Pull bpf updates from Alexei Starovoitov: "Major changes: - Recover from BPF arena page faults using a scratch page and add ptep_try_set() for lockless empty-slot installs on x86 and arm64. This allows BPF kfuncs to access arena pointers directly. The 'arena_direct_access' stable branch was created for this work and was pulled into sched-ext and bpf-next trees (Tejun Heo, Kumar Kartikeya Dwivedi) - Lift old restriction and support 6+ arguments in BPF programs and kfuncs on x86 and arm64 (Yonghong Song, Puranjay Mohan) Other features and fixes: - Add 24-bit BTF vlen and reclaim unused bits in the BTF UAPI to ease addition of new BTF kinds (Alan Maguire) - Raise the maximum BPF call chain depth from 8 to 16 frames (Alexei Starovoitov) - Refactor object relationship tracking in the verifier and fix a dynptr use-after-free bug (Amery Hung) - Harden the signed program loader and reject exclusive maps as inner maps (Daniel Borkmann) - Replace the verifier min/max bounds fields with a circular number (cnum) representation and improve 32->64 bit range refinements (Eduard Zingerman) - Introduce the arena library and runtime (libarena) with a buddy allocator, rbtree and SPMC queue data structures, ASAN support and a parallel test harness. Allow subprograms to return arena pointers and switch to a BTF type-tag based __arena annotation (Emil Tsalapatis) - Cache build IDs in the sleepable stackmap path and avoid faultable build ID reads under mm locks (Ihor Solodrai) - Introduce the tracing_multi link to attach a single BPF program to many kernel functions at once. Allow specifying the uprobe_multi target via FD (Jiri Olsa) - Extend the bpf_list family of kfuncs with bpf_list_add/del(), and bpf_list_is_first/is_last/empty() (Kaitao Cheng) - Extend the BPF syscall with common attributes support for prog_load, btf_load and map_create (Leon Hwang) - Wrap rhashtable as BPF map (Mykyta Yatsenko, Herbert Xu) - Add sleepable support for tracepoint programs and fix deadlocks in LRU map due to NMI reentry (Mykyta Yatsenko) - Fix OOB access in bpf_flow_keys, fix nullness analysis of inner arrays, enforce write checks for global subprograms (Nuoqi Gui) - Report the maximum combined stack depth and print a breakdown of instructions processed per subprogram (Paul Chaignon) - Add an XDP load-balancer benchmark and arm64 JIT support for stack arguments (Puranjay Mohan) - Add kfuncs to traverse over wakeup_sources (Samuel Wu) - Allow sleepable BPF programs to use LPM trie maps directly (Vlad Poenaru) - Many more fixes and cleanups across the verifier, BTF, sockmap, devmap, bpffs, security hooks, s390/riscv/loongarch JITs, rqspinlock, libbpf, bpftool, selftests" * tag 'bpf-next-7.2' of git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next: (336 commits) selftests/bpf: Work around llvm stack overflow in crypto progs selftests/bpf: add test for bpf_msg_pop_data() overflow bpf, sockmap: fix integer overflow in bpf_msg_pop_data() bounds check sockmap: Fix use-after-free in udp_bpf_recvmsg() bpf, sockmap: keep sk_msg copy state in sync bpf, sockmap: Fix wrong rsge offset in bpf_msg_push_data() bpf, sockmap: reject overflowing copy + len in bpf_msg_push_data() selftsets/bpf: Retry map update on helper_fill_hashmap() selftests/bpf: Add test for sleepable lsm_cgroup rejection selftests/bpf: Add test to verify the fix for bpf_setsockopt() helper bpf: Fix bpf_get/setsockopt to tos for ipv4-mapped ipv6 socket selftests/bpf: Avoid static LLVM linking for cross builds selftests/bpf: Use common CFLAGS for urandom_read selftests/bpf: Initialize operation name before use tools/bpf: build: Append extra cflags libbpf: Initialize CFLAGS before including Makefile.include bpftool: Append extra host flags bpftool: Avoid adding EXTRA_CFLAGS to HOST_CFLAGS bpftool: Pass host flags to bootstrap libbpf selftests/bpf: correct CONFIG_PPC64 macro name in comment ...
Unified · Split
| @@ -97,10 +97,8 @@ Each type contains the following common data:: · 96 unmodified lines | |||
| 97 | struct btf_type { | 97 | struct btf_type { |
| 98 | __u32 name_off; | 98 | __u32 name_off; |
| 99 | /* "info" bits arrangement | 99 | /* "info" bits arrangement |
| 100 | * bits 0-15: vlen (e.g. # of struct's members) | 100 | * bits 0-23: vlen (e.g. # of struct's members) |
| 101 | * bits 16-23: unused | 101 | * bits 24-30: kind (e.g. int, ptr, array...etc) |
| 102 | * bits 24-28: kind (e.g. int, ptr, array...etc) | ||
| 103 | * bits 29-30: unused | ||
| 104 | * bit 31: kind_flag, currently used by | 102 | * bit 31: kind_flag, currently used by |
| 105 | * struct, union, enum, fwd, enum64, | 103 | * struct, union, enum, fwd, enum64, |
| 106 | * decl_tag and type_tag | 104 | * decl_tag and type_tag |
| @@ -207,8 +207,26 @@ Here, the buffer may be NULL. If the buffer is not NULL, it must be at least · 206 unmodified lines | |||
| 207 | buffer__szk bytes in size. The kfunc is responsible for checking if the buffer | 207 | buffer__szk bytes in size. The kfunc is responsible for checking if the buffer |
| 208 | is NULL before using it. | 208 | is NULL before using it. |
| 209 | 209 | ||
| 210 | 2.3.5 __str Annotation | 210 | 2.3.5 __nonown_allowed Annotation |
| 211 | ---------------------------- | 211 | --------------------------------- |
| 212 | |||
| 213 | This annotation is used to indicate that the parameter may be a non-owning reference. | ||
| 214 | |||
| 215 | An example is given below:: | ||
| 216 | |||
| 217 | __bpf_kfunc int bpf_list_add(..., struct bpf_list_node | ||
| 218 | *prev__nonown_allowed, ...) | ||
| 219 | { | ||
| 220 | ... | ||
| 221 | } | ||
| 222 | |||
| 223 | For the ``prev__nonown_allowed`` parameter (resolved as ``KF_ARG_PTR_TO_LIST_NODE``), | ||
| 224 | suffix ``__nonown_allowed`` retains the usual owning-pointer rules and also | ||
| 225 | permits a non-owning reference with no ref_obj_id (e.g. the return value of | ||
| 226 | bpf_list_front() / bpf_list_back()). | ||
| 227 | |||
| 228 | 2.3.6 __str Annotation | ||
| 229 | ---------------------- | ||
| 212 | This annotation is used to indicate that the argument is a constant string. | 230 | This annotation is used to indicate that the argument is a constant string. |
| 213 | 231 | ||
| 214 | An example is given below:: | 232 | An example is given below:: |
| @@ -462,6 +480,20 @@ In order to accommodate such requirements, the verifier will enforce strict · 247 unmodified lines | |||
| 462 | PTR_TO_BTF_ID type matching if two types have the exact same name, with one | 480 | PTR_TO_BTF_ID type matching if two types have the exact same name, with one |
| 463 | being suffixed with ``___init``. | 481 | being suffixed with ``___init``. |
| 464 | 482 | ||
| 483 | 2.8 Accessing arena memory through kfunc arguments | ||
| 484 | -------------------------------------------------- | ||
| 485 | |||
| 486 | A read or write at any address inside an arena does not oops the kernel. | ||
| 487 | Unallocated arena pages are lazily backed by a scratch page and the | ||
| 488 | access is reported through the program's BPF stream as an error. Only | ||
| 489 | the BPF program's correctness is affected; the kernel itself remains | ||
| 490 | intact. | ||
| 491 | |||
| 492 | The arena is followed by a ``GUARD_SZ / 2`` (32 KiB) guard region that | ||
| 493 | is also covered by this recovery. A kfunc handed an arena pointer may | ||
| 494 | therefore access up to ``GUARD_SZ / 2`` past it without bounds-checking | ||
| 495 | against the arena. Larger accesses must verify the range explicitly. | ||
| 496 | |||
| 465 | .. _BPF_kfunc_lifecycle_expectations: | 497 | .. _BPF_kfunc_lifecycle_expectations: |
| 466 | 498 | ||
| 467 | 3. kfunc lifecycle expectations | 499 | 3. kfunc lifecycle expectations |
| @@ -21,10 +21,18 @@ digraph { · 20 unmodified lines | |||
| 21 | // names that initiate the corresponding logic in kernel/bpf/bpf_lru_list.c. | 21 | // names that initiate the corresponding logic in kernel/bpf/bpf_lru_list.c. |
| 22 | // Number suffixes and errno suffixes handle subsections of the corresponding | 22 | // Number suffixes and errno suffixes handle subsections of the corresponding |
| 23 | // logic in the function as of the writing of this dot. | 23 | // logic in the function as of the writing of this dot. |
| 24 | // | ||
| 25 | // All LRU locks are rqspinlock_t. Every acquire can fail (AA self-deadlock | ||
| 26 | // or contention timeout); on failure the corresponding helper returns NULL | ||
| 27 | // and the caller propagates -ENOMEM. The "rqspinlock acquire failed" | ||
| 28 | // terminal below is reached via the dashed arrows from each acquire site. | ||
| 29 | |||
| 30 | rqspinlock_failed [shape=rectangle, | ||
| 31 | label="Any LRU rqspinlock\nacquire fails\n(AA or timeout)"] | ||
| 24 | 32 | ||
| 25 | // cf. __local_list_pop_free() / bpf_percpu_lru_pop_free() | 33 | // cf. __local_list_pop_free() / bpf_percpu_lru_pop_free() |
| 26 | local_freelist_check [shape=diamond,fillcolor=1, | 34 | local_freelist_check [shape=diamond,fillcolor=1, |
| 27 | label="Local freelist\nnode available?"]; | 35 | label="Local freelist\nnode available?\n(lockless free_llist)"]; |
| 28 | use_local_node [shape=rectangle, | 36 | use_local_node [shape=rectangle, |
| 29 | label="Use node owned\nby this CPU"] | 37 | label="Use node owned\nby this CPU"] |
| 30 | 38 | ||
| @@ -82,6 +90,15 @@ digraph { · 51 unmodified lines | |||
| 82 | // fn__local_list_pop_pending() | 90 | // fn__local_list_pop_pending() |
| 83 | } | 91 | } |
| 84 | 92 | ||
| 93 | // Post-steal: re-acquire local loc_l->lock to insert the stolen node into | ||
| 94 | // the local pending list. If the acquire fails, the stolen node is published | ||
| 95 | // to the lockless local free_llist so the next pop on this CPU picks it up | ||
| 96 | // instead of orphaning it. | ||
| 97 | post_steal_lock [shape=diamond,fillcolor=1, | ||
| 98 | label="Acquire local\nloc_l->lock\nto add pending"] | ||
| 99 | post_steal_to_free_llist [shape=rectangle, | ||
| 100 | label="Publish stolen node to\nlocal free_llist (lockless)"] | ||
| 101 | |||
| 85 | fn_bpf_lru_list_pop_free_to_local2 [shape=rectangle, | 102 | fn_bpf_lru_list_pop_free_to_local2 [shape=rectangle, |
| 86 | label="Use node that was\nnot recently referenced"] | 103 | label="Use node that was\nnot recently referenced"] |
| 87 | local_freelist_check4 [shape=rectangle, | 104 | local_freelist_check4 [shape=rectangle, |
| @@ -97,10 +114,19 @@ digraph { · 9 unmodified lines | |||
| 97 | fn_htab_lru_map_update_elem_ENOENT [shape=oval,label="return -ENOENT"] | 114 | fn_htab_lru_map_update_elem_ENOENT [shape=oval,label="return -ENOENT"] |
| 98 | 115 | ||
| 99 | begin -> local_freelist_check | 116 | begin -> local_freelist_check |
| 117 | // The initial per-CPU lock (loc_l->lock for common, l->lock for percpu) is | ||
| 118 | // acquired before the local freelist check; rqspinlock failure here exits | ||
| 119 | // directly to -ENOMEM (no recovery needed: nothing was removed yet). | ||
| 120 | local_freelist_check -> rqspinlock_failed [style=dashed, | ||
| 121 | xlabel="acquire fails"] | ||
| 100 | local_freelist_check -> use_local_node [xlabel="Y"] | 122 | local_freelist_check -> use_local_node [xlabel="Y"] |
| 101 | local_freelist_check -> common_lru_check [xlabel="N"] | 123 | local_freelist_check -> common_lru_check [xlabel="N"] |
| 102 | common_lru_check -> fn_bpf_lru_list_pop_free_to_local [xlabel="Y"] | 124 | common_lru_check -> fn_bpf_lru_list_pop_free_to_local [xlabel="Y"] |
| 103 | common_lru_check -> fn___bpf_lru_list_shrink_inactive [xlabel="N"] | 125 | common_lru_check -> fn___bpf_lru_list_shrink_inactive [xlabel="N"] |
| 126 | // Global lru_list lock acquire failure in pop_free_to_local: skip refill, | ||
| 127 | // fall through to the steal path. Not ENOMEM by itself. | ||
| 128 | fn_bpf_lru_list_pop_free_to_local -> common_lru_check2 [style=dashed, | ||
| 129 | xlabel="global lru_lock\nacquire fails"] | ||
| 104 | fn_bpf_lru_list_pop_free_to_local -> fn___bpf_lru_node_move_to_free | 130 | fn_bpf_lru_list_pop_free_to_local -> fn___bpf_lru_node_move_to_free |
| 105 | fn___bpf_lru_node_move_to_free -> | 131 | fn___bpf_lru_node_move_to_free -> |
| 106 | fn_bpf_lru_list_pop_free_to_local2 [xlabel="Y"] | 132 | fn_bpf_lru_list_pop_free_to_local2 [xlabel="Y"] |
| @@ -120,13 +146,27 @@ digraph { · 13 unmodified lines | |||
| 120 | local_freelist_check6 -> local_freelist_check7 | 146 | local_freelist_check6 -> local_freelist_check7 |
| 121 | local_freelist_check7 -> fn_htab_lru_map_update_elem | 147 | local_freelist_check7 -> fn_htab_lru_map_update_elem |
| 122 | 148 | ||
| 123 | fn_htab_lru_map_update_elem -> fn_htab_lru_map_update_elem3 [xlabel = "Y"] | 149 | // Steal-loop victim lock failure is silent: treat as "no node found here" |
| 150 | // and continue to next CPU; same edge as the existing "N" path. | ||
| 151 | local_freelist_check5 -> fn_htab_lru_map_update_elem2 [style=dashed, | ||
| 152 | xlabel="victim's lock\nfails: skip"] | ||
| 153 | // After a successful steal, re-acquire the local loc_l->lock. On failure | ||
| 154 | // the stolen node is published to free_llist (recovered, not orphaned) | ||
| 155 | // and the update returns -ENOMEM. | ||
| 156 | fn_htab_lru_map_update_elem -> post_steal_lock [xlabel = "Y"] | ||
| 157 | post_steal_lock -> fn_htab_lru_map_update_elem3 [xlabel = "OK"] | ||
| 158 | post_steal_lock -> post_steal_to_free_llist [style=dashed, | ||
| 159 | xlabel="loc_l->lock\nacquire fails"] | ||
| 160 | post_steal_to_free_llist -> fn_htab_lru_map_update_elem_ENOMEM | ||
| 124 | fn_htab_lru_map_update_elem -> fn_htab_lru_map_update_elem2 [xlabel = "N"] | 161 | fn_htab_lru_map_update_elem -> fn_htab_lru_map_update_elem2 [xlabel = "N"] |
| 125 | fn_htab_lru_map_update_elem2 -> | 162 | fn_htab_lru_map_update_elem2 -> |
| 126 | fn_htab_lru_map_update_elem_ENOMEM [xlabel = "Y"] | 163 | fn_htab_lru_map_update_elem_ENOMEM [xlabel = "Y"] |
| 127 | fn_htab_lru_map_update_elem2 -> local_freelist_check5 [xlabel = "N"] | 164 | fn_htab_lru_map_update_elem2 -> local_freelist_check5 [xlabel = "N"] |
| 128 | fn_htab_lru_map_update_elem3 -> fn_htab_lru_map_update_elem4 | 165 | fn_htab_lru_map_update_elem3 -> fn_htab_lru_map_update_elem4 |
| 129 | 166 | ||
| 167 | // Shared rqspinlock-failure terminal collapses to the same -ENOMEM exit. | ||
| 168 | rqspinlock_failed -> fn_htab_lru_map_update_elem_ENOMEM | ||
| 169 | |||
| 130 | use_local_node -> fn_htab_lru_map_update_elem4 | 170 | use_local_node -> fn_htab_lru_map_update_elem4 |
| 131 | fn_bpf_lru_list_pop_free_to_local2 -> fn_htab_lru_map_update_elem4 | 171 | fn_bpf_lru_list_pop_free_to_local2 -> fn_htab_lru_map_update_elem4 |
| 132 | local_freelist_check4 -> fn_htab_lru_map_update_elem4 | 172 | local_freelist_check4 -> fn_htab_lru_map_update_elem4 |
| @@ -4797,13 +4797,6 @@ S: Supported · 4796 unmodified lines | |||
| 4797 | F: arch/x86/net/ | 4797 | F: arch/x86/net/ |
| 4798 | X: arch/x86/net/bpf_jit_comp32.c | 4798 | X: arch/x86/net/bpf_jit_comp32.c |
| 4799 | 4799 | ||
| 4800 | BPF [BTF] | ||
| 4801 | M: Martin KaFai Lau <martin.lau@linux.dev> | ||
| 4802 | L: bpf@vger.kernel.org | ||
| 4803 | S: Maintained | ||
| 4804 | F: include/linux/btf* | ||
| 4805 | F: kernel/bpf/btf.c | ||
| 4806 | |||
| 4807 | BPF [CORE] | 4800 | BPF [CORE] |
| 4808 | M: Alexei Starovoitov <ast@kernel.org> | 4801 | M: Alexei Starovoitov <ast@kernel.org> |
| 4809 | M: Daniel Borkmann <daniel@iogearbox.net> | 4802 | M: Daniel Borkmann <daniel@iogearbox.net> |
| @@ -4840,12 +4833,13 @@ BPF [GENERAL] (Safe Dynamic Programs and Tools) · 30 unmodified lines | |||
| 4840 | M: Alexei Starovoitov <ast@kernel.org> | 4833 | M: Alexei Starovoitov <ast@kernel.org> |
| 4841 | M: Daniel Borkmann <daniel@iogearbox.net> | 4834 | M: Daniel Borkmann <daniel@iogearbox.net> |
| 4842 | M: Andrii Nakryiko <andrii@kernel.org> | 4835 | M: Andrii Nakryiko <andrii@kernel.org> |
| 4843 | R: Martin KaFai Lau <martin.lau@linux.dev> | ||
| 4844 | M: Eduard Zingerman <eddyz87@gmail.com> | 4836 | M: Eduard Zingerman <eddyz87@gmail.com> |
| 4845 | M: Kumar Kartikeya Dwivedi <memxor@gmail.com> | 4837 | M: Kumar Kartikeya Dwivedi <memxor@gmail.com> |
| 4838 | R: Martin KaFai Lau <martin.lau@linux.dev> | ||
| 4846 | R: Song Liu <song@kernel.org> | 4839 | R: Song Liu <song@kernel.org> |
| 4847 | R: Yonghong Song <yonghong.song@linux.dev> | 4840 | R: Yonghong Song <yonghong.song@linux.dev> |
| 4848 | R: Jiri Olsa <jolsa@kernel.org> | 4841 | R: Jiri Olsa <jolsa@kernel.org> |
| 4842 | R: Emil Tsalapatis <emil@etsalapatis.com> | ||
| 4849 | L: bpf@vger.kernel.org | 4843 | L: bpf@vger.kernel.org |
| 4850 | S: Supported | 4844 | S: Supported |
| 4851 | W: https://bpf.io/ | 4845 | W: https://bpf.io/ |
| @@ -4855,7 +4849,9 @@ T: git git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git · 3 unmodified lines | |||
| 4855 | F: Documentation/bpf/ | 4849 | F: Documentation/bpf/ |
| 4856 | F: Documentation/networking/filter.rst | 4850 | F: Documentation/networking/filter.rst |
| 4857 | F: Documentation/userspace-api/ebpf/ | 4851 | F: Documentation/userspace-api/ebpf/ |
| 4852 | F: arch/*/include/asm/rqspinlock.h | ||
| 4858 | F: arch/*/net/* | 4853 | F: arch/*/net/* |
| 4854 | F: include/asm-generic/rqspinlock.h | ||
| 4859 | F: include/linux/bpf* | 4855 | F: include/linux/bpf* |
| 4860 | F: include/linux/btf* | 4856 | F: include/linux/btf* |
| 4861 | F: include/linux/buildid.h | 4857 | F: include/linux/buildid.h |
| @@ -4867,17 +4863,15 @@ F: include/uapi/linux/filter.h · 5 unmodified lines | |||
| 4867 | F: kernel/bpf/ | 4863 | F: kernel/bpf/ |
| 4868 | F: kernel/trace/bpf_trace.c | 4864 | F: kernel/trace/bpf_trace.c |
| 4869 | F: lib/buildid.c | 4865 | F: lib/buildid.c |
| 4870 | F: arch/*/include/asm/rqspinlock.h | ||
| 4871 | F: include/asm-generic/rqspinlock.h | ||
| 4872 | F: lib/test_bpf.c | 4866 | F: lib/test_bpf.c |
| 4873 | F: net/bpf/ | 4867 | F: net/bpf/ |
| 4874 | F: net/core/filter.c | 4868 | F: net/core/filter.c |
| 4875 | F: net/sched/act_bpf.c | 4869 | F: net/sched/act_bpf.c |
| 4876 | F: net/sched/cls_bpf.c | 4870 | F: net/sched/cls_bpf.c |
| 4877 | F: samples/bpf/ | 4871 | F: samples/bpf/ |
| 4872 | F: scripts/Makefile.btf | ||
| 4878 | F: scripts/bpf_doc.py | 4873 | F: scripts/bpf_doc.py |
| 4879 | F: scripts/gen-btf.sh | 4874 | F: scripts/gen-btf.sh |
| 4880 | F: scripts/Makefile.btf | ||
| 4881 | F: scripts/pahole-version.sh | 4875 | F: scripts/pahole-version.sh |
| 4882 | F: tools/bpf/ | 4876 | F: tools/bpf/ |
| 4883 | F: tools/lib/bpf/ | 4877 | F: tools/lib/bpf/ |
| @@ -4892,6 +4886,7 @@ F: kernel/bpf/*iter.c · 8 unmodified lines | |||
| 4892 | BPF [L7 FRAMEWORK] (sockmap) | 4886 | BPF [L7 FRAMEWORK] (sockmap) |
| 4893 | M: John Fastabend <john.fastabend@gmail.com> | 4887 | M: John Fastabend <john.fastabend@gmail.com> |
| 4894 | M: Jakub Sitnicki <jakub@cloudflare.com> | 4888 | M: Jakub Sitnicki <jakub@cloudflare.com> |
| 4889 | M: Jiayuan Chen <jiayuan.chen@linux.dev> | ||
| 4895 | L: netdev@vger.kernel.org | 4890 | L: netdev@vger.kernel.org |
| 4896 | L: bpf@vger.kernel.org | 4891 | L: bpf@vger.kernel.org |
| 4897 | S: Maintained | 4892 | S: Maintained |
| @@ -4935,18 +4930,11 @@ F: tools/testing/selftests/bpf/prog_tests/tc_netkit.c · 37 unmodified lines | |||
| 4935 | F: tools/testing/selftests/drivers/net/hw/nk_qlease.py | 4930 | F: tools/testing/selftests/drivers/net/hw/nk_qlease.py |
| 4936 | F: tools/testing/selftests/net/nk_qlease.py | 4931 | F: tools/testing/selftests/net/nk_qlease.py |
| 4937 | 4932 | ||
| 4938 | BPF [NETWORKING] (struct_ops, reuseport) | ||
| 4939 | M: Martin KaFai Lau <martin.lau@linux.dev> | ||
| 4940 | L: bpf@vger.kernel.org | ||
| 4941 | L: netdev@vger.kernel.org | ||
| 4942 | S: Maintained | ||
| 4943 | F: kernel/bpf/bpf_struct* | ||
| 4944 | |||
| 4945 | BPF [NETWORKING] (tcx & tc BPF, sock_addr) | 4933 | BPF [NETWORKING] (tcx & tc BPF, sock_addr) |
| 4946 | M: Martin KaFai Lau <martin.lau@linux.dev> | ||
| 4947 | M: Daniel Borkmann <daniel@iogearbox.net> | 4934 | M: Daniel Borkmann <daniel@iogearbox.net> |
| 4948 | R: John Fastabend <john.fastabend@gmail.com> | 4935 | R: John Fastabend <john.fastabend@gmail.com> |
| 4949 | R: Stanislav Fomichev <sdf@fomichev.me> | 4936 | R: Stanislav Fomichev <sdf@fomichev.me> |
| 4937 | R: Martin KaFai Lau <martin.lau@linux.dev> | ||
| 4950 | L: bpf@vger.kernel.org | 4938 | L: bpf@vger.kernel.org |
| 4951 | L: netdev@vger.kernel.org | 4939 | L: netdev@vger.kernel.org |
| 4952 | S: Maintained | 4940 | S: Maintained |
| @@ -4981,14 +4969,6 @@ L: bpf@vger.kernel.org · 28 unmodified lines | |||
| 4981 | S: Maintained | 4969 | S: Maintained |
| 4982 | F: tools/testing/selftests/bpf/ | 4970 | F: tools/testing/selftests/bpf/ |
| 4983 | 4971 | ||
| 4984 | BPF [STORAGE & CGROUPS] | ||
| 4985 | M: Martin KaFai Lau <martin.lau@linux.dev> | ||
| 4986 | L: bpf@vger.kernel.org | ||
| 4987 | S: Maintained | ||
| 4988 | F: kernel/bpf/*storage.c | ||
| 4989 | F: kernel/bpf/bpf_lru* | ||
| 4990 | F: kernel/bpf/cgroup.c | ||
| 4991 | |||
| 4992 | BPF [TOOLING] (bpftool) | 4972 | BPF [TOOLING] (bpftool) |
| 4993 | M: Quentin Monnet <qmo@kernel.org> | 4973 | M: Quentin Monnet <qmo@kernel.org> |
| 4994 | L: bpf@vger.kernel.org | 4974 | L: bpf@vger.kernel.org |
| @@ -1830,6 +1830,34 @@ static inline pte_t ptep_get_and_clear(struct mm_struct *mm, · 1829 unmodified lines | |||
| 1830 | return __ptep_get_and_clear(mm, addr, ptep); | 1830 | return __ptep_get_and_clear(mm, addr, ptep); |
| 1831 | } | 1831 | } |
| 1832 | 1832 | ||
| 1833 | static inline bool ptep_try_set(pte_t *ptep, pte_t new_pte) | ||
| 1834 | { | ||
| 1835 | pteval_t old = 0; | ||
| 1836 | |||
| 1837 | if (!try_cmpxchg(&pte_val(*ptep), &old, pte_val(new_pte))) | ||
| 1838 | return false; | ||
| 1839 | |||
| 1840 | /* | ||
| 1841 | * The store must be complete by the time this returns, but the caller | ||
| 1842 | * may be in lazy MMU mode, where __set_pte_complete() would defer the | ||
| 1843 | * barriers. Issue them directly. | ||
| 1844 | */ | ||
| 1845 | emit_pte_barriers(); | ||
| 1846 | return true; | ||
| 1847 | } | ||
| 1848 | #define ptep_try_set ptep_try_set | ||
| 1849 | |||
| 1850 | /* | ||
| 1851 | * arm64 mandates break-before-make: a cleared kernel PTE must have its TLB | ||
| 1852 | * invalidated before a different page is installed in its place. The broadcast | ||
| 1853 | * TLBI is an instruction, not an IPI, so this is safe with interrupts disabled. | ||
| 1854 | */ | ||
| 1855 | static inline void flush_tlb_before_set(unsigned long addr) | ||
| 1856 | { | ||
| 1857 | flush_tlb_kernel_range(addr, addr + PAGE_SIZE); | ||
| 1858 | } | ||
| 1859 | #define flush_tlb_before_set flush_tlb_before_set | ||
| 1860 | |||
| 1833 | #define test_and_clear_young_ptes test_and_clear_young_ptes | 1861 | #define test_and_clear_young_ptes test_and_clear_young_ptes |
| 1834 | static inline bool test_and_clear_young_ptes(struct vm_area_struct *vma, | 1862 | static inline bool test_and_clear_young_ptes(struct vm_area_struct *vma, |
| 1835 | unsigned long addr, pte_t *ptep, unsigned int nr) | 1863 | unsigned long addr, pte_t *ptep, unsigned int nr) |
| @@ -9,6 +9,7 @@ · 8 unmodified lines | |||
| 9 | 9 | ||
| 10 | #include <linux/acpi.h> | 10 | #include <linux/acpi.h> |
| 11 | #include <linux/bitfield.h> | 11 | #include <linux/bitfield.h> |
| 12 | #include <linux/bpf_defs.h> | ||
| 12 | #include <linux/extable.h> | 13 | #include <linux/extable.h> |
| 13 | #include <linux/kfence.h> | 14 | #include <linux/kfence.h> |
| 14 | #include <linux/signal.h> | 15 | #include <linux/signal.h> |
| @@ -436,9 +437,12 @@ static void __do_kernel_fault(unsigned long addr, unsigned long esr, · 421 unmodified lines | |||
| 436 | } else if (is_pkvm_stage2_abort(esr)) { | 437 | } else if (is_pkvm_stage2_abort(esr)) { |
| 437 | msg = "access to hypervisor-protected memory"; | 438 | msg = "access to hypervisor-protected memory"; |
| 438 | } else { | 439 | } else { |
| 439 | if (esr_fsc_is_translation_fault(esr) && | 440 | if (esr_fsc_is_translation_fault(esr)) { |
| 440 | kfence_handle_page_fault(addr, esr & ESR_ELx_WNR, regs)) | 441 | if (kfence_handle_page_fault(addr, esr & ESR_ELx_WNR, regs)) |
| 441 | return; | 442 | return; |
| 443 | if (bpf_arena_handle_page_fault(addr, esr & ESR_ELx_WNR, regs->pc)) | ||
| 444 | return; | ||
| 445 | } | ||
| 442 | 446 | ||
| 443 | msg = "paging request"; | 447 | msg = "paging request"; |
| 444 | } | 448 | } |
| @@ -47,7 +47,7 @@ · 46 unmodified lines | |||
| 47 | /* Map BPF registers to A64 registers */ | 47 | /* Map BPF registers to A64 registers */ |
| 48 | static const int bpf2a64[] = { | 48 | static const int bpf2a64[] = { |
| 49 | /* return value from in-kernel function, and exit value from eBPF */ | 49 | /* return value from in-kernel function, and exit value from eBPF */ |
| 50 | [BPF_REG_0] = A64_R(7), | 50 | [BPF_REG_0] = A64_R(8), |
| 51 | /* arguments from eBPF program to in-kernel function */ | 51 | /* arguments from eBPF program to in-kernel function */ |
| 52 | [BPF_REG_1] = A64_R(0), | 52 | [BPF_REG_1] = A64_R(0), |
| 53 | [BPF_REG_2] = A64_R(1), | 53 | [BPF_REG_2] = A64_R(1), |
| @@ -86,6 +86,7 @@ struct jit_ctx { · 32 unmodified lines | |||
| 86 | __le32 *image; | 86 | __le32 *image; |
| 87 | __le32 *ro_image; | 87 | __le32 *ro_image; |
| 88 | u32 stack_size; | 88 | u32 stack_size; |
| 89 | u16 stack_arg_size; | ||
| 89 | u64 user_vm_start; | 90 | u64 user_vm_start; |
| 90 | u64 arena_vm_start; | 91 | u64 arena_vm_start; |
| 91 | bool fp_used; | 92 | bool fp_used; |
| @@ -533,13 +534,20 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf) · 441 unmodified lines | |||
| 533 | * | | | 534 | * | | |
| 534 | * +-----+ <= (BPF_FP - prog->aux->stack_depth) | 535 | * +-----+ <= (BPF_FP - prog->aux->stack_depth) |
| 535 | * |RSVD | padding | 536 | * |RSVD | padding |
| 536 | * current A64_SP => +-----+ <= (BPF_FP - ctx->stack_size) | 537 | * +-----+ <= (BPF_FP - ctx->stack_size) |
| 538 | * | | | ||
| 539 | * | ... | outgoing stack args (9+, if any) | ||
| 540 | * | | | ||
| 541 | * current A64_SP => +-----+ | ||
| 537 | * | | | 542 | * | | |
| 538 | * | ... | Function call stack | 543 | * | ... | Function call stack |
| 539 | * | | | 544 | * | | |
| 540 | * +-----+ | 545 | * +-----+ |
| 541 | * low | 546 | * low |
| 542 | * | 547 | * |
| 548 | * Stack args 6-8 are passed in x5-x7, args 9+ at [SP]. | ||
| 549 | * Incoming args 9+ are at [A64_FP + 16], [A64_FP + 24], ... | ||
| 550 | * (above the saved FP/LR pair pushed in the callee prologue). | ||
| 543 | */ | 551 | */ |
| 544 | 552 | ||
| 545 | emit_kcfi(is_main_prog ? cfi_bpf_hash : cfi_bpf_subprog_hash, ctx); | 553 | emit_kcfi(is_main_prog ? cfi_bpf_hash : cfi_bpf_subprog_hash, ctx); |
| @@ -613,6 +621,9 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf) · 67 unmodified lines | |||
| 613 | if (ctx->stack_size && !ctx->priv_sp_used) | 621 | if (ctx->stack_size && !ctx->priv_sp_used) |
| 614 | emit(A64_SUB_I(1, A64_SP, A64_SP, ctx->stack_size), ctx); | 622 | emit(A64_SUB_I(1, A64_SP, A64_SP, ctx->stack_size), ctx); |
| 615 | 623 | ||
| 624 | if (ctx->stack_arg_size) | ||
| 625 | emit(A64_SUB_I(1, A64_SP, A64_SP, ctx->stack_arg_size), ctx); | ||
| 626 | |||
| 616 | if (ctx->arena_vm_start) | 627 | if (ctx->arena_vm_start) |
| 617 | emit_a64_mov_i64(arena_vm_base, ctx->arena_vm_start, ctx); | 628 | emit_a64_mov_i64(arena_vm_base, ctx->arena_vm_start, ctx); |
| 618 | 629 | ||
| @@ -673,6 +684,9 @@ static int emit_bpf_tail_call(struct jit_ctx *ctx) · 54 unmodified lines | |||
| 673 | /* Update tail_call_cnt if the slot is populated. */ | 684 | /* Update tail_call_cnt if the slot is populated. */ |
| 674 | emit(A64_STR64I(tcc, ptr, 0), ctx); | 685 | emit(A64_STR64I(tcc, ptr, 0), ctx); |
| 675 | 686 | ||
| 687 | if (ctx->stack_arg_size) | ||
| 688 | emit(A64_ADD_I(1, A64_SP, A64_SP, ctx->stack_arg_size), ctx); | ||
| 689 | |||
| 676 | /* restore SP */ | 690 | /* restore SP */ |
| 677 | if (ctx->stack_size && !ctx->priv_sp_used) | 691 | if (ctx->stack_size && !ctx->priv_sp_used) |
| 678 | emit(A64_ADD_I(1, A64_SP, A64_SP, ctx->stack_size), ctx); | 692 | emit(A64_ADD_I(1, A64_SP, A64_SP, ctx->stack_size), ctx); |
| @@ -1034,6 +1048,9 @@ static void build_epilogue(struct jit_ctx *ctx, bool was_classic) · 355 unmodified lines | |||
| 1034 | const u8 r0 = bpf2a64[BPF_REG_0]; | 1048 | const u8 r0 = bpf2a64[BPF_REG_0]; |
| 1035 | const u8 ptr = bpf2a64[TCCNT_PTR]; | 1049 | const u8 ptr = bpf2a64[TCCNT_PTR]; |
| 1036 | 1050 | ||
| 1051 | if (ctx->stack_arg_size) | ||
| 1052 | emit(A64_ADD_I(1, A64_SP, A64_SP, ctx->stack_arg_size), ctx); | ||
| 1053 | |||
| 1037 | /* We're done with BPF stack */ | 1054 | /* We're done with BPF stack */ |
| 1038 | if (ctx->stack_size && !ctx->priv_sp_used) | 1055 | if (ctx->stack_size && !ctx->priv_sp_used) |
| 1039 | emit(A64_ADD_I(1, A64_SP, A64_SP, ctx->stack_size), ctx); | 1056 | emit(A64_ADD_I(1, A64_SP, A64_SP, ctx->stack_size), ctx); |
| @@ -1048,7 +1065,7 @@ static void build_epilogue(struct jit_ctx *ctx, bool was_classic) · 8 unmodified lines | |||
| 1048 | /* Restore FP/LR registers */ | 1065 | /* Restore FP/LR registers */ |
| 1049 | emit(A64_POP(A64_FP, A64_LR, A64_SP), ctx); | 1066 | emit(A64_POP(A64_FP, A64_LR, A64_SP), ctx); |
| 1050 | 1067 | ||
| 1051 | /* Move the return value from bpf:r0 (aka x7) to x0 */ | 1068 | /* Move the return value from bpf:r0 (aka x8) to x0 */ |
| 1052 | emit(A64_MOV(1, A64_R(0), r0), ctx); | 1069 | emit(A64_MOV(1, A64_R(0), r0), ctx); |
| 1053 | 1070 | ||
| 1054 | /* Authenticate lr */ | 1071 | /* Authenticate lr */ |
| @@ -1191,6 +1208,42 @@ static int add_exception_handler(const struct bpf_insn *insn, · 136 unmodified lines | |||
| 1191 | return 0; | 1208 | return 0; |
| 1192 | } | 1209 | } |
| 1193 | 1210 | ||
| 1211 | static const u8 stack_arg_reg[] = { A64_R(5), A64_R(6), A64_R(7) }; | ||
| 1212 | |||
| 1213 | #define NR_STACK_ARG_REGS ARRAY_SIZE(stack_arg_reg) | ||
| 1214 | |||
| 1215 | static void emit_stack_arg_load(u8 dst, s16 bpf_off, struct jit_ctx *ctx) | ||
| 1216 | { | ||
| 1217 | int idx = bpf_off / sizeof(u64) - 1; | ||
| 1218 | |||
| 1219 | if (idx < NR_STACK_ARG_REGS) | ||
| 1220 | emit(A64_MOV(1, dst, stack_arg_reg[idx]), ctx); | ||
| 1221 | else | ||
| 1222 | emit(A64_LDR64I(dst, A64_FP, (idx - NR_STACK_ARG_REGS) * sizeof(u64) + 16), ctx); | ||
| 1223 | } | ||
| 1224 | |||
| 1225 | static void emit_stack_arg_store(u8 src_a64, s16 bpf_off, struct jit_ctx *ctx) | ||
| 1226 | { | ||
| 1227 | int idx = -bpf_off / sizeof(u64) - 1; | ||
| 1228 | |||
| 1229 | if (idx < NR_STACK_ARG_REGS) | ||
| 1230 | emit(A64_MOV(1, stack_arg_reg[idx], src_a64), ctx); | ||
| 1231 | else | ||
| 1232 | emit(A64_STR64I(src_a64, A64_SP, (idx - NR_STACK_ARG_REGS) * sizeof(u64)), ctx); | ||
| 1233 | } | ||
| 1234 | |||
| 1235 | static void emit_stack_arg_store_imm(s32 imm, s16 bpf_off, const u8 tmp, struct jit_ctx *ctx) | ||
| 1236 | { | ||
| 1237 | int idx = -bpf_off / sizeof(u64) - 1; | ||
| 1238 | |||
| 1239 | if (idx < NR_STACK_ARG_REGS) { | ||
| 1240 | emit_a64_mov_i(1, stack_arg_reg[idx], imm, ctx); | ||
| 1241 | } else { | ||
| 1242 | emit_a64_mov_i(1, tmp, imm, ctx); | ||
| 1243 | emit(A64_STR64I(tmp, A64_SP, (idx - NR_STACK_ARG_REGS) * sizeof(u64)), ctx); | ||
| 1244 | } | ||
| 1245 | } | ||
| 1246 | |||
| 1194 | /* JITs an eBPF instruction. | 1247 | /* JITs an eBPF instruction. |
| 1195 | * Returns: | 1248 | * Returns: |
| 1196 | * 0 - successfully JITed an 8-byte eBPF instruction. | 1249 | * 0 - successfully JITed an 8-byte eBPF instruction. |
| @@ -1646,6 +1699,11 @@ emit_cond_jmp: · 449 unmodified lines | |||
| 1646 | case BPF_LDX | BPF_MEM | BPF_H: | 1699 | case BPF_LDX | BPF_MEM | BPF_H: |
| 1647 | case BPF_LDX | BPF_MEM | BPF_B: | 1700 | case BPF_LDX | BPF_MEM | BPF_B: |
| 1648 | case BPF_LDX | BPF_MEM | BPF_DW: | 1701 | case BPF_LDX | BPF_MEM | BPF_DW: |
| 1702 | if (insn->src_reg == BPF_REG_PARAMS) { | ||
| 1703 | emit_stack_arg_load(dst, off, ctx); | ||
| 1704 | break; | ||
| 1705 | } | ||
| 1706 | fallthrough; | ||
| 1649 | case BPF_LDX | BPF_PROBE_MEM | BPF_DW: | 1707 | case BPF_LDX | BPF_PROBE_MEM | BPF_DW: |
| 1650 | case BPF_LDX | BPF_PROBE_MEM | BPF_W: | 1708 | case BPF_LDX | BPF_PROBE_MEM | BPF_W: |
| 1651 | case BPF_LDX | BPF_PROBE_MEM | BPF_H: | 1709 | case BPF_LDX | BPF_PROBE_MEM | BPF_H: |
| @@ -1672,6 +1730,8 @@ emit_cond_jmp: · 20 unmodified lines | |||
| 1672 | if (src == fp) { | 1730 | if (src == fp) { |
| 1673 | src_adj = ctx->priv_sp_used ? priv_sp : A64_SP; | 1731 | src_adj = ctx->priv_sp_used ? priv_sp : A64_SP; |
| 1674 | off_adj = off + ctx->stack_size; | 1732 | off_adj = off + ctx->stack_size; |
| 1733 | if (!ctx->priv_sp_used) | ||
| 1734 | off_adj += ctx->stack_arg_size; | ||
| 1675 | } else { | 1735 | } else { |
| 1676 | src_adj = src; | 1736 | src_adj = src; |
| 1677 | off_adj = off; | 1737 | off_adj = off; |
| @@ -1752,6 +1812,11 @@ emit_cond_jmp: · 74 unmodified lines | |||
| 1752 | case BPF_ST | BPF_MEM | BPF_H: | 1812 | case BPF_ST | BPF_MEM | BPF_H: |
| 1753 | case BPF_ST | BPF_MEM | BPF_B: | 1813 | case BPF_ST | BPF_MEM | BPF_B: |
| 1754 | case BPF_ST | BPF_MEM | BPF_DW: | 1814 | case BPF_ST | BPF_MEM | BPF_DW: |
| 1815 | if (insn->dst_reg == BPF_REG_PARAMS) { | ||
| 1816 | emit_stack_arg_store_imm(imm, off, tmp, ctx); | ||
| 1817 | break; | ||
| 1818 | } | ||
| 1819 | fallthrough; | ||
| 1755 | case BPF_ST | BPF_PROBE_MEM32 | BPF_B: | 1820 | case BPF_ST | BPF_PROBE_MEM32 | BPF_B: |
| 1756 | case BPF_ST | BPF_PROBE_MEM32 | BPF_H: | 1821 | case BPF_ST | BPF_PROBE_MEM32 | BPF_H: |
| 1757 | case BPF_ST | BPF_PROBE_MEM32 | BPF_W: | 1822 | case BPF_ST | BPF_PROBE_MEM32 | BPF_W: |
| @@ -1763,6 +1828,8 @@ emit_cond_jmp: · 5 unmodified lines | |||
| 1763 | if (dst == fp) { | 1828 | if (dst == fp) { |
| 1764 | dst_adj = ctx->priv_sp_used ? priv_sp : A64_SP; | 1829 | dst_adj = ctx->priv_sp_used ? priv_sp : A64_SP; |
| 1765 | off_adj = off + ctx->stack_size; | 1830 | off_adj = off + ctx->stack_size; |
| 1831 | if (!ctx->priv_sp_used) | ||
| 1832 | off_adj += ctx->stack_arg_size; | ||
| 1766 | } else { | 1833 | } else { |
| 1767 | dst_adj = dst; | 1834 | dst_adj = dst; |
| 1768 | off_adj = off; | 1835 | off_adj = off; |
| @@ -1814,6 +1881,11 @@ emit_cond_jmp: · 45 unmodified lines | |||
| 1814 | case BPF_STX | BPF_MEM | BPF_H: | 1881 | case BPF_STX | BPF_MEM | BPF_H: |
| 1815 | case BPF_STX | BPF_MEM | BPF_B: | 1882 | case BPF_STX | BPF_MEM | BPF_B: |
| 1816 | case BPF_STX | BPF_MEM | BPF_DW: | 1883 | case BPF_STX | BPF_MEM | BPF_DW: |
| 1884 | if (insn->dst_reg == BPF_REG_PARAMS) { | ||
| 1885 | emit_stack_arg_store(src, off, ctx); | ||
| 1886 | break; | ||
| 1887 | } | ||
| 1888 | fallthrough; | ||
| 1817 | case BPF_STX | BPF_PROBE_MEM32 | BPF_B: | 1889 | case BPF_STX | BPF_PROBE_MEM32 | BPF_B: |
| 1818 | case BPF_STX | BPF_PROBE_MEM32 | BPF_H: | 1890 | case BPF_STX | BPF_PROBE_MEM32 | BPF_H: |
| 1819 | case BPF_STX | BPF_PROBE_MEM32 | BPF_W: | 1891 | case BPF_STX | BPF_PROBE_MEM32 | BPF_W: |
| @@ -1825,6 +1897,8 @@ emit_cond_jmp: · 5 unmodified lines | |||
| 1825 | if (dst == fp) { | 1897 | if (dst == fp) { |
| 1826 | dst_adj = ctx->priv_sp_used ? priv_sp : A64_SP; | 1898 | dst_adj = ctx->priv_sp_used ? priv_sp : A64_SP; |
| 1827 | off_adj = off + ctx->stack_size; | 1899 | off_adj = off + ctx->stack_size; |
| 1900 | if (!ctx->priv_sp_used) | ||
| 1901 | off_adj += ctx->stack_arg_size; | ||
| 1828 | } else { | 1902 | } else { |
| 1829 | dst_adj = dst; | 1903 | dst_adj = dst; |
| 1830 | off_adj = off; | 1904 | off_adj = off; |
| @@ -2018,6 +2092,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr · 187 unmodified lines | |||
| 2018 | u8 *ro_image_ptr; | 2092 | u8 *ro_image_ptr; |
| 2019 | int body_idx; | 2093 | int body_idx; |
| 2020 | int exentry_idx; | 2094 | int exentry_idx; |
| 2095 | int out_cnt; | ||
| 2021 | 2096 | ||
| 2022 | if (!prog->jit_requested) | 2097 | if (!prog->jit_requested) |
| 2023 | return prog; | 2098 | return prog; |
| @@ -2065,6 +2140,14 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr · 41 unmodified lines | |||
| 2065 | ctx.user_vm_start = bpf_arena_get_user_vm_start(prog->aux->arena); | 2140 | ctx.user_vm_start = bpf_arena_get_user_vm_start(prog->aux->arena); |
| 2066 | ctx.arena_vm_start = bpf_arena_get_kern_vm_start(prog->aux->arena); | 2141 | ctx.arena_vm_start = bpf_arena_get_kern_vm_start(prog->aux->arena); |
| 2067 | 2142 | ||
| 2143 | out_cnt = bpf_out_stack_arg_cnt(env, prog); | ||
| 2144 | if (out_cnt) { | ||
| 2145 | int nr_on_stack = out_cnt - NR_STACK_ARG_REGS; | ||
| 2146 | |||
| 2147 | if (nr_on_stack > 0) | ||
| 2148 | ctx.stack_arg_size = round_up(nr_on_stack * sizeof(u64), 16); | ||
| 2149 | } | ||
| 2150 | |||
| 2068 | if (priv_stack_ptr) | 2151 | if (priv_stack_ptr) |
| 2069 | ctx.priv_sp_used = true; | 2152 | ctx.priv_sp_used = true; |
| 2070 | 2153 | ||
| @@ -2229,6 +2312,11 @@ bool bpf_jit_supports_kfunc_call(void) · 158 unmodified lines | |||
| 2229 | return true; | 2312 | return true; |
| 2230 | } | 2313 | } |
| 2231 | 2314 | ||
| 2315 | bool bpf_jit_supports_stack_args(void) | ||
| 2316 | { | ||
| 2317 | return true; | ||
| 2318 | } | ||
| 2319 | |||
| 2232 | void *bpf_arch_text_copy(void *dst, void *src, size_t len) | 2320 | void *bpf_arch_text_copy(void *dst, void *src, size_t len) |
| 2233 | { | 2321 | { |
| 2234 | if (!aarch64_insn_copy(dst, src, len)) | 2322 | if (!aarch64_insn_copy(dst, src, len)) |
| @@ -2247,24 +2335,24 @@ bool bpf_jit_supports_subprog_tailcalls(void) · 12 unmodified lines | |||
| 2247 | return true; | 2335 | return true; |
| 2248 | } | 2336 | } |
| 2249 | 2337 | ||
| 2250 | static void invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_link *l, | 2338 | static void invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_node *node, |
| 2251 | int bargs_off, int retval_off, int run_ctx_off, | 2339 | int bargs_off, int retval_off, int run_ctx_off, |
| 2252 | bool save_ret) | 2340 | bool save_ret) |
| 2253 | { | 2341 | { |
| 2254 | __le32 *branch; | 2342 | __le32 *branch; |
| 2255 | u64 enter_prog; | 2343 | u64 enter_prog; |
| 2256 | u64 exit_prog; | 2344 | u64 exit_prog; |
| 2257 | struct bpf_prog *p = l->link.prog; | 2345 | struct bpf_prog *p = node->link->prog; |
| 2258 | int cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie); | 2346 | int cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie); |
| 2259 | 2347 | ||
| 2260 | enter_prog = (u64)bpf_trampoline_enter(p); | 2348 | enter_prog = (u64)bpf_trampoline_enter(p); |
| 2261 | exit_prog = (u64)bpf_trampoline_exit(p); | 2349 | exit_prog = (u64)bpf_trampoline_exit(p); |
| 2262 | 2350 | ||
| 2263 | if (l->cookie == 0) { | 2351 | if (node->cookie == 0) { |
| 2264 | /* if cookie is zero, one instruction is enough to store it */ | 2352 | /* if cookie is zero, one instruction is enough to store it */ |
| 2265 | emit(A64_STR64I(A64_ZR, A64_SP, run_ctx_off + cookie_off), ctx); | 2353 | emit(A64_STR64I(A64_ZR, A64_SP, run_ctx_off + cookie_off), ctx); |
| 2266 | } else { | 2354 | } else { |
| 2267 | emit_a64_mov_i64(A64_R(10), l->cookie, ctx); | 2355 | emit_a64_mov_i64(A64_R(10), node->cookie, ctx); |
| 2268 | emit(A64_STR64I(A64_R(10), A64_SP, run_ctx_off + cookie_off), | 2356 | emit(A64_STR64I(A64_R(10), A64_SP, run_ctx_off + cookie_off), |
| 2269 | ctx); | 2357 | ctx); |
| 2270 | } | 2358 | } |
| @@ -2314,7 +2402,7 @@ static void invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_link *l, · 43 unmodified lines | |||
| 2314 | emit_call(exit_prog, ctx); | 2402 | emit_call(exit_prog, ctx); |
| 2315 | } | 2403 | } |
| 2316 | 2404 | ||
| 2317 | static void invoke_bpf_mod_ret(struct jit_ctx *ctx, struct bpf_tramp_links *tl, | 2405 | static void invoke_bpf_mod_ret(struct jit_ctx *ctx, struct bpf_tramp_nodes *tn, |
| 2318 | int bargs_off, int retval_off, int run_ctx_off, | 2406 | int bargs_off, int retval_off, int run_ctx_off, |
| 2319 | __le32 **branches) | 2407 | __le32 **branches) |
| 2320 | { | 2408 | { |
| @@ -2324,8 +2412,8 @@ static void invoke_bpf_mod_ret(struct jit_ctx *ctx, struct bpf_tramp_links *tl, · 3 unmodified lines | |||
| 2324 | * Set this to 0 to avoid confusing the program. | 2412 | * Set this to 0 to avoid confusing the program. |
| 2325 | */ | 2413 | */ |
| 2326 | emit(A64_STR64I(A64_ZR, A64_SP, retval_off), ctx); | 2414 | emit(A64_STR64I(A64_ZR, A64_SP, retval_off), ctx); |
| 2327 | for (i = 0; i < tl->nr_links; i++) { | 2415 | for (i = 0; i < tn->nr_nodes; i++) { |
| 2328 | invoke_bpf_prog(ctx, tl->links[i], bargs_off, retval_off, | 2416 | invoke_bpf_prog(ctx, tn->nodes[i], bargs_off, retval_off, |
| 2329 | run_ctx_off, true); | 2417 | run_ctx_off, true); |
| 2330 | /* if (*(u64 *)(sp + retval_off) != 0) | 2418 | /* if (*(u64 *)(sp + retval_off) != 0) |
| 2331 | * goto do_fexit; | 2419 | * goto do_fexit; |
| @@ -2456,10 +2544,10 @@ static void restore_args(struct jit_ctx *ctx, int bargs_off, int nregs) · 124 unmodified lines | |||
| 2456 | } | 2544 | } |
| 2457 | } | 2545 | } |
| 2458 | 2546 | ||
| 2459 | static bool is_struct_ops_tramp(const struct bpf_tramp_links *fentry_links) | 2547 | static bool is_struct_ops_tramp(const struct bpf_tramp_nodes *fentry_nodes) |
| 2460 | { | 2548 | { |
| 2461 | return fentry_links->nr_links == 1 && | 2549 | return fentry_nodes->nr_nodes == 1 && |
| 2462 | fentry_links->links[0]->link.type == BPF_LINK_TYPE_STRUCT_OPS; | 2550 | fentry_nodes->nodes[0]->link->type == BPF_LINK_TYPE_STRUCT_OPS; |
| 2463 | } | 2551 | } |
| 2464 | 2552 | ||
| 2465 | static void store_func_meta(struct jit_ctx *ctx, u64 func_meta, int func_meta_off) | 2553 | static void store_func_meta(struct jit_ctx *ctx, u64 func_meta, int func_meta_off) |
| @@ -2480,7 +2568,7 @@ static void store_func_meta(struct jit_ctx *ctx, u64 func_meta, int func_meta_of · 14 unmodified lines | |||
| 2480 | * | 2568 | * |
| 2481 | */ | 2569 | */ |
| 2482 | static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im, | 2570 | static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im, |
| 2483 | struct bpf_tramp_links *tlinks, void *func_addr, | 2571 | struct bpf_tramp_nodes *tnodes, void *func_addr, |
| 2484 | const struct btf_func_model *m, | 2572 | const struct btf_func_model *m, |
| 2485 | const struct arg_aux *a, | 2573 | const struct arg_aux *a, |
| 2486 | u32 flags) | 2574 | u32 flags) |
| @@ -2496,14 +2584,14 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im, · 9 unmodified lines | |||
| 2496 | int run_ctx_off; | 2584 | int run_ctx_off; |
| 2497 | int oargs_off; | 2585 | int oargs_off; |
| 2498 | int nfuncargs; | 2586 | int nfuncargs; |
| 2499 | struct bpf_tramp_links *fentry = &tlinks[BPF_TRAMP_FENTRY]; | 2587 | struct bpf_tramp_nodes *fentry = &tnodes[BPF_TRAMP_FENTRY]; |
| 2500 | struct bpf_tramp_links *fexit = &tlinks[BPF_TRAMP_FEXIT]; | 2588 | struct bpf_tramp_nodes *fexit = &tnodes[BPF_TRAMP_FEXIT]; |
| 2501 | struct bpf_tramp_links *fmod_ret = &tlinks[BPF_TRAMP_MODIFY_RETURN]; | 2589 | struct bpf_tramp_nodes *fmod_ret = &tnodes[BPF_TRAMP_MODIFY_RETURN]; |
| 2502 | bool save_ret; | 2590 | bool save_ret; |
| 2503 | __le32 **branches = NULL; | 2591 | __le32 **branches = NULL; |
| 2504 | bool is_struct_ops = is_struct_ops_tramp(fentry); | 2592 | bool is_struct_ops = is_struct_ops_tramp(fentry); |
| 2505 | int cookie_off, cookie_cnt, cookie_bargs_off; | 2593 | int cookie_off, cookie_cnt, cookie_bargs_off; |
| 2506 | int fsession_cnt = bpf_fsession_cnt(tlinks); | 2594 | int fsession_cnt = bpf_fsession_cnt(tnodes); |
| 2507 | u64 func_meta; | 2595 | u64 func_meta; |
| 2508 | 2596 | ||
| 2509 | /* trampoline stack layout: | 2597 | /* trampoline stack layout: |
| @@ -2549,7 +2637,7 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im, · 39 unmodified lines | |||
| 2549 | 2637 | ||
| 2550 | cookie_off = stack_size; | 2638 | cookie_off = stack_size; |
| 2551 | /* room for session cookies */ | 2639 | /* room for session cookies */ |
| 2552 | cookie_cnt = bpf_fsession_cookie_cnt(tlinks); | 2640 | cookie_cnt = bpf_fsession_cookie_cnt(tnodes); |
| 2553 | stack_size += cookie_cnt * 8; | 2641 | stack_size += cookie_cnt * 8; |
| 2554 | 2642 | ||
| 2555 | ip_off = stack_size; | 2643 | ip_off = stack_size; |
| @@ -2646,20 +2734,20 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im, · 90 unmodified lines | |||
| 2646 | } | 2734 | } |
| 2647 | 2735 | ||
| 2648 | cookie_bargs_off = (bargs_off - cookie_off) / 8; | 2736 | cookie_bargs_off = (bargs_off - cookie_off) / 8; |
| 2649 | for (i = 0; i < fentry->nr_links; i++) { | 2737 | for (i = 0; i < fentry->nr_nodes; i++) { |
| 2650 | if (bpf_prog_calls_session_cookie(fentry->links[i])) { | 2738 | if (bpf_prog_calls_session_cookie(fentry->nodes[i])) { |
| 2651 | u64 meta = func_meta | (cookie_bargs_off << BPF_TRAMP_COOKIE_INDEX_SHIFT); | 2739 | u64 meta = func_meta | (cookie_bargs_off << BPF_TRAMP_COOKIE_INDEX_SHIFT); |
| 2652 | 2740 | ||
| 2653 | store_func_meta(ctx, meta, func_meta_off); | 2741 | store_func_meta(ctx, meta, func_meta_off); |
| 2654 | cookie_bargs_off--; | 2742 | cookie_bargs_off--; |
| 2655 | } | 2743 | } |
| 2656 | invoke_bpf_prog(ctx, fentry->links[i], bargs_off, | 2744 | invoke_bpf_prog(ctx, fentry->nodes[i], bargs_off, |
| 2657 | retval_off, run_ctx_off, | 2745 | retval_off, run_ctx_off, |
| 2658 | flags & BPF_TRAMP_F_RET_FENTRY_RET); | 2746 | flags & BPF_TRAMP_F_RET_FENTRY_RET); |
| 2659 | } | 2747 | } |
| 2660 | 2748 | ||
| 2661 | if (fmod_ret->nr_links) { | 2749 | if (fmod_ret->nr_nodes) { |
| 2662 | branches = kcalloc(fmod_ret->nr_links, sizeof(__le32 *), | 2750 | branches = kcalloc(fmod_ret->nr_nodes, sizeof(__le32 *), |
| 2663 | GFP_KERNEL); | 2751 | GFP_KERNEL); |
| 2664 | if (!branches) | 2752 | if (!branches) |
| 2665 | return -ENOMEM; | 2753 | return -ENOMEM; |
| @@ -2683,7 +2771,7 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im, · 17 unmodified lines | |||
| 2683 | } | 2771 | } |
| 2684 | 2772 | ||
| 2685 | /* update the branches saved in invoke_bpf_mod_ret with cbnz */ | 2773 | /* update the branches saved in invoke_bpf_mod_ret with cbnz */ |
| 2686 | for (i = 0; i < fmod_ret->nr_links && ctx->image != NULL; i++) { | 2774 | for (i = 0; i < fmod_ret->nr_nodes && ctx->image != NULL; i++) { |
| 2687 | int offset = &ctx->image[ctx->idx] - branches[i]; | 2775 | int offset = &ctx->image[ctx->idx] - branches[i]; |
| 2688 | *branches[i] = cpu_to_le32(A64_CBNZ(1, A64_R(10), offset)); | 2776 | *branches[i] = cpu_to_le32(A64_CBNZ(1, A64_R(10), offset)); |
| 2689 | } | 2777 | } |
| @@ -2694,14 +2782,14 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im, · 4 unmodified lines | |||
| 2694 | store_func_meta(ctx, func_meta, func_meta_off); | 2782 | store_func_meta(ctx, func_meta, func_meta_off); |
| 2695 | 2783 | ||
| 2696 | cookie_bargs_off = (bargs_off - cookie_off) / 8; | 2784 | cookie_bargs_off = (bargs_off - cookie_off) / 8; |
| 2697 | for (i = 0; i < fexit->nr_links; i++) { | 2785 | for (i = 0; i < fexit->nr_nodes; i++) { |
| 2698 | if (bpf_prog_calls_session_cookie(fexit->links[i])) { | 2786 | if (bpf_prog_calls_session_cookie(fexit->nodes[i])) { |
| 2699 | u64 meta = func_meta | (cookie_bargs_off << BPF_TRAMP_COOKIE_INDEX_SHIFT); | 2787 | u64 meta = func_meta | (cookie_bargs_off << BPF_TRAMP_COOKIE_INDEX_SHIFT); |
| 2700 | 2788 | ||
| 2701 | store_func_meta(ctx, meta, func_meta_off); | 2789 | store_func_meta(ctx, meta, func_meta_off); |
| 2702 | cookie_bargs_off--; | 2790 | cookie_bargs_off--; |
| 2703 | } | 2791 | } |
| 2704 | invoke_bpf_prog(ctx, fexit->links[i], bargs_off, retval_off, | 2792 | invoke_bpf_prog(ctx, fexit->nodes[i], bargs_off, retval_off, |
| 2705 | run_ctx_off, false); | 2793 | run_ctx_off, false); |
| 2706 | } | 2794 | } |
| 2707 | 2795 | ||
| @@ -2759,7 +2847,7 @@ bool bpf_jit_supports_fsession(void) · 51 unmodified lines | |||
| 2759 | } | 2847 | } |
| 2760 | 2848 | ||
| 2761 | int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, | 2849 | int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, |
| 2762 | struct bpf_tramp_links *tlinks, void *func_addr) | 2850 | struct bpf_tramp_nodes *tnodes, void *func_addr) |
| 2763 | { | 2851 | { |
| 2764 | struct jit_ctx ctx = { | 2852 | struct jit_ctx ctx = { |
| 2765 | .image = NULL, | 2853 | .image = NULL, |
| @@ -2773,7 +2861,7 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, · 7 unmodified lines | |||
| 2773 | if (ret < 0) | 2861 | if (ret < 0) |
| 2774 | return ret; | 2862 | return ret; |
| 2775 | 2863 | ||
| 2776 | ret = prepare_trampoline(&ctx, &im, tlinks, func_addr, m, &aaux, flags); | 2864 | ret = prepare_trampoline(&ctx, &im, tnodes, func_addr, m, &aaux, flags); |
| 2777 | if (ret < 0) | 2865 | if (ret < 0) |
| 2778 | return ret; | 2866 | return ret; |
| 2779 | 2867 | ||
| @@ -2797,7 +2885,7 @@ int arch_protect_bpf_trampoline(void *image, unsigned int size) · 17 unmodified lines | |||
| 2797 | 2885 | ||
| 2798 | int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image, | 2886 | int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image, |
| 2799 | void *ro_image_end, const struct btf_func_model *m, | 2887 | void *ro_image_end, const struct btf_func_model *m, |
| 2800 | u32 flags, struct bpf_tramp_links *tlinks, | 2888 | u32 flags, struct bpf_tramp_nodes *tnodes, |
| 2801 | void *func_addr) | 2889 | void *func_addr) |
| 2802 | { | 2890 | { |
| 2803 | u32 size = ro_image_end - ro_image; | 2891 | u32 size = ro_image_end - ro_image; |
| @@ -2824,7 +2912,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image, · 20 unmodified lines | |||
| 2824 | ret = calc_arg_aux(m, &aaux); | 2912 | ret = calc_arg_aux(m, &aaux); |
| 2825 | if (ret) | 2913 | if (ret) |
| 2826 | goto out; | 2914 | goto out; |
| 2827 | ret = prepare_trampoline(&ctx, im, tlinks, func_addr, m, &aaux, flags); | 2915 | ret = prepare_trampoline(&ctx, im, tnodes, func_addr, m, &aaux, flags); |
| 2828 | 2916 | ||
| 2829 | if (ret > 0 && validate_code(&ctx) < 0) { | 2917 | if (ret > 0 && validate_code(&ctx) < 0) { |
| 2830 | ret = -EINVAL; | 2918 | ret = -EINVAL; |
| @@ -8,8 +8,8 @@ SYM_FUNC_START(arch_bpf_timed_may_goto) · 7 unmodified lines | |||
| 8 | stp x29, x30, [sp, #-64]! | 8 | stp x29, x30, [sp, #-64]! |
| 9 | mov x29, sp | 9 | mov x29, sp |
| 10 | 10 | ||
| 11 | /* Save BPF registers R0 - R5 (x7, x0-x4)*/ | 11 | /* Save BPF registers R0 - R5 (x8, x0-x4)*/ |
| 12 | stp x7, x0, [sp, #16] | 12 | stp x8, x0, [sp, #16] |
| 13 | stp x1, x2, [sp, #32] | 13 | stp x1, x2, [sp, #32] |
| 14 | stp x3, x4, [sp, #48] | 14 | stp x3, x4, [sp, #48] |
| 15 | 15 | ||
| @@ -28,8 +28,8 @@ SYM_FUNC_START(arch_bpf_timed_may_goto) · 12 unmodified lines | |||
| 28 | /* BPF_REG_AX(x9) will be stored into count, so move return value to it. */ | 28 | /* BPF_REG_AX(x9) will be stored into count, so move return value to it. */ |
| 29 | mov x9, x0 | 29 | mov x9, x0 |
| 30 | 30 | ||
| 31 | /* Restore BPF registers R0 - R5 (x7, x0-x4) */ | 31 | /* Restore BPF registers R0 - R5 (x8, x0-x4) */ |
| 32 | ldp x7, x0, [sp, #16] | 32 | ldp x8, x0, [sp, #16] |
| 33 | ldp x1, x2, [sp, #32] | 33 | ldp x1, x2, [sp, #32] |
| 34 | ldp x3, x4, [sp, #48] | 34 | ldp x3, x4, [sp, #48] |
| 35 | 35 | ||
| @@ -1674,17 +1674,17 @@ static void restore_stk_args(struct jit_ctx *ctx, int nr_stk_args, int args_off, · 1673 unmodified lines | |||
| 1674 | } | 1674 | } |
| 1675 | } | 1675 | } |
| 1676 | 1676 | ||
| 1677 | static int invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_link *l, | 1677 | static int invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_node *n, |
| 1678 | int args_off, int retval_off, int run_ctx_off, bool save_ret) | 1678 | int args_off, int retval_off, int run_ctx_off, bool save_ret) |
| 1679 | { | 1679 | { |
| 1680 | int ret; | 1680 | int ret; |
| 1681 | u32 *branch; | 1681 | u32 *branch; |
| 1682 | struct bpf_prog *p = l->link.prog; | 1682 | struct bpf_prog *p = n->link->prog; |
| 1683 | int cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie); | 1683 | int cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie); |
| 1684 | 1684 | ||
| 1685 | if (l->cookie) | 1685 | if (n->cookie) |
| 1686 | emit_store_stack_imm64(ctx, LOONGARCH_GPR_T1, | 1686 | emit_store_stack_imm64(ctx, LOONGARCH_GPR_T1, |
| 1687 | -run_ctx_off + cookie_off, l->cookie); | 1687 | -run_ctx_off + cookie_off, n->cookie); |
| 1688 | else | 1688 | else |
| 1689 | emit_insn(ctx, std, LOONGARCH_GPR_ZERO, LOONGARCH_GPR_FP, -run_ctx_off + cookie_off); | 1689 | emit_insn(ctx, std, LOONGARCH_GPR_ZERO, LOONGARCH_GPR_FP, -run_ctx_off + cookie_off); |
| 1690 | 1690 | ||
| @@ -1737,22 +1737,22 @@ static int invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_link *l, · 46 unmodified lines | |||
| 1737 | return ret; | 1737 | return ret; |
| 1738 | } | 1738 | } |
| 1739 | 1739 | ||
| 1740 | static int invoke_bpf(struct jit_ctx *ctx, struct bpf_tramp_links *tl, | 1740 | static int invoke_bpf(struct jit_ctx *ctx, struct bpf_tramp_nodes *tn, |
| 1741 | int args_off, int retval_off, int run_ctx_off, | 1741 | int args_off, int retval_off, int run_ctx_off, |
| 1742 | int func_meta_off, bool save_ret, u64 func_meta, int cookie_off) | 1742 | int func_meta_off, bool save_ret, u64 func_meta, int cookie_off) |
| 1743 | { | 1743 | { |
| 1744 | int i, cur_cookie = (cookie_off - args_off) / 8; | 1744 | int i, cur_cookie = (cookie_off - args_off) / 8; |
| 1745 | 1745 | ||
| 1746 | for (i = 0; i < tl->nr_links; i++) { | 1746 | for (i = 0; i < tn->nr_nodes; i++) { |
| 1747 | int err; | 1747 | int err; |
| 1748 | 1748 | ||
| 1749 | if (bpf_prog_calls_session_cookie(tl->links[i])) { | 1749 | if (bpf_prog_calls_session_cookie(tn->nodes[i])) { |
| 1750 | u64 meta = func_meta | ((u64)cur_cookie << BPF_TRAMP_COOKIE_INDEX_SHIFT); | 1750 | u64 meta = func_meta | ((u64)cur_cookie << BPF_TRAMP_COOKIE_INDEX_SHIFT); |
| 1751 | 1751 | ||
| 1752 | emit_store_stack_imm64(ctx, LOONGARCH_GPR_T1, -func_meta_off, meta); | 1752 | emit_store_stack_imm64(ctx, LOONGARCH_GPR_T1, -func_meta_off, meta); |
| 1753 | cur_cookie--; | 1753 | cur_cookie--; |
| 1754 | } | 1754 | } |
| 1755 | err = invoke_bpf_prog(ctx, tl->links[i], args_off, retval_off, run_ctx_off, save_ret); | 1755 | err = invoke_bpf_prog(ctx, tn->nodes[i], args_off, retval_off, run_ctx_off, save_ret); |
| 1756 | if (err) | 1756 | if (err) |
| 1757 | return err; | 1757 | return err; |
| 1758 | } | 1758 | } |
| @@ -1807,7 +1807,7 @@ static void sign_extend(struct jit_ctx *ctx, int rd, int rj, u8 size, bool sign) · 48 unmodified lines | |||
| 1807 | } | 1807 | } |
| 1808 | 1808 | ||
| 1809 | static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im, | 1809 | static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im, |
| 1810 | const struct btf_func_model *m, struct bpf_tramp_links *tlinks, | 1810 | const struct btf_func_model *m, struct bpf_tramp_nodes *tnodes, |
| 1811 | void *func_addr, u32 flags) | 1811 | void *func_addr, u32 flags) |
| 1812 | { | 1812 | { |
| 1813 | int i, ret, save_ret; | 1813 | int i, ret, save_ret; |
| @@ -1817,9 +1817,9 @@ static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_i · 3 unmodified lines | |||
| 1817 | unsigned long long func_meta; | 1817 | unsigned long long func_meta; |
| 1818 | bool is_struct_ops = flags & BPF_TRAMP_F_INDIRECT; | 1818 | bool is_struct_ops = flags & BPF_TRAMP_F_INDIRECT; |
| 1819 | void *orig_call = func_addr; | 1819 | void *orig_call = func_addr; |
| 1820 | struct bpf_tramp_links *fentry = &tlinks[BPF_TRAMP_FENTRY]; | 1820 | struct bpf_tramp_nodes *fentry = &tnodes[BPF_TRAMP_FENTRY]; |
| 1821 | struct bpf_tramp_links *fexit = &tlinks[BPF_TRAMP_FEXIT]; | 1821 | struct bpf_tramp_nodes *fexit = &tnodes[BPF_TRAMP_FEXIT]; |
| 1822 | struct bpf_tramp_links *fmod_ret = &tlinks[BPF_TRAMP_MODIFY_RETURN]; | 1822 | struct bpf_tramp_nodes *fmod_ret = &tnodes[BPF_TRAMP_MODIFY_RETURN]; |
| 1823 | u32 **branches = NULL; | 1823 | u32 **branches = NULL; |
| 1824 | 1824 | ||
| 1825 | /* | 1825 | /* |
| @@ -1898,7 +1898,7 @@ static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_i · 72 unmodified lines | |||
| 1898 | ip_off = stack_size; | 1898 | ip_off = stack_size; |
| 1899 | } | 1899 | } |
| 1900 | 1900 | ||
| 1901 | cookie_cnt = bpf_fsession_cookie_cnt(tlinks); | 1901 | cookie_cnt = bpf_fsession_cookie_cnt(tnodes); |
| 1902 | 1902 | ||
| 1903 | /* Room for session cookies */ | 1903 | /* Room for session cookies */ |
| 1904 | stack_size += cookie_cnt * 8; | 1904 | stack_size += cookie_cnt * 8; |
| @@ -1969,7 +1969,7 @@ static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_i · 64 unmodified lines | |||
| 1969 | 1969 | ||
| 1970 | store_args(ctx, nr_arg_slots, args_off); | 1970 | store_args(ctx, nr_arg_slots, args_off); |
| 1971 | 1971 | ||
| 1972 | if (bpf_fsession_cnt(tlinks)) { | 1972 | if (bpf_fsession_cnt(tnodes)) { |
| 1973 | /* clear all session cookies' value */ | 1973 | /* clear all session cookies' value */ |
| 1974 | for (i = 0; i < cookie_cnt; i++) | 1974 | for (i = 0; i < cookie_cnt; i++) |
| 1975 | emit_insn(ctx, std, LOONGARCH_GPR_ZERO, LOONGARCH_GPR_FP, -cookie_off + 8 * i); | 1975 | emit_insn(ctx, std, LOONGARCH_GPR_ZERO, LOONGARCH_GPR_FP, -cookie_off + 8 * i); |
| @@ -1994,20 +1994,20 @@ static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_i · 18 unmodified lines | |||
| 1994 | return ret; | 1994 | return ret; |
| 1995 | } | 1995 | } |
| 1996 | 1996 | ||
| 1997 | if (fentry->nr_links) { | 1997 | if (fentry->nr_nodes) { |
| 1998 | ret = invoke_bpf(ctx, fentry, args_off, retval_off, run_ctx_off, func_meta_off, | 1998 | ret = invoke_bpf(ctx, fentry, args_off, retval_off, run_ctx_off, func_meta_off, |
| 1999 | flags & BPF_TRAMP_F_RET_FENTRY_RET, func_meta, cookie_off); | 1999 | flags & BPF_TRAMP_F_RET_FENTRY_RET, func_meta, cookie_off); |
| 2000 | if (ret) | 2000 | if (ret) |
| 2001 | return ret; | 2001 | return ret; |
| 2002 | } | 2002 | } |
| 2003 | if (fmod_ret->nr_links) { | 2003 | if (fmod_ret->nr_nodes) { |
| 2004 | branches = kcalloc(fmod_ret->nr_links, sizeof(u32 *), GFP_KERNEL); | 2004 | branches = kcalloc(fmod_ret->nr_nodes, sizeof(u32 *), GFP_KERNEL); |
| 2005 | if (!branches) | 2005 | if (!branches) |
| 2006 | return -ENOMEM; | 2006 | return -ENOMEM; |
| 2007 | 2007 | ||
| 2008 | emit_insn(ctx, std, LOONGARCH_GPR_ZERO, LOONGARCH_GPR_FP, -retval_off); | 2008 | emit_insn(ctx, std, LOONGARCH_GPR_ZERO, LOONGARCH_GPR_FP, -retval_off); |
| 2009 | for (i = 0; i < fmod_ret->nr_links; i++) { | 2009 | for (i = 0; i < fmod_ret->nr_nodes; i++) { |
| 2010 | ret = invoke_bpf_prog(ctx, fmod_ret->links[i], | 2010 | ret = invoke_bpf_prog(ctx, fmod_ret->nodes[i], |
| 2011 | args_off, retval_off, run_ctx_off, true); | 2011 | args_off, retval_off, run_ctx_off, true); |
| 2012 | if (ret) | 2012 | if (ret) |
| 2013 | goto out; | 2013 | goto out; |
| @@ -2035,17 +2035,17 @@ static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_i · 21 unmodified lines | |||
| 2035 | emit_insn(ctx, nop); | 2035 | emit_insn(ctx, nop); |
| 2036 | } | 2036 | } |
| 2037 | 2037 | ||
| 2038 | for (i = 0; ctx->image && i < fmod_ret->nr_links; i++) { | 2038 | for (i = 0; ctx->image && i < fmod_ret->nr_nodes; i++) { |
| 2039 | int offset = (void *)(&ctx->image[ctx->idx]) - (void *)branches[i]; | 2039 | int offset = (void *)(&ctx->image[ctx->idx]) - (void *)branches[i]; |
| 2040 | *branches[i] = larch_insn_gen_bne(LOONGARCH_GPR_T1, LOONGARCH_GPR_ZERO, offset); | 2040 | *branches[i] = larch_insn_gen_bne(LOONGARCH_GPR_T1, LOONGARCH_GPR_ZERO, offset); |
| 2041 | } | 2041 | } |
| 2042 | 2042 | ||
| 2043 | /* Set "is_return" flag for fsession */ | 2043 | /* Set "is_return" flag for fsession */ |
| 2044 | func_meta |= (1ULL << BPF_TRAMP_IS_RETURN_SHIFT); | 2044 | func_meta |= (1ULL << BPF_TRAMP_IS_RETURN_SHIFT); |
| 2045 | if (bpf_fsession_cnt(tlinks)) | 2045 | if (bpf_fsession_cnt(tnodes)) |
| 2046 | emit_store_stack_imm64(ctx, LOONGARCH_GPR_T1, -func_meta_off, func_meta); | 2046 | emit_store_stack_imm64(ctx, LOONGARCH_GPR_T1, -func_meta_off, func_meta); |
| 2047 | 2047 | ||
| 2048 | if (fexit->nr_links) { | 2048 | if (fexit->nr_nodes) { |
| 2049 | ret = invoke_bpf(ctx, fexit, args_off, retval_off, run_ctx_off, | 2049 | ret = invoke_bpf(ctx, fexit, args_off, retval_off, run_ctx_off, |
| 2050 | func_meta_off, false, func_meta, cookie_off); | 2050 | func_meta_off, false, func_meta, cookie_off); |
| 2051 | if (ret) | 2051 | if (ret) |
| @@ -2115,7 +2115,7 @@ out: · 63 unmodified lines | |||
| 2115 | 2115 | ||
| 2116 | int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image, | 2116 | int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image, |
| 2117 | void *ro_image_end, const struct btf_func_model *m, | 2117 | void *ro_image_end, const struct btf_func_model *m, |
| 2118 | u32 flags, struct bpf_tramp_links *tlinks, void *func_addr) | 2118 | u32 flags, struct bpf_tramp_nodes *tnodes, void *func_addr) |
| 2119 | { | 2119 | { |
| 2120 | int ret, size; | 2120 | int ret, size; |
| 2121 | void *image, *tmp; | 2121 | void *image, *tmp; |
| @@ -2131,7 +2131,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image, · 9 unmodified lines | |||
| 2131 | ctx.idx = 0; | 2131 | ctx.idx = 0; |
| 2132 | 2132 | ||
| 2133 | jit_fill_hole(image, (unsigned int)(ro_image_end - ro_image)); | 2133 | jit_fill_hole(image, (unsigned int)(ro_image_end - ro_image)); |
| 2134 | ret = __arch_prepare_bpf_trampoline(&ctx, im, m, tlinks, func_addr, flags); | 2134 | ret = __arch_prepare_bpf_trampoline(&ctx, im, m, tnodes, func_addr, flags); |
| 2135 | if (ret < 0) | 2135 | if (ret < 0) |
| 2136 | goto out; | 2136 | goto out; |
| 2137 | 2137 | ||
| @@ -2152,7 +2152,7 @@ out: · 14 unmodified lines | |||
| 2152 | } | 2152 | } |
| 2153 | 2153 | ||
| 2154 | int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, | 2154 | int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, |
| 2155 | struct bpf_tramp_links *tlinks, void *func_addr) | 2155 | struct bpf_tramp_nodes *tnodes, void *func_addr) |
| 2156 | { | 2156 | { |
| 2157 | int ret; | 2157 | int ret; |
| 2158 | struct jit_ctx ctx; | 2158 | struct jit_ctx ctx; |
| @@ -2161,7 +2161,7 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, · 2 unmodified lines | |||
| 2161 | ctx.image = NULL; | 2161 | ctx.image = NULL; |
| 2162 | ctx.idx = 0; | 2162 | ctx.idx = 0; |
| 2163 | 2163 | ||
| 2164 | ret = __arch_prepare_bpf_trampoline(&ctx, &im, m, tlinks, func_addr, flags); | 2164 | ret = __arch_prepare_bpf_trampoline(&ctx, &im, m, tnodes, func_addr, flags); |
| 2165 | 2165 | ||
| 2166 | return ret < 0 ? ret : ret * LOONGARCH_INSN_SIZE; | 2166 | return ret < 0 ? ret : ret * LOONGARCH_INSN_SIZE; |
| 2167 | } | 2167 | } |
| @@ -597,22 +597,22 @@ int arch_protect_bpf_trampoline(void *image, unsigned int size) · 596 unmodified lines | |||
| 597 | } | 597 | } |
| 598 | 598 | ||
| 599 | static int invoke_bpf_prog(u32 *image, u32 *ro_image, struct codegen_context *ctx, | 599 | static int invoke_bpf_prog(u32 *image, u32 *ro_image, struct codegen_context *ctx, |
| 600 | struct bpf_tramp_link *l, int regs_off, int retval_off, | 600 | struct bpf_tramp_node *n, int regs_off, int retval_off, |
| 601 | int run_ctx_off, bool save_ret) | 601 | int run_ctx_off, bool save_ret) |
| 602 | { | 602 | { |
| 603 | struct bpf_prog *p = l->link.prog; | 603 | struct bpf_prog *p = n->link->prog; |
| 604 | ppc_inst_t branch_insn; | 604 | ppc_inst_t branch_insn; |
| 605 | u32 jmp_idx; | 605 | u32 jmp_idx; |
| 606 | int ret = 0; | 606 | int ret = 0; |
| 607 | 607 | ||
| 608 | /* Save cookie */ | 608 | /* Save cookie */ |
| 609 | if (IS_ENABLED(CONFIG_PPC64)) { | 609 | if (IS_ENABLED(CONFIG_PPC64)) { |
| 610 | PPC_LI64(_R3, l->cookie); | 610 | PPC_LI64(_R3, n->cookie); |
| 611 | EMIT(PPC_RAW_STD(_R3, _R1, run_ctx_off + offsetof(struct bpf_tramp_run_ctx, | 611 | EMIT(PPC_RAW_STD(_R3, _R1, run_ctx_off + offsetof(struct bpf_tramp_run_ctx, |
| 612 | bpf_cookie))); | 612 | bpf_cookie))); |
| 613 | } else { | 613 | } else { |
| 614 | PPC_LI32(_R3, l->cookie >> 32); | 614 | PPC_LI32(_R3, n->cookie >> 32); |
| 615 | PPC_LI32(_R4, l->cookie); | 615 | PPC_LI32(_R4, n->cookie); |
| 616 | EMIT(PPC_RAW_STW(_R3, _R1, | 616 | EMIT(PPC_RAW_STW(_R3, _R1, |
| 617 | run_ctx_off + offsetof(struct bpf_tramp_run_ctx, bpf_cookie))); | 617 | run_ctx_off + offsetof(struct bpf_tramp_run_ctx, bpf_cookie))); |
| 618 | EMIT(PPC_RAW_STW(_R4, _R1, | 618 | EMIT(PPC_RAW_STW(_R4, _R1, |
| @@ -679,7 +679,7 @@ static int invoke_bpf_prog(u32 *image, u32 *ro_image, struct codegen_context *ct · 60 unmodified lines | |||
| 679 | } | 679 | } |
| 680 | 680 | ||
| 681 | static int invoke_bpf_mod_ret(u32 *image, u32 *ro_image, struct codegen_context *ctx, | 681 | static int invoke_bpf_mod_ret(u32 *image, u32 *ro_image, struct codegen_context *ctx, |
| 682 | struct bpf_tramp_links *tl, int regs_off, int retval_off, | 682 | struct bpf_tramp_nodes *tn, int regs_off, int retval_off, |
| 683 | int run_ctx_off, u32 *branches) | 683 | int run_ctx_off, u32 *branches) |
| 684 | { | 684 | { |
| 685 | int i; | 685 | int i; |
| @@ -690,8 +690,8 @@ static int invoke_bpf_mod_ret(u32 *image, u32 *ro_image, struct codegen_context · 4 unmodified lines | |||
| 690 | */ | 690 | */ |
| 691 | EMIT(PPC_RAW_LI(_R3, 0)); | 691 | EMIT(PPC_RAW_LI(_R3, 0)); |
| 692 | EMIT(PPC_RAW_STL(_R3, _R1, retval_off)); | 692 | EMIT(PPC_RAW_STL(_R3, _R1, retval_off)); |
| 693 | for (i = 0; i < tl->nr_links; i++) { | 693 | for (i = 0; i < tn->nr_nodes; i++) { |
| 694 | if (invoke_bpf_prog(image, ro_image, ctx, tl->links[i], regs_off, retval_off, | 694 | if (invoke_bpf_prog(image, ro_image, ctx, tn->nodes[i], regs_off, retval_off, |
| 695 | run_ctx_off, true)) | 695 | run_ctx_off, true)) |
| 696 | return -EINVAL; | 696 | return -EINVAL; |
| 697 | 697 | ||
| @@ -807,18 +807,18 @@ static void bpf_trampoline_restore_args_stack(u32 *image, struct codegen_context · 109 unmodified lines | |||
| 807 | static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_image, | 807 | static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_image, |
| 808 | void *rw_image_end, void *ro_image, | 808 | void *rw_image_end, void *ro_image, |
| 809 | const struct btf_func_model *m, u32 flags, | 809 | const struct btf_func_model *m, u32 flags, |
| 810 | struct bpf_tramp_links *tlinks, | 810 | struct bpf_tramp_nodes *tnodes, |
| 811 | void *func_addr) | 811 | void *func_addr) |
| 812 | { | 812 | { |
| 813 | int regs_off, func_meta_off, ip_off, run_ctx_off, retval_off; | 813 | int regs_off, func_meta_off, ip_off, run_ctx_off, retval_off; |
| 814 | int nvr_off, alt_lr_off, r4_off = 0; | 814 | int nvr_off, alt_lr_off, r4_off = 0; |
| 815 | struct bpf_tramp_links *fmod_ret = &tlinks[BPF_TRAMP_MODIFY_RETURN]; | 815 | struct bpf_tramp_nodes *fmod_ret = &tnodes[BPF_TRAMP_MODIFY_RETURN]; |
| 816 | struct bpf_tramp_links *fentry = &tlinks[BPF_TRAMP_FENTRY]; | 816 | struct bpf_tramp_nodes *fentry = &tnodes[BPF_TRAMP_FENTRY]; |
| 817 | struct bpf_tramp_links *fexit = &tlinks[BPF_TRAMP_FEXIT]; | 817 | struct bpf_tramp_nodes *fexit = &tnodes[BPF_TRAMP_FEXIT]; |
| 818 | int i, ret, nr_regs, retaddr_off, bpf_frame_size = 0; | 818 | int i, ret, nr_regs, retaddr_off, bpf_frame_size = 0; |
| 819 | struct codegen_context codegen_ctx, *ctx; | 819 | struct codegen_context codegen_ctx, *ctx; |
| 820 | int cookie_off, cookie_cnt, cookie_ctx_off; | 820 | int cookie_off, cookie_cnt, cookie_ctx_off; |
| 821 | int fsession_cnt = bpf_fsession_cnt(tlinks); | 821 | int fsession_cnt = bpf_fsession_cnt(tnodes); |
| 822 | u64 func_meta; | 822 | u64 func_meta; |
| 823 | u32 *image = (u32 *)rw_image; | 823 | u32 *image = (u32 *)rw_image; |
| 824 | ppc_inst_t branch_insn; | 824 | ppc_inst_t branch_insn; |
| @@ -893,7 +893,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 68 unmodified lines | |||
| 893 | 893 | ||
| 894 | /* room for session cookies */ | 894 | /* room for session cookies */ |
| 895 | cookie_off = bpf_frame_size; | 895 | cookie_off = bpf_frame_size; |
| 896 | cookie_cnt = bpf_fsession_cookie_cnt(tlinks); | 896 | cookie_cnt = bpf_fsession_cookie_cnt(tnodes); |
| 897 | bpf_frame_size += cookie_cnt * 8; | 897 | bpf_frame_size += cookie_cnt * 8; |
| 898 | 898 | ||
| 899 | /* Room for IP address argument */ | 899 | /* Room for IP address argument */ |
| @@ -1030,21 +1030,21 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 130 unmodified lines | |||
| 1030 | 1030 | ||
| 1031 | cookie_ctx_off = (regs_off - cookie_off) / 8; | 1031 | cookie_ctx_off = (regs_off - cookie_off) / 8; |
| 1032 | 1032 | ||
| 1033 | for (i = 0; i < fentry->nr_links; i++) { | 1033 | for (i = 0; i < fentry->nr_nodes; i++) { |
| 1034 | if (bpf_prog_calls_session_cookie(fentry->links[i])) { | 1034 | if (bpf_prog_calls_session_cookie(fentry->nodes[i])) { |
| 1035 | u64 meta = func_meta | (cookie_ctx_off << BPF_TRAMP_COOKIE_INDEX_SHIFT); | 1035 | u64 meta = func_meta | (cookie_ctx_off << BPF_TRAMP_COOKIE_INDEX_SHIFT); |
| 1036 | 1036 | ||
| 1037 | store_func_meta(image, ctx, meta, func_meta_off); | 1037 | store_func_meta(image, ctx, meta, func_meta_off); |
| 1038 | cookie_ctx_off--; | 1038 | cookie_ctx_off--; |
| 1039 | } | 1039 | } |
| 1040 | 1040 | ||
| 1041 | if (invoke_bpf_prog(image, ro_image, ctx, fentry->links[i], regs_off, retval_off, | 1041 | if (invoke_bpf_prog(image, ro_image, ctx, fentry->nodes[i], regs_off, retval_off, |
| 1042 | run_ctx_off, flags & BPF_TRAMP_F_RET_FENTRY_RET)) | 1042 | run_ctx_off, flags & BPF_TRAMP_F_RET_FENTRY_RET)) |
| 1043 | return -EINVAL; | 1043 | return -EINVAL; |
| 1044 | } | 1044 | } |
| 1045 | 1045 | ||
| 1046 | if (fmod_ret->nr_links) { | 1046 | if (fmod_ret->nr_nodes) { |
| 1047 | branches = kcalloc(fmod_ret->nr_links, sizeof(u32), GFP_KERNEL); | 1047 | branches = kcalloc(fmod_ret->nr_nodes, sizeof(u32), GFP_KERNEL); |
| 1048 | if (!branches) | 1048 | if (!branches) |
| 1049 | return -ENOMEM; | 1049 | return -ENOMEM; |
| 1050 | 1050 | ||
| @@ -1093,7 +1093,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 42 unmodified lines | |||
| 1093 | } | 1093 | } |
| 1094 | 1094 | ||
| 1095 | /* Update branches saved in invoke_bpf_mod_ret with address of do_fexit */ | 1095 | /* Update branches saved in invoke_bpf_mod_ret with address of do_fexit */ |
| 1096 | for (i = 0; i < fmod_ret->nr_links && image; i++) { | 1096 | for (i = 0; i < fmod_ret->nr_nodes && image; i++) { |
| 1097 | if (create_cond_branch(&branch_insn, &image[branches[i]], | 1097 | if (create_cond_branch(&branch_insn, &image[branches[i]], |
| 1098 | (unsigned long)&image[ctx->idx], COND_NE << 16)) { | 1098 | (unsigned long)&image[ctx->idx], COND_NE << 16)) { |
| 1099 | ret = -EINVAL; | 1099 | ret = -EINVAL; |
| @@ -1110,15 +1110,15 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 10 unmodified lines | |||
| 1110 | 1110 | ||
| 1111 | cookie_ctx_off = (regs_off - cookie_off) / 8; | 1111 | cookie_ctx_off = (regs_off - cookie_off) / 8; |
| 1112 | 1112 | ||
| 1113 | for (i = 0; i < fexit->nr_links; i++) { | 1113 | for (i = 0; i < fexit->nr_nodes; i++) { |
| 1114 | if (bpf_prog_calls_session_cookie(fexit->links[i])) { | 1114 | if (bpf_prog_calls_session_cookie(fexit->nodes[i])) { |
| 1115 | u64 meta = func_meta | (cookie_ctx_off << BPF_TRAMP_COOKIE_INDEX_SHIFT); | 1115 | u64 meta = func_meta | (cookie_ctx_off << BPF_TRAMP_COOKIE_INDEX_SHIFT); |
| 1116 | 1116 | ||
| 1117 | store_func_meta(image, ctx, meta, func_meta_off); | 1117 | store_func_meta(image, ctx, meta, func_meta_off); |
| 1118 | cookie_ctx_off--; | 1118 | cookie_ctx_off--; |
| 1119 | } | 1119 | } |
| 1120 | 1120 | ||
| 1121 | if (invoke_bpf_prog(image, ro_image, ctx, fexit->links[i], regs_off, retval_off, | 1121 | if (invoke_bpf_prog(image, ro_image, ctx, fexit->nodes[i], regs_off, retval_off, |
| 1122 | run_ctx_off, false)) { | 1122 | run_ctx_off, false)) { |
| 1123 | ret = -EINVAL; | 1123 | ret = -EINVAL; |
| 1124 | goto cleanup; | 1124 | goto cleanup; |
| @@ -1185,18 +1185,18 @@ cleanup: · 60 unmodified lines | |||
| 1185 | } | 1185 | } |
| 1186 | 1186 | ||
| 1187 | int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, | 1187 | int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, |
| 1188 | struct bpf_tramp_links *tlinks, void *func_addr) | 1188 | struct bpf_tramp_nodes *tnodes, void *func_addr) |
| 1189 | { | 1189 | { |
| 1190 | struct bpf_tramp_image im; | 1190 | struct bpf_tramp_image im; |
| 1191 | int ret; | 1191 | int ret; |
| 1192 | 1192 | ||
| 1193 | ret = __arch_prepare_bpf_trampoline(&im, NULL, NULL, NULL, m, flags, tlinks, func_addr); | 1193 | ret = __arch_prepare_bpf_trampoline(&im, NULL, NULL, NULL, m, flags, tnodes, func_addr); |
| 1194 | return ret; | 1194 | return ret; |
| 1195 | } | 1195 | } |
| 1196 | 1196 | ||
| 1197 | int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *image_end, | 1197 | int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *image_end, |
| 1198 | const struct btf_func_model *m, u32 flags, | 1198 | const struct btf_func_model *m, u32 flags, |
| 1199 | struct bpf_tramp_links *tlinks, | 1199 | struct bpf_tramp_nodes *tnodes, |
| 1200 | void *func_addr) | 1200 | void *func_addr) |
| 1201 | { | 1201 | { |
| 1202 | u32 size = image_end - image; | 1202 | u32 size = image_end - image; |
| @@ -1212,7 +1212,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i · 9 unmodified lines | |||
| 1212 | return -ENOMEM; | 1212 | return -ENOMEM; |
| 1213 | 1213 | ||
| 1214 | ret = __arch_prepare_bpf_trampoline(im, rw_image, rw_image + size, image, m, | 1214 | ret = __arch_prepare_bpf_trampoline(im, rw_image, rw_image + size, image, m, |
| 1215 | flags, tlinks, func_addr); | 1215 | flags, tnodes, func_addr); |
| 1216 | if (ret < 0) | 1216 | if (ret < 0) |
| 1217 | goto out; | 1217 | goto out; |
| 1218 | 1218 | ||
| @@ -934,15 +934,15 @@ static void emit_store_stack_imm64(u8 reg, int stack_off, u64 imm64, · 933 unmodified lines | |||
| 934 | emit_sd(RV_REG_FP, stack_off, reg, ctx); | 934 | emit_sd(RV_REG_FP, stack_off, reg, ctx); |
| 935 | } | 935 | } |
| 936 | 936 | ||
| 937 | static int invoke_bpf_prog(struct bpf_tramp_link *l, int args_off, int retval_off, | 937 | static int invoke_bpf_prog(struct bpf_tramp_node *node, int args_off, int retval_off, |
| 938 | int run_ctx_off, bool save_ret, struct rv_jit_context *ctx) | 938 | int run_ctx_off, bool save_ret, struct rv_jit_context *ctx) |
| 939 | { | 939 | { |
| 940 | int ret, branch_off; | 940 | int ret, branch_off; |
| 941 | struct bpf_prog *p = l->link.prog; | 941 | struct bpf_prog *p = node->link->prog; |
| 942 | int cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie); | 942 | int cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie); |
| 943 | 943 | ||
| 944 | if (l->cookie) | 944 | if (node->cookie) |
| 945 | emit_store_stack_imm64(RV_REG_T1, -run_ctx_off + cookie_off, l->cookie, ctx); | 945 | emit_store_stack_imm64(RV_REG_T1, -run_ctx_off + cookie_off, node->cookie, ctx); |
| 946 | else | 946 | else |
| 947 | emit_sd(RV_REG_FP, -run_ctx_off + cookie_off, RV_REG_ZERO, ctx); | 947 | emit_sd(RV_REG_FP, -run_ctx_off + cookie_off, RV_REG_ZERO, ctx); |
| 948 | 948 | ||
| @@ -996,22 +996,22 @@ static int invoke_bpf_prog(struct bpf_tramp_link *l, int args_off, int retval_of · 47 unmodified lines | |||
| 996 | return ret; | 996 | return ret; |
| 997 | } | 997 | } |
| 998 | 998 | ||
| 999 | static int invoke_bpf(struct bpf_tramp_links *tl, int args_off, int retval_off, | 999 | static int invoke_bpf(struct bpf_tramp_nodes *tn, int args_off, int retval_off, |
| 1000 | int run_ctx_off, int func_meta_off, bool save_ret, u64 func_meta, | 1000 | int run_ctx_off, int func_meta_off, bool save_ret, u64 func_meta, |
| 1001 | int cookie_off, struct rv_jit_context *ctx) | 1001 | int cookie_off, struct rv_jit_context *ctx) |
| 1002 | { | 1002 | { |
| 1003 | int i, cur_cookie = (cookie_off - args_off) / 8; | 1003 | int i, cur_cookie = (cookie_off - args_off) / 8; |
| 1004 | 1004 | ||
| 1005 | for (i = 0; i < tl->nr_links; i++) { | 1005 | for (i = 0; i < tn->nr_nodes; i++) { |
| 1006 | int err; | 1006 | int err; |
| 1007 | 1007 | ||
| 1008 | if (bpf_prog_calls_session_cookie(tl->links[i])) { | 1008 | if (bpf_prog_calls_session_cookie(tn->nodes[i])) { |
| 1009 | u64 meta = func_meta | ((u64)cur_cookie << BPF_TRAMP_COOKIE_INDEX_SHIFT); | 1009 | u64 meta = func_meta | ((u64)cur_cookie << BPF_TRAMP_COOKIE_INDEX_SHIFT); |
| 1010 | 1010 | ||
| 1011 | emit_store_stack_imm64(RV_REG_T1, -func_meta_off, meta, ctx); | 1011 | emit_store_stack_imm64(RV_REG_T1, -func_meta_off, meta, ctx); |
| 1012 | cur_cookie--; | 1012 | cur_cookie--; |
| 1013 | } | 1013 | } |
| 1014 | err = invoke_bpf_prog(tl->links[i], args_off, retval_off, run_ctx_off, | 1014 | err = invoke_bpf_prog(tn->nodes[i], args_off, retval_off, run_ctx_off, |
| 1015 | save_ret, ctx); | 1015 | save_ret, ctx); |
| 1016 | if (err) | 1016 | if (err) |
| 1017 | return err; | 1017 | return err; |
| @@ -1021,7 +1021,7 @@ static int invoke_bpf(struct bpf_tramp_links *tl, int args_off, int retval_off, · 3 unmodified lines | |||
| 1021 | 1021 | ||
| 1022 | static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, | 1022 | static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, |
| 1023 | const struct btf_func_model *m, | 1023 | const struct btf_func_model *m, |
| 1024 | struct bpf_tramp_links *tlinks, | 1024 | struct bpf_tramp_nodes *tnodes, |
| 1025 | void *func_addr, u32 flags, | 1025 | void *func_addr, u32 flags, |
| 1026 | struct rv_jit_context *ctx) | 1026 | struct rv_jit_context *ctx) |
| 1027 | { | 1027 | { |
| @@ -1030,9 +1030,9 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, · 2 unmodified lines | |||
| 1030 | int stack_size = 0, nr_arg_slots = 0; | 1030 | int stack_size = 0, nr_arg_slots = 0; |
| 1031 | int retval_off, args_off, func_meta_off, ip_off, run_ctx_off, sreg_off, stk_arg_off; | 1031 | int retval_off, args_off, func_meta_off, ip_off, run_ctx_off, sreg_off, stk_arg_off; |
| 1032 | int cookie_off, cookie_cnt; | 1032 | int cookie_off, cookie_cnt; |
| 1033 | struct bpf_tramp_links *fentry = &tlinks[BPF_TRAMP_FENTRY]; | 1033 | struct bpf_tramp_nodes *fentry = &tnodes[BPF_TRAMP_FENTRY]; |
| 1034 | struct bpf_tramp_links *fexit = &tlinks[BPF_TRAMP_FEXIT]; | 1034 | struct bpf_tramp_nodes *fexit = &tnodes[BPF_TRAMP_FEXIT]; |
| 1035 | struct bpf_tramp_links *fmod_ret = &tlinks[BPF_TRAMP_MODIFY_RETURN]; | 1035 | struct bpf_tramp_nodes *fmod_ret = &tnodes[BPF_TRAMP_MODIFY_RETURN]; |
| 1036 | bool is_struct_ops = flags & BPF_TRAMP_F_INDIRECT; | 1036 | bool is_struct_ops = flags & BPF_TRAMP_F_INDIRECT; |
| 1037 | void *orig_call = func_addr; | 1037 | void *orig_call = func_addr; |
| 1038 | bool save_ret; | 1038 | bool save_ret; |
| @@ -1115,7 +1115,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, · 76 unmodified lines | |||
| 1115 | ip_off = stack_size; | 1115 | ip_off = stack_size; |
| 1116 | } | 1116 | } |
| 1117 | 1117 | ||
| 1118 | cookie_cnt = bpf_fsession_cookie_cnt(tlinks); | 1118 | cookie_cnt = bpf_fsession_cookie_cnt(tnodes); |
| 1119 | /* room for session cookies */ | 1119 | /* room for session cookies */ |
| 1120 | stack_size += cookie_cnt * 8; | 1120 | stack_size += cookie_cnt * 8; |
| 1121 | cookie_off = stack_size; | 1121 | cookie_off = stack_size; |
| @@ -1172,7 +1172,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, · 50 unmodified lines | |||
| 1172 | 1172 | ||
| 1173 | store_args(nr_arg_slots, args_off, ctx); | 1173 | store_args(nr_arg_slots, args_off, ctx); |
| 1174 | 1174 | ||
| 1175 | if (bpf_fsession_cnt(tlinks)) { | 1175 | if (bpf_fsession_cnt(tnodes)) { |
| 1176 | /* clear all session cookies' value */ | 1176 | /* clear all session cookies' value */ |
| 1177 | for (i = 0; i < cookie_cnt; i++) | 1177 | for (i = 0; i < cookie_cnt; i++) |
| 1178 | emit_sd(RV_REG_FP, -cookie_off + 8 * i, RV_REG_ZERO, ctx); | 1178 | emit_sd(RV_REG_FP, -cookie_off + 8 * i, RV_REG_ZERO, ctx); |
| @@ -1187,22 +1187,22 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, · 8 unmodified lines | |||
| 1187 | return ret; | 1187 | return ret; |
| 1188 | } | 1188 | } |
| 1189 | 1189 | ||
| 1190 | if (fentry->nr_links) { | 1190 | if (fentry->nr_nodes) { |
| 1191 | ret = invoke_bpf(fentry, args_off, retval_off, run_ctx_off, func_meta_off, | 1191 | ret = invoke_bpf(fentry, args_off, retval_off, run_ctx_off, func_meta_off, |
| 1192 | flags & BPF_TRAMP_F_RET_FENTRY_RET, func_meta, cookie_off, ctx); | 1192 | flags & BPF_TRAMP_F_RET_FENTRY_RET, func_meta, cookie_off, ctx); |
| 1193 | if (ret) | 1193 | if (ret) |
| 1194 | return ret; | 1194 | return ret; |
| 1195 | } | 1195 | } |
| 1196 | 1196 | ||
| 1197 | if (fmod_ret->nr_links) { | 1197 | if (fmod_ret->nr_nodes) { |
| 1198 | branches_off = kzalloc_objs(int, fmod_ret->nr_links); | 1198 | branches_off = kzalloc_objs(int, fmod_ret->nr_nodes); |
| 1199 | if (!branches_off) | 1199 | if (!branches_off) |
| 1200 | return -ENOMEM; | 1200 | return -ENOMEM; |
| 1201 | 1201 | ||
| 1202 | /* cleanup to avoid garbage return value confusion */ | 1202 | /* cleanup to avoid garbage return value confusion */ |
| 1203 | emit_sd(RV_REG_FP, -retval_off, RV_REG_ZERO, ctx); | 1203 | emit_sd(RV_REG_FP, -retval_off, RV_REG_ZERO, ctx); |
| 1204 | for (i = 0; i < fmod_ret->nr_links; i++) { | 1204 | for (i = 0; i < fmod_ret->nr_nodes; i++) { |
| 1205 | ret = invoke_bpf_prog(fmod_ret->links[i], args_off, retval_off, | 1205 | ret = invoke_bpf_prog(fmod_ret->nodes[i], args_off, retval_off, |
| 1206 | run_ctx_off, true, ctx); | 1206 | run_ctx_off, true, ctx); |
| 1207 | if (ret) | 1207 | if (ret) |
| 1208 | goto out; | 1208 | goto out; |
| @@ -1230,7 +1230,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, · 21 unmodified lines | |||
| 1230 | } | 1230 | } |
| 1231 | 1231 | ||
| 1232 | /* update branches saved in invoke_bpf_mod_ret with bnez */ | 1232 | /* update branches saved in invoke_bpf_mod_ret with bnez */ |
| 1233 | for (i = 0; ctx->insns && i < fmod_ret->nr_links; i++) { | 1233 | for (i = 0; ctx->insns && i < fmod_ret->nr_nodes; i++) { |
| 1234 | offset = ninsns_rvoff(ctx->ninsns - branches_off[i]); | 1234 | offset = ninsns_rvoff(ctx->ninsns - branches_off[i]); |
| 1235 | insn = rv_bne(RV_REG_T1, RV_REG_ZERO, offset >> 1); | 1235 | insn = rv_bne(RV_REG_T1, RV_REG_ZERO, offset >> 1); |
| 1236 | *(u32 *)(ctx->insns + branches_off[i]) = insn; | 1236 | *(u32 *)(ctx->insns + branches_off[i]) = insn; |
| @@ -1238,10 +1238,10 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, · 1 unmodified line | |||
| 1238 | 1238 | ||
| 1239 | /* set "is_return" flag for fsession */ | 1239 | /* set "is_return" flag for fsession */ |
| 1240 | func_meta |= (1ULL << BPF_TRAMP_IS_RETURN_SHIFT); | 1240 | func_meta |= (1ULL << BPF_TRAMP_IS_RETURN_SHIFT); |
| 1241 | if (bpf_fsession_cnt(tlinks)) | 1241 | if (bpf_fsession_cnt(tnodes)) |
| 1242 | emit_store_stack_imm64(RV_REG_T1, -func_meta_off, func_meta, ctx); | 1242 | emit_store_stack_imm64(RV_REG_T1, -func_meta_off, func_meta, ctx); |
| 1243 | 1243 | ||
| 1244 | if (fexit->nr_links) { | 1244 | if (fexit->nr_nodes) { |
| 1245 | ret = invoke_bpf(fexit, args_off, retval_off, run_ctx_off, func_meta_off, | 1245 | ret = invoke_bpf(fexit, args_off, retval_off, run_ctx_off, func_meta_off, |
| 1246 | false, func_meta, cookie_off, ctx); | 1246 | false, func_meta, cookie_off, ctx); |
| 1247 | if (ret) | 1247 | if (ret) |
| @@ -1305,7 +1305,7 @@ out: · 57 unmodified lines | |||
| 1305 | } | 1305 | } |
| 1306 | 1306 | ||
| 1307 | int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, | 1307 | int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, |
| 1308 | struct bpf_tramp_links *tlinks, void *func_addr) | 1308 | struct bpf_tramp_nodes *tnodes, void *func_addr) |
| 1309 | { | 1309 | { |
| 1310 | struct bpf_tramp_image im; | 1310 | struct bpf_tramp_image im; |
| 1311 | struct rv_jit_context ctx; | 1311 | struct rv_jit_context ctx; |
| @@ -1314,7 +1314,7 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, · 2 unmodified lines | |||
| 1314 | ctx.ninsns = 0; | 1314 | ctx.ninsns = 0; |
| 1315 | ctx.insns = NULL; | 1315 | ctx.insns = NULL; |
| 1316 | ctx.ro_insns = NULL; | 1316 | ctx.ro_insns = NULL; |
| 1317 | ret = __arch_prepare_bpf_trampoline(&im, m, tlinks, func_addr, flags, &ctx); | 1317 | ret = __arch_prepare_bpf_trampoline(&im, m, tnodes, func_addr, flags, &ctx); |
| 1318 | 1318 | ||
| 1319 | return ret < 0 ? ret : ninsns_rvoff(ctx.ninsns); | 1319 | return ret < 0 ? ret : ninsns_rvoff(ctx.ninsns); |
| 1320 | } | 1320 | } |
| @@ -1331,7 +1331,7 @@ void arch_free_bpf_trampoline(void *image, unsigned int size) · 10 unmodified lines | |||
| 1331 | 1331 | ||
| 1332 | int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image, | 1332 | int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image, |
| 1333 | void *ro_image_end, const struct btf_func_model *m, | 1333 | void *ro_image_end, const struct btf_func_model *m, |
| 1334 | u32 flags, struct bpf_tramp_links *tlinks, | 1334 | u32 flags, struct bpf_tramp_nodes *tnodes, |
| 1335 | void *func_addr) | 1335 | void *func_addr) |
| 1336 | { | 1336 | { |
| 1337 | int ret; | 1337 | int ret; |
| @@ -1346,7 +1346,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image, · 8 unmodified lines | |||
| 1346 | ctx.ninsns = 0; | 1346 | ctx.ninsns = 0; |
| 1347 | ctx.insns = image; | 1347 | ctx.insns = image; |
| 1348 | ctx.ro_insns = ro_image; | 1348 | ctx.ro_insns = ro_image; |
| 1349 | ret = __arch_prepare_bpf_trampoline(im, m, tlinks, func_addr, flags, &ctx); | 1349 | ret = __arch_prepare_bpf_trampoline(im, m, tnodes, func_addr, flags, &ctx); |
| 1350 | if (ret < 0) | 1350 | if (ret < 0) |
| 1351 | goto out; | 1351 | goto out; |
| 1352 | 1352 | ||
| @@ -1808,6 +1808,13 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, · 455 unmodified lines | |||
| 1808 | break; | 1808 | break; |
| 1809 | } | 1809 | } |
| 1810 | 1810 | ||
| 1811 | /* Implement helper call to bpf_get_current_task/_btf() inline */ | ||
| 1812 | if (insn->src_reg == 0 && (insn->imm == BPF_FUNC_get_current_task || | ||
| 1813 | insn->imm == BPF_FUNC_get_current_task_btf)) { | ||
| 1814 | emit_mv(bpf_to_rv_reg(BPF_REG_0, ctx), RV_REG_TP, ctx); | ||
| 1815 | break; | ||
| 1816 | } | ||
| 1817 | |||
| 1811 | mark_call(ctx); | 1818 | mark_call(ctx); |
| 1812 | ret = bpf_jit_get_func_addr(ctx->prog, insn, extra_pass, | 1819 | ret = bpf_jit_get_func_addr(ctx->prog, insn, extra_pass, |
| 1813 | &addr, &fixed_addr); | 1820 | &addr, &fixed_addr); |
| @@ -2138,6 +2145,8 @@ bool bpf_jit_inlines_helper_call(s32 imm) · 324 unmodified lines | |||
| 2138 | { | 2145 | { |
| 2139 | switch (imm) { | 2146 | switch (imm) { |
| 2140 | case BPF_FUNC_get_smp_processor_id: | 2147 | case BPF_FUNC_get_smp_processor_id: |
| 2148 | case BPF_FUNC_get_current_task: | ||
| 2149 | case BPF_FUNC_get_current_task_btf: | ||
| 2141 | return true; | 2150 | return true; |
| 2142 | default: | 2151 | default: |
| 2143 | return false; | 2152 | return false; |
| @@ -27,6 +27,7 @@ · 26 unmodified lines | |||
| 27 | #include <asm/extable.h> | 27 | #include <asm/extable.h> |
| 28 | #include <asm/dis.h> | 28 | #include <asm/dis.h> |
| 29 | #include <asm/facility.h> | 29 | #include <asm/facility.h> |
| 30 | #include <asm/lowcore.h> | ||
| 30 | #include <asm/nospec-branch.h> | 31 | #include <asm/nospec-branch.h> |
| 31 | #include <asm/set_memory.h> | 32 | #include <asm/set_memory.h> |
| 32 | #include <asm/text-patching.h> | 33 | #include <asm/text-patching.h> |
| @@ -1777,6 +1778,30 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp, · 1744 unmodified lines | |||
| 1777 | int j, ret; | 1778 | int j, ret; |
| 1778 | u64 func; | 1779 | u64 func; |
| 1779 | 1780 | ||
| 1781 | /* Implement helper call to bpf_get_smp_processor_id() inline */ | ||
| 1782 | if (insn->src_reg == 0 && | ||
| 1783 | insn->imm == BPF_FUNC_get_smp_processor_id) { | ||
| 1784 | const u32 *cpu_nr = &get_lowcore()->cpu_nr; | ||
| 1785 | |||
| 1786 | /* ly %b0, cpu_nr */ | ||
| 1787 | EMIT6_DISP_LH(0xe3000000, 0x0058, BPF_REG_0, REG_0, REG_0, | ||
| 1788 | (unsigned long)cpu_nr); | ||
| 1789 | break; | ||
| 1790 | } | ||
| 1791 | |||
| 1792 | /* Implement helper call to bpf_get_current_task/_btf() inline */ | ||
| 1793 | if (insn->src_reg == 0 && | ||
| 1794 | (insn->imm == BPF_FUNC_get_current_task || | ||
| 1795 | insn->imm == BPF_FUNC_get_current_task_btf)) { | ||
| 1796 | const u64 *current_task = | ||
| 1797 | &get_lowcore()->current_task; | ||
| 1798 | |||
| 1799 | /* lg %b0, current_task */ | ||
| 1800 | EMIT6_DISP_LH(0xe3000000, 0x0004, BPF_REG_0, REG_0, REG_0, | ||
| 1801 | (unsigned long)current_task); | ||
| 1802 | break; | ||
| 1803 | } | ||
| 1804 | |||
| 1780 | ret = bpf_jit_get_func_addr(fp, insn, extra_pass, | 1805 | ret = bpf_jit_get_func_addr(fp, insn, extra_pass, |
| 1781 | &func, &func_addr_fixed); | 1806 | &func, &func_addr_fixed); |
| 1782 | if (ret < 0) | 1807 | if (ret < 0) |
| @@ -2512,19 +2537,19 @@ static void emit_store_stack_imm64(struct bpf_jit *jit, int tmp_reg, int stack_o · 729 unmodified lines | |||
| 2512 | 2537 | ||
| 2513 | static int invoke_bpf_prog(struct bpf_tramp_jit *tjit, | 2538 | static int invoke_bpf_prog(struct bpf_tramp_jit *tjit, |
| 2514 | const struct btf_func_model *m, | 2539 | const struct btf_func_model *m, |
| 2515 | struct bpf_tramp_link *tlink, bool save_ret) | 2540 | struct bpf_tramp_node *node, bool save_ret) |
| 2516 | { | 2541 | { |
| 2517 | struct bpf_jit *jit = &tjit->common; | 2542 | struct bpf_jit *jit = &tjit->common; |
| 2518 | int cookie_off = tjit->run_ctx_off + | 2543 | int cookie_off = tjit->run_ctx_off + |
| 2519 | offsetof(struct bpf_tramp_run_ctx, bpf_cookie); | 2544 | offsetof(struct bpf_tramp_run_ctx, bpf_cookie); |
| 2520 | struct bpf_prog *p = tlink->link.prog; | 2545 | struct bpf_prog *p = node->link->prog; |
| 2521 | int patch; | 2546 | int patch; |
| 2522 | 2547 | ||
| 2523 | /* | 2548 | /* |
| 2524 | * run_ctx.cookie = tlink->cookie; | 2549 | * run_ctx.cookie = node->cookie; |
| 2525 | */ | 2550 | */ |
| 2526 | 2551 | ||
| 2527 | emit_store_stack_imm64(jit, REG_W0, cookie_off, tlink->cookie); | 2552 | emit_store_stack_imm64(jit, REG_W0, cookie_off, node->cookie); |
| 2528 | 2553 | ||
| 2529 | /* | 2554 | /* |
| 2530 | * if ((start = __bpf_prog_enter(p, &run_ctx)) == 0) | 2555 | * if ((start = __bpf_prog_enter(p, &run_ctx)) == 0) |
| @@ -2584,20 +2609,20 @@ static int invoke_bpf_prog(struct bpf_tramp_jit *tjit, · 53 unmodified lines | |||
| 2584 | 2609 | ||
| 2585 | static int invoke_bpf(struct bpf_tramp_jit *tjit, | 2610 | static int invoke_bpf(struct bpf_tramp_jit *tjit, |
| 2586 | const struct btf_func_model *m, | 2611 | const struct btf_func_model *m, |
| 2587 | struct bpf_tramp_links *tl, bool save_ret, | 2612 | struct bpf_tramp_nodes *tn, bool save_ret, |
| 2588 | u64 func_meta, int cookie_off) | 2613 | u64 func_meta, int cookie_off) |
| 2589 | { | 2614 | { |
| 2590 | int i, cur_cookie = (tjit->bpf_args_off - cookie_off) / sizeof(u64); | 2615 | int i, cur_cookie = (tjit->bpf_args_off - cookie_off) / sizeof(u64); |
| 2591 | struct bpf_jit *jit = &tjit->common; | 2616 | struct bpf_jit *jit = &tjit->common; |
| 2592 | 2617 | ||
| 2593 | for (i = 0; i < tl->nr_links; i++) { | 2618 | for (i = 0; i < tn->nr_nodes; i++) { |
| 2594 | if (bpf_prog_calls_session_cookie(tl->links[i])) { | 2619 | if (bpf_prog_calls_session_cookie(tn->nodes[i])) { |
| 2595 | u64 meta = func_meta | ((u64)cur_cookie << BPF_TRAMP_COOKIE_INDEX_SHIFT); | 2620 | u64 meta = func_meta | ((u64)cur_cookie << BPF_TRAMP_COOKIE_INDEX_SHIFT); |
| 2596 | 2621 | ||
| 2597 | emit_store_stack_imm64(jit, REG_0, tjit->func_meta_off, meta); | 2622 | emit_store_stack_imm64(jit, REG_0, tjit->func_meta_off, meta); |
| 2598 | cur_cookie--; | 2623 | cur_cookie--; |
| 2599 | } | 2624 | } |
| 2600 | if (invoke_bpf_prog(tjit, m, tl->links[i], save_ret)) | 2625 | if (invoke_bpf_prog(tjit, m, tn->nodes[i], save_ret)) |
| 2601 | return -EINVAL; | 2626 | return -EINVAL; |
| 2602 | } | 2627 | } |
| 2603 | 2628 | ||
| @@ -2626,12 +2651,12 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, · 22 unmodified lines | |||
| 2626 | struct bpf_tramp_jit *tjit, | 2651 | struct bpf_tramp_jit *tjit, |
| 2627 | const struct btf_func_model *m, | 2652 | const struct btf_func_model *m, |
| 2628 | u32 flags, | 2653 | u32 flags, |
| 2629 | struct bpf_tramp_links *tlinks, | 2654 | struct bpf_tramp_nodes *tnodes, |
| 2630 | void *func_addr) | 2655 | void *func_addr) |
| 2631 | { | 2656 | { |
| 2632 | struct bpf_tramp_links *fmod_ret = &tlinks[BPF_TRAMP_MODIFY_RETURN]; | 2657 | struct bpf_tramp_nodes *fmod_ret = &tnodes[BPF_TRAMP_MODIFY_RETURN]; |
| 2633 | struct bpf_tramp_links *fentry = &tlinks[BPF_TRAMP_FENTRY]; | 2658 | struct bpf_tramp_nodes *fentry = &tnodes[BPF_TRAMP_FENTRY]; |
| 2634 | struct bpf_tramp_links *fexit = &tlinks[BPF_TRAMP_FEXIT]; | 2659 | struct bpf_tramp_nodes *fexit = &tnodes[BPF_TRAMP_FEXIT]; |
| 2635 | int nr_bpf_args, nr_reg_args, nr_stack_args; | 2660 | int nr_bpf_args, nr_reg_args, nr_stack_args; |
| 2636 | int cookie_cnt, cookie_off, fsession_cnt; | 2661 | int cookie_cnt, cookie_off, fsession_cnt; |
| 2637 | struct bpf_jit *jit = &tjit->common; | 2662 | struct bpf_jit *jit = &tjit->common; |
| @@ -2668,8 +2693,8 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, · 30 unmodified lines | |||
| 2668 | return -ENOTSUPP; | 2693 | return -ENOTSUPP; |
| 2669 | } | 2694 | } |
| 2670 | 2695 | ||
| 2671 | cookie_cnt = bpf_fsession_cookie_cnt(tlinks); | 2696 | cookie_cnt = bpf_fsession_cookie_cnt(tnodes); |
| 2672 | fsession_cnt = bpf_fsession_cnt(tlinks); | 2697 | fsession_cnt = bpf_fsession_cnt(tnodes); |
| 2673 | 2698 | ||
| 2674 | /* | 2699 | /* |
| 2675 | * Calculate the stack layout. | 2700 | * Calculate the stack layout. |
| @@ -2804,7 +2829,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, · 128 unmodified lines | |||
| 2804 | func_meta, cookie_off)) | 2829 | func_meta, cookie_off)) |
| 2805 | return -EINVAL; | 2830 | return -EINVAL; |
| 2806 | 2831 | ||
| 2807 | if (fmod_ret->nr_links) { | 2832 | if (fmod_ret->nr_nodes) { |
| 2808 | /* | 2833 | /* |
| 2809 | * retval = 0; | 2834 | * retval = 0; |
| 2810 | */ | 2835 | */ |
| @@ -2813,8 +2838,8 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, · 2 unmodified lines | |||
| 2813 | _EMIT6(0xd707f000 | tjit->retval_off, | 2838 | _EMIT6(0xd707f000 | tjit->retval_off, |
| 2814 | 0xf000 | tjit->retval_off); | 2839 | 0xf000 | tjit->retval_off); |
| 2815 | 2840 | ||
| 2816 | for (i = 0; i < fmod_ret->nr_links; i++) { | 2841 | for (i = 0; i < fmod_ret->nr_nodes; i++) { |
| 2817 | if (invoke_bpf_prog(tjit, m, fmod_ret->links[i], true)) | 2842 | if (invoke_bpf_prog(tjit, m, fmod_ret->nodes[i], true)) |
| 2818 | return -EINVAL; | 2843 | return -EINVAL; |
| 2819 | 2844 | ||
| 2820 | /* | 2845 | /* |
| @@ -2939,7 +2964,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, · 118 unmodified lines | |||
| 2939 | } | 2964 | } |
| 2940 | 2965 | ||
| 2941 | int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, | 2966 | int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, |
| 2942 | struct bpf_tramp_links *tlinks, void *orig_call) | 2967 | struct bpf_tramp_nodes *tnodes, void *orig_call) |
| 2943 | { | 2968 | { |
| 2944 | struct bpf_tramp_image im; | 2969 | struct bpf_tramp_image im; |
| 2945 | struct bpf_tramp_jit tjit; | 2970 | struct bpf_tramp_jit tjit; |
| @@ -2948,14 +2973,14 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, · 2 unmodified lines | |||
| 2948 | memset(&tjit, 0, sizeof(tjit)); | 2973 | memset(&tjit, 0, sizeof(tjit)); |
| 2949 | 2974 | ||
| 2950 | ret = __arch_prepare_bpf_trampoline(&im, &tjit, m, flags, | 2975 | ret = __arch_prepare_bpf_trampoline(&im, &tjit, m, flags, |
| 2951 | tlinks, orig_call); | 2976 | tnodes, orig_call); |
| 2952 | 2977 | ||
| 2953 | return ret < 0 ? ret : tjit.common.prg; | 2978 | return ret < 0 ? ret : tjit.common.prg; |
| 2954 | } | 2979 | } |
| 2955 | 2980 | ||
| 2956 | int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, | 2981 | int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, |
| 2957 | void *image_end, const struct btf_func_model *m, | 2982 | void *image_end, const struct btf_func_model *m, |
| 2958 | u32 flags, struct bpf_tramp_links *tlinks, | 2983 | u32 flags, struct bpf_tramp_nodes *tnodes, |
| 2959 | void *func_addr) | 2984 | void *func_addr) |
| 2960 | { | 2985 | { |
| 2961 | struct bpf_tramp_jit tjit; | 2986 | struct bpf_tramp_jit tjit; |
| @@ -2964,7 +2989,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, · 2 unmodified lines | |||
| 2964 | /* Compute offsets, check whether the code fits. */ | 2989 | /* Compute offsets, check whether the code fits. */ |
| 2965 | memset(&tjit, 0, sizeof(tjit)); | 2990 | memset(&tjit, 0, sizeof(tjit)); |
| 2966 | ret = __arch_prepare_bpf_trampoline(im, &tjit, m, flags, | 2991 | ret = __arch_prepare_bpf_trampoline(im, &tjit, m, flags, |
| 2967 | tlinks, func_addr); | 2992 | tnodes, func_addr); |
| 2968 | 2993 | ||
| 2969 | if (ret < 0) | 2994 | if (ret < 0) |
| 2970 | return ret; | 2995 | return ret; |
| @@ -2978,7 +3003,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, · 7 unmodified lines | |||
| 2978 | tjit.common.prg = 0; | 3003 | tjit.common.prg = 0; |
| 2979 | tjit.common.prg_buf = image; | 3004 | tjit.common.prg_buf = image; |
| 2980 | ret = __arch_prepare_bpf_trampoline(im, &tjit, m, flags, | 3005 | ret = __arch_prepare_bpf_trampoline(im, &tjit, m, flags, |
| 2981 | tlinks, func_addr); | 3006 | tnodes, func_addr); |
| 2982 | 3007 | ||
| 2983 | return ret < 0 ? ret : tjit.common.prg; | 3008 | return ret < 0 ? ret : tjit.common.prg; |
| 2984 | } | 3009 | } |
| @@ -3057,3 +3082,15 @@ bool bpf_jit_supports_timed_may_goto(void) · 72 unmodified lines | |||
| 3057 | { | 3082 | { |
| 3058 | return true; | 3083 | return true; |
| 3059 | } | 3084 | } |
| 3085 | |||
| 3086 | bool bpf_jit_inlines_helper_call(s32 imm) | ||
| 3087 | { | ||
| 3088 | switch (imm) { | ||
| 3089 | case BPF_FUNC_get_smp_processor_id: | ||
| 3090 | case BPF_FUNC_get_current_task: | ||
| 3091 | case BPF_FUNC_get_current_task_btf: | ||
| 3092 | return true; | ||
| 3093 | default: | ||
| 3094 | return false; | ||
| 3095 | } | ||
| 3096 | } | ||
| @@ -1284,6 +1284,18 @@ static inline void ptep_set_wrprotect(struct mm_struct *mm, · 1283 unmodified lines | |||
| 1284 | } while (!try_cmpxchg((long *)&ptep->pte, (long *)&old_pte, *(long *)&new_pte)); | 1284 | } while (!try_cmpxchg((long *)&ptep->pte, (long *)&old_pte, *(long *)&new_pte)); |
| 1285 | } | 1285 | } |
| 1286 | 1286 | ||
| 1287 | /* | ||
| 1288 | * Note: strictly-zero compare is narrower than pte_none(), but the gap is | ||
| 1289 | * harmless: _PAGE_DIRTY and _PAGE_ACCESSED aren't set on untouched kernel PTEs. | ||
| 1290 | */ | ||
| 1291 | static inline bool ptep_try_set(pte_t *ptep, pte_t new_pte) | ||
| 1292 | { | ||
| 1293 | pte_t old_pte = __pte(0); | ||
| 1294 | |||
| 1295 | return try_cmpxchg((long *)&ptep->pte, (long *)&old_pte, *(long *)&new_pte); | ||
| 1296 | } | ||
| 1297 | #define ptep_try_set ptep_try_set | ||
| 1298 | |||
| 1287 | #define flush_tlb_fix_spurious_fault(vma, address, ptep) do { } while (0) | 1299 | #define flush_tlb_fix_spurious_fault(vma, address, ptep) do { } while (0) |
| 1288 | 1300 | ||
| 1289 | #define __HAVE_ARCH_PMDP_SET_ACCESS_FLAGS | 1301 | #define __HAVE_ARCH_PMDP_SET_ACCESS_FLAGS |
| @@ -8,6 +8,7 @@ · 7 unmodified lines | |||
| 8 | #include <linux/sched/task_stack.h> /* task_stack_*(), ... */ | 8 | #include <linux/sched/task_stack.h> /* task_stack_*(), ... */ |
| 9 | #include <linux/kdebug.h> /* oops_begin/end, ... */ | 9 | #include <linux/kdebug.h> /* oops_begin/end, ... */ |
| 10 | #include <linux/memblock.h> /* max_low_pfn */ | 10 | #include <linux/memblock.h> /* max_low_pfn */ |
| 11 | #include <linux/bpf_defs.h> /* bpf_arena_handle_page_fault */ | ||
| 11 | #include <linux/kfence.h> /* kfence_handle_page_fault */ | 12 | #include <linux/kfence.h> /* kfence_handle_page_fault */ |
| 12 | #include <linux/kprobes.h> /* NOKPROBE_SYMBOL, ... */ | 13 | #include <linux/kprobes.h> /* NOKPROBE_SYMBOL, ... */ |
| 13 | #include <linux/mmiotrace.h> /* kmmio_handler, ... */ | 14 | #include <linux/mmiotrace.h> /* kmmio_handler, ... */ |
| @@ -693,10 +694,13 @@ page_fault_oops(struct pt_regs *regs, unsigned long error_code, · 679 unmodified lines | |||
| 693 | if (IS_ENABLED(CONFIG_EFI)) | 694 | if (IS_ENABLED(CONFIG_EFI)) |
| 694 | efi_crash_gracefully_on_page_fault(address, regs); | 695 | efi_crash_gracefully_on_page_fault(address, regs); |
| 695 | 696 | ||
| 696 | /* Only not-present faults should be handled by KFENCE. */ | 697 | /* Only not-present faults should be handled by KFENCE or BPF arena. */ |
| 697 | if (!(error_code & X86_PF_PROT) && | 698 | if (!(error_code & X86_PF_PROT)) { |
| 698 | kfence_handle_page_fault(address, error_code & X86_PF_WRITE, regs)) | 699 | if (kfence_handle_page_fault(address, error_code & X86_PF_WRITE, regs)) |
| 699 | return; | 700 | return; |
| 701 | if (bpf_arena_handle_page_fault(address, error_code & X86_PF_WRITE, regs->ip)) | ||
| 702 | return; | ||
| 703 | } | ||
| 700 | 704 | ||
| 701 | oops: | 705 | oops: |
| 702 | /* | 706 | /* |
| @@ -10,6 +10,7 @@ · 9 unmodified lines | |||
| 10 | #include <linux/if_vlan.h> | 10 | #include <linux/if_vlan.h> |
| 11 | #include <linux/bitfield.h> | 11 | #include <linux/bitfield.h> |
| 12 | #include <linux/bpf.h> | 12 | #include <linux/bpf.h> |
| 13 | #include <linux/bpf_verifier.h> | ||
| 13 | #include <linux/memory.h> | 14 | #include <linux/memory.h> |
| 14 | #include <linux/sort.h> | 15 | #include <linux/sort.h> |
| 15 | #include <asm/extable.h> | 16 | #include <asm/extable.h> |
| @@ -390,6 +391,34 @@ static void pop_callee_regs(u8 **pprog, bool *callee_regs_used) · 374 unmodified lines | |||
| 390 | *pprog = prog; | 391 | *pprog = prog; |
| 391 | } | 392 | } |
| 392 | 393 | ||
| 394 | /* add rsp, depth */ | ||
| 395 | static void emit_add_rsp(u8 **pprog, u16 depth) | ||
| 396 | { | ||
| 397 | u8 *prog = *pprog; | ||
| 398 | |||
| 399 | if (!depth) | ||
| 400 | return; | ||
| 401 | if (is_imm8(depth)) | ||
| 402 | EMIT4(0x48, 0x83, 0xC4, depth); /* add rsp, imm8 */ | ||
| 403 | else | ||
| 404 | EMIT3_off32(0x48, 0x81, 0xC4, depth); /* add rsp, imm32 */ | ||
| 405 | *pprog = prog; | ||
| 406 | } | ||
| 407 | |||
| 408 | /* sub rsp, depth */ | ||
| 409 | static void emit_sub_rsp(u8 **pprog, u16 depth) | ||
| 410 | { | ||
| 411 | u8 *prog = *pprog; | ||
| 412 | |||
| 413 | if (!depth) | ||
| 414 | return; | ||
| 415 | if (is_imm8(depth)) | ||
| 416 | EMIT4(0x48, 0x83, 0xEC, depth); /* sub rsp, imm8 */ | ||
| 417 | else | ||
| 418 | EMIT3_off32(0x48, 0x81, 0xEC, depth); /* sub rsp, imm32 */ | ||
| 419 | *pprog = prog; | ||
| 420 | } | ||
| 421 | |||
| 393 | static void emit_nops(u8 **pprog, int len) | 422 | static void emit_nops(u8 **pprog, int len) |
| 394 | { | 423 | { |
| 395 | u8 *prog = *pprog; | 424 | u8 *prog = *pprog; |
| @@ -1659,21 +1688,47 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int * · 1263 unmodified lines | |||
| 1659 | bool seen_exit = false; | 1688 | bool seen_exit = false; |
| 1660 | u8 temp[BPF_MAX_INSN_SIZE + BPF_INSN_SAFETY]; | 1689 | u8 temp[BPF_MAX_INSN_SIZE + BPF_INSN_SAFETY]; |
| 1661 | void __percpu *priv_frame_ptr = NULL; | 1690 | void __percpu *priv_frame_ptr = NULL; |
| 1691 | u16 out_stack_arg_cnt, outgoing_rsp; | ||
| 1662 | u64 arena_vm_start, user_vm_start; | 1692 | u64 arena_vm_start, user_vm_start; |
| 1663 | void __percpu *priv_stack_ptr; | 1693 | void __percpu *priv_stack_ptr; |
| 1664 | int i, excnt = 0; | 1694 | int i, excnt = 0; |
| 1665 | int ilen, proglen = 0; | 1695 | int ilen, proglen = 0; |
| 1666 | u8 *ip, *prog = temp; | 1696 | u8 *ip, *prog = temp; |
| 1667 | u32 stack_depth; | 1697 | u32 stack_depth; |
| 1698 | int callee_saved_size; | ||
| 1699 | s32 outgoing_arg_base; | ||
| 1668 | int err; | 1700 | int err; |
| 1669 | 1701 | ||
| 1670 | stack_depth = bpf_prog->aux->stack_depth; | 1702 | stack_depth = bpf_prog->aux->stack_depth; |
| 1703 | out_stack_arg_cnt = bpf_out_stack_arg_cnt(env, bpf_prog); | ||
| 1671 | priv_stack_ptr = bpf_prog->aux->priv_stack_ptr; | 1704 | priv_stack_ptr = bpf_prog->aux->priv_stack_ptr; |
| 1672 | if (priv_stack_ptr) { | 1705 | if (priv_stack_ptr) { |
| 1673 | priv_frame_ptr = priv_stack_ptr + PRIV_STACK_GUARD_SZ + round_up(stack_depth, 8); | 1706 | priv_frame_ptr = priv_stack_ptr + PRIV_STACK_GUARD_SZ + round_up(stack_depth, 8); |
| 1674 | stack_depth = 0; | 1707 | stack_depth = 0; |
| 1675 | } | 1708 | } |
| 1676 | 1709 | ||
| 1710 | /* | ||
| 1711 | * Follow x86-64 calling convention for both BPF-to-BPF and | ||
| 1712 | * kfunc calls: | ||
| 1713 | * - Arg 6 is passed in R9 register | ||
| 1714 | * - Args 7+ are passed on the stack at [rsp] | ||
| 1715 | * | ||
| 1716 | * Incoming arg 6 is read from R9 (BPF r11+8 → MOV from R9). | ||
| 1717 | * Incoming args 7+ are read from [rbp + 16], [rbp + 24], ... | ||
| 1718 | * (BPF r11+16, r11+24, ... map directly with no offset change). | ||
| 1719 | * | ||
| 1720 | * tail_call_reachable is rejected by the verifier and priv_stack | ||
| 1721 | * is disabled by the JIT when stack args exist, so R9 is always | ||
| 1722 | * available. | ||
| 1723 | * | ||
| 1724 | * Stack layout (high to low): | ||
| 1725 | * [rbp + 16 + ...] incoming stack args 7+ (from caller) | ||
| 1726 | * [rbp + 8] return address | ||
| 1727 | * [rbp] saved rbp | ||
| 1728 | * [rbp - prog_stack] program stack | ||
| 1729 | * [below] callee-saved regs | ||
| 1730 | * [below] outgoing args 7+ (= rsp) | ||
| 1731 | */ | ||
| 1677 | arena_vm_start = bpf_arena_get_kern_vm_start(bpf_prog->aux->arena); | 1732 | arena_vm_start = bpf_arena_get_kern_vm_start(bpf_prog->aux->arena); |
| 1678 | user_vm_start = bpf_arena_get_user_vm_start(bpf_prog->aux->arena); | 1733 | user_vm_start = bpf_arena_get_user_vm_start(bpf_prog->aux->arena); |
| 1679 | 1734 | ||
| @@ -1700,6 +1755,44 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int * · 20 unmodified lines | |||
| 1700 | push_r12(&prog); | 1755 | push_r12(&prog); |
| 1701 | push_callee_regs(&prog, callee_regs_used); | 1756 | push_callee_regs(&prog, callee_regs_used); |
| 1702 | } | 1757 | } |
| 1758 | |||
| 1759 | /* Compute callee-saved register area size. */ | ||
| 1760 | callee_saved_size = 0; | ||
| 1761 | if (bpf_prog->aux->exception_boundary || arena_vm_start) | ||
| 1762 | callee_saved_size += 8; /* r12 */ | ||
| 1763 | if (bpf_prog->aux->exception_boundary) { | ||
| 1764 | callee_saved_size += 4 * 8; /* rbx, r13, r14, r15 */ | ||
| 1765 | } else { | ||
| 1766 | int j; | ||
| 1767 | |||
| 1768 | for (j = 0; j < 4; j++) | ||
| 1769 | if (callee_regs_used[j]) | ||
| 1770 | callee_saved_size += 8; | ||
| 1771 | } | ||
| 1772 | /* | ||
| 1773 | * Base offset from rbp for translating BPF outgoing args 7+ | ||
| 1774 | * to native offsets. BPF uses negative offsets from r11 | ||
| 1775 | * (r11-8 for arg6, r11-16 for arg7, ...) while x86 uses | ||
| 1776 | * positive offsets from rsp ([rsp+0] for arg7, [rsp+8] for | ||
| 1777 | * arg8, ...). Arg 6 goes to R9 directly. | ||
| 1778 | * | ||
| 1779 | * The translation reverses direction: | ||
| 1780 | * native_off = outgoing_arg_base - outgoing_rsp - bpf_off - 16 | ||
| 1781 | * | ||
| 1782 | * Note that tail_call_reachable is guaranteed to be false when | ||
| 1783 | * stack args exist, so tcc pushes need not be accounted for. | ||
| 1784 | */ | ||
| 1785 | outgoing_arg_base = -(round_up(stack_depth, 8) + callee_saved_size); | ||
| 1786 | |||
| 1787 | /* | ||
| 1788 | * Allocate outgoing stack arg area for args 7+ only. | ||
| 1789 | * Arg 6 goes into r9 register, not on stack. | ||
| 1790 | */ | ||
| 1791 | outgoing_rsp = out_stack_arg_cnt > 1 ? (out_stack_arg_cnt - 1) * 8 : 0; | ||
| 1792 | if (bpf_prog->aux->exception_boundary) | ||
| 1793 | bpf_prog->aux->stack_arg_sp_adjust = outgoing_rsp; | ||
| 1794 | emit_sub_rsp(&prog, outgoing_rsp); | ||
| 1795 | |||
| 1703 | if (arena_vm_start) | 1796 | if (arena_vm_start) |
| 1704 | emit_mov_imm64(&prog, X86_REG_R12, | 1797 | emit_mov_imm64(&prog, X86_REG_R12, |
| 1705 | arena_vm_start >> 32, (u32) arena_vm_start); | 1798 | arena_vm_start >> 32, (u32) arena_vm_start); |
| @@ -1721,7 +1814,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int * · 15 unmodified lines | |||
| 1721 | u8 b2 = 0, b3 = 0; | 1814 | u8 b2 = 0, b3 = 0; |
| 1722 | u8 *start_of_ldx; | 1815 | u8 *start_of_ldx; |
| 1723 | s64 jmp_offset; | 1816 | s64 jmp_offset; |
| 1724 | s16 insn_off; | 1817 | s32 insn_off; |
| 1725 | u8 jmp_cond; | 1818 | u8 jmp_cond; |
| 1726 | u8 *func; | 1819 | u8 *func; |
| 1727 | int nops; | 1820 | int nops; |
| @@ -2134,12 +2227,27 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int * · 406 unmodified lines | |||
| 2134 | EMIT1(0xC7); | 2227 | EMIT1(0xC7); |
| 2135 | goto st; | 2228 | goto st; |
| 2136 | case BPF_ST | BPF_MEM | BPF_DW: | 2229 | case BPF_ST | BPF_MEM | BPF_DW: |
| 2230 | if (dst_reg == BPF_REG_PARAMS && insn->off == -8) { | ||
| 2231 | /* Arg 6: store immediate in r9 register */ | ||
| 2232 | emit_mov_imm64(&prog, X86_REG_R9, imm32 >> 31, (u32)imm32); | ||
| 2233 | break; | ||
| 2234 | } | ||
| 2137 | EMIT2(add_1mod(0x48, dst_reg), 0xC7); | 2235 | EMIT2(add_1mod(0x48, dst_reg), 0xC7); |
| 2138 | 2236 | ||
| 2139 | st: if (is_imm8(insn->off)) | 2237 | st: insn_off = insn->off; |
| 2140 | EMIT2(add_1reg(0x40, dst_reg), insn->off); | 2238 | if (dst_reg == BPF_REG_PARAMS) { |
| 2239 | /* | ||
| 2240 | * Args 7+: reverse BPF negative offsets to | ||
| 2241 | * x86 positive rsp offsets. | ||
| 2242 | * BPF off=-16 → [rsp+0], off=-24 → [rsp+8], ... | ||
| 2243 | */ | ||
| 2244 | insn_off = outgoing_arg_base - outgoing_rsp - insn_off - 16; | ||
| 2245 | dst_reg = BPF_REG_FP; | ||
| 2246 | } | ||
| 2247 | if (is_imm8(insn_off)) | ||
| 2248 | EMIT2(add_1reg(0x40, dst_reg), insn_off); | ||
| 2141 | else | 2249 | else |
| 2142 | EMIT1_off32(add_1reg(0x80, dst_reg), insn->off); | 2250 | EMIT1_off32(add_1reg(0x80, dst_reg), insn_off); |
| 2143 | 2251 | ||
| 2144 | EMIT(imm32, bpf_size_to_x86_bytes(BPF_SIZE(insn->code))); | 2252 | EMIT(imm32, bpf_size_to_x86_bytes(BPF_SIZE(insn->code))); |
| 2145 | break; | 2253 | break; |
| @@ -2149,7 +2257,17 @@ st: if (is_imm8(insn->off)) · 3 unmodified lines | |||
| 2149 | case BPF_STX | BPF_MEM | BPF_H: | 2257 | case BPF_STX | BPF_MEM | BPF_H: |
| 2150 | case BPF_STX | BPF_MEM | BPF_W: | 2258 | case BPF_STX | BPF_MEM | BPF_W: |
| 2151 | case BPF_STX | BPF_MEM | BPF_DW: | 2259 | case BPF_STX | BPF_MEM | BPF_DW: |
| 2152 | emit_stx(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn->off); | 2260 | if (dst_reg == BPF_REG_PARAMS && insn->off == -8) { |
| 2261 | /* Arg 6: store register value in r9 */ | ||
| 2262 | EMIT_mov(X86_REG_R9, src_reg); | ||
| 2263 | break; | ||
| 2264 | } | ||
| 2265 | insn_off = insn->off; | ||
| 2266 | if (dst_reg == BPF_REG_PARAMS) { | ||
| 2267 | insn_off = outgoing_arg_base - outgoing_rsp - insn_off - 16; | ||
| 2268 | dst_reg = BPF_REG_FP; | ||
| 2269 | } | ||
| 2270 | emit_stx(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn_off); | ||
| 2153 | break; | 2271 | break; |
| 2154 | 2272 | ||
| 2155 | case BPF_ST | BPF_PROBE_MEM32 | BPF_B: | 2273 | case BPF_ST | BPF_PROBE_MEM32 | BPF_B: |
| @@ -2248,6 +2366,19 @@ populate_extable: · 92 unmodified lines | |||
| 2248 | case BPF_LDX | BPF_PROBE_MEMSX | BPF_H: | 2366 | case BPF_LDX | BPF_PROBE_MEMSX | BPF_H: |
| 2249 | case BPF_LDX | BPF_PROBE_MEMSX | BPF_W: | 2367 | case BPF_LDX | BPF_PROBE_MEMSX | BPF_W: |
| 2250 | insn_off = insn->off; | 2368 | insn_off = insn->off; |
| 2369 | if (src_reg == BPF_REG_PARAMS) { | ||
| 2370 | if (insn_off == 8) { | ||
| 2371 | /* Incoming arg 6: read from r9 */ | ||
| 2372 | EMIT_mov(dst_reg, X86_REG_R9); | ||
| 2373 | break; | ||
| 2374 | } | ||
| 2375 | src_reg = BPF_REG_FP; | ||
| 2376 | /* | ||
| 2377 | * Incoming args 7+: native_off == bpf_off | ||
| 2378 | * (r11+16 → [rbp+16], r11+24 → [rbp+24], ...) | ||
| 2379 | * No offset adjustment needed. | ||
| 2380 | */ | ||
| 2381 | } | ||
| 2251 | 2382 | ||
| 2252 | if (BPF_MODE(insn->code) == BPF_PROBE_MEM || | 2383 | if (BPF_MODE(insn->code) == BPF_PROBE_MEM || |
| 2253 | BPF_MODE(insn->code) == BPF_PROBE_MEMSX) { | 2384 | BPF_MODE(insn->code) == BPF_PROBE_MEMSX) { |
| @@ -2736,6 +2867,8 @@ emit_jmp: · 482 unmodified lines | |||
| 2736 | if (emit_spectre_bhb_barrier(&prog, ip, bpf_prog)) | 2867 | if (emit_spectre_bhb_barrier(&prog, ip, bpf_prog)) |
| 2737 | return -EINVAL; | 2868 | return -EINVAL; |
| 2738 | } | 2869 | } |
| 2870 | /* Deallocate outgoing args 7+ area. */ | ||
| 2871 | emit_add_rsp(&prog, outgoing_rsp); | ||
| 2739 | if (bpf_prog->aux->exception_boundary) { | 2872 | if (bpf_prog->aux->exception_boundary) { |
| 2740 | pop_callee_regs(&prog, all_callee_regs_used); | 2873 | pop_callee_regs(&prog, all_callee_regs_used); |
| 2741 | pop_r12(&prog); | 2874 | pop_r12(&prog); |
| @@ -2971,15 +3104,15 @@ static void restore_regs(const struct btf_func_model *m, u8 **prog, · 229 unmodified lines | |||
| 2971 | } | 3104 | } |
| 2972 | 3105 | ||
| 2973 | static int invoke_bpf_prog(const struct btf_func_model *m, u8 **pprog, | 3106 | static int invoke_bpf_prog(const struct btf_func_model *m, u8 **pprog, |
| 2974 | struct bpf_tramp_link *l, int stack_size, | 3107 | struct bpf_tramp_node *node, int stack_size, |
| 2975 | int run_ctx_off, bool save_ret, | 3108 | int run_ctx_off, bool save_ret, |
| 2976 | void *image, void *rw_image) | 3109 | void *image, void *rw_image) |
| 2977 | { | 3110 | { |
| 2978 | u8 *prog = *pprog; | 3111 | u8 *prog = *pprog; |
| 2979 | u8 *jmp_insn; | 3112 | u8 *jmp_insn; |
| 2980 | int ctx_cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie); | 3113 | int ctx_cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie); |
| 2981 | struct bpf_prog *p = l->link.prog; | 3114 | struct bpf_prog *p = node->link->prog; |
| 2982 | u64 cookie = l->cookie; | 3115 | u64 cookie = node->cookie; |
| 2983 | 3116 | ||
| 2984 | /* mov rdi, cookie */ | 3117 | /* mov rdi, cookie */ |
| 2985 | emit_mov_imm64(&prog, BPF_REG_1, (long) cookie >> 32, (u32) (long) cookie); | 3118 | emit_mov_imm64(&prog, BPF_REG_1, (long) cookie >> 32, (u32) (long) cookie); |
| @@ -3086,7 +3219,7 @@ static int emit_cond_near_jump(u8 **pprog, void *func, void *ip, u8 jmp_cond) · 100 unmodified lines | |||
| 3086 | } | 3219 | } |
| 3087 | 3220 | ||
| 3088 | static int invoke_bpf(const struct btf_func_model *m, u8 **pprog, | 3221 | static int invoke_bpf(const struct btf_func_model *m, u8 **pprog, |
| 3089 | struct bpf_tramp_links *tl, int stack_size, | 3222 | struct bpf_tramp_nodes *tl, int stack_size, |
| 3090 | int run_ctx_off, int func_meta_off, bool save_ret, | 3223 | int run_ctx_off, int func_meta_off, bool save_ret, |
| 3091 | void *image, void *rw_image, u64 func_meta, | 3224 | void *image, void *rw_image, u64 func_meta, |
| 3092 | int cookie_off) | 3225 | int cookie_off) |
| @@ -3094,13 +3227,13 @@ static int invoke_bpf(const struct btf_func_model *m, u8 **pprog, · 1 unmodified line | |||
| 3094 | int i, cur_cookie = (cookie_off - stack_size) / 8; | 3227 | int i, cur_cookie = (cookie_off - stack_size) / 8; |
| 3095 | u8 *prog = *pprog; | 3228 | u8 *prog = *pprog; |
| 3096 | 3229 | ||
| 3097 | for (i = 0; i < tl->nr_links; i++) { | 3230 | for (i = 0; i < tl->nr_nodes; i++) { |
| 3098 | if (tl->links[i]->link.prog->call_session_cookie) { | 3231 | if (tl->nodes[i]->link->prog->call_session_cookie) { |
| 3099 | emit_store_stack_imm64(&prog, BPF_REG_0, -func_meta_off, | 3232 | emit_store_stack_imm64(&prog, BPF_REG_0, -func_meta_off, |
| 3100 | func_meta | (cur_cookie << BPF_TRAMP_COOKIE_INDEX_SHIFT)); | 3233 | func_meta | (cur_cookie << BPF_TRAMP_COOKIE_INDEX_SHIFT)); |
| 3101 | cur_cookie--; | 3234 | cur_cookie--; |
| 3102 | } | 3235 | } |
| 3103 | if (invoke_bpf_prog(m, &prog, tl->links[i], stack_size, | 3236 | if (invoke_bpf_prog(m, &prog, tl->nodes[i], stack_size, |
| 3104 | run_ctx_off, save_ret, image, rw_image)) | 3237 | run_ctx_off, save_ret, image, rw_image)) |
| 3105 | return -EINVAL; | 3238 | return -EINVAL; |
| 3106 | } | 3239 | } |
| @@ -3109,7 +3242,7 @@ static int invoke_bpf(const struct btf_func_model *m, u8 **pprog, · 2 unmodified lines | |||
| 3109 | } | 3242 | } |
| 3110 | 3243 | ||
| 3111 | static int invoke_bpf_mod_ret(const struct btf_func_model *m, u8 **pprog, | 3244 | static int invoke_bpf_mod_ret(const struct btf_func_model *m, u8 **pprog, |
| 3112 | struct bpf_tramp_links *tl, int stack_size, | 3245 | struct bpf_tramp_nodes *tl, int stack_size, |
| 3113 | int run_ctx_off, u8 **branches, | 3246 | int run_ctx_off, u8 **branches, |
| 3114 | void *image, void *rw_image) | 3247 | void *image, void *rw_image) |
| 3115 | { | 3248 | { |
| @@ -3121,8 +3254,8 @@ static int invoke_bpf_mod_ret(const struct btf_func_model *m, u8 **pprog, · 5 unmodified lines | |||
| 3121 | */ | 3254 | */ |
| 3122 | emit_mov_imm32(&prog, false, BPF_REG_0, 0); | 3255 | emit_mov_imm32(&prog, false, BPF_REG_0, 0); |
| 3123 | emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8); | 3256 | emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8); |
| 3124 | for (i = 0; i < tl->nr_links; i++) { | 3257 | for (i = 0; i < tl->nr_nodes; i++) { |
| 3125 | if (invoke_bpf_prog(m, &prog, tl->links[i], stack_size, run_ctx_off, true, | 3258 | if (invoke_bpf_prog(m, &prog, tl->nodes[i], stack_size, run_ctx_off, true, |
| 3126 | image, rw_image)) | 3259 | image, rw_image)) |
| 3127 | return -EINVAL; | 3260 | return -EINVAL; |
| 3128 | 3261 | ||
| @@ -3213,14 +3346,14 @@ static int invoke_bpf_mod_ret(const struct btf_func_model *m, u8 **pprog, · 84 unmodified lines | |||
| 3213 | static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_image, | 3346 | static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_image, |
| 3214 | void *rw_image_end, void *image, | 3347 | void *rw_image_end, void *image, |
| 3215 | const struct btf_func_model *m, u32 flags, | 3348 | const struct btf_func_model *m, u32 flags, |
| 3216 | struct bpf_tramp_links *tlinks, | 3349 | struct bpf_tramp_nodes *tnodes, |
| 3217 | void *func_addr) | 3350 | void *func_addr) |
| 3218 | { | 3351 | { |
| 3219 | int i, ret, nr_regs = m->nr_args, stack_size = 0; | 3352 | int i, ret, nr_regs = m->nr_args, stack_size = 0; |
| 3220 | int regs_off, func_meta_off, ip_off, run_ctx_off, arg_stack_off, rbx_off; | 3353 | int regs_off, func_meta_off, ip_off, run_ctx_off, arg_stack_off, rbx_off; |
| 3221 | struct bpf_tramp_links *fentry = &tlinks[BPF_TRAMP_FENTRY]; | 3354 | struct bpf_tramp_nodes *fentry = &tnodes[BPF_TRAMP_FENTRY]; |
| 3222 | struct bpf_tramp_links *fexit = &tlinks[BPF_TRAMP_FEXIT]; | 3355 | struct bpf_tramp_nodes *fexit = &tnodes[BPF_TRAMP_FEXIT]; |
| 3223 | struct bpf_tramp_links *fmod_ret = &tlinks[BPF_TRAMP_MODIFY_RETURN]; | 3356 | struct bpf_tramp_nodes *fmod_ret = &tnodes[BPF_TRAMP_MODIFY_RETURN]; |
| 3224 | void *orig_call = func_addr; | 3357 | void *orig_call = func_addr; |
| 3225 | int cookie_off, cookie_cnt; | 3358 | int cookie_off, cookie_cnt; |
| 3226 | u8 **branches = NULL; | 3359 | u8 **branches = NULL; |
| @@ -3292,7 +3425,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 65 unmodified lines | |||
| 3292 | 3425 | ||
| 3293 | ip_off = stack_size; | 3426 | ip_off = stack_size; |
| 3294 | 3427 | ||
| 3295 | cookie_cnt = bpf_fsession_cookie_cnt(tlinks); | 3428 | cookie_cnt = bpf_fsession_cookie_cnt(tnodes); |
| 3296 | /* room for session cookies */ | 3429 | /* room for session cookies */ |
| 3297 | stack_size += cookie_cnt * 8; | 3430 | stack_size += cookie_cnt * 8; |
| 3298 | cookie_off = stack_size; | 3431 | cookie_off = stack_size; |
| @@ -3385,7 +3518,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 86 unmodified lines | |||
| 3385 | } | 3518 | } |
| 3386 | } | 3519 | } |
| 3387 | 3520 | ||
| 3388 | if (bpf_fsession_cnt(tlinks)) { | 3521 | if (bpf_fsession_cnt(tnodes)) { |
| 3389 | /* clear all the session cookies' value */ | 3522 | /* clear all the session cookies' value */ |
| 3390 | for (int i = 0; i < cookie_cnt; i++) | 3523 | for (int i = 0; i < cookie_cnt; i++) |
| 3391 | emit_store_stack_imm64(&prog, BPF_REG_0, -cookie_off + 8 * i, 0); | 3524 | emit_store_stack_imm64(&prog, BPF_REG_0, -cookie_off + 8 * i, 0); |
| @@ -3393,15 +3526,15 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 1 unmodified line | |||
| 3393 | emit_store_stack_imm64(&prog, BPF_REG_0, -8, 0); | 3526 | emit_store_stack_imm64(&prog, BPF_REG_0, -8, 0); |
| 3394 | } | 3527 | } |
| 3395 | 3528 | ||
| 3396 | if (fentry->nr_links) { | 3529 | if (fentry->nr_nodes) { |
| 3397 | if (invoke_bpf(m, &prog, fentry, regs_off, run_ctx_off, func_meta_off, | 3530 | if (invoke_bpf(m, &prog, fentry, regs_off, run_ctx_off, func_meta_off, |
| 3398 | flags & BPF_TRAMP_F_RET_FENTRY_RET, image, rw_image, | 3531 | flags & BPF_TRAMP_F_RET_FENTRY_RET, image, rw_image, |
| 3399 | func_meta, cookie_off)) | 3532 | func_meta, cookie_off)) |
| 3400 | return -EINVAL; | 3533 | return -EINVAL; |
| 3401 | } | 3534 | } |
| 3402 | 3535 | ||
| 3403 | if (fmod_ret->nr_links) { | 3536 | if (fmod_ret->nr_nodes) { |
| 3404 | branches = kcalloc(fmod_ret->nr_links, sizeof(u8 *), | 3537 | branches = kcalloc(fmod_ret->nr_nodes, sizeof(u8 *), |
| 3405 | GFP_KERNEL); | 3538 | GFP_KERNEL); |
| 3406 | if (!branches) | 3539 | if (!branches) |
| 3407 | return -ENOMEM; | 3540 | return -ENOMEM; |
| @@ -3440,7 +3573,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 32 unmodified lines | |||
| 3440 | emit_nops(&prog, X86_PATCH_SIZE); | 3573 | emit_nops(&prog, X86_PATCH_SIZE); |
| 3441 | } | 3574 | } |
| 3442 | 3575 | ||
| 3443 | if (fmod_ret->nr_links) { | 3576 | if (fmod_ret->nr_nodes) { |
| 3444 | /* From Intel 64 and IA-32 Architectures Optimization | 3577 | /* From Intel 64 and IA-32 Architectures Optimization |
| 3445 | * Reference Manual, 3.4.1.4 Code Alignment, Assembly/Compiler | 3578 | * Reference Manual, 3.4.1.4 Code Alignment, Assembly/Compiler |
| 3446 | * Coding Rule 11: All branch targets should be 16-byte | 3579 | * Coding Rule 11: All branch targets should be 16-byte |
| @@ -3450,7 +3583,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 3 unmodified lines | |||
| 3450 | /* Update the branches saved in invoke_bpf_mod_ret with the | 3583 | /* Update the branches saved in invoke_bpf_mod_ret with the |
| 3451 | * aligned address of do_fexit. | 3584 | * aligned address of do_fexit. |
| 3452 | */ | 3585 | */ |
| 3453 | for (i = 0; i < fmod_ret->nr_links; i++) { | 3586 | for (i = 0; i < fmod_ret->nr_nodes; i++) { |
| 3454 | emit_cond_near_jump(&branches[i], image + (prog - (u8 *)rw_image), | 3587 | emit_cond_near_jump(&branches[i], image + (prog - (u8 *)rw_image), |
| 3455 | image + (branches[i] - (u8 *)rw_image), X86_JNE); | 3588 | image + (branches[i] - (u8 *)rw_image), X86_JNE); |
| 3456 | } | 3589 | } |
| @@ -3458,10 +3591,10 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 1 unmodified line | |||
| 3458 | 3591 | ||
| 3459 | /* set the "is_return" flag for fsession */ | 3592 | /* set the "is_return" flag for fsession */ |
| 3460 | func_meta |= (1ULL << BPF_TRAMP_IS_RETURN_SHIFT); | 3593 | func_meta |= (1ULL << BPF_TRAMP_IS_RETURN_SHIFT); |
| 3461 | if (bpf_fsession_cnt(tlinks)) | 3594 | if (bpf_fsession_cnt(tnodes)) |
| 3462 | emit_store_stack_imm64(&prog, BPF_REG_0, -func_meta_off, func_meta); | 3595 | emit_store_stack_imm64(&prog, BPF_REG_0, -func_meta_off, func_meta); |
| 3463 | 3596 | ||
| 3464 | if (fexit->nr_links) { | 3597 | if (fexit->nr_nodes) { |
| 3465 | if (invoke_bpf(m, &prog, fexit, regs_off, run_ctx_off, func_meta_off, | 3598 | if (invoke_bpf(m, &prog, fexit, regs_off, run_ctx_off, func_meta_off, |
| 3466 | false, image, rw_image, func_meta, cookie_off)) { | 3599 | false, image, rw_image, func_meta, cookie_off)) { |
| 3467 | ret = -EINVAL; | 3600 | ret = -EINVAL; |
| @@ -3535,7 +3668,7 @@ int arch_protect_bpf_trampoline(void *image, unsigned int size) · 67 unmodified lines | |||
| 3535 | 3668 | ||
| 3536 | int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *image_end, | 3669 | int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *image_end, |
| 3537 | const struct btf_func_model *m, u32 flags, | 3670 | const struct btf_func_model *m, u32 flags, |
| 3538 | struct bpf_tramp_links *tlinks, | 3671 | struct bpf_tramp_nodes *tnodes, |
| 3539 | void *func_addr) | 3672 | void *func_addr) |
| 3540 | { | 3673 | { |
| 3541 | void *rw_image, *tmp; | 3674 | void *rw_image, *tmp; |
| @@ -3550,7 +3683,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i · 8 unmodified lines | |||
| 3550 | return -ENOMEM; | 3683 | return -ENOMEM; |
| 3551 | 3684 | ||
| 3552 | ret = __arch_prepare_bpf_trampoline(im, rw_image, rw_image + size, image, m, | 3685 | ret = __arch_prepare_bpf_trampoline(im, rw_image, rw_image + size, image, m, |
| 3553 | flags, tlinks, func_addr); | 3686 | flags, tnodes, func_addr); |
| 3554 | if (ret < 0) | 3687 | if (ret < 0) |
| 3555 | goto out; | 3688 | goto out; |
| 3556 | 3689 | ||
| @@ -3563,7 +3696,7 @@ out: · 6 unmodified lines | |||
| 3563 | } | 3696 | } |
| 3564 | 3697 | ||
| 3565 | int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, | 3698 | int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, |
| 3566 | struct bpf_tramp_links *tlinks, void *func_addr) | 3699 | struct bpf_tramp_nodes *tnodes, void *func_addr) |
| 3567 | { | 3700 | { |
| 3568 | struct bpf_tramp_image im; | 3701 | struct bpf_tramp_image im; |
| 3569 | void *image; | 3702 | void *image; |
| @@ -3581,7 +3714,7 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, · 11 unmodified lines | |||
| 3581 | return -ENOMEM; | 3714 | return -ENOMEM; |
| 3582 | 3715 | ||
| 3583 | ret = __arch_prepare_bpf_trampoline(&im, image, image + PAGE_SIZE, image, | 3716 | ret = __arch_prepare_bpf_trampoline(&im, image, image + PAGE_SIZE, image, |
| 3584 | m, flags, tlinks, func_addr); | 3717 | m, flags, tnodes, func_addr); |
| 3585 | bpf_jit_free_exec(image); | 3718 | bpf_jit_free_exec(image); |
| 3586 | return ret; | 3719 | return ret; |
| 3587 | } | 3720 | } |
| @@ -3793,7 +3926,8 @@ skip_init_addrs: · 205 unmodified lines | |||
| 3793 | for (pass = 0; pass < MAX_PASSES || image; pass++) { | 3926 | for (pass = 0; pass < MAX_PASSES || image; pass++) { |
| 3794 | if (!padding && pass >= PADDING_PASSES) | 3927 | if (!padding && pass >= PADDING_PASSES) |
| 3795 | padding = true; | 3928 | padding = true; |
| 3796 | proglen = do_jit(env, prog, addrs, image, rw_image, oldproglen, &ctx, padding); | 3929 | proglen = do_jit(env, prog, addrs, image, rw_image, oldproglen, |
| 3930 | &ctx, padding); | ||
| 3797 | if (proglen <= 0) { | 3931 | if (proglen <= 0) { |
| 3798 | out_image: | 3932 | out_image: |
| 3799 | image = NULL; | 3933 | image = NULL; |
| @@ -3910,6 +4044,11 @@ bool bpf_jit_supports_kfunc_call(void) · 110 unmodified lines | |||
| 3910 | return true; | 4044 | return true; |
| 3911 | } | 4045 | } |
| 3912 | 4046 | ||
| 4047 | bool bpf_jit_supports_stack_args(void) | ||
| 4048 | { | ||
| 4049 | return true; | ||
| 4050 | } | ||
| 4051 | |||
| 3913 | void *bpf_arch_text_copy(void *dst, void *src, size_t len) | 4052 | void *bpf_arch_text_copy(void *dst, void *src, size_t len) |
| 3914 | { | 4053 | { |
| 3915 | if (text_poke_copy(dst, src, len) == NULL) | 4054 | if (text_poke_copy(dst, src, len) == NULL) |
| @@ -168,3 +168,10 @@ static inline void device_pm_init(struct device *dev) · 167 unmodified lines | |||
| 168 | device_pm_sleep_init(dev); | 168 | device_pm_sleep_init(dev); |
| 169 | pm_runtime_init(dev); | 169 | pm_runtime_init(dev); |
| 170 | } | 170 | } |
| 171 | |||
| 172 | #ifdef CONFIG_BPF_SYSCALL | ||
| 173 | struct bpf_ws_lock { }; | ||
| 174 | struct bpf_ws_lock *bpf_wakeup_sources_read_lock(void); | ||
| 175 | void bpf_wakeup_sources_read_unlock(struct bpf_ws_lock *lock); | ||
| 176 | void *bpf_wakeup_sources_get_head(void); | ||
| 177 | #endif | ||
| @@ -1168,11 +1168,78 @@ static const struct file_operations wakeup_sources_stats_fops = { · 1167 unmodified lines | |||
| 1168 | .release = seq_release_private, | 1168 | .release = seq_release_private, |
| 1169 | }; | 1169 | }; |
| 1170 | 1170 | ||
| 1171 | static int __init wakeup_sources_debugfs_init(void) | 1171 | #ifdef CONFIG_BPF_SYSCALL |
| 1172 | #include <linux/btf.h> | ||
| 1173 | |||
| 1174 | __bpf_kfunc_start_defs(); | ||
| 1175 | |||
| 1176 | /** | ||
| 1177 | * bpf_wakeup_sources_read_lock - Acquire the SRCU lock for wakeup sources | ||
| 1178 | * | ||
| 1179 | * The underlying SRCU lock returns an integer index. However, the BPF verifier | ||
| 1180 | * requires a pointer (PTR_TO_BTF_ID) to strictly track the state of acquired | ||
| 1181 | * resources using KF_ACQUIRE and KF_RELEASE semantics. We use an opaque | ||
| 1182 | * structure pointer (struct bpf_ws_lock *) to satisfy the verifier while | ||
| 1183 | * safely encoding the integer index within the pointer address itself. | ||
| 1184 | * | ||
| 1185 | * Return: An opaque pointer encoding the SRCU lock index + 1 (to avoid NULL). | ||
| 1186 | */ | ||
| 1187 | __bpf_kfunc struct bpf_ws_lock *bpf_wakeup_sources_read_lock(void) | ||
| 1188 | { | ||
| 1189 | return (struct bpf_ws_lock *)(long)(wakeup_sources_read_lock() + 1); | ||
| 1190 | } | ||
| 1191 | |||
| 1192 | /** | ||
| 1193 | * bpf_wakeup_sources_read_unlock - Release the SRCU lock for wakeup sources | ||
| 1194 | * @lock: The opaque pointer returned by bpf_wakeup_sources_read_lock() | ||
| 1195 | * | ||
| 1196 | * The BPF verifier guarantees that @lock is a valid, unreleased pointer from | ||
| 1197 | * the acquire function. We decode the pointer back into the integer SRCU index | ||
| 1198 | * by subtracting 1 and release the lock. | ||
| 1199 | */ | ||
| 1200 | __bpf_kfunc void bpf_wakeup_sources_read_unlock(struct bpf_ws_lock *lock) | ||
| 1201 | { | ||
| 1202 | wakeup_sources_read_unlock((int)(long)lock - 1); | ||
| 1203 | } | ||
| 1204 | |||
| 1205 | /** | ||
| 1206 | * bpf_wakeup_sources_get_head - Get the head of the wakeup sources list | ||
| 1207 | * | ||
| 1208 | * Return: The head of the wakeup sources list. | ||
| 1209 | */ | ||
| 1210 | __bpf_kfunc void *bpf_wakeup_sources_get_head(void) | ||
| 1211 | { | ||
| 1212 | return &wakeup_sources; | ||
| 1213 | } | ||
| 1214 | |||
| 1215 | __bpf_kfunc_end_defs(); | ||
| 1216 | |||
| 1217 | BTF_KFUNCS_START(wakeup_source_kfunc_ids) | ||
| 1218 | BTF_ID_FLAGS(func, bpf_wakeup_sources_read_lock, KF_ACQUIRE) | ||
| 1219 | BTF_ID_FLAGS(func, bpf_wakeup_sources_read_unlock, KF_RELEASE) | ||
| 1220 | BTF_ID_FLAGS(func, bpf_wakeup_sources_get_head) | ||
| 1221 | BTF_KFUNCS_END(wakeup_source_kfunc_ids) | ||
| 1222 | |||
| 1223 | static const struct btf_kfunc_id_set wakeup_source_kfunc_set = { | ||
| 1224 | .set = &wakeup_source_kfunc_ids, | ||
| 1225 | }; | ||
| 1226 | |||
| 1227 | static void __init wakeup_sources_bpf_init(void) | ||
| 1228 | { | ||
| 1229 | if (register_btf_kfunc_id_set(BPF_PROG_TYPE_SYSCALL, &wakeup_source_kfunc_set)) | ||
| 1230 | pm_pr_dbg("Wakeup: failed to register BTF kfuncs\n"); | ||
| 1231 | } | ||
| 1232 | #else | ||
| 1233 | static inline void wakeup_sources_bpf_init(void) {} | ||
| 1234 | #endif /* CONFIG_BPF_SYSCALL */ | ||
| 1235 | |||
| 1236 | static int __init wakeup_sources_init(void) | ||
| 1172 | { | 1237 | { |
| 1173 | debugfs_create_file("wakeup_sources", 0444, NULL, NULL, | 1238 | debugfs_create_file("wakeup_sources", 0444, NULL, NULL, |
| 1174 | &wakeup_sources_stats_fops); | 1239 | &wakeup_sources_stats_fops); |
| 1240 | wakeup_sources_bpf_init(); | ||
| 1241 | |||
| 1175 | return 0; | 1242 | return 0; |
| 1176 | } | 1243 | } |
| 1177 | 1244 | ||
| 1178 | postcore_initcall(wakeup_sources_debugfs_init); | 1245 | postcore_initcall(wakeup_sources_init); |
| @@ -561,10 +561,10 @@ nfp_bpf_check_alu(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta, · 560 unmodified lines | |||
| 561 | const struct bpf_reg_state *dreg = | 561 | const struct bpf_reg_state *dreg = |
| 562 | cur_regs(env) + meta->insn.dst_reg; | 562 | cur_regs(env) + meta->insn.dst_reg; |
| 563 | 563 | ||
| 564 | meta->umin_src = min(meta->umin_src, sreg->umin_value); | 564 | meta->umin_src = min(meta->umin_src, reg_umin(sreg)); |
| 565 | meta->umax_src = max(meta->umax_src, sreg->umax_value); | 565 | meta->umax_src = max(meta->umax_src, reg_umax(sreg)); |
| 566 | meta->umin_dst = min(meta->umin_dst, dreg->umin_value); | 566 | meta->umin_dst = min(meta->umin_dst, reg_umin(dreg)); |
| 567 | meta->umax_dst = max(meta->umax_dst, dreg->umax_value); | 567 | meta->umax_dst = max(meta->umax_dst, reg_umax(dreg)); |
| 568 | 568 | ||
| 569 | /* NFP supports u16 and u32 multiplication. | 569 | /* NFP supports u16 and u32 multiplication. |
| 570 | * | 570 | * |
| @@ -200,7 +200,7 @@ int bpf_set_dentry_xattr_locked(struct dentry *dentry, const char *name__str, · 199 unmodified lines | |||
| 200 | const struct bpf_dynptr *value_p, int flags) | 200 | const struct bpf_dynptr *value_p, int flags) |
| 201 | { | 201 | { |
| 202 | 202 | ||
| 203 | struct bpf_dynptr_kern *value_ptr = (struct bpf_dynptr_kern *)value_p; | 203 | const struct bpf_dynptr_kern *value_ptr = (struct bpf_dynptr_kern *)value_p; |
| 204 | struct inode *inode = d_inode(dentry); | 204 | struct inode *inode = d_inode(dentry); |
| 205 | const void *value; | 205 | const void *value; |
| 206 | u32 value_len; | 206 | u32 value_len; |
| @@ -118,9 +118,9 @@ __bpf_kfunc_start_defs(); · 117 unmodified lines | |||
| 118 | * | 118 | * |
| 119 | * Return: 0 on success, a negative value on error. | 119 | * Return: 0 on success, a negative value on error. |
| 120 | */ | 120 | */ |
| 121 | __bpf_kfunc int bpf_get_fsverity_digest(struct file *file, struct bpf_dynptr *digest_p) | 121 | __bpf_kfunc int bpf_get_fsverity_digest(struct file *file, const struct bpf_dynptr *digest_p) |
| 122 | { | 122 | { |
| 123 | struct bpf_dynptr_kern *digest_ptr = (struct bpf_dynptr_kern *)digest_p; | 123 | const struct bpf_dynptr_kern *digest_ptr = (struct bpf_dynptr_kern *)digest_p; |
| 124 | const struct inode *inode = file_inode(file); | 124 | const struct inode *inode = file_inode(file); |
| 125 | u32 dynptr_sz = __bpf_dynptr_size(digest_ptr); | 125 | u32 dynptr_sz = __bpf_dynptr_size(digest_ptr); |
| 126 | struct fsverity_digest *arg; | 126 | struct fsverity_digest *arg; |
| @@ -243,12 +243,20 @@ static __always_inline void res_spin_unlock(rqspinlock_t *lock) · 242 unmodified lines | |||
| 243 | ({ \ | 243 | ({ \ |
| 244 | int __ret; \ | 244 | int __ret; \ |
| 245 | local_irq_save(flags); \ | 245 | local_irq_save(flags); \ |
| 246 | __ret = raw_res_spin_lock(lock); \ | 246 | preempt_disable(); \ |
| 247 | if (__ret) \ | 247 | __ret = res_spin_lock(lock); \ |
| 248 | if (__ret) { \ | ||
| 248 | local_irq_restore(flags); \ | 249 | local_irq_restore(flags); \ |
| 250 | preempt_enable(); \ | ||
| 251 | } \ | ||
| 249 | __ret; \ | 252 | __ret; \ |
| 250 | }) | 253 | }) |
| 251 | 254 | ||
| 252 | #define raw_res_spin_unlock_irqrestore(lock, flags) ({ raw_res_spin_unlock(lock); local_irq_restore(flags); }) | 255 | #define raw_res_spin_unlock_irqrestore(lock, flags) \ |
| 256 | ({ \ | ||
| 257 | res_spin_unlock(lock); \ | ||
| 258 | local_irq_restore(flags); \ | ||
| 259 | preempt_enable(); \ | ||
| 260 | }) | ||
| 253 | 261 | ||
| 254 | #endif /* __ASM_GENERIC_RQSPINLOCK_H */ | 262 | #endif /* __ASM_GENERIC_RQSPINLOCK_H */ |
| @@ -421,7 +421,7 @@ int cgroup_bpf_prog_detach(const union bpf_attr *attr, · 420 unmodified lines | |||
| 421 | enum bpf_prog_type ptype); | 421 | enum bpf_prog_type ptype); |
| 422 | int cgroup_bpf_link_attach(const union bpf_attr *attr, struct bpf_prog *prog); | 422 | int cgroup_bpf_link_attach(const union bpf_attr *attr, struct bpf_prog *prog); |
| 423 | int cgroup_bpf_prog_query(const union bpf_attr *attr, | 423 | int cgroup_bpf_prog_query(const union bpf_attr *attr, |
| 424 | union bpf_attr __user *uattr); | 424 | union bpf_attr __user *uattr, u32 uattr_size); |
| 425 | 425 | ||
| 426 | const struct bpf_func_proto * | 426 | const struct bpf_func_proto * |
| 427 | cgroup_common_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog); | 427 | cgroup_common_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog); |
| @@ -452,7 +452,8 @@ static inline int cgroup_bpf_link_attach(const union bpf_attr *attr, · 24 unmodified lines | |||
| 452 | } | 452 | } |
| 453 | 453 | ||
| 454 | static inline int cgroup_bpf_prog_query(const union bpf_attr *attr, | 454 | static inline int cgroup_bpf_prog_query(const union bpf_attr *attr, |
| 455 | union bpf_attr __user *uattr) | 455 | union bpf_attr __user *uattr, |
| 456 | u32 uattr_size) | ||
| 456 | { | 457 | { |
| 457 | return -EINVAL; | 458 | return -EINVAL; |
| 458 | } | 459 | } |
| @@ -6,6 +6,7 @@ · 5 unmodified lines | |||
| 6 | 6 | ||
| 7 | #include <uapi/linux/bpf.h> | 7 | #include <uapi/linux/bpf.h> |
| 8 | #include <uapi/linux/filter.h> | 8 | #include <uapi/linux/filter.h> |
| 9 | #include <linux/bpf_defs.h> | ||
| 9 | 10 | ||
| 10 | #include <crypto/sha2.h> | 11 | #include <crypto/sha2.h> |
| 11 | #include <linux/workqueue.h> | 12 | #include <linux/workqueue.h> |
| @@ -32,6 +33,8 @@ · 20 unmodified lines | |||
| 32 | #include <linux/memcontrol.h> | 33 | #include <linux/memcontrol.h> |
| 33 | #include <linux/cfi.h> | 34 | #include <linux/cfi.h> |
| 34 | #include <linux/xattr.h> | 35 | #include <linux/xattr.h> |
| 36 | #include <linux/key.h> | ||
| 37 | #include <linux/ftrace.h> | ||
| 35 | #include <asm/rqspinlock.h> | 38 | #include <asm/rqspinlock.h> |
| 36 | 39 | ||
| 37 | struct bpf_verifier_env; | 40 | struct bpf_verifier_env; |
| @@ -111,7 +114,7 @@ struct bpf_map_ops { · 73 unmodified lines | |||
| 111 | long (*map_pop_elem)(struct bpf_map *map, void *value); | 114 | long (*map_pop_elem)(struct bpf_map *map, void *value); |
| 112 | long (*map_peek_elem)(struct bpf_map *map, void *value); | 115 | long (*map_peek_elem)(struct bpf_map *map, void *value); |
| 113 | void *(*map_lookup_percpu_elem)(struct bpf_map *map, void *key, u32 cpu); | 116 | void *(*map_lookup_percpu_elem)(struct bpf_map *map, void *key, u32 cpu); |
| 114 | int (*map_get_hash)(struct bpf_map *map, u32 hash_buf_size, void *hash_buf); | 117 | int (*map_get_hash)(struct bpf_map *map); |
| 115 | 118 | ||
| 116 | /* funcs called by prog_array and perf_event_array map */ | 119 | /* funcs called by prog_array and perf_event_array map */ |
| 117 | void *(*map_fd_get_ptr)(struct bpf_map *map, struct file *map_file, | 120 | void *(*map_fd_get_ptr)(struct bpf_map *map, struct file *map_file, |
| @@ -296,6 +299,7 @@ struct bpf_map_owner { · 178 unmodified lines | |||
| 296 | 299 | ||
| 297 | struct bpf_map { | 300 | struct bpf_map { |
| 298 | u8 sha[SHA256_DIGEST_SIZE]; | 301 | u8 sha[SHA256_DIGEST_SIZE]; |
| 302 | u32 excl; | ||
| 299 | const struct bpf_map_ops *ops; | 303 | const struct bpf_map_ops *ops; |
| 300 | struct bpf_map *inner_map_meta; | 304 | struct bpf_map *inner_map_meta; |
| 301 | #ifdef CONFIG_SECURITY | 305 | #ifdef CONFIG_SECURITY |
| @@ -489,6 +493,35 @@ static inline bool btf_record_has_field(const struct btf_record *rec, enum btf_f · 187 unmodified lines | |||
| 489 | return rec->field_mask & type; | 493 | return rec->field_mask & type; |
| 490 | } | 494 | } |
| 491 | 495 | ||
| 496 | static inline bool btf_field_is_nmi_safe(enum btf_field_type type) | ||
| 497 | { | ||
| 498 | switch (type) { | ||
| 499 | case BPF_SPIN_LOCK: | ||
| 500 | case BPF_RES_SPIN_LOCK: | ||
| 501 | case BPF_TIMER: | ||
| 502 | case BPF_WORKQUEUE: | ||
| 503 | case BPF_TASK_WORK: | ||
| 504 | case BPF_KPTR_UNREF: | ||
| 505 | case BPF_REFCOUNT: | ||
| 506 | return true; | ||
| 507 | default: | ||
| 508 | return false; | ||
| 509 | } | ||
| 510 | } | ||
| 511 | |||
| 512 | static inline bool btf_record_has_nmi_unsafe_fields(const struct btf_record *rec) | ||
| 513 | { | ||
| 514 | int i; | ||
| 515 | |||
| 516 | if (IS_ERR_OR_NULL(rec)) | ||
| 517 | return false; | ||
| 518 | for (i = 0; i < rec->cnt; i++) { | ||
| 519 | if (!btf_field_is_nmi_safe(rec->fields[i].type)) | ||
| 520 | return true; | ||
| 521 | } | ||
| 522 | return false; | ||
| 523 | } | ||
| 524 | |||
| 492 | static inline void bpf_obj_init(const struct btf_record *rec, void *obj) | 525 | static inline void bpf_obj_init(const struct btf_record *rec, void *obj) |
| 493 | { | 526 | { |
| 494 | int i; | 527 | int i; |
| @@ -618,6 +651,8 @@ void bpf_rb_root_free(const struct btf_field *field, void *rb_root, · 123 unmodified lines | |||
| 618 | struct bpf_spin_lock *spin_lock); | 651 | struct bpf_spin_lock *spin_lock); |
| 619 | u64 bpf_arena_get_kern_vm_start(struct bpf_arena *arena); | 652 | u64 bpf_arena_get_kern_vm_start(struct bpf_arena *arena); |
| 620 | u64 bpf_arena_get_user_vm_start(struct bpf_arena *arena); | 653 | u64 bpf_arena_get_user_vm_start(struct bpf_arena *arena); |
| 654 | u64 bpf_arena_map_kern_vm_start(struct bpf_map *map); | ||
| 655 | struct bpf_map *bpf_prog_arena(struct bpf_prog *prog); | ||
| 621 | int bpf_obj_name_cpy(char *dst, const char *src, unsigned int size); | 656 | int bpf_obj_name_cpy(char *dst, const char *src, unsigned int size); |
| 622 | 657 | ||
| 623 | struct bpf_offload_dev; | 658 | struct bpf_offload_dev; |
| @@ -679,6 +714,8 @@ int bpf_dynptr_from_file_sleepable(struct file *file, u32 flags, · 55 unmodified lines | |||
| 679 | void *bpf_arena_alloc_pages_non_sleepable(void *p__map, void *addr__ign, u32 page_cnt, int node_id, | 714 | void *bpf_arena_alloc_pages_non_sleepable(void *p__map, void *addr__ign, u32 page_cnt, int node_id, |
| 680 | u64 flags); | 715 | u64 flags); |
| 681 | void bpf_arena_free_pages_non_sleepable(void *p__map, void *ptr__ign, u32 page_cnt); | 716 | void bpf_arena_free_pages_non_sleepable(void *p__map, void *ptr__ign, u32 page_cnt); |
| 717 | void *bpf_arena_alloc_pages_sleepable(void *p__map, void *addr__ign, u32 page_cnt, int node_id, | ||
| 718 | u64 flags); | ||
| 682 | #else | 719 | #else |
| 683 | static inline void *bpf_arena_alloc_pages_non_sleepable(void *p__map, void *addr__ign, u32 page_cnt, | 720 | static inline void *bpf_arena_alloc_pages_non_sleepable(void *p__map, void *addr__ign, u32 page_cnt, |
| 684 | int node_id, u64 flags) | 721 | int node_id, u64 flags) |
| @@ -689,6 +726,12 @@ static inline void *bpf_arena_alloc_pages_non_sleepable(void *p__map, void *addr · 4 unmodified lines | |||
| 689 | static inline void bpf_arena_free_pages_non_sleepable(void *p__map, void *ptr__ign, u32 page_cnt) | 726 | static inline void bpf_arena_free_pages_non_sleepable(void *p__map, void *ptr__ign, u32 page_cnt) |
| 690 | { | 727 | { |
| 691 | } | 728 | } |
| 729 | |||
| 730 | static inline void *bpf_arena_alloc_pages_sleepable(void *p__map, void *addr__ign, u32 page_cnt, | ||
| 731 | int node_id, u64 flags) | ||
| 732 | { | ||
| 733 | return NULL; | ||
| 734 | } | ||
| 692 | #endif | 735 | #endif |
| 693 | 736 | ||
| 694 | extern const struct bpf_map_ops bpf_map_offload_ops; | 737 | extern const struct bpf_map_ops bpf_map_offload_ops; |
| @@ -1052,7 +1095,7 @@ struct bpf_insn_access_aux { · 357 unmodified lines | |||
| 1052 | struct { | 1095 | struct { |
| 1053 | struct btf *btf; | 1096 | struct btf *btf; |
| 1054 | u32 btf_id; | 1097 | u32 btf_id; |
| 1055 | u32 ref_obj_id; | 1098 | u32 ref_id; |
| 1056 | }; | 1099 | }; |
| 1057 | }; | 1100 | }; |
| 1058 | struct bpf_verifier_log *log; /* for verbose logs */ | 1101 | struct bpf_verifier_log *log; /* for verbose logs */ |
| @@ -1152,6 +1195,11 @@ struct bpf_prog_offload { · 93 unmodified lines | |||
| 1152 | 1195 | ||
| 1153 | /* The longest tracepoint has 12 args. | 1196 | /* The longest tracepoint has 12 args. |
| 1154 | * See include/trace/bpf_probe.h | 1197 | * See include/trace/bpf_probe.h |
| 1198 | * | ||
| 1199 | * Also reuse this macro for maximum number of arguments a BPF function | ||
| 1200 | * or a kfunc can have. Args 1-5 are passed in registers, args 6-12 via | ||
| 1201 | * stack arg slots. The JIT may map some stack arg slots to registers based | ||
| 1202 | * on the native calling convention (e.g., arg 6 to R9 on x86-64). | ||
| 1155 | */ | 1203 | */ |
| 1156 | #define MAX_BPF_FUNC_ARGS 12 | 1204 | #define MAX_BPF_FUNC_ARGS 12 |
| 1157 | 1205 | ||
| @@ -1234,9 +1282,9 @@ enum { · 76 unmodified lines | |||
| 1234 | #define BPF_TRAMP_COOKIE_INDEX_SHIFT 8 | 1282 | #define BPF_TRAMP_COOKIE_INDEX_SHIFT 8 |
| 1235 | #define BPF_TRAMP_IS_RETURN_SHIFT 63 | 1283 | #define BPF_TRAMP_IS_RETURN_SHIFT 63 |
| 1236 | 1284 | ||
| 1237 | struct bpf_tramp_links { | 1285 | struct bpf_tramp_nodes { |
| 1238 | struct bpf_tramp_link *links[BPF_MAX_TRAMP_LINKS]; | 1286 | struct bpf_tramp_node *nodes[BPF_MAX_TRAMP_LINKS]; |
| 1239 | int nr_links; | 1287 | int nr_nodes; |
| 1240 | }; | 1288 | }; |
| 1241 | 1289 | ||
| 1242 | struct bpf_tramp_run_ctx; | 1290 | struct bpf_tramp_run_ctx; |
| @@ -1264,13 +1312,13 @@ struct bpf_tramp_run_ctx; · 21 unmodified lines | |||
| 1264 | struct bpf_tramp_image; | 1312 | struct bpf_tramp_image; |
| 1265 | int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *image_end, | 1313 | int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *image_end, |
| 1266 | const struct btf_func_model *m, u32 flags, | 1314 | const struct btf_func_model *m, u32 flags, |
| 1267 | struct bpf_tramp_links *tlinks, | 1315 | struct bpf_tramp_nodes *tnodes, |
| 1268 | void *func_addr); | 1316 | void *func_addr); |
| 1269 | void *arch_alloc_bpf_trampoline(unsigned int size); | 1317 | void *arch_alloc_bpf_trampoline(unsigned int size); |
| 1270 | void arch_free_bpf_trampoline(void *image, unsigned int size); | 1318 | void arch_free_bpf_trampoline(void *image, unsigned int size); |
| 1271 | int __must_check arch_protect_bpf_trampoline(void *image, unsigned int size); | 1319 | int __must_check arch_protect_bpf_trampoline(void *image, unsigned int size); |
| 1272 | int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, | 1320 | int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, |
| 1273 | struct bpf_tramp_links *tlinks, void *func_addr); | 1321 | struct bpf_tramp_nodes *tnodes, void *func_addr); |
| 1274 | 1322 | ||
| 1275 | u64 notrace __bpf_prog_enter_sleepable_recur(struct bpf_prog *prog, | 1323 | u64 notrace __bpf_prog_enter_sleepable_recur(struct bpf_prog *prog, |
| 1276 | struct bpf_tramp_run_ctx *run_ctx); | 1324 | struct bpf_tramp_run_ctx *run_ctx); |
| @@ -1336,8 +1384,6 @@ struct bpf_trampoline { · 59 unmodified lines | |||
| 1336 | /* hlist for trampoline_ip_table */ | 1384 | /* hlist for trampoline_ip_table */ |
| 1337 | struct hlist_node hlist_ip; | 1385 | struct hlist_node hlist_ip; |
| 1338 | struct ftrace_ops *fops; | 1386 | struct ftrace_ops *fops; |
| 1339 | /* serializes access to fields of this trampoline */ | ||
| 1340 | struct mutex mutex; | ||
| 1341 | refcount_t refcnt; | 1387 | refcount_t refcnt; |
| 1342 | u32 flags; | 1388 | u32 flags; |
| 1343 | u64 key; | 1389 | u64 key; |
| @@ -1358,6 +1404,11 @@ struct bpf_trampoline { · 14 unmodified lines | |||
| 1358 | int progs_cnt[BPF_TRAMP_MAX]; | 1404 | int progs_cnt[BPF_TRAMP_MAX]; |
| 1359 | /* Executable image of trampoline */ | 1405 | /* Executable image of trampoline */ |
| 1360 | struct bpf_tramp_image *cur_image; | 1406 | struct bpf_tramp_image *cur_image; |
| 1407 | /* Used as temporary old image storage for multi_attach */ | ||
| 1408 | struct { | ||
| 1409 | struct bpf_tramp_image *old_image; | ||
| 1410 | u32 old_flags; | ||
| 1411 | } multi_attach; | ||
| 1361 | }; | 1412 | }; |
| 1362 | 1413 | ||
| 1363 | struct bpf_attach_target_info { | 1414 | struct bpf_attach_target_info { |
| @@ -1455,11 +1506,13 @@ static inline int bpf_dynptr_check_off_len(const struct bpf_dynptr_kern *ptr, u6 · 91 unmodified lines | |||
| 1455 | return 0; | 1506 | return 0; |
| 1456 | } | 1507 | } |
| 1457 | 1508 | ||
| 1509 | struct bpf_tracing_multi_link; | ||
| 1510 | |||
| 1458 | #ifdef CONFIG_BPF_JIT | 1511 | #ifdef CONFIG_BPF_JIT |
| 1459 | int bpf_trampoline_link_prog(struct bpf_tramp_link *link, | 1512 | int bpf_trampoline_link_prog(struct bpf_tramp_node *node, |
| 1460 | struct bpf_trampoline *tr, | 1513 | struct bpf_trampoline *tr, |
| 1461 | struct bpf_prog *tgt_prog); | 1514 | struct bpf_prog *tgt_prog); |
| 1462 | int bpf_trampoline_unlink_prog(struct bpf_tramp_link *link, | 1515 | int bpf_trampoline_unlink_prog(struct bpf_tramp_node *node, |
| 1463 | struct bpf_trampoline *tr, | 1516 | struct bpf_trampoline *tr, |
| 1464 | struct bpf_prog *tgt_prog); | 1517 | struct bpf_prog *tgt_prog); |
| 1465 | struct bpf_trampoline *bpf_trampoline_get(u64 key, | 1518 | struct bpf_trampoline *bpf_trampoline_get(u64 key, |
| @@ -1467,6 +1520,11 @@ struct bpf_trampoline *bpf_trampoline_get(u64 key, · 1 unmodified line | |||
| 1467 | void bpf_trampoline_put(struct bpf_trampoline *tr); | 1520 | void bpf_trampoline_put(struct bpf_trampoline *tr); |
| 1468 | int arch_prepare_bpf_dispatcher(void *image, void *buf, s64 *funcs, int num_funcs); | 1521 | int arch_prepare_bpf_dispatcher(void *image, void *buf, s64 *funcs, int num_funcs); |
| 1469 | 1522 | ||
| 1523 | int bpf_trampoline_multi_attach(struct bpf_prog *prog, u32 *ids, | ||
| 1524 | struct bpf_tracing_multi_link *link); | ||
| 1525 | int bpf_trampoline_multi_detach(struct bpf_prog *prog, | ||
| 1526 | struct bpf_tracing_multi_link *link); | ||
| 1527 | |||
| 1470 | /* | 1528 | /* |
| 1471 | * When the architecture supports STATIC_CALL replace the bpf_dispatcher_fn | 1529 | * When the architecture supports STATIC_CALL replace the bpf_dispatcher_fn |
| 1472 | * indirection with a direct call to the bpf program. If the architecture does | 1530 | * indirection with a direct call to the bpf program. If the architecture does |
| @@ -1544,14 +1602,15 @@ void bpf_jit_uncharge_modmem(u32 size); · 71 unmodified lines | |||
| 1544 | bool bpf_prog_has_trampoline(const struct bpf_prog *prog); | 1602 | bool bpf_prog_has_trampoline(const struct bpf_prog *prog); |
| 1545 | bool bpf_insn_is_indirect_target(const struct bpf_verifier_env *env, const struct bpf_prog *prog, | 1603 | bool bpf_insn_is_indirect_target(const struct bpf_verifier_env *env, const struct bpf_prog *prog, |
| 1546 | int insn_idx); | 1604 | int insn_idx); |
| 1605 | u16 bpf_out_stack_arg_cnt(const struct bpf_verifier_env *env, const struct bpf_prog *prog); | ||
| 1547 | #else | 1606 | #else |
| 1548 | static inline int bpf_trampoline_link_prog(struct bpf_tramp_link *link, | 1607 | static inline int bpf_trampoline_link_prog(struct bpf_tramp_node *node, |
| 1549 | struct bpf_trampoline *tr, | 1608 | struct bpf_trampoline *tr, |
| 1550 | struct bpf_prog *tgt_prog) | 1609 | struct bpf_prog *tgt_prog) |
| 1551 | { | 1610 | { |
| 1552 | return -ENOTSUPP; | 1611 | return -ENOTSUPP; |
| 1553 | } | 1612 | } |
| 1554 | static inline int bpf_trampoline_unlink_prog(struct bpf_tramp_link *link, | 1613 | static inline int bpf_trampoline_unlink_prog(struct bpf_tramp_node *node, |
| 1555 | struct bpf_trampoline *tr, | 1614 | struct bpf_trampoline *tr, |
| 1556 | struct bpf_prog *tgt_prog) | 1615 | struct bpf_prog *tgt_prog) |
| 1557 | { | 1616 | { |
| @@ -1578,6 +1637,16 @@ static inline bool bpf_prog_has_trampoline(const struct bpf_prog *prog) · 20 unmodified lines | |||
| 1578 | { | 1637 | { |
| 1579 | return false; | 1638 | return false; |
| 1580 | } | 1639 | } |
| 1640 | static inline int bpf_trampoline_multi_attach(struct bpf_prog *prog, u32 *ids, | ||
| 1641 | struct bpf_tracing_multi_link *link) | ||
| 1642 | { | ||
| 1643 | return -ENOTSUPP; | ||
| 1644 | } | ||
| 1645 | static inline int bpf_trampoline_multi_detach(struct bpf_prog *prog, | ||
| 1646 | struct bpf_tracing_multi_link *link) | ||
| 1647 | { | ||
| 1648 | return -ENOTSUPP; | ||
| 1649 | } | ||
| 1581 | #endif | 1650 | #endif |
| 1582 | 1651 | ||
| 1583 | struct bpf_func_info_aux { | 1652 | struct bpf_func_info_aux { |
| @@ -1615,7 +1684,7 @@ struct bpf_ctx_arg_aux { · 31 unmodified lines | |||
| 1615 | enum bpf_reg_type reg_type; | 1684 | enum bpf_reg_type reg_type; |
| 1616 | struct btf *btf; | 1685 | struct btf *btf; |
| 1617 | u32 btf_id; | 1686 | u32 btf_id; |
| 1618 | u32 ref_obj_id; | 1687 | u32 ref_id; |
| 1619 | bool refcounted; | 1688 | bool refcounted; |
| 1620 | }; | 1689 | }; |
| 1621 | 1690 | ||
| @@ -1657,6 +1726,19 @@ struct bpf_stream_stage { · 35 unmodified lines | |||
| 1657 | int len; | 1726 | int len; |
| 1658 | }; | 1727 | }; |
| 1659 | 1728 | ||
| 1729 | enum bpf_sig_verdict { | ||
| 1730 | BPF_SIG_UNSIGNED = 0, | ||
| 1731 | BPF_SIG_VERIFIED, | ||
| 1732 | }; | ||
| 1733 | |||
| 1734 | enum bpf_sig_keyring { | ||
| 1735 | BPF_SIG_KEYRING_NONE = 0, | ||
| 1736 | BPF_SIG_KEYRING_BUILTIN, | ||
| 1737 | BPF_SIG_KEYRING_SECONDARY, | ||
| 1738 | BPF_SIG_KEYRING_PLATFORM, | ||
| 1739 | BPF_SIG_KEYRING_USER, | ||
| 1740 | }; | ||
| 1741 | |||
| 1660 | struct bpf_prog_aux { | 1742 | struct bpf_prog_aux { |
| 1661 | atomic64_t refcnt; | 1743 | atomic64_t refcnt; |
| 1662 | u32 used_map_cnt; | 1744 | u32 used_map_cnt; |
| @@ -1699,6 +1781,11 @@ struct bpf_prog_aux { · 36 unmodified lines | |||
| 1699 | bool changes_pkt_data; | 1781 | bool changes_pkt_data; |
| 1700 | bool might_sleep; | 1782 | bool might_sleep; |
| 1701 | bool kprobe_write_ctx; | 1783 | bool kprobe_write_ctx; |
| 1784 | struct { | ||
| 1785 | s32 keyring_serial; | ||
| 1786 | u8 keyring_type; | ||
| 1787 | u8 verdict; | ||
| 1788 | } sig; | ||
| 1702 | u64 prog_array_member_cnt; /* counts how many times as member of prog_array */ | 1789 | u64 prog_array_member_cnt; /* counts how many times as member of prog_array */ |
| 1703 | struct mutex ext_mutex; /* mutex for is_extended and prog_array_member_cnt */ | 1790 | struct mutex ext_mutex; /* mutex for is_extended and prog_array_member_cnt */ |
| 1704 | struct bpf_arena *arena; | 1791 | struct bpf_arena *arena; |
| @@ -1731,6 +1818,7 @@ struct bpf_prog_aux { · 26 unmodified lines | |||
| 1731 | struct bpf_map *cgroup_storage[MAX_BPF_CGROUP_STORAGE_TYPE]; | 1818 | struct bpf_map *cgroup_storage[MAX_BPF_CGROUP_STORAGE_TYPE]; |
| 1732 | char name[BPF_OBJ_NAME_LEN]; | 1819 | char name[BPF_OBJ_NAME_LEN]; |
| 1733 | u64 (*bpf_exception_cb)(u64 cookie, u64 sp, u64 bp, u64, u64); | 1820 | u64 (*bpf_exception_cb)(u64 cookie, u64 sp, u64 bp, u64, u64); |
| 1821 | u16 stack_arg_sp_adjust; | ||
| 1734 | #ifdef CONFIG_SECURITY | 1822 | #ifdef CONFIG_SECURITY |
| 1735 | void *security; | 1823 | void *security; |
| 1736 | #endif | 1824 | #endif |
| @@ -1874,12 +1962,17 @@ struct bpf_link_ops { · 137 unmodified lines | |||
| 1874 | __poll_t (*poll)(struct file *file, struct poll_table_struct *pts); | 1962 | __poll_t (*poll)(struct file *file, struct poll_table_struct *pts); |
| 1875 | }; | 1963 | }; |
| 1876 | 1964 | ||
| 1877 | struct bpf_tramp_link { | 1965 | struct bpf_tramp_node { |
| 1878 | struct bpf_link link; | 1966 | struct bpf_link *link; |
| 1879 | struct hlist_node tramp_hlist; | 1967 | struct hlist_node tramp_hlist; |
| 1880 | u64 cookie; | 1968 | u64 cookie; |
| 1881 | }; | 1969 | }; |
| 1882 | 1970 | ||
| 1971 | struct bpf_tramp_link { | ||
| 1972 | struct bpf_link link; | ||
| 1973 | struct bpf_tramp_node node; | ||
| 1974 | }; | ||
| 1975 | |||
| 1883 | struct bpf_shim_tramp_link { | 1976 | struct bpf_shim_tramp_link { |
| 1884 | struct bpf_tramp_link link; | 1977 | struct bpf_tramp_link link; |
| 1885 | struct bpf_trampoline *trampoline; | 1978 | struct bpf_trampoline *trampoline; |
| @@ -1887,13 +1980,31 @@ struct bpf_shim_tramp_link { · 1 unmodified line | |||
| 1887 | 1980 | ||
| 1888 | struct bpf_tracing_link { | 1981 | struct bpf_tracing_link { |
| 1889 | struct bpf_tramp_link link; | 1982 | struct bpf_tramp_link link; |
| 1983 | struct bpf_tramp_node fexit; | ||
| 1890 | struct bpf_trampoline *trampoline; | 1984 | struct bpf_trampoline *trampoline; |
| 1891 | struct bpf_prog *tgt_prog; | 1985 | struct bpf_prog *tgt_prog; |
| 1892 | }; | 1986 | }; |
| 1893 | 1987 | ||
| 1894 | struct bpf_fsession_link { | 1988 | struct bpf_tracing_multi_node { |
| 1895 | struct bpf_tracing_link link; | 1989 | struct bpf_tramp_node node; |
| 1896 | struct bpf_tramp_link fexit; | 1990 | struct bpf_trampoline *trampoline; |
| 1991 | struct ftrace_func_entry entry; | ||
| 1992 | }; | ||
| 1993 | |||
| 1994 | struct bpf_tracing_multi_data { | ||
| 1995 | struct ftrace_hash *unreg; | ||
| 1996 | struct ftrace_hash *modify; | ||
| 1997 | struct ftrace_hash *reg; | ||
| 1998 | struct ftrace_func_entry *entry; | ||
| 1999 | }; | ||
| 2000 | |||
| 2001 | struct bpf_tracing_multi_link { | ||
| 2002 | struct bpf_link link; | ||
| 2003 | struct bpf_tracing_multi_data data; | ||
| 2004 | u64 *cookies; | ||
| 2005 | struct bpf_tramp_node *fexits; | ||
| 2006 | int nodes_cnt; | ||
| 2007 | struct bpf_tracing_multi_node nodes[] __counted_by(nodes_cnt); | ||
| 1897 | }; | 2008 | }; |
| 1898 | 2009 | ||
| 1899 | struct bpf_raw_tp_link { | 2010 | struct bpf_raw_tp_link { |
| @@ -2079,6 +2190,12 @@ static inline void bpf_prog_put_recursion_context(struct bpf_prog *prog) · 179 unmodified lines | |||
| 2079 | #endif | 2190 | #endif |
| 2080 | } | 2191 | } |
| 2081 | 2192 | ||
| 2193 | static inline bool is_tracing_multi(enum bpf_attach_type type) | ||
| 2194 | { | ||
| 2195 | return type == BPF_TRACE_FENTRY_MULTI || type == BPF_TRACE_FEXIT_MULTI || | ||
| 2196 | type == BPF_TRACE_FSESSION_MULTI; | ||
| 2197 | } | ||
| 2198 | |||
| 2082 | #if defined(CONFIG_BPF_JIT) && defined(CONFIG_BPF_SYSCALL) | 2199 | #if defined(CONFIG_BPF_JIT) && defined(CONFIG_BPF_SYSCALL) |
| 2083 | /* This macro helps developer to register a struct_ops type and generate | 2200 | /* This macro helps developer to register a struct_ops type and generate |
| 2084 | * type information correctly. Developers should use this macro to register | 2201 | * type information correctly. Developers should use this macro to register |
| @@ -2099,8 +2216,8 @@ void bpf_struct_ops_put(const void *kdata); · 14 unmodified lines | |||
| 2099 | int bpf_struct_ops_supported(const struct bpf_struct_ops *st_ops, u32 moff); | 2216 | int bpf_struct_ops_supported(const struct bpf_struct_ops *st_ops, u32 moff); |
| 2100 | int bpf_struct_ops_map_sys_lookup_elem(struct bpf_map *map, void *key, | 2217 | int bpf_struct_ops_map_sys_lookup_elem(struct bpf_map *map, void *key, |
| 2101 | void *value); | 2218 | void *value); |
| 2102 | int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_links *tlinks, | 2219 | int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_nodes *tnodes, |
| 2103 | struct bpf_tramp_link *link, | 2220 | struct bpf_tramp_node *node, |
| 2104 | const struct btf_func_model *model, | 2221 | const struct btf_func_model *model, |
| 2105 | void *stub_func, | 2222 | void *stub_func, |
| 2106 | void **image, u32 *image_off, | 2223 | void **image, u32 *image_off, |
| @@ -2125,6 +2242,9 @@ int bpf_prog_assoc_struct_ops(struct bpf_prog *prog, struct bpf_map *map); · 18 unmodified lines | |||
| 2125 | void bpf_prog_disassoc_struct_ops(struct bpf_prog *prog); | 2242 | void bpf_prog_disassoc_struct_ops(struct bpf_prog *prog); |
| 2126 | void *bpf_prog_get_assoc_struct_ops(const struct bpf_prog_aux *aux); | 2243 | void *bpf_prog_get_assoc_struct_ops(const struct bpf_prog_aux *aux); |
| 2127 | u32 bpf_struct_ops_id(const void *kdata); | 2244 | u32 bpf_struct_ops_id(const void *kdata); |
| 2245 | int bpf_struct_ops_for_each_prog(const void *kdata, | ||
| 2246 | int (*cb)(struct bpf_prog *prog, void *data), | ||
| 2247 | void *data); | ||
| 2128 | 2248 | ||
| 2129 | #ifdef CONFIG_NET | 2249 | #ifdef CONFIG_NET |
| 2130 | /* Define it here to avoid the use of forward declaration */ | 2250 | /* Define it here to avoid the use of forward declaration */ |
| @@ -2192,31 +2312,33 @@ static inline void bpf_struct_ops_desc_release(struct bpf_struct_ops_desc *st_op · 61 unmodified lines | |||
| 2192 | 2312 | ||
| 2193 | #endif | 2313 | #endif |
| 2194 | 2314 | ||
| 2195 | static inline int bpf_fsession_cnt(struct bpf_tramp_links *links) | 2315 | static inline int bpf_fsession_cnt(struct bpf_tramp_nodes *nodes) |
| 2196 | { | 2316 | { |
| 2197 | struct bpf_tramp_links fentries = links[BPF_TRAMP_FENTRY]; | 2317 | struct bpf_tramp_nodes fentries = nodes[BPF_TRAMP_FENTRY]; |
| 2198 | int cnt = 0; | 2318 | int cnt = 0; |
| 2199 | 2319 | ||
| 2200 | for (int i = 0; i < links[BPF_TRAMP_FENTRY].nr_links; i++) { | 2320 | for (int i = 0; i < nodes[BPF_TRAMP_FENTRY].nr_nodes; i++) { |
| 2201 | if (fentries.links[i]->link.prog->expected_attach_type == BPF_TRACE_FSESSION) | 2321 | if (fentries.nodes[i]->link->prog->expected_attach_type == BPF_TRACE_FSESSION) |
| 2322 | cnt++; | ||
| 2323 | if (fentries.nodes[i]->link->prog->expected_attach_type == BPF_TRACE_FSESSION_MULTI) | ||
| 2202 | cnt++; | 2324 | cnt++; |
| 2203 | } | 2325 | } |
| 2204 | 2326 | ||
| 2205 | return cnt; | 2327 | return cnt; |
| 2206 | } | 2328 | } |
| 2207 | 2329 | ||
| 2208 | static inline bool bpf_prog_calls_session_cookie(struct bpf_tramp_link *link) | 2330 | static inline bool bpf_prog_calls_session_cookie(struct bpf_tramp_node *node) |
| 2209 | { | 2331 | { |
| 2210 | return link->link.prog->call_session_cookie; | 2332 | return node->link->prog->call_session_cookie; |
| 2211 | } | 2333 | } |
| 2212 | 2334 | ||
| 2213 | static inline int bpf_fsession_cookie_cnt(struct bpf_tramp_links *links) | 2335 | static inline int bpf_fsession_cookie_cnt(struct bpf_tramp_nodes *nodes) |
| 2214 | { | 2336 | { |
| 2215 | struct bpf_tramp_links fentries = links[BPF_TRAMP_FENTRY]; | 2337 | struct bpf_tramp_nodes fentries = nodes[BPF_TRAMP_FENTRY]; |
| 2216 | int cnt = 0; | 2338 | int cnt = 0; |
| 2217 | 2339 | ||
| 2218 | for (int i = 0; i < links[BPF_TRAMP_FENTRY].nr_links; i++) { | 2340 | for (int i = 0; i < nodes[BPF_TRAMP_FENTRY].nr_nodes; i++) { |
| 2219 | if (bpf_prog_calls_session_cookie(fentries.links[i])) | 2341 | if (bpf_prog_calls_session_cookie(fentries.nodes[i])) |
| 2220 | cnt++; | 2342 | cnt++; |
| 2221 | } | 2343 | } |
| 2222 | 2344 | ||
| @@ -2598,6 +2720,7 @@ bool btf_record_equal(const struct btf_record *rec_a, const struct btf_record *r · 375 unmodified lines | |||
| 2598 | void bpf_obj_free_timer(const struct btf_record *rec, void *obj); | 2720 | void bpf_obj_free_timer(const struct btf_record *rec, void *obj); |
| 2599 | void bpf_obj_free_workqueue(const struct btf_record *rec, void *obj); | 2721 | void bpf_obj_free_workqueue(const struct btf_record *rec, void *obj); |
| 2600 | void bpf_obj_free_task_work(const struct btf_record *rec, void *obj); | 2722 | void bpf_obj_free_task_work(const struct btf_record *rec, void *obj); |
| 2723 | void bpf_obj_cancel_fields(struct bpf_map *map, void *obj); | ||
| 2601 | void bpf_obj_free_fields(const struct btf_record *rec, void *obj); | 2724 | void bpf_obj_free_fields(const struct btf_record *rec, void *obj); |
| 2602 | void __bpf_obj_drop_impl(void *p, const struct btf_record *rec, bool percpu); | 2725 | void __bpf_obj_drop_impl(void *p, const struct btf_record *rec, bool percpu); |
| 2603 | 2726 | ||
| @@ -2764,6 +2887,9 @@ void bpf_link_init(struct bpf_link *link, enum bpf_link_type type, · 160 unmodified lines | |||
| 2764 | void bpf_link_init_sleepable(struct bpf_link *link, enum bpf_link_type type, | 2887 | void bpf_link_init_sleepable(struct bpf_link *link, enum bpf_link_type type, |
| 2765 | const struct bpf_link_ops *ops, struct bpf_prog *prog, | 2888 | const struct bpf_link_ops *ops, struct bpf_prog *prog, |
| 2766 | enum bpf_attach_type attach_type, bool sleepable); | 2889 | enum bpf_attach_type attach_type, bool sleepable); |
| 2890 | void bpf_tramp_link_init(struct bpf_tramp_link *link, enum bpf_link_type type, | ||
| 2891 | const struct bpf_link_ops *ops, struct bpf_prog *prog, | ||
| 2892 | enum bpf_attach_type attach_type, u64 cookie); | ||
| 2767 | int bpf_link_prime(struct bpf_link *link, struct bpf_link_primer *primer); | 2893 | int bpf_link_prime(struct bpf_link *link, struct bpf_link_primer *primer); |
| 2768 | int bpf_link_settle(struct bpf_link_primer *primer); | 2894 | int bpf_link_settle(struct bpf_link_primer *primer); |
| 2769 | void bpf_link_cleanup(struct bpf_link_primer *primer); | 2895 | void bpf_link_cleanup(struct bpf_link_primer *primer); |
| @@ -2917,7 +3043,9 @@ int bpf_check_uarg_tail_zero(bpfptr_t uaddr, size_t expected_size, · 147 unmodified lines | |||
| 2917 | size_t actual_size); | 3043 | size_t actual_size); |
| 2918 | 3044 | ||
| 2919 | /* verify correctness of eBPF program */ | 3045 | /* verify correctness of eBPF program */ |
| 2920 | int bpf_check(struct bpf_prog **fp, union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size); | 3046 | struct bpf_log_attr; |
| 3047 | int bpf_check(struct bpf_prog **fp, union bpf_attr *attr, bpfptr_t uattr, | ||
| 3048 | struct bpf_log_attr *attr_log); | ||
| 2921 | 3049 | ||
| 2922 | #ifndef CONFIG_BPF_JIT_ALWAYS_ON | 3050 | #ifndef CONFIG_BPF_JIT_ALWAYS_ON |
| 2923 | int bpf_patch_call_args(struct bpf_insn *insn, u32 stack_depth); | 3051 | int bpf_patch_call_args(struct bpf_insn *insn, u32 stack_depth); |
| @@ -3088,6 +3216,56 @@ void bpf_dynptr_set_null(struct bpf_dynptr_kern *ptr); · 164 unmodified lines | |||
| 3088 | void bpf_dynptr_set_rdonly(struct bpf_dynptr_kern *ptr); | 3216 | void bpf_dynptr_set_rdonly(struct bpf_dynptr_kern *ptr); |
| 3089 | void bpf_prog_report_arena_violation(bool write, unsigned long addr, unsigned long fault_ip); | 3217 | void bpf_prog_report_arena_violation(bool write, unsigned long addr, unsigned long fault_ip); |
| 3090 | 3218 | ||
| 3219 | static __always_inline u32 | ||
| 3220 | bpf_prog_run_array_sleepable(const struct bpf_prog_array *array, | ||
| 3221 | const void *ctx, bpf_prog_run_fn run_prog) | ||
| 3222 | { | ||
| 3223 | const struct bpf_prog_array_item *item; | ||
| 3224 | struct bpf_prog *prog; | ||
| 3225 | struct bpf_run_ctx *old_run_ctx; | ||
| 3226 | struct bpf_trace_run_ctx run_ctx; | ||
| 3227 | u32 ret = 1; | ||
| 3228 | |||
| 3229 | if (unlikely(!array)) | ||
| 3230 | return ret; | ||
| 3231 | |||
| 3232 | migrate_disable(); | ||
| 3233 | |||
| 3234 | run_ctx.is_uprobe = false; | ||
| 3235 | |||
| 3236 | old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx); | ||
| 3237 | item = &array->items[0]; | ||
| 3238 | while ((prog = READ_ONCE(item->prog))) { | ||
| 3239 | /* Skip dummy_bpf_prog placeholder (len == 0) */ | ||
| 3240 | if (unlikely(!prog->len)) { | ||
| 3241 | item++; | ||
| 3242 | continue; | ||
| 3243 | } | ||
| 3244 | |||
| 3245 | if (unlikely(!bpf_prog_get_recursion_context(prog))) { | ||
| 3246 | bpf_prog_inc_misses_counter(prog); | ||
| 3247 | bpf_prog_put_recursion_context(prog); | ||
| 3248 | item++; | ||
| 3249 | continue; | ||
| 3250 | } | ||
| 3251 | |||
| 3252 | run_ctx.bpf_cookie = item->bpf_cookie; | ||
| 3253 | |||
| 3254 | if (!prog->sleepable) { | ||
| 3255 | guard(rcu)(); | ||
| 3256 | ret &= run_prog(prog, ctx); | ||
| 3257 | } else { | ||
| 3258 | ret &= run_prog(prog, ctx); | ||
| 3259 | } | ||
| 3260 | |||
| 3261 | bpf_prog_put_recursion_context(prog); | ||
| 3262 | item++; | ||
| 3263 | } | ||
| 3264 | bpf_reset_run_ctx(old_run_ctx); | ||
| 3265 | migrate_enable(); | ||
| 3266 | return ret; | ||
| 3267 | } | ||
| 3268 | |||
| 3091 | #else /* !CONFIG_BPF_SYSCALL */ | 3269 | #else /* !CONFIG_BPF_SYSCALL */ |
| 3092 | static inline struct bpf_prog *bpf_prog_get(u32 ufd) | 3270 | static inline struct bpf_prog *bpf_prog_get(u32 ufd) |
| 3093 | { | 3271 | { |
| @@ -3135,6 +3313,12 @@ static inline void bpf_link_init_sleepable(struct bpf_link *link, enum bpf_link_ · 41 unmodified lines | |||
| 3135 | { | 3313 | { |
| 3136 | } | 3314 | } |
| 3137 | 3315 | ||
| 3316 | static inline void bpf_tramp_link_init(struct bpf_tramp_link *link, enum bpf_link_type type, | ||
| 3317 | const struct bpf_link_ops *ops, struct bpf_prog *prog, | ||
| 3318 | enum bpf_attach_type attach_type, u64 cookie) | ||
| 3319 | { | ||
| 3320 | } | ||
| 3321 | |||
| 3138 | static inline int bpf_link_prime(struct bpf_link *link, | 3322 | static inline int bpf_link_prime(struct bpf_link *link, |
| 3139 | struct bpf_link_primer *primer) | 3323 | struct bpf_link_primer *primer) |
| 3140 | { | 3324 | { |
| @@ -3626,15 +3810,25 @@ static inline int bpf_fd_reuseport_array_update_elem(struct bpf_map *map, · 485 unmodified lines | |||
| 3626 | #endif /* CONFIG_BPF_SYSCALL */ | 3810 | #endif /* CONFIG_BPF_SYSCALL */ |
| 3627 | #endif /* defined(CONFIG_INET) && defined(CONFIG_BPF_SYSCALL) */ | 3811 | #endif /* defined(CONFIG_INET) && defined(CONFIG_BPF_SYSCALL) */ |
| 3628 | 3812 | ||
| 3629 | #if defined(CONFIG_KEYS) && defined(CONFIG_BPF_SYSCALL) | 3813 | #ifdef CONFIG_KEYS |
| 3814 | struct bpf_key { | ||
| 3815 | struct key *key; | ||
| 3816 | bool has_ref; | ||
| 3817 | }; | ||
| 3818 | #endif /* CONFIG_KEYS */ | ||
| 3630 | 3819 | ||
| 3820 | #if defined(CONFIG_KEYS) && defined(CONFIG_BPF_SYSCALL) | ||
| 3631 | struct bpf_key *bpf_lookup_user_key(s32 serial, u64 flags); | 3821 | struct bpf_key *bpf_lookup_user_key(s32 serial, u64 flags); |
| 3632 | struct bpf_key *bpf_lookup_system_key(u64 id); | 3822 | struct bpf_key *bpf_lookup_system_key(u64 id); |
| 3633 | void bpf_key_put(struct bpf_key *bkey); | 3823 | void bpf_key_put(struct bpf_key *bkey); |
| 3634 | int bpf_verify_pkcs7_signature(struct bpf_dynptr *data_p, | 3824 | int bpf_verify_pkcs7_signature(const struct bpf_dynptr *data_p, |
| 3635 | struct bpf_dynptr *sig_p, | 3825 | const struct bpf_dynptr *sig_p, |
| 3636 | struct bpf_key *trusted_keyring); | 3826 | struct bpf_key *trusted_keyring); |
| 3637 | 3827 | ||
| 3828 | static inline s32 bpf_key_serial(const struct bpf_key *key) | ||
| 3829 | { | ||
| 3830 | return key->has_ref ? key->key->serial : 0; | ||
| 3831 | } | ||
| 3638 | #else | 3832 | #else |
| 3639 | static inline struct bpf_key *bpf_lookup_user_key(u32 serial, u64 flags) | 3833 | static inline struct bpf_key *bpf_lookup_user_key(u32 serial, u64 flags) |
| 3640 | { | 3834 | { |
| @@ -3650,12 +3844,17 @@ static inline void bpf_key_put(struct bpf_key *bkey) · 9 unmodified lines | |||
| 3650 | { | 3844 | { |
| 3651 | } | 3845 | } |
| 3652 | 3846 | ||
| 3653 | static inline int bpf_verify_pkcs7_signature(struct bpf_dynptr *data_p, | 3847 | static inline int bpf_verify_pkcs7_signature(const struct bpf_dynptr *data_p, |
| 3654 | struct bpf_dynptr *sig_p, | 3848 | const struct bpf_dynptr *sig_p, |
| 3655 | struct bpf_key *trusted_keyring) | 3849 | struct bpf_key *trusted_keyring) |
| 3656 | { | 3850 | { |
| 3657 | return -EOPNOTSUPP; | 3851 | return -EOPNOTSUPP; |
| 3658 | } | 3852 | } |
| 3853 | |||
| 3854 | static inline s32 bpf_key_serial(const struct bpf_key *key) | ||
| 3855 | { | ||
| 3856 | return 0; | ||
| 3857 | } | ||
| 3659 | #endif /* defined(CONFIG_KEYS) && defined(CONFIG_BPF_SYSCALL) */ | 3858 | #endif /* defined(CONFIG_KEYS) && defined(CONFIG_BPF_SYSCALL) */ |
| 3660 | 3859 | ||
| 3661 | /* verifier prototypes for helper functions called from eBPF programs */ | 3860 | /* verifier prototypes for helper functions called from eBPF programs */ |
| @@ -3931,15 +4130,6 @@ static inline void bpf_cgroup_atype_get(u32 attach_btf_id, int cgroup_atype) {} · 269 unmodified lines | |||
| 3931 | static inline void bpf_cgroup_atype_put(int cgroup_atype) {} | 4130 | static inline void bpf_cgroup_atype_put(int cgroup_atype) {} |
| 3932 | #endif /* CONFIG_BPF_LSM */ | 4131 | #endif /* CONFIG_BPF_LSM */ |
| 3933 | 4132 | ||
| 3934 | struct key; | ||
| 3935 | |||
| 3936 | #ifdef CONFIG_KEYS | ||
| 3937 | struct bpf_key { | ||
| 3938 | struct key *key; | ||
| 3939 | bool has_ref; | ||
| 3940 | }; | ||
| 3941 | #endif /* CONFIG_KEYS */ | ||
| 3942 | |||
| 3943 | static inline bool type_is_alloc(u32 type) | 4133 | static inline bool type_is_alloc(u32 type) |
| 3944 | { | 4134 | { |
| 3945 | return type & MEM_ALLOC; | 4135 | return type & MEM_ALLOC; |
| @@ -0,0 +1,19 @@ | |||
| 1 | /* SPDX-License-Identifier: GPL-2.0-or-later */ | ||
| 2 | /* | ||
| 3 | * Subset of bpf.h declarations, split out so files that need only these | ||
| 4 | * declarations can avoid bpf.h's full include cost. | ||
| 5 | */ | ||
| 6 | #ifndef _LINUX_BPF_DEFS_H | ||
| 7 | #define _LINUX_BPF_DEFS_H | ||
| 8 | |||
| 9 | #ifdef CONFIG_BPF_SYSCALL | ||
| 10 | bool bpf_arena_handle_page_fault(unsigned long addr, bool is_write, unsigned long fault_ip); | ||
| 11 | #else | ||
| 12 | static inline bool bpf_arena_handle_page_fault(unsigned long addr, bool is_write, | ||
| 13 | unsigned long fault_ip) | ||
| 14 | { | ||
| 15 | return false; | ||
| 16 | } | ||
| 17 | #endif | ||
| 18 | |||
| 19 | #endif /* _LINUX_BPF_DEFS_H */ | ||
| @@ -52,6 +52,7 @@ int bpf_set_dentry_xattr_locked(struct dentry *dentry, const char *name__str, · 51 unmodified lines | |||
| 52 | const struct bpf_dynptr *value_p, int flags); | 52 | const struct bpf_dynptr *value_p, int flags); |
| 53 | int bpf_remove_dentry_xattr_locked(struct dentry *dentry, const char *name__str); | 53 | int bpf_remove_dentry_xattr_locked(struct dentry *dentry, const char *name__str); |
| 54 | bool bpf_lsm_has_d_inode_locked(const struct bpf_prog *prog); | 54 | bool bpf_lsm_has_d_inode_locked(const struct bpf_prog *prog); |
| 55 | bool bpf_lsm_hook_returns_errno(u32 btf_id); | ||
| 55 | 56 | ||
| 56 | #else /* !CONFIG_BPF_LSM */ | 57 | #else /* !CONFIG_BPF_LSM */ |
| 57 | 58 | ||
| @@ -104,6 +105,11 @@ static inline bool bpf_lsm_has_d_inode_locked(const struct bpf_prog *prog) · 46 unmodified lines | |||
| 104 | { | 105 | { |
| 105 | return false; | 106 | return false; |
| 106 | } | 107 | } |
| 108 | |||
| 109 | static inline bool bpf_lsm_hook_returns_errno(u32 btf_id) | ||
| 110 | { | ||
| 111 | return true; | ||
| 112 | } | ||
| 107 | #endif /* CONFIG_BPF_LSM */ | 113 | #endif /* CONFIG_BPF_LSM */ |
| 108 | 114 | ||
| 109 | #endif /* _LINUX_BPF_LSM_H */ | 115 | #endif /* _LINUX_BPF_LSM_H */ |
| @@ -134,6 +134,7 @@ BPF_MAP_TYPE(BPF_MAP_TYPE_BLOOM_FILTER, bloom_filter_map_ops) · 133 unmodified lines | |||
| 134 | BPF_MAP_TYPE(BPF_MAP_TYPE_USER_RINGBUF, user_ringbuf_map_ops) | 134 | BPF_MAP_TYPE(BPF_MAP_TYPE_USER_RINGBUF, user_ringbuf_map_ops) |
| 135 | BPF_MAP_TYPE(BPF_MAP_TYPE_ARENA, arena_map_ops) | 135 | BPF_MAP_TYPE(BPF_MAP_TYPE_ARENA, arena_map_ops) |
| 136 | BPF_MAP_TYPE(BPF_MAP_TYPE_INSN_ARRAY, insn_array_map_ops) | 136 | BPF_MAP_TYPE(BPF_MAP_TYPE_INSN_ARRAY, insn_array_map_ops) |
| 137 | BPF_MAP_TYPE(BPF_MAP_TYPE_RHASH, rhtab_map_ops) | ||
| 137 | 138 | ||
| 138 | BPF_LINK_TYPE(BPF_LINK_TYPE_RAW_TRACEPOINT, raw_tracepoint) | 139 | BPF_LINK_TYPE(BPF_LINK_TYPE_RAW_TRACEPOINT, raw_tracepoint) |
| 139 | BPF_LINK_TYPE(BPF_LINK_TYPE_TRACING, tracing) | 140 | BPF_LINK_TYPE(BPF_LINK_TYPE_TRACING, tracing) |
| @@ -155,3 +156,4 @@ BPF_LINK_TYPE(BPF_LINK_TYPE_PERF_EVENT, perf) · 15 unmodified lines | |||
| 155 | BPF_LINK_TYPE(BPF_LINK_TYPE_KPROBE_MULTI, kprobe_multi) | 156 | BPF_LINK_TYPE(BPF_LINK_TYPE_KPROBE_MULTI, kprobe_multi) |
| 156 | BPF_LINK_TYPE(BPF_LINK_TYPE_STRUCT_OPS, struct_ops) | 157 | BPF_LINK_TYPE(BPF_LINK_TYPE_STRUCT_OPS, struct_ops) |
| 157 | BPF_LINK_TYPE(BPF_LINK_TYPE_UPROBE_MULTI, uprobe_multi) | 158 | BPF_LINK_TYPE(BPF_LINK_TYPE_UPROBE_MULTI, uprobe_multi) |
| 159 | BPF_LINK_TYPE(BPF_LINK_TYPE_TRACING_MULTI, tracing_multi) | ||
| @@ -8,6 +8,7 @@ · 7 unmodified lines | |||
| 8 | #include <linux/btf.h> /* for struct btf and btf_id() */ | 8 | #include <linux/btf.h> /* for struct btf and btf_id() */ |
| 9 | #include <linux/filter.h> /* for MAX_BPF_STACK */ | 9 | #include <linux/filter.h> /* for MAX_BPF_STACK */ |
| 10 | #include <linux/tnum.h> | 10 | #include <linux/tnum.h> |
| 11 | #include <linux/cnum.h> | ||
| 11 | 12 | ||
| 12 | /* Maximum variable offset umax_value permitted when resolving memory accesses. | 13 | /* Maximum variable offset umax_value permitted when resolving memory accesses. |
| 13 | * In practice this is far bigger than any realistic pointer offset; this limit | 14 | * In practice this is far bigger than any realistic pointer offset; this limit |
| @@ -65,7 +66,6 @@ struct bpf_reg_state { · 51 unmodified lines | |||
| 65 | 66 | ||
| 66 | struct { /* for PTR_TO_MEM | PTR_TO_MEM_OR_NULL */ | 67 | struct { /* for PTR_TO_MEM | PTR_TO_MEM_OR_NULL */ |
| 67 | u32 mem_size; | 68 | u32 mem_size; |
| 68 | u32 dynptr_id; /* for dynptr slices */ | ||
| 69 | }; | 69 | }; |
| 70 | 70 | ||
| 71 | /* For dynptr stack slots */ | 71 | /* For dynptr stack slots */ |
| @@ -120,14 +120,8 @@ struct bpf_reg_state { · 48 unmodified lines | |||
| 120 | * These refer to the same value as var_off, not necessarily the actual | 120 | * These refer to the same value as var_off, not necessarily the actual |
| 121 | * contents of the register. | 121 | * contents of the register. |
| 122 | */ | 122 | */ |
| 123 | s64 smin_value; /* minimum possible (s64)value */ | 123 | struct cnum64 r64; /* 64-bit range as circular number */ |
| 124 | s64 smax_value; /* maximum possible (s64)value */ | 124 | struct cnum32 r32; /* 32-bit range as circular number */ |
| 125 | u64 umin_value; /* minimum possible (u64)value */ | ||
| 126 | u64 umax_value; /* maximum possible (u64)value */ | ||
| 127 | s32 s32_min_value; /* minimum possible (s32)value */ | ||
| 128 | s32 s32_max_value; /* maximum possible (s32)value */ | ||
| 129 | u32 u32_min_value; /* minimum possible (u32)value */ | ||
| 130 | u32 u32_max_value; /* maximum possible (u32)value */ | ||
| 131 | /* For PTR_TO_PACKET, used to find other pointers with the same variable | 125 | /* For PTR_TO_PACKET, used to find other pointers with the same variable |
| 132 | * offset, so they can share range knowledge. | 126 | * offset, so they can share range knowledge. |
| 133 | * For PTR_TO_MAP_VALUE_OR_NULL this is used to share which map value we | 127 | * For PTR_TO_MAP_VALUE_OR_NULL this is used to share which map value we |
| @@ -153,46 +147,14 @@ struct bpf_reg_state { · 19 unmodified lines | |||
| 153 | #define BPF_ADD_CONST32 (1U << 30) | 147 | #define BPF_ADD_CONST32 (1U << 30) |
| 154 | #define BPF_ADD_CONST (BPF_ADD_CONST64 | BPF_ADD_CONST32) | 148 | #define BPF_ADD_CONST (BPF_ADD_CONST64 | BPF_ADD_CONST32) |
| 155 | u32 id; | 149 | u32 id; |
| 156 | /* PTR_TO_SOCKET and PTR_TO_TCP_SOCK could be a ptr returned | 150 | /* |
| 157 | * from a pointer-cast helper, bpf_sk_fullsock() and | 151 | * Tracks the parent object this register was derived from. |
| 158 | * bpf_tcp_sock(). | 152 | * Used for cascading invalidation: when the parent object is |
| 159 | * | 153 | * released or invalidated, all registers with matching parent_id |
| 160 | * Consider the following where "sk" is a reference counted | 154 | * are also invalidated. For example, a slice from bpf_dynptr_data() |
| 161 | * pointer returned from "sk = bpf_sk_lookup_tcp();": | 155 | * gets parent_id set to the dynptr's id. |
| 162 | * | ||
| 163 | * 1: sk = bpf_sk_lookup_tcp(); | ||
| 164 | * 2: if (!sk) { return 0; } | ||
| 165 | * 3: fullsock = bpf_sk_fullsock(sk); | ||
| 166 | * 4: if (!fullsock) { bpf_sk_release(sk); return 0; } | ||
| 167 | * 5: tp = bpf_tcp_sock(fullsock); | ||
| 168 | * 6: if (!tp) { bpf_sk_release(sk); return 0; } | ||
| 169 | * 7: bpf_sk_release(sk); | ||
| 170 | * 8: snd_cwnd = tp->snd_cwnd; // verifier will complain | ||
| 171 | * | ||
| 172 | * After bpf_sk_release(sk) at line 7, both "fullsock" ptr and | ||
| 173 | * "tp" ptr should be invalidated also. In order to do that, | ||
| 174 | * the reg holding "fullsock" and "sk" need to remember | ||
| 175 | * the original refcounted ptr id (i.e. sk_reg->id) in ref_obj_id | ||
| 176 | * such that the verifier can reset all regs which have | ||
| 177 | * ref_obj_id matching the sk_reg->id. | ||
| 178 | * | ||
| 179 | * sk_reg->ref_obj_id is set to sk_reg->id at line 1. | ||
| 180 | * sk_reg->id will stay as NULL-marking purpose only. | ||
| 181 | * After NULL-marking is done, sk_reg->id can be reset to 0. | ||
| 182 | * | ||
| 183 | * After "fullsock = bpf_sk_fullsock(sk);" at line 3, | ||
| 184 | * fullsock_reg->ref_obj_id is set to sk_reg->ref_obj_id. | ||
| 185 | * | ||
| 186 | * After "tp = bpf_tcp_sock(fullsock);" at line 5, | ||
| 187 | * tp_reg->ref_obj_id is set to fullsock_reg->ref_obj_id | ||
| 188 | * which is the same as sk_reg->ref_obj_id. | ||
| 189 | * | ||
| 190 | * From the verifier perspective, if sk, fullsock and tp | ||
| 191 | * are not NULL, they are the same ptr with different | ||
| 192 | * reg->type. In particular, bpf_sk_release(tp) is also | ||
| 193 | * allowed and has the same effect as bpf_sk_release(sk). | ||
| 194 | */ | 156 | */ |
| 195 | u32 ref_obj_id; | 157 | u32 parent_id; |
| 196 | /* Inside the callee two registers can be both PTR_TO_STACK like | 158 | /* Inside the callee two registers can be both PTR_TO_STACK like |
| 197 | * R1=fp-8 and R2=fp-8, but one of them points to this function stack | 159 | * R1=fp-8 and R2=fp-8, but one of them points to this function stack |
| 198 | * while another to the caller's stack. To differentiate them 'frameno' | 160 | * while another to the caller's stack. To differentiate them 'frameno' |
| @@ -209,6 +171,66 @@ struct bpf_reg_state { · 10 unmodified lines | |||
| 209 | bool precise; | 171 | bool precise; |
| 210 | }; | 172 | }; |
| 211 | 173 | ||
| 174 | static inline s64 reg_smin(const struct bpf_reg_state *reg) | ||
| 175 | { | ||
| 176 | return cnum64_smin(reg->r64); | ||
| 177 | } | ||
| 178 | |||
| 179 | static inline s64 reg_smax(const struct bpf_reg_state *reg) | ||
| 180 | { | ||
| 181 | return cnum64_smax(reg->r64); | ||
| 182 | } | ||
| 183 | |||
| 184 | static inline u64 reg_umin(const struct bpf_reg_state *reg) | ||
| 185 | { | ||
| 186 | return cnum64_umin(reg->r64); | ||
| 187 | } | ||
| 188 | |||
| 189 | static inline u64 reg_umax(const struct bpf_reg_state *reg) | ||
| 190 | { | ||
| 191 | return cnum64_umax(reg->r64); | ||
| 192 | } | ||
| 193 | |||
| 194 | static inline s32 reg_s32_min(const struct bpf_reg_state *reg) | ||
| 195 | { | ||
| 196 | return cnum32_smin(reg->r32); | ||
| 197 | } | ||
| 198 | |||
| 199 | static inline s32 reg_s32_max(const struct bpf_reg_state *reg) | ||
| 200 | { | ||
| 201 | return cnum32_smax(reg->r32); | ||
| 202 | } | ||
| 203 | |||
| 204 | static inline u32 reg_u32_min(const struct bpf_reg_state *reg) | ||
| 205 | { | ||
| 206 | return cnum32_umin(reg->r32); | ||
| 207 | } | ||
| 208 | |||
| 209 | static inline u32 reg_u32_max(const struct bpf_reg_state *reg) | ||
| 210 | { | ||
| 211 | return cnum32_umax(reg->r32); | ||
| 212 | } | ||
| 213 | |||
| 214 | static inline void reg_set_srange32(struct bpf_reg_state *reg, s32 smin, s32 smax) | ||
| 215 | { | ||
| 216 | reg->r32 = cnum32_from_srange(smin, smax); | ||
| 217 | } | ||
| 218 | |||
| 219 | static inline void reg_set_urange32(struct bpf_reg_state *reg, u32 umin, u32 umax) | ||
| 220 | { | ||
| 221 | reg->r32 = cnum32_from_urange(umin, umax); | ||
| 222 | } | ||
| 223 | |||
| 224 | static inline void reg_set_srange64(struct bpf_reg_state *reg, s64 smin, s64 smax) | ||
| 225 | { | ||
| 226 | reg->r64 = cnum64_from_srange(smin, smax); | ||
| 227 | } | ||
| 228 | |||
| 229 | static inline void reg_set_urange64(struct bpf_reg_state *reg, u64 umin, u64 umax) | ||
| 230 | { | ||
| 231 | reg->r64 = cnum64_from_urange(umin, umax); | ||
| 232 | } | ||
| 233 | |||
| 212 | enum bpf_stack_slot_type { | 234 | enum bpf_stack_slot_type { |
| 213 | STACK_INVALID, /* nothing was stored in this stack slot */ | 235 | STACK_INVALID, /* nothing was stored in this stack slot */ |
| 214 | STACK_SPILL, /* register spilled into stack */ | 236 | STACK_SPILL, /* register spilled into stack */ |
| @@ -309,10 +331,14 @@ struct bpf_reference_state { · 94 unmodified lines | |||
| 309 | * is used purely to inform the user of a reference leak. | 331 | * is used purely to inform the user of a reference leak. |
| 310 | */ | 332 | */ |
| 311 | int insn_idx; | 333 | int insn_idx; |
| 312 | /* Use to keep track of the source object of a lock, to ensure | 334 | union { |
| 313 | * it matches on unlock. | 335 | /* For REF_TYPE_PTR */ |
| 314 | */ | 336 | int parent_id; |
| 315 | void *ptr; | 337 | /* Use to keep track of the source object of a lock, to ensure |
| 338 | * it matches on unlock. | ||
| 339 | */ | ||
| 340 | void *ptr; | ||
| 341 | }; | ||
| 316 | }; | 342 | }; |
| 317 | 343 | ||
| 318 | struct bpf_retval_range { | 344 | struct bpf_retval_range { |
| @@ -347,6 +373,7 @@ struct bpf_func_state { · 28 unmodified lines | |||
| 347 | bool in_callback_fn; | 373 | bool in_callback_fn; |
| 348 | bool in_async_callback_fn; | 374 | bool in_async_callback_fn; |
| 349 | bool in_exception_callback_fn; | 375 | bool in_exception_callback_fn; |
| 376 | bool no_stack_arg_load; | ||
| 350 | /* For callback calling functions that limit number of possible | 377 | /* For callback calling functions that limit number of possible |
| 351 | * callback executions (e.g. bpf_loop) keeps track of current | 378 | * callback executions (e.g. bpf_loop) keeps track of current |
| 352 | * simulated iteration number. | 379 | * simulated iteration number. |
| @@ -372,46 +399,49 @@ struct bpf_func_state { · 19 unmodified lines | |||
| 372 | * `stack`. allocated_stack is always a multiple of BPF_REG_SIZE. | 399 | * `stack`. allocated_stack is always a multiple of BPF_REG_SIZE. |
| 373 | */ | 400 | */ |
| 374 | int allocated_stack; | 401 | int allocated_stack; |
| 402 | |||
| 403 | u16 out_stack_arg_cnt; /* Number of outgoing on-stack argument slots */ | ||
| 404 | struct bpf_reg_state *stack_arg_regs; /* Outgoing on-stack arguments */ | ||
| 375 | }; | 405 | }; |
| 376 | 406 | ||
| 377 | #define MAX_CALL_FRAMES 8 | 407 | #define MAX_CALL_FRAMES 16 |
| 378 | 408 | ||
| 379 | /* instruction history flags, used in bpf_jmp_history_entry.flags field */ | 409 | /* instruction history flags, used in bpf_jmp_history_entry.flags field. |
| 410 | * Frame number and SPI are stored in dedicated fields of bpf_jmp_history_entry. | ||
| 411 | */ | ||
| 380 | enum { | 412 | enum { |
| 381 | /* instruction references stack slot through PTR_TO_STACK register; | 413 | INSN_F_STACK_ACCESS = BIT(0), |
| 382 | * we also store stack's frame number in lower 3 bits (MAX_CALL_FRAMES is 8) | ||
| 383 | * and accessed stack slot's index in next 6 bits (MAX_BPF_STACK is 512, | ||
| 384 | * 8 bytes per slot, so slot index (spi) is [0, 63]) | ||
| 385 | */ | ||
| 386 | INSN_F_FRAMENO_MASK = 0x7, /* 3 bits */ | ||
| 387 | |||
| 388 | INSN_F_SPI_MASK = 0x3f, /* 6 bits */ | ||
| 389 | INSN_F_SPI_SHIFT = 3, /* shifted 3 bits to the left */ | ||
| 390 | 414 | ||
| 391 | INSN_F_STACK_ACCESS = BIT(9), | 415 | INSN_F_DST_REG_STACK = BIT(1), /* dst_reg is PTR_TO_STACK */ |
| 416 | INSN_F_SRC_REG_STACK = BIT(2), /* src_reg is PTR_TO_STACK */ | ||
| 392 | 417 | ||
| 393 | INSN_F_DST_REG_STACK = BIT(10), /* dst_reg is PTR_TO_STACK */ | 418 | INSN_F_STACK_ARG_ACCESS = BIT(3), |
| 394 | INSN_F_SRC_REG_STACK = BIT(11), /* src_reg is PTR_TO_STACK */ | ||
| 395 | /* total 12 bits are used now. */ | ||
| 396 | }; | 419 | }; |
| 397 | 420 | ||
| 398 | static_assert(INSN_F_FRAMENO_MASK + 1 >= MAX_CALL_FRAMES); | ||
| 399 | static_assert(INSN_F_SPI_MASK + 1 >= MAX_BPF_STACK / 8); | ||
| 400 | |||
| 401 | struct bpf_jmp_history_entry { | 421 | struct bpf_jmp_history_entry { |
| 402 | u32 idx; | ||
| 403 | /* insn idx can't be bigger than 1 million */ | 422 | /* insn idx can't be bigger than 1 million */ |
| 423 | u32 idx : 20; | ||
| 424 | u32 frame : 4; /* stack access frame number */ | ||
| 425 | u32 spi : 6; /* stack slot index (0..63) */ | ||
| 426 | u32 : 2; | ||
| 404 | u32 prev_idx : 20; | 427 | u32 prev_idx : 20; |
| 405 | /* special INSN_F_xxx flags */ | 428 | /* special INSN_F_xxx flags */ |
| 406 | u32 flags : 12; | 429 | u32 flags : 4; |
| 407 | /* additional registers that need precision tracking when this | 430 | u32 : 8; |
| 408 | * jump is backtracked, vector of six 10-bit records | 431 | /* |
| 432 | * additional registers that need precision tracking when this | ||
| 433 | * jump is backtracked, vector of five 11-bit records | ||
| 409 | */ | 434 | */ |
| 410 | u64 linked_regs; | 435 | u64 linked_regs; |
| 411 | }; | 436 | }; |
| 412 | 437 | ||
| 413 | /* Maximum number of register states that can exist at once */ | 438 | static_assert(MAX_CALL_FRAMES <= (1 << 4)); |
| 414 | #define BPF_ID_MAP_SIZE ((MAX_BPF_REG + MAX_BPF_STACK / BPF_REG_SIZE) * MAX_CALL_FRAMES) | 439 | static_assert(MAX_BPF_STACK / 8 <= (1 << 6)); |
| 440 | |||
| 441 | /* Maximum number of bpf_reg_state objects that can exist at once */ | ||
| 442 | #define MAX_STACK_ARG_SLOTS (MAX_BPF_FUNC_ARGS - MAX_BPF_FUNC_REG_ARGS) | ||
| 443 | #define BPF_ID_MAP_SIZE ((MAX_BPF_REG + MAX_BPF_STACK / BPF_REG_SIZE + \ | ||
| 444 | MAX_STACK_ARG_SLOTS) * MAX_CALL_FRAMES) | ||
| 415 | struct bpf_verifier_state { | 445 | struct bpf_verifier_state { |
| 416 | /* call stack tracking */ | 446 | /* call stack tracking */ |
| 417 | struct bpf_func_state *frame[MAX_CALL_FRAMES]; | 447 | struct bpf_func_state *frame[MAX_CALL_FRAMES]; |
| @@ -497,10 +527,23 @@ struct bpf_verifier_state { · 79 unmodified lines | |||
| 497 | u32 may_goto_depth; | 527 | u32 may_goto_depth; |
| 498 | }; | 528 | }; |
| 499 | 529 | ||
| 500 | #define bpf_get_spilled_reg(slot, frame, mask) \ | 530 | static inline struct bpf_reg_state * |
| 501 | (((slot < frame->allocated_stack / BPF_REG_SIZE) && \ | 531 | bpf_get_spilled_reg(int slot, struct bpf_func_state *frame, u32 mask) |
| 502 | ((1 << frame->stack[slot].slot_type[BPF_REG_SIZE - 1]) & (mask))) \ | 532 | { |
| 503 | ? &frame->stack[slot].spilled_ptr : NULL) | 533 | if (slot < frame->allocated_stack / BPF_REG_SIZE && |
| 534 | (1 << frame->stack[slot].slot_type[BPF_REG_SIZE - 1]) & mask) | ||
| 535 | return &frame->stack[slot].spilled_ptr; | ||
| 536 | return NULL; | ||
| 537 | } | ||
| 538 | |||
| 539 | static inline struct bpf_reg_state * | ||
| 540 | bpf_get_spilled_stack_arg(int slot, struct bpf_func_state *frame) | ||
| 541 | { | ||
| 542 | if (slot < frame->out_stack_arg_cnt && | ||
| 543 | frame->stack_arg_regs[slot].type != NOT_INIT) | ||
| 544 | return &frame->stack_arg_regs[slot]; | ||
| 545 | return NULL; | ||
| 546 | } | ||
| 504 | 547 | ||
| 505 | /* Iterate over 'frame', setting 'reg' to either NULL or a spilled register. */ | 548 | /* Iterate over 'frame', setting 'reg' to either NULL or a spilled register. */ |
| 506 | #define bpf_for_each_spilled_reg(iter, frame, reg, mask) \ | 549 | #define bpf_for_each_spilled_reg(iter, frame, reg, mask) \ |
| @@ -508,7 +551,13 @@ struct bpf_verifier_state { · 1 unmodified line | |||
| 508 | iter < frame->allocated_stack / BPF_REG_SIZE; \ | 551 | iter < frame->allocated_stack / BPF_REG_SIZE; \ |
| 509 | iter++, reg = bpf_get_spilled_reg(iter, frame, mask)) | 552 | iter++, reg = bpf_get_spilled_reg(iter, frame, mask)) |
| 510 | 553 | ||
| 511 | #define bpf_for_each_reg_in_vstate_mask(__vst, __state, __reg, __mask, __expr) \ | 554 | /* Iterate over 'frame', setting 'reg' to either NULL or a spilled stack arg. */ |
| 555 | #define bpf_for_each_spilled_stack_arg(iter, frame, reg) \ | ||
| 556 | for (iter = 0, reg = bpf_get_spilled_stack_arg(iter, frame); \ | ||
| 557 | iter < frame->out_stack_arg_cnt; \ | ||
| 558 | iter++, reg = bpf_get_spilled_stack_arg(iter, frame)) | ||
| 559 | |||
| 560 | #define bpf_for_each_reg_in_vstate_mask(__vst, __state, __reg, __stack, __mask, __expr) \ | ||
| 512 | ({ \ | 561 | ({ \ |
| 513 | struct bpf_verifier_state *___vstate = __vst; \ | 562 | struct bpf_verifier_state *___vstate = __vst; \ |
| 514 | int ___i, ___j; \ | 563 | int ___i, ___j; \ |
| @@ -516,6 +565,7 @@ struct bpf_verifier_state { · 1 unmodified line | |||
| 516 | struct bpf_reg_state *___regs; \ | 565 | struct bpf_reg_state *___regs; \ |
| 517 | __state = ___vstate->frame[___i]; \ | 566 | __state = ___vstate->frame[___i]; \ |
| 518 | ___regs = __state->regs; \ | 567 | ___regs = __state->regs; \ |
| 568 | __stack = NULL; \ | ||
| 519 | for (___j = 0; ___j < MAX_BPF_REG; ___j++) { \ | 569 | for (___j = 0; ___j < MAX_BPF_REG; ___j++) { \ |
| 520 | __reg = &___regs[___j]; \ | 570 | __reg = &___regs[___j]; \ |
| 521 | (void)(__expr); \ | 571 | (void)(__expr); \ |
| @@ -523,14 +573,27 @@ struct bpf_verifier_state { · 1 unmodified line | |||
| 523 | bpf_for_each_spilled_reg(___j, __state, __reg, __mask) { \ | 573 | bpf_for_each_spilled_reg(___j, __state, __reg, __mask) { \ |
| 524 | if (!__reg) \ | 574 | if (!__reg) \ |
| 525 | continue; \ | 575 | continue; \ |
| 576 | __stack = &__state->stack[___j]; \ | ||
| 526 | (void)(__expr); \ | 577 | (void)(__expr); \ |
| 527 | } \ | 578 | } \ |
| 579 | __stack = NULL; \ | ||
| 580 | bpf_for_each_spilled_stack_arg(___j, __state, __reg) { \ | ||
| 581 | if (!__reg) \ | ||
| 582 | continue; \ | ||
| 583 | (void)(__expr); \ | ||
| 584 | } \ | ||
| 528 | } \ | 585 | } \ |
| 586 | (void)__stack; \ | ||
| 529 | }) | 587 | }) |
| 530 | 588 | ||
| 531 | /* Invoke __expr over regsiters in __vst, setting __state and __reg */ | 589 | /* Invoke __expr over regsiters in __vst, setting __state and __reg */ |
| 532 | #define bpf_for_each_reg_in_vstate(__vst, __state, __reg, __expr) \ | 590 | #define bpf_for_each_reg_in_vstate(__vst, __state, __reg, __expr) \ |
| 533 | bpf_for_each_reg_in_vstate_mask(__vst, __state, __reg, 1 << STACK_SPILL, __expr) | 591 | ({ \ |
| 592 | struct bpf_stack_state * ___stack; \ | ||
| 593 | (void)___stack; \ | ||
| 594 | bpf_for_each_reg_in_vstate_mask(__vst, __state, __reg, ___stack,\ | ||
| 595 | 1 << STACK_SPILL, __expr); \ | ||
| 596 | }) | ||
| 534 | 597 | ||
| 535 | /* linked list of verifier states used to prune search */ | 598 | /* linked list of verifier states used to prune search */ |
| 536 | struct bpf_verifier_state_list { | 599 | struct bpf_verifier_state_list { |
| @@ -700,6 +763,22 @@ static inline bool bpf_verifier_log_needed(const struct bpf_verifier_log *log) · 163 unmodified lines | |||
| 700 | return log && log->level; | 763 | return log && log->level; |
| 701 | } | 764 | } |
| 702 | 765 | ||
| 766 | struct bpf_log_attr { | ||
| 767 | char __user *ubuf; | ||
| 768 | u32 size; | ||
| 769 | u32 level; | ||
| 770 | u32 offsetof_true_size; | ||
| 771 | bpfptr_t uattr; | ||
| 772 | }; | ||
| 773 | |||
| 774 | int bpf_log_attr_init(struct bpf_log_attr *log, u64 log_buf, u32 log_size, u32 log_level, | ||
| 775 | u32 offsetof_log_true_size, bpfptr_t uattr, struct bpf_common_attr *common, | ||
| 776 | bpfptr_t uattr_common, u32 size_common); | ||
| 777 | struct bpf_verifier_log *bpf_log_attr_create_vlog(struct bpf_log_attr *attr_log, | ||
| 778 | struct bpf_common_attr *common, bpfptr_t uattr, | ||
| 779 | u32 size); | ||
| 780 | int bpf_log_attr_finalize(struct bpf_log_attr *attr, struct bpf_verifier_log *log); | ||
| 781 | |||
| 703 | #define BPF_MAX_SUBPROGS 256 | 782 | #define BPF_MAX_SUBPROGS 256 |
| 704 | 783 | ||
| 705 | struct bpf_subprog_arg_info { | 784 | struct bpf_subprog_arg_info { |
| @@ -724,6 +803,7 @@ struct bpf_subprog_info { · 18 unmodified lines | |||
| 724 | u32 exit_idx; /* Index of one of the BPF_EXIT instructions in this subprogram */ | 803 | u32 exit_idx; /* Index of one of the BPF_EXIT instructions in this subprogram */ |
| 725 | u16 stack_depth; /* max. stack depth used by this function */ | 804 | u16 stack_depth; /* max. stack depth used by this function */ |
| 726 | u16 stack_extra; | 805 | u16 stack_extra; |
| 806 | u32 insn_processed; | ||
| 727 | /* offsets in range [stack_depth .. fastcall_stack_off) | 807 | /* offsets in range [stack_depth .. fastcall_stack_off) |
| 728 | * are used for bpf_fastcall spills and fills. | 808 | * are used for bpf_fastcall spills and fills. |
| 729 | */ | 809 | */ |
| @@ -740,12 +820,21 @@ struct bpf_subprog_info { · 10 unmodified lines | |||
| 740 | bool keep_fastcall_stack: 1; | 820 | bool keep_fastcall_stack: 1; |
| 741 | bool changes_pkt_data: 1; | 821 | bool changes_pkt_data: 1; |
| 742 | bool might_sleep: 1; | 822 | bool might_sleep: 1; |
| 743 | u8 arg_cnt:3; | 823 | u8 arg_cnt:4; |
| 744 | 824 | ||
| 745 | enum priv_stack_mode priv_stack_mode; | 825 | enum priv_stack_mode priv_stack_mode; |
| 746 | struct bpf_subprog_arg_info args[MAX_BPF_FUNC_REG_ARGS]; | 826 | struct bpf_subprog_arg_info args[MAX_BPF_FUNC_ARGS]; |
| 827 | u16 stack_arg_cnt; /* incoming + max outgoing */ | ||
| 828 | u16 max_out_stack_arg_cnt; | ||
| 747 | }; | 829 | }; |
| 748 | 830 | ||
| 831 | static inline u16 bpf_in_stack_arg_cnt(const struct bpf_subprog_info *sub) | ||
| 832 | { | ||
| 833 | if (sub->arg_cnt > MAX_BPF_FUNC_REG_ARGS) | ||
| 834 | return sub->arg_cnt - MAX_BPF_FUNC_REG_ARGS; | ||
| 835 | return 0; | ||
| 836 | } | ||
| 837 | |||
| 749 | struct bpf_verifier_env; | 838 | struct bpf_verifier_env; |
| 750 | 839 | ||
| 751 | struct backtrack_state { | 840 | struct backtrack_state { |
| @@ -753,6 +842,7 @@ struct backtrack_state { · 1 unmodified line | |||
| 753 | u32 frame; | 842 | u32 frame; |
| 754 | u32 reg_masks[MAX_CALL_FRAMES]; | 843 | u32 reg_masks[MAX_CALL_FRAMES]; |
| 755 | u64 stack_masks[MAX_CALL_FRAMES]; | 844 | u64 stack_masks[MAX_CALL_FRAMES]; |
| 845 | u8 stack_arg_masks[MAX_CALL_FRAMES]; | ||
| 756 | }; | 846 | }; |
| 757 | 847 | ||
| 758 | struct bpf_id_pair { | 848 | struct bpf_id_pair { |
| @@ -881,6 +971,8 @@ struct bpf_verifier_env { · 122 unmodified lines | |||
| 881 | u32 prev_insn_processed, insn_processed; | 971 | u32 prev_insn_processed, insn_processed; |
| 882 | /* number of jmps, calls, exits analyzed so far */ | 972 | /* number of jmps, calls, exits analyzed so far */ |
| 883 | u32 prev_jmps_processed, jmps_processed; | 973 | u32 prev_jmps_processed, jmps_processed; |
| 974 | /* maximum combined stack depth */ | ||
| 975 | u32 max_stack_depth; | ||
| 884 | /* total verification time */ | 976 | /* total verification time */ |
| 885 | u64 verification_time; | 977 | u64 verification_time; |
| 886 | /* maximum number of verifier states kept in 'branching' instructions */ | 978 | /* maximum number of verifier states kept in 'branching' instructions */ |
| @@ -914,6 +1006,7 @@ struct bpf_verifier_env { · 27 unmodified lines | |||
| 914 | * e.g., in reg_type_str() to generate reg_type string | 1006 | * e.g., in reg_type_str() to generate reg_type string |
| 915 | */ | 1007 | */ |
| 916 | char tmp_str_buf[TMP_STR_BUF_LEN]; | 1008 | char tmp_str_buf[TMP_STR_BUF_LEN]; |
| 1009 | char tmp_arg_name[32]; | ||
| 917 | struct bpf_insn insn_buf[INSN_BUF_SIZE]; | 1010 | struct bpf_insn insn_buf[INSN_BUF_SIZE]; |
| 918 | struct bpf_insn epilogue_buf[INSN_BUF_SIZE]; | 1011 | struct bpf_insn epilogue_buf[INSN_BUF_SIZE]; |
| 919 | struct bpf_scc_callchain callchain_buf; | 1012 | struct bpf_scc_callchain callchain_buf; |
| @@ -1087,7 +1180,7 @@ struct list_head *bpf_explored_state(struct bpf_verifier_env *env, int idx); · 167 unmodified lines | |||
| 1087 | void bpf_free_verifier_state(struct bpf_verifier_state *state, bool free_self); | 1180 | void bpf_free_verifier_state(struct bpf_verifier_state *state, bool free_self); |
| 1088 | void bpf_free_backedges(struct bpf_scc_visit *visit); | 1181 | void bpf_free_backedges(struct bpf_scc_visit *visit); |
| 1089 | int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state *cur, | 1182 | int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state *cur, |
| 1090 | int insn_flags, u64 linked_regs); | 1183 | int insn_flags, int spi, int frame, u64 linked_regs); |
| 1091 | void bpf_bt_sync_linked_regs(struct backtrack_state *bt, struct bpf_jmp_history_entry *hist); | 1184 | void bpf_bt_sync_linked_regs(struct backtrack_state *bt, struct bpf_jmp_history_entry *hist); |
| 1092 | void bpf_mark_reg_not_init(const struct bpf_verifier_env *env, | 1185 | void bpf_mark_reg_not_init(const struct bpf_verifier_env *env, |
| 1093 | struct bpf_reg_state *reg); | 1186 | struct bpf_reg_state *reg); |
| @@ -1150,6 +1243,11 @@ static inline void bpf_bt_set_frame_slot(struct backtrack_state *bt, u32 frame, · 56 unmodified lines | |||
| 1150 | bt->stack_masks[frame] |= 1ull << slot; | 1243 | bt->stack_masks[frame] |= 1ull << slot; |
| 1151 | } | 1244 | } |
| 1152 | 1245 | ||
| 1246 | static inline void bt_set_frame_stack_arg_slot(struct backtrack_state *bt, u32 frame, u32 slot) | ||
| 1247 | { | ||
| 1248 | bt->stack_arg_masks[frame] |= 1 << slot; | ||
| 1249 | } | ||
| 1250 | |||
| 1153 | static inline bool bt_is_frame_reg_set(struct backtrack_state *bt, u32 frame, u32 reg) | 1251 | static inline bool bt_is_frame_reg_set(struct backtrack_state *bt, u32 frame, u32 reg) |
| 1154 | { | 1252 | { |
| 1155 | return bt->reg_masks[frame] & (1 << reg); | 1253 | return bt->reg_masks[frame] & (1 << reg); |
| @@ -1321,6 +1419,25 @@ struct bpf_map_desc { · 165 unmodified lines | |||
| 1321 | int uid; | 1419 | int uid; |
| 1322 | }; | 1420 | }; |
| 1323 | 1421 | ||
| 1422 | /* The last initialized dynptr; Populated by process_dynptr_func() */ | ||
| 1423 | struct bpf_dynptr_desc { | ||
| 1424 | enum bpf_dynptr_type type; | ||
| 1425 | u32 id; | ||
| 1426 | u32 parent_id; | ||
| 1427 | }; | ||
| 1428 | |||
| 1429 | /* | ||
| 1430 | * The last seen rereferenced object; Updated by update_ref_obj() when a register refers to a | ||
| 1431 | * referenced object. Used when the helper or kfunc is casting a referenced object, returning | ||
| 1432 | * allocated memory derived from referenced object or creating a dynptr with a referenced | ||
| 1433 | * object as parent. | ||
| 1434 | */ | ||
| 1435 | struct ref_obj_desc { | ||
| 1436 | u32 id; | ||
| 1437 | u32 parent_id; | ||
| 1438 | u8 cnt; | ||
| 1439 | }; | ||
| 1440 | |||
| 1324 | struct bpf_kfunc_call_arg_meta { | 1441 | struct bpf_kfunc_call_arg_meta { |
| 1325 | /* In parameters */ | 1442 | /* In parameters */ |
| 1326 | struct btf *btf; | 1443 | struct btf *btf; |
| @@ -1329,7 +1446,6 @@ struct bpf_kfunc_call_arg_meta { · 2 unmodified lines | |||
| 1329 | const struct btf_type *func_proto; | 1446 | const struct btf_type *func_proto; |
| 1330 | const char *func_name; | 1447 | const char *func_name; |
| 1331 | /* Out parameters */ | 1448 | /* Out parameters */ |
| 1332 | u32 ref_obj_id; | ||
| 1333 | u8 release_regno; | 1449 | u8 release_regno; |
| 1334 | bool r0_rdonly; | 1450 | bool r0_rdonly; |
| 1335 | u32 ret_btf_id; | 1451 | u32 ret_btf_id; |
| @@ -1361,16 +1477,13 @@ struct bpf_kfunc_call_arg_meta { · 25 unmodified lines | |||
| 1361 | struct { | 1477 | struct { |
| 1362 | struct btf_field *field; | 1478 | struct btf_field *field; |
| 1363 | } arg_rbtree_root; | 1479 | } arg_rbtree_root; |
| 1364 | struct { | ||
| 1365 | enum bpf_dynptr_type type; | ||
| 1366 | u32 id; | ||
| 1367 | u32 ref_obj_id; | ||
| 1368 | } initialized_dynptr; | ||
| 1369 | struct { | 1480 | struct { |
| 1370 | u8 spi; | 1481 | u8 spi; |
| 1371 | u8 frameno; | 1482 | u8 frameno; |
| 1372 | } iter; | 1483 | } iter; |
| 1373 | struct bpf_map_desc map; | 1484 | struct bpf_map_desc map; |
| 1485 | struct bpf_dynptr_desc dynptr; | ||
| 1486 | struct ref_obj_desc ref_obj; | ||
| 1374 | u64 mem_size; | 1487 | u64 mem_size; |
| 1375 | }; | 1488 | }; |
| 1376 | 1489 | ||
| @@ -1479,6 +1592,10 @@ int bpf_add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, u16 offset); · 102 unmodified lines | |||
| 1479 | int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, | 1592 | int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, |
| 1480 | struct bpf_insn *insn_buf, int insn_idx, int *cnt); | 1593 | struct bpf_insn *insn_buf, int insn_idx, int *cnt); |
| 1481 | 1594 | ||
| 1595 | /* Functions exported from verifier.c, used by trampoline.c */ | ||
| 1596 | int bpf_check_attach_btf_id_multi(struct btf *btf, struct bpf_prog *prog, u32 btf_id, | ||
| 1597 | struct bpf_attach_target_info *tgt_info); | ||
| 1598 | |||
| 1482 | /* Functions in fixups.c, called from bpf_check() */ | 1599 | /* Functions in fixups.c, called from bpf_check() */ |
| 1483 | int bpf_remove_fastcall_spills_fills(struct bpf_verifier_env *env); | 1600 | int bpf_remove_fastcall_spills_fills(struct bpf_verifier_env *env); |
| 1484 | int bpf_optimize_bpf_loop(struct bpf_verifier_env *env); | 1601 | int bpf_optimize_bpf_loop(struct bpf_verifier_env *env); |
| @@ -145,7 +145,8 @@ const char *btf_get_name(const struct btf *btf); · 144 unmodified lines | |||
| 145 | void btf_get(struct btf *btf); | 145 | void btf_get(struct btf *btf); |
| 146 | void btf_put(struct btf *btf); | 146 | void btf_put(struct btf *btf); |
| 147 | const struct btf_header *btf_header(const struct btf *btf); | 147 | const struct btf_header *btf_header(const struct btf *btf); |
| 148 | int btf_new_fd(const union bpf_attr *attr, bpfptr_t uattr, u32 uattr_sz); | 148 | struct bpf_log_attr; |
| 149 | int btf_new_fd(const union bpf_attr *attr, bpfptr_t uattr, struct bpf_log_attr *attr_log); | ||
| 149 | struct btf *btf_get_by_fd(int fd); | 150 | struct btf *btf_get_by_fd(int fd); |
| 150 | int btf_get_info_by_fd(const struct btf *btf, | 151 | int btf_get_info_by_fd(const struct btf *btf, |
| 151 | const union bpf_attr *attr, | 152 | const union bpf_attr *attr, |
| @@ -415,12 +416,12 @@ static inline bool btf_type_is_array(const struct btf_type *t) · 263 unmodified lines | |||
| 415 | return BTF_INFO_KIND(t->info) == BTF_KIND_ARRAY; | 416 | return BTF_INFO_KIND(t->info) == BTF_KIND_ARRAY; |
| 416 | } | 417 | } |
| 417 | 418 | ||
| 418 | static inline u16 btf_type_vlen(const struct btf_type *t) | 419 | static inline u32 btf_type_vlen(const struct btf_type *t) |
| 419 | { | 420 | { |
| 420 | return BTF_INFO_VLEN(t->info); | 421 | return BTF_INFO_VLEN(t->info); |
| 421 | } | 422 | } |
| 422 | 423 | ||
| 423 | static inline u16 btf_vlen(const struct btf_type *t) | 424 | static inline u32 btf_vlen(const struct btf_type *t) |
| 424 | { | 425 | { |
| 425 | return btf_type_vlen(t); | 426 | return btf_type_vlen(t); |
| 426 | } | 427 | } |
| @@ -284,5 +284,6 @@ extern u32 bpf_cgroup_btf_id[]; · 283 unmodified lines | |||
| 284 | extern u32 bpf_local_storage_map_btf_id[]; | 284 | extern u32 bpf_local_storage_map_btf_id[]; |
| 285 | extern u32 btf_bpf_map_id[]; | 285 | extern u32 btf_bpf_map_id[]; |
| 286 | extern u32 bpf_kmem_cache_btf_id[]; | 286 | extern u32 bpf_kmem_cache_btf_id[]; |
| 287 | extern u32 bpf_multi_func_btf_id[]; | ||
| 287 | 288 | ||
| 288 | #endif | 289 | #endif |
| @@ -0,0 +1,82 @@ | |||
| 1 | /* SPDX-License-Identifier: GPL-2.0-only */ | ||
| 2 | /* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ | ||
| 3 | |||
| 4 | #ifndef _LINUX_CNUM_H | ||
| 5 | #define _LINUX_CNUM_H | ||
| 6 | |||
| 7 | #include <linux/types.h> | ||
| 8 | |||
| 9 | /* | ||
| 10 | * cnum32: a circular number. | ||
| 11 | * A unified representation for signed and unsigned ranges. | ||
| 12 | * | ||
| 13 | * Assume that a 32-bit range is a circle, with 0 being in the 12 o'clock | ||
| 14 | * position, numbers placed sequentially in clockwise order and U32_MAX | ||
| 15 | * in the 11 o'clock position. Signed values map onto the same circle: | ||
| 16 | * S32_MAX sits at 5 o'clock, S32_MIN sits at 6 o'clock (opposite 0), | ||
| 17 | * negative values occupy the left half and positive values the right half. | ||
| 18 | * | ||
| 19 | * @cnum32 represents an arc on this circle drawn clockwise. | ||
| 20 | * @base corresponds to the first value of the range. | ||
| 21 | * @size corresponds to the number of integers in the range excluding @base. | ||
| 22 | * (The @base is excluded to avoid integer overflow when representing the full | ||
| 23 | * 0..U32_MAX range, which corresponds to 2^32, which can't be stored in u32). | ||
| 24 | * | ||
| 25 | * For example: {U32_MAX, 1} corresponds to signed range [-1, 0], | ||
| 26 | * {S32_MAX, 1} corresponds to unsigned range [S32_MAX, S32_MIN]. | ||
| 27 | */ | ||
| 28 | struct cnum32 { | ||
| 29 | u32 base; | ||
| 30 | u32 size; | ||
| 31 | }; | ||
| 32 | |||
| 33 | #define CNUM32_UNBOUNDED ((struct cnum32){ .base = 0, .size = U32_MAX }) | ||
| 34 | #define CNUM32_EMPTY ((struct cnum32){ .base = U32_MAX, .size = U32_MAX }) | ||
| 35 | |||
| 36 | struct cnum32 cnum32_from_urange(u32 min, u32 max); | ||
| 37 | struct cnum32 cnum32_from_srange(s32 min, s32 max); | ||
| 38 | u32 cnum32_umin(struct cnum32 cnum); | ||
| 39 | u32 cnum32_umax(struct cnum32 cnum); | ||
| 40 | s32 cnum32_smin(struct cnum32 cnum); | ||
| 41 | s32 cnum32_smax(struct cnum32 cnum); | ||
| 42 | struct cnum32 cnum32_intersect(struct cnum32 a, struct cnum32 b); | ||
| 43 | void cnum32_intersect_with(struct cnum32 *dst, struct cnum32 src); | ||
| 44 | void cnum32_intersect_with_urange(struct cnum32 *dst, u32 min, u32 max); | ||
| 45 | void cnum32_intersect_with_srange(struct cnum32 *dst, s32 min, s32 max); | ||
| 46 | bool cnum32_contains(struct cnum32 cnum, u32 v); | ||
| 47 | bool cnum32_is_const(struct cnum32 cnum); | ||
| 48 | bool cnum32_is_empty(struct cnum32 cnum); | ||
| 49 | struct cnum32 cnum32_add(struct cnum32 a, struct cnum32 b); | ||
| 50 | struct cnum32 cnum32_negate(struct cnum32 a); | ||
| 51 | bool cnum32_is_subset(struct cnum32 outer, struct cnum32 inner); | ||
| 52 | |||
| 53 | /* Same as cnum32 but for 64-bit ranges */ | ||
| 54 | struct cnum64 { | ||
| 55 | u64 base; | ||
| 56 | u64 size; | ||
| 57 | }; | ||
| 58 | |||
| 59 | #define CNUM64_UNBOUNDED ((struct cnum64){ .base = 0, .size = U64_MAX }) | ||
| 60 | #define CNUM64_EMPTY ((struct cnum64){ .base = U64_MAX, .size = U64_MAX }) | ||
| 61 | |||
| 62 | struct cnum64 cnum64_from_urange(u64 min, u64 max); | ||
| 63 | struct cnum64 cnum64_from_srange(s64 min, s64 max); | ||
| 64 | u64 cnum64_umin(struct cnum64 cnum); | ||
| 65 | u64 cnum64_umax(struct cnum64 cnum); | ||
| 66 | s64 cnum64_smin(struct cnum64 cnum); | ||
| 67 | s64 cnum64_smax(struct cnum64 cnum); | ||
| 68 | struct cnum64 cnum64_intersect(struct cnum64 a, struct cnum64 b); | ||
| 69 | void cnum64_intersect_with(struct cnum64 *dst, struct cnum64 src); | ||
| 70 | void cnum64_intersect_with_urange(struct cnum64 *dst, u64 min, u64 max); | ||
| 71 | void cnum64_intersect_with_srange(struct cnum64 *dst, s64 min, s64 max); | ||
| 72 | bool cnum64_contains(struct cnum64 cnum, u64 v); | ||
| 73 | bool cnum64_is_const(struct cnum64 cnum); | ||
| 74 | bool cnum64_is_empty(struct cnum64 cnum); | ||
| 75 | struct cnum64 cnum64_add(struct cnum64 a, struct cnum64 b); | ||
| 76 | struct cnum64 cnum64_negate(struct cnum64 a); | ||
| 77 | bool cnum64_is_subset(struct cnum64 outer, struct cnum64 inner); | ||
| 78 | |||
| 79 | struct cnum32 cnum32_from_cnum64(struct cnum64 cnum); | ||
| 80 | struct cnum64 cnum64_cnum32_intersect(struct cnum64 a, struct cnum32 b); | ||
| 81 | |||
| 82 | #endif /* _LINUX_CNUM_H */ | ||
| @@ -58,8 +58,9 @@ struct ctl_table_header; · 57 unmodified lines | |||
| 58 | #define BPF_REG_H BPF_REG_9 /* hlen, callee-saved */ | 58 | #define BPF_REG_H BPF_REG_9 /* hlen, callee-saved */ |
| 59 | 59 | ||
| 60 | /* Kernel hidden auxiliary/helper register. */ | 60 | /* Kernel hidden auxiliary/helper register. */ |
| 61 | #define BPF_REG_AX MAX_BPF_REG | 61 | #define BPF_REG_PARAMS MAX_BPF_REG |
| 62 | #define MAX_BPF_EXT_REG (MAX_BPF_REG + 1) | 62 | #define BPF_REG_AX (MAX_BPF_REG + 1) |
| 63 | #define MAX_BPF_EXT_REG (MAX_BPF_REG + 2) | ||
| 63 | #define MAX_BPF_JIT_REG MAX_BPF_EXT_REG | 64 | #define MAX_BPF_JIT_REG MAX_BPF_EXT_REG |
| 64 | 65 | ||
| 65 | /* unused opcode to mark special call to bpf_tail_call() helper */ | 66 | /* unused opcode to mark special call to bpf_tail_call() helper */ |
| @@ -748,6 +749,27 @@ static inline u32 bpf_prog_run_pin_on_cpu(const struct bpf_prog *prog, · 682 unmodified lines | |||
| 748 | return ret; | 749 | return ret; |
| 749 | } | 750 | } |
| 750 | 751 | ||
| 752 | static inline bool is_stack_arg_ldx(const struct bpf_insn *insn) | ||
| 753 | { | ||
| 754 | return insn->code == (BPF_LDX | BPF_MEM | BPF_DW) && | ||
| 755 | insn->src_reg == BPF_REG_PARAMS && | ||
| 756 | insn->off > 0 && insn->off % 8 == 0; | ||
| 757 | } | ||
| 758 | |||
| 759 | static inline bool is_stack_arg_st(const struct bpf_insn *insn) | ||
| 760 | { | ||
| 761 | return insn->code == (BPF_ST | BPF_MEM | BPF_DW) && | ||
| 762 | insn->dst_reg == BPF_REG_PARAMS && | ||
| 763 | insn->off < 0 && insn->off % 8 == 0; | ||
| 764 | } | ||
| 765 | |||
| 766 | static inline bool is_stack_arg_stx(const struct bpf_insn *insn) | ||
| 767 | { | ||
| 768 | return insn->code == (BPF_STX | BPF_MEM | BPF_DW) && | ||
| 769 | insn->dst_reg == BPF_REG_PARAMS && | ||
| 770 | insn->off < 0 && insn->off % 8 == 0; | ||
| 771 | } | ||
| 772 | |||
| 751 | #define BPF_SKB_CB_LEN QDISC_CB_PRIV_LEN | 773 | #define BPF_SKB_CB_LEN QDISC_CB_PRIV_LEN |
| 752 | 774 | ||
| 753 | struct bpf_skb_data_end { | 775 | struct bpf_skb_data_end { |
| @@ -1159,6 +1181,7 @@ bool bpf_jit_inlines_helper_call(s32 imm); · 405 unmodified lines | |||
| 1159 | bool bpf_jit_supports_subprog_tailcalls(void); | 1181 | bool bpf_jit_supports_subprog_tailcalls(void); |
| 1160 | bool bpf_jit_supports_percpu_insn(void); | 1182 | bool bpf_jit_supports_percpu_insn(void); |
| 1161 | bool bpf_jit_supports_kfunc_call(void); | 1183 | bool bpf_jit_supports_kfunc_call(void); |
| 1184 | bool bpf_jit_supports_stack_args(void); | ||
| 1162 | bool bpf_jit_supports_far_kfunc_call(void); | 1185 | bool bpf_jit_supports_far_kfunc_call(void); |
| 1163 | bool bpf_jit_supports_exceptions(void); | 1186 | bool bpf_jit_supports_exceptions(void); |
| 1164 | bool bpf_jit_supports_ptr_xchg(void); | 1187 | bool bpf_jit_supports_ptr_xchg(void); |
| @@ -415,6 +415,8 @@ struct ftrace_hash *alloc_ftrace_hash(int size_bits); · 414 unmodified lines | |||
| 415 | void free_ftrace_hash(struct ftrace_hash *hash); | 415 | void free_ftrace_hash(struct ftrace_hash *hash); |
| 416 | struct ftrace_func_entry *add_ftrace_hash_entry_direct(struct ftrace_hash *hash, | 416 | struct ftrace_func_entry *add_ftrace_hash_entry_direct(struct ftrace_hash *hash, |
| 417 | unsigned long ip, unsigned long direct); | 417 | unsigned long ip, unsigned long direct); |
| 418 | void add_ftrace_hash_entry(struct ftrace_hash *hash, struct ftrace_func_entry *entry); | ||
| 419 | void ftrace_hash_remove(struct ftrace_hash *hash); | ||
| 418 | 420 | ||
| 419 | /* The hash used to know what functions callbacks trace */ | 421 | /* The hash used to know what functions callbacks trace */ |
| 420 | struct ftrace_ops_hash { | 422 | struct ftrace_ops_hash { |
| @@ -551,6 +553,8 @@ int update_ftrace_direct_mod(struct ftrace_ops *ops, struct ftrace_hash *hash, b · 130 unmodified lines | |||
| 551 | 553 | ||
| 552 | void ftrace_stub_direct_tramp(void); | 554 | void ftrace_stub_direct_tramp(void); |
| 553 | 555 | ||
| 556 | unsigned long ftrace_hash_count(struct ftrace_hash *hash); | ||
| 557 | |||
| 554 | #else | 558 | #else |
| 555 | struct ftrace_ops; | 559 | struct ftrace_ops; |
| 556 | static inline unsigned long ftrace_find_rec_direct(unsigned long ip) | 560 | static inline unsigned long ftrace_find_rec_direct(unsigned long ip) |
| @@ -590,6 +594,11 @@ static inline int update_ftrace_direct_mod(struct ftrace_ops *ops, struct ftrace · 33 unmodified lines | |||
| 590 | return -ENODEV; | 594 | return -ENODEV; |
| 591 | } | 595 | } |
| 592 | 596 | ||
| 597 | static inline unsigned long ftrace_hash_count(struct ftrace_hash *hash) | ||
| 598 | { | ||
| 599 | return 0; | ||
| 600 | } | ||
| 601 | |||
| 593 | /* | 602 | /* |
| 594 | * This must be implemented by the architecture. | 603 | * This must be implemented by the architecture. |
| 595 | * It is the way the ftrace direct_ops helper, when called | 604 | * It is the way the ftrace direct_ops helper, when called |
| @@ -1070,6 +1070,49 @@ static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long addres · 1069 unmodified lines | |||
| 1070 | } | 1070 | } |
| 1071 | #endif | 1071 | #endif |
| 1072 | 1072 | ||
| 1073 | #ifndef ptep_try_set | ||
| 1074 | /** | ||
| 1075 | * ptep_try_set - atomically set an empty kernel PTE | ||
| 1076 | * @ptep: page table entry | ||
| 1077 | * @new_pte: value to install | ||
| 1078 | * | ||
| 1079 | * Atomically set *@ptep to @new_pte iff *@ptep is pte_none(). Return true on | ||
| 1080 | * success, false if the slot was already populated or the arch has no | ||
| 1081 | * implementation. | ||
| 1082 | * | ||
| 1083 | * For special kernel page tables only - never user page tables. The caller must | ||
| 1084 | * prevent concurrent teardown of @ptep and must accept that other writers may | ||
| 1085 | * race. Concurrent clearers must use ptep_get_and_clear() so racing accesses | ||
| 1086 | * agree on the outcome. | ||
| 1087 | * | ||
| 1088 | * Architectures opt in by providing a cmpxchg-based override and defining | ||
| 1089 | * ptep_try_set as an identity macro. The generic stub returns false, which is | ||
| 1090 | * correct for callers that fall through to oops on failure. | ||
| 1091 | */ | ||
| 1092 | static inline bool ptep_try_set(pte_t *ptep, pte_t new_pte) | ||
| 1093 | { | ||
| 1094 | return false; | ||
| 1095 | } | ||
| 1096 | #endif | ||
| 1097 | |||
| 1098 | #ifndef flush_tlb_before_set | ||
| 1099 | /** | ||
| 1100 | * flush_tlb_before_set - invalidate a kernel PTE's TLB before re-setting it | ||
| 1101 | * @addr: kernel virtual address whose PTE was just cleared | ||
| 1102 | * | ||
| 1103 | * Some architectures (e.g. arm64) do not allow a live page-table entry to be | ||
| 1104 | * repointed at a different page in one step. The old entry must first be made | ||
| 1105 | * invalid and its translation flushed from every TLB, and only then may the new | ||
| 1106 | * entry be written. | ||
| 1107 | * | ||
| 1108 | * This is only for the lockless atomic kernel-PTE installers (ptep_try_set()). | ||
| 1109 | * It must be callable with interrupts disabled. | ||
| 1110 | */ | ||
| 1111 | static inline void flush_tlb_before_set(unsigned long addr) | ||
| 1112 | { | ||
| 1113 | } | ||
| 1114 | #endif | ||
| 1115 | |||
| 1073 | #ifndef wrprotect_ptes | 1116 | #ifndef wrprotect_ptes |
| 1074 | /** | 1117 | /** |
| 1075 | * wrprotect_ptes - Write-protect PTEs that map consecutive pages of the same | 1118 | * wrprotect_ptes - Write-protect PTEs that map consecutive pages of the same |
| @@ -263,6 +263,8 @@ struct rhash_lock_head __rcu **__rht_bucket_nested( · 262 unmodified lines | |||
| 263 | struct rhash_lock_head __rcu **rht_bucket_nested_insert( | 263 | struct rhash_lock_head __rcu **rht_bucket_nested_insert( |
| 264 | struct rhashtable *ht, struct bucket_table *tbl, unsigned int hash); | 264 | struct rhashtable *ht, struct bucket_table *tbl, unsigned int hash); |
| 265 | 265 | ||
| 266 | void *rhashtable_next_key(struct rhashtable *ht, const void *prev_key); | ||
| 267 | |||
| 266 | #define rht_dereference(p, ht) \ | 268 | #define rht_dereference(p, ht) \ |
| 267 | rcu_dereference_protected(p, lockdep_rht_mutex_is_held(ht)) | 269 | rcu_dereference_protected(p, lockdep_rht_mutex_is_held(ht)) |
| 268 | 270 | ||
| @@ -1117,7 +1119,7 @@ unlocked: · 848 unmodified lines | |||
| 1117 | atomic_dec(&ht->nelems); | 1119 | atomic_dec(&ht->nelems); |
| 1118 | if (unlikely(ht->p.automatic_shrinking && | 1120 | if (unlikely(ht->p.automatic_shrinking && |
| 1119 | rht_shrink_below_30(ht, tbl))) | 1121 | rht_shrink_below_30(ht, tbl))) |
| 1120 | schedule_work(&ht->run_work); | 1122 | irq_work_queue(&ht->run_irq_work); |
| 1121 | err = 0; | 1123 | err = 0; |
| 1122 | } | 1124 | } |
| 1123 | 1125 | ||
| @@ -940,7 +940,8 @@ asmlinkage long sys_seccomp(unsigned int op, unsigned int flags, · 939 unmodified lines | |||
| 940 | asmlinkage long sys_getrandom(char __user *buf, size_t count, | 940 | asmlinkage long sys_getrandom(char __user *buf, size_t count, |
| 941 | unsigned int flags); | 941 | unsigned int flags); |
| 942 | asmlinkage long sys_memfd_create(const char __user *uname_ptr, unsigned int flags); | 942 | asmlinkage long sys_memfd_create(const char __user *uname_ptr, unsigned int flags); |
| 943 | asmlinkage long sys_bpf(int cmd, union bpf_attr __user *attr, unsigned int size); | 943 | asmlinkage long sys_bpf(int cmd, union bpf_attr __user *attr, unsigned int size, |
| 944 | struct bpf_common_attr __user *attr_common, unsigned int size_common); | ||
| 944 | asmlinkage long sys_execveat(int dfd, const char __user *filename, | 945 | asmlinkage long sys_execveat(int dfd, const char __user *filename, |
| 945 | const char __user *const __user *argv, | 946 | const char __user *const __user *argv, |
| 946 | const char __user *const __user *envp, int flags); | 947 | const char __user *const __user *envp, int flags); |
| @@ -770,6 +770,7 @@ trace_trigger_soft_disabled(struct trace_event_file *file) · 769 unmodified lines | |||
| 770 | 770 | ||
| 771 | #ifdef CONFIG_BPF_EVENTS | 771 | #ifdef CONFIG_BPF_EVENTS |
| 772 | unsigned int trace_call_bpf(struct trace_event_call *call, void *ctx); | 772 | unsigned int trace_call_bpf(struct trace_event_call *call, void *ctx); |
| 773 | unsigned int trace_call_bpf_faultable(struct trace_event_call *call, void *ctx); | ||
| 773 | int perf_event_attach_bpf_prog(struct perf_event *event, struct bpf_prog *prog, u64 bpf_cookie); | 774 | int perf_event_attach_bpf_prog(struct perf_event *event, struct bpf_prog *prog, u64 bpf_cookie); |
| 774 | void perf_event_detach_bpf_prog(struct perf_event *event); | 775 | void perf_event_detach_bpf_prog(struct perf_event *event); |
| 775 | int perf_event_query_prog_array(struct perf_event *event, void __user *info); | 776 | int perf_event_query_prog_array(struct perf_event *event, void __user *info); |
| @@ -786,12 +787,18 @@ int bpf_get_perf_event_info(const struct perf_event *event, u32 *prog_id, · 10 unmodified lines | |||
| 786 | unsigned long *missed); | 787 | unsigned long *missed); |
| 787 | int bpf_kprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog); | 788 | int bpf_kprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog); |
| 788 | int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog); | 789 | int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog); |
| 790 | int bpf_tracing_multi_attach(struct bpf_prog *prog, const union bpf_attr *attr); | ||
| 789 | #else | 791 | #else |
| 790 | static inline unsigned int trace_call_bpf(struct trace_event_call *call, void *ctx) | 792 | static inline unsigned int trace_call_bpf(struct trace_event_call *call, void *ctx) |
| 791 | { | 793 | { |
| 792 | return 1; | 794 | return 1; |
| 793 | } | 795 | } |
| 794 | 796 | ||
| 797 | static inline unsigned int trace_call_bpf_faultable(struct trace_event_call *call, void *ctx) | ||
| 798 | { | ||
| 799 | return 1; | ||
| 800 | } | ||
| 801 | |||
| 795 | static inline int | 802 | static inline int |
| 796 | perf_event_attach_bpf_prog(struct perf_event *event, struct bpf_prog *prog, u64 bpf_cookie) | 803 | perf_event_attach_bpf_prog(struct perf_event *event, struct bpf_prog *prog, u64 bpf_cookie) |
| 797 | { | 804 | { |
| @@ -838,6 +845,11 @@ bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog) · 40 unmodified lines | |||
| 838 | { | 845 | { |
| 839 | return -EOPNOTSUPP; | 846 | return -EOPNOTSUPP; |
| 840 | } | 847 | } |
| 848 | static inline int | ||
| 849 | bpf_tracing_multi_attach(struct bpf_prog *prog, const union bpf_attr *attr) | ||
| 850 | { | ||
| 851 | return -EOPNOTSUPP; | ||
| 852 | } | ||
| 841 | #endif | 853 | #endif |
| 842 | 854 | ||
| 843 | enum { | 855 | enum { |
| @@ -58,9 +58,7 @@ static notrace void \ · 57 unmodified lines | |||
| 58 | __bpf_trace_##call(void *__data, proto) \ | 58 | __bpf_trace_##call(void *__data, proto) \ |
| 59 | { \ | 59 | { \ |
| 60 | might_fault(); \ | 60 | might_fault(); \ |
| 61 | preempt_disable_notrace(); \ | ||
| 62 | CONCATENATE(bpf_trace_run, COUNT_ARGS(args))(__data, CAST_TO_U64(args)); \ | 61 | CONCATENATE(bpf_trace_run, COUNT_ARGS(args))(__data, CAST_TO_U64(args)); \ |
| 63 | preempt_enable_notrace(); \ | ||
| 64 | } | 62 | } |
| 65 | 63 | ||
| 66 | #undef DECLARE_EVENT_SYSCALL_CLASS | 64 | #undef DECLARE_EVENT_SYSCALL_CLASS |
| @@ -994,6 +994,7 @@ enum bpf_cmd { · 993 unmodified lines | |||
| 994 | BPF_PROG_STREAM_READ_BY_FD, | 994 | BPF_PROG_STREAM_READ_BY_FD, |
| 995 | BPF_PROG_ASSOC_STRUCT_OPS, | 995 | BPF_PROG_ASSOC_STRUCT_OPS, |
| 996 | __MAX_BPF_CMD, | 996 | __MAX_BPF_CMD, |
| 997 | BPF_COMMON_ATTRS = 1 << 16, /* Indicate carrying syscall common attrs. */ | ||
| 997 | }; | 998 | }; |
| 998 | 999 | ||
| 999 | enum bpf_map_type { | 1000 | enum bpf_map_type { |
| @@ -1046,6 +1047,7 @@ enum bpf_map_type { · 46 unmodified lines | |||
| 1046 | BPF_MAP_TYPE_CGRP_STORAGE, | 1047 | BPF_MAP_TYPE_CGRP_STORAGE, |
| 1047 | BPF_MAP_TYPE_ARENA, | 1048 | BPF_MAP_TYPE_ARENA, |
| 1048 | BPF_MAP_TYPE_INSN_ARRAY, | 1049 | BPF_MAP_TYPE_INSN_ARRAY, |
| 1050 | BPF_MAP_TYPE_RHASH, | ||
| 1049 | __MAX_BPF_MAP_TYPE | 1051 | __MAX_BPF_MAP_TYPE |
| 1050 | }; | 1052 | }; |
| 1051 | 1053 | ||
| @@ -1154,6 +1156,9 @@ enum bpf_attach_type { · 102 unmodified lines | |||
| 1154 | BPF_TRACE_KPROBE_SESSION, | 1156 | BPF_TRACE_KPROBE_SESSION, |
| 1155 | BPF_TRACE_UPROBE_SESSION, | 1157 | BPF_TRACE_UPROBE_SESSION, |
| 1156 | BPF_TRACE_FSESSION, | 1158 | BPF_TRACE_FSESSION, |
| 1159 | BPF_TRACE_FENTRY_MULTI, | ||
| 1160 | BPF_TRACE_FEXIT_MULTI, | ||
| 1161 | BPF_TRACE_FSESSION_MULTI, | ||
| 1157 | __MAX_BPF_ATTACH_TYPE | 1162 | __MAX_BPF_ATTACH_TYPE |
| 1158 | }; | 1163 | }; |
| 1159 | 1164 | ||
| @@ -1178,6 +1183,7 @@ enum bpf_link_type { · 18 unmodified lines | |||
| 1178 | BPF_LINK_TYPE_UPROBE_MULTI = 12, | 1183 | BPF_LINK_TYPE_UPROBE_MULTI = 12, |
| 1179 | BPF_LINK_TYPE_NETKIT = 13, | 1184 | BPF_LINK_TYPE_NETKIT = 13, |
| 1180 | BPF_LINK_TYPE_SOCKMAP = 14, | 1185 | BPF_LINK_TYPE_SOCKMAP = 14, |
| 1186 | BPF_LINK_TYPE_TRACING_MULTI = 15, | ||
| 1181 | __MAX_BPF_LINK_TYPE, | 1187 | __MAX_BPF_LINK_TYPE, |
| 1182 | }; | 1188 | }; |
| 1183 | 1189 | ||
| @@ -1321,7 +1327,11 @@ enum { · 137 unmodified lines | |||
| 1321 | * BPF_TRACE_UPROBE_MULTI attach type to create return probe. | 1327 | * BPF_TRACE_UPROBE_MULTI attach type to create return probe. |
| 1322 | */ | 1328 | */ |
| 1323 | enum { | 1329 | enum { |
| 1324 | BPF_F_UPROBE_MULTI_RETURN = (1U << 0) | 1330 | /* Get return uprobe. */ |
| 1331 | BPF_F_UPROBE_MULTI_RETURN = (1U << 0), | ||
| 1332 | |||
| 1333 | /* Get path from provided path_fd. */ | ||
| 1334 | BPF_F_UPROBE_MULTI_PATH_FD = (1U << 1), | ||
| 1325 | }; | 1335 | }; |
| 1326 | 1336 | ||
| 1327 | /* link_create.netfilter.flags used in LINK_CREATE command for | 1337 | /* link_create.netfilter.flags used in LINK_CREATE command for |
| @@ -1500,6 +1510,13 @@ struct bpf_stack_build_id { · 172 unmodified lines | |||
| 1500 | }; | 1510 | }; |
| 1501 | }; | 1511 | }; |
| 1502 | 1512 | ||
| 1513 | struct bpf_common_attr { | ||
| 1514 | __aligned_u64 log_buf; | ||
| 1515 | __u32 log_size; | ||
| 1516 | __u32 log_level; | ||
| 1517 | __u32 log_true_size; | ||
| 1518 | }; | ||
| 1519 | |||
| 1503 | #define BPF_OBJ_NAME_LEN 16U | 1520 | #define BPF_OBJ_NAME_LEN 16U |
| 1504 | 1521 | ||
| 1505 | enum { | 1522 | enum { |
| @@ -1537,6 +1554,11 @@ union bpf_attr { · 31 unmodified lines | |||
| 1537 | * | 1554 | * |
| 1538 | * BPF_MAP_TYPE_ARENA - contains the address where user space | 1555 | * BPF_MAP_TYPE_ARENA - contains the address where user space |
| 1539 | * is going to mmap() the arena. It has to be page aligned. | 1556 | * is going to mmap() the arena. It has to be page aligned. |
| 1557 | * | ||
| 1558 | * BPF_MAP_TYPE_RHASH - initial table size hint | ||
| 1559 | * (nelem_hint). 0 = use rhashtable default. Must be | ||
| 1560 | * <= min(max_entries, U16_MAX). Upper 32 bits reserved, | ||
| 1561 | * must be zero. | ||
| 1540 | */ | 1562 | */ |
| 1541 | __u64 map_extra; | 1563 | __u64 map_extra; |
| 1542 | 1564 | ||
| @@ -1846,6 +1868,7 @@ union bpf_attr { · 303 unmodified lines | |||
| 1846 | __u32 cnt; | 1868 | __u32 cnt; |
| 1847 | __u32 flags; | 1869 | __u32 flags; |
| 1848 | __u32 pid; | 1870 | __u32 pid; |
| 1871 | __u32 path_fd; | ||
| 1849 | } uprobe_multi; | 1872 | } uprobe_multi; |
| 1850 | struct { | 1873 | struct { |
| 1851 | union { | 1874 | union { |
| @@ -1861,6 +1884,11 @@ union bpf_attr { · 9 unmodified lines | |||
| 1861 | }; | 1884 | }; |
| 1862 | __u64 expected_revision; | 1885 | __u64 expected_revision; |
| 1863 | } cgroup; | 1886 | } cgroup; |
| 1887 | struct { | ||
| 1888 | __aligned_u64 ids; | ||
| 1889 | __aligned_u64 cookies; | ||
| 1890 | __u32 cnt; | ||
| 1891 | } tracing_multi; | ||
| 1864 | }; | 1892 | }; |
| 1865 | } link_create; | 1893 | } link_create; |
| 1866 | 1894 | ||
| @@ -6698,6 +6726,7 @@ struct bpf_prog_info { · 4831 unmodified lines | |||
| 6698 | __u32 verified_insns; | 6726 | __u32 verified_insns; |
| 6699 | __u32 attach_btf_obj_id; | 6727 | __u32 attach_btf_obj_id; |
| 6700 | __u32 attach_btf_id; | 6728 | __u32 attach_btf_id; |
| 6729 | __u32 :32; | ||
| 6701 | } __attribute__((aligned(8))); | 6730 | } __attribute__((aligned(8))); |
| 6702 | 6731 | ||
| 6703 | struct bpf_map_info { | 6732 | struct bpf_map_info { |
| @@ -6719,6 +6748,7 @@ struct bpf_map_info { · 15 unmodified lines | |||
| 6719 | __u64 map_extra; | 6748 | __u64 map_extra; |
| 6720 | __aligned_u64 hash; | 6749 | __aligned_u64 hash; |
| 6721 | __u32 hash_size; | 6750 | __u32 hash_size; |
| 6751 | __u32 :32; | ||
| 6722 | } __attribute__((aligned(8))); | 6752 | } __attribute__((aligned(8))); |
| 6723 | 6753 | ||
| 6724 | struct bpf_btf_info { | 6754 | struct bpf_btf_info { |
| @@ -33,20 +33,22 @@ struct btf_header { · 32 unmodified lines | |||
| 33 | __u32 layout_len; /* length of layout section */ | 33 | __u32 layout_len; /* length of layout section */ |
| 34 | }; | 34 | }; |
| 35 | 35 | ||
| 36 | /* Max # of type identifier */ | 36 | enum btf_max { |
| 37 | #define BTF_MAX_TYPE 0x000fffff | 37 | /* Max possible kind */ |
| 38 | /* Max offset into the string section */ | 38 | BTF_MAX_KIND = 0x0000007f, |
| 39 | #define BTF_MAX_NAME_OFFSET 0x00ffffff | 39 | /* Max # of type identifier */ |
| 40 | /* Max # of struct/union/enum members or func args */ | 40 | BTF_MAX_TYPE = 0x000fffff, |
| 41 | #define BTF_MAX_VLEN 0xffff | 41 | /* Max offset into the string section */ |
| 42 | BTF_MAX_NAME_OFFSET = 0x00ffffff, | ||
| 43 | /* Max # of struct/union/enum members or func args */ | ||
| 44 | BTF_MAX_VLEN = 0x00ffffff, | ||
| 45 | }; | ||
| 42 | 46 | ||
| 43 | struct btf_type { | 47 | struct btf_type { |
| 44 | __u32 name_off; | 48 | __u32 name_off; |
| 45 | /* "info" bits arrangement | 49 | /* "info" bits arrangement |
| 46 | * bits 0-15: vlen (e.g. # of struct's members) | 50 | * bits 0-23: vlen (e.g. # of struct's members) |
| 47 | * bits 16-23: unused | 51 | * bits 24-30: kind (e.g. int, ptr, array...etc) |
| 48 | * bits 24-28: kind (e.g. int, ptr, array...etc) | ||
| 49 | * bits 29-30: unused | ||
| 50 | * bit 31: kind_flag, currently used by | 52 | * bit 31: kind_flag, currently used by |
| 51 | * struct, union, enum, fwd, enum64, | 53 | * struct, union, enum, fwd, enum64, |
| 52 | * decl_tag and type_tag | 54 | * decl_tag and type_tag |
| @@ -65,8 +67,8 @@ struct btf_type { · 12 unmodified lines | |||
| 65 | }; | 67 | }; |
| 66 | }; | 68 | }; |
| 67 | 69 | ||
| 68 | #define BTF_INFO_KIND(info) (((info) >> 24) & 0x1f) | 70 | #define BTF_INFO_KIND(info) (((info) >> 24) & 0x7f) |
| 69 | #define BTF_INFO_VLEN(info) ((info) & 0xffff) | 71 | #define BTF_INFO_VLEN(info) ((info) & 0xffffff) |
| 70 | #define BTF_INFO_KFLAG(info) ((info) >> 31) | 72 | #define BTF_INFO_KFLAG(info) ((info) >> 31) |
| 71 | 73 | ||
| 72 | enum { | 74 | enum { |
| @@ -6,7 +6,7 @@ cflags-nogcse-$(CONFIG_X86)$(CONFIG_CC_IS_GCC) := -fno-gcse · 5 unmodified lines | |||
| 6 | endif | 6 | endif |
| 7 | CFLAGS_core.o += -Wno-override-init $(cflags-nogcse-yy) | 7 | CFLAGS_core.o += -Wno-override-init $(cflags-nogcse-yy) |
| 8 | 8 | ||
| 9 | obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o tnum.o log.o token.o liveness.o const_fold.o | 9 | obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o tnum.o cnum.o log.o token.o liveness.o const_fold.o |
| 10 | obj-$(CONFIG_BPF_SYSCALL) += bpf_iter.o map_iter.o task_iter.o prog_iter.o link_iter.o | 10 | obj-$(CONFIG_BPF_SYSCALL) += bpf_iter.o map_iter.o task_iter.o prog_iter.o link_iter.o |
| 11 | obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o bloom_filter.o | 11 | obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o bloom_filter.o |
| 12 | obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o bpf_insn_array.o | 12 | obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o bpf_insn_array.o |
| @@ -53,12 +53,15 @@ struct bpf_arena { · 52 unmodified lines | |||
| 53 | u64 user_vm_start; | 53 | u64 user_vm_start; |
| 54 | u64 user_vm_end; | 54 | u64 user_vm_end; |
| 55 | struct vm_struct *kern_vm; | 55 | struct vm_struct *kern_vm; |
| 56 | struct page *scratch_page; | ||
| 56 | struct range_tree rt; | 57 | struct range_tree rt; |
| 57 | /* protects rt */ | 58 | /* protects rt */ |
| 58 | rqspinlock_t spinlock; | 59 | rqspinlock_t spinlock; |
| 59 | struct list_head vma_list; | 60 | struct list_head vma_list; |
| 60 | /* protects vma_list */ | 61 | /* protects vma_list */ |
| 61 | struct mutex lock; | 62 | struct mutex lock; |
| 63 | u64 zap_gen; | ||
| 64 | struct mutex zap_mutex; | ||
| 62 | struct irq_work free_irq; | 65 | struct irq_work free_irq; |
| 63 | struct work_struct free_work; | 66 | struct work_struct free_work; |
| 64 | struct llist_head free_spans; | 67 | struct llist_head free_spans; |
| @@ -83,6 +86,32 @@ u64 bpf_arena_get_user_vm_start(struct bpf_arena *arena) · 18 unmodified lines | |||
| 83 | return arena ? arena->user_vm_start : 0; | 86 | return arena ? arena->user_vm_start : 0; |
| 84 | } | 87 | } |
| 85 | 88 | ||
| 89 | /** | ||
| 90 | * bpf_arena_map_kern_vm_start - kern_vm_start lookup by struct bpf_map * | ||
| 91 | * @map: a BPF_MAP_TYPE_ARENA map | ||
| 92 | * | ||
| 93 | * Return @map's kern_vm_start. | ||
| 94 | */ | ||
| 95 | u64 bpf_arena_map_kern_vm_start(struct bpf_map *map) | ||
| 96 | { | ||
| 97 | return bpf_arena_get_kern_vm_start(container_of(map, struct bpf_arena, map)); | ||
| 98 | } | ||
| 99 | |||
| 100 | /** | ||
| 101 | * bpf_prog_arena - return the bpf_map of the arena referenced by @prog | ||
| 102 | * @prog: a loaded BPF program | ||
| 103 | * | ||
| 104 | * The verifier enforces at most one arena per program and stores it in | ||
| 105 | * prog->aux->arena. Return that arena's underlying bpf_map, or NULL if | ||
| 106 | * @prog does not reference an arena. | ||
| 107 | */ | ||
| 108 | struct bpf_map *bpf_prog_arena(struct bpf_prog *prog) | ||
| 109 | { | ||
| 110 | struct bpf_arena *arena = prog->aux->arena; | ||
| 111 | |||
| 112 | return arena ? &arena->map : NULL; | ||
| 113 | } | ||
| 114 | |||
| 86 | static long arena_map_peek_elem(struct bpf_map *map, void *value) | 115 | static long arena_map_peek_elem(struct bpf_map *map, void *value) |
| 87 | { | 116 | { |
| 88 | return -EOPNOTSUPP; | 117 | return -EOPNOTSUPP; |
| @@ -115,26 +144,57 @@ static long compute_pgoff(struct bpf_arena *arena, long uaddr) · 26 unmodified lines | |||
| 115 | 144 | ||
| 116 | struct apply_range_data { | 145 | struct apply_range_data { |
| 117 | struct page **pages; | 146 | struct page **pages; |
| 147 | struct page *scratch_page; | ||
| 118 | int i; | 148 | int i; |
| 119 | }; | 149 | }; |
| 120 | 150 | ||
| 151 | struct clear_range_data { | ||
| 152 | struct llist_head *free_pages; | ||
| 153 | struct page *scratch_page; | ||
| 154 | }; | ||
| 155 | |||
| 121 | static int apply_range_set_cb(pte_t *pte, unsigned long addr, void *data) | 156 | static int apply_range_set_cb(pte_t *pte, unsigned long addr, void *data) |
| 122 | { | 157 | { |
| 123 | struct apply_range_data *d = data; | 158 | struct apply_range_data *d = data; |
| 124 | struct page *page; | 159 | struct page *page; |
| 160 | pte_t pteval; | ||
| 125 | 161 | ||
| 126 | if (!data) | 162 | if (!data) |
| 127 | return 0; | 163 | return 0; |
| 128 | /* sanity check */ | ||
| 129 | if (unlikely(!pte_none(ptep_get(pte)))) | ||
| 130 | return -EBUSY; | ||
| 131 | 164 | ||
| 132 | page = d->pages[d->i]; | 165 | page = d->pages[d->i]; |
| 133 | /* paranoia, similar to vmap_pages_pte_range() */ | 166 | /* paranoia, similar to vmap_pages_pte_range() */ |
| 134 | if (WARN_ON_ONCE(!pfn_valid(page_to_pfn(page)))) | 167 | if (WARN_ON_ONCE(!pfn_valid(page_to_pfn(page)))) |
| 135 | return -EINVAL; | 168 | return -EINVAL; |
| 136 | 169 | ||
| 137 | set_pte_at(&init_mm, addr, pte, mk_pte(page, PAGE_KERNEL)); | 170 | pteval = mk_pte(page, PAGE_KERNEL); |
| 171 | #ifdef ptep_try_set | ||
| 172 | /* | ||
| 173 | * Kernel-fault recovery may have installed the scratch page here, and | ||
| 174 | * some architectures (arm64) prohibit valid->valid PTE transitions. | ||
| 175 | * Install atomically into a none slot. If scratch is present, clear it | ||
| 176 | * and flush_tlb_before_set() (break-before-make) before retrying. | ||
| 177 | */ | ||
| 178 | while (!ptep_try_set(pte, pteval)) { | ||
| 179 | pte_t old = ptep_get(pte); | ||
| 180 | |||
| 181 | if (pte_none(old)) | ||
| 182 | continue; | ||
| 183 | if (WARN_ON_ONCE(pte_page(old) != d->scratch_page)) | ||
| 184 | return -EBUSY; | ||
| 185 | ptep_get_and_clear(&init_mm, addr, pte); | ||
| 186 | flush_tlb_before_set(addr); | ||
| 187 | } | ||
| 188 | #else | ||
| 189 | /* | ||
| 190 | * Without ptep_try_set() there is no atomic installer, but such arches | ||
| 191 | * also do not wire up bpf_arena_handle_page_fault(), so no scratch page | ||
| 192 | * is ever installed and the slot is always none here. | ||
| 193 | */ | ||
| 194 | if (unlikely(!pte_none(ptep_get(pte)))) | ||
| 195 | return -EBUSY; | ||
| 196 | set_pte_at(&init_mm, addr, pte, pteval); | ||
| 197 | #endif | ||
| 138 | d->i++; | 198 | d->i++; |
| 139 | return 0; | 199 | return 0; |
| 140 | } | 200 | } |
| @@ -144,33 +204,59 @@ static void flush_vmap_cache(unsigned long start, unsigned long size) · 3 unmodified lines | |||
| 144 | flush_cache_vmap(start, start + size); | 204 | flush_cache_vmap(start, start + size); |
| 145 | } | 205 | } |
| 146 | 206 | ||
| 147 | static int apply_range_clear_cb(pte_t *pte, unsigned long addr, void *free_pages) | 207 | static int apply_range_clear_cb(pte_t *pte, unsigned long addr, void *data) |
| 148 | { | 208 | { |
| 209 | struct clear_range_data *d = data; | ||
| 149 | pte_t old_pte; | 210 | pte_t old_pte; |
| 150 | struct page *page; | 211 | struct page *page; |
| 151 | 212 | ||
| 152 | /* sanity check */ | 213 | /* |
| 153 | old_pte = ptep_get(pte); | 214 | * Pairs with ptep_try_set() in the kernel-fault scratch installer. |
| 215 | * Both sides must be atomic. | ||
| 216 | */ | ||
| 217 | old_pte = ptep_get_and_clear(&init_mm, addr, pte); | ||
| 154 | if (pte_none(old_pte) || !pte_present(old_pte)) | 218 | if (pte_none(old_pte) || !pte_present(old_pte)) |
| 155 | return 0; /* nothing to do */ | 219 | return 0; |
| 156 | 220 | ||
| 157 | page = pte_page(old_pte); | 221 | page = pte_page(old_pte); |
| 158 | if (WARN_ON_ONCE(!page)) | 222 | if (WARN_ON_ONCE(!page)) |
| 159 | return -EINVAL; | 223 | return -EINVAL; |
| 160 | 224 | ||
| 161 | pte_clear(&init_mm, addr, pte); | 225 | /* |
| 226 | * Skip the per-arena scratch page. A kernel fault on an unallocated uaddr | ||
| 227 | * scratches its PTE. A later bpf_arena_free_pages() over that range walks | ||
| 228 | * here. Without the skip, scratch_page would be freed. | ||
| 229 | */ | ||
| 230 | if (page == d->scratch_page) | ||
| 231 | return 0; | ||
| 232 | |||
| 233 | __llist_add(&page->pcp_llist, d->free_pages); | ||
| 234 | return 0; | ||
| 235 | } | ||
| 162 | 236 | ||
| 163 | /* Add page to the list so it is freed later */ | 237 | static int apply_range_set_scratch_cb(pte_t *pte, unsigned long addr, void *data) |
| 164 | if (free_pages) | 238 | { |
| 165 | __llist_add(&page->pcp_llist, free_pages); | 239 | struct page *scratch_page = data; |
| 166 | 240 | ||
| 241 | if (!pte_none(ptep_get(pte))) | ||
| 242 | return 0; | ||
| 243 | /* | ||
| 244 | * Best-effort install. ptep_try_set() returns false only if another | ||
| 245 | * installer (real allocation or concurrent fault) won the cmpxchg. | ||
| 246 | * Their PTE is already valid, so the access retry succeeds. | ||
| 247 | * | ||
| 248 | * No flush_tlb_kernel_range() needed. Stale "not mapped" entries just | ||
| 249 | * cause one extra re-fault through this same path. | ||
| 250 | */ | ||
| 251 | ptep_try_set(pte, mk_pte(scratch_page, PAGE_KERNEL)); | ||
| 167 | return 0; | 252 | return 0; |
| 168 | } | 253 | } |
| 169 | 254 | ||
| 170 | static int populate_pgtable_except_pte(struct bpf_arena *arena) | 255 | static int populate_pgtable_except_pte(struct bpf_arena *arena) |
| 171 | { | 256 | { |
| 257 | /* Populate intermediates for the recovery range (4 GiB + upper half-guard). */ | ||
| 172 | return apply_to_page_range(&init_mm, bpf_arena_get_kern_vm_start(arena), | 258 | return apply_to_page_range(&init_mm, bpf_arena_get_kern_vm_start(arena), |
| 173 | KERN_VM_SZ - GUARD_SZ, apply_range_set_cb, NULL); | 259 | SZ_4G + GUARD_SZ / 2, apply_range_set_cb, NULL); |
| 174 | } | 260 | } |
| 175 | 261 | ||
| 176 | static struct bpf_map *arena_map_alloc(union bpf_attr *attr) | 262 | static struct bpf_map *arena_map_alloc(union bpf_attr *attr) |
| @@ -221,22 +307,30 @@ static struct bpf_map *arena_map_alloc(union bpf_attr *attr) · 44 unmodified lines | |||
| 221 | init_irq_work(&arena->free_irq, arena_free_irq); | 307 | init_irq_work(&arena->free_irq, arena_free_irq); |
| 222 | INIT_WORK(&arena->free_work, arena_free_worker); | 308 | INIT_WORK(&arena->free_work, arena_free_worker); |
| 223 | bpf_map_init_from_attr(&arena->map, attr); | 309 | bpf_map_init_from_attr(&arena->map, attr); |
| 310 | |||
| 311 | err = bpf_map_alloc_pages(&arena->map, NUMA_NO_NODE, 1, &arena->scratch_page); | ||
| 312 | if (err) | ||
| 313 | goto err_free_arena; | ||
| 314 | |||
| 224 | range_tree_init(&arena->rt); | 315 | range_tree_init(&arena->rt); |
| 225 | err = range_tree_set(&arena->rt, 0, attr->max_entries); | 316 | err = range_tree_set(&arena->rt, 0, attr->max_entries); |
| 226 | if (err) { | 317 | if (err) |
| 227 | bpf_map_area_free(arena); | 318 | goto err_free_scratch; |
| 228 | goto err; | ||
| 229 | } | ||
| 230 | mutex_init(&arena->lock); | 319 | mutex_init(&arena->lock); |
| 320 | mutex_init(&arena->zap_mutex); | ||
| 231 | raw_res_spin_lock_init(&arena->spinlock); | 321 | raw_res_spin_lock_init(&arena->spinlock); |
| 232 | err = populate_pgtable_except_pte(arena); | 322 | err = populate_pgtable_except_pte(arena); |
| 233 | if (err) { | 323 | if (err) |
| 234 | range_tree_destroy(&arena->rt); | 324 | goto err_destroy_rt; |
| 235 | bpf_map_area_free(arena); | ||
| 236 | goto err; | ||
| 237 | } | ||
| 238 | 325 | ||
| 239 | return &arena->map; | 326 | return &arena->map; |
| 327 | |||
| 328 | err_destroy_rt: | ||
| 329 | range_tree_destroy(&arena->rt); | ||
| 330 | err_free_scratch: | ||
| 331 | __free_page(arena->scratch_page); | ||
| 332 | err_free_arena: | ||
| 333 | bpf_map_area_free(arena); | ||
| 240 | err: | 334 | err: |
| 241 | free_vm_area(kern_vm); | 335 | free_vm_area(kern_vm); |
| 242 | return ERR_PTR(err); | 336 | return ERR_PTR(err); |
| @@ -244,6 +338,7 @@ err: · 1 unmodified line | |||
| 244 | 338 | ||
| 245 | static int existing_page_cb(pte_t *ptep, unsigned long addr, void *data) | 339 | static int existing_page_cb(pte_t *ptep, unsigned long addr, void *data) |
| 246 | { | 340 | { |
| 341 | struct bpf_arena *arena = data; | ||
| 247 | struct page *page; | 342 | struct page *page; |
| 248 | pte_t pte; | 343 | pte_t pte; |
| 249 | 344 | ||
| @@ -251,6 +346,12 @@ static int existing_page_cb(pte_t *ptep, unsigned long addr, void *data) · 1 unmodified line | |||
| 251 | if (!pte_present(pte)) /* sanity check */ | 346 | if (!pte_present(pte)) /* sanity check */ |
| 252 | return 0; | 347 | return 0; |
| 253 | page = pte_page(pte); | 348 | page = pte_page(pte); |
| 349 | /* | ||
| 350 | * Skip the scratch page. The walk is page-table-driven, not range-tree-driven, | ||
| 351 | * so it can visit scratch PTEs at uaddrs the BPF program never allocated. | ||
| 352 | */ | ||
| 353 | if (page == arena->scratch_page) | ||
| 354 | return 0; | ||
| 254 | /* | 355 | /* |
| 255 | * We do not update pte here: | 356 | * We do not update pte here: |
| 256 | * 1. Nobody should be accessing bpf_arena's range outside of a kernel bug | 357 | * 1. Nobody should be accessing bpf_arena's range outside of a kernel bug |
| @@ -286,9 +387,10 @@ static void arena_map_free(struct bpf_map *map) · 29 unmodified lines | |||
| 286 | * free those pages. | 387 | * free those pages. |
| 287 | */ | 388 | */ |
| 288 | apply_to_existing_page_range(&init_mm, bpf_arena_get_kern_vm_start(arena), | 389 | apply_to_existing_page_range(&init_mm, bpf_arena_get_kern_vm_start(arena), |
| 289 | KERN_VM_SZ - GUARD_SZ, existing_page_cb, NULL); | 390 | SZ_4G + GUARD_SZ / 2, existing_page_cb, arena); |
| 290 | free_vm_area(arena->kern_vm); | 391 | free_vm_area(arena->kern_vm); |
| 291 | range_tree_destroy(&arena->rt); | 392 | range_tree_destroy(&arena->rt); |
| 393 | __free_page(arena->scratch_page); | ||
| 292 | bpf_map_area_free(arena); | 394 | bpf_map_area_free(arena); |
| 293 | } | 395 | } |
| 294 | 396 | ||
| @@ -318,6 +420,7 @@ struct vma_list { · 23 unmodified lines | |||
| 318 | struct vm_area_struct *vma; | 420 | struct vm_area_struct *vma; |
| 319 | struct list_head head; | 421 | struct list_head head; |
| 320 | refcount_t mmap_count; | 422 | refcount_t mmap_count; |
| 423 | u64 zap_gen; | ||
| 321 | }; | 424 | }; |
| 322 | 425 | ||
| 323 | static int remember_vma(struct bpf_arena *arena, struct vm_area_struct *vma) | 426 | static int remember_vma(struct bpf_arena *arena, struct vm_area_struct *vma) |
| @@ -330,6 +433,7 @@ static int remember_vma(struct bpf_arena *arena, struct vm_area_struct *vma) · 6 unmodified lines | |||
| 330 | refcount_set(&vml->mmap_count, 1); | 433 | refcount_set(&vml->mmap_count, 1); |
| 331 | vma->vm_private_data = vml; | 434 | vma->vm_private_data = vml; |
| 332 | vml->vma = vma; | 435 | vml->vma = vma; |
| 436 | vml->zap_gen = 0; | ||
| 333 | list_add(&vml->head, &arena->vma_list); | 437 | list_add(&vml->head, &arena->vma_list); |
| 334 | return 0; | 438 | return 0; |
| 335 | } | 439 | } |
| @@ -384,33 +488,38 @@ static vm_fault_t arena_vm_fault(struct vm_fault *vmf) · 48 unmodified lines | |||
| 384 | return VM_FAULT_RETRY; | 488 | return VM_FAULT_RETRY; |
| 385 | 489 | ||
| 386 | page = vmalloc_to_page((void *)kaddr); | 490 | page = vmalloc_to_page((void *)kaddr); |
| 387 | if (page) | 491 | if (page) { |
| 492 | if (page == arena->scratch_page) | ||
| 493 | /* BPF triggered scratch here; don't lazy-alloc over it */ | ||
| 494 | goto out_sigsegv; | ||
| 388 | /* already have a page vmap-ed */ | 495 | /* already have a page vmap-ed */ |
| 389 | goto out; | 496 | goto out; |
| 497 | } | ||
| 390 | 498 | ||
| 391 | bpf_map_memcg_enter(&arena->map, &old_memcg, &new_memcg); | 499 | bpf_map_memcg_enter(&arena->map, &old_memcg, &new_memcg); |
| 392 | 500 | ||
| 393 | if (arena->map.map_flags & BPF_F_SEGV_ON_FAULT) | 501 | if (arena->map.map_flags & BPF_F_SEGV_ON_FAULT) |
| 394 | /* User space requested to segfault when page is not allocated by bpf prog */ | 502 | /* User space requested to segfault when page is not allocated by bpf prog */ |
| 395 | goto out_unlock_sigsegv; | 503 | goto out_sigsegv_memcg; |
| 396 | 504 | ||
| 397 | ret = range_tree_clear(&arena->rt, vmf->pgoff, 1); | 505 | ret = range_tree_clear(&arena->rt, vmf->pgoff, 1); |
| 398 | if (ret) | 506 | if (ret) |
| 399 | goto out_unlock_sigsegv; | 507 | goto out_sigsegv_memcg; |
| 400 | 508 | ||
| 401 | struct apply_range_data data = { .pages = &page, .i = 0 }; | 509 | struct apply_range_data data = { .pages = &page, .i = 0, |
| 510 | .scratch_page = arena->scratch_page }; | ||
| 402 | /* Account into memcg of the process that created bpf_arena */ | 511 | /* Account into memcg of the process that created bpf_arena */ |
| 403 | ret = bpf_map_alloc_pages(map, NUMA_NO_NODE, 1, &page); | 512 | ret = bpf_map_alloc_pages(map, NUMA_NO_NODE, 1, &page); |
| 404 | if (ret) { | 513 | if (ret) { |
| 405 | range_tree_set(&arena->rt, vmf->pgoff, 1); | 514 | range_tree_set(&arena->rt, vmf->pgoff, 1); |
| 406 | goto out_unlock_sigsegv; | 515 | goto out_sigsegv_memcg; |
| 407 | } | 516 | } |
| 408 | 517 | ||
| 409 | ret = apply_to_page_range(&init_mm, kaddr, PAGE_SIZE, apply_range_set_cb, &data); | 518 | ret = apply_to_page_range(&init_mm, kaddr, PAGE_SIZE, apply_range_set_cb, &data); |
| 410 | if (ret) { | 519 | if (ret) { |
| 411 | range_tree_set(&arena->rt, vmf->pgoff, 1); | 520 | range_tree_set(&arena->rt, vmf->pgoff, 1); |
| 412 | free_pages_nolock(page, 0); | 521 | free_pages_nolock(page, 0); |
| 413 | goto out_unlock_sigsegv; | 522 | goto out_sigsegv_memcg; |
| 414 | } | 523 | } |
| 415 | flush_vmap_cache(kaddr, PAGE_SIZE); | 524 | flush_vmap_cache(kaddr, PAGE_SIZE); |
| 416 | bpf_map_memcg_exit(old_memcg, new_memcg); | 525 | bpf_map_memcg_exit(old_memcg, new_memcg); |
| @@ -419,8 +528,9 @@ out: · 2 unmodified lines | |||
| 419 | raw_res_spin_unlock_irqrestore(&arena->spinlock, flags); | 528 | raw_res_spin_unlock_irqrestore(&arena->spinlock, flags); |
| 420 | vmf->page = page; | 529 | vmf->page = page; |
| 421 | return 0; | 530 | return 0; |
| 422 | out_unlock_sigsegv: | 531 | out_sigsegv_memcg: |
| 423 | bpf_map_memcg_exit(old_memcg, new_memcg); | 532 | bpf_map_memcg_exit(old_memcg, new_memcg); |
| 533 | out_sigsegv: | ||
| 424 | raw_res_spin_unlock_irqrestore(&arena->spinlock, flags); | 534 | raw_res_spin_unlock_irqrestore(&arena->spinlock, flags); |
| 425 | return VM_FAULT_SIGSEGV; | 535 | return VM_FAULT_SIGSEGV; |
| 426 | } | 536 | } |
| @@ -587,6 +697,7 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt · 160 unmodified lines | |||
| 587 | return 0; | 697 | return 0; |
| 588 | } | 698 | } |
| 589 | data.pages = pages; | 699 | data.pages = pages; |
| 700 | data.scratch_page = arena->scratch_page; | ||
| 590 | 701 | ||
| 591 | if (raw_res_spin_lock_irqsave(&arena->spinlock, flags)) | 702 | if (raw_res_spin_lock_irqsave(&arena->spinlock, flags)) |
| 592 | goto out_free_pages; | 703 | goto out_free_pages; |
| @@ -668,12 +779,60 @@ out_free_pages: · 75 unmodified lines | |||
| 668 | */ | 779 | */ |
| 669 | static void zap_pages(struct bpf_arena *arena, long uaddr, long page_cnt) | 780 | static void zap_pages(struct bpf_arena *arena, long uaddr, long page_cnt) |
| 670 | { | 781 | { |
| 782 | unsigned long size = (unsigned long)page_cnt << PAGE_SHIFT; | ||
| 783 | struct vm_area_struct *vma; | ||
| 784 | struct mm_struct *mm; | ||
| 671 | struct vma_list *vml; | 785 | struct vma_list *vml; |
| 786 | unsigned long vm_start; | ||
| 787 | u64 my_gen; | ||
| 672 | 788 | ||
| 673 | guard(mutex)(&arena->lock); | 789 | /* |
| 674 | /* iterate link list under lock */ | 790 | * Taking mmap_read_lock() under arena->lock would deadlock against |
| 675 | list_for_each_entry(vml, &arena->vma_list, head) | 791 | * arena_vm_close(), which runs with mmap_write_lock held and then |
| 676 | zap_vma_range(vml->vma, uaddr, PAGE_SIZE * page_cnt); | 792 | * acquires arena->lock. Drop arena->lock for mmap_read_lock(). |
| 793 | * | ||
| 794 | * Use per-call my_gen, recorded in vml->zap_gen, to remember which | ||
| 795 | * vmls this invocation has already processed across the lock drop. | ||
| 796 | * Hold zap_mutex around the whole walk so concurrent zap_pages() | ||
| 797 | * callers cannot overwrite each other's marks on shared vmls -- | ||
| 798 | * otherwise call B's mark would make call A skip a vml that A has | ||
| 799 | * not yet zapped for A's uaddr range. | ||
| 800 | */ | ||
| 801 | mutex_lock(&arena->zap_mutex); | ||
| 802 | mutex_lock(&arena->lock); | ||
| 803 | my_gen = ++arena->zap_gen; | ||
| 804 | for (;;) { | ||
| 805 | mm = NULL; | ||
| 806 | list_for_each_entry(vml, &arena->vma_list, head) { | ||
| 807 | if (vml->zap_gen >= my_gen) | ||
| 808 | continue; | ||
| 809 | vml->zap_gen = my_gen; | ||
| 810 | if (!mmget_not_zero(vml->vma->vm_mm)) | ||
| 811 | continue; | ||
| 812 | mm = vml->vma->vm_mm; | ||
| 813 | vm_start = vml->vma->vm_start; | ||
| 814 | break; | ||
| 815 | } | ||
| 816 | if (!mm) | ||
| 817 | break; | ||
| 818 | mutex_unlock(&arena->lock); | ||
| 819 | |||
| 820 | mmap_read_lock(mm); | ||
| 821 | /* | ||
| 822 | * Re-resolve: while we waited the VMA could have been unmapped | ||
| 823 | * and a different mapping installed at the same address. | ||
| 824 | */ | ||
| 825 | vma = find_vma(mm, vm_start); | ||
| 826 | if (vma && vma->vm_start == vm_start && | ||
| 827 | vma->vm_file && vma->vm_file->private_data == &arena->map) | ||
| 828 | zap_vma_range(vma, uaddr, size); | ||
| 829 | mmap_read_unlock(mm); | ||
| 830 | mmput(mm); | ||
| 831 | |||
| 832 | mutex_lock(&arena->lock); | ||
| 833 | } | ||
| 834 | mutex_unlock(&arena->lock); | ||
| 835 | mutex_unlock(&arena->zap_mutex); | ||
| 677 | } | 836 | } |
| 678 | 837 | ||
| 679 | static void arena_free_pages(struct bpf_arena *arena, long uaddr, long page_cnt, bool sleepable) | 838 | static void arena_free_pages(struct bpf_arena *arena, long uaddr, long page_cnt, bool sleepable) |
| @@ -685,6 +844,7 @@ static void arena_free_pages(struct bpf_arena *arena, long uaddr, long page_cnt, · 5 unmodified lines | |||
| 685 | struct llist_head free_pages; | 844 | struct llist_head free_pages; |
| 686 | struct llist_node *pos, *t; | 845 | struct llist_node *pos, *t; |
| 687 | struct arena_free_span *s; | 846 | struct arena_free_span *s; |
| 847 | struct clear_range_data cdata; | ||
| 688 | unsigned long flags; | 848 | unsigned long flags; |
| 689 | int ret = 0; | 849 | int ret = 0; |
| 690 | 850 | ||
| @@ -713,9 +873,11 @@ static void arena_free_pages(struct bpf_arena *arena, long uaddr, long page_cnt, · 22 unmodified lines | |||
| 713 | range_tree_set(&arena->rt, pgoff, page_cnt); | 873 | range_tree_set(&arena->rt, pgoff, page_cnt); |
| 714 | 874 | ||
| 715 | init_llist_head(&free_pages); | 875 | init_llist_head(&free_pages); |
| 876 | cdata.free_pages = &free_pages; | ||
| 877 | cdata.scratch_page = arena->scratch_page; | ||
| 716 | /* clear ptes and collect struct pages */ | 878 | /* clear ptes and collect struct pages */ |
| 717 | apply_to_existing_page_range(&init_mm, kaddr, page_cnt << PAGE_SHIFT, | 879 | apply_to_existing_page_range(&init_mm, kaddr, page_cnt << PAGE_SHIFT, |
| 718 | apply_range_clear_cb, &free_pages); | 880 | apply_range_clear_cb, &cdata); |
| 719 | 881 | ||
| 720 | /* drop the lock to do the tlb flush and zap pages */ | 882 | /* drop the lock to do the tlb flush and zap pages */ |
| 721 | raw_res_spin_unlock_irqrestore(&arena->spinlock, flags); | 883 | raw_res_spin_unlock_irqrestore(&arena->spinlock, flags); |
| @@ -805,6 +967,7 @@ static void arena_free_worker(struct work_struct *work) · 83 unmodified lines | |||
| 805 | struct arena_free_span *s; | 967 | struct arena_free_span *s; |
| 806 | u64 arena_vm_start, user_vm_start; | 968 | u64 arena_vm_start, user_vm_start; |
| 807 | struct llist_head free_pages; | 969 | struct llist_head free_pages; |
| 970 | struct clear_range_data cdata; | ||
| 808 | struct page *page; | 971 | struct page *page; |
| 809 | unsigned long full_uaddr; | 972 | unsigned long full_uaddr; |
| 810 | long kaddr, page_cnt, pgoff; | 973 | long kaddr, page_cnt, pgoff; |
| @@ -818,6 +981,8 @@ static void arena_free_worker(struct work_struct *work) · 7 unmodified lines | |||
| 818 | bpf_map_memcg_enter(&arena->map, &old_memcg, &new_memcg); | 981 | bpf_map_memcg_enter(&arena->map, &old_memcg, &new_memcg); |
| 819 | 982 | ||
| 820 | init_llist_head(&free_pages); | 983 | init_llist_head(&free_pages); |
| 984 | cdata.free_pages = &free_pages; | ||
| 985 | cdata.scratch_page = arena->scratch_page; | ||
| 821 | arena_vm_start = bpf_arena_get_kern_vm_start(arena); | 986 | arena_vm_start = bpf_arena_get_kern_vm_start(arena); |
| 822 | user_vm_start = bpf_arena_get_user_vm_start(arena); | 987 | user_vm_start = bpf_arena_get_user_vm_start(arena); |
| 823 | 988 | ||
| @@ -830,7 +995,7 @@ static void arena_free_worker(struct work_struct *work) · 6 unmodified lines | |||
| 830 | 995 | ||
| 831 | /* clear ptes and collect pages in free_pages llist */ | 996 | /* clear ptes and collect pages in free_pages llist */ |
| 832 | apply_to_existing_page_range(&init_mm, kaddr, page_cnt << PAGE_SHIFT, | 997 | apply_to_existing_page_range(&init_mm, kaddr, page_cnt << PAGE_SHIFT, |
| 833 | apply_range_clear_cb, &free_pages); | 998 | apply_range_clear_cb, &cdata); |
| 834 | 999 | ||
| 835 | range_tree_set(&arena->rt, pgoff, page_cnt); | 1000 | range_tree_set(&arena->rt, pgoff, page_cnt); |
| 836 | } | 1001 | } |
| @@ -893,6 +1058,19 @@ void *bpf_arena_alloc_pages_non_sleepable(void *p__map, void *addr__ign, u32 pag · 56 unmodified lines | |||
| 893 | 1058 | ||
| 894 | return (void *)arena_alloc_pages(arena, (long)addr__ign, page_cnt, node_id, false); | 1059 | return (void *)arena_alloc_pages(arena, (long)addr__ign, page_cnt, node_id, false); |
| 895 | } | 1060 | } |
| 1061 | |||
| 1062 | void *bpf_arena_alloc_pages_sleepable(void *p__map, void *addr__ign, u32 page_cnt, | ||
| 1063 | int node_id, u64 flags) | ||
| 1064 | { | ||
| 1065 | struct bpf_map *map = p__map; | ||
| 1066 | struct bpf_arena *arena = container_of(map, struct bpf_arena, map); | ||
| 1067 | |||
| 1068 | if (map->map_type != BPF_MAP_TYPE_ARENA || flags || !page_cnt) | ||
| 1069 | return NULL; | ||
| 1070 | |||
| 1071 | return (void *)arena_alloc_pages(arena, (long)addr__ign, page_cnt, node_id, true); | ||
| 1072 | } | ||
| 1073 | |||
| 896 | __bpf_kfunc void bpf_arena_free_pages(void *p__map, void *ptr__ign, u32 page_cnt) | 1074 | __bpf_kfunc void bpf_arena_free_pages(void *p__map, void *ptr__ign, u32 page_cnt) |
| 897 | { | 1075 | { |
| 898 | struct bpf_map *map = p__map; | 1076 | struct bpf_map *map = p__map; |
| @@ -945,23 +1123,12 @@ static int __init kfunc_init(void) · 46 unmodified lines | |||
| 945 | } | 1123 | } |
| 946 | late_initcall(kfunc_init); | 1124 | late_initcall(kfunc_init); |
| 947 | 1125 | ||
| 948 | void bpf_prog_report_arena_violation(bool write, unsigned long addr, unsigned long fault_ip) | 1126 | static void __bpf_prog_report_arena_violation(struct bpf_prog *prog, bool write, |
| 1127 | unsigned long addr, unsigned long fault_ip) | ||
| 949 | { | 1128 | { |
| 950 | struct bpf_stream_stage ss; | 1129 | struct bpf_stream_stage ss; |
| 951 | struct bpf_prog *prog; | ||
| 952 | u64 user_vm_start; | 1130 | u64 user_vm_start; |
| 953 | 1131 | ||
| 954 | /* | ||
| 955 | * The RCU read lock is held to safely traverse the latch tree, but we | ||
| 956 | * don't need its protection when accessing the prog, since it will not | ||
| 957 | * disappear while we are handling the fault. | ||
| 958 | */ | ||
| 959 | rcu_read_lock(); | ||
| 960 | prog = bpf_prog_ksym_find(fault_ip); | ||
| 961 | rcu_read_unlock(); | ||
| 962 | if (!prog) | ||
| 963 | return; | ||
| 964 | |||
| 965 | /* Use main prog for stream access */ | 1132 | /* Use main prog for stream access */ |
| 966 | prog = prog->aux->main_prog_aux->prog; | 1133 | prog = prog->aux->main_prog_aux->prog; |
| 967 | 1134 | ||
| @@ -974,3 +1141,53 @@ void bpf_prog_report_arena_violation(bool write, unsigned long addr, unsigned lo · 6 unmodified lines | |||
| 974 | bpf_stream_dump_stack(ss); | 1141 | bpf_stream_dump_stack(ss); |
| 975 | })); | 1142 | })); |
| 976 | } | 1143 | } |
| 1144 | |||
| 1145 | bool bpf_arena_handle_page_fault(unsigned long addr, bool is_write, unsigned long fault_ip) | ||
| 1146 | { | ||
| 1147 | struct bpf_arena *arena; | ||
| 1148 | struct bpf_prog *prog; | ||
| 1149 | unsigned long kbase; | ||
| 1150 | unsigned long page_addr = addr & PAGE_MASK; | ||
| 1151 | |||
| 1152 | prog = bpf_prog_find_from_stack(); | ||
| 1153 | if (!prog) | ||
| 1154 | return false; | ||
| 1155 | |||
| 1156 | arena = prog->aux->arena; | ||
| 1157 | /* a prog not using arena may be on stack, so arena can be NULL */ | ||
| 1158 | if (!arena) | ||
| 1159 | return false; | ||
| 1160 | |||
| 1161 | kbase = bpf_arena_get_kern_vm_start(arena); | ||
| 1162 | |||
| 1163 | /* | ||
| 1164 | * Recovery covers the 4 GiB mappable band plus the upper half-guard. | ||
| 1165 | * Lower guard is unreachable from kfuncs; an address there indicates | ||
| 1166 | * a different bug class - leave it to the regular kernel oops path. | ||
| 1167 | */ | ||
| 1168 | if (page_addr < kbase || page_addr >= kbase + SZ_4G + GUARD_SZ / 2) | ||
| 1169 | return false; | ||
| 1170 | |||
| 1171 | apply_to_page_range(&init_mm, page_addr, PAGE_SIZE, | ||
| 1172 | apply_range_set_scratch_cb, arena->scratch_page); | ||
| 1173 | flush_vmap_cache(page_addr, PAGE_SIZE); | ||
| 1174 | __bpf_prog_report_arena_violation(prog, is_write, page_addr - kbase, fault_ip); | ||
| 1175 | return true; | ||
| 1176 | } | ||
| 1177 | |||
| 1178 | void bpf_prog_report_arena_violation(bool write, unsigned long addr, unsigned long fault_ip) | ||
| 1179 | { | ||
| 1180 | struct bpf_prog *prog; | ||
| 1181 | |||
| 1182 | /* | ||
| 1183 | * The RCU read lock is held to safely traverse the latch tree, but we | ||
| 1184 | * don't need its protection when accessing the prog, since it will not | ||
| 1185 | * disappear while we are handling the fault. | ||
| 1186 | */ | ||
| 1187 | rcu_read_lock(); | ||
| 1188 | prog = bpf_prog_ksym_find(fault_ip); | ||
| 1189 | rcu_read_unlock(); | ||
| 1190 | if (!prog) | ||
| 1191 | return; | ||
| 1192 | __bpf_prog_report_arena_violation(prog, write, addr, fault_ip); | ||
| 1193 | } | ||
| @@ -175,14 +175,12 @@ static void *array_map_lookup_elem(struct bpf_map *map, void *key) · 174 unmodified lines | |||
| 175 | return array->value + (u64)array->elem_size * (index & array->index_mask); | 175 | return array->value + (u64)array->elem_size * (index & array->index_mask); |
| 176 | } | 176 | } |
| 177 | 177 | ||
| 178 | static int array_map_get_hash(struct bpf_map *map, u32 hash_buf_size, | 178 | static int array_map_get_hash(struct bpf_map *map) |
| 179 | void *hash_buf) | ||
| 180 | { | 179 | { |
| 181 | struct bpf_array *array = container_of(map, struct bpf_array, map); | 180 | struct bpf_array *array = container_of(map, struct bpf_array, map); |
| 182 | 181 | ||
| 183 | sha256(array->value, (u64)array->elem_size * array->map.max_entries, | 182 | sha256(array->value, (u64)array->elem_size * array->map.max_entries, |
| 184 | hash_buf); | 183 | array->map.sha); |
| 185 | memcpy(array->map.sha, hash_buf, sizeof(array->map.sha)); | ||
| 186 | return 0; | 184 | return 0; |
| 187 | } | 185 | } |
| 188 | 186 | ||
| @@ -386,7 +384,7 @@ static long array_map_update_elem(struct bpf_map *map, void *key, void *value, · 197 unmodified lines | |||
| 386 | if (array->map.map_type == BPF_MAP_TYPE_PERCPU_ARRAY) { | 384 | if (array->map.map_type == BPF_MAP_TYPE_PERCPU_ARRAY) { |
| 387 | val = this_cpu_ptr(array->pptrs[index & array->index_mask]); | 385 | val = this_cpu_ptr(array->pptrs[index & array->index_mask]); |
| 388 | copy_map_value(map, val, value); | 386 | copy_map_value(map, val, value); |
| 389 | bpf_obj_free_fields(array->map.record, val); | 387 | bpf_obj_cancel_fields(map, val); |
| 390 | } else { | 388 | } else { |
| 391 | val = array->value + | 389 | val = array->value + |
| 392 | (u64)array->elem_size * (index & array->index_mask); | 390 | (u64)array->elem_size * (index & array->index_mask); |
| @@ -394,7 +392,7 @@ static long array_map_update_elem(struct bpf_map *map, void *key, void *value, · 1 unmodified line | |||
| 394 | copy_map_value_locked(map, val, value, false); | 392 | copy_map_value_locked(map, val, value, false); |
| 395 | else | 393 | else |
| 396 | copy_map_value(map, val, value); | 394 | copy_map_value(map, val, value); |
| 397 | bpf_obj_free_fields(array->map.record, val); | 395 | bpf_obj_cancel_fields(map, val); |
| 398 | } | 396 | } |
| 399 | return 0; | 397 | return 0; |
| 400 | } | 398 | } |
| @@ -434,14 +432,14 @@ int bpf_percpu_array_update(struct bpf_map *map, void *key, void *value, · 33 unmodified lines | |||
| 434 | cpu = map_flags >> 32; | 432 | cpu = map_flags >> 32; |
| 435 | ptr = per_cpu_ptr(pptr, cpu); | 433 | ptr = per_cpu_ptr(pptr, cpu); |
| 436 | copy_map_value(map, ptr, value); | 434 | copy_map_value(map, ptr, value); |
| 437 | bpf_obj_free_fields(array->map.record, ptr); | 435 | bpf_obj_cancel_fields(map, ptr); |
| 438 | goto unlock; | 436 | goto unlock; |
| 439 | } | 437 | } |
| 440 | for_each_possible_cpu(cpu) { | 438 | for_each_possible_cpu(cpu) { |
| 441 | ptr = per_cpu_ptr(pptr, cpu); | 439 | ptr = per_cpu_ptr(pptr, cpu); |
| 442 | val = (map_flags & BPF_F_ALL_CPUS) ? value : value + size * cpu; | 440 | val = (map_flags & BPF_F_ALL_CPUS) ? value : value + size * cpu; |
| 443 | copy_map_value(map, ptr, val); | 441 | copy_map_value(map, ptr, val); |
| 444 | bpf_obj_free_fields(array->map.record, ptr); | 442 | bpf_obj_cancel_fields(map, ptr); |
| 445 | } | 443 | } |
| 446 | unlock: | 444 | unlock: |
| 447 | rcu_read_unlock(); | 445 | rcu_read_unlock(); |
| @@ -9,7 +9,7 @@ · 8 unmodified lines | |||
| 9 | 9 | ||
| 10 | /* for any branch, call, exit record the history of jmps in the given state */ | 10 | /* for any branch, call, exit record the history of jmps in the given state */ |
| 11 | int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state *cur, | 11 | int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state *cur, |
| 12 | int insn_flags, u64 linked_regs) | 12 | int insn_flags, int spi, int frame, u64 linked_regs) |
| 13 | { | 13 | { |
| 14 | u32 cnt = cur->jmp_history_cnt; | 14 | u32 cnt = cur->jmp_history_cnt; |
| 15 | struct bpf_jmp_history_entry *p; | 15 | struct bpf_jmp_history_entry *p; |
| @@ -25,6 +25,8 @@ int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state · 9 unmodified lines | |||
| 25 | env, "insn history: insn_idx %d cur flags %x new flags %x", | 25 | env, "insn history: insn_idx %d cur flags %x new flags %x", |
| 26 | env->insn_idx, env->cur_hist_ent->flags, insn_flags); | 26 | env->insn_idx, env->cur_hist_ent->flags, insn_flags); |
| 27 | env->cur_hist_ent->flags |= insn_flags; | 27 | env->cur_hist_ent->flags |= insn_flags; |
| 28 | env->cur_hist_ent->spi = spi; | ||
| 29 | env->cur_hist_ent->frame = frame; | ||
| 28 | verifier_bug_if(env->cur_hist_ent->linked_regs != 0, env, | 30 | verifier_bug_if(env->cur_hist_ent->linked_regs != 0, env, |
| 29 | "insn history: insn_idx %d linked_regs: %#llx", | 31 | "insn history: insn_idx %d linked_regs: %#llx", |
| 30 | env->insn_idx, env->cur_hist_ent->linked_regs); | 32 | env->insn_idx, env->cur_hist_ent->linked_regs); |
| @@ -43,6 +45,8 @@ int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state · 12 unmodified lines | |||
| 43 | p->idx = env->insn_idx; | 45 | p->idx = env->insn_idx; |
| 44 | p->prev_idx = env->prev_insn_idx; | 46 | p->prev_idx = env->prev_insn_idx; |
| 45 | p->flags = insn_flags; | 47 | p->flags = insn_flags; |
| 48 | p->spi = spi; | ||
| 49 | p->frame = frame; | ||
| 46 | p->linked_regs = linked_regs; | 50 | p->linked_regs = linked_regs; |
| 47 | cur->jmp_history_cnt = cnt; | 51 | cur->jmp_history_cnt = cnt; |
| 48 | env->cur_hist_ent = p; | 52 | env->cur_hist_ent = p; |
| @@ -64,16 +68,6 @@ static bool is_atomic_fetch_insn(const struct bpf_insn *insn) · 15 unmodified lines | |||
| 64 | (insn->imm & BPF_FETCH); | 68 | (insn->imm & BPF_FETCH); |
| 65 | } | 69 | } |
| 66 | 70 | ||
| 67 | static int insn_stack_access_spi(int insn_flags) | ||
| 68 | { | ||
| 69 | return (insn_flags >> INSN_F_SPI_SHIFT) & INSN_F_SPI_MASK; | ||
| 70 | } | ||
| 71 | |||
| 72 | static int insn_stack_access_frameno(int insn_flags) | ||
| 73 | { | ||
| 74 | return insn_flags & INSN_F_FRAMENO_MASK; | ||
| 75 | } | ||
| 76 | |||
| 77 | /* Backtrack one insn at a time. If idx is not at the top of recorded | 71 | /* Backtrack one insn at a time. If idx is not at the top of recorded |
| 78 | * history then previous instruction came from straight line execution. | 72 | * history then previous instruction came from straight line execution. |
| 79 | * Return -ENOENT if we exhausted all instructions within given state. | 73 | * Return -ENOENT if we exhausted all instructions within given state. |
| @@ -135,11 +129,21 @@ static inline u32 bt_empty(struct backtrack_state *bt) · 55 unmodified lines | |||
| 135 | int i; | 129 | int i; |
| 136 | 130 | ||
| 137 | for (i = 0; i <= bt->frame; i++) | 131 | for (i = 0; i <= bt->frame; i++) |
| 138 | mask |= bt->reg_masks[i] | bt->stack_masks[i]; | 132 | mask |= bt->reg_masks[i] | bt->stack_masks[i] | bt->stack_arg_masks[i]; |
| 139 | 133 | ||
| 140 | return mask == 0; | 134 | return mask == 0; |
| 141 | } | 135 | } |
| 142 | 136 | ||
| 137 | static inline void bt_clear_frame_stack_arg_slot(struct backtrack_state *bt, u32 frame, u32 slot) | ||
| 138 | { | ||
| 139 | bt->stack_arg_masks[frame] &= ~(1 << slot); | ||
| 140 | } | ||
| 141 | |||
| 142 | static inline bool bt_is_frame_stack_arg_slot_set(struct backtrack_state *bt, u32 frame, u32 slot) | ||
| 143 | { | ||
| 144 | return bt->stack_arg_masks[frame] & (1 << slot); | ||
| 145 | } | ||
| 146 | |||
| 143 | static inline int bt_subprog_enter(struct backtrack_state *bt) | 147 | static inline int bt_subprog_enter(struct backtrack_state *bt) |
| 144 | { | 148 | { |
| 145 | if (bt->frame == MAX_CALL_FRAMES - 1) { | 149 | if (bt->frame == MAX_CALL_FRAMES - 1) { |
| @@ -200,6 +204,11 @@ static inline u64 bt_stack_mask(struct backtrack_state *bt) · 54 unmodified lines | |||
| 200 | return bt->stack_masks[bt->frame]; | 204 | return bt->stack_masks[bt->frame]; |
| 201 | } | 205 | } |
| 202 | 206 | ||
| 207 | static inline u8 bt_stack_arg_mask(struct backtrack_state *bt) | ||
| 208 | { | ||
| 209 | return bt->stack_arg_masks[bt->frame]; | ||
| 210 | } | ||
| 211 | |||
| 203 | static inline bool bt_is_reg_set(struct backtrack_state *bt, u32 reg) | 212 | static inline bool bt_is_reg_set(struct backtrack_state *bt, u32 reg) |
| 204 | { | 213 | { |
| 205 | return bt->reg_masks[bt->frame] & (1 << reg); | 214 | return bt->reg_masks[bt->frame] & (1 << reg); |
| @@ -341,6 +350,19 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx, · 135 unmodified lines | |||
| 341 | return 0; | 350 | return 0; |
| 342 | bt_clear_reg(bt, load_reg); | 351 | bt_clear_reg(bt, load_reg); |
| 343 | 352 | ||
| 353 | if (hist && hist->flags & INSN_F_STACK_ARG_ACCESS) { | ||
| 354 | spi = hist->spi; | ||
| 355 | /* | ||
| 356 | * Stack arg read: callee reads from r11+off, but | ||
| 357 | * the data lives in the caller's stack_arg_regs. | ||
| 358 | * Set the mask in the caller frame so precision | ||
| 359 | * is marked in the caller's slot at the callee | ||
| 360 | * entry checkpoint. | ||
| 361 | */ | ||
| 362 | bt_set_frame_stack_arg_slot(bt, bt->frame - 1, spi); | ||
| 363 | return 0; | ||
| 364 | } | ||
| 365 | |||
| 344 | /* scalars can only be spilled into stack w/o losing precision. | 366 | /* scalars can only be spilled into stack w/o losing precision. |
| 345 | * Load from any other memory can be zero extended. | 367 | * Load from any other memory can be zero extended. |
| 346 | * The desire to keep that precision is already indicated | 368 | * The desire to keep that precision is already indicated |
| @@ -353,8 +375,8 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx, · 6 unmodified lines | |||
| 353 | * that [fp - off] slot contains scalar that needs to be | 375 | * that [fp - off] slot contains scalar that needs to be |
| 354 | * tracked with precision | 376 | * tracked with precision |
| 355 | */ | 377 | */ |
| 356 | spi = insn_stack_access_spi(hist->flags); | 378 | spi = hist->spi; |
| 357 | fr = insn_stack_access_frameno(hist->flags); | 379 | fr = hist->frame; |
| 358 | bpf_bt_set_frame_slot(bt, fr, spi); | 380 | bpf_bt_set_frame_slot(bt, fr, spi); |
| 359 | } else if (class == BPF_STX || class == BPF_ST) { | 381 | } else if (class == BPF_STX || class == BPF_ST) { |
| 360 | if (bt_is_reg_set(bt, dreg)) | 382 | if (bt_is_reg_set(bt, dreg)) |
| @@ -363,11 +385,22 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx, · 2 unmodified lines | |||
| 363 | * encountered a case of pointer subtraction. | 385 | * encountered a case of pointer subtraction. |
| 364 | */ | 386 | */ |
| 365 | return -ENOTSUPP; | 387 | return -ENOTSUPP; |
| 388 | |||
| 389 | if (hist && hist->flags & INSN_F_STACK_ARG_ACCESS) { | ||
| 390 | spi = hist->spi; | ||
| 391 | if (!bt_is_frame_stack_arg_slot_set(bt, bt->frame, spi)) | ||
| 392 | return 0; | ||
| 393 | bt_clear_frame_stack_arg_slot(bt, bt->frame, spi); | ||
| 394 | if (class == BPF_STX) | ||
| 395 | bt_set_reg(bt, sreg); | ||
| 396 | return 0; | ||
| 397 | } | ||
| 398 | |||
| 366 | /* scalars can only be spilled into stack */ | 399 | /* scalars can only be spilled into stack */ |
| 367 | if (!hist || !(hist->flags & INSN_F_STACK_ACCESS)) | 400 | if (!hist || !(hist->flags & INSN_F_STACK_ACCESS)) |
| 368 | return 0; | 401 | return 0; |
| 369 | spi = insn_stack_access_spi(hist->flags); | 402 | spi = hist->spi; |
| 370 | fr = insn_stack_access_frameno(hist->flags); | 403 | fr = hist->frame; |
| 371 | if (!bt_is_frame_slot_set(bt, fr, spi)) | 404 | if (!bt_is_frame_slot_set(bt, fr, spi)) |
| 372 | return 0; | 405 | return 0; |
| 373 | bt_clear_frame_slot(bt, fr, spi); | 406 | bt_clear_frame_slot(bt, fr, spi); |
| @@ -431,6 +464,12 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx, · 57 unmodified lines | |||
| 431 | bpf_bt_set_frame_reg(bt, bt->frame - 1, i); | 464 | bpf_bt_set_frame_reg(bt, bt->frame - 1, i); |
| 432 | } | 465 | } |
| 433 | } | 466 | } |
| 467 | if (bt_stack_arg_mask(bt)) { | ||
| 468 | verifier_bug(env, | ||
| 469 | "static subprog leftover stack arg slots %x", | ||
| 470 | bt_stack_arg_mask(bt)); | ||
| 471 | return -EFAULT; | ||
| 472 | } | ||
| 434 | if (bt_subprog_exit(bt)) | 473 | if (bt_subprog_exit(bt)) |
| 435 | return -EFAULT; | 474 | return -EFAULT; |
| 436 | return 0; | 475 | return 0; |
| @@ -901,6 +940,17 @@ int bpf_mark_chain_precision(struct bpf_verifier_env *env, · 464 unmodified lines | |||
| 901 | *changed = true; | 940 | *changed = true; |
| 902 | } | 941 | } |
| 903 | } | 942 | } |
| 943 | for (i = 0; i < func->out_stack_arg_cnt; i++) { | ||
| 944 | if (!bt_is_frame_stack_arg_slot_set(bt, fr, i)) | ||
| 945 | continue; | ||
| 946 | reg = &func->stack_arg_regs[i]; | ||
| 947 | if (reg->type != SCALAR_VALUE || reg->precise) { | ||
| 948 | bt_clear_frame_stack_arg_slot(bt, fr, i); | ||
| 949 | } else { | ||
| 950 | reg->precise = true; | ||
| 951 | *changed = true; | ||
| 952 | } | ||
| 953 | } | ||
| 904 | if (env->log.level & BPF_LOG_LEVEL2) { | 954 | if (env->log.level & BPF_LOG_LEVEL2) { |
| 905 | fmt_reg_mask(env->tmp_str_buf, TMP_STR_BUF_LEN, | 955 | fmt_reg_mask(env->tmp_str_buf, TMP_STR_BUF_LEN, |
| 906 | bt_frame_reg_mask(bt, fr)); | 956 | bt_frame_reg_mask(bt, fr)); |
| @@ -13,23 +13,8 @@ · 12 unmodified lines | |||
| 13 | #define PERCPU_FREE_TARGET (4) | 13 | #define PERCPU_FREE_TARGET (4) |
| 14 | #define PERCPU_NR_SCANS PERCPU_FREE_TARGET | 14 | #define PERCPU_NR_SCANS PERCPU_FREE_TARGET |
| 15 | 15 | ||
| 16 | /* Helpers to get the local list index */ | ||
| 17 | #define LOCAL_LIST_IDX(t) ((t) - BPF_LOCAL_LIST_T_OFFSET) | ||
| 18 | #define LOCAL_FREE_LIST_IDX LOCAL_LIST_IDX(BPF_LRU_LOCAL_LIST_T_FREE) | ||
| 19 | #define LOCAL_PENDING_LIST_IDX LOCAL_LIST_IDX(BPF_LRU_LOCAL_LIST_T_PENDING) | ||
| 20 | #define IS_LOCAL_LIST_TYPE(t) ((t) >= BPF_LOCAL_LIST_T_OFFSET) | 16 | #define IS_LOCAL_LIST_TYPE(t) ((t) >= BPF_LOCAL_LIST_T_OFFSET) |
| 21 | 17 | ||
| 22 | /* Local list helpers */ | ||
| 23 | static struct list_head *local_free_list(struct bpf_lru_locallist *loc_l) | ||
| 24 | { | ||
| 25 | return &loc_l->lists[LOCAL_FREE_LIST_IDX]; | ||
| 26 | } | ||
| 27 | |||
| 28 | static struct list_head *local_pending_list(struct bpf_lru_locallist *loc_l) | ||
| 29 | { | ||
| 30 | return &loc_l->lists[LOCAL_PENDING_LIST_IDX]; | ||
| 31 | } | ||
| 32 | |||
| 33 | /* bpf_lru_node helpers */ | 18 | /* bpf_lru_node helpers */ |
| 34 | static bool bpf_lru_node_is_ref(const struct bpf_lru_node *node) | 19 | static bool bpf_lru_node_is_ref(const struct bpf_lru_node *node) |
| 35 | { | 20 | { |
| @@ -72,6 +57,7 @@ static void __bpf_lru_node_move_to_free(struct bpf_lru_list *l, · 36 unmodified lines | |||
| 72 | bpf_lru_list_count_dec(l, node->type); | 57 | bpf_lru_list_count_dec(l, node->type); |
| 73 | 58 | ||
| 74 | node->type = tgt_free_type; | 59 | node->type = tgt_free_type; |
| 60 | WRITE_ONCE(node->pending_free, 0); | ||
| 75 | list_move(&node->list, free_list); | 61 | list_move(&node->list, free_list); |
| 76 | } | 62 | } |
| 77 | 63 | ||
| @@ -87,6 +73,9 @@ static void __bpf_lru_node_move_in(struct bpf_lru_list *l, · 9 unmodified lines | |||
| 87 | bpf_lru_list_count_inc(l, tgt_type); | 73 | bpf_lru_list_count_inc(l, tgt_type); |
| 88 | node->type = tgt_type; | 74 | node->type = tgt_type; |
| 89 | bpf_lru_node_clear_ref(node); | 75 | bpf_lru_node_clear_ref(node); |
| 76 | /* Reset pending_free only when moving to the free list */ | ||
| 77 | if (tgt_type == BPF_LRU_LIST_T_FREE) | ||
| 78 | WRITE_ONCE(node->pending_free, 0); | ||
| 90 | list_move(&node->list, &l->lists[tgt_type]); | 79 | list_move(&node->list, &l->lists[tgt_type]); |
| 91 | } | 80 | } |
| 92 | 81 | ||
| @@ -212,9 +201,11 @@ __bpf_lru_list_shrink_inactive(struct bpf_lru *lru, · 119 unmodified lines | |||
| 212 | unsigned int i = 0; | 201 | unsigned int i = 0; |
| 213 | 202 | ||
| 214 | list_for_each_entry_safe_reverse(node, tmp_node, inactive, list) { | 203 | list_for_each_entry_safe_reverse(node, tmp_node, inactive, list) { |
| 215 | if (bpf_lru_node_is_ref(node)) { | 204 | if (bpf_lru_node_is_ref(node) && |
| 205 | !READ_ONCE(node->pending_free)) { | ||
| 216 | __bpf_lru_node_move(l, node, BPF_LRU_LIST_T_ACTIVE); | 206 | __bpf_lru_node_move(l, node, BPF_LRU_LIST_T_ACTIVE); |
| 217 | } else if (lru->del_from_htab(lru->del_arg, node)) { | 207 | } else if (READ_ONCE(node->pending_free) || |
| 208 | lru->del_from_htab(lru->del_arg, node)) { | ||
| 218 | __bpf_lru_node_move_to_free(l, node, free_list, | 209 | __bpf_lru_node_move_to_free(l, node, free_list, |
| 219 | tgt_free_type); | 210 | tgt_free_type); |
| 220 | if (++nshrinked == tgt_nshrink) | 211 | if (++nshrinked == tgt_nshrink) |
| @@ -273,7 +264,8 @@ static unsigned int __bpf_lru_list_shrink(struct bpf_lru *lru, · 52 unmodified lines | |||
| 273 | 264 | ||
| 274 | list_for_each_entry_safe_reverse(node, tmp_node, force_shrink_list, | 265 | list_for_each_entry_safe_reverse(node, tmp_node, force_shrink_list, |
| 275 | list) { | 266 | list) { |
| 276 | if (lru->del_from_htab(lru->del_arg, node)) { | 267 | if (READ_ONCE(node->pending_free) || |
| 268 | lru->del_from_htab(lru->del_arg, node)) { | ||
| 277 | __bpf_lru_node_move_to_free(l, node, free_list, | 269 | __bpf_lru_node_move_to_free(l, node, free_list, |
| 278 | tgt_free_type); | 270 | tgt_free_type); |
| 279 | return 1; | 271 | return 1; |
| @@ -290,8 +282,10 @@ static void __local_list_flush(struct bpf_lru_list *l, · 10 unmodified lines | |||
| 290 | struct bpf_lru_node *node, *tmp_node; | 282 | struct bpf_lru_node *node, *tmp_node; |
| 291 | 283 | ||
| 292 | list_for_each_entry_safe_reverse(node, tmp_node, | 284 | list_for_each_entry_safe_reverse(node, tmp_node, |
| 293 | local_pending_list(loc_l), list) { | 285 | &loc_l->pending_list, list) { |
| 294 | if (bpf_lru_node_is_ref(node)) | 286 | if (READ_ONCE(node->pending_free)) |
| 287 | __bpf_lru_node_move_in(l, node, BPF_LRU_LIST_T_FREE); | ||
| 288 | else if (bpf_lru_node_is_ref(node)) | ||
| 295 | __bpf_lru_node_move_in(l, node, BPF_LRU_LIST_T_ACTIVE); | 289 | __bpf_lru_node_move_in(l, node, BPF_LRU_LIST_T_ACTIVE); |
| 296 | else | 290 | else |
| 297 | __bpf_lru_node_move_in(l, node, | 291 | __bpf_lru_node_move_in(l, node, |
| @@ -307,9 +301,12 @@ static void bpf_lru_list_push_free(struct bpf_lru_list *l, · 9 unmodified lines | |||
| 307 | if (WARN_ON_ONCE(IS_LOCAL_LIST_TYPE(node->type))) | 301 | if (WARN_ON_ONCE(IS_LOCAL_LIST_TYPE(node->type))) |
| 308 | return; | 302 | return; |
| 309 | 303 | ||
| 310 | raw_spin_lock_irqsave(&l->lock, flags); | 304 | if (raw_res_spin_lock_irqsave(&l->lock, flags)) { |
| 305 | WRITE_ONCE(node->pending_free, 1); | ||
| 306 | return; | ||
| 307 | } | ||
| 311 | __bpf_lru_node_move(l, node, BPF_LRU_LIST_T_FREE); | 308 | __bpf_lru_node_move(l, node, BPF_LRU_LIST_T_FREE); |
| 312 | raw_spin_unlock_irqrestore(&l->lock, flags); | 309 | raw_res_spin_unlock_irqrestore(&l->lock, flags); |
| 313 | } | 310 | } |
| 314 | 311 | ||
| 315 | static void bpf_lru_list_pop_free_to_local(struct bpf_lru *lru, | 312 | static void bpf_lru_list_pop_free_to_local(struct bpf_lru *lru, |
| @@ -318,8 +315,10 @@ static void bpf_lru_list_pop_free_to_local(struct bpf_lru *lru, · 2 unmodified lines | |||
| 318 | struct bpf_lru_list *l = &lru->common_lru.lru_list; | 315 | struct bpf_lru_list *l = &lru->common_lru.lru_list; |
| 319 | struct bpf_lru_node *node, *tmp_node; | 316 | struct bpf_lru_node *node, *tmp_node; |
| 320 | unsigned int nfree = 0; | 317 | unsigned int nfree = 0; |
| 318 | LIST_HEAD(tmp_free); | ||
| 321 | 319 | ||
| 322 | raw_spin_lock(&l->lock); | 320 | if (raw_res_spin_lock(&l->lock)) |
| 321 | return; | ||
| 323 | 322 | ||
| 324 | __local_list_flush(l, loc_l); | 323 | __local_list_flush(l, loc_l); |
| 325 | 324 | ||
| @@ -327,7 +326,7 @@ static void bpf_lru_list_pop_free_to_local(struct bpf_lru *lru, · 1 unmodified line | |||
| 327 | 326 | ||
| 328 | list_for_each_entry_safe(node, tmp_node, &l->lists[BPF_LRU_LIST_T_FREE], | 327 | list_for_each_entry_safe(node, tmp_node, &l->lists[BPF_LRU_LIST_T_FREE], |
| 329 | list) { | 328 | list) { |
| 330 | __bpf_lru_node_move_to_free(l, node, local_free_list(loc_l), | 329 | __bpf_lru_node_move_to_free(l, node, &tmp_free, |
| 331 | BPF_LRU_LOCAL_LIST_T_FREE); | 330 | BPF_LRU_LOCAL_LIST_T_FREE); |
| 332 | if (++nfree == lru->target_free) | 331 | if (++nfree == lru->target_free) |
| 333 | break; | 332 | break; |
| @@ -335,10 +334,19 @@ static void bpf_lru_list_pop_free_to_local(struct bpf_lru *lru, · 1 unmodified line | |||
| 335 | 334 | ||
| 336 | if (nfree < lru->target_free) | 335 | if (nfree < lru->target_free) |
| 337 | __bpf_lru_list_shrink(lru, l, lru->target_free - nfree, | 336 | __bpf_lru_list_shrink(lru, l, lru->target_free - nfree, |
| 338 | local_free_list(loc_l), | 337 | &tmp_free, |
| 339 | BPF_LRU_LOCAL_LIST_T_FREE); | 338 | BPF_LRU_LOCAL_LIST_T_FREE); |
| 340 | 339 | ||
| 341 | raw_spin_unlock(&l->lock); | 340 | raw_res_spin_unlock(&l->lock); |
| 341 | |||
| 342 | /* | ||
| 343 | * Transfer the harvested nodes from the temporary list_head into | ||
| 344 | * the lockless per-CPU free llist. | ||
| 345 | */ | ||
| 346 | list_for_each_entry_safe(node, tmp_node, &tmp_free, list) { | ||
| 347 | list_del(&node->list); | ||
| 348 | llist_add(&node->llist, &loc_l->free_llist); | ||
| 349 | } | ||
| 342 | } | 350 | } |
| 343 | 351 | ||
| 344 | static void __local_list_add_pending(struct bpf_lru *lru, | 352 | static void __local_list_add_pending(struct bpf_lru *lru, |
| @@ -350,22 +358,21 @@ static void __local_list_add_pending(struct bpf_lru *lru, · 5 unmodified lines | |||
| 350 | *(u32 *)((void *)node + lru->hash_offset) = hash; | 358 | *(u32 *)((void *)node + lru->hash_offset) = hash; |
| 351 | node->cpu = cpu; | 359 | node->cpu = cpu; |
| 352 | node->type = BPF_LRU_LOCAL_LIST_T_PENDING; | 360 | node->type = BPF_LRU_LOCAL_LIST_T_PENDING; |
| 361 | WRITE_ONCE(node->pending_free, 0); | ||
| 353 | bpf_lru_node_clear_ref(node); | 362 | bpf_lru_node_clear_ref(node); |
| 354 | list_add(&node->list, local_pending_list(loc_l)); | 363 | list_add(&node->list, &loc_l->pending_list); |
| 355 | } | 364 | } |
| 356 | 365 | ||
| 357 | static struct bpf_lru_node * | 366 | static struct bpf_lru_node * |
| 358 | __local_list_pop_free(struct bpf_lru_locallist *loc_l) | 367 | __local_list_pop_free(struct bpf_lru_locallist *loc_l) |
| 359 | { | 368 | { |
| 360 | struct bpf_lru_node *node; | 369 | struct llist_node *llnode; |
| 361 | 370 | ||
| 362 | node = list_first_entry_or_null(local_free_list(loc_l), | 371 | llnode = llist_del_first(&loc_l->free_llist); |
| 363 | struct bpf_lru_node, | 372 | if (!llnode) |
| 364 | list); | 373 | return NULL; |
| 365 | if (node) | ||
| 366 | list_del(&node->list); | ||
| 367 | 374 | ||
| 368 | return node; | 375 | return container_of(llnode, struct bpf_lru_node, llist); |
| 369 | } | 376 | } |
| 370 | 377 | ||
| 371 | static struct bpf_lru_node * | 378 | static struct bpf_lru_node * |
| @@ -376,10 +383,10 @@ __local_list_pop_pending(struct bpf_lru *lru, struct bpf_lru_locallist *loc_l) · 4 unmodified lines | |||
| 376 | 383 | ||
| 377 | ignore_ref: | 384 | ignore_ref: |
| 378 | /* Get from the tail (i.e. older element) of the pending list. */ | 385 | /* Get from the tail (i.e. older element) of the pending list. */ |
| 379 | list_for_each_entry_reverse(node, local_pending_list(loc_l), | 386 | list_for_each_entry_reverse(node, &loc_l->pending_list, list) { |
| 380 | list) { | ||
| 381 | if ((!bpf_lru_node_is_ref(node) || force) && | 387 | if ((!bpf_lru_node_is_ref(node) || force) && |
| 382 | lru->del_from_htab(lru->del_arg, node)) { | 388 | (READ_ONCE(node->pending_free) || |
| 389 | lru->del_from_htab(lru->del_arg, node))) { | ||
| 383 | list_del(&node->list); | 390 | list_del(&node->list); |
| 384 | return node; | 391 | return node; |
| 385 | } | 392 | } |
| @@ -404,7 +411,8 @@ static struct bpf_lru_node *bpf_percpu_lru_pop_free(struct bpf_lru *lru, · 18 unmodified lines | |||
| 404 | 411 | ||
| 405 | l = per_cpu_ptr(lru->percpu_lru, cpu); | 412 | l = per_cpu_ptr(lru->percpu_lru, cpu); |
| 406 | 413 | ||
| 407 | raw_spin_lock_irqsave(&l->lock, flags); | 414 | if (raw_res_spin_lock_irqsave(&l->lock, flags)) |
| 415 | return NULL; | ||
| 408 | 416 | ||
| 409 | __bpf_lru_list_rotate(lru, l); | 417 | __bpf_lru_list_rotate(lru, l); |
| 410 | 418 | ||
| @@ -420,7 +428,7 @@ static struct bpf_lru_node *bpf_percpu_lru_pop_free(struct bpf_lru *lru, · 9 unmodified lines | |||
| 420 | __bpf_lru_node_move(l, node, BPF_LRU_LIST_T_INACTIVE); | 428 | __bpf_lru_node_move(l, node, BPF_LRU_LIST_T_INACTIVE); |
| 421 | } | 429 | } |
| 422 | 430 | ||
| 423 | raw_spin_unlock_irqrestore(&l->lock, flags); | 431 | raw_res_spin_unlock_irqrestore(&l->lock, flags); |
| 424 | 432 | ||
| 425 | return node; | 433 | return node; |
| 426 | } | 434 | } |
| @@ -437,7 +445,8 @@ static struct bpf_lru_node *bpf_common_lru_pop_free(struct bpf_lru *lru, · 10 unmodified lines | |||
| 437 | 445 | ||
| 438 | loc_l = per_cpu_ptr(clru->local_list, cpu); | 446 | loc_l = per_cpu_ptr(clru->local_list, cpu); |
| 439 | 447 | ||
| 440 | raw_spin_lock_irqsave(&loc_l->lock, flags); | 448 | if (raw_res_spin_lock_irqsave(&loc_l->lock, flags)) |
| 449 | return NULL; | ||
| 441 | 450 | ||
| 442 | node = __local_list_pop_free(loc_l); | 451 | node = __local_list_pop_free(loc_l); |
| 443 | if (!node) { | 452 | if (!node) { |
| @@ -448,17 +457,22 @@ static struct bpf_lru_node *bpf_common_lru_pop_free(struct bpf_lru *lru, · 4 unmodified lines | |||
| 448 | if (node) | 457 | if (node) |
| 449 | __local_list_add_pending(lru, loc_l, cpu, node, hash); | 458 | __local_list_add_pending(lru, loc_l, cpu, node, hash); |
| 450 | 459 | ||
| 451 | raw_spin_unlock_irqrestore(&loc_l->lock, flags); | 460 | raw_res_spin_unlock_irqrestore(&loc_l->lock, flags); |
| 452 | 461 | ||
| 453 | if (node) | 462 | if (node) |
| 454 | return node; | 463 | return node; |
| 455 | 464 | ||
| 456 | /* No free nodes found from the local free list and | 465 | /* |
| 466 | * No free nodes found from the local free list and | ||
| 457 | * the global LRU list. | 467 | * the global LRU list. |
| 458 | * | 468 | * |
| 459 | * Steal from the local free/pending list of the | 469 | * Steal from the local free/pending list of the |
| 460 | * current CPU and remote CPU in RR. It starts | 470 | * current CPU and remote CPU in RR. It starts |
| 461 | * with the loc_l->next_steal CPU. | 471 | * with the loc_l->next_steal CPU. |
| 472 | * | ||
| 473 | * Acquire the victim's lock before touching either list. On | ||
| 474 | * acquisition failure (rqspinlock AA or timeout) skip the victim | ||
| 475 | * and try the next CPU. | ||
| 462 | */ | 476 | */ |
| 463 | 477 | ||
| 464 | first_steal = loc_l->next_steal; | 478 | first_steal = loc_l->next_steal; |
| @@ -466,24 +480,36 @@ static struct bpf_lru_node *bpf_common_lru_pop_free(struct bpf_lru *lru, · 1 unmodified line | |||
| 466 | do { | 480 | do { |
| 467 | steal_loc_l = per_cpu_ptr(clru->local_list, steal); | 481 | steal_loc_l = per_cpu_ptr(clru->local_list, steal); |
| 468 | 482 | ||
| 469 | raw_spin_lock_irqsave(&steal_loc_l->lock, flags); | 483 | if (!raw_res_spin_lock_irqsave(&steal_loc_l->lock, flags)) { |
| 470 | 484 | node = __local_list_pop_free(steal_loc_l); | |
| 471 | node = __local_list_pop_free(steal_loc_l); | 485 | if (!node) |
| 472 | if (!node) | 486 | node = __local_list_pop_pending(lru, steal_loc_l); |
| 473 | node = __local_list_pop_pending(lru, steal_loc_l); | 487 | raw_res_spin_unlock_irqrestore(&steal_loc_l->lock, flags); |
| 474 | 488 | } | |
| 475 | raw_spin_unlock_irqrestore(&steal_loc_l->lock, flags); | ||
| 476 | 489 | ||
| 477 | steal = cpumask_next_wrap(steal, cpu_possible_mask); | 490 | steal = cpumask_next_wrap(steal, cpu_possible_mask); |
| 478 | } while (!node && steal != first_steal); | 491 | } while (!node && steal != first_steal); |
| 479 | 492 | ||
| 480 | loc_l->next_steal = steal; | 493 | loc_l->next_steal = steal; |
| 481 | 494 | ||
| 482 | if (node) { | 495 | if (!node) |
| 483 | raw_spin_lock_irqsave(&loc_l->lock, flags); | 496 | return NULL; |
| 484 | __local_list_add_pending(lru, loc_l, cpu, node, hash); | 497 | |
| 485 | raw_spin_unlock_irqrestore(&loc_l->lock, flags); | 498 | if (raw_res_spin_lock_irqsave(&loc_l->lock, flags)) { |
| 499 | /* | ||
| 500 | * The local pending lock can't be acquired (rqspinlock AA | ||
| 501 | * or timeout). Return the stolen node to the per-CPU | ||
| 502 | * free_llist instead of orphaning it; the next pop_free on | ||
| 503 | * this CPU will pick it up. | ||
| 504 | */ | ||
| 505 | node->type = BPF_LRU_LOCAL_LIST_T_FREE; | ||
| 506 | bpf_lru_node_clear_ref(node); | ||
| 507 | WRITE_ONCE(node->pending_free, 0); | ||
| 508 | llist_add(&node->llist, &loc_l->free_llist); | ||
| 509 | return NULL; | ||
| 486 | } | 510 | } |
| 511 | __local_list_add_pending(lru, loc_l, cpu, node, hash); | ||
| 512 | raw_res_spin_unlock_irqrestore(&loc_l->lock, flags); | ||
| 487 | 513 | ||
| 488 | return node; | 514 | return node; |
| 489 | } | 515 | } |
| @@ -511,18 +537,24 @@ static void bpf_common_lru_push_free(struct bpf_lru *lru, · 21 unmodified lines | |||
| 511 | 537 | ||
| 512 | loc_l = per_cpu_ptr(lru->common_lru.local_list, node->cpu); | 538 | loc_l = per_cpu_ptr(lru->common_lru.local_list, node->cpu); |
| 513 | 539 | ||
| 514 | raw_spin_lock_irqsave(&loc_l->lock, flags); | 540 | if (raw_res_spin_lock_irqsave(&loc_l->lock, flags)) { |
| 541 | WRITE_ONCE(node->pending_free, 1); | ||
| 542 | return; | ||
| 543 | } | ||
| 515 | 544 | ||
| 516 | if (unlikely(node->type != BPF_LRU_LOCAL_LIST_T_PENDING)) { | 545 | if (unlikely(node->type != BPF_LRU_LOCAL_LIST_T_PENDING)) { |
| 517 | raw_spin_unlock_irqrestore(&loc_l->lock, flags); | 546 | raw_res_spin_unlock_irqrestore(&loc_l->lock, |
| 547 | flags); | ||
| 518 | goto check_lru_list; | 548 | goto check_lru_list; |
| 519 | } | 549 | } |
| 520 | 550 | ||
| 521 | node->type = BPF_LRU_LOCAL_LIST_T_FREE; | 551 | node->type = BPF_LRU_LOCAL_LIST_T_FREE; |
| 522 | bpf_lru_node_clear_ref(node); | 552 | bpf_lru_node_clear_ref(node); |
| 523 | list_move(&node->list, local_free_list(loc_l)); | 553 | list_del(&node->list); |
| 554 | |||
| 555 | raw_res_spin_unlock_irqrestore(&loc_l->lock, flags); | ||
| 524 | 556 | ||
| 525 | raw_spin_unlock_irqrestore(&loc_l->lock, flags); | 557 | llist_add(&node->llist, &loc_l->free_llist); |
| 526 | return; | 558 | return; |
| 527 | } | 559 | } |
| 528 | 560 | ||
| @@ -538,11 +570,14 @@ static void bpf_percpu_lru_push_free(struct bpf_lru *lru, · 9 unmodified lines | |||
| 538 | 570 | ||
| 539 | l = per_cpu_ptr(lru->percpu_lru, node->cpu); | 571 | l = per_cpu_ptr(lru->percpu_lru, node->cpu); |
| 540 | 572 | ||
| 541 | raw_spin_lock_irqsave(&l->lock, flags); | 573 | if (raw_res_spin_lock_irqsave(&l->lock, flags)) { |
| 574 | WRITE_ONCE(node->pending_free, 1); | ||
| 575 | return; | ||
| 576 | } | ||
| 542 | 577 | ||
| 543 | __bpf_lru_node_move(l, node, BPF_LRU_LIST_T_FREE); | 578 | __bpf_lru_node_move(l, node, BPF_LRU_LIST_T_FREE); |
| 544 | 579 | ||
| 545 | raw_spin_unlock_irqrestore(&l->lock, flags); | 580 | raw_res_spin_unlock_irqrestore(&l->lock, flags); |
| 546 | } | 581 | } |
| 547 | 582 | ||
| 548 | void bpf_lru_push_free(struct bpf_lru *lru, struct bpf_lru_node *node) | 583 | void bpf_lru_push_free(struct bpf_lru *lru, struct bpf_lru_node *node) |
| @@ -565,6 +600,7 @@ static void bpf_common_lru_populate(struct bpf_lru *lru, void *buf, · 16 unmodified lines | |||
| 565 | 600 | ||
| 566 | node = (struct bpf_lru_node *)(buf + node_offset); | 601 | node = (struct bpf_lru_node *)(buf + node_offset); |
| 567 | node->type = BPF_LRU_LIST_T_FREE; | 602 | node->type = BPF_LRU_LIST_T_FREE; |
| 603 | node->pending_free = 0; | ||
| 568 | bpf_lru_node_clear_ref(node); | 604 | bpf_lru_node_clear_ref(node); |
| 569 | list_add(&node->list, &l->lists[BPF_LRU_LIST_T_FREE]); | 605 | list_add(&node->list, &l->lists[BPF_LRU_LIST_T_FREE]); |
| 570 | buf += elem_size; | 606 | buf += elem_size; |
| @@ -594,6 +630,7 @@ again: · 23 unmodified lines | |||
| 594 | node = (struct bpf_lru_node *)(buf + node_offset); | 630 | node = (struct bpf_lru_node *)(buf + node_offset); |
| 595 | node->cpu = cpu; | 631 | node->cpu = cpu; |
| 596 | node->type = BPF_LRU_LIST_T_FREE; | 632 | node->type = BPF_LRU_LIST_T_FREE; |
| 633 | node->pending_free = 0; | ||
| 597 | bpf_lru_node_clear_ref(node); | 634 | bpf_lru_node_clear_ref(node); |
| 598 | list_add(&node->list, &l->lists[BPF_LRU_LIST_T_FREE]); | 635 | list_add(&node->list, &l->lists[BPF_LRU_LIST_T_FREE]); |
| 599 | i++; | 636 | i++; |
| @@ -618,14 +655,12 @@ void bpf_lru_populate(struct bpf_lru *lru, void *buf, u32 node_offset, · 18 unmodified lines | |||
| 618 | 655 | ||
| 619 | static void bpf_lru_locallist_init(struct bpf_lru_locallist *loc_l, int cpu) | 656 | static void bpf_lru_locallist_init(struct bpf_lru_locallist *loc_l, int cpu) |
| 620 | { | 657 | { |
| 621 | int i; | 658 | INIT_LIST_HEAD(&loc_l->pending_list); |
| 622 | 659 | init_llist_head(&loc_l->free_llist); | |
| 623 | for (i = 0; i < NR_BPF_LRU_LOCAL_LIST_T; i++) | ||
| 624 | INIT_LIST_HEAD(&loc_l->lists[i]); | ||
| 625 | 660 | ||
| 626 | loc_l->next_steal = cpu; | 661 | loc_l->next_steal = cpu; |
| 627 | 662 | ||
| 628 | raw_spin_lock_init(&loc_l->lock); | 663 | raw_res_spin_lock_init(&loc_l->lock); |
| 629 | } | 664 | } |
| 630 | 665 | ||
| 631 | static void bpf_lru_list_init(struct bpf_lru_list *l) | 666 | static void bpf_lru_list_init(struct bpf_lru_list *l) |
| @@ -640,7 +675,7 @@ static void bpf_lru_list_init(struct bpf_lru_list *l) · 8 unmodified lines | |||
| 640 | 675 | ||
| 641 | l->next_inactive_rotation = &l->lists[BPF_LRU_LIST_T_INACTIVE]; | 676 | l->next_inactive_rotation = &l->lists[BPF_LRU_LIST_T_INACTIVE]; |
| 642 | 677 | ||
| 643 | raw_spin_lock_init(&l->lock); | 678 | raw_res_spin_lock_init(&l->lock); |
| 644 | } | 679 | } |
| 645 | 680 | ||
| 646 | int bpf_lru_init(struct bpf_lru *lru, bool percpu, u32 hash_offset, | 681 | int bpf_lru_init(struct bpf_lru *lru, bool percpu, u32 hash_offset, |
| @@ -6,11 +6,11 @@ · 5 unmodified lines | |||
| 6 | 6 | ||
| 7 | #include <linux/cache.h> | 7 | #include <linux/cache.h> |
| 8 | #include <linux/list.h> | 8 | #include <linux/list.h> |
| 9 | #include <linux/spinlock_types.h> | 9 | #include <linux/llist.h> |
| 10 | #include <asm/rqspinlock.h> | ||
| 10 | 11 | ||
| 11 | #define NR_BPF_LRU_LIST_T (3) | 12 | #define NR_BPF_LRU_LIST_T (3) |
| 12 | #define NR_BPF_LRU_LIST_COUNT (2) | 13 | #define NR_BPF_LRU_LIST_COUNT (2) |
| 13 | #define NR_BPF_LRU_LOCAL_LIST_T (2) | ||
| 14 | #define BPF_LOCAL_LIST_T_OFFSET NR_BPF_LRU_LIST_T | 14 | #define BPF_LOCAL_LIST_T_OFFSET NR_BPF_LRU_LIST_T |
| 15 | 15 | ||
| 16 | enum bpf_lru_list_type { | 16 | enum bpf_lru_list_type { |
| @@ -22,10 +22,22 @@ enum bpf_lru_list_type { · 5 unmodified lines | |||
| 22 | }; | 22 | }; |
| 23 | 23 | ||
| 24 | struct bpf_lru_node { | 24 | struct bpf_lru_node { |
| 25 | struct list_head list; | 25 | /* |
| 26 | * A node is in at most one list at a time. The free path on the | ||
| 27 | * per-CPU locallist uses an llist, so share storage via a union. | ||
| 28 | */ | ||
| 29 | union { | ||
| 30 | struct list_head list; | ||
| 31 | struct llist_node llist; | ||
| 32 | }; | ||
| 26 | u16 cpu; | 33 | u16 cpu; |
| 27 | u8 type; | 34 | u8 type; |
| 28 | u8 ref; | 35 | u8 ref; |
| 36 | /* | ||
| 37 | * Marks nodes whose *_push_free() lock acquire failed; reclaimed | ||
| 38 | * by flush/shrink which honor the flag instead of del_from_htab(). | ||
| 39 | */ | ||
| 40 | u8 pending_free; | ||
| 29 | }; | 41 | }; |
| 30 | 42 | ||
| 31 | struct bpf_lru_list { | 43 | struct bpf_lru_list { |
| @@ -34,13 +46,14 @@ struct bpf_lru_list { · 2 unmodified lines | |||
| 34 | /* The next inactive list rotation starts from here */ | 46 | /* The next inactive list rotation starts from here */ |
| 35 | struct list_head *next_inactive_rotation; | 47 | struct list_head *next_inactive_rotation; |
| 36 | 48 | ||
| 37 | raw_spinlock_t lock ____cacheline_aligned_in_smp; | 49 | rqspinlock_t lock ____cacheline_aligned_in_smp; |
| 38 | }; | 50 | }; |
| 39 | 51 | ||
| 40 | struct bpf_lru_locallist { | 52 | struct bpf_lru_locallist { |
| 41 | struct list_head lists[NR_BPF_LRU_LOCAL_LIST_T]; | 53 | struct list_head pending_list; |
| 54 | struct llist_head free_llist; | ||
| 42 | u16 next_steal; | 55 | u16 next_steal; |
| 43 | raw_spinlock_t lock; | 56 | rqspinlock_t lock; |
| 44 | }; | 57 | }; |
| 45 | 58 | ||
| 46 | struct bpf_common_lru { | 59 | struct bpf_common_lru { |
| @@ -427,6 +427,26 @@ BTF_ID(func, bpf_lsm_audit_rule_known) · 426 unmodified lines | |||
| 427 | BTF_ID(func, bpf_lsm_inode_xattr_skipcap) | 427 | BTF_ID(func, bpf_lsm_inode_xattr_skipcap) |
| 428 | BTF_SET_END(bool_lsm_hooks) | 428 | BTF_SET_END(bool_lsm_hooks) |
| 429 | 429 | ||
| 430 | /* hooks returning void */ | ||
| 431 | #define LSM_HOOK_void(DEFAULT, NAME, ...) BTF_ID(func, bpf_lsm_##NAME) | ||
| 432 | #define LSM_HOOK_int(DEFAULT, NAME, ...) /* nothing */ | ||
| 433 | #define LSM_HOOK(RET, DEFAULT, NAME, ...) LSM_HOOK_##RET(DEFAULT, NAME, __VA_ARGS__) | ||
| 434 | BTF_SET_START(void_lsm_hooks) | ||
| 435 | #include <linux/lsm_hook_defs.h> | ||
| 436 | #undef LSM_HOOK | ||
| 437 | #undef LSM_HOOK_void | ||
| 438 | #undef LSM_HOOK_int | ||
| 439 | BTF_SET_END(void_lsm_hooks) | ||
| 440 | |||
| 441 | bool bpf_lsm_hook_returns_errno(u32 btf_id) | ||
| 442 | { | ||
| 443 | if (btf_id_set_contains(&bool_lsm_hooks, btf_id)) | ||
| 444 | return false; | ||
| 445 | if (btf_id_set_contains(&void_lsm_hooks, btf_id)) | ||
| 446 | return false; | ||
| 447 | return true; | ||
| 448 | } | ||
| 449 | |||
| 430 | int bpf_lsm_get_retval_range(const struct bpf_prog *prog, | 450 | int bpf_lsm_get_retval_range(const struct bpf_prog *prog, |
| 431 | struct bpf_retval_range *retval_range) | 451 | struct bpf_retval_range *retval_range) |
| 432 | { | 452 | { |
| @@ -594,8 +594,8 @@ const struct bpf_link_ops bpf_struct_ops_link_lops = { · 593 unmodified lines | |||
| 594 | .dealloc = bpf_struct_ops_link_dealloc, | 594 | .dealloc = bpf_struct_ops_link_dealloc, |
| 595 | }; | 595 | }; |
| 596 | 596 | ||
| 597 | int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_links *tlinks, | 597 | int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_nodes *tnodes, |
| 598 | struct bpf_tramp_link *link, | 598 | struct bpf_tramp_node *node, |
| 599 | const struct btf_func_model *model, | 599 | const struct btf_func_model *model, |
| 600 | void *stub_func, | 600 | void *stub_func, |
| 601 | void **_image, u32 *_image_off, | 601 | void **_image, u32 *_image_off, |
| @@ -605,13 +605,13 @@ int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_links *tlinks, · 3 unmodified lines | |||
| 605 | void *image = *_image; | 605 | void *image = *_image; |
| 606 | int size; | 606 | int size; |
| 607 | 607 | ||
| 608 | tlinks[BPF_TRAMP_FENTRY].links[0] = link; | 608 | tnodes[BPF_TRAMP_FENTRY].nodes[0] = node; |
| 609 | tlinks[BPF_TRAMP_FENTRY].nr_links = 1; | 609 | tnodes[BPF_TRAMP_FENTRY].nr_nodes = 1; |
| 610 | 610 | ||
| 611 | if (model->ret_size > 0) | 611 | if (model->ret_size > 0) |
| 612 | flags |= BPF_TRAMP_F_RET_FENTRY_RET; | 612 | flags |= BPF_TRAMP_F_RET_FENTRY_RET; |
| 613 | 613 | ||
| 614 | size = arch_bpf_trampoline_size(model, flags, tlinks, stub_func); | 614 | size = arch_bpf_trampoline_size(model, flags, tnodes, stub_func); |
| 615 | if (size <= 0) | 615 | if (size <= 0) |
| 616 | return size ? : -EFAULT; | 616 | return size ? : -EFAULT; |
| 617 | 617 | ||
| @@ -628,7 +628,7 @@ int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_links *tlinks, · 10 unmodified lines | |||
| 628 | 628 | ||
| 629 | size = arch_prepare_bpf_trampoline(NULL, image + image_off, | 629 | size = arch_prepare_bpf_trampoline(NULL, image + image_off, |
| 630 | image + image_off + size, | 630 | image + image_off + size, |
| 631 | model, flags, tlinks, stub_func); | 631 | model, flags, tnodes, stub_func); |
| 632 | if (size <= 0) { | 632 | if (size <= 0) { |
| 633 | if (image != *_image) | 633 | if (image != *_image) |
| 634 | bpf_struct_ops_image_free(image); | 634 | bpf_struct_ops_image_free(image); |
| @@ -693,7 +693,7 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, · 58 unmodified lines | |||
| 693 | const struct btf_type *module_type; | 693 | const struct btf_type *module_type; |
| 694 | const struct btf_member *member; | 694 | const struct btf_member *member; |
| 695 | const struct btf_type *t = st_ops_desc->type; | 695 | const struct btf_type *t = st_ops_desc->type; |
| 696 | struct bpf_tramp_links *tlinks; | 696 | struct bpf_tramp_nodes *tnodes; |
| 697 | void *udata, *kdata; | 697 | void *udata, *kdata; |
| 698 | int prog_fd, err; | 698 | int prog_fd, err; |
| 699 | u32 i, trampoline_start, image_off = 0; | 699 | u32 i, trampoline_start, image_off = 0; |
| @@ -720,8 +720,8 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, · 20 unmodified lines | |||
| 720 | if (uvalue->common.state || refcount_read(&uvalue->common.refcnt)) | 720 | if (uvalue->common.state || refcount_read(&uvalue->common.refcnt)) |
| 721 | return -EINVAL; | 721 | return -EINVAL; |
| 722 | 722 | ||
| 723 | tlinks = kzalloc_objs(*tlinks, BPF_TRAMP_MAX); | 723 | tnodes = kzalloc_objs(*tnodes, BPF_TRAMP_MAX); |
| 724 | if (!tlinks) | 724 | if (!tnodes) |
| 725 | return -ENOMEM; | 725 | return -ENOMEM; |
| 726 | 726 | ||
| 727 | uvalue = (struct bpf_struct_ops_value *)st_map->uvalue; | 727 | uvalue = (struct bpf_struct_ops_value *)st_map->uvalue; |
| @@ -817,8 +817,9 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, · 89 unmodified lines | |||
| 817 | err = -ENOMEM; | 817 | err = -ENOMEM; |
| 818 | goto reset_unlock; | 818 | goto reset_unlock; |
| 819 | } | 819 | } |
| 820 | bpf_link_init(&link->link, BPF_LINK_TYPE_STRUCT_OPS, | 820 | bpf_tramp_link_init(link, BPF_LINK_TYPE_STRUCT_OPS, |
| 821 | &bpf_struct_ops_link_lops, prog, prog->expected_attach_type); | 821 | &bpf_struct_ops_link_lops, prog, prog->expected_attach_type, 0); |
| 822 | |||
| 822 | *plink++ = &link->link; | 823 | *plink++ = &link->link; |
| 823 | 824 | ||
| 824 | /* Poison pointer on error instead of return for backward compatibility */ | 825 | /* Poison pointer on error instead of return for backward compatibility */ |
| @@ -832,7 +833,7 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, · 7 unmodified lines | |||
| 832 | *pksym++ = ksym; | 833 | *pksym++ = ksym; |
| 833 | 834 | ||
| 834 | trampoline_start = image_off; | 835 | trampoline_start = image_off; |
| 835 | err = bpf_struct_ops_prepare_trampoline(tlinks, link, | 836 | err = bpf_struct_ops_prepare_trampoline(tnodes, &link->node, |
| 836 | &st_ops->func_models[i], | 837 | &st_ops->func_models[i], |
| 837 | *(void **)(st_ops->cfi_stubs + moff), | 838 | *(void **)(st_ops->cfi_stubs + moff), |
| 838 | &image, &image_off, | 839 | &image, &image_off, |
| @@ -911,7 +912,7 @@ reset_unlock: · 72 unmodified lines | |||
| 911 | memset(uvalue, 0, map->value_size); | 912 | memset(uvalue, 0, map->value_size); |
| 912 | memset(kvalue, 0, map->value_size); | 913 | memset(kvalue, 0, map->value_size); |
| 913 | unlock: | 914 | unlock: |
| 914 | kfree(tlinks); | 915 | kfree(tnodes); |
| 915 | mutex_unlock(&st_map->lock); | 916 | mutex_unlock(&st_map->lock); |
| 916 | if (!err) | 917 | if (!err) |
| 917 | bpf_struct_ops_map_add_ksyms(st_map); | 918 | bpf_struct_ops_map_add_ksyms(st_map); |
| @@ -1204,6 +1205,42 @@ u32 bpf_struct_ops_id(const void *kdata) · 286 unmodified lines | |||
| 1204 | } | 1205 | } |
| 1205 | EXPORT_SYMBOL_GPL(bpf_struct_ops_id); | 1206 | EXPORT_SYMBOL_GPL(bpf_struct_ops_id); |
| 1206 | 1207 | ||
| 1208 | /** | ||
| 1209 | * bpf_struct_ops_for_each_prog - Invoke @cb for each member prog | ||
| 1210 | * @kdata: kernel-side struct_ops vmtable (the @kdata arg to ->reg/->update/->unreg) | ||
| 1211 | * @cb: callback invoked once per member prog; non-zero return stops iteration | ||
| 1212 | * @data: opaque argument passed to @cb | ||
| 1213 | * | ||
| 1214 | * Walks the struct_ops member progs registered on the map containing @kdata. | ||
| 1215 | * Intended for use from struct_ops ->reg() callbacks (and similar) that need to | ||
| 1216 | * inspect the loaded BPF programs (for example to discover maps they reference | ||
| 1217 | * via @prog->aux->used_maps). | ||
| 1218 | * | ||
| 1219 | * Return 0 if iteration completed, otherwise the first non-zero @cb return. | ||
| 1220 | */ | ||
| 1221 | int bpf_struct_ops_for_each_prog(const void *kdata, | ||
| 1222 | int (*cb)(struct bpf_prog *prog, void *data), | ||
| 1223 | void *data) | ||
| 1224 | { | ||
| 1225 | struct bpf_struct_ops_value *kvalue; | ||
| 1226 | struct bpf_struct_ops_map *st_map; | ||
| 1227 | u32 i; | ||
| 1228 | int ret; | ||
| 1229 | |||
| 1230 | kvalue = container_of(kdata, struct bpf_struct_ops_value, data); | ||
| 1231 | st_map = container_of(kvalue, struct bpf_struct_ops_map, kvalue); | ||
| 1232 | |||
| 1233 | for (i = 0; i < st_map->funcs_cnt; i++) { | ||
| 1234 | if (!st_map->links[i]) | ||
| 1235 | continue; | ||
| 1236 | ret = cb(st_map->links[i]->prog, data); | ||
| 1237 | if (ret) | ||
| 1238 | return ret; | ||
| 1239 | } | ||
| 1240 | return 0; | ||
| 1241 | } | ||
| 1242 | EXPORT_SYMBOL_GPL(bpf_struct_ops_for_each_prog); | ||
| 1243 | |||
| 1207 | static bool bpf_struct_ops_valid_to_reg(struct bpf_map *map) | 1244 | static bool bpf_struct_ops_valid_to_reg(struct bpf_map *map) |
| 1208 | { | 1245 | { |
| 1209 | struct bpf_struct_ops_map *st_map = (struct bpf_struct_ops_map *)map; | 1246 | struct bpf_struct_ops_map *st_map = (struct bpf_struct_ops_map *)map; |
| @@ -182,7 +182,6 @@ · 181 unmodified lines | |||
| 182 | #define BITS_ROUNDUP_BYTES(bits) \ | 182 | #define BITS_ROUNDUP_BYTES(bits) \ |
| 183 | (BITS_ROUNDDOWN_BYTES(bits) + !!BITS_PER_BYTE_MASKED(bits)) | 183 | (BITS_ROUNDDOWN_BYTES(bits) + !!BITS_PER_BYTE_MASKED(bits)) |
| 184 | 184 | ||
| 185 | #define BTF_INFO_MASK 0x9f00ffff | ||
| 186 | #define BTF_INT_MASK 0x0fffffff | 185 | #define BTF_INT_MASK 0x0fffffff |
| 187 | #define BTF_TYPE_ID_VALID(type_id) ((type_id) <= BTF_MAX_TYPE) | 186 | #define BTF_TYPE_ID_VALID(type_id) ((type_id) <= BTF_MAX_TYPE) |
| 188 | #define BTF_STR_OFFSET_VALID(name_off) ((name_off) <= BTF_MAX_NAME_OFFSET) | 187 | #define BTF_STR_OFFSET_VALID(name_off) ((name_off) <= BTF_MAX_NAME_OFFSET) |
| @@ -289,7 +288,7 @@ enum verifier_phase { · 100 unmodified lines | |||
| 289 | struct resolve_vertex { | 288 | struct resolve_vertex { |
| 290 | const struct btf_type *t; | 289 | const struct btf_type *t; |
| 291 | u32 type_id; | 290 | u32 type_id; |
| 292 | u16 next_member; | 291 | u32 next_member; |
| 293 | }; | 292 | }; |
| 294 | 293 | ||
| 295 | enum visit_state { | 294 | enum visit_state { |
| @@ -2031,7 +2030,7 @@ static int env_stack_push(struct btf_verifier_env *env, · 1735 unmodified lines | |||
| 2031 | } | 2030 | } |
| 2032 | 2031 | ||
| 2033 | static void env_stack_set_next_member(struct btf_verifier_env *env, | 2032 | static void env_stack_set_next_member(struct btf_verifier_env *env, |
| 2034 | u16 next_member) | 2033 | u32 next_member) |
| 2035 | { | 2034 | { |
| 2036 | env->stack[env->top_stack - 1].next_member = next_member; | 2035 | env->stack[env->top_stack - 1].next_member = next_member; |
| 2037 | } | 2036 | } |
| @@ -3293,7 +3292,7 @@ static s32 btf_struct_check_meta(struct btf_verifier_env *env, · 1255 unmodified lines | |||
| 3293 | struct btf *btf = env->btf; | 3292 | struct btf *btf = env->btf; |
| 3294 | u32 struct_size = t->size; | 3293 | u32 struct_size = t->size; |
| 3295 | u32 offset; | 3294 | u32 offset; |
| 3296 | u16 i; | 3295 | u32 i; |
| 3297 | 3296 | ||
| 3298 | meta_needed = btf_type_vlen(t) * sizeof(*member); | 3297 | meta_needed = btf_type_vlen(t) * sizeof(*member); |
| 3299 | if (meta_left < meta_needed) { | 3298 | if (meta_left < meta_needed) { |
| @@ -3369,7 +3368,7 @@ static int btf_struct_resolve(struct btf_verifier_env *env, · 69 unmodified lines | |||
| 3369 | { | 3368 | { |
| 3370 | const struct btf_member *member; | 3369 | const struct btf_member *member; |
| 3371 | int err; | 3370 | int err; |
| 3372 | u16 i; | 3371 | u32 i; |
| 3373 | 3372 | ||
| 3374 | /* Before continue resolving the next_member, | 3373 | /* Before continue resolving the next_member, |
| 3375 | * ensure the last member is indeed resolved to a | 3374 | * ensure the last member is indeed resolved to a |
| @@ -3668,7 +3667,7 @@ end: · 292 unmodified lines | |||
| 3668 | static int btf_repeat_fields(struct btf_field_info *info, int info_cnt, | 3667 | static int btf_repeat_fields(struct btf_field_info *info, int info_cnt, |
| 3669 | u32 field_cnt, u32 repeat_cnt, u32 elem_size) | 3668 | u32 field_cnt, u32 repeat_cnt, u32 elem_size) |
| 3670 | { | 3669 | { |
| 3671 | u32 i, j; | 3670 | u32 i, j, total_cnt, total_repeats; |
| 3672 | u32 cur; | 3671 | u32 cur; |
| 3673 | 3672 | ||
| 3674 | /* Ensure not repeating fields that should not be repeated. */ | 3673 | /* Ensure not repeating fields that should not be repeated. */ |
| @@ -3686,10 +3685,9 @@ static int btf_repeat_fields(struct btf_field_info *info, int info_cnt, · 11 unmodified lines | |||
| 3686 | } | 3685 | } |
| 3687 | } | 3686 | } |
| 3688 | 3687 | ||
| 3689 | /* The type of struct size or variable size is u32, | 3688 | if (check_add_overflow(repeat_cnt, 1, &total_repeats) || |
| 3690 | * so the multiplication will not overflow. | 3689 | check_mul_overflow(field_cnt, total_repeats, &total_cnt) || |
| 3691 | */ | 3690 | total_cnt > (u32)info_cnt) |
| 3692 | if (field_cnt * (repeat_cnt + 1) > info_cnt) | ||
| 3693 | return -E2BIG; | 3691 | return -E2BIG; |
| 3694 | 3692 | ||
| 3695 | cur = field_cnt; | 3693 | cur = field_cnt; |
| @@ -4447,7 +4445,7 @@ static s32 btf_enum_check_meta(struct btf_verifier_env *env, · 751 unmodified lines | |||
| 4447 | const struct btf_enum *enums = btf_type_enum(t); | 4445 | const struct btf_enum *enums = btf_type_enum(t); |
| 4448 | struct btf *btf = env->btf; | 4446 | struct btf *btf = env->btf; |
| 4449 | const char *fmt_str; | 4447 | const char *fmt_str; |
| 4450 | u16 i, nr_enums; | 4448 | u32 i, nr_enums; |
| 4451 | u32 meta_needed; | 4449 | u32 meta_needed; |
| 4452 | 4450 | ||
| 4453 | nr_enums = btf_type_vlen(t); | 4451 | nr_enums = btf_type_vlen(t); |
| @@ -4555,7 +4553,7 @@ static s32 btf_enum64_check_meta(struct btf_verifier_env *env, · 101 unmodified lines | |||
| 4555 | const struct btf_enum64 *enums = btf_type_enum64(t); | 4553 | const struct btf_enum64 *enums = btf_type_enum64(t); |
| 4556 | struct btf *btf = env->btf; | 4554 | struct btf *btf = env->btf; |
| 4557 | const char *fmt_str; | 4555 | const char *fmt_str; |
| 4558 | u16 i, nr_enums; | 4556 | u32 i, nr_enums; |
| 4559 | u32 meta_needed; | 4557 | u32 meta_needed; |
| 4560 | 4558 | ||
| 4561 | nr_enums = btf_type_vlen(t); | 4559 | nr_enums = btf_type_vlen(t); |
| @@ -4683,7 +4681,7 @@ static void btf_func_proto_log(struct btf_verifier_env *env, · 121 unmodified lines | |||
| 4683 | const struct btf_type *t) | 4681 | const struct btf_type *t) |
| 4684 | { | 4682 | { |
| 4685 | const struct btf_param *args = (const struct btf_param *)(t + 1); | 4683 | const struct btf_param *args = (const struct btf_param *)(t + 1); |
| 4686 | u16 nr_args = btf_type_vlen(t), i; | 4684 | u32 nr_args = btf_type_vlen(t), i; |
| 4687 | 4685 | ||
| 4688 | btf_verifier_log(env, "return=%u args=(", t->type); | 4686 | btf_verifier_log(env, "return=%u args=(", t->type); |
| 4689 | if (!nr_args) { | 4687 | if (!nr_args) { |
| @@ -4929,7 +4927,7 @@ static int btf_datasec_resolve(struct btf_verifier_env *env, · 239 unmodified lines | |||
| 4929 | { | 4927 | { |
| 4930 | const struct btf_var_secinfo *vsi; | 4928 | const struct btf_var_secinfo *vsi; |
| 4931 | struct btf *btf = env->btf; | 4929 | struct btf *btf = env->btf; |
| 4932 | u16 i; | 4930 | u32 i; |
| 4933 | 4931 | ||
| 4934 | env->resolve_mode = RESOLVE_TBD; | 4932 | env->resolve_mode = RESOLVE_TBD; |
| 4935 | for_each_vsi_from(i, v->next_member, v->t, vsi) { | 4933 | for_each_vsi_from(i, v->next_member, v->t, vsi) { |
| @@ -5183,7 +5181,7 @@ static int btf_func_proto_check(struct btf_verifier_env *env, · 247 unmodified lines | |||
| 5183 | const struct btf_type *ret_type; | 5181 | const struct btf_type *ret_type; |
| 5184 | const struct btf_param *args; | 5182 | const struct btf_param *args; |
| 5185 | const struct btf *btf; | 5183 | const struct btf *btf; |
| 5186 | u16 nr_args, i; | 5184 | u32 nr_args, i; |
| 5187 | int err; | 5185 | int err; |
| 5188 | 5186 | ||
| 5189 | btf = env->btf; | 5187 | btf = env->btf; |
| @@ -5278,7 +5276,7 @@ static int btf_func_check(struct btf_verifier_env *env, · 88 unmodified lines | |||
| 5278 | const struct btf_type *proto_type; | 5276 | const struct btf_type *proto_type; |
| 5279 | const struct btf_param *args; | 5277 | const struct btf_param *args; |
| 5280 | const struct btf *btf; | 5278 | const struct btf *btf; |
| 5281 | u16 nr_args, i; | 5279 | u32 nr_args, i; |
| 5282 | 5280 | ||
| 5283 | btf = env->btf; | 5281 | btf = env->btf; |
| 5284 | proto_type = btf_type_by_id(btf, t->type); | 5282 | proto_type = btf_type_by_id(btf, t->type); |
| @@ -5336,12 +5334,6 @@ static s32 btf_check_meta(struct btf_verifier_env *env, · 51 unmodified lines | |||
| 5336 | } | 5334 | } |
| 5337 | meta_left -= sizeof(*t); | 5335 | meta_left -= sizeof(*t); |
| 5338 | 5336 | ||
| 5339 | if (t->info & ~BTF_INFO_MASK) { | ||
| 5340 | btf_verifier_log(env, "[%u] Invalid btf_info:%x", | ||
| 5341 | env->log_type_id, t->info); | ||
| 5342 | return -EINVAL; | ||
| 5343 | } | ||
| 5344 | |||
| 5345 | if (BTF_INFO_KIND(t->info) > BTF_KIND_MAX || | 5337 | if (BTF_INFO_KIND(t->info) > BTF_KIND_MAX || |
| 5346 | BTF_INFO_KIND(t->info) == BTF_KIND_UNKN) { | 5338 | BTF_INFO_KIND(t->info) == BTF_KIND_UNKN) { |
| 5347 | btf_verifier_log(env, "[%u] Invalid kind:%u", | 5339 | btf_verifier_log(env, "[%u] Invalid kind:%u", |
| @@ -5914,25 +5906,10 @@ static int btf_check_type_tags(struct btf_verifier_env *env, · 566 unmodified lines | |||
| 5914 | return 0; | 5906 | return 0; |
| 5915 | } | 5907 | } |
| 5916 | 5908 | ||
| 5917 | static int finalize_log(struct bpf_verifier_log *log, bpfptr_t uattr, u32 uattr_size) | 5909 | static struct btf *btf_parse(const union bpf_attr *attr, bpfptr_t uattr, |
| 5918 | { | 5910 | struct bpf_log_attr *attr_log) |
| 5919 | u32 log_true_size; | ||
| 5920 | int err; | ||
| 5921 | |||
| 5922 | err = bpf_vlog_finalize(log, &log_true_size); | ||
| 5923 | |||
| 5924 | if (uattr_size >= offsetofend(union bpf_attr, btf_log_true_size) && | ||
| 5925 | copy_to_bpfptr_offset(uattr, offsetof(union bpf_attr, btf_log_true_size), | ||
| 5926 | &log_true_size, sizeof(log_true_size))) | ||
| 5927 | err = -EFAULT; | ||
| 5928 | |||
| 5929 | return err; | ||
| 5930 | } | ||
| 5931 | |||
| 5932 | static struct btf *btf_parse(const union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size) | ||
| 5933 | { | 5911 | { |
| 5934 | bpfptr_t btf_data = make_bpfptr(attr->btf, uattr.is_kernel); | 5912 | bpfptr_t btf_data = make_bpfptr(attr->btf, uattr.is_kernel); |
| 5935 | char __user *log_ubuf = u64_to_user_ptr(attr->btf_log_buf); | ||
| 5936 | struct btf_struct_metas *struct_meta_tab; | 5913 | struct btf_struct_metas *struct_meta_tab; |
| 5937 | struct btf_verifier_env *env = NULL; | 5914 | struct btf_verifier_env *env = NULL; |
| 5938 | struct btf *btf = NULL; | 5915 | struct btf *btf = NULL; |
| @@ -5949,8 +5926,7 @@ static struct btf *btf_parse(const union bpf_attr *attr, bpfptr_t uattr, u32 uat · 10 unmodified lines | |||
| 5949 | /* user could have requested verbose verifier output | 5926 | /* user could have requested verbose verifier output |
| 5950 | * and supplied buffer to store the verification trace | 5927 | * and supplied buffer to store the verification trace |
| 5951 | */ | 5928 | */ |
| 5952 | err = bpf_vlog_init(&env->log, attr->btf_log_level, | 5929 | err = bpf_vlog_init(&env->log, attr_log->level, attr_log->ubuf, attr_log->size); |
| 5953 | log_ubuf, attr->btf_log_size); | ||
| 5954 | if (err) | 5930 | if (err) |
| 5955 | goto errout_free; | 5931 | goto errout_free; |
| 5956 | 5932 | ||
| @@ -6015,7 +5991,7 @@ static struct btf *btf_parse(const union bpf_attr *attr, bpfptr_t uattr, u32 uat · 58 unmodified lines | |||
| 6015 | } | 5991 | } |
| 6016 | } | 5992 | } |
| 6017 | 5993 | ||
| 6018 | err = finalize_log(&env->log, uattr, uattr_size); | 5994 | err = bpf_log_attr_finalize(attr_log, &env->log); |
| 6019 | if (err) | 5995 | if (err) |
| 6020 | goto errout_free; | 5996 | goto errout_free; |
| 6021 | 5997 | ||
| @@ -6027,7 +6003,7 @@ errout_meta: · 5 unmodified lines | |||
| 6027 | btf_free_struct_meta_tab(btf); | 6003 | btf_free_struct_meta_tab(btf); |
| 6028 | errout: | 6004 | errout: |
| 6029 | /* overwrite err with -ENOSPC or -EFAULT */ | 6005 | /* overwrite err with -ENOSPC or -EFAULT */ |
| 6030 | ret = finalize_log(&env->log, uattr, uattr_size); | 6006 | ret = bpf_log_attr_finalize(attr_log, &env->log); |
| 6031 | if (ret) | 6007 | if (ret) |
| 6032 | err = ret; | 6008 | err = ret; |
| 6033 | errout_free: | 6009 | errout_free: |
| @@ -6980,7 +6956,7 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type, · 946 unmodified lines | |||
| 6980 | info->reg_type = ctx_arg_info->reg_type; | 6956 | info->reg_type = ctx_arg_info->reg_type; |
| 6981 | info->btf = ctx_arg_info->btf ? : btf_vmlinux; | 6957 | info->btf = ctx_arg_info->btf ? : btf_vmlinux; |
| 6982 | info->btf_id = ctx_arg_info->btf_id; | 6958 | info->btf_id = ctx_arg_info->btf_id; |
| 6983 | info->ref_obj_id = ctx_arg_info->ref_obj_id; | 6959 | info->ref_id = ctx_arg_info->ref_id; |
| 6984 | return true; | 6960 | return true; |
| 6985 | } | 6961 | } |
| 6986 | } | 6962 | } |
| @@ -7825,6 +7801,134 @@ enum btf_arg_tag { · 838 unmodified lines | |||
| 7825 | ARG_TAG_ARENA = BIT_ULL(5), | 7801 | ARG_TAG_ARENA = BIT_ULL(5), |
| 7826 | }; | 7802 | }; |
| 7827 | 7803 | ||
| 7804 | static int btf_scan_decl_tags(struct bpf_verifier_env *env, | ||
| 7805 | const struct btf *btf, | ||
| 7806 | const struct btf_type *fn_t, | ||
| 7807 | u32 arg_idx, bool is_global, u32 *tags) | ||
| 7808 | { | ||
| 7809 | int id = btf_named_start_id(btf, false) - 1; | ||
| 7810 | const char tag_key[] = "arg:"; | ||
| 7811 | static const struct { | ||
| 7812 | const char *tag_value; | ||
| 7813 | enum btf_arg_tag arg_tag; | ||
| 7814 | } tag_values[] = { | ||
| 7815 | { "ctx", ARG_TAG_CTX }, | ||
| 7816 | { "trusted", ARG_TAG_TRUSTED }, | ||
| 7817 | { "untrusted", ARG_TAG_UNTRUSTED }, | ||
| 7818 | { "nonnull", ARG_TAG_NONNULL }, | ||
| 7819 | { "nullable", ARG_TAG_NULLABLE }, | ||
| 7820 | { "arena", ARG_TAG_ARENA }, | ||
| 7821 | }; | ||
| 7822 | |||
| 7823 | /* | ||
| 7824 | * The 'arg:<tag>' decl_tag takes precedence over the derivation | ||
| 7825 | * of the register type from the BTF type itself. | ||
| 7826 | */ | ||
| 7827 | while ((id = btf_find_next_decl_tag(btf, fn_t, arg_idx, tag_key, id)) > 0) { | ||
| 7828 | const struct btf_type *tag_t; | ||
| 7829 | const char *tag; | ||
| 7830 | int i; | ||
| 7831 | bool found; | ||
| 7832 | |||
| 7833 | /* disallow arg tags in static subprogs */ | ||
| 7834 | if (!is_global) { | ||
| 7835 | bpf_log(&env->log, | ||
| 7836 | "arg#%d type tag is not supported in static functions\n", | ||
| 7837 | arg_idx); | ||
| 7838 | return -EOPNOTSUPP; | ||
| 7839 | } | ||
| 7840 | |||
| 7841 | tag_t = btf_type_by_id(btf, id); | ||
| 7842 | tag = __btf_name_by_offset(btf, tag_t->name_off) + (sizeof(tag_key) - 1); | ||
| 7843 | |||
| 7844 | found = false; | ||
| 7845 | for (i = 0; i < ARRAY_SIZE(tag_values); ++i) { | ||
| 7846 | if (!strcmp(tag, tag_values[i].tag_value)) { | ||
| 7847 | *tags |= tag_values[i].arg_tag; | ||
| 7848 | found = true; | ||
| 7849 | break; | ||
| 7850 | } | ||
| 7851 | } | ||
| 7852 | |||
| 7853 | if (!found) { | ||
| 7854 | bpf_log(&env->log, "arg#%d has unsupported set of tags\n", arg_idx); | ||
| 7855 | return -EOPNOTSUPP; | ||
| 7856 | } | ||
| 7857 | } | ||
| 7858 | if (id != -ENOENT) { | ||
| 7859 | bpf_log(&env->log, "arg#%d type tag fetching failure: %d\n", arg_idx, id); | ||
| 7860 | return id; | ||
| 7861 | } | ||
| 7862 | |||
| 7863 | return 0; | ||
| 7864 | } | ||
| 7865 | |||
| 7866 | static int btf_scan_type_tags(struct bpf_verifier_env *env, | ||
| 7867 | const struct btf *btf, u32 type_id, | ||
| 7868 | u32 *tags) | ||
| 7869 | { | ||
| 7870 | const struct btf_type *t; | ||
| 7871 | |||
| 7872 | /* Find the first pointer type in the chain. */ | ||
| 7873 | t = btf_type_skip_modifiers(btf, type_id, NULL); | ||
| 7874 | |||
| 7875 | /* | ||
| 7876 | * We currently reject type tags on non-pointer types, | ||
| 7877 | * which neither LLVM nor GCC support anyway. | ||
| 7878 | */ | ||
| 7879 | if (!t || !btf_type_is_ptr(t)) | ||
| 7880 | return 0; | ||
| 7881 | |||
| 7882 | /* We got a pointer, get all associated type tags. */ | ||
| 7883 | for (t = btf_type_by_id(btf, t->type); t && btf_type_is_modifier(t); | ||
| 7884 | t = btf_type_by_id(btf, t->type)) { | ||
| 7885 | |||
| 7886 | /* Skip non-type tag modifiers. */ | ||
| 7887 | if (!btf_type_is_type_tag(t)) | ||
| 7888 | continue; | ||
| 7889 | |||
| 7890 | const char *tag = __btf_name_by_offset(btf, t->name_off); | ||
| 7891 | |||
| 7892 | if (strcmp(tag, "arena") == 0) { | ||
| 7893 | *tags |= ARG_TAG_ARENA; | ||
| 7894 | } else { | ||
| 7895 | bpf_log(&env->log, "function signature member has unsupported type tag '%s'\n", | ||
| 7896 | tag); | ||
| 7897 | return -EOPNOTSUPP; | ||
| 7898 | } | ||
| 7899 | } | ||
| 7900 | |||
| 7901 | return 0; | ||
| 7902 | } | ||
| 7903 | |||
| 7904 | /* Check whether the type is a valid return type. */ | ||
| 7905 | static int btf_validate_return_type(struct bpf_verifier_env *env, struct btf *btf, | ||
| 7906 | const struct btf_type *t, int subprog) | ||
| 7907 | { | ||
| 7908 | u32 tags = 0; | ||
| 7909 | int err; | ||
| 7910 | |||
| 7911 | err = btf_scan_type_tags(env, btf, t->type, &tags); | ||
| 7912 | if (err) | ||
| 7913 | return err; | ||
| 7914 | |||
| 7915 | t = btf_type_skip_modifiers(btf, t->type, NULL); | ||
| 7916 | |||
| 7917 | /* | ||
| 7918 | * We allow all subprogs except for the main one to return any kind of arena pointer. | ||
| 7919 | * General arena variables are not allowed, since it makes no sense to return by value | ||
| 7920 | * a variable that's on the heap in the first place. | ||
| 7921 | */ | ||
| 7922 | if (subprog && (tags & ARG_TAG_ARENA) && btf_type_is_ptr(t)) | ||
| 7923 | return 0; | ||
| 7924 | |||
| 7925 | /* We always accept void or scalars. */ | ||
| 7926 | if (btf_type_is_void(t) || btf_type_is_int(t) || btf_is_any_enum(t)) | ||
| 7927 | return 0; | ||
| 7928 | |||
| 7929 | return -EOPNOTSUPP; | ||
| 7930 | } | ||
| 7931 | |||
| 7828 | /* Process BTF of a function to produce high-level expectation of function | 7932 | /* Process BTF of a function to produce high-level expectation of function |
| 7829 | * arguments (like ARG_PTR_TO_CTX, or ARG_PTR_TO_MEM, etc). This information | 7933 | * arguments (like ARG_PTR_TO_CTX, or ARG_PTR_TO_MEM, etc). This information |
| 7830 | * is cached in subprog info for reuse. | 7934 | * is cached in subprog info for reuse. |
| @@ -7843,6 +7947,7 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog) · 12 unmodified lines | |||
| 7843 | struct btf *btf = prog->aux->btf; | 7947 | struct btf *btf = prog->aux->btf; |
| 7844 | const struct btf_param *args; | 7948 | const struct btf_param *args; |
| 7845 | const struct btf_type *t, *ref_t, *fn_t; | 7949 | const struct btf_type *t, *ref_t, *fn_t; |
| 7950 | int err; | ||
| 7846 | u32 i, nargs, btf_id; | 7951 | u32 i, nargs, btf_id; |
| 7847 | const char *tname; | 7952 | const char *tname; |
| 7848 | 7953 | ||
| @@ -7887,25 +7992,36 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog) · 38 unmodified lines | |||
| 7887 | } | 7992 | } |
| 7888 | args = (const struct btf_param *)(t + 1); | 7993 | args = (const struct btf_param *)(t + 1); |
| 7889 | nargs = btf_type_vlen(t); | 7994 | nargs = btf_type_vlen(t); |
| 7995 | sub->arg_cnt = nargs; | ||
| 7996 | if (nargs > MAX_BPF_FUNC_ARGS) { | ||
| 7997 | bpf_log(log, "kernel supports at most %d parameters, function %s has %d\n", | ||
| 7998 | MAX_BPF_FUNC_ARGS, tname, nargs); | ||
| 7999 | return -EFAULT; | ||
| 8000 | } | ||
| 7890 | if (nargs > MAX_BPF_FUNC_REG_ARGS) { | 8001 | if (nargs > MAX_BPF_FUNC_REG_ARGS) { |
| 7891 | if (!is_global) | 8002 | if (!bpf_jit_supports_stack_args()) { |
| 7892 | return -EINVAL; | 8003 | bpf_log(log, "JIT does not support function %s() with %d args\n", |
| 7893 | bpf_log(log, "Global function %s() with %d > %d args. Buggy compiler.\n", | 8004 | tname, nargs); |
| 8005 | return -EFAULT; | ||
| 8006 | } | ||
| 8007 | sub->stack_arg_cnt = nargs - MAX_BPF_FUNC_REG_ARGS; | ||
| 8008 | } | ||
| 8009 | |||
| 8010 | if (is_global && nargs > MAX_BPF_FUNC_REG_ARGS) { | ||
| 8011 | bpf_log(log, "global function %s has %d > %d args, stack args not supported\n", | ||
| 7894 | tname, nargs, MAX_BPF_FUNC_REG_ARGS); | 8012 | tname, nargs, MAX_BPF_FUNC_REG_ARGS); |
| 7895 | return -EINVAL; | 8013 | return -EINVAL; |
| 7896 | } | 8014 | } |
| 7897 | /* check that function is void or returns int, exception cb also requires this */ | 8015 | |
| 7898 | t = btf_type_by_id(btf, t->type); | 8016 | err = btf_validate_return_type(env, btf, t, subprog); |
| 7899 | while (btf_type_is_modifier(t)) | 8017 | if (err) { |
| 7900 | t = btf_type_by_id(btf, t->type); | 8018 | if (is_global) { |
| 7901 | if (!btf_type_is_void(t) && !btf_type_is_int(t) && !btf_is_any_enum(t)) { | 8019 | bpf_log(log, |
| 7902 | if (!is_global) | 8020 | "Global function %s() return value not void or scalar. " |
| 7903 | return -EINVAL; | 8021 | "Only those are supported.\n", |
| 7904 | bpf_log(log, | 8022 | tname); |
| 7905 | "Global function %s() return value not void or scalar. " | 8023 | } |
| 7906 | "Only those are supported.\n", | 8024 | return err; |
| 7907 | tname); | ||
| 7908 | return -EINVAL; | ||
| 7909 | } | 8025 | } |
| 7910 | 8026 | ||
| 7911 | /* Convert BTF function arguments into verifier types. | 8027 | /* Convert BTF function arguments into verifier types. |
| @@ -7913,42 +8029,13 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog) · 1 unmodified line | |||
| 7913 | */ | 8029 | */ |
| 7914 | for (i = 0; i < nargs; i++) { | 8030 | for (i = 0; i < nargs; i++) { |
| 7915 | u32 tags = 0; | 8031 | u32 tags = 0; |
| 7916 | int id = btf_named_start_id(btf, false) - 1; | 8032 | err = btf_scan_decl_tags(env, btf, fn_t, i, is_global, &tags); |
| 7917 | 8033 | if (err) | |
| 7918 | /* 'arg:<tag>' decl_tag takes precedence over derivation of | 8034 | return err; |
| 7919 | * register type from BTF type itself | ||
| 7920 | */ | ||
| 7921 | while ((id = btf_find_next_decl_tag(btf, fn_t, i, "arg:", id)) > 0) { | ||
| 7922 | const struct btf_type *tag_t = btf_type_by_id(btf, id); | ||
| 7923 | const char *tag = __btf_name_by_offset(btf, tag_t->name_off) + 4; | ||
| 7924 | |||
| 7925 | /* disallow arg tags in static subprogs */ | ||
| 7926 | if (!is_global) { | ||
| 7927 | bpf_log(log, "arg#%d type tag is not supported in static functions\n", i); | ||
| 7928 | return -EOPNOTSUPP; | ||
| 7929 | } | ||
| 7930 | 8035 | ||
| 7931 | if (strcmp(tag, "ctx") == 0) { | 8036 | err = btf_scan_type_tags(env, btf, args[i].type, &tags); |
| 7932 | tags |= ARG_TAG_CTX; | 8037 | if (err) |
| 7933 | } else if (strcmp(tag, "trusted") == 0) { | 8038 | return err; |
| 7934 | tags |= ARG_TAG_TRUSTED; | ||
| 7935 | } else if (strcmp(tag, "untrusted") == 0) { | ||
| 7936 | tags |= ARG_TAG_UNTRUSTED; | ||
| 7937 | } else if (strcmp(tag, "nonnull") == 0) { | ||
| 7938 | tags |= ARG_TAG_NONNULL; | ||
| 7939 | } else if (strcmp(tag, "nullable") == 0) { | ||
| 7940 | tags |= ARG_TAG_NULLABLE; | ||
| 7941 | } else if (strcmp(tag, "arena") == 0) { | ||
| 7942 | tags |= ARG_TAG_ARENA; | ||
| 7943 | } else { | ||
| 7944 | bpf_log(log, "arg#%d has unsupported set of tags\n", i); | ||
| 7945 | return -EOPNOTSUPP; | ||
| 7946 | } | ||
| 7947 | } | ||
| 7948 | if (id != -ENOENT) { | ||
| 7949 | bpf_log(log, "arg#%d type tag fetching failure: %d\n", i, id); | ||
| 7950 | return id; | ||
| 7951 | } | ||
| 7952 | 8039 | ||
| 7953 | t = btf_type_by_id(btf, args[i].type); | 8040 | t = btf_type_by_id(btf, args[i].type); |
| 7954 | while (btf_type_is_modifier(t)) | 8041 | while (btf_type_is_modifier(t)) |
| @@ -7973,7 +8060,7 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog) · 18 unmodified lines | |||
| 7973 | bpf_log(log, "arg#%d has invalid combination of tags\n", i); | 8060 | bpf_log(log, "arg#%d has invalid combination of tags\n", i); |
| 7974 | return -EINVAL; | 8061 | return -EINVAL; |
| 7975 | } | 8062 | } |
| 7976 | sub->args[i].arg_type = ARG_PTR_TO_DYNPTR | MEM_RDONLY; | 8063 | sub->args[i].arg_type = ARG_PTR_TO_DYNPTR; |
| 7977 | continue; | 8064 | continue; |
| 7978 | } | 8065 | } |
| 7979 | if (tags & ARG_TAG_TRUSTED) { | 8066 | if (tags & ARG_TAG_TRUSTED) { |
| @@ -8074,7 +8161,6 @@ skip_pointer: · 94 unmodified lines | |||
| 8074 | return -EINVAL; | 8161 | return -EINVAL; |
| 8075 | } | 8162 | } |
| 8076 | 8163 | ||
| 8077 | sub->arg_cnt = nargs; | ||
| 8078 | sub->args_cached = true; | 8164 | sub->args_cached = true; |
| 8079 | 8165 | ||
| 8080 | return 0; | 8166 | return 0; |
| @@ -8196,12 +8282,12 @@ static int __btf_new_fd(struct btf *btf) · 115 unmodified lines | |||
| 8196 | return anon_inode_getfd("btf", &btf_fops, btf, O_RDONLY | O_CLOEXEC); | 8282 | return anon_inode_getfd("btf", &btf_fops, btf, O_RDONLY | O_CLOEXEC); |
| 8197 | } | 8283 | } |
| 8198 | 8284 | ||
| 8199 | int btf_new_fd(const union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size) | 8285 | int btf_new_fd(const union bpf_attr *attr, bpfptr_t uattr, struct bpf_log_attr *attr_log) |
| 8200 | { | 8286 | { |
| 8201 | struct btf *btf; | 8287 | struct btf *btf; |
| 8202 | int ret; | 8288 | int ret; |
| 8203 | 8289 | ||
| 8204 | btf = btf_parse(attr, uattr, uattr_size); | 8290 | btf = btf_parse(attr, uattr, attr_log); |
| 8205 | if (IS_ERR(btf)) | 8291 | if (IS_ERR(btf)) |
| 8206 | return PTR_ERR(btf); | 8292 | return PTR_ERR(btf); |
| 8207 | 8293 | ||
| @@ -8684,6 +8770,39 @@ static int btf_check_iter_kfuncs(struct btf *btf, const char *func_name, · 476 unmodified lines | |||
| 8684 | return 0; | 8770 | return 0; |
| 8685 | } | 8771 | } |
| 8686 | 8772 | ||
| 8773 | static int btf_check_kfunc_name(struct btf *btf, const char *func_name, u32 kind) | ||
| 8774 | { | ||
| 8775 | #ifdef CONFIG_DEBUG_INFO_BTF_MODULES | ||
| 8776 | struct btf_module *btf_mod, *tmp; | ||
| 8777 | #endif | ||
| 8778 | s32 id; | ||
| 8779 | |||
| 8780 | if (!btf_is_module(btf)) | ||
| 8781 | return 0; | ||
| 8782 | |||
| 8783 | id = btf_find_by_name_kind(bpf_get_btf_vmlinux(), func_name, kind); | ||
| 8784 | if (id >= 0) { | ||
| 8785 | pr_err("kfunc %s (id: %d) is already present in vmlinux.\n", | ||
| 8786 | func_name, id); | ||
| 8787 | return -EINVAL; | ||
| 8788 | } | ||
| 8789 | |||
| 8790 | #ifdef CONFIG_DEBUG_INFO_BTF_MODULES | ||
| 8791 | guard(mutex)(&btf_module_mutex); | ||
| 8792 | list_for_each_entry_safe(btf_mod, tmp, &btf_modules, list) { | ||
| 8793 | if (btf_mod->btf == btf) | ||
| 8794 | continue; | ||
| 8795 | id = btf_find_by_name_kind(btf_mod->btf, func_name, kind); | ||
| 8796 | if (id >= 0) { | ||
| 8797 | pr_err("kfunc %s (id: %d) is already present in module %s.\n", | ||
| 8798 | func_name, id, btf_mod->module->name); | ||
| 8799 | return -EINVAL; | ||
| 8800 | } | ||
| 8801 | } | ||
| 8802 | #endif | ||
| 8803 | return 0; | ||
| 8804 | } | ||
| 8805 | |||
| 8687 | static int btf_check_kfunc_protos(struct btf *btf, u32 func_id, u32 func_flags) | 8806 | static int btf_check_kfunc_protos(struct btf *btf, u32 func_id, u32 func_flags) |
| 8688 | { | 8807 | { |
| 8689 | const struct btf_type *func; | 8808 | const struct btf_type *func; |
| @@ -8697,7 +8816,8 @@ static int btf_check_kfunc_protos(struct btf *btf, u32 func_id, u32 func_flags) · 7 unmodified lines | |||
| 8697 | 8816 | ||
| 8698 | /* sanity check kfunc name */ | 8817 | /* sanity check kfunc name */ |
| 8699 | func_name = btf_name_by_offset(btf, func->name_off); | 8818 | func_name = btf_name_by_offset(btf, func->name_off); |
| 8700 | if (!func_name || !func_name[0]) | 8819 | if (!func_name || !func_name[0] || |
| 8820 | btf_check_kfunc_name(btf, func_name, BTF_INFO_KIND(func->info))) | ||
| 8701 | return -EINVAL; | 8821 | return -EINVAL; |
| 8702 | 8822 | ||
| 8703 | func = btf_type_by_id(btf, func->type); | 8823 | func = btf_type_by_id(btf, func->type); |
| @@ -55,6 +55,28 @@ void __init cgroup_bpf_lifetime_notifier_init(void) · 54 unmodified lines | |||
| 55 | &cgroup_bpf_lifetime_nb)); | 55 | &cgroup_bpf_lifetime_nb)); |
| 56 | } | 56 | } |
| 57 | 57 | ||
| 58 | #ifdef CONFIG_BPF_LSM | ||
| 59 | struct cgroup_lsm_atype { | ||
| 60 | u32 attach_btf_id; | ||
| 61 | int refcnt; | ||
| 62 | bool returns_errno; | ||
| 63 | }; | ||
| 64 | |||
| 65 | static struct cgroup_lsm_atype cgroup_lsm_atype[CGROUP_LSM_NUM]; | ||
| 66 | |||
| 67 | static bool cgroup_bpf_hook_returns_errno(enum cgroup_bpf_attach_type atype) | ||
| 68 | { | ||
| 69 | if (atype >= CGROUP_LSM_START && atype <= CGROUP_LSM_END) | ||
| 70 | return READ_ONCE(cgroup_lsm_atype[atype - CGROUP_LSM_START].returns_errno); | ||
| 71 | return true; | ||
| 72 | } | ||
| 73 | #else | ||
| 74 | static bool cgroup_bpf_hook_returns_errno(enum cgroup_bpf_attach_type atype) | ||
| 75 | { | ||
| 76 | return true; | ||
| 77 | } | ||
| 78 | #endif | ||
| 79 | |||
| 58 | /* __always_inline is necessary to prevent indirect call through run_prog | 80 | /* __always_inline is necessary to prevent indirect call through run_prog |
| 59 | * function pointer. | 81 | * function pointer. |
| 60 | */ | 82 | */ |
| @@ -83,7 +105,8 @@ bpf_prog_run_array_cg(const struct cgroup_bpf *cgrp, · 22 unmodified lines | |||
| 83 | *(ret_flags) |= (func_ret >> 1); | 105 | *(ret_flags) |= (func_ret >> 1); |
| 84 | func_ret &= 1; | 106 | func_ret &= 1; |
| 85 | } | 107 | } |
| 86 | if (!func_ret && !IS_ERR_VALUE((long)run_ctx.retval)) | 108 | if (!func_ret && cgroup_bpf_hook_returns_errno(atype) && |
| 109 | !IS_ERR_VALUE((long)run_ctx.retval)) | ||
| 87 | run_ctx.retval = -EPERM; | 110 | run_ctx.retval = -EPERM; |
| 88 | item++; | 111 | item++; |
| 89 | } | 112 | } |
| @@ -156,13 +179,6 @@ unsigned int __cgroup_bpf_run_lsm_current(const void *ctx, · 66 unmodified lines | |||
| 156 | } | 179 | } |
| 157 | 180 | ||
| 158 | #ifdef CONFIG_BPF_LSM | 181 | #ifdef CONFIG_BPF_LSM |
| 159 | struct cgroup_lsm_atype { | ||
| 160 | u32 attach_btf_id; | ||
| 161 | int refcnt; | ||
| 162 | }; | ||
| 163 | |||
| 164 | static struct cgroup_lsm_atype cgroup_lsm_atype[CGROUP_LSM_NUM]; | ||
| 165 | |||
| 166 | static enum cgroup_bpf_attach_type | 182 | static enum cgroup_bpf_attach_type |
| 167 | bpf_cgroup_atype_find(enum bpf_attach_type attach_type, u32 attach_btf_id) | 183 | bpf_cgroup_atype_find(enum bpf_attach_type attach_type, u32 attach_btf_id) |
| 168 | { | 184 | { |
| @@ -191,10 +207,13 @@ void bpf_cgroup_atype_get(u32 attach_btf_id, int cgroup_atype) · 22 unmodified lines | |||
| 191 | 207 | ||
| 192 | lockdep_assert_held(&cgroup_mutex); | 208 | lockdep_assert_held(&cgroup_mutex); |
| 193 | 209 | ||
| 194 | WARN_ON_ONCE(cgroup_lsm_atype[i].attach_btf_id && | 210 | if (!cgroup_lsm_atype[i].attach_btf_id) { |
| 195 | cgroup_lsm_atype[i].attach_btf_id != attach_btf_id); | 211 | cgroup_lsm_atype[i].attach_btf_id = attach_btf_id; |
| 196 | 212 | WRITE_ONCE(cgroup_lsm_atype[i].returns_errno, | |
| 197 | cgroup_lsm_atype[i].attach_btf_id = attach_btf_id; | 213 | bpf_lsm_hook_returns_errno(attach_btf_id)); |
| 214 | } else { | ||
| 215 | WARN_ON_ONCE(cgroup_lsm_atype[i].attach_btf_id != attach_btf_id); | ||
| 216 | } | ||
| 198 | cgroup_lsm_atype[i].refcnt++; | 217 | cgroup_lsm_atype[i].refcnt++; |
| 199 | } | 218 | } |
| 200 | 219 | ||
| @@ -203,8 +222,10 @@ void bpf_cgroup_atype_put(int cgroup_atype) · 2 unmodified lines | |||
| 203 | int i = cgroup_atype - CGROUP_LSM_START; | 222 | int i = cgroup_atype - CGROUP_LSM_START; |
| 204 | 223 | ||
| 205 | cgroup_lock(); | 224 | cgroup_lock(); |
| 206 | if (--cgroup_lsm_atype[i].refcnt <= 0) | 225 | if (--cgroup_lsm_atype[i].refcnt <= 0) { |
| 226 | WRITE_ONCE(cgroup_lsm_atype[i].returns_errno, true); | ||
| 207 | cgroup_lsm_atype[i].attach_btf_id = 0; | 227 | cgroup_lsm_atype[i].attach_btf_id = 0; |
| 228 | } | ||
| 208 | WARN_ON_ONCE(cgroup_lsm_atype[i].refcnt < 0); | 229 | WARN_ON_ONCE(cgroup_lsm_atype[i].refcnt < 0); |
| 209 | cgroup_unlock(); | 230 | cgroup_unlock(); |
| 210 | } | 231 | } |
| @@ -1208,7 +1229,7 @@ static int cgroup_bpf_detach(struct cgroup *cgrp, struct bpf_prog *prog, · 997 unmodified lines | |||
| 1208 | 1229 | ||
| 1209 | /* Must be called with cgroup_mutex held to avoid races. */ | 1230 | /* Must be called with cgroup_mutex held to avoid races. */ |
| 1210 | static int __cgroup_bpf_query(struct cgroup *cgrp, const union bpf_attr *attr, | 1231 | static int __cgroup_bpf_query(struct cgroup *cgrp, const union bpf_attr *attr, |
| 1211 | union bpf_attr __user *uattr) | 1232 | union bpf_attr __user *uattr, u32 uattr_size) |
| 1212 | { | 1233 | { |
| 1213 | __u32 __user *prog_attach_flags = u64_to_user_ptr(attr->query.prog_attach_flags); | 1234 | __u32 __user *prog_attach_flags = u64_to_user_ptr(attr->query.prog_attach_flags); |
| 1214 | bool effective_query = attr->query.query_flags & BPF_F_QUERY_EFFECTIVE; | 1235 | bool effective_query = attr->query.query_flags & BPF_F_QUERY_EFFECTIVE; |
| @@ -1259,7 +1280,8 @@ static int __cgroup_bpf_query(struct cgroup *cgrp, const union bpf_attr *attr, · 44 unmodified lines | |||
| 1259 | return -EFAULT; | 1280 | return -EFAULT; |
| 1260 | if (!effective_query && from_atype == to_atype) | 1281 | if (!effective_query && from_atype == to_atype) |
| 1261 | revision = cgrp->bpf.revisions[from_atype]; | 1282 | revision = cgrp->bpf.revisions[from_atype]; |
| 1262 | if (copy_to_user(&uattr->query.revision, &revision, sizeof(revision))) | 1283 | if (uattr_size >= offsetofend(union bpf_attr, query.revision) && |
| 1284 | copy_to_user(&uattr->query.revision, &revision, sizeof(revision))) | ||
| 1263 | return -EFAULT; | 1285 | return -EFAULT; |
| 1264 | if (attr->query.prog_cnt == 0 || !prog_ids || !total_cnt) | 1286 | if (attr->query.prog_cnt == 0 || !prog_ids || !total_cnt) |
| 1265 | /* return early if user requested only program count + flags */ | 1287 | /* return early if user requested only program count + flags */ |
| @@ -1312,12 +1334,12 @@ static int __cgroup_bpf_query(struct cgroup *cgrp, const union bpf_attr *attr, · 46 unmodified lines | |||
| 1312 | } | 1334 | } |
| 1313 | 1335 | ||
| 1314 | static int cgroup_bpf_query(struct cgroup *cgrp, const union bpf_attr *attr, | 1336 | static int cgroup_bpf_query(struct cgroup *cgrp, const union bpf_attr *attr, |
| 1315 | union bpf_attr __user *uattr) | 1337 | union bpf_attr __user *uattr, u32 uattr_size) |
| 1316 | { | 1338 | { |
| 1317 | int ret; | 1339 | int ret; |
| 1318 | 1340 | ||
| 1319 | cgroup_lock(); | 1341 | cgroup_lock(); |
| 1320 | ret = __cgroup_bpf_query(cgrp, attr, uattr); | 1342 | ret = __cgroup_bpf_query(cgrp, attr, uattr, uattr_size); |
| 1321 | cgroup_unlock(); | 1343 | cgroup_unlock(); |
| 1322 | return ret; | 1344 | return ret; |
| 1323 | } | 1345 | } |
| @@ -1520,7 +1542,7 @@ out_put_cgroup: · 196 unmodified lines | |||
| 1520 | } | 1542 | } |
| 1521 | 1543 | ||
| 1522 | int cgroup_bpf_prog_query(const union bpf_attr *attr, | 1544 | int cgroup_bpf_prog_query(const union bpf_attr *attr, |
| 1523 | union bpf_attr __user *uattr) | 1545 | union bpf_attr __user *uattr, u32 uattr_size) |
| 1524 | { | 1546 | { |
| 1525 | struct cgroup *cgrp; | 1547 | struct cgroup *cgrp; |
| 1526 | int ret; | 1548 | int ret; |
| @@ -1529,7 +1551,7 @@ int cgroup_bpf_prog_query(const union bpf_attr *attr, · 2 unmodified lines | |||
| 1529 | if (IS_ERR(cgrp)) | 1551 | if (IS_ERR(cgrp)) |
| 1530 | return PTR_ERR(cgrp); | 1552 | return PTR_ERR(cgrp); |
| 1531 | 1553 | ||
| 1532 | ret = cgroup_bpf_query(cgrp, attr, uattr); | 1554 | ret = cgroup_bpf_query(cgrp, attr, uattr, uattr_size); |
| 1533 | 1555 | ||
| 1534 | cgroup_put(cgrp); | 1556 | cgroup_put(cgrp); |
| 1535 | return ret; | 1557 | return ret; |
| @@ -1935,8 +1957,8 @@ int __cgroup_bpf_run_filter_sysctl(struct ctl_table_header *head, · 399 unmodified lines | |||
| 1935 | 1957 | ||
| 1936 | kfree(ctx.cur_val); | 1958 | kfree(ctx.cur_val); |
| 1937 | 1959 | ||
| 1938 | if (ret == 1 && ctx.new_updated) { | 1960 | if (!ret && ctx.new_updated) { |
| 1939 | kfree(*buf); | 1961 | kvfree(*buf); |
| 1940 | *buf = ctx.new_val; | 1962 | *buf = ctx.new_val; |
| 1941 | *pcount = ctx.new_len; | 1963 | *pcount = ctx.new_len; |
| 1942 | } else { | 1964 | } else { |
| @@ -2342,6 +2364,7 @@ BPF_CALL_3(bpf_sysctl_set_new_value, struct bpf_sysctl_kern *, ctx, · 399 unmodified lines | |||
| 2342 | return -E2BIG; | 2364 | return -E2BIG; |
| 2343 | 2365 | ||
| 2344 | memcpy(ctx->new_val, buf, buf_len); | 2366 | memcpy(ctx->new_val, buf, buf_len); |
| 2367 | ((char *)ctx->new_val)[buf_len] = '\0'; | ||
| 2345 | ctx->new_len = buf_len; | 2368 | ctx->new_len = buf_len; |
| 2346 | ctx->new_updated = 1; | 2369 | ctx->new_updated = 1; |
| 2347 | 2370 | ||
| @@ -0,0 +1,120 @@ | |||
| 1 | // SPDX-License-Identifier: GPL-2.0-only | ||
| 2 | /* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ | ||
| 3 | |||
| 4 | #include <linux/bits.h> | ||
| 5 | |||
| 6 | #define T 32 | ||
| 7 | #include "cnum_defs.h" | ||
| 8 | #undef T | ||
| 9 | |||
| 10 | #define T 64 | ||
| 11 | #include "cnum_defs.h" | ||
| 12 | #undef T | ||
| 13 | |||
| 14 | struct cnum32 cnum32_from_cnum64(struct cnum64 cnum) | ||
| 15 | { | ||
| 16 | if (cnum64_is_empty(cnum)) | ||
| 17 | return CNUM32_EMPTY; | ||
| 18 | |||
| 19 | if (cnum.size >= U32_MAX) | ||
| 20 | return (struct cnum32){ .base = 0, .size = U32_MAX }; | ||
| 21 | else | ||
| 22 | return (struct cnum32){ .base = (u32)cnum.base, .size = cnum.size }; | ||
| 23 | } | ||
| 24 | |||
| 25 | /* | ||
| 26 | * Suppose 'a' and 'b' are laid out as follows: | ||
| 27 | * | ||
| 28 | * 64-bit number axis ---> | ||
| 29 | * | ||
| 30 | * N*2^32 (N+1)*2^32 (N+2)*2^32 (N+3)*2^32 | ||
| 31 | * ||------|---|=====|-------||----------|=====|-------||----------|=====|----|--|| | ||
| 32 | * | |< b >| |< b >| |< b >| | | ||
| 33 | * | | | | | ||
| 34 | * |<--+--------------------------- a ---------------------------+--->| | ||
| 35 | * | | | ||
| 36 | * |<-------------------------- t -------------------------->| | ||
| 37 | * | ||
| 38 | * In such a case it is possible to infer a more tight representation t | ||
| 39 | * such that ∀ v ∈ a, (u32)v ∈ b: v ∈ t. | ||
| 40 | */ | ||
| 41 | struct cnum64 cnum64_cnum32_intersect(struct cnum64 a, struct cnum32 b) | ||
| 42 | { | ||
| 43 | /* | ||
| 44 | * To simplify reasoning, rotate the circles so that [virtual] a1 starts | ||
| 45 | * at u32 boundary, b1 represents b in this new frame of reference. | ||
| 46 | */ | ||
| 47 | struct cnum32 b1 = { b.base - (u32)a.base, b.size }; | ||
| 48 | struct cnum64 t = a; | ||
| 49 | u64 d, b1_max; | ||
| 50 | |||
| 51 | if (cnum64_is_empty(a) || cnum32_is_empty(b)) | ||
| 52 | return CNUM64_EMPTY; | ||
| 53 | |||
| 54 | if (cnum32_urange_overflow(b1)) { | ||
| 55 | b1_max = (u32)b1.base + (u32)b1.size; /* overflow here is fine and necessary */ | ||
| 56 | if ((u32)a.size > b1_max && (u32)a.size < b1.base) { | ||
| 57 | /* | ||
| 58 | * N*2^32 (N+1)*2^32 | ||
| 59 | * ||=====|------------|=====||=====|---------|---|=====|| | ||
| 60 | * |b1 ->| |<- b1||b1 ->| | |<- b1| | ||
| 61 | * |<----------------- a1 ------------------>| | ||
| 62 | * |<-------------- t ------------>|<-- d -->| (after adjustment) | ||
| 63 | * ^ | ||
| 64 | * b1_max | ||
| 65 | */ | ||
| 66 | d = (u32)a.size - b1_max; | ||
| 67 | t.size -= d; | ||
| 68 | } else { | ||
| 69 | /* | ||
| 70 | * No adjustments possible in the following cases: | ||
| 71 | * | ||
| 72 | * ||=====|------------|=====||===|=|-------------|=|===|| | ||
| 73 | * |b1 ->| |<- b1||b1 +>| |<+ b1| | ||
| 74 | * |<----------------- a1 ------>| | | ||
| 75 | * |<----------------- (or) a1 ------------------->| | ||
| 76 | */ | ||
| 77 | } | ||
| 78 | } else { | ||
| 79 | if (t.size < b1.base) | ||
| 80 | /* | ||
| 81 | * N*2^32 (N+1)*2^32 | ||
| 82 | * ||----------|--|=======|--||------> | ||
| 83 | * |<-- a1 -->| |<- b ->| | ||
| 84 | */ | ||
| 85 | return CNUM64_EMPTY; | ||
| 86 | /* | ||
| 87 | * N*2^32 (N+1)*2^32 | ||
| 88 | * ||-------------|========|-||-----| -------|========|-|| | ||
| 89 | * | |<- b1 ->| | |<- b1 ->| | ||
| 90 | * |<------------+ a1 ------------>| | ||
| 91 | * |<------ t ------>| (after adjustment) | ||
| 92 | */ | ||
| 93 | t.base += b1.base; | ||
| 94 | t.size -= b1.base; | ||
| 95 | b1_max = b1.base + b1.size; | ||
| 96 | d = 0; | ||
| 97 | if ((u32)a.size < b1.base) | ||
| 98 | /* | ||
| 99 | * N*2^32 (N+1)*2^32 | ||
| 100 | * ||-------------|========|-||------|-------|========|-|| | ||
| 101 | * | |<- b1 ->| | |<- b1 ->| | ||
| 102 | * |<------------+-- a1 --+-------->| | ||
| 103 | * |<- t ->|<-- d -->| (after adjustment) | ||
| 104 | */ | ||
| 105 | d = (u32)a.size + (BIT_ULL(32) - b1_max); | ||
| 106 | else if ((u32)a.size >= b1_max) | ||
| 107 | /* | ||
| 108 | * N*2^32 (N+1)*2^32 | ||
| 109 | * ||--|========|------------||--|========|-------|-----|| | ||
| 110 | * | |<- b1 ->| |<- b1 ->| | | ||
| 111 | * |<-+------------------ a1 ------------+------>| | ||
| 112 | * |<-------------- t --------------->|<- d ->| (after adjustment) | ||
| 113 | */ | ||
| 114 | d = (u32)a.size - b1_max; | ||
| 115 | if (t.size < d) | ||
| 116 | return CNUM64_EMPTY; | ||
| 117 | t.size -= d; | ||
| 118 | } | ||
| 119 | return t; | ||
| 120 | } | ||
| @@ -0,0 +1,247 @@ | |||
| 1 | /* SPDX-License-Identifier: GPL-2.0-only */ | ||
| 2 | /* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ | ||
| 3 | |||
| 4 | #ifndef T | ||
| 5 | #error "Define T (bit width: 32, 64) before including cnum_defs.h" | ||
| 6 | #endif | ||
| 7 | |||
| 8 | #include <linux/cnum.h> | ||
| 9 | #include <linux/kernel.h> | ||
| 10 | #include <linux/limits.h> | ||
| 11 | #include <linux/minmax.h> | ||
| 12 | #include <linux/compiler_types.h> | ||
| 13 | |||
| 14 | #define cnum_t __PASTE(cnum, T) | ||
| 15 | #define ut __PASTE(u, T) | ||
| 16 | #define st __PASTE(s, T) | ||
| 17 | #define UT_MAX __PASTE(__PASTE(U, T), _MAX) | ||
| 18 | #define ST_MAX __PASTE(__PASTE(S, T), _MAX) | ||
| 19 | #define ST_MIN __PASTE(__PASTE(S, T), _MIN) | ||
| 20 | #define EMPTY __PASTE(__PASTE(CNUM, T), _EMPTY) | ||
| 21 | #define FN(name) __PASTE(__PASTE(cnum, T), __PASTE(_, name)) | ||
| 22 | |||
| 23 | struct cnum_t FN(from_urange)(ut min, ut max) | ||
| 24 | { | ||
| 25 | return (struct cnum_t){ .base = min, .size = (ut)max - min }; | ||
| 26 | } | ||
| 27 | |||
| 28 | struct cnum_t FN(from_srange)(st min, st max) | ||
| 29 | { | ||
| 30 | ut size = (ut)max - (ut)min; | ||
| 31 | ut base = size == UT_MAX ? 0 : (ut)min; | ||
| 32 | |||
| 33 | return (struct cnum_t){ .base = base, .size = size }; | ||
| 34 | } | ||
| 35 | |||
| 36 | /* True if this cnum represents two unsigned ranges. */ | ||
| 37 | static inline bool FN(urange_overflow)(struct cnum_t cnum) | ||
| 38 | { | ||
| 39 | /* Same as cnum.base + cnum.size > UT_MAX but avoids overflow */ | ||
| 40 | return cnum.size > UT_MAX - (ut)cnum.base; | ||
| 41 | } | ||
| 42 | |||
| 43 | /* | ||
| 44 | * cnum{T}_umin / cnum{T}_umax query an unsigned range represented by this cnum. | ||
| 45 | * If cnum represents a range crossing the UT_MAX/0 boundary, the unbound range | ||
| 46 | * [0..UT_MAX] is returned. | ||
| 47 | */ | ||
| 48 | ut FN(umin)(struct cnum_t cnum) | ||
| 49 | { | ||
| 50 | return FN(urange_overflow)(cnum) ? 0 : cnum.base; | ||
| 51 | } | ||
| 52 | EXPORT_SYMBOL_GPL(FN(umin)); | ||
| 53 | |||
| 54 | ut FN(umax)(struct cnum_t cnum) | ||
| 55 | { | ||
| 56 | return FN(urange_overflow)(cnum) ? UT_MAX : cnum.base + cnum.size; | ||
| 57 | } | ||
| 58 | EXPORT_SYMBOL_GPL(FN(umax)); | ||
| 59 | |||
| 60 | /* True if this cnum represents two signed ranges. */ | ||
| 61 | static inline bool FN(srange_overflow)(struct cnum_t cnum) | ||
| 62 | { | ||
| 63 | return FN(contains)(cnum, (ut)ST_MAX) && FN(contains)(cnum, (ut)ST_MIN); | ||
| 64 | } | ||
| 65 | |||
| 66 | /* | ||
| 67 | * cnum{T}_smin / cnum{T}_smax query a signed range represented by this cnum. | ||
| 68 | * If cnum represents a range crossing the ST_MAX/ST_MIN boundary, the unbound range | ||
| 69 | * [ST_MIN..ST_MAX] is returned. | ||
| 70 | */ | ||
| 71 | st FN(smin)(struct cnum_t cnum) | ||
| 72 | { | ||
| 73 | return FN(srange_overflow)(cnum) | ||
| 74 | ? ST_MIN | ||
| 75 | : min((st)cnum.base, (st)(cnum.base + cnum.size)); | ||
| 76 | } | ||
| 77 | |||
| 78 | st FN(smax)(struct cnum_t cnum) | ||
| 79 | { | ||
| 80 | return FN(srange_overflow)(cnum) | ||
| 81 | ? ST_MAX | ||
| 82 | : max((st)cnum.base, (st)(cnum.base + cnum.size)); | ||
| 83 | } | ||
| 84 | |||
| 85 | /* | ||
| 86 | * Returns a possibly empty intersection of cnums 'a' and 'b'. | ||
| 87 | * If 'a' and 'b' intersect in two sub-arcs, the function over-approximates | ||
| 88 | * and returns either 'a' or 'b', whichever is smaller. | ||
| 89 | */ | ||
| 90 | struct cnum_t FN(intersect)(struct cnum_t a, struct cnum_t b) | ||
| 91 | { | ||
| 92 | struct cnum_t b1; | ||
| 93 | ut dbase; | ||
| 94 | |||
| 95 | if (FN(is_empty)(a) || FN(is_empty)(b)) | ||
| 96 | return EMPTY; | ||
| 97 | |||
| 98 | if (a.base > b.base) | ||
| 99 | swap(a, b); | ||
| 100 | |||
| 101 | /* | ||
| 102 | * Rotate frame of reference such that a.base is 0. | ||
| 103 | * 'b1' is 'b' in this frame of reference. | ||
| 104 | */ | ||
| 105 | dbase = b.base - a.base; | ||
| 106 | b1 = (struct cnum_t){ dbase, b.size }; | ||
| 107 | if (FN(urange_overflow)(b1)) { | ||
| 108 | if (b1.base <= a.size) { | ||
| 109 | /* | ||
| 110 | * Rotated frame (a.base at origin): | ||
| 111 | * | ||
| 112 | * 0 UT_MAX | ||
| 113 | * |--------------------------------------------| | ||
| 114 | * [=== a ==========================] | | ||
| 115 | * [= b1 tail =] [========= b1 main ==========>] | ||
| 116 | * ^-- b1.base <= a.size | ||
| 117 | * | ||
| 118 | * 'a' and 'b' intersect in two disjoint arcs, | ||
| 119 | * can't represent as single cnum, over-approximate | ||
| 120 | * the result. | ||
| 121 | */ | ||
| 122 | return a.size <= b.size ? a : b; | ||
| 123 | } else { | ||
| 124 | /* | ||
| 125 | * Rotated frame (a.base at origin): | ||
| 126 | * | ||
| 127 | * 0 UT_MAX | ||
| 128 | * |--------------------------------------------| | ||
| 129 | * [=== a =============] | | | ||
| 130 | * [= b1 tail =] [======= b1 main ====>] | ||
| 131 | * ^-- b1.base > a.size | ||
| 132 | * | ||
| 133 | * Only 'b' tail intersects 'a'. | ||
| 134 | */ | ||
| 135 | return (struct cnum_t) { | ||
| 136 | .base = a.base, | ||
| 137 | .size = min(a.size, (ut)(b1.base + b1.size)), | ||
| 138 | }; | ||
| 139 | } | ||
| 140 | } else if (a.size >= b1.base) { | ||
| 141 | /* | ||
| 142 | * Rotated frame (a.base at origin): | ||
| 143 | * | ||
| 144 | * 0 UT_MAX | ||
| 145 | * |--------------------------------------------------| | ||
| 146 | * [=== a ==================================] | | ||
| 147 | * [== b1 =====================] | ||
| 148 | * | ||
| 149 | * 0 UT_MAX | ||
| 150 | * |--------------------------------------------------| | ||
| 151 | * [=== a ==================================] | | ||
| 152 | * [== b1 ====] | ||
| 153 | * ^-- b1.base <= a.size | ||
| 154 | * |<-- a.size - dbase -->| | ||
| 155 | * | ||
| 156 | * 'a' and 'b' intersect as one cnum. | ||
| 157 | */ | ||
| 158 | return (struct cnum_t) { | ||
| 159 | .base = b.base, | ||
| 160 | .size = min((ut)(a.size - dbase), b.size), | ||
| 161 | }; | ||
| 162 | } else { | ||
| 163 | return EMPTY; | ||
| 164 | } | ||
| 165 | } | ||
| 166 | |||
| 167 | void FN(intersect_with)(struct cnum_t *dst, struct cnum_t src) | ||
| 168 | { | ||
| 169 | *dst = FN(intersect)(*dst, src); | ||
| 170 | } | ||
| 171 | |||
| 172 | void FN(intersect_with_urange)(struct cnum_t *dst, ut min, ut max) | ||
| 173 | { | ||
| 174 | FN(intersect_with)(dst, FN(from_urange)(min, max)); | ||
| 175 | } | ||
| 176 | |||
| 177 | void FN(intersect_with_srange)(struct cnum_t *dst, st min, st max) | ||
| 178 | { | ||
| 179 | FN(intersect_with)(dst, FN(from_srange)(min, max)); | ||
| 180 | } | ||
| 181 | |||
| 182 | static inline struct cnum_t FN(normalize)(struct cnum_t cnum) | ||
| 183 | { | ||
| 184 | if (cnum.size == UT_MAX && cnum.base != 0 && cnum.base != (ut)ST_MAX) | ||
| 185 | cnum.base = 0; | ||
| 186 | return cnum; | ||
| 187 | } | ||
| 188 | |||
| 189 | struct cnum_t FN(add)(struct cnum_t a, struct cnum_t b) | ||
| 190 | { | ||
| 191 | if (FN(is_empty)(a) || FN(is_empty)(b)) | ||
| 192 | return EMPTY; | ||
| 193 | if (a.size > UT_MAX - b.size) | ||
| 194 | return (struct cnum_t){ 0, (ut)UT_MAX }; | ||
| 195 | else | ||
| 196 | return FN(normalize)((struct cnum_t){ a.base + b.base, a.size + b.size }); | ||
| 197 | } | ||
| 198 | |||
| 199 | struct cnum_t FN(negate)(struct cnum_t a) | ||
| 200 | { | ||
| 201 | if (FN(is_empty)(a)) | ||
| 202 | return EMPTY; | ||
| 203 | return FN(normalize)((struct cnum_t){ -((ut)a.base + a.size), a.size }); | ||
| 204 | } | ||
| 205 | |||
| 206 | bool FN(is_empty)(struct cnum_t cnum) | ||
| 207 | { | ||
| 208 | return cnum.base == EMPTY.base && cnum.size == EMPTY.size; | ||
| 209 | } | ||
| 210 | |||
| 211 | bool FN(contains)(struct cnum_t cnum, ut v) | ||
| 212 | { | ||
| 213 | if (FN(is_empty)(cnum)) | ||
| 214 | return false; | ||
| 215 | if (FN(urange_overflow)(cnum)) | ||
| 216 | return v >= cnum.base || v <= (ut)cnum.base + cnum.size; | ||
| 217 | else | ||
| 218 | return v >= cnum.base && v <= (ut)cnum.base + cnum.size; | ||
| 219 | } | ||
| 220 | |||
| 221 | bool FN(is_const)(struct cnum_t cnum) | ||
| 222 | { | ||
| 223 | return cnum.size == 0; | ||
| 224 | } | ||
| 225 | |||
| 226 | bool FN(is_subset)(struct cnum_t bigger, struct cnum_t smaller) | ||
| 227 | { | ||
| 228 | if (FN(is_empty(smaller))) | ||
| 229 | return true; | ||
| 230 | if (FN(is_empty(bigger))) | ||
| 231 | return false; | ||
| 232 | /* rotate both arcs such that 'bigger' starts at origin, hence does not overflow */ | ||
| 233 | smaller.base -= bigger.base; | ||
| 234 | bigger.base = 0; | ||
| 235 | if (FN(urange_overflow)(smaller) && bigger.size < UT_MAX) | ||
| 236 | return false; | ||
| 237 | return smaller.base + smaller.size <= bigger.size; | ||
| 238 | } | ||
| 239 | |||
| 240 | #undef EMPTY | ||
| 241 | #undef cnum_t | ||
| 242 | #undef ut | ||
| 243 | #undef st | ||
| 244 | #undef UT_MAX | ||
| 245 | #undef ST_MAX | ||
| 246 | #undef ST_MIN | ||
| 247 | #undef FN | ||
| @@ -58,6 +58,14 @@ static void const_reg_xfer(struct bpf_verifier_env *env, struct const_arg_info * · 57 unmodified lines | |||
| 58 | u8 opcode = BPF_OP(insn->code) | BPF_SRC(insn->code); | 58 | u8 opcode = BPF_OP(insn->code) | BPF_SRC(insn->code); |
| 59 | int r; | 59 | int r; |
| 60 | 60 | ||
| 61 | /* Stack arg stores (r11-based) are outside the tracked register set. */ | ||
| 62 | if (is_stack_arg_st(insn) || is_stack_arg_stx(insn)) | ||
| 63 | return; | ||
| 64 | if (is_stack_arg_ldx(insn)) { | ||
| 65 | ci_out[insn->dst_reg] = unknown; | ||
| 66 | return; | ||
| 67 | } | ||
| 68 | |||
| 61 | switch (class) { | 69 | switch (class) { |
| 62 | case BPF_ALU: | 70 | case BPF_ALU: |
| 63 | case BPF_ALU64: | 71 | case BPF_ALU64: |
| @@ -1299,8 +1299,8 @@ static int bpf_jit_blind_insn(const struct bpf_insn *from, · 1298 unmodified lines | |||
| 1299 | u32 imm_rnd = get_random_u32(); | 1299 | u32 imm_rnd = get_random_u32(); |
| 1300 | s16 off; | 1300 | s16 off; |
| 1301 | 1301 | ||
| 1302 | BUILD_BUG_ON(BPF_REG_AX + 1 != MAX_BPF_JIT_REG); | 1302 | BUILD_BUG_ON(BPF_REG_PARAMS + 2 != MAX_BPF_JIT_REG); |
| 1303 | BUILD_BUG_ON(MAX_BPF_REG + 1 != MAX_BPF_JIT_REG); | 1303 | BUILD_BUG_ON(BPF_REG_AX + 1 != MAX_BPF_JIT_REG); |
| 1304 | 1304 | ||
| 1305 | /* Constraints on AX register: | 1305 | /* Constraints on AX register: |
| 1306 | * | 1306 | * |
| @@ -1582,6 +1582,16 @@ bool bpf_insn_is_indirect_target(const struct bpf_verifier_env *env, const struc · 275 unmodified lines | |||
| 1582 | insn_idx += prog->aux->subprog_start; | 1582 | insn_idx += prog->aux->subprog_start; |
| 1583 | return env->insn_aux_data[insn_idx].indirect_target; | 1583 | return env->insn_aux_data[insn_idx].indirect_target; |
| 1584 | } | 1584 | } |
| 1585 | |||
| 1586 | u16 bpf_out_stack_arg_cnt(const struct bpf_verifier_env *env, const struct bpf_prog *prog) | ||
| 1587 | { | ||
| 1588 | const struct bpf_subprog_info *sub; | ||
| 1589 | |||
| 1590 | if (!env) | ||
| 1591 | return 0; | ||
| 1592 | sub = &env->subprog_info[prog->aux->func_idx]; | ||
| 1593 | return sub->stack_arg_cnt - bpf_in_stack_arg_cnt(sub); | ||
| 1594 | } | ||
| 1585 | #endif /* CONFIG_BPF_JIT */ | 1595 | #endif /* CONFIG_BPF_JIT */ |
| 1586 | 1596 | ||
| 1587 | /* Base function for offset calculation. Needs to go into .text section, | 1597 | /* Base function for offset calculation. Needs to go into .text section, |
| @@ -2471,7 +2481,7 @@ static bool __bpf_prog_map_compatible(struct bpf_map *map, · 883 unmodified lines | |||
| 2471 | cookie = aux->cgroup_storage[i] ? | 2481 | cookie = aux->cgroup_storage[i] ? |
| 2472 | aux->cgroup_storage[i]->cookie : 0; | 2482 | aux->cgroup_storage[i]->cookie : 0; |
| 2473 | ret = map->owner->storage_cookie[i] == cookie || | 2483 | ret = map->owner->storage_cookie[i] == cookie || |
| 2474 | !cookie; | 2484 | (!cookie && !aux->tail_call_reachable); |
| 2475 | } | 2485 | } |
| 2476 | if (ret && | 2486 | if (ret && |
| 2477 | map->owner->attach_func_proto != aux->attach_func_proto) { | 2487 | map->owner->attach_func_proto != aux->attach_func_proto) { |
| @@ -3228,6 +3238,11 @@ bool __weak bpf_jit_supports_kfunc_call(void) · 750 unmodified lines | |||
| 3228 | return false; | 3238 | return false; |
| 3229 | } | 3239 | } |
| 3230 | 3240 | ||
| 3241 | bool __weak bpf_jit_supports_stack_args(void) | ||
| 3242 | { | ||
| 3243 | return false; | ||
| 3244 | } | ||
| 3245 | |||
| 3231 | bool __weak bpf_jit_supports_far_kfunc_call(void) | 3246 | bool __weak bpf_jit_supports_far_kfunc_call(void) |
| 3232 | { | 3247 | { |
| 3233 | return false; | 3248 | return false; |
| @@ -3363,6 +3378,12 @@ __weak u64 bpf_arena_get_kern_vm_start(struct bpf_arena *arena) · 129 unmodified lines | |||
| 3363 | } | 3378 | } |
| 3364 | 3379 | ||
| 3365 | #ifdef CONFIG_BPF_SYSCALL | 3380 | #ifdef CONFIG_BPF_SYSCALL |
| 3381 | __weak bool bpf_arena_handle_page_fault(unsigned long addr, bool is_write, | ||
| 3382 | unsigned long fault_ip) | ||
| 3383 | { | ||
| 3384 | return false; | ||
| 3385 | } | ||
| 3386 | |||
| 3366 | static int __init bpf_global_ma_init(void) | 3387 | static int __init bpf_global_ma_init(void) |
| 3367 | { | 3388 | { |
| 3368 | int ret; | 3389 | int ret; |
| @@ -581,6 +581,10 @@ static int dev_map_enqueue_clone(struct bpf_dtab_netdev *obj, · 580 unmodified lines | |||
| 581 | { | 581 | { |
| 582 | struct xdp_frame *nxdpf; | 582 | struct xdp_frame *nxdpf; |
| 583 | 583 | ||
| 584 | /* Frags live outside the linear frame and cannot be cloned safely. */ | ||
| 585 | if (unlikely(xdp_frame_has_frags(xdpf))) | ||
| 586 | return -EOPNOTSUPP; | ||
| 587 | |||
| 584 | nxdpf = xdpf_clone(xdpf); | 588 | nxdpf = xdpf_clone(xdpf); |
| 585 | if (!nxdpf) | 589 | if (!nxdpf) |
| 586 | return -ENOMEM; | 590 | return -ENOMEM; |
| @@ -706,6 +710,18 @@ int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, · 119 unmodified lines | |||
| 706 | if (unlikely(err)) | 710 | if (unlikely(err)) |
| 707 | return err; | 711 | return err; |
| 708 | 712 | ||
| 713 | if (dst->xdp_prog && skb_cloned(skb)) { | ||
| 714 | struct sk_buff *nskb; | ||
| 715 | |||
| 716 | nskb = skb_copy(skb, GFP_ATOMIC); | ||
| 717 | if (!nskb) | ||
| 718 | return -ENOMEM; | ||
| 719 | |||
| 720 | nskb->mac_len = skb->mac_len; | ||
| 721 | consume_skb(skb); | ||
| 722 | skb = nskb; | ||
| 723 | } | ||
| 724 | |||
| 709 | /* Redirect has already succeeded semantically at this point, so we just | 725 | /* Redirect has already succeeded semantically at this point, so we just |
| 710 | * return 0 even if packet is dropped. Helper below takes care of | 726 | * return 0 even if packet is dropped. Helper below takes care of |
| 711 | * freeing skb. | 727 | * freeing skb. |
| @@ -726,6 +742,9 @@ static int dev_map_redirect_clone(struct bpf_dtab_netdev *dst, · 14 unmodified lines | |||
| 726 | struct sk_buff *nskb; | 742 | struct sk_buff *nskb; |
| 727 | int err; | 743 | int err; |
| 728 | 744 | ||
| 745 | if (unlikely(skb_is_nonlinear(skb))) | ||
| 746 | return -EOPNOTSUPP; | ||
| 747 | |||
| 729 | nskb = skb_clone(skb, GFP_ATOMIC); | 748 | nskb = skb_clone(skb, GFP_ATOMIC); |
| 730 | if (!nskb) | 749 | if (!nskb) |
| 731 | return -ENOMEM; | 750 | return -ENOMEM; |
| @@ -870,7 +870,7 @@ int bpf_convert_ctx_accesses(struct bpf_verifier_env *env) · 869 unmodified lines | |||
| 870 | case PTR_TO_BTF_ID: | 870 | case PTR_TO_BTF_ID: |
| 871 | case PTR_TO_BTF_ID | PTR_UNTRUSTED: | 871 | case PTR_TO_BTF_ID | PTR_UNTRUSTED: |
| 872 | /* PTR_TO_BTF_ID | MEM_ALLOC always has a valid lifetime, unlike | 872 | /* PTR_TO_BTF_ID | MEM_ALLOC always has a valid lifetime, unlike |
| 873 | * PTR_TO_BTF_ID, and an active ref_obj_id, but the same cannot | 873 | * PTR_TO_BTF_ID, and an active referenced id, but the same cannot |
| 874 | * be said once it is marked PTR_UNTRUSTED, hence we must handle | 874 | * be said once it is marked PTR_UNTRUSTED, hence we must handle |
| 875 | * any faults for loads into such types. BPF_WRITE is disallowed | 875 | * any faults for loads into such types. BPF_WRITE is disallowed |
| 876 | * for this case. | 876 | * for this case. |
| @@ -1265,6 +1265,7 @@ static int jit_subprogs(struct bpf_verifier_env *env) · 388 unmodified lines | |||
| 1265 | prog->aux->real_func_cnt = env->subprog_cnt; | 1265 | prog->aux->real_func_cnt = env->subprog_cnt; |
| 1266 | prog->aux->bpf_exception_cb = (void *)func[env->exception_callback_subprog]->bpf_func; | 1266 | prog->aux->bpf_exception_cb = (void *)func[env->exception_callback_subprog]->bpf_func; |
| 1267 | prog->aux->exception_boundary = func[0]->aux->exception_boundary; | 1267 | prog->aux->exception_boundary = func[0]->aux->exception_boundary; |
| 1268 | prog->aux->stack_arg_sp_adjust = func[0]->aux->stack_arg_sp_adjust; | ||
| 1268 | bpf_prog_jit_attempt_done(prog); | 1269 | bpf_prog_jit_attempt_done(prog); |
| 1269 | return 0; | 1270 | return 0; |
| 1270 | out_free: | 1271 | out_free: |
| @@ -1378,9 +1379,21 @@ int bpf_fixup_call_args(struct bpf_verifier_env *env) · 107 unmodified lines | |||
| 1378 | struct bpf_prog *prog = env->prog; | 1379 | struct bpf_prog *prog = env->prog; |
| 1379 | struct bpf_insn *insn = prog->insnsi; | 1380 | struct bpf_insn *insn = prog->insnsi; |
| 1380 | bool has_kfunc_call = bpf_prog_has_kfunc_call(prog); | 1381 | bool has_kfunc_call = bpf_prog_has_kfunc_call(prog); |
| 1381 | int i, depth; | 1382 | int depth; |
| 1382 | #endif | 1383 | #endif |
| 1383 | int err = 0; | 1384 | int i, err = 0; |
| 1385 | |||
| 1386 | for (i = 0; i < env->subprog_cnt; i++) { | ||
| 1387 | struct bpf_subprog_info *subprog = &env->subprog_info[i]; | ||
| 1388 | u16 outgoing = subprog->stack_arg_cnt - bpf_in_stack_arg_cnt(subprog); | ||
| 1389 | |||
| 1390 | if (subprog->max_out_stack_arg_cnt > outgoing) { | ||
| 1391 | verbose(env, | ||
| 1392 | "func#%d writes %u stack arg slots, but calls only require %u\n", | ||
| 1393 | i, subprog->max_out_stack_arg_cnt, outgoing); | ||
| 1394 | return -EINVAL; | ||
| 1395 | } | ||
| 1396 | } | ||
| 1384 | 1397 | ||
| 1385 | if (env->prog->jit_requested && | 1398 | if (env->prog->jit_requested && |
| 1386 | !bpf_prog_is_offloaded(env->prog->aux)) { | 1399 | !bpf_prog_is_offloaded(env->prog->aux)) { |
| @@ -1395,6 +1408,12 @@ int bpf_fixup_call_args(struct bpf_verifier_env *env) · 8 unmodified lines | |||
| 1395 | verbose(env, "calling kernel functions are not allowed in non-JITed programs\n"); | 1408 | verbose(env, "calling kernel functions are not allowed in non-JITed programs\n"); |
| 1396 | return -EINVAL; | 1409 | return -EINVAL; |
| 1397 | } | 1410 | } |
| 1411 | for (i = 0; i < env->subprog_cnt; i++) { | ||
| 1412 | if (bpf_in_stack_arg_cnt(&env->subprog_info[i])) { | ||
| 1413 | verbose(env, "stack args are not supported in non-JITed programs\n"); | ||
| 1414 | return -EINVAL; | ||
| 1415 | } | ||
| 1416 | } | ||
| 1398 | if (env->subprog_cnt > 1 && env->prog->aux->tail_call_reachable) { | 1417 | if (env->subprog_cnt > 1 && env->prog->aux->tail_call_reachable) { |
| 1399 | /* When JIT fails the progs with bpf2bpf calls and tail_calls | 1418 | /* When JIT fails the progs with bpf2bpf calls and tail_calls |
| 1400 | * have to be rejected, since interpreter doesn't support them yet. | 1419 | * have to be rejected, since interpreter doesn't support them yet. |
| @@ -2167,6 +2186,8 @@ patch_map_ops_generic: · 766 unmodified lines | |||
| 2167 | insn->imm == BPF_FUNC_get_func_ret) { | 2186 | insn->imm == BPF_FUNC_get_func_ret) { |
| 2168 | if (eatype == BPF_TRACE_FEXIT || | 2187 | if (eatype == BPF_TRACE_FEXIT || |
| 2169 | eatype == BPF_TRACE_FSESSION || | 2188 | eatype == BPF_TRACE_FSESSION || |
| 2189 | eatype == BPF_TRACE_FEXIT_MULTI || | ||
| 2190 | eatype == BPF_TRACE_FSESSION_MULTI || | ||
| 2170 | eatype == BPF_MODIFY_RETURN) { | 2191 | eatype == BPF_MODIFY_RETURN) { |
| 2171 | /* Load nr_args from ctx - 8 */ | 2192 | /* Load nr_args from ctx - 8 */ |
| 2172 | insn_buf[0] = BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, -8); | 2193 | insn_buf[0] = BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, -8); |
| @@ -9,6 +9,7 @@ · 8 unmodified lines | |||
| 9 | #include <linux/rculist_nulls.h> | 9 | #include <linux/rculist_nulls.h> |
| 10 | #include <linux/rcupdate_wait.h> | 10 | #include <linux/rcupdate_wait.h> |
| 11 | #include <linux/random.h> | 11 | #include <linux/random.h> |
| 12 | #include <linux/rhashtable.h> | ||
| 12 | #include <uapi/linux/btf.h> | 13 | #include <uapi/linux/btf.h> |
| 13 | #include <linux/rcupdate_trace.h> | 14 | #include <linux/rcupdate_trace.h> |
| 14 | #include <linux/btf_ids.h> | 15 | #include <linux/btf_ids.h> |
| @@ -242,6 +243,10 @@ static void htab_free_prealloced_fields(struct bpf_htab *htab) · 227 unmodified lines | |||
| 242 | 243 | ||
| 243 | if (IS_ERR_OR_NULL(htab->map.record)) | 244 | if (IS_ERR_OR_NULL(htab->map.record)) |
| 244 | return; | 245 | return; |
| 246 | /* | ||
| 247 | * Preallocated maps do not have a bpf_mem_alloc destructor, so fully | ||
| 248 | * destroy every element, including the extra elements. | ||
| 249 | */ | ||
| 245 | if (htab_has_extra_elems(htab)) | 250 | if (htab_has_extra_elems(htab)) |
| 246 | num_entries += num_possible_cpus(); | 251 | num_entries += num_possible_cpus(); |
| 247 | for (i = 0; i < num_entries; i++) { | 252 | for (i = 0; i < num_entries; i++) { |
| @@ -496,28 +501,26 @@ static void htab_dtor_ctx_free(void *ctx) · 248 unmodified lines | |||
| 496 | kfree(ctx); | 501 | kfree(ctx); |
| 497 | } | 502 | } |
| 498 | 503 | ||
| 499 | static int htab_set_dtor(struct bpf_htab *htab, void (*dtor)(void *, void *)) | 504 | static int bpf_ma_set_dtor(struct bpf_map *map, struct bpf_mem_alloc *ma, |
| 505 | void (*dtor)(void *, void *)) | ||
| 500 | { | 506 | { |
| 501 | u32 key_size = htab->map.key_size; | ||
| 502 | struct bpf_mem_alloc *ma; | ||
| 503 | struct htab_btf_record *hrec; | 507 | struct htab_btf_record *hrec; |
| 504 | int err; | 508 | int err; |
| 505 | 509 | ||
| 506 | /* No need for dtors. */ | 510 | /* No need for dtors. */ |
| 507 | if (IS_ERR_OR_NULL(htab->map.record)) | 511 | if (IS_ERR_OR_NULL(map->record)) |
| 508 | return 0; | 512 | return 0; |
| 509 | 513 | ||
| 510 | hrec = kzalloc(sizeof(*hrec), GFP_KERNEL); | 514 | hrec = kzalloc(sizeof(*hrec), GFP_KERNEL); |
| 511 | if (!hrec) | 515 | if (!hrec) |
| 512 | return -ENOMEM; | 516 | return -ENOMEM; |
| 513 | hrec->key_size = key_size; | 517 | hrec->key_size = map->key_size; |
| 514 | hrec->record = btf_record_dup(htab->map.record); | 518 | hrec->record = btf_record_dup(map->record); |
| 515 | if (IS_ERR(hrec->record)) { | 519 | if (IS_ERR(hrec->record)) { |
| 516 | err = PTR_ERR(hrec->record); | 520 | err = PTR_ERR(hrec->record); |
| 517 | kfree(hrec); | 521 | kfree(hrec); |
| 518 | return err; | 522 | return err; |
| 519 | } | 523 | } |
| 520 | ma = htab_is_percpu(htab) ? &htab->pcpu_ma : &htab->ma; | ||
| 521 | bpf_mem_alloc_set_dtor(ma, dtor, htab_dtor_ctx_free, hrec); | 524 | bpf_mem_alloc_set_dtor(ma, dtor, htab_dtor_ctx_free, hrec); |
| 522 | return 0; | 525 | return 0; |
| 523 | } | 526 | } |
| @@ -534,9 +537,9 @@ static int htab_map_check_btf(struct bpf_map *map, const struct btf *btf, · 10 unmodified lines | |||
| 534 | * populated in htab_map_alloc(), so it will always appear as NULL. | 537 | * populated in htab_map_alloc(), so it will always appear as NULL. |
| 535 | */ | 538 | */ |
| 536 | if (htab_is_percpu(htab)) | 539 | if (htab_is_percpu(htab)) |
| 537 | return htab_set_dtor(htab, htab_pcpu_mem_dtor); | 540 | return bpf_ma_set_dtor(map, &htab->pcpu_ma, htab_pcpu_mem_dtor); |
| 538 | else | 541 | else |
| 539 | return htab_set_dtor(htab, htab_mem_dtor); | 542 | return bpf_ma_set_dtor(map, &htab->ma, htab_mem_dtor); |
| 540 | } | 543 | } |
| 541 | 544 | ||
| 542 | static struct bpf_map *htab_map_alloc(union bpf_attr *attr) | 545 | static struct bpf_map *htab_map_alloc(union bpf_attr *attr) |
| @@ -834,8 +837,8 @@ static int htab_lru_map_gen_lookup(struct bpf_map *map, · 291 unmodified lines | |||
| 834 | return insn - insn_buf; | 837 | return insn - insn_buf; |
| 835 | } | 838 | } |
| 836 | 839 | ||
| 837 | static void check_and_free_fields(struct bpf_htab *htab, | 840 | static void check_and_cancel_fields(struct bpf_htab *htab, |
| 838 | struct htab_elem *elem) | 841 | struct htab_elem *elem) |
| 839 | { | 842 | { |
| 840 | if (IS_ERR_OR_NULL(htab->map.record)) | 843 | if (IS_ERR_OR_NULL(htab->map.record)) |
| 841 | return; | 844 | return; |
| @@ -845,11 +848,11 @@ static void check_and_free_fields(struct bpf_htab *htab, · 3 unmodified lines | |||
| 845 | int cpu; | 848 | int cpu; |
| 846 | 849 | ||
| 847 | for_each_possible_cpu(cpu) | 850 | for_each_possible_cpu(cpu) |
| 848 | bpf_obj_free_fields(htab->map.record, per_cpu_ptr(pptr, cpu)); | 851 | bpf_obj_cancel_fields(&htab->map, per_cpu_ptr(pptr, cpu)); |
| 849 | } else { | 852 | } else { |
| 850 | void *map_value = htab_elem_value(elem, htab->map.key_size); | 853 | void *map_value = htab_elem_value(elem, htab->map.key_size); |
| 851 | 854 | ||
| 852 | bpf_obj_free_fields(htab->map.record, map_value); | 855 | bpf_obj_cancel_fields(&htab->map, map_value); |
| 853 | } | 856 | } |
| 854 | } | 857 | } |
| 855 | 858 | ||
| @@ -884,7 +887,7 @@ static bool htab_lru_map_delete_node(void *arg, struct bpf_lru_node *node) · 28 unmodified lines | |||
| 884 | htab_unlock_bucket(b, flags); | 887 | htab_unlock_bucket(b, flags); |
| 885 | 888 | ||
| 886 | if (l == tgt_l) | 889 | if (l == tgt_l) |
| 887 | check_and_free_fields(htab, l); | 890 | check_and_cancel_fields(htab, l); |
| 888 | return l == tgt_l; | 891 | return l == tgt_l; |
| 889 | } | 892 | } |
| 890 | 893 | ||
| @@ -949,7 +952,7 @@ find_first_elem: · 58 unmodified lines | |||
| 949 | 952 | ||
| 950 | static void htab_elem_free(struct bpf_htab *htab, struct htab_elem *l) | 953 | static void htab_elem_free(struct bpf_htab *htab, struct htab_elem *l) |
| 951 | { | 954 | { |
| 952 | check_and_free_fields(htab, l); | 955 | check_and_cancel_fields(htab, l); |
| 953 | 956 | ||
| 954 | if (htab->map.map_type == BPF_MAP_TYPE_PERCPU_HASH) | 957 | if (htab->map.map_type == BPF_MAP_TYPE_PERCPU_HASH) |
| 955 | bpf_mem_cache_free(&htab->pcpu_ma, l->ptr_to_pptr); | 958 | bpf_mem_cache_free(&htab->pcpu_ma, l->ptr_to_pptr); |
| @@ -1002,7 +1005,7 @@ static void free_htab_elem(struct bpf_htab *htab, struct htab_elem *l) · 46 unmodified lines | |||
| 1002 | 1005 | ||
| 1003 | if (htab_is_prealloc(htab)) { | 1006 | if (htab_is_prealloc(htab)) { |
| 1004 | bpf_map_dec_elem_count(&htab->map); | 1007 | bpf_map_dec_elem_count(&htab->map); |
| 1005 | check_and_free_fields(htab, l); | 1008 | check_and_cancel_fields(htab, l); |
| 1006 | pcpu_freelist_push(&htab->freelist, &l->fnode); | 1009 | pcpu_freelist_push(&htab->freelist, &l->fnode); |
| 1007 | } else { | 1010 | } else { |
| 1008 | dec_elem_count(htab); | 1011 | dec_elem_count(htab); |
| @@ -1019,7 +1022,7 @@ static void pcpu_copy_value(struct bpf_htab *htab, void __percpu *pptr, · 10 unmodified lines | |||
| 1019 | /* copy true value_size bytes */ | 1022 | /* copy true value_size bytes */ |
| 1020 | ptr = this_cpu_ptr(pptr); | 1023 | ptr = this_cpu_ptr(pptr); |
| 1021 | copy_map_value(&htab->map, ptr, value); | 1024 | copy_map_value(&htab->map, ptr, value); |
| 1022 | bpf_obj_free_fields(htab->map.record, ptr); | 1025 | bpf_obj_cancel_fields(&htab->map, ptr); |
| 1023 | } else { | 1026 | } else { |
| 1024 | u32 size = round_up(htab->map.value_size, 8); | 1027 | u32 size = round_up(htab->map.value_size, 8); |
| 1025 | void *val; | 1028 | void *val; |
| @@ -1029,7 +1032,7 @@ static void pcpu_copy_value(struct bpf_htab *htab, void __percpu *pptr, · 3 unmodified lines | |||
| 1029 | cpu = map_flags >> 32; | 1032 | cpu = map_flags >> 32; |
| 1030 | ptr = per_cpu_ptr(pptr, cpu); | 1033 | ptr = per_cpu_ptr(pptr, cpu); |
| 1031 | copy_map_value(&htab->map, ptr, value); | 1034 | copy_map_value(&htab->map, ptr, value); |
| 1032 | bpf_obj_free_fields(htab->map.record, ptr); | 1035 | bpf_obj_cancel_fields(&htab->map, ptr); |
| 1033 | return; | 1036 | return; |
| 1034 | } | 1037 | } |
| 1035 | 1038 | ||
| @@ -1037,7 +1040,7 @@ static void pcpu_copy_value(struct bpf_htab *htab, void __percpu *pptr, · 1 unmodified line | |||
| 1037 | ptr = per_cpu_ptr(pptr, cpu); | 1040 | ptr = per_cpu_ptr(pptr, cpu); |
| 1038 | val = (map_flags & BPF_F_ALL_CPUS) ? value : value + size * cpu; | 1041 | val = (map_flags & BPF_F_ALL_CPUS) ? value : value + size * cpu; |
| 1039 | copy_map_value(&htab->map, ptr, val); | 1042 | copy_map_value(&htab->map, ptr, val); |
| 1040 | bpf_obj_free_fields(htab->map.record, ptr); | 1043 | bpf_obj_cancel_fields(&htab->map, ptr); |
| 1041 | } | 1044 | } |
| 1042 | } | 1045 | } |
| 1043 | } | 1046 | } |
| @@ -1253,11 +1256,11 @@ static long htab_map_update_elem(struct bpf_map *map, void *key, void *value, · 209 unmodified lines | |||
| 1253 | if (l_old) { | 1256 | if (l_old) { |
| 1254 | hlist_nulls_del_rcu(&l_old->hash_node); | 1257 | hlist_nulls_del_rcu(&l_old->hash_node); |
| 1255 | 1258 | ||
| 1256 | /* l_old has already been stashed in htab->extra_elems, free | 1259 | /* l_old has already been stashed in htab->extra_elems, cancel |
| 1257 | * its special fields before it is available for reuse. | 1260 | * its reusable special fields before it is available for reuse. |
| 1258 | */ | 1261 | */ |
| 1259 | if (htab_is_prealloc(htab)) | 1262 | if (htab_is_prealloc(htab)) |
| 1260 | check_and_free_fields(htab, l_old); | 1263 | check_and_cancel_fields(htab, l_old); |
| 1261 | } | 1264 | } |
| 1262 | htab_unlock_bucket(b, flags); | 1265 | htab_unlock_bucket(b, flags); |
| 1263 | if (l_old && !htab_is_prealloc(htab)) | 1266 | if (l_old && !htab_is_prealloc(htab)) |
| @@ -1270,7 +1273,7 @@ err: · 6 unmodified lines | |||
| 1270 | 1273 | ||
| 1271 | static void htab_lru_push_free(struct bpf_htab *htab, struct htab_elem *elem) | 1274 | static void htab_lru_push_free(struct bpf_htab *htab, struct htab_elem *elem) |
| 1272 | { | 1275 | { |
| 1273 | check_and_free_fields(htab, elem); | 1276 | check_and_cancel_fields(htab, elem); |
| 1274 | bpf_map_dec_elem_count(&htab->map); | 1277 | bpf_map_dec_elem_count(&htab->map); |
| 1275 | bpf_lru_push_free(&htab->lru, &elem->lru_node); | 1278 | bpf_lru_push_free(&htab->lru, &elem->lru_node); |
| 1276 | } | 1279 | } |
| @@ -2739,3 +2742,794 @@ const struct bpf_map_ops htab_of_maps_map_ops = { · 1462 unmodified lines | |||
| 2739 | BATCH_OPS(htab), | 2742 | BATCH_OPS(htab), |
| 2740 | .map_btf_id = &htab_map_btf_ids[0], | 2743 | .map_btf_id = &htab_map_btf_ids[0], |
| 2741 | }; | 2744 | }; |
| 2745 | |||
| 2746 | struct rhtab_elem { | ||
| 2747 | struct rhash_head node; | ||
| 2748 | /* key bytes, then value bytes follow */ | ||
| 2749 | u8 data[] __aligned(8); | ||
| 2750 | }; | ||
| 2751 | |||
| 2752 | struct bpf_rhtab { | ||
| 2753 | struct bpf_map map; | ||
| 2754 | struct rhashtable ht; | ||
| 2755 | struct bpf_mem_alloc ma; | ||
| 2756 | u32 elem_size; | ||
| 2757 | bool freeing_internal; | ||
| 2758 | }; | ||
| 2759 | |||
| 2760 | static const struct rhashtable_params rhtab_params = { | ||
| 2761 | .head_offset = offsetof(struct rhtab_elem, node), | ||
| 2762 | .key_offset = offsetof(struct rhtab_elem, data), | ||
| 2763 | }; | ||
| 2764 | |||
| 2765 | static inline void *rhtab_elem_value(struct rhtab_elem *l, u32 key_size) | ||
| 2766 | { | ||
| 2767 | return l->data + round_up(key_size, 8); | ||
| 2768 | } | ||
| 2769 | |||
| 2770 | /* Specialize hash function and objcmp for long sized key */ | ||
| 2771 | static __always_inline int rhtab_key_cmp_long(struct rhashtable_compare_arg *arg, | ||
| 2772 | const void *ptr) | ||
| 2773 | { | ||
| 2774 | const unsigned long key1 = *(const unsigned long *)arg->key; | ||
| 2775 | const struct rhtab_elem *key2 = ptr; | ||
| 2776 | |||
| 2777 | return key1 != *(const unsigned long *)key2->data; | ||
| 2778 | } | ||
| 2779 | |||
| 2780 | static __always_inline u32 rhtab_hashfn_long(const void *data, u32 len, u32 seed) | ||
| 2781 | { | ||
| 2782 | u64 k = *(const unsigned long *)data; | ||
| 2783 | |||
| 2784 | return (u32)(k ^ (k >> 32)) ^ seed; | ||
| 2785 | } | ||
| 2786 | |||
| 2787 | static const struct rhashtable_params rhtab_params_long = { | ||
| 2788 | .head_offset = offsetof(struct rhtab_elem, node), | ||
| 2789 | .key_offset = offsetof(struct rhtab_elem, data), | ||
| 2790 | .key_len = sizeof(long), | ||
| 2791 | .hashfn = rhtab_hashfn_long, | ||
| 2792 | .obj_cmpfn = rhtab_key_cmp_long, | ||
| 2793 | }; | ||
| 2794 | |||
| 2795 | static struct bpf_map *rhtab_map_alloc(union bpf_attr *attr) | ||
| 2796 | { | ||
| 2797 | struct rhashtable_params params; | ||
| 2798 | struct bpf_rhtab *rhtab; | ||
| 2799 | int err = 0; | ||
| 2800 | |||
| 2801 | rhtab = bpf_map_area_alloc(sizeof(*rhtab), NUMA_NO_NODE); | ||
| 2802 | if (!rhtab) | ||
| 2803 | return ERR_PTR(-ENOMEM); | ||
| 2804 | |||
| 2805 | bpf_map_init_from_attr(&rhtab->map, attr); | ||
| 2806 | |||
| 2807 | if (rhtab->map.max_entries > 1UL << 31) { | ||
| 2808 | err = -E2BIG; | ||
| 2809 | goto free_rhtab; | ||
| 2810 | } | ||
| 2811 | |||
| 2812 | rhtab->elem_size = sizeof(struct rhtab_elem) + round_up(rhtab->map.key_size, 8) + | ||
| 2813 | round_up(rhtab->map.value_size, 8); | ||
| 2814 | |||
| 2815 | params = rhtab_params; | ||
| 2816 | params.key_len = rhtab->map.key_size; | ||
| 2817 | params.nelem_hint = (u32)attr->map_extra; | ||
| 2818 | params.automatic_shrinking = true; | ||
| 2819 | |||
| 2820 | if (rhtab->map.key_size == sizeof(long)) { | ||
| 2821 | params.hashfn = rhtab_hashfn_long; | ||
| 2822 | params.obj_cmpfn = rhtab_key_cmp_long; | ||
| 2823 | } | ||
| 2824 | |||
| 2825 | err = rhashtable_init(&rhtab->ht, ¶ms); | ||
| 2826 | if (err) | ||
| 2827 | goto free_rhtab; | ||
| 2828 | |||
| 2829 | /* Set max_elems after rhashtable_init() since init zeroes the struct */ | ||
| 2830 | rhtab->ht.max_elems = rhtab->map.max_entries; | ||
| 2831 | |||
| 2832 | err = bpf_mem_alloc_init(&rhtab->ma, rhtab->elem_size, false); | ||
| 2833 | if (err) | ||
| 2834 | goto destroy_rhtab; | ||
| 2835 | |||
| 2836 | return &rhtab->map; | ||
| 2837 | |||
| 2838 | destroy_rhtab: | ||
| 2839 | rhashtable_destroy(&rhtab->ht); | ||
| 2840 | free_rhtab: | ||
| 2841 | bpf_map_area_free(rhtab); | ||
| 2842 | return ERR_PTR(err); | ||
| 2843 | } | ||
| 2844 | |||
| 2845 | static int rhtab_map_alloc_check(union bpf_attr *attr) | ||
| 2846 | { | ||
| 2847 | if (!(attr->map_flags & BPF_F_NO_PREALLOC)) | ||
| 2848 | return -EINVAL; | ||
| 2849 | |||
| 2850 | if (attr->map_flags & BPF_F_ZERO_SEED) | ||
| 2851 | return -EINVAL; | ||
| 2852 | |||
| 2853 | if (attr->key_size > U16_MAX) | ||
| 2854 | return -E2BIG; | ||
| 2855 | |||
| 2856 | if (attr->map_extra >> 32) | ||
| 2857 | return -EINVAL; | ||
| 2858 | |||
| 2859 | if ((u32)attr->map_extra > U16_MAX) | ||
| 2860 | return -E2BIG; | ||
| 2861 | |||
| 2862 | if ((u32)attr->map_extra > attr->max_entries) | ||
| 2863 | return -EINVAL; | ||
| 2864 | |||
| 2865 | return htab_map_alloc_check(attr); | ||
| 2866 | } | ||
| 2867 | |||
| 2868 | static void rhtab_check_and_free_fields(struct bpf_rhtab *rhtab, | ||
| 2869 | struct rhtab_elem *elem) | ||
| 2870 | { | ||
| 2871 | if (IS_ERR_OR_NULL(rhtab->map.record)) | ||
| 2872 | return; | ||
| 2873 | |||
| 2874 | bpf_obj_free_fields(rhtab->map.record, | ||
| 2875 | rhtab_elem_value(elem, rhtab->map.key_size)); | ||
| 2876 | } | ||
| 2877 | |||
| 2878 | static void rhtab_mem_dtor(void *obj, void *ctx) | ||
| 2879 | { | ||
| 2880 | struct htab_btf_record *hrec = ctx; | ||
| 2881 | struct rhtab_elem *elem = obj; | ||
| 2882 | |||
| 2883 | if (IS_ERR_OR_NULL(hrec->record)) | ||
| 2884 | return; | ||
| 2885 | |||
| 2886 | bpf_obj_free_fields(hrec->record, | ||
| 2887 | rhtab_elem_value(elem, hrec->key_size)); | ||
| 2888 | } | ||
| 2889 | |||
| 2890 | static void rhtab_free_elem(void *ptr, void *arg) | ||
| 2891 | { | ||
| 2892 | struct bpf_rhtab *rhtab = arg; | ||
| 2893 | struct rhtab_elem *elem = ptr; | ||
| 2894 | |||
| 2895 | bpf_map_free_internal_structs(&rhtab->map, rhtab_elem_value(elem, rhtab->map.key_size)); | ||
| 2896 | bpf_mem_cache_free_rcu(&rhtab->ma, elem); | ||
| 2897 | } | ||
| 2898 | |||
| 2899 | static void rhtab_map_free(struct bpf_map *map) | ||
| 2900 | { | ||
| 2901 | struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map); | ||
| 2902 | |||
| 2903 | rhashtable_free_and_destroy(&rhtab->ht, rhtab_free_elem, rhtab); | ||
| 2904 | bpf_mem_alloc_destroy(&rhtab->ma); | ||
| 2905 | bpf_map_area_free(rhtab); | ||
| 2906 | } | ||
| 2907 | |||
| 2908 | static void *rhtab_lookup_elem(struct bpf_map *map, void *key) | ||
| 2909 | { | ||
| 2910 | struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map); | ||
| 2911 | |||
| 2912 | /* Hold RCU lock in case sleepable program calls via gen_lookup */ | ||
| 2913 | guard(rcu)(); | ||
| 2914 | |||
| 2915 | if (map->key_size == sizeof(long)) | ||
| 2916 | return rhashtable_lookup_likely(&rhtab->ht, key, rhtab_params_long); | ||
| 2917 | |||
| 2918 | return rhashtable_lookup_likely(&rhtab->ht, key, rhtab_params); | ||
| 2919 | } | ||
| 2920 | |||
| 2921 | static void *rhtab_map_lookup_elem(struct bpf_map *map, void *key) __must_hold(RCU) | ||
| 2922 | { | ||
| 2923 | struct rhtab_elem *l; | ||
| 2924 | |||
| 2925 | l = rhtab_lookup_elem(map, key); | ||
| 2926 | return l ? rhtab_elem_value(l, map->key_size) : NULL; | ||
| 2927 | } | ||
| 2928 | |||
| 2929 | static void rhtab_read_elem_value(struct bpf_map *map, void *dst, struct rhtab_elem *elem, | ||
| 2930 | u64 flags) | ||
| 2931 | { | ||
| 2932 | void *src = rhtab_elem_value(elem, map->key_size); | ||
| 2933 | |||
| 2934 | if (flags & BPF_F_LOCK) | ||
| 2935 | copy_map_value_locked(map, dst, src, true); | ||
| 2936 | else | ||
| 2937 | copy_map_value(map, dst, src); | ||
| 2938 | } | ||
| 2939 | |||
| 2940 | static int rhtab_delete_elem(struct bpf_rhtab *rhtab, struct rhtab_elem *elem, void *copy, | ||
| 2941 | u64 flags) | ||
| 2942 | { | ||
| 2943 | int err; | ||
| 2944 | |||
| 2945 | /* | ||
| 2946 | * disable_instrumentation() mitigates the deadlock for programs running in NMI context. | ||
| 2947 | * rhashtable locks bucket with local_irq_save(). Only NMI programs may reenter | ||
| 2948 | * rhashtable code, bpf_disable_instrumentation() disables programs running in NMI, except | ||
| 2949 | * raw tracepoints, which we don't have in rhashtable. | ||
| 2950 | */ | ||
| 2951 | bpf_disable_instrumentation(); | ||
| 2952 | |||
| 2953 | if (rhtab->map.key_size == sizeof(long)) | ||
| 2954 | err = rhashtable_remove_fast(&rhtab->ht, &elem->node, rhtab_params_long); | ||
| 2955 | else | ||
| 2956 | err = rhashtable_remove_fast(&rhtab->ht, &elem->node, rhtab_params); | ||
| 2957 | |||
| 2958 | bpf_enable_instrumentation(); | ||
| 2959 | |||
| 2960 | if (err) | ||
| 2961 | return err; | ||
| 2962 | |||
| 2963 | if (copy) { | ||
| 2964 | rhtab_read_elem_value(&rhtab->map, copy, elem, flags); | ||
| 2965 | check_and_init_map_value(&rhtab->map, copy); | ||
| 2966 | } | ||
| 2967 | /* Release internal structs: kptr, bpf_timer, task_work, wq */ | ||
| 2968 | rhtab_check_and_free_fields(rhtab, elem); | ||
| 2969 | bpf_mem_cache_free_rcu(&rhtab->ma, elem); | ||
| 2970 | return 0; | ||
| 2971 | } | ||
| 2972 | |||
| 2973 | |||
| 2974 | static long rhtab_map_delete_elem(struct bpf_map *map, void *key) | ||
| 2975 | { | ||
| 2976 | struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map); | ||
| 2977 | struct rhtab_elem *elem; | ||
| 2978 | |||
| 2979 | guard(rcu)(); | ||
| 2980 | |||
| 2981 | elem = rhtab_lookup_elem(map, key); | ||
| 2982 | if (!elem) | ||
| 2983 | return -ENOENT; | ||
| 2984 | |||
| 2985 | return rhtab_delete_elem(rhtab, elem, NULL, 0); | ||
| 2986 | } | ||
| 2987 | |||
| 2988 | static int rhtab_map_lookup_and_delete_elem(struct bpf_map *map, void *key, void *value, u64 flags) | ||
| 2989 | { | ||
| 2990 | struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map); | ||
| 2991 | struct rhtab_elem *elem; | ||
| 2992 | int err; | ||
| 2993 | |||
| 2994 | err = bpf_map_check_op_flags(map, flags, BPF_F_LOCK); | ||
| 2995 | if (err) | ||
| 2996 | return err; | ||
| 2997 | |||
| 2998 | guard(rcu)(); | ||
| 2999 | |||
| 3000 | elem = rhtab_lookup_elem(map, key); | ||
| 3001 | if (!elem) | ||
| 3002 | return -ENOENT; | ||
| 3003 | |||
| 3004 | return rhtab_delete_elem(rhtab, elem, value, flags); | ||
| 3005 | } | ||
| 3006 | |||
| 3007 | static long rhtab_map_update_existing(struct bpf_map *map, struct rhtab_elem *elem, void *value, | ||
| 3008 | u64 map_flags) | ||
| 3009 | { | ||
| 3010 | struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map); | ||
| 3011 | void *old_val = rhtab_elem_value(elem, map->key_size); | ||
| 3012 | |||
| 3013 | if (map_flags & BPF_NOEXIST) | ||
| 3014 | return -EEXIST; | ||
| 3015 | |||
| 3016 | if (map_flags & BPF_F_LOCK) | ||
| 3017 | copy_map_value_locked(map, old_val, value, false); | ||
| 3018 | else | ||
| 3019 | copy_map_value(map, old_val, value); | ||
| 3020 | |||
| 3021 | /* | ||
| 3022 | * Torn reads: a concurrent reader without BPF_F_LOCK may observe | ||
| 3023 | * the value mid-copy. Callers requiring consistent reads must use | ||
| 3024 | * BPF_F_LOCK, matching arraymap semantics. | ||
| 3025 | * | ||
| 3026 | * copy_map_value() skips special-field offsets, so old timers/ | ||
| 3027 | * kptrs/etc. still sit in the slot. Cancel them after the copy | ||
| 3028 | * to match arraymap's update semantics. | ||
| 3029 | */ | ||
| 3030 | rhtab_check_and_free_fields(rhtab, elem); | ||
| 3031 | return 0; | ||
| 3032 | } | ||
| 3033 | |||
| 3034 | static long rhtab_map_update_elem(struct bpf_map *map, void *key, void *value, u64 map_flags) | ||
| 3035 | { | ||
| 3036 | struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map); | ||
| 3037 | struct rhtab_elem *elem, *tmp; | ||
| 3038 | |||
| 3039 | if (unlikely((map_flags & ~BPF_F_LOCK) > BPF_EXIST)) | ||
| 3040 | return -EINVAL; | ||
| 3041 | |||
| 3042 | if ((map_flags & BPF_F_LOCK) && !btf_record_has_field(map->record, BPF_SPIN_LOCK)) | ||
| 3043 | return -EINVAL; | ||
| 3044 | |||
| 3045 | guard(rcu)(); | ||
| 3046 | elem = rhtab_lookup_elem(map, key); | ||
| 3047 | if (elem) | ||
| 3048 | return rhtab_map_update_existing(map, elem, value, map_flags); | ||
| 3049 | |||
| 3050 | if (map_flags & BPF_EXIST) | ||
| 3051 | return -ENOENT; | ||
| 3052 | |||
| 3053 | /* | ||
| 3054 | * Reject new insertions while map_release_uref cleanup walks the | ||
| 3055 | * table. Without this, new elements could keep triggering rehash | ||
| 3056 | * and prevent the walk from terminating. | ||
| 3057 | */ | ||
| 3058 | if (READ_ONCE(rhtab->freeing_internal)) | ||
| 3059 | return -EBUSY; | ||
| 3060 | |||
| 3061 | /* Check max_entries limit before inserting new element */ | ||
| 3062 | if (atomic_read(&rhtab->ht.nelems) >= map->max_entries) | ||
| 3063 | return -E2BIG; | ||
| 3064 | |||
| 3065 | elem = bpf_mem_cache_alloc(&rhtab->ma); | ||
| 3066 | if (!elem) | ||
| 3067 | return -ENOMEM; | ||
| 3068 | |||
| 3069 | memcpy(elem->data, key, map->key_size); | ||
| 3070 | copy_map_value(map, rhtab_elem_value(elem, map->key_size), value); | ||
| 3071 | check_and_init_map_value(map, rhtab_elem_value(elem, map->key_size)); | ||
| 3072 | |||
| 3073 | /* Prevent deadlock for NMI programs attempting to take bucket lock */ | ||
| 3074 | bpf_disable_instrumentation(); | ||
| 3075 | |||
| 3076 | if (map->key_size == sizeof(long)) | ||
| 3077 | tmp = rhashtable_lookup_get_insert_fast(&rhtab->ht, &elem->node, rhtab_params_long); | ||
| 3078 | else | ||
| 3079 | tmp = rhashtable_lookup_get_insert_fast(&rhtab->ht, &elem->node, rhtab_params); | ||
| 3080 | |||
| 3081 | bpf_enable_instrumentation(); | ||
| 3082 | |||
| 3083 | if (tmp) { | ||
| 3084 | bpf_mem_cache_free(&rhtab->ma, elem); | ||
| 3085 | if (IS_ERR(tmp)) | ||
| 3086 | return PTR_ERR(tmp); | ||
| 3087 | |||
| 3088 | return rhtab_map_update_existing(map, tmp, value, map_flags); | ||
| 3089 | } | ||
| 3090 | |||
| 3091 | return 0; | ||
| 3092 | } | ||
| 3093 | |||
| 3094 | static int rhtab_map_gen_lookup(struct bpf_map *map, struct bpf_insn *insn_buf) | ||
| 3095 | { | ||
| 3096 | struct bpf_insn *insn = insn_buf; | ||
| 3097 | const int ret = BPF_REG_0; | ||
| 3098 | |||
| 3099 | BUILD_BUG_ON(!__same_type(&rhtab_lookup_elem, | ||
| 3100 | (void *(*)(struct bpf_map *map, void *key)) NULL)); | ||
| 3101 | *insn++ = BPF_EMIT_CALL(rhtab_lookup_elem); | ||
| 3102 | *insn++ = BPF_JMP_IMM(BPF_JEQ, ret, 0, 1); | ||
| 3103 | *insn++ = BPF_ALU64_IMM(BPF_ADD, ret, | ||
| 3104 | offsetof(struct rhtab_elem, data) + round_up(map->key_size, 8)); | ||
| 3105 | |||
| 3106 | return insn - insn_buf; | ||
| 3107 | } | ||
| 3108 | |||
| 3109 | static int rhtab_map_check_btf(struct bpf_map *map, const struct btf *btf, | ||
| 3110 | const struct btf_type *key_type, | ||
| 3111 | const struct btf_type *value_type) | ||
| 3112 | { | ||
| 3113 | struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map); | ||
| 3114 | |||
| 3115 | return bpf_ma_set_dtor(map, &rhtab->ma, rhtab_mem_dtor); | ||
| 3116 | } | ||
| 3117 | |||
| 3118 | static void rhtab_map_free_internal_structs(struct bpf_map *map) | ||
| 3119 | { | ||
| 3120 | struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map); | ||
| 3121 | struct rhashtable_iter iter; | ||
| 3122 | struct rhtab_elem *elem; | ||
| 3123 | |||
| 3124 | if (!bpf_map_has_internal_structs(map)) | ||
| 3125 | return; | ||
| 3126 | |||
| 3127 | /* | ||
| 3128 | * Block new insertions. Once observed, no new growth is triggered, | ||
| 3129 | * so any in-flight rehash will drain and the walker is guaranteed | ||
| 3130 | * to stop returning -EAGAIN. Treat -EAGAIN as "rehash in progress, | ||
| 3131 | * retry"; do not wait for the worker. | ||
| 3132 | */ | ||
| 3133 | WRITE_ONCE(rhtab->freeing_internal, true); | ||
| 3134 | |||
| 3135 | rhashtable_walk_enter(&rhtab->ht, &iter); | ||
| 3136 | rhashtable_walk_start(&iter); | ||
| 3137 | |||
| 3138 | while ((elem = rhashtable_walk_next(&iter))) { | ||
| 3139 | if (IS_ERR(elem)) { | ||
| 3140 | if (PTR_ERR(elem) == -EAGAIN) | ||
| 3141 | continue; | ||
| 3142 | break; | ||
| 3143 | } | ||
| 3144 | |||
| 3145 | bpf_map_free_internal_structs(map, rhtab_elem_value(elem, map->key_size)); | ||
| 3146 | |||
| 3147 | if (need_resched()) { /* Avoid stalls on large maps */ | ||
| 3148 | rhashtable_walk_stop(&iter); | ||
| 3149 | cond_resched(); | ||
| 3150 | rhashtable_walk_start(&iter); | ||
| 3151 | } | ||
| 3152 | } | ||
| 3153 | |||
| 3154 | rhashtable_walk_stop(&iter); | ||
| 3155 | rhashtable_walk_exit(&iter); | ||
| 3156 | WRITE_ONCE(rhtab->freeing_internal, false); | ||
| 3157 | } | ||
| 3158 | |||
| 3159 | static int rhtab_map_get_next_key(struct bpf_map *map, void *key, void *next_key) | ||
| 3160 | __must_hold_shared(RCU) | ||
| 3161 | { | ||
| 3162 | struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map); | ||
| 3163 | struct rhtab_elem *elem; | ||
| 3164 | |||
| 3165 | elem = rhashtable_next_key(&rhtab->ht, key); | ||
| 3166 | |||
| 3167 | /* if not found, return the first key */ | ||
| 3168 | if (PTR_ERR(elem) == -ENOENT) | ||
| 3169 | elem = rhashtable_next_key(&rhtab->ht, NULL); | ||
| 3170 | |||
| 3171 | if (IS_ERR(elem)) | ||
| 3172 | return PTR_ERR(elem); | ||
| 3173 | if (!elem) | ||
| 3174 | return -ENOENT; | ||
| 3175 | |||
| 3176 | memcpy(next_key, elem->data, map->key_size); | ||
| 3177 | return 0; | ||
| 3178 | } | ||
| 3179 | |||
| 3180 | static void rhtab_map_seq_show_elem(struct bpf_map *map, void *key, struct seq_file *m) | ||
| 3181 | { | ||
| 3182 | void *value; | ||
| 3183 | |||
| 3184 | /* Guarantee that hashtab value is not freed */ | ||
| 3185 | guard(rcu)(); | ||
| 3186 | |||
| 3187 | value = rhtab_map_lookup_elem(map, key); | ||
| 3188 | if (!value) | ||
| 3189 | return; | ||
| 3190 | |||
| 3191 | btf_type_seq_show(map->btf, map->btf_key_type_id, key, m); | ||
| 3192 | seq_puts(m, ": "); | ||
| 3193 | btf_type_seq_show(map->btf, map->btf_value_type_id, value, m); | ||
| 3194 | seq_putc(m, '\n'); | ||
| 3195 | } | ||
| 3196 | |||
| 3197 | static long bpf_each_rhash_elem(struct bpf_map *map, bpf_callback_t callback_fn, | ||
| 3198 | void *callback_ctx, u64 flags) | ||
| 3199 | { | ||
| 3200 | struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map); | ||
| 3201 | void *prev_key = NULL; | ||
| 3202 | struct rhtab_elem *elem; | ||
| 3203 | int num_elems = 0; | ||
| 3204 | u64 ret = 0; | ||
| 3205 | |||
| 3206 | cant_migrate(); | ||
| 3207 | |||
| 3208 | if (flags != 0) | ||
| 3209 | return -EINVAL; | ||
| 3210 | |||
| 3211 | rcu_read_lock(); | ||
| 3212 | /* | ||
| 3213 | * Best-effort iteration: if rhashtable is concurrently resized or | ||
| 3214 | * elements are deleted/inserted, there may be missed or duplicate | ||
| 3215 | * elements visited. | ||
| 3216 | */ | ||
| 3217 | while ((elem = rhashtable_next_key(&rhtab->ht, prev_key))) { | ||
| 3218 | if (IS_ERR(elem)) | ||
| 3219 | break; | ||
| 3220 | num_elems++; | ||
| 3221 | ret = callback_fn((u64)(long)map, | ||
| 3222 | (u64)(long)elem->data, | ||
| 3223 | (u64)(long)rhtab_elem_value(elem, map->key_size), | ||
| 3224 | (u64)(long)callback_ctx, 0); | ||
| 3225 | if (ret) | ||
| 3226 | break; | ||
| 3227 | |||
| 3228 | prev_key = elem->data; /* valid while RCU held */ | ||
| 3229 | } | ||
| 3230 | rcu_read_unlock(); | ||
| 3231 | |||
| 3232 | return num_elems; | ||
| 3233 | } | ||
| 3234 | |||
| 3235 | static u64 rhtab_map_mem_usage(const struct bpf_map *map) | ||
| 3236 | { | ||
| 3237 | struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map); | ||
| 3238 | u64 num_entries; | ||
| 3239 | |||
| 3240 | /* Excludes rhashtable bucket overhead (~ nelems * sizeof(void *) at 75% load). */ | ||
| 3241 | num_entries = atomic_read(&rhtab->ht.nelems); | ||
| 3242 | return sizeof(struct bpf_rhtab) + rhtab->elem_size * num_entries; | ||
| 3243 | } | ||
| 3244 | |||
| 3245 | static int __rhtab_map_lookup_and_delete_batch(struct bpf_map *map, | ||
| 3246 | const union bpf_attr *attr, | ||
| 3247 | union bpf_attr __user *uattr, | ||
| 3248 | bool do_delete) | ||
| 3249 | { | ||
| 3250 | struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map); | ||
| 3251 | void __user *uvalues = u64_to_user_ptr(attr->batch.values); | ||
| 3252 | void __user *ukeys = u64_to_user_ptr(attr->batch.keys); | ||
| 3253 | void __user *ubatch = u64_to_user_ptr(attr->batch.in_batch); | ||
| 3254 | void *cursor = NULL, *keys = NULL, *values = NULL, *dst_key, *dst_val; | ||
| 3255 | struct rhtab_elem **del_elems = NULL; | ||
| 3256 | u32 max_count, total, key_size, value_size, i; | ||
| 3257 | bool has_next_cursor = false; | ||
| 3258 | struct rhtab_elem *elem; | ||
| 3259 | u64 elem_map_flags, map_flags; | ||
| 3260 | int ret = 0; | ||
| 3261 | |||
| 3262 | elem_map_flags = attr->batch.elem_flags; | ||
| 3263 | ret = bpf_map_check_op_flags(map, elem_map_flags, BPF_F_LOCK); | ||
| 3264 | if (ret) | ||
| 3265 | return ret; | ||
| 3266 | |||
| 3267 | map_flags = attr->batch.flags; | ||
| 3268 | if (map_flags) | ||
| 3269 | return -EINVAL; | ||
| 3270 | |||
| 3271 | max_count = attr->batch.count; | ||
| 3272 | if (!max_count) | ||
| 3273 | return 0; | ||
| 3274 | |||
| 3275 | if (put_user(0, &uattr->batch.count)) | ||
| 3276 | return -EFAULT; | ||
| 3277 | |||
| 3278 | key_size = map->key_size; | ||
| 3279 | value_size = map->value_size; | ||
| 3280 | |||
| 3281 | keys = kvmalloc_array(max_count, key_size, GFP_USER | __GFP_NOWARN); | ||
| 3282 | values = kvmalloc_array(max_count, value_size, GFP_USER | __GFP_NOWARN); | ||
| 3283 | if (do_delete) | ||
| 3284 | del_elems = kvmalloc_array(max_count, sizeof(void *), | ||
| 3285 | GFP_USER | __GFP_NOWARN); | ||
| 3286 | cursor = kmalloc(key_size, GFP_USER | __GFP_NOWARN); | ||
| 3287 | |||
| 3288 | if (!keys || !values || !cursor || (do_delete && !del_elems)) { | ||
| 3289 | ret = -ENOMEM; | ||
| 3290 | goto free; | ||
| 3291 | } | ||
| 3292 | |||
| 3293 | if (ubatch && copy_from_user(cursor, ubatch, key_size)) { | ||
| 3294 | ret = -EFAULT; | ||
| 3295 | goto free; | ||
| 3296 | } | ||
| 3297 | |||
| 3298 | dst_key = keys; | ||
| 3299 | dst_val = values; | ||
| 3300 | total = 0; | ||
| 3301 | |||
| 3302 | rcu_read_lock(); | ||
| 3303 | |||
| 3304 | /* | ||
| 3305 | * Cursor stores the key of the next-to-process element (stashed by | ||
| 3306 | * the previous batch). Look it up directly so the element is included | ||
| 3307 | * here rather than skipped by next_key(). If the cursor was deleted | ||
| 3308 | * concurrently (or by the previous do_delete batch), return -EAGAIN | ||
| 3309 | * so userspace can distinguish a lost cursor from end-of-iteration | ||
| 3310 | * (-ENOENT) and restart from a NULL cursor. | ||
| 3311 | */ | ||
| 3312 | if (ubatch) { | ||
| 3313 | elem = rhtab_lookup_elem(map, cursor); | ||
| 3314 | if (!elem) { | ||
| 3315 | rcu_read_unlock(); | ||
| 3316 | ret = -EAGAIN; | ||
| 3317 | goto free; | ||
| 3318 | } | ||
| 3319 | } else { | ||
| 3320 | elem = rhashtable_next_key(&rhtab->ht, NULL); | ||
| 3321 | } | ||
| 3322 | |||
| 3323 | while (elem && !IS_ERR(elem) && total < max_count) { | ||
| 3324 | memcpy(dst_key, elem->data, key_size); | ||
| 3325 | rhtab_read_elem_value(map, dst_val, elem, elem_map_flags); | ||
| 3326 | check_and_init_map_value(map, dst_val); | ||
| 3327 | |||
| 3328 | if (do_delete) | ||
| 3329 | del_elems[total] = elem; | ||
| 3330 | |||
| 3331 | elem = rhashtable_next_key(&rhtab->ht, dst_key); | ||
| 3332 | dst_key += key_size; | ||
| 3333 | dst_val += value_size; | ||
| 3334 | total++; | ||
| 3335 | |||
| 3336 | /* Bail to userspace to avoid stalls. */ | ||
| 3337 | if (need_resched()) | ||
| 3338 | break; | ||
| 3339 | } | ||
| 3340 | |||
| 3341 | if (elem && !IS_ERR(elem)) { | ||
| 3342 | /* Stash next-to-process key as cursor for the next batch. */ | ||
| 3343 | memcpy(cursor, elem->data, key_size); | ||
| 3344 | has_next_cursor = true; | ||
| 3345 | } | ||
| 3346 | |||
| 3347 | if (do_delete) { | ||
| 3348 | for (i = 0; i < total; i++) | ||
| 3349 | rhtab_delete_elem(rhtab, del_elems[i], NULL, 0); | ||
| 3350 | } | ||
| 3351 | |||
| 3352 | rcu_read_unlock(); | ||
| 3353 | |||
| 3354 | if (total == 0) { | ||
| 3355 | ret = -ENOENT; | ||
| 3356 | goto free; | ||
| 3357 | } | ||
| 3358 | |||
| 3359 | /* No more elements after this batch. */ | ||
| 3360 | if (!has_next_cursor) | ||
| 3361 | ret = -ENOENT; | ||
| 3362 | |||
| 3363 | if (copy_to_user(ukeys, keys, (size_t)total * key_size) || | ||
| 3364 | copy_to_user(uvalues, values, (size_t)total * value_size) || | ||
| 3365 | put_user(total, &uattr->batch.count) || | ||
| 3366 | (has_next_cursor && | ||
| 3367 | copy_to_user(u64_to_user_ptr(attr->batch.out_batch), | ||
| 3368 | cursor, key_size))) { | ||
| 3369 | ret = -EFAULT; | ||
| 3370 | goto free; | ||
| 3371 | } | ||
| 3372 | |||
| 3373 | free: | ||
| 3374 | kfree(cursor); | ||
| 3375 | kvfree(keys); | ||
| 3376 | kvfree(values); | ||
| 3377 | kvfree(del_elems); | ||
| 3378 | return ret; | ||
| 3379 | } | ||
| 3380 | |||
| 3381 | static int rhtab_map_lookup_batch(struct bpf_map *map, const union bpf_attr *attr, | ||
| 3382 | union bpf_attr __user *uattr) | ||
| 3383 | { | ||
| 3384 | return __rhtab_map_lookup_and_delete_batch(map, attr, uattr, false); | ||
| 3385 | } | ||
| 3386 | |||
| 3387 | static int rhtab_map_lookup_and_delete_batch(struct bpf_map *map, const union bpf_attr *attr, | ||
| 3388 | union bpf_attr __user *uattr) | ||
| 3389 | { | ||
| 3390 | return __rhtab_map_lookup_and_delete_batch(map, attr, uattr, true); | ||
| 3391 | } | ||
| 3392 | |||
| 3393 | struct bpf_iter_seq_rhash_map_info { | ||
| 3394 | struct bpf_map *map; | ||
| 3395 | struct bpf_rhtab *rhtab; | ||
| 3396 | struct rhashtable_iter iter; | ||
| 3397 | }; | ||
| 3398 | |||
| 3399 | static void *bpf_rhash_map_seq_start(struct seq_file *seq, loff_t *pos) | ||
| 3400 | __acquires(RCU) | ||
| 3401 | { | ||
| 3402 | struct bpf_iter_seq_rhash_map_info *info = seq->private; | ||
| 3403 | struct rhtab_elem *elem; | ||
| 3404 | |||
| 3405 | rhashtable_walk_start(&info->iter); | ||
| 3406 | /* | ||
| 3407 | * Re-deliver the element returned by walk_next() at the end of the | ||
| 3408 | * previous read() — bpf_seq_read may have stopped before show() | ||
| 3409 | * consumed it. Rehash rewinds the walker; retry on -EAGAIN. | ||
| 3410 | */ | ||
| 3411 | do { | ||
| 3412 | elem = rhashtable_walk_peek(&info->iter); | ||
| 3413 | } while (PTR_ERR(elem) == -EAGAIN); | ||
| 3414 | |||
| 3415 | if (IS_ERR(elem)) | ||
| 3416 | return NULL; | ||
| 3417 | |||
| 3418 | if (elem && *pos == 0) | ||
| 3419 | ++*pos; | ||
| 3420 | return elem; | ||
| 3421 | } | ||
| 3422 | |||
| 3423 | static void *bpf_rhash_map_seq_next(struct seq_file *seq, void *v, loff_t *pos) | ||
| 3424 | { | ||
| 3425 | struct bpf_iter_seq_rhash_map_info *info = seq->private; | ||
| 3426 | struct rhtab_elem *elem; | ||
| 3427 | |||
| 3428 | ++*pos; | ||
| 3429 | |||
| 3430 | /* Rehash rewinds the walker; retry until it stops returning -EAGAIN. */ | ||
| 3431 | do { | ||
| 3432 | elem = rhashtable_walk_next(&info->iter); | ||
| 3433 | } while (PTR_ERR(elem) == -EAGAIN); | ||
| 3434 | |||
| 3435 | if (IS_ERR(elem)) | ||
| 3436 | return NULL; | ||
| 3437 | return elem; | ||
| 3438 | } | ||
| 3439 | |||
| 3440 | static int __bpf_rhash_map_seq_show(struct seq_file *seq, | ||
| 3441 | struct rhtab_elem *elem) | ||
| 3442 | { | ||
| 3443 | struct bpf_iter_seq_rhash_map_info *info = seq->private; | ||
| 3444 | struct bpf_iter__bpf_map_elem ctx = {}; | ||
| 3445 | struct bpf_iter_meta meta; | ||
| 3446 | struct bpf_prog *prog; | ||
| 3447 | int ret = 0; | ||
| 3448 | |||
| 3449 | meta.seq = seq; | ||
| 3450 | prog = bpf_iter_get_info(&meta, elem == NULL); | ||
| 3451 | if (prog) { | ||
| 3452 | ctx.meta = &meta; | ||
| 3453 | ctx.map = info->map; | ||
| 3454 | if (elem) { | ||
| 3455 | ctx.key = elem->data; | ||
| 3456 | ctx.value = rhtab_elem_value(elem, info->map->key_size); | ||
| 3457 | } | ||
| 3458 | ret = bpf_iter_run_prog(prog, &ctx); | ||
| 3459 | } | ||
| 3460 | |||
| 3461 | return ret; | ||
| 3462 | } | ||
| 3463 | |||
| 3464 | static int bpf_rhash_map_seq_show(struct seq_file *seq, void *v) | ||
| 3465 | { | ||
| 3466 | return __bpf_rhash_map_seq_show(seq, v); | ||
| 3467 | } | ||
| 3468 | |||
| 3469 | static void bpf_rhash_map_seq_stop(struct seq_file *seq, void *v) | ||
| 3470 | __releases(RCU) | ||
| 3471 | { | ||
| 3472 | struct bpf_iter_seq_rhash_map_info *info = seq->private; | ||
| 3473 | |||
| 3474 | if (!v) | ||
| 3475 | (void)__bpf_rhash_map_seq_show(seq, NULL); | ||
| 3476 | |||
| 3477 | rhashtable_walk_stop(&info->iter); | ||
| 3478 | } | ||
| 3479 | |||
| 3480 | static int bpf_iter_init_rhash_map(void *priv_data, struct bpf_iter_aux_info *aux) | ||
| 3481 | { | ||
| 3482 | struct bpf_iter_seq_rhash_map_info *info = priv_data; | ||
| 3483 | struct bpf_map *map = aux->map; | ||
| 3484 | |||
| 3485 | bpf_map_inc_with_uref(map); | ||
| 3486 | info->map = map; | ||
| 3487 | info->rhtab = container_of(map, struct bpf_rhtab, map); | ||
| 3488 | rhashtable_walk_enter(&info->rhtab->ht, &info->iter); | ||
| 3489 | return 0; | ||
| 3490 | } | ||
| 3491 | |||
| 3492 | static void bpf_iter_fini_rhash_map(void *priv_data) | ||
| 3493 | { | ||
| 3494 | struct bpf_iter_seq_rhash_map_info *info = priv_data; | ||
| 3495 | |||
| 3496 | rhashtable_walk_exit(&info->iter); | ||
| 3497 | bpf_map_put_with_uref(info->map); | ||
| 3498 | } | ||
| 3499 | |||
| 3500 | static const struct seq_operations bpf_rhash_map_seq_ops = { | ||
| 3501 | .start = bpf_rhash_map_seq_start, | ||
| 3502 | .next = bpf_rhash_map_seq_next, | ||
| 3503 | .stop = bpf_rhash_map_seq_stop, | ||
| 3504 | .show = bpf_rhash_map_seq_show, | ||
| 3505 | }; | ||
| 3506 | |||
| 3507 | static const struct bpf_iter_seq_info rhash_iter_seq_info = { | ||
| 3508 | .seq_ops = &bpf_rhash_map_seq_ops, | ||
| 3509 | .init_seq_private = bpf_iter_init_rhash_map, | ||
| 3510 | .fini_seq_private = bpf_iter_fini_rhash_map, | ||
| 3511 | .seq_priv_size = sizeof(struct bpf_iter_seq_rhash_map_info), | ||
| 3512 | }; | ||
| 3513 | |||
| 3514 | BTF_ID_LIST_SINGLE(rhtab_map_btf_ids, struct, bpf_rhtab) | ||
| 3515 | const struct bpf_map_ops rhtab_map_ops = { | ||
| 3516 | .map_meta_equal = bpf_map_meta_equal, | ||
| 3517 | .map_alloc_check = rhtab_map_alloc_check, | ||
| 3518 | .map_alloc = rhtab_map_alloc, | ||
| 3519 | .map_free = rhtab_map_free, | ||
| 3520 | .map_get_next_key = rhtab_map_get_next_key, | ||
| 3521 | .map_release_uref = rhtab_map_free_internal_structs, | ||
| 3522 | .map_check_btf = rhtab_map_check_btf, | ||
| 3523 | .map_lookup_elem = rhtab_map_lookup_elem, | ||
| 3524 | .map_lookup_and_delete_elem = rhtab_map_lookup_and_delete_elem, | ||
| 3525 | .map_update_elem = rhtab_map_update_elem, | ||
| 3526 | .map_delete_elem = rhtab_map_delete_elem, | ||
| 3527 | .map_gen_lookup = rhtab_map_gen_lookup, | ||
| 3528 | .map_seq_show_elem = rhtab_map_seq_show_elem, | ||
| 3529 | .map_set_for_each_callback_args = map_set_for_each_callback_args, | ||
| 3530 | .map_for_each_callback = bpf_each_rhash_elem, | ||
| 3531 | .map_mem_usage = rhtab_map_mem_usage, | ||
| 3532 | BATCH_OPS(rhtab), | ||
| 3533 | .map_btf_id = &rhtab_map_btf_ids[0], | ||
| 3534 | .iter_seq_info = &rhash_iter_seq_info, | ||
| 3535 | }; | ||
| @@ -1944,7 +1944,7 @@ static const struct bpf_func_proto bpf_dynptr_read_proto = { · 1943 unmodified lines | |||
| 1944 | .ret_type = RET_INTEGER, | 1944 | .ret_type = RET_INTEGER, |
| 1945 | .arg1_type = ARG_PTR_TO_UNINIT_MEM, | 1945 | .arg1_type = ARG_PTR_TO_UNINIT_MEM, |
| 1946 | .arg2_type = ARG_CONST_SIZE_OR_ZERO, | 1946 | .arg2_type = ARG_CONST_SIZE_OR_ZERO, |
| 1947 | .arg3_type = ARG_PTR_TO_DYNPTR | MEM_RDONLY, | 1947 | .arg3_type = ARG_PTR_TO_DYNPTR, |
| 1948 | .arg4_type = ARG_ANYTHING, | 1948 | .arg4_type = ARG_ANYTHING, |
| 1949 | .arg5_type = ARG_ANYTHING, | 1949 | .arg5_type = ARG_ANYTHING, |
| 1950 | }; | 1950 | }; |
| @@ -2001,7 +2001,7 @@ static const struct bpf_func_proto bpf_dynptr_write_proto = { · 50 unmodified lines | |||
| 2001 | .func = bpf_dynptr_write, | 2001 | .func = bpf_dynptr_write, |
| 2002 | .gpl_only = false, | 2002 | .gpl_only = false, |
| 2003 | .ret_type = RET_INTEGER, | 2003 | .ret_type = RET_INTEGER, |
| 2004 | .arg1_type = ARG_PTR_TO_DYNPTR | MEM_RDONLY, | 2004 | .arg1_type = ARG_PTR_TO_DYNPTR, |
| 2005 | .arg2_type = ARG_ANYTHING, | 2005 | .arg2_type = ARG_ANYTHING, |
| 2006 | .arg3_type = ARG_PTR_TO_MEM | MEM_RDONLY, | 2006 | .arg3_type = ARG_PTR_TO_MEM | MEM_RDONLY, |
| 2007 | .arg4_type = ARG_CONST_SIZE_OR_ZERO, | 2007 | .arg4_type = ARG_CONST_SIZE_OR_ZERO, |
| @@ -2044,7 +2044,7 @@ static const struct bpf_func_proto bpf_dynptr_data_proto = { · 36 unmodified lines | |||
| 2044 | .func = bpf_dynptr_data, | 2044 | .func = bpf_dynptr_data, |
| 2045 | .gpl_only = false, | 2045 | .gpl_only = false, |
| 2046 | .ret_type = RET_PTR_TO_DYNPTR_MEM_OR_NULL, | 2046 | .ret_type = RET_PTR_TO_DYNPTR_MEM_OR_NULL, |
| 2047 | .arg1_type = ARG_PTR_TO_DYNPTR | MEM_RDONLY, | 2047 | .arg1_type = ARG_PTR_TO_DYNPTR, |
| 2048 | .arg2_type = ARG_ANYTHING, | 2048 | .arg2_type = ARG_ANYTHING, |
| 2049 | .arg3_type = ARG_CONST_ALLOC_SIZE_OR_ZERO, | 2049 | .arg3_type = ARG_CONST_ALLOC_SIZE_OR_ZERO, |
| 2050 | }; | 2050 | }; |
| @@ -2247,10 +2247,11 @@ EXPORT_SYMBOL_GPL(bpf_base_func_proto); · 196 unmodified lines | |||
| 2247 | void bpf_list_head_free(const struct btf_field *field, void *list_head, | 2247 | void bpf_list_head_free(const struct btf_field *field, void *list_head, |
| 2248 | struct bpf_spin_lock *spin_lock) | 2248 | struct bpf_spin_lock *spin_lock) |
| 2249 | { | 2249 | { |
| 2250 | struct list_head *head = list_head, *orig_head = list_head; | 2250 | struct list_head *head = list_head, drain, *pos, *n; |
| 2251 | 2251 | ||
| 2252 | BUILD_BUG_ON(sizeof(struct list_head) > sizeof(struct bpf_list_head)); | 2252 | BUILD_BUG_ON(sizeof(struct list_head) > sizeof(struct bpf_list_head)); |
| 2253 | BUILD_BUG_ON(__alignof__(struct list_head) > __alignof__(struct bpf_list_head)); | 2253 | BUILD_BUG_ON(__alignof__(struct list_head) > __alignof__(struct bpf_list_head)); |
| 2254 | INIT_LIST_HEAD(&drain); | ||
| 2254 | 2255 | ||
| 2255 | /* Do the actual list draining outside the lock to not hold the lock for | 2256 | /* Do the actual list draining outside the lock to not hold the lock for |
| 2256 | * too long, and also prevent deadlocks if tracing programs end up | 2257 | * too long, and also prevent deadlocks if tracing programs end up |
| @@ -2261,20 +2262,30 @@ void bpf_list_head_free(const struct btf_field *field, void *list_head, · 4 unmodified lines | |||
| 2261 | __bpf_spin_lock_irqsave(spin_lock); | 2262 | __bpf_spin_lock_irqsave(spin_lock); |
| 2262 | if (!head->next || list_empty(head)) | 2263 | if (!head->next || list_empty(head)) |
| 2263 | goto unlock; | 2264 | goto unlock; |
| 2264 | head = head->next; | 2265 | list_for_each_safe(pos, n, head) { |
| 2266 | struct bpf_list_node_kern *node; | ||
| 2267 | |||
| 2268 | node = container_of(pos, struct bpf_list_node_kern, list_head); | ||
| 2269 | WRITE_ONCE(node->owner, BPF_PTR_POISON); | ||
| 2270 | list_move_tail(pos, &drain); | ||
| 2271 | } | ||
| 2265 | unlock: | 2272 | unlock: |
| 2266 | INIT_LIST_HEAD(orig_head); | 2273 | INIT_LIST_HEAD(head); |
| 2267 | __bpf_spin_unlock_irqrestore(spin_lock); | 2274 | __bpf_spin_unlock_irqrestore(spin_lock); |
| 2268 | 2275 | ||
| 2269 | while (head != orig_head) { | 2276 | while (!list_empty(&drain)) { |
| 2270 | void *obj = head; | 2277 | struct bpf_list_node_kern *node; |
| 2271 | 2278 | ||
| 2272 | obj -= field->graph_root.node_offset; | 2279 | pos = drain.next; |
| 2273 | head = head->next; | 2280 | node = container_of(pos, struct bpf_list_node_kern, list_head); |
| 2281 | list_del_init(pos); | ||
| 2282 | /* Ensure __bpf_list_add() sees the node as unlinked. */ | ||
| 2283 | smp_store_release(&node->owner, NULL); | ||
| 2274 | /* The contained type can also have resources, including a | 2284 | /* The contained type can also have resources, including a |
| 2275 | * bpf_list_head which needs to be freed. | 2285 | * bpf_list_head which needs to be freed. |
| 2276 | */ | 2286 | */ |
| 2277 | __bpf_obj_drop_impl(obj, field->graph_root.value_rec, false); | 2287 | __bpf_obj_drop_impl((char *)pos - field->graph_root.node_offset, |
| 2288 | field->graph_root.value_rec, false); | ||
| 2278 | } | 2289 | } |
| 2279 | } | 2290 | } |
| 2280 | 2291 | ||
| @@ -2295,6 +2306,7 @@ void bpf_rb_root_free(const struct btf_field *field, void *rb_root, · 14 unmodified lines | |||
| 2295 | struct bpf_spin_lock *spin_lock) | 2306 | struct bpf_spin_lock *spin_lock) |
| 2296 | { | 2307 | { |
| 2297 | struct rb_root_cached orig_root, *root = rb_root; | 2308 | struct rb_root_cached orig_root, *root = rb_root; |
| 2309 | struct bpf_rb_node_kern *node; | ||
| 2298 | struct rb_node *pos, *n; | 2310 | struct rb_node *pos, *n; |
| 2299 | void *obj; | 2311 | void *obj; |
| 2300 | 2312 | ||
| @@ -2303,14 +2315,20 @@ void bpf_rb_root_free(const struct btf_field *field, void *rb_root, · 2 unmodified lines | |||
| 2303 | 2315 | ||
| 2304 | __bpf_spin_lock_irqsave(spin_lock); | 2316 | __bpf_spin_lock_irqsave(spin_lock); |
| 2305 | orig_root = *root; | 2317 | orig_root = *root; |
| 2318 | bpf_rbtree_postorder_for_each_entry_safe(pos, n, &orig_root.rb_root) { | ||
| 2319 | node = rb_entry(pos, struct bpf_rb_node_kern, rb_node); | ||
| 2320 | WRITE_ONCE(node->owner, BPF_PTR_POISON); | ||
| 2321 | } | ||
| 2306 | *root = RB_ROOT_CACHED; | 2322 | *root = RB_ROOT_CACHED; |
| 2307 | __bpf_spin_unlock_irqrestore(spin_lock); | 2323 | __bpf_spin_unlock_irqrestore(spin_lock); |
| 2308 | 2324 | ||
| 2309 | bpf_rbtree_postorder_for_each_entry_safe(pos, n, &orig_root.rb_root) { | 2325 | bpf_rbtree_postorder_for_each_entry_safe(pos, n, &orig_root.rb_root) { |
| 2310 | obj = pos; | 2326 | obj = pos; |
| 2311 | obj -= field->graph_root.node_offset; | 2327 | obj -= field->graph_root.node_offset; |
| 2312 | 2328 | node = rb_entry(pos, struct bpf_rb_node_kern, rb_node); | |
| 2313 | 2329 | RB_CLEAR_NODE(pos); | |
| 2330 | /* Ensure __bpf_rbtree_add() sees the node as unlinked. */ | ||
| 2331 | smp_store_release(&node->owner, NULL); | ||
| 2314 | __bpf_obj_drop_impl(obj, field->graph_root.value_rec, false); | 2332 | __bpf_obj_drop_impl(obj, field->graph_root.value_rec, false); |
| 2315 | } | 2333 | } |
| 2316 | } | 2334 | } |
| @@ -2467,9 +2485,11 @@ __bpf_kfunc void *bpf_refcount_acquire_impl(void *p__refcounted_kptr, void *meta · 150 unmodified lines | |||
| 2467 | 2485 | ||
| 2468 | static int __bpf_list_add(struct bpf_list_node_kern *node, | 2486 | static int __bpf_list_add(struct bpf_list_node_kern *node, |
| 2469 | struct bpf_list_head *head, | 2487 | struct bpf_list_head *head, |
| 2470 | bool tail, struct btf_record *rec, u64 off) | 2488 | struct list_head **prev_ptr, |
| 2489 | struct btf_record *rec, u64 off) | ||
| 2471 | { | 2490 | { |
| 2472 | struct list_head *n = &node->list_head, *h = (void *)head; | 2491 | struct list_head *n = &node->list_head, *h = (void *)head; |
| 2492 | struct list_head *prev; | ||
| 2473 | 2493 | ||
| 2474 | /* If list_head was 0-initialized by map, bpf_obj_init_field wasn't | 2494 | /* If list_head was 0-initialized by map, bpf_obj_init_field wasn't |
| 2475 | * called on its fields, so init here | 2495 | * called on its fields, so init here |
| @@ -2477,19 +2497,31 @@ static int __bpf_list_add(struct bpf_list_node_kern *node, · 1 unmodified line | |||
| 2477 | if (unlikely(!h->next)) | 2497 | if (unlikely(!h->next)) |
| 2478 | INIT_LIST_HEAD(h); | 2498 | INIT_LIST_HEAD(h); |
| 2479 | 2499 | ||
| 2500 | prev = *prev_ptr; | ||
| 2501 | |||
| 2502 | /* When prev is not the list head, it must be a node in this list. */ | ||
| 2503 | if (prev != h) { | ||
| 2504 | struct bpf_list_node_kern *prev_kn = | ||
| 2505 | container_of(prev, struct bpf_list_node_kern, list_head); | ||
| 2506 | |||
| 2507 | if (unlikely(READ_ONCE(prev_kn->owner) != head)) | ||
| 2508 | goto fail; | ||
| 2509 | } | ||
| 2510 | |||
| 2480 | /* node->owner != NULL implies !list_empty(n), no need to separately | 2511 | /* node->owner != NULL implies !list_empty(n), no need to separately |
| 2481 | * check the latter | 2512 | * check the latter |
| 2482 | */ | 2513 | */ |
| 2483 | if (cmpxchg(&node->owner, NULL, BPF_PTR_POISON)) { | 2514 | if (cmpxchg(&node->owner, NULL, BPF_PTR_POISON)) |
| 2484 | /* Only called from BPF prog, no need to migrate_disable */ | 2515 | goto fail; |
| 2485 | __bpf_obj_drop_impl((void *)n - off, rec, false); | ||
| 2486 | return -EINVAL; | ||
| 2487 | } | ||
| 2488 | 2516 | ||
| 2489 | tail ? list_add_tail(n, h) : list_add(n, h); | 2517 | list_add(n, prev); |
| 2490 | WRITE_ONCE(node->owner, head); | 2518 | WRITE_ONCE(node->owner, head); |
| 2491 | |||
| 2492 | return 0; | 2519 | return 0; |
| 2520 | |||
| 2521 | fail: | ||
| 2522 | /* Only called from BPF prog, no need to migrate_disable */ | ||
| 2523 | __bpf_obj_drop_impl((void *)n - off, rec, false); | ||
| 2524 | return -EINVAL; | ||
| 2493 | } | 2525 | } |
| 2494 | 2526 | ||
| 2495 | /** | 2527 | /** |
| @@ -2510,8 +2542,9 @@ __bpf_kfunc int bpf_list_push_front(struct bpf_list_head *head, · 14 unmodified lines | |||
| 2510 | u64 off) | 2542 | u64 off) |
| 2511 | { | 2543 | { |
| 2512 | struct bpf_list_node_kern *n = (void *)node; | 2544 | struct bpf_list_node_kern *n = (void *)node; |
| 2545 | struct list_head *h = (void *)head; | ||
| 2513 | 2546 | ||
| 2514 | return __bpf_list_add(n, head, false, meta ? meta->record : NULL, off); | 2547 | return __bpf_list_add(n, head, &h, meta ? meta->record : NULL, off); |
| 2515 | } | 2548 | } |
| 2516 | 2549 | ||
| 2517 | __bpf_kfunc int bpf_list_push_front_impl(struct bpf_list_head *head, | 2550 | __bpf_kfunc int bpf_list_push_front_impl(struct bpf_list_head *head, |
| @@ -2539,8 +2572,9 @@ __bpf_kfunc int bpf_list_push_back(struct bpf_list_head *head, · 21 unmodified lines | |||
| 2539 | u64 off) | 2572 | u64 off) |
| 2540 | { | 2573 | { |
| 2541 | struct bpf_list_node_kern *n = (void *)node; | 2574 | struct bpf_list_node_kern *n = (void *)node; |
| 2575 | struct list_head *h = (void *)head; | ||
| 2542 | 2576 | ||
| 2543 | return __bpf_list_add(n, head, true, meta ? meta->record : NULL, off); | 2577 | return __bpf_list_add(n, head, &h->prev, meta ? meta->record : NULL, off); |
| 2544 | } | 2578 | } |
| 2545 | 2579 | ||
| 2546 | __bpf_kfunc int bpf_list_push_back_impl(struct bpf_list_head *head, | 2580 | __bpf_kfunc int bpf_list_push_back_impl(struct bpf_list_head *head, |
| @@ -2550,37 +2584,63 @@ __bpf_kfunc int bpf_list_push_back_impl(struct bpf_list_head *head, · 3 unmodified lines | |||
| 2550 | return bpf_list_push_back(head, node, meta__ign, off); | 2584 | return bpf_list_push_back(head, node, meta__ign, off); |
| 2551 | } | 2585 | } |
| 2552 | 2586 | ||
| 2553 | static struct bpf_list_node *__bpf_list_del(struct bpf_list_head *head, bool tail) | 2587 | __bpf_kfunc int bpf_list_add(struct bpf_list_head *head, struct bpf_list_node *new, |
| 2588 | struct bpf_list_node *prev__nonown_allowed, | ||
| 2589 | struct btf_struct_meta *meta, u64 off) | ||
| 2590 | { | ||
| 2591 | struct bpf_list_node_kern *n = (void *)new, *p = (void *)prev__nonown_allowed; | ||
| 2592 | struct list_head *prev_ptr = &p->list_head; | ||
| 2593 | |||
| 2594 | return __bpf_list_add(n, head, &prev_ptr, meta ? meta->record : NULL, off); | ||
| 2595 | } | ||
| 2596 | |||
| 2597 | static struct bpf_list_node *__bpf_list_del(struct bpf_list_head *head, | ||
| 2598 | struct list_head *n) | ||
| 2554 | { | 2599 | { |
| 2555 | struct list_head *n, *h = (void *)head; | 2600 | struct list_head *h = (void *)head; |
| 2556 | struct bpf_list_node_kern *node; | 2601 | struct bpf_list_node_kern *node; |
| 2557 | 2602 | ||
| 2558 | /* If list_head was 0-initialized by map, bpf_obj_init_field wasn't | 2603 | /* If list_head was 0-initialized by map, bpf_obj_init_field wasn't |
| 2559 | * called on its fields, so init here | 2604 | * called on its fields, so init here |
| 2560 | */ | 2605 | */ |
| 2561 | if (unlikely(!h->next)) | 2606 | if (unlikely(!h->next)) { |
| 2562 | INIT_LIST_HEAD(h); | 2607 | INIT_LIST_HEAD(h); |
| 2608 | return NULL; | ||
| 2609 | } | ||
| 2563 | if (list_empty(h)) | 2610 | if (list_empty(h)) |
| 2564 | return NULL; | 2611 | return NULL; |
| 2565 | 2612 | ||
| 2566 | n = tail ? h->prev : h->next; | ||
| 2567 | node = container_of(n, struct bpf_list_node_kern, list_head); | 2613 | node = container_of(n, struct bpf_list_node_kern, list_head); |
| 2568 | if (WARN_ON_ONCE(READ_ONCE(node->owner) != head)) | 2614 | if (unlikely(READ_ONCE(node->owner) != head)) |
| 2569 | return NULL; | 2615 | return NULL; |
| 2570 | 2616 | ||
| 2571 | list_del_init(n); | 2617 | list_del_init(n); |
| 2572 | WRITE_ONCE(node->owner, NULL); | 2618 | /* Ensure __bpf_list_add() sees the node as unlinked. */ |
| 2619 | smp_store_release(&node->owner, NULL); | ||
| 2573 | return (struct bpf_list_node *)n; | 2620 | return (struct bpf_list_node *)n; |
| 2574 | } | 2621 | } |
| 2575 | 2622 | ||
| 2576 | __bpf_kfunc struct bpf_list_node *bpf_list_pop_front(struct bpf_list_head *head) | 2623 | __bpf_kfunc struct bpf_list_node *bpf_list_pop_front(struct bpf_list_head *head) |
| 2577 | { | 2624 | { |
| 2578 | return __bpf_list_del(head, false); | 2625 | struct list_head *h = (void *)head; |
| 2626 | |||
| 2627 | return __bpf_list_del(head, h->next); | ||
| 2579 | } | 2628 | } |
| 2580 | 2629 | ||
| 2581 | __bpf_kfunc struct bpf_list_node *bpf_list_pop_back(struct bpf_list_head *head) | 2630 | __bpf_kfunc struct bpf_list_node *bpf_list_pop_back(struct bpf_list_head *head) |
| 2582 | { | 2631 | { |
| 2583 | return __bpf_list_del(head, true); | 2632 | struct list_head *h = (void *)head; |
| 2633 | |||
| 2634 | return __bpf_list_del(head, h->prev); | ||
| 2635 | } | ||
| 2636 | |||
| 2637 | __bpf_kfunc struct bpf_list_node *bpf_list_del(struct bpf_list_head *head, | ||
| 2638 | struct bpf_list_node *node__nonown_allowed) | ||
| 2639 | { | ||
| 2640 | struct bpf_list_node_kern *kn = (void *)node__nonown_allowed; | ||
| 2641 | |||
| 2642 | /* verifier guarantees node is a list node rather than list head */ | ||
| 2643 | return __bpf_list_del(head, &kn->list_head); | ||
| 2584 | } | 2644 | } |
| 2585 | 2645 | ||
| 2586 | __bpf_kfunc struct bpf_list_node *bpf_list_front(struct bpf_list_head *head) | 2646 | __bpf_kfunc struct bpf_list_node *bpf_list_front(struct bpf_list_head *head) |
| @@ -2603,6 +2663,43 @@ __bpf_kfunc struct bpf_list_node *bpf_list_back(struct bpf_list_head *head) · 16 unmodified lines | |||
| 2603 | return (struct bpf_list_node *)h->prev; | 2663 | return (struct bpf_list_node *)h->prev; |
| 2604 | } | 2664 | } |
| 2605 | 2665 | ||
| 2666 | __bpf_kfunc bool bpf_list_is_first(struct bpf_list_head *head, | ||
| 2667 | struct bpf_list_node *node__nonown_allowed) | ||
| 2668 | { | ||
| 2669 | struct list_head *h = (struct list_head *)head; | ||
| 2670 | struct bpf_list_node_kern *kn = (struct bpf_list_node_kern *)node__nonown_allowed; | ||
| 2671 | |||
| 2672 | if (READ_ONCE(kn->owner) != head) | ||
| 2673 | return false; | ||
| 2674 | |||
| 2675 | return list_is_first(&kn->list_head, h); | ||
| 2676 | } | ||
| 2677 | |||
| 2678 | __bpf_kfunc bool bpf_list_is_last(struct bpf_list_head *head, | ||
| 2679 | struct bpf_list_node *node__nonown_allowed) | ||
| 2680 | { | ||
| 2681 | struct list_head *h = (struct list_head *)head; | ||
| 2682 | struct bpf_list_node_kern *kn = (struct bpf_list_node_kern *)node__nonown_allowed; | ||
| 2683 | |||
| 2684 | if (READ_ONCE(kn->owner) != head) | ||
| 2685 | return false; | ||
| 2686 | |||
| 2687 | return list_is_last(&kn->list_head, h); | ||
| 2688 | } | ||
| 2689 | |||
| 2690 | __bpf_kfunc bool bpf_list_empty(struct bpf_list_head *head) | ||
| 2691 | { | ||
| 2692 | struct list_head *h = (struct list_head *)head; | ||
| 2693 | |||
| 2694 | /* If list_head was 0-initialized by map, bpf_obj_init_field wasn't | ||
| 2695 | * called on its fields, so init here | ||
| 2696 | */ | ||
| 2697 | if (unlikely(!h->next)) | ||
| 2698 | INIT_LIST_HEAD(h); | ||
| 2699 | |||
| 2700 | return list_empty(h); | ||
| 2701 | } | ||
| 2702 | |||
| 2606 | __bpf_kfunc struct bpf_rb_node *bpf_rbtree_remove(struct bpf_rb_root *root, | 2703 | __bpf_kfunc struct bpf_rb_node *bpf_rbtree_remove(struct bpf_rb_root *root, |
| 2607 | struct bpf_rb_node *node) | 2704 | struct bpf_rb_node *node) |
| 2608 | { | 2705 | { |
| @@ -2912,11 +3009,13 @@ __bpf_kfunc struct task_struct *bpf_task_from_vpid(s32 vpid) · 303 unmodified lines | |||
| 2912 | { | 3009 | { |
| 2913 | struct task_struct *p; | 3010 | struct task_struct *p; |
| 2914 | 3011 | ||
| 2915 | rcu_read_lock(); | 3012 | guard(rcu)(); |
| 3013 | if (!task_active_pid_ns(current)) | ||
| 3014 | return NULL; | ||
| 3015 | |||
| 2916 | p = find_task_by_vpid(vpid); | 3016 | p = find_task_by_vpid(vpid); |
| 2917 | if (p) | 3017 | if (p) |
| 2918 | p = bpf_task_acquire(p); | 3018 | p = bpf_task_acquire(p); |
| 2919 | rcu_read_unlock(); | ||
| 2920 | 3019 | ||
| 2921 | return p; | 3020 | return p; |
| 2922 | } | 3021 | } |
| @@ -3072,7 +3171,7 @@ __bpf_kfunc void *bpf_dynptr_slice_rdwr(const struct bpf_dynptr *p, u64 offset, · 149 unmodified lines | |||
| 3072 | return bpf_dynptr_slice(p, offset, buffer__nullable, buffer__szk); | 3171 | return bpf_dynptr_slice(p, offset, buffer__nullable, buffer__szk); |
| 3073 | } | 3172 | } |
| 3074 | 3173 | ||
| 3075 | __bpf_kfunc int bpf_dynptr_adjust(const struct bpf_dynptr *p, u64 start, u64 end) | 3174 | __bpf_kfunc int bpf_dynptr_adjust(struct bpf_dynptr *p, u64 start, u64 end) |
| 3076 | { | 3175 | { |
| 3077 | struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p; | 3176 | struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p; |
| 3078 | u64 size; | 3177 | u64 size; |
| @@ -3093,14 +3192,14 @@ __bpf_kfunc int bpf_dynptr_adjust(const struct bpf_dynptr *p, u64 start, u64 end · 14 unmodified lines | |||
| 3093 | 3192 | ||
| 3094 | __bpf_kfunc bool bpf_dynptr_is_null(const struct bpf_dynptr *p) | 3193 | __bpf_kfunc bool bpf_dynptr_is_null(const struct bpf_dynptr *p) |
| 3095 | { | 3194 | { |
| 3096 | struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p; | 3195 | const struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p; |
| 3097 | 3196 | ||
| 3098 | return !ptr->data; | 3197 | return !ptr->data; |
| 3099 | } | 3198 | } |
| 3100 | 3199 | ||
| 3101 | __bpf_kfunc bool bpf_dynptr_is_rdonly(const struct bpf_dynptr *p) | 3200 | __bpf_kfunc bool bpf_dynptr_is_rdonly(const struct bpf_dynptr *p) |
| 3102 | { | 3201 | { |
| 3103 | struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p; | 3202 | const struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p; |
| 3104 | 3203 | ||
| 3105 | if (!ptr->data) | 3204 | if (!ptr->data) |
| 3106 | return false; | 3205 | return false; |
| @@ -3110,7 +3209,7 @@ __bpf_kfunc bool bpf_dynptr_is_rdonly(const struct bpf_dynptr *p) · 3 unmodified lines | |||
| 3110 | 3209 | ||
| 3111 | __bpf_kfunc u64 bpf_dynptr_size(const struct bpf_dynptr *p) | 3210 | __bpf_kfunc u64 bpf_dynptr_size(const struct bpf_dynptr *p) |
| 3112 | { | 3211 | { |
| 3113 | struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p; | 3212 | const struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p; |
| 3114 | 3213 | ||
| 3115 | if (!ptr->data) | 3214 | if (!ptr->data) |
| 3116 | return -EINVAL; | 3215 | return -EINVAL; |
| @@ -3122,7 +3221,7 @@ __bpf_kfunc int bpf_dynptr_clone(const struct bpf_dynptr *p, · 5 unmodified lines | |||
| 3122 | struct bpf_dynptr *clone__uninit) | 3221 | struct bpf_dynptr *clone__uninit) |
| 3123 | { | 3222 | { |
| 3124 | struct bpf_dynptr_kern *clone = (struct bpf_dynptr_kern *)clone__uninit; | 3223 | struct bpf_dynptr_kern *clone = (struct bpf_dynptr_kern *)clone__uninit; |
| 3125 | struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p; | 3224 | const struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p; |
| 3126 | 3225 | ||
| 3127 | if (!ptr->data) { | 3226 | if (!ptr->data) { |
| 3128 | bpf_dynptr_set_null(clone); | 3227 | bpf_dynptr_set_null(clone); |
| @@ -3145,11 +3244,11 @@ __bpf_kfunc int bpf_dynptr_clone(const struct bpf_dynptr *p, · 16 unmodified lines | |||
| 3145 | * Copies data from source dynptr to destination dynptr. | 3244 | * Copies data from source dynptr to destination dynptr. |
| 3146 | * Returns 0 on success; negative error, otherwise. | 3245 | * Returns 0 on success; negative error, otherwise. |
| 3147 | */ | 3246 | */ |
| 3148 | __bpf_kfunc int bpf_dynptr_copy(struct bpf_dynptr *dst_ptr, u64 dst_off, | 3247 | __bpf_kfunc int bpf_dynptr_copy(const struct bpf_dynptr *dst_ptr, u64 dst_off, |
| 3149 | struct bpf_dynptr *src_ptr, u64 src_off, u64 size) | 3248 | const struct bpf_dynptr *src_ptr, u64 src_off, u64 size) |
| 3150 | { | 3249 | { |
| 3151 | struct bpf_dynptr_kern *dst = (struct bpf_dynptr_kern *)dst_ptr; | 3250 | const struct bpf_dynptr_kern *dst = (struct bpf_dynptr_kern *)dst_ptr; |
| 3152 | struct bpf_dynptr_kern *src = (struct bpf_dynptr_kern *)src_ptr; | 3251 | const struct bpf_dynptr_kern *src = (struct bpf_dynptr_kern *)src_ptr; |
| 3153 | void *src_slice, *dst_slice; | 3252 | void *src_slice, *dst_slice; |
| 3154 | char buf[256]; | 3253 | char buf[256]; |
| 3155 | u64 off; | 3254 | u64 off; |
| @@ -3200,9 +3299,9 @@ __bpf_kfunc int bpf_dynptr_copy(struct bpf_dynptr *dst_ptr, u64 dst_off, · 44 unmodified lines | |||
| 3200 | * at @offset with the constant byte @val. | 3299 | * at @offset with the constant byte @val. |
| 3201 | * Returns 0 on success; negative error, otherwise. | 3300 | * Returns 0 on success; negative error, otherwise. |
| 3202 | */ | 3301 | */ |
| 3203 | __bpf_kfunc int bpf_dynptr_memset(struct bpf_dynptr *p, u64 offset, u64 size, u8 val) | 3302 | __bpf_kfunc int bpf_dynptr_memset(const struct bpf_dynptr *p, u64 offset, u64 size, u8 val) |
| 3204 | { | 3303 | { |
| 3205 | struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p; | 3304 | const struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p; |
| 3206 | u64 chunk_sz, write_off; | 3305 | u64 chunk_sz, write_off; |
| 3207 | char buf[256]; | 3306 | char buf[256]; |
| 3208 | void* slice; | 3307 | void* slice; |
| @@ -3301,7 +3400,7 @@ __bpf_kfunc void bpf_throw(u64 cookie) · 92 unmodified lines | |||
| 3301 | * which skips compiler generated instrumentation to do the same. | 3400 | * which skips compiler generated instrumentation to do the same. |
| 3302 | */ | 3401 | */ |
| 3303 | kasan_unpoison_task_stack_below((void *)(long)ctx.sp); | 3402 | kasan_unpoison_task_stack_below((void *)(long)ctx.sp); |
| 3304 | ctx.aux->bpf_exception_cb(cookie, ctx.sp, ctx.bp, 0, 0); | 3403 | ctx.aux->bpf_exception_cb(cookie, ctx.sp + ctx.aux->stack_arg_sp_adjust, ctx.bp, 0, 0); |
| 3305 | WARN(1, "A call to BPF exception callback should never return\n"); | 3404 | WARN(1, "A call to BPF exception callback should never return\n"); |
| 3306 | } | 3405 | } |
| 3307 | 3406 | ||
| @@ -4214,13 +4313,13 @@ __bpf_kfunc void bpf_key_put(struct bpf_key *bkey) · 906 unmodified lines | |||
| 4214 | * | 4313 | * |
| 4215 | * Return: 0 on success, a negative value on error. | 4314 | * Return: 0 on success, a negative value on error. |
| 4216 | */ | 4315 | */ |
| 4217 | __bpf_kfunc int bpf_verify_pkcs7_signature(struct bpf_dynptr *data_p, | 4316 | __bpf_kfunc int bpf_verify_pkcs7_signature(const struct bpf_dynptr *data_p, |
| 4218 | struct bpf_dynptr *sig_p, | 4317 | const struct bpf_dynptr *sig_p, |
| 4219 | struct bpf_key *trusted_keyring) | 4318 | struct bpf_key *trusted_keyring) |
| 4220 | { | 4319 | { |
| 4221 | #ifdef CONFIG_SYSTEM_DATA_VERIFICATION | 4320 | #ifdef CONFIG_SYSTEM_DATA_VERIFICATION |
| 4222 | struct bpf_dynptr_kern *data_ptr = (struct bpf_dynptr_kern *)data_p; | 4321 | const struct bpf_dynptr_kern *data_ptr = (struct bpf_dynptr_kern *)data_p; |
| 4223 | struct bpf_dynptr_kern *sig_ptr = (struct bpf_dynptr_kern *)sig_p; | 4322 | const struct bpf_dynptr_kern *sig_ptr = (struct bpf_dynptr_kern *)sig_p; |
| 4224 | const void *data, *sig; | 4323 | const void *data, *sig; |
| 4225 | u32 data_len, sig_len; | 4324 | u32 data_len, sig_len; |
| 4226 | int ret; | 4325 | int ret; |
| @@ -4718,10 +4817,15 @@ BTF_ID_FLAGS(func, bpf_list_push_front, KF_IMPLICIT_ARGS) · 491 unmodified lines | |||
| 4718 | BTF_ID_FLAGS(func, bpf_list_push_front_impl) | 4817 | BTF_ID_FLAGS(func, bpf_list_push_front_impl) |
| 4719 | BTF_ID_FLAGS(func, bpf_list_push_back, KF_IMPLICIT_ARGS) | 4818 | BTF_ID_FLAGS(func, bpf_list_push_back, KF_IMPLICIT_ARGS) |
| 4720 | BTF_ID_FLAGS(func, bpf_list_push_back_impl) | 4819 | BTF_ID_FLAGS(func, bpf_list_push_back_impl) |
| 4820 | BTF_ID_FLAGS(func, bpf_list_add, KF_IMPLICIT_ARGS) | ||
| 4721 | BTF_ID_FLAGS(func, bpf_list_pop_front, KF_ACQUIRE | KF_RET_NULL) | 4821 | BTF_ID_FLAGS(func, bpf_list_pop_front, KF_ACQUIRE | KF_RET_NULL) |
| 4722 | BTF_ID_FLAGS(func, bpf_list_pop_back, KF_ACQUIRE | KF_RET_NULL) | 4822 | BTF_ID_FLAGS(func, bpf_list_pop_back, KF_ACQUIRE | KF_RET_NULL) |
| 4823 | BTF_ID_FLAGS(func, bpf_list_del, KF_ACQUIRE | KF_RET_NULL) | ||
| 4723 | BTF_ID_FLAGS(func, bpf_list_front, KF_RET_NULL) | 4824 | BTF_ID_FLAGS(func, bpf_list_front, KF_RET_NULL) |
| 4724 | BTF_ID_FLAGS(func, bpf_list_back, KF_RET_NULL) | 4825 | BTF_ID_FLAGS(func, bpf_list_back, KF_RET_NULL) |
| 4826 | BTF_ID_FLAGS(func, bpf_list_is_first) | ||
| 4827 | BTF_ID_FLAGS(func, bpf_list_is_last) | ||
| 4828 | BTF_ID_FLAGS(func, bpf_list_empty) | ||
| 4725 | BTF_ID_FLAGS(func, bpf_task_acquire, KF_ACQUIRE | KF_RCU | KF_RET_NULL) | 4829 | BTF_ID_FLAGS(func, bpf_task_acquire, KF_ACQUIRE | KF_RCU | KF_RET_NULL) |
| 4726 | BTF_ID_FLAGS(func, bpf_task_release, KF_RELEASE) | 4830 | BTF_ID_FLAGS(func, bpf_task_release, KF_RELEASE) |
| 4727 | BTF_ID_FLAGS(func, bpf_rbtree_remove, KF_ACQUIRE | KF_RET_NULL) | 4831 | BTF_ID_FLAGS(func, bpf_rbtree_remove, KF_ACQUIRE | KF_RET_NULL) |
| @@ -4862,7 +4966,7 @@ BTF_ID_FLAGS(func, bpf_stream_print_stack, KF_IMPLICIT_ARGS) · 134 unmodified lines | |||
| 4862 | BTF_ID_FLAGS(func, bpf_task_work_schedule_signal, KF_IMPLICIT_ARGS) | 4966 | BTF_ID_FLAGS(func, bpf_task_work_schedule_signal, KF_IMPLICIT_ARGS) |
| 4863 | BTF_ID_FLAGS(func, bpf_task_work_schedule_resume, KF_IMPLICIT_ARGS) | 4967 | BTF_ID_FLAGS(func, bpf_task_work_schedule_resume, KF_IMPLICIT_ARGS) |
| 4864 | BTF_ID_FLAGS(func, bpf_dynptr_from_file) | 4968 | BTF_ID_FLAGS(func, bpf_dynptr_from_file) |
| 4865 | BTF_ID_FLAGS(func, bpf_dynptr_file_discard) | 4969 | BTF_ID_FLAGS(func, bpf_dynptr_file_discard, KF_RELEASE) |
| 4866 | BTF_ID_FLAGS(func, bpf_timer_cancel_async) | 4970 | BTF_ID_FLAGS(func, bpf_timer_cancel_async) |
| 4867 | BTF_KFUNCS_END(common_btf_ids) | 4971 | BTF_KFUNCS_END(common_btf_ids) |
| 4868 | 4972 | ||
| @@ -845,6 +845,10 @@ static void bpf_destroy_inode(struct inode *inode) · 844 unmodified lines | |||
| 845 | simple_xattrs_free(&opts->xa_cache, &bi->xattrs, NULL); | 845 | simple_xattrs_free(&opts->xa_cache, &bi->xattrs, NULL); |
| 846 | } | 846 | } |
| 847 | 847 | ||
| 848 | /* | ||
| 849 | * Called after RCU grace period - safe to free inode and anything | ||
| 850 | * that might be accessed by RCU pathwalk (inode fields, i_link). | ||
| 851 | */ | ||
| 848 | static void bpf_free_inode(struct inode *inode) | 852 | static void bpf_free_inode(struct inode *inode) |
| 849 | { | 853 | { |
| 850 | if (S_ISLNK(inode->i_mode)) | 854 | if (S_ISLNK(inode->i_mode)) |
| @@ -610,6 +610,21 @@ enum arg_track_state { · 609 unmodified lines | |||
| 610 | /* Track callee stack slots fp-8 through fp-512 (64 slots of 8 bytes each) */ | 610 | /* Track callee stack slots fp-8 through fp-512 (64 slots of 8 bytes each) */ |
| 611 | #define MAX_ARG_SPILL_SLOTS 64 | 611 | #define MAX_ARG_SPILL_SLOTS 64 |
| 612 | 612 | ||
| 613 | /* | ||
| 614 | * Combined register + stack arg tracking: R0-R10 at indices 0-10, | ||
| 615 | * outgoing stack arg slots at indices MAX_BPF_REG..MAX_BPF_REG+6. | ||
| 616 | */ | ||
| 617 | #define MAX_AT_TRACK_REGS (MAX_BPF_REG + MAX_STACK_ARG_SLOTS) | ||
| 618 | |||
| 619 | static int stack_arg_off_to_slot(s16 off) | ||
| 620 | { | ||
| 621 | int aoff = off < 0 ? -off : off; | ||
| 622 | |||
| 623 | if (aoff / 8 > MAX_STACK_ARG_SLOTS) | ||
| 624 | return -1; | ||
| 625 | return aoff / 8 - 1; | ||
| 626 | } | ||
| 627 | |||
| 613 | static bool arg_is_visited(const struct arg_track *at) | 628 | static bool arg_is_visited(const struct arg_track *at) |
| 614 | { | 629 | { |
| 615 | return at->frame != ARG_UNVISITED; | 630 | return at->frame != ARG_UNVISITED; |
| @@ -791,7 +806,9 @@ static bool arg_track_join(struct bpf_verifier_env *env, int idx, int target, in · 175 unmodified lines | |||
| 791 | return true; | 806 | return true; |
| 792 | 807 | ||
| 793 | verbose(env, "arg JOIN insn %d -> %d ", idx, target); | 808 | verbose(env, "arg JOIN insn %d -> %d ", idx, target); |
| 794 | if (r >= 0) | 809 | if (r >= MAX_BPF_REG) |
| 810 | verbose(env, "sa%d: ", r - MAX_BPF_REG); | ||
| 811 | else if (r >= 0) | ||
| 795 | verbose(env, "r%d: ", r); | 812 | verbose(env, "r%d: ", r); |
| 796 | else | 813 | else |
| 797 | verbose(env, "fp%+d: ", r * 8); | 814 | verbose(env, "fp%+d: ", r * 8); |
| @@ -1032,6 +1049,21 @@ static void arg_track_log(struct bpf_verifier_env *env, struct bpf_insn *insn, i · 234 unmodified lines | |||
| 1032 | verbose(env, "\tr%d: ", i); verbose_arg_track(env, &at_in[i]); | 1049 | verbose(env, "\tr%d: ", i); verbose_arg_track(env, &at_in[i]); |
| 1033 | verbose(env, " -> "); verbose_arg_track(env, &at_out[i]); | 1050 | verbose(env, " -> "); verbose_arg_track(env, &at_out[i]); |
| 1034 | } | 1051 | } |
| 1052 | /* Log outgoing stack arg slot transitions at indices MAX_BPF_REG..MAX_AT_TRACK_REGS-1 */ | ||
| 1053 | for (i = 0; i < MAX_STACK_ARG_SLOTS; i++) { | ||
| 1054 | int ai = MAX_BPF_REG + i; | ||
| 1055 | |||
| 1056 | if (arg_track_eq(&at_out[ai], &at_in[ai])) | ||
| 1057 | continue; | ||
| 1058 | if (!printed) { | ||
| 1059 | verbose(env, "%3d: ", idx); | ||
| 1060 | bpf_verbose_insn(env, insn); | ||
| 1061 | bpf_vlog_reset(&env->log, env->log.end_pos - 1); | ||
| 1062 | printed = true; | ||
| 1063 | } | ||
| 1064 | verbose(env, "\tsa%d: ", i); verbose_arg_track(env, &at_in[ai]); | ||
| 1065 | verbose(env, " -> "); verbose_arg_track(env, &at_out[ai]); | ||
| 1066 | } | ||
| 1035 | for (i = 0; i < MAX_ARG_SPILL_SLOTS; i++) { | 1067 | for (i = 0; i < MAX_ARG_SPILL_SLOTS; i++) { |
| 1036 | if (arg_track_eq(&at_stack_out[i], &at_stack_in[i])) | 1068 | if (arg_track_eq(&at_stack_out[i], &at_stack_in[i])) |
| 1037 | continue; | 1069 | continue; |
| @@ -1062,6 +1094,7 @@ static bool can_be_local_fp(int depth, int regno, struct arg_track *at) · 24 unmodified lines | |||
| 1062 | static void arg_track_xfer(struct bpf_verifier_env *env, struct bpf_insn *insn, | 1094 | static void arg_track_xfer(struct bpf_verifier_env *env, struct bpf_insn *insn, |
| 1063 | int insn_idx, | 1095 | int insn_idx, |
| 1064 | struct arg_track *at_out, struct arg_track *at_stack_out, | 1096 | struct arg_track *at_out, struct arg_track *at_stack_out, |
| 1097 | const struct arg_track *at_stack_arg_entry, | ||
| 1065 | struct func_instance *instance, | 1098 | struct func_instance *instance, |
| 1066 | u32 *callsites) | 1099 | u32 *callsites) |
| 1067 | { | 1100 | { |
| @@ -1071,9 +1104,21 @@ static void arg_track_xfer(struct bpf_verifier_env *env, struct bpf_insn *insn, · 3 unmodified lines | |||
| 1071 | struct arg_track *dst = &at_out[insn->dst_reg]; | 1104 | struct arg_track *dst = &at_out[insn->dst_reg]; |
| 1072 | struct arg_track *src = &at_out[insn->src_reg]; | 1105 | struct arg_track *src = &at_out[insn->src_reg]; |
| 1073 | struct arg_track none = { .frame = ARG_NONE }; | 1106 | struct arg_track none = { .frame = ARG_NONE }; |
| 1074 | int r; | 1107 | int r, slot; |
| 1075 | 1108 | ||
| 1076 | if (class == BPF_ALU64 && BPF_SRC(insn->code) == BPF_K) { | 1109 | /* Handle stack arg stores and loads. */ |
| 1110 | if (is_stack_arg_st(insn) || is_stack_arg_stx(insn)) { | ||
| 1111 | slot = stack_arg_off_to_slot(insn->off); | ||
| 1112 | if (slot >= 0) { | ||
| 1113 | if (is_stack_arg_stx(insn)) | ||
| 1114 | at_out[MAX_BPF_REG + slot] = at_out[insn->src_reg]; | ||
| 1115 | else | ||
| 1116 | at_out[MAX_BPF_REG + slot] = none; | ||
| 1117 | } | ||
| 1118 | } else if (is_stack_arg_ldx(insn)) { | ||
| 1119 | slot = stack_arg_off_to_slot(insn->off); | ||
| 1120 | at_out[insn->dst_reg] = (slot >= 0) ? at_stack_arg_entry[slot] : none; | ||
| 1121 | } else if (class == BPF_ALU64 && BPF_SRC(insn->code) == BPF_K) { | ||
| 1077 | if (code == BPF_MOV) { | 1122 | if (code == BPF_MOV) { |
| 1078 | *dst = none; | 1123 | *dst = none; |
| 1079 | } else if (dst->frame >= 0) { | 1124 | } else if (dst->frame >= 0) { |
| @@ -1297,6 +1342,16 @@ static int record_load_store_access(struct bpf_verifier_env *env, · 217 unmodified lines | |||
| 1297 | struct arg_track resolved, *ptr; | 1342 | struct arg_track resolved, *ptr; |
| 1298 | int oi; | 1343 | int oi; |
| 1299 | 1344 | ||
| 1345 | /* | ||
| 1346 | * Stack arg insns use dst_reg/src_reg=BPF_REG_PARAMS(11). Since at[] | ||
| 1347 | * is extended to MAX_AT_TRACK_REGS, at[11] holds the arg_track for | ||
| 1348 | * outgoing stack arg slot 0 — not the pointer used for the memory | ||
| 1349 | * access. Skip so the slot's tracked value isn't confused with the | ||
| 1350 | * base register that record_stack_access() expects. | ||
| 1351 | */ | ||
| 1352 | if (is_stack_arg_stx(insn) || is_stack_arg_st(insn) || is_stack_arg_ldx(insn)) | ||
| 1353 | return 0; | ||
| 1354 | |||
| 1300 | switch (class) { | 1355 | switch (class) { |
| 1301 | case BPF_LDX: | 1356 | case BPF_LDX: |
| 1302 | ptr = &at[insn->src_reg]; | 1357 | ptr = &at[insn->src_reg]; |
| @@ -1343,6 +1398,42 @@ static int record_load_store_access(struct bpf_verifier_env *env, · 40 unmodified lines | |||
| 1343 | return 0; | 1398 | return 0; |
| 1344 | } | 1399 | } |
| 1345 | 1400 | ||
| 1401 | static int record_arg_access(struct bpf_verifier_env *env, | ||
| 1402 | struct func_instance *instance, | ||
| 1403 | struct bpf_insn *insn, | ||
| 1404 | struct arg_track *at, int arg_idx, | ||
| 1405 | int insn_idx) | ||
| 1406 | { | ||
| 1407 | int depth = instance->depth; | ||
| 1408 | int frame = at->frame; | ||
| 1409 | int err = 0; | ||
| 1410 | s64 bytes; | ||
| 1411 | |||
| 1412 | if (!arg_is_fp(at)) | ||
| 1413 | return 0; | ||
| 1414 | |||
| 1415 | if (bpf_helper_call(insn)) { | ||
| 1416 | bytes = bpf_helper_stack_access_bytes(env, insn, arg_idx, insn_idx); | ||
| 1417 | } else if (bpf_pseudo_kfunc_call(insn)) { | ||
| 1418 | bytes = bpf_kfunc_stack_access_bytes(env, insn, arg_idx, insn_idx); | ||
| 1419 | } else { | ||
| 1420 | for (int f = 0; f <= depth; f++) { | ||
| 1421 | err = mark_stack_read(instance, f, insn_idx, SPIS_ALL); | ||
| 1422 | if (err) | ||
| 1423 | return err; | ||
| 1424 | } | ||
| 1425 | return 0; | ||
| 1426 | } | ||
| 1427 | if (bytes == 0) | ||
| 1428 | return 0; | ||
| 1429 | |||
| 1430 | if (frame >= 0 && frame <= depth) | ||
| 1431 | err = record_stack_access(instance, at, bytes, frame, insn_idx); | ||
| 1432 | else if (frame == ARG_IMPRECISE) | ||
| 1433 | err = record_imprecise(instance, at->mask, insn_idx); | ||
| 1434 | return err; | ||
| 1435 | } | ||
| 1436 | |||
| 1346 | /* Record stack access for a given 'at' state of helper/kfunc 'insn' */ | 1437 | /* Record stack access for a given 'at' state of helper/kfunc 'insn' */ |
| 1347 | static int record_call_access(struct bpf_verifier_env *env, | 1438 | static int record_call_access(struct bpf_verifier_env *env, |
| 1348 | struct func_instance *instance, | 1439 | struct func_instance *instance, |
| @@ -1350,9 +1441,8 @@ static int record_call_access(struct bpf_verifier_env *env, · 1 unmodified line | |||
| 1350 | int insn_idx) | 1441 | int insn_idx) |
| 1351 | { | 1442 | { |
| 1352 | struct bpf_insn *insn = &env->prog->insnsi[insn_idx]; | 1443 | struct bpf_insn *insn = &env->prog->insnsi[insn_idx]; |
| 1353 | int depth = instance->depth; | ||
| 1354 | struct bpf_call_summary cs; | 1444 | struct bpf_call_summary cs; |
| 1355 | int r, err = 0, num_params = 5; | 1445 | int r, err, num_params = 5; |
| 1356 | 1446 | ||
| 1357 | if (bpf_pseudo_call(insn)) | 1447 | if (bpf_pseudo_call(insn)) |
| 1358 | return 0; | 1448 | return 0; |
| @@ -1360,32 +1450,15 @@ static int record_call_access(struct bpf_verifier_env *env, · 1 unmodified line | |||
| 1360 | if (bpf_get_call_summary(env, insn, &cs)) | 1450 | if (bpf_get_call_summary(env, insn, &cs)) |
| 1361 | num_params = cs.num_params; | 1451 | num_params = cs.num_params; |
| 1362 | 1452 | ||
| 1363 | for (r = BPF_REG_1; r < BPF_REG_1 + num_params; r++) { | 1453 | for (r = BPF_REG_1; r < BPF_REG_1 + min(num_params, MAX_BPF_FUNC_REG_ARGS); r++) { |
| 1364 | int frame = at[r].frame; | 1454 | err = record_arg_access(env, instance, insn, &at[r], r - 1, insn_idx); |
| 1365 | s64 bytes; | 1455 | if (err) |
| 1366 | 1456 | return err; | |
| 1367 | if (!arg_is_fp(&at[r])) | 1457 | } |
| 1368 | continue; | ||
| 1369 | |||
| 1370 | if (bpf_helper_call(insn)) { | ||
| 1371 | bytes = bpf_helper_stack_access_bytes(env, insn, r - 1, insn_idx); | ||
| 1372 | } else if (bpf_pseudo_kfunc_call(insn)) { | ||
| 1373 | bytes = bpf_kfunc_stack_access_bytes(env, insn, r - 1, insn_idx); | ||
| 1374 | } else { | ||
| 1375 | for (int f = 0; f <= depth; f++) { | ||
| 1376 | err = mark_stack_read(instance, f, insn_idx, SPIS_ALL); | ||
| 1377 | if (err) | ||
| 1378 | return err; | ||
| 1379 | } | ||
| 1380 | return 0; | ||
| 1381 | } | ||
| 1382 | if (bytes == 0) | ||
| 1383 | continue; | ||
| 1384 | 1458 | ||
| 1385 | if (frame >= 0 && frame <= depth) | 1459 | for (r = 0; r < MAX_STACK_ARG_SLOTS && r < num_params - MAX_BPF_FUNC_REG_ARGS; r++) { |
| 1386 | err = record_stack_access(instance, &at[r], bytes, frame, insn_idx); | 1460 | err = record_arg_access(env, instance, insn, &at[MAX_BPF_REG + r], |
| 1387 | else if (frame == ARG_IMPRECISE) | 1461 | r + MAX_BPF_FUNC_REG_ARGS, insn_idx); |
| 1388 | err = record_imprecise(instance, at[r].mask, insn_idx); | ||
| 1389 | if (err) | 1462 | if (err) |
| 1390 | return err; | 1463 | return err; |
| 1391 | } | 1464 | } |
| @@ -1445,7 +1518,7 @@ static int find_callback_subprog(struct bpf_verifier_env *env, · 53 unmodified lines | |||
| 1445 | 1518 | ||
| 1446 | /* Per-subprog intermediate state kept alive across analysis phases */ | 1519 | /* Per-subprog intermediate state kept alive across analysis phases */ |
| 1447 | struct subprog_at_info { | 1520 | struct subprog_at_info { |
| 1448 | struct arg_track (*at_in)[MAX_BPF_REG]; | 1521 | struct arg_track (*at_in)[MAX_AT_TRACK_REGS]; |
| 1449 | int len; | 1522 | int len; |
| 1450 | }; | 1523 | }; |
| 1451 | 1524 | ||
| @@ -1479,6 +1552,9 @@ static void print_subprog_arg_access(struct bpf_verifier_env *env, · 27 unmodified lines | |||
| 1479 | for (r = 0; r < MAX_BPF_REG - 1; r++) | 1552 | for (r = 0; r < MAX_BPF_REG - 1; r++) |
| 1480 | if (arg_is_fp(&info->at_in[i][r])) | 1553 | if (arg_is_fp(&info->at_in[i][r])) |
| 1481 | has_extra = true; | 1554 | has_extra = true; |
| 1555 | for (r = 0; r < MAX_STACK_ARG_SLOTS; r++) | ||
| 1556 | if (arg_is_fp(&info->at_in[i][MAX_BPF_REG + r])) | ||
| 1557 | has_extra = true; | ||
| 1482 | } | 1558 | } |
| 1483 | if (is_ldx_stx_call) { | 1559 | if (is_ldx_stx_call) { |
| 1484 | for (r = 0; r < MAX_ARG_SPILL_SLOTS; r++) | 1560 | for (r = 0; r < MAX_ARG_SPILL_SLOTS; r++) |
| @@ -1503,6 +1579,12 @@ static void print_subprog_arg_access(struct bpf_verifier_env *env, · 18 unmodified lines | |||
| 1503 | verbose(env, " r%d=", r); | 1579 | verbose(env, " r%d=", r); |
| 1504 | verbose_arg_track(env, &info->at_in[i][r]); | 1580 | verbose_arg_track(env, &info->at_in[i][r]); |
| 1505 | } | 1581 | } |
| 1582 | for (r = 0; r < MAX_STACK_ARG_SLOTS; r++) { | ||
| 1583 | if (!arg_is_fp(&info->at_in[i][MAX_BPF_REG + r])) | ||
| 1584 | continue; | ||
| 1585 | verbose(env, " sa%d=", r); | ||
| 1586 | verbose_arg_track(env, &info->at_in[i][MAX_BPF_REG + r]); | ||
| 1587 | } | ||
| 1506 | } | 1588 | } |
| 1507 | 1589 | ||
| 1508 | if (is_ldx_stx_call) { | 1590 | if (is_ldx_stx_call) { |
| @@ -1525,7 +1607,7 @@ static void print_subprog_arg_access(struct bpf_verifier_env *env, · 16 unmodified lines | |||
| 1525 | * Runs forward fixed-point with arg_track_xfer(), then records | 1607 | * Runs forward fixed-point with arg_track_xfer(), then records |
| 1526 | * memory accesses in a single linear pass over converged state. | 1608 | * memory accesses in a single linear pass over converged state. |
| 1527 | * | 1609 | * |
| 1528 | * @callee_entry: pre-populated entry state for R1-R5 | 1610 | * @callee_entry: pre-populated entry state for R1-R5 and stack args |
| 1529 | * NULL for main (subprog 0). | 1611 | * NULL for main (subprog 0). |
| 1530 | * @info: stores at_in, len for debug printing. | 1612 | * @info: stores at_in, len for debug printing. |
| 1531 | */ | 1613 | */ |
| @@ -1543,10 +1625,11 @@ static int compute_subprog_args(struct bpf_verifier_env *env, · 11 unmodified lines | |||
| 1543 | int end = env->subprog_info[subprog + 1].start; | 1625 | int end = env->subprog_info[subprog + 1].start; |
| 1544 | int po_end = env->subprog_info[subprog + 1].postorder_start; | 1626 | int po_end = env->subprog_info[subprog + 1].postorder_start; |
| 1545 | int len = end - start; | 1627 | int len = end - start; |
| 1546 | struct arg_track (*at_in)[MAX_BPF_REG] = NULL; | 1628 | struct arg_track (*at_in)[MAX_AT_TRACK_REGS] = NULL; |
| 1547 | struct arg_track at_out[MAX_BPF_REG]; | 1629 | struct arg_track at_out[MAX_AT_TRACK_REGS]; |
| 1548 | struct arg_track (*at_stack_in)[MAX_ARG_SPILL_SLOTS] = NULL; | 1630 | struct arg_track (*at_stack_in)[MAX_ARG_SPILL_SLOTS] = NULL; |
| 1549 | struct arg_track *at_stack_out = NULL; | 1631 | struct arg_track *at_stack_out = NULL; |
| 1632 | struct arg_track at_stack_arg_entry[MAX_STACK_ARG_SLOTS]; | ||
| 1550 | struct arg_track unvisited = { .frame = ARG_UNVISITED }; | 1633 | struct arg_track unvisited = { .frame = ARG_UNVISITED }; |
| 1551 | struct arg_track none = { .frame = ARG_NONE }; | 1634 | struct arg_track none = { .frame = ARG_NONE }; |
| 1552 | bool changed; | 1635 | bool changed; |
| @@ -1565,13 +1648,13 @@ static int compute_subprog_args(struct bpf_verifier_env *env, · 12 unmodified lines | |||
| 1565 | goto err_free; | 1648 | goto err_free; |
| 1566 | 1649 | ||
| 1567 | for (i = 0; i < len; i++) { | 1650 | for (i = 0; i < len; i++) { |
| 1568 | for (r = 0; r < MAX_BPF_REG; r++) | 1651 | for (r = 0; r < MAX_AT_TRACK_REGS; r++) |
| 1569 | at_in[i][r] = unvisited; | 1652 | at_in[i][r] = unvisited; |
| 1570 | for (r = 0; r < MAX_ARG_SPILL_SLOTS; r++) | 1653 | for (r = 0; r < MAX_ARG_SPILL_SLOTS; r++) |
| 1571 | at_stack_in[i][r] = unvisited; | 1654 | at_stack_in[i][r] = unvisited; |
| 1572 | } | 1655 | } |
| 1573 | 1656 | ||
| 1574 | for (r = 0; r < MAX_BPF_REG; r++) | 1657 | for (r = 0; r < MAX_AT_TRACK_REGS; r++) |
| 1575 | at_in[0][r] = none; | 1658 | at_in[0][r] = none; |
| 1576 | 1659 | ||
| 1577 | /* Entry: R10 is always precisely the current frame's FP */ | 1660 | /* Entry: R10 is always precisely the current frame's FP */ |
| @@ -1587,6 +1670,10 @@ static int compute_subprog_args(struct bpf_verifier_env *env, · 9 unmodified lines | |||
| 1587 | for (r = 0; r < MAX_ARG_SPILL_SLOTS; r++) | 1670 | for (r = 0; r < MAX_ARG_SPILL_SLOTS; r++) |
| 1588 | at_stack_in[0][r] = none; | 1671 | at_stack_in[0][r] = none; |
| 1589 | 1672 | ||
| 1673 | /* Entry: incoming stack args from caller, or ARG_NONE for main */ | ||
| 1674 | for (r = 0; r < MAX_STACK_ARG_SLOTS; r++) | ||
| 1675 | at_stack_arg_entry[r] = callee_entry ? callee_entry[MAX_BPF_REG + r] : none; | ||
| 1676 | |||
| 1590 | if (env->log.level & BPF_LOG_LEVEL2) | 1677 | if (env->log.level & BPF_LOG_LEVEL2) |
| 1591 | verbose(env, "subprog#%d: analyzing (depth %d)...\n", subprog, depth); | 1678 | verbose(env, "subprog#%d: analyzing (depth %d)...\n", subprog, depth); |
| 1592 | 1679 | ||
| @@ -1605,7 +1692,8 @@ redo: · 12 unmodified lines | |||
| 1605 | memcpy(at_out, at_in[i], sizeof(at_out)); | 1692 | memcpy(at_out, at_in[i], sizeof(at_out)); |
| 1606 | memcpy(at_stack_out, at_stack_in[i], MAX_ARG_SPILL_SLOTS * sizeof(*at_stack_out)); | 1693 | memcpy(at_stack_out, at_stack_in[i], MAX_ARG_SPILL_SLOTS * sizeof(*at_stack_out)); |
| 1607 | 1694 | ||
| 1608 | arg_track_xfer(env, insn, idx, at_out, at_stack_out, instance, callsites); | 1695 | arg_track_xfer(env, insn, idx, at_out, at_stack_out, |
| 1696 | at_stack_arg_entry, instance, callsites); | ||
| 1609 | arg_track_log(env, insn, idx, at_in[i], at_stack_in[i], at_out, at_stack_out); | 1697 | arg_track_log(env, insn, idx, at_in[i], at_stack_in[i], at_out, at_stack_out); |
| 1610 | 1698 | ||
| 1611 | /* Propagate to successors within this subprogram */ | 1699 | /* Propagate to successors within this subprogram */ |
| @@ -1619,7 +1707,7 @@ redo: · 7 unmodified lines | |||
| 1619 | continue; | 1707 | continue; |
| 1620 | ti = target - start; | 1708 | ti = target - start; |
| 1621 | 1709 | ||
| 1622 | for (r = 0; r < MAX_BPF_REG; r++) | 1710 | for (r = 0; r < MAX_AT_TRACK_REGS; r++) |
| 1623 | changed |= arg_track_join(env, idx, target, r, | 1711 | changed |= arg_track_join(env, idx, target, r, |
| 1624 | &at_in[ti][r], at_out[r]); | 1712 | &at_in[ti][r], at_out[r]); |
| 1625 | 1713 | ||
| @@ -1674,11 +1762,14 @@ err_free: · 48 unmodified lines | |||
| 1674 | return err; | 1762 | return err; |
| 1675 | } | 1763 | } |
| 1676 | 1764 | ||
| 1677 | /* Return true if any of R1-R5 is derived from a frame pointer. */ | 1765 | /* Return true if any of R1-R5 or stack args is derived from a frame pointer. */ |
| 1678 | static bool has_fp_args(struct arg_track *args) | 1766 | static bool has_fp_args(struct arg_track *args) |
| 1679 | { | 1767 | { |
| 1680 | for (int r = BPF_REG_1; r <= BPF_REG_5; r++) | 1768 | for (int r = BPF_REG_1; r <= BPF_REG_5; r++) |
| 1681 | if (args[r].frame != ARG_NONE) | 1769 | if (arg_is_fp(&args[r])) |
| 1770 | return true; | ||
| 1771 | for (int r = 0; r < MAX_STACK_ARG_SLOTS; r++) | ||
| 1772 | if (arg_is_fp(&args[MAX_BPF_REG + r])) | ||
| 1682 | return true; | 1773 | return true; |
| 1683 | return false; | 1774 | return false; |
| 1684 | } | 1775 | } |
| @@ -1803,7 +1894,7 @@ static int analyze_subprog(struct bpf_verifier_env *env, · 118 unmodified lines | |||
| 1803 | /* For each reachable call site in the subprog, recurse into callees */ | 1894 | /* For each reachable call site in the subprog, recurse into callees */ |
| 1804 | for (int p = po_start; p < po_end; p++) { | 1895 | for (int p = po_start; p < po_end; p++) { |
| 1805 | int idx = env->cfg.insn_postorder[p]; | 1896 | int idx = env->cfg.insn_postorder[p]; |
| 1806 | struct arg_track callee_args[BPF_REG_5 + 1]; | 1897 | struct arg_track callee_args[MAX_AT_TRACK_REGS] = {}; |
| 1807 | struct arg_track none = { .frame = ARG_NONE }; | 1898 | struct arg_track none = { .frame = ARG_NONE }; |
| 1808 | struct bpf_insn *insn = &insns[idx]; | 1899 | struct bpf_insn *insn = &insns[idx]; |
| 1809 | struct func_instance *callee_instance; | 1900 | struct func_instance *callee_instance; |
| @@ -1818,9 +1909,11 @@ static int analyze_subprog(struct bpf_verifier_env *env, · 8 unmodified lines | |||
| 1818 | if (callee < 0) | 1909 | if (callee < 0) |
| 1819 | continue; | 1910 | continue; |
| 1820 | 1911 | ||
| 1821 | /* Build entry args: R1-R5 from at_in at call site */ | 1912 | /* Build entry args: R1-R5 and stack args from at_in at call site */ |
| 1822 | for (int r = BPF_REG_1; r <= BPF_REG_5; r++) | 1913 | for (int r = BPF_REG_1; r <= BPF_REG_5; r++) |
| 1823 | callee_args[r] = info[subprog].at_in[j][r]; | 1914 | callee_args[r] = info[subprog].at_in[j][r]; |
| 1915 | for (int r = 0; r < MAX_STACK_ARG_SLOTS; r++) | ||
| 1916 | callee_args[MAX_BPF_REG + r] = info[subprog].at_in[j][MAX_BPF_REG + r]; | ||
| 1824 | } else if (bpf_calls_callback(env, idx)) { | 1917 | } else if (bpf_calls_callback(env, idx)) { |
| 1825 | callee = find_callback_subprog(env, insn, idx, &caller_reg, &cb_callee_reg); | 1918 | callee = find_callback_subprog(env, insn, idx, &caller_reg, &cb_callee_reg); |
| 1826 | if (callee == -2) { | 1919 | if (callee == -2) { |
| @@ -1842,6 +1935,8 @@ static int analyze_subprog(struct bpf_verifier_env *env, · 15 unmodified lines | |||
| 1842 | 1935 | ||
| 1843 | for (int r = BPF_REG_1; r <= BPF_REG_5; r++) | 1936 | for (int r = BPF_REG_1; r <= BPF_REG_5; r++) |
| 1844 | callee_args[r] = none; | 1937 | callee_args[r] = none; |
| 1938 | for (int r = 0; r < MAX_STACK_ARG_SLOTS; r++) | ||
| 1939 | callee_args[MAX_BPF_REG + r] = none; | ||
| 1845 | callee_args[cb_callee_reg] = info[subprog].at_in[j][caller_reg]; | 1940 | callee_args[cb_callee_reg] = info[subprog].at_in[j][caller_reg]; |
| 1846 | } else { | 1941 | } else { |
| 1847 | continue; | 1942 | continue; |
| @@ -2085,7 +2180,7 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env, · 237 unmodified lines | |||
| 2085 | def = ALL_CALLER_SAVED_REGS; | 2180 | def = ALL_CALLER_SAVED_REGS; |
| 2086 | use = def & ~BIT(BPF_REG_0); | 2181 | use = def & ~BIT(BPF_REG_0); |
| 2087 | if (bpf_get_call_summary(env, insn, &cs)) | 2182 | if (bpf_get_call_summary(env, insn, &cs)) |
| 2088 | use = GENMASK(cs.num_params, 1); | 2183 | use = GENMASK(min_t(u8, cs.num_params, MAX_BPF_FUNC_REG_ARGS), 1); |
| 2089 | break; | 2184 | break; |
| 2090 | default: | 2185 | default: |
| 2091 | def = 0; | 2186 | def = 0; |
| @@ -13,17 +13,17 @@ · 12 unmodified lines | |||
| 13 | 13 | ||
| 14 | #define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args) | 14 | #define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args) |
| 15 | 15 | ||
| 16 | static bool bpf_verifier_log_attr_valid(const struct bpf_verifier_log *log) | 16 | static bool bpf_verifier_log_attr_valid(u32 log_level, char __user *log_buf, u32 log_size) |
| 17 | { | 17 | { |
| 18 | /* ubuf and len_total should both be specified (or not) together */ | 18 | /* ubuf and len_total should both be specified (or not) together */ |
| 19 | if (!!log->ubuf != !!log->len_total) | 19 | if (!!log_buf != !!log_size) |
| 20 | return false; | 20 | return false; |
| 21 | /* log buf without log_level is meaningless */ | 21 | /* log buf without log_level is meaningless */ |
| 22 | if (log->ubuf && log->level == 0) | 22 | if (log_buf && log_level == 0) |
| 23 | return false; | 23 | return false; |
| 24 | if (log->level & ~BPF_LOG_MASK) | 24 | if (log_level & ~BPF_LOG_MASK) |
| 25 | return false; | 25 | return false; |
| 26 | if (log->len_total > UINT_MAX >> 2) | 26 | if (log_size > UINT_MAX >> 2) |
| 27 | return false; | 27 | return false; |
| 28 | return true; | 28 | return true; |
| 29 | } | 29 | } |
| @@ -36,7 +36,7 @@ int bpf_vlog_init(struct bpf_verifier_log *log, u32 log_level, · 6 unmodified lines | |||
| 36 | log->len_total = log_size; | 36 | log->len_total = log_size; |
| 37 | 37 | ||
| 38 | /* log attributes have to be sane */ | 38 | /* log attributes have to be sane */ |
| 39 | if (!bpf_verifier_log_attr_valid(log)) | 39 | if (!bpf_verifier_log_attr_valid(log_level, log_buf, log_size)) |
| 40 | return -EINVAL; | 40 | return -EINVAL; |
| 41 | 41 | ||
| 42 | return 0; | 42 | return 0; |
| @@ -571,20 +571,20 @@ static void print_scalar_ranges(struct bpf_verifier_env *env, · 528 unmodified lines | |||
| 571 | u64 val; | 571 | u64 val; |
| 572 | bool omit; | 572 | bool omit; |
| 573 | } minmaxs[] = { | 573 | } minmaxs[] = { |
| 574 | {"smin", reg->smin_value, reg->smin_value == S64_MIN}, | 574 | {"smin", reg_smin(reg), reg_smin(reg) == S64_MIN}, |
| 575 | {"smax", reg->smax_value, reg->smax_value == S64_MAX}, | 575 | {"smax", reg_smax(reg), reg_smax(reg) == S64_MAX}, |
| 576 | {"umin", reg->umin_value, reg->umin_value == 0}, | 576 | {"umin", reg_umin(reg), reg_umin(reg) == 0}, |
| 577 | {"umax", reg->umax_value, reg->umax_value == U64_MAX}, | 577 | {"umax", reg_umax(reg), reg_umax(reg) == U64_MAX}, |
| 578 | {"smin32", | 578 | {"smin32", |
| 579 | is_snum_decimal((s64)reg->s32_min_value) | 579 | is_snum_decimal((s64)reg_s32_min(reg)) |
| 580 | ? (s64)reg->s32_min_value | 580 | ? (s64)reg_s32_min(reg) |
| 581 | : (u32)reg->s32_min_value, reg->s32_min_value == S32_MIN}, | 581 | : (u32)reg_s32_min(reg), reg_s32_min(reg) == S32_MIN}, |
| 582 | {"smax32", | 582 | {"smax32", |
| 583 | is_snum_decimal((s64)reg->s32_max_value) | 583 | is_snum_decimal((s64)reg_s32_max(reg)) |
| 584 | ? (s64)reg->s32_max_value | 584 | ? (s64)reg_s32_max(reg) |
| 585 | : (u32)reg->s32_max_value, reg->s32_max_value == S32_MAX}, | 585 | : (u32)reg_s32_max(reg), reg_s32_max(reg) == S32_MAX}, |
| 586 | {"umin32", reg->u32_min_value, reg->u32_min_value == 0}, | 586 | {"umin32", reg_u32_min(reg), reg_u32_min(reg) == 0}, |
| 587 | {"umax32", reg->u32_max_value, reg->u32_max_value == U32_MAX}, | 587 | {"umax32", reg_u32_max(reg), reg_u32_max(reg) == U32_MAX}, |
| 588 | }, *m1, *m2, *mend = &minmaxs[ARRAY_SIZE(minmaxs)]; | 588 | }, *m1, *m2, *mend = &minmaxs[ARRAY_SIZE(minmaxs)]; |
| 589 | bool neg1, neg2; | 589 | bool neg1, neg2; |
| 590 | 590 | ||
| @@ -665,8 +665,8 @@ static void print_reg_state(struct bpf_verifier_env *env, · 74 unmodified lines | |||
| 665 | verbose_a("id=%d", reg->id & ~BPF_ADD_CONST); | 665 | verbose_a("id=%d", reg->id & ~BPF_ADD_CONST); |
| 666 | if (reg->id & BPF_ADD_CONST) | 666 | if (reg->id & BPF_ADD_CONST) |
| 667 | verbose(env, "%+d", reg->delta); | 667 | verbose(env, "%+d", reg->delta); |
| 668 | if (reg->ref_obj_id) | 668 | if (reg->parent_id) |
| 669 | verbose_a("ref_obj_id=%d", reg->ref_obj_id); | 669 | verbose_a("parent_id=%d", reg->parent_id); |
| 670 | if (type_is_non_owning_ref(reg->type)) | 670 | if (type_is_non_owning_ref(reg->type)) |
| 671 | verbose_a("%s", "non_own_ref"); | 671 | verbose_a("%s", "non_own_ref"); |
| 672 | if (type_is_map_ptr(t)) { | 672 | if (type_is_map_ptr(t)) { |
| @@ -768,21 +768,19 @@ void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_verifie · 95 unmodified lines | |||
| 768 | verbose(env, "=dynptr_%s(", dynptr_type_str(reg->dynptr.type)); | 768 | verbose(env, "=dynptr_%s(", dynptr_type_str(reg->dynptr.type)); |
| 769 | if (reg->id) | 769 | if (reg->id) |
| 770 | verbose_a("id=%d", reg->id); | 770 | verbose_a("id=%d", reg->id); |
| 771 | if (reg->ref_obj_id) | 771 | if (reg->parent_id) |
| 772 | verbose_a("ref_id=%d", reg->ref_obj_id); | 772 | verbose_a("parent_id=%d", reg->parent_id); |
| 773 | if (reg->dynptr_id) | ||
| 774 | verbose_a("dynptr_id=%d", reg->dynptr_id); | ||
| 775 | verbose(env, ")"); | 773 | verbose(env, ")"); |
| 776 | break; | 774 | break; |
| 777 | case STACK_ITER: | 775 | case STACK_ITER: |
| 778 | /* only main slot has ref_obj_id set; skip others */ | 776 | /* only main slot has id set; skip others */ |
| 779 | if (!reg->ref_obj_id) | 777 | if (!reg->id) |
| 780 | continue; | 778 | continue; |
| 781 | 779 | ||
| 782 | verbose(env, " fp%d=iter_%s(ref_id=%d,state=%s,depth=%u)", | 780 | verbose(env, " fp%d=iter_%s(id=%d,state=%s,depth=%u)", |
| 783 | (-i - 1) * BPF_REG_SIZE, | 781 | (-i - 1) * BPF_REG_SIZE, |
| 784 | iter_type_str(reg->iter.btf, reg->iter.btf_id), | 782 | iter_type_str(reg->iter.btf, reg->iter.btf_id), |
| 785 | reg->ref_obj_id, iter_state_str(reg->iter.state), | 783 | reg->id, iter_state_str(reg->iter.state), |
| 786 | reg->iter.depth); | 784 | reg->iter.depth); |
| 787 | break; | 785 | break; |
| 788 | case STACK_MISC: | 786 | case STACK_MISC: |
| @@ -825,3 +823,81 @@ void print_insn_state(struct bpf_verifier_env *env, const struct bpf_verifier_st · 36 unmodified lines | |||
| 825 | } | 823 | } |
| 826 | print_verifier_state(env, vstate, frameno, false); | 824 | print_verifier_state(env, vstate, frameno, false); |
| 827 | } | 825 | } |
| 826 | |||
| 827 | int bpf_log_attr_init(struct bpf_log_attr *log, u64 log_buf, u32 log_size, u32 log_level, | ||
| 828 | u32 offsetof_log_true_size, bpfptr_t uattr, struct bpf_common_attr *common, | ||
| 829 | bpfptr_t uattr_common, u32 size_common) | ||
| 830 | { | ||
| 831 | char __user *ubuf_common = u64_to_user_ptr(common->log_buf); | ||
| 832 | char __user *ubuf = u64_to_user_ptr(log_buf); | ||
| 833 | |||
| 834 | if (!bpf_verifier_log_attr_valid(common->log_level, ubuf_common, common->log_size) || | ||
| 835 | !bpf_verifier_log_attr_valid(log_level, ubuf, log_size)) | ||
| 836 | return -EINVAL; | ||
| 837 | |||
| 838 | if (ubuf && ubuf_common && (ubuf != ubuf_common || log_size != common->log_size || | ||
| 839 | log_level != common->log_level)) | ||
| 840 | return -EINVAL; | ||
| 841 | |||
| 842 | memset(log, 0, sizeof(*log)); | ||
| 843 | log->ubuf = ubuf; | ||
| 844 | log->size = log_size; | ||
| 845 | log->level = log_level; | ||
| 846 | log->offsetof_true_size = offsetof_log_true_size; | ||
| 847 | log->uattr = uattr; | ||
| 848 | |||
| 849 | if (!ubuf && ubuf_common) { | ||
| 850 | log->ubuf = ubuf_common; | ||
| 851 | log->size = common->log_size; | ||
| 852 | log->level = common->log_level; | ||
| 853 | log->uattr = uattr_common; | ||
| 854 | log->offsetof_true_size = 0; | ||
| 855 | if (size_common >= offsetofend(struct bpf_common_attr, log_true_size)) | ||
| 856 | log->offsetof_true_size = offsetof(struct bpf_common_attr, log_true_size); | ||
| 857 | } | ||
| 858 | return 0; | ||
| 859 | } | ||
| 860 | |||
| 861 | struct bpf_verifier_log *bpf_log_attr_create_vlog(struct bpf_log_attr *attr_log, | ||
| 862 | struct bpf_common_attr *common, bpfptr_t uattr, | ||
| 863 | u32 size) | ||
| 864 | { | ||
| 865 | struct bpf_verifier_log *log; | ||
| 866 | int err; | ||
| 867 | |||
| 868 | memset(attr_log, 0, sizeof(*attr_log)); | ||
| 869 | attr_log->uattr = uattr; | ||
| 870 | if (size >= offsetofend(struct bpf_common_attr, log_true_size)) | ||
| 871 | attr_log->offsetof_true_size = offsetof(struct bpf_common_attr, log_true_size); | ||
| 872 | |||
| 873 | if (!size) | ||
| 874 | return NULL; | ||
| 875 | |||
| 876 | log = kzalloc_obj(*log, GFP_KERNEL); | ||
| 877 | if (!log) | ||
| 878 | return ERR_PTR(-ENOMEM); | ||
| 879 | |||
| 880 | err = bpf_vlog_init(log, common->log_level, u64_to_user_ptr(common->log_buf), | ||
| 881 | common->log_size); | ||
| 882 | if (err) { | ||
| 883 | kfree(log); | ||
| 884 | return ERR_PTR(err); | ||
| 885 | } | ||
| 886 | |||
| 887 | return log; | ||
| 888 | } | ||
| 889 | |||
| 890 | int bpf_log_attr_finalize(struct bpf_log_attr *attr, struct bpf_verifier_log *log) | ||
| 891 | { | ||
| 892 | u32 log_true_size; | ||
| 893 | int err; | ||
| 894 | |||
| 895 | err = bpf_vlog_finalize(log, &log_true_size); | ||
| 896 | |||
| 897 | if (attr->offsetof_true_size && | ||
| 898 | copy_to_bpfptr_offset(attr->uattr, attr->offsetof_true_size, &log_true_size, | ||
| 899 | sizeof(log_true_size))) | ||
| 900 | return -EFAULT; | ||
| 901 | |||
| 902 | return err; | ||
| 903 | } | ||
| @@ -246,7 +246,7 @@ static void *trie_lookup_elem(struct bpf_map *map, void *_key) · 245 unmodified lines | |||
| 246 | 246 | ||
| 247 | /* Start walking the trie from the root node ... */ | 247 | /* Start walking the trie from the root node ... */ |
| 248 | 248 | ||
| 249 | for (node = rcu_dereference_check(trie->root, rcu_read_lock_bh_held()); | 249 | for (node = rcu_dereference_check(trie->root, bpf_rcu_lock_held()); |
| 250 | node;) { | 250 | node;) { |
| 251 | unsigned int next_bit; | 251 | unsigned int next_bit; |
| 252 | size_t matchlen; | 252 | size_t matchlen; |
| @@ -280,7 +280,7 @@ static void *trie_lookup_elem(struct bpf_map *map, void *_key) · 27 unmodified lines | |||
| 280 | */ | 280 | */ |
| 281 | next_bit = extract_bit(key->data, node->prefixlen); | 281 | next_bit = extract_bit(key->data, node->prefixlen); |
| 282 | node = rcu_dereference_check(node->child[next_bit], | 282 | node = rcu_dereference_check(node->child[next_bit], |
| 283 | rcu_read_lock_bh_held()); | 283 | bpf_rcu_lock_held()); |
| 284 | } | 284 | } |
| 285 | 285 | ||
| 286 | if (!found) | 286 | if (!found) |
| @@ -359,7 +359,7 @@ static long trie_update_elem(struct bpf_map *map, · 72 unmodified lines | |||
| 359 | */ | 359 | */ |
| 360 | slot = &trie->root; | 360 | slot = &trie->root; |
| 361 | 361 | ||
| 362 | while ((node = rcu_dereference(*slot))) { | 362 | while ((node = rcu_dereference_protected(*slot, 1))) { |
| 363 | matchlen = longest_prefix_match(trie, node, key); | 363 | matchlen = longest_prefix_match(trie, node, key); |
| 364 | 364 | ||
| 365 | if (node->prefixlen != matchlen || | 365 | if (node->prefixlen != matchlen || |
| @@ -482,7 +482,7 @@ static long trie_delete_elem(struct bpf_map *map, void *_key) · 116 unmodified lines | |||
| 482 | trim = &trie->root; | 482 | trim = &trie->root; |
| 483 | trim2 = trim; | 483 | trim2 = trim; |
| 484 | parent = NULL; | 484 | parent = NULL; |
| 485 | while ((node = rcu_dereference(*trim))) { | 485 | while ((node = rcu_dereference_protected(*trim, 1))) { |
| 486 | matchlen = longest_prefix_match(trie, node, key); | 486 | matchlen = longest_prefix_match(trie, node, key); |
| 487 | 487 | ||
| 488 | if (node->prefixlen != matchlen || | 488 | if (node->prefixlen != matchlen || |
| @@ -20,7 +20,8 @@ struct bpf_map *bpf_map_meta_alloc(int inner_map_ufd) · 19 unmodified lines | |||
| 20 | /* Does not support >1 level map-in-map */ | 20 | /* Does not support >1 level map-in-map */ |
| 21 | if (inner_map->inner_map_meta) | 21 | if (inner_map->inner_map_meta) |
| 22 | return ERR_PTR(-EINVAL); | 22 | return ERR_PTR(-EINVAL); |
| 23 | 23 | if (inner_map->excl_prog_sha) | |
| 24 | return ERR_PTR(-ENOTSUPP); | ||
| 24 | if (!inner_map->ops->map_meta_equal) | 25 | if (!inner_map->ops->map_meta_equal) |
| 25 | return ERR_PTR(-ENOTSUPP); | 26 | return ERR_PTR(-ENOTSUPP); |
| 26 | 27 | ||
| @@ -101,6 +102,8 @@ void *bpf_map_fd_get_ptr(struct bpf_map *map, · 74 unmodified lines | |||
| 101 | inner_map = __bpf_map_get(f); | 102 | inner_map = __bpf_map_get(f); |
| 102 | if (IS_ERR(inner_map)) | 103 | if (IS_ERR(inner_map)) |
| 103 | return inner_map; | 104 | return inner_map; |
| 105 | if (inner_map->excl_prog_sha) | ||
| 106 | return ERR_PTR(-ENOTSUPP); | ||
| 104 | 107 | ||
| 105 | inner_map_meta = map->inner_map_meta; | 108 | inner_map_meta = map->inner_map_meta; |
| 106 | if (inner_map_meta->ops->map_meta_equal(inner_map_meta, inner_map)) | 109 | if (inner_map_meta->ops->map_meta_equal(inner_map_meta, inner_map)) |
| @@ -112,6 +112,10 @@ static int bpf_iter_attach_map(struct bpf_prog *prog, · 111 unmodified lines | |||
| 112 | map = bpf_map_get_with_uref(linfo->map.map_fd); | 112 | map = bpf_map_get_with_uref(linfo->map.map_fd); |
| 113 | if (IS_ERR(map)) | 113 | if (IS_ERR(map)) |
| 114 | return PTR_ERR(map); | 114 | return PTR_ERR(map); |
| 115 | if (map->excl_prog_sha) { | ||
| 116 | err = -EPERM; | ||
| 117 | goto put_map; | ||
| 118 | } | ||
| 115 | 119 | ||
| 116 | if (map->map_type == BPF_MAP_TYPE_PERCPU_HASH || | 120 | if (map->map_type == BPF_MAP_TYPE_PERCPU_HASH || |
| 117 | map->map_type == BPF_MAP_TYPE_LRU_PERCPU_HASH || | 121 | map->map_type == BPF_MAP_TYPE_LRU_PERCPU_HASH || |
| @@ -119,7 +123,8 @@ static int bpf_iter_attach_map(struct bpf_prog *prog, · 1 unmodified line | |||
| 119 | is_percpu = true; | 123 | is_percpu = true; |
| 120 | else if (map->map_type != BPF_MAP_TYPE_HASH && | 124 | else if (map->map_type != BPF_MAP_TYPE_HASH && |
| 121 | map->map_type != BPF_MAP_TYPE_LRU_HASH && | 125 | map->map_type != BPF_MAP_TYPE_LRU_HASH && |
| 122 | map->map_type != BPF_MAP_TYPE_ARRAY) | 126 | map->map_type != BPF_MAP_TYPE_ARRAY && |
| 127 | map->map_type != BPF_MAP_TYPE_RHASH) | ||
| 123 | goto put_map; | 128 | goto put_map; |
| 124 | 129 | ||
| 125 | key_acc_size = prog->aux->max_rdonly_access; | 130 | key_acc_size = prog->aux->max_rdonly_access; |
| @@ -9,6 +9,7 @@ · 8 unmodified lines | |||
| 9 | #include <linux/perf_event.h> | 9 | #include <linux/perf_event.h> |
| 10 | #include <linux/btf_ids.h> | 10 | #include <linux/btf_ids.h> |
| 11 | #include <linux/buildid.h> | 11 | #include <linux/buildid.h> |
| 12 | #include <linux/mmap_lock.h> | ||
| 12 | #include "percpu_freelist.h" | 13 | #include "percpu_freelist.h" |
| 13 | #include "mmap_unlock_work.h" | 14 | #include "mmap_unlock_work.h" |
| 14 | 15 | ||
| @@ -152,6 +153,180 @@ static int fetch_build_id(struct vm_area_struct *vma, unsigned char *build_id, b · 137 unmodified lines | |||
| 152 | : build_id_parse_nofault(vma, build_id, NULL); | 153 | : build_id_parse_nofault(vma, build_id, NULL); |
| 153 | } | 154 | } |
| 154 | 155 | ||
| 156 | static inline void stack_map_build_id_set_ip(struct bpf_stack_build_id *id) | ||
| 157 | { | ||
| 158 | id->status = BPF_STACK_BUILD_ID_IP; | ||
| 159 | memset(id->build_id, 0, BUILD_ID_SIZE_MAX); | ||
| 160 | } | ||
| 161 | |||
| 162 | static inline u64 stack_map_build_id_offset(unsigned long vm_pgoff, | ||
| 163 | unsigned long vm_start, u64 ip) | ||
| 164 | { | ||
| 165 | return (vm_pgoff << PAGE_SHIFT) + ip - vm_start; | ||
| 166 | } | ||
| 167 | |||
| 168 | static inline void stack_map_build_id_set_valid(struct bpf_stack_build_id *id, | ||
| 169 | u64 offset, | ||
| 170 | const unsigned char *build_id) | ||
| 171 | { | ||
| 172 | id->status = BPF_STACK_BUILD_ID_VALID; | ||
| 173 | id->offset = offset; | ||
| 174 | if (id->build_id != build_id) | ||
| 175 | memcpy(id->build_id, build_id, BUILD_ID_SIZE_MAX); | ||
| 176 | } | ||
| 177 | |||
| 178 | struct stack_map_vma_lock { | ||
| 179 | struct vm_area_struct *vma; | ||
| 180 | struct mm_struct *mm; | ||
| 181 | }; | ||
| 182 | |||
| 183 | /* | ||
| 184 | * Acquire a stable read-side reference on the VMA covering @ip. | ||
| 185 | * | ||
| 186 | * With CONFIG_PER_VMA_LOCK=y this returns a VMA with its per-VMA read | ||
| 187 | * lock held and mmap_lock dropped, so the caller may sleep. | ||
| 188 | * | ||
| 189 | * With CONFIG_PER_VMA_LOCK=n it returns a VMA with mmap_lock still | ||
| 190 | * held; the caller must snapshot any fields it needs and pin vm_file | ||
| 191 | * with get_file() before stack_map_unlock_vma() drops mmap_lock, as | ||
| 192 | * the VMA may be split, merged, or freed after that. | ||
| 193 | * | ||
| 194 | * Returns NULL on failure, in which case no lock is held. | ||
| 195 | */ | ||
| 196 | static struct vm_area_struct * | ||
| 197 | stack_map_lock_vma(struct stack_map_vma_lock *lock, unsigned long ip) | ||
| 198 | { | ||
| 199 | struct mm_struct *mm = lock->mm; | ||
| 200 | struct vm_area_struct *vma; | ||
| 201 | |||
| 202 | /* noop under !CONFIG_PER_VMA_LOCK */ | ||
| 203 | vma = lock_vma_under_rcu(mm, ip); | ||
| 204 | if (vma) { | ||
| 205 | lock->vma = vma; | ||
| 206 | return vma; | ||
| 207 | } | ||
| 208 | |||
| 209 | /* | ||
| 210 | * Taking mmap_read_lock() is unsafe here, because the caller BPF | ||
| 211 | * program might already hold it, causing a deadlock. | ||
| 212 | */ | ||
| 213 | if (!mmap_read_trylock(mm)) | ||
| 214 | return NULL; | ||
| 215 | |||
| 216 | vma = vma_lookup(mm, ip); | ||
| 217 | if (!vma) { | ||
| 218 | mmap_read_unlock(mm); | ||
| 219 | return NULL; | ||
| 220 | } | ||
| 221 | |||
| 222 | #ifdef CONFIG_PER_VMA_LOCK | ||
| 223 | if (!vma_start_read_locked(vma)) { | ||
| 224 | mmap_read_unlock(mm); | ||
| 225 | return NULL; | ||
| 226 | } | ||
| 227 | mmap_read_unlock(mm); | ||
| 228 | #endif | ||
| 229 | |||
| 230 | lock->vma = vma; | ||
| 231 | return vma; | ||
| 232 | } | ||
| 233 | |||
| 234 | static void stack_map_unlock_vma(struct stack_map_vma_lock *lock) | ||
| 235 | { | ||
| 236 | #ifdef CONFIG_PER_VMA_LOCK | ||
| 237 | vma_end_read(lock->vma); | ||
| 238 | #else | ||
| 239 | mmap_read_unlock(lock->mm); | ||
| 240 | #endif | ||
| 241 | lock->vma = NULL; | ||
| 242 | } | ||
| 243 | |||
| 244 | static void stack_map_get_build_id_offset_sleepable(struct bpf_stack_build_id *id_offs, | ||
| 245 | u32 trace_nr) | ||
| 246 | { | ||
| 247 | struct mm_struct *mm = current->mm; | ||
| 248 | struct stack_map_vma_lock lock = { .mm = mm }; | ||
| 249 | struct { | ||
| 250 | struct file *file; | ||
| 251 | const unsigned char *build_id; | ||
| 252 | unsigned long vm_start; | ||
| 253 | unsigned long vm_end; | ||
| 254 | unsigned long vm_pgoff; | ||
| 255 | } cache = {}; | ||
| 256 | unsigned long vm_pgoff, vm_start, vm_end; | ||
| 257 | struct vm_area_struct *vma; | ||
| 258 | struct file *file; | ||
| 259 | u64 offset; | ||
| 260 | u64 ip; | ||
| 261 | |||
| 262 | for (u32 i = 0; i < trace_nr; i++) { | ||
| 263 | ip = READ_ONCE(id_offs[i].ip); | ||
| 264 | |||
| 265 | /* | ||
| 266 | * Range cache fast path: if ip falls within the previously | ||
| 267 | * resolved VMA range, reuse the cache build_id without | ||
| 268 | * re-acquiring the VMA lock. | ||
| 269 | */ | ||
| 270 | if (cache.build_id && ip >= cache.vm_start && ip < cache.vm_end) { | ||
| 271 | offset = stack_map_build_id_offset(cache.vm_pgoff, cache.vm_start, ip); | ||
| 272 | stack_map_build_id_set_valid(&id_offs[i], offset, cache.build_id); | ||
| 273 | continue; | ||
| 274 | } | ||
| 275 | |||
| 276 | vma = stack_map_lock_vma(&lock, ip); | ||
| 277 | if (!vma) { | ||
| 278 | stack_map_build_id_set_ip(&id_offs[i]); | ||
| 279 | continue; | ||
| 280 | } | ||
| 281 | if (vma_is_anonymous(vma) || !vma->vm_file) { | ||
| 282 | stack_map_build_id_set_ip(&id_offs[i]); | ||
| 283 | stack_map_unlock_vma(&lock); | ||
| 284 | continue; | ||
| 285 | } | ||
| 286 | |||
| 287 | file = vma->vm_file; | ||
| 288 | vm_pgoff = vma->vm_pgoff; | ||
| 289 | vm_start = vma->vm_start; | ||
| 290 | vm_end = vma->vm_end; | ||
| 291 | offset = stack_map_build_id_offset(vm_pgoff, vm_start, ip); | ||
| 292 | |||
| 293 | /* | ||
| 294 | * Same backing file as previous (e.g. different VMAs | ||
| 295 | * of the same ELF binary). Reuse the cache build_id. | ||
| 296 | */ | ||
| 297 | if (file == cache.file) { | ||
| 298 | stack_map_unlock_vma(&lock); | ||
| 299 | stack_map_build_id_set_valid(&id_offs[i], offset, cache.build_id); | ||
| 300 | cache.vm_start = vm_start; | ||
| 301 | cache.vm_end = vm_end; | ||
| 302 | cache.vm_pgoff = vm_pgoff; | ||
| 303 | continue; | ||
| 304 | } | ||
| 305 | |||
| 306 | file = get_file(file); | ||
| 307 | stack_map_unlock_vma(&lock); | ||
| 308 | |||
| 309 | /* build_id_parse_file() may block on filesystem reads */ | ||
| 310 | if (build_id_parse_file(file, id_offs[i].build_id, NULL)) { | ||
| 311 | stack_map_build_id_set_ip(&id_offs[i]); | ||
| 312 | fput(file); | ||
| 313 | continue; | ||
| 314 | } | ||
| 315 | |||
| 316 | stack_map_build_id_set_valid(&id_offs[i], offset, id_offs[i].build_id); | ||
| 317 | if (cache.file) | ||
| 318 | fput(cache.file); | ||
| 319 | cache.file = file; | ||
| 320 | cache.build_id = id_offs[i].build_id; | ||
| 321 | cache.vm_start = vm_start; | ||
| 322 | cache.vm_end = vm_end; | ||
| 323 | cache.vm_pgoff = vm_pgoff; | ||
| 324 | } | ||
| 325 | |||
| 326 | if (cache.file) | ||
| 327 | fput(cache.file); | ||
| 328 | } | ||
| 329 | |||
| 155 | /* | 330 | /* |
| 156 | * Expects all id_offs[i].ip values to be set to correct initial IPs. | 331 | * Expects all id_offs[i].ip values to be set to correct initial IPs. |
| 157 | * They will be subsequently: | 332 | * They will be subsequently: |
| @@ -165,44 +340,50 @@ static int fetch_build_id(struct vm_area_struct *vma, unsigned char *build_id, b · 7 unmodified lines | |||
| 165 | static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs, | 340 | static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs, |
| 166 | u32 trace_nr, bool user, bool may_fault) | 341 | u32 trace_nr, bool user, bool may_fault) |
| 167 | { | 342 | { |
| 168 | int i; | ||
| 169 | struct mmap_unlock_irq_work *work = NULL; | 343 | struct mmap_unlock_irq_work *work = NULL; |
| 170 | bool irq_work_busy = bpf_mmap_unlock_get_irq_work(&work); | 344 | bool irq_work_busy = bpf_mmap_unlock_get_irq_work(&work); |
| 345 | bool has_user_ctx = user && current && current->mm; | ||
| 171 | struct vm_area_struct *vma, *prev_vma = NULL; | 346 | struct vm_area_struct *vma, *prev_vma = NULL; |
| 172 | const char *prev_build_id; | 347 | const unsigned char *prev_build_id = NULL; |
| 348 | int i; | ||
| 349 | |||
| 350 | if (may_fault && has_user_ctx) { | ||
| 351 | stack_map_get_build_id_offset_sleepable(id_offs, trace_nr); | ||
| 352 | return; | ||
| 353 | } | ||
| 173 | 354 | ||
| 174 | /* If the irq_work is in use, fall back to report ips. Same | 355 | /* If the irq_work is in use, fall back to report ips. Same |
| 175 | * fallback is used for kernel stack (!user) on a stackmap with | 356 | * fallback is used for kernel stack (!user) on a stackmap with |
| 176 | * build_id. | 357 | * build_id. |
| 177 | */ | 358 | */ |
| 178 | if (!user || !current || !current->mm || irq_work_busy || | 359 | if (!has_user_ctx || irq_work_busy || !mmap_read_trylock(current->mm)) { |
| 179 | !mmap_read_trylock(current->mm)) { | ||
| 180 | /* cannot access current->mm, fall back to ips */ | 360 | /* cannot access current->mm, fall back to ips */ |
| 181 | for (i = 0; i < trace_nr; i++) { | 361 | for (i = 0; i < trace_nr; i++) |
| 182 | id_offs[i].status = BPF_STACK_BUILD_ID_IP; | 362 | stack_map_build_id_set_ip(&id_offs[i]); |
| 183 | memset(id_offs[i].build_id, 0, BUILD_ID_SIZE_MAX); | ||
| 184 | } | ||
| 185 | return; | 363 | return; |
| 186 | } | 364 | } |
| 187 | 365 | ||
| 188 | for (i = 0; i < trace_nr; i++) { | 366 | for (i = 0; i < trace_nr; i++) { |
| 189 | u64 ip = READ_ONCE(id_offs[i].ip); | 367 | u64 ip = READ_ONCE(id_offs[i].ip); |
| 368 | u64 offset; | ||
| 190 | 369 | ||
| 191 | if (range_in_vma(prev_vma, ip, ip)) { | 370 | if (prev_build_id && range_in_vma(prev_vma, ip, ip)) { |
| 192 | vma = prev_vma; | 371 | vma = prev_vma; |
| 193 | memcpy(id_offs[i].build_id, prev_build_id, BUILD_ID_SIZE_MAX); | 372 | offset = stack_map_build_id_offset(vma->vm_pgoff, vma->vm_start, ip); |
| 194 | goto build_id_valid; | 373 | stack_map_build_id_set_valid(&id_offs[i], offset, prev_build_id); |
| 374 | continue; | ||
| 195 | } | 375 | } |
| 196 | vma = find_vma(current->mm, ip); | 376 | vma = find_vma(current->mm, ip); |
| 197 | if (!vma || fetch_build_id(vma, id_offs[i].build_id, may_fault)) { | 377 | if (!vma || vma_is_anonymous(vma) || |
| 378 | fetch_build_id(vma, id_offs[i].build_id, may_fault)) { | ||
| 198 | /* per entry fall back to ips */ | 379 | /* per entry fall back to ips */ |
| 199 | id_offs[i].status = BPF_STACK_BUILD_ID_IP; | 380 | stack_map_build_id_set_ip(&id_offs[i]); |
| 200 | memset(id_offs[i].build_id, 0, BUILD_ID_SIZE_MAX); | 381 | prev_vma = vma; |
| 382 | prev_build_id = NULL; | ||
| 201 | continue; | 383 | continue; |
| 202 | } | 384 | } |
| 203 | build_id_valid: | 385 | offset = stack_map_build_id_offset(vma->vm_pgoff, vma->vm_start, ip); |
| 204 | id_offs[i].offset = (vma->vm_pgoff << PAGE_SHIFT) + ip - vma->vm_start; | 386 | stack_map_build_id_set_valid(&id_offs[i], offset, id_offs[i].build_id); |
| 205 | id_offs[i].status = BPF_STACK_BUILD_ID_VALID; | ||
| 206 | prev_vma = vma; | 387 | prev_vma = vma; |
| 207 | prev_build_id = id_offs[i].build_id; | 388 | prev_build_id = id_offs[i].build_id; |
| 208 | } | 389 | } |
| @@ -2,6 +2,7 @@ · 1 unmodified line | |||
| 2 | /* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ | 2 | /* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ |
| 3 | #include <linux/bpf.h> | 3 | #include <linux/bpf.h> |
| 4 | #include <linux/bpf_verifier.h> | 4 | #include <linux/bpf_verifier.h> |
| 5 | #include <linux/cnum.h> | ||
| 5 | #include <linux/filter.h> | 6 | #include <linux/filter.h> |
| 6 | 7 | ||
| 7 | #define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args) | 8 | #define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args) |
| @@ -301,14 +302,8 @@ int bpf_update_branch_counts(struct bpf_verifier_env *env, struct bpf_verifier_s · 293 unmodified lines | |||
| 301 | static bool range_within(const struct bpf_reg_state *old, | 302 | static bool range_within(const struct bpf_reg_state *old, |
| 302 | const struct bpf_reg_state *cur) | 303 | const struct bpf_reg_state *cur) |
| 303 | { | 304 | { |
| 304 | return old->umin_value <= cur->umin_value && | 305 | return cnum64_is_subset(old->r64, cur->r64) && |
| 305 | old->umax_value >= cur->umax_value && | 306 | cnum32_is_subset(old->r32, cur->r32); |
| 306 | old->smin_value <= cur->smin_value && | ||
| 307 | old->smax_value >= cur->smax_value && | ||
| 308 | old->u32_min_value <= cur->u32_min_value && | ||
| 309 | old->u32_max_value >= cur->u32_max_value && | ||
| 310 | old->s32_min_value <= cur->s32_min_value && | ||
| 311 | old->s32_max_value >= cur->s32_max_value; | ||
| 312 | } | 307 | } |
| 313 | 308 | ||
| 314 | /* If in the old state two registers had the same id, then they need to have | 309 | /* If in the old state two registers had the same id, then they need to have |
| @@ -348,8 +343,12 @@ static bool check_ids(u32 old_id, u32 cur_id, struct bpf_idmap *idmap) · 33 unmodified lines | |||
| 348 | return true; | 343 | return true; |
| 349 | } | 344 | } |
| 350 | 345 | ||
| 351 | /* We ran out of idmap slots, which should be impossible */ | 346 | /* |
| 352 | WARN_ON_ONCE(1); | 347 | * idmap slots are bounded by the number of registers and stack slots. |
| 348 | * Since referenced dynptrs acquire intermediate references that do | ||
| 349 | * not live in either, so the map can be exhausted. Since it is unlikely, | ||
| 350 | * fail the verification by treating the states as not equivalent. | ||
| 351 | */ | ||
| 353 | return false; | 352 | return false; |
| 354 | } | 353 | } |
| 355 | 354 | ||
| @@ -494,7 +493,7 @@ static bool regs_exact(const struct bpf_reg_state *rold, · 138 unmodified lines | |||
| 494 | { | 493 | { |
| 495 | return memcmp(rold, rcur, offsetof(struct bpf_reg_state, id)) == 0 && | 494 | return memcmp(rold, rcur, offsetof(struct bpf_reg_state, id)) == 0 && |
| 496 | check_ids(rold->id, rcur->id, idmap) && | 495 | check_ids(rold->id, rcur->id, idmap) && |
| 497 | check_ids(rold->ref_obj_id, rcur->ref_obj_id, idmap); | 496 | check_ids(rold->parent_id, rcur->parent_id, idmap); |
| 498 | } | 497 | } |
| 499 | 498 | ||
| 500 | enum exact_level { | 499 | enum exact_level { |
| @@ -619,7 +618,7 @@ static bool regsafe(struct bpf_verifier_env *env, struct bpf_reg_state *rold, · 118 unmodified lines | |||
| 619 | range_within(rold, rcur) && | 618 | range_within(rold, rcur) && |
| 620 | tnum_in(rold->var_off, rcur->var_off) && | 619 | tnum_in(rold->var_off, rcur->var_off) && |
| 621 | check_ids(rold->id, rcur->id, idmap) && | 620 | check_ids(rold->id, rcur->id, idmap) && |
| 622 | check_ids(rold->ref_obj_id, rcur->ref_obj_id, idmap); | 621 | check_ids(rold->parent_id, rcur->parent_id, idmap); |
| 623 | case PTR_TO_PACKET_META: | 622 | case PTR_TO_PACKET_META: |
| 624 | case PTR_TO_PACKET: | 623 | case PTR_TO_PACKET: |
| 625 | /* We must have at least as much range as the old ptr | 624 | /* We must have at least as much range as the old ptr |
| @@ -799,7 +798,8 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old, · 173 unmodified lines | |||
| 799 | cur_reg = &cur->stack[spi].spilled_ptr; | 798 | cur_reg = &cur->stack[spi].spilled_ptr; |
| 800 | if (old_reg->dynptr.type != cur_reg->dynptr.type || | 799 | if (old_reg->dynptr.type != cur_reg->dynptr.type || |
| 801 | old_reg->dynptr.first_slot != cur_reg->dynptr.first_slot || | 800 | old_reg->dynptr.first_slot != cur_reg->dynptr.first_slot || |
| 802 | !check_ids(old_reg->ref_obj_id, cur_reg->ref_obj_id, idmap)) | 801 | !check_ids(old_reg->id, cur_reg->id, idmap) || |
| 802 | !check_ids(old_reg->parent_id, cur_reg->parent_id, idmap)) | ||
| 803 | return false; | 803 | return false; |
| 804 | break; | 804 | break; |
| 805 | case STACK_ITER: | 805 | case STACK_ITER: |
| @@ -815,13 +815,13 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old, · 9 unmodified lines | |||
| 815 | old_reg->iter.btf_id != cur_reg->iter.btf_id || | 815 | old_reg->iter.btf_id != cur_reg->iter.btf_id || |
| 816 | old_reg->iter.state != cur_reg->iter.state || | 816 | old_reg->iter.state != cur_reg->iter.state || |
| 817 | /* ignore {old_reg,cur_reg}->iter.depth, see above */ | 817 | /* ignore {old_reg,cur_reg}->iter.depth, see above */ |
| 818 | !check_ids(old_reg->ref_obj_id, cur_reg->ref_obj_id, idmap)) | 818 | !check_ids(old_reg->id, cur_reg->id, idmap)) |
| 819 | return false; | 819 | return false; |
| 820 | break; | 820 | break; |
| 821 | case STACK_IRQ_FLAG: | 821 | case STACK_IRQ_FLAG: |
| 822 | old_reg = &old->stack[spi].spilled_ptr; | 822 | old_reg = &old->stack[spi].spilled_ptr; |
| 823 | cur_reg = &cur->stack[spi].spilled_ptr; | 823 | cur_reg = &cur->stack[spi].spilled_ptr; |
| 824 | if (!check_ids(old_reg->ref_obj_id, cur_reg->ref_obj_id, idmap) || | 824 | if (!check_ids(old_reg->id, cur_reg->id, idmap) || |
| 825 | old_reg->irq.kfunc_class != cur_reg->irq.kfunc_class) | 825 | old_reg->irq.kfunc_class != cur_reg->irq.kfunc_class) |
| 826 | return false; | 826 | return false; |
| 827 | break; | 827 | break; |
| @@ -838,6 +838,32 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old, · 10 unmodified lines | |||
| 838 | return true; | 838 | return true; |
| 839 | } | 839 | } |
| 840 | 840 | ||
| 841 | /* | ||
| 842 | * Compare stack arg slots between old and current states. | ||
| 843 | * Outgoing stack args are path-local state and must agree for pruning. | ||
| 844 | */ | ||
| 845 | static bool stack_arg_safe(struct bpf_verifier_env *env, struct bpf_func_state *old, | ||
| 846 | struct bpf_func_state *cur, struct bpf_idmap *idmap, | ||
| 847 | enum exact_level exact) | ||
| 848 | { | ||
| 849 | int i, nslots; | ||
| 850 | |||
| 851 | nslots = max(old->out_stack_arg_cnt, cur->out_stack_arg_cnt); | ||
| 852 | for (i = 0; i < nslots; i++) { | ||
| 853 | struct bpf_reg_state *old_arg, *cur_arg; | ||
| 854 | struct bpf_reg_state not_init = { .type = NOT_INIT }; | ||
| 855 | |||
| 856 | old_arg = i < old->out_stack_arg_cnt ? | ||
| 857 | &old->stack_arg_regs[i] : ¬_init; | ||
| 858 | cur_arg = i < cur->out_stack_arg_cnt ? | ||
| 859 | &cur->stack_arg_regs[i] : ¬_init; | ||
| 860 | if (!regsafe(env, old_arg, cur_arg, idmap, exact)) | ||
| 861 | return false; | ||
| 862 | } | ||
| 863 | |||
| 864 | return true; | ||
| 865 | } | ||
| 866 | |||
| 841 | static bool refsafe(struct bpf_verifier_state *old, struct bpf_verifier_state *cur, | 867 | static bool refsafe(struct bpf_verifier_state *old, struct bpf_verifier_state *cur, |
| 842 | struct bpf_idmap *idmap) | 868 | struct bpf_idmap *idmap) |
| 843 | { | 869 | { |
| @@ -868,6 +894,9 @@ static bool refsafe(struct bpf_verifier_state *old, struct bpf_verifier_state *c · 24 unmodified lines | |||
| 868 | return false; | 894 | return false; |
| 869 | switch (old->refs[i].type) { | 895 | switch (old->refs[i].type) { |
| 870 | case REF_TYPE_PTR: | 896 | case REF_TYPE_PTR: |
| 897 | if (!check_ids(old->refs[i].parent_id, cur->refs[i].parent_id, idmap)) | ||
| 898 | return false; | ||
| 899 | break; | ||
| 871 | case REF_TYPE_IRQ: | 900 | case REF_TYPE_IRQ: |
| 872 | break; | 901 | break; |
| 873 | case REF_TYPE_LOCK: | 902 | case REF_TYPE_LOCK: |
| @@ -920,6 +949,9 @@ static bool func_states_equal(struct bpf_verifier_env *env, struct bpf_func_stat · 46 unmodified lines | |||
| 920 | if (old->callback_depth > cur->callback_depth) | 949 | if (old->callback_depth > cur->callback_depth) |
| 921 | return false; | 950 | return false; |
| 922 | 951 | ||
| 952 | if (!old->no_stack_arg_load && cur->no_stack_arg_load) | ||
| 953 | return false; | ||
| 954 | |||
| 923 | for (i = 0; i < MAX_BPF_REG; i++) | 955 | for (i = 0; i < MAX_BPF_REG; i++) |
| 924 | if (((1 << i) & live_regs) && | 956 | if (((1 << i) & live_regs) && |
| 925 | !regsafe(env, &old->regs[i], &cur->regs[i], | 957 | !regsafe(env, &old->regs[i], &cur->regs[i], |
| @@ -929,6 +961,9 @@ static bool func_states_equal(struct bpf_verifier_env *env, struct bpf_func_stat · 3 unmodified lines | |||
| 929 | if (!stacksafe(env, old, cur, &env->idmap_scratch, exact)) | 961 | if (!stacksafe(env, old, cur, &env->idmap_scratch, exact)) |
| 930 | return false; | 962 | return false; |
| 931 | 963 | ||
| 964 | if (!stack_arg_safe(env, old, cur, &env->idmap_scratch, exact)) | ||
| 965 | return false; | ||
| 966 | |||
| 932 | return true; | 967 | return true; |
| 933 | } | 968 | } |
| 934 | 969 | ||
| @@ -1376,7 +1411,7 @@ hit: · 441 unmodified lines | |||
| 1376 | */ | 1411 | */ |
| 1377 | err = 0; | 1412 | err = 0; |
| 1378 | if (bpf_is_jmp_point(env, env->insn_idx)) | 1413 | if (bpf_is_jmp_point(env, env->insn_idx)) |
| 1379 | err = bpf_push_jmp_history(env, cur, 0, 0); | 1414 | err = bpf_push_jmp_history(env, cur, 0, 0, 0, 0); |
| 1380 | err = err ? : propagate_precision(env, &sl->state, cur, NULL); | 1415 | err = err ? : propagate_precision(env, &sl->state, cur, NULL); |
| 1381 | if (err) | 1416 | if (err) |
| 1382 | return err; | 1417 | return err; |
| @@ -41,6 +41,7 @@ · 40 unmodified lines | |||
| 41 | #include <linux/overflow.h> | 41 | #include <linux/overflow.h> |
| 42 | #include <linux/cookie.h> | 42 | #include <linux/cookie.h> |
| 43 | #include <linux/verification.h> | 43 | #include <linux/verification.h> |
| 44 | #include <linux/btf_ids.h> | ||
| 44 | 45 | ||
| 45 | #include <net/netfilter/nf_bpf_link.h> | 46 | #include <net/netfilter/nf_bpf_link.h> |
| 46 | #include <net/netkit.h> | 47 | #include <net/netkit.h> |
| @@ -807,6 +808,11 @@ void bpf_obj_free_task_work(const struct btf_record *rec, void *obj) · 760 unmodified lines | |||
| 807 | bpf_task_work_cancel_and_free(obj + rec->task_work_off); | 808 | bpf_task_work_cancel_and_free(obj + rec->task_work_off); |
| 808 | } | 809 | } |
| 809 | 810 | ||
| 811 | void bpf_obj_cancel_fields(struct bpf_map *map, void *obj) | ||
| 812 | { | ||
| 813 | bpf_map_free_internal_structs(map, obj); | ||
| 814 | } | ||
| 815 | |||
| 810 | void bpf_obj_free_fields(const struct btf_record *rec, void *obj) | 816 | void bpf_obj_free_fields(const struct btf_record *rec, void *obj) |
| 811 | { | 817 | { |
| 812 | const struct btf_field *fields; | 818 | const struct btf_field *fields; |
| @@ -1280,6 +1286,7 @@ static int map_check_btf(struct bpf_map *map, struct bpf_token *token, · 467 unmodified lines | |||
| 1280 | case BPF_SPIN_LOCK: | 1286 | case BPF_SPIN_LOCK: |
| 1281 | case BPF_RES_SPIN_LOCK: | 1287 | case BPF_RES_SPIN_LOCK: |
| 1282 | if (map->map_type != BPF_MAP_TYPE_HASH && | 1288 | if (map->map_type != BPF_MAP_TYPE_HASH && |
| 1289 | map->map_type != BPF_MAP_TYPE_RHASH && | ||
| 1283 | map->map_type != BPF_MAP_TYPE_ARRAY && | 1290 | map->map_type != BPF_MAP_TYPE_ARRAY && |
| 1284 | map->map_type != BPF_MAP_TYPE_CGROUP_STORAGE && | 1291 | map->map_type != BPF_MAP_TYPE_CGROUP_STORAGE && |
| 1285 | map->map_type != BPF_MAP_TYPE_SK_STORAGE && | 1292 | map->map_type != BPF_MAP_TYPE_SK_STORAGE && |
| @@ -1294,6 +1301,7 @@ static int map_check_btf(struct bpf_map *map, struct bpf_token *token, · 8 unmodified lines | |||
| 1294 | case BPF_WORKQUEUE: | 1301 | case BPF_WORKQUEUE: |
| 1295 | case BPF_TASK_WORK: | 1302 | case BPF_TASK_WORK: |
| 1296 | if (map->map_type != BPF_MAP_TYPE_HASH && | 1303 | if (map->map_type != BPF_MAP_TYPE_HASH && |
| 1304 | map->map_type != BPF_MAP_TYPE_RHASH && | ||
| 1297 | map->map_type != BPF_MAP_TYPE_LRU_HASH && | 1305 | map->map_type != BPF_MAP_TYPE_LRU_HASH && |
| 1298 | map->map_type != BPF_MAP_TYPE_ARRAY) { | 1306 | map->map_type != BPF_MAP_TYPE_ARRAY) { |
| 1299 | ret = -EOPNOTSUPP; | 1307 | ret = -EOPNOTSUPP; |
| @@ -1305,6 +1313,7 @@ static int map_check_btf(struct bpf_map *map, struct bpf_token *token, · 5 unmodified lines | |||
| 1305 | case BPF_KPTR_PERCPU: | 1313 | case BPF_KPTR_PERCPU: |
| 1306 | case BPF_REFCOUNT: | 1314 | case BPF_REFCOUNT: |
| 1307 | if (map->map_type != BPF_MAP_TYPE_HASH && | 1315 | if (map->map_type != BPF_MAP_TYPE_HASH && |
| 1316 | map->map_type != BPF_MAP_TYPE_RHASH && | ||
| 1308 | map->map_type != BPF_MAP_TYPE_PERCPU_HASH && | 1317 | map->map_type != BPF_MAP_TYPE_PERCPU_HASH && |
| 1309 | map->map_type != BPF_MAP_TYPE_LRU_HASH && | 1318 | map->map_type != BPF_MAP_TYPE_LRU_HASH && |
| 1310 | map->map_type != BPF_MAP_TYPE_LRU_PERCPU_HASH && | 1319 | map->map_type != BPF_MAP_TYPE_LRU_PERCPU_HASH && |
| @@ -1359,7 +1368,8 @@ free_map_tab: · 48 unmodified lines | |||
| 1359 | 1368 | ||
| 1360 | #define BPF_MAP_CREATE_LAST_FIELD excl_prog_hash_size | 1369 | #define BPF_MAP_CREATE_LAST_FIELD excl_prog_hash_size |
| 1361 | /* called via syscall */ | 1370 | /* called via syscall */ |
| 1362 | static int map_create(union bpf_attr *attr, bpfptr_t uattr) | 1371 | static int map_create_alloc(union bpf_attr *attr, bpfptr_t uattr, struct bpf_verifier_log *log, |
| 1372 | struct bpf_map **mapp, struct bpf_token **tokenp) | ||
| 1363 | { | 1373 | { |
| 1364 | const struct bpf_map_ops *ops; | 1374 | const struct bpf_map_ops *ops; |
| 1365 | struct bpf_token *token = NULL; | 1375 | struct bpf_token *token = NULL; |
| @@ -1367,12 +1377,13 @@ static int map_create(union bpf_attr *attr, bpfptr_t uattr) · 1 unmodified line | |||
| 1367 | u32 map_type = attr->map_type; | 1377 | u32 map_type = attr->map_type; |
| 1368 | struct bpf_map *map; | 1378 | struct bpf_map *map; |
| 1369 | bool token_flag; | 1379 | bool token_flag; |
| 1370 | int f_flags; | ||
| 1371 | int err; | 1380 | int err; |
| 1372 | 1381 | ||
| 1373 | err = CHECK_ATTR(BPF_MAP_CREATE); | 1382 | err = CHECK_ATTR(BPF_MAP_CREATE); |
| 1374 | if (err) | 1383 | if (err) { |
| 1384 | bpf_log(log, "Invalid attr.\n"); | ||
| 1375 | return -EINVAL; | 1385 | return -EINVAL; |
| 1386 | } | ||
| 1376 | 1387 | ||
| 1377 | /* check BPF_F_TOKEN_FD flag, remember if it's set, and then clear it | 1388 | /* check BPF_F_TOKEN_FD flag, remember if it's set, and then clear it |
| 1378 | * to avoid per-map type checks tripping on unknown flag | 1389 | * to avoid per-map type checks tripping on unknown flag |
| @@ -1381,31 +1392,40 @@ static int map_create(union bpf_attr *attr, bpfptr_t uattr) · 2 unmodified lines | |||
| 1381 | attr->map_flags &= ~BPF_F_TOKEN_FD; | 1392 | attr->map_flags &= ~BPF_F_TOKEN_FD; |
| 1382 | 1393 | ||
| 1383 | if (attr->btf_vmlinux_value_type_id) { | 1394 | if (attr->btf_vmlinux_value_type_id) { |
| 1384 | if (attr->map_type != BPF_MAP_TYPE_STRUCT_OPS || | 1395 | if (attr->map_type != BPF_MAP_TYPE_STRUCT_OPS) { |
| 1385 | attr->btf_key_type_id || attr->btf_value_type_id) | 1396 | bpf_log(log, "btf_vmlinux_value_type_id can only be used with struct_ops maps.\n"); |
| 1386 | return -EINVAL; | 1397 | return -EINVAL; |
| 1398 | } | ||
| 1399 | if (attr->btf_key_type_id || attr->btf_value_type_id) { | ||
| 1400 | bpf_log(log, "btf_vmlinux_value_type_id is mutually exclusive with btf_key_type_id and btf_value_type_id.\n"); | ||
| 1401 | return -EINVAL; | ||
| 1402 | } | ||
| 1387 | } else if (attr->btf_key_type_id && !attr->btf_value_type_id) { | 1403 | } else if (attr->btf_key_type_id && !attr->btf_value_type_id) { |
| 1404 | bpf_log(log, "Invalid btf_value_type_id.\n"); | ||
| 1388 | return -EINVAL; | 1405 | return -EINVAL; |
| 1389 | } | 1406 | } |
| 1390 | 1407 | ||
| 1391 | if (attr->map_type != BPF_MAP_TYPE_BLOOM_FILTER && | 1408 | if (attr->map_type != BPF_MAP_TYPE_BLOOM_FILTER && |
| 1392 | attr->map_type != BPF_MAP_TYPE_ARENA && | 1409 | attr->map_type != BPF_MAP_TYPE_ARENA && |
| 1393 | attr->map_extra != 0) | 1410 | attr->map_type != BPF_MAP_TYPE_RHASH && |
| 1411 | attr->map_extra != 0) { | ||
| 1412 | bpf_log(log, "Invalid map_extra.\n"); | ||
| 1394 | return -EINVAL; | 1413 | return -EINVAL; |
| 1395 | 1414 | } | |
| 1396 | f_flags = bpf_get_file_flag(attr->map_flags); | ||
| 1397 | if (f_flags < 0) | ||
| 1398 | return f_flags; | ||
| 1399 | 1415 | ||
| 1400 | if (numa_node != NUMA_NO_NODE && | 1416 | if (numa_node != NUMA_NO_NODE && |
| 1401 | ((unsigned int)numa_node >= nr_node_ids || | 1417 | ((unsigned int)numa_node >= nr_node_ids || |
| 1402 | !node_online(numa_node))) | 1418 | !node_online(numa_node))) { |
| 1419 | bpf_log(log, "Invalid numa_node.\n"); | ||
| 1403 | return -EINVAL; | 1420 | return -EINVAL; |
| 1421 | } | ||
| 1404 | 1422 | ||
| 1405 | /* find map type and init map: hashtable vs rbtree vs bloom vs ... */ | 1423 | /* find map type and init map: hashtable vs rbtree vs bloom vs ... */ |
| 1406 | map_type = attr->map_type; | 1424 | map_type = attr->map_type; |
| 1407 | if (map_type >= ARRAY_SIZE(bpf_map_types)) | 1425 | if (map_type >= ARRAY_SIZE(bpf_map_types)) { |
| 1426 | bpf_log(log, "Invalid map_type.\n"); | ||
| 1408 | return -EINVAL; | 1427 | return -EINVAL; |
| 1428 | } | ||
| 1409 | map_type = array_index_nospec(map_type, ARRAY_SIZE(bpf_map_types)); | 1429 | map_type = array_index_nospec(map_type, ARRAY_SIZE(bpf_map_types)); |
| 1410 | ops = bpf_map_types[map_type]; | 1430 | ops = bpf_map_types[map_type]; |
| 1411 | if (!ops) | 1431 | if (!ops) |
| @@ -1423,8 +1443,10 @@ static int map_create(union bpf_attr *attr, bpfptr_t uattr) · 11 unmodified lines | |||
| 1423 | 1443 | ||
| 1424 | if (token_flag) { | 1444 | if (token_flag) { |
| 1425 | token = bpf_token_get_from_fd(attr->map_token_fd); | 1445 | token = bpf_token_get_from_fd(attr->map_token_fd); |
| 1426 | if (IS_ERR(token)) | 1446 | if (IS_ERR(token)) { |
| 1447 | bpf_log(log, "Invalid map_token_fd.\n"); | ||
| 1427 | return PTR_ERR(token); | 1448 | return PTR_ERR(token); |
| 1449 | } | ||
| 1428 | 1450 | ||
| 1429 | /* if current token doesn't grant map creation permissions, | 1451 | /* if current token doesn't grant map creation permissions, |
| 1430 | * then we can't use this token, so ignore it and rely on | 1452 | * then we can't use this token, so ignore it and rely on |
| @@ -1457,6 +1479,7 @@ static int map_create(union bpf_attr *attr, bpfptr_t uattr) · 26 unmodified lines | |||
| 1457 | case BPF_MAP_TYPE_CGROUP_ARRAY: | 1479 | case BPF_MAP_TYPE_CGROUP_ARRAY: |
| 1458 | case BPF_MAP_TYPE_ARRAY_OF_MAPS: | 1480 | case BPF_MAP_TYPE_ARRAY_OF_MAPS: |
| 1459 | case BPF_MAP_TYPE_HASH: | 1481 | case BPF_MAP_TYPE_HASH: |
| 1482 | case BPF_MAP_TYPE_RHASH: | ||
| 1460 | case BPF_MAP_TYPE_PERCPU_HASH: | 1483 | case BPF_MAP_TYPE_PERCPU_HASH: |
| 1461 | case BPF_MAP_TYPE_HASH_OF_MAPS: | 1484 | case BPF_MAP_TYPE_HASH_OF_MAPS: |
| 1462 | case BPF_MAP_TYPE_RINGBUF: | 1485 | case BPF_MAP_TYPE_RINGBUF: |
| @@ -1507,8 +1530,10 @@ static int map_create(union bpf_attr *attr, bpfptr_t uattr) · 44 unmodified lines | |||
| 1507 | 1530 | ||
| 1508 | err = bpf_obj_name_cpy(map->name, attr->map_name, | 1531 | err = bpf_obj_name_cpy(map->name, attr->map_name, |
| 1509 | sizeof(attr->map_name)); | 1532 | sizeof(attr->map_name)); |
| 1510 | if (err < 0) | 1533 | if (err < 0) { |
| 1534 | bpf_log(log, "Invalid map_name.\n"); | ||
| 1511 | goto free_map; | 1535 | goto free_map; |
| 1536 | } | ||
| 1512 | 1537 | ||
| 1513 | preempt_disable(); | 1538 | preempt_disable(); |
| 1514 | map->cookie = gen_cookie_next(&bpf_map_cookie); | 1539 | map->cookie = gen_cookie_next(&bpf_map_cookie); |
| @@ -1531,6 +1556,7 @@ static int map_create(union bpf_attr *attr, bpfptr_t uattr) · 16 unmodified lines | |||
| 1531 | 1556 | ||
| 1532 | btf = btf_get_by_fd(attr->btf_fd); | 1557 | btf = btf_get_by_fd(attr->btf_fd); |
| 1533 | if (IS_ERR(btf)) { | 1558 | if (IS_ERR(btf)) { |
| 1559 | bpf_log(log, "Invalid btf_fd.\n"); | ||
| 1534 | err = PTR_ERR(btf); | 1560 | err = PTR_ERR(btf); |
| 1535 | goto free_map; | 1561 | goto free_map; |
| 1536 | } | 1562 | } |
| @@ -1558,6 +1584,7 @@ static int map_create(union bpf_attr *attr, bpfptr_t uattr) · 21 unmodified lines | |||
| 1558 | bpfptr_t uprog_hash = make_bpfptr(attr->excl_prog_hash, uattr.is_kernel); | 1584 | bpfptr_t uprog_hash = make_bpfptr(attr->excl_prog_hash, uattr.is_kernel); |
| 1559 | 1585 | ||
| 1560 | if (attr->excl_prog_hash_size != SHA256_DIGEST_SIZE) { | 1586 | if (attr->excl_prog_hash_size != SHA256_DIGEST_SIZE) { |
| 1587 | bpf_log(log, "Invalid excl_prog_hash_size.\n"); | ||
| 1561 | err = -EINVAL; | 1588 | err = -EINVAL; |
| 1562 | goto free_map; | 1589 | goto free_map; |
| 1563 | } | 1590 | } |
| @@ -1572,11 +1599,62 @@ static int map_create(union bpf_attr *attr, bpfptr_t uattr) · 8 unmodified lines | |||
| 1572 | err = -EFAULT; | 1599 | err = -EFAULT; |
| 1573 | goto free_map; | 1600 | goto free_map; |
| 1574 | } | 1601 | } |
| 1602 | |||
| 1603 | /* See libbpf: emit_signature_match() */ | ||
| 1604 | BUILD_BUG_ON(offsetof(struct bpf_map, excl) != SHA256_DIGEST_SIZE); | ||
| 1605 | BUILD_BUG_ON(!__same_type(map->excl, u32)); | ||
| 1606 | BUILD_BUG_ON(offsetof(struct bpf_map, sha) != 0); | ||
| 1607 | BUILD_BUG_ON(!__same_type(map->sha, u8[SHA256_DIGEST_SIZE])); | ||
| 1608 | map->excl = 1; | ||
| 1575 | } else if (attr->excl_prog_hash_size) { | 1609 | } else if (attr->excl_prog_hash_size) { |
| 1610 | bpf_log(log, "Invalid excl_prog_hash_size.\n"); | ||
| 1576 | err = -EINVAL; | 1611 | err = -EINVAL; |
| 1577 | goto free_map; | 1612 | goto free_map; |
| 1578 | } | 1613 | } |
| 1579 | 1614 | ||
| 1615 | *mapp = map; | ||
| 1616 | *tokenp = token; | ||
| 1617 | return 0; | ||
| 1618 | |||
| 1619 | free_map: | ||
| 1620 | bpf_map_free(map); | ||
| 1621 | put_token: | ||
| 1622 | bpf_token_put(token); | ||
| 1623 | return err; | ||
| 1624 | } | ||
| 1625 | |||
| 1626 | static int map_create(union bpf_attr *attr, bpfptr_t uattr, struct bpf_common_attr *attr_common, | ||
| 1627 | bpfptr_t uattr_common, u32 size_common) | ||
| 1628 | { | ||
| 1629 | struct bpf_token *token = NULL; | ||
| 1630 | struct bpf_verifier_log *log; | ||
| 1631 | struct bpf_log_attr attr_log; | ||
| 1632 | struct bpf_map *map = NULL; | ||
| 1633 | int err, ret; | ||
| 1634 | int f_flags; | ||
| 1635 | |||
| 1636 | log = bpf_log_attr_create_vlog(&attr_log, attr_common, uattr_common, size_common); | ||
| 1637 | if (IS_ERR(log)) | ||
| 1638 | return PTR_ERR(log); | ||
| 1639 | |||
| 1640 | err = map_create_alloc(attr, uattr, log, &map, &token); | ||
| 1641 | |||
| 1642 | /* preserve original error even if log finalization is successful */ | ||
| 1643 | ret = bpf_log_attr_finalize(&attr_log, log); | ||
| 1644 | if (ret) | ||
| 1645 | err = ret; | ||
| 1646 | |||
| 1647 | kfree(log); | ||
| 1648 | |||
| 1649 | if (err) | ||
| 1650 | goto free_map; | ||
| 1651 | |||
| 1652 | f_flags = bpf_get_file_flag(attr->map_flags); | ||
| 1653 | if (f_flags < 0) { | ||
| 1654 | err = f_flags; | ||
| 1655 | goto free_map; | ||
| 1656 | } | ||
| 1657 | |||
| 1580 | err = security_bpf_map_create(map, attr, token, uattr.is_kernel); | 1658 | err = security_bpf_map_create(map, attr, token, uattr.is_kernel); |
| 1581 | if (err) | 1659 | if (err) |
| 1582 | goto free_map_sec; | 1660 | goto free_map_sec; |
| @@ -1605,8 +1683,8 @@ static int map_create(union bpf_attr *attr, bpfptr_t uattr) · 22 unmodified lines | |||
| 1605 | free_map_sec: | 1683 | free_map_sec: |
| 1606 | security_bpf_map_free(map); | 1684 | security_bpf_map_free(map); |
| 1607 | free_map: | 1685 | free_map: |
| 1608 | bpf_map_free(map); | 1686 | if (map) |
| 1609 | put_token: | 1687 | bpf_map_free(map); |
| 1610 | bpf_token_put(token); | 1688 | bpf_token_put(token); |
| 1611 | return err; | 1689 | return err; |
| 1612 | } | 1690 | } |
| @@ -2192,6 +2270,7 @@ static int map_lookup_and_delete_elem(union bpf_attr *attr) · 579 unmodified lines | |||
| 2192 | map->map_type == BPF_MAP_TYPE_PERCPU_HASH || | 2270 | map->map_type == BPF_MAP_TYPE_PERCPU_HASH || |
| 2193 | map->map_type == BPF_MAP_TYPE_LRU_HASH || | 2271 | map->map_type == BPF_MAP_TYPE_LRU_HASH || |
| 2194 | map->map_type == BPF_MAP_TYPE_LRU_PERCPU_HASH || | 2272 | map->map_type == BPF_MAP_TYPE_LRU_PERCPU_HASH || |
| 2273 | map->map_type == BPF_MAP_TYPE_RHASH || | ||
| 2195 | map->map_type == BPF_MAP_TYPE_STACK_TRACE) { | 2274 | map->map_type == BPF_MAP_TYPE_STACK_TRACE) { |
| 2196 | if (!bpf_map_is_offloaded(map)) { | 2275 | if (!bpf_map_is_offloaded(map)) { |
| 2197 | bpf_disable_instrumentation(); | 2276 | bpf_disable_instrumentation(); |
| @@ -2646,7 +2725,8 @@ static int · 448 unmodified lines | |||
| 2646 | bpf_prog_load_check_attach(enum bpf_prog_type prog_type, | 2725 | bpf_prog_load_check_attach(enum bpf_prog_type prog_type, |
| 2647 | enum bpf_attach_type expected_attach_type, | 2726 | enum bpf_attach_type expected_attach_type, |
| 2648 | struct btf *attach_btf, u32 btf_id, | 2727 | struct btf *attach_btf, u32 btf_id, |
| 2649 | struct bpf_prog *dst_prog) | 2728 | struct bpf_prog *dst_prog, |
| 2729 | bool multi_func) | ||
| 2650 | { | 2730 | { |
| 2651 | if (btf_id) { | 2731 | if (btf_id) { |
| 2652 | if (btf_id > BTF_MAX_TYPE) | 2732 | if (btf_id > BTF_MAX_TYPE) |
| @@ -2666,6 +2746,14 @@ bpf_prog_load_check_attach(enum bpf_prog_type prog_type, · 13 unmodified lines | |||
| 2666 | } | 2746 | } |
| 2667 | } | 2747 | } |
| 2668 | 2748 | ||
| 2749 | if (multi_func) { | ||
| 2750 | if (prog_type != BPF_PROG_TYPE_TRACING) | ||
| 2751 | return -EINVAL; | ||
| 2752 | if (!attach_btf || btf_id) | ||
| 2753 | return -EINVAL; | ||
| 2754 | return 0; | ||
| 2755 | } | ||
| 2756 | |||
| 2669 | if (attach_btf && (!btf_id || dst_prog)) | 2757 | if (attach_btf && (!btf_id || dst_prog)) |
| 2670 | return -EINVAL; | 2758 | return -EINVAL; |
| 2671 | 2759 | ||
| @@ -2798,8 +2886,22 @@ static bool is_perfmon_prog_type(enum bpf_prog_type prog_type) · 126 unmodified lines | |||
| 2798 | } | 2886 | } |
| 2799 | } | 2887 | } |
| 2800 | 2888 | ||
| 2889 | static enum bpf_sig_keyring bpf_classify_keyring(s32 keyring_id) | ||
| 2890 | { | ||
| 2891 | switch (keyring_id) { | ||
| 2892 | case 0: | ||
| 2893 | return BPF_SIG_KEYRING_BUILTIN; | ||
| 2894 | case (s32)(unsigned long)VERIFY_USE_SECONDARY_KEYRING: | ||
| 2895 | return BPF_SIG_KEYRING_SECONDARY; | ||
| 2896 | case (s32)(unsigned long)VERIFY_USE_PLATFORM_KEYRING: | ||
| 2897 | return BPF_SIG_KEYRING_PLATFORM; | ||
| 2898 | default: | ||
| 2899 | return BPF_SIG_KEYRING_USER; | ||
| 2900 | } | ||
| 2901 | } | ||
| 2902 | |||
| 2801 | static int bpf_prog_verify_signature(struct bpf_prog *prog, union bpf_attr *attr, | 2903 | static int bpf_prog_verify_signature(struct bpf_prog *prog, union bpf_attr *attr, |
| 2802 | bool is_kernel) | 2904 | bool is_kernel, s32 *keyring_serial) |
| 2803 | { | 2905 | { |
| 2804 | bpfptr_t usig = make_bpfptr(attr->signature, is_kernel); | 2906 | bpfptr_t usig = make_bpfptr(attr->signature, is_kernel); |
| 2805 | struct bpf_dynptr_kern sig_ptr, insns_ptr; | 2907 | struct bpf_dynptr_kern sig_ptr, insns_ptr; |
| @@ -2835,7 +2937,8 @@ static int bpf_prog_verify_signature(struct bpf_prog *prog, union bpf_attr *attr · 29 unmodified lines | |||
| 2835 | 2937 | ||
| 2836 | err = bpf_verify_pkcs7_signature((struct bpf_dynptr *)&insns_ptr, | 2938 | err = bpf_verify_pkcs7_signature((struct bpf_dynptr *)&insns_ptr, |
| 2837 | (struct bpf_dynptr *)&sig_ptr, key); | 2939 | (struct bpf_dynptr *)&sig_ptr, key); |
| 2838 | 2940 | if (!err) | |
| 2941 | *keyring_serial = bpf_key_serial(key); | ||
| 2839 | bpf_key_put(key); | 2942 | bpf_key_put(key); |
| 2840 | kvfree(sig); | 2943 | kvfree(sig); |
| 2841 | return err; | 2944 | return err; |
| @@ -2858,10 +2961,15 @@ static int bpf_prog_mark_insn_arrays_ready(struct bpf_prog *prog) · 16 unmodified lines | |||
| 2858 | return 0; | 2961 | return 0; |
| 2859 | } | 2962 | } |
| 2860 | 2963 | ||
| 2964 | extern int bpf_multi_func(void); | ||
| 2965 | int __init __used bpf_multi_func(void) { return 0; } | ||
| 2966 | |||
| 2967 | BTF_ID_LIST_GLOBAL_SINGLE(bpf_multi_func_btf_id, func, bpf_multi_func) | ||
| 2968 | |||
| 2861 | /* last field in 'union bpf_attr' used by this command */ | 2969 | /* last field in 'union bpf_attr' used by this command */ |
| 2862 | #define BPF_PROG_LOAD_LAST_FIELD keyring_id | 2970 | #define BPF_PROG_LOAD_LAST_FIELD keyring_id |
| 2863 | 2971 | ||
| 2864 | static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size) | 2972 | static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, struct bpf_log_attr *attr_log) |
| 2865 | { | 2973 | { |
| 2866 | enum bpf_prog_type type = attr->prog_type; | 2974 | enum bpf_prog_type type = attr->prog_type; |
| 2867 | struct bpf_prog *prog, *dst_prog = NULL; | 2975 | struct bpf_prog *prog, *dst_prog = NULL; |
| @@ -2870,6 +2978,7 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size) · 2 unmodified lines | |||
| 2870 | bool bpf_cap; | 2978 | bool bpf_cap; |
| 2871 | int err; | 2979 | int err; |
| 2872 | char license[128]; | 2980 | char license[128]; |
| 2981 | bool multi_func; | ||
| 2873 | 2982 | ||
| 2874 | if (CHECK_ATTR(BPF_PROG_LOAD)) | 2983 | if (CHECK_ATTR(BPF_PROG_LOAD)) |
| 2875 | return -EINVAL; | 2984 | return -EINVAL; |
| @@ -2936,6 +3045,8 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size) · 60 unmodified lines | |||
| 2936 | if (is_perfmon_prog_type(type) && !bpf_token_capable(token, CAP_PERFMON)) | 3045 | if (is_perfmon_prog_type(type) && !bpf_token_capable(token, CAP_PERFMON)) |
| 2937 | goto put_token; | 3046 | goto put_token; |
| 2938 | 3047 | ||
| 3048 | multi_func = is_tracing_multi(attr->expected_attach_type); | ||
| 3049 | |||
| 2939 | /* attach_prog_fd/attach_btf_obj_fd can specify fd of either bpf_prog | 3050 | /* attach_prog_fd/attach_btf_obj_fd can specify fd of either bpf_prog |
| 2940 | * or btf, we need to check which one it is | 3051 | * or btf, we need to check which one it is |
| 2941 | */ | 3052 | */ |
| @@ -2957,7 +3068,7 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size) · 15 unmodified lines | |||
| 2957 | goto put_token; | 3068 | goto put_token; |
| 2958 | } | 3069 | } |
| 2959 | } | 3070 | } |
| 2960 | } else if (attr->attach_btf_id) { | 3071 | } else if (attr->attach_btf_id || multi_func) { |
| 2961 | /* fall back to vmlinux BTF, if BTF type ID is specified */ | 3072 | /* fall back to vmlinux BTF, if BTF type ID is specified */ |
| 2962 | attach_btf = bpf_get_btf_vmlinux(); | 3073 | attach_btf = bpf_get_btf_vmlinux(); |
| 2963 | if (IS_ERR(attach_btf)) { | 3074 | if (IS_ERR(attach_btf)) { |
| @@ -2973,7 +3084,7 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size) · 9 unmodified lines | |||
| 2973 | 3084 | ||
| 2974 | if (bpf_prog_load_check_attach(type, attr->expected_attach_type, | 3085 | if (bpf_prog_load_check_attach(type, attr->expected_attach_type, |
| 2975 | attach_btf, attr->attach_btf_id, | 3086 | attach_btf, attr->attach_btf_id, |
| 2976 | dst_prog)) { | 3087 | dst_prog, multi_func)) { |
| 2977 | if (dst_prog) | 3088 | if (dst_prog) |
| 2978 | bpf_prog_put(dst_prog); | 3089 | bpf_prog_put(dst_prog); |
| 2979 | if (attach_btf) | 3090 | if (attach_btf) |
| @@ -2996,7 +3107,7 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size) · 16 unmodified lines | |||
| 2996 | prog->expected_attach_type = attr->expected_attach_type; | 3107 | prog->expected_attach_type = attr->expected_attach_type; |
| 2997 | prog->sleepable = !!(attr->prog_flags & BPF_F_SLEEPABLE); | 3108 | prog->sleepable = !!(attr->prog_flags & BPF_F_SLEEPABLE); |
| 2998 | prog->aux->attach_btf = attach_btf; | 3109 | prog->aux->attach_btf = attach_btf; |
| 2999 | prog->aux->attach_btf_id = attr->attach_btf_id; | 3110 | prog->aux->attach_btf_id = multi_func ? bpf_multi_func_btf_id[0] : attr->attach_btf_id; |
| 3000 | prog->aux->dst_prog = dst_prog; | 3111 | prog->aux->dst_prog = dst_prog; |
| 3001 | prog->aux->dev_bound = !!attr->prog_ifindex; | 3112 | prog->aux->dev_bound = !!attr->prog_ifindex; |
| 3002 | prog->aux->xdp_has_frags = attr->prog_flags & BPF_F_XDP_HAS_FRAGS; | 3113 | prog->aux->xdp_has_frags = attr->prog_flags & BPF_F_XDP_HAS_FRAGS; |
| @@ -3022,13 +3133,17 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size) · 19 unmodified lines | |||
| 3022 | 3133 | ||
| 3023 | /* eBPF programs must be GPL compatible to use GPL-ed functions */ | 3134 | /* eBPF programs must be GPL compatible to use GPL-ed functions */ |
| 3024 | prog->gpl_compatible = license_is_gpl_compatible(license) ? 1 : 0; | 3135 | prog->gpl_compatible = license_is_gpl_compatible(license) ? 1 : 0; |
| 3025 | |||
| 3026 | if (attr->signature) { | 3136 | if (attr->signature) { |
| 3027 | err = bpf_prog_verify_signature(prog, attr, uattr.is_kernel); | 3137 | err = bpf_prog_verify_signature(prog, attr, uattr.is_kernel, |
| 3138 | &prog->aux->sig.keyring_serial); | ||
| 3028 | if (err) | 3139 | if (err) |
| 3029 | goto free_prog; | 3140 | goto free_prog; |
| 3141 | prog->aux->sig.keyring_type = bpf_classify_keyring(attr->keyring_id); | ||
| 3142 | prog->aux->sig.verdict = BPF_SIG_VERIFIED; | ||
| 3143 | } else { | ||
| 3144 | prog->aux->sig.keyring_type = BPF_SIG_KEYRING_NONE; | ||
| 3145 | prog->aux->sig.verdict = BPF_SIG_UNSIGNED; | ||
| 3030 | } | 3146 | } |
| 3031 | |||
| 3032 | prog->orig_prog = NULL; | 3147 | prog->orig_prog = NULL; |
| 3033 | prog->jited = 0; | 3148 | prog->jited = 0; |
| 3034 | 3149 | ||
| @@ -3076,10 +3191,10 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size) · 41 unmodified lines | |||
| 3076 | 3191 | ||
| 3077 | err = security_bpf_prog_load(prog, attr, token, uattr.is_kernel); | 3192 | err = security_bpf_prog_load(prog, attr, token, uattr.is_kernel); |
| 3078 | if (err) | 3193 | if (err) |
| 3079 | goto free_prog_sec; | 3194 | goto free_prog; |
| 3080 | 3195 | ||
| 3081 | /* run eBPF verifier */ | 3196 | /* run eBPF verifier */ |
| 3082 | err = bpf_check(&prog, attr, uattr, uattr_size); | 3197 | err = bpf_check(&prog, attr, uattr, attr_log); |
| 3083 | if (err < 0) | 3198 | if (err < 0) |
| 3084 | goto free_used_maps; | 3199 | goto free_used_maps; |
| 3085 | 3200 | ||
| @@ -3122,8 +3237,6 @@ free_used_maps: · 36 unmodified lines | |||
| 3122 | __bpf_prog_put_noref(prog, prog->aux->real_func_cnt); | 3237 | __bpf_prog_put_noref(prog, prog->aux->real_func_cnt); |
| 3123 | return err; | 3238 | return err; |
| 3124 | 3239 | ||
| 3125 | free_prog_sec: | ||
| 3126 | security_bpf_prog_free(prog); | ||
| 3127 | free_prog: | 3240 | free_prog: |
| 3128 | free_uid(prog->aux->user); | 3241 | free_uid(prog->aux->user); |
| 3129 | if (prog->aux->attach_btf) | 3242 | if (prog->aux->attach_btf) |
| @@ -3198,6 +3311,15 @@ void bpf_link_init(struct bpf_link *link, enum bpf_link_type type, · 68 unmodified lines | |||
| 3198 | bpf_link_init_sleepable(link, type, ops, prog, attach_type, false); | 3311 | bpf_link_init_sleepable(link, type, ops, prog, attach_type, false); |
| 3199 | } | 3312 | } |
| 3200 | 3313 | ||
| 3314 | void bpf_tramp_link_init(struct bpf_tramp_link *link, enum bpf_link_type type, | ||
| 3315 | const struct bpf_link_ops *ops, struct bpf_prog *prog, | ||
| 3316 | enum bpf_attach_type attach_type, u64 cookie) | ||
| 3317 | { | ||
| 3318 | bpf_link_init(&link->link, type, ops, prog, attach_type); | ||
| 3319 | link->node.link = &link->link; | ||
| 3320 | link->node.cookie = cookie; | ||
| 3321 | } | ||
| 3322 | |||
| 3201 | static void bpf_link_free_id(int id) | 3323 | static void bpf_link_free_id(int id) |
| 3202 | { | 3324 | { |
| 3203 | if (!id) | 3325 | if (!id) |
| @@ -3358,7 +3480,7 @@ static void bpf_link_show_fdinfo(struct seq_file *m, struct file *filp) · 154 unmodified lines | |||
| 3358 | seq_printf(m, "link_type:\t%s\n", link->flags == BPF_F_KPROBE_MULTI_RETURN ? | 3480 | seq_printf(m, "link_type:\t%s\n", link->flags == BPF_F_KPROBE_MULTI_RETURN ? |
| 3359 | "kretprobe_multi" : "kprobe_multi"); | 3481 | "kretprobe_multi" : "kprobe_multi"); |
| 3360 | else if (link->type == BPF_LINK_TYPE_UPROBE_MULTI) | 3482 | else if (link->type == BPF_LINK_TYPE_UPROBE_MULTI) |
| 3361 | seq_printf(m, "link_type:\t%s\n", link->flags == BPF_F_UPROBE_MULTI_RETURN ? | 3483 | seq_printf(m, "link_type:\t%s\n", link->flags & BPF_F_UPROBE_MULTI_RETURN ? |
| 3362 | "uretprobe_multi" : "uprobe_multi"); | 3484 | "uretprobe_multi" : "uprobe_multi"); |
| 3363 | else | 3485 | else |
| 3364 | seq_printf(m, "link_type:\t%s\n", bpf_link_type_strs[type]); | 3486 | seq_printf(m, "link_type:\t%s\n", bpf_link_type_strs[type]); |
| @@ -3505,7 +3627,7 @@ static void bpf_tracing_link_release(struct bpf_link *link) · 140 unmodified lines | |||
| 3505 | struct bpf_tracing_link *tr_link = | 3627 | struct bpf_tracing_link *tr_link = |
| 3506 | container_of(link, struct bpf_tracing_link, link.link); | 3628 | container_of(link, struct bpf_tracing_link, link.link); |
| 3507 | 3629 | ||
| 3508 | WARN_ON_ONCE(bpf_trampoline_unlink_prog(&tr_link->link, | 3630 | WARN_ON_ONCE(bpf_trampoline_unlink_prog(&tr_link->link.node, |
| 3509 | tr_link->trampoline, | 3631 | tr_link->trampoline, |
| 3510 | tr_link->tgt_prog)); | 3632 | tr_link->tgt_prog)); |
| 3511 | 3633 | ||
| @@ -3518,8 +3640,7 @@ static void bpf_tracing_link_release(struct bpf_link *link) · 6 unmodified lines | |||
| 3518 | 3640 | ||
| 3519 | static void bpf_tracing_link_dealloc(struct bpf_link *link) | 3641 | static void bpf_tracing_link_dealloc(struct bpf_link *link) |
| 3520 | { | 3642 | { |
| 3521 | struct bpf_tracing_link *tr_link = | 3643 | struct bpf_tracing_link *tr_link = container_of(link, struct bpf_tracing_link, link.link); |
| 3522 | container_of(link, struct bpf_tracing_link, link.link); | ||
| 3523 | 3644 | ||
| 3524 | kfree(tr_link); | 3645 | kfree(tr_link); |
| 3525 | } | 3646 | } |
| @@ -3527,8 +3648,8 @@ static void bpf_tracing_link_dealloc(struct bpf_link *link) · 1 unmodified line | |||
| 3527 | static void bpf_tracing_link_show_fdinfo(const struct bpf_link *link, | 3648 | static void bpf_tracing_link_show_fdinfo(const struct bpf_link *link, |
| 3528 | struct seq_file *seq) | 3649 | struct seq_file *seq) |
| 3529 | { | 3650 | { |
| 3530 | struct bpf_tracing_link *tr_link = | 3651 | struct bpf_tracing_link *tr_link = container_of(link, struct bpf_tracing_link, link.link); |
| 3531 | container_of(link, struct bpf_tracing_link, link.link); | 3652 | |
| 3532 | u32 target_btf_id, target_obj_id; | 3653 | u32 target_btf_id, target_obj_id; |
| 3533 | 3654 | ||
| 3534 | bpf_trampoline_unpack_key(tr_link->trampoline->key, | 3655 | bpf_trampoline_unpack_key(tr_link->trampoline->key, |
| @@ -3541,17 +3662,16 @@ static void bpf_tracing_link_show_fdinfo(const struct bpf_link *link, · 6 unmodified lines | |||
| 3541 | link->attach_type, | 3662 | link->attach_type, |
| 3542 | target_obj_id, | 3663 | target_obj_id, |
| 3543 | target_btf_id, | 3664 | target_btf_id, |
| 3544 | tr_link->link.cookie); | 3665 | tr_link->link.node.cookie); |
| 3545 | } | 3666 | } |
| 3546 | 3667 | ||
| 3547 | static int bpf_tracing_link_fill_link_info(const struct bpf_link *link, | 3668 | static int bpf_tracing_link_fill_link_info(const struct bpf_link *link, |
| 3548 | struct bpf_link_info *info) | 3669 | struct bpf_link_info *info) |
| 3549 | { | 3670 | { |
| 3550 | struct bpf_tracing_link *tr_link = | 3671 | struct bpf_tracing_link *tr_link = container_of(link, struct bpf_tracing_link, link.link); |
| 3551 | container_of(link, struct bpf_tracing_link, link.link); | ||
| 3552 | 3672 | ||
| 3553 | info->tracing.attach_type = link->attach_type; | 3673 | info->tracing.attach_type = link->attach_type; |
| 3554 | info->tracing.cookie = tr_link->link.cookie; | 3674 | info->tracing.cookie = tr_link->link.node.cookie; |
| 3555 | bpf_trampoline_unpack_key(tr_link->trampoline->key, | 3675 | bpf_trampoline_unpack_key(tr_link->trampoline->key, |
| 3556 | &info->tracing.target_obj_id, | 3676 | &info->tracing.target_obj_id, |
| 3557 | &info->tracing.target_btf_id); | 3677 | &info->tracing.target_btf_id); |
| @@ -3633,29 +3753,18 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, · 75 unmodified lines | |||
| 3633 | key = bpf_trampoline_compute_key(tgt_prog, NULL, btf_id); | 3753 | key = bpf_trampoline_compute_key(tgt_prog, NULL, btf_id); |
| 3634 | } | 3754 | } |
| 3635 | 3755 | ||
| 3636 | if (prog->expected_attach_type == BPF_TRACE_FSESSION) { | 3756 | link = kzalloc_obj(*link, GFP_USER); |
| 3637 | struct bpf_fsession_link *fslink; | ||
| 3638 | |||
| 3639 | fslink = kzalloc_obj(*fslink, GFP_USER); | ||
| 3640 | if (fslink) { | ||
| 3641 | bpf_link_init(&fslink->fexit.link, BPF_LINK_TYPE_TRACING, | ||
| 3642 | &bpf_tracing_link_lops, prog, attach_type); | ||
| 3643 | fslink->fexit.cookie = bpf_cookie; | ||
| 3644 | link = &fslink->link; | ||
| 3645 | } else { | ||
| 3646 | link = NULL; | ||
| 3647 | } | ||
| 3648 | } else { | ||
| 3649 | link = kzalloc_obj(*link, GFP_USER); | ||
| 3650 | } | ||
| 3651 | if (!link) { | 3757 | if (!link) { |
| 3652 | err = -ENOMEM; | 3758 | err = -ENOMEM; |
| 3653 | goto out_put_prog; | 3759 | goto out_put_prog; |
| 3654 | } | 3760 | } |
| 3655 | bpf_link_init(&link->link.link, BPF_LINK_TYPE_TRACING, | 3761 | bpf_tramp_link_init(&link->link, BPF_LINK_TYPE_TRACING, |
| 3656 | &bpf_tracing_link_lops, prog, attach_type); | 3762 | &bpf_tracing_link_lops, prog, attach_type, bpf_cookie); |
| 3657 | 3763 | ||
| 3658 | link->link.cookie = bpf_cookie; | 3764 | if (prog->expected_attach_type == BPF_TRACE_FSESSION) { |
| 3765 | link->fexit.link = &link->link.link; | ||
| 3766 | link->fexit.cookie = bpf_cookie; | ||
| 3767 | } | ||
| 3659 | 3768 | ||
| 3660 | mutex_lock(&prog->aux->dst_mutex); | 3769 | mutex_lock(&prog->aux->dst_mutex); |
| 3661 | 3770 | ||
| @@ -3758,7 +3867,7 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, · 96 unmodified lines | |||
| 3758 | if (err) | 3867 | if (err) |
| 3759 | goto out_unlock; | 3868 | goto out_unlock; |
| 3760 | 3869 | ||
| 3761 | err = bpf_trampoline_link_prog(&link->link, tr, tgt_prog); | 3870 | err = bpf_trampoline_link_prog(&link->link.node, tr, tgt_prog); |
| 3762 | if (err) { | 3871 | if (err) { |
| 3763 | bpf_link_cleanup(&link_primer); | 3872 | bpf_link_cleanup(&link_primer); |
| 3764 | link = NULL; | 3873 | link = NULL; |
| @@ -4281,6 +4390,11 @@ static int bpf_raw_tp_link_attach(struct bpf_prog *prog, · 516 unmodified lines | |||
| 4281 | if (!btp) | 4390 | if (!btp) |
| 4282 | return -ENOENT; | 4391 | return -ENOENT; |
| 4283 | 4392 | ||
| 4393 | if (prog->sleepable && !tracepoint_is_faultable(btp->tp)) { | ||
| 4394 | bpf_put_raw_tracepoint(btp); | ||
| 4395 | return -EINVAL; | ||
| 4396 | } | ||
| 4397 | |||
| 4284 | link = kzalloc_obj(*link, GFP_USER); | 4398 | link = kzalloc_obj(*link, GFP_USER); |
| 4285 | if (!link) { | 4399 | if (!link) { |
| 4286 | err = -ENOMEM; | 4400 | err = -ENOMEM; |
| @@ -4389,6 +4503,9 @@ attach_type_to_prog_type(enum bpf_attach_type attach_type) · 102 unmodified lines | |||
| 4389 | case BPF_TRACE_FENTRY: | 4503 | case BPF_TRACE_FENTRY: |
| 4390 | case BPF_TRACE_FEXIT: | 4504 | case BPF_TRACE_FEXIT: |
| 4391 | case BPF_TRACE_FSESSION: | 4505 | case BPF_TRACE_FSESSION: |
| 4506 | case BPF_TRACE_FSESSION_MULTI: | ||
| 4507 | case BPF_TRACE_FENTRY_MULTI: | ||
| 4508 | case BPF_TRACE_FEXIT_MULTI: | ||
| 4392 | case BPF_MODIFY_RETURN: | 4509 | case BPF_MODIFY_RETURN: |
| 4393 | return BPF_PROG_TYPE_TRACING; | 4510 | return BPF_PROG_TYPE_TRACING; |
| 4394 | case BPF_LSM_MAC: | 4511 | case BPF_LSM_MAC: |
| @@ -4654,7 +4771,7 @@ static int bpf_prog_detach(const union bpf_attr *attr) · 259 unmodified lines | |||
| 4654 | #define BPF_PROG_QUERY_LAST_FIELD query.revision | 4771 | #define BPF_PROG_QUERY_LAST_FIELD query.revision |
| 4655 | 4772 | ||
| 4656 | static int bpf_prog_query(const union bpf_attr *attr, | 4773 | static int bpf_prog_query(const union bpf_attr *attr, |
| 4657 | union bpf_attr __user *uattr) | 4774 | union bpf_attr __user *uattr, u32 uattr_size) |
| 4658 | { | 4775 | { |
| 4659 | if (!bpf_net_capable()) | 4776 | if (!bpf_net_capable()) |
| 4660 | return -EPERM; | 4777 | return -EPERM; |
| @@ -4693,7 +4810,7 @@ static int bpf_prog_query(const union bpf_attr *attr, · 32 unmodified lines | |||
| 4693 | case BPF_CGROUP_GETSOCKOPT: | 4810 | case BPF_CGROUP_GETSOCKOPT: |
| 4694 | case BPF_CGROUP_SETSOCKOPT: | 4811 | case BPF_CGROUP_SETSOCKOPT: |
| 4695 | case BPF_LSM_CGROUP: | 4812 | case BPF_LSM_CGROUP: |
| 4696 | return cgroup_bpf_prog_query(attr, uattr); | 4813 | return cgroup_bpf_prog_query(attr, uattr, uattr_size); |
| 4697 | case BPF_LIRC_MODE2: | 4814 | case BPF_LIRC_MODE2: |
| 4698 | return lirc_prog_query(attr, uattr); | 4815 | return lirc_prog_query(attr, uattr); |
| 4699 | case BPF_FLOW_DISSECTOR: | 4816 | case BPF_FLOW_DISSECTOR: |
| @@ -5045,10 +5162,11 @@ static int bpf_prog_get_info_by_fd(struct file *file, · 345 unmodified lines | |||
| 5045 | u32 info_len = attr->info.info_len; | 5162 | u32 info_len = attr->info.info_len; |
| 5046 | struct bpf_prog_kstats stats; | 5163 | struct bpf_prog_kstats stats; |
| 5047 | char __user *uinsns; | 5164 | char __user *uinsns; |
| 5048 | u32 ulen; | 5165 | u32 ulen, len; |
| 5049 | int err; | 5166 | int err; |
| 5050 | 5167 | ||
| 5051 | err = bpf_check_uarg_tail_zero(USER_BPFPTR(uinfo), sizeof(info), info_len); | 5168 | len = offsetofend(struct bpf_prog_info, attach_btf_id); |
| 5169 | err = bpf_check_uarg_tail_zero(USER_BPFPTR(uinfo), len, info_len); | ||
| 5052 | if (err) | 5170 | if (err) |
| 5053 | return err; | 5171 | return err; |
| 5054 | info_len = min_t(u32, sizeof(info), info_len); | 5172 | info_len = min_t(u32, sizeof(info), info_len); |
| @@ -5330,10 +5448,11 @@ static int bpf_map_get_info_by_fd(struct file *file, · 275 unmodified lines | |||
| 5330 | { | 5448 | { |
| 5331 | struct bpf_map_info __user *uinfo = u64_to_user_ptr(attr->info.info); | 5449 | struct bpf_map_info __user *uinfo = u64_to_user_ptr(attr->info.info); |
| 5332 | struct bpf_map_info info; | 5450 | struct bpf_map_info info; |
| 5333 | u32 info_len = attr->info.info_len; | 5451 | u32 info_len = attr->info.info_len, len; |
| 5334 | int err; | 5452 | int err; |
| 5335 | 5453 | ||
| 5336 | err = bpf_check_uarg_tail_zero(USER_BPFPTR(uinfo), sizeof(info), info_len); | 5454 | len = offsetofend(struct bpf_map_info, hash_size); |
| 5455 | err = bpf_check_uarg_tail_zero(USER_BPFPTR(uinfo), len, info_len); | ||
| 5337 | if (err) | 5456 | if (err) |
| 5338 | return err; | 5457 | return err; |
| 5339 | info_len = min_t(u32, sizeof(info), info_len); | 5458 | info_len = min_t(u32, sizeof(info), info_len); |
| @@ -5371,18 +5490,16 @@ static int bpf_map_get_info_by_fd(struct file *file, · 31 unmodified lines | |||
| 5371 | 5490 | ||
| 5372 | if (!map->ops->map_get_hash) | 5491 | if (!map->ops->map_get_hash) |
| 5373 | return -EINVAL; | 5492 | return -EINVAL; |
| 5374 | 5493 | if (info.hash_size != sizeof(map->sha)) | |
| 5375 | if (info.hash_size != SHA256_DIGEST_SIZE) | ||
| 5376 | return -EINVAL; | 5494 | return -EINVAL; |
| 5377 | |||
| 5378 | if (!READ_ONCE(map->frozen)) | 5495 | if (!READ_ONCE(map->frozen)) |
| 5379 | return -EPERM; | 5496 | return -EPERM; |
| 5380 | 5497 | ||
| 5381 | err = map->ops->map_get_hash(map, SHA256_DIGEST_SIZE, map->sha); | 5498 | err = map->ops->map_get_hash(map); |
| 5382 | if (err != 0) | 5499 | if (err != 0) |
| 5383 | return err; | 5500 | return err; |
| 5384 | 5501 | ||
| 5385 | if (copy_to_user(uhash, map->sha, SHA256_DIGEST_SIZE) != 0) | 5502 | if (copy_to_user(uhash, map->sha, sizeof(map->sha)) != 0) |
| 5386 | return -EFAULT; | 5503 | return -EFAULT; |
| 5387 | } else if (info.hash_size) { | 5504 | } else if (info.hash_size) { |
| 5388 | return -EINVAL; | 5505 | return -EINVAL; |
| @@ -5495,7 +5612,7 @@ static int bpf_obj_get_info_by_fd(const union bpf_attr *attr, · 106 unmodified lines | |||
| 5495 | 5612 | ||
| 5496 | #define BPF_BTF_LOAD_LAST_FIELD btf_token_fd | 5613 | #define BPF_BTF_LOAD_LAST_FIELD btf_token_fd |
| 5497 | 5614 | ||
| 5498 | static int bpf_btf_load(const union bpf_attr *attr, bpfptr_t uattr, __u32 uattr_size) | 5615 | static int bpf_btf_load(const union bpf_attr *attr, bpfptr_t uattr, struct bpf_log_attr *attr_log) |
| 5499 | { | 5616 | { |
| 5500 | struct bpf_token *token = NULL; | 5617 | struct bpf_token *token = NULL; |
| 5501 | 5618 | ||
| @@ -5522,7 +5639,7 @@ static int bpf_btf_load(const union bpf_attr *attr, bpfptr_t uattr, __u32 uattr_ · 20 unmodified lines | |||
| 5522 | 5639 | ||
| 5523 | bpf_token_put(token); | 5640 | bpf_token_put(token); |
| 5524 | 5641 | ||
| 5525 | return btf_new_fd(attr, uattr, uattr_size); | 5642 | return btf_new_fd(attr, uattr, attr_log); |
| 5526 | } | 5643 | } |
| 5527 | 5644 | ||
| 5528 | #define BPF_BTF_GET_FD_BY_ID_LAST_FIELD fd_by_id_token_fd | 5645 | #define BPF_BTF_GET_FD_BY_ID_LAST_FIELD fd_by_id_token_fd |
| @@ -5723,7 +5840,7 @@ err_put: · 194 unmodified lines | |||
| 5723 | return err; | 5840 | return err; |
| 5724 | } | 5841 | } |
| 5725 | 5842 | ||
| 5726 | #define BPF_LINK_CREATE_LAST_FIELD link_create.uprobe_multi.pid | 5843 | #define BPF_LINK_CREATE_LAST_FIELD link_create.uprobe_multi.path_fd |
| 5727 | static int link_create(union bpf_attr *attr, bpfptr_t uattr) | 5844 | static int link_create(union bpf_attr *attr, bpfptr_t uattr) |
| 5728 | { | 5845 | { |
| 5729 | struct bpf_prog *prog; | 5846 | struct bpf_prog *prog; |
| @@ -5774,6 +5891,8 @@ static int link_create(union bpf_attr *attr, bpfptr_t uattr) · 44 unmodified lines | |||
| 5774 | ret = bpf_iter_link_attach(attr, uattr, prog); | 5891 | ret = bpf_iter_link_attach(attr, uattr, prog); |
| 5775 | else if (prog->expected_attach_type == BPF_LSM_CGROUP) | 5892 | else if (prog->expected_attach_type == BPF_LSM_CGROUP) |
| 5776 | ret = cgroup_bpf_link_attach(attr, prog); | 5893 | ret = cgroup_bpf_link_attach(attr, prog); |
| 5894 | else if (is_tracing_multi(prog->expected_attach_type)) | ||
| 5895 | ret = bpf_tracing_multi_attach(prog, attr); | ||
| 5777 | else | 5896 | else |
| 5778 | ret = bpf_tracing_prog_attach(prog, | 5897 | ret = bpf_tracing_prog_attach(prog, |
| 5779 | attr->link_create.target_fd, | 5898 | attr->link_create.target_fd, |
| @@ -6232,8 +6351,12 @@ put_prog: · 452 unmodified lines | |||
| 6232 | return ret; | 6351 | return ret; |
| 6233 | } | 6352 | } |
| 6234 | 6353 | ||
| 6235 | static int __sys_bpf(enum bpf_cmd cmd, bpfptr_t uattr, unsigned int size) | 6354 | static int __sys_bpf(enum bpf_cmd cmd, bpfptr_t uattr, unsigned int size, |
| 6355 | bpfptr_t uattr_common, unsigned int size_common) | ||
| 6236 | { | 6356 | { |
| 6357 | struct bpf_common_attr attr_common; | ||
| 6358 | u32 offsetof_log_true_size = 0; | ||
| 6359 | struct bpf_log_attr attr_log; | ||
| 6237 | union bpf_attr attr; | 6360 | union bpf_attr attr; |
| 6238 | int err; | 6361 | int err; |
| 6239 | 6362 | ||
| @@ -6247,13 +6370,29 @@ static int __sys_bpf(enum bpf_cmd cmd, bpfptr_t uattr, unsigned int size) · 7 unmodified lines | |||
| 6247 | if (copy_from_bpfptr(&attr, uattr, size) != 0) | 6370 | if (copy_from_bpfptr(&attr, uattr, size) != 0) |
| 6248 | return -EFAULT; | 6371 | return -EFAULT; |
| 6249 | 6372 | ||
| 6373 | memset(&attr_common, 0, sizeof(attr_common)); | ||
| 6374 | if (cmd & BPF_COMMON_ATTRS) { | ||
| 6375 | err = bpf_check_uarg_tail_zero(uattr_common, | ||
| 6376 | offsetofend(struct bpf_common_attr, log_true_size), | ||
| 6377 | size_common); | ||
| 6378 | if (err) | ||
| 6379 | return err; | ||
| 6380 | |||
| 6381 | cmd &= ~BPF_COMMON_ATTRS; | ||
| 6382 | size_common = min_t(u32, size_common, sizeof(attr_common)); | ||
| 6383 | if (copy_from_bpfptr(&attr_common, uattr_common, size_common) != 0) | ||
| 6384 | return -EFAULT; | ||
| 6385 | } else { | ||
| 6386 | size_common = 0; | ||
| 6387 | } | ||
| 6388 | |||
| 6250 | err = security_bpf(cmd, &attr, size, uattr.is_kernel); | 6389 | err = security_bpf(cmd, &attr, size, uattr.is_kernel); |
| 6251 | if (err < 0) | 6390 | if (err < 0) |
| 6252 | return err; | 6391 | return err; |
| 6253 | 6392 | ||
| 6254 | switch (cmd) { | 6393 | switch (cmd) { |
| 6255 | case BPF_MAP_CREATE: | 6394 | case BPF_MAP_CREATE: |
| 6256 | err = map_create(&attr, uattr); | 6395 | err = map_create(&attr, uattr, &attr_common, uattr_common, size_common); |
| 6257 | break; | 6396 | break; |
| 6258 | case BPF_MAP_LOOKUP_ELEM: | 6397 | case BPF_MAP_LOOKUP_ELEM: |
| 6259 | err = map_lookup_elem(&attr); | 6398 | err = map_lookup_elem(&attr); |
| @@ -6271,7 +6410,12 @@ static int __sys_bpf(enum bpf_cmd cmd, bpfptr_t uattr, unsigned int size) · 11 unmodified lines | |||
| 6271 | err = map_freeze(&attr); | 6410 | err = map_freeze(&attr); |
| 6272 | break; | 6411 | break; |
| 6273 | case BPF_PROG_LOAD: | 6412 | case BPF_PROG_LOAD: |
| 6274 | err = bpf_prog_load(&attr, uattr, size); | 6413 | if (size >= offsetofend(union bpf_attr, log_true_size)) |
| 6414 | offsetof_log_true_size = offsetof(union bpf_attr, log_true_size); | ||
| 6415 | err = bpf_log_attr_init(&attr_log, attr.log_buf, attr.log_size, attr.log_level, | ||
| 6416 | offsetof_log_true_size, uattr, &attr_common, uattr_common, | ||
| 6417 | size_common); | ||
| 6418 | err = err ?: bpf_prog_load(&attr, uattr, &attr_log); | ||
| 6275 | break; | 6419 | break; |
| 6276 | case BPF_OBJ_PIN: | 6420 | case BPF_OBJ_PIN: |
| 6277 | err = bpf_obj_pin(&attr); | 6421 | err = bpf_obj_pin(&attr); |
| @@ -6286,7 +6430,7 @@ static int __sys_bpf(enum bpf_cmd cmd, bpfptr_t uattr, unsigned int size) · 8 unmodified lines | |||
| 6286 | err = bpf_prog_detach(&attr); | 6430 | err = bpf_prog_detach(&attr); |
| 6287 | break; | 6431 | break; |
| 6288 | case BPF_PROG_QUERY: | 6432 | case BPF_PROG_QUERY: |
| 6289 | err = bpf_prog_query(&attr, uattr.user); | 6433 | err = bpf_prog_query(&attr, uattr.user, size); |
| 6290 | break; | 6434 | break; |
| 6291 | case BPF_PROG_TEST_RUN: | 6435 | case BPF_PROG_TEST_RUN: |
| 6292 | err = bpf_prog_test_run(&attr, uattr.user); | 6436 | err = bpf_prog_test_run(&attr, uattr.user); |
| @@ -6316,7 +6460,12 @@ static int __sys_bpf(enum bpf_cmd cmd, bpfptr_t uattr, unsigned int size) · 23 unmodified lines | |||
| 6316 | err = bpf_raw_tracepoint_open(&attr); | 6460 | err = bpf_raw_tracepoint_open(&attr); |
| 6317 | break; | 6461 | break; |
| 6318 | case BPF_BTF_LOAD: | 6462 | case BPF_BTF_LOAD: |
| 6319 | err = bpf_btf_load(&attr, uattr, size); | 6463 | if (size >= offsetofend(union bpf_attr, btf_log_true_size)) |
| 6464 | offsetof_log_true_size = offsetof(union bpf_attr, btf_log_true_size); | ||
| 6465 | err = bpf_log_attr_init(&attr_log, attr.btf_log_buf, attr.btf_log_size, | ||
| 6466 | attr.btf_log_level, offsetof_log_true_size, uattr, | ||
| 6467 | &attr_common, uattr_common, size_common); | ||
| 6468 | err = err ?: bpf_btf_load(&attr, uattr, &attr_log); | ||
| 6320 | break; | 6469 | break; |
| 6321 | case BPF_BTF_GET_FD_BY_ID: | 6470 | case BPF_BTF_GET_FD_BY_ID: |
| 6322 | err = bpf_btf_get_fd_by_id(&attr); | 6471 | err = bpf_btf_get_fd_by_id(&attr); |
| @@ -6382,9 +6531,10 @@ static int __sys_bpf(enum bpf_cmd cmd, bpfptr_t uattr, unsigned int size) · 59 unmodified lines | |||
| 6382 | return err; | 6531 | return err; |
| 6383 | } | 6532 | } |
| 6384 | 6533 | ||
| 6385 | SYSCALL_DEFINE3(bpf, int, cmd, union bpf_attr __user *, uattr, unsigned int, size) | 6534 | SYSCALL_DEFINE5(bpf, int, cmd, union bpf_attr __user *, uattr, unsigned int, size, |
| 6535 | struct bpf_common_attr __user *, uattr_common, unsigned int, size_common) | ||
| 6386 | { | 6536 | { |
| 6387 | return __sys_bpf(cmd, USER_BPFPTR(uattr), size); | 6537 | return __sys_bpf(cmd, USER_BPFPTR(uattr), size, USER_BPFPTR(uattr_common), size_common); |
| 6388 | } | 6538 | } |
| 6389 | 6539 | ||
| 6390 | static bool syscall_prog_is_valid_access(int off, int size, | 6540 | static bool syscall_prog_is_valid_access(int off, int size, |
| @@ -6414,7 +6564,7 @@ BPF_CALL_3(bpf_sys_bpf, int, cmd, union bpf_attr *, attr, u32, attr_size) · 23 unmodified lines | |||
| 6414 | default: | 6564 | default: |
| 6415 | return -EINVAL; | 6565 | return -EINVAL; |
| 6416 | } | 6566 | } |
| 6417 | return __sys_bpf(cmd, KERNEL_BPFPTR(attr), attr_size); | 6567 | return __sys_bpf(cmd, KERNEL_BPFPTR(attr), attr_size, KERNEL_BPFPTR(NULL), 0); |
| 6418 | } | 6568 | } |
| 6419 | 6569 | ||
| 6420 | 6570 | ||
| @@ -30,8 +30,46 @@ static struct hlist_head trampoline_ip_table[TRAMPOLINE_TABLE_SIZE]; · 29 unmodified lines | |||
| 30 | /* serializes access to trampoline tables */ | 30 | /* serializes access to trampoline tables */ |
| 31 | static DEFINE_MUTEX(trampoline_mutex); | 31 | static DEFINE_MUTEX(trampoline_mutex); |
| 32 | 32 | ||
| 33 | /* | ||
| 34 | * Keep 32 trampoline locks (5 bits) in the pool so trampoline_lock_all() | ||
| 35 | * stays below MAX_LOCK_DEPTH. Each pool slot has a distinct lockdep | ||
| 36 | * class because trampoline_lock_all() takes all pool mutexes at once; | ||
| 37 | * otherwise lockdep would report recursive locking on same-class mutexes. | ||
| 38 | */ | ||
| 39 | #define TRAMPOLINE_LOCKS_BITS 5 | ||
| 40 | #define TRAMPOLINE_LOCKS_TABLE_SIZE (1 << TRAMPOLINE_LOCKS_BITS) | ||
| 41 | |||
| 42 | static struct { | ||
| 43 | struct mutex mutex; | ||
| 44 | struct lock_class_key key; | ||
| 45 | } trampoline_locks[TRAMPOLINE_LOCKS_TABLE_SIZE]; | ||
| 46 | |||
| 47 | static struct mutex *select_trampoline_lock(struct bpf_trampoline *tr) | ||
| 48 | { | ||
| 49 | return &trampoline_locks[hash_ptr(tr, TRAMPOLINE_LOCKS_BITS)].mutex; | ||
| 50 | } | ||
| 51 | |||
| 52 | static void trampoline_lock(struct bpf_trampoline *tr) | ||
| 53 | { | ||
| 54 | mutex_lock(select_trampoline_lock(tr)); | ||
| 55 | } | ||
| 56 | |||
| 57 | static void trampoline_unlock(struct bpf_trampoline *tr) | ||
| 58 | { | ||
| 59 | mutex_unlock(select_trampoline_lock(tr)); | ||
| 60 | } | ||
| 61 | |||
| 62 | struct bpf_trampoline_ops { | ||
| 63 | int (*register_fentry)(struct bpf_trampoline *tr, struct bpf_tramp_image *im, void *data); | ||
| 64 | int (*unregister_fentry)(struct bpf_trampoline *tr, u32 orig_flags, void *data); | ||
| 65 | int (*modify_fentry)(struct bpf_trampoline *tr, u32 orig_flags, struct bpf_tramp_image *im, | ||
| 66 | bool lock_direct_mutex, void *data); | ||
| 67 | }; | ||
| 68 | |||
| 33 | #ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS | 69 | #ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS |
| 34 | static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mutex); | 70 | static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mutex, |
| 71 | const struct bpf_trampoline_ops *ops, void *data); | ||
| 72 | static const struct bpf_trampoline_ops trampoline_ops; | ||
| 35 | 73 | ||
| 36 | #ifdef CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS | 74 | #ifdef CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS |
| 37 | static struct bpf_trampoline *direct_ops_ip_lookup(struct ftrace_ops *ops, unsigned long ip) | 75 | static struct bpf_trampoline *direct_ops_ip_lookup(struct ftrace_ops *ops, unsigned long ip) |
| @@ -69,9 +107,9 @@ static int bpf_tramp_ftrace_ops_func(struct ftrace_ops *ops, unsigned long ip, · 31 unmodified lines | |||
| 69 | 107 | ||
| 70 | if (cmd == FTRACE_OPS_CMD_ENABLE_SHARE_IPMODIFY_SELF) { | 108 | if (cmd == FTRACE_OPS_CMD_ENABLE_SHARE_IPMODIFY_SELF) { |
| 71 | /* This is called inside register_ftrace_direct_multi(), so | 109 | /* This is called inside register_ftrace_direct_multi(), so |
| 72 | * tr->mutex is already locked. | 110 | * trampoline's mutex is already locked. |
| 73 | */ | 111 | */ |
| 74 | lockdep_assert_held_once(&tr->mutex); | 112 | lockdep_assert_held_once(select_trampoline_lock(tr)); |
| 75 | 113 | ||
| 76 | /* Instead of updating the trampoline here, we propagate | 114 | /* Instead of updating the trampoline here, we propagate |
| 77 | * -EAGAIN to register_ftrace_direct(). Then we can | 115 | * -EAGAIN to register_ftrace_direct(). Then we can |
| @@ -91,7 +129,7 @@ static int bpf_tramp_ftrace_ops_func(struct ftrace_ops *ops, unsigned long ip, · 13 unmodified lines | |||
| 91 | } | 129 | } |
| 92 | 130 | ||
| 93 | /* The normal locking order is | 131 | /* The normal locking order is |
| 94 | * tr->mutex => direct_mutex (ftrace.c) => ftrace_lock (ftrace.c) | 132 | * select_trampoline_lock(tr) => direct_mutex (ftrace.c) => ftrace_lock (ftrace.c) |
| 95 | * | 133 | * |
| 96 | * The following two commands are called from | 134 | * The following two commands are called from |
| 97 | * | 135 | * |
| @@ -99,12 +137,12 @@ static int bpf_tramp_ftrace_ops_func(struct ftrace_ops *ops, unsigned long ip, · 1 unmodified line | |||
| 99 | * cleanup_direct_functions_after_ipmodify | 137 | * cleanup_direct_functions_after_ipmodify |
| 100 | * | 138 | * |
| 101 | * In both cases, direct_mutex is already locked. Use | 139 | * In both cases, direct_mutex is already locked. Use |
| 102 | * mutex_trylock(&tr->mutex) to avoid deadlock in race condition | 140 | * mutex_trylock(select_trampoline_lock(tr)) to avoid deadlock in race condition |
| 103 | * (something else is making changes to this same trampoline). | 141 | * (something else holds the same pool lock). |
| 104 | */ | 142 | */ |
| 105 | if (!mutex_trylock(&tr->mutex)) { | 143 | if (!mutex_trylock(select_trampoline_lock(tr))) { |
| 106 | /* sleep 1 ms to make sure whatever holding tr->mutex makes | 144 | /* sleep 1 ms to make sure whatever holding select_trampoline_lock(tr) |
| 107 | * some progress. | 145 | * makes some progress. |
| 108 | */ | 146 | */ |
| 109 | msleep(1); | 147 | msleep(1); |
| 110 | return -EAGAIN; | 148 | return -EAGAIN; |
| @@ -116,20 +154,22 @@ static int bpf_tramp_ftrace_ops_func(struct ftrace_ops *ops, unsigned long ip, · 5 unmodified lines | |||
| 116 | 154 | ||
| 117 | if ((tr->flags & BPF_TRAMP_F_CALL_ORIG) && | 155 | if ((tr->flags & BPF_TRAMP_F_CALL_ORIG) && |
| 118 | !(tr->flags & BPF_TRAMP_F_ORIG_STACK)) | 156 | !(tr->flags & BPF_TRAMP_F_ORIG_STACK)) |
| 119 | ret = bpf_trampoline_update(tr, false /* lock_direct_mutex */); | 157 | ret = bpf_trampoline_update(tr, false /* lock_direct_mutex */, |
| 158 | &trampoline_ops, NULL); | ||
| 120 | break; | 159 | break; |
| 121 | case FTRACE_OPS_CMD_DISABLE_SHARE_IPMODIFY_PEER: | 160 | case FTRACE_OPS_CMD_DISABLE_SHARE_IPMODIFY_PEER: |
| 122 | tr->flags &= ~BPF_TRAMP_F_SHARE_IPMODIFY; | 161 | tr->flags &= ~BPF_TRAMP_F_SHARE_IPMODIFY; |
| 123 | 162 | ||
| 124 | if (tr->flags & BPF_TRAMP_F_ORIG_STACK) | 163 | if (tr->flags & BPF_TRAMP_F_ORIG_STACK) |
| 125 | ret = bpf_trampoline_update(tr, false /* lock_direct_mutex */); | 164 | ret = bpf_trampoline_update(tr, false /* lock_direct_mutex */, |
| 165 | &trampoline_ops, NULL); | ||
| 126 | break; | 166 | break; |
| 127 | default: | 167 | default: |
| 128 | ret = -EINVAL; | 168 | ret = -EINVAL; |
| 129 | break; | 169 | break; |
| 130 | } | 170 | } |
| 131 | 171 | ||
| 132 | mutex_unlock(&tr->mutex); | 172 | trampoline_unlock(tr); |
| 133 | return ret; | 173 | return ret; |
| 134 | } | 174 | } |
| 135 | #endif | 175 | #endif |
| @@ -142,7 +182,9 @@ bool bpf_prog_has_trampoline(const struct bpf_prog *prog) · 6 unmodified lines | |||
| 142 | switch (ptype) { | 182 | switch (ptype) { |
| 143 | case BPF_PROG_TYPE_TRACING: | 183 | case BPF_PROG_TYPE_TRACING: |
| 144 | if (eatype == BPF_TRACE_FENTRY || eatype == BPF_TRACE_FEXIT || | 184 | if (eatype == BPF_TRACE_FENTRY || eatype == BPF_TRACE_FEXIT || |
| 145 | eatype == BPF_MODIFY_RETURN || eatype == BPF_TRACE_FSESSION) | 185 | eatype == BPF_MODIFY_RETURN || eatype == BPF_TRACE_FSESSION || |
| 186 | eatype == BPF_TRACE_FENTRY_MULTI || eatype == BPF_TRACE_FEXIT_MULTI || | ||
| 187 | eatype == BPF_TRACE_FSESSION_MULTI) | ||
| 146 | return true; | 188 | return true; |
| 147 | return false; | 189 | return false; |
| 148 | case BPF_PROG_TYPE_LSM: | 190 | case BPF_PROG_TYPE_LSM: |
| @@ -359,7 +401,6 @@ static struct bpf_trampoline *bpf_trampoline_lookup(u64 key, unsigned long ip) · 210 unmodified lines | |||
| 359 | head = &trampoline_ip_table[hash_64(tr->ip, TRAMPOLINE_HASH_BITS)]; | 401 | head = &trampoline_ip_table[hash_64(tr->ip, TRAMPOLINE_HASH_BITS)]; |
| 360 | hlist_add_head(&tr->hlist_ip, head); | 402 | hlist_add_head(&tr->hlist_ip, head); |
| 361 | refcount_set(&tr->refcnt, 1); | 403 | refcount_set(&tr->refcnt, 1); |
| 362 | mutex_init(&tr->mutex); | ||
| 363 | for (i = 0; i < BPF_TRAMP_MAX; i++) | 404 | for (i = 0; i < BPF_TRAMP_MAX; i++) |
| 364 | INIT_HLIST_HEAD(&tr->progs_hlist[i]); | 405 | INIT_HLIST_HEAD(&tr->progs_hlist[i]); |
| 365 | out: | 406 | out: |
| @@ -386,9 +427,11 @@ static int bpf_trampoline_update_fentry(struct bpf_trampoline *tr, u32 orig_flag · 20 unmodified lines | |||
| 386 | return bpf_arch_text_poke(ip, old_t, new_t, old_addr, new_addr); | 427 | return bpf_arch_text_poke(ip, old_t, new_t, old_addr, new_addr); |
| 387 | } | 428 | } |
| 388 | 429 | ||
| 389 | static int unregister_fentry(struct bpf_trampoline *tr, u32 orig_flags, | 430 | static void bpf_tramp_image_put(struct bpf_tramp_image *im); |
| 390 | void *old_addr) | 431 | |
| 432 | static int unregister_fentry(struct bpf_trampoline *tr, u32 orig_flags, void *data __maybe_unused) | ||
| 391 | { | 433 | { |
| 434 | void *old_addr = tr->cur_image->image; | ||
| 392 | int ret; | 435 | int ret; |
| 393 | 436 | ||
| 394 | if (tr->func.ftrace_managed) | 437 | if (tr->func.ftrace_managed) |
| @@ -396,13 +439,19 @@ static int unregister_fentry(struct bpf_trampoline *tr, u32 orig_flags, · 1 unmodified line | |||
| 396 | else | 439 | else |
| 397 | ret = bpf_trampoline_update_fentry(tr, orig_flags, old_addr, NULL); | 440 | ret = bpf_trampoline_update_fentry(tr, orig_flags, old_addr, NULL); |
| 398 | 441 | ||
| 399 | return ret; | 442 | if (ret) |
| 443 | return ret; | ||
| 444 | |||
| 445 | bpf_tramp_image_put(tr->cur_image); | ||
| 446 | tr->cur_image = NULL; | ||
| 447 | return 0; | ||
| 400 | } | 448 | } |
| 401 | 449 | ||
| 402 | static int modify_fentry(struct bpf_trampoline *tr, u32 orig_flags, | 450 | static int modify_fentry(struct bpf_trampoline *tr, u32 orig_flags, struct bpf_tramp_image *im, |
| 403 | void *old_addr, void *new_addr, | 451 | bool lock_direct_mutex, void *data __maybe_unused) |
| 404 | bool lock_direct_mutex) | ||
| 405 | { | 452 | { |
| 453 | void *old_addr = tr->cur_image->image; | ||
| 454 | void *new_addr = im->image; | ||
| 406 | int ret; | 455 | int ret; |
| 407 | 456 | ||
| 408 | if (tr->func.ftrace_managed) { | 457 | if (tr->func.ftrace_managed) { |
| @@ -411,12 +460,20 @@ static int modify_fentry(struct bpf_trampoline *tr, u32 orig_flags, · 2 unmodified lines | |||
| 411 | ret = bpf_trampoline_update_fentry(tr, orig_flags, old_addr, | 460 | ret = bpf_trampoline_update_fentry(tr, orig_flags, old_addr, |
| 412 | new_addr); | 461 | new_addr); |
| 413 | } | 462 | } |
| 414 | return ret; | 463 | |
| 464 | if (ret) | ||
| 465 | return ret; | ||
| 466 | |||
| 467 | bpf_tramp_image_put(tr->cur_image); | ||
| 468 | tr->cur_image = im; | ||
| 469 | return 0; | ||
| 415 | } | 470 | } |
| 416 | 471 | ||
| 417 | /* first time registering */ | 472 | /* first time registering */ |
| 418 | static int register_fentry(struct bpf_trampoline *tr, void *new_addr) | 473 | static int register_fentry(struct bpf_trampoline *tr, struct bpf_tramp_image *im, |
| 474 | void *data __maybe_unused) | ||
| 419 | { | 475 | { |
| 476 | void *new_addr = im->image; | ||
| 420 | void *ip = tr->func.addr; | 477 | void *ip = tr->func.addr; |
| 421 | unsigned long faddr; | 478 | unsigned long faddr; |
| 422 | int ret; | 479 | int ret; |
| @@ -434,33 +491,42 @@ static int register_fentry(struct bpf_trampoline *tr, void *new_addr) · 11 unmodified lines | |||
| 434 | ret = bpf_trampoline_update_fentry(tr, 0, NULL, new_addr); | 491 | ret = bpf_trampoline_update_fentry(tr, 0, NULL, new_addr); |
| 435 | } | 492 | } |
| 436 | 493 | ||
| 437 | return ret; | 494 | if (ret) |
| 495 | return ret; | ||
| 496 | |||
| 497 | tr->cur_image = im; | ||
| 498 | return 0; | ||
| 438 | } | 499 | } |
| 439 | 500 | ||
| 440 | static struct bpf_tramp_links * | 501 | static const struct bpf_trampoline_ops trampoline_ops = { |
| 502 | .register_fentry = register_fentry, | ||
| 503 | .unregister_fentry = unregister_fentry, | ||
| 504 | .modify_fentry = modify_fentry, | ||
| 505 | }; | ||
| 506 | |||
| 507 | static struct bpf_tramp_nodes * | ||
| 441 | bpf_trampoline_get_progs(const struct bpf_trampoline *tr, int *total, bool *ip_arg) | 508 | bpf_trampoline_get_progs(const struct bpf_trampoline *tr, int *total, bool *ip_arg) |
| 442 | { | 509 | { |
| 443 | struct bpf_tramp_link *link; | 510 | struct bpf_tramp_node *node, **nodes; |
| 444 | struct bpf_tramp_links *tlinks; | 511 | struct bpf_tramp_nodes *tnodes; |
| 445 | struct bpf_tramp_link **links; | ||
| 446 | int kind; | 512 | int kind; |
| 447 | 513 | ||
| 448 | *total = 0; | 514 | *total = 0; |
| 449 | tlinks = kzalloc_objs(*tlinks, BPF_TRAMP_MAX); | 515 | tnodes = kzalloc_objs(*tnodes, BPF_TRAMP_MAX); |
| 450 | if (!tlinks) | 516 | if (!tnodes) |
| 451 | return ERR_PTR(-ENOMEM); | 517 | return ERR_PTR(-ENOMEM); |
| 452 | 518 | ||
| 453 | for (kind = 0; kind < BPF_TRAMP_MAX; kind++) { | 519 | for (kind = 0; kind < BPF_TRAMP_MAX; kind++) { |
| 454 | tlinks[kind].nr_links = tr->progs_cnt[kind]; | 520 | tnodes[kind].nr_nodes = tr->progs_cnt[kind]; |
| 455 | *total += tr->progs_cnt[kind]; | 521 | *total += tr->progs_cnt[kind]; |
| 456 | links = tlinks[kind].links; | 522 | nodes = tnodes[kind].nodes; |
| 457 | 523 | ||
| 458 | hlist_for_each_entry(link, &tr->progs_hlist[kind], tramp_hlist) { | 524 | hlist_for_each_entry(node, &tr->progs_hlist[kind], tramp_hlist) { |
| 459 | *ip_arg |= link->link.prog->call_get_func_ip; | 525 | *ip_arg |= node->link->prog->call_get_func_ip; |
| 460 | *links++ = link; | 526 | *nodes++ = node; |
| 461 | } | 527 | } |
| 462 | } | 528 | } |
| 463 | return tlinks; | 529 | return tnodes; |
| 464 | } | 530 | } |
| 465 | 531 | ||
| 466 | static void bpf_tramp_image_free(struct bpf_tramp_image *im) | 532 | static void bpf_tramp_image_free(struct bpf_tramp_image *im) |
| @@ -604,30 +670,29 @@ out: · 137 unmodified lines | |||
| 604 | return ERR_PTR(err); | 670 | return ERR_PTR(err); |
| 605 | } | 671 | } |
| 606 | 672 | ||
| 607 | static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mutex) | 673 | static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mutex, |
| 674 | const struct bpf_trampoline_ops *ops, void *data) | ||
| 608 | { | 675 | { |
| 609 | struct bpf_tramp_image *im; | 676 | struct bpf_tramp_image *im; |
| 610 | struct bpf_tramp_links *tlinks; | 677 | struct bpf_tramp_nodes *tnodes; |
| 611 | u32 orig_flags = tr->flags; | 678 | u32 orig_flags = tr->flags; |
| 612 | bool ip_arg = false; | 679 | bool ip_arg = false; |
| 613 | int err, total, size; | 680 | int err, total, size; |
| 614 | 681 | ||
| 615 | tlinks = bpf_trampoline_get_progs(tr, &total, &ip_arg); | 682 | tnodes = bpf_trampoline_get_progs(tr, &total, &ip_arg); |
| 616 | if (IS_ERR(tlinks)) | 683 | if (IS_ERR(tnodes)) |
| 617 | return PTR_ERR(tlinks); | 684 | return PTR_ERR(tnodes); |
| 618 | 685 | ||
| 619 | if (total == 0) { | 686 | if (total == 0) { |
| 620 | err = unregister_fentry(tr, orig_flags, tr->cur_image->image); | 687 | err = ops->unregister_fentry(tr, orig_flags, data); |
| 621 | bpf_tramp_image_put(tr->cur_image); | ||
| 622 | tr->cur_image = NULL; | ||
| 623 | goto out; | 688 | goto out; |
| 624 | } | 689 | } |
| 625 | 690 | ||
| 626 | /* clear all bits except SHARE_IPMODIFY and TAIL_CALL_CTX */ | 691 | /* clear all bits except SHARE_IPMODIFY and TAIL_CALL_CTX */ |
| 627 | tr->flags &= (BPF_TRAMP_F_SHARE_IPMODIFY | BPF_TRAMP_F_TAIL_CALL_CTX); | 692 | tr->flags &= (BPF_TRAMP_F_SHARE_IPMODIFY | BPF_TRAMP_F_TAIL_CALL_CTX); |
| 628 | 693 | ||
| 629 | if (tlinks[BPF_TRAMP_FEXIT].nr_links || | 694 | if (tnodes[BPF_TRAMP_FEXIT].nr_nodes || |
| 630 | tlinks[BPF_TRAMP_MODIFY_RETURN].nr_links) { | 695 | tnodes[BPF_TRAMP_MODIFY_RETURN].nr_nodes) { |
| 631 | /* NOTE: BPF_TRAMP_F_RESTORE_REGS and BPF_TRAMP_F_SKIP_FRAME | 696 | /* NOTE: BPF_TRAMP_F_RESTORE_REGS and BPF_TRAMP_F_SKIP_FRAME |
| 632 | * should not be set together. | 697 | * should not be set together. |
| 633 | */ | 698 | */ |
| @@ -658,7 +723,7 @@ again: · 24 unmodified lines | |||
| 658 | #endif | 723 | #endif |
| 659 | 724 | ||
| 660 | size = arch_bpf_trampoline_size(&tr->func.model, tr->flags, | 725 | size = arch_bpf_trampoline_size(&tr->func.model, tr->flags, |
| 661 | tlinks, tr->func.addr); | 726 | tnodes, tr->func.addr); |
| 662 | if (size < 0) { | 727 | if (size < 0) { |
| 663 | err = size; | 728 | err = size; |
| 664 | goto out; | 729 | goto out; |
| @@ -676,7 +741,7 @@ again: · 11 unmodified lines | |||
| 676 | } | 741 | } |
| 677 | 742 | ||
| 678 | err = arch_prepare_bpf_trampoline(im, im->image, im->image + size, | 743 | err = arch_prepare_bpf_trampoline(im, im->image, im->image + size, |
| 679 | &tr->func.model, tr->flags, tlinks, | 744 | &tr->func.model, tr->flags, tnodes, |
| 680 | tr->func.addr); | 745 | tr->func.addr); |
| 681 | if (err < 0) | 746 | if (err < 0) |
| 682 | goto out_free; | 747 | goto out_free; |
| @@ -685,14 +750,12 @@ again: · 2 unmodified lines | |||
| 685 | if (err) | 750 | if (err) |
| 686 | goto out_free; | 751 | goto out_free; |
| 687 | 752 | ||
| 688 | WARN_ON(tr->cur_image && total == 0); | ||
| 689 | if (tr->cur_image) | 753 | if (tr->cur_image) |
| 690 | /* progs already running at this address */ | 754 | /* progs already running at this address */ |
| 691 | err = modify_fentry(tr, orig_flags, tr->cur_image->image, | 755 | err = ops->modify_fentry(tr, orig_flags, im, lock_direct_mutex, data); |
| 692 | im->image, lock_direct_mutex); | ||
| 693 | else | 756 | else |
| 694 | /* first time registering */ | 757 | /* first time registering */ |
| 695 | err = register_fentry(tr, im->image); | 758 | err = ops->register_fentry(tr, im, data); |
| 696 | 759 | ||
| 697 | #ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS | 760 | #ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS |
| 698 | if (err == -EAGAIN) { | 761 | if (err == -EAGAIN) { |
| @@ -704,34 +767,31 @@ again: · 5 unmodified lines | |||
| 704 | goto again; | 767 | goto again; |
| 705 | } | 768 | } |
| 706 | #endif | 769 | #endif |
| 707 | if (err) | ||
| 708 | goto out_free; | ||
| 709 | 770 | ||
| 710 | if (tr->cur_image) | 771 | out_free: |
| 711 | bpf_tramp_image_put(tr->cur_image); | 772 | if (err) |
| 712 | tr->cur_image = im; | 773 | bpf_tramp_image_free(im); |
| 713 | out: | 774 | out: |
| 714 | /* If any error happens, restore previous flags */ | 775 | /* If any error happens, restore previous flags */ |
| 715 | if (err) | 776 | if (err) |
| 716 | tr->flags = orig_flags; | 777 | tr->flags = orig_flags; |
| 717 | kfree(tlinks); | 778 | kfree(tnodes); |
| 718 | return err; | 779 | return err; |
| 719 | |||
| 720 | out_free: | ||
| 721 | bpf_tramp_image_free(im); | ||
| 722 | goto out; | ||
| 723 | } | 780 | } |
| 724 | 781 | ||
| 725 | static enum bpf_tramp_prog_type bpf_attach_type_to_tramp(struct bpf_prog *prog) | 782 | static enum bpf_tramp_prog_type bpf_attach_type_to_tramp(struct bpf_prog *prog) |
| 726 | { | 783 | { |
| 727 | switch (prog->expected_attach_type) { | 784 | switch (prog->expected_attach_type) { |
| 728 | case BPF_TRACE_FENTRY: | 785 | case BPF_TRACE_FENTRY: |
| 786 | case BPF_TRACE_FENTRY_MULTI: | ||
| 729 | return BPF_TRAMP_FENTRY; | 787 | return BPF_TRAMP_FENTRY; |
| 730 | case BPF_MODIFY_RETURN: | 788 | case BPF_MODIFY_RETURN: |
| 731 | return BPF_TRAMP_MODIFY_RETURN; | 789 | return BPF_TRAMP_MODIFY_RETURN; |
| 732 | case BPF_TRACE_FEXIT: | 790 | case BPF_TRACE_FEXIT: |
| 791 | case BPF_TRACE_FEXIT_MULTI: | ||
| 733 | return BPF_TRAMP_FEXIT; | 792 | return BPF_TRAMP_FEXIT; |
| 734 | case BPF_TRACE_FSESSION: | 793 | case BPF_TRACE_FSESSION: |
| 794 | case BPF_TRACE_FSESSION_MULTI: | ||
| 735 | return BPF_TRAMP_FSESSION; | 795 | return BPF_TRAMP_FSESSION; |
| 736 | case BPF_LSM_MAC: | 796 | case BPF_LSM_MAC: |
| 737 | if (!prog->aux->attach_func_proto->type) | 797 | if (!prog->aux->attach_func_proto->type) |
| @@ -764,39 +824,33 @@ static int bpf_freplace_check_tgt_prog(struct bpf_prog *tgt_prog) · 26 unmodified lines | |||
| 764 | return 0; | 824 | return 0; |
| 765 | } | 825 | } |
| 766 | 826 | ||
| 767 | static int __bpf_trampoline_link_prog(struct bpf_tramp_link *link, | 827 | static struct bpf_tramp_node *fsession_exit(struct bpf_tramp_node *node) |
| 768 | struct bpf_trampoline *tr, | ||
| 769 | struct bpf_prog *tgt_prog) | ||
| 770 | { | 828 | { |
| 771 | struct bpf_fsession_link *fslink = NULL; | 829 | if (node->link->type == BPF_LINK_TYPE_TRACING) { |
| 772 | enum bpf_tramp_prog_type kind; | 830 | struct bpf_tracing_link *link; |
| 773 | struct bpf_tramp_link *link_exiting; | ||
| 774 | struct hlist_head *prog_list; | ||
| 775 | int err = 0; | ||
| 776 | int cnt = 0, i; | ||
| 777 | 831 | ||
| 778 | kind = bpf_attach_type_to_tramp(link->link.prog); | 832 | link = container_of(node->link, struct bpf_tracing_link, link.link); |
| 779 | if (tr->extension_prog) | 833 | return &link->fexit; |
| 780 | /* cannot attach fentry/fexit if extension prog is attached. | 834 | } else if (node->link->type == BPF_LINK_TYPE_TRACING_MULTI) { |
| 781 | * cannot overwrite extension prog either. | 835 | struct bpf_tracing_multi_link *link; |
| 782 | */ | 836 | struct bpf_tracing_multi_node *mnode; |
| 783 | return -EBUSY; | ||
| 784 | 837 | ||
| 785 | for (i = 0; i < BPF_TRAMP_MAX; i++) | 838 | link = container_of(node->link, struct bpf_tracing_multi_link, link); |
| 786 | cnt += tr->progs_cnt[i]; | 839 | mnode = container_of(node, struct bpf_tracing_multi_node, node); |
| 787 | 840 | return &link->fexits[mnode - link->nodes]; | |
| 788 | if (kind == BPF_TRAMP_REPLACE) { | ||
| 789 | /* Cannot attach extension if fentry/fexit are in use. */ | ||
| 790 | if (cnt) | ||
| 791 | return -EBUSY; | ||
| 792 | err = bpf_freplace_check_tgt_prog(tgt_prog); | ||
| 793 | if (err) | ||
| 794 | return err; | ||
| 795 | tr->extension_prog = link->link.prog; | ||
| 796 | return bpf_arch_text_poke(tr->func.addr, BPF_MOD_NOP, | ||
| 797 | BPF_MOD_JUMP, NULL, | ||
| 798 | link->link.prog->bpf_func); | ||
| 799 | } | 841 | } |
| 842 | return NULL; | ||
| 843 | } | ||
| 844 | |||
| 845 | static int bpf_trampoline_add_prog(struct bpf_trampoline *tr, | ||
| 846 | struct bpf_tramp_node *node, | ||
| 847 | int cnt) | ||
| 848 | { | ||
| 849 | enum bpf_tramp_prog_type kind; | ||
| 850 | struct bpf_tramp_node *node_existing, *fexit; | ||
| 851 | struct hlist_head *prog_list; | ||
| 852 | |||
| 853 | kind = bpf_attach_type_to_tramp(node->link->prog); | ||
| 800 | if (kind == BPF_TRAMP_FSESSION) { | 854 | if (kind == BPF_TRAMP_FSESSION) { |
| 801 | prog_list = &tr->progs_hlist[BPF_TRAMP_FENTRY]; | 855 | prog_list = &tr->progs_hlist[BPF_TRAMP_FENTRY]; |
| 802 | cnt++; | 856 | cnt++; |
| @@ -805,59 +859,112 @@ static int __bpf_trampoline_link_prog(struct bpf_tramp_link *link, · 2 unmodified lines | |||
| 805 | } | 859 | } |
| 806 | if (cnt >= BPF_MAX_TRAMP_LINKS) | 860 | if (cnt >= BPF_MAX_TRAMP_LINKS) |
| 807 | return -E2BIG; | 861 | return -E2BIG; |
| 808 | if (!hlist_unhashed(&link->tramp_hlist)) | 862 | if (!hlist_unhashed(&node->tramp_hlist)) |
| 809 | /* prog already linked */ | 863 | /* prog already linked */ |
| 810 | return -EBUSY; | 864 | return -EBUSY; |
| 811 | hlist_for_each_entry(link_exiting, prog_list, tramp_hlist) { | 865 | hlist_for_each_entry(node_existing, prog_list, tramp_hlist) { |
| 812 | if (link_exiting->link.prog != link->link.prog) | 866 | if (node_existing->link->prog != node->link->prog) |
| 813 | continue; | 867 | continue; |
| 814 | /* prog already linked */ | 868 | /* prog already linked */ |
| 815 | return -EBUSY; | 869 | return -EBUSY; |
| 816 | } | 870 | } |
| 817 | 871 | ||
| 818 | hlist_add_head(&link->tramp_hlist, prog_list); | 872 | hlist_add_head(&node->tramp_hlist, prog_list); |
| 819 | if (kind == BPF_TRAMP_FSESSION) { | 873 | if (kind == BPF_TRAMP_FSESSION) { |
| 820 | tr->progs_cnt[BPF_TRAMP_FENTRY]++; | 874 | tr->progs_cnt[BPF_TRAMP_FENTRY]++; |
| 821 | fslink = container_of(link, struct bpf_fsession_link, link.link); | 875 | fexit = fsession_exit(node); |
| 822 | hlist_add_head(&fslink->fexit.tramp_hlist, &tr->progs_hlist[BPF_TRAMP_FEXIT]); | 876 | if (WARN_ON_ONCE(!fexit)) |
| 877 | return -EINVAL; | ||
| 878 | hlist_add_head(&fexit->tramp_hlist, &tr->progs_hlist[BPF_TRAMP_FEXIT]); | ||
| 823 | tr->progs_cnt[BPF_TRAMP_FEXIT]++; | 879 | tr->progs_cnt[BPF_TRAMP_FEXIT]++; |
| 824 | } else { | 880 | } else { |
| 825 | tr->progs_cnt[kind]++; | 881 | tr->progs_cnt[kind]++; |
| 826 | } | 882 | } |
| 827 | err = bpf_trampoline_update(tr, true /* lock_direct_mutex */); | 883 | return 0; |
| 828 | if (err) { | 884 | } |
| 829 | hlist_del_init(&link->tramp_hlist); | 885 | |
| 830 | if (kind == BPF_TRAMP_FSESSION) { | 886 | static void bpf_trampoline_remove_prog(struct bpf_trampoline *tr, |
| 831 | tr->progs_cnt[BPF_TRAMP_FENTRY]--; | 887 | struct bpf_tramp_node *node) |
| 832 | hlist_del_init(&fslink->fexit.tramp_hlist); | 888 | { |
| 833 | tr->progs_cnt[BPF_TRAMP_FEXIT]--; | 889 | enum bpf_tramp_prog_type kind; |
| 834 | } else { | 890 | struct bpf_tramp_node *fexit; |
| 835 | tr->progs_cnt[kind]--; | 891 | |
| 836 | } | 892 | kind = bpf_attach_type_to_tramp(node->link->prog); |
| 893 | if (kind == BPF_TRAMP_FSESSION) { | ||
| 894 | fexit = fsession_exit(node); | ||
| 895 | if (WARN_ON_ONCE(!fexit)) | ||
| 896 | return; | ||
| 897 | hlist_del_init(&fexit->tramp_hlist); | ||
| 898 | tr->progs_cnt[BPF_TRAMP_FEXIT]--; | ||
| 899 | kind = BPF_TRAMP_FENTRY; | ||
| 837 | } | 900 | } |
| 901 | hlist_del_init(&node->tramp_hlist); | ||
| 902 | tr->progs_cnt[kind]--; | ||
| 903 | } | ||
| 904 | |||
| 905 | static int __bpf_trampoline_link_prog(struct bpf_tramp_node *node, | ||
| 906 | struct bpf_trampoline *tr, | ||
| 907 | struct bpf_prog *tgt_prog, | ||
| 908 | const struct bpf_trampoline_ops *ops, | ||
| 909 | void *data) | ||
| 910 | { | ||
| 911 | enum bpf_tramp_prog_type kind; | ||
| 912 | int err = 0; | ||
| 913 | int cnt = 0, i; | ||
| 914 | |||
| 915 | kind = bpf_attach_type_to_tramp(node->link->prog); | ||
| 916 | if (tr->extension_prog) | ||
| 917 | /* cannot attach fentry/fexit if extension prog is attached. | ||
| 918 | * cannot overwrite extension prog either. | ||
| 919 | */ | ||
| 920 | return -EBUSY; | ||
| 921 | |||
| 922 | for (i = 0; i < BPF_TRAMP_MAX; i++) | ||
| 923 | cnt += tr->progs_cnt[i]; | ||
| 924 | |||
| 925 | if (kind == BPF_TRAMP_REPLACE) { | ||
| 926 | /* Cannot attach extension if fentry/fexit are in use. */ | ||
| 927 | if (cnt) | ||
| 928 | return -EBUSY; | ||
| 929 | err = bpf_freplace_check_tgt_prog(tgt_prog); | ||
| 930 | if (err) | ||
| 931 | return err; | ||
| 932 | tr->extension_prog = node->link->prog; | ||
| 933 | return bpf_arch_text_poke(tr->func.addr, BPF_MOD_NOP, | ||
| 934 | BPF_MOD_JUMP, NULL, | ||
| 935 | node->link->prog->bpf_func); | ||
| 936 | } | ||
| 937 | err = bpf_trampoline_add_prog(tr, node, cnt); | ||
| 938 | if (err) | ||
| 939 | return err; | ||
| 940 | err = bpf_trampoline_update(tr, true /* lock_direct_mutex */, ops, data); | ||
| 941 | if (err) | ||
| 942 | bpf_trampoline_remove_prog(tr, node); | ||
| 838 | return err; | 943 | return err; |
| 839 | } | 944 | } |
| 840 | 945 | ||
| 841 | int bpf_trampoline_link_prog(struct bpf_tramp_link *link, | 946 | int bpf_trampoline_link_prog(struct bpf_tramp_node *node, |
| 842 | struct bpf_trampoline *tr, | 947 | struct bpf_trampoline *tr, |
| 843 | struct bpf_prog *tgt_prog) | 948 | struct bpf_prog *tgt_prog) |
| 844 | { | 949 | { |
| 845 | int err; | 950 | int err; |
| 846 | 951 | ||
| 847 | mutex_lock(&tr->mutex); | 952 | trampoline_lock(tr); |
| 848 | err = __bpf_trampoline_link_prog(link, tr, tgt_prog); | 953 | err = __bpf_trampoline_link_prog(node, tr, tgt_prog, &trampoline_ops, NULL); |
| 849 | mutex_unlock(&tr->mutex); | 954 | trampoline_unlock(tr); |
| 850 | return err; | 955 | return err; |
| 851 | } | 956 | } |
| 852 | 957 | ||
| 853 | static int __bpf_trampoline_unlink_prog(struct bpf_tramp_link *link, | 958 | static int __bpf_trampoline_unlink_prog(struct bpf_tramp_node *node, |
| 854 | struct bpf_trampoline *tr, | 959 | struct bpf_trampoline *tr, |
| 855 | struct bpf_prog *tgt_prog) | 960 | struct bpf_prog *tgt_prog, |
| 961 | const struct bpf_trampoline_ops *ops, | ||
| 962 | void *data) | ||
| 856 | { | 963 | { |
| 857 | enum bpf_tramp_prog_type kind; | 964 | enum bpf_tramp_prog_type kind; |
| 858 | int err; | 965 | int err; |
| 859 | 966 | ||
| 860 | kind = bpf_attach_type_to_tramp(link->link.prog); | 967 | kind = bpf_attach_type_to_tramp(node->link->prog); |
| 861 | if (kind == BPF_TRAMP_REPLACE) { | 968 | if (kind == BPF_TRAMP_REPLACE) { |
| 862 | WARN_ON_ONCE(!tr->extension_prog); | 969 | WARN_ON_ONCE(!tr->extension_prog); |
| 863 | err = bpf_arch_text_poke(tr->func.addr, BPF_MOD_JUMP, | 970 | err = bpf_arch_text_poke(tr->func.addr, BPF_MOD_JUMP, |
| @@ -867,29 +974,21 @@ static int __bpf_trampoline_unlink_prog(struct bpf_tramp_link *link, · 3 unmodified lines | |||
| 867 | guard(mutex)(&tgt_prog->aux->ext_mutex); | 974 | guard(mutex)(&tgt_prog->aux->ext_mutex); |
| 868 | tgt_prog->aux->is_extended = false; | 975 | tgt_prog->aux->is_extended = false; |
| 869 | return err; | 976 | return err; |
| 870 | } else if (kind == BPF_TRAMP_FSESSION) { | ||
| 871 | struct bpf_fsession_link *fslink = | ||
| 872 | container_of(link, struct bpf_fsession_link, link.link); | ||
| 873 | |||
| 874 | hlist_del_init(&fslink->fexit.tramp_hlist); | ||
| 875 | tr->progs_cnt[BPF_TRAMP_FEXIT]--; | ||
| 876 | kind = BPF_TRAMP_FENTRY; | ||
| 877 | } | 977 | } |
| 878 | hlist_del_init(&link->tramp_hlist); | 978 | bpf_trampoline_remove_prog(tr, node); |
| 879 | tr->progs_cnt[kind]--; | 979 | return bpf_trampoline_update(tr, true /* lock_direct_mutex */, ops, data); |
| 880 | return bpf_trampoline_update(tr, true /* lock_direct_mutex */); | ||
| 881 | } | 980 | } |
| 882 | 981 | ||
| 883 | /* bpf_trampoline_unlink_prog() should never fail. */ | 982 | /* bpf_trampoline_unlink_prog() should never fail. */ |
| 884 | int bpf_trampoline_unlink_prog(struct bpf_tramp_link *link, | 983 | int bpf_trampoline_unlink_prog(struct bpf_tramp_node *node, |
| 885 | struct bpf_trampoline *tr, | 984 | struct bpf_trampoline *tr, |
| 886 | struct bpf_prog *tgt_prog) | 985 | struct bpf_prog *tgt_prog) |
| 887 | { | 986 | { |
| 888 | int err; | 987 | int err; |
| 889 | 988 | ||
| 890 | mutex_lock(&tr->mutex); | 989 | trampoline_lock(tr); |
| 891 | err = __bpf_trampoline_unlink_prog(link, tr, tgt_prog); | 990 | err = __bpf_trampoline_unlink_prog(node, tr, tgt_prog, &trampoline_ops, NULL); |
| 892 | mutex_unlock(&tr->mutex); | 991 | trampoline_unlock(tr); |
| 893 | return err; | 992 | return err; |
| 894 | } | 993 | } |
| 895 | 994 | ||
| @@ -903,7 +1002,7 @@ static void bpf_shim_tramp_link_release(struct bpf_link *link) · 7 unmodified lines | |||
| 903 | if (!shim_link->trampoline) | 1002 | if (!shim_link->trampoline) |
| 904 | return; | 1003 | return; |
| 905 | 1004 | ||
| 906 | WARN_ON_ONCE(bpf_trampoline_unlink_prog(&shim_link->link, shim_link->trampoline, NULL)); | 1005 | WARN_ON_ONCE(bpf_trampoline_unlink_prog(&shim_link->link.node, shim_link->trampoline, NULL)); |
| 907 | bpf_trampoline_put(shim_link->trampoline); | 1006 | bpf_trampoline_put(shim_link->trampoline); |
| 908 | } | 1007 | } |
| 909 | 1008 | ||
| @@ -949,8 +1048,8 @@ static struct bpf_shim_tramp_link *cgroup_shim_alloc(const struct bpf_prog *prog · 39 unmodified lines | |||
| 949 | p->type = BPF_PROG_TYPE_LSM; | 1048 | p->type = BPF_PROG_TYPE_LSM; |
| 950 | p->expected_attach_type = BPF_LSM_MAC; | 1049 | p->expected_attach_type = BPF_LSM_MAC; |
| 951 | bpf_prog_inc(p); | 1050 | bpf_prog_inc(p); |
| 952 | bpf_link_init(&shim_link->link.link, BPF_LINK_TYPE_UNSPEC, | 1051 | bpf_tramp_link_init(&shim_link->link, BPF_LINK_TYPE_UNSPEC, |
| 953 | &bpf_shim_tramp_link_lops, p, attach_type); | 1052 | &bpf_shim_tramp_link_lops, p, attach_type, 0); |
| 954 | bpf_cgroup_atype_get(p->aux->attach_btf_id, cgroup_atype); | 1053 | bpf_cgroup_atype_get(p->aux->attach_btf_id, cgroup_atype); |
| 955 | 1054 | ||
| 956 | return shim_link; | 1055 | return shim_link; |
| @@ -959,15 +1058,15 @@ static struct bpf_shim_tramp_link *cgroup_shim_alloc(const struct bpf_prog *prog · 2 unmodified lines | |||
| 959 | static struct bpf_shim_tramp_link *cgroup_shim_find(struct bpf_trampoline *tr, | 1058 | static struct bpf_shim_tramp_link *cgroup_shim_find(struct bpf_trampoline *tr, |
| 960 | bpf_func_t bpf_func) | 1059 | bpf_func_t bpf_func) |
| 961 | { | 1060 | { |
| 962 | struct bpf_tramp_link *link; | 1061 | struct bpf_tramp_node *node; |
| 963 | int kind; | 1062 | int kind; |
| 964 | 1063 | ||
| 965 | for (kind = 0; kind < BPF_TRAMP_MAX; kind++) { | 1064 | for (kind = 0; kind < BPF_TRAMP_MAX; kind++) { |
| 966 | hlist_for_each_entry(link, &tr->progs_hlist[kind], tramp_hlist) { | 1065 | hlist_for_each_entry(node, &tr->progs_hlist[kind], tramp_hlist) { |
| 967 | struct bpf_prog *p = link->link.prog; | 1066 | struct bpf_prog *p = node->link->prog; |
| 968 | 1067 | ||
| 969 | if (p->bpf_func == bpf_func) | 1068 | if (p->bpf_func == bpf_func) |
| 970 | return container_of(link, struct bpf_shim_tramp_link, link); | 1069 | return container_of(node, struct bpf_shim_tramp_link, link.node); |
| 971 | } | 1070 | } |
| 972 | } | 1071 | } |
| 973 | 1072 | ||
| @@ -999,12 +1098,12 @@ int bpf_trampoline_link_cgroup_shim(struct bpf_prog *prog, · 25 unmodified lines | |||
| 999 | if (!tr) | 1098 | if (!tr) |
| 1000 | return -ENOMEM; | 1099 | return -ENOMEM; |
| 1001 | 1100 | ||
| 1002 | mutex_lock(&tr->mutex); | 1101 | trampoline_lock(tr); |
| 1003 | 1102 | ||
| 1004 | shim_link = cgroup_shim_find(tr, bpf_func); | 1103 | shim_link = cgroup_shim_find(tr, bpf_func); |
| 1005 | if (shim_link && !IS_ERR(bpf_link_inc_not_zero(&shim_link->link.link))) { | 1104 | if (shim_link && !IS_ERR(bpf_link_inc_not_zero(&shim_link->link.link))) { |
| 1006 | /* Reusing existing shim attached by the other program. */ | 1105 | /* Reusing existing shim attached by the other program. */ |
| 1007 | mutex_unlock(&tr->mutex); | 1106 | trampoline_unlock(tr); |
| 1008 | bpf_trampoline_put(tr); /* bpf_trampoline_get above */ | 1107 | bpf_trampoline_put(tr); /* bpf_trampoline_get above */ |
| 1009 | return 0; | 1108 | return 0; |
| 1010 | } | 1109 | } |
| @@ -1017,23 +1116,23 @@ int bpf_trampoline_link_cgroup_shim(struct bpf_prog *prog, · 6 unmodified lines | |||
| 1017 | goto err; | 1116 | goto err; |
| 1018 | } | 1117 | } |
| 1019 | 1118 | ||
| 1020 | err = __bpf_trampoline_link_prog(&shim_link->link, tr, NULL); | 1119 | err = __bpf_trampoline_link_prog(&shim_link->link.node, tr, NULL, &trampoline_ops, NULL); |
| 1021 | if (err) | 1120 | if (err) |
| 1022 | goto err; | 1121 | goto err; |
| 1023 | 1122 | ||
| 1024 | shim_link->trampoline = tr; | 1123 | shim_link->trampoline = tr; |
| 1025 | /* note, we're still holding tr refcnt from above */ | 1124 | /* note, we're still holding tr refcnt from above */ |
| 1026 | 1125 | ||
| 1027 | mutex_unlock(&tr->mutex); | 1126 | trampoline_unlock(tr); |
| 1028 | 1127 | ||
| 1029 | return 0; | 1128 | return 0; |
| 1030 | err: | 1129 | err: |
| 1031 | mutex_unlock(&tr->mutex); | 1130 | trampoline_unlock(tr); |
| 1032 | 1131 | ||
| 1033 | if (shim_link) | 1132 | if (shim_link) |
| 1034 | bpf_link_put(&shim_link->link.link); | 1133 | bpf_link_put(&shim_link->link.link); |
| 1035 | 1134 | ||
| 1036 | /* have to release tr while _not_ holding its mutex */ | 1135 | /* have to release tr while _not_ holding pool mutex for trampoline */ |
| 1037 | bpf_trampoline_put(tr); /* bpf_trampoline_get above */ | 1136 | bpf_trampoline_put(tr); /* bpf_trampoline_get above */ |
| 1038 | 1137 | ||
| 1039 | return err; | 1138 | return err; |
| @@ -1054,9 +1153,9 @@ void bpf_trampoline_unlink_cgroup_shim(struct bpf_prog *prog) · 14 unmodified lines | |||
| 1054 | if (WARN_ON_ONCE(!tr)) | 1153 | if (WARN_ON_ONCE(!tr)) |
| 1055 | return; | 1154 | return; |
| 1056 | 1155 | ||
| 1057 | mutex_lock(&tr->mutex); | 1156 | trampoline_lock(tr); |
| 1058 | shim_link = cgroup_shim_find(tr, bpf_func); | 1157 | shim_link = cgroup_shim_find(tr, bpf_func); |
| 1059 | mutex_unlock(&tr->mutex); | 1158 | trampoline_unlock(tr); |
| 1060 | 1159 | ||
| 1061 | if (shim_link) | 1160 | if (shim_link) |
| 1062 | bpf_link_put(&shim_link->link.link); | 1161 | bpf_link_put(&shim_link->link.link); |
| @@ -1074,14 +1173,14 @@ struct bpf_trampoline *bpf_trampoline_get(u64 key, · 11 unmodified lines | |||
| 1074 | if (!tr) | 1173 | if (!tr) |
| 1075 | return NULL; | 1174 | return NULL; |
| 1076 | 1175 | ||
| 1077 | mutex_lock(&tr->mutex); | 1176 | trampoline_lock(tr); |
| 1078 | if (tr->func.addr) | 1177 | if (tr->func.addr) |
| 1079 | goto out; | 1178 | goto out; |
| 1080 | 1179 | ||
| 1081 | memcpy(&tr->func.model, &tgt_info->fmodel, sizeof(tgt_info->fmodel)); | 1180 | memcpy(&tr->func.model, &tgt_info->fmodel, sizeof(tgt_info->fmodel)); |
| 1082 | tr->func.addr = (void *)tgt_info->tgt_addr; | 1181 | tr->func.addr = (void *)tgt_info->tgt_addr; |
| 1083 | out: | 1182 | out: |
| 1084 | mutex_unlock(&tr->mutex); | 1183 | trampoline_unlock(tr); |
| 1085 | return tr; | 1184 | return tr; |
| 1086 | } | 1185 | } |
| 1087 | 1186 | ||
| @@ -1094,7 +1193,6 @@ void bpf_trampoline_put(struct bpf_trampoline *tr) · 6 unmodified lines | |||
| 1094 | mutex_lock(&trampoline_mutex); | 1193 | mutex_lock(&trampoline_mutex); |
| 1095 | if (!refcount_dec_and_test(&tr->refcnt)) | 1194 | if (!refcount_dec_and_test(&tr->refcnt)) |
| 1096 | goto out; | 1195 | goto out; |
| 1097 | WARN_ON_ONCE(mutex_is_locked(&tr->mutex)); | ||
| 1098 | 1196 | ||
| 1099 | for (i = 0; i < BPF_TRAMP_MAX; i++) | 1197 | for (i = 0; i < BPF_TRAMP_MAX; i++) |
| 1100 | if (WARN_ON_ONCE(!hlist_empty(&tr->progs_hlist[i]))) | 1198 | if (WARN_ON_ONCE(!hlist_empty(&tr->progs_hlist[i]))) |
| @@ -1333,7 +1431,7 @@ bpf_trampoline_exit_t bpf_trampoline_exit(const struct bpf_prog *prog) · 232 unmodified lines | |||
| 1333 | int __weak | 1431 | int __weak |
| 1334 | arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *image_end, | 1432 | arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *image_end, |
| 1335 | const struct btf_func_model *m, u32 flags, | 1433 | const struct btf_func_model *m, u32 flags, |
| 1336 | struct bpf_tramp_links *tlinks, | 1434 | struct bpf_tramp_nodes *tnodes, |
| 1337 | void *func_addr) | 1435 | void *func_addr) |
| 1338 | { | 1436 | { |
| 1339 | return -ENOTSUPP; | 1437 | return -ENOTSUPP; |
| @@ -1367,11 +1465,288 @@ int __weak arch_protect_bpf_trampoline(void *image, unsigned int size) · 27 unmodified lines | |||
| 1367 | } | 1465 | } |
| 1368 | 1466 | ||
| 1369 | int __weak arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, | 1467 | int __weak arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, |
| 1370 | struct bpf_tramp_links *tlinks, void *func_addr) | 1468 | struct bpf_tramp_nodes *tnodes, void *func_addr) |
| 1371 | { | 1469 | { |
| 1372 | return -ENOTSUPP; | 1470 | return -ENOTSUPP; |
| 1373 | } | 1471 | } |
| 1374 | 1472 | ||
| 1473 | #if defined(CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS) && \ | ||
| 1474 | defined(CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS) && \ | ||
| 1475 | defined(CONFIG_BPF_SYSCALL) | ||
| 1476 | |||
| 1477 | static void trampoline_lock_all(void) | ||
| 1478 | { | ||
| 1479 | int i; | ||
| 1480 | |||
| 1481 | for (i = 0; i < TRAMPOLINE_LOCKS_TABLE_SIZE; i++) | ||
| 1482 | mutex_lock(&trampoline_locks[i].mutex); | ||
| 1483 | } | ||
| 1484 | |||
| 1485 | static void trampoline_unlock_all(void) | ||
| 1486 | { | ||
| 1487 | int i; | ||
| 1488 | |||
| 1489 | for (i = 0; i < TRAMPOLINE_LOCKS_TABLE_SIZE; i++) | ||
| 1490 | mutex_unlock(&trampoline_locks[i].mutex); | ||
| 1491 | } | ||
| 1492 | |||
| 1493 | static void remove_tracing_multi_data(struct bpf_tracing_multi_data *data) | ||
| 1494 | { | ||
| 1495 | ftrace_hash_remove(data->reg); | ||
| 1496 | ftrace_hash_remove(data->unreg); | ||
| 1497 | ftrace_hash_remove(data->modify); | ||
| 1498 | } | ||
| 1499 | |||
| 1500 | static void clear_tracing_multi_data(struct bpf_tracing_multi_data *data) | ||
| 1501 | { | ||
| 1502 | remove_tracing_multi_data(data); | ||
| 1503 | |||
| 1504 | free_ftrace_hash(data->reg); | ||
| 1505 | free_ftrace_hash(data->unreg); | ||
| 1506 | free_ftrace_hash(data->modify); | ||
| 1507 | } | ||
| 1508 | |||
| 1509 | static int init_tracing_multi_data(struct bpf_tracing_multi_data *data) | ||
| 1510 | { | ||
| 1511 | data->reg = alloc_ftrace_hash(FTRACE_HASH_DEFAULT_BITS); | ||
| 1512 | data->unreg = alloc_ftrace_hash(FTRACE_HASH_DEFAULT_BITS); | ||
| 1513 | data->modify = alloc_ftrace_hash(FTRACE_HASH_DEFAULT_BITS); | ||
| 1514 | |||
| 1515 | if (!data->reg || !data->unreg || !data->modify) { | ||
| 1516 | clear_tracing_multi_data(data); | ||
| 1517 | return -ENOMEM; | ||
| 1518 | } | ||
| 1519 | return 0; | ||
| 1520 | } | ||
| 1521 | |||
| 1522 | static void ftrace_hash_add(struct ftrace_hash *hash, struct ftrace_func_entry *entry, | ||
| 1523 | unsigned long ip, unsigned long direct) | ||
| 1524 | { | ||
| 1525 | entry->ip = ip; | ||
| 1526 | entry->direct = direct; | ||
| 1527 | add_ftrace_hash_entry(hash, entry); | ||
| 1528 | } | ||
| 1529 | |||
| 1530 | static int register_fentry_multi(struct bpf_trampoline *tr, struct bpf_tramp_image *im, void *ptr) | ||
| 1531 | { | ||
| 1532 | unsigned long addr = (unsigned long) im->image; | ||
| 1533 | unsigned long ip = ftrace_location(tr->ip); | ||
| 1534 | struct bpf_tracing_multi_data *data = ptr; | ||
| 1535 | |||
| 1536 | if (bpf_trampoline_use_jmp(tr->flags)) | ||
| 1537 | addr = ftrace_jmp_set(addr); | ||
| 1538 | |||
| 1539 | ftrace_hash_add(data->reg, data->entry, ip, addr); | ||
| 1540 | tr->cur_image = im; | ||
| 1541 | return 0; | ||
| 1542 | } | ||
| 1543 | |||
| 1544 | static int unregister_fentry_multi(struct bpf_trampoline *tr, u32 orig_flags, void *ptr) | ||
| 1545 | { | ||
| 1546 | unsigned long addr = (unsigned long) tr->cur_image->image; | ||
| 1547 | unsigned long ip = ftrace_location(tr->ip); | ||
| 1548 | struct bpf_tracing_multi_data *data = ptr; | ||
| 1549 | |||
| 1550 | if (bpf_trampoline_use_jmp(tr->flags)) | ||
| 1551 | addr = ftrace_jmp_set(addr); | ||
| 1552 | |||
| 1553 | ftrace_hash_add(data->unreg, data->entry, ip, addr); | ||
| 1554 | tr->cur_image = NULL; | ||
| 1555 | return 0; | ||
| 1556 | } | ||
| 1557 | |||
| 1558 | static int modify_fentry_multi(struct bpf_trampoline *tr, u32 orig_flags, struct bpf_tramp_image *im, | ||
| 1559 | bool lock_direct_mutex, void *ptr) | ||
| 1560 | { | ||
| 1561 | unsigned long addr = (unsigned long) im->image; | ||
| 1562 | unsigned long ip = ftrace_location(tr->ip); | ||
| 1563 | struct bpf_tracing_multi_data *data = ptr; | ||
| 1564 | |||
| 1565 | if (bpf_trampoline_use_jmp(tr->flags)) | ||
| 1566 | addr = ftrace_jmp_set(addr); | ||
| 1567 | |||
| 1568 | ftrace_hash_add(data->modify, data->entry, ip, addr); | ||
| 1569 | tr->cur_image = im; | ||
| 1570 | return 0; | ||
| 1571 | } | ||
| 1572 | |||
| 1573 | static const struct bpf_trampoline_ops trampoline_multi_ops = { | ||
| 1574 | .register_fentry = register_fentry_multi, | ||
| 1575 | .unregister_fentry = unregister_fentry_multi, | ||
| 1576 | .modify_fentry = modify_fentry_multi, | ||
| 1577 | }; | ||
| 1578 | |||
| 1579 | static void bpf_trampoline_multi_attach_init(struct bpf_trampoline *tr) | ||
| 1580 | { | ||
| 1581 | tr->multi_attach.old_image = tr->cur_image; | ||
| 1582 | tr->multi_attach.old_flags = tr->flags; | ||
| 1583 | } | ||
| 1584 | |||
| 1585 | static void bpf_trampoline_multi_attach_free(struct bpf_trampoline *tr) | ||
| 1586 | { | ||
| 1587 | if (tr->multi_attach.old_image) | ||
| 1588 | bpf_tramp_image_put(tr->multi_attach.old_image); | ||
| 1589 | |||
| 1590 | tr->multi_attach.old_image = NULL; | ||
| 1591 | tr->multi_attach.old_flags = 0; | ||
| 1592 | } | ||
| 1593 | |||
| 1594 | static void bpf_trampoline_multi_attach_rollback(struct bpf_trampoline *tr) | ||
| 1595 | { | ||
| 1596 | if (tr->cur_image) | ||
| 1597 | bpf_tramp_image_put(tr->cur_image); | ||
| 1598 | tr->cur_image = tr->multi_attach.old_image; | ||
| 1599 | tr->flags = tr->multi_attach.old_flags; | ||
| 1600 | |||
| 1601 | tr->multi_attach.old_image = NULL; | ||
| 1602 | tr->multi_attach.old_flags = 0; | ||
| 1603 | } | ||
| 1604 | |||
| 1605 | #define for_each_mnode_cnt(mnode, link, cnt) \ | ||
| 1606 | for (i = 0, mnode = &link->nodes[i]; i < cnt; i++, mnode = &link->nodes[i]) | ||
| 1607 | |||
| 1608 | #define for_each_mnode(mnode, link) \ | ||
| 1609 | for_each_mnode_cnt(mnode, link, link->nodes_cnt) | ||
| 1610 | |||
| 1611 | int bpf_trampoline_multi_attach(struct bpf_prog *prog, u32 *ids, | ||
| 1612 | struct bpf_tracing_multi_link *link) | ||
| 1613 | { | ||
| 1614 | struct bpf_tracing_multi_data *data = &link->data; | ||
| 1615 | struct bpf_attach_target_info tgt_info = {}; | ||
| 1616 | struct btf *btf = prog->aux->attach_btf; | ||
| 1617 | struct bpf_tracing_multi_node *mnode; | ||
| 1618 | struct bpf_trampoline *tr; | ||
| 1619 | int i, err, rollback_cnt; | ||
| 1620 | u64 key; | ||
| 1621 | |||
| 1622 | for_each_mnode(mnode, link) { | ||
| 1623 | rollback_cnt = i; | ||
| 1624 | |||
| 1625 | err = bpf_check_attach_btf_id_multi(btf, prog, ids[i], &tgt_info); | ||
| 1626 | if (err) | ||
| 1627 | goto rollback_put; | ||
| 1628 | |||
| 1629 | key = bpf_trampoline_compute_key(NULL, btf, ids[i]); | ||
| 1630 | |||
| 1631 | tr = bpf_trampoline_get(key, &tgt_info); | ||
| 1632 | if (!tr) { | ||
| 1633 | err = -ENOMEM; | ||
| 1634 | goto rollback_put; | ||
| 1635 | } | ||
| 1636 | |||
| 1637 | mnode->trampoline = tr; | ||
| 1638 | mnode->node.link = &link->link; | ||
| 1639 | mnode->node.cookie = link->cookies ? link->cookies[i] : 0; | ||
| 1640 | |||
| 1641 | if (prog->expected_attach_type == BPF_TRACE_FSESSION_MULTI) { | ||
| 1642 | link->fexits[i].link = &link->link; | ||
| 1643 | link->fexits[i].cookie = link->cookies ? link->cookies[i] : 0; | ||
| 1644 | } | ||
| 1645 | |||
| 1646 | cond_resched(); | ||
| 1647 | } | ||
| 1648 | |||
| 1649 | err = init_tracing_multi_data(data); | ||
| 1650 | if (err) { | ||
| 1651 | rollback_cnt = link->nodes_cnt; | ||
| 1652 | goto rollback_put; | ||
| 1653 | } | ||
| 1654 | |||
| 1655 | trampoline_lock_all(); | ||
| 1656 | |||
| 1657 | for_each_mnode(mnode, link) { | ||
| 1658 | bpf_trampoline_multi_attach_init(mnode->trampoline); | ||
| 1659 | |||
| 1660 | data->entry = &mnode->entry; | ||
| 1661 | err = __bpf_trampoline_link_prog(&mnode->node, mnode->trampoline, NULL, | ||
| 1662 | &trampoline_multi_ops, data); | ||
| 1663 | if (err) { | ||
| 1664 | rollback_cnt = i; | ||
| 1665 | goto rollback_unlink; | ||
| 1666 | } | ||
| 1667 | } | ||
| 1668 | |||
| 1669 | rollback_cnt = link->nodes_cnt; | ||
| 1670 | if (ftrace_hash_count(data->reg)) { | ||
| 1671 | err = update_ftrace_direct_add(&direct_ops, data->reg); | ||
| 1672 | if (err) | ||
| 1673 | goto rollback_unlink; | ||
| 1674 | } | ||
| 1675 | |||
| 1676 | if (ftrace_hash_count(data->modify)) { | ||
| 1677 | err = update_ftrace_direct_mod(&direct_ops, data->modify, true); | ||
| 1678 | if (err) { | ||
| 1679 | if (ftrace_hash_count(data->reg)) | ||
| 1680 | WARN_ON_ONCE(update_ftrace_direct_del(&direct_ops, data->reg)); | ||
| 1681 | goto rollback_unlink; | ||
| 1682 | } | ||
| 1683 | } | ||
| 1684 | |||
| 1685 | for_each_mnode(mnode, link) | ||
| 1686 | bpf_trampoline_multi_attach_free(mnode->trampoline); | ||
| 1687 | |||
| 1688 | trampoline_unlock_all(); | ||
| 1689 | |||
| 1690 | remove_tracing_multi_data(data); | ||
| 1691 | return 0; | ||
| 1692 | |||
| 1693 | rollback_unlink: | ||
| 1694 | for_each_mnode_cnt(mnode, link, rollback_cnt) { | ||
| 1695 | bpf_trampoline_remove_prog(mnode->trampoline, &mnode->node); | ||
| 1696 | bpf_trampoline_multi_attach_rollback(mnode->trampoline); | ||
| 1697 | } | ||
| 1698 | |||
| 1699 | trampoline_unlock_all(); | ||
| 1700 | |||
| 1701 | clear_tracing_multi_data(data); | ||
| 1702 | rollback_cnt = link->nodes_cnt; | ||
| 1703 | |||
| 1704 | rollback_put: | ||
| 1705 | for_each_mnode_cnt(mnode, link, rollback_cnt) | ||
| 1706 | bpf_trampoline_put(mnode->trampoline); | ||
| 1707 | |||
| 1708 | return err; | ||
| 1709 | } | ||
| 1710 | |||
| 1711 | int bpf_trampoline_multi_detach(struct bpf_prog *prog, struct bpf_tracing_multi_link *link) | ||
| 1712 | { | ||
| 1713 | struct bpf_tracing_multi_data *data = &link->data; | ||
| 1714 | struct bpf_tracing_multi_node *mnode; | ||
| 1715 | int i; | ||
| 1716 | |||
| 1717 | trampoline_lock_all(); | ||
| 1718 | |||
| 1719 | for_each_mnode(mnode, link) { | ||
| 1720 | data->entry = &mnode->entry; | ||
| 1721 | bpf_trampoline_multi_attach_init(mnode->trampoline); | ||
| 1722 | WARN_ON_ONCE(__bpf_trampoline_unlink_prog(&mnode->node, mnode->trampoline, | ||
| 1723 | NULL, &trampoline_multi_ops, data)); | ||
| 1724 | } | ||
| 1725 | |||
| 1726 | if (ftrace_hash_count(data->unreg)) | ||
| 1727 | WARN_ON_ONCE(update_ftrace_direct_del(&direct_ops, data->unreg)); | ||
| 1728 | if (ftrace_hash_count(data->modify)) | ||
| 1729 | WARN_ON_ONCE(update_ftrace_direct_mod(&direct_ops, data->modify, true)); | ||
| 1730 | |||
| 1731 | for_each_mnode(mnode, link) | ||
| 1732 | bpf_trampoline_multi_attach_free(mnode->trampoline); | ||
| 1733 | |||
| 1734 | trampoline_unlock_all(); | ||
| 1735 | |||
| 1736 | for_each_mnode(mnode, link) | ||
| 1737 | bpf_trampoline_put(mnode->trampoline); | ||
| 1738 | |||
| 1739 | clear_tracing_multi_data(data); | ||
| 1740 | return 0; | ||
| 1741 | } | ||
| 1742 | |||
| 1743 | #undef for_each_mnode_cnt | ||
| 1744 | #undef for_each_mnode | ||
| 1745 | |||
| 1746 | #endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS && | ||
| 1747 | CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS && | ||
| 1748 | CONFIG_BPF_SYSCALL */ | ||
| 1749 | |||
| 1375 | static int __init init_trampolines(void) | 1750 | static int __init init_trampolines(void) |
| 1376 | { | 1751 | { |
| 1377 | int i; | 1752 | int i; |
| @@ -1380,6 +1755,8 @@ static int __init init_trampolines(void) · 2 unmodified lines | |||
| 1380 | INIT_HLIST_HEAD(&trampoline_key_table[i]); | 1755 | INIT_HLIST_HEAD(&trampoline_key_table[i]); |
| 1381 | for (i = 0; i < TRAMPOLINE_TABLE_SIZE; i++) | 1756 | for (i = 0; i < TRAMPOLINE_TABLE_SIZE; i++) |
| 1382 | INIT_HLIST_HEAD(&trampoline_ip_table[i]); | 1757 | INIT_HLIST_HEAD(&trampoline_ip_table[i]); |
| 1758 | for (i = 0; i < TRAMPOLINE_LOCKS_TABLE_SIZE; i++) | ||
| 1759 | __mutex_init(&trampoline_locks[i].mutex, "trampoline_lock", &trampoline_locks[i].key); | ||
| 1383 | return 0; | 1760 | return 0; |
| 1384 | } | 1761 | } |
| 1385 | late_initcall(init_trampolines); | 1762 | late_initcall(init_trampolines); |
| @@ -26,6 +26,7 @@ · 25 unmodified lines | |||
| 26 | #include <linux/poison.h> | 26 | #include <linux/poison.h> |
| 27 | #include <linux/module.h> | 27 | #include <linux/module.h> |
| 28 | #include <linux/cpumask.h> | 28 | #include <linux/cpumask.h> |
| 29 | #include <linux/cnum.h> | ||
| 29 | #include <linux/bpf_mem_alloc.h> | 30 | #include <linux/bpf_mem_alloc.h> |
| 30 | #include <net/xdp.h> | 31 | #include <net/xdp.h> |
| 31 | #include <linux/trace_events.h> | 32 | #include <linux/trace_events.h> |
| @@ -199,14 +200,15 @@ struct bpf_verifier_stack_elem { · 167 unmodified lines | |||
| 199 | 200 | ||
| 200 | #define BPF_PRIV_STACK_MIN_SIZE 64 | 201 | #define BPF_PRIV_STACK_MIN_SIZE 64 |
| 201 | 202 | ||
| 202 | static int acquire_reference(struct bpf_verifier_env *env, int insn_idx); | 203 | static int acquire_reference(struct bpf_verifier_env *env, int insn_idx, int parent_id); |
| 203 | static int release_reference_nomark(struct bpf_verifier_state *state, int ref_obj_id); | 204 | static int release_reference_nomark(struct bpf_verifier_state *state, int id); |
| 204 | static int release_reference(struct bpf_verifier_env *env, int ref_obj_id); | 205 | static int release_reference(struct bpf_verifier_env *env, int id); |
| 205 | static void invalidate_non_owning_refs(struct bpf_verifier_env *env); | 206 | static void invalidate_non_owning_refs(struct bpf_verifier_env *env); |
| 206 | static bool in_rbtree_lock_required_cb(struct bpf_verifier_env *env); | 207 | static bool in_rbtree_lock_required_cb(struct bpf_verifier_env *env); |
| 208 | static bool is_tracing_prog_type(enum bpf_prog_type type); | ||
| 207 | static int ref_set_non_owning(struct bpf_verifier_env *env, | 209 | static int ref_set_non_owning(struct bpf_verifier_env *env, |
| 208 | struct bpf_reg_state *reg); | 210 | struct bpf_reg_state *reg); |
| 209 | static bool is_trusted_reg(const struct bpf_reg_state *reg); | 211 | static bool is_trusted_reg(struct bpf_verifier_env *env, const struct bpf_reg_state *reg); |
| 210 | static inline bool in_sleepable_context(struct bpf_verifier_env *env); | 212 | static inline bool in_sleepable_context(struct bpf_verifier_env *env); |
| 211 | static const char *non_sleepable_context_description(struct bpf_verifier_env *env); | 213 | static const char *non_sleepable_context_description(struct bpf_verifier_env *env); |
| 212 | static void scalar32_min_max_add(struct bpf_reg_state *dst_reg, struct bpf_reg_state *src_reg); | 214 | static void scalar32_min_max_add(struct bpf_reg_state *dst_reg, struct bpf_reg_state *src_reg); |
| @@ -230,8 +232,28 @@ static void bpf_map_key_store(struct bpf_insn_aux_data *aux, u64 state) · 17 unmodified lines | |||
| 230 | (poisoned ? BPF_MAP_KEY_POISON : 0ULL); | 232 | (poisoned ? BPF_MAP_KEY_POISON : 0ULL); |
| 231 | } | 233 | } |
| 232 | 234 | ||
| 235 | static void update_ref_obj(struct ref_obj_desc *ref_obj, struct bpf_reg_state *reg) | ||
| 236 | { | ||
| 237 | ref_obj->id = reg->id; | ||
| 238 | ref_obj->parent_id = reg->parent_id; | ||
| 239 | ref_obj->cnt++; | ||
| 240 | } | ||
| 241 | |||
| 242 | static int validate_ref_obj(struct bpf_verifier_env *env, struct ref_obj_desc *ref_obj) | ||
| 243 | { | ||
| 244 | if (ref_obj->cnt > 1) { | ||
| 245 | verifier_bug(env, "function expects only one referenced object but got %d\n", | ||
| 246 | ref_obj->cnt); | ||
| 247 | return -EFAULT; | ||
| 248 | } | ||
| 249 | |||
| 250 | return 0; | ||
| 251 | } | ||
| 252 | |||
| 233 | struct bpf_call_arg_meta { | 253 | struct bpf_call_arg_meta { |
| 234 | struct bpf_map_desc map; | 254 | struct bpf_map_desc map; |
| 255 | struct bpf_dynptr_desc dynptr; | ||
| 256 | struct ref_obj_desc ref_obj; | ||
| 235 | bool raw_mode; | 257 | bool raw_mode; |
| 236 | bool pkt_access; | 258 | bool pkt_access; |
| 237 | u8 release_regno; | 259 | u8 release_regno; |
| @@ -239,8 +261,6 @@ struct bpf_call_arg_meta { · 1 unmodified line | |||
| 239 | int access_size; | 261 | int access_size; |
| 240 | int mem_size; | 262 | int mem_size; |
| 241 | u64 msize_max_value; | 263 | u64 msize_max_value; |
| 242 | int ref_obj_id; | ||
| 243 | int dynptr_id; | ||
| 244 | int func_id; | 264 | int func_id; |
| 245 | struct btf *btf; | 265 | struct btf *btf; |
| 246 | u32 btf_id; | 266 | u32 btf_id; |
| @@ -261,6 +281,41 @@ struct bpf_kfunc_meta { · 14 unmodified lines | |||
| 261 | 281 | ||
| 262 | struct btf *btf_vmlinux; | 282 | struct btf *btf_vmlinux; |
| 263 | 283 | ||
| 284 | typedef struct argno { | ||
| 285 | int argno; | ||
| 286 | } argno_t; | ||
| 287 | |||
| 288 | static argno_t argno_from_reg(u32 regno) | ||
| 289 | { | ||
| 290 | return (argno_t){ .argno = regno }; | ||
| 291 | } | ||
| 292 | |||
| 293 | static argno_t argno_from_arg(u32 arg) | ||
| 294 | { | ||
| 295 | return (argno_t){ .argno = -arg }; | ||
| 296 | } | ||
| 297 | |||
| 298 | static int reg_from_argno(argno_t a) | ||
| 299 | { | ||
| 300 | if (a.argno >= 0) | ||
| 301 | return a.argno; | ||
| 302 | if (a.argno >= -MAX_BPF_FUNC_REG_ARGS) | ||
| 303 | return -a.argno; | ||
| 304 | return -1; | ||
| 305 | } | ||
| 306 | |||
| 307 | static int arg_from_argno(argno_t a) | ||
| 308 | { | ||
| 309 | if (a.argno < 0) | ||
| 310 | return -a.argno; | ||
| 311 | return -1; | ||
| 312 | } | ||
| 313 | |||
| 314 | static int arg_idx_from_argno(argno_t a) | ||
| 315 | { | ||
| 316 | return arg_from_argno(a) - 1; | ||
| 317 | } | ||
| 318 | |||
| 264 | static const char *btf_type_name(const struct btf *btf, u32 id) | 319 | static const char *btf_type_name(const struct btf *btf, u32 id) |
| 265 | { | 320 | { |
| 266 | return btf_name_by_offset(btf, btf_type_by_id(btf, id)->name_off); | 321 | return btf_name_by_offset(btf, btf_type_by_id(btf, id)->name_off); |
| @@ -290,12 +345,12 @@ static void verbose_invalid_scalar(struct bpf_verifier_env *env, · 23 unmodified lines | |||
| 290 | bool unknown = true; | 345 | bool unknown = true; |
| 291 | 346 | ||
| 292 | verbose(env, "%s the register %s has", ctx, reg_name); | 347 | verbose(env, "%s the register %s has", ctx, reg_name); |
| 293 | if (reg->smin_value > S64_MIN) { | 348 | if (reg_smin(reg) > S64_MIN) { |
| 294 | verbose(env, " smin=%lld", reg->smin_value); | 349 | verbose(env, " smin=%lld", reg_smin(reg)); |
| 295 | unknown = false; | 350 | unknown = false; |
| 296 | } | 351 | } |
| 297 | if (reg->smax_value < S64_MAX) { | 352 | if (reg_smax(reg) < S64_MAX) { |
| 298 | verbose(env, " smax=%lld", reg->smax_value); | 353 | verbose(env, " smax=%lld", reg_smax(reg)); |
| 299 | unknown = false; | 354 | unknown = false; |
| 300 | } | 355 | } |
| 301 | if (unknown) | 356 | if (unknown) |
| @@ -303,7 +358,7 @@ static void verbose_invalid_scalar(struct bpf_verifier_env *env, · 1 unmodified line | |||
| 303 | verbose(env, " should have been in [%d, %d]\n", range.minval, range.maxval); | 358 | verbose(env, " should have been in [%d, %d]\n", range.minval, range.maxval); |
| 304 | } | 359 | } |
| 305 | 360 | ||
| 306 | static bool reg_not_null(const struct bpf_reg_state *reg) | 361 | static bool reg_not_null(struct bpf_verifier_env *env, const struct bpf_reg_state *reg) |
| 307 | { | 362 | { |
| 308 | enum bpf_reg_type type; | 363 | enum bpf_reg_type type; |
| 309 | 364 | ||
| @@ -317,7 +372,7 @@ static bool reg_not_null(const struct bpf_reg_state *reg) · 7 unmodified lines | |||
| 317 | type == PTR_TO_MAP_VALUE || | 372 | type == PTR_TO_MAP_VALUE || |
| 318 | type == PTR_TO_MAP_KEY || | 373 | type == PTR_TO_MAP_KEY || |
| 319 | type == PTR_TO_SOCK_COMMON || | 374 | type == PTR_TO_SOCK_COMMON || |
| 320 | (type == PTR_TO_BTF_ID && is_trusted_reg(reg)) || | 375 | (type == PTR_TO_BTF_ID && is_trusted_reg(env, reg)) || |
| 321 | (type == PTR_TO_MEM && !(reg->type & PTR_UNTRUSTED)) || | 376 | (type == PTR_TO_MEM && !(reg->type & PTR_UNTRUSTED)) || |
| 322 | type == CONST_PTR_TO_MAP; | 377 | type == CONST_PTR_TO_MAP; |
| 323 | } | 378 | } |
| @@ -434,11 +489,6 @@ static bool is_ptr_cast_function(enum bpf_func_id func_id) · 110 unmodified lines | |||
| 434 | func_id == BPF_FUNC_skc_to_tcp_request_sock; | 489 | func_id == BPF_FUNC_skc_to_tcp_request_sock; |
| 435 | } | 490 | } |
| 436 | 491 | ||
| 437 | static bool is_dynptr_ref_function(enum bpf_func_id func_id) | ||
| 438 | { | ||
| 439 | return func_id == BPF_FUNC_dynptr_data; | ||
| 440 | } | ||
| 441 | |||
| 442 | static bool is_sync_callback_calling_kfunc(u32 btf_id); | 492 | static bool is_sync_callback_calling_kfunc(u32 btf_id); |
| 443 | static bool is_async_callback_calling_kfunc(u32 btf_id); | 493 | static bool is_async_callback_calling_kfunc(u32 btf_id); |
| 444 | static bool is_callback_calling_kfunc(u32 btf_id); | 494 | static bool is_callback_calling_kfunc(u32 btf_id); |
| @@ -497,22 +547,6 @@ bool bpf_is_may_goto_insn(struct bpf_insn *insn) · 52 unmodified lines | |||
| 497 | return insn->code == (BPF_JMP | BPF_JCOND) && insn->src_reg == BPF_MAY_GOTO; | 547 | return insn->code == (BPF_JMP | BPF_JCOND) && insn->src_reg == BPF_MAY_GOTO; |
| 498 | } | 548 | } |
| 499 | 549 | ||
| 500 | static bool helper_multiple_ref_obj_use(enum bpf_func_id func_id, | ||
| 501 | const struct bpf_map *map) | ||
| 502 | { | ||
| 503 | int ref_obj_uses = 0; | ||
| 504 | |||
| 505 | if (is_ptr_cast_function(func_id)) | ||
| 506 | ref_obj_uses++; | ||
| 507 | if (is_acquire_function(func_id, map)) | ||
| 508 | ref_obj_uses++; | ||
| 509 | if (is_dynptr_ref_function(func_id)) | ||
| 510 | ref_obj_uses++; | ||
| 511 | |||
| 512 | return ref_obj_uses > 1; | ||
| 513 | } | ||
| 514 | |||
| 515 | |||
| 516 | static bool is_spi_bounds_valid(struct bpf_func_state *state, int spi, int nr_slots) | 550 | static bool is_spi_bounds_valid(struct bpf_func_state *state, int spi, int nr_slots) |
| 517 | { | 551 | { |
| 518 | int allocated_slots = state->allocated_stack / BPF_REG_SIZE; | 552 | int allocated_slots = state->allocated_stack / BPF_REG_SIZE; |
| @@ -609,43 +643,44 @@ static enum bpf_type_flag get_dynptr_type_flag(enum bpf_dynptr_type type) · 90 unmodified lines | |||
| 609 | } | 643 | } |
| 610 | } | 644 | } |
| 611 | 645 | ||
| 612 | static bool dynptr_type_refcounted(enum bpf_dynptr_type type) | 646 | static bool dynptr_type_referenced(enum bpf_dynptr_type type) |
| 613 | { | 647 | { |
| 614 | return type == BPF_DYNPTR_TYPE_RINGBUF || type == BPF_DYNPTR_TYPE_FILE; | 648 | return type == BPF_DYNPTR_TYPE_RINGBUF || type == BPF_DYNPTR_TYPE_FILE; |
| 615 | } | 649 | } |
| 616 | 650 | ||
| 617 | static void __mark_dynptr_reg(struct bpf_reg_state *reg, | 651 | static void __mark_dynptr_reg(struct bpf_reg_state *reg, |
| 618 | enum bpf_dynptr_type type, | 652 | enum bpf_dynptr_type type, |
| 619 | bool first_slot, int dynptr_id); | 653 | bool first_slot, int id, int parent_id); |
| 620 | 654 | ||
| 621 | 655 | ||
| 622 | static void mark_dynptr_stack_regs(struct bpf_verifier_env *env, | 656 | static void mark_dynptr_stack_regs(struct bpf_verifier_env *env, |
| 623 | struct bpf_reg_state *sreg1, | 657 | struct bpf_reg_state *sreg1, |
| 624 | struct bpf_reg_state *sreg2, | 658 | struct bpf_reg_state *sreg2, |
| 625 | enum bpf_dynptr_type type) | 659 | enum bpf_dynptr_type type, int parent_id) |
| 626 | { | 660 | { |
| 627 | int id = ++env->id_gen; | 661 | int id = ++env->id_gen; |
| 628 | 662 | ||
| 629 | __mark_dynptr_reg(sreg1, type, true, id); | 663 | __mark_dynptr_reg(sreg1, type, true, id, parent_id); |
| 630 | __mark_dynptr_reg(sreg2, type, false, id); | 664 | __mark_dynptr_reg(sreg2, type, false, id, parent_id); |
| 631 | } | 665 | } |
| 632 | 666 | ||
| 633 | static void mark_dynptr_cb_reg(struct bpf_verifier_env *env, | 667 | static void mark_dynptr_cb_reg(struct bpf_verifier_env *env, |
| 634 | struct bpf_reg_state *reg, | 668 | struct bpf_reg_state *reg, |
| 635 | enum bpf_dynptr_type type) | 669 | enum bpf_dynptr_type type) |
| 636 | { | 670 | { |
| 637 | __mark_dynptr_reg(reg, type, true, ++env->id_gen); | 671 | __mark_dynptr_reg(reg, type, true, ++env->id_gen, 0); |
| 638 | } | 672 | } |
| 639 | 673 | ||
| 640 | static int destroy_if_dynptr_stack_slot(struct bpf_verifier_env *env, | 674 | static int destroy_if_dynptr_stack_slot(struct bpf_verifier_env *env, |
| 641 | struct bpf_func_state *state, int spi); | 675 | struct bpf_func_state *state, int spi); |
| 642 | 676 | ||
| 643 | static int mark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_state *reg, | 677 | static int mark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_state *reg, |
| 644 | enum bpf_arg_type arg_type, int insn_idx, int clone_ref_obj_id) | 678 | enum bpf_arg_type arg_type, int insn_idx, |
| 679 | struct ref_obj_desc *ref_obj, struct bpf_dynptr_desc *dynptr) | ||
| 645 | { | 680 | { |
| 646 | struct bpf_func_state *state = bpf_func(env, reg); | 681 | struct bpf_func_state *state = bpf_func(env, reg); |
| 682 | int spi, i, err, parent_id = 0; | ||
| 647 | enum bpf_dynptr_type type; | 683 | enum bpf_dynptr_type type; |
| 648 | int spi, i, err; | ||
| 649 | 684 | ||
| 650 | spi = dynptr_get_spi(env, reg); | 685 | spi = dynptr_get_spi(env, reg); |
| 651 | if (spi < 0) | 686 | if (spi < 0) |
| @@ -676,94 +711,69 @@ static int mark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_ · 24 unmodified lines | |||
| 676 | if (type == BPF_DYNPTR_TYPE_INVALID) | 711 | if (type == BPF_DYNPTR_TYPE_INVALID) |
| 677 | return -EINVAL; | 712 | return -EINVAL; |
| 678 | 713 | ||
| 679 | mark_dynptr_stack_regs(env, &state->stack[spi].spilled_ptr, | 714 | if (dynptr->type == BPF_DYNPTR_TYPE_INVALID) { /* dynptr constructors */ |
| 680 | &state->stack[spi - 1].spilled_ptr, type); | 715 | err = validate_ref_obj(env, ref_obj); |
| 716 | if (err) | ||
| 717 | return err; | ||
| 681 | 718 | ||
| 682 | if (dynptr_type_refcounted(type)) { | 719 | /* Track parent's id if the parent is a referenced object */ |
| 683 | /* The id is used to track proper releasing */ | 720 | parent_id = ref_obj->id; |
| 684 | int id; | ||
| 685 | 721 | ||
| 686 | if (clone_ref_obj_id) | 722 | if (dynptr_type_referenced(type)) { |
| 687 | id = clone_ref_obj_id; | 723 | int id; |
| 688 | else | ||
| 689 | id = acquire_reference(env, insn_idx); | ||
| 690 | 724 | ||
| 691 | if (id < 0) | 725 | /* |
| 692 | return id; | 726 | * Create an intermediate reference that tracks the referenced |
| 727 | * object for the referenced dynptr. Freeing a referenced dynptr | ||
| 728 | * through helpers/kfuncs will invalidate all clones. | ||
| 729 | */ | ||
| 730 | id = acquire_reference(env, insn_idx, parent_id); | ||
| 731 | if (id < 0) | ||
| 732 | return id; | ||
| 693 | 733 | ||
| 694 | state->stack[spi].spilled_ptr.ref_obj_id = id; | 734 | parent_id = id; |
| 695 | state->stack[spi - 1].spilled_ptr.ref_obj_id = id; | 735 | } |
| 736 | } else { /* bpf_dynptr_clone() */ | ||
| 737 | parent_id = dynptr->parent_id; | ||
| 696 | } | 738 | } |
| 697 | 739 | ||
| 740 | mark_dynptr_stack_regs(env, &state->stack[spi].spilled_ptr, | ||
| 741 | &state->stack[spi - 1].spilled_ptr, type, parent_id); | ||
| 742 | |||
| 698 | return 0; | 743 | return 0; |
| 699 | } | 744 | } |
| 700 | 745 | ||
| 701 | static void invalidate_dynptr(struct bpf_verifier_env *env, struct bpf_func_state *state, int spi) | 746 | static void invalidate_dynptr(struct bpf_verifier_env *env, struct bpf_stack_state *stack) |
| 702 | { | 747 | { |
| 703 | int i; | 748 | int i; |
| 704 | 749 | ||
| 705 | for (i = 0; i < BPF_REG_SIZE; i++) { | 750 | for (i = 0; i < BPF_REG_SIZE; i++) { |
| 706 | state->stack[spi].slot_type[i] = STACK_INVALID; | 751 | stack[0].slot_type[i] = STACK_INVALID; |
| 707 | state->stack[spi - 1].slot_type[i] = STACK_INVALID; | 752 | stack[1].slot_type[i] = STACK_INVALID; |
| 708 | } | 753 | } |
| 709 | 754 | ||
| 710 | bpf_mark_reg_not_init(env, &state->stack[spi].spilled_ptr); | 755 | bpf_mark_reg_not_init(env, &stack[0].spilled_ptr); |
| 711 | bpf_mark_reg_not_init(env, &state->stack[spi - 1].spilled_ptr); | 756 | bpf_mark_reg_not_init(env, &stack[1].spilled_ptr); |
| 712 | } | 757 | } |
| 713 | 758 | ||
| 714 | static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_state *reg) | 759 | static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_state *reg) |
| 715 | { | 760 | { |
| 716 | struct bpf_func_state *state = bpf_func(env, reg); | 761 | struct bpf_func_state *state = bpf_func(env, reg); |
| 717 | int spi, ref_obj_id, i; | 762 | int spi; |
| 718 | 763 | ||
| 719 | /* | ||
| 720 | * This can only be set for PTR_TO_STACK, as CONST_PTR_TO_DYNPTR cannot | ||
| 721 | * be released by any dynptr helper. Hence, unmark_stack_slots_dynptr | ||
| 722 | * is safe to do directly. | ||
| 723 | */ | ||
| 724 | if (reg->type == CONST_PTR_TO_DYNPTR) { | ||
| 725 | verifier_bug(env, "CONST_PTR_TO_DYNPTR cannot be released"); | ||
| 726 | return -EFAULT; | ||
| 727 | } | ||
| 728 | spi = dynptr_get_spi(env, reg); | 764 | spi = dynptr_get_spi(env, reg); |
| 729 | if (spi < 0) | 765 | if (spi < 0) |
| 730 | return spi; | 766 | return spi; |
| 731 | 767 | ||
| 732 | if (!dynptr_type_refcounted(state->stack[spi].spilled_ptr.dynptr.type)) { | 768 | /* |
| 733 | invalidate_dynptr(env, state, spi); | 769 | * For referenced dynptr, release the parent ref which cascades to |
| 734 | return 0; | 770 | * all clones and derived slices. For non-referenced dynptr, only |
| 735 | } | 771 | * the dynptr and slices derived from it will be invalidated. |
| 736 | |||
| 737 | ref_obj_id = state->stack[spi].spilled_ptr.ref_obj_id; | ||
| 738 | |||
| 739 | /* If the dynptr has a ref_obj_id, then we need to invalidate | ||
| 740 | * two things: | ||
| 741 | * | ||
| 742 | * 1) Any dynptrs with a matching ref_obj_id (clones) | ||
| 743 | * 2) Any slices derived from this dynptr. | ||
| 744 | */ | 772 | */ |
| 745 | 773 | reg = &state->stack[spi].spilled_ptr; | |
| 746 | /* Invalidate any slices associated with this dynptr */ | 774 | return release_reference(env, dynptr_type_referenced(reg->dynptr.type) |
| 747 | WARN_ON_ONCE(release_reference(env, ref_obj_id)); | 775 | ? reg->parent_id |
| 748 | 776 | : reg->id); | |
| 749 | /* Invalidate any dynptr clones */ | ||
| 750 | for (i = 1; i < state->allocated_stack / BPF_REG_SIZE; i++) { | ||
| 751 | if (state->stack[i].spilled_ptr.ref_obj_id != ref_obj_id) | ||
| 752 | continue; | ||
| 753 | |||
| 754 | /* it should always be the case that if the ref obj id | ||
| 755 | * matches then the stack slot also belongs to a | ||
| 756 | * dynptr | ||
| 757 | */ | ||
| 758 | if (state->stack[i].slot_type[0] != STACK_DYNPTR) { | ||
| 759 | verifier_bug(env, "misconfigured ref_obj_id"); | ||
| 760 | return -EFAULT; | ||
| 761 | } | ||
| 762 | if (state->stack[i].spilled_ptr.dynptr.first_slot) | ||
| 763 | invalidate_dynptr(env, state, i); | ||
| 764 | } | ||
| 765 | |||
| 766 | return 0; | ||
| 767 | } | 777 | } |
| 768 | 778 | ||
| 769 | static void __mark_reg_unknown(const struct bpf_verifier_env *env, | 779 | static void __mark_reg_unknown(const struct bpf_verifier_env *env, |
| @@ -777,12 +787,29 @@ static void mark_reg_invalid(const struct bpf_verifier_env *env, struct bpf_reg_ · 7 unmodified lines | |||
| 777 | __mark_reg_unknown(env, reg); | 787 | __mark_reg_unknown(env, reg); |
| 778 | } | 788 | } |
| 779 | 789 | ||
| 790 | static int dynptr_ref_cnt(struct bpf_verifier_env *env, int v_parent_id) | ||
| 791 | { | ||
| 792 | struct bpf_stack_state *stack; | ||
| 793 | struct bpf_func_state *state; | ||
| 794 | struct bpf_reg_state *reg; | ||
| 795 | int ref_cnt = 0; | ||
| 796 | |||
| 797 | bpf_for_each_reg_in_vstate_mask(env->cur_state, state, reg, stack, 1 << STACK_DYNPTR, ({ | ||
| 798 | if (!stack || stack->slot_type[0] != STACK_DYNPTR) | ||
| 799 | continue; | ||
| 800 | if (!stack->spilled_ptr.dynptr.first_slot) | ||
| 801 | continue; | ||
| 802 | if (stack->spilled_ptr.parent_id == v_parent_id) | ||
| 803 | ref_cnt++; | ||
| 804 | })); | ||
| 805 | |||
| 806 | return ref_cnt; | ||
| 807 | } | ||
| 808 | |||
| 780 | static int destroy_if_dynptr_stack_slot(struct bpf_verifier_env *env, | 809 | static int destroy_if_dynptr_stack_slot(struct bpf_verifier_env *env, |
| 781 | struct bpf_func_state *state, int spi) | 810 | struct bpf_func_state *state, int spi) |
| 782 | { | 811 | { |
| 783 | struct bpf_func_state *fstate; | 812 | int err = 0; |
| 784 | struct bpf_reg_state *dreg; | ||
| 785 | int i, dynptr_id; | ||
| 786 | 813 | ||
| 787 | /* We always ensure that STACK_DYNPTR is never set partially, | 814 | /* We always ensure that STACK_DYNPTR is never set partially, |
| 788 | * hence just checking for slot_type[0] is enough. This is | 815 | * hence just checking for slot_type[0] is enough. This is |
| @@ -796,56 +823,25 @@ static int destroy_if_dynptr_stack_slot(struct bpf_verifier_env *env, · 7 unmodified lines | |||
| 796 | if (!state->stack[spi].spilled_ptr.dynptr.first_slot) | 823 | if (!state->stack[spi].spilled_ptr.dynptr.first_slot) |
| 797 | spi = spi + 1; | 824 | spi = spi + 1; |
| 798 | 825 | ||
| 799 | if (dynptr_type_refcounted(state->stack[spi].spilled_ptr.dynptr.type)) { | 826 | /* |
| 800 | int ref_obj_id = state->stack[spi].spilled_ptr.ref_obj_id; | 827 | * A referenced dynptr can be overwritten only if there is at |
| 801 | int ref_cnt = 0; | 828 | * least one other dynptr sharing the same virtual ref parent, |
| 802 | 829 | * ensuring the reference can still be properly released. | |
| 803 | /* | 830 | */ |
| 804 | * A referenced dynptr can be overwritten only if there is at | 831 | if (dynptr_type_referenced(state->stack[spi].spilled_ptr.dynptr.type) && |
| 805 | * least one other dynptr sharing the same ref_obj_id, | 832 | dynptr_ref_cnt(env, state->stack[spi].spilled_ptr.parent_id) <= 1) { |
| 806 | * ensuring the reference can still be properly released. | 833 | verbose(env, "cannot overwrite referenced dynptr\n"); |
| 807 | */ | 834 | return -EINVAL; |
| 808 | for (i = 0; i < state->allocated_stack / BPF_REG_SIZE; i++) { | ||
| 809 | if (state->stack[i].slot_type[0] != STACK_DYNPTR) | ||
| 810 | continue; | ||
| 811 | if (!state->stack[i].spilled_ptr.dynptr.first_slot) | ||
| 812 | continue; | ||
| 813 | if (state->stack[i].spilled_ptr.ref_obj_id == ref_obj_id) | ||
| 814 | ref_cnt++; | ||
| 815 | } | ||
| 816 | |||
| 817 | if (ref_cnt <= 1) { | ||
| 818 | verbose(env, "cannot overwrite referenced dynptr\n"); | ||
| 819 | return -EINVAL; | ||
| 820 | } | ||
| 821 | } | 835 | } |
| 822 | 836 | ||
| 823 | mark_stack_slot_scratched(env, spi); | 837 | /* Invalidate the dynptr and any derived slices */ |
| 824 | mark_stack_slot_scratched(env, spi - 1); | 838 | err = release_reference(env, state->stack[spi].spilled_ptr.id); |
| 825 | 839 | if (!err) { | |
| 826 | /* Writing partially to one dynptr stack slot destroys both. */ | 840 | mark_stack_slot_scratched(env, spi); |
| 827 | for (i = 0; i < BPF_REG_SIZE; i++) { | 841 | mark_stack_slot_scratched(env, spi - 1); |
| 828 | state->stack[spi].slot_type[i] = STACK_INVALID; | ||
| 829 | state->stack[spi - 1].slot_type[i] = STACK_INVALID; | ||
| 830 | } | 842 | } |
| 831 | 843 | ||
| 832 | dynptr_id = state->stack[spi].spilled_ptr.id; | 844 | return err; |
| 833 | /* Invalidate any slices associated with this dynptr */ | ||
| 834 | bpf_for_each_reg_in_vstate(env->cur_state, fstate, dreg, ({ | ||
| 835 | /* Dynptr slices are only PTR_TO_MEM_OR_NULL and PTR_TO_MEM */ | ||
| 836 | if (dreg->type != (PTR_TO_MEM | PTR_MAYBE_NULL) && dreg->type != PTR_TO_MEM) | ||
| 837 | continue; | ||
| 838 | if (dreg->dynptr_id == dynptr_id) | ||
| 839 | mark_reg_invalid(env, dreg); | ||
| 840 | })); | ||
| 841 | |||
| 842 | /* Do not release reference state, we are destroying dynptr on stack, | ||
| 843 | * not using some helper to release it. Just reset register. | ||
| 844 | */ | ||
| 845 | bpf_mark_reg_not_init(env, &state->stack[spi].spilled_ptr); | ||
| 846 | bpf_mark_reg_not_init(env, &state->stack[spi - 1].spilled_ptr); | ||
| 847 | |||
| 848 | return 0; | ||
| 849 | } | 845 | } |
| 850 | 846 | ||
| 851 | static bool is_dynptr_reg_valid_uninit(struct bpf_verifier_env *env, struct bpf_reg_state *reg) | 847 | static bool is_dynptr_reg_valid_uninit(struct bpf_verifier_env *env, struct bpf_reg_state *reg) |
| @@ -945,7 +941,7 @@ static int mark_stack_slots_iter(struct bpf_verifier_env *env, · 93 unmodified lines | |||
| 945 | if (spi < 0) | 941 | if (spi < 0) |
| 946 | return spi; | 942 | return spi; |
| 947 | 943 | ||
| 948 | id = acquire_reference(env, insn_idx); | 944 | id = acquire_reference(env, insn_idx, 0); |
| 949 | if (id < 0) | 945 | if (id < 0) |
| 950 | return id; | 946 | return id; |
| 951 | 947 | ||
| @@ -961,7 +957,7 @@ static int mark_stack_slots_iter(struct bpf_verifier_env *env, · 9 unmodified lines | |||
| 961 | else | 957 | else |
| 962 | st->type |= PTR_UNTRUSTED; | 958 | st->type |= PTR_UNTRUSTED; |
| 963 | } | 959 | } |
| 964 | st->ref_obj_id = i == 0 ? id : 0; | 960 | st->id = i == 0 ? id : 0; |
| 965 | st->iter.btf = btf; | 961 | st->iter.btf = btf; |
| 966 | st->iter.btf_id = btf_id; | 962 | st->iter.btf_id = btf_id; |
| 967 | st->iter.state = BPF_ITER_STATE_ACTIVE; | 963 | st->iter.state = BPF_ITER_STATE_ACTIVE; |
| @@ -991,7 +987,7 @@ static int unmark_stack_slots_iter(struct bpf_verifier_env *env, · 23 unmodified lines | |||
| 991 | struct bpf_reg_state *st = &slot->spilled_ptr; | 987 | struct bpf_reg_state *st = &slot->spilled_ptr; |
| 992 | 988 | ||
| 993 | if (i == 0) | 989 | if (i == 0) |
| 994 | WARN_ON_ONCE(release_reference(env, st->ref_obj_id)); | 990 | WARN_ON_ONCE(release_reference(env, st->id)); |
| 995 | 991 | ||
| 996 | bpf_mark_reg_not_init(env, st); | 992 | bpf_mark_reg_not_init(env, st); |
| 997 | 993 | ||
| @@ -1047,10 +1043,10 @@ static int is_iter_reg_valid_init(struct bpf_verifier_env *env, struct bpf_reg_s · 49 unmodified lines | |||
| 1047 | 1043 | ||
| 1048 | if (st->type & PTR_UNTRUSTED) | 1044 | if (st->type & PTR_UNTRUSTED) |
| 1049 | return -EPROTO; | 1045 | return -EPROTO; |
| 1050 | /* only main (first) slot has ref_obj_id set */ | 1046 | /* only main (first) slot has id set */ |
| 1051 | if (i == 0 && !st->ref_obj_id) | 1047 | if (i == 0 && !st->id) |
| 1052 | return -EINVAL; | 1048 | return -EINVAL; |
| 1053 | if (i != 0 && st->ref_obj_id) | 1049 | if (i != 0 && st->id) |
| 1054 | return -EINVAL; | 1050 | return -EINVAL; |
| 1055 | if (st->iter.btf != btf || st->iter.btf_id != btf_id) | 1051 | if (st->iter.btf != btf || st->iter.btf_id != btf_id) |
| 1056 | return -EINVAL; | 1052 | return -EINVAL; |
| @@ -1089,7 +1085,7 @@ static int mark_stack_slot_irq_flag(struct bpf_verifier_env *env, · 32 unmodified lines | |||
| 1089 | 1085 | ||
| 1090 | __mark_reg_known_zero(st); | 1086 | __mark_reg_known_zero(st); |
| 1091 | st->type = PTR_TO_STACK; /* we don't have dedicated reg type */ | 1087 | st->type = PTR_TO_STACK; /* we don't have dedicated reg type */ |
| 1092 | st->ref_obj_id = id; | 1088 | st->id = id; |
| 1093 | st->irq.kfunc_class = kfunc_class; | 1089 | st->irq.kfunc_class = kfunc_class; |
| 1094 | 1090 | ||
| 1095 | for (i = 0; i < BPF_REG_SIZE; i++) | 1091 | for (i = 0; i < BPF_REG_SIZE; i++) |
| @@ -1123,7 +1119,7 @@ static int unmark_stack_slot_irq_flag(struct bpf_verifier_env *env, struct bpf_r · 27 unmodified lines | |||
| 1123 | return -EINVAL; | 1119 | return -EINVAL; |
| 1124 | } | 1120 | } |
| 1125 | 1121 | ||
| 1126 | err = release_irq_state(env->cur_state, st->ref_obj_id); | 1122 | err = release_irq_state(env->cur_state, st->id); |
| 1127 | WARN_ON_ONCE(err && err != -EACCES); | 1123 | WARN_ON_ONCE(err && err != -EACCES); |
| 1128 | if (err) { | 1124 | if (err) { |
| 1129 | int insn_idx = 0; | 1125 | int insn_idx = 0; |
| @@ -1187,7 +1183,7 @@ static int is_irq_flag_reg_valid_init(struct bpf_verifier_env *env, struct bpf_r · 57 unmodified lines | |||
| 1187 | slot = &state->stack[spi]; | 1183 | slot = &state->stack[spi]; |
| 1188 | st = &slot->spilled_ptr; | 1184 | st = &slot->spilled_ptr; |
| 1189 | 1185 | ||
| 1190 | if (!st->ref_obj_id) | 1186 | if (!st->id) |
| 1191 | return -EINVAL; | 1187 | return -EINVAL; |
| 1192 | 1188 | ||
| 1193 | for (i = 0; i < BPF_REG_SIZE; i++) | 1189 | for (i = 0; i < BPF_REG_SIZE; i++) |
| @@ -1339,6 +1335,18 @@ static int copy_stack_state(struct bpf_func_state *dst, const struct bpf_func_st · 145 unmodified lines | |||
| 1339 | return -ENOMEM; | 1335 | return -ENOMEM; |
| 1340 | 1336 | ||
| 1341 | dst->allocated_stack = src->allocated_stack; | 1337 | dst->allocated_stack = src->allocated_stack; |
| 1338 | |||
| 1339 | /* copy stack args state */ | ||
| 1340 | n = src->out_stack_arg_cnt; | ||
| 1341 | if (n) { | ||
| 1342 | dst->stack_arg_regs = copy_array(dst->stack_arg_regs, src->stack_arg_regs, n, | ||
| 1343 | sizeof(struct bpf_reg_state), | ||
| 1344 | GFP_KERNEL_ACCOUNT); | ||
| 1345 | if (!dst->stack_arg_regs) | ||
| 1346 | return -ENOMEM; | ||
| 1347 | } | ||
| 1348 | |||
| 1349 | dst->out_stack_arg_cnt = src->out_stack_arg_cnt; | ||
| 1342 | return 0; | 1350 | return 0; |
| 1343 | } | 1351 | } |
| 1344 | 1352 | ||
| @@ -1380,6 +1388,23 @@ static int grow_stack_state(struct bpf_verifier_env *env, struct bpf_func_state · 35 unmodified lines | |||
| 1380 | return 0; | 1388 | return 0; |
| 1381 | } | 1389 | } |
| 1382 | 1390 | ||
| 1391 | static int grow_stack_arg_slots(struct bpf_verifier_env *env, | ||
| 1392 | struct bpf_func_state *state, int cnt) | ||
| 1393 | { | ||
| 1394 | size_t old_n = state->out_stack_arg_cnt; | ||
| 1395 | |||
| 1396 | if (old_n >= cnt) | ||
| 1397 | return 0; | ||
| 1398 | |||
| 1399 | state->stack_arg_regs = realloc_array(state->stack_arg_regs, old_n, cnt, | ||
| 1400 | sizeof(struct bpf_reg_state)); | ||
| 1401 | if (!state->stack_arg_regs) | ||
| 1402 | return -ENOMEM; | ||
| 1403 | |||
| 1404 | state->out_stack_arg_cnt = cnt; | ||
| 1405 | return 0; | ||
| 1406 | } | ||
| 1407 | |||
| 1383 | /* Acquire a pointer id from the env and update the state->refs to include | 1408 | /* Acquire a pointer id from the env and update the state->refs to include |
| 1384 | * this new pointer reference. | 1409 | * this new pointer reference. |
| 1385 | * On success, returns a valid pointer id to associate with the register | 1410 | * On success, returns a valid pointer id to associate with the register |
| @@ -1399,7 +1424,7 @@ static struct bpf_reference_state *acquire_reference_state(struct bpf_verifier_e · 13 unmodified lines | |||
| 1399 | return &state->refs[new_ofs]; | 1424 | return &state->refs[new_ofs]; |
| 1400 | } | 1425 | } |
| 1401 | 1426 | ||
| 1402 | static int acquire_reference(struct bpf_verifier_env *env, int insn_idx) | 1427 | static int acquire_reference(struct bpf_verifier_env *env, int insn_idx, int parent_id) |
| 1403 | { | 1428 | { |
| 1404 | struct bpf_reference_state *s; | 1429 | struct bpf_reference_state *s; |
| 1405 | 1430 | ||
| @@ -1408,6 +1433,7 @@ static int acquire_reference(struct bpf_verifier_env *env, int insn_idx) · 2 unmodified lines | |||
| 1408 | return -ENOMEM; | 1433 | return -ENOMEM; |
| 1409 | s->type = REF_TYPE_PTR; | 1434 | s->type = REF_TYPE_PTR; |
| 1410 | s->id = ++env->id_gen; | 1435 | s->id = ++env->id_gen; |
| 1436 | s->parent_id = parent_id; | ||
| 1411 | return s->id; | 1437 | return s->id; |
| 1412 | } | 1438 | } |
| 1413 | 1439 | ||
| @@ -1464,17 +1490,25 @@ static void release_reference_state(struct bpf_verifier_state *state, int idx) · 50 unmodified lines | |||
| 1464 | return; | 1490 | return; |
| 1465 | } | 1491 | } |
| 1466 | 1492 | ||
| 1467 | static bool find_reference_state(struct bpf_verifier_state *state, int ptr_id) | 1493 | static bool find_reference_state(struct bpf_verifier_state *state, int id) |
| 1468 | { | 1494 | { |
| 1469 | int i; | 1495 | int i; |
| 1470 | 1496 | ||
| 1471 | for (i = 0; i < state->acquired_refs; i++) | 1497 | for (i = 0; i < state->acquired_refs; i++) { |
| 1472 | if (state->refs[i].id == ptr_id) | 1498 | if (state->refs[i].type != REF_TYPE_PTR) |
| 1499 | continue; | ||
| 1500 | if (state->refs[i].id == id) | ||
| 1473 | return true; | 1501 | return true; |
| 1502 | } | ||
| 1474 | 1503 | ||
| 1475 | return false; | 1504 | return false; |
| 1476 | } | 1505 | } |
| 1477 | 1506 | ||
| 1507 | static bool reg_is_referenced(struct bpf_verifier_env *env, const struct bpf_reg_state *reg) | ||
| 1508 | { | ||
| 1509 | return find_reference_state(env->cur_state, reg->id); | ||
| 1510 | } | ||
| 1511 | |||
| 1478 | static int release_lock_state(struct bpf_verifier_state *state, int type, int id, void *ptr) | 1512 | static int release_lock_state(struct bpf_verifier_state *state, int type, int id, void *ptr) |
| 1479 | { | 1513 | { |
| 1480 | void *prev_ptr = NULL; | 1514 | void *prev_ptr = NULL; |
| @@ -1542,6 +1576,7 @@ static void free_func_state(struct bpf_func_state *state) · 61 unmodified lines | |||
| 1542 | { | 1576 | { |
| 1543 | if (!state) | 1577 | if (!state) |
| 1544 | return; | 1578 | return; |
| 1579 | kfree(state->stack_arg_regs); | ||
| 1545 | kfree(state->stack); | 1580 | kfree(state->stack); |
| 1546 | kfree(state); | 1581 | kfree(state); |
| 1547 | } | 1582 | } |
| @@ -1750,6 +1785,22 @@ static struct bpf_verifier_state *push_stack(struct bpf_verifier_env *env, · 202 unmodified lines | |||
| 1750 | return &elem->st; | 1785 | return &elem->st; |
| 1751 | } | 1786 | } |
| 1752 | 1787 | ||
| 1788 | static const char *reg_arg_name(struct bpf_verifier_env *env, argno_t argno) | ||
| 1789 | { | ||
| 1790 | char *buf = env->tmp_arg_name; | ||
| 1791 | int len = sizeof(env->tmp_arg_name); | ||
| 1792 | int arg, regno = reg_from_argno(argno); | ||
| 1793 | |||
| 1794 | if (regno >= 0) { | ||
| 1795 | snprintf(buf, len, "R%d", regno); | ||
| 1796 | } else { | ||
| 1797 | arg = arg_from_argno(argno); | ||
| 1798 | snprintf(buf, len, "*(R11-%u)", (arg - MAX_BPF_FUNC_REG_ARGS) * BPF_REG_SIZE); | ||
| 1799 | } | ||
| 1800 | |||
| 1801 | return buf; | ||
| 1802 | } | ||
| 1803 | |||
| 1753 | static const int caller_saved[CALLER_SAVED_REGS] = { | 1804 | static const int caller_saved[CALLER_SAVED_REGS] = { |
| 1754 | BPF_REG_0, BPF_REG_1, BPF_REG_2, BPF_REG_3, BPF_REG_4, BPF_REG_5 | 1805 | BPF_REG_0, BPF_REG_1, BPF_REG_2, BPF_REG_3, BPF_REG_4, BPF_REG_5 |
| 1755 | }; | 1806 | }; |
| @@ -1758,15 +1809,8 @@ static const int caller_saved[CALLER_SAVED_REGS] = { · 2 unmodified lines | |||
| 1758 | static void ___mark_reg_known(struct bpf_reg_state *reg, u64 imm) | 1809 | static void ___mark_reg_known(struct bpf_reg_state *reg, u64 imm) |
| 1759 | { | 1810 | { |
| 1760 | reg->var_off = tnum_const(imm); | 1811 | reg->var_off = tnum_const(imm); |
| 1761 | reg->smin_value = (s64)imm; | 1812 | reg->r64 = cnum64_from_urange(imm, imm); |
| 1762 | reg->smax_value = (s64)imm; | 1813 | reg->r32 = cnum32_from_urange((u32)imm, (u32)imm); |
| 1763 | reg->umin_value = imm; | ||
| 1764 | reg->umax_value = imm; | ||
| 1765 | |||
| 1766 | reg->s32_min_value = (s32)imm; | ||
| 1767 | reg->s32_max_value = (s32)imm; | ||
| 1768 | reg->u32_min_value = (u32)imm; | ||
| 1769 | reg->u32_max_value = (u32)imm; | ||
| 1770 | } | 1814 | } |
| 1771 | 1815 | ||
| 1772 | /* Mark the unknown part of a register (variable offset or scalar value) as | 1816 | /* Mark the unknown part of a register (variable offset or scalar value) as |
| @@ -1778,17 +1822,14 @@ static void __mark_reg_known(struct bpf_reg_state *reg, u64 imm) · 5 unmodified lines | |||
| 1778 | memset(((u8 *)reg) + sizeof(reg->type), 0, | 1822 | memset(((u8 *)reg) + sizeof(reg->type), 0, |
| 1779 | offsetof(struct bpf_reg_state, var_off) - sizeof(reg->type)); | 1823 | offsetof(struct bpf_reg_state, var_off) - sizeof(reg->type)); |
| 1780 | reg->id = 0; | 1824 | reg->id = 0; |
| 1781 | reg->ref_obj_id = 0; | 1825 | reg->parent_id = 0; |
| 1782 | ___mark_reg_known(reg, imm); | 1826 | ___mark_reg_known(reg, imm); |
| 1783 | } | 1827 | } |
| 1784 | 1828 | ||
| 1785 | static void __mark_reg32_known(struct bpf_reg_state *reg, u64 imm) | 1829 | static void __mark_reg32_known(struct bpf_reg_state *reg, u64 imm) |
| 1786 | { | 1830 | { |
| 1787 | reg->var_off = tnum_const_subreg(reg->var_off, imm); | 1831 | reg->var_off = tnum_const_subreg(reg->var_off, imm); |
| 1788 | reg->s32_min_value = (s32)imm; | 1832 | reg->r32 = cnum32_from_urange((u32)imm, (u32)imm); |
| 1789 | reg->s32_max_value = (s32)imm; | ||
| 1790 | reg->u32_min_value = (u32)imm; | ||
| 1791 | reg->u32_max_value = (u32)imm; | ||
| 1792 | } | 1833 | } |
| 1793 | 1834 | ||
| 1794 | /* Mark the 'variable offset' part of a register as zero. This should be | 1835 | /* Mark the 'variable offset' part of a register as zero. This should be |
| @@ -1816,7 +1857,7 @@ static void mark_reg_known_zero(struct bpf_verifier_env *env, · 21 unmodified lines | |||
| 1816 | } | 1857 | } |
| 1817 | 1858 | ||
| 1818 | static void __mark_dynptr_reg(struct bpf_reg_state *reg, enum bpf_dynptr_type type, | 1859 | static void __mark_dynptr_reg(struct bpf_reg_state *reg, enum bpf_dynptr_type type, |
| 1819 | bool first_slot, int dynptr_id) | 1860 | bool first_slot, int id, int parent_id) |
| 1820 | { | 1861 | { |
| 1821 | /* reg->type has no meaning for STACK_DYNPTR, but when we set reg for | 1862 | /* reg->type has no meaning for STACK_DYNPTR, but when we set reg for |
| 1822 | * callback arguments, it does need to be CONST_PTR_TO_DYNPTR, so simply | 1863 | * callback arguments, it does need to be CONST_PTR_TO_DYNPTR, so simply |
| @@ -1825,7 +1866,8 @@ static void __mark_dynptr_reg(struct bpf_reg_state *reg, enum bpf_dynptr_type ty · 2 unmodified lines | |||
| 1825 | __mark_reg_known_zero(reg); | 1866 | __mark_reg_known_zero(reg); |
| 1826 | reg->type = CONST_PTR_TO_DYNPTR; | 1867 | reg->type = CONST_PTR_TO_DYNPTR; |
| 1827 | /* Give each dynptr a unique id to uniquely associate slices to it. */ | 1868 | /* Give each dynptr a unique id to uniquely associate slices to it. */ |
| 1828 | reg->id = dynptr_id; | 1869 | reg->id = id; |
| 1870 | reg->parent_id = parent_id; | ||
| 1829 | reg->dynptr.type = type; | 1871 | reg->dynptr.type = type; |
| 1830 | reg->dynptr.first_slot = first_slot; | 1872 | reg->dynptr.first_slot = first_slot; |
| 1831 | } | 1873 | } |
| @@ -1899,34 +1941,21 @@ static bool reg_is_init_pkt_pointer(const struct bpf_reg_state *reg, · 67 unmodified lines | |||
| 1899 | tnum_equals_const(reg->var_off, 0); | 1941 | tnum_equals_const(reg->var_off, 0); |
| 1900 | } | 1942 | } |
| 1901 | 1943 | ||
| 1902 | /* Reset the min/max bounds of a register */ | 1944 | static void __mark_reg32_unbounded(struct bpf_reg_state *reg) |
| 1903 | static void __mark_reg_unbounded(struct bpf_reg_state *reg) | ||
| 1904 | { | 1945 | { |
| 1905 | reg->smin_value = S64_MIN; | 1946 | reg->r32 = CNUM32_UNBOUNDED; |
| 1906 | reg->smax_value = S64_MAX; | ||
| 1907 | reg->umin_value = 0; | ||
| 1908 | reg->umax_value = U64_MAX; | ||
| 1909 | |||
| 1910 | reg->s32_min_value = S32_MIN; | ||
| 1911 | reg->s32_max_value = S32_MAX; | ||
| 1912 | reg->u32_min_value = 0; | ||
| 1913 | reg->u32_max_value = U32_MAX; | ||
| 1914 | } | 1947 | } |
| 1915 | 1948 | ||
| 1916 | static void __mark_reg64_unbounded(struct bpf_reg_state *reg) | 1949 | static void __mark_reg64_unbounded(struct bpf_reg_state *reg) |
| 1917 | { | 1950 | { |
| 1918 | reg->smin_value = S64_MIN; | 1951 | reg->r64 = CNUM64_UNBOUNDED; |
| 1919 | reg->smax_value = S64_MAX; | ||
| 1920 | reg->umin_value = 0; | ||
| 1921 | reg->umax_value = U64_MAX; | ||
| 1922 | } | 1952 | } |
| 1923 | 1953 | ||
| 1924 | static void __mark_reg32_unbounded(struct bpf_reg_state *reg) | 1954 | /* Reset the min/max bounds of a register */ |
| 1955 | static void __mark_reg_unbounded(struct bpf_reg_state *reg) | ||
| 1925 | { | 1956 | { |
| 1926 | reg->s32_min_value = S32_MIN; | 1957 | __mark_reg64_unbounded(reg); |
| 1927 | reg->s32_max_value = S32_MAX; | 1958 | __mark_reg32_unbounded(reg); |
| 1928 | reg->u32_min_value = 0; | ||
| 1929 | reg->u32_max_value = U32_MAX; | ||
| 1930 | } | 1959 | } |
| 1931 | 1960 | ||
| 1932 | static void reset_reg64_and_tnum(struct bpf_reg_state *reg) | 1961 | static void reset_reg64_and_tnum(struct bpf_reg_state *reg) |
| @@ -1941,19 +1970,32 @@ static void reset_reg32_and_tnum(struct bpf_reg_state *reg) · 8 unmodified lines | |||
| 1941 | reg->var_off = tnum_unknown; | 1970 | reg->var_off = tnum_unknown; |
| 1942 | } | 1971 | } |
| 1943 | 1972 | ||
| 1944 | static void __update_reg32_bounds(struct bpf_reg_state *reg) | 1973 | static struct cnum32 cnum32_from_tnum(struct tnum tnum) |
| 1945 | { | 1974 | { |
| 1946 | struct tnum var32_off = tnum_subreg(reg->var_off); | 1975 | tnum = tnum_subreg(tnum); |
| 1976 | if ((tnum.mask & S32_MIN) || (tnum.value & S32_MIN)) | ||
| 1977 | /* min signed is max(sign bit) | min(other bits) */ | ||
| 1978 | /* max signed is min(sign bit) | max(other bits) */ | ||
| 1979 | return cnum32_from_srange(tnum.value | (tnum.mask & S32_MIN), | ||
| 1980 | tnum.value | (tnum.mask & S32_MAX)); | ||
| 1981 | else | ||
| 1982 | return cnum32_from_urange(tnum.value, (tnum.value | tnum.mask)); | ||
| 1983 | } | ||
| 1947 | 1984 | ||
| 1948 | /* min signed is max(sign bit) | min(other bits) */ | 1985 | static struct cnum64 cnum64_from_tnum(struct tnum tnum) |
| 1949 | reg->s32_min_value = max_t(s32, reg->s32_min_value, | 1986 | { |
| 1950 | var32_off.value | (var32_off.mask & S32_MIN)); | 1987 | if ((tnum.mask & S64_MIN) || (tnum.value & S64_MIN)) |
| 1951 | /* max signed is min(sign bit) | max(other bits) */ | 1988 | /* min signed is max(sign bit) | min(other bits) */ |
| 1952 | reg->s32_max_value = min_t(s32, reg->s32_max_value, | 1989 | /* max signed is min(sign bit) | max(other bits) */ |
| 1953 | var32_off.value | (var32_off.mask & S32_MAX)); | 1990 | return cnum64_from_srange(tnum.value | (tnum.mask & S64_MIN), |
| 1954 | reg->u32_min_value = max_t(u32, reg->u32_min_value, (u32)var32_off.value); | 1991 | tnum.value | (tnum.mask & S64_MAX)); |
| 1955 | reg->u32_max_value = min(reg->u32_max_value, | 1992 | else |
| 1956 | (u32)(var32_off.value | var32_off.mask)); | 1993 | return cnum64_from_urange(tnum.value, (tnum.value | tnum.mask)); |
| 1994 | } | ||
| 1995 | |||
| 1996 | static void __update_reg32_bounds(struct bpf_reg_state *reg) | ||
| 1997 | { | ||
| 1998 | cnum32_intersect_with(®->r32, cnum32_from_tnum(reg->var_off)); | ||
| 1957 | } | 1999 | } |
| 1958 | 2000 | ||
| 1959 | static void __update_reg64_bounds(struct bpf_reg_state *reg) | 2001 | static void __update_reg64_bounds(struct bpf_reg_state *reg) |
| @@ -1961,26 +2003,18 @@ static void __update_reg64_bounds(struct bpf_reg_state *reg) · 1 unmodified line | |||
| 1961 | u64 tnum_next, tmax; | 2003 | u64 tnum_next, tmax; |
| 1962 | bool umin_in_tnum; | 2004 | bool umin_in_tnum; |
| 1963 | 2005 | ||
| 1964 | /* min signed is max(sign bit) | min(other bits) */ | 2006 | cnum64_intersect_with(®->r64, cnum64_from_tnum(reg->var_off)); |
| 1965 | reg->smin_value = max_t(s64, reg->smin_value, | ||
| 1966 | reg->var_off.value | (reg->var_off.mask & S64_MIN)); | ||
| 1967 | /* max signed is min(sign bit) | max(other bits) */ | ||
| 1968 | reg->smax_value = min_t(s64, reg->smax_value, | ||
| 1969 | reg->var_off.value | (reg->var_off.mask & S64_MAX)); | ||
| 1970 | reg->umin_value = max(reg->umin_value, reg->var_off.value); | ||
| 1971 | reg->umax_value = min(reg->umax_value, | ||
| 1972 | reg->var_off.value | reg->var_off.mask); | ||
| 1973 | 2007 | ||
| 1974 | /* Check if u64 and tnum overlap in a single value */ | 2008 | /* Check if u64 and tnum overlap in a single value */ |
| 1975 | tnum_next = tnum_step(reg->var_off, reg->umin_value); | 2009 | tnum_next = tnum_step(reg->var_off, reg_umin(reg)); |
| 1976 | umin_in_tnum = (reg->umin_value & ~reg->var_off.mask) == reg->var_off.value; | 2010 | umin_in_tnum = (reg_umin(reg) & ~reg->var_off.mask) == reg->var_off.value; |
| 1977 | tmax = reg->var_off.value | reg->var_off.mask; | 2011 | tmax = reg->var_off.value | reg->var_off.mask; |
| 1978 | if (umin_in_tnum && tnum_next > reg->umax_value) { | 2012 | if (umin_in_tnum && tnum_next > reg_umax(reg)) { |
| 1979 | /* The u64 range and the tnum only overlap in umin. | 2013 | /* The u64 range and the tnum only overlap in umin. |
| 1980 | * u64: ---[xxxxxx]----- | 2014 | * u64: ---[xxxxxx]----- |
| 1981 | * tnum: --xx----------x- | 2015 | * tnum: --xx----------x- |
| 1982 | */ | 2016 | */ |
| 1983 | ___mark_reg_known(reg, reg->umin_value); | 2017 | ___mark_reg_known(reg, reg_umin(reg)); |
| 1984 | } else if (!umin_in_tnum && tnum_next == tmax) { | 2018 | } else if (!umin_in_tnum && tnum_next == tmax) { |
| 1985 | /* The u64 range and the tnum only overlap in the maximum value | 2019 | /* The u64 range and the tnum only overlap in the maximum value |
| 1986 | * represented by the tnum, called tmax. | 2020 | * represented by the tnum, called tmax. |
| @@ -1988,8 +2022,8 @@ static void __update_reg64_bounds(struct bpf_reg_state *reg) · 1 unmodified line | |||
| 1988 | * tnum: xx-----x-------- | 2022 | * tnum: xx-----x-------- |
| 1989 | */ | 2023 | */ |
| 1990 | ___mark_reg_known(reg, tmax); | 2024 | ___mark_reg_known(reg, tmax); |
| 1991 | } else if (!umin_in_tnum && tnum_next <= reg->umax_value && | 2025 | } else if (!umin_in_tnum && tnum_next <= reg_umax(reg) && |
| 1992 | tnum_step(reg->var_off, tnum_next) > reg->umax_value) { | 2026 | tnum_step(reg->var_off, tnum_next) > reg_umax(reg)) { |
| 1993 | /* The u64 range and the tnum only overlap in between umin | 2027 | /* The u64 range and the tnum only overlap in between umin |
| 1994 | * (excluded) and umax. | 2028 | * (excluded) and umax. |
| 1995 | * u64: ---[xxxxxx]----- | 2029 | * u64: ---[xxxxxx]----- |
| @@ -2005,329 +2039,19 @@ static void __update_reg_bounds(struct bpf_reg_state *reg) · 9 unmodified lines | |||
| 2005 | __update_reg64_bounds(reg); | 2039 | __update_reg64_bounds(reg); |
| 2006 | } | 2040 | } |
| 2007 | 2041 | ||
| 2008 | /* Uses signed min/max values to inform unsigned, and vice-versa */ | ||
| 2009 | static void deduce_bounds_32_from_64(struct bpf_reg_state *reg) | 2042 | static void deduce_bounds_32_from_64(struct bpf_reg_state *reg) |
| 2010 | { | 2043 | { |
| 2011 | /* If upper 32 bits of u64/s64 range don't change, we can use lower 32 | 2044 | cnum32_intersect_with(®->r32, cnum32_from_cnum64(reg->r64)); |
| 2012 | * bits to improve our u32/s32 boundaries. | ||
| 2013 | * | ||
| 2014 | * E.g., the case where we have upper 32 bits as zero ([10, 20] in | ||
| 2015 | * u64) is pretty trivial, it's obvious that in u32 we'll also have | ||
| 2016 | * [10, 20] range. But this property holds for any 64-bit range as | ||
| 2017 | * long as upper 32 bits in that entire range of values stay the same. | ||
| 2018 | * | ||
| 2019 | * E.g., u64 range [0x10000000A, 0x10000000F] ([4294967306, 4294967311] | ||
| 2020 | * in decimal) has the same upper 32 bits throughout all the values in | ||
| 2021 | * that range. As such, lower 32 bits form a valid [0xA, 0xF] ([10, 15]) | ||
| 2022 | * range. | ||
| 2023 | * | ||
| 2024 | * Note also, that [0xA, 0xF] is a valid range both in u32 and in s32, | ||
| 2025 | * following the rules outlined below about u64/s64 correspondence | ||
| 2026 | * (which equally applies to u32 vs s32 correspondence). In general it | ||
| 2027 | * depends on actual hexadecimal values of 32-bit range. They can form | ||
| 2028 | * only valid u32, or only valid s32 ranges in some cases. | ||
| 2029 | * | ||
| 2030 | * So we use all these insights to derive bounds for subregisters here. | ||
| 2031 | */ | ||
| 2032 | if ((reg->umin_value >> 32) == (reg->umax_value >> 32)) { | ||
| 2033 | /* u64 to u32 casting preserves validity of low 32 bits as | ||
| 2034 | * a range, if upper 32 bits are the same | ||
| 2035 | */ | ||
| 2036 | reg->u32_min_value = max_t(u32, reg->u32_min_value, (u32)reg->umin_value); | ||
| 2037 | reg->u32_max_value = min_t(u32, reg->u32_max_value, (u32)reg->umax_value); | ||
| 2038 | |||
| 2039 | if ((s32)reg->umin_value <= (s32)reg->umax_value) { | ||
| 2040 | reg->s32_min_value = max_t(s32, reg->s32_min_value, (s32)reg->umin_value); | ||
| 2041 | reg->s32_max_value = min_t(s32, reg->s32_max_value, (s32)reg->umax_value); | ||
| 2042 | } | ||
| 2043 | } | ||
| 2044 | if ((reg->smin_value >> 32) == (reg->smax_value >> 32)) { | ||
| 2045 | /* low 32 bits should form a proper u32 range */ | ||
| 2046 | if ((u32)reg->smin_value <= (u32)reg->smax_value) { | ||
| 2047 | reg->u32_min_value = max_t(u32, reg->u32_min_value, (u32)reg->smin_value); | ||
| 2048 | reg->u32_max_value = min_t(u32, reg->u32_max_value, (u32)reg->smax_value); | ||
| 2049 | } | ||
| 2050 | /* low 32 bits should form a proper s32 range */ | ||
| 2051 | if ((s32)reg->smin_value <= (s32)reg->smax_value) { | ||
| 2052 | reg->s32_min_value = max_t(s32, reg->s32_min_value, (s32)reg->smin_value); | ||
| 2053 | reg->s32_max_value = min_t(s32, reg->s32_max_value, (s32)reg->smax_value); | ||
| 2054 | } | ||
| 2055 | } | ||
| 2056 | /* Special case where upper bits form a small sequence of two | ||
| 2057 | * sequential numbers (in 32-bit unsigned space, so 0xffffffff to | ||
| 2058 | * 0x00000000 is also valid), while lower bits form a proper s32 range | ||
| 2059 | * going from negative numbers to positive numbers. E.g., let's say we | ||
| 2060 | * have s64 range [-1, 1] ([0xffffffffffffffff, 0x0000000000000001]). | ||
| 2061 | * Possible s64 values are {-1, 0, 1} ({0xffffffffffffffff, | ||
| 2062 | * 0x0000000000000000, 0x00000000000001}). Ignoring upper 32 bits, | ||
| 2063 | * we still get a valid s32 range [-1, 1] ([0xffffffff, 0x00000001]). | ||
| 2064 | * Note that it doesn't have to be 0xffffffff going to 0x00000000 in | ||
| 2065 | * upper 32 bits. As a random example, s64 range | ||
| 2066 | * [0xfffffff0fffffff0; 0xfffffff100000010], forms a valid s32 range | ||
| 2067 | * [-16, 16] ([0xfffffff0; 0x00000010]) in its 32 bit subregister. | ||
| 2068 | */ | ||
| 2069 | if ((u32)(reg->umin_value >> 32) + 1 == (u32)(reg->umax_value >> 32) && | ||
| 2070 | (s32)reg->umin_value < 0 && (s32)reg->umax_value >= 0) { | ||
| 2071 | reg->s32_min_value = max_t(s32, reg->s32_min_value, (s32)reg->umin_value); | ||
| 2072 | reg->s32_max_value = min_t(s32, reg->s32_max_value, (s32)reg->umax_value); | ||
| 2073 | } | ||
| 2074 | if ((u32)(reg->smin_value >> 32) + 1 == (u32)(reg->smax_value >> 32) && | ||
| 2075 | (s32)reg->smin_value < 0 && (s32)reg->smax_value >= 0) { | ||
| 2076 | reg->s32_min_value = max_t(s32, reg->s32_min_value, (s32)reg->smin_value); | ||
| 2077 | reg->s32_max_value = min_t(s32, reg->s32_max_value, (s32)reg->smax_value); | ||
| 2078 | } | ||
| 2079 | } | ||
| 2080 | |||
| 2081 | static void deduce_bounds_32_from_32(struct bpf_reg_state *reg) | ||
| 2082 | { | ||
| 2083 | /* if u32 range forms a valid s32 range (due to matching sign bit), | ||
| 2084 | * try to learn from that | ||
| 2085 | */ | ||
| 2086 | if ((s32)reg->u32_min_value <= (s32)reg->u32_max_value) { | ||
| 2087 | reg->s32_min_value = max_t(s32, reg->s32_min_value, reg->u32_min_value); | ||
| 2088 | reg->s32_max_value = min_t(s32, reg->s32_max_value, reg->u32_max_value); | ||
| 2089 | } | ||
| 2090 | /* If we cannot cross the sign boundary, then signed and unsigned bounds | ||
| 2091 | * are the same, so combine. This works even in the negative case, e.g. | ||
| 2092 | * -3 s<= x s<= -1 implies 0xf...fd u<= x u<= 0xf...ff. | ||
| 2093 | */ | ||
| 2094 | if ((u32)reg->s32_min_value <= (u32)reg->s32_max_value) { | ||
| 2095 | reg->u32_min_value = max_t(u32, reg->s32_min_value, reg->u32_min_value); | ||
| 2096 | reg->u32_max_value = min_t(u32, reg->s32_max_value, reg->u32_max_value); | ||
| 2097 | } else { | ||
| 2098 | if (reg->u32_max_value < (u32)reg->s32_min_value) { | ||
| 2099 | /* See __reg64_deduce_bounds() for detailed explanation. | ||
| 2100 | * Refine ranges in the following situation: | ||
| 2101 | * | ||
| 2102 | * 0 U32_MAX | ||
| 2103 | * | [xxxxxxxxxxxxxx u32 range xxxxxxxxxxxxxx] | | ||
| 2104 | * |----------------------------|----------------------------| | ||
| 2105 | * |xxxxx s32 range xxxxxxxxx] [xxxxxxx| | ||
| 2106 | * 0 S32_MAX S32_MIN -1 | ||
| 2107 | */ | ||
| 2108 | reg->s32_min_value = (s32)reg->u32_min_value; | ||
| 2109 | reg->u32_max_value = min_t(u32, reg->u32_max_value, reg->s32_max_value); | ||
| 2110 | } else if ((u32)reg->s32_max_value < reg->u32_min_value) { | ||
| 2111 | /* | ||
| 2112 | * 0 U32_MAX | ||
| 2113 | * | [xxxxxxxxxxxxxx u32 range xxxxxxxxxxxxxx] | | ||
| 2114 | * |----------------------------|----------------------------| | ||
| 2115 | * |xxxxxxxxx] [xxxxxxxxxxxx s32 range | | ||
| 2116 | * 0 S32_MAX S32_MIN -1 | ||
| 2117 | */ | ||
| 2118 | reg->s32_max_value = (s32)reg->u32_max_value; | ||
| 2119 | reg->u32_min_value = max_t(u32, reg->u32_min_value, reg->s32_min_value); | ||
| 2120 | } | ||
| 2121 | } | ||
| 2122 | } | ||
| 2123 | |||
| 2124 | static void deduce_bounds_64_from_64(struct bpf_reg_state *reg) | ||
| 2125 | { | ||
| 2126 | /* If u64 range forms a valid s64 range (due to matching sign bit), | ||
| 2127 | * try to learn from that. Let's do a bit of ASCII art to see when | ||
| 2128 | * this is happening. Let's take u64 range first: | ||
| 2129 | * | ||
| 2130 | * 0 0x7fffffffffffffff 0x8000000000000000 U64_MAX | ||
| 2131 | * |-------------------------------|--------------------------------| | ||
| 2132 | * | ||
| 2133 | * Valid u64 range is formed when umin and umax are anywhere in the | ||
| 2134 | * range [0, U64_MAX], and umin <= umax. u64 case is simple and | ||
| 2135 | * straightforward. Let's see how s64 range maps onto the same range | ||
| 2136 | * of values, annotated below the line for comparison: | ||
| 2137 | * | ||
| 2138 | * 0 0x7fffffffffffffff 0x8000000000000000 U64_MAX | ||
| 2139 | * |-------------------------------|--------------------------------| | ||
| 2140 | * 0 S64_MAX S64_MIN -1 | ||
| 2141 | * | ||
| 2142 | * So s64 values basically start in the middle and they are logically | ||
| 2143 | * contiguous to the right of it, wrapping around from -1 to 0, and | ||
| 2144 | * then finishing as S64_MAX (0x7fffffffffffffff) right before | ||
| 2145 | * S64_MIN. We can try drawing the continuity of u64 vs s64 values | ||
| 2146 | * more visually as mapped to sign-agnostic range of hex values. | ||
| 2147 | * | ||
| 2148 | * u64 start u64 end | ||
| 2149 | * _______________________________________________________________ | ||
| 2150 | * / \ | ||
| 2151 | * 0 0x7fffffffffffffff 0x8000000000000000 U64_MAX | ||
| 2152 | * |-------------------------------|--------------------------------| | ||
| 2153 | * 0 S64_MAX S64_MIN -1 | ||
| 2154 | * / \ | ||
| 2155 | * >------------------------------ -------------------------------> | ||
| 2156 | * s64 continues... s64 end s64 start s64 "midpoint" | ||
| 2157 | * | ||
| 2158 | * What this means is that, in general, we can't always derive | ||
| 2159 | * something new about u64 from any random s64 range, and vice versa. | ||
| 2160 | * | ||
| 2161 | * But we can do that in two particular cases. One is when entire | ||
| 2162 | * u64/s64 range is *entirely* contained within left half of the above | ||
| 2163 | * diagram or when it is *entirely* contained in the right half. I.e.: | ||
| 2164 | * | ||
| 2165 | * |-------------------------------|--------------------------------| | ||
| 2166 | * ^ ^ ^ ^ | ||
| 2167 | * A B C D | ||
| 2168 | * | ||
| 2169 | * [A, B] and [C, D] are contained entirely in their respective halves | ||
| 2170 | * and form valid contiguous ranges as both u64 and s64 values. [A, B] | ||
| 2171 | * will be non-negative both as u64 and s64 (and in fact it will be | ||
| 2172 | * identical ranges no matter the signedness). [C, D] treated as s64 | ||
| 2173 | * will be a range of negative values, while in u64 it will be | ||
| 2174 | * non-negative range of values larger than 0x8000000000000000. | ||
| 2175 | * | ||
| 2176 | * Now, any other range here can't be represented in both u64 and s64 | ||
| 2177 | * simultaneously. E.g., [A, C], [A, D], [B, C], [B, D] are valid | ||
| 2178 | * contiguous u64 ranges, but they are discontinuous in s64. [B, C] | ||
| 2179 | * in s64 would be properly presented as [S64_MIN, C] and [B, S64_MAX], | ||
| 2180 | * for example. Similarly, valid s64 range [D, A] (going from negative | ||
| 2181 | * to positive values), would be two separate [D, U64_MAX] and [0, A] | ||
| 2182 | * ranges as u64. Currently reg_state can't represent two segments per | ||
| 2183 | * numeric domain, so in such situations we can only derive maximal | ||
| 2184 | * possible range ([0, U64_MAX] for u64, and [S64_MIN, S64_MAX] for s64). | ||
| 2185 | * | ||
| 2186 | * So we use these facts to derive umin/umax from smin/smax and vice | ||
| 2187 | * versa only if they stay within the same "half". This is equivalent | ||
| 2188 | * to checking sign bit: lower half will have sign bit as zero, upper | ||
| 2189 | * half have sign bit 1. Below in code we simplify this by just | ||
| 2190 | * casting umin/umax as smin/smax and checking if they form valid | ||
| 2191 | * range, and vice versa. Those are equivalent checks. | ||
| 2192 | */ | ||
| 2193 | if ((s64)reg->umin_value <= (s64)reg->umax_value) { | ||
| 2194 | reg->smin_value = max_t(s64, reg->smin_value, reg->umin_value); | ||
| 2195 | reg->smax_value = min_t(s64, reg->smax_value, reg->umax_value); | ||
| 2196 | } | ||
| 2197 | /* If we cannot cross the sign boundary, then signed and unsigned bounds | ||
| 2198 | * are the same, so combine. This works even in the negative case, e.g. | ||
| 2199 | * -3 s<= x s<= -1 implies 0xf...fd u<= x u<= 0xf...ff. | ||
| 2200 | */ | ||
| 2201 | if ((u64)reg->smin_value <= (u64)reg->smax_value) { | ||
| 2202 | reg->umin_value = max_t(u64, reg->smin_value, reg->umin_value); | ||
| 2203 | reg->umax_value = min_t(u64, reg->smax_value, reg->umax_value); | ||
| 2204 | } else { | ||
| 2205 | /* If the s64 range crosses the sign boundary, then it's split | ||
| 2206 | * between the beginning and end of the U64 domain. In that | ||
| 2207 | * case, we can derive new bounds if the u64 range overlaps | ||
| 2208 | * with only one end of the s64 range. | ||
| 2209 | * | ||
| 2210 | * In the following example, the u64 range overlaps only with | ||
| 2211 | * positive portion of the s64 range. | ||
| 2212 | * | ||
| 2213 | * 0 U64_MAX | ||
| 2214 | * | [xxxxxxxxxxxxxx u64 range xxxxxxxxxxxxxx] | | ||
| 2215 | * |----------------------------|----------------------------| | ||
| 2216 | * |xxxxx s64 range xxxxxxxxx] [xxxxxxx| | ||
| 2217 | * 0 S64_MAX S64_MIN -1 | ||
| 2218 | * | ||
| 2219 | * We can thus derive the following new s64 and u64 ranges. | ||
| 2220 | * | ||
| 2221 | * 0 U64_MAX | ||
| 2222 | * | [xxxxxx u64 range xxxxx] | | ||
| 2223 | * |----------------------------|----------------------------| | ||
| 2224 | * | [xxxxxx s64 range xxxxx] | | ||
| 2225 | * 0 S64_MAX S64_MIN -1 | ||
| 2226 | * | ||
| 2227 | * If they overlap in two places, we can't derive anything | ||
| 2228 | * because reg_state can't represent two ranges per numeric | ||
| 2229 | * domain. | ||
| 2230 | * | ||
| 2231 | * 0 U64_MAX | ||
| 2232 | * | [xxxxxxxxxxxxxxxxx u64 range xxxxxxxxxxxxxxxxx] | | ||
| 2233 | * |----------------------------|----------------------------| | ||
| 2234 | * |xxxxx s64 range xxxxxxxxx] [xxxxxxxxxx| | ||
| 2235 | * 0 S64_MAX S64_MIN -1 | ||
| 2236 | * | ||
| 2237 | * The first condition below corresponds to the first diagram | ||
| 2238 | * above. | ||
| 2239 | */ | ||
| 2240 | if (reg->umax_value < (u64)reg->smin_value) { | ||
| 2241 | reg->smin_value = (s64)reg->umin_value; | ||
| 2242 | reg->umax_value = min_t(u64, reg->umax_value, reg->smax_value); | ||
| 2243 | } else if ((u64)reg->smax_value < reg->umin_value) { | ||
| 2244 | /* This second condition considers the case where the u64 range | ||
| 2245 | * overlaps with the negative portion of the s64 range: | ||
| 2246 | * | ||
| 2247 | * 0 U64_MAX | ||
| 2248 | * | [xxxxxxxxxxxxxx u64 range xxxxxxxxxxxxxx] | | ||
| 2249 | * |----------------------------|----------------------------| | ||
| 2250 | * |xxxxxxxxx] [xxxxxxxxxxxx s64 range | | ||
| 2251 | * 0 S64_MAX S64_MIN -1 | ||
| 2252 | */ | ||
| 2253 | reg->smax_value = (s64)reg->umax_value; | ||
| 2254 | reg->umin_value = max_t(u64, reg->umin_value, reg->smin_value); | ||
| 2255 | } | ||
| 2256 | } | ||
| 2257 | } | 2045 | } |
| 2258 | 2046 | ||
| 2259 | static void deduce_bounds_64_from_32(struct bpf_reg_state *reg) | 2047 | static void deduce_bounds_64_from_32(struct bpf_reg_state *reg) |
| 2260 | { | 2048 | { |
| 2261 | /* Try to tighten 64-bit bounds from 32-bit knowledge, using 32-bit | 2049 | reg->r64 = cnum64_cnum32_intersect(reg->r64, reg->r32); |
| 2262 | * values on both sides of 64-bit range in hope to have tighter range. | ||
| 2263 | * E.g., if r1 is [0x1'00000000, 0x3'80000000], and we learn from | ||
| 2264 | * 32-bit signed > 0 operation that s32 bounds are now [1; 0x7fffffff]. | ||
| 2265 | * With this, we can substitute 1 as low 32-bits of _low_ 64-bit bound | ||
| 2266 | * (0x100000000 -> 0x100000001) and 0x7fffffff as low 32-bits of | ||
| 2267 | * _high_ 64-bit bound (0x380000000 -> 0x37fffffff) and arrive at a | ||
| 2268 | * better overall bounds for r1 as [0x1'000000001; 0x3'7fffffff]. | ||
| 2269 | * We just need to make sure that derived bounds we are intersecting | ||
| 2270 | * with are well-formed ranges in respective s64 or u64 domain, just | ||
| 2271 | * like we do with similar kinds of 32-to-64 or 64-to-32 adjustments. | ||
| 2272 | */ | ||
| 2273 | __u64 new_umin, new_umax; | ||
| 2274 | __s64 new_smin, new_smax; | ||
| 2275 | |||
| 2276 | /* u32 -> u64 tightening, it's always well-formed */ | ||
| 2277 | new_umin = (reg->umin_value & ~0xffffffffULL) | reg->u32_min_value; | ||
| 2278 | new_umax = (reg->umax_value & ~0xffffffffULL) | reg->u32_max_value; | ||
| 2279 | reg->umin_value = max_t(u64, reg->umin_value, new_umin); | ||
| 2280 | reg->umax_value = min_t(u64, reg->umax_value, new_umax); | ||
| 2281 | /* u32 -> s64 tightening, u32 range embedded into s64 preserves range validity */ | ||
| 2282 | new_smin = (reg->smin_value & ~0xffffffffULL) | reg->u32_min_value; | ||
| 2283 | new_smax = (reg->smax_value & ~0xffffffffULL) | reg->u32_max_value; | ||
| 2284 | reg->smin_value = max_t(s64, reg->smin_value, new_smin); | ||
| 2285 | reg->smax_value = min_t(s64, reg->smax_value, new_smax); | ||
| 2286 | |||
| 2287 | /* Here we would like to handle a special case after sign extending load, | ||
| 2288 | * when upper bits for a 64-bit range are all 1s or all 0s. | ||
| 2289 | * | ||
| 2290 | * Upper bits are all 1s when register is in a range: | ||
| 2291 | * [0xffff_ffff_0000_0000, 0xffff_ffff_ffff_ffff] | ||
| 2292 | * Upper bits are all 0s when register is in a range: | ||
| 2293 | * [0x0000_0000_0000_0000, 0x0000_0000_ffff_ffff] | ||
| 2294 | * Together this forms are continuous range: | ||
| 2295 | * [0xffff_ffff_0000_0000, 0x0000_0000_ffff_ffff] | ||
| 2296 | * | ||
| 2297 | * Now, suppose that register range is in fact tighter: | ||
| 2298 | * [0xffff_ffff_8000_0000, 0x0000_0000_ffff_ffff] (R) | ||
| 2299 | * Also suppose that it's 32-bit range is positive, | ||
| 2300 | * meaning that lower 32-bits of the full 64-bit register | ||
| 2301 | * are in the range: | ||
| 2302 | * [0x0000_0000, 0x7fff_ffff] (W) | ||
| 2303 | * | ||
| 2304 | * If this happens, then any value in a range: | ||
| 2305 | * [0xffff_ffff_0000_0000, 0xffff_ffff_7fff_ffff] | ||
| 2306 | * is smaller than a lowest bound of the range (R): | ||
| 2307 | * 0xffff_ffff_8000_0000 | ||
| 2308 | * which means that upper bits of the full 64-bit register | ||
| 2309 | * can't be all 1s, when lower bits are in range (W). | ||
| 2310 | * | ||
| 2311 | * Note that: | ||
| 2312 | * - 0xffff_ffff_8000_0000 == (s64)S32_MIN | ||
| 2313 | * - 0x0000_0000_7fff_ffff == (s64)S32_MAX | ||
| 2314 | * These relations are used in the conditions below. | ||
| 2315 | */ | ||
| 2316 | if (reg->s32_min_value >= 0 && reg->smin_value >= S32_MIN && reg->smax_value <= S32_MAX) { | ||
| 2317 | reg->smin_value = reg->s32_min_value; | ||
| 2318 | reg->smax_value = reg->s32_max_value; | ||
| 2319 | reg->umin_value = reg->s32_min_value; | ||
| 2320 | reg->umax_value = reg->s32_max_value; | ||
| 2321 | reg->var_off = tnum_intersect(reg->var_off, | ||
| 2322 | tnum_range(reg->smin_value, reg->smax_value)); | ||
| 2323 | } | ||
| 2324 | } | 2050 | } |
| 2325 | 2051 | ||
| 2326 | static void __reg_deduce_bounds(struct bpf_reg_state *reg) | 2052 | static void __reg_deduce_bounds(struct bpf_reg_state *reg) |
| 2327 | { | 2053 | { |
| 2328 | deduce_bounds_64_from_64(reg); | ||
| 2329 | deduce_bounds_32_from_64(reg); | 2054 | deduce_bounds_32_from_64(reg); |
| 2330 | deduce_bounds_32_from_32(reg); | ||
| 2331 | deduce_bounds_64_from_32(reg); | 2055 | deduce_bounds_64_from_32(reg); |
| 2332 | } | 2056 | } |
| 2333 | 2057 | ||
| @@ -2335,11 +2059,11 @@ static void __reg_deduce_bounds(struct bpf_reg_state *reg) · 1 unmodified line | |||
| 2335 | static void __reg_bound_offset(struct bpf_reg_state *reg) | 2059 | static void __reg_bound_offset(struct bpf_reg_state *reg) |
| 2336 | { | 2060 | { |
| 2337 | struct tnum var64_off = tnum_intersect(reg->var_off, | 2061 | struct tnum var64_off = tnum_intersect(reg->var_off, |
| 2338 | tnum_range(reg->umin_value, | 2062 | tnum_range(reg_umin(reg), |
| 2339 | reg->umax_value)); | 2063 | reg_umax(reg))); |
| 2340 | struct tnum var32_off = tnum_intersect(tnum_subreg(var64_off), | 2064 | struct tnum var32_off = tnum_intersect(tnum_subreg(var64_off), |
| 2341 | tnum_range(reg->u32_min_value, | 2065 | tnum_range(reg_u32_min(reg), |
| 2342 | reg->u32_max_value)); | 2066 | reg_u32_max(reg))); |
| 2343 | 2067 | ||
| 2344 | reg->var_off = tnum_or(tnum_clear_subreg(var64_off), var32_off); | 2068 | reg->var_off = tnum_or(tnum_clear_subreg(var64_off), var32_off); |
| 2345 | } | 2069 | } |
| @@ -2365,35 +2089,25 @@ static void reg_bounds_sync(struct bpf_reg_state *reg) · 19 unmodified lines | |||
| 2365 | __update_reg_bounds(reg); | 2089 | __update_reg_bounds(reg); |
| 2366 | } | 2090 | } |
| 2367 | 2091 | ||
| 2368 | static bool range_bounds_violation(struct bpf_reg_state *reg) | ||
| 2369 | { | ||
| 2370 | return (reg->umin_value > reg->umax_value || reg->smin_value > reg->smax_value || | ||
| 2371 | reg->u32_min_value > reg->u32_max_value || | ||
| 2372 | reg->s32_min_value > reg->s32_max_value); | ||
| 2373 | } | ||
| 2374 | |||
| 2375 | static bool const_tnum_range_mismatch(struct bpf_reg_state *reg) | 2092 | static bool const_tnum_range_mismatch(struct bpf_reg_state *reg) |
| 2376 | { | 2093 | { |
| 2377 | u64 uval = reg->var_off.value; | ||
| 2378 | s64 sval = (s64)uval; | ||
| 2379 | |||
| 2380 | if (!tnum_is_const(reg->var_off)) | 2094 | if (!tnum_is_const(reg->var_off)) |
| 2381 | return false; | 2095 | return false; |
| 2382 | 2096 | ||
| 2383 | return reg->umin_value != uval || reg->umax_value != uval || | 2097 | return !cnum64_is_const(reg->r64) || reg->r64.base != reg->var_off.value; |
| 2384 | reg->smin_value != sval || reg->smax_value != sval; | ||
| 2385 | } | 2098 | } |
| 2386 | 2099 | ||
| 2387 | static bool const_tnum_range_mismatch_32(struct bpf_reg_state *reg) | 2100 | static bool const_tnum_range_mismatch_32(struct bpf_reg_state *reg) |
| 2388 | { | 2101 | { |
| 2389 | u32 uval32 = tnum_subreg(reg->var_off).value; | ||
| 2390 | s32 sval32 = (s32)uval32; | ||
| 2391 | |||
| 2392 | if (!tnum_subreg_is_const(reg->var_off)) | 2102 | if (!tnum_subreg_is_const(reg->var_off)) |
| 2393 | return false; | 2103 | return false; |
| 2394 | 2104 | ||
| 2395 | return reg->u32_min_value != uval32 || reg->u32_max_value != uval32 || | 2105 | return !cnum32_is_const(reg->r32) || reg->r32.base != tnum_subreg(reg->var_off).value; |
| 2396 | reg->s32_min_value != sval32 || reg->s32_max_value != sval32; | 2106 | } |
| 2107 | |||
| 2108 | static bool range_bounds_violation(struct bpf_reg_state *reg) | ||
| 2109 | { | ||
| 2110 | return cnum32_is_empty(reg->r32) || cnum64_is_empty(reg->r64); | ||
| 2397 | } | 2111 | } |
| 2398 | 2112 | ||
| 2399 | static int reg_bounds_sanity_check(struct bpf_verifier_env *env, | 2113 | static int reg_bounds_sanity_check(struct bpf_verifier_env *env, |
| @@ -2418,12 +2132,11 @@ static int reg_bounds_sanity_check(struct bpf_verifier_env *env, · 18 unmodified lines | |||
| 2418 | 2132 | ||
| 2419 | return 0; | 2133 | return 0; |
| 2420 | out: | 2134 | out: |
| 2421 | verifier_bug(env, "REG INVARIANTS VIOLATION (%s): %s u64=[%#llx, %#llx] " | 2135 | verifier_bug(env, "REG INVARIANTS VIOLATION (%s): %s r64={.base=%#llx, .size=%#llx} " |
| 2422 | "s64=[%#llx, %#llx] u32=[%#x, %#x] s32=[%#x, %#x] var_off=(%#llx, %#llx)", | 2136 | "r32={.base=%#x, .size=%#x} var_off=(%#llx, %#llx)", |
| 2423 | ctx, msg, reg->umin_value, reg->umax_value, | 2137 | ctx, msg, |
| 2424 | reg->smin_value, reg->smax_value, | 2138 | reg->r64.base, reg->r64.size, |
| 2425 | reg->u32_min_value, reg->u32_max_value, | 2139 | reg->r32.base, reg->r32.size, |
| 2426 | reg->s32_min_value, reg->s32_max_value, | ||
| 2427 | reg->var_off.value, reg->var_off.mask); | 2140 | reg->var_off.value, reg->var_off.mask); |
| 2428 | if (env->test_reg_invariants) | 2141 | if (env->test_reg_invariants) |
| 2429 | return -EFAULT; | 2142 | return -EFAULT; |
| @@ -2431,44 +2144,15 @@ out: · 1 unmodified line | |||
| 2431 | return 0; | 2144 | return 0; |
| 2432 | } | 2145 | } |
| 2433 | 2146 | ||
| 2434 | static bool __reg32_bound_s64(s32 a) | ||
| 2435 | { | ||
| 2436 | return a >= 0 && a <= S32_MAX; | ||
| 2437 | } | ||
| 2438 | |||
| 2439 | static void __reg_assign_32_into_64(struct bpf_reg_state *reg) | ||
| 2440 | { | ||
| 2441 | reg->umin_value = reg->u32_min_value; | ||
| 2442 | reg->umax_value = reg->u32_max_value; | ||
| 2443 | |||
| 2444 | /* Attempt to pull 32-bit signed bounds into 64-bit bounds but must | ||
| 2445 | * be positive otherwise set to worse case bounds and refine later | ||
| 2446 | * from tnum. | ||
| 2447 | */ | ||
| 2448 | if (__reg32_bound_s64(reg->s32_min_value) && | ||
| 2449 | __reg32_bound_s64(reg->s32_max_value)) { | ||
| 2450 | reg->smin_value = reg->s32_min_value; | ||
| 2451 | reg->smax_value = reg->s32_max_value; | ||
| 2452 | } else { | ||
| 2453 | reg->smin_value = 0; | ||
| 2454 | reg->smax_value = U32_MAX; | ||
| 2455 | } | ||
| 2456 | } | ||
| 2457 | |||
| 2458 | /* Mark a register as having a completely unknown (scalar) value. */ | 2147 | /* Mark a register as having a completely unknown (scalar) value. */ |
| 2459 | void bpf_mark_reg_unknown_imprecise(struct bpf_reg_state *reg) | 2148 | void bpf_mark_reg_unknown_imprecise(struct bpf_reg_state *reg) |
| 2460 | { | 2149 | { |
| 2461 | /* | 2150 | s32 subreg_def = reg->subreg_def; |
| 2462 | * Clear type, off, and union(map_ptr, range) and | 2151 | |
| 2463 | * padding between 'type' and union | 2152 | memset(reg, 0, sizeof(*reg)); |
| 2464 | */ | ||
| 2465 | memset(reg, 0, offsetof(struct bpf_reg_state, var_off)); | ||
| 2466 | reg->type = SCALAR_VALUE; | 2153 | reg->type = SCALAR_VALUE; |
| 2467 | reg->id = 0; | ||
| 2468 | reg->ref_obj_id = 0; | ||
| 2469 | reg->var_off = tnum_unknown; | 2154 | reg->var_off = tnum_unknown; |
| 2470 | reg->frameno = 0; | 2155 | reg->subreg_def = subreg_def; |
| 2471 | reg->precise = false; | ||
| 2472 | __mark_reg_unbounded(reg); | 2156 | __mark_reg_unbounded(reg); |
| 2473 | } | 2157 | } |
| 2474 | 2158 | ||
| @@ -2496,11 +2180,12 @@ static int __mark_reg_s32_range(struct bpf_verifier_env *env, · 21 unmodified lines | |||
| 2496 | { | 2180 | { |
| 2497 | struct bpf_reg_state *reg = regs + regno; | 2181 | struct bpf_reg_state *reg = regs + regno; |
| 2498 | 2182 | ||
| 2499 | reg->s32_min_value = max_t(s32, reg->s32_min_value, s32_min); | 2183 | reg_set_srange32(reg, |
| 2500 | reg->s32_max_value = min_t(s32, reg->s32_max_value, s32_max); | 2184 | max_t(s32, reg_s32_min(reg), s32_min), |
| 2501 | 2185 | min_t(s32, reg_s32_max(reg), s32_max)); | |
| 2502 | reg->smin_value = max_t(s64, reg->smin_value, s32_min); | 2186 | reg_set_srange64(reg, |
| 2503 | reg->smax_value = min_t(s64, reg->smax_value, s32_max); | 2187 | max_t(s64, reg_smin(reg), s32_min), |
| 2188 | min_t(s64, reg_smax(reg), s32_max)); | ||
| 2504 | 2189 | ||
| 2505 | reg_bounds_sync(reg); | 2190 | reg_bounds_sync(reg); |
| 2506 | 2191 | ||
| @@ -3295,50 +2980,13 @@ out: · 788 unmodified lines | |||
| 3295 | return ret; | 2980 | return ret; |
| 3296 | } | 2981 | } |
| 3297 | 2982 | ||
| 3298 | static int mark_stack_slot_obj_read(struct bpf_verifier_env *env, struct bpf_reg_state *reg, | 2983 | static void mark_stack_slots_scratched(struct bpf_verifier_env *env, |
| 3299 | int spi, int nr_slots) | 2984 | int spi, int nr_slots) |
| 3300 | { | 2985 | { |
| 3301 | int i; | 2986 | int i; |
| 3302 | 2987 | ||
| 3303 | for (i = 0; i < nr_slots; i++) | 2988 | for (i = 0; i < nr_slots; i++) |
| 3304 | mark_stack_slot_scratched(env, spi - i); | 2989 | mark_stack_slot_scratched(env, spi - i); |
| 3305 | return 0; | ||
| 3306 | } | ||
| 3307 | |||
| 3308 | static int mark_dynptr_read(struct bpf_verifier_env *env, struct bpf_reg_state *reg) | ||
| 3309 | { | ||
| 3310 | int spi; | ||
| 3311 | |||
| 3312 | /* For CONST_PTR_TO_DYNPTR, it must have already been done by | ||
| 3313 | * check_reg_arg in check_helper_call and mark_btf_func_reg_size in | ||
| 3314 | * check_kfunc_call. | ||
| 3315 | */ | ||
| 3316 | if (reg->type == CONST_PTR_TO_DYNPTR) | ||
| 3317 | return 0; | ||
| 3318 | spi = dynptr_get_spi(env, reg); | ||
| 3319 | if (spi < 0) | ||
| 3320 | return spi; | ||
| 3321 | /* Caller ensures dynptr is valid and initialized, which means spi is in | ||
| 3322 | * bounds and spi is the first dynptr slot. Simply mark stack slot as | ||
| 3323 | * read. | ||
| 3324 | */ | ||
| 3325 | return mark_stack_slot_obj_read(env, reg, spi, BPF_DYNPTR_NR_SLOTS); | ||
| 3326 | } | ||
| 3327 | |||
| 3328 | static int mark_iter_read(struct bpf_verifier_env *env, struct bpf_reg_state *reg, | ||
| 3329 | int spi, int nr_slots) | ||
| 3330 | { | ||
| 3331 | return mark_stack_slot_obj_read(env, reg, spi, nr_slots); | ||
| 3332 | } | ||
| 3333 | |||
| 3334 | static int mark_irq_flag_read(struct bpf_verifier_env *env, struct bpf_reg_state *reg) | ||
| 3335 | { | ||
| 3336 | int spi; | ||
| 3337 | |||
| 3338 | spi = irq_flag_get_spi(env, reg); | ||
| 3339 | if (spi < 0) | ||
| 3340 | return spi; | ||
| 3341 | return mark_stack_slot_obj_read(env, reg, spi, 1); | ||
| 3342 | } | 2990 | } |
| 3343 | 2991 | ||
| 3344 | /* This function is supposed to be used by the following 32-bit optimization | 2992 | /* This function is supposed to be used by the following 32-bit optimization |
| @@ -3491,17 +3139,12 @@ static int check_reg_arg(struct bpf_verifier_env *env, u32 regno, · 146 unmodified lines | |||
| 3491 | return __check_reg_arg(env, state->regs, regno, t); | 3139 | return __check_reg_arg(env, state->regs, regno, t); |
| 3492 | } | 3140 | } |
| 3493 | 3141 | ||
| 3494 | static int insn_stack_access_flags(int frameno, int spi) | ||
| 3495 | { | ||
| 3496 | return INSN_F_STACK_ACCESS | (spi << INSN_F_SPI_SHIFT) | frameno; | ||
| 3497 | } | ||
| 3498 | |||
| 3499 | static void mark_indirect_target(struct bpf_verifier_env *env, int idx) | 3142 | static void mark_indirect_target(struct bpf_verifier_env *env, int idx) |
| 3500 | { | 3143 | { |
| 3501 | env->insn_aux_data[idx].indirect_target = true; | 3144 | env->insn_aux_data[idx].indirect_target = true; |
| 3502 | } | 3145 | } |
| 3503 | 3146 | ||
| 3504 | #define LR_FRAMENO_BITS 3 | 3147 | #define LR_FRAMENO_BITS 4 |
| 3505 | #define LR_SPI_BITS 6 | 3148 | #define LR_SPI_BITS 6 |
| 3506 | #define LR_ENTRY_BITS (LR_SPI_BITS + LR_FRAMENO_BITS + 1) | 3149 | #define LR_ENTRY_BITS (LR_SPI_BITS + LR_FRAMENO_BITS + 1) |
| 3507 | #define LR_SIZE_BITS 4 | 3150 | #define LR_SIZE_BITS 4 |
| @@ -3510,7 +3153,11 @@ static void mark_indirect_target(struct bpf_verifier_env *env, int idx) · 2 unmodified lines | |||
| 3510 | #define LR_SIZE_MASK ((1ull << LR_SIZE_BITS) - 1) | 3153 | #define LR_SIZE_MASK ((1ull << LR_SIZE_BITS) - 1) |
| 3511 | #define LR_SPI_OFF LR_FRAMENO_BITS | 3154 | #define LR_SPI_OFF LR_FRAMENO_BITS |
| 3512 | #define LR_IS_REG_OFF (LR_SPI_BITS + LR_FRAMENO_BITS) | 3155 | #define LR_IS_REG_OFF (LR_SPI_BITS + LR_FRAMENO_BITS) |
| 3513 | #define LINKED_REGS_MAX 6 | 3156 | #define LINKED_REGS_MAX 5 |
| 3157 | |||
| 3158 | static_assert(MAX_CALL_FRAMES <= (1 << LR_FRAMENO_BITS)); | ||
| 3159 | static_assert(LINKED_REGS_MAX < (1 << LR_SIZE_BITS)); | ||
| 3160 | static_assert(LINKED_REGS_MAX * LR_ENTRY_BITS + LR_SIZE_BITS <= 64); | ||
| 3514 | 3161 | ||
| 3515 | struct linked_reg { | 3162 | struct linked_reg { |
| 3516 | u8 frameno; | 3163 | u8 frameno; |
| @@ -3534,10 +3181,11 @@ static struct linked_reg *linked_regs_push(struct linked_regs *s) · 17 unmodified lines | |||
| 3534 | return NULL; | 3181 | return NULL; |
| 3535 | } | 3182 | } |
| 3536 | 3183 | ||
| 3537 | /* Use u64 as a vector of 6 10-bit values, use first 4-bits to track | 3184 | /* |
| 3185 | * Use u64 as a vector of 5 11-bit values, use first 4-bits to track | ||
| 3538 | * number of elements currently in stack. | 3186 | * number of elements currently in stack. |
| 3539 | * Pack one history entry for linked registers as 10 bits in the following format: | 3187 | * Pack one history entry for linked registers as 11 bits in the following format: |
| 3540 | * - 3-bits frameno | 3188 | * - 4-bits frameno |
| 3541 | * - 6-bits spi_or_reg | 3189 | * - 6-bits spi_or_reg |
| 3542 | * - 1-bit is_reg | 3190 | * - 1-bit is_reg |
| 3543 | */ | 3191 | */ |
| @@ -3733,12 +3381,6 @@ static void assign_scalar_id_before_mov(struct bpf_verifier_env *env, · 189 unmodified lines | |||
| 3733 | src_reg->id = ++env->id_gen; | 3381 | src_reg->id = ++env->id_gen; |
| 3734 | } | 3382 | } |
| 3735 | 3383 | ||
| 3736 | /* Copy src state preserving dst->parent and dst->live fields */ | ||
| 3737 | static void copy_register_state(struct bpf_reg_state *dst, const struct bpf_reg_state *src) | ||
| 3738 | { | ||
| 3739 | *dst = *src; | ||
| 3740 | } | ||
| 3741 | |||
| 3742 | static void save_register_state(struct bpf_verifier_env *env, | 3384 | static void save_register_state(struct bpf_verifier_env *env, |
| 3743 | struct bpf_func_state *state, | 3385 | struct bpf_func_state *state, |
| 3744 | int spi, struct bpf_reg_state *reg, | 3386 | int spi, struct bpf_reg_state *reg, |
| @@ -3746,7 +3388,7 @@ static void save_register_state(struct bpf_verifier_env *env, · 1 unmodified line | |||
| 3746 | { | 3388 | { |
| 3747 | int i; | 3389 | int i; |
| 3748 | 3390 | ||
| 3749 | copy_register_state(&state->stack[spi].spilled_ptr, reg); | 3391 | state->stack[spi].spilled_ptr = *reg; |
| 3750 | 3392 | ||
| 3751 | for (i = BPF_REG_SIZE; i > BPF_REG_SIZE - size; i--) | 3393 | for (i = BPF_REG_SIZE; i > BPF_REG_SIZE - size; i--) |
| 3752 | state->stack[spi].slot_type[i - 1] = STACK_SPILL; | 3394 | state->stack[spi].slot_type[i - 1] = STACK_SPILL; |
| @@ -3763,7 +3405,7 @@ static bool is_bpf_st_mem(struct bpf_insn *insn) · 10 unmodified lines | |||
| 3763 | 3405 | ||
| 3764 | static int get_reg_width(struct bpf_reg_state *reg) | 3406 | static int get_reg_width(struct bpf_reg_state *reg) |
| 3765 | { | 3407 | { |
| 3766 | return fls64(reg->umax_value); | 3408 | return fls64(reg_umax(reg)); |
| 3767 | } | 3409 | } |
| 3768 | 3410 | ||
| 3769 | /* See comment for mark_fastcall_pattern_for_call() */ | 3411 | /* See comment for mark_fastcall_pattern_for_call() */ |
| @@ -3816,7 +3458,8 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env, · 46 unmodified lines | |||
| 3816 | int i, slot = -off - 1, spi = slot / BPF_REG_SIZE, err; | 3458 | int i, slot = -off - 1, spi = slot / BPF_REG_SIZE, err; |
| 3817 | struct bpf_insn *insn = &env->prog->insnsi[insn_idx]; | 3459 | struct bpf_insn *insn = &env->prog->insnsi[insn_idx]; |
| 3818 | struct bpf_reg_state *reg = NULL; | 3460 | struct bpf_reg_state *reg = NULL; |
| 3819 | int insn_flags = insn_stack_access_flags(state->frameno, spi); | 3461 | int insn_flags = INSN_F_STACK_ACCESS; |
| 3462 | int hist_spi = spi, hist_frame = state->frameno; | ||
| 3820 | 3463 | ||
| 3821 | /* caller checked that off % size == 0 and -MAX_BPF_STACK <= off < 0, | 3464 | /* caller checked that off % size == 0 and -MAX_BPF_STACK <= off < 0, |
| 3822 | * so it's aligned access and [off, off + size) are within stack limits | 3465 | * so it's aligned access and [off, off + size) are within stack limits |
| @@ -3912,11 +3555,12 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env, · 89 unmodified lines | |||
| 3912 | } | 3555 | } |
| 3913 | 3556 | ||
| 3914 | if (insn_flags) | 3557 | if (insn_flags) |
| 3915 | return bpf_push_jmp_history(env, env->cur_state, insn_flags, 0); | 3558 | return bpf_push_jmp_history(env, env->cur_state, insn_flags, |
| 3559 | hist_spi, hist_frame, 0); | ||
| 3916 | return 0; | 3560 | return 0; |
| 3917 | } | 3561 | } |
| 3918 | 3562 | ||
| 3919 | /* Write the stack: 'stack[ptr_regno + off] = value_regno'. 'ptr_regno' is | 3563 | /* Write the stack: 'stack[ptr_reg + off] = value_regno'. 'ptr_reg' is |
| 3920 | * known to contain a variable offset. | 3564 | * known to contain a variable offset. |
| 3921 | * This function checks whether the write is permitted and conservatively | 3565 | * This function checks whether the write is permitted and conservatively |
| 3922 | * tracks the effects of the write, considering that each stack slot in the | 3566 | * tracks the effects of the write, considering that each stack slot in the |
| @@ -3937,13 +3581,13 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env, · 14 unmodified lines | |||
| 3937 | static int check_stack_write_var_off(struct bpf_verifier_env *env, | 3581 | static int check_stack_write_var_off(struct bpf_verifier_env *env, |
| 3938 | /* func where register points to */ | 3582 | /* func where register points to */ |
| 3939 | struct bpf_func_state *state, | 3583 | struct bpf_func_state *state, |
| 3940 | int ptr_regno, int off, int size, | 3584 | struct bpf_reg_state *ptr_reg, int off, int size, |
| 3941 | int value_regno, int insn_idx) | 3585 | int value_regno, int insn_idx) |
| 3942 | { | 3586 | { |
| 3943 | struct bpf_func_state *cur; /* state of the current function */ | 3587 | struct bpf_func_state *cur; /* state of the current function */ |
| 3944 | int min_off, max_off; | 3588 | int min_off, max_off; |
| 3945 | int i, err; | 3589 | int i, err; |
| 3946 | struct bpf_reg_state *ptr_reg = NULL, *value_reg = NULL; | 3590 | struct bpf_reg_state *value_reg = NULL; |
| 3947 | struct bpf_insn *insn = &env->prog->insnsi[insn_idx]; | 3591 | struct bpf_insn *insn = &env->prog->insnsi[insn_idx]; |
| 3948 | bool writing_zero = false; | 3592 | bool writing_zero = false; |
| 3949 | /* set if the fact that we're writing a zero is used to let any | 3593 | /* set if the fact that we're writing a zero is used to let any |
| @@ -3952,9 +3596,8 @@ static int check_stack_write_var_off(struct bpf_verifier_env *env, · 2 unmodified lines | |||
| 3952 | bool zero_used = false; | 3596 | bool zero_used = false; |
| 3953 | 3597 | ||
| 3954 | cur = env->cur_state->frame[env->cur_state->curframe]; | 3598 | cur = env->cur_state->frame[env->cur_state->curframe]; |
| 3955 | ptr_reg = &cur->regs[ptr_regno]; | 3599 | min_off = reg_smin(ptr_reg) + off; |
| 3956 | min_off = ptr_reg->smin_value + off; | 3600 | max_off = reg_smax(ptr_reg) + off + size; |
| 3957 | max_off = ptr_reg->smax_value + off + size; | ||
| 3958 | if (value_regno >= 0) | 3601 | if (value_regno >= 0) |
| 3959 | value_reg = &cur->regs[value_regno]; | 3602 | value_reg = &cur->regs[value_regno]; |
| 3960 | if ((value_reg && bpf_register_is_null(value_reg)) || | 3603 | if ((value_reg && bpf_register_is_null(value_reg)) || |
| @@ -4109,7 +3752,8 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env, · 148 unmodified lines | |||
| 4109 | int i, slot = -off - 1, spi = slot / BPF_REG_SIZE; | 3752 | int i, slot = -off - 1, spi = slot / BPF_REG_SIZE; |
| 4110 | struct bpf_reg_state *reg; | 3753 | struct bpf_reg_state *reg; |
| 4111 | u8 *stype, type; | 3754 | u8 *stype, type; |
| 4112 | int insn_flags = insn_stack_access_flags(reg_state->frameno, spi); | 3755 | int insn_flags = INSN_F_STACK_ACCESS; |
| 3756 | int hist_spi = spi, hist_frame = reg_state->frameno; | ||
| 4113 | 3757 | ||
| 4114 | stype = reg_state->stack[spi].slot_type; | 3758 | stype = reg_state->stack[spi].slot_type; |
| 4115 | reg = ®_state->stack[spi].spilled_ptr; | 3759 | reg = ®_state->stack[spi].spilled_ptr; |
| @@ -4146,7 +3790,7 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env, · 30 unmodified lines | |||
| 4146 | * with the destination register on fill. | 3790 | * with the destination register on fill. |
| 4147 | */ | 3791 | */ |
| 4148 | assign_scalar_id_before_mov(env, reg); | 3792 | assign_scalar_id_before_mov(env, reg); |
| 4149 | copy_register_state(&state->regs[dst_regno], reg); | 3793 | state->regs[dst_regno] = *reg; |
| 4150 | state->regs[dst_regno].subreg_def = subreg_def; | 3794 | state->regs[dst_regno].subreg_def = subreg_def; |
| 4151 | 3795 | ||
| 4152 | /* Break the relation on a narrowing fill. | 3796 | /* Break the relation on a narrowing fill. |
| @@ -4201,7 +3845,7 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env, · 48 unmodified lines | |||
| 4201 | * with the destination register on fill. | 3845 | * with the destination register on fill. |
| 4202 | */ | 3846 | */ |
| 4203 | assign_scalar_id_before_mov(env, reg); | 3847 | assign_scalar_id_before_mov(env, reg); |
| 4204 | copy_register_state(&state->regs[dst_regno], reg); | 3848 | state->regs[dst_regno] = *reg; |
| 4205 | /* mark reg as written since spilled pointer state likely | 3849 | /* mark reg as written since spilled pointer state likely |
| 4206 | * has its liveness marks cleared by is_state_visited() | 3850 | * has its liveness marks cleared by is_state_visited() |
| 4207 | * which resets stack/reg liveness for state transitions | 3851 | * which resets stack/reg liveness for state transitions |
| @@ -4240,7 +3884,8 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env, · 32 unmodified lines | |||
| 4240 | insn_flags = 0; /* we are not restoring spilled register */ | 3884 | insn_flags = 0; /* we are not restoring spilled register */ |
| 4241 | } | 3885 | } |
| 4242 | if (insn_flags) | 3886 | if (insn_flags) |
| 4243 | return bpf_push_jmp_history(env, env->cur_state, insn_flags, 0); | 3887 | return bpf_push_jmp_history(env, env->cur_state, insn_flags, |
| 3888 | hist_spi, hist_frame, 0); | ||
| 4244 | return 0; | 3889 | return 0; |
| 4245 | } | 3890 | } |
| 4246 | 3891 | ||
| @@ -4249,8 +3894,8 @@ enum bpf_access_src { · 2 unmodified lines | |||
| 4249 | ACCESS_HELPER = 2, /* the access is performed by a helper */ | 3894 | ACCESS_HELPER = 2, /* the access is performed by a helper */ |
| 4250 | }; | 3895 | }; |
| 4251 | 3896 | ||
| 4252 | static int check_stack_range_initialized(struct bpf_verifier_env *env, | 3897 | static int check_stack_range_initialized(struct bpf_verifier_env *env, struct bpf_reg_state *reg, |
| 4253 | int regno, int off, int access_size, | 3898 | argno_t argno, int off, int access_size, |
| 4254 | bool zero_size_allowed, | 3899 | bool zero_size_allowed, |
| 4255 | enum bpf_access_type type, | 3900 | enum bpf_access_type type, |
| 4256 | struct bpf_call_arg_meta *meta); | 3901 | struct bpf_call_arg_meta *meta); |
| @@ -4260,37 +3905,35 @@ static struct bpf_reg_state *reg_state(struct bpf_verifier_env *env, int regno) · 3 unmodified lines | |||
| 4260 | return cur_regs(env) + regno; | 3905 | return cur_regs(env) + regno; |
| 4261 | } | 3906 | } |
| 4262 | 3907 | ||
| 4263 | /* Read the stack at 'ptr_regno + off' and put the result into the register | 3908 | /* Read the stack at 'reg + off' and put the result into the register |
| 4264 | * 'dst_regno'. | 3909 | * 'dst_regno'. |
| 4265 | * 'off' includes the pointer register's fixed offset(i.e. 'ptr_regno.off'), | 3910 | * 'off' includes the pointer register's fixed offset(i.e. 'reg->off'), |
| 4266 | * but not its variable offset. | 3911 | * but not its variable offset. |
| 4267 | * 'size' is assumed to be <= reg size and the access is assumed to be aligned. | 3912 | * 'size' is assumed to be <= reg size and the access is assumed to be aligned. |
| 4268 | * | 3913 | * |
| 4269 | * As opposed to check_stack_read_fixed_off, this function doesn't deal with | 3914 | * As opposed to check_stack_read_fixed_off, this function doesn't deal with |
| 4270 | * filling registers (i.e. reads of spilled register cannot be detected when | 3915 | * filling registers (i.e. reads of spilled register cannot be detected when |
| 4271 | * the offset is not fixed). We conservatively mark 'dst_regno' as containing | 3916 | * the offset is not fixed). We conservatively mark 'dst_regno' as containing |
| 4272 | * SCALAR_VALUE. That's why we assert that the 'ptr_regno' has a variable | 3917 | * SCALAR_VALUE. That's why we assert that the 'reg' has a variable |
| 4273 | * offset; for a fixed offset check_stack_read_fixed_off should be used | 3918 | * offset; for a fixed offset check_stack_read_fixed_off should be used |
| 4274 | * instead. | 3919 | * instead. |
| 4275 | */ | 3920 | */ |
| 4276 | static int check_stack_read_var_off(struct bpf_verifier_env *env, | 3921 | static int check_stack_read_var_off(struct bpf_verifier_env *env, struct bpf_reg_state *reg, |
| 4277 | int ptr_regno, int off, int size, int dst_regno) | 3922 | argno_t ptr_argno, int off, int size, int dst_regno) |
| 4278 | { | 3923 | { |
| 4279 | /* The state of the source register. */ | ||
| 4280 | struct bpf_reg_state *reg = reg_state(env, ptr_regno); | ||
| 4281 | struct bpf_func_state *ptr_state = bpf_func(env, reg); | 3924 | struct bpf_func_state *ptr_state = bpf_func(env, reg); |
| 4282 | int err; | 3925 | int err; |
| 4283 | int min_off, max_off; | 3926 | int min_off, max_off; |
| 4284 | 3927 | ||
| 4285 | /* Note that we pass a NULL meta, so raw access will not be permitted. | 3928 | /* Note that we pass a NULL meta, so raw access will not be permitted. |
| 4286 | */ | 3929 | */ |
| 4287 | err = check_stack_range_initialized(env, ptr_regno, off, size, | 3930 | err = check_stack_range_initialized(env, reg, ptr_argno, off, size, |
| 4288 | false, BPF_READ, NULL); | 3931 | false, BPF_READ, NULL); |
| 4289 | if (err) | 3932 | if (err) |
| 4290 | return err; | 3933 | return err; |
| 4291 | 3934 | ||
| 4292 | min_off = reg->smin_value + off; | 3935 | min_off = reg_smin(reg) + off; |
| 4293 | max_off = reg->smax_value + off; | 3936 | max_off = reg_smax(reg) + off; |
| 4294 | mark_reg_stack_read(env, ptr_state, min_off, max_off + size, dst_regno); | 3937 | mark_reg_stack_read(env, ptr_state, min_off, max_off + size, dst_regno); |
| 4295 | check_fastcall_stack_contract(env, ptr_state, env->insn_idx, min_off); | 3938 | check_fastcall_stack_contract(env, ptr_state, env->insn_idx, min_off); |
| 4296 | return 0; | 3939 | return 0; |
| @@ -4306,10 +3949,9 @@ static int check_stack_read_var_off(struct bpf_verifier_env *env, · 9 unmodified lines | |||
| 4306 | * can be -1, meaning that the read value is not going to a register. | 3949 | * can be -1, meaning that the read value is not going to a register. |
| 4307 | */ | 3950 | */ |
| 4308 | static int check_stack_read(struct bpf_verifier_env *env, | 3951 | static int check_stack_read(struct bpf_verifier_env *env, |
| 4309 | int ptr_regno, int off, int size, | 3952 | struct bpf_reg_state *reg, argno_t ptr_argno, int off, int size, |
| 4310 | int dst_regno) | 3953 | int dst_regno) |
| 4311 | { | 3954 | { |
| 4312 | struct bpf_reg_state *reg = reg_state(env, ptr_regno); | ||
| 4313 | struct bpf_func_state *state = bpf_func(env, reg); | 3955 | struct bpf_func_state *state = bpf_func(env, reg); |
| 4314 | int err; | 3956 | int err; |
| 4315 | /* Some accesses are only permitted with a static offset. */ | 3957 | /* Some accesses are only permitted with a static offset. */ |
| @@ -4345,7 +3987,7 @@ static int check_stack_read(struct bpf_verifier_env *env, · 29 unmodified lines | |||
| 4345 | * than fixed offset ones. Note that dst_regno >= 0 on this | 3987 | * than fixed offset ones. Note that dst_regno >= 0 on this |
| 4346 | * branch. | 3988 | * branch. |
| 4347 | */ | 3989 | */ |
| 4348 | err = check_stack_read_var_off(env, ptr_regno, off, size, | 3990 | err = check_stack_read_var_off(env, reg, ptr_argno, off, size, |
| 4349 | dst_regno); | 3991 | dst_regno); |
| 4350 | } | 3992 | } |
| 4351 | return err; | 3993 | return err; |
| @@ -4355,17 +3997,16 @@ static int check_stack_read(struct bpf_verifier_env *env, · 3 unmodified lines | |||
| 4355 | /* check_stack_write dispatches to check_stack_write_fixed_off or | 3997 | /* check_stack_write dispatches to check_stack_write_fixed_off or |
| 4356 | * check_stack_write_var_off. | 3998 | * check_stack_write_var_off. |
| 4357 | * | 3999 | * |
| 4358 | * 'ptr_regno' is the register used as a pointer into the stack. | 4000 | * 'reg' is the register used as a pointer into the stack. |
| 4359 | * 'value_regno' is the register whose value we're writing to the stack. It can | 4001 | * 'value_regno' is the register whose value we're writing to the stack. It can |
| 4360 | * be -1, meaning that we're not writing from a register. | 4002 | * be -1, meaning that we're not writing from a register. |
| 4361 | * | 4003 | * |
| 4362 | * The caller must ensure that the offset falls within the maximum stack size. | 4004 | * The caller must ensure that the offset falls within the maximum stack size. |
| 4363 | */ | 4005 | */ |
| 4364 | static int check_stack_write(struct bpf_verifier_env *env, | 4006 | static int check_stack_write(struct bpf_verifier_env *env, |
| 4365 | int ptr_regno, int off, int size, | 4007 | struct bpf_reg_state *reg, int off, int size, |
| 4366 | int value_regno, int insn_idx) | 4008 | int value_regno, int insn_idx) |
| 4367 | { | 4009 | { |
| 4368 | struct bpf_reg_state *reg = reg_state(env, ptr_regno); | ||
| 4369 | struct bpf_func_state *state = bpf_func(env, reg); | 4010 | struct bpf_func_state *state = bpf_func(env, reg); |
| 4370 | int err; | 4011 | int err; |
| 4371 | 4012 | ||
| @@ -4378,28 +4019,135 @@ static int check_stack_write(struct bpf_verifier_env *env, · 6 unmodified lines | |||
| 4378 | * than fixed offset ones. | 4019 | * than fixed offset ones. |
| 4379 | */ | 4020 | */ |
| 4380 | err = check_stack_write_var_off(env, state, | 4021 | err = check_stack_write_var_off(env, state, |
| 4381 | ptr_regno, off, size, | 4022 | reg, off, size, |
| 4382 | value_regno, insn_idx); | 4023 | value_regno, insn_idx); |
| 4383 | } | 4024 | } |
| 4384 | return err; | 4025 | return err; |
| 4385 | } | 4026 | } |
| 4386 | 4027 | ||
| 4387 | static int check_map_access_type(struct bpf_verifier_env *env, u32 regno, | 4028 | /* |
| 4029 | * Write a value to the outgoing stack arg area. | ||
| 4030 | * off is a negative offset from r11 (e.g. -8 for arg6, -16 for arg7). | ||
| 4031 | */ | ||
| 4032 | static int check_stack_arg_write(struct bpf_verifier_env *env, struct bpf_func_state *state, | ||
| 4033 | int off, struct bpf_reg_state *value_reg) | ||
| 4034 | { | ||
| 4035 | int max_stack_arg_regs = MAX_BPF_FUNC_ARGS - MAX_BPF_FUNC_REG_ARGS; | ||
| 4036 | struct bpf_subprog_info *subprog = &env->subprog_info[state->subprogno]; | ||
| 4037 | int spi = -off / BPF_REG_SIZE - 1; | ||
| 4038 | struct bpf_reg_state *arg; | ||
| 4039 | int err; | ||
| 4040 | |||
| 4041 | if (spi >= max_stack_arg_regs) { | ||
| 4042 | verbose(env, "stack arg write offset %d exceeds max %d stack args\n", | ||
| 4043 | off, max_stack_arg_regs); | ||
| 4044 | return -EINVAL; | ||
| 4045 | } | ||
| 4046 | |||
| 4047 | err = grow_stack_arg_slots(env, state, spi + 1); | ||
| 4048 | if (err) | ||
| 4049 | return err; | ||
| 4050 | |||
| 4051 | /* Track the max outgoing stack arg slot count. */ | ||
| 4052 | if (spi + 1 > subprog->max_out_stack_arg_cnt) | ||
| 4053 | subprog->max_out_stack_arg_cnt = spi + 1; | ||
| 4054 | |||
| 4055 | if (value_reg) { | ||
| 4056 | state->stack_arg_regs[spi] = *value_reg; | ||
| 4057 | } else { | ||
| 4058 | /* BPF_ST: store immediate, treat as scalar */ | ||
| 4059 | arg = &state->stack_arg_regs[spi]; | ||
| 4060 | arg->type = SCALAR_VALUE; | ||
| 4061 | __mark_reg_known(arg, env->prog->insnsi[env->insn_idx].imm); | ||
| 4062 | } | ||
| 4063 | state->no_stack_arg_load = true; | ||
| 4064 | return bpf_push_jmp_history(env, env->cur_state, | ||
| 4065 | INSN_F_STACK_ARG_ACCESS, spi, 0, 0); | ||
| 4066 | } | ||
| 4067 | |||
| 4068 | /* | ||
| 4069 | * Read a value from the incoming stack arg area. | ||
| 4070 | * off is a positive offset from r11 (e.g. +8 for arg6, +16 for arg7). | ||
| 4071 | */ | ||
| 4072 | static int check_stack_arg_read(struct bpf_verifier_env *env, struct bpf_func_state *state, | ||
| 4073 | int off, int dst_regno) | ||
| 4074 | { | ||
| 4075 | struct bpf_subprog_info *subprog = &env->subprog_info[state->subprogno]; | ||
| 4076 | struct bpf_verifier_state *vstate = env->cur_state; | ||
| 4077 | int spi = off / BPF_REG_SIZE - 1; | ||
| 4078 | struct bpf_func_state *caller, *cur; | ||
| 4079 | struct bpf_reg_state *arg; | ||
| 4080 | |||
| 4081 | if (state->no_stack_arg_load) { | ||
| 4082 | verbose(env, "r11 load must be before any r11 store or call insn\n"); | ||
| 4083 | return -EINVAL; | ||
| 4084 | } | ||
| 4085 | |||
| 4086 | if (spi + 1 > bpf_in_stack_arg_cnt(subprog)) { | ||
| 4087 | verbose(env, "invalid read from stack arg off %d depth %d\n", | ||
| 4088 | off, bpf_in_stack_arg_cnt(subprog) * BPF_REG_SIZE); | ||
| 4089 | return -EACCES; | ||
| 4090 | } | ||
| 4091 | |||
| 4092 | caller = vstate->frame[vstate->curframe - 1]; | ||
| 4093 | arg = &caller->stack_arg_regs[spi]; | ||
| 4094 | cur = vstate->frame[vstate->curframe]; | ||
| 4095 | cur->regs[dst_regno] = *arg; | ||
| 4096 | return bpf_push_jmp_history(env, env->cur_state, | ||
| 4097 | INSN_F_STACK_ARG_ACCESS, spi, 0, 0); | ||
| 4098 | } | ||
| 4099 | |||
| 4100 | static int mark_stack_arg_precision(struct bpf_verifier_env *env, int arg_idx) | ||
| 4101 | { | ||
| 4102 | struct bpf_func_state *caller = cur_func(env); | ||
| 4103 | int spi = arg_idx - MAX_BPF_FUNC_REG_ARGS; | ||
| 4104 | |||
| 4105 | bt_set_frame_stack_arg_slot(&env->bt, caller->frameno, spi); | ||
| 4106 | return mark_chain_precision_batch(env, env->cur_state); | ||
| 4107 | } | ||
| 4108 | |||
| 4109 | static int check_outgoing_stack_args(struct bpf_verifier_env *env, struct bpf_func_state *caller, | ||
| 4110 | int nargs) | ||
| 4111 | { | ||
| 4112 | int i, spi; | ||
| 4113 | |||
| 4114 | for (i = MAX_BPF_FUNC_REG_ARGS; i < nargs; i++) { | ||
| 4115 | spi = i - MAX_BPF_FUNC_REG_ARGS; | ||
| 4116 | if (spi >= caller->out_stack_arg_cnt || | ||
| 4117 | caller->stack_arg_regs[spi].type == NOT_INIT) { | ||
| 4118 | verbose(env, "callee expects %d args, stack arg%d is not initialized\n", | ||
| 4119 | nargs, spi + 1); | ||
| 4120 | return -EFAULT; | ||
| 4121 | } | ||
| 4122 | } | ||
| 4123 | |||
| 4124 | return 0; | ||
| 4125 | } | ||
| 4126 | |||
| 4127 | static struct bpf_reg_state *get_func_arg_reg(struct bpf_func_state *caller, | ||
| 4128 | struct bpf_reg_state *regs, int arg) | ||
| 4129 | { | ||
| 4130 | if (arg < MAX_BPF_FUNC_REG_ARGS) | ||
| 4131 | return ®s[arg + 1]; | ||
| 4132 | |||
| 4133 | return &caller->stack_arg_regs[arg - MAX_BPF_FUNC_REG_ARGS]; | ||
| 4134 | } | ||
| 4135 | |||
| 4136 | static int check_map_access_type(struct bpf_verifier_env *env, struct bpf_reg_state *reg, | ||
| 4388 | int off, int size, enum bpf_access_type type) | 4137 | int off, int size, enum bpf_access_type type) |
| 4389 | { | 4138 | { |
| 4390 | struct bpf_reg_state *reg = reg_state(env, regno); | ||
| 4391 | struct bpf_map *map = reg->map_ptr; | 4139 | struct bpf_map *map = reg->map_ptr; |
| 4392 | u32 cap = bpf_map_flags_to_cap(map); | 4140 | u32 cap = bpf_map_flags_to_cap(map); |
| 4393 | 4141 | ||
| 4394 | if (type == BPF_WRITE && !(cap & BPF_MAP_CAN_WRITE)) { | 4142 | if (type == BPF_WRITE && !(cap & BPF_MAP_CAN_WRITE)) { |
| 4395 | verbose(env, "write into map forbidden, value_size=%d off=%lld size=%d\n", | 4143 | verbose(env, "write into map forbidden, value_size=%d off=%lld size=%d\n", |
| 4396 | map->value_size, reg->smin_value + off, size); | 4144 | map->value_size, reg_smin(reg) + off, size); |
| 4397 | return -EACCES; | 4145 | return -EACCES; |
| 4398 | } | 4146 | } |
| 4399 | 4147 | ||
| 4400 | if (type == BPF_READ && !(cap & BPF_MAP_CAN_READ)) { | 4148 | if (type == BPF_READ && !(cap & BPF_MAP_CAN_READ)) { |
| 4401 | verbose(env, "read from map forbidden, value_size=%d off=%lld size=%d\n", | 4149 | verbose(env, "read from map forbidden, value_size=%d off=%lld size=%d\n", |
| 4402 | map->value_size, reg->smin_value + off, size); | 4150 | map->value_size, reg_smin(reg) + off, size); |
| 4403 | return -EACCES; | 4151 | return -EACCES; |
| 4404 | } | 4152 | } |
| 4405 | 4153 | ||
| @@ -4407,17 +4155,15 @@ static int check_map_access_type(struct bpf_verifier_env *env, u32 regno, · 1 unmodified line | |||
| 4407 | } | 4155 | } |
| 4408 | 4156 | ||
| 4409 | /* check read/write into memory region (e.g., map value, ringbuf sample, etc) */ | 4157 | /* check read/write into memory region (e.g., map value, ringbuf sample, etc) */ |
| 4410 | static int __check_mem_access(struct bpf_verifier_env *env, int regno, | 4158 | static int __check_mem_access(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno, |
| 4411 | int off, int size, u32 mem_size, | 4159 | int off, int size, u32 mem_size, |
| 4412 | bool zero_size_allowed) | 4160 | bool zero_size_allowed) |
| 4413 | { | 4161 | { |
| 4414 | bool size_ok = size > 0 || (size == 0 && zero_size_allowed); | 4162 | bool size_ok = size > 0 || (size == 0 && zero_size_allowed); |
| 4415 | struct bpf_reg_state *reg; | ||
| 4416 | 4163 | ||
| 4417 | if (off >= 0 && size_ok && (u64)off + size <= mem_size) | 4164 | if (off >= 0 && size_ok && (u64)off + size <= mem_size) |
| 4418 | return 0; | 4165 | return 0; |
| 4419 | 4166 | ||
| 4420 | reg = &cur_regs(env)[regno]; | ||
| 4421 | switch (reg->type) { | 4167 | switch (reg->type) { |
| 4422 | case PTR_TO_MAP_KEY: | 4168 | case PTR_TO_MAP_KEY: |
| 4423 | verbose(env, "invalid access to map key, key_size=%d off=%d size=%d\n", | 4169 | verbose(env, "invalid access to map key, key_size=%d off=%d size=%d\n", |
| @@ -4430,8 +4176,8 @@ static int __check_mem_access(struct bpf_verifier_env *env, int regno, · 6 unmodified lines | |||
| 4430 | case PTR_TO_PACKET: | 4176 | case PTR_TO_PACKET: |
| 4431 | case PTR_TO_PACKET_META: | 4177 | case PTR_TO_PACKET_META: |
| 4432 | case PTR_TO_PACKET_END: | 4178 | case PTR_TO_PACKET_END: |
| 4433 | verbose(env, "invalid access to packet, off=%d size=%d, R%d(id=%d,off=%d,r=%d)\n", | 4179 | verbose(env, "invalid access to packet, off=%d size=%d, %s(id=%d,off=%d,r=%d)\n", |
| 4434 | off, size, regno, reg->id, off, mem_size); | 4180 | off, size, reg_arg_name(env, argno), reg->id, off, mem_size); |
| 4435 | break; | 4181 | break; |
| 4436 | case PTR_TO_CTX: | 4182 | case PTR_TO_CTX: |
| 4437 | verbose(env, "invalid access to context, ctx_size=%d off=%d size=%d\n", | 4183 | verbose(env, "invalid access to context, ctx_size=%d off=%d size=%d\n", |
| @@ -4447,13 +4193,10 @@ static int __check_mem_access(struct bpf_verifier_env *env, int regno, · 9 unmodified lines | |||
| 4447 | } | 4193 | } |
| 4448 | 4194 | ||
| 4449 | /* check read/write into a memory region with possible variable offset */ | 4195 | /* check read/write into a memory region with possible variable offset */ |
| 4450 | static int check_mem_region_access(struct bpf_verifier_env *env, u32 regno, | 4196 | static int check_mem_region_access(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno, |
| 4451 | int off, int size, u32 mem_size, | 4197 | int off, int size, u32 mem_size, |
| 4452 | bool zero_size_allowed) | 4198 | bool zero_size_allowed) |
| 4453 | { | 4199 | { |
| 4454 | struct bpf_verifier_state *vstate = env->cur_state; | ||
| 4455 | struct bpf_func_state *state = vstate->frame[vstate->curframe]; | ||
| 4456 | struct bpf_reg_state *reg = &state->regs[regno]; | ||
| 4457 | int err; | 4200 | int err; |
| 4458 | 4201 | ||
| 4459 | /* We may have adjusted the register pointing to memory region, so we | 4202 | /* We may have adjusted the register pointing to memory region, so we |
| @@ -4466,36 +4209,36 @@ static int check_mem_region_access(struct bpf_verifier_env *env, u32 regno, · 6 unmodified lines | |||
| 4466 | * index'es we need to make sure that whatever we use | 4209 | * index'es we need to make sure that whatever we use |
| 4467 | * will have a set floor within our range. | 4210 | * will have a set floor within our range. |
| 4468 | */ | 4211 | */ |
| 4469 | if (reg->smin_value < 0 && | 4212 | if (reg_smin(reg) < 0 && |
| 4470 | (reg->smin_value == S64_MIN || | 4213 | (reg_smin(reg) == S64_MIN || |
| 4471 | (off + reg->smin_value != (s64)(s32)(off + reg->smin_value)) || | 4214 | (off + reg_smin(reg) != (s64)(s32)(off + reg_smin(reg))) || |
| 4472 | reg->smin_value + off < 0)) { | 4215 | reg_smin(reg) + off < 0)) { |
| 4473 | verbose(env, "R%d min value is negative, either use unsigned index or do a if (index >=0) check.\n", | 4216 | verbose(env, "%s min value is negative, either use unsigned index or do a if (index >=0) check.\n", |
| 4474 | regno); | 4217 | reg_arg_name(env, argno)); |
| 4475 | return -EACCES; | 4218 | return -EACCES; |
| 4476 | } | 4219 | } |
| 4477 | err = __check_mem_access(env, regno, reg->smin_value + off, size, | 4220 | err = __check_mem_access(env, reg, argno, reg_smin(reg) + off, size, |
| 4478 | mem_size, zero_size_allowed); | 4221 | mem_size, zero_size_allowed); |
| 4479 | if (err) { | 4222 | if (err) { |
| 4480 | verbose(env, "R%d min value is outside of the allowed memory range\n", | 4223 | verbose(env, "%s min value is outside of the allowed memory range\n", |
| 4481 | regno); | 4224 | reg_arg_name(env, argno)); |
| 4482 | return err; | 4225 | return err; |
| 4483 | } | 4226 | } |
| 4484 | 4227 | ||
| 4485 | /* If we haven't set a max value then we need to bail since we can't be | 4228 | /* If we haven't set a max value then we need to bail since we can't be |
| 4486 | * sure we won't do bad things. | 4229 | * sure we won't do bad things. |
| 4487 | * If reg->umax_value + off could overflow, treat that as unbounded too. | 4230 | * If reg_umax(reg) + off could overflow, treat that as unbounded too. |
| 4488 | */ | 4231 | */ |
| 4489 | if (reg->umax_value >= BPF_MAX_VAR_OFF) { | 4232 | if (reg_umax(reg) >= BPF_MAX_VAR_OFF) { |
| 4490 | verbose(env, "R%d unbounded memory access, make sure to bounds check any such access\n", | 4233 | verbose(env, "%s unbounded memory access, make sure to bounds check any such access\n", |
| 4491 | regno); | 4234 | reg_arg_name(env, argno)); |
| 4492 | return -EACCES; | 4235 | return -EACCES; |
| 4493 | } | 4236 | } |
| 4494 | err = __check_mem_access(env, regno, reg->umax_value + off, size, | 4237 | err = __check_mem_access(env, reg, argno, reg_umax(reg) + off, size, |
| 4495 | mem_size, zero_size_allowed); | 4238 | mem_size, zero_size_allowed); |
| 4496 | if (err) { | 4239 | if (err) { |
| 4497 | verbose(env, "R%d max value is outside of the allowed memory range\n", | 4240 | verbose(env, "%s max value is outside of the allowed memory range\n", |
| 4498 | regno); | 4241 | reg_arg_name(env, argno)); |
| 4499 | return err; | 4242 | return err; |
| 4500 | } | 4243 | } |
| 4501 | 4244 | ||
| @@ -4503,7 +4246,7 @@ static int check_mem_region_access(struct bpf_verifier_env *env, u32 regno, · 1 unmodified line | |||
| 4503 | } | 4246 | } |
| 4504 | 4247 | ||
| 4505 | static int __check_ptr_off_reg(struct bpf_verifier_env *env, | 4248 | static int __check_ptr_off_reg(struct bpf_verifier_env *env, |
| 4506 | const struct bpf_reg_state *reg, int regno, | 4249 | const struct bpf_reg_state *reg, argno_t argno, |
| 4507 | bool fixed_off_ok) | 4250 | bool fixed_off_ok) |
| 4508 | { | 4251 | { |
| 4509 | /* Access to this pointer-typed register or passing it to a helper | 4252 | /* Access to this pointer-typed register or passing it to a helper |
| @@ -4519,15 +4262,15 @@ static int __check_ptr_off_reg(struct bpf_verifier_env *env, · 9 unmodified lines | |||
| 4519 | return -EACCES; | 4262 | return -EACCES; |
| 4520 | } | 4263 | } |
| 4521 | 4264 | ||
| 4522 | if (reg->smin_value < 0) { | 4265 | if (reg_smin(reg) < 0) { |
| 4523 | verbose(env, "negative offset %s ptr R%d off=%lld disallowed\n", | 4266 | verbose(env, "negative offset %s ptr %s off=%lld disallowed\n", |
| 4524 | reg_type_str(env, reg->type), regno, reg->var_off.value); | 4267 | reg_type_str(env, reg->type), reg_arg_name(env, argno), reg->var_off.value); |
| 4525 | return -EACCES; | 4268 | return -EACCES; |
| 4526 | } | 4269 | } |
| 4527 | 4270 | ||
| 4528 | if (!fixed_off_ok && reg->var_off.value != 0) { | 4271 | if (!fixed_off_ok && reg->var_off.value != 0) { |
| 4529 | verbose(env, "dereference of modified %s ptr R%d off=%lld disallowed\n", | 4272 | verbose(env, "dereference of modified %s ptr %s off=%lld disallowed\n", |
| 4530 | reg_type_str(env, reg->type), regno, reg->var_off.value); | 4273 | reg_type_str(env, reg->type), reg_arg_name(env, argno), reg->var_off.value); |
| 4531 | return -EACCES; | 4274 | return -EACCES; |
| 4532 | } | 4275 | } |
| 4533 | 4276 | ||
| @@ -4537,7 +4280,7 @@ static int __check_ptr_off_reg(struct bpf_verifier_env *env, · 3 unmodified lines | |||
| 4537 | static int check_ptr_off_reg(struct bpf_verifier_env *env, | 4280 | static int check_ptr_off_reg(struct bpf_verifier_env *env, |
| 4538 | const struct bpf_reg_state *reg, int regno) | 4281 | const struct bpf_reg_state *reg, int regno) |
| 4539 | { | 4282 | { |
| 4540 | return __check_ptr_off_reg(env, reg, regno, false); | 4283 | return __check_ptr_off_reg(env, reg, argno_from_reg(regno), false); |
| 4541 | } | 4284 | } |
| 4542 | 4285 | ||
| 4543 | static int map_kptr_match_type(struct bpf_verifier_env *env, | 4286 | static int map_kptr_match_type(struct bpf_verifier_env *env, |
| @@ -4573,9 +4316,9 @@ static int map_kptr_match_type(struct bpf_verifier_env *env, · 29 unmodified lines | |||
| 4573 | * referenced PTR_TO_BTF_ID, and that its fixed offset is 0. For the | 4316 | * referenced PTR_TO_BTF_ID, and that its fixed offset is 0. For the |
| 4574 | * normal store of unreferenced kptr, we must ensure var_off is zero. | 4317 | * normal store of unreferenced kptr, we must ensure var_off is zero. |
| 4575 | * Since ref_ptr cannot be accessed directly by BPF insns, check for | 4318 | * Since ref_ptr cannot be accessed directly by BPF insns, check for |
| 4576 | * reg->ref_obj_id is not needed here. | 4319 | * reg->id is not needed here. |
| 4577 | */ | 4320 | */ |
| 4578 | if (__check_ptr_off_reg(env, reg, regno, true)) | 4321 | if (__check_ptr_off_reg(env, reg, argno_from_reg(regno), true)) |
| 4579 | return -EACCES; | 4322 | return -EACCES; |
| 4580 | 4323 | ||
| 4581 | /* A full type match is needed, as BTF can be vmlinux, module or prog BTF, and | 4324 | /* A full type match is needed, as BTF can be vmlinux, module or prog BTF, and |
| @@ -4718,7 +4461,7 @@ static int mark_uptr_ld_reg(struct bpf_verifier_env *env, u32 regno, · 136 unmodified lines | |||
| 4718 | return 0; | 4461 | return 0; |
| 4719 | } | 4462 | } |
| 4720 | 4463 | ||
| 4721 | static int check_map_kptr_access(struct bpf_verifier_env *env, u32 regno, | 4464 | static int check_map_kptr_access(struct bpf_verifier_env *env, |
| 4722 | int value_regno, int insn_idx, | 4465 | int value_regno, int insn_idx, |
| 4723 | struct btf_field *kptr_field) | 4466 | struct btf_field *kptr_field) |
| 4724 | { | 4467 | { |
| @@ -4795,19 +4538,16 @@ static u32 map_mem_size(const struct bpf_map *map) · 70 unmodified lines | |||
| 4795 | } | 4538 | } |
| 4796 | 4539 | ||
| 4797 | /* check read/write into a map element with possible variable offset */ | 4540 | /* check read/write into a map element with possible variable offset */ |
| 4798 | static int check_map_access(struct bpf_verifier_env *env, u32 regno, | 4541 | static int check_map_access(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno, |
| 4799 | int off, int size, bool zero_size_allowed, | 4542 | int off, int size, bool zero_size_allowed, |
| 4800 | enum bpf_access_src src) | 4543 | enum bpf_access_src src) |
| 4801 | { | 4544 | { |
| 4802 | struct bpf_verifier_state *vstate = env->cur_state; | ||
| 4803 | struct bpf_func_state *state = vstate->frame[vstate->curframe]; | ||
| 4804 | struct bpf_reg_state *reg = &state->regs[regno]; | ||
| 4805 | struct bpf_map *map = reg->map_ptr; | 4545 | struct bpf_map *map = reg->map_ptr; |
| 4806 | u32 mem_size = map_mem_size(map); | 4546 | u32 mem_size = map_mem_size(map); |
| 4807 | struct btf_record *rec; | 4547 | struct btf_record *rec; |
| 4808 | int err, i; | 4548 | int err, i; |
| 4809 | 4549 | ||
| 4810 | err = check_mem_region_access(env, regno, off, size, mem_size, zero_size_allowed); | 4550 | err = check_mem_region_access(env, reg, argno, off, size, mem_size, zero_size_allowed); |
| 4811 | if (err) | 4551 | if (err) |
| 4812 | return err; | 4552 | return err; |
| 4813 | 4553 | ||
| @@ -4822,8 +4562,8 @@ static int check_map_access(struct bpf_verifier_env *env, u32 regno, · 8 unmodified lines | |||
| 4822 | * this program. To check that [x1, x2) overlaps with [y1, y2), | 4562 | * this program. To check that [x1, x2) overlaps with [y1, y2), |
| 4823 | * it is sufficient to check x1 < y2 && y1 < x2. | 4563 | * it is sufficient to check x1 < y2 && y1 < x2. |
| 4824 | */ | 4564 | */ |
| 4825 | if (reg->smin_value + off < p + field->size && | 4565 | if (reg_smin(reg) + off < p + field->size && |
| 4826 | p < reg->umax_value + off + size) { | 4566 | p < reg_umax(reg) + off + size) { |
| 4827 | switch (field->type) { | 4567 | switch (field->type) { |
| 4828 | case BPF_KPTR_UNREF: | 4568 | case BPF_KPTR_UNREF: |
| 4829 | case BPF_KPTR_REF: | 4569 | case BPF_KPTR_REF: |
| @@ -4903,30 +4643,29 @@ static bool may_access_direct_pkt_data(struct bpf_verifier_env *env, · 73 unmodified lines | |||
| 4903 | } | 4643 | } |
| 4904 | } | 4644 | } |
| 4905 | 4645 | ||
| 4906 | static int check_packet_access(struct bpf_verifier_env *env, u32 regno, int off, | 4646 | static int check_packet_access(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno, int off, |
| 4907 | int size, bool zero_size_allowed) | 4647 | int size, bool zero_size_allowed) |
| 4908 | { | 4648 | { |
| 4909 | struct bpf_reg_state *reg = reg_state(env, regno); | ||
| 4910 | int err; | 4649 | int err; |
| 4911 | 4650 | ||
| 4912 | if (reg->range < 0) { | 4651 | if (reg->range < 0) { |
| 4913 | verbose(env, "R%d offset is outside of the packet\n", regno); | 4652 | verbose(env, "%s offset is outside of the packet\n", reg_arg_name(env, argno)); |
| 4914 | return -EINVAL; | 4653 | return -EINVAL; |
| 4915 | } | 4654 | } |
| 4916 | 4655 | ||
| 4917 | err = check_mem_region_access(env, regno, off, size, reg->range, zero_size_allowed); | 4656 | err = check_mem_region_access(env, reg, argno, off, size, reg->range, zero_size_allowed); |
| 4918 | if (err) | 4657 | if (err) |
| 4919 | return err; | 4658 | return err; |
| 4920 | 4659 | ||
| 4921 | /* __check_mem_access has made sure "off + size - 1" is within u16. | 4660 | /* __check_mem_access has made sure "off + size - 1" is within u16. |
| 4922 | * reg->umax_value can't be bigger than MAX_PACKET_OFF which is 0xffff, | 4661 | * reg_umax(reg) can't be bigger than MAX_PACKET_OFF which is 0xffff, |
| 4923 | * otherwise find_good_pkt_pointers would have refused to set range info | 4662 | * otherwise find_good_pkt_pointers would have refused to set range info |
| 4924 | * that __check_mem_access would have rejected this pkt access. | 4663 | * that __check_mem_access would have rejected this pkt access. |
| 4925 | * Therefore, "off + reg->umax_value + size - 1" won't overflow u32. | 4664 | * Therefore, "off + reg_umax(reg) + size - 1" won't overflow u32. |
| 4926 | */ | 4665 | */ |
| 4927 | env->prog->aux->max_pkt_offset = | 4666 | env->prog->aux->max_pkt_offset = |
| 4928 | max_t(u32, env->prog->aux->max_pkt_offset, | 4667 | max_t(u32, env->prog->aux->max_pkt_offset, |
| 4929 | off + reg->umax_value + size - 1); | 4668 | off + reg_umax(reg) + size - 1); |
| 4930 | 4669 | ||
| 4931 | return 0; | 4670 | return 0; |
| 4932 | } | 4671 | } |
| @@ -4950,8 +4689,8 @@ static int __check_ctx_access(struct bpf_verifier_env *env, int insn_idx, int of · 17 unmodified lines | |||
| 4950 | * type of narrower access. | 4689 | * type of narrower access. |
| 4951 | */ | 4690 | */ |
| 4952 | if (base_type(info->reg_type) == PTR_TO_BTF_ID) { | 4691 | if (base_type(info->reg_type) == PTR_TO_BTF_ID) { |
| 4953 | if (info->ref_obj_id && | 4692 | if (info->ref_id && |
| 4954 | !find_reference_state(env->cur_state, info->ref_obj_id)) { | 4693 | !find_reference_state(env->cur_state, info->ref_id)) { |
| 4955 | verbose(env, "invalid bpf_context access off=%d. Reference may already be released\n", | 4694 | verbose(env, "invalid bpf_context access off=%d. Reference may already be released\n", |
| 4956 | off); | 4695 | off); |
| 4957 | return -EACCES; | 4696 | return -EACCES; |
| @@ -4969,7 +4708,7 @@ static int __check_ctx_access(struct bpf_verifier_env *env, int insn_idx, int of · 11 unmodified lines | |||
| 4969 | return -EACCES; | 4708 | return -EACCES; |
| 4970 | } | 4709 | } |
| 4971 | 4710 | ||
| 4972 | static int check_ctx_access(struct bpf_verifier_env *env, int insn_idx, u32 regno, | 4711 | static int check_ctx_access(struct bpf_verifier_env *env, int insn_idx, struct bpf_reg_state *reg, argno_t argno, |
| 4973 | int off, int access_size, enum bpf_access_type t, | 4712 | int off, int access_size, enum bpf_access_type t, |
| 4974 | struct bpf_insn_access_aux *info) | 4713 | struct bpf_insn_access_aux *info) |
| 4975 | { | 4714 | { |
| @@ -4979,17 +4718,15 @@ static int check_ctx_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 3 unmodified lines | |||
| 4979 | */ | 4718 | */ |
| 4980 | bool var_off_ok = is_var_ctx_off_allowed(env->prog); | 4719 | bool var_off_ok = is_var_ctx_off_allowed(env->prog); |
| 4981 | bool fixed_off_ok = !env->ops->convert_ctx_access; | 4720 | bool fixed_off_ok = !env->ops->convert_ctx_access; |
| 4982 | struct bpf_reg_state *regs = cur_regs(env); | ||
| 4983 | struct bpf_reg_state *reg = regs + regno; | ||
| 4984 | int err; | 4721 | int err; |
| 4985 | 4722 | ||
| 4986 | if (var_off_ok) | 4723 | if (var_off_ok) |
| 4987 | err = check_mem_region_access(env, regno, off, access_size, U16_MAX, false); | 4724 | err = check_mem_region_access(env, reg, argno, off, access_size, U16_MAX, false); |
| 4988 | else | 4725 | else |
| 4989 | err = __check_ptr_off_reg(env, reg, regno, fixed_off_ok); | 4726 | err = __check_ptr_off_reg(env, reg, argno, fixed_off_ok); |
| 4990 | if (err) | 4727 | if (err) |
| 4991 | return err; | 4728 | return err; |
| 4992 | off += reg->umax_value; | 4729 | off += reg_umax(reg); |
| 4993 | 4730 | ||
| 4994 | err = __check_ctx_access(env, insn_idx, off, access_size, t, info); | 4731 | err = __check_ctx_access(env, insn_idx, off, access_size, t, info); |
| 4995 | if (err) | 4732 | if (err) |
| @@ -4997,9 +4734,21 @@ static int check_ctx_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 1 unmodified line | |||
| 4997 | return err; | 4734 | return err; |
| 4998 | } | 4735 | } |
| 4999 | 4736 | ||
| 5000 | static int check_flow_keys_access(struct bpf_verifier_env *env, int off, | 4737 | static int check_flow_keys_access(struct bpf_verifier_env *env, |
| 5001 | int size) | 4738 | struct bpf_reg_state *reg, argno_t argno, |
| 4739 | int off, int size) | ||
| 5002 | { | 4740 | { |
| 4741 | /* Only a constant offset is allowed here; fold it into off. */ | ||
| 4742 | if (!tnum_is_const(reg->var_off)) { | ||
| 4743 | char tn_buf[48]; | ||
| 4744 | |||
| 4745 | tnum_strn(tn_buf, sizeof(tn_buf), reg->var_off); | ||
| 4746 | verbose(env, "%s invalid variable offset to flow keys: off=%d, var_off=%s\n", | ||
| 4747 | reg_arg_name(env, argno), off, tn_buf); | ||
| 4748 | return -EACCES; | ||
| 4749 | } | ||
| 4750 | off += reg->var_off.value; | ||
| 4751 | |||
| 5003 | if (size < 0 || off < 0 || | 4752 | if (size < 0 || off < 0 || |
| 5004 | (u64)off + size > sizeof(struct bpf_flow_keys)) { | 4753 | (u64)off + size > sizeof(struct bpf_flow_keys)) { |
| 5005 | verbose(env, "invalid access to flow keys off=%d size=%d\n", | 4754 | verbose(env, "invalid access to flow keys off=%d size=%d\n", |
| @@ -5010,16 +4759,15 @@ static int check_flow_keys_access(struct bpf_verifier_env *env, int off, · 4 unmodified lines | |||
| 5010 | } | 4759 | } |
| 5011 | 4760 | ||
| 5012 | static int check_sock_access(struct bpf_verifier_env *env, int insn_idx, | 4761 | static int check_sock_access(struct bpf_verifier_env *env, int insn_idx, |
| 5013 | u32 regno, int off, int size, | 4762 | struct bpf_reg_state *reg, argno_t argno, int off, int size, |
| 5014 | enum bpf_access_type t) | 4763 | enum bpf_access_type t) |
| 5015 | { | 4764 | { |
| 5016 | struct bpf_reg_state *reg = reg_state(env, regno); | ||
| 5017 | struct bpf_insn_access_aux info = {}; | 4765 | struct bpf_insn_access_aux info = {}; |
| 5018 | bool valid; | 4766 | bool valid; |
| 5019 | 4767 | ||
| 5020 | if (reg->smin_value < 0) { | 4768 | if (reg_smin(reg) < 0) { |
| 5021 | verbose(env, "R%d min value is negative, either use unsigned index or do a if (index >=0) check.\n", | 4769 | verbose(env, "%s min value is negative, either use unsigned index or do a if (index >=0) check.\n", |
| 5022 | regno); | 4770 | reg_arg_name(env, argno)); |
| 5023 | return -EACCES; | 4771 | return -EACCES; |
| 5024 | } | 4772 | } |
| 5025 | 4773 | ||
| @@ -5047,8 +4795,8 @@ static int check_sock_access(struct bpf_verifier_env *env, int insn_idx, · 21 unmodified lines | |||
| 5047 | return 0; | 4795 | return 0; |
| 5048 | } | 4796 | } |
| 5049 | 4797 | ||
| 5050 | verbose(env, "R%d invalid %s access off=%d size=%d\n", | 4798 | verbose(env, "%s invalid %s access off=%d size=%d\n", |
| 5051 | regno, reg_type_str(env, reg->type), off, size); | 4799 | reg_arg_name(env, argno), reg_type_str(env, reg->type), off, size); |
| 5052 | 4800 | ||
| 5053 | return -EACCES; | 4801 | return -EACCES; |
| 5054 | } | 4802 | } |
| @@ -5123,10 +4871,10 @@ static u32 *reg2btf_ids[__BPF_REG_TYPE_MAX] = { · 68 unmodified lines | |||
| 5123 | [CONST_PTR_TO_MAP] = btf_bpf_map_id, | 4871 | [CONST_PTR_TO_MAP] = btf_bpf_map_id, |
| 5124 | }; | 4872 | }; |
| 5125 | 4873 | ||
| 5126 | static bool is_trusted_reg(const struct bpf_reg_state *reg) | 4874 | static bool is_trusted_reg(struct bpf_verifier_env *env, const struct bpf_reg_state *reg) |
| 5127 | { | 4875 | { |
| 5128 | /* A referenced register is always trusted. */ | 4876 | /* A referenced register is always trusted. */ |
| 5129 | if (reg->ref_obj_id) | 4877 | if (reg_is_referenced(env, reg)) |
| 5130 | return true; | 4878 | return true; |
| 5131 | 4879 | ||
| 5132 | /* Types listed in the reg2btf_ids are always trusted */ | 4880 | /* Types listed in the reg2btf_ids are always trusted */ |
| @@ -5368,7 +5116,10 @@ process_func: · 235 unmodified lines | |||
| 5368 | } | 5116 | } |
| 5369 | 5117 | ||
| 5370 | subprog_depth = round_up_stack_depth(env, subprog[idx].stack_depth); | 5118 | subprog_depth = round_up_stack_depth(env, subprog[idx].stack_depth); |
| 5371 | if (priv_stack_supported) { | 5119 | if (IS_ENABLED(CONFIG_X86_64) && subprog[idx].stack_arg_cnt) { |
| 5120 | /* x86-64 uses R9 for both private stack frame pointer and arg6. */ | ||
| 5121 | subprog[idx].priv_stack_mode = NO_PRIV_STACK; | ||
| 5122 | } else if (priv_stack_supported) { | ||
| 5372 | /* Request private stack support only if the subprog stack | 5123 | /* Request private stack support only if the subprog stack |
| 5373 | * depth is no less than BPF_PRIV_STACK_MIN_SIZE. This is to | 5124 | * depth is no less than BPF_PRIV_STACK_MIN_SIZE. This is to |
| 5374 | * avoid jit penalty if the stack usage is small. | 5125 | * avoid jit penalty if the stack usage is small. |
| @@ -5379,6 +5130,8 @@ process_func: · 4 unmodified lines | |||
| 5379 | } | 5130 | } |
| 5380 | 5131 | ||
| 5381 | if (subprog[idx].priv_stack_mode == PRIV_STACK_ADAPTIVE) { | 5132 | if (subprog[idx].priv_stack_mode == PRIV_STACK_ADAPTIVE) { |
| 5133 | if (subprog_depth > env->max_stack_depth) | ||
| 5134 | env->max_stack_depth = subprog_depth; | ||
| 5382 | if (subprog_depth > MAX_BPF_STACK) { | 5135 | if (subprog_depth > MAX_BPF_STACK) { |
| 5383 | verbose(env, "stack size of subprog %d is %d. Too large\n", | 5136 | verbose(env, "stack size of subprog %d is %d. Too large\n", |
| 5384 | idx, subprog_depth); | 5137 | idx, subprog_depth); |
| @@ -5386,6 +5139,8 @@ process_func: · 1 unmodified line | |||
| 5386 | } | 5139 | } |
| 5387 | } else { | 5140 | } else { |
| 5388 | depth += subprog_depth; | 5141 | depth += subprog_depth; |
| 5142 | if (depth > env->max_stack_depth) | ||
| 5143 | env->max_stack_depth = depth; | ||
| 5389 | if (depth > MAX_BPF_STACK) { | 5144 | if (depth > MAX_BPF_STACK) { |
| 5390 | total = 0; | 5145 | total = 0; |
| 5391 | for (tmp = idx; tmp >= 0; tmp = dinfo[tmp].caller) | 5146 | for (tmp = idx; tmp >= 0; tmp = dinfo[tmp].caller) |
| @@ -5472,14 +5227,23 @@ continue_func: · 80 unmodified lines | |||
| 5472 | * this info will be utilized by JIT so that we will be preserving the | 5227 | * this info will be utilized by JIT so that we will be preserving the |
| 5473 | * tail call counter throughout bpf2bpf calls combined with tailcalls | 5228 | * tail call counter throughout bpf2bpf calls combined with tailcalls |
| 5474 | */ | 5229 | */ |
| 5475 | if (tail_call_reachable) | 5230 | if (tail_call_reachable) { |
| 5476 | for (tmp = idx; tmp >= 0; tmp = dinfo[tmp].caller) { | 5231 | for (tmp = idx; tmp >= 0; tmp = dinfo[tmp].caller) { |
| 5477 | if (subprog[tmp].is_exception_cb) { | 5232 | if (subprog[tmp].is_exception_cb) { |
| 5478 | verbose(env, "cannot tail call within exception cb\n"); | 5233 | verbose(env, "cannot tail call within exception cb\n"); |
| 5479 | return -EINVAL; | 5234 | return -EINVAL; |
| 5480 | } | 5235 | } |
| 5236 | if (subprog[tmp].stack_arg_cnt) { | ||
| 5237 | verbose(env, "tail_calls are not allowed in programs with stack args\n"); | ||
| 5238 | return -EINVAL; | ||
| 5239 | } | ||
| 5481 | subprog[tmp].tail_call_reachable = true; | 5240 | subprog[tmp].tail_call_reachable = true; |
| 5482 | } | 5241 | } |
| 5242 | } else if (!idx && subprog[0].has_tail_call && subprog[0].stack_arg_cnt) { | ||
| 5243 | verbose(env, "tail_calls are not allowed in programs with stack args\n"); | ||
| 5244 | return -EINVAL; | ||
| 5245 | } | ||
| 5246 | |||
| 5483 | if (subprog[0].tail_call_reachable) | 5247 | if (subprog[0].tail_call_reachable) |
| 5484 | env->prog->aux->tail_call_reachable = true; | 5248 | env->prog->aux->tail_call_reachable = true; |
| 5485 | 5249 | ||
| @@ -5498,6 +5262,9 @@ continue_func: · 12 unmodified lines | |||
| 5498 | frame = dinfo[idx].frame; | 5262 | frame = dinfo[idx].frame; |
| 5499 | i = dinfo[idx].ret_insn; | 5263 | i = dinfo[idx].ret_insn; |
| 5500 | 5264 | ||
| 5265 | /* reset tail_call_reachable to the parent's actual state */ | ||
| 5266 | tail_call_reachable = subprog[idx].tail_call_reachable; | ||
| 5267 | |||
| 5501 | goto continue_func; | 5268 | goto continue_func; |
| 5502 | } | 5269 | } |
| 5503 | 5270 | ||
| @@ -5558,12 +5325,12 @@ static int check_max_stack_depth(struct bpf_verifier_env *env) · 54 unmodified lines | |||
| 5558 | static int __check_buffer_access(struct bpf_verifier_env *env, | 5325 | static int __check_buffer_access(struct bpf_verifier_env *env, |
| 5559 | const char *buf_info, | 5326 | const char *buf_info, |
| 5560 | const struct bpf_reg_state *reg, | 5327 | const struct bpf_reg_state *reg, |
| 5561 | int regno, int off, int size) | 5328 | argno_t argno, int off, int size) |
| 5562 | { | 5329 | { |
| 5563 | if (off < 0) { | 5330 | if (off < 0) { |
| 5564 | verbose(env, | 5331 | verbose(env, |
| 5565 | "R%d invalid %s buffer access: off=%d, size=%d\n", | 5332 | "%s invalid %s buffer access: off=%d, size=%d\n", |
| 5566 | regno, buf_info, off, size); | 5333 | reg_arg_name(env, argno), buf_info, off, size); |
| 5567 | return -EACCES; | 5334 | return -EACCES; |
| 5568 | } | 5335 | } |
| 5569 | if (!tnum_is_const(reg->var_off)) { | 5336 | if (!tnum_is_const(reg->var_off)) { |
| @@ -5571,8 +5338,8 @@ static int __check_buffer_access(struct bpf_verifier_env *env, · 1 unmodified line | |||
| 5571 | 5338 | ||
| 5572 | tnum_strn(tn_buf, sizeof(tn_buf), reg->var_off); | 5339 | tnum_strn(tn_buf, sizeof(tn_buf), reg->var_off); |
| 5573 | verbose(env, | 5340 | verbose(env, |
| 5574 | "R%d invalid variable buffer offset: off=%d, var_off=%s\n", | 5341 | "%s invalid variable buffer offset: off=%d, var_off=%s\n", |
| 5575 | regno, off, tn_buf); | 5342 | reg_arg_name(env, argno), off, tn_buf); |
| 5576 | return -EACCES; | 5343 | return -EACCES; |
| 5577 | } | 5344 | } |
| 5578 | 5345 | ||
| @@ -5581,11 +5348,11 @@ static int __check_buffer_access(struct bpf_verifier_env *env, · 2 unmodified lines | |||
| 5581 | 5348 | ||
| 5582 | static int check_tp_buffer_access(struct bpf_verifier_env *env, | 5349 | static int check_tp_buffer_access(struct bpf_verifier_env *env, |
| 5583 | const struct bpf_reg_state *reg, | 5350 | const struct bpf_reg_state *reg, |
| 5584 | int regno, int off, int size) | 5351 | argno_t argno, int off, int size) |
| 5585 | { | 5352 | { |
| 5586 | int err; | 5353 | int err; |
| 5587 | 5354 | ||
| 5588 | err = __check_buffer_access(env, "tracepoint", reg, regno, off, size); | 5355 | err = __check_buffer_access(env, "tracepoint", reg, argno, off, size); |
| 5589 | if (err) | 5356 | if (err) |
| 5590 | return err; | 5357 | return err; |
| 5591 | 5358 | ||
| @@ -5597,14 +5364,14 @@ static int check_tp_buffer_access(struct bpf_verifier_env *env, · 5 unmodified lines | |||
| 5597 | 5364 | ||
| 5598 | static int check_buffer_access(struct bpf_verifier_env *env, | 5365 | static int check_buffer_access(struct bpf_verifier_env *env, |
| 5599 | const struct bpf_reg_state *reg, | 5366 | const struct bpf_reg_state *reg, |
| 5600 | int regno, int off, int size, | 5367 | argno_t argno, int off, int size, |
| 5601 | bool zero_size_allowed, | 5368 | bool zero_size_allowed, |
| 5602 | u32 *max_access) | 5369 | u32 *max_access) |
| 5603 | { | 5370 | { |
| 5604 | const char *buf_info = type_is_rdonly_mem(reg->type) ? "rdonly" : "rdwr"; | 5371 | const char *buf_info = type_is_rdonly_mem(reg->type) ? "rdonly" : "rdwr"; |
| 5605 | int err; | 5372 | int err; |
| 5606 | 5373 | ||
| 5607 | err = __check_buffer_access(env, buf_info, reg, regno, off, size); | 5374 | err = __check_buffer_access(env, buf_info, reg, argno, off, size); |
| 5608 | if (err) | 5375 | if (err) |
| 5609 | return err; | 5376 | return err; |
| 5610 | 5377 | ||
| @@ -5617,7 +5384,7 @@ static int check_buffer_access(struct bpf_verifier_env *env, · 6 unmodified lines | |||
| 5617 | static void zext_32_to_64(struct bpf_reg_state *reg) | 5384 | static void zext_32_to_64(struct bpf_reg_state *reg) |
| 5618 | { | 5385 | { |
| 5619 | reg->var_off = tnum_subreg(reg->var_off); | 5386 | reg->var_off = tnum_subreg(reg->var_off); |
| 5620 | __reg_assign_32_into_64(reg); | 5387 | reg_set_urange64(reg, reg_u32_min(reg), reg_u32_max(reg)); |
| 5621 | } | 5388 | } |
| 5622 | 5389 | ||
| 5623 | /* truncate register to smaller size (in bytes) | 5390 | /* truncate register to smaller size (in bytes) |
| @@ -5632,15 +5399,10 @@ static void coerce_reg_to_size(struct bpf_reg_state *reg, int size) · 8 unmodified lines | |||
| 5632 | 5399 | ||
| 5633 | /* fix arithmetic bounds */ | 5400 | /* fix arithmetic bounds */ |
| 5634 | mask = ((u64)1 << (size * 8)) - 1; | 5401 | mask = ((u64)1 << (size * 8)) - 1; |
| 5635 | if ((reg->umin_value & ~mask) == (reg->umax_value & ~mask)) { | 5402 | if ((reg_umin(reg) & ~mask) == (reg_umax(reg) & ~mask)) |
| 5636 | reg->umin_value &= mask; | 5403 | reg_set_urange64(reg, reg_umin(reg) & mask, reg_umax(reg) & mask); |
| 5637 | reg->umax_value &= mask; | 5404 | else |
| 5638 | } else { | 5405 | reg_set_urange64(reg, 0, mask); |
| 5639 | reg->umin_value = 0; | ||
| 5640 | reg->umax_value = mask; | ||
| 5641 | } | ||
| 5642 | reg->smin_value = reg->umin_value; | ||
| 5643 | reg->smax_value = reg->umax_value; | ||
| 5644 | 5406 | ||
| 5645 | /* If size is smaller than 32bit register the 32bit register | 5407 | /* If size is smaller than 32bit register the 32bit register |
| 5646 | * values are also truncated so we push 64-bit bounds into | 5408 | * values are also truncated so we push 64-bit bounds into |
| @@ -5655,19 +5417,16 @@ static void coerce_reg_to_size(struct bpf_reg_state *reg, int size) · 8 unmodified lines | |||
| 5655 | static void set_sext64_default_val(struct bpf_reg_state *reg, int size) | 5417 | static void set_sext64_default_val(struct bpf_reg_state *reg, int size) |
| 5656 | { | 5418 | { |
| 5657 | if (size == 1) { | 5419 | if (size == 1) { |
| 5658 | reg->smin_value = reg->s32_min_value = S8_MIN; | 5420 | reg_set_srange64(reg, S8_MIN, S8_MAX); |
| 5659 | reg->smax_value = reg->s32_max_value = S8_MAX; | 5421 | reg_set_srange32(reg, S8_MIN, S8_MAX); |
| 5660 | } else if (size == 2) { | 5422 | } else if (size == 2) { |
| 5661 | reg->smin_value = reg->s32_min_value = S16_MIN; | 5423 | reg_set_srange64(reg, S16_MIN, S16_MAX); |
| 5662 | reg->smax_value = reg->s32_max_value = S16_MAX; | 5424 | reg_set_srange32(reg, S16_MIN, S16_MAX); |
| 5663 | } else { | 5425 | } else { |
| 5664 | /* size == 4 */ | 5426 | /* size == 4 */ |
| 5665 | reg->smin_value = reg->s32_min_value = S32_MIN; | 5427 | reg_set_srange64(reg, S32_MIN, S32_MAX); |
| 5666 | reg->smax_value = reg->s32_max_value = S32_MAX; | 5428 | reg_set_srange32(reg, S32_MIN, S32_MAX); |
| 5667 | } | 5429 | } |
| 5668 | reg->umin_value = reg->u32_min_value = 0; | ||
| 5669 | reg->umax_value = U64_MAX; | ||
| 5670 | reg->u32_max_value = U32_MAX; | ||
| 5671 | reg->var_off = tnum_unknown; | 5430 | reg->var_off = tnum_unknown; |
| 5672 | } | 5431 | } |
| 5673 | 5432 | ||
| @@ -5688,29 +5447,27 @@ static void coerce_reg_to_size_sx(struct bpf_reg_state *reg, int size) · 14 unmodified lines | |||
| 5688 | reg->var_off = tnum_const((s32)u64_cval); | 5447 | reg->var_off = tnum_const((s32)u64_cval); |
| 5689 | 5448 | ||
| 5690 | u64_cval = reg->var_off.value; | 5449 | u64_cval = reg->var_off.value; |
| 5691 | reg->smax_value = reg->smin_value = u64_cval; | 5450 | reg->r64 = cnum64_from_urange(u64_cval, u64_cval); |
| 5692 | reg->umax_value = reg->umin_value = u64_cval; | 5451 | reg->r32 = cnum32_from_urange((u32)u64_cval, (u32)u64_cval); |
| 5693 | reg->s32_max_value = reg->s32_min_value = u64_cval; | ||
| 5694 | reg->u32_max_value = reg->u32_min_value = u64_cval; | ||
| 5695 | return; | 5452 | return; |
| 5696 | } | 5453 | } |
| 5697 | 5454 | ||
| 5698 | top_smax_value = ((u64)reg->smax_value >> num_bits) << num_bits; | 5455 | top_smax_value = ((u64)reg_smax(reg) >> num_bits) << num_bits; |
| 5699 | top_smin_value = ((u64)reg->smin_value >> num_bits) << num_bits; | 5456 | top_smin_value = ((u64)reg_smin(reg) >> num_bits) << num_bits; |
| 5700 | 5457 | ||
| 5701 | if (top_smax_value != top_smin_value) | 5458 | if (top_smax_value != top_smin_value) |
| 5702 | goto out; | 5459 | goto out; |
| 5703 | 5460 | ||
| 5704 | /* find the s64_min and s64_min after sign extension */ | 5461 | /* find the s64_min and s64_min after sign extension */ |
| 5705 | if (size == 1) { | 5462 | if (size == 1) { |
| 5706 | init_s64_max = (s8)reg->smax_value; | 5463 | init_s64_max = (s8)reg_smax(reg); |
| 5707 | init_s64_min = (s8)reg->smin_value; | 5464 | init_s64_min = (s8)reg_smin(reg); |
| 5708 | } else if (size == 2) { | 5465 | } else if (size == 2) { |
| 5709 | init_s64_max = (s16)reg->smax_value; | 5466 | init_s64_max = (s16)reg_smax(reg); |
| 5710 | init_s64_min = (s16)reg->smin_value; | 5467 | init_s64_min = (s16)reg_smin(reg); |
| 5711 | } else { | 5468 | } else { |
| 5712 | init_s64_max = (s32)reg->smax_value; | 5469 | init_s64_max = (s32)reg_smax(reg); |
| 5713 | init_s64_min = (s32)reg->smin_value; | 5470 | init_s64_min = (s32)reg_smin(reg); |
| 5714 | } | 5471 | } |
| 5715 | 5472 | ||
| 5716 | s64_max = max(init_s64_max, init_s64_min); | 5473 | s64_max = max(init_s64_max, init_s64_min); |
| @@ -5718,10 +5475,8 @@ static void coerce_reg_to_size_sx(struct bpf_reg_state *reg, int size) · 1 unmodified line | |||
| 5718 | 5475 | ||
| 5719 | /* both of s64_max/s64_min positive or negative */ | 5476 | /* both of s64_max/s64_min positive or negative */ |
| 5720 | if ((s64_max >= 0) == (s64_min >= 0)) { | 5477 | if ((s64_max >= 0) == (s64_min >= 0)) { |
| 5721 | reg->s32_min_value = reg->smin_value = s64_min; | 5478 | reg_set_srange64(reg, s64_min, s64_max); |
| 5722 | reg->s32_max_value = reg->smax_value = s64_max; | 5479 | reg_set_srange32(reg, s64_min, s64_max); |
| 5723 | reg->u32_min_value = reg->umin_value = s64_min; | ||
| 5724 | reg->u32_max_value = reg->umax_value = s64_max; | ||
| 5725 | reg->var_off = tnum_range(s64_min, s64_max); | 5480 | reg->var_off = tnum_range(s64_min, s64_max); |
| 5726 | return; | 5481 | return; |
| 5727 | } | 5482 | } |
| @@ -5732,16 +5487,11 @@ out: · 4 unmodified lines | |||
| 5732 | 5487 | ||
| 5733 | static void set_sext32_default_val(struct bpf_reg_state *reg, int size) | 5488 | static void set_sext32_default_val(struct bpf_reg_state *reg, int size) |
| 5734 | { | 5489 | { |
| 5735 | if (size == 1) { | 5490 | if (size == 1) |
| 5736 | reg->s32_min_value = S8_MIN; | 5491 | reg_set_srange32(reg, S8_MIN, S8_MAX); |
| 5737 | reg->s32_max_value = S8_MAX; | 5492 | else |
| 5738 | } else { | ||
| 5739 | /* size == 2 */ | 5493 | /* size == 2 */ |
| 5740 | reg->s32_min_value = S16_MIN; | 5494 | reg_set_srange32(reg, S16_MIN, S16_MAX); |
| 5741 | reg->s32_max_value = S16_MAX; | ||
| 5742 | } | ||
| 5743 | reg->u32_min_value = 0; | ||
| 5744 | reg->u32_max_value = U32_MAX; | ||
| 5745 | reg->var_off = tnum_subreg(tnum_unknown); | 5495 | reg->var_off = tnum_subreg(tnum_unknown); |
| 5746 | } | 5496 | } |
| 5747 | 5497 | ||
| @@ -5759,34 +5509,30 @@ static void coerce_subreg_to_size_sx(struct bpf_reg_state *reg, int size) · 11 unmodified lines | |||
| 5759 | reg->var_off = tnum_const((s16)u32_val); | 5509 | reg->var_off = tnum_const((s16)u32_val); |
| 5760 | 5510 | ||
| 5761 | u32_val = reg->var_off.value; | 5511 | u32_val = reg->var_off.value; |
| 5762 | reg->s32_min_value = reg->s32_max_value = u32_val; | 5512 | reg_set_srange32(reg, u32_val, u32_val); |
| 5763 | reg->u32_min_value = reg->u32_max_value = u32_val; | ||
| 5764 | return; | 5513 | return; |
| 5765 | } | 5514 | } |
| 5766 | 5515 | ||
| 5767 | top_smax_value = ((u32)reg->s32_max_value >> num_bits) << num_bits; | 5516 | top_smax_value = ((u32)reg_s32_max(reg) >> num_bits) << num_bits; |
| 5768 | top_smin_value = ((u32)reg->s32_min_value >> num_bits) << num_bits; | 5517 | top_smin_value = ((u32)reg_s32_min(reg) >> num_bits) << num_bits; |
| 5769 | 5518 | ||
| 5770 | if (top_smax_value != top_smin_value) | 5519 | if (top_smax_value != top_smin_value) |
| 5771 | goto out; | 5520 | goto out; |
| 5772 | 5521 | ||
| 5773 | /* find the s32_min and s32_min after sign extension */ | 5522 | /* find the s32_min and s32_min after sign extension */ |
| 5774 | if (size == 1) { | 5523 | if (size == 1) { |
| 5775 | init_s32_max = (s8)reg->s32_max_value; | 5524 | init_s32_max = (s8)reg_s32_max(reg); |
| 5776 | init_s32_min = (s8)reg->s32_min_value; | 5525 | init_s32_min = (s8)reg_s32_min(reg); |
| 5777 | } else { | 5526 | } else { |
| 5778 | /* size == 2 */ | 5527 | /* size == 2 */ |
| 5779 | init_s32_max = (s16)reg->s32_max_value; | 5528 | init_s32_max = (s16)reg_s32_max(reg); |
| 5780 | init_s32_min = (s16)reg->s32_min_value; | 5529 | init_s32_min = (s16)reg_s32_min(reg); |
| 5781 | } | 5530 | } |
| 5782 | s32_max = max(init_s32_max, init_s32_min); | 5531 | s32_max = max(init_s32_max, init_s32_min); |
| 5783 | s32_min = min(init_s32_max, init_s32_min); | 5532 | s32_min = min(init_s32_max, init_s32_min); |
| 5784 | 5533 | ||
| 5785 | if ((s32_min >= 0) == (s32_max >= 0)) { | 5534 | if ((s32_min >= 0) == (s32_max >= 0)) { |
| 5786 | reg->s32_min_value = s32_min; | 5535 | reg_set_srange32(reg, s32_min, s32_max); |
| 5787 | reg->s32_max_value = s32_max; | ||
| 5788 | reg->u32_min_value = (u32)s32_min; | ||
| 5789 | reg->u32_max_value = (u32)s32_max; | ||
| 5790 | reg->var_off = tnum_subreg(tnum_range(s32_min, s32_max)); | 5536 | reg->var_off = tnum_subreg(tnum_range(s32_min, s32_max)); |
| 5791 | return; | 5537 | return; |
| 5792 | } | 5538 | } |
| @@ -5976,12 +5722,11 @@ static bool type_is_trusted_or_null(struct bpf_verifier_env *env, · 183 unmodified lines | |||
| 5976 | } | 5722 | } |
| 5977 | 5723 | ||
| 5978 | static int check_ptr_to_btf_access(struct bpf_verifier_env *env, | 5724 | static int check_ptr_to_btf_access(struct bpf_verifier_env *env, |
| 5979 | struct bpf_reg_state *regs, | 5725 | struct bpf_reg_state *regs, struct bpf_reg_state *reg, |
| 5980 | int regno, int off, int size, | 5726 | argno_t argno, int off, int size, |
| 5981 | enum bpf_access_type atype, | 5727 | enum bpf_access_type atype, |
| 5982 | int value_regno) | 5728 | int value_regno) |
| 5983 | { | 5729 | { |
| 5984 | struct bpf_reg_state *reg = regs + regno; | ||
| 5985 | const struct btf_type *t = btf_type_by_id(reg->btf, reg->btf_id); | 5730 | const struct btf_type *t = btf_type_by_id(reg->btf, reg->btf_id); |
| 5986 | const char *tname = btf_name_by_offset(reg->btf, t->name_off); | 5731 | const char *tname = btf_name_by_offset(reg->btf, t->name_off); |
| 5987 | const char *field_name = NULL; | 5732 | const char *field_name = NULL; |
| @@ -6007,8 +5752,8 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env, · 19 unmodified lines | |||
| 6007 | 5752 | ||
| 6008 | tnum_strn(tn_buf, sizeof(tn_buf), reg->var_off); | 5753 | tnum_strn(tn_buf, sizeof(tn_buf), reg->var_off); |
| 6009 | verbose(env, | 5754 | verbose(env, |
| 6010 | "R%d is ptr_%s invalid variable offset: off=%d, var_off=%s\n", | 5755 | "%s is ptr_%s invalid variable offset: off=%d, var_off=%s\n", |
| 6011 | regno, tname, off, tn_buf); | 5756 | reg_arg_name(env, argno), tname, off, tn_buf); |
| 6012 | return -EACCES; | 5757 | return -EACCES; |
| 6013 | } | 5758 | } |
| 6014 | 5759 | ||
| @@ -6016,22 +5761,22 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env, · 1 unmodified line | |||
| 6016 | 5761 | ||
| 6017 | if (off < 0) { | 5762 | if (off < 0) { |
| 6018 | verbose(env, | 5763 | verbose(env, |
| 6019 | "R%d is ptr_%s invalid negative access: off=%d\n", | 5764 | "%s is ptr_%s invalid negative access: off=%d\n", |
| 6020 | regno, tname, off); | 5765 | reg_arg_name(env, argno), tname, off); |
| 6021 | return -EACCES; | 5766 | return -EACCES; |
| 6022 | } | 5767 | } |
| 6023 | 5768 | ||
| 6024 | if (reg->type & MEM_USER) { | 5769 | if (reg->type & MEM_USER) { |
| 6025 | verbose(env, | 5770 | verbose(env, |
| 6026 | "R%d is ptr_%s access user memory: off=%d\n", | 5771 | "%s is ptr_%s access user memory: off=%d\n", |
| 6027 | regno, tname, off); | 5772 | reg_arg_name(env, argno), tname, off); |
| 6028 | return -EACCES; | 5773 | return -EACCES; |
| 6029 | } | 5774 | } |
| 6030 | 5775 | ||
| 6031 | if (reg->type & MEM_PERCPU) { | 5776 | if (reg->type & MEM_PERCPU) { |
| 6032 | verbose(env, | 5777 | verbose(env, |
| 6033 | "R%d is ptr_%s access percpu memory: off=%d\n", | 5778 | "%s is ptr_%s access percpu memory: off=%d\n", |
| 6034 | regno, tname, off); | 5779 | reg_arg_name(env, argno), tname, off); |
| 6035 | return -EACCES; | 5780 | return -EACCES; |
| 6036 | } | 5781 | } |
| 6037 | 5782 | ||
| @@ -6043,7 +5788,7 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env, · 5 unmodified lines | |||
| 6043 | ret = env->ops->btf_struct_access(&env->log, reg, off, size); | 5788 | ret = env->ops->btf_struct_access(&env->log, reg, off, size); |
| 6044 | } else { | 5789 | } else { |
| 6045 | /* Writes are permitted with default btf_struct_access for | 5790 | /* Writes are permitted with default btf_struct_access for |
| 6046 | * program allocated objects (which always have ref_obj_id > 0), | 5791 | * program allocated objects (which always have id > 0), |
| 6047 | * but not for untrusted PTR_TO_BTF_ID | MEM_ALLOC. | 5792 | * but not for untrusted PTR_TO_BTF_ID | MEM_ALLOC. |
| 6048 | */ | 5793 | */ |
| 6049 | if (atype != BPF_READ && !type_is_ptr_alloc_obj(reg->type)) { | 5794 | if (atype != BPF_READ && !type_is_ptr_alloc_obj(reg->type)) { |
| @@ -6052,8 +5797,8 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env, · 2 unmodified lines | |||
| 6052 | } | 5797 | } |
| 6053 | 5798 | ||
| 6054 | if (type_is_alloc(reg->type) && !type_is_non_owning_ref(reg->type) && | 5799 | if (type_is_alloc(reg->type) && !type_is_non_owning_ref(reg->type) && |
| 6055 | !(reg->type & MEM_RCU) && !reg->ref_obj_id) { | 5800 | !(reg->type & MEM_RCU) && !reg_is_referenced(env, reg)) { |
| 6056 | verifier_bug(env, "ref_obj_id for allocated object must be non-zero"); | 5801 | verifier_bug(env, "allocated object must have a referenced id"); |
| 6057 | return -EFAULT; | 5802 | return -EFAULT; |
| 6058 | } | 5803 | } |
| 6059 | 5804 | ||
| @@ -6072,7 +5817,7 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env, · 12 unmodified lines | |||
| 6072 | */ | 5817 | */ |
| 6073 | flag = PTR_UNTRUSTED; | 5818 | flag = PTR_UNTRUSTED; |
| 6074 | 5819 | ||
| 6075 | } else if (is_trusted_reg(reg) || is_rcu_reg(reg)) { | 5820 | } else if (is_trusted_reg(env, reg) || is_rcu_reg(reg)) { |
| 6076 | /* By default any pointer obtained from walking a trusted pointer is no | 5821 | /* By default any pointer obtained from walking a trusted pointer is no |
| 6077 | * longer trusted, unless the field being accessed has explicitly been | 5822 | * longer trusted, unless the field being accessed has explicitly been |
| 6078 | * marked as inheriting its parent's state of trust (either full or RCU). | 5823 | * marked as inheriting its parent's state of trust (either full or RCU). |
| @@ -6133,12 +5878,11 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env, · 54 unmodified lines | |||
| 6133 | } | 5878 | } |
| 6134 | 5879 | ||
| 6135 | static int check_ptr_to_map_access(struct bpf_verifier_env *env, | 5880 | static int check_ptr_to_map_access(struct bpf_verifier_env *env, |
| 6136 | struct bpf_reg_state *regs, | 5881 | struct bpf_reg_state *regs, struct bpf_reg_state *reg, |
| 6137 | int regno, int off, int size, | 5882 | argno_t argno, int off, int size, |
| 6138 | enum bpf_access_type atype, | 5883 | enum bpf_access_type atype, |
| 6139 | int value_regno) | 5884 | int value_regno) |
| 6140 | { | 5885 | { |
| 6141 | struct bpf_reg_state *reg = regs + regno; | ||
| 6142 | struct bpf_map *map = reg->map_ptr; | 5886 | struct bpf_map *map = reg->map_ptr; |
| 6143 | struct bpf_reg_state map_reg; | 5887 | struct bpf_reg_state map_reg; |
| 6144 | enum bpf_type_flag flag = 0; | 5888 | enum bpf_type_flag flag = 0; |
| @@ -6169,8 +5913,8 @@ static int check_ptr_to_map_access(struct bpf_verifier_env *env, · 24 unmodified lines | |||
| 6169 | } | 5913 | } |
| 6170 | 5914 | ||
| 6171 | if (off < 0) { | 5915 | if (off < 0) { |
| 6172 | verbose(env, "R%d is %s invalid negative access: off=%d\n", | 5916 | verbose(env, "%s is %s invalid negative access: off=%d\n", |
| 6173 | regno, tname, off); | 5917 | reg_arg_name(env, argno), tname, off); |
| 6174 | return -EACCES; | 5918 | return -EACCES; |
| 6175 | } | 5919 | } |
| 6176 | 5920 | ||
| @@ -6227,11 +5971,10 @@ static int check_stack_slot_within_bounds(struct bpf_verifier_env *env, · 50 unmodified lines | |||
| 6227 | * 'off' includes `regno->offset`, but not its dynamic part (if any). | 5971 | * 'off' includes `regno->offset`, but not its dynamic part (if any). |
| 6228 | */ | 5972 | */ |
| 6229 | static int check_stack_access_within_bounds( | 5973 | static int check_stack_access_within_bounds( |
| 6230 | struct bpf_verifier_env *env, | 5974 | struct bpf_verifier_env *env, struct bpf_reg_state *reg, |
| 6231 | int regno, int off, int access_size, | 5975 | argno_t argno, int off, int access_size, |
| 6232 | enum bpf_access_type type) | 5976 | enum bpf_access_type type) |
| 6233 | { | 5977 | { |
| 6234 | struct bpf_reg_state *reg = reg_state(env, regno); | ||
| 6235 | struct bpf_func_state *state = bpf_func(env, reg); | 5978 | struct bpf_func_state *state = bpf_func(env, reg); |
| 6236 | s64 min_off, max_off; | 5979 | s64 min_off, max_off; |
| 6237 | int err; | 5980 | int err; |
| @@ -6246,14 +5989,14 @@ static int check_stack_access_within_bounds( · 8 unmodified lines | |||
| 6246 | min_off = (s64)reg->var_off.value + off; | 5989 | min_off = (s64)reg->var_off.value + off; |
| 6247 | max_off = min_off + access_size; | 5990 | max_off = min_off + access_size; |
| 6248 | } else { | 5991 | } else { |
| 6249 | if (reg->smax_value >= BPF_MAX_VAR_OFF || | 5992 | if (reg_smax(reg) >= BPF_MAX_VAR_OFF || |
| 6250 | reg->smin_value <= -BPF_MAX_VAR_OFF) { | 5993 | reg_smin(reg) <= -BPF_MAX_VAR_OFF) { |
| 6251 | verbose(env, "invalid unbounded variable-offset%s stack R%d\n", | 5994 | verbose(env, "invalid unbounded variable-offset%s stack %s\n", |
| 6252 | err_extra, regno); | 5995 | err_extra, reg_arg_name(env, argno)); |
| 6253 | return -EACCES; | 5996 | return -EACCES; |
| 6254 | } | 5997 | } |
| 6255 | min_off = reg->smin_value + off; | 5998 | min_off = reg_smin(reg) + off; |
| 6256 | max_off = reg->smax_value + off + access_size; | 5999 | max_off = reg_smax(reg) + off + access_size; |
| 6257 | } | 6000 | } |
| 6258 | 6001 | ||
| 6259 | err = check_stack_slot_within_bounds(env, min_off, state, type); | 6002 | err = check_stack_slot_within_bounds(env, min_off, state, type); |
| @@ -6267,14 +6010,14 @@ static int check_stack_access_within_bounds( · 7 unmodified lines | |||
| 6267 | 6010 | ||
| 6268 | if (err) { | 6011 | if (err) { |
| 6269 | if (tnum_is_const(reg->var_off)) { | 6012 | if (tnum_is_const(reg->var_off)) { |
| 6270 | verbose(env, "invalid%s stack R%d off=%lld size=%d\n", | 6013 | verbose(env, "invalid%s stack %s off=%lld size=%d\n", |
| 6271 | err_extra, regno, min_off, access_size); | 6014 | err_extra, reg_arg_name(env, argno), min_off, access_size); |
| 6272 | } else { | 6015 | } else { |
| 6273 | char tn_buf[48]; | 6016 | char tn_buf[48]; |
| 6274 | 6017 | ||
| 6275 | tnum_strn(tn_buf, sizeof(tn_buf), reg->var_off); | 6018 | tnum_strn(tn_buf, sizeof(tn_buf), reg->var_off); |
| 6276 | verbose(env, "invalid variable-offset%s stack R%d var_off=%s off=%d size=%d\n", | 6019 | verbose(env, "invalid variable-offset%s stack %s var_off=%s off=%d size=%d\n", |
| 6277 | err_extra, regno, tn_buf, off, access_size); | 6020 | err_extra, reg_arg_name(env, argno), tn_buf, off, access_size); |
| 6278 | } | 6021 | } |
| 6279 | return err; | 6022 | return err; |
| 6280 | } | 6023 | } |
| @@ -6319,12 +6062,11 @@ static void add_scalar_to_reg(struct bpf_reg_state *dst_reg, s64 val) · 38 unmodified lines | |||
| 6319 | * if t==write && value_regno==-1, some unknown value is stored into memory | 6062 | * if t==write && value_regno==-1, some unknown value is stored into memory |
| 6320 | * if t==read && value_regno==-1, don't care what we read from memory | 6063 | * if t==read && value_regno==-1, don't care what we read from memory |
| 6321 | */ | 6064 | */ |
| 6322 | static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regno, | 6065 | static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, struct bpf_reg_state *reg, argno_t argno, |
| 6323 | int off, int bpf_size, enum bpf_access_type t, | 6066 | int off, int bpf_size, enum bpf_access_type t, |
| 6324 | int value_regno, bool strict_alignment_once, bool is_ldsx) | 6067 | int value_regno, bool strict_alignment_once, bool is_ldsx) |
| 6325 | { | 6068 | { |
| 6326 | struct bpf_reg_state *regs = cur_regs(env); | 6069 | struct bpf_reg_state *regs = cur_regs(env); |
| 6327 | struct bpf_reg_state *reg = regs + regno; | ||
| 6328 | int size, err = 0; | 6070 | int size, err = 0; |
| 6329 | 6071 | ||
| 6330 | size = bpf_size_to_bytes(bpf_size); | 6072 | size = bpf_size_to_bytes(bpf_size); |
| @@ -6337,11 +6079,12 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 6 unmodified lines | |||
| 6337 | 6079 | ||
| 6338 | if (reg->type == PTR_TO_MAP_KEY) { | 6080 | if (reg->type == PTR_TO_MAP_KEY) { |
| 6339 | if (t == BPF_WRITE) { | 6081 | if (t == BPF_WRITE) { |
| 6340 | verbose(env, "write to change key R%d not allowed\n", regno); | 6082 | verbose(env, "write to change key %s not allowed\n", |
| 6083 | reg_arg_name(env, argno)); | ||
| 6341 | return -EACCES; | 6084 | return -EACCES; |
| 6342 | } | 6085 | } |
| 6343 | 6086 | ||
| 6344 | err = check_mem_region_access(env, regno, off, size, | 6087 | err = check_mem_region_access(env, reg, argno, off, size, |
| 6345 | reg->map_ptr->key_size, false); | 6088 | reg->map_ptr->key_size, false); |
| 6346 | if (err) | 6089 | if (err) |
| 6347 | return err; | 6090 | return err; |
| @@ -6355,17 +6098,17 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 7 unmodified lines | |||
| 6355 | verbose(env, "R%d leaks addr into map\n", value_regno); | 6098 | verbose(env, "R%d leaks addr into map\n", value_regno); |
| 6356 | return -EACCES; | 6099 | return -EACCES; |
| 6357 | } | 6100 | } |
| 6358 | err = check_map_access_type(env, regno, off, size, t); | 6101 | err = check_map_access_type(env, reg, off, size, t); |
| 6359 | if (err) | 6102 | if (err) |
| 6360 | return err; | 6103 | return err; |
| 6361 | err = check_map_access(env, regno, off, size, false, ACCESS_DIRECT); | 6104 | err = check_map_access(env, reg, argno, off, size, false, ACCESS_DIRECT); |
| 6362 | if (err) | 6105 | if (err) |
| 6363 | return err; | 6106 | return err; |
| 6364 | if (tnum_is_const(reg->var_off)) | 6107 | if (tnum_is_const(reg->var_off)) |
| 6365 | kptr_field = btf_record_find(reg->map_ptr->record, | 6108 | kptr_field = btf_record_find(reg->map_ptr->record, |
| 6366 | off + reg->var_off.value, BPF_KPTR | BPF_UPTR); | 6109 | off + reg->var_off.value, BPF_KPTR | BPF_UPTR); |
| 6367 | if (kptr_field) { | 6110 | if (kptr_field) { |
| 6368 | err = check_map_kptr_access(env, regno, value_regno, insn_idx, kptr_field); | 6111 | err = check_map_kptr_access(env, value_regno, insn_idx, kptr_field); |
| 6369 | } else if (t == BPF_READ && value_regno >= 0) { | 6112 | } else if (t == BPF_READ && value_regno >= 0) { |
| 6370 | struct bpf_map *map = reg->map_ptr; | 6113 | struct bpf_map *map = reg->map_ptr; |
| 6371 | 6114 | ||
| @@ -6393,7 +6136,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 21 unmodified lines | |||
| 6393 | size); | 6136 | size); |
| 6394 | return -EACCES; | 6137 | return -EACCES; |
| 6395 | } | 6138 | } |
| 6396 | copy_register_state(®s[value_regno], reg); | 6139 | regs[value_regno] = *reg; |
| 6397 | add_scalar_to_reg(®s[value_regno], off); | 6140 | add_scalar_to_reg(®s[value_regno], off); |
| 6398 | regs[value_regno].type = PTR_TO_INSN; | 6141 | regs[value_regno].type = PTR_TO_INSN; |
| 6399 | } else { | 6142 | } else { |
| @@ -6405,14 +6148,14 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 5 unmodified lines | |||
| 6405 | bool rdonly_untrusted = rdonly_mem && (reg->type & PTR_UNTRUSTED); | 6148 | bool rdonly_untrusted = rdonly_mem && (reg->type & PTR_UNTRUSTED); |
| 6406 | 6149 | ||
| 6407 | if (type_may_be_null(reg->type)) { | 6150 | if (type_may_be_null(reg->type)) { |
| 6408 | verbose(env, "R%d invalid mem access '%s'\n", regno, | 6151 | verbose(env, "%s invalid mem access '%s'\n", reg_arg_name(env, argno), |
| 6409 | reg_type_str(env, reg->type)); | 6152 | reg_type_str(env, reg->type)); |
| 6410 | return -EACCES; | 6153 | return -EACCES; |
| 6411 | } | 6154 | } |
| 6412 | 6155 | ||
| 6413 | if (t == BPF_WRITE && rdonly_mem) { | 6156 | if (t == BPF_WRITE && rdonly_mem) { |
| 6414 | verbose(env, "R%d cannot write into %s\n", | 6157 | verbose(env, "%s cannot write into %s\n", |
| 6415 | regno, reg_type_str(env, reg->type)); | 6158 | reg_arg_name(env, argno), reg_type_str(env, reg->type)); |
| 6416 | return -EACCES; | 6159 | return -EACCES; |
| 6417 | } | 6160 | } |
| 6418 | 6161 | ||
| @@ -6427,7 +6170,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 8 unmodified lines | |||
| 6427 | * instructions, hence no need to check bounds in that case. | 6170 | * instructions, hence no need to check bounds in that case. |
| 6428 | */ | 6171 | */ |
| 6429 | if (!rdonly_untrusted) | 6172 | if (!rdonly_untrusted) |
| 6430 | err = check_mem_region_access(env, regno, off, size, | 6173 | err = check_mem_region_access(env, reg, argno, off, size, |
| 6431 | reg->mem_size, false); | 6174 | reg->mem_size, false); |
| 6432 | if (!err && value_regno >= 0 && (t == BPF_READ || rdonly_mem)) | 6175 | if (!err && value_regno >= 0 && (t == BPF_READ || rdonly_mem)) |
| 6433 | mark_reg_unknown(env, regs, value_regno); | 6176 | mark_reg_unknown(env, regs, value_regno); |
| @@ -6445,7 +6188,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 11 unmodified lines | |||
| 6445 | return -EACCES; | 6188 | return -EACCES; |
| 6446 | } | 6189 | } |
| 6447 | 6190 | ||
| 6448 | err = check_ctx_access(env, insn_idx, regno, off, size, t, &info); | 6191 | err = check_ctx_access(env, insn_idx, reg, argno, off, size, t, &info); |
| 6449 | if (!err && t == BPF_READ && value_regno >= 0) { | 6192 | if (!err && t == BPF_READ && value_regno >= 0) { |
| 6450 | /* ctx access returns either a scalar, or a | 6193 | /* ctx access returns either a scalar, or a |
| 6451 | * PTR_TO_PACKET[_META,_END]. In the latter | 6194 | * PTR_TO_PACKET[_META,_END]. In the latter |
| @@ -6463,8 +6206,6 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 11 unmodified lines | |||
| 6463 | } else { | 6206 | } else { |
| 6464 | mark_reg_known_zero(env, regs, | 6207 | mark_reg_known_zero(env, regs, |
| 6465 | value_regno); | 6208 | value_regno); |
| 6466 | if (type_may_be_null(info.reg_type)) | ||
| 6467 | regs[value_regno].id = ++env->id_gen; | ||
| 6468 | /* A load of ctx field could have different | 6209 | /* A load of ctx field could have different |
| 6469 | * actual load size with the one encoded in the | 6210 | * actual load size with the one encoded in the |
| 6470 | * insn. When the dst is PTR, it is for sure not | 6211 | * insn. When the dst is PTR, it is for sure not |
| @@ -6474,23 +6215,25 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 3 unmodified lines | |||
| 6474 | if (base_type(info.reg_type) == PTR_TO_BTF_ID) { | 6215 | if (base_type(info.reg_type) == PTR_TO_BTF_ID) { |
| 6475 | regs[value_regno].btf = info.btf; | 6216 | regs[value_regno].btf = info.btf; |
| 6476 | regs[value_regno].btf_id = info.btf_id; | 6217 | regs[value_regno].btf_id = info.btf_id; |
| 6477 | regs[value_regno].ref_obj_id = info.ref_obj_id; | 6218 | regs[value_regno].id = info.ref_id; |
| 6478 | } | 6219 | } |
| 6220 | if (type_may_be_null(info.reg_type) && !regs[value_regno].id) | ||
| 6221 | regs[value_regno].id = ++env->id_gen; | ||
| 6479 | } | 6222 | } |
| 6480 | regs[value_regno].type = info.reg_type; | 6223 | regs[value_regno].type = info.reg_type; |
| 6481 | } | 6224 | } |
| 6482 | 6225 | ||
| 6483 | } else if (reg->type == PTR_TO_STACK) { | 6226 | } else if (reg->type == PTR_TO_STACK) { |
| 6484 | /* Basic bounds checks. */ | 6227 | /* Basic bounds checks. */ |
| 6485 | err = check_stack_access_within_bounds(env, regno, off, size, t); | 6228 | err = check_stack_access_within_bounds(env, reg, argno, off, size, t); |
| 6486 | if (err) | 6229 | if (err) |
| 6487 | return err; | 6230 | return err; |
| 6488 | 6231 | ||
| 6489 | if (t == BPF_READ) | 6232 | if (t == BPF_READ) |
| 6490 | err = check_stack_read(env, regno, off, size, | 6233 | err = check_stack_read(env, reg, argno, off, size, |
| 6491 | value_regno); | 6234 | value_regno); |
| 6492 | else | 6235 | else |
| 6493 | err = check_stack_write(env, regno, off, size, | 6236 | err = check_stack_write(env, reg, off, size, |
| 6494 | value_regno, insn_idx); | 6237 | value_regno, insn_idx); |
| 6495 | } else if (reg_is_pkt_pointer(reg)) { | 6238 | } else if (reg_is_pkt_pointer(reg)) { |
| 6496 | if (t == BPF_WRITE && !may_access_direct_pkt_data(env, NULL, t)) { | 6239 | if (t == BPF_WRITE && !may_access_direct_pkt_data(env, NULL, t)) { |
| @@ -6503,7 +6246,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 6 unmodified lines | |||
| 6503 | value_regno); | 6246 | value_regno); |
| 6504 | return -EACCES; | 6247 | return -EACCES; |
| 6505 | } | 6248 | } |
| 6506 | err = check_packet_access(env, regno, off, size, false); | 6249 | err = check_packet_access(env, reg, argno, off, size, false); |
| 6507 | if (!err && t == BPF_READ && value_regno >= 0) | 6250 | if (!err && t == BPF_READ && value_regno >= 0) |
| 6508 | mark_reg_unknown(env, regs, value_regno); | 6251 | mark_reg_unknown(env, regs, value_regno); |
| 6509 | } else if (reg->type == PTR_TO_FLOW_KEYS) { | 6252 | } else if (reg->type == PTR_TO_FLOW_KEYS) { |
| @@ -6514,28 +6257,28 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 4 unmodified lines | |||
| 6514 | return -EACCES; | 6257 | return -EACCES; |
| 6515 | } | 6258 | } |
| 6516 | 6259 | ||
| 6517 | err = check_flow_keys_access(env, off, size); | 6260 | err = check_flow_keys_access(env, reg, argno, off, size); |
| 6518 | if (!err && t == BPF_READ && value_regno >= 0) | 6261 | if (!err && t == BPF_READ && value_regno >= 0) |
| 6519 | mark_reg_unknown(env, regs, value_regno); | 6262 | mark_reg_unknown(env, regs, value_regno); |
| 6520 | } else if (type_is_sk_pointer(reg->type)) { | 6263 | } else if (type_is_sk_pointer(reg->type)) { |
| 6521 | if (t == BPF_WRITE) { | 6264 | if (t == BPF_WRITE) { |
| 6522 | verbose(env, "R%d cannot write into %s\n", | 6265 | verbose(env, "%s cannot write into %s\n", |
| 6523 | regno, reg_type_str(env, reg->type)); | 6266 | reg_arg_name(env, argno), reg_type_str(env, reg->type)); |
| 6524 | return -EACCES; | 6267 | return -EACCES; |
| 6525 | } | 6268 | } |
| 6526 | err = check_sock_access(env, insn_idx, regno, off, size, t); | 6269 | err = check_sock_access(env, insn_idx, reg, argno, off, size, t); |
| 6527 | if (!err && value_regno >= 0) | 6270 | if (!err && value_regno >= 0) |
| 6528 | mark_reg_unknown(env, regs, value_regno); | 6271 | mark_reg_unknown(env, regs, value_regno); |
| 6529 | } else if (reg->type == PTR_TO_TP_BUFFER) { | 6272 | } else if (reg->type == PTR_TO_TP_BUFFER) { |
| 6530 | err = check_tp_buffer_access(env, reg, regno, off, size); | 6273 | err = check_tp_buffer_access(env, reg, argno, off, size); |
| 6531 | if (!err && t == BPF_READ && value_regno >= 0) | 6274 | if (!err && t == BPF_READ && value_regno >= 0) |
| 6532 | mark_reg_unknown(env, regs, value_regno); | 6275 | mark_reg_unknown(env, regs, value_regno); |
| 6533 | } else if (base_type(reg->type) == PTR_TO_BTF_ID && | 6276 | } else if (base_type(reg->type) == PTR_TO_BTF_ID && |
| 6534 | !type_may_be_null(reg->type)) { | 6277 | !type_may_be_null(reg->type)) { |
| 6535 | err = check_ptr_to_btf_access(env, regs, regno, off, size, t, | 6278 | err = check_ptr_to_btf_access(env, regs, reg, argno, off, size, t, |
| 6536 | value_regno); | 6279 | value_regno); |
| 6537 | } else if (reg->type == CONST_PTR_TO_MAP) { | 6280 | } else if (reg->type == CONST_PTR_TO_MAP) { |
| 6538 | err = check_ptr_to_map_access(env, regs, regno, off, size, t, | 6281 | err = check_ptr_to_map_access(env, regs, reg, argno, off, size, t, |
| 6539 | value_regno); | 6282 | value_regno); |
| 6540 | } else if (base_type(reg->type) == PTR_TO_BUF && | 6283 | } else if (base_type(reg->type) == PTR_TO_BUF && |
| 6541 | !type_may_be_null(reg->type)) { | 6284 | !type_may_be_null(reg->type)) { |
| @@ -6544,8 +6287,8 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 2 unmodified lines | |||
| 6544 | 6287 | ||
| 6545 | if (rdonly_mem) { | 6288 | if (rdonly_mem) { |
| 6546 | if (t == BPF_WRITE) { | 6289 | if (t == BPF_WRITE) { |
| 6547 | verbose(env, "R%d cannot write into %s\n", | 6290 | verbose(env, "%s cannot write into %s\n", |
| 6548 | regno, reg_type_str(env, reg->type)); | 6291 | reg_arg_name(env, argno), reg_type_str(env, reg->type)); |
| 6549 | return -EACCES; | 6292 | return -EACCES; |
| 6550 | } | 6293 | } |
| 6551 | max_access = &env->prog->aux->max_rdonly_access; | 6294 | max_access = &env->prog->aux->max_rdonly_access; |
| @@ -6553,7 +6296,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 1 unmodified line | |||
| 6553 | max_access = &env->prog->aux->max_rdwr_access; | 6296 | max_access = &env->prog->aux->max_rdwr_access; |
| 6554 | } | 6297 | } |
| 6555 | 6298 | ||
| 6556 | err = check_buffer_access(env, reg, regno, off, size, false, | 6299 | err = check_buffer_access(env, reg, argno, off, size, false, |
| 6557 | max_access); | 6300 | max_access); |
| 6558 | 6301 | ||
| 6559 | if (!err && value_regno >= 0 && (rdonly_mem || t == BPF_READ)) | 6302 | if (!err && value_regno >= 0 && (rdonly_mem || t == BPF_READ)) |
| @@ -6562,7 +6305,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 2 unmodified lines | |||
| 6562 | if (t == BPF_READ && value_regno >= 0) | 6305 | if (t == BPF_READ && value_regno >= 0) |
| 6563 | mark_reg_unknown(env, regs, value_regno); | 6306 | mark_reg_unknown(env, regs, value_regno); |
| 6564 | } else { | 6307 | } else { |
| 6565 | verbose(env, "R%d invalid mem access '%s'\n", regno, | 6308 | verbose(env, "%s invalid mem access '%s'\n", reg_arg_name(env, argno), |
| 6566 | reg_type_str(env, reg->type)); | 6309 | reg_type_str(env, reg->type)); |
| 6567 | return -EACCES; | 6310 | return -EACCES; |
| 6568 | } | 6311 | } |
| @@ -6585,10 +6328,20 @@ static int check_load_mem(struct bpf_verifier_env *env, struct bpf_insn *insn, · 16 unmodified lines | |||
| 6585 | bool strict_alignment_once, bool is_ldsx, | 6328 | bool strict_alignment_once, bool is_ldsx, |
| 6586 | bool allow_trust_mismatch, const char *ctx) | 6329 | bool allow_trust_mismatch, const char *ctx) |
| 6587 | { | 6330 | { |
| 6331 | struct bpf_verifier_state *vstate = env->cur_state; | ||
| 6332 | struct bpf_func_state *state = vstate->frame[vstate->curframe]; | ||
| 6588 | struct bpf_reg_state *regs = cur_regs(env); | 6333 | struct bpf_reg_state *regs = cur_regs(env); |
| 6589 | enum bpf_reg_type src_reg_type; | 6334 | enum bpf_reg_type src_reg_type; |
| 6590 | int err; | 6335 | int err; |
| 6591 | 6336 | ||
| 6337 | /* Handle stack arg read */ | ||
| 6338 | if (is_stack_arg_ldx(insn)) { | ||
| 6339 | err = check_reg_arg(env, insn->dst_reg, DST_OP_NO_MARK); | ||
| 6340 | if (err) | ||
| 6341 | return err; | ||
| 6342 | return check_stack_arg_read(env, state, insn->off, insn->dst_reg); | ||
| 6343 | } | ||
| 6344 | |||
| 6592 | /* check src operand */ | 6345 | /* check src operand */ |
| 6593 | err = check_reg_arg(env, insn->src_reg, SRC_OP); | 6346 | err = check_reg_arg(env, insn->src_reg, SRC_OP); |
| 6594 | if (err) | 6347 | if (err) |
| @@ -6604,7 +6357,7 @@ static int check_load_mem(struct bpf_verifier_env *env, struct bpf_insn *insn, · 9 unmodified lines | |||
| 6604 | /* Check if (src_reg + off) is readable. The state of dst_reg will be | 6357 | /* Check if (src_reg + off) is readable. The state of dst_reg will be |
| 6605 | * updated by this call. | 6358 | * updated by this call. |
| 6606 | */ | 6359 | */ |
| 6607 | err = check_mem_access(env, env->insn_idx, insn->src_reg, insn->off, | 6360 | err = check_mem_access(env, env->insn_idx, regs + insn->src_reg, argno_from_reg(insn->src_reg), insn->off, |
| 6608 | BPF_SIZE(insn->code), BPF_READ, insn->dst_reg, | 6361 | BPF_SIZE(insn->code), BPF_READ, insn->dst_reg, |
| 6609 | strict_alignment_once, is_ldsx); | 6362 | strict_alignment_once, is_ldsx); |
| 6610 | err = err ?: save_aux_ptr_type(env, src_reg_type, | 6363 | err = err ?: save_aux_ptr_type(env, src_reg_type, |
| @@ -6617,10 +6370,20 @@ static int check_load_mem(struct bpf_verifier_env *env, struct bpf_insn *insn, · 6 unmodified lines | |||
| 6617 | static int check_store_reg(struct bpf_verifier_env *env, struct bpf_insn *insn, | 6370 | static int check_store_reg(struct bpf_verifier_env *env, struct bpf_insn *insn, |
| 6618 | bool strict_alignment_once) | 6371 | bool strict_alignment_once) |
| 6619 | { | 6372 | { |
| 6373 | struct bpf_verifier_state *vstate = env->cur_state; | ||
| 6374 | struct bpf_func_state *state = vstate->frame[vstate->curframe]; | ||
| 6620 | struct bpf_reg_state *regs = cur_regs(env); | 6375 | struct bpf_reg_state *regs = cur_regs(env); |
| 6621 | enum bpf_reg_type dst_reg_type; | 6376 | enum bpf_reg_type dst_reg_type; |
| 6622 | int err; | 6377 | int err; |
| 6623 | 6378 | ||
| 6379 | /* Handle stack arg write */ | ||
| 6380 | if (is_stack_arg_stx(insn)) { | ||
| 6381 | err = check_reg_arg(env, insn->src_reg, SRC_OP); | ||
| 6382 | if (err) | ||
| 6383 | return err; | ||
| 6384 | return check_stack_arg_write(env, state, insn->off, regs + insn->src_reg); | ||
| 6385 | } | ||
| 6386 | |||
| 6624 | /* check src1 operand */ | 6387 | /* check src1 operand */ |
| 6625 | err = check_reg_arg(env, insn->src_reg, SRC_OP); | 6388 | err = check_reg_arg(env, insn->src_reg, SRC_OP); |
| 6626 | if (err) | 6389 | if (err) |
| @@ -6634,7 +6397,7 @@ static int check_store_reg(struct bpf_verifier_env *env, struct bpf_insn *insn, · 7 unmodified lines | |||
| 6634 | dst_reg_type = regs[insn->dst_reg].type; | 6397 | dst_reg_type = regs[insn->dst_reg].type; |
| 6635 | 6398 | ||
| 6636 | /* Check if (dst_reg + off) is writeable. */ | 6399 | /* Check if (dst_reg + off) is writeable. */ |
| 6637 | err = check_mem_access(env, env->insn_idx, insn->dst_reg, insn->off, | 6400 | err = check_mem_access(env, env->insn_idx, regs + insn->dst_reg, argno_from_reg(insn->dst_reg), insn->off, |
| 6638 | BPF_SIZE(insn->code), BPF_WRITE, insn->src_reg, | 6401 | BPF_SIZE(insn->code), BPF_WRITE, insn->src_reg, |
| 6639 | strict_alignment_once, false); | 6402 | strict_alignment_once, false); |
| 6640 | err = err ?: save_aux_ptr_type(env, dst_reg_type, false); | 6403 | err = err ?: save_aux_ptr_type(env, dst_reg_type, false); |
| @@ -6645,6 +6408,7 @@ static int check_store_reg(struct bpf_verifier_env *env, struct bpf_insn *insn, · 4 unmodified lines | |||
| 6645 | static int check_atomic_rmw(struct bpf_verifier_env *env, | 6408 | static int check_atomic_rmw(struct bpf_verifier_env *env, |
| 6646 | struct bpf_insn *insn) | 6409 | struct bpf_insn *insn) |
| 6647 | { | 6410 | { |
| 6411 | struct bpf_reg_state *dst_reg; | ||
| 6648 | int load_reg; | 6412 | int load_reg; |
| 6649 | int err; | 6413 | int err; |
| 6650 | 6414 | ||
| @@ -6706,13 +6470,15 @@ static int check_atomic_rmw(struct bpf_verifier_env *env, · 55 unmodified lines | |||
| 6706 | load_reg = -1; | 6470 | load_reg = -1; |
| 6707 | } | 6471 | } |
| 6708 | 6472 | ||
| 6473 | dst_reg = cur_regs(env) + insn->dst_reg; | ||
| 6474 | |||
| 6709 | /* Check whether we can read the memory, with second call for fetch | 6475 | /* Check whether we can read the memory, with second call for fetch |
| 6710 | * case to simulate the register fill. | 6476 | * case to simulate the register fill. |
| 6711 | */ | 6477 | */ |
| 6712 | err = check_mem_access(env, env->insn_idx, insn->dst_reg, insn->off, | 6478 | err = check_mem_access(env, env->insn_idx, dst_reg, argno_from_reg(insn->dst_reg), insn->off, |
| 6713 | BPF_SIZE(insn->code), BPF_READ, -1, true, false); | 6479 | BPF_SIZE(insn->code), BPF_READ, -1, true, false); |
| 6714 | if (!err && load_reg >= 0) | 6480 | if (!err && load_reg >= 0) |
| 6715 | err = check_mem_access(env, env->insn_idx, insn->dst_reg, | 6481 | err = check_mem_access(env, env->insn_idx, dst_reg, argno_from_reg(insn->dst_reg), |
| 6716 | insn->off, BPF_SIZE(insn->code), | 6482 | insn->off, BPF_SIZE(insn->code), |
| 6717 | BPF_READ, load_reg, true, false); | 6483 | BPF_READ, load_reg, true, false); |
| 6718 | if (err) | 6484 | if (err) |
| @@ -6724,7 +6490,7 @@ static int check_atomic_rmw(struct bpf_verifier_env *env, · 5 unmodified lines | |||
| 6724 | return err; | 6490 | return err; |
| 6725 | } | 6491 | } |
| 6726 | /* Check whether we can write into the same memory. */ | 6492 | /* Check whether we can write into the same memory. */ |
| 6727 | err = check_mem_access(env, env->insn_idx, insn->dst_reg, insn->off, | 6493 | err = check_mem_access(env, env->insn_idx, dst_reg, argno_from_reg(insn->dst_reg), insn->off, |
| 6728 | BPF_SIZE(insn->code), BPF_WRITE, -1, true, false); | 6494 | BPF_SIZE(insn->code), BPF_WRITE, -1, true, false); |
| 6729 | if (err) | 6495 | if (err) |
| 6730 | return err; | 6496 | return err; |
| @@ -6813,11 +6579,10 @@ static int check_atomic(struct bpf_verifier_env *env, struct bpf_insn *insn) · 82 unmodified lines | |||
| 6813 | * read offsets are marked as read. | 6579 | * read offsets are marked as read. |
| 6814 | */ | 6580 | */ |
| 6815 | static int check_stack_range_initialized( | 6581 | static int check_stack_range_initialized( |
| 6816 | struct bpf_verifier_env *env, int regno, int off, | 6582 | struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno, int off, |
| 6817 | int access_size, bool zero_size_allowed, | 6583 | int access_size, bool zero_size_allowed, |
| 6818 | enum bpf_access_type type, struct bpf_call_arg_meta *meta) | 6584 | enum bpf_access_type type, struct bpf_call_arg_meta *meta) |
| 6819 | { | 6585 | { |
| 6820 | struct bpf_reg_state *reg = reg_state(env, regno); | ||
| 6821 | struct bpf_func_state *state = bpf_func(env, reg); | 6586 | struct bpf_func_state *state = bpf_func(env, reg); |
| 6822 | int err, min_off, max_off, i, j, slot, spi; | 6587 | int err, min_off, max_off, i, j, slot, spi; |
| 6823 | /* Some accesses can write anything into the stack, others are | 6588 | /* Some accesses can write anything into the stack, others are |
| @@ -6839,7 +6604,7 @@ static int check_stack_range_initialized( · 15 unmodified lines | |||
| 6839 | return -EACCES; | 6604 | return -EACCES; |
| 6840 | } | 6605 | } |
| 6841 | 6606 | ||
| 6842 | err = check_stack_access_within_bounds(env, regno, off, access_size, type); | 6607 | err = check_stack_access_within_bounds(env, reg, argno, off, access_size, type); |
| 6843 | if (err) | 6608 | if (err) |
| 6844 | return err; | 6609 | return err; |
| 6845 | 6610 | ||
| @@ -6856,8 +6621,8 @@ static int check_stack_range_initialized( · 10 unmodified lines | |||
| 6856 | char tn_buf[48]; | 6621 | char tn_buf[48]; |
| 6857 | 6622 | ||
| 6858 | tnum_strn(tn_buf, sizeof(tn_buf), reg->var_off); | 6623 | tnum_strn(tn_buf, sizeof(tn_buf), reg->var_off); |
| 6859 | verbose(env, "R%d variable offset stack access prohibited for !root, var_off=%s\n", | 6624 | verbose(env, "%s variable offset stack access prohibited for !root, var_off=%s\n", |
| 6860 | regno, tn_buf); | 6625 | reg_arg_name(env, argno), tn_buf); |
| 6861 | return -EACCES; | 6626 | return -EACCES; |
| 6862 | } | 6627 | } |
| 6863 | /* Only initialized buffer on stack is allowed to be accessed | 6628 | /* Only initialized buffer on stack is allowed to be accessed |
| @@ -6869,8 +6634,8 @@ static int check_stack_range_initialized( · 5 unmodified lines | |||
| 6869 | if (meta && meta->raw_mode) | 6634 | if (meta && meta->raw_mode) |
| 6870 | meta = NULL; | 6635 | meta = NULL; |
| 6871 | 6636 | ||
| 6872 | min_off = reg->smin_value + off; | 6637 | min_off = reg_smin(reg) + off; |
| 6873 | max_off = reg->smax_value + off; | 6638 | max_off = reg_smax(reg) + off; |
| 6874 | } | 6639 | } |
| 6875 | 6640 | ||
| 6876 | if (meta && meta->raw_mode) { | 6641 | if (meta && meta->raw_mode) { |
| @@ -6900,7 +6665,7 @@ static int check_stack_range_initialized( · 23 unmodified lines | |||
| 6900 | } | 6665 | } |
| 6901 | } | 6666 | } |
| 6902 | meta->access_size = access_size; | 6667 | meta->access_size = access_size; |
| 6903 | meta->regno = regno; | 6668 | meta->regno = reg_from_argno(argno); |
| 6904 | return 0; | 6669 | return 0; |
| 6905 | } | 6670 | } |
| 6906 | 6671 | ||
| @@ -6940,17 +6705,17 @@ static int check_stack_range_initialized( · 33 unmodified lines | |||
| 6940 | if (*stype == STACK_POISON) { | 6705 | if (*stype == STACK_POISON) { |
| 6941 | if (allow_poison) | 6706 | if (allow_poison) |
| 6942 | goto mark; | 6707 | goto mark; |
| 6943 | verbose(env, "reading from stack R%d off %d+%d size %d, slot poisoned by dead code elimination\n", | 6708 | verbose(env, "reading from stack %s off %d+%d size %d, slot poisoned by dead code elimination\n", |
| 6944 | regno, min_off, i - min_off, access_size); | 6709 | reg_arg_name(env, argno), min_off, i - min_off, access_size); |
| 6945 | } else if (tnum_is_const(reg->var_off)) { | 6710 | } else if (tnum_is_const(reg->var_off)) { |
| 6946 | verbose(env, "invalid read from stack R%d off %d+%d size %d\n", | 6711 | verbose(env, "invalid read from stack %s off %d+%d size %d\n", |
| 6947 | regno, min_off, i - min_off, access_size); | 6712 | reg_arg_name(env, argno), min_off, i - min_off, access_size); |
| 6948 | } else { | 6713 | } else { |
| 6949 | char tn_buf[48]; | 6714 | char tn_buf[48]; |
| 6950 | 6715 | ||
| 6951 | tnum_strn(tn_buf, sizeof(tn_buf), reg->var_off); | 6716 | tnum_strn(tn_buf, sizeof(tn_buf), reg->var_off); |
| 6952 | verbose(env, "invalid read from stack R%d var_off %s+%d size %d\n", | 6717 | verbose(env, "invalid read from stack %s var_off %s+%d size %d\n", |
| 6953 | regno, tn_buf, i - min_off, access_size); | 6718 | reg_arg_name(env, argno), tn_buf, i - min_off, access_size); |
| 6954 | } | 6719 | } |
| 6955 | return -EACCES; | 6720 | return -EACCES; |
| 6956 | mark: | 6721 | mark: |
| @@ -6959,48 +6724,48 @@ mark: · 2 unmodified lines | |||
| 6959 | return 0; | 6724 | return 0; |
| 6960 | } | 6725 | } |
| 6961 | 6726 | ||
| 6962 | static int check_helper_mem_access(struct bpf_verifier_env *env, int regno, | 6727 | static int check_helper_mem_access(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno, |
| 6963 | int access_size, enum bpf_access_type access_type, | 6728 | int access_size, enum bpf_access_type access_type, |
| 6964 | bool zero_size_allowed, | 6729 | bool zero_size_allowed, |
| 6965 | struct bpf_call_arg_meta *meta) | 6730 | struct bpf_call_arg_meta *meta) |
| 6966 | { | 6731 | { |
| 6967 | struct bpf_reg_state *regs = cur_regs(env), *reg = ®s[regno]; | 6732 | struct bpf_reg_state *regs = cur_regs(env); |
| 6968 | u32 *max_access; | 6733 | u32 *max_access; |
| 6969 | 6734 | ||
| 6970 | switch (base_type(reg->type)) { | 6735 | switch (base_type(reg->type)) { |
| 6971 | case PTR_TO_PACKET: | 6736 | case PTR_TO_PACKET: |
| 6972 | case PTR_TO_PACKET_META: | 6737 | case PTR_TO_PACKET_META: |
| 6973 | return check_packet_access(env, regno, 0, access_size, | 6738 | return check_packet_access(env, reg, argno, 0, access_size, |
| 6974 | zero_size_allowed); | 6739 | zero_size_allowed); |
| 6975 | case PTR_TO_MAP_KEY: | 6740 | case PTR_TO_MAP_KEY: |
| 6976 | if (access_type == BPF_WRITE) { | 6741 | if (access_type == BPF_WRITE) { |
| 6977 | verbose(env, "R%d cannot write into %s\n", regno, | 6742 | verbose(env, "%s cannot write into %s\n", |
| 6978 | reg_type_str(env, reg->type)); | 6743 | reg_arg_name(env, argno), reg_type_str(env, reg->type)); |
| 6979 | return -EACCES; | 6744 | return -EACCES; |
| 6980 | } | 6745 | } |
| 6981 | return check_mem_region_access(env, regno, 0, access_size, | 6746 | return check_mem_region_access(env, reg, argno, 0, access_size, |
| 6982 | reg->map_ptr->key_size, false); | 6747 | reg->map_ptr->key_size, false); |
| 6983 | case PTR_TO_MAP_VALUE: | 6748 | case PTR_TO_MAP_VALUE: |
| 6984 | if (check_map_access_type(env, regno, 0, access_size, access_type)) | 6749 | if (check_map_access_type(env, reg, 0, access_size, access_type)) |
| 6985 | return -EACCES; | 6750 | return -EACCES; |
| 6986 | return check_map_access(env, regno, 0, access_size, | 6751 | return check_map_access(env, reg, argno, 0, access_size, |
| 6987 | zero_size_allowed, ACCESS_HELPER); | 6752 | zero_size_allowed, ACCESS_HELPER); |
| 6988 | case PTR_TO_MEM: | 6753 | case PTR_TO_MEM: |
| 6989 | if (type_is_rdonly_mem(reg->type)) { | 6754 | if (type_is_rdonly_mem(reg->type)) { |
| 6990 | if (access_type == BPF_WRITE) { | 6755 | if (access_type == BPF_WRITE) { |
| 6991 | verbose(env, "R%d cannot write into %s\n", regno, | 6756 | verbose(env, "%s cannot write into %s\n", |
| 6992 | reg_type_str(env, reg->type)); | 6757 | reg_arg_name(env, argno), reg_type_str(env, reg->type)); |
| 6993 | return -EACCES; | 6758 | return -EACCES; |
| 6994 | } | 6759 | } |
| 6995 | } | 6760 | } |
| 6996 | return check_mem_region_access(env, regno, 0, | 6761 | return check_mem_region_access(env, reg, argno, 0, |
| 6997 | access_size, reg->mem_size, | 6762 | access_size, reg->mem_size, |
| 6998 | zero_size_allowed); | 6763 | zero_size_allowed); |
| 6999 | case PTR_TO_BUF: | 6764 | case PTR_TO_BUF: |
| 7000 | if (type_is_rdonly_mem(reg->type)) { | 6765 | if (type_is_rdonly_mem(reg->type)) { |
| 7001 | if (access_type == BPF_WRITE) { | 6766 | if (access_type == BPF_WRITE) { |
| 7002 | verbose(env, "R%d cannot write into %s\n", regno, | 6767 | verbose(env, "%s cannot write into %s\n", |
| 7003 | reg_type_str(env, reg->type)); | 6768 | reg_arg_name(env, argno), reg_type_str(env, reg->type)); |
| 7004 | return -EACCES; | 6769 | return -EACCES; |
| 7005 | } | 6770 | } |
| 7006 | 6771 | ||
| @@ -7008,26 +6773,26 @@ static int check_helper_mem_access(struct bpf_verifier_env *env, int regno, · 1 unmodified line | |||
| 7008 | } else { | 6773 | } else { |
| 7009 | max_access = &env->prog->aux->max_rdwr_access; | 6774 | max_access = &env->prog->aux->max_rdwr_access; |
| 7010 | } | 6775 | } |
| 7011 | return check_buffer_access(env, reg, regno, 0, | 6776 | return check_buffer_access(env, reg, argno, 0, |
| 7012 | access_size, zero_size_allowed, | 6777 | access_size, zero_size_allowed, |
| 7013 | max_access); | 6778 | max_access); |
| 7014 | case PTR_TO_STACK: | 6779 | case PTR_TO_STACK: |
| 7015 | return check_stack_range_initialized( | 6780 | return check_stack_range_initialized( |
| 7016 | env, | 6781 | env, reg, |
| 7017 | regno, 0, access_size, | 6782 | argno, 0, access_size, |
| 7018 | zero_size_allowed, access_type, meta); | 6783 | zero_size_allowed, access_type, meta); |
| 7019 | case PTR_TO_BTF_ID: | 6784 | case PTR_TO_BTF_ID: |
| 7020 | return check_ptr_to_btf_access(env, regs, regno, 0, | 6785 | return check_ptr_to_btf_access(env, regs, reg, argno, 0, |
| 7021 | access_size, BPF_READ, -1); | 6786 | access_size, access_type, -1); |
| 7022 | case PTR_TO_CTX: | 6787 | case PTR_TO_CTX: |
| 7023 | /* Only permit reading or writing syscall context using helper calls. */ | 6788 | /* Only permit reading or writing syscall context using helper calls. */ |
| 7024 | if (is_var_ctx_off_allowed(env->prog)) { | 6789 | if (is_var_ctx_off_allowed(env->prog)) { |
| 7025 | int err = check_mem_region_access(env, regno, 0, access_size, U16_MAX, | 6790 | int err = check_mem_region_access(env, reg, argno, 0, access_size, U16_MAX, |
| 7026 | zero_size_allowed); | 6791 | zero_size_allowed); |
| 7027 | if (err) | 6792 | if (err) |
| 7028 | return err; | 6793 | return err; |
| 7029 | if (env->prog->aux->max_ctx_offset < reg->umax_value + access_size) | 6794 | if (env->prog->aux->max_ctx_offset < reg_umax(reg) + access_size) |
| 7030 | env->prog->aux->max_ctx_offset = reg->umax_value + access_size; | 6795 | env->prog->aux->max_ctx_offset = reg_umax(reg) + access_size; |
| 7031 | return 0; | 6796 | return 0; |
| 7032 | } | 6797 | } |
| 7033 | fallthrough; | 6798 | fallthrough; |
| @@ -7037,7 +6802,7 @@ static int check_helper_mem_access(struct bpf_verifier_env *env, int regno, · 3 unmodified lines | |||
| 7037 | bpf_register_is_null(reg)) | 6802 | bpf_register_is_null(reg)) |
| 7038 | return 0; | 6803 | return 0; |
| 7039 | 6804 | ||
| 7040 | verbose(env, "R%d type=%s ", regno, | 6805 | verbose(env, "%s type=%s ", reg_arg_name(env, argno), |
| 7041 | reg_type_str(env, reg->type)); | 6806 | reg_type_str(env, reg->type)); |
| 7042 | verbose(env, "expected=%s\n", reg_type_str(env, PTR_TO_STACK)); | 6807 | verbose(env, "expected=%s\n", reg_type_str(env, PTR_TO_STACK)); |
| 7043 | return -EACCES; | 6808 | return -EACCES; |
| @@ -7047,12 +6812,12 @@ static int check_helper_mem_access(struct bpf_verifier_env *env, int regno, · 3 unmodified lines | |||
| 7047 | /* verify arguments to helpers or kfuncs consisting of a pointer and an access | 6812 | /* verify arguments to helpers or kfuncs consisting of a pointer and an access |
| 7048 | * size. | 6813 | * size. |
| 7049 | * | 6814 | * |
| 7050 | * @regno is the register containing the access size. regno-1 is the register | 6815 | * @mem_reg contains the pointer, @size_reg contains the access size. |
| 7051 | * containing the pointer. | ||
| 7052 | */ | 6816 | */ |
| 7053 | static int check_mem_size_reg(struct bpf_verifier_env *env, | 6817 | static int check_mem_size_reg(struct bpf_verifier_env *env, |
| 7054 | struct bpf_reg_state *reg, u32 regno, | 6818 | struct bpf_reg_state *mem_reg, |
| 7055 | enum bpf_access_type access_type, | 6819 | struct bpf_reg_state *size_reg, argno_t mem_argno, |
| 6820 | argno_t size_argno, enum bpf_access_type access_type, | ||
| 7056 | bool zero_size_allowed, | 6821 | bool zero_size_allowed, |
| 7057 | struct bpf_call_arg_meta *meta) | 6822 | struct bpf_call_arg_meta *meta) |
| 7058 | { | 6823 | { |
| @@ -7066,42 +6831,48 @@ static int check_mem_size_reg(struct bpf_verifier_env *env, · 7 unmodified lines | |||
| 7066 | * out. Only upper bounds can be learned because retval is an | 6831 | * out. Only upper bounds can be learned because retval is an |
| 7067 | * int type and negative retvals are allowed. | 6832 | * int type and negative retvals are allowed. |
| 7068 | */ | 6833 | */ |
| 7069 | meta->msize_max_value = reg->umax_value; | 6834 | meta->msize_max_value = reg_umax(size_reg); |
| 7070 | 6835 | ||
| 7071 | /* The register is SCALAR_VALUE; the access check happens using | 6836 | /* The register is SCALAR_VALUE; the access check happens using |
| 7072 | * its boundaries. For unprivileged variable accesses, disable | 6837 | * its boundaries. For unprivileged variable accesses, disable |
| 7073 | * raw mode so that the program is required to initialize all | 6838 | * raw mode so that the program is required to initialize all |
| 7074 | * the memory that the helper could just partially fill up. | 6839 | * the memory that the helper could just partially fill up. |
| 7075 | */ | 6840 | */ |
| 7076 | if (!tnum_is_const(reg->var_off)) | 6841 | if (!tnum_is_const(size_reg->var_off)) |
| 7077 | meta = NULL; | 6842 | meta = NULL; |
| 7078 | 6843 | ||
| 7079 | if (reg->smin_value < 0) { | 6844 | if (reg_smin(size_reg) < 0) { |
| 7080 | verbose(env, "R%d min value is negative, either use unsigned or 'var &= const'\n", | 6845 | verbose(env, "%s min value is negative, either use unsigned or 'var &= const'\n", |
| 7081 | regno); | 6846 | reg_arg_name(env, size_argno)); |
| 7082 | return -EACCES; | 6847 | return -EACCES; |
| 7083 | } | 6848 | } |
| 7084 | 6849 | ||
| 7085 | if (reg->umin_value == 0 && !zero_size_allowed) { | 6850 | if (reg_umin(size_reg) == 0 && !zero_size_allowed) { |
| 7086 | verbose(env, "R%d invalid zero-sized read: u64=[%lld,%lld]\n", | 6851 | verbose(env, "%s invalid zero-sized read: u64=[%lld,%lld]\n", |
| 7087 | regno, reg->umin_value, reg->umax_value); | 6852 | reg_arg_name(env, size_argno), reg_umin(size_reg), reg_umax(size_reg)); |
| 7088 | return -EACCES; | 6853 | return -EACCES; |
| 7089 | } | 6854 | } |
| 7090 | 6855 | ||
| 7091 | if (reg->umax_value >= BPF_MAX_VAR_SIZ) { | 6856 | if (reg_umax(size_reg) >= BPF_MAX_VAR_SIZ) { |
| 7092 | verbose(env, "R%d unbounded memory access, use 'var &= const' or 'if (var < const)'\n", | 6857 | verbose(env, "%s unbounded memory access, use 'var &= const' or 'if (var < const)'\n", |
| 7093 | regno); | 6858 | reg_arg_name(env, size_argno)); |
| 7094 | return -EACCES; | 6859 | return -EACCES; |
| 7095 | } | 6860 | } |
| 7096 | err = check_helper_mem_access(env, regno - 1, reg->umax_value, | 6861 | err = check_helper_mem_access(env, mem_reg, mem_argno, reg_umax(size_reg), |
| 7097 | access_type, zero_size_allowed, meta); | 6862 | access_type, zero_size_allowed, meta); |
| 7098 | if (!err) | 6863 | if (!err) { |
| 7099 | err = mark_chain_precision(env, regno); | 6864 | int regno = reg_from_argno(size_argno); |
| 6865 | |||
| 6866 | if (regno >= 0) | ||
| 6867 | err = mark_chain_precision(env, regno); | ||
| 6868 | else | ||
| 6869 | err = mark_stack_arg_precision(env, arg_idx_from_argno(size_argno)); | ||
| 6870 | } | ||
| 7100 | return err; | 6871 | return err; |
| 7101 | } | 6872 | } |
| 7102 | 6873 | ||
| 7103 | static int check_mem_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg, | 6874 | static int check_mem_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg, |
| 7104 | u32 regno, u32 mem_size) | 6875 | argno_t argno, u32 mem_size) |
| 7105 | { | 6876 | { |
| 7106 | bool may_be_null = type_may_be_null(reg->type); | 6877 | bool may_be_null = type_may_be_null(reg->type); |
| 7107 | struct bpf_reg_state saved_reg; | 6878 | struct bpf_reg_state saved_reg; |
| @@ -7110,6 +6881,12 @@ static int check_mem_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg · 2 unmodified lines | |||
| 7110 | if (bpf_register_is_null(reg)) | 6881 | if (bpf_register_is_null(reg)) |
| 7111 | return 0; | 6882 | return 0; |
| 7112 | 6883 | ||
| 6884 | if (mem_size > S32_MAX) { | ||
| 6885 | verbose(env, "%s memory size %u is too large\n", | ||
| 6886 | reg_arg_name(env, argno), mem_size); | ||
| 6887 | return -EACCES; | ||
| 6888 | } | ||
| 6889 | |||
| 7113 | /* Assuming that the register contains a value check if the memory | 6890 | /* Assuming that the register contains a value check if the memory |
| 7114 | * access is safe. Temporarily save and restore the register's state as | 6891 | * access is safe. Temporarily save and restore the register's state as |
| 7115 | * the conversion shouldn't be visible to a caller. | 6892 | * the conversion shouldn't be visible to a caller. |
| @@ -7121,8 +6898,8 @@ static int check_mem_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg · 5 unmodified lines | |||
| 7121 | 6898 | ||
| 7122 | int size = base_type(reg->type) == PTR_TO_STACK ? -(int)mem_size : mem_size; | 6899 | int size = base_type(reg->type) == PTR_TO_STACK ? -(int)mem_size : mem_size; |
| 7123 | 6900 | ||
| 7124 | err = check_helper_mem_access(env, regno, size, BPF_READ, true, NULL); | 6901 | err = check_helper_mem_access(env, reg, argno, size, BPF_READ, true, NULL); |
| 7125 | err = err ?: check_helper_mem_access(env, regno, size, BPF_WRITE, true, NULL); | 6902 | err = err ?: check_helper_mem_access(env, reg, argno, size, BPF_WRITE, true, NULL); |
| 7126 | 6903 | ||
| 7127 | if (may_be_null) | 6904 | if (may_be_null) |
| 7128 | *reg = saved_reg; | 6905 | *reg = saved_reg; |
| @@ -7130,17 +6907,14 @@ static int check_mem_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg · 1 unmodified line | |||
| 7130 | return err; | 6907 | return err; |
| 7131 | } | 6908 | } |
| 7132 | 6909 | ||
| 7133 | static int check_kfunc_mem_size_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg, | 6910 | static int check_kfunc_mem_size_reg(struct bpf_verifier_env *env, struct bpf_reg_state *mem_reg, |
| 7134 | u32 regno) | 6911 | struct bpf_reg_state *size_reg, argno_t mem_argno, argno_t size_argno) |
| 7135 | { | 6912 | { |
| 7136 | struct bpf_reg_state *mem_reg = &cur_regs(env)[regno - 1]; | ||
| 7137 | bool may_be_null = type_may_be_null(mem_reg->type); | 6913 | bool may_be_null = type_may_be_null(mem_reg->type); |
| 7138 | struct bpf_reg_state saved_reg; | 6914 | struct bpf_reg_state saved_reg; |
| 7139 | struct bpf_call_arg_meta meta; | 6915 | struct bpf_call_arg_meta meta; |
| 7140 | int err; | 6916 | int err; |
| 7141 | 6917 | ||
| 7142 | WARN_ON_ONCE(regno < BPF_REG_2 || regno > BPF_REG_5); | ||
| 7143 | |||
| 7144 | memset(&meta, 0, sizeof(meta)); | 6918 | memset(&meta, 0, sizeof(meta)); |
| 7145 | 6919 | ||
| 7146 | if (may_be_null) { | 6920 | if (may_be_null) { |
| @@ -7148,8 +6922,8 @@ static int check_kfunc_mem_size_reg(struct bpf_verifier_env *env, struct bpf_reg · 1 unmodified line | |||
| 7148 | mark_ptr_not_null_reg(mem_reg); | 6922 | mark_ptr_not_null_reg(mem_reg); |
| 7149 | } | 6923 | } |
| 7150 | 6924 | ||
| 7151 | err = check_mem_size_reg(env, reg, regno, BPF_READ, true, &meta); | 6925 | err = check_mem_size_reg(env, mem_reg, size_reg, mem_argno, size_argno, BPF_READ, true, &meta); |
| 7152 | err = err ?: check_mem_size_reg(env, reg, regno, BPF_WRITE, true, &meta); | 6926 | err = err ?: check_mem_size_reg(env, mem_reg, size_reg, mem_argno, size_argno, BPF_WRITE, true, &meta); |
| 7153 | 6927 | ||
| 7154 | if (may_be_null) | 6928 | if (may_be_null) |
| 7155 | *mem_reg = saved_reg; | 6929 | *mem_reg = saved_reg; |
| @@ -7185,11 +6959,10 @@ enum { · 29 unmodified lines | |||
| 7185 | * env->cur_state->active_locks remembers which map value element or allocated | 6959 | * env->cur_state->active_locks remembers which map value element or allocated |
| 7186 | * object got locked and clears it after bpf_spin_unlock. | 6960 | * object got locked and clears it after bpf_spin_unlock. |
| 7187 | */ | 6961 | */ |
| 7188 | static int process_spin_lock(struct bpf_verifier_env *env, int regno, int flags) | 6962 | static int process_spin_lock(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno, int flags) |
| 7189 | { | 6963 | { |
| 7190 | bool is_lock = flags & PROCESS_SPIN_LOCK, is_res_lock = flags & PROCESS_RES_LOCK; | 6964 | bool is_lock = flags & PROCESS_SPIN_LOCK, is_res_lock = flags & PROCESS_RES_LOCK; |
| 7191 | const char *lock_str = is_res_lock ? "bpf_res_spin" : "bpf_spin"; | 6965 | const char *lock_str = is_res_lock ? "bpf_res_spin" : "bpf_spin"; |
| 7192 | struct bpf_reg_state *reg = reg_state(env, regno); | ||
| 7193 | struct bpf_verifier_state *cur = env->cur_state; | 6966 | struct bpf_verifier_state *cur = env->cur_state; |
| 7194 | bool is_const = tnum_is_const(reg->var_off); | 6967 | bool is_const = tnum_is_const(reg->var_off); |
| 7195 | bool is_irq = flags & PROCESS_LOCK_IRQ; | 6968 | bool is_irq = flags & PROCESS_LOCK_IRQ; |
| @@ -7202,8 +6975,8 @@ static int process_spin_lock(struct bpf_verifier_env *env, int regno, int flags) · 6 unmodified lines | |||
| 7202 | 6975 | ||
| 7203 | if (!is_const) { | 6976 | if (!is_const) { |
| 7204 | verbose(env, | 6977 | verbose(env, |
| 7205 | "R%d doesn't have constant offset. %s_lock has to be at the constant offset\n", | 6978 | "%s doesn't have constant offset. %s_lock has to be at the constant offset\n", |
| 7206 | regno, lock_str); | 6979 | reg_arg_name(env, argno), lock_str); |
| 7207 | return -EINVAL; | 6980 | return -EINVAL; |
| 7208 | } | 6981 | } |
| 7209 | if (reg->type == PTR_TO_MAP_VALUE) { | 6982 | if (reg->type == PTR_TO_MAP_VALUE) { |
| @@ -7302,11 +7075,10 @@ static int process_spin_lock(struct bpf_verifier_env *env, int regno, int flags) · 92 unmodified lines | |||
| 7302 | } | 7075 | } |
| 7303 | 7076 | ||
| 7304 | /* Check if @regno is a pointer to a specific field in a map value */ | 7077 | /* Check if @regno is a pointer to a specific field in a map value */ |
| 7305 | static int check_map_field_pointer(struct bpf_verifier_env *env, u32 regno, | 7078 | static int check_map_field_pointer(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno, |
| 7306 | enum btf_field_type field_type, | 7079 | enum btf_field_type field_type, |
| 7307 | struct bpf_map_desc *map_desc) | 7080 | struct bpf_map_desc *map_desc) |
| 7308 | { | 7081 | { |
| 7309 | struct bpf_reg_state *reg = reg_state(env, regno); | ||
| 7310 | bool is_const = tnum_is_const(reg->var_off); | 7082 | bool is_const = tnum_is_const(reg->var_off); |
| 7311 | struct bpf_map *map = reg->map_ptr; | 7083 | struct bpf_map *map = reg->map_ptr; |
| 7312 | u64 val = reg->var_off.value; | 7084 | u64 val = reg->var_off.value; |
| @@ -7315,8 +7087,8 @@ static int check_map_field_pointer(struct bpf_verifier_env *env, u32 regno, · 2 unmodified lines | |||
| 7315 | 7087 | ||
| 7316 | if (!is_const) { | 7088 | if (!is_const) { |
| 7317 | verbose(env, | 7089 | verbose(env, |
| 7318 | "R%d doesn't have constant offset. %s has to be at the constant offset\n", | 7090 | "%s doesn't have constant offset. %s has to be at the constant offset\n", |
| 7319 | regno, struct_name); | 7091 | reg_arg_name(env, argno), struct_name); |
| 7320 | return -EINVAL; | 7092 | return -EINVAL; |
| 7321 | } | 7093 | } |
| 7322 | if (!map->btf) { | 7094 | if (!map->btf) { |
| @@ -7356,26 +7128,26 @@ static int check_map_field_pointer(struct bpf_verifier_env *env, u32 regno, · 33 unmodified lines | |||
| 7356 | return 0; | 7128 | return 0; |
| 7357 | } | 7129 | } |
| 7358 | 7130 | ||
| 7359 | static int process_timer_func(struct bpf_verifier_env *env, int regno, | 7131 | static int process_timer_func(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno, |
| 7360 | struct bpf_map_desc *map) | 7132 | struct bpf_map_desc *map) |
| 7361 | { | 7133 | { |
| 7362 | if (IS_ENABLED(CONFIG_PREEMPT_RT)) { | 7134 | if (IS_ENABLED(CONFIG_PREEMPT_RT)) { |
| 7363 | verbose(env, "bpf_timer cannot be used for PREEMPT_RT.\n"); | 7135 | verbose(env, "bpf_timer cannot be used for PREEMPT_RT.\n"); |
| 7364 | return -EOPNOTSUPP; | 7136 | return -EOPNOTSUPP; |
| 7365 | } | 7137 | } |
| 7366 | return check_map_field_pointer(env, regno, BPF_TIMER, map); | 7138 | return check_map_field_pointer(env, reg, argno, BPF_TIMER, map); |
| 7367 | } | 7139 | } |
| 7368 | 7140 | ||
| 7369 | static int process_timer_helper(struct bpf_verifier_env *env, int regno, | 7141 | static int process_timer_helper(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno, |
| 7370 | struct bpf_call_arg_meta *meta) | 7142 | struct bpf_call_arg_meta *meta) |
| 7371 | { | 7143 | { |
| 7372 | return process_timer_func(env, regno, &meta->map); | 7144 | return process_timer_func(env, reg, argno, &meta->map); |
| 7373 | } | 7145 | } |
| 7374 | 7146 | ||
| 7375 | static int process_timer_kfunc(struct bpf_verifier_env *env, int regno, | 7147 | static int process_timer_kfunc(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno, |
| 7376 | struct bpf_kfunc_call_arg_meta *meta) | 7148 | struct bpf_kfunc_call_arg_meta *meta) |
| 7377 | { | 7149 | { |
| 7378 | return process_timer_func(env, regno, &meta->map); | 7150 | return process_timer_func(env, reg, argno, &meta->map); |
| 7379 | } | 7151 | } |
| 7380 | 7152 | ||
| 7381 | static int process_kptr_func(struct bpf_verifier_env *env, int regno, | 7153 | static int process_kptr_func(struct bpf_verifier_env *env, int regno, |
| @@ -7426,52 +7198,42 @@ static int process_kptr_func(struct bpf_verifier_env *env, int regno, · 44 unmodified lines | |||
| 7426 | return 0; | 7198 | return 0; |
| 7427 | } | 7199 | } |
| 7428 | 7200 | ||
| 7429 | /* There are two register types representing a bpf_dynptr, one is PTR_TO_STACK | 7201 | /* |
| 7202 | * Validate dynptr arguments for helper, kfunc and subprog. | ||
| 7203 | * | ||
| 7204 | * @dynptr is both input and output. It is populated when the argument is | ||
| 7205 | * tagged with MEM_UNINIT (i.e., the dynptr argument that will be constructed) | ||
| 7206 | * and consumed when the argument is expecting to be an initialized dynptr. | ||
| 7207 | * @parent_id is used to track the referenced parent object (e.g., file or skb in | ||
| 7208 | * qdisc program) when constructing a dynptr. | ||
| 7209 | * | ||
| 7210 | * There are two register types representing a bpf_dynptr, one is PTR_TO_STACK | ||
| 7430 | * which points to a stack slot, and the other is CONST_PTR_TO_DYNPTR. | 7211 | * which points to a stack slot, and the other is CONST_PTR_TO_DYNPTR. |
| 7431 | * | 7212 | * |
| 7432 | * In both cases we deal with the first 8 bytes, but need to mark the next 8 | 7213 | * In both cases we deal with the first 8 bytes, but need to mark the next 8 |
| 7433 | * bytes as STACK_DYNPTR in case of PTR_TO_STACK. In case of | 7214 | * bytes as STACK_DYNPTR in case of PTR_TO_STACK. In case of |
| 7434 | * CONST_PTR_TO_DYNPTR, we are guaranteed to get the beginning of the object. | 7215 | * CONST_PTR_TO_DYNPTR, we are guaranteed to get the beginning of the object. |
| 7435 | * | 7216 | * |
| 7436 | * Mutability of bpf_dynptr is at two levels, one is at the level of struct | 7217 | * Mutability of bpf_dynptr is at two levels: the dynptr and the memory the |
| 7437 | * bpf_dynptr itself, i.e. whether the helper is receiving a pointer to struct | 7218 | * dynptr points to. At the first level, the verifier will make sure a |
| 7438 | * bpf_dynptr or pointer to const struct bpf_dynptr. In the former case, it can | 7219 | * CONST_PTR_TO_DYNPTR cannot be reinitialized or destroyed. The mutability of |
| 7439 | * mutate the view of the dynptr and also possibly destroy it. In the latter | 7220 | * a dynptr's view (i.e., start and offset) is not tracked as there is not such |
| 7440 | * case, it cannot mutate the bpf_dynptr itself but it can still mutate the | 7221 | * use case. The second level is tracked using the upper bit of bpf_dynptr->size |
| 7441 | * memory that dynptr points to. | 7222 | * and checked dynamically during runtime. |
| 7442 | * | ||
| 7443 | * The verifier will keep track both levels of mutation (bpf_dynptr's in | ||
| 7444 | * reg->type and the memory's in reg->dynptr.type), but there is no support for | ||
| 7445 | * readonly dynptr view yet, hence only the first case is tracked and checked. | ||
| 7446 | * | ||
| 7447 | * This is consistent with how C applies the const modifier to a struct object, | ||
| 7448 | * where the pointer itself inside bpf_dynptr becomes const but not what it | ||
| 7449 | * points to. | ||
| 7450 | * | ||
| 7451 | * Helpers which do not mutate the bpf_dynptr set MEM_RDONLY in their argument | ||
| 7452 | * type, and declare it as 'const struct bpf_dynptr *' in their prototype. | ||
| 7453 | */ | 7223 | */ |
| 7454 | static int process_dynptr_func(struct bpf_verifier_env *env, int regno, int insn_idx, | 7224 | static int process_dynptr_func(struct bpf_verifier_env *env, struct bpf_reg_state *reg, |
| 7455 | enum bpf_arg_type arg_type, int clone_ref_obj_id) | 7225 | argno_t argno, int insn_idx, enum bpf_arg_type arg_type, |
| 7226 | struct ref_obj_desc *ref_obj, struct bpf_dynptr_desc *dynptr) | ||
| 7456 | { | 7227 | { |
| 7457 | struct bpf_reg_state *reg = reg_state(env, regno); | 7228 | int spi, err = 0; |
| 7458 | int err; | ||
| 7459 | 7229 | ||
| 7460 | if (reg->type != PTR_TO_STACK && reg->type != CONST_PTR_TO_DYNPTR) { | 7230 | if (reg->type != PTR_TO_STACK && reg->type != CONST_PTR_TO_DYNPTR) { |
| 7461 | verbose(env, | 7231 | verbose(env, |
| 7462 | "arg#%d expected pointer to stack or const struct bpf_dynptr\n", | 7232 | "%s expected pointer to stack or const struct bpf_dynptr\n", |
| 7463 | regno - 1); | 7233 | reg_arg_name(env, argno)); |
| 7464 | return -EINVAL; | 7234 | return -EINVAL; |
| 7465 | } | 7235 | } |
| 7466 | 7236 | ||
| 7467 | /* MEM_UNINIT and MEM_RDONLY are exclusive, when applied to an | ||
| 7468 | * ARG_PTR_TO_DYNPTR (or ARG_PTR_TO_DYNPTR | DYNPTR_TYPE_*): | ||
| 7469 | */ | ||
| 7470 | if ((arg_type & (MEM_UNINIT | MEM_RDONLY)) == (MEM_UNINIT | MEM_RDONLY)) { | ||
| 7471 | verifier_bug(env, "misconfigured dynptr helper type flags"); | ||
| 7472 | return -EFAULT; | ||
| 7473 | } | ||
| 7474 | |||
| 7475 | /* MEM_UNINIT - Points to memory that is an appropriate candidate for | 7237 | /* MEM_UNINIT - Points to memory that is an appropriate candidate for |
| 7476 | * constructing a mutable bpf_dynptr object. | 7238 | * constructing a mutable bpf_dynptr object. |
| 7477 | * | 7239 | * |
| @@ -7479,13 +7241,12 @@ static int process_dynptr_func(struct bpf_verifier_env *env, int regno, int insn · 1 unmodified line | |||
| 7479 | * pointing to a region of at least 16 bytes which doesn't | 7241 | * pointing to a region of at least 16 bytes which doesn't |
| 7480 | * contain an existing bpf_dynptr. | 7242 | * contain an existing bpf_dynptr. |
| 7481 | * | 7243 | * |
| 7482 | * MEM_RDONLY - Points to a initialized bpf_dynptr that will not be | 7244 | * OBJ_RELEASE - Points to a initialized bpf_dynptr that will be |
| 7483 | * mutated or destroyed. However, the memory it points to | 7245 | * destroyed. |
| 7484 | * may be mutated. | ||
| 7485 | * | 7246 | * |
| 7486 | * None - Points to a initialized dynptr that can be mutated and | 7247 | * None - Points to a initialized dynptr that cannot be |
| 7487 | * destroyed, including mutation of the memory it points | 7248 | * reinitialized or destroyed. However, the view of the |
| 7488 | * to. | 7249 | * dynptr and the memory it points to may be mutated. |
| 7489 | */ | 7250 | */ |
| 7490 | if (arg_type & MEM_UNINIT) { | 7251 | if (arg_type & MEM_UNINIT) { |
| 7491 | int i; | 7252 | int i; |
| @@ -7497,45 +7258,58 @@ static int process_dynptr_func(struct bpf_verifier_env *env, int regno, int insn · 5 unmodified lines | |||
| 7497 | 7258 | ||
| 7498 | /* we write BPF_DW bits (8 bytes) at a time */ | 7259 | /* we write BPF_DW bits (8 bytes) at a time */ |
| 7499 | for (i = 0; i < BPF_DYNPTR_SIZE; i += 8) { | 7260 | for (i = 0; i < BPF_DYNPTR_SIZE; i += 8) { |
| 7500 | err = check_mem_access(env, insn_idx, regno, | 7261 | err = check_mem_access(env, insn_idx, reg, argno, |
| 7501 | i, BPF_DW, BPF_WRITE, -1, false, false); | 7262 | i, BPF_DW, BPF_WRITE, -1, false, false); |
| 7502 | if (err) | 7263 | if (err) |
| 7503 | return err; | 7264 | return err; |
| 7504 | } | 7265 | } |
| 7505 | 7266 | ||
| 7506 | err = mark_stack_slots_dynptr(env, reg, arg_type, insn_idx, clone_ref_obj_id); | 7267 | err = mark_stack_slots_dynptr(env, reg, arg_type, insn_idx, ref_obj, dynptr); |
| 7507 | } else /* MEM_RDONLY and None case from above */ { | 7268 | } else /* OBJ_RELEASE and None case from above */ { |
| 7508 | /* For the reg->type == PTR_TO_STACK case, bpf_dynptr is never const */ | 7269 | /* For the reg->type == PTR_TO_STACK case, bpf_dynptr is never const */ |
| 7509 | if (reg->type == CONST_PTR_TO_DYNPTR && !(arg_type & MEM_RDONLY)) { | 7270 | if (reg->type == CONST_PTR_TO_DYNPTR && (arg_type & OBJ_RELEASE)) { |
| 7510 | verbose(env, "cannot pass pointer to const bpf_dynptr, the helper mutates it\n"); | 7271 | verbose(env, "CONST_PTR_TO_DYNPTR cannot be released\n"); |
| 7511 | return -EINVAL; | 7272 | return -EINVAL; |
| 7512 | } | 7273 | } |
| 7513 | 7274 | ||
| 7514 | if (!is_dynptr_reg_valid_init(env, reg)) { | 7275 | if (!is_dynptr_reg_valid_init(env, reg)) { |
| 7515 | verbose(env, | 7276 | verbose(env, "Expected an initialized dynptr as %s\n", |
| 7516 | "Expected an initialized dynptr as arg #%d\n", | 7277 | reg_arg_name(env, argno)); |
| 7517 | regno - 1); | ||
| 7518 | return -EINVAL; | 7278 | return -EINVAL; |
| 7519 | } | 7279 | } |
| 7520 | 7280 | ||
| 7521 | /* Fold modifiers (in this case, MEM_RDONLY) when checking expected type */ | 7281 | /* Fold modifiers (in this case, OBJ_RELEASE) when checking expected type */ |
| 7522 | if (!is_dynptr_type_expected(env, reg, arg_type & ~MEM_RDONLY)) { | 7282 | if (!is_dynptr_type_expected(env, reg, arg_type & ~OBJ_RELEASE)) { |
| 7523 | verbose(env, | 7283 | verbose(env, |
| 7524 | "Expected a dynptr of type %s as arg #%d\n", | 7284 | "Expected a dynptr of type %s as %s\n", |
| 7525 | dynptr_type_str(arg_to_dynptr_type(arg_type)), regno - 1); | 7285 | dynptr_type_str(arg_to_dynptr_type(arg_type)), |
| 7286 | reg_arg_name(env, argno)); | ||
| 7526 | return -EINVAL; | 7287 | return -EINVAL; |
| 7527 | } | 7288 | } |
| 7528 | 7289 | ||
| 7529 | err = mark_dynptr_read(env, reg); | 7290 | if (reg->type != CONST_PTR_TO_DYNPTR) { |
| 7530 | } | 7291 | struct bpf_func_state *state = bpf_func(env, reg); |
| 7531 | return err; | ||
| 7532 | } | ||
| 7533 | 7292 | ||
| 7534 | static u32 iter_ref_obj_id(struct bpf_verifier_env *env, struct bpf_reg_state *reg, int spi) | 7293 | spi = dynptr_get_spi(env, reg); |
| 7535 | { | 7294 | if (spi < 0) |
| 7536 | struct bpf_func_state *state = bpf_func(env, reg); | 7295 | return spi; |
| 7296 | |||
| 7297 | /* | ||
| 7298 | * For CONST_PTR_TO_DYNPTR, reg is already scratched by check_reg_arg | ||
| 7299 | * in check_helper_call and mark_btf_func_reg_size in check_kfunc_call. | ||
| 7300 | */ | ||
| 7301 | mark_stack_slots_scratched(env, spi, BPF_DYNPTR_NR_SLOTS); | ||
| 7537 | 7302 | ||
| 7538 | return state->stack[spi].spilled_ptr.ref_obj_id; | 7303 | reg = &state->stack[spi].spilled_ptr; |
| 7304 | } | ||
| 7305 | |||
| 7306 | if (dynptr) { | ||
| 7307 | dynptr->type = reg->dynptr.type; | ||
| 7308 | dynptr->id = reg->id; | ||
| 7309 | dynptr->parent_id = reg->parent_id; | ||
| 7310 | } | ||
| 7311 | } | ||
| 7312 | return err; | ||
| 7539 | } | 7313 | } |
| 7540 | 7314 | ||
| 7541 | static bool is_iter_kfunc(struct bpf_kfunc_call_arg_meta *meta) | 7315 | static bool is_iter_kfunc(struct bpf_kfunc_call_arg_meta *meta) |
| @@ -7567,15 +7341,17 @@ static bool is_kfunc_arg_iter(struct bpf_kfunc_call_arg_meta *meta, int arg_idx, · 25 unmodified lines | |||
| 7567 | return btf_param_match_suffix(meta->btf, arg, "__iter"); | 7341 | return btf_param_match_suffix(meta->btf, arg, "__iter"); |
| 7568 | } | 7342 | } |
| 7569 | 7343 | ||
| 7570 | static int process_iter_arg(struct bpf_verifier_env *env, int regno, int insn_idx, | 7344 | static int process_iter_arg(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno, int insn_idx, |
| 7571 | struct bpf_kfunc_call_arg_meta *meta) | 7345 | struct bpf_kfunc_call_arg_meta *meta) |
| 7572 | { | 7346 | { |
| 7573 | struct bpf_reg_state *reg = reg_state(env, regno); | 7347 | struct bpf_func_state *state = bpf_func(env, reg); |
| 7574 | const struct btf_type *t; | 7348 | const struct btf_type *t; |
| 7349 | u32 arg_idx = arg_idx_from_argno(argno); | ||
| 7575 | int spi, err, i, nr_slots, btf_id; | 7350 | int spi, err, i, nr_slots, btf_id; |
| 7576 | 7351 | ||
| 7577 | if (reg->type != PTR_TO_STACK) { | 7352 | if (reg->type != PTR_TO_STACK) { |
| 7578 | verbose(env, "arg#%d expected pointer to an iterator on stack\n", regno - 1); | 7353 | verbose(env, "%s expected pointer to an iterator on stack\n", |
| 7354 | reg_arg_name(env, argno)); | ||
| 7579 | return -EINVAL; | 7355 | return -EINVAL; |
| 7580 | } | 7356 | } |
| 7581 | 7357 | ||
| @@ -7585,9 +7361,10 @@ static int process_iter_arg(struct bpf_verifier_env *env, int regno, int insn_id · 3 unmodified lines | |||
| 7585 | * to any kfunc, if arg has "__iter" suffix, we need to be a bit more | 7361 | * to any kfunc, if arg has "__iter" suffix, we need to be a bit more |
| 7586 | * conservative here. | 7362 | * conservative here. |
| 7587 | */ | 7363 | */ |
| 7588 | btf_id = btf_check_iter_arg(meta->btf, meta->func_proto, regno - 1); | 7364 | btf_id = btf_check_iter_arg(meta->btf, meta->func_proto, arg_idx); |
| 7589 | if (btf_id < 0) { | 7365 | if (btf_id < 0) { |
| 7590 | verbose(env, "expected valid iter pointer as arg #%d\n", regno - 1); | 7366 | verbose(env, "expected valid iter pointer as %s\n", |
| 7367 | reg_arg_name(env, argno)); | ||
| 7591 | return -EINVAL; | 7368 | return -EINVAL; |
| 7592 | } | 7369 | } |
| 7593 | t = btf_type_by_id(meta->btf, btf_id); | 7370 | t = btf_type_by_id(meta->btf, btf_id); |
| @@ -7596,13 +7373,13 @@ static int process_iter_arg(struct bpf_verifier_env *env, int regno, int insn_id · 2 unmodified lines | |||
| 7596 | if (is_iter_new_kfunc(meta)) { | 7373 | if (is_iter_new_kfunc(meta)) { |
| 7597 | /* bpf_iter_<type>_new() expects pointer to uninit iter state */ | 7374 | /* bpf_iter_<type>_new() expects pointer to uninit iter state */ |
| 7598 | if (!is_iter_reg_valid_uninit(env, reg, nr_slots)) { | 7375 | if (!is_iter_reg_valid_uninit(env, reg, nr_slots)) { |
| 7599 | verbose(env, "expected uninitialized iter_%s as arg #%d\n", | 7376 | verbose(env, "expected uninitialized iter_%s as %s\n", |
| 7600 | iter_type_str(meta->btf, btf_id), regno - 1); | 7377 | iter_type_str(meta->btf, btf_id), reg_arg_name(env, argno)); |
| 7601 | return -EINVAL; | 7378 | return -EINVAL; |
| 7602 | } | 7379 | } |
| 7603 | 7380 | ||
| 7604 | for (i = 0; i < nr_slots * 8; i += BPF_REG_SIZE) { | 7381 | for (i = 0; i < nr_slots * 8; i += BPF_REG_SIZE) { |
| 7605 | err = check_mem_access(env, insn_idx, regno, | 7382 | err = check_mem_access(env, insn_idx, reg, argno, |
| 7606 | i, BPF_DW, BPF_WRITE, -1, false, false); | 7383 | i, BPF_DW, BPF_WRITE, -1, false, false); |
| 7607 | if (err) | 7384 | if (err) |
| 7608 | return err; | 7385 | return err; |
| @@ -7620,8 +7397,8 @@ static int process_iter_arg(struct bpf_verifier_env *env, int regno, int insn_id · 11 unmodified lines | |||
| 7620 | case 0: | 7397 | case 0: |
| 7621 | break; | 7398 | break; |
| 7622 | case -EINVAL: | 7399 | case -EINVAL: |
| 7623 | verbose(env, "expected an initialized iter_%s as arg #%d\n", | 7400 | verbose(env, "expected an initialized iter_%s as %s\n", |
| 7624 | iter_type_str(meta->btf, btf_id), regno - 1); | 7401 | iter_type_str(meta->btf, btf_id), reg_arg_name(env, argno)); |
| 7625 | return err; | 7402 | return err; |
| 7626 | case -EPROTO: | 7403 | case -EPROTO: |
| 7627 | verbose(env, "expected an RCU CS when using %s\n", meta->func_name); | 7404 | verbose(env, "expected an RCU CS when using %s\n", meta->func_name); |
| @@ -7634,14 +7411,12 @@ static int process_iter_arg(struct bpf_verifier_env *env, int regno, int insn_id · 6 unmodified lines | |||
| 7634 | if (spi < 0) | 7411 | if (spi < 0) |
| 7635 | return spi; | 7412 | return spi; |
| 7636 | 7413 | ||
| 7637 | err = mark_iter_read(env, reg, spi, nr_slots); | 7414 | mark_stack_slots_scratched(env, spi, nr_slots); |
| 7638 | if (err) | ||
| 7639 | return err; | ||
| 7640 | 7415 | ||
| 7641 | /* remember meta->iter info for process_iter_next_call() */ | 7416 | /* remember meta->iter info for process_iter_next_call() */ |
| 7642 | meta->iter.spi = spi; | 7417 | meta->iter.spi = spi; |
| 7643 | meta->iter.frameno = reg->frameno; | 7418 | meta->iter.frameno = reg->frameno; |
| 7644 | meta->ref_obj_id = iter_ref_obj_id(env, reg, spi); | 7419 | update_ref_obj(&meta->ref_obj, &state->stack[spi].spilled_ptr); |
| 7645 | 7420 | ||
| 7646 | if (is_iter_destroy_kfunc(meta)) { | 7421 | if (is_iter_destroy_kfunc(meta)) { |
| 7647 | err = unmark_stack_slots_iter(env, reg, nr_slots); | 7422 | err = unmark_stack_slots_iter(env, reg, nr_slots); |
| @@ -8041,12 +7816,11 @@ static const struct bpf_reg_types *compatible_reg_types[__BPF_ARG_TYPE_MAX] = { · 393 unmodified lines | |||
| 8041 | [ARG_PTR_TO_DYNPTR] = &dynptr_types, | 7816 | [ARG_PTR_TO_DYNPTR] = &dynptr_types, |
| 8042 | }; | 7817 | }; |
| 8043 | 7818 | ||
| 8044 | static int check_reg_type(struct bpf_verifier_env *env, u32 regno, | 7819 | static int check_reg_type(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno, |
| 8045 | enum bpf_arg_type arg_type, | 7820 | enum bpf_arg_type arg_type, |
| 8046 | const u32 *arg_btf_id, | 7821 | const u32 *arg_btf_id, |
| 8047 | struct bpf_call_arg_meta *meta) | 7822 | struct bpf_call_arg_meta *meta) |
| 8048 | { | 7823 | { |
| 8049 | struct bpf_reg_state *reg = reg_state(env, regno); | ||
| 8050 | enum bpf_reg_type expected, type = reg->type; | 7824 | enum bpf_reg_type expected, type = reg->type; |
| 8051 | const struct bpf_reg_types *compatible; | 7825 | const struct bpf_reg_types *compatible; |
| 8052 | int i, j, err; | 7826 | int i, j, err; |
| @@ -8077,7 +7851,7 @@ static int check_reg_type(struct bpf_verifier_env *env, u32 regno, · 24 unmodified lines | |||
| 8077 | type &= ~DYNPTR_TYPE_FLAG_MASK; | 7851 | type &= ~DYNPTR_TYPE_FLAG_MASK; |
| 8078 | 7852 | ||
| 8079 | /* Local kptr types are allowed as the source argument of bpf_kptr_xchg */ | 7853 | /* Local kptr types are allowed as the source argument of bpf_kptr_xchg */ |
| 8080 | if (meta->func_id == BPF_FUNC_kptr_xchg && type_is_alloc(type) && regno == BPF_REG_2) { | 7854 | if (meta->func_id == BPF_FUNC_kptr_xchg && type_is_alloc(type) && reg_from_argno(argno) == BPF_REG_2) { |
| 8081 | type &= ~MEM_ALLOC; | 7855 | type &= ~MEM_ALLOC; |
| 8082 | type &= ~MEM_PERCPU; | 7856 | type &= ~MEM_PERCPU; |
| 8083 | } | 7857 | } |
| @@ -8091,7 +7865,7 @@ static int check_reg_type(struct bpf_verifier_env *env, u32 regno, · 7 unmodified lines | |||
| 8091 | goto found; | 7865 | goto found; |
| 8092 | } | 7866 | } |
| 8093 | 7867 | ||
| 8094 | verbose(env, "R%d type=%s expected=", regno, reg_type_str(env, reg->type)); | 7868 | verbose(env, "%s type=%s expected=", reg_arg_name(env, argno), reg_type_str(env, reg->type)); |
| 8095 | for (j = 0; j + 1 < i; j++) | 7869 | for (j = 0; j + 1 < i; j++) |
| 8096 | verbose(env, "%s, ", reg_type_str(env, compatible->types[j])); | 7870 | verbose(env, "%s, ", reg_type_str(env, compatible->types[j])); |
| 8097 | verbose(env, "%s\n", reg_type_str(env, compatible->types[j])); | 7871 | verbose(env, "%s\n", reg_type_str(env, compatible->types[j])); |
| @@ -8104,9 +7878,9 @@ found: · 6 unmodified lines | |||
| 8104 | if (compatible == &mem_types) { | 7878 | if (compatible == &mem_types) { |
| 8105 | if (!(arg_type & MEM_RDONLY)) { | 7879 | if (!(arg_type & MEM_RDONLY)) { |
| 8106 | verbose(env, | 7880 | verbose(env, |
| 8107 | "%s() may write into memory pointed by R%d type=%s\n", | 7881 | "%s() may write into memory pointed by %s type=%s\n", |
| 8108 | func_id_name(meta->func_id), | 7882 | func_id_name(meta->func_id), |
| 8109 | regno, reg_type_str(env, reg->type)); | 7883 | reg_arg_name(env, argno), reg_type_str(env, reg->type)); |
| 8110 | return -EACCES; | 7884 | return -EACCES; |
| 8111 | } | 7885 | } |
| 8112 | return 0; | 7886 | return 0; |
| @@ -8129,7 +7903,8 @@ found: · 16 unmodified lines | |||
| 8129 | 7903 | ||
| 8130 | if (type_may_be_null(reg->type) && | 7904 | if (type_may_be_null(reg->type) && |
| 8131 | (!type_may_be_null(arg_type) || arg_type_is_release(arg_type))) { | 7905 | (!type_may_be_null(arg_type) || arg_type_is_release(arg_type))) { |
| 8132 | verbose(env, "Possibly NULL pointer passed to helper arg%d\n", regno); | 7906 | verbose(env, "Possibly NULL pointer passed to helper %s\n", |
| 7907 | reg_arg_name(env, argno)); | ||
| 8133 | return -EACCES; | 7908 | return -EACCES; |
| 8134 | } | 7909 | } |
| 8135 | 7910 | ||
| @@ -8142,25 +7917,26 @@ found: · 6 unmodified lines | |||
| 8142 | } | 7917 | } |
| 8143 | 7918 | ||
| 8144 | if (meta->func_id == BPF_FUNC_kptr_xchg) { | 7919 | if (meta->func_id == BPF_FUNC_kptr_xchg) { |
| 8145 | if (map_kptr_match_type(env, meta->kptr_field, reg, regno)) | 7920 | if (map_kptr_match_type(env, meta->kptr_field, reg, reg_from_argno(argno))) |
| 8146 | return -EACCES; | 7921 | return -EACCES; |
| 8147 | } else { | 7922 | } else { |
| 8148 | if (arg_btf_id == BPF_PTR_POISON) { | 7923 | if (arg_btf_id == BPF_PTR_POISON) { |
| 8149 | verbose(env, "verifier internal error:"); | 7924 | verbose(env, "verifier internal error:"); |
| 8150 | verbose(env, "R%d has non-overwritten BPF_PTR_POISON type\n", | 7925 | verbose(env, "%s has non-overwritten BPF_PTR_POISON type\n", |
| 8151 | regno); | 7926 | reg_arg_name(env, argno)); |
| 8152 | return -EACCES; | 7927 | return -EACCES; |
| 8153 | } | 7928 | } |
| 8154 | 7929 | ||
| 8155 | err = __check_ptr_off_reg(env, reg, regno, true); | 7930 | err = __check_ptr_off_reg(env, reg, argno, true); |
| 8156 | if (err) | 7931 | if (err) |
| 8157 | return err; | 7932 | return err; |
| 8158 | 7933 | ||
| 8159 | if (!btf_struct_ids_match(&env->log, reg->btf, reg->btf_id, | 7934 | if (!btf_struct_ids_match(&env->log, reg->btf, reg->btf_id, |
| 8160 | reg->var_off.value, btf_vmlinux, *arg_btf_id, | 7935 | reg->var_off.value, btf_vmlinux, *arg_btf_id, |
| 8161 | strict_type_match)) { | 7936 | strict_type_match)) { |
| 8162 | verbose(env, "R%d is of type %s but %s is expected\n", | 7937 | verbose(env, "%s is of type %s but %s is expected\n", |
| 8163 | regno, btf_type_name(reg->btf, reg->btf_id), | 7938 | reg_arg_name(env, argno), |
| 7939 | btf_type_name(reg->btf, reg->btf_id), | ||
| 8164 | btf_type_name(btf_vmlinux, *arg_btf_id)); | 7940 | btf_type_name(btf_vmlinux, *arg_btf_id)); |
| 8165 | return -EACCES; | 7941 | return -EACCES; |
| 8166 | } | 7942 | } |
| @@ -8177,8 +7953,11 @@ found: · 10 unmodified lines | |||
| 8177 | return -EFAULT; | 7953 | return -EFAULT; |
| 8178 | } | 7954 | } |
| 8179 | /* Check if local kptr in src arg matches kptr in dst arg */ | 7955 | /* Check if local kptr in src arg matches kptr in dst arg */ |
| 8180 | if (meta->func_id == BPF_FUNC_kptr_xchg && regno == BPF_REG_2) { | 7956 | if (meta->func_id == BPF_FUNC_kptr_xchg) { |
| 8181 | if (map_kptr_match_type(env, meta->kptr_field, reg, regno)) | 7957 | int regno = reg_from_argno(argno); |
| 7958 | |||
| 7959 | if (regno == BPF_REG_2 && | ||
| 7960 | map_kptr_match_type(env, meta->kptr_field, reg, regno)) | ||
| 8182 | return -EACCES; | 7961 | return -EACCES; |
| 8183 | } | 7962 | } |
| 8184 | break; | 7963 | break; |
| @@ -8212,7 +7991,7 @@ reg_find_field_offset(const struct bpf_reg_state *reg, s32 off, u32 fields) · 27 unmodified lines | |||
| 8212 | } | 7991 | } |
| 8213 | 7992 | ||
| 8214 | static int check_func_arg_reg_off(struct bpf_verifier_env *env, | 7993 | static int check_func_arg_reg_off(struct bpf_verifier_env *env, |
| 8215 | const struct bpf_reg_state *reg, int regno, | 7994 | const struct bpf_reg_state *reg, argno_t argno, |
| 8216 | enum bpf_arg_type arg_type) | 7995 | enum bpf_arg_type arg_type) |
| 8217 | { | 7996 | { |
| 8218 | u32 type = reg->type; | 7997 | u32 type = reg->type; |
| @@ -8220,7 +7999,7 @@ static int check_func_arg_reg_off(struct bpf_verifier_env *env, · 1 unmodified line | |||
| 8220 | /* When referenced register is passed to release function, its fixed | 7999 | /* When referenced register is passed to release function, its fixed |
| 8221 | * offset must be 0. | 8000 | * offset must be 0. |
| 8222 | * | 8001 | * |
| 8223 | * We will check arg_type_is_release reg has ref_obj_id when storing | 8002 | * We will check arg_type_is_release reg has id when storing |
| 8224 | * meta->release_regno. | 8003 | * meta->release_regno. |
| 8225 | */ | 8004 | */ |
| 8226 | if (arg_type_is_release(arg_type)) { | 8005 | if (arg_type_is_release(arg_type)) { |
| @@ -8238,8 +8017,8 @@ static int check_func_arg_reg_off(struct bpf_verifier_env *env, · 11 unmodified lines | |||
| 8238 | * to give the user a better error message. | 8017 | * to give the user a better error message. |
| 8239 | */ | 8018 | */ |
| 8240 | if (!tnum_is_const(reg->var_off) || reg->var_off.value != 0) { | 8019 | if (!tnum_is_const(reg->var_off) || reg->var_off.value != 0) { |
| 8241 | verbose(env, "R%d must have zero offset when passed to release func or trusted arg to kfunc\n", | 8020 | verbose(env, "%s must have zero offset when passed to release func or trusted arg to kfunc\n", |
| 8242 | regno); | 8021 | reg_arg_name(env, argno)); |
| 8243 | return -EINVAL; | 8022 | return -EINVAL; |
| 8244 | } | 8023 | } |
| 8245 | } | 8024 | } |
| @@ -8272,93 +8051,27 @@ static int check_func_arg_reg_off(struct bpf_verifier_env *env, · 26 unmodified lines | |||
| 8272 | * its fixed offset must be 0. In the other cases, fixed offset | 8051 | * its fixed offset must be 0. In the other cases, fixed offset |
| 8273 | * can be non-zero. This was already checked above. So pass | 8052 | * can be non-zero. This was already checked above. So pass |
| 8274 | * fixed_off_ok as true to allow fixed offset for all other | 8053 | * fixed_off_ok as true to allow fixed offset for all other |
| 8275 | * cases. var_off always must be 0 for PTR_TO_BTF_ID, hence we | 8054 | * cases. var_off always must be 0 for PTR_TO_BTF_ID, hence we |
| 8276 | * still need to do checks instead of returning. | 8055 | * still need to do checks instead of returning. |
| 8277 | */ | 8056 | */ |
| 8278 | return __check_ptr_off_reg(env, reg, regno, true); | 8057 | return __check_ptr_off_reg(env, reg, argno, true); |
| 8279 | case PTR_TO_CTX: | 8058 | case PTR_TO_CTX: |
| 8280 | /* | 8059 | /* |
| 8281 | * Allow fixed and variable offsets for syscall context, but | 8060 | * Allow fixed and variable offsets for syscall context, but |
| 8282 | * only when the argument is passed as memory, not ctx, | 8061 | * only when the argument is passed as memory, not ctx, |
| 8283 | * otherwise we may get modified ctx in tail called programs and | 8062 | * otherwise we may get modified ctx in tail called programs and |
| 8284 | * global subprogs (that may act as extension prog hooks). | 8063 | * global subprogs (that may act as extension prog hooks). |
| 8285 | */ | 8064 | */ |
| 8286 | if (arg_type != ARG_PTR_TO_CTX && is_var_ctx_off_allowed(env->prog)) | 8065 | if (arg_type != ARG_PTR_TO_CTX && is_var_ctx_off_allowed(env->prog)) |
| 8287 | return 0; | 8066 | return 0; |
| 8288 | fallthrough; | 8067 | fallthrough; |
| 8289 | default: | 8068 | default: |
| 8290 | return __check_ptr_off_reg(env, reg, regno, false); | 8069 | return __check_ptr_off_reg(env, reg, argno, false); |
| 8291 | } | ||
| 8292 | } | ||
| 8293 | |||
| 8294 | static struct bpf_reg_state *get_dynptr_arg_reg(struct bpf_verifier_env *env, | ||
| 8295 | const struct bpf_func_proto *fn, | ||
| 8296 | struct bpf_reg_state *regs) | ||
| 8297 | { | ||
| 8298 | struct bpf_reg_state *state = NULL; | ||
| 8299 | int i; | ||
| 8300 | |||
| 8301 | for (i = 0; i < MAX_BPF_FUNC_REG_ARGS; i++) | ||
| 8302 | if (arg_type_is_dynptr(fn->arg_type[i])) { | ||
| 8303 | if (state) { | ||
| 8304 | verbose(env, "verifier internal error: multiple dynptr args\n"); | ||
| 8305 | return NULL; | ||
| 8306 | } | ||
| 8307 | state = ®s[BPF_REG_1 + i]; | ||
| 8308 | } | ||
| 8309 | |||
| 8310 | if (!state) | ||
| 8311 | verbose(env, "verifier internal error: no dynptr arg found\n"); | ||
| 8312 | |||
| 8313 | return state; | ||
| 8314 | } | ||
| 8315 | |||
| 8316 | static int dynptr_id(struct bpf_verifier_env *env, struct bpf_reg_state *reg) | ||
| 8317 | { | ||
| 8318 | struct bpf_func_state *state = bpf_func(env, reg); | ||
| 8319 | int spi; | ||
| 8320 | |||
| 8321 | if (reg->type == CONST_PTR_TO_DYNPTR) | ||
| 8322 | return reg->id; | ||
| 8323 | spi = dynptr_get_spi(env, reg); | ||
| 8324 | if (spi < 0) | ||
| 8325 | return spi; | ||
| 8326 | return state->stack[spi].spilled_ptr.id; | ||
| 8327 | } | ||
| 8328 | |||
| 8329 | static int dynptr_ref_obj_id(struct bpf_verifier_env *env, struct bpf_reg_state *reg) | ||
| 8330 | { | ||
| 8331 | struct bpf_func_state *state = bpf_func(env, reg); | ||
| 8332 | int spi; | ||
| 8333 | |||
| 8334 | if (reg->type == CONST_PTR_TO_DYNPTR) | ||
| 8335 | return reg->ref_obj_id; | ||
| 8336 | spi = dynptr_get_spi(env, reg); | ||
| 8337 | if (spi < 0) | ||
| 8338 | return spi; | ||
| 8339 | return state->stack[spi].spilled_ptr.ref_obj_id; | ||
| 8340 | } | ||
| 8341 | |||
| 8342 | static enum bpf_dynptr_type dynptr_get_type(struct bpf_verifier_env *env, | ||
| 8343 | struct bpf_reg_state *reg) | ||
| 8344 | { | ||
| 8345 | struct bpf_func_state *state = bpf_func(env, reg); | ||
| 8346 | int spi; | ||
| 8347 | |||
| 8348 | if (reg->type == CONST_PTR_TO_DYNPTR) | ||
| 8349 | return reg->dynptr.type; | ||
| 8350 | |||
| 8351 | spi = bpf_get_spi(reg->var_off.value); | ||
| 8352 | if (spi < 0) { | ||
| 8353 | verbose(env, "verifier internal error: invalid spi when querying dynptr type\n"); | ||
| 8354 | return BPF_DYNPTR_TYPE_INVALID; | ||
| 8355 | } | 8070 | } |
| 8356 | |||
| 8357 | return state->stack[spi].spilled_ptr.dynptr.type; | ||
| 8358 | } | 8071 | } |
| 8359 | 8072 | ||
| 8360 | static int check_reg_const_str(struct bpf_verifier_env *env, | 8073 | static int check_arg_const_str(struct bpf_verifier_env *env, |
| 8361 | struct bpf_reg_state *reg, u32 regno) | 8074 | struct bpf_reg_state *reg, argno_t argno) |
| 8362 | { | 8075 | { |
| 8363 | struct bpf_map *map = reg->map_ptr; | 8076 | struct bpf_map *map = reg->map_ptr; |
| 8364 | int err; | 8077 | int err; |
| @@ -8370,17 +8083,18 @@ static int check_reg_const_str(struct bpf_verifier_env *env, · 5 unmodified lines | |||
| 8370 | return -EINVAL; | 8083 | return -EINVAL; |
| 8371 | 8084 | ||
| 8372 | if (map->map_type == BPF_MAP_TYPE_INSN_ARRAY) { | 8085 | if (map->map_type == BPF_MAP_TYPE_INSN_ARRAY) { |
| 8373 | verbose(env, "R%d points to insn_array map which cannot be used as const string\n", regno); | 8086 | verbose(env, "%s points to insn_array map which cannot be used as const string\n", |
| 8087 | reg_arg_name(env, argno)); | ||
| 8374 | return -EACCES; | 8088 | return -EACCES; |
| 8375 | } | 8089 | } |
| 8376 | 8090 | ||
| 8377 | if (!bpf_map_is_rdonly(map)) { | 8091 | if (!bpf_map_is_rdonly(map)) { |
| 8378 | verbose(env, "R%d does not point to a readonly map'\n", regno); | 8092 | verbose(env, "%s does not point to a readonly map'\n", reg_arg_name(env, argno)); |
| 8379 | return -EACCES; | 8093 | return -EACCES; |
| 8380 | } | 8094 | } |
| 8381 | 8095 | ||
| 8382 | if (!tnum_is_const(reg->var_off)) { | 8096 | if (!tnum_is_const(reg->var_off)) { |
| 8383 | verbose(env, "R%d is not a constant address'\n", regno); | 8097 | verbose(env, "%s is not a constant address'\n", reg_arg_name(env, argno)); |
| 8384 | return -EACCES; | 8098 | return -EACCES; |
| 8385 | } | 8099 | } |
| 8386 | 8100 | ||
| @@ -8389,7 +8103,7 @@ static int check_reg_const_str(struct bpf_verifier_env *env, · 2 unmodified lines | |||
| 8389 | return -EACCES; | 8103 | return -EACCES; |
| 8390 | } | 8104 | } |
| 8391 | 8105 | ||
| 8392 | err = check_map_access(env, regno, 0, | 8106 | err = check_map_access(env, reg, argno, 0, |
| 8393 | map->value_size - reg->var_off.value, false, | 8107 | map->value_size - reg->var_off.value, false, |
| 8394 | ACCESS_HELPER); | 8108 | ACCESS_HELPER); |
| 8395 | if (err) | 8109 | if (err) |
| @@ -8471,7 +8185,7 @@ static int get_constant_map_key(struct bpf_verifier_env *env, · 75 unmodified lines | |||
| 8471 | return 0; | 8185 | return 0; |
| 8472 | } | 8186 | } |
| 8473 | 8187 | ||
| 8474 | static bool can_elide_value_nullness(enum bpf_map_type type); | 8188 | static bool can_elide_value_nullness(const struct bpf_map *map); |
| 8475 | 8189 | ||
| 8476 | static int check_func_arg(struct bpf_verifier_env *env, u32 arg, | 8190 | static int check_func_arg(struct bpf_verifier_env *env, u32 arg, |
| 8477 | struct bpf_call_arg_meta *meta, | 8191 | struct bpf_call_arg_meta *meta, |
| @@ -8481,6 +8195,7 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, · 3 unmodified lines | |||
| 8481 | u32 regno = BPF_REG_1 + arg; | 8195 | u32 regno = BPF_REG_1 + arg; |
| 8482 | struct bpf_reg_state *reg = reg_state(env, regno); | 8196 | struct bpf_reg_state *reg = reg_state(env, regno); |
| 8483 | enum bpf_arg_type arg_type = fn->arg_type[arg]; | 8197 | enum bpf_arg_type arg_type = fn->arg_type[arg]; |
| 8198 | argno_t argno = argno_from_arg(arg + 1); | ||
| 8484 | enum bpf_reg_type type = reg->type; | 8199 | enum bpf_reg_type type = reg->type; |
| 8485 | u32 *arg_btf_id = NULL; | 8200 | u32 *arg_btf_id = NULL; |
| 8486 | u32 key_size; | 8201 | u32 key_size; |
| @@ -8525,56 +8240,24 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, · 38 unmodified lines | |||
| 8525 | base_type(arg_type) == ARG_PTR_TO_SPIN_LOCK) | 8240 | base_type(arg_type) == ARG_PTR_TO_SPIN_LOCK) |
| 8526 | arg_btf_id = fn->arg_btf_id[arg]; | 8241 | arg_btf_id = fn->arg_btf_id[arg]; |
| 8527 | 8242 | ||
| 8528 | err = check_reg_type(env, regno, arg_type, arg_btf_id, meta); | 8243 | err = check_reg_type(env, reg, argno_from_reg(regno), arg_type, arg_btf_id, meta); |
| 8529 | if (err) | 8244 | if (err) |
| 8530 | return err; | 8245 | return err; |
| 8531 | 8246 | ||
| 8532 | err = check_func_arg_reg_off(env, reg, regno, arg_type); | 8247 | err = check_func_arg_reg_off(env, reg, argno_from_reg(regno), arg_type); |
| 8533 | if (err) | 8248 | if (err) |
| 8534 | return err; | 8249 | return err; |
| 8535 | 8250 | ||
| 8536 | skip_type_check: | 8251 | skip_type_check: |
| 8537 | if (arg_type_is_release(arg_type)) { | 8252 | if (arg_type_is_release(arg_type) && !arg_type_is_dynptr(arg_type) && |
| 8538 | if (arg_type_is_dynptr(arg_type)) { | 8253 | !reg_is_referenced(env, reg) && !bpf_register_is_null(reg)) { |
| 8539 | struct bpf_func_state *state = bpf_func(env, reg); | 8254 | verbose(env, "release helper %s expects referenced PTR_TO_BTF_ID passed to %s\n", |
| 8540 | int spi; | 8255 | func_id_name(meta->func_id), reg_arg_name(env, argno)); |
| 8541 | 8256 | return -EINVAL; | |
| 8542 | /* Only dynptr created on stack can be released, thus | ||
| 8543 | * the get_spi and stack state checks for spilled_ptr | ||
| 8544 | * should only be done before process_dynptr_func for | ||
| 8545 | * PTR_TO_STACK. | ||
| 8546 | */ | ||
| 8547 | if (reg->type == PTR_TO_STACK) { | ||
| 8548 | spi = dynptr_get_spi(env, reg); | ||
| 8549 | if (spi < 0 || !state->stack[spi].spilled_ptr.ref_obj_id) { | ||
| 8550 | verbose(env, "arg %d is an unacquired reference\n", regno); | ||
| 8551 | return -EINVAL; | ||
| 8552 | } | ||
| 8553 | } else { | ||
| 8554 | verbose(env, "cannot release unowned const bpf_dynptr\n"); | ||
| 8555 | return -EINVAL; | ||
| 8556 | } | ||
| 8557 | } else if (!reg->ref_obj_id && !bpf_register_is_null(reg)) { | ||
| 8558 | verbose(env, "R%d must be referenced when passed to release function\n", | ||
| 8559 | regno); | ||
| 8560 | return -EINVAL; | ||
| 8561 | } | ||
| 8562 | if (meta->release_regno) { | ||
| 8563 | verifier_bug(env, "more than one release argument"); | ||
| 8564 | return -EFAULT; | ||
| 8565 | } | ||
| 8566 | meta->release_regno = regno; | ||
| 8567 | } | 8257 | } |
| 8568 | 8258 | ||
| 8569 | if (reg->ref_obj_id && base_type(arg_type) != ARG_KPTR_XCHG_DEST) { | 8259 | if (reg_is_referenced(env, reg)) |
| 8570 | if (meta->ref_obj_id) { | 8260 | update_ref_obj(&meta->ref_obj, reg); |
| 8571 | verbose(env, "more than one arg with ref_obj_id R%d %u %u", | ||
| 8572 | regno, reg->ref_obj_id, | ||
| 8573 | meta->ref_obj_id); | ||
| 8574 | return -EACCES; | ||
| 8575 | } | ||
| 8576 | meta->ref_obj_id = reg->ref_obj_id; | ||
| 8577 | } | ||
| 8578 | 8261 | ||
| 8579 | switch (base_type(arg_type)) { | 8262 | switch (base_type(arg_type)) { |
| 8580 | case ARG_CONST_MAP_PTR: | 8263 | case ARG_CONST_MAP_PTR: |
| @@ -8618,10 +8301,10 @@ skip_type_check: · 37 unmodified lines | |||
| 8618 | return -EFAULT; | 8301 | return -EFAULT; |
| 8619 | } | 8302 | } |
| 8620 | key_size = meta->map.ptr->key_size; | 8303 | key_size = meta->map.ptr->key_size; |
| 8621 | err = check_helper_mem_access(env, regno, key_size, BPF_READ, false, NULL); | 8304 | err = check_helper_mem_access(env, reg, argno_from_reg(regno), key_size, BPF_READ, false, NULL); |
| 8622 | if (err) | 8305 | if (err) |
| 8623 | return err; | 8306 | return err; |
| 8624 | if (can_elide_value_nullness(meta->map.ptr->map_type)) { | 8307 | if (can_elide_value_nullness(meta->map.ptr)) { |
| 8625 | err = get_constant_map_key(env, reg, key_size, &meta->const_map_key); | 8308 | err = get_constant_map_key(env, reg, key_size, &meta->const_map_key); |
| 8626 | if (err < 0) { | 8309 | if (err < 0) { |
| 8627 | meta->const_map_key = -1; | 8310 | meta->const_map_key = -1; |
| @@ -8645,7 +8328,7 @@ skip_type_check: · 17 unmodified lines | |||
| 8645 | return -EFAULT; | 8328 | return -EFAULT; |
| 8646 | } | 8329 | } |
| 8647 | meta->raw_mode = arg_type & MEM_UNINIT; | 8330 | meta->raw_mode = arg_type & MEM_UNINIT; |
| 8648 | err = check_helper_mem_access(env, regno, meta->map.ptr->value_size, | 8331 | err = check_helper_mem_access(env, reg, argno_from_reg(regno), meta->map.ptr->value_size, |
| 8649 | arg_type & MEM_WRITE ? BPF_WRITE : BPF_READ, | 8332 | arg_type & MEM_WRITE ? BPF_WRITE : BPF_READ, |
| 8650 | false, meta); | 8333 | false, meta); |
| 8651 | break; | 8334 | break; |
| @@ -8663,11 +8346,11 @@ skip_type_check: · 11 unmodified lines | |||
| 8663 | return -EACCES; | 8346 | return -EACCES; |
| 8664 | } | 8347 | } |
| 8665 | if (meta->func_id == BPF_FUNC_spin_lock) { | 8348 | if (meta->func_id == BPF_FUNC_spin_lock) { |
| 8666 | err = process_spin_lock(env, regno, PROCESS_SPIN_LOCK); | 8349 | err = process_spin_lock(env, reg, argno_from_reg(regno), PROCESS_SPIN_LOCK); |
| 8667 | if (err) | 8350 | if (err) |
| 8668 | return err; | 8351 | return err; |
| 8669 | } else if (meta->func_id == BPF_FUNC_spin_unlock) { | 8352 | } else if (meta->func_id == BPF_FUNC_spin_unlock) { |
| 8670 | err = process_spin_lock(env, regno, 0); | 8353 | err = process_spin_lock(env, reg, argno_from_reg(regno), 0); |
| 8671 | if (err) | 8354 | if (err) |
| 8672 | return err; | 8355 | return err; |
| 8673 | } else { | 8356 | } else { |
| @@ -8676,7 +8359,7 @@ skip_type_check: · 2 unmodified lines | |||
| 8676 | } | 8359 | } |
| 8677 | break; | 8360 | break; |
| 8678 | case ARG_PTR_TO_TIMER: | 8361 | case ARG_PTR_TO_TIMER: |
| 8679 | err = process_timer_helper(env, regno, meta); | 8362 | err = process_timer_helper(env, reg, argno_from_reg(regno), meta); |
| 8680 | if (err) | 8363 | if (err) |
| 8681 | return err; | 8364 | return err; |
| 8682 | break; | 8365 | break; |
| @@ -8689,7 +8372,7 @@ skip_type_check: · 6 unmodified lines | |||
| 8689 | */ | 8372 | */ |
| 8690 | meta->raw_mode = arg_type & MEM_UNINIT; | 8373 | meta->raw_mode = arg_type & MEM_UNINIT; |
| 8691 | if (arg_type & MEM_FIXED_SIZE) { | 8374 | if (arg_type & MEM_FIXED_SIZE) { |
| 8692 | err = check_helper_mem_access(env, regno, fn->arg_size[arg], | 8375 | err = check_helper_mem_access(env, reg, argno_from_reg(regno), fn->arg_size[arg], |
| 8693 | arg_type & MEM_WRITE ? BPF_WRITE : BPF_READ, | 8376 | arg_type & MEM_WRITE ? BPF_WRITE : BPF_READ, |
| 8694 | false, meta); | 8377 | false, meta); |
| 8695 | if (err) | 8378 | if (err) |
| @@ -8699,19 +8382,22 @@ skip_type_check: · 3 unmodified lines | |||
| 8699 | } | 8382 | } |
| 8700 | break; | 8383 | break; |
| 8701 | case ARG_CONST_SIZE: | 8384 | case ARG_CONST_SIZE: |
| 8702 | err = check_mem_size_reg(env, reg, regno, | 8385 | err = check_mem_size_reg(env, reg_state(env, regno - 1), reg, argno_from_reg(regno - 1), |
| 8386 | argno_from_reg(regno), | ||
| 8703 | fn->arg_type[arg - 1] & MEM_WRITE ? | 8387 | fn->arg_type[arg - 1] & MEM_WRITE ? |
| 8704 | BPF_WRITE : BPF_READ, | 8388 | BPF_WRITE : BPF_READ, |
| 8705 | false, meta); | 8389 | false, meta); |
| 8706 | break; | 8390 | break; |
| 8707 | case ARG_CONST_SIZE_OR_ZERO: | 8391 | case ARG_CONST_SIZE_OR_ZERO: |
| 8708 | err = check_mem_size_reg(env, reg, regno, | 8392 | err = check_mem_size_reg(env, reg_state(env, regno - 1), reg, argno_from_reg(regno - 1), |
| 8393 | argno_from_reg(regno), | ||
| 8709 | fn->arg_type[arg - 1] & MEM_WRITE ? | 8394 | fn->arg_type[arg - 1] & MEM_WRITE ? |
| 8710 | BPF_WRITE : BPF_READ, | 8395 | BPF_WRITE : BPF_READ, |
| 8711 | true, meta); | 8396 | true, meta); |
| 8712 | break; | 8397 | break; |
| 8713 | case ARG_PTR_TO_DYNPTR: | 8398 | case ARG_PTR_TO_DYNPTR: |
| 8714 | err = process_dynptr_func(env, regno, insn_idx, arg_type, 0); | 8399 | err = process_dynptr_func(env, reg, argno_from_reg(regno), insn_idx, arg_type, &meta->ref_obj, |
| 8400 | &meta->dynptr); | ||
| 8715 | if (err) | 8401 | if (err) |
| 8716 | return err; | 8402 | return err; |
| 8717 | break; | 8403 | break; |
| @@ -8728,7 +8414,7 @@ skip_type_check: · 10 unmodified lines | |||
| 8728 | break; | 8414 | break; |
| 8729 | case ARG_PTR_TO_CONST_STR: | 8415 | case ARG_PTR_TO_CONST_STR: |
| 8730 | { | 8416 | { |
| 8731 | err = check_reg_const_str(env, reg, regno); | 8417 | err = check_arg_const_str(env, reg, argno_from_reg(regno)); |
| 8732 | if (err) | 8418 | if (err) |
| 8733 | return err; | 8419 | return err; |
| 8734 | break; | 8420 | break; |
| @@ -9130,11 +8816,29 @@ static bool check_mem_arg_rw_flag_ok(const struct bpf_func_proto *fn) · 395 unmodified lines | |||
| 9130 | return true; | 8816 | return true; |
| 9131 | } | 8817 | } |
| 9132 | 8818 | ||
| 9133 | static int check_func_proto(const struct bpf_func_proto *fn) | 8819 | static bool check_proto_release_reg(const struct bpf_func_proto *fn, struct bpf_call_arg_meta *meta) |
| 8820 | { | ||
| 8821 | int i; | ||
| 8822 | |||
| 8823 | for (i = 0; i < ARRAY_SIZE(fn->arg_type); i++) { | ||
| 8824 | enum bpf_arg_type arg_type = fn->arg_type[i]; | ||
| 8825 | |||
| 8826 | if (arg_type_is_release(arg_type)) { | ||
| 8827 | if (meta->release_regno) | ||
| 8828 | return false; | ||
| 8829 | meta->release_regno = i + 1; | ||
| 8830 | } | ||
| 8831 | } | ||
| 8832 | |||
| 8833 | return true; | ||
| 8834 | } | ||
| 8835 | |||
| 8836 | static int check_func_proto(const struct bpf_func_proto *fn, struct bpf_call_arg_meta *meta) | ||
| 9134 | { | 8837 | { |
| 9135 | return check_raw_mode_ok(fn) && | 8838 | return check_raw_mode_ok(fn) && |
| 9136 | check_arg_pair_ok(fn) && | 8839 | check_arg_pair_ok(fn) && |
| 9137 | check_mem_arg_rw_flag_ok(fn) && | 8840 | check_mem_arg_rw_flag_ok(fn) && |
| 8841 | check_proto_release_reg(fn, meta) && | ||
| 9138 | check_btf_id_ok(fn) ? 0 : -EINVAL; | 8842 | check_btf_id_ok(fn) ? 0 : -EINVAL; |
| 9139 | } | 8843 | } |
| 9140 | 8844 | ||
| @@ -9181,14 +8885,14 @@ static void mark_pkt_end(struct bpf_verifier_state *vstate, int regn, bool range · 40 unmodified lines | |||
| 9181 | reg->range = AT_PKT_END; | 8885 | reg->range = AT_PKT_END; |
| 9182 | } | 8886 | } |
| 9183 | 8887 | ||
| 9184 | static int release_reference_nomark(struct bpf_verifier_state *state, int ref_obj_id) | 8888 | static int release_reference_nomark(struct bpf_verifier_state *state, int id) |
| 9185 | { | 8889 | { |
| 9186 | int i; | 8890 | int i; |
| 9187 | 8891 | ||
| 9188 | for (i = 0; i < state->acquired_refs; i++) { | 8892 | for (i = 0; i < state->acquired_refs; i++) { |
| 9189 | if (state->refs[i].type != REF_TYPE_PTR) | 8893 | if (state->refs[i].type != REF_TYPE_PTR) |
| 9190 | continue; | 8894 | continue; |
| 9191 | if (state->refs[i].id == ref_obj_id) { | 8895 | if (state->refs[i].id == id) { |
| 9192 | release_reference_state(state, i); | 8896 | release_reference_state(state, i); |
| 9193 | return 0; | 8897 | return 0; |
| 9194 | } | 8898 | } |
| @@ -9196,26 +8900,83 @@ static int release_reference_nomark(struct bpf_verifier_state *state, int ref_ob · 1 unmodified line | |||
| 9196 | return -EINVAL; | 8900 | return -EINVAL; |
| 9197 | } | 8901 | } |
| 9198 | 8902 | ||
| 9199 | /* The pointer with the specified id has released its reference to kernel | 8903 | static int idstack_push(struct bpf_idmap *idmap, u32 id) |
| 9200 | * resources. Identify all copies of the same pointer and clear the reference. | 8904 | { |
| 9201 | * | 8905 | int i; |
| 9202 | * This is the release function corresponding to acquire_reference(). Idempotent. | 8906 | |
| 9203 | */ | 8907 | if (!id) |
| 9204 | static int release_reference(struct bpf_verifier_env *env, int ref_obj_id) | 8908 | return 0; |
| 8909 | |||
| 8910 | for (i = 0; i < idmap->cnt; i++) | ||
| 8911 | if (idmap->map[i].old == id) | ||
| 8912 | return 0; | ||
| 8913 | |||
| 8914 | if (WARN_ON_ONCE(idmap->cnt >= BPF_ID_MAP_SIZE)) | ||
| 8915 | return -EFAULT; | ||
| 8916 | |||
| 8917 | idmap->map[idmap->cnt++].old = id; | ||
| 8918 | return 0; | ||
| 8919 | } | ||
| 8920 | |||
| 8921 | static int idstack_pop(struct bpf_idmap *idmap) | ||
| 8922 | { | ||
| 8923 | if (!idmap->cnt) | ||
| 8924 | return 0; | ||
| 8925 | |||
| 8926 | return idmap->map[--idmap->cnt].old; | ||
| 8927 | } | ||
| 8928 | |||
| 8929 | /* Release id and objects derived from it iteratively in a DFS manner */ | ||
| 8930 | static int release_reference(struct bpf_verifier_env *env, int id) | ||
| 9205 | { | 8931 | { |
| 8932 | u32 mask = (1 << STACK_SPILL) | (1 << STACK_DYNPTR); | ||
| 9206 | struct bpf_verifier_state *vstate = env->cur_state; | 8933 | struct bpf_verifier_state *vstate = env->cur_state; |
| 8934 | struct bpf_idmap *idstack = &env->idmap_scratch; | ||
| 8935 | struct bpf_stack_state *stack; | ||
| 9207 | struct bpf_func_state *state; | 8936 | struct bpf_func_state *state; |
| 9208 | struct bpf_reg_state *reg; | 8937 | struct bpf_reg_state *reg; |
| 9209 | int err; | 8938 | int i, err; |
| 9210 | 8939 | ||
| 9211 | err = release_reference_nomark(vstate, ref_obj_id); | 8940 | idstack->cnt = 0; |
| 8941 | err = idstack_push(idstack, id); | ||
| 9212 | if (err) | 8942 | if (err) |
| 9213 | return err; | 8943 | return err; |
| 9214 | 8944 | ||
| 9215 | bpf_for_each_reg_in_vstate(vstate, state, reg, ({ | 8945 | if (find_reference_state(vstate, id)) |
| 9216 | if (reg->ref_obj_id == ref_obj_id) | 8946 | WARN_ON_ONCE(release_reference_nomark(vstate, id)); |
| 9217 | mark_reg_invalid(env, reg); | 8947 | |
| 9218 | })); | 8948 | while ((id = idstack_pop(idstack))) { |
| 8949 | /* | ||
| 8950 | * Child references are inaccessible after parent is released, | ||
| 8951 | * any child references that exist at this point are a leak. | ||
| 8952 | */ | ||
| 8953 | for (i = 0; i < vstate->acquired_refs; i++) { | ||
| 8954 | if (vstate->refs[i].type != REF_TYPE_PTR) | ||
| 8955 | continue; | ||
| 8956 | if (vstate->refs[i].parent_id != id) | ||
| 8957 | continue; | ||
| 8958 | verbose(env, "Leaking reference id=%d alloc_insn=%d. Release it first.\n", | ||
| 8959 | vstate->refs[i].id, vstate->refs[i].insn_idx); | ||
| 8960 | return -EINVAL; | ||
| 8961 | } | ||
| 8962 | |||
| 8963 | bpf_for_each_reg_in_vstate_mask(vstate, state, reg, stack, mask, ({ | ||
| 8964 | if (reg->id != id && reg->parent_id != id) | ||
| 8965 | continue; | ||
| 8966 | |||
| 8967 | /* Free objects derived from the current object */ | ||
| 8968 | if (reg->parent_id == id) { | ||
| 8969 | err = idstack_push(idstack, reg->id); | ||
| 8970 | if (err) | ||
| 8971 | return err; | ||
| 8972 | } | ||
| 8973 | |||
| 8974 | if (!stack || stack->slot_type[BPF_REG_SIZE - 1] == STACK_SPILL) | ||
| 8975 | mark_reg_invalid(env, reg); | ||
| 8976 | else if (stack->slot_type[BPF_REG_SIZE - 1] == STACK_DYNPTR) | ||
| 8977 | invalidate_dynptr(env, stack); | ||
| 8978 | })); | ||
| 8979 | } | ||
| 9219 | 8980 | ||
| 9220 | return 0; | 8981 | return 0; |
| 9221 | } | 8982 | } |
| @@ -9231,6 +8992,42 @@ static void invalidate_non_owning_refs(struct bpf_verifier_env *env) · 9 unmodified lines | |||
| 9231 | })); | 8992 | })); |
| 9232 | } | 8993 | } |
| 9233 | 8994 | ||
| 8995 | static void invalidate_rcu_protected_refs(struct bpf_verifier_env *env) | ||
| 8996 | { | ||
| 8997 | struct bpf_stack_state *stack; | ||
| 8998 | struct bpf_func_state *state; | ||
| 8999 | struct bpf_reg_state *reg; | ||
| 9000 | u32 clear_mask = (1 << STACK_SPILL) | (1 << STACK_ITER); | ||
| 9001 | |||
| 9002 | bpf_for_each_reg_in_vstate_mask(env->cur_state, state, reg, stack, clear_mask, ({ | ||
| 9003 | if (reg->type & MEM_RCU) { | ||
| 9004 | reg->type &= ~(MEM_RCU | PTR_MAYBE_NULL); | ||
| 9005 | reg->type |= PTR_UNTRUSTED; | ||
| 9006 | } | ||
| 9007 | })); | ||
| 9008 | } | ||
| 9009 | |||
| 9010 | static int ref_convert_alloc_rcu_protected(struct bpf_verifier_env *env, u32 id) | ||
| 9011 | { | ||
| 9012 | struct bpf_func_state *state; | ||
| 9013 | struct bpf_reg_state *reg; | ||
| 9014 | int err; | ||
| 9015 | |||
| 9016 | err = release_reference_nomark(env->cur_state, id); | ||
| 9017 | |||
| 9018 | bpf_for_each_reg_in_vstate(env->cur_state, state, reg, ({ | ||
| 9019 | if (reg->id != id) | ||
| 9020 | continue; | ||
| 9021 | if ((reg->type & MEM_ALLOC) && (reg->type & MEM_PERCPU)) { | ||
| 9022 | reg->id = 0; | ||
| 9023 | reg->type &= ~MEM_ALLOC; | ||
| 9024 | reg->type |= MEM_RCU; | ||
| 9025 | } | ||
| 9026 | })); | ||
| 9027 | |||
| 9028 | return err; | ||
| 9029 | } | ||
| 9030 | |||
| 9234 | static void clear_caller_saved_regs(struct bpf_verifier_env *env, | 9031 | static void clear_caller_saved_regs(struct bpf_verifier_env *env, |
| 9235 | struct bpf_reg_state *regs) | 9032 | struct bpf_reg_state *regs) |
| 9236 | { | 9033 | { |
| @@ -9243,6 +9040,15 @@ static void clear_caller_saved_regs(struct bpf_verifier_env *env, · 6 unmodified lines | |||
| 9243 | } | 9040 | } |
| 9244 | } | 9041 | } |
| 9245 | 9042 | ||
| 9043 | static void invalidate_outgoing_stack_args(const struct bpf_verifier_env *env, | ||
| 9044 | struct bpf_func_state *state) | ||
| 9045 | { | ||
| 9046 | int i, nslots = state->out_stack_arg_cnt; | ||
| 9047 | |||
| 9048 | for (i = 0; i < nslots; i++) | ||
| 9049 | bpf_mark_reg_not_init(env, &state->stack_arg_regs[i]); | ||
| 9050 | } | ||
| 9051 | |||
| 9246 | typedef int (*set_callee_state_fn)(struct bpf_verifier_env *env, | 9052 | typedef int (*set_callee_state_fn)(struct bpf_verifier_env *env, |
| 9247 | struct bpf_func_state *caller, | 9053 | struct bpf_func_state *caller, |
| 9248 | struct bpf_func_state *callee, | 9054 | struct bpf_func_state *callee, |
| @@ -9305,11 +9111,23 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog, · 56 unmodified lines | |||
| 9305 | struct bpf_reg_state *regs) | 9111 | struct bpf_reg_state *regs) |
| 9306 | { | 9112 | { |
| 9307 | struct bpf_subprog_info *sub = subprog_info(env, subprog); | 9113 | struct bpf_subprog_info *sub = subprog_info(env, subprog); |
| 9114 | struct bpf_func_state *caller = cur_func(env); | ||
| 9308 | struct bpf_verifier_log *log = &env->log; | 9115 | struct bpf_verifier_log *log = &env->log; |
| 9116 | struct ref_obj_desc ref_obj = {}; | ||
| 9309 | u32 i; | 9117 | u32 i; |
| 9310 | int ret; | 9118 | int ret, err; |
| 9311 | 9119 | ||
| 9312 | ret = btf_prepare_func_args(env, subprog); | 9120 | ret = btf_prepare_func_args(env, subprog); |
| 9121 | if (ret) { | ||
| 9122 | if (bpf_in_stack_arg_cnt(sub) > 0) { | ||
| 9123 | err = check_outgoing_stack_args(env, caller, sub->arg_cnt); | ||
| 9124 | if (err) | ||
| 9125 | return err; | ||
| 9126 | } | ||
| 9127 | return ret; | ||
| 9128 | } | ||
| 9129 | |||
| 9130 | ret = check_outgoing_stack_args(env, caller, sub->arg_cnt); | ||
| 9313 | if (ret) | 9131 | if (ret) |
| 9314 | return ret; | 9132 | return ret; |
| 9315 | 9133 | ||
| @@ -9317,13 +9135,13 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog, · 1 unmodified line | |||
| 9317 | * verifier sees. | 9135 | * verifier sees. |
| 9318 | */ | 9136 | */ |
| 9319 | for (i = 0; i < sub->arg_cnt; i++) { | 9137 | for (i = 0; i < sub->arg_cnt; i++) { |
| 9320 | u32 regno = i + 1; | 9138 | argno_t argno = argno_from_arg(i + 1); |
| 9321 | struct bpf_reg_state *reg = ®s[regno]; | 9139 | struct bpf_reg_state *reg = get_func_arg_reg(caller, regs, i); |
| 9322 | struct bpf_subprog_arg_info *arg = &sub->args[i]; | 9140 | struct bpf_subprog_arg_info *arg = &sub->args[i]; |
| 9323 | 9141 | ||
| 9324 | if (arg->arg_type == ARG_ANYTHING) { | 9142 | if (arg->arg_type == ARG_ANYTHING) { |
| 9325 | if (reg->type != SCALAR_VALUE) { | 9143 | if (reg->type != SCALAR_VALUE) { |
| 9326 | bpf_log(log, "R%d is not a scalar\n", regno); | 9144 | bpf_log(log, "%s is not a scalar\n", reg_arg_name(env, argno)); |
| 9327 | return -EINVAL; | 9145 | return -EINVAL; |
| 9328 | } | 9146 | } |
| 9329 | } else if (arg->arg_type & PTR_UNTRUSTED) { | 9147 | } else if (arg->arg_type & PTR_UNTRUSTED) { |
| @@ -9333,24 +9151,26 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog, · 3 unmodified lines | |||
| 9333 | * invalid memory access. | 9151 | * invalid memory access. |
| 9334 | */ | 9152 | */ |
| 9335 | } else if (arg->arg_type == ARG_PTR_TO_CTX) { | 9153 | } else if (arg->arg_type == ARG_PTR_TO_CTX) { |
| 9336 | ret = check_func_arg_reg_off(env, reg, regno, ARG_PTR_TO_CTX); | 9154 | ret = check_func_arg_reg_off(env, reg, argno, ARG_PTR_TO_CTX); |
| 9337 | if (ret < 0) | 9155 | if (ret < 0) |
| 9338 | return ret; | 9156 | return ret; |
| 9339 | /* If function expects ctx type in BTF check that caller | 9157 | /* If function expects ctx type in BTF check that caller |
| 9340 | * is passing PTR_TO_CTX. | 9158 | * is passing PTR_TO_CTX. |
| 9341 | */ | 9159 | */ |
| 9342 | if (reg->type != PTR_TO_CTX) { | 9160 | if (reg->type != PTR_TO_CTX) { |
| 9343 | bpf_log(log, "arg#%d expects pointer to ctx\n", i); | 9161 | bpf_log(log, "%s expects pointer to ctx\n", |
| 9162 | reg_arg_name(env, argno)); | ||
| 9344 | return -EINVAL; | 9163 | return -EINVAL; |
| 9345 | } | 9164 | } |
| 9346 | } else if (base_type(arg->arg_type) == ARG_PTR_TO_MEM) { | 9165 | } else if (base_type(arg->arg_type) == ARG_PTR_TO_MEM) { |
| 9347 | ret = check_func_arg_reg_off(env, reg, regno, ARG_DONTCARE); | 9166 | ret = check_func_arg_reg_off(env, reg, argno, ARG_DONTCARE); |
| 9348 | if (ret < 0) | 9167 | if (ret < 0) |
| 9349 | return ret; | 9168 | return ret; |
| 9350 | if (check_mem_reg(env, reg, regno, arg->mem_size)) | 9169 | if (check_mem_reg(env, reg, argno, arg->mem_size)) |
| 9351 | return -EINVAL; | 9170 | return -EINVAL; |
| 9352 | if (!(arg->arg_type & PTR_MAYBE_NULL) && (reg->type & PTR_MAYBE_NULL)) { | 9171 | if (!(arg->arg_type & PTR_MAYBE_NULL) && (reg->type & PTR_MAYBE_NULL)) { |
| 9353 | bpf_log(log, "arg#%d is expected to be non-NULL\n", i); | 9172 | bpf_log(log, "%s is expected to be non-NULL\n", |
| 9173 | reg_arg_name(env, argno)); | ||
| 9354 | return -EINVAL; | 9174 | return -EINVAL; |
| 9355 | } | 9175 | } |
| 9356 | } else if (base_type(arg->arg_type) == ARG_PTR_TO_ARENA) { | 9176 | } else if (base_type(arg->arg_type) == ARG_PTR_TO_ARENA) { |
| @@ -9362,15 +9182,16 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog, · 5 unmodified lines | |||
| 9362 | * run-time debug nightmare. | 9182 | * run-time debug nightmare. |
| 9363 | */ | 9183 | */ |
| 9364 | if (reg->type != PTR_TO_ARENA && reg->type != SCALAR_VALUE) { | 9184 | if (reg->type != PTR_TO_ARENA && reg->type != SCALAR_VALUE) { |
| 9365 | bpf_log(log, "R%d is not a pointer to arena or scalar.\n", regno); | 9185 | bpf_log(log, "%s is not a pointer to arena or scalar.\n", |
| 9186 | reg_arg_name(env, argno)); | ||
| 9366 | return -EINVAL; | 9187 | return -EINVAL; |
| 9367 | } | 9188 | } |
| 9368 | } else if (arg->arg_type == (ARG_PTR_TO_DYNPTR | MEM_RDONLY)) { | 9189 | } else if (arg->arg_type == ARG_PTR_TO_DYNPTR) { |
| 9369 | ret = check_func_arg_reg_off(env, reg, regno, ARG_PTR_TO_DYNPTR); | 9190 | ret = check_func_arg_reg_off(env, reg, argno, ARG_PTR_TO_DYNPTR); |
| 9370 | if (ret) | 9191 | if (ret) |
| 9371 | return ret; | 9192 | return ret; |
| 9372 | 9193 | ||
| 9373 | ret = process_dynptr_func(env, regno, -1, arg->arg_type, 0); | 9194 | ret = process_dynptr_func(env, reg, argno, -1, arg->arg_type, &ref_obj, NULL); |
| 9374 | if (ret) | 9195 | if (ret) |
| 9375 | return ret; | 9196 | return ret; |
| 9376 | } else if (base_type(arg->arg_type) == ARG_PTR_TO_BTF_ID) { | 9197 | } else if (base_type(arg->arg_type) == ARG_PTR_TO_BTF_ID) { |
| @@ -9381,12 +9202,13 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog, · 4 unmodified lines | |||
| 9381 | continue; | 9202 | continue; |
| 9382 | 9203 | ||
| 9383 | memset(&meta, 0, sizeof(meta)); /* leave func_id as zero */ | 9204 | memset(&meta, 0, sizeof(meta)); /* leave func_id as zero */ |
| 9384 | err = check_reg_type(env, regno, arg->arg_type, &arg->btf_id, &meta); | 9205 | err = check_reg_type(env, reg, argno, arg->arg_type, &arg->btf_id, &meta); |
| 9385 | err = err ?: check_func_arg_reg_off(env, reg, regno, arg->arg_type); | 9206 | err = err ?: check_func_arg_reg_off(env, reg, argno, arg->arg_type); |
| 9386 | if (err) | 9207 | if (err) |
| 9387 | return err; | 9208 | return err; |
| 9388 | } else { | 9209 | } else { |
| 9389 | verifier_bug(env, "unrecognized arg#%d type %d", i, arg->arg_type); | 9210 | verifier_bug(env, "unrecognized %s type %d", |
| 9211 | reg_arg_name(env, argno), arg->arg_type); | ||
| 9390 | return -EFAULT; | 9212 | return -EFAULT; |
| 9391 | } | 9213 | } |
| 9392 | } | 9214 | } |
| @@ -9505,6 +9327,8 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn, · 112 unmodified lines | |||
| 9505 | int *insn_idx) | 9327 | int *insn_idx) |
| 9506 | { | 9328 | { |
| 9507 | struct bpf_verifier_state *state = env->cur_state; | 9329 | struct bpf_verifier_state *state = env->cur_state; |
| 9330 | struct bpf_subprog_info *caller_info; | ||
| 9331 | u16 callee_incoming, stack_arg_cnt; | ||
| 9508 | struct bpf_func_state *caller; | 9332 | struct bpf_func_state *caller; |
| 9509 | int err, subprog, target_insn; | 9333 | int err, subprog, target_insn; |
| 9510 | 9334 | ||
| @@ -9547,6 +9371,7 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn, · 36 unmodified lines | |||
| 9547 | /* mark global subprog for verifying after main prog */ | 9371 | /* mark global subprog for verifying after main prog */ |
| 9548 | subprog_aux(env, subprog)->called = true; | 9372 | subprog_aux(env, subprog)->called = true; |
| 9549 | clear_caller_saved_regs(env, caller->regs); | 9373 | clear_caller_saved_regs(env, caller->regs); |
| 9374 | invalidate_outgoing_stack_args(env, cur_func(env)); | ||
| 9550 | 9375 | ||
| 9551 | /* All non-void global functions return a 64-bit SCALAR_VALUE. */ | 9376 | /* All non-void global functions return a 64-bit SCALAR_VALUE. */ |
| 9552 | if (!subprog_returns_void(env, subprog)) { | 9377 | if (!subprog_returns_void(env, subprog)) { |
| @@ -9569,6 +9394,16 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn, · 16 unmodified lines | |||
| 9569 | return 0; | 9394 | return 0; |
| 9570 | } | 9395 | } |
| 9571 | 9396 | ||
| 9397 | /* | ||
| 9398 | * Track caller's total stack arg count (incoming + max outgoing). | ||
| 9399 | * This is needed so the JIT knows how much stack arg space to allocate. | ||
| 9400 | */ | ||
| 9401 | caller_info = &env->subprog_info[caller->subprogno]; | ||
| 9402 | callee_incoming = bpf_in_stack_arg_cnt(&env->subprog_info[subprog]); | ||
| 9403 | stack_arg_cnt = bpf_in_stack_arg_cnt(caller_info) + callee_incoming; | ||
| 9404 | if (stack_arg_cnt > caller_info->stack_arg_cnt) | ||
| 9405 | caller_info->stack_arg_cnt = stack_arg_cnt; | ||
| 9406 | |||
| 9572 | /* for regular function entry setup new frame and continue | 9407 | /* for regular function entry setup new frame and continue |
| 9573 | * from that frame. | 9408 | * from that frame. |
| 9574 | */ | 9409 | */ |
| @@ -9852,9 +9687,9 @@ static bool in_rbtree_lock_required_cb(struct bpf_verifier_env *env) · 277 unmodified lines | |||
| 9852 | static bool retval_range_within(struct bpf_retval_range range, const struct bpf_reg_state *reg) | 9687 | static bool retval_range_within(struct bpf_retval_range range, const struct bpf_reg_state *reg) |
| 9853 | { | 9688 | { |
| 9854 | if (range.return_32bit) | 9689 | if (range.return_32bit) |
| 9855 | return range.minval <= reg->s32_min_value && reg->s32_max_value <= range.maxval; | 9690 | return range.minval <= reg_s32_min(reg) && reg_s32_max(reg) <= range.maxval; |
| 9856 | else | 9691 | else |
| 9857 | return range.minval <= reg->smin_value && reg->smax_value <= range.maxval; | 9692 | return range.minval <= reg_smin(reg) && reg_smax(reg) <= range.maxval; |
| 9858 | } | 9693 | } |
| 9859 | 9694 | ||
| 9860 | static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx) | 9695 | static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx) |
| @@ -9926,6 +9761,7 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx) · 65 unmodified lines | |||
| 9926 | * bpf_throw, this will be done by copy_verifier_state for extra frames. */ | 9761 | * bpf_throw, this will be done by copy_verifier_state for extra frames. */ |
| 9927 | free_func_state(callee); | 9762 | free_func_state(callee); |
| 9928 | state->frame[state->curframe--] = NULL; | 9763 | state->frame[state->curframe--] = NULL; |
| 9764 | invalidate_outgoing_stack_args(env, caller); | ||
| 9929 | 9765 | ||
| 9930 | /* for callbacks widen imprecise scalars to make programs like below verify: | 9766 | /* for callbacks widen imprecise scalars to make programs like below verify: |
| 9931 | * | 9767 | * |
| @@ -9952,7 +9788,9 @@ static int do_refine_retval_range(struct bpf_verifier_env *env, · 20 unmodified lines | |||
| 9952 | int func_id, | 9788 | int func_id, |
| 9953 | struct bpf_call_arg_meta *meta) | 9789 | struct bpf_call_arg_meta *meta) |
| 9954 | { | 9790 | { |
| 9791 | struct bpf_retval_range range; | ||
| 9955 | struct bpf_reg_state *ret_reg = ®s[BPF_REG_0]; | 9792 | struct bpf_reg_state *ret_reg = ®s[BPF_REG_0]; |
| 9793 | enum bpf_prog_type prog_type = resolve_prog_type(env->prog); | ||
| 9956 | 9794 | ||
| 9957 | if (ret_type != RET_INTEGER) | 9795 | if (ret_type != RET_INTEGER) |
| 9958 | return 0; | 9796 | return 0; |
| @@ -9963,21 +9801,36 @@ static int do_refine_retval_range(struct bpf_verifier_env *env, · 4 unmodified lines | |||
| 9963 | case BPF_FUNC_probe_read_str: | 9801 | case BPF_FUNC_probe_read_str: |
| 9964 | case BPF_FUNC_probe_read_kernel_str: | 9802 | case BPF_FUNC_probe_read_kernel_str: |
| 9965 | case BPF_FUNC_probe_read_user_str: | 9803 | case BPF_FUNC_probe_read_user_str: |
| 9966 | ret_reg->smax_value = meta->msize_max_value; | 9804 | reg_set_srange64(ret_reg, -MAX_ERRNO, meta->msize_max_value); |
| 9967 | ret_reg->s32_max_value = meta->msize_max_value; | 9805 | reg_set_srange32(ret_reg, -MAX_ERRNO, meta->msize_max_value); |
| 9968 | ret_reg->smin_value = -MAX_ERRNO; | ||
| 9969 | ret_reg->s32_min_value = -MAX_ERRNO; | ||
| 9970 | reg_bounds_sync(ret_reg); | 9806 | reg_bounds_sync(ret_reg); |
| 9971 | break; | 9807 | break; |
| 9972 | case BPF_FUNC_get_smp_processor_id: | 9808 | case BPF_FUNC_get_smp_processor_id: |
| 9973 | ret_reg->umax_value = nr_cpu_ids - 1; | 9809 | reg_set_urange64(ret_reg, 0, nr_cpu_ids - 1); |
| 9974 | ret_reg->u32_max_value = nr_cpu_ids - 1; | 9810 | reg_set_urange32(ret_reg, 0, nr_cpu_ids - 1); |
| 9975 | ret_reg->smax_value = nr_cpu_ids - 1; | 9811 | reg_bounds_sync(ret_reg); |
| 9976 | ret_reg->s32_max_value = nr_cpu_ids - 1; | 9812 | break; |
| 9977 | ret_reg->umin_value = 0; | 9813 | case BPF_FUNC_get_retval: |
| 9978 | ret_reg->u32_min_value = 0; | 9814 | /* |
| 9979 | ret_reg->smin_value = 0; | 9815 | * bpf_get_retval may see arbitrary value passed by bpf_prog_run_array_cg for |
| 9980 | ret_reg->s32_min_value = 0; | 9816 | * CGROUP_GETSOCKOPT type. |
| 9817 | */ | ||
| 9818 | if (prog_type == BPF_PROG_TYPE_CGROUP_SOCKOPT && | ||
| 9819 | env->prog->expected_attach_type == BPF_CGROUP_GETSOCKOPT) | ||
| 9820 | break; | ||
| 9821 | |||
| 9822 | if (prog_type == BPF_PROG_TYPE_LSM && | ||
| 9823 | env->prog->expected_attach_type == BPF_LSM_CGROUP) { | ||
| 9824 | if (!env->prog->aux->attach_func_proto->type) | ||
| 9825 | break; | ||
| 9826 | bpf_lsm_get_retval_range(env->prog, &range); | ||
| 9827 | } else { | ||
| 9828 | range.minval = -MAX_ERRNO; | ||
| 9829 | range.maxval = 0; | ||
| 9830 | } | ||
| 9831 | |||
| 9832 | reg_set_srange64(ret_reg, range.minval, range.maxval); | ||
| 9833 | reg_set_srange32(ret_reg, range.minval, range.maxval); | ||
| 9981 | reg_bounds_sync(ret_reg); | 9834 | reg_bounds_sync(ret_reg); |
| 9982 | break; | 9835 | break; |
| 9983 | } | 9836 | } |
| @@ -10086,7 +9939,7 @@ static int check_reference_leak(struct bpf_verifier_env *env, bool exception_exi · 102 unmodified lines | |||
| 10086 | * kernel. Type checks are performed later in check_return_code. | 9939 | * kernel. Type checks are performed later in check_return_code. |
| 10087 | */ | 9940 | */ |
| 10088 | if (type == BPF_PROG_TYPE_STRUCT_OPS && !exception_exit && | 9941 | if (type == BPF_PROG_TYPE_STRUCT_OPS && !exception_exit && |
| 10089 | reg->ref_obj_id == state->refs[i].id) | 9942 | reg->id == state->refs[i].id) |
| 10090 | continue; | 9943 | continue; |
| 10091 | verbose(env, "Unreleased reference id=%d alloc_insn=%d\n", | 9944 | verbose(env, "Unreleased reference id=%d alloc_insn=%d\n", |
| 10092 | state->refs[i].id, state->refs[i].insn_idx); | 9945 | state->refs[i].id, state->refs[i].insn_idx); |
| @@ -10221,13 +10074,16 @@ static void update_loop_inline_state(struct bpf_verifier_env *env, u32 subprogno · 128 unmodified lines | |||
| 10221 | state->callback_subprogno == subprogno); | 10074 | state->callback_subprogno == subprogno); |
| 10222 | } | 10075 | } |
| 10223 | 10076 | ||
| 10224 | /* Returns whether or not the given map type can potentially elide | 10077 | /* Returns whether or not the given map can potentially elide |
| 10225 | * lookup return value nullness check. This is possible if the key | 10078 | * lookup return value nullness check. This is possible if the key |
| 10226 | * is statically known. | 10079 | * is statically known. |
| 10227 | */ | 10080 | */ |
| 10228 | static bool can_elide_value_nullness(enum bpf_map_type type) | 10081 | static bool can_elide_value_nullness(const struct bpf_map *map) |
| 10229 | { | 10082 | { |
| 10230 | switch (type) { | 10083 | if (map->map_flags & BPF_F_INNER_MAP) |
| 10084 | return false; | ||
| 10085 | |||
| 10086 | switch (map->map_type) { | ||
| 10231 | case BPF_MAP_TYPE_ARRAY: | 10087 | case BPF_MAP_TYPE_ARRAY: |
| 10232 | case BPF_MAP_TYPE_PERCPU_ARRAY: | 10088 | case BPF_MAP_TYPE_PERCPU_ARRAY: |
| 10233 | return true; | 10089 | return true; |
| @@ -10272,6 +10128,24 @@ static const char *non_sleepable_context_description(struct bpf_verifier_env *en · 38 unmodified lines | |||
| 10272 | return "non-sleepable prog"; | 10128 | return "non-sleepable prog"; |
| 10273 | } | 10129 | } |
| 10274 | 10130 | ||
| 10131 | static int release_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg, | ||
| 10132 | bool convert_rcu, bool release_dynptr) | ||
| 10133 | { | ||
| 10134 | int err = -EINVAL; | ||
| 10135 | |||
| 10136 | if (bpf_register_is_null(reg)) | ||
| 10137 | return 0; | ||
| 10138 | |||
| 10139 | if (release_dynptr) | ||
| 10140 | err = unmark_stack_slots_dynptr(env, reg); | ||
| 10141 | else if (convert_rcu) | ||
| 10142 | err = ref_convert_alloc_rcu_protected(env, reg->id); | ||
| 10143 | else if (reg_is_referenced(env, reg)) | ||
| 10144 | err = release_reference(env, reg->id); | ||
| 10145 | |||
| 10146 | return err; | ||
| 10147 | } | ||
| 10148 | |||
| 10275 | static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn, | 10149 | static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn, |
| 10276 | int *insn_idx_p) | 10150 | int *insn_idx_p) |
| 10277 | { | 10151 | { |
| @@ -10321,7 +10195,7 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn · 43 unmodified lines | |||
| 10321 | memset(&meta, 0, sizeof(meta)); | 10195 | memset(&meta, 0, sizeof(meta)); |
| 10322 | meta.pkt_access = fn->pkt_access; | 10196 | meta.pkt_access = fn->pkt_access; |
| 10323 | 10197 | ||
| 10324 | err = check_func_proto(fn); | 10198 | err = check_func_proto(fn, &meta); |
| 10325 | if (err) { | 10199 | if (err) { |
| 10326 | verifier_bug(env, "incorrect func proto %s#%d", func_id_name(func_id), func_id); | 10200 | verifier_bug(env, "incorrect func proto %s#%d", func_id_name(func_id), func_id); |
| 10327 | return err; | 10201 | return err; |
| @@ -10353,55 +10227,26 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn · 25 unmodified lines | |||
| 10353 | if (err) | 10227 | if (err) |
| 10354 | return err; | 10228 | return err; |
| 10355 | 10229 | ||
| 10230 | regs = cur_regs(env); | ||
| 10231 | |||
| 10356 | /* Mark slots with STACK_MISC in case of raw mode, stack offset | 10232 | /* Mark slots with STACK_MISC in case of raw mode, stack offset |
| 10357 | * is inferred from register state. | 10233 | * is inferred from register state. |
| 10358 | */ | 10234 | */ |
| 10359 | for (i = 0; i < meta.access_size; i++) { | 10235 | for (i = 0; i < meta.access_size; i++) { |
| 10360 | err = check_mem_access(env, insn_idx, meta.regno, i, BPF_B, | 10236 | err = check_mem_access(env, insn_idx, regs + meta.regno, argno_from_reg(meta.regno), i, BPF_B, |
| 10361 | BPF_WRITE, -1, false, false); | 10237 | BPF_WRITE, -1, false, false); |
| 10362 | if (err) | 10238 | if (err) |
| 10363 | return err; | 10239 | return err; |
| 10364 | } | 10240 | } |
| 10365 | 10241 | ||
| 10366 | regs = cur_regs(env); | ||
| 10367 | |||
| 10368 | if (meta.release_regno) { | 10242 | if (meta.release_regno) { |
| 10369 | err = -EINVAL; | 10243 | struct bpf_reg_state *reg = ®s[meta.release_regno]; |
| 10370 | if (arg_type_is_dynptr(fn->arg_type[meta.release_regno - BPF_REG_1])) { | 10244 | bool convert_rcu = (func_id == BPF_FUNC_kptr_xchg) && in_rcu_cs(env) && |
| 10371 | err = unmark_stack_slots_dynptr(env, ®s[meta.release_regno]); | 10245 | (reg->type & MEM_ALLOC) && (reg->type & MEM_PERCPU); |
| 10372 | } else if (func_id == BPF_FUNC_kptr_xchg && meta.ref_obj_id) { | 10246 | |
| 10373 | u32 ref_obj_id = meta.ref_obj_id; | 10247 | err = release_reg(env, reg, convert_rcu, !!meta.dynptr.id); |
| 10374 | bool in_rcu = in_rcu_cs(env); | 10248 | if (err) |
| 10375 | struct bpf_func_state *state; | ||
| 10376 | struct bpf_reg_state *reg; | ||
| 10377 | |||
| 10378 | err = release_reference_nomark(env->cur_state, ref_obj_id); | ||
| 10379 | if (!err) { | ||
| 10380 | bpf_for_each_reg_in_vstate(env->cur_state, state, reg, ({ | ||
| 10381 | if (reg->ref_obj_id == ref_obj_id) { | ||
| 10382 | if (in_rcu && (reg->type & MEM_ALLOC) && (reg->type & MEM_PERCPU)) { | ||
| 10383 | reg->ref_obj_id = 0; | ||
| 10384 | reg->type &= ~MEM_ALLOC; | ||
| 10385 | reg->type |= MEM_RCU; | ||
| 10386 | } else { | ||
| 10387 | mark_reg_invalid(env, reg); | ||
| 10388 | } | ||
| 10389 | } | ||
| 10390 | })); | ||
| 10391 | } | ||
| 10392 | } else if (meta.ref_obj_id) { | ||
| 10393 | err = release_reference(env, meta.ref_obj_id); | ||
| 10394 | } else if (bpf_register_is_null(®s[meta.release_regno])) { | ||
| 10395 | /* meta.ref_obj_id can only be 0 if register that is meant to be | ||
| 10396 | * released is NULL, which must be > R0. | ||
| 10397 | */ | ||
| 10398 | err = 0; | ||
| 10399 | } | ||
| 10400 | if (err) { | ||
| 10401 | verbose(env, "func %s#%d reference has not been acquired before\n", | ||
| 10402 | func_id_name(func_id), func_id); | ||
| 10403 | return err; | 10249 | return err; |
| 10404 | } | ||
| 10405 | } | 10250 | } |
| 10406 | 10251 | ||
| 10407 | switch (func_id) { | 10252 | switch (func_id) { |
| @@ -10442,7 +10287,7 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn · 34 unmodified lines | |||
| 10442 | err = mark_chain_precision(env, BPF_REG_1); | 10287 | err = mark_chain_precision(env, BPF_REG_1); |
| 10443 | if (err) | 10288 | if (err) |
| 10444 | return err; | 10289 | return err; |
| 10445 | if (cur_func(env)->callback_depth < regs[BPF_REG_1].umax_value) { | 10290 | if (cur_func(env)->callback_depth < reg_umax(®s[BPF_REG_1])) { |
| 10446 | err = push_callback_call(env, insn, insn_idx, meta.subprogno, | 10291 | err = push_callback_call(env, insn, insn_idx, meta.subprogno, |
| 10447 | set_loop_callback_state); | 10292 | set_loop_callback_state); |
| 10448 | } else { | 10293 | } else { |
| @@ -10460,6 +10305,24 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn · 11 unmodified lines | |||
| 10460 | } | 10305 | } |
| 10461 | break; | 10306 | break; |
| 10462 | case BPF_FUNC_set_retval: | 10307 | case BPF_FUNC_set_retval: |
| 10308 | { | ||
| 10309 | struct bpf_retval_range range = { | ||
| 10310 | .minval = -MAX_ERRNO, | ||
| 10311 | .maxval = 0, | ||
| 10312 | .return_32bit = true | ||
| 10313 | }; | ||
| 10314 | struct bpf_reg_state *r1 = ®s[BPF_REG_1]; | ||
| 10315 | |||
| 10316 | if (r1->type != SCALAR_VALUE) { | ||
| 10317 | verbose(env, "R1 is not a scalar\n"); | ||
| 10318 | return -EINVAL; | ||
| 10319 | } | ||
| 10320 | |||
| 10321 | /* CGROUP_GETSOCKOPT is allowed to return arbitrary value */ | ||
| 10322 | if (prog_type == BPF_PROG_TYPE_CGROUP_SOCKOPT && | ||
| 10323 | env->prog->expected_attach_type == BPF_CGROUP_GETSOCKOPT) | ||
| 10324 | break; | ||
| 10325 | |||
| 10463 | if (prog_type == BPF_PROG_TYPE_LSM && | 10326 | if (prog_type == BPF_PROG_TYPE_LSM && |
| 10464 | env->prog->expected_attach_type == BPF_LSM_CGROUP) { | 10327 | env->prog->expected_attach_type == BPF_LSM_CGROUP) { |
| 10465 | if (!env->prog->aux->attach_func_proto->type) { | 10328 | if (!env->prog->aux->attach_func_proto->type) { |
| @@ -10469,54 +10332,24 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn · 3 unmodified lines | |||
| 10469 | verbose(env, "BPF_LSM_CGROUP that attach to void LSM hooks can't modify return value!\n"); | 10332 | verbose(env, "BPF_LSM_CGROUP that attach to void LSM hooks can't modify return value!\n"); |
| 10470 | return -EINVAL; | 10333 | return -EINVAL; |
| 10471 | } | 10334 | } |
| 10472 | } | 10335 | bpf_lsm_get_retval_range(env->prog, &range); |
| 10473 | break; | ||
| 10474 | case BPF_FUNC_dynptr_data: | ||
| 10475 | { | ||
| 10476 | struct bpf_reg_state *reg; | ||
| 10477 | int id, ref_obj_id; | ||
| 10478 | |||
| 10479 | reg = get_dynptr_arg_reg(env, fn, regs); | ||
| 10480 | if (!reg) | ||
| 10481 | return -EFAULT; | ||
| 10482 | |||
| 10483 | |||
| 10484 | if (meta.dynptr_id) { | ||
| 10485 | verifier_bug(env, "meta.dynptr_id already set"); | ||
| 10486 | return -EFAULT; | ||
| 10487 | } | ||
| 10488 | if (meta.ref_obj_id) { | ||
| 10489 | verifier_bug(env, "meta.ref_obj_id already set"); | ||
| 10490 | return -EFAULT; | ||
| 10491 | } | 10336 | } |
| 10492 | 10337 | ||
| 10493 | id = dynptr_id(env, reg); | 10338 | err = mark_chain_precision(env, BPF_REG_1); |
| 10494 | if (id < 0) { | 10339 | if (err) |
| 10495 | verifier_bug(env, "failed to obtain dynptr id"); | 10340 | return err; |
| 10496 | return id; | ||
| 10497 | } | ||
| 10498 | 10341 | ||
| 10499 | ref_obj_id = dynptr_ref_obj_id(env, reg); | 10342 | if (!retval_range_within(range, r1)) { |
| 10500 | if (ref_obj_id < 0) { | 10343 | verbose_invalid_scalar(env, r1, range, "At bpf_set_retval", "R1"); |
| 10501 | verifier_bug(env, "failed to obtain dynptr ref_obj_id"); | 10344 | return -EINVAL; |
| 10502 | return ref_obj_id; | ||
| 10503 | } | 10345 | } |
| 10504 | 10346 | ||
| 10505 | meta.dynptr_id = id; | ||
| 10506 | meta.ref_obj_id = ref_obj_id; | ||
| 10507 | |||
| 10508 | break; | 10347 | break; |
| 10509 | } | 10348 | } |
| 10510 | case BPF_FUNC_dynptr_write: | 10349 | case BPF_FUNC_dynptr_write: |
| 10511 | { | 10350 | { |
| 10512 | enum bpf_dynptr_type dynptr_type; | 10351 | enum bpf_dynptr_type dynptr_type = meta.dynptr.type; |
| 10513 | struct bpf_reg_state *reg; | ||
| 10514 | |||
| 10515 | reg = get_dynptr_arg_reg(env, fn, regs); | ||
| 10516 | if (!reg) | ||
| 10517 | return -EFAULT; | ||
| 10518 | 10352 | ||
| 10519 | dynptr_type = dynptr_get_type(env, reg); | ||
| 10520 | if (dynptr_type == BPF_DYNPTR_TYPE_INVALID) | 10353 | if (dynptr_type == BPF_DYNPTR_TYPE_INVALID) |
| 10521 | return -EFAULT; | 10354 | return -EFAULT; |
| 10522 | 10355 | ||
| @@ -10560,6 +10393,7 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn · 37 unmodified lines | |||
| 10560 | bpf_mark_reg_not_init(env, ®s[caller_saved[i]]); | 10393 | bpf_mark_reg_not_init(env, ®s[caller_saved[i]]); |
| 10561 | check_reg_arg(env, caller_saved[i], DST_OP_NO_MARK); | 10394 | check_reg_arg(env, caller_saved[i], DST_OP_NO_MARK); |
| 10562 | } | 10395 | } |
| 10396 | invalidate_outgoing_stack_args(env, cur_func(env)); | ||
| 10563 | 10397 | ||
| 10564 | /* helper call returns 64-bit value. */ | 10398 | /* helper call returns 64-bit value. */ |
| 10565 | regs[BPF_REG_0].subreg_def = DEF_NOT_SUBREG; | 10399 | regs[BPF_REG_0].subreg_def = DEF_NOT_SUBREG; |
| @@ -10589,7 +10423,7 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn · 23 unmodified lines | |||
| 10589 | } | 10423 | } |
| 10590 | 10424 | ||
| 10591 | if (func_id == BPF_FUNC_map_lookup_elem && | 10425 | if (func_id == BPF_FUNC_map_lookup_elem && |
| 10592 | can_elide_value_nullness(meta.map.ptr->map_type) && | 10426 | can_elide_value_nullness(meta.map.ptr) && |
| 10593 | meta.const_map_key >= 0 && | 10427 | meta.const_map_key >= 0 && |
| 10594 | meta.const_map_key < meta.map.ptr->max_entries) | 10428 | meta.const_map_key < meta.map.ptr->max_entries) |
| 10595 | ret_flag &= ~PTR_MAYBE_NULL; | 10429 | ret_flag &= ~PTR_MAYBE_NULL; |
| @@ -10701,29 +10535,45 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn · 105 unmodified lines | |||
| 10701 | if (type_may_be_null(regs[BPF_REG_0].type)) | 10535 | if (type_may_be_null(regs[BPF_REG_0].type)) |
| 10702 | regs[BPF_REG_0].id = ++env->id_gen; | 10536 | regs[BPF_REG_0].id = ++env->id_gen; |
| 10703 | 10537 | ||
| 10704 | if (helper_multiple_ref_obj_use(func_id, meta.map.ptr)) { | 10538 | if (is_ptr_cast_function(func_id) && |
| 10705 | verifier_bug(env, "func %s#%d sets ref_obj_id more than once", | 10539 | find_reference_state(env->cur_state, meta.ref_obj.id)) { |
| 10706 | func_id_name(func_id), func_id); | 10540 | struct bpf_verifier_state *branch; |
| 10707 | return -EFAULT; | 10541 | struct bpf_reg_state *r0; |
| 10708 | } | 10542 | |
| 10543 | err = validate_ref_obj(env, &meta.ref_obj); | ||
| 10544 | if (err) | ||
| 10545 | return err; | ||
| 10546 | |||
| 10547 | /* | ||
| 10548 | * In order for a release of any of the original or cast pointers | ||
| 10549 | * to invalidate all other pointers, reuse the same reference id for | ||
| 10550 | * the cast result. | ||
| 10551 | * This reference id can't be used for nullness propagation, | ||
| 10552 | * as cast might return NULL for a non-NULL input. | ||
| 10553 | * Hence, explore the NULL case as a separate branch. | ||
| 10554 | */ | ||
| 10555 | branch = push_stack(env, env->insn_idx + 1, env->insn_idx, false); | ||
| 10556 | if (IS_ERR(branch)) | ||
| 10557 | return PTR_ERR(branch); | ||
| 10709 | 10558 | ||
| 10710 | if (is_dynptr_ref_function(func_id)) | 10559 | r0 = &branch->frame[branch->curframe]->regs[BPF_REG_0]; |
| 10711 | regs[BPF_REG_0].dynptr_id = meta.dynptr_id; | 10560 | __mark_reg_known_zero(r0); |
| 10561 | r0->type = SCALAR_VALUE; | ||
| 10712 | 10562 | ||
| 10713 | if (is_ptr_cast_function(func_id) || is_dynptr_ref_function(func_id)) { | 10563 | regs[BPF_REG_0].type &= ~PTR_MAYBE_NULL; |
| 10714 | /* For release_reference() */ | 10564 | regs[BPF_REG_0].id = meta.ref_obj.id; |
| 10715 | regs[BPF_REG_0].ref_obj_id = meta.ref_obj_id; | ||
| 10716 | } else if (is_acquire_function(func_id, meta.map.ptr)) { | 10565 | } else if (is_acquire_function(func_id, meta.map.ptr)) { |
| 10717 | int id = acquire_reference(env, insn_idx); | 10566 | int id = acquire_reference(env, insn_idx, 0); |
| 10718 | 10567 | ||
| 10719 | if (id < 0) | 10568 | if (id < 0) |
| 10720 | return id; | 10569 | return id; |
| 10721 | /* For mark_ptr_or_null_reg() */ | 10570 | |
| 10722 | regs[BPF_REG_0].id = id; | 10571 | regs[BPF_REG_0].id = id; |
| 10723 | /* For release_reference() */ | ||
| 10724 | regs[BPF_REG_0].ref_obj_id = id; | ||
| 10725 | } | 10572 | } |
| 10726 | 10573 | ||
| 10574 | if (func_id == BPF_FUNC_dynptr_data) | ||
| 10575 | regs[BPF_REG_0].parent_id = meta.dynptr.id; | ||
| 10576 | |||
| 10727 | err = do_refine_retval_range(env, regs, fn->ret_type, func_id, &meta); | 10577 | err = do_refine_retval_range(env, regs, fn->ret_type, func_id, &meta); |
| 10728 | if (err) | 10578 | if (err) |
| 10729 | return err; | 10579 | return err; |
| @@ -10819,7 +10669,6 @@ static bool is_kfunc_release(struct bpf_kfunc_call_arg_meta *meta) · 89 unmodified lines | |||
| 10819 | return meta->kfunc_flags & KF_RELEASE; | 10669 | return meta->kfunc_flags & KF_RELEASE; |
| 10820 | } | 10670 | } |
| 10821 | 10671 | ||
| 10822 | |||
| 10823 | static bool is_kfunc_destructive(struct bpf_kfunc_call_arg_meta *meta) | 10672 | static bool is_kfunc_destructive(struct bpf_kfunc_call_arg_meta *meta) |
| 10824 | { | 10673 | { |
| 10825 | return meta->kfunc_flags & KF_DESTRUCTIVE; | 10674 | return meta->kfunc_flags & KF_DESTRUCTIVE; |
| @@ -10896,6 +10745,11 @@ static bool is_kfunc_arg_nullable(const struct btf *btf, const struct btf_param · 70 unmodified lines | |||
| 10896 | return btf_param_match_suffix(btf, arg, "__nullable"); | 10745 | return btf_param_match_suffix(btf, arg, "__nullable"); |
| 10897 | } | 10746 | } |
| 10898 | 10747 | ||
| 10748 | static bool is_kfunc_arg_nonown_allowed(const struct btf *btf, const struct btf_param *arg) | ||
| 10749 | { | ||
| 10750 | return btf_param_match_suffix(btf, arg, "__nonown_allowed"); | ||
| 10751 | } | ||
| 10752 | |||
| 10899 | static bool is_kfunc_arg_const_str(const struct btf *btf, const struct btf_param *arg) | 10753 | static bool is_kfunc_arg_const_str(const struct btf *btf, const struct btf_param *arg) |
| 10900 | { | 10754 | { |
| 10901 | return btf_param_match_suffix(btf, arg, "__str"); | 10755 | return btf_param_match_suffix(btf, arg, "__str"); |
| @@ -11136,10 +10990,15 @@ enum special_kfunc_type { · 234 unmodified lines | |||
| 11136 | KF_bpf_list_push_front, | 10990 | KF_bpf_list_push_front, |
| 11137 | KF_bpf_list_push_back_impl, | 10991 | KF_bpf_list_push_back_impl, |
| 11138 | KF_bpf_list_push_back, | 10992 | KF_bpf_list_push_back, |
| 10993 | KF_bpf_list_add, | ||
| 11139 | KF_bpf_list_pop_front, | 10994 | KF_bpf_list_pop_front, |
| 11140 | KF_bpf_list_pop_back, | 10995 | KF_bpf_list_pop_back, |
| 10996 | KF_bpf_list_del, | ||
| 11141 | KF_bpf_list_front, | 10997 | KF_bpf_list_front, |
| 11142 | KF_bpf_list_back, | 10998 | KF_bpf_list_back, |
| 10999 | KF_bpf_list_is_first, | ||
| 11000 | KF_bpf_list_is_last, | ||
| 11001 | KF_bpf_list_empty, | ||
| 11143 | KF_bpf_cast_to_kern_ctx, | 11002 | KF_bpf_cast_to_kern_ctx, |
| 11144 | KF_bpf_rdonly_cast, | 11003 | KF_bpf_rdonly_cast, |
| 11145 | KF_bpf_rcu_read_lock, | 11004 | KF_bpf_rcu_read_lock, |
| @@ -11204,10 +11063,15 @@ BTF_ID(func, bpf_list_push_front_impl) · 58 unmodified lines | |||
| 11204 | BTF_ID(func, bpf_list_push_front) | 11063 | BTF_ID(func, bpf_list_push_front) |
| 11205 | BTF_ID(func, bpf_list_push_back_impl) | 11064 | BTF_ID(func, bpf_list_push_back_impl) |
| 11206 | BTF_ID(func, bpf_list_push_back) | 11065 | BTF_ID(func, bpf_list_push_back) |
| 11066 | BTF_ID(func, bpf_list_add) | ||
| 11207 | BTF_ID(func, bpf_list_pop_front) | 11067 | BTF_ID(func, bpf_list_pop_front) |
| 11208 | BTF_ID(func, bpf_list_pop_back) | 11068 | BTF_ID(func, bpf_list_pop_back) |
| 11069 | BTF_ID(func, bpf_list_del) | ||
| 11209 | BTF_ID(func, bpf_list_front) | 11070 | BTF_ID(func, bpf_list_front) |
| 11210 | BTF_ID(func, bpf_list_back) | 11071 | BTF_ID(func, bpf_list_back) |
| 11072 | BTF_ID(func, bpf_list_is_first) | ||
| 11073 | BTF_ID(func, bpf_list_is_last) | ||
| 11074 | BTF_ID(func, bpf_list_empty) | ||
| 11211 | BTF_ID(func, bpf_cast_to_kern_ctx) | 11075 | BTF_ID(func, bpf_cast_to_kern_ctx) |
| 11212 | BTF_ID(func, bpf_rdonly_cast) | 11076 | BTF_ID(func, bpf_rdonly_cast) |
| 11213 | BTF_ID(func, bpf_rcu_read_lock) | 11077 | BTF_ID(func, bpf_rcu_read_lock) |
| @@ -11319,7 +11183,8 @@ static bool is_bpf_list_push_kfunc(u32 func_id) · 105 unmodified lines | |||
| 11319 | return func_id == special_kfunc_list[KF_bpf_list_push_front] || | 11183 | return func_id == special_kfunc_list[KF_bpf_list_push_front] || |
| 11320 | func_id == special_kfunc_list[KF_bpf_list_push_front_impl] || | 11184 | func_id == special_kfunc_list[KF_bpf_list_push_front_impl] || |
| 11321 | func_id == special_kfunc_list[KF_bpf_list_push_back] || | 11185 | func_id == special_kfunc_list[KF_bpf_list_push_back] || |
| 11322 | func_id == special_kfunc_list[KF_bpf_list_push_back_impl]; | 11186 | func_id == special_kfunc_list[KF_bpf_list_push_back_impl] || |
| 11187 | func_id == special_kfunc_list[KF_bpf_list_add]; | ||
| 11323 | } | 11188 | } |
| 11324 | 11189 | ||
| 11325 | static bool is_bpf_rbtree_add_kfunc(u32 func_id) | 11190 | static bool is_bpf_rbtree_add_kfunc(u32 func_id) |
| @@ -11368,15 +11233,12 @@ bool bpf_is_kfunc_pkt_changing(struct bpf_kfunc_call_arg_meta *meta) · 42 unmodified lines | |||
| 11368 | } | 11233 | } |
| 11369 | 11234 | ||
| 11370 | static enum kfunc_ptr_arg_type | 11235 | static enum kfunc_ptr_arg_type |
| 11371 | get_kfunc_ptr_arg_type(struct bpf_verifier_env *env, | 11236 | get_kfunc_ptr_arg_type(struct bpf_verifier_env *env, struct bpf_func_state *caller, |
| 11372 | struct bpf_kfunc_call_arg_meta *meta, | 11237 | struct bpf_reg_state *regs, struct bpf_kfunc_call_arg_meta *meta, |
| 11373 | const struct btf_type *t, const struct btf_type *ref_t, | 11238 | const struct btf_type *t, const struct btf_type *ref_t, |
| 11374 | const char *ref_tname, const struct btf_param *args, | 11239 | const char *ref_tname, const struct btf_param *args, |
| 11375 | int argno, int nargs) | 11240 | int arg, int nargs, argno_t argno, struct bpf_reg_state *reg) |
| 11376 | { | 11241 | { |
| 11377 | u32 regno = argno + 1; | ||
| 11378 | struct bpf_reg_state *regs = cur_regs(env); | ||
| 11379 | struct bpf_reg_state *reg = ®s[regno]; | ||
| 11380 | bool arg_mem_size = false; | 11242 | bool arg_mem_size = false; |
| 11381 | 11243 | ||
| 11382 | if (meta->func_id == special_kfunc_list[KF_bpf_cast_to_kern_ctx] || | 11244 | if (meta->func_id == special_kfunc_list[KF_bpf_cast_to_kern_ctx] || |
| @@ -11384,9 +11246,9 @@ get_kfunc_ptr_arg_type(struct bpf_verifier_env *env, · 1 unmodified line | |||
| 11384 | meta->func_id == special_kfunc_list[KF_bpf_session_cookie]) | 11246 | meta->func_id == special_kfunc_list[KF_bpf_session_cookie]) |
| 11385 | return KF_ARG_PTR_TO_CTX; | 11247 | return KF_ARG_PTR_TO_CTX; |
| 11386 | 11248 | ||
| 11387 | if (argno + 1 < nargs && | 11249 | if (arg + 1 < nargs && |
| 11388 | (is_kfunc_arg_mem_size(meta->btf, &args[argno + 1], ®s[regno + 1]) || | 11250 | (is_kfunc_arg_mem_size(meta->btf, &args[arg + 1], get_func_arg_reg(caller, regs, arg + 1)) || |
| 11389 | is_kfunc_arg_const_mem_size(meta->btf, &args[argno + 1], ®s[regno + 1]))) | 11251 | is_kfunc_arg_const_mem_size(meta->btf, &args[arg + 1], get_func_arg_reg(caller, regs, arg + 1)))) |
| 11390 | arg_mem_size = true; | 11252 | arg_mem_size = true; |
| 11391 | 11253 | ||
| 11392 | /* In this function, we verify the kfunc's BTF as per the argument type, | 11254 | /* In this function, we verify the kfunc's BTF as per the argument type, |
| @@ -11394,68 +11256,69 @@ get_kfunc_ptr_arg_type(struct bpf_verifier_env *env, · 1 unmodified line | |||
| 11394 | * type to our caller. When a set of conditions hold in the BTF type of | 11256 | * type to our caller. When a set of conditions hold in the BTF type of |
| 11395 | * arguments, we resolve it to a known kfunc_ptr_arg_type. | 11257 | * arguments, we resolve it to a known kfunc_ptr_arg_type. |
| 11396 | */ | 11258 | */ |
| 11397 | if (btf_is_prog_ctx_type(&env->log, meta->btf, t, resolve_prog_type(env->prog), argno)) | 11259 | if (btf_is_prog_ctx_type(&env->log, meta->btf, t, resolve_prog_type(env->prog), arg)) |
| 11398 | return KF_ARG_PTR_TO_CTX; | 11260 | return KF_ARG_PTR_TO_CTX; |
| 11399 | 11261 | ||
| 11400 | if (is_kfunc_arg_nullable(meta->btf, &args[argno]) && bpf_register_is_null(reg) && | 11262 | if (is_kfunc_arg_nullable(meta->btf, &args[arg]) && bpf_register_is_null(reg) && |
| 11401 | !arg_mem_size) | 11263 | !arg_mem_size) |
| 11402 | return KF_ARG_PTR_TO_NULL; | 11264 | return KF_ARG_PTR_TO_NULL; |
| 11403 | 11265 | ||
| 11404 | if (is_kfunc_arg_alloc_obj(meta->btf, &args[argno])) | 11266 | if (is_kfunc_arg_alloc_obj(meta->btf, &args[arg])) |
| 11405 | return KF_ARG_PTR_TO_ALLOC_BTF_ID; | 11267 | return KF_ARG_PTR_TO_ALLOC_BTF_ID; |
| 11406 | 11268 | ||
| 11407 | if (is_kfunc_arg_refcounted_kptr(meta->btf, &args[argno])) | 11269 | if (is_kfunc_arg_refcounted_kptr(meta->btf, &args[arg])) |
| 11408 | return KF_ARG_PTR_TO_REFCOUNTED_KPTR; | 11270 | return KF_ARG_PTR_TO_REFCOUNTED_KPTR; |
| 11409 | 11271 | ||
| 11410 | if (is_kfunc_arg_dynptr(meta->btf, &args[argno])) | 11272 | if (is_kfunc_arg_dynptr(meta->btf, &args[arg])) |
| 11411 | return KF_ARG_PTR_TO_DYNPTR; | 11273 | return KF_ARG_PTR_TO_DYNPTR; |
| 11412 | 11274 | ||
| 11413 | if (is_kfunc_arg_iter(meta, argno, &args[argno])) | 11275 | if (is_kfunc_arg_iter(meta, arg, &args[arg])) |
| 11414 | return KF_ARG_PTR_TO_ITER; | 11276 | return KF_ARG_PTR_TO_ITER; |
| 11415 | 11277 | ||
| 11416 | if (is_kfunc_arg_list_head(meta->btf, &args[argno])) | 11278 | if (is_kfunc_arg_list_head(meta->btf, &args[arg])) |
| 11417 | return KF_ARG_PTR_TO_LIST_HEAD; | 11279 | return KF_ARG_PTR_TO_LIST_HEAD; |
| 11418 | 11280 | ||
| 11419 | if (is_kfunc_arg_list_node(meta->btf, &args[argno])) | 11281 | if (is_kfunc_arg_list_node(meta->btf, &args[arg])) |
| 11420 | return KF_ARG_PTR_TO_LIST_NODE; | 11282 | return KF_ARG_PTR_TO_LIST_NODE; |
| 11421 | 11283 | ||
| 11422 | if (is_kfunc_arg_rbtree_root(meta->btf, &args[argno])) | 11284 | if (is_kfunc_arg_rbtree_root(meta->btf, &args[arg])) |
| 11423 | return KF_ARG_PTR_TO_RB_ROOT; | 11285 | return KF_ARG_PTR_TO_RB_ROOT; |
| 11424 | 11286 | ||
| 11425 | if (is_kfunc_arg_rbtree_node(meta->btf, &args[argno])) | 11287 | if (is_kfunc_arg_rbtree_node(meta->btf, &args[arg])) |
| 11426 | return KF_ARG_PTR_TO_RB_NODE; | 11288 | return KF_ARG_PTR_TO_RB_NODE; |
| 11427 | 11289 | ||
| 11428 | if (is_kfunc_arg_const_str(meta->btf, &args[argno])) | 11290 | if (is_kfunc_arg_const_str(meta->btf, &args[arg])) |
| 11429 | return KF_ARG_PTR_TO_CONST_STR; | 11291 | return KF_ARG_PTR_TO_CONST_STR; |
| 11430 | 11292 | ||
| 11431 | if (is_kfunc_arg_map(meta->btf, &args[argno])) | 11293 | if (is_kfunc_arg_map(meta->btf, &args[arg])) |
| 11432 | return KF_ARG_PTR_TO_MAP; | 11294 | return KF_ARG_PTR_TO_MAP; |
| 11433 | 11295 | ||
| 11434 | if (is_kfunc_arg_wq(meta->btf, &args[argno])) | 11296 | if (is_kfunc_arg_wq(meta->btf, &args[arg])) |
| 11435 | return KF_ARG_PTR_TO_WORKQUEUE; | 11297 | return KF_ARG_PTR_TO_WORKQUEUE; |
| 11436 | 11298 | ||
| 11437 | if (is_kfunc_arg_timer(meta->btf, &args[argno])) | 11299 | if (is_kfunc_arg_timer(meta->btf, &args[arg])) |
| 11438 | return KF_ARG_PTR_TO_TIMER; | 11300 | return KF_ARG_PTR_TO_TIMER; |
| 11439 | 11301 | ||
| 11440 | if (is_kfunc_arg_task_work(meta->btf, &args[argno])) | 11302 | if (is_kfunc_arg_task_work(meta->btf, &args[arg])) |
| 11441 | return KF_ARG_PTR_TO_TASK_WORK; | 11303 | return KF_ARG_PTR_TO_TASK_WORK; |
| 11442 | 11304 | ||
| 11443 | if (is_kfunc_arg_irq_flag(meta->btf, &args[argno])) | 11305 | if (is_kfunc_arg_irq_flag(meta->btf, &args[arg])) |
| 11444 | return KF_ARG_PTR_TO_IRQ_FLAG; | 11306 | return KF_ARG_PTR_TO_IRQ_FLAG; |
| 11445 | 11307 | ||
| 11446 | if (is_kfunc_arg_res_spin_lock(meta->btf, &args[argno])) | 11308 | if (is_kfunc_arg_res_spin_lock(meta->btf, &args[arg])) |
| 11447 | return KF_ARG_PTR_TO_RES_SPIN_LOCK; | 11309 | return KF_ARG_PTR_TO_RES_SPIN_LOCK; |
| 11448 | 11310 | ||
| 11449 | if ((base_type(reg->type) == PTR_TO_BTF_ID || reg2btf_ids[base_type(reg->type)])) { | 11311 | if ((base_type(reg->type) == PTR_TO_BTF_ID || reg2btf_ids[base_type(reg->type)])) { |
| 11450 | if (!btf_type_is_struct(ref_t)) { | 11312 | if (!btf_type_is_struct(ref_t)) { |
| 11451 | verbose(env, "kernel function %s args#%d pointer type %s %s is not supported\n", | 11313 | verbose(env, "kernel function %s %s pointer type %s %s is not supported\n", |
| 11452 | meta->func_name, argno, btf_type_str(ref_t), ref_tname); | 11314 | meta->func_name, reg_arg_name(env, argno), |
| 11315 | btf_type_str(ref_t), ref_tname); | ||
| 11453 | return -EINVAL; | 11316 | return -EINVAL; |
| 11454 | } | 11317 | } |
| 11455 | return KF_ARG_PTR_TO_BTF_ID; | 11318 | return KF_ARG_PTR_TO_BTF_ID; |
| 11456 | } | 11319 | } |
| 11457 | 11320 | ||
| 11458 | if (is_kfunc_arg_callback(env, meta->btf, &args[argno])) | 11321 | if (is_kfunc_arg_callback(env, meta->btf, &args[arg])) |
| 11459 | return KF_ARG_PTR_TO_CALLBACK; | 11322 | return KF_ARG_PTR_TO_CALLBACK; |
| 11460 | 11323 | ||
| 11461 | /* This is the catch all argument type of register types supported by | 11324 | /* This is the catch all argument type of register types supported by |
| @@ -11465,8 +11328,9 @@ get_kfunc_ptr_arg_type(struct bpf_verifier_env *env, · 3 unmodified lines | |||
| 11465 | */ | 11328 | */ |
| 11466 | if (!btf_type_is_scalar(ref_t) && !__btf_type_is_scalar_struct(env, meta->btf, ref_t, 0) && | 11329 | if (!btf_type_is_scalar(ref_t) && !__btf_type_is_scalar_struct(env, meta->btf, ref_t, 0) && |
| 11467 | (arg_mem_size ? !btf_type_is_void(ref_t) : 1)) { | 11330 | (arg_mem_size ? !btf_type_is_void(ref_t) : 1)) { |
| 11468 | verbose(env, "arg#%d pointer type %s %s must point to %sscalar, or struct with scalar\n", | 11331 | verbose(env, "%s pointer type %s %s must point to %sscalar, or struct with scalar\n", |
| 11469 | argno, btf_type_str(ref_t), ref_tname, arg_mem_size ? "void, " : ""); | 11332 | reg_arg_name(env, argno), |
| 11333 | btf_type_str(ref_t), ref_tname, arg_mem_size ? "void, " : ""); | ||
| 11470 | return -EINVAL; | 11334 | return -EINVAL; |
| 11471 | } | 11335 | } |
| 11472 | return arg_mem_size ? KF_ARG_PTR_TO_MEM_SIZE : KF_ARG_PTR_TO_MEM; | 11336 | return arg_mem_size ? KF_ARG_PTR_TO_MEM_SIZE : KF_ARG_PTR_TO_MEM; |
| @@ -11477,7 +11341,7 @@ static int process_kf_arg_ptr_to_btf_id(struct bpf_verifier_env *env, · 4 unmodified lines | |||
| 11477 | const struct btf_type *ref_t, | 11341 | const struct btf_type *ref_t, |
| 11478 | const char *ref_tname, u32 ref_id, | 11342 | const char *ref_tname, u32 ref_id, |
| 11479 | struct bpf_kfunc_call_arg_meta *meta, | 11343 | struct bpf_kfunc_call_arg_meta *meta, |
| 11480 | int argno) | 11344 | int arg, argno_t argno) |
| 11481 | { | 11345 | { |
| 11482 | const struct btf_type *reg_ref_t; | 11346 | const struct btf_type *reg_ref_t; |
| 11483 | bool strict_type_match = false; | 11347 | bool strict_type_match = false; |
| @@ -11519,7 +11383,7 @@ static int process_kf_arg_ptr_to_btf_id(struct bpf_verifier_env *env, · 35 unmodified lines | |||
| 11519 | * btf_struct_ids_match() to walk the struct at the 0th offset, and | 11383 | * btf_struct_ids_match() to walk the struct at the 0th offset, and |
| 11520 | * resolve types. | 11384 | * resolve types. |
| 11521 | */ | 11385 | */ |
| 11522 | if ((is_kfunc_release(meta) && reg->ref_obj_id) || | 11386 | if ((is_kfunc_release(meta) && reg_is_referenced(env, reg)) || |
| 11523 | btf_type_ids_nocast_alias(&env->log, reg_btf, reg_ref_id, meta->btf, ref_id)) | 11387 | btf_type_ids_nocast_alias(&env->log, reg_btf, reg_ref_id, meta->btf, ref_id)) |
| 11524 | strict_type_match = true; | 11388 | strict_type_match = true; |
| 11525 | 11389 | ||
| @@ -11535,19 +11399,19 @@ static int process_kf_arg_ptr_to_btf_id(struct bpf_verifier_env *env, · 9 unmodified lines | |||
| 11535 | */ | 11399 | */ |
| 11536 | taking_projection = btf_is_projection_of(ref_tname, reg_ref_tname); | 11400 | taking_projection = btf_is_projection_of(ref_tname, reg_ref_tname); |
| 11537 | if (!taking_projection && !struct_same) { | 11401 | if (!taking_projection && !struct_same) { |
| 11538 | verbose(env, "kernel function %s args#%d expected pointer to %s %s but R%d has a pointer to %s %s\n", | 11402 | verbose(env, "kernel function %s %s expected pointer to %s %s but %s has a pointer to %s %s\n", |
| 11539 | meta->func_name, argno, btf_type_str(ref_t), ref_tname, argno + 1, | 11403 | meta->func_name, reg_arg_name(env, argno), |
| 11404 | btf_type_str(ref_t), ref_tname, reg_arg_name(env, argno), | ||
| 11540 | btf_type_str(reg_ref_t), reg_ref_tname); | 11405 | btf_type_str(reg_ref_t), reg_ref_tname); |
| 11541 | return -EINVAL; | 11406 | return -EINVAL; |
| 11542 | } | 11407 | } |
| 11543 | return 0; | 11408 | return 0; |
| 11544 | } | 11409 | } |
| 11545 | 11410 | ||
| 11546 | static int process_irq_flag(struct bpf_verifier_env *env, int regno, | 11411 | static int process_irq_flag(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno, |
| 11547 | struct bpf_kfunc_call_arg_meta *meta) | 11412 | struct bpf_kfunc_call_arg_meta *meta) |
| 11548 | { | 11413 | { |
| 11549 | struct bpf_reg_state *reg = reg_state(env, regno); | 11414 | int err, spi, kfunc_class = IRQ_NATIVE_KFUNC; |
| 11550 | int err, kfunc_class = IRQ_NATIVE_KFUNC; | ||
| 11551 | bool irq_save; | 11415 | bool irq_save; |
| 11552 | 11416 | ||
| 11553 | if (meta->func_id == special_kfunc_list[KF_bpf_local_irq_save] || | 11417 | if (meta->func_id == special_kfunc_list[KF_bpf_local_irq_save] || |
| @@ -11567,11 +11431,13 @@ static int process_irq_flag(struct bpf_verifier_env *env, int regno, · 13 unmodified lines | |||
| 11567 | 11431 | ||
| 11568 | if (irq_save) { | 11432 | if (irq_save) { |
| 11569 | if (!is_irq_flag_reg_valid_uninit(env, reg)) { | 11433 | if (!is_irq_flag_reg_valid_uninit(env, reg)) { |
| 11570 | verbose(env, "expected uninitialized irq flag as arg#%d\n", regno - 1); | 11434 | verbose(env, "expected uninitialized irq flag as %s\n", |
| 11435 | reg_arg_name(env, argno)); | ||
| 11571 | return -EINVAL; | 11436 | return -EINVAL; |
| 11572 | } | 11437 | } |
| 11573 | 11438 | ||
| 11574 | err = check_mem_access(env, env->insn_idx, regno, 0, BPF_DW, BPF_WRITE, -1, false, false); | 11439 | err = check_mem_access(env, env->insn_idx, reg, argno, 0, BPF_DW, |
| 11440 | BPF_WRITE, -1, false, false); | ||
| 11575 | if (err) | 11441 | if (err) |
| 11576 | return err; | 11442 | return err; |
| 11577 | 11443 | ||
| @@ -11581,13 +11447,16 @@ static int process_irq_flag(struct bpf_verifier_env *env, int regno, · 3 unmodified lines | |||
| 11581 | } else { | 11447 | } else { |
| 11582 | err = is_irq_flag_reg_valid_init(env, reg); | 11448 | err = is_irq_flag_reg_valid_init(env, reg); |
| 11583 | if (err) { | 11449 | if (err) { |
| 11584 | verbose(env, "expected an initialized irq flag as arg#%d\n", regno - 1); | 11450 | verbose(env, "expected an initialized irq flag as %s\n", |
| 11451 | reg_arg_name(env, argno)); | ||
| 11585 | return err; | 11452 | return err; |
| 11586 | } | 11453 | } |
| 11587 | 11454 | ||
| 11588 | err = mark_irq_flag_read(env, reg); | 11455 | spi = irq_flag_get_spi(env, reg); |
| 11589 | if (err) | 11456 | if (spi < 0) |
| 11590 | return err; | 11457 | return spi; |
| 11458 | |||
| 11459 | mark_stack_slots_scratched(env, spi, 1); | ||
| 11591 | 11460 | ||
| 11592 | err = unmark_stack_slot_irq_flag(env, reg, kfunc_class); | 11461 | err = unmark_stack_slot_irq_flag(env, reg, kfunc_class); |
| 11593 | if (err) | 11462 | if (err) |
| @@ -11618,36 +11487,21 @@ static int ref_set_non_owning(struct bpf_verifier_env *env, struct bpf_reg_state · 24 unmodified lines | |||
| 11618 | return 0; | 11487 | return 0; |
| 11619 | } | 11488 | } |
| 11620 | 11489 | ||
| 11621 | static int ref_convert_owning_non_owning(struct bpf_verifier_env *env, u32 ref_obj_id) | 11490 | static void ref_convert_owning_non_owning(struct bpf_verifier_env *env, u32 id) |
| 11622 | { | 11491 | { |
| 11623 | struct bpf_verifier_state *state = env->cur_state; | ||
| 11624 | struct bpf_func_state *unused; | 11492 | struct bpf_func_state *unused; |
| 11625 | struct bpf_reg_state *reg; | 11493 | struct bpf_reg_state *reg; |
| 11626 | int i; | ||
| 11627 | |||
| 11628 | if (!ref_obj_id) { | ||
| 11629 | verifier_bug(env, "ref_obj_id is zero for owning -> non-owning conversion"); | ||
| 11630 | return -EFAULT; | ||
| 11631 | } | ||
| 11632 | 11494 | ||
| 11633 | for (i = 0; i < state->acquired_refs; i++) { | ||
2971 more lines aren’t shown here.
| @@ -11712,6 +11712,15 @@ static int __perf_event_set_bpf_prog(struct perf_event *event, · 11711 unmodified lines | |||
| 11712 | /* only uprobe programs are allowed to be sleepable */ | 11712 | /* only uprobe programs are allowed to be sleepable */ |
| 11713 | return -EINVAL; | 11713 | return -EINVAL; |
| 11714 | 11714 | ||
| 11715 | if (prog->type == BPF_PROG_TYPE_TRACEPOINT && prog->sleepable) { | ||
| 11716 | /* | ||
| 11717 | * Sleepable tracepoint programs can only attach to faultable | ||
| 11718 | * tracepoints. Currently only syscall tracepoints are faultable. | ||
| 11719 | */ | ||
| 11720 | if (!is_syscall_tp) | ||
| 11721 | return -EINVAL; | ||
| 11722 | } | ||
| 11723 | |||
| 11715 | /* Kprobe override only works for kprobes, not uprobes. */ | 11724 | /* Kprobe override only works for kprobes, not uprobes. */ |
| 11716 | if (prog->kprobe_override && !is_kprobe) | 11725 | if (prog->kprobe_override && !is_kprobe) |
| 11717 | return -EINVAL; | 11726 | return -EINVAL; |
| @@ -23,6 +23,7 @@ · 22 unmodified lines | |||
| 23 | #include <linux/sort.h> | 23 | #include <linux/sort.h> |
| 24 | #include <linux/key.h> | 24 | #include <linux/key.h> |
| 25 | #include <linux/namei.h> | 25 | #include <linux/namei.h> |
| 26 | #include <linux/file.h> | ||
| 26 | 27 | ||
| 27 | #include <net/bpf_sk_storage.h> | 28 | #include <net/bpf_sk_storage.h> |
| 28 | 29 | ||
| @@ -42,6 +43,7 @@ · 13 unmodified lines | |||
| 42 | 43 | ||
| 43 | #define MAX_UPROBE_MULTI_CNT (1U << 20) | 44 | #define MAX_UPROBE_MULTI_CNT (1U << 20) |
| 44 | #define MAX_KPROBE_MULTI_CNT (1U << 20) | 45 | #define MAX_KPROBE_MULTI_CNT (1U << 20) |
| 46 | #define MAX_TRACING_MULTI_CNT (1U << 20) | ||
| 45 | 47 | ||
| 46 | #ifdef CONFIG_MODULES | 48 | #ifdef CONFIG_MODULES |
| 47 | struct bpf_trace_module { | 49 | struct bpf_trace_module { |
| @@ -152,6 +154,34 @@ unsigned int trace_call_bpf(struct trace_event_call *call, void *ctx) · 104 unmodified lines | |||
| 152 | return ret; | 154 | return ret; |
| 153 | } | 155 | } |
| 154 | 156 | ||
| 157 | /** | ||
| 158 | * trace_call_bpf_faultable - invoke BPF program in faultable context | ||
| 159 | * @call: tracepoint event | ||
| 160 | * @ctx: opaque context pointer | ||
| 161 | * | ||
| 162 | * Variant of trace_call_bpf() for faultable tracepoints (syscall | ||
| 163 | * tracepoints). Supports sleepable BPF programs by using rcu_tasks_trace | ||
| 164 | * for lifetime protection and bpf_prog_run_array_sleepable() for per-program | ||
| 165 | * RCU flavor selection, following the uprobe pattern. | ||
| 166 | * | ||
| 167 | * Per-program recursion protection is provided by | ||
| 168 | * bpf_prog_run_array_sleepable(). Global bpf_prog_active is not | ||
| 169 | * needed because syscall tracepoints cannot self-recurse. | ||
| 170 | * | ||
| 171 | * Must be called from a faultable/preemptible context. | ||
| 172 | */ | ||
| 173 | unsigned int trace_call_bpf_faultable(struct trace_event_call *call, void *ctx) | ||
| 174 | { | ||
| 175 | struct bpf_prog_array *prog_array; | ||
| 176 | |||
| 177 | might_fault(); | ||
| 178 | guard(rcu_tasks_trace)(); | ||
| 179 | |||
| 180 | prog_array = rcu_dereference_check(call->prog_array, | ||
| 181 | rcu_read_lock_trace_held()); | ||
| 182 | return bpf_prog_run_array_sleepable(prog_array, ctx, bpf_prog_run); | ||
| 183 | } | ||
| 184 | |||
| 155 | #ifdef CONFIG_BPF_KPROBE_OVERRIDE | 185 | #ifdef CONFIG_BPF_KPROBE_OVERRIDE |
| 156 | BPF_CALL_2(bpf_override_return, struct pt_regs *, regs, unsigned long, rc) | 186 | BPF_CALL_2(bpf_override_return, struct pt_regs *, regs, unsigned long, rc) |
| 157 | { | 187 | { |
| @@ -1305,7 +1335,8 @@ static inline bool is_uprobe_session(const struct bpf_prog *prog) · 1147 unmodified lines | |||
| 1305 | static inline bool is_trace_fsession(const struct bpf_prog *prog) | 1335 | static inline bool is_trace_fsession(const struct bpf_prog *prog) |
| 1306 | { | 1336 | { |
| 1307 | return prog->type == BPF_PROG_TYPE_TRACING && | 1337 | return prog->type == BPF_PROG_TYPE_TRACING && |
| 1308 | prog->expected_attach_type == BPF_TRACE_FSESSION; | 1338 | (prog->expected_attach_type == BPF_TRACE_FSESSION || |
| 1339 | prog->expected_attach_type == BPF_TRACE_FSESSION_MULTI); | ||
| 1309 | } | 1340 | } |
| 1310 | 1341 | ||
| 1311 | static const struct bpf_func_proto * | 1342 | static const struct bpf_func_proto * |
| @@ -2072,11 +2103,19 @@ void bpf_put_raw_tracepoint(struct bpf_raw_event_map *btp) · 760 unmodified lines | |||
| 2072 | static __always_inline | 2103 | static __always_inline |
| 2073 | void __bpf_trace_run(struct bpf_raw_tp_link *link, u64 *args) | 2104 | void __bpf_trace_run(struct bpf_raw_tp_link *link, u64 *args) |
| 2074 | { | 2105 | { |
| 2106 | struct srcu_ctr __percpu *scp = NULL; | ||
| 2075 | struct bpf_prog *prog = link->link.prog; | 2107 | struct bpf_prog *prog = link->link.prog; |
| 2108 | bool sleepable = prog->sleepable; | ||
| 2076 | struct bpf_run_ctx *old_run_ctx; | 2109 | struct bpf_run_ctx *old_run_ctx; |
| 2077 | struct bpf_trace_run_ctx run_ctx; | 2110 | struct bpf_trace_run_ctx run_ctx; |
| 2078 | 2111 | ||
| 2079 | rcu_read_lock_dont_migrate(); | 2112 | if (sleepable) { |
| 2113 | scp = rcu_read_lock_tasks_trace(); | ||
| 2114 | migrate_disable(); | ||
| 2115 | } else { | ||
| 2116 | rcu_read_lock_dont_migrate(); | ||
| 2117 | } | ||
| 2118 | |||
| 2080 | if (unlikely(!bpf_prog_get_recursion_context(prog))) { | 2119 | if (unlikely(!bpf_prog_get_recursion_context(prog))) { |
| 2081 | bpf_prog_inc_misses_counter(prog); | 2120 | bpf_prog_inc_misses_counter(prog); |
| 2082 | goto out; | 2121 | goto out; |
| @@ -2085,12 +2124,18 @@ void __bpf_trace_run(struct bpf_raw_tp_link *link, u64 *args) · 2 unmodified lines | |||
| 2085 | run_ctx.bpf_cookie = link->cookie; | 2124 | run_ctx.bpf_cookie = link->cookie; |
| 2086 | old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx); | 2125 | old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx); |
| 2087 | 2126 | ||
| 2088 | (void) bpf_prog_run(prog, args); | 2127 | (void)bpf_prog_run(prog, args); |
| 2089 | 2128 | ||
| 2090 | bpf_reset_run_ctx(old_run_ctx); | 2129 | bpf_reset_run_ctx(old_run_ctx); |
| 2091 | out: | 2130 | out: |
| 2092 | bpf_prog_put_recursion_context(prog); | 2131 | bpf_prog_put_recursion_context(prog); |
| 2093 | rcu_read_unlock_migrate(); | 2132 | |
| 2133 | if (sleepable) { | ||
| 2134 | migrate_enable(); | ||
| 2135 | rcu_read_unlock_tasks_trace(scp); | ||
| 2136 | } else { | ||
| 2137 | rcu_read_unlock_migrate(); | ||
| 2138 | } | ||
| 2094 | } | 2139 | } |
| 2095 | 2140 | ||
| 2096 | #define UNPACK(...) __VA_ARGS__ | 2141 | #define UNPACK(...) __VA_ARGS__ |
| @@ -3170,6 +3215,38 @@ static u64 bpf_uprobe_multi_cookie(struct bpf_run_ctx *ctx) · 1073 unmodified lines | |||
| 3170 | return run_ctx->uprobe->cookie; | 3215 | return run_ctx->uprobe->cookie; |
| 3171 | } | 3216 | } |
| 3172 | 3217 | ||
| 3218 | static int bpf_uprobe_multi_get_path(const union bpf_attr *attr, struct path *path) | ||
| 3219 | { | ||
| 3220 | void __user *upath = u64_to_user_ptr(attr->link_create.uprobe_multi.path); | ||
| 3221 | u32 path_fd = attr->link_create.uprobe_multi.path_fd; | ||
| 3222 | u32 flags = attr->link_create.uprobe_multi.flags; | ||
| 3223 | |||
| 3224 | if (flags & BPF_F_UPROBE_MULTI_PATH_FD) { | ||
| 3225 | /* | ||
| 3226 | * When BPF_F_UPROBE_MULTI_PATH_FD is set, the executable is | ||
| 3227 | * identified by path_fd, upath must be NULL. | ||
| 3228 | */ | ||
| 3229 | if (upath) | ||
| 3230 | return -EINVAL; | ||
| 3231 | |||
| 3232 | CLASS(fd, f)(path_fd); | ||
| 3233 | if (fd_empty(f)) | ||
| 3234 | return -EBADF; | ||
| 3235 | *path = fd_file(f)->f_path; | ||
| 3236 | path_get(path); | ||
| 3237 | return 0; | ||
| 3238 | } | ||
| 3239 | |||
| 3240 | /* | ||
| 3241 | * When BPF_F_UPROBE_MULTI_PATH_FD is not set, the path is resolved | ||
| 3242 | * relative to the cwd (AT_FDCWD) or absolute using the upath string. | ||
| 3243 | */ | ||
| 3244 | if (!upath || path_fd) | ||
| 3245 | return -EINVAL; | ||
| 3246 | |||
| 3247 | return user_path_at(AT_FDCWD, upath, LOOKUP_FOLLOW, path); | ||
| 3248 | } | ||
| 3249 | |||
| 3173 | int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog) | 3250 | int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog) |
| 3174 | { | 3251 | { |
| 3175 | struct bpf_uprobe_multi_link *link = NULL; | 3252 | struct bpf_uprobe_multi_link *link = NULL; |
| @@ -3179,10 +3256,9 @@ int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *pr · 3 unmodified lines | |||
| 3179 | struct task_struct *task = NULL; | 3256 | struct task_struct *task = NULL; |
| 3180 | unsigned long __user *uoffsets; | 3257 | unsigned long __user *uoffsets; |
| 3181 | u64 __user *ucookies; | 3258 | u64 __user *ucookies; |
| 3182 | void __user *upath; | 3259 | unsigned long size; |
| 3183 | u32 flags, cnt, i; | 3260 | u32 flags, cnt, i; |
| 3184 | struct path path; | 3261 | struct path path; |
| 3185 | char *name; | ||
| 3186 | pid_t pid; | 3262 | pid_t pid; |
| 3187 | int err; | 3263 | int err; |
| 3188 | 3264 | ||
| @@ -3197,19 +3273,18 @@ int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *pr · 8 unmodified lines | |||
| 3197 | return -EINVAL; | 3273 | return -EINVAL; |
| 3198 | 3274 | ||
| 3199 | flags = attr->link_create.uprobe_multi.flags; | 3275 | flags = attr->link_create.uprobe_multi.flags; |
| 3200 | if (flags & ~BPF_F_UPROBE_MULTI_RETURN) | 3276 | if (flags & ~(BPF_F_UPROBE_MULTI_RETURN | BPF_F_UPROBE_MULTI_PATH_FD)) |
| 3201 | return -EINVAL; | 3277 | return -EINVAL; |
| 3202 | 3278 | ||
| 3203 | /* | 3279 | /* |
| 3204 | * path, offsets and cnt are mandatory, | 3280 | * offsets and cnt are mandatory, |
| 3205 | * ref_ctr_offsets and cookies are optional | 3281 | * ref_ctr_offsets and cookies are optional |
| 3206 | */ | 3282 | */ |
| 3207 | upath = u64_to_user_ptr(attr->link_create.uprobe_multi.path); | ||
| 3208 | uoffsets = u64_to_user_ptr(attr->link_create.uprobe_multi.offsets); | 3283 | uoffsets = u64_to_user_ptr(attr->link_create.uprobe_multi.offsets); |
| 3209 | cnt = attr->link_create.uprobe_multi.cnt; | 3284 | cnt = attr->link_create.uprobe_multi.cnt; |
| 3210 | pid = attr->link_create.uprobe_multi.pid; | 3285 | pid = attr->link_create.uprobe_multi.pid; |
| 3211 | 3286 | ||
| 3212 | if (!upath || !uoffsets || !cnt || pid < 0) | 3287 | if (!uoffsets || !cnt || pid < 0) |
| 3213 | return -EINVAL; | 3288 | return -EINVAL; |
| 3214 | if (cnt > MAX_UPROBE_MULTI_CNT) | 3289 | if (cnt > MAX_UPROBE_MULTI_CNT) |
| 3215 | return -E2BIG; | 3290 | return -E2BIG; |
| @@ -3217,14 +3292,17 @@ int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *pr · 1 unmodified line | |||
| 3217 | uref_ctr_offsets = u64_to_user_ptr(attr->link_create.uprobe_multi.ref_ctr_offsets); | 3292 | uref_ctr_offsets = u64_to_user_ptr(attr->link_create.uprobe_multi.ref_ctr_offsets); |
| 3218 | ucookies = u64_to_user_ptr(attr->link_create.uprobe_multi.cookies); | 3293 | ucookies = u64_to_user_ptr(attr->link_create.uprobe_multi.cookies); |
| 3219 | 3294 | ||
| 3220 | name = strndup_user(upath, PATH_MAX); | 3295 | /* |
| 3221 | if (IS_ERR(name)) { | 3296 | * All uoffsets/uref_ctr_offsets/ucookies arrays have the same value |
| 3222 | err = PTR_ERR(name); | 3297 | * size, we need to check their address range is safe for __get_user |
| 3223 | return err; | 3298 | * calls. |
| 3224 | } | 3299 | */ |
| 3300 | size = sizeof(*uoffsets) * cnt; | ||
| 3301 | if (!access_ok(uoffsets, size) || !access_ok(uref_ctr_offsets, size) || | ||
| 3302 | !access_ok(ucookies, size)) | ||
| 3303 | return -EFAULT; | ||
| 3225 | 3304 | ||
| 3226 | err = kern_path(name, LOOKUP_FOLLOW, &path); | 3305 | err = bpf_uprobe_multi_get_path(attr, &path); |
| 3227 | kfree(name); | ||
| 3228 | if (err) | 3306 | if (err) |
| 3229 | return err; | 3307 | return err; |
| 3230 | 3308 | ||
| @@ -3398,12 +3476,12 @@ typedef int (*copy_fn_t)(void *dst, const void *src, u32 size, struct task_struc · 167 unmodified lines | |||
| 3398 | * direct calls into all the specific callback implementations | 3476 | * direct calls into all the specific callback implementations |
| 3399 | * (copy_user_data_sleepable, copy_user_data_nofault, and so on) | 3477 | * (copy_user_data_sleepable, copy_user_data_nofault, and so on) |
| 3400 | */ | 3478 | */ |
| 3401 | static __always_inline int __bpf_dynptr_copy_str(struct bpf_dynptr *dptr, u64 doff, u64 size, | 3479 | static __always_inline int __bpf_dynptr_copy_str(const struct bpf_dynptr *dptr, u64 doff, u64 size, |
| 3402 | const void *unsafe_src, | 3480 | const void *unsafe_src, |
| 3403 | copy_fn_t str_copy_fn, | 3481 | copy_fn_t str_copy_fn, |
| 3404 | struct task_struct *tsk) | 3482 | struct task_struct *tsk) |
| 3405 | { | 3483 | { |
| 3406 | struct bpf_dynptr_kern *dst; | 3484 | const struct bpf_dynptr_kern *dst; |
| 3407 | u64 chunk_sz, off; | 3485 | u64 chunk_sz, off; |
| 3408 | void *dst_slice; | 3486 | void *dst_slice; |
| 3409 | int cnt, err; | 3487 | int cnt, err; |
| @@ -3439,7 +3517,7 @@ static __always_inline int __bpf_dynptr_copy(const struct bpf_dynptr *dptr, u64 · 29 unmodified lines | |||
| 3439 | u64 size, const void *unsafe_src, | 3517 | u64 size, const void *unsafe_src, |
| 3440 | copy_fn_t copy_fn, struct task_struct *tsk) | 3518 | copy_fn_t copy_fn, struct task_struct *tsk) |
| 3441 | { | 3519 | { |
| 3442 | struct bpf_dynptr_kern *dst; | 3520 | const struct bpf_dynptr_kern *dst; |
| 3443 | void *dst_slice; | 3521 | void *dst_slice; |
| 3444 | char buf[256]; | 3522 | char buf[256]; |
| 3445 | u64 off, chunk_sz; | 3523 | u64 off, chunk_sz; |
| @@ -3540,49 +3618,49 @@ __bpf_kfunc int bpf_send_signal_task(struct task_struct *task, int sig, enum pid · 94 unmodified lines | |||
| 3540 | return bpf_send_signal_common(sig, type, task, value); | 3618 | return bpf_send_signal_common(sig, type, task, value); |
| 3541 | } | 3619 | } |
| 3542 | 3620 | ||
| 3543 | __bpf_kfunc int bpf_probe_read_user_dynptr(struct bpf_dynptr *dptr, u64 off, | 3621 | __bpf_kfunc int bpf_probe_read_user_dynptr(const struct bpf_dynptr *dptr, u64 off, |
| 3544 | u64 size, const void __user *unsafe_ptr__ign) | 3622 | u64 size, const void __user *unsafe_ptr__ign) |
| 3545 | { | 3623 | { |
| 3546 | return __bpf_dynptr_copy(dptr, off, size, (const void __force *)unsafe_ptr__ign, | 3624 | return __bpf_dynptr_copy(dptr, off, size, (const void __force *)unsafe_ptr__ign, |
| 3547 | copy_user_data_nofault, NULL); | 3625 | copy_user_data_nofault, NULL); |
| 3548 | } | 3626 | } |
| 3549 | 3627 | ||
| 3550 | __bpf_kfunc int bpf_probe_read_kernel_dynptr(struct bpf_dynptr *dptr, u64 off, | 3628 | __bpf_kfunc int bpf_probe_read_kernel_dynptr(const struct bpf_dynptr *dptr, u64 off, |
| 3551 | u64 size, const void *unsafe_ptr__ign) | 3629 | u64 size, const void *unsafe_ptr__ign) |
| 3552 | { | 3630 | { |
| 3553 | return __bpf_dynptr_copy(dptr, off, size, unsafe_ptr__ign, | 3631 | return __bpf_dynptr_copy(dptr, off, size, unsafe_ptr__ign, |
| 3554 | copy_kernel_data_nofault, NULL); | 3632 | copy_kernel_data_nofault, NULL); |
| 3555 | } | 3633 | } |
| 3556 | 3634 | ||
| 3557 | __bpf_kfunc int bpf_probe_read_user_str_dynptr(struct bpf_dynptr *dptr, u64 off, | 3635 | __bpf_kfunc int bpf_probe_read_user_str_dynptr(const struct bpf_dynptr *dptr, u64 off, |
| 3558 | u64 size, const void __user *unsafe_ptr__ign) | 3636 | u64 size, const void __user *unsafe_ptr__ign) |
| 3559 | { | 3637 | { |
| 3560 | return __bpf_dynptr_copy_str(dptr, off, size, (const void __force *)unsafe_ptr__ign, | 3638 | return __bpf_dynptr_copy_str(dptr, off, size, (const void __force *)unsafe_ptr__ign, |
| 3561 | copy_user_str_nofault, NULL); | 3639 | copy_user_str_nofault, NULL); |
| 3562 | } | 3640 | } |
| 3563 | 3641 | ||
| 3564 | __bpf_kfunc int bpf_probe_read_kernel_str_dynptr(struct bpf_dynptr *dptr, u64 off, | 3642 | __bpf_kfunc int bpf_probe_read_kernel_str_dynptr(const struct bpf_dynptr *dptr, u64 off, |
| 3565 | u64 size, const void *unsafe_ptr__ign) | 3643 | u64 size, const void *unsafe_ptr__ign) |
| 3566 | { | 3644 | { |
| 3567 | return __bpf_dynptr_copy_str(dptr, off, size, unsafe_ptr__ign, | 3645 | return __bpf_dynptr_copy_str(dptr, off, size, unsafe_ptr__ign, |
| 3568 | copy_kernel_str_nofault, NULL); | 3646 | copy_kernel_str_nofault, NULL); |
| 3569 | } | 3647 | } |
| 3570 | 3648 | ||
| 3571 | __bpf_kfunc int bpf_copy_from_user_dynptr(struct bpf_dynptr *dptr, u64 off, | 3649 | __bpf_kfunc int bpf_copy_from_user_dynptr(const struct bpf_dynptr *dptr, u64 off, |
| 3572 | u64 size, const void __user *unsafe_ptr__ign) | 3650 | u64 size, const void __user *unsafe_ptr__ign) |
| 3573 | { | 3651 | { |
| 3574 | return __bpf_dynptr_copy(dptr, off, size, (const void __force *)unsafe_ptr__ign, | 3652 | return __bpf_dynptr_copy(dptr, off, size, (const void __force *)unsafe_ptr__ign, |
| 3575 | copy_user_data_sleepable, NULL); | 3653 | copy_user_data_sleepable, NULL); |
| 3576 | } | 3654 | } |
| 3577 | 3655 | ||
| 3578 | __bpf_kfunc int bpf_copy_from_user_str_dynptr(struct bpf_dynptr *dptr, u64 off, | 3656 | __bpf_kfunc int bpf_copy_from_user_str_dynptr(const struct bpf_dynptr *dptr, u64 off, |
| 3579 | u64 size, const void __user *unsafe_ptr__ign) | 3657 | u64 size, const void __user *unsafe_ptr__ign) |
| 3580 | { | 3658 | { |
| 3581 | return __bpf_dynptr_copy_str(dptr, off, size, (const void __force *)unsafe_ptr__ign, | 3659 | return __bpf_dynptr_copy_str(dptr, off, size, (const void __force *)unsafe_ptr__ign, |
| 3582 | copy_user_str_sleepable, NULL); | 3660 | copy_user_str_sleepable, NULL); |
| 3583 | } | 3661 | } |
| 3584 | 3662 | ||
| 3585 | __bpf_kfunc int bpf_copy_from_user_task_dynptr(struct bpf_dynptr *dptr, u64 off, | 3663 | __bpf_kfunc int bpf_copy_from_user_task_dynptr(const struct bpf_dynptr *dptr, u64 off, |
| 3586 | u64 size, const void __user *unsafe_ptr__ign, | 3664 | u64 size, const void __user *unsafe_ptr__ign, |
| 3587 | struct task_struct *tsk) | 3665 | struct task_struct *tsk) |
| 3588 | { | 3666 | { |
| @@ -3590,7 +3668,7 @@ __bpf_kfunc int bpf_copy_from_user_task_dynptr(struct bpf_dynptr *dptr, u64 off, · 1 unmodified line | |||
| 3590 | copy_user_data_sleepable, tsk); | 3668 | copy_user_data_sleepable, tsk); |
| 3591 | } | 3669 | } |
| 3592 | 3670 | ||
| 3593 | __bpf_kfunc int bpf_copy_from_user_task_str_dynptr(struct bpf_dynptr *dptr, u64 off, | 3671 | __bpf_kfunc int bpf_copy_from_user_task_str_dynptr(const struct bpf_dynptr *dptr, u64 off, |
| 3594 | u64 size, const void __user *unsafe_ptr__ign, | 3672 | u64 size, const void __user *unsafe_ptr__ign, |
| 3595 | struct task_struct *tsk) | 3673 | struct task_struct *tsk) |
| 3596 | { | 3674 | { |
| @@ -3599,3 +3677,203 @@ __bpf_kfunc int bpf_copy_from_user_task_str_dynptr(struct bpf_dynptr *dptr, u64 · 2 unmodified lines | |||
| 3599 | } | 3677 | } |
| 3600 | 3678 | ||
| 3601 | __bpf_kfunc_end_defs(); | 3679 | __bpf_kfunc_end_defs(); |
| 3680 | |||
| 3681 | #if defined(CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS) && \ | ||
| 3682 | defined(CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS) | ||
| 3683 | |||
| 3684 | static void bpf_tracing_multi_link_release(struct bpf_link *link) | ||
| 3685 | { | ||
| 3686 | struct bpf_tracing_multi_link *tr_link = | ||
| 3687 | container_of(link, struct bpf_tracing_multi_link, link); | ||
| 3688 | |||
| 3689 | WARN_ON_ONCE(bpf_trampoline_multi_detach(link->prog, tr_link)); | ||
| 3690 | } | ||
| 3691 | |||
| 3692 | static void bpf_tracing_multi_link_dealloc(struct bpf_link *link) | ||
| 3693 | { | ||
| 3694 | struct bpf_tracing_multi_link *tr_link = | ||
| 3695 | container_of(link, struct bpf_tracing_multi_link, link); | ||
| 3696 | |||
| 3697 | kvfree(tr_link->fexits); | ||
| 3698 | kvfree(tr_link->cookies); | ||
| 3699 | kvfree(tr_link); | ||
| 3700 | } | ||
| 3701 | |||
| 3702 | #ifdef CONFIG_PROC_FS | ||
| 3703 | static void bpf_tracing_multi_show_fdinfo(const struct bpf_link *link, | ||
| 3704 | struct seq_file *seq) | ||
| 3705 | { | ||
| 3706 | struct bpf_tracing_multi_link *tr_link = | ||
| 3707 | container_of(link, struct bpf_tracing_multi_link, link); | ||
| 3708 | bool has_cookies = !!tr_link->cookies; | ||
| 3709 | |||
| 3710 | seq_printf(seq, "attach_type:\t%u\n", tr_link->link.attach_type); | ||
| 3711 | seq_printf(seq, "cnt:\t%u\n", tr_link->nodes_cnt); | ||
| 3712 | |||
| 3713 | seq_printf(seq, "%s\t %s\t %s\t %s\n", "obj-id", "btf-id", "cookie", "func"); | ||
| 3714 | for (int i = 0; i < tr_link->nodes_cnt; i++) { | ||
| 3715 | struct bpf_tracing_multi_node *mnode = &tr_link->nodes[i]; | ||
| 3716 | u32 btf_id, obj_id; | ||
| 3717 | |||
| 3718 | bpf_trampoline_unpack_key(mnode->trampoline->key, &obj_id, &btf_id); | ||
| 3719 | seq_printf(seq, "%u\t %u\t %llu\t %pS\n", | ||
| 3720 | obj_id, btf_id, | ||
| 3721 | has_cookies ? tr_link->cookies[i] : 0, | ||
| 3722 | (void *) mnode->trampoline->ip); | ||
| 3723 | |||
| 3724 | cond_resched(); | ||
| 3725 | } | ||
| 3726 | } | ||
| 3727 | #endif | ||
| 3728 | |||
| 3729 | static const struct bpf_link_ops bpf_tracing_multi_link_lops = { | ||
| 3730 | .release = bpf_tracing_multi_link_release, | ||
| 3731 | .dealloc_deferred = bpf_tracing_multi_link_dealloc, | ||
| 3732 | #ifdef CONFIG_PROC_FS | ||
| 3733 | .show_fdinfo = bpf_tracing_multi_show_fdinfo, | ||
| 3734 | #endif | ||
| 3735 | }; | ||
| 3736 | |||
| 3737 | static int ids_cmp_r(const void *pa, const void *pb, const void *priv __maybe_unused) | ||
| 3738 | { | ||
| 3739 | u32 a = *(u32 *) pa; | ||
| 3740 | u32 b = *(u32 *) pb; | ||
| 3741 | |||
| 3742 | return (a > b) - (a < b); | ||
| 3743 | } | ||
| 3744 | |||
| 3745 | static void ids_swap_r(void *a, void *b, int size __maybe_unused, | ||
| 3746 | const void *priv __maybe_unused) | ||
| 3747 | { | ||
| 3748 | u64 *cookie_a, *cookie_b, *cookies; | ||
| 3749 | u32 *id_a = a, *id_b = b, *ids; | ||
| 3750 | void **data = (void **) priv; | ||
| 3751 | |||
| 3752 | ids = data[0]; | ||
| 3753 | cookies = data[1]; | ||
| 3754 | |||
| 3755 | if (cookies) { | ||
| 3756 | cookie_a = cookies + (id_a - ids); | ||
| 3757 | cookie_b = cookies + (id_b - ids); | ||
| 3758 | swap(*cookie_a, *cookie_b); | ||
| 3759 | } | ||
| 3760 | swap(*id_a, *id_b); | ||
| 3761 | } | ||
| 3762 | |||
| 3763 | static int check_dup_ids(u32 *ids, u64 *cookies, u32 cnt) | ||
| 3764 | { | ||
| 3765 | void *data[2] = { ids, cookies }; | ||
| 3766 | int err = 0; | ||
| 3767 | |||
| 3768 | /* | ||
| 3769 | * Sort ids array (together with cookies array if defined) | ||
| 3770 | * and check it for duplicates. The ids and cookies arrays | ||
| 3771 | * are left sorted. | ||
| 3772 | */ | ||
| 3773 | sort_r_nonatomic(ids, cnt, sizeof(ids[0]), ids_cmp_r, ids_swap_r, data); | ||
| 3774 | |||
| 3775 | for (int i = 1; i < cnt; i++) { | ||
| 3776 | if (ids[i] == ids[i - 1]) { | ||
| 3777 | err = -EINVAL; | ||
| 3778 | break; | ||
| 3779 | } | ||
| 3780 | } | ||
| 3781 | return err; | ||
| 3782 | } | ||
| 3783 | |||
| 3784 | int bpf_tracing_multi_attach(struct bpf_prog *prog, const union bpf_attr *attr) | ||
| 3785 | { | ||
| 3786 | struct bpf_tracing_multi_link *link = NULL; | ||
| 3787 | struct bpf_tramp_node *fexits = NULL; | ||
| 3788 | struct bpf_link_primer link_primer; | ||
| 3789 | u32 cnt, *ids = NULL; | ||
| 3790 | u64 __user *ucookies; | ||
| 3791 | u64 *cookies = NULL; | ||
| 3792 | u32 __user *uids; | ||
| 3793 | int err; | ||
| 3794 | |||
| 3795 | uids = u64_to_user_ptr(attr->link_create.tracing_multi.ids); | ||
| 3796 | cnt = attr->link_create.tracing_multi.cnt; | ||
| 3797 | |||
| 3798 | if (!cnt || !uids) | ||
| 3799 | return -EINVAL; | ||
| 3800 | if (cnt > MAX_TRACING_MULTI_CNT) | ||
| 3801 | return -E2BIG; | ||
| 3802 | if (attr->link_create.flags || attr->link_create.target_fd) | ||
| 3803 | return -EINVAL; | ||
| 3804 | |||
| 3805 | ids = kvmalloc_objs(*ids, cnt); | ||
| 3806 | if (!ids) | ||
| 3807 | return -ENOMEM; | ||
| 3808 | |||
| 3809 | if (copy_from_user(ids, uids, cnt * sizeof(*ids))) { | ||
| 3810 | err = -EFAULT; | ||
| 3811 | goto error; | ||
| 3812 | } | ||
| 3813 | |||
| 3814 | ucookies = u64_to_user_ptr(attr->link_create.tracing_multi.cookies); | ||
| 3815 | if (ucookies) { | ||
| 3816 | cookies = kvmalloc_objs(*cookies, cnt); | ||
| 3817 | if (!cookies) { | ||
| 3818 | err = -ENOMEM; | ||
| 3819 | goto error; | ||
| 3820 | } | ||
| 3821 | if (copy_from_user(cookies, ucookies, cnt * sizeof(*cookies))) { | ||
| 3822 | err = -EFAULT; | ||
| 3823 | goto error; | ||
| 3824 | } | ||
| 3825 | } | ||
| 3826 | |||
| 3827 | err = check_dup_ids(ids, cookies, cnt); | ||
| 3828 | if (err) | ||
| 3829 | goto error; | ||
| 3830 | |||
| 3831 | if (prog->expected_attach_type == BPF_TRACE_FSESSION_MULTI) { | ||
| 3832 | fexits = kvmalloc_objs(*fexits, cnt); | ||
| 3833 | if (!fexits) { | ||
| 3834 | err = -ENOMEM; | ||
| 3835 | goto error; | ||
| 3836 | } | ||
| 3837 | } | ||
| 3838 | |||
| 3839 | link = kvzalloc_flex(*link, nodes, cnt); | ||
| 3840 | if (!link) { | ||
| 3841 | err = -ENOMEM; | ||
| 3842 | goto error; | ||
| 3843 | } | ||
| 3844 | |||
| 3845 | bpf_link_init(&link->link, BPF_LINK_TYPE_TRACING_MULTI, | ||
| 3846 | &bpf_tracing_multi_link_lops, prog, prog->expected_attach_type); | ||
| 3847 | |||
| 3848 | err = bpf_link_prime(&link->link, &link_primer); | ||
| 3849 | if (err) | ||
| 3850 | goto error; | ||
| 3851 | |||
| 3852 | link->nodes_cnt = cnt; | ||
| 3853 | link->cookies = cookies; | ||
| 3854 | link->fexits = fexits; | ||
| 3855 | |||
| 3856 | err = bpf_trampoline_multi_attach(prog, ids, link); | ||
| 3857 | kvfree(ids); | ||
| 3858 | if (err) { | ||
| 3859 | bpf_link_cleanup(&link_primer); | ||
| 3860 | return err; | ||
| 3861 | } | ||
| 3862 | return bpf_link_settle(&link_primer); | ||
| 3863 | |||
| 3864 | error: | ||
| 3865 | kvfree(fexits); | ||
| 3866 | kvfree(cookies); | ||
| 3867 | kvfree(ids); | ||
| 3868 | kvfree(link); | ||
| 3869 | return err; | ||
| 3870 | } | ||
| 3871 | |||
| 3872 | #else | ||
| 3873 | |||
| 3874 | int bpf_tracing_multi_attach(struct bpf_prog *prog, const union bpf_attr *attr) | ||
| 3875 | { | ||
| 3876 | return -EOPNOTSUPP; | ||
| 3877 | } | ||
| 3878 | |||
| 3879 | #endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS && CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS */ | ||
| @@ -1198,8 +1198,7 @@ ftrace_lookup_ip(struct ftrace_hash *hash, unsigned long ip) · 1197 unmodified lines | |||
| 1198 | return __ftrace_lookup_ip(hash, ip); | 1198 | return __ftrace_lookup_ip(hash, ip); |
| 1199 | } | 1199 | } |
| 1200 | 1200 | ||
| 1201 | static void __add_hash_entry(struct ftrace_hash *hash, | 1201 | void add_ftrace_hash_entry(struct ftrace_hash *hash, struct ftrace_func_entry *entry) |
| 1202 | struct ftrace_func_entry *entry) | ||
| 1203 | { | 1202 | { |
| 1204 | struct hlist_head *hhd; | 1203 | struct hlist_head *hhd; |
| 1205 | unsigned long key; | 1204 | unsigned long key; |
| @@ -1221,7 +1220,7 @@ add_ftrace_hash_entry_direct(struct ftrace_hash *hash, unsigned long ip, unsigne · 15 unmodified lines | |||
| 1221 | 1220 | ||
| 1222 | entry->ip = ip; | 1221 | entry->ip = ip; |
| 1223 | entry->direct = direct; | 1222 | entry->direct = direct; |
| 1224 | __add_hash_entry(hash, entry); | 1223 | add_ftrace_hash_entry(hash, entry); |
| 1225 | 1224 | ||
| 1226 | return entry; | 1225 | return entry; |
| 1227 | } | 1226 | } |
| @@ -1249,6 +1248,25 @@ remove_hash_entry(struct ftrace_hash *hash, · 21 unmodified lines | |||
| 1249 | hash->count--; | 1248 | hash->count--; |
| 1250 | } | 1249 | } |
| 1251 | 1250 | ||
| 1251 | void ftrace_hash_remove(struct ftrace_hash *hash) | ||
| 1252 | { | ||
| 1253 | struct ftrace_func_entry *entry; | ||
| 1254 | struct hlist_head *hhd; | ||
| 1255 | struct hlist_node *tn; | ||
| 1256 | int size; | ||
| 1257 | int i; | ||
| 1258 | |||
| 1259 | if (!hash || !hash->count) | ||
| 1260 | return; | ||
| 1261 | size = 1 << hash->size_bits; | ||
| 1262 | for (i = 0; i < size; i++) { | ||
| 1263 | hhd = &hash->buckets[i]; | ||
| 1264 | hlist_for_each_entry_safe(entry, tn, hhd, hlist) | ||
| 1265 | remove_hash_entry(hash, entry); | ||
| 1266 | } | ||
| 1267 | FTRACE_WARN_ON(hash->count); | ||
| 1268 | } | ||
| 1269 | |||
| 1252 | static void ftrace_hash_clear(struct ftrace_hash *hash) | 1270 | static void ftrace_hash_clear(struct ftrace_hash *hash) |
| 1253 | { | 1271 | { |
| 1254 | struct hlist_head *hhd; | 1272 | struct hlist_head *hhd; |
| @@ -1458,7 +1476,7 @@ static struct ftrace_hash *__move_hash(struct ftrace_hash *src, int size) · 203 unmodified lines | |||
| 1458 | hhd = &src->buckets[i]; | 1476 | hhd = &src->buckets[i]; |
| 1459 | hlist_for_each_entry_safe(entry, tn, hhd, hlist) { | 1477 | hlist_for_each_entry_safe(entry, tn, hhd, hlist) { |
| 1460 | remove_hash_entry(src, entry); | 1478 | remove_hash_entry(src, entry); |
| 1461 | __add_hash_entry(new_hash, entry); | 1479 | add_ftrace_hash_entry(new_hash, entry); |
| 1462 | } | 1480 | } |
| 1463 | } | 1481 | } |
| 1464 | return new_hash; | 1482 | return new_hash; |
| @@ -5341,7 +5359,7 @@ int ftrace_func_mapper_add_ip(struct ftrace_func_mapper *mapper, · 3876 unmodified lines | |||
| 5341 | map->entry.ip = ip; | 5359 | map->entry.ip = ip; |
| 5342 | map->data = data; | 5360 | map->data = data; |
| 5343 | 5361 | ||
| 5344 | __add_hash_entry(&mapper->hash, &map->entry); | 5362 | add_ftrace_hash_entry(&mapper->hash, &map->entry); |
| 5345 | 5363 | ||
| 5346 | return 0; | 5364 | return 0; |
| 5347 | } | 5365 | } |
| @@ -6288,11 +6306,16 @@ int modify_ftrace_direct(struct ftrace_ops *ops, unsigned long addr) · 940 unmodified lines | |||
| 6288 | } | 6306 | } |
| 6289 | EXPORT_SYMBOL_GPL(modify_ftrace_direct); | 6307 | EXPORT_SYMBOL_GPL(modify_ftrace_direct); |
| 6290 | 6308 | ||
| 6291 | static unsigned long hash_count(struct ftrace_hash *hash) | 6309 | static inline unsigned long hash_count(struct ftrace_hash *hash) |
| 6292 | { | 6310 | { |
| 6293 | return hash ? hash->count : 0; | 6311 | return hash ? hash->count : 0; |
| 6294 | } | 6312 | } |
| 6295 | 6313 | ||
| 6314 | unsigned long ftrace_hash_count(struct ftrace_hash *hash) | ||
| 6315 | { | ||
| 6316 | return hash_count(hash); | ||
| 6317 | } | ||
| 6318 | |||
| 6296 | /** | 6319 | /** |
| 6297 | * hash_add - adds two struct ftrace_hash and returns the result | 6320 | * hash_add - adds two struct ftrace_hash and returns the result |
| 6298 | * @a: struct ftrace_hash object | 6321 | * @a: struct ftrace_hash object |
| @@ -1371,33 +1371,33 @@ static DECLARE_BITMAP(enabled_perf_exit_syscalls, NR_syscalls); · 1370 unmodified lines | |||
| 1371 | static int sys_perf_refcount_enter; | 1371 | static int sys_perf_refcount_enter; |
| 1372 | static int sys_perf_refcount_exit; | 1372 | static int sys_perf_refcount_exit; |
| 1373 | 1373 | ||
| 1374 | static int perf_call_bpf_enter(struct trace_event_call *call, struct pt_regs *regs, | 1374 | static int perf_call_bpf_enter(struct trace_event_call *call, |
| 1375 | struct syscall_metadata *sys_data, | 1375 | struct syscall_metadata *sys_data, |
| 1376 | struct syscall_trace_enter *rec) | 1376 | int syscall_nr, unsigned long *args) |
| 1377 | { | 1377 | { |
| 1378 | struct syscall_tp_t { | 1378 | struct syscall_tp_t { |
| 1379 | struct trace_entry ent; | 1379 | struct trace_entry ent; |
| 1380 | int syscall_nr; | 1380 | int syscall_nr; |
| 1381 | unsigned long args[SYSCALL_DEFINE_MAXARGS]; | 1381 | unsigned long args[SYSCALL_DEFINE_MAXARGS]; |
| 1382 | } __aligned(8) param; | 1382 | } __aligned(8) param; |
| 1383 | struct pt_regs regs = {}; | ||
| 1383 | int i; | 1384 | int i; |
| 1384 | 1385 | ||
| 1385 | BUILD_BUG_ON(sizeof(param.ent) < sizeof(void *)); | 1386 | BUILD_BUG_ON(sizeof(param.ent) < sizeof(void *)); |
| 1386 | 1387 | ||
| 1387 | /* bpf prog requires 'regs' to be the first member in the ctx (a.k.a. ¶m) */ | 1388 | /* bpf prog requires 'regs' to be the first member in the ctx */ |
| 1388 | perf_fetch_caller_regs(regs); | 1389 | perf_fetch_caller_regs(®s); |
| 1389 | *(struct pt_regs **)¶m = regs; | 1390 | *(struct pt_regs **)¶m = ®s; |
| 1390 | param.syscall_nr = rec->nr; | 1391 | param.syscall_nr = syscall_nr; |
| 1391 | for (i = 0; i < sys_data->nb_args; i++) | 1392 | for (i = 0; i < sys_data->nb_args; i++) |
| 1392 | param.args[i] = rec->args[i]; | 1393 | param.args[i] = args[i]; |
| 1393 | return trace_call_bpf(call, ¶m); | 1394 | return trace_call_bpf_faultable(call, ¶m); |
| 1394 | } | 1395 | } |
| 1395 | 1396 | ||
| 1396 | static void perf_syscall_enter(void *ignore, struct pt_regs *regs, long id) | 1397 | static void perf_syscall_enter(void *ignore, struct pt_regs *regs, long id) |
| 1397 | { | 1398 | { |
| 1398 | struct syscall_metadata *sys_data; | 1399 | struct syscall_metadata *sys_data; |
| 1399 | struct syscall_trace_enter *rec; | 1400 | struct syscall_trace_enter *rec; |
| 1400 | struct pt_regs *fake_regs; | ||
| 1401 | struct hlist_head *head; | 1401 | struct hlist_head *head; |
| 1402 | unsigned long args[6]; | 1402 | unsigned long args[6]; |
| 1403 | bool valid_prog_array; | 1403 | bool valid_prog_array; |
| @@ -1410,12 +1410,7 @@ static void perf_syscall_enter(void *ignore, struct pt_regs *regs, long id) · 6 unmodified lines | |||
| 1410 | int size = 0; | 1410 | int size = 0; |
| 1411 | int uargs = 0; | 1411 | int uargs = 0; |
| 1412 | 1412 | ||
| 1413 | /* | ||
| 1414 | * Syscall probe called with preemption enabled, but the ring | ||
| 1415 | * buffer and per-cpu data require preemption to be disabled. | ||
| 1416 | */ | ||
| 1417 | might_fault(); | 1413 | might_fault(); |
| 1418 | guard(preempt_notrace)(); | ||
| 1419 | 1414 | ||
| 1420 | syscall_nr = trace_get_syscall_nr(current, regs); | 1415 | syscall_nr = trace_get_syscall_nr(current, regs); |
| 1421 | if (syscall_nr < 0 || syscall_nr >= NR_syscalls) | 1416 | if (syscall_nr < 0 || syscall_nr >= NR_syscalls) |
| @@ -1429,6 +1424,26 @@ static void perf_syscall_enter(void *ignore, struct pt_regs *regs, long id) · 7 unmodified lines | |||
| 1429 | 1424 | ||
| 1430 | syscall_get_arguments(current, regs, args); | 1425 | syscall_get_arguments(current, regs, args); |
| 1431 | 1426 | ||
| 1427 | /* | ||
| 1428 | * Run BPF program in faultable context before per-cpu buffer | ||
| 1429 | * allocation, allowing sleepable BPF programs to execute. | ||
| 1430 | */ | ||
| 1431 | valid_prog_array = bpf_prog_array_valid(sys_data->enter_event); | ||
| 1432 | if (valid_prog_array && | ||
| 1433 | !perf_call_bpf_enter(sys_data->enter_event, sys_data, | ||
| 1434 | syscall_nr, args)) | ||
| 1435 | return; | ||
| 1436 | |||
| 1437 | /* | ||
| 1438 | * Per-cpu ring buffer and perf event list operations require | ||
| 1439 | * preemption to be disabled. | ||
| 1440 | */ | ||
| 1441 | guard(preempt_notrace)(); | ||
| 1442 | |||
| 1443 | head = this_cpu_ptr(sys_data->enter_event->perf_events); | ||
| 1444 | if (hlist_empty(head)) | ||
| 1445 | return; | ||
| 1446 | |||
| 1432 | /* Check if this syscall event faults in user space memory */ | 1447 | /* Check if this syscall event faults in user space memory */ |
| 1433 | mayfault = sys_data->user_mask != 0; | 1448 | mayfault = sys_data->user_mask != 0; |
| 1434 | 1449 | ||
| @@ -1438,17 +1453,12 @@ static void perf_syscall_enter(void *ignore, struct pt_regs *regs, long id) · 3 unmodified lines | |||
| 1438 | return; | 1453 | return; |
| 1439 | } | 1454 | } |
| 1440 | 1455 | ||
| 1441 | head = this_cpu_ptr(sys_data->enter_event->perf_events); | ||
| 1442 | valid_prog_array = bpf_prog_array_valid(sys_data->enter_event); | ||
| 1443 | if (!valid_prog_array && hlist_empty(head)) | ||
| 1444 | return; | ||
| 1445 | |||
| 1446 | /* get the size after alignment with the u32 buffer size field */ | 1456 | /* get the size after alignment with the u32 buffer size field */ |
| 1447 | size += sizeof(unsigned long) * sys_data->nb_args + sizeof(*rec); | 1457 | size += sizeof(unsigned long) * sys_data->nb_args + sizeof(*rec); |
| 1448 | size = ALIGN(size + sizeof(u32), sizeof(u64)); | 1458 | size = ALIGN(size + sizeof(u32), sizeof(u64)); |
| 1449 | size -= sizeof(u32); | 1459 | size -= sizeof(u32); |
| 1450 | 1460 | ||
| 1451 | rec = perf_trace_buf_alloc(size, &fake_regs, &rctx); | 1461 | rec = perf_trace_buf_alloc(size, NULL, &rctx); |
| 1452 | if (!rec) | 1462 | if (!rec) |
| 1453 | return; | 1463 | return; |
| 1454 | 1464 | ||
| @@ -1458,13 +1468,6 @@ static void perf_syscall_enter(void *ignore, struct pt_regs *regs, long id) · 3 unmodified lines | |||
| 1458 | if (mayfault) | 1468 | if (mayfault) |
| 1459 | syscall_put_data(sys_data, rec, user_ptr, size, user_sizes, uargs); | 1469 | syscall_put_data(sys_data, rec, user_ptr, size, user_sizes, uargs); |
| 1460 | 1470 | ||
| 1461 | if ((valid_prog_array && | ||
| 1462 | !perf_call_bpf_enter(sys_data->enter_event, fake_regs, sys_data, rec)) || | ||
| 1463 | hlist_empty(head)) { | ||
| 1464 | perf_swevent_put_recursion_context(rctx); | ||
| 1465 | return; | ||
| 1466 | } | ||
| 1467 | |||
| 1468 | perf_trace_buf_submit(rec, size, rctx, | 1471 | perf_trace_buf_submit(rec, size, rctx, |
| 1469 | sys_data->enter_event->event.type, 1, regs, | 1472 | sys_data->enter_event->event.type, 1, regs, |
| 1470 | head, NULL); | 1473 | head, NULL); |
| @@ -1514,40 +1517,35 @@ static void perf_sysenter_disable(struct trace_event_call *call) · 43 unmodified lines | |||
| 1514 | syscall_fault_buffer_disable(); | 1517 | syscall_fault_buffer_disable(); |
| 1515 | } | 1518 | } |
| 1516 | 1519 | ||
| 1517 | static int perf_call_bpf_exit(struct trace_event_call *call, struct pt_regs *regs, | 1520 | static int perf_call_bpf_exit(struct trace_event_call *call, |
| 1518 | struct syscall_trace_exit *rec) | 1521 | int syscall_nr, long ret_val) |
| 1519 | { | 1522 | { |
| 1520 | struct syscall_tp_t { | 1523 | struct syscall_tp_t { |
| 1521 | struct trace_entry ent; | 1524 | struct trace_entry ent; |
| 1522 | int syscall_nr; | 1525 | int syscall_nr; |
| 1523 | unsigned long ret; | 1526 | unsigned long ret; |
| 1524 | } __aligned(8) param; | 1527 | } __aligned(8) param; |
| 1525 | 1528 | struct pt_regs regs = {}; | |
| 1526 | /* bpf prog requires 'regs' to be the first member in the ctx (a.k.a. ¶m) */ | 1529 | |
| 1527 | perf_fetch_caller_regs(regs); | 1530 | /* bpf prog requires 'regs' to be the first member in the ctx */ |
| 1528 | *(struct pt_regs **)¶m = regs; | 1531 | perf_fetch_caller_regs(®s); |
| 1529 | param.syscall_nr = rec->nr; | 1532 | *(struct pt_regs **)¶m = ®s; |
| 1530 | param.ret = rec->ret; | 1533 | param.syscall_nr = syscall_nr; |
| 1531 | return trace_call_bpf(call, ¶m); | 1534 | param.ret = ret_val; |
| 1535 | return trace_call_bpf_faultable(call, ¶m); | ||
| 1532 | } | 1536 | } |
| 1533 | 1537 | ||
| 1534 | static void perf_syscall_exit(void *ignore, struct pt_regs *regs, long ret) | 1538 | static void perf_syscall_exit(void *ignore, struct pt_regs *regs, long ret) |
| 1535 | { | 1539 | { |
| 1536 | struct syscall_metadata *sys_data; | 1540 | struct syscall_metadata *sys_data; |
| 1537 | struct syscall_trace_exit *rec; | 1541 | struct syscall_trace_exit *rec; |
| 1538 | struct pt_regs *fake_regs; | ||
| 1539 | struct hlist_head *head; | 1542 | struct hlist_head *head; |
| 1540 | bool valid_prog_array; | 1543 | bool valid_prog_array; |
| 1541 | int syscall_nr; | 1544 | int syscall_nr; |
| 1542 | int rctx; | 1545 | int rctx; |
| 1543 | int size; | 1546 | int size; |
| 1544 | 1547 | ||
| 1545 | /* | ||
| 1546 | * Syscall probe called with preemption enabled, but the ring | ||
| 1547 | * buffer and per-cpu data require preemption to be disabled. | ||
| 1548 | */ | ||
| 1549 | might_fault(); | 1548 | might_fault(); |
| 1550 | guard(preempt_notrace)(); | ||
| 1551 | 1549 | ||
| 1552 | syscall_nr = trace_get_syscall_nr(current, regs); | 1550 | syscall_nr = trace_get_syscall_nr(current, regs); |
| 1553 | if (syscall_nr < 0 || syscall_nr >= NR_syscalls) | 1551 | if (syscall_nr < 0 || syscall_nr >= NR_syscalls) |
| @@ -1559,29 +1557,37 @@ static void perf_syscall_exit(void *ignore, struct pt_regs *regs, long ret) · 5 unmodified lines | |||
| 1559 | if (!sys_data) | 1557 | if (!sys_data) |
| 1560 | return; | 1558 | return; |
| 1561 | 1559 | ||
| 1562 | head = this_cpu_ptr(sys_data->exit_event->perf_events); | 1560 | /* |
| 1561 | * Run BPF program in faultable context before per-cpu buffer | ||
| 1562 | * allocation, allowing sleepable BPF programs to execute. | ||
| 1563 | */ | ||
| 1563 | valid_prog_array = bpf_prog_array_valid(sys_data->exit_event); | 1564 | valid_prog_array = bpf_prog_array_valid(sys_data->exit_event); |
| 1564 | if (!valid_prog_array && hlist_empty(head)) | 1565 | if (valid_prog_array && |
| 1566 | !perf_call_bpf_exit(sys_data->exit_event, syscall_nr, | ||
| 1567 | syscall_get_return_value(current, regs))) | ||
| 1568 | return; | ||
| 1569 | |||
| 1570 | /* | ||
| 1571 | * Per-cpu ring buffer and perf event list operations require | ||
| 1572 | * preemption to be disabled. | ||
| 1573 | */ | ||
| 1574 | guard(preempt_notrace)(); | ||
| 1575 | |||
| 1576 | head = this_cpu_ptr(sys_data->exit_event->perf_events); | ||
| 1577 | if (hlist_empty(head)) | ||
| 1565 | return; | 1578 | return; |
| 1566 | 1579 | ||
| 1567 | /* We can probably do that at build time */ | 1580 | /* We can probably do that at build time */ |
| 1568 | size = ALIGN(sizeof(*rec) + sizeof(u32), sizeof(u64)); | 1581 | size = ALIGN(sizeof(*rec) + sizeof(u32), sizeof(u64)); |
| 1569 | size -= sizeof(u32); | 1582 | size -= sizeof(u32); |
| 1570 | 1583 | ||
| 1571 | rec = perf_trace_buf_alloc(size, &fake_regs, &rctx); | 1584 | rec = perf_trace_buf_alloc(size, NULL, &rctx); |
| 1572 | if (!rec) | 1585 | if (!rec) |
| 1573 | return; | 1586 | return; |
| 1574 | 1587 | ||
| 1575 | rec->nr = syscall_nr; | 1588 | rec->nr = syscall_nr; |
| 1576 | rec->ret = syscall_get_return_value(current, regs); | 1589 | rec->ret = syscall_get_return_value(current, regs); |
| 1577 | 1590 | ||
| 1578 | if ((valid_prog_array && | ||
| 1579 | !perf_call_bpf_exit(sys_data->exit_event, fake_regs, rec)) || | ||
| 1580 | hlist_empty(head)) { | ||
| 1581 | perf_swevent_put_recursion_context(rctx); | ||
| 1582 | return; | ||
| 1583 | } | ||
| 1584 | |||
| 1585 | perf_trace_buf_submit(rec, size, rctx, sys_data->exit_event->event.type, | 1591 | perf_trace_buf_submit(rec, size, rctx, sys_data->exit_event->event.type, |
| 1586 | 1, regs, head, NULL); | 1592 | 1, regs, head, NULL); |
| 1587 | } | 1593 | } |
| @@ -687,6 +687,108 @@ void *rhashtable_insert_slow(struct rhashtable *ht, const void *key, · 686 unmodified lines | |||
| 687 | } | 687 | } |
| 688 | EXPORT_SYMBOL_GPL(rhashtable_insert_slow); | 688 | EXPORT_SYMBOL_GPL(rhashtable_insert_slow); |
| 689 | 689 | ||
| 690 | /* Scan one element forward from prev_key's position in @tbl. | ||
| 691 | * Returns first rhash_head whose bucket > prev_key's bucket, or the | ||
| 692 | * element immediately after prev_key inside prev_key's bucket. | ||
| 693 | * Returns the first element if prev_key is NULL, NULL when @tbl is | ||
| 694 | * exhausted, or ERR_PTR(-ENOENT) if prev_key is not found in @tbl. | ||
| 695 | */ | ||
| 696 | static struct rhash_head *__rhashtable_next_in_table( | ||
| 697 | struct rhashtable *ht, struct bucket_table *tbl, | ||
| 698 | const void *prev_key) | ||
| 699 | { | ||
| 700 | struct rhashtable_compare_arg arg = { .ht = ht, .key = prev_key }; | ||
| 701 | const struct rhashtable_params params = ht->p; | ||
| 702 | struct rhash_head *he; | ||
| 703 | unsigned int b = 0; | ||
| 704 | bool found = false; | ||
| 705 | |||
| 706 | if (prev_key) { | ||
| 707 | b = rht_key_hashfn(ht, tbl, prev_key, params); | ||
| 708 | rht_for_each_rcu(he, tbl, b) { | ||
| 709 | bool match = params.obj_cmpfn | ||
| 710 | ? !params.obj_cmpfn(&arg, rht_obj(ht, he)) | ||
| 711 | : !rhashtable_compare(&arg, rht_obj(ht, he)); | ||
| 712 | if (found) { | ||
| 713 | if (match) | ||
| 714 | continue; | ||
| 715 | return he; | ||
| 716 | } | ||
| 717 | if (match) | ||
| 718 | found = true; | ||
| 719 | } | ||
| 720 | if (!found) | ||
| 721 | return ERR_PTR(-ENOENT); | ||
| 722 | b++; | ||
| 723 | } | ||
| 724 | |||
| 725 | for (; b < tbl->size; b++) | ||
| 726 | rht_for_each_rcu(he, tbl, b) | ||
| 727 | return he; | ||
| 728 | return NULL; | ||
| 729 | } | ||
| 730 | |||
| 731 | /** | ||
| 732 | * rhashtable_next_key - return next element after a given key | ||
| 733 | * @ht: hash table | ||
| 734 | * @prev_key: pointer to previous key, or NULL for the first element | ||
| 735 | * | ||
| 736 | * WARNING: this walk is highly unstable. Unlike rhashtable_walk_*(), | ||
| 737 | * it cannot detect a concurrent resize or rehash, so a full iteration | ||
| 738 | * is NOT guaranteed to terminate under adversarial or sustained | ||
| 739 | * rehashing. Callers MUST tolerate skipped and duplicated elements and | ||
| 740 | * SHOULD bound their loop externally. | ||
| 741 | * | ||
| 742 | * Returns the next element in best-effort iteration order, walking the | ||
| 743 | * @tbl chain (including any future_tbl in flight). Caller must hold RCU. | ||
| 744 | * | ||
| 745 | * Pass @prev_key == NULL to obtain the first element. To iterate, set | ||
| 746 | * @prev_key to the key of the previously returned element on each call, | ||
| 747 | * and stop when NULL is returned. | ||
| 748 | * | ||
| 749 | * Best-effort semantics: | ||
| 750 | * - Across the tbl->future_tbl chain, an element being migrated may | ||
| 751 | * transiently appear in both tables and be observed twice. | ||
| 752 | * - Concurrent inserts may or may not be observed. | ||
| 753 | * - Termination of a full iteration loop is NOT guaranteed under | ||
| 754 | * adversarial continuous rehash; callers MUST tolerate skips and | ||
| 755 | * repeats and SHOULD bound their loop externally. | ||
| 756 | * - Behavior on tables that contain duplicate keys is undefined: | ||
| 757 | * duplicates may be skipped, repeated, or trap the walk in a | ||
| 758 | * cycle. Callers requiring duplicate-key iteration must use | ||
| 759 | * rhashtable_walk_*() instead. | ||
| 760 | * - rhltable instances are not supported and return | ||
| 761 | * ERR_PTR(-EOPNOTSUPP). | ||
| 762 | * - If prev_key was concurrently deleted and is not present in any | ||
| 763 | * in-flight table, returns ERR_PTR(-ENOENT). | ||
| 764 | * | ||
| 765 | * Returns entry of the next element, or NULL when iteration is exhausted, | ||
| 766 | * or ERR_PTR(-ENOENT) if prev_key is not found, or | ||
| 767 | * ERR_PTR(-EOPNOTSUPP) if @ht is an rhltable. | ||
| 768 | */ | ||
| 769 | void *rhashtable_next_key(struct rhashtable *ht, const void *prev_key) | ||
| 770 | { | ||
| 771 | struct bucket_table *tbl; | ||
| 772 | struct rhash_head *he; | ||
| 773 | |||
| 774 | if (unlikely(ht->rhlist)) | ||
| 775 | return ERR_PTR(-EOPNOTSUPP); | ||
| 776 | |||
| 777 | tbl = rht_dereference_rcu(ht->tbl, ht); | ||
| 778 | do { | ||
| 779 | he = __rhashtable_next_in_table(ht, tbl, prev_key); | ||
| 780 | if (!IS_ERR_OR_NULL(he)) | ||
| 781 | return rht_obj(ht, he); | ||
| 782 | if (!he) | ||
| 783 | prev_key = NULL; | ||
| 784 | /* See any new future_tbl attached during a rehash. */ | ||
| 785 | smp_rmb(); | ||
| 786 | tbl = rht_dereference_rcu(tbl->future_tbl, ht); | ||
| 787 | } while (tbl); | ||
| 788 | return he; /* NULL or -ENOENT */ | ||
| 789 | } | ||
| 790 | EXPORT_SYMBOL_GPL(rhashtable_next_key); | ||
| 791 | |||
| 690 | /** | 792 | /** |
| 691 | * rhashtable_walk_enter - Initialise an iterator | 793 | * rhashtable_walk_enter - Initialise an iterator |
| 692 | * @ht: Table to walk over | 794 | * @ht: Table to walk over |
| @@ -560,8 +560,23 @@ static int bpf_fill_max_jmp_never_taken(struct bpf_test *self) · 559 unmodified lines | |||
| 560 | } | 560 | } |
| 561 | 561 | ||
| 562 | /* ALU result computation used in tests */ | 562 | /* ALU result computation used in tests */ |
| 563 | static bool __bpf_alu_result(u64 *res, u64 v1, u64 v2, u8 op) | 563 | enum { F_ALU32 = 1, F_SIGNED = 2 }; |
| 564 | |||
| 565 | static bool __bpf_alu_result(u64 *res, u64 v1, u64 v2, u8 op, u32 flags) | ||
| 564 | { | 566 | { |
| 567 | bool is_signed = flags & F_SIGNED; | ||
| 568 | |||
| 569 | /* Narrow operands for ALU32 */ | ||
| 570 | if (flags & F_ALU32) { | ||
| 571 | if (is_signed) { | ||
| 572 | v1 = (u64)(s32)v1; | ||
| 573 | v2 = (u64)(s32)v2; | ||
| 574 | } else { | ||
| 575 | v1 = (u32)v1; | ||
| 576 | v2 = (u32)v2; | ||
| 577 | } | ||
| 578 | } | ||
| 579 | |||
| 565 | *res = 0; | 580 | *res = 0; |
| 566 | switch (op) { | 581 | switch (op) { |
| 567 | case BPF_MOV: | 582 | case BPF_MOV: |
| @@ -599,12 +614,28 @@ static bool __bpf_alu_result(u64 *res, u64 v1, u64 v2, u8 op) · 31 unmodified lines | |||
| 599 | case BPF_DIV: | 614 | case BPF_DIV: |
| 600 | if (v2 == 0) | 615 | if (v2 == 0) |
| 601 | return false; | 616 | return false; |
| 602 | *res = div64_u64(v1, v2); | 617 | if (!is_signed) { |
| 618 | *res = div64_u64(v1, v2); | ||
| 619 | } else { | ||
| 620 | if ((s64)v2 == -1) /* Handled by verifier */ | ||
| 621 | return false; | ||
| 622 | *res = (u64)div64_s64(v1, v2); | ||
| 623 | } | ||
| 603 | break; | 624 | break; |
| 604 | case BPF_MOD: | 625 | case BPF_MOD: |
| 605 | if (v2 == 0) | 626 | if (v2 == 0) |
| 606 | return false; | 627 | return false; |
| 607 | div64_u64_rem(v1, v2, res); | 628 | if (!is_signed) { |
| 629 | div64_u64_rem(v1, v2, res); | ||
| 630 | } else { | ||
| 631 | if ((s64)v2 == -1) | ||
| 632 | return false; | ||
| 633 | /* | ||
| 634 | * Avoid s64 % s64 which generates __moddi3 on | ||
| 635 | * 32-bit architectures. Use div64_s64 instead. | ||
| 636 | */ | ||
| 637 | *res = (u64)((s64)v1 - div64_s64(v1, v2) * (s64)v2); | ||
| 638 | } | ||
| 608 | break; | 639 | break; |
| 609 | } | 640 | } |
| 610 | return true; | 641 | return true; |
| @@ -612,7 +643,7 @@ static bool __bpf_alu_result(u64 *res, u64 v1, u64 v2, u8 op) · 1 unmodified line | |||
| 612 | 643 | ||
| 613 | /* Test an ALU shift operation for all valid shift values */ | 644 | /* Test an ALU shift operation for all valid shift values */ |
| 614 | static int __bpf_fill_alu_shift(struct bpf_test *self, u8 op, | 645 | static int __bpf_fill_alu_shift(struct bpf_test *self, u8 op, |
| 615 | u8 mode, bool alu32) | 646 | u8 mode, u32 flags) |
| 616 | { | 647 | { |
| 617 | static const s64 regs[] = { | 648 | static const s64 regs[] = { |
| 618 | 0x0123456789abcdefLL, /* dword > 0, word < 0 */ | 649 | 0x0123456789abcdefLL, /* dword > 0, word < 0 */ |
| @@ -620,7 +651,7 @@ static int __bpf_fill_alu_shift(struct bpf_test *self, u8 op, · 1 unmodified line | |||
| 620 | 0xfedcba0198765432LL, /* dword < 0, word < 0 */ | 651 | 0xfedcba0198765432LL, /* dword < 0, word < 0 */ |
| 621 | 0x0123458967abcdefLL, /* dword > 0, word > 0 */ | 652 | 0x0123458967abcdefLL, /* dword > 0, word > 0 */ |
| 622 | }; | 653 | }; |
| 623 | int bits = alu32 ? 32 : 64; | 654 | int bits = (flags & F_ALU32) ? 32 : 64; |
| 624 | int len = (2 + 7 * bits) * ARRAY_SIZE(regs) + 3; | 655 | int len = (2 + 7 * bits) * ARRAY_SIZE(regs) + 3; |
| 625 | struct bpf_insn *insn; | 656 | struct bpf_insn *insn; |
| 626 | int imm, k; | 657 | int imm, k; |
| @@ -643,7 +674,7 @@ static int __bpf_fill_alu_shift(struct bpf_test *self, u8 op, · 16 unmodified lines | |||
| 643 | /* Perform operation */ | 674 | /* Perform operation */ |
| 644 | insn[i++] = BPF_ALU64_REG(BPF_MOV, R1, R3); | 675 | insn[i++] = BPF_ALU64_REG(BPF_MOV, R1, R3); |
| 645 | insn[i++] = BPF_ALU64_IMM(BPF_MOV, R2, imm); | 676 | insn[i++] = BPF_ALU64_IMM(BPF_MOV, R2, imm); |
| 646 | if (alu32) { | 677 | if (flags & F_ALU32) { |
| 647 | if (mode == BPF_K) | 678 | if (mode == BPF_K) |
| 648 | insn[i++] = BPF_ALU32_IMM(op, R1, imm); | 679 | insn[i++] = BPF_ALU32_IMM(op, R1, imm); |
| 649 | else | 680 | else |
| @@ -653,14 +684,14 @@ static int __bpf_fill_alu_shift(struct bpf_test *self, u8 op, · 3 unmodified lines | |||
| 653 | reg = (s32)reg; | 684 | reg = (s32)reg; |
| 654 | else | 685 | else |
| 655 | reg = (u32)reg; | 686 | reg = (u32)reg; |
| 656 | __bpf_alu_result(&val, reg, imm, op); | 687 | __bpf_alu_result(&val, reg, imm, op, 0); |
| 657 | val = (u32)val; | 688 | val = (u32)val; |
| 658 | } else { | 689 | } else { |
| 659 | if (mode == BPF_K) | 690 | if (mode == BPF_K) |
| 660 | insn[i++] = BPF_ALU64_IMM(op, R1, imm); | 691 | insn[i++] = BPF_ALU64_IMM(op, R1, imm); |
| 661 | else | 692 | else |
| 662 | insn[i++] = BPF_ALU64_REG(op, R1, R2); | 693 | insn[i++] = BPF_ALU64_REG(op, R1, R2); |
| 663 | __bpf_alu_result(&val, reg, imm, op); | 694 | __bpf_alu_result(&val, reg, imm, op, 0); |
| 664 | } | 695 | } |
| 665 | 696 | ||
| 666 | /* | 697 | /* |
| @@ -688,62 +719,62 @@ static int __bpf_fill_alu_shift(struct bpf_test *self, u8 op, · 21 unmodified lines | |||
| 688 | 719 | ||
| 689 | static int bpf_fill_alu64_lsh_imm(struct bpf_test *self) | 720 | static int bpf_fill_alu64_lsh_imm(struct bpf_test *self) |
| 690 | { | 721 | { |
| 691 | return __bpf_fill_alu_shift(self, BPF_LSH, BPF_K, false); | 722 | return __bpf_fill_alu_shift(self, BPF_LSH, BPF_K, 0); |
| 692 | } | 723 | } |
| 693 | 724 | ||
| 694 | static int bpf_fill_alu64_rsh_imm(struct bpf_test *self) | 725 | static int bpf_fill_alu64_rsh_imm(struct bpf_test *self) |
| 695 | { | 726 | { |
| 696 | return __bpf_fill_alu_shift(self, BPF_RSH, BPF_K, false); | 727 | return __bpf_fill_alu_shift(self, BPF_RSH, BPF_K, 0); |
| 697 | } | 728 | } |
| 698 | 729 | ||
| 699 | static int bpf_fill_alu64_arsh_imm(struct bpf_test *self) | 730 | static int bpf_fill_alu64_arsh_imm(struct bpf_test *self) |
| 700 | { | 731 | { |
| 701 | return __bpf_fill_alu_shift(self, BPF_ARSH, BPF_K, false); | 732 | return __bpf_fill_alu_shift(self, BPF_ARSH, BPF_K, 0); |
| 702 | } | 733 | } |
| 703 | 734 | ||
| 704 | static int bpf_fill_alu64_lsh_reg(struct bpf_test *self) | 735 | static int bpf_fill_alu64_lsh_reg(struct bpf_test *self) |
| 705 | { | 736 | { |
| 706 | return __bpf_fill_alu_shift(self, BPF_LSH, BPF_X, false); | 737 | return __bpf_fill_alu_shift(self, BPF_LSH, BPF_X, 0); |
| 707 | } | 738 | } |
| 708 | 739 | ||
| 709 | static int bpf_fill_alu64_rsh_reg(struct bpf_test *self) | 740 | static int bpf_fill_alu64_rsh_reg(struct bpf_test *self) |
| 710 | { | 741 | { |
| 711 | return __bpf_fill_alu_shift(self, BPF_RSH, BPF_X, false); | 742 | return __bpf_fill_alu_shift(self, BPF_RSH, BPF_X, 0); |
| 712 | } | 743 | } |
| 713 | 744 | ||
| 714 | static int bpf_fill_alu64_arsh_reg(struct bpf_test *self) | 745 | static int bpf_fill_alu64_arsh_reg(struct bpf_test *self) |
| 715 | { | 746 | { |
| 716 | return __bpf_fill_alu_shift(self, BPF_ARSH, BPF_X, false); | 747 | return __bpf_fill_alu_shift(self, BPF_ARSH, BPF_X, 0); |
| 717 | } | 748 | } |
| 718 | 749 | ||
| 719 | static int bpf_fill_alu32_lsh_imm(struct bpf_test *self) | 750 | static int bpf_fill_alu32_lsh_imm(struct bpf_test *self) |
| 720 | { | 751 | { |
| 721 | return __bpf_fill_alu_shift(self, BPF_LSH, BPF_K, true); | 752 | return __bpf_fill_alu_shift(self, BPF_LSH, BPF_K, F_ALU32); |
| 722 | } | 753 | } |
| 723 | 754 | ||
| 724 | static int bpf_fill_alu32_rsh_imm(struct bpf_test *self) | 755 | static int bpf_fill_alu32_rsh_imm(struct bpf_test *self) |
| 725 | { | 756 | { |
| 726 | return __bpf_fill_alu_shift(self, BPF_RSH, BPF_K, true); | 757 | return __bpf_fill_alu_shift(self, BPF_RSH, BPF_K, F_ALU32); |
| 727 | } | 758 | } |
| 728 | 759 | ||
| 729 | static int bpf_fill_alu32_arsh_imm(struct bpf_test *self) | 760 | static int bpf_fill_alu32_arsh_imm(struct bpf_test *self) |
| 730 | { | 761 | { |
| 731 | return __bpf_fill_alu_shift(self, BPF_ARSH, BPF_K, true); | 762 | return __bpf_fill_alu_shift(self, BPF_ARSH, BPF_K, F_ALU32); |
| 732 | } | 763 | } |
| 733 | 764 | ||
| 734 | static int bpf_fill_alu32_lsh_reg(struct bpf_test *self) | 765 | static int bpf_fill_alu32_lsh_reg(struct bpf_test *self) |
| 735 | { | 766 | { |
| 736 | return __bpf_fill_alu_shift(self, BPF_LSH, BPF_X, true); | 767 | return __bpf_fill_alu_shift(self, BPF_LSH, BPF_X, F_ALU32); |
| 737 | } | 768 | } |
| 738 | 769 | ||
| 739 | static int bpf_fill_alu32_rsh_reg(struct bpf_test *self) | 770 | static int bpf_fill_alu32_rsh_reg(struct bpf_test *self) |
| 740 | { | 771 | { |
| 741 | return __bpf_fill_alu_shift(self, BPF_RSH, BPF_X, true); | 772 | return __bpf_fill_alu_shift(self, BPF_RSH, BPF_X, F_ALU32); |
| 742 | } | 773 | } |
| 743 | 774 | ||
| 744 | static int bpf_fill_alu32_arsh_reg(struct bpf_test *self) | 775 | static int bpf_fill_alu32_arsh_reg(struct bpf_test *self) |
| 745 | { | 776 | { |
| 746 | return __bpf_fill_alu_shift(self, BPF_ARSH, BPF_X, true); | 777 | return __bpf_fill_alu_shift(self, BPF_ARSH, BPF_X, F_ALU32); |
| 747 | } | 778 | } |
| 748 | 779 | ||
| 749 | /* | 780 | /* |
| @@ -751,9 +782,9 @@ static int bpf_fill_alu32_arsh_reg(struct bpf_test *self) · 1 unmodified line | |||
| 751 | * for the case when the source and destination are the same. | 782 | * for the case when the source and destination are the same. |
| 752 | */ | 783 | */ |
| 753 | static int __bpf_fill_alu_shift_same_reg(struct bpf_test *self, u8 op, | 784 | static int __bpf_fill_alu_shift_same_reg(struct bpf_test *self, u8 op, |
| 754 | bool alu32) | 785 | u32 flags) |
| 755 | { | 786 | { |
| 756 | int bits = alu32 ? 32 : 64; | 787 | int bits = (flags & F_ALU32) ? 32 : 64; |
| 757 | int len = 3 + 6 * bits; | 788 | int len = 3 + 6 * bits; |
| 758 | struct bpf_insn *insn; | 789 | struct bpf_insn *insn; |
| 759 | int i = 0; | 790 | int i = 0; |
| @@ -770,14 +801,14 @@ static int __bpf_fill_alu_shift_same_reg(struct bpf_test *self, u8 op, · 10 unmodified lines | |||
| 770 | 801 | ||
| 771 | /* Perform operation */ | 802 | /* Perform operation */ |
| 772 | insn[i++] = BPF_ALU64_IMM(BPF_MOV, R1, val); | 803 | insn[i++] = BPF_ALU64_IMM(BPF_MOV, R1, val); |
| 773 | if (alu32) | 804 | if (flags & F_ALU32) |
| 774 | insn[i++] = BPF_ALU32_REG(op, R1, R1); | 805 | insn[i++] = BPF_ALU32_REG(op, R1, R1); |
| 775 | else | 806 | else |
| 776 | insn[i++] = BPF_ALU64_REG(op, R1, R1); | 807 | insn[i++] = BPF_ALU64_REG(op, R1, R1); |
| 777 | 808 | ||
| 778 | /* Compute the reference result */ | 809 | /* Compute the reference result */ |
| 779 | __bpf_alu_result(&res, val, val, op); | 810 | __bpf_alu_result(&res, val, val, op, 0); |
| 780 | if (alu32) | 811 | if (flags & F_ALU32) |
| 781 | res = (u32)res; | 812 | res = (u32)res; |
| 782 | i += __bpf_ld_imm64(&insn[i], R2, res); | 813 | i += __bpf_ld_imm64(&insn[i], R2, res); |
| 783 | 814 | ||
| @@ -798,32 +829,32 @@ static int __bpf_fill_alu_shift_same_reg(struct bpf_test *self, u8 op, · 14 unmodified lines | |||
| 798 | 829 | ||
| 799 | static int bpf_fill_alu64_lsh_same_reg(struct bpf_test *self) | 830 | static int bpf_fill_alu64_lsh_same_reg(struct bpf_test *self) |
| 800 | { | 831 | { |
| 801 | return __bpf_fill_alu_shift_same_reg(self, BPF_LSH, false); | 832 | return __bpf_fill_alu_shift_same_reg(self, BPF_LSH, 0); |
| 802 | } | 833 | } |
| 803 | 834 | ||
| 804 | static int bpf_fill_alu64_rsh_same_reg(struct bpf_test *self) | 835 | static int bpf_fill_alu64_rsh_same_reg(struct bpf_test *self) |
| 805 | { | 836 | { |
| 806 | return __bpf_fill_alu_shift_same_reg(self, BPF_RSH, false); | 837 | return __bpf_fill_alu_shift_same_reg(self, BPF_RSH, 0); |
| 807 | } | 838 | } |
| 808 | 839 | ||
| 809 | static int bpf_fill_alu64_arsh_same_reg(struct bpf_test *self) | 840 | static int bpf_fill_alu64_arsh_same_reg(struct bpf_test *self) |
| 810 | { | 841 | { |
| 811 | return __bpf_fill_alu_shift_same_reg(self, BPF_ARSH, false); | 842 | return __bpf_fill_alu_shift_same_reg(self, BPF_ARSH, 0); |
| 812 | } | 843 | } |
| 813 | 844 | ||
| 814 | static int bpf_fill_alu32_lsh_same_reg(struct bpf_test *self) | 845 | static int bpf_fill_alu32_lsh_same_reg(struct bpf_test *self) |
| 815 | { | 846 | { |
| 816 | return __bpf_fill_alu_shift_same_reg(self, BPF_LSH, true); | 847 | return __bpf_fill_alu_shift_same_reg(self, BPF_LSH, F_ALU32); |
| 817 | } | 848 | } |
| 818 | 849 | ||
| 819 | static int bpf_fill_alu32_rsh_same_reg(struct bpf_test *self) | 850 | static int bpf_fill_alu32_rsh_same_reg(struct bpf_test *self) |
| 820 | { | 851 | { |
| 821 | return __bpf_fill_alu_shift_same_reg(self, BPF_RSH, true); | 852 | return __bpf_fill_alu_shift_same_reg(self, BPF_RSH, F_ALU32); |
| 822 | } | 853 | } |
| 823 | 854 | ||
| 824 | static int bpf_fill_alu32_arsh_same_reg(struct bpf_test *self) | 855 | static int bpf_fill_alu32_arsh_same_reg(struct bpf_test *self) |
| 825 | { | 856 | { |
| 826 | return __bpf_fill_alu_shift_same_reg(self, BPF_ARSH, true); | 857 | return __bpf_fill_alu_shift_same_reg(self, BPF_ARSH, F_ALU32); |
| 827 | } | 858 | } |
| 828 | 859 | ||
| 829 | /* | 860 | /* |
| @@ -936,17 +967,20 @@ static int __bpf_fill_pattern(struct bpf_test *self, void *arg, · 106 unmodified lines | |||
| 936 | static int __bpf_emit_alu64_imm(struct bpf_test *self, void *arg, | 967 | static int __bpf_emit_alu64_imm(struct bpf_test *self, void *arg, |
| 937 | struct bpf_insn *insns, s64 dst, s64 imm) | 968 | struct bpf_insn *insns, s64 dst, s64 imm) |
| 938 | { | 969 | { |
| 939 | int op = *(int *)arg; | 970 | int *a = arg; |
| 971 | int op = a[0]; | ||
| 972 | u32 flags = a[1]; | ||
| 973 | s16 off = (flags & F_SIGNED) ? 1 : 0; | ||
| 940 | int i = 0; | 974 | int i = 0; |
| 941 | u64 res; | 975 | u64 res; |
| 942 | 976 | ||
| 943 | if (!insns) | 977 | if (!insns) |
| 944 | return 7; | 978 | return 7; |
| 945 | 979 | ||
| 946 | if (__bpf_alu_result(&res, dst, (s32)imm, op)) { | 980 | if (__bpf_alu_result(&res, dst, (s32)imm, op, flags)) { |
| 947 | i += __bpf_ld_imm64(&insns[i], R1, dst); | 981 | i += __bpf_ld_imm64(&insns[i], R1, dst); |
| 948 | i += __bpf_ld_imm64(&insns[i], R3, res); | 982 | i += __bpf_ld_imm64(&insns[i], R3, res); |
| 949 | insns[i++] = BPF_ALU64_IMM(op, R1, imm); | 983 | insns[i++] = BPF_ALU64_IMM_OFF(op, R1, imm, off); |
| 950 | insns[i++] = BPF_JMP_REG(BPF_JEQ, R1, R3, 1); | 984 | insns[i++] = BPF_JMP_REG(BPF_JEQ, R1, R3, 1); |
| 951 | insns[i++] = BPF_EXIT_INSN(); | 985 | insns[i++] = BPF_EXIT_INSN(); |
| 952 | } | 986 | } |
| @@ -957,17 +991,20 @@ static int __bpf_emit_alu64_imm(struct bpf_test *self, void *arg, · 4 unmodified lines | |||
| 957 | static int __bpf_emit_alu32_imm(struct bpf_test *self, void *arg, | 991 | static int __bpf_emit_alu32_imm(struct bpf_test *self, void *arg, |
| 958 | struct bpf_insn *insns, s64 dst, s64 imm) | 992 | struct bpf_insn *insns, s64 dst, s64 imm) |
| 959 | { | 993 | { |
| 960 | int op = *(int *)arg; | 994 | int *a = arg; |
| 995 | int op = a[0]; | ||
| 996 | u32 flags = a[1]; | ||
| 997 | s16 off = (flags & F_SIGNED) ? 1 : 0; | ||
| 961 | int i = 0; | 998 | int i = 0; |
| 962 | u64 res; | 999 | u64 res; |
| 963 | 1000 | ||
| 964 | if (!insns) | 1001 | if (!insns) |
| 965 | return 7; | 1002 | return 7; |
| 966 | 1003 | ||
| 967 | if (__bpf_alu_result(&res, (u32)dst, (u32)imm, op)) { | 1004 | if (__bpf_alu_result(&res, dst, (s32)imm, op, flags | F_ALU32)) { |
| 968 | i += __bpf_ld_imm64(&insns[i], R1, dst); | 1005 | i += __bpf_ld_imm64(&insns[i], R1, dst); |
| 969 | i += __bpf_ld_imm64(&insns[i], R3, (u32)res); | 1006 | i += __bpf_ld_imm64(&insns[i], R3, (u32)res); |
| 970 | insns[i++] = BPF_ALU32_IMM(op, R1, imm); | 1007 | insns[i++] = BPF_ALU32_IMM_OFF(op, R1, imm, off); |
| 971 | insns[i++] = BPF_JMP_REG(BPF_JEQ, R1, R3, 1); | 1008 | insns[i++] = BPF_JMP_REG(BPF_JEQ, R1, R3, 1); |
| 972 | insns[i++] = BPF_EXIT_INSN(); | 1009 | insns[i++] = BPF_EXIT_INSN(); |
| 973 | } | 1010 | } |
| @@ -985,7 +1022,7 @@ static int __bpf_emit_alu64_reg(struct bpf_test *self, void *arg, · 11 unmodified lines | |||
| 985 | if (!insns) | 1022 | if (!insns) |
| 986 | return 9; | 1023 | return 9; |
| 987 | 1024 | ||
| 988 | if (__bpf_alu_result(&res, dst, src, op)) { | 1025 | if (__bpf_alu_result(&res, dst, src, op, 0)) { |
| 989 | i += __bpf_ld_imm64(&insns[i], R1, dst); | 1026 | i += __bpf_ld_imm64(&insns[i], R1, dst); |
| 990 | i += __bpf_ld_imm64(&insns[i], R2, src); | 1027 | i += __bpf_ld_imm64(&insns[i], R2, src); |
| 991 | i += __bpf_ld_imm64(&insns[i], R3, res); | 1028 | i += __bpf_ld_imm64(&insns[i], R3, res); |
| @@ -1007,7 +1044,7 @@ static int __bpf_emit_alu32_reg(struct bpf_test *self, void *arg, · 15 unmodified lines | |||
| 1007 | if (!insns) | 1044 | if (!insns) |
| 1008 | return 9; | 1045 | return 9; |
| 1009 | 1046 | ||
| 1010 | if (__bpf_alu_result(&res, (u32)dst, (u32)src, op)) { | 1047 | if (__bpf_alu_result(&res, (u32)dst, (u32)src, op, 0)) { |
| 1011 | i += __bpf_ld_imm64(&insns[i], R1, dst); | 1048 | i += __bpf_ld_imm64(&insns[i], R1, dst); |
| 1012 | i += __bpf_ld_imm64(&insns[i], R2, src); | 1049 | i += __bpf_ld_imm64(&insns[i], R2, src); |
| 1013 | i += __bpf_ld_imm64(&insns[i], R3, (u32)res); | 1050 | i += __bpf_ld_imm64(&insns[i], R3, (u32)res); |
| @@ -1019,16 +1056,20 @@ static int __bpf_emit_alu32_reg(struct bpf_test *self, void *arg, · 5 unmodified lines | |||
| 1019 | return i; | 1056 | return i; |
| 1020 | } | 1057 | } |
| 1021 | 1058 | ||
| 1022 | static int __bpf_fill_alu64_imm(struct bpf_test *self, int op) | 1059 | static int __bpf_fill_alu64_imm(struct bpf_test *self, int op, u32 flags) |
| 1023 | { | 1060 | { |
| 1024 | return __bpf_fill_pattern(self, &op, 64, 32, | 1061 | int arg[2] = {op, flags}; |
| 1062 | |||
| 1063 | return __bpf_fill_pattern(self, &arg, 64, 32, | ||
| 1025 | PATTERN_BLOCK1, PATTERN_BLOCK2, | 1064 | PATTERN_BLOCK1, PATTERN_BLOCK2, |
| 1026 | &__bpf_emit_alu64_imm); | 1065 | &__bpf_emit_alu64_imm); |
| 1027 | } | 1066 | } |
| 1028 | 1067 | ||
| 1029 | static int __bpf_fill_alu32_imm(struct bpf_test *self, int op) | 1068 | static int __bpf_fill_alu32_imm(struct bpf_test *self, int op, u32 flags) |
| 1030 | { | 1069 | { |
| 1031 | return __bpf_fill_pattern(self, &op, 64, 32, | 1070 | int arg[2] = {op, flags}; |
| 1071 | |||
| 1072 | return __bpf_fill_pattern(self, &arg, 64, 32, | ||
| 1032 | PATTERN_BLOCK1, PATTERN_BLOCK2, | 1073 | PATTERN_BLOCK1, PATTERN_BLOCK2, |
| 1033 | &__bpf_emit_alu32_imm); | 1074 | &__bpf_emit_alu32_imm); |
| 1034 | } | 1075 | } |
| @@ -1050,93 +1091,115 @@ static int __bpf_fill_alu32_reg(struct bpf_test *self, int op) · 15 unmodified lines | |||
| 1050 | /* ALU64 immediate operations */ | 1091 | /* ALU64 immediate operations */ |
| 1051 | static int bpf_fill_alu64_mov_imm(struct bpf_test *self) | 1092 | static int bpf_fill_alu64_mov_imm(struct bpf_test *self) |
| 1052 | { | 1093 | { |
| 1053 | return __bpf_fill_alu64_imm(self, BPF_MOV); | 1094 | return __bpf_fill_alu64_imm(self, BPF_MOV, 0); |
| 1054 | } | 1095 | } |
| 1055 | 1096 | ||
| 1056 | static int bpf_fill_alu64_and_imm(struct bpf_test *self) | 1097 | static int bpf_fill_alu64_and_imm(struct bpf_test *self) |
| 1057 | { | 1098 | { |
| 1058 | return __bpf_fill_alu64_imm(self, BPF_AND); | 1099 | return __bpf_fill_alu64_imm(self, BPF_AND, 0); |
| 1059 | } | 1100 | } |
| 1060 | 1101 | ||
| 1061 | static int bpf_fill_alu64_or_imm(struct bpf_test *self) | 1102 | static int bpf_fill_alu64_or_imm(struct bpf_test *self) |
| 1062 | { | 1103 | { |
| 1063 | return __bpf_fill_alu64_imm(self, BPF_OR); | 1104 | return __bpf_fill_alu64_imm(self, BPF_OR, 0); |
| 1064 | } | 1105 | } |
| 1065 | 1106 | ||
| 1066 | static int bpf_fill_alu64_xor_imm(struct bpf_test *self) | 1107 | static int bpf_fill_alu64_xor_imm(struct bpf_test *self) |
| 1067 | { | 1108 | { |
| 1068 | return __bpf_fill_alu64_imm(self, BPF_XOR); | 1109 | return __bpf_fill_alu64_imm(self, BPF_XOR, 0); |
| 1069 | } | 1110 | } |
| 1070 | 1111 | ||
| 1071 | static int bpf_fill_alu64_add_imm(struct bpf_test *self) | 1112 | static int bpf_fill_alu64_add_imm(struct bpf_test *self) |
| 1072 | { | 1113 | { |
| 1073 | return __bpf_fill_alu64_imm(self, BPF_ADD); | 1114 | return __bpf_fill_alu64_imm(self, BPF_ADD, 0); |
| 1074 | } | 1115 | } |
| 1075 | 1116 | ||
| 1076 | static int bpf_fill_alu64_sub_imm(struct bpf_test *self) | 1117 | static int bpf_fill_alu64_sub_imm(struct bpf_test *self) |
| 1077 | { | 1118 | { |
| 1078 | return __bpf_fill_alu64_imm(self, BPF_SUB); | 1119 | return __bpf_fill_alu64_imm(self, BPF_SUB, 0); |
| 1079 | } | 1120 | } |
| 1080 | 1121 | ||
| 1081 | static int bpf_fill_alu64_mul_imm(struct bpf_test *self) | 1122 | static int bpf_fill_alu64_mul_imm(struct bpf_test *self) |
| 1082 | { | 1123 | { |
| 1083 | return __bpf_fill_alu64_imm(self, BPF_MUL); | 1124 | return __bpf_fill_alu64_imm(self, BPF_MUL, 0); |
| 1084 | } | 1125 | } |
| 1085 | 1126 | ||
| 1086 | static int bpf_fill_alu64_div_imm(struct bpf_test *self) | 1127 | static int bpf_fill_alu64_div_imm(struct bpf_test *self) |
| 1087 | { | 1128 | { |
| 1088 | return __bpf_fill_alu64_imm(self, BPF_DIV); | 1129 | return __bpf_fill_alu64_imm(self, BPF_DIV, 0); |
| 1089 | } | 1130 | } |
| 1090 | 1131 | ||
| 1091 | static int bpf_fill_alu64_mod_imm(struct bpf_test *self) | 1132 | static int bpf_fill_alu64_mod_imm(struct bpf_test *self) |
| 1092 | { | 1133 | { |
| 1093 | return __bpf_fill_alu64_imm(self, BPF_MOD); | 1134 | return __bpf_fill_alu64_imm(self, BPF_MOD, 0); |
| 1135 | } | ||
| 1136 | |||
| 1137 | /* Signed ALU64 immediate operations */ | ||
| 1138 | static int bpf_fill_alu64_sdiv_imm(struct bpf_test *self) | ||
| 1139 | { | ||
| 1140 | return __bpf_fill_alu64_imm(self, BPF_DIV, F_SIGNED); | ||
| 1141 | } | ||
| 1142 | |||
| 1143 | static int bpf_fill_alu64_smod_imm(struct bpf_test *self) | ||
| 1144 | { | ||
| 1145 | return __bpf_fill_alu64_imm(self, BPF_MOD, F_SIGNED); | ||
| 1146 | } | ||
| 1147 | |||
| 1148 | /* Signed ALU32 immediate operations */ | ||
| 1149 | static int bpf_fill_alu32_sdiv_imm(struct bpf_test *self) | ||
| 1150 | { | ||
| 1151 | return __bpf_fill_alu32_imm(self, BPF_DIV, F_SIGNED); | ||
| 1152 | } | ||
| 1153 | |||
| 1154 | static int bpf_fill_alu32_smod_imm(struct bpf_test *self) | ||
| 1155 | { | ||
| 1156 | return __bpf_fill_alu32_imm(self, BPF_MOD, F_SIGNED); | ||
| 1094 | } | 1157 | } |
| 1095 | 1158 | ||
| 1096 | /* ALU32 immediate operations */ | 1159 | /* ALU32 immediate operations */ |
| 1097 | static int bpf_fill_alu32_mov_imm(struct bpf_test *self) | 1160 | static int bpf_fill_alu32_mov_imm(struct bpf_test *self) |
| 1098 | { | 1161 | { |
| 1099 | return __bpf_fill_alu32_imm(self, BPF_MOV); | 1162 | return __bpf_fill_alu32_imm(self, BPF_MOV, 0); |
| 1100 | } | 1163 | } |
| 1101 | 1164 | ||
| 1102 | static int bpf_fill_alu32_and_imm(struct bpf_test *self) | 1165 | static int bpf_fill_alu32_and_imm(struct bpf_test *self) |
| 1103 | { | 1166 | { |
| 1104 | return __bpf_fill_alu32_imm(self, BPF_AND); | 1167 | return __bpf_fill_alu32_imm(self, BPF_AND, 0); |
| 1105 | } | 1168 | } |
| 1106 | 1169 | ||
| 1107 | static int bpf_fill_alu32_or_imm(struct bpf_test *self) | 1170 | static int bpf_fill_alu32_or_imm(struct bpf_test *self) |
| 1108 | { | 1171 | { |
| 1109 | return __bpf_fill_alu32_imm(self, BPF_OR); | 1172 | return __bpf_fill_alu32_imm(self, BPF_OR, 0); |
| 1110 | } | 1173 | } |
| 1111 | 1174 | ||
| 1112 | static int bpf_fill_alu32_xor_imm(struct bpf_test *self) | 1175 | static int bpf_fill_alu32_xor_imm(struct bpf_test *self) |
| 1113 | { | 1176 | { |
| 1114 | return __bpf_fill_alu32_imm(self, BPF_XOR); | 1177 | return __bpf_fill_alu32_imm(self, BPF_XOR, 0); |
| 1115 | } | 1178 | } |
| 1116 | 1179 | ||
| 1117 | static int bpf_fill_alu32_add_imm(struct bpf_test *self) | 1180 | static int bpf_fill_alu32_add_imm(struct bpf_test *self) |
| 1118 | { | 1181 | { |
| 1119 | return __bpf_fill_alu32_imm(self, BPF_ADD); | 1182 | return __bpf_fill_alu32_imm(self, BPF_ADD, 0); |
| 1120 | } | 1183 | } |
| 1121 | 1184 | ||
| 1122 | static int bpf_fill_alu32_sub_imm(struct bpf_test *self) | 1185 | static int bpf_fill_alu32_sub_imm(struct bpf_test *self) |
| 1123 | { | 1186 | { |
| 1124 | return __bpf_fill_alu32_imm(self, BPF_SUB); | 1187 | return __bpf_fill_alu32_imm(self, BPF_SUB, 0); |
| 1125 | } | 1188 | } |
| 1126 | 1189 | ||
| 1127 | static int bpf_fill_alu32_mul_imm(struct bpf_test *self) | 1190 | static int bpf_fill_alu32_mul_imm(struct bpf_test *self) |
| 1128 | { | 1191 | { |
| 1129 | return __bpf_fill_alu32_imm(self, BPF_MUL); | 1192 | return __bpf_fill_alu32_imm(self, BPF_MUL, 0); |
| 1130 | } | 1193 | } |
| 1131 | 1194 | ||
| 1132 | static int bpf_fill_alu32_div_imm(struct bpf_test *self) | 1195 | static int bpf_fill_alu32_div_imm(struct bpf_test *self) |
| 1133 | { | 1196 | { |
| 1134 | return __bpf_fill_alu32_imm(self, BPF_DIV); | 1197 | return __bpf_fill_alu32_imm(self, BPF_DIV, 0); |
| 1135 | } | 1198 | } |
| 1136 | 1199 | ||
| 1137 | static int bpf_fill_alu32_mod_imm(struct bpf_test *self) | 1200 | static int bpf_fill_alu32_mod_imm(struct bpf_test *self) |
| 1138 | { | 1201 | { |
| 1139 | return __bpf_fill_alu32_imm(self, BPF_MOD); | 1202 | return __bpf_fill_alu32_imm(self, BPF_MOD, 0); |
| 1140 | } | 1203 | } |
| 1141 | 1204 | ||
| 1142 | /* ALU64 register operations */ | 1205 | /* ALU64 register operations */ |
| @@ -1235,7 +1298,8 @@ static int bpf_fill_alu32_mod_reg(struct bpf_test *self) · 92 unmodified lines | |||
| 1235 | * Test JITs that implement complex ALU operations as function | 1298 | * Test JITs that implement complex ALU operations as function |
| 1236 | * calls, and must re-arrange operands for argument passing. | 1299 | * calls, and must re-arrange operands for argument passing. |
| 1237 | */ | 1300 | */ |
| 1238 | static int __bpf_fill_alu_imm_regs(struct bpf_test *self, u8 op, bool alu32) | 1301 | static int __bpf_fill_alu_imm_regs(struct bpf_test *self, u8 op, |
| 1302 | u32 flags) | ||
| 1239 | { | 1303 | { |
| 1240 | int len = 2 + 10 * 10; | 1304 | int len = 2 + 10 * 10; |
| 1241 | struct bpf_insn *insns; | 1305 | struct bpf_insn *insns; |
| @@ -1249,28 +1313,37 @@ static int __bpf_fill_alu_imm_regs(struct bpf_test *self, u8 op, bool alu32) · 7 unmodified lines | |||
| 1249 | return -ENOMEM; | 1313 | return -ENOMEM; |
| 1250 | 1314 | ||
| 1251 | /* Operand and result values according to operation */ | 1315 | /* Operand and result values according to operation */ |
| 1252 | if (alu32) | 1316 | if (flags & F_SIGNED) { |
| 1253 | dst = 0x76543210U; | 1317 | if (flags & F_ALU32) |
| 1254 | else | 1318 | dst = -76543210; |
| 1255 | dst = 0x7edcba9876543210ULL; | 1319 | else |
| 1320 | dst = -7654321076543210LL; | ||
| 1321 | } else { | ||
| 1322 | if (flags & F_ALU32) | ||
| 1323 | dst = 0x76543210U; | ||
| 1324 | else | ||
| 1325 | dst = 0x7edcba9876543210ULL; | ||
| 1326 | } | ||
| 1256 | imm = 0x01234567U; | 1327 | imm = 0x01234567U; |
| 1257 | 1328 | ||
| 1258 | if (op == BPF_LSH || op == BPF_RSH || op == BPF_ARSH) | 1329 | if (op == BPF_LSH || op == BPF_RSH || op == BPF_ARSH) |
| 1259 | imm &= 31; | 1330 | imm &= 31; |
| 1260 | 1331 | ||
| 1261 | __bpf_alu_result(&res, dst, imm, op); | 1332 | __bpf_alu_result(&res, dst, imm, op, flags); |
| 1262 | 1333 | ||
| 1263 | if (alu32) | 1334 | if (flags & F_ALU32) |
| 1264 | res = (u32)res; | 1335 | res = (u32)res; |
| 1265 | 1336 | ||
| 1266 | /* Check all operand registers */ | 1337 | /* Check all operand registers */ |
| 1267 | for (rd = R0; rd <= R9; rd++) { | 1338 | for (rd = R0; rd <= R9; rd++) { |
| 1268 | i += __bpf_ld_imm64(&insns[i], rd, dst); | 1339 | i += __bpf_ld_imm64(&insns[i], rd, dst); |
| 1269 | 1340 | ||
| 1270 | if (alu32) | 1341 | s16 off = (flags & F_SIGNED) ? 1 : 0; |
| 1271 | insns[i++] = BPF_ALU32_IMM(op, rd, imm); | 1342 | |
| 1343 | if (flags & F_ALU32) | ||
| 1344 | insns[i++] = BPF_ALU32_IMM_OFF(op, rd, imm, off); | ||
| 1272 | else | 1345 | else |
| 1273 | insns[i++] = BPF_ALU64_IMM(op, rd, imm); | 1346 | insns[i++] = BPF_ALU64_IMM_OFF(op, rd, imm, off); |
| 1274 | 1347 | ||
| 1275 | insns[i++] = BPF_JMP32_IMM(BPF_JEQ, rd, res, 2); | 1348 | insns[i++] = BPF_JMP32_IMM(BPF_JEQ, rd, res, 2); |
| 1276 | insns[i++] = BPF_MOV64_IMM(R0, __LINE__); | 1349 | insns[i++] = BPF_MOV64_IMM(R0, __LINE__); |
| @@ -1295,123 +1368,145 @@ static int __bpf_fill_alu_imm_regs(struct bpf_test *self, u8 op, bool alu32) · 18 unmodified lines | |||
| 1295 | /* ALU64 K registers */ | 1368 | /* ALU64 K registers */ |
| 1296 | static int bpf_fill_alu64_mov_imm_regs(struct bpf_test *self) | 1369 | static int bpf_fill_alu64_mov_imm_regs(struct bpf_test *self) |
| 1297 | { | 1370 | { |
| 1298 | return __bpf_fill_alu_imm_regs(self, BPF_MOV, false); | 1371 | return __bpf_fill_alu_imm_regs(self, BPF_MOV, 0); |
| 1299 | } | 1372 | } |
| 1300 | 1373 | ||
| 1301 | static int bpf_fill_alu64_and_imm_regs(struct bpf_test *self) | 1374 | static int bpf_fill_alu64_and_imm_regs(struct bpf_test *self) |
| 1302 | { | 1375 | { |
| 1303 | return __bpf_fill_alu_imm_regs(self, BPF_AND, false); | 1376 | return __bpf_fill_alu_imm_regs(self, BPF_AND, 0); |
| 1304 | } | 1377 | } |
| 1305 | 1378 | ||
| 1306 | static int bpf_fill_alu64_or_imm_regs(struct bpf_test *self) | 1379 | static int bpf_fill_alu64_or_imm_regs(struct bpf_test *self) |
| 1307 | { | 1380 | { |
| 1308 | return __bpf_fill_alu_imm_regs(self, BPF_OR, false); | 1381 | return __bpf_fill_alu_imm_regs(self, BPF_OR, 0); |
| 1309 | } | 1382 | } |
| 1310 | 1383 | ||
| 1311 | static int bpf_fill_alu64_xor_imm_regs(struct bpf_test *self) | 1384 | static int bpf_fill_alu64_xor_imm_regs(struct bpf_test *self) |
| 1312 | { | 1385 | { |
| 1313 | return __bpf_fill_alu_imm_regs(self, BPF_XOR, false); | 1386 | return __bpf_fill_alu_imm_regs(self, BPF_XOR, 0); |
| 1314 | } | 1387 | } |
| 1315 | 1388 | ||
| 1316 | static int bpf_fill_alu64_lsh_imm_regs(struct bpf_test *self) | 1389 | static int bpf_fill_alu64_lsh_imm_regs(struct bpf_test *self) |
| 1317 | { | 1390 | { |
| 1318 | return __bpf_fill_alu_imm_regs(self, BPF_LSH, false); | 1391 | return __bpf_fill_alu_imm_regs(self, BPF_LSH, 0); |
| 1319 | } | 1392 | } |
| 1320 | 1393 | ||
| 1321 | static int bpf_fill_alu64_rsh_imm_regs(struct bpf_test *self) | 1394 | static int bpf_fill_alu64_rsh_imm_regs(struct bpf_test *self) |
| 1322 | { | 1395 | { |
| 1323 | return __bpf_fill_alu_imm_regs(self, BPF_RSH, false); | 1396 | return __bpf_fill_alu_imm_regs(self, BPF_RSH, 0); |
| 1324 | } | 1397 | } |
| 1325 | 1398 | ||
| 1326 | static int bpf_fill_alu64_arsh_imm_regs(struct bpf_test *self) | 1399 | static int bpf_fill_alu64_arsh_imm_regs(struct bpf_test *self) |
| 1327 | { | 1400 | { |
| 1328 | return __bpf_fill_alu_imm_regs(self, BPF_ARSH, false); | 1401 | return __bpf_fill_alu_imm_regs(self, BPF_ARSH, 0); |
| 1329 | } | 1402 | } |
| 1330 | 1403 | ||
| 1331 | static int bpf_fill_alu64_add_imm_regs(struct bpf_test *self) | 1404 | static int bpf_fill_alu64_add_imm_regs(struct bpf_test *self) |
| 1332 | { | 1405 | { |
| 1333 | return __bpf_fill_alu_imm_regs(self, BPF_ADD, false); | 1406 | return __bpf_fill_alu_imm_regs(self, BPF_ADD, 0); |
| 1334 | } | 1407 | } |
| 1335 | 1408 | ||
| 1336 | static int bpf_fill_alu64_sub_imm_regs(struct bpf_test *self) | 1409 | static int bpf_fill_alu64_sub_imm_regs(struct bpf_test *self) |
| 1337 | { | 1410 | { |
| 1338 | return __bpf_fill_alu_imm_regs(self, BPF_SUB, false); | 1411 | return __bpf_fill_alu_imm_regs(self, BPF_SUB, 0); |
| 1339 | } | 1412 | } |
| 1340 | 1413 | ||
| 1341 | static int bpf_fill_alu64_mul_imm_regs(struct bpf_test *self) | 1414 | static int bpf_fill_alu64_mul_imm_regs(struct bpf_test *self) |
| 1342 | { | 1415 | { |
| 1343 | return __bpf_fill_alu_imm_regs(self, BPF_MUL, false); | 1416 | return __bpf_fill_alu_imm_regs(self, BPF_MUL, 0); |
| 1344 | } | 1417 | } |
| 1345 | 1418 | ||
| 1346 | static int bpf_fill_alu64_div_imm_regs(struct bpf_test *self) | 1419 | static int bpf_fill_alu64_div_imm_regs(struct bpf_test *self) |
| 1347 | { | 1420 | { |
| 1348 | return __bpf_fill_alu_imm_regs(self, BPF_DIV, false); | 1421 | return __bpf_fill_alu_imm_regs(self, BPF_DIV, 0); |
| 1349 | } | 1422 | } |
| 1350 | 1423 | ||
| 1351 | static int bpf_fill_alu64_mod_imm_regs(struct bpf_test *self) | 1424 | static int bpf_fill_alu64_mod_imm_regs(struct bpf_test *self) |
| 1352 | { | 1425 | { |
| 1353 | return __bpf_fill_alu_imm_regs(self, BPF_MOD, false); | 1426 | return __bpf_fill_alu_imm_regs(self, BPF_MOD, 0); |
| 1427 | } | ||
| 1428 | |||
| 1429 | /* Signed ALU64 K registers */ | ||
| 1430 | static int bpf_fill_alu64_sdiv_imm_regs(struct bpf_test *self) | ||
| 1431 | { | ||
| 1432 | return __bpf_fill_alu_imm_regs(self, BPF_DIV, F_SIGNED); | ||
| 1433 | } | ||
| 1434 | |||
| 1435 | static int bpf_fill_alu64_smod_imm_regs(struct bpf_test *self) | ||
| 1436 | { | ||
| 1437 | return __bpf_fill_alu_imm_regs(self, BPF_MOD, F_SIGNED); | ||
| 1354 | } | 1438 | } |
| 1355 | 1439 | ||
| 1356 | /* ALU32 K registers */ | 1440 | /* ALU32 K registers */ |
| 1357 | static int bpf_fill_alu32_mov_imm_regs(struct bpf_test *self) | 1441 | static int bpf_fill_alu32_mov_imm_regs(struct bpf_test *self) |
| 1358 | { | 1442 | { |
| 1359 | return __bpf_fill_alu_imm_regs(self, BPF_MOV, true); | 1443 | return __bpf_fill_alu_imm_regs(self, BPF_MOV, F_ALU32); |
| 1360 | } | 1444 | } |
| 1361 | 1445 | ||
| 1362 | static int bpf_fill_alu32_and_imm_regs(struct bpf_test *self) | 1446 | static int bpf_fill_alu32_and_imm_regs(struct bpf_test *self) |
| 1363 | { | 1447 | { |
| 1364 | return __bpf_fill_alu_imm_regs(self, BPF_AND, true); | 1448 | return __bpf_fill_alu_imm_regs(self, BPF_AND, F_ALU32); |
| 1365 | } | 1449 | } |
| 1366 | 1450 | ||
| 1367 | static int bpf_fill_alu32_or_imm_regs(struct bpf_test *self) | 1451 | static int bpf_fill_alu32_or_imm_regs(struct bpf_test *self) |
| 1368 | { | 1452 | { |
| 1369 | return __bpf_fill_alu_imm_regs(self, BPF_OR, true); | 1453 | return __bpf_fill_alu_imm_regs(self, BPF_OR, F_ALU32); |
| 1370 | } | 1454 | } |
| 1371 | 1455 | ||
| 1372 | static int bpf_fill_alu32_xor_imm_regs(struct bpf_test *self) | 1456 | static int bpf_fill_alu32_xor_imm_regs(struct bpf_test *self) |
| 1373 | { | 1457 | { |
| 1374 | return __bpf_fill_alu_imm_regs(self, BPF_XOR, true); | 1458 | return __bpf_fill_alu_imm_regs(self, BPF_XOR, F_ALU32); |
| 1375 | } | 1459 | } |
| 1376 | 1460 | ||
| 1377 | static int bpf_fill_alu32_lsh_imm_regs(struct bpf_test *self) | 1461 | static int bpf_fill_alu32_lsh_imm_regs(struct bpf_test *self) |
| 1378 | { | 1462 | { |
| 1379 | return __bpf_fill_alu_imm_regs(self, BPF_LSH, true); | 1463 | return __bpf_fill_alu_imm_regs(self, BPF_LSH, F_ALU32); |
| 1380 | } | 1464 | } |
| 1381 | 1465 | ||
| 1382 | static int bpf_fill_alu32_rsh_imm_regs(struct bpf_test *self) | 1466 | static int bpf_fill_alu32_rsh_imm_regs(struct bpf_test *self) |
| 1383 | { | 1467 | { |
| 1384 | return __bpf_fill_alu_imm_regs(self, BPF_RSH, true); | 1468 | return __bpf_fill_alu_imm_regs(self, BPF_RSH, F_ALU32); |
| 1385 | } | 1469 | } |
| 1386 | 1470 | ||
| 1387 | static int bpf_fill_alu32_arsh_imm_regs(struct bpf_test *self) | 1471 | static int bpf_fill_alu32_arsh_imm_regs(struct bpf_test *self) |
| 1388 | { | 1472 | { |
| 1389 | return __bpf_fill_alu_imm_regs(self, BPF_ARSH, true); | 1473 | return __bpf_fill_alu_imm_regs(self, BPF_ARSH, F_ALU32); |
| 1390 | } | 1474 | } |
| 1391 | 1475 | ||
| 1392 | static int bpf_fill_alu32_add_imm_regs(struct bpf_test *self) | 1476 | static int bpf_fill_alu32_add_imm_regs(struct bpf_test *self) |
| 1393 | { | 1477 | { |
| 1394 | return __bpf_fill_alu_imm_regs(self, BPF_ADD, true); | 1478 | return __bpf_fill_alu_imm_regs(self, BPF_ADD, F_ALU32); |
| 1395 | } | 1479 | } |
| 1396 | 1480 | ||
| 1397 | static int bpf_fill_alu32_sub_imm_regs(struct bpf_test *self) | 1481 | static int bpf_fill_alu32_sub_imm_regs(struct bpf_test *self) |
| 1398 | { | 1482 | { |
| 1399 | return __bpf_fill_alu_imm_regs(self, BPF_SUB, true); | 1483 | return __bpf_fill_alu_imm_regs(self, BPF_SUB, F_ALU32); |
| 1400 | } | 1484 | } |
| 1401 | 1485 | ||
| 1402 | static int bpf_fill_alu32_mul_imm_regs(struct bpf_test *self) | 1486 | static int bpf_fill_alu32_mul_imm_regs(struct bpf_test *self) |
| 1403 | { | 1487 | { |
| 1404 | return __bpf_fill_alu_imm_regs(self, BPF_MUL, true); | 1488 | return __bpf_fill_alu_imm_regs(self, BPF_MUL, F_ALU32); |
| 1405 | } | 1489 | } |
| 1406 | 1490 | ||
| 1407 | static int bpf_fill_alu32_div_imm_regs(struct bpf_test *self) | 1491 | static int bpf_fill_alu32_div_imm_regs(struct bpf_test *self) |
| 1408 | { | 1492 | { |
| 1409 | return __bpf_fill_alu_imm_regs(self, BPF_DIV, true); | 1493 | return __bpf_fill_alu_imm_regs(self, BPF_DIV, F_ALU32); |
| 1410 | } | 1494 | } |
| 1411 | 1495 | ||
| 1412 | static int bpf_fill_alu32_mod_imm_regs(struct bpf_test *self) | 1496 | static int bpf_fill_alu32_mod_imm_regs(struct bpf_test *self) |
| 1413 | { | 1497 | { |
| 1414 | return __bpf_fill_alu_imm_regs(self, BPF_MOD, true); | 1498 | return __bpf_fill_alu_imm_regs(self, BPF_MOD, F_ALU32); |
| 1499 | } | ||
| 1500 | |||
| 1501 | /* Signed ALU32 K registers */ | ||
| 1502 | static int bpf_fill_alu32_sdiv_imm_regs(struct bpf_test *self) | ||
| 1503 | { | ||
| 1504 | return __bpf_fill_alu_imm_regs(self, BPF_DIV, F_ALU32 | F_SIGNED); | ||
| 1505 | } | ||
| 1506 | |||
| 1507 | static int bpf_fill_alu32_smod_imm_regs(struct bpf_test *self) | ||
| 1508 | { | ||
| 1509 | return __bpf_fill_alu_imm_regs(self, BPF_MOD, F_ALU32 | F_SIGNED); | ||
| 1415 | } | 1510 | } |
| 1416 | 1511 | ||
| 1417 | /* | 1512 | /* |
| @@ -1442,8 +1537,8 @@ static int __bpf_fill_alu_reg_pairs(struct bpf_test *self, u8 op, bool alu32) · 24 unmodified lines | |||
| 1442 | if (op == BPF_LSH || op == BPF_RSH || op == BPF_ARSH) | 1537 | if (op == BPF_LSH || op == BPF_RSH || op == BPF_ARSH) |
| 1443 | src &= 31; | 1538 | src &= 31; |
| 1444 | 1539 | ||
| 1445 | __bpf_alu_result(&res, dst, src, op); | 1540 | __bpf_alu_result(&res, dst, src, op, 0); |
| 1446 | __bpf_alu_result(&same, src, src, op); | 1541 | __bpf_alu_result(&same, src, src, op, 0); |
| 1447 | 1542 | ||
| 1448 | if (alu32) { | 1543 | if (alu32) { |
| 1449 | res = (u32)res; | 1544 | res = (u32)res; |
| @@ -1626,7 +1721,7 @@ static int __bpf_emit_atomic64(struct bpf_test *self, void *arg, · 176 unmodified lines | |||
| 1626 | res = src; | 1721 | res = src; |
| 1627 | break; | 1722 | break; |
| 1628 | default: | 1723 | default: |
| 1629 | __bpf_alu_result(&res, dst, src, BPF_OP(op)); | 1724 | __bpf_alu_result(&res, dst, src, BPF_OP(op), 0); |
| 1630 | } | 1725 | } |
| 1631 | 1726 | ||
| 1632 | keep = 0x0123456789abcdefULL; | 1727 | keep = 0x0123456789abcdefULL; |
| @@ -1673,7 +1768,7 @@ static int __bpf_emit_atomic32(struct bpf_test *self, void *arg, · 40 unmodified lines | |||
| 1673 | res = src; | 1768 | res = src; |
| 1674 | break; | 1769 | break; |
| 1675 | default: | 1770 | default: |
| 1676 | __bpf_alu_result(&res, (u32)dst, (u32)src, BPF_OP(op)); | 1771 | __bpf_alu_result(&res, (u32)dst, (u32)src, BPF_OP(op), 0); |
| 1677 | } | 1772 | } |
| 1678 | 1773 | ||
| 1679 | keep = 0x0123456789abcdefULL; | 1774 | keep = 0x0123456789abcdefULL; |
| @@ -1939,7 +2034,7 @@ static int __bpf_fill_atomic_reg_pairs(struct bpf_test *self, u8 width, u8 op) · 259 unmodified lines | |||
| 1939 | res = mem; | 2034 | res = mem; |
| 1940 | break; | 2035 | break; |
| 1941 | default: | 2036 | default: |
| 1942 | __bpf_alu_result(&res, mem, upd, BPF_OP(op)); | 2037 | __bpf_alu_result(&res, mem, upd, BPF_OP(op), 0); |
| 1943 | } | 2038 | } |
| 1944 | 2039 | ||
| 1945 | /* Test all operand registers */ | 2040 | /* Test all operand registers */ |
| @@ -12354,6 +12449,22 @@ static struct bpf_test tests[] = { · 10408 unmodified lines | |||
| 12354 | { { 0, 1 } }, | 12449 | { { 0, 1 } }, |
| 12355 | .fill_helper = bpf_fill_alu64_mod_imm_regs, | 12450 | .fill_helper = bpf_fill_alu64_mod_imm_regs, |
| 12356 | }, | 12451 | }, |
| 12452 | { | ||
| 12453 | "ALU64_SDIV_K: registers", | ||
| 12454 | { }, | ||
| 12455 | INTERNAL, | ||
| 12456 | { }, | ||
| 12457 | { { 0, 1 } }, | ||
| 12458 | .fill_helper = bpf_fill_alu64_sdiv_imm_regs, | ||
| 12459 | }, | ||
| 12460 | { | ||
| 12461 | "ALU64_SMOD_K: registers", | ||
| 12462 | { }, | ||
| 12463 | INTERNAL, | ||
| 12464 | { }, | ||
| 12465 | { { 0, 1 } }, | ||
| 12466 | .fill_helper = bpf_fill_alu64_smod_imm_regs, | ||
| 12467 | }, | ||
| 12357 | /* ALU32 K registers */ | 12468 | /* ALU32 K registers */ |
| 12358 | { | 12469 | { |
| 12359 | "ALU32_MOV_K: registers", | 12470 | "ALU32_MOV_K: registers", |
| @@ -12451,6 +12562,22 @@ static struct bpf_test tests[] = { · 91 unmodified lines | |||
| 12451 | { { 0, 1 } }, | 12562 | { { 0, 1 } }, |
| 12452 | .fill_helper = bpf_fill_alu32_mod_imm_regs, | 12563 | .fill_helper = bpf_fill_alu32_mod_imm_regs, |
| 12453 | }, | 12564 | }, |
| 12565 | { | ||
| 12566 | "ALU32_SDIV_K: registers", | ||
| 12567 | { }, | ||
| 12568 | INTERNAL, | ||
| 12569 | { }, | ||
| 12570 | { { 0, 1 } }, | ||
| 12571 | .fill_helper = bpf_fill_alu32_sdiv_imm_regs, | ||
| 12572 | }, | ||
| 12573 | { | ||
| 12574 | "ALU32_SMOD_K: registers", | ||
| 12575 | { }, | ||
| 12576 | INTERNAL, | ||
| 12577 | { }, | ||
| 12578 | { { 0, 1 } }, | ||
| 12579 | .fill_helper = bpf_fill_alu32_smod_imm_regs, | ||
| 12580 | }, | ||
| 12454 | /* ALU64 X register combinations */ | 12581 | /* ALU64 X register combinations */ |
| 12455 | { | 12582 | { |
| 12456 | "ALU64_MOV_X: register combinations", | 12583 | "ALU64_MOV_X: register combinations", |
| @@ -12881,6 +13008,24 @@ static struct bpf_test tests[] = { · 424 unmodified lines | |||
| 12881 | .fill_helper = bpf_fill_alu64_mod_imm, | 13008 | .fill_helper = bpf_fill_alu64_mod_imm, |
| 12882 | .nr_testruns = NR_PATTERN_RUNS, | 13009 | .nr_testruns = NR_PATTERN_RUNS, |
| 12883 | }, | 13010 | }, |
| 13011 | { | ||
| 13012 | "ALU64_SDIV_K: all immediate value magnitudes", | ||
| 13013 | { }, | ||
| 13014 | INTERNAL | FLAG_NO_DATA, | ||
| 13015 | { }, | ||
| 13016 | { { 0, 1 } }, | ||
| 13017 | .fill_helper = bpf_fill_alu64_sdiv_imm, | ||
| 13018 | .nr_testruns = NR_PATTERN_RUNS, | ||
| 13019 | }, | ||
| 13020 | { | ||
| 13021 | "ALU64_SMOD_K: all immediate value magnitudes", | ||
| 13022 | { }, | ||
| 13023 | INTERNAL | FLAG_NO_DATA, | ||
| 13024 | { }, | ||
| 13025 | { { 0, 1 } }, | ||
| 13026 | .fill_helper = bpf_fill_alu64_smod_imm, | ||
| 13027 | .nr_testruns = NR_PATTERN_RUNS, | ||
| 13028 | }, | ||
| 12884 | /* ALU32 immediate magnitudes */ | 13029 | /* ALU32 immediate magnitudes */ |
| 12885 | { | 13030 | { |
| 12886 | "ALU32_MOV_K: all immediate value magnitudes", | 13031 | "ALU32_MOV_K: all immediate value magnitudes", |
| @@ -12963,6 +13108,24 @@ static struct bpf_test tests[] = { · 76 unmodified lines | |||
| 12963 | .fill_helper = bpf_fill_alu32_mod_imm, | 13108 | .fill_helper = bpf_fill_alu32_mod_imm, |
| 12964 | .nr_testruns = NR_PATTERN_RUNS, | 13109 | .nr_testruns = NR_PATTERN_RUNS, |
| 12965 | }, | 13110 | }, |
| 13111 | { | ||
| 13112 | "ALU32_SDIV_K: all immediate value magnitudes", | ||
| 13113 | { }, | ||
| 13114 | INTERNAL | FLAG_NO_DATA, | ||
| 13115 | { }, | ||
| 13116 | { { 0, 1 } }, | ||
| 13117 | .fill_helper = bpf_fill_alu32_sdiv_imm, | ||
| 13118 | .nr_testruns = NR_PATTERN_RUNS, | ||
| 13119 | }, | ||
| 13120 | { | ||
| 13121 | "ALU32_SMOD_K: all immediate value magnitudes", | ||
| 13122 | { }, | ||
| 13123 | INTERNAL | FLAG_NO_DATA, | ||
| 13124 | { }, | ||
| 13125 | { { 0, 1 } }, | ||
| 13126 | .fill_helper = bpf_fill_alu32_smod_imm, | ||
| 13127 | .nr_testruns = NR_PATTERN_RUNS, | ||
| 13128 | }, | ||
| 12966 | /* ALU64 register magnitudes */ | 13129 | /* ALU64 register magnitudes */ |
| 12967 | { | 13130 | { |
| 12968 | "ALU64_MOV_X: all register value magnitudes", | 13131 | "ALU64_MOV_X: all register value magnitudes", |
| @@ -679,6 +679,78 @@ out: · 678 unmodified lines | |||
| 679 | return err; | 679 | return err; |
| 680 | } | 680 | } |
| 681 | 681 | ||
| 682 | static int __init test_rhashtable_next_key(void) | ||
| 683 | { | ||
| 684 | struct rhashtable_params params = test_rht_params; | ||
| 685 | struct test_obj_val key_missing = { .id = 99999, .tid = 0 }; | ||
| 686 | struct test_obj_val *prev_key = NULL; | ||
| 687 | struct rhashtable ht; | ||
| 688 | struct test_obj *objs, *cur; | ||
| 689 | int i, count = 0, err; | ||
| 690 | int visited_keys[8] = { 0 }; | ||
| 691 | const int n = ARRAY_SIZE(visited_keys); | ||
| 692 | |||
| 693 | params.nelem_hint = n; | ||
| 694 | |||
| 695 | err = rhashtable_init(&ht, ¶ms); | ||
| 696 | if (err) | ||
| 697 | return err; | ||
| 698 | |||
| 699 | objs = kcalloc(n, sizeof(*objs), GFP_KERNEL); | ||
| 700 | if (!objs) { | ||
| 701 | rhashtable_destroy(&ht); | ||
| 702 | return -ENOMEM; | ||
| 703 | } | ||
| 704 | |||
| 705 | for (i = 0; i < n; i++) { | ||
| 706 | objs[i].value.id = i; | ||
| 707 | err = rhashtable_insert_fast(&ht, &objs[i].node, params); | ||
| 708 | if (err) | ||
| 709 | goto out; | ||
| 710 | } | ||
| 711 | |||
| 712 | rcu_read_lock(); | ||
| 713 | |||
| 714 | /* NULL prev_key: walk from the beginning, expect all n elements. */ | ||
| 715 | while ((cur = rhashtable_next_key(&ht, prev_key))) { | ||
| 716 | if (IS_ERR(cur)) { | ||
| 717 | err = -EINVAL; | ||
| 718 | goto unlock; | ||
| 719 | } | ||
| 720 | count++; | ||
| 721 | prev_key = &cur->value; | ||
| 722 | visited_keys[cur->value.id] = 1; | ||
| 723 | if (count > n) | ||
| 724 | break; | ||
| 725 | } | ||
| 726 | |||
| 727 | if (count != n) { | ||
| 728 | err = -EINVAL; | ||
| 729 | goto unlock; | ||
| 730 | } | ||
| 731 | |||
| 732 | for (i = 0; i < n; i++) { | ||
| 733 | if (!visited_keys[i]) { | ||
| 734 | err = -EINVAL; | ||
| 735 | goto unlock; | ||
| 736 | } | ||
| 737 | } | ||
| 738 | |||
| 739 | /* Non-existing prev_key: must return ERR_PTR(-ENOENT). */ | ||
| 740 | cur = rhashtable_next_key(&ht, &key_missing); | ||
| 741 | if (!IS_ERR(cur) || PTR_ERR(cur) != -ENOENT) | ||
| 742 | err = -EINVAL; | ||
| 743 | |||
| 744 | unlock: | ||
| 745 | rcu_read_unlock(); | ||
| 746 | out: | ||
| 747 | for (i = 0; i < n; i++) | ||
| 748 | rhashtable_remove_fast(&ht, &objs[i].node, params); | ||
| 749 | kfree(objs); | ||
| 750 | rhashtable_destroy(&ht); | ||
| 751 | return err; | ||
| 752 | } | ||
| 753 | |||
| 682 | static int __init test_rht_init(void) | 754 | static int __init test_rht_init(void) |
| 683 | { | 755 | { |
| 684 | unsigned int entries; | 756 | unsigned int entries; |
| @@ -738,6 +810,9 @@ static int __init test_rht_init(void) · 53 unmodified lines | |||
| 738 | 810 | ||
| 739 | test_insert_duplicates_run(); | 811 | test_insert_duplicates_run(); |
| 740 | 812 | ||
| 813 | pr_info("Testing rhashtable_next_key: %s\n", | ||
| 814 | test_rhashtable_next_key() == 0 ? "pass" : "FAIL"); | ||
| 815 | |||
| 741 | if (!tcount) | 816 | if (!tcount) |
| 742 | return 0; | 817 | return 0; |
| 743 | 818 | ||
| @@ -132,7 +132,7 @@ int bpf_struct_ops_test_run(struct bpf_prog *prog, const union bpf_attr *kattr, · 131 unmodified lines | |||
| 132 | const struct bpf_struct_ops *st_ops = &bpf_bpf_dummy_ops; | 132 | const struct bpf_struct_ops *st_ops = &bpf_bpf_dummy_ops; |
| 133 | const struct btf_type *func_proto; | 133 | const struct btf_type *func_proto; |
| 134 | struct bpf_dummy_ops_test_args *args; | 134 | struct bpf_dummy_ops_test_args *args; |
| 135 | struct bpf_tramp_links *tlinks = NULL; | 135 | struct bpf_tramp_nodes *tnodes = NULL; |
| 136 | struct bpf_tramp_link *link = NULL; | 136 | struct bpf_tramp_link *link = NULL; |
| 137 | void *image = NULL; | 137 | void *image = NULL; |
| 138 | unsigned int op_idx; | 138 | unsigned int op_idx; |
| @@ -158,8 +158,8 @@ int bpf_struct_ops_test_run(struct bpf_prog *prog, const union bpf_attr *kattr, · 19 unmodified lines | |||
| 158 | if (err) | 158 | if (err) |
| 159 | goto out; | 159 | goto out; |
| 160 | 160 | ||
| 161 | tlinks = kzalloc_objs(*tlinks, BPF_TRAMP_MAX); | 161 | tnodes = kzalloc_objs(*tnodes, BPF_TRAMP_MAX); |
| 162 | if (!tlinks) { | 162 | if (!tnodes) { |
| 163 | err = -ENOMEM; | 163 | err = -ENOMEM; |
| 164 | goto out; | 164 | goto out; |
| 165 | } | 165 | } |
| @@ -171,11 +171,11 @@ int bpf_struct_ops_test_run(struct bpf_prog *prog, const union bpf_attr *kattr, · 5 unmodified lines | |||
| 171 | } | 171 | } |
| 172 | /* prog doesn't take the ownership of the reference from caller */ | 172 | /* prog doesn't take the ownership of the reference from caller */ |
| 173 | bpf_prog_inc(prog); | 173 | bpf_prog_inc(prog); |
| 174 | bpf_link_init(&link->link, BPF_LINK_TYPE_STRUCT_OPS, &bpf_struct_ops_link_lops, prog, | 174 | bpf_tramp_link_init(link, BPF_LINK_TYPE_STRUCT_OPS, &bpf_struct_ops_link_lops, |
| 175 | prog->expected_attach_type); | 175 | prog, prog->expected_attach_type, 0); |
| 176 | 176 | ||
| 177 | op_idx = prog->expected_attach_type; | 177 | op_idx = prog->expected_attach_type; |
| 178 | err = bpf_struct_ops_prepare_trampoline(tlinks, link, | 178 | err = bpf_struct_ops_prepare_trampoline(tnodes, &link->node, |
| 179 | &st_ops->func_models[op_idx], | 179 | &st_ops->func_models[op_idx], |
| 180 | &dummy_ops_test_ret_function, | 180 | &dummy_ops_test_ret_function, |
| 181 | &image, &image_off, | 181 | &image, &image_off, |
| @@ -198,7 +198,7 @@ out: · 16 unmodified lines | |||
| 198 | bpf_struct_ops_image_free(image); | 198 | bpf_struct_ops_image_free(image); |
| 199 | if (link) | 199 | if (link) |
| 200 | bpf_link_put(&link->link); | 200 | bpf_link_put(&link->link); |
| 201 | kfree(tlinks); | 201 | kfree(tnodes); |
| 202 | return err; | 202 | return err; |
| 203 | } | 203 | } |
| 204 | 204 | ||
| @@ -702,6 +702,9 @@ int bpf_prog_test_run_tracing(struct bpf_prog *prog, · 701 unmodified lines | |||
| 702 | case BPF_TRACE_FENTRY: | 702 | case BPF_TRACE_FENTRY: |
| 703 | case BPF_TRACE_FEXIT: | 703 | case BPF_TRACE_FEXIT: |
| 704 | case BPF_TRACE_FSESSION: | 704 | case BPF_TRACE_FSESSION: |
| 705 | case BPF_TRACE_FENTRY_MULTI: | ||
| 706 | case BPF_TRACE_FEXIT_MULTI: | ||
| 707 | case BPF_TRACE_FSESSION_MULTI: | ||
| 705 | if (bpf_fentry_test1(1) != 2 || | 708 | if (bpf_fentry_test1(1) != 2 || |
| 706 | bpf_fentry_test2(2, 3) != 5 || | 709 | bpf_fentry_test2(2, 3) != 5 || |
| 707 | bpf_fentry_test3(4, 5, 6) != 15 || | 710 | bpf_fentry_test3(4, 5, 6) != 15 || |
| @@ -747,14 +750,35 @@ static void · 39 unmodified lines | |||
| 747 | __bpf_prog_test_run_raw_tp(void *data) | 750 | __bpf_prog_test_run_raw_tp(void *data) |
| 748 | { | 751 | { |
| 749 | struct bpf_raw_tp_test_run_info *info = data; | 752 | struct bpf_raw_tp_test_run_info *info = data; |
| 753 | struct srcu_ctr __percpu *scp = NULL; | ||
| 750 | struct bpf_trace_run_ctx run_ctx = {}; | 754 | struct bpf_trace_run_ctx run_ctx = {}; |
| 751 | struct bpf_run_ctx *old_run_ctx; | 755 | struct bpf_run_ctx *old_run_ctx; |
| 752 | 756 | ||
| 753 | old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx); | 757 | old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx); |
| 754 | 758 | ||
| 755 | rcu_read_lock(); | 759 | if (info->prog->sleepable) { |
| 760 | scp = rcu_read_lock_tasks_trace(); | ||
| 761 | migrate_disable(); | ||
| 762 | } else { | ||
| 763 | rcu_read_lock(); | ||
| 764 | } | ||
| 765 | |||
| 766 | if (unlikely(!bpf_prog_get_recursion_context(info->prog))) { | ||
| 767 | bpf_prog_inc_misses_counter(info->prog); | ||
| 768 | goto out; | ||
| 769 | } | ||
| 770 | |||
| 756 | info->retval = bpf_prog_run(info->prog, info->ctx); | 771 | info->retval = bpf_prog_run(info->prog, info->ctx); |
| 757 | rcu_read_unlock(); | 772 | |
| 773 | out: | ||
| 774 | bpf_prog_put_recursion_context(info->prog); | ||
| 775 | |||
| 776 | if (info->prog->sleepable) { | ||
| 777 | migrate_enable(); | ||
| 778 | rcu_read_unlock_tasks_trace(scp); | ||
| 779 | } else { | ||
| 780 | rcu_read_unlock(); | ||
| 781 | } | ||
| 758 | 782 | ||
| 759 | bpf_reset_run_ctx(old_run_ctx); | 783 | bpf_reset_run_ctx(old_run_ctx); |
| 760 | } | 784 | } |
| @@ -782,6 +806,13 @@ int bpf_prog_test_run_raw_tp(struct bpf_prog *prog, · 21 unmodified lines | |||
| 782 | if ((kattr->test.flags & BPF_F_TEST_RUN_ON_CPU) == 0 && cpu != 0) | 806 | if ((kattr->test.flags & BPF_F_TEST_RUN_ON_CPU) == 0 && cpu != 0) |
| 783 | return -EINVAL; | 807 | return -EINVAL; |
| 784 | 808 | ||
| 809 | /* | ||
| 810 | * Sleepable programs cannot run with preemption disabled or in | ||
| 811 | * hardirq context (smp_call_function_single), reject the flag. | ||
| 812 | */ | ||
| 813 | if (prog->sleepable && (kattr->test.flags & BPF_F_TEST_RUN_ON_CPU)) | ||
| 814 | return -EINVAL; | ||
| 815 | |||
| 785 | if (ctx_size_in) { | 816 | if (ctx_size_in) { |
| 786 | info.ctx = memdup_user(ctx_in, ctx_size_in); | 817 | info.ctx = memdup_user(ctx_in, ctx_size_in); |
| 787 | if (IS_ERR(info.ctx)) | 818 | if (IS_ERR(info.ctx)) |
| @@ -790,24 +821,31 @@ int bpf_prog_test_run_raw_tp(struct bpf_prog *prog, · 2 unmodified lines | |||
| 790 | info.ctx = NULL; | 821 | info.ctx = NULL; |
| 791 | } | 822 | } |
| 792 | 823 | ||
| 824 | info.retval = 0; | ||
| 793 | info.prog = prog; | 825 | info.prog = prog; |
| 794 | 826 | ||
| 795 | current_cpu = get_cpu(); | 827 | if (prog->sleepable) { |
| 796 | if ((kattr->test.flags & BPF_F_TEST_RUN_ON_CPU) == 0 || | ||
| 797 | cpu == current_cpu) { | ||
| 798 | __bpf_prog_test_run_raw_tp(&info); | 828 | __bpf_prog_test_run_raw_tp(&info); |
| 799 | } else if (cpu >= nr_cpu_ids || !cpu_online(cpu)) { | ||
| 800 | /* smp_call_function_single() also checks cpu_online() | ||
| 801 | * after csd_lock(). However, since cpu is from user | ||
| 802 | * space, let's do an extra quick check to filter out | ||
| 803 | * invalid value before smp_call_function_single(). | ||
| 804 | */ | ||
| 805 | err = -ENXIO; | ||
| 806 | } else { | 829 | } else { |
| 807 | err = smp_call_function_single(cpu, __bpf_prog_test_run_raw_tp, | 830 | current_cpu = get_cpu(); |
| 808 | &info, 1); | 831 | if ((kattr->test.flags & BPF_F_TEST_RUN_ON_CPU) == 0 || |
| 832 | cpu == current_cpu) { | ||
| 833 | __bpf_prog_test_run_raw_tp(&info); | ||
| 834 | } else if (cpu >= nr_cpu_ids || !cpu_online(cpu)) { | ||
| 835 | /* | ||
| 836 | * smp_call_function_single() also checks cpu_online() | ||
| 837 | * after csd_lock(). However, since cpu is from user | ||
| 838 | * space, let's do an extra quick check to filter out | ||
| 839 | * invalid value before smp_call_function_single(). | ||
| 840 | */ | ||
| 841 | err = -ENXIO; | ||
| 842 | } else { | ||
| 843 | err = smp_call_function_single(cpu, | ||
| 844 | __bpf_prog_test_run_raw_tp, | ||
| 845 | &info, 1); | ||
| 846 | } | ||
| 847 | put_cpu(); | ||
| 809 | } | 848 | } |
| 810 | put_cpu(); | ||
| 811 | 849 | ||
| 812 | if (!err && | 850 | if (!err && |
| 813 | copy_to_user(&uattr->test.retval, &info.retval, sizeof(u32))) | 851 | copy_to_user(&uattr->test.retval, &info.retval, sizeof(u32))) |
| @@ -2654,6 +2654,37 @@ static void sk_msg_reset_curr(struct sk_msg *msg) · 2653 unmodified lines | |||
| 2654 | } | 2654 | } |
| 2655 | } | 2655 | } |
| 2656 | 2656 | ||
| 2657 | static bool sk_msg_elem_is_copy(const struct sk_msg *msg, u32 i) | ||
| 2658 | { | ||
| 2659 | return test_bit(i, msg->sg.copy); | ||
| 2660 | } | ||
| 2661 | |||
| 2662 | static void sk_msg_clear_elem_copy(struct sk_msg *msg, u32 i) | ||
| 2663 | { | ||
| 2664 | __clear_bit(i, msg->sg.copy); | ||
| 2665 | } | ||
| 2666 | |||
| 2667 | static void sk_msg_set_elem_copy(struct sk_msg *msg, u32 i, bool sg_copy) | ||
| 2668 | { | ||
| 2669 | __assign_bit(i, msg->sg.copy, sg_copy); | ||
| 2670 | } | ||
| 2671 | |||
| 2672 | static void sk_msg_clear_copy_range(struct sk_msg *msg, u32 start, u32 end) | ||
| 2673 | { | ||
| 2674 | while (start != end) { | ||
| 2675 | sk_msg_clear_elem_copy(msg, start); | ||
| 2676 | sk_msg_iter_var_next(start); | ||
| 2677 | } | ||
| 2678 | } | ||
| 2679 | |||
| 2680 | static void sk_msg_sg_move(struct sk_msg *msg, u32 dst, u32 src) | ||
| 2681 | { | ||
| 2682 | msg->sg.data[dst] = msg->sg.data[src]; | ||
| 2683 | |||
| 2684 | sk_msg_set_elem_copy(msg, dst, | ||
| 2685 | sk_msg_elem_is_copy(msg, src)); | ||
| 2686 | } | ||
| 2687 | |||
| 2657 | static const struct bpf_func_proto bpf_msg_cork_bytes_proto = { | 2688 | static const struct bpf_func_proto bpf_msg_cork_bytes_proto = { |
| 2658 | .func = bpf_msg_cork_bytes, | 2689 | .func = bpf_msg_cork_bytes, |
| 2659 | .gpl_only = false, | 2690 | .gpl_only = false, |
| @@ -2692,7 +2723,7 @@ BPF_CALL_4(bpf_msg_pull_data, struct sk_msg *, msg, u32, start, · 32 unmodified lines | |||
| 2692 | * account for the headroom. | 2723 | * account for the headroom. |
| 2693 | */ | 2724 | */ |
| 2694 | bytes_sg_total = start - offset + bytes; | 2725 | bytes_sg_total = start - offset + bytes; |
| 2695 | if (!test_bit(i, msg->sg.copy) && bytes_sg_total <= len) | 2726 | if (!sk_msg_elem_is_copy(msg, i) && bytes_sg_total <= len) |
| 2696 | goto out; | 2727 | goto out; |
| 2697 | 2728 | ||
| 2698 | /* At this point we need to linearize multiple scatterlist | 2729 | /* At this point we need to linearize multiple scatterlist |
| @@ -2733,13 +2764,13 @@ BPF_CALL_4(bpf_msg_pull_data, struct sk_msg *, msg, u32, start, · 34 unmodified lines | |||
| 2733 | poffset += len; | 2764 | poffset += len; |
| 2734 | sge->length = 0; | 2765 | sge->length = 0; |
| 2735 | put_page(sg_page(sge)); | 2766 | put_page(sg_page(sge)); |
| 2736 | __clear_bit(i, msg->sg.copy); | 2767 | sk_msg_clear_elem_copy(msg, i); |
| 2737 | 2768 | ||
| 2738 | sk_msg_iter_var_next(i); | 2769 | sk_msg_iter_var_next(i); |
| 2739 | } while (i != last_sge); | 2770 | } while (i != last_sge); |
| 2740 | 2771 | ||
| 2741 | sg_set_page(&msg->sg.data[first_sge], page, copy, 0); | 2772 | sg_set_page(&msg->sg.data[first_sge], page, copy, 0); |
| 2742 | __clear_bit(first_sge, msg->sg.copy); | 2773 | sk_msg_clear_elem_copy(msg, first_sge); |
| 2743 | 2774 | ||
| 2744 | /* To repair sg ring we need to shift entries. If we only | 2775 | /* To repair sg ring we need to shift entries. If we only |
| 2745 | * had a single entry though we can just replace it and | 2776 | * had a single entry though we can just replace it and |
| @@ -2749,8 +2780,14 @@ BPF_CALL_4(bpf_msg_pull_data, struct sk_msg *, msg, u32, start, · 3 unmodified lines | |||
| 2749 | shift = last_sge > first_sge ? | 2780 | shift = last_sge > first_sge ? |
| 2750 | last_sge - first_sge - 1 : | 2781 | last_sge - first_sge - 1 : |
| 2751 | NR_MSG_FRAG_IDS - first_sge + last_sge - 1; | 2782 | NR_MSG_FRAG_IDS - first_sge + last_sge - 1; |
| 2752 | if (!shift) | 2783 | if (!shift) { |
| 2784 | sk_msg_clear_elem_copy(msg, msg->sg.end); | ||
| 2753 | goto out; | 2785 | goto out; |
| 2786 | } | ||
| 2787 | |||
| 2788 | i = first_sge; | ||
| 2789 | sk_msg_iter_var_next(i); | ||
| 2790 | sk_msg_clear_copy_range(msg, i, last_sge); | ||
| 2754 | 2791 | ||
| 2755 | i = first_sge; | 2792 | i = first_sge; |
| 2756 | sk_msg_iter_var_next(i); | 2793 | sk_msg_iter_var_next(i); |
| @@ -2764,18 +2801,18 @@ BPF_CALL_4(bpf_msg_pull_data, struct sk_msg *, msg, u32, start, · 7 unmodified lines | |||
| 2764 | if (move_from == msg->sg.end) | 2801 | if (move_from == msg->sg.end) |
| 2765 | break; | 2802 | break; |
| 2766 | 2803 | ||
| 2767 | msg->sg.data[i] = msg->sg.data[move_from]; | 2804 | sk_msg_sg_move(msg, i, move_from); |
| 2768 | sk_msg_sg_copy_assign(msg, i, msg, move_from); | ||
| 2769 | msg->sg.data[move_from].length = 0; | 2805 | msg->sg.data[move_from].length = 0; |
| 2770 | msg->sg.data[move_from].page_link = 0; | 2806 | msg->sg.data[move_from].page_link = 0; |
| 2771 | msg->sg.data[move_from].offset = 0; | 2807 | msg->sg.data[move_from].offset = 0; |
| 2772 | __clear_bit(move_from, msg->sg.copy); | 2808 | sk_msg_clear_elem_copy(msg, move_from); |
| 2773 | sk_msg_iter_var_next(i); | 2809 | sk_msg_iter_var_next(i); |
| 2774 | } while (1); | 2810 | } while (1); |
| 2775 | 2811 | ||
| 2776 | msg->sg.end = msg->sg.end - shift > msg->sg.end ? | 2812 | msg->sg.end = msg->sg.end - shift > msg->sg.end ? |
| 2777 | msg->sg.end - shift + NR_MSG_FRAG_IDS : | 2813 | msg->sg.end - shift + NR_MSG_FRAG_IDS : |
| 2778 | msg->sg.end - shift; | 2814 | msg->sg.end - shift; |
| 2815 | sk_msg_clear_elem_copy(msg, msg->sg.end); | ||
| 2779 | out: | 2816 | out: |
| 2780 | sk_msg_reset_curr(msg); | 2817 | sk_msg_reset_curr(msg); |
| 2781 | msg->data = sg_virt(&msg->sg.data[first_sge]) + start - offset; | 2818 | msg->data = sg_virt(&msg->sg.data[first_sge]) + start - offset; |
| @@ -2796,9 +2833,10 @@ static const struct bpf_func_proto bpf_msg_pull_data_proto = { · 14 unmodified lines | |||
| 2796 | BPF_CALL_4(bpf_msg_push_data, struct sk_msg *, msg, u32, start, | 2833 | BPF_CALL_4(bpf_msg_push_data, struct sk_msg *, msg, u32, start, |
| 2797 | u32, len, u64, flags) | 2834 | u32, len, u64, flags) |
| 2798 | { | 2835 | { |
| 2836 | bool sge_copy = false, nsge_copy = false, nnsge_copy = false; | ||
| 2799 | struct scatterlist sge, nsge, nnsge, rsge = {0}, *psge; | 2837 | struct scatterlist sge, nsge, nnsge, rsge = {0}, *psge; |
| 2800 | u32 new, i = 0, l = 0, space, copy = 0, offset = 0; | 2838 | u32 new, i = 0, l = 0, space, copy = 0, offset = 0; |
| 2801 | bool sge_copy, nsge_copy, nnsge_copy, rsge_copy = false; | 2839 | bool rsge_copy = false; |
| 2802 | u8 *raw, *to, *from; | 2840 | u8 *raw, *to, *from; |
| 2803 | struct page *page; | 2841 | struct page *page; |
| 2804 | 2842 | ||
| @@ -2834,6 +2872,9 @@ BPF_CALL_4(bpf_msg_push_data, struct sk_msg *, msg, u32, start, · 29 unmodified lines | |||
| 2834 | if (!space || (space == 1 && start != offset)) | 2872 | if (!space || (space == 1 && start != offset)) |
| 2835 | copy = msg->sg.data[i].length; | 2873 | copy = msg->sg.data[i].length; |
| 2836 | 2874 | ||
| 2875 | if (unlikely(copy + len < copy)) | ||
| 2876 | return -EINVAL; | ||
| 2877 | |||
| 2837 | page = alloc_pages(__GFP_NOWARN | GFP_ATOMIC | __GFP_COMP, | 2878 | page = alloc_pages(__GFP_NOWARN | GFP_ATOMIC | __GFP_COMP, |
| 2838 | get_order(copy + len)); | 2879 | get_order(copy + len)); |
| 2839 | if (unlikely(!page)) | 2880 | if (unlikely(!page)) |
| @@ -2871,7 +2912,7 @@ BPF_CALL_4(bpf_msg_push_data, struct sk_msg *, msg, u32, start, · 31 unmodified lines | |||
| 2871 | sk_msg_iter_var_prev(i); | 2912 | sk_msg_iter_var_prev(i); |
| 2872 | psge = sk_msg_elem(msg, i); | 2913 | psge = sk_msg_elem(msg, i); |
| 2873 | rsge = sk_msg_elem_cpy(msg, i); | 2914 | rsge = sk_msg_elem_cpy(msg, i); |
| 2874 | rsge_copy = test_bit(i, msg->sg.copy); | 2915 | rsge_copy = sk_msg_elem_is_copy(msg, i); |
| 2875 | 2916 | ||
| 2876 | psge->length = start - offset; | 2917 | psge->length = start - offset; |
| 2877 | rsge.length -= psge->length; | 2918 | rsge.length -= psge->length; |
| @@ -2896,21 +2937,21 @@ BPF_CALL_4(bpf_msg_push_data, struct sk_msg *, msg, u32, start, · 18 unmodified lines | |||
| 2896 | 2937 | ||
| 2897 | /* Shift one or two slots as needed */ | 2938 | /* Shift one or two slots as needed */ |
| 2898 | sge = sk_msg_elem_cpy(msg, new); | 2939 | sge = sk_msg_elem_cpy(msg, new); |
| 2899 | sge_copy = test_bit(new, msg->sg.copy); | ||
| 2900 | sg_unmark_end(&sge); | 2940 | sg_unmark_end(&sge); |
| 2941 | sge_copy = sk_msg_elem_is_copy(msg, new); | ||
| 2901 | 2942 | ||
| 2902 | nsge = sk_msg_elem_cpy(msg, i); | 2943 | nsge = sk_msg_elem_cpy(msg, i); |
| 2903 | nsge_copy = test_bit(i, msg->sg.copy); | 2944 | nsge_copy = sk_msg_elem_is_copy(msg, i); |
| 2904 | if (rsge.length) { | 2945 | if (rsge.length) { |
| 2905 | sk_msg_iter_var_next(i); | 2946 | sk_msg_iter_var_next(i); |
| 2906 | nnsge = sk_msg_elem_cpy(msg, i); | 2947 | nnsge = sk_msg_elem_cpy(msg, i); |
| 2907 | nnsge_copy = test_bit(i, msg->sg.copy); | 2948 | nnsge_copy = sk_msg_elem_is_copy(msg, i); |
| 2908 | sk_msg_iter_next(msg, end); | 2949 | sk_msg_iter_next(msg, end); |
| 2909 | } | 2950 | } |
| 2910 | 2951 | ||
| 2911 | while (i != msg->sg.end) { | 2952 | while (i != msg->sg.end) { |
| 2912 | msg->sg.data[i] = sge; | 2953 | msg->sg.data[i] = sge; |
| 2913 | __assign_bit(i, msg->sg.copy, sge_copy); | 2954 | sk_msg_set_elem_copy(msg, i, sge_copy); |
| 2914 | sge = nsge; | 2955 | sge = nsge; |
| 2915 | sge_copy = nsge_copy; | 2956 | sge_copy = nsge_copy; |
| 2916 | sk_msg_iter_var_next(i); | 2957 | sk_msg_iter_var_next(i); |
| @@ -2918,10 +2959,10 @@ BPF_CALL_4(bpf_msg_push_data, struct sk_msg *, msg, u32, start, · 1 unmodified line | |||
| 2918 | nsge = nnsge; | 2959 | nsge = nnsge; |
| 2919 | nsge_copy = nnsge_copy; | 2960 | nsge_copy = nnsge_copy; |
| 2920 | nnsge = sk_msg_elem_cpy(msg, i); | 2961 | nnsge = sk_msg_elem_cpy(msg, i); |
| 2921 | nnsge_copy = test_bit(i, msg->sg.copy); | 2962 | nnsge_copy = sk_msg_elem_is_copy(msg, i); |
| 2922 | } else { | 2963 | } else { |
| 2923 | nsge = sk_msg_elem_cpy(msg, i); | 2964 | nsge = sk_msg_elem_cpy(msg, i); |
| 2924 | nsge_copy = test_bit(i, msg->sg.copy); | 2965 | nsge_copy = sk_msg_elem_is_copy(msg, i); |
| 2925 | } | 2966 | } |
| 2926 | } | 2967 | } |
| 2927 | 2968 | ||
| @@ -2929,14 +2970,15 @@ place_new: · 1 unmodified line | |||
| 2929 | /* Place newly allocated data buffer */ | 2970 | /* Place newly allocated data buffer */ |
| 2930 | sk_mem_charge(msg->sk, len); | 2971 | sk_mem_charge(msg->sk, len); |
| 2931 | msg->sg.size += len; | 2972 | msg->sg.size += len; |
| 2932 | __clear_bit(new, msg->sg.copy); | 2973 | sk_msg_clear_elem_copy(msg, new); |
| 2933 | sg_set_page(&msg->sg.data[new], page, len + copy, 0); | 2974 | sg_set_page(&msg->sg.data[new], page, len + copy, 0); |
| 2934 | if (rsge.length) { | 2975 | if (rsge.length) { |
| 2935 | get_page(sg_page(&rsge)); | 2976 | get_page(sg_page(&rsge)); |
| 2936 | sk_msg_iter_var_next(new); | 2977 | sk_msg_iter_var_next(new); |
| 2937 | msg->sg.data[new] = rsge; | 2978 | msg->sg.data[new] = rsge; |
| 2938 | __assign_bit(new, msg->sg.copy, rsge_copy); | 2979 | sk_msg_set_elem_copy(msg, new, rsge_copy); |
| 2939 | } | 2980 | } |
| 2981 | sk_msg_clear_elem_copy(msg, msg->sg.end); | ||
| 2940 | 2982 | ||
| 2941 | sk_msg_reset_curr(msg); | 2983 | sk_msg_reset_curr(msg); |
| 2942 | sk_msg_compute_data_pointers(msg); | 2984 | sk_msg_compute_data_pointers(msg); |
| @@ -2962,12 +3004,11 @@ static void sk_msg_shift_left(struct sk_msg *msg, int i) · 19 unmodified lines | |||
| 2962 | do { | 3004 | do { |
| 2963 | prev = i; | 3005 | prev = i; |
| 2964 | sk_msg_iter_var_next(i); | 3006 | sk_msg_iter_var_next(i); |
| 2965 | msg->sg.data[prev] = msg->sg.data[i]; | 3007 | sk_msg_sg_move(msg, prev, i); |
| 2966 | sk_msg_sg_copy_assign(msg, prev, msg, i); | ||
| 2967 | } while (i != msg->sg.end); | 3008 | } while (i != msg->sg.end); |
| 2968 | 3009 | ||
| 2969 | sk_msg_iter_prev(msg, end); | 3010 | sk_msg_iter_prev(msg, end); |
| 2970 | __clear_bit(msg->sg.end, msg->sg.copy); | 3011 | sk_msg_clear_elem_copy(msg, msg->sg.end); |
| 2971 | } | 3012 | } |
| 2972 | 3013 | ||
| 2973 | static void sk_msg_shift_right(struct sk_msg *msg, int i) | 3014 | static void sk_msg_shift_right(struct sk_msg *msg, int i) |
| @@ -2977,28 +3018,29 @@ static void sk_msg_shift_right(struct sk_msg *msg, int i) · 3 unmodified lines | |||
| 2977 | 3018 | ||
| 2978 | sk_msg_iter_next(msg, end); | 3019 | sk_msg_iter_next(msg, end); |
| 2979 | sge = sk_msg_elem_cpy(msg, i); | 3020 | sge = sk_msg_elem_cpy(msg, i); |
| 2980 | sge_copy = test_bit(i, msg->sg.copy); | 3021 | sge_copy = sk_msg_elem_is_copy(msg, i); |
| 2981 | sk_msg_iter_var_next(i); | 3022 | sk_msg_iter_var_next(i); |
| 2982 | tmp = sk_msg_elem_cpy(msg, i); | 3023 | tmp = sk_msg_elem_cpy(msg, i); |
| 2983 | tmp_copy = test_bit(i, msg->sg.copy); | 3024 | tmp_copy = sk_msg_elem_is_copy(msg, i); |
| 2984 | 3025 | ||
| 2985 | while (i != msg->sg.end) { | 3026 | while (i != msg->sg.end) { |
| 2986 | msg->sg.data[i] = sge; | 3027 | msg->sg.data[i] = sge; |
| 2987 | __assign_bit(i, msg->sg.copy, sge_copy); | 3028 | sk_msg_set_elem_copy(msg, i, sge_copy); |
| 2988 | sk_msg_iter_var_next(i); | 3029 | sk_msg_iter_var_next(i); |
| 2989 | sge = tmp; | 3030 | sge = tmp; |
| 2990 | sge_copy = tmp_copy; | 3031 | sge_copy = tmp_copy; |
| 2991 | tmp = sk_msg_elem_cpy(msg, i); | 3032 | tmp = sk_msg_elem_cpy(msg, i); |
| 2992 | tmp_copy = test_bit(i, msg->sg.copy); | 3033 | tmp_copy = sk_msg_elem_is_copy(msg, i); |
| 2993 | } | 3034 | } |
| 3035 | sk_msg_clear_elem_copy(msg, msg->sg.end); | ||
| 2994 | } | 3036 | } |
| 2995 | 3037 | ||
| 2996 | BPF_CALL_4(bpf_msg_pop_data, struct sk_msg *, msg, u32, start, | 3038 | BPF_CALL_4(bpf_msg_pop_data, struct sk_msg *, msg, u32, start, |
| 2997 | u32, len, u64, flags) | 3039 | u32, len, u64, flags) |
| 2998 | { | 3040 | { |
| 2999 | u32 i = 0, l = 0, space, offset = 0; | 3041 | u32 i = 0, l = 0, space, offset = 0; |
| 3000 | u64 last = start + len; | 3042 | u64 last = (u64)start + len; |
| 3001 | int pop; | 3043 | u32 pop; |
| 3002 | 3044 | ||
| 3003 | if (unlikely(flags)) | 3045 | if (unlikely(flags)) |
| 3004 | return -EINVAL; | 3046 | return -EINVAL; |
| @@ -3047,10 +3089,10 @@ BPF_CALL_4(bpf_msg_pop_data, struct sk_msg *, msg, u32, start, · 42 unmodified lines | |||
| 3047 | */ | 3089 | */ |
| 3048 | if (start != offset) { | 3090 | if (start != offset) { |
| 3049 | struct scatterlist *nsge, *sge = sk_msg_elem(msg, i); | 3091 | struct scatterlist *nsge, *sge = sk_msg_elem(msg, i); |
| 3092 | bool sge_copy = sk_msg_elem_is_copy(msg, i); | ||
| 3050 | int a = start - offset; | 3093 | int a = start - offset; |
| 3051 | int b = sge->length - pop - a; | 3094 | int b = sge->length - pop - a; |
| 3052 | u32 sge_i = i; | 3095 | u32 sge_idx = i; |
| 3053 | bool sge_copy = test_bit(i, msg->sg.copy); | ||
| 3054 | 3096 | ||
| 3055 | sk_msg_iter_var_next(i); | 3097 | sk_msg_iter_var_next(i); |
| 3056 | 3098 | ||
| @@ -3063,7 +3105,7 @@ BPF_CALL_4(bpf_msg_pop_data, struct sk_msg *, msg, u32, start, · 6 unmodified lines | |||
| 3063 | sg_set_page(nsge, | 3105 | sg_set_page(nsge, |
| 3064 | sg_page(sge), | 3106 | sg_page(sge), |
| 3065 | b, sge->offset + pop + a); | 3107 | b, sge->offset + pop + a); |
| 3066 | __assign_bit(i, msg->sg.copy, sge_copy); | 3108 | sk_msg_set_elem_copy(msg, i, sge_copy); |
| 3067 | } else { | 3109 | } else { |
| 3068 | struct page *page, *orig; | 3110 | struct page *page, *orig; |
| 3069 | u8 *to, *from; | 3111 | u8 *to, *from; |
| @@ -3080,7 +3122,7 @@ BPF_CALL_4(bpf_msg_pop_data, struct sk_msg *, msg, u32, start, · 10 unmodified lines | |||
| 3080 | memcpy(to, from, a); | 3122 | memcpy(to, from, a); |
| 3081 | memcpy(to + a, from + a + pop, b); | 3123 | memcpy(to + a, from + a + pop, b); |
| 3082 | sg_set_page(sge, page, a + b, 0); | 3124 | sg_set_page(sge, page, a + b, 0); |
| 3083 | __clear_bit(sge_i, msg->sg.copy); | 3125 | sk_msg_clear_elem_copy(msg, sge_idx); |
| 3084 | put_page(orig); | 3126 | put_page(orig); |
| 3085 | } | 3127 | } |
| 3086 | pop = 0; | 3128 | pop = 0; |
| @@ -5571,11 +5613,24 @@ static int sol_tcp_sockopt(struct sock *sk, int optname, · 2484 unmodified lines | |||
| 5571 | KERNEL_SOCKPTR(optval), *optlen); | 5613 | KERNEL_SOCKPTR(optval), *optlen); |
| 5572 | } | 5614 | } |
| 5573 | 5615 | ||
| 5616 | static bool sk_allows_sol_ip_sockopt(struct sock *sk) | ||
| 5617 | { | ||
| 5618 | switch (sk->sk_family) { | ||
| 5619 | case AF_INET: | ||
| 5620 | return true; | ||
| 5621 | case AF_INET6: | ||
| 5622 | /* Allow getting/setting sockopt for possible ipv4-mapped ipv6 socket. */ | ||
| 5623 | return sk->sk_type != SOCK_RAW && !ipv6_only_sock(sk); | ||
| 5624 | default: | ||
| 5625 | return false; | ||
| 5626 | } | ||
| 5627 | } | ||
| 5628 | |||
| 5574 | static int sol_ip_sockopt(struct sock *sk, int optname, | 5629 | static int sol_ip_sockopt(struct sock *sk, int optname, |
| 5575 | char *optval, int *optlen, | 5630 | char *optval, int *optlen, |
| 5576 | bool getopt) | 5631 | bool getopt) |
| 5577 | { | 5632 | { |
| 5578 | if (sk->sk_family != AF_INET) | 5633 | if (!sk_allows_sol_ip_sockopt(sk)) |
| 5579 | return -EINVAL; | 5634 | return -EINVAL; |
| 5580 | 5635 | ||
| 5581 | switch (optname) { | 5636 | switch (optname) { |
| @@ -599,6 +599,7 @@ static int handle_gso_encap(struct sk_buff *skb, bool ipv4, int encap_len) · 598 unmodified lines | |||
| 599 | 599 | ||
| 600 | int bpf_lwt_push_ip_encap(struct sk_buff *skb, void *hdr, u32 len, bool ingress) | 600 | int bpf_lwt_push_ip_encap(struct sk_buff *skb, void *hdr, u32 len, bool ingress) |
| 601 | { | 601 | { |
| 602 | bool is_udp_tunnel; | ||
| 602 | struct iphdr *iph; | 603 | struct iphdr *iph; |
| 603 | bool ipv4; | 604 | bool ipv4; |
| 604 | int err; | 605 | int err; |
| @@ -612,10 +613,16 @@ int bpf_lwt_push_ip_encap(struct sk_buff *skb, void *hdr, u32 len, bool ingress) · 7 unmodified lines | |||
| 612 | ipv4 = true; | 613 | ipv4 = true; |
| 613 | if (unlikely(len < iph->ihl * 4)) | 614 | if (unlikely(len < iph->ihl * 4)) |
| 614 | return -EINVAL; | 615 | return -EINVAL; |
| 616 | is_udp_tunnel = iph->protocol == IPPROTO_UDP; | ||
| 617 | if (unlikely(is_udp_tunnel && len < iph->ihl * 4 + sizeof(struct udphdr))) | ||
| 618 | return -EINVAL; | ||
| 615 | } else if (iph->version == 6) { | 619 | } else if (iph->version == 6) { |
| 616 | ipv4 = false; | 620 | ipv4 = false; |
| 617 | if (unlikely(len < sizeof(struct ipv6hdr))) | 621 | if (unlikely(len < sizeof(struct ipv6hdr))) |
| 618 | return -EINVAL; | 622 | return -EINVAL; |
| 623 | is_udp_tunnel = ((struct ipv6hdr *)iph)->nexthdr == NEXTHDR_UDP; | ||
| 624 | if (unlikely(is_udp_tunnel && len < sizeof(struct ipv6hdr) + sizeof(struct udphdr))) | ||
| 625 | return -EINVAL; | ||
| 619 | } else { | 626 | } else { |
| 620 | return -EINVAL; | 627 | return -EINVAL; |
| 621 | } | 628 | } |
| @@ -637,6 +644,11 @@ int bpf_lwt_push_ip_encap(struct sk_buff *skb, void *hdr, u32 len, bool ingress) · 15 unmodified lines | |||
| 637 | if (ingress) | 644 | if (ingress) |
| 638 | skb_postpush_rcsum(skb, iph, len); | 645 | skb_postpush_rcsum(skb, iph, len); |
| 639 | skb_reset_network_header(skb); | 646 | skb_reset_network_header(skb); |
| 647 | if (is_udp_tunnel) { | ||
| 648 | size_t iph_sz = ipv4 ? iph->ihl * 4 : sizeof(struct ipv6hdr); | ||
| 649 | |||
| 650 | skb_set_transport_header(skb, skb_network_offset(skb) + iph_sz); | ||
| 651 | } | ||
| 640 | memcpy(skb_network_header(skb), hdr, len); | 652 | memcpy(skb_network_header(skb), hdr, len); |
| 641 | bpf_compute_data_pointers(skb); | 653 | bpf_compute_data_pointers(skb); |
| 642 | skb_clear_hash(skb); | 654 | skb_clear_hash(skb); |
| @@ -50,7 +50,9 @@ static int udp_msg_wait_data(struct sock *sk, struct sk_psock *psock, · 49 unmodified lines | |||
| 50 | sk_set_bit(SOCKWQ_ASYNC_WAITDATA, sk); | 50 | sk_set_bit(SOCKWQ_ASYNC_WAITDATA, sk); |
| 51 | ret = udp_msg_has_data(sk, psock); | 51 | ret = udp_msg_has_data(sk, psock); |
| 52 | if (!ret) { | 52 | if (!ret) { |
| 53 | release_sock(sk); | ||
| 53 | wait_woken(&wait, TASK_INTERRUPTIBLE, timeo); | 54 | wait_woken(&wait, TASK_INTERRUPTIBLE, timeo); |
| 55 | lock_sock(sk); | ||
| 54 | ret = udp_msg_has_data(sk, psock); | 56 | ret = udp_msg_has_data(sk, psock); |
| 55 | } | 57 | } |
| 56 | sk_clear_bit(SOCKWQ_ASYNC_WAITDATA, sk); | 58 | sk_clear_bit(SOCKWQ_ASYNC_WAITDATA, sk); |
| @@ -79,6 +81,7 @@ static int udp_bpf_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, · 22 unmodified lines | |||
| 79 | goto out; | 81 | goto out; |
| 80 | } | 82 | } |
| 81 | 83 | ||
| 84 | lock_sock(sk); | ||
| 82 | msg_bytes_ready: | 85 | msg_bytes_ready: |
| 83 | copied = sk_msg_recvmsg(sk, psock, msg, len, flags); | 86 | copied = sk_msg_recvmsg(sk, psock, msg, len, flags); |
| 84 | if (!copied) { | 87 | if (!copied) { |
| @@ -90,11 +93,17 @@ msg_bytes_ready: · 5 unmodified lines | |||
| 90 | if (data) { | 93 | if (data) { |
| 91 | if (psock_has_data(psock)) | 94 | if (psock_has_data(psock)) |
| 92 | goto msg_bytes_ready; | 95 | goto msg_bytes_ready; |
| 96 | |||
| 97 | release_sock(sk); | ||
| 98 | |||
| 93 | ret = sk_udp_recvmsg(sk, msg, len, flags); | 99 | ret = sk_udp_recvmsg(sk, msg, len, flags); |
| 94 | goto out; | 100 | goto out; |
| 95 | } | 101 | } |
| 96 | copied = -EAGAIN; | 102 | copied = -EAGAIN; |
| 97 | } | 103 | } |
| 104 | |||
| 105 | release_sock(sk); | ||
| 106 | |||
| 98 | ret = copied; | 107 | ret = copied; |
| 99 | out: | 108 | out: |
| 100 | sk_psock_put(sk, psock); | 109 | sk_psock_put(sk, psock); |
| @@ -11,6 +11,7 @@ INSTALL ?= install · 10 unmodified lines | |||
| 11 | CFLAGS += -Wall -O2 | 11 | CFLAGS += -Wall -O2 |
| 12 | CFLAGS += -D__EXPORTED_HEADERS__ -I$(srctree)/tools/include/uapi \ | 12 | CFLAGS += -D__EXPORTED_HEADERS__ -I$(srctree)/tools/include/uapi \ |
| 13 | -I$(srctree)/tools/include | 13 | -I$(srctree)/tools/include |
| 14 | CFLAGS += $(EXTRA_CFLAGS) | ||
| 14 | 15 | ||
| 15 | # This will work when bpf is built in tools env. where srctree | 16 | # This will work when bpf is built in tools env. where srctree |
| 16 | # isn't set and when invoked from selftests build, where srctree | 17 | # isn't set and when invoked from selftests build, where srctree |
| @@ -56,7 +56,7 @@ MAP COMMANDS · 55 unmodified lines | |||
| 56 | | | **cgroup_storage** | **reuseport_sockarray** | **percpu_cgroup_storage** | 56 | | | **cgroup_storage** | **reuseport_sockarray** | **percpu_cgroup_storage** |
| 57 | | | **queue** | **stack** | **sk_storage** | **struct_ops** | **ringbuf** | **inode_storage** | 57 | | | **queue** | **stack** | **sk_storage** | **struct_ops** | **ringbuf** | **inode_storage** |
| 58 | | | **task_storage** | **bloom_filter** | **user_ringbuf** | **cgrp_storage** | **arena** | 58 | | | **task_storage** | **bloom_filter** | **user_ringbuf** | **cgrp_storage** | **arena** |
| 59 | | | **insn_array** } | 59 | | | **insn_array** | **rhash** } |
| 60 | 60 | ||
| 61 | DESCRIPTION | 61 | DESCRIPTION |
| 62 | =========== | 62 | =========== |
| @@ -47,7 +47,8 @@ $(LIBBPF_INTERNAL_HDRS): $(LIBBPF_HDRS_DIR)/%.h: $(BPF_DIR)/%.h | $(LIBBPF_HDRS_ · 46 unmodified lines | |||
| 47 | $(LIBBPF_BOOTSTRAP): $(wildcard $(BPF_DIR)/*.[ch] $(BPF_DIR)/Makefile) | $(LIBBPF_BOOTSTRAP_OUTPUT) | 47 | $(LIBBPF_BOOTSTRAP): $(wildcard $(BPF_DIR)/*.[ch] $(BPF_DIR)/Makefile) | $(LIBBPF_BOOTSTRAP_OUTPUT) |
| 48 | $(Q)$(MAKE) -C $(BPF_DIR) OUTPUT=$(LIBBPF_BOOTSTRAP_OUTPUT) \ | 48 | $(Q)$(MAKE) -C $(BPF_DIR) OUTPUT=$(LIBBPF_BOOTSTRAP_OUTPUT) \ |
| 49 | DESTDIR=$(LIBBPF_BOOTSTRAP_DESTDIR:/=) prefix= \ | 49 | DESTDIR=$(LIBBPF_BOOTSTRAP_DESTDIR:/=) prefix= \ |
| 50 | ARCH= CROSS_COMPILE= CC="$(HOSTCC)" LD="$(HOSTLD)" AR="$(HOSTAR)" $@ install_headers | 50 | ARCH= CROSS_COMPILE= CC="$(HOSTCC)" LD="$(HOSTLD)" AR="$(HOSTAR)" \ |
| 51 | CFLAGS="$(LIBBPF_BOOTSTRAP_CFLAGS)" EXTRA_CFLAGS= $@ install_headers | ||
| 51 | 52 | ||
| 52 | $(LIBBPF_BOOTSTRAP_INTERNAL_HDRS): $(LIBBPF_BOOTSTRAP_HDRS_DIR)/%.h: $(BPF_DIR)/%.h | $(LIBBPF_BOOTSTRAP_HDRS_DIR) | 53 | $(LIBBPF_BOOTSTRAP_INTERNAL_HDRS): $(LIBBPF_BOOTSTRAP_HDRS_DIR)/%.h: $(BPF_DIR)/%.h | $(LIBBPF_BOOTSTRAP_HDRS_DIR) |
| 53 | $(call QUIET_INSTALL, $@) | 54 | $(call QUIET_INSTALL, $@) |
| @@ -81,6 +82,13 @@ CFLAGS += -DPACKAGE='"bpftool"' -D__EXPORTED_HEADERS__ \ · 27 unmodified lines | |||
| 81 | ifneq ($(BPFTOOL_VERSION),) | 82 | ifneq ($(BPFTOOL_VERSION),) |
| 82 | CFLAGS += -DBPFTOOL_VERSION='"$(BPFTOOL_VERSION)"' | 83 | CFLAGS += -DBPFTOOL_VERSION='"$(BPFTOOL_VERSION)"' |
| 83 | endif | 84 | endif |
| 85 | |||
| 86 | # This must be done before appending EXTRA_CFLAGS to CFLAGS to avoid | ||
| 87 | # including flags that are not applicable to the host compiler. | ||
| 88 | HOST_CFLAGS := $(subst -I$(LIBBPF_INCLUDE),-I$(LIBBPF_BOOTSTRAP_INCLUDE),\ | ||
| 89 | $(subst $(CLANG_CROSS_FLAGS),,$(CFLAGS))) | ||
| 90 | HOST_CFLAGS += $(HOST_EXTRACFLAGS) | ||
| 91 | |||
| 84 | ifneq ($(EXTRA_CFLAGS),) | 92 | ifneq ($(EXTRA_CFLAGS),) |
| 85 | CFLAGS += $(EXTRA_CFLAGS) | 93 | CFLAGS += $(EXTRA_CFLAGS) |
| 86 | endif | 94 | endif |
| @@ -88,10 +96,11 @@ ifneq ($(EXTRA_LDFLAGS),) · 1 unmodified line | |||
| 88 | LDFLAGS += $(EXTRA_LDFLAGS) | 96 | LDFLAGS += $(EXTRA_LDFLAGS) |
| 89 | endif | 97 | endif |
| 90 | 98 | ||
| 91 | HOST_CFLAGS := $(subst -I$(LIBBPF_INCLUDE),-I$(LIBBPF_BOOTSTRAP_INCLUDE),\ | ||
| 92 | $(subst $(CLANG_CROSS_FLAGS),,$(CFLAGS))) | ||
| 93 | HOST_LDFLAGS := $(LDFLAGS) | 99 | HOST_LDFLAGS := $(LDFLAGS) |
| 94 | 100 | ||
| 101 | # Remove warnings for libbpf bootstrap build | ||
| 102 | LIBBPF_BOOTSTRAP_CFLAGS := $(filter-out -W -Wall -Wextra -Wformat -Wformat-signedness,$(HOST_CFLAGS)) | ||
| 103 | |||
| 95 | INSTALL ?= install | 104 | INSTALL ?= install |
| 96 | RM ?= rm -f | 105 | RM ?= rm -f |
| 97 | 106 | ||
| @@ -106,6 +115,10 @@ ifneq ($(SKIP_CRYPTO),1) · 8 unmodified lines | |||
| 106 | CRYPTO_LIBS := -lcrypto | 115 | CRYPTO_LIBS := -lcrypto |
| 107 | endif | 116 | endif |
| 108 | 117 | ||
| 118 | ifeq ($(MAKECMDGOALS),bootstrap) | ||
| 119 | FEATURE_TESTS := libelf-zstd | ||
| 120 | FEATURE_DISPLAY := | ||
| 121 | else | ||
| 109 | FEATURE_TESTS := clang-bpf-co-re | 122 | FEATURE_TESTS := clang-bpf-co-re |
| 110 | FEATURE_TESTS += llvm | 123 | FEATURE_TESTS += llvm |
| 111 | FEATURE_TESTS += libcap | 124 | FEATURE_TESTS += libcap |
| @@ -122,6 +135,7 @@ FEATURE_DISPLAY += libcap · 10 unmodified lines | |||
| 122 | FEATURE_DISPLAY += libbfd | 135 | FEATURE_DISPLAY += libbfd |
| 123 | FEATURE_DISPLAY += libbfd-liberty | 136 | FEATURE_DISPLAY += libbfd-liberty |
| 124 | FEATURE_DISPLAY += libbfd-liberty-z | 137 | FEATURE_DISPLAY += libbfd-liberty-z |
| 138 | endif | ||
| 125 | 139 | ||
| 126 | check_feat := 1 | 140 | check_feat := 1 |
| 127 | NON_CHECK_FEAT_TARGETS := clean uninstall doc doc-clean doc-install doc-uninstall | 141 | NON_CHECK_FEAT_TARGETS := clean uninstall doc doc-clean doc-install doc-uninstall |
| @@ -179,8 +179,7 @@ static int dump_btf_type(const struct btf *btf, __u32 id, · 178 unmodified lines | |||
| 179 | case BTF_KIND_STRUCT: | 179 | case BTF_KIND_STRUCT: |
| 180 | case BTF_KIND_UNION: { | 180 | case BTF_KIND_UNION: { |
| 181 | const struct btf_member *m = (const void *)(t + 1); | 181 | const struct btf_member *m = (const void *)(t + 1); |
| 182 | __u16 vlen = BTF_INFO_VLEN(t->info); | 182 | __u32 i, vlen = BTF_INFO_VLEN(t->info); |
| 183 | int i; | ||
| 184 | 183 | ||
| 185 | if (json_output) { | 184 | if (json_output) { |
| 186 | jsonw_uint_field(w, "size", t->size); | 185 | jsonw_uint_field(w, "size", t->size); |
| @@ -225,9 +224,8 @@ static int dump_btf_type(const struct btf *btf, __u32 id, · 38 unmodified lines | |||
| 225 | } | 224 | } |
| 226 | case BTF_KIND_ENUM: { | 225 | case BTF_KIND_ENUM: { |
| 227 | const struct btf_enum *v = (const void *)(t + 1); | 226 | const struct btf_enum *v = (const void *)(t + 1); |
| 228 | __u16 vlen = BTF_INFO_VLEN(t->info); | 227 | __u32 i, vlen = BTF_INFO_VLEN(t->info); |
| 229 | const char *encoding; | 228 | const char *encoding; |
| 230 | int i; | ||
| 231 | 229 | ||
| 232 | encoding = btf_kflag(t) ? "SIGNED" : "UNSIGNED"; | 230 | encoding = btf_kflag(t) ? "SIGNED" : "UNSIGNED"; |
| 233 | if (json_output) { | 231 | if (json_output) { |
| @@ -263,9 +261,8 @@ static int dump_btf_type(const struct btf *btf, __u32 id, · 29 unmodified lines | |||
| 263 | } | 261 | } |
| 264 | case BTF_KIND_ENUM64: { | 262 | case BTF_KIND_ENUM64: { |
| 265 | const struct btf_enum64 *v = btf_enum64(t); | 263 | const struct btf_enum64 *v = btf_enum64(t); |
| 266 | __u16 vlen = btf_vlen(t); | 264 | __u32 i, vlen = btf_vlen(t); |
| 267 | const char *encoding; | 265 | const char *encoding; |
| 268 | int i; | ||
| 269 | 266 | ||
| 270 | encoding = btf_kflag(t) ? "SIGNED" : "UNSIGNED"; | 267 | encoding = btf_kflag(t) ? "SIGNED" : "UNSIGNED"; |
| 271 | if (json_output) { | 268 | if (json_output) { |
| @@ -325,8 +322,7 @@ static int dump_btf_type(const struct btf *btf, __u32 id, · 53 unmodified lines | |||
| 325 | } | 322 | } |
| 326 | case BTF_KIND_FUNC_PROTO: { | 323 | case BTF_KIND_FUNC_PROTO: { |
| 327 | const struct btf_param *p = (const void *)(t + 1); | 324 | const struct btf_param *p = (const void *)(t + 1); |
| 328 | __u16 vlen = BTF_INFO_VLEN(t->info); | 325 | __u32 i, vlen = BTF_INFO_VLEN(t->info); |
| 329 | int i; | ||
| 330 | 326 | ||
| 331 | if (json_output) { | 327 | if (json_output) { |
| 332 | jsonw_uint_field(w, "ret_type_id", t->type); | 328 | jsonw_uint_field(w, "ret_type_id", t->type); |
| @@ -369,8 +365,7 @@ static int dump_btf_type(const struct btf *btf, __u32 id, · 36 unmodified lines | |||
| 369 | case BTF_KIND_DATASEC: { | 365 | case BTF_KIND_DATASEC: { |
| 370 | const struct btf_var_secinfo *v = (const void *)(t + 1); | 366 | const struct btf_var_secinfo *v = (const void *)(t + 1); |
| 371 | const struct btf_type *vt; | 367 | const struct btf_type *vt; |
| 372 | __u16 vlen = BTF_INFO_VLEN(t->info); | 368 | __u32 i, vlen = BTF_INFO_VLEN(t->info); |
| 373 | int i; | ||
| 374 | 369 | ||
| 375 | if (json_output) { | 370 | if (json_output) { |
| 376 | jsonw_uint_field(w, "size", t->size); | 371 | jsonw_uint_field(w, "size", t->size); |
| @@ -675,7 +670,7 @@ static __u64 btf_name_hasher(__u64 hash, const struct btf *btf, __u32 name_off) · 298 unmodified lines | |||
| 675 | static __u64 btf_type_disambig_hash(const struct btf *btf, __u32 id, bool include_members) | 670 | static __u64 btf_type_disambig_hash(const struct btf *btf, __u32 id, bool include_members) |
| 676 | { | 671 | { |
| 677 | const struct btf_type *t = btf__type_by_id(btf, id); | 672 | const struct btf_type *t = btf__type_by_id(btf, id); |
| 678 | int i; | 673 | __u32 i; |
| 679 | size_t hash = 0; | 674 | size_t hash = 0; |
| 680 | 675 | ||
| 681 | hash = btf_name_hasher(hash, btf, t->name_off); | 676 | hash = btf_name_hasher(hash, btf, t->name_off); |
| @@ -150,7 +150,7 @@ static int btf_dumper_enum(const struct btf_dumper *d, · 149 unmodified lines | |||
| 150 | { | 150 | { |
| 151 | const struct btf_enum *enums = btf_enum(t); | 151 | const struct btf_enum *enums = btf_enum(t); |
| 152 | __s64 value; | 152 | __s64 value; |
| 153 | __u16 i; | 153 | __u32 i; |
| 154 | 154 | ||
| 155 | switch (t->size) { | 155 | switch (t->size) { |
| 156 | case 8: | 156 | case 8: |
| @@ -189,7 +189,7 @@ static int btf_dumper_enum64(const struct btf_dumper *d, · 32 unmodified lines | |||
| 189 | const struct btf_enum64 *enums = btf_enum64(t); | 189 | const struct btf_enum64 *enums = btf_enum64(t); |
| 190 | __u32 val_lo32, val_hi32; | 190 | __u32 val_lo32, val_hi32; |
| 191 | __u64 value; | 191 | __u64 value; |
| 192 | __u16 i; | 192 | __u32 i; |
| 193 | 193 | ||
| 194 | value = *(__u64 *)data; | 194 | value = *(__u64 *)data; |
| 195 | val_lo32 = (__u32)value; | 195 | val_lo32 = (__u32)value; |
| @@ -1399,7 +1399,7 @@ static int do_skeleton(int argc, char **argv) · 1398 unmodified lines | |||
| 1399 | continue; | 1399 | continue; |
| 1400 | 1400 | ||
| 1401 | if (use_loader) | 1401 | if (use_loader) |
| 1402 | printf("t\tint %s_fd;\n", ident); | 1402 | printf("\t\tint %s_fd;\n", ident); |
| 1403 | else | 1403 | else |
| 1404 | printf("\t\tstruct bpf_link *%s;\n", ident); | 1404 | printf("\t\tstruct bpf_link *%s;\n", ident); |
| 1405 | } | 1405 | } |
| @@ -2094,7 +2094,8 @@ btfgen_mark_type(struct btfgen_info *info, unsigned int type_id, bool follow_poi · 688 unmodified lines | |||
| 2094 | struct btf_type *cloned_type; | 2094 | struct btf_type *cloned_type; |
| 2095 | struct btf_param *param; | 2095 | struct btf_param *param; |
| 2096 | struct btf_array *array; | 2096 | struct btf_array *array; |
| 2097 | int err, i; | 2097 | __u32 i; |
| 2098 | int err; | ||
| 2098 | 2099 | ||
| 2099 | if (type_id == 0) | 2100 | if (type_id == 0) |
| 2100 | return 0; | 2101 | return 0; |
| @@ -2229,7 +2230,8 @@ static int btfgen_mark_type_match(struct btfgen_info *info, __u32 type_id, bool · 128 unmodified lines | |||
| 2229 | const struct btf_type *btf_type; | 2230 | const struct btf_type *btf_type; |
| 2230 | struct btf *btf = info->src_btf; | 2231 | struct btf *btf = info->src_btf; |
| 2231 | struct btf_type *cloned_type; | 2232 | struct btf_type *cloned_type; |
| 2232 | int i, err; | 2233 | int err; |
| 2234 | __u32 i; | ||
| 2233 | 2235 | ||
| 2234 | if (type_id == 0) | 2236 | if (type_id == 0) |
| 2235 | return 0; | 2237 | return 0; |
| @@ -2249,7 +2251,7 @@ static int btfgen_mark_type_match(struct btfgen_info *info, __u32 type_id, bool · 13 unmodified lines | |||
| 2249 | case BTF_KIND_STRUCT: | 2251 | case BTF_KIND_STRUCT: |
| 2250 | case BTF_KIND_UNION: { | 2252 | case BTF_KIND_UNION: { |
| 2251 | struct btf_member *m = btf_members(btf_type); | 2253 | struct btf_member *m = btf_members(btf_type); |
| 2252 | __u16 vlen = btf_vlen(btf_type); | 2254 | __u32 vlen = btf_vlen(btf_type); |
| 2253 | 2255 | ||
| 2254 | if (behind_ptr) | 2256 | if (behind_ptr) |
| 2255 | break; | 2257 | break; |
| @@ -2286,7 +2288,7 @@ static int btfgen_mark_type_match(struct btfgen_info *info, __u32 type_id, bool · 30 unmodified lines | |||
| 2286 | break; | 2288 | break; |
| 2287 | } | 2289 | } |
| 2288 | case BTF_KIND_FUNC_PROTO: { | 2290 | case BTF_KIND_FUNC_PROTO: { |
| 2289 | __u16 vlen = btf_vlen(btf_type); | 2291 | __u32 vlen = btf_vlen(btf_type); |
| 2290 | struct btf_param *param; | 2292 | struct btf_param *param; |
| 2291 | 2293 | ||
| 2292 | /* mark ret type */ | 2294 | /* mark ret type */ |
| @@ -2492,8 +2494,9 @@ static struct btf *btfgen_get_btf(struct btfgen_info *info) · 199 unmodified lines | |||
| 2492 | { | 2494 | { |
| 2493 | struct btf *btf_new = NULL; | 2495 | struct btf *btf_new = NULL; |
| 2494 | unsigned int *ids = NULL; | 2496 | unsigned int *ids = NULL; |
| 2495 | unsigned int i, n = btf__type_cnt(info->marked_btf); | 2497 | unsigned int n = btf__type_cnt(info->marked_btf); |
| 2496 | int err = 0; | 2498 | int err = 0; |
| 2499 | __u32 i; | ||
| 2497 | 2500 | ||
| 2498 | btf_new = btf__new_empty(); | 2501 | btf_new = btf__new_empty(); |
| 2499 | if (!btf_new) { | 2502 | if (!btf_new) { |
| @@ -2523,8 +2526,7 @@ static struct btf *btfgen_get_btf(struct btfgen_info *info) · 23 unmodified lines | |||
| 2523 | /* add members for struct and union */ | 2526 | /* add members for struct and union */ |
| 2524 | if (btf_is_composite(type)) { | 2527 | if (btf_is_composite(type)) { |
| 2525 | struct btf_member *cloned_m, *m; | 2528 | struct btf_member *cloned_m, *m; |
| 2526 | unsigned short vlen; | 2529 | __u32 vlen, idx_src; |
| 2527 | int idx_src; | ||
| 2528 | 2530 | ||
| 2529 | name = btf__str_by_offset(info->src_btf, type->name_off); | 2531 | name = btf__str_by_offset(info->src_btf, type->name_off); |
| 2530 | 2532 | ||
| @@ -1478,7 +1478,7 @@ static int do_help(int argc, char **argv) · 1477 unmodified lines | |||
| 1478 | " cgroup_storage | reuseport_sockarray | percpu_cgroup_storage |\n" | 1478 | " cgroup_storage | reuseport_sockarray | percpu_cgroup_storage |\n" |
| 1479 | " queue | stack | sk_storage | struct_ops | ringbuf | inode_storage |\n" | 1479 | " queue | stack | sk_storage | struct_ops | ringbuf | inode_storage |\n" |
| 1480 | " task_storage | bloom_filter | user_ringbuf | cgrp_storage | arena |\n" | 1480 | " task_storage | bloom_filter | user_ringbuf | cgrp_storage | arena |\n" |
| 1481 | " insn_array }\n" | 1481 | " insn_array | rhash }\n" |
| 1482 | " " HELP_SPEC_OPTIONS " |\n" | 1482 | " " HELP_SPEC_OPTIONS " |\n" |
| 1483 | " {-f|--bpffs} | {-n|--nomount} }\n" | 1483 | " {-f|--bpffs} | {-n|--nomount} }\n" |
| 1484 | "", | 1484 | "", |
| @@ -603,14 +603,14 @@ static int query_flow_dissector(struct bpf_attach_info *attach_info) · 602 unmodified lines | |||
| 603 | &attach_flags, prog_ids, &prog_cnt); | 603 | &attach_flags, prog_ids, &prog_cnt); |
| 604 | close(fd); | 604 | close(fd); |
| 605 | if (err) { | 605 | if (err) { |
| 606 | if (errno == EINVAL) { | 606 | if (err == -EINVAL) { |
| 607 | /* Older kernel's don't support querying | 607 | /* Older kernel's don't support querying |
| 608 | * flow dissector programs. | 608 | * flow dissector programs. |
| 609 | */ | 609 | */ |
| 610 | errno = 0; | 610 | errno = 0; |
| 611 | return 0; | 611 | return 0; |
| 612 | } | 612 | } |
| 613 | p_err("can't query prog: %s", strerror(errno)); | 613 | p_err("can't query prog: %s", strerror(-err)); |
| 614 | return -1; | 614 | return -1; |
| 615 | } | 615 | } |
| 616 | 616 | ||
| @@ -20,6 +20,8 @@ · 19 unmodified lines | |||
| 20 | # define __NR_bpf 6319 | 20 | # define __NR_bpf 6319 |
| 21 | # elif defined(__mips__) && defined(_ABI64) | 21 | # elif defined(__mips__) && defined(_ABI64) |
| 22 | # define __NR_bpf 5315 | 22 | # define __NR_bpf 5315 |
| 23 | # elif defined(__loongarch__) | ||
| 24 | # define __NR_bpf 280 | ||
| 23 | # else | 25 | # else |
| 24 | # error __NR_bpf not defined. libbpf does not support your arch. | 26 | # error __NR_bpf not defined. libbpf does not support your arch. |
| 25 | # endif | 27 | # endif |
| @@ -994,6 +994,7 @@ enum bpf_cmd { · 993 unmodified lines | |||
| 994 | BPF_PROG_STREAM_READ_BY_FD, | 994 | BPF_PROG_STREAM_READ_BY_FD, |
| 995 | BPF_PROG_ASSOC_STRUCT_OPS, | 995 | BPF_PROG_ASSOC_STRUCT_OPS, |
| 996 | __MAX_BPF_CMD, | 996 | __MAX_BPF_CMD, |
| 997 | BPF_COMMON_ATTRS = 1 << 16, /* Indicate carrying syscall common attrs. */ | ||
| 997 | }; | 998 | }; |
| 998 | 999 | ||
| 999 | enum bpf_map_type { | 1000 | enum bpf_map_type { |
| @@ -1046,6 +1047,7 @@ enum bpf_map_type { · 46 unmodified lines | |||
| 1046 | BPF_MAP_TYPE_CGRP_STORAGE, | 1047 | BPF_MAP_TYPE_CGRP_STORAGE, |
| 1047 | BPF_MAP_TYPE_ARENA, | 1048 | BPF_MAP_TYPE_ARENA, |
| 1048 | BPF_MAP_TYPE_INSN_ARRAY, | 1049 | BPF_MAP_TYPE_INSN_ARRAY, |
| 1050 | BPF_MAP_TYPE_RHASH, | ||
| 1049 | __MAX_BPF_MAP_TYPE | 1051 | __MAX_BPF_MAP_TYPE |
| 1050 | }; | 1052 | }; |
| 1051 | 1053 | ||
| @@ -1154,6 +1156,9 @@ enum bpf_attach_type { · 102 unmodified lines | |||
| 1154 | BPF_TRACE_KPROBE_SESSION, | 1156 | BPF_TRACE_KPROBE_SESSION, |
| 1155 | BPF_TRACE_UPROBE_SESSION, | 1157 | BPF_TRACE_UPROBE_SESSION, |
| 1156 | BPF_TRACE_FSESSION, | 1158 | BPF_TRACE_FSESSION, |
| 1159 | BPF_TRACE_FENTRY_MULTI, | ||
| 1160 | BPF_TRACE_FEXIT_MULTI, | ||
| 1161 | BPF_TRACE_FSESSION_MULTI, | ||
| 1157 | __MAX_BPF_ATTACH_TYPE | 1162 | __MAX_BPF_ATTACH_TYPE |
| 1158 | }; | 1163 | }; |
| 1159 | 1164 | ||
| @@ -1178,6 +1183,7 @@ enum bpf_link_type { · 18 unmodified lines | |||
| 1178 | BPF_LINK_TYPE_UPROBE_MULTI = 12, | 1183 | BPF_LINK_TYPE_UPROBE_MULTI = 12, |
| 1179 | BPF_LINK_TYPE_NETKIT = 13, | 1184 | BPF_LINK_TYPE_NETKIT = 13, |
| 1180 | BPF_LINK_TYPE_SOCKMAP = 14, | 1185 | BPF_LINK_TYPE_SOCKMAP = 14, |
| 1186 | BPF_LINK_TYPE_TRACING_MULTI = 15, | ||
| 1181 | __MAX_BPF_LINK_TYPE, | 1187 | __MAX_BPF_LINK_TYPE, |
| 1182 | }; | 1188 | }; |
| 1183 | 1189 | ||
| @@ -1321,7 +1327,11 @@ enum { · 137 unmodified lines | |||
| 1321 | * BPF_TRACE_UPROBE_MULTI attach type to create return probe. | 1327 | * BPF_TRACE_UPROBE_MULTI attach type to create return probe. |
| 1322 | */ | 1328 | */ |
| 1323 | enum { | 1329 | enum { |
| 1324 | BPF_F_UPROBE_MULTI_RETURN = (1U << 0) | 1330 | /* Get return uprobe. */ |
| 1331 | BPF_F_UPROBE_MULTI_RETURN = (1U << 0), | ||
| 1332 | |||
| 1333 | /* Get path from provided path_fd. */ | ||
| 1334 | BPF_F_UPROBE_MULTI_PATH_FD = (1U << 1), | ||
| 1325 | }; | 1335 | }; |
| 1326 | 1336 | ||
| 1327 | /* link_create.netfilter.flags used in LINK_CREATE command for | 1337 | /* link_create.netfilter.flags used in LINK_CREATE command for |
| @@ -1500,6 +1510,13 @@ struct bpf_stack_build_id { · 172 unmodified lines | |||
| 1500 | }; | 1510 | }; |
| 1501 | }; | 1511 | }; |
| 1502 | 1512 | ||
| 1513 | struct bpf_common_attr { | ||
| 1514 | __aligned_u64 log_buf; | ||
| 1515 | __u32 log_size; | ||
| 1516 | __u32 log_level; | ||
| 1517 | __u32 log_true_size; | ||
| 1518 | }; | ||
| 1519 | |||
| 1503 | #define BPF_OBJ_NAME_LEN 16U | 1520 | #define BPF_OBJ_NAME_LEN 16U |
| 1504 | 1521 | ||
| 1505 | enum { | 1522 | enum { |
| @@ -1537,6 +1554,11 @@ union bpf_attr { · 31 unmodified lines | |||
| 1537 | * | 1554 | * |
| 1538 | * BPF_MAP_TYPE_ARENA - contains the address where user space | 1555 | * BPF_MAP_TYPE_ARENA - contains the address where user space |
| 1539 | * is going to mmap() the arena. It has to be page aligned. | 1556 | * is going to mmap() the arena. It has to be page aligned. |
| 1557 | * | ||
| 1558 | * BPF_MAP_TYPE_RHASH - initial table size hint | ||
| 1559 | * (nelem_hint). 0 = use rhashtable default. Must be | ||
| 1560 | * <= min(max_entries, U16_MAX). Upper 32 bits reserved, | ||
| 1561 | * must be zero. | ||
| 1540 | */ | 1562 | */ |
| 1541 | __u64 map_extra; | 1563 | __u64 map_extra; |
| 1542 | 1564 | ||
| @@ -1846,6 +1868,7 @@ union bpf_attr { · 303 unmodified lines | |||
| 1846 | __u32 cnt; | 1868 | __u32 cnt; |
| 1847 | __u32 flags; | 1869 | __u32 flags; |
| 1848 | __u32 pid; | 1870 | __u32 pid; |
| 1871 | __u32 path_fd; | ||
| 1849 | } uprobe_multi; | 1872 | } uprobe_multi; |
| 1850 | struct { | 1873 | struct { |
| 1851 | union { | 1874 | union { |
| @@ -1861,6 +1884,11 @@ union bpf_attr { · 9 unmodified lines | |||
| 1861 | }; | 1884 | }; |
| 1862 | __u64 expected_revision; | 1885 | __u64 expected_revision; |
| 1863 | } cgroup; | 1886 | } cgroup; |
| 1887 | struct { | ||
| 1888 | __aligned_u64 ids; | ||
| 1889 | __aligned_u64 cookies; | ||
| 1890 | __u32 cnt; | ||
| 1891 | } tracing_multi; | ||
| 1864 | }; | 1892 | }; |
| 1865 | } link_create; | 1893 | } link_create; |
| 1866 | 1894 | ||
| @@ -6698,6 +6726,7 @@ struct bpf_prog_info { · 4831 unmodified lines | |||
| 6698 | __u32 verified_insns; | 6726 | __u32 verified_insns; |
| 6699 | __u32 attach_btf_obj_id; | 6727 | __u32 attach_btf_obj_id; |
| 6700 | __u32 attach_btf_id; | 6728 | __u32 attach_btf_id; |
| 6729 | __u32 :32; | ||
| 6701 | } __attribute__((aligned(8))); | 6730 | } __attribute__((aligned(8))); |
| 6702 | 6731 | ||
| 6703 | struct bpf_map_info { | 6732 | struct bpf_map_info { |
| @@ -6719,6 +6748,7 @@ struct bpf_map_info { · 15 unmodified lines | |||
| 6719 | __u64 map_extra; | 6748 | __u64 map_extra; |
| 6720 | __aligned_u64 hash; | 6749 | __aligned_u64 hash; |
| 6721 | __u32 hash_size; | 6750 | __u32 hash_size; |
| 6751 | __u32 :32; | ||
| 6722 | } __attribute__((aligned(8))); | 6752 | } __attribute__((aligned(8))); |
| 6723 | 6753 | ||
| 6724 | struct bpf_btf_info { | 6754 | struct bpf_btf_info { |
| @@ -7236,6 +7266,7 @@ enum { · 511 unmodified lines | |||
| 7236 | TCP_BPF_SOCK_OPS_CB_FLAGS = 1008, /* Get or Set TCP sock ops flags */ | 7266 | TCP_BPF_SOCK_OPS_CB_FLAGS = 1008, /* Get or Set TCP sock ops flags */ |
| 7237 | SK_BPF_CB_FLAGS = 1009, /* Get or set sock ops flags in socket */ | 7267 | SK_BPF_CB_FLAGS = 1009, /* Get or set sock ops flags in socket */ |
| 7238 | SK_BPF_BYPASS_PROT_MEM = 1010, /* Get or Set sk->sk_bypass_prot_mem */ | 7268 | SK_BPF_BYPASS_PROT_MEM = 1010, /* Get or Set sk->sk_bypass_prot_mem */ |
| 7269 | |||
| 7239 | }; | 7270 | }; |
| 7240 | 7271 | ||
| 7241 | enum { | 7272 | enum { |
| @@ -33,20 +33,22 @@ struct btf_header { · 32 unmodified lines | |||
| 33 | __u32 layout_len; /* length of layout section */ | 33 | __u32 layout_len; /* length of layout section */ |
| 34 | }; | 34 | }; |
| 35 | 35 | ||
| 36 | /* Max # of type identifier */ | 36 | enum btf_max { |
| 37 | #define BTF_MAX_TYPE 0x000fffff | 37 | /* Max possible kind */ |
| 38 | /* Max offset into the string section */ | 38 | BTF_MAX_KIND = 0x0000007f, |
| 39 | #define BTF_MAX_NAME_OFFSET 0x00ffffff | 39 | /* Max # of type identifier */ |
| 40 | /* Max # of struct/union/enum members or func args */ | 40 | BTF_MAX_TYPE = 0x000fffff, |
| 41 | #define BTF_MAX_VLEN 0xffff | 41 | /* Max offset into the string section */ |
| 42 | BTF_MAX_NAME_OFFSET = 0x00ffffff, | ||
| 43 | /* Max # of struct/union/enum members or func args */ | ||
| 44 | BTF_MAX_VLEN = 0x00ffffff, | ||
| 45 | }; | ||
| 42 | 46 | ||
| 43 | struct btf_type { | 47 | struct btf_type { |
| 44 | __u32 name_off; | 48 | __u32 name_off; |
| 45 | /* "info" bits arrangement | 49 | /* "info" bits arrangement |
| 46 | * bits 0-15: vlen (e.g. # of struct's members) | 50 | * bits 0-23: vlen (e.g. # of struct's members) |
| 47 | * bits 16-23: unused | 51 | * bits 24-30: kind (e.g. int, ptr, array...etc) |
| 48 | * bits 24-28: kind (e.g. int, ptr, array...etc) | ||
| 49 | * bits 29-30: unused | ||
| 50 | * bit 31: kind_flag, currently used by | 52 | * bit 31: kind_flag, currently used by |
| 51 | * struct, union, enum, fwd, enum64, | 53 | * struct, union, enum, fwd, enum64, |
| 52 | * decl_tag and type_tag | 54 | * decl_tag and type_tag |
| @@ -65,8 +67,8 @@ struct btf_type { · 12 unmodified lines | |||
| 65 | }; | 67 | }; |
| 66 | }; | 68 | }; |
| 67 | 69 | ||
| 68 | #define BTF_INFO_KIND(info) (((info) >> 24) & 0x1f) | 70 | #define BTF_INFO_KIND(info) (((info) >> 24) & 0x7f) |
| 69 | #define BTF_INFO_VLEN(info) ((info) & 0xffff) | 71 | #define BTF_INFO_VLEN(info) ((info) & 0xffffff) |
| 70 | #define BTF_INFO_KFLAG(info) ((info) >> 31) | 72 | #define BTF_INFO_KFLAG(info) ((info) >> 31) |
| 71 | 73 | ||
| 72 | enum { | 74 | enum { |
| @@ -49,6 +49,14 @@ man_dir_SQ = '$(subst ','\'',$(man_dir))' · 48 unmodified lines | |||
| 49 | export man_dir man_dir_SQ INSTALL | 49 | export man_dir man_dir_SQ INSTALL |
| 50 | export DESTDIR DESTDIR_SQ | 50 | export DESTDIR DESTDIR_SQ |
| 51 | 51 | ||
| 52 | # Defer assigning EXTRA_CFLAGS to CFLAGS until after including | ||
| 53 | # tools/scripts/Makefile.include, as it may add flags to EXTRA_CFLAGS. | ||
| 54 | ifdef EXTRA_CFLAGS | ||
| 55 | CFLAGS := | ||
| 56 | else | ||
| 57 | CFLAGS := -g -O2 | ||
| 58 | endif | ||
| 59 | |||
| 52 | include $(srctree)/tools/scripts/Makefile.include | 60 | include $(srctree)/tools/scripts/Makefile.include |
| 53 | 61 | ||
| 54 | # copy a bit from Linux kbuild | 62 | # copy a bit from Linux kbuild |
| @@ -70,13 +78,6 @@ LIB_TARGET = libbpf.a libbpf.so.$(LIBBPF_VERSION) · 15 unmodified lines | |||
| 70 | LIB_FILE = libbpf.a libbpf.so* | 78 | LIB_FILE = libbpf.a libbpf.so* |
| 71 | PC_FILE = libbpf.pc | 79 | PC_FILE = libbpf.pc |
| 72 | 80 | ||
| 73 | # Set compile option CFLAGS | ||
| 74 | ifdef EXTRA_CFLAGS | ||
| 75 | CFLAGS := $(EXTRA_CFLAGS) | ||
| 76 | else | ||
| 77 | CFLAGS := -g -O2 | ||
| 78 | endif | ||
| 79 | |||
| 80 | # Append required CFLAGS | 81 | # Append required CFLAGS |
| 81 | override CFLAGS += -std=gnu89 | 82 | override CFLAGS += -std=gnu89 |
| 82 | override CFLAGS += $(EXTRA_WARNINGS) -Wno-switch-enum | 83 | override CFLAGS += $(EXTRA_WARNINGS) -Wno-switch-enum |
| @@ -84,7 +85,7 @@ override CFLAGS += -Werror -Wall · 1 unmodified line | |||
| 84 | override CFLAGS += $(INCLUDES) | 85 | override CFLAGS += $(INCLUDES) |
| 85 | override CFLAGS += -fvisibility=hidden | 86 | override CFLAGS += -fvisibility=hidden |
| 86 | override CFLAGS += -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 | 87 | override CFLAGS += -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 |
| 87 | override CFLAGS += $(CLANG_CROSS_FLAGS) | 88 | override CFLAGS += $(EXTRA_CFLAGS) |
| 88 | 89 | ||
| 89 | # flags specific for shared library | 90 | # flags specific for shared library |
| 90 | SHLIB_FLAGS := -DSHARED -fPIC | 91 | SHLIB_FLAGS := -DSHARED -fPIC |
| @@ -59,6 +59,8 @@ · 58 unmodified lines | |||
| 59 | # define __NR_bpf 6319 | 59 | # define __NR_bpf 6319 |
| 60 | # elif defined(__mips__) && defined(_ABI64) | 60 | # elif defined(__mips__) && defined(_ABI64) |
| 61 | # define __NR_bpf 5315 | 61 | # define __NR_bpf 5315 |
| 62 | # elif defined(__loongarch__) | ||
| 63 | # define __NR_bpf 280 | ||
| 62 | # else | 64 | # else |
| 63 | # error __NR_bpf not defined. libbpf does not support your arch. | 65 | # error __NR_bpf not defined. libbpf does not support your arch. |
| 64 | # endif | 66 | # endif |
| @@ -69,6 +71,42 @@ static inline __u64 ptr_to_u64(const void *ptr) · 4 unmodified lines | |||
| 69 | return (__u64) (unsigned long) ptr; | 71 | return (__u64) (unsigned long) ptr; |
| 70 | } | 72 | } |
| 71 | 73 | ||
| 74 | static inline int sys_bpf_ext(enum bpf_cmd cmd, union bpf_attr *attr, | ||
| 75 | unsigned int size, | ||
| 76 | struct bpf_common_attr *attr_common, | ||
| 77 | unsigned int size_common) | ||
| 78 | { | ||
| 79 | cmd = attr_common ? (cmd | BPF_COMMON_ATTRS) : (cmd & ~BPF_COMMON_ATTRS); | ||
| 80 | return syscall(__NR_bpf, cmd, attr, size, attr_common, size_common); | ||
| 81 | } | ||
| 82 | |||
| 83 | static inline int sys_bpf_ext_fd(enum bpf_cmd cmd, union bpf_attr *attr, | ||
| 84 | unsigned int size, | ||
| 85 | struct bpf_common_attr *attr_common, | ||
| 86 | unsigned int size_common) | ||
| 87 | { | ||
| 88 | int fd; | ||
| 89 | |||
| 90 | fd = sys_bpf_ext(cmd, attr, size, attr_common, size_common); | ||
| 91 | return ensure_good_fd(fd); | ||
| 92 | } | ||
| 93 | |||
| 94 | int probe_sys_bpf_ext(void) | ||
| 95 | { | ||
| 96 | const size_t attr_sz = offsetofend(union bpf_attr, prog_token_fd); | ||
| 97 | union bpf_attr attr; | ||
| 98 | int fd; | ||
| 99 | |||
| 100 | memset(&attr, 0, attr_sz); | ||
| 101 | fd = syscall(__NR_bpf, BPF_PROG_LOAD | BPF_COMMON_ATTRS, &attr, attr_sz, NULL, | ||
| 102 | sizeof(struct bpf_common_attr)); | ||
| 103 | if (fd >= 0) { | ||
| 104 | close(fd); | ||
| 105 | return -EINVAL; | ||
| 106 | } | ||
| 107 | return errno == EFAULT ? 1 : 0; | ||
| 108 | } | ||
| 109 | |||
| 72 | static inline int sys_bpf(enum bpf_cmd cmd, union bpf_attr *attr, | 110 | static inline int sys_bpf(enum bpf_cmd cmd, union bpf_attr *attr, |
| 73 | unsigned int size) | 111 | unsigned int size) |
| 74 | { | 112 | { |
| @@ -173,6 +211,9 @@ int bpf_map_create(enum bpf_map_type map_type, · 98 unmodified lines | |||
| 173 | const struct bpf_map_create_opts *opts) | 211 | const struct bpf_map_create_opts *opts) |
| 174 | { | 212 | { |
| 175 | const size_t attr_sz = offsetofend(union bpf_attr, excl_prog_hash_size); | 213 | const size_t attr_sz = offsetofend(union bpf_attr, excl_prog_hash_size); |
| 214 | const size_t attr_common_sz = sizeof(struct bpf_common_attr); | ||
| 215 | struct bpf_common_attr attr_common; | ||
| 216 | struct bpf_log_opts *log_opts; | ||
| 176 | union bpf_attr attr; | 217 | union bpf_attr attr; |
| 177 | int fd; | 218 | int fd; |
| 178 | 219 | ||
| @@ -206,7 +247,21 @@ int bpf_map_create(enum bpf_map_type map_type, · 27 unmodified lines | |||
| 206 | attr.excl_prog_hash = ptr_to_u64(OPTS_GET(opts, excl_prog_hash, NULL)); | 247 | attr.excl_prog_hash = ptr_to_u64(OPTS_GET(opts, excl_prog_hash, NULL)); |
| 207 | attr.excl_prog_hash_size = OPTS_GET(opts, excl_prog_hash_size, 0); | 248 | attr.excl_prog_hash_size = OPTS_GET(opts, excl_prog_hash_size, 0); |
| 208 | 249 | ||
| 209 | fd = sys_bpf_fd(BPF_MAP_CREATE, &attr, attr_sz); | 250 | log_opts = OPTS_GET(opts, log_opts, NULL); |
| 251 | if (!OPTS_VALID(log_opts, bpf_log_opts)) | ||
| 252 | return libbpf_err(-EINVAL); | ||
| 253 | |||
| 254 | if (log_opts && feat_supported(NULL, FEAT_BPF_SYSCALL_COMMON_ATTRS)) { | ||
| 255 | memset(&attr_common, 0, attr_common_sz); | ||
| 256 | attr_common.log_buf = ptr_to_u64(OPTS_GET(log_opts, buf, NULL)); | ||
| 257 | attr_common.log_size = OPTS_GET(log_opts, size, 0); | ||
| 258 | attr_common.log_level = OPTS_GET(log_opts, level, 0); | ||
| 259 | fd = sys_bpf_ext_fd(BPF_MAP_CREATE, &attr, attr_sz, &attr_common, attr_common_sz); | ||
| 260 | OPTS_SET(log_opts, true_size, attr_common.log_true_size); | ||
| 261 | } else { | ||
| 262 | fd = sys_bpf_fd(BPF_MAP_CREATE, &attr, attr_sz); | ||
| 263 | OPTS_SET(log_opts, true_size, 0); | ||
| 264 | } | ||
| 210 | return libbpf_err_errno(fd); | 265 | return libbpf_err_errno(fd); |
| 211 | } | 266 | } |
| 212 | 267 | ||
| @@ -787,9 +842,19 @@ int bpf_link_create(int prog_fd, int target_fd, · 574 unmodified lines | |||
| 787 | attr.link_create.uprobe_multi.ref_ctr_offsets = ptr_to_u64(OPTS_GET(opts, uprobe_multi.ref_ctr_offsets, 0)); | 842 | attr.link_create.uprobe_multi.ref_ctr_offsets = ptr_to_u64(OPTS_GET(opts, uprobe_multi.ref_ctr_offsets, 0)); |
| 788 | attr.link_create.uprobe_multi.cookies = ptr_to_u64(OPTS_GET(opts, uprobe_multi.cookies, 0)); | 843 | attr.link_create.uprobe_multi.cookies = ptr_to_u64(OPTS_GET(opts, uprobe_multi.cookies, 0)); |
| 789 | attr.link_create.uprobe_multi.pid = OPTS_GET(opts, uprobe_multi.pid, 0); | 844 | attr.link_create.uprobe_multi.pid = OPTS_GET(opts, uprobe_multi.pid, 0); |
| 845 | attr.link_create.uprobe_multi.path_fd = OPTS_GET(opts, uprobe_multi.path_fd, 0); | ||
| 790 | if (!OPTS_ZEROED(opts, uprobe_multi)) | 846 | if (!OPTS_ZEROED(opts, uprobe_multi)) |
| 791 | return libbpf_err(-EINVAL); | 847 | return libbpf_err(-EINVAL); |
| 792 | break; | 848 | break; |
| 849 | case BPF_TRACE_FENTRY_MULTI: | ||
| 850 | case BPF_TRACE_FEXIT_MULTI: | ||
| 851 | case BPF_TRACE_FSESSION_MULTI: | ||
| 852 | attr.link_create.tracing_multi.ids = ptr_to_u64(OPTS_GET(opts, tracing_multi.ids, 0)); | ||
| 853 | attr.link_create.tracing_multi.cookies = ptr_to_u64(OPTS_GET(opts, tracing_multi.cookies, 0)); | ||
| 854 | attr.link_create.tracing_multi.cnt = OPTS_GET(opts, tracing_multi.cnt, 0); | ||
| 855 | if (!OPTS_ZEROED(opts, tracing_multi)) | ||
| 856 | return libbpf_err(-EINVAL); | ||
| 857 | break; | ||
| 793 | case BPF_TRACE_RAW_TP: | 858 | case BPF_TRACE_RAW_TP: |
| 794 | case BPF_TRACE_FENTRY: | 859 | case BPF_TRACE_FENTRY: |
| 795 | case BPF_TRACE_FEXIT: | 860 | case BPF_TRACE_FEXIT: |
| @@ -37,6 +37,18 @@ extern "C" { · 36 unmodified lines | |||
| 37 | 37 | ||
| 38 | LIBBPF_API int libbpf_set_memlock_rlim(size_t memlock_bytes); | 38 | LIBBPF_API int libbpf_set_memlock_rlim(size_t memlock_bytes); |
| 39 | 39 | ||
| 40 | struct bpf_log_opts { | ||
| 41 | size_t sz; /* size of this struct for forward/backward compatibility */ | ||
| 42 | |||
| 43 | char *buf; | ||
| 44 | __u32 size; | ||
| 45 | __u32 level; | ||
| 46 | __u32 true_size; /* out parameter set by kernel */ | ||
| 47 | |||
| 48 | size_t :0; | ||
| 49 | }; | ||
| 50 | #define bpf_log_opts__last_field true_size | ||
| 51 | |||
| 40 | struct bpf_map_create_opts { | 52 | struct bpf_map_create_opts { |
| 41 | size_t sz; /* size of this struct for forward/backward compatibility */ | 53 | size_t sz; /* size of this struct for forward/backward compatibility */ |
| 42 | 54 | ||
| @@ -57,9 +69,12 @@ struct bpf_map_create_opts { · 14 unmodified lines | |||
| 57 | 69 | ||
| 58 | const void *excl_prog_hash; | 70 | const void *excl_prog_hash; |
| 59 | __u32 excl_prog_hash_size; | 71 | __u32 excl_prog_hash_size; |
| 72 | |||
| 73 | struct bpf_log_opts *log_opts; | ||
| 74 | |||
| 60 | size_t :0; | 75 | size_t :0; |
| 61 | }; | 76 | }; |
| 62 | #define bpf_map_create_opts__last_field excl_prog_hash_size | 77 | #define bpf_map_create_opts__last_field log_opts |
| 63 | 78 | ||
| 64 | LIBBPF_API int bpf_map_create(enum bpf_map_type map_type, | 79 | LIBBPF_API int bpf_map_create(enum bpf_map_type map_type, |
| 65 | const char *map_name, | 80 | const char *map_name, |
| @@ -429,6 +444,7 @@ struct bpf_link_create_opts { · 363 unmodified lines | |||
| 429 | const unsigned long *ref_ctr_offsets; | 444 | const unsigned long *ref_ctr_offsets; |
| 430 | const __u64 *cookies; | 445 | const __u64 *cookies; |
| 431 | __u32 pid; | 446 | __u32 pid; |
| 447 | __u32 path_fd; | ||
| 432 | } uprobe_multi; | 448 | } uprobe_multi; |
| 433 | struct { | 449 | struct { |
| 434 | __u64 cookie; | 450 | __u64 cookie; |
| @@ -454,10 +470,15 @@ struct bpf_link_create_opts { · 19 unmodified lines | |||
| 454 | __u32 relative_id; | 470 | __u32 relative_id; |
| 455 | __u64 expected_revision; | 471 | __u64 expected_revision; |
| 456 | } cgroup; | 472 | } cgroup; |
| 473 | struct { | ||
| 474 | const __u32 *ids; | ||
| 475 | const __u64 *cookies; | ||
| 476 | __u32 cnt; | ||
| 477 | } tracing_multi; | ||
| 457 | }; | 478 | }; |
| 458 | size_t :0; | 479 | size_t :0; |
| 459 | }; | 480 | }; |
| 460 | #define bpf_link_create_opts__last_field uprobe_multi.pid | 481 | #define bpf_link_create_opts__last_field uprobe_multi.path_fd |
| 461 | 482 | ||
| 462 | LIBBPF_API int bpf_link_create(int prog_fd, int target_fd, | 483 | LIBBPF_API int bpf_link_create(int prog_fd, int target_fd, |
| 463 | enum bpf_attach_type attach_type, | 484 | enum bpf_attach_type attach_type, |
| @@ -421,7 +421,7 @@ static int btf_type_size_unknown(const struct btf *btf, const struct btf_type *t · 420 unmodified lines | |||
| 421 | { | 421 | { |
| 422 | __u32 l_cnt = btf->hdr.layout_len / sizeof(struct btf_layout); | 422 | __u32 l_cnt = btf->hdr.layout_len / sizeof(struct btf_layout); |
| 423 | struct btf_layout *l = btf->layout; | 423 | struct btf_layout *l = btf->layout; |
| 424 | __u16 vlen = btf_vlen(t); | 424 | __u32 vlen = btf_vlen(t); |
| 425 | __u32 kind = btf_kind(t); | 425 | __u32 kind = btf_kind(t); |
| 426 | 426 | ||
| 427 | /* Fall back to base BTF if needed as they share layout information */ | 427 | /* Fall back to base BTF if needed as they share layout information */ |
| @@ -454,7 +454,7 @@ static int btf_type_size_unknown(const struct btf *btf, const struct btf_type *t · 26 unmodified lines | |||
| 454 | static int btf_type_size(const struct btf *btf, const struct btf_type *t) | 454 | static int btf_type_size(const struct btf *btf, const struct btf_type *t) |
| 455 | { | 455 | { |
| 456 | const int base_size = sizeof(struct btf_type); | 456 | const int base_size = sizeof(struct btf_type); |
| 457 | __u16 vlen = btf_vlen(t); | 457 | __u32 vlen = btf_vlen(t); |
| 458 | 458 | ||
| 459 | switch (btf_kind(t)) { | 459 | switch (btf_kind(t)) { |
| 460 | case BTF_KIND_FWD: | 460 | case BTF_KIND_FWD: |
| @@ -506,7 +506,7 @@ static int btf_bswap_type_rest(struct btf_type *t) · 45 unmodified lines | |||
| 506 | struct btf_array *a; | 506 | struct btf_array *a; |
| 507 | struct btf_param *p; | 507 | struct btf_param *p; |
| 508 | struct btf_enum *e; | 508 | struct btf_enum *e; |
| 509 | __u16 vlen = btf_vlen(t); | 509 | __u32 vlen = btf_vlen(t); |
| 510 | int i; | 510 | int i; |
| 511 | 511 | ||
| 512 | switch (btf_kind(t)) { | 512 | switch (btf_kind(t)) { |
| @@ -1007,7 +1007,7 @@ int btf__align_of(const struct btf *btf, __u32 id) · 494 unmodified lines | |||
| 1007 | case BTF_KIND_STRUCT: | 1007 | case BTF_KIND_STRUCT: |
| 1008 | case BTF_KIND_UNION: { | 1008 | case BTF_KIND_UNION: { |
| 1009 | const struct btf_member *m = btf_members(t); | 1009 | const struct btf_member *m = btf_members(t); |
| 1010 | __u16 vlen = btf_vlen(t); | 1010 | __u32 vlen = btf_vlen(t); |
| 1011 | int i, max_align = 1, align; | 1011 | int i, max_align = 1, align; |
| 1012 | 1012 | ||
| 1013 | for (i = 0; i < vlen; i++, m++) { | 1013 | for (i = 0; i < vlen; i++, m++) { |
| @@ -2121,9 +2121,12 @@ static void *btf_add_type_mem(struct btf *btf, size_t add_sz) · 1107 unmodified lines | |||
| 2121 | btf->hdr.type_len, UINT_MAX, add_sz); | 2121 | btf->hdr.type_len, UINT_MAX, add_sz); |
| 2122 | } | 2122 | } |
| 2123 | 2123 | ||
| 2124 | static void btf_type_inc_vlen(struct btf_type *t) | 2124 | static int btf_type_inc_vlen(struct btf_type *t) |
| 2125 | { | 2125 | { |
| 2126 | if (btf_vlen(t) == BTF_MAX_VLEN) | ||
| 2127 | return -ENOSPC; | ||
| 2126 | t->info = btf_type_info(btf_kind(t), btf_vlen(t) + 1, btf_kflag(t)); | 2128 | t->info = btf_type_info(btf_kind(t), btf_vlen(t) + 1, btf_kflag(t)); |
| 2129 | return 0; | ||
| 2127 | } | 2130 | } |
| 2128 | 2131 | ||
| 2129 | static void btf_hdr_update_type_len(struct btf *btf, int new_len) | 2132 | static void btf_hdr_update_type_len(struct btf *btf, int new_len) |
| @@ -2652,6 +2655,8 @@ int btf__add_field(struct btf *btf, const char *name, int type_id, · 522 unmodified lines | |||
| 2652 | t = btf_last_type(btf); | 2655 | t = btf_last_type(btf); |
| 2653 | if (!btf_is_composite(t)) | 2656 | if (!btf_is_composite(t)) |
| 2654 | return libbpf_err(-EINVAL); | 2657 | return libbpf_err(-EINVAL); |
| 2658 | if (btf_vlen(t) == BTF_MAX_VLEN) | ||
| 2659 | return libbpf_err(-ENOSPC); | ||
| 2655 | 2660 | ||
| 2656 | if (validate_type_id(type_id)) | 2661 | if (validate_type_id(type_id)) |
| 2657 | return libbpf_err(-EINVAL); | 2662 | return libbpf_err(-EINVAL); |
| @@ -2686,6 +2691,7 @@ int btf__add_field(struct btf *btf, const char *name, int type_id, · 28 unmodified lines | |||
| 2686 | 2691 | ||
| 2687 | /* btf_add_type_mem can invalidate t pointer */ | 2692 | /* btf_add_type_mem can invalidate t pointer */ |
| 2688 | t = btf_last_type(btf); | 2693 | t = btf_last_type(btf); |
| 2694 | |||
| 2689 | /* update parent type's vlen and kflag */ | 2695 | /* update parent type's vlen and kflag */ |
| 2690 | t->info = btf_type_info(btf_kind(t), btf_vlen(t) + 1, is_bitfield || btf_kflag(t)); | 2696 | t->info = btf_type_info(btf_kind(t), btf_vlen(t) + 1, is_bitfield || btf_kflag(t)); |
| 2691 | 2697 | ||
| @@ -2796,7 +2802,9 @@ int btf__add_enum_value(struct btf *btf, const char *name, __s64 value) · 104 unmodified lines | |||
| 2796 | 2802 | ||
| 2797 | /* update parent type's vlen */ | 2803 | /* update parent type's vlen */ |
| 2798 | t = btf_last_type(btf); | 2804 | t = btf_last_type(btf); |
| 2799 | btf_type_inc_vlen(t); | 2805 | err = btf_type_inc_vlen(t); |
| 2806 | if (err) | ||
| 2807 | return libbpf_err(err); | ||
| 2800 | 2808 | ||
| 2801 | /* if negative value, set signedness to signed */ | 2809 | /* if negative value, set signedness to signed */ |
| 2802 | if (value < 0) | 2810 | if (value < 0) |
| @@ -2873,7 +2881,9 @@ int btf__add_enum64_value(struct btf *btf, const char *name, __u64 value) · 70 unmodified lines | |||
| 2873 | 2881 | ||
| 2874 | /* update parent type's vlen */ | 2882 | /* update parent type's vlen */ |
| 2875 | t = btf_last_type(btf); | 2883 | t = btf_last_type(btf); |
| 2876 | btf_type_inc_vlen(t); | 2884 | err = btf_type_inc_vlen(t); |
| 2885 | if (err) | ||
| 2886 | return libbpf_err(err); | ||
| 2877 | 2887 | ||
| 2878 | btf_hdr_update_type_len(btf, btf->hdr.type_len + sz); | 2888 | btf_hdr_update_type_len(btf, btf->hdr.type_len + sz); |
| 2879 | return 0; | 2889 | return 0; |
| @@ -3115,7 +3125,9 @@ int btf__add_func_param(struct btf *btf, const char *name, int type_id) · 235 unmodified lines | |||
| 3115 | 3125 | ||
| 3116 | /* update parent type's vlen */ | 3126 | /* update parent type's vlen */ |
| 3117 | t = btf_last_type(btf); | 3127 | t = btf_last_type(btf); |
| 3118 | btf_type_inc_vlen(t); | 3128 | err = btf_type_inc_vlen(t); |
| 3129 | if (err) | ||
| 3130 | return libbpf_err(err); | ||
| 3119 | 3131 | ||
| 3120 | btf_hdr_update_type_len(btf, btf->hdr.type_len + sz); | 3132 | btf_hdr_update_type_len(btf, btf->hdr.type_len + sz); |
| 3121 | return 0; | 3133 | return 0; |
| @@ -3257,7 +3269,9 @@ int btf__add_datasec_var_info(struct btf *btf, int var_type_id, __u32 offset, __ · 135 unmodified lines | |||
| 3257 | 3269 | ||
| 3258 | /* update parent type's vlen */ | 3270 | /* update parent type's vlen */ |
| 3259 | t = btf_last_type(btf); | 3271 | t = btf_last_type(btf); |
| 3260 | btf_type_inc_vlen(t); | 3272 | err = btf_type_inc_vlen(t); |
| 3273 | if (err) | ||
| 3274 | return libbpf_err(err); | ||
| 3261 | 3275 | ||
| 3262 | btf_hdr_update_type_len(btf, btf->hdr.type_len + sz); | 3276 | btf_hdr_update_type_len(btf, btf->hdr.type_len + sz); |
| 3263 | return 0; | 3277 | return 0; |
| @@ -4311,7 +4325,7 @@ static long btf_hash_enum(struct btf_type *t) · 1047 unmodified lines | |||
| 4311 | static bool btf_equal_enum_members(struct btf_type *t1, struct btf_type *t2) | 4325 | static bool btf_equal_enum_members(struct btf_type *t1, struct btf_type *t2) |
| 4312 | { | 4326 | { |
| 4313 | const struct btf_enum *m1, *m2; | 4327 | const struct btf_enum *m1, *m2; |
| 4314 | __u16 vlen; | 4328 | __u32 vlen; |
| 4315 | int i; | 4329 | int i; |
| 4316 | 4330 | ||
| 4317 | vlen = btf_vlen(t1); | 4331 | vlen = btf_vlen(t1); |
| @@ -4329,7 +4343,7 @@ static bool btf_equal_enum_members(struct btf_type *t1, struct btf_type *t2) · 11 unmodified lines | |||
| 4329 | static bool btf_equal_enum64_members(struct btf_type *t1, struct btf_type *t2) | 4343 | static bool btf_equal_enum64_members(struct btf_type *t1, struct btf_type *t2) |
| 4330 | { | 4344 | { |
| 4331 | const struct btf_enum64 *m1, *m2; | 4345 | const struct btf_enum64 *m1, *m2; |
| 4332 | __u16 vlen; | 4346 | __u32 vlen; |
| 4333 | int i; | 4347 | int i; |
| 4334 | 4348 | ||
| 4335 | vlen = btf_vlen(t1); | 4349 | vlen = btf_vlen(t1); |
| @@ -4406,7 +4420,7 @@ static long btf_hash_struct(struct btf_type *t) · 70 unmodified lines | |||
| 4406 | static bool btf_shallow_equal_struct(struct btf_type *t1, struct btf_type *t2) | 4420 | static bool btf_shallow_equal_struct(struct btf_type *t1, struct btf_type *t2) |
| 4407 | { | 4421 | { |
| 4408 | const struct btf_member *m1, *m2; | 4422 | const struct btf_member *m1, *m2; |
| 4409 | __u16 vlen; | 4423 | __u32 vlen; |
| 4410 | int i; | 4424 | int i; |
| 4411 | 4425 | ||
| 4412 | if (!btf_equal_common(t1, t2)) | 4426 | if (!btf_equal_common(t1, t2)) |
| @@ -4482,7 +4496,7 @@ static bool btf_compat_array(struct btf_type *t1, struct btf_type *t2) · 69 unmodified lines | |||
| 4482 | static long btf_hash_fnproto(struct btf_type *t) | 4496 | static long btf_hash_fnproto(struct btf_type *t) |
| 4483 | { | 4497 | { |
| 4484 | const struct btf_param *member = btf_params(t); | 4498 | const struct btf_param *member = btf_params(t); |
| 4485 | __u16 vlen = btf_vlen(t); | 4499 | __u32 vlen = btf_vlen(t); |
| 4486 | long h = btf_hash_common(t); | 4500 | long h = btf_hash_common(t); |
| 4487 | int i; | 4501 | int i; |
| 4488 | 4502 | ||
| @@ -4504,7 +4518,7 @@ static long btf_hash_fnproto(struct btf_type *t) · 15 unmodified lines | |||
| 4504 | static bool btf_equal_fnproto(struct btf_type *t1, struct btf_type *t2) | 4518 | static bool btf_equal_fnproto(struct btf_type *t1, struct btf_type *t2) |
| 4505 | { | 4519 | { |
| 4506 | const struct btf_param *m1, *m2; | 4520 | const struct btf_param *m1, *m2; |
| 4507 | __u16 vlen; | 4521 | __u32 vlen; |
| 4508 | int i; | 4522 | int i; |
| 4509 | 4523 | ||
| 4510 | if (!btf_equal_common(t1, t2)) | 4524 | if (!btf_equal_common(t1, t2)) |
| @@ -4530,7 +4544,7 @@ static bool btf_equal_fnproto(struct btf_type *t1, struct btf_type *t2) · 19 unmodified lines | |||
| 4530 | static bool btf_compat_fnproto(struct btf_type *t1, struct btf_type *t2) | 4544 | static bool btf_compat_fnproto(struct btf_type *t1, struct btf_type *t2) |
| 4531 | { | 4545 | { |
| 4532 | const struct btf_param *m1, *m2; | 4546 | const struct btf_param *m1, *m2; |
| 4533 | __u16 vlen; | 4547 | __u32 vlen; |
| 4534 | int i; | 4548 | int i; |
| 4535 | 4549 | ||
| 4536 | /* skip return type ID */ | 4550 | /* skip return type ID */ |
| @@ -4578,12 +4592,14 @@ static int btf_dedup_prep(struct btf_dedup *d) · 41 unmodified lines | |||
| 4578 | case BTF_KIND_RESTRICT: | 4592 | case BTF_KIND_RESTRICT: |
| 4579 | case BTF_KIND_PTR: | 4593 | case BTF_KIND_PTR: |
| 4580 | case BTF_KIND_FWD: | 4594 | case BTF_KIND_FWD: |
| 4581 | case BTF_KIND_TYPEDEF: | ||
| 4582 | case BTF_KIND_FUNC: | 4595 | case BTF_KIND_FUNC: |
| 4583 | case BTF_KIND_FLOAT: | 4596 | case BTF_KIND_FLOAT: |
| 4584 | case BTF_KIND_TYPE_TAG: | 4597 | case BTF_KIND_TYPE_TAG: |
| 4585 | h = btf_hash_common(t); | 4598 | h = btf_hash_common(t); |
| 4586 | break; | 4599 | break; |
| 4600 | case BTF_KIND_TYPEDEF: | ||
| 4601 | h = btf_hash_typedef(t); | ||
| 4602 | break; | ||
| 4587 | case BTF_KIND_INT: | 4603 | case BTF_KIND_INT: |
| 4588 | case BTF_KIND_DECL_TAG: | 4604 | case BTF_KIND_DECL_TAG: |
| 4589 | h = btf_hash_int_decl_tag(t); | 4605 | h = btf_hash_int_decl_tag(t); |
| @@ -5077,7 +5093,7 @@ static int btf_dedup_is_equiv(struct btf_dedup *d, __u32 cand_id, · 487 unmodified lines | |||
| 5077 | case BTF_KIND_STRUCT: | 5093 | case BTF_KIND_STRUCT: |
| 5078 | case BTF_KIND_UNION: { | 5094 | case BTF_KIND_UNION: { |
| 5079 | const struct btf_member *cand_m, *canon_m; | 5095 | const struct btf_member *cand_m, *canon_m; |
| 5080 | __u16 vlen; | 5096 | __u32 vlen; |
| 5081 | 5097 | ||
| 5082 | if (!btf_shallow_equal_struct(cand_type, canon_type)) | 5098 | if (!btf_shallow_equal_struct(cand_type, canon_type)) |
| 5083 | return 0; | 5099 | return 0; |
| @@ -5105,7 +5121,7 @@ static int btf_dedup_is_equiv(struct btf_dedup *d, __u32 cand_id, · 21 unmodified lines | |||
| 5105 | 5121 | ||
| 5106 | case BTF_KIND_FUNC_PROTO: { | 5122 | case BTF_KIND_FUNC_PROTO: { |
| 5107 | const struct btf_param *cand_p, *canon_p; | 5123 | const struct btf_param *cand_p, *canon_p; |
| 5108 | __u16 vlen; | 5124 | __u32 vlen; |
| 5109 | 5125 | ||
| 5110 | if (!btf_compat_fnproto(cand_type, canon_type)) | 5126 | if (!btf_compat_fnproto(cand_type, canon_type)) |
| 5111 | return 0; | 5127 | return 0; |
| @@ -5439,7 +5455,7 @@ static int btf_dedup_ref_type(struct btf_dedup *d, __u32 type_id) · 327 unmodified lines | |||
| 5439 | 5455 | ||
| 5440 | case BTF_KIND_FUNC_PROTO: { | 5456 | case BTF_KIND_FUNC_PROTO: { |
| 5441 | struct btf_param *param; | 5457 | struct btf_param *param; |
| 5442 | __u16 vlen; | 5458 | __u32 vlen; |
| 5443 | int i; | 5459 | int i; |
| 5444 | 5460 | ||
| 5445 | ref_type_id = btf_dedup_ref_type(d, t->type); | 5461 | ref_type_id = btf_dedup_ref_type(d, t->type); |
| @@ -435,7 +435,7 @@ static inline __u16 btf_kind(const struct btf_type *t) · 434 unmodified lines | |||
| 435 | return BTF_INFO_KIND(t->info); | 435 | return BTF_INFO_KIND(t->info); |
| 436 | } | 436 | } |
| 437 | 437 | ||
| 438 | static inline __u16 btf_vlen(const struct btf_type *t) | 438 | static inline __u32 btf_vlen(const struct btf_type *t) |
| 439 | { | 439 | { |
| 440 | return BTF_INFO_VLEN(t->info); | 440 | return BTF_INFO_VLEN(t->info); |
| 441 | } | 441 | } |
| @@ -316,7 +316,7 @@ static int btf_dump_mark_referenced(struct btf_dump *d) · 315 unmodified lines | |||
| 316 | { | 316 | { |
| 317 | int i, j, n = btf__type_cnt(d->btf); | 317 | int i, j, n = btf__type_cnt(d->btf); |
| 318 | const struct btf_type *t; | 318 | const struct btf_type *t; |
| 319 | __u16 vlen; | 319 | __u32 vlen; |
| 320 | 320 | ||
| 321 | for (i = d->last_id + 1; i < n; i++) { | 321 | for (i = d->last_id + 1; i < n; i++) { |
| 322 | t = btf__type_by_id(d->btf, i); | 322 | t = btf__type_by_id(d->btf, i); |
| @@ -485,7 +485,7 @@ static int btf_dump_order_type(struct btf_dump *d, __u32 id, bool through_ptr) · 162 unmodified lines | |||
| 485 | */ | 485 | */ |
| 486 | struct btf_dump_type_aux_state *tstate = &d->type_states[id]; | 486 | struct btf_dump_type_aux_state *tstate = &d->type_states[id]; |
| 487 | const struct btf_type *t; | 487 | const struct btf_type *t; |
| 488 | __u16 vlen; | 488 | __u32 vlen; |
| 489 | int err, i; | 489 | int err, i; |
| 490 | 490 | ||
| 491 | /* return true, letting typedefs know that it's ok to be emitted */ | 491 | /* return true, letting typedefs know that it's ok to be emitted */ |
| @@ -798,7 +798,7 @@ static void btf_dump_emit_type(struct btf_dump *d, __u32 id, __u32 cont_id) · 306 unmodified lines | |||
| 798 | */ | 798 | */ |
| 799 | if (top_level_def || t->name_off == 0) { | 799 | if (top_level_def || t->name_off == 0) { |
| 800 | const struct btf_member *m = btf_members(t); | 800 | const struct btf_member *m = btf_members(t); |
| 801 | __u16 vlen = btf_vlen(t); | 801 | __u32 vlen = btf_vlen(t); |
| 802 | int i, new_cont_id; | 802 | int i, new_cont_id; |
| 803 | 803 | ||
| 804 | new_cont_id = t->name_off == 0 ? cont_id : id; | 804 | new_cont_id = t->name_off == 0 ? cont_id : id; |
| @@ -820,7 +820,7 @@ static void btf_dump_emit_type(struct btf_dump *d, __u32 id, __u32 cont_id) · 15 unmodified lines | |||
| 820 | break; | 820 | break; |
| 821 | case BTF_KIND_FUNC_PROTO: { | 821 | case BTF_KIND_FUNC_PROTO: { |
| 822 | const struct btf_param *p = btf_params(t); | 822 | const struct btf_param *p = btf_params(t); |
| 823 | __u16 n = btf_vlen(t); | 823 | __u32 n = btf_vlen(t); |
| 824 | int i; | 824 | int i; |
| 825 | 825 | ||
| 826 | btf_dump_emit_type(d, t->type, cont_id); | 826 | btf_dump_emit_type(d, t->type, cont_id); |
| @@ -839,7 +839,7 @@ static bool btf_is_struct_packed(const struct btf *btf, __u32 id, · 12 unmodified lines | |||
| 839 | { | 839 | { |
| 840 | const struct btf_member *m; | 840 | const struct btf_member *m; |
| 841 | int max_align = 1, align, i, bit_sz; | 841 | int max_align = 1, align, i, bit_sz; |
| 842 | __u16 vlen; | 842 | __u32 vlen; |
| 843 | 843 | ||
| 844 | m = btf_members(t); | 844 | m = btf_members(t); |
| 845 | vlen = btf_vlen(t); | 845 | vlen = btf_vlen(t); |
| @@ -973,7 +973,7 @@ static void btf_dump_emit_struct_def(struct btf_dump *d, · 127 unmodified lines | |||
| 973 | bool is_struct = btf_is_struct(t); | 973 | bool is_struct = btf_is_struct(t); |
| 974 | bool packed, prev_bitfield = false; | 974 | bool packed, prev_bitfield = false; |
| 975 | int align, i, off = 0; | 975 | int align, i, off = 0; |
| 976 | __u16 vlen = btf_vlen(t); | 976 | __u32 vlen = btf_vlen(t); |
| 977 | 977 | ||
| 978 | align = btf__align_of(d->btf, id); | 978 | align = btf__align_of(d->btf, id); |
| 979 | packed = is_struct ? btf_is_struct_packed(d->btf, id, t) : 0; | 979 | packed = is_struct ? btf_is_struct_packed(d->btf, id, t) : 0; |
| @@ -1064,7 +1064,7 @@ static void btf_dump_emit_enum_fwd(struct btf_dump *d, __u32 id, · 84 unmodified lines | |||
| 1064 | 1064 | ||
| 1065 | static void btf_dump_emit_enum32_val(struct btf_dump *d, | 1065 | static void btf_dump_emit_enum32_val(struct btf_dump *d, |
| 1066 | const struct btf_type *t, | 1066 | const struct btf_type *t, |
| 1067 | int lvl, __u16 vlen) | 1067 | int lvl, __u32 vlen) |
| 1068 | { | 1068 | { |
| 1069 | const struct btf_enum *v = btf_enum(t); | 1069 | const struct btf_enum *v = btf_enum(t); |
| 1070 | bool is_signed = btf_kflag(t); | 1070 | bool is_signed = btf_kflag(t); |
| @@ -1089,7 +1089,7 @@ static void btf_dump_emit_enum32_val(struct btf_dump *d, · 18 unmodified lines | |||
| 1089 | 1089 | ||
| 1090 | static void btf_dump_emit_enum64_val(struct btf_dump *d, | 1090 | static void btf_dump_emit_enum64_val(struct btf_dump *d, |
| 1091 | const struct btf_type *t, | 1091 | const struct btf_type *t, |
| 1092 | int lvl, __u16 vlen) | 1092 | int lvl, __u32 vlen) |
| 1093 | { | 1093 | { |
| 1094 | const struct btf_enum64 *v = btf_enum64(t); | 1094 | const struct btf_enum64 *v = btf_enum64(t); |
| 1095 | bool is_signed = btf_kflag(t); | 1095 | bool is_signed = btf_kflag(t); |
| @@ -1122,7 +1122,7 @@ static void btf_dump_emit_enum_def(struct btf_dump *d, __u32 id, · 26 unmodified lines | |||
| 1122 | const struct btf_type *t, | 1122 | const struct btf_type *t, |
| 1123 | int lvl) | 1123 | int lvl) |
| 1124 | { | 1124 | { |
| 1125 | __u16 vlen = btf_vlen(t); | 1125 | __u32 vlen = btf_vlen(t); |
| 1126 | 1126 | ||
| 1127 | btf_dump_printf(d, "enum%s%s", | 1127 | btf_dump_printf(d, "enum%s%s", |
| 1128 | t->name_off ? " " : "", | 1128 | t->name_off ? " " : "", |
| @@ -1542,7 +1542,7 @@ static void btf_dump_emit_type_chain(struct btf_dump *d, · 413 unmodified lines | |||
| 1542 | } | 1542 | } |
| 1543 | case BTF_KIND_FUNC_PROTO: { | 1543 | case BTF_KIND_FUNC_PROTO: { |
| 1544 | const struct btf_param *p = btf_params(t); | 1544 | const struct btf_param *p = btf_params(t); |
| 1545 | __u16 vlen = btf_vlen(t); | 1545 | __u32 vlen = btf_vlen(t); |
| 1546 | int i; | 1546 | int i; |
| 1547 | 1547 | ||
| 1548 | /* | 1548 | /* |
| @@ -2159,7 +2159,7 @@ static int btf_dump_struct_data(struct btf_dump *d, · 610 unmodified lines | |||
| 2159 | const void *data) | 2159 | const void *data) |
| 2160 | { | 2160 | { |
| 2161 | const struct btf_member *m = btf_members(t); | 2161 | const struct btf_member *m = btf_members(t); |
| 2162 | __u16 n = btf_vlen(t); | 2162 | __u32 n = btf_vlen(t); |
| 2163 | int i, err = 0; | 2163 | int i, err = 0; |
| 2164 | 2164 | ||
| 2165 | /* note that we increment depth before calling btf_dump_print() below; | 2165 | /* note that we increment depth before calling btf_dump_print() below; |
| @@ -2449,7 +2449,7 @@ static int btf_dump_type_data_check_zero(struct btf_dump *d, · 283 unmodified lines | |||
| 2449 | case BTF_KIND_STRUCT: | 2449 | case BTF_KIND_STRUCT: |
| 2450 | case BTF_KIND_UNION: { | 2450 | case BTF_KIND_UNION: { |
| 2451 | const struct btf_member *m = btf_members(t); | 2451 | const struct btf_member *m = btf_members(t); |
| 2452 | __u16 n = btf_vlen(t); | 2452 | __u32 n = btf_vlen(t); |
| 2453 | 2453 | ||
| 2454 | /* if any struct/union member is non-zero, the struct/union | 2454 | /* if any struct/union member is non-zero, the struct/union |
| 2455 | * is considered non-zero and dumped. | 2455 | * is considered non-zero and dumped. |
| @@ -615,6 +615,11 @@ static int probe_kern_btf_layout(int token_fd) · 614 unmodified lines | |||
| 615 | (char *)layout, token_fd)); | 615 | (char *)layout, token_fd)); |
| 616 | } | 616 | } |
| 617 | 617 | ||
| 618 | static int probe_bpf_syscall_common_attrs(int token_fd) | ||
| 619 | { | ||
| 620 | return probe_sys_bpf_ext(); | ||
| 621 | } | ||
| 622 | |||
| 618 | typedef int (*feature_probe_fn)(int /* token_fd */); | 623 | typedef int (*feature_probe_fn)(int /* token_fd */); |
| 619 | 624 | ||
| 620 | static struct kern_feature_cache feature_cache; | 625 | static struct kern_feature_cache feature_cache; |
| @@ -699,6 +704,9 @@ static struct kern_feature_desc { · 78 unmodified lines | |||
| 699 | [FEAT_BTF_LAYOUT] = { | 704 | [FEAT_BTF_LAYOUT] = { |
| 700 | "kernel supports BTF layout", probe_kern_btf_layout, | 705 | "kernel supports BTF layout", probe_kern_btf_layout, |
| 701 | }, | 706 | }, |
| 707 | [FEAT_BPF_SYSCALL_COMMON_ATTRS] = { | ||
| 708 | "BPF syscall common attributes support", probe_bpf_syscall_common_attrs, | ||
| 709 | }, | ||
| 702 | }; | 710 | }; |
| 703 | 711 | ||
| 704 | bool feat_supported(struct kern_feature_cache *cache, enum kern_feature_id feat_id) | 712 | bool feat_supported(struct kern_feature_cache *cache, enum kern_feature_id feat_id) |
| @@ -63,6 +63,7 @@ static int realloc_insn_buf(struct bpf_gen *gen, __u32 size) · 62 unmodified lines | |||
| 63 | gen->error = -ENOMEM; | 63 | gen->error = -ENOMEM; |
| 64 | free(gen->insn_start); | 64 | free(gen->insn_start); |
| 65 | gen->insn_start = NULL; | 65 | gen->insn_start = NULL; |
| 66 | gen->insn_cur = NULL; | ||
| 66 | return -ENOMEM; | 67 | return -ENOMEM; |
| 67 | } | 68 | } |
| 68 | gen->insn_start = insn_start; | 69 | gen->insn_start = insn_start; |
| @@ -86,6 +87,7 @@ static int realloc_data_buf(struct bpf_gen *gen, __u32 size) · 17 unmodified lines | |||
| 86 | gen->error = -ENOMEM; | 87 | gen->error = -ENOMEM; |
| 87 | free(gen->data_start); | 88 | free(gen->data_start); |
| 88 | gen->data_start = NULL; | 89 | gen->data_start = NULL; |
| 90 | gen->data_cur = NULL; | ||
| 89 | return -ENOMEM; | 91 | return -ENOMEM; |
| 90 | } | 92 | } |
| 91 | gen->data_start = data_start; | 93 | gen->data_start = data_start; |
| @@ -158,10 +160,16 @@ void bpf_gen__init(struct bpf_gen *gen, int log_level, int nr_progs, int nr_maps · 66 unmodified lines | |||
| 158 | 160 | ||
| 159 | static int add_data(struct bpf_gen *gen, const void *data, __u32 size) | 161 | static int add_data(struct bpf_gen *gen, const void *data, __u32 size) |
| 160 | { | 162 | { |
| 161 | __u32 size8 = roundup(size, 8); | ||
| 162 | __u64 zero = 0; | 163 | __u64 zero = 0; |
| 164 | __u32 size8; | ||
| 163 | void *prev; | 165 | void *prev; |
| 164 | 166 | ||
| 167 | if (size > INT32_MAX) { | ||
| 168 | gen->error = -ERANGE; | ||
| 169 | return 0; | ||
| 170 | } | ||
| 171 | size8 = roundup(size, 8); | ||
| 172 | |||
| 165 | if (realloc_data_buf(gen, size8)) | 173 | if (realloc_data_buf(gen, size8)) |
| 166 | return 0; | 174 | return 0; |
| 167 | prev = gen->data_cur; | 175 | prev = gen->data_cur; |
| @@ -293,7 +301,6 @@ static void emit_check_err(struct bpf_gen *gen) · 125 unmodified lines | |||
| 293 | emit(gen, BPF_JMP_IMM(BPF_JSLT, BPF_REG_7, 0, off)); | 301 | emit(gen, BPF_JMP_IMM(BPF_JSLT, BPF_REG_7, 0, off)); |
| 294 | } else { | 302 | } else { |
| 295 | gen->error = -ERANGE; | 303 | gen->error = -ERANGE; |
| 296 | emit(gen, BPF_JMP_IMM(BPF_JA, 0, 0, -1)); | ||
| 297 | } | 304 | } |
| 298 | } | 305 | } |
| 299 | 306 | ||
| @@ -398,13 +405,12 @@ int bpf_gen__finish(struct bpf_gen *gen, int nr_progs, int nr_maps) · 98 unmodified lines | |||
| 398 | blob_fd_array_off(gen, i)); | 405 | blob_fd_array_off(gen, i)); |
| 399 | emit(gen, BPF_MOV64_IMM(BPF_REG_0, 0)); | 406 | emit(gen, BPF_MOV64_IMM(BPF_REG_0, 0)); |
| 400 | emit(gen, BPF_EXIT_INSN()); | 407 | emit(gen, BPF_EXIT_INSN()); |
| 401 | if (OPTS_GET(gen->opts, gen_hash, false)) | ||
| 402 | compute_sha_update_offsets(gen); | ||
| 403 | |||
| 404 | pr_debug("gen: finish %s\n", errstr(gen->error)); | ||
| 405 | if (!gen->error) { | 408 | if (!gen->error) { |
| 406 | struct gen_loader_opts *opts = gen->opts; | 409 | struct gen_loader_opts *opts = gen->opts; |
| 407 | 410 | ||
| 411 | if (OPTS_GET(opts, gen_hash, false)) | ||
| 412 | compute_sha_update_offsets(gen); | ||
| 413 | |||
| 408 | opts->insns = gen->insn_start; | 414 | opts->insns = gen->insn_start; |
| 409 | opts->insns_sz = gen->insn_cur - gen->insn_start; | 415 | opts->insns_sz = gen->insn_cur - gen->insn_start; |
| 410 | opts->data = gen->data_start; | 416 | opts->data = gen->data_start; |
| @@ -419,6 +425,7 @@ int bpf_gen__finish(struct bpf_gen *gen, int nr_progs, int nr_maps) · 8 unmodified lines | |||
| 419 | bpf_insn_bswap(insn++); | 425 | bpf_insn_bswap(insn++); |
| 420 | } | 426 | } |
| 421 | } | 427 | } |
| 428 | pr_debug("gen: finish %s\n", errstr(gen->error)); | ||
| 422 | return gen->error; | 429 | return gen->error; |
| 423 | } | 430 | } |
| 424 | 431 | ||
| @@ -545,13 +552,22 @@ void bpf_gen__map_create(struct bpf_gen *gen, · 120 unmodified lines | |||
| 545 | default: | 552 | default: |
| 546 | break; | 553 | break; |
| 547 | } | 554 | } |
| 548 | /* conditionally update max_entries */ | 555 | |
| 549 | if (map_idx >= 0) | 556 | /* |
| 557 | * Conditionally update max_entries from the host-supplied loader | ||
| 558 | * ctx. This sizes the map at runtime, but for a signed loader | ||
| 559 | * (gen_hash) it would let an untrusted host re-dimension the | ||
| 560 | * program's maps after emit_signature_match(), outside what the | ||
| 561 | * signature attests to. Keep the signer-provided max_entries | ||
| 562 | * baked into the blob in that case. | ||
| 563 | */ | ||
| 564 | if (map_idx >= 0 && !OPTS_GET(gen->opts, gen_hash, false)) | ||
| 550 | move_ctx2blob(gen, attr_field(map_create_attr, max_entries), 4, | 565 | move_ctx2blob(gen, attr_field(map_create_attr, max_entries), 4, |
| 551 | sizeof(struct bpf_loader_ctx) + | 566 | sizeof(struct bpf_loader_ctx) + |
| 552 | sizeof(struct bpf_map_desc) * map_idx + | 567 | sizeof(struct bpf_map_desc) * map_idx + |
| 553 | offsetof(struct bpf_map_desc, max_entries), | 568 | offsetof(struct bpf_map_desc, max_entries), |
| 554 | true /* check that max_entries != 0 */); | 569 | true /* check that max_entries != 0 */); |
| 570 | |||
| 555 | /* emit MAP_CREATE command */ | 571 | /* emit MAP_CREATE command */ |
| 556 | emit_sys_bpf(gen, BPF_MAP_CREATE, map_create_attr, attr_size); | 572 | emit_sys_bpf(gen, BPF_MAP_CREATE, map_create_attr, attr_size); |
| 557 | debug_ret(gen, "map_create %s idx %d type %d value_size %d value_btf_id %d", | 573 | debug_ret(gen, "map_create %s idx %d type %d value_size %d value_btf_id %d", |
| @@ -585,6 +601,23 @@ static void emit_signature_match(struct bpf_gen *gen) · 27 unmodified lines | |||
| 585 | __s64 off; | 601 | __s64 off; |
| 586 | int i; | 602 | int i; |
| 587 | 603 | ||
| 604 | /* | ||
| 605 | * Reject if the metadata map is not exclusive. Without exclusivity | ||
| 606 | * the cached map->sha[] verified above can be stale: another BPF | ||
| 607 | * program with map access could have mutated the contents between | ||
| 608 | * BPF_OBJ_GET_INFO_BY_FD and loader execution. | ||
| 609 | */ | ||
| 610 | emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_IDX, | ||
| 611 | 0, 0, 0, 0)); | ||
| 612 | emit(gen, BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, SHA256_DIGEST_LENGTH)); | ||
| 613 | off = -(gen->insn_cur - gen->insn_start - gen->cleanup_label) / 8 - 2; | ||
| 614 | if (is_simm16(off)) { | ||
| 615 | emit(gen, BPF_MOV64_IMM(BPF_REG_7, -EINVAL)); | ||
| 616 | emit(gen, BPF_JMP_IMM(BPF_JNE, BPF_REG_2, 1, off)); | ||
| 617 | } else { | ||
| 618 | gen->error = -ERANGE; | ||
| 619 | } | ||
| 620 | |||
| 588 | for (i = 0; i < SHA256_DWORD_SIZE; i++) { | 621 | for (i = 0; i < SHA256_DWORD_SIZE; i++) { |
| 589 | emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_IDX, | 622 | emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_IDX, |
| 590 | 0, 0, 0, 0)); | 623 | 0, 0, 0, 0)); |
| @@ -1053,7 +1086,7 @@ void bpf_gen__prog_load(struct bpf_gen *gen, · 462 unmodified lines | |||
| 1053 | prog_idx, prog_type, insns_off, insn_cnt, license_off); | 1086 | prog_idx, prog_type, insns_off, insn_cnt, license_off); |
| 1054 | 1087 | ||
| 1055 | /* convert blob insns to target endianness */ | 1088 | /* convert blob insns to target endianness */ |
| 1056 | if (gen->swapped_endian) { | 1089 | if (gen->swapped_endian && !gen->error) { |
| 1057 | struct bpf_insn *insn = gen->data_start + insns_off; | 1090 | struct bpf_insn *insn = gen->data_start + insns_off; |
| 1058 | int i; | 1091 | int i; |
| 1059 | 1092 | ||
| @@ -1091,7 +1124,7 @@ void bpf_gen__prog_load(struct bpf_gen *gen, · 31 unmodified lines | |||
| 1091 | sizeof(struct bpf_core_relo)); | 1124 | sizeof(struct bpf_core_relo)); |
| 1092 | 1125 | ||
| 1093 | /* convert all info blobs to target endianness */ | 1126 | /* convert all info blobs to target endianness */ |
| 1094 | if (gen->swapped_endian) | 1127 | if (gen->swapped_endian && !gen->error) |
| 1095 | info_blob_bswap(gen, func_info, line_info, core_relos, load_attr); | 1128 | info_blob_bswap(gen, func_info, line_info, core_relos, load_attr); |
| 1096 | 1129 | ||
| 1097 | libbpf_strlcpy(attr.prog_name, prog_name, sizeof(attr.prog_name)); | 1130 | libbpf_strlcpy(attr.prog_name, prog_name, sizeof(attr.prog_name)); |
| @@ -1169,27 +1202,36 @@ void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *pvalue, · 71 unmodified lines | |||
| 1169 | value = add_data(gen, pvalue, value_size); | 1202 | value = add_data(gen, pvalue, value_size); |
| 1170 | key = add_data(gen, &zero, sizeof(zero)); | 1203 | key = add_data(gen, &zero, sizeof(zero)); |
| 1171 | 1204 | ||
| 1172 | /* if (map_desc[map_idx].initial_value) { | 1205 | /* |
| 1206 | * if (map_desc[map_idx].initial_value) { | ||
| 1173 | * if (ctx->flags & BPF_SKEL_KERNEL) | 1207 | * if (ctx->flags & BPF_SKEL_KERNEL) |
| 1174 | * bpf_probe_read_kernel(value, value_size, initial_value); | 1208 | * bpf_probe_read_kernel(value, value_size, initial_value); |
| 1175 | * else | 1209 | * else |
| 1176 | * bpf_copy_from_user(value, value_size, initial_value); | 1210 | * bpf_copy_from_user(value, value_size, initial_value); |
| 1177 | * } | 1211 | * } |
| 1212 | * | ||
| 1213 | * The runtime initial_value comes from the host-supplied loader | ||
| 1214 | * ctx and would overwrite the blob value after emit_signature_match() | ||
| 1215 | * has already validated map->sha[]. For a signed loader (gen_hash) | ||
| 1216 | * the attested blob value must be authoritative, so skip the override | ||
| 1217 | * and leave the hashed value in place. | ||
| 1178 | */ | 1218 | */ |
| 1179 | emit(gen, BPF_LDX_MEM(BPF_DW, BPF_REG_3, BPF_REG_6, | 1219 | if (!OPTS_GET(gen->opts, gen_hash, false)) { |
| 1180 | sizeof(struct bpf_loader_ctx) + | 1220 | emit(gen, BPF_LDX_MEM(BPF_DW, BPF_REG_3, BPF_REG_6, |
| 1181 | sizeof(struct bpf_map_desc) * map_idx + | 1221 | sizeof(struct bpf_loader_ctx) + |
| 1182 | offsetof(struct bpf_map_desc, initial_value))); | 1222 | sizeof(struct bpf_map_desc) * map_idx + |
| 1183 | emit(gen, BPF_JMP_IMM(BPF_JEQ, BPF_REG_3, 0, 8)); | 1223 | offsetof(struct bpf_map_desc, initial_value))); |
| 1184 | emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_IDX_VALUE, | 1224 | emit(gen, BPF_JMP_IMM(BPF_JEQ, BPF_REG_3, 0, 8)); |
| 1185 | 0, 0, 0, value)); | 1225 | emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_IDX_VALUE, |
| 1186 | emit(gen, BPF_MOV64_IMM(BPF_REG_2, value_size)); | 1226 | 0, 0, 0, value)); |
| 1187 | emit(gen, BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_6, | 1227 | emit(gen, BPF_MOV64_IMM(BPF_REG_2, value_size)); |
| 1188 | offsetof(struct bpf_loader_ctx, flags))); | 1228 | emit(gen, BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_6, |
| 1189 | emit(gen, BPF_JMP_IMM(BPF_JSET, BPF_REG_0, BPF_SKEL_KERNEL, 2)); | 1229 | offsetof(struct bpf_loader_ctx, flags))); |
| 1190 | emit(gen, BPF_EMIT_CALL(BPF_FUNC_copy_from_user)); | 1230 | emit(gen, BPF_JMP_IMM(BPF_JSET, BPF_REG_0, BPF_SKEL_KERNEL, 2)); |
| 1191 | emit(gen, BPF_JMP_IMM(BPF_JA, 0, 0, 1)); | 1231 | emit(gen, BPF_EMIT_CALL(BPF_FUNC_copy_from_user)); |
| 1192 | emit(gen, BPF_EMIT_CALL(BPF_FUNC_probe_read_kernel)); | 1232 | emit(gen, BPF_JMP_IMM(BPF_JA, 0, 0, 1)); |
| 1233 | emit(gen, BPF_EMIT_CALL(BPF_FUNC_probe_read_kernel)); | ||
| 1234 | } | ||
| 1193 | 1235 | ||
| 1194 | map_update_attr = add_data(gen, &attr, attr_size); | 1236 | map_update_attr = add_data(gen, &attr, attr_size); |
| 1195 | pr_debug("gen: map_update_elem: idx %d, value: off %d size %d, attr: off %d size %d\n", | 1237 | pr_debug("gen: map_update_elem: idx %d, value: off %d size %d, attr: off %d size %d\n", |
| @@ -136,6 +136,9 @@ static const char * const attach_type_name[] = { · 135 unmodified lines | |||
| 136 | [BPF_NETKIT_PEER] = "netkit_peer", | 136 | [BPF_NETKIT_PEER] = "netkit_peer", |
| 137 | [BPF_TRACE_KPROBE_SESSION] = "trace_kprobe_session", | 137 | [BPF_TRACE_KPROBE_SESSION] = "trace_kprobe_session", |
| 138 | [BPF_TRACE_UPROBE_SESSION] = "trace_uprobe_session", | 138 | [BPF_TRACE_UPROBE_SESSION] = "trace_uprobe_session", |
| 139 | [BPF_TRACE_FENTRY_MULTI] = "trace_fentry_multi", | ||
| 140 | [BPF_TRACE_FEXIT_MULTI] = "trace_fexit_multi", | ||
| 141 | [BPF_TRACE_FSESSION_MULTI] = "trace_fsession_multi", | ||
| 139 | }; | 142 | }; |
| 140 | 143 | ||
| 141 | static const char * const link_type_name[] = { | 144 | static const char * const link_type_name[] = { |
| @@ -154,6 +157,7 @@ static const char * const link_type_name[] = { · 12 unmodified lines | |||
| 154 | [BPF_LINK_TYPE_UPROBE_MULTI] = "uprobe_multi", | 157 | [BPF_LINK_TYPE_UPROBE_MULTI] = "uprobe_multi", |
| 155 | [BPF_LINK_TYPE_NETKIT] = "netkit", | 158 | [BPF_LINK_TYPE_NETKIT] = "netkit", |
| 156 | [BPF_LINK_TYPE_SOCKMAP] = "sockmap", | 159 | [BPF_LINK_TYPE_SOCKMAP] = "sockmap", |
| 160 | [BPF_LINK_TYPE_TRACING_MULTI] = "tracing_multi", | ||
| 157 | }; | 161 | }; |
| 158 | 162 | ||
| 159 | static const char * const map_type_name[] = { | 163 | static const char * const map_type_name[] = { |
| @@ -192,6 +196,7 @@ static const char * const map_type_name[] = { · 32 unmodified lines | |||
| 192 | [BPF_MAP_TYPE_CGRP_STORAGE] = "cgrp_storage", | 196 | [BPF_MAP_TYPE_CGRP_STORAGE] = "cgrp_storage", |
| 193 | [BPF_MAP_TYPE_ARENA] = "arena", | 197 | [BPF_MAP_TYPE_ARENA] = "arena", |
| 194 | [BPF_MAP_TYPE_INSN_ARRAY] = "insn_array", | 198 | [BPF_MAP_TYPE_INSN_ARRAY] = "insn_array", |
| 199 | [BPF_MAP_TYPE_RHASH] = "rhash", | ||
| 195 | }; | 200 | }; |
| 196 | 201 | ||
| 197 | static const char * const prog_type_name[] = { | 202 | static const char * const prog_type_name[] = { |
| @@ -7767,6 +7772,69 @@ static int bpf_object__sanitize_prog(struct bpf_object *obj, struct bpf_program · 7569 unmodified lines | |||
| 7767 | static int libbpf_find_attach_btf_id(struct bpf_program *prog, const char *attach_name, | 7772 | static int libbpf_find_attach_btf_id(struct bpf_program *prog, const char *attach_name, |
| 7768 | int *btf_obj_fd, int *btf_type_id); | 7773 | int *btf_obj_fd, int *btf_type_id); |
| 7769 | 7774 | ||
| 7775 | static inline bool is_tracing_multi(enum bpf_attach_type type) | ||
| 7776 | { | ||
| 7777 | return type == BPF_TRACE_FENTRY_MULTI || type == BPF_TRACE_FEXIT_MULTI || | ||
| 7778 | type == BPF_TRACE_FSESSION_MULTI; | ||
| 7779 | } | ||
| 7780 | |||
| 7781 | static const struct module_btf *find_attach_module(struct bpf_object *obj, const char *attach) | ||
| 7782 | { | ||
| 7783 | const char *sep, *mod_name = NULL; | ||
| 7784 | int i, mod_len, err; | ||
| 7785 | |||
| 7786 | /* | ||
| 7787 | * We expect attach string in the form of either | ||
| 7788 | * - function_pattern or | ||
| 7789 | * - <module>:function_pattern | ||
| 7790 | */ | ||
| 7791 | sep = strchr(attach, ':'); | ||
| 7792 | if (sep) { | ||
| 7793 | mod_name = attach; | ||
| 7794 | mod_len = sep - mod_name; | ||
| 7795 | } | ||
| 7796 | if (!mod_name) | ||
| 7797 | return NULL; | ||
| 7798 | |||
| 7799 | err = load_module_btfs(obj); | ||
| 7800 | if (err) | ||
| 7801 | return NULL; | ||
| 7802 | |||
| 7803 | for (i = 0; i < obj->btf_module_cnt; i++) { | ||
| 7804 | const struct module_btf *mod = &obj->btf_modules[i]; | ||
| 7805 | |||
| 7806 | if (strncmp(mod->name, mod_name, mod_len) == 0 && mod->name[mod_len] == '\0') | ||
| 7807 | return mod; | ||
| 7808 | } | ||
| 7809 | return NULL; | ||
| 7810 | } | ||
| 7811 | |||
| 7812 | static int tracing_multi_mod_fd(struct bpf_program *prog, int *btf_obj_fd) | ||
| 7813 | { | ||
| 7814 | const char *attach_name, *sep; | ||
| 7815 | const struct module_btf *mod; | ||
| 7816 | |||
| 7817 | *btf_obj_fd = 0; | ||
| 7818 | attach_name = strchr(prog->sec_name, '/'); | ||
| 7819 | |||
| 7820 | /* Program with no details in spec, using kernel btf. */ | ||
| 7821 | if (!attach_name) | ||
| 7822 | return 0; | ||
| 7823 | |||
| 7824 | /* Program with no module section, using kernel btf. */ | ||
| 7825 | sep = strchr(++attach_name, ':'); | ||
| 7826 | if (!sep) | ||
| 7827 | return 0; | ||
| 7828 | |||
| 7829 | /* Program with module specified, get its btf fd. */ | ||
| 7830 | mod = find_attach_module(prog->obj, attach_name); | ||
| 7831 | if (!mod) | ||
| 7832 | return -EINVAL; | ||
| 7833 | |||
| 7834 | *btf_obj_fd = mod->fd; | ||
| 7835 | return 0; | ||
| 7836 | } | ||
| 7837 | |||
| 7770 | /* this is called as prog->sec_def->prog_prepare_load_fn for libbpf-supported sec_defs */ | 7838 | /* this is called as prog->sec_def->prog_prepare_load_fn for libbpf-supported sec_defs */ |
| 7771 | static int libbpf_prepare_prog_load(struct bpf_program *prog, | 7839 | static int libbpf_prepare_prog_load(struct bpf_program *prog, |
| 7772 | struct bpf_prog_load_opts *opts, long cookie) | 7840 | struct bpf_prog_load_opts *opts, long cookie) |
| @@ -7830,6 +7898,18 @@ static int libbpf_prepare_prog_load(struct bpf_program *prog, · 57 unmodified lines | |||
| 7830 | opts->attach_btf_obj_fd = btf_obj_fd; | 7898 | opts->attach_btf_obj_fd = btf_obj_fd; |
| 7831 | opts->attach_btf_id = btf_type_id; | 7899 | opts->attach_btf_id = btf_type_id; |
| 7832 | } | 7900 | } |
| 7901 | |||
| 7902 | if (is_tracing_multi(prog->expected_attach_type)) { | ||
| 7903 | int err, btf_obj_fd = 0; | ||
| 7904 | |||
| 7905 | err = tracing_multi_mod_fd(prog, &btf_obj_fd); | ||
| 7906 | if (err < 0) | ||
| 7907 | return err; | ||
| 7908 | |||
| 7909 | prog->attach_btf_obj_fd = btf_obj_fd; | ||
| 7910 | opts->attach_btf_obj_fd = btf_obj_fd; | ||
| 7911 | } | ||
| 7912 | |||
| 7833 | return 0; | 7913 | return 0; |
| 7834 | } | 7914 | } |
| 7835 | 7915 | ||
| @@ -8936,13 +9016,10 @@ static void bpf_object_unpin(struct bpf_object *obj) · 1100 unmodified lines | |||
| 8936 | bpf_map__unpin(&obj->maps[i], NULL); | 9016 | bpf_map__unpin(&obj->maps[i], NULL); |
| 8937 | } | 9017 | } |
| 8938 | 9018 | ||
| 8939 | static void bpf_object_post_load_cleanup(struct bpf_object *obj) | 9019 | static void bpf_object_cleanup_btf(struct bpf_object *obj) |
| 8940 | { | 9020 | { |
| 8941 | int i; | 9021 | int i; |
| 8942 | 9022 | ||
| 8943 | /* clean up fd_array */ | ||
| 8944 | zfree(&obj->fd_array); | ||
| 8945 | |||
| 8946 | /* clean up module BTFs */ | 9023 | /* clean up module BTFs */ |
| 8947 | for (i = 0; i < obj->btf_module_cnt; i++) { | 9024 | for (i = 0; i < obj->btf_module_cnt; i++) { |
| 8948 | close(obj->btf_modules[i].fd); | 9025 | close(obj->btf_modules[i].fd); |
| @@ -8950,6 +9027,8 @@ static void bpf_object_post_load_cleanup(struct bpf_object *obj) · 1 unmodified line | |||
| 8950 | free(obj->btf_modules[i].name); | 9027 | free(obj->btf_modules[i].name); |
| 8951 | } | 9028 | } |
| 8952 | obj->btf_module_cnt = 0; | 9029 | obj->btf_module_cnt = 0; |
| 9030 | obj->btf_module_cap = 0; | ||
| 9031 | obj->btf_modules_loaded = false; | ||
| 8953 | zfree(&obj->btf_modules); | 9032 | zfree(&obj->btf_modules); |
| 8954 | 9033 | ||
| 8955 | /* clean up vmlinux BTF */ | 9034 | /* clean up vmlinux BTF */ |
| @@ -8957,6 +9036,15 @@ static void bpf_object_post_load_cleanup(struct bpf_object *obj) · 1 unmodified line | |||
| 8957 | obj->btf_vmlinux = NULL; | 9036 | obj->btf_vmlinux = NULL; |
| 8958 | } | 9037 | } |
| 8959 | 9038 | ||
| 9039 | static void bpf_object_post_load_cleanup(struct bpf_object *obj) | ||
| 9040 | { | ||
| 9041 | /* clean up fd_array */ | ||
| 9042 | zfree(&obj->fd_array); | ||
| 9043 | |||
| 9044 | /* clean up BTF */ | ||
| 9045 | bpf_object_cleanup_btf(obj); | ||
| 9046 | } | ||
| 9047 | |||
| 8960 | static int bpf_object_prepare(struct bpf_object *obj, const char *target_btf_path) | 9048 | static int bpf_object_prepare(struct bpf_object *obj, const char *target_btf_path) |
| 8961 | { | 9049 | { |
| 8962 | int err; | 9050 | int err; |
| @@ -9983,6 +10071,7 @@ static int attach_kprobe_session(const struct bpf_program *prog, long cookie, st · 1020 unmodified lines | |||
| 9983 | static int attach_uprobe_multi(const struct bpf_program *prog, long cookie, struct bpf_link **link); | 10071 | static int attach_uprobe_multi(const struct bpf_program *prog, long cookie, struct bpf_link **link); |
| 9984 | static int attach_lsm(const struct bpf_program *prog, long cookie, struct bpf_link **link); | 10072 | static int attach_lsm(const struct bpf_program *prog, long cookie, struct bpf_link **link); |
| 9985 | static int attach_iter(const struct bpf_program *prog, long cookie, struct bpf_link **link); | 10073 | static int attach_iter(const struct bpf_program *prog, long cookie, struct bpf_link **link); |
| 10074 | static int attach_tracing_multi(const struct bpf_program *prog, long cookie, struct bpf_link **link); | ||
| 9986 | 10075 | ||
| 9987 | static const struct bpf_sec_def section_defs[] = { | 10076 | static const struct bpf_sec_def section_defs[] = { |
| 9988 | SEC_DEF("socket", SOCKET_FILTER, 0, SEC_NONE), | 10077 | SEC_DEF("socket", SOCKET_FILTER, 0, SEC_NONE), |
| @@ -10018,11 +10107,16 @@ static const struct bpf_sec_def section_defs[] = { · 29 unmodified lines | |||
| 10018 | SEC_DEF("netkit/peer", SCHED_CLS, BPF_NETKIT_PEER, SEC_NONE), | 10107 | SEC_DEF("netkit/peer", SCHED_CLS, BPF_NETKIT_PEER, SEC_NONE), |
| 10019 | SEC_DEF("tracepoint+", TRACEPOINT, 0, SEC_NONE, attach_tp), | 10108 | SEC_DEF("tracepoint+", TRACEPOINT, 0, SEC_NONE, attach_tp), |
| 10020 | SEC_DEF("tp+", TRACEPOINT, 0, SEC_NONE, attach_tp), | 10109 | SEC_DEF("tp+", TRACEPOINT, 0, SEC_NONE, attach_tp), |
| 10110 | SEC_DEF("tracepoint.s+", TRACEPOINT, 0, SEC_SLEEPABLE, attach_tp), | ||
| 10111 | SEC_DEF("tp.s+", TRACEPOINT, 0, SEC_SLEEPABLE, attach_tp), | ||
| 10021 | SEC_DEF("raw_tracepoint+", RAW_TRACEPOINT, 0, SEC_NONE, attach_raw_tp), | 10112 | SEC_DEF("raw_tracepoint+", RAW_TRACEPOINT, 0, SEC_NONE, attach_raw_tp), |
| 10022 | SEC_DEF("raw_tp+", RAW_TRACEPOINT, 0, SEC_NONE, attach_raw_tp), | 10113 | SEC_DEF("raw_tp+", RAW_TRACEPOINT, 0, SEC_NONE, attach_raw_tp), |
| 10114 | SEC_DEF("raw_tracepoint.s+", RAW_TRACEPOINT, 0, SEC_SLEEPABLE, attach_raw_tp), | ||
| 10115 | SEC_DEF("raw_tp.s+", RAW_TRACEPOINT, 0, SEC_SLEEPABLE, attach_raw_tp), | ||
| 10023 | SEC_DEF("raw_tracepoint.w+", RAW_TRACEPOINT_WRITABLE, 0, SEC_NONE, attach_raw_tp), | 10116 | SEC_DEF("raw_tracepoint.w+", RAW_TRACEPOINT_WRITABLE, 0, SEC_NONE, attach_raw_tp), |
| 10024 | SEC_DEF("raw_tp.w+", RAW_TRACEPOINT_WRITABLE, 0, SEC_NONE, attach_raw_tp), | 10117 | SEC_DEF("raw_tp.w+", RAW_TRACEPOINT_WRITABLE, 0, SEC_NONE, attach_raw_tp), |
| 10025 | SEC_DEF("tp_btf+", TRACING, BPF_TRACE_RAW_TP, SEC_ATTACH_BTF, attach_trace), | 10118 | SEC_DEF("tp_btf+", TRACING, BPF_TRACE_RAW_TP, SEC_ATTACH_BTF, attach_trace), |
| 10119 | SEC_DEF("tp_btf.s+", TRACING, BPF_TRACE_RAW_TP, SEC_ATTACH_BTF | SEC_SLEEPABLE, attach_trace), | ||
| 10026 | SEC_DEF("fentry+", TRACING, BPF_TRACE_FENTRY, SEC_ATTACH_BTF, attach_trace), | 10120 | SEC_DEF("fentry+", TRACING, BPF_TRACE_FENTRY, SEC_ATTACH_BTF, attach_trace), |
| 10027 | SEC_DEF("fmod_ret+", TRACING, BPF_MODIFY_RETURN, SEC_ATTACH_BTF, attach_trace), | 10121 | SEC_DEF("fmod_ret+", TRACING, BPF_MODIFY_RETURN, SEC_ATTACH_BTF, attach_trace), |
| 10028 | SEC_DEF("fexit+", TRACING, BPF_TRACE_FEXIT, SEC_ATTACH_BTF, attach_trace), | 10122 | SEC_DEF("fexit+", TRACING, BPF_TRACE_FEXIT, SEC_ATTACH_BTF, attach_trace), |
| @@ -10031,6 +10125,12 @@ static const struct bpf_sec_def section_defs[] = { · 2 unmodified lines | |||
| 10031 | SEC_DEF("fexit.s+", TRACING, BPF_TRACE_FEXIT, SEC_ATTACH_BTF | SEC_SLEEPABLE, attach_trace), | 10125 | SEC_DEF("fexit.s+", TRACING, BPF_TRACE_FEXIT, SEC_ATTACH_BTF | SEC_SLEEPABLE, attach_trace), |
| 10032 | SEC_DEF("fsession+", TRACING, BPF_TRACE_FSESSION, SEC_ATTACH_BTF, attach_trace), | 10126 | SEC_DEF("fsession+", TRACING, BPF_TRACE_FSESSION, SEC_ATTACH_BTF, attach_trace), |
| 10033 | SEC_DEF("fsession.s+", TRACING, BPF_TRACE_FSESSION, SEC_ATTACH_BTF | SEC_SLEEPABLE, attach_trace), | 10127 | SEC_DEF("fsession.s+", TRACING, BPF_TRACE_FSESSION, SEC_ATTACH_BTF | SEC_SLEEPABLE, attach_trace), |
| 10128 | SEC_DEF("fsession.multi+", TRACING, BPF_TRACE_FSESSION_MULTI, 0, attach_tracing_multi), | ||
| 10129 | SEC_DEF("fsession.multi.s+", TRACING, BPF_TRACE_FSESSION_MULTI, SEC_SLEEPABLE, attach_tracing_multi), | ||
| 10130 | SEC_DEF("fentry.multi+", TRACING, BPF_TRACE_FENTRY_MULTI, 0, attach_tracing_multi), | ||
| 10131 | SEC_DEF("fexit.multi+", TRACING, BPF_TRACE_FEXIT_MULTI, 0, attach_tracing_multi), | ||
| 10132 | SEC_DEF("fentry.multi.s+", TRACING, BPF_TRACE_FENTRY_MULTI, SEC_SLEEPABLE, attach_tracing_multi), | ||
| 10133 | SEC_DEF("fexit.multi.s+", TRACING, BPF_TRACE_FEXIT_MULTI, SEC_SLEEPABLE, attach_tracing_multi), | ||
| 10034 | SEC_DEF("freplace+", EXT, 0, SEC_ATTACH_BTF, attach_trace), | 10134 | SEC_DEF("freplace+", EXT, 0, SEC_ATTACH_BTF, attach_trace), |
| 10035 | SEC_DEF("lsm+", LSM, BPF_LSM_MAC, SEC_ATTACH_BTF, attach_lsm), | 10135 | SEC_DEF("lsm+", LSM, BPF_LSM_MAC, SEC_ATTACH_BTF, attach_lsm), |
| 10036 | SEC_DEF("lsm.s+", LSM, BPF_LSM_MAC, SEC_ATTACH_BTF | SEC_SLEEPABLE, attach_lsm), | 10136 | SEC_DEF("lsm.s+", LSM, BPF_LSM_MAC, SEC_ATTACH_BTF | SEC_SLEEPABLE, attach_lsm), |
| @@ -12280,7 +12380,7 @@ error: · 2243 unmodified lines | |||
| 12280 | static int attach_kprobe(const struct bpf_program *prog, long cookie, struct bpf_link **link) | 12380 | static int attach_kprobe(const struct bpf_program *prog, long cookie, struct bpf_link **link) |
| 12281 | { | 12381 | { |
| 12282 | DECLARE_LIBBPF_OPTS(bpf_kprobe_opts, opts); | 12382 | DECLARE_LIBBPF_OPTS(bpf_kprobe_opts, opts); |
| 12283 | unsigned long offset = 0; | 12383 | long offset = 0; |
| 12284 | const char *func_name; | 12384 | const char *func_name; |
| 12285 | char *func; | 12385 | char *func; |
| 12286 | int n; | 12386 | int n; |
| @@ -12302,6 +12402,13 @@ static int attach_kprobe(const struct bpf_program *prog, long cookie, struct bpf · 15 unmodified lines | |||
| 12302 | pr_warn("kprobe name is invalid: %s\n", func_name); | 12402 | pr_warn("kprobe name is invalid: %s\n", func_name); |
| 12303 | return -EINVAL; | 12403 | return -EINVAL; |
| 12304 | } | 12404 | } |
| 12405 | |||
| 12406 | if (offset < 0) { | ||
| 12407 | free(func); | ||
| 12408 | pr_warn("kprobe offset must be a non-negative integer: %li\n", offset); | ||
| 12409 | return -EINVAL; | ||
| 12410 | } | ||
| 12411 | |||
| 12305 | if (opts.retprobe && offset != 0) { | 12412 | if (opts.retprobe && offset != 0) { |
| 12306 | free(func); | 12413 | free(func); |
| 12307 | pr_warn("kretprobes do not support offset specification\n"); | 12414 | pr_warn("kretprobes do not support offset specification\n"); |
| @@ -12425,6 +12532,279 @@ static int attach_uprobe_multi(const struct bpf_program *prog, long cookie, stru · 117 unmodified lines | |||
| 12425 | return ret; | 12532 | return ret; |
| 12426 | } | 12533 | } |
| 12427 | 12534 | ||
| 12535 | #define MAX_BPF_FUNC_ARGS 12 | ||
| 12536 | |||
| 12537 | static bool btf_type_is_modifier(const struct btf_type *t) | ||
| 12538 | { | ||
| 12539 | switch (BTF_INFO_KIND(t->info)) { | ||
| 12540 | case BTF_KIND_TYPEDEF: | ||
| 12541 | case BTF_KIND_VOLATILE: | ||
| 12542 | case BTF_KIND_CONST: | ||
| 12543 | case BTF_KIND_RESTRICT: | ||
| 12544 | case BTF_KIND_TYPE_TAG: | ||
| 12545 | return true; | ||
| 12546 | default: | ||
| 12547 | return false; | ||
| 12548 | } | ||
| 12549 | } | ||
| 12550 | |||
| 12551 | #define MAX_RESOLVE_DEPTH 32 | ||
| 12552 | |||
| 12553 | static int btf_get_type_size(const struct btf *btf, __u32 type_id, | ||
| 12554 | const struct btf_type **ret_type) | ||
| 12555 | { | ||
| 12556 | const struct btf_type *t; | ||
| 12557 | int i; | ||
| 12558 | |||
| 12559 | *ret_type = btf__type_by_id(btf, 0); | ||
| 12560 | if (!type_id) | ||
| 12561 | return 0; | ||
| 12562 | t = btf__type_by_id(btf, type_id); | ||
| 12563 | for (i = 0; i < MAX_RESOLVE_DEPTH && t && btf_type_is_modifier(t); i++) | ||
| 12564 | t = btf__type_by_id(btf, t->type); | ||
| 12565 | if (!t || i == MAX_RESOLVE_DEPTH) | ||
| 12566 | return -EINVAL; | ||
| 12567 | *ret_type = t; | ||
| 12568 | if (btf_is_ptr(t)) | ||
| 12569 | return btf__pointer_size(btf); | ||
| 12570 | if (btf_is_int(t) || btf_is_any_enum(t) || btf_is_struct(t) || btf_is_union(t)) | ||
| 12571 | return t->size; | ||
| 12572 | return -EINVAL; | ||
| 12573 | } | ||
| 12574 | |||
| 12575 | bool btf_type_is_traceable_func(const struct btf *btf, const struct btf_type *t) | ||
| 12576 | { | ||
| 12577 | const struct btf_param *args; | ||
| 12578 | const struct btf_type *proto; | ||
| 12579 | __u32 i, nargs; | ||
| 12580 | int ret; | ||
| 12581 | |||
| 12582 | if (!btf_is_func(t)) | ||
| 12583 | return false; | ||
| 12584 | proto = btf__type_by_id(btf, t->type); | ||
| 12585 | if (!proto || !btf_is_func_proto(proto)) | ||
| 12586 | return false; | ||
| 12587 | |||
| 12588 | args = (const struct btf_param *)(proto + 1); | ||
| 12589 | nargs = btf_vlen(proto); | ||
| 12590 | if (nargs > MAX_BPF_FUNC_ARGS) | ||
| 12591 | return false; | ||
| 12592 | |||
| 12593 | /* No support for struct return type. */ | ||
| 12594 | ret = btf_get_type_size(btf, proto->type, &t); | ||
| 12595 | if (ret < 0 || btf_is_struct(t) || btf_is_union(t)) | ||
| 12596 | return false; | ||
| 12597 | |||
| 12598 | for (i = 0; i < nargs; i++) { | ||
| 12599 | /* No support for variable args. */ | ||
| 12600 | if (i == nargs - 1 && args[i].type == 0) | ||
| 12601 | return false; | ||
| 12602 | ret = btf_get_type_size(btf, args[i].type, &t); | ||
| 12603 | /* No support of struct argument size greater than 16 bytes. */ | ||
| 12604 | if (ret < 0 || ret > 16) | ||
| 12605 | return false; | ||
| 12606 | /* No support for void argument. */ | ||
| 12607 | if (ret == 0) | ||
| 12608 | return false; | ||
| 12609 | } | ||
| 12610 | |||
| 12611 | return true; | ||
| 12612 | } | ||
| 12613 | |||
| 12614 | static int | ||
| 12615 | collect_btf_func_ids_by_glob(const struct btf *btf, const char *pattern, __u32 **ids) | ||
| 12616 | { | ||
| 12617 | __u32 type_id, nr_types = btf__type_cnt(btf); | ||
| 12618 | size_t cap = 0, cnt = 0; | ||
| 12619 | |||
| 12620 | if (!pattern) | ||
| 12621 | return -EINVAL; | ||
| 12622 | |||
| 12623 | for (type_id = 1; type_id < nr_types; type_id++) { | ||
| 12624 | const struct btf_type *t = btf__type_by_id(btf, type_id); | ||
| 12625 | const char *name; | ||
| 12626 | int err; | ||
| 12627 | |||
| 12628 | if (btf_kind(t) != BTF_KIND_FUNC) | ||
| 12629 | continue; | ||
| 12630 | name = btf__name_by_offset(btf, t->name_off); | ||
| 12631 | if (!name) | ||
| 12632 | continue; | ||
| 12633 | |||
| 12634 | if (!glob_match(name, pattern)) | ||
| 12635 | continue; | ||
| 12636 | if (!btf_type_is_traceable_func(btf, t)) | ||
| 12637 | continue; | ||
| 12638 | |||
| 12639 | err = libbpf_ensure_mem((void **) ids, &cap, sizeof(**ids), cnt + 1); | ||
| 12640 | if (err) { | ||
| 12641 | free(*ids); | ||
| 12642 | return -ENOMEM; | ||
| 12643 | } | ||
| 12644 | (*ids)[cnt++] = type_id; | ||
| 12645 | } | ||
| 12646 | |||
| 12647 | return cnt; | ||
| 12648 | } | ||
| 12649 | |||
| 12650 | static int collect_func_ids_by_glob(const struct bpf_program *prog, const char *pattern, __u32 **ids) | ||
| 12651 | { | ||
| 12652 | struct bpf_object *obj = prog->obj; | ||
| 12653 | const struct module_btf *mod; | ||
| 12654 | struct btf *btf = NULL; | ||
| 12655 | const char *sep; | ||
| 12656 | int err; | ||
| 12657 | |||
| 12658 | err = bpf_object__load_vmlinux_btf(obj, true); | ||
| 12659 | if (err) | ||
| 12660 | return err; | ||
| 12661 | |||
| 12662 | /* In case we have module specified, we will find its btf and use that. */ | ||
| 12663 | sep = strchr(pattern, ':'); | ||
| 12664 | if (sep) { | ||
| 12665 | mod = find_attach_module(obj, pattern); | ||
| 12666 | if (!mod) { | ||
| 12667 | err = -EINVAL; | ||
| 12668 | goto cleanup; | ||
| 12669 | } | ||
| 12670 | btf = mod->btf; | ||
| 12671 | pattern = sep + 1; | ||
| 12672 | } else { | ||
| 12673 | /* Program is loaded for kernel module. */ | ||
| 12674 | if (prog->attach_btf_obj_fd) { | ||
| 12675 | err = -EINVAL; | ||
| 12676 | goto cleanup; | ||
| 12677 | } | ||
| 12678 | btf = obj->btf_vmlinux; | ||
| 12679 | } | ||
| 12680 | |||
| 12681 | err = collect_btf_func_ids_by_glob(btf, pattern, ids); | ||
| 12682 | |||
| 12683 | cleanup: | ||
| 12684 | bpf_object_cleanup_btf(obj); | ||
| 12685 | return err; | ||
| 12686 | } | ||
| 12687 | |||
| 12688 | struct bpf_link * | ||
| 12689 | bpf_program__attach_tracing_multi(const struct bpf_program *prog, const char *pattern, | ||
| 12690 | const struct bpf_tracing_multi_opts *opts) | ||
| 12691 | { | ||
| 12692 | LIBBPF_OPTS(bpf_link_create_opts, lopts); | ||
| 12693 | int prog_fd, link_fd, err, cnt; | ||
| 12694 | __u32 *free_ids = NULL; | ||
| 12695 | struct bpf_link *link; | ||
| 12696 | const __u64 *cookies; | ||
| 12697 | const __u32 *ids; | ||
| 12698 | |||
| 12699 | if (!OPTS_VALID(opts, bpf_tracing_multi_opts)) | ||
| 12700 | return libbpf_err_ptr(-EINVAL); | ||
| 12701 | |||
| 12702 | prog_fd = bpf_program__fd(prog); | ||
| 12703 | if (prog_fd < 0) { | ||
| 12704 | pr_warn("prog '%s': can't attach BPF program without FD (was it loaded?)\n", | ||
| 12705 | prog->name); | ||
| 12706 | return libbpf_err_ptr(-EINVAL); | ||
| 12707 | } | ||
| 12708 | |||
| 12709 | cnt = OPTS_GET(opts, cnt, 0); | ||
| 12710 | ids = OPTS_GET(opts, ids, NULL); | ||
| 12711 | cookies = OPTS_GET(opts, cookies, NULL); | ||
| 12712 | |||
| 12713 | if (!!ids != !!cnt) | ||
| 12714 | return libbpf_err_ptr(-EINVAL); | ||
| 12715 | if (pattern && (ids || cookies)) | ||
| 12716 | return libbpf_err_ptr(-EINVAL); | ||
| 12717 | if (!pattern && !ids) | ||
| 12718 | return libbpf_err_ptr(-EINVAL); | ||
| 12719 | |||
| 12720 | if (pattern) { | ||
| 12721 | cnt = collect_func_ids_by_glob(prog, pattern, &free_ids); | ||
| 12722 | if (cnt < 0) | ||
| 12723 | return libbpf_err_ptr(cnt); | ||
| 12724 | if (cnt == 0) | ||
| 12725 | return libbpf_err_ptr(-EINVAL); | ||
| 12726 | ids = (const __u32 *) free_ids; | ||
| 12727 | } | ||
| 12728 | |||
| 12729 | lopts.tracing_multi.ids = ids; | ||
| 12730 | lopts.tracing_multi.cookies = cookies; | ||
| 12731 | lopts.tracing_multi.cnt = cnt; | ||
| 12732 | |||
| 12733 | link = calloc(1, sizeof(*link)); | ||
| 12734 | if (!link) { | ||
| 12735 | err = -ENOMEM; | ||
| 12736 | goto error; | ||
| 12737 | } | ||
| 12738 | link->detach = &bpf_link__detach_fd; | ||
| 12739 | |||
| 12740 | link_fd = bpf_link_create(prog_fd, 0, prog->expected_attach_type, &lopts); | ||
| 12741 | if (link_fd < 0) { | ||
| 12742 | err = -errno; | ||
| 12743 | pr_warn("prog '%s': failed to attach: %s\n", prog->name, errstr(err)); | ||
| 12744 | goto error; | ||
| 12745 | } | ||
| 12746 | link->fd = link_fd; | ||
| 12747 | free(free_ids); | ||
| 12748 | return link; | ||
| 12749 | |||
| 12750 | error: | ||
| 12751 | free(link); | ||
| 12752 | free(free_ids); | ||
| 12753 | return libbpf_err_ptr(err); | ||
| 12754 | } | ||
| 12755 | |||
| 12756 | static int attach_tracing_multi(const struct bpf_program *prog, long cookie, struct bpf_link **link) | ||
| 12757 | { | ||
| 12758 | static const char *const prefixes[] = { | ||
| 12759 | "fentry.multi", | ||
| 12760 | "fexit.multi", | ||
| 12761 | "fsession.multi", | ||
| 12762 | "fentry.multi.s", | ||
| 12763 | "fexit.multi.s", | ||
| 12764 | "fsession.multi.s", | ||
| 12765 | }; | ||
| 12766 | const char *spec = NULL; | ||
| 12767 | char *pattern; | ||
| 12768 | size_t i; | ||
| 12769 | int n; | ||
| 12770 | |||
| 12771 | *link = NULL; | ||
| 12772 | |||
| 12773 | for (i = 0; i < ARRAY_SIZE(prefixes); i++) { | ||
| 12774 | size_t pfx_len; | ||
| 12775 | |||
| 12776 | if (!str_has_pfx(prog->sec_name, prefixes[i])) | ||
| 12777 | continue; | ||
| 12778 | |||
| 12779 | pfx_len = strlen(prefixes[i]); | ||
| 12780 | /* no auto-attach case of, e.g., SEC("fentry.multi") */ | ||
| 12781 | if (prog->sec_name[pfx_len] == '\0') | ||
| 12782 | return 0; | ||
| 12783 | |||
| 12784 | if (prog->sec_name[pfx_len] != '/') | ||
| 12785 | continue; | ||
| 12786 | |||
| 12787 | spec = prog->sec_name + pfx_len + 1; | ||
| 12788 | break; | ||
| 12789 | } | ||
| 12790 | |||
| 12791 | if (!spec) { | ||
| 12792 | pr_warn("prog '%s': invalid section name '%s'\n", | ||
| 12793 | prog->name, prog->sec_name); | ||
| 12794 | return -EINVAL; | ||
| 12795 | } | ||
| 12796 | |||
| 12797 | n = sscanf(spec, "%m[a-zA-Z0-9_.*?:]", &pattern); | ||
| 12798 | if (n < 1) { | ||
| 12799 | pr_warn("tracing multi pattern is invalid: %s\n", spec); | ||
| 12800 | return -EINVAL; | ||
| 12801 | } | ||
| 12802 | |||
| 12803 | *link = bpf_program__attach_tracing_multi(prog, pattern, NULL); | ||
| 12804 | free(pattern); | ||
| 12805 | return libbpf_get_error(*link); | ||
| 12806 | } | ||
| 12807 | |||
| 12428 | static inline int add_uprobe_event_legacy(const char *probe_name, bool retprobe, | 12808 | static inline int add_uprobe_event_legacy(const char *probe_name, bool retprobe, |
| 12429 | const char *binary_path, size_t offset) | 12809 | const char *binary_path, size_t offset) |
| 12430 | { | 12810 | { |
| @@ -13145,25 +13525,61 @@ struct bpf_link *bpf_program__attach_tracepoint(const struct bpf_program *prog, · 714 unmodified lines | |||
| 13145 | return bpf_program__attach_tracepoint_opts(prog, tp_category, tp_name, NULL); | 13525 | return bpf_program__attach_tracepoint_opts(prog, tp_category, tp_name, NULL); |
| 13146 | } | 13526 | } |
| 13147 | 13527 | ||
| 13528 | /* | ||
| 13529 | * Match section name against a prefix array. Returns pointer past | ||
| 13530 | * "prefix/" on match, empty string for bare sections (exact prefix | ||
| 13531 | * match), or NULL if no prefix matches. | ||
| 13532 | */ | ||
| 13533 | static const char *sec_name_match_prefix(const char *sec_name, | ||
| 13534 | const char *const *prefixes, | ||
| 13535 | size_t n) | ||
| 13536 | { | ||
| 13537 | size_t i; | ||
| 13538 | |||
| 13539 | for (i = 0; i < n; i++) { | ||
| 13540 | size_t pfx_len; | ||
| 13541 | |||
| 13542 | if (!str_has_pfx(sec_name, prefixes[i])) | ||
| 13543 | continue; | ||
| 13544 | |||
| 13545 | pfx_len = strlen(prefixes[i]); | ||
| 13546 | if (sec_name[pfx_len] == '\0') | ||
| 13547 | return sec_name + pfx_len; | ||
| 13548 | |||
| 13549 | if (sec_name[pfx_len] != '/' || sec_name[pfx_len + 1] == '\0') | ||
| 13550 | continue; | ||
| 13551 | |||
| 13552 | return sec_name + pfx_len + 1; | ||
| 13553 | } | ||
| 13554 | return NULL; | ||
| 13555 | } | ||
| 13556 | |||
| 13148 | static int attach_tp(const struct bpf_program *prog, long cookie, struct bpf_link **link) | 13557 | static int attach_tp(const struct bpf_program *prog, long cookie, struct bpf_link **link) |
| 13149 | { | 13558 | { |
| 13559 | static const char *const prefixes[] = { | ||
| 13560 | "tp.s", | ||
| 13561 | "tp", | ||
| 13562 | "tracepoint.s", | ||
| 13563 | "tracepoint", | ||
| 13564 | }; | ||
| 13150 | char *sec_name, *tp_cat, *tp_name; | 13565 | char *sec_name, *tp_cat, *tp_name; |
| 13566 | const char *match; | ||
| 13151 | 13567 | ||
| 13152 | *link = NULL; | 13568 | *link = NULL; |
| 13153 | 13569 | ||
| 13154 | /* no auto-attach for SEC("tp") or SEC("tracepoint") */ | 13570 | match = sec_name_match_prefix(prog->sec_name, prefixes, ARRAY_SIZE(prefixes)); |
| 13155 | if (strcmp(prog->sec_name, "tp") == 0 || strcmp(prog->sec_name, "tracepoint") == 0) | 13571 | if (!match) { |
| 13572 | pr_warn("prog '%s': invalid section name '%s'\n", prog->name, prog->sec_name); | ||
| 13573 | return -EINVAL; | ||
| 13574 | } | ||
| 13575 | if (!match[0]) /* bare section name no autoattach */ | ||
| 13156 | return 0; | 13576 | return 0; |
| 13157 | 13577 | ||
| 13158 | sec_name = strdup(prog->sec_name); | 13578 | sec_name = strdup(prog->sec_name); |
| 13159 | if (!sec_name) | 13579 | if (!sec_name) |
| 13160 | return -ENOMEM; | 13580 | return -ENOMEM; |
| 13161 | 13581 | ||
| 13162 | /* extract "tp/<category>/<name>" or "tracepoint/<category>/<name>" */ | 13582 | tp_cat = sec_name + (match - prog->sec_name); |
| 13163 | if (str_has_pfx(prog->sec_name, "tp/")) | ||
| 13164 | tp_cat = sec_name + sizeof("tp/") - 1; | ||
| 13165 | else | ||
| 13166 | tp_cat = sec_name + sizeof("tracepoint/") - 1; | ||
| 13167 | tp_name = strchr(tp_cat, '/'); | 13583 | tp_name = strchr(tp_cat, '/'); |
| 13168 | if (!tp_name) { | 13584 | if (!tp_name) { |
| 13169 | free(sec_name); | 13585 | free(sec_name); |
| @@ -13227,37 +13643,22 @@ static int attach_raw_tp(const struct bpf_program *prog, long cookie, struct bpf · 57 unmodified lines | |||
| 13227 | "raw_tracepoint", | 13643 | "raw_tracepoint", |
| 13228 | "raw_tp.w", | 13644 | "raw_tp.w", |
| 13229 | "raw_tracepoint.w", | 13645 | "raw_tracepoint.w", |
| 13646 | "raw_tp.s", | ||
| 13647 | "raw_tracepoint.s", | ||
| 13230 | }; | 13648 | }; |
| 13231 | size_t i; | 13649 | const char *match; |
| 13232 | const char *tp_name = NULL; | ||
| 13233 | 13650 | ||
| 13234 | *link = NULL; | 13651 | *link = NULL; |
| 13235 | 13652 | ||
| 13236 | for (i = 0; i < ARRAY_SIZE(prefixes); i++) { | 13653 | match = sec_name_match_prefix(prog->sec_name, prefixes, ARRAY_SIZE(prefixes)); |
| 13237 | size_t pfx_len; | 13654 | if (!match) { |
| 13238 | 13655 | pr_warn("prog '%s': invalid section name '%s'\n", prog->name, prog->sec_name); | |
| 13239 | if (!str_has_pfx(prog->sec_name, prefixes[i])) | ||
| 13240 | continue; | ||
| 13241 | |||
| 13242 | pfx_len = strlen(prefixes[i]); | ||
| 13243 | /* no auto-attach case of, e.g., SEC("raw_tp") */ | ||
| 13244 | if (prog->sec_name[pfx_len] == '\0') | ||
| 13245 | return 0; | ||
| 13246 | |||
| 13247 | if (prog->sec_name[pfx_len] != '/') | ||
| 13248 | continue; | ||
| 13249 | |||
| 13250 | tp_name = prog->sec_name + pfx_len + 1; | ||
| 13251 | break; | ||
| 13252 | } | ||
| 13253 | |||
| 13254 | if (!tp_name) { | ||
| 13255 | pr_warn("prog '%s': invalid section name '%s'\n", | ||
| 13256 | prog->name, prog->sec_name); | ||
| 13257 | return -EINVAL; | 13656 | return -EINVAL; |
| 13258 | } | 13657 | } |
| 13658 | if (!match[0]) | ||
| 13659 | return 0; | ||
| 13259 | 13660 | ||
| 13260 | *link = bpf_program__attach_raw_tracepoint(prog, tp_name); | 13661 | *link = bpf_program__attach_raw_tracepoint(prog, match); |
| 13261 | return libbpf_get_error(*link); | 13662 | return libbpf_get_error(*link); |
| 13262 | } | 13663 | } |
| 13263 | 13664 | ||
| @@ -726,6 +726,21 @@ bpf_program__attach_ksyscall(const struct bpf_program *prog, · 725 unmodified lines | |||
| 726 | const char *syscall_name, | 726 | const char *syscall_name, |
| 727 | const struct bpf_ksyscall_opts *opts); | 727 | const struct bpf_ksyscall_opts *opts); |
| 728 | 728 | ||
| 729 | struct bpf_tracing_multi_opts { | ||
| 730 | /* size of this struct, for forward/backward compatibility */ | ||
| 731 | size_t sz; | ||
| 732 | const __u32 *ids; | ||
| 733 | const __u64 *cookies; | ||
| 734 | size_t cnt; | ||
| 735 | size_t :0; | ||
| 736 | }; | ||
| 737 | |||
| 738 | #define bpf_tracing_multi_opts__last_field cnt | ||
| 739 | |||
| 740 | LIBBPF_API struct bpf_link * | ||
| 741 | bpf_program__attach_tracing_multi(const struct bpf_program *prog, const char *pattern, | ||
| 742 | const struct bpf_tracing_multi_opts *opts); | ||
| 743 | |||
| 729 | struct bpf_uprobe_opts { | 744 | struct bpf_uprobe_opts { |
| 730 | /* size of this struct, for forward/backward compatibility */ | 745 | /* size of this struct, for forward/backward compatibility */ |
| 731 | size_t sz; | 746 | size_t sz; |
| @@ -458,6 +458,7 @@ LIBBPF_1.7.0 { · 457 unmodified lines | |||
| 458 | 458 | ||
| 459 | LIBBPF_1.8.0 { | 459 | LIBBPF_1.8.0 { |
| 460 | global: | 460 | global: |
| 461 | bpf_program__attach_tracing_multi; | ||
| 461 | bpf_program__clone; | 462 | bpf_program__clone; |
| 462 | btf__new_empty_opts; | 463 | btf__new_empty_opts; |
| 463 | } LIBBPF_1.7.0; | 464 | } LIBBPF_1.7.0; |
| @@ -250,6 +250,7 @@ const struct btf_type *skip_mods_and_typedefs(const struct btf *btf, __u32 id, _ · 249 unmodified lines | |||
| 250 | const struct btf_header *btf_header(const struct btf *btf); | 250 | const struct btf_header *btf_header(const struct btf *btf); |
| 251 | void btf_set_base_btf(struct btf *btf, const struct btf *base_btf); | 251 | void btf_set_base_btf(struct btf *btf, const struct btf *base_btf); |
| 252 | int btf_relocate(struct btf *btf, const struct btf *base_btf, __u32 **id_map); | 252 | int btf_relocate(struct btf *btf, const struct btf *base_btf, __u32 **id_map); |
| 253 | bool btf_type_is_traceable_func(const struct btf *btf, const struct btf_type *t); | ||
| 253 | 254 | ||
| 254 | static inline enum btf_func_linkage btf_func_linkage(const struct btf_type *t) | 255 | static inline enum btf_func_linkage btf_func_linkage(const struct btf_type *t) |
| 255 | { | 256 | { |
| @@ -398,6 +399,8 @@ enum kern_feature_id { · 142 unmodified lines | |||
| 398 | FEAT_UPROBE_SYSCALL, | 399 | FEAT_UPROBE_SYSCALL, |
| 399 | /* Kernel supports BTF layout information */ | 400 | /* Kernel supports BTF layout information */ |
| 400 | FEAT_BTF_LAYOUT, | 401 | FEAT_BTF_LAYOUT, |
| 402 | /* Kernel supports BPF syscall common attributes */ | ||
| 403 | FEAT_BPF_SYSCALL_COMMON_ATTRS, | ||
| 401 | __FEAT_CNT, | 404 | __FEAT_CNT, |
| 402 | }; | 405 | }; |
| 403 | 406 | ||
| @@ -768,4 +771,5 @@ int probe_fd(int fd); · 364 unmodified lines | |||
| 768 | #define SHA256_DWORD_SIZE SHA256_DIGEST_LENGTH / sizeof(__u64) | 771 | #define SHA256_DWORD_SIZE SHA256_DIGEST_LENGTH / sizeof(__u64) |
| 769 | 772 | ||
| 770 | void libbpf_sha256(const void *data, size_t len, __u8 out[SHA256_DIGEST_LENGTH]); | 773 | void libbpf_sha256(const void *data, size_t len, __u8 out[SHA256_DIGEST_LENGTH]); |
| 774 | int probe_sys_bpf_ext(void); | ||
| 771 | #endif /* __LIBBPF_LIBBPF_INTERNAL_H */ | 775 | #endif /* __LIBBPF_LIBBPF_INTERNAL_H */ |
| @@ -309,6 +309,9 @@ static int probe_map_create(enum bpf_map_type map_type) · 308 unmodified lines | |||
| 309 | value_size = sizeof(__u64); | 309 | value_size = sizeof(__u64); |
| 310 | opts.map_flags = BPF_F_NO_PREALLOC; | 310 | opts.map_flags = BPF_F_NO_PREALLOC; |
| 311 | break; | 311 | break; |
| 312 | case BPF_MAP_TYPE_RHASH: | ||
| 313 | opts.map_flags = BPF_F_NO_PREALLOC; | ||
| 314 | break; | ||
| 312 | case BPF_MAP_TYPE_CGROUP_STORAGE: | 315 | case BPF_MAP_TYPE_CGROUP_STORAGE: |
| 313 | case BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE: | 316 | case BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE: |
| 314 | key_size = sizeof(struct bpf_cgroup_storage_key); | 317 | key_size = sizeof(struct bpf_cgroup_storage_key); |
| @@ -191,8 +191,8 @@ recur: · 190 unmodified lines | |||
| 191 | case BTF_KIND_FUNC_PROTO: { | 191 | case BTF_KIND_FUNC_PROTO: { |
| 192 | struct btf_param *local_p = btf_params(local_type); | 192 | struct btf_param *local_p = btf_params(local_type); |
| 193 | struct btf_param *targ_p = btf_params(targ_type); | 193 | struct btf_param *targ_p = btf_params(targ_type); |
| 194 | __u16 local_vlen = btf_vlen(local_type); | 194 | __u32 local_vlen = btf_vlen(local_type); |
| 195 | __u16 targ_vlen = btf_vlen(targ_type); | 195 | __u32 targ_vlen = btf_vlen(targ_type); |
| 196 | int i, err; | 196 | int i, err; |
| 197 | 197 | ||
| 198 | if (local_vlen != targ_vlen) | 198 | if (local_vlen != targ_vlen) |
| @@ -1457,8 +1457,8 @@ static bool bpf_core_names_match(const struct btf *local_btf, size_t local_name_ · 1258 unmodified lines | |||
| 1457 | static int bpf_core_enums_match(const struct btf *local_btf, const struct btf_type *local_t, | 1457 | static int bpf_core_enums_match(const struct btf *local_btf, const struct btf_type *local_t, |
| 1458 | const struct btf *targ_btf, const struct btf_type *targ_t) | 1458 | const struct btf *targ_btf, const struct btf_type *targ_t) |
| 1459 | { | 1459 | { |
| 1460 | __u16 local_vlen = btf_vlen(local_t); | 1460 | __u32 local_vlen = btf_vlen(local_t); |
| 1461 | __u16 targ_vlen = btf_vlen(targ_t); | 1461 | __u32 targ_vlen = btf_vlen(targ_t); |
| 1462 | int i, j; | 1462 | int i, j; |
| 1463 | 1463 | ||
| 1464 | if (local_t->size != targ_t->size) | 1464 | if (local_t->size != targ_t->size) |
| @@ -1498,8 +1498,8 @@ static int bpf_core_composites_match(const struct btf *local_btf, const struct b · 33 unmodified lines | |||
| 1498 | bool behind_ptr, int level) | 1498 | bool behind_ptr, int level) |
| 1499 | { | 1499 | { |
| 1500 | const struct btf_member *local_m = btf_members(local_t); | 1500 | const struct btf_member *local_m = btf_members(local_t); |
| 1501 | __u16 local_vlen = btf_vlen(local_t); | 1501 | __u32 local_vlen = btf_vlen(local_t); |
| 1502 | __u16 targ_vlen = btf_vlen(targ_t); | 1502 | __u32 targ_vlen = btf_vlen(targ_t); |
| 1503 | int i, j, err; | 1503 | int i, j, err; |
| 1504 | 1504 | ||
| 1505 | if (local_vlen > targ_vlen) | 1505 | if (local_vlen > targ_vlen) |
| @@ -1674,8 +1674,8 @@ recur: · 168 unmodified lines | |||
| 1674 | case BTF_KIND_FUNC_PROTO: { | 1674 | case BTF_KIND_FUNC_PROTO: { |
| 1675 | struct btf_param *local_p = btf_params(local_t); | 1675 | struct btf_param *local_p = btf_params(local_t); |
| 1676 | struct btf_param *targ_p = btf_params(targ_t); | 1676 | struct btf_param *targ_p = btf_params(targ_t); |
| 1677 | __u16 local_vlen = btf_vlen(local_t); | 1677 | __u32 local_vlen = btf_vlen(local_t); |
| 1678 | __u16 targ_vlen = btf_vlen(targ_t); | 1678 | __u32 targ_vlen = btf_vlen(targ_t); |
| 1679 | int i, err; | 1679 | int i, err; |
| 1680 | 1680 | ||
| 1681 | if (local_k != targ_k) | 1681 | if (local_k != targ_k) |
| @@ -243,7 +243,12 @@ static inline int skel_map_create(enum bpf_map_type map_type, · 242 unmodified lines | |||
| 243 | attr.excl_prog_hash = (unsigned long) excl_prog_hash; | 243 | attr.excl_prog_hash = (unsigned long) excl_prog_hash; |
| 244 | attr.excl_prog_hash_size = excl_prog_hash_sz; | 244 | attr.excl_prog_hash_size = excl_prog_hash_sz; |
| 245 | 245 | ||
| 246 | #ifdef __KERNEL__ | ||
| 247 | if (strscpy(attr.map_name, map_name) < 0) | ||
| 248 | return -EINVAL; | ||
| 249 | #else | ||
| 246 | strncpy(attr.map_name, map_name, sizeof(attr.map_name)); | 250 | strncpy(attr.map_name, map_name, sizeof(attr.map_name)); |
| 251 | #endif | ||
| 247 | attr.key_size = key_size; | 252 | attr.key_size = key_size; |
| 248 | attr.value_size = value_size; | 253 | attr.value_size = value_size; |
| 249 | attr.max_entries = max_entries; | 254 | attr.max_entries = max_entries; |
| @@ -107,6 +107,41 @@ static void *strset_add_str_mem(struct strset *set, size_t add_sz) · 106 unmodified lines | |||
| 107 | set->strs_data_len, set->strs_data_max_len, add_sz); | 107 | set->strs_data_len, set->strs_data_max_len, add_sz); |
| 108 | } | 108 | } |
| 109 | 109 | ||
| 110 | static long strset_str_append(struct strset *set, const char *s) | ||
| 111 | { | ||
| 112 | uintptr_t old_data = (uintptr_t)set->strs_data; | ||
| 113 | size_t old_data_len = set->strs_data_len; | ||
| 114 | uintptr_t old_s = (uintptr_t)s; | ||
| 115 | long len = strlen(s) + 1; | ||
| 116 | void *p; | ||
| 117 | |||
| 118 | /* | ||
| 119 | * Hashmap keys are always offsets within set->strs_data, so to even | ||
| 120 | * look up some string from the "outside", we need to first append it | ||
| 121 | * at the end, so that it can be addressed with an offset. Luckily, | ||
| 122 | * until set->strs_data_len is incremented, that string is just a piece | ||
| 123 | * of garbage for the rest of the code, so no harm, no foul. On the | ||
| 124 | * other hand, if the string is unique, it's already appended and | ||
| 125 | * ready to be used, only a simple set->strs_data_len increment away. | ||
| 126 | */ | ||
| 127 | p = strset_add_str_mem(set, len); | ||
| 128 | if (!p) | ||
| 129 | return -ENOMEM; | ||
| 130 | |||
| 131 | /* | ||
| 132 | * The set->strs_data might have reallocated and if 's' pointed | ||
| 133 | * to an internal string within the old buffer, then it became | ||
| 134 | * dangling and needs to be reconstructed before the copy. | ||
| 135 | */ | ||
| 136 | if (old_data && old_data != (uintptr_t)set->strs_data && | ||
| 137 | old_s >= old_data && old_s < old_data + old_data_len) | ||
| 138 | s = set->strs_data + (old_s - old_data); | ||
| 139 | |||
| 140 | memcpy(p, s, len); | ||
| 141 | |||
| 142 | return len; | ||
| 143 | } | ||
| 144 | |||
| 110 | /* Find string offset that corresponds to a given string *s*. | 145 | /* Find string offset that corresponds to a given string *s*. |
| 111 | * Returns: | 146 | * Returns: |
| 112 | * - >0 offset into string data, if string is found; | 147 | * - >0 offset into string data, if string is found; |
| @@ -116,16 +151,12 @@ static void *strset_add_str_mem(struct strset *set, size_t add_sz) · 3 unmodified lines | |||
| 116 | int strset__find_str(struct strset *set, const char *s) | 151 | int strset__find_str(struct strset *set, const char *s) |
| 117 | { | 152 | { |
| 118 | long old_off, new_off, len; | 153 | long old_off, new_off, len; |
| 119 | void *p; | ||
| 120 | 154 | ||
| 121 | /* see strset__add_str() for why we do this */ | 155 | len = strset_str_append(set, s); |
| 122 | len = strlen(s) + 1; | 156 | if (len < 0) |
| 123 | p = strset_add_str_mem(set, len); | 157 | return len; |
| 124 | if (!p) | ||
| 125 | return -ENOMEM; | ||
| 126 | 158 | ||
| 127 | new_off = set->strs_data_len; | 159 | new_off = set->strs_data_len; |
| 128 | memcpy(p, s, len); | ||
| 129 | 160 | ||
| 130 | if (hashmap__find(set->strs_hash, new_off, &old_off)) | 161 | if (hashmap__find(set->strs_hash, new_off, &old_off)) |
| 131 | return old_off; | 162 | return old_off; |
| @@ -142,24 +173,13 @@ int strset__find_str(struct strset *set, const char *s) · 10 unmodified lines | |||
| 142 | int strset__add_str(struct strset *set, const char *s) | 173 | int strset__add_str(struct strset *set, const char *s) |
| 143 | { | 174 | { |
| 144 | long old_off, new_off, len; | 175 | long old_off, new_off, len; |
| 145 | void *p; | ||
| 146 | int err; | 176 | int err; |
| 147 | 177 | ||
| 148 | /* Hashmap keys are always offsets within set->strs_data, so to even | 178 | len = strset_str_append(set, s); |
| 149 | * look up some string from the "outside", we need to first append it | 179 | if (len < 0) |
| 150 | * at the end, so that it can be addressed with an offset. Luckily, | 180 | return len; |
| 151 | * until set->strs_data_len is incremented, that string is just a piece | ||
| 152 | * of garbage for the rest of the code, so no harm, no foul. On the | ||
| 153 | * other hand, if the string is unique, it's already appended and | ||
| 154 | * ready to be used, only a simple set->strs_data_len increment away. | ||
| 155 | */ | ||
| 156 | len = strlen(s) + 1; | ||
| 157 | p = strset_add_str_mem(set, len); | ||
| 158 | if (!p) | ||
| 159 | return -ENOMEM; | ||
| 160 | 181 | ||
| 161 | new_off = set->strs_data_len; | 182 | new_off = set->strs_data_len; |
| 162 | memcpy(p, s, len); | ||
| 163 | 183 | ||
| 164 | /* Now attempt to add the string, but only if the string with the same | 184 | /* Now attempt to add the string, but only if the string with the same |
| 165 | * contents doesn't exist already (HASHMAP_ADD strategy). If such | 185 | * contents doesn't exist already (HASHMAP_ADD strategy). If such |
| @@ -468,10 +468,10 @@ static int parse_elf_segs(Elf *elf, const char *path, struct elf_seg **segs, siz · 467 unmodified lines | |||
| 468 | 468 | ||
| 469 | static int parse_vma_segs(int pid, const char *lib_path, struct elf_seg **segs, size_t *seg_cnt) | 469 | static int parse_vma_segs(int pid, const char *lib_path, struct elf_seg **segs, size_t *seg_cnt) |
| 470 | { | 470 | { |
| 471 | char path[PATH_MAX], line[PATH_MAX], mode[16]; | 471 | char path[PATH_MAX], line[4096], mode[16]; |
| 472 | size_t seg_start, seg_end, seg_off; | 472 | size_t seg_start, seg_end, seg_off; |
| 473 | struct elf_seg *seg; | 473 | struct elf_seg *seg; |
| 474 | int tmp_pid, i, err; | 474 | int tmp_pid, n, i, err; |
| 475 | FILE *f; | 475 | FILE *f; |
| 476 | 476 | ||
| 477 | *seg_cnt = 0; | 477 | *seg_cnt = 0; |
| @@ -480,8 +480,13 @@ static int parse_vma_segs(int pid, const char *lib_path, struct elf_seg **segs, · 2 unmodified lines | |||
| 480 | * /proc/<pid>/root/<path>. They will be reported as just /<path> in | 480 | * /proc/<pid>/root/<path>. They will be reported as just /<path> in |
| 481 | * /proc/<pid>/maps. | 481 | * /proc/<pid>/maps. |
| 482 | */ | 482 | */ |
| 483 | if (sscanf(lib_path, "/proc/%d/root%s", &tmp_pid, path) == 2 && pid == tmp_pid) | 483 | /* %n is not counted in sscanf() return value, so initialize it. */ |
| 484 | n = 0; | ||
| 485 | if (sscanf(lib_path, "/proc/%d/root%n", &tmp_pid, &n) == 1 && | ||
| 486 | n > 0 && pid == tmp_pid && lib_path[n] == '/') { | ||
| 487 | libbpf_strlcpy(path, lib_path + n, sizeof(path)); | ||
| 484 | goto proceed; | 488 | goto proceed; |
| 489 | } | ||
| 485 | 490 | ||
| 486 | if (!realpath(lib_path, path)) { | 491 | if (!realpath(lib_path, path)) { |
| 487 | pr_warn("usdt: failed to get absolute path of '%s' (err %s), using path as is...\n", | 492 | pr_warn("usdt: failed to get absolute path of '%s' (err %s), using path as is...\n", |
| @@ -504,8 +509,11 @@ proceed: · 16 unmodified lines | |||
| 504 | * 7f5c6f5d1000-7f5c6f5d3000 rw-p 001c7000 08:04 21238613 /usr/lib64/libc-2.17.so | 509 | * 7f5c6f5d1000-7f5c6f5d3000 rw-p 001c7000 08:04 21238613 /usr/lib64/libc-2.17.so |
| 505 | * 7f5c6f5d3000-7f5c6f5d8000 rw-p 00000000 00:00 0 | 510 | * 7f5c6f5d3000-7f5c6f5d8000 rw-p 00000000 00:00 0 |
| 506 | * 7f5c6f5d8000-7f5c6f5d9000 r-xp 00000000 103:01 362990598 /data/users/andriin/linux/tools/bpf/usdt/libhello_usdt.so | 511 | * 7f5c6f5d8000-7f5c6f5d9000 r-xp 00000000 103:01 362990598 /data/users/andriin/linux/tools/bpf/usdt/libhello_usdt.so |
| 512 | * | ||
| 513 | * Some VMA names can be longer than the local buffer. Bound the | ||
| 514 | * writes, but still consume the rest of the line. | ||
| 507 | */ | 515 | */ |
| 508 | while (fscanf(f, "%zx-%zx %s %zx %*s %*d%[^\n]\n", | 516 | while (fscanf(f, "%zx-%zx %15s %zx %*s %*d%4095[^\n]%*[^\n]\n", |
| 509 | &seg_start, &seg_end, mode, &seg_off, line) == 5) { | 517 | &seg_start, &seg_end, mode, &seg_off, line) == 5) { |
| 510 | void *tmp; | 518 | void *tmp; |
| 511 | 519 | ||
| @@ -21,7 +21,6 @@ test_lirc_mode2_user · 20 unmodified lines | |||
| 21 | flow_dissector_load | 21 | flow_dissector_load |
| 22 | test_tcpnotify_user | 22 | test_tcpnotify_user |
| 23 | test_libbpf | 23 | test_libbpf |
| 24 | xdping | ||
| 25 | test_cpp | 24 | test_cpp |
| 26 | *.d | 25 | *.d |
| 27 | *.subskel.h | 26 | *.subskel.h |
| @@ -44,6 +44,12 @@ SKIP_LLVM ?= · 43 unmodified lines | |||
| 44 | SKIP_LIBBFD ?= | 44 | SKIP_LIBBFD ?= |
| 45 | SKIP_CRYPTO ?= | 45 | SKIP_CRYPTO ?= |
| 46 | 46 | ||
| 47 | # When BPF_STRICT_BUILD is 1, any BPF object, skeleton, test object, or | ||
| 48 | # benchmark compilation failure is fatal. Set to 0 to tolerate failures | ||
| 49 | # and continue building the remaining tests. | ||
| 50 | BPF_STRICT_BUILD ?= 1 | ||
| 51 | PERMISSIVE := $(filter 0,$(BPF_STRICT_BUILD)) | ||
| 52 | |||
| 47 | ifeq ($(srctree),) | 53 | ifeq ($(srctree),) |
| 48 | srctree := $(patsubst %/,%,$(dir $(CURDIR))) | 54 | srctree := $(patsubst %/,%,$(dir $(CURDIR))) |
| 49 | srctree := $(patsubst %/,%,$(dir $(srctree))) | 55 | srctree := $(patsubst %/,%,$(dir $(srctree))) |
| @@ -51,19 +57,20 @@ srctree := $(patsubst %/,%,$(dir $(srctree))) · 1 unmodified line | |||
| 51 | srctree := $(patsubst %/,%,$(dir $(srctree))) | 57 | srctree := $(patsubst %/,%,$(dir $(srctree))) |
| 52 | endif | 58 | endif |
| 53 | 59 | ||
| 54 | CFLAGS += -g $(OPT_FLAGS) -rdynamic -std=gnu11 \ | 60 | COMMON_CFLAGS = -g $(OPT_FLAGS) -rdynamic -std=gnu11 \ |
| 55 | -Wall -Werror -fno-omit-frame-pointer \ | 61 | -Wall -Werror -fno-omit-frame-pointer \ |
| 56 | -Wno-unused-but-set-variable \ | 62 | -Wno-unused-but-set-variable \ |
| 57 | $(GENFLAGS) $(SAN_CFLAGS) $(LIBELF_CFLAGS) \ | 63 | $(GENFLAGS) $(SAN_CFLAGS) $(LIBELF_CFLAGS) \ |
| 58 | -I$(CURDIR) -I$(INCLUDE_DIR) -I$(GENDIR) -I$(LIBDIR) \ | 64 | -I$(CURDIR) -I$(INCLUDE_DIR) -I$(GENDIR) -I$(LIBDIR) \ |
| 59 | -I$(TOOLSINCDIR) -I$(TOOLSARCHINCDIR) -I$(APIDIR) -I$(OUTPUT) | 65 | -I$(TOOLSINCDIR) -I$(TOOLSARCHINCDIR) -I$(APIDIR) -I$(OUTPUT) \ |
| 66 | -I$(CURDIR)/libarena/include | ||
| 60 | LDFLAGS += $(SAN_LDFLAGS) | 67 | LDFLAGS += $(SAN_LDFLAGS) |
| 61 | LDLIBS += $(LIBELF_LIBS) -lz -lrt -lpthread | 68 | LDLIBS += $(LIBELF_LIBS) -lz -lrt -lpthread |
| 62 | 69 | ||
| 63 | PCAP_CFLAGS := $(shell $(PKG_CONFIG) --cflags libpcap 2>/dev/null && echo "-DTRAFFIC_MONITOR=1") | 70 | PCAP_CFLAGS := $(shell $(PKG_CONFIG) --cflags libpcap 2>/dev/null && echo "-DTRAFFIC_MONITOR=1") |
| 64 | PCAP_LIBS := $(shell $(PKG_CONFIG) --libs libpcap 2>/dev/null) | 71 | PCAP_LIBS := $(shell $(PKG_CONFIG) --libs libpcap 2>/dev/null) |
| 65 | LDLIBS += $(PCAP_LIBS) | 72 | LDLIBS += $(PCAP_LIBS) |
| 66 | CFLAGS += $(PCAP_CFLAGS) | 73 | CFLAGS += $(COMMON_CFLAGS) $(PCAP_CFLAGS) |
| 67 | 74 | ||
| 68 | # Some utility functions use LLVM libraries | 75 | # Some utility functions use LLVM libraries |
| 69 | jit_disasm_helpers.c-CFLAGS = $(LLVM_CFLAGS) | 76 | jit_disasm_helpers.c-CFLAGS = $(LLVM_CFLAGS) |
| @@ -78,6 +85,12 @@ ifneq ($(shell $(CLANG) --target=bpf -mcpu=help 2>&1 | grep 'v4'),) · 8 unmodified lines | |||
| 78 | CLANG_CPUV4 := 1 | 85 | CLANG_CPUV4 := 1 |
| 79 | endif | 86 | endif |
| 80 | 87 | ||
| 88 | # Check whether clang supports BPF address sanitizer (requires LLVM 22+) | ||
| 89 | CLANG_HAS_ARENA_ASAN := $(shell echo 'int x;' | \ | ||
| 90 | $(CLANG) --target=bpf -fsanitize=kernel-address \ | ||
| 91 | -mllvm -asan-shadow-addr-space=1 \ | ||
| 92 | -x c -c - -o /dev/null 2>/dev/null && echo 1) | ||
| 93 | |||
| 81 | # Order correspond to 'make run_tests' order | 94 | # Order correspond to 'make run_tests' order |
| 82 | TEST_GEN_PROGS = test_verifier test_tag test_maps test_lru_map test_progs \ | 95 | TEST_GEN_PROGS = test_verifier test_tag test_maps test_lru_map test_progs \ |
| 83 | test_sockmap \ | 96 | test_sockmap \ |
| @@ -111,7 +124,6 @@ TEST_FILES = xsk_prereqs.sh $(wildcard progs/btf_dump_test_case_*.c) · 27 unmodified lines | |||
| 111 | # Order correspond to 'make run_tests' order | 124 | # Order correspond to 'make run_tests' order |
| 112 | TEST_PROGS := test_kmod.sh \ | 125 | TEST_PROGS := test_kmod.sh \ |
| 113 | test_lirc_mode2.sh \ | 126 | test_lirc_mode2.sh \ |
| 114 | test_xdping.sh \ | ||
| 115 | test_bpftool_build.sh \ | 127 | test_bpftool_build.sh \ |
| 116 | test_doc_build.sh \ | 128 | test_doc_build.sh \ |
| 117 | test_xsk.sh \ | 129 | test_xsk.sh \ |
| @@ -134,7 +146,6 @@ TEST_GEN_PROGS_EXTENDED = \ · 16 unmodified lines | |||
| 134 | xdp_features \ | 146 | xdp_features \ |
| 135 | xdp_hw_metadata \ | 147 | xdp_hw_metadata \ |
| 136 | xdp_synproxy \ | 148 | xdp_synproxy \ |
| 137 | xdping \ | ||
| 138 | xskxceiver | 149 | xskxceiver |
| 139 | 150 | ||
| 140 | TEST_GEN_FILES += $(TEST_KMODS) liburandom_read.so urandom_read sign-file uprobe_multi | 151 | TEST_GEN_FILES += $(TEST_KMODS) liburandom_read.so urandom_read sign-file uprobe_multi |
| @@ -153,12 +164,13 @@ override define CLEAN · 12 unmodified lines | |||
| 153 | $(Q)$(RM) -r $(TEST_KMODS) | 164 | $(Q)$(RM) -r $(TEST_KMODS) |
| 154 | $(Q)$(RM) -r $(EXTRA_CLEAN) | 165 | $(Q)$(RM) -r $(EXTRA_CLEAN) |
445 more lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 96 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 51 lines aren’t shown here.
This diff is large: its 63 lines aren’t shown here.
This diff is large: its 61 lines aren’t shown here.
This diff is large: its 85 lines aren’t shown here.
This diff is large: its 299 lines aren’t shown here.
This diff is large: its 70 lines aren’t shown here.
This diff is large: its 1125 lines aren’t shown here.
This diff is large: its 80 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 47 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 98 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 7 lines aren’t shown here.
Binary file not shown.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 93 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 45 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 85 lines aren’t shown here.
This diff is large: its 102 lines aren’t shown here.
This diff is large: its 82 lines aren’t shown here.
This diff is large: its 94 lines aren’t shown here.
This diff is large: its 84 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 139 lines aren’t shown here.
This diff is large: its 259 lines aren’t shown here.
This diff is large: its 53 lines aren’t shown here.
This diff is large: its 210 lines aren’t shown here.
This diff is large: its 670 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 969 lines aren’t shown here.
This diff is large: its 195 lines aren’t shown here.
This diff is large: its 554 lines aren’t shown here.
This diff is large: its 904 lines aren’t shown here.
This diff is large: its 53 lines aren’t shown here.
This diff is large: its 1048 lines aren’t shown here.
This diff is large: its 235 lines aren’t shown here.
This diff is large: its 74 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 125 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 76 lines aren’t shown here.
This diff is large: its 125 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 50 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 254 lines aren’t shown here.
This diff is large: its 94 lines aren’t shown here.
This diff is large: its 114 lines aren’t shown here.
This diff is large: its 244 lines aren’t shown here.
This diff is large: its 98 lines aren’t shown here.
This diff is large: its 163 lines aren’t shown here.
This diff is large: its 140 lines aren’t shown here.
This diff is large: its 196 lines aren’t shown here.
This diff is large: its 136 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 120 lines aren’t shown here.
This diff is large: its 184 lines aren’t shown here.
This diff is large: its 92 lines aren’t shown here.
This diff is large: its 1136 lines aren’t shown here.
This diff is large: its 143 lines aren’t shown here.
This diff is large: its 69 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 140 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 205 lines aren’t shown here.
This diff is large: its 70 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 200 lines aren’t shown here.
This diff is large: its 961 lines aren’t shown here.
This diff is large: its 122 lines aren’t shown here.
This diff is large: its 50 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 71 lines aren’t shown here.
This diff is large: its 119 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 70 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 75 lines aren’t shown here.
This diff is large: its 69 lines aren’t shown here.
This diff is large: its 75 lines aren’t shown here.
This diff is large: its 71 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 50 lines aren’t shown here.
This diff is large: its 81 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 45 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 44 lines aren’t shown here.
This diff is large: its 45 lines aren’t shown here.
This diff is large: its 193 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 119 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 82 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 81 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 51 lines aren’t shown here.
This diff is large: its 78 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 41 lines aren’t shown here.
This diff is large: its 107 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 461 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 249 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 274 lines aren’t shown here.
This diff is large: its 115 lines aren’t shown here.
This diff is large: its 167 lines aren’t shown here.
This diff is large: its 136 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 57 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 45 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 146 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 60 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 176 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 113 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 50 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 93 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 215 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 42 lines aren’t shown here.
This diff is large: its 44 lines aren’t shown here.
This diff is large: its 66 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 81 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 175 lines aren’t shown here.
This diff is large: its 50 lines aren’t shown here.
This diff is large: its 98 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 39 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 54 lines aren’t shown here.
This diff is large: its 58 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 108 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 50 lines aren’t shown here.
This diff is large: its 120 lines aren’t shown here.
This diff is large: its 108 lines aren’t shown here.
This diff is large: its 54 lines aren’t shown here.
This diff is large: its 448 lines aren’t shown here.
This diff is large: its 186 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 45 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 77 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 648 lines aren’t shown here.
This diff is large: its 184 lines aren’t shown here.
This diff is large: its 42 lines aren’t shown here.
This diff is large: its 93 lines aren’t shown here.
This diff is large: its 54 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 242 lines aren’t shown here.
This diff is large: its 49 lines aren’t shown here.
This diff is large: its 125 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 104 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 7 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 6 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 153 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 67 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 113 lines aren’t shown here.
This diff is large: its 255 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.