Pull bpf updates from Alexei Starovoitov: "Major changes: - Recover from BPF arena page faults using a scratch page and add ptep_try_set() for lockless empty-slot installs on x86 and arm64. This allows BPF kfuncs to access arena pointers directly. The 'arena_direct_access' stable branch was created for this work and was pulled into sched-ext and bpf-next trees (Tejun Heo, Kumar Kartikeya Dwivedi) - Lift old restriction and support 6+ arguments in BPF programs and kfuncs on x86 and arm64 (Yonghong Song, Puranjay Mohan) Other features and fixes: - Add 24-bit BTF vlen and reclaim unused bits in the BTF UAPI to ease addition of new BTF kinds (Alan Maguire) - Raise the maximum BPF call chain depth from 8 to 16 frames (Alexei Starovoitov) - Refactor object relationship tracking in the verifier and fix a dynptr use-after-free bug (Amery Hung) - Harden the signed program loader and reject exclusive maps as inner maps (Daniel Borkmann) - Replace the verifier min/max bounds fields with a circular number (cnum) representation and improve 32->64 bit range refinements (Eduard Zingerman) - Introduce the arena library and runtime (libarena) with a buddy allocator, rbtree and SPMC queue data structures, ASAN support and a parallel test harness. Allow subprograms to return arena pointers and switch to a BTF type-tag based __arena annotation (Emil Tsalapatis) - Cache build IDs in the sleepable stackmap path and avoid faultable build ID reads under mm locks (Ihor Solodrai) - Introduce the tracing_multi link to attach a single BPF program to many kernel functions at once. Allow specifying the uprobe_multi target via FD (Jiri Olsa) - Extend the bpf_list family of kfuncs with bpf_list_add/del(), and bpf_list_is_first/is_last/empty() (Kaitao Cheng) - Extend the BPF syscall with common attributes support for prog_load, btf_load and map_create (Leon Hwang) - Wrap rhashtable as BPF map (Mykyta Yatsenko, Herbert Xu) - Add sleepable support for tracepoint programs and fix deadlocks in LRU map due to NMI reentry (Mykyta Yatsenko) - Fix OOB access in bpf_flow_keys, fix nullness analysis of inner arrays, enforce write checks for global subprograms (Nuoqi Gui) - Report the maximum combined stack depth and print a breakdown of instructions processed per subprogram (Paul Chaignon) - Add an XDP load-balancer benchmark and arm64 JIT support for stack arguments (Puranjay Mohan) - Add kfuncs to traverse over wakeup_sources (Samuel Wu) - Allow sleepable BPF programs to use LPM trie maps directly (Vlad Poenaru) - Many more fixes and cleanups across the verifier, BTF, sockmap, devmap, bpffs, security hooks, s390/riscv/loongarch JITs, rqspinlock, libbpf, bpftool, selftests" * tag 'bpf-next-7.2' of git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next: (336 commits) selftests/bpf: Work around llvm stack overflow in crypto progs selftests/bpf: add test for bpf_msg_pop_data() overflow bpf, sockmap: fix integer overflow in bpf_msg_pop_data() bounds check sockmap: Fix use-after-free in udp_bpf_recvmsg() bpf, sockmap: keep sk_msg copy state in sync bpf, sockmap: Fix wrong rsge offset in bpf_msg_push_data() bpf, sockmap: reject overflowing copy + len in bpf_msg_push_data() selftsets/bpf: Retry map update on helper_fill_hashmap() selftests/bpf: Add test for sleepable lsm_cgroup rejection selftests/bpf: Add test to verify the fix for bpf_setsockopt() helper bpf: Fix bpf_get/setsockopt to tos for ipv4-mapped ipv6 socket selftests/bpf: Avoid static LLVM linking for cross builds selftests/bpf: Use common CFLAGS for urandom_read selftests/bpf: Initialize operation name before use tools/bpf: build: Append extra cflags libbpf: Initialize CFLAGS before including Makefile.include bpftool: Append extra host flags bpftool: Avoid adding EXTRA_CFLAGS to HOST_CFLAGS bpftool: Pass host flags to bootstrap libbpf selftests/bpf: correct CONFIG_PPC64 macro name in comment ...
Unified · Split
| @@ -97,10 +97,8 @@ Each type contains the following common data:: · 96 unmodified lines | ||
| 97 | 97 | struct btf_type { |
| 98 | 98 | __u32 name_off; |
| 99 | 99 | /* "info" bits arrangement |
| 100 | − * bits 0-15: vlen (e.g. # of struct's members) | |
| 101 | − * bits 16-23: unused | |
| 102 | − * bits 24-28: kind (e.g. int, ptr, array...etc) | |
| 103 | − * bits 29-30: unused | |
| 100 | + * bits 0-23: vlen (e.g. # of struct's members) | |
| 101 | + * bits 24-30: kind (e.g. int, ptr, array...etc) | |
| 104 | 102 | * bit 31: kind_flag, currently used by |
| 105 | 103 | * struct, union, enum, fwd, enum64, |
| 106 | 104 | * decl_tag and type_tag |
| @@ -207,8 +207,26 @@ Here, the buffer may be NULL. If the buffer is not NULL, it must be at least · 206 unmodified lines | ||
| 207 | 207 | buffer__szk bytes in size. The kfunc is responsible for checking if the buffer |
| 208 | 208 | is NULL before using it. |
| 209 | 209 | |
| 210 | −2.3.5 __str Annotation | |
| 211 | −---------------------------- | |
| 210 | +2.3.5 __nonown_allowed Annotation | |
| 211 | +--------------------------------- | |
| 212 | + | |
| 213 | +This annotation is used to indicate that the parameter may be a non-owning reference. | |
| 214 | + | |
| 215 | +An example is given below:: | |
| 216 | + | |
| 217 | + __bpf_kfunc int bpf_list_add(..., struct bpf_list_node | |
| 218 | + *prev__nonown_allowed, ...) | |
| 219 | + { | |
| 220 | + ... | |
| 221 | + } | |
| 222 | + | |
| 223 | +For the ``prev__nonown_allowed`` parameter (resolved as ``KF_ARG_PTR_TO_LIST_NODE``), | |
| 224 | +suffix ``__nonown_allowed`` retains the usual owning-pointer rules and also | |
| 225 | +permits a non-owning reference with no ref_obj_id (e.g. the return value of | |
| 226 | +bpf_list_front() / bpf_list_back()). | |
| 227 | + | |
| 228 | +2.3.6 __str Annotation | |
| 229 | +---------------------- | |
| 212 | 230 | This annotation is used to indicate that the argument is a constant string. |
| 213 | 231 | |
| 214 | 232 | An example is given below:: |
| @@ -462,6 +480,20 @@ In order to accommodate such requirements, the verifier will enforce strict · 247 unmodified lines | ||
| 462 | 480 | PTR_TO_BTF_ID type matching if two types have the exact same name, with one |
| 463 | 481 | being suffixed with ``___init``. |
| 464 | 482 | |
| 483 | +2.8 Accessing arena memory through kfunc arguments | |
| 484 | +-------------------------------------------------- | |
| 485 | + | |
| 486 | +A read or write at any address inside an arena does not oops the kernel. | |
| 487 | +Unallocated arena pages are lazily backed by a scratch page and the | |
| 488 | +access is reported through the program's BPF stream as an error. Only | |
| 489 | +the BPF program's correctness is affected; the kernel itself remains | |
| 490 | +intact. | |
| 491 | + | |
| 492 | +The arena is followed by a ``GUARD_SZ / 2`` (32 KiB) guard region that | |
| 493 | +is also covered by this recovery. A kfunc handed an arena pointer may | |
| 494 | +therefore access up to ``GUARD_SZ / 2`` past it without bounds-checking | |
| 495 | +against the arena. Larger accesses must verify the range explicitly. | |
| 496 | + | |
| 465 | 497 | .. _BPF_kfunc_lifecycle_expectations: |
| 466 | 498 | |
| 467 | 499 | 3. kfunc lifecycle expectations |
| @@ -21,10 +21,18 @@ digraph { · 20 unmodified lines | ||
| 21 | 21 | // names that initiate the corresponding logic in kernel/bpf/bpf_lru_list.c. |
| 22 | 22 | // Number suffixes and errno suffixes handle subsections of the corresponding |
| 23 | 23 | // logic in the function as of the writing of this dot. |
| 24 | + // | |
| 25 | + // All LRU locks are rqspinlock_t. Every acquire can fail (AA self-deadlock | |
| 26 | + // or contention timeout); on failure the corresponding helper returns NULL | |
| 27 | + // and the caller propagates -ENOMEM. The "rqspinlock acquire failed" | |
| 28 | + // terminal below is reached via the dashed arrows from each acquire site. | |
| 29 | + | |
| 30 | + rqspinlock_failed [shape=rectangle, | |
| 31 | + label="Any LRU rqspinlock\nacquire fails\n(AA or timeout)"] | |
| 24 | 32 | |
| 25 | 33 | // cf. __local_list_pop_free() / bpf_percpu_lru_pop_free() |
| 26 | 34 | local_freelist_check [shape=diamond,fillcolor=1, |
| 27 | − label="Local freelist\nnode available?"]; | |
| 35 | + label="Local freelist\nnode available?\n(lockless free_llist)"]; | |
| 28 | 36 | use_local_node [shape=rectangle, |
| 29 | 37 | label="Use node owned\nby this CPU"] |
| 30 | 38 | |
| @@ -82,6 +90,15 @@ digraph { · 51 unmodified lines | ||
| 82 | 90 | // fn__local_list_pop_pending() |
| 83 | 91 | } |
| 84 | 92 | |
| 93 | + // Post-steal: re-acquire local loc_l->lock to insert the stolen node into | |
| 94 | + // the local pending list. If the acquire fails, the stolen node is published | |
| 95 | + // to the lockless local free_llist so the next pop on this CPU picks it up | |
| 96 | + // instead of orphaning it. | |
| 97 | + post_steal_lock [shape=diamond,fillcolor=1, | |
| 98 | + label="Acquire local\nloc_l->lock\nto add pending"] | |
| 99 | + post_steal_to_free_llist [shape=rectangle, | |
| 100 | + label="Publish stolen node to\nlocal free_llist (lockless)"] | |
| 101 | + | |
| 85 | 102 | fn_bpf_lru_list_pop_free_to_local2 [shape=rectangle, |
| 86 | 103 | label="Use node that was\nnot recently referenced"] |
| 87 | 104 | local_freelist_check4 [shape=rectangle, |
| @@ -97,10 +114,19 @@ digraph { · 9 unmodified lines | ||
| 97 | 114 | fn_htab_lru_map_update_elem_ENOENT [shape=oval,label="return -ENOENT"] |
| 98 | 115 | |
| 99 | 116 | begin -> local_freelist_check |
| 117 | + // The initial per-CPU lock (loc_l->lock for common, l->lock for percpu) is | |
| 118 | + // acquired before the local freelist check; rqspinlock failure here exits | |
| 119 | + // directly to -ENOMEM (no recovery needed: nothing was removed yet). | |
| 120 | + local_freelist_check -> rqspinlock_failed [style=dashed, | |
| 121 | + xlabel="acquire fails"] | |
| 100 | 122 | local_freelist_check -> use_local_node [xlabel="Y"] |
| 101 | 123 | local_freelist_check -> common_lru_check [xlabel="N"] |
| 102 | 124 | common_lru_check -> fn_bpf_lru_list_pop_free_to_local [xlabel="Y"] |
| 103 | 125 | common_lru_check -> fn___bpf_lru_list_shrink_inactive [xlabel="N"] |
| 126 | + // Global lru_list lock acquire failure in pop_free_to_local: skip refill, | |
| 127 | + // fall through to the steal path. Not ENOMEM by itself. | |
| 128 | + fn_bpf_lru_list_pop_free_to_local -> common_lru_check2 [style=dashed, | |
| 129 | + xlabel="global lru_lock\nacquire fails"] | |
| 104 | 130 | fn_bpf_lru_list_pop_free_to_local -> fn___bpf_lru_node_move_to_free |
| 105 | 131 | fn___bpf_lru_node_move_to_free -> |
| 106 | 132 | fn_bpf_lru_list_pop_free_to_local2 [xlabel="Y"] |
| @@ -120,13 +146,27 @@ digraph { · 13 unmodified lines | ||
| 120 | 146 | local_freelist_check6 -> local_freelist_check7 |
| 121 | 147 | local_freelist_check7 -> fn_htab_lru_map_update_elem |
| 122 | 148 | |
| 123 | − fn_htab_lru_map_update_elem -> fn_htab_lru_map_update_elem3 [xlabel = "Y"] | |
| 149 | + // Steal-loop victim lock failure is silent: treat as "no node found here" | |
| 150 | + // and continue to next CPU; same edge as the existing "N" path. | |
| 151 | + local_freelist_check5 -> fn_htab_lru_map_update_elem2 [style=dashed, | |
| 152 | + xlabel="victim's lock\nfails: skip"] | |
| 153 | + // After a successful steal, re-acquire the local loc_l->lock. On failure | |
| 154 | + // the stolen node is published to free_llist (recovered, not orphaned) | |
| 155 | + // and the update returns -ENOMEM. | |
| 156 | + fn_htab_lru_map_update_elem -> post_steal_lock [xlabel = "Y"] | |
| 157 | + post_steal_lock -> fn_htab_lru_map_update_elem3 [xlabel = "OK"] | |
| 158 | + post_steal_lock -> post_steal_to_free_llist [style=dashed, | |
| 159 | + xlabel="loc_l->lock\nacquire fails"] | |
| 160 | + post_steal_to_free_llist -> fn_htab_lru_map_update_elem_ENOMEM | |
| 124 | 161 | fn_htab_lru_map_update_elem -> fn_htab_lru_map_update_elem2 [xlabel = "N"] |
| 125 | 162 | fn_htab_lru_map_update_elem2 -> |
| 126 | 163 | fn_htab_lru_map_update_elem_ENOMEM [xlabel = "Y"] |
| 127 | 164 | fn_htab_lru_map_update_elem2 -> local_freelist_check5 [xlabel = "N"] |
| 128 | 165 | fn_htab_lru_map_update_elem3 -> fn_htab_lru_map_update_elem4 |
| 129 | 166 | |
| 167 | + // Shared rqspinlock-failure terminal collapses to the same -ENOMEM exit. | |
| 168 | + rqspinlock_failed -> fn_htab_lru_map_update_elem_ENOMEM | |
| 169 | + | |
| 130 | 170 | use_local_node -> fn_htab_lru_map_update_elem4 |
| 131 | 171 | fn_bpf_lru_list_pop_free_to_local2 -> fn_htab_lru_map_update_elem4 |
| 132 | 172 | local_freelist_check4 -> fn_htab_lru_map_update_elem4 |
| @@ -4797,13 +4797,6 @@ S: Supported · 4796 unmodified lines | ||
| 4797 | 4797 | F: arch/x86/net/ |
| 4798 | 4798 | X: arch/x86/net/bpf_jit_comp32.c |
| 4799 | 4799 | |
| 4800 | −BPF [BTF] | |
| 4801 | −M: Martin KaFai Lau <martin.lau@linux.dev> | |
| 4802 | −L: bpf@vger.kernel.org | |
| 4803 | −S: Maintained | |
| 4804 | −F: include/linux/btf* | |
| 4805 | −F: kernel/bpf/btf.c | |
| 4806 | − | |
| 4807 | 4800 | BPF [CORE] |
| 4808 | 4801 | M: Alexei Starovoitov <ast@kernel.org> |
| 4809 | 4802 | M: Daniel Borkmann <daniel@iogearbox.net> |
| @@ -4840,12 +4833,13 @@ BPF [GENERAL] (Safe Dynamic Programs and Tools) · 30 unmodified lines | ||
| 4840 | 4833 | M: Alexei Starovoitov <ast@kernel.org> |
| 4841 | 4834 | M: Daniel Borkmann <daniel@iogearbox.net> |
| 4842 | 4835 | M: Andrii Nakryiko <andrii@kernel.org> |
| 4843 | −R: Martin KaFai Lau <martin.lau@linux.dev> | |
| 4844 | 4836 | M: Eduard Zingerman <eddyz87@gmail.com> |
| 4845 | 4837 | M: Kumar Kartikeya Dwivedi <memxor@gmail.com> |
| 4838 | +R: Martin KaFai Lau <martin.lau@linux.dev> | |
| 4846 | 4839 | R: Song Liu <song@kernel.org> |
| 4847 | 4840 | R: Yonghong Song <yonghong.song@linux.dev> |
| 4848 | 4841 | R: Jiri Olsa <jolsa@kernel.org> |
| 4842 | +R: Emil Tsalapatis <emil@etsalapatis.com> | |
| 4849 | 4843 | L: bpf@vger.kernel.org |
| 4850 | 4844 | S: Supported |
| 4851 | 4845 | W: https://bpf.io/ |
| @@ -4855,7 +4849,9 @@ T: git git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git · 3 unmodified lines | ||
| 4855 | 4849 | F: Documentation/bpf/ |
| 4856 | 4850 | F: Documentation/networking/filter.rst |
| 4857 | 4851 | F: Documentation/userspace-api/ebpf/ |
| 4852 | +F: arch/*/include/asm/rqspinlock.h | |
| 4858 | 4853 | F: arch/*/net/* |
| 4854 | +F: include/asm-generic/rqspinlock.h | |
| 4859 | 4855 | F: include/linux/bpf* |
| 4860 | 4856 | F: include/linux/btf* |
| 4861 | 4857 | F: include/linux/buildid.h |
| @@ -4867,17 +4863,15 @@ F: include/uapi/linux/filter.h · 5 unmodified lines | ||
| 4867 | 4863 | F: kernel/bpf/ |
| 4868 | 4864 | F: kernel/trace/bpf_trace.c |
| 4869 | 4865 | F: lib/buildid.c |
| 4870 | −F: arch/*/include/asm/rqspinlock.h | |
| 4871 | −F: include/asm-generic/rqspinlock.h | |
| 4872 | 4866 | F: lib/test_bpf.c |
| 4873 | 4867 | F: net/bpf/ |
| 4874 | 4868 | F: net/core/filter.c |
| 4875 | 4869 | F: net/sched/act_bpf.c |
| 4876 | 4870 | F: net/sched/cls_bpf.c |
| 4877 | 4871 | F: samples/bpf/ |
| 4872 | +F: scripts/Makefile.btf | |
| 4878 | 4873 | F: scripts/bpf_doc.py |
| 4879 | 4874 | F: scripts/gen-btf.sh |
| 4880 | −F: scripts/Makefile.btf | |
| 4881 | 4875 | F: scripts/pahole-version.sh |
| 4882 | 4876 | F: tools/bpf/ |
| 4883 | 4877 | F: tools/lib/bpf/ |
| @@ -4892,6 +4886,7 @@ F: kernel/bpf/*iter.c · 8 unmodified lines | ||
| 4892 | 4886 | BPF [L7 FRAMEWORK] (sockmap) |
| 4893 | 4887 | M: John Fastabend <john.fastabend@gmail.com> |
| 4894 | 4888 | M: Jakub Sitnicki <jakub@cloudflare.com> |
| 4889 | +M: Jiayuan Chen <jiayuan.chen@linux.dev> | |
| 4895 | 4890 | L: netdev@vger.kernel.org |
| 4896 | 4891 | L: bpf@vger.kernel.org |
| 4897 | 4892 | S: Maintained |
| @@ -4935,18 +4930,11 @@ F: tools/testing/selftests/bpf/prog_tests/tc_netkit.c · 37 unmodified lines | ||
| 4935 | 4930 | F: tools/testing/selftests/drivers/net/hw/nk_qlease.py |
| 4936 | 4931 | F: tools/testing/selftests/net/nk_qlease.py |
| 4937 | 4932 | |
| 4938 | −BPF [NETWORKING] (struct_ops, reuseport) | |
| 4939 | −M: Martin KaFai Lau <martin.lau@linux.dev> | |
| 4940 | −L: bpf@vger.kernel.org | |
| 4941 | −L: netdev@vger.kernel.org | |
| 4942 | −S: Maintained | |
| 4943 | −F: kernel/bpf/bpf_struct* | |
| 4944 | − | |
| 4945 | 4933 | BPF [NETWORKING] (tcx & tc BPF, sock_addr) |
| 4946 | −M: Martin KaFai Lau <martin.lau@linux.dev> | |
| 4947 | 4934 | M: Daniel Borkmann <daniel@iogearbox.net> |
| 4948 | 4935 | R: John Fastabend <john.fastabend@gmail.com> |
| 4949 | 4936 | R: Stanislav Fomichev <sdf@fomichev.me> |
| 4937 | +R: Martin KaFai Lau <martin.lau@linux.dev> | |
| 4950 | 4938 | L: bpf@vger.kernel.org |
| 4951 | 4939 | L: netdev@vger.kernel.org |
| 4952 | 4940 | S: Maintained |
| @@ -4981,14 +4969,6 @@ L: bpf@vger.kernel.org · 28 unmodified lines | ||
| 4981 | 4969 | S: Maintained |
| 4982 | 4970 | F: tools/testing/selftests/bpf/ |
| 4983 | 4971 | |
| 4984 | −BPF [STORAGE & CGROUPS] | |
| 4985 | −M: Martin KaFai Lau <martin.lau@linux.dev> | |
| 4986 | −L: bpf@vger.kernel.org | |
| 4987 | −S: Maintained | |
| 4988 | −F: kernel/bpf/*storage.c | |
| 4989 | −F: kernel/bpf/bpf_lru* | |
| 4990 | −F: kernel/bpf/cgroup.c | |
| 4991 | − | |
| 4992 | 4972 | BPF [TOOLING] (bpftool) |
| 4993 | 4973 | M: Quentin Monnet <qmo@kernel.org> |
| 4994 | 4974 | L: bpf@vger.kernel.org |
| @@ -1830,6 +1830,34 @@ static inline pte_t ptep_get_and_clear(struct mm_struct *mm, · 1829 unmodified lines | ||
| 1830 | 1830 | return __ptep_get_and_clear(mm, addr, ptep); |
| 1831 | 1831 | } |
| 1832 | 1832 | |
| 1833 | +static inline bool ptep_try_set(pte_t *ptep, pte_t new_pte) | |
| 1834 | +{ | |
| 1835 | + pteval_t old = 0; | |
| 1836 | + | |
| 1837 | + if (!try_cmpxchg(&pte_val(*ptep), &old, pte_val(new_pte))) | |
| 1838 | + return false; | |
| 1839 | + | |
| 1840 | + /* | |
| 1841 | + * The store must be complete by the time this returns, but the caller | |
| 1842 | + * may be in lazy MMU mode, where __set_pte_complete() would defer the | |
| 1843 | + * barriers. Issue them directly. | |
| 1844 | + */ | |
| 1845 | + emit_pte_barriers(); | |
| 1846 | + return true; | |
| 1847 | +} | |
| 1848 | +#define ptep_try_set ptep_try_set | |
| 1849 | + | |
| 1850 | +/* | |
| 1851 | + * arm64 mandates break-before-make: a cleared kernel PTE must have its TLB | |
| 1852 | + * invalidated before a different page is installed in its place. The broadcast | |
| 1853 | + * TLBI is an instruction, not an IPI, so this is safe with interrupts disabled. | |
| 1854 | + */ | |
| 1855 | +static inline void flush_tlb_before_set(unsigned long addr) | |
| 1856 | +{ | |
| 1857 | + flush_tlb_kernel_range(addr, addr + PAGE_SIZE); | |
| 1858 | +} | |
| 1859 | +#define flush_tlb_before_set flush_tlb_before_set | |
| 1860 | + | |
| 1833 | 1861 | #define test_and_clear_young_ptes test_and_clear_young_ptes |
| 1834 | 1862 | static inline bool test_and_clear_young_ptes(struct vm_area_struct *vma, |
| 1835 | 1863 | unsigned long addr, pte_t *ptep, unsigned int nr) |
| @@ -9,6 +9,7 @@ · 8 unmodified lines | ||
| 9 | 9 | |
| 10 | 10 | #include <linux/acpi.h> |
| 11 | 11 | #include <linux/bitfield.h> |
| 12 | +#include <linux/bpf_defs.h> | |
| 12 | 13 | #include <linux/extable.h> |
| 13 | 14 | #include <linux/kfence.h> |
| 14 | 15 | #include <linux/signal.h> |
| @@ -436,9 +437,12 @@ static void __do_kernel_fault(unsigned long addr, unsigned long esr, · 421 unmodified lines | ||
| 436 | 437 | } else if (is_pkvm_stage2_abort(esr)) { |
| 437 | 438 | msg = "access to hypervisor-protected memory"; |
| 438 | 439 | } else { |
| 439 | − if (esr_fsc_is_translation_fault(esr) && | |
| 440 | − kfence_handle_page_fault(addr, esr & ESR_ELx_WNR, regs)) | |
| 441 | − return; | |
| 440 | + if (esr_fsc_is_translation_fault(esr)) { | |
| 441 | + if (kfence_handle_page_fault(addr, esr & ESR_ELx_WNR, regs)) | |
| 442 | + return; | |
| 443 | + if (bpf_arena_handle_page_fault(addr, esr & ESR_ELx_WNR, regs->pc)) | |
| 444 | + return; | |
| 445 | + } | |
| 442 | 446 | |
| 443 | 447 | msg = "paging request"; |
| 444 | 448 | } |
| @@ -47,7 +47,7 @@ · 46 unmodified lines | ||
| 47 | 47 | /* Map BPF registers to A64 registers */ |
| 48 | 48 | static const int bpf2a64[] = { |
| 49 | 49 | /* return value from in-kernel function, and exit value from eBPF */ |
| 50 | − [BPF_REG_0] = A64_R(7), | |
| 50 | + [BPF_REG_0] = A64_R(8), | |
| 51 | 51 | /* arguments from eBPF program to in-kernel function */ |
| 52 | 52 | [BPF_REG_1] = A64_R(0), |
| 53 | 53 | [BPF_REG_2] = A64_R(1), |
| @@ -86,6 +86,7 @@ struct jit_ctx { · 32 unmodified lines | ||
| 86 | 86 | __le32 *image; |
| 87 | 87 | __le32 *ro_image; |
| 88 | 88 | u32 stack_size; |
| 89 | + u16 stack_arg_size; | |
| 89 | 90 | u64 user_vm_start; |
| 90 | 91 | u64 arena_vm_start; |
| 91 | 92 | bool fp_used; |
| @@ -533,13 +534,20 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf) · 441 unmodified lines | ||
| 533 | 534 | * | | |
| 534 | 535 | * +-----+ <= (BPF_FP - prog->aux->stack_depth) |
| 535 | 536 | * |RSVD | padding |
| 536 | − * current A64_SP => +-----+ <= (BPF_FP - ctx->stack_size) | |
| 537 | + * +-----+ <= (BPF_FP - ctx->stack_size) | |
| 538 | + * | | | |
| 539 | + * | ... | outgoing stack args (9+, if any) | |
| 540 | + * | | | |
| 541 | + * current A64_SP => +-----+ | |
| 537 | 542 | * | | |
| 538 | 543 | * | ... | Function call stack |
| 539 | 544 | * | | |
| 540 | 545 | * +-----+ |
| 541 | 546 | * low |
| 542 | 547 | * |
| 548 | + * Stack args 6-8 are passed in x5-x7, args 9+ at [SP]. | |
| 549 | + * Incoming args 9+ are at [A64_FP + 16], [A64_FP + 24], ... | |
| 550 | + * (above the saved FP/LR pair pushed in the callee prologue). | |
| 543 | 551 | */ |
| 544 | 552 | |
| 545 | 553 | emit_kcfi(is_main_prog ? cfi_bpf_hash : cfi_bpf_subprog_hash, ctx); |
| @@ -613,6 +621,9 @@ static int build_prologue(struct jit_ctx *ctx, bool ebpf_from_cbpf) · 67 unmodified lines | ||
| 613 | 621 | if (ctx->stack_size && !ctx->priv_sp_used) |
| 614 | 622 | emit(A64_SUB_I(1, A64_SP, A64_SP, ctx->stack_size), ctx); |
| 615 | 623 | |
| 624 | + if (ctx->stack_arg_size) | |
| 625 | + emit(A64_SUB_I(1, A64_SP, A64_SP, ctx->stack_arg_size), ctx); | |
| 626 | + | |
| 616 | 627 | if (ctx->arena_vm_start) |
| 617 | 628 | emit_a64_mov_i64(arena_vm_base, ctx->arena_vm_start, ctx); |
| 618 | 629 | |
| @@ -673,6 +684,9 @@ static int emit_bpf_tail_call(struct jit_ctx *ctx) · 54 unmodified lines | ||
| 673 | 684 | /* Update tail_call_cnt if the slot is populated. */ |
| 674 | 685 | emit(A64_STR64I(tcc, ptr, 0), ctx); |
| 675 | 686 | |
| 687 | + if (ctx->stack_arg_size) | |
| 688 | + emit(A64_ADD_I(1, A64_SP, A64_SP, ctx->stack_arg_size), ctx); | |
| 689 | + | |
| 676 | 690 | /* restore SP */ |
| 677 | 691 | if (ctx->stack_size && !ctx->priv_sp_used) |
| 678 | 692 | emit(A64_ADD_I(1, A64_SP, A64_SP, ctx->stack_size), ctx); |
| @@ -1034,6 +1048,9 @@ static void build_epilogue(struct jit_ctx *ctx, bool was_classic) · 355 unmodified lines | ||
| 1034 | 1048 | const u8 r0 = bpf2a64[BPF_REG_0]; |
| 1035 | 1049 | const u8 ptr = bpf2a64[TCCNT_PTR]; |
| 1036 | 1050 | |
| 1051 | + if (ctx->stack_arg_size) | |
| 1052 | + emit(A64_ADD_I(1, A64_SP, A64_SP, ctx->stack_arg_size), ctx); | |
| 1053 | + | |
| 1037 | 1054 | /* We're done with BPF stack */ |
| 1038 | 1055 | if (ctx->stack_size && !ctx->priv_sp_used) |
| 1039 | 1056 | emit(A64_ADD_I(1, A64_SP, A64_SP, ctx->stack_size), ctx); |
| @@ -1048,7 +1065,7 @@ static void build_epilogue(struct jit_ctx *ctx, bool was_classic) · 8 unmodified lines | ||
| 1048 | 1065 | /* Restore FP/LR registers */ |
| 1049 | 1066 | emit(A64_POP(A64_FP, A64_LR, A64_SP), ctx); |
| 1050 | 1067 | |
| 1051 | − /* Move the return value from bpf:r0 (aka x7) to x0 */ | |
| 1068 | + /* Move the return value from bpf:r0 (aka x8) to x0 */ | |
| 1052 | 1069 | emit(A64_MOV(1, A64_R(0), r0), ctx); |
| 1053 | 1070 | |
| 1054 | 1071 | /* Authenticate lr */ |
| @@ -1191,6 +1208,42 @@ static int add_exception_handler(const struct bpf_insn *insn, · 136 unmodified lines | ||
| 1191 | 1208 | return 0; |
| 1192 | 1209 | } |
| 1193 | 1210 | |
| 1211 | +static const u8 stack_arg_reg[] = { A64_R(5), A64_R(6), A64_R(7) }; | |
| 1212 | + | |
| 1213 | +#define NR_STACK_ARG_REGS ARRAY_SIZE(stack_arg_reg) | |
| 1214 | + | |
| 1215 | +static void emit_stack_arg_load(u8 dst, s16 bpf_off, struct jit_ctx *ctx) | |
| 1216 | +{ | |
| 1217 | + int idx = bpf_off / sizeof(u64) - 1; | |
| 1218 | + | |
| 1219 | + if (idx < NR_STACK_ARG_REGS) | |
| 1220 | + emit(A64_MOV(1, dst, stack_arg_reg[idx]), ctx); | |
| 1221 | + else | |
| 1222 | + emit(A64_LDR64I(dst, A64_FP, (idx - NR_STACK_ARG_REGS) * sizeof(u64) + 16), ctx); | |
| 1223 | +} | |
| 1224 | + | |
| 1225 | +static void emit_stack_arg_store(u8 src_a64, s16 bpf_off, struct jit_ctx *ctx) | |
| 1226 | +{ | |
| 1227 | + int idx = -bpf_off / sizeof(u64) - 1; | |
| 1228 | + | |
| 1229 | + if (idx < NR_STACK_ARG_REGS) | |
| 1230 | + emit(A64_MOV(1, stack_arg_reg[idx], src_a64), ctx); | |
| 1231 | + else | |
| 1232 | + emit(A64_STR64I(src_a64, A64_SP, (idx - NR_STACK_ARG_REGS) * sizeof(u64)), ctx); | |
| 1233 | +} | |
| 1234 | + | |
| 1235 | +static void emit_stack_arg_store_imm(s32 imm, s16 bpf_off, const u8 tmp, struct jit_ctx *ctx) | |
| 1236 | +{ | |
| 1237 | + int idx = -bpf_off / sizeof(u64) - 1; | |
| 1238 | + | |
| 1239 | + if (idx < NR_STACK_ARG_REGS) { | |
| 1240 | + emit_a64_mov_i(1, stack_arg_reg[idx], imm, ctx); | |
| 1241 | + } else { | |
| 1242 | + emit_a64_mov_i(1, tmp, imm, ctx); | |
| 1243 | + emit(A64_STR64I(tmp, A64_SP, (idx - NR_STACK_ARG_REGS) * sizeof(u64)), ctx); | |
| 1244 | + } | |
| 1245 | +} | |
| 1246 | + | |
| 1194 | 1247 | /* JITs an eBPF instruction. |
| 1195 | 1248 | * Returns: |
| 1196 | 1249 | * 0 - successfully JITed an 8-byte eBPF instruction. |
| @@ -1646,6 +1699,11 @@ emit_cond_jmp: · 449 unmodified lines | ||
| 1646 | 1699 | case BPF_LDX | BPF_MEM | BPF_H: |
| 1647 | 1700 | case BPF_LDX | BPF_MEM | BPF_B: |
| 1648 | 1701 | case BPF_LDX | BPF_MEM | BPF_DW: |
| 1702 | + if (insn->src_reg == BPF_REG_PARAMS) { | |
| 1703 | + emit_stack_arg_load(dst, off, ctx); | |
| 1704 | + break; | |
| 1705 | + } | |
| 1706 | + fallthrough; | |
| 1649 | 1707 | case BPF_LDX | BPF_PROBE_MEM | BPF_DW: |
| 1650 | 1708 | case BPF_LDX | BPF_PROBE_MEM | BPF_W: |
| 1651 | 1709 | case BPF_LDX | BPF_PROBE_MEM | BPF_H: |
| @@ -1672,6 +1730,8 @@ emit_cond_jmp: · 20 unmodified lines | ||
| 1672 | 1730 | if (src == fp) { |
| 1673 | 1731 | src_adj = ctx->priv_sp_used ? priv_sp : A64_SP; |
| 1674 | 1732 | off_adj = off + ctx->stack_size; |
| 1733 | + if (!ctx->priv_sp_used) | |
| 1734 | + off_adj += ctx->stack_arg_size; | |
| 1675 | 1735 | } else { |
| 1676 | 1736 | src_adj = src; |
| 1677 | 1737 | off_adj = off; |
| @@ -1752,6 +1812,11 @@ emit_cond_jmp: · 74 unmodified lines | ||
| 1752 | 1812 | case BPF_ST | BPF_MEM | BPF_H: |
| 1753 | 1813 | case BPF_ST | BPF_MEM | BPF_B: |
| 1754 | 1814 | case BPF_ST | BPF_MEM | BPF_DW: |
| 1815 | + if (insn->dst_reg == BPF_REG_PARAMS) { | |
| 1816 | + emit_stack_arg_store_imm(imm, off, tmp, ctx); | |
| 1817 | + break; | |
| 1818 | + } | |
| 1819 | + fallthrough; | |
| 1755 | 1820 | case BPF_ST | BPF_PROBE_MEM32 | BPF_B: |
| 1756 | 1821 | case BPF_ST | BPF_PROBE_MEM32 | BPF_H: |
| 1757 | 1822 | case BPF_ST | BPF_PROBE_MEM32 | BPF_W: |
| @@ -1763,6 +1828,8 @@ emit_cond_jmp: · 5 unmodified lines | ||
| 1763 | 1828 | if (dst == fp) { |
| 1764 | 1829 | dst_adj = ctx->priv_sp_used ? priv_sp : A64_SP; |
| 1765 | 1830 | off_adj = off + ctx->stack_size; |
| 1831 | + if (!ctx->priv_sp_used) | |
| 1832 | + off_adj += ctx->stack_arg_size; | |
| 1766 | 1833 | } else { |
| 1767 | 1834 | dst_adj = dst; |
| 1768 | 1835 | off_adj = off; |
| @@ -1814,6 +1881,11 @@ emit_cond_jmp: · 45 unmodified lines | ||
| 1814 | 1881 | case BPF_STX | BPF_MEM | BPF_H: |
| 1815 | 1882 | case BPF_STX | BPF_MEM | BPF_B: |
| 1816 | 1883 | case BPF_STX | BPF_MEM | BPF_DW: |
| 1884 | + if (insn->dst_reg == BPF_REG_PARAMS) { | |
| 1885 | + emit_stack_arg_store(src, off, ctx); | |
| 1886 | + break; | |
| 1887 | + } | |
| 1888 | + fallthrough; | |
| 1817 | 1889 | case BPF_STX | BPF_PROBE_MEM32 | BPF_B: |
| 1818 | 1890 | case BPF_STX | BPF_PROBE_MEM32 | BPF_H: |
| 1819 | 1891 | case BPF_STX | BPF_PROBE_MEM32 | BPF_W: |
| @@ -1825,6 +1897,8 @@ emit_cond_jmp: · 5 unmodified lines | ||
| 1825 | 1897 | if (dst == fp) { |
| 1826 | 1898 | dst_adj = ctx->priv_sp_used ? priv_sp : A64_SP; |
| 1827 | 1899 | off_adj = off + ctx->stack_size; |
| 1900 | + if (!ctx->priv_sp_used) | |
| 1901 | + off_adj += ctx->stack_arg_size; | |
| 1828 | 1902 | } else { |
| 1829 | 1903 | dst_adj = dst; |
| 1830 | 1904 | off_adj = off; |
| @@ -2018,6 +2092,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr · 187 unmodified lines | ||
| 2018 | 2092 | u8 *ro_image_ptr; |
| 2019 | 2093 | int body_idx; |
| 2020 | 2094 | int exentry_idx; |
| 2095 | + int out_cnt; | |
| 2021 | 2096 | |
| 2022 | 2097 | if (!prog->jit_requested) |
| 2023 | 2098 | return prog; |
| @@ -2065,6 +2140,14 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_pr · 41 unmodified lines | ||
| 2065 | 2140 | ctx.user_vm_start = bpf_arena_get_user_vm_start(prog->aux->arena); |
| 2066 | 2141 | ctx.arena_vm_start = bpf_arena_get_kern_vm_start(prog->aux->arena); |
| 2067 | 2142 | |
| 2143 | + out_cnt = bpf_out_stack_arg_cnt(env, prog); | |
| 2144 | + if (out_cnt) { | |
| 2145 | + int nr_on_stack = out_cnt - NR_STACK_ARG_REGS; | |
| 2146 | + | |
| 2147 | + if (nr_on_stack > 0) | |
| 2148 | + ctx.stack_arg_size = round_up(nr_on_stack * sizeof(u64), 16); | |
| 2149 | + } | |
| 2150 | + | |
| 2068 | 2151 | if (priv_stack_ptr) |
| 2069 | 2152 | ctx.priv_sp_used = true; |
| 2070 | 2153 | |
| @@ -2229,6 +2312,11 @@ bool bpf_jit_supports_kfunc_call(void) · 158 unmodified lines | ||
| 2229 | 2312 | return true; |
| 2230 | 2313 | } |
| 2231 | 2314 | |
| 2315 | +bool bpf_jit_supports_stack_args(void) | |
| 2316 | +{ | |
| 2317 | + return true; | |
| 2318 | +} | |
| 2319 | + | |
| 2232 | 2320 | void *bpf_arch_text_copy(void *dst, void *src, size_t len) |
| 2233 | 2321 | { |
| 2234 | 2322 | if (!aarch64_insn_copy(dst, src, len)) |
| @@ -2247,24 +2335,24 @@ bool bpf_jit_supports_subprog_tailcalls(void) · 12 unmodified lines | ||
| 2247 | 2335 | return true; |
| 2248 | 2336 | } |
| 2249 | 2337 | |
| 2250 | −static void invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_link *l, | |
| 2338 | +static void invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_node *node, | |
| 2251 | 2339 | int bargs_off, int retval_off, int run_ctx_off, |
| 2252 | 2340 | bool save_ret) |
| 2253 | 2341 | { |
| 2254 | 2342 | __le32 *branch; |
| 2255 | 2343 | u64 enter_prog; |
| 2256 | 2344 | u64 exit_prog; |
| 2257 | − struct bpf_prog *p = l->link.prog; | |
| 2345 | + struct bpf_prog *p = node->link->prog; | |
| 2258 | 2346 | int cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie); |
| 2259 | 2347 | |
| 2260 | 2348 | enter_prog = (u64)bpf_trampoline_enter(p); |
| 2261 | 2349 | exit_prog = (u64)bpf_trampoline_exit(p); |
| 2262 | 2350 | |
| 2263 | − if (l->cookie == 0) { | |
| 2351 | + if (node->cookie == 0) { | |
| 2264 | 2352 | /* if cookie is zero, one instruction is enough to store it */ |
| 2265 | 2353 | emit(A64_STR64I(A64_ZR, A64_SP, run_ctx_off + cookie_off), ctx); |
| 2266 | 2354 | } else { |
| 2267 | − emit_a64_mov_i64(A64_R(10), l->cookie, ctx); | |
| 2355 | + emit_a64_mov_i64(A64_R(10), node->cookie, ctx); | |
| 2268 | 2356 | emit(A64_STR64I(A64_R(10), A64_SP, run_ctx_off + cookie_off), |
| 2269 | 2357 | ctx); |
| 2270 | 2358 | } |
| @@ -2314,7 +2402,7 @@ static void invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_link *l, · 43 unmodified lines | ||
| 2314 | 2402 | emit_call(exit_prog, ctx); |
| 2315 | 2403 | } |
| 2316 | 2404 | |
| 2317 | −static void invoke_bpf_mod_ret(struct jit_ctx *ctx, struct bpf_tramp_links *tl, | |
| 2405 | +static void invoke_bpf_mod_ret(struct jit_ctx *ctx, struct bpf_tramp_nodes *tn, | |
| 2318 | 2406 | int bargs_off, int retval_off, int run_ctx_off, |
| 2319 | 2407 | __le32 **branches) |
| 2320 | 2408 | { |
| @@ -2324,8 +2412,8 @@ static void invoke_bpf_mod_ret(struct jit_ctx *ctx, struct bpf_tramp_links *tl, · 3 unmodified lines | ||
| 2324 | 2412 | * Set this to 0 to avoid confusing the program. |
| 2325 | 2413 | */ |
| 2326 | 2414 | emit(A64_STR64I(A64_ZR, A64_SP, retval_off), ctx); |
| 2327 | − for (i = 0; i < tl->nr_links; i++) { | |
| 2328 | − invoke_bpf_prog(ctx, tl->links[i], bargs_off, retval_off, | |
| 2415 | + for (i = 0; i < tn->nr_nodes; i++) { | |
| 2416 | + invoke_bpf_prog(ctx, tn->nodes[i], bargs_off, retval_off, | |
| 2329 | 2417 | run_ctx_off, true); |
| 2330 | 2418 | /* if (*(u64 *)(sp + retval_off) != 0) |
| 2331 | 2419 | * goto do_fexit; |
| @@ -2456,10 +2544,10 @@ static void restore_args(struct jit_ctx *ctx, int bargs_off, int nregs) · 124 unmodified lines | ||
| 2456 | 2544 | } |
| 2457 | 2545 | } |
| 2458 | 2546 | |
| 2459 | −static bool is_struct_ops_tramp(const struct bpf_tramp_links *fentry_links) | |
| 2547 | +static bool is_struct_ops_tramp(const struct bpf_tramp_nodes *fentry_nodes) | |
| 2460 | 2548 | { |
| 2461 | − return fentry_links->nr_links == 1 && | |
| 2462 | − fentry_links->links[0]->link.type == BPF_LINK_TYPE_STRUCT_OPS; | |
| 2549 | + return fentry_nodes->nr_nodes == 1 && | |
| 2550 | + fentry_nodes->nodes[0]->link->type == BPF_LINK_TYPE_STRUCT_OPS; | |
| 2463 | 2551 | } |
| 2464 | 2552 | |
| 2465 | 2553 | static void store_func_meta(struct jit_ctx *ctx, u64 func_meta, int func_meta_off) |
| @@ -2480,7 +2568,7 @@ static void store_func_meta(struct jit_ctx *ctx, u64 func_meta, int func_meta_of · 14 unmodified lines | ||
| 2480 | 2568 | * |
| 2481 | 2569 | */ |
| 2482 | 2570 | static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im, |
| 2483 | − struct bpf_tramp_links *tlinks, void *func_addr, | |
| 2571 | + struct bpf_tramp_nodes *tnodes, void *func_addr, | |
| 2484 | 2572 | const struct btf_func_model *m, |
| 2485 | 2573 | const struct arg_aux *a, |
| 2486 | 2574 | u32 flags) |
| @@ -2496,14 +2584,14 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im, · 9 unmodified lines | ||
| 2496 | 2584 | int run_ctx_off; |
| 2497 | 2585 | int oargs_off; |
| 2498 | 2586 | int nfuncargs; |
| 2499 | − struct bpf_tramp_links *fentry = &tlinks[BPF_TRAMP_FENTRY]; | |
| 2500 | − struct bpf_tramp_links *fexit = &tlinks[BPF_TRAMP_FEXIT]; | |
| 2501 | − struct bpf_tramp_links *fmod_ret = &tlinks[BPF_TRAMP_MODIFY_RETURN]; | |
| 2587 | + struct bpf_tramp_nodes *fentry = &tnodes[BPF_TRAMP_FENTRY]; | |
| 2588 | + struct bpf_tramp_nodes *fexit = &tnodes[BPF_TRAMP_FEXIT]; | |
| 2589 | + struct bpf_tramp_nodes *fmod_ret = &tnodes[BPF_TRAMP_MODIFY_RETURN]; | |
| 2502 | 2590 | bool save_ret; |
| 2503 | 2591 | __le32 **branches = NULL; |
| 2504 | 2592 | bool is_struct_ops = is_struct_ops_tramp(fentry); |
| 2505 | 2593 | int cookie_off, cookie_cnt, cookie_bargs_off; |
| 2506 | − int fsession_cnt = bpf_fsession_cnt(tlinks); | |
| 2594 | + int fsession_cnt = bpf_fsession_cnt(tnodes); | |
| 2507 | 2595 | u64 func_meta; |
| 2508 | 2596 | |
| 2509 | 2597 | /* trampoline stack layout: |
| @@ -2549,7 +2637,7 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im, · 39 unmodified lines | ||
| 2549 | 2637 | |
| 2550 | 2638 | cookie_off = stack_size; |
| 2551 | 2639 | /* room for session cookies */ |
| 2552 | − cookie_cnt = bpf_fsession_cookie_cnt(tlinks); | |
| 2640 | + cookie_cnt = bpf_fsession_cookie_cnt(tnodes); | |
| 2553 | 2641 | stack_size += cookie_cnt * 8; |
| 2554 | 2642 | |
| 2555 | 2643 | ip_off = stack_size; |
| @@ -2646,20 +2734,20 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im, · 90 unmodified lines | ||
| 2646 | 2734 | } |
| 2647 | 2735 | |
| 2648 | 2736 | cookie_bargs_off = (bargs_off - cookie_off) / 8; |
| 2649 | − for (i = 0; i < fentry->nr_links; i++) { | |
| 2650 | − if (bpf_prog_calls_session_cookie(fentry->links[i])) { | |
| 2737 | + for (i = 0; i < fentry->nr_nodes; i++) { | |
| 2738 | + if (bpf_prog_calls_session_cookie(fentry->nodes[i])) { | |
| 2651 | 2739 | u64 meta = func_meta | (cookie_bargs_off << BPF_TRAMP_COOKIE_INDEX_SHIFT); |
| 2652 | 2740 | |
| 2653 | 2741 | store_func_meta(ctx, meta, func_meta_off); |
| 2654 | 2742 | cookie_bargs_off--; |
| 2655 | 2743 | } |
| 2656 | − invoke_bpf_prog(ctx, fentry->links[i], bargs_off, | |
| 2744 | + invoke_bpf_prog(ctx, fentry->nodes[i], bargs_off, | |
| 2657 | 2745 | retval_off, run_ctx_off, |
| 2658 | 2746 | flags & BPF_TRAMP_F_RET_FENTRY_RET); |
| 2659 | 2747 | } |
| 2660 | 2748 | |
| 2661 | − if (fmod_ret->nr_links) { | |
| 2662 | − branches = kcalloc(fmod_ret->nr_links, sizeof(__le32 *), | |
| 2749 | + if (fmod_ret->nr_nodes) { | |
| 2750 | + branches = kcalloc(fmod_ret->nr_nodes, sizeof(__le32 *), | |
| 2663 | 2751 | GFP_KERNEL); |
| 2664 | 2752 | if (!branches) |
| 2665 | 2753 | return -ENOMEM; |
| @@ -2683,7 +2771,7 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im, · 17 unmodified lines | ||
| 2683 | 2771 | } |
| 2684 | 2772 | |
| 2685 | 2773 | /* update the branches saved in invoke_bpf_mod_ret with cbnz */ |
| 2686 | − for (i = 0; i < fmod_ret->nr_links && ctx->image != NULL; i++) { | |
| 2774 | + for (i = 0; i < fmod_ret->nr_nodes && ctx->image != NULL; i++) { | |
| 2687 | 2775 | int offset = &ctx->image[ctx->idx] - branches[i]; |
| 2688 | 2776 | *branches[i] = cpu_to_le32(A64_CBNZ(1, A64_R(10), offset)); |
| 2689 | 2777 | } |
| @@ -2694,14 +2782,14 @@ static int prepare_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im, · 4 unmodified lines | ||
| 2694 | 2782 | store_func_meta(ctx, func_meta, func_meta_off); |
| 2695 | 2783 | |
| 2696 | 2784 | cookie_bargs_off = (bargs_off - cookie_off) / 8; |
| 2697 | − for (i = 0; i < fexit->nr_links; i++) { | |
| 2698 | − if (bpf_prog_calls_session_cookie(fexit->links[i])) { | |
| 2785 | + for (i = 0; i < fexit->nr_nodes; i++) { | |
| 2786 | + if (bpf_prog_calls_session_cookie(fexit->nodes[i])) { | |
| 2699 | 2787 | u64 meta = func_meta | (cookie_bargs_off << BPF_TRAMP_COOKIE_INDEX_SHIFT); |
| 2700 | 2788 | |
| 2701 | 2789 | store_func_meta(ctx, meta, func_meta_off); |
| 2702 | 2790 | cookie_bargs_off--; |
| 2703 | 2791 | } |
| 2704 | − invoke_bpf_prog(ctx, fexit->links[i], bargs_off, retval_off, | |
| 2792 | + invoke_bpf_prog(ctx, fexit->nodes[i], bargs_off, retval_off, | |
| 2705 | 2793 | run_ctx_off, false); |
| 2706 | 2794 | } |
| 2707 | 2795 | |
| @@ -2759,7 +2847,7 @@ bool bpf_jit_supports_fsession(void) · 51 unmodified lines | ||
| 2759 | 2847 | } |
| 2760 | 2848 | |
| 2761 | 2849 | int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, |
| 2762 | − struct bpf_tramp_links *tlinks, void *func_addr) | |
| 2850 | + struct bpf_tramp_nodes *tnodes, void *func_addr) | |
| 2763 | 2851 | { |
| 2764 | 2852 | struct jit_ctx ctx = { |
| 2765 | 2853 | .image = NULL, |
| @@ -2773,7 +2861,7 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, · 7 unmodified lines | ||
| 2773 | 2861 | if (ret < 0) |
| 2774 | 2862 | return ret; |
| 2775 | 2863 | |
| 2776 | − ret = prepare_trampoline(&ctx, &im, tlinks, func_addr, m, &aaux, flags); | |
| 2864 | + ret = prepare_trampoline(&ctx, &im, tnodes, func_addr, m, &aaux, flags); | |
| 2777 | 2865 | if (ret < 0) |
| 2778 | 2866 | return ret; |
| 2779 | 2867 | |
| @@ -2797,7 +2885,7 @@ int arch_protect_bpf_trampoline(void *image, unsigned int size) · 17 unmodified lines | ||
| 2797 | 2885 | |
| 2798 | 2886 | int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image, |
| 2799 | 2887 | void *ro_image_end, const struct btf_func_model *m, |
| 2800 | − u32 flags, struct bpf_tramp_links *tlinks, | |
| 2888 | + u32 flags, struct bpf_tramp_nodes *tnodes, | |
| 2801 | 2889 | void *func_addr) |
| 2802 | 2890 | { |
| 2803 | 2891 | u32 size = ro_image_end - ro_image; |
| @@ -2824,7 +2912,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image, · 20 unmodified lines | ||
| 2824 | 2912 | ret = calc_arg_aux(m, &aaux); |
| 2825 | 2913 | if (ret) |
| 2826 | 2914 | goto out; |
| 2827 | − ret = prepare_trampoline(&ctx, im, tlinks, func_addr, m, &aaux, flags); | |
| 2915 | + ret = prepare_trampoline(&ctx, im, tnodes, func_addr, m, &aaux, flags); | |
| 2828 | 2916 | |
| 2829 | 2917 | if (ret > 0 && validate_code(&ctx) < 0) { |
| 2830 | 2918 | ret = -EINVAL; |
| @@ -8,8 +8,8 @@ SYM_FUNC_START(arch_bpf_timed_may_goto) · 7 unmodified lines | ||
| 8 | 8 | stp x29, x30, [sp, #-64]! |
| 9 | 9 | mov x29, sp |
| 10 | 10 | |
| 11 | − /* Save BPF registers R0 - R5 (x7, x0-x4)*/ | |
| 12 | − stp x7, x0, [sp, #16] | |
| 11 | + /* Save BPF registers R0 - R5 (x8, x0-x4)*/ | |
| 12 | + stp x8, x0, [sp, #16] | |
| 13 | 13 | stp x1, x2, [sp, #32] |
| 14 | 14 | stp x3, x4, [sp, #48] |
| 15 | 15 | |
| @@ -28,8 +28,8 @@ SYM_FUNC_START(arch_bpf_timed_may_goto) · 12 unmodified lines | ||
| 28 | 28 | /* BPF_REG_AX(x9) will be stored into count, so move return value to it. */ |
| 29 | 29 | mov x9, x0 |
| 30 | 30 | |
| 31 | − /* Restore BPF registers R0 - R5 (x7, x0-x4) */ | |
| 32 | − ldp x7, x0, [sp, #16] | |
| 31 | + /* Restore BPF registers R0 - R5 (x8, x0-x4) */ | |
| 32 | + ldp x8, x0, [sp, #16] | |
| 33 | 33 | ldp x1, x2, [sp, #32] |
| 34 | 34 | ldp x3, x4, [sp, #48] |
| 35 | 35 | |
| @@ -1674,17 +1674,17 @@ static void restore_stk_args(struct jit_ctx *ctx, int nr_stk_args, int args_off, · 1673 unmodified lines | ||
| 1674 | 1674 | } |
| 1675 | 1675 | } |
| 1676 | 1676 | |
| 1677 | −static int invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_link *l, | |
| 1677 | +static int invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_node *n, | |
| 1678 | 1678 | int args_off, int retval_off, int run_ctx_off, bool save_ret) |
| 1679 | 1679 | { |
| 1680 | 1680 | int ret; |
| 1681 | 1681 | u32 *branch; |
| 1682 | − struct bpf_prog *p = l->link.prog; | |
| 1682 | + struct bpf_prog *p = n->link->prog; | |
| 1683 | 1683 | int cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie); |
| 1684 | 1684 | |
| 1685 | − if (l->cookie) | |
| 1685 | + if (n->cookie) | |
| 1686 | 1686 | emit_store_stack_imm64(ctx, LOONGARCH_GPR_T1, |
| 1687 | − -run_ctx_off + cookie_off, l->cookie); | |
| 1687 | + -run_ctx_off + cookie_off, n->cookie); | |
| 1688 | 1688 | else |
| 1689 | 1689 | emit_insn(ctx, std, LOONGARCH_GPR_ZERO, LOONGARCH_GPR_FP, -run_ctx_off + cookie_off); |
| 1690 | 1690 | |
| @@ -1737,22 +1737,22 @@ static int invoke_bpf_prog(struct jit_ctx *ctx, struct bpf_tramp_link *l, · 46 unmodified lines | ||
| 1737 | 1737 | return ret; |
| 1738 | 1738 | } |
| 1739 | 1739 | |
| 1740 | −static int invoke_bpf(struct jit_ctx *ctx, struct bpf_tramp_links *tl, | |
| 1740 | +static int invoke_bpf(struct jit_ctx *ctx, struct bpf_tramp_nodes *tn, | |
| 1741 | 1741 | int args_off, int retval_off, int run_ctx_off, |
| 1742 | 1742 | int func_meta_off, bool save_ret, u64 func_meta, int cookie_off) |
| 1743 | 1743 | { |
| 1744 | 1744 | int i, cur_cookie = (cookie_off - args_off) / 8; |
| 1745 | 1745 | |
| 1746 | − for (i = 0; i < tl->nr_links; i++) { | |
| 1746 | + for (i = 0; i < tn->nr_nodes; i++) { | |
| 1747 | 1747 | int err; |
| 1748 | 1748 | |
| 1749 | − if (bpf_prog_calls_session_cookie(tl->links[i])) { | |
| 1749 | + if (bpf_prog_calls_session_cookie(tn->nodes[i])) { | |
| 1750 | 1750 | u64 meta = func_meta | ((u64)cur_cookie << BPF_TRAMP_COOKIE_INDEX_SHIFT); |
| 1751 | 1751 | |
| 1752 | 1752 | emit_store_stack_imm64(ctx, LOONGARCH_GPR_T1, -func_meta_off, meta); |
| 1753 | 1753 | cur_cookie--; |
| 1754 | 1754 | } |
| 1755 | − err = invoke_bpf_prog(ctx, tl->links[i], args_off, retval_off, run_ctx_off, save_ret); | |
| 1755 | + err = invoke_bpf_prog(ctx, tn->nodes[i], args_off, retval_off, run_ctx_off, save_ret); | |
| 1756 | 1756 | if (err) |
| 1757 | 1757 | return err; |
| 1758 | 1758 | } |
| @@ -1807,7 +1807,7 @@ static void sign_extend(struct jit_ctx *ctx, int rd, int rj, u8 size, bool sign) · 48 unmodified lines | ||
| 1807 | 1807 | } |
| 1808 | 1808 | |
| 1809 | 1809 | static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_image *im, |
| 1810 | − const struct btf_func_model *m, struct bpf_tramp_links *tlinks, | |
| 1810 | + const struct btf_func_model *m, struct bpf_tramp_nodes *tnodes, | |
| 1811 | 1811 | void *func_addr, u32 flags) |
| 1812 | 1812 | { |
| 1813 | 1813 | int i, ret, save_ret; |
| @@ -1817,9 +1817,9 @@ static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_i · 3 unmodified lines | ||
| 1817 | 1817 | unsigned long long func_meta; |
| 1818 | 1818 | bool is_struct_ops = flags & BPF_TRAMP_F_INDIRECT; |
| 1819 | 1819 | void *orig_call = func_addr; |
| 1820 | − struct bpf_tramp_links *fentry = &tlinks[BPF_TRAMP_FENTRY]; | |
| 1821 | − struct bpf_tramp_links *fexit = &tlinks[BPF_TRAMP_FEXIT]; | |
| 1822 | − struct bpf_tramp_links *fmod_ret = &tlinks[BPF_TRAMP_MODIFY_RETURN]; | |
| 1820 | + struct bpf_tramp_nodes *fentry = &tnodes[BPF_TRAMP_FENTRY]; | |
| 1821 | + struct bpf_tramp_nodes *fexit = &tnodes[BPF_TRAMP_FEXIT]; | |
| 1822 | + struct bpf_tramp_nodes *fmod_ret = &tnodes[BPF_TRAMP_MODIFY_RETURN]; | |
| 1823 | 1823 | u32 **branches = NULL; |
| 1824 | 1824 | |
| 1825 | 1825 | /* |
| @@ -1898,7 +1898,7 @@ static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_i · 72 unmodified lines | ||
| 1898 | 1898 | ip_off = stack_size; |
| 1899 | 1899 | } |
| 1900 | 1900 | |
| 1901 | − cookie_cnt = bpf_fsession_cookie_cnt(tlinks); | |
| 1901 | + cookie_cnt = bpf_fsession_cookie_cnt(tnodes); | |
| 1902 | 1902 | |
| 1903 | 1903 | /* Room for session cookies */ |
| 1904 | 1904 | stack_size += cookie_cnt * 8; |
| @@ -1969,7 +1969,7 @@ static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_i · 64 unmodified lines | ||
| 1969 | 1969 | |
| 1970 | 1970 | store_args(ctx, nr_arg_slots, args_off); |
| 1971 | 1971 | |
| 1972 | − if (bpf_fsession_cnt(tlinks)) { | |
| 1972 | + if (bpf_fsession_cnt(tnodes)) { | |
| 1973 | 1973 | /* clear all session cookies' value */ |
| 1974 | 1974 | for (i = 0; i < cookie_cnt; i++) |
| 1975 | 1975 | emit_insn(ctx, std, LOONGARCH_GPR_ZERO, LOONGARCH_GPR_FP, -cookie_off + 8 * i); |
| @@ -1994,20 +1994,20 @@ static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_i · 18 unmodified lines | ||
| 1994 | 1994 | return ret; |
| 1995 | 1995 | } |
| 1996 | 1996 | |
| 1997 | − if (fentry->nr_links) { | |
| 1997 | + if (fentry->nr_nodes) { | |
| 1998 | 1998 | ret = invoke_bpf(ctx, fentry, args_off, retval_off, run_ctx_off, func_meta_off, |
| 1999 | 1999 | flags & BPF_TRAMP_F_RET_FENTRY_RET, func_meta, cookie_off); |
| 2000 | 2000 | if (ret) |
| 2001 | 2001 | return ret; |
| 2002 | 2002 | } |
| 2003 | − if (fmod_ret->nr_links) { | |
| 2004 | − branches = kcalloc(fmod_ret->nr_links, sizeof(u32 *), GFP_KERNEL); | |
| 2003 | + if (fmod_ret->nr_nodes) { | |
| 2004 | + branches = kcalloc(fmod_ret->nr_nodes, sizeof(u32 *), GFP_KERNEL); | |
| 2005 | 2005 | if (!branches) |
| 2006 | 2006 | return -ENOMEM; |
| 2007 | 2007 | |
| 2008 | 2008 | emit_insn(ctx, std, LOONGARCH_GPR_ZERO, LOONGARCH_GPR_FP, -retval_off); |
| 2009 | − for (i = 0; i < fmod_ret->nr_links; i++) { | |
| 2010 | − ret = invoke_bpf_prog(ctx, fmod_ret->links[i], | |
| 2009 | + for (i = 0; i < fmod_ret->nr_nodes; i++) { | |
| 2010 | + ret = invoke_bpf_prog(ctx, fmod_ret->nodes[i], | |
| 2011 | 2011 | args_off, retval_off, run_ctx_off, true); |
| 2012 | 2012 | if (ret) |
| 2013 | 2013 | goto out; |
| @@ -2035,17 +2035,17 @@ static int __arch_prepare_bpf_trampoline(struct jit_ctx *ctx, struct bpf_tramp_i · 21 unmodified lines | ||
| 2035 | 2035 | emit_insn(ctx, nop); |
| 2036 | 2036 | } |
| 2037 | 2037 | |
| 2038 | − for (i = 0; ctx->image && i < fmod_ret->nr_links; i++) { | |
| 2038 | + for (i = 0; ctx->image && i < fmod_ret->nr_nodes; i++) { | |
| 2039 | 2039 | int offset = (void *)(&ctx->image[ctx->idx]) - (void *)branches[i]; |
| 2040 | 2040 | *branches[i] = larch_insn_gen_bne(LOONGARCH_GPR_T1, LOONGARCH_GPR_ZERO, offset); |
| 2041 | 2041 | } |
| 2042 | 2042 | |
| 2043 | 2043 | /* Set "is_return" flag for fsession */ |
| 2044 | 2044 | func_meta |= (1ULL << BPF_TRAMP_IS_RETURN_SHIFT); |
| 2045 | − if (bpf_fsession_cnt(tlinks)) | |
| 2045 | + if (bpf_fsession_cnt(tnodes)) | |
| 2046 | 2046 | emit_store_stack_imm64(ctx, LOONGARCH_GPR_T1, -func_meta_off, func_meta); |
| 2047 | 2047 | |
| 2048 | − if (fexit->nr_links) { | |
| 2048 | + if (fexit->nr_nodes) { | |
| 2049 | 2049 | ret = invoke_bpf(ctx, fexit, args_off, retval_off, run_ctx_off, |
| 2050 | 2050 | func_meta_off, false, func_meta, cookie_off); |
| 2051 | 2051 | if (ret) |
| @@ -2115,7 +2115,7 @@ out: · 63 unmodified lines | ||
| 2115 | 2115 | |
| 2116 | 2116 | int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image, |
| 2117 | 2117 | void *ro_image_end, const struct btf_func_model *m, |
| 2118 | − u32 flags, struct bpf_tramp_links *tlinks, void *func_addr) | |
| 2118 | + u32 flags, struct bpf_tramp_nodes *tnodes, void *func_addr) | |
| 2119 | 2119 | { |
| 2120 | 2120 | int ret, size; |
| 2121 | 2121 | void *image, *tmp; |
| @@ -2131,7 +2131,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image, · 9 unmodified lines | ||
| 2131 | 2131 | ctx.idx = 0; |
| 2132 | 2132 | |
| 2133 | 2133 | jit_fill_hole(image, (unsigned int)(ro_image_end - ro_image)); |
| 2134 | − ret = __arch_prepare_bpf_trampoline(&ctx, im, m, tlinks, func_addr, flags); | |
| 2134 | + ret = __arch_prepare_bpf_trampoline(&ctx, im, m, tnodes, func_addr, flags); | |
| 2135 | 2135 | if (ret < 0) |
| 2136 | 2136 | goto out; |
| 2137 | 2137 | |
| @@ -2152,7 +2152,7 @@ out: · 14 unmodified lines | ||
| 2152 | 2152 | } |
| 2153 | 2153 | |
| 2154 | 2154 | int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, |
| 2155 | − struct bpf_tramp_links *tlinks, void *func_addr) | |
| 2155 | + struct bpf_tramp_nodes *tnodes, void *func_addr) | |
| 2156 | 2156 | { |
| 2157 | 2157 | int ret; |
| 2158 | 2158 | struct jit_ctx ctx; |
| @@ -2161,7 +2161,7 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, · 2 unmodified lines | ||
| 2161 | 2161 | ctx.image = NULL; |
| 2162 | 2162 | ctx.idx = 0; |
| 2163 | 2163 | |
| 2164 | − ret = __arch_prepare_bpf_trampoline(&ctx, &im, m, tlinks, func_addr, flags); | |
| 2164 | + ret = __arch_prepare_bpf_trampoline(&ctx, &im, m, tnodes, func_addr, flags); | |
| 2165 | 2165 | |
| 2166 | 2166 | return ret < 0 ? ret : ret * LOONGARCH_INSN_SIZE; |
| 2167 | 2167 | } |
| @@ -597,22 +597,22 @@ int arch_protect_bpf_trampoline(void *image, unsigned int size) · 596 unmodified lines | ||
| 597 | 597 | } |
| 598 | 598 | |
| 599 | 599 | static int invoke_bpf_prog(u32 *image, u32 *ro_image, struct codegen_context *ctx, |
| 600 | − struct bpf_tramp_link *l, int regs_off, int retval_off, | |
| 600 | + struct bpf_tramp_node *n, int regs_off, int retval_off, | |
| 601 | 601 | int run_ctx_off, bool save_ret) |
| 602 | 602 | { |
| 603 | − struct bpf_prog *p = l->link.prog; | |
| 603 | + struct bpf_prog *p = n->link->prog; | |
| 604 | 604 | ppc_inst_t branch_insn; |
| 605 | 605 | u32 jmp_idx; |
| 606 | 606 | int ret = 0; |
| 607 | 607 | |
| 608 | 608 | /* Save cookie */ |
| 609 | 609 | if (IS_ENABLED(CONFIG_PPC64)) { |
| 610 | − PPC_LI64(_R3, l->cookie); | |
| 610 | + PPC_LI64(_R3, n->cookie); | |
| 611 | 611 | EMIT(PPC_RAW_STD(_R3, _R1, run_ctx_off + offsetof(struct bpf_tramp_run_ctx, |
| 612 | 612 | bpf_cookie))); |
| 613 | 613 | } else { |
| 614 | − PPC_LI32(_R3, l->cookie >> 32); | |
| 615 | − PPC_LI32(_R4, l->cookie); | |
| 614 | + PPC_LI32(_R3, n->cookie >> 32); | |
| 615 | + PPC_LI32(_R4, n->cookie); | |
| 616 | 616 | EMIT(PPC_RAW_STW(_R3, _R1, |
| 617 | 617 | run_ctx_off + offsetof(struct bpf_tramp_run_ctx, bpf_cookie))); |
| 618 | 618 | EMIT(PPC_RAW_STW(_R4, _R1, |
| @@ -679,7 +679,7 @@ static int invoke_bpf_prog(u32 *image, u32 *ro_image, struct codegen_context *ct · 60 unmodified lines | ||
| 679 | 679 | } |
| 680 | 680 | |
| 681 | 681 | static int invoke_bpf_mod_ret(u32 *image, u32 *ro_image, struct codegen_context *ctx, |
| 682 | − struct bpf_tramp_links *tl, int regs_off, int retval_off, | |
| 682 | + struct bpf_tramp_nodes *tn, int regs_off, int retval_off, | |
| 683 | 683 | int run_ctx_off, u32 *branches) |
| 684 | 684 | { |
| 685 | 685 | int i; |
| @@ -690,8 +690,8 @@ static int invoke_bpf_mod_ret(u32 *image, u32 *ro_image, struct codegen_context · 4 unmodified lines | ||
| 690 | 690 | */ |
| 691 | 691 | EMIT(PPC_RAW_LI(_R3, 0)); |
| 692 | 692 | EMIT(PPC_RAW_STL(_R3, _R1, retval_off)); |
| 693 | − for (i = 0; i < tl->nr_links; i++) { | |
| 694 | − if (invoke_bpf_prog(image, ro_image, ctx, tl->links[i], regs_off, retval_off, | |
| 693 | + for (i = 0; i < tn->nr_nodes; i++) { | |
| 694 | + if (invoke_bpf_prog(image, ro_image, ctx, tn->nodes[i], regs_off, retval_off, | |
| 695 | 695 | run_ctx_off, true)) |
| 696 | 696 | return -EINVAL; |
| 697 | 697 | |
| @@ -807,18 +807,18 @@ static void bpf_trampoline_restore_args_stack(u32 *image, struct codegen_context · 109 unmodified lines | ||
| 807 | 807 | static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_image, |
| 808 | 808 | void *rw_image_end, void *ro_image, |
| 809 | 809 | const struct btf_func_model *m, u32 flags, |
| 810 | − struct bpf_tramp_links *tlinks, | |
| 810 | + struct bpf_tramp_nodes *tnodes, | |
| 811 | 811 | void *func_addr) |
| 812 | 812 | { |
| 813 | 813 | int regs_off, func_meta_off, ip_off, run_ctx_off, retval_off; |
| 814 | 814 | int nvr_off, alt_lr_off, r4_off = 0; |
| 815 | − struct bpf_tramp_links *fmod_ret = &tlinks[BPF_TRAMP_MODIFY_RETURN]; | |
| 816 | − struct bpf_tramp_links *fentry = &tlinks[BPF_TRAMP_FENTRY]; | |
| 817 | − struct bpf_tramp_links *fexit = &tlinks[BPF_TRAMP_FEXIT]; | |
| 815 | + struct bpf_tramp_nodes *fmod_ret = &tnodes[BPF_TRAMP_MODIFY_RETURN]; | |
| 816 | + struct bpf_tramp_nodes *fentry = &tnodes[BPF_TRAMP_FENTRY]; | |
| 817 | + struct bpf_tramp_nodes *fexit = &tnodes[BPF_TRAMP_FEXIT]; | |
| 818 | 818 | int i, ret, nr_regs, retaddr_off, bpf_frame_size = 0; |
| 819 | 819 | struct codegen_context codegen_ctx, *ctx; |
| 820 | 820 | int cookie_off, cookie_cnt, cookie_ctx_off; |
| 821 | − int fsession_cnt = bpf_fsession_cnt(tlinks); | |
| 821 | + int fsession_cnt = bpf_fsession_cnt(tnodes); | |
| 822 | 822 | u64 func_meta; |
| 823 | 823 | u32 *image = (u32 *)rw_image; |
| 824 | 824 | ppc_inst_t branch_insn; |
| @@ -893,7 +893,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 68 unmodified lines | ||
| 893 | 893 | |
| 894 | 894 | /* room for session cookies */ |
| 895 | 895 | cookie_off = bpf_frame_size; |
| 896 | − cookie_cnt = bpf_fsession_cookie_cnt(tlinks); | |
| 896 | + cookie_cnt = bpf_fsession_cookie_cnt(tnodes); | |
| 897 | 897 | bpf_frame_size += cookie_cnt * 8; |
| 898 | 898 | |
| 899 | 899 | /* Room for IP address argument */ |
| @@ -1030,21 +1030,21 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 130 unmodified lines | ||
| 1030 | 1030 | |
| 1031 | 1031 | cookie_ctx_off = (regs_off - cookie_off) / 8; |
| 1032 | 1032 | |
| 1033 | − for (i = 0; i < fentry->nr_links; i++) { | |
| 1034 | − if (bpf_prog_calls_session_cookie(fentry->links[i])) { | |
| 1033 | + for (i = 0; i < fentry->nr_nodes; i++) { | |
| 1034 | + if (bpf_prog_calls_session_cookie(fentry->nodes[i])) { | |
| 1035 | 1035 | u64 meta = func_meta | (cookie_ctx_off << BPF_TRAMP_COOKIE_INDEX_SHIFT); |
| 1036 | 1036 | |
| 1037 | 1037 | store_func_meta(image, ctx, meta, func_meta_off); |
| 1038 | 1038 | cookie_ctx_off--; |
| 1039 | 1039 | } |
| 1040 | 1040 | |
| 1041 | − if (invoke_bpf_prog(image, ro_image, ctx, fentry->links[i], regs_off, retval_off, | |
| 1041 | + if (invoke_bpf_prog(image, ro_image, ctx, fentry->nodes[i], regs_off, retval_off, | |
| 1042 | 1042 | run_ctx_off, flags & BPF_TRAMP_F_RET_FENTRY_RET)) |
| 1043 | 1043 | return -EINVAL; |
| 1044 | 1044 | } |
| 1045 | 1045 | |
| 1046 | − if (fmod_ret->nr_links) { | |
| 1047 | − branches = kcalloc(fmod_ret->nr_links, sizeof(u32), GFP_KERNEL); | |
| 1046 | + if (fmod_ret->nr_nodes) { | |
| 1047 | + branches = kcalloc(fmod_ret->nr_nodes, sizeof(u32), GFP_KERNEL); | |
| 1048 | 1048 | if (!branches) |
| 1049 | 1049 | return -ENOMEM; |
| 1050 | 1050 | |
| @@ -1093,7 +1093,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 42 unmodified lines | ||
| 1093 | 1093 | } |
| 1094 | 1094 | |
| 1095 | 1095 | /* Update branches saved in invoke_bpf_mod_ret with address of do_fexit */ |
| 1096 | − for (i = 0; i < fmod_ret->nr_links && image; i++) { | |
| 1096 | + for (i = 0; i < fmod_ret->nr_nodes && image; i++) { | |
| 1097 | 1097 | if (create_cond_branch(&branch_insn, &image[branches[i]], |
| 1098 | 1098 | (unsigned long)&image[ctx->idx], COND_NE << 16)) { |
| 1099 | 1099 | ret = -EINVAL; |
| @@ -1110,15 +1110,15 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 10 unmodified lines | ||
| 1110 | 1110 | |
| 1111 | 1111 | cookie_ctx_off = (regs_off - cookie_off) / 8; |
| 1112 | 1112 | |
| 1113 | − for (i = 0; i < fexit->nr_links; i++) { | |
| 1114 | − if (bpf_prog_calls_session_cookie(fexit->links[i])) { | |
| 1113 | + for (i = 0; i < fexit->nr_nodes; i++) { | |
| 1114 | + if (bpf_prog_calls_session_cookie(fexit->nodes[i])) { | |
| 1115 | 1115 | u64 meta = func_meta | (cookie_ctx_off << BPF_TRAMP_COOKIE_INDEX_SHIFT); |
| 1116 | 1116 | |
| 1117 | 1117 | store_func_meta(image, ctx, meta, func_meta_off); |
| 1118 | 1118 | cookie_ctx_off--; |
| 1119 | 1119 | } |
| 1120 | 1120 | |
| 1121 | − if (invoke_bpf_prog(image, ro_image, ctx, fexit->links[i], regs_off, retval_off, | |
| 1121 | + if (invoke_bpf_prog(image, ro_image, ctx, fexit->nodes[i], regs_off, retval_off, | |
| 1122 | 1122 | run_ctx_off, false)) { |
| 1123 | 1123 | ret = -EINVAL; |
| 1124 | 1124 | goto cleanup; |
| @@ -1185,18 +1185,18 @@ cleanup: · 60 unmodified lines | ||
| 1185 | 1185 | } |
| 1186 | 1186 | |
| 1187 | 1187 | int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, |
| 1188 | − struct bpf_tramp_links *tlinks, void *func_addr) | |
| 1188 | + struct bpf_tramp_nodes *tnodes, void *func_addr) | |
| 1189 | 1189 | { |
| 1190 | 1190 | struct bpf_tramp_image im; |
| 1191 | 1191 | int ret; |
| 1192 | 1192 | |
| 1193 | − ret = __arch_prepare_bpf_trampoline(&im, NULL, NULL, NULL, m, flags, tlinks, func_addr); | |
| 1193 | + ret = __arch_prepare_bpf_trampoline(&im, NULL, NULL, NULL, m, flags, tnodes, func_addr); | |
| 1194 | 1194 | return ret; |
| 1195 | 1195 | } |
| 1196 | 1196 | |
| 1197 | 1197 | int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *image_end, |
| 1198 | 1198 | const struct btf_func_model *m, u32 flags, |
| 1199 | − struct bpf_tramp_links *tlinks, | |
| 1199 | + struct bpf_tramp_nodes *tnodes, | |
| 1200 | 1200 | void *func_addr) |
| 1201 | 1201 | { |
| 1202 | 1202 | u32 size = image_end - image; |
| @@ -1212,7 +1212,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i · 9 unmodified lines | ||
| 1212 | 1212 | return -ENOMEM; |
| 1213 | 1213 | |
| 1214 | 1214 | ret = __arch_prepare_bpf_trampoline(im, rw_image, rw_image + size, image, m, |
| 1215 | − flags, tlinks, func_addr); | |
| 1215 | + flags, tnodes, func_addr); | |
| 1216 | 1216 | if (ret < 0) |
| 1217 | 1217 | goto out; |
| 1218 | 1218 | |
| @@ -934,15 +934,15 @@ static void emit_store_stack_imm64(u8 reg, int stack_off, u64 imm64, · 933 unmodified lines | ||
| 934 | 934 | emit_sd(RV_REG_FP, stack_off, reg, ctx); |
| 935 | 935 | } |
| 936 | 936 | |
| 937 | −static int invoke_bpf_prog(struct bpf_tramp_link *l, int args_off, int retval_off, | |
| 937 | +static int invoke_bpf_prog(struct bpf_tramp_node *node, int args_off, int retval_off, | |
| 938 | 938 | int run_ctx_off, bool save_ret, struct rv_jit_context *ctx) |
| 939 | 939 | { |
| 940 | 940 | int ret, branch_off; |
| 941 | − struct bpf_prog *p = l->link.prog; | |
| 941 | + struct bpf_prog *p = node->link->prog; | |
| 942 | 942 | int cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie); |
| 943 | 943 | |
| 944 | − if (l->cookie) | |
| 945 | − emit_store_stack_imm64(RV_REG_T1, -run_ctx_off + cookie_off, l->cookie, ctx); | |
| 944 | + if (node->cookie) | |
| 945 | + emit_store_stack_imm64(RV_REG_T1, -run_ctx_off + cookie_off, node->cookie, ctx); | |
| 946 | 946 | else |
| 947 | 947 | emit_sd(RV_REG_FP, -run_ctx_off + cookie_off, RV_REG_ZERO, ctx); |
| 948 | 948 | |
| @@ -996,22 +996,22 @@ static int invoke_bpf_prog(struct bpf_tramp_link *l, int args_off, int retval_of · 47 unmodified lines | ||
| 996 | 996 | return ret; |
| 997 | 997 | } |
| 998 | 998 | |
| 999 | −static int invoke_bpf(struct bpf_tramp_links *tl, int args_off, int retval_off, | |
| 999 | +static int invoke_bpf(struct bpf_tramp_nodes *tn, int args_off, int retval_off, | |
| 1000 | 1000 | int run_ctx_off, int func_meta_off, bool save_ret, u64 func_meta, |
| 1001 | 1001 | int cookie_off, struct rv_jit_context *ctx) |
| 1002 | 1002 | { |
| 1003 | 1003 | int i, cur_cookie = (cookie_off - args_off) / 8; |
| 1004 | 1004 | |
| 1005 | − for (i = 0; i < tl->nr_links; i++) { | |
| 1005 | + for (i = 0; i < tn->nr_nodes; i++) { | |
| 1006 | 1006 | int err; |
| 1007 | 1007 | |
| 1008 | − if (bpf_prog_calls_session_cookie(tl->links[i])) { | |
| 1008 | + if (bpf_prog_calls_session_cookie(tn->nodes[i])) { | |
| 1009 | 1009 | u64 meta = func_meta | ((u64)cur_cookie << BPF_TRAMP_COOKIE_INDEX_SHIFT); |
| 1010 | 1010 | |
| 1011 | 1011 | emit_store_stack_imm64(RV_REG_T1, -func_meta_off, meta, ctx); |
| 1012 | 1012 | cur_cookie--; |
| 1013 | 1013 | } |
| 1014 | − err = invoke_bpf_prog(tl->links[i], args_off, retval_off, run_ctx_off, | |
| 1014 | + err = invoke_bpf_prog(tn->nodes[i], args_off, retval_off, run_ctx_off, | |
| 1015 | 1015 | save_ret, ctx); |
| 1016 | 1016 | if (err) |
| 1017 | 1017 | return err; |
| @@ -1021,7 +1021,7 @@ static int invoke_bpf(struct bpf_tramp_links *tl, int args_off, int retval_off, · 3 unmodified lines | ||
| 1021 | 1021 | |
| 1022 | 1022 | static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, |
| 1023 | 1023 | const struct btf_func_model *m, |
| 1024 | − struct bpf_tramp_links *tlinks, | |
| 1024 | + struct bpf_tramp_nodes *tnodes, | |
| 1025 | 1025 | void *func_addr, u32 flags, |
| 1026 | 1026 | struct rv_jit_context *ctx) |
| 1027 | 1027 | { |
| @@ -1030,9 +1030,9 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, · 2 unmodified lines | ||
| 1030 | 1030 | int stack_size = 0, nr_arg_slots = 0; |
| 1031 | 1031 | int retval_off, args_off, func_meta_off, ip_off, run_ctx_off, sreg_off, stk_arg_off; |
| 1032 | 1032 | int cookie_off, cookie_cnt; |
| 1033 | − struct bpf_tramp_links *fentry = &tlinks[BPF_TRAMP_FENTRY]; | |
| 1034 | − struct bpf_tramp_links *fexit = &tlinks[BPF_TRAMP_FEXIT]; | |
| 1035 | − struct bpf_tramp_links *fmod_ret = &tlinks[BPF_TRAMP_MODIFY_RETURN]; | |
| 1033 | + struct bpf_tramp_nodes *fentry = &tnodes[BPF_TRAMP_FENTRY]; | |
| 1034 | + struct bpf_tramp_nodes *fexit = &tnodes[BPF_TRAMP_FEXIT]; | |
| 1035 | + struct bpf_tramp_nodes *fmod_ret = &tnodes[BPF_TRAMP_MODIFY_RETURN]; | |
| 1036 | 1036 | bool is_struct_ops = flags & BPF_TRAMP_F_INDIRECT; |
| 1037 | 1037 | void *orig_call = func_addr; |
| 1038 | 1038 | bool save_ret; |
| @@ -1115,7 +1115,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, · 76 unmodified lines | ||
| 1115 | 1115 | ip_off = stack_size; |
| 1116 | 1116 | } |
| 1117 | 1117 | |
| 1118 | − cookie_cnt = bpf_fsession_cookie_cnt(tlinks); | |
| 1118 | + cookie_cnt = bpf_fsession_cookie_cnt(tnodes); | |
| 1119 | 1119 | /* room for session cookies */ |
| 1120 | 1120 | stack_size += cookie_cnt * 8; |
| 1121 | 1121 | cookie_off = stack_size; |
| @@ -1172,7 +1172,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, · 50 unmodified lines | ||
| 1172 | 1172 | |
| 1173 | 1173 | store_args(nr_arg_slots, args_off, ctx); |
| 1174 | 1174 | |
| 1175 | − if (bpf_fsession_cnt(tlinks)) { | |
| 1175 | + if (bpf_fsession_cnt(tnodes)) { | |
| 1176 | 1176 | /* clear all session cookies' value */ |
| 1177 | 1177 | for (i = 0; i < cookie_cnt; i++) |
| 1178 | 1178 | emit_sd(RV_REG_FP, -cookie_off + 8 * i, RV_REG_ZERO, ctx); |
| @@ -1187,22 +1187,22 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, · 8 unmodified lines | ||
| 1187 | 1187 | return ret; |
| 1188 | 1188 | } |
| 1189 | 1189 | |
| 1190 | − if (fentry->nr_links) { | |
| 1190 | + if (fentry->nr_nodes) { | |
| 1191 | 1191 | ret = invoke_bpf(fentry, args_off, retval_off, run_ctx_off, func_meta_off, |
| 1192 | 1192 | flags & BPF_TRAMP_F_RET_FENTRY_RET, func_meta, cookie_off, ctx); |
| 1193 | 1193 | if (ret) |
| 1194 | 1194 | return ret; |
| 1195 | 1195 | } |
| 1196 | 1196 | |
| 1197 | − if (fmod_ret->nr_links) { | |
| 1198 | − branches_off = kzalloc_objs(int, fmod_ret->nr_links); | |
| 1197 | + if (fmod_ret->nr_nodes) { | |
| 1198 | + branches_off = kzalloc_objs(int, fmod_ret->nr_nodes); | |
| 1199 | 1199 | if (!branches_off) |
| 1200 | 1200 | return -ENOMEM; |
| 1201 | 1201 | |
| 1202 | 1202 | /* cleanup to avoid garbage return value confusion */ |
| 1203 | 1203 | emit_sd(RV_REG_FP, -retval_off, RV_REG_ZERO, ctx); |
| 1204 | − for (i = 0; i < fmod_ret->nr_links; i++) { | |
| 1205 | − ret = invoke_bpf_prog(fmod_ret->links[i], args_off, retval_off, | |
| 1204 | + for (i = 0; i < fmod_ret->nr_nodes; i++) { | |
| 1205 | + ret = invoke_bpf_prog(fmod_ret->nodes[i], args_off, retval_off, | |
| 1206 | 1206 | run_ctx_off, true, ctx); |
| 1207 | 1207 | if (ret) |
| 1208 | 1208 | goto out; |
| @@ -1230,7 +1230,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, · 21 unmodified lines | ||
| 1230 | 1230 | } |
| 1231 | 1231 | |
| 1232 | 1232 | /* update branches saved in invoke_bpf_mod_ret with bnez */ |
| 1233 | − for (i = 0; ctx->insns && i < fmod_ret->nr_links; i++) { | |
| 1233 | + for (i = 0; ctx->insns && i < fmod_ret->nr_nodes; i++) { | |
| 1234 | 1234 | offset = ninsns_rvoff(ctx->ninsns - branches_off[i]); |
| 1235 | 1235 | insn = rv_bne(RV_REG_T1, RV_REG_ZERO, offset >> 1); |
| 1236 | 1236 | *(u32 *)(ctx->insns + branches_off[i]) = insn; |
| @@ -1238,10 +1238,10 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, · 1 unmodified line | ||
| 1238 | 1238 | |
| 1239 | 1239 | /* set "is_return" flag for fsession */ |
| 1240 | 1240 | func_meta |= (1ULL << BPF_TRAMP_IS_RETURN_SHIFT); |
| 1241 | − if (bpf_fsession_cnt(tlinks)) | |
| 1241 | + if (bpf_fsession_cnt(tnodes)) | |
| 1242 | 1242 | emit_store_stack_imm64(RV_REG_T1, -func_meta_off, func_meta, ctx); |
| 1243 | 1243 | |
| 1244 | − if (fexit->nr_links) { | |
| 1244 | + if (fexit->nr_nodes) { | |
| 1245 | 1245 | ret = invoke_bpf(fexit, args_off, retval_off, run_ctx_off, func_meta_off, |
| 1246 | 1246 | false, func_meta, cookie_off, ctx); |
| 1247 | 1247 | if (ret) |
| @@ -1305,7 +1305,7 @@ out: · 57 unmodified lines | ||
| 1305 | 1305 | } |
| 1306 | 1306 | |
| 1307 | 1307 | int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, |
| 1308 | − struct bpf_tramp_links *tlinks, void *func_addr) | |
| 1308 | + struct bpf_tramp_nodes *tnodes, void *func_addr) | |
| 1309 | 1309 | { |
| 1310 | 1310 | struct bpf_tramp_image im; |
| 1311 | 1311 | struct rv_jit_context ctx; |
| @@ -1314,7 +1314,7 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, · 2 unmodified lines | ||
| 1314 | 1314 | ctx.ninsns = 0; |
| 1315 | 1315 | ctx.insns = NULL; |
| 1316 | 1316 | ctx.ro_insns = NULL; |
| 1317 | − ret = __arch_prepare_bpf_trampoline(&im, m, tlinks, func_addr, flags, &ctx); | |
| 1317 | + ret = __arch_prepare_bpf_trampoline(&im, m, tnodes, func_addr, flags, &ctx); | |
| 1318 | 1318 | |
| 1319 | 1319 | return ret < 0 ? ret : ninsns_rvoff(ctx.ninsns); |
| 1320 | 1320 | } |
| @@ -1331,7 +1331,7 @@ void arch_free_bpf_trampoline(void *image, unsigned int size) · 10 unmodified lines | ||
| 1331 | 1331 | |
| 1332 | 1332 | int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image, |
| 1333 | 1333 | void *ro_image_end, const struct btf_func_model *m, |
| 1334 | − u32 flags, struct bpf_tramp_links *tlinks, | |
| 1334 | + u32 flags, struct bpf_tramp_nodes *tnodes, | |
| 1335 | 1335 | void *func_addr) |
| 1336 | 1336 | { |
| 1337 | 1337 | int ret; |
| @@ -1346,7 +1346,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *ro_image, · 8 unmodified lines | ||
| 1346 | 1346 | ctx.ninsns = 0; |
| 1347 | 1347 | ctx.insns = image; |
| 1348 | 1348 | ctx.ro_insns = ro_image; |
| 1349 | − ret = __arch_prepare_bpf_trampoline(im, m, tlinks, func_addr, flags, &ctx); | |
| 1349 | + ret = __arch_prepare_bpf_trampoline(im, m, tnodes, func_addr, flags, &ctx); | |
| 1350 | 1350 | if (ret < 0) |
| 1351 | 1351 | goto out; |
| 1352 | 1352 | |
| @@ -1808,6 +1808,13 @@ int bpf_jit_emit_insn(const struct bpf_insn *insn, struct rv_jit_context *ctx, · 455 unmodified lines | ||
| 1808 | 1808 | break; |
| 1809 | 1809 | } |
| 1810 | 1810 | |
| 1811 | + /* Implement helper call to bpf_get_current_task/_btf() inline */ | |
| 1812 | + if (insn->src_reg == 0 && (insn->imm == BPF_FUNC_get_current_task || | |
| 1813 | + insn->imm == BPF_FUNC_get_current_task_btf)) { | |
| 1814 | + emit_mv(bpf_to_rv_reg(BPF_REG_0, ctx), RV_REG_TP, ctx); | |
| 1815 | + break; | |
| 1816 | + } | |
| 1817 | + | |
| 1811 | 1818 | mark_call(ctx); |
| 1812 | 1819 | ret = bpf_jit_get_func_addr(ctx->prog, insn, extra_pass, |
| 1813 | 1820 | &addr, &fixed_addr); |
| @@ -2138,6 +2145,8 @@ bool bpf_jit_inlines_helper_call(s32 imm) · 324 unmodified lines | ||
| 2138 | 2145 | { |
| 2139 | 2146 | switch (imm) { |
| 2140 | 2147 | case BPF_FUNC_get_smp_processor_id: |
| 2148 | + case BPF_FUNC_get_current_task: | |
| 2149 | + case BPF_FUNC_get_current_task_btf: | |
| 2141 | 2150 | return true; |
| 2142 | 2151 | default: |
| 2143 | 2152 | return false; |
| @@ -27,6 +27,7 @@ · 26 unmodified lines | ||
| 27 | 27 | #include <asm/extable.h> |
| 28 | 28 | #include <asm/dis.h> |
| 29 | 29 | #include <asm/facility.h> |
| 30 | +#include <asm/lowcore.h> | |
| 30 | 31 | #include <asm/nospec-branch.h> |
| 31 | 32 | #include <asm/set_memory.h> |
| 32 | 33 | #include <asm/text-patching.h> |
| @@ -1777,6 +1778,30 @@ static noinline int bpf_jit_insn(struct bpf_jit *jit, struct bpf_prog *fp, · 1744 unmodified lines | ||
| 1777 | 1778 | int j, ret; |
| 1778 | 1779 | u64 func; |
| 1779 | 1780 | |
| 1781 | + /* Implement helper call to bpf_get_smp_processor_id() inline */ | |
| 1782 | + if (insn->src_reg == 0 && | |
| 1783 | + insn->imm == BPF_FUNC_get_smp_processor_id) { | |
| 1784 | + const u32 *cpu_nr = &get_lowcore()->cpu_nr; | |
| 1785 | + | |
| 1786 | + /* ly %b0, cpu_nr */ | |
| 1787 | + EMIT6_DISP_LH(0xe3000000, 0x0058, BPF_REG_0, REG_0, REG_0, | |
| 1788 | + (unsigned long)cpu_nr); | |
| 1789 | + break; | |
| 1790 | + } | |
| 1791 | + | |
| 1792 | + /* Implement helper call to bpf_get_current_task/_btf() inline */ | |
| 1793 | + if (insn->src_reg == 0 && | |
| 1794 | + (insn->imm == BPF_FUNC_get_current_task || | |
| 1795 | + insn->imm == BPF_FUNC_get_current_task_btf)) { | |
| 1796 | + const u64 *current_task = | |
| 1797 | + &get_lowcore()->current_task; | |
| 1798 | + | |
| 1799 | + /* lg %b0, current_task */ | |
| 1800 | + EMIT6_DISP_LH(0xe3000000, 0x0004, BPF_REG_0, REG_0, REG_0, | |
| 1801 | + (unsigned long)current_task); | |
| 1802 | + break; | |
| 1803 | + } | |
| 1804 | + | |
| 1780 | 1805 | ret = bpf_jit_get_func_addr(fp, insn, extra_pass, |
| 1781 | 1806 | &func, &func_addr_fixed); |
| 1782 | 1807 | if (ret < 0) |
| @@ -2512,19 +2537,19 @@ static void emit_store_stack_imm64(struct bpf_jit *jit, int tmp_reg, int stack_o · 729 unmodified lines | ||
| 2512 | 2537 | |
| 2513 | 2538 | static int invoke_bpf_prog(struct bpf_tramp_jit *tjit, |
| 2514 | 2539 | const struct btf_func_model *m, |
| 2515 | − struct bpf_tramp_link *tlink, bool save_ret) | |
| 2540 | + struct bpf_tramp_node *node, bool save_ret) | |
| 2516 | 2541 | { |
| 2517 | 2542 | struct bpf_jit *jit = &tjit->common; |
| 2518 | 2543 | int cookie_off = tjit->run_ctx_off + |
| 2519 | 2544 | offsetof(struct bpf_tramp_run_ctx, bpf_cookie); |
| 2520 | − struct bpf_prog *p = tlink->link.prog; | |
| 2545 | + struct bpf_prog *p = node->link->prog; | |
| 2521 | 2546 | int patch; |
| 2522 | 2547 | |
| 2523 | 2548 | /* |
| 2524 | − * run_ctx.cookie = tlink->cookie; | |
| 2549 | + * run_ctx.cookie = node->cookie; | |
| 2525 | 2550 | */ |
| 2526 | 2551 | |
| 2527 | − emit_store_stack_imm64(jit, REG_W0, cookie_off, tlink->cookie); | |
| 2552 | + emit_store_stack_imm64(jit, REG_W0, cookie_off, node->cookie); | |
| 2528 | 2553 | |
| 2529 | 2554 | /* |
| 2530 | 2555 | * if ((start = __bpf_prog_enter(p, &run_ctx)) == 0) |
| @@ -2584,20 +2609,20 @@ static int invoke_bpf_prog(struct bpf_tramp_jit *tjit, · 53 unmodified lines | ||
| 2584 | 2609 | |
| 2585 | 2610 | static int invoke_bpf(struct bpf_tramp_jit *tjit, |
| 2586 | 2611 | const struct btf_func_model *m, |
| 2587 | − struct bpf_tramp_links *tl, bool save_ret, | |
| 2612 | + struct bpf_tramp_nodes *tn, bool save_ret, | |
| 2588 | 2613 | u64 func_meta, int cookie_off) |
| 2589 | 2614 | { |
| 2590 | 2615 | int i, cur_cookie = (tjit->bpf_args_off - cookie_off) / sizeof(u64); |
| 2591 | 2616 | struct bpf_jit *jit = &tjit->common; |
| 2592 | 2617 | |
| 2593 | − for (i = 0; i < tl->nr_links; i++) { | |
| 2594 | − if (bpf_prog_calls_session_cookie(tl->links[i])) { | |
| 2618 | + for (i = 0; i < tn->nr_nodes; i++) { | |
| 2619 | + if (bpf_prog_calls_session_cookie(tn->nodes[i])) { | |
| 2595 | 2620 | u64 meta = func_meta | ((u64)cur_cookie << BPF_TRAMP_COOKIE_INDEX_SHIFT); |
| 2596 | 2621 | |
| 2597 | 2622 | emit_store_stack_imm64(jit, REG_0, tjit->func_meta_off, meta); |
| 2598 | 2623 | cur_cookie--; |
| 2599 | 2624 | } |
| 2600 | − if (invoke_bpf_prog(tjit, m, tl->links[i], save_ret)) | |
| 2625 | + if (invoke_bpf_prog(tjit, m, tn->nodes[i], save_ret)) | |
| 2601 | 2626 | return -EINVAL; |
| 2602 | 2627 | } |
| 2603 | 2628 | |
| @@ -2626,12 +2651,12 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, · 22 unmodified lines | ||
| 2626 | 2651 | struct bpf_tramp_jit *tjit, |
| 2627 | 2652 | const struct btf_func_model *m, |
| 2628 | 2653 | u32 flags, |
| 2629 | − struct bpf_tramp_links *tlinks, | |
| 2654 | + struct bpf_tramp_nodes *tnodes, | |
| 2630 | 2655 | void *func_addr) |
| 2631 | 2656 | { |
| 2632 | − struct bpf_tramp_links *fmod_ret = &tlinks[BPF_TRAMP_MODIFY_RETURN]; | |
| 2633 | − struct bpf_tramp_links *fentry = &tlinks[BPF_TRAMP_FENTRY]; | |
| 2634 | − struct bpf_tramp_links *fexit = &tlinks[BPF_TRAMP_FEXIT]; | |
| 2657 | + struct bpf_tramp_nodes *fmod_ret = &tnodes[BPF_TRAMP_MODIFY_RETURN]; | |
| 2658 | + struct bpf_tramp_nodes *fentry = &tnodes[BPF_TRAMP_FENTRY]; | |
| 2659 | + struct bpf_tramp_nodes *fexit = &tnodes[BPF_TRAMP_FEXIT]; | |
| 2635 | 2660 | int nr_bpf_args, nr_reg_args, nr_stack_args; |
| 2636 | 2661 | int cookie_cnt, cookie_off, fsession_cnt; |
| 2637 | 2662 | struct bpf_jit *jit = &tjit->common; |
| @@ -2668,8 +2693,8 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, · 30 unmodified lines | ||
| 2668 | 2693 | return -ENOTSUPP; |
| 2669 | 2694 | } |
| 2670 | 2695 | |
| 2671 | − cookie_cnt = bpf_fsession_cookie_cnt(tlinks); | |
| 2672 | − fsession_cnt = bpf_fsession_cnt(tlinks); | |
| 2696 | + cookie_cnt = bpf_fsession_cookie_cnt(tnodes); | |
| 2697 | + fsession_cnt = bpf_fsession_cnt(tnodes); | |
| 2673 | 2698 | |
| 2674 | 2699 | /* |
| 2675 | 2700 | * Calculate the stack layout. |
| @@ -2804,7 +2829,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, · 128 unmodified lines | ||
| 2804 | 2829 | func_meta, cookie_off)) |
| 2805 | 2830 | return -EINVAL; |
| 2806 | 2831 | |
| 2807 | − if (fmod_ret->nr_links) { | |
| 2832 | + if (fmod_ret->nr_nodes) { | |
| 2808 | 2833 | /* |
| 2809 | 2834 | * retval = 0; |
| 2810 | 2835 | */ |
| @@ -2813,8 +2838,8 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, · 2 unmodified lines | ||
| 2813 | 2838 | _EMIT6(0xd707f000 | tjit->retval_off, |
| 2814 | 2839 | 0xf000 | tjit->retval_off); |
| 2815 | 2840 | |
| 2816 | − for (i = 0; i < fmod_ret->nr_links; i++) { | |
| 2817 | − if (invoke_bpf_prog(tjit, m, fmod_ret->links[i], true)) | |
| 2841 | + for (i = 0; i < fmod_ret->nr_nodes; i++) { | |
| 2842 | + if (invoke_bpf_prog(tjit, m, fmod_ret->nodes[i], true)) | |
| 2818 | 2843 | return -EINVAL; |
| 2819 | 2844 | |
| 2820 | 2845 | /* |
| @@ -2939,7 +2964,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, · 118 unmodified lines | ||
| 2939 | 2964 | } |
| 2940 | 2965 | |
| 2941 | 2966 | int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, |
| 2942 | − struct bpf_tramp_links *tlinks, void *orig_call) | |
| 2967 | + struct bpf_tramp_nodes *tnodes, void *orig_call) | |
| 2943 | 2968 | { |
| 2944 | 2969 | struct bpf_tramp_image im; |
| 2945 | 2970 | struct bpf_tramp_jit tjit; |
| @@ -2948,14 +2973,14 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, · 2 unmodified lines | ||
| 2948 | 2973 | memset(&tjit, 0, sizeof(tjit)); |
| 2949 | 2974 | |
| 2950 | 2975 | ret = __arch_prepare_bpf_trampoline(&im, &tjit, m, flags, |
| 2951 | − tlinks, orig_call); | |
| 2976 | + tnodes, orig_call); | |
| 2952 | 2977 | |
| 2953 | 2978 | return ret < 0 ? ret : tjit.common.prg; |
| 2954 | 2979 | } |
| 2955 | 2980 | |
| 2956 | 2981 | int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, |
| 2957 | 2982 | void *image_end, const struct btf_func_model *m, |
| 2958 | − u32 flags, struct bpf_tramp_links *tlinks, | |
| 2983 | + u32 flags, struct bpf_tramp_nodes *tnodes, | |
| 2959 | 2984 | void *func_addr) |
| 2960 | 2985 | { |
| 2961 | 2986 | struct bpf_tramp_jit tjit; |
| @@ -2964,7 +2989,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, · 2 unmodified lines | ||
| 2964 | 2989 | /* Compute offsets, check whether the code fits. */ |
| 2965 | 2990 | memset(&tjit, 0, sizeof(tjit)); |
| 2966 | 2991 | ret = __arch_prepare_bpf_trampoline(im, &tjit, m, flags, |
| 2967 | − tlinks, func_addr); | |
| 2992 | + tnodes, func_addr); | |
| 2968 | 2993 | |
| 2969 | 2994 | if (ret < 0) |
| 2970 | 2995 | return ret; |
| @@ -2978,7 +3003,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, · 7 unmodified lines | ||
| 2978 | 3003 | tjit.common.prg = 0; |
| 2979 | 3004 | tjit.common.prg_buf = image; |
| 2980 | 3005 | ret = __arch_prepare_bpf_trampoline(im, &tjit, m, flags, |
| 2981 | − tlinks, func_addr); | |
| 3006 | + tnodes, func_addr); | |
| 2982 | 3007 | |
| 2983 | 3008 | return ret < 0 ? ret : tjit.common.prg; |
| 2984 | 3009 | } |
| @@ -3057,3 +3082,15 @@ bool bpf_jit_supports_timed_may_goto(void) · 72 unmodified lines | ||
| 3057 | 3082 | { |
| 3058 | 3083 | return true; |
| 3059 | 3084 | } |
| 3085 | + | |
| 3086 | +bool bpf_jit_inlines_helper_call(s32 imm) | |
| 3087 | +{ | |
| 3088 | + switch (imm) { | |
| 3089 | + case BPF_FUNC_get_smp_processor_id: | |
| 3090 | + case BPF_FUNC_get_current_task: | |
| 3091 | + case BPF_FUNC_get_current_task_btf: | |
| 3092 | + return true; | |
| 3093 | + default: | |
| 3094 | + return false; | |
| 3095 | + } | |
| 3096 | +} | |
| @@ -1284,6 +1284,18 @@ static inline void ptep_set_wrprotect(struct mm_struct *mm, · 1283 unmodified lines | ||
| 1284 | 1284 | } while (!try_cmpxchg((long *)&ptep->pte, (long *)&old_pte, *(long *)&new_pte)); |
| 1285 | 1285 | } |
| 1286 | 1286 | |
| 1287 | +/* | |
| 1288 | + * Note: strictly-zero compare is narrower than pte_none(), but the gap is | |
| 1289 | + * harmless: _PAGE_DIRTY and _PAGE_ACCESSED aren't set on untouched kernel PTEs. | |
| 1290 | + */ | |
| 1291 | +static inline bool ptep_try_set(pte_t *ptep, pte_t new_pte) | |
| 1292 | +{ | |
| 1293 | + pte_t old_pte = __pte(0); | |
| 1294 | + | |
| 1295 | + return try_cmpxchg((long *)&ptep->pte, (long *)&old_pte, *(long *)&new_pte); | |
| 1296 | +} | |
| 1297 | +#define ptep_try_set ptep_try_set | |
| 1298 | + | |
| 1287 | 1299 | #define flush_tlb_fix_spurious_fault(vma, address, ptep) do { } while (0) |
| 1288 | 1300 | |
| 1289 | 1301 | #define __HAVE_ARCH_PMDP_SET_ACCESS_FLAGS |
| @@ -8,6 +8,7 @@ · 7 unmodified lines | ||
| 8 | 8 | #include <linux/sched/task_stack.h> /* task_stack_*(), ... */ |
| 9 | 9 | #include <linux/kdebug.h> /* oops_begin/end, ... */ |
| 10 | 10 | #include <linux/memblock.h> /* max_low_pfn */ |
| 11 | +#include <linux/bpf_defs.h> /* bpf_arena_handle_page_fault */ | |
| 11 | 12 | #include <linux/kfence.h> /* kfence_handle_page_fault */ |
| 12 | 13 | #include <linux/kprobes.h> /* NOKPROBE_SYMBOL, ... */ |
| 13 | 14 | #include <linux/mmiotrace.h> /* kmmio_handler, ... */ |
| @@ -693,10 +694,13 @@ page_fault_oops(struct pt_regs *regs, unsigned long error_code, · 679 unmodified lines | ||
| 693 | 694 | if (IS_ENABLED(CONFIG_EFI)) |
| 694 | 695 | efi_crash_gracefully_on_page_fault(address, regs); |
| 695 | 696 | |
| 696 | − /* Only not-present faults should be handled by KFENCE. */ | |
| 697 | − if (!(error_code & X86_PF_PROT) && | |
| 698 | − kfence_handle_page_fault(address, error_code & X86_PF_WRITE, regs)) | |
| 699 | − return; | |
| 697 | + /* Only not-present faults should be handled by KFENCE or BPF arena. */ | |
| 698 | + if (!(error_code & X86_PF_PROT)) { | |
| 699 | + if (kfence_handle_page_fault(address, error_code & X86_PF_WRITE, regs)) | |
| 700 | + return; | |
| 701 | + if (bpf_arena_handle_page_fault(address, error_code & X86_PF_WRITE, regs->ip)) | |
| 702 | + return; | |
| 703 | + } | |
| 700 | 704 | |
| 701 | 705 | oops: |
| 702 | 706 | /* |
| @@ -10,6 +10,7 @@ · 9 unmodified lines | ||
| 10 | 10 | #include <linux/if_vlan.h> |
| 11 | 11 | #include <linux/bitfield.h> |
| 12 | 12 | #include <linux/bpf.h> |
| 13 | +#include <linux/bpf_verifier.h> | |
| 13 | 14 | #include <linux/memory.h> |
| 14 | 15 | #include <linux/sort.h> |
| 15 | 16 | #include <asm/extable.h> |
| @@ -390,6 +391,34 @@ static void pop_callee_regs(u8 **pprog, bool *callee_regs_used) · 374 unmodified lines | ||
| 390 | 391 | *pprog = prog; |
| 391 | 392 | } |
| 392 | 393 | |
| 394 | +/* add rsp, depth */ | |
| 395 | +static void emit_add_rsp(u8 **pprog, u16 depth) | |
| 396 | +{ | |
| 397 | + u8 *prog = *pprog; | |
| 398 | + | |
| 399 | + if (!depth) | |
| 400 | + return; | |
| 401 | + if (is_imm8(depth)) | |
| 402 | + EMIT4(0x48, 0x83, 0xC4, depth); /* add rsp, imm8 */ | |
| 403 | + else | |
| 404 | + EMIT3_off32(0x48, 0x81, 0xC4, depth); /* add rsp, imm32 */ | |
| 405 | + *pprog = prog; | |
| 406 | +} | |
| 407 | + | |
| 408 | +/* sub rsp, depth */ | |
| 409 | +static void emit_sub_rsp(u8 **pprog, u16 depth) | |
| 410 | +{ | |
| 411 | + u8 *prog = *pprog; | |
| 412 | + | |
| 413 | + if (!depth) | |
| 414 | + return; | |
| 415 | + if (is_imm8(depth)) | |
| 416 | + EMIT4(0x48, 0x83, 0xEC, depth); /* sub rsp, imm8 */ | |
| 417 | + else | |
| 418 | + EMIT3_off32(0x48, 0x81, 0xEC, depth); /* sub rsp, imm32 */ | |
| 419 | + *pprog = prog; | |
| 420 | +} | |
| 421 | + | |
| 393 | 422 | static void emit_nops(u8 **pprog, int len) |
| 394 | 423 | { |
| 395 | 424 | u8 *prog = *pprog; |
| @@ -1659,21 +1688,47 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int * · 1263 unmodified lines | ||
| 1659 | 1688 | bool seen_exit = false; |
| 1660 | 1689 | u8 temp[BPF_MAX_INSN_SIZE + BPF_INSN_SAFETY]; |
| 1661 | 1690 | void __percpu *priv_frame_ptr = NULL; |
| 1691 | + u16 out_stack_arg_cnt, outgoing_rsp; | |
| 1662 | 1692 | u64 arena_vm_start, user_vm_start; |
| 1663 | 1693 | void __percpu *priv_stack_ptr; |
| 1664 | 1694 | int i, excnt = 0; |
| 1665 | 1695 | int ilen, proglen = 0; |
| 1666 | 1696 | u8 *ip, *prog = temp; |
| 1667 | 1697 | u32 stack_depth; |
| 1698 | + int callee_saved_size; | |
| 1699 | + s32 outgoing_arg_base; | |
| 1668 | 1700 | int err; |
| 1669 | 1701 | |
| 1670 | 1702 | stack_depth = bpf_prog->aux->stack_depth; |
| 1703 | + out_stack_arg_cnt = bpf_out_stack_arg_cnt(env, bpf_prog); | |
| 1671 | 1704 | priv_stack_ptr = bpf_prog->aux->priv_stack_ptr; |
| 1672 | 1705 | if (priv_stack_ptr) { |
| 1673 | 1706 | priv_frame_ptr = priv_stack_ptr + PRIV_STACK_GUARD_SZ + round_up(stack_depth, 8); |
| 1674 | 1707 | stack_depth = 0; |
| 1675 | 1708 | } |
| 1676 | 1709 | |
| 1710 | + /* | |
| 1711 | + * Follow x86-64 calling convention for both BPF-to-BPF and | |
| 1712 | + * kfunc calls: | |
| 1713 | + * - Arg 6 is passed in R9 register | |
| 1714 | + * - Args 7+ are passed on the stack at [rsp] | |
| 1715 | + * | |
| 1716 | + * Incoming arg 6 is read from R9 (BPF r11+8 → MOV from R9). | |
| 1717 | + * Incoming args 7+ are read from [rbp + 16], [rbp + 24], ... | |
| 1718 | + * (BPF r11+16, r11+24, ... map directly with no offset change). | |
| 1719 | + * | |
| 1720 | + * tail_call_reachable is rejected by the verifier and priv_stack | |
| 1721 | + * is disabled by the JIT when stack args exist, so R9 is always | |
| 1722 | + * available. | |
| 1723 | + * | |
| 1724 | + * Stack layout (high to low): | |
| 1725 | + * [rbp + 16 + ...] incoming stack args 7+ (from caller) | |
| 1726 | + * [rbp + 8] return address | |
| 1727 | + * [rbp] saved rbp | |
| 1728 | + * [rbp - prog_stack] program stack | |
| 1729 | + * [below] callee-saved regs | |
| 1730 | + * [below] outgoing args 7+ (= rsp) | |
| 1731 | + */ | |
| 1677 | 1732 | arena_vm_start = bpf_arena_get_kern_vm_start(bpf_prog->aux->arena); |
| 1678 | 1733 | user_vm_start = bpf_arena_get_user_vm_start(bpf_prog->aux->arena); |
| 1679 | 1734 | |
| @@ -1700,6 +1755,44 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int * · 20 unmodified lines | ||
| 1700 | 1755 | push_r12(&prog); |
| 1701 | 1756 | push_callee_regs(&prog, callee_regs_used); |
| 1702 | 1757 | } |
| 1758 | + | |
| 1759 | + /* Compute callee-saved register area size. */ | |
| 1760 | + callee_saved_size = 0; | |
| 1761 | + if (bpf_prog->aux->exception_boundary || arena_vm_start) | |
| 1762 | + callee_saved_size += 8; /* r12 */ | |
| 1763 | + if (bpf_prog->aux->exception_boundary) { | |
| 1764 | + callee_saved_size += 4 * 8; /* rbx, r13, r14, r15 */ | |
| 1765 | + } else { | |
| 1766 | + int j; | |
| 1767 | + | |
| 1768 | + for (j = 0; j < 4; j++) | |
| 1769 | + if (callee_regs_used[j]) | |
| 1770 | + callee_saved_size += 8; | |
| 1771 | + } | |
| 1772 | + /* | |
| 1773 | + * Base offset from rbp for translating BPF outgoing args 7+ | |
| 1774 | + * to native offsets. BPF uses negative offsets from r11 | |
| 1775 | + * (r11-8 for arg6, r11-16 for arg7, ...) while x86 uses | |
| 1776 | + * positive offsets from rsp ([rsp+0] for arg7, [rsp+8] for | |
| 1777 | + * arg8, ...). Arg 6 goes to R9 directly. | |
| 1778 | + * | |
| 1779 | + * The translation reverses direction: | |
| 1780 | + * native_off = outgoing_arg_base - outgoing_rsp - bpf_off - 16 | |
| 1781 | + * | |
| 1782 | + * Note that tail_call_reachable is guaranteed to be false when | |
| 1783 | + * stack args exist, so tcc pushes need not be accounted for. | |
| 1784 | + */ | |
| 1785 | + outgoing_arg_base = -(round_up(stack_depth, 8) + callee_saved_size); | |
| 1786 | + | |
| 1787 | + /* | |
| 1788 | + * Allocate outgoing stack arg area for args 7+ only. | |
| 1789 | + * Arg 6 goes into r9 register, not on stack. | |
| 1790 | + */ | |
| 1791 | + outgoing_rsp = out_stack_arg_cnt > 1 ? (out_stack_arg_cnt - 1) * 8 : 0; | |
| 1792 | + if (bpf_prog->aux->exception_boundary) | |
| 1793 | + bpf_prog->aux->stack_arg_sp_adjust = outgoing_rsp; | |
| 1794 | + emit_sub_rsp(&prog, outgoing_rsp); | |
| 1795 | + | |
| 1703 | 1796 | if (arena_vm_start) |
| 1704 | 1797 | emit_mov_imm64(&prog, X86_REG_R12, |
| 1705 | 1798 | arena_vm_start >> 32, (u32) arena_vm_start); |
| @@ -1721,7 +1814,7 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int * · 15 unmodified lines | ||
| 1721 | 1814 | u8 b2 = 0, b3 = 0; |
| 1722 | 1815 | u8 *start_of_ldx; |
| 1723 | 1816 | s64 jmp_offset; |
| 1724 | − s16 insn_off; | |
| 1817 | + s32 insn_off; | |
| 1725 | 1818 | u8 jmp_cond; |
| 1726 | 1819 | u8 *func; |
| 1727 | 1820 | int nops; |
| @@ -2134,12 +2227,27 @@ static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int * · 406 unmodified lines | ||
| 2134 | 2227 | EMIT1(0xC7); |
| 2135 | 2228 | goto st; |
| 2136 | 2229 | case BPF_ST | BPF_MEM | BPF_DW: |
| 2230 | + if (dst_reg == BPF_REG_PARAMS && insn->off == -8) { | |
| 2231 | + /* Arg 6: store immediate in r9 register */ | |
| 2232 | + emit_mov_imm64(&prog, X86_REG_R9, imm32 >> 31, (u32)imm32); | |
| 2233 | + break; | |
| 2234 | + } | |
| 2137 | 2235 | EMIT2(add_1mod(0x48, dst_reg), 0xC7); |
| 2138 | 2236 | |
| 2139 | −st: if (is_imm8(insn->off)) | |
| 2140 | − EMIT2(add_1reg(0x40, dst_reg), insn->off); | |
| 2237 | +st: insn_off = insn->off; | |
| 2238 | + if (dst_reg == BPF_REG_PARAMS) { | |
| 2239 | + /* | |
| 2240 | + * Args 7+: reverse BPF negative offsets to | |
| 2241 | + * x86 positive rsp offsets. | |
| 2242 | + * BPF off=-16 → [rsp+0], off=-24 → [rsp+8], ... | |
| 2243 | + */ | |
| 2244 | + insn_off = outgoing_arg_base - outgoing_rsp - insn_off - 16; | |
| 2245 | + dst_reg = BPF_REG_FP; | |
| 2246 | + } | |
| 2247 | + if (is_imm8(insn_off)) | |
| 2248 | + EMIT2(add_1reg(0x40, dst_reg), insn_off); | |
| 2141 | 2249 | else |
| 2142 | − EMIT1_off32(add_1reg(0x80, dst_reg), insn->off); | |
| 2250 | + EMIT1_off32(add_1reg(0x80, dst_reg), insn_off); | |
| 2143 | 2251 | |
| 2144 | 2252 | EMIT(imm32, bpf_size_to_x86_bytes(BPF_SIZE(insn->code))); |
| 2145 | 2253 | break; |
| @@ -2149,7 +2257,17 @@ st: if (is_imm8(insn->off)) · 3 unmodified lines | ||
| 2149 | 2257 | case BPF_STX | BPF_MEM | BPF_H: |
| 2150 | 2258 | case BPF_STX | BPF_MEM | BPF_W: |
| 2151 | 2259 | case BPF_STX | BPF_MEM | BPF_DW: |
| 2152 | − emit_stx(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn->off); | |
| 2260 | + if (dst_reg == BPF_REG_PARAMS && insn->off == -8) { | |
| 2261 | + /* Arg 6: store register value in r9 */ | |
| 2262 | + EMIT_mov(X86_REG_R9, src_reg); | |
| 2263 | + break; | |
| 2264 | + } | |
| 2265 | + insn_off = insn->off; | |
| 2266 | + if (dst_reg == BPF_REG_PARAMS) { | |
| 2267 | + insn_off = outgoing_arg_base - outgoing_rsp - insn_off - 16; | |
| 2268 | + dst_reg = BPF_REG_FP; | |
| 2269 | + } | |
| 2270 | + emit_stx(&prog, BPF_SIZE(insn->code), dst_reg, src_reg, insn_off); | |
| 2153 | 2271 | break; |
| 2154 | 2272 | |
| 2155 | 2273 | case BPF_ST | BPF_PROBE_MEM32 | BPF_B: |
| @@ -2248,6 +2366,19 @@ populate_extable: · 92 unmodified lines | ||
| 2248 | 2366 | case BPF_LDX | BPF_PROBE_MEMSX | BPF_H: |
| 2249 | 2367 | case BPF_LDX | BPF_PROBE_MEMSX | BPF_W: |
| 2250 | 2368 | insn_off = insn->off; |
| 2369 | + if (src_reg == BPF_REG_PARAMS) { | |
| 2370 | + if (insn_off == 8) { | |
| 2371 | + /* Incoming arg 6: read from r9 */ | |
| 2372 | + EMIT_mov(dst_reg, X86_REG_R9); | |
| 2373 | + break; | |
| 2374 | + } | |
| 2375 | + src_reg = BPF_REG_FP; | |
| 2376 | + /* | |
| 2377 | + * Incoming args 7+: native_off == bpf_off | |
| 2378 | + * (r11+16 → [rbp+16], r11+24 → [rbp+24], ...) | |
| 2379 | + * No offset adjustment needed. | |
| 2380 | + */ | |
| 2381 | + } | |
| 2251 | 2382 | |
| 2252 | 2383 | if (BPF_MODE(insn->code) == BPF_PROBE_MEM || |
| 2253 | 2384 | BPF_MODE(insn->code) == BPF_PROBE_MEMSX) { |
| @@ -2736,6 +2867,8 @@ emit_jmp: · 482 unmodified lines | ||
| 2736 | 2867 | if (emit_spectre_bhb_barrier(&prog, ip, bpf_prog)) |
| 2737 | 2868 | return -EINVAL; |
| 2738 | 2869 | } |
| 2870 | + /* Deallocate outgoing args 7+ area. */ | |
| 2871 | + emit_add_rsp(&prog, outgoing_rsp); | |
| 2739 | 2872 | if (bpf_prog->aux->exception_boundary) { |
| 2740 | 2873 | pop_callee_regs(&prog, all_callee_regs_used); |
| 2741 | 2874 | pop_r12(&prog); |
| @@ -2971,15 +3104,15 @@ static void restore_regs(const struct btf_func_model *m, u8 **prog, · 229 unmodified lines | ||
| 2971 | 3104 | } |
| 2972 | 3105 | |
| 2973 | 3106 | static int invoke_bpf_prog(const struct btf_func_model *m, u8 **pprog, |
| 2974 | − struct bpf_tramp_link *l, int stack_size, | |
| 3107 | + struct bpf_tramp_node *node, int stack_size, | |
| 2975 | 3108 | int run_ctx_off, bool save_ret, |
| 2976 | 3109 | void *image, void *rw_image) |
| 2977 | 3110 | { |
| 2978 | 3111 | u8 *prog = *pprog; |
| 2979 | 3112 | u8 *jmp_insn; |
| 2980 | 3113 | int ctx_cookie_off = offsetof(struct bpf_tramp_run_ctx, bpf_cookie); |
| 2981 | − struct bpf_prog *p = l->link.prog; | |
| 2982 | − u64 cookie = l->cookie; | |
| 3114 | + struct bpf_prog *p = node->link->prog; | |
| 3115 | + u64 cookie = node->cookie; | |
| 2983 | 3116 | |
| 2984 | 3117 | /* mov rdi, cookie */ |
| 2985 | 3118 | emit_mov_imm64(&prog, BPF_REG_1, (long) cookie >> 32, (u32) (long) cookie); |
| @@ -3086,7 +3219,7 @@ static int emit_cond_near_jump(u8 **pprog, void *func, void *ip, u8 jmp_cond) · 100 unmodified lines | ||
| 3086 | 3219 | } |
| 3087 | 3220 | |
| 3088 | 3221 | static int invoke_bpf(const struct btf_func_model *m, u8 **pprog, |
| 3089 | − struct bpf_tramp_links *tl, int stack_size, | |
| 3222 | + struct bpf_tramp_nodes *tl, int stack_size, | |
| 3090 | 3223 | int run_ctx_off, int func_meta_off, bool save_ret, |
| 3091 | 3224 | void *image, void *rw_image, u64 func_meta, |
| 3092 | 3225 | int cookie_off) |
| @@ -3094,13 +3227,13 @@ static int invoke_bpf(const struct btf_func_model *m, u8 **pprog, · 1 unmodified line | ||
| 3094 | 3227 | int i, cur_cookie = (cookie_off - stack_size) / 8; |
| 3095 | 3228 | u8 *prog = *pprog; |
| 3096 | 3229 | |
| 3097 | − for (i = 0; i < tl->nr_links; i++) { | |
| 3098 | − if (tl->links[i]->link.prog->call_session_cookie) { | |
| 3230 | + for (i = 0; i < tl->nr_nodes; i++) { | |
| 3231 | + if (tl->nodes[i]->link->prog->call_session_cookie) { | |
| 3099 | 3232 | emit_store_stack_imm64(&prog, BPF_REG_0, -func_meta_off, |
| 3100 | 3233 | func_meta | (cur_cookie << BPF_TRAMP_COOKIE_INDEX_SHIFT)); |
| 3101 | 3234 | cur_cookie--; |
| 3102 | 3235 | } |
| 3103 | − if (invoke_bpf_prog(m, &prog, tl->links[i], stack_size, | |
| 3236 | + if (invoke_bpf_prog(m, &prog, tl->nodes[i], stack_size, | |
| 3104 | 3237 | run_ctx_off, save_ret, image, rw_image)) |
| 3105 | 3238 | return -EINVAL; |
| 3106 | 3239 | } |
| @@ -3109,7 +3242,7 @@ static int invoke_bpf(const struct btf_func_model *m, u8 **pprog, · 2 unmodified lines | ||
| 3109 | 3242 | } |
| 3110 | 3243 | |
| 3111 | 3244 | static int invoke_bpf_mod_ret(const struct btf_func_model *m, u8 **pprog, |
| 3112 | − struct bpf_tramp_links *tl, int stack_size, | |
| 3245 | + struct bpf_tramp_nodes *tl, int stack_size, | |
| 3113 | 3246 | int run_ctx_off, u8 **branches, |
| 3114 | 3247 | void *image, void *rw_image) |
| 3115 | 3248 | { |
| @@ -3121,8 +3254,8 @@ static int invoke_bpf_mod_ret(const struct btf_func_model *m, u8 **pprog, · 5 unmodified lines | ||
| 3121 | 3254 | */ |
| 3122 | 3255 | emit_mov_imm32(&prog, false, BPF_REG_0, 0); |
| 3123 | 3256 | emit_stx(&prog, BPF_DW, BPF_REG_FP, BPF_REG_0, -8); |
| 3124 | − for (i = 0; i < tl->nr_links; i++) { | |
| 3125 | − if (invoke_bpf_prog(m, &prog, tl->links[i], stack_size, run_ctx_off, true, | |
| 3257 | + for (i = 0; i < tl->nr_nodes; i++) { | |
| 3258 | + if (invoke_bpf_prog(m, &prog, tl->nodes[i], stack_size, run_ctx_off, true, | |
| 3126 | 3259 | image, rw_image)) |
| 3127 | 3260 | return -EINVAL; |
| 3128 | 3261 | |
| @@ -3213,14 +3346,14 @@ static int invoke_bpf_mod_ret(const struct btf_func_model *m, u8 **pprog, · 84 unmodified lines | ||
| 3213 | 3346 | static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_image, |
| 3214 | 3347 | void *rw_image_end, void *image, |
| 3215 | 3348 | const struct btf_func_model *m, u32 flags, |
| 3216 | − struct bpf_tramp_links *tlinks, | |
| 3349 | + struct bpf_tramp_nodes *tnodes, | |
| 3217 | 3350 | void *func_addr) |
| 3218 | 3351 | { |
| 3219 | 3352 | int i, ret, nr_regs = m->nr_args, stack_size = 0; |
| 3220 | 3353 | int regs_off, func_meta_off, ip_off, run_ctx_off, arg_stack_off, rbx_off; |
| 3221 | − struct bpf_tramp_links *fentry = &tlinks[BPF_TRAMP_FENTRY]; | |
| 3222 | − struct bpf_tramp_links *fexit = &tlinks[BPF_TRAMP_FEXIT]; | |
| 3223 | − struct bpf_tramp_links *fmod_ret = &tlinks[BPF_TRAMP_MODIFY_RETURN]; | |
| 3354 | + struct bpf_tramp_nodes *fentry = &tnodes[BPF_TRAMP_FENTRY]; | |
| 3355 | + struct bpf_tramp_nodes *fexit = &tnodes[BPF_TRAMP_FEXIT]; | |
| 3356 | + struct bpf_tramp_nodes *fmod_ret = &tnodes[BPF_TRAMP_MODIFY_RETURN]; | |
| 3224 | 3357 | void *orig_call = func_addr; |
| 3225 | 3358 | int cookie_off, cookie_cnt; |
| 3226 | 3359 | u8 **branches = NULL; |
| @@ -3292,7 +3425,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 65 unmodified lines | ||
| 3292 | 3425 | |
| 3293 | 3426 | ip_off = stack_size; |
| 3294 | 3427 | |
| 3295 | − cookie_cnt = bpf_fsession_cookie_cnt(tlinks); | |
| 3428 | + cookie_cnt = bpf_fsession_cookie_cnt(tnodes); | |
| 3296 | 3429 | /* room for session cookies */ |
| 3297 | 3430 | stack_size += cookie_cnt * 8; |
| 3298 | 3431 | cookie_off = stack_size; |
| @@ -3385,7 +3518,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 86 unmodified lines | ||
| 3385 | 3518 | } |
| 3386 | 3519 | } |
| 3387 | 3520 | |
| 3388 | − if (bpf_fsession_cnt(tlinks)) { | |
| 3521 | + if (bpf_fsession_cnt(tnodes)) { | |
| 3389 | 3522 | /* clear all the session cookies' value */ |
| 3390 | 3523 | for (int i = 0; i < cookie_cnt; i++) |
| 3391 | 3524 | emit_store_stack_imm64(&prog, BPF_REG_0, -cookie_off + 8 * i, 0); |
| @@ -3393,15 +3526,15 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 1 unmodified line | ||
| 3393 | 3526 | emit_store_stack_imm64(&prog, BPF_REG_0, -8, 0); |
| 3394 | 3527 | } |
| 3395 | 3528 | |
| 3396 | − if (fentry->nr_links) { | |
| 3529 | + if (fentry->nr_nodes) { | |
| 3397 | 3530 | if (invoke_bpf(m, &prog, fentry, regs_off, run_ctx_off, func_meta_off, |
| 3398 | 3531 | flags & BPF_TRAMP_F_RET_FENTRY_RET, image, rw_image, |
| 3399 | 3532 | func_meta, cookie_off)) |
| 3400 | 3533 | return -EINVAL; |
| 3401 | 3534 | } |
| 3402 | 3535 | |
| 3403 | − if (fmod_ret->nr_links) { | |
| 3404 | − branches = kcalloc(fmod_ret->nr_links, sizeof(u8 *), | |
| 3536 | + if (fmod_ret->nr_nodes) { | |
| 3537 | + branches = kcalloc(fmod_ret->nr_nodes, sizeof(u8 *), | |
| 3405 | 3538 | GFP_KERNEL); |
| 3406 | 3539 | if (!branches) |
| 3407 | 3540 | return -ENOMEM; |
| @@ -3440,7 +3573,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 32 unmodified lines | ||
| 3440 | 3573 | emit_nops(&prog, X86_PATCH_SIZE); |
| 3441 | 3574 | } |
| 3442 | 3575 | |
| 3443 | − if (fmod_ret->nr_links) { | |
| 3576 | + if (fmod_ret->nr_nodes) { | |
| 3444 | 3577 | /* From Intel 64 and IA-32 Architectures Optimization |
| 3445 | 3578 | * Reference Manual, 3.4.1.4 Code Alignment, Assembly/Compiler |
| 3446 | 3579 | * Coding Rule 11: All branch targets should be 16-byte |
| @@ -3450,7 +3583,7 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 3 unmodified lines | ||
| 3450 | 3583 | /* Update the branches saved in invoke_bpf_mod_ret with the |
| 3451 | 3584 | * aligned address of do_fexit. |
| 3452 | 3585 | */ |
| 3453 | − for (i = 0; i < fmod_ret->nr_links; i++) { | |
| 3586 | + for (i = 0; i < fmod_ret->nr_nodes; i++) { | |
| 3454 | 3587 | emit_cond_near_jump(&branches[i], image + (prog - (u8 *)rw_image), |
| 3455 | 3588 | image + (branches[i] - (u8 *)rw_image), X86_JNE); |
| 3456 | 3589 | } |
| @@ -3458,10 +3591,10 @@ static int __arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *rw_im · 1 unmodified line | ||
| 3458 | 3591 | |
| 3459 | 3592 | /* set the "is_return" flag for fsession */ |
| 3460 | 3593 | func_meta |= (1ULL << BPF_TRAMP_IS_RETURN_SHIFT); |
| 3461 | − if (bpf_fsession_cnt(tlinks)) | |
| 3594 | + if (bpf_fsession_cnt(tnodes)) | |
| 3462 | 3595 | emit_store_stack_imm64(&prog, BPF_REG_0, -func_meta_off, func_meta); |
| 3463 | 3596 | |
| 3464 | − if (fexit->nr_links) { | |
| 3597 | + if (fexit->nr_nodes) { | |
| 3465 | 3598 | if (invoke_bpf(m, &prog, fexit, regs_off, run_ctx_off, func_meta_off, |
| 3466 | 3599 | false, image, rw_image, func_meta, cookie_off)) { |
| 3467 | 3600 | ret = -EINVAL; |
| @@ -3535,7 +3668,7 @@ int arch_protect_bpf_trampoline(void *image, unsigned int size) · 67 unmodified lines | ||
| 3535 | 3668 | |
| 3536 | 3669 | int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *image_end, |
| 3537 | 3670 | const struct btf_func_model *m, u32 flags, |
| 3538 | − struct bpf_tramp_links *tlinks, | |
| 3671 | + struct bpf_tramp_nodes *tnodes, | |
| 3539 | 3672 | void *func_addr) |
| 3540 | 3673 | { |
| 3541 | 3674 | void *rw_image, *tmp; |
| @@ -3550,7 +3683,7 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i · 8 unmodified lines | ||
| 3550 | 3683 | return -ENOMEM; |
| 3551 | 3684 | |
| 3552 | 3685 | ret = __arch_prepare_bpf_trampoline(im, rw_image, rw_image + size, image, m, |
| 3553 | − flags, tlinks, func_addr); | |
| 3686 | + flags, tnodes, func_addr); | |
| 3554 | 3687 | if (ret < 0) |
| 3555 | 3688 | goto out; |
| 3556 | 3689 | |
| @@ -3563,7 +3696,7 @@ out: · 6 unmodified lines | ||
| 3563 | 3696 | } |
| 3564 | 3697 | |
| 3565 | 3698 | int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, |
| 3566 | − struct bpf_tramp_links *tlinks, void *func_addr) | |
| 3699 | + struct bpf_tramp_nodes *tnodes, void *func_addr) | |
| 3567 | 3700 | { |
| 3568 | 3701 | struct bpf_tramp_image im; |
| 3569 | 3702 | void *image; |
| @@ -3581,7 +3714,7 @@ int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, · 11 unmodified lines | ||
| 3581 | 3714 | return -ENOMEM; |
| 3582 | 3715 | |
| 3583 | 3716 | ret = __arch_prepare_bpf_trampoline(&im, image, image + PAGE_SIZE, image, |
| 3584 | − m, flags, tlinks, func_addr); | |
| 3717 | + m, flags, tnodes, func_addr); | |
| 3585 | 3718 | bpf_jit_free_exec(image); |
| 3586 | 3719 | return ret; |
| 3587 | 3720 | } |
| @@ -3793,7 +3926,8 @@ skip_init_addrs: · 205 unmodified lines | ||
| 3793 | 3926 | for (pass = 0; pass < MAX_PASSES || image; pass++) { |
| 3794 | 3927 | if (!padding && pass >= PADDING_PASSES) |
| 3795 | 3928 | padding = true; |
| 3796 | − proglen = do_jit(env, prog, addrs, image, rw_image, oldproglen, &ctx, padding); | |
| 3929 | + proglen = do_jit(env, prog, addrs, image, rw_image, oldproglen, | |
| 3930 | + &ctx, padding); | |
| 3797 | 3931 | if (proglen <= 0) { |
| 3798 | 3932 | out_image: |
| 3799 | 3933 | image = NULL; |
| @@ -3910,6 +4044,11 @@ bool bpf_jit_supports_kfunc_call(void) · 110 unmodified lines | ||
| 3910 | 4044 | return true; |
| 3911 | 4045 | } |
| 3912 | 4046 | |
| 4047 | +bool bpf_jit_supports_stack_args(void) | |
| 4048 | +{ | |
| 4049 | + return true; | |
| 4050 | +} | |
| 4051 | + | |
| 3913 | 4052 | void *bpf_arch_text_copy(void *dst, void *src, size_t len) |
| 3914 | 4053 | { |
| 3915 | 4054 | if (text_poke_copy(dst, src, len) == NULL) |
| @@ -168,3 +168,10 @@ static inline void device_pm_init(struct device *dev) · 167 unmodified lines | ||
| 168 | 168 | device_pm_sleep_init(dev); |
| 169 | 169 | pm_runtime_init(dev); |
| 170 | 170 | } |
| 171 | + | |
| 172 | +#ifdef CONFIG_BPF_SYSCALL | |
| 173 | +struct bpf_ws_lock { }; | |
| 174 | +struct bpf_ws_lock *bpf_wakeup_sources_read_lock(void); | |
| 175 | +void bpf_wakeup_sources_read_unlock(struct bpf_ws_lock *lock); | |
| 176 | +void *bpf_wakeup_sources_get_head(void); | |
| 177 | +#endif | |
| @@ -1168,11 +1168,78 @@ static const struct file_operations wakeup_sources_stats_fops = { · 1167 unmodified lines | ||
| 1168 | 1168 | .release = seq_release_private, |
| 1169 | 1169 | }; |
| 1170 | 1170 | |
| 1171 | −static int __init wakeup_sources_debugfs_init(void) | |
| 1171 | +#ifdef CONFIG_BPF_SYSCALL | |
| 1172 | +#include <linux/btf.h> | |
| 1173 | + | |
| 1174 | +__bpf_kfunc_start_defs(); | |
| 1175 | + | |
| 1176 | +/** | |
| 1177 | + * bpf_wakeup_sources_read_lock - Acquire the SRCU lock for wakeup sources | |
| 1178 | + * | |
| 1179 | + * The underlying SRCU lock returns an integer index. However, the BPF verifier | |
| 1180 | + * requires a pointer (PTR_TO_BTF_ID) to strictly track the state of acquired | |
| 1181 | + * resources using KF_ACQUIRE and KF_RELEASE semantics. We use an opaque | |
| 1182 | + * structure pointer (struct bpf_ws_lock *) to satisfy the verifier while | |
| 1183 | + * safely encoding the integer index within the pointer address itself. | |
| 1184 | + * | |
| 1185 | + * Return: An opaque pointer encoding the SRCU lock index + 1 (to avoid NULL). | |
| 1186 | + */ | |
| 1187 | +__bpf_kfunc struct bpf_ws_lock *bpf_wakeup_sources_read_lock(void) | |
| 1188 | +{ | |
| 1189 | + return (struct bpf_ws_lock *)(long)(wakeup_sources_read_lock() + 1); | |
| 1190 | +} | |
| 1191 | + | |
| 1192 | +/** | |
| 1193 | + * bpf_wakeup_sources_read_unlock - Release the SRCU lock for wakeup sources | |
| 1194 | + * @lock: The opaque pointer returned by bpf_wakeup_sources_read_lock() | |
| 1195 | + * | |
| 1196 | + * The BPF verifier guarantees that @lock is a valid, unreleased pointer from | |
| 1197 | + * the acquire function. We decode the pointer back into the integer SRCU index | |
| 1198 | + * by subtracting 1 and release the lock. | |
| 1199 | + */ | |
| 1200 | +__bpf_kfunc void bpf_wakeup_sources_read_unlock(struct bpf_ws_lock *lock) | |
| 1201 | +{ | |
| 1202 | + wakeup_sources_read_unlock((int)(long)lock - 1); | |
| 1203 | +} | |
| 1204 | + | |
| 1205 | +/** | |
| 1206 | + * bpf_wakeup_sources_get_head - Get the head of the wakeup sources list | |
| 1207 | + * | |
| 1208 | + * Return: The head of the wakeup sources list. | |
| 1209 | + */ | |
| 1210 | +__bpf_kfunc void *bpf_wakeup_sources_get_head(void) | |
| 1211 | +{ | |
| 1212 | + return &wakeup_sources; | |
| 1213 | +} | |
| 1214 | + | |
| 1215 | +__bpf_kfunc_end_defs(); | |
| 1216 | + | |
| 1217 | +BTF_KFUNCS_START(wakeup_source_kfunc_ids) | |
| 1218 | +BTF_ID_FLAGS(func, bpf_wakeup_sources_read_lock, KF_ACQUIRE) | |
| 1219 | +BTF_ID_FLAGS(func, bpf_wakeup_sources_read_unlock, KF_RELEASE) | |
| 1220 | +BTF_ID_FLAGS(func, bpf_wakeup_sources_get_head) | |
| 1221 | +BTF_KFUNCS_END(wakeup_source_kfunc_ids) | |
| 1222 | + | |
| 1223 | +static const struct btf_kfunc_id_set wakeup_source_kfunc_set = { | |
| 1224 | + .set = &wakeup_source_kfunc_ids, | |
| 1225 | +}; | |
| 1226 | + | |
| 1227 | +static void __init wakeup_sources_bpf_init(void) | |
| 1228 | +{ | |
| 1229 | + if (register_btf_kfunc_id_set(BPF_PROG_TYPE_SYSCALL, &wakeup_source_kfunc_set)) | |
| 1230 | + pm_pr_dbg("Wakeup: failed to register BTF kfuncs\n"); | |
| 1231 | +} | |
| 1232 | +#else | |
| 1233 | +static inline void wakeup_sources_bpf_init(void) {} | |
| 1234 | +#endif /* CONFIG_BPF_SYSCALL */ | |
| 1235 | + | |
| 1236 | +static int __init wakeup_sources_init(void) | |
| 1172 | 1237 | { |
| 1173 | 1238 | debugfs_create_file("wakeup_sources", 0444, NULL, NULL, |
| 1174 | 1239 | &wakeup_sources_stats_fops); |
| 1240 | + wakeup_sources_bpf_init(); | |
| 1241 | + | |
| 1175 | 1242 | return 0; |
| 1176 | 1243 | } |
| 1177 | 1244 | |
| 1178 | −postcore_initcall(wakeup_sources_debugfs_init); | |
| 1245 | +postcore_initcall(wakeup_sources_init); | |
| @@ -561,10 +561,10 @@ nfp_bpf_check_alu(struct nfp_prog *nfp_prog, struct nfp_insn_meta *meta, · 560 unmodified lines | ||
| 561 | 561 | const struct bpf_reg_state *dreg = |
| 562 | 562 | cur_regs(env) + meta->insn.dst_reg; |
| 563 | 563 | |
| 564 | − meta->umin_src = min(meta->umin_src, sreg->umin_value); | |
| 565 | − meta->umax_src = max(meta->umax_src, sreg->umax_value); | |
| 566 | − meta->umin_dst = min(meta->umin_dst, dreg->umin_value); | |
| 567 | − meta->umax_dst = max(meta->umax_dst, dreg->umax_value); | |
| 564 | + meta->umin_src = min(meta->umin_src, reg_umin(sreg)); | |
| 565 | + meta->umax_src = max(meta->umax_src, reg_umax(sreg)); | |
| 566 | + meta->umin_dst = min(meta->umin_dst, reg_umin(dreg)); | |
| 567 | + meta->umax_dst = max(meta->umax_dst, reg_umax(dreg)); | |
| 568 | 568 | |
| 569 | 569 | /* NFP supports u16 and u32 multiplication. |
| 570 | 570 | * |
| @@ -200,7 +200,7 @@ int bpf_set_dentry_xattr_locked(struct dentry *dentry, const char *name__str, · 199 unmodified lines | ||
| 200 | 200 | const struct bpf_dynptr *value_p, int flags) |
| 201 | 201 | { |
| 202 | 202 | |
| 203 | − struct bpf_dynptr_kern *value_ptr = (struct bpf_dynptr_kern *)value_p; | |
| 203 | + const struct bpf_dynptr_kern *value_ptr = (struct bpf_dynptr_kern *)value_p; | |
| 204 | 204 | struct inode *inode = d_inode(dentry); |
| 205 | 205 | const void *value; |
| 206 | 206 | u32 value_len; |
| @@ -118,9 +118,9 @@ __bpf_kfunc_start_defs(); · 117 unmodified lines | ||
| 118 | 118 | * |
| 119 | 119 | * Return: 0 on success, a negative value on error. |
| 120 | 120 | */ |
| 121 | −__bpf_kfunc int bpf_get_fsverity_digest(struct file *file, struct bpf_dynptr *digest_p) | |
| 121 | +__bpf_kfunc int bpf_get_fsverity_digest(struct file *file, const struct bpf_dynptr *digest_p) | |
| 122 | 122 | { |
| 123 | − struct bpf_dynptr_kern *digest_ptr = (struct bpf_dynptr_kern *)digest_p; | |
| 123 | + const struct bpf_dynptr_kern *digest_ptr = (struct bpf_dynptr_kern *)digest_p; | |
| 124 | 124 | const struct inode *inode = file_inode(file); |
| 125 | 125 | u32 dynptr_sz = __bpf_dynptr_size(digest_ptr); |
| 126 | 126 | struct fsverity_digest *arg; |
| @@ -243,12 +243,20 @@ static __always_inline void res_spin_unlock(rqspinlock_t *lock) · 242 unmodified lines | ||
| 243 | 243 | ({ \ |
| 244 | 244 | int __ret; \ |
| 245 | 245 | local_irq_save(flags); \ |
| 246 | − __ret = raw_res_spin_lock(lock); \ | |
| 247 | − if (__ret) \ | |
| 246 | + preempt_disable(); \ | |
| 247 | + __ret = res_spin_lock(lock); \ | |
| 248 | + if (__ret) { \ | |
| 248 | 249 | local_irq_restore(flags); \ |
| 250 | + preempt_enable(); \ | |
| 251 | + } \ | |
| 249 | 252 | __ret; \ |
| 250 | 253 | }) |
| 251 | 254 | |
| 252 | −#define raw_res_spin_unlock_irqrestore(lock, flags) ({ raw_res_spin_unlock(lock); local_irq_restore(flags); }) | |
| 255 | +#define raw_res_spin_unlock_irqrestore(lock, flags) \ | |
| 256 | + ({ \ | |
| 257 | + res_spin_unlock(lock); \ | |
| 258 | + local_irq_restore(flags); \ | |
| 259 | + preempt_enable(); \ | |
| 260 | + }) | |
| 253 | 261 | |
| 254 | 262 | #endif /* __ASM_GENERIC_RQSPINLOCK_H */ |
| @@ -421,7 +421,7 @@ int cgroup_bpf_prog_detach(const union bpf_attr *attr, · 420 unmodified lines | ||
| 421 | 421 | enum bpf_prog_type ptype); |
| 422 | 422 | int cgroup_bpf_link_attach(const union bpf_attr *attr, struct bpf_prog *prog); |
| 423 | 423 | int cgroup_bpf_prog_query(const union bpf_attr *attr, |
| 424 | − union bpf_attr __user *uattr); | |
| 424 | + union bpf_attr __user *uattr, u32 uattr_size); | |
| 425 | 425 | |
| 426 | 426 | const struct bpf_func_proto * |
| 427 | 427 | cgroup_common_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog); |
| @@ -452,7 +452,8 @@ static inline int cgroup_bpf_link_attach(const union bpf_attr *attr, · 24 unmodified lines | ||
| 452 | 452 | } |
| 453 | 453 | |
| 454 | 454 | static inline int cgroup_bpf_prog_query(const union bpf_attr *attr, |
| 455 | − union bpf_attr __user *uattr) | |
| 455 | + union bpf_attr __user *uattr, | |
| 456 | + u32 uattr_size) | |
| 456 | 457 | { |
| 457 | 458 | return -EINVAL; |
| 458 | 459 | } |
| @@ -6,6 +6,7 @@ · 5 unmodified lines | ||
| 6 | 6 | |
| 7 | 7 | #include <uapi/linux/bpf.h> |
| 8 | 8 | #include <uapi/linux/filter.h> |
| 9 | +#include <linux/bpf_defs.h> | |
| 9 | 10 | |
| 10 | 11 | #include <crypto/sha2.h> |
| 11 | 12 | #include <linux/workqueue.h> |
| @@ -32,6 +33,8 @@ · 20 unmodified lines | ||
| 32 | 33 | #include <linux/memcontrol.h> |
| 33 | 34 | #include <linux/cfi.h> |
| 34 | 35 | #include <linux/xattr.h> |
| 36 | +#include <linux/key.h> | |
| 37 | +#include <linux/ftrace.h> | |
| 35 | 38 | #include <asm/rqspinlock.h> |
| 36 | 39 | |
| 37 | 40 | struct bpf_verifier_env; |
| @@ -111,7 +114,7 @@ struct bpf_map_ops { · 73 unmodified lines | ||
| 111 | 114 | long (*map_pop_elem)(struct bpf_map *map, void *value); |
| 112 | 115 | long (*map_peek_elem)(struct bpf_map *map, void *value); |
| 113 | 116 | void *(*map_lookup_percpu_elem)(struct bpf_map *map, void *key, u32 cpu); |
| 114 | − int (*map_get_hash)(struct bpf_map *map, u32 hash_buf_size, void *hash_buf); | |
| 117 | + int (*map_get_hash)(struct bpf_map *map); | |
| 115 | 118 | |
| 116 | 119 | /* funcs called by prog_array and perf_event_array map */ |
| 117 | 120 | void *(*map_fd_get_ptr)(struct bpf_map *map, struct file *map_file, |
| @@ -296,6 +299,7 @@ struct bpf_map_owner { · 178 unmodified lines | ||
| 296 | 299 | |
| 297 | 300 | struct bpf_map { |
| 298 | 301 | u8 sha[SHA256_DIGEST_SIZE]; |
| 302 | + u32 excl; | |
| 299 | 303 | const struct bpf_map_ops *ops; |
| 300 | 304 | struct bpf_map *inner_map_meta; |
| 301 | 305 | #ifdef CONFIG_SECURITY |
| @@ -489,6 +493,35 @@ static inline bool btf_record_has_field(const struct btf_record *rec, enum btf_f · 187 unmodified lines | ||
| 489 | 493 | return rec->field_mask & type; |
| 490 | 494 | } |
| 491 | 495 | |
| 496 | +static inline bool btf_field_is_nmi_safe(enum btf_field_type type) | |
| 497 | +{ | |
| 498 | + switch (type) { | |
| 499 | + case BPF_SPIN_LOCK: | |
| 500 | + case BPF_RES_SPIN_LOCK: | |
| 501 | + case BPF_TIMER: | |
| 502 | + case BPF_WORKQUEUE: | |
| 503 | + case BPF_TASK_WORK: | |
| 504 | + case BPF_KPTR_UNREF: | |
| 505 | + case BPF_REFCOUNT: | |
| 506 | + return true; | |
| 507 | + default: | |
| 508 | + return false; | |
| 509 | + } | |
| 510 | +} | |
| 511 | + | |
| 512 | +static inline bool btf_record_has_nmi_unsafe_fields(const struct btf_record *rec) | |
| 513 | +{ | |
| 514 | + int i; | |
| 515 | + | |
| 516 | + if (IS_ERR_OR_NULL(rec)) | |
| 517 | + return false; | |
| 518 | + for (i = 0; i < rec->cnt; i++) { | |
| 519 | + if (!btf_field_is_nmi_safe(rec->fields[i].type)) | |
| 520 | + return true; | |
| 521 | + } | |
| 522 | + return false; | |
| 523 | +} | |
| 524 | + | |
| 492 | 525 | static inline void bpf_obj_init(const struct btf_record *rec, void *obj) |
| 493 | 526 | { |
| 494 | 527 | int i; |
| @@ -618,6 +651,8 @@ void bpf_rb_root_free(const struct btf_field *field, void *rb_root, · 123 unmodified lines | ||
| 618 | 651 | struct bpf_spin_lock *spin_lock); |
| 619 | 652 | u64 bpf_arena_get_kern_vm_start(struct bpf_arena *arena); |
| 620 | 653 | u64 bpf_arena_get_user_vm_start(struct bpf_arena *arena); |
| 654 | +u64 bpf_arena_map_kern_vm_start(struct bpf_map *map); | |
| 655 | +struct bpf_map *bpf_prog_arena(struct bpf_prog *prog); | |
| 621 | 656 | int bpf_obj_name_cpy(char *dst, const char *src, unsigned int size); |
| 622 | 657 | |
| 623 | 658 | struct bpf_offload_dev; |
| @@ -679,6 +714,8 @@ int bpf_dynptr_from_file_sleepable(struct file *file, u32 flags, · 55 unmodified lines | ||
| 679 | 714 | void *bpf_arena_alloc_pages_non_sleepable(void *p__map, void *addr__ign, u32 page_cnt, int node_id, |
| 680 | 715 | u64 flags); |
| 681 | 716 | void bpf_arena_free_pages_non_sleepable(void *p__map, void *ptr__ign, u32 page_cnt); |
| 717 | +void *bpf_arena_alloc_pages_sleepable(void *p__map, void *addr__ign, u32 page_cnt, int node_id, | |
| 718 | + u64 flags); | |
| 682 | 719 | #else |
| 683 | 720 | static inline void *bpf_arena_alloc_pages_non_sleepable(void *p__map, void *addr__ign, u32 page_cnt, |
| 684 | 721 | int node_id, u64 flags) |
| @@ -689,6 +726,12 @@ static inline void *bpf_arena_alloc_pages_non_sleepable(void *p__map, void *addr · 4 unmodified lines | ||
| 689 | 726 | static inline void bpf_arena_free_pages_non_sleepable(void *p__map, void *ptr__ign, u32 page_cnt) |
| 690 | 727 | { |
| 691 | 728 | } |
| 729 | + | |
| 730 | +static inline void *bpf_arena_alloc_pages_sleepable(void *p__map, void *addr__ign, u32 page_cnt, | |
| 731 | + int node_id, u64 flags) | |
| 732 | +{ | |
| 733 | + return NULL; | |
| 734 | +} | |
| 692 | 735 | #endif |
| 693 | 736 | |
| 694 | 737 | extern const struct bpf_map_ops bpf_map_offload_ops; |
| @@ -1052,7 +1095,7 @@ struct bpf_insn_access_aux { · 357 unmodified lines | ||
| 1052 | 1095 | struct { |
| 1053 | 1096 | struct btf *btf; |
| 1054 | 1097 | u32 btf_id; |
| 1055 | − u32 ref_obj_id; | |
| 1098 | + u32 ref_id; | |
| 1056 | 1099 | }; |
| 1057 | 1100 | }; |
| 1058 | 1101 | struct bpf_verifier_log *log; /* for verbose logs */ |
| @@ -1152,6 +1195,11 @@ struct bpf_prog_offload { · 93 unmodified lines | ||
| 1152 | 1195 | |
| 1153 | 1196 | /* The longest tracepoint has 12 args. |
| 1154 | 1197 | * See include/trace/bpf_probe.h |
| 1198 | + * | |
| 1199 | + * Also reuse this macro for maximum number of arguments a BPF function | |
| 1200 | + * or a kfunc can have. Args 1-5 are passed in registers, args 6-12 via | |
| 1201 | + * stack arg slots. The JIT may map some stack arg slots to registers based | |
| 1202 | + * on the native calling convention (e.g., arg 6 to R9 on x86-64). | |
| 1155 | 1203 | */ |
| 1156 | 1204 | #define MAX_BPF_FUNC_ARGS 12 |
| 1157 | 1205 | |
| @@ -1234,9 +1282,9 @@ enum { · 76 unmodified lines | ||
| 1234 | 1282 | #define BPF_TRAMP_COOKIE_INDEX_SHIFT 8 |
| 1235 | 1283 | #define BPF_TRAMP_IS_RETURN_SHIFT 63 |
| 1236 | 1284 | |
| 1237 | −struct bpf_tramp_links { | |
| 1238 | − struct bpf_tramp_link *links[BPF_MAX_TRAMP_LINKS]; | |
| 1239 | − int nr_links; | |
| 1285 | +struct bpf_tramp_nodes { | |
| 1286 | + struct bpf_tramp_node *nodes[BPF_MAX_TRAMP_LINKS]; | |
| 1287 | + int nr_nodes; | |
| 1240 | 1288 | }; |
| 1241 | 1289 | |
| 1242 | 1290 | struct bpf_tramp_run_ctx; |
| @@ -1264,13 +1312,13 @@ struct bpf_tramp_run_ctx; · 21 unmodified lines | ||
| 1264 | 1312 | struct bpf_tramp_image; |
| 1265 | 1313 | int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *image_end, |
| 1266 | 1314 | const struct btf_func_model *m, u32 flags, |
| 1267 | − struct bpf_tramp_links *tlinks, | |
| 1315 | + struct bpf_tramp_nodes *tnodes, | |
| 1268 | 1316 | void *func_addr); |
| 1269 | 1317 | void *arch_alloc_bpf_trampoline(unsigned int size); |
| 1270 | 1318 | void arch_free_bpf_trampoline(void *image, unsigned int size); |
| 1271 | 1319 | int __must_check arch_protect_bpf_trampoline(void *image, unsigned int size); |
| 1272 | 1320 | int arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, |
| 1273 | − struct bpf_tramp_links *tlinks, void *func_addr); | |
| 1321 | + struct bpf_tramp_nodes *tnodes, void *func_addr); | |
| 1274 | 1322 | |
| 1275 | 1323 | u64 notrace __bpf_prog_enter_sleepable_recur(struct bpf_prog *prog, |
| 1276 | 1324 | struct bpf_tramp_run_ctx *run_ctx); |
| @@ -1336,8 +1384,6 @@ struct bpf_trampoline { · 59 unmodified lines | ||
| 1336 | 1384 | /* hlist for trampoline_ip_table */ |
| 1337 | 1385 | struct hlist_node hlist_ip; |
| 1338 | 1386 | struct ftrace_ops *fops; |
| 1339 | − /* serializes access to fields of this trampoline */ | |
| 1340 | − struct mutex mutex; | |
| 1341 | 1387 | refcount_t refcnt; |
| 1342 | 1388 | u32 flags; |
| 1343 | 1389 | u64 key; |
| @@ -1358,6 +1404,11 @@ struct bpf_trampoline { · 14 unmodified lines | ||
| 1358 | 1404 | int progs_cnt[BPF_TRAMP_MAX]; |
| 1359 | 1405 | /* Executable image of trampoline */ |
| 1360 | 1406 | struct bpf_tramp_image *cur_image; |
| 1407 | + /* Used as temporary old image storage for multi_attach */ | |
| 1408 | + struct { | |
| 1409 | + struct bpf_tramp_image *old_image; | |
| 1410 | + u32 old_flags; | |
| 1411 | + } multi_attach; | |
| 1361 | 1412 | }; |
| 1362 | 1413 | |
| 1363 | 1414 | struct bpf_attach_target_info { |
| @@ -1455,11 +1506,13 @@ static inline int bpf_dynptr_check_off_len(const struct bpf_dynptr_kern *ptr, u6 · 91 unmodified lines | ||
| 1455 | 1506 | return 0; |
| 1456 | 1507 | } |
| 1457 | 1508 | |
| 1509 | +struct bpf_tracing_multi_link; | |
| 1510 | + | |
| 1458 | 1511 | #ifdef CONFIG_BPF_JIT |
| 1459 | −int bpf_trampoline_link_prog(struct bpf_tramp_link *link, | |
| 1512 | +int bpf_trampoline_link_prog(struct bpf_tramp_node *node, | |
| 1460 | 1513 | struct bpf_trampoline *tr, |
| 1461 | 1514 | struct bpf_prog *tgt_prog); |
| 1462 | −int bpf_trampoline_unlink_prog(struct bpf_tramp_link *link, | |
| 1515 | +int bpf_trampoline_unlink_prog(struct bpf_tramp_node *node, | |
| 1463 | 1516 | struct bpf_trampoline *tr, |
| 1464 | 1517 | struct bpf_prog *tgt_prog); |
| 1465 | 1518 | struct bpf_trampoline *bpf_trampoline_get(u64 key, |
| @@ -1467,6 +1520,11 @@ struct bpf_trampoline *bpf_trampoline_get(u64 key, · 1 unmodified line | ||
| 1467 | 1520 | void bpf_trampoline_put(struct bpf_trampoline *tr); |
| 1468 | 1521 | int arch_prepare_bpf_dispatcher(void *image, void *buf, s64 *funcs, int num_funcs); |
| 1469 | 1522 | |
| 1523 | +int bpf_trampoline_multi_attach(struct bpf_prog *prog, u32 *ids, | |
| 1524 | + struct bpf_tracing_multi_link *link); | |
| 1525 | +int bpf_trampoline_multi_detach(struct bpf_prog *prog, | |
| 1526 | + struct bpf_tracing_multi_link *link); | |
| 1527 | + | |
| 1470 | 1528 | /* |
| 1471 | 1529 | * When the architecture supports STATIC_CALL replace the bpf_dispatcher_fn |
| 1472 | 1530 | * indirection with a direct call to the bpf program. If the architecture does |
| @@ -1544,14 +1602,15 @@ void bpf_jit_uncharge_modmem(u32 size); · 71 unmodified lines | ||
| 1544 | 1602 | bool bpf_prog_has_trampoline(const struct bpf_prog *prog); |
| 1545 | 1603 | bool bpf_insn_is_indirect_target(const struct bpf_verifier_env *env, const struct bpf_prog *prog, |
| 1546 | 1604 | int insn_idx); |
| 1605 | +u16 bpf_out_stack_arg_cnt(const struct bpf_verifier_env *env, const struct bpf_prog *prog); | |
| 1547 | 1606 | #else |
| 1548 | −static inline int bpf_trampoline_link_prog(struct bpf_tramp_link *link, | |
| 1607 | +static inline int bpf_trampoline_link_prog(struct bpf_tramp_node *node, | |
| 1549 | 1608 | struct bpf_trampoline *tr, |
| 1550 | 1609 | struct bpf_prog *tgt_prog) |
| 1551 | 1610 | { |
| 1552 | 1611 | return -ENOTSUPP; |
| 1553 | 1612 | } |
| 1554 | −static inline int bpf_trampoline_unlink_prog(struct bpf_tramp_link *link, | |
| 1613 | +static inline int bpf_trampoline_unlink_prog(struct bpf_tramp_node *node, | |
| 1555 | 1614 | struct bpf_trampoline *tr, |
| 1556 | 1615 | struct bpf_prog *tgt_prog) |
| 1557 | 1616 | { |
| @@ -1578,6 +1637,16 @@ static inline bool bpf_prog_has_trampoline(const struct bpf_prog *prog) · 20 unmodified lines | ||
| 1578 | 1637 | { |
| 1579 | 1638 | return false; |
| 1580 | 1639 | } |
| 1640 | +static inline int bpf_trampoline_multi_attach(struct bpf_prog *prog, u32 *ids, | |
| 1641 | + struct bpf_tracing_multi_link *link) | |
| 1642 | +{ | |
| 1643 | + return -ENOTSUPP; | |
| 1644 | +} | |
| 1645 | +static inline int bpf_trampoline_multi_detach(struct bpf_prog *prog, | |
| 1646 | + struct bpf_tracing_multi_link *link) | |
| 1647 | +{ | |
| 1648 | + return -ENOTSUPP; | |
| 1649 | +} | |
| 1581 | 1650 | #endif |
| 1582 | 1651 | |
| 1583 | 1652 | struct bpf_func_info_aux { |
| @@ -1615,7 +1684,7 @@ struct bpf_ctx_arg_aux { · 31 unmodified lines | ||
| 1615 | 1684 | enum bpf_reg_type reg_type; |
| 1616 | 1685 | struct btf *btf; |
| 1617 | 1686 | u32 btf_id; |
| 1618 | − u32 ref_obj_id; | |
| 1687 | + u32 ref_id; | |
| 1619 | 1688 | bool refcounted; |
| 1620 | 1689 | }; |
| 1621 | 1690 | |
| @@ -1657,6 +1726,19 @@ struct bpf_stream_stage { · 35 unmodified lines | ||
| 1657 | 1726 | int len; |
| 1658 | 1727 | }; |
| 1659 | 1728 | |
| 1729 | +enum bpf_sig_verdict { | |
| 1730 | + BPF_SIG_UNSIGNED = 0, | |
| 1731 | + BPF_SIG_VERIFIED, | |
| 1732 | +}; | |
| 1733 | + | |
| 1734 | +enum bpf_sig_keyring { | |
| 1735 | + BPF_SIG_KEYRING_NONE = 0, | |
| 1736 | + BPF_SIG_KEYRING_BUILTIN, | |
| 1737 | + BPF_SIG_KEYRING_SECONDARY, | |
| 1738 | + BPF_SIG_KEYRING_PLATFORM, | |
| 1739 | + BPF_SIG_KEYRING_USER, | |
| 1740 | +}; | |
| 1741 | + | |
| 1660 | 1742 | struct bpf_prog_aux { |
| 1661 | 1743 | atomic64_t refcnt; |
| 1662 | 1744 | u32 used_map_cnt; |
| @@ -1699,6 +1781,11 @@ struct bpf_prog_aux { · 36 unmodified lines | ||
| 1699 | 1781 | bool changes_pkt_data; |
| 1700 | 1782 | bool might_sleep; |
| 1701 | 1783 | bool kprobe_write_ctx; |
| 1784 | + struct { | |
| 1785 | + s32 keyring_serial; | |
| 1786 | + u8 keyring_type; | |
| 1787 | + u8 verdict; | |
| 1788 | + } sig; | |
| 1702 | 1789 | u64 prog_array_member_cnt; /* counts how many times as member of prog_array */ |
| 1703 | 1790 | struct mutex ext_mutex; /* mutex for is_extended and prog_array_member_cnt */ |
| 1704 | 1791 | struct bpf_arena *arena; |
| @@ -1731,6 +1818,7 @@ struct bpf_prog_aux { · 26 unmodified lines | ||
| 1731 | 1818 | struct bpf_map *cgroup_storage[MAX_BPF_CGROUP_STORAGE_TYPE]; |
| 1732 | 1819 | char name[BPF_OBJ_NAME_LEN]; |
| 1733 | 1820 | u64 (*bpf_exception_cb)(u64 cookie, u64 sp, u64 bp, u64, u64); |
| 1821 | + u16 stack_arg_sp_adjust; | |
| 1734 | 1822 | #ifdef CONFIG_SECURITY |
| 1735 | 1823 | void *security; |
| 1736 | 1824 | #endif |
| @@ -1874,12 +1962,17 @@ struct bpf_link_ops { · 137 unmodified lines | ||
| 1874 | 1962 | __poll_t (*poll)(struct file *file, struct poll_table_struct *pts); |
| 1875 | 1963 | }; |
| 1876 | 1964 | |
| 1877 | −struct bpf_tramp_link { | |
| 1878 | − struct bpf_link link; | |
| 1965 | +struct bpf_tramp_node { | |
| 1966 | + struct bpf_link *link; | |
| 1879 | 1967 | struct hlist_node tramp_hlist; |
| 1880 | 1968 | u64 cookie; |
| 1881 | 1969 | }; |
| 1882 | 1970 | |
| 1971 | +struct bpf_tramp_link { | |
| 1972 | + struct bpf_link link; | |
| 1973 | + struct bpf_tramp_node node; | |
| 1974 | +}; | |
| 1975 | + | |
| 1883 | 1976 | struct bpf_shim_tramp_link { |
| 1884 | 1977 | struct bpf_tramp_link link; |
| 1885 | 1978 | struct bpf_trampoline *trampoline; |
| @@ -1887,13 +1980,31 @@ struct bpf_shim_tramp_link { · 1 unmodified line | ||
| 1887 | 1980 | |
| 1888 | 1981 | struct bpf_tracing_link { |
| 1889 | 1982 | struct bpf_tramp_link link; |
| 1983 | + struct bpf_tramp_node fexit; | |
| 1890 | 1984 | struct bpf_trampoline *trampoline; |
| 1891 | 1985 | struct bpf_prog *tgt_prog; |
| 1892 | 1986 | }; |
| 1893 | 1987 | |
| 1894 | −struct bpf_fsession_link { | |
| 1895 | − struct bpf_tracing_link link; | |
| 1896 | − struct bpf_tramp_link fexit; | |
| 1988 | +struct bpf_tracing_multi_node { | |
| 1989 | + struct bpf_tramp_node node; | |
| 1990 | + struct bpf_trampoline *trampoline; | |
| 1991 | + struct ftrace_func_entry entry; | |
| 1992 | +}; | |
| 1993 | + | |
| 1994 | +struct bpf_tracing_multi_data { | |
| 1995 | + struct ftrace_hash *unreg; | |
| 1996 | + struct ftrace_hash *modify; | |
| 1997 | + struct ftrace_hash *reg; | |
| 1998 | + struct ftrace_func_entry *entry; | |
| 1999 | +}; | |
| 2000 | + | |
| 2001 | +struct bpf_tracing_multi_link { | |
| 2002 | + struct bpf_link link; | |
| 2003 | + struct bpf_tracing_multi_data data; | |
| 2004 | + u64 *cookies; | |
| 2005 | + struct bpf_tramp_node *fexits; | |
| 2006 | + int nodes_cnt; | |
| 2007 | + struct bpf_tracing_multi_node nodes[] __counted_by(nodes_cnt); | |
| 1897 | 2008 | }; |
| 1898 | 2009 | |
| 1899 | 2010 | struct bpf_raw_tp_link { |
| @@ -2079,6 +2190,12 @@ static inline void bpf_prog_put_recursion_context(struct bpf_prog *prog) · 179 unmodified lines | ||
| 2079 | 2190 | #endif |
| 2080 | 2191 | } |
| 2081 | 2192 | |
| 2193 | +static inline bool is_tracing_multi(enum bpf_attach_type type) | |
| 2194 | +{ | |
| 2195 | + return type == BPF_TRACE_FENTRY_MULTI || type == BPF_TRACE_FEXIT_MULTI || | |
| 2196 | + type == BPF_TRACE_FSESSION_MULTI; | |
| 2197 | +} | |
| 2198 | + | |
| 2082 | 2199 | #if defined(CONFIG_BPF_JIT) && defined(CONFIG_BPF_SYSCALL) |
| 2083 | 2200 | /* This macro helps developer to register a struct_ops type and generate |
| 2084 | 2201 | * type information correctly. Developers should use this macro to register |
| @@ -2099,8 +2216,8 @@ void bpf_struct_ops_put(const void *kdata); · 14 unmodified lines | ||
| 2099 | 2216 | int bpf_struct_ops_supported(const struct bpf_struct_ops *st_ops, u32 moff); |
| 2100 | 2217 | int bpf_struct_ops_map_sys_lookup_elem(struct bpf_map *map, void *key, |
| 2101 | 2218 | void *value); |
| 2102 | −int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_links *tlinks, | |
| 2103 | − struct bpf_tramp_link *link, | |
| 2219 | +int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_nodes *tnodes, | |
| 2220 | + struct bpf_tramp_node *node, | |
| 2104 | 2221 | const struct btf_func_model *model, |
| 2105 | 2222 | void *stub_func, |
| 2106 | 2223 | void **image, u32 *image_off, |
| @@ -2125,6 +2242,9 @@ int bpf_prog_assoc_struct_ops(struct bpf_prog *prog, struct bpf_map *map); · 18 unmodified lines | ||
| 2125 | 2242 | void bpf_prog_disassoc_struct_ops(struct bpf_prog *prog); |
| 2126 | 2243 | void *bpf_prog_get_assoc_struct_ops(const struct bpf_prog_aux *aux); |
| 2127 | 2244 | u32 bpf_struct_ops_id(const void *kdata); |
| 2245 | +int bpf_struct_ops_for_each_prog(const void *kdata, | |
| 2246 | + int (*cb)(struct bpf_prog *prog, void *data), | |
| 2247 | + void *data); | |
| 2128 | 2248 | |
| 2129 | 2249 | #ifdef CONFIG_NET |
| 2130 | 2250 | /* Define it here to avoid the use of forward declaration */ |
| @@ -2192,31 +2312,33 @@ static inline void bpf_struct_ops_desc_release(struct bpf_struct_ops_desc *st_op · 61 unmodified lines | ||
| 2192 | 2312 | |
| 2193 | 2313 | #endif |
| 2194 | 2314 | |
| 2195 | −static inline int bpf_fsession_cnt(struct bpf_tramp_links *links) | |
| 2315 | +static inline int bpf_fsession_cnt(struct bpf_tramp_nodes *nodes) | |
| 2196 | 2316 | { |
| 2197 | − struct bpf_tramp_links fentries = links[BPF_TRAMP_FENTRY]; | |
| 2317 | + struct bpf_tramp_nodes fentries = nodes[BPF_TRAMP_FENTRY]; | |
| 2198 | 2318 | int cnt = 0; |
| 2199 | 2319 | |
| 2200 | − for (int i = 0; i < links[BPF_TRAMP_FENTRY].nr_links; i++) { | |
| 2201 | − if (fentries.links[i]->link.prog->expected_attach_type == BPF_TRACE_FSESSION) | |
| 2320 | + for (int i = 0; i < nodes[BPF_TRAMP_FENTRY].nr_nodes; i++) { | |
| 2321 | + if (fentries.nodes[i]->link->prog->expected_attach_type == BPF_TRACE_FSESSION) | |
| 2322 | + cnt++; | |
| 2323 | + if (fentries.nodes[i]->link->prog->expected_attach_type == BPF_TRACE_FSESSION_MULTI) | |
| 2202 | 2324 | cnt++; |
| 2203 | 2325 | } |
| 2204 | 2326 | |
| 2205 | 2327 | return cnt; |
| 2206 | 2328 | } |
| 2207 | 2329 | |
| 2208 | −static inline bool bpf_prog_calls_session_cookie(struct bpf_tramp_link *link) | |
| 2330 | +static inline bool bpf_prog_calls_session_cookie(struct bpf_tramp_node *node) | |
| 2209 | 2331 | { |
| 2210 | − return link->link.prog->call_session_cookie; | |
| 2332 | + return node->link->prog->call_session_cookie; | |
| 2211 | 2333 | } |
| 2212 | 2334 | |
| 2213 | −static inline int bpf_fsession_cookie_cnt(struct bpf_tramp_links *links) | |
| 2335 | +static inline int bpf_fsession_cookie_cnt(struct bpf_tramp_nodes *nodes) | |
| 2214 | 2336 | { |
| 2215 | − struct bpf_tramp_links fentries = links[BPF_TRAMP_FENTRY]; | |
| 2337 | + struct bpf_tramp_nodes fentries = nodes[BPF_TRAMP_FENTRY]; | |
| 2216 | 2338 | int cnt = 0; |
| 2217 | 2339 | |
| 2218 | − for (int i = 0; i < links[BPF_TRAMP_FENTRY].nr_links; i++) { | |
| 2219 | − if (bpf_prog_calls_session_cookie(fentries.links[i])) | |
| 2340 | + for (int i = 0; i < nodes[BPF_TRAMP_FENTRY].nr_nodes; i++) { | |
| 2341 | + if (bpf_prog_calls_session_cookie(fentries.nodes[i])) | |
| 2220 | 2342 | cnt++; |
| 2221 | 2343 | } |
| 2222 | 2344 | |
| @@ -2598,6 +2720,7 @@ bool btf_record_equal(const struct btf_record *rec_a, const struct btf_record *r · 375 unmodified lines | ||
| 2598 | 2720 | void bpf_obj_free_timer(const struct btf_record *rec, void *obj); |
| 2599 | 2721 | void bpf_obj_free_workqueue(const struct btf_record *rec, void *obj); |
| 2600 | 2722 | void bpf_obj_free_task_work(const struct btf_record *rec, void *obj); |
| 2723 | +void bpf_obj_cancel_fields(struct bpf_map *map, void *obj); | |
| 2601 | 2724 | void bpf_obj_free_fields(const struct btf_record *rec, void *obj); |
| 2602 | 2725 | void __bpf_obj_drop_impl(void *p, const struct btf_record *rec, bool percpu); |
| 2603 | 2726 | |
| @@ -2764,6 +2887,9 @@ void bpf_link_init(struct bpf_link *link, enum bpf_link_type type, · 160 unmodified lines | ||
| 2764 | 2887 | void bpf_link_init_sleepable(struct bpf_link *link, enum bpf_link_type type, |
| 2765 | 2888 | const struct bpf_link_ops *ops, struct bpf_prog *prog, |
| 2766 | 2889 | enum bpf_attach_type attach_type, bool sleepable); |
| 2890 | +void bpf_tramp_link_init(struct bpf_tramp_link *link, enum bpf_link_type type, | |
| 2891 | + const struct bpf_link_ops *ops, struct bpf_prog *prog, | |
| 2892 | + enum bpf_attach_type attach_type, u64 cookie); | |
| 2767 | 2893 | int bpf_link_prime(struct bpf_link *link, struct bpf_link_primer *primer); |
| 2768 | 2894 | int bpf_link_settle(struct bpf_link_primer *primer); |
| 2769 | 2895 | void bpf_link_cleanup(struct bpf_link_primer *primer); |
| @@ -2917,7 +3043,9 @@ int bpf_check_uarg_tail_zero(bpfptr_t uaddr, size_t expected_size, · 147 unmodified lines | ||
| 2917 | 3043 | size_t actual_size); |
| 2918 | 3044 | |
| 2919 | 3045 | /* verify correctness of eBPF program */ |
| 2920 | −int bpf_check(struct bpf_prog **fp, union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size); | |
| 3046 | +struct bpf_log_attr; | |
| 3047 | +int bpf_check(struct bpf_prog **fp, union bpf_attr *attr, bpfptr_t uattr, | |
| 3048 | + struct bpf_log_attr *attr_log); | |
| 2921 | 3049 | |
| 2922 | 3050 | #ifndef CONFIG_BPF_JIT_ALWAYS_ON |
| 2923 | 3051 | int bpf_patch_call_args(struct bpf_insn *insn, u32 stack_depth); |
| @@ -3088,6 +3216,56 @@ void bpf_dynptr_set_null(struct bpf_dynptr_kern *ptr); · 164 unmodified lines | ||
| 3088 | 3216 | void bpf_dynptr_set_rdonly(struct bpf_dynptr_kern *ptr); |
| 3089 | 3217 | void bpf_prog_report_arena_violation(bool write, unsigned long addr, unsigned long fault_ip); |
| 3090 | 3218 | |
| 3219 | +static __always_inline u32 | |
| 3220 | +bpf_prog_run_array_sleepable(const struct bpf_prog_array *array, | |
| 3221 | + const void *ctx, bpf_prog_run_fn run_prog) | |
| 3222 | +{ | |
| 3223 | + const struct bpf_prog_array_item *item; | |
| 3224 | + struct bpf_prog *prog; | |
| 3225 | + struct bpf_run_ctx *old_run_ctx; | |
| 3226 | + struct bpf_trace_run_ctx run_ctx; | |
| 3227 | + u32 ret = 1; | |
| 3228 | + | |
| 3229 | + if (unlikely(!array)) | |
| 3230 | + return ret; | |
| 3231 | + | |
| 3232 | + migrate_disable(); | |
| 3233 | + | |
| 3234 | + run_ctx.is_uprobe = false; | |
| 3235 | + | |
| 3236 | + old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx); | |
| 3237 | + item = &array->items[0]; | |
| 3238 | + while ((prog = READ_ONCE(item->prog))) { | |
| 3239 | + /* Skip dummy_bpf_prog placeholder (len == 0) */ | |
| 3240 | + if (unlikely(!prog->len)) { | |
| 3241 | + item++; | |
| 3242 | + continue; | |
| 3243 | + } | |
| 3244 | + | |
| 3245 | + if (unlikely(!bpf_prog_get_recursion_context(prog))) { | |
| 3246 | + bpf_prog_inc_misses_counter(prog); | |
| 3247 | + bpf_prog_put_recursion_context(prog); | |
| 3248 | + item++; | |
| 3249 | + continue; | |
| 3250 | + } | |
| 3251 | + | |
| 3252 | + run_ctx.bpf_cookie = item->bpf_cookie; | |
| 3253 | + | |
| 3254 | + if (!prog->sleepable) { | |
| 3255 | + guard(rcu)(); | |
| 3256 | + ret &= run_prog(prog, ctx); | |
| 3257 | + } else { | |
| 3258 | + ret &= run_prog(prog, ctx); | |
| 3259 | + } | |
| 3260 | + | |
| 3261 | + bpf_prog_put_recursion_context(prog); | |
| 3262 | + item++; | |
| 3263 | + } | |
| 3264 | + bpf_reset_run_ctx(old_run_ctx); | |
| 3265 | + migrate_enable(); | |
| 3266 | + return ret; | |
| 3267 | +} | |
| 3268 | + | |
| 3091 | 3269 | #else /* !CONFIG_BPF_SYSCALL */ |
| 3092 | 3270 | static inline struct bpf_prog *bpf_prog_get(u32 ufd) |
| 3093 | 3271 | { |
| @@ -3135,6 +3313,12 @@ static inline void bpf_link_init_sleepable(struct bpf_link *link, enum bpf_link_ · 41 unmodified lines | ||
| 3135 | 3313 | { |
| 3136 | 3314 | } |
| 3137 | 3315 | |
| 3316 | +static inline void bpf_tramp_link_init(struct bpf_tramp_link *link, enum bpf_link_type type, | |
| 3317 | + const struct bpf_link_ops *ops, struct bpf_prog *prog, | |
| 3318 | + enum bpf_attach_type attach_type, u64 cookie) | |
| 3319 | +{ | |
| 3320 | +} | |
| 3321 | + | |
| 3138 | 3322 | static inline int bpf_link_prime(struct bpf_link *link, |
| 3139 | 3323 | struct bpf_link_primer *primer) |
| 3140 | 3324 | { |
| @@ -3626,15 +3810,25 @@ static inline int bpf_fd_reuseport_array_update_elem(struct bpf_map *map, · 485 unmodified lines | ||
| 3626 | 3810 | #endif /* CONFIG_BPF_SYSCALL */ |
| 3627 | 3811 | #endif /* defined(CONFIG_INET) && defined(CONFIG_BPF_SYSCALL) */ |
| 3628 | 3812 | |
| 3629 | −#if defined(CONFIG_KEYS) && defined(CONFIG_BPF_SYSCALL) | |
| 3813 | +#ifdef CONFIG_KEYS | |
| 3814 | +struct bpf_key { | |
| 3815 | + struct key *key; | |
| 3816 | + bool has_ref; | |
| 3817 | +}; | |
| 3818 | +#endif /* CONFIG_KEYS */ | |
| 3630 | 3819 | |
| 3820 | +#if defined(CONFIG_KEYS) && defined(CONFIG_BPF_SYSCALL) | |
| 3631 | 3821 | struct bpf_key *bpf_lookup_user_key(s32 serial, u64 flags); |
| 3632 | 3822 | struct bpf_key *bpf_lookup_system_key(u64 id); |
| 3633 | 3823 | void bpf_key_put(struct bpf_key *bkey); |
| 3634 | −int bpf_verify_pkcs7_signature(struct bpf_dynptr *data_p, | |
| 3635 | − struct bpf_dynptr *sig_p, | |
| 3824 | +int bpf_verify_pkcs7_signature(const struct bpf_dynptr *data_p, | |
| 3825 | + const struct bpf_dynptr *sig_p, | |
| 3636 | 3826 | struct bpf_key *trusted_keyring); |
| 3637 | 3827 | |
| 3828 | +static inline s32 bpf_key_serial(const struct bpf_key *key) | |
| 3829 | +{ | |
| 3830 | + return key->has_ref ? key->key->serial : 0; | |
| 3831 | +} | |
| 3638 | 3832 | #else |
| 3639 | 3833 | static inline struct bpf_key *bpf_lookup_user_key(u32 serial, u64 flags) |
| 3640 | 3834 | { |
| @@ -3650,12 +3844,17 @@ static inline void bpf_key_put(struct bpf_key *bkey) · 9 unmodified lines | ||
| 3650 | 3844 | { |
| 3651 | 3845 | } |
| 3652 | 3846 | |
| 3653 | −static inline int bpf_verify_pkcs7_signature(struct bpf_dynptr *data_p, | |
| 3654 | − struct bpf_dynptr *sig_p, | |
| 3847 | +static inline int bpf_verify_pkcs7_signature(const struct bpf_dynptr *data_p, | |
| 3848 | + const struct bpf_dynptr *sig_p, | |
| 3655 | 3849 | struct bpf_key *trusted_keyring) |
| 3656 | 3850 | { |
| 3657 | 3851 | return -EOPNOTSUPP; |
| 3658 | 3852 | } |
| 3853 | + | |
| 3854 | +static inline s32 bpf_key_serial(const struct bpf_key *key) | |
| 3855 | +{ | |
| 3856 | + return 0; | |
| 3857 | +} | |
| 3659 | 3858 | #endif /* defined(CONFIG_KEYS) && defined(CONFIG_BPF_SYSCALL) */ |
| 3660 | 3859 | |
| 3661 | 3860 | /* verifier prototypes for helper functions called from eBPF programs */ |
| @@ -3931,15 +4130,6 @@ static inline void bpf_cgroup_atype_get(u32 attach_btf_id, int cgroup_atype) {} · 269 unmodified lines | ||
| 3931 | 4130 | static inline void bpf_cgroup_atype_put(int cgroup_atype) {} |
| 3932 | 4131 | #endif /* CONFIG_BPF_LSM */ |
| 3933 | 4132 | |
| 3934 | −struct key; | |
| 3935 | − | |
| 3936 | −#ifdef CONFIG_KEYS | |
| 3937 | −struct bpf_key { | |
| 3938 | − struct key *key; | |
| 3939 | − bool has_ref; | |
| 3940 | −}; | |
| 3941 | −#endif /* CONFIG_KEYS */ | |
| 3942 | − | |
| 3943 | 4133 | static inline bool type_is_alloc(u32 type) |
| 3944 | 4134 | { |
| 3945 | 4135 | return type & MEM_ALLOC; |
| @@ -0,0 +1,19 @@ | ||
| 1 | +/* SPDX-License-Identifier: GPL-2.0-or-later */ | |
| 2 | +/* | |
| 3 | + * Subset of bpf.h declarations, split out so files that need only these | |
| 4 | + * declarations can avoid bpf.h's full include cost. | |
| 5 | + */ | |
| 6 | +#ifndef _LINUX_BPF_DEFS_H | |
| 7 | +#define _LINUX_BPF_DEFS_H | |
| 8 | + | |
| 9 | +#ifdef CONFIG_BPF_SYSCALL | |
| 10 | +bool bpf_arena_handle_page_fault(unsigned long addr, bool is_write, unsigned long fault_ip); | |
| 11 | +#else | |
| 12 | +static inline bool bpf_arena_handle_page_fault(unsigned long addr, bool is_write, | |
| 13 | + unsigned long fault_ip) | |
| 14 | +{ | |
| 15 | + return false; | |
| 16 | +} | |
| 17 | +#endif | |
| 18 | + | |
| 19 | +#endif /* _LINUX_BPF_DEFS_H */ | |
| @@ -52,6 +52,7 @@ int bpf_set_dentry_xattr_locked(struct dentry *dentry, const char *name__str, · 51 unmodified lines | ||
| 52 | 52 | const struct bpf_dynptr *value_p, int flags); |
| 53 | 53 | int bpf_remove_dentry_xattr_locked(struct dentry *dentry, const char *name__str); |
| 54 | 54 | bool bpf_lsm_has_d_inode_locked(const struct bpf_prog *prog); |
| 55 | +bool bpf_lsm_hook_returns_errno(u32 btf_id); | |
| 55 | 56 | |
| 56 | 57 | #else /* !CONFIG_BPF_LSM */ |
| 57 | 58 | |
| @@ -104,6 +105,11 @@ static inline bool bpf_lsm_has_d_inode_locked(const struct bpf_prog *prog) · 46 unmodified lines | ||
| 104 | 105 | { |
| 105 | 106 | return false; |
| 106 | 107 | } |
| 108 | + | |
| 109 | +static inline bool bpf_lsm_hook_returns_errno(u32 btf_id) | |
| 110 | +{ | |
| 111 | + return true; | |
| 112 | +} | |
| 107 | 113 | #endif /* CONFIG_BPF_LSM */ |
| 108 | 114 | |
| 109 | 115 | #endif /* _LINUX_BPF_LSM_H */ |
| @@ -134,6 +134,7 @@ BPF_MAP_TYPE(BPF_MAP_TYPE_BLOOM_FILTER, bloom_filter_map_ops) · 133 unmodified lines | ||
| 134 | 134 | BPF_MAP_TYPE(BPF_MAP_TYPE_USER_RINGBUF, user_ringbuf_map_ops) |
| 135 | 135 | BPF_MAP_TYPE(BPF_MAP_TYPE_ARENA, arena_map_ops) |
| 136 | 136 | BPF_MAP_TYPE(BPF_MAP_TYPE_INSN_ARRAY, insn_array_map_ops) |
| 137 | +BPF_MAP_TYPE(BPF_MAP_TYPE_RHASH, rhtab_map_ops) | |
| 137 | 138 | |
| 138 | 139 | BPF_LINK_TYPE(BPF_LINK_TYPE_RAW_TRACEPOINT, raw_tracepoint) |
| 139 | 140 | BPF_LINK_TYPE(BPF_LINK_TYPE_TRACING, tracing) |
| @@ -155,3 +156,4 @@ BPF_LINK_TYPE(BPF_LINK_TYPE_PERF_EVENT, perf) · 15 unmodified lines | ||
| 155 | 156 | BPF_LINK_TYPE(BPF_LINK_TYPE_KPROBE_MULTI, kprobe_multi) |
| 156 | 157 | BPF_LINK_TYPE(BPF_LINK_TYPE_STRUCT_OPS, struct_ops) |
| 157 | 158 | BPF_LINK_TYPE(BPF_LINK_TYPE_UPROBE_MULTI, uprobe_multi) |
| 159 | +BPF_LINK_TYPE(BPF_LINK_TYPE_TRACING_MULTI, tracing_multi) | |
| @@ -8,6 +8,7 @@ · 7 unmodified lines | ||
| 8 | 8 | #include <linux/btf.h> /* for struct btf and btf_id() */ |
| 9 | 9 | #include <linux/filter.h> /* for MAX_BPF_STACK */ |
| 10 | 10 | #include <linux/tnum.h> |
| 11 | +#include <linux/cnum.h> | |
| 11 | 12 | |
| 12 | 13 | /* Maximum variable offset umax_value permitted when resolving memory accesses. |
| 13 | 14 | * In practice this is far bigger than any realistic pointer offset; this limit |
| @@ -65,7 +66,6 @@ struct bpf_reg_state { · 51 unmodified lines | ||
| 65 | 66 | |
| 66 | 67 | struct { /* for PTR_TO_MEM | PTR_TO_MEM_OR_NULL */ |
| 67 | 68 | u32 mem_size; |
| 68 | − u32 dynptr_id; /* for dynptr slices */ | |
| 69 | 69 | }; |
| 70 | 70 | |
| 71 | 71 | /* For dynptr stack slots */ |
| @@ -120,14 +120,8 @@ struct bpf_reg_state { · 48 unmodified lines | ||
| 120 | 120 | * These refer to the same value as var_off, not necessarily the actual |
| 121 | 121 | * contents of the register. |
| 122 | 122 | */ |
| 123 | − s64 smin_value; /* minimum possible (s64)value */ | |
| 124 | − s64 smax_value; /* maximum possible (s64)value */ | |
| 125 | − u64 umin_value; /* minimum possible (u64)value */ | |
| 126 | − u64 umax_value; /* maximum possible (u64)value */ | |
| 127 | − s32 s32_min_value; /* minimum possible (s32)value */ | |
| 128 | − s32 s32_max_value; /* maximum possible (s32)value */ | |
| 129 | − u32 u32_min_value; /* minimum possible (u32)value */ | |
| 130 | − u32 u32_max_value; /* maximum possible (u32)value */ | |
| 123 | + struct cnum64 r64; /* 64-bit range as circular number */ | |
| 124 | + struct cnum32 r32; /* 32-bit range as circular number */ | |
| 131 | 125 | /* For PTR_TO_PACKET, used to find other pointers with the same variable |
| 132 | 126 | * offset, so they can share range knowledge. |
| 133 | 127 | * For PTR_TO_MAP_VALUE_OR_NULL this is used to share which map value we |
| @@ -153,46 +147,14 @@ struct bpf_reg_state { · 19 unmodified lines | ||
| 153 | 147 | #define BPF_ADD_CONST32 (1U << 30) |
| 154 | 148 | #define BPF_ADD_CONST (BPF_ADD_CONST64 | BPF_ADD_CONST32) |
| 155 | 149 | u32 id; |
| 156 | − /* PTR_TO_SOCKET and PTR_TO_TCP_SOCK could be a ptr returned | |
| 157 | − * from a pointer-cast helper, bpf_sk_fullsock() and | |
| 158 | − * bpf_tcp_sock(). | |
| 159 | − * | |
| 160 | − * Consider the following where "sk" is a reference counted | |
| 161 | − * pointer returned from "sk = bpf_sk_lookup_tcp();": | |
| 162 | − * | |
| 163 | − * 1: sk = bpf_sk_lookup_tcp(); | |
| 164 | − * 2: if (!sk) { return 0; } | |
| 165 | − * 3: fullsock = bpf_sk_fullsock(sk); | |
| 166 | − * 4: if (!fullsock) { bpf_sk_release(sk); return 0; } | |
| 167 | − * 5: tp = bpf_tcp_sock(fullsock); | |
| 168 | − * 6: if (!tp) { bpf_sk_release(sk); return 0; } | |
| 169 | − * 7: bpf_sk_release(sk); | |
| 170 | − * 8: snd_cwnd = tp->snd_cwnd; // verifier will complain | |
| 171 | − * | |
| 172 | − * After bpf_sk_release(sk) at line 7, both "fullsock" ptr and | |
| 173 | − * "tp" ptr should be invalidated also. In order to do that, | |
| 174 | − * the reg holding "fullsock" and "sk" need to remember | |
| 175 | − * the original refcounted ptr id (i.e. sk_reg->id) in ref_obj_id | |
| 176 | − * such that the verifier can reset all regs which have | |
| 177 | − * ref_obj_id matching the sk_reg->id. | |
| 178 | − * | |
| 179 | − * sk_reg->ref_obj_id is set to sk_reg->id at line 1. | |
| 180 | − * sk_reg->id will stay as NULL-marking purpose only. | |
| 181 | − * After NULL-marking is done, sk_reg->id can be reset to 0. | |
| 182 | − * | |
| 183 | − * After "fullsock = bpf_sk_fullsock(sk);" at line 3, | |
| 184 | − * fullsock_reg->ref_obj_id is set to sk_reg->ref_obj_id. | |
| 185 | − * | |
| 186 | − * After "tp = bpf_tcp_sock(fullsock);" at line 5, | |
| 187 | − * tp_reg->ref_obj_id is set to fullsock_reg->ref_obj_id | |
| 188 | − * which is the same as sk_reg->ref_obj_id. | |
| 189 | − * | |
| 190 | − * From the verifier perspective, if sk, fullsock and tp | |
| 191 | − * are not NULL, they are the same ptr with different | |
| 192 | − * reg->type. In particular, bpf_sk_release(tp) is also | |
| 193 | − * allowed and has the same effect as bpf_sk_release(sk). | |
| 150 | + /* | |
| 151 | + * Tracks the parent object this register was derived from. | |
| 152 | + * Used for cascading invalidation: when the parent object is | |
| 153 | + * released or invalidated, all registers with matching parent_id | |
| 154 | + * are also invalidated. For example, a slice from bpf_dynptr_data() | |
| 155 | + * gets parent_id set to the dynptr's id. | |
| 194 | 156 | */ |
| 195 | − u32 ref_obj_id; | |
| 157 | + u32 parent_id; | |
| 196 | 158 | /* Inside the callee two registers can be both PTR_TO_STACK like |
| 197 | 159 | * R1=fp-8 and R2=fp-8, but one of them points to this function stack |
| 198 | 160 | * while another to the caller's stack. To differentiate them 'frameno' |
| @@ -209,6 +171,66 @@ struct bpf_reg_state { · 10 unmodified lines | ||
| 209 | 171 | bool precise; |
| 210 | 172 | }; |
| 211 | 173 | |
| 174 | +static inline s64 reg_smin(const struct bpf_reg_state *reg) | |
| 175 | +{ | |
| 176 | + return cnum64_smin(reg->r64); | |
| 177 | +} | |
| 178 | + | |
| 179 | +static inline s64 reg_smax(const struct bpf_reg_state *reg) | |
| 180 | +{ | |
| 181 | + return cnum64_smax(reg->r64); | |
| 182 | +} | |
| 183 | + | |
| 184 | +static inline u64 reg_umin(const struct bpf_reg_state *reg) | |
| 185 | +{ | |
| 186 | + return cnum64_umin(reg->r64); | |
| 187 | +} | |
| 188 | + | |
| 189 | +static inline u64 reg_umax(const struct bpf_reg_state *reg) | |
| 190 | +{ | |
| 191 | + return cnum64_umax(reg->r64); | |
| 192 | +} | |
| 193 | + | |
| 194 | +static inline s32 reg_s32_min(const struct bpf_reg_state *reg) | |
| 195 | +{ | |
| 196 | + return cnum32_smin(reg->r32); | |
| 197 | +} | |
| 198 | + | |
| 199 | +static inline s32 reg_s32_max(const struct bpf_reg_state *reg) | |
| 200 | +{ | |
| 201 | + return cnum32_smax(reg->r32); | |
| 202 | +} | |
| 203 | + | |
| 204 | +static inline u32 reg_u32_min(const struct bpf_reg_state *reg) | |
| 205 | +{ | |
| 206 | + return cnum32_umin(reg->r32); | |
| 207 | +} | |
| 208 | + | |
| 209 | +static inline u32 reg_u32_max(const struct bpf_reg_state *reg) | |
| 210 | +{ | |
| 211 | + return cnum32_umax(reg->r32); | |
| 212 | +} | |
| 213 | + | |
| 214 | +static inline void reg_set_srange32(struct bpf_reg_state *reg, s32 smin, s32 smax) | |
| 215 | +{ | |
| 216 | + reg->r32 = cnum32_from_srange(smin, smax); | |
| 217 | +} | |
| 218 | + | |
| 219 | +static inline void reg_set_urange32(struct bpf_reg_state *reg, u32 umin, u32 umax) | |
| 220 | +{ | |
| 221 | + reg->r32 = cnum32_from_urange(umin, umax); | |
| 222 | +} | |
| 223 | + | |
| 224 | +static inline void reg_set_srange64(struct bpf_reg_state *reg, s64 smin, s64 smax) | |
| 225 | +{ | |
| 226 | + reg->r64 = cnum64_from_srange(smin, smax); | |
| 227 | +} | |
| 228 | + | |
| 229 | +static inline void reg_set_urange64(struct bpf_reg_state *reg, u64 umin, u64 umax) | |
| 230 | +{ | |
| 231 | + reg->r64 = cnum64_from_urange(umin, umax); | |
| 232 | +} | |
| 233 | + | |
| 212 | 234 | enum bpf_stack_slot_type { |
| 213 | 235 | STACK_INVALID, /* nothing was stored in this stack slot */ |
| 214 | 236 | STACK_SPILL, /* register spilled into stack */ |
| @@ -309,10 +331,14 @@ struct bpf_reference_state { · 94 unmodified lines | ||
| 309 | 331 | * is used purely to inform the user of a reference leak. |
| 310 | 332 | */ |
| 311 | 333 | int insn_idx; |
| 312 | − /* Use to keep track of the source object of a lock, to ensure | |
| 313 | − * it matches on unlock. | |
| 314 | − */ | |
| 315 | − void *ptr; | |
| 334 | + union { | |
| 335 | + /* For REF_TYPE_PTR */ | |
| 336 | + int parent_id; | |
| 337 | + /* Use to keep track of the source object of a lock, to ensure | |
| 338 | + * it matches on unlock. | |
| 339 | + */ | |
| 340 | + void *ptr; | |
| 341 | + }; | |
| 316 | 342 | }; |
| 317 | 343 | |
| 318 | 344 | struct bpf_retval_range { |
| @@ -347,6 +373,7 @@ struct bpf_func_state { · 28 unmodified lines | ||
| 347 | 373 | bool in_callback_fn; |
| 348 | 374 | bool in_async_callback_fn; |
| 349 | 375 | bool in_exception_callback_fn; |
| 376 | + bool no_stack_arg_load; | |
| 350 | 377 | /* For callback calling functions that limit number of possible |
| 351 | 378 | * callback executions (e.g. bpf_loop) keeps track of current |
| 352 | 379 | * simulated iteration number. |
| @@ -372,46 +399,49 @@ struct bpf_func_state { · 19 unmodified lines | ||
| 372 | 399 | * `stack`. allocated_stack is always a multiple of BPF_REG_SIZE. |
| 373 | 400 | */ |
| 374 | 401 | int allocated_stack; |
| 402 | + | |
| 403 | + u16 out_stack_arg_cnt; /* Number of outgoing on-stack argument slots */ | |
| 404 | + struct bpf_reg_state *stack_arg_regs; /* Outgoing on-stack arguments */ | |
| 375 | 405 | }; |
| 376 | 406 | |
| 377 | −#define MAX_CALL_FRAMES 8 | |
| 407 | +#define MAX_CALL_FRAMES 16 | |
| 378 | 408 | |
| 379 | −/* instruction history flags, used in bpf_jmp_history_entry.flags field */ | |
| 409 | +/* instruction history flags, used in bpf_jmp_history_entry.flags field. | |
| 410 | + * Frame number and SPI are stored in dedicated fields of bpf_jmp_history_entry. | |
| 411 | + */ | |
| 380 | 412 | enum { |
| 381 | − /* instruction references stack slot through PTR_TO_STACK register; | |
| 382 | − * we also store stack's frame number in lower 3 bits (MAX_CALL_FRAMES is 8) | |
| 383 | − * and accessed stack slot's index in next 6 bits (MAX_BPF_STACK is 512, | |
| 384 | − * 8 bytes per slot, so slot index (spi) is [0, 63]) | |
| 385 | − */ | |
| 386 | − INSN_F_FRAMENO_MASK = 0x7, /* 3 bits */ | |
| 387 | − | |
| 388 | − INSN_F_SPI_MASK = 0x3f, /* 6 bits */ | |
| 389 | − INSN_F_SPI_SHIFT = 3, /* shifted 3 bits to the left */ | |
| 413 | + INSN_F_STACK_ACCESS = BIT(0), | |
| 390 | 414 | |
| 391 | − INSN_F_STACK_ACCESS = BIT(9), | |
| 415 | + INSN_F_DST_REG_STACK = BIT(1), /* dst_reg is PTR_TO_STACK */ | |
| 416 | + INSN_F_SRC_REG_STACK = BIT(2), /* src_reg is PTR_TO_STACK */ | |
| 392 | 417 | |
| 393 | − INSN_F_DST_REG_STACK = BIT(10), /* dst_reg is PTR_TO_STACK */ | |
| 394 | − INSN_F_SRC_REG_STACK = BIT(11), /* src_reg is PTR_TO_STACK */ | |
| 395 | − /* total 12 bits are used now. */ | |
| 418 | + INSN_F_STACK_ARG_ACCESS = BIT(3), | |
| 396 | 419 | }; |
| 397 | 420 | |
| 398 | −static_assert(INSN_F_FRAMENO_MASK + 1 >= MAX_CALL_FRAMES); | |
| 399 | −static_assert(INSN_F_SPI_MASK + 1 >= MAX_BPF_STACK / 8); | |
| 400 | − | |
| 401 | 421 | struct bpf_jmp_history_entry { |
| 402 | − u32 idx; | |
| 403 | 422 | /* insn idx can't be bigger than 1 million */ |
| 423 | + u32 idx : 20; | |
| 424 | + u32 frame : 4; /* stack access frame number */ | |
| 425 | + u32 spi : 6; /* stack slot index (0..63) */ | |
| 426 | + u32 : 2; | |
| 404 | 427 | u32 prev_idx : 20; |
| 405 | 428 | /* special INSN_F_xxx flags */ |
| 406 | − u32 flags : 12; | |
| 407 | − /* additional registers that need precision tracking when this | |
| 408 | − * jump is backtracked, vector of six 10-bit records | |
| 429 | + u32 flags : 4; | |
| 430 | + u32 : 8; | |
| 431 | + /* | |
| 432 | + * additional registers that need precision tracking when this | |
| 433 | + * jump is backtracked, vector of five 11-bit records | |
| 409 | 434 | */ |
| 410 | 435 | u64 linked_regs; |
| 411 | 436 | }; |
| 412 | 437 | |
| 413 | −/* Maximum number of register states that can exist at once */ | |
| 414 | −#define BPF_ID_MAP_SIZE ((MAX_BPF_REG + MAX_BPF_STACK / BPF_REG_SIZE) * MAX_CALL_FRAMES) | |
| 438 | +static_assert(MAX_CALL_FRAMES <= (1 << 4)); | |
| 439 | +static_assert(MAX_BPF_STACK / 8 <= (1 << 6)); | |
| 440 | + | |
| 441 | +/* Maximum number of bpf_reg_state objects that can exist at once */ | |
| 442 | +#define MAX_STACK_ARG_SLOTS (MAX_BPF_FUNC_ARGS - MAX_BPF_FUNC_REG_ARGS) | |
| 443 | +#define BPF_ID_MAP_SIZE ((MAX_BPF_REG + MAX_BPF_STACK / BPF_REG_SIZE + \ | |
| 444 | + MAX_STACK_ARG_SLOTS) * MAX_CALL_FRAMES) | |
| 415 | 445 | struct bpf_verifier_state { |
| 416 | 446 | /* call stack tracking */ |
| 417 | 447 | struct bpf_func_state *frame[MAX_CALL_FRAMES]; |
| @@ -497,10 +527,23 @@ struct bpf_verifier_state { · 79 unmodified lines | ||
| 497 | 527 | u32 may_goto_depth; |
| 498 | 528 | }; |
| 499 | 529 | |
| 500 | −#define bpf_get_spilled_reg(slot, frame, mask) \ | |
| 501 | − (((slot < frame->allocated_stack / BPF_REG_SIZE) && \ | |
| 502 | − ((1 << frame->stack[slot].slot_type[BPF_REG_SIZE - 1]) & (mask))) \ | |
| 503 | − ? &frame->stack[slot].spilled_ptr : NULL) | |
| 530 | +static inline struct bpf_reg_state * | |
| 531 | +bpf_get_spilled_reg(int slot, struct bpf_func_state *frame, u32 mask) | |
| 532 | +{ | |
| 533 | + if (slot < frame->allocated_stack / BPF_REG_SIZE && | |
| 534 | + (1 << frame->stack[slot].slot_type[BPF_REG_SIZE - 1]) & mask) | |
| 535 | + return &frame->stack[slot].spilled_ptr; | |
| 536 | + return NULL; | |
| 537 | +} | |
| 538 | + | |
| 539 | +static inline struct bpf_reg_state * | |
| 540 | +bpf_get_spilled_stack_arg(int slot, struct bpf_func_state *frame) | |
| 541 | +{ | |
| 542 | + if (slot < frame->out_stack_arg_cnt && | |
| 543 | + frame->stack_arg_regs[slot].type != NOT_INIT) | |
| 544 | + return &frame->stack_arg_regs[slot]; | |
| 545 | + return NULL; | |
| 546 | +} | |
| 504 | 547 | |
| 505 | 548 | /* Iterate over 'frame', setting 'reg' to either NULL or a spilled register. */ |
| 506 | 549 | #define bpf_for_each_spilled_reg(iter, frame, reg, mask) \ |
| @@ -508,7 +551,13 @@ struct bpf_verifier_state { · 1 unmodified line | ||
| 508 | 551 | iter < frame->allocated_stack / BPF_REG_SIZE; \ |
| 509 | 552 | iter++, reg = bpf_get_spilled_reg(iter, frame, mask)) |
| 510 | 553 | |
| 511 | −#define bpf_for_each_reg_in_vstate_mask(__vst, __state, __reg, __mask, __expr) \ | |
| 554 | +/* Iterate over 'frame', setting 'reg' to either NULL or a spilled stack arg. */ | |
| 555 | +#define bpf_for_each_spilled_stack_arg(iter, frame, reg) \ | |
| 556 | + for (iter = 0, reg = bpf_get_spilled_stack_arg(iter, frame); \ | |
| 557 | + iter < frame->out_stack_arg_cnt; \ | |
| 558 | + iter++, reg = bpf_get_spilled_stack_arg(iter, frame)) | |
| 559 | + | |
| 560 | +#define bpf_for_each_reg_in_vstate_mask(__vst, __state, __reg, __stack, __mask, __expr) \ | |
| 512 | 561 | ({ \ |
| 513 | 562 | struct bpf_verifier_state *___vstate = __vst; \ |
| 514 | 563 | int ___i, ___j; \ |
| @@ -516,6 +565,7 @@ struct bpf_verifier_state { · 1 unmodified line | ||
| 516 | 565 | struct bpf_reg_state *___regs; \ |
| 517 | 566 | __state = ___vstate->frame[___i]; \ |
| 518 | 567 | ___regs = __state->regs; \ |
| 568 | + __stack = NULL; \ | |
| 519 | 569 | for (___j = 0; ___j < MAX_BPF_REG; ___j++) { \ |
| 520 | 570 | __reg = &___regs[___j]; \ |
| 521 | 571 | (void)(__expr); \ |
| @@ -523,14 +573,27 @@ struct bpf_verifier_state { · 1 unmodified line | ||
| 523 | 573 | bpf_for_each_spilled_reg(___j, __state, __reg, __mask) { \ |
| 524 | 574 | if (!__reg) \ |
| 525 | 575 | continue; \ |
| 576 | + __stack = &__state->stack[___j]; \ | |
| 526 | 577 | (void)(__expr); \ |
| 527 | 578 | } \ |
| 579 | + __stack = NULL; \ | |
| 580 | + bpf_for_each_spilled_stack_arg(___j, __state, __reg) { \ | |
| 581 | + if (!__reg) \ | |
| 582 | + continue; \ | |
| 583 | + (void)(__expr); \ | |
| 584 | + } \ | |
| 528 | 585 | } \ |
| 586 | + (void)__stack; \ | |
| 529 | 587 | }) |
| 530 | 588 | |
| 531 | 589 | /* Invoke __expr over regsiters in __vst, setting __state and __reg */ |
| 532 | −#define bpf_for_each_reg_in_vstate(__vst, __state, __reg, __expr) \ | |
| 533 | − bpf_for_each_reg_in_vstate_mask(__vst, __state, __reg, 1 << STACK_SPILL, __expr) | |
| 590 | +#define bpf_for_each_reg_in_vstate(__vst, __state, __reg, __expr) \ | |
| 591 | + ({ \ | |
| 592 | + struct bpf_stack_state * ___stack; \ | |
| 593 | + (void)___stack; \ | |
| 594 | + bpf_for_each_reg_in_vstate_mask(__vst, __state, __reg, ___stack,\ | |
| 595 | + 1 << STACK_SPILL, __expr); \ | |
| 596 | + }) | |
| 534 | 597 | |
| 535 | 598 | /* linked list of verifier states used to prune search */ |
| 536 | 599 | struct bpf_verifier_state_list { |
| @@ -700,6 +763,22 @@ static inline bool bpf_verifier_log_needed(const struct bpf_verifier_log *log) · 163 unmodified lines | ||
| 700 | 763 | return log && log->level; |
| 701 | 764 | } |
| 702 | 765 | |
| 766 | +struct bpf_log_attr { | |
| 767 | + char __user *ubuf; | |
| 768 | + u32 size; | |
| 769 | + u32 level; | |
| 770 | + u32 offsetof_true_size; | |
| 771 | + bpfptr_t uattr; | |
| 772 | +}; | |
| 773 | + | |
| 774 | +int bpf_log_attr_init(struct bpf_log_attr *log, u64 log_buf, u32 log_size, u32 log_level, | |
| 775 | + u32 offsetof_log_true_size, bpfptr_t uattr, struct bpf_common_attr *common, | |
| 776 | + bpfptr_t uattr_common, u32 size_common); | |
| 777 | +struct bpf_verifier_log *bpf_log_attr_create_vlog(struct bpf_log_attr *attr_log, | |
| 778 | + struct bpf_common_attr *common, bpfptr_t uattr, | |
| 779 | + u32 size); | |
| 780 | +int bpf_log_attr_finalize(struct bpf_log_attr *attr, struct bpf_verifier_log *log); | |
| 781 | + | |
| 703 | 782 | #define BPF_MAX_SUBPROGS 256 |
| 704 | 783 | |
| 705 | 784 | struct bpf_subprog_arg_info { |
| @@ -724,6 +803,7 @@ struct bpf_subprog_info { · 18 unmodified lines | ||
| 724 | 803 | u32 exit_idx; /* Index of one of the BPF_EXIT instructions in this subprogram */ |
| 725 | 804 | u16 stack_depth; /* max. stack depth used by this function */ |
| 726 | 805 | u16 stack_extra; |
| 806 | + u32 insn_processed; | |
| 727 | 807 | /* offsets in range [stack_depth .. fastcall_stack_off) |
| 728 | 808 | * are used for bpf_fastcall spills and fills. |
| 729 | 809 | */ |
| @@ -740,12 +820,21 @@ struct bpf_subprog_info { · 10 unmodified lines | ||
| 740 | 820 | bool keep_fastcall_stack: 1; |
| 741 | 821 | bool changes_pkt_data: 1; |
| 742 | 822 | bool might_sleep: 1; |
| 743 | − u8 arg_cnt:3; | |
| 823 | + u8 arg_cnt:4; | |
| 744 | 824 | |
| 745 | 825 | enum priv_stack_mode priv_stack_mode; |
| 746 | − struct bpf_subprog_arg_info args[MAX_BPF_FUNC_REG_ARGS]; | |
| 826 | + struct bpf_subprog_arg_info args[MAX_BPF_FUNC_ARGS]; | |
| 827 | + u16 stack_arg_cnt; /* incoming + max outgoing */ | |
| 828 | + u16 max_out_stack_arg_cnt; | |
| 747 | 829 | }; |
| 748 | 830 | |
| 831 | +static inline u16 bpf_in_stack_arg_cnt(const struct bpf_subprog_info *sub) | |
| 832 | +{ | |
| 833 | + if (sub->arg_cnt > MAX_BPF_FUNC_REG_ARGS) | |
| 834 | + return sub->arg_cnt - MAX_BPF_FUNC_REG_ARGS; | |
| 835 | + return 0; | |
| 836 | +} | |
| 837 | + | |
| 749 | 838 | struct bpf_verifier_env; |
| 750 | 839 | |
| 751 | 840 | struct backtrack_state { |
| @@ -753,6 +842,7 @@ struct backtrack_state { · 1 unmodified line | ||
| 753 | 842 | u32 frame; |
| 754 | 843 | u32 reg_masks[MAX_CALL_FRAMES]; |
| 755 | 844 | u64 stack_masks[MAX_CALL_FRAMES]; |
| 845 | + u8 stack_arg_masks[MAX_CALL_FRAMES]; | |
| 756 | 846 | }; |
| 757 | 847 | |
| 758 | 848 | struct bpf_id_pair { |
| @@ -881,6 +971,8 @@ struct bpf_verifier_env { · 122 unmodified lines | ||
| 881 | 971 | u32 prev_insn_processed, insn_processed; |
| 882 | 972 | /* number of jmps, calls, exits analyzed so far */ |
| 883 | 973 | u32 prev_jmps_processed, jmps_processed; |
| 974 | + /* maximum combined stack depth */ | |
| 975 | + u32 max_stack_depth; | |
| 884 | 976 | /* total verification time */ |
| 885 | 977 | u64 verification_time; |
| 886 | 978 | /* maximum number of verifier states kept in 'branching' instructions */ |
| @@ -914,6 +1006,7 @@ struct bpf_verifier_env { · 27 unmodified lines | ||
| 914 | 1006 | * e.g., in reg_type_str() to generate reg_type string |
| 915 | 1007 | */ |
| 916 | 1008 | char tmp_str_buf[TMP_STR_BUF_LEN]; |
| 1009 | + char tmp_arg_name[32]; | |
| 917 | 1010 | struct bpf_insn insn_buf[INSN_BUF_SIZE]; |
| 918 | 1011 | struct bpf_insn epilogue_buf[INSN_BUF_SIZE]; |
| 919 | 1012 | struct bpf_scc_callchain callchain_buf; |
| @@ -1087,7 +1180,7 @@ struct list_head *bpf_explored_state(struct bpf_verifier_env *env, int idx); · 167 unmodified lines | ||
| 1087 | 1180 | void bpf_free_verifier_state(struct bpf_verifier_state *state, bool free_self); |
| 1088 | 1181 | void bpf_free_backedges(struct bpf_scc_visit *visit); |
| 1089 | 1182 | int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state *cur, |
| 1090 | − int insn_flags, u64 linked_regs); | |
| 1183 | + int insn_flags, int spi, int frame, u64 linked_regs); | |
| 1091 | 1184 | void bpf_bt_sync_linked_regs(struct backtrack_state *bt, struct bpf_jmp_history_entry *hist); |
| 1092 | 1185 | void bpf_mark_reg_not_init(const struct bpf_verifier_env *env, |
| 1093 | 1186 | struct bpf_reg_state *reg); |
| @@ -1150,6 +1243,11 @@ static inline void bpf_bt_set_frame_slot(struct backtrack_state *bt, u32 frame, · 56 unmodified lines | ||
| 1150 | 1243 | bt->stack_masks[frame] |= 1ull << slot; |
| 1151 | 1244 | } |
| 1152 | 1245 | |
| 1246 | +static inline void bt_set_frame_stack_arg_slot(struct backtrack_state *bt, u32 frame, u32 slot) | |
| 1247 | +{ | |
| 1248 | + bt->stack_arg_masks[frame] |= 1 << slot; | |
| 1249 | +} | |
| 1250 | + | |
| 1153 | 1251 | static inline bool bt_is_frame_reg_set(struct backtrack_state *bt, u32 frame, u32 reg) |
| 1154 | 1252 | { |
| 1155 | 1253 | return bt->reg_masks[frame] & (1 << reg); |
| @@ -1321,6 +1419,25 @@ struct bpf_map_desc { · 165 unmodified lines | ||
| 1321 | 1419 | int uid; |
| 1322 | 1420 | }; |
| 1323 | 1421 | |
| 1422 | +/* The last initialized dynptr; Populated by process_dynptr_func() */ | |
| 1423 | +struct bpf_dynptr_desc { | |
| 1424 | + enum bpf_dynptr_type type; | |
| 1425 | + u32 id; | |
| 1426 | + u32 parent_id; | |
| 1427 | +}; | |
| 1428 | + | |
| 1429 | +/* | |
| 1430 | + * The last seen rereferenced object; Updated by update_ref_obj() when a register refers to a | |
| 1431 | + * referenced object. Used when the helper or kfunc is casting a referenced object, returning | |
| 1432 | + * allocated memory derived from referenced object or creating a dynptr with a referenced | |
| 1433 | + * object as parent. | |
| 1434 | + */ | |
| 1435 | +struct ref_obj_desc { | |
| 1436 | + u32 id; | |
| 1437 | + u32 parent_id; | |
| 1438 | + u8 cnt; | |
| 1439 | +}; | |
| 1440 | + | |
| 1324 | 1441 | struct bpf_kfunc_call_arg_meta { |
| 1325 | 1442 | /* In parameters */ |
| 1326 | 1443 | struct btf *btf; |
| @@ -1329,7 +1446,6 @@ struct bpf_kfunc_call_arg_meta { · 2 unmodified lines | ||
| 1329 | 1446 | const struct btf_type *func_proto; |
| 1330 | 1447 | const char *func_name; |
| 1331 | 1448 | /* Out parameters */ |
| 1332 | − u32 ref_obj_id; | |
| 1333 | 1449 | u8 release_regno; |
| 1334 | 1450 | bool r0_rdonly; |
| 1335 | 1451 | u32 ret_btf_id; |
| @@ -1361,16 +1477,13 @@ struct bpf_kfunc_call_arg_meta { · 25 unmodified lines | ||
| 1361 | 1477 | struct { |
| 1362 | 1478 | struct btf_field *field; |
| 1363 | 1479 | } arg_rbtree_root; |
| 1364 | − struct { | |
| 1365 | − enum bpf_dynptr_type type; | |
| 1366 | − u32 id; | |
| 1367 | − u32 ref_obj_id; | |
| 1368 | − } initialized_dynptr; | |
| 1369 | 1480 | struct { |
| 1370 | 1481 | u8 spi; |
| 1371 | 1482 | u8 frameno; |
| 1372 | 1483 | } iter; |
| 1373 | 1484 | struct bpf_map_desc map; |
| 1485 | + struct bpf_dynptr_desc dynptr; | |
| 1486 | + struct ref_obj_desc ref_obj; | |
| 1374 | 1487 | u64 mem_size; |
| 1375 | 1488 | }; |
| 1376 | 1489 | |
| @@ -1479,6 +1592,10 @@ int bpf_add_kfunc_call(struct bpf_verifier_env *env, u32 func_id, u16 offset); · 102 unmodified lines | ||
| 1479 | 1592 | int bpf_fixup_kfunc_call(struct bpf_verifier_env *env, struct bpf_insn *insn, |
| 1480 | 1593 | struct bpf_insn *insn_buf, int insn_idx, int *cnt); |
| 1481 | 1594 | |
| 1595 | +/* Functions exported from verifier.c, used by trampoline.c */ | |
| 1596 | +int bpf_check_attach_btf_id_multi(struct btf *btf, struct bpf_prog *prog, u32 btf_id, | |
| 1597 | + struct bpf_attach_target_info *tgt_info); | |
| 1598 | + | |
| 1482 | 1599 | /* Functions in fixups.c, called from bpf_check() */ |
| 1483 | 1600 | int bpf_remove_fastcall_spills_fills(struct bpf_verifier_env *env); |
| 1484 | 1601 | int bpf_optimize_bpf_loop(struct bpf_verifier_env *env); |
| @@ -145,7 +145,8 @@ const char *btf_get_name(const struct btf *btf); · 144 unmodified lines | ||
| 145 | 145 | void btf_get(struct btf *btf); |
| 146 | 146 | void btf_put(struct btf *btf); |
| 147 | 147 | const struct btf_header *btf_header(const struct btf *btf); |
| 148 | −int btf_new_fd(const union bpf_attr *attr, bpfptr_t uattr, u32 uattr_sz); | |
| 148 | +struct bpf_log_attr; | |
| 149 | +int btf_new_fd(const union bpf_attr *attr, bpfptr_t uattr, struct bpf_log_attr *attr_log); | |
| 149 | 150 | struct btf *btf_get_by_fd(int fd); |
| 150 | 151 | int btf_get_info_by_fd(const struct btf *btf, |
| 151 | 152 | const union bpf_attr *attr, |
| @@ -415,12 +416,12 @@ static inline bool btf_type_is_array(const struct btf_type *t) · 263 unmodified lines | ||
| 415 | 416 | return BTF_INFO_KIND(t->info) == BTF_KIND_ARRAY; |
| 416 | 417 | } |
| 417 | 418 | |
| 418 | −static inline u16 btf_type_vlen(const struct btf_type *t) | |
| 419 | +static inline u32 btf_type_vlen(const struct btf_type *t) | |
| 419 | 420 | { |
| 420 | 421 | return BTF_INFO_VLEN(t->info); |
| 421 | 422 | } |
| 422 | 423 | |
| 423 | −static inline u16 btf_vlen(const struct btf_type *t) | |
| 424 | +static inline u32 btf_vlen(const struct btf_type *t) | |
| 424 | 425 | { |
| 425 | 426 | return btf_type_vlen(t); |
| 426 | 427 | } |
| @@ -284,5 +284,6 @@ extern u32 bpf_cgroup_btf_id[]; · 283 unmodified lines | ||
| 284 | 284 | extern u32 bpf_local_storage_map_btf_id[]; |
| 285 | 285 | extern u32 btf_bpf_map_id[]; |
| 286 | 286 | extern u32 bpf_kmem_cache_btf_id[]; |
| 287 | +extern u32 bpf_multi_func_btf_id[]; | |
| 287 | 288 | |
| 288 | 289 | #endif |
| @@ -0,0 +1,82 @@ | ||
| 1 | +/* SPDX-License-Identifier: GPL-2.0-only */ | |
| 2 | +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ | |
| 3 | + | |
| 4 | +#ifndef _LINUX_CNUM_H | |
| 5 | +#define _LINUX_CNUM_H | |
| 6 | + | |
| 7 | +#include <linux/types.h> | |
| 8 | + | |
| 9 | +/* | |
| 10 | + * cnum32: a circular number. | |
| 11 | + * A unified representation for signed and unsigned ranges. | |
| 12 | + * | |
| 13 | + * Assume that a 32-bit range is a circle, with 0 being in the 12 o'clock | |
| 14 | + * position, numbers placed sequentially in clockwise order and U32_MAX | |
| 15 | + * in the 11 o'clock position. Signed values map onto the same circle: | |
| 16 | + * S32_MAX sits at 5 o'clock, S32_MIN sits at 6 o'clock (opposite 0), | |
| 17 | + * negative values occupy the left half and positive values the right half. | |
| 18 | + * | |
| 19 | + * @cnum32 represents an arc on this circle drawn clockwise. | |
| 20 | + * @base corresponds to the first value of the range. | |
| 21 | + * @size corresponds to the number of integers in the range excluding @base. | |
| 22 | + * (The @base is excluded to avoid integer overflow when representing the full | |
| 23 | + * 0..U32_MAX range, which corresponds to 2^32, which can't be stored in u32). | |
| 24 | + * | |
| 25 | + * For example: {U32_MAX, 1} corresponds to signed range [-1, 0], | |
| 26 | + * {S32_MAX, 1} corresponds to unsigned range [S32_MAX, S32_MIN]. | |
| 27 | + */ | |
| 28 | +struct cnum32 { | |
| 29 | + u32 base; | |
| 30 | + u32 size; | |
| 31 | +}; | |
| 32 | + | |
| 33 | +#define CNUM32_UNBOUNDED ((struct cnum32){ .base = 0, .size = U32_MAX }) | |
| 34 | +#define CNUM32_EMPTY ((struct cnum32){ .base = U32_MAX, .size = U32_MAX }) | |
| 35 | + | |
| 36 | +struct cnum32 cnum32_from_urange(u32 min, u32 max); | |
| 37 | +struct cnum32 cnum32_from_srange(s32 min, s32 max); | |
| 38 | +u32 cnum32_umin(struct cnum32 cnum); | |
| 39 | +u32 cnum32_umax(struct cnum32 cnum); | |
| 40 | +s32 cnum32_smin(struct cnum32 cnum); | |
| 41 | +s32 cnum32_smax(struct cnum32 cnum); | |
| 42 | +struct cnum32 cnum32_intersect(struct cnum32 a, struct cnum32 b); | |
| 43 | +void cnum32_intersect_with(struct cnum32 *dst, struct cnum32 src); | |
| 44 | +void cnum32_intersect_with_urange(struct cnum32 *dst, u32 min, u32 max); | |
| 45 | +void cnum32_intersect_with_srange(struct cnum32 *dst, s32 min, s32 max); | |
| 46 | +bool cnum32_contains(struct cnum32 cnum, u32 v); | |
| 47 | +bool cnum32_is_const(struct cnum32 cnum); | |
| 48 | +bool cnum32_is_empty(struct cnum32 cnum); | |
| 49 | +struct cnum32 cnum32_add(struct cnum32 a, struct cnum32 b); | |
| 50 | +struct cnum32 cnum32_negate(struct cnum32 a); | |
| 51 | +bool cnum32_is_subset(struct cnum32 outer, struct cnum32 inner); | |
| 52 | + | |
| 53 | +/* Same as cnum32 but for 64-bit ranges */ | |
| 54 | +struct cnum64 { | |
| 55 | + u64 base; | |
| 56 | + u64 size; | |
| 57 | +}; | |
| 58 | + | |
| 59 | +#define CNUM64_UNBOUNDED ((struct cnum64){ .base = 0, .size = U64_MAX }) | |
| 60 | +#define CNUM64_EMPTY ((struct cnum64){ .base = U64_MAX, .size = U64_MAX }) | |
| 61 | + | |
| 62 | +struct cnum64 cnum64_from_urange(u64 min, u64 max); | |
| 63 | +struct cnum64 cnum64_from_srange(s64 min, s64 max); | |
| 64 | +u64 cnum64_umin(struct cnum64 cnum); | |
| 65 | +u64 cnum64_umax(struct cnum64 cnum); | |
| 66 | +s64 cnum64_smin(struct cnum64 cnum); | |
| 67 | +s64 cnum64_smax(struct cnum64 cnum); | |
| 68 | +struct cnum64 cnum64_intersect(struct cnum64 a, struct cnum64 b); | |
| 69 | +void cnum64_intersect_with(struct cnum64 *dst, struct cnum64 src); | |
| 70 | +void cnum64_intersect_with_urange(struct cnum64 *dst, u64 min, u64 max); | |
| 71 | +void cnum64_intersect_with_srange(struct cnum64 *dst, s64 min, s64 max); | |
| 72 | +bool cnum64_contains(struct cnum64 cnum, u64 v); | |
| 73 | +bool cnum64_is_const(struct cnum64 cnum); | |
| 74 | +bool cnum64_is_empty(struct cnum64 cnum); | |
| 75 | +struct cnum64 cnum64_add(struct cnum64 a, struct cnum64 b); | |
| 76 | +struct cnum64 cnum64_negate(struct cnum64 a); | |
| 77 | +bool cnum64_is_subset(struct cnum64 outer, struct cnum64 inner); | |
| 78 | + | |
| 79 | +struct cnum32 cnum32_from_cnum64(struct cnum64 cnum); | |
| 80 | +struct cnum64 cnum64_cnum32_intersect(struct cnum64 a, struct cnum32 b); | |
| 81 | + | |
| 82 | +#endif /* _LINUX_CNUM_H */ | |
| @@ -58,8 +58,9 @@ struct ctl_table_header; · 57 unmodified lines | ||
| 58 | 58 | #define BPF_REG_H BPF_REG_9 /* hlen, callee-saved */ |
| 59 | 59 | |
| 60 | 60 | /* Kernel hidden auxiliary/helper register. */ |
| 61 | −#define BPF_REG_AX MAX_BPF_REG | |
| 62 | −#define MAX_BPF_EXT_REG (MAX_BPF_REG + 1) | |
| 61 | +#define BPF_REG_PARAMS MAX_BPF_REG | |
| 62 | +#define BPF_REG_AX (MAX_BPF_REG + 1) | |
| 63 | +#define MAX_BPF_EXT_REG (MAX_BPF_REG + 2) | |
| 63 | 64 | #define MAX_BPF_JIT_REG MAX_BPF_EXT_REG |
| 64 | 65 | |
| 65 | 66 | /* unused opcode to mark special call to bpf_tail_call() helper */ |
| @@ -748,6 +749,27 @@ static inline u32 bpf_prog_run_pin_on_cpu(const struct bpf_prog *prog, · 682 unmodified lines | ||
| 748 | 749 | return ret; |
| 749 | 750 | } |
| 750 | 751 | |
| 752 | +static inline bool is_stack_arg_ldx(const struct bpf_insn *insn) | |
| 753 | +{ | |
| 754 | + return insn->code == (BPF_LDX | BPF_MEM | BPF_DW) && | |
| 755 | + insn->src_reg == BPF_REG_PARAMS && | |
| 756 | + insn->off > 0 && insn->off % 8 == 0; | |
| 757 | +} | |
| 758 | + | |
| 759 | +static inline bool is_stack_arg_st(const struct bpf_insn *insn) | |
| 760 | +{ | |
| 761 | + return insn->code == (BPF_ST | BPF_MEM | BPF_DW) && | |
| 762 | + insn->dst_reg == BPF_REG_PARAMS && | |
| 763 | + insn->off < 0 && insn->off % 8 == 0; | |
| 764 | +} | |
| 765 | + | |
| 766 | +static inline bool is_stack_arg_stx(const struct bpf_insn *insn) | |
| 767 | +{ | |
| 768 | + return insn->code == (BPF_STX | BPF_MEM | BPF_DW) && | |
| 769 | + insn->dst_reg == BPF_REG_PARAMS && | |
| 770 | + insn->off < 0 && insn->off % 8 == 0; | |
| 771 | +} | |
| 772 | + | |
| 751 | 773 | #define BPF_SKB_CB_LEN QDISC_CB_PRIV_LEN |
| 752 | 774 | |
| 753 | 775 | struct bpf_skb_data_end { |
| @@ -1159,6 +1181,7 @@ bool bpf_jit_inlines_helper_call(s32 imm); · 405 unmodified lines | ||
| 1159 | 1181 | bool bpf_jit_supports_subprog_tailcalls(void); |
| 1160 | 1182 | bool bpf_jit_supports_percpu_insn(void); |
| 1161 | 1183 | bool bpf_jit_supports_kfunc_call(void); |
| 1184 | +bool bpf_jit_supports_stack_args(void); | |
| 1162 | 1185 | bool bpf_jit_supports_far_kfunc_call(void); |
| 1163 | 1186 | bool bpf_jit_supports_exceptions(void); |
| 1164 | 1187 | bool bpf_jit_supports_ptr_xchg(void); |
| @@ -415,6 +415,8 @@ struct ftrace_hash *alloc_ftrace_hash(int size_bits); · 414 unmodified lines | ||
| 415 | 415 | void free_ftrace_hash(struct ftrace_hash *hash); |
| 416 | 416 | struct ftrace_func_entry *add_ftrace_hash_entry_direct(struct ftrace_hash *hash, |
| 417 | 417 | unsigned long ip, unsigned long direct); |
| 418 | +void add_ftrace_hash_entry(struct ftrace_hash *hash, struct ftrace_func_entry *entry); | |
| 419 | +void ftrace_hash_remove(struct ftrace_hash *hash); | |
| 418 | 420 | |
| 419 | 421 | /* The hash used to know what functions callbacks trace */ |
| 420 | 422 | struct ftrace_ops_hash { |
| @@ -551,6 +553,8 @@ int update_ftrace_direct_mod(struct ftrace_ops *ops, struct ftrace_hash *hash, b · 130 unmodified lines | ||
| 551 | 553 | |
| 552 | 554 | void ftrace_stub_direct_tramp(void); |
| 553 | 555 | |
| 556 | +unsigned long ftrace_hash_count(struct ftrace_hash *hash); | |
| 557 | + | |
| 554 | 558 | #else |
| 555 | 559 | struct ftrace_ops; |
| 556 | 560 | static inline unsigned long ftrace_find_rec_direct(unsigned long ip) |
| @@ -590,6 +594,11 @@ static inline int update_ftrace_direct_mod(struct ftrace_ops *ops, struct ftrace · 33 unmodified lines | ||
| 590 | 594 | return -ENODEV; |
| 591 | 595 | } |
| 592 | 596 | |
| 597 | +static inline unsigned long ftrace_hash_count(struct ftrace_hash *hash) | |
| 598 | +{ | |
| 599 | + return 0; | |
| 600 | +} | |
| 601 | + | |
| 593 | 602 | /* |
| 594 | 603 | * This must be implemented by the architecture. |
| 595 | 604 | * It is the way the ftrace direct_ops helper, when called |
| @@ -1070,6 +1070,49 @@ static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long addres · 1069 unmodified lines | ||
| 1070 | 1070 | } |
| 1071 | 1071 | #endif |
| 1072 | 1072 | |
| 1073 | +#ifndef ptep_try_set | |
| 1074 | +/** | |
| 1075 | + * ptep_try_set - atomically set an empty kernel PTE | |
| 1076 | + * @ptep: page table entry | |
| 1077 | + * @new_pte: value to install | |
| 1078 | + * | |
| 1079 | + * Atomically set *@ptep to @new_pte iff *@ptep is pte_none(). Return true on | |
| 1080 | + * success, false if the slot was already populated or the arch has no | |
| 1081 | + * implementation. | |
| 1082 | + * | |
| 1083 | + * For special kernel page tables only - never user page tables. The caller must | |
| 1084 | + * prevent concurrent teardown of @ptep and must accept that other writers may | |
| 1085 | + * race. Concurrent clearers must use ptep_get_and_clear() so racing accesses | |
| 1086 | + * agree on the outcome. | |
| 1087 | + * | |
| 1088 | + * Architectures opt in by providing a cmpxchg-based override and defining | |
| 1089 | + * ptep_try_set as an identity macro. The generic stub returns false, which is | |
| 1090 | + * correct for callers that fall through to oops on failure. | |
| 1091 | + */ | |
| 1092 | +static inline bool ptep_try_set(pte_t *ptep, pte_t new_pte) | |
| 1093 | +{ | |
| 1094 | + return false; | |
| 1095 | +} | |
| 1096 | +#endif | |
| 1097 | + | |
| 1098 | +#ifndef flush_tlb_before_set | |
| 1099 | +/** | |
| 1100 | + * flush_tlb_before_set - invalidate a kernel PTE's TLB before re-setting it | |
| 1101 | + * @addr: kernel virtual address whose PTE was just cleared | |
| 1102 | + * | |
| 1103 | + * Some architectures (e.g. arm64) do not allow a live page-table entry to be | |
| 1104 | + * repointed at a different page in one step. The old entry must first be made | |
| 1105 | + * invalid and its translation flushed from every TLB, and only then may the new | |
| 1106 | + * entry be written. | |
| 1107 | + * | |
| 1108 | + * This is only for the lockless atomic kernel-PTE installers (ptep_try_set()). | |
| 1109 | + * It must be callable with interrupts disabled. | |
| 1110 | + */ | |
| 1111 | +static inline void flush_tlb_before_set(unsigned long addr) | |
| 1112 | +{ | |
| 1113 | +} | |
| 1114 | +#endif | |
| 1115 | + | |
| 1073 | 1116 | #ifndef wrprotect_ptes |
| 1074 | 1117 | /** |
| 1075 | 1118 | * wrprotect_ptes - Write-protect PTEs that map consecutive pages of the same |
| @@ -263,6 +263,8 @@ struct rhash_lock_head __rcu **__rht_bucket_nested( · 262 unmodified lines | ||
| 263 | 263 | struct rhash_lock_head __rcu **rht_bucket_nested_insert( |
| 264 | 264 | struct rhashtable *ht, struct bucket_table *tbl, unsigned int hash); |
| 265 | 265 | |
| 266 | +void *rhashtable_next_key(struct rhashtable *ht, const void *prev_key); | |
| 267 | + | |
| 266 | 268 | #define rht_dereference(p, ht) \ |
| 267 | 269 | rcu_dereference_protected(p, lockdep_rht_mutex_is_held(ht)) |
| 268 | 270 | |
| @@ -1117,7 +1119,7 @@ unlocked: · 848 unmodified lines | ||
| 1117 | 1119 | atomic_dec(&ht->nelems); |
| 1118 | 1120 | if (unlikely(ht->p.automatic_shrinking && |
| 1119 | 1121 | rht_shrink_below_30(ht, tbl))) |
| 1120 | − schedule_work(&ht->run_work); | |
| 1122 | + irq_work_queue(&ht->run_irq_work); | |
| 1121 | 1123 | err = 0; |
| 1122 | 1124 | } |
| 1123 | 1125 | |
| @@ -940,7 +940,8 @@ asmlinkage long sys_seccomp(unsigned int op, unsigned int flags, · 939 unmodified lines | ||
| 940 | 940 | asmlinkage long sys_getrandom(char __user *buf, size_t count, |
| 941 | 941 | unsigned int flags); |
| 942 | 942 | asmlinkage long sys_memfd_create(const char __user *uname_ptr, unsigned int flags); |
| 943 | −asmlinkage long sys_bpf(int cmd, union bpf_attr __user *attr, unsigned int size); | |
| 943 | +asmlinkage long sys_bpf(int cmd, union bpf_attr __user *attr, unsigned int size, | |
| 944 | + struct bpf_common_attr __user *attr_common, unsigned int size_common); | |
| 944 | 945 | asmlinkage long sys_execveat(int dfd, const char __user *filename, |
| 945 | 946 | const char __user *const __user *argv, |
| 946 | 947 | const char __user *const __user *envp, int flags); |
| @@ -770,6 +770,7 @@ trace_trigger_soft_disabled(struct trace_event_file *file) · 769 unmodified lines | ||
| 770 | 770 | |
| 771 | 771 | #ifdef CONFIG_BPF_EVENTS |
| 772 | 772 | unsigned int trace_call_bpf(struct trace_event_call *call, void *ctx); |
| 773 | +unsigned int trace_call_bpf_faultable(struct trace_event_call *call, void *ctx); | |
| 773 | 774 | int perf_event_attach_bpf_prog(struct perf_event *event, struct bpf_prog *prog, u64 bpf_cookie); |
| 774 | 775 | void perf_event_detach_bpf_prog(struct perf_event *event); |
| 775 | 776 | int perf_event_query_prog_array(struct perf_event *event, void __user *info); |
| @@ -786,12 +787,18 @@ int bpf_get_perf_event_info(const struct perf_event *event, u32 *prog_id, · 10 unmodified lines | ||
| 786 | 787 | unsigned long *missed); |
| 787 | 788 | int bpf_kprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog); |
| 788 | 789 | int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog); |
| 790 | +int bpf_tracing_multi_attach(struct bpf_prog *prog, const union bpf_attr *attr); | |
| 789 | 791 | #else |
| 790 | 792 | static inline unsigned int trace_call_bpf(struct trace_event_call *call, void *ctx) |
| 791 | 793 | { |
| 792 | 794 | return 1; |
| 793 | 795 | } |
| 794 | 796 | |
| 797 | +static inline unsigned int trace_call_bpf_faultable(struct trace_event_call *call, void *ctx) | |
| 798 | +{ | |
| 799 | + return 1; | |
| 800 | +} | |
| 801 | + | |
| 795 | 802 | static inline int |
| 796 | 803 | perf_event_attach_bpf_prog(struct perf_event *event, struct bpf_prog *prog, u64 bpf_cookie) |
| 797 | 804 | { |
| @@ -838,6 +845,11 @@ bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog) · 40 unmodified lines | ||
| 838 | 845 | { |
| 839 | 846 | return -EOPNOTSUPP; |
| 840 | 847 | } |
| 848 | +static inline int | |
| 849 | +bpf_tracing_multi_attach(struct bpf_prog *prog, const union bpf_attr *attr) | |
| 850 | +{ | |
| 851 | + return -EOPNOTSUPP; | |
| 852 | +} | |
| 841 | 853 | #endif |
| 842 | 854 | |
| 843 | 855 | enum { |
| @@ -58,9 +58,7 @@ static notrace void \ · 57 unmodified lines | ||
| 58 | 58 | __bpf_trace_##call(void *__data, proto) \ |
| 59 | 59 | { \ |
| 60 | 60 | might_fault(); \ |
| 61 | − preempt_disable_notrace(); \ | |
| 62 | 61 | CONCATENATE(bpf_trace_run, COUNT_ARGS(args))(__data, CAST_TO_U64(args)); \ |
| 63 | − preempt_enable_notrace(); \ | |
| 64 | 62 | } |
| 65 | 63 | |
| 66 | 64 | #undef DECLARE_EVENT_SYSCALL_CLASS |
| @@ -994,6 +994,7 @@ enum bpf_cmd { · 993 unmodified lines | ||
| 994 | 994 | BPF_PROG_STREAM_READ_BY_FD, |
| 995 | 995 | BPF_PROG_ASSOC_STRUCT_OPS, |
| 996 | 996 | __MAX_BPF_CMD, |
| 997 | + BPF_COMMON_ATTRS = 1 << 16, /* Indicate carrying syscall common attrs. */ | |
| 997 | 998 | }; |
| 998 | 999 | |
| 999 | 1000 | enum bpf_map_type { |
| @@ -1046,6 +1047,7 @@ enum bpf_map_type { · 46 unmodified lines | ||
| 1046 | 1047 | BPF_MAP_TYPE_CGRP_STORAGE, |
| 1047 | 1048 | BPF_MAP_TYPE_ARENA, |
| 1048 | 1049 | BPF_MAP_TYPE_INSN_ARRAY, |
| 1050 | + BPF_MAP_TYPE_RHASH, | |
| 1049 | 1051 | __MAX_BPF_MAP_TYPE |
| 1050 | 1052 | }; |
| 1051 | 1053 | |
| @@ -1154,6 +1156,9 @@ enum bpf_attach_type { · 102 unmodified lines | ||
| 1154 | 1156 | BPF_TRACE_KPROBE_SESSION, |
| 1155 | 1157 | BPF_TRACE_UPROBE_SESSION, |
| 1156 | 1158 | BPF_TRACE_FSESSION, |
| 1159 | + BPF_TRACE_FENTRY_MULTI, | |
| 1160 | + BPF_TRACE_FEXIT_MULTI, | |
| 1161 | + BPF_TRACE_FSESSION_MULTI, | |
| 1157 | 1162 | __MAX_BPF_ATTACH_TYPE |
| 1158 | 1163 | }; |
| 1159 | 1164 | |
| @@ -1178,6 +1183,7 @@ enum bpf_link_type { · 18 unmodified lines | ||
| 1178 | 1183 | BPF_LINK_TYPE_UPROBE_MULTI = 12, |
| 1179 | 1184 | BPF_LINK_TYPE_NETKIT = 13, |
| 1180 | 1185 | BPF_LINK_TYPE_SOCKMAP = 14, |
| 1186 | + BPF_LINK_TYPE_TRACING_MULTI = 15, | |
| 1181 | 1187 | __MAX_BPF_LINK_TYPE, |
| 1182 | 1188 | }; |
| 1183 | 1189 | |
| @@ -1321,7 +1327,11 @@ enum { · 137 unmodified lines | ||
| 1321 | 1327 | * BPF_TRACE_UPROBE_MULTI attach type to create return probe. |
| 1322 | 1328 | */ |
| 1323 | 1329 | enum { |
| 1324 | − BPF_F_UPROBE_MULTI_RETURN = (1U << 0) | |
| 1330 | + /* Get return uprobe. */ | |
| 1331 | + BPF_F_UPROBE_MULTI_RETURN = (1U << 0), | |
| 1332 | + | |
| 1333 | + /* Get path from provided path_fd. */ | |
| 1334 | + BPF_F_UPROBE_MULTI_PATH_FD = (1U << 1), | |
| 1325 | 1335 | }; |
| 1326 | 1336 | |
| 1327 | 1337 | /* link_create.netfilter.flags used in LINK_CREATE command for |
| @@ -1500,6 +1510,13 @@ struct bpf_stack_build_id { · 172 unmodified lines | ||
| 1500 | 1510 | }; |
| 1501 | 1511 | }; |
| 1502 | 1512 | |
| 1513 | +struct bpf_common_attr { | |
| 1514 | + __aligned_u64 log_buf; | |
| 1515 | + __u32 log_size; | |
| 1516 | + __u32 log_level; | |
| 1517 | + __u32 log_true_size; | |
| 1518 | +}; | |
| 1519 | + | |
| 1503 | 1520 | #define BPF_OBJ_NAME_LEN 16U |
| 1504 | 1521 | |
| 1505 | 1522 | enum { |
| @@ -1537,6 +1554,11 @@ union bpf_attr { · 31 unmodified lines | ||
| 1537 | 1554 | * |
| 1538 | 1555 | * BPF_MAP_TYPE_ARENA - contains the address where user space |
| 1539 | 1556 | * is going to mmap() the arena. It has to be page aligned. |
| 1557 | + * | |
| 1558 | + * BPF_MAP_TYPE_RHASH - initial table size hint | |
| 1559 | + * (nelem_hint). 0 = use rhashtable default. Must be | |
| 1560 | + * <= min(max_entries, U16_MAX). Upper 32 bits reserved, | |
| 1561 | + * must be zero. | |
| 1540 | 1562 | */ |
| 1541 | 1563 | __u64 map_extra; |
| 1542 | 1564 | |
| @@ -1846,6 +1868,7 @@ union bpf_attr { · 303 unmodified lines | ||
| 1846 | 1868 | __u32 cnt; |
| 1847 | 1869 | __u32 flags; |
| 1848 | 1870 | __u32 pid; |
| 1871 | + __u32 path_fd; | |
| 1849 | 1872 | } uprobe_multi; |
| 1850 | 1873 | struct { |
| 1851 | 1874 | union { |
| @@ -1861,6 +1884,11 @@ union bpf_attr { · 9 unmodified lines | ||
| 1861 | 1884 | }; |
| 1862 | 1885 | __u64 expected_revision; |
| 1863 | 1886 | } cgroup; |
| 1887 | + struct { | |
| 1888 | + __aligned_u64 ids; | |
| 1889 | + __aligned_u64 cookies; | |
| 1890 | + __u32 cnt; | |
| 1891 | + } tracing_multi; | |
| 1864 | 1892 | }; |
| 1865 | 1893 | } link_create; |
| 1866 | 1894 | |
| @@ -6698,6 +6726,7 @@ struct bpf_prog_info { · 4831 unmodified lines | ||
| 6698 | 6726 | __u32 verified_insns; |
| 6699 | 6727 | __u32 attach_btf_obj_id; |
| 6700 | 6728 | __u32 attach_btf_id; |
| 6729 | + __u32 :32; | |
| 6701 | 6730 | } __attribute__((aligned(8))); |
| 6702 | 6731 | |
| 6703 | 6732 | struct bpf_map_info { |
| @@ -6719,6 +6748,7 @@ struct bpf_map_info { · 15 unmodified lines | ||
| 6719 | 6748 | __u64 map_extra; |
| 6720 | 6749 | __aligned_u64 hash; |
| 6721 | 6750 | __u32 hash_size; |
| 6751 | + __u32 :32; | |
| 6722 | 6752 | } __attribute__((aligned(8))); |
| 6723 | 6753 | |
| 6724 | 6754 | struct bpf_btf_info { |
| @@ -33,20 +33,22 @@ struct btf_header { · 32 unmodified lines | ||
| 33 | 33 | __u32 layout_len; /* length of layout section */ |
| 34 | 34 | }; |
| 35 | 35 | |
| 36 | −/* Max # of type identifier */ | |
| 37 | −#define BTF_MAX_TYPE 0x000fffff | |
| 38 | −/* Max offset into the string section */ | |
| 39 | −#define BTF_MAX_NAME_OFFSET 0x00ffffff | |
| 40 | −/* Max # of struct/union/enum members or func args */ | |
| 41 | −#define BTF_MAX_VLEN 0xffff | |
| 36 | +enum btf_max { | |
| 37 | + /* Max possible kind */ | |
| 38 | + BTF_MAX_KIND = 0x0000007f, | |
| 39 | + /* Max # of type identifier */ | |
| 40 | + BTF_MAX_TYPE = 0x000fffff, | |
| 41 | + /* Max offset into the string section */ | |
| 42 | + BTF_MAX_NAME_OFFSET = 0x00ffffff, | |
| 43 | + /* Max # of struct/union/enum members or func args */ | |
| 44 | + BTF_MAX_VLEN = 0x00ffffff, | |
| 45 | +}; | |
| 42 | 46 | |
| 43 | 47 | struct btf_type { |
| 44 | 48 | __u32 name_off; |
| 45 | 49 | /* "info" bits arrangement |
| 46 | − * bits 0-15: vlen (e.g. # of struct's members) | |
| 47 | − * bits 16-23: unused | |
| 48 | − * bits 24-28: kind (e.g. int, ptr, array...etc) | |
| 49 | − * bits 29-30: unused | |
| 50 | + * bits 0-23: vlen (e.g. # of struct's members) | |
| 51 | + * bits 24-30: kind (e.g. int, ptr, array...etc) | |
| 50 | 52 | * bit 31: kind_flag, currently used by |
| 51 | 53 | * struct, union, enum, fwd, enum64, |
| 52 | 54 | * decl_tag and type_tag |
| @@ -65,8 +67,8 @@ struct btf_type { · 12 unmodified lines | ||
| 65 | 67 | }; |
| 66 | 68 | }; |
| 67 | 69 | |
| 68 | −#define BTF_INFO_KIND(info) (((info) >> 24) & 0x1f) | |
| 69 | −#define BTF_INFO_VLEN(info) ((info) & 0xffff) | |
| 70 | +#define BTF_INFO_KIND(info) (((info) >> 24) & 0x7f) | |
| 71 | +#define BTF_INFO_VLEN(info) ((info) & 0xffffff) | |
| 70 | 72 | #define BTF_INFO_KFLAG(info) ((info) >> 31) |
| 71 | 73 | |
| 72 | 74 | enum { |
| @@ -6,7 +6,7 @@ cflags-nogcse-$(CONFIG_X86)$(CONFIG_CC_IS_GCC) := -fno-gcse · 5 unmodified lines | ||
| 6 | 6 | endif |
| 7 | 7 | CFLAGS_core.o += -Wno-override-init $(cflags-nogcse-yy) |
| 8 | 8 | |
| 9 | −obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o tnum.o log.o token.o liveness.o const_fold.o | |
| 9 | +obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o tnum.o cnum.o log.o token.o liveness.o const_fold.o | |
| 10 | 10 | obj-$(CONFIG_BPF_SYSCALL) += bpf_iter.o map_iter.o task_iter.o prog_iter.o link_iter.o |
| 11 | 11 | obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o bloom_filter.o |
| 12 | 12 | obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o bpf_insn_array.o |
| @@ -53,12 +53,15 @@ struct bpf_arena { · 52 unmodified lines | ||
| 53 | 53 | u64 user_vm_start; |
| 54 | 54 | u64 user_vm_end; |
| 55 | 55 | struct vm_struct *kern_vm; |
| 56 | + struct page *scratch_page; | |
| 56 | 57 | struct range_tree rt; |
| 57 | 58 | /* protects rt */ |
| 58 | 59 | rqspinlock_t spinlock; |
| 59 | 60 | struct list_head vma_list; |
| 60 | 61 | /* protects vma_list */ |
| 61 | 62 | struct mutex lock; |
| 63 | + u64 zap_gen; | |
| 64 | + struct mutex zap_mutex; | |
| 62 | 65 | struct irq_work free_irq; |
| 63 | 66 | struct work_struct free_work; |
| 64 | 67 | struct llist_head free_spans; |
| @@ -83,6 +86,32 @@ u64 bpf_arena_get_user_vm_start(struct bpf_arena *arena) · 18 unmodified lines | ||
| 83 | 86 | return arena ? arena->user_vm_start : 0; |
| 84 | 87 | } |
| 85 | 88 | |
| 89 | +/** | |
| 90 | + * bpf_arena_map_kern_vm_start - kern_vm_start lookup by struct bpf_map * | |
| 91 | + * @map: a BPF_MAP_TYPE_ARENA map | |
| 92 | + * | |
| 93 | + * Return @map's kern_vm_start. | |
| 94 | + */ | |
| 95 | +u64 bpf_arena_map_kern_vm_start(struct bpf_map *map) | |
| 96 | +{ | |
| 97 | + return bpf_arena_get_kern_vm_start(container_of(map, struct bpf_arena, map)); | |
| 98 | +} | |
| 99 | + | |
| 100 | +/** | |
| 101 | + * bpf_prog_arena - return the bpf_map of the arena referenced by @prog | |
| 102 | + * @prog: a loaded BPF program | |
| 103 | + * | |
| 104 | + * The verifier enforces at most one arena per program and stores it in | |
| 105 | + * prog->aux->arena. Return that arena's underlying bpf_map, or NULL if | |
| 106 | + * @prog does not reference an arena. | |
| 107 | + */ | |
| 108 | +struct bpf_map *bpf_prog_arena(struct bpf_prog *prog) | |
| 109 | +{ | |
| 110 | + struct bpf_arena *arena = prog->aux->arena; | |
| 111 | + | |
| 112 | + return arena ? &arena->map : NULL; | |
| 113 | +} | |
| 114 | + | |
| 86 | 115 | static long arena_map_peek_elem(struct bpf_map *map, void *value) |
| 87 | 116 | { |
| 88 | 117 | return -EOPNOTSUPP; |
| @@ -115,26 +144,57 @@ static long compute_pgoff(struct bpf_arena *arena, long uaddr) · 26 unmodified lines | ||
| 115 | 144 | |
| 116 | 145 | struct apply_range_data { |
| 117 | 146 | struct page **pages; |
| 147 | + struct page *scratch_page; | |
| 118 | 148 | int i; |
| 119 | 149 | }; |
| 120 | 150 | |
| 151 | +struct clear_range_data { | |
| 152 | + struct llist_head *free_pages; | |
| 153 | + struct page *scratch_page; | |
| 154 | +}; | |
| 155 | + | |
| 121 | 156 | static int apply_range_set_cb(pte_t *pte, unsigned long addr, void *data) |
| 122 | 157 | { |
| 123 | 158 | struct apply_range_data *d = data; |
| 124 | 159 | struct page *page; |
| 160 | + pte_t pteval; | |
| 125 | 161 | |
| 126 | 162 | if (!data) |
| 127 | 163 | return 0; |
| 128 | − /* sanity check */ | |
| 129 | − if (unlikely(!pte_none(ptep_get(pte)))) | |
| 130 | − return -EBUSY; | |
| 131 | 164 | |
| 132 | 165 | page = d->pages[d->i]; |
| 133 | 166 | /* paranoia, similar to vmap_pages_pte_range() */ |
| 134 | 167 | if (WARN_ON_ONCE(!pfn_valid(page_to_pfn(page)))) |
| 135 | 168 | return -EINVAL; |
| 136 | 169 | |
| 137 | − set_pte_at(&init_mm, addr, pte, mk_pte(page, PAGE_KERNEL)); | |
| 170 | + pteval = mk_pte(page, PAGE_KERNEL); | |
| 171 | +#ifdef ptep_try_set | |
| 172 | + /* | |
| 173 | + * Kernel-fault recovery may have installed the scratch page here, and | |
| 174 | + * some architectures (arm64) prohibit valid->valid PTE transitions. | |
| 175 | + * Install atomically into a none slot. If scratch is present, clear it | |
| 176 | + * and flush_tlb_before_set() (break-before-make) before retrying. | |
| 177 | + */ | |
| 178 | + while (!ptep_try_set(pte, pteval)) { | |
| 179 | + pte_t old = ptep_get(pte); | |
| 180 | + | |
| 181 | + if (pte_none(old)) | |
| 182 | + continue; | |
| 183 | + if (WARN_ON_ONCE(pte_page(old) != d->scratch_page)) | |
| 184 | + return -EBUSY; | |
| 185 | + ptep_get_and_clear(&init_mm, addr, pte); | |
| 186 | + flush_tlb_before_set(addr); | |
| 187 | + } | |
| 188 | +#else | |
| 189 | + /* | |
| 190 | + * Without ptep_try_set() there is no atomic installer, but such arches | |
| 191 | + * also do not wire up bpf_arena_handle_page_fault(), so no scratch page | |
| 192 | + * is ever installed and the slot is always none here. | |
| 193 | + */ | |
| 194 | + if (unlikely(!pte_none(ptep_get(pte)))) | |
| 195 | + return -EBUSY; | |
| 196 | + set_pte_at(&init_mm, addr, pte, pteval); | |
| 197 | +#endif | |
| 138 | 198 | d->i++; |
| 139 | 199 | return 0; |
| 140 | 200 | } |
| @@ -144,33 +204,59 @@ static void flush_vmap_cache(unsigned long start, unsigned long size) · 3 unmodified lines | ||
| 144 | 204 | flush_cache_vmap(start, start + size); |
| 145 | 205 | } |
| 146 | 206 | |
| 147 | −static int apply_range_clear_cb(pte_t *pte, unsigned long addr, void *free_pages) | |
| 207 | +static int apply_range_clear_cb(pte_t *pte, unsigned long addr, void *data) | |
| 148 | 208 | { |
| 209 | + struct clear_range_data *d = data; | |
| 149 | 210 | pte_t old_pte; |
| 150 | 211 | struct page *page; |
| 151 | 212 | |
| 152 | − /* sanity check */ | |
| 153 | − old_pte = ptep_get(pte); | |
| 213 | + /* | |
| 214 | + * Pairs with ptep_try_set() in the kernel-fault scratch installer. | |
| 215 | + * Both sides must be atomic. | |
| 216 | + */ | |
| 217 | + old_pte = ptep_get_and_clear(&init_mm, addr, pte); | |
| 154 | 218 | if (pte_none(old_pte) || !pte_present(old_pte)) |
| 155 | − return 0; /* nothing to do */ | |
| 219 | + return 0; | |
| 156 | 220 | |
| 157 | 221 | page = pte_page(old_pte); |
| 158 | 222 | if (WARN_ON_ONCE(!page)) |
| 159 | 223 | return -EINVAL; |
| 160 | 224 | |
| 161 | − pte_clear(&init_mm, addr, pte); | |
| 225 | + /* | |
| 226 | + * Skip the per-arena scratch page. A kernel fault on an unallocated uaddr | |
| 227 | + * scratches its PTE. A later bpf_arena_free_pages() over that range walks | |
| 228 | + * here. Without the skip, scratch_page would be freed. | |
| 229 | + */ | |
| 230 | + if (page == d->scratch_page) | |
| 231 | + return 0; | |
| 232 | + | |
| 233 | + __llist_add(&page->pcp_llist, d->free_pages); | |
| 234 | + return 0; | |
| 235 | +} | |
| 162 | 236 | |
| 163 | − /* Add page to the list so it is freed later */ | |
| 164 | − if (free_pages) | |
| 165 | − __llist_add(&page->pcp_llist, free_pages); | |
| 237 | +static int apply_range_set_scratch_cb(pte_t *pte, unsigned long addr, void *data) | |
| 238 | +{ | |
| 239 | + struct page *scratch_page = data; | |
| 166 | 240 | |
| 241 | + if (!pte_none(ptep_get(pte))) | |
| 242 | + return 0; | |
| 243 | + /* | |
| 244 | + * Best-effort install. ptep_try_set() returns false only if another | |
| 245 | + * installer (real allocation or concurrent fault) won the cmpxchg. | |
| 246 | + * Their PTE is already valid, so the access retry succeeds. | |
| 247 | + * | |
| 248 | + * No flush_tlb_kernel_range() needed. Stale "not mapped" entries just | |
| 249 | + * cause one extra re-fault through this same path. | |
| 250 | + */ | |
| 251 | + ptep_try_set(pte, mk_pte(scratch_page, PAGE_KERNEL)); | |
| 167 | 252 | return 0; |
| 168 | 253 | } |
| 169 | 254 | |
| 170 | 255 | static int populate_pgtable_except_pte(struct bpf_arena *arena) |
| 171 | 256 | { |
| 257 | + /* Populate intermediates for the recovery range (4 GiB + upper half-guard). */ | |
| 172 | 258 | return apply_to_page_range(&init_mm, bpf_arena_get_kern_vm_start(arena), |
| 173 | − KERN_VM_SZ - GUARD_SZ, apply_range_set_cb, NULL); | |
| 259 | + SZ_4G + GUARD_SZ / 2, apply_range_set_cb, NULL); | |
| 174 | 260 | } |
| 175 | 261 | |
| 176 | 262 | static struct bpf_map *arena_map_alloc(union bpf_attr *attr) |
| @@ -221,22 +307,30 @@ static struct bpf_map *arena_map_alloc(union bpf_attr *attr) · 44 unmodified lines | ||
| 221 | 307 | init_irq_work(&arena->free_irq, arena_free_irq); |
| 222 | 308 | INIT_WORK(&arena->free_work, arena_free_worker); |
| 223 | 309 | bpf_map_init_from_attr(&arena->map, attr); |
| 310 | + | |
| 311 | + err = bpf_map_alloc_pages(&arena->map, NUMA_NO_NODE, 1, &arena->scratch_page); | |
| 312 | + if (err) | |
| 313 | + goto err_free_arena; | |
| 314 | + | |
| 224 | 315 | range_tree_init(&arena->rt); |
| 225 | 316 | err = range_tree_set(&arena->rt, 0, attr->max_entries); |
| 226 | − if (err) { | |
| 227 | − bpf_map_area_free(arena); | |
| 228 | − goto err; | |
| 229 | − } | |
| 317 | + if (err) | |
| 318 | + goto err_free_scratch; | |
| 230 | 319 | mutex_init(&arena->lock); |
| 320 | + mutex_init(&arena->zap_mutex); | |
| 231 | 321 | raw_res_spin_lock_init(&arena->spinlock); |
| 232 | 322 | err = populate_pgtable_except_pte(arena); |
| 233 | − if (err) { | |
| 234 | − range_tree_destroy(&arena->rt); | |
| 235 | − bpf_map_area_free(arena); | |
| 236 | − goto err; | |
| 237 | − } | |
| 323 | + if (err) | |
| 324 | + goto err_destroy_rt; | |
| 238 | 325 | |
| 239 | 326 | return &arena->map; |
| 327 | + | |
| 328 | +err_destroy_rt: | |
| 329 | + range_tree_destroy(&arena->rt); | |
| 330 | +err_free_scratch: | |
| 331 | + __free_page(arena->scratch_page); | |
| 332 | +err_free_arena: | |
| 333 | + bpf_map_area_free(arena); | |
| 240 | 334 | err: |
| 241 | 335 | free_vm_area(kern_vm); |
| 242 | 336 | return ERR_PTR(err); |
| @@ -244,6 +338,7 @@ err: · 1 unmodified line | ||
| 244 | 338 | |
| 245 | 339 | static int existing_page_cb(pte_t *ptep, unsigned long addr, void *data) |
| 246 | 340 | { |
| 341 | + struct bpf_arena *arena = data; | |
| 247 | 342 | struct page *page; |
| 248 | 343 | pte_t pte; |
| 249 | 344 | |
| @@ -251,6 +346,12 @@ static int existing_page_cb(pte_t *ptep, unsigned long addr, void *data) · 1 unmodified line | ||
| 251 | 346 | if (!pte_present(pte)) /* sanity check */ |
| 252 | 347 | return 0; |
| 253 | 348 | page = pte_page(pte); |
| 349 | + /* | |
| 350 | + * Skip the scratch page. The walk is page-table-driven, not range-tree-driven, | |
| 351 | + * so it can visit scratch PTEs at uaddrs the BPF program never allocated. | |
| 352 | + */ | |
| 353 | + if (page == arena->scratch_page) | |
| 354 | + return 0; | |
| 254 | 355 | /* |
| 255 | 356 | * We do not update pte here: |
| 256 | 357 | * 1. Nobody should be accessing bpf_arena's range outside of a kernel bug |
| @@ -286,9 +387,10 @@ static void arena_map_free(struct bpf_map *map) · 29 unmodified lines | ||
| 286 | 387 | * free those pages. |
| 287 | 388 | */ |
| 288 | 389 | apply_to_existing_page_range(&init_mm, bpf_arena_get_kern_vm_start(arena), |
| 289 | − KERN_VM_SZ - GUARD_SZ, existing_page_cb, NULL); | |
| 390 | + SZ_4G + GUARD_SZ / 2, existing_page_cb, arena); | |
| 290 | 391 | free_vm_area(arena->kern_vm); |
| 291 | 392 | range_tree_destroy(&arena->rt); |
| 393 | + __free_page(arena->scratch_page); | |
| 292 | 394 | bpf_map_area_free(arena); |
| 293 | 395 | } |
| 294 | 396 | |
| @@ -318,6 +420,7 @@ struct vma_list { · 23 unmodified lines | ||
| 318 | 420 | struct vm_area_struct *vma; |
| 319 | 421 | struct list_head head; |
| 320 | 422 | refcount_t mmap_count; |
| 423 | + u64 zap_gen; | |
| 321 | 424 | }; |
| 322 | 425 | |
| 323 | 426 | static int remember_vma(struct bpf_arena *arena, struct vm_area_struct *vma) |
| @@ -330,6 +433,7 @@ static int remember_vma(struct bpf_arena *arena, struct vm_area_struct *vma) · 6 unmodified lines | ||
| 330 | 433 | refcount_set(&vml->mmap_count, 1); |
| 331 | 434 | vma->vm_private_data = vml; |
| 332 | 435 | vml->vma = vma; |
| 436 | + vml->zap_gen = 0; | |
| 333 | 437 | list_add(&vml->head, &arena->vma_list); |
| 334 | 438 | return 0; |
| 335 | 439 | } |
| @@ -384,33 +488,38 @@ static vm_fault_t arena_vm_fault(struct vm_fault *vmf) · 48 unmodified lines | ||
| 384 | 488 | return VM_FAULT_RETRY; |
| 385 | 489 | |
| 386 | 490 | page = vmalloc_to_page((void *)kaddr); |
| 387 | − if (page) | |
| 491 | + if (page) { | |
| 492 | + if (page == arena->scratch_page) | |
| 493 | + /* BPF triggered scratch here; don't lazy-alloc over it */ | |
| 494 | + goto out_sigsegv; | |
| 388 | 495 | /* already have a page vmap-ed */ |
| 389 | 496 | goto out; |
| 497 | + } | |
| 390 | 498 | |
| 391 | 499 | bpf_map_memcg_enter(&arena->map, &old_memcg, &new_memcg); |
| 392 | 500 | |
| 393 | 501 | if (arena->map.map_flags & BPF_F_SEGV_ON_FAULT) |
| 394 | 502 | /* User space requested to segfault when page is not allocated by bpf prog */ |
| 395 | − goto out_unlock_sigsegv; | |
| 503 | + goto out_sigsegv_memcg; | |
| 396 | 504 | |
| 397 | 505 | ret = range_tree_clear(&arena->rt, vmf->pgoff, 1); |
| 398 | 506 | if (ret) |
| 399 | − goto out_unlock_sigsegv; | |
| 507 | + goto out_sigsegv_memcg; | |
| 400 | 508 | |
| 401 | − struct apply_range_data data = { .pages = &page, .i = 0 }; | |
| 509 | + struct apply_range_data data = { .pages = &page, .i = 0, | |
| 510 | + .scratch_page = arena->scratch_page }; | |
| 402 | 511 | /* Account into memcg of the process that created bpf_arena */ |
| 403 | 512 | ret = bpf_map_alloc_pages(map, NUMA_NO_NODE, 1, &page); |
| 404 | 513 | if (ret) { |
| 405 | 514 | range_tree_set(&arena->rt, vmf->pgoff, 1); |
| 406 | − goto out_unlock_sigsegv; | |
| 515 | + goto out_sigsegv_memcg; | |
| 407 | 516 | } |
| 408 | 517 | |
| 409 | 518 | ret = apply_to_page_range(&init_mm, kaddr, PAGE_SIZE, apply_range_set_cb, &data); |
| 410 | 519 | if (ret) { |
| 411 | 520 | range_tree_set(&arena->rt, vmf->pgoff, 1); |
| 412 | 521 | free_pages_nolock(page, 0); |
| 413 | − goto out_unlock_sigsegv; | |
| 522 | + goto out_sigsegv_memcg; | |
| 414 | 523 | } |
| 415 | 524 | flush_vmap_cache(kaddr, PAGE_SIZE); |
| 416 | 525 | bpf_map_memcg_exit(old_memcg, new_memcg); |
| @@ -419,8 +528,9 @@ out: · 2 unmodified lines | ||
| 419 | 528 | raw_res_spin_unlock_irqrestore(&arena->spinlock, flags); |
| 420 | 529 | vmf->page = page; |
| 421 | 530 | return 0; |
| 422 | −out_unlock_sigsegv: | |
| 531 | +out_sigsegv_memcg: | |
| 423 | 532 | bpf_map_memcg_exit(old_memcg, new_memcg); |
| 533 | +out_sigsegv: | |
| 424 | 534 | raw_res_spin_unlock_irqrestore(&arena->spinlock, flags); |
| 425 | 535 | return VM_FAULT_SIGSEGV; |
| 426 | 536 | } |
| @@ -587,6 +697,7 @@ static long arena_alloc_pages(struct bpf_arena *arena, long uaddr, long page_cnt · 160 unmodified lines | ||
| 587 | 697 | return 0; |
| 588 | 698 | } |
| 589 | 699 | data.pages = pages; |
| 700 | + data.scratch_page = arena->scratch_page; | |
| 590 | 701 | |
| 591 | 702 | if (raw_res_spin_lock_irqsave(&arena->spinlock, flags)) |
| 592 | 703 | goto out_free_pages; |
| @@ -668,12 +779,60 @@ out_free_pages: · 75 unmodified lines | ||
| 668 | 779 | */ |
| 669 | 780 | static void zap_pages(struct bpf_arena *arena, long uaddr, long page_cnt) |
| 670 | 781 | { |
| 782 | + unsigned long size = (unsigned long)page_cnt << PAGE_SHIFT; | |
| 783 | + struct vm_area_struct *vma; | |
| 784 | + struct mm_struct *mm; | |
| 671 | 785 | struct vma_list *vml; |
| 786 | + unsigned long vm_start; | |
| 787 | + u64 my_gen; | |
| 672 | 788 | |
| 673 | − guard(mutex)(&arena->lock); | |
| 674 | − /* iterate link list under lock */ | |
| 675 | − list_for_each_entry(vml, &arena->vma_list, head) | |
| 676 | − zap_vma_range(vml->vma, uaddr, PAGE_SIZE * page_cnt); | |
| 789 | + /* | |
| 790 | + * Taking mmap_read_lock() under arena->lock would deadlock against | |
| 791 | + * arena_vm_close(), which runs with mmap_write_lock held and then | |
| 792 | + * acquires arena->lock. Drop arena->lock for mmap_read_lock(). | |
| 793 | + * | |
| 794 | + * Use per-call my_gen, recorded in vml->zap_gen, to remember which | |
| 795 | + * vmls this invocation has already processed across the lock drop. | |
| 796 | + * Hold zap_mutex around the whole walk so concurrent zap_pages() | |
| 797 | + * callers cannot overwrite each other's marks on shared vmls -- | |
| 798 | + * otherwise call B's mark would make call A skip a vml that A has | |
| 799 | + * not yet zapped for A's uaddr range. | |
| 800 | + */ | |
| 801 | + mutex_lock(&arena->zap_mutex); | |
| 802 | + mutex_lock(&arena->lock); | |
| 803 | + my_gen = ++arena->zap_gen; | |
| 804 | + for (;;) { | |
| 805 | + mm = NULL; | |
| 806 | + list_for_each_entry(vml, &arena->vma_list, head) { | |
| 807 | + if (vml->zap_gen >= my_gen) | |
| 808 | + continue; | |
| 809 | + vml->zap_gen = my_gen; | |
| 810 | + if (!mmget_not_zero(vml->vma->vm_mm)) | |
| 811 | + continue; | |
| 812 | + mm = vml->vma->vm_mm; | |
| 813 | + vm_start = vml->vma->vm_start; | |
| 814 | + break; | |
| 815 | + } | |
| 816 | + if (!mm) | |
| 817 | + break; | |
| 818 | + mutex_unlock(&arena->lock); | |
| 819 | + | |
| 820 | + mmap_read_lock(mm); | |
| 821 | + /* | |
| 822 | + * Re-resolve: while we waited the VMA could have been unmapped | |
| 823 | + * and a different mapping installed at the same address. | |
| 824 | + */ | |
| 825 | + vma = find_vma(mm, vm_start); | |
| 826 | + if (vma && vma->vm_start == vm_start && | |
| 827 | + vma->vm_file && vma->vm_file->private_data == &arena->map) | |
| 828 | + zap_vma_range(vma, uaddr, size); | |
| 829 | + mmap_read_unlock(mm); | |
| 830 | + mmput(mm); | |
| 831 | + | |
| 832 | + mutex_lock(&arena->lock); | |
| 833 | + } | |
| 834 | + mutex_unlock(&arena->lock); | |
| 835 | + mutex_unlock(&arena->zap_mutex); | |
| 677 | 836 | } |
| 678 | 837 | |
| 679 | 838 | static void arena_free_pages(struct bpf_arena *arena, long uaddr, long page_cnt, bool sleepable) |
| @@ -685,6 +844,7 @@ static void arena_free_pages(struct bpf_arena *arena, long uaddr, long page_cnt, · 5 unmodified lines | ||
| 685 | 844 | struct llist_head free_pages; |
| 686 | 845 | struct llist_node *pos, *t; |
| 687 | 846 | struct arena_free_span *s; |
| 847 | + struct clear_range_data cdata; | |
| 688 | 848 | unsigned long flags; |
| 689 | 849 | int ret = 0; |
| 690 | 850 | |
| @@ -713,9 +873,11 @@ static void arena_free_pages(struct bpf_arena *arena, long uaddr, long page_cnt, · 22 unmodified lines | ||
| 713 | 873 | range_tree_set(&arena->rt, pgoff, page_cnt); |
| 714 | 874 | |
| 715 | 875 | init_llist_head(&free_pages); |
| 876 | + cdata.free_pages = &free_pages; | |
| 877 | + cdata.scratch_page = arena->scratch_page; | |
| 716 | 878 | /* clear ptes and collect struct pages */ |
| 717 | 879 | apply_to_existing_page_range(&init_mm, kaddr, page_cnt << PAGE_SHIFT, |
| 718 | − apply_range_clear_cb, &free_pages); | |
| 880 | + apply_range_clear_cb, &cdata); | |
| 719 | 881 | |
| 720 | 882 | /* drop the lock to do the tlb flush and zap pages */ |
| 721 | 883 | raw_res_spin_unlock_irqrestore(&arena->spinlock, flags); |
| @@ -805,6 +967,7 @@ static void arena_free_worker(struct work_struct *work) · 83 unmodified lines | ||
| 805 | 967 | struct arena_free_span *s; |
| 806 | 968 | u64 arena_vm_start, user_vm_start; |
| 807 | 969 | struct llist_head free_pages; |
| 970 | + struct clear_range_data cdata; | |
| 808 | 971 | struct page *page; |
| 809 | 972 | unsigned long full_uaddr; |
| 810 | 973 | long kaddr, page_cnt, pgoff; |
| @@ -818,6 +981,8 @@ static void arena_free_worker(struct work_struct *work) · 7 unmodified lines | ||
| 818 | 981 | bpf_map_memcg_enter(&arena->map, &old_memcg, &new_memcg); |
| 819 | 982 | |
| 820 | 983 | init_llist_head(&free_pages); |
| 984 | + cdata.free_pages = &free_pages; | |
| 985 | + cdata.scratch_page = arena->scratch_page; | |
| 821 | 986 | arena_vm_start = bpf_arena_get_kern_vm_start(arena); |
| 822 | 987 | user_vm_start = bpf_arena_get_user_vm_start(arena); |
| 823 | 988 | |
| @@ -830,7 +995,7 @@ static void arena_free_worker(struct work_struct *work) · 6 unmodified lines | ||
| 830 | 995 | |
| 831 | 996 | /* clear ptes and collect pages in free_pages llist */ |
| 832 | 997 | apply_to_existing_page_range(&init_mm, kaddr, page_cnt << PAGE_SHIFT, |
| 833 | − apply_range_clear_cb, &free_pages); | |
| 998 | + apply_range_clear_cb, &cdata); | |
| 834 | 999 | |
| 835 | 1000 | range_tree_set(&arena->rt, pgoff, page_cnt); |
| 836 | 1001 | } |
| @@ -893,6 +1058,19 @@ void *bpf_arena_alloc_pages_non_sleepable(void *p__map, void *addr__ign, u32 pag · 56 unmodified lines | ||
| 893 | 1058 | |
| 894 | 1059 | return (void *)arena_alloc_pages(arena, (long)addr__ign, page_cnt, node_id, false); |
| 895 | 1060 | } |
| 1061 | + | |
| 1062 | +void *bpf_arena_alloc_pages_sleepable(void *p__map, void *addr__ign, u32 page_cnt, | |
| 1063 | + int node_id, u64 flags) | |
| 1064 | +{ | |
| 1065 | + struct bpf_map *map = p__map; | |
| 1066 | + struct bpf_arena *arena = container_of(map, struct bpf_arena, map); | |
| 1067 | + | |
| 1068 | + if (map->map_type != BPF_MAP_TYPE_ARENA || flags || !page_cnt) | |
| 1069 | + return NULL; | |
| 1070 | + | |
| 1071 | + return (void *)arena_alloc_pages(arena, (long)addr__ign, page_cnt, node_id, true); | |
| 1072 | +} | |
| 1073 | + | |
| 896 | 1074 | __bpf_kfunc void bpf_arena_free_pages(void *p__map, void *ptr__ign, u32 page_cnt) |
| 897 | 1075 | { |
| 898 | 1076 | struct bpf_map *map = p__map; |
| @@ -945,23 +1123,12 @@ static int __init kfunc_init(void) · 46 unmodified lines | ||
| 945 | 1123 | } |
| 946 | 1124 | late_initcall(kfunc_init); |
| 947 | 1125 | |
| 948 | −void bpf_prog_report_arena_violation(bool write, unsigned long addr, unsigned long fault_ip) | |
| 1126 | +static void __bpf_prog_report_arena_violation(struct bpf_prog *prog, bool write, | |
| 1127 | + unsigned long addr, unsigned long fault_ip) | |
| 949 | 1128 | { |
| 950 | 1129 | struct bpf_stream_stage ss; |
| 951 | − struct bpf_prog *prog; | |
| 952 | 1130 | u64 user_vm_start; |
| 953 | 1131 | |
| 954 | − /* | |
| 955 | − * The RCU read lock is held to safely traverse the latch tree, but we | |
| 956 | − * don't need its protection when accessing the prog, since it will not | |
| 957 | − * disappear while we are handling the fault. | |
| 958 | − */ | |
| 959 | − rcu_read_lock(); | |
| 960 | − prog = bpf_prog_ksym_find(fault_ip); | |
| 961 | − rcu_read_unlock(); | |
| 962 | − if (!prog) | |
| 963 | − return; | |
| 964 | − | |
| 965 | 1132 | /* Use main prog for stream access */ |
| 966 | 1133 | prog = prog->aux->main_prog_aux->prog; |
| 967 | 1134 | |
| @@ -974,3 +1141,53 @@ void bpf_prog_report_arena_violation(bool write, unsigned long addr, unsigned lo · 6 unmodified lines | ||
| 974 | 1141 | bpf_stream_dump_stack(ss); |
| 975 | 1142 | })); |
| 976 | 1143 | } |
| 1144 | + | |
| 1145 | +bool bpf_arena_handle_page_fault(unsigned long addr, bool is_write, unsigned long fault_ip) | |
| 1146 | +{ | |
| 1147 | + struct bpf_arena *arena; | |
| 1148 | + struct bpf_prog *prog; | |
| 1149 | + unsigned long kbase; | |
| 1150 | + unsigned long page_addr = addr & PAGE_MASK; | |
| 1151 | + | |
| 1152 | + prog = bpf_prog_find_from_stack(); | |
| 1153 | + if (!prog) | |
| 1154 | + return false; | |
| 1155 | + | |
| 1156 | + arena = prog->aux->arena; | |
| 1157 | + /* a prog not using arena may be on stack, so arena can be NULL */ | |
| 1158 | + if (!arena) | |
| 1159 | + return false; | |
| 1160 | + | |
| 1161 | + kbase = bpf_arena_get_kern_vm_start(arena); | |
| 1162 | + | |
| 1163 | + /* | |
| 1164 | + * Recovery covers the 4 GiB mappable band plus the upper half-guard. | |
| 1165 | + * Lower guard is unreachable from kfuncs; an address there indicates | |
| 1166 | + * a different bug class - leave it to the regular kernel oops path. | |
| 1167 | + */ | |
| 1168 | + if (page_addr < kbase || page_addr >= kbase + SZ_4G + GUARD_SZ / 2) | |
| 1169 | + return false; | |
| 1170 | + | |
| 1171 | + apply_to_page_range(&init_mm, page_addr, PAGE_SIZE, | |
| 1172 | + apply_range_set_scratch_cb, arena->scratch_page); | |
| 1173 | + flush_vmap_cache(page_addr, PAGE_SIZE); | |
| 1174 | + __bpf_prog_report_arena_violation(prog, is_write, page_addr - kbase, fault_ip); | |
| 1175 | + return true; | |
| 1176 | +} | |
| 1177 | + | |
| 1178 | +void bpf_prog_report_arena_violation(bool write, unsigned long addr, unsigned long fault_ip) | |
| 1179 | +{ | |
| 1180 | + struct bpf_prog *prog; | |
| 1181 | + | |
| 1182 | + /* | |
| 1183 | + * The RCU read lock is held to safely traverse the latch tree, but we | |
| 1184 | + * don't need its protection when accessing the prog, since it will not | |
| 1185 | + * disappear while we are handling the fault. | |
| 1186 | + */ | |
| 1187 | + rcu_read_lock(); | |
| 1188 | + prog = bpf_prog_ksym_find(fault_ip); | |
| 1189 | + rcu_read_unlock(); | |
| 1190 | + if (!prog) | |
| 1191 | + return; | |
| 1192 | + __bpf_prog_report_arena_violation(prog, write, addr, fault_ip); | |
| 1193 | +} | |
| @@ -175,14 +175,12 @@ static void *array_map_lookup_elem(struct bpf_map *map, void *key) · 174 unmodified lines | ||
| 175 | 175 | return array->value + (u64)array->elem_size * (index & array->index_mask); |
| 176 | 176 | } |
| 177 | 177 | |
| 178 | −static int array_map_get_hash(struct bpf_map *map, u32 hash_buf_size, | |
| 179 | − void *hash_buf) | |
| 178 | +static int array_map_get_hash(struct bpf_map *map) | |
| 180 | 179 | { |
| 181 | 180 | struct bpf_array *array = container_of(map, struct bpf_array, map); |
| 182 | 181 | |
| 183 | 182 | sha256(array->value, (u64)array->elem_size * array->map.max_entries, |
| 184 | − hash_buf); | |
| 185 | − memcpy(array->map.sha, hash_buf, sizeof(array->map.sha)); | |
| 183 | + array->map.sha); | |
| 186 | 184 | return 0; |
| 187 | 185 | } |
| 188 | 186 | |
| @@ -386,7 +384,7 @@ static long array_map_update_elem(struct bpf_map *map, void *key, void *value, · 197 unmodified lines | ||
| 386 | 384 | if (array->map.map_type == BPF_MAP_TYPE_PERCPU_ARRAY) { |
| 387 | 385 | val = this_cpu_ptr(array->pptrs[index & array->index_mask]); |
| 388 | 386 | copy_map_value(map, val, value); |
| 389 | − bpf_obj_free_fields(array->map.record, val); | |
| 387 | + bpf_obj_cancel_fields(map, val); | |
| 390 | 388 | } else { |
| 391 | 389 | val = array->value + |
| 392 | 390 | (u64)array->elem_size * (index & array->index_mask); |
| @@ -394,7 +392,7 @@ static long array_map_update_elem(struct bpf_map *map, void *key, void *value, · 1 unmodified line | ||
| 394 | 392 | copy_map_value_locked(map, val, value, false); |
| 395 | 393 | else |
| 396 | 394 | copy_map_value(map, val, value); |
| 397 | − bpf_obj_free_fields(array->map.record, val); | |
| 395 | + bpf_obj_cancel_fields(map, val); | |
| 398 | 396 | } |
| 399 | 397 | return 0; |
| 400 | 398 | } |
| @@ -434,14 +432,14 @@ int bpf_percpu_array_update(struct bpf_map *map, void *key, void *value, · 33 unmodified lines | ||
| 434 | 432 | cpu = map_flags >> 32; |
| 435 | 433 | ptr = per_cpu_ptr(pptr, cpu); |
| 436 | 434 | copy_map_value(map, ptr, value); |
| 437 | − bpf_obj_free_fields(array->map.record, ptr); | |
| 435 | + bpf_obj_cancel_fields(map, ptr); | |
| 438 | 436 | goto unlock; |
| 439 | 437 | } |
| 440 | 438 | for_each_possible_cpu(cpu) { |
| 441 | 439 | ptr = per_cpu_ptr(pptr, cpu); |
| 442 | 440 | val = (map_flags & BPF_F_ALL_CPUS) ? value : value + size * cpu; |
| 443 | 441 | copy_map_value(map, ptr, val); |
| 444 | − bpf_obj_free_fields(array->map.record, ptr); | |
| 442 | + bpf_obj_cancel_fields(map, ptr); | |
| 445 | 443 | } |
| 446 | 444 | unlock: |
| 447 | 445 | rcu_read_unlock(); |
| @@ -9,7 +9,7 @@ · 8 unmodified lines | ||
| 9 | 9 | |
| 10 | 10 | /* for any branch, call, exit record the history of jmps in the given state */ |
| 11 | 11 | int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state *cur, |
| 12 | − int insn_flags, u64 linked_regs) | |
| 12 | + int insn_flags, int spi, int frame, u64 linked_regs) | |
| 13 | 13 | { |
| 14 | 14 | u32 cnt = cur->jmp_history_cnt; |
| 15 | 15 | struct bpf_jmp_history_entry *p; |
| @@ -25,6 +25,8 @@ int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state · 9 unmodified lines | ||
| 25 | 25 | env, "insn history: insn_idx %d cur flags %x new flags %x", |
| 26 | 26 | env->insn_idx, env->cur_hist_ent->flags, insn_flags); |
| 27 | 27 | env->cur_hist_ent->flags |= insn_flags; |
| 28 | + env->cur_hist_ent->spi = spi; | |
| 29 | + env->cur_hist_ent->frame = frame; | |
| 28 | 30 | verifier_bug_if(env->cur_hist_ent->linked_regs != 0, env, |
| 29 | 31 | "insn history: insn_idx %d linked_regs: %#llx", |
| 30 | 32 | env->insn_idx, env->cur_hist_ent->linked_regs); |
| @@ -43,6 +45,8 @@ int bpf_push_jmp_history(struct bpf_verifier_env *env, struct bpf_verifier_state · 12 unmodified lines | ||
| 43 | 45 | p->idx = env->insn_idx; |
| 44 | 46 | p->prev_idx = env->prev_insn_idx; |
| 45 | 47 | p->flags = insn_flags; |
| 48 | + p->spi = spi; | |
| 49 | + p->frame = frame; | |
| 46 | 50 | p->linked_regs = linked_regs; |
| 47 | 51 | cur->jmp_history_cnt = cnt; |
| 48 | 52 | env->cur_hist_ent = p; |
| @@ -64,16 +68,6 @@ static bool is_atomic_fetch_insn(const struct bpf_insn *insn) · 15 unmodified lines | ||
| 64 | 68 | (insn->imm & BPF_FETCH); |
| 65 | 69 | } |
| 66 | 70 | |
| 67 | −static int insn_stack_access_spi(int insn_flags) | |
| 68 | −{ | |
| 69 | − return (insn_flags >> INSN_F_SPI_SHIFT) & INSN_F_SPI_MASK; | |
| 70 | −} | |
| 71 | − | |
| 72 | −static int insn_stack_access_frameno(int insn_flags) | |
| 73 | −{ | |
| 74 | − return insn_flags & INSN_F_FRAMENO_MASK; | |
| 75 | −} | |
| 76 | − | |
| 77 | 71 | /* Backtrack one insn at a time. If idx is not at the top of recorded |
| 78 | 72 | * history then previous instruction came from straight line execution. |
| 79 | 73 | * Return -ENOENT if we exhausted all instructions within given state. |
| @@ -135,11 +129,21 @@ static inline u32 bt_empty(struct backtrack_state *bt) · 55 unmodified lines | ||
| 135 | 129 | int i; |
| 136 | 130 | |
| 137 | 131 | for (i = 0; i <= bt->frame; i++) |
| 138 | − mask |= bt->reg_masks[i] | bt->stack_masks[i]; | |
| 132 | + mask |= bt->reg_masks[i] | bt->stack_masks[i] | bt->stack_arg_masks[i]; | |
| 139 | 133 | |
| 140 | 134 | return mask == 0; |
| 141 | 135 | } |
| 142 | 136 | |
| 137 | +static inline void bt_clear_frame_stack_arg_slot(struct backtrack_state *bt, u32 frame, u32 slot) | |
| 138 | +{ | |
| 139 | + bt->stack_arg_masks[frame] &= ~(1 << slot); | |
| 140 | +} | |
| 141 | + | |
| 142 | +static inline bool bt_is_frame_stack_arg_slot_set(struct backtrack_state *bt, u32 frame, u32 slot) | |
| 143 | +{ | |
| 144 | + return bt->stack_arg_masks[frame] & (1 << slot); | |
| 145 | +} | |
| 146 | + | |
| 143 | 147 | static inline int bt_subprog_enter(struct backtrack_state *bt) |
| 144 | 148 | { |
| 145 | 149 | if (bt->frame == MAX_CALL_FRAMES - 1) { |
| @@ -200,6 +204,11 @@ static inline u64 bt_stack_mask(struct backtrack_state *bt) · 54 unmodified lines | ||
| 200 | 204 | return bt->stack_masks[bt->frame]; |
| 201 | 205 | } |
| 202 | 206 | |
| 207 | +static inline u8 bt_stack_arg_mask(struct backtrack_state *bt) | |
| 208 | +{ | |
| 209 | + return bt->stack_arg_masks[bt->frame]; | |
| 210 | +} | |
| 211 | + | |
| 203 | 212 | static inline bool bt_is_reg_set(struct backtrack_state *bt, u32 reg) |
| 204 | 213 | { |
| 205 | 214 | return bt->reg_masks[bt->frame] & (1 << reg); |
| @@ -341,6 +350,19 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx, · 135 unmodified lines | ||
| 341 | 350 | return 0; |
| 342 | 351 | bt_clear_reg(bt, load_reg); |
| 343 | 352 | |
| 353 | + if (hist && hist->flags & INSN_F_STACK_ARG_ACCESS) { | |
| 354 | + spi = hist->spi; | |
| 355 | + /* | |
| 356 | + * Stack arg read: callee reads from r11+off, but | |
| 357 | + * the data lives in the caller's stack_arg_regs. | |
| 358 | + * Set the mask in the caller frame so precision | |
| 359 | + * is marked in the caller's slot at the callee | |
| 360 | + * entry checkpoint. | |
| 361 | + */ | |
| 362 | + bt_set_frame_stack_arg_slot(bt, bt->frame - 1, spi); | |
| 363 | + return 0; | |
| 364 | + } | |
| 365 | + | |
| 344 | 366 | /* scalars can only be spilled into stack w/o losing precision. |
| 345 | 367 | * Load from any other memory can be zero extended. |
| 346 | 368 | * The desire to keep that precision is already indicated |
| @@ -353,8 +375,8 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx, · 6 unmodified lines | ||
| 353 | 375 | * that [fp - off] slot contains scalar that needs to be |
| 354 | 376 | * tracked with precision |
| 355 | 377 | */ |
| 356 | − spi = insn_stack_access_spi(hist->flags); | |
| 357 | − fr = insn_stack_access_frameno(hist->flags); | |
| 378 | + spi = hist->spi; | |
| 379 | + fr = hist->frame; | |
| 358 | 380 | bpf_bt_set_frame_slot(bt, fr, spi); |
| 359 | 381 | } else if (class == BPF_STX || class == BPF_ST) { |
| 360 | 382 | if (bt_is_reg_set(bt, dreg)) |
| @@ -363,11 +385,22 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx, · 2 unmodified lines | ||
| 363 | 385 | * encountered a case of pointer subtraction. |
| 364 | 386 | */ |
| 365 | 387 | return -ENOTSUPP; |
| 388 | + | |
| 389 | + if (hist && hist->flags & INSN_F_STACK_ARG_ACCESS) { | |
| 390 | + spi = hist->spi; | |
| 391 | + if (!bt_is_frame_stack_arg_slot_set(bt, bt->frame, spi)) | |
| 392 | + return 0; | |
| 393 | + bt_clear_frame_stack_arg_slot(bt, bt->frame, spi); | |
| 394 | + if (class == BPF_STX) | |
| 395 | + bt_set_reg(bt, sreg); | |
| 396 | + return 0; | |
| 397 | + } | |
| 398 | + | |
| 366 | 399 | /* scalars can only be spilled into stack */ |
| 367 | 400 | if (!hist || !(hist->flags & INSN_F_STACK_ACCESS)) |
| 368 | 401 | return 0; |
| 369 | − spi = insn_stack_access_spi(hist->flags); | |
| 370 | − fr = insn_stack_access_frameno(hist->flags); | |
| 402 | + spi = hist->spi; | |
| 403 | + fr = hist->frame; | |
| 371 | 404 | if (!bt_is_frame_slot_set(bt, fr, spi)) |
| 372 | 405 | return 0; |
| 373 | 406 | bt_clear_frame_slot(bt, fr, spi); |
| @@ -431,6 +464,12 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx, · 57 unmodified lines | ||
| 431 | 464 | bpf_bt_set_frame_reg(bt, bt->frame - 1, i); |
| 432 | 465 | } |
| 433 | 466 | } |
| 467 | + if (bt_stack_arg_mask(bt)) { | |
| 468 | + verifier_bug(env, | |
| 469 | + "static subprog leftover stack arg slots %x", | |
| 470 | + bt_stack_arg_mask(bt)); | |
| 471 | + return -EFAULT; | |
| 472 | + } | |
| 434 | 473 | if (bt_subprog_exit(bt)) |
| 435 | 474 | return -EFAULT; |
| 436 | 475 | return 0; |
| @@ -901,6 +940,17 @@ int bpf_mark_chain_precision(struct bpf_verifier_env *env, · 464 unmodified lines | ||
| 901 | 940 | *changed = true; |
| 902 | 941 | } |
| 903 | 942 | } |
| 943 | + for (i = 0; i < func->out_stack_arg_cnt; i++) { | |
| 944 | + if (!bt_is_frame_stack_arg_slot_set(bt, fr, i)) | |
| 945 | + continue; | |
| 946 | + reg = &func->stack_arg_regs[i]; | |
| 947 | + if (reg->type != SCALAR_VALUE || reg->precise) { | |
| 948 | + bt_clear_frame_stack_arg_slot(bt, fr, i); | |
| 949 | + } else { | |
| 950 | + reg->precise = true; | |
| 951 | + *changed = true; | |
| 952 | + } | |
| 953 | + } | |
| 904 | 954 | if (env->log.level & BPF_LOG_LEVEL2) { |
| 905 | 955 | fmt_reg_mask(env->tmp_str_buf, TMP_STR_BUF_LEN, |
| 906 | 956 | bt_frame_reg_mask(bt, fr)); |
| @@ -13,23 +13,8 @@ · 12 unmodified lines | ||
| 13 | 13 | #define PERCPU_FREE_TARGET (4) |
| 14 | 14 | #define PERCPU_NR_SCANS PERCPU_FREE_TARGET |
| 15 | 15 | |
| 16 | −/* Helpers to get the local list index */ | |
| 17 | −#define LOCAL_LIST_IDX(t) ((t) - BPF_LOCAL_LIST_T_OFFSET) | |
| 18 | −#define LOCAL_FREE_LIST_IDX LOCAL_LIST_IDX(BPF_LRU_LOCAL_LIST_T_FREE) | |
| 19 | −#define LOCAL_PENDING_LIST_IDX LOCAL_LIST_IDX(BPF_LRU_LOCAL_LIST_T_PENDING) | |
| 20 | 16 | #define IS_LOCAL_LIST_TYPE(t) ((t) >= BPF_LOCAL_LIST_T_OFFSET) |
| 21 | 17 | |
| 22 | −/* Local list helpers */ | |
| 23 | −static struct list_head *local_free_list(struct bpf_lru_locallist *loc_l) | |
| 24 | −{ | |
| 25 | − return &loc_l->lists[LOCAL_FREE_LIST_IDX]; | |
| 26 | −} | |
| 27 | − | |
| 28 | −static struct list_head *local_pending_list(struct bpf_lru_locallist *loc_l) | |
| 29 | −{ | |
| 30 | − return &loc_l->lists[LOCAL_PENDING_LIST_IDX]; | |
| 31 | −} | |
| 32 | − | |
| 33 | 18 | /* bpf_lru_node helpers */ |
| 34 | 19 | static bool bpf_lru_node_is_ref(const struct bpf_lru_node *node) |
| 35 | 20 | { |
| @@ -72,6 +57,7 @@ static void __bpf_lru_node_move_to_free(struct bpf_lru_list *l, · 36 unmodified lines | ||
| 72 | 57 | bpf_lru_list_count_dec(l, node->type); |
| 73 | 58 | |
| 74 | 59 | node->type = tgt_free_type; |
| 60 | + WRITE_ONCE(node->pending_free, 0); | |
| 75 | 61 | list_move(&node->list, free_list); |
| 76 | 62 | } |
| 77 | 63 | |
| @@ -87,6 +73,9 @@ static void __bpf_lru_node_move_in(struct bpf_lru_list *l, · 9 unmodified lines | ||
| 87 | 73 | bpf_lru_list_count_inc(l, tgt_type); |
| 88 | 74 | node->type = tgt_type; |
| 89 | 75 | bpf_lru_node_clear_ref(node); |
| 76 | + /* Reset pending_free only when moving to the free list */ | |
| 77 | + if (tgt_type == BPF_LRU_LIST_T_FREE) | |
| 78 | + WRITE_ONCE(node->pending_free, 0); | |
| 90 | 79 | list_move(&node->list, &l->lists[tgt_type]); |
| 91 | 80 | } |
| 92 | 81 | |
| @@ -212,9 +201,11 @@ __bpf_lru_list_shrink_inactive(struct bpf_lru *lru, · 119 unmodified lines | ||
| 212 | 201 | unsigned int i = 0; |
| 213 | 202 | |
| 214 | 203 | list_for_each_entry_safe_reverse(node, tmp_node, inactive, list) { |
| 215 | − if (bpf_lru_node_is_ref(node)) { | |
| 204 | + if (bpf_lru_node_is_ref(node) && | |
| 205 | + !READ_ONCE(node->pending_free)) { | |
| 216 | 206 | __bpf_lru_node_move(l, node, BPF_LRU_LIST_T_ACTIVE); |
| 217 | − } else if (lru->del_from_htab(lru->del_arg, node)) { | |
| 207 | + } else if (READ_ONCE(node->pending_free) || | |
| 208 | + lru->del_from_htab(lru->del_arg, node)) { | |
| 218 | 209 | __bpf_lru_node_move_to_free(l, node, free_list, |
| 219 | 210 | tgt_free_type); |
| 220 | 211 | if (++nshrinked == tgt_nshrink) |
| @@ -273,7 +264,8 @@ static unsigned int __bpf_lru_list_shrink(struct bpf_lru *lru, · 52 unmodified lines | ||
| 273 | 264 | |
| 274 | 265 | list_for_each_entry_safe_reverse(node, tmp_node, force_shrink_list, |
| 275 | 266 | list) { |
| 276 | − if (lru->del_from_htab(lru->del_arg, node)) { | |
| 267 | + if (READ_ONCE(node->pending_free) || | |
| 268 | + lru->del_from_htab(lru->del_arg, node)) { | |
| 277 | 269 | __bpf_lru_node_move_to_free(l, node, free_list, |
| 278 | 270 | tgt_free_type); |
| 279 | 271 | return 1; |
| @@ -290,8 +282,10 @@ static void __local_list_flush(struct bpf_lru_list *l, · 10 unmodified lines | ||
| 290 | 282 | struct bpf_lru_node *node, *tmp_node; |
| 291 | 283 | |
| 292 | 284 | list_for_each_entry_safe_reverse(node, tmp_node, |
| 293 | − local_pending_list(loc_l), list) { | |
| 294 | − if (bpf_lru_node_is_ref(node)) | |
| 285 | + &loc_l->pending_list, list) { | |
| 286 | + if (READ_ONCE(node->pending_free)) | |
| 287 | + __bpf_lru_node_move_in(l, node, BPF_LRU_LIST_T_FREE); | |
| 288 | + else if (bpf_lru_node_is_ref(node)) | |
| 295 | 289 | __bpf_lru_node_move_in(l, node, BPF_LRU_LIST_T_ACTIVE); |
| 296 | 290 | else |
| 297 | 291 | __bpf_lru_node_move_in(l, node, |
| @@ -307,9 +301,12 @@ static void bpf_lru_list_push_free(struct bpf_lru_list *l, · 9 unmodified lines | ||
| 307 | 301 | if (WARN_ON_ONCE(IS_LOCAL_LIST_TYPE(node->type))) |
| 308 | 302 | return; |
| 309 | 303 | |
| 310 | − raw_spin_lock_irqsave(&l->lock, flags); | |
| 304 | + if (raw_res_spin_lock_irqsave(&l->lock, flags)) { | |
| 305 | + WRITE_ONCE(node->pending_free, 1); | |
| 306 | + return; | |
| 307 | + } | |
| 311 | 308 | __bpf_lru_node_move(l, node, BPF_LRU_LIST_T_FREE); |
| 312 | − raw_spin_unlock_irqrestore(&l->lock, flags); | |
| 309 | + raw_res_spin_unlock_irqrestore(&l->lock, flags); | |
| 313 | 310 | } |
| 314 | 311 | |
| 315 | 312 | static void bpf_lru_list_pop_free_to_local(struct bpf_lru *lru, |
| @@ -318,8 +315,10 @@ static void bpf_lru_list_pop_free_to_local(struct bpf_lru *lru, · 2 unmodified lines | ||
| 318 | 315 | struct bpf_lru_list *l = &lru->common_lru.lru_list; |
| 319 | 316 | struct bpf_lru_node *node, *tmp_node; |
| 320 | 317 | unsigned int nfree = 0; |
| 318 | + LIST_HEAD(tmp_free); | |
| 321 | 319 | |
| 322 | − raw_spin_lock(&l->lock); | |
| 320 | + if (raw_res_spin_lock(&l->lock)) | |
| 321 | + return; | |
| 323 | 322 | |
| 324 | 323 | __local_list_flush(l, loc_l); |
| 325 | 324 | |
| @@ -327,7 +326,7 @@ static void bpf_lru_list_pop_free_to_local(struct bpf_lru *lru, · 1 unmodified line | ||
| 327 | 326 | |
| 328 | 327 | list_for_each_entry_safe(node, tmp_node, &l->lists[BPF_LRU_LIST_T_FREE], |
| 329 | 328 | list) { |
| 330 | − __bpf_lru_node_move_to_free(l, node, local_free_list(loc_l), | |
| 329 | + __bpf_lru_node_move_to_free(l, node, &tmp_free, | |
| 331 | 330 | BPF_LRU_LOCAL_LIST_T_FREE); |
| 332 | 331 | if (++nfree == lru->target_free) |
| 333 | 332 | break; |
| @@ -335,10 +334,19 @@ static void bpf_lru_list_pop_free_to_local(struct bpf_lru *lru, · 1 unmodified line | ||
| 335 | 334 | |
| 336 | 335 | if (nfree < lru->target_free) |
| 337 | 336 | __bpf_lru_list_shrink(lru, l, lru->target_free - nfree, |
| 338 | − local_free_list(loc_l), | |
| 337 | + &tmp_free, | |
| 339 | 338 | BPF_LRU_LOCAL_LIST_T_FREE); |
| 340 | 339 | |
| 341 | − raw_spin_unlock(&l->lock); | |
| 340 | + raw_res_spin_unlock(&l->lock); | |
| 341 | + | |
| 342 | + /* | |
| 343 | + * Transfer the harvested nodes from the temporary list_head into | |
| 344 | + * the lockless per-CPU free llist. | |
| 345 | + */ | |
| 346 | + list_for_each_entry_safe(node, tmp_node, &tmp_free, list) { | |
| 347 | + list_del(&node->list); | |
| 348 | + llist_add(&node->llist, &loc_l->free_llist); | |
| 349 | + } | |
| 342 | 350 | } |
| 343 | 351 | |
| 344 | 352 | static void __local_list_add_pending(struct bpf_lru *lru, |
| @@ -350,22 +358,21 @@ static void __local_list_add_pending(struct bpf_lru *lru, · 5 unmodified lines | ||
| 350 | 358 | *(u32 *)((void *)node + lru->hash_offset) = hash; |
| 351 | 359 | node->cpu = cpu; |
| 352 | 360 | node->type = BPF_LRU_LOCAL_LIST_T_PENDING; |
| 361 | + WRITE_ONCE(node->pending_free, 0); | |
| 353 | 362 | bpf_lru_node_clear_ref(node); |
| 354 | − list_add(&node->list, local_pending_list(loc_l)); | |
| 363 | + list_add(&node->list, &loc_l->pending_list); | |
| 355 | 364 | } |
| 356 | 365 | |
| 357 | 366 | static struct bpf_lru_node * |
| 358 | 367 | __local_list_pop_free(struct bpf_lru_locallist *loc_l) |
| 359 | 368 | { |
| 360 | − struct bpf_lru_node *node; | |
| 369 | + struct llist_node *llnode; | |
| 361 | 370 | |
| 362 | − node = list_first_entry_or_null(local_free_list(loc_l), | |
| 363 | − struct bpf_lru_node, | |
| 364 | − list); | |
| 365 | − if (node) | |
| 366 | − list_del(&node->list); | |
| 371 | + llnode = llist_del_first(&loc_l->free_llist); | |
| 372 | + if (!llnode) | |
| 373 | + return NULL; | |
| 367 | 374 | |
| 368 | − return node; | |
| 375 | + return container_of(llnode, struct bpf_lru_node, llist); | |
| 369 | 376 | } |
| 370 | 377 | |
| 371 | 378 | static struct bpf_lru_node * |
| @@ -376,10 +383,10 @@ __local_list_pop_pending(struct bpf_lru *lru, struct bpf_lru_locallist *loc_l) · 4 unmodified lines | ||
| 376 | 383 | |
| 377 | 384 | ignore_ref: |
| 378 | 385 | /* Get from the tail (i.e. older element) of the pending list. */ |
| 379 | − list_for_each_entry_reverse(node, local_pending_list(loc_l), | |
| 380 | − list) { | |
| 386 | + list_for_each_entry_reverse(node, &loc_l->pending_list, list) { | |
| 381 | 387 | if ((!bpf_lru_node_is_ref(node) || force) && |
| 382 | − lru->del_from_htab(lru->del_arg, node)) { | |
| 388 | + (READ_ONCE(node->pending_free) || | |
| 389 | + lru->del_from_htab(lru->del_arg, node))) { | |
| 383 | 390 | list_del(&node->list); |
| 384 | 391 | return node; |
| 385 | 392 | } |
| @@ -404,7 +411,8 @@ static struct bpf_lru_node *bpf_percpu_lru_pop_free(struct bpf_lru *lru, · 18 unmodified lines | ||
| 404 | 411 | |
| 405 | 412 | l = per_cpu_ptr(lru->percpu_lru, cpu); |
| 406 | 413 | |
| 407 | − raw_spin_lock_irqsave(&l->lock, flags); | |
| 414 | + if (raw_res_spin_lock_irqsave(&l->lock, flags)) | |
| 415 | + return NULL; | |
| 408 | 416 | |
| 409 | 417 | __bpf_lru_list_rotate(lru, l); |
| 410 | 418 | |
| @@ -420,7 +428,7 @@ static struct bpf_lru_node *bpf_percpu_lru_pop_free(struct bpf_lru *lru, · 9 unmodified lines | ||
| 420 | 428 | __bpf_lru_node_move(l, node, BPF_LRU_LIST_T_INACTIVE); |
| 421 | 429 | } |
| 422 | 430 | |
| 423 | − raw_spin_unlock_irqrestore(&l->lock, flags); | |
| 431 | + raw_res_spin_unlock_irqrestore(&l->lock, flags); | |
| 424 | 432 | |
| 425 | 433 | return node; |
| 426 | 434 | } |
| @@ -437,7 +445,8 @@ static struct bpf_lru_node *bpf_common_lru_pop_free(struct bpf_lru *lru, · 10 unmodified lines | ||
| 437 | 445 | |
| 438 | 446 | loc_l = per_cpu_ptr(clru->local_list, cpu); |
| 439 | 447 | |
| 440 | − raw_spin_lock_irqsave(&loc_l->lock, flags); | |
| 448 | + if (raw_res_spin_lock_irqsave(&loc_l->lock, flags)) | |
| 449 | + return NULL; | |
| 441 | 450 | |
| 442 | 451 | node = __local_list_pop_free(loc_l); |
| 443 | 452 | if (!node) { |
| @@ -448,17 +457,22 @@ static struct bpf_lru_node *bpf_common_lru_pop_free(struct bpf_lru *lru, · 4 unmodified lines | ||
| 448 | 457 | if (node) |
| 449 | 458 | __local_list_add_pending(lru, loc_l, cpu, node, hash); |
| 450 | 459 | |
| 451 | − raw_spin_unlock_irqrestore(&loc_l->lock, flags); | |
| 460 | + raw_res_spin_unlock_irqrestore(&loc_l->lock, flags); | |
| 452 | 461 | |
| 453 | 462 | if (node) |
| 454 | 463 | return node; |
| 455 | 464 | |
| 456 | − /* No free nodes found from the local free list and | |
| 465 | + /* | |
| 466 | + * No free nodes found from the local free list and | |
| 457 | 467 | * the global LRU list. |
| 458 | 468 | * |
| 459 | 469 | * Steal from the local free/pending list of the |
| 460 | 470 | * current CPU and remote CPU in RR. It starts |
| 461 | 471 | * with the loc_l->next_steal CPU. |
| 472 | + * | |
| 473 | + * Acquire the victim's lock before touching either list. On | |
| 474 | + * acquisition failure (rqspinlock AA or timeout) skip the victim | |
| 475 | + * and try the next CPU. | |
| 462 | 476 | */ |
| 463 | 477 | |
| 464 | 478 | first_steal = loc_l->next_steal; |
| @@ -466,24 +480,36 @@ static struct bpf_lru_node *bpf_common_lru_pop_free(struct bpf_lru *lru, · 1 unmodified line | ||
| 466 | 480 | do { |
| 467 | 481 | steal_loc_l = per_cpu_ptr(clru->local_list, steal); |
| 468 | 482 | |
| 469 | − raw_spin_lock_irqsave(&steal_loc_l->lock, flags); | |
| 470 | − | |
| 471 | − node = __local_list_pop_free(steal_loc_l); | |
| 472 | − if (!node) | |
| 473 | − node = __local_list_pop_pending(lru, steal_loc_l); | |
| 474 | − | |
| 475 | − raw_spin_unlock_irqrestore(&steal_loc_l->lock, flags); | |
| 483 | + if (!raw_res_spin_lock_irqsave(&steal_loc_l->lock, flags)) { | |
| 484 | + node = __local_list_pop_free(steal_loc_l); | |
| 485 | + if (!node) | |
| 486 | + node = __local_list_pop_pending(lru, steal_loc_l); | |
| 487 | + raw_res_spin_unlock_irqrestore(&steal_loc_l->lock, flags); | |
| 488 | + } | |
| 476 | 489 | |
| 477 | 490 | steal = cpumask_next_wrap(steal, cpu_possible_mask); |
| 478 | 491 | } while (!node && steal != first_steal); |
| 479 | 492 | |
| 480 | 493 | loc_l->next_steal = steal; |
| 481 | 494 | |
| 482 | − if (node) { | |
| 483 | − raw_spin_lock_irqsave(&loc_l->lock, flags); | |
| 484 | − __local_list_add_pending(lru, loc_l, cpu, node, hash); | |
| 485 | − raw_spin_unlock_irqrestore(&loc_l->lock, flags); | |
| 495 | + if (!node) | |
| 496 | + return NULL; | |
| 497 | + | |
| 498 | + if (raw_res_spin_lock_irqsave(&loc_l->lock, flags)) { | |
| 499 | + /* | |
| 500 | + * The local pending lock can't be acquired (rqspinlock AA | |
| 501 | + * or timeout). Return the stolen node to the per-CPU | |
| 502 | + * free_llist instead of orphaning it; the next pop_free on | |
| 503 | + * this CPU will pick it up. | |
| 504 | + */ | |
| 505 | + node->type = BPF_LRU_LOCAL_LIST_T_FREE; | |
| 506 | + bpf_lru_node_clear_ref(node); | |
| 507 | + WRITE_ONCE(node->pending_free, 0); | |
| 508 | + llist_add(&node->llist, &loc_l->free_llist); | |
| 509 | + return NULL; | |
| 486 | 510 | } |
| 511 | + __local_list_add_pending(lru, loc_l, cpu, node, hash); | |
| 512 | + raw_res_spin_unlock_irqrestore(&loc_l->lock, flags); | |
| 487 | 513 | |
| 488 | 514 | return node; |
| 489 | 515 | } |
| @@ -511,18 +537,24 @@ static void bpf_common_lru_push_free(struct bpf_lru *lru, · 21 unmodified lines | ||
| 511 | 537 | |
| 512 | 538 | loc_l = per_cpu_ptr(lru->common_lru.local_list, node->cpu); |
| 513 | 539 | |
| 514 | − raw_spin_lock_irqsave(&loc_l->lock, flags); | |
| 540 | + if (raw_res_spin_lock_irqsave(&loc_l->lock, flags)) { | |
| 541 | + WRITE_ONCE(node->pending_free, 1); | |
| 542 | + return; | |
| 543 | + } | |
| 515 | 544 | |
| 516 | 545 | if (unlikely(node->type != BPF_LRU_LOCAL_LIST_T_PENDING)) { |
| 517 | − raw_spin_unlock_irqrestore(&loc_l->lock, flags); | |
| 546 | + raw_res_spin_unlock_irqrestore(&loc_l->lock, | |
| 547 | + flags); | |
| 518 | 548 | goto check_lru_list; |
| 519 | 549 | } |
| 520 | 550 | |
| 521 | 551 | node->type = BPF_LRU_LOCAL_LIST_T_FREE; |
| 522 | 552 | bpf_lru_node_clear_ref(node); |
| 523 | − list_move(&node->list, local_free_list(loc_l)); | |
| 553 | + list_del(&node->list); | |
| 554 | + | |
| 555 | + raw_res_spin_unlock_irqrestore(&loc_l->lock, flags); | |
| 524 | 556 | |
| 525 | − raw_spin_unlock_irqrestore(&loc_l->lock, flags); | |
| 557 | + llist_add(&node->llist, &loc_l->free_llist); | |
| 526 | 558 | return; |
| 527 | 559 | } |
| 528 | 560 | |
| @@ -538,11 +570,14 @@ static void bpf_percpu_lru_push_free(struct bpf_lru *lru, · 9 unmodified lines | ||
| 538 | 570 | |
| 539 | 571 | l = per_cpu_ptr(lru->percpu_lru, node->cpu); |
| 540 | 572 | |
| 541 | − raw_spin_lock_irqsave(&l->lock, flags); | |
| 573 | + if (raw_res_spin_lock_irqsave(&l->lock, flags)) { | |
| 574 | + WRITE_ONCE(node->pending_free, 1); | |
| 575 | + return; | |
| 576 | + } | |
| 542 | 577 | |
| 543 | 578 | __bpf_lru_node_move(l, node, BPF_LRU_LIST_T_FREE); |
| 544 | 579 | |
| 545 | − raw_spin_unlock_irqrestore(&l->lock, flags); | |
| 580 | + raw_res_spin_unlock_irqrestore(&l->lock, flags); | |
| 546 | 581 | } |
| 547 | 582 | |
| 548 | 583 | void bpf_lru_push_free(struct bpf_lru *lru, struct bpf_lru_node *node) |
| @@ -565,6 +600,7 @@ static void bpf_common_lru_populate(struct bpf_lru *lru, void *buf, · 16 unmodified lines | ||
| 565 | 600 | |
| 566 | 601 | node = (struct bpf_lru_node *)(buf + node_offset); |
| 567 | 602 | node->type = BPF_LRU_LIST_T_FREE; |
| 603 | + node->pending_free = 0; | |
| 568 | 604 | bpf_lru_node_clear_ref(node); |
| 569 | 605 | list_add(&node->list, &l->lists[BPF_LRU_LIST_T_FREE]); |
| 570 | 606 | buf += elem_size; |
| @@ -594,6 +630,7 @@ again: · 23 unmodified lines | ||
| 594 | 630 | node = (struct bpf_lru_node *)(buf + node_offset); |
| 595 | 631 | node->cpu = cpu; |
| 596 | 632 | node->type = BPF_LRU_LIST_T_FREE; |
| 633 | + node->pending_free = 0; | |
| 597 | 634 | bpf_lru_node_clear_ref(node); |
| 598 | 635 | list_add(&node->list, &l->lists[BPF_LRU_LIST_T_FREE]); |
| 599 | 636 | i++; |
| @@ -618,14 +655,12 @@ void bpf_lru_populate(struct bpf_lru *lru, void *buf, u32 node_offset, · 18 unmodified lines | ||
| 618 | 655 | |
| 619 | 656 | static void bpf_lru_locallist_init(struct bpf_lru_locallist *loc_l, int cpu) |
| 620 | 657 | { |
| 621 | − int i; | |
| 622 | − | |
| 623 | − for (i = 0; i < NR_BPF_LRU_LOCAL_LIST_T; i++) | |
| 624 | − INIT_LIST_HEAD(&loc_l->lists[i]); | |
| 658 | + INIT_LIST_HEAD(&loc_l->pending_list); | |
| 659 | + init_llist_head(&loc_l->free_llist); | |
| 625 | 660 | |
| 626 | 661 | loc_l->next_steal = cpu; |
| 627 | 662 | |
| 628 | − raw_spin_lock_init(&loc_l->lock); | |
| 663 | + raw_res_spin_lock_init(&loc_l->lock); | |
| 629 | 664 | } |
| 630 | 665 | |
| 631 | 666 | static void bpf_lru_list_init(struct bpf_lru_list *l) |
| @@ -640,7 +675,7 @@ static void bpf_lru_list_init(struct bpf_lru_list *l) · 8 unmodified lines | ||
| 640 | 675 | |
| 641 | 676 | l->next_inactive_rotation = &l->lists[BPF_LRU_LIST_T_INACTIVE]; |
| 642 | 677 | |
| 643 | − raw_spin_lock_init(&l->lock); | |
| 678 | + raw_res_spin_lock_init(&l->lock); | |
| 644 | 679 | } |
| 645 | 680 | |
| 646 | 681 | int bpf_lru_init(struct bpf_lru *lru, bool percpu, u32 hash_offset, |
| @@ -6,11 +6,11 @@ · 5 unmodified lines | ||
| 6 | 6 | |
| 7 | 7 | #include <linux/cache.h> |
| 8 | 8 | #include <linux/list.h> |
| 9 | −#include <linux/spinlock_types.h> | |
| 9 | +#include <linux/llist.h> | |
| 10 | +#include <asm/rqspinlock.h> | |
| 10 | 11 | |
| 11 | 12 | #define NR_BPF_LRU_LIST_T (3) |
| 12 | 13 | #define NR_BPF_LRU_LIST_COUNT (2) |
| 13 | −#define NR_BPF_LRU_LOCAL_LIST_T (2) | |
| 14 | 14 | #define BPF_LOCAL_LIST_T_OFFSET NR_BPF_LRU_LIST_T |
| 15 | 15 | |
| 16 | 16 | enum bpf_lru_list_type { |
| @@ -22,10 +22,22 @@ enum bpf_lru_list_type { · 5 unmodified lines | ||
| 22 | 22 | }; |
| 23 | 23 | |
| 24 | 24 | struct bpf_lru_node { |
| 25 | − struct list_head list; | |
| 25 | + /* | |
| 26 | + * A node is in at most one list at a time. The free path on the | |
| 27 | + * per-CPU locallist uses an llist, so share storage via a union. | |
| 28 | + */ | |
| 29 | + union { | |
| 30 | + struct list_head list; | |
| 31 | + struct llist_node llist; | |
| 32 | + }; | |
| 26 | 33 | u16 cpu; |
| 27 | 34 | u8 type; |
| 28 | 35 | u8 ref; |
| 36 | + /* | |
| 37 | + * Marks nodes whose *_push_free() lock acquire failed; reclaimed | |
| 38 | + * by flush/shrink which honor the flag instead of del_from_htab(). | |
| 39 | + */ | |
| 40 | + u8 pending_free; | |
| 29 | 41 | }; |
| 30 | 42 | |
| 31 | 43 | struct bpf_lru_list { |
| @@ -34,13 +46,14 @@ struct bpf_lru_list { · 2 unmodified lines | ||
| 34 | 46 | /* The next inactive list rotation starts from here */ |
| 35 | 47 | struct list_head *next_inactive_rotation; |
| 36 | 48 | |
| 37 | − raw_spinlock_t lock ____cacheline_aligned_in_smp; | |
| 49 | + rqspinlock_t lock ____cacheline_aligned_in_smp; | |
| 38 | 50 | }; |
| 39 | 51 | |
| 40 | 52 | struct bpf_lru_locallist { |
| 41 | − struct list_head lists[NR_BPF_LRU_LOCAL_LIST_T]; | |
| 53 | + struct list_head pending_list; | |
| 54 | + struct llist_head free_llist; | |
| 42 | 55 | u16 next_steal; |
| 43 | − raw_spinlock_t lock; | |
| 56 | + rqspinlock_t lock; | |
| 44 | 57 | }; |
| 45 | 58 | |
| 46 | 59 | struct bpf_common_lru { |
| @@ -427,6 +427,26 @@ BTF_ID(func, bpf_lsm_audit_rule_known) · 426 unmodified lines | ||
| 427 | 427 | BTF_ID(func, bpf_lsm_inode_xattr_skipcap) |
| 428 | 428 | BTF_SET_END(bool_lsm_hooks) |
| 429 | 429 | |
| 430 | +/* hooks returning void */ | |
| 431 | +#define LSM_HOOK_void(DEFAULT, NAME, ...) BTF_ID(func, bpf_lsm_##NAME) | |
| 432 | +#define LSM_HOOK_int(DEFAULT, NAME, ...) /* nothing */ | |
| 433 | +#define LSM_HOOK(RET, DEFAULT, NAME, ...) LSM_HOOK_##RET(DEFAULT, NAME, __VA_ARGS__) | |
| 434 | +BTF_SET_START(void_lsm_hooks) | |
| 435 | +#include <linux/lsm_hook_defs.h> | |
| 436 | +#undef LSM_HOOK | |
| 437 | +#undef LSM_HOOK_void | |
| 438 | +#undef LSM_HOOK_int | |
| 439 | +BTF_SET_END(void_lsm_hooks) | |
| 440 | + | |
| 441 | +bool bpf_lsm_hook_returns_errno(u32 btf_id) | |
| 442 | +{ | |
| 443 | + if (btf_id_set_contains(&bool_lsm_hooks, btf_id)) | |
| 444 | + return false; | |
| 445 | + if (btf_id_set_contains(&void_lsm_hooks, btf_id)) | |
| 446 | + return false; | |
| 447 | + return true; | |
| 448 | +} | |
| 449 | + | |
| 430 | 450 | int bpf_lsm_get_retval_range(const struct bpf_prog *prog, |
| 431 | 451 | struct bpf_retval_range *retval_range) |
| 432 | 452 | { |
| @@ -594,8 +594,8 @@ const struct bpf_link_ops bpf_struct_ops_link_lops = { · 593 unmodified lines | ||
| 594 | 594 | .dealloc = bpf_struct_ops_link_dealloc, |
| 595 | 595 | }; |
| 596 | 596 | |
| 597 | −int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_links *tlinks, | |
| 598 | − struct bpf_tramp_link *link, | |
| 597 | +int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_nodes *tnodes, | |
| 598 | + struct bpf_tramp_node *node, | |
| 599 | 599 | const struct btf_func_model *model, |
| 600 | 600 | void *stub_func, |
| 601 | 601 | void **_image, u32 *_image_off, |
| @@ -605,13 +605,13 @@ int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_links *tlinks, · 3 unmodified lines | ||
| 605 | 605 | void *image = *_image; |
| 606 | 606 | int size; |
| 607 | 607 | |
| 608 | − tlinks[BPF_TRAMP_FENTRY].links[0] = link; | |
| 609 | − tlinks[BPF_TRAMP_FENTRY].nr_links = 1; | |
| 608 | + tnodes[BPF_TRAMP_FENTRY].nodes[0] = node; | |
| 609 | + tnodes[BPF_TRAMP_FENTRY].nr_nodes = 1; | |
| 610 | 610 | |
| 611 | 611 | if (model->ret_size > 0) |
| 612 | 612 | flags |= BPF_TRAMP_F_RET_FENTRY_RET; |
| 613 | 613 | |
| 614 | − size = arch_bpf_trampoline_size(model, flags, tlinks, stub_func); | |
| 614 | + size = arch_bpf_trampoline_size(model, flags, tnodes, stub_func); | |
| 615 | 615 | if (size <= 0) |
| 616 | 616 | return size ? : -EFAULT; |
| 617 | 617 | |
| @@ -628,7 +628,7 @@ int bpf_struct_ops_prepare_trampoline(struct bpf_tramp_links *tlinks, · 10 unmodified lines | ||
| 628 | 628 | |
| 629 | 629 | size = arch_prepare_bpf_trampoline(NULL, image + image_off, |
| 630 | 630 | image + image_off + size, |
| 631 | − model, flags, tlinks, stub_func); | |
| 631 | + model, flags, tnodes, stub_func); | |
| 632 | 632 | if (size <= 0) { |
| 633 | 633 | if (image != *_image) |
| 634 | 634 | bpf_struct_ops_image_free(image); |
| @@ -693,7 +693,7 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, · 58 unmodified lines | ||
| 693 | 693 | const struct btf_type *module_type; |
| 694 | 694 | const struct btf_member *member; |
| 695 | 695 | const struct btf_type *t = st_ops_desc->type; |
| 696 | − struct bpf_tramp_links *tlinks; | |
| 696 | + struct bpf_tramp_nodes *tnodes; | |
| 697 | 697 | void *udata, *kdata; |
| 698 | 698 | int prog_fd, err; |
| 699 | 699 | u32 i, trampoline_start, image_off = 0; |
| @@ -720,8 +720,8 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, · 20 unmodified lines | ||
| 720 | 720 | if (uvalue->common.state || refcount_read(&uvalue->common.refcnt)) |
| 721 | 721 | return -EINVAL; |
| 722 | 722 | |
| 723 | − tlinks = kzalloc_objs(*tlinks, BPF_TRAMP_MAX); | |
| 724 | − if (!tlinks) | |
| 723 | + tnodes = kzalloc_objs(*tnodes, BPF_TRAMP_MAX); | |
| 724 | + if (!tnodes) | |
| 725 | 725 | return -ENOMEM; |
| 726 | 726 | |
| 727 | 727 | uvalue = (struct bpf_struct_ops_value *)st_map->uvalue; |
| @@ -817,8 +817,9 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, · 89 unmodified lines | ||
| 817 | 817 | err = -ENOMEM; |
| 818 | 818 | goto reset_unlock; |
| 819 | 819 | } |
| 820 | − bpf_link_init(&link->link, BPF_LINK_TYPE_STRUCT_OPS, | |
| 821 | − &bpf_struct_ops_link_lops, prog, prog->expected_attach_type); | |
| 820 | + bpf_tramp_link_init(link, BPF_LINK_TYPE_STRUCT_OPS, | |
| 821 | + &bpf_struct_ops_link_lops, prog, prog->expected_attach_type, 0); | |
| 822 | + | |
| 822 | 823 | *plink++ = &link->link; |
| 823 | 824 | |
| 824 | 825 | /* Poison pointer on error instead of return for backward compatibility */ |
| @@ -832,7 +833,7 @@ static long bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, · 7 unmodified lines | ||
| 832 | 833 | *pksym++ = ksym; |
| 833 | 834 | |
| 834 | 835 | trampoline_start = image_off; |
| 835 | − err = bpf_struct_ops_prepare_trampoline(tlinks, link, | |
| 836 | + err = bpf_struct_ops_prepare_trampoline(tnodes, &link->node, | |
| 836 | 837 | &st_ops->func_models[i], |
| 837 | 838 | *(void **)(st_ops->cfi_stubs + moff), |
| 838 | 839 | &image, &image_off, |
| @@ -911,7 +912,7 @@ reset_unlock: · 72 unmodified lines | ||
| 911 | 912 | memset(uvalue, 0, map->value_size); |
| 912 | 913 | memset(kvalue, 0, map->value_size); |
| 913 | 914 | unlock: |
| 914 | − kfree(tlinks); | |
| 915 | + kfree(tnodes); | |
| 915 | 916 | mutex_unlock(&st_map->lock); |
| 916 | 917 | if (!err) |
| 917 | 918 | bpf_struct_ops_map_add_ksyms(st_map); |
| @@ -1204,6 +1205,42 @@ u32 bpf_struct_ops_id(const void *kdata) · 286 unmodified lines | ||
| 1204 | 1205 | } |
| 1205 | 1206 | EXPORT_SYMBOL_GPL(bpf_struct_ops_id); |
| 1206 | 1207 | |
| 1208 | +/** | |
| 1209 | + * bpf_struct_ops_for_each_prog - Invoke @cb for each member prog | |
| 1210 | + * @kdata: kernel-side struct_ops vmtable (the @kdata arg to ->reg/->update/->unreg) | |
| 1211 | + * @cb: callback invoked once per member prog; non-zero return stops iteration | |
| 1212 | + * @data: opaque argument passed to @cb | |
| 1213 | + * | |
| 1214 | + * Walks the struct_ops member progs registered on the map containing @kdata. | |
| 1215 | + * Intended for use from struct_ops ->reg() callbacks (and similar) that need to | |
| 1216 | + * inspect the loaded BPF programs (for example to discover maps they reference | |
| 1217 | + * via @prog->aux->used_maps). | |
| 1218 | + * | |
| 1219 | + * Return 0 if iteration completed, otherwise the first non-zero @cb return. | |
| 1220 | + */ | |
| 1221 | +int bpf_struct_ops_for_each_prog(const void *kdata, | |
| 1222 | + int (*cb)(struct bpf_prog *prog, void *data), | |
| 1223 | + void *data) | |
| 1224 | +{ | |
| 1225 | + struct bpf_struct_ops_value *kvalue; | |
| 1226 | + struct bpf_struct_ops_map *st_map; | |
| 1227 | + u32 i; | |
| 1228 | + int ret; | |
| 1229 | + | |
| 1230 | + kvalue = container_of(kdata, struct bpf_struct_ops_value, data); | |
| 1231 | + st_map = container_of(kvalue, struct bpf_struct_ops_map, kvalue); | |
| 1232 | + | |
| 1233 | + for (i = 0; i < st_map->funcs_cnt; i++) { | |
| 1234 | + if (!st_map->links[i]) | |
| 1235 | + continue; | |
| 1236 | + ret = cb(st_map->links[i]->prog, data); | |
| 1237 | + if (ret) | |
| 1238 | + return ret; | |
| 1239 | + } | |
| 1240 | + return 0; | |
| 1241 | +} | |
| 1242 | +EXPORT_SYMBOL_GPL(bpf_struct_ops_for_each_prog); | |
| 1243 | + | |
| 1207 | 1244 | static bool bpf_struct_ops_valid_to_reg(struct bpf_map *map) |
| 1208 | 1245 | { |
| 1209 | 1246 | struct bpf_struct_ops_map *st_map = (struct bpf_struct_ops_map *)map; |
| @@ -182,7 +182,6 @@ · 181 unmodified lines | ||
| 182 | 182 | #define BITS_ROUNDUP_BYTES(bits) \ |
| 183 | 183 | (BITS_ROUNDDOWN_BYTES(bits) + !!BITS_PER_BYTE_MASKED(bits)) |
| 184 | 184 | |
| 185 | −#define BTF_INFO_MASK 0x9f00ffff | |
| 186 | 185 | #define BTF_INT_MASK 0x0fffffff |
| 187 | 186 | #define BTF_TYPE_ID_VALID(type_id) ((type_id) <= BTF_MAX_TYPE) |
| 188 | 187 | #define BTF_STR_OFFSET_VALID(name_off) ((name_off) <= BTF_MAX_NAME_OFFSET) |
| @@ -289,7 +288,7 @@ enum verifier_phase { · 100 unmodified lines | ||
| 289 | 288 | struct resolve_vertex { |
| 290 | 289 | const struct btf_type *t; |
| 291 | 290 | u32 type_id; |
| 292 | − u16 next_member; | |
| 291 | + u32 next_member; | |
| 293 | 292 | }; |
| 294 | 293 | |
| 295 | 294 | enum visit_state { |
| @@ -2031,7 +2030,7 @@ static int env_stack_push(struct btf_verifier_env *env, · 1735 unmodified lines | ||
| 2031 | 2030 | } |
| 2032 | 2031 | |
| 2033 | 2032 | static void env_stack_set_next_member(struct btf_verifier_env *env, |
| 2034 | − u16 next_member) | |
| 2033 | + u32 next_member) | |
| 2035 | 2034 | { |
| 2036 | 2035 | env->stack[env->top_stack - 1].next_member = next_member; |
| 2037 | 2036 | } |
| @@ -3293,7 +3292,7 @@ static s32 btf_struct_check_meta(struct btf_verifier_env *env, · 1255 unmodified lines | ||
| 3293 | 3292 | struct btf *btf = env->btf; |
| 3294 | 3293 | u32 struct_size = t->size; |
| 3295 | 3294 | u32 offset; |
| 3296 | − u16 i; | |
| 3295 | + u32 i; | |
| 3297 | 3296 | |
| 3298 | 3297 | meta_needed = btf_type_vlen(t) * sizeof(*member); |
| 3299 | 3298 | if (meta_left < meta_needed) { |
| @@ -3369,7 +3368,7 @@ static int btf_struct_resolve(struct btf_verifier_env *env, · 69 unmodified lines | ||
| 3369 | 3368 | { |
| 3370 | 3369 | const struct btf_member *member; |
| 3371 | 3370 | int err; |
| 3372 | − u16 i; | |
| 3371 | + u32 i; | |
| 3373 | 3372 | |
| 3374 | 3373 | /* Before continue resolving the next_member, |
| 3375 | 3374 | * ensure the last member is indeed resolved to a |
| @@ -3668,7 +3667,7 @@ end: · 292 unmodified lines | ||
| 3668 | 3667 | static int btf_repeat_fields(struct btf_field_info *info, int info_cnt, |
| 3669 | 3668 | u32 field_cnt, u32 repeat_cnt, u32 elem_size) |
| 3670 | 3669 | { |
| 3671 | − u32 i, j; | |
| 3670 | + u32 i, j, total_cnt, total_repeats; | |
| 3672 | 3671 | u32 cur; |
| 3673 | 3672 | |
| 3674 | 3673 | /* Ensure not repeating fields that should not be repeated. */ |
| @@ -3686,10 +3685,9 @@ static int btf_repeat_fields(struct btf_field_info *info, int info_cnt, · 11 unmodified lines | ||
| 3686 | 3685 | } |
| 3687 | 3686 | } |
| 3688 | 3687 | |
| 3689 | − /* The type of struct size or variable size is u32, | |
| 3690 | − * so the multiplication will not overflow. | |
| 3691 | − */ | |
| 3692 | − if (field_cnt * (repeat_cnt + 1) > info_cnt) | |
| 3688 | + if (check_add_overflow(repeat_cnt, 1, &total_repeats) || | |
| 3689 | + check_mul_overflow(field_cnt, total_repeats, &total_cnt) || | |
| 3690 | + total_cnt > (u32)info_cnt) | |
| 3693 | 3691 | return -E2BIG; |
| 3694 | 3692 | |
| 3695 | 3693 | cur = field_cnt; |
| @@ -4447,7 +4445,7 @@ static s32 btf_enum_check_meta(struct btf_verifier_env *env, · 751 unmodified lines | ||
| 4447 | 4445 | const struct btf_enum *enums = btf_type_enum(t); |
| 4448 | 4446 | struct btf *btf = env->btf; |
| 4449 | 4447 | const char *fmt_str; |
| 4450 | − u16 i, nr_enums; | |
| 4448 | + u32 i, nr_enums; | |
| 4451 | 4449 | u32 meta_needed; |
| 4452 | 4450 | |
| 4453 | 4451 | nr_enums = btf_type_vlen(t); |
| @@ -4555,7 +4553,7 @@ static s32 btf_enum64_check_meta(struct btf_verifier_env *env, · 101 unmodified lines | ||
| 4555 | 4553 | const struct btf_enum64 *enums = btf_type_enum64(t); |
| 4556 | 4554 | struct btf *btf = env->btf; |
| 4557 | 4555 | const char *fmt_str; |
| 4558 | − u16 i, nr_enums; | |
| 4556 | + u32 i, nr_enums; | |
| 4559 | 4557 | u32 meta_needed; |
| 4560 | 4558 | |
| 4561 | 4559 | nr_enums = btf_type_vlen(t); |
| @@ -4683,7 +4681,7 @@ static void btf_func_proto_log(struct btf_verifier_env *env, · 121 unmodified lines | ||
| 4683 | 4681 | const struct btf_type *t) |
| 4684 | 4682 | { |
| 4685 | 4683 | const struct btf_param *args = (const struct btf_param *)(t + 1); |
| 4686 | − u16 nr_args = btf_type_vlen(t), i; | |
| 4684 | + u32 nr_args = btf_type_vlen(t), i; | |
| 4687 | 4685 | |
| 4688 | 4686 | btf_verifier_log(env, "return=%u args=(", t->type); |
| 4689 | 4687 | if (!nr_args) { |
| @@ -4929,7 +4927,7 @@ static int btf_datasec_resolve(struct btf_verifier_env *env, · 239 unmodified lines | ||
| 4929 | 4927 | { |
| 4930 | 4928 | const struct btf_var_secinfo *vsi; |
| 4931 | 4929 | struct btf *btf = env->btf; |
| 4932 | − u16 i; | |
| 4930 | + u32 i; | |
| 4933 | 4931 | |
| 4934 | 4932 | env->resolve_mode = RESOLVE_TBD; |
| 4935 | 4933 | for_each_vsi_from(i, v->next_member, v->t, vsi) { |
| @@ -5183,7 +5181,7 @@ static int btf_func_proto_check(struct btf_verifier_env *env, · 247 unmodified lines | ||
| 5183 | 5181 | const struct btf_type *ret_type; |
| 5184 | 5182 | const struct btf_param *args; |
| 5185 | 5183 | const struct btf *btf; |
| 5186 | − u16 nr_args, i; | |
| 5184 | + u32 nr_args, i; | |
| 5187 | 5185 | int err; |
| 5188 | 5186 | |
| 5189 | 5187 | btf = env->btf; |
| @@ -5278,7 +5276,7 @@ static int btf_func_check(struct btf_verifier_env *env, · 88 unmodified lines | ||
| 5278 | 5276 | const struct btf_type *proto_type; |
| 5279 | 5277 | const struct btf_param *args; |
| 5280 | 5278 | const struct btf *btf; |
| 5281 | − u16 nr_args, i; | |
| 5279 | + u32 nr_args, i; | |
| 5282 | 5280 | |
| 5283 | 5281 | btf = env->btf; |
| 5284 | 5282 | proto_type = btf_type_by_id(btf, t->type); |
| @@ -5336,12 +5334,6 @@ static s32 btf_check_meta(struct btf_verifier_env *env, · 51 unmodified lines | ||
| 5336 | 5334 | } |
| 5337 | 5335 | meta_left -= sizeof(*t); |
| 5338 | 5336 | |
| 5339 | − if (t->info & ~BTF_INFO_MASK) { | |
| 5340 | − btf_verifier_log(env, "[%u] Invalid btf_info:%x", | |
| 5341 | − env->log_type_id, t->info); | |
| 5342 | − return -EINVAL; | |
| 5343 | − } | |
| 5344 | − | |
| 5345 | 5337 | if (BTF_INFO_KIND(t->info) > BTF_KIND_MAX || |
| 5346 | 5338 | BTF_INFO_KIND(t->info) == BTF_KIND_UNKN) { |
| 5347 | 5339 | btf_verifier_log(env, "[%u] Invalid kind:%u", |
| @@ -5914,25 +5906,10 @@ static int btf_check_type_tags(struct btf_verifier_env *env, · 566 unmodified lines | ||
| 5914 | 5906 | return 0; |
| 5915 | 5907 | } |
| 5916 | 5908 | |
| 5917 | −static int finalize_log(struct bpf_verifier_log *log, bpfptr_t uattr, u32 uattr_size) | |
| 5918 | −{ | |
| 5919 | − u32 log_true_size; | |
| 5920 | − int err; | |
| 5921 | − | |
| 5922 | − err = bpf_vlog_finalize(log, &log_true_size); | |
| 5923 | − | |
| 5924 | − if (uattr_size >= offsetofend(union bpf_attr, btf_log_true_size) && | |
| 5925 | − copy_to_bpfptr_offset(uattr, offsetof(union bpf_attr, btf_log_true_size), | |
| 5926 | − &log_true_size, sizeof(log_true_size))) | |
| 5927 | − err = -EFAULT; | |
| 5928 | − | |
| 5929 | − return err; | |
| 5930 | −} | |
| 5931 | − | |
| 5932 | −static struct btf *btf_parse(const union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size) | |
| 5909 | +static struct btf *btf_parse(const union bpf_attr *attr, bpfptr_t uattr, | |
| 5910 | + struct bpf_log_attr *attr_log) | |
| 5933 | 5911 | { |
| 5934 | 5912 | bpfptr_t btf_data = make_bpfptr(attr->btf, uattr.is_kernel); |
| 5935 | − char __user *log_ubuf = u64_to_user_ptr(attr->btf_log_buf); | |
| 5936 | 5913 | struct btf_struct_metas *struct_meta_tab; |
| 5937 | 5914 | struct btf_verifier_env *env = NULL; |
| 5938 | 5915 | struct btf *btf = NULL; |
| @@ -5949,8 +5926,7 @@ static struct btf *btf_parse(const union bpf_attr *attr, bpfptr_t uattr, u32 uat · 10 unmodified lines | ||
| 5949 | 5926 | /* user could have requested verbose verifier output |
| 5950 | 5927 | * and supplied buffer to store the verification trace |
| 5951 | 5928 | */ |
| 5952 | − err = bpf_vlog_init(&env->log, attr->btf_log_level, | |
| 5953 | − log_ubuf, attr->btf_log_size); | |
| 5929 | + err = bpf_vlog_init(&env->log, attr_log->level, attr_log->ubuf, attr_log->size); | |
| 5954 | 5930 | if (err) |
| 5955 | 5931 | goto errout_free; |
| 5956 | 5932 | |
| @@ -6015,7 +5991,7 @@ static struct btf *btf_parse(const union bpf_attr *attr, bpfptr_t uattr, u32 uat · 58 unmodified lines | ||
| 6015 | 5991 | } |
| 6016 | 5992 | } |
| 6017 | 5993 | |
| 6018 | − err = finalize_log(&env->log, uattr, uattr_size); | |
| 5994 | + err = bpf_log_attr_finalize(attr_log, &env->log); | |
| 6019 | 5995 | if (err) |
| 6020 | 5996 | goto errout_free; |
| 6021 | 5997 | |
| @@ -6027,7 +6003,7 @@ errout_meta: · 5 unmodified lines | ||
| 6027 | 6003 | btf_free_struct_meta_tab(btf); |
| 6028 | 6004 | errout: |
| 6029 | 6005 | /* overwrite err with -ENOSPC or -EFAULT */ |
| 6030 | − ret = finalize_log(&env->log, uattr, uattr_size); | |
| 6006 | + ret = bpf_log_attr_finalize(attr_log, &env->log); | |
| 6031 | 6007 | if (ret) |
| 6032 | 6008 | err = ret; |
| 6033 | 6009 | errout_free: |
| @@ -6980,7 +6956,7 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type, · 946 unmodified lines | ||
| 6980 | 6956 | info->reg_type = ctx_arg_info->reg_type; |
| 6981 | 6957 | info->btf = ctx_arg_info->btf ? : btf_vmlinux; |
| 6982 | 6958 | info->btf_id = ctx_arg_info->btf_id; |
| 6983 | − info->ref_obj_id = ctx_arg_info->ref_obj_id; | |
| 6959 | + info->ref_id = ctx_arg_info->ref_id; | |
| 6984 | 6960 | return true; |
| 6985 | 6961 | } |
| 6986 | 6962 | } |
| @@ -7825,6 +7801,134 @@ enum btf_arg_tag { · 838 unmodified lines | ||
| 7825 | 7801 | ARG_TAG_ARENA = BIT_ULL(5), |
| 7826 | 7802 | }; |
| 7827 | 7803 | |
| 7804 | +static int btf_scan_decl_tags(struct bpf_verifier_env *env, | |
| 7805 | + const struct btf *btf, | |
| 7806 | + const struct btf_type *fn_t, | |
| 7807 | + u32 arg_idx, bool is_global, u32 *tags) | |
| 7808 | +{ | |
| 7809 | + int id = btf_named_start_id(btf, false) - 1; | |
| 7810 | + const char tag_key[] = "arg:"; | |
| 7811 | + static const struct { | |
| 7812 | + const char *tag_value; | |
| 7813 | + enum btf_arg_tag arg_tag; | |
| 7814 | + } tag_values[] = { | |
| 7815 | + { "ctx", ARG_TAG_CTX }, | |
| 7816 | + { "trusted", ARG_TAG_TRUSTED }, | |
| 7817 | + { "untrusted", ARG_TAG_UNTRUSTED }, | |
| 7818 | + { "nonnull", ARG_TAG_NONNULL }, | |
| 7819 | + { "nullable", ARG_TAG_NULLABLE }, | |
| 7820 | + { "arena", ARG_TAG_ARENA }, | |
| 7821 | + }; | |
| 7822 | + | |
| 7823 | + /* | |
| 7824 | + * The 'arg:<tag>' decl_tag takes precedence over the derivation | |
| 7825 | + * of the register type from the BTF type itself. | |
| 7826 | + */ | |
| 7827 | + while ((id = btf_find_next_decl_tag(btf, fn_t, arg_idx, tag_key, id)) > 0) { | |
| 7828 | + const struct btf_type *tag_t; | |
| 7829 | + const char *tag; | |
| 7830 | + int i; | |
| 7831 | + bool found; | |
| 7832 | + | |
| 7833 | + /* disallow arg tags in static subprogs */ | |
| 7834 | + if (!is_global) { | |
| 7835 | + bpf_log(&env->log, | |
| 7836 | + "arg#%d type tag is not supported in static functions\n", | |
| 7837 | + arg_idx); | |
| 7838 | + return -EOPNOTSUPP; | |
| 7839 | + } | |
| 7840 | + | |
| 7841 | + tag_t = btf_type_by_id(btf, id); | |
| 7842 | + tag = __btf_name_by_offset(btf, tag_t->name_off) + (sizeof(tag_key) - 1); | |
| 7843 | + | |
| 7844 | + found = false; | |
| 7845 | + for (i = 0; i < ARRAY_SIZE(tag_values); ++i) { | |
| 7846 | + if (!strcmp(tag, tag_values[i].tag_value)) { | |
| 7847 | + *tags |= tag_values[i].arg_tag; | |
| 7848 | + found = true; | |
| 7849 | + break; | |
| 7850 | + } | |
| 7851 | + } | |
| 7852 | + | |
| 7853 | + if (!found) { | |
| 7854 | + bpf_log(&env->log, "arg#%d has unsupported set of tags\n", arg_idx); | |
| 7855 | + return -EOPNOTSUPP; | |
| 7856 | + } | |
| 7857 | + } | |
| 7858 | + if (id != -ENOENT) { | |
| 7859 | + bpf_log(&env->log, "arg#%d type tag fetching failure: %d\n", arg_idx, id); | |
| 7860 | + return id; | |
| 7861 | + } | |
| 7862 | + | |
| 7863 | + return 0; | |
| 7864 | +} | |
| 7865 | + | |
| 7866 | +static int btf_scan_type_tags(struct bpf_verifier_env *env, | |
| 7867 | + const struct btf *btf, u32 type_id, | |
| 7868 | + u32 *tags) | |
| 7869 | +{ | |
| 7870 | + const struct btf_type *t; | |
| 7871 | + | |
| 7872 | + /* Find the first pointer type in the chain. */ | |
| 7873 | + t = btf_type_skip_modifiers(btf, type_id, NULL); | |
| 7874 | + | |
| 7875 | + /* | |
| 7876 | + * We currently reject type tags on non-pointer types, | |
| 7877 | + * which neither LLVM nor GCC support anyway. | |
| 7878 | + */ | |
| 7879 | + if (!t || !btf_type_is_ptr(t)) | |
| 7880 | + return 0; | |
| 7881 | + | |
| 7882 | + /* We got a pointer, get all associated type tags. */ | |
| 7883 | + for (t = btf_type_by_id(btf, t->type); t && btf_type_is_modifier(t); | |
| 7884 | + t = btf_type_by_id(btf, t->type)) { | |
| 7885 | + | |
| 7886 | + /* Skip non-type tag modifiers. */ | |
| 7887 | + if (!btf_type_is_type_tag(t)) | |
| 7888 | + continue; | |
| 7889 | + | |
| 7890 | + const char *tag = __btf_name_by_offset(btf, t->name_off); | |
| 7891 | + | |
| 7892 | + if (strcmp(tag, "arena") == 0) { | |
| 7893 | + *tags |= ARG_TAG_ARENA; | |
| 7894 | + } else { | |
| 7895 | + bpf_log(&env->log, "function signature member has unsupported type tag '%s'\n", | |
| 7896 | + tag); | |
| 7897 | + return -EOPNOTSUPP; | |
| 7898 | + } | |
| 7899 | + } | |
| 7900 | + | |
| 7901 | + return 0; | |
| 7902 | +} | |
| 7903 | + | |
| 7904 | +/* Check whether the type is a valid return type. */ | |
| 7905 | +static int btf_validate_return_type(struct bpf_verifier_env *env, struct btf *btf, | |
| 7906 | + const struct btf_type *t, int subprog) | |
| 7907 | +{ | |
| 7908 | + u32 tags = 0; | |
| 7909 | + int err; | |
| 7910 | + | |
| 7911 | + err = btf_scan_type_tags(env, btf, t->type, &tags); | |
| 7912 | + if (err) | |
| 7913 | + return err; | |
| 7914 | + | |
| 7915 | + t = btf_type_skip_modifiers(btf, t->type, NULL); | |
| 7916 | + | |
| 7917 | + /* | |
| 7918 | + * We allow all subprogs except for the main one to return any kind of arena pointer. | |
| 7919 | + * General arena variables are not allowed, since it makes no sense to return by value | |
| 7920 | + * a variable that's on the heap in the first place. | |
| 7921 | + */ | |
| 7922 | + if (subprog && (tags & ARG_TAG_ARENA) && btf_type_is_ptr(t)) | |
| 7923 | + return 0; | |
| 7924 | + | |
| 7925 | + /* We always accept void or scalars. */ | |
| 7926 | + if (btf_type_is_void(t) || btf_type_is_int(t) || btf_is_any_enum(t)) | |
| 7927 | + return 0; | |
| 7928 | + | |
| 7929 | + return -EOPNOTSUPP; | |
| 7930 | +} | |
| 7931 | + | |
| 7828 | 7932 | /* Process BTF of a function to produce high-level expectation of function |
| 7829 | 7933 | * arguments (like ARG_PTR_TO_CTX, or ARG_PTR_TO_MEM, etc). This information |
| 7830 | 7934 | * is cached in subprog info for reuse. |
| @@ -7843,6 +7947,7 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog) · 12 unmodified lines | ||
| 7843 | 7947 | struct btf *btf = prog->aux->btf; |
| 7844 | 7948 | const struct btf_param *args; |
| 7845 | 7949 | const struct btf_type *t, *ref_t, *fn_t; |
| 7950 | + int err; | |
| 7846 | 7951 | u32 i, nargs, btf_id; |
| 7847 | 7952 | const char *tname; |
| 7848 | 7953 | |
| @@ -7887,25 +7992,36 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog) · 38 unmodified lines | ||
| 7887 | 7992 | } |
| 7888 | 7993 | args = (const struct btf_param *)(t + 1); |
| 7889 | 7994 | nargs = btf_type_vlen(t); |
| 7995 | + sub->arg_cnt = nargs; | |
| 7996 | + if (nargs > MAX_BPF_FUNC_ARGS) { | |
| 7997 | + bpf_log(log, "kernel supports at most %d parameters, function %s has %d\n", | |
| 7998 | + MAX_BPF_FUNC_ARGS, tname, nargs); | |
| 7999 | + return -EFAULT; | |
| 8000 | + } | |
| 7890 | 8001 | if (nargs > MAX_BPF_FUNC_REG_ARGS) { |
| 7891 | − if (!is_global) | |
| 7892 | − return -EINVAL; | |
| 7893 | − bpf_log(log, "Global function %s() with %d > %d args. Buggy compiler.\n", | |
| 8002 | + if (!bpf_jit_supports_stack_args()) { | |
| 8003 | + bpf_log(log, "JIT does not support function %s() with %d args\n", | |
| 8004 | + tname, nargs); | |
| 8005 | + return -EFAULT; | |
| 8006 | + } | |
| 8007 | + sub->stack_arg_cnt = nargs - MAX_BPF_FUNC_REG_ARGS; | |
| 8008 | + } | |
| 8009 | + | |
| 8010 | + if (is_global && nargs > MAX_BPF_FUNC_REG_ARGS) { | |
| 8011 | + bpf_log(log, "global function %s has %d > %d args, stack args not supported\n", | |
| 7894 | 8012 | tname, nargs, MAX_BPF_FUNC_REG_ARGS); |
| 7895 | 8013 | return -EINVAL; |
| 7896 | 8014 | } |
| 7897 | − /* check that function is void or returns int, exception cb also requires this */ | |
| 7898 | − t = btf_type_by_id(btf, t->type); | |
| 7899 | − while (btf_type_is_modifier(t)) | |
| 7900 | − t = btf_type_by_id(btf, t->type); | |
| 7901 | − if (!btf_type_is_void(t) && !btf_type_is_int(t) && !btf_is_any_enum(t)) { | |
| 7902 | − if (!is_global) | |
| 7903 | − return -EINVAL; | |
| 7904 | − bpf_log(log, | |
| 7905 | − "Global function %s() return value not void or scalar. " | |
| 7906 | − "Only those are supported.\n", | |
| 7907 | − tname); | |
| 7908 | − return -EINVAL; | |
| 8015 | + | |
| 8016 | + err = btf_validate_return_type(env, btf, t, subprog); | |
| 8017 | + if (err) { | |
| 8018 | + if (is_global) { | |
| 8019 | + bpf_log(log, | |
| 8020 | + "Global function %s() return value not void or scalar. " | |
| 8021 | + "Only those are supported.\n", | |
| 8022 | + tname); | |
| 8023 | + } | |
| 8024 | + return err; | |
| 7909 | 8025 | } |
| 7910 | 8026 | |
| 7911 | 8027 | /* Convert BTF function arguments into verifier types. |
| @@ -7913,42 +8029,13 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog) · 1 unmodified line | ||
| 7913 | 8029 | */ |
| 7914 | 8030 | for (i = 0; i < nargs; i++) { |
| 7915 | 8031 | u32 tags = 0; |
| 7916 | − int id = btf_named_start_id(btf, false) - 1; | |
| 7917 | − | |
| 7918 | − /* 'arg:<tag>' decl_tag takes precedence over derivation of | |
| 7919 | − * register type from BTF type itself | |
| 7920 | − */ | |
| 7921 | − while ((id = btf_find_next_decl_tag(btf, fn_t, i, "arg:", id)) > 0) { | |
| 7922 | − const struct btf_type *tag_t = btf_type_by_id(btf, id); | |
| 7923 | − const char *tag = __btf_name_by_offset(btf, tag_t->name_off) + 4; | |
| 7924 | − | |
| 7925 | − /* disallow arg tags in static subprogs */ | |
| 7926 | − if (!is_global) { | |
| 7927 | − bpf_log(log, "arg#%d type tag is not supported in static functions\n", i); | |
| 7928 | − return -EOPNOTSUPP; | |
| 7929 | − } | |
| 8032 | + err = btf_scan_decl_tags(env, btf, fn_t, i, is_global, &tags); | |
| 8033 | + if (err) | |
| 8034 | + return err; | |
| 7930 | 8035 | |
| 7931 | − if (strcmp(tag, "ctx") == 0) { | |
| 7932 | − tags |= ARG_TAG_CTX; | |
| 7933 | − } else if (strcmp(tag, "trusted") == 0) { | |
| 7934 | − tags |= ARG_TAG_TRUSTED; | |
| 7935 | − } else if (strcmp(tag, "untrusted") == 0) { | |
| 7936 | − tags |= ARG_TAG_UNTRUSTED; | |
| 7937 | − } else if (strcmp(tag, "nonnull") == 0) { | |
| 7938 | − tags |= ARG_TAG_NONNULL; | |
| 7939 | − } else if (strcmp(tag, "nullable") == 0) { | |
| 7940 | − tags |= ARG_TAG_NULLABLE; | |
| 7941 | − } else if (strcmp(tag, "arena") == 0) { | |
| 7942 | − tags |= ARG_TAG_ARENA; | |
| 7943 | − } else { | |
| 7944 | − bpf_log(log, "arg#%d has unsupported set of tags\n", i); | |
| 7945 | − return -EOPNOTSUPP; | |
| 7946 | − } | |
| 7947 | − } | |
| 7948 | − if (id != -ENOENT) { | |
| 7949 | − bpf_log(log, "arg#%d type tag fetching failure: %d\n", i, id); | |
| 7950 | − return id; | |
| 7951 | − } | |
| 8036 | + err = btf_scan_type_tags(env, btf, args[i].type, &tags); | |
| 8037 | + if (err) | |
| 8038 | + return err; | |
| 7952 | 8039 | |
| 7953 | 8040 | t = btf_type_by_id(btf, args[i].type); |
| 7954 | 8041 | while (btf_type_is_modifier(t)) |
| @@ -7973,7 +8060,7 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog) · 18 unmodified lines | ||
| 7973 | 8060 | bpf_log(log, "arg#%d has invalid combination of tags\n", i); |
| 7974 | 8061 | return -EINVAL; |
| 7975 | 8062 | } |
| 7976 | − sub->args[i].arg_type = ARG_PTR_TO_DYNPTR | MEM_RDONLY; | |
| 8063 | + sub->args[i].arg_type = ARG_PTR_TO_DYNPTR; | |
| 7977 | 8064 | continue; |
| 7978 | 8065 | } |
| 7979 | 8066 | if (tags & ARG_TAG_TRUSTED) { |
| @@ -8074,7 +8161,6 @@ skip_pointer: · 94 unmodified lines | ||
| 8074 | 8161 | return -EINVAL; |
| 8075 | 8162 | } |
| 8076 | 8163 | |
| 8077 | − sub->arg_cnt = nargs; | |
| 8078 | 8164 | sub->args_cached = true; |
| 8079 | 8165 | |
| 8080 | 8166 | return 0; |
| @@ -8196,12 +8282,12 @@ static int __btf_new_fd(struct btf *btf) · 115 unmodified lines | ||
| 8196 | 8282 | return anon_inode_getfd("btf", &btf_fops, btf, O_RDONLY | O_CLOEXEC); |
| 8197 | 8283 | } |
| 8198 | 8284 | |
| 8199 | −int btf_new_fd(const union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size) | |
| 8285 | +int btf_new_fd(const union bpf_attr *attr, bpfptr_t uattr, struct bpf_log_attr *attr_log) | |
| 8200 | 8286 | { |
| 8201 | 8287 | struct btf *btf; |
| 8202 | 8288 | int ret; |
| 8203 | 8289 | |
| 8204 | − btf = btf_parse(attr, uattr, uattr_size); | |
| 8290 | + btf = btf_parse(attr, uattr, attr_log); | |
| 8205 | 8291 | if (IS_ERR(btf)) |
| 8206 | 8292 | return PTR_ERR(btf); |
| 8207 | 8293 | |
| @@ -8684,6 +8770,39 @@ static int btf_check_iter_kfuncs(struct btf *btf, const char *func_name, · 476 unmodified lines | ||
| 8684 | 8770 | return 0; |
| 8685 | 8771 | } |
| 8686 | 8772 | |
| 8773 | +static int btf_check_kfunc_name(struct btf *btf, const char *func_name, u32 kind) | |
| 8774 | +{ | |
| 8775 | +#ifdef CONFIG_DEBUG_INFO_BTF_MODULES | |
| 8776 | + struct btf_module *btf_mod, *tmp; | |
| 8777 | +#endif | |
| 8778 | + s32 id; | |
| 8779 | + | |
| 8780 | + if (!btf_is_module(btf)) | |
| 8781 | + return 0; | |
| 8782 | + | |
| 8783 | + id = btf_find_by_name_kind(bpf_get_btf_vmlinux(), func_name, kind); | |
| 8784 | + if (id >= 0) { | |
| 8785 | + pr_err("kfunc %s (id: %d) is already present in vmlinux.\n", | |
| 8786 | + func_name, id); | |
| 8787 | + return -EINVAL; | |
| 8788 | + } | |
| 8789 | + | |
| 8790 | +#ifdef CONFIG_DEBUG_INFO_BTF_MODULES | |
| 8791 | + guard(mutex)(&btf_module_mutex); | |
| 8792 | + list_for_each_entry_safe(btf_mod, tmp, &btf_modules, list) { | |
| 8793 | + if (btf_mod->btf == btf) | |
| 8794 | + continue; | |
| 8795 | + id = btf_find_by_name_kind(btf_mod->btf, func_name, kind); | |
| 8796 | + if (id >= 0) { | |
| 8797 | + pr_err("kfunc %s (id: %d) is already present in module %s.\n", | |
| 8798 | + func_name, id, btf_mod->module->name); | |
| 8799 | + return -EINVAL; | |
| 8800 | + } | |
| 8801 | + } | |
| 8802 | +#endif | |
| 8803 | + return 0; | |
| 8804 | +} | |
| 8805 | + | |
| 8687 | 8806 | static int btf_check_kfunc_protos(struct btf *btf, u32 func_id, u32 func_flags) |
| 8688 | 8807 | { |
| 8689 | 8808 | const struct btf_type *func; |
| @@ -8697,7 +8816,8 @@ static int btf_check_kfunc_protos(struct btf *btf, u32 func_id, u32 func_flags) · 7 unmodified lines | ||
| 8697 | 8816 | |
| 8698 | 8817 | /* sanity check kfunc name */ |
| 8699 | 8818 | func_name = btf_name_by_offset(btf, func->name_off); |
| 8700 | − if (!func_name || !func_name[0]) | |
| 8819 | + if (!func_name || !func_name[0] || | |
| 8820 | + btf_check_kfunc_name(btf, func_name, BTF_INFO_KIND(func->info))) | |
| 8701 | 8821 | return -EINVAL; |
| 8702 | 8822 | |
| 8703 | 8823 | func = btf_type_by_id(btf, func->type); |
| @@ -55,6 +55,28 @@ void __init cgroup_bpf_lifetime_notifier_init(void) · 54 unmodified lines | ||
| 55 | 55 | &cgroup_bpf_lifetime_nb)); |
| 56 | 56 | } |
| 57 | 57 | |
| 58 | +#ifdef CONFIG_BPF_LSM | |
| 59 | +struct cgroup_lsm_atype { | |
| 60 | + u32 attach_btf_id; | |
| 61 | + int refcnt; | |
| 62 | + bool returns_errno; | |
| 63 | +}; | |
| 64 | + | |
| 65 | +static struct cgroup_lsm_atype cgroup_lsm_atype[CGROUP_LSM_NUM]; | |
| 66 | + | |
| 67 | +static bool cgroup_bpf_hook_returns_errno(enum cgroup_bpf_attach_type atype) | |
| 68 | +{ | |
| 69 | + if (atype >= CGROUP_LSM_START && atype <= CGROUP_LSM_END) | |
| 70 | + return READ_ONCE(cgroup_lsm_atype[atype - CGROUP_LSM_START].returns_errno); | |
| 71 | + return true; | |
| 72 | +} | |
| 73 | +#else | |
| 74 | +static bool cgroup_bpf_hook_returns_errno(enum cgroup_bpf_attach_type atype) | |
| 75 | +{ | |
| 76 | + return true; | |
| 77 | +} | |
| 78 | +#endif | |
| 79 | + | |
| 58 | 80 | /* __always_inline is necessary to prevent indirect call through run_prog |
| 59 | 81 | * function pointer. |
| 60 | 82 | */ |
| @@ -83,7 +105,8 @@ bpf_prog_run_array_cg(const struct cgroup_bpf *cgrp, · 22 unmodified lines | ||
| 83 | 105 | *(ret_flags) |= (func_ret >> 1); |
| 84 | 106 | func_ret &= 1; |
| 85 | 107 | } |
| 86 | − if (!func_ret && !IS_ERR_VALUE((long)run_ctx.retval)) | |
| 108 | + if (!func_ret && cgroup_bpf_hook_returns_errno(atype) && | |
| 109 | + !IS_ERR_VALUE((long)run_ctx.retval)) | |
| 87 | 110 | run_ctx.retval = -EPERM; |
| 88 | 111 | item++; |
| 89 | 112 | } |
| @@ -156,13 +179,6 @@ unsigned int __cgroup_bpf_run_lsm_current(const void *ctx, · 66 unmodified lines | ||
| 156 | 179 | } |
| 157 | 180 | |
| 158 | 181 | #ifdef CONFIG_BPF_LSM |
| 159 | −struct cgroup_lsm_atype { | |
| 160 | − u32 attach_btf_id; | |
| 161 | − int refcnt; | |
| 162 | −}; | |
| 163 | − | |
| 164 | −static struct cgroup_lsm_atype cgroup_lsm_atype[CGROUP_LSM_NUM]; | |
| 165 | − | |
| 166 | 182 | static enum cgroup_bpf_attach_type |
| 167 | 183 | bpf_cgroup_atype_find(enum bpf_attach_type attach_type, u32 attach_btf_id) |
| 168 | 184 | { |
| @@ -191,10 +207,13 @@ void bpf_cgroup_atype_get(u32 attach_btf_id, int cgroup_atype) · 22 unmodified lines | ||
| 191 | 207 | |
| 192 | 208 | lockdep_assert_held(&cgroup_mutex); |
| 193 | 209 | |
| 194 | − WARN_ON_ONCE(cgroup_lsm_atype[i].attach_btf_id && | |
| 195 | − cgroup_lsm_atype[i].attach_btf_id != attach_btf_id); | |
| 196 | − | |
| 197 | − cgroup_lsm_atype[i].attach_btf_id = attach_btf_id; | |
| 210 | + if (!cgroup_lsm_atype[i].attach_btf_id) { | |
| 211 | + cgroup_lsm_atype[i].attach_btf_id = attach_btf_id; | |
| 212 | + WRITE_ONCE(cgroup_lsm_atype[i].returns_errno, | |
| 213 | + bpf_lsm_hook_returns_errno(attach_btf_id)); | |
| 214 | + } else { | |
| 215 | + WARN_ON_ONCE(cgroup_lsm_atype[i].attach_btf_id != attach_btf_id); | |
| 216 | + } | |
| 198 | 217 | cgroup_lsm_atype[i].refcnt++; |
| 199 | 218 | } |
| 200 | 219 | |
| @@ -203,8 +222,10 @@ void bpf_cgroup_atype_put(int cgroup_atype) · 2 unmodified lines | ||
| 203 | 222 | int i = cgroup_atype - CGROUP_LSM_START; |
| 204 | 223 | |
| 205 | 224 | cgroup_lock(); |
| 206 | − if (--cgroup_lsm_atype[i].refcnt <= 0) | |
| 225 | + if (--cgroup_lsm_atype[i].refcnt <= 0) { | |
| 226 | + WRITE_ONCE(cgroup_lsm_atype[i].returns_errno, true); | |
| 207 | 227 | cgroup_lsm_atype[i].attach_btf_id = 0; |
| 228 | + } | |
| 208 | 229 | WARN_ON_ONCE(cgroup_lsm_atype[i].refcnt < 0); |
| 209 | 230 | cgroup_unlock(); |
| 210 | 231 | } |
| @@ -1208,7 +1229,7 @@ static int cgroup_bpf_detach(struct cgroup *cgrp, struct bpf_prog *prog, · 997 unmodified lines | ||
| 1208 | 1229 | |
| 1209 | 1230 | /* Must be called with cgroup_mutex held to avoid races. */ |
| 1210 | 1231 | static int __cgroup_bpf_query(struct cgroup *cgrp, const union bpf_attr *attr, |
| 1211 | − union bpf_attr __user *uattr) | |
| 1232 | + union bpf_attr __user *uattr, u32 uattr_size) | |
| 1212 | 1233 | { |
| 1213 | 1234 | __u32 __user *prog_attach_flags = u64_to_user_ptr(attr->query.prog_attach_flags); |
| 1214 | 1235 | bool effective_query = attr->query.query_flags & BPF_F_QUERY_EFFECTIVE; |
| @@ -1259,7 +1280,8 @@ static int __cgroup_bpf_query(struct cgroup *cgrp, const union bpf_attr *attr, · 44 unmodified lines | ||
| 1259 | 1280 | return -EFAULT; |
| 1260 | 1281 | if (!effective_query && from_atype == to_atype) |
| 1261 | 1282 | revision = cgrp->bpf.revisions[from_atype]; |
| 1262 | − if (copy_to_user(&uattr->query.revision, &revision, sizeof(revision))) | |
| 1283 | + if (uattr_size >= offsetofend(union bpf_attr, query.revision) && | |
| 1284 | + copy_to_user(&uattr->query.revision, &revision, sizeof(revision))) | |
| 1263 | 1285 | return -EFAULT; |
| 1264 | 1286 | if (attr->query.prog_cnt == 0 || !prog_ids || !total_cnt) |
| 1265 | 1287 | /* return early if user requested only program count + flags */ |
| @@ -1312,12 +1334,12 @@ static int __cgroup_bpf_query(struct cgroup *cgrp, const union bpf_attr *attr, · 46 unmodified lines | ||
| 1312 | 1334 | } |
| 1313 | 1335 | |
| 1314 | 1336 | static int cgroup_bpf_query(struct cgroup *cgrp, const union bpf_attr *attr, |
| 1315 | − union bpf_attr __user *uattr) | |
| 1337 | + union bpf_attr __user *uattr, u32 uattr_size) | |
| 1316 | 1338 | { |
| 1317 | 1339 | int ret; |
| 1318 | 1340 | |
| 1319 | 1341 | cgroup_lock(); |
| 1320 | − ret = __cgroup_bpf_query(cgrp, attr, uattr); | |
| 1342 | + ret = __cgroup_bpf_query(cgrp, attr, uattr, uattr_size); | |
| 1321 | 1343 | cgroup_unlock(); |
| 1322 | 1344 | return ret; |
| 1323 | 1345 | } |
| @@ -1520,7 +1542,7 @@ out_put_cgroup: · 196 unmodified lines | ||
| 1520 | 1542 | } |
| 1521 | 1543 | |
| 1522 | 1544 | int cgroup_bpf_prog_query(const union bpf_attr *attr, |
| 1523 | − union bpf_attr __user *uattr) | |
| 1545 | + union bpf_attr __user *uattr, u32 uattr_size) | |
| 1524 | 1546 | { |
| 1525 | 1547 | struct cgroup *cgrp; |
| 1526 | 1548 | int ret; |
| @@ -1529,7 +1551,7 @@ int cgroup_bpf_prog_query(const union bpf_attr *attr, · 2 unmodified lines | ||
| 1529 | 1551 | if (IS_ERR(cgrp)) |
| 1530 | 1552 | return PTR_ERR(cgrp); |
| 1531 | 1553 | |
| 1532 | − ret = cgroup_bpf_query(cgrp, attr, uattr); | |
| 1554 | + ret = cgroup_bpf_query(cgrp, attr, uattr, uattr_size); | |
| 1533 | 1555 | |
| 1534 | 1556 | cgroup_put(cgrp); |
| 1535 | 1557 | return ret; |
| @@ -1935,8 +1957,8 @@ int __cgroup_bpf_run_filter_sysctl(struct ctl_table_header *head, · 399 unmodified lines | ||
| 1935 | 1957 | |
| 1936 | 1958 | kfree(ctx.cur_val); |
| 1937 | 1959 | |
| 1938 | − if (ret == 1 && ctx.new_updated) { | |
| 1939 | − kfree(*buf); | |
| 1960 | + if (!ret && ctx.new_updated) { | |
| 1961 | + kvfree(*buf); | |
| 1940 | 1962 | *buf = ctx.new_val; |
| 1941 | 1963 | *pcount = ctx.new_len; |
| 1942 | 1964 | } else { |
| @@ -2342,6 +2364,7 @@ BPF_CALL_3(bpf_sysctl_set_new_value, struct bpf_sysctl_kern *, ctx, · 399 unmodified lines | ||
| 2342 | 2364 | return -E2BIG; |
| 2343 | 2365 | |
| 2344 | 2366 | memcpy(ctx->new_val, buf, buf_len); |
| 2367 | + ((char *)ctx->new_val)[buf_len] = '\0'; | |
| 2345 | 2368 | ctx->new_len = buf_len; |
| 2346 | 2369 | ctx->new_updated = 1; |
| 2347 | 2370 | |
| @@ -0,0 +1,120 @@ | ||
| 1 | +// SPDX-License-Identifier: GPL-2.0-only | |
| 2 | +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ | |
| 3 | + | |
| 4 | +#include <linux/bits.h> | |
| 5 | + | |
| 6 | +#define T 32 | |
| 7 | +#include "cnum_defs.h" | |
| 8 | +#undef T | |
| 9 | + | |
| 10 | +#define T 64 | |
| 11 | +#include "cnum_defs.h" | |
| 12 | +#undef T | |
| 13 | + | |
| 14 | +struct cnum32 cnum32_from_cnum64(struct cnum64 cnum) | |
| 15 | +{ | |
| 16 | + if (cnum64_is_empty(cnum)) | |
| 17 | + return CNUM32_EMPTY; | |
| 18 | + | |
| 19 | + if (cnum.size >= U32_MAX) | |
| 20 | + return (struct cnum32){ .base = 0, .size = U32_MAX }; | |
| 21 | + else | |
| 22 | + return (struct cnum32){ .base = (u32)cnum.base, .size = cnum.size }; | |
| 23 | +} | |
| 24 | + | |
| 25 | +/* | |
| 26 | + * Suppose 'a' and 'b' are laid out as follows: | |
| 27 | + * | |
| 28 | + * 64-bit number axis ---> | |
| 29 | + * | |
| 30 | + * N*2^32 (N+1)*2^32 (N+2)*2^32 (N+3)*2^32 | |
| 31 | + * ||------|---|=====|-------||----------|=====|-------||----------|=====|----|--|| | |
| 32 | + * | |< b >| |< b >| |< b >| | | |
| 33 | + * | | | | | |
| 34 | + * |<--+--------------------------- a ---------------------------+--->| | |
| 35 | + * | | | |
| 36 | + * |<-------------------------- t -------------------------->| | |
| 37 | + * | |
| 38 | + * In such a case it is possible to infer a more tight representation t | |
| 39 | + * such that ∀ v ∈ a, (u32)v ∈ b: v ∈ t. | |
| 40 | + */ | |
| 41 | +struct cnum64 cnum64_cnum32_intersect(struct cnum64 a, struct cnum32 b) | |
| 42 | +{ | |
| 43 | + /* | |
| 44 | + * To simplify reasoning, rotate the circles so that [virtual] a1 starts | |
| 45 | + * at u32 boundary, b1 represents b in this new frame of reference. | |
| 46 | + */ | |
| 47 | + struct cnum32 b1 = { b.base - (u32)a.base, b.size }; | |
| 48 | + struct cnum64 t = a; | |
| 49 | + u64 d, b1_max; | |
| 50 | + | |
| 51 | + if (cnum64_is_empty(a) || cnum32_is_empty(b)) | |
| 52 | + return CNUM64_EMPTY; | |
| 53 | + | |
| 54 | + if (cnum32_urange_overflow(b1)) { | |
| 55 | + b1_max = (u32)b1.base + (u32)b1.size; /* overflow here is fine and necessary */ | |
| 56 | + if ((u32)a.size > b1_max && (u32)a.size < b1.base) { | |
| 57 | + /* | |
| 58 | + * N*2^32 (N+1)*2^32 | |
| 59 | + * ||=====|------------|=====||=====|---------|---|=====|| | |
| 60 | + * |b1 ->| |<- b1||b1 ->| | |<- b1| | |
| 61 | + * |<----------------- a1 ------------------>| | |
| 62 | + * |<-------------- t ------------>|<-- d -->| (after adjustment) | |
| 63 | + * ^ | |
| 64 | + * b1_max | |
| 65 | + */ | |
| 66 | + d = (u32)a.size - b1_max; | |
| 67 | + t.size -= d; | |
| 68 | + } else { | |
| 69 | + /* | |
| 70 | + * No adjustments possible in the following cases: | |
| 71 | + * | |
| 72 | + * ||=====|------------|=====||===|=|-------------|=|===|| | |
| 73 | + * |b1 ->| |<- b1||b1 +>| |<+ b1| | |
| 74 | + * |<----------------- a1 ------>| | | |
| 75 | + * |<----------------- (or) a1 ------------------->| | |
| 76 | + */ | |
| 77 | + } | |
| 78 | + } else { | |
| 79 | + if (t.size < b1.base) | |
| 80 | + /* | |
| 81 | + * N*2^32 (N+1)*2^32 | |
| 82 | + * ||----------|--|=======|--||------> | |
| 83 | + * |<-- a1 -->| |<- b ->| | |
| 84 | + */ | |
| 85 | + return CNUM64_EMPTY; | |
| 86 | + /* | |
| 87 | + * N*2^32 (N+1)*2^32 | |
| 88 | + * ||-------------|========|-||-----| -------|========|-|| | |
| 89 | + * | |<- b1 ->| | |<- b1 ->| | |
| 90 | + * |<------------+ a1 ------------>| | |
| 91 | + * |<------ t ------>| (after adjustment) | |
| 92 | + */ | |
| 93 | + t.base += b1.base; | |
| 94 | + t.size -= b1.base; | |
| 95 | + b1_max = b1.base + b1.size; | |
| 96 | + d = 0; | |
| 97 | + if ((u32)a.size < b1.base) | |
| 98 | + /* | |
| 99 | + * N*2^32 (N+1)*2^32 | |
| 100 | + * ||-------------|========|-||------|-------|========|-|| | |
| 101 | + * | |<- b1 ->| | |<- b1 ->| | |
| 102 | + * |<------------+-- a1 --+-------->| | |
| 103 | + * |<- t ->|<-- d -->| (after adjustment) | |
| 104 | + */ | |
| 105 | + d = (u32)a.size + (BIT_ULL(32) - b1_max); | |
| 106 | + else if ((u32)a.size >= b1_max) | |
| 107 | + /* | |
| 108 | + * N*2^32 (N+1)*2^32 | |
| 109 | + * ||--|========|------------||--|========|-------|-----|| | |
| 110 | + * | |<- b1 ->| |<- b1 ->| | | |
| 111 | + * |<-+------------------ a1 ------------+------>| | |
| 112 | + * |<-------------- t --------------->|<- d ->| (after adjustment) | |
| 113 | + */ | |
| 114 | + d = (u32)a.size - b1_max; | |
| 115 | + if (t.size < d) | |
| 116 | + return CNUM64_EMPTY; | |
| 117 | + t.size -= d; | |
| 118 | + } | |
| 119 | + return t; | |
| 120 | +} | |
| @@ -0,0 +1,247 @@ | ||
| 1 | +/* SPDX-License-Identifier: GPL-2.0-only */ | |
| 2 | +/* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ | |
| 3 | + | |
| 4 | +#ifndef T | |
| 5 | +#error "Define T (bit width: 32, 64) before including cnum_defs.h" | |
| 6 | +#endif | |
| 7 | + | |
| 8 | +#include <linux/cnum.h> | |
| 9 | +#include <linux/kernel.h> | |
| 10 | +#include <linux/limits.h> | |
| 11 | +#include <linux/minmax.h> | |
| 12 | +#include <linux/compiler_types.h> | |
| 13 | + | |
| 14 | +#define cnum_t __PASTE(cnum, T) | |
| 15 | +#define ut __PASTE(u, T) | |
| 16 | +#define st __PASTE(s, T) | |
| 17 | +#define UT_MAX __PASTE(__PASTE(U, T), _MAX) | |
| 18 | +#define ST_MAX __PASTE(__PASTE(S, T), _MAX) | |
| 19 | +#define ST_MIN __PASTE(__PASTE(S, T), _MIN) | |
| 20 | +#define EMPTY __PASTE(__PASTE(CNUM, T), _EMPTY) | |
| 21 | +#define FN(name) __PASTE(__PASTE(cnum, T), __PASTE(_, name)) | |
| 22 | + | |
| 23 | +struct cnum_t FN(from_urange)(ut min, ut max) | |
| 24 | +{ | |
| 25 | + return (struct cnum_t){ .base = min, .size = (ut)max - min }; | |
| 26 | +} | |
| 27 | + | |
| 28 | +struct cnum_t FN(from_srange)(st min, st max) | |
| 29 | +{ | |
| 30 | + ut size = (ut)max - (ut)min; | |
| 31 | + ut base = size == UT_MAX ? 0 : (ut)min; | |
| 32 | + | |
| 33 | + return (struct cnum_t){ .base = base, .size = size }; | |
| 34 | +} | |
| 35 | + | |
| 36 | +/* True if this cnum represents two unsigned ranges. */ | |
| 37 | +static inline bool FN(urange_overflow)(struct cnum_t cnum) | |
| 38 | +{ | |
| 39 | + /* Same as cnum.base + cnum.size > UT_MAX but avoids overflow */ | |
| 40 | + return cnum.size > UT_MAX - (ut)cnum.base; | |
| 41 | +} | |
| 42 | + | |
| 43 | +/* | |
| 44 | + * cnum{T}_umin / cnum{T}_umax query an unsigned range represented by this cnum. | |
| 45 | + * If cnum represents a range crossing the UT_MAX/0 boundary, the unbound range | |
| 46 | + * [0..UT_MAX] is returned. | |
| 47 | + */ | |
| 48 | +ut FN(umin)(struct cnum_t cnum) | |
| 49 | +{ | |
| 50 | + return FN(urange_overflow)(cnum) ? 0 : cnum.base; | |
| 51 | +} | |
| 52 | +EXPORT_SYMBOL_GPL(FN(umin)); | |
| 53 | + | |
| 54 | +ut FN(umax)(struct cnum_t cnum) | |
| 55 | +{ | |
| 56 | + return FN(urange_overflow)(cnum) ? UT_MAX : cnum.base + cnum.size; | |
| 57 | +} | |
| 58 | +EXPORT_SYMBOL_GPL(FN(umax)); | |
| 59 | + | |
| 60 | +/* True if this cnum represents two signed ranges. */ | |
| 61 | +static inline bool FN(srange_overflow)(struct cnum_t cnum) | |
| 62 | +{ | |
| 63 | + return FN(contains)(cnum, (ut)ST_MAX) && FN(contains)(cnum, (ut)ST_MIN); | |
| 64 | +} | |
| 65 | + | |
| 66 | +/* | |
| 67 | + * cnum{T}_smin / cnum{T}_smax query a signed range represented by this cnum. | |
| 68 | + * If cnum represents a range crossing the ST_MAX/ST_MIN boundary, the unbound range | |
| 69 | + * [ST_MIN..ST_MAX] is returned. | |
| 70 | + */ | |
| 71 | +st FN(smin)(struct cnum_t cnum) | |
| 72 | +{ | |
| 73 | + return FN(srange_overflow)(cnum) | |
| 74 | + ? ST_MIN | |
| 75 | + : min((st)cnum.base, (st)(cnum.base + cnum.size)); | |
| 76 | +} | |
| 77 | + | |
| 78 | +st FN(smax)(struct cnum_t cnum) | |
| 79 | +{ | |
| 80 | + return FN(srange_overflow)(cnum) | |
| 81 | + ? ST_MAX | |
| 82 | + : max((st)cnum.base, (st)(cnum.base + cnum.size)); | |
| 83 | +} | |
| 84 | + | |
| 85 | +/* | |
| 86 | + * Returns a possibly empty intersection of cnums 'a' and 'b'. | |
| 87 | + * If 'a' and 'b' intersect in two sub-arcs, the function over-approximates | |
| 88 | + * and returns either 'a' or 'b', whichever is smaller. | |
| 89 | + */ | |
| 90 | +struct cnum_t FN(intersect)(struct cnum_t a, struct cnum_t b) | |
| 91 | +{ | |
| 92 | + struct cnum_t b1; | |
| 93 | + ut dbase; | |
| 94 | + | |
| 95 | + if (FN(is_empty)(a) || FN(is_empty)(b)) | |
| 96 | + return EMPTY; | |
| 97 | + | |
| 98 | + if (a.base > b.base) | |
| 99 | + swap(a, b); | |
| 100 | + | |
| 101 | + /* | |
| 102 | + * Rotate frame of reference such that a.base is 0. | |
| 103 | + * 'b1' is 'b' in this frame of reference. | |
| 104 | + */ | |
| 105 | + dbase = b.base - a.base; | |
| 106 | + b1 = (struct cnum_t){ dbase, b.size }; | |
| 107 | + if (FN(urange_overflow)(b1)) { | |
| 108 | + if (b1.base <= a.size) { | |
| 109 | + /* | |
| 110 | + * Rotated frame (a.base at origin): | |
| 111 | + * | |
| 112 | + * 0 UT_MAX | |
| 113 | + * |--------------------------------------------| | |
| 114 | + * [=== a ==========================] | | |
| 115 | + * [= b1 tail =] [========= b1 main ==========>] | |
| 116 | + * ^-- b1.base <= a.size | |
| 117 | + * | |
| 118 | + * 'a' and 'b' intersect in two disjoint arcs, | |
| 119 | + * can't represent as single cnum, over-approximate | |
| 120 | + * the result. | |
| 121 | + */ | |
| 122 | + return a.size <= b.size ? a : b; | |
| 123 | + } else { | |
| 124 | + /* | |
| 125 | + * Rotated frame (a.base at origin): | |
| 126 | + * | |
| 127 | + * 0 UT_MAX | |
| 128 | + * |--------------------------------------------| | |
| 129 | + * [=== a =============] | | | |
| 130 | + * [= b1 tail =] [======= b1 main ====>] | |
| 131 | + * ^-- b1.base > a.size | |
| 132 | + * | |
| 133 | + * Only 'b' tail intersects 'a'. | |
| 134 | + */ | |
| 135 | + return (struct cnum_t) { | |
| 136 | + .base = a.base, | |
| 137 | + .size = min(a.size, (ut)(b1.base + b1.size)), | |
| 138 | + }; | |
| 139 | + } | |
| 140 | + } else if (a.size >= b1.base) { | |
| 141 | + /* | |
| 142 | + * Rotated frame (a.base at origin): | |
| 143 | + * | |
| 144 | + * 0 UT_MAX | |
| 145 | + * |--------------------------------------------------| | |
| 146 | + * [=== a ==================================] | | |
| 147 | + * [== b1 =====================] | |
| 148 | + * | |
| 149 | + * 0 UT_MAX | |
| 150 | + * |--------------------------------------------------| | |
| 151 | + * [=== a ==================================] | | |
| 152 | + * [== b1 ====] | |
| 153 | + * ^-- b1.base <= a.size | |
| 154 | + * |<-- a.size - dbase -->| | |
| 155 | + * | |
| 156 | + * 'a' and 'b' intersect as one cnum. | |
| 157 | + */ | |
| 158 | + return (struct cnum_t) { | |
| 159 | + .base = b.base, | |
| 160 | + .size = min((ut)(a.size - dbase), b.size), | |
| 161 | + }; | |
| 162 | + } else { | |
| 163 | + return EMPTY; | |
| 164 | + } | |
| 165 | +} | |
| 166 | + | |
| 167 | +void FN(intersect_with)(struct cnum_t *dst, struct cnum_t src) | |
| 168 | +{ | |
| 169 | + *dst = FN(intersect)(*dst, src); | |
| 170 | +} | |
| 171 | + | |
| 172 | +void FN(intersect_with_urange)(struct cnum_t *dst, ut min, ut max) | |
| 173 | +{ | |
| 174 | + FN(intersect_with)(dst, FN(from_urange)(min, max)); | |
| 175 | +} | |
| 176 | + | |
| 177 | +void FN(intersect_with_srange)(struct cnum_t *dst, st min, st max) | |
| 178 | +{ | |
| 179 | + FN(intersect_with)(dst, FN(from_srange)(min, max)); | |
| 180 | +} | |
| 181 | + | |
| 182 | +static inline struct cnum_t FN(normalize)(struct cnum_t cnum) | |
| 183 | +{ | |
| 184 | + if (cnum.size == UT_MAX && cnum.base != 0 && cnum.base != (ut)ST_MAX) | |
| 185 | + cnum.base = 0; | |
| 186 | + return cnum; | |
| 187 | +} | |
| 188 | + | |
| 189 | +struct cnum_t FN(add)(struct cnum_t a, struct cnum_t b) | |
| 190 | +{ | |
| 191 | + if (FN(is_empty)(a) || FN(is_empty)(b)) | |
| 192 | + return EMPTY; | |
| 193 | + if (a.size > UT_MAX - b.size) | |
| 194 | + return (struct cnum_t){ 0, (ut)UT_MAX }; | |
| 195 | + else | |
| 196 | + return FN(normalize)((struct cnum_t){ a.base + b.base, a.size + b.size }); | |
| 197 | +} | |
| 198 | + | |
| 199 | +struct cnum_t FN(negate)(struct cnum_t a) | |
| 200 | +{ | |
| 201 | + if (FN(is_empty)(a)) | |
| 202 | + return EMPTY; | |
| 203 | + return FN(normalize)((struct cnum_t){ -((ut)a.base + a.size), a.size }); | |
| 204 | +} | |
| 205 | + | |
| 206 | +bool FN(is_empty)(struct cnum_t cnum) | |
| 207 | +{ | |
| 208 | + return cnum.base == EMPTY.base && cnum.size == EMPTY.size; | |
| 209 | +} | |
| 210 | + | |
| 211 | +bool FN(contains)(struct cnum_t cnum, ut v) | |
| 212 | +{ | |
| 213 | + if (FN(is_empty)(cnum)) | |
| 214 | + return false; | |
| 215 | + if (FN(urange_overflow)(cnum)) | |
| 216 | + return v >= cnum.base || v <= (ut)cnum.base + cnum.size; | |
| 217 | + else | |
| 218 | + return v >= cnum.base && v <= (ut)cnum.base + cnum.size; | |
| 219 | +} | |
| 220 | + | |
| 221 | +bool FN(is_const)(struct cnum_t cnum) | |
| 222 | +{ | |
| 223 | + return cnum.size == 0; | |
| 224 | +} | |
| 225 | + | |
| 226 | +bool FN(is_subset)(struct cnum_t bigger, struct cnum_t smaller) | |
| 227 | +{ | |
| 228 | + if (FN(is_empty(smaller))) | |
| 229 | + return true; | |
| 230 | + if (FN(is_empty(bigger))) | |
| 231 | + return false; | |
| 232 | + /* rotate both arcs such that 'bigger' starts at origin, hence does not overflow */ | |
| 233 | + smaller.base -= bigger.base; | |
| 234 | + bigger.base = 0; | |
| 235 | + if (FN(urange_overflow)(smaller) && bigger.size < UT_MAX) | |
| 236 | + return false; | |
| 237 | + return smaller.base + smaller.size <= bigger.size; | |
| 238 | +} | |
| 239 | + | |
| 240 | +#undef EMPTY | |
| 241 | +#undef cnum_t | |
| 242 | +#undef ut | |
| 243 | +#undef st | |
| 244 | +#undef UT_MAX | |
| 245 | +#undef ST_MAX | |
| 246 | +#undef ST_MIN | |
| 247 | +#undef FN | |
| @@ -58,6 +58,14 @@ static void const_reg_xfer(struct bpf_verifier_env *env, struct const_arg_info * · 57 unmodified lines | ||
| 58 | 58 | u8 opcode = BPF_OP(insn->code) | BPF_SRC(insn->code); |
| 59 | 59 | int r; |
| 60 | 60 | |
| 61 | + /* Stack arg stores (r11-based) are outside the tracked register set. */ | |
| 62 | + if (is_stack_arg_st(insn) || is_stack_arg_stx(insn)) | |
| 63 | + return; | |
| 64 | + if (is_stack_arg_ldx(insn)) { | |
| 65 | + ci_out[insn->dst_reg] = unknown; | |
| 66 | + return; | |
| 67 | + } | |
| 68 | + | |
| 61 | 69 | switch (class) { |
| 62 | 70 | case BPF_ALU: |
| 63 | 71 | case BPF_ALU64: |
| @@ -1299,8 +1299,8 @@ static int bpf_jit_blind_insn(const struct bpf_insn *from, · 1298 unmodified lines | ||
| 1299 | 1299 | u32 imm_rnd = get_random_u32(); |
| 1300 | 1300 | s16 off; |
| 1301 | 1301 | |
| 1302 | − BUILD_BUG_ON(BPF_REG_AX + 1 != MAX_BPF_JIT_REG); | |
| 1303 | − BUILD_BUG_ON(MAX_BPF_REG + 1 != MAX_BPF_JIT_REG); | |
| 1302 | + BUILD_BUG_ON(BPF_REG_PARAMS + 2 != MAX_BPF_JIT_REG); | |
| 1303 | + BUILD_BUG_ON(BPF_REG_AX + 1 != MAX_BPF_JIT_REG); | |
| 1304 | 1304 | |
| 1305 | 1305 | /* Constraints on AX register: |
| 1306 | 1306 | * |
| @@ -1582,6 +1582,16 @@ bool bpf_insn_is_indirect_target(const struct bpf_verifier_env *env, const struc · 275 unmodified lines | ||
| 1582 | 1582 | insn_idx += prog->aux->subprog_start; |
| 1583 | 1583 | return env->insn_aux_data[insn_idx].indirect_target; |
| 1584 | 1584 | } |
| 1585 | + | |
| 1586 | +u16 bpf_out_stack_arg_cnt(const struct bpf_verifier_env *env, const struct bpf_prog *prog) | |
| 1587 | +{ | |
| 1588 | + const struct bpf_subprog_info *sub; | |
| 1589 | + | |
| 1590 | + if (!env) | |
| 1591 | + return 0; | |
| 1592 | + sub = &env->subprog_info[prog->aux->func_idx]; | |
| 1593 | + return sub->stack_arg_cnt - bpf_in_stack_arg_cnt(sub); | |
| 1594 | +} | |
| 1585 | 1595 | #endif /* CONFIG_BPF_JIT */ |
| 1586 | 1596 | |
| 1587 | 1597 | /* Base function for offset calculation. Needs to go into .text section, |
| @@ -2471,7 +2481,7 @@ static bool __bpf_prog_map_compatible(struct bpf_map *map, · 883 unmodified lines | ||
| 2471 | 2481 | cookie = aux->cgroup_storage[i] ? |
| 2472 | 2482 | aux->cgroup_storage[i]->cookie : 0; |
| 2473 | 2483 | ret = map->owner->storage_cookie[i] == cookie || |
| 2474 | − !cookie; | |
| 2484 | + (!cookie && !aux->tail_call_reachable); | |
| 2475 | 2485 | } |
| 2476 | 2486 | if (ret && |
| 2477 | 2487 | map->owner->attach_func_proto != aux->attach_func_proto) { |
| @@ -3228,6 +3238,11 @@ bool __weak bpf_jit_supports_kfunc_call(void) · 750 unmodified lines | ||
| 3228 | 3238 | return false; |
| 3229 | 3239 | } |
| 3230 | 3240 | |
| 3241 | +bool __weak bpf_jit_supports_stack_args(void) | |
| 3242 | +{ | |
| 3243 | + return false; | |
| 3244 | +} | |
| 3245 | + | |
| 3231 | 3246 | bool __weak bpf_jit_supports_far_kfunc_call(void) |
| 3232 | 3247 | { |
| 3233 | 3248 | return false; |
| @@ -3363,6 +3378,12 @@ __weak u64 bpf_arena_get_kern_vm_start(struct bpf_arena *arena) · 129 unmodified lines | ||
| 3363 | 3378 | } |
| 3364 | 3379 | |
| 3365 | 3380 | #ifdef CONFIG_BPF_SYSCALL |
| 3381 | +__weak bool bpf_arena_handle_page_fault(unsigned long addr, bool is_write, | |
| 3382 | + unsigned long fault_ip) | |
| 3383 | +{ | |
| 3384 | + return false; | |
| 3385 | +} | |
| 3386 | + | |
| 3366 | 3387 | static int __init bpf_global_ma_init(void) |
| 3367 | 3388 | { |
| 3368 | 3389 | int ret; |
| @@ -581,6 +581,10 @@ static int dev_map_enqueue_clone(struct bpf_dtab_netdev *obj, · 580 unmodified lines | ||
| 581 | 581 | { |
| 582 | 582 | struct xdp_frame *nxdpf; |
| 583 | 583 | |
| 584 | + /* Frags live outside the linear frame and cannot be cloned safely. */ | |
| 585 | + if (unlikely(xdp_frame_has_frags(xdpf))) | |
| 586 | + return -EOPNOTSUPP; | |
| 587 | + | |
| 584 | 588 | nxdpf = xdpf_clone(xdpf); |
| 585 | 589 | if (!nxdpf) |
| 586 | 590 | return -ENOMEM; |
| @@ -706,6 +710,18 @@ int dev_map_generic_redirect(struct bpf_dtab_netdev *dst, struct sk_buff *skb, · 119 unmodified lines | ||
| 706 | 710 | if (unlikely(err)) |
| 707 | 711 | return err; |
| 708 | 712 | |
| 713 | + if (dst->xdp_prog && skb_cloned(skb)) { | |
| 714 | + struct sk_buff *nskb; | |
| 715 | + | |
| 716 | + nskb = skb_copy(skb, GFP_ATOMIC); | |
| 717 | + if (!nskb) | |
| 718 | + return -ENOMEM; | |
| 719 | + | |
| 720 | + nskb->mac_len = skb->mac_len; | |
| 721 | + consume_skb(skb); | |
| 722 | + skb = nskb; | |
| 723 | + } | |
| 724 | + | |
| 709 | 725 | /* Redirect has already succeeded semantically at this point, so we just |
| 710 | 726 | * return 0 even if packet is dropped. Helper below takes care of |
| 711 | 727 | * freeing skb. |
| @@ -726,6 +742,9 @@ static int dev_map_redirect_clone(struct bpf_dtab_netdev *dst, · 14 unmodified lines | ||
| 726 | 742 | struct sk_buff *nskb; |
| 727 | 743 | int err; |
| 728 | 744 | |
| 745 | + if (unlikely(skb_is_nonlinear(skb))) | |
| 746 | + return -EOPNOTSUPP; | |
| 747 | + | |
| 729 | 748 | nskb = skb_clone(skb, GFP_ATOMIC); |
| 730 | 749 | if (!nskb) |
| 731 | 750 | return -ENOMEM; |
| @@ -870,7 +870,7 @@ int bpf_convert_ctx_accesses(struct bpf_verifier_env *env) · 869 unmodified lines | ||
| 870 | 870 | case PTR_TO_BTF_ID: |
| 871 | 871 | case PTR_TO_BTF_ID | PTR_UNTRUSTED: |
| 872 | 872 | /* PTR_TO_BTF_ID | MEM_ALLOC always has a valid lifetime, unlike |
| 873 | − * PTR_TO_BTF_ID, and an active ref_obj_id, but the same cannot | |
| 873 | + * PTR_TO_BTF_ID, and an active referenced id, but the same cannot | |
| 874 | 874 | * be said once it is marked PTR_UNTRUSTED, hence we must handle |
| 875 | 875 | * any faults for loads into such types. BPF_WRITE is disallowed |
| 876 | 876 | * for this case. |
| @@ -1265,6 +1265,7 @@ static int jit_subprogs(struct bpf_verifier_env *env) · 388 unmodified lines | ||
| 1265 | 1265 | prog->aux->real_func_cnt = env->subprog_cnt; |
| 1266 | 1266 | prog->aux->bpf_exception_cb = (void *)func[env->exception_callback_subprog]->bpf_func; |
| 1267 | 1267 | prog->aux->exception_boundary = func[0]->aux->exception_boundary; |
| 1268 | + prog->aux->stack_arg_sp_adjust = func[0]->aux->stack_arg_sp_adjust; | |
| 1268 | 1269 | bpf_prog_jit_attempt_done(prog); |
| 1269 | 1270 | return 0; |
| 1270 | 1271 | out_free: |
| @@ -1378,9 +1379,21 @@ int bpf_fixup_call_args(struct bpf_verifier_env *env) · 107 unmodified lines | ||
| 1378 | 1379 | struct bpf_prog *prog = env->prog; |
| 1379 | 1380 | struct bpf_insn *insn = prog->insnsi; |
| 1380 | 1381 | bool has_kfunc_call = bpf_prog_has_kfunc_call(prog); |
| 1381 | − int i, depth; | |
| 1382 | + int depth; | |
| 1382 | 1383 | #endif |
| 1383 | − int err = 0; | |
| 1384 | + int i, err = 0; | |
| 1385 | + | |
| 1386 | + for (i = 0; i < env->subprog_cnt; i++) { | |
| 1387 | + struct bpf_subprog_info *subprog = &env->subprog_info[i]; | |
| 1388 | + u16 outgoing = subprog->stack_arg_cnt - bpf_in_stack_arg_cnt(subprog); | |
| 1389 | + | |
| 1390 | + if (subprog->max_out_stack_arg_cnt > outgoing) { | |
| 1391 | + verbose(env, | |
| 1392 | + "func#%d writes %u stack arg slots, but calls only require %u\n", | |
| 1393 | + i, subprog->max_out_stack_arg_cnt, outgoing); | |
| 1394 | + return -EINVAL; | |
| 1395 | + } | |
| 1396 | + } | |
| 1384 | 1397 | |
| 1385 | 1398 | if (env->prog->jit_requested && |
| 1386 | 1399 | !bpf_prog_is_offloaded(env->prog->aux)) { |
| @@ -1395,6 +1408,12 @@ int bpf_fixup_call_args(struct bpf_verifier_env *env) · 8 unmodified lines | ||
| 1395 | 1408 | verbose(env, "calling kernel functions are not allowed in non-JITed programs\n"); |
| 1396 | 1409 | return -EINVAL; |
| 1397 | 1410 | } |
| 1411 | + for (i = 0; i < env->subprog_cnt; i++) { | |
| 1412 | + if (bpf_in_stack_arg_cnt(&env->subprog_info[i])) { | |
| 1413 | + verbose(env, "stack args are not supported in non-JITed programs\n"); | |
| 1414 | + return -EINVAL; | |
| 1415 | + } | |
| 1416 | + } | |
| 1398 | 1417 | if (env->subprog_cnt > 1 && env->prog->aux->tail_call_reachable) { |
| 1399 | 1418 | /* When JIT fails the progs with bpf2bpf calls and tail_calls |
| 1400 | 1419 | * have to be rejected, since interpreter doesn't support them yet. |
| @@ -2167,6 +2186,8 @@ patch_map_ops_generic: · 766 unmodified lines | ||
| 2167 | 2186 | insn->imm == BPF_FUNC_get_func_ret) { |
| 2168 | 2187 | if (eatype == BPF_TRACE_FEXIT || |
| 2169 | 2188 | eatype == BPF_TRACE_FSESSION || |
| 2189 | + eatype == BPF_TRACE_FEXIT_MULTI || | |
| 2190 | + eatype == BPF_TRACE_FSESSION_MULTI || | |
| 2170 | 2191 | eatype == BPF_MODIFY_RETURN) { |
| 2171 | 2192 | /* Load nr_args from ctx - 8 */ |
| 2172 | 2193 | insn_buf[0] = BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_1, -8); |
| @@ -9,6 +9,7 @@ · 8 unmodified lines | ||
| 9 | 9 | #include <linux/rculist_nulls.h> |
| 10 | 10 | #include <linux/rcupdate_wait.h> |
| 11 | 11 | #include <linux/random.h> |
| 12 | +#include <linux/rhashtable.h> | |
| 12 | 13 | #include <uapi/linux/btf.h> |
| 13 | 14 | #include <linux/rcupdate_trace.h> |
| 14 | 15 | #include <linux/btf_ids.h> |
| @@ -242,6 +243,10 @@ static void htab_free_prealloced_fields(struct bpf_htab *htab) · 227 unmodified lines | ||
| 242 | 243 | |
| 243 | 244 | if (IS_ERR_OR_NULL(htab->map.record)) |
| 244 | 245 | return; |
| 246 | + /* | |
| 247 | + * Preallocated maps do not have a bpf_mem_alloc destructor, so fully | |
| 248 | + * destroy every element, including the extra elements. | |
| 249 | + */ | |
| 245 | 250 | if (htab_has_extra_elems(htab)) |
| 246 | 251 | num_entries += num_possible_cpus(); |
| 247 | 252 | for (i = 0; i < num_entries; i++) { |
| @@ -496,28 +501,26 @@ static void htab_dtor_ctx_free(void *ctx) · 248 unmodified lines | ||
| 496 | 501 | kfree(ctx); |
| 497 | 502 | } |
| 498 | 503 | |
| 499 | −static int htab_set_dtor(struct bpf_htab *htab, void (*dtor)(void *, void *)) | |
| 504 | +static int bpf_ma_set_dtor(struct bpf_map *map, struct bpf_mem_alloc *ma, | |
| 505 | + void (*dtor)(void *, void *)) | |
| 500 | 506 | { |
| 501 | − u32 key_size = htab->map.key_size; | |
| 502 | − struct bpf_mem_alloc *ma; | |
| 503 | 507 | struct htab_btf_record *hrec; |
| 504 | 508 | int err; |
| 505 | 509 | |
| 506 | 510 | /* No need for dtors. */ |
| 507 | − if (IS_ERR_OR_NULL(htab->map.record)) | |
| 511 | + if (IS_ERR_OR_NULL(map->record)) | |
| 508 | 512 | return 0; |
| 509 | 513 | |
| 510 | 514 | hrec = kzalloc(sizeof(*hrec), GFP_KERNEL); |
| 511 | 515 | if (!hrec) |
| 512 | 516 | return -ENOMEM; |
| 513 | − hrec->key_size = key_size; | |
| 514 | − hrec->record = btf_record_dup(htab->map.record); | |
| 517 | + hrec->key_size = map->key_size; | |
| 518 | + hrec->record = btf_record_dup(map->record); | |
| 515 | 519 | if (IS_ERR(hrec->record)) { |
| 516 | 520 | err = PTR_ERR(hrec->record); |
| 517 | 521 | kfree(hrec); |
| 518 | 522 | return err; |
| 519 | 523 | } |
| 520 | − ma = htab_is_percpu(htab) ? &htab->pcpu_ma : &htab->ma; | |
| 521 | 524 | bpf_mem_alloc_set_dtor(ma, dtor, htab_dtor_ctx_free, hrec); |
| 522 | 525 | return 0; |
| 523 | 526 | } |
| @@ -534,9 +537,9 @@ static int htab_map_check_btf(struct bpf_map *map, const struct btf *btf, · 10 unmodified lines | ||
| 534 | 537 | * populated in htab_map_alloc(), so it will always appear as NULL. |
| 535 | 538 | */ |
| 536 | 539 | if (htab_is_percpu(htab)) |
| 537 | − return htab_set_dtor(htab, htab_pcpu_mem_dtor); | |
| 540 | + return bpf_ma_set_dtor(map, &htab->pcpu_ma, htab_pcpu_mem_dtor); | |
| 538 | 541 | else |
| 539 | − return htab_set_dtor(htab, htab_mem_dtor); | |
| 542 | + return bpf_ma_set_dtor(map, &htab->ma, htab_mem_dtor); | |
| 540 | 543 | } |
| 541 | 544 | |
| 542 | 545 | static struct bpf_map *htab_map_alloc(union bpf_attr *attr) |
| @@ -834,8 +837,8 @@ static int htab_lru_map_gen_lookup(struct bpf_map *map, · 291 unmodified lines | ||
| 834 | 837 | return insn - insn_buf; |
| 835 | 838 | } |
| 836 | 839 | |
| 837 | −static void check_and_free_fields(struct bpf_htab *htab, | |
| 838 | − struct htab_elem *elem) | |
| 840 | +static void check_and_cancel_fields(struct bpf_htab *htab, | |
| 841 | + struct htab_elem *elem) | |
| 839 | 842 | { |
| 840 | 843 | if (IS_ERR_OR_NULL(htab->map.record)) |
| 841 | 844 | return; |
| @@ -845,11 +848,11 @@ static void check_and_free_fields(struct bpf_htab *htab, · 3 unmodified lines | ||
| 845 | 848 | int cpu; |
| 846 | 849 | |
| 847 | 850 | for_each_possible_cpu(cpu) |
| 848 | − bpf_obj_free_fields(htab->map.record, per_cpu_ptr(pptr, cpu)); | |
| 851 | + bpf_obj_cancel_fields(&htab->map, per_cpu_ptr(pptr, cpu)); | |
| 849 | 852 | } else { |
| 850 | 853 | void *map_value = htab_elem_value(elem, htab->map.key_size); |
| 851 | 854 | |
| 852 | − bpf_obj_free_fields(htab->map.record, map_value); | |
| 855 | + bpf_obj_cancel_fields(&htab->map, map_value); | |
| 853 | 856 | } |
| 854 | 857 | } |
| 855 | 858 | |
| @@ -884,7 +887,7 @@ static bool htab_lru_map_delete_node(void *arg, struct bpf_lru_node *node) · 28 unmodified lines | ||
| 884 | 887 | htab_unlock_bucket(b, flags); |
| 885 | 888 | |
| 886 | 889 | if (l == tgt_l) |
| 887 | − check_and_free_fields(htab, l); | |
| 890 | + check_and_cancel_fields(htab, l); | |
| 888 | 891 | return l == tgt_l; |
| 889 | 892 | } |
| 890 | 893 | |
| @@ -949,7 +952,7 @@ find_first_elem: · 58 unmodified lines | ||
| 949 | 952 | |
| 950 | 953 | static void htab_elem_free(struct bpf_htab *htab, struct htab_elem *l) |
| 951 | 954 | { |
| 952 | − check_and_free_fields(htab, l); | |
| 955 | + check_and_cancel_fields(htab, l); | |
| 953 | 956 | |
| 954 | 957 | if (htab->map.map_type == BPF_MAP_TYPE_PERCPU_HASH) |
| 955 | 958 | bpf_mem_cache_free(&htab->pcpu_ma, l->ptr_to_pptr); |
| @@ -1002,7 +1005,7 @@ static void free_htab_elem(struct bpf_htab *htab, struct htab_elem *l) · 46 unmodified lines | ||
| 1002 | 1005 | |
| 1003 | 1006 | if (htab_is_prealloc(htab)) { |
| 1004 | 1007 | bpf_map_dec_elem_count(&htab->map); |
| 1005 | − check_and_free_fields(htab, l); | |
| 1008 | + check_and_cancel_fields(htab, l); | |
| 1006 | 1009 | pcpu_freelist_push(&htab->freelist, &l->fnode); |
| 1007 | 1010 | } else { |
| 1008 | 1011 | dec_elem_count(htab); |
| @@ -1019,7 +1022,7 @@ static void pcpu_copy_value(struct bpf_htab *htab, void __percpu *pptr, · 10 unmodified lines | ||
| 1019 | 1022 | /* copy true value_size bytes */ |
| 1020 | 1023 | ptr = this_cpu_ptr(pptr); |
| 1021 | 1024 | copy_map_value(&htab->map, ptr, value); |
| 1022 | − bpf_obj_free_fields(htab->map.record, ptr); | |
| 1025 | + bpf_obj_cancel_fields(&htab->map, ptr); | |
| 1023 | 1026 | } else { |
| 1024 | 1027 | u32 size = round_up(htab->map.value_size, 8); |
| 1025 | 1028 | void *val; |
| @@ -1029,7 +1032,7 @@ static void pcpu_copy_value(struct bpf_htab *htab, void __percpu *pptr, · 3 unmodified lines | ||
| 1029 | 1032 | cpu = map_flags >> 32; |
| 1030 | 1033 | ptr = per_cpu_ptr(pptr, cpu); |
| 1031 | 1034 | copy_map_value(&htab->map, ptr, value); |
| 1032 | − bpf_obj_free_fields(htab->map.record, ptr); | |
| 1035 | + bpf_obj_cancel_fields(&htab->map, ptr); | |
| 1033 | 1036 | return; |
| 1034 | 1037 | } |
| 1035 | 1038 | |
| @@ -1037,7 +1040,7 @@ static void pcpu_copy_value(struct bpf_htab *htab, void __percpu *pptr, · 1 unmodified line | ||
| 1037 | 1040 | ptr = per_cpu_ptr(pptr, cpu); |
| 1038 | 1041 | val = (map_flags & BPF_F_ALL_CPUS) ? value : value + size * cpu; |
| 1039 | 1042 | copy_map_value(&htab->map, ptr, val); |
| 1040 | − bpf_obj_free_fields(htab->map.record, ptr); | |
| 1043 | + bpf_obj_cancel_fields(&htab->map, ptr); | |
| 1041 | 1044 | } |
| 1042 | 1045 | } |
| 1043 | 1046 | } |
| @@ -1253,11 +1256,11 @@ static long htab_map_update_elem(struct bpf_map *map, void *key, void *value, · 209 unmodified lines | ||
| 1253 | 1256 | if (l_old) { |
| 1254 | 1257 | hlist_nulls_del_rcu(&l_old->hash_node); |
| 1255 | 1258 | |
| 1256 | − /* l_old has already been stashed in htab->extra_elems, free | |
| 1257 | − * its special fields before it is available for reuse. | |
| 1259 | + /* l_old has already been stashed in htab->extra_elems, cancel | |
| 1260 | + * its reusable special fields before it is available for reuse. | |
| 1258 | 1261 | */ |
| 1259 | 1262 | if (htab_is_prealloc(htab)) |
| 1260 | − check_and_free_fields(htab, l_old); | |
| 1263 | + check_and_cancel_fields(htab, l_old); | |
| 1261 | 1264 | } |
| 1262 | 1265 | htab_unlock_bucket(b, flags); |
| 1263 | 1266 | if (l_old && !htab_is_prealloc(htab)) |
| @@ -1270,7 +1273,7 @@ err: · 6 unmodified lines | ||
| 1270 | 1273 | |
| 1271 | 1274 | static void htab_lru_push_free(struct bpf_htab *htab, struct htab_elem *elem) |
| 1272 | 1275 | { |
| 1273 | − check_and_free_fields(htab, elem); | |
| 1276 | + check_and_cancel_fields(htab, elem); | |
| 1274 | 1277 | bpf_map_dec_elem_count(&htab->map); |
| 1275 | 1278 | bpf_lru_push_free(&htab->lru, &elem->lru_node); |
| 1276 | 1279 | } |
| @@ -2739,3 +2742,794 @@ const struct bpf_map_ops htab_of_maps_map_ops = { · 1462 unmodified lines | ||
| 2739 | 2742 | BATCH_OPS(htab), |
| 2740 | 2743 | .map_btf_id = &htab_map_btf_ids[0], |
| 2741 | 2744 | }; |
| 2745 | + | |
| 2746 | +struct rhtab_elem { | |
| 2747 | + struct rhash_head node; | |
| 2748 | + /* key bytes, then value bytes follow */ | |
| 2749 | + u8 data[] __aligned(8); | |
| 2750 | +}; | |
| 2751 | + | |
| 2752 | +struct bpf_rhtab { | |
| 2753 | + struct bpf_map map; | |
| 2754 | + struct rhashtable ht; | |
| 2755 | + struct bpf_mem_alloc ma; | |
| 2756 | + u32 elem_size; | |
| 2757 | + bool freeing_internal; | |
| 2758 | +}; | |
| 2759 | + | |
| 2760 | +static const struct rhashtable_params rhtab_params = { | |
| 2761 | + .head_offset = offsetof(struct rhtab_elem, node), | |
| 2762 | + .key_offset = offsetof(struct rhtab_elem, data), | |
| 2763 | +}; | |
| 2764 | + | |
| 2765 | +static inline void *rhtab_elem_value(struct rhtab_elem *l, u32 key_size) | |
| 2766 | +{ | |
| 2767 | + return l->data + round_up(key_size, 8); | |
| 2768 | +} | |
| 2769 | + | |
| 2770 | +/* Specialize hash function and objcmp for long sized key */ | |
| 2771 | +static __always_inline int rhtab_key_cmp_long(struct rhashtable_compare_arg *arg, | |
| 2772 | + const void *ptr) | |
| 2773 | +{ | |
| 2774 | + const unsigned long key1 = *(const unsigned long *)arg->key; | |
| 2775 | + const struct rhtab_elem *key2 = ptr; | |
| 2776 | + | |
| 2777 | + return key1 != *(const unsigned long *)key2->data; | |
| 2778 | +} | |
| 2779 | + | |
| 2780 | +static __always_inline u32 rhtab_hashfn_long(const void *data, u32 len, u32 seed) | |
| 2781 | +{ | |
| 2782 | + u64 k = *(const unsigned long *)data; | |
| 2783 | + | |
| 2784 | + return (u32)(k ^ (k >> 32)) ^ seed; | |
| 2785 | +} | |
| 2786 | + | |
| 2787 | +static const struct rhashtable_params rhtab_params_long = { | |
| 2788 | + .head_offset = offsetof(struct rhtab_elem, node), | |
| 2789 | + .key_offset = offsetof(struct rhtab_elem, data), | |
| 2790 | + .key_len = sizeof(long), | |
| 2791 | + .hashfn = rhtab_hashfn_long, | |
| 2792 | + .obj_cmpfn = rhtab_key_cmp_long, | |
| 2793 | +}; | |
| 2794 | + | |
| 2795 | +static struct bpf_map *rhtab_map_alloc(union bpf_attr *attr) | |
| 2796 | +{ | |
| 2797 | + struct rhashtable_params params; | |
| 2798 | + struct bpf_rhtab *rhtab; | |
| 2799 | + int err = 0; | |
| 2800 | + | |
| 2801 | + rhtab = bpf_map_area_alloc(sizeof(*rhtab), NUMA_NO_NODE); | |
| 2802 | + if (!rhtab) | |
| 2803 | + return ERR_PTR(-ENOMEM); | |
| 2804 | + | |
| 2805 | + bpf_map_init_from_attr(&rhtab->map, attr); | |
| 2806 | + | |
| 2807 | + if (rhtab->map.max_entries > 1UL << 31) { | |
| 2808 | + err = -E2BIG; | |
| 2809 | + goto free_rhtab; | |
| 2810 | + } | |
| 2811 | + | |
| 2812 | + rhtab->elem_size = sizeof(struct rhtab_elem) + round_up(rhtab->map.key_size, 8) + | |
| 2813 | + round_up(rhtab->map.value_size, 8); | |
| 2814 | + | |
| 2815 | + params = rhtab_params; | |
| 2816 | + params.key_len = rhtab->map.key_size; | |
| 2817 | + params.nelem_hint = (u32)attr->map_extra; | |
| 2818 | + params.automatic_shrinking = true; | |
| 2819 | + | |
| 2820 | + if (rhtab->map.key_size == sizeof(long)) { | |
| 2821 | + params.hashfn = rhtab_hashfn_long; | |
| 2822 | + params.obj_cmpfn = rhtab_key_cmp_long; | |
| 2823 | + } | |
| 2824 | + | |
| 2825 | + err = rhashtable_init(&rhtab->ht, ¶ms); | |
| 2826 | + if (err) | |
| 2827 | + goto free_rhtab; | |
| 2828 | + | |
| 2829 | + /* Set max_elems after rhashtable_init() since init zeroes the struct */ | |
| 2830 | + rhtab->ht.max_elems = rhtab->map.max_entries; | |
| 2831 | + | |
| 2832 | + err = bpf_mem_alloc_init(&rhtab->ma, rhtab->elem_size, false); | |
| 2833 | + if (err) | |
| 2834 | + goto destroy_rhtab; | |
| 2835 | + | |
| 2836 | + return &rhtab->map; | |
| 2837 | + | |
| 2838 | +destroy_rhtab: | |
| 2839 | + rhashtable_destroy(&rhtab->ht); | |
| 2840 | +free_rhtab: | |
| 2841 | + bpf_map_area_free(rhtab); | |
| 2842 | + return ERR_PTR(err); | |
| 2843 | +} | |
| 2844 | + | |
| 2845 | +static int rhtab_map_alloc_check(union bpf_attr *attr) | |
| 2846 | +{ | |
| 2847 | + if (!(attr->map_flags & BPF_F_NO_PREALLOC)) | |
| 2848 | + return -EINVAL; | |
| 2849 | + | |
| 2850 | + if (attr->map_flags & BPF_F_ZERO_SEED) | |
| 2851 | + return -EINVAL; | |
| 2852 | + | |
| 2853 | + if (attr->key_size > U16_MAX) | |
| 2854 | + return -E2BIG; | |
| 2855 | + | |
| 2856 | + if (attr->map_extra >> 32) | |
| 2857 | + return -EINVAL; | |
| 2858 | + | |
| 2859 | + if ((u32)attr->map_extra > U16_MAX) | |
| 2860 | + return -E2BIG; | |
| 2861 | + | |
| 2862 | + if ((u32)attr->map_extra > attr->max_entries) | |
| 2863 | + return -EINVAL; | |
| 2864 | + | |
| 2865 | + return htab_map_alloc_check(attr); | |
| 2866 | +} | |
| 2867 | + | |
| 2868 | +static void rhtab_check_and_free_fields(struct bpf_rhtab *rhtab, | |
| 2869 | + struct rhtab_elem *elem) | |
| 2870 | +{ | |
| 2871 | + if (IS_ERR_OR_NULL(rhtab->map.record)) | |
| 2872 | + return; | |
| 2873 | + | |
| 2874 | + bpf_obj_free_fields(rhtab->map.record, | |
| 2875 | + rhtab_elem_value(elem, rhtab->map.key_size)); | |
| 2876 | +} | |
| 2877 | + | |
| 2878 | +static void rhtab_mem_dtor(void *obj, void *ctx) | |
| 2879 | +{ | |
| 2880 | + struct htab_btf_record *hrec = ctx; | |
| 2881 | + struct rhtab_elem *elem = obj; | |
| 2882 | + | |
| 2883 | + if (IS_ERR_OR_NULL(hrec->record)) | |
| 2884 | + return; | |
| 2885 | + | |
| 2886 | + bpf_obj_free_fields(hrec->record, | |
| 2887 | + rhtab_elem_value(elem, hrec->key_size)); | |
| 2888 | +} | |
| 2889 | + | |
| 2890 | +static void rhtab_free_elem(void *ptr, void *arg) | |
| 2891 | +{ | |
| 2892 | + struct bpf_rhtab *rhtab = arg; | |
| 2893 | + struct rhtab_elem *elem = ptr; | |
| 2894 | + | |
| 2895 | + bpf_map_free_internal_structs(&rhtab->map, rhtab_elem_value(elem, rhtab->map.key_size)); | |
| 2896 | + bpf_mem_cache_free_rcu(&rhtab->ma, elem); | |
| 2897 | +} | |
| 2898 | + | |
| 2899 | +static void rhtab_map_free(struct bpf_map *map) | |
| 2900 | +{ | |
| 2901 | + struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map); | |
| 2902 | + | |
| 2903 | + rhashtable_free_and_destroy(&rhtab->ht, rhtab_free_elem, rhtab); | |
| 2904 | + bpf_mem_alloc_destroy(&rhtab->ma); | |
| 2905 | + bpf_map_area_free(rhtab); | |
| 2906 | +} | |
| 2907 | + | |
| 2908 | +static void *rhtab_lookup_elem(struct bpf_map *map, void *key) | |
| 2909 | +{ | |
| 2910 | + struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map); | |
| 2911 | + | |
| 2912 | + /* Hold RCU lock in case sleepable program calls via gen_lookup */ | |
| 2913 | + guard(rcu)(); | |
| 2914 | + | |
| 2915 | + if (map->key_size == sizeof(long)) | |
| 2916 | + return rhashtable_lookup_likely(&rhtab->ht, key, rhtab_params_long); | |
| 2917 | + | |
| 2918 | + return rhashtable_lookup_likely(&rhtab->ht, key, rhtab_params); | |
| 2919 | +} | |
| 2920 | + | |
| 2921 | +static void *rhtab_map_lookup_elem(struct bpf_map *map, void *key) __must_hold(RCU) | |
| 2922 | +{ | |
| 2923 | + struct rhtab_elem *l; | |
| 2924 | + | |
| 2925 | + l = rhtab_lookup_elem(map, key); | |
| 2926 | + return l ? rhtab_elem_value(l, map->key_size) : NULL; | |
| 2927 | +} | |
| 2928 | + | |
| 2929 | +static void rhtab_read_elem_value(struct bpf_map *map, void *dst, struct rhtab_elem *elem, | |
| 2930 | + u64 flags) | |
| 2931 | +{ | |
| 2932 | + void *src = rhtab_elem_value(elem, map->key_size); | |
| 2933 | + | |
| 2934 | + if (flags & BPF_F_LOCK) | |
| 2935 | + copy_map_value_locked(map, dst, src, true); | |
| 2936 | + else | |
| 2937 | + copy_map_value(map, dst, src); | |
| 2938 | +} | |
| 2939 | + | |
| 2940 | +static int rhtab_delete_elem(struct bpf_rhtab *rhtab, struct rhtab_elem *elem, void *copy, | |
| 2941 | + u64 flags) | |
| 2942 | +{ | |
| 2943 | + int err; | |
| 2944 | + | |
| 2945 | + /* | |
| 2946 | + * disable_instrumentation() mitigates the deadlock for programs running in NMI context. | |
| 2947 | + * rhashtable locks bucket with local_irq_save(). Only NMI programs may reenter | |
| 2948 | + * rhashtable code, bpf_disable_instrumentation() disables programs running in NMI, except | |
| 2949 | + * raw tracepoints, which we don't have in rhashtable. | |
| 2950 | + */ | |
| 2951 | + bpf_disable_instrumentation(); | |
| 2952 | + | |
| 2953 | + if (rhtab->map.key_size == sizeof(long)) | |
| 2954 | + err = rhashtable_remove_fast(&rhtab->ht, &elem->node, rhtab_params_long); | |
| 2955 | + else | |
| 2956 | + err = rhashtable_remove_fast(&rhtab->ht, &elem->node, rhtab_params); | |
| 2957 | + | |
| 2958 | + bpf_enable_instrumentation(); | |
| 2959 | + | |
| 2960 | + if (err) | |
| 2961 | + return err; | |
| 2962 | + | |
| 2963 | + if (copy) { | |
| 2964 | + rhtab_read_elem_value(&rhtab->map, copy, elem, flags); | |
| 2965 | + check_and_init_map_value(&rhtab->map, copy); | |
| 2966 | + } | |
| 2967 | + /* Release internal structs: kptr, bpf_timer, task_work, wq */ | |
| 2968 | + rhtab_check_and_free_fields(rhtab, elem); | |
| 2969 | + bpf_mem_cache_free_rcu(&rhtab->ma, elem); | |
| 2970 | + return 0; | |
| 2971 | +} | |
| 2972 | + | |
| 2973 | + | |
| 2974 | +static long rhtab_map_delete_elem(struct bpf_map *map, void *key) | |
| 2975 | +{ | |
| 2976 | + struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map); | |
| 2977 | + struct rhtab_elem *elem; | |
| 2978 | + | |
| 2979 | + guard(rcu)(); | |
| 2980 | + | |
| 2981 | + elem = rhtab_lookup_elem(map, key); | |
| 2982 | + if (!elem) | |
| 2983 | + return -ENOENT; | |
| 2984 | + | |
| 2985 | + return rhtab_delete_elem(rhtab, elem, NULL, 0); | |
| 2986 | +} | |
| 2987 | + | |
| 2988 | +static int rhtab_map_lookup_and_delete_elem(struct bpf_map *map, void *key, void *value, u64 flags) | |
| 2989 | +{ | |
| 2990 | + struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map); | |
| 2991 | + struct rhtab_elem *elem; | |
| 2992 | + int err; | |
| 2993 | + | |
| 2994 | + err = bpf_map_check_op_flags(map, flags, BPF_F_LOCK); | |
| 2995 | + if (err) | |
| 2996 | + return err; | |
| 2997 | + | |
| 2998 | + guard(rcu)(); | |
| 2999 | + | |
| 3000 | + elem = rhtab_lookup_elem(map, key); | |
| 3001 | + if (!elem) | |
| 3002 | + return -ENOENT; | |
| 3003 | + | |
| 3004 | + return rhtab_delete_elem(rhtab, elem, value, flags); | |
| 3005 | +} | |
| 3006 | + | |
| 3007 | +static long rhtab_map_update_existing(struct bpf_map *map, struct rhtab_elem *elem, void *value, | |
| 3008 | + u64 map_flags) | |
| 3009 | +{ | |
| 3010 | + struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map); | |
| 3011 | + void *old_val = rhtab_elem_value(elem, map->key_size); | |
| 3012 | + | |
| 3013 | + if (map_flags & BPF_NOEXIST) | |
| 3014 | + return -EEXIST; | |
| 3015 | + | |
| 3016 | + if (map_flags & BPF_F_LOCK) | |
| 3017 | + copy_map_value_locked(map, old_val, value, false); | |
| 3018 | + else | |
| 3019 | + copy_map_value(map, old_val, value); | |
| 3020 | + | |
| 3021 | + /* | |
| 3022 | + * Torn reads: a concurrent reader without BPF_F_LOCK may observe | |
| 3023 | + * the value mid-copy. Callers requiring consistent reads must use | |
| 3024 | + * BPF_F_LOCK, matching arraymap semantics. | |
| 3025 | + * | |
| 3026 | + * copy_map_value() skips special-field offsets, so old timers/ | |
| 3027 | + * kptrs/etc. still sit in the slot. Cancel them after the copy | |
| 3028 | + * to match arraymap's update semantics. | |
| 3029 | + */ | |
| 3030 | + rhtab_check_and_free_fields(rhtab, elem); | |
| 3031 | + return 0; | |
| 3032 | +} | |
| 3033 | + | |
| 3034 | +static long rhtab_map_update_elem(struct bpf_map *map, void *key, void *value, u64 map_flags) | |
| 3035 | +{ | |
| 3036 | + struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map); | |
| 3037 | + struct rhtab_elem *elem, *tmp; | |
| 3038 | + | |
| 3039 | + if (unlikely((map_flags & ~BPF_F_LOCK) > BPF_EXIST)) | |
| 3040 | + return -EINVAL; | |
| 3041 | + | |
| 3042 | + if ((map_flags & BPF_F_LOCK) && !btf_record_has_field(map->record, BPF_SPIN_LOCK)) | |
| 3043 | + return -EINVAL; | |
| 3044 | + | |
| 3045 | + guard(rcu)(); | |
| 3046 | + elem = rhtab_lookup_elem(map, key); | |
| 3047 | + if (elem) | |
| 3048 | + return rhtab_map_update_existing(map, elem, value, map_flags); | |
| 3049 | + | |
| 3050 | + if (map_flags & BPF_EXIST) | |
| 3051 | + return -ENOENT; | |
| 3052 | + | |
| 3053 | + /* | |
| 3054 | + * Reject new insertions while map_release_uref cleanup walks the | |
| 3055 | + * table. Without this, new elements could keep triggering rehash | |
| 3056 | + * and prevent the walk from terminating. | |
| 3057 | + */ | |
| 3058 | + if (READ_ONCE(rhtab->freeing_internal)) | |
| 3059 | + return -EBUSY; | |
| 3060 | + | |
| 3061 | + /* Check max_entries limit before inserting new element */ | |
| 3062 | + if (atomic_read(&rhtab->ht.nelems) >= map->max_entries) | |
| 3063 | + return -E2BIG; | |
| 3064 | + | |
| 3065 | + elem = bpf_mem_cache_alloc(&rhtab->ma); | |
| 3066 | + if (!elem) | |
| 3067 | + return -ENOMEM; | |
| 3068 | + | |
| 3069 | + memcpy(elem->data, key, map->key_size); | |
| 3070 | + copy_map_value(map, rhtab_elem_value(elem, map->key_size), value); | |
| 3071 | + check_and_init_map_value(map, rhtab_elem_value(elem, map->key_size)); | |
| 3072 | + | |
| 3073 | + /* Prevent deadlock for NMI programs attempting to take bucket lock */ | |
| 3074 | + bpf_disable_instrumentation(); | |
| 3075 | + | |
| 3076 | + if (map->key_size == sizeof(long)) | |
| 3077 | + tmp = rhashtable_lookup_get_insert_fast(&rhtab->ht, &elem->node, rhtab_params_long); | |
| 3078 | + else | |
| 3079 | + tmp = rhashtable_lookup_get_insert_fast(&rhtab->ht, &elem->node, rhtab_params); | |
| 3080 | + | |
| 3081 | + bpf_enable_instrumentation(); | |
| 3082 | + | |
| 3083 | + if (tmp) { | |
| 3084 | + bpf_mem_cache_free(&rhtab->ma, elem); | |
| 3085 | + if (IS_ERR(tmp)) | |
| 3086 | + return PTR_ERR(tmp); | |
| 3087 | + | |
| 3088 | + return rhtab_map_update_existing(map, tmp, value, map_flags); | |
| 3089 | + } | |
| 3090 | + | |
| 3091 | + return 0; | |
| 3092 | +} | |
| 3093 | + | |
| 3094 | +static int rhtab_map_gen_lookup(struct bpf_map *map, struct bpf_insn *insn_buf) | |
| 3095 | +{ | |
| 3096 | + struct bpf_insn *insn = insn_buf; | |
| 3097 | + const int ret = BPF_REG_0; | |
| 3098 | + | |
| 3099 | + BUILD_BUG_ON(!__same_type(&rhtab_lookup_elem, | |
| 3100 | + (void *(*)(struct bpf_map *map, void *key)) NULL)); | |
| 3101 | + *insn++ = BPF_EMIT_CALL(rhtab_lookup_elem); | |
| 3102 | + *insn++ = BPF_JMP_IMM(BPF_JEQ, ret, 0, 1); | |
| 3103 | + *insn++ = BPF_ALU64_IMM(BPF_ADD, ret, | |
| 3104 | + offsetof(struct rhtab_elem, data) + round_up(map->key_size, 8)); | |
| 3105 | + | |
| 3106 | + return insn - insn_buf; | |
| 3107 | +} | |
| 3108 | + | |
| 3109 | +static int rhtab_map_check_btf(struct bpf_map *map, const struct btf *btf, | |
| 3110 | + const struct btf_type *key_type, | |
| 3111 | + const struct btf_type *value_type) | |
| 3112 | +{ | |
| 3113 | + struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map); | |
| 3114 | + | |
| 3115 | + return bpf_ma_set_dtor(map, &rhtab->ma, rhtab_mem_dtor); | |
| 3116 | +} | |
| 3117 | + | |
| 3118 | +static void rhtab_map_free_internal_structs(struct bpf_map *map) | |
| 3119 | +{ | |
| 3120 | + struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map); | |
| 3121 | + struct rhashtable_iter iter; | |
| 3122 | + struct rhtab_elem *elem; | |
| 3123 | + | |
| 3124 | + if (!bpf_map_has_internal_structs(map)) | |
| 3125 | + return; | |
| 3126 | + | |
| 3127 | + /* | |
| 3128 | + * Block new insertions. Once observed, no new growth is triggered, | |
| 3129 | + * so any in-flight rehash will drain and the walker is guaranteed | |
| 3130 | + * to stop returning -EAGAIN. Treat -EAGAIN as "rehash in progress, | |
| 3131 | + * retry"; do not wait for the worker. | |
| 3132 | + */ | |
| 3133 | + WRITE_ONCE(rhtab->freeing_internal, true); | |
| 3134 | + | |
| 3135 | + rhashtable_walk_enter(&rhtab->ht, &iter); | |
| 3136 | + rhashtable_walk_start(&iter); | |
| 3137 | + | |
| 3138 | + while ((elem = rhashtable_walk_next(&iter))) { | |
| 3139 | + if (IS_ERR(elem)) { | |
| 3140 | + if (PTR_ERR(elem) == -EAGAIN) | |
| 3141 | + continue; | |
| 3142 | + break; | |
| 3143 | + } | |
| 3144 | + | |
| 3145 | + bpf_map_free_internal_structs(map, rhtab_elem_value(elem, map->key_size)); | |
| 3146 | + | |
| 3147 | + if (need_resched()) { /* Avoid stalls on large maps */ | |
| 3148 | + rhashtable_walk_stop(&iter); | |
| 3149 | + cond_resched(); | |
| 3150 | + rhashtable_walk_start(&iter); | |
| 3151 | + } | |
| 3152 | + } | |
| 3153 | + | |
| 3154 | + rhashtable_walk_stop(&iter); | |
| 3155 | + rhashtable_walk_exit(&iter); | |
| 3156 | + WRITE_ONCE(rhtab->freeing_internal, false); | |
| 3157 | +} | |
| 3158 | + | |
| 3159 | +static int rhtab_map_get_next_key(struct bpf_map *map, void *key, void *next_key) | |
| 3160 | + __must_hold_shared(RCU) | |
| 3161 | +{ | |
| 3162 | + struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map); | |
| 3163 | + struct rhtab_elem *elem; | |
| 3164 | + | |
| 3165 | + elem = rhashtable_next_key(&rhtab->ht, key); | |
| 3166 | + | |
| 3167 | + /* if not found, return the first key */ | |
| 3168 | + if (PTR_ERR(elem) == -ENOENT) | |
| 3169 | + elem = rhashtable_next_key(&rhtab->ht, NULL); | |
| 3170 | + | |
| 3171 | + if (IS_ERR(elem)) | |
| 3172 | + return PTR_ERR(elem); | |
| 3173 | + if (!elem) | |
| 3174 | + return -ENOENT; | |
| 3175 | + | |
| 3176 | + memcpy(next_key, elem->data, map->key_size); | |
| 3177 | + return 0; | |
| 3178 | +} | |
| 3179 | + | |
| 3180 | +static void rhtab_map_seq_show_elem(struct bpf_map *map, void *key, struct seq_file *m) | |
| 3181 | +{ | |
| 3182 | + void *value; | |
| 3183 | + | |
| 3184 | + /* Guarantee that hashtab value is not freed */ | |
| 3185 | + guard(rcu)(); | |
| 3186 | + | |
| 3187 | + value = rhtab_map_lookup_elem(map, key); | |
| 3188 | + if (!value) | |
| 3189 | + return; | |
| 3190 | + | |
| 3191 | + btf_type_seq_show(map->btf, map->btf_key_type_id, key, m); | |
| 3192 | + seq_puts(m, ": "); | |
| 3193 | + btf_type_seq_show(map->btf, map->btf_value_type_id, value, m); | |
| 3194 | + seq_putc(m, '\n'); | |
| 3195 | +} | |
| 3196 | + | |
| 3197 | +static long bpf_each_rhash_elem(struct bpf_map *map, bpf_callback_t callback_fn, | |
| 3198 | + void *callback_ctx, u64 flags) | |
| 3199 | +{ | |
| 3200 | + struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map); | |
| 3201 | + void *prev_key = NULL; | |
| 3202 | + struct rhtab_elem *elem; | |
| 3203 | + int num_elems = 0; | |
| 3204 | + u64 ret = 0; | |
| 3205 | + | |
| 3206 | + cant_migrate(); | |
| 3207 | + | |
| 3208 | + if (flags != 0) | |
| 3209 | + return -EINVAL; | |
| 3210 | + | |
| 3211 | + rcu_read_lock(); | |
| 3212 | + /* | |
| 3213 | + * Best-effort iteration: if rhashtable is concurrently resized or | |
| 3214 | + * elements are deleted/inserted, there may be missed or duplicate | |
| 3215 | + * elements visited. | |
| 3216 | + */ | |
| 3217 | + while ((elem = rhashtable_next_key(&rhtab->ht, prev_key))) { | |
| 3218 | + if (IS_ERR(elem)) | |
| 3219 | + break; | |
| 3220 | + num_elems++; | |
| 3221 | + ret = callback_fn((u64)(long)map, | |
| 3222 | + (u64)(long)elem->data, | |
| 3223 | + (u64)(long)rhtab_elem_value(elem, map->key_size), | |
| 3224 | + (u64)(long)callback_ctx, 0); | |
| 3225 | + if (ret) | |
| 3226 | + break; | |
| 3227 | + | |
| 3228 | + prev_key = elem->data; /* valid while RCU held */ | |
| 3229 | + } | |
| 3230 | + rcu_read_unlock(); | |
| 3231 | + | |
| 3232 | + return num_elems; | |
| 3233 | +} | |
| 3234 | + | |
| 3235 | +static u64 rhtab_map_mem_usage(const struct bpf_map *map) | |
| 3236 | +{ | |
| 3237 | + struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map); | |
| 3238 | + u64 num_entries; | |
| 3239 | + | |
| 3240 | + /* Excludes rhashtable bucket overhead (~ nelems * sizeof(void *) at 75% load). */ | |
| 3241 | + num_entries = atomic_read(&rhtab->ht.nelems); | |
| 3242 | + return sizeof(struct bpf_rhtab) + rhtab->elem_size * num_entries; | |
| 3243 | +} | |
| 3244 | + | |
| 3245 | +static int __rhtab_map_lookup_and_delete_batch(struct bpf_map *map, | |
| 3246 | + const union bpf_attr *attr, | |
| 3247 | + union bpf_attr __user *uattr, | |
| 3248 | + bool do_delete) | |
| 3249 | +{ | |
| 3250 | + struct bpf_rhtab *rhtab = container_of(map, struct bpf_rhtab, map); | |
| 3251 | + void __user *uvalues = u64_to_user_ptr(attr->batch.values); | |
| 3252 | + void __user *ukeys = u64_to_user_ptr(attr->batch.keys); | |
| 3253 | + void __user *ubatch = u64_to_user_ptr(attr->batch.in_batch); | |
| 3254 | + void *cursor = NULL, *keys = NULL, *values = NULL, *dst_key, *dst_val; | |
| 3255 | + struct rhtab_elem **del_elems = NULL; | |
| 3256 | + u32 max_count, total, key_size, value_size, i; | |
| 3257 | + bool has_next_cursor = false; | |
| 3258 | + struct rhtab_elem *elem; | |
| 3259 | + u64 elem_map_flags, map_flags; | |
| 3260 | + int ret = 0; | |
| 3261 | + | |
| 3262 | + elem_map_flags = attr->batch.elem_flags; | |
| 3263 | + ret = bpf_map_check_op_flags(map, elem_map_flags, BPF_F_LOCK); | |
| 3264 | + if (ret) | |
| 3265 | + return ret; | |
| 3266 | + | |
| 3267 | + map_flags = attr->batch.flags; | |
| 3268 | + if (map_flags) | |
| 3269 | + return -EINVAL; | |
| 3270 | + | |
| 3271 | + max_count = attr->batch.count; | |
| 3272 | + if (!max_count) | |
| 3273 | + return 0; | |
| 3274 | + | |
| 3275 | + if (put_user(0, &uattr->batch.count)) | |
| 3276 | + return -EFAULT; | |
| 3277 | + | |
| 3278 | + key_size = map->key_size; | |
| 3279 | + value_size = map->value_size; | |
| 3280 | + | |
| 3281 | + keys = kvmalloc_array(max_count, key_size, GFP_USER | __GFP_NOWARN); | |
| 3282 | + values = kvmalloc_array(max_count, value_size, GFP_USER | __GFP_NOWARN); | |
| 3283 | + if (do_delete) | |
| 3284 | + del_elems = kvmalloc_array(max_count, sizeof(void *), | |
| 3285 | + GFP_USER | __GFP_NOWARN); | |
| 3286 | + cursor = kmalloc(key_size, GFP_USER | __GFP_NOWARN); | |
| 3287 | + | |
| 3288 | + if (!keys || !values || !cursor || (do_delete && !del_elems)) { | |
| 3289 | + ret = -ENOMEM; | |
| 3290 | + goto free; | |
| 3291 | + } | |
| 3292 | + | |
| 3293 | + if (ubatch && copy_from_user(cursor, ubatch, key_size)) { | |
| 3294 | + ret = -EFAULT; | |
| 3295 | + goto free; | |
| 3296 | + } | |
| 3297 | + | |
| 3298 | + dst_key = keys; | |
| 3299 | + dst_val = values; | |
| 3300 | + total = 0; | |
| 3301 | + | |
| 3302 | + rcu_read_lock(); | |
| 3303 | + | |
| 3304 | + /* | |
| 3305 | + * Cursor stores the key of the next-to-process element (stashed by | |
| 3306 | + * the previous batch). Look it up directly so the element is included | |
| 3307 | + * here rather than skipped by next_key(). If the cursor was deleted | |
| 3308 | + * concurrently (or by the previous do_delete batch), return -EAGAIN | |
| 3309 | + * so userspace can distinguish a lost cursor from end-of-iteration | |
| 3310 | + * (-ENOENT) and restart from a NULL cursor. | |
| 3311 | + */ | |
| 3312 | + if (ubatch) { | |
| 3313 | + elem = rhtab_lookup_elem(map, cursor); | |
| 3314 | + if (!elem) { | |
| 3315 | + rcu_read_unlock(); | |
| 3316 | + ret = -EAGAIN; | |
| 3317 | + goto free; | |
| 3318 | + } | |
| 3319 | + } else { | |
| 3320 | + elem = rhashtable_next_key(&rhtab->ht, NULL); | |
| 3321 | + } | |
| 3322 | + | |
| 3323 | + while (elem && !IS_ERR(elem) && total < max_count) { | |
| 3324 | + memcpy(dst_key, elem->data, key_size); | |
| 3325 | + rhtab_read_elem_value(map, dst_val, elem, elem_map_flags); | |
| 3326 | + check_and_init_map_value(map, dst_val); | |
| 3327 | + | |
| 3328 | + if (do_delete) | |
| 3329 | + del_elems[total] = elem; | |
| 3330 | + | |
| 3331 | + elem = rhashtable_next_key(&rhtab->ht, dst_key); | |
| 3332 | + dst_key += key_size; | |
| 3333 | + dst_val += value_size; | |
| 3334 | + total++; | |
| 3335 | + | |
| 3336 | + /* Bail to userspace to avoid stalls. */ | |
| 3337 | + if (need_resched()) | |
| 3338 | + break; | |
| 3339 | + } | |
| 3340 | + | |
| 3341 | + if (elem && !IS_ERR(elem)) { | |
| 3342 | + /* Stash next-to-process key as cursor for the next batch. */ | |
| 3343 | + memcpy(cursor, elem->data, key_size); | |
| 3344 | + has_next_cursor = true; | |
| 3345 | + } | |
| 3346 | + | |
| 3347 | + if (do_delete) { | |
| 3348 | + for (i = 0; i < total; i++) | |
| 3349 | + rhtab_delete_elem(rhtab, del_elems[i], NULL, 0); | |
| 3350 | + } | |
| 3351 | + | |
| 3352 | + rcu_read_unlock(); | |
| 3353 | + | |
| 3354 | + if (total == 0) { | |
| 3355 | + ret = -ENOENT; | |
| 3356 | + goto free; | |
| 3357 | + } | |
| 3358 | + | |
| 3359 | + /* No more elements after this batch. */ | |
| 3360 | + if (!has_next_cursor) | |
| 3361 | + ret = -ENOENT; | |
| 3362 | + | |
| 3363 | + if (copy_to_user(ukeys, keys, (size_t)total * key_size) || | |
| 3364 | + copy_to_user(uvalues, values, (size_t)total * value_size) || | |
| 3365 | + put_user(total, &uattr->batch.count) || | |
| 3366 | + (has_next_cursor && | |
| 3367 | + copy_to_user(u64_to_user_ptr(attr->batch.out_batch), | |
| 3368 | + cursor, key_size))) { | |
| 3369 | + ret = -EFAULT; | |
| 3370 | + goto free; | |
| 3371 | + } | |
| 3372 | + | |
| 3373 | +free: | |
| 3374 | + kfree(cursor); | |
| 3375 | + kvfree(keys); | |
| 3376 | + kvfree(values); | |
| 3377 | + kvfree(del_elems); | |
| 3378 | + return ret; | |
| 3379 | +} | |
| 3380 | + | |
| 3381 | +static int rhtab_map_lookup_batch(struct bpf_map *map, const union bpf_attr *attr, | |
| 3382 | + union bpf_attr __user *uattr) | |
| 3383 | +{ | |
| 3384 | + return __rhtab_map_lookup_and_delete_batch(map, attr, uattr, false); | |
| 3385 | +} | |
| 3386 | + | |
| 3387 | +static int rhtab_map_lookup_and_delete_batch(struct bpf_map *map, const union bpf_attr *attr, | |
| 3388 | + union bpf_attr __user *uattr) | |
| 3389 | +{ | |
| 3390 | + return __rhtab_map_lookup_and_delete_batch(map, attr, uattr, true); | |
| 3391 | +} | |
| 3392 | + | |
| 3393 | +struct bpf_iter_seq_rhash_map_info { | |
| 3394 | + struct bpf_map *map; | |
| 3395 | + struct bpf_rhtab *rhtab; | |
| 3396 | + struct rhashtable_iter iter; | |
| 3397 | +}; | |
| 3398 | + | |
| 3399 | +static void *bpf_rhash_map_seq_start(struct seq_file *seq, loff_t *pos) | |
| 3400 | + __acquires(RCU) | |
| 3401 | +{ | |
| 3402 | + struct bpf_iter_seq_rhash_map_info *info = seq->private; | |
| 3403 | + struct rhtab_elem *elem; | |
| 3404 | + | |
| 3405 | + rhashtable_walk_start(&info->iter); | |
| 3406 | + /* | |
| 3407 | + * Re-deliver the element returned by walk_next() at the end of the | |
| 3408 | + * previous read() — bpf_seq_read may have stopped before show() | |
| 3409 | + * consumed it. Rehash rewinds the walker; retry on -EAGAIN. | |
| 3410 | + */ | |
| 3411 | + do { | |
| 3412 | + elem = rhashtable_walk_peek(&info->iter); | |
| 3413 | + } while (PTR_ERR(elem) == -EAGAIN); | |
| 3414 | + | |
| 3415 | + if (IS_ERR(elem)) | |
| 3416 | + return NULL; | |
| 3417 | + | |
| 3418 | + if (elem && *pos == 0) | |
| 3419 | + ++*pos; | |
| 3420 | + return elem; | |
| 3421 | +} | |
| 3422 | + | |
| 3423 | +static void *bpf_rhash_map_seq_next(struct seq_file *seq, void *v, loff_t *pos) | |
| 3424 | +{ | |
| 3425 | + struct bpf_iter_seq_rhash_map_info *info = seq->private; | |
| 3426 | + struct rhtab_elem *elem; | |
| 3427 | + | |
| 3428 | + ++*pos; | |
| 3429 | + | |
| 3430 | + /* Rehash rewinds the walker; retry until it stops returning -EAGAIN. */ | |
| 3431 | + do { | |
| 3432 | + elem = rhashtable_walk_next(&info->iter); | |
| 3433 | + } while (PTR_ERR(elem) == -EAGAIN); | |
| 3434 | + | |
| 3435 | + if (IS_ERR(elem)) | |
| 3436 | + return NULL; | |
| 3437 | + return elem; | |
| 3438 | +} | |
| 3439 | + | |
| 3440 | +static int __bpf_rhash_map_seq_show(struct seq_file *seq, | |
| 3441 | + struct rhtab_elem *elem) | |
| 3442 | +{ | |
| 3443 | + struct bpf_iter_seq_rhash_map_info *info = seq->private; | |
| 3444 | + struct bpf_iter__bpf_map_elem ctx = {}; | |
| 3445 | + struct bpf_iter_meta meta; | |
| 3446 | + struct bpf_prog *prog; | |
| 3447 | + int ret = 0; | |
| 3448 | + | |
| 3449 | + meta.seq = seq; | |
| 3450 | + prog = bpf_iter_get_info(&meta, elem == NULL); | |
| 3451 | + if (prog) { | |
| 3452 | + ctx.meta = &meta; | |
| 3453 | + ctx.map = info->map; | |
| 3454 | + if (elem) { | |
| 3455 | + ctx.key = elem->data; | |
| 3456 | + ctx.value = rhtab_elem_value(elem, info->map->key_size); | |
| 3457 | + } | |
| 3458 | + ret = bpf_iter_run_prog(prog, &ctx); | |
| 3459 | + } | |
| 3460 | + | |
| 3461 | + return ret; | |
| 3462 | +} | |
| 3463 | + | |
| 3464 | +static int bpf_rhash_map_seq_show(struct seq_file *seq, void *v) | |
| 3465 | +{ | |
| 3466 | + return __bpf_rhash_map_seq_show(seq, v); | |
| 3467 | +} | |
| 3468 | + | |
| 3469 | +static void bpf_rhash_map_seq_stop(struct seq_file *seq, void *v) | |
| 3470 | + __releases(RCU) | |
| 3471 | +{ | |
| 3472 | + struct bpf_iter_seq_rhash_map_info *info = seq->private; | |
| 3473 | + | |
| 3474 | + if (!v) | |
| 3475 | + (void)__bpf_rhash_map_seq_show(seq, NULL); | |
| 3476 | + | |
| 3477 | + rhashtable_walk_stop(&info->iter); | |
| 3478 | +} | |
| 3479 | + | |
| 3480 | +static int bpf_iter_init_rhash_map(void *priv_data, struct bpf_iter_aux_info *aux) | |
| 3481 | +{ | |
| 3482 | + struct bpf_iter_seq_rhash_map_info *info = priv_data; | |
| 3483 | + struct bpf_map *map = aux->map; | |
| 3484 | + | |
| 3485 | + bpf_map_inc_with_uref(map); | |
| 3486 | + info->map = map; | |
| 3487 | + info->rhtab = container_of(map, struct bpf_rhtab, map); | |
| 3488 | + rhashtable_walk_enter(&info->rhtab->ht, &info->iter); | |
| 3489 | + return 0; | |
| 3490 | +} | |
| 3491 | + | |
| 3492 | +static void bpf_iter_fini_rhash_map(void *priv_data) | |
| 3493 | +{ | |
| 3494 | + struct bpf_iter_seq_rhash_map_info *info = priv_data; | |
| 3495 | + | |
| 3496 | + rhashtable_walk_exit(&info->iter); | |
| 3497 | + bpf_map_put_with_uref(info->map); | |
| 3498 | +} | |
| 3499 | + | |
| 3500 | +static const struct seq_operations bpf_rhash_map_seq_ops = { | |
| 3501 | + .start = bpf_rhash_map_seq_start, | |
| 3502 | + .next = bpf_rhash_map_seq_next, | |
| 3503 | + .stop = bpf_rhash_map_seq_stop, | |
| 3504 | + .show = bpf_rhash_map_seq_show, | |
| 3505 | +}; | |
| 3506 | + | |
| 3507 | +static const struct bpf_iter_seq_info rhash_iter_seq_info = { | |
| 3508 | + .seq_ops = &bpf_rhash_map_seq_ops, | |
| 3509 | + .init_seq_private = bpf_iter_init_rhash_map, | |
| 3510 | + .fini_seq_private = bpf_iter_fini_rhash_map, | |
| 3511 | + .seq_priv_size = sizeof(struct bpf_iter_seq_rhash_map_info), | |
| 3512 | +}; | |
| 3513 | + | |
| 3514 | +BTF_ID_LIST_SINGLE(rhtab_map_btf_ids, struct, bpf_rhtab) | |
| 3515 | +const struct bpf_map_ops rhtab_map_ops = { | |
| 3516 | + .map_meta_equal = bpf_map_meta_equal, | |
| 3517 | + .map_alloc_check = rhtab_map_alloc_check, | |
| 3518 | + .map_alloc = rhtab_map_alloc, | |
| 3519 | + .map_free = rhtab_map_free, | |
| 3520 | + .map_get_next_key = rhtab_map_get_next_key, | |
| 3521 | + .map_release_uref = rhtab_map_free_internal_structs, | |
| 3522 | + .map_check_btf = rhtab_map_check_btf, | |
| 3523 | + .map_lookup_elem = rhtab_map_lookup_elem, | |
| 3524 | + .map_lookup_and_delete_elem = rhtab_map_lookup_and_delete_elem, | |
| 3525 | + .map_update_elem = rhtab_map_update_elem, | |
| 3526 | + .map_delete_elem = rhtab_map_delete_elem, | |
| 3527 | + .map_gen_lookup = rhtab_map_gen_lookup, | |
| 3528 | + .map_seq_show_elem = rhtab_map_seq_show_elem, | |
| 3529 | + .map_set_for_each_callback_args = map_set_for_each_callback_args, | |
| 3530 | + .map_for_each_callback = bpf_each_rhash_elem, | |
| 3531 | + .map_mem_usage = rhtab_map_mem_usage, | |
| 3532 | + BATCH_OPS(rhtab), | |
| 3533 | + .map_btf_id = &rhtab_map_btf_ids[0], | |
| 3534 | + .iter_seq_info = &rhash_iter_seq_info, | |
| 3535 | +}; | |
| @@ -1944,7 +1944,7 @@ static const struct bpf_func_proto bpf_dynptr_read_proto = { · 1943 unmodified lines | ||
| 1944 | 1944 | .ret_type = RET_INTEGER, |
| 1945 | 1945 | .arg1_type = ARG_PTR_TO_UNINIT_MEM, |
| 1946 | 1946 | .arg2_type = ARG_CONST_SIZE_OR_ZERO, |
| 1947 | − .arg3_type = ARG_PTR_TO_DYNPTR | MEM_RDONLY, | |
| 1947 | + .arg3_type = ARG_PTR_TO_DYNPTR, | |
| 1948 | 1948 | .arg4_type = ARG_ANYTHING, |
| 1949 | 1949 | .arg5_type = ARG_ANYTHING, |
| 1950 | 1950 | }; |
| @@ -2001,7 +2001,7 @@ static const struct bpf_func_proto bpf_dynptr_write_proto = { · 50 unmodified lines | ||
| 2001 | 2001 | .func = bpf_dynptr_write, |
| 2002 | 2002 | .gpl_only = false, |
| 2003 | 2003 | .ret_type = RET_INTEGER, |
| 2004 | − .arg1_type = ARG_PTR_TO_DYNPTR | MEM_RDONLY, | |
| 2004 | + .arg1_type = ARG_PTR_TO_DYNPTR, | |
| 2005 | 2005 | .arg2_type = ARG_ANYTHING, |
| 2006 | 2006 | .arg3_type = ARG_PTR_TO_MEM | MEM_RDONLY, |
| 2007 | 2007 | .arg4_type = ARG_CONST_SIZE_OR_ZERO, |
| @@ -2044,7 +2044,7 @@ static const struct bpf_func_proto bpf_dynptr_data_proto = { · 36 unmodified lines | ||
| 2044 | 2044 | .func = bpf_dynptr_data, |
| 2045 | 2045 | .gpl_only = false, |
| 2046 | 2046 | .ret_type = RET_PTR_TO_DYNPTR_MEM_OR_NULL, |
| 2047 | − .arg1_type = ARG_PTR_TO_DYNPTR | MEM_RDONLY, | |
| 2047 | + .arg1_type = ARG_PTR_TO_DYNPTR, | |
| 2048 | 2048 | .arg2_type = ARG_ANYTHING, |
| 2049 | 2049 | .arg3_type = ARG_CONST_ALLOC_SIZE_OR_ZERO, |
| 2050 | 2050 | }; |
| @@ -2247,10 +2247,11 @@ EXPORT_SYMBOL_GPL(bpf_base_func_proto); · 196 unmodified lines | ||
| 2247 | 2247 | void bpf_list_head_free(const struct btf_field *field, void *list_head, |
| 2248 | 2248 | struct bpf_spin_lock *spin_lock) |
| 2249 | 2249 | { |
| 2250 | − struct list_head *head = list_head, *orig_head = list_head; | |
| 2250 | + struct list_head *head = list_head, drain, *pos, *n; | |
| 2251 | 2251 | |
| 2252 | 2252 | BUILD_BUG_ON(sizeof(struct list_head) > sizeof(struct bpf_list_head)); |
| 2253 | 2253 | BUILD_BUG_ON(__alignof__(struct list_head) > __alignof__(struct bpf_list_head)); |
| 2254 | + INIT_LIST_HEAD(&drain); | |
| 2254 | 2255 | |
| 2255 | 2256 | /* Do the actual list draining outside the lock to not hold the lock for |
| 2256 | 2257 | * too long, and also prevent deadlocks if tracing programs end up |
| @@ -2261,20 +2262,30 @@ void bpf_list_head_free(const struct btf_field *field, void *list_head, · 4 unmodified lines | ||
| 2261 | 2262 | __bpf_spin_lock_irqsave(spin_lock); |
| 2262 | 2263 | if (!head->next || list_empty(head)) |
| 2263 | 2264 | goto unlock; |
| 2264 | − head = head->next; | |
| 2265 | + list_for_each_safe(pos, n, head) { | |
| 2266 | + struct bpf_list_node_kern *node; | |
| 2267 | + | |
| 2268 | + node = container_of(pos, struct bpf_list_node_kern, list_head); | |
| 2269 | + WRITE_ONCE(node->owner, BPF_PTR_POISON); | |
| 2270 | + list_move_tail(pos, &drain); | |
| 2271 | + } | |
| 2265 | 2272 | unlock: |
| 2266 | − INIT_LIST_HEAD(orig_head); | |
| 2273 | + INIT_LIST_HEAD(head); | |
| 2267 | 2274 | __bpf_spin_unlock_irqrestore(spin_lock); |
| 2268 | 2275 | |
| 2269 | − while (head != orig_head) { | |
| 2270 | − void *obj = head; | |
| 2276 | + while (!list_empty(&drain)) { | |
| 2277 | + struct bpf_list_node_kern *node; | |
| 2271 | 2278 | |
| 2272 | − obj -= field->graph_root.node_offset; | |
| 2273 | − head = head->next; | |
| 2279 | + pos = drain.next; | |
| 2280 | + node = container_of(pos, struct bpf_list_node_kern, list_head); | |
| 2281 | + list_del_init(pos); | |
| 2282 | + /* Ensure __bpf_list_add() sees the node as unlinked. */ | |
| 2283 | + smp_store_release(&node->owner, NULL); | |
| 2274 | 2284 | /* The contained type can also have resources, including a |
| 2275 | 2285 | * bpf_list_head which needs to be freed. |
| 2276 | 2286 | */ |
| 2277 | − __bpf_obj_drop_impl(obj, field->graph_root.value_rec, false); | |
| 2287 | + __bpf_obj_drop_impl((char *)pos - field->graph_root.node_offset, | |
| 2288 | + field->graph_root.value_rec, false); | |
| 2278 | 2289 | } |
| 2279 | 2290 | } |
| 2280 | 2291 | |
| @@ -2295,6 +2306,7 @@ void bpf_rb_root_free(const struct btf_field *field, void *rb_root, · 14 unmodified lines | ||
| 2295 | 2306 | struct bpf_spin_lock *spin_lock) |
| 2296 | 2307 | { |
| 2297 | 2308 | struct rb_root_cached orig_root, *root = rb_root; |
| 2309 | + struct bpf_rb_node_kern *node; | |
| 2298 | 2310 | struct rb_node *pos, *n; |
| 2299 | 2311 | void *obj; |
| 2300 | 2312 | |
| @@ -2303,14 +2315,20 @@ void bpf_rb_root_free(const struct btf_field *field, void *rb_root, · 2 unmodified lines | ||
| 2303 | 2315 | |
| 2304 | 2316 | __bpf_spin_lock_irqsave(spin_lock); |
| 2305 | 2317 | orig_root = *root; |
| 2318 | + bpf_rbtree_postorder_for_each_entry_safe(pos, n, &orig_root.rb_root) { | |
| 2319 | + node = rb_entry(pos, struct bpf_rb_node_kern, rb_node); | |
| 2320 | + WRITE_ONCE(node->owner, BPF_PTR_POISON); | |
| 2321 | + } | |
| 2306 | 2322 | *root = RB_ROOT_CACHED; |
| 2307 | 2323 | __bpf_spin_unlock_irqrestore(spin_lock); |
| 2308 | 2324 | |
| 2309 | 2325 | bpf_rbtree_postorder_for_each_entry_safe(pos, n, &orig_root.rb_root) { |
| 2310 | 2326 | obj = pos; |
| 2311 | 2327 | obj -= field->graph_root.node_offset; |
| 2312 | − | |
| 2313 | − | |
| 2328 | + node = rb_entry(pos, struct bpf_rb_node_kern, rb_node); | |
| 2329 | + RB_CLEAR_NODE(pos); | |
| 2330 | + /* Ensure __bpf_rbtree_add() sees the node as unlinked. */ | |
| 2331 | + smp_store_release(&node->owner, NULL); | |
| 2314 | 2332 | __bpf_obj_drop_impl(obj, field->graph_root.value_rec, false); |
| 2315 | 2333 | } |
| 2316 | 2334 | } |
| @@ -2467,9 +2485,11 @@ __bpf_kfunc void *bpf_refcount_acquire_impl(void *p__refcounted_kptr, void *meta · 150 unmodified lines | ||
| 2467 | 2485 | |
| 2468 | 2486 | static int __bpf_list_add(struct bpf_list_node_kern *node, |
| 2469 | 2487 | struct bpf_list_head *head, |
| 2470 | − bool tail, struct btf_record *rec, u64 off) | |
| 2488 | + struct list_head **prev_ptr, | |
| 2489 | + struct btf_record *rec, u64 off) | |
| 2471 | 2490 | { |
| 2472 | 2491 | struct list_head *n = &node->list_head, *h = (void *)head; |
| 2492 | + struct list_head *prev; | |
| 2473 | 2493 | |
| 2474 | 2494 | /* If list_head was 0-initialized by map, bpf_obj_init_field wasn't |
| 2475 | 2495 | * called on its fields, so init here |
| @@ -2477,19 +2497,31 @@ static int __bpf_list_add(struct bpf_list_node_kern *node, · 1 unmodified line | ||
| 2477 | 2497 | if (unlikely(!h->next)) |
| 2478 | 2498 | INIT_LIST_HEAD(h); |
| 2479 | 2499 | |
| 2500 | + prev = *prev_ptr; | |
| 2501 | + | |
| 2502 | + /* When prev is not the list head, it must be a node in this list. */ | |
| 2503 | + if (prev != h) { | |
| 2504 | + struct bpf_list_node_kern *prev_kn = | |
| 2505 | + container_of(prev, struct bpf_list_node_kern, list_head); | |
| 2506 | + | |
| 2507 | + if (unlikely(READ_ONCE(prev_kn->owner) != head)) | |
| 2508 | + goto fail; | |
| 2509 | + } | |
| 2510 | + | |
| 2480 | 2511 | /* node->owner != NULL implies !list_empty(n), no need to separately |
| 2481 | 2512 | * check the latter |
| 2482 | 2513 | */ |
| 2483 | − if (cmpxchg(&node->owner, NULL, BPF_PTR_POISON)) { | |
| 2484 | − /* Only called from BPF prog, no need to migrate_disable */ | |
| 2485 | − __bpf_obj_drop_impl((void *)n - off, rec, false); | |
| 2486 | − return -EINVAL; | |
| 2487 | − } | |
| 2514 | + if (cmpxchg(&node->owner, NULL, BPF_PTR_POISON)) | |
| 2515 | + goto fail; | |
| 2488 | 2516 | |
| 2489 | − tail ? list_add_tail(n, h) : list_add(n, h); | |
| 2517 | + list_add(n, prev); | |
| 2490 | 2518 | WRITE_ONCE(node->owner, head); |
| 2491 | − | |
| 2492 | 2519 | return 0; |
| 2520 | + | |
| 2521 | +fail: | |
| 2522 | + /* Only called from BPF prog, no need to migrate_disable */ | |
| 2523 | + __bpf_obj_drop_impl((void *)n - off, rec, false); | |
| 2524 | + return -EINVAL; | |
| 2493 | 2525 | } |
| 2494 | 2526 | |
| 2495 | 2527 | /** |
| @@ -2510,8 +2542,9 @@ __bpf_kfunc int bpf_list_push_front(struct bpf_list_head *head, · 14 unmodified lines | ||
| 2510 | 2542 | u64 off) |
| 2511 | 2543 | { |
| 2512 | 2544 | struct bpf_list_node_kern *n = (void *)node; |
| 2545 | + struct list_head *h = (void *)head; | |
| 2513 | 2546 | |
| 2514 | − return __bpf_list_add(n, head, false, meta ? meta->record : NULL, off); | |
| 2547 | + return __bpf_list_add(n, head, &h, meta ? meta->record : NULL, off); | |
| 2515 | 2548 | } |
| 2516 | 2549 | |
| 2517 | 2550 | __bpf_kfunc int bpf_list_push_front_impl(struct bpf_list_head *head, |
| @@ -2539,8 +2572,9 @@ __bpf_kfunc int bpf_list_push_back(struct bpf_list_head *head, · 21 unmodified lines | ||
| 2539 | 2572 | u64 off) |
| 2540 | 2573 | { |
| 2541 | 2574 | struct bpf_list_node_kern *n = (void *)node; |
| 2575 | + struct list_head *h = (void *)head; | |
| 2542 | 2576 | |
| 2543 | − return __bpf_list_add(n, head, true, meta ? meta->record : NULL, off); | |
| 2577 | + return __bpf_list_add(n, head, &h->prev, meta ? meta->record : NULL, off); | |
| 2544 | 2578 | } |
| 2545 | 2579 | |
| 2546 | 2580 | __bpf_kfunc int bpf_list_push_back_impl(struct bpf_list_head *head, |
| @@ -2550,37 +2584,63 @@ __bpf_kfunc int bpf_list_push_back_impl(struct bpf_list_head *head, · 3 unmodified lines | ||
| 2550 | 2584 | return bpf_list_push_back(head, node, meta__ign, off); |
| 2551 | 2585 | } |
| 2552 | 2586 | |
| 2553 | −static struct bpf_list_node *__bpf_list_del(struct bpf_list_head *head, bool tail) | |
| 2587 | +__bpf_kfunc int bpf_list_add(struct bpf_list_head *head, struct bpf_list_node *new, | |
| 2588 | + struct bpf_list_node *prev__nonown_allowed, | |
| 2589 | + struct btf_struct_meta *meta, u64 off) | |
| 2590 | +{ | |
| 2591 | + struct bpf_list_node_kern *n = (void *)new, *p = (void *)prev__nonown_allowed; | |
| 2592 | + struct list_head *prev_ptr = &p->list_head; | |
| 2593 | + | |
| 2594 | + return __bpf_list_add(n, head, &prev_ptr, meta ? meta->record : NULL, off); | |
| 2595 | +} | |
| 2596 | + | |
| 2597 | +static struct bpf_list_node *__bpf_list_del(struct bpf_list_head *head, | |
| 2598 | + struct list_head *n) | |
| 2554 | 2599 | { |
| 2555 | − struct list_head *n, *h = (void *)head; | |
| 2600 | + struct list_head *h = (void *)head; | |
| 2556 | 2601 | struct bpf_list_node_kern *node; |
| 2557 | 2602 | |
| 2558 | 2603 | /* If list_head was 0-initialized by map, bpf_obj_init_field wasn't |
| 2559 | 2604 | * called on its fields, so init here |
| 2560 | 2605 | */ |
| 2561 | − if (unlikely(!h->next)) | |
| 2606 | + if (unlikely(!h->next)) { | |
| 2562 | 2607 | INIT_LIST_HEAD(h); |
| 2608 | + return NULL; | |
| 2609 | + } | |
| 2563 | 2610 | if (list_empty(h)) |
| 2564 | 2611 | return NULL; |
| 2565 | 2612 | |
| 2566 | − n = tail ? h->prev : h->next; | |
| 2567 | 2613 | node = container_of(n, struct bpf_list_node_kern, list_head); |
| 2568 | − if (WARN_ON_ONCE(READ_ONCE(node->owner) != head)) | |
| 2614 | + if (unlikely(READ_ONCE(node->owner) != head)) | |
| 2569 | 2615 | return NULL; |
| 2570 | 2616 | |
| 2571 | 2617 | list_del_init(n); |
| 2572 | − WRITE_ONCE(node->owner, NULL); | |
| 2618 | + /* Ensure __bpf_list_add() sees the node as unlinked. */ | |
| 2619 | + smp_store_release(&node->owner, NULL); | |
| 2573 | 2620 | return (struct bpf_list_node *)n; |
| 2574 | 2621 | } |
| 2575 | 2622 | |
| 2576 | 2623 | __bpf_kfunc struct bpf_list_node *bpf_list_pop_front(struct bpf_list_head *head) |
| 2577 | 2624 | { |
| 2578 | − return __bpf_list_del(head, false); | |
| 2625 | + struct list_head *h = (void *)head; | |
| 2626 | + | |
| 2627 | + return __bpf_list_del(head, h->next); | |
| 2579 | 2628 | } |
| 2580 | 2629 | |
| 2581 | 2630 | __bpf_kfunc struct bpf_list_node *bpf_list_pop_back(struct bpf_list_head *head) |
| 2582 | 2631 | { |
| 2583 | − return __bpf_list_del(head, true); | |
| 2632 | + struct list_head *h = (void *)head; | |
| 2633 | + | |
| 2634 | + return __bpf_list_del(head, h->prev); | |
| 2635 | +} | |
| 2636 | + | |
| 2637 | +__bpf_kfunc struct bpf_list_node *bpf_list_del(struct bpf_list_head *head, | |
| 2638 | + struct bpf_list_node *node__nonown_allowed) | |
| 2639 | +{ | |
| 2640 | + struct bpf_list_node_kern *kn = (void *)node__nonown_allowed; | |
| 2641 | + | |
| 2642 | + /* verifier guarantees node is a list node rather than list head */ | |
| 2643 | + return __bpf_list_del(head, &kn->list_head); | |
| 2584 | 2644 | } |
| 2585 | 2645 | |
| 2586 | 2646 | __bpf_kfunc struct bpf_list_node *bpf_list_front(struct bpf_list_head *head) |
| @@ -2603,6 +2663,43 @@ __bpf_kfunc struct bpf_list_node *bpf_list_back(struct bpf_list_head *head) · 16 unmodified lines | ||
| 2603 | 2663 | return (struct bpf_list_node *)h->prev; |
| 2604 | 2664 | } |
| 2605 | 2665 | |
| 2666 | +__bpf_kfunc bool bpf_list_is_first(struct bpf_list_head *head, | |
| 2667 | + struct bpf_list_node *node__nonown_allowed) | |
| 2668 | +{ | |
| 2669 | + struct list_head *h = (struct list_head *)head; | |
| 2670 | + struct bpf_list_node_kern *kn = (struct bpf_list_node_kern *)node__nonown_allowed; | |
| 2671 | + | |
| 2672 | + if (READ_ONCE(kn->owner) != head) | |
| 2673 | + return false; | |
| 2674 | + | |
| 2675 | + return list_is_first(&kn->list_head, h); | |
| 2676 | +} | |
| 2677 | + | |
| 2678 | +__bpf_kfunc bool bpf_list_is_last(struct bpf_list_head *head, | |
| 2679 | + struct bpf_list_node *node__nonown_allowed) | |
| 2680 | +{ | |
| 2681 | + struct list_head *h = (struct list_head *)head; | |
| 2682 | + struct bpf_list_node_kern *kn = (struct bpf_list_node_kern *)node__nonown_allowed; | |
| 2683 | + | |
| 2684 | + if (READ_ONCE(kn->owner) != head) | |
| 2685 | + return false; | |
| 2686 | + | |
| 2687 | + return list_is_last(&kn->list_head, h); | |
| 2688 | +} | |
| 2689 | + | |
| 2690 | +__bpf_kfunc bool bpf_list_empty(struct bpf_list_head *head) | |
| 2691 | +{ | |
| 2692 | + struct list_head *h = (struct list_head *)head; | |
| 2693 | + | |
| 2694 | + /* If list_head was 0-initialized by map, bpf_obj_init_field wasn't | |
| 2695 | + * called on its fields, so init here | |
| 2696 | + */ | |
| 2697 | + if (unlikely(!h->next)) | |
| 2698 | + INIT_LIST_HEAD(h); | |
| 2699 | + | |
| 2700 | + return list_empty(h); | |
| 2701 | +} | |
| 2702 | + | |
| 2606 | 2703 | __bpf_kfunc struct bpf_rb_node *bpf_rbtree_remove(struct bpf_rb_root *root, |
| 2607 | 2704 | struct bpf_rb_node *node) |
| 2608 | 2705 | { |
| @@ -2912,11 +3009,13 @@ __bpf_kfunc struct task_struct *bpf_task_from_vpid(s32 vpid) · 303 unmodified lines | ||
| 2912 | 3009 | { |
| 2913 | 3010 | struct task_struct *p; |
| 2914 | 3011 | |
| 2915 | − rcu_read_lock(); | |
| 3012 | + guard(rcu)(); | |
| 3013 | + if (!task_active_pid_ns(current)) | |
| 3014 | + return NULL; | |
| 3015 | + | |
| 2916 | 3016 | p = find_task_by_vpid(vpid); |
| 2917 | 3017 | if (p) |
| 2918 | 3018 | p = bpf_task_acquire(p); |
| 2919 | − rcu_read_unlock(); | |
| 2920 | 3019 | |
| 2921 | 3020 | return p; |
| 2922 | 3021 | } |
| @@ -3072,7 +3171,7 @@ __bpf_kfunc void *bpf_dynptr_slice_rdwr(const struct bpf_dynptr *p, u64 offset, · 149 unmodified lines | ||
| 3072 | 3171 | return bpf_dynptr_slice(p, offset, buffer__nullable, buffer__szk); |
| 3073 | 3172 | } |
| 3074 | 3173 | |
| 3075 | −__bpf_kfunc int bpf_dynptr_adjust(const struct bpf_dynptr *p, u64 start, u64 end) | |
| 3174 | +__bpf_kfunc int bpf_dynptr_adjust(struct bpf_dynptr *p, u64 start, u64 end) | |
| 3076 | 3175 | { |
| 3077 | 3176 | struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p; |
| 3078 | 3177 | u64 size; |
| @@ -3093,14 +3192,14 @@ __bpf_kfunc int bpf_dynptr_adjust(const struct bpf_dynptr *p, u64 start, u64 end · 14 unmodified lines | ||
| 3093 | 3192 | |
| 3094 | 3193 | __bpf_kfunc bool bpf_dynptr_is_null(const struct bpf_dynptr *p) |
| 3095 | 3194 | { |
| 3096 | − struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p; | |
| 3195 | + const struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p; | |
| 3097 | 3196 | |
| 3098 | 3197 | return !ptr->data; |
| 3099 | 3198 | } |
| 3100 | 3199 | |
| 3101 | 3200 | __bpf_kfunc bool bpf_dynptr_is_rdonly(const struct bpf_dynptr *p) |
| 3102 | 3201 | { |
| 3103 | − struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p; | |
| 3202 | + const struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p; | |
| 3104 | 3203 | |
| 3105 | 3204 | if (!ptr->data) |
| 3106 | 3205 | return false; |
| @@ -3110,7 +3209,7 @@ __bpf_kfunc bool bpf_dynptr_is_rdonly(const struct bpf_dynptr *p) · 3 unmodified lines | ||
| 3110 | 3209 | |
| 3111 | 3210 | __bpf_kfunc u64 bpf_dynptr_size(const struct bpf_dynptr *p) |
| 3112 | 3211 | { |
| 3113 | − struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p; | |
| 3212 | + const struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p; | |
| 3114 | 3213 | |
| 3115 | 3214 | if (!ptr->data) |
| 3116 | 3215 | return -EINVAL; |
| @@ -3122,7 +3221,7 @@ __bpf_kfunc int bpf_dynptr_clone(const struct bpf_dynptr *p, · 5 unmodified lines | ||
| 3122 | 3221 | struct bpf_dynptr *clone__uninit) |
| 3123 | 3222 | { |
| 3124 | 3223 | struct bpf_dynptr_kern *clone = (struct bpf_dynptr_kern *)clone__uninit; |
| 3125 | − struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p; | |
| 3224 | + const struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p; | |
| 3126 | 3225 | |
| 3127 | 3226 | if (!ptr->data) { |
| 3128 | 3227 | bpf_dynptr_set_null(clone); |
| @@ -3145,11 +3244,11 @@ __bpf_kfunc int bpf_dynptr_clone(const struct bpf_dynptr *p, · 16 unmodified lines | ||
| 3145 | 3244 | * Copies data from source dynptr to destination dynptr. |
| 3146 | 3245 | * Returns 0 on success; negative error, otherwise. |
| 3147 | 3246 | */ |
| 3148 | −__bpf_kfunc int bpf_dynptr_copy(struct bpf_dynptr *dst_ptr, u64 dst_off, | |
| 3149 | − struct bpf_dynptr *src_ptr, u64 src_off, u64 size) | |
| 3247 | +__bpf_kfunc int bpf_dynptr_copy(const struct bpf_dynptr *dst_ptr, u64 dst_off, | |
| 3248 | + const struct bpf_dynptr *src_ptr, u64 src_off, u64 size) | |
| 3150 | 3249 | { |
| 3151 | − struct bpf_dynptr_kern *dst = (struct bpf_dynptr_kern *)dst_ptr; | |
| 3152 | − struct bpf_dynptr_kern *src = (struct bpf_dynptr_kern *)src_ptr; | |
| 3250 | + const struct bpf_dynptr_kern *dst = (struct bpf_dynptr_kern *)dst_ptr; | |
| 3251 | + const struct bpf_dynptr_kern *src = (struct bpf_dynptr_kern *)src_ptr; | |
| 3153 | 3252 | void *src_slice, *dst_slice; |
| 3154 | 3253 | char buf[256]; |
| 3155 | 3254 | u64 off; |
| @@ -3200,9 +3299,9 @@ __bpf_kfunc int bpf_dynptr_copy(struct bpf_dynptr *dst_ptr, u64 dst_off, · 44 unmodified lines | ||
| 3200 | 3299 | * at @offset with the constant byte @val. |
| 3201 | 3300 | * Returns 0 on success; negative error, otherwise. |
| 3202 | 3301 | */ |
| 3203 | −__bpf_kfunc int bpf_dynptr_memset(struct bpf_dynptr *p, u64 offset, u64 size, u8 val) | |
| 3302 | +__bpf_kfunc int bpf_dynptr_memset(const struct bpf_dynptr *p, u64 offset, u64 size, u8 val) | |
| 3204 | 3303 | { |
| 3205 | − struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p; | |
| 3304 | + const struct bpf_dynptr_kern *ptr = (struct bpf_dynptr_kern *)p; | |
| 3206 | 3305 | u64 chunk_sz, write_off; |
| 3207 | 3306 | char buf[256]; |
| 3208 | 3307 | void* slice; |
| @@ -3301,7 +3400,7 @@ __bpf_kfunc void bpf_throw(u64 cookie) · 92 unmodified lines | ||
| 3301 | 3400 | * which skips compiler generated instrumentation to do the same. |
| 3302 | 3401 | */ |
| 3303 | 3402 | kasan_unpoison_task_stack_below((void *)(long)ctx.sp); |
| 3304 | − ctx.aux->bpf_exception_cb(cookie, ctx.sp, ctx.bp, 0, 0); | |
| 3403 | + ctx.aux->bpf_exception_cb(cookie, ctx.sp + ctx.aux->stack_arg_sp_adjust, ctx.bp, 0, 0); | |
| 3305 | 3404 | WARN(1, "A call to BPF exception callback should never return\n"); |
| 3306 | 3405 | } |
| 3307 | 3406 | |
| @@ -4214,13 +4313,13 @@ __bpf_kfunc void bpf_key_put(struct bpf_key *bkey) · 906 unmodified lines | ||
| 4214 | 4313 | * |
| 4215 | 4314 | * Return: 0 on success, a negative value on error. |
| 4216 | 4315 | */ |
| 4217 | −__bpf_kfunc int bpf_verify_pkcs7_signature(struct bpf_dynptr *data_p, | |
| 4218 | − struct bpf_dynptr *sig_p, | |
| 4316 | +__bpf_kfunc int bpf_verify_pkcs7_signature(const struct bpf_dynptr *data_p, | |
| 4317 | + const struct bpf_dynptr *sig_p, | |
| 4219 | 4318 | struct bpf_key *trusted_keyring) |
| 4220 | 4319 | { |
| 4221 | 4320 | #ifdef CONFIG_SYSTEM_DATA_VERIFICATION |
| 4222 | − struct bpf_dynptr_kern *data_ptr = (struct bpf_dynptr_kern *)data_p; | |
| 4223 | − struct bpf_dynptr_kern *sig_ptr = (struct bpf_dynptr_kern *)sig_p; | |
| 4321 | + const struct bpf_dynptr_kern *data_ptr = (struct bpf_dynptr_kern *)data_p; | |
| 4322 | + const struct bpf_dynptr_kern *sig_ptr = (struct bpf_dynptr_kern *)sig_p; | |
| 4224 | 4323 | const void *data, *sig; |
| 4225 | 4324 | u32 data_len, sig_len; |
| 4226 | 4325 | int ret; |
| @@ -4718,10 +4817,15 @@ BTF_ID_FLAGS(func, bpf_list_push_front, KF_IMPLICIT_ARGS) · 491 unmodified lines | ||
| 4718 | 4817 | BTF_ID_FLAGS(func, bpf_list_push_front_impl) |
| 4719 | 4818 | BTF_ID_FLAGS(func, bpf_list_push_back, KF_IMPLICIT_ARGS) |
| 4720 | 4819 | BTF_ID_FLAGS(func, bpf_list_push_back_impl) |
| 4820 | +BTF_ID_FLAGS(func, bpf_list_add, KF_IMPLICIT_ARGS) | |
| 4721 | 4821 | BTF_ID_FLAGS(func, bpf_list_pop_front, KF_ACQUIRE | KF_RET_NULL) |
| 4722 | 4822 | BTF_ID_FLAGS(func, bpf_list_pop_back, KF_ACQUIRE | KF_RET_NULL) |
| 4823 | +BTF_ID_FLAGS(func, bpf_list_del, KF_ACQUIRE | KF_RET_NULL) | |
| 4723 | 4824 | BTF_ID_FLAGS(func, bpf_list_front, KF_RET_NULL) |
| 4724 | 4825 | BTF_ID_FLAGS(func, bpf_list_back, KF_RET_NULL) |
| 4826 | +BTF_ID_FLAGS(func, bpf_list_is_first) | |
| 4827 | +BTF_ID_FLAGS(func, bpf_list_is_last) | |
| 4828 | +BTF_ID_FLAGS(func, bpf_list_empty) | |
| 4725 | 4829 | BTF_ID_FLAGS(func, bpf_task_acquire, KF_ACQUIRE | KF_RCU | KF_RET_NULL) |
| 4726 | 4830 | BTF_ID_FLAGS(func, bpf_task_release, KF_RELEASE) |
| 4727 | 4831 | BTF_ID_FLAGS(func, bpf_rbtree_remove, KF_ACQUIRE | KF_RET_NULL) |
| @@ -4862,7 +4966,7 @@ BTF_ID_FLAGS(func, bpf_stream_print_stack, KF_IMPLICIT_ARGS) · 134 unmodified lines | ||
| 4862 | 4966 | BTF_ID_FLAGS(func, bpf_task_work_schedule_signal, KF_IMPLICIT_ARGS) |
| 4863 | 4967 | BTF_ID_FLAGS(func, bpf_task_work_schedule_resume, KF_IMPLICIT_ARGS) |
| 4864 | 4968 | BTF_ID_FLAGS(func, bpf_dynptr_from_file) |
| 4865 | −BTF_ID_FLAGS(func, bpf_dynptr_file_discard) | |
| 4969 | +BTF_ID_FLAGS(func, bpf_dynptr_file_discard, KF_RELEASE) | |
| 4866 | 4970 | BTF_ID_FLAGS(func, bpf_timer_cancel_async) |
| 4867 | 4971 | BTF_KFUNCS_END(common_btf_ids) |
| 4868 | 4972 | |
| @@ -845,6 +845,10 @@ static void bpf_destroy_inode(struct inode *inode) · 844 unmodified lines | ||
| 845 | 845 | simple_xattrs_free(&opts->xa_cache, &bi->xattrs, NULL); |
| 846 | 846 | } |
| 847 | 847 | |
| 848 | +/* | |
| 849 | + * Called after RCU grace period - safe to free inode and anything | |
| 850 | + * that might be accessed by RCU pathwalk (inode fields, i_link). | |
| 851 | + */ | |
| 848 | 852 | static void bpf_free_inode(struct inode *inode) |
| 849 | 853 | { |
| 850 | 854 | if (S_ISLNK(inode->i_mode)) |
| @@ -610,6 +610,21 @@ enum arg_track_state { · 609 unmodified lines | ||
| 610 | 610 | /* Track callee stack slots fp-8 through fp-512 (64 slots of 8 bytes each) */ |
| 611 | 611 | #define MAX_ARG_SPILL_SLOTS 64 |
| 612 | 612 | |
| 613 | +/* | |
| 614 | + * Combined register + stack arg tracking: R0-R10 at indices 0-10, | |
| 615 | + * outgoing stack arg slots at indices MAX_BPF_REG..MAX_BPF_REG+6. | |
| 616 | + */ | |
| 617 | +#define MAX_AT_TRACK_REGS (MAX_BPF_REG + MAX_STACK_ARG_SLOTS) | |
| 618 | + | |
| 619 | +static int stack_arg_off_to_slot(s16 off) | |
| 620 | +{ | |
| 621 | + int aoff = off < 0 ? -off : off; | |
| 622 | + | |
| 623 | + if (aoff / 8 > MAX_STACK_ARG_SLOTS) | |
| 624 | + return -1; | |
| 625 | + return aoff / 8 - 1; | |
| 626 | +} | |
| 627 | + | |
| 613 | 628 | static bool arg_is_visited(const struct arg_track *at) |
| 614 | 629 | { |
| 615 | 630 | return at->frame != ARG_UNVISITED; |
| @@ -791,7 +806,9 @@ static bool arg_track_join(struct bpf_verifier_env *env, int idx, int target, in · 175 unmodified lines | ||
| 791 | 806 | return true; |
| 792 | 807 | |
| 793 | 808 | verbose(env, "arg JOIN insn %d -> %d ", idx, target); |
| 794 | − if (r >= 0) | |
| 809 | + if (r >= MAX_BPF_REG) | |
| 810 | + verbose(env, "sa%d: ", r - MAX_BPF_REG); | |
| 811 | + else if (r >= 0) | |
| 795 | 812 | verbose(env, "r%d: ", r); |
| 796 | 813 | else |
| 797 | 814 | verbose(env, "fp%+d: ", r * 8); |
| @@ -1032,6 +1049,21 @@ static void arg_track_log(struct bpf_verifier_env *env, struct bpf_insn *insn, i · 234 unmodified lines | ||
| 1032 | 1049 | verbose(env, "\tr%d: ", i); verbose_arg_track(env, &at_in[i]); |
| 1033 | 1050 | verbose(env, " -> "); verbose_arg_track(env, &at_out[i]); |
| 1034 | 1051 | } |
| 1052 | + /* Log outgoing stack arg slot transitions at indices MAX_BPF_REG..MAX_AT_TRACK_REGS-1 */ | |
| 1053 | + for (i = 0; i < MAX_STACK_ARG_SLOTS; i++) { | |
| 1054 | + int ai = MAX_BPF_REG + i; | |
| 1055 | + | |
| 1056 | + if (arg_track_eq(&at_out[ai], &at_in[ai])) | |
| 1057 | + continue; | |
| 1058 | + if (!printed) { | |
| 1059 | + verbose(env, "%3d: ", idx); | |
| 1060 | + bpf_verbose_insn(env, insn); | |
| 1061 | + bpf_vlog_reset(&env->log, env->log.end_pos - 1); | |
| 1062 | + printed = true; | |
| 1063 | + } | |
| 1064 | + verbose(env, "\tsa%d: ", i); verbose_arg_track(env, &at_in[ai]); | |
| 1065 | + verbose(env, " -> "); verbose_arg_track(env, &at_out[ai]); | |
| 1066 | + } | |
| 1035 | 1067 | for (i = 0; i < MAX_ARG_SPILL_SLOTS; i++) { |
| 1036 | 1068 | if (arg_track_eq(&at_stack_out[i], &at_stack_in[i])) |
| 1037 | 1069 | continue; |
| @@ -1062,6 +1094,7 @@ static bool can_be_local_fp(int depth, int regno, struct arg_track *at) · 24 unmodified lines | ||
| 1062 | 1094 | static void arg_track_xfer(struct bpf_verifier_env *env, struct bpf_insn *insn, |
| 1063 | 1095 | int insn_idx, |
| 1064 | 1096 | struct arg_track *at_out, struct arg_track *at_stack_out, |
| 1097 | + const struct arg_track *at_stack_arg_entry, | |
| 1065 | 1098 | struct func_instance *instance, |
| 1066 | 1099 | u32 *callsites) |
| 1067 | 1100 | { |
| @@ -1071,9 +1104,21 @@ static void arg_track_xfer(struct bpf_verifier_env *env, struct bpf_insn *insn, · 3 unmodified lines | ||
| 1071 | 1104 | struct arg_track *dst = &at_out[insn->dst_reg]; |
| 1072 | 1105 | struct arg_track *src = &at_out[insn->src_reg]; |
| 1073 | 1106 | struct arg_track none = { .frame = ARG_NONE }; |
| 1074 | − int r; | |
| 1075 | − | |
| 1076 | − if (class == BPF_ALU64 && BPF_SRC(insn->code) == BPF_K) { | |
| 1107 | + int r, slot; | |
| 1108 | + | |
| 1109 | + /* Handle stack arg stores and loads. */ | |
| 1110 | + if (is_stack_arg_st(insn) || is_stack_arg_stx(insn)) { | |
| 1111 | + slot = stack_arg_off_to_slot(insn->off); | |
| 1112 | + if (slot >= 0) { | |
| 1113 | + if (is_stack_arg_stx(insn)) | |
| 1114 | + at_out[MAX_BPF_REG + slot] = at_out[insn->src_reg]; | |
| 1115 | + else | |
| 1116 | + at_out[MAX_BPF_REG + slot] = none; | |
| 1117 | + } | |
| 1118 | + } else if (is_stack_arg_ldx(insn)) { | |
| 1119 | + slot = stack_arg_off_to_slot(insn->off); | |
| 1120 | + at_out[insn->dst_reg] = (slot >= 0) ? at_stack_arg_entry[slot] : none; | |
| 1121 | + } else if (class == BPF_ALU64 && BPF_SRC(insn->code) == BPF_K) { | |
| 1077 | 1122 | if (code == BPF_MOV) { |
| 1078 | 1123 | *dst = none; |
| 1079 | 1124 | } else if (dst->frame >= 0) { |
| @@ -1297,6 +1342,16 @@ static int record_load_store_access(struct bpf_verifier_env *env, · 217 unmodified lines | ||
| 1297 | 1342 | struct arg_track resolved, *ptr; |
| 1298 | 1343 | int oi; |
| 1299 | 1344 | |
| 1345 | + /* | |
| 1346 | + * Stack arg insns use dst_reg/src_reg=BPF_REG_PARAMS(11). Since at[] | |
| 1347 | + * is extended to MAX_AT_TRACK_REGS, at[11] holds the arg_track for | |
| 1348 | + * outgoing stack arg slot 0 — not the pointer used for the memory | |
| 1349 | + * access. Skip so the slot's tracked value isn't confused with the | |
| 1350 | + * base register that record_stack_access() expects. | |
| 1351 | + */ | |
| 1352 | + if (is_stack_arg_stx(insn) || is_stack_arg_st(insn) || is_stack_arg_ldx(insn)) | |
| 1353 | + return 0; | |
| 1354 | + | |
| 1300 | 1355 | switch (class) { |
| 1301 | 1356 | case BPF_LDX: |
| 1302 | 1357 | ptr = &at[insn->src_reg]; |
| @@ -1343,6 +1398,42 @@ static int record_load_store_access(struct bpf_verifier_env *env, · 40 unmodified lines | ||
| 1343 | 1398 | return 0; |
| 1344 | 1399 | } |
| 1345 | 1400 | |
| 1401 | +static int record_arg_access(struct bpf_verifier_env *env, | |
| 1402 | + struct func_instance *instance, | |
| 1403 | + struct bpf_insn *insn, | |
| 1404 | + struct arg_track *at, int arg_idx, | |
| 1405 | + int insn_idx) | |
| 1406 | +{ | |
| 1407 | + int depth = instance->depth; | |
| 1408 | + int frame = at->frame; | |
| 1409 | + int err = 0; | |
| 1410 | + s64 bytes; | |
| 1411 | + | |
| 1412 | + if (!arg_is_fp(at)) | |
| 1413 | + return 0; | |
| 1414 | + | |
| 1415 | + if (bpf_helper_call(insn)) { | |
| 1416 | + bytes = bpf_helper_stack_access_bytes(env, insn, arg_idx, insn_idx); | |
| 1417 | + } else if (bpf_pseudo_kfunc_call(insn)) { | |
| 1418 | + bytes = bpf_kfunc_stack_access_bytes(env, insn, arg_idx, insn_idx); | |
| 1419 | + } else { | |
| 1420 | + for (int f = 0; f <= depth; f++) { | |
| 1421 | + err = mark_stack_read(instance, f, insn_idx, SPIS_ALL); | |
| 1422 | + if (err) | |
| 1423 | + return err; | |
| 1424 | + } | |
| 1425 | + return 0; | |
| 1426 | + } | |
| 1427 | + if (bytes == 0) | |
| 1428 | + return 0; | |
| 1429 | + | |
| 1430 | + if (frame >= 0 && frame <= depth) | |
| 1431 | + err = record_stack_access(instance, at, bytes, frame, insn_idx); | |
| 1432 | + else if (frame == ARG_IMPRECISE) | |
| 1433 | + err = record_imprecise(instance, at->mask, insn_idx); | |
| 1434 | + return err; | |
| 1435 | +} | |
| 1436 | + | |
| 1346 | 1437 | /* Record stack access for a given 'at' state of helper/kfunc 'insn' */ |
| 1347 | 1438 | static int record_call_access(struct bpf_verifier_env *env, |
| 1348 | 1439 | struct func_instance *instance, |
| @@ -1350,9 +1441,8 @@ static int record_call_access(struct bpf_verifier_env *env, · 1 unmodified line | ||
| 1350 | 1441 | int insn_idx) |
| 1351 | 1442 | { |
| 1352 | 1443 | struct bpf_insn *insn = &env->prog->insnsi[insn_idx]; |
| 1353 | − int depth = instance->depth; | |
| 1354 | 1444 | struct bpf_call_summary cs; |
| 1355 | − int r, err = 0, num_params = 5; | |
| 1445 | + int r, err, num_params = 5; | |
| 1356 | 1446 | |
| 1357 | 1447 | if (bpf_pseudo_call(insn)) |
| 1358 | 1448 | return 0; |
| @@ -1360,32 +1450,15 @@ static int record_call_access(struct bpf_verifier_env *env, · 1 unmodified line | ||
| 1360 | 1450 | if (bpf_get_call_summary(env, insn, &cs)) |
| 1361 | 1451 | num_params = cs.num_params; |
| 1362 | 1452 | |
| 1363 | − for (r = BPF_REG_1; r < BPF_REG_1 + num_params; r++) { | |
| 1364 | − int frame = at[r].frame; | |
| 1365 | − s64 bytes; | |
| 1366 | − | |
| 1367 | − if (!arg_is_fp(&at[r])) | |
| 1368 | − continue; | |
| 1369 | − | |
| 1370 | − if (bpf_helper_call(insn)) { | |
| 1371 | − bytes = bpf_helper_stack_access_bytes(env, insn, r - 1, insn_idx); | |
| 1372 | − } else if (bpf_pseudo_kfunc_call(insn)) { | |
| 1373 | − bytes = bpf_kfunc_stack_access_bytes(env, insn, r - 1, insn_idx); | |
| 1374 | − } else { | |
| 1375 | − for (int f = 0; f <= depth; f++) { | |
| 1376 | − err = mark_stack_read(instance, f, insn_idx, SPIS_ALL); | |
| 1377 | − if (err) | |
| 1378 | − return err; | |
| 1379 | − } | |
| 1380 | − return 0; | |
| 1381 | − } | |
| 1382 | − if (bytes == 0) | |
| 1383 | − continue; | |
| 1453 | + for (r = BPF_REG_1; r < BPF_REG_1 + min(num_params, MAX_BPF_FUNC_REG_ARGS); r++) { | |
| 1454 | + err = record_arg_access(env, instance, insn, &at[r], r - 1, insn_idx); | |
| 1455 | + if (err) | |
| 1456 | + return err; | |
| 1457 | + } | |
| 1384 | 1458 | |
| 1385 | − if (frame >= 0 && frame <= depth) | |
| 1386 | − err = record_stack_access(instance, &at[r], bytes, frame, insn_idx); | |
| 1387 | − else if (frame == ARG_IMPRECISE) | |
| 1388 | − err = record_imprecise(instance, at[r].mask, insn_idx); | |
| 1459 | + for (r = 0; r < MAX_STACK_ARG_SLOTS && r < num_params - MAX_BPF_FUNC_REG_ARGS; r++) { | |
| 1460 | + err = record_arg_access(env, instance, insn, &at[MAX_BPF_REG + r], | |
| 1461 | + r + MAX_BPF_FUNC_REG_ARGS, insn_idx); | |
| 1389 | 1462 | if (err) |
| 1390 | 1463 | return err; |
| 1391 | 1464 | } |
| @@ -1445,7 +1518,7 @@ static int find_callback_subprog(struct bpf_verifier_env *env, · 53 unmodified lines | ||
| 1445 | 1518 | |
| 1446 | 1519 | /* Per-subprog intermediate state kept alive across analysis phases */ |
| 1447 | 1520 | struct subprog_at_info { |
| 1448 | − struct arg_track (*at_in)[MAX_BPF_REG]; | |
| 1521 | + struct arg_track (*at_in)[MAX_AT_TRACK_REGS]; | |
| 1449 | 1522 | int len; |
| 1450 | 1523 | }; |
| 1451 | 1524 | |
| @@ -1479,6 +1552,9 @@ static void print_subprog_arg_access(struct bpf_verifier_env *env, · 27 unmodified lines | ||
| 1479 | 1552 | for (r = 0; r < MAX_BPF_REG - 1; r++) |
| 1480 | 1553 | if (arg_is_fp(&info->at_in[i][r])) |
| 1481 | 1554 | has_extra = true; |
| 1555 | + for (r = 0; r < MAX_STACK_ARG_SLOTS; r++) | |
| 1556 | + if (arg_is_fp(&info->at_in[i][MAX_BPF_REG + r])) | |
| 1557 | + has_extra = true; | |
| 1482 | 1558 | } |
| 1483 | 1559 | if (is_ldx_stx_call) { |
| 1484 | 1560 | for (r = 0; r < MAX_ARG_SPILL_SLOTS; r++) |
| @@ -1503,6 +1579,12 @@ static void print_subprog_arg_access(struct bpf_verifier_env *env, · 18 unmodified lines | ||
| 1503 | 1579 | verbose(env, " r%d=", r); |
| 1504 | 1580 | verbose_arg_track(env, &info->at_in[i][r]); |
| 1505 | 1581 | } |
| 1582 | + for (r = 0; r < MAX_STACK_ARG_SLOTS; r++) { | |
| 1583 | + if (!arg_is_fp(&info->at_in[i][MAX_BPF_REG + r])) | |
| 1584 | + continue; | |
| 1585 | + verbose(env, " sa%d=", r); | |
| 1586 | + verbose_arg_track(env, &info->at_in[i][MAX_BPF_REG + r]); | |
| 1587 | + } | |
| 1506 | 1588 | } |
| 1507 | 1589 | |
| 1508 | 1590 | if (is_ldx_stx_call) { |
| @@ -1525,7 +1607,7 @@ static void print_subprog_arg_access(struct bpf_verifier_env *env, · 16 unmodified lines | ||
| 1525 | 1607 | * Runs forward fixed-point with arg_track_xfer(), then records |
| 1526 | 1608 | * memory accesses in a single linear pass over converged state. |
| 1527 | 1609 | * |
| 1528 | − * @callee_entry: pre-populated entry state for R1-R5 | |
| 1610 | + * @callee_entry: pre-populated entry state for R1-R5 and stack args | |
| 1529 | 1611 | * NULL for main (subprog 0). |
| 1530 | 1612 | * @info: stores at_in, len for debug printing. |
| 1531 | 1613 | */ |
| @@ -1543,10 +1625,11 @@ static int compute_subprog_args(struct bpf_verifier_env *env, · 11 unmodified lines | ||
| 1543 | 1625 | int end = env->subprog_info[subprog + 1].start; |
| 1544 | 1626 | int po_end = env->subprog_info[subprog + 1].postorder_start; |
| 1545 | 1627 | int len = end - start; |
| 1546 | − struct arg_track (*at_in)[MAX_BPF_REG] = NULL; | |
| 1547 | − struct arg_track at_out[MAX_BPF_REG]; | |
| 1628 | + struct arg_track (*at_in)[MAX_AT_TRACK_REGS] = NULL; | |
| 1629 | + struct arg_track at_out[MAX_AT_TRACK_REGS]; | |
| 1548 | 1630 | struct arg_track (*at_stack_in)[MAX_ARG_SPILL_SLOTS] = NULL; |
| 1549 | 1631 | struct arg_track *at_stack_out = NULL; |
| 1632 | + struct arg_track at_stack_arg_entry[MAX_STACK_ARG_SLOTS]; | |
| 1550 | 1633 | struct arg_track unvisited = { .frame = ARG_UNVISITED }; |
| 1551 | 1634 | struct arg_track none = { .frame = ARG_NONE }; |
| 1552 | 1635 | bool changed; |
| @@ -1565,13 +1648,13 @@ static int compute_subprog_args(struct bpf_verifier_env *env, · 12 unmodified lines | ||
| 1565 | 1648 | goto err_free; |
| 1566 | 1649 | |
| 1567 | 1650 | for (i = 0; i < len; i++) { |
| 1568 | − for (r = 0; r < MAX_BPF_REG; r++) | |
| 1651 | + for (r = 0; r < MAX_AT_TRACK_REGS; r++) | |
| 1569 | 1652 | at_in[i][r] = unvisited; |
| 1570 | 1653 | for (r = 0; r < MAX_ARG_SPILL_SLOTS; r++) |
| 1571 | 1654 | at_stack_in[i][r] = unvisited; |
| 1572 | 1655 | } |
| 1573 | 1656 | |
| 1574 | − for (r = 0; r < MAX_BPF_REG; r++) | |
| 1657 | + for (r = 0; r < MAX_AT_TRACK_REGS; r++) | |
| 1575 | 1658 | at_in[0][r] = none; |
| 1576 | 1659 | |
| 1577 | 1660 | /* Entry: R10 is always precisely the current frame's FP */ |
| @@ -1587,6 +1670,10 @@ static int compute_subprog_args(struct bpf_verifier_env *env, · 9 unmodified lines | ||
| 1587 | 1670 | for (r = 0; r < MAX_ARG_SPILL_SLOTS; r++) |
| 1588 | 1671 | at_stack_in[0][r] = none; |
| 1589 | 1672 | |
| 1673 | + /* Entry: incoming stack args from caller, or ARG_NONE for main */ | |
| 1674 | + for (r = 0; r < MAX_STACK_ARG_SLOTS; r++) | |
| 1675 | + at_stack_arg_entry[r] = callee_entry ? callee_entry[MAX_BPF_REG + r] : none; | |
| 1676 | + | |
| 1590 | 1677 | if (env->log.level & BPF_LOG_LEVEL2) |
| 1591 | 1678 | verbose(env, "subprog#%d: analyzing (depth %d)...\n", subprog, depth); |
| 1592 | 1679 | |
| @@ -1605,7 +1692,8 @@ redo: · 12 unmodified lines | ||
| 1605 | 1692 | memcpy(at_out, at_in[i], sizeof(at_out)); |
| 1606 | 1693 | memcpy(at_stack_out, at_stack_in[i], MAX_ARG_SPILL_SLOTS * sizeof(*at_stack_out)); |
| 1607 | 1694 | |
| 1608 | − arg_track_xfer(env, insn, idx, at_out, at_stack_out, instance, callsites); | |
| 1695 | + arg_track_xfer(env, insn, idx, at_out, at_stack_out, | |
| 1696 | + at_stack_arg_entry, instance, callsites); | |
| 1609 | 1697 | arg_track_log(env, insn, idx, at_in[i], at_stack_in[i], at_out, at_stack_out); |
| 1610 | 1698 | |
| 1611 | 1699 | /* Propagate to successors within this subprogram */ |
| @@ -1619,7 +1707,7 @@ redo: · 7 unmodified lines | ||
| 1619 | 1707 | continue; |
| 1620 | 1708 | ti = target - start; |
| 1621 | 1709 | |
| 1622 | − for (r = 0; r < MAX_BPF_REG; r++) | |
| 1710 | + for (r = 0; r < MAX_AT_TRACK_REGS; r++) | |
| 1623 | 1711 | changed |= arg_track_join(env, idx, target, r, |
| 1624 | 1712 | &at_in[ti][r], at_out[r]); |
| 1625 | 1713 | |
| @@ -1674,11 +1762,14 @@ err_free: · 48 unmodified lines | ||
| 1674 | 1762 | return err; |
| 1675 | 1763 | } |
| 1676 | 1764 | |
| 1677 | −/* Return true if any of R1-R5 is derived from a frame pointer. */ | |
| 1765 | +/* Return true if any of R1-R5 or stack args is derived from a frame pointer. */ | |
| 1678 | 1766 | static bool has_fp_args(struct arg_track *args) |
| 1679 | 1767 | { |
| 1680 | 1768 | for (int r = BPF_REG_1; r <= BPF_REG_5; r++) |
| 1681 | − if (args[r].frame != ARG_NONE) | |
| 1769 | + if (arg_is_fp(&args[r])) | |
| 1770 | + return true; | |
| 1771 | + for (int r = 0; r < MAX_STACK_ARG_SLOTS; r++) | |
| 1772 | + if (arg_is_fp(&args[MAX_BPF_REG + r])) | |
| 1682 | 1773 | return true; |
| 1683 | 1774 | return false; |
| 1684 | 1775 | } |
| @@ -1803,7 +1894,7 @@ static int analyze_subprog(struct bpf_verifier_env *env, · 118 unmodified lines | ||
| 1803 | 1894 | /* For each reachable call site in the subprog, recurse into callees */ |
| 1804 | 1895 | for (int p = po_start; p < po_end; p++) { |
| 1805 | 1896 | int idx = env->cfg.insn_postorder[p]; |
| 1806 | − struct arg_track callee_args[BPF_REG_5 + 1]; | |
| 1897 | + struct arg_track callee_args[MAX_AT_TRACK_REGS] = {}; | |
| 1807 | 1898 | struct arg_track none = { .frame = ARG_NONE }; |
| 1808 | 1899 | struct bpf_insn *insn = &insns[idx]; |
| 1809 | 1900 | struct func_instance *callee_instance; |
| @@ -1818,9 +1909,11 @@ static int analyze_subprog(struct bpf_verifier_env *env, · 8 unmodified lines | ||
| 1818 | 1909 | if (callee < 0) |
| 1819 | 1910 | continue; |
| 1820 | 1911 | |
| 1821 | − /* Build entry args: R1-R5 from at_in at call site */ | |
| 1912 | + /* Build entry args: R1-R5 and stack args from at_in at call site */ | |
| 1822 | 1913 | for (int r = BPF_REG_1; r <= BPF_REG_5; r++) |
| 1823 | 1914 | callee_args[r] = info[subprog].at_in[j][r]; |
| 1915 | + for (int r = 0; r < MAX_STACK_ARG_SLOTS; r++) | |
| 1916 | + callee_args[MAX_BPF_REG + r] = info[subprog].at_in[j][MAX_BPF_REG + r]; | |
| 1824 | 1917 | } else if (bpf_calls_callback(env, idx)) { |
| 1825 | 1918 | callee = find_callback_subprog(env, insn, idx, &caller_reg, &cb_callee_reg); |
| 1826 | 1919 | if (callee == -2) { |
| @@ -1842,6 +1935,8 @@ static int analyze_subprog(struct bpf_verifier_env *env, · 15 unmodified lines | ||
| 1842 | 1935 | |
| 1843 | 1936 | for (int r = BPF_REG_1; r <= BPF_REG_5; r++) |
| 1844 | 1937 | callee_args[r] = none; |
| 1938 | + for (int r = 0; r < MAX_STACK_ARG_SLOTS; r++) | |
| 1939 | + callee_args[MAX_BPF_REG + r] = none; | |
| 1845 | 1940 | callee_args[cb_callee_reg] = info[subprog].at_in[j][caller_reg]; |
| 1846 | 1941 | } else { |
| 1847 | 1942 | continue; |
| @@ -2085,7 +2180,7 @@ static void compute_insn_live_regs(struct bpf_verifier_env *env, · 237 unmodified lines | ||
| 2085 | 2180 | def = ALL_CALLER_SAVED_REGS; |
| 2086 | 2181 | use = def & ~BIT(BPF_REG_0); |
| 2087 | 2182 | if (bpf_get_call_summary(env, insn, &cs)) |
| 2088 | − use = GENMASK(cs.num_params, 1); | |
| 2183 | + use = GENMASK(min_t(u8, cs.num_params, MAX_BPF_FUNC_REG_ARGS), 1); | |
| 2089 | 2184 | break; |
| 2090 | 2185 | default: |
| 2091 | 2186 | def = 0; |
| @@ -13,17 +13,17 @@ · 12 unmodified lines | ||
| 13 | 13 | |
| 14 | 14 | #define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args) |
| 15 | 15 | |
| 16 | −static bool bpf_verifier_log_attr_valid(const struct bpf_verifier_log *log) | |
| 16 | +static bool bpf_verifier_log_attr_valid(u32 log_level, char __user *log_buf, u32 log_size) | |
| 17 | 17 | { |
| 18 | 18 | /* ubuf and len_total should both be specified (or not) together */ |
| 19 | − if (!!log->ubuf != !!log->len_total) | |
| 19 | + if (!!log_buf != !!log_size) | |
| 20 | 20 | return false; |
| 21 | 21 | /* log buf without log_level is meaningless */ |
| 22 | − if (log->ubuf && log->level == 0) | |
| 22 | + if (log_buf && log_level == 0) | |
| 23 | 23 | return false; |
| 24 | − if (log->level & ~BPF_LOG_MASK) | |
| 24 | + if (log_level & ~BPF_LOG_MASK) | |
| 25 | 25 | return false; |
| 26 | − if (log->len_total > UINT_MAX >> 2) | |
| 26 | + if (log_size > UINT_MAX >> 2) | |
| 27 | 27 | return false; |
| 28 | 28 | return true; |
| 29 | 29 | } |
| @@ -36,7 +36,7 @@ int bpf_vlog_init(struct bpf_verifier_log *log, u32 log_level, · 6 unmodified lines | ||
| 36 | 36 | log->len_total = log_size; |
| 37 | 37 | |
| 38 | 38 | /* log attributes have to be sane */ |
| 39 | − if (!bpf_verifier_log_attr_valid(log)) | |
| 39 | + if (!bpf_verifier_log_attr_valid(log_level, log_buf, log_size)) | |
| 40 | 40 | return -EINVAL; |
| 41 | 41 | |
| 42 | 42 | return 0; |
| @@ -571,20 +571,20 @@ static void print_scalar_ranges(struct bpf_verifier_env *env, · 528 unmodified lines | ||
| 571 | 571 | u64 val; |
| 572 | 572 | bool omit; |
| 573 | 573 | } minmaxs[] = { |
| 574 | − {"smin", reg->smin_value, reg->smin_value == S64_MIN}, | |
| 575 | − {"smax", reg->smax_value, reg->smax_value == S64_MAX}, | |
| 576 | − {"umin", reg->umin_value, reg->umin_value == 0}, | |
| 577 | − {"umax", reg->umax_value, reg->umax_value == U64_MAX}, | |
| 574 | + {"smin", reg_smin(reg), reg_smin(reg) == S64_MIN}, | |
| 575 | + {"smax", reg_smax(reg), reg_smax(reg) == S64_MAX}, | |
| 576 | + {"umin", reg_umin(reg), reg_umin(reg) == 0}, | |
| 577 | + {"umax", reg_umax(reg), reg_umax(reg) == U64_MAX}, | |
| 578 | 578 | {"smin32", |
| 579 | − is_snum_decimal((s64)reg->s32_min_value) | |
| 580 | − ? (s64)reg->s32_min_value | |
| 581 | − : (u32)reg->s32_min_value, reg->s32_min_value == S32_MIN}, | |
| 579 | + is_snum_decimal((s64)reg_s32_min(reg)) | |
| 580 | + ? (s64)reg_s32_min(reg) | |
| 581 | + : (u32)reg_s32_min(reg), reg_s32_min(reg) == S32_MIN}, | |
| 582 | 582 | {"smax32", |
| 583 | − is_snum_decimal((s64)reg->s32_max_value) | |
| 584 | − ? (s64)reg->s32_max_value | |
| 585 | − : (u32)reg->s32_max_value, reg->s32_max_value == S32_MAX}, | |
| 586 | − {"umin32", reg->u32_min_value, reg->u32_min_value == 0}, | |
| 587 | − {"umax32", reg->u32_max_value, reg->u32_max_value == U32_MAX}, | |
| 583 | + is_snum_decimal((s64)reg_s32_max(reg)) | |
| 584 | + ? (s64)reg_s32_max(reg) | |
| 585 | + : (u32)reg_s32_max(reg), reg_s32_max(reg) == S32_MAX}, | |
| 586 | + {"umin32", reg_u32_min(reg), reg_u32_min(reg) == 0}, | |
| 587 | + {"umax32", reg_u32_max(reg), reg_u32_max(reg) == U32_MAX}, | |
| 588 | 588 | }, *m1, *m2, *mend = &minmaxs[ARRAY_SIZE(minmaxs)]; |
| 589 | 589 | bool neg1, neg2; |
| 590 | 590 | |
| @@ -665,8 +665,8 @@ static void print_reg_state(struct bpf_verifier_env *env, · 74 unmodified lines | ||
| 665 | 665 | verbose_a("id=%d", reg->id & ~BPF_ADD_CONST); |
| 666 | 666 | if (reg->id & BPF_ADD_CONST) |
| 667 | 667 | verbose(env, "%+d", reg->delta); |
| 668 | − if (reg->ref_obj_id) | |
| 669 | − verbose_a("ref_obj_id=%d", reg->ref_obj_id); | |
| 668 | + if (reg->parent_id) | |
| 669 | + verbose_a("parent_id=%d", reg->parent_id); | |
| 670 | 670 | if (type_is_non_owning_ref(reg->type)) |
| 671 | 671 | verbose_a("%s", "non_own_ref"); |
| 672 | 672 | if (type_is_map_ptr(t)) { |
| @@ -768,21 +768,19 @@ void print_verifier_state(struct bpf_verifier_env *env, const struct bpf_verifie · 95 unmodified lines | ||
| 768 | 768 | verbose(env, "=dynptr_%s(", dynptr_type_str(reg->dynptr.type)); |
| 769 | 769 | if (reg->id) |
| 770 | 770 | verbose_a("id=%d", reg->id); |
| 771 | − if (reg->ref_obj_id) | |
| 772 | − verbose_a("ref_id=%d", reg->ref_obj_id); | |
| 773 | − if (reg->dynptr_id) | |
| 774 | − verbose_a("dynptr_id=%d", reg->dynptr_id); | |
| 771 | + if (reg->parent_id) | |
| 772 | + verbose_a("parent_id=%d", reg->parent_id); | |
| 775 | 773 | verbose(env, ")"); |
| 776 | 774 | break; |
| 777 | 775 | case STACK_ITER: |
| 778 | − /* only main slot has ref_obj_id set; skip others */ | |
| 779 | − if (!reg->ref_obj_id) | |
| 776 | + /* only main slot has id set; skip others */ | |
| 777 | + if (!reg->id) | |
| 780 | 778 | continue; |
| 781 | 779 | |
| 782 | − verbose(env, " fp%d=iter_%s(ref_id=%d,state=%s,depth=%u)", | |
| 780 | + verbose(env, " fp%d=iter_%s(id=%d,state=%s,depth=%u)", | |
| 783 | 781 | (-i - 1) * BPF_REG_SIZE, |
| 784 | 782 | iter_type_str(reg->iter.btf, reg->iter.btf_id), |
| 785 | − reg->ref_obj_id, iter_state_str(reg->iter.state), | |
| 783 | + reg->id, iter_state_str(reg->iter.state), | |
| 786 | 784 | reg->iter.depth); |
| 787 | 785 | break; |
| 788 | 786 | case STACK_MISC: |
| @@ -825,3 +823,81 @@ void print_insn_state(struct bpf_verifier_env *env, const struct bpf_verifier_st · 36 unmodified lines | ||
| 825 | 823 | } |
| 826 | 824 | print_verifier_state(env, vstate, frameno, false); |
| 827 | 825 | } |
| 826 | + | |
| 827 | +int bpf_log_attr_init(struct bpf_log_attr *log, u64 log_buf, u32 log_size, u32 log_level, | |
| 828 | + u32 offsetof_log_true_size, bpfptr_t uattr, struct bpf_common_attr *common, | |
| 829 | + bpfptr_t uattr_common, u32 size_common) | |
| 830 | +{ | |
| 831 | + char __user *ubuf_common = u64_to_user_ptr(common->log_buf); | |
| 832 | + char __user *ubuf = u64_to_user_ptr(log_buf); | |
| 833 | + | |
| 834 | + if (!bpf_verifier_log_attr_valid(common->log_level, ubuf_common, common->log_size) || | |
| 835 | + !bpf_verifier_log_attr_valid(log_level, ubuf, log_size)) | |
| 836 | + return -EINVAL; | |
| 837 | + | |
| 838 | + if (ubuf && ubuf_common && (ubuf != ubuf_common || log_size != common->log_size || | |
| 839 | + log_level != common->log_level)) | |
| 840 | + return -EINVAL; | |
| 841 | + | |
| 842 | + memset(log, 0, sizeof(*log)); | |
| 843 | + log->ubuf = ubuf; | |
| 844 | + log->size = log_size; | |
| 845 | + log->level = log_level; | |
| 846 | + log->offsetof_true_size = offsetof_log_true_size; | |
| 847 | + log->uattr = uattr; | |
| 848 | + | |
| 849 | + if (!ubuf && ubuf_common) { | |
| 850 | + log->ubuf = ubuf_common; | |
| 851 | + log->size = common->log_size; | |
| 852 | + log->level = common->log_level; | |
| 853 | + log->uattr = uattr_common; | |
| 854 | + log->offsetof_true_size = 0; | |
| 855 | + if (size_common >= offsetofend(struct bpf_common_attr, log_true_size)) | |
| 856 | + log->offsetof_true_size = offsetof(struct bpf_common_attr, log_true_size); | |
| 857 | + } | |
| 858 | + return 0; | |
| 859 | +} | |
| 860 | + | |
| 861 | +struct bpf_verifier_log *bpf_log_attr_create_vlog(struct bpf_log_attr *attr_log, | |
| 862 | + struct bpf_common_attr *common, bpfptr_t uattr, | |
| 863 | + u32 size) | |
| 864 | +{ | |
| 865 | + struct bpf_verifier_log *log; | |
| 866 | + int err; | |
| 867 | + | |
| 868 | + memset(attr_log, 0, sizeof(*attr_log)); | |
| 869 | + attr_log->uattr = uattr; | |
| 870 | + if (size >= offsetofend(struct bpf_common_attr, log_true_size)) | |
| 871 | + attr_log->offsetof_true_size = offsetof(struct bpf_common_attr, log_true_size); | |
| 872 | + | |
| 873 | + if (!size) | |
| 874 | + return NULL; | |
| 875 | + | |
| 876 | + log = kzalloc_obj(*log, GFP_KERNEL); | |
| 877 | + if (!log) | |
| 878 | + return ERR_PTR(-ENOMEM); | |
| 879 | + | |
| 880 | + err = bpf_vlog_init(log, common->log_level, u64_to_user_ptr(common->log_buf), | |
| 881 | + common->log_size); | |
| 882 | + if (err) { | |
| 883 | + kfree(log); | |
| 884 | + return ERR_PTR(err); | |
| 885 | + } | |
| 886 | + | |
| 887 | + return log; | |
| 888 | +} | |
| 889 | + | |
| 890 | +int bpf_log_attr_finalize(struct bpf_log_attr *attr, struct bpf_verifier_log *log) | |
| 891 | +{ | |
| 892 | + u32 log_true_size; | |
| 893 | + int err; | |
| 894 | + | |
| 895 | + err = bpf_vlog_finalize(log, &log_true_size); | |
| 896 | + | |
| 897 | + if (attr->offsetof_true_size && | |
| 898 | + copy_to_bpfptr_offset(attr->uattr, attr->offsetof_true_size, &log_true_size, | |
| 899 | + sizeof(log_true_size))) | |
| 900 | + return -EFAULT; | |
| 901 | + | |
| 902 | + return err; | |
| 903 | +} | |
| @@ -246,7 +246,7 @@ static void *trie_lookup_elem(struct bpf_map *map, void *_key) · 245 unmodified lines | ||
| 246 | 246 | |
| 247 | 247 | /* Start walking the trie from the root node ... */ |
| 248 | 248 | |
| 249 | − for (node = rcu_dereference_check(trie->root, rcu_read_lock_bh_held()); | |
| 249 | + for (node = rcu_dereference_check(trie->root, bpf_rcu_lock_held()); | |
| 250 | 250 | node;) { |
| 251 | 251 | unsigned int next_bit; |
| 252 | 252 | size_t matchlen; |
| @@ -280,7 +280,7 @@ static void *trie_lookup_elem(struct bpf_map *map, void *_key) · 27 unmodified lines | ||
| 280 | 280 | */ |
| 281 | 281 | next_bit = extract_bit(key->data, node->prefixlen); |
| 282 | 282 | node = rcu_dereference_check(node->child[next_bit], |
| 283 | − rcu_read_lock_bh_held()); | |
| 283 | + bpf_rcu_lock_held()); | |
| 284 | 284 | } |
| 285 | 285 | |
| 286 | 286 | if (!found) |
| @@ -359,7 +359,7 @@ static long trie_update_elem(struct bpf_map *map, · 72 unmodified lines | ||
| 359 | 359 | */ |
| 360 | 360 | slot = &trie->root; |
| 361 | 361 | |
| 362 | − while ((node = rcu_dereference(*slot))) { | |
| 362 | + while ((node = rcu_dereference_protected(*slot, 1))) { | |
| 363 | 363 | matchlen = longest_prefix_match(trie, node, key); |
| 364 | 364 | |
| 365 | 365 | if (node->prefixlen != matchlen || |
| @@ -482,7 +482,7 @@ static long trie_delete_elem(struct bpf_map *map, void *_key) · 116 unmodified lines | ||
| 482 | 482 | trim = &trie->root; |
| 483 | 483 | trim2 = trim; |
| 484 | 484 | parent = NULL; |
| 485 | − while ((node = rcu_dereference(*trim))) { | |
| 485 | + while ((node = rcu_dereference_protected(*trim, 1))) { | |
| 486 | 486 | matchlen = longest_prefix_match(trie, node, key); |
| 487 | 487 | |
| 488 | 488 | if (node->prefixlen != matchlen || |
| @@ -20,7 +20,8 @@ struct bpf_map *bpf_map_meta_alloc(int inner_map_ufd) · 19 unmodified lines | ||
| 20 | 20 | /* Does not support >1 level map-in-map */ |
| 21 | 21 | if (inner_map->inner_map_meta) |
| 22 | 22 | return ERR_PTR(-EINVAL); |
| 23 | − | |
| 23 | + if (inner_map->excl_prog_sha) | |
| 24 | + return ERR_PTR(-ENOTSUPP); | |
| 24 | 25 | if (!inner_map->ops->map_meta_equal) |
| 25 | 26 | return ERR_PTR(-ENOTSUPP); |
| 26 | 27 | |
| @@ -101,6 +102,8 @@ void *bpf_map_fd_get_ptr(struct bpf_map *map, · 74 unmodified lines | ||
| 101 | 102 | inner_map = __bpf_map_get(f); |
| 102 | 103 | if (IS_ERR(inner_map)) |
| 103 | 104 | return inner_map; |
| 105 | + if (inner_map->excl_prog_sha) | |
| 106 | + return ERR_PTR(-ENOTSUPP); | |
| 104 | 107 | |
| 105 | 108 | inner_map_meta = map->inner_map_meta; |
| 106 | 109 | if (inner_map_meta->ops->map_meta_equal(inner_map_meta, inner_map)) |
| @@ -112,6 +112,10 @@ static int bpf_iter_attach_map(struct bpf_prog *prog, · 111 unmodified lines | ||
| 112 | 112 | map = bpf_map_get_with_uref(linfo->map.map_fd); |
| 113 | 113 | if (IS_ERR(map)) |
| 114 | 114 | return PTR_ERR(map); |
| 115 | + if (map->excl_prog_sha) { | |
| 116 | + err = -EPERM; | |
| 117 | + goto put_map; | |
| 118 | + } | |
| 115 | 119 | |
| 116 | 120 | if (map->map_type == BPF_MAP_TYPE_PERCPU_HASH || |
| 117 | 121 | map->map_type == BPF_MAP_TYPE_LRU_PERCPU_HASH || |
| @@ -119,7 +123,8 @@ static int bpf_iter_attach_map(struct bpf_prog *prog, · 1 unmodified line | ||
| 119 | 123 | is_percpu = true; |
| 120 | 124 | else if (map->map_type != BPF_MAP_TYPE_HASH && |
| 121 | 125 | map->map_type != BPF_MAP_TYPE_LRU_HASH && |
| 122 | − map->map_type != BPF_MAP_TYPE_ARRAY) | |
| 126 | + map->map_type != BPF_MAP_TYPE_ARRAY && | |
| 127 | + map->map_type != BPF_MAP_TYPE_RHASH) | |
| 123 | 128 | goto put_map; |
| 124 | 129 | |
| 125 | 130 | key_acc_size = prog->aux->max_rdonly_access; |
| @@ -9,6 +9,7 @@ · 8 unmodified lines | ||
| 9 | 9 | #include <linux/perf_event.h> |
| 10 | 10 | #include <linux/btf_ids.h> |
| 11 | 11 | #include <linux/buildid.h> |
| 12 | +#include <linux/mmap_lock.h> | |
| 12 | 13 | #include "percpu_freelist.h" |
| 13 | 14 | #include "mmap_unlock_work.h" |
| 14 | 15 | |
| @@ -152,6 +153,180 @@ static int fetch_build_id(struct vm_area_struct *vma, unsigned char *build_id, b · 137 unmodified lines | ||
| 152 | 153 | : build_id_parse_nofault(vma, build_id, NULL); |
| 153 | 154 | } |
| 154 | 155 | |
| 156 | +static inline void stack_map_build_id_set_ip(struct bpf_stack_build_id *id) | |
| 157 | +{ | |
| 158 | + id->status = BPF_STACK_BUILD_ID_IP; | |
| 159 | + memset(id->build_id, 0, BUILD_ID_SIZE_MAX); | |
| 160 | +} | |
| 161 | + | |
| 162 | +static inline u64 stack_map_build_id_offset(unsigned long vm_pgoff, | |
| 163 | + unsigned long vm_start, u64 ip) | |
| 164 | +{ | |
| 165 | + return (vm_pgoff << PAGE_SHIFT) + ip - vm_start; | |
| 166 | +} | |
| 167 | + | |
| 168 | +static inline void stack_map_build_id_set_valid(struct bpf_stack_build_id *id, | |
| 169 | + u64 offset, | |
| 170 | + const unsigned char *build_id) | |
| 171 | +{ | |
| 172 | + id->status = BPF_STACK_BUILD_ID_VALID; | |
| 173 | + id->offset = offset; | |
| 174 | + if (id->build_id != build_id) | |
| 175 | + memcpy(id->build_id, build_id, BUILD_ID_SIZE_MAX); | |
| 176 | +} | |
| 177 | + | |
| 178 | +struct stack_map_vma_lock { | |
| 179 | + struct vm_area_struct *vma; | |
| 180 | + struct mm_struct *mm; | |
| 181 | +}; | |
| 182 | + | |
| 183 | +/* | |
| 184 | + * Acquire a stable read-side reference on the VMA covering @ip. | |
| 185 | + * | |
| 186 | + * With CONFIG_PER_VMA_LOCK=y this returns a VMA with its per-VMA read | |
| 187 | + * lock held and mmap_lock dropped, so the caller may sleep. | |
| 188 | + * | |
| 189 | + * With CONFIG_PER_VMA_LOCK=n it returns a VMA with mmap_lock still | |
| 190 | + * held; the caller must snapshot any fields it needs and pin vm_file | |
| 191 | + * with get_file() before stack_map_unlock_vma() drops mmap_lock, as | |
| 192 | + * the VMA may be split, merged, or freed after that. | |
| 193 | + * | |
| 194 | + * Returns NULL on failure, in which case no lock is held. | |
| 195 | + */ | |
| 196 | +static struct vm_area_struct * | |
| 197 | +stack_map_lock_vma(struct stack_map_vma_lock *lock, unsigned long ip) | |
| 198 | +{ | |
| 199 | + struct mm_struct *mm = lock->mm; | |
| 200 | + struct vm_area_struct *vma; | |
| 201 | + | |
| 202 | + /* noop under !CONFIG_PER_VMA_LOCK */ | |
| 203 | + vma = lock_vma_under_rcu(mm, ip); | |
| 204 | + if (vma) { | |
| 205 | + lock->vma = vma; | |
| 206 | + return vma; | |
| 207 | + } | |
| 208 | + | |
| 209 | + /* | |
| 210 | + * Taking mmap_read_lock() is unsafe here, because the caller BPF | |
| 211 | + * program might already hold it, causing a deadlock. | |
| 212 | + */ | |
| 213 | + if (!mmap_read_trylock(mm)) | |
| 214 | + return NULL; | |
| 215 | + | |
| 216 | + vma = vma_lookup(mm, ip); | |
| 217 | + if (!vma) { | |
| 218 | + mmap_read_unlock(mm); | |
| 219 | + return NULL; | |
| 220 | + } | |
| 221 | + | |
| 222 | +#ifdef CONFIG_PER_VMA_LOCK | |
| 223 | + if (!vma_start_read_locked(vma)) { | |
| 224 | + mmap_read_unlock(mm); | |
| 225 | + return NULL; | |
| 226 | + } | |
| 227 | + mmap_read_unlock(mm); | |
| 228 | +#endif | |
| 229 | + | |
| 230 | + lock->vma = vma; | |
| 231 | + return vma; | |
| 232 | +} | |
| 233 | + | |
| 234 | +static void stack_map_unlock_vma(struct stack_map_vma_lock *lock) | |
| 235 | +{ | |
| 236 | +#ifdef CONFIG_PER_VMA_LOCK | |
| 237 | + vma_end_read(lock->vma); | |
| 238 | +#else | |
| 239 | + mmap_read_unlock(lock->mm); | |
| 240 | +#endif | |
| 241 | + lock->vma = NULL; | |
| 242 | +} | |
| 243 | + | |
| 244 | +static void stack_map_get_build_id_offset_sleepable(struct bpf_stack_build_id *id_offs, | |
| 245 | + u32 trace_nr) | |
| 246 | +{ | |
| 247 | + struct mm_struct *mm = current->mm; | |
| 248 | + struct stack_map_vma_lock lock = { .mm = mm }; | |
| 249 | + struct { | |
| 250 | + struct file *file; | |
| 251 | + const unsigned char *build_id; | |
| 252 | + unsigned long vm_start; | |
| 253 | + unsigned long vm_end; | |
| 254 | + unsigned long vm_pgoff; | |
| 255 | + } cache = {}; | |
| 256 | + unsigned long vm_pgoff, vm_start, vm_end; | |
| 257 | + struct vm_area_struct *vma; | |
| 258 | + struct file *file; | |
| 259 | + u64 offset; | |
| 260 | + u64 ip; | |
| 261 | + | |
| 262 | + for (u32 i = 0; i < trace_nr; i++) { | |
| 263 | + ip = READ_ONCE(id_offs[i].ip); | |
| 264 | + | |
| 265 | + /* | |
| 266 | + * Range cache fast path: if ip falls within the previously | |
| 267 | + * resolved VMA range, reuse the cache build_id without | |
| 268 | + * re-acquiring the VMA lock. | |
| 269 | + */ | |
| 270 | + if (cache.build_id && ip >= cache.vm_start && ip < cache.vm_end) { | |
| 271 | + offset = stack_map_build_id_offset(cache.vm_pgoff, cache.vm_start, ip); | |
| 272 | + stack_map_build_id_set_valid(&id_offs[i], offset, cache.build_id); | |
| 273 | + continue; | |
| 274 | + } | |
| 275 | + | |
| 276 | + vma = stack_map_lock_vma(&lock, ip); | |
| 277 | + if (!vma) { | |
| 278 | + stack_map_build_id_set_ip(&id_offs[i]); | |
| 279 | + continue; | |
| 280 | + } | |
| 281 | + if (vma_is_anonymous(vma) || !vma->vm_file) { | |
| 282 | + stack_map_build_id_set_ip(&id_offs[i]); | |
| 283 | + stack_map_unlock_vma(&lock); | |
| 284 | + continue; | |
| 285 | + } | |
| 286 | + | |
| 287 | + file = vma->vm_file; | |
| 288 | + vm_pgoff = vma->vm_pgoff; | |
| 289 | + vm_start = vma->vm_start; | |
| 290 | + vm_end = vma->vm_end; | |
| 291 | + offset = stack_map_build_id_offset(vm_pgoff, vm_start, ip); | |
| 292 | + | |
| 293 | + /* | |
| 294 | + * Same backing file as previous (e.g. different VMAs | |
| 295 | + * of the same ELF binary). Reuse the cache build_id. | |
| 296 | + */ | |
| 297 | + if (file == cache.file) { | |
| 298 | + stack_map_unlock_vma(&lock); | |
| 299 | + stack_map_build_id_set_valid(&id_offs[i], offset, cache.build_id); | |
| 300 | + cache.vm_start = vm_start; | |
| 301 | + cache.vm_end = vm_end; | |
| 302 | + cache.vm_pgoff = vm_pgoff; | |
| 303 | + continue; | |
| 304 | + } | |
| 305 | + | |
| 306 | + file = get_file(file); | |
| 307 | + stack_map_unlock_vma(&lock); | |
| 308 | + | |
| 309 | + /* build_id_parse_file() may block on filesystem reads */ | |
| 310 | + if (build_id_parse_file(file, id_offs[i].build_id, NULL)) { | |
| 311 | + stack_map_build_id_set_ip(&id_offs[i]); | |
| 312 | + fput(file); | |
| 313 | + continue; | |
| 314 | + } | |
| 315 | + | |
| 316 | + stack_map_build_id_set_valid(&id_offs[i], offset, id_offs[i].build_id); | |
| 317 | + if (cache.file) | |
| 318 | + fput(cache.file); | |
| 319 | + cache.file = file; | |
| 320 | + cache.build_id = id_offs[i].build_id; | |
| 321 | + cache.vm_start = vm_start; | |
| 322 | + cache.vm_end = vm_end; | |
| 323 | + cache.vm_pgoff = vm_pgoff; | |
| 324 | + } | |
| 325 | + | |
| 326 | + if (cache.file) | |
| 327 | + fput(cache.file); | |
| 328 | +} | |
| 329 | + | |
| 155 | 330 | /* |
| 156 | 331 | * Expects all id_offs[i].ip values to be set to correct initial IPs. |
| 157 | 332 | * They will be subsequently: |
| @@ -165,44 +340,50 @@ static int fetch_build_id(struct vm_area_struct *vma, unsigned char *build_id, b · 7 unmodified lines | ||
| 165 | 340 | static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs, |
| 166 | 341 | u32 trace_nr, bool user, bool may_fault) |
| 167 | 342 | { |
| 168 | − int i; | |
| 169 | 343 | struct mmap_unlock_irq_work *work = NULL; |
| 170 | 344 | bool irq_work_busy = bpf_mmap_unlock_get_irq_work(&work); |
| 345 | + bool has_user_ctx = user && current && current->mm; | |
| 171 | 346 | struct vm_area_struct *vma, *prev_vma = NULL; |
| 172 | − const char *prev_build_id; | |
| 347 | + const unsigned char *prev_build_id = NULL; | |
| 348 | + int i; | |
| 349 | + | |
| 350 | + if (may_fault && has_user_ctx) { | |
| 351 | + stack_map_get_build_id_offset_sleepable(id_offs, trace_nr); | |
| 352 | + return; | |
| 353 | + } | |
| 173 | 354 | |
| 174 | 355 | /* If the irq_work is in use, fall back to report ips. Same |
| 175 | 356 | * fallback is used for kernel stack (!user) on a stackmap with |
| 176 | 357 | * build_id. |
| 177 | 358 | */ |
| 178 | − if (!user || !current || !current->mm || irq_work_busy || | |
| 179 | − !mmap_read_trylock(current->mm)) { | |
| 359 | + if (!has_user_ctx || irq_work_busy || !mmap_read_trylock(current->mm)) { | |
| 180 | 360 | /* cannot access current->mm, fall back to ips */ |
| 181 | − for (i = 0; i < trace_nr; i++) { | |
| 182 | − id_offs[i].status = BPF_STACK_BUILD_ID_IP; | |
| 183 | − memset(id_offs[i].build_id, 0, BUILD_ID_SIZE_MAX); | |
| 184 | − } | |
| 361 | + for (i = 0; i < trace_nr; i++) | |
| 362 | + stack_map_build_id_set_ip(&id_offs[i]); | |
| 185 | 363 | return; |
| 186 | 364 | } |
| 187 | 365 | |
| 188 | 366 | for (i = 0; i < trace_nr; i++) { |
| 189 | 367 | u64 ip = READ_ONCE(id_offs[i].ip); |
| 368 | + u64 offset; | |
| 190 | 369 | |
| 191 | − if (range_in_vma(prev_vma, ip, ip)) { | |
| 370 | + if (prev_build_id && range_in_vma(prev_vma, ip, ip)) { | |
| 192 | 371 | vma = prev_vma; |
| 193 | − memcpy(id_offs[i].build_id, prev_build_id, BUILD_ID_SIZE_MAX); | |
| 194 | − goto build_id_valid; | |
| 372 | + offset = stack_map_build_id_offset(vma->vm_pgoff, vma->vm_start, ip); | |
| 373 | + stack_map_build_id_set_valid(&id_offs[i], offset, prev_build_id); | |
| 374 | + continue; | |
| 195 | 375 | } |
| 196 | 376 | vma = find_vma(current->mm, ip); |
| 197 | − if (!vma || fetch_build_id(vma, id_offs[i].build_id, may_fault)) { | |
| 377 | + if (!vma || vma_is_anonymous(vma) || | |
| 378 | + fetch_build_id(vma, id_offs[i].build_id, may_fault)) { | |
| 198 | 379 | /* per entry fall back to ips */ |
| 199 | − id_offs[i].status = BPF_STACK_BUILD_ID_IP; | |
| 200 | − memset(id_offs[i].build_id, 0, BUILD_ID_SIZE_MAX); | |
| 380 | + stack_map_build_id_set_ip(&id_offs[i]); | |
| 381 | + prev_vma = vma; | |
| 382 | + prev_build_id = NULL; | |
| 201 | 383 | continue; |
| 202 | 384 | } |
| 203 | −build_id_valid: | |
| 204 | − id_offs[i].offset = (vma->vm_pgoff << PAGE_SHIFT) + ip - vma->vm_start; | |
| 205 | − id_offs[i].status = BPF_STACK_BUILD_ID_VALID; | |
| 385 | + offset = stack_map_build_id_offset(vma->vm_pgoff, vma->vm_start, ip); | |
| 386 | + stack_map_build_id_set_valid(&id_offs[i], offset, id_offs[i].build_id); | |
| 206 | 387 | prev_vma = vma; |
| 207 | 388 | prev_build_id = id_offs[i].build_id; |
| 208 | 389 | } |
| @@ -2,6 +2,7 @@ · 1 unmodified line | ||
| 2 | 2 | /* Copyright (c) 2026 Meta Platforms, Inc. and affiliates. */ |
| 3 | 3 | #include <linux/bpf.h> |
| 4 | 4 | #include <linux/bpf_verifier.h> |
| 5 | +#include <linux/cnum.h> | |
| 5 | 6 | #include <linux/filter.h> |
| 6 | 7 | |
| 7 | 8 | #define verbose(env, fmt, args...) bpf_verifier_log_write(env, fmt, ##args) |
| @@ -301,14 +302,8 @@ int bpf_update_branch_counts(struct bpf_verifier_env *env, struct bpf_verifier_s · 293 unmodified lines | ||
| 301 | 302 | static bool range_within(const struct bpf_reg_state *old, |
| 302 | 303 | const struct bpf_reg_state *cur) |
| 303 | 304 | { |
| 304 | − return old->umin_value <= cur->umin_value && | |
| 305 | − old->umax_value >= cur->umax_value && | |
| 306 | − old->smin_value <= cur->smin_value && | |
| 307 | − old->smax_value >= cur->smax_value && | |
| 308 | − old->u32_min_value <= cur->u32_min_value && | |
| 309 | − old->u32_max_value >= cur->u32_max_value && | |
| 310 | − old->s32_min_value <= cur->s32_min_value && | |
| 311 | − old->s32_max_value >= cur->s32_max_value; | |
| 305 | + return cnum64_is_subset(old->r64, cur->r64) && | |
| 306 | + cnum32_is_subset(old->r32, cur->r32); | |
| 312 | 307 | } |
| 313 | 308 | |
| 314 | 309 | /* If in the old state two registers had the same id, then they need to have |
| @@ -348,8 +343,12 @@ static bool check_ids(u32 old_id, u32 cur_id, struct bpf_idmap *idmap) · 33 unmodified lines | ||
| 348 | 343 | return true; |
| 349 | 344 | } |
| 350 | 345 | |
| 351 | − /* We ran out of idmap slots, which should be impossible */ | |
| 352 | − WARN_ON_ONCE(1); | |
| 346 | + /* | |
| 347 | + * idmap slots are bounded by the number of registers and stack slots. | |
| 348 | + * Since referenced dynptrs acquire intermediate references that do | |
| 349 | + * not live in either, so the map can be exhausted. Since it is unlikely, | |
| 350 | + * fail the verification by treating the states as not equivalent. | |
| 351 | + */ | |
| 353 | 352 | return false; |
| 354 | 353 | } |
| 355 | 354 | |
| @@ -494,7 +493,7 @@ static bool regs_exact(const struct bpf_reg_state *rold, · 138 unmodified lines | ||
| 494 | 493 | { |
| 495 | 494 | return memcmp(rold, rcur, offsetof(struct bpf_reg_state, id)) == 0 && |
| 496 | 495 | check_ids(rold->id, rcur->id, idmap) && |
| 497 | − check_ids(rold->ref_obj_id, rcur->ref_obj_id, idmap); | |
| 496 | + check_ids(rold->parent_id, rcur->parent_id, idmap); | |
| 498 | 497 | } |
| 499 | 498 | |
| 500 | 499 | enum exact_level { |
| @@ -619,7 +618,7 @@ static bool regsafe(struct bpf_verifier_env *env, struct bpf_reg_state *rold, · 118 unmodified lines | ||
| 619 | 618 | range_within(rold, rcur) && |
| 620 | 619 | tnum_in(rold->var_off, rcur->var_off) && |
| 621 | 620 | check_ids(rold->id, rcur->id, idmap) && |
| 622 | − check_ids(rold->ref_obj_id, rcur->ref_obj_id, idmap); | |
| 621 | + check_ids(rold->parent_id, rcur->parent_id, idmap); | |
| 623 | 622 | case PTR_TO_PACKET_META: |
| 624 | 623 | case PTR_TO_PACKET: |
| 625 | 624 | /* We must have at least as much range as the old ptr |
| @@ -799,7 +798,8 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old, · 173 unmodified lines | ||
| 799 | 798 | cur_reg = &cur->stack[spi].spilled_ptr; |
| 800 | 799 | if (old_reg->dynptr.type != cur_reg->dynptr.type || |
| 801 | 800 | old_reg->dynptr.first_slot != cur_reg->dynptr.first_slot || |
| 802 | − !check_ids(old_reg->ref_obj_id, cur_reg->ref_obj_id, idmap)) | |
| 801 | + !check_ids(old_reg->id, cur_reg->id, idmap) || | |
| 802 | + !check_ids(old_reg->parent_id, cur_reg->parent_id, idmap)) | |
| 803 | 803 | return false; |
| 804 | 804 | break; |
| 805 | 805 | case STACK_ITER: |
| @@ -815,13 +815,13 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old, · 9 unmodified lines | ||
| 815 | 815 | old_reg->iter.btf_id != cur_reg->iter.btf_id || |
| 816 | 816 | old_reg->iter.state != cur_reg->iter.state || |
| 817 | 817 | /* ignore {old_reg,cur_reg}->iter.depth, see above */ |
| 818 | − !check_ids(old_reg->ref_obj_id, cur_reg->ref_obj_id, idmap)) | |
| 818 | + !check_ids(old_reg->id, cur_reg->id, idmap)) | |
| 819 | 819 | return false; |
| 820 | 820 | break; |
| 821 | 821 | case STACK_IRQ_FLAG: |
| 822 | 822 | old_reg = &old->stack[spi].spilled_ptr; |
| 823 | 823 | cur_reg = &cur->stack[spi].spilled_ptr; |
| 824 | − if (!check_ids(old_reg->ref_obj_id, cur_reg->ref_obj_id, idmap) || | |
| 824 | + if (!check_ids(old_reg->id, cur_reg->id, idmap) || | |
| 825 | 825 | old_reg->irq.kfunc_class != cur_reg->irq.kfunc_class) |
| 826 | 826 | return false; |
| 827 | 827 | break; |
| @@ -838,6 +838,32 @@ static bool stacksafe(struct bpf_verifier_env *env, struct bpf_func_state *old, · 10 unmodified lines | ||
| 838 | 838 | return true; |
| 839 | 839 | } |
| 840 | 840 | |
| 841 | +/* | |
| 842 | + * Compare stack arg slots between old and current states. | |
| 843 | + * Outgoing stack args are path-local state and must agree for pruning. | |
| 844 | + */ | |
| 845 | +static bool stack_arg_safe(struct bpf_verifier_env *env, struct bpf_func_state *old, | |
| 846 | + struct bpf_func_state *cur, struct bpf_idmap *idmap, | |
| 847 | + enum exact_level exact) | |
| 848 | +{ | |
| 849 | + int i, nslots; | |
| 850 | + | |
| 851 | + nslots = max(old->out_stack_arg_cnt, cur->out_stack_arg_cnt); | |
| 852 | + for (i = 0; i < nslots; i++) { | |
| 853 | + struct bpf_reg_state *old_arg, *cur_arg; | |
| 854 | + struct bpf_reg_state not_init = { .type = NOT_INIT }; | |
| 855 | + | |
| 856 | + old_arg = i < old->out_stack_arg_cnt ? | |
| 857 | + &old->stack_arg_regs[i] : ¬_init; | |
| 858 | + cur_arg = i < cur->out_stack_arg_cnt ? | |
| 859 | + &cur->stack_arg_regs[i] : ¬_init; | |
| 860 | + if (!regsafe(env, old_arg, cur_arg, idmap, exact)) | |
| 861 | + return false; | |
| 862 | + } | |
| 863 | + | |
| 864 | + return true; | |
| 865 | +} | |
| 866 | + | |
| 841 | 867 | static bool refsafe(struct bpf_verifier_state *old, struct bpf_verifier_state *cur, |
| 842 | 868 | struct bpf_idmap *idmap) |
| 843 | 869 | { |
| @@ -868,6 +894,9 @@ static bool refsafe(struct bpf_verifier_state *old, struct bpf_verifier_state *c · 24 unmodified lines | ||
| 868 | 894 | return false; |
| 869 | 895 | switch (old->refs[i].type) { |
| 870 | 896 | case REF_TYPE_PTR: |
| 897 | + if (!check_ids(old->refs[i].parent_id, cur->refs[i].parent_id, idmap)) | |
| 898 | + return false; | |
| 899 | + break; | |
| 871 | 900 | case REF_TYPE_IRQ: |
| 872 | 901 | break; |
| 873 | 902 | case REF_TYPE_LOCK: |
| @@ -920,6 +949,9 @@ static bool func_states_equal(struct bpf_verifier_env *env, struct bpf_func_stat · 46 unmodified lines | ||
| 920 | 949 | if (old->callback_depth > cur->callback_depth) |
| 921 | 950 | return false; |
| 922 | 951 | |
| 952 | + if (!old->no_stack_arg_load && cur->no_stack_arg_load) | |
| 953 | + return false; | |
| 954 | + | |
| 923 | 955 | for (i = 0; i < MAX_BPF_REG; i++) |
| 924 | 956 | if (((1 << i) & live_regs) && |
| 925 | 957 | !regsafe(env, &old->regs[i], &cur->regs[i], |
| @@ -929,6 +961,9 @@ static bool func_states_equal(struct bpf_verifier_env *env, struct bpf_func_stat · 3 unmodified lines | ||
| 929 | 961 | if (!stacksafe(env, old, cur, &env->idmap_scratch, exact)) |
| 930 | 962 | return false; |
| 931 | 963 | |
| 964 | + if (!stack_arg_safe(env, old, cur, &env->idmap_scratch, exact)) | |
| 965 | + return false; | |
| 966 | + | |
| 932 | 967 | return true; |
| 933 | 968 | } |
| 934 | 969 | |
| @@ -1376,7 +1411,7 @@ hit: · 441 unmodified lines | ||
| 1376 | 1411 | */ |
| 1377 | 1412 | err = 0; |
| 1378 | 1413 | if (bpf_is_jmp_point(env, env->insn_idx)) |
| 1379 | − err = bpf_push_jmp_history(env, cur, 0, 0); | |
| 1414 | + err = bpf_push_jmp_history(env, cur, 0, 0, 0, 0); | |
| 1380 | 1415 | err = err ? : propagate_precision(env, &sl->state, cur, NULL); |
| 1381 | 1416 | if (err) |
| 1382 | 1417 | return err; |
| @@ -41,6 +41,7 @@ · 40 unmodified lines | ||
| 41 | 41 | #include <linux/overflow.h> |
| 42 | 42 | #include <linux/cookie.h> |
| 43 | 43 | #include <linux/verification.h> |
| 44 | +#include <linux/btf_ids.h> | |
| 44 | 45 | |
| 45 | 46 | #include <net/netfilter/nf_bpf_link.h> |
| 46 | 47 | #include <net/netkit.h> |
| @@ -807,6 +808,11 @@ void bpf_obj_free_task_work(const struct btf_record *rec, void *obj) · 760 unmodified lines | ||
| 807 | 808 | bpf_task_work_cancel_and_free(obj + rec->task_work_off); |
| 808 | 809 | } |
| 809 | 810 | |
| 811 | +void bpf_obj_cancel_fields(struct bpf_map *map, void *obj) | |
| 812 | +{ | |
| 813 | + bpf_map_free_internal_structs(map, obj); | |
| 814 | +} | |
| 815 | + | |
| 810 | 816 | void bpf_obj_free_fields(const struct btf_record *rec, void *obj) |
| 811 | 817 | { |
| 812 | 818 | const struct btf_field *fields; |
| @@ -1280,6 +1286,7 @@ static int map_check_btf(struct bpf_map *map, struct bpf_token *token, · 467 unmodified lines | ||
| 1280 | 1286 | case BPF_SPIN_LOCK: |
| 1281 | 1287 | case BPF_RES_SPIN_LOCK: |
| 1282 | 1288 | if (map->map_type != BPF_MAP_TYPE_HASH && |
| 1289 | + map->map_type != BPF_MAP_TYPE_RHASH && | |
| 1283 | 1290 | map->map_type != BPF_MAP_TYPE_ARRAY && |
| 1284 | 1291 | map->map_type != BPF_MAP_TYPE_CGROUP_STORAGE && |
| 1285 | 1292 | map->map_type != BPF_MAP_TYPE_SK_STORAGE && |
| @@ -1294,6 +1301,7 @@ static int map_check_btf(struct bpf_map *map, struct bpf_token *token, · 8 unmodified lines | ||
| 1294 | 1301 | case BPF_WORKQUEUE: |
| 1295 | 1302 | case BPF_TASK_WORK: |
| 1296 | 1303 | if (map->map_type != BPF_MAP_TYPE_HASH && |
| 1304 | + map->map_type != BPF_MAP_TYPE_RHASH && | |
| 1297 | 1305 | map->map_type != BPF_MAP_TYPE_LRU_HASH && |
| 1298 | 1306 | map->map_type != BPF_MAP_TYPE_ARRAY) { |
| 1299 | 1307 | ret = -EOPNOTSUPP; |
| @@ -1305,6 +1313,7 @@ static int map_check_btf(struct bpf_map *map, struct bpf_token *token, · 5 unmodified lines | ||
| 1305 | 1313 | case BPF_KPTR_PERCPU: |
| 1306 | 1314 | case BPF_REFCOUNT: |
| 1307 | 1315 | if (map->map_type != BPF_MAP_TYPE_HASH && |
| 1316 | + map->map_type != BPF_MAP_TYPE_RHASH && | |
| 1308 | 1317 | map->map_type != BPF_MAP_TYPE_PERCPU_HASH && |
| 1309 | 1318 | map->map_type != BPF_MAP_TYPE_LRU_HASH && |
| 1310 | 1319 | map->map_type != BPF_MAP_TYPE_LRU_PERCPU_HASH && |
| @@ -1359,7 +1368,8 @@ free_map_tab: · 48 unmodified lines | ||
| 1359 | 1368 | |
| 1360 | 1369 | #define BPF_MAP_CREATE_LAST_FIELD excl_prog_hash_size |
| 1361 | 1370 | /* called via syscall */ |
| 1362 | −static int map_create(union bpf_attr *attr, bpfptr_t uattr) | |
| 1371 | +static int map_create_alloc(union bpf_attr *attr, bpfptr_t uattr, struct bpf_verifier_log *log, | |
| 1372 | + struct bpf_map **mapp, struct bpf_token **tokenp) | |
| 1363 | 1373 | { |
| 1364 | 1374 | const struct bpf_map_ops *ops; |
| 1365 | 1375 | struct bpf_token *token = NULL; |
| @@ -1367,12 +1377,13 @@ static int map_create(union bpf_attr *attr, bpfptr_t uattr) · 1 unmodified line | ||
| 1367 | 1377 | u32 map_type = attr->map_type; |
| 1368 | 1378 | struct bpf_map *map; |
| 1369 | 1379 | bool token_flag; |
| 1370 | − int f_flags; | |
| 1371 | 1380 | int err; |
| 1372 | 1381 | |
| 1373 | 1382 | err = CHECK_ATTR(BPF_MAP_CREATE); |
| 1374 | − if (err) | |
| 1383 | + if (err) { | |
| 1384 | + bpf_log(log, "Invalid attr.\n"); | |
| 1375 | 1385 | return -EINVAL; |
| 1386 | + } | |
| 1376 | 1387 | |
| 1377 | 1388 | /* check BPF_F_TOKEN_FD flag, remember if it's set, and then clear it |
| 1378 | 1389 | * to avoid per-map type checks tripping on unknown flag |
| @@ -1381,31 +1392,40 @@ static int map_create(union bpf_attr *attr, bpfptr_t uattr) · 2 unmodified lines | ||
| 1381 | 1392 | attr->map_flags &= ~BPF_F_TOKEN_FD; |
| 1382 | 1393 | |
| 1383 | 1394 | if (attr->btf_vmlinux_value_type_id) { |
| 1384 | − if (attr->map_type != BPF_MAP_TYPE_STRUCT_OPS || | |
| 1385 | − attr->btf_key_type_id || attr->btf_value_type_id) | |
| 1395 | + if (attr->map_type != BPF_MAP_TYPE_STRUCT_OPS) { | |
| 1396 | + bpf_log(log, "btf_vmlinux_value_type_id can only be used with struct_ops maps.\n"); | |
| 1386 | 1397 | return -EINVAL; |
| 1398 | + } | |
| 1399 | + if (attr->btf_key_type_id || attr->btf_value_type_id) { | |
| 1400 | + bpf_log(log, "btf_vmlinux_value_type_id is mutually exclusive with btf_key_type_id and btf_value_type_id.\n"); | |
| 1401 | + return -EINVAL; | |
| 1402 | + } | |
| 1387 | 1403 | } else if (attr->btf_key_type_id && !attr->btf_value_type_id) { |
| 1404 | + bpf_log(log, "Invalid btf_value_type_id.\n"); | |
| 1388 | 1405 | return -EINVAL; |
| 1389 | 1406 | } |
| 1390 | 1407 | |
| 1391 | 1408 | if (attr->map_type != BPF_MAP_TYPE_BLOOM_FILTER && |
| 1392 | 1409 | attr->map_type != BPF_MAP_TYPE_ARENA && |
| 1393 | − attr->map_extra != 0) | |
| 1410 | + attr->map_type != BPF_MAP_TYPE_RHASH && | |
| 1411 | + attr->map_extra != 0) { | |
| 1412 | + bpf_log(log, "Invalid map_extra.\n"); | |
| 1394 | 1413 | return -EINVAL; |
| 1395 | − | |
| 1396 | − f_flags = bpf_get_file_flag(attr->map_flags); | |
| 1397 | − if (f_flags < 0) | |
| 1398 | − return f_flags; | |
| 1414 | + } | |
| 1399 | 1415 | |
| 1400 | 1416 | if (numa_node != NUMA_NO_NODE && |
| 1401 | 1417 | ((unsigned int)numa_node >= nr_node_ids || |
| 1402 | − !node_online(numa_node))) | |
| 1418 | + !node_online(numa_node))) { | |
| 1419 | + bpf_log(log, "Invalid numa_node.\n"); | |
| 1403 | 1420 | return -EINVAL; |
| 1421 | + } | |
| 1404 | 1422 | |
| 1405 | 1423 | /* find map type and init map: hashtable vs rbtree vs bloom vs ... */ |
| 1406 | 1424 | map_type = attr->map_type; |
| 1407 | − if (map_type >= ARRAY_SIZE(bpf_map_types)) | |
| 1425 | + if (map_type >= ARRAY_SIZE(bpf_map_types)) { | |
| 1426 | + bpf_log(log, "Invalid map_type.\n"); | |
| 1408 | 1427 | return -EINVAL; |
| 1428 | + } | |
| 1409 | 1429 | map_type = array_index_nospec(map_type, ARRAY_SIZE(bpf_map_types)); |
| 1410 | 1430 | ops = bpf_map_types[map_type]; |
| 1411 | 1431 | if (!ops) |
| @@ -1423,8 +1443,10 @@ static int map_create(union bpf_attr *attr, bpfptr_t uattr) · 11 unmodified lines | ||
| 1423 | 1443 | |
| 1424 | 1444 | if (token_flag) { |
| 1425 | 1445 | token = bpf_token_get_from_fd(attr->map_token_fd); |
| 1426 | − if (IS_ERR(token)) | |
| 1446 | + if (IS_ERR(token)) { | |
| 1447 | + bpf_log(log, "Invalid map_token_fd.\n"); | |
| 1427 | 1448 | return PTR_ERR(token); |
| 1449 | + } | |
| 1428 | 1450 | |
| 1429 | 1451 | /* if current token doesn't grant map creation permissions, |
| 1430 | 1452 | * then we can't use this token, so ignore it and rely on |
| @@ -1457,6 +1479,7 @@ static int map_create(union bpf_attr *attr, bpfptr_t uattr) · 26 unmodified lines | ||
| 1457 | 1479 | case BPF_MAP_TYPE_CGROUP_ARRAY: |
| 1458 | 1480 | case BPF_MAP_TYPE_ARRAY_OF_MAPS: |
| 1459 | 1481 | case BPF_MAP_TYPE_HASH: |
| 1482 | + case BPF_MAP_TYPE_RHASH: | |
| 1460 | 1483 | case BPF_MAP_TYPE_PERCPU_HASH: |
| 1461 | 1484 | case BPF_MAP_TYPE_HASH_OF_MAPS: |
| 1462 | 1485 | case BPF_MAP_TYPE_RINGBUF: |
| @@ -1507,8 +1530,10 @@ static int map_create(union bpf_attr *attr, bpfptr_t uattr) · 44 unmodified lines | ||
| 1507 | 1530 | |
| 1508 | 1531 | err = bpf_obj_name_cpy(map->name, attr->map_name, |
| 1509 | 1532 | sizeof(attr->map_name)); |
| 1510 | − if (err < 0) | |
| 1533 | + if (err < 0) { | |
| 1534 | + bpf_log(log, "Invalid map_name.\n"); | |
| 1511 | 1535 | goto free_map; |
| 1536 | + } | |
| 1512 | 1537 | |
| 1513 | 1538 | preempt_disable(); |
| 1514 | 1539 | map->cookie = gen_cookie_next(&bpf_map_cookie); |
| @@ -1531,6 +1556,7 @@ static int map_create(union bpf_attr *attr, bpfptr_t uattr) · 16 unmodified lines | ||
| 1531 | 1556 | |
| 1532 | 1557 | btf = btf_get_by_fd(attr->btf_fd); |
| 1533 | 1558 | if (IS_ERR(btf)) { |
| 1559 | + bpf_log(log, "Invalid btf_fd.\n"); | |
| 1534 | 1560 | err = PTR_ERR(btf); |
| 1535 | 1561 | goto free_map; |
| 1536 | 1562 | } |
| @@ -1558,6 +1584,7 @@ static int map_create(union bpf_attr *attr, bpfptr_t uattr) · 21 unmodified lines | ||
| 1558 | 1584 | bpfptr_t uprog_hash = make_bpfptr(attr->excl_prog_hash, uattr.is_kernel); |
| 1559 | 1585 | |
| 1560 | 1586 | if (attr->excl_prog_hash_size != SHA256_DIGEST_SIZE) { |
| 1587 | + bpf_log(log, "Invalid excl_prog_hash_size.\n"); | |
| 1561 | 1588 | err = -EINVAL; |
| 1562 | 1589 | goto free_map; |
| 1563 | 1590 | } |
| @@ -1572,11 +1599,62 @@ static int map_create(union bpf_attr *attr, bpfptr_t uattr) · 8 unmodified lines | ||
| 1572 | 1599 | err = -EFAULT; |
| 1573 | 1600 | goto free_map; |
| 1574 | 1601 | } |
| 1602 | + | |
| 1603 | + /* See libbpf: emit_signature_match() */ | |
| 1604 | + BUILD_BUG_ON(offsetof(struct bpf_map, excl) != SHA256_DIGEST_SIZE); | |
| 1605 | + BUILD_BUG_ON(!__same_type(map->excl, u32)); | |
| 1606 | + BUILD_BUG_ON(offsetof(struct bpf_map, sha) != 0); | |
| 1607 | + BUILD_BUG_ON(!__same_type(map->sha, u8[SHA256_DIGEST_SIZE])); | |
| 1608 | + map->excl = 1; | |
| 1575 | 1609 | } else if (attr->excl_prog_hash_size) { |
| 1610 | + bpf_log(log, "Invalid excl_prog_hash_size.\n"); | |
| 1576 | 1611 | err = -EINVAL; |
| 1577 | 1612 | goto free_map; |
| 1578 | 1613 | } |
| 1579 | 1614 | |
| 1615 | + *mapp = map; | |
| 1616 | + *tokenp = token; | |
| 1617 | + return 0; | |
| 1618 | + | |
| 1619 | +free_map: | |
| 1620 | + bpf_map_free(map); | |
| 1621 | +put_token: | |
| 1622 | + bpf_token_put(token); | |
| 1623 | + return err; | |
| 1624 | +} | |
| 1625 | + | |
| 1626 | +static int map_create(union bpf_attr *attr, bpfptr_t uattr, struct bpf_common_attr *attr_common, | |
| 1627 | + bpfptr_t uattr_common, u32 size_common) | |
| 1628 | +{ | |
| 1629 | + struct bpf_token *token = NULL; | |
| 1630 | + struct bpf_verifier_log *log; | |
| 1631 | + struct bpf_log_attr attr_log; | |
| 1632 | + struct bpf_map *map = NULL; | |
| 1633 | + int err, ret; | |
| 1634 | + int f_flags; | |
| 1635 | + | |
| 1636 | + log = bpf_log_attr_create_vlog(&attr_log, attr_common, uattr_common, size_common); | |
| 1637 | + if (IS_ERR(log)) | |
| 1638 | + return PTR_ERR(log); | |
| 1639 | + | |
| 1640 | + err = map_create_alloc(attr, uattr, log, &map, &token); | |
| 1641 | + | |
| 1642 | + /* preserve original error even if log finalization is successful */ | |
| 1643 | + ret = bpf_log_attr_finalize(&attr_log, log); | |
| 1644 | + if (ret) | |
| 1645 | + err = ret; | |
| 1646 | + | |
| 1647 | + kfree(log); | |
| 1648 | + | |
| 1649 | + if (err) | |
| 1650 | + goto free_map; | |
| 1651 | + | |
| 1652 | + f_flags = bpf_get_file_flag(attr->map_flags); | |
| 1653 | + if (f_flags < 0) { | |
| 1654 | + err = f_flags; | |
| 1655 | + goto free_map; | |
| 1656 | + } | |
| 1657 | + | |
| 1580 | 1658 | err = security_bpf_map_create(map, attr, token, uattr.is_kernel); |
| 1581 | 1659 | if (err) |
| 1582 | 1660 | goto free_map_sec; |
| @@ -1605,8 +1683,8 @@ static int map_create(union bpf_attr *attr, bpfptr_t uattr) · 22 unmodified lines | ||
| 1605 | 1683 | free_map_sec: |
| 1606 | 1684 | security_bpf_map_free(map); |
| 1607 | 1685 | free_map: |
| 1608 | − bpf_map_free(map); | |
| 1609 | −put_token: | |
| 1686 | + if (map) | |
| 1687 | + bpf_map_free(map); | |
| 1610 | 1688 | bpf_token_put(token); |
| 1611 | 1689 | return err; |
| 1612 | 1690 | } |
| @@ -2192,6 +2270,7 @@ static int map_lookup_and_delete_elem(union bpf_attr *attr) · 579 unmodified lines | ||
| 2192 | 2270 | map->map_type == BPF_MAP_TYPE_PERCPU_HASH || |
| 2193 | 2271 | map->map_type == BPF_MAP_TYPE_LRU_HASH || |
| 2194 | 2272 | map->map_type == BPF_MAP_TYPE_LRU_PERCPU_HASH || |
| 2273 | + map->map_type == BPF_MAP_TYPE_RHASH || | |
| 2195 | 2274 | map->map_type == BPF_MAP_TYPE_STACK_TRACE) { |
| 2196 | 2275 | if (!bpf_map_is_offloaded(map)) { |
| 2197 | 2276 | bpf_disable_instrumentation(); |
| @@ -2646,7 +2725,8 @@ static int · 448 unmodified lines | ||
| 2646 | 2725 | bpf_prog_load_check_attach(enum bpf_prog_type prog_type, |
| 2647 | 2726 | enum bpf_attach_type expected_attach_type, |
| 2648 | 2727 | struct btf *attach_btf, u32 btf_id, |
| 2649 | − struct bpf_prog *dst_prog) | |
| 2728 | + struct bpf_prog *dst_prog, | |
| 2729 | + bool multi_func) | |
| 2650 | 2730 | { |
| 2651 | 2731 | if (btf_id) { |
| 2652 | 2732 | if (btf_id > BTF_MAX_TYPE) |
| @@ -2666,6 +2746,14 @@ bpf_prog_load_check_attach(enum bpf_prog_type prog_type, · 13 unmodified lines | ||
| 2666 | 2746 | } |
| 2667 | 2747 | } |
| 2668 | 2748 | |
| 2749 | + if (multi_func) { | |
| 2750 | + if (prog_type != BPF_PROG_TYPE_TRACING) | |
| 2751 | + return -EINVAL; | |
| 2752 | + if (!attach_btf || btf_id) | |
| 2753 | + return -EINVAL; | |
| 2754 | + return 0; | |
| 2755 | + } | |
| 2756 | + | |
| 2669 | 2757 | if (attach_btf && (!btf_id || dst_prog)) |
| 2670 | 2758 | return -EINVAL; |
| 2671 | 2759 | |
| @@ -2798,8 +2886,22 @@ static bool is_perfmon_prog_type(enum bpf_prog_type prog_type) · 126 unmodified lines | ||
| 2798 | 2886 | } |
| 2799 | 2887 | } |
| 2800 | 2888 | |
| 2889 | +static enum bpf_sig_keyring bpf_classify_keyring(s32 keyring_id) | |
| 2890 | +{ | |
| 2891 | + switch (keyring_id) { | |
| 2892 | + case 0: | |
| 2893 | + return BPF_SIG_KEYRING_BUILTIN; | |
| 2894 | + case (s32)(unsigned long)VERIFY_USE_SECONDARY_KEYRING: | |
| 2895 | + return BPF_SIG_KEYRING_SECONDARY; | |
| 2896 | + case (s32)(unsigned long)VERIFY_USE_PLATFORM_KEYRING: | |
| 2897 | + return BPF_SIG_KEYRING_PLATFORM; | |
| 2898 | + default: | |
| 2899 | + return BPF_SIG_KEYRING_USER; | |
| 2900 | + } | |
| 2901 | +} | |
| 2902 | + | |
| 2801 | 2903 | static int bpf_prog_verify_signature(struct bpf_prog *prog, union bpf_attr *attr, |
| 2802 | − bool is_kernel) | |
| 2904 | + bool is_kernel, s32 *keyring_serial) | |
| 2803 | 2905 | { |
| 2804 | 2906 | bpfptr_t usig = make_bpfptr(attr->signature, is_kernel); |
| 2805 | 2907 | struct bpf_dynptr_kern sig_ptr, insns_ptr; |
| @@ -2835,7 +2937,8 @@ static int bpf_prog_verify_signature(struct bpf_prog *prog, union bpf_attr *attr · 29 unmodified lines | ||
| 2835 | 2937 | |
| 2836 | 2938 | err = bpf_verify_pkcs7_signature((struct bpf_dynptr *)&insns_ptr, |
| 2837 | 2939 | (struct bpf_dynptr *)&sig_ptr, key); |
| 2838 | − | |
| 2940 | + if (!err) | |
| 2941 | + *keyring_serial = bpf_key_serial(key); | |
| 2839 | 2942 | bpf_key_put(key); |
| 2840 | 2943 | kvfree(sig); |
| 2841 | 2944 | return err; |
| @@ -2858,10 +2961,15 @@ static int bpf_prog_mark_insn_arrays_ready(struct bpf_prog *prog) · 16 unmodified lines | ||
| 2858 | 2961 | return 0; |
| 2859 | 2962 | } |
| 2860 | 2963 | |
| 2964 | +extern int bpf_multi_func(void); | |
| 2965 | +int __init __used bpf_multi_func(void) { return 0; } | |
| 2966 | + | |
| 2967 | +BTF_ID_LIST_GLOBAL_SINGLE(bpf_multi_func_btf_id, func, bpf_multi_func) | |
| 2968 | + | |
| 2861 | 2969 | /* last field in 'union bpf_attr' used by this command */ |
| 2862 | 2970 | #define BPF_PROG_LOAD_LAST_FIELD keyring_id |
| 2863 | 2971 | |
| 2864 | −static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size) | |
| 2972 | +static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, struct bpf_log_attr *attr_log) | |
| 2865 | 2973 | { |
| 2866 | 2974 | enum bpf_prog_type type = attr->prog_type; |
| 2867 | 2975 | struct bpf_prog *prog, *dst_prog = NULL; |
| @@ -2870,6 +2978,7 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size) · 2 unmodified lines | ||
| 2870 | 2978 | bool bpf_cap; |
| 2871 | 2979 | int err; |
| 2872 | 2980 | char license[128]; |
| 2981 | + bool multi_func; | |
| 2873 | 2982 | |
| 2874 | 2983 | if (CHECK_ATTR(BPF_PROG_LOAD)) |
| 2875 | 2984 | return -EINVAL; |
| @@ -2936,6 +3045,8 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size) · 60 unmodified lines | ||
| 2936 | 3045 | if (is_perfmon_prog_type(type) && !bpf_token_capable(token, CAP_PERFMON)) |
| 2937 | 3046 | goto put_token; |
| 2938 | 3047 | |
| 3048 | + multi_func = is_tracing_multi(attr->expected_attach_type); | |
| 3049 | + | |
| 2939 | 3050 | /* attach_prog_fd/attach_btf_obj_fd can specify fd of either bpf_prog |
| 2940 | 3051 | * or btf, we need to check which one it is |
| 2941 | 3052 | */ |
| @@ -2957,7 +3068,7 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size) · 15 unmodified lines | ||
| 2957 | 3068 | goto put_token; |
| 2958 | 3069 | } |
| 2959 | 3070 | } |
| 2960 | − } else if (attr->attach_btf_id) { | |
| 3071 | + } else if (attr->attach_btf_id || multi_func) { | |
| 2961 | 3072 | /* fall back to vmlinux BTF, if BTF type ID is specified */ |
| 2962 | 3073 | attach_btf = bpf_get_btf_vmlinux(); |
| 2963 | 3074 | if (IS_ERR(attach_btf)) { |
| @@ -2973,7 +3084,7 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size) · 9 unmodified lines | ||
| 2973 | 3084 | |
| 2974 | 3085 | if (bpf_prog_load_check_attach(type, attr->expected_attach_type, |
| 2975 | 3086 | attach_btf, attr->attach_btf_id, |
| 2976 | − dst_prog)) { | |
| 3087 | + dst_prog, multi_func)) { | |
| 2977 | 3088 | if (dst_prog) |
| 2978 | 3089 | bpf_prog_put(dst_prog); |
| 2979 | 3090 | if (attach_btf) |
| @@ -2996,7 +3107,7 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size) · 16 unmodified lines | ||
| 2996 | 3107 | prog->expected_attach_type = attr->expected_attach_type; |
| 2997 | 3108 | prog->sleepable = !!(attr->prog_flags & BPF_F_SLEEPABLE); |
| 2998 | 3109 | prog->aux->attach_btf = attach_btf; |
| 2999 | − prog->aux->attach_btf_id = attr->attach_btf_id; | |
| 3110 | + prog->aux->attach_btf_id = multi_func ? bpf_multi_func_btf_id[0] : attr->attach_btf_id; | |
| 3000 | 3111 | prog->aux->dst_prog = dst_prog; |
| 3001 | 3112 | prog->aux->dev_bound = !!attr->prog_ifindex; |
| 3002 | 3113 | prog->aux->xdp_has_frags = attr->prog_flags & BPF_F_XDP_HAS_FRAGS; |
| @@ -3022,13 +3133,17 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size) · 19 unmodified lines | ||
| 3022 | 3133 | |
| 3023 | 3134 | /* eBPF programs must be GPL compatible to use GPL-ed functions */ |
| 3024 | 3135 | prog->gpl_compatible = license_is_gpl_compatible(license) ? 1 : 0; |
| 3025 | − | |
| 3026 | 3136 | if (attr->signature) { |
| 3027 | − err = bpf_prog_verify_signature(prog, attr, uattr.is_kernel); | |
| 3137 | + err = bpf_prog_verify_signature(prog, attr, uattr.is_kernel, | |
| 3138 | + &prog->aux->sig.keyring_serial); | |
| 3028 | 3139 | if (err) |
| 3029 | 3140 | goto free_prog; |
| 3141 | + prog->aux->sig.keyring_type = bpf_classify_keyring(attr->keyring_id); | |
| 3142 | + prog->aux->sig.verdict = BPF_SIG_VERIFIED; | |
| 3143 | + } else { | |
| 3144 | + prog->aux->sig.keyring_type = BPF_SIG_KEYRING_NONE; | |
| 3145 | + prog->aux->sig.verdict = BPF_SIG_UNSIGNED; | |
| 3030 | 3146 | } |
| 3031 | − | |
| 3032 | 3147 | prog->orig_prog = NULL; |
| 3033 | 3148 | prog->jited = 0; |
| 3034 | 3149 | |
| @@ -3076,10 +3191,10 @@ static int bpf_prog_load(union bpf_attr *attr, bpfptr_t uattr, u32 uattr_size) · 41 unmodified lines | ||
| 3076 | 3191 | |
| 3077 | 3192 | err = security_bpf_prog_load(prog, attr, token, uattr.is_kernel); |
| 3078 | 3193 | if (err) |
| 3079 | − goto free_prog_sec; | |
| 3194 | + goto free_prog; | |
| 3080 | 3195 | |
| 3081 | 3196 | /* run eBPF verifier */ |
| 3082 | − err = bpf_check(&prog, attr, uattr, uattr_size); | |
| 3197 | + err = bpf_check(&prog, attr, uattr, attr_log); | |
| 3083 | 3198 | if (err < 0) |
| 3084 | 3199 | goto free_used_maps; |
| 3085 | 3200 | |
| @@ -3122,8 +3237,6 @@ free_used_maps: · 36 unmodified lines | ||
| 3122 | 3237 | __bpf_prog_put_noref(prog, prog->aux->real_func_cnt); |
| 3123 | 3238 | return err; |
| 3124 | 3239 | |
| 3125 | −free_prog_sec: | |
| 3126 | − security_bpf_prog_free(prog); | |
| 3127 | 3240 | free_prog: |
| 3128 | 3241 | free_uid(prog->aux->user); |
| 3129 | 3242 | if (prog->aux->attach_btf) |
| @@ -3198,6 +3311,15 @@ void bpf_link_init(struct bpf_link *link, enum bpf_link_type type, · 68 unmodified lines | ||
| 3198 | 3311 | bpf_link_init_sleepable(link, type, ops, prog, attach_type, false); |
| 3199 | 3312 | } |
| 3200 | 3313 | |
| 3314 | +void bpf_tramp_link_init(struct bpf_tramp_link *link, enum bpf_link_type type, | |
| 3315 | + const struct bpf_link_ops *ops, struct bpf_prog *prog, | |
| 3316 | + enum bpf_attach_type attach_type, u64 cookie) | |
| 3317 | +{ | |
| 3318 | + bpf_link_init(&link->link, type, ops, prog, attach_type); | |
| 3319 | + link->node.link = &link->link; | |
| 3320 | + link->node.cookie = cookie; | |
| 3321 | +} | |
| 3322 | + | |
| 3201 | 3323 | static void bpf_link_free_id(int id) |
| 3202 | 3324 | { |
| 3203 | 3325 | if (!id) |
| @@ -3358,7 +3480,7 @@ static void bpf_link_show_fdinfo(struct seq_file *m, struct file *filp) · 154 unmodified lines | ||
| 3358 | 3480 | seq_printf(m, "link_type:\t%s\n", link->flags == BPF_F_KPROBE_MULTI_RETURN ? |
| 3359 | 3481 | "kretprobe_multi" : "kprobe_multi"); |
| 3360 | 3482 | else if (link->type == BPF_LINK_TYPE_UPROBE_MULTI) |
| 3361 | − seq_printf(m, "link_type:\t%s\n", link->flags == BPF_F_UPROBE_MULTI_RETURN ? | |
| 3483 | + seq_printf(m, "link_type:\t%s\n", link->flags & BPF_F_UPROBE_MULTI_RETURN ? | |
| 3362 | 3484 | "uretprobe_multi" : "uprobe_multi"); |
| 3363 | 3485 | else |
| 3364 | 3486 | seq_printf(m, "link_type:\t%s\n", bpf_link_type_strs[type]); |
| @@ -3505,7 +3627,7 @@ static void bpf_tracing_link_release(struct bpf_link *link) · 140 unmodified lines | ||
| 3505 | 3627 | struct bpf_tracing_link *tr_link = |
| 3506 | 3628 | container_of(link, struct bpf_tracing_link, link.link); |
| 3507 | 3629 | |
| 3508 | − WARN_ON_ONCE(bpf_trampoline_unlink_prog(&tr_link->link, | |
| 3630 | + WARN_ON_ONCE(bpf_trampoline_unlink_prog(&tr_link->link.node, | |
| 3509 | 3631 | tr_link->trampoline, |
| 3510 | 3632 | tr_link->tgt_prog)); |
| 3511 | 3633 | |
| @@ -3518,8 +3640,7 @@ static void bpf_tracing_link_release(struct bpf_link *link) · 6 unmodified lines | ||
| 3518 | 3640 | |
| 3519 | 3641 | static void bpf_tracing_link_dealloc(struct bpf_link *link) |
| 3520 | 3642 | { |
| 3521 | − struct bpf_tracing_link *tr_link = | |
| 3522 | − container_of(link, struct bpf_tracing_link, link.link); | |
| 3643 | + struct bpf_tracing_link *tr_link = container_of(link, struct bpf_tracing_link, link.link); | |
| 3523 | 3644 | |
| 3524 | 3645 | kfree(tr_link); |
| 3525 | 3646 | } |
| @@ -3527,8 +3648,8 @@ static void bpf_tracing_link_dealloc(struct bpf_link *link) · 1 unmodified line | ||
| 3527 | 3648 | static void bpf_tracing_link_show_fdinfo(const struct bpf_link *link, |
| 3528 | 3649 | struct seq_file *seq) |
| 3529 | 3650 | { |
| 3530 | − struct bpf_tracing_link *tr_link = | |
| 3531 | − container_of(link, struct bpf_tracing_link, link.link); | |
| 3651 | + struct bpf_tracing_link *tr_link = container_of(link, struct bpf_tracing_link, link.link); | |
| 3652 | + | |
| 3532 | 3653 | u32 target_btf_id, target_obj_id; |
| 3533 | 3654 | |
| 3534 | 3655 | bpf_trampoline_unpack_key(tr_link->trampoline->key, |
| @@ -3541,17 +3662,16 @@ static void bpf_tracing_link_show_fdinfo(const struct bpf_link *link, · 6 unmodified lines | ||
| 3541 | 3662 | link->attach_type, |
| 3542 | 3663 | target_obj_id, |
| 3543 | 3664 | target_btf_id, |
| 3544 | − tr_link->link.cookie); | |
| 3665 | + tr_link->link.node.cookie); | |
| 3545 | 3666 | } |
| 3546 | 3667 | |
| 3547 | 3668 | static int bpf_tracing_link_fill_link_info(const struct bpf_link *link, |
| 3548 | 3669 | struct bpf_link_info *info) |
| 3549 | 3670 | { |
| 3550 | − struct bpf_tracing_link *tr_link = | |
| 3551 | − container_of(link, struct bpf_tracing_link, link.link); | |
| 3671 | + struct bpf_tracing_link *tr_link = container_of(link, struct bpf_tracing_link, link.link); | |
| 3552 | 3672 | |
| 3553 | 3673 | info->tracing.attach_type = link->attach_type; |
| 3554 | − info->tracing.cookie = tr_link->link.cookie; | |
| 3674 | + info->tracing.cookie = tr_link->link.node.cookie; | |
| 3555 | 3675 | bpf_trampoline_unpack_key(tr_link->trampoline->key, |
| 3556 | 3676 | &info->tracing.target_obj_id, |
| 3557 | 3677 | &info->tracing.target_btf_id); |
| @@ -3633,29 +3753,18 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, · 75 unmodified lines | ||
| 3633 | 3753 | key = bpf_trampoline_compute_key(tgt_prog, NULL, btf_id); |
| 3634 | 3754 | } |
| 3635 | 3755 | |
| 3636 | − if (prog->expected_attach_type == BPF_TRACE_FSESSION) { | |
| 3637 | − struct bpf_fsession_link *fslink; | |
| 3638 | − | |
| 3639 | − fslink = kzalloc_obj(*fslink, GFP_USER); | |
| 3640 | − if (fslink) { | |
| 3641 | − bpf_link_init(&fslink->fexit.link, BPF_LINK_TYPE_TRACING, | |
| 3642 | − &bpf_tracing_link_lops, prog, attach_type); | |
| 3643 | − fslink->fexit.cookie = bpf_cookie; | |
| 3644 | − link = &fslink->link; | |
| 3645 | − } else { | |
| 3646 | − link = NULL; | |
| 3647 | − } | |
| 3648 | − } else { | |
| 3649 | − link = kzalloc_obj(*link, GFP_USER); | |
| 3650 | − } | |
| 3756 | + link = kzalloc_obj(*link, GFP_USER); | |
| 3651 | 3757 | if (!link) { |
| 3652 | 3758 | err = -ENOMEM; |
| 3653 | 3759 | goto out_put_prog; |
| 3654 | 3760 | } |
| 3655 | − bpf_link_init(&link->link.link, BPF_LINK_TYPE_TRACING, | |
| 3656 | − &bpf_tracing_link_lops, prog, attach_type); | |
| 3761 | + bpf_tramp_link_init(&link->link, BPF_LINK_TYPE_TRACING, | |
| 3762 | + &bpf_tracing_link_lops, prog, attach_type, bpf_cookie); | |
| 3657 | 3763 | |
| 3658 | − link->link.cookie = bpf_cookie; | |
| 3764 | + if (prog->expected_attach_type == BPF_TRACE_FSESSION) { | |
| 3765 | + link->fexit.link = &link->link.link; | |
| 3766 | + link->fexit.cookie = bpf_cookie; | |
| 3767 | + } | |
| 3659 | 3768 | |
| 3660 | 3769 | mutex_lock(&prog->aux->dst_mutex); |
| 3661 | 3770 | |
| @@ -3758,7 +3867,7 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, · 96 unmodified lines | ||
| 3758 | 3867 | if (err) |
| 3759 | 3868 | goto out_unlock; |
| 3760 | 3869 | |
| 3761 | − err = bpf_trampoline_link_prog(&link->link, tr, tgt_prog); | |
| 3870 | + err = bpf_trampoline_link_prog(&link->link.node, tr, tgt_prog); | |
| 3762 | 3871 | if (err) { |
| 3763 | 3872 | bpf_link_cleanup(&link_primer); |
| 3764 | 3873 | link = NULL; |
| @@ -4281,6 +4390,11 @@ static int bpf_raw_tp_link_attach(struct bpf_prog *prog, · 516 unmodified lines | ||
| 4281 | 4390 | if (!btp) |
| 4282 | 4391 | return -ENOENT; |
| 4283 | 4392 | |
| 4393 | + if (prog->sleepable && !tracepoint_is_faultable(btp->tp)) { | |
| 4394 | + bpf_put_raw_tracepoint(btp); | |
| 4395 | + return -EINVAL; | |
| 4396 | + } | |
| 4397 | + | |
| 4284 | 4398 | link = kzalloc_obj(*link, GFP_USER); |
| 4285 | 4399 | if (!link) { |
| 4286 | 4400 | err = -ENOMEM; |
| @@ -4389,6 +4503,9 @@ attach_type_to_prog_type(enum bpf_attach_type attach_type) · 102 unmodified lines | ||
| 4389 | 4503 | case BPF_TRACE_FENTRY: |
| 4390 | 4504 | case BPF_TRACE_FEXIT: |
| 4391 | 4505 | case BPF_TRACE_FSESSION: |
| 4506 | + case BPF_TRACE_FSESSION_MULTI: | |
| 4507 | + case BPF_TRACE_FENTRY_MULTI: | |
| 4508 | + case BPF_TRACE_FEXIT_MULTI: | |
| 4392 | 4509 | case BPF_MODIFY_RETURN: |
| 4393 | 4510 | return BPF_PROG_TYPE_TRACING; |
| 4394 | 4511 | case BPF_LSM_MAC: |
| @@ -4654,7 +4771,7 @@ static int bpf_prog_detach(const union bpf_attr *attr) · 259 unmodified lines | ||
| 4654 | 4771 | #define BPF_PROG_QUERY_LAST_FIELD query.revision |
| 4655 | 4772 | |
| 4656 | 4773 | static int bpf_prog_query(const union bpf_attr *attr, |
| 4657 | − union bpf_attr __user *uattr) | |
| 4774 | + union bpf_attr __user *uattr, u32 uattr_size) | |
| 4658 | 4775 | { |
| 4659 | 4776 | if (!bpf_net_capable()) |
| 4660 | 4777 | return -EPERM; |
| @@ -4693,7 +4810,7 @@ static int bpf_prog_query(const union bpf_attr *attr, · 32 unmodified lines | ||
| 4693 | 4810 | case BPF_CGROUP_GETSOCKOPT: |
| 4694 | 4811 | case BPF_CGROUP_SETSOCKOPT: |
| 4695 | 4812 | case BPF_LSM_CGROUP: |
| 4696 | − return cgroup_bpf_prog_query(attr, uattr); | |
| 4813 | + return cgroup_bpf_prog_query(attr, uattr, uattr_size); | |
| 4697 | 4814 | case BPF_LIRC_MODE2: |
| 4698 | 4815 | return lirc_prog_query(attr, uattr); |
| 4699 | 4816 | case BPF_FLOW_DISSECTOR: |
| @@ -5045,10 +5162,11 @@ static int bpf_prog_get_info_by_fd(struct file *file, · 345 unmodified lines | ||
| 5045 | 5162 | u32 info_len = attr->info.info_len; |
| 5046 | 5163 | struct bpf_prog_kstats stats; |
| 5047 | 5164 | char __user *uinsns; |
| 5048 | − u32 ulen; | |
| 5165 | + u32 ulen, len; | |
| 5049 | 5166 | int err; |
| 5050 | 5167 | |
| 5051 | − err = bpf_check_uarg_tail_zero(USER_BPFPTR(uinfo), sizeof(info), info_len); | |
| 5168 | + len = offsetofend(struct bpf_prog_info, attach_btf_id); | |
| 5169 | + err = bpf_check_uarg_tail_zero(USER_BPFPTR(uinfo), len, info_len); | |
| 5052 | 5170 | if (err) |
| 5053 | 5171 | return err; |
| 5054 | 5172 | info_len = min_t(u32, sizeof(info), info_len); |
| @@ -5330,10 +5448,11 @@ static int bpf_map_get_info_by_fd(struct file *file, · 275 unmodified lines | ||
| 5330 | 5448 | { |
| 5331 | 5449 | struct bpf_map_info __user *uinfo = u64_to_user_ptr(attr->info.info); |
| 5332 | 5450 | struct bpf_map_info info; |
| 5333 | − u32 info_len = attr->info.info_len; | |
| 5451 | + u32 info_len = attr->info.info_len, len; | |
| 5334 | 5452 | int err; |
| 5335 | 5453 | |
| 5336 | − err = bpf_check_uarg_tail_zero(USER_BPFPTR(uinfo), sizeof(info), info_len); | |
| 5454 | + len = offsetofend(struct bpf_map_info, hash_size); | |
| 5455 | + err = bpf_check_uarg_tail_zero(USER_BPFPTR(uinfo), len, info_len); | |
| 5337 | 5456 | if (err) |
| 5338 | 5457 | return err; |
| 5339 | 5458 | info_len = min_t(u32, sizeof(info), info_len); |
| @@ -5371,18 +5490,16 @@ static int bpf_map_get_info_by_fd(struct file *file, · 31 unmodified lines | ||
| 5371 | 5490 | |
| 5372 | 5491 | if (!map->ops->map_get_hash) |
| 5373 | 5492 | return -EINVAL; |
| 5374 | − | |
| 5375 | − if (info.hash_size != SHA256_DIGEST_SIZE) | |
| 5493 | + if (info.hash_size != sizeof(map->sha)) | |
| 5376 | 5494 | return -EINVAL; |
| 5377 | − | |
| 5378 | 5495 | if (!READ_ONCE(map->frozen)) |
| 5379 | 5496 | return -EPERM; |
| 5380 | 5497 | |
| 5381 | − err = map->ops->map_get_hash(map, SHA256_DIGEST_SIZE, map->sha); | |
| 5498 | + err = map->ops->map_get_hash(map); | |
| 5382 | 5499 | if (err != 0) |
| 5383 | 5500 | return err; |
| 5384 | 5501 | |
| 5385 | − if (copy_to_user(uhash, map->sha, SHA256_DIGEST_SIZE) != 0) | |
| 5502 | + if (copy_to_user(uhash, map->sha, sizeof(map->sha)) != 0) | |
| 5386 | 5503 | return -EFAULT; |
| 5387 | 5504 | } else if (info.hash_size) { |
| 5388 | 5505 | return -EINVAL; |
| @@ -5495,7 +5612,7 @@ static int bpf_obj_get_info_by_fd(const union bpf_attr *attr, · 106 unmodified lines | ||
| 5495 | 5612 | |
| 5496 | 5613 | #define BPF_BTF_LOAD_LAST_FIELD btf_token_fd |
| 5497 | 5614 | |
| 5498 | −static int bpf_btf_load(const union bpf_attr *attr, bpfptr_t uattr, __u32 uattr_size) | |
| 5615 | +static int bpf_btf_load(const union bpf_attr *attr, bpfptr_t uattr, struct bpf_log_attr *attr_log) | |
| 5499 | 5616 | { |
| 5500 | 5617 | struct bpf_token *token = NULL; |
| 5501 | 5618 | |
| @@ -5522,7 +5639,7 @@ static int bpf_btf_load(const union bpf_attr *attr, bpfptr_t uattr, __u32 uattr_ · 20 unmodified lines | ||
| 5522 | 5639 | |
| 5523 | 5640 | bpf_token_put(token); |
| 5524 | 5641 | |
| 5525 | − return btf_new_fd(attr, uattr, uattr_size); | |
| 5642 | + return btf_new_fd(attr, uattr, attr_log); | |
| 5526 | 5643 | } |
| 5527 | 5644 | |
| 5528 | 5645 | #define BPF_BTF_GET_FD_BY_ID_LAST_FIELD fd_by_id_token_fd |
| @@ -5723,7 +5840,7 @@ err_put: · 194 unmodified lines | ||
| 5723 | 5840 | return err; |
| 5724 | 5841 | } |
| 5725 | 5842 | |
| 5726 | −#define BPF_LINK_CREATE_LAST_FIELD link_create.uprobe_multi.pid | |
| 5843 | +#define BPF_LINK_CREATE_LAST_FIELD link_create.uprobe_multi.path_fd | |
| 5727 | 5844 | static int link_create(union bpf_attr *attr, bpfptr_t uattr) |
| 5728 | 5845 | { |
| 5729 | 5846 | struct bpf_prog *prog; |
| @@ -5774,6 +5891,8 @@ static int link_create(union bpf_attr *attr, bpfptr_t uattr) · 44 unmodified lines | ||
| 5774 | 5891 | ret = bpf_iter_link_attach(attr, uattr, prog); |
| 5775 | 5892 | else if (prog->expected_attach_type == BPF_LSM_CGROUP) |
| 5776 | 5893 | ret = cgroup_bpf_link_attach(attr, prog); |
| 5894 | + else if (is_tracing_multi(prog->expected_attach_type)) | |
| 5895 | + ret = bpf_tracing_multi_attach(prog, attr); | |
| 5777 | 5896 | else |
| 5778 | 5897 | ret = bpf_tracing_prog_attach(prog, |
| 5779 | 5898 | attr->link_create.target_fd, |
| @@ -6232,8 +6351,12 @@ put_prog: · 452 unmodified lines | ||
| 6232 | 6351 | return ret; |
| 6233 | 6352 | } |
| 6234 | 6353 | |
| 6235 | −static int __sys_bpf(enum bpf_cmd cmd, bpfptr_t uattr, unsigned int size) | |
| 6354 | +static int __sys_bpf(enum bpf_cmd cmd, bpfptr_t uattr, unsigned int size, | |
| 6355 | + bpfptr_t uattr_common, unsigned int size_common) | |
| 6236 | 6356 | { |
| 6357 | + struct bpf_common_attr attr_common; | |
| 6358 | + u32 offsetof_log_true_size = 0; | |
| 6359 | + struct bpf_log_attr attr_log; | |
| 6237 | 6360 | union bpf_attr attr; |
| 6238 | 6361 | int err; |
| 6239 | 6362 | |
| @@ -6247,13 +6370,29 @@ static int __sys_bpf(enum bpf_cmd cmd, bpfptr_t uattr, unsigned int size) · 7 unmodified lines | ||
| 6247 | 6370 | if (copy_from_bpfptr(&attr, uattr, size) != 0) |
| 6248 | 6371 | return -EFAULT; |
| 6249 | 6372 | |
| 6373 | + memset(&attr_common, 0, sizeof(attr_common)); | |
| 6374 | + if (cmd & BPF_COMMON_ATTRS) { | |
| 6375 | + err = bpf_check_uarg_tail_zero(uattr_common, | |
| 6376 | + offsetofend(struct bpf_common_attr, log_true_size), | |
| 6377 | + size_common); | |
| 6378 | + if (err) | |
| 6379 | + return err; | |
| 6380 | + | |
| 6381 | + cmd &= ~BPF_COMMON_ATTRS; | |
| 6382 | + size_common = min_t(u32, size_common, sizeof(attr_common)); | |
| 6383 | + if (copy_from_bpfptr(&attr_common, uattr_common, size_common) != 0) | |
| 6384 | + return -EFAULT; | |
| 6385 | + } else { | |
| 6386 | + size_common = 0; | |
| 6387 | + } | |
| 6388 | + | |
| 6250 | 6389 | err = security_bpf(cmd, &attr, size, uattr.is_kernel); |
| 6251 | 6390 | if (err < 0) |
| 6252 | 6391 | return err; |
| 6253 | 6392 | |
| 6254 | 6393 | switch (cmd) { |
| 6255 | 6394 | case BPF_MAP_CREATE: |
| 6256 | − err = map_create(&attr, uattr); | |
| 6395 | + err = map_create(&attr, uattr, &attr_common, uattr_common, size_common); | |
| 6257 | 6396 | break; |
| 6258 | 6397 | case BPF_MAP_LOOKUP_ELEM: |
| 6259 | 6398 | err = map_lookup_elem(&attr); |
| @@ -6271,7 +6410,12 @@ static int __sys_bpf(enum bpf_cmd cmd, bpfptr_t uattr, unsigned int size) · 11 unmodified lines | ||
| 6271 | 6410 | err = map_freeze(&attr); |
| 6272 | 6411 | break; |
| 6273 | 6412 | case BPF_PROG_LOAD: |
| 6274 | − err = bpf_prog_load(&attr, uattr, size); | |
| 6413 | + if (size >= offsetofend(union bpf_attr, log_true_size)) | |
| 6414 | + offsetof_log_true_size = offsetof(union bpf_attr, log_true_size); | |
| 6415 | + err = bpf_log_attr_init(&attr_log, attr.log_buf, attr.log_size, attr.log_level, | |
| 6416 | + offsetof_log_true_size, uattr, &attr_common, uattr_common, | |
| 6417 | + size_common); | |
| 6418 | + err = err ?: bpf_prog_load(&attr, uattr, &attr_log); | |
| 6275 | 6419 | break; |
| 6276 | 6420 | case BPF_OBJ_PIN: |
| 6277 | 6421 | err = bpf_obj_pin(&attr); |
| @@ -6286,7 +6430,7 @@ static int __sys_bpf(enum bpf_cmd cmd, bpfptr_t uattr, unsigned int size) · 8 unmodified lines | ||
| 6286 | 6430 | err = bpf_prog_detach(&attr); |
| 6287 | 6431 | break; |
| 6288 | 6432 | case BPF_PROG_QUERY: |
| 6289 | − err = bpf_prog_query(&attr, uattr.user); | |
| 6433 | + err = bpf_prog_query(&attr, uattr.user, size); | |
| 6290 | 6434 | break; |
| 6291 | 6435 | case BPF_PROG_TEST_RUN: |
| 6292 | 6436 | err = bpf_prog_test_run(&attr, uattr.user); |
| @@ -6316,7 +6460,12 @@ static int __sys_bpf(enum bpf_cmd cmd, bpfptr_t uattr, unsigned int size) · 23 unmodified lines | ||
| 6316 | 6460 | err = bpf_raw_tracepoint_open(&attr); |
| 6317 | 6461 | break; |
| 6318 | 6462 | case BPF_BTF_LOAD: |
| 6319 | − err = bpf_btf_load(&attr, uattr, size); | |
| 6463 | + if (size >= offsetofend(union bpf_attr, btf_log_true_size)) | |
| 6464 | + offsetof_log_true_size = offsetof(union bpf_attr, btf_log_true_size); | |
| 6465 | + err = bpf_log_attr_init(&attr_log, attr.btf_log_buf, attr.btf_log_size, | |
| 6466 | + attr.btf_log_level, offsetof_log_true_size, uattr, | |
| 6467 | + &attr_common, uattr_common, size_common); | |
| 6468 | + err = err ?: bpf_btf_load(&attr, uattr, &attr_log); | |
| 6320 | 6469 | break; |
| 6321 | 6470 | case BPF_BTF_GET_FD_BY_ID: |
| 6322 | 6471 | err = bpf_btf_get_fd_by_id(&attr); |
| @@ -6382,9 +6531,10 @@ static int __sys_bpf(enum bpf_cmd cmd, bpfptr_t uattr, unsigned int size) · 59 unmodified lines | ||
| 6382 | 6531 | return err; |
| 6383 | 6532 | } |
| 6384 | 6533 | |
| 6385 | −SYSCALL_DEFINE3(bpf, int, cmd, union bpf_attr __user *, uattr, unsigned int, size) | |
| 6534 | +SYSCALL_DEFINE5(bpf, int, cmd, union bpf_attr __user *, uattr, unsigned int, size, | |
| 6535 | + struct bpf_common_attr __user *, uattr_common, unsigned int, size_common) | |
| 6386 | 6536 | { |
| 6387 | − return __sys_bpf(cmd, USER_BPFPTR(uattr), size); | |
| 6537 | + return __sys_bpf(cmd, USER_BPFPTR(uattr), size, USER_BPFPTR(uattr_common), size_common); | |
| 6388 | 6538 | } |
| 6389 | 6539 | |
| 6390 | 6540 | static bool syscall_prog_is_valid_access(int off, int size, |
| @@ -6414,7 +6564,7 @@ BPF_CALL_3(bpf_sys_bpf, int, cmd, union bpf_attr *, attr, u32, attr_size) · 23 unmodified lines | ||
| 6414 | 6564 | default: |
| 6415 | 6565 | return -EINVAL; |
| 6416 | 6566 | } |
| 6417 | − return __sys_bpf(cmd, KERNEL_BPFPTR(attr), attr_size); | |
| 6567 | + return __sys_bpf(cmd, KERNEL_BPFPTR(attr), attr_size, KERNEL_BPFPTR(NULL), 0); | |
| 6418 | 6568 | } |
| 6419 | 6569 | |
| 6420 | 6570 | |
| @@ -30,8 +30,46 @@ static struct hlist_head trampoline_ip_table[TRAMPOLINE_TABLE_SIZE]; · 29 unmodified lines | ||
| 30 | 30 | /* serializes access to trampoline tables */ |
| 31 | 31 | static DEFINE_MUTEX(trampoline_mutex); |
| 32 | 32 | |
| 33 | +/* | |
| 34 | + * Keep 32 trampoline locks (5 bits) in the pool so trampoline_lock_all() | |
| 35 | + * stays below MAX_LOCK_DEPTH. Each pool slot has a distinct lockdep | |
| 36 | + * class because trampoline_lock_all() takes all pool mutexes at once; | |
| 37 | + * otherwise lockdep would report recursive locking on same-class mutexes. | |
| 38 | + */ | |
| 39 | +#define TRAMPOLINE_LOCKS_BITS 5 | |
| 40 | +#define TRAMPOLINE_LOCKS_TABLE_SIZE (1 << TRAMPOLINE_LOCKS_BITS) | |
| 41 | + | |
| 42 | +static struct { | |
| 43 | + struct mutex mutex; | |
| 44 | + struct lock_class_key key; | |
| 45 | +} trampoline_locks[TRAMPOLINE_LOCKS_TABLE_SIZE]; | |
| 46 | + | |
| 47 | +static struct mutex *select_trampoline_lock(struct bpf_trampoline *tr) | |
| 48 | +{ | |
| 49 | + return &trampoline_locks[hash_ptr(tr, TRAMPOLINE_LOCKS_BITS)].mutex; | |
| 50 | +} | |
| 51 | + | |
| 52 | +static void trampoline_lock(struct bpf_trampoline *tr) | |
| 53 | +{ | |
| 54 | + mutex_lock(select_trampoline_lock(tr)); | |
| 55 | +} | |
| 56 | + | |
| 57 | +static void trampoline_unlock(struct bpf_trampoline *tr) | |
| 58 | +{ | |
| 59 | + mutex_unlock(select_trampoline_lock(tr)); | |
| 60 | +} | |
| 61 | + | |
| 62 | +struct bpf_trampoline_ops { | |
| 63 | + int (*register_fentry)(struct bpf_trampoline *tr, struct bpf_tramp_image *im, void *data); | |
| 64 | + int (*unregister_fentry)(struct bpf_trampoline *tr, u32 orig_flags, void *data); | |
| 65 | + int (*modify_fentry)(struct bpf_trampoline *tr, u32 orig_flags, struct bpf_tramp_image *im, | |
| 66 | + bool lock_direct_mutex, void *data); | |
| 67 | +}; | |
| 68 | + | |
| 33 | 69 | #ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS |
| 34 | −static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mutex); | |
| 70 | +static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mutex, | |
| 71 | + const struct bpf_trampoline_ops *ops, void *data); | |
| 72 | +static const struct bpf_trampoline_ops trampoline_ops; | |
| 35 | 73 | |
| 36 | 74 | #ifdef CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS |
| 37 | 75 | static struct bpf_trampoline *direct_ops_ip_lookup(struct ftrace_ops *ops, unsigned long ip) |
| @@ -69,9 +107,9 @@ static int bpf_tramp_ftrace_ops_func(struct ftrace_ops *ops, unsigned long ip, · 31 unmodified lines | ||
| 69 | 107 | |
| 70 | 108 | if (cmd == FTRACE_OPS_CMD_ENABLE_SHARE_IPMODIFY_SELF) { |
| 71 | 109 | /* This is called inside register_ftrace_direct_multi(), so |
| 72 | − * tr->mutex is already locked. | |
| 110 | + * trampoline's mutex is already locked. | |
| 73 | 111 | */ |
| 74 | − lockdep_assert_held_once(&tr->mutex); | |
| 112 | + lockdep_assert_held_once(select_trampoline_lock(tr)); | |
| 75 | 113 | |
| 76 | 114 | /* Instead of updating the trampoline here, we propagate |
| 77 | 115 | * -EAGAIN to register_ftrace_direct(). Then we can |
| @@ -91,7 +129,7 @@ static int bpf_tramp_ftrace_ops_func(struct ftrace_ops *ops, unsigned long ip, · 13 unmodified lines | ||
| 91 | 129 | } |
| 92 | 130 | |
| 93 | 131 | /* The normal locking order is |
| 94 | − * tr->mutex => direct_mutex (ftrace.c) => ftrace_lock (ftrace.c) | |
| 132 | + * select_trampoline_lock(tr) => direct_mutex (ftrace.c) => ftrace_lock (ftrace.c) | |
| 95 | 133 | * |
| 96 | 134 | * The following two commands are called from |
| 97 | 135 | * |
| @@ -99,12 +137,12 @@ static int bpf_tramp_ftrace_ops_func(struct ftrace_ops *ops, unsigned long ip, · 1 unmodified line | ||
| 99 | 137 | * cleanup_direct_functions_after_ipmodify |
| 100 | 138 | * |
| 101 | 139 | * In both cases, direct_mutex is already locked. Use |
| 102 | − * mutex_trylock(&tr->mutex) to avoid deadlock in race condition | |
| 103 | − * (something else is making changes to this same trampoline). | |
| 140 | + * mutex_trylock(select_trampoline_lock(tr)) to avoid deadlock in race condition | |
| 141 | + * (something else holds the same pool lock). | |
| 104 | 142 | */ |
| 105 | − if (!mutex_trylock(&tr->mutex)) { | |
| 106 | − /* sleep 1 ms to make sure whatever holding tr->mutex makes | |
| 107 | − * some progress. | |
| 143 | + if (!mutex_trylock(select_trampoline_lock(tr))) { | |
| 144 | + /* sleep 1 ms to make sure whatever holding select_trampoline_lock(tr) | |
| 145 | + * makes some progress. | |
| 108 | 146 | */ |
| 109 | 147 | msleep(1); |
| 110 | 148 | return -EAGAIN; |
| @@ -116,20 +154,22 @@ static int bpf_tramp_ftrace_ops_func(struct ftrace_ops *ops, unsigned long ip, · 5 unmodified lines | ||
| 116 | 154 | |
| 117 | 155 | if ((tr->flags & BPF_TRAMP_F_CALL_ORIG) && |
| 118 | 156 | !(tr->flags & BPF_TRAMP_F_ORIG_STACK)) |
| 119 | − ret = bpf_trampoline_update(tr, false /* lock_direct_mutex */); | |
| 157 | + ret = bpf_trampoline_update(tr, false /* lock_direct_mutex */, | |
| 158 | + &trampoline_ops, NULL); | |
| 120 | 159 | break; |
| 121 | 160 | case FTRACE_OPS_CMD_DISABLE_SHARE_IPMODIFY_PEER: |
| 122 | 161 | tr->flags &= ~BPF_TRAMP_F_SHARE_IPMODIFY; |
| 123 | 162 | |
| 124 | 163 | if (tr->flags & BPF_TRAMP_F_ORIG_STACK) |
| 125 | − ret = bpf_trampoline_update(tr, false /* lock_direct_mutex */); | |
| 164 | + ret = bpf_trampoline_update(tr, false /* lock_direct_mutex */, | |
| 165 | + &trampoline_ops, NULL); | |
| 126 | 166 | break; |
| 127 | 167 | default: |
| 128 | 168 | ret = -EINVAL; |
| 129 | 169 | break; |
| 130 | 170 | } |
| 131 | 171 | |
| 132 | − mutex_unlock(&tr->mutex); | |
| 172 | + trampoline_unlock(tr); | |
| 133 | 173 | return ret; |
| 134 | 174 | } |
| 135 | 175 | #endif |
| @@ -142,7 +182,9 @@ bool bpf_prog_has_trampoline(const struct bpf_prog *prog) · 6 unmodified lines | ||
| 142 | 182 | switch (ptype) { |
| 143 | 183 | case BPF_PROG_TYPE_TRACING: |
| 144 | 184 | if (eatype == BPF_TRACE_FENTRY || eatype == BPF_TRACE_FEXIT || |
| 145 | − eatype == BPF_MODIFY_RETURN || eatype == BPF_TRACE_FSESSION) | |
| 185 | + eatype == BPF_MODIFY_RETURN || eatype == BPF_TRACE_FSESSION || | |
| 186 | + eatype == BPF_TRACE_FENTRY_MULTI || eatype == BPF_TRACE_FEXIT_MULTI || | |
| 187 | + eatype == BPF_TRACE_FSESSION_MULTI) | |
| 146 | 188 | return true; |
| 147 | 189 | return false; |
| 148 | 190 | case BPF_PROG_TYPE_LSM: |
| @@ -359,7 +401,6 @@ static struct bpf_trampoline *bpf_trampoline_lookup(u64 key, unsigned long ip) · 210 unmodified lines | ||
| 359 | 401 | head = &trampoline_ip_table[hash_64(tr->ip, TRAMPOLINE_HASH_BITS)]; |
| 360 | 402 | hlist_add_head(&tr->hlist_ip, head); |
| 361 | 403 | refcount_set(&tr->refcnt, 1); |
| 362 | − mutex_init(&tr->mutex); | |
| 363 | 404 | for (i = 0; i < BPF_TRAMP_MAX; i++) |
| 364 | 405 | INIT_HLIST_HEAD(&tr->progs_hlist[i]); |
| 365 | 406 | out: |
| @@ -386,9 +427,11 @@ static int bpf_trampoline_update_fentry(struct bpf_trampoline *tr, u32 orig_flag · 20 unmodified lines | ||
| 386 | 427 | return bpf_arch_text_poke(ip, old_t, new_t, old_addr, new_addr); |
| 387 | 428 | } |
| 388 | 429 | |
| 389 | −static int unregister_fentry(struct bpf_trampoline *tr, u32 orig_flags, | |
| 390 | − void *old_addr) | |
| 430 | +static void bpf_tramp_image_put(struct bpf_tramp_image *im); | |
| 431 | + | |
| 432 | +static int unregister_fentry(struct bpf_trampoline *tr, u32 orig_flags, void *data __maybe_unused) | |
| 391 | 433 | { |
| 434 | + void *old_addr = tr->cur_image->image; | |
| 392 | 435 | int ret; |
| 393 | 436 | |
| 394 | 437 | if (tr->func.ftrace_managed) |
| @@ -396,13 +439,19 @@ static int unregister_fentry(struct bpf_trampoline *tr, u32 orig_flags, · 1 unmodified line | ||
| 396 | 439 | else |
| 397 | 440 | ret = bpf_trampoline_update_fentry(tr, orig_flags, old_addr, NULL); |
| 398 | 441 | |
| 399 | − return ret; | |
| 442 | + if (ret) | |
| 443 | + return ret; | |
| 444 | + | |
| 445 | + bpf_tramp_image_put(tr->cur_image); | |
| 446 | + tr->cur_image = NULL; | |
| 447 | + return 0; | |
| 400 | 448 | } |
| 401 | 449 | |
| 402 | −static int modify_fentry(struct bpf_trampoline *tr, u32 orig_flags, | |
| 403 | − void *old_addr, void *new_addr, | |
| 404 | − bool lock_direct_mutex) | |
| 450 | +static int modify_fentry(struct bpf_trampoline *tr, u32 orig_flags, struct bpf_tramp_image *im, | |
| 451 | + bool lock_direct_mutex, void *data __maybe_unused) | |
| 405 | 452 | { |
| 453 | + void *old_addr = tr->cur_image->image; | |
| 454 | + void *new_addr = im->image; | |
| 406 | 455 | int ret; |
| 407 | 456 | |
| 408 | 457 | if (tr->func.ftrace_managed) { |
| @@ -411,12 +460,20 @@ static int modify_fentry(struct bpf_trampoline *tr, u32 orig_flags, · 2 unmodified lines | ||
| 411 | 460 | ret = bpf_trampoline_update_fentry(tr, orig_flags, old_addr, |
| 412 | 461 | new_addr); |
| 413 | 462 | } |
| 414 | − return ret; | |
| 463 | + | |
| 464 | + if (ret) | |
| 465 | + return ret; | |
| 466 | + | |
| 467 | + bpf_tramp_image_put(tr->cur_image); | |
| 468 | + tr->cur_image = im; | |
| 469 | + return 0; | |
| 415 | 470 | } |
| 416 | 471 | |
| 417 | 472 | /* first time registering */ |
| 418 | −static int register_fentry(struct bpf_trampoline *tr, void *new_addr) | |
| 473 | +static int register_fentry(struct bpf_trampoline *tr, struct bpf_tramp_image *im, | |
| 474 | + void *data __maybe_unused) | |
| 419 | 475 | { |
| 476 | + void *new_addr = im->image; | |
| 420 | 477 | void *ip = tr->func.addr; |
| 421 | 478 | unsigned long faddr; |
| 422 | 479 | int ret; |
| @@ -434,33 +491,42 @@ static int register_fentry(struct bpf_trampoline *tr, void *new_addr) · 11 unmodified lines | ||
| 434 | 491 | ret = bpf_trampoline_update_fentry(tr, 0, NULL, new_addr); |
| 435 | 492 | } |
| 436 | 493 | |
| 437 | − return ret; | |
| 494 | + if (ret) | |
| 495 | + return ret; | |
| 496 | + | |
| 497 | + tr->cur_image = im; | |
| 498 | + return 0; | |
| 438 | 499 | } |
| 439 | 500 | |
| 440 | −static struct bpf_tramp_links * | |
| 501 | +static const struct bpf_trampoline_ops trampoline_ops = { | |
| 502 | + .register_fentry = register_fentry, | |
| 503 | + .unregister_fentry = unregister_fentry, | |
| 504 | + .modify_fentry = modify_fentry, | |
| 505 | +}; | |
| 506 | + | |
| 507 | +static struct bpf_tramp_nodes * | |
| 441 | 508 | bpf_trampoline_get_progs(const struct bpf_trampoline *tr, int *total, bool *ip_arg) |
| 442 | 509 | { |
| 443 | − struct bpf_tramp_link *link; | |
| 444 | − struct bpf_tramp_links *tlinks; | |
| 445 | − struct bpf_tramp_link **links; | |
| 510 | + struct bpf_tramp_node *node, **nodes; | |
| 511 | + struct bpf_tramp_nodes *tnodes; | |
| 446 | 512 | int kind; |
| 447 | 513 | |
| 448 | 514 | *total = 0; |
| 449 | − tlinks = kzalloc_objs(*tlinks, BPF_TRAMP_MAX); | |
| 450 | − if (!tlinks) | |
| 515 | + tnodes = kzalloc_objs(*tnodes, BPF_TRAMP_MAX); | |
| 516 | + if (!tnodes) | |
| 451 | 517 | return ERR_PTR(-ENOMEM); |
| 452 | 518 | |
| 453 | 519 | for (kind = 0; kind < BPF_TRAMP_MAX; kind++) { |
| 454 | − tlinks[kind].nr_links = tr->progs_cnt[kind]; | |
| 520 | + tnodes[kind].nr_nodes = tr->progs_cnt[kind]; | |
| 455 | 521 | *total += tr->progs_cnt[kind]; |
| 456 | − links = tlinks[kind].links; | |
| 522 | + nodes = tnodes[kind].nodes; | |
| 457 | 523 | |
| 458 | − hlist_for_each_entry(link, &tr->progs_hlist[kind], tramp_hlist) { | |
| 459 | − *ip_arg |= link->link.prog->call_get_func_ip; | |
| 460 | − *links++ = link; | |
| 524 | + hlist_for_each_entry(node, &tr->progs_hlist[kind], tramp_hlist) { | |
| 525 | + *ip_arg |= node->link->prog->call_get_func_ip; | |
| 526 | + *nodes++ = node; | |
| 461 | 527 | } |
| 462 | 528 | } |
| 463 | − return tlinks; | |
| 529 | + return tnodes; | |
| 464 | 530 | } |
| 465 | 531 | |
| 466 | 532 | static void bpf_tramp_image_free(struct bpf_tramp_image *im) |
| @@ -604,30 +670,29 @@ out: · 137 unmodified lines | ||
| 604 | 670 | return ERR_PTR(err); |
| 605 | 671 | } |
| 606 | 672 | |
| 607 | −static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mutex) | |
| 673 | +static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mutex, | |
| 674 | + const struct bpf_trampoline_ops *ops, void *data) | |
| 608 | 675 | { |
| 609 | 676 | struct bpf_tramp_image *im; |
| 610 | − struct bpf_tramp_links *tlinks; | |
| 677 | + struct bpf_tramp_nodes *tnodes; | |
| 611 | 678 | u32 orig_flags = tr->flags; |
| 612 | 679 | bool ip_arg = false; |
| 613 | 680 | int err, total, size; |
| 614 | 681 | |
| 615 | − tlinks = bpf_trampoline_get_progs(tr, &total, &ip_arg); | |
| 616 | − if (IS_ERR(tlinks)) | |
| 617 | − return PTR_ERR(tlinks); | |
| 682 | + tnodes = bpf_trampoline_get_progs(tr, &total, &ip_arg); | |
| 683 | + if (IS_ERR(tnodes)) | |
| 684 | + return PTR_ERR(tnodes); | |
| 618 | 685 | |
| 619 | 686 | if (total == 0) { |
| 620 | − err = unregister_fentry(tr, orig_flags, tr->cur_image->image); | |
| 621 | − bpf_tramp_image_put(tr->cur_image); | |
| 622 | − tr->cur_image = NULL; | |
| 687 | + err = ops->unregister_fentry(tr, orig_flags, data); | |
| 623 | 688 | goto out; |
| 624 | 689 | } |
| 625 | 690 | |
| 626 | 691 | /* clear all bits except SHARE_IPMODIFY and TAIL_CALL_CTX */ |
| 627 | 692 | tr->flags &= (BPF_TRAMP_F_SHARE_IPMODIFY | BPF_TRAMP_F_TAIL_CALL_CTX); |
| 628 | 693 | |
| 629 | − if (tlinks[BPF_TRAMP_FEXIT].nr_links || | |
| 630 | − tlinks[BPF_TRAMP_MODIFY_RETURN].nr_links) { | |
| 694 | + if (tnodes[BPF_TRAMP_FEXIT].nr_nodes || | |
| 695 | + tnodes[BPF_TRAMP_MODIFY_RETURN].nr_nodes) { | |
| 631 | 696 | /* NOTE: BPF_TRAMP_F_RESTORE_REGS and BPF_TRAMP_F_SKIP_FRAME |
| 632 | 697 | * should not be set together. |
| 633 | 698 | */ |
| @@ -658,7 +723,7 @@ again: · 24 unmodified lines | ||
| 658 | 723 | #endif |
| 659 | 724 | |
| 660 | 725 | size = arch_bpf_trampoline_size(&tr->func.model, tr->flags, |
| 661 | − tlinks, tr->func.addr); | |
| 726 | + tnodes, tr->func.addr); | |
| 662 | 727 | if (size < 0) { |
| 663 | 728 | err = size; |
| 664 | 729 | goto out; |
| @@ -676,7 +741,7 @@ again: · 11 unmodified lines | ||
| 676 | 741 | } |
| 677 | 742 | |
| 678 | 743 | err = arch_prepare_bpf_trampoline(im, im->image, im->image + size, |
| 679 | − &tr->func.model, tr->flags, tlinks, | |
| 744 | + &tr->func.model, tr->flags, tnodes, | |
| 680 | 745 | tr->func.addr); |
| 681 | 746 | if (err < 0) |
| 682 | 747 | goto out_free; |
| @@ -685,14 +750,12 @@ again: · 2 unmodified lines | ||
| 685 | 750 | if (err) |
| 686 | 751 | goto out_free; |
| 687 | 752 | |
| 688 | − WARN_ON(tr->cur_image && total == 0); | |
| 689 | 753 | if (tr->cur_image) |
| 690 | 754 | /* progs already running at this address */ |
| 691 | − err = modify_fentry(tr, orig_flags, tr->cur_image->image, | |
| 692 | − im->image, lock_direct_mutex); | |
| 755 | + err = ops->modify_fentry(tr, orig_flags, im, lock_direct_mutex, data); | |
| 693 | 756 | else |
| 694 | 757 | /* first time registering */ |
| 695 | − err = register_fentry(tr, im->image); | |
| 758 | + err = ops->register_fentry(tr, im, data); | |
| 696 | 759 | |
| 697 | 760 | #ifdef CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS |
| 698 | 761 | if (err == -EAGAIN) { |
| @@ -704,34 +767,31 @@ again: · 5 unmodified lines | ||
| 704 | 767 | goto again; |
| 705 | 768 | } |
| 706 | 769 | #endif |
| 707 | − if (err) | |
| 708 | − goto out_free; | |
| 709 | 770 | |
| 710 | − if (tr->cur_image) | |
| 711 | − bpf_tramp_image_put(tr->cur_image); | |
| 712 | − tr->cur_image = im; | |
| 771 | +out_free: | |
| 772 | + if (err) | |
| 773 | + bpf_tramp_image_free(im); | |
| 713 | 774 | out: |
| 714 | 775 | /* If any error happens, restore previous flags */ |
| 715 | 776 | if (err) |
| 716 | 777 | tr->flags = orig_flags; |
| 717 | − kfree(tlinks); | |
| 778 | + kfree(tnodes); | |
| 718 | 779 | return err; |
| 719 | − | |
| 720 | −out_free: | |
| 721 | − bpf_tramp_image_free(im); | |
| 722 | − goto out; | |
| 723 | 780 | } |
| 724 | 781 | |
| 725 | 782 | static enum bpf_tramp_prog_type bpf_attach_type_to_tramp(struct bpf_prog *prog) |
| 726 | 783 | { |
| 727 | 784 | switch (prog->expected_attach_type) { |
| 728 | 785 | case BPF_TRACE_FENTRY: |
| 786 | + case BPF_TRACE_FENTRY_MULTI: | |
| 729 | 787 | return BPF_TRAMP_FENTRY; |
| 730 | 788 | case BPF_MODIFY_RETURN: |
| 731 | 789 | return BPF_TRAMP_MODIFY_RETURN; |
| 732 | 790 | case BPF_TRACE_FEXIT: |
| 791 | + case BPF_TRACE_FEXIT_MULTI: | |
| 733 | 792 | return BPF_TRAMP_FEXIT; |
| 734 | 793 | case BPF_TRACE_FSESSION: |
| 794 | + case BPF_TRACE_FSESSION_MULTI: | |
| 735 | 795 | return BPF_TRAMP_FSESSION; |
| 736 | 796 | case BPF_LSM_MAC: |
| 737 | 797 | if (!prog->aux->attach_func_proto->type) |
| @@ -764,39 +824,33 @@ static int bpf_freplace_check_tgt_prog(struct bpf_prog *tgt_prog) · 26 unmodified lines | ||
| 764 | 824 | return 0; |
| 765 | 825 | } |
| 766 | 826 | |
| 767 | −static int __bpf_trampoline_link_prog(struct bpf_tramp_link *link, | |
| 768 | − struct bpf_trampoline *tr, | |
| 769 | − struct bpf_prog *tgt_prog) | |
| 827 | +static struct bpf_tramp_node *fsession_exit(struct bpf_tramp_node *node) | |
| 770 | 828 | { |
| 771 | − struct bpf_fsession_link *fslink = NULL; | |
| 772 | − enum bpf_tramp_prog_type kind; | |
| 773 | − struct bpf_tramp_link *link_exiting; | |
| 774 | − struct hlist_head *prog_list; | |
| 775 | − int err = 0; | |
| 776 | − int cnt = 0, i; | |
| 829 | + if (node->link->type == BPF_LINK_TYPE_TRACING) { | |
| 830 | + struct bpf_tracing_link *link; | |
| 777 | 831 | |
| 778 | − kind = bpf_attach_type_to_tramp(link->link.prog); | |
| 779 | − if (tr->extension_prog) | |
| 780 | − /* cannot attach fentry/fexit if extension prog is attached. | |
| 781 | − * cannot overwrite extension prog either. | |
| 782 | − */ | |
| 783 | − return -EBUSY; | |
| 832 | + link = container_of(node->link, struct bpf_tracing_link, link.link); | |
| 833 | + return &link->fexit; | |
| 834 | + } else if (node->link->type == BPF_LINK_TYPE_TRACING_MULTI) { | |
| 835 | + struct bpf_tracing_multi_link *link; | |
| 836 | + struct bpf_tracing_multi_node *mnode; | |
| 784 | 837 | |
| 785 | − for (i = 0; i < BPF_TRAMP_MAX; i++) | |
| 786 | − cnt += tr->progs_cnt[i]; | |
| 787 | − | |
| 788 | − if (kind == BPF_TRAMP_REPLACE) { | |
| 789 | − /* Cannot attach extension if fentry/fexit are in use. */ | |
| 790 | − if (cnt) | |
| 791 | − return -EBUSY; | |
| 792 | − err = bpf_freplace_check_tgt_prog(tgt_prog); | |
| 793 | − if (err) | |
| 794 | − return err; | |
| 795 | − tr->extension_prog = link->link.prog; | |
| 796 | − return bpf_arch_text_poke(tr->func.addr, BPF_MOD_NOP, | |
| 797 | − BPF_MOD_JUMP, NULL, | |
| 798 | − link->link.prog->bpf_func); | |
| 838 | + link = container_of(node->link, struct bpf_tracing_multi_link, link); | |
| 839 | + mnode = container_of(node, struct bpf_tracing_multi_node, node); | |
| 840 | + return &link->fexits[mnode - link->nodes]; | |
| 799 | 841 | } |
| 842 | + return NULL; | |
| 843 | +} | |
| 844 | + | |
| 845 | +static int bpf_trampoline_add_prog(struct bpf_trampoline *tr, | |
| 846 | + struct bpf_tramp_node *node, | |
| 847 | + int cnt) | |
| 848 | +{ | |
| 849 | + enum bpf_tramp_prog_type kind; | |
| 850 | + struct bpf_tramp_node *node_existing, *fexit; | |
| 851 | + struct hlist_head *prog_list; | |
| 852 | + | |
| 853 | + kind = bpf_attach_type_to_tramp(node->link->prog); | |
| 800 | 854 | if (kind == BPF_TRAMP_FSESSION) { |
| 801 | 855 | prog_list = &tr->progs_hlist[BPF_TRAMP_FENTRY]; |
| 802 | 856 | cnt++; |
| @@ -805,59 +859,112 @@ static int __bpf_trampoline_link_prog(struct bpf_tramp_link *link, · 2 unmodified lines | ||
| 805 | 859 | } |
| 806 | 860 | if (cnt >= BPF_MAX_TRAMP_LINKS) |
| 807 | 861 | return -E2BIG; |
| 808 | − if (!hlist_unhashed(&link->tramp_hlist)) | |
| 862 | + if (!hlist_unhashed(&node->tramp_hlist)) | |
| 809 | 863 | /* prog already linked */ |
| 810 | 864 | return -EBUSY; |
| 811 | − hlist_for_each_entry(link_exiting, prog_list, tramp_hlist) { | |
| 812 | − if (link_exiting->link.prog != link->link.prog) | |
| 865 | + hlist_for_each_entry(node_existing, prog_list, tramp_hlist) { | |
| 866 | + if (node_existing->link->prog != node->link->prog) | |
| 813 | 867 | continue; |
| 814 | 868 | /* prog already linked */ |
| 815 | 869 | return -EBUSY; |
| 816 | 870 | } |
| 817 | 871 | |
| 818 | − hlist_add_head(&link->tramp_hlist, prog_list); | |
| 872 | + hlist_add_head(&node->tramp_hlist, prog_list); | |
| 819 | 873 | if (kind == BPF_TRAMP_FSESSION) { |
| 820 | 874 | tr->progs_cnt[BPF_TRAMP_FENTRY]++; |
| 821 | − fslink = container_of(link, struct bpf_fsession_link, link.link); | |
| 822 | − hlist_add_head(&fslink->fexit.tramp_hlist, &tr->progs_hlist[BPF_TRAMP_FEXIT]); | |
| 875 | + fexit = fsession_exit(node); | |
| 876 | + if (WARN_ON_ONCE(!fexit)) | |
| 877 | + return -EINVAL; | |
| 878 | + hlist_add_head(&fexit->tramp_hlist, &tr->progs_hlist[BPF_TRAMP_FEXIT]); | |
| 823 | 879 | tr->progs_cnt[BPF_TRAMP_FEXIT]++; |
| 824 | 880 | } else { |
| 825 | 881 | tr->progs_cnt[kind]++; |
| 826 | 882 | } |
| 827 | − err = bpf_trampoline_update(tr, true /* lock_direct_mutex */); | |
| 828 | − if (err) { | |
| 829 | − hlist_del_init(&link->tramp_hlist); | |
| 830 | − if (kind == BPF_TRAMP_FSESSION) { | |
| 831 | − tr->progs_cnt[BPF_TRAMP_FENTRY]--; | |
| 832 | − hlist_del_init(&fslink->fexit.tramp_hlist); | |
| 833 | − tr->progs_cnt[BPF_TRAMP_FEXIT]--; | |
| 834 | − } else { | |
| 835 | − tr->progs_cnt[kind]--; | |
| 836 | − } | |
| 883 | + return 0; | |
| 884 | +} | |
| 885 | + | |
| 886 | +static void bpf_trampoline_remove_prog(struct bpf_trampoline *tr, | |
| 887 | + struct bpf_tramp_node *node) | |
| 888 | +{ | |
| 889 | + enum bpf_tramp_prog_type kind; | |
| 890 | + struct bpf_tramp_node *fexit; | |
| 891 | + | |
| 892 | + kind = bpf_attach_type_to_tramp(node->link->prog); | |
| 893 | + if (kind == BPF_TRAMP_FSESSION) { | |
| 894 | + fexit = fsession_exit(node); | |
| 895 | + if (WARN_ON_ONCE(!fexit)) | |
| 896 | + return; | |
| 897 | + hlist_del_init(&fexit->tramp_hlist); | |
| 898 | + tr->progs_cnt[BPF_TRAMP_FEXIT]--; | |
| 899 | + kind = BPF_TRAMP_FENTRY; | |
| 837 | 900 | } |
| 901 | + hlist_del_init(&node->tramp_hlist); | |
| 902 | + tr->progs_cnt[kind]--; | |
| 903 | +} | |
| 904 | + | |
| 905 | +static int __bpf_trampoline_link_prog(struct bpf_tramp_node *node, | |
| 906 | + struct bpf_trampoline *tr, | |
| 907 | + struct bpf_prog *tgt_prog, | |
| 908 | + const struct bpf_trampoline_ops *ops, | |
| 909 | + void *data) | |
| 910 | +{ | |
| 911 | + enum bpf_tramp_prog_type kind; | |
| 912 | + int err = 0; | |
| 913 | + int cnt = 0, i; | |
| 914 | + | |
| 915 | + kind = bpf_attach_type_to_tramp(node->link->prog); | |
| 916 | + if (tr->extension_prog) | |
| 917 | + /* cannot attach fentry/fexit if extension prog is attached. | |
| 918 | + * cannot overwrite extension prog either. | |
| 919 | + */ | |
| 920 | + return -EBUSY; | |
| 921 | + | |
| 922 | + for (i = 0; i < BPF_TRAMP_MAX; i++) | |
| 923 | + cnt += tr->progs_cnt[i]; | |
| 924 | + | |
| 925 | + if (kind == BPF_TRAMP_REPLACE) { | |
| 926 | + /* Cannot attach extension if fentry/fexit are in use. */ | |
| 927 | + if (cnt) | |
| 928 | + return -EBUSY; | |
| 929 | + err = bpf_freplace_check_tgt_prog(tgt_prog); | |
| 930 | + if (err) | |
| 931 | + return err; | |
| 932 | + tr->extension_prog = node->link->prog; | |
| 933 | + return bpf_arch_text_poke(tr->func.addr, BPF_MOD_NOP, | |
| 934 | + BPF_MOD_JUMP, NULL, | |
| 935 | + node->link->prog->bpf_func); | |
| 936 | + } | |
| 937 | + err = bpf_trampoline_add_prog(tr, node, cnt); | |
| 938 | + if (err) | |
| 939 | + return err; | |
| 940 | + err = bpf_trampoline_update(tr, true /* lock_direct_mutex */, ops, data); | |
| 941 | + if (err) | |
| 942 | + bpf_trampoline_remove_prog(tr, node); | |
| 838 | 943 | return err; |
| 839 | 944 | } |
| 840 | 945 | |
| 841 | −int bpf_trampoline_link_prog(struct bpf_tramp_link *link, | |
| 946 | +int bpf_trampoline_link_prog(struct bpf_tramp_node *node, | |
| 842 | 947 | struct bpf_trampoline *tr, |
| 843 | 948 | struct bpf_prog *tgt_prog) |
| 844 | 949 | { |
| 845 | 950 | int err; |
| 846 | 951 | |
| 847 | − mutex_lock(&tr->mutex); | |
| 848 | − err = __bpf_trampoline_link_prog(link, tr, tgt_prog); | |
| 849 | − mutex_unlock(&tr->mutex); | |
| 952 | + trampoline_lock(tr); | |
| 953 | + err = __bpf_trampoline_link_prog(node, tr, tgt_prog, &trampoline_ops, NULL); | |
| 954 | + trampoline_unlock(tr); | |
| 850 | 955 | return err; |
| 851 | 956 | } |
| 852 | 957 | |
| 853 | −static int __bpf_trampoline_unlink_prog(struct bpf_tramp_link *link, | |
| 958 | +static int __bpf_trampoline_unlink_prog(struct bpf_tramp_node *node, | |
| 854 | 959 | struct bpf_trampoline *tr, |
| 855 | − struct bpf_prog *tgt_prog) | |
| 960 | + struct bpf_prog *tgt_prog, | |
| 961 | + const struct bpf_trampoline_ops *ops, | |
| 962 | + void *data) | |
| 856 | 963 | { |
| 857 | 964 | enum bpf_tramp_prog_type kind; |
| 858 | 965 | int err; |
| 859 | 966 | |
| 860 | − kind = bpf_attach_type_to_tramp(link->link.prog); | |
| 967 | + kind = bpf_attach_type_to_tramp(node->link->prog); | |
| 861 | 968 | if (kind == BPF_TRAMP_REPLACE) { |
| 862 | 969 | WARN_ON_ONCE(!tr->extension_prog); |
| 863 | 970 | err = bpf_arch_text_poke(tr->func.addr, BPF_MOD_JUMP, |
| @@ -867,29 +974,21 @@ static int __bpf_trampoline_unlink_prog(struct bpf_tramp_link *link, · 3 unmodified lines | ||
| 867 | 974 | guard(mutex)(&tgt_prog->aux->ext_mutex); |
| 868 | 975 | tgt_prog->aux->is_extended = false; |
| 869 | 976 | return err; |
| 870 | − } else if (kind == BPF_TRAMP_FSESSION) { | |
| 871 | − struct bpf_fsession_link *fslink = | |
| 872 | − container_of(link, struct bpf_fsession_link, link.link); | |
| 873 | − | |
| 874 | − hlist_del_init(&fslink->fexit.tramp_hlist); | |
| 875 | − tr->progs_cnt[BPF_TRAMP_FEXIT]--; | |
| 876 | − kind = BPF_TRAMP_FENTRY; | |
| 877 | 977 | } |
| 878 | − hlist_del_init(&link->tramp_hlist); | |
| 879 | − tr->progs_cnt[kind]--; | |
| 880 | − return bpf_trampoline_update(tr, true /* lock_direct_mutex */); | |
| 978 | + bpf_trampoline_remove_prog(tr, node); | |
| 979 | + return bpf_trampoline_update(tr, true /* lock_direct_mutex */, ops, data); | |
| 881 | 980 | } |
| 882 | 981 | |
| 883 | 982 | /* bpf_trampoline_unlink_prog() should never fail. */ |
| 884 | −int bpf_trampoline_unlink_prog(struct bpf_tramp_link *link, | |
| 983 | +int bpf_trampoline_unlink_prog(struct bpf_tramp_node *node, | |
| 885 | 984 | struct bpf_trampoline *tr, |
| 886 | 985 | struct bpf_prog *tgt_prog) |
| 887 | 986 | { |
| 888 | 987 | int err; |
| 889 | 988 | |
| 890 | − mutex_lock(&tr->mutex); | |
| 891 | − err = __bpf_trampoline_unlink_prog(link, tr, tgt_prog); | |
| 892 | − mutex_unlock(&tr->mutex); | |
| 989 | + trampoline_lock(tr); | |
| 990 | + err = __bpf_trampoline_unlink_prog(node, tr, tgt_prog, &trampoline_ops, NULL); | |
| 991 | + trampoline_unlock(tr); | |
| 893 | 992 | return err; |
| 894 | 993 | } |
| 895 | 994 | |
| @@ -903,7 +1002,7 @@ static void bpf_shim_tramp_link_release(struct bpf_link *link) · 7 unmodified lines | ||
| 903 | 1002 | if (!shim_link->trampoline) |
| 904 | 1003 | return; |
| 905 | 1004 | |
| 906 | − WARN_ON_ONCE(bpf_trampoline_unlink_prog(&shim_link->link, shim_link->trampoline, NULL)); | |
| 1005 | + WARN_ON_ONCE(bpf_trampoline_unlink_prog(&shim_link->link.node, shim_link->trampoline, NULL)); | |
| 907 | 1006 | bpf_trampoline_put(shim_link->trampoline); |
| 908 | 1007 | } |
| 909 | 1008 | |
| @@ -949,8 +1048,8 @@ static struct bpf_shim_tramp_link *cgroup_shim_alloc(const struct bpf_prog *prog · 39 unmodified lines | ||
| 949 | 1048 | p->type = BPF_PROG_TYPE_LSM; |
| 950 | 1049 | p->expected_attach_type = BPF_LSM_MAC; |
| 951 | 1050 | bpf_prog_inc(p); |
| 952 | − bpf_link_init(&shim_link->link.link, BPF_LINK_TYPE_UNSPEC, | |
| 953 | − &bpf_shim_tramp_link_lops, p, attach_type); | |
| 1051 | + bpf_tramp_link_init(&shim_link->link, BPF_LINK_TYPE_UNSPEC, | |
| 1052 | + &bpf_shim_tramp_link_lops, p, attach_type, 0); | |
| 954 | 1053 | bpf_cgroup_atype_get(p->aux->attach_btf_id, cgroup_atype); |
| 955 | 1054 | |
| 956 | 1055 | return shim_link; |
| @@ -959,15 +1058,15 @@ static struct bpf_shim_tramp_link *cgroup_shim_alloc(const struct bpf_prog *prog · 2 unmodified lines | ||
| 959 | 1058 | static struct bpf_shim_tramp_link *cgroup_shim_find(struct bpf_trampoline *tr, |
| 960 | 1059 | bpf_func_t bpf_func) |
| 961 | 1060 | { |
| 962 | − struct bpf_tramp_link *link; | |
| 1061 | + struct bpf_tramp_node *node; | |
| 963 | 1062 | int kind; |
| 964 | 1063 | |
| 965 | 1064 | for (kind = 0; kind < BPF_TRAMP_MAX; kind++) { |
| 966 | − hlist_for_each_entry(link, &tr->progs_hlist[kind], tramp_hlist) { | |
| 967 | − struct bpf_prog *p = link->link.prog; | |
| 1065 | + hlist_for_each_entry(node, &tr->progs_hlist[kind], tramp_hlist) { | |
| 1066 | + struct bpf_prog *p = node->link->prog; | |
| 968 | 1067 | |
| 969 | 1068 | if (p->bpf_func == bpf_func) |
| 970 | − return container_of(link, struct bpf_shim_tramp_link, link); | |
| 1069 | + return container_of(node, struct bpf_shim_tramp_link, link.node); | |
| 971 | 1070 | } |
| 972 | 1071 | } |
| 973 | 1072 | |
| @@ -999,12 +1098,12 @@ int bpf_trampoline_link_cgroup_shim(struct bpf_prog *prog, · 25 unmodified lines | ||
| 999 | 1098 | if (!tr) |
| 1000 | 1099 | return -ENOMEM; |
| 1001 | 1100 | |
| 1002 | − mutex_lock(&tr->mutex); | |
| 1101 | + trampoline_lock(tr); | |
| 1003 | 1102 | |
| 1004 | 1103 | shim_link = cgroup_shim_find(tr, bpf_func); |
| 1005 | 1104 | if (shim_link && !IS_ERR(bpf_link_inc_not_zero(&shim_link->link.link))) { |
| 1006 | 1105 | /* Reusing existing shim attached by the other program. */ |
| 1007 | − mutex_unlock(&tr->mutex); | |
| 1106 | + trampoline_unlock(tr); | |
| 1008 | 1107 | bpf_trampoline_put(tr); /* bpf_trampoline_get above */ |
| 1009 | 1108 | return 0; |
| 1010 | 1109 | } |
| @@ -1017,23 +1116,23 @@ int bpf_trampoline_link_cgroup_shim(struct bpf_prog *prog, · 6 unmodified lines | ||
| 1017 | 1116 | goto err; |
| 1018 | 1117 | } |
| 1019 | 1118 | |
| 1020 | − err = __bpf_trampoline_link_prog(&shim_link->link, tr, NULL); | |
| 1119 | + err = __bpf_trampoline_link_prog(&shim_link->link.node, tr, NULL, &trampoline_ops, NULL); | |
| 1021 | 1120 | if (err) |
| 1022 | 1121 | goto err; |
| 1023 | 1122 | |
| 1024 | 1123 | shim_link->trampoline = tr; |
| 1025 | 1124 | /* note, we're still holding tr refcnt from above */ |
| 1026 | 1125 | |
| 1027 | − mutex_unlock(&tr->mutex); | |
| 1126 | + trampoline_unlock(tr); | |
| 1028 | 1127 | |
| 1029 | 1128 | return 0; |
| 1030 | 1129 | err: |
| 1031 | − mutex_unlock(&tr->mutex); | |
| 1130 | + trampoline_unlock(tr); | |
| 1032 | 1131 | |
| 1033 | 1132 | if (shim_link) |
| 1034 | 1133 | bpf_link_put(&shim_link->link.link); |
| 1035 | 1134 | |
| 1036 | − /* have to release tr while _not_ holding its mutex */ | |
| 1135 | + /* have to release tr while _not_ holding pool mutex for trampoline */ | |
| 1037 | 1136 | bpf_trampoline_put(tr); /* bpf_trampoline_get above */ |
| 1038 | 1137 | |
| 1039 | 1138 | return err; |
| @@ -1054,9 +1153,9 @@ void bpf_trampoline_unlink_cgroup_shim(struct bpf_prog *prog) · 14 unmodified lines | ||
| 1054 | 1153 | if (WARN_ON_ONCE(!tr)) |
| 1055 | 1154 | return; |
| 1056 | 1155 | |
| 1057 | − mutex_lock(&tr->mutex); | |
| 1156 | + trampoline_lock(tr); | |
| 1058 | 1157 | shim_link = cgroup_shim_find(tr, bpf_func); |
| 1059 | − mutex_unlock(&tr->mutex); | |
| 1158 | + trampoline_unlock(tr); | |
| 1060 | 1159 | |
| 1061 | 1160 | if (shim_link) |
| 1062 | 1161 | bpf_link_put(&shim_link->link.link); |
| @@ -1074,14 +1173,14 @@ struct bpf_trampoline *bpf_trampoline_get(u64 key, · 11 unmodified lines | ||
| 1074 | 1173 | if (!tr) |
| 1075 | 1174 | return NULL; |
| 1076 | 1175 | |
| 1077 | − mutex_lock(&tr->mutex); | |
| 1176 | + trampoline_lock(tr); | |
| 1078 | 1177 | if (tr->func.addr) |
| 1079 | 1178 | goto out; |
| 1080 | 1179 | |
| 1081 | 1180 | memcpy(&tr->func.model, &tgt_info->fmodel, sizeof(tgt_info->fmodel)); |
| 1082 | 1181 | tr->func.addr = (void *)tgt_info->tgt_addr; |
| 1083 | 1182 | out: |
| 1084 | − mutex_unlock(&tr->mutex); | |
| 1183 | + trampoline_unlock(tr); | |
| 1085 | 1184 | return tr; |
| 1086 | 1185 | } |
| 1087 | 1186 | |
| @@ -1094,7 +1193,6 @@ void bpf_trampoline_put(struct bpf_trampoline *tr) · 6 unmodified lines | ||
| 1094 | 1193 | mutex_lock(&trampoline_mutex); |
| 1095 | 1194 | if (!refcount_dec_and_test(&tr->refcnt)) |
| 1096 | 1195 | goto out; |
| 1097 | − WARN_ON_ONCE(mutex_is_locked(&tr->mutex)); | |
| 1098 | 1196 | |
| 1099 | 1197 | for (i = 0; i < BPF_TRAMP_MAX; i++) |
| 1100 | 1198 | if (WARN_ON_ONCE(!hlist_empty(&tr->progs_hlist[i]))) |
| @@ -1333,7 +1431,7 @@ bpf_trampoline_exit_t bpf_trampoline_exit(const struct bpf_prog *prog) · 232 unmodified lines | ||
| 1333 | 1431 | int __weak |
| 1334 | 1432 | arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *image_end, |
| 1335 | 1433 | const struct btf_func_model *m, u32 flags, |
| 1336 | − struct bpf_tramp_links *tlinks, | |
| 1434 | + struct bpf_tramp_nodes *tnodes, | |
| 1337 | 1435 | void *func_addr) |
| 1338 | 1436 | { |
| 1339 | 1437 | return -ENOTSUPP; |
| @@ -1367,11 +1465,288 @@ int __weak arch_protect_bpf_trampoline(void *image, unsigned int size) · 27 unmodified lines | ||
| 1367 | 1465 | } |
| 1368 | 1466 | |
| 1369 | 1467 | int __weak arch_bpf_trampoline_size(const struct btf_func_model *m, u32 flags, |
| 1370 | − struct bpf_tramp_links *tlinks, void *func_addr) | |
| 1468 | + struct bpf_tramp_nodes *tnodes, void *func_addr) | |
| 1371 | 1469 | { |
| 1372 | 1470 | return -ENOTSUPP; |
| 1373 | 1471 | } |
| 1374 | 1472 | |
| 1473 | +#if defined(CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS) && \ | |
| 1474 | + defined(CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS) && \ | |
| 1475 | + defined(CONFIG_BPF_SYSCALL) | |
| 1476 | + | |
| 1477 | +static void trampoline_lock_all(void) | |
| 1478 | +{ | |
| 1479 | + int i; | |
| 1480 | + | |
| 1481 | + for (i = 0; i < TRAMPOLINE_LOCKS_TABLE_SIZE; i++) | |
| 1482 | + mutex_lock(&trampoline_locks[i].mutex); | |
| 1483 | +} | |
| 1484 | + | |
| 1485 | +static void trampoline_unlock_all(void) | |
| 1486 | +{ | |
| 1487 | + int i; | |
| 1488 | + | |
| 1489 | + for (i = 0; i < TRAMPOLINE_LOCKS_TABLE_SIZE; i++) | |
| 1490 | + mutex_unlock(&trampoline_locks[i].mutex); | |
| 1491 | +} | |
| 1492 | + | |
| 1493 | +static void remove_tracing_multi_data(struct bpf_tracing_multi_data *data) | |
| 1494 | +{ | |
| 1495 | + ftrace_hash_remove(data->reg); | |
| 1496 | + ftrace_hash_remove(data->unreg); | |
| 1497 | + ftrace_hash_remove(data->modify); | |
| 1498 | +} | |
| 1499 | + | |
| 1500 | +static void clear_tracing_multi_data(struct bpf_tracing_multi_data *data) | |
| 1501 | +{ | |
| 1502 | + remove_tracing_multi_data(data); | |
| 1503 | + | |
| 1504 | + free_ftrace_hash(data->reg); | |
| 1505 | + free_ftrace_hash(data->unreg); | |
| 1506 | + free_ftrace_hash(data->modify); | |
| 1507 | +} | |
| 1508 | + | |
| 1509 | +static int init_tracing_multi_data(struct bpf_tracing_multi_data *data) | |
| 1510 | +{ | |
| 1511 | + data->reg = alloc_ftrace_hash(FTRACE_HASH_DEFAULT_BITS); | |
| 1512 | + data->unreg = alloc_ftrace_hash(FTRACE_HASH_DEFAULT_BITS); | |
| 1513 | + data->modify = alloc_ftrace_hash(FTRACE_HASH_DEFAULT_BITS); | |
| 1514 | + | |
| 1515 | + if (!data->reg || !data->unreg || !data->modify) { | |
| 1516 | + clear_tracing_multi_data(data); | |
| 1517 | + return -ENOMEM; | |
| 1518 | + } | |
| 1519 | + return 0; | |
| 1520 | +} | |
| 1521 | + | |
| 1522 | +static void ftrace_hash_add(struct ftrace_hash *hash, struct ftrace_func_entry *entry, | |
| 1523 | + unsigned long ip, unsigned long direct) | |
| 1524 | +{ | |
| 1525 | + entry->ip = ip; | |
| 1526 | + entry->direct = direct; | |
| 1527 | + add_ftrace_hash_entry(hash, entry); | |
| 1528 | +} | |
| 1529 | + | |
| 1530 | +static int register_fentry_multi(struct bpf_trampoline *tr, struct bpf_tramp_image *im, void *ptr) | |
| 1531 | +{ | |
| 1532 | + unsigned long addr = (unsigned long) im->image; | |
| 1533 | + unsigned long ip = ftrace_location(tr->ip); | |
| 1534 | + struct bpf_tracing_multi_data *data = ptr; | |
| 1535 | + | |
| 1536 | + if (bpf_trampoline_use_jmp(tr->flags)) | |
| 1537 | + addr = ftrace_jmp_set(addr); | |
| 1538 | + | |
| 1539 | + ftrace_hash_add(data->reg, data->entry, ip, addr); | |
| 1540 | + tr->cur_image = im; | |
| 1541 | + return 0; | |
| 1542 | +} | |
| 1543 | + | |
| 1544 | +static int unregister_fentry_multi(struct bpf_trampoline *tr, u32 orig_flags, void *ptr) | |
| 1545 | +{ | |
| 1546 | + unsigned long addr = (unsigned long) tr->cur_image->image; | |
| 1547 | + unsigned long ip = ftrace_location(tr->ip); | |
| 1548 | + struct bpf_tracing_multi_data *data = ptr; | |
| 1549 | + | |
| 1550 | + if (bpf_trampoline_use_jmp(tr->flags)) | |
| 1551 | + addr = ftrace_jmp_set(addr); | |
| 1552 | + | |
| 1553 | + ftrace_hash_add(data->unreg, data->entry, ip, addr); | |
| 1554 | + tr->cur_image = NULL; | |
| 1555 | + return 0; | |
| 1556 | +} | |
| 1557 | + | |
| 1558 | +static int modify_fentry_multi(struct bpf_trampoline *tr, u32 orig_flags, struct bpf_tramp_image *im, | |
| 1559 | + bool lock_direct_mutex, void *ptr) | |
| 1560 | +{ | |
| 1561 | + unsigned long addr = (unsigned long) im->image; | |
| 1562 | + unsigned long ip = ftrace_location(tr->ip); | |
| 1563 | + struct bpf_tracing_multi_data *data = ptr; | |
| 1564 | + | |
| 1565 | + if (bpf_trampoline_use_jmp(tr->flags)) | |
| 1566 | + addr = ftrace_jmp_set(addr); | |
| 1567 | + | |
| 1568 | + ftrace_hash_add(data->modify, data->entry, ip, addr); | |
| 1569 | + tr->cur_image = im; | |
| 1570 | + return 0; | |
| 1571 | +} | |
| 1572 | + | |
| 1573 | +static const struct bpf_trampoline_ops trampoline_multi_ops = { | |
| 1574 | + .register_fentry = register_fentry_multi, | |
| 1575 | + .unregister_fentry = unregister_fentry_multi, | |
| 1576 | + .modify_fentry = modify_fentry_multi, | |
| 1577 | +}; | |
| 1578 | + | |
| 1579 | +static void bpf_trampoline_multi_attach_init(struct bpf_trampoline *tr) | |
| 1580 | +{ | |
| 1581 | + tr->multi_attach.old_image = tr->cur_image; | |
| 1582 | + tr->multi_attach.old_flags = tr->flags; | |
| 1583 | +} | |
| 1584 | + | |
| 1585 | +static void bpf_trampoline_multi_attach_free(struct bpf_trampoline *tr) | |
| 1586 | +{ | |
| 1587 | + if (tr->multi_attach.old_image) | |
| 1588 | + bpf_tramp_image_put(tr->multi_attach.old_image); | |
| 1589 | + | |
| 1590 | + tr->multi_attach.old_image = NULL; | |
| 1591 | + tr->multi_attach.old_flags = 0; | |
| 1592 | +} | |
| 1593 | + | |
| 1594 | +static void bpf_trampoline_multi_attach_rollback(struct bpf_trampoline *tr) | |
| 1595 | +{ | |
| 1596 | + if (tr->cur_image) | |
| 1597 | + bpf_tramp_image_put(tr->cur_image); | |
| 1598 | + tr->cur_image = tr->multi_attach.old_image; | |
| 1599 | + tr->flags = tr->multi_attach.old_flags; | |
| 1600 | + | |
| 1601 | + tr->multi_attach.old_image = NULL; | |
| 1602 | + tr->multi_attach.old_flags = 0; | |
| 1603 | +} | |
| 1604 | + | |
| 1605 | +#define for_each_mnode_cnt(mnode, link, cnt) \ | |
| 1606 | + for (i = 0, mnode = &link->nodes[i]; i < cnt; i++, mnode = &link->nodes[i]) | |
| 1607 | + | |
| 1608 | +#define for_each_mnode(mnode, link) \ | |
| 1609 | + for_each_mnode_cnt(mnode, link, link->nodes_cnt) | |
| 1610 | + | |
| 1611 | +int bpf_trampoline_multi_attach(struct bpf_prog *prog, u32 *ids, | |
| 1612 | + struct bpf_tracing_multi_link *link) | |
| 1613 | +{ | |
| 1614 | + struct bpf_tracing_multi_data *data = &link->data; | |
| 1615 | + struct bpf_attach_target_info tgt_info = {}; | |
| 1616 | + struct btf *btf = prog->aux->attach_btf; | |
| 1617 | + struct bpf_tracing_multi_node *mnode; | |
| 1618 | + struct bpf_trampoline *tr; | |
| 1619 | + int i, err, rollback_cnt; | |
| 1620 | + u64 key; | |
| 1621 | + | |
| 1622 | + for_each_mnode(mnode, link) { | |
| 1623 | + rollback_cnt = i; | |
| 1624 | + | |
| 1625 | + err = bpf_check_attach_btf_id_multi(btf, prog, ids[i], &tgt_info); | |
| 1626 | + if (err) | |
| 1627 | + goto rollback_put; | |
| 1628 | + | |
| 1629 | + key = bpf_trampoline_compute_key(NULL, btf, ids[i]); | |
| 1630 | + | |
| 1631 | + tr = bpf_trampoline_get(key, &tgt_info); | |
| 1632 | + if (!tr) { | |
| 1633 | + err = -ENOMEM; | |
| 1634 | + goto rollback_put; | |
| 1635 | + } | |
| 1636 | + | |
| 1637 | + mnode->trampoline = tr; | |
| 1638 | + mnode->node.link = &link->link; | |
| 1639 | + mnode->node.cookie = link->cookies ? link->cookies[i] : 0; | |
| 1640 | + | |
| 1641 | + if (prog->expected_attach_type == BPF_TRACE_FSESSION_MULTI) { | |
| 1642 | + link->fexits[i].link = &link->link; | |
| 1643 | + link->fexits[i].cookie = link->cookies ? link->cookies[i] : 0; | |
| 1644 | + } | |
| 1645 | + | |
| 1646 | + cond_resched(); | |
| 1647 | + } | |
| 1648 | + | |
| 1649 | + err = init_tracing_multi_data(data); | |
| 1650 | + if (err) { | |
| 1651 | + rollback_cnt = link->nodes_cnt; | |
| 1652 | + goto rollback_put; | |
| 1653 | + } | |
| 1654 | + | |
| 1655 | + trampoline_lock_all(); | |
| 1656 | + | |
| 1657 | + for_each_mnode(mnode, link) { | |
| 1658 | + bpf_trampoline_multi_attach_init(mnode->trampoline); | |
| 1659 | + | |
| 1660 | + data->entry = &mnode->entry; | |
| 1661 | + err = __bpf_trampoline_link_prog(&mnode->node, mnode->trampoline, NULL, | |
| 1662 | + &trampoline_multi_ops, data); | |
| 1663 | + if (err) { | |
| 1664 | + rollback_cnt = i; | |
| 1665 | + goto rollback_unlink; | |
| 1666 | + } | |
| 1667 | + } | |
| 1668 | + | |
| 1669 | + rollback_cnt = link->nodes_cnt; | |
| 1670 | + if (ftrace_hash_count(data->reg)) { | |
| 1671 | + err = update_ftrace_direct_add(&direct_ops, data->reg); | |
| 1672 | + if (err) | |
| 1673 | + goto rollback_unlink; | |
| 1674 | + } | |
| 1675 | + | |
| 1676 | + if (ftrace_hash_count(data->modify)) { | |
| 1677 | + err = update_ftrace_direct_mod(&direct_ops, data->modify, true); | |
| 1678 | + if (err) { | |
| 1679 | + if (ftrace_hash_count(data->reg)) | |
| 1680 | + WARN_ON_ONCE(update_ftrace_direct_del(&direct_ops, data->reg)); | |
| 1681 | + goto rollback_unlink; | |
| 1682 | + } | |
| 1683 | + } | |
| 1684 | + | |
| 1685 | + for_each_mnode(mnode, link) | |
| 1686 | + bpf_trampoline_multi_attach_free(mnode->trampoline); | |
| 1687 | + | |
| 1688 | + trampoline_unlock_all(); | |
| 1689 | + | |
| 1690 | + remove_tracing_multi_data(data); | |
| 1691 | + return 0; | |
| 1692 | + | |
| 1693 | +rollback_unlink: | |
| 1694 | + for_each_mnode_cnt(mnode, link, rollback_cnt) { | |
| 1695 | + bpf_trampoline_remove_prog(mnode->trampoline, &mnode->node); | |
| 1696 | + bpf_trampoline_multi_attach_rollback(mnode->trampoline); | |
| 1697 | + } | |
| 1698 | + | |
| 1699 | + trampoline_unlock_all(); | |
| 1700 | + | |
| 1701 | + clear_tracing_multi_data(data); | |
| 1702 | + rollback_cnt = link->nodes_cnt; | |
| 1703 | + | |
| 1704 | +rollback_put: | |
| 1705 | + for_each_mnode_cnt(mnode, link, rollback_cnt) | |
| 1706 | + bpf_trampoline_put(mnode->trampoline); | |
| 1707 | + | |
| 1708 | + return err; | |
| 1709 | +} | |
| 1710 | + | |
| 1711 | +int bpf_trampoline_multi_detach(struct bpf_prog *prog, struct bpf_tracing_multi_link *link) | |
| 1712 | +{ | |
| 1713 | + struct bpf_tracing_multi_data *data = &link->data; | |
| 1714 | + struct bpf_tracing_multi_node *mnode; | |
| 1715 | + int i; | |
| 1716 | + | |
| 1717 | + trampoline_lock_all(); | |
| 1718 | + | |
| 1719 | + for_each_mnode(mnode, link) { | |
| 1720 | + data->entry = &mnode->entry; | |
| 1721 | + bpf_trampoline_multi_attach_init(mnode->trampoline); | |
| 1722 | + WARN_ON_ONCE(__bpf_trampoline_unlink_prog(&mnode->node, mnode->trampoline, | |
| 1723 | + NULL, &trampoline_multi_ops, data)); | |
| 1724 | + } | |
| 1725 | + | |
| 1726 | + if (ftrace_hash_count(data->unreg)) | |
| 1727 | + WARN_ON_ONCE(update_ftrace_direct_del(&direct_ops, data->unreg)); | |
| 1728 | + if (ftrace_hash_count(data->modify)) | |
| 1729 | + WARN_ON_ONCE(update_ftrace_direct_mod(&direct_ops, data->modify, true)); | |
| 1730 | + | |
| 1731 | + for_each_mnode(mnode, link) | |
| 1732 | + bpf_trampoline_multi_attach_free(mnode->trampoline); | |
| 1733 | + | |
| 1734 | + trampoline_unlock_all(); | |
| 1735 | + | |
| 1736 | + for_each_mnode(mnode, link) | |
| 1737 | + bpf_trampoline_put(mnode->trampoline); | |
| 1738 | + | |
| 1739 | + clear_tracing_multi_data(data); | |
| 1740 | + return 0; | |
| 1741 | +} | |
| 1742 | + | |
| 1743 | +#undef for_each_mnode_cnt | |
| 1744 | +#undef for_each_mnode | |
| 1745 | + | |
| 1746 | +#endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS && | |
| 1747 | + CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS && | |
| 1748 | + CONFIG_BPF_SYSCALL */ | |
| 1749 | + | |
| 1375 | 1750 | static int __init init_trampolines(void) |
| 1376 | 1751 | { |
| 1377 | 1752 | int i; |
| @@ -1380,6 +1755,8 @@ static int __init init_trampolines(void) · 2 unmodified lines | ||
| 1380 | 1755 | INIT_HLIST_HEAD(&trampoline_key_table[i]); |
| 1381 | 1756 | for (i = 0; i < TRAMPOLINE_TABLE_SIZE; i++) |
| 1382 | 1757 | INIT_HLIST_HEAD(&trampoline_ip_table[i]); |
| 1758 | + for (i = 0; i < TRAMPOLINE_LOCKS_TABLE_SIZE; i++) | |
| 1759 | + __mutex_init(&trampoline_locks[i].mutex, "trampoline_lock", &trampoline_locks[i].key); | |
| 1383 | 1760 | return 0; |
| 1384 | 1761 | } |
| 1385 | 1762 | late_initcall(init_trampolines); |
| @@ -26,6 +26,7 @@ · 25 unmodified lines | ||
| 26 | 26 | #include <linux/poison.h> |
| 27 | 27 | #include <linux/module.h> |
| 28 | 28 | #include <linux/cpumask.h> |
| 29 | +#include <linux/cnum.h> | |
| 29 | 30 | #include <linux/bpf_mem_alloc.h> |
| 30 | 31 | #include <net/xdp.h> |
| 31 | 32 | #include <linux/trace_events.h> |
| @@ -199,14 +200,15 @@ struct bpf_verifier_stack_elem { · 167 unmodified lines | ||
| 199 | 200 | |
| 200 | 201 | #define BPF_PRIV_STACK_MIN_SIZE 64 |
| 201 | 202 | |
| 202 | −static int acquire_reference(struct bpf_verifier_env *env, int insn_idx); | |
| 203 | −static int release_reference_nomark(struct bpf_verifier_state *state, int ref_obj_id); | |
| 204 | −static int release_reference(struct bpf_verifier_env *env, int ref_obj_id); | |
| 203 | +static int acquire_reference(struct bpf_verifier_env *env, int insn_idx, int parent_id); | |
| 204 | +static int release_reference_nomark(struct bpf_verifier_state *state, int id); | |
| 205 | +static int release_reference(struct bpf_verifier_env *env, int id); | |
| 205 | 206 | static void invalidate_non_owning_refs(struct bpf_verifier_env *env); |
| 206 | 207 | static bool in_rbtree_lock_required_cb(struct bpf_verifier_env *env); |
| 208 | +static bool is_tracing_prog_type(enum bpf_prog_type type); | |
| 207 | 209 | static int ref_set_non_owning(struct bpf_verifier_env *env, |
| 208 | 210 | struct bpf_reg_state *reg); |
| 209 | −static bool is_trusted_reg(const struct bpf_reg_state *reg); | |
| 211 | +static bool is_trusted_reg(struct bpf_verifier_env *env, const struct bpf_reg_state *reg); | |
| 210 | 212 | static inline bool in_sleepable_context(struct bpf_verifier_env *env); |
| 211 | 213 | static const char *non_sleepable_context_description(struct bpf_verifier_env *env); |
| 212 | 214 | static void scalar32_min_max_add(struct bpf_reg_state *dst_reg, struct bpf_reg_state *src_reg); |
| @@ -230,8 +232,28 @@ static void bpf_map_key_store(struct bpf_insn_aux_data *aux, u64 state) · 17 unmodified lines | ||
| 230 | 232 | (poisoned ? BPF_MAP_KEY_POISON : 0ULL); |
| 231 | 233 | } |
| 232 | 234 | |
| 235 | +static void update_ref_obj(struct ref_obj_desc *ref_obj, struct bpf_reg_state *reg) | |
| 236 | +{ | |
| 237 | + ref_obj->id = reg->id; | |
| 238 | + ref_obj->parent_id = reg->parent_id; | |
| 239 | + ref_obj->cnt++; | |
| 240 | +} | |
| 241 | + | |
| 242 | +static int validate_ref_obj(struct bpf_verifier_env *env, struct ref_obj_desc *ref_obj) | |
| 243 | +{ | |
| 244 | + if (ref_obj->cnt > 1) { | |
| 245 | + verifier_bug(env, "function expects only one referenced object but got %d\n", | |
| 246 | + ref_obj->cnt); | |
| 247 | + return -EFAULT; | |
| 248 | + } | |
| 249 | + | |
| 250 | + return 0; | |
| 251 | +} | |
| 252 | + | |
| 233 | 253 | struct bpf_call_arg_meta { |
| 234 | 254 | struct bpf_map_desc map; |
| 255 | + struct bpf_dynptr_desc dynptr; | |
| 256 | + struct ref_obj_desc ref_obj; | |
| 235 | 257 | bool raw_mode; |
| 236 | 258 | bool pkt_access; |
| 237 | 259 | u8 release_regno; |
| @@ -239,8 +261,6 @@ struct bpf_call_arg_meta { · 1 unmodified line | ||
| 239 | 261 | int access_size; |
| 240 | 262 | int mem_size; |
| 241 | 263 | u64 msize_max_value; |
| 242 | − int ref_obj_id; | |
| 243 | − int dynptr_id; | |
| 244 | 264 | int func_id; |
| 245 | 265 | struct btf *btf; |
| 246 | 266 | u32 btf_id; |
| @@ -261,6 +281,41 @@ struct bpf_kfunc_meta { · 14 unmodified lines | ||
| 261 | 281 | |
| 262 | 282 | struct btf *btf_vmlinux; |
| 263 | 283 | |
| 284 | +typedef struct argno { | |
| 285 | + int argno; | |
| 286 | +} argno_t; | |
| 287 | + | |
| 288 | +static argno_t argno_from_reg(u32 regno) | |
| 289 | +{ | |
| 290 | + return (argno_t){ .argno = regno }; | |
| 291 | +} | |
| 292 | + | |
| 293 | +static argno_t argno_from_arg(u32 arg) | |
| 294 | +{ | |
| 295 | + return (argno_t){ .argno = -arg }; | |
| 296 | +} | |
| 297 | + | |
| 298 | +static int reg_from_argno(argno_t a) | |
| 299 | +{ | |
| 300 | + if (a.argno >= 0) | |
| 301 | + return a.argno; | |
| 302 | + if (a.argno >= -MAX_BPF_FUNC_REG_ARGS) | |
| 303 | + return -a.argno; | |
| 304 | + return -1; | |
| 305 | +} | |
| 306 | + | |
| 307 | +static int arg_from_argno(argno_t a) | |
| 308 | +{ | |
| 309 | + if (a.argno < 0) | |
| 310 | + return -a.argno; | |
| 311 | + return -1; | |
| 312 | +} | |
| 313 | + | |
| 314 | +static int arg_idx_from_argno(argno_t a) | |
| 315 | +{ | |
| 316 | + return arg_from_argno(a) - 1; | |
| 317 | +} | |
| 318 | + | |
| 264 | 319 | static const char *btf_type_name(const struct btf *btf, u32 id) |
| 265 | 320 | { |
| 266 | 321 | return btf_name_by_offset(btf, btf_type_by_id(btf, id)->name_off); |
| @@ -290,12 +345,12 @@ static void verbose_invalid_scalar(struct bpf_verifier_env *env, · 23 unmodified lines | ||
| 290 | 345 | bool unknown = true; |
| 291 | 346 | |
| 292 | 347 | verbose(env, "%s the register %s has", ctx, reg_name); |
| 293 | − if (reg->smin_value > S64_MIN) { | |
| 294 | − verbose(env, " smin=%lld", reg->smin_value); | |
| 348 | + if (reg_smin(reg) > S64_MIN) { | |
| 349 | + verbose(env, " smin=%lld", reg_smin(reg)); | |
| 295 | 350 | unknown = false; |
| 296 | 351 | } |
| 297 | − if (reg->smax_value < S64_MAX) { | |
| 298 | − verbose(env, " smax=%lld", reg->smax_value); | |
| 352 | + if (reg_smax(reg) < S64_MAX) { | |
| 353 | + verbose(env, " smax=%lld", reg_smax(reg)); | |
| 299 | 354 | unknown = false; |
| 300 | 355 | } |
| 301 | 356 | if (unknown) |
| @@ -303,7 +358,7 @@ static void verbose_invalid_scalar(struct bpf_verifier_env *env, · 1 unmodified line | ||
| 303 | 358 | verbose(env, " should have been in [%d, %d]\n", range.minval, range.maxval); |
| 304 | 359 | } |
| 305 | 360 | |
| 306 | −static bool reg_not_null(const struct bpf_reg_state *reg) | |
| 361 | +static bool reg_not_null(struct bpf_verifier_env *env, const struct bpf_reg_state *reg) | |
| 307 | 362 | { |
| 308 | 363 | enum bpf_reg_type type; |
| 309 | 364 | |
| @@ -317,7 +372,7 @@ static bool reg_not_null(const struct bpf_reg_state *reg) · 7 unmodified lines | ||
| 317 | 372 | type == PTR_TO_MAP_VALUE || |
| 318 | 373 | type == PTR_TO_MAP_KEY || |
| 319 | 374 | type == PTR_TO_SOCK_COMMON || |
| 320 | − (type == PTR_TO_BTF_ID && is_trusted_reg(reg)) || | |
| 375 | + (type == PTR_TO_BTF_ID && is_trusted_reg(env, reg)) || | |
| 321 | 376 | (type == PTR_TO_MEM && !(reg->type & PTR_UNTRUSTED)) || |
| 322 | 377 | type == CONST_PTR_TO_MAP; |
| 323 | 378 | } |
| @@ -434,11 +489,6 @@ static bool is_ptr_cast_function(enum bpf_func_id func_id) · 110 unmodified lines | ||
| 434 | 489 | func_id == BPF_FUNC_skc_to_tcp_request_sock; |
| 435 | 490 | } |
| 436 | 491 | |
| 437 | −static bool is_dynptr_ref_function(enum bpf_func_id func_id) | |
| 438 | −{ | |
| 439 | − return func_id == BPF_FUNC_dynptr_data; | |
| 440 | −} | |
| 441 | − | |
| 442 | 492 | static bool is_sync_callback_calling_kfunc(u32 btf_id); |
| 443 | 493 | static bool is_async_callback_calling_kfunc(u32 btf_id); |
| 444 | 494 | static bool is_callback_calling_kfunc(u32 btf_id); |
| @@ -497,22 +547,6 @@ bool bpf_is_may_goto_insn(struct bpf_insn *insn) · 52 unmodified lines | ||
| 497 | 547 | return insn->code == (BPF_JMP | BPF_JCOND) && insn->src_reg == BPF_MAY_GOTO; |
| 498 | 548 | } |
| 499 | 549 | |
| 500 | −static bool helper_multiple_ref_obj_use(enum bpf_func_id func_id, | |
| 501 | − const struct bpf_map *map) | |
| 502 | −{ | |
| 503 | − int ref_obj_uses = 0; | |
| 504 | − | |
| 505 | − if (is_ptr_cast_function(func_id)) | |
| 506 | − ref_obj_uses++; | |
| 507 | − if (is_acquire_function(func_id, map)) | |
| 508 | − ref_obj_uses++; | |
| 509 | − if (is_dynptr_ref_function(func_id)) | |
| 510 | − ref_obj_uses++; | |
| 511 | − | |
| 512 | − return ref_obj_uses > 1; | |
| 513 | −} | |
| 514 | − | |
| 515 | − | |
| 516 | 550 | static bool is_spi_bounds_valid(struct bpf_func_state *state, int spi, int nr_slots) |
| 517 | 551 | { |
| 518 | 552 | int allocated_slots = state->allocated_stack / BPF_REG_SIZE; |
| @@ -609,43 +643,44 @@ static enum bpf_type_flag get_dynptr_type_flag(enum bpf_dynptr_type type) · 90 unmodified lines | ||
| 609 | 643 | } |
| 610 | 644 | } |
| 611 | 645 | |
| 612 | −static bool dynptr_type_refcounted(enum bpf_dynptr_type type) | |
| 646 | +static bool dynptr_type_referenced(enum bpf_dynptr_type type) | |
| 613 | 647 | { |
| 614 | 648 | return type == BPF_DYNPTR_TYPE_RINGBUF || type == BPF_DYNPTR_TYPE_FILE; |
| 615 | 649 | } |
| 616 | 650 | |
| 617 | 651 | static void __mark_dynptr_reg(struct bpf_reg_state *reg, |
| 618 | 652 | enum bpf_dynptr_type type, |
| 619 | − bool first_slot, int dynptr_id); | |
| 653 | + bool first_slot, int id, int parent_id); | |
| 620 | 654 | |
| 621 | 655 | |
| 622 | 656 | static void mark_dynptr_stack_regs(struct bpf_verifier_env *env, |
| 623 | 657 | struct bpf_reg_state *sreg1, |
| 624 | 658 | struct bpf_reg_state *sreg2, |
| 625 | − enum bpf_dynptr_type type) | |
| 659 | + enum bpf_dynptr_type type, int parent_id) | |
| 626 | 660 | { |
| 627 | 661 | int id = ++env->id_gen; |
| 628 | 662 | |
| 629 | − __mark_dynptr_reg(sreg1, type, true, id); | |
| 630 | − __mark_dynptr_reg(sreg2, type, false, id); | |
| 663 | + __mark_dynptr_reg(sreg1, type, true, id, parent_id); | |
| 664 | + __mark_dynptr_reg(sreg2, type, false, id, parent_id); | |
| 631 | 665 | } |
| 632 | 666 | |
| 633 | 667 | static void mark_dynptr_cb_reg(struct bpf_verifier_env *env, |
| 634 | 668 | struct bpf_reg_state *reg, |
| 635 | 669 | enum bpf_dynptr_type type) |
| 636 | 670 | { |
| 637 | − __mark_dynptr_reg(reg, type, true, ++env->id_gen); | |
| 671 | + __mark_dynptr_reg(reg, type, true, ++env->id_gen, 0); | |
| 638 | 672 | } |
| 639 | 673 | |
| 640 | 674 | static int destroy_if_dynptr_stack_slot(struct bpf_verifier_env *env, |
| 641 | 675 | struct bpf_func_state *state, int spi); |
| 642 | 676 | |
| 643 | 677 | static int mark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_state *reg, |
| 644 | − enum bpf_arg_type arg_type, int insn_idx, int clone_ref_obj_id) | |
| 678 | + enum bpf_arg_type arg_type, int insn_idx, | |
| 679 | + struct ref_obj_desc *ref_obj, struct bpf_dynptr_desc *dynptr) | |
| 645 | 680 | { |
| 646 | 681 | struct bpf_func_state *state = bpf_func(env, reg); |
| 682 | + int spi, i, err, parent_id = 0; | |
| 647 | 683 | enum bpf_dynptr_type type; |
| 648 | − int spi, i, err; | |
| 649 | 684 | |
| 650 | 685 | spi = dynptr_get_spi(env, reg); |
| 651 | 686 | if (spi < 0) |
| @@ -676,94 +711,69 @@ static int mark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_ · 24 unmodified lines | ||
| 676 | 711 | if (type == BPF_DYNPTR_TYPE_INVALID) |
| 677 | 712 | return -EINVAL; |
| 678 | 713 | |
| 679 | − mark_dynptr_stack_regs(env, &state->stack[spi].spilled_ptr, | |
| 680 | − &state->stack[spi - 1].spilled_ptr, type); | |
| 714 | + if (dynptr->type == BPF_DYNPTR_TYPE_INVALID) { /* dynptr constructors */ | |
| 715 | + err = validate_ref_obj(env, ref_obj); | |
| 716 | + if (err) | |
| 717 | + return err; | |
| 681 | 718 | |
| 682 | − if (dynptr_type_refcounted(type)) { | |
| 683 | − /* The id is used to track proper releasing */ | |
| 684 | − int id; | |
| 719 | + /* Track parent's id if the parent is a referenced object */ | |
| 720 | + parent_id = ref_obj->id; | |
| 685 | 721 | |
| 686 | − if (clone_ref_obj_id) | |
| 687 | − id = clone_ref_obj_id; | |
| 688 | − else | |
| 689 | − id = acquire_reference(env, insn_idx); | |
| 722 | + if (dynptr_type_referenced(type)) { | |
| 723 | + int id; | |
| 690 | 724 | |
| 691 | − if (id < 0) | |
| 692 | − return id; | |
| 725 | + /* | |
| 726 | + * Create an intermediate reference that tracks the referenced | |
| 727 | + * object for the referenced dynptr. Freeing a referenced dynptr | |
| 728 | + * through helpers/kfuncs will invalidate all clones. | |
| 729 | + */ | |
| 730 | + id = acquire_reference(env, insn_idx, parent_id); | |
| 731 | + if (id < 0) | |
| 732 | + return id; | |
| 693 | 733 | |
| 694 | − state->stack[spi].spilled_ptr.ref_obj_id = id; | |
| 695 | − state->stack[spi - 1].spilled_ptr.ref_obj_id = id; | |
| 734 | + parent_id = id; | |
| 735 | + } | |
| 736 | + } else { /* bpf_dynptr_clone() */ | |
| 737 | + parent_id = dynptr->parent_id; | |
| 696 | 738 | } |
| 697 | 739 | |
| 740 | + mark_dynptr_stack_regs(env, &state->stack[spi].spilled_ptr, | |
| 741 | + &state->stack[spi - 1].spilled_ptr, type, parent_id); | |
| 742 | + | |
| 698 | 743 | return 0; |
| 699 | 744 | } |
| 700 | 745 | |
| 701 | −static void invalidate_dynptr(struct bpf_verifier_env *env, struct bpf_func_state *state, int spi) | |
| 746 | +static void invalidate_dynptr(struct bpf_verifier_env *env, struct bpf_stack_state *stack) | |
| 702 | 747 | { |
| 703 | 748 | int i; |
| 704 | 749 | |
| 705 | 750 | for (i = 0; i < BPF_REG_SIZE; i++) { |
| 706 | − state->stack[spi].slot_type[i] = STACK_INVALID; | |
| 707 | − state->stack[spi - 1].slot_type[i] = STACK_INVALID; | |
| 751 | + stack[0].slot_type[i] = STACK_INVALID; | |
| 752 | + stack[1].slot_type[i] = STACK_INVALID; | |
| 708 | 753 | } |
| 709 | 754 | |
| 710 | − bpf_mark_reg_not_init(env, &state->stack[spi].spilled_ptr); | |
| 711 | − bpf_mark_reg_not_init(env, &state->stack[spi - 1].spilled_ptr); | |
| 755 | + bpf_mark_reg_not_init(env, &stack[0].spilled_ptr); | |
| 756 | + bpf_mark_reg_not_init(env, &stack[1].spilled_ptr); | |
| 712 | 757 | } |
| 713 | 758 | |
| 714 | 759 | static int unmark_stack_slots_dynptr(struct bpf_verifier_env *env, struct bpf_reg_state *reg) |
| 715 | 760 | { |
| 716 | 761 | struct bpf_func_state *state = bpf_func(env, reg); |
| 717 | − int spi, ref_obj_id, i; | |
| 762 | + int spi; | |
| 718 | 763 | |
| 719 | − /* | |
| 720 | − * This can only be set for PTR_TO_STACK, as CONST_PTR_TO_DYNPTR cannot | |
| 721 | − * be released by any dynptr helper. Hence, unmark_stack_slots_dynptr | |
| 722 | − * is safe to do directly. | |
| 723 | − */ | |
| 724 | − if (reg->type == CONST_PTR_TO_DYNPTR) { | |
| 725 | − verifier_bug(env, "CONST_PTR_TO_DYNPTR cannot be released"); | |
| 726 | − return -EFAULT; | |
| 727 | − } | |
| 728 | 764 | spi = dynptr_get_spi(env, reg); |
| 729 | 765 | if (spi < 0) |
| 730 | 766 | return spi; |
| 731 | 767 | |
| 732 | − if (!dynptr_type_refcounted(state->stack[spi].spilled_ptr.dynptr.type)) { | |
| 733 | − invalidate_dynptr(env, state, spi); | |
| 734 | − return 0; | |
| 735 | − } | |
| 736 | − | |
| 737 | − ref_obj_id = state->stack[spi].spilled_ptr.ref_obj_id; | |
| 738 | − | |
| 739 | − /* If the dynptr has a ref_obj_id, then we need to invalidate | |
| 740 | − * two things: | |
| 741 | − * | |
| 742 | − * 1) Any dynptrs with a matching ref_obj_id (clones) | |
| 743 | − * 2) Any slices derived from this dynptr. | |
| 768 | + /* | |
| 769 | + * For referenced dynptr, release the parent ref which cascades to | |
| 770 | + * all clones and derived slices. For non-referenced dynptr, only | |
| 771 | + * the dynptr and slices derived from it will be invalidated. | |
| 744 | 772 | */ |
| 745 | − | |
| 746 | − /* Invalidate any slices associated with this dynptr */ | |
| 747 | − WARN_ON_ONCE(release_reference(env, ref_obj_id)); | |
| 748 | − | |
| 749 | − /* Invalidate any dynptr clones */ | |
| 750 | − for (i = 1; i < state->allocated_stack / BPF_REG_SIZE; i++) { | |
| 751 | − if (state->stack[i].spilled_ptr.ref_obj_id != ref_obj_id) | |
| 752 | − continue; | |
| 753 | − | |
| 754 | − /* it should always be the case that if the ref obj id | |
| 755 | − * matches then the stack slot also belongs to a | |
| 756 | − * dynptr | |
| 757 | − */ | |
| 758 | − if (state->stack[i].slot_type[0] != STACK_DYNPTR) { | |
| 759 | − verifier_bug(env, "misconfigured ref_obj_id"); | |
| 760 | − return -EFAULT; | |
| 761 | − } | |
| 762 | − if (state->stack[i].spilled_ptr.dynptr.first_slot) | |
| 763 | − invalidate_dynptr(env, state, i); | |
| 764 | − } | |
| 765 | − | |
| 766 | − return 0; | |
| 773 | + reg = &state->stack[spi].spilled_ptr; | |
| 774 | + return release_reference(env, dynptr_type_referenced(reg->dynptr.type) | |
| 775 | + ? reg->parent_id | |
| 776 | + : reg->id); | |
| 767 | 777 | } |
| 768 | 778 | |
| 769 | 779 | static void __mark_reg_unknown(const struct bpf_verifier_env *env, |
| @@ -777,12 +787,29 @@ static void mark_reg_invalid(const struct bpf_verifier_env *env, struct bpf_reg_ · 7 unmodified lines | ||
| 777 | 787 | __mark_reg_unknown(env, reg); |
| 778 | 788 | } |
| 779 | 789 | |
| 790 | +static int dynptr_ref_cnt(struct bpf_verifier_env *env, int v_parent_id) | |
| 791 | +{ | |
| 792 | + struct bpf_stack_state *stack; | |
| 793 | + struct bpf_func_state *state; | |
| 794 | + struct bpf_reg_state *reg; | |
| 795 | + int ref_cnt = 0; | |
| 796 | + | |
| 797 | + bpf_for_each_reg_in_vstate_mask(env->cur_state, state, reg, stack, 1 << STACK_DYNPTR, ({ | |
| 798 | + if (!stack || stack->slot_type[0] != STACK_DYNPTR) | |
| 799 | + continue; | |
| 800 | + if (!stack->spilled_ptr.dynptr.first_slot) | |
| 801 | + continue; | |
| 802 | + if (stack->spilled_ptr.parent_id == v_parent_id) | |
| 803 | + ref_cnt++; | |
| 804 | + })); | |
| 805 | + | |
| 806 | + return ref_cnt; | |
| 807 | +} | |
| 808 | + | |
| 780 | 809 | static int destroy_if_dynptr_stack_slot(struct bpf_verifier_env *env, |
| 781 | 810 | struct bpf_func_state *state, int spi) |
| 782 | 811 | { |
| 783 | − struct bpf_func_state *fstate; | |
| 784 | − struct bpf_reg_state *dreg; | |
| 785 | − int i, dynptr_id; | |
| 812 | + int err = 0; | |
| 786 | 813 | |
| 787 | 814 | /* We always ensure that STACK_DYNPTR is never set partially, |
| 788 | 815 | * hence just checking for slot_type[0] is enough. This is |
| @@ -796,56 +823,25 @@ static int destroy_if_dynptr_stack_slot(struct bpf_verifier_env *env, · 7 unmodified lines | ||
| 796 | 823 | if (!state->stack[spi].spilled_ptr.dynptr.first_slot) |
| 797 | 824 | spi = spi + 1; |
| 798 | 825 | |
| 799 | − if (dynptr_type_refcounted(state->stack[spi].spilled_ptr.dynptr.type)) { | |
| 800 | − int ref_obj_id = state->stack[spi].spilled_ptr.ref_obj_id; | |
| 801 | − int ref_cnt = 0; | |
| 802 | − | |
| 803 | − /* | |
| 804 | − * A referenced dynptr can be overwritten only if there is at | |
| 805 | − * least one other dynptr sharing the same ref_obj_id, | |
| 806 | − * ensuring the reference can still be properly released. | |
| 807 | − */ | |
| 808 | − for (i = 0; i < state->allocated_stack / BPF_REG_SIZE; i++) { | |
| 809 | − if (state->stack[i].slot_type[0] != STACK_DYNPTR) | |
| 810 | − continue; | |
| 811 | − if (!state->stack[i].spilled_ptr.dynptr.first_slot) | |
| 812 | − continue; | |
| 813 | − if (state->stack[i].spilled_ptr.ref_obj_id == ref_obj_id) | |
| 814 | − ref_cnt++; | |
| 815 | − } | |
| 816 | − | |
| 817 | − if (ref_cnt <= 1) { | |
| 818 | − verbose(env, "cannot overwrite referenced dynptr\n"); | |
| 819 | − return -EINVAL; | |
| 820 | − } | |
| 826 | + /* | |
| 827 | + * A referenced dynptr can be overwritten only if there is at | |
| 828 | + * least one other dynptr sharing the same virtual ref parent, | |
| 829 | + * ensuring the reference can still be properly released. | |
| 830 | + */ | |
| 831 | + if (dynptr_type_referenced(state->stack[spi].spilled_ptr.dynptr.type) && | |
| 832 | + dynptr_ref_cnt(env, state->stack[spi].spilled_ptr.parent_id) <= 1) { | |
| 833 | + verbose(env, "cannot overwrite referenced dynptr\n"); | |
| 834 | + return -EINVAL; | |
| 821 | 835 | } |
| 822 | 836 | |
| 823 | − mark_stack_slot_scratched(env, spi); | |
| 824 | − mark_stack_slot_scratched(env, spi - 1); | |
| 825 | − | |
| 826 | − /* Writing partially to one dynptr stack slot destroys both. */ | |
| 827 | − for (i = 0; i < BPF_REG_SIZE; i++) { | |
| 828 | − state->stack[spi].slot_type[i] = STACK_INVALID; | |
| 829 | − state->stack[spi - 1].slot_type[i] = STACK_INVALID; | |
| 837 | + /* Invalidate the dynptr and any derived slices */ | |
| 838 | + err = release_reference(env, state->stack[spi].spilled_ptr.id); | |
| 839 | + if (!err) { | |
| 840 | + mark_stack_slot_scratched(env, spi); | |
| 841 | + mark_stack_slot_scratched(env, spi - 1); | |
| 830 | 842 | } |
| 831 | 843 | |
| 832 | − dynptr_id = state->stack[spi].spilled_ptr.id; | |
| 833 | − /* Invalidate any slices associated with this dynptr */ | |
| 834 | − bpf_for_each_reg_in_vstate(env->cur_state, fstate, dreg, ({ | |
| 835 | − /* Dynptr slices are only PTR_TO_MEM_OR_NULL and PTR_TO_MEM */ | |
| 836 | − if (dreg->type != (PTR_TO_MEM | PTR_MAYBE_NULL) && dreg->type != PTR_TO_MEM) | |
| 837 | − continue; | |
| 838 | − if (dreg->dynptr_id == dynptr_id) | |
| 839 | − mark_reg_invalid(env, dreg); | |
| 840 | − })); | |
| 841 | − | |
| 842 | − /* Do not release reference state, we are destroying dynptr on stack, | |
| 843 | − * not using some helper to release it. Just reset register. | |
| 844 | − */ | |
| 845 | − bpf_mark_reg_not_init(env, &state->stack[spi].spilled_ptr); | |
| 846 | − bpf_mark_reg_not_init(env, &state->stack[spi - 1].spilled_ptr); | |
| 847 | − | |
| 848 | − return 0; | |
| 844 | + return err; | |
| 849 | 845 | } |
| 850 | 846 | |
| 851 | 847 | static bool is_dynptr_reg_valid_uninit(struct bpf_verifier_env *env, struct bpf_reg_state *reg) |
| @@ -945,7 +941,7 @@ static int mark_stack_slots_iter(struct bpf_verifier_env *env, · 93 unmodified lines | ||
| 945 | 941 | if (spi < 0) |
| 946 | 942 | return spi; |
| 947 | 943 | |
| 948 | − id = acquire_reference(env, insn_idx); | |
| 944 | + id = acquire_reference(env, insn_idx, 0); | |
| 949 | 945 | if (id < 0) |
| 950 | 946 | return id; |
| 951 | 947 | |
| @@ -961,7 +957,7 @@ static int mark_stack_slots_iter(struct bpf_verifier_env *env, · 9 unmodified lines | ||
| 961 | 957 | else |
| 962 | 958 | st->type |= PTR_UNTRUSTED; |
| 963 | 959 | } |
| 964 | − st->ref_obj_id = i == 0 ? id : 0; | |
| 960 | + st->id = i == 0 ? id : 0; | |
| 965 | 961 | st->iter.btf = btf; |
| 966 | 962 | st->iter.btf_id = btf_id; |
| 967 | 963 | st->iter.state = BPF_ITER_STATE_ACTIVE; |
| @@ -991,7 +987,7 @@ static int unmark_stack_slots_iter(struct bpf_verifier_env *env, · 23 unmodified lines | ||
| 991 | 987 | struct bpf_reg_state *st = &slot->spilled_ptr; |
| 992 | 988 | |
| 993 | 989 | if (i == 0) |
| 994 | − WARN_ON_ONCE(release_reference(env, st->ref_obj_id)); | |
| 990 | + WARN_ON_ONCE(release_reference(env, st->id)); | |
| 995 | 991 | |
| 996 | 992 | bpf_mark_reg_not_init(env, st); |
| 997 | 993 | |
| @@ -1047,10 +1043,10 @@ static int is_iter_reg_valid_init(struct bpf_verifier_env *env, struct bpf_reg_s · 49 unmodified lines | ||
| 1047 | 1043 | |
| 1048 | 1044 | if (st->type & PTR_UNTRUSTED) |
| 1049 | 1045 | return -EPROTO; |
| 1050 | − /* only main (first) slot has ref_obj_id set */ | |
| 1051 | − if (i == 0 && !st->ref_obj_id) | |
| 1046 | + /* only main (first) slot has id set */ | |
| 1047 | + if (i == 0 && !st->id) | |
| 1052 | 1048 | return -EINVAL; |
| 1053 | − if (i != 0 && st->ref_obj_id) | |
| 1049 | + if (i != 0 && st->id) | |
| 1054 | 1050 | return -EINVAL; |
| 1055 | 1051 | if (st->iter.btf != btf || st->iter.btf_id != btf_id) |
| 1056 | 1052 | return -EINVAL; |
| @@ -1089,7 +1085,7 @@ static int mark_stack_slot_irq_flag(struct bpf_verifier_env *env, · 32 unmodified lines | ||
| 1089 | 1085 | |
| 1090 | 1086 | __mark_reg_known_zero(st); |
| 1091 | 1087 | st->type = PTR_TO_STACK; /* we don't have dedicated reg type */ |
| 1092 | − st->ref_obj_id = id; | |
| 1088 | + st->id = id; | |
| 1093 | 1089 | st->irq.kfunc_class = kfunc_class; |
| 1094 | 1090 | |
| 1095 | 1091 | for (i = 0; i < BPF_REG_SIZE; i++) |
| @@ -1123,7 +1119,7 @@ static int unmark_stack_slot_irq_flag(struct bpf_verifier_env *env, struct bpf_r · 27 unmodified lines | ||
| 1123 | 1119 | return -EINVAL; |
| 1124 | 1120 | } |
| 1125 | 1121 | |
| 1126 | − err = release_irq_state(env->cur_state, st->ref_obj_id); | |
| 1122 | + err = release_irq_state(env->cur_state, st->id); | |
| 1127 | 1123 | WARN_ON_ONCE(err && err != -EACCES); |
| 1128 | 1124 | if (err) { |
| 1129 | 1125 | int insn_idx = 0; |
| @@ -1187,7 +1183,7 @@ static int is_irq_flag_reg_valid_init(struct bpf_verifier_env *env, struct bpf_r · 57 unmodified lines | ||
| 1187 | 1183 | slot = &state->stack[spi]; |
| 1188 | 1184 | st = &slot->spilled_ptr; |
| 1189 | 1185 | |
| 1190 | − if (!st->ref_obj_id) | |
| 1186 | + if (!st->id) | |
| 1191 | 1187 | return -EINVAL; |
| 1192 | 1188 | |
| 1193 | 1189 | for (i = 0; i < BPF_REG_SIZE; i++) |
| @@ -1339,6 +1335,18 @@ static int copy_stack_state(struct bpf_func_state *dst, const struct bpf_func_st · 145 unmodified lines | ||
| 1339 | 1335 | return -ENOMEM; |
| 1340 | 1336 | |
| 1341 | 1337 | dst->allocated_stack = src->allocated_stack; |
| 1338 | + | |
| 1339 | + /* copy stack args state */ | |
| 1340 | + n = src->out_stack_arg_cnt; | |
| 1341 | + if (n) { | |
| 1342 | + dst->stack_arg_regs = copy_array(dst->stack_arg_regs, src->stack_arg_regs, n, | |
| 1343 | + sizeof(struct bpf_reg_state), | |
| 1344 | + GFP_KERNEL_ACCOUNT); | |
| 1345 | + if (!dst->stack_arg_regs) | |
| 1346 | + return -ENOMEM; | |
| 1347 | + } | |
| 1348 | + | |
| 1349 | + dst->out_stack_arg_cnt = src->out_stack_arg_cnt; | |
| 1342 | 1350 | return 0; |
| 1343 | 1351 | } |
| 1344 | 1352 | |
| @@ -1380,6 +1388,23 @@ static int grow_stack_state(struct bpf_verifier_env *env, struct bpf_func_state · 35 unmodified lines | ||
| 1380 | 1388 | return 0; |
| 1381 | 1389 | } |
| 1382 | 1390 | |
| 1391 | +static int grow_stack_arg_slots(struct bpf_verifier_env *env, | |
| 1392 | + struct bpf_func_state *state, int cnt) | |
| 1393 | +{ | |
| 1394 | + size_t old_n = state->out_stack_arg_cnt; | |
| 1395 | + | |
| 1396 | + if (old_n >= cnt) | |
| 1397 | + return 0; | |
| 1398 | + | |
| 1399 | + state->stack_arg_regs = realloc_array(state->stack_arg_regs, old_n, cnt, | |
| 1400 | + sizeof(struct bpf_reg_state)); | |
| 1401 | + if (!state->stack_arg_regs) | |
| 1402 | + return -ENOMEM; | |
| 1403 | + | |
| 1404 | + state->out_stack_arg_cnt = cnt; | |
| 1405 | + return 0; | |
| 1406 | +} | |
| 1407 | + | |
| 1383 | 1408 | /* Acquire a pointer id from the env and update the state->refs to include |
| 1384 | 1409 | * this new pointer reference. |
| 1385 | 1410 | * On success, returns a valid pointer id to associate with the register |
| @@ -1399,7 +1424,7 @@ static struct bpf_reference_state *acquire_reference_state(struct bpf_verifier_e · 13 unmodified lines | ||
| 1399 | 1424 | return &state->refs[new_ofs]; |
| 1400 | 1425 | } |
| 1401 | 1426 | |
| 1402 | −static int acquire_reference(struct bpf_verifier_env *env, int insn_idx) | |
| 1427 | +static int acquire_reference(struct bpf_verifier_env *env, int insn_idx, int parent_id) | |
| 1403 | 1428 | { |
| 1404 | 1429 | struct bpf_reference_state *s; |
| 1405 | 1430 | |
| @@ -1408,6 +1433,7 @@ static int acquire_reference(struct bpf_verifier_env *env, int insn_idx) · 2 unmodified lines | ||
| 1408 | 1433 | return -ENOMEM; |
| 1409 | 1434 | s->type = REF_TYPE_PTR; |
| 1410 | 1435 | s->id = ++env->id_gen; |
| 1436 | + s->parent_id = parent_id; | |
| 1411 | 1437 | return s->id; |
| 1412 | 1438 | } |
| 1413 | 1439 | |
| @@ -1464,17 +1490,25 @@ static void release_reference_state(struct bpf_verifier_state *state, int idx) · 50 unmodified lines | ||
| 1464 | 1490 | return; |
| 1465 | 1491 | } |
| 1466 | 1492 | |
| 1467 | −static bool find_reference_state(struct bpf_verifier_state *state, int ptr_id) | |
| 1493 | +static bool find_reference_state(struct bpf_verifier_state *state, int id) | |
| 1468 | 1494 | { |
| 1469 | 1495 | int i; |
| 1470 | 1496 | |
| 1471 | − for (i = 0; i < state->acquired_refs; i++) | |
| 1472 | − if (state->refs[i].id == ptr_id) | |
| 1497 | + for (i = 0; i < state->acquired_refs; i++) { | |
| 1498 | + if (state->refs[i].type != REF_TYPE_PTR) | |
| 1499 | + continue; | |
| 1500 | + if (state->refs[i].id == id) | |
| 1473 | 1501 | return true; |
| 1502 | + } | |
| 1474 | 1503 | |
| 1475 | 1504 | return false; |
| 1476 | 1505 | } |
| 1477 | 1506 | |
| 1507 | +static bool reg_is_referenced(struct bpf_verifier_env *env, const struct bpf_reg_state *reg) | |
| 1508 | +{ | |
| 1509 | + return find_reference_state(env->cur_state, reg->id); | |
| 1510 | +} | |
| 1511 | + | |
| 1478 | 1512 | static int release_lock_state(struct bpf_verifier_state *state, int type, int id, void *ptr) |
| 1479 | 1513 | { |
| 1480 | 1514 | void *prev_ptr = NULL; |
| @@ -1542,6 +1576,7 @@ static void free_func_state(struct bpf_func_state *state) · 61 unmodified lines | ||
| 1542 | 1576 | { |
| 1543 | 1577 | if (!state) |
| 1544 | 1578 | return; |
| 1579 | + kfree(state->stack_arg_regs); | |
| 1545 | 1580 | kfree(state->stack); |
| 1546 | 1581 | kfree(state); |
| 1547 | 1582 | } |
| @@ -1750,6 +1785,22 @@ static struct bpf_verifier_state *push_stack(struct bpf_verifier_env *env, · 202 unmodified lines | ||
| 1750 | 1785 | return &elem->st; |
| 1751 | 1786 | } |
| 1752 | 1787 | |
| 1788 | +static const char *reg_arg_name(struct bpf_verifier_env *env, argno_t argno) | |
| 1789 | +{ | |
| 1790 | + char *buf = env->tmp_arg_name; | |
| 1791 | + int len = sizeof(env->tmp_arg_name); | |
| 1792 | + int arg, regno = reg_from_argno(argno); | |
| 1793 | + | |
| 1794 | + if (regno >= 0) { | |
| 1795 | + snprintf(buf, len, "R%d", regno); | |
| 1796 | + } else { | |
| 1797 | + arg = arg_from_argno(argno); | |
| 1798 | + snprintf(buf, len, "*(R11-%u)", (arg - MAX_BPF_FUNC_REG_ARGS) * BPF_REG_SIZE); | |
| 1799 | + } | |
| 1800 | + | |
| 1801 | + return buf; | |
| 1802 | +} | |
| 1803 | + | |
| 1753 | 1804 | static const int caller_saved[CALLER_SAVED_REGS] = { |
| 1754 | 1805 | BPF_REG_0, BPF_REG_1, BPF_REG_2, BPF_REG_3, BPF_REG_4, BPF_REG_5 |
| 1755 | 1806 | }; |
| @@ -1758,15 +1809,8 @@ static const int caller_saved[CALLER_SAVED_REGS] = { · 2 unmodified lines | ||
| 1758 | 1809 | static void ___mark_reg_known(struct bpf_reg_state *reg, u64 imm) |
| 1759 | 1810 | { |
| 1760 | 1811 | reg->var_off = tnum_const(imm); |
| 1761 | − reg->smin_value = (s64)imm; | |
| 1762 | − reg->smax_value = (s64)imm; | |
| 1763 | − reg->umin_value = imm; | |
| 1764 | − reg->umax_value = imm; | |
| 1765 | − | |
| 1766 | − reg->s32_min_value = (s32)imm; | |
| 1767 | − reg->s32_max_value = (s32)imm; | |
| 1768 | − reg->u32_min_value = (u32)imm; | |
| 1769 | − reg->u32_max_value = (u32)imm; | |
| 1812 | + reg->r64 = cnum64_from_urange(imm, imm); | |
| 1813 | + reg->r32 = cnum32_from_urange((u32)imm, (u32)imm); | |
| 1770 | 1814 | } |
| 1771 | 1815 | |
| 1772 | 1816 | /* Mark the unknown part of a register (variable offset or scalar value) as |
| @@ -1778,17 +1822,14 @@ static void __mark_reg_known(struct bpf_reg_state *reg, u64 imm) · 5 unmodified lines | ||
| 1778 | 1822 | memset(((u8 *)reg) + sizeof(reg->type), 0, |
| 1779 | 1823 | offsetof(struct bpf_reg_state, var_off) - sizeof(reg->type)); |
| 1780 | 1824 | reg->id = 0; |
| 1781 | − reg->ref_obj_id = 0; | |
| 1825 | + reg->parent_id = 0; | |
| 1782 | 1826 | ___mark_reg_known(reg, imm); |
| 1783 | 1827 | } |
| 1784 | 1828 | |
| 1785 | 1829 | static void __mark_reg32_known(struct bpf_reg_state *reg, u64 imm) |
| 1786 | 1830 | { |
| 1787 | 1831 | reg->var_off = tnum_const_subreg(reg->var_off, imm); |
| 1788 | − reg->s32_min_value = (s32)imm; | |
| 1789 | − reg->s32_max_value = (s32)imm; | |
| 1790 | − reg->u32_min_value = (u32)imm; | |
| 1791 | − reg->u32_max_value = (u32)imm; | |
| 1832 | + reg->r32 = cnum32_from_urange((u32)imm, (u32)imm); | |
| 1792 | 1833 | } |
| 1793 | 1834 | |
| 1794 | 1835 | /* Mark the 'variable offset' part of a register as zero. This should be |
| @@ -1816,7 +1857,7 @@ static void mark_reg_known_zero(struct bpf_verifier_env *env, · 21 unmodified lines | ||
| 1816 | 1857 | } |
| 1817 | 1858 | |
| 1818 | 1859 | static void __mark_dynptr_reg(struct bpf_reg_state *reg, enum bpf_dynptr_type type, |
| 1819 | − bool first_slot, int dynptr_id) | |
| 1860 | + bool first_slot, int id, int parent_id) | |
| 1820 | 1861 | { |
| 1821 | 1862 | /* reg->type has no meaning for STACK_DYNPTR, but when we set reg for |
| 1822 | 1863 | * callback arguments, it does need to be CONST_PTR_TO_DYNPTR, so simply |
| @@ -1825,7 +1866,8 @@ static void __mark_dynptr_reg(struct bpf_reg_state *reg, enum bpf_dynptr_type ty · 2 unmodified lines | ||
| 1825 | 1866 | __mark_reg_known_zero(reg); |
| 1826 | 1867 | reg->type = CONST_PTR_TO_DYNPTR; |
| 1827 | 1868 | /* Give each dynptr a unique id to uniquely associate slices to it. */ |
| 1828 | − reg->id = dynptr_id; | |
| 1869 | + reg->id = id; | |
| 1870 | + reg->parent_id = parent_id; | |
| 1829 | 1871 | reg->dynptr.type = type; |
| 1830 | 1872 | reg->dynptr.first_slot = first_slot; |
| 1831 | 1873 | } |
| @@ -1899,34 +1941,21 @@ static bool reg_is_init_pkt_pointer(const struct bpf_reg_state *reg, · 67 unmodified lines | ||
| 1899 | 1941 | tnum_equals_const(reg->var_off, 0); |
| 1900 | 1942 | } |
| 1901 | 1943 | |
| 1902 | −/* Reset the min/max bounds of a register */ | |
| 1903 | −static void __mark_reg_unbounded(struct bpf_reg_state *reg) | |
| 1944 | +static void __mark_reg32_unbounded(struct bpf_reg_state *reg) | |
| 1904 | 1945 | { |
| 1905 | − reg->smin_value = S64_MIN; | |
| 1906 | − reg->smax_value = S64_MAX; | |
| 1907 | − reg->umin_value = 0; | |
| 1908 | − reg->umax_value = U64_MAX; | |
| 1909 | − | |
| 1910 | − reg->s32_min_value = S32_MIN; | |
| 1911 | − reg->s32_max_value = S32_MAX; | |
| 1912 | − reg->u32_min_value = 0; | |
| 1913 | − reg->u32_max_value = U32_MAX; | |
| 1946 | + reg->r32 = CNUM32_UNBOUNDED; | |
| 1914 | 1947 | } |
| 1915 | 1948 | |
| 1916 | 1949 | static void __mark_reg64_unbounded(struct bpf_reg_state *reg) |
| 1917 | 1950 | { |
| 1918 | − reg->smin_value = S64_MIN; | |
| 1919 | − reg->smax_value = S64_MAX; | |
| 1920 | − reg->umin_value = 0; | |
| 1921 | − reg->umax_value = U64_MAX; | |
| 1951 | + reg->r64 = CNUM64_UNBOUNDED; | |
| 1922 | 1952 | } |
| 1923 | 1953 | |
| 1924 | −static void __mark_reg32_unbounded(struct bpf_reg_state *reg) | |
| 1954 | +/* Reset the min/max bounds of a register */ | |
| 1955 | +static void __mark_reg_unbounded(struct bpf_reg_state *reg) | |
| 1925 | 1956 | { |
| 1926 | − reg->s32_min_value = S32_MIN; | |
| 1927 | − reg->s32_max_value = S32_MAX; | |
| 1928 | − reg->u32_min_value = 0; | |
| 1929 | − reg->u32_max_value = U32_MAX; | |
| 1957 | + __mark_reg64_unbounded(reg); | |
| 1958 | + __mark_reg32_unbounded(reg); | |
| 1930 | 1959 | } |
| 1931 | 1960 | |
| 1932 | 1961 | static void reset_reg64_and_tnum(struct bpf_reg_state *reg) |
| @@ -1941,19 +1970,32 @@ static void reset_reg32_and_tnum(struct bpf_reg_state *reg) · 8 unmodified lines | ||
| 1941 | 1970 | reg->var_off = tnum_unknown; |
| 1942 | 1971 | } |
| 1943 | 1972 | |
| 1944 | −static void __update_reg32_bounds(struct bpf_reg_state *reg) | |
| 1973 | +static struct cnum32 cnum32_from_tnum(struct tnum tnum) | |
| 1945 | 1974 | { |
| 1946 | − struct tnum var32_off = tnum_subreg(reg->var_off); | |
| 1975 | + tnum = tnum_subreg(tnum); | |
| 1976 | + if ((tnum.mask & S32_MIN) || (tnum.value & S32_MIN)) | |
| 1977 | + /* min signed is max(sign bit) | min(other bits) */ | |
| 1978 | + /* max signed is min(sign bit) | max(other bits) */ | |
| 1979 | + return cnum32_from_srange(tnum.value | (tnum.mask & S32_MIN), | |
| 1980 | + tnum.value | (tnum.mask & S32_MAX)); | |
| 1981 | + else | |
| 1982 | + return cnum32_from_urange(tnum.value, (tnum.value | tnum.mask)); | |
| 1983 | +} | |
| 1947 | 1984 | |
| 1948 | − /* min signed is max(sign bit) | min(other bits) */ | |
| 1949 | − reg->s32_min_value = max_t(s32, reg->s32_min_value, | |
| 1950 | − var32_off.value | (var32_off.mask & S32_MIN)); | |
| 1951 | − /* max signed is min(sign bit) | max(other bits) */ | |
| 1952 | − reg->s32_max_value = min_t(s32, reg->s32_max_value, | |
| 1953 | − var32_off.value | (var32_off.mask & S32_MAX)); | |
| 1954 | − reg->u32_min_value = max_t(u32, reg->u32_min_value, (u32)var32_off.value); | |
| 1955 | − reg->u32_max_value = min(reg->u32_max_value, | |
| 1956 | − (u32)(var32_off.value | var32_off.mask)); | |
| 1985 | +static struct cnum64 cnum64_from_tnum(struct tnum tnum) | |
| 1986 | +{ | |
| 1987 | + if ((tnum.mask & S64_MIN) || (tnum.value & S64_MIN)) | |
| 1988 | + /* min signed is max(sign bit) | min(other bits) */ | |
| 1989 | + /* max signed is min(sign bit) | max(other bits) */ | |
| 1990 | + return cnum64_from_srange(tnum.value | (tnum.mask & S64_MIN), | |
| 1991 | + tnum.value | (tnum.mask & S64_MAX)); | |
| 1992 | + else | |
| 1993 | + return cnum64_from_urange(tnum.value, (tnum.value | tnum.mask)); | |
| 1994 | +} | |
| 1995 | + | |
| 1996 | +static void __update_reg32_bounds(struct bpf_reg_state *reg) | |
| 1997 | +{ | |
| 1998 | + cnum32_intersect_with(®->r32, cnum32_from_tnum(reg->var_off)); | |
| 1957 | 1999 | } |
| 1958 | 2000 | |
| 1959 | 2001 | static void __update_reg64_bounds(struct bpf_reg_state *reg) |
| @@ -1961,26 +2003,18 @@ static void __update_reg64_bounds(struct bpf_reg_state *reg) · 1 unmodified line | ||
| 1961 | 2003 | u64 tnum_next, tmax; |
| 1962 | 2004 | bool umin_in_tnum; |
| 1963 | 2005 | |
| 1964 | − /* min signed is max(sign bit) | min(other bits) */ | |
| 1965 | − reg->smin_value = max_t(s64, reg->smin_value, | |
| 1966 | − reg->var_off.value | (reg->var_off.mask & S64_MIN)); | |
| 1967 | − /* max signed is min(sign bit) | max(other bits) */ | |
| 1968 | − reg->smax_value = min_t(s64, reg->smax_value, | |
| 1969 | − reg->var_off.value | (reg->var_off.mask & S64_MAX)); | |
| 1970 | − reg->umin_value = max(reg->umin_value, reg->var_off.value); | |
| 1971 | − reg->umax_value = min(reg->umax_value, | |
| 1972 | − reg->var_off.value | reg->var_off.mask); | |
| 2006 | + cnum64_intersect_with(®->r64, cnum64_from_tnum(reg->var_off)); | |
| 1973 | 2007 | |
| 1974 | 2008 | /* Check if u64 and tnum overlap in a single value */ |
| 1975 | − tnum_next = tnum_step(reg->var_off, reg->umin_value); | |
| 1976 | − umin_in_tnum = (reg->umin_value & ~reg->var_off.mask) == reg->var_off.value; | |
| 2009 | + tnum_next = tnum_step(reg->var_off, reg_umin(reg)); | |
| 2010 | + umin_in_tnum = (reg_umin(reg) & ~reg->var_off.mask) == reg->var_off.value; | |
| 1977 | 2011 | tmax = reg->var_off.value | reg->var_off.mask; |
| 1978 | − if (umin_in_tnum && tnum_next > reg->umax_value) { | |
| 2012 | + if (umin_in_tnum && tnum_next > reg_umax(reg)) { | |
| 1979 | 2013 | /* The u64 range and the tnum only overlap in umin. |
| 1980 | 2014 | * u64: ---[xxxxxx]----- |
| 1981 | 2015 | * tnum: --xx----------x- |
| 1982 | 2016 | */ |
| 1983 | − ___mark_reg_known(reg, reg->umin_value); | |
| 2017 | + ___mark_reg_known(reg, reg_umin(reg)); | |
| 1984 | 2018 | } else if (!umin_in_tnum && tnum_next == tmax) { |
| 1985 | 2019 | /* The u64 range and the tnum only overlap in the maximum value |
| 1986 | 2020 | * represented by the tnum, called tmax. |
| @@ -1988,8 +2022,8 @@ static void __update_reg64_bounds(struct bpf_reg_state *reg) · 1 unmodified line | ||
| 1988 | 2022 | * tnum: xx-----x-------- |
| 1989 | 2023 | */ |
| 1990 | 2024 | ___mark_reg_known(reg, tmax); |
| 1991 | − } else if (!umin_in_tnum && tnum_next <= reg->umax_value && | |
| 1992 | − tnum_step(reg->var_off, tnum_next) > reg->umax_value) { | |
| 2025 | + } else if (!umin_in_tnum && tnum_next <= reg_umax(reg) && | |
| 2026 | + tnum_step(reg->var_off, tnum_next) > reg_umax(reg)) { | |
| 1993 | 2027 | /* The u64 range and the tnum only overlap in between umin |
| 1994 | 2028 | * (excluded) and umax. |
| 1995 | 2029 | * u64: ---[xxxxxx]----- |
| @@ -2005,329 +2039,19 @@ static void __update_reg_bounds(struct bpf_reg_state *reg) · 9 unmodified lines | ||
| 2005 | 2039 | __update_reg64_bounds(reg); |
| 2006 | 2040 | } |
| 2007 | 2041 | |
| 2008 | −/* Uses signed min/max values to inform unsigned, and vice-versa */ | |
| 2009 | 2042 | static void deduce_bounds_32_from_64(struct bpf_reg_state *reg) |
| 2010 | 2043 | { |
| 2011 | − /* If upper 32 bits of u64/s64 range don't change, we can use lower 32 | |
| 2012 | − * bits to improve our u32/s32 boundaries. | |
| 2013 | − * | |
| 2014 | − * E.g., the case where we have upper 32 bits as zero ([10, 20] in | |
| 2015 | − * u64) is pretty trivial, it's obvious that in u32 we'll also have | |
| 2016 | − * [10, 20] range. But this property holds for any 64-bit range as | |
| 2017 | − * long as upper 32 bits in that entire range of values stay the same. | |
| 2018 | − * | |
| 2019 | − * E.g., u64 range [0x10000000A, 0x10000000F] ([4294967306, 4294967311] | |
| 2020 | − * in decimal) has the same upper 32 bits throughout all the values in | |
| 2021 | − * that range. As such, lower 32 bits form a valid [0xA, 0xF] ([10, 15]) | |
| 2022 | − * range. | |
| 2023 | − * | |
| 2024 | − * Note also, that [0xA, 0xF] is a valid range both in u32 and in s32, | |
| 2025 | − * following the rules outlined below about u64/s64 correspondence | |
| 2026 | − * (which equally applies to u32 vs s32 correspondence). In general it | |
| 2027 | − * depends on actual hexadecimal values of 32-bit range. They can form | |
| 2028 | − * only valid u32, or only valid s32 ranges in some cases. | |
| 2029 | − * | |
| 2030 | − * So we use all these insights to derive bounds for subregisters here. | |
| 2031 | − */ | |
| 2032 | − if ((reg->umin_value >> 32) == (reg->umax_value >> 32)) { | |
| 2033 | − /* u64 to u32 casting preserves validity of low 32 bits as | |
| 2034 | − * a range, if upper 32 bits are the same | |
| 2035 | − */ | |
| 2036 | − reg->u32_min_value = max_t(u32, reg->u32_min_value, (u32)reg->umin_value); | |
| 2037 | − reg->u32_max_value = min_t(u32, reg->u32_max_value, (u32)reg->umax_value); | |
| 2038 | − | |
| 2039 | − if ((s32)reg->umin_value <= (s32)reg->umax_value) { | |
| 2040 | − reg->s32_min_value = max_t(s32, reg->s32_min_value, (s32)reg->umin_value); | |
| 2041 | − reg->s32_max_value = min_t(s32, reg->s32_max_value, (s32)reg->umax_value); | |
| 2042 | − } | |
| 2043 | − } | |
| 2044 | − if ((reg->smin_value >> 32) == (reg->smax_value >> 32)) { | |
| 2045 | − /* low 32 bits should form a proper u32 range */ | |
| 2046 | − if ((u32)reg->smin_value <= (u32)reg->smax_value) { | |
| 2047 | − reg->u32_min_value = max_t(u32, reg->u32_min_value, (u32)reg->smin_value); | |
| 2048 | − reg->u32_max_value = min_t(u32, reg->u32_max_value, (u32)reg->smax_value); | |
| 2049 | − } | |
| 2050 | − /* low 32 bits should form a proper s32 range */ | |
| 2051 | − if ((s32)reg->smin_value <= (s32)reg->smax_value) { | |
| 2052 | − reg->s32_min_value = max_t(s32, reg->s32_min_value, (s32)reg->smin_value); | |
| 2053 | − reg->s32_max_value = min_t(s32, reg->s32_max_value, (s32)reg->smax_value); | |
| 2054 | − } | |
| 2055 | − } | |
| 2056 | − /* Special case where upper bits form a small sequence of two | |
| 2057 | − * sequential numbers (in 32-bit unsigned space, so 0xffffffff to | |
| 2058 | − * 0x00000000 is also valid), while lower bits form a proper s32 range | |
| 2059 | − * going from negative numbers to positive numbers. E.g., let's say we | |
| 2060 | − * have s64 range [-1, 1] ([0xffffffffffffffff, 0x0000000000000001]). | |
| 2061 | − * Possible s64 values are {-1, 0, 1} ({0xffffffffffffffff, | |
| 2062 | − * 0x0000000000000000, 0x00000000000001}). Ignoring upper 32 bits, | |
| 2063 | − * we still get a valid s32 range [-1, 1] ([0xffffffff, 0x00000001]). | |
| 2064 | − * Note that it doesn't have to be 0xffffffff going to 0x00000000 in | |
| 2065 | − * upper 32 bits. As a random example, s64 range | |
| 2066 | − * [0xfffffff0fffffff0; 0xfffffff100000010], forms a valid s32 range | |
| 2067 | − * [-16, 16] ([0xfffffff0; 0x00000010]) in its 32 bit subregister. | |
| 2068 | − */ | |
| 2069 | − if ((u32)(reg->umin_value >> 32) + 1 == (u32)(reg->umax_value >> 32) && | |
| 2070 | − (s32)reg->umin_value < 0 && (s32)reg->umax_value >= 0) { | |
| 2071 | − reg->s32_min_value = max_t(s32, reg->s32_min_value, (s32)reg->umin_value); | |
| 2072 | − reg->s32_max_value = min_t(s32, reg->s32_max_value, (s32)reg->umax_value); | |
| 2073 | − } | |
| 2074 | − if ((u32)(reg->smin_value >> 32) + 1 == (u32)(reg->smax_value >> 32) && | |
| 2075 | − (s32)reg->smin_value < 0 && (s32)reg->smax_value >= 0) { | |
| 2076 | − reg->s32_min_value = max_t(s32, reg->s32_min_value, (s32)reg->smin_value); | |
| 2077 | − reg->s32_max_value = min_t(s32, reg->s32_max_value, (s32)reg->smax_value); | |
| 2078 | − } | |
| 2079 | −} | |
| 2080 | − | |
| 2081 | −static void deduce_bounds_32_from_32(struct bpf_reg_state *reg) | |
| 2082 | −{ | |
| 2083 | − /* if u32 range forms a valid s32 range (due to matching sign bit), | |
| 2084 | − * try to learn from that | |
| 2085 | − */ | |
| 2086 | − if ((s32)reg->u32_min_value <= (s32)reg->u32_max_value) { | |
| 2087 | − reg->s32_min_value = max_t(s32, reg->s32_min_value, reg->u32_min_value); | |
| 2088 | − reg->s32_max_value = min_t(s32, reg->s32_max_value, reg->u32_max_value); | |
| 2089 | − } | |
| 2090 | − /* If we cannot cross the sign boundary, then signed and unsigned bounds | |
| 2091 | − * are the same, so combine. This works even in the negative case, e.g. | |
| 2092 | − * -3 s<= x s<= -1 implies 0xf...fd u<= x u<= 0xf...ff. | |
| 2093 | − */ | |
| 2094 | − if ((u32)reg->s32_min_value <= (u32)reg->s32_max_value) { | |
| 2095 | − reg->u32_min_value = max_t(u32, reg->s32_min_value, reg->u32_min_value); | |
| 2096 | − reg->u32_max_value = min_t(u32, reg->s32_max_value, reg->u32_max_value); | |
| 2097 | − } else { | |
| 2098 | − if (reg->u32_max_value < (u32)reg->s32_min_value) { | |
| 2099 | − /* See __reg64_deduce_bounds() for detailed explanation. | |
| 2100 | − * Refine ranges in the following situation: | |
| 2101 | − * | |
| 2102 | − * 0 U32_MAX | |
| 2103 | − * | [xxxxxxxxxxxxxx u32 range xxxxxxxxxxxxxx] | | |
| 2104 | − * |----------------------------|----------------------------| | |
| 2105 | − * |xxxxx s32 range xxxxxxxxx] [xxxxxxx| | |
| 2106 | − * 0 S32_MAX S32_MIN -1 | |
| 2107 | − */ | |
| 2108 | − reg->s32_min_value = (s32)reg->u32_min_value; | |
| 2109 | − reg->u32_max_value = min_t(u32, reg->u32_max_value, reg->s32_max_value); | |
| 2110 | − } else if ((u32)reg->s32_max_value < reg->u32_min_value) { | |
| 2111 | − /* | |
| 2112 | − * 0 U32_MAX | |
| 2113 | − * | [xxxxxxxxxxxxxx u32 range xxxxxxxxxxxxxx] | | |
| 2114 | − * |----------------------------|----------------------------| | |
| 2115 | − * |xxxxxxxxx] [xxxxxxxxxxxx s32 range | | |
| 2116 | − * 0 S32_MAX S32_MIN -1 | |
| 2117 | − */ | |
| 2118 | − reg->s32_max_value = (s32)reg->u32_max_value; | |
| 2119 | − reg->u32_min_value = max_t(u32, reg->u32_min_value, reg->s32_min_value); | |
| 2120 | − } | |
| 2121 | − } | |
| 2122 | −} | |
| 2123 | − | |
| 2124 | −static void deduce_bounds_64_from_64(struct bpf_reg_state *reg) | |
| 2125 | −{ | |
| 2126 | − /* If u64 range forms a valid s64 range (due to matching sign bit), | |
| 2127 | − * try to learn from that. Let's do a bit of ASCII art to see when | |
| 2128 | − * this is happening. Let's take u64 range first: | |
| 2129 | − * | |
| 2130 | − * 0 0x7fffffffffffffff 0x8000000000000000 U64_MAX | |
| 2131 | − * |-------------------------------|--------------------------------| | |
| 2132 | − * | |
| 2133 | − * Valid u64 range is formed when umin and umax are anywhere in the | |
| 2134 | − * range [0, U64_MAX], and umin <= umax. u64 case is simple and | |
| 2135 | − * straightforward. Let's see how s64 range maps onto the same range | |
| 2136 | − * of values, annotated below the line for comparison: | |
| 2137 | − * | |
| 2138 | − * 0 0x7fffffffffffffff 0x8000000000000000 U64_MAX | |
| 2139 | − * |-------------------------------|--------------------------------| | |
| 2140 | − * 0 S64_MAX S64_MIN -1 | |
| 2141 | − * | |
| 2142 | − * So s64 values basically start in the middle and they are logically | |
| 2143 | − * contiguous to the right of it, wrapping around from -1 to 0, and | |
| 2144 | − * then finishing as S64_MAX (0x7fffffffffffffff) right before | |
| 2145 | − * S64_MIN. We can try drawing the continuity of u64 vs s64 values | |
| 2146 | − * more visually as mapped to sign-agnostic range of hex values. | |
| 2147 | − * | |
| 2148 | − * u64 start u64 end | |
| 2149 | − * _______________________________________________________________ | |
| 2150 | − * / \ | |
| 2151 | − * 0 0x7fffffffffffffff 0x8000000000000000 U64_MAX | |
| 2152 | − * |-------------------------------|--------------------------------| | |
| 2153 | − * 0 S64_MAX S64_MIN -1 | |
| 2154 | − * / \ | |
| 2155 | − * >------------------------------ -------------------------------> | |
| 2156 | − * s64 continues... s64 end s64 start s64 "midpoint" | |
| 2157 | − * | |
| 2158 | − * What this means is that, in general, we can't always derive | |
| 2159 | − * something new about u64 from any random s64 range, and vice versa. | |
| 2160 | − * | |
| 2161 | − * But we can do that in two particular cases. One is when entire | |
| 2162 | − * u64/s64 range is *entirely* contained within left half of the above | |
| 2163 | − * diagram or when it is *entirely* contained in the right half. I.e.: | |
| 2164 | − * | |
| 2165 | − * |-------------------------------|--------------------------------| | |
| 2166 | − * ^ ^ ^ ^ | |
| 2167 | − * A B C D | |
| 2168 | − * | |
| 2169 | − * [A, B] and [C, D] are contained entirely in their respective halves | |
| 2170 | − * and form valid contiguous ranges as both u64 and s64 values. [A, B] | |
| 2171 | − * will be non-negative both as u64 and s64 (and in fact it will be | |
| 2172 | − * identical ranges no matter the signedness). [C, D] treated as s64 | |
| 2173 | − * will be a range of negative values, while in u64 it will be | |
| 2174 | − * non-negative range of values larger than 0x8000000000000000. | |
| 2175 | − * | |
| 2176 | − * Now, any other range here can't be represented in both u64 and s64 | |
| 2177 | − * simultaneously. E.g., [A, C], [A, D], [B, C], [B, D] are valid | |
| 2178 | − * contiguous u64 ranges, but they are discontinuous in s64. [B, C] | |
| 2179 | − * in s64 would be properly presented as [S64_MIN, C] and [B, S64_MAX], | |
| 2180 | − * for example. Similarly, valid s64 range [D, A] (going from negative | |
| 2181 | − * to positive values), would be two separate [D, U64_MAX] and [0, A] | |
| 2182 | − * ranges as u64. Currently reg_state can't represent two segments per | |
| 2183 | − * numeric domain, so in such situations we can only derive maximal | |
| 2184 | − * possible range ([0, U64_MAX] for u64, and [S64_MIN, S64_MAX] for s64). | |
| 2185 | − * | |
| 2186 | − * So we use these facts to derive umin/umax from smin/smax and vice | |
| 2187 | − * versa only if they stay within the same "half". This is equivalent | |
| 2188 | − * to checking sign bit: lower half will have sign bit as zero, upper | |
| 2189 | − * half have sign bit 1. Below in code we simplify this by just | |
| 2190 | − * casting umin/umax as smin/smax and checking if they form valid | |
| 2191 | − * range, and vice versa. Those are equivalent checks. | |
| 2192 | − */ | |
| 2193 | − if ((s64)reg->umin_value <= (s64)reg->umax_value) { | |
| 2194 | − reg->smin_value = max_t(s64, reg->smin_value, reg->umin_value); | |
| 2195 | − reg->smax_value = min_t(s64, reg->smax_value, reg->umax_value); | |
| 2196 | − } | |
| 2197 | − /* If we cannot cross the sign boundary, then signed and unsigned bounds | |
| 2198 | − * are the same, so combine. This works even in the negative case, e.g. | |
| 2199 | − * -3 s<= x s<= -1 implies 0xf...fd u<= x u<= 0xf...ff. | |
| 2200 | − */ | |
| 2201 | − if ((u64)reg->smin_value <= (u64)reg->smax_value) { | |
| 2202 | − reg->umin_value = max_t(u64, reg->smin_value, reg->umin_value); | |
| 2203 | − reg->umax_value = min_t(u64, reg->smax_value, reg->umax_value); | |
| 2204 | − } else { | |
| 2205 | − /* If the s64 range crosses the sign boundary, then it's split | |
| 2206 | − * between the beginning and end of the U64 domain. In that | |
| 2207 | − * case, we can derive new bounds if the u64 range overlaps | |
| 2208 | − * with only one end of the s64 range. | |
| 2209 | − * | |
| 2210 | − * In the following example, the u64 range overlaps only with | |
| 2211 | − * positive portion of the s64 range. | |
| 2212 | − * | |
| 2213 | − * 0 U64_MAX | |
| 2214 | − * | [xxxxxxxxxxxxxx u64 range xxxxxxxxxxxxxx] | | |
| 2215 | − * |----------------------------|----------------------------| | |
| 2216 | − * |xxxxx s64 range xxxxxxxxx] [xxxxxxx| | |
| 2217 | − * 0 S64_MAX S64_MIN -1 | |
| 2218 | − * | |
| 2219 | − * We can thus derive the following new s64 and u64 ranges. | |
| 2220 | − * | |
| 2221 | − * 0 U64_MAX | |
| 2222 | − * | [xxxxxx u64 range xxxxx] | | |
| 2223 | − * |----------------------------|----------------------------| | |
| 2224 | − * | [xxxxxx s64 range xxxxx] | | |
| 2225 | − * 0 S64_MAX S64_MIN -1 | |
| 2226 | − * | |
| 2227 | − * If they overlap in two places, we can't derive anything | |
| 2228 | − * because reg_state can't represent two ranges per numeric | |
| 2229 | − * domain. | |
| 2230 | − * | |
| 2231 | − * 0 U64_MAX | |
| 2232 | − * | [xxxxxxxxxxxxxxxxx u64 range xxxxxxxxxxxxxxxxx] | | |
| 2233 | − * |----------------------------|----------------------------| | |
| 2234 | − * |xxxxx s64 range xxxxxxxxx] [xxxxxxxxxx| | |
| 2235 | − * 0 S64_MAX S64_MIN -1 | |
| 2236 | − * | |
| 2237 | − * The first condition below corresponds to the first diagram | |
| 2238 | − * above. | |
| 2239 | − */ | |
| 2240 | − if (reg->umax_value < (u64)reg->smin_value) { | |
| 2241 | − reg->smin_value = (s64)reg->umin_value; | |
| 2242 | − reg->umax_value = min_t(u64, reg->umax_value, reg->smax_value); | |
| 2243 | − } else if ((u64)reg->smax_value < reg->umin_value) { | |
| 2244 | − /* This second condition considers the case where the u64 range | |
| 2245 | − * overlaps with the negative portion of the s64 range: | |
| 2246 | − * | |
| 2247 | − * 0 U64_MAX | |
| 2248 | − * | [xxxxxxxxxxxxxx u64 range xxxxxxxxxxxxxx] | | |
| 2249 | − * |----------------------------|----------------------------| | |
| 2250 | − * |xxxxxxxxx] [xxxxxxxxxxxx s64 range | | |
| 2251 | − * 0 S64_MAX S64_MIN -1 | |
| 2252 | − */ | |
| 2253 | − reg->smax_value = (s64)reg->umax_value; | |
| 2254 | − reg->umin_value = max_t(u64, reg->umin_value, reg->smin_value); | |
| 2255 | − } | |
| 2256 | − } | |
| 2044 | + cnum32_intersect_with(®->r32, cnum32_from_cnum64(reg->r64)); | |
| 2257 | 2045 | } |
| 2258 | 2046 | |
| 2259 | 2047 | static void deduce_bounds_64_from_32(struct bpf_reg_state *reg) |
| 2260 | 2048 | { |
| 2261 | − /* Try to tighten 64-bit bounds from 32-bit knowledge, using 32-bit | |
| 2262 | − * values on both sides of 64-bit range in hope to have tighter range. | |
| 2263 | − * E.g., if r1 is [0x1'00000000, 0x3'80000000], and we learn from | |
| 2264 | − * 32-bit signed > 0 operation that s32 bounds are now [1; 0x7fffffff]. | |
| 2265 | − * With this, we can substitute 1 as low 32-bits of _low_ 64-bit bound | |
| 2266 | − * (0x100000000 -> 0x100000001) and 0x7fffffff as low 32-bits of | |
| 2267 | − * _high_ 64-bit bound (0x380000000 -> 0x37fffffff) and arrive at a | |
| 2268 | − * better overall bounds for r1 as [0x1'000000001; 0x3'7fffffff]. | |
| 2269 | − * We just need to make sure that derived bounds we are intersecting | |
| 2270 | − * with are well-formed ranges in respective s64 or u64 domain, just | |
| 2271 | − * like we do with similar kinds of 32-to-64 or 64-to-32 adjustments. | |
| 2272 | − */ | |
| 2273 | − __u64 new_umin, new_umax; | |
| 2274 | − __s64 new_smin, new_smax; | |
| 2275 | − | |
| 2276 | − /* u32 -> u64 tightening, it's always well-formed */ | |
| 2277 | − new_umin = (reg->umin_value & ~0xffffffffULL) | reg->u32_min_value; | |
| 2278 | − new_umax = (reg->umax_value & ~0xffffffffULL) | reg->u32_max_value; | |
| 2279 | − reg->umin_value = max_t(u64, reg->umin_value, new_umin); | |
| 2280 | − reg->umax_value = min_t(u64, reg->umax_value, new_umax); | |
| 2281 | − /* u32 -> s64 tightening, u32 range embedded into s64 preserves range validity */ | |
| 2282 | − new_smin = (reg->smin_value & ~0xffffffffULL) | reg->u32_min_value; | |
| 2283 | − new_smax = (reg->smax_value & ~0xffffffffULL) | reg->u32_max_value; | |
| 2284 | − reg->smin_value = max_t(s64, reg->smin_value, new_smin); | |
| 2285 | − reg->smax_value = min_t(s64, reg->smax_value, new_smax); | |
| 2286 | − | |
| 2287 | − /* Here we would like to handle a special case after sign extending load, | |
| 2288 | − * when upper bits for a 64-bit range are all 1s or all 0s. | |
| 2289 | − * | |
| 2290 | − * Upper bits are all 1s when register is in a range: | |
| 2291 | − * [0xffff_ffff_0000_0000, 0xffff_ffff_ffff_ffff] | |
| 2292 | − * Upper bits are all 0s when register is in a range: | |
| 2293 | − * [0x0000_0000_0000_0000, 0x0000_0000_ffff_ffff] | |
| 2294 | − * Together this forms are continuous range: | |
| 2295 | − * [0xffff_ffff_0000_0000, 0x0000_0000_ffff_ffff] | |
| 2296 | − * | |
| 2297 | − * Now, suppose that register range is in fact tighter: | |
| 2298 | − * [0xffff_ffff_8000_0000, 0x0000_0000_ffff_ffff] (R) | |
| 2299 | − * Also suppose that it's 32-bit range is positive, | |
| 2300 | − * meaning that lower 32-bits of the full 64-bit register | |
| 2301 | − * are in the range: | |
| 2302 | − * [0x0000_0000, 0x7fff_ffff] (W) | |
| 2303 | − * | |
| 2304 | − * If this happens, then any value in a range: | |
| 2305 | − * [0xffff_ffff_0000_0000, 0xffff_ffff_7fff_ffff] | |
| 2306 | − * is smaller than a lowest bound of the range (R): | |
| 2307 | − * 0xffff_ffff_8000_0000 | |
| 2308 | − * which means that upper bits of the full 64-bit register | |
| 2309 | − * can't be all 1s, when lower bits are in range (W). | |
| 2310 | − * | |
| 2311 | − * Note that: | |
| 2312 | − * - 0xffff_ffff_8000_0000 == (s64)S32_MIN | |
| 2313 | − * - 0x0000_0000_7fff_ffff == (s64)S32_MAX | |
| 2314 | − * These relations are used in the conditions below. | |
| 2315 | − */ | |
| 2316 | − if (reg->s32_min_value >= 0 && reg->smin_value >= S32_MIN && reg->smax_value <= S32_MAX) { | |
| 2317 | − reg->smin_value = reg->s32_min_value; | |
| 2318 | − reg->smax_value = reg->s32_max_value; | |
| 2319 | − reg->umin_value = reg->s32_min_value; | |
| 2320 | − reg->umax_value = reg->s32_max_value; | |
| 2321 | − reg->var_off = tnum_intersect(reg->var_off, | |
| 2322 | − tnum_range(reg->smin_value, reg->smax_value)); | |
| 2323 | − } | |
| 2049 | + reg->r64 = cnum64_cnum32_intersect(reg->r64, reg->r32); | |
| 2324 | 2050 | } |
| 2325 | 2051 | |
| 2326 | 2052 | static void __reg_deduce_bounds(struct bpf_reg_state *reg) |
| 2327 | 2053 | { |
| 2328 | − deduce_bounds_64_from_64(reg); | |
| 2329 | 2054 | deduce_bounds_32_from_64(reg); |
| 2330 | − deduce_bounds_32_from_32(reg); | |
| 2331 | 2055 | deduce_bounds_64_from_32(reg); |
| 2332 | 2056 | } |
| 2333 | 2057 | |
| @@ -2335,11 +2059,11 @@ static void __reg_deduce_bounds(struct bpf_reg_state *reg) · 1 unmodified line | ||
| 2335 | 2059 | static void __reg_bound_offset(struct bpf_reg_state *reg) |
| 2336 | 2060 | { |
| 2337 | 2061 | struct tnum var64_off = tnum_intersect(reg->var_off, |
| 2338 | − tnum_range(reg->umin_value, | |
| 2339 | − reg->umax_value)); | |
| 2062 | + tnum_range(reg_umin(reg), | |
| 2063 | + reg_umax(reg))); | |
| 2340 | 2064 | struct tnum var32_off = tnum_intersect(tnum_subreg(var64_off), |
| 2341 | − tnum_range(reg->u32_min_value, | |
| 2342 | − reg->u32_max_value)); | |
| 2065 | + tnum_range(reg_u32_min(reg), | |
| 2066 | + reg_u32_max(reg))); | |
| 2343 | 2067 | |
| 2344 | 2068 | reg->var_off = tnum_or(tnum_clear_subreg(var64_off), var32_off); |
| 2345 | 2069 | } |
| @@ -2365,35 +2089,25 @@ static void reg_bounds_sync(struct bpf_reg_state *reg) · 19 unmodified lines | ||
| 2365 | 2089 | __update_reg_bounds(reg); |
| 2366 | 2090 | } |
| 2367 | 2091 | |
| 2368 | −static bool range_bounds_violation(struct bpf_reg_state *reg) | |
| 2369 | −{ | |
| 2370 | − return (reg->umin_value > reg->umax_value || reg->smin_value > reg->smax_value || | |
| 2371 | − reg->u32_min_value > reg->u32_max_value || | |
| 2372 | − reg->s32_min_value > reg->s32_max_value); | |
| 2373 | −} | |
| 2374 | − | |
| 2375 | 2092 | static bool const_tnum_range_mismatch(struct bpf_reg_state *reg) |
| 2376 | 2093 | { |
| 2377 | − u64 uval = reg->var_off.value; | |
| 2378 | − s64 sval = (s64)uval; | |
| 2379 | − | |
| 2380 | 2094 | if (!tnum_is_const(reg->var_off)) |
| 2381 | 2095 | return false; |
| 2382 | 2096 | |
| 2383 | − return reg->umin_value != uval || reg->umax_value != uval || | |
| 2384 | − reg->smin_value != sval || reg->smax_value != sval; | |
| 2097 | + return !cnum64_is_const(reg->r64) || reg->r64.base != reg->var_off.value; | |
| 2385 | 2098 | } |
| 2386 | 2099 | |
| 2387 | 2100 | static bool const_tnum_range_mismatch_32(struct bpf_reg_state *reg) |
| 2388 | 2101 | { |
| 2389 | − u32 uval32 = tnum_subreg(reg->var_off).value; | |
| 2390 | − s32 sval32 = (s32)uval32; | |
| 2391 | − | |
| 2392 | 2102 | if (!tnum_subreg_is_const(reg->var_off)) |
| 2393 | 2103 | return false; |
| 2394 | 2104 | |
| 2395 | − return reg->u32_min_value != uval32 || reg->u32_max_value != uval32 || | |
| 2396 | − reg->s32_min_value != sval32 || reg->s32_max_value != sval32; | |
| 2105 | + return !cnum32_is_const(reg->r32) || reg->r32.base != tnum_subreg(reg->var_off).value; | |
| 2106 | +} | |
| 2107 | + | |
| 2108 | +static bool range_bounds_violation(struct bpf_reg_state *reg) | |
| 2109 | +{ | |
| 2110 | + return cnum32_is_empty(reg->r32) || cnum64_is_empty(reg->r64); | |
| 2397 | 2111 | } |
| 2398 | 2112 | |
| 2399 | 2113 | static int reg_bounds_sanity_check(struct bpf_verifier_env *env, |
| @@ -2418,12 +2132,11 @@ static int reg_bounds_sanity_check(struct bpf_verifier_env *env, · 18 unmodified lines | ||
| 2418 | 2132 | |
| 2419 | 2133 | return 0; |
| 2420 | 2134 | out: |
| 2421 | − verifier_bug(env, "REG INVARIANTS VIOLATION (%s): %s u64=[%#llx, %#llx] " | |
| 2422 | − "s64=[%#llx, %#llx] u32=[%#x, %#x] s32=[%#x, %#x] var_off=(%#llx, %#llx)", | |
| 2423 | − ctx, msg, reg->umin_value, reg->umax_value, | |
| 2424 | − reg->smin_value, reg->smax_value, | |
| 2425 | − reg->u32_min_value, reg->u32_max_value, | |
| 2426 | − reg->s32_min_value, reg->s32_max_value, | |
| 2135 | + verifier_bug(env, "REG INVARIANTS VIOLATION (%s): %s r64={.base=%#llx, .size=%#llx} " | |
| 2136 | + "r32={.base=%#x, .size=%#x} var_off=(%#llx, %#llx)", | |
| 2137 | + ctx, msg, | |
| 2138 | + reg->r64.base, reg->r64.size, | |
| 2139 | + reg->r32.base, reg->r32.size, | |
| 2427 | 2140 | reg->var_off.value, reg->var_off.mask); |
| 2428 | 2141 | if (env->test_reg_invariants) |
| 2429 | 2142 | return -EFAULT; |
| @@ -2431,44 +2144,15 @@ out: · 1 unmodified line | ||
| 2431 | 2144 | return 0; |
| 2432 | 2145 | } |
| 2433 | 2146 | |
| 2434 | −static bool __reg32_bound_s64(s32 a) | |
| 2435 | −{ | |
| 2436 | − return a >= 0 && a <= S32_MAX; | |
| 2437 | −} | |
| 2438 | − | |
| 2439 | −static void __reg_assign_32_into_64(struct bpf_reg_state *reg) | |
| 2440 | −{ | |
| 2441 | − reg->umin_value = reg->u32_min_value; | |
| 2442 | − reg->umax_value = reg->u32_max_value; | |
| 2443 | − | |
| 2444 | − /* Attempt to pull 32-bit signed bounds into 64-bit bounds but must | |
| 2445 | − * be positive otherwise set to worse case bounds and refine later | |
| 2446 | − * from tnum. | |
| 2447 | − */ | |
| 2448 | − if (__reg32_bound_s64(reg->s32_min_value) && | |
| 2449 | − __reg32_bound_s64(reg->s32_max_value)) { | |
| 2450 | − reg->smin_value = reg->s32_min_value; | |
| 2451 | − reg->smax_value = reg->s32_max_value; | |
| 2452 | − } else { | |
| 2453 | − reg->smin_value = 0; | |
| 2454 | − reg->smax_value = U32_MAX; | |
| 2455 | − } | |
| 2456 | −} | |
| 2457 | − | |
| 2458 | 2147 | /* Mark a register as having a completely unknown (scalar) value. */ |
| 2459 | 2148 | void bpf_mark_reg_unknown_imprecise(struct bpf_reg_state *reg) |
| 2460 | 2149 | { |
| 2461 | − /* | |
| 2462 | − * Clear type, off, and union(map_ptr, range) and | |
| 2463 | − * padding between 'type' and union | |
| 2464 | − */ | |
| 2465 | − memset(reg, 0, offsetof(struct bpf_reg_state, var_off)); | |
| 2150 | + s32 subreg_def = reg->subreg_def; | |
| 2151 | + | |
| 2152 | + memset(reg, 0, sizeof(*reg)); | |
| 2466 | 2153 | reg->type = SCALAR_VALUE; |
| 2467 | − reg->id = 0; | |
| 2468 | − reg->ref_obj_id = 0; | |
| 2469 | 2154 | reg->var_off = tnum_unknown; |
| 2470 | − reg->frameno = 0; | |
| 2471 | − reg->precise = false; | |
| 2155 | + reg->subreg_def = subreg_def; | |
| 2472 | 2156 | __mark_reg_unbounded(reg); |
| 2473 | 2157 | } |
| 2474 | 2158 | |
| @@ -2496,11 +2180,12 @@ static int __mark_reg_s32_range(struct bpf_verifier_env *env, · 21 unmodified lines | ||
| 2496 | 2180 | { |
| 2497 | 2181 | struct bpf_reg_state *reg = regs + regno; |
| 2498 | 2182 | |
| 2499 | − reg->s32_min_value = max_t(s32, reg->s32_min_value, s32_min); | |
| 2500 | − reg->s32_max_value = min_t(s32, reg->s32_max_value, s32_max); | |
| 2501 | − | |
| 2502 | − reg->smin_value = max_t(s64, reg->smin_value, s32_min); | |
| 2503 | − reg->smax_value = min_t(s64, reg->smax_value, s32_max); | |
| 2183 | + reg_set_srange32(reg, | |
| 2184 | + max_t(s32, reg_s32_min(reg), s32_min), | |
| 2185 | + min_t(s32, reg_s32_max(reg), s32_max)); | |
| 2186 | + reg_set_srange64(reg, | |
| 2187 | + max_t(s64, reg_smin(reg), s32_min), | |
| 2188 | + min_t(s64, reg_smax(reg), s32_max)); | |
| 2504 | 2189 | |
| 2505 | 2190 | reg_bounds_sync(reg); |
| 2506 | 2191 | |
| @@ -3295,50 +2980,13 @@ out: · 788 unmodified lines | ||
| 3295 | 2980 | return ret; |
| 3296 | 2981 | } |
| 3297 | 2982 | |
| 3298 | −static int mark_stack_slot_obj_read(struct bpf_verifier_env *env, struct bpf_reg_state *reg, | |
| 3299 | − int spi, int nr_slots) | |
| 2983 | +static void mark_stack_slots_scratched(struct bpf_verifier_env *env, | |
| 2984 | + int spi, int nr_slots) | |
| 3300 | 2985 | { |
| 3301 | 2986 | int i; |
| 3302 | 2987 | |
| 3303 | 2988 | for (i = 0; i < nr_slots; i++) |
| 3304 | 2989 | mark_stack_slot_scratched(env, spi - i); |
| 3305 | − return 0; | |
| 3306 | −} | |
| 3307 | − | |
| 3308 | −static int mark_dynptr_read(struct bpf_verifier_env *env, struct bpf_reg_state *reg) | |
| 3309 | −{ | |
| 3310 | − int spi; | |
| 3311 | − | |
| 3312 | − /* For CONST_PTR_TO_DYNPTR, it must have already been done by | |
| 3313 | − * check_reg_arg in check_helper_call and mark_btf_func_reg_size in | |
| 3314 | − * check_kfunc_call. | |
| 3315 | − */ | |
| 3316 | − if (reg->type == CONST_PTR_TO_DYNPTR) | |
| 3317 | − return 0; | |
| 3318 | − spi = dynptr_get_spi(env, reg); | |
| 3319 | − if (spi < 0) | |
| 3320 | − return spi; | |
| 3321 | − /* Caller ensures dynptr is valid and initialized, which means spi is in | |
| 3322 | − * bounds and spi is the first dynptr slot. Simply mark stack slot as | |
| 3323 | − * read. | |
| 3324 | − */ | |
| 3325 | − return mark_stack_slot_obj_read(env, reg, spi, BPF_DYNPTR_NR_SLOTS); | |
| 3326 | −} | |
| 3327 | − | |
| 3328 | −static int mark_iter_read(struct bpf_verifier_env *env, struct bpf_reg_state *reg, | |
| 3329 | − int spi, int nr_slots) | |
| 3330 | −{ | |
| 3331 | − return mark_stack_slot_obj_read(env, reg, spi, nr_slots); | |
| 3332 | −} | |
| 3333 | − | |
| 3334 | −static int mark_irq_flag_read(struct bpf_verifier_env *env, struct bpf_reg_state *reg) | |
| 3335 | −{ | |
| 3336 | − int spi; | |
| 3337 | − | |
| 3338 | − spi = irq_flag_get_spi(env, reg); | |
| 3339 | − if (spi < 0) | |
| 3340 | − return spi; | |
| 3341 | − return mark_stack_slot_obj_read(env, reg, spi, 1); | |
| 3342 | 2990 | } |
| 3343 | 2991 | |
| 3344 | 2992 | /* This function is supposed to be used by the following 32-bit optimization |
| @@ -3491,17 +3139,12 @@ static int check_reg_arg(struct bpf_verifier_env *env, u32 regno, · 146 unmodified lines | ||
| 3491 | 3139 | return __check_reg_arg(env, state->regs, regno, t); |
| 3492 | 3140 | } |
| 3493 | 3141 | |
| 3494 | −static int insn_stack_access_flags(int frameno, int spi) | |
| 3495 | −{ | |
| 3496 | − return INSN_F_STACK_ACCESS | (spi << INSN_F_SPI_SHIFT) | frameno; | |
| 3497 | −} | |
| 3498 | − | |
| 3499 | 3142 | static void mark_indirect_target(struct bpf_verifier_env *env, int idx) |
| 3500 | 3143 | { |
| 3501 | 3144 | env->insn_aux_data[idx].indirect_target = true; |
| 3502 | 3145 | } |
| 3503 | 3146 | |
| 3504 | −#define LR_FRAMENO_BITS 3 | |
| 3147 | +#define LR_FRAMENO_BITS 4 | |
| 3505 | 3148 | #define LR_SPI_BITS 6 |
| 3506 | 3149 | #define LR_ENTRY_BITS (LR_SPI_BITS + LR_FRAMENO_BITS + 1) |
| 3507 | 3150 | #define LR_SIZE_BITS 4 |
| @@ -3510,7 +3153,11 @@ static void mark_indirect_target(struct bpf_verifier_env *env, int idx) · 2 unmodified lines | ||
| 3510 | 3153 | #define LR_SIZE_MASK ((1ull << LR_SIZE_BITS) - 1) |
| 3511 | 3154 | #define LR_SPI_OFF LR_FRAMENO_BITS |
| 3512 | 3155 | #define LR_IS_REG_OFF (LR_SPI_BITS + LR_FRAMENO_BITS) |
| 3513 | −#define LINKED_REGS_MAX 6 | |
| 3156 | +#define LINKED_REGS_MAX 5 | |
| 3157 | + | |
| 3158 | +static_assert(MAX_CALL_FRAMES <= (1 << LR_FRAMENO_BITS)); | |
| 3159 | +static_assert(LINKED_REGS_MAX < (1 << LR_SIZE_BITS)); | |
| 3160 | +static_assert(LINKED_REGS_MAX * LR_ENTRY_BITS + LR_SIZE_BITS <= 64); | |
| 3514 | 3161 | |
| 3515 | 3162 | struct linked_reg { |
| 3516 | 3163 | u8 frameno; |
| @@ -3534,10 +3181,11 @@ static struct linked_reg *linked_regs_push(struct linked_regs *s) · 17 unmodified lines | ||
| 3534 | 3181 | return NULL; |
| 3535 | 3182 | } |
| 3536 | 3183 | |
| 3537 | −/* Use u64 as a vector of 6 10-bit values, use first 4-bits to track | |
| 3184 | +/* | |
| 3185 | + * Use u64 as a vector of 5 11-bit values, use first 4-bits to track | |
| 3538 | 3186 | * number of elements currently in stack. |
| 3539 | − * Pack one history entry for linked registers as 10 bits in the following format: | |
| 3540 | − * - 3-bits frameno | |
| 3187 | + * Pack one history entry for linked registers as 11 bits in the following format: | |
| 3188 | + * - 4-bits frameno | |
| 3541 | 3189 | * - 6-bits spi_or_reg |
| 3542 | 3190 | * - 1-bit is_reg |
| 3543 | 3191 | */ |
| @@ -3733,12 +3381,6 @@ static void assign_scalar_id_before_mov(struct bpf_verifier_env *env, · 189 unmodified lines | ||
| 3733 | 3381 | src_reg->id = ++env->id_gen; |
| 3734 | 3382 | } |
| 3735 | 3383 | |
| 3736 | −/* Copy src state preserving dst->parent and dst->live fields */ | |
| 3737 | −static void copy_register_state(struct bpf_reg_state *dst, const struct bpf_reg_state *src) | |
| 3738 | −{ | |
| 3739 | − *dst = *src; | |
| 3740 | −} | |
| 3741 | − | |
| 3742 | 3384 | static void save_register_state(struct bpf_verifier_env *env, |
| 3743 | 3385 | struct bpf_func_state *state, |
| 3744 | 3386 | int spi, struct bpf_reg_state *reg, |
| @@ -3746,7 +3388,7 @@ static void save_register_state(struct bpf_verifier_env *env, · 1 unmodified line | ||
| 3746 | 3388 | { |
| 3747 | 3389 | int i; |
| 3748 | 3390 | |
| 3749 | − copy_register_state(&state->stack[spi].spilled_ptr, reg); | |
| 3391 | + state->stack[spi].spilled_ptr = *reg; | |
| 3750 | 3392 | |
| 3751 | 3393 | for (i = BPF_REG_SIZE; i > BPF_REG_SIZE - size; i--) |
| 3752 | 3394 | state->stack[spi].slot_type[i - 1] = STACK_SPILL; |
| @@ -3763,7 +3405,7 @@ static bool is_bpf_st_mem(struct bpf_insn *insn) · 10 unmodified lines | ||
| 3763 | 3405 | |
| 3764 | 3406 | static int get_reg_width(struct bpf_reg_state *reg) |
| 3765 | 3407 | { |
| 3766 | − return fls64(reg->umax_value); | |
| 3408 | + return fls64(reg_umax(reg)); | |
| 3767 | 3409 | } |
| 3768 | 3410 | |
| 3769 | 3411 | /* See comment for mark_fastcall_pattern_for_call() */ |
| @@ -3816,7 +3458,8 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env, · 46 unmodified lines | ||
| 3816 | 3458 | int i, slot = -off - 1, spi = slot / BPF_REG_SIZE, err; |
| 3817 | 3459 | struct bpf_insn *insn = &env->prog->insnsi[insn_idx]; |
| 3818 | 3460 | struct bpf_reg_state *reg = NULL; |
| 3819 | − int insn_flags = insn_stack_access_flags(state->frameno, spi); | |
| 3461 | + int insn_flags = INSN_F_STACK_ACCESS; | |
| 3462 | + int hist_spi = spi, hist_frame = state->frameno; | |
| 3820 | 3463 | |
| 3821 | 3464 | /* caller checked that off % size == 0 and -MAX_BPF_STACK <= off < 0, |
| 3822 | 3465 | * so it's aligned access and [off, off + size) are within stack limits |
| @@ -3912,11 +3555,12 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env, · 89 unmodified lines | ||
| 3912 | 3555 | } |
| 3913 | 3556 | |
| 3914 | 3557 | if (insn_flags) |
| 3915 | − return bpf_push_jmp_history(env, env->cur_state, insn_flags, 0); | |
| 3558 | + return bpf_push_jmp_history(env, env->cur_state, insn_flags, | |
| 3559 | + hist_spi, hist_frame, 0); | |
| 3916 | 3560 | return 0; |
| 3917 | 3561 | } |
| 3918 | 3562 | |
| 3919 | −/* Write the stack: 'stack[ptr_regno + off] = value_regno'. 'ptr_regno' is | |
| 3563 | +/* Write the stack: 'stack[ptr_reg + off] = value_regno'. 'ptr_reg' is | |
| 3920 | 3564 | * known to contain a variable offset. |
| 3921 | 3565 | * This function checks whether the write is permitted and conservatively |
| 3922 | 3566 | * tracks the effects of the write, considering that each stack slot in the |
| @@ -3937,13 +3581,13 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env, · 14 unmodified lines | ||
| 3937 | 3581 | static int check_stack_write_var_off(struct bpf_verifier_env *env, |
| 3938 | 3582 | /* func where register points to */ |
| 3939 | 3583 | struct bpf_func_state *state, |
| 3940 | − int ptr_regno, int off, int size, | |
| 3584 | + struct bpf_reg_state *ptr_reg, int off, int size, | |
| 3941 | 3585 | int value_regno, int insn_idx) |
| 3942 | 3586 | { |
| 3943 | 3587 | struct bpf_func_state *cur; /* state of the current function */ |
| 3944 | 3588 | int min_off, max_off; |
| 3945 | 3589 | int i, err; |
| 3946 | − struct bpf_reg_state *ptr_reg = NULL, *value_reg = NULL; | |
| 3590 | + struct bpf_reg_state *value_reg = NULL; | |
| 3947 | 3591 | struct bpf_insn *insn = &env->prog->insnsi[insn_idx]; |
| 3948 | 3592 | bool writing_zero = false; |
| 3949 | 3593 | /* set if the fact that we're writing a zero is used to let any |
| @@ -3952,9 +3596,8 @@ static int check_stack_write_var_off(struct bpf_verifier_env *env, · 2 unmodified lines | ||
| 3952 | 3596 | bool zero_used = false; |
| 3953 | 3597 | |
| 3954 | 3598 | cur = env->cur_state->frame[env->cur_state->curframe]; |
| 3955 | − ptr_reg = &cur->regs[ptr_regno]; | |
| 3956 | − min_off = ptr_reg->smin_value + off; | |
| 3957 | − max_off = ptr_reg->smax_value + off + size; | |
| 3599 | + min_off = reg_smin(ptr_reg) + off; | |
| 3600 | + max_off = reg_smax(ptr_reg) + off + size; | |
| 3958 | 3601 | if (value_regno >= 0) |
| 3959 | 3602 | value_reg = &cur->regs[value_regno]; |
| 3960 | 3603 | if ((value_reg && bpf_register_is_null(value_reg)) || |
| @@ -4109,7 +3752,8 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env, · 148 unmodified lines | ||
| 4109 | 3752 | int i, slot = -off - 1, spi = slot / BPF_REG_SIZE; |
| 4110 | 3753 | struct bpf_reg_state *reg; |
| 4111 | 3754 | u8 *stype, type; |
| 4112 | − int insn_flags = insn_stack_access_flags(reg_state->frameno, spi); | |
| 3755 | + int insn_flags = INSN_F_STACK_ACCESS; | |
| 3756 | + int hist_spi = spi, hist_frame = reg_state->frameno; | |
| 4113 | 3757 | |
| 4114 | 3758 | stype = reg_state->stack[spi].slot_type; |
| 4115 | 3759 | reg = ®_state->stack[spi].spilled_ptr; |
| @@ -4146,7 +3790,7 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env, · 30 unmodified lines | ||
| 4146 | 3790 | * with the destination register on fill. |
| 4147 | 3791 | */ |
| 4148 | 3792 | assign_scalar_id_before_mov(env, reg); |
| 4149 | − copy_register_state(&state->regs[dst_regno], reg); | |
| 3793 | + state->regs[dst_regno] = *reg; | |
| 4150 | 3794 | state->regs[dst_regno].subreg_def = subreg_def; |
| 4151 | 3795 | |
| 4152 | 3796 | /* Break the relation on a narrowing fill. |
| @@ -4201,7 +3845,7 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env, · 48 unmodified lines | ||
| 4201 | 3845 | * with the destination register on fill. |
| 4202 | 3846 | */ |
| 4203 | 3847 | assign_scalar_id_before_mov(env, reg); |
| 4204 | − copy_register_state(&state->regs[dst_regno], reg); | |
| 3848 | + state->regs[dst_regno] = *reg; | |
| 4205 | 3849 | /* mark reg as written since spilled pointer state likely |
| 4206 | 3850 | * has its liveness marks cleared by is_state_visited() |
| 4207 | 3851 | * which resets stack/reg liveness for state transitions |
| @@ -4240,7 +3884,8 @@ static int check_stack_read_fixed_off(struct bpf_verifier_env *env, · 32 unmodified lines | ||
| 4240 | 3884 | insn_flags = 0; /* we are not restoring spilled register */ |
| 4241 | 3885 | } |
| 4242 | 3886 | if (insn_flags) |
| 4243 | − return bpf_push_jmp_history(env, env->cur_state, insn_flags, 0); | |
| 3887 | + return bpf_push_jmp_history(env, env->cur_state, insn_flags, | |
| 3888 | + hist_spi, hist_frame, 0); | |
| 4244 | 3889 | return 0; |
| 4245 | 3890 | } |
| 4246 | 3891 | |
| @@ -4249,8 +3894,8 @@ enum bpf_access_src { · 2 unmodified lines | ||
| 4249 | 3894 | ACCESS_HELPER = 2, /* the access is performed by a helper */ |
| 4250 | 3895 | }; |
| 4251 | 3896 | |
| 4252 | −static int check_stack_range_initialized(struct bpf_verifier_env *env, | |
| 4253 | − int regno, int off, int access_size, | |
| 3897 | +static int check_stack_range_initialized(struct bpf_verifier_env *env, struct bpf_reg_state *reg, | |
| 3898 | + argno_t argno, int off, int access_size, | |
| 4254 | 3899 | bool zero_size_allowed, |
| 4255 | 3900 | enum bpf_access_type type, |
| 4256 | 3901 | struct bpf_call_arg_meta *meta); |
| @@ -4260,37 +3905,35 @@ static struct bpf_reg_state *reg_state(struct bpf_verifier_env *env, int regno) · 3 unmodified lines | ||
| 4260 | 3905 | return cur_regs(env) + regno; |
| 4261 | 3906 | } |
| 4262 | 3907 | |
| 4263 | −/* Read the stack at 'ptr_regno + off' and put the result into the register | |
| 3908 | +/* Read the stack at 'reg + off' and put the result into the register | |
| 4264 | 3909 | * 'dst_regno'. |
| 4265 | − * 'off' includes the pointer register's fixed offset(i.e. 'ptr_regno.off'), | |
| 3910 | + * 'off' includes the pointer register's fixed offset(i.e. 'reg->off'), | |
| 4266 | 3911 | * but not its variable offset. |
| 4267 | 3912 | * 'size' is assumed to be <= reg size and the access is assumed to be aligned. |
| 4268 | 3913 | * |
| 4269 | 3914 | * As opposed to check_stack_read_fixed_off, this function doesn't deal with |
| 4270 | 3915 | * filling registers (i.e. reads of spilled register cannot be detected when |
| 4271 | 3916 | * the offset is not fixed). We conservatively mark 'dst_regno' as containing |
| 4272 | − * SCALAR_VALUE. That's why we assert that the 'ptr_regno' has a variable | |
| 3917 | + * SCALAR_VALUE. That's why we assert that the 'reg' has a variable | |
| 4273 | 3918 | * offset; for a fixed offset check_stack_read_fixed_off should be used |
| 4274 | 3919 | * instead. |
| 4275 | 3920 | */ |
| 4276 | −static int check_stack_read_var_off(struct bpf_verifier_env *env, | |
| 4277 | − int ptr_regno, int off, int size, int dst_regno) | |
| 3921 | +static int check_stack_read_var_off(struct bpf_verifier_env *env, struct bpf_reg_state *reg, | |
| 3922 | + argno_t ptr_argno, int off, int size, int dst_regno) | |
| 4278 | 3923 | { |
| 4279 | − /* The state of the source register. */ | |
| 4280 | − struct bpf_reg_state *reg = reg_state(env, ptr_regno); | |
| 4281 | 3924 | struct bpf_func_state *ptr_state = bpf_func(env, reg); |
| 4282 | 3925 | int err; |
| 4283 | 3926 | int min_off, max_off; |
| 4284 | 3927 | |
| 4285 | 3928 | /* Note that we pass a NULL meta, so raw access will not be permitted. |
| 4286 | 3929 | */ |
| 4287 | − err = check_stack_range_initialized(env, ptr_regno, off, size, | |
| 3930 | + err = check_stack_range_initialized(env, reg, ptr_argno, off, size, | |
| 4288 | 3931 | false, BPF_READ, NULL); |
| 4289 | 3932 | if (err) |
| 4290 | 3933 | return err; |
| 4291 | 3934 | |
| 4292 | − min_off = reg->smin_value + off; | |
| 4293 | − max_off = reg->smax_value + off; | |
| 3935 | + min_off = reg_smin(reg) + off; | |
| 3936 | + max_off = reg_smax(reg) + off; | |
| 4294 | 3937 | mark_reg_stack_read(env, ptr_state, min_off, max_off + size, dst_regno); |
| 4295 | 3938 | check_fastcall_stack_contract(env, ptr_state, env->insn_idx, min_off); |
| 4296 | 3939 | return 0; |
| @@ -4306,10 +3949,9 @@ static int check_stack_read_var_off(struct bpf_verifier_env *env, · 9 unmodified lines | ||
| 4306 | 3949 | * can be -1, meaning that the read value is not going to a register. |
| 4307 | 3950 | */ |
| 4308 | 3951 | static int check_stack_read(struct bpf_verifier_env *env, |
| 4309 | − int ptr_regno, int off, int size, | |
| 3952 | + struct bpf_reg_state *reg, argno_t ptr_argno, int off, int size, | |
| 4310 | 3953 | int dst_regno) |
| 4311 | 3954 | { |
| 4312 | − struct bpf_reg_state *reg = reg_state(env, ptr_regno); | |
| 4313 | 3955 | struct bpf_func_state *state = bpf_func(env, reg); |
| 4314 | 3956 | int err; |
| 4315 | 3957 | /* Some accesses are only permitted with a static offset. */ |
| @@ -4345,7 +3987,7 @@ static int check_stack_read(struct bpf_verifier_env *env, · 29 unmodified lines | ||
| 4345 | 3987 | * than fixed offset ones. Note that dst_regno >= 0 on this |
| 4346 | 3988 | * branch. |
| 4347 | 3989 | */ |
| 4348 | − err = check_stack_read_var_off(env, ptr_regno, off, size, | |
| 3990 | + err = check_stack_read_var_off(env, reg, ptr_argno, off, size, | |
| 4349 | 3991 | dst_regno); |
| 4350 | 3992 | } |
| 4351 | 3993 | return err; |
| @@ -4355,17 +3997,16 @@ static int check_stack_read(struct bpf_verifier_env *env, · 3 unmodified lines | ||
| 4355 | 3997 | /* check_stack_write dispatches to check_stack_write_fixed_off or |
| 4356 | 3998 | * check_stack_write_var_off. |
| 4357 | 3999 | * |
| 4358 | − * 'ptr_regno' is the register used as a pointer into the stack. | |
| 4000 | + * 'reg' is the register used as a pointer into the stack. | |
| 4359 | 4001 | * 'value_regno' is the register whose value we're writing to the stack. It can |
| 4360 | 4002 | * be -1, meaning that we're not writing from a register. |
| 4361 | 4003 | * |
| 4362 | 4004 | * The caller must ensure that the offset falls within the maximum stack size. |
| 4363 | 4005 | */ |
| 4364 | 4006 | static int check_stack_write(struct bpf_verifier_env *env, |
| 4365 | − int ptr_regno, int off, int size, | |
| 4007 | + struct bpf_reg_state *reg, int off, int size, | |
| 4366 | 4008 | int value_regno, int insn_idx) |
| 4367 | 4009 | { |
| 4368 | − struct bpf_reg_state *reg = reg_state(env, ptr_regno); | |
| 4369 | 4010 | struct bpf_func_state *state = bpf_func(env, reg); |
| 4370 | 4011 | int err; |
| 4371 | 4012 | |
| @@ -4378,28 +4019,135 @@ static int check_stack_write(struct bpf_verifier_env *env, · 6 unmodified lines | ||
| 4378 | 4019 | * than fixed offset ones. |
| 4379 | 4020 | */ |
| 4380 | 4021 | err = check_stack_write_var_off(env, state, |
| 4381 | − ptr_regno, off, size, | |
| 4022 | + reg, off, size, | |
| 4382 | 4023 | value_regno, insn_idx); |
| 4383 | 4024 | } |
| 4384 | 4025 | return err; |
| 4385 | 4026 | } |
| 4386 | 4027 | |
| 4387 | −static int check_map_access_type(struct bpf_verifier_env *env, u32 regno, | |
| 4028 | +/* | |
| 4029 | + * Write a value to the outgoing stack arg area. | |
| 4030 | + * off is a negative offset from r11 (e.g. -8 for arg6, -16 for arg7). | |
| 4031 | + */ | |
| 4032 | +static int check_stack_arg_write(struct bpf_verifier_env *env, struct bpf_func_state *state, | |
| 4033 | + int off, struct bpf_reg_state *value_reg) | |
| 4034 | +{ | |
| 4035 | + int max_stack_arg_regs = MAX_BPF_FUNC_ARGS - MAX_BPF_FUNC_REG_ARGS; | |
| 4036 | + struct bpf_subprog_info *subprog = &env->subprog_info[state->subprogno]; | |
| 4037 | + int spi = -off / BPF_REG_SIZE - 1; | |
| 4038 | + struct bpf_reg_state *arg; | |
| 4039 | + int err; | |
| 4040 | + | |
| 4041 | + if (spi >= max_stack_arg_regs) { | |
| 4042 | + verbose(env, "stack arg write offset %d exceeds max %d stack args\n", | |
| 4043 | + off, max_stack_arg_regs); | |
| 4044 | + return -EINVAL; | |
| 4045 | + } | |
| 4046 | + | |
| 4047 | + err = grow_stack_arg_slots(env, state, spi + 1); | |
| 4048 | + if (err) | |
| 4049 | + return err; | |
| 4050 | + | |
| 4051 | + /* Track the max outgoing stack arg slot count. */ | |
| 4052 | + if (spi + 1 > subprog->max_out_stack_arg_cnt) | |
| 4053 | + subprog->max_out_stack_arg_cnt = spi + 1; | |
| 4054 | + | |
| 4055 | + if (value_reg) { | |
| 4056 | + state->stack_arg_regs[spi] = *value_reg; | |
| 4057 | + } else { | |
| 4058 | + /* BPF_ST: store immediate, treat as scalar */ | |
| 4059 | + arg = &state->stack_arg_regs[spi]; | |
| 4060 | + arg->type = SCALAR_VALUE; | |
| 4061 | + __mark_reg_known(arg, env->prog->insnsi[env->insn_idx].imm); | |
| 4062 | + } | |
| 4063 | + state->no_stack_arg_load = true; | |
| 4064 | + return bpf_push_jmp_history(env, env->cur_state, | |
| 4065 | + INSN_F_STACK_ARG_ACCESS, spi, 0, 0); | |
| 4066 | +} | |
| 4067 | + | |
| 4068 | +/* | |
| 4069 | + * Read a value from the incoming stack arg area. | |
| 4070 | + * off is a positive offset from r11 (e.g. +8 for arg6, +16 for arg7). | |
| 4071 | + */ | |
| 4072 | +static int check_stack_arg_read(struct bpf_verifier_env *env, struct bpf_func_state *state, | |
| 4073 | + int off, int dst_regno) | |
| 4074 | +{ | |
| 4075 | + struct bpf_subprog_info *subprog = &env->subprog_info[state->subprogno]; | |
| 4076 | + struct bpf_verifier_state *vstate = env->cur_state; | |
| 4077 | + int spi = off / BPF_REG_SIZE - 1; | |
| 4078 | + struct bpf_func_state *caller, *cur; | |
| 4079 | + struct bpf_reg_state *arg; | |
| 4080 | + | |
| 4081 | + if (state->no_stack_arg_load) { | |
| 4082 | + verbose(env, "r11 load must be before any r11 store or call insn\n"); | |
| 4083 | + return -EINVAL; | |
| 4084 | + } | |
| 4085 | + | |
| 4086 | + if (spi + 1 > bpf_in_stack_arg_cnt(subprog)) { | |
| 4087 | + verbose(env, "invalid read from stack arg off %d depth %d\n", | |
| 4088 | + off, bpf_in_stack_arg_cnt(subprog) * BPF_REG_SIZE); | |
| 4089 | + return -EACCES; | |
| 4090 | + } | |
| 4091 | + | |
| 4092 | + caller = vstate->frame[vstate->curframe - 1]; | |
| 4093 | + arg = &caller->stack_arg_regs[spi]; | |
| 4094 | + cur = vstate->frame[vstate->curframe]; | |
| 4095 | + cur->regs[dst_regno] = *arg; | |
| 4096 | + return bpf_push_jmp_history(env, env->cur_state, | |
| 4097 | + INSN_F_STACK_ARG_ACCESS, spi, 0, 0); | |
| 4098 | +} | |
| 4099 | + | |
| 4100 | +static int mark_stack_arg_precision(struct bpf_verifier_env *env, int arg_idx) | |
| 4101 | +{ | |
| 4102 | + struct bpf_func_state *caller = cur_func(env); | |
| 4103 | + int spi = arg_idx - MAX_BPF_FUNC_REG_ARGS; | |
| 4104 | + | |
| 4105 | + bt_set_frame_stack_arg_slot(&env->bt, caller->frameno, spi); | |
| 4106 | + return mark_chain_precision_batch(env, env->cur_state); | |
| 4107 | +} | |
| 4108 | + | |
| 4109 | +static int check_outgoing_stack_args(struct bpf_verifier_env *env, struct bpf_func_state *caller, | |
| 4110 | + int nargs) | |
| 4111 | +{ | |
| 4112 | + int i, spi; | |
| 4113 | + | |
| 4114 | + for (i = MAX_BPF_FUNC_REG_ARGS; i < nargs; i++) { | |
| 4115 | + spi = i - MAX_BPF_FUNC_REG_ARGS; | |
| 4116 | + if (spi >= caller->out_stack_arg_cnt || | |
| 4117 | + caller->stack_arg_regs[spi].type == NOT_INIT) { | |
| 4118 | + verbose(env, "callee expects %d args, stack arg%d is not initialized\n", | |
| 4119 | + nargs, spi + 1); | |
| 4120 | + return -EFAULT; | |
| 4121 | + } | |
| 4122 | + } | |
| 4123 | + | |
| 4124 | + return 0; | |
| 4125 | +} | |
| 4126 | + | |
| 4127 | +static struct bpf_reg_state *get_func_arg_reg(struct bpf_func_state *caller, | |
| 4128 | + struct bpf_reg_state *regs, int arg) | |
| 4129 | +{ | |
| 4130 | + if (arg < MAX_BPF_FUNC_REG_ARGS) | |
| 4131 | + return ®s[arg + 1]; | |
| 4132 | + | |
| 4133 | + return &caller->stack_arg_regs[arg - MAX_BPF_FUNC_REG_ARGS]; | |
| 4134 | +} | |
| 4135 | + | |
| 4136 | +static int check_map_access_type(struct bpf_verifier_env *env, struct bpf_reg_state *reg, | |
| 4388 | 4137 | int off, int size, enum bpf_access_type type) |
| 4389 | 4138 | { |
| 4390 | − struct bpf_reg_state *reg = reg_state(env, regno); | |
| 4391 | 4139 | struct bpf_map *map = reg->map_ptr; |
| 4392 | 4140 | u32 cap = bpf_map_flags_to_cap(map); |
| 4393 | 4141 | |
| 4394 | 4142 | if (type == BPF_WRITE && !(cap & BPF_MAP_CAN_WRITE)) { |
| 4395 | 4143 | verbose(env, "write into map forbidden, value_size=%d off=%lld size=%d\n", |
| 4396 | − map->value_size, reg->smin_value + off, size); | |
| 4144 | + map->value_size, reg_smin(reg) + off, size); | |
| 4397 | 4145 | return -EACCES; |
| 4398 | 4146 | } |
| 4399 | 4147 | |
| 4400 | 4148 | if (type == BPF_READ && !(cap & BPF_MAP_CAN_READ)) { |
| 4401 | 4149 | verbose(env, "read from map forbidden, value_size=%d off=%lld size=%d\n", |
| 4402 | − map->value_size, reg->smin_value + off, size); | |
| 4150 | + map->value_size, reg_smin(reg) + off, size); | |
| 4403 | 4151 | return -EACCES; |
| 4404 | 4152 | } |
| 4405 | 4153 | |
| @@ -4407,17 +4155,15 @@ static int check_map_access_type(struct bpf_verifier_env *env, u32 regno, · 1 unmodified line | ||
| 4407 | 4155 | } |
| 4408 | 4156 | |
| 4409 | 4157 | /* check read/write into memory region (e.g., map value, ringbuf sample, etc) */ |
| 4410 | −static int __check_mem_access(struct bpf_verifier_env *env, int regno, | |
| 4158 | +static int __check_mem_access(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno, | |
| 4411 | 4159 | int off, int size, u32 mem_size, |
| 4412 | 4160 | bool zero_size_allowed) |
| 4413 | 4161 | { |
| 4414 | 4162 | bool size_ok = size > 0 || (size == 0 && zero_size_allowed); |
| 4415 | − struct bpf_reg_state *reg; | |
| 4416 | 4163 | |
| 4417 | 4164 | if (off >= 0 && size_ok && (u64)off + size <= mem_size) |
| 4418 | 4165 | return 0; |
| 4419 | 4166 | |
| 4420 | − reg = &cur_regs(env)[regno]; | |
| 4421 | 4167 | switch (reg->type) { |
| 4422 | 4168 | case PTR_TO_MAP_KEY: |
| 4423 | 4169 | verbose(env, "invalid access to map key, key_size=%d off=%d size=%d\n", |
| @@ -4430,8 +4176,8 @@ static int __check_mem_access(struct bpf_verifier_env *env, int regno, · 6 unmodified lines | ||
| 4430 | 4176 | case PTR_TO_PACKET: |
| 4431 | 4177 | case PTR_TO_PACKET_META: |
| 4432 | 4178 | case PTR_TO_PACKET_END: |
| 4433 | − verbose(env, "invalid access to packet, off=%d size=%d, R%d(id=%d,off=%d,r=%d)\n", | |
| 4434 | − off, size, regno, reg->id, off, mem_size); | |
| 4179 | + verbose(env, "invalid access to packet, off=%d size=%d, %s(id=%d,off=%d,r=%d)\n", | |
| 4180 | + off, size, reg_arg_name(env, argno), reg->id, off, mem_size); | |
| 4435 | 4181 | break; |
| 4436 | 4182 | case PTR_TO_CTX: |
| 4437 | 4183 | verbose(env, "invalid access to context, ctx_size=%d off=%d size=%d\n", |
| @@ -4447,13 +4193,10 @@ static int __check_mem_access(struct bpf_verifier_env *env, int regno, · 9 unmodified lines | ||
| 4447 | 4193 | } |
| 4448 | 4194 | |
| 4449 | 4195 | /* check read/write into a memory region with possible variable offset */ |
| 4450 | −static int check_mem_region_access(struct bpf_verifier_env *env, u32 regno, | |
| 4196 | +static int check_mem_region_access(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno, | |
| 4451 | 4197 | int off, int size, u32 mem_size, |
| 4452 | 4198 | bool zero_size_allowed) |
| 4453 | 4199 | { |
| 4454 | − struct bpf_verifier_state *vstate = env->cur_state; | |
| 4455 | − struct bpf_func_state *state = vstate->frame[vstate->curframe]; | |
| 4456 | − struct bpf_reg_state *reg = &state->regs[regno]; | |
| 4457 | 4200 | int err; |
| 4458 | 4201 | |
| 4459 | 4202 | /* We may have adjusted the register pointing to memory region, so we |
| @@ -4466,36 +4209,36 @@ static int check_mem_region_access(struct bpf_verifier_env *env, u32 regno, · 6 unmodified lines | ||
| 4466 | 4209 | * index'es we need to make sure that whatever we use |
| 4467 | 4210 | * will have a set floor within our range. |
| 4468 | 4211 | */ |
| 4469 | − if (reg->smin_value < 0 && | |
| 4470 | − (reg->smin_value == S64_MIN || | |
| 4471 | − (off + reg->smin_value != (s64)(s32)(off + reg->smin_value)) || | |
| 4472 | − reg->smin_value + off < 0)) { | |
| 4473 | − verbose(env, "R%d min value is negative, either use unsigned index or do a if (index >=0) check.\n", | |
| 4474 | − regno); | |
| 4212 | + if (reg_smin(reg) < 0 && | |
| 4213 | + (reg_smin(reg) == S64_MIN || | |
| 4214 | + (off + reg_smin(reg) != (s64)(s32)(off + reg_smin(reg))) || | |
| 4215 | + reg_smin(reg) + off < 0)) { | |
| 4216 | + verbose(env, "%s min value is negative, either use unsigned index or do a if (index >=0) check.\n", | |
| 4217 | + reg_arg_name(env, argno)); | |
| 4475 | 4218 | return -EACCES; |
| 4476 | 4219 | } |
| 4477 | − err = __check_mem_access(env, regno, reg->smin_value + off, size, | |
| 4220 | + err = __check_mem_access(env, reg, argno, reg_smin(reg) + off, size, | |
| 4478 | 4221 | mem_size, zero_size_allowed); |
| 4479 | 4222 | if (err) { |
| 4480 | − verbose(env, "R%d min value is outside of the allowed memory range\n", | |
| 4481 | − regno); | |
| 4223 | + verbose(env, "%s min value is outside of the allowed memory range\n", | |
| 4224 | + reg_arg_name(env, argno)); | |
| 4482 | 4225 | return err; |
| 4483 | 4226 | } |
| 4484 | 4227 | |
| 4485 | 4228 | /* If we haven't set a max value then we need to bail since we can't be |
| 4486 | 4229 | * sure we won't do bad things. |
| 4487 | − * If reg->umax_value + off could overflow, treat that as unbounded too. | |
| 4230 | + * If reg_umax(reg) + off could overflow, treat that as unbounded too. | |
| 4488 | 4231 | */ |
| 4489 | − if (reg->umax_value >= BPF_MAX_VAR_OFF) { | |
| 4490 | − verbose(env, "R%d unbounded memory access, make sure to bounds check any such access\n", | |
| 4491 | − regno); | |
| 4232 | + if (reg_umax(reg) >= BPF_MAX_VAR_OFF) { | |
| 4233 | + verbose(env, "%s unbounded memory access, make sure to bounds check any such access\n", | |
| 4234 | + reg_arg_name(env, argno)); | |
| 4492 | 4235 | return -EACCES; |
| 4493 | 4236 | } |
| 4494 | − err = __check_mem_access(env, regno, reg->umax_value + off, size, | |
| 4237 | + err = __check_mem_access(env, reg, argno, reg_umax(reg) + off, size, | |
| 4495 | 4238 | mem_size, zero_size_allowed); |
| 4496 | 4239 | if (err) { |
| 4497 | − verbose(env, "R%d max value is outside of the allowed memory range\n", | |
| 4498 | − regno); | |
| 4240 | + verbose(env, "%s max value is outside of the allowed memory range\n", | |
| 4241 | + reg_arg_name(env, argno)); | |
| 4499 | 4242 | return err; |
| 4500 | 4243 | } |
| 4501 | 4244 | |
| @@ -4503,7 +4246,7 @@ static int check_mem_region_access(struct bpf_verifier_env *env, u32 regno, · 1 unmodified line | ||
| 4503 | 4246 | } |
| 4504 | 4247 | |
| 4505 | 4248 | static int __check_ptr_off_reg(struct bpf_verifier_env *env, |
| 4506 | − const struct bpf_reg_state *reg, int regno, | |
| 4249 | + const struct bpf_reg_state *reg, argno_t argno, | |
| 4507 | 4250 | bool fixed_off_ok) |
| 4508 | 4251 | { |
| 4509 | 4252 | /* Access to this pointer-typed register or passing it to a helper |
| @@ -4519,15 +4262,15 @@ static int __check_ptr_off_reg(struct bpf_verifier_env *env, · 9 unmodified lines | ||
| 4519 | 4262 | return -EACCES; |
| 4520 | 4263 | } |
| 4521 | 4264 | |
| 4522 | − if (reg->smin_value < 0) { | |
| 4523 | − verbose(env, "negative offset %s ptr R%d off=%lld disallowed\n", | |
| 4524 | − reg_type_str(env, reg->type), regno, reg->var_off.value); | |
| 4265 | + if (reg_smin(reg) < 0) { | |
| 4266 | + verbose(env, "negative offset %s ptr %s off=%lld disallowed\n", | |
| 4267 | + reg_type_str(env, reg->type), reg_arg_name(env, argno), reg->var_off.value); | |
| 4525 | 4268 | return -EACCES; |
| 4526 | 4269 | } |
| 4527 | 4270 | |
| 4528 | 4271 | if (!fixed_off_ok && reg->var_off.value != 0) { |
| 4529 | − verbose(env, "dereference of modified %s ptr R%d off=%lld disallowed\n", | |
| 4530 | − reg_type_str(env, reg->type), regno, reg->var_off.value); | |
| 4272 | + verbose(env, "dereference of modified %s ptr %s off=%lld disallowed\n", | |
| 4273 | + reg_type_str(env, reg->type), reg_arg_name(env, argno), reg->var_off.value); | |
| 4531 | 4274 | return -EACCES; |
| 4532 | 4275 | } |
| 4533 | 4276 | |
| @@ -4537,7 +4280,7 @@ static int __check_ptr_off_reg(struct bpf_verifier_env *env, · 3 unmodified lines | ||
| 4537 | 4280 | static int check_ptr_off_reg(struct bpf_verifier_env *env, |
| 4538 | 4281 | const struct bpf_reg_state *reg, int regno) |
| 4539 | 4282 | { |
| 4540 | − return __check_ptr_off_reg(env, reg, regno, false); | |
| 4283 | + return __check_ptr_off_reg(env, reg, argno_from_reg(regno), false); | |
| 4541 | 4284 | } |
| 4542 | 4285 | |
| 4543 | 4286 | static int map_kptr_match_type(struct bpf_verifier_env *env, |
| @@ -4573,9 +4316,9 @@ static int map_kptr_match_type(struct bpf_verifier_env *env, · 29 unmodified lines | ||
| 4573 | 4316 | * referenced PTR_TO_BTF_ID, and that its fixed offset is 0. For the |
| 4574 | 4317 | * normal store of unreferenced kptr, we must ensure var_off is zero. |
| 4575 | 4318 | * Since ref_ptr cannot be accessed directly by BPF insns, check for |
| 4576 | − * reg->ref_obj_id is not needed here. | |
| 4319 | + * reg->id is not needed here. | |
| 4577 | 4320 | */ |
| 4578 | − if (__check_ptr_off_reg(env, reg, regno, true)) | |
| 4321 | + if (__check_ptr_off_reg(env, reg, argno_from_reg(regno), true)) | |
| 4579 | 4322 | return -EACCES; |
| 4580 | 4323 | |
| 4581 | 4324 | /* A full type match is needed, as BTF can be vmlinux, module or prog BTF, and |
| @@ -4718,7 +4461,7 @@ static int mark_uptr_ld_reg(struct bpf_verifier_env *env, u32 regno, · 136 unmodified lines | ||
| 4718 | 4461 | return 0; |
| 4719 | 4462 | } |
| 4720 | 4463 | |
| 4721 | −static int check_map_kptr_access(struct bpf_verifier_env *env, u32 regno, | |
| 4464 | +static int check_map_kptr_access(struct bpf_verifier_env *env, | |
| 4722 | 4465 | int value_regno, int insn_idx, |
| 4723 | 4466 | struct btf_field *kptr_field) |
| 4724 | 4467 | { |
| @@ -4795,19 +4538,16 @@ static u32 map_mem_size(const struct bpf_map *map) · 70 unmodified lines | ||
| 4795 | 4538 | } |
| 4796 | 4539 | |
| 4797 | 4540 | /* check read/write into a map element with possible variable offset */ |
| 4798 | −static int check_map_access(struct bpf_verifier_env *env, u32 regno, | |
| 4541 | +static int check_map_access(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno, | |
| 4799 | 4542 | int off, int size, bool zero_size_allowed, |
| 4800 | 4543 | enum bpf_access_src src) |
| 4801 | 4544 | { |
| 4802 | − struct bpf_verifier_state *vstate = env->cur_state; | |
| 4803 | − struct bpf_func_state *state = vstate->frame[vstate->curframe]; | |
| 4804 | − struct bpf_reg_state *reg = &state->regs[regno]; | |
| 4805 | 4545 | struct bpf_map *map = reg->map_ptr; |
| 4806 | 4546 | u32 mem_size = map_mem_size(map); |
| 4807 | 4547 | struct btf_record *rec; |
| 4808 | 4548 | int err, i; |
| 4809 | 4549 | |
| 4810 | − err = check_mem_region_access(env, regno, off, size, mem_size, zero_size_allowed); | |
| 4550 | + err = check_mem_region_access(env, reg, argno, off, size, mem_size, zero_size_allowed); | |
| 4811 | 4551 | if (err) |
| 4812 | 4552 | return err; |
| 4813 | 4553 | |
| @@ -4822,8 +4562,8 @@ static int check_map_access(struct bpf_verifier_env *env, u32 regno, · 8 unmodified lines | ||
| 4822 | 4562 | * this program. To check that [x1, x2) overlaps with [y1, y2), |
| 4823 | 4563 | * it is sufficient to check x1 < y2 && y1 < x2. |
| 4824 | 4564 | */ |
| 4825 | − if (reg->smin_value + off < p + field->size && | |
| 4826 | − p < reg->umax_value + off + size) { | |
| 4565 | + if (reg_smin(reg) + off < p + field->size && | |
| 4566 | + p < reg_umax(reg) + off + size) { | |
| 4827 | 4567 | switch (field->type) { |
| 4828 | 4568 | case BPF_KPTR_UNREF: |
| 4829 | 4569 | case BPF_KPTR_REF: |
| @@ -4903,30 +4643,29 @@ static bool may_access_direct_pkt_data(struct bpf_verifier_env *env, · 73 unmodified lines | ||
| 4903 | 4643 | } |
| 4904 | 4644 | } |
| 4905 | 4645 | |
| 4906 | −static int check_packet_access(struct bpf_verifier_env *env, u32 regno, int off, | |
| 4646 | +static int check_packet_access(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno, int off, | |
| 4907 | 4647 | int size, bool zero_size_allowed) |
| 4908 | 4648 | { |
| 4909 | − struct bpf_reg_state *reg = reg_state(env, regno); | |
| 4910 | 4649 | int err; |
| 4911 | 4650 | |
| 4912 | 4651 | if (reg->range < 0) { |
| 4913 | − verbose(env, "R%d offset is outside of the packet\n", regno); | |
| 4652 | + verbose(env, "%s offset is outside of the packet\n", reg_arg_name(env, argno)); | |
| 4914 | 4653 | return -EINVAL; |
| 4915 | 4654 | } |
| 4916 | 4655 | |
| 4917 | − err = check_mem_region_access(env, regno, off, size, reg->range, zero_size_allowed); | |
| 4656 | + err = check_mem_region_access(env, reg, argno, off, size, reg->range, zero_size_allowed); | |
| 4918 | 4657 | if (err) |
| 4919 | 4658 | return err; |
| 4920 | 4659 | |
| 4921 | 4660 | /* __check_mem_access has made sure "off + size - 1" is within u16. |
| 4922 | − * reg->umax_value can't be bigger than MAX_PACKET_OFF which is 0xffff, | |
| 4661 | + * reg_umax(reg) can't be bigger than MAX_PACKET_OFF which is 0xffff, | |
| 4923 | 4662 | * otherwise find_good_pkt_pointers would have refused to set range info |
| 4924 | 4663 | * that __check_mem_access would have rejected this pkt access. |
| 4925 | − * Therefore, "off + reg->umax_value + size - 1" won't overflow u32. | |
| 4664 | + * Therefore, "off + reg_umax(reg) + size - 1" won't overflow u32. | |
| 4926 | 4665 | */ |
| 4927 | 4666 | env->prog->aux->max_pkt_offset = |
| 4928 | 4667 | max_t(u32, env->prog->aux->max_pkt_offset, |
| 4929 | − off + reg->umax_value + size - 1); | |
| 4668 | + off + reg_umax(reg) + size - 1); | |
| 4930 | 4669 | |
| 4931 | 4670 | return 0; |
| 4932 | 4671 | } |
| @@ -4950,8 +4689,8 @@ static int __check_ctx_access(struct bpf_verifier_env *env, int insn_idx, int of · 17 unmodified lines | ||
| 4950 | 4689 | * type of narrower access. |
| 4951 | 4690 | */ |
| 4952 | 4691 | if (base_type(info->reg_type) == PTR_TO_BTF_ID) { |
| 4953 | − if (info->ref_obj_id && | |
| 4954 | − !find_reference_state(env->cur_state, info->ref_obj_id)) { | |
| 4692 | + if (info->ref_id && | |
| 4693 | + !find_reference_state(env->cur_state, info->ref_id)) { | |
| 4955 | 4694 | verbose(env, "invalid bpf_context access off=%d. Reference may already be released\n", |
| 4956 | 4695 | off); |
| 4957 | 4696 | return -EACCES; |
| @@ -4969,7 +4708,7 @@ static int __check_ctx_access(struct bpf_verifier_env *env, int insn_idx, int of · 11 unmodified lines | ||
| 4969 | 4708 | return -EACCES; |
| 4970 | 4709 | } |
| 4971 | 4710 | |
| 4972 | −static int check_ctx_access(struct bpf_verifier_env *env, int insn_idx, u32 regno, | |
| 4711 | +static int check_ctx_access(struct bpf_verifier_env *env, int insn_idx, struct bpf_reg_state *reg, argno_t argno, | |
| 4973 | 4712 | int off, int access_size, enum bpf_access_type t, |
| 4974 | 4713 | struct bpf_insn_access_aux *info) |
| 4975 | 4714 | { |
| @@ -4979,17 +4718,15 @@ static int check_ctx_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 3 unmodified lines | ||
| 4979 | 4718 | */ |
| 4980 | 4719 | bool var_off_ok = is_var_ctx_off_allowed(env->prog); |
| 4981 | 4720 | bool fixed_off_ok = !env->ops->convert_ctx_access; |
| 4982 | − struct bpf_reg_state *regs = cur_regs(env); | |
| 4983 | − struct bpf_reg_state *reg = regs + regno; | |
| 4984 | 4721 | int err; |
| 4985 | 4722 | |
| 4986 | 4723 | if (var_off_ok) |
| 4987 | − err = check_mem_region_access(env, regno, off, access_size, U16_MAX, false); | |
| 4724 | + err = check_mem_region_access(env, reg, argno, off, access_size, U16_MAX, false); | |
| 4988 | 4725 | else |
| 4989 | − err = __check_ptr_off_reg(env, reg, regno, fixed_off_ok); | |
| 4726 | + err = __check_ptr_off_reg(env, reg, argno, fixed_off_ok); | |
| 4990 | 4727 | if (err) |
| 4991 | 4728 | return err; |
| 4992 | − off += reg->umax_value; | |
| 4729 | + off += reg_umax(reg); | |
| 4993 | 4730 | |
| 4994 | 4731 | err = __check_ctx_access(env, insn_idx, off, access_size, t, info); |
| 4995 | 4732 | if (err) |
| @@ -4997,9 +4734,21 @@ static int check_ctx_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 1 unmodified line | ||
| 4997 | 4734 | return err; |
| 4998 | 4735 | } |
| 4999 | 4736 | |
| 5000 | −static int check_flow_keys_access(struct bpf_verifier_env *env, int off, | |
| 5001 | − int size) | |
| 4737 | +static int check_flow_keys_access(struct bpf_verifier_env *env, | |
| 4738 | + struct bpf_reg_state *reg, argno_t argno, | |
| 4739 | + int off, int size) | |
| 5002 | 4740 | { |
| 4741 | + /* Only a constant offset is allowed here; fold it into off. */ | |
| 4742 | + if (!tnum_is_const(reg->var_off)) { | |
| 4743 | + char tn_buf[48]; | |
| 4744 | + | |
| 4745 | + tnum_strn(tn_buf, sizeof(tn_buf), reg->var_off); | |
| 4746 | + verbose(env, "%s invalid variable offset to flow keys: off=%d, var_off=%s\n", | |
| 4747 | + reg_arg_name(env, argno), off, tn_buf); | |
| 4748 | + return -EACCES; | |
| 4749 | + } | |
| 4750 | + off += reg->var_off.value; | |
| 4751 | + | |
| 5003 | 4752 | if (size < 0 || off < 0 || |
| 5004 | 4753 | (u64)off + size > sizeof(struct bpf_flow_keys)) { |
| 5005 | 4754 | verbose(env, "invalid access to flow keys off=%d size=%d\n", |
| @@ -5010,16 +4759,15 @@ static int check_flow_keys_access(struct bpf_verifier_env *env, int off, · 4 unmodified lines | ||
| 5010 | 4759 | } |
| 5011 | 4760 | |
| 5012 | 4761 | static int check_sock_access(struct bpf_verifier_env *env, int insn_idx, |
| 5013 | − u32 regno, int off, int size, | |
| 4762 | + struct bpf_reg_state *reg, argno_t argno, int off, int size, | |
| 5014 | 4763 | enum bpf_access_type t) |
| 5015 | 4764 | { |
| 5016 | − struct bpf_reg_state *reg = reg_state(env, regno); | |
| 5017 | 4765 | struct bpf_insn_access_aux info = {}; |
| 5018 | 4766 | bool valid; |
| 5019 | 4767 | |
| 5020 | − if (reg->smin_value < 0) { | |
| 5021 | − verbose(env, "R%d min value is negative, either use unsigned index or do a if (index >=0) check.\n", | |
| 5022 | − regno); | |
| 4768 | + if (reg_smin(reg) < 0) { | |
| 4769 | + verbose(env, "%s min value is negative, either use unsigned index or do a if (index >=0) check.\n", | |
| 4770 | + reg_arg_name(env, argno)); | |
| 5023 | 4771 | return -EACCES; |
| 5024 | 4772 | } |
| 5025 | 4773 | |
| @@ -5047,8 +4795,8 @@ static int check_sock_access(struct bpf_verifier_env *env, int insn_idx, · 21 unmodified lines | ||
| 5047 | 4795 | return 0; |
| 5048 | 4796 | } |
| 5049 | 4797 | |
| 5050 | − verbose(env, "R%d invalid %s access off=%d size=%d\n", | |
| 5051 | − regno, reg_type_str(env, reg->type), off, size); | |
| 4798 | + verbose(env, "%s invalid %s access off=%d size=%d\n", | |
| 4799 | + reg_arg_name(env, argno), reg_type_str(env, reg->type), off, size); | |
| 5052 | 4800 | |
| 5053 | 4801 | return -EACCES; |
| 5054 | 4802 | } |
| @@ -5123,10 +4871,10 @@ static u32 *reg2btf_ids[__BPF_REG_TYPE_MAX] = { · 68 unmodified lines | ||
| 5123 | 4871 | [CONST_PTR_TO_MAP] = btf_bpf_map_id, |
| 5124 | 4872 | }; |
| 5125 | 4873 | |
| 5126 | −static bool is_trusted_reg(const struct bpf_reg_state *reg) | |
| 4874 | +static bool is_trusted_reg(struct bpf_verifier_env *env, const struct bpf_reg_state *reg) | |
| 5127 | 4875 | { |
| 5128 | 4876 | /* A referenced register is always trusted. */ |
| 5129 | − if (reg->ref_obj_id) | |
| 4877 | + if (reg_is_referenced(env, reg)) | |
| 5130 | 4878 | return true; |
| 5131 | 4879 | |
| 5132 | 4880 | /* Types listed in the reg2btf_ids are always trusted */ |
| @@ -5368,7 +5116,10 @@ process_func: · 235 unmodified lines | ||
| 5368 | 5116 | } |
| 5369 | 5117 | |
| 5370 | 5118 | subprog_depth = round_up_stack_depth(env, subprog[idx].stack_depth); |
| 5371 | − if (priv_stack_supported) { | |
| 5119 | + if (IS_ENABLED(CONFIG_X86_64) && subprog[idx].stack_arg_cnt) { | |
| 5120 | + /* x86-64 uses R9 for both private stack frame pointer and arg6. */ | |
| 5121 | + subprog[idx].priv_stack_mode = NO_PRIV_STACK; | |
| 5122 | + } else if (priv_stack_supported) { | |
| 5372 | 5123 | /* Request private stack support only if the subprog stack |
| 5373 | 5124 | * depth is no less than BPF_PRIV_STACK_MIN_SIZE. This is to |
| 5374 | 5125 | * avoid jit penalty if the stack usage is small. |
| @@ -5379,6 +5130,8 @@ process_func: · 4 unmodified lines | ||
| 5379 | 5130 | } |
| 5380 | 5131 | |
| 5381 | 5132 | if (subprog[idx].priv_stack_mode == PRIV_STACK_ADAPTIVE) { |
| 5133 | + if (subprog_depth > env->max_stack_depth) | |
| 5134 | + env->max_stack_depth = subprog_depth; | |
| 5382 | 5135 | if (subprog_depth > MAX_BPF_STACK) { |
| 5383 | 5136 | verbose(env, "stack size of subprog %d is %d. Too large\n", |
| 5384 | 5137 | idx, subprog_depth); |
| @@ -5386,6 +5139,8 @@ process_func: · 1 unmodified line | ||
| 5386 | 5139 | } |
| 5387 | 5140 | } else { |
| 5388 | 5141 | depth += subprog_depth; |
| 5142 | + if (depth > env->max_stack_depth) | |
| 5143 | + env->max_stack_depth = depth; | |
| 5389 | 5144 | if (depth > MAX_BPF_STACK) { |
| 5390 | 5145 | total = 0; |
| 5391 | 5146 | for (tmp = idx; tmp >= 0; tmp = dinfo[tmp].caller) |
| @@ -5472,14 +5227,23 @@ continue_func: · 80 unmodified lines | ||
| 5472 | 5227 | * this info will be utilized by JIT so that we will be preserving the |
| 5473 | 5228 | * tail call counter throughout bpf2bpf calls combined with tailcalls |
| 5474 | 5229 | */ |
| 5475 | − if (tail_call_reachable) | |
| 5230 | + if (tail_call_reachable) { | |
| 5476 | 5231 | for (tmp = idx; tmp >= 0; tmp = dinfo[tmp].caller) { |
| 5477 | 5232 | if (subprog[tmp].is_exception_cb) { |
| 5478 | 5233 | verbose(env, "cannot tail call within exception cb\n"); |
| 5479 | 5234 | return -EINVAL; |
| 5480 | 5235 | } |
| 5236 | + if (subprog[tmp].stack_arg_cnt) { | |
| 5237 | + verbose(env, "tail_calls are not allowed in programs with stack args\n"); | |
| 5238 | + return -EINVAL; | |
| 5239 | + } | |
| 5481 | 5240 | subprog[tmp].tail_call_reachable = true; |
| 5482 | 5241 | } |
| 5242 | + } else if (!idx && subprog[0].has_tail_call && subprog[0].stack_arg_cnt) { | |
| 5243 | + verbose(env, "tail_calls are not allowed in programs with stack args\n"); | |
| 5244 | + return -EINVAL; | |
| 5245 | + } | |
| 5246 | + | |
| 5483 | 5247 | if (subprog[0].tail_call_reachable) |
| 5484 | 5248 | env->prog->aux->tail_call_reachable = true; |
| 5485 | 5249 | |
| @@ -5498,6 +5262,9 @@ continue_func: · 12 unmodified lines | ||
| 5498 | 5262 | frame = dinfo[idx].frame; |
| 5499 | 5263 | i = dinfo[idx].ret_insn; |
| 5500 | 5264 | |
| 5265 | + /* reset tail_call_reachable to the parent's actual state */ | |
| 5266 | + tail_call_reachable = subprog[idx].tail_call_reachable; | |
| 5267 | + | |
| 5501 | 5268 | goto continue_func; |
| 5502 | 5269 | } |
| 5503 | 5270 | |
| @@ -5558,12 +5325,12 @@ static int check_max_stack_depth(struct bpf_verifier_env *env) · 54 unmodified lines | ||
| 5558 | 5325 | static int __check_buffer_access(struct bpf_verifier_env *env, |
| 5559 | 5326 | const char *buf_info, |
| 5560 | 5327 | const struct bpf_reg_state *reg, |
| 5561 | − int regno, int off, int size) | |
| 5328 | + argno_t argno, int off, int size) | |
| 5562 | 5329 | { |
| 5563 | 5330 | if (off < 0) { |
| 5564 | 5331 | verbose(env, |
| 5565 | − "R%d invalid %s buffer access: off=%d, size=%d\n", | |
| 5566 | − regno, buf_info, off, size); | |
| 5332 | + "%s invalid %s buffer access: off=%d, size=%d\n", | |
| 5333 | + reg_arg_name(env, argno), buf_info, off, size); | |
| 5567 | 5334 | return -EACCES; |
| 5568 | 5335 | } |
| 5569 | 5336 | if (!tnum_is_const(reg->var_off)) { |
| @@ -5571,8 +5338,8 @@ static int __check_buffer_access(struct bpf_verifier_env *env, · 1 unmodified line | ||
| 5571 | 5338 | |
| 5572 | 5339 | tnum_strn(tn_buf, sizeof(tn_buf), reg->var_off); |
| 5573 | 5340 | verbose(env, |
| 5574 | − "R%d invalid variable buffer offset: off=%d, var_off=%s\n", | |
| 5575 | − regno, off, tn_buf); | |
| 5341 | + "%s invalid variable buffer offset: off=%d, var_off=%s\n", | |
| 5342 | + reg_arg_name(env, argno), off, tn_buf); | |
| 5576 | 5343 | return -EACCES; |
| 5577 | 5344 | } |
| 5578 | 5345 | |
| @@ -5581,11 +5348,11 @@ static int __check_buffer_access(struct bpf_verifier_env *env, · 2 unmodified lines | ||
| 5581 | 5348 | |
| 5582 | 5349 | static int check_tp_buffer_access(struct bpf_verifier_env *env, |
| 5583 | 5350 | const struct bpf_reg_state *reg, |
| 5584 | − int regno, int off, int size) | |
| 5351 | + argno_t argno, int off, int size) | |
| 5585 | 5352 | { |
| 5586 | 5353 | int err; |
| 5587 | 5354 | |
| 5588 | − err = __check_buffer_access(env, "tracepoint", reg, regno, off, size); | |
| 5355 | + err = __check_buffer_access(env, "tracepoint", reg, argno, off, size); | |
| 5589 | 5356 | if (err) |
| 5590 | 5357 | return err; |
| 5591 | 5358 | |
| @@ -5597,14 +5364,14 @@ static int check_tp_buffer_access(struct bpf_verifier_env *env, · 5 unmodified lines | ||
| 5597 | 5364 | |
| 5598 | 5365 | static int check_buffer_access(struct bpf_verifier_env *env, |
| 5599 | 5366 | const struct bpf_reg_state *reg, |
| 5600 | − int regno, int off, int size, | |
| 5367 | + argno_t argno, int off, int size, | |
| 5601 | 5368 | bool zero_size_allowed, |
| 5602 | 5369 | u32 *max_access) |
| 5603 | 5370 | { |
| 5604 | 5371 | const char *buf_info = type_is_rdonly_mem(reg->type) ? "rdonly" : "rdwr"; |
| 5605 | 5372 | int err; |
| 5606 | 5373 | |
| 5607 | − err = __check_buffer_access(env, buf_info, reg, regno, off, size); | |
| 5374 | + err = __check_buffer_access(env, buf_info, reg, argno, off, size); | |
| 5608 | 5375 | if (err) |
| 5609 | 5376 | return err; |
| 5610 | 5377 | |
| @@ -5617,7 +5384,7 @@ static int check_buffer_access(struct bpf_verifier_env *env, · 6 unmodified lines | ||
| 5617 | 5384 | static void zext_32_to_64(struct bpf_reg_state *reg) |
| 5618 | 5385 | { |
| 5619 | 5386 | reg->var_off = tnum_subreg(reg->var_off); |
| 5620 | − __reg_assign_32_into_64(reg); | |
| 5387 | + reg_set_urange64(reg, reg_u32_min(reg), reg_u32_max(reg)); | |
| 5621 | 5388 | } |
| 5622 | 5389 | |
| 5623 | 5390 | /* truncate register to smaller size (in bytes) |
| @@ -5632,15 +5399,10 @@ static void coerce_reg_to_size(struct bpf_reg_state *reg, int size) · 8 unmodified lines | ||
| 5632 | 5399 | |
| 5633 | 5400 | /* fix arithmetic bounds */ |
| 5634 | 5401 | mask = ((u64)1 << (size * 8)) - 1; |
| 5635 | − if ((reg->umin_value & ~mask) == (reg->umax_value & ~mask)) { | |
| 5636 | − reg->umin_value &= mask; | |
| 5637 | − reg->umax_value &= mask; | |
| 5638 | − } else { | |
| 5639 | − reg->umin_value = 0; | |
| 5640 | − reg->umax_value = mask; | |
| 5641 | − } | |
| 5642 | − reg->smin_value = reg->umin_value; | |
| 5643 | − reg->smax_value = reg->umax_value; | |
| 5402 | + if ((reg_umin(reg) & ~mask) == (reg_umax(reg) & ~mask)) | |
| 5403 | + reg_set_urange64(reg, reg_umin(reg) & mask, reg_umax(reg) & mask); | |
| 5404 | + else | |
| 5405 | + reg_set_urange64(reg, 0, mask); | |
| 5644 | 5406 | |
| 5645 | 5407 | /* If size is smaller than 32bit register the 32bit register |
| 5646 | 5408 | * values are also truncated so we push 64-bit bounds into |
| @@ -5655,19 +5417,16 @@ static void coerce_reg_to_size(struct bpf_reg_state *reg, int size) · 8 unmodified lines | ||
| 5655 | 5417 | static void set_sext64_default_val(struct bpf_reg_state *reg, int size) |
| 5656 | 5418 | { |
| 5657 | 5419 | if (size == 1) { |
| 5658 | − reg->smin_value = reg->s32_min_value = S8_MIN; | |
| 5659 | − reg->smax_value = reg->s32_max_value = S8_MAX; | |
| 5420 | + reg_set_srange64(reg, S8_MIN, S8_MAX); | |
| 5421 | + reg_set_srange32(reg, S8_MIN, S8_MAX); | |
| 5660 | 5422 | } else if (size == 2) { |
| 5661 | − reg->smin_value = reg->s32_min_value = S16_MIN; | |
| 5662 | − reg->smax_value = reg->s32_max_value = S16_MAX; | |
| 5423 | + reg_set_srange64(reg, S16_MIN, S16_MAX); | |
| 5424 | + reg_set_srange32(reg, S16_MIN, S16_MAX); | |
| 5663 | 5425 | } else { |
| 5664 | 5426 | /* size == 4 */ |
| 5665 | − reg->smin_value = reg->s32_min_value = S32_MIN; | |
| 5666 | − reg->smax_value = reg->s32_max_value = S32_MAX; | |
| 5427 | + reg_set_srange64(reg, S32_MIN, S32_MAX); | |
| 5428 | + reg_set_srange32(reg, S32_MIN, S32_MAX); | |
| 5667 | 5429 | } |
| 5668 | − reg->umin_value = reg->u32_min_value = 0; | |
| 5669 | − reg->umax_value = U64_MAX; | |
| 5670 | − reg->u32_max_value = U32_MAX; | |
| 5671 | 5430 | reg->var_off = tnum_unknown; |
| 5672 | 5431 | } |
| 5673 | 5432 | |
| @@ -5688,29 +5447,27 @@ static void coerce_reg_to_size_sx(struct bpf_reg_state *reg, int size) · 14 unmodified lines | ||
| 5688 | 5447 | reg->var_off = tnum_const((s32)u64_cval); |
| 5689 | 5448 | |
| 5690 | 5449 | u64_cval = reg->var_off.value; |
| 5691 | − reg->smax_value = reg->smin_value = u64_cval; | |
| 5692 | − reg->umax_value = reg->umin_value = u64_cval; | |
| 5693 | − reg->s32_max_value = reg->s32_min_value = u64_cval; | |
| 5694 | − reg->u32_max_value = reg->u32_min_value = u64_cval; | |
| 5450 | + reg->r64 = cnum64_from_urange(u64_cval, u64_cval); | |
| 5451 | + reg->r32 = cnum32_from_urange((u32)u64_cval, (u32)u64_cval); | |
| 5695 | 5452 | return; |
| 5696 | 5453 | } |
| 5697 | 5454 | |
| 5698 | − top_smax_value = ((u64)reg->smax_value >> num_bits) << num_bits; | |
| 5699 | − top_smin_value = ((u64)reg->smin_value >> num_bits) << num_bits; | |
| 5455 | + top_smax_value = ((u64)reg_smax(reg) >> num_bits) << num_bits; | |
| 5456 | + top_smin_value = ((u64)reg_smin(reg) >> num_bits) << num_bits; | |
| 5700 | 5457 | |
| 5701 | 5458 | if (top_smax_value != top_smin_value) |
| 5702 | 5459 | goto out; |
| 5703 | 5460 | |
| 5704 | 5461 | /* find the s64_min and s64_min after sign extension */ |
| 5705 | 5462 | if (size == 1) { |
| 5706 | − init_s64_max = (s8)reg->smax_value; | |
| 5707 | − init_s64_min = (s8)reg->smin_value; | |
| 5463 | + init_s64_max = (s8)reg_smax(reg); | |
| 5464 | + init_s64_min = (s8)reg_smin(reg); | |
| 5708 | 5465 | } else if (size == 2) { |
| 5709 | − init_s64_max = (s16)reg->smax_value; | |
| 5710 | − init_s64_min = (s16)reg->smin_value; | |
| 5466 | + init_s64_max = (s16)reg_smax(reg); | |
| 5467 | + init_s64_min = (s16)reg_smin(reg); | |
| 5711 | 5468 | } else { |
| 5712 | − init_s64_max = (s32)reg->smax_value; | |
| 5713 | − init_s64_min = (s32)reg->smin_value; | |
| 5469 | + init_s64_max = (s32)reg_smax(reg); | |
| 5470 | + init_s64_min = (s32)reg_smin(reg); | |
| 5714 | 5471 | } |
| 5715 | 5472 | |
| 5716 | 5473 | s64_max = max(init_s64_max, init_s64_min); |
| @@ -5718,10 +5475,8 @@ static void coerce_reg_to_size_sx(struct bpf_reg_state *reg, int size) · 1 unmodified line | ||
| 5718 | 5475 | |
| 5719 | 5476 | /* both of s64_max/s64_min positive or negative */ |
| 5720 | 5477 | if ((s64_max >= 0) == (s64_min >= 0)) { |
| 5721 | − reg->s32_min_value = reg->smin_value = s64_min; | |
| 5722 | − reg->s32_max_value = reg->smax_value = s64_max; | |
| 5723 | − reg->u32_min_value = reg->umin_value = s64_min; | |
| 5724 | − reg->u32_max_value = reg->umax_value = s64_max; | |
| 5478 | + reg_set_srange64(reg, s64_min, s64_max); | |
| 5479 | + reg_set_srange32(reg, s64_min, s64_max); | |
| 5725 | 5480 | reg->var_off = tnum_range(s64_min, s64_max); |
| 5726 | 5481 | return; |
| 5727 | 5482 | } |
| @@ -5732,16 +5487,11 @@ out: · 4 unmodified lines | ||
| 5732 | 5487 | |
| 5733 | 5488 | static void set_sext32_default_val(struct bpf_reg_state *reg, int size) |
| 5734 | 5489 | { |
| 5735 | − if (size == 1) { | |
| 5736 | − reg->s32_min_value = S8_MIN; | |
| 5737 | − reg->s32_max_value = S8_MAX; | |
| 5738 | − } else { | |
| 5490 | + if (size == 1) | |
| 5491 | + reg_set_srange32(reg, S8_MIN, S8_MAX); | |
| 5492 | + else | |
| 5739 | 5493 | /* size == 2 */ |
| 5740 | − reg->s32_min_value = S16_MIN; | |
| 5741 | − reg->s32_max_value = S16_MAX; | |
| 5742 | − } | |
| 5743 | − reg->u32_min_value = 0; | |
| 5744 | − reg->u32_max_value = U32_MAX; | |
| 5494 | + reg_set_srange32(reg, S16_MIN, S16_MAX); | |
| 5745 | 5495 | reg->var_off = tnum_subreg(tnum_unknown); |
| 5746 | 5496 | } |
| 5747 | 5497 | |
| @@ -5759,34 +5509,30 @@ static void coerce_subreg_to_size_sx(struct bpf_reg_state *reg, int size) · 11 unmodified lines | ||
| 5759 | 5509 | reg->var_off = tnum_const((s16)u32_val); |
| 5760 | 5510 | |
| 5761 | 5511 | u32_val = reg->var_off.value; |
| 5762 | − reg->s32_min_value = reg->s32_max_value = u32_val; | |
| 5763 | − reg->u32_min_value = reg->u32_max_value = u32_val; | |
| 5512 | + reg_set_srange32(reg, u32_val, u32_val); | |
| 5764 | 5513 | return; |
| 5765 | 5514 | } |
| 5766 | 5515 | |
| 5767 | − top_smax_value = ((u32)reg->s32_max_value >> num_bits) << num_bits; | |
| 5768 | − top_smin_value = ((u32)reg->s32_min_value >> num_bits) << num_bits; | |
| 5516 | + top_smax_value = ((u32)reg_s32_max(reg) >> num_bits) << num_bits; | |
| 5517 | + top_smin_value = ((u32)reg_s32_min(reg) >> num_bits) << num_bits; | |
| 5769 | 5518 | |
| 5770 | 5519 | if (top_smax_value != top_smin_value) |
| 5771 | 5520 | goto out; |
| 5772 | 5521 | |
| 5773 | 5522 | /* find the s32_min and s32_min after sign extension */ |
| 5774 | 5523 | if (size == 1) { |
| 5775 | − init_s32_max = (s8)reg->s32_max_value; | |
| 5776 | − init_s32_min = (s8)reg->s32_min_value; | |
| 5524 | + init_s32_max = (s8)reg_s32_max(reg); | |
| 5525 | + init_s32_min = (s8)reg_s32_min(reg); | |
| 5777 | 5526 | } else { |
| 5778 | 5527 | /* size == 2 */ |
| 5779 | − init_s32_max = (s16)reg->s32_max_value; | |
| 5780 | − init_s32_min = (s16)reg->s32_min_value; | |
| 5528 | + init_s32_max = (s16)reg_s32_max(reg); | |
| 5529 | + init_s32_min = (s16)reg_s32_min(reg); | |
| 5781 | 5530 | } |
| 5782 | 5531 | s32_max = max(init_s32_max, init_s32_min); |
| 5783 | 5532 | s32_min = min(init_s32_max, init_s32_min); |
| 5784 | 5533 | |
| 5785 | 5534 | if ((s32_min >= 0) == (s32_max >= 0)) { |
| 5786 | − reg->s32_min_value = s32_min; | |
| 5787 | − reg->s32_max_value = s32_max; | |
| 5788 | − reg->u32_min_value = (u32)s32_min; | |
| 5789 | − reg->u32_max_value = (u32)s32_max; | |
| 5535 | + reg_set_srange32(reg, s32_min, s32_max); | |
| 5790 | 5536 | reg->var_off = tnum_subreg(tnum_range(s32_min, s32_max)); |
| 5791 | 5537 | return; |
| 5792 | 5538 | } |
| @@ -5976,12 +5722,11 @@ static bool type_is_trusted_or_null(struct bpf_verifier_env *env, · 183 unmodified lines | ||
| 5976 | 5722 | } |
| 5977 | 5723 | |
| 5978 | 5724 | static int check_ptr_to_btf_access(struct bpf_verifier_env *env, |
| 5979 | − struct bpf_reg_state *regs, | |
| 5980 | − int regno, int off, int size, | |
| 5725 | + struct bpf_reg_state *regs, struct bpf_reg_state *reg, | |
| 5726 | + argno_t argno, int off, int size, | |
| 5981 | 5727 | enum bpf_access_type atype, |
| 5982 | 5728 | int value_regno) |
| 5983 | 5729 | { |
| 5984 | − struct bpf_reg_state *reg = regs + regno; | |
| 5985 | 5730 | const struct btf_type *t = btf_type_by_id(reg->btf, reg->btf_id); |
| 5986 | 5731 | const char *tname = btf_name_by_offset(reg->btf, t->name_off); |
| 5987 | 5732 | const char *field_name = NULL; |
| @@ -6007,8 +5752,8 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env, · 19 unmodified lines | ||
| 6007 | 5752 | |
| 6008 | 5753 | tnum_strn(tn_buf, sizeof(tn_buf), reg->var_off); |
| 6009 | 5754 | verbose(env, |
| 6010 | − "R%d is ptr_%s invalid variable offset: off=%d, var_off=%s\n", | |
| 6011 | − regno, tname, off, tn_buf); | |
| 5755 | + "%s is ptr_%s invalid variable offset: off=%d, var_off=%s\n", | |
| 5756 | + reg_arg_name(env, argno), tname, off, tn_buf); | |
| 6012 | 5757 | return -EACCES; |
| 6013 | 5758 | } |
| 6014 | 5759 | |
| @@ -6016,22 +5761,22 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env, · 1 unmodified line | ||
| 6016 | 5761 | |
| 6017 | 5762 | if (off < 0) { |
| 6018 | 5763 | verbose(env, |
| 6019 | − "R%d is ptr_%s invalid negative access: off=%d\n", | |
| 6020 | − regno, tname, off); | |
| 5764 | + "%s is ptr_%s invalid negative access: off=%d\n", | |
| 5765 | + reg_arg_name(env, argno), tname, off); | |
| 6021 | 5766 | return -EACCES; |
| 6022 | 5767 | } |
| 6023 | 5768 | |
| 6024 | 5769 | if (reg->type & MEM_USER) { |
| 6025 | 5770 | verbose(env, |
| 6026 | − "R%d is ptr_%s access user memory: off=%d\n", | |
| 6027 | − regno, tname, off); | |
| 5771 | + "%s is ptr_%s access user memory: off=%d\n", | |
| 5772 | + reg_arg_name(env, argno), tname, off); | |
| 6028 | 5773 | return -EACCES; |
| 6029 | 5774 | } |
| 6030 | 5775 | |
| 6031 | 5776 | if (reg->type & MEM_PERCPU) { |
| 6032 | 5777 | verbose(env, |
| 6033 | − "R%d is ptr_%s access percpu memory: off=%d\n", | |
| 6034 | − regno, tname, off); | |
| 5778 | + "%s is ptr_%s access percpu memory: off=%d\n", | |
| 5779 | + reg_arg_name(env, argno), tname, off); | |
| 6035 | 5780 | return -EACCES; |
| 6036 | 5781 | } |
| 6037 | 5782 | |
| @@ -6043,7 +5788,7 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env, · 5 unmodified lines | ||
| 6043 | 5788 | ret = env->ops->btf_struct_access(&env->log, reg, off, size); |
| 6044 | 5789 | } else { |
| 6045 | 5790 | /* Writes are permitted with default btf_struct_access for |
| 6046 | − * program allocated objects (which always have ref_obj_id > 0), | |
| 5791 | + * program allocated objects (which always have id > 0), | |
| 6047 | 5792 | * but not for untrusted PTR_TO_BTF_ID | MEM_ALLOC. |
| 6048 | 5793 | */ |
| 6049 | 5794 | if (atype != BPF_READ && !type_is_ptr_alloc_obj(reg->type)) { |
| @@ -6052,8 +5797,8 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env, · 2 unmodified lines | ||
| 6052 | 5797 | } |
| 6053 | 5798 | |
| 6054 | 5799 | if (type_is_alloc(reg->type) && !type_is_non_owning_ref(reg->type) && |
| 6055 | − !(reg->type & MEM_RCU) && !reg->ref_obj_id) { | |
| 6056 | − verifier_bug(env, "ref_obj_id for allocated object must be non-zero"); | |
| 5800 | + !(reg->type & MEM_RCU) && !reg_is_referenced(env, reg)) { | |
| 5801 | + verifier_bug(env, "allocated object must have a referenced id"); | |
| 6057 | 5802 | return -EFAULT; |
| 6058 | 5803 | } |
| 6059 | 5804 | |
| @@ -6072,7 +5817,7 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env, · 12 unmodified lines | ||
| 6072 | 5817 | */ |
| 6073 | 5818 | flag = PTR_UNTRUSTED; |
| 6074 | 5819 | |
| 6075 | − } else if (is_trusted_reg(reg) || is_rcu_reg(reg)) { | |
| 5820 | + } else if (is_trusted_reg(env, reg) || is_rcu_reg(reg)) { | |
| 6076 | 5821 | /* By default any pointer obtained from walking a trusted pointer is no |
| 6077 | 5822 | * longer trusted, unless the field being accessed has explicitly been |
| 6078 | 5823 | * marked as inheriting its parent's state of trust (either full or RCU). |
| @@ -6133,12 +5878,11 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env, · 54 unmodified lines | ||
| 6133 | 5878 | } |
| 6134 | 5879 | |
| 6135 | 5880 | static int check_ptr_to_map_access(struct bpf_verifier_env *env, |
| 6136 | − struct bpf_reg_state *regs, | |
| 6137 | − int regno, int off, int size, | |
| 5881 | + struct bpf_reg_state *regs, struct bpf_reg_state *reg, | |
| 5882 | + argno_t argno, int off, int size, | |
| 6138 | 5883 | enum bpf_access_type atype, |
| 6139 | 5884 | int value_regno) |
| 6140 | 5885 | { |
| 6141 | − struct bpf_reg_state *reg = regs + regno; | |
| 6142 | 5886 | struct bpf_map *map = reg->map_ptr; |
| 6143 | 5887 | struct bpf_reg_state map_reg; |
| 6144 | 5888 | enum bpf_type_flag flag = 0; |
| @@ -6169,8 +5913,8 @@ static int check_ptr_to_map_access(struct bpf_verifier_env *env, · 24 unmodified lines | ||
| 6169 | 5913 | } |
| 6170 | 5914 | |
| 6171 | 5915 | if (off < 0) { |
| 6172 | − verbose(env, "R%d is %s invalid negative access: off=%d\n", | |
| 6173 | − regno, tname, off); | |
| 5916 | + verbose(env, "%s is %s invalid negative access: off=%d\n", | |
| 5917 | + reg_arg_name(env, argno), tname, off); | |
| 6174 | 5918 | return -EACCES; |
| 6175 | 5919 | } |
| 6176 | 5920 | |
| @@ -6227,11 +5971,10 @@ static int check_stack_slot_within_bounds(struct bpf_verifier_env *env, · 50 unmodified lines | ||
| 6227 | 5971 | * 'off' includes `regno->offset`, but not its dynamic part (if any). |
| 6228 | 5972 | */ |
| 6229 | 5973 | static int check_stack_access_within_bounds( |
| 6230 | − struct bpf_verifier_env *env, | |
| 6231 | − int regno, int off, int access_size, | |
| 5974 | + struct bpf_verifier_env *env, struct bpf_reg_state *reg, | |
| 5975 | + argno_t argno, int off, int access_size, | |
| 6232 | 5976 | enum bpf_access_type type) |
| 6233 | 5977 | { |
| 6234 | − struct bpf_reg_state *reg = reg_state(env, regno); | |
| 6235 | 5978 | struct bpf_func_state *state = bpf_func(env, reg); |
| 6236 | 5979 | s64 min_off, max_off; |
| 6237 | 5980 | int err; |
| @@ -6246,14 +5989,14 @@ static int check_stack_access_within_bounds( · 8 unmodified lines | ||
| 6246 | 5989 | min_off = (s64)reg->var_off.value + off; |
| 6247 | 5990 | max_off = min_off + access_size; |
| 6248 | 5991 | } else { |
| 6249 | − if (reg->smax_value >= BPF_MAX_VAR_OFF || | |
| 6250 | − reg->smin_value <= -BPF_MAX_VAR_OFF) { | |
| 6251 | − verbose(env, "invalid unbounded variable-offset%s stack R%d\n", | |
| 6252 | − err_extra, regno); | |
| 5992 | + if (reg_smax(reg) >= BPF_MAX_VAR_OFF || | |
| 5993 | + reg_smin(reg) <= -BPF_MAX_VAR_OFF) { | |
| 5994 | + verbose(env, "invalid unbounded variable-offset%s stack %s\n", | |
| 5995 | + err_extra, reg_arg_name(env, argno)); | |
| 6253 | 5996 | return -EACCES; |
| 6254 | 5997 | } |
| 6255 | − min_off = reg->smin_value + off; | |
| 6256 | − max_off = reg->smax_value + off + access_size; | |
| 5998 | + min_off = reg_smin(reg) + off; | |
| 5999 | + max_off = reg_smax(reg) + off + access_size; | |
| 6257 | 6000 | } |
| 6258 | 6001 | |
| 6259 | 6002 | err = check_stack_slot_within_bounds(env, min_off, state, type); |
| @@ -6267,14 +6010,14 @@ static int check_stack_access_within_bounds( · 7 unmodified lines | ||
| 6267 | 6010 | |
| 6268 | 6011 | if (err) { |
| 6269 | 6012 | if (tnum_is_const(reg->var_off)) { |
| 6270 | − verbose(env, "invalid%s stack R%d off=%lld size=%d\n", | |
| 6271 | − err_extra, regno, min_off, access_size); | |
| 6013 | + verbose(env, "invalid%s stack %s off=%lld size=%d\n", | |
| 6014 | + err_extra, reg_arg_name(env, argno), min_off, access_size); | |
| 6272 | 6015 | } else { |
| 6273 | 6016 | char tn_buf[48]; |
| 6274 | 6017 | |
| 6275 | 6018 | tnum_strn(tn_buf, sizeof(tn_buf), reg->var_off); |
| 6276 | − verbose(env, "invalid variable-offset%s stack R%d var_off=%s off=%d size=%d\n", | |
| 6277 | − err_extra, regno, tn_buf, off, access_size); | |
| 6019 | + verbose(env, "invalid variable-offset%s stack %s var_off=%s off=%d size=%d\n", | |
| 6020 | + err_extra, reg_arg_name(env, argno), tn_buf, off, access_size); | |
| 6278 | 6021 | } |
| 6279 | 6022 | return err; |
| 6280 | 6023 | } |
| @@ -6319,12 +6062,11 @@ static void add_scalar_to_reg(struct bpf_reg_state *dst_reg, s64 val) · 38 unmodified lines | ||
| 6319 | 6062 | * if t==write && value_regno==-1, some unknown value is stored into memory |
| 6320 | 6063 | * if t==read && value_regno==-1, don't care what we read from memory |
| 6321 | 6064 | */ |
| 6322 | −static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regno, | |
| 6065 | +static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, struct bpf_reg_state *reg, argno_t argno, | |
| 6323 | 6066 | int off, int bpf_size, enum bpf_access_type t, |
| 6324 | 6067 | int value_regno, bool strict_alignment_once, bool is_ldsx) |
| 6325 | 6068 | { |
| 6326 | 6069 | struct bpf_reg_state *regs = cur_regs(env); |
| 6327 | − struct bpf_reg_state *reg = regs + regno; | |
| 6328 | 6070 | int size, err = 0; |
| 6329 | 6071 | |
| 6330 | 6072 | size = bpf_size_to_bytes(bpf_size); |
| @@ -6337,11 +6079,12 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 6 unmodified lines | ||
| 6337 | 6079 | |
| 6338 | 6080 | if (reg->type == PTR_TO_MAP_KEY) { |
| 6339 | 6081 | if (t == BPF_WRITE) { |
| 6340 | − verbose(env, "write to change key R%d not allowed\n", regno); | |
| 6082 | + verbose(env, "write to change key %s not allowed\n", | |
| 6083 | + reg_arg_name(env, argno)); | |
| 6341 | 6084 | return -EACCES; |
| 6342 | 6085 | } |
| 6343 | 6086 | |
| 6344 | − err = check_mem_region_access(env, regno, off, size, | |
| 6087 | + err = check_mem_region_access(env, reg, argno, off, size, | |
| 6345 | 6088 | reg->map_ptr->key_size, false); |
| 6346 | 6089 | if (err) |
| 6347 | 6090 | return err; |
| @@ -6355,17 +6098,17 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 7 unmodified lines | ||
| 6355 | 6098 | verbose(env, "R%d leaks addr into map\n", value_regno); |
| 6356 | 6099 | return -EACCES; |
| 6357 | 6100 | } |
| 6358 | − err = check_map_access_type(env, regno, off, size, t); | |
| 6101 | + err = check_map_access_type(env, reg, off, size, t); | |
| 6359 | 6102 | if (err) |
| 6360 | 6103 | return err; |
| 6361 | − err = check_map_access(env, regno, off, size, false, ACCESS_DIRECT); | |
| 6104 | + err = check_map_access(env, reg, argno, off, size, false, ACCESS_DIRECT); | |
| 6362 | 6105 | if (err) |
| 6363 | 6106 | return err; |
| 6364 | 6107 | if (tnum_is_const(reg->var_off)) |
| 6365 | 6108 | kptr_field = btf_record_find(reg->map_ptr->record, |
| 6366 | 6109 | off + reg->var_off.value, BPF_KPTR | BPF_UPTR); |
| 6367 | 6110 | if (kptr_field) { |
| 6368 | − err = check_map_kptr_access(env, regno, value_regno, insn_idx, kptr_field); | |
| 6111 | + err = check_map_kptr_access(env, value_regno, insn_idx, kptr_field); | |
| 6369 | 6112 | } else if (t == BPF_READ && value_regno >= 0) { |
| 6370 | 6113 | struct bpf_map *map = reg->map_ptr; |
| 6371 | 6114 | |
| @@ -6393,7 +6136,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 21 unmodified lines | ||
| 6393 | 6136 | size); |
| 6394 | 6137 | return -EACCES; |
| 6395 | 6138 | } |
| 6396 | − copy_register_state(®s[value_regno], reg); | |
| 6139 | + regs[value_regno] = *reg; | |
| 6397 | 6140 | add_scalar_to_reg(®s[value_regno], off); |
| 6398 | 6141 | regs[value_regno].type = PTR_TO_INSN; |
| 6399 | 6142 | } else { |
| @@ -6405,14 +6148,14 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 5 unmodified lines | ||
| 6405 | 6148 | bool rdonly_untrusted = rdonly_mem && (reg->type & PTR_UNTRUSTED); |
| 6406 | 6149 | |
| 6407 | 6150 | if (type_may_be_null(reg->type)) { |
| 6408 | − verbose(env, "R%d invalid mem access '%s'\n", regno, | |
| 6151 | + verbose(env, "%s invalid mem access '%s'\n", reg_arg_name(env, argno), | |
| 6409 | 6152 | reg_type_str(env, reg->type)); |
| 6410 | 6153 | return -EACCES; |
| 6411 | 6154 | } |
| 6412 | 6155 | |
| 6413 | 6156 | if (t == BPF_WRITE && rdonly_mem) { |
| 6414 | − verbose(env, "R%d cannot write into %s\n", | |
| 6415 | − regno, reg_type_str(env, reg->type)); | |
| 6157 | + verbose(env, "%s cannot write into %s\n", | |
| 6158 | + reg_arg_name(env, argno), reg_type_str(env, reg->type)); | |
| 6416 | 6159 | return -EACCES; |
| 6417 | 6160 | } |
| 6418 | 6161 | |
| @@ -6427,7 +6170,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 8 unmodified lines | ||
| 6427 | 6170 | * instructions, hence no need to check bounds in that case. |
| 6428 | 6171 | */ |
| 6429 | 6172 | if (!rdonly_untrusted) |
| 6430 | − err = check_mem_region_access(env, regno, off, size, | |
| 6173 | + err = check_mem_region_access(env, reg, argno, off, size, | |
| 6431 | 6174 | reg->mem_size, false); |
| 6432 | 6175 | if (!err && value_regno >= 0 && (t == BPF_READ || rdonly_mem)) |
| 6433 | 6176 | mark_reg_unknown(env, regs, value_regno); |
| @@ -6445,7 +6188,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 11 unmodified lines | ||
| 6445 | 6188 | return -EACCES; |
| 6446 | 6189 | } |
| 6447 | 6190 | |
| 6448 | − err = check_ctx_access(env, insn_idx, regno, off, size, t, &info); | |
| 6191 | + err = check_ctx_access(env, insn_idx, reg, argno, off, size, t, &info); | |
| 6449 | 6192 | if (!err && t == BPF_READ && value_regno >= 0) { |
| 6450 | 6193 | /* ctx access returns either a scalar, or a |
| 6451 | 6194 | * PTR_TO_PACKET[_META,_END]. In the latter |
| @@ -6463,8 +6206,6 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 11 unmodified lines | ||
| 6463 | 6206 | } else { |
| 6464 | 6207 | mark_reg_known_zero(env, regs, |
| 6465 | 6208 | value_regno); |
| 6466 | − if (type_may_be_null(info.reg_type)) | |
| 6467 | − regs[value_regno].id = ++env->id_gen; | |
| 6468 | 6209 | /* A load of ctx field could have different |
| 6469 | 6210 | * actual load size with the one encoded in the |
| 6470 | 6211 | * insn. When the dst is PTR, it is for sure not |
| @@ -6474,23 +6215,25 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 3 unmodified lines | ||
| 6474 | 6215 | if (base_type(info.reg_type) == PTR_TO_BTF_ID) { |
| 6475 | 6216 | regs[value_regno].btf = info.btf; |
| 6476 | 6217 | regs[value_regno].btf_id = info.btf_id; |
| 6477 | − regs[value_regno].ref_obj_id = info.ref_obj_id; | |
| 6218 | + regs[value_regno].id = info.ref_id; | |
| 6478 | 6219 | } |
| 6220 | + if (type_may_be_null(info.reg_type) && !regs[value_regno].id) | |
| 6221 | + regs[value_regno].id = ++env->id_gen; | |
| 6479 | 6222 | } |
| 6480 | 6223 | regs[value_regno].type = info.reg_type; |
| 6481 | 6224 | } |
| 6482 | 6225 | |
| 6483 | 6226 | } else if (reg->type == PTR_TO_STACK) { |
| 6484 | 6227 | /* Basic bounds checks. */ |
| 6485 | − err = check_stack_access_within_bounds(env, regno, off, size, t); | |
| 6228 | + err = check_stack_access_within_bounds(env, reg, argno, off, size, t); | |
| 6486 | 6229 | if (err) |
| 6487 | 6230 | return err; |
| 6488 | 6231 | |
| 6489 | 6232 | if (t == BPF_READ) |
| 6490 | − err = check_stack_read(env, regno, off, size, | |
| 6233 | + err = check_stack_read(env, reg, argno, off, size, | |
| 6491 | 6234 | value_regno); |
| 6492 | 6235 | else |
| 6493 | − err = check_stack_write(env, regno, off, size, | |
| 6236 | + err = check_stack_write(env, reg, off, size, | |
| 6494 | 6237 | value_regno, insn_idx); |
| 6495 | 6238 | } else if (reg_is_pkt_pointer(reg)) { |
| 6496 | 6239 | if (t == BPF_WRITE && !may_access_direct_pkt_data(env, NULL, t)) { |
| @@ -6503,7 +6246,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 6 unmodified lines | ||
| 6503 | 6246 | value_regno); |
| 6504 | 6247 | return -EACCES; |
| 6505 | 6248 | } |
| 6506 | − err = check_packet_access(env, regno, off, size, false); | |
| 6249 | + err = check_packet_access(env, reg, argno, off, size, false); | |
| 6507 | 6250 | if (!err && t == BPF_READ && value_regno >= 0) |
| 6508 | 6251 | mark_reg_unknown(env, regs, value_regno); |
| 6509 | 6252 | } else if (reg->type == PTR_TO_FLOW_KEYS) { |
| @@ -6514,28 +6257,28 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 4 unmodified lines | ||
| 6514 | 6257 | return -EACCES; |
| 6515 | 6258 | } |
| 6516 | 6259 | |
| 6517 | − err = check_flow_keys_access(env, off, size); | |
| 6260 | + err = check_flow_keys_access(env, reg, argno, off, size); | |
| 6518 | 6261 | if (!err && t == BPF_READ && value_regno >= 0) |
| 6519 | 6262 | mark_reg_unknown(env, regs, value_regno); |
| 6520 | 6263 | } else if (type_is_sk_pointer(reg->type)) { |
| 6521 | 6264 | if (t == BPF_WRITE) { |
| 6522 | − verbose(env, "R%d cannot write into %s\n", | |
| 6523 | − regno, reg_type_str(env, reg->type)); | |
| 6265 | + verbose(env, "%s cannot write into %s\n", | |
| 6266 | + reg_arg_name(env, argno), reg_type_str(env, reg->type)); | |
| 6524 | 6267 | return -EACCES; |
| 6525 | 6268 | } |
| 6526 | − err = check_sock_access(env, insn_idx, regno, off, size, t); | |
| 6269 | + err = check_sock_access(env, insn_idx, reg, argno, off, size, t); | |
| 6527 | 6270 | if (!err && value_regno >= 0) |
| 6528 | 6271 | mark_reg_unknown(env, regs, value_regno); |
| 6529 | 6272 | } else if (reg->type == PTR_TO_TP_BUFFER) { |
| 6530 | − err = check_tp_buffer_access(env, reg, regno, off, size); | |
| 6273 | + err = check_tp_buffer_access(env, reg, argno, off, size); | |
| 6531 | 6274 | if (!err && t == BPF_READ && value_regno >= 0) |
| 6532 | 6275 | mark_reg_unknown(env, regs, value_regno); |
| 6533 | 6276 | } else if (base_type(reg->type) == PTR_TO_BTF_ID && |
| 6534 | 6277 | !type_may_be_null(reg->type)) { |
| 6535 | − err = check_ptr_to_btf_access(env, regs, regno, off, size, t, | |
| 6278 | + err = check_ptr_to_btf_access(env, regs, reg, argno, off, size, t, | |
| 6536 | 6279 | value_regno); |
| 6537 | 6280 | } else if (reg->type == CONST_PTR_TO_MAP) { |
| 6538 | − err = check_ptr_to_map_access(env, regs, regno, off, size, t, | |
| 6281 | + err = check_ptr_to_map_access(env, regs, reg, argno, off, size, t, | |
| 6539 | 6282 | value_regno); |
| 6540 | 6283 | } else if (base_type(reg->type) == PTR_TO_BUF && |
| 6541 | 6284 | !type_may_be_null(reg->type)) { |
| @@ -6544,8 +6287,8 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 2 unmodified lines | ||
| 6544 | 6287 | |
| 6545 | 6288 | if (rdonly_mem) { |
| 6546 | 6289 | if (t == BPF_WRITE) { |
| 6547 | − verbose(env, "R%d cannot write into %s\n", | |
| 6548 | − regno, reg_type_str(env, reg->type)); | |
| 6290 | + verbose(env, "%s cannot write into %s\n", | |
| 6291 | + reg_arg_name(env, argno), reg_type_str(env, reg->type)); | |
| 6549 | 6292 | return -EACCES; |
| 6550 | 6293 | } |
| 6551 | 6294 | max_access = &env->prog->aux->max_rdonly_access; |
| @@ -6553,7 +6296,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 1 unmodified line | ||
| 6553 | 6296 | max_access = &env->prog->aux->max_rdwr_access; |
| 6554 | 6297 | } |
| 6555 | 6298 | |
| 6556 | − err = check_buffer_access(env, reg, regno, off, size, false, | |
| 6299 | + err = check_buffer_access(env, reg, argno, off, size, false, | |
| 6557 | 6300 | max_access); |
| 6558 | 6301 | |
| 6559 | 6302 | if (!err && value_regno >= 0 && (rdonly_mem || t == BPF_READ)) |
| @@ -6562,7 +6305,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn · 2 unmodified lines | ||
| 6562 | 6305 | if (t == BPF_READ && value_regno >= 0) |
| 6563 | 6306 | mark_reg_unknown(env, regs, value_regno); |
| 6564 | 6307 | } else { |
| 6565 | − verbose(env, "R%d invalid mem access '%s'\n", regno, | |
| 6308 | + verbose(env, "%s invalid mem access '%s'\n", reg_arg_name(env, argno), | |
| 6566 | 6309 | reg_type_str(env, reg->type)); |
| 6567 | 6310 | return -EACCES; |
| 6568 | 6311 | } |
| @@ -6585,10 +6328,20 @@ static int check_load_mem(struct bpf_verifier_env *env, struct bpf_insn *insn, · 16 unmodified lines | ||
| 6585 | 6328 | bool strict_alignment_once, bool is_ldsx, |
| 6586 | 6329 | bool allow_trust_mismatch, const char *ctx) |
| 6587 | 6330 | { |
| 6331 | + struct bpf_verifier_state *vstate = env->cur_state; | |
| 6332 | + struct bpf_func_state *state = vstate->frame[vstate->curframe]; | |
| 6588 | 6333 | struct bpf_reg_state *regs = cur_regs(env); |
| 6589 | 6334 | enum bpf_reg_type src_reg_type; |
| 6590 | 6335 | int err; |
| 6591 | 6336 | |
| 6337 | + /* Handle stack arg read */ | |
| 6338 | + if (is_stack_arg_ldx(insn)) { | |
| 6339 | + err = check_reg_arg(env, insn->dst_reg, DST_OP_NO_MARK); | |
| 6340 | + if (err) | |
| 6341 | + return err; | |
| 6342 | + return check_stack_arg_read(env, state, insn->off, insn->dst_reg); | |
| 6343 | + } | |
| 6344 | + | |
| 6592 | 6345 | /* check src operand */ |
| 6593 | 6346 | err = check_reg_arg(env, insn->src_reg, SRC_OP); |
| 6594 | 6347 | if (err) |
| @@ -6604,7 +6357,7 @@ static int check_load_mem(struct bpf_verifier_env *env, struct bpf_insn *insn, · 9 unmodified lines | ||
| 6604 | 6357 | /* Check if (src_reg + off) is readable. The state of dst_reg will be |
| 6605 | 6358 | * updated by this call. |
| 6606 | 6359 | */ |
| 6607 | − err = check_mem_access(env, env->insn_idx, insn->src_reg, insn->off, | |
| 6360 | + err = check_mem_access(env, env->insn_idx, regs + insn->src_reg, argno_from_reg(insn->src_reg), insn->off, | |
| 6608 | 6361 | BPF_SIZE(insn->code), BPF_READ, insn->dst_reg, |
| 6609 | 6362 | strict_alignment_once, is_ldsx); |
| 6610 | 6363 | err = err ?: save_aux_ptr_type(env, src_reg_type, |
| @@ -6617,10 +6370,20 @@ static int check_load_mem(struct bpf_verifier_env *env, struct bpf_insn *insn, · 6 unmodified lines | ||
| 6617 | 6370 | static int check_store_reg(struct bpf_verifier_env *env, struct bpf_insn *insn, |
| 6618 | 6371 | bool strict_alignment_once) |
| 6619 | 6372 | { |
| 6373 | + struct bpf_verifier_state *vstate = env->cur_state; | |
| 6374 | + struct bpf_func_state *state = vstate->frame[vstate->curframe]; | |
| 6620 | 6375 | struct bpf_reg_state *regs = cur_regs(env); |
| 6621 | 6376 | enum bpf_reg_type dst_reg_type; |
| 6622 | 6377 | int err; |
| 6623 | 6378 | |
| 6379 | + /* Handle stack arg write */ | |
| 6380 | + if (is_stack_arg_stx(insn)) { | |
| 6381 | + err = check_reg_arg(env, insn->src_reg, SRC_OP); | |
| 6382 | + if (err) | |
| 6383 | + return err; | |
| 6384 | + return check_stack_arg_write(env, state, insn->off, regs + insn->src_reg); | |
| 6385 | + } | |
| 6386 | + | |
| 6624 | 6387 | /* check src1 operand */ |
| 6625 | 6388 | err = check_reg_arg(env, insn->src_reg, SRC_OP); |
| 6626 | 6389 | if (err) |
| @@ -6634,7 +6397,7 @@ static int check_store_reg(struct bpf_verifier_env *env, struct bpf_insn *insn, · 7 unmodified lines | ||
| 6634 | 6397 | dst_reg_type = regs[insn->dst_reg].type; |
| 6635 | 6398 | |
| 6636 | 6399 | /* Check if (dst_reg + off) is writeable. */ |
| 6637 | − err = check_mem_access(env, env->insn_idx, insn->dst_reg, insn->off, | |
| 6400 | + err = check_mem_access(env, env->insn_idx, regs + insn->dst_reg, argno_from_reg(insn->dst_reg), insn->off, | |
| 6638 | 6401 | BPF_SIZE(insn->code), BPF_WRITE, insn->src_reg, |
| 6639 | 6402 | strict_alignment_once, false); |
| 6640 | 6403 | err = err ?: save_aux_ptr_type(env, dst_reg_type, false); |
| @@ -6645,6 +6408,7 @@ static int check_store_reg(struct bpf_verifier_env *env, struct bpf_insn *insn, · 4 unmodified lines | ||
| 6645 | 6408 | static int check_atomic_rmw(struct bpf_verifier_env *env, |
| 6646 | 6409 | struct bpf_insn *insn) |
| 6647 | 6410 | { |
| 6411 | + struct bpf_reg_state *dst_reg; | |
| 6648 | 6412 | int load_reg; |
| 6649 | 6413 | int err; |
| 6650 | 6414 | |
| @@ -6706,13 +6470,15 @@ static int check_atomic_rmw(struct bpf_verifier_env *env, · 55 unmodified lines | ||
| 6706 | 6470 | load_reg = -1; |
| 6707 | 6471 | } |
| 6708 | 6472 | |
| 6473 | + dst_reg = cur_regs(env) + insn->dst_reg; | |
| 6474 | + | |
| 6709 | 6475 | /* Check whether we can read the memory, with second call for fetch |
| 6710 | 6476 | * case to simulate the register fill. |
| 6711 | 6477 | */ |
| 6712 | − err = check_mem_access(env, env->insn_idx, insn->dst_reg, insn->off, | |
| 6478 | + err = check_mem_access(env, env->insn_idx, dst_reg, argno_from_reg(insn->dst_reg), insn->off, | |
| 6713 | 6479 | BPF_SIZE(insn->code), BPF_READ, -1, true, false); |
| 6714 | 6480 | if (!err && load_reg >= 0) |
| 6715 | − err = check_mem_access(env, env->insn_idx, insn->dst_reg, | |
| 6481 | + err = check_mem_access(env, env->insn_idx, dst_reg, argno_from_reg(insn->dst_reg), | |
| 6716 | 6482 | insn->off, BPF_SIZE(insn->code), |
| 6717 | 6483 | BPF_READ, load_reg, true, false); |
| 6718 | 6484 | if (err) |
| @@ -6724,7 +6490,7 @@ static int check_atomic_rmw(struct bpf_verifier_env *env, · 5 unmodified lines | ||
| 6724 | 6490 | return err; |
| 6725 | 6491 | } |
| 6726 | 6492 | /* Check whether we can write into the same memory. */ |
| 6727 | − err = check_mem_access(env, env->insn_idx, insn->dst_reg, insn->off, | |
| 6493 | + err = check_mem_access(env, env->insn_idx, dst_reg, argno_from_reg(insn->dst_reg), insn->off, | |
| 6728 | 6494 | BPF_SIZE(insn->code), BPF_WRITE, -1, true, false); |
| 6729 | 6495 | if (err) |
| 6730 | 6496 | return err; |
| @@ -6813,11 +6579,10 @@ static int check_atomic(struct bpf_verifier_env *env, struct bpf_insn *insn) · 82 unmodified lines | ||
| 6813 | 6579 | * read offsets are marked as read. |
| 6814 | 6580 | */ |
| 6815 | 6581 | static int check_stack_range_initialized( |
| 6816 | − struct bpf_verifier_env *env, int regno, int off, | |
| 6582 | + struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno, int off, | |
| 6817 | 6583 | int access_size, bool zero_size_allowed, |
| 6818 | 6584 | enum bpf_access_type type, struct bpf_call_arg_meta *meta) |
| 6819 | 6585 | { |
| 6820 | − struct bpf_reg_state *reg = reg_state(env, regno); | |
| 6821 | 6586 | struct bpf_func_state *state = bpf_func(env, reg); |
| 6822 | 6587 | int err, min_off, max_off, i, j, slot, spi; |
| 6823 | 6588 | /* Some accesses can write anything into the stack, others are |
| @@ -6839,7 +6604,7 @@ static int check_stack_range_initialized( · 15 unmodified lines | ||
| 6839 | 6604 | return -EACCES; |
| 6840 | 6605 | } |
| 6841 | 6606 | |
| 6842 | − err = check_stack_access_within_bounds(env, regno, off, access_size, type); | |
| 6607 | + err = check_stack_access_within_bounds(env, reg, argno, off, access_size, type); | |
| 6843 | 6608 | if (err) |
| 6844 | 6609 | return err; |
| 6845 | 6610 | |
| @@ -6856,8 +6621,8 @@ static int check_stack_range_initialized( · 10 unmodified lines | ||
| 6856 | 6621 | char tn_buf[48]; |
| 6857 | 6622 | |
| 6858 | 6623 | tnum_strn(tn_buf, sizeof(tn_buf), reg->var_off); |
| 6859 | − verbose(env, "R%d variable offset stack access prohibited for !root, var_off=%s\n", | |
| 6860 | − regno, tn_buf); | |
| 6624 | + verbose(env, "%s variable offset stack access prohibited for !root, var_off=%s\n", | |
| 6625 | + reg_arg_name(env, argno), tn_buf); | |
| 6861 | 6626 | return -EACCES; |
| 6862 | 6627 | } |
| 6863 | 6628 | /* Only initialized buffer on stack is allowed to be accessed |
| @@ -6869,8 +6634,8 @@ static int check_stack_range_initialized( · 5 unmodified lines | ||
| 6869 | 6634 | if (meta && meta->raw_mode) |
| 6870 | 6635 | meta = NULL; |
| 6871 | 6636 | |
| 6872 | − min_off = reg->smin_value + off; | |
| 6873 | − max_off = reg->smax_value + off; | |
| 6637 | + min_off = reg_smin(reg) + off; | |
| 6638 | + max_off = reg_smax(reg) + off; | |
| 6874 | 6639 | } |
| 6875 | 6640 | |
| 6876 | 6641 | if (meta && meta->raw_mode) { |
| @@ -6900,7 +6665,7 @@ static int check_stack_range_initialized( · 23 unmodified lines | ||
| 6900 | 6665 | } |
| 6901 | 6666 | } |
| 6902 | 6667 | meta->access_size = access_size; |
| 6903 | − meta->regno = regno; | |
| 6668 | + meta->regno = reg_from_argno(argno); | |
| 6904 | 6669 | return 0; |
| 6905 | 6670 | } |
| 6906 | 6671 | |
| @@ -6940,17 +6705,17 @@ static int check_stack_range_initialized( · 33 unmodified lines | ||
| 6940 | 6705 | if (*stype == STACK_POISON) { |
| 6941 | 6706 | if (allow_poison) |
| 6942 | 6707 | goto mark; |
| 6943 | − verbose(env, "reading from stack R%d off %d+%d size %d, slot poisoned by dead code elimination\n", | |
| 6944 | − regno, min_off, i - min_off, access_size); | |
| 6708 | + verbose(env, "reading from stack %s off %d+%d size %d, slot poisoned by dead code elimination\n", | |
| 6709 | + reg_arg_name(env, argno), min_off, i - min_off, access_size); | |
| 6945 | 6710 | } else if (tnum_is_const(reg->var_off)) { |
| 6946 | − verbose(env, "invalid read from stack R%d off %d+%d size %d\n", | |
| 6947 | − regno, min_off, i - min_off, access_size); | |
| 6711 | + verbose(env, "invalid read from stack %s off %d+%d size %d\n", | |
| 6712 | + reg_arg_name(env, argno), min_off, i - min_off, access_size); | |
| 6948 | 6713 | } else { |
| 6949 | 6714 | char tn_buf[48]; |
| 6950 | 6715 | |
| 6951 | 6716 | tnum_strn(tn_buf, sizeof(tn_buf), reg->var_off); |
| 6952 | − verbose(env, "invalid read from stack R%d var_off %s+%d size %d\n", | |
| 6953 | − regno, tn_buf, i - min_off, access_size); | |
| 6717 | + verbose(env, "invalid read from stack %s var_off %s+%d size %d\n", | |
| 6718 | + reg_arg_name(env, argno), tn_buf, i - min_off, access_size); | |
| 6954 | 6719 | } |
| 6955 | 6720 | return -EACCES; |
| 6956 | 6721 | mark: |
| @@ -6959,48 +6724,48 @@ mark: · 2 unmodified lines | ||
| 6959 | 6724 | return 0; |
| 6960 | 6725 | } |
| 6961 | 6726 | |
| 6962 | −static int check_helper_mem_access(struct bpf_verifier_env *env, int regno, | |
| 6727 | +static int check_helper_mem_access(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno, | |
| 6963 | 6728 | int access_size, enum bpf_access_type access_type, |
| 6964 | 6729 | bool zero_size_allowed, |
| 6965 | 6730 | struct bpf_call_arg_meta *meta) |
| 6966 | 6731 | { |
| 6967 | − struct bpf_reg_state *regs = cur_regs(env), *reg = ®s[regno]; | |
| 6732 | + struct bpf_reg_state *regs = cur_regs(env); | |
| 6968 | 6733 | u32 *max_access; |
| 6969 | 6734 | |
| 6970 | 6735 | switch (base_type(reg->type)) { |
| 6971 | 6736 | case PTR_TO_PACKET: |
| 6972 | 6737 | case PTR_TO_PACKET_META: |
| 6973 | − return check_packet_access(env, regno, 0, access_size, | |
| 6738 | + return check_packet_access(env, reg, argno, 0, access_size, | |
| 6974 | 6739 | zero_size_allowed); |
| 6975 | 6740 | case PTR_TO_MAP_KEY: |
| 6976 | 6741 | if (access_type == BPF_WRITE) { |
| 6977 | − verbose(env, "R%d cannot write into %s\n", regno, | |
| 6978 | − reg_type_str(env, reg->type)); | |
| 6742 | + verbose(env, "%s cannot write into %s\n", | |
| 6743 | + reg_arg_name(env, argno), reg_type_str(env, reg->type)); | |
| 6979 | 6744 | return -EACCES; |
| 6980 | 6745 | } |
| 6981 | − return check_mem_region_access(env, regno, 0, access_size, | |
| 6746 | + return check_mem_region_access(env, reg, argno, 0, access_size, | |
| 6982 | 6747 | reg->map_ptr->key_size, false); |
| 6983 | 6748 | case PTR_TO_MAP_VALUE: |
| 6984 | − if (check_map_access_type(env, regno, 0, access_size, access_type)) | |
| 6749 | + if (check_map_access_type(env, reg, 0, access_size, access_type)) | |
| 6985 | 6750 | return -EACCES; |
| 6986 | − return check_map_access(env, regno, 0, access_size, | |
| 6751 | + return check_map_access(env, reg, argno, 0, access_size, | |
| 6987 | 6752 | zero_size_allowed, ACCESS_HELPER); |
| 6988 | 6753 | case PTR_TO_MEM: |
| 6989 | 6754 | if (type_is_rdonly_mem(reg->type)) { |
| 6990 | 6755 | if (access_type == BPF_WRITE) { |
| 6991 | − verbose(env, "R%d cannot write into %s\n", regno, | |
| 6992 | − reg_type_str(env, reg->type)); | |
| 6756 | + verbose(env, "%s cannot write into %s\n", | |
| 6757 | + reg_arg_name(env, argno), reg_type_str(env, reg->type)); | |
| 6993 | 6758 | return -EACCES; |
| 6994 | 6759 | } |
| 6995 | 6760 | } |
| 6996 | − return check_mem_region_access(env, regno, 0, | |
| 6761 | + return check_mem_region_access(env, reg, argno, 0, | |
| 6997 | 6762 | access_size, reg->mem_size, |
| 6998 | 6763 | zero_size_allowed); |
| 6999 | 6764 | case PTR_TO_BUF: |
| 7000 | 6765 | if (type_is_rdonly_mem(reg->type)) { |
| 7001 | 6766 | if (access_type == BPF_WRITE) { |
| 7002 | − verbose(env, "R%d cannot write into %s\n", regno, | |
| 7003 | − reg_type_str(env, reg->type)); | |
| 6767 | + verbose(env, "%s cannot write into %s\n", | |
| 6768 | + reg_arg_name(env, argno), reg_type_str(env, reg->type)); | |
| 7004 | 6769 | return -EACCES; |
| 7005 | 6770 | } |
| 7006 | 6771 | |
| @@ -7008,26 +6773,26 @@ static int check_helper_mem_access(struct bpf_verifier_env *env, int regno, · 1 unmodified line | ||
| 7008 | 6773 | } else { |
| 7009 | 6774 | max_access = &env->prog->aux->max_rdwr_access; |
| 7010 | 6775 | } |
| 7011 | − return check_buffer_access(env, reg, regno, 0, | |
| 6776 | + return check_buffer_access(env, reg, argno, 0, | |
| 7012 | 6777 | access_size, zero_size_allowed, |
| 7013 | 6778 | max_access); |
| 7014 | 6779 | case PTR_TO_STACK: |
| 7015 | 6780 | return check_stack_range_initialized( |
| 7016 | − env, | |
| 7017 | − regno, 0, access_size, | |
| 6781 | + env, reg, | |
| 6782 | + argno, 0, access_size, | |
| 7018 | 6783 | zero_size_allowed, access_type, meta); |
| 7019 | 6784 | case PTR_TO_BTF_ID: |
| 7020 | − return check_ptr_to_btf_access(env, regs, regno, 0, | |
| 7021 | − access_size, BPF_READ, -1); | |
| 6785 | + return check_ptr_to_btf_access(env, regs, reg, argno, 0, | |
| 6786 | + access_size, access_type, -1); | |
| 7022 | 6787 | case PTR_TO_CTX: |
| 7023 | 6788 | /* Only permit reading or writing syscall context using helper calls. */ |
| 7024 | 6789 | if (is_var_ctx_off_allowed(env->prog)) { |
| 7025 | − int err = check_mem_region_access(env, regno, 0, access_size, U16_MAX, | |
| 6790 | + int err = check_mem_region_access(env, reg, argno, 0, access_size, U16_MAX, | |
| 7026 | 6791 | zero_size_allowed); |
| 7027 | 6792 | if (err) |
| 7028 | 6793 | return err; |
| 7029 | − if (env->prog->aux->max_ctx_offset < reg->umax_value + access_size) | |
| 7030 | − env->prog->aux->max_ctx_offset = reg->umax_value + access_size; | |
| 6794 | + if (env->prog->aux->max_ctx_offset < reg_umax(reg) + access_size) | |
| 6795 | + env->prog->aux->max_ctx_offset = reg_umax(reg) + access_size; | |
| 7031 | 6796 | return 0; |
| 7032 | 6797 | } |
| 7033 | 6798 | fallthrough; |
| @@ -7037,7 +6802,7 @@ static int check_helper_mem_access(struct bpf_verifier_env *env, int regno, · 3 unmodified lines | ||
| 7037 | 6802 | bpf_register_is_null(reg)) |
| 7038 | 6803 | return 0; |
| 7039 | 6804 | |
| 7040 | − verbose(env, "R%d type=%s ", regno, | |
| 6805 | + verbose(env, "%s type=%s ", reg_arg_name(env, argno), | |
| 7041 | 6806 | reg_type_str(env, reg->type)); |
| 7042 | 6807 | verbose(env, "expected=%s\n", reg_type_str(env, PTR_TO_STACK)); |
| 7043 | 6808 | return -EACCES; |
| @@ -7047,12 +6812,12 @@ static int check_helper_mem_access(struct bpf_verifier_env *env, int regno, · 3 unmodified lines | ||
| 7047 | 6812 | /* verify arguments to helpers or kfuncs consisting of a pointer and an access |
| 7048 | 6813 | * size. |
| 7049 | 6814 | * |
| 7050 | − * @regno is the register containing the access size. regno-1 is the register | |
| 7051 | − * containing the pointer. | |
| 6815 | + * @mem_reg contains the pointer, @size_reg contains the access size. | |
| 7052 | 6816 | */ |
| 7053 | 6817 | static int check_mem_size_reg(struct bpf_verifier_env *env, |
| 7054 | − struct bpf_reg_state *reg, u32 regno, | |
| 7055 | − enum bpf_access_type access_type, | |
| 6818 | + struct bpf_reg_state *mem_reg, | |
| 6819 | + struct bpf_reg_state *size_reg, argno_t mem_argno, | |
| 6820 | + argno_t size_argno, enum bpf_access_type access_type, | |
| 7056 | 6821 | bool zero_size_allowed, |
| 7057 | 6822 | struct bpf_call_arg_meta *meta) |
| 7058 | 6823 | { |
| @@ -7066,42 +6831,48 @@ static int check_mem_size_reg(struct bpf_verifier_env *env, · 7 unmodified lines | ||
| 7066 | 6831 | * out. Only upper bounds can be learned because retval is an |
| 7067 | 6832 | * int type and negative retvals are allowed. |
| 7068 | 6833 | */ |
| 7069 | − meta->msize_max_value = reg->umax_value; | |
| 6834 | + meta->msize_max_value = reg_umax(size_reg); | |
| 7070 | 6835 | |
| 7071 | 6836 | /* The register is SCALAR_VALUE; the access check happens using |
| 7072 | 6837 | * its boundaries. For unprivileged variable accesses, disable |
| 7073 | 6838 | * raw mode so that the program is required to initialize all |
| 7074 | 6839 | * the memory that the helper could just partially fill up. |
| 7075 | 6840 | */ |
| 7076 | − if (!tnum_is_const(reg->var_off)) | |
| 6841 | + if (!tnum_is_const(size_reg->var_off)) | |
| 7077 | 6842 | meta = NULL; |
| 7078 | 6843 | |
| 7079 | − if (reg->smin_value < 0) { | |
| 7080 | − verbose(env, "R%d min value is negative, either use unsigned or 'var &= const'\n", | |
| 7081 | − regno); | |
| 6844 | + if (reg_smin(size_reg) < 0) { | |
| 6845 | + verbose(env, "%s min value is negative, either use unsigned or 'var &= const'\n", | |
| 6846 | + reg_arg_name(env, size_argno)); | |
| 7082 | 6847 | return -EACCES; |
| 7083 | 6848 | } |
| 7084 | 6849 | |
| 7085 | − if (reg->umin_value == 0 && !zero_size_allowed) { | |
| 7086 | − verbose(env, "R%d invalid zero-sized read: u64=[%lld,%lld]\n", | |
| 7087 | − regno, reg->umin_value, reg->umax_value); | |
| 6850 | + if (reg_umin(size_reg) == 0 && !zero_size_allowed) { | |
| 6851 | + verbose(env, "%s invalid zero-sized read: u64=[%lld,%lld]\n", | |
| 6852 | + reg_arg_name(env, size_argno), reg_umin(size_reg), reg_umax(size_reg)); | |
| 7088 | 6853 | return -EACCES; |
| 7089 | 6854 | } |
| 7090 | 6855 | |
| 7091 | − if (reg->umax_value >= BPF_MAX_VAR_SIZ) { | |
| 7092 | − verbose(env, "R%d unbounded memory access, use 'var &= const' or 'if (var < const)'\n", | |
| 7093 | − regno); | |
| 6856 | + if (reg_umax(size_reg) >= BPF_MAX_VAR_SIZ) { | |
| 6857 | + verbose(env, "%s unbounded memory access, use 'var &= const' or 'if (var < const)'\n", | |
| 6858 | + reg_arg_name(env, size_argno)); | |
| 7094 | 6859 | return -EACCES; |
| 7095 | 6860 | } |
| 7096 | − err = check_helper_mem_access(env, regno - 1, reg->umax_value, | |
| 6861 | + err = check_helper_mem_access(env, mem_reg, mem_argno, reg_umax(size_reg), | |
| 7097 | 6862 | access_type, zero_size_allowed, meta); |
| 7098 | − if (!err) | |
| 7099 | − err = mark_chain_precision(env, regno); | |
| 6863 | + if (!err) { | |
| 6864 | + int regno = reg_from_argno(size_argno); | |
| 6865 | + | |
| 6866 | + if (regno >= 0) | |
| 6867 | + err = mark_chain_precision(env, regno); | |
| 6868 | + else | |
| 6869 | + err = mark_stack_arg_precision(env, arg_idx_from_argno(size_argno)); | |
| 6870 | + } | |
| 7100 | 6871 | return err; |
| 7101 | 6872 | } |
| 7102 | 6873 | |
| 7103 | 6874 | static int check_mem_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg, |
| 7104 | − u32 regno, u32 mem_size) | |
| 6875 | + argno_t argno, u32 mem_size) | |
| 7105 | 6876 | { |
| 7106 | 6877 | bool may_be_null = type_may_be_null(reg->type); |
| 7107 | 6878 | struct bpf_reg_state saved_reg; |
| @@ -7110,6 +6881,12 @@ static int check_mem_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg · 2 unmodified lines | ||
| 7110 | 6881 | if (bpf_register_is_null(reg)) |
| 7111 | 6882 | return 0; |
| 7112 | 6883 | |
| 6884 | + if (mem_size > S32_MAX) { | |
| 6885 | + verbose(env, "%s memory size %u is too large\n", | |
| 6886 | + reg_arg_name(env, argno), mem_size); | |
| 6887 | + return -EACCES; | |
| 6888 | + } | |
| 6889 | + | |
| 7113 | 6890 | /* Assuming that the register contains a value check if the memory |
| 7114 | 6891 | * access is safe. Temporarily save and restore the register's state as |
| 7115 | 6892 | * the conversion shouldn't be visible to a caller. |
| @@ -7121,8 +6898,8 @@ static int check_mem_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg · 5 unmodified lines | ||
| 7121 | 6898 | |
| 7122 | 6899 | int size = base_type(reg->type) == PTR_TO_STACK ? -(int)mem_size : mem_size; |
| 7123 | 6900 | |
| 7124 | − err = check_helper_mem_access(env, regno, size, BPF_READ, true, NULL); | |
| 7125 | − err = err ?: check_helper_mem_access(env, regno, size, BPF_WRITE, true, NULL); | |
| 6901 | + err = check_helper_mem_access(env, reg, argno, size, BPF_READ, true, NULL); | |
| 6902 | + err = err ?: check_helper_mem_access(env, reg, argno, size, BPF_WRITE, true, NULL); | |
| 7126 | 6903 | |
| 7127 | 6904 | if (may_be_null) |
| 7128 | 6905 | *reg = saved_reg; |
| @@ -7130,17 +6907,14 @@ static int check_mem_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg · 1 unmodified line | ||
| 7130 | 6907 | return err; |
| 7131 | 6908 | } |
| 7132 | 6909 | |
| 7133 | −static int check_kfunc_mem_size_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg, | |
| 7134 | − u32 regno) | |
| 6910 | +static int check_kfunc_mem_size_reg(struct bpf_verifier_env *env, struct bpf_reg_state *mem_reg, | |
| 6911 | + struct bpf_reg_state *size_reg, argno_t mem_argno, argno_t size_argno) | |
| 7135 | 6912 | { |
| 7136 | − struct bpf_reg_state *mem_reg = &cur_regs(env)[regno - 1]; | |
| 7137 | 6913 | bool may_be_null = type_may_be_null(mem_reg->type); |
| 7138 | 6914 | struct bpf_reg_state saved_reg; |
| 7139 | 6915 | struct bpf_call_arg_meta meta; |
| 7140 | 6916 | int err; |
| 7141 | 6917 | |
| 7142 | − WARN_ON_ONCE(regno < BPF_REG_2 || regno > BPF_REG_5); | |
| 7143 | − | |
| 7144 | 6918 | memset(&meta, 0, sizeof(meta)); |
| 7145 | 6919 | |
| 7146 | 6920 | if (may_be_null) { |
| @@ -7148,8 +6922,8 @@ static int check_kfunc_mem_size_reg(struct bpf_verifier_env *env, struct bpf_reg · 1 unmodified line | ||
| 7148 | 6922 | mark_ptr_not_null_reg(mem_reg); |
| 7149 | 6923 | } |
| 7150 | 6924 | |
| 7151 | − err = check_mem_size_reg(env, reg, regno, BPF_READ, true, &meta); | |
| 7152 | − err = err ?: check_mem_size_reg(env, reg, regno, BPF_WRITE, true, &meta); | |
| 6925 | + err = check_mem_size_reg(env, mem_reg, size_reg, mem_argno, size_argno, BPF_READ, true, &meta); | |
| 6926 | + err = err ?: check_mem_size_reg(env, mem_reg, size_reg, mem_argno, size_argno, BPF_WRITE, true, &meta); | |
| 7153 | 6927 | |
| 7154 | 6928 | if (may_be_null) |
| 7155 | 6929 | *mem_reg = saved_reg; |
| @@ -7185,11 +6959,10 @@ enum { · 29 unmodified lines | ||
| 7185 | 6959 | * env->cur_state->active_locks remembers which map value element or allocated |
| 7186 | 6960 | * object got locked and clears it after bpf_spin_unlock. |
| 7187 | 6961 | */ |
| 7188 | −static int process_spin_lock(struct bpf_verifier_env *env, int regno, int flags) | |
| 6962 | +static int process_spin_lock(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno, int flags) | |
| 7189 | 6963 | { |
| 7190 | 6964 | bool is_lock = flags & PROCESS_SPIN_LOCK, is_res_lock = flags & PROCESS_RES_LOCK; |
| 7191 | 6965 | const char *lock_str = is_res_lock ? "bpf_res_spin" : "bpf_spin"; |
| 7192 | − struct bpf_reg_state *reg = reg_state(env, regno); | |
| 7193 | 6966 | struct bpf_verifier_state *cur = env->cur_state; |
| 7194 | 6967 | bool is_const = tnum_is_const(reg->var_off); |
| 7195 | 6968 | bool is_irq = flags & PROCESS_LOCK_IRQ; |
| @@ -7202,8 +6975,8 @@ static int process_spin_lock(struct bpf_verifier_env *env, int regno, int flags) · 6 unmodified lines | ||
| 7202 | 6975 | |
| 7203 | 6976 | if (!is_const) { |
| 7204 | 6977 | verbose(env, |
| 7205 | − "R%d doesn't have constant offset. %s_lock has to be at the constant offset\n", | |
| 7206 | − regno, lock_str); | |
| 6978 | + "%s doesn't have constant offset. %s_lock has to be at the constant offset\n", | |
| 6979 | + reg_arg_name(env, argno), lock_str); | |
| 7207 | 6980 | return -EINVAL; |
| 7208 | 6981 | } |
| 7209 | 6982 | if (reg->type == PTR_TO_MAP_VALUE) { |
| @@ -7302,11 +7075,10 @@ static int process_spin_lock(struct bpf_verifier_env *env, int regno, int flags) · 92 unmodified lines | ||
| 7302 | 7075 | } |
| 7303 | 7076 | |
| 7304 | 7077 | /* Check if @regno is a pointer to a specific field in a map value */ |
| 7305 | −static int check_map_field_pointer(struct bpf_verifier_env *env, u32 regno, | |
| 7078 | +static int check_map_field_pointer(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno, | |
| 7306 | 7079 | enum btf_field_type field_type, |
| 7307 | 7080 | struct bpf_map_desc *map_desc) |
| 7308 | 7081 | { |
| 7309 | − struct bpf_reg_state *reg = reg_state(env, regno); | |
| 7310 | 7082 | bool is_const = tnum_is_const(reg->var_off); |
| 7311 | 7083 | struct bpf_map *map = reg->map_ptr; |
| 7312 | 7084 | u64 val = reg->var_off.value; |
| @@ -7315,8 +7087,8 @@ static int check_map_field_pointer(struct bpf_verifier_env *env, u32 regno, · 2 unmodified lines | ||
| 7315 | 7087 | |
| 7316 | 7088 | if (!is_const) { |
| 7317 | 7089 | verbose(env, |
| 7318 | − "R%d doesn't have constant offset. %s has to be at the constant offset\n", | |
| 7319 | − regno, struct_name); | |
| 7090 | + "%s doesn't have constant offset. %s has to be at the constant offset\n", | |
| 7091 | + reg_arg_name(env, argno), struct_name); | |
| 7320 | 7092 | return -EINVAL; |
| 7321 | 7093 | } |
| 7322 | 7094 | if (!map->btf) { |
| @@ -7356,26 +7128,26 @@ static int check_map_field_pointer(struct bpf_verifier_env *env, u32 regno, · 33 unmodified lines | ||
| 7356 | 7128 | return 0; |
| 7357 | 7129 | } |
| 7358 | 7130 | |
| 7359 | −static int process_timer_func(struct bpf_verifier_env *env, int regno, | |
| 7131 | +static int process_timer_func(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno, | |
| 7360 | 7132 | struct bpf_map_desc *map) |
| 7361 | 7133 | { |
| 7362 | 7134 | if (IS_ENABLED(CONFIG_PREEMPT_RT)) { |
| 7363 | 7135 | verbose(env, "bpf_timer cannot be used for PREEMPT_RT.\n"); |
| 7364 | 7136 | return -EOPNOTSUPP; |
| 7365 | 7137 | } |
| 7366 | − return check_map_field_pointer(env, regno, BPF_TIMER, map); | |
| 7138 | + return check_map_field_pointer(env, reg, argno, BPF_TIMER, map); | |
| 7367 | 7139 | } |
| 7368 | 7140 | |
| 7369 | −static int process_timer_helper(struct bpf_verifier_env *env, int regno, | |
| 7141 | +static int process_timer_helper(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno, | |
| 7370 | 7142 | struct bpf_call_arg_meta *meta) |
| 7371 | 7143 | { |
| 7372 | − return process_timer_func(env, regno, &meta->map); | |
| 7144 | + return process_timer_func(env, reg, argno, &meta->map); | |
| 7373 | 7145 | } |
| 7374 | 7146 | |
| 7375 | −static int process_timer_kfunc(struct bpf_verifier_env *env, int regno, | |
| 7147 | +static int process_timer_kfunc(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno, | |
| 7376 | 7148 | struct bpf_kfunc_call_arg_meta *meta) |
| 7377 | 7149 | { |
| 7378 | − return process_timer_func(env, regno, &meta->map); | |
| 7150 | + return process_timer_func(env, reg, argno, &meta->map); | |
| 7379 | 7151 | } |
| 7380 | 7152 | |
| 7381 | 7153 | static int process_kptr_func(struct bpf_verifier_env *env, int regno, |
| @@ -7426,52 +7198,42 @@ static int process_kptr_func(struct bpf_verifier_env *env, int regno, · 44 unmodified lines | ||
| 7426 | 7198 | return 0; |
| 7427 | 7199 | } |
| 7428 | 7200 | |
| 7429 | −/* There are two register types representing a bpf_dynptr, one is PTR_TO_STACK | |
| 7201 | +/* | |
| 7202 | + * Validate dynptr arguments for helper, kfunc and subprog. | |
| 7203 | + * | |
| 7204 | + * @dynptr is both input and output. It is populated when the argument is | |
| 7205 | + * tagged with MEM_UNINIT (i.e., the dynptr argument that will be constructed) | |
| 7206 | + * and consumed when the argument is expecting to be an initialized dynptr. | |
| 7207 | + * @parent_id is used to track the referenced parent object (e.g., file or skb in | |
| 7208 | + * qdisc program) when constructing a dynptr. | |
| 7209 | + * | |
| 7210 | + * There are two register types representing a bpf_dynptr, one is PTR_TO_STACK | |
| 7430 | 7211 | * which points to a stack slot, and the other is CONST_PTR_TO_DYNPTR. |
| 7431 | 7212 | * |
| 7432 | 7213 | * In both cases we deal with the first 8 bytes, but need to mark the next 8 |
| 7433 | 7214 | * bytes as STACK_DYNPTR in case of PTR_TO_STACK. In case of |
| 7434 | 7215 | * CONST_PTR_TO_DYNPTR, we are guaranteed to get the beginning of the object. |
| 7435 | 7216 | * |
| 7436 | − * Mutability of bpf_dynptr is at two levels, one is at the level of struct | |
| 7437 | − * bpf_dynptr itself, i.e. whether the helper is receiving a pointer to struct | |
| 7438 | − * bpf_dynptr or pointer to const struct bpf_dynptr. In the former case, it can | |
| 7439 | − * mutate the view of the dynptr and also possibly destroy it. In the latter | |
| 7440 | − * case, it cannot mutate the bpf_dynptr itself but it can still mutate the | |
| 7441 | − * memory that dynptr points to. | |
| 7442 | − * | |
| 7443 | − * The verifier will keep track both levels of mutation (bpf_dynptr's in | |
| 7444 | − * reg->type and the memory's in reg->dynptr.type), but there is no support for | |
| 7445 | − * readonly dynptr view yet, hence only the first case is tracked and checked. | |
| 7446 | − * | |
| 7447 | − * This is consistent with how C applies the const modifier to a struct object, | |
| 7448 | − * where the pointer itself inside bpf_dynptr becomes const but not what it | |
| 7449 | − * points to. | |
| 7450 | − * | |
| 7451 | − * Helpers which do not mutate the bpf_dynptr set MEM_RDONLY in their argument | |
| 7452 | − * type, and declare it as 'const struct bpf_dynptr *' in their prototype. | |
| 7217 | + * Mutability of bpf_dynptr is at two levels: the dynptr and the memory the | |
| 7218 | + * dynptr points to. At the first level, the verifier will make sure a | |
| 7219 | + * CONST_PTR_TO_DYNPTR cannot be reinitialized or destroyed. The mutability of | |
| 7220 | + * a dynptr's view (i.e., start and offset) is not tracked as there is not such | |
| 7221 | + * use case. The second level is tracked using the upper bit of bpf_dynptr->size | |
| 7222 | + * and checked dynamically during runtime. | |
| 7453 | 7223 | */ |
| 7454 | −static int process_dynptr_func(struct bpf_verifier_env *env, int regno, int insn_idx, | |
| 7455 | − enum bpf_arg_type arg_type, int clone_ref_obj_id) | |
| 7224 | +static int process_dynptr_func(struct bpf_verifier_env *env, struct bpf_reg_state *reg, | |
| 7225 | + argno_t argno, int insn_idx, enum bpf_arg_type arg_type, | |
| 7226 | + struct ref_obj_desc *ref_obj, struct bpf_dynptr_desc *dynptr) | |
| 7456 | 7227 | { |
| 7457 | − struct bpf_reg_state *reg = reg_state(env, regno); | |
| 7458 | − int err; | |
| 7228 | + int spi, err = 0; | |
| 7459 | 7229 | |
| 7460 | 7230 | if (reg->type != PTR_TO_STACK && reg->type != CONST_PTR_TO_DYNPTR) { |
| 7461 | 7231 | verbose(env, |
| 7462 | − "arg#%d expected pointer to stack or const struct bpf_dynptr\n", | |
| 7463 | − regno - 1); | |
| 7232 | + "%s expected pointer to stack or const struct bpf_dynptr\n", | |
| 7233 | + reg_arg_name(env, argno)); | |
| 7464 | 7234 | return -EINVAL; |
| 7465 | 7235 | } |
| 7466 | 7236 | |
| 7467 | − /* MEM_UNINIT and MEM_RDONLY are exclusive, when applied to an | |
| 7468 | − * ARG_PTR_TO_DYNPTR (or ARG_PTR_TO_DYNPTR | DYNPTR_TYPE_*): | |
| 7469 | − */ | |
| 7470 | − if ((arg_type & (MEM_UNINIT | MEM_RDONLY)) == (MEM_UNINIT | MEM_RDONLY)) { | |
| 7471 | − verifier_bug(env, "misconfigured dynptr helper type flags"); | |
| 7472 | − return -EFAULT; | |
| 7473 | − } | |
| 7474 | − | |
| 7475 | 7237 | /* MEM_UNINIT - Points to memory that is an appropriate candidate for |
| 7476 | 7238 | * constructing a mutable bpf_dynptr object. |
| 7477 | 7239 | * |
| @@ -7479,13 +7241,12 @@ static int process_dynptr_func(struct bpf_verifier_env *env, int regno, int insn · 1 unmodified line | ||
| 7479 | 7241 | * pointing to a region of at least 16 bytes which doesn't |
| 7480 | 7242 | * contain an existing bpf_dynptr. |
| 7481 | 7243 | * |
| 7482 | − * MEM_RDONLY - Points to a initialized bpf_dynptr that will not be | |
| 7483 | − * mutated or destroyed. However, the memory it points to | |
| 7484 | − * may be mutated. | |
| 7244 | + * OBJ_RELEASE - Points to a initialized bpf_dynptr that will be | |
| 7245 | + * destroyed. | |
| 7485 | 7246 | * |
| 7486 | − * None - Points to a initialized dynptr that can be mutated and | |
| 7487 | − * destroyed, including mutation of the memory it points | |
| 7488 | − * to. | |
| 7247 | + * None - Points to a initialized dynptr that cannot be | |
| 7248 | + * reinitialized or destroyed. However, the view of the | |
| 7249 | + * dynptr and the memory it points to may be mutated. | |
| 7489 | 7250 | */ |
| 7490 | 7251 | if (arg_type & MEM_UNINIT) { |
| 7491 | 7252 | int i; |
| @@ -7497,45 +7258,58 @@ static int process_dynptr_func(struct bpf_verifier_env *env, int regno, int insn · 5 unmodified lines | ||
| 7497 | 7258 | |
| 7498 | 7259 | /* we write BPF_DW bits (8 bytes) at a time */ |
| 7499 | 7260 | for (i = 0; i < BPF_DYNPTR_SIZE; i += 8) { |
| 7500 | − err = check_mem_access(env, insn_idx, regno, | |
| 7261 | + err = check_mem_access(env, insn_idx, reg, argno, | |
| 7501 | 7262 | i, BPF_DW, BPF_WRITE, -1, false, false); |
| 7502 | 7263 | if (err) |
| 7503 | 7264 | return err; |
| 7504 | 7265 | } |
| 7505 | 7266 | |
| 7506 | − err = mark_stack_slots_dynptr(env, reg, arg_type, insn_idx, clone_ref_obj_id); | |
| 7507 | − } else /* MEM_RDONLY and None case from above */ { | |
| 7267 | + err = mark_stack_slots_dynptr(env, reg, arg_type, insn_idx, ref_obj, dynptr); | |
| 7268 | + } else /* OBJ_RELEASE and None case from above */ { | |
| 7508 | 7269 | /* For the reg->type == PTR_TO_STACK case, bpf_dynptr is never const */ |
| 7509 | − if (reg->type == CONST_PTR_TO_DYNPTR && !(arg_type & MEM_RDONLY)) { | |
| 7510 | − verbose(env, "cannot pass pointer to const bpf_dynptr, the helper mutates it\n"); | |
| 7270 | + if (reg->type == CONST_PTR_TO_DYNPTR && (arg_type & OBJ_RELEASE)) { | |
| 7271 | + verbose(env, "CONST_PTR_TO_DYNPTR cannot be released\n"); | |
| 7511 | 7272 | return -EINVAL; |
| 7512 | 7273 | } |
| 7513 | 7274 | |
| 7514 | 7275 | if (!is_dynptr_reg_valid_init(env, reg)) { |
| 7515 | − verbose(env, | |
| 7516 | − "Expected an initialized dynptr as arg #%d\n", | |
| 7517 | − regno - 1); | |
| 7276 | + verbose(env, "Expected an initialized dynptr as %s\n", | |
| 7277 | + reg_arg_name(env, argno)); | |
| 7518 | 7278 | return -EINVAL; |
| 7519 | 7279 | } |
| 7520 | 7280 | |
| 7521 | − /* Fold modifiers (in this case, MEM_RDONLY) when checking expected type */ | |
| 7522 | − if (!is_dynptr_type_expected(env, reg, arg_type & ~MEM_RDONLY)) { | |
| 7281 | + /* Fold modifiers (in this case, OBJ_RELEASE) when checking expected type */ | |
| 7282 | + if (!is_dynptr_type_expected(env, reg, arg_type & ~OBJ_RELEASE)) { | |
| 7523 | 7283 | verbose(env, |
| 7524 | − "Expected a dynptr of type %s as arg #%d\n", | |
| 7525 | − dynptr_type_str(arg_to_dynptr_type(arg_type)), regno - 1); | |
| 7284 | + "Expected a dynptr of type %s as %s\n", | |
| 7285 | + dynptr_type_str(arg_to_dynptr_type(arg_type)), | |
| 7286 | + reg_arg_name(env, argno)); | |
| 7526 | 7287 | return -EINVAL; |
| 7527 | 7288 | } |
| 7528 | 7289 | |
| 7529 | − err = mark_dynptr_read(env, reg); | |
| 7530 | − } | |
| 7531 | − return err; | |
| 7532 | −} | |
| 7290 | + if (reg->type != CONST_PTR_TO_DYNPTR) { | |
| 7291 | + struct bpf_func_state *state = bpf_func(env, reg); | |
| 7533 | 7292 | |
| 7534 | −static u32 iter_ref_obj_id(struct bpf_verifier_env *env, struct bpf_reg_state *reg, int spi) | |
| 7535 | −{ | |
| 7536 | − struct bpf_func_state *state = bpf_func(env, reg); | |
| 7293 | + spi = dynptr_get_spi(env, reg); | |
| 7294 | + if (spi < 0) | |
| 7295 | + return spi; | |
| 7296 | + | |
| 7297 | + /* | |
| 7298 | + * For CONST_PTR_TO_DYNPTR, reg is already scratched by check_reg_arg | |
| 7299 | + * in check_helper_call and mark_btf_func_reg_size in check_kfunc_call. | |
| 7300 | + */ | |
| 7301 | + mark_stack_slots_scratched(env, spi, BPF_DYNPTR_NR_SLOTS); | |
| 7537 | 7302 | |
| 7538 | − return state->stack[spi].spilled_ptr.ref_obj_id; | |
| 7303 | + reg = &state->stack[spi].spilled_ptr; | |
| 7304 | + } | |
| 7305 | + | |
| 7306 | + if (dynptr) { | |
| 7307 | + dynptr->type = reg->dynptr.type; | |
| 7308 | + dynptr->id = reg->id; | |
| 7309 | + dynptr->parent_id = reg->parent_id; | |
| 7310 | + } | |
| 7311 | + } | |
| 7312 | + return err; | |
| 7539 | 7313 | } |
| 7540 | 7314 | |
| 7541 | 7315 | static bool is_iter_kfunc(struct bpf_kfunc_call_arg_meta *meta) |
| @@ -7567,15 +7341,17 @@ static bool is_kfunc_arg_iter(struct bpf_kfunc_call_arg_meta *meta, int arg_idx, · 25 unmodified lines | ||
| 7567 | 7341 | return btf_param_match_suffix(meta->btf, arg, "__iter"); |
| 7568 | 7342 | } |
| 7569 | 7343 | |
| 7570 | −static int process_iter_arg(struct bpf_verifier_env *env, int regno, int insn_idx, | |
| 7344 | +static int process_iter_arg(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno, int insn_idx, | |
| 7571 | 7345 | struct bpf_kfunc_call_arg_meta *meta) |
| 7572 | 7346 | { |
| 7573 | − struct bpf_reg_state *reg = reg_state(env, regno); | |
| 7347 | + struct bpf_func_state *state = bpf_func(env, reg); | |
| 7574 | 7348 | const struct btf_type *t; |
| 7349 | + u32 arg_idx = arg_idx_from_argno(argno); | |
| 7575 | 7350 | int spi, err, i, nr_slots, btf_id; |
| 7576 | 7351 | |
| 7577 | 7352 | if (reg->type != PTR_TO_STACK) { |
| 7578 | − verbose(env, "arg#%d expected pointer to an iterator on stack\n", regno - 1); | |
| 7353 | + verbose(env, "%s expected pointer to an iterator on stack\n", | |
| 7354 | + reg_arg_name(env, argno)); | |
| 7579 | 7355 | return -EINVAL; |
| 7580 | 7356 | } |
| 7581 | 7357 | |
| @@ -7585,9 +7361,10 @@ static int process_iter_arg(struct bpf_verifier_env *env, int regno, int insn_id · 3 unmodified lines | ||
| 7585 | 7361 | * to any kfunc, if arg has "__iter" suffix, we need to be a bit more |
| 7586 | 7362 | * conservative here. |
| 7587 | 7363 | */ |
| 7588 | − btf_id = btf_check_iter_arg(meta->btf, meta->func_proto, regno - 1); | |
| 7364 | + btf_id = btf_check_iter_arg(meta->btf, meta->func_proto, arg_idx); | |
| 7589 | 7365 | if (btf_id < 0) { |
| 7590 | − verbose(env, "expected valid iter pointer as arg #%d\n", regno - 1); | |
| 7366 | + verbose(env, "expected valid iter pointer as %s\n", | |
| 7367 | + reg_arg_name(env, argno)); | |
| 7591 | 7368 | return -EINVAL; |
| 7592 | 7369 | } |
| 7593 | 7370 | t = btf_type_by_id(meta->btf, btf_id); |
| @@ -7596,13 +7373,13 @@ static int process_iter_arg(struct bpf_verifier_env *env, int regno, int insn_id · 2 unmodified lines | ||
| 7596 | 7373 | if (is_iter_new_kfunc(meta)) { |
| 7597 | 7374 | /* bpf_iter_<type>_new() expects pointer to uninit iter state */ |
| 7598 | 7375 | if (!is_iter_reg_valid_uninit(env, reg, nr_slots)) { |
| 7599 | − verbose(env, "expected uninitialized iter_%s as arg #%d\n", | |
| 7600 | − iter_type_str(meta->btf, btf_id), regno - 1); | |
| 7376 | + verbose(env, "expected uninitialized iter_%s as %s\n", | |
| 7377 | + iter_type_str(meta->btf, btf_id), reg_arg_name(env, argno)); | |
| 7601 | 7378 | return -EINVAL; |
| 7602 | 7379 | } |
| 7603 | 7380 | |
| 7604 | 7381 | for (i = 0; i < nr_slots * 8; i += BPF_REG_SIZE) { |
| 7605 | − err = check_mem_access(env, insn_idx, regno, | |
| 7382 | + err = check_mem_access(env, insn_idx, reg, argno, | |
| 7606 | 7383 | i, BPF_DW, BPF_WRITE, -1, false, false); |
| 7607 | 7384 | if (err) |
| 7608 | 7385 | return err; |
| @@ -7620,8 +7397,8 @@ static int process_iter_arg(struct bpf_verifier_env *env, int regno, int insn_id · 11 unmodified lines | ||
| 7620 | 7397 | case 0: |
| 7621 | 7398 | break; |
| 7622 | 7399 | case -EINVAL: |
| 7623 | − verbose(env, "expected an initialized iter_%s as arg #%d\n", | |
| 7624 | − iter_type_str(meta->btf, btf_id), regno - 1); | |
| 7400 | + verbose(env, "expected an initialized iter_%s as %s\n", | |
| 7401 | + iter_type_str(meta->btf, btf_id), reg_arg_name(env, argno)); | |
| 7625 | 7402 | return err; |
| 7626 | 7403 | case -EPROTO: |
| 7627 | 7404 | verbose(env, "expected an RCU CS when using %s\n", meta->func_name); |
| @@ -7634,14 +7411,12 @@ static int process_iter_arg(struct bpf_verifier_env *env, int regno, int insn_id · 6 unmodified lines | ||
| 7634 | 7411 | if (spi < 0) |
| 7635 | 7412 | return spi; |
| 7636 | 7413 | |
| 7637 | − err = mark_iter_read(env, reg, spi, nr_slots); | |
| 7638 | − if (err) | |
| 7639 | − return err; | |
| 7414 | + mark_stack_slots_scratched(env, spi, nr_slots); | |
| 7640 | 7415 | |
| 7641 | 7416 | /* remember meta->iter info for process_iter_next_call() */ |
| 7642 | 7417 | meta->iter.spi = spi; |
| 7643 | 7418 | meta->iter.frameno = reg->frameno; |
| 7644 | − meta->ref_obj_id = iter_ref_obj_id(env, reg, spi); | |
| 7419 | + update_ref_obj(&meta->ref_obj, &state->stack[spi].spilled_ptr); | |
| 7645 | 7420 | |
| 7646 | 7421 | if (is_iter_destroy_kfunc(meta)) { |
| 7647 | 7422 | err = unmark_stack_slots_iter(env, reg, nr_slots); |
| @@ -8041,12 +7816,11 @@ static const struct bpf_reg_types *compatible_reg_types[__BPF_ARG_TYPE_MAX] = { · 393 unmodified lines | ||
| 8041 | 7816 | [ARG_PTR_TO_DYNPTR] = &dynptr_types, |
| 8042 | 7817 | }; |
| 8043 | 7818 | |
| 8044 | −static int check_reg_type(struct bpf_verifier_env *env, u32 regno, | |
| 7819 | +static int check_reg_type(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno, | |
| 8045 | 7820 | enum bpf_arg_type arg_type, |
| 8046 | 7821 | const u32 *arg_btf_id, |
| 8047 | 7822 | struct bpf_call_arg_meta *meta) |
| 8048 | 7823 | { |
| 8049 | − struct bpf_reg_state *reg = reg_state(env, regno); | |
| 8050 | 7824 | enum bpf_reg_type expected, type = reg->type; |
| 8051 | 7825 | const struct bpf_reg_types *compatible; |
| 8052 | 7826 | int i, j, err; |
| @@ -8077,7 +7851,7 @@ static int check_reg_type(struct bpf_verifier_env *env, u32 regno, · 24 unmodified lines | ||
| 8077 | 7851 | type &= ~DYNPTR_TYPE_FLAG_MASK; |
| 8078 | 7852 | |
| 8079 | 7853 | /* Local kptr types are allowed as the source argument of bpf_kptr_xchg */ |
| 8080 | − if (meta->func_id == BPF_FUNC_kptr_xchg && type_is_alloc(type) && regno == BPF_REG_2) { | |
| 7854 | + if (meta->func_id == BPF_FUNC_kptr_xchg && type_is_alloc(type) && reg_from_argno(argno) == BPF_REG_2) { | |
| 8081 | 7855 | type &= ~MEM_ALLOC; |
| 8082 | 7856 | type &= ~MEM_PERCPU; |
| 8083 | 7857 | } |
| @@ -8091,7 +7865,7 @@ static int check_reg_type(struct bpf_verifier_env *env, u32 regno, · 7 unmodified lines | ||
| 8091 | 7865 | goto found; |
| 8092 | 7866 | } |
| 8093 | 7867 | |
| 8094 | − verbose(env, "R%d type=%s expected=", regno, reg_type_str(env, reg->type)); | |
| 7868 | + verbose(env, "%s type=%s expected=", reg_arg_name(env, argno), reg_type_str(env, reg->type)); | |
| 8095 | 7869 | for (j = 0; j + 1 < i; j++) |
| 8096 | 7870 | verbose(env, "%s, ", reg_type_str(env, compatible->types[j])); |
| 8097 | 7871 | verbose(env, "%s\n", reg_type_str(env, compatible->types[j])); |
| @@ -8104,9 +7878,9 @@ found: · 6 unmodified lines | ||
| 8104 | 7878 | if (compatible == &mem_types) { |
| 8105 | 7879 | if (!(arg_type & MEM_RDONLY)) { |
| 8106 | 7880 | verbose(env, |
| 8107 | − "%s() may write into memory pointed by R%d type=%s\n", | |
| 7881 | + "%s() may write into memory pointed by %s type=%s\n", | |
| 8108 | 7882 | func_id_name(meta->func_id), |
| 8109 | − regno, reg_type_str(env, reg->type)); | |
| 7883 | + reg_arg_name(env, argno), reg_type_str(env, reg->type)); | |
| 8110 | 7884 | return -EACCES; |
| 8111 | 7885 | } |
| 8112 | 7886 | return 0; |
| @@ -8129,7 +7903,8 @@ found: · 16 unmodified lines | ||
| 8129 | 7903 | |
| 8130 | 7904 | if (type_may_be_null(reg->type) && |
| 8131 | 7905 | (!type_may_be_null(arg_type) || arg_type_is_release(arg_type))) { |
| 8132 | − verbose(env, "Possibly NULL pointer passed to helper arg%d\n", regno); | |
| 7906 | + verbose(env, "Possibly NULL pointer passed to helper %s\n", | |
| 7907 | + reg_arg_name(env, argno)); | |
| 8133 | 7908 | return -EACCES; |
| 8134 | 7909 | } |
| 8135 | 7910 | |
| @@ -8142,25 +7917,26 @@ found: · 6 unmodified lines | ||
| 8142 | 7917 | } |
| 8143 | 7918 | |
| 8144 | 7919 | if (meta->func_id == BPF_FUNC_kptr_xchg) { |
| 8145 | − if (map_kptr_match_type(env, meta->kptr_field, reg, regno)) | |
| 7920 | + if (map_kptr_match_type(env, meta->kptr_field, reg, reg_from_argno(argno))) | |
| 8146 | 7921 | return -EACCES; |
| 8147 | 7922 | } else { |
| 8148 | 7923 | if (arg_btf_id == BPF_PTR_POISON) { |
| 8149 | 7924 | verbose(env, "verifier internal error:"); |
| 8150 | − verbose(env, "R%d has non-overwritten BPF_PTR_POISON type\n", | |
| 8151 | − regno); | |
| 7925 | + verbose(env, "%s has non-overwritten BPF_PTR_POISON type\n", | |
| 7926 | + reg_arg_name(env, argno)); | |
| 8152 | 7927 | return -EACCES; |
| 8153 | 7928 | } |
| 8154 | 7929 | |
| 8155 | − err = __check_ptr_off_reg(env, reg, regno, true); | |
| 7930 | + err = __check_ptr_off_reg(env, reg, argno, true); | |
| 8156 | 7931 | if (err) |
| 8157 | 7932 | return err; |
| 8158 | 7933 | |
| 8159 | 7934 | if (!btf_struct_ids_match(&env->log, reg->btf, reg->btf_id, |
| 8160 | 7935 | reg->var_off.value, btf_vmlinux, *arg_btf_id, |
| 8161 | 7936 | strict_type_match)) { |
| 8162 | − verbose(env, "R%d is of type %s but %s is expected\n", | |
| 8163 | − regno, btf_type_name(reg->btf, reg->btf_id), | |
| 7937 | + verbose(env, "%s is of type %s but %s is expected\n", | |
| 7938 | + reg_arg_name(env, argno), | |
| 7939 | + btf_type_name(reg->btf, reg->btf_id), | |
| 8164 | 7940 | btf_type_name(btf_vmlinux, *arg_btf_id)); |
| 8165 | 7941 | return -EACCES; |
| 8166 | 7942 | } |
| @@ -8177,8 +7953,11 @@ found: · 10 unmodified lines | ||
| 8177 | 7953 | return -EFAULT; |
| 8178 | 7954 | } |
| 8179 | 7955 | /* Check if local kptr in src arg matches kptr in dst arg */ |
| 8180 | − if (meta->func_id == BPF_FUNC_kptr_xchg && regno == BPF_REG_2) { | |
| 8181 | − if (map_kptr_match_type(env, meta->kptr_field, reg, regno)) | |
| 7956 | + if (meta->func_id == BPF_FUNC_kptr_xchg) { | |
| 7957 | + int regno = reg_from_argno(argno); | |
| 7958 | + | |
| 7959 | + if (regno == BPF_REG_2 && | |
| 7960 | + map_kptr_match_type(env, meta->kptr_field, reg, regno)) | |
| 8182 | 7961 | return -EACCES; |
| 8183 | 7962 | } |
| 8184 | 7963 | break; |
| @@ -8212,7 +7991,7 @@ reg_find_field_offset(const struct bpf_reg_state *reg, s32 off, u32 fields) · 27 unmodified lines | ||
| 8212 | 7991 | } |
| 8213 | 7992 | |
| 8214 | 7993 | static int check_func_arg_reg_off(struct bpf_verifier_env *env, |
| 8215 | − const struct bpf_reg_state *reg, int regno, | |
| 7994 | + const struct bpf_reg_state *reg, argno_t argno, | |
| 8216 | 7995 | enum bpf_arg_type arg_type) |
| 8217 | 7996 | { |
| 8218 | 7997 | u32 type = reg->type; |
| @@ -8220,7 +7999,7 @@ static int check_func_arg_reg_off(struct bpf_verifier_env *env, · 1 unmodified line | ||
| 8220 | 7999 | /* When referenced register is passed to release function, its fixed |
| 8221 | 8000 | * offset must be 0. |
| 8222 | 8001 | * |
| 8223 | − * We will check arg_type_is_release reg has ref_obj_id when storing | |
| 8002 | + * We will check arg_type_is_release reg has id when storing | |
| 8224 | 8003 | * meta->release_regno. |
| 8225 | 8004 | */ |
| 8226 | 8005 | if (arg_type_is_release(arg_type)) { |
| @@ -8238,8 +8017,8 @@ static int check_func_arg_reg_off(struct bpf_verifier_env *env, · 11 unmodified lines | ||
| 8238 | 8017 | * to give the user a better error message. |
| 8239 | 8018 | */ |
| 8240 | 8019 | if (!tnum_is_const(reg->var_off) || reg->var_off.value != 0) { |
| 8241 | − verbose(env, "R%d must have zero offset when passed to release func or trusted arg to kfunc\n", | |
| 8242 | − regno); | |
| 8020 | + verbose(env, "%s must have zero offset when passed to release func or trusted arg to kfunc\n", | |
| 8021 | + reg_arg_name(env, argno)); | |
| 8243 | 8022 | return -EINVAL; |
| 8244 | 8023 | } |
| 8245 | 8024 | } |
| @@ -8272,93 +8051,27 @@ static int check_func_arg_reg_off(struct bpf_verifier_env *env, · 26 unmodified lines | ||
| 8272 | 8051 | * its fixed offset must be 0. In the other cases, fixed offset |
| 8273 | 8052 | * can be non-zero. This was already checked above. So pass |
| 8274 | 8053 | * fixed_off_ok as true to allow fixed offset for all other |
| 8275 | − * cases. var_off always must be 0 for PTR_TO_BTF_ID, hence we | |
| 8276 | − * still need to do checks instead of returning. | |
| 8277 | − */ | |
| 8278 | − return __check_ptr_off_reg(env, reg, regno, true); | |
| 8279 | − case PTR_TO_CTX: | |
| 8280 | − /* | |
| 8281 | − * Allow fixed and variable offsets for syscall context, but | |
| 8282 | − * only when the argument is passed as memory, not ctx, | |
| 8283 | − * otherwise we may get modified ctx in tail called programs and | |
| 8284 | − * global subprogs (that may act as extension prog hooks). | |
| 8285 | − */ | |
| 8286 | − if (arg_type != ARG_PTR_TO_CTX && is_var_ctx_off_allowed(env->prog)) | |
| 8287 | − return 0; | |
| 8288 | − fallthrough; | |
| 8289 | − default: | |
| 8290 | − return __check_ptr_off_reg(env, reg, regno, false); | |
| 8291 | − } | |
| 8292 | −} | |
| 8293 | − | |
| 8294 | −static struct bpf_reg_state *get_dynptr_arg_reg(struct bpf_verifier_env *env, | |
| 8295 | − const struct bpf_func_proto *fn, | |
| 8296 | − struct bpf_reg_state *regs) | |
| 8297 | −{ | |
| 8298 | − struct bpf_reg_state *state = NULL; | |
| 8299 | − int i; | |
| 8300 | − | |
| 8301 | − for (i = 0; i < MAX_BPF_FUNC_REG_ARGS; i++) | |
| 8302 | − if (arg_type_is_dynptr(fn->arg_type[i])) { | |
| 8303 | − if (state) { | |
| 8304 | − verbose(env, "verifier internal error: multiple dynptr args\n"); | |
| 8305 | − return NULL; | |
| 8306 | − } | |
| 8307 | − state = ®s[BPF_REG_1 + i]; | |
| 8308 | − } | |
| 8309 | − | |
| 8310 | − if (!state) | |
| 8311 | − verbose(env, "verifier internal error: no dynptr arg found\n"); | |
| 8312 | − | |
| 8313 | − return state; | |
| 8314 | −} | |
| 8315 | − | |
| 8316 | −static int dynptr_id(struct bpf_verifier_env *env, struct bpf_reg_state *reg) | |
| 8317 | −{ | |
| 8318 | − struct bpf_func_state *state = bpf_func(env, reg); | |
| 8319 | − int spi; | |
| 8320 | − | |
| 8321 | − if (reg->type == CONST_PTR_TO_DYNPTR) | |
| 8322 | − return reg->id; | |
| 8323 | − spi = dynptr_get_spi(env, reg); | |
| 8324 | − if (spi < 0) | |
| 8325 | − return spi; | |
| 8326 | − return state->stack[spi].spilled_ptr.id; | |
| 8327 | −} | |
| 8328 | − | |
| 8329 | −static int dynptr_ref_obj_id(struct bpf_verifier_env *env, struct bpf_reg_state *reg) | |
| 8330 | −{ | |
| 8331 | − struct bpf_func_state *state = bpf_func(env, reg); | |
| 8332 | − int spi; | |
| 8333 | − | |
| 8334 | − if (reg->type == CONST_PTR_TO_DYNPTR) | |
| 8335 | − return reg->ref_obj_id; | |
| 8336 | − spi = dynptr_get_spi(env, reg); | |
| 8337 | − if (spi < 0) | |
| 8338 | − return spi; | |
| 8339 | − return state->stack[spi].spilled_ptr.ref_obj_id; | |
| 8340 | −} | |
| 8341 | − | |
| 8342 | −static enum bpf_dynptr_type dynptr_get_type(struct bpf_verifier_env *env, | |
| 8343 | − struct bpf_reg_state *reg) | |
| 8344 | −{ | |
| 8345 | − struct bpf_func_state *state = bpf_func(env, reg); | |
| 8346 | − int spi; | |
| 8347 | − | |
| 8348 | − if (reg->type == CONST_PTR_TO_DYNPTR) | |
| 8349 | − return reg->dynptr.type; | |
| 8350 | − | |
| 8351 | − spi = bpf_get_spi(reg->var_off.value); | |
| 8352 | − if (spi < 0) { | |
| 8353 | − verbose(env, "verifier internal error: invalid spi when querying dynptr type\n"); | |
| 8354 | − return BPF_DYNPTR_TYPE_INVALID; | |
| 8054 | + * cases. var_off always must be 0 for PTR_TO_BTF_ID, hence we | |
| 8055 | + * still need to do checks instead of returning. | |
| 8056 | + */ | |
| 8057 | + return __check_ptr_off_reg(env, reg, argno, true); | |
| 8058 | + case PTR_TO_CTX: | |
| 8059 | + /* | |
| 8060 | + * Allow fixed and variable offsets for syscall context, but | |
| 8061 | + * only when the argument is passed as memory, not ctx, | |
| 8062 | + * otherwise we may get modified ctx in tail called programs and | |
| 8063 | + * global subprogs (that may act as extension prog hooks). | |
| 8064 | + */ | |
| 8065 | + if (arg_type != ARG_PTR_TO_CTX && is_var_ctx_off_allowed(env->prog)) | |
| 8066 | + return 0; | |
| 8067 | + fallthrough; | |
| 8068 | + default: | |
| 8069 | + return __check_ptr_off_reg(env, reg, argno, false); | |
| 8355 | 8070 | } |
| 8356 | − | |
| 8357 | − return state->stack[spi].spilled_ptr.dynptr.type; | |
| 8358 | 8071 | } |
| 8359 | 8072 | |
| 8360 | −static int check_reg_const_str(struct bpf_verifier_env *env, | |
| 8361 | − struct bpf_reg_state *reg, u32 regno) | |
| 8073 | +static int check_arg_const_str(struct bpf_verifier_env *env, | |
| 8074 | + struct bpf_reg_state *reg, argno_t argno) | |
| 8362 | 8075 | { |
| 8363 | 8076 | struct bpf_map *map = reg->map_ptr; |
| 8364 | 8077 | int err; |
| @@ -8370,17 +8083,18 @@ static int check_reg_const_str(struct bpf_verifier_env *env, · 5 unmodified lines | ||
| 8370 | 8083 | return -EINVAL; |
| 8371 | 8084 | |
| 8372 | 8085 | if (map->map_type == BPF_MAP_TYPE_INSN_ARRAY) { |
| 8373 | − verbose(env, "R%d points to insn_array map which cannot be used as const string\n", regno); | |
| 8086 | + verbose(env, "%s points to insn_array map which cannot be used as const string\n", | |
| 8087 | + reg_arg_name(env, argno)); | |
| 8374 | 8088 | return -EACCES; |
| 8375 | 8089 | } |
| 8376 | 8090 | |
| 8377 | 8091 | if (!bpf_map_is_rdonly(map)) { |
| 8378 | − verbose(env, "R%d does not point to a readonly map'\n", regno); | |
| 8092 | + verbose(env, "%s does not point to a readonly map'\n", reg_arg_name(env, argno)); | |
| 8379 | 8093 | return -EACCES; |
| 8380 | 8094 | } |
| 8381 | 8095 | |
| 8382 | 8096 | if (!tnum_is_const(reg->var_off)) { |
| 8383 | − verbose(env, "R%d is not a constant address'\n", regno); | |
| 8097 | + verbose(env, "%s is not a constant address'\n", reg_arg_name(env, argno)); | |
| 8384 | 8098 | return -EACCES; |
| 8385 | 8099 | } |
| 8386 | 8100 | |
| @@ -8389,7 +8103,7 @@ static int check_reg_const_str(struct bpf_verifier_env *env, · 2 unmodified lines | ||
| 8389 | 8103 | return -EACCES; |
| 8390 | 8104 | } |
| 8391 | 8105 | |
| 8392 | − err = check_map_access(env, regno, 0, | |
| 8106 | + err = check_map_access(env, reg, argno, 0, | |
| 8393 | 8107 | map->value_size - reg->var_off.value, false, |
| 8394 | 8108 | ACCESS_HELPER); |
| 8395 | 8109 | if (err) |
| @@ -8471,7 +8185,7 @@ static int get_constant_map_key(struct bpf_verifier_env *env, · 75 unmodified lines | ||
| 8471 | 8185 | return 0; |
| 8472 | 8186 | } |
| 8473 | 8187 | |
| 8474 | −static bool can_elide_value_nullness(enum bpf_map_type type); | |
| 8188 | +static bool can_elide_value_nullness(const struct bpf_map *map); | |
| 8475 | 8189 | |
| 8476 | 8190 | static int check_func_arg(struct bpf_verifier_env *env, u32 arg, |
| 8477 | 8191 | struct bpf_call_arg_meta *meta, |
| @@ -8481,6 +8195,7 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, · 3 unmodified lines | ||
| 8481 | 8195 | u32 regno = BPF_REG_1 + arg; |
| 8482 | 8196 | struct bpf_reg_state *reg = reg_state(env, regno); |
| 8483 | 8197 | enum bpf_arg_type arg_type = fn->arg_type[arg]; |
| 8198 | + argno_t argno = argno_from_arg(arg + 1); | |
| 8484 | 8199 | enum bpf_reg_type type = reg->type; |
| 8485 | 8200 | u32 *arg_btf_id = NULL; |
| 8486 | 8201 | u32 key_size; |
| @@ -8525,56 +8240,24 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, · 38 unmodified lines | ||
| 8525 | 8240 | base_type(arg_type) == ARG_PTR_TO_SPIN_LOCK) |
| 8526 | 8241 | arg_btf_id = fn->arg_btf_id[arg]; |
| 8527 | 8242 | |
| 8528 | − err = check_reg_type(env, regno, arg_type, arg_btf_id, meta); | |
| 8243 | + err = check_reg_type(env, reg, argno_from_reg(regno), arg_type, arg_btf_id, meta); | |
| 8529 | 8244 | if (err) |
| 8530 | 8245 | return err; |
| 8531 | 8246 | |
| 8532 | − err = check_func_arg_reg_off(env, reg, regno, arg_type); | |
| 8247 | + err = check_func_arg_reg_off(env, reg, argno_from_reg(regno), arg_type); | |
| 8533 | 8248 | if (err) |
| 8534 | 8249 | return err; |
| 8535 | 8250 | |
| 8536 | 8251 | skip_type_check: |
| 8537 | − if (arg_type_is_release(arg_type)) { | |
| 8538 | − if (arg_type_is_dynptr(arg_type)) { | |
| 8539 | − struct bpf_func_state *state = bpf_func(env, reg); | |
| 8540 | − int spi; | |
| 8541 | − | |
| 8542 | − /* Only dynptr created on stack can be released, thus | |
| 8543 | − * the get_spi and stack state checks for spilled_ptr | |
| 8544 | − * should only be done before process_dynptr_func for | |
| 8545 | − * PTR_TO_STACK. | |
| 8546 | − */ | |
| 8547 | − if (reg->type == PTR_TO_STACK) { | |
| 8548 | − spi = dynptr_get_spi(env, reg); | |
| 8549 | − if (spi < 0 || !state->stack[spi].spilled_ptr.ref_obj_id) { | |
| 8550 | − verbose(env, "arg %d is an unacquired reference\n", regno); | |
| 8551 | − return -EINVAL; | |
| 8552 | − } | |
| 8553 | − } else { | |
| 8554 | − verbose(env, "cannot release unowned const bpf_dynptr\n"); | |
| 8555 | − return -EINVAL; | |
| 8556 | − } | |
| 8557 | − } else if (!reg->ref_obj_id && !bpf_register_is_null(reg)) { | |
| 8558 | − verbose(env, "R%d must be referenced when passed to release function\n", | |
| 8559 | − regno); | |
| 8560 | − return -EINVAL; | |
| 8561 | − } | |
| 8562 | − if (meta->release_regno) { | |
| 8563 | − verifier_bug(env, "more than one release argument"); | |
| 8564 | − return -EFAULT; | |
| 8565 | − } | |
| 8566 | − meta->release_regno = regno; | |
| 8252 | + if (arg_type_is_release(arg_type) && !arg_type_is_dynptr(arg_type) && | |
| 8253 | + !reg_is_referenced(env, reg) && !bpf_register_is_null(reg)) { | |
| 8254 | + verbose(env, "release helper %s expects referenced PTR_TO_BTF_ID passed to %s\n", | |
| 8255 | + func_id_name(meta->func_id), reg_arg_name(env, argno)); | |
| 8256 | + return -EINVAL; | |
| 8567 | 8257 | } |
| 8568 | 8258 | |
| 8569 | − if (reg->ref_obj_id && base_type(arg_type) != ARG_KPTR_XCHG_DEST) { | |
| 8570 | − if (meta->ref_obj_id) { | |
| 8571 | − verbose(env, "more than one arg with ref_obj_id R%d %u %u", | |
| 8572 | − regno, reg->ref_obj_id, | |
| 8573 | − meta->ref_obj_id); | |
| 8574 | − return -EACCES; | |
| 8575 | − } | |
| 8576 | − meta->ref_obj_id = reg->ref_obj_id; | |
| 8577 | − } | |
| 8259 | + if (reg_is_referenced(env, reg)) | |
| 8260 | + update_ref_obj(&meta->ref_obj, reg); | |
| 8578 | 8261 | |
| 8579 | 8262 | switch (base_type(arg_type)) { |
| 8580 | 8263 | case ARG_CONST_MAP_PTR: |
| @@ -8618,10 +8301,10 @@ skip_type_check: · 37 unmodified lines | ||
| 8618 | 8301 | return -EFAULT; |
| 8619 | 8302 | } |
| 8620 | 8303 | key_size = meta->map.ptr->key_size; |
| 8621 | − err = check_helper_mem_access(env, regno, key_size, BPF_READ, false, NULL); | |
| 8304 | + err = check_helper_mem_access(env, reg, argno_from_reg(regno), key_size, BPF_READ, false, NULL); | |
| 8622 | 8305 | if (err) |
| 8623 | 8306 | return err; |
| 8624 | − if (can_elide_value_nullness(meta->map.ptr->map_type)) { | |
| 8307 | + if (can_elide_value_nullness(meta->map.ptr)) { | |
| 8625 | 8308 | err = get_constant_map_key(env, reg, key_size, &meta->const_map_key); |
| 8626 | 8309 | if (err < 0) { |
| 8627 | 8310 | meta->const_map_key = -1; |
| @@ -8645,7 +8328,7 @@ skip_type_check: · 17 unmodified lines | ||
| 8645 | 8328 | return -EFAULT; |
| 8646 | 8329 | } |
| 8647 | 8330 | meta->raw_mode = arg_type & MEM_UNINIT; |
| 8648 | − err = check_helper_mem_access(env, regno, meta->map.ptr->value_size, | |
| 8331 | + err = check_helper_mem_access(env, reg, argno_from_reg(regno), meta->map.ptr->value_size, | |
| 8649 | 8332 | arg_type & MEM_WRITE ? BPF_WRITE : BPF_READ, |
| 8650 | 8333 | false, meta); |
| 8651 | 8334 | break; |
| @@ -8663,11 +8346,11 @@ skip_type_check: · 11 unmodified lines | ||
| 8663 | 8346 | return -EACCES; |
| 8664 | 8347 | } |
| 8665 | 8348 | if (meta->func_id == BPF_FUNC_spin_lock) { |
| 8666 | − err = process_spin_lock(env, regno, PROCESS_SPIN_LOCK); | |
| 8349 | + err = process_spin_lock(env, reg, argno_from_reg(regno), PROCESS_SPIN_LOCK); | |
| 8667 | 8350 | if (err) |
| 8668 | 8351 | return err; |
| 8669 | 8352 | } else if (meta->func_id == BPF_FUNC_spin_unlock) { |
| 8670 | − err = process_spin_lock(env, regno, 0); | |
| 8353 | + err = process_spin_lock(env, reg, argno_from_reg(regno), 0); | |
| 8671 | 8354 | if (err) |
| 8672 | 8355 | return err; |
| 8673 | 8356 | } else { |
| @@ -8676,7 +8359,7 @@ skip_type_check: · 2 unmodified lines | ||
| 8676 | 8359 | } |
| 8677 | 8360 | break; |
| 8678 | 8361 | case ARG_PTR_TO_TIMER: |
| 8679 | − err = process_timer_helper(env, regno, meta); | |
| 8362 | + err = process_timer_helper(env, reg, argno_from_reg(regno), meta); | |
| 8680 | 8363 | if (err) |
| 8681 | 8364 | return err; |
| 8682 | 8365 | break; |
| @@ -8689,7 +8372,7 @@ skip_type_check: · 6 unmodified lines | ||
| 8689 | 8372 | */ |
| 8690 | 8373 | meta->raw_mode = arg_type & MEM_UNINIT; |
| 8691 | 8374 | if (arg_type & MEM_FIXED_SIZE) { |
| 8692 | − err = check_helper_mem_access(env, regno, fn->arg_size[arg], | |
| 8375 | + err = check_helper_mem_access(env, reg, argno_from_reg(regno), fn->arg_size[arg], | |
| 8693 | 8376 | arg_type & MEM_WRITE ? BPF_WRITE : BPF_READ, |
| 8694 | 8377 | false, meta); |
| 8695 | 8378 | if (err) |
| @@ -8699,19 +8382,22 @@ skip_type_check: · 3 unmodified lines | ||
| 8699 | 8382 | } |
| 8700 | 8383 | break; |
| 8701 | 8384 | case ARG_CONST_SIZE: |
| 8702 | − err = check_mem_size_reg(env, reg, regno, | |
| 8385 | + err = check_mem_size_reg(env, reg_state(env, regno - 1), reg, argno_from_reg(regno - 1), | |
| 8386 | + argno_from_reg(regno), | |
| 8703 | 8387 | fn->arg_type[arg - 1] & MEM_WRITE ? |
| 8704 | 8388 | BPF_WRITE : BPF_READ, |
| 8705 | 8389 | false, meta); |
| 8706 | 8390 | break; |
| 8707 | 8391 | case ARG_CONST_SIZE_OR_ZERO: |
| 8708 | − err = check_mem_size_reg(env, reg, regno, | |
| 8392 | + err = check_mem_size_reg(env, reg_state(env, regno - 1), reg, argno_from_reg(regno - 1), | |
| 8393 | + argno_from_reg(regno), | |
| 8709 | 8394 | fn->arg_type[arg - 1] & MEM_WRITE ? |
| 8710 | 8395 | BPF_WRITE : BPF_READ, |
| 8711 | 8396 | true, meta); |
| 8712 | 8397 | break; |
| 8713 | 8398 | case ARG_PTR_TO_DYNPTR: |
| 8714 | − err = process_dynptr_func(env, regno, insn_idx, arg_type, 0); | |
| 8399 | + err = process_dynptr_func(env, reg, argno_from_reg(regno), insn_idx, arg_type, &meta->ref_obj, | |
| 8400 | + &meta->dynptr); | |
| 8715 | 8401 | if (err) |
| 8716 | 8402 | return err; |
| 8717 | 8403 | break; |
| @@ -8728,7 +8414,7 @@ skip_type_check: · 10 unmodified lines | ||
| 8728 | 8414 | break; |
| 8729 | 8415 | case ARG_PTR_TO_CONST_STR: |
| 8730 | 8416 | { |
| 8731 | − err = check_reg_const_str(env, reg, regno); | |
| 8417 | + err = check_arg_const_str(env, reg, argno_from_reg(regno)); | |
| 8732 | 8418 | if (err) |
| 8733 | 8419 | return err; |
| 8734 | 8420 | break; |
| @@ -9130,11 +8816,29 @@ static bool check_mem_arg_rw_flag_ok(const struct bpf_func_proto *fn) · 395 unmodified lines | ||
| 9130 | 8816 | return true; |
| 9131 | 8817 | } |
| 9132 | 8818 | |
| 9133 | −static int check_func_proto(const struct bpf_func_proto *fn) | |
| 8819 | +static bool check_proto_release_reg(const struct bpf_func_proto *fn, struct bpf_call_arg_meta *meta) | |
| 8820 | +{ | |
| 8821 | + int i; | |
| 8822 | + | |
| 8823 | + for (i = 0; i < ARRAY_SIZE(fn->arg_type); i++) { | |
| 8824 | + enum bpf_arg_type arg_type = fn->arg_type[i]; | |
| 8825 | + | |
| 8826 | + if (arg_type_is_release(arg_type)) { | |
| 8827 | + if (meta->release_regno) | |
| 8828 | + return false; | |
| 8829 | + meta->release_regno = i + 1; | |
| 8830 | + } | |
| 8831 | + } | |
| 8832 | + | |
| 8833 | + return true; | |
| 8834 | +} | |
| 8835 | + | |
| 8836 | +static int check_func_proto(const struct bpf_func_proto *fn, struct bpf_call_arg_meta *meta) | |
| 9134 | 8837 | { |
| 9135 | 8838 | return check_raw_mode_ok(fn) && |
| 9136 | 8839 | check_arg_pair_ok(fn) && |
| 9137 | 8840 | check_mem_arg_rw_flag_ok(fn) && |
| 8841 | + check_proto_release_reg(fn, meta) && | |
| 9138 | 8842 | check_btf_id_ok(fn) ? 0 : -EINVAL; |
| 9139 | 8843 | } |
| 9140 | 8844 | |
| @@ -9181,14 +8885,14 @@ static void mark_pkt_end(struct bpf_verifier_state *vstate, int regn, bool range · 40 unmodified lines | ||
| 9181 | 8885 | reg->range = AT_PKT_END; |
| 9182 | 8886 | } |
| 9183 | 8887 | |
| 9184 | −static int release_reference_nomark(struct bpf_verifier_state *state, int ref_obj_id) | |
| 8888 | +static int release_reference_nomark(struct bpf_verifier_state *state, int id) | |
| 9185 | 8889 | { |
| 9186 | 8890 | int i; |
| 9187 | 8891 | |
| 9188 | 8892 | for (i = 0; i < state->acquired_refs; i++) { |
| 9189 | 8893 | if (state->refs[i].type != REF_TYPE_PTR) |
| 9190 | 8894 | continue; |
| 9191 | − if (state->refs[i].id == ref_obj_id) { | |
| 8895 | + if (state->refs[i].id == id) { | |
| 9192 | 8896 | release_reference_state(state, i); |
| 9193 | 8897 | return 0; |
| 9194 | 8898 | } |
| @@ -9196,26 +8900,83 @@ static int release_reference_nomark(struct bpf_verifier_state *state, int ref_ob · 1 unmodified line | ||
| 9196 | 8900 | return -EINVAL; |
| 9197 | 8901 | } |
| 9198 | 8902 | |
| 9199 | −/* The pointer with the specified id has released its reference to kernel | |
| 9200 | − * resources. Identify all copies of the same pointer and clear the reference. | |
| 9201 | − * | |
| 9202 | − * This is the release function corresponding to acquire_reference(). Idempotent. | |
| 9203 | − */ | |
| 9204 | −static int release_reference(struct bpf_verifier_env *env, int ref_obj_id) | |
| 8903 | +static int idstack_push(struct bpf_idmap *idmap, u32 id) | |
| 8904 | +{ | |
| 8905 | + int i; | |
| 8906 | + | |
| 8907 | + if (!id) | |
| 8908 | + return 0; | |
| 8909 | + | |
| 8910 | + for (i = 0; i < idmap->cnt; i++) | |
| 8911 | + if (idmap->map[i].old == id) | |
| 8912 | + return 0; | |
| 8913 | + | |
| 8914 | + if (WARN_ON_ONCE(idmap->cnt >= BPF_ID_MAP_SIZE)) | |
| 8915 | + return -EFAULT; | |
| 8916 | + | |
| 8917 | + idmap->map[idmap->cnt++].old = id; | |
| 8918 | + return 0; | |
| 8919 | +} | |
| 8920 | + | |
| 8921 | +static int idstack_pop(struct bpf_idmap *idmap) | |
| 8922 | +{ | |
| 8923 | + if (!idmap->cnt) | |
| 8924 | + return 0; | |
| 8925 | + | |
| 8926 | + return idmap->map[--idmap->cnt].old; | |
| 8927 | +} | |
| 8928 | + | |
| 8929 | +/* Release id and objects derived from it iteratively in a DFS manner */ | |
| 8930 | +static int release_reference(struct bpf_verifier_env *env, int id) | |
| 9205 | 8931 | { |
| 8932 | + u32 mask = (1 << STACK_SPILL) | (1 << STACK_DYNPTR); | |
| 9206 | 8933 | struct bpf_verifier_state *vstate = env->cur_state; |
| 8934 | + struct bpf_idmap *idstack = &env->idmap_scratch; | |
| 8935 | + struct bpf_stack_state *stack; | |
| 9207 | 8936 | struct bpf_func_state *state; |
| 9208 | 8937 | struct bpf_reg_state *reg; |
| 9209 | − int err; | |
| 8938 | + int i, err; | |
| 9210 | 8939 | |
| 9211 | − err = release_reference_nomark(vstate, ref_obj_id); | |
| 8940 | + idstack->cnt = 0; | |
| 8941 | + err = idstack_push(idstack, id); | |
| 9212 | 8942 | if (err) |
| 9213 | 8943 | return err; |
| 9214 | 8944 | |
| 9215 | − bpf_for_each_reg_in_vstate(vstate, state, reg, ({ | |
| 9216 | − if (reg->ref_obj_id == ref_obj_id) | |
| 9217 | − mark_reg_invalid(env, reg); | |
| 9218 | − })); | |
| 8945 | + if (find_reference_state(vstate, id)) | |
| 8946 | + WARN_ON_ONCE(release_reference_nomark(vstate, id)); | |
| 8947 | + | |
| 8948 | + while ((id = idstack_pop(idstack))) { | |
| 8949 | + /* | |
| 8950 | + * Child references are inaccessible after parent is released, | |
| 8951 | + * any child references that exist at this point are a leak. | |
| 8952 | + */ | |
| 8953 | + for (i = 0; i < vstate->acquired_refs; i++) { | |
| 8954 | + if (vstate->refs[i].type != REF_TYPE_PTR) | |
| 8955 | + continue; | |
| 8956 | + if (vstate->refs[i].parent_id != id) | |
| 8957 | + continue; | |
| 8958 | + verbose(env, "Leaking reference id=%d alloc_insn=%d. Release it first.\n", | |
| 8959 | + vstate->refs[i].id, vstate->refs[i].insn_idx); | |
| 8960 | + return -EINVAL; | |
| 8961 | + } | |
| 8962 | + | |
| 8963 | + bpf_for_each_reg_in_vstate_mask(vstate, state, reg, stack, mask, ({ | |
| 8964 | + if (reg->id != id && reg->parent_id != id) | |
| 8965 | + continue; | |
| 8966 | + | |
| 8967 | + /* Free objects derived from the current object */ | |
| 8968 | + if (reg->parent_id == id) { | |
| 8969 | + err = idstack_push(idstack, reg->id); | |
| 8970 | + if (err) | |
| 8971 | + return err; | |
| 8972 | + } | |
| 8973 | + | |
| 8974 | + if (!stack || stack->slot_type[BPF_REG_SIZE - 1] == STACK_SPILL) | |
| 8975 | + mark_reg_invalid(env, reg); | |
| 8976 | + else if (stack->slot_type[BPF_REG_SIZE - 1] == STACK_DYNPTR) | |
| 8977 | + invalidate_dynptr(env, stack); | |
| 8978 | + })); | |
| 8979 | + } | |
| 9219 | 8980 | |
| 9220 | 8981 | return 0; |
| 9221 | 8982 | } |
| @@ -9231,6 +8992,42 @@ static void invalidate_non_owning_refs(struct bpf_verifier_env *env) · 9 unmodified lines | ||
| 9231 | 8992 | })); |
| 9232 | 8993 | } |
| 9233 | 8994 | |
| 8995 | +static void invalidate_rcu_protected_refs(struct bpf_verifier_env *env) | |
| 8996 | +{ | |
| 8997 | + struct bpf_stack_state *stack; | |
| 8998 | + struct bpf_func_state *state; | |
| 8999 | + struct bpf_reg_state *reg; | |
| 9000 | + u32 clear_mask = (1 << STACK_SPILL) | (1 << STACK_ITER); | |
| 9001 | + | |
| 9002 | + bpf_for_each_reg_in_vstate_mask(env->cur_state, state, reg, stack, clear_mask, ({ | |
| 9003 | + if (reg->type & MEM_RCU) { | |
| 9004 | + reg->type &= ~(MEM_RCU | PTR_MAYBE_NULL); | |
| 9005 | + reg->type |= PTR_UNTRUSTED; | |
| 9006 | + } | |
| 9007 | + })); | |
| 9008 | +} | |
| 9009 | + | |
| 9010 | +static int ref_convert_alloc_rcu_protected(struct bpf_verifier_env *env, u32 id) | |
| 9011 | +{ | |
| 9012 | + struct bpf_func_state *state; | |
| 9013 | + struct bpf_reg_state *reg; | |
| 9014 | + int err; | |
| 9015 | + | |
| 9016 | + err = release_reference_nomark(env->cur_state, id); | |
| 9017 | + | |
| 9018 | + bpf_for_each_reg_in_vstate(env->cur_state, state, reg, ({ | |
| 9019 | + if (reg->id != id) | |
| 9020 | + continue; | |
| 9021 | + if ((reg->type & MEM_ALLOC) && (reg->type & MEM_PERCPU)) { | |
| 9022 | + reg->id = 0; | |
| 9023 | + reg->type &= ~MEM_ALLOC; | |
| 9024 | + reg->type |= MEM_RCU; | |
| 9025 | + } | |
| 9026 | + })); | |
| 9027 | + | |
| 9028 | + return err; | |
| 9029 | +} | |
| 9030 | + | |
| 9234 | 9031 | static void clear_caller_saved_regs(struct bpf_verifier_env *env, |
| 9235 | 9032 | struct bpf_reg_state *regs) |
| 9236 | 9033 | { |
| @@ -9243,6 +9040,15 @@ static void clear_caller_saved_regs(struct bpf_verifier_env *env, · 6 unmodified lines | ||
| 9243 | 9040 | } |
| 9244 | 9041 | } |
| 9245 | 9042 | |
| 9043 | +static void invalidate_outgoing_stack_args(const struct bpf_verifier_env *env, | |
| 9044 | + struct bpf_func_state *state) | |
| 9045 | +{ | |
| 9046 | + int i, nslots = state->out_stack_arg_cnt; | |
| 9047 | + | |
| 9048 | + for (i = 0; i < nslots; i++) | |
| 9049 | + bpf_mark_reg_not_init(env, &state->stack_arg_regs[i]); | |
| 9050 | +} | |
| 9051 | + | |
| 9246 | 9052 | typedef int (*set_callee_state_fn)(struct bpf_verifier_env *env, |
| 9247 | 9053 | struct bpf_func_state *caller, |
| 9248 | 9054 | struct bpf_func_state *callee, |
| @@ -9305,11 +9111,23 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog, · 56 unmodified lines | ||
| 9305 | 9111 | struct bpf_reg_state *regs) |
| 9306 | 9112 | { |
| 9307 | 9113 | struct bpf_subprog_info *sub = subprog_info(env, subprog); |
| 9114 | + struct bpf_func_state *caller = cur_func(env); | |
| 9308 | 9115 | struct bpf_verifier_log *log = &env->log; |
| 9116 | + struct ref_obj_desc ref_obj = {}; | |
| 9309 | 9117 | u32 i; |
| 9310 | − int ret; | |
| 9118 | + int ret, err; | |
| 9311 | 9119 | |
| 9312 | 9120 | ret = btf_prepare_func_args(env, subprog); |
| 9121 | + if (ret) { | |
| 9122 | + if (bpf_in_stack_arg_cnt(sub) > 0) { | |
| 9123 | + err = check_outgoing_stack_args(env, caller, sub->arg_cnt); | |
| 9124 | + if (err) | |
| 9125 | + return err; | |
| 9126 | + } | |
| 9127 | + return ret; | |
| 9128 | + } | |
| 9129 | + | |
| 9130 | + ret = check_outgoing_stack_args(env, caller, sub->arg_cnt); | |
| 9313 | 9131 | if (ret) |
| 9314 | 9132 | return ret; |
| 9315 | 9133 | |
| @@ -9317,13 +9135,13 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog, · 1 unmodified line | ||
| 9317 | 9135 | * verifier sees. |
| 9318 | 9136 | */ |
| 9319 | 9137 | for (i = 0; i < sub->arg_cnt; i++) { |
| 9320 | − u32 regno = i + 1; | |
| 9321 | − struct bpf_reg_state *reg = ®s[regno]; | |
| 9138 | + argno_t argno = argno_from_arg(i + 1); | |
| 9139 | + struct bpf_reg_state *reg = get_func_arg_reg(caller, regs, i); | |
| 9322 | 9140 | struct bpf_subprog_arg_info *arg = &sub->args[i]; |
| 9323 | 9141 | |
| 9324 | 9142 | if (arg->arg_type == ARG_ANYTHING) { |
| 9325 | 9143 | if (reg->type != SCALAR_VALUE) { |
| 9326 | − bpf_log(log, "R%d is not a scalar\n", regno); | |
| 9144 | + bpf_log(log, "%s is not a scalar\n", reg_arg_name(env, argno)); | |
| 9327 | 9145 | return -EINVAL; |
| 9328 | 9146 | } |
| 9329 | 9147 | } else if (arg->arg_type & PTR_UNTRUSTED) { |
| @@ -9333,24 +9151,26 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog, · 3 unmodified lines | ||
| 9333 | 9151 | * invalid memory access. |
| 9334 | 9152 | */ |
| 9335 | 9153 | } else if (arg->arg_type == ARG_PTR_TO_CTX) { |
| 9336 | − ret = check_func_arg_reg_off(env, reg, regno, ARG_PTR_TO_CTX); | |
| 9154 | + ret = check_func_arg_reg_off(env, reg, argno, ARG_PTR_TO_CTX); | |
| 9337 | 9155 | if (ret < 0) |
| 9338 | 9156 | return ret; |
| 9339 | 9157 | /* If function expects ctx type in BTF check that caller |
| 9340 | 9158 | * is passing PTR_TO_CTX. |
| 9341 | 9159 | */ |
| 9342 | 9160 | if (reg->type != PTR_TO_CTX) { |
| 9343 | − bpf_log(log, "arg#%d expects pointer to ctx\n", i); | |
| 9161 | + bpf_log(log, "%s expects pointer to ctx\n", | |
| 9162 | + reg_arg_name(env, argno)); | |
| 9344 | 9163 | return -EINVAL; |
| 9345 | 9164 | } |
| 9346 | 9165 | } else if (base_type(arg->arg_type) == ARG_PTR_TO_MEM) { |
| 9347 | − ret = check_func_arg_reg_off(env, reg, regno, ARG_DONTCARE); | |
| 9166 | + ret = check_func_arg_reg_off(env, reg, argno, ARG_DONTCARE); | |
| 9348 | 9167 | if (ret < 0) |
| 9349 | 9168 | return ret; |
| 9350 | − if (check_mem_reg(env, reg, regno, arg->mem_size)) | |
| 9169 | + if (check_mem_reg(env, reg, argno, arg->mem_size)) | |
| 9351 | 9170 | return -EINVAL; |
| 9352 | 9171 | if (!(arg->arg_type & PTR_MAYBE_NULL) && (reg->type & PTR_MAYBE_NULL)) { |
| 9353 | − bpf_log(log, "arg#%d is expected to be non-NULL\n", i); | |
| 9172 | + bpf_log(log, "%s is expected to be non-NULL\n", | |
| 9173 | + reg_arg_name(env, argno)); | |
| 9354 | 9174 | return -EINVAL; |
| 9355 | 9175 | } |
| 9356 | 9176 | } else if (base_type(arg->arg_type) == ARG_PTR_TO_ARENA) { |
| @@ -9362,15 +9182,16 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog, · 5 unmodified lines | ||
| 9362 | 9182 | * run-time debug nightmare. |
| 9363 | 9183 | */ |
| 9364 | 9184 | if (reg->type != PTR_TO_ARENA && reg->type != SCALAR_VALUE) { |
| 9365 | − bpf_log(log, "R%d is not a pointer to arena or scalar.\n", regno); | |
| 9185 | + bpf_log(log, "%s is not a pointer to arena or scalar.\n", | |
| 9186 | + reg_arg_name(env, argno)); | |
| 9366 | 9187 | return -EINVAL; |
| 9367 | 9188 | } |
| 9368 | − } else if (arg->arg_type == (ARG_PTR_TO_DYNPTR | MEM_RDONLY)) { | |
| 9369 | − ret = check_func_arg_reg_off(env, reg, regno, ARG_PTR_TO_DYNPTR); | |
| 9189 | + } else if (arg->arg_type == ARG_PTR_TO_DYNPTR) { | |
| 9190 | + ret = check_func_arg_reg_off(env, reg, argno, ARG_PTR_TO_DYNPTR); | |
| 9370 | 9191 | if (ret) |
| 9371 | 9192 | return ret; |
| 9372 | 9193 | |
| 9373 | − ret = process_dynptr_func(env, regno, -1, arg->arg_type, 0); | |
| 9194 | + ret = process_dynptr_func(env, reg, argno, -1, arg->arg_type, &ref_obj, NULL); | |
| 9374 | 9195 | if (ret) |
| 9375 | 9196 | return ret; |
| 9376 | 9197 | } else if (base_type(arg->arg_type) == ARG_PTR_TO_BTF_ID) { |
| @@ -9381,12 +9202,13 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env, int subprog, · 4 unmodified lines | ||
| 9381 | 9202 | continue; |
| 9382 | 9203 | |
| 9383 | 9204 | memset(&meta, 0, sizeof(meta)); /* leave func_id as zero */ |
| 9384 | − err = check_reg_type(env, regno, arg->arg_type, &arg->btf_id, &meta); | |
| 9385 | − err = err ?: check_func_arg_reg_off(env, reg, regno, arg->arg_type); | |
| 9205 | + err = check_reg_type(env, reg, argno, arg->arg_type, &arg->btf_id, &meta); | |
| 9206 | + err = err ?: check_func_arg_reg_off(env, reg, argno, arg->arg_type); | |
| 9386 | 9207 | if (err) |
| 9387 | 9208 | return err; |
| 9388 | 9209 | } else { |
| 9389 | − verifier_bug(env, "unrecognized arg#%d type %d", i, arg->arg_type); | |
| 9210 | + verifier_bug(env, "unrecognized %s type %d", | |
| 9211 | + reg_arg_name(env, argno), arg->arg_type); | |
| 9390 | 9212 | return -EFAULT; |
| 9391 | 9213 | } |
| 9392 | 9214 | } |
| @@ -9505,6 +9327,8 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn, · 112 unmodified lines | ||
| 9505 | 9327 | int *insn_idx) |
| 9506 | 9328 | { |
| 9507 | 9329 | struct bpf_verifier_state *state = env->cur_state; |
| 9330 | + struct bpf_subprog_info *caller_info; | |
| 9331 | + u16 callee_incoming, stack_arg_cnt; | |
| 9508 | 9332 | struct bpf_func_state *caller; |
| 9509 | 9333 | int err, subprog, target_insn; |
| 9510 | 9334 | |
| @@ -9547,6 +9371,7 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn, · 36 unmodified lines | ||
| 9547 | 9371 | /* mark global subprog for verifying after main prog */ |
| 9548 | 9372 | subprog_aux(env, subprog)->called = true; |
| 9549 | 9373 | clear_caller_saved_regs(env, caller->regs); |
| 9374 | + invalidate_outgoing_stack_args(env, cur_func(env)); | |
| 9550 | 9375 | |
| 9551 | 9376 | /* All non-void global functions return a 64-bit SCALAR_VALUE. */ |
| 9552 | 9377 | if (!subprog_returns_void(env, subprog)) { |
| @@ -9569,6 +9394,16 @@ static int check_func_call(struct bpf_verifier_env *env, struct bpf_insn *insn, · 16 unmodified lines | ||
| 9569 | 9394 | return 0; |
| 9570 | 9395 | } |
| 9571 | 9396 | |
| 9397 | + /* | |
| 9398 | + * Track caller's total stack arg count (incoming + max outgoing). | |
| 9399 | + * This is needed so the JIT knows how much stack arg space to allocate. | |
| 9400 | + */ | |
| 9401 | + caller_info = &env->subprog_info[caller->subprogno]; | |
| 9402 | + callee_incoming = bpf_in_stack_arg_cnt(&env->subprog_info[subprog]); | |
| 9403 | + stack_arg_cnt = bpf_in_stack_arg_cnt(caller_info) + callee_incoming; | |
| 9404 | + if (stack_arg_cnt > caller_info->stack_arg_cnt) | |
| 9405 | + caller_info->stack_arg_cnt = stack_arg_cnt; | |
| 9406 | + | |
| 9572 | 9407 | /* for regular function entry setup new frame and continue |
| 9573 | 9408 | * from that frame. |
| 9574 | 9409 | */ |
| @@ -9852,9 +9687,9 @@ static bool in_rbtree_lock_required_cb(struct bpf_verifier_env *env) · 277 unmodified lines | ||
| 9852 | 9687 | static bool retval_range_within(struct bpf_retval_range range, const struct bpf_reg_state *reg) |
| 9853 | 9688 | { |
| 9854 | 9689 | if (range.return_32bit) |
| 9855 | − return range.minval <= reg->s32_min_value && reg->s32_max_value <= range.maxval; | |
| 9690 | + return range.minval <= reg_s32_min(reg) && reg_s32_max(reg) <= range.maxval; | |
| 9856 | 9691 | else |
| 9857 | − return range.minval <= reg->smin_value && reg->smax_value <= range.maxval; | |
| 9692 | + return range.minval <= reg_smin(reg) && reg_smax(reg) <= range.maxval; | |
| 9858 | 9693 | } |
| 9859 | 9694 | |
| 9860 | 9695 | static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx) |
| @@ -9926,6 +9761,7 @@ static int prepare_func_exit(struct bpf_verifier_env *env, int *insn_idx) · 65 unmodified lines | ||
| 9926 | 9761 | * bpf_throw, this will be done by copy_verifier_state for extra frames. */ |
| 9927 | 9762 | free_func_state(callee); |
| 9928 | 9763 | state->frame[state->curframe--] = NULL; |
| 9764 | + invalidate_outgoing_stack_args(env, caller); | |
| 9929 | 9765 | |
| 9930 | 9766 | /* for callbacks widen imprecise scalars to make programs like below verify: |
| 9931 | 9767 | * |
| @@ -9952,7 +9788,9 @@ static int do_refine_retval_range(struct bpf_verifier_env *env, · 20 unmodified lines | ||
| 9952 | 9788 | int func_id, |
| 9953 | 9789 | struct bpf_call_arg_meta *meta) |
| 9954 | 9790 | { |
| 9791 | + struct bpf_retval_range range; | |
| 9955 | 9792 | struct bpf_reg_state *ret_reg = ®s[BPF_REG_0]; |
| 9793 | + enum bpf_prog_type prog_type = resolve_prog_type(env->prog); | |
| 9956 | 9794 | |
| 9957 | 9795 | if (ret_type != RET_INTEGER) |
| 9958 | 9796 | return 0; |
| @@ -9963,21 +9801,36 @@ static int do_refine_retval_range(struct bpf_verifier_env *env, · 4 unmodified lines | ||
| 9963 | 9801 | case BPF_FUNC_probe_read_str: |
| 9964 | 9802 | case BPF_FUNC_probe_read_kernel_str: |
| 9965 | 9803 | case BPF_FUNC_probe_read_user_str: |
| 9966 | − ret_reg->smax_value = meta->msize_max_value; | |
| 9967 | − ret_reg->s32_max_value = meta->msize_max_value; | |
| 9968 | − ret_reg->smin_value = -MAX_ERRNO; | |
| 9969 | − ret_reg->s32_min_value = -MAX_ERRNO; | |
| 9804 | + reg_set_srange64(ret_reg, -MAX_ERRNO, meta->msize_max_value); | |
| 9805 | + reg_set_srange32(ret_reg, -MAX_ERRNO, meta->msize_max_value); | |
| 9970 | 9806 | reg_bounds_sync(ret_reg); |
| 9971 | 9807 | break; |
| 9972 | 9808 | case BPF_FUNC_get_smp_processor_id: |
| 9973 | − ret_reg->umax_value = nr_cpu_ids - 1; | |
| 9974 | − ret_reg->u32_max_value = nr_cpu_ids - 1; | |
| 9975 | − ret_reg->smax_value = nr_cpu_ids - 1; | |
| 9976 | − ret_reg->s32_max_value = nr_cpu_ids - 1; | |
| 9977 | − ret_reg->umin_value = 0; | |
| 9978 | − ret_reg->u32_min_value = 0; | |
| 9979 | − ret_reg->smin_value = 0; | |
| 9980 | − ret_reg->s32_min_value = 0; | |
| 9809 | + reg_set_urange64(ret_reg, 0, nr_cpu_ids - 1); | |
| 9810 | + reg_set_urange32(ret_reg, 0, nr_cpu_ids - 1); | |
| 9811 | + reg_bounds_sync(ret_reg); | |
| 9812 | + break; | |
| 9813 | + case BPF_FUNC_get_retval: | |
| 9814 | + /* | |
| 9815 | + * bpf_get_retval may see arbitrary value passed by bpf_prog_run_array_cg for | |
| 9816 | + * CGROUP_GETSOCKOPT type. | |
| 9817 | + */ | |
| 9818 | + if (prog_type == BPF_PROG_TYPE_CGROUP_SOCKOPT && | |
| 9819 | + env->prog->expected_attach_type == BPF_CGROUP_GETSOCKOPT) | |
| 9820 | + break; | |
| 9821 | + | |
| 9822 | + if (prog_type == BPF_PROG_TYPE_LSM && | |
| 9823 | + env->prog->expected_attach_type == BPF_LSM_CGROUP) { | |
| 9824 | + if (!env->prog->aux->attach_func_proto->type) | |
| 9825 | + break; | |
| 9826 | + bpf_lsm_get_retval_range(env->prog, &range); | |
| 9827 | + } else { | |
| 9828 | + range.minval = -MAX_ERRNO; | |
| 9829 | + range.maxval = 0; | |
| 9830 | + } | |
| 9831 | + | |
| 9832 | + reg_set_srange64(ret_reg, range.minval, range.maxval); | |
| 9833 | + reg_set_srange32(ret_reg, range.minval, range.maxval); | |
| 9981 | 9834 | reg_bounds_sync(ret_reg); |
| 9982 | 9835 | break; |
| 9983 | 9836 | } |
| @@ -10086,7 +9939,7 @@ static int check_reference_leak(struct bpf_verifier_env *env, bool exception_exi · 102 unmodified lines | ||
| 10086 | 9939 | * kernel. Type checks are performed later in check_return_code. |
| 10087 | 9940 | */ |
| 10088 | 9941 | if (type == BPF_PROG_TYPE_STRUCT_OPS && !exception_exit && |
| 10089 | − reg->ref_obj_id == state->refs[i].id) | |
| 9942 | + reg->id == state->refs[i].id) | |
| 10090 | 9943 | continue; |
| 10091 | 9944 | verbose(env, "Unreleased reference id=%d alloc_insn=%d\n", |
| 10092 | 9945 | state->refs[i].id, state->refs[i].insn_idx); |
| @@ -10221,13 +10074,16 @@ static void update_loop_inline_state(struct bpf_verifier_env *env, u32 subprogno · 128 unmodified lines | ||
| 10221 | 10074 | state->callback_subprogno == subprogno); |
| 10222 | 10075 | } |
| 10223 | 10076 | |
| 10224 | −/* Returns whether or not the given map type can potentially elide | |
| 10077 | +/* Returns whether or not the given map can potentially elide | |
| 10225 | 10078 | * lookup return value nullness check. This is possible if the key |
| 10226 | 10079 | * is statically known. |
| 10227 | 10080 | */ |
| 10228 | −static bool can_elide_value_nullness(enum bpf_map_type type) | |
| 10081 | +static bool can_elide_value_nullness(const struct bpf_map *map) | |
| 10229 | 10082 | { |
| 10230 | − switch (type) { | |
| 10083 | + if (map->map_flags & BPF_F_INNER_MAP) | |
| 10084 | + return false; | |
| 10085 | + | |
| 10086 | + switch (map->map_type) { | |
| 10231 | 10087 | case BPF_MAP_TYPE_ARRAY: |
| 10232 | 10088 | case BPF_MAP_TYPE_PERCPU_ARRAY: |
| 10233 | 10089 | return true; |
| @@ -10272,6 +10128,24 @@ static const char *non_sleepable_context_description(struct bpf_verifier_env *en · 38 unmodified lines | ||
| 10272 | 10128 | return "non-sleepable prog"; |
| 10273 | 10129 | } |
| 10274 | 10130 | |
| 10131 | +static int release_reg(struct bpf_verifier_env *env, struct bpf_reg_state *reg, | |
| 10132 | + bool convert_rcu, bool release_dynptr) | |
| 10133 | +{ | |
| 10134 | + int err = -EINVAL; | |
| 10135 | + | |
| 10136 | + if (bpf_register_is_null(reg)) | |
| 10137 | + return 0; | |
| 10138 | + | |
| 10139 | + if (release_dynptr) | |
| 10140 | + err = unmark_stack_slots_dynptr(env, reg); | |
| 10141 | + else if (convert_rcu) | |
| 10142 | + err = ref_convert_alloc_rcu_protected(env, reg->id); | |
| 10143 | + else if (reg_is_referenced(env, reg)) | |
| 10144 | + err = release_reference(env, reg->id); | |
| 10145 | + | |
| 10146 | + return err; | |
| 10147 | +} | |
| 10148 | + | |
| 10275 | 10149 | static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn, |
| 10276 | 10150 | int *insn_idx_p) |
| 10277 | 10151 | { |
| @@ -10321,7 +10195,7 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn · 43 unmodified lines | ||
| 10321 | 10195 | memset(&meta, 0, sizeof(meta)); |
| 10322 | 10196 | meta.pkt_access = fn->pkt_access; |
| 10323 | 10197 | |
| 10324 | − err = check_func_proto(fn); | |
| 10198 | + err = check_func_proto(fn, &meta); | |
| 10325 | 10199 | if (err) { |
| 10326 | 10200 | verifier_bug(env, "incorrect func proto %s#%d", func_id_name(func_id), func_id); |
| 10327 | 10201 | return err; |
| @@ -10353,55 +10227,26 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn · 25 unmodified lines | ||
| 10353 | 10227 | if (err) |
| 10354 | 10228 | return err; |
| 10355 | 10229 | |
| 10230 | + regs = cur_regs(env); | |
| 10231 | + | |
| 10356 | 10232 | /* Mark slots with STACK_MISC in case of raw mode, stack offset |
| 10357 | 10233 | * is inferred from register state. |
| 10358 | 10234 | */ |
| 10359 | 10235 | for (i = 0; i < meta.access_size; i++) { |
| 10360 | − err = check_mem_access(env, insn_idx, meta.regno, i, BPF_B, | |
| 10236 | + err = check_mem_access(env, insn_idx, regs + meta.regno, argno_from_reg(meta.regno), i, BPF_B, | |
| 10361 | 10237 | BPF_WRITE, -1, false, false); |
| 10362 | 10238 | if (err) |
| 10363 | 10239 | return err; |
| 10364 | 10240 | } |
| 10365 | 10241 | |
| 10366 | − regs = cur_regs(env); | |
| 10367 | − | |
| 10368 | 10242 | if (meta.release_regno) { |
| 10369 | − err = -EINVAL; | |
| 10370 | − if (arg_type_is_dynptr(fn->arg_type[meta.release_regno - BPF_REG_1])) { | |
| 10371 | − err = unmark_stack_slots_dynptr(env, ®s[meta.release_regno]); | |
| 10372 | − } else if (func_id == BPF_FUNC_kptr_xchg && meta.ref_obj_id) { | |
| 10373 | − u32 ref_obj_id = meta.ref_obj_id; | |
| 10374 | − bool in_rcu = in_rcu_cs(env); | |
| 10375 | − struct bpf_func_state *state; | |
| 10376 | − struct bpf_reg_state *reg; | |
| 10377 | − | |
| 10378 | − err = release_reference_nomark(env->cur_state, ref_obj_id); | |
| 10379 | − if (!err) { | |
| 10380 | − bpf_for_each_reg_in_vstate(env->cur_state, state, reg, ({ | |
| 10381 | − if (reg->ref_obj_id == ref_obj_id) { | |
| 10382 | − if (in_rcu && (reg->type & MEM_ALLOC) && (reg->type & MEM_PERCPU)) { | |
| 10383 | − reg->ref_obj_id = 0; | |
| 10384 | − reg->type &= ~MEM_ALLOC; | |
| 10385 | − reg->type |= MEM_RCU; | |
| 10386 | − } else { | |
| 10387 | − mark_reg_invalid(env, reg); | |
| 10388 | − } | |
| 10389 | − } | |
| 10390 | − })); | |
| 10391 | − } | |
| 10392 | − } else if (meta.ref_obj_id) { | |
| 10393 | − err = release_reference(env, meta.ref_obj_id); | |
| 10394 | − } else if (bpf_register_is_null(®s[meta.release_regno])) { | |
| 10395 | − /* meta.ref_obj_id can only be 0 if register that is meant to be | |
| 10396 | − * released is NULL, which must be > R0. | |
| 10397 | − */ | |
| 10398 | − err = 0; | |
| 10399 | − } | |
| 10400 | − if (err) { | |
| 10401 | − verbose(env, "func %s#%d reference has not been acquired before\n", | |
| 10402 | − func_id_name(func_id), func_id); | |
| 10243 | + struct bpf_reg_state *reg = ®s[meta.release_regno]; | |
| 10244 | + bool convert_rcu = (func_id == BPF_FUNC_kptr_xchg) && in_rcu_cs(env) && | |
| 10245 | + (reg->type & MEM_ALLOC) && (reg->type & MEM_PERCPU); | |
| 10246 | + | |
| 10247 | + err = release_reg(env, reg, convert_rcu, !!meta.dynptr.id); | |
| 10248 | + if (err) | |
| 10403 | 10249 | return err; |
| 10404 | − } | |
| 10405 | 10250 | } |
| 10406 | 10251 | |
| 10407 | 10252 | switch (func_id) { |
| @@ -10442,7 +10287,7 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn · 34 unmodified lines | ||
| 10442 | 10287 | err = mark_chain_precision(env, BPF_REG_1); |
| 10443 | 10288 | if (err) |
| 10444 | 10289 | return err; |
| 10445 | − if (cur_func(env)->callback_depth < regs[BPF_REG_1].umax_value) { | |
| 10290 | + if (cur_func(env)->callback_depth < reg_umax(®s[BPF_REG_1])) { | |
| 10446 | 10291 | err = push_callback_call(env, insn, insn_idx, meta.subprogno, |
| 10447 | 10292 | set_loop_callback_state); |
| 10448 | 10293 | } else { |
| @@ -10460,6 +10305,24 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn · 11 unmodified lines | ||
| 10460 | 10305 | } |
| 10461 | 10306 | break; |
| 10462 | 10307 | case BPF_FUNC_set_retval: |
| 10308 | + { | |
| 10309 | + struct bpf_retval_range range = { | |
| 10310 | + .minval = -MAX_ERRNO, | |
| 10311 | + .maxval = 0, | |
| 10312 | + .return_32bit = true | |
| 10313 | + }; | |
| 10314 | + struct bpf_reg_state *r1 = ®s[BPF_REG_1]; | |
| 10315 | + | |
| 10316 | + if (r1->type != SCALAR_VALUE) { | |
| 10317 | + verbose(env, "R1 is not a scalar\n"); | |
| 10318 | + return -EINVAL; | |
| 10319 | + } | |
| 10320 | + | |
| 10321 | + /* CGROUP_GETSOCKOPT is allowed to return arbitrary value */ | |
| 10322 | + if (prog_type == BPF_PROG_TYPE_CGROUP_SOCKOPT && | |
| 10323 | + env->prog->expected_attach_type == BPF_CGROUP_GETSOCKOPT) | |
| 10324 | + break; | |
| 10325 | + | |
| 10463 | 10326 | if (prog_type == BPF_PROG_TYPE_LSM && |
| 10464 | 10327 | env->prog->expected_attach_type == BPF_LSM_CGROUP) { |
| 10465 | 10328 | if (!env->prog->aux->attach_func_proto->type) { |
| @@ -10469,54 +10332,24 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn · 3 unmodified lines | ||
| 10469 | 10332 | verbose(env, "BPF_LSM_CGROUP that attach to void LSM hooks can't modify return value!\n"); |
| 10470 | 10333 | return -EINVAL; |
| 10471 | 10334 | } |
| 10472 | − } | |
| 10473 | − break; | |
| 10474 | − case BPF_FUNC_dynptr_data: | |
| 10475 | − { | |
| 10476 | − struct bpf_reg_state *reg; | |
| 10477 | − int id, ref_obj_id; | |
| 10478 | − | |
| 10479 | − reg = get_dynptr_arg_reg(env, fn, regs); | |
| 10480 | − if (!reg) | |
| 10481 | − return -EFAULT; | |
| 10482 | − | |
| 10483 | − | |
| 10484 | − if (meta.dynptr_id) { | |
| 10485 | − verifier_bug(env, "meta.dynptr_id already set"); | |
| 10486 | − return -EFAULT; | |
| 10487 | − } | |
| 10488 | − if (meta.ref_obj_id) { | |
| 10489 | − verifier_bug(env, "meta.ref_obj_id already set"); | |
| 10490 | − return -EFAULT; | |
| 10335 | + bpf_lsm_get_retval_range(env->prog, &range); | |
| 10491 | 10336 | } |
| 10492 | 10337 | |
| 10493 | − id = dynptr_id(env, reg); | |
| 10494 | − if (id < 0) { | |
| 10495 | − verifier_bug(env, "failed to obtain dynptr id"); | |
| 10496 | − return id; | |
| 10497 | − } | |
| 10338 | + err = mark_chain_precision(env, BPF_REG_1); | |
| 10339 | + if (err) | |
| 10340 | + return err; | |
| 10498 | 10341 | |
| 10499 | − ref_obj_id = dynptr_ref_obj_id(env, reg); | |
| 10500 | − if (ref_obj_id < 0) { | |
| 10501 | − verifier_bug(env, "failed to obtain dynptr ref_obj_id"); | |
| 10502 | − return ref_obj_id; | |
| 10342 | + if (!retval_range_within(range, r1)) { | |
| 10343 | + verbose_invalid_scalar(env, r1, range, "At bpf_set_retval", "R1"); | |
| 10344 | + return -EINVAL; | |
| 10503 | 10345 | } |
| 10504 | 10346 | |
| 10505 | − meta.dynptr_id = id; | |
| 10506 | − meta.ref_obj_id = ref_obj_id; | |
| 10507 | − | |
| 10508 | 10347 | break; |
| 10509 | 10348 | } |
| 10510 | 10349 | case BPF_FUNC_dynptr_write: |
| 10511 | 10350 | { |
| 10512 | − enum bpf_dynptr_type dynptr_type; | |
| 10513 | − struct bpf_reg_state *reg; | |
| 10514 | − | |
| 10515 | − reg = get_dynptr_arg_reg(env, fn, regs); | |
| 10516 | − if (!reg) | |
| 10517 | − return -EFAULT; | |
| 10351 | + enum bpf_dynptr_type dynptr_type = meta.dynptr.type; | |
| 10518 | 10352 | |
| 10519 | − dynptr_type = dynptr_get_type(env, reg); | |
| 10520 | 10353 | if (dynptr_type == BPF_DYNPTR_TYPE_INVALID) |
| 10521 | 10354 | return -EFAULT; |
| 10522 | 10355 | |
| @@ -10560,6 +10393,7 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn · 37 unmodified lines | ||
| 10560 | 10393 | bpf_mark_reg_not_init(env, ®s[caller_saved[i]]); |
| 10561 | 10394 | check_reg_arg(env, caller_saved[i], DST_OP_NO_MARK); |
| 10562 | 10395 | } |
| 10396 | + invalidate_outgoing_stack_args(env, cur_func(env)); | |
| 10563 | 10397 | |
| 10564 | 10398 | /* helper call returns 64-bit value. */ |
| 10565 | 10399 | regs[BPF_REG_0].subreg_def = DEF_NOT_SUBREG; |
| @@ -10589,7 +10423,7 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn · 23 unmodified lines | ||
| 10589 | 10423 | } |
| 10590 | 10424 | |
| 10591 | 10425 | if (func_id == BPF_FUNC_map_lookup_elem && |
| 10592 | − can_elide_value_nullness(meta.map.ptr->map_type) && | |
| 10426 | + can_elide_value_nullness(meta.map.ptr) && | |
| 10593 | 10427 | meta.const_map_key >= 0 && |
| 10594 | 10428 | meta.const_map_key < meta.map.ptr->max_entries) |
| 10595 | 10429 | ret_flag &= ~PTR_MAYBE_NULL; |
| @@ -10701,29 +10535,45 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn · 105 unmodified lines | ||
| 10701 | 10535 | if (type_may_be_null(regs[BPF_REG_0].type)) |
| 10702 | 10536 | regs[BPF_REG_0].id = ++env->id_gen; |
| 10703 | 10537 | |
| 10704 | − if (helper_multiple_ref_obj_use(func_id, meta.map.ptr)) { | |
| 10705 | − verifier_bug(env, "func %s#%d sets ref_obj_id more than once", | |
| 10706 | − func_id_name(func_id), func_id); | |
| 10707 | − return -EFAULT; | |
| 10708 | − } | |
| 10538 | + if (is_ptr_cast_function(func_id) && | |
| 10539 | + find_reference_state(env->cur_state, meta.ref_obj.id)) { | |
| 10540 | + struct bpf_verifier_state *branch; | |
| 10541 | + struct bpf_reg_state *r0; | |
| 10542 | + | |
| 10543 | + err = validate_ref_obj(env, &meta.ref_obj); | |
| 10544 | + if (err) | |
| 10545 | + return err; | |
| 10546 | + | |
| 10547 | + /* | |
| 10548 | + * In order for a release of any of the original or cast pointers | |
| 10549 | + * to invalidate all other pointers, reuse the same reference id for | |
| 10550 | + * the cast result. | |
| 10551 | + * This reference id can't be used for nullness propagation, | |
| 10552 | + * as cast might return NULL for a non-NULL input. | |
| 10553 | + * Hence, explore the NULL case as a separate branch. | |
| 10554 | + */ | |
| 10555 | + branch = push_stack(env, env->insn_idx + 1, env->insn_idx, false); | |
| 10556 | + if (IS_ERR(branch)) | |
| 10557 | + return PTR_ERR(branch); | |
| 10709 | 10558 | |
| 10710 | − if (is_dynptr_ref_function(func_id)) | |
| 10711 | − regs[BPF_REG_0].dynptr_id = meta.dynptr_id; | |
| 10559 | + r0 = &branch->frame[branch->curframe]->regs[BPF_REG_0]; | |
| 10560 | + __mark_reg_known_zero(r0); | |
| 10561 | + r0->type = SCALAR_VALUE; | |
| 10712 | 10562 | |
| 10713 | − if (is_ptr_cast_function(func_id) || is_dynptr_ref_function(func_id)) { | |
| 10714 | − /* For release_reference() */ | |
| 10715 | − regs[BPF_REG_0].ref_obj_id = meta.ref_obj_id; | |
| 10563 | + regs[BPF_REG_0].type &= ~PTR_MAYBE_NULL; | |
| 10564 | + regs[BPF_REG_0].id = meta.ref_obj.id; | |
| 10716 | 10565 | } else if (is_acquire_function(func_id, meta.map.ptr)) { |
| 10717 | − int id = acquire_reference(env, insn_idx); | |
| 10566 | + int id = acquire_reference(env, insn_idx, 0); | |
| 10718 | 10567 | |
| 10719 | 10568 | if (id < 0) |
| 10720 | 10569 | return id; |
| 10721 | − /* For mark_ptr_or_null_reg() */ | |
| 10570 | + | |
| 10722 | 10571 | regs[BPF_REG_0].id = id; |
| 10723 | − /* For release_reference() */ | |
| 10724 | − regs[BPF_REG_0].ref_obj_id = id; | |
| 10725 | 10572 | } |
| 10726 | 10573 | |
| 10574 | + if (func_id == BPF_FUNC_dynptr_data) | |
| 10575 | + regs[BPF_REG_0].parent_id = meta.dynptr.id; | |
| 10576 | + | |
| 10727 | 10577 | err = do_refine_retval_range(env, regs, fn->ret_type, func_id, &meta); |
| 10728 | 10578 | if (err) |
| 10729 | 10579 | return err; |
| @@ -10819,7 +10669,6 @@ static bool is_kfunc_release(struct bpf_kfunc_call_arg_meta *meta) · 89 unmodified lines | ||
| 10819 | 10669 | return meta->kfunc_flags & KF_RELEASE; |
| 10820 | 10670 | } |
| 10821 | 10671 | |
| 10822 | − | |
| 10823 | 10672 | static bool is_kfunc_destructive(struct bpf_kfunc_call_arg_meta *meta) |
| 10824 | 10673 | { |
| 10825 | 10674 | return meta->kfunc_flags & KF_DESTRUCTIVE; |
| @@ -10896,6 +10745,11 @@ static bool is_kfunc_arg_nullable(const struct btf *btf, const struct btf_param · 70 unmodified lines | ||
| 10896 | 10745 | return btf_param_match_suffix(btf, arg, "__nullable"); |
| 10897 | 10746 | } |
| 10898 | 10747 | |
| 10748 | +static bool is_kfunc_arg_nonown_allowed(const struct btf *btf, const struct btf_param *arg) | |
| 10749 | +{ | |
| 10750 | + return btf_param_match_suffix(btf, arg, "__nonown_allowed"); | |
| 10751 | +} | |
| 10752 | + | |
| 10899 | 10753 | static bool is_kfunc_arg_const_str(const struct btf *btf, const struct btf_param *arg) |
| 10900 | 10754 | { |
| 10901 | 10755 | return btf_param_match_suffix(btf, arg, "__str"); |
| @@ -11136,10 +10990,15 @@ enum special_kfunc_type { · 234 unmodified lines | ||
| 11136 | 10990 | KF_bpf_list_push_front, |
| 11137 | 10991 | KF_bpf_list_push_back_impl, |
| 11138 | 10992 | KF_bpf_list_push_back, |
| 10993 | + KF_bpf_list_add, | |
| 11139 | 10994 | KF_bpf_list_pop_front, |
| 11140 | 10995 | KF_bpf_list_pop_back, |
| 10996 | + KF_bpf_list_del, | |
| 11141 | 10997 | KF_bpf_list_front, |
| 11142 | 10998 | KF_bpf_list_back, |
| 10999 | + KF_bpf_list_is_first, | |
| 11000 | + KF_bpf_list_is_last, | |
| 11001 | + KF_bpf_list_empty, | |
| 11143 | 11002 | KF_bpf_cast_to_kern_ctx, |
| 11144 | 11003 | KF_bpf_rdonly_cast, |
| 11145 | 11004 | KF_bpf_rcu_read_lock, |
| @@ -11204,10 +11063,15 @@ BTF_ID(func, bpf_list_push_front_impl) · 58 unmodified lines | ||
| 11204 | 11063 | BTF_ID(func, bpf_list_push_front) |
| 11205 | 11064 | BTF_ID(func, bpf_list_push_back_impl) |
| 11206 | 11065 | BTF_ID(func, bpf_list_push_back) |
| 11066 | +BTF_ID(func, bpf_list_add) | |
| 11207 | 11067 | BTF_ID(func, bpf_list_pop_front) |
| 11208 | 11068 | BTF_ID(func, bpf_list_pop_back) |
| 11069 | +BTF_ID(func, bpf_list_del) | |
| 11209 | 11070 | BTF_ID(func, bpf_list_front) |
| 11210 | 11071 | BTF_ID(func, bpf_list_back) |
| 11072 | +BTF_ID(func, bpf_list_is_first) | |
| 11073 | +BTF_ID(func, bpf_list_is_last) | |
| 11074 | +BTF_ID(func, bpf_list_empty) | |
| 11211 | 11075 | BTF_ID(func, bpf_cast_to_kern_ctx) |
| 11212 | 11076 | BTF_ID(func, bpf_rdonly_cast) |
| 11213 | 11077 | BTF_ID(func, bpf_rcu_read_lock) |
| @@ -11319,7 +11183,8 @@ static bool is_bpf_list_push_kfunc(u32 func_id) · 105 unmodified lines | ||
| 11319 | 11183 | return func_id == special_kfunc_list[KF_bpf_list_push_front] || |
| 11320 | 11184 | func_id == special_kfunc_list[KF_bpf_list_push_front_impl] || |
| 11321 | 11185 | func_id == special_kfunc_list[KF_bpf_list_push_back] || |
| 11322 | − func_id == special_kfunc_list[KF_bpf_list_push_back_impl]; | |
| 11186 | + func_id == special_kfunc_list[KF_bpf_list_push_back_impl] || | |
| 11187 | + func_id == special_kfunc_list[KF_bpf_list_add]; | |
| 11323 | 11188 | } |
| 11324 | 11189 | |
| 11325 | 11190 | static bool is_bpf_rbtree_add_kfunc(u32 func_id) |
| @@ -11368,15 +11233,12 @@ bool bpf_is_kfunc_pkt_changing(struct bpf_kfunc_call_arg_meta *meta) · 42 unmodified lines | ||
| 11368 | 11233 | } |
| 11369 | 11234 | |
| 11370 | 11235 | static enum kfunc_ptr_arg_type |
| 11371 | −get_kfunc_ptr_arg_type(struct bpf_verifier_env *env, | |
| 11372 | − struct bpf_kfunc_call_arg_meta *meta, | |
| 11236 | +get_kfunc_ptr_arg_type(struct bpf_verifier_env *env, struct bpf_func_state *caller, | |
| 11237 | + struct bpf_reg_state *regs, struct bpf_kfunc_call_arg_meta *meta, | |
| 11373 | 11238 | const struct btf_type *t, const struct btf_type *ref_t, |
| 11374 | 11239 | const char *ref_tname, const struct btf_param *args, |
| 11375 | − int argno, int nargs) | |
| 11240 | + int arg, int nargs, argno_t argno, struct bpf_reg_state *reg) | |
| 11376 | 11241 | { |
| 11377 | − u32 regno = argno + 1; | |
| 11378 | − struct bpf_reg_state *regs = cur_regs(env); | |
| 11379 | − struct bpf_reg_state *reg = ®s[regno]; | |
| 11380 | 11242 | bool arg_mem_size = false; |
| 11381 | 11243 | |
| 11382 | 11244 | if (meta->func_id == special_kfunc_list[KF_bpf_cast_to_kern_ctx] || |
| @@ -11384,9 +11246,9 @@ get_kfunc_ptr_arg_type(struct bpf_verifier_env *env, · 1 unmodified line | ||
| 11384 | 11246 | meta->func_id == special_kfunc_list[KF_bpf_session_cookie]) |
| 11385 | 11247 | return KF_ARG_PTR_TO_CTX; |
| 11386 | 11248 | |
| 11387 | − if (argno + 1 < nargs && | |
| 11388 | − (is_kfunc_arg_mem_size(meta->btf, &args[argno + 1], ®s[regno + 1]) || | |
| 11389 | − is_kfunc_arg_const_mem_size(meta->btf, &args[argno + 1], ®s[regno + 1]))) | |
| 11249 | + if (arg + 1 < nargs && | |
| 11250 | + (is_kfunc_arg_mem_size(meta->btf, &args[arg + 1], get_func_arg_reg(caller, regs, arg + 1)) || | |
| 11251 | + is_kfunc_arg_const_mem_size(meta->btf, &args[arg + 1], get_func_arg_reg(caller, regs, arg + 1)))) | |
| 11390 | 11252 | arg_mem_size = true; |
| 11391 | 11253 | |
| 11392 | 11254 | /* In this function, we verify the kfunc's BTF as per the argument type, |
| @@ -11394,68 +11256,69 @@ get_kfunc_ptr_arg_type(struct bpf_verifier_env *env, · 1 unmodified line | ||
| 11394 | 11256 | * type to our caller. When a set of conditions hold in the BTF type of |
| 11395 | 11257 | * arguments, we resolve it to a known kfunc_ptr_arg_type. |
| 11396 | 11258 | */ |
| 11397 | − if (btf_is_prog_ctx_type(&env->log, meta->btf, t, resolve_prog_type(env->prog), argno)) | |
| 11259 | + if (btf_is_prog_ctx_type(&env->log, meta->btf, t, resolve_prog_type(env->prog), arg)) | |
| 11398 | 11260 | return KF_ARG_PTR_TO_CTX; |
| 11399 | 11261 | |
| 11400 | − if (is_kfunc_arg_nullable(meta->btf, &args[argno]) && bpf_register_is_null(reg) && | |
| 11262 | + if (is_kfunc_arg_nullable(meta->btf, &args[arg]) && bpf_register_is_null(reg) && | |
| 11401 | 11263 | !arg_mem_size) |
| 11402 | 11264 | return KF_ARG_PTR_TO_NULL; |
| 11403 | 11265 | |
| 11404 | − if (is_kfunc_arg_alloc_obj(meta->btf, &args[argno])) | |
| 11266 | + if (is_kfunc_arg_alloc_obj(meta->btf, &args[arg])) | |
| 11405 | 11267 | return KF_ARG_PTR_TO_ALLOC_BTF_ID; |
| 11406 | 11268 | |
| 11407 | − if (is_kfunc_arg_refcounted_kptr(meta->btf, &args[argno])) | |
| 11269 | + if (is_kfunc_arg_refcounted_kptr(meta->btf, &args[arg])) | |
| 11408 | 11270 | return KF_ARG_PTR_TO_REFCOUNTED_KPTR; |
| 11409 | 11271 | |
| 11410 | − if (is_kfunc_arg_dynptr(meta->btf, &args[argno])) | |
| 11272 | + if (is_kfunc_arg_dynptr(meta->btf, &args[arg])) | |
| 11411 | 11273 | return KF_ARG_PTR_TO_DYNPTR; |
| 11412 | 11274 | |
| 11413 | − if (is_kfunc_arg_iter(meta, argno, &args[argno])) | |
| 11275 | + if (is_kfunc_arg_iter(meta, arg, &args[arg])) | |
| 11414 | 11276 | return KF_ARG_PTR_TO_ITER; |
| 11415 | 11277 | |
| 11416 | − if (is_kfunc_arg_list_head(meta->btf, &args[argno])) | |
| 11278 | + if (is_kfunc_arg_list_head(meta->btf, &args[arg])) | |
| 11417 | 11279 | return KF_ARG_PTR_TO_LIST_HEAD; |
| 11418 | 11280 | |
| 11419 | − if (is_kfunc_arg_list_node(meta->btf, &args[argno])) | |
| 11281 | + if (is_kfunc_arg_list_node(meta->btf, &args[arg])) | |
| 11420 | 11282 | return KF_ARG_PTR_TO_LIST_NODE; |
| 11421 | 11283 | |
| 11422 | − if (is_kfunc_arg_rbtree_root(meta->btf, &args[argno])) | |
| 11284 | + if (is_kfunc_arg_rbtree_root(meta->btf, &args[arg])) | |
| 11423 | 11285 | return KF_ARG_PTR_TO_RB_ROOT; |
| 11424 | 11286 | |
| 11425 | − if (is_kfunc_arg_rbtree_node(meta->btf, &args[argno])) | |
| 11287 | + if (is_kfunc_arg_rbtree_node(meta->btf, &args[arg])) | |
| 11426 | 11288 | return KF_ARG_PTR_TO_RB_NODE; |
| 11427 | 11289 | |
| 11428 | − if (is_kfunc_arg_const_str(meta->btf, &args[argno])) | |
| 11290 | + if (is_kfunc_arg_const_str(meta->btf, &args[arg])) | |
| 11429 | 11291 | return KF_ARG_PTR_TO_CONST_STR; |
| 11430 | 11292 | |
| 11431 | − if (is_kfunc_arg_map(meta->btf, &args[argno])) | |
| 11293 | + if (is_kfunc_arg_map(meta->btf, &args[arg])) | |
| 11432 | 11294 | return KF_ARG_PTR_TO_MAP; |
| 11433 | 11295 | |
| 11434 | − if (is_kfunc_arg_wq(meta->btf, &args[argno])) | |
| 11296 | + if (is_kfunc_arg_wq(meta->btf, &args[arg])) | |
| 11435 | 11297 | return KF_ARG_PTR_TO_WORKQUEUE; |
| 11436 | 11298 | |
| 11437 | − if (is_kfunc_arg_timer(meta->btf, &args[argno])) | |
| 11299 | + if (is_kfunc_arg_timer(meta->btf, &args[arg])) | |
| 11438 | 11300 | return KF_ARG_PTR_TO_TIMER; |
| 11439 | 11301 | |
| 11440 | − if (is_kfunc_arg_task_work(meta->btf, &args[argno])) | |
| 11302 | + if (is_kfunc_arg_task_work(meta->btf, &args[arg])) | |
| 11441 | 11303 | return KF_ARG_PTR_TO_TASK_WORK; |
| 11442 | 11304 | |
| 11443 | − if (is_kfunc_arg_irq_flag(meta->btf, &args[argno])) | |
| 11305 | + if (is_kfunc_arg_irq_flag(meta->btf, &args[arg])) | |
| 11444 | 11306 | return KF_ARG_PTR_TO_IRQ_FLAG; |
| 11445 | 11307 | |
| 11446 | − if (is_kfunc_arg_res_spin_lock(meta->btf, &args[argno])) | |
| 11308 | + if (is_kfunc_arg_res_spin_lock(meta->btf, &args[arg])) | |
| 11447 | 11309 | return KF_ARG_PTR_TO_RES_SPIN_LOCK; |
| 11448 | 11310 | |
| 11449 | 11311 | if ((base_type(reg->type) == PTR_TO_BTF_ID || reg2btf_ids[base_type(reg->type)])) { |
| 11450 | 11312 | if (!btf_type_is_struct(ref_t)) { |
| 11451 | − verbose(env, "kernel function %s args#%d pointer type %s %s is not supported\n", | |
| 11452 | − meta->func_name, argno, btf_type_str(ref_t), ref_tname); | |
| 11313 | + verbose(env, "kernel function %s %s pointer type %s %s is not supported\n", | |
| 11314 | + meta->func_name, reg_arg_name(env, argno), | |
| 11315 | + btf_type_str(ref_t), ref_tname); | |
| 11453 | 11316 | return -EINVAL; |
| 11454 | 11317 | } |
| 11455 | 11318 | return KF_ARG_PTR_TO_BTF_ID; |
| 11456 | 11319 | } |
| 11457 | 11320 | |
| 11458 | − if (is_kfunc_arg_callback(env, meta->btf, &args[argno])) | |
| 11321 | + if (is_kfunc_arg_callback(env, meta->btf, &args[arg])) | |
| 11459 | 11322 | return KF_ARG_PTR_TO_CALLBACK; |
| 11460 | 11323 | |
| 11461 | 11324 | /* This is the catch all argument type of register types supported by |
| @@ -11465,8 +11328,9 @@ get_kfunc_ptr_arg_type(struct bpf_verifier_env *env, · 3 unmodified lines | ||
| 11465 | 11328 | */ |
| 11466 | 11329 | if (!btf_type_is_scalar(ref_t) && !__btf_type_is_scalar_struct(env, meta->btf, ref_t, 0) && |
| 11467 | 11330 | (arg_mem_size ? !btf_type_is_void(ref_t) : 1)) { |
| 11468 | − verbose(env, "arg#%d pointer type %s %s must point to %sscalar, or struct with scalar\n", | |
| 11469 | − argno, btf_type_str(ref_t), ref_tname, arg_mem_size ? "void, " : ""); | |
| 11331 | + verbose(env, "%s pointer type %s %s must point to %sscalar, or struct with scalar\n", | |
| 11332 | + reg_arg_name(env, argno), | |
| 11333 | + btf_type_str(ref_t), ref_tname, arg_mem_size ? "void, " : ""); | |
| 11470 | 11334 | return -EINVAL; |
| 11471 | 11335 | } |
| 11472 | 11336 | return arg_mem_size ? KF_ARG_PTR_TO_MEM_SIZE : KF_ARG_PTR_TO_MEM; |
| @@ -11477,7 +11341,7 @@ static int process_kf_arg_ptr_to_btf_id(struct bpf_verifier_env *env, · 4 unmodified lines | ||
| 11477 | 11341 | const struct btf_type *ref_t, |
| 11478 | 11342 | const char *ref_tname, u32 ref_id, |
| 11479 | 11343 | struct bpf_kfunc_call_arg_meta *meta, |
| 11480 | − int argno) | |
| 11344 | + int arg, argno_t argno) | |
| 11481 | 11345 | { |
| 11482 | 11346 | const struct btf_type *reg_ref_t; |
| 11483 | 11347 | bool strict_type_match = false; |
| @@ -11519,7 +11383,7 @@ static int process_kf_arg_ptr_to_btf_id(struct bpf_verifier_env *env, · 35 unmodified lines | ||
| 11519 | 11383 | * btf_struct_ids_match() to walk the struct at the 0th offset, and |
| 11520 | 11384 | * resolve types. |
| 11521 | 11385 | */ |
| 11522 | − if ((is_kfunc_release(meta) && reg->ref_obj_id) || | |
| 11386 | + if ((is_kfunc_release(meta) && reg_is_referenced(env, reg)) || | |
| 11523 | 11387 | btf_type_ids_nocast_alias(&env->log, reg_btf, reg_ref_id, meta->btf, ref_id)) |
| 11524 | 11388 | strict_type_match = true; |
| 11525 | 11389 | |
| @@ -11535,19 +11399,19 @@ static int process_kf_arg_ptr_to_btf_id(struct bpf_verifier_env *env, · 9 unmodified lines | ||
| 11535 | 11399 | */ |
| 11536 | 11400 | taking_projection = btf_is_projection_of(ref_tname, reg_ref_tname); |
| 11537 | 11401 | if (!taking_projection && !struct_same) { |
| 11538 | − verbose(env, "kernel function %s args#%d expected pointer to %s %s but R%d has a pointer to %s %s\n", | |
| 11539 | − meta->func_name, argno, btf_type_str(ref_t), ref_tname, argno + 1, | |
| 11402 | + verbose(env, "kernel function %s %s expected pointer to %s %s but %s has a pointer to %s %s\n", | |
| 11403 | + meta->func_name, reg_arg_name(env, argno), | |
| 11404 | + btf_type_str(ref_t), ref_tname, reg_arg_name(env, argno), | |
| 11540 | 11405 | btf_type_str(reg_ref_t), reg_ref_tname); |
| 11541 | 11406 | return -EINVAL; |
| 11542 | 11407 | } |
| 11543 | 11408 | return 0; |
| 11544 | 11409 | } |
| 11545 | 11410 | |
| 11546 | −static int process_irq_flag(struct bpf_verifier_env *env, int regno, | |
| 11411 | +static int process_irq_flag(struct bpf_verifier_env *env, struct bpf_reg_state *reg, argno_t argno, | |
| 11547 | 11412 | struct bpf_kfunc_call_arg_meta *meta) |
| 11548 | 11413 | { |
| 11549 | − struct bpf_reg_state *reg = reg_state(env, regno); | |
| 11550 | − int err, kfunc_class = IRQ_NATIVE_KFUNC; | |
| 11414 | + int err, spi, kfunc_class = IRQ_NATIVE_KFUNC; | |
| 11551 | 11415 | bool irq_save; |
| 11552 | 11416 | |
| 11553 | 11417 | if (meta->func_id == special_kfunc_list[KF_bpf_local_irq_save] || |
| @@ -11567,11 +11431,13 @@ static int process_irq_flag(struct bpf_verifier_env *env, int regno, · 13 unmodified lines | ||
| 11567 | 11431 | |
| 11568 | 11432 | if (irq_save) { |
| 11569 | 11433 | if (!is_irq_flag_reg_valid_uninit(env, reg)) { |
| 11570 | − verbose(env, "expected uninitialized irq flag as arg#%d\n", regno - 1); | |
| 11434 | + verbose(env, "expected uninitialized irq flag as %s\n", | |
| 11435 | + reg_arg_name(env, argno)); | |
| 11571 | 11436 | return -EINVAL; |
| 11572 | 11437 | } |
| 11573 | 11438 | |
| 11574 | − err = check_mem_access(env, env->insn_idx, regno, 0, BPF_DW, BPF_WRITE, -1, false, false); | |
| 11439 | + err = check_mem_access(env, env->insn_idx, reg, argno, 0, BPF_DW, | |
| 11440 | + BPF_WRITE, -1, false, false); | |
| 11575 | 11441 | if (err) |
| 11576 | 11442 | return err; |
| 11577 | 11443 | |
| @@ -11581,13 +11447,16 @@ static int process_irq_flag(struct bpf_verifier_env *env, int regno, · 3 unmodified lines | ||
| 11581 | 11447 | } else { |
| 11582 | 11448 | err = is_irq_flag_reg_valid_init(env, reg); |
| 11583 | 11449 | if (err) { |
| 11584 | − verbose(env, "expected an initialized irq flag as arg#%d\n", regno - 1); | |
| 11450 | + verbose(env, "expected an initialized irq flag as %s\n", | |
| 11451 | + reg_arg_name(env, argno)); | |
| 11585 | 11452 | return err; |
| 11586 | 11453 | } |
| 11587 | 11454 | |
| 11588 | − err = mark_irq_flag_read(env, reg); | |
| 11589 | − if (err) | |
| 11590 | − return err; | |
| 11455 | + spi = irq_flag_get_spi(env, reg); | |
| 11456 | + if (spi < 0) | |
| 11457 | + return spi; | |
| 11458 | + | |
| 11459 | + mark_stack_slots_scratched(env, spi, 1); | |
| 11591 | 11460 | |
| 11592 | 11461 | err = unmark_stack_slot_irq_flag(env, reg, kfunc_class); |
| 11593 | 11462 | if (err) |
| @@ -11618,36 +11487,21 @@ static int ref_set_non_owning(struct bpf_verifier_env *env, struct bpf_reg_state · 24 unmodified lines | ||
| 11618 | 11487 | return 0; |
| 11619 | 11488 | } |
| 11620 | 11489 | |
| 11621 | −static int ref_convert_owning_non_owning(struct bpf_verifier_env *env, u32 ref_obj_id) | |
| 11490 | +static void ref_convert_owning_non_owning(struct bpf_verifier_env *env, u32 id) | |
| 11622 | 11491 | { |
| 11623 | − struct bpf_verifier_state *state = env->cur_state; | |
| 11624 | 11492 | struct bpf_func_state *unused; |
| 11625 | 11493 | struct bpf_reg_state *reg; |
| 11626 | − int i; | |
| 11627 | − | |
| 11628 | − if (!ref_obj_id) { | |
| 11629 | − verifier_bug(env, "ref_obj_id is zero for owning -> non-owning conversion"); | |
| 11630 | − return -EFAULT; | |
| 11631 | − } | |
| 11632 | 11494 | |
| 11633 | − for (i = 0; i < state->acquired_refs; i++) { | |
2971 more lines aren’t shown here.
| @@ -11712,6 +11712,15 @@ static int __perf_event_set_bpf_prog(struct perf_event *event, · 11711 unmodified lines | ||
| 11712 | 11712 | /* only uprobe programs are allowed to be sleepable */ |
| 11713 | 11713 | return -EINVAL; |
| 11714 | 11714 | |
| 11715 | + if (prog->type == BPF_PROG_TYPE_TRACEPOINT && prog->sleepable) { | |
| 11716 | + /* | |
| 11717 | + * Sleepable tracepoint programs can only attach to faultable | |
| 11718 | + * tracepoints. Currently only syscall tracepoints are faultable. | |
| 11719 | + */ | |
| 11720 | + if (!is_syscall_tp) | |
| 11721 | + return -EINVAL; | |
| 11722 | + } | |
| 11723 | + | |
| 11715 | 11724 | /* Kprobe override only works for kprobes, not uprobes. */ |
| 11716 | 11725 | if (prog->kprobe_override && !is_kprobe) |
| 11717 | 11726 | return -EINVAL; |
| @@ -23,6 +23,7 @@ · 22 unmodified lines | ||
| 23 | 23 | #include <linux/sort.h> |
| 24 | 24 | #include <linux/key.h> |
| 25 | 25 | #include <linux/namei.h> |
| 26 | +#include <linux/file.h> | |
| 26 | 27 | |
| 27 | 28 | #include <net/bpf_sk_storage.h> |
| 28 | 29 | |
| @@ -42,6 +43,7 @@ · 13 unmodified lines | ||
| 42 | 43 | |
| 43 | 44 | #define MAX_UPROBE_MULTI_CNT (1U << 20) |
| 44 | 45 | #define MAX_KPROBE_MULTI_CNT (1U << 20) |
| 46 | +#define MAX_TRACING_MULTI_CNT (1U << 20) | |
| 45 | 47 | |
| 46 | 48 | #ifdef CONFIG_MODULES |
| 47 | 49 | struct bpf_trace_module { |
| @@ -152,6 +154,34 @@ unsigned int trace_call_bpf(struct trace_event_call *call, void *ctx) · 104 unmodified lines | ||
| 152 | 154 | return ret; |
| 153 | 155 | } |
| 154 | 156 | |
| 157 | +/** | |
| 158 | + * trace_call_bpf_faultable - invoke BPF program in faultable context | |
| 159 | + * @call: tracepoint event | |
| 160 | + * @ctx: opaque context pointer | |
| 161 | + * | |
| 162 | + * Variant of trace_call_bpf() for faultable tracepoints (syscall | |
| 163 | + * tracepoints). Supports sleepable BPF programs by using rcu_tasks_trace | |
| 164 | + * for lifetime protection and bpf_prog_run_array_sleepable() for per-program | |
| 165 | + * RCU flavor selection, following the uprobe pattern. | |
| 166 | + * | |
| 167 | + * Per-program recursion protection is provided by | |
| 168 | + * bpf_prog_run_array_sleepable(). Global bpf_prog_active is not | |
| 169 | + * needed because syscall tracepoints cannot self-recurse. | |
| 170 | + * | |
| 171 | + * Must be called from a faultable/preemptible context. | |
| 172 | + */ | |
| 173 | +unsigned int trace_call_bpf_faultable(struct trace_event_call *call, void *ctx) | |
| 174 | +{ | |
| 175 | + struct bpf_prog_array *prog_array; | |
| 176 | + | |
| 177 | + might_fault(); | |
| 178 | + guard(rcu_tasks_trace)(); | |
| 179 | + | |
| 180 | + prog_array = rcu_dereference_check(call->prog_array, | |
| 181 | + rcu_read_lock_trace_held()); | |
| 182 | + return bpf_prog_run_array_sleepable(prog_array, ctx, bpf_prog_run); | |
| 183 | +} | |
| 184 | + | |
| 155 | 185 | #ifdef CONFIG_BPF_KPROBE_OVERRIDE |
| 156 | 186 | BPF_CALL_2(bpf_override_return, struct pt_regs *, regs, unsigned long, rc) |
| 157 | 187 | { |
| @@ -1305,7 +1335,8 @@ static inline bool is_uprobe_session(const struct bpf_prog *prog) · 1147 unmodified lines | ||
| 1305 | 1335 | static inline bool is_trace_fsession(const struct bpf_prog *prog) |
| 1306 | 1336 | { |
| 1307 | 1337 | return prog->type == BPF_PROG_TYPE_TRACING && |
| 1308 | − prog->expected_attach_type == BPF_TRACE_FSESSION; | |
| 1338 | + (prog->expected_attach_type == BPF_TRACE_FSESSION || | |
| 1339 | + prog->expected_attach_type == BPF_TRACE_FSESSION_MULTI); | |
| 1309 | 1340 | } |
| 1310 | 1341 | |
| 1311 | 1342 | static const struct bpf_func_proto * |
| @@ -2072,11 +2103,19 @@ void bpf_put_raw_tracepoint(struct bpf_raw_event_map *btp) · 760 unmodified lines | ||
| 2072 | 2103 | static __always_inline |
| 2073 | 2104 | void __bpf_trace_run(struct bpf_raw_tp_link *link, u64 *args) |
| 2074 | 2105 | { |
| 2106 | + struct srcu_ctr __percpu *scp = NULL; | |
| 2075 | 2107 | struct bpf_prog *prog = link->link.prog; |
| 2108 | + bool sleepable = prog->sleepable; | |
| 2076 | 2109 | struct bpf_run_ctx *old_run_ctx; |
| 2077 | 2110 | struct bpf_trace_run_ctx run_ctx; |
| 2078 | 2111 | |
| 2079 | − rcu_read_lock_dont_migrate(); | |
| 2112 | + if (sleepable) { | |
| 2113 | + scp = rcu_read_lock_tasks_trace(); | |
| 2114 | + migrate_disable(); | |
| 2115 | + } else { | |
| 2116 | + rcu_read_lock_dont_migrate(); | |
| 2117 | + } | |
| 2118 | + | |
| 2080 | 2119 | if (unlikely(!bpf_prog_get_recursion_context(prog))) { |
| 2081 | 2120 | bpf_prog_inc_misses_counter(prog); |
| 2082 | 2121 | goto out; |
| @@ -2085,12 +2124,18 @@ void __bpf_trace_run(struct bpf_raw_tp_link *link, u64 *args) · 2 unmodified lines | ||
| 2085 | 2124 | run_ctx.bpf_cookie = link->cookie; |
| 2086 | 2125 | old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx); |
| 2087 | 2126 | |
| 2088 | − (void) bpf_prog_run(prog, args); | |
| 2127 | + (void)bpf_prog_run(prog, args); | |
| 2089 | 2128 | |
| 2090 | 2129 | bpf_reset_run_ctx(old_run_ctx); |
| 2091 | 2130 | out: |
| 2092 | 2131 | bpf_prog_put_recursion_context(prog); |
| 2093 | − rcu_read_unlock_migrate(); | |
| 2132 | + | |
| 2133 | + if (sleepable) { | |
| 2134 | + migrate_enable(); | |
| 2135 | + rcu_read_unlock_tasks_trace(scp); | |
| 2136 | + } else { | |
| 2137 | + rcu_read_unlock_migrate(); | |
| 2138 | + } | |
| 2094 | 2139 | } |
| 2095 | 2140 | |
| 2096 | 2141 | #define UNPACK(...) __VA_ARGS__ |
| @@ -3170,6 +3215,38 @@ static u64 bpf_uprobe_multi_cookie(struct bpf_run_ctx *ctx) · 1073 unmodified lines | ||
| 3170 | 3215 | return run_ctx->uprobe->cookie; |
| 3171 | 3216 | } |
| 3172 | 3217 | |
| 3218 | +static int bpf_uprobe_multi_get_path(const union bpf_attr *attr, struct path *path) | |
| 3219 | +{ | |
| 3220 | + void __user *upath = u64_to_user_ptr(attr->link_create.uprobe_multi.path); | |
| 3221 | + u32 path_fd = attr->link_create.uprobe_multi.path_fd; | |
| 3222 | + u32 flags = attr->link_create.uprobe_multi.flags; | |
| 3223 | + | |
| 3224 | + if (flags & BPF_F_UPROBE_MULTI_PATH_FD) { | |
| 3225 | + /* | |
| 3226 | + * When BPF_F_UPROBE_MULTI_PATH_FD is set, the executable is | |
| 3227 | + * identified by path_fd, upath must be NULL. | |
| 3228 | + */ | |
| 3229 | + if (upath) | |
| 3230 | + return -EINVAL; | |
| 3231 | + | |
| 3232 | + CLASS(fd, f)(path_fd); | |
| 3233 | + if (fd_empty(f)) | |
| 3234 | + return -EBADF; | |
| 3235 | + *path = fd_file(f)->f_path; | |
| 3236 | + path_get(path); | |
| 3237 | + return 0; | |
| 3238 | + } | |
| 3239 | + | |
| 3240 | + /* | |
| 3241 | + * When BPF_F_UPROBE_MULTI_PATH_FD is not set, the path is resolved | |
| 3242 | + * relative to the cwd (AT_FDCWD) or absolute using the upath string. | |
| 3243 | + */ | |
| 3244 | + if (!upath || path_fd) | |
| 3245 | + return -EINVAL; | |
| 3246 | + | |
| 3247 | + return user_path_at(AT_FDCWD, upath, LOOKUP_FOLLOW, path); | |
| 3248 | +} | |
| 3249 | + | |
| 3173 | 3250 | int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *prog) |
| 3174 | 3251 | { |
| 3175 | 3252 | struct bpf_uprobe_multi_link *link = NULL; |
| @@ -3179,10 +3256,9 @@ int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *pr · 3 unmodified lines | ||
| 3179 | 3256 | struct task_struct *task = NULL; |
| 3180 | 3257 | unsigned long __user *uoffsets; |
| 3181 | 3258 | u64 __user *ucookies; |
| 3182 | − void __user *upath; | |
| 3259 | + unsigned long size; | |
| 3183 | 3260 | u32 flags, cnt, i; |
| 3184 | 3261 | struct path path; |
| 3185 | − char *name; | |
| 3186 | 3262 | pid_t pid; |
| 3187 | 3263 | int err; |
| 3188 | 3264 | |
| @@ -3197,19 +3273,18 @@ int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *pr · 8 unmodified lines | ||
| 3197 | 3273 | return -EINVAL; |
| 3198 | 3274 | |
| 3199 | 3275 | flags = attr->link_create.uprobe_multi.flags; |
| 3200 | − if (flags & ~BPF_F_UPROBE_MULTI_RETURN) | |
| 3276 | + if (flags & ~(BPF_F_UPROBE_MULTI_RETURN | BPF_F_UPROBE_MULTI_PATH_FD)) | |
| 3201 | 3277 | return -EINVAL; |
| 3202 | 3278 | |
| 3203 | 3279 | /* |
| 3204 | − * path, offsets and cnt are mandatory, | |
| 3280 | + * offsets and cnt are mandatory, | |
| 3205 | 3281 | * ref_ctr_offsets and cookies are optional |
| 3206 | 3282 | */ |
| 3207 | − upath = u64_to_user_ptr(attr->link_create.uprobe_multi.path); | |
| 3208 | 3283 | uoffsets = u64_to_user_ptr(attr->link_create.uprobe_multi.offsets); |
| 3209 | 3284 | cnt = attr->link_create.uprobe_multi.cnt; |
| 3210 | 3285 | pid = attr->link_create.uprobe_multi.pid; |
| 3211 | 3286 | |
| 3212 | − if (!upath || !uoffsets || !cnt || pid < 0) | |
| 3287 | + if (!uoffsets || !cnt || pid < 0) | |
| 3213 | 3288 | return -EINVAL; |
| 3214 | 3289 | if (cnt > MAX_UPROBE_MULTI_CNT) |
| 3215 | 3290 | return -E2BIG; |
| @@ -3217,14 +3292,17 @@ int bpf_uprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *pr · 1 unmodified line | ||
| 3217 | 3292 | uref_ctr_offsets = u64_to_user_ptr(attr->link_create.uprobe_multi.ref_ctr_offsets); |
| 3218 | 3293 | ucookies = u64_to_user_ptr(attr->link_create.uprobe_multi.cookies); |
| 3219 | 3294 | |
| 3220 | − name = strndup_user(upath, PATH_MAX); | |
| 3221 | − if (IS_ERR(name)) { | |
| 3222 | − err = PTR_ERR(name); | |
| 3223 | − return err; | |
| 3224 | − } | |
| 3295 | + /* | |
| 3296 | + * All uoffsets/uref_ctr_offsets/ucookies arrays have the same value | |
| 3297 | + * size, we need to check their address range is safe for __get_user | |
| 3298 | + * calls. | |
| 3299 | + */ | |
| 3300 | + size = sizeof(*uoffsets) * cnt; | |
| 3301 | + if (!access_ok(uoffsets, size) || !access_ok(uref_ctr_offsets, size) || | |
| 3302 | + !access_ok(ucookies, size)) | |
| 3303 | + return -EFAULT; | |
| 3225 | 3304 | |
| 3226 | − err = kern_path(name, LOOKUP_FOLLOW, &path); | |
| 3227 | − kfree(name); | |
| 3305 | + err = bpf_uprobe_multi_get_path(attr, &path); | |
| 3228 | 3306 | if (err) |
| 3229 | 3307 | return err; |
| 3230 | 3308 | |
| @@ -3398,12 +3476,12 @@ typedef int (*copy_fn_t)(void *dst, const void *src, u32 size, struct task_struc · 167 unmodified lines | ||
| 3398 | 3476 | * direct calls into all the specific callback implementations |
| 3399 | 3477 | * (copy_user_data_sleepable, copy_user_data_nofault, and so on) |
| 3400 | 3478 | */ |
| 3401 | −static __always_inline int __bpf_dynptr_copy_str(struct bpf_dynptr *dptr, u64 doff, u64 size, | |
| 3479 | +static __always_inline int __bpf_dynptr_copy_str(const struct bpf_dynptr *dptr, u64 doff, u64 size, | |
| 3402 | 3480 | const void *unsafe_src, |
| 3403 | 3481 | copy_fn_t str_copy_fn, |
| 3404 | 3482 | struct task_struct *tsk) |
| 3405 | 3483 | { |
| 3406 | − struct bpf_dynptr_kern *dst; | |
| 3484 | + const struct bpf_dynptr_kern *dst; | |
| 3407 | 3485 | u64 chunk_sz, off; |
| 3408 | 3486 | void *dst_slice; |
| 3409 | 3487 | int cnt, err; |
| @@ -3439,7 +3517,7 @@ static __always_inline int __bpf_dynptr_copy(const struct bpf_dynptr *dptr, u64 · 29 unmodified lines | ||
| 3439 | 3517 | u64 size, const void *unsafe_src, |
| 3440 | 3518 | copy_fn_t copy_fn, struct task_struct *tsk) |
| 3441 | 3519 | { |
| 3442 | − struct bpf_dynptr_kern *dst; | |
| 3520 | + const struct bpf_dynptr_kern *dst; | |
| 3443 | 3521 | void *dst_slice; |
| 3444 | 3522 | char buf[256]; |
| 3445 | 3523 | u64 off, chunk_sz; |
| @@ -3540,49 +3618,49 @@ __bpf_kfunc int bpf_send_signal_task(struct task_struct *task, int sig, enum pid · 94 unmodified lines | ||
| 3540 | 3618 | return bpf_send_signal_common(sig, type, task, value); |
| 3541 | 3619 | } |
| 3542 | 3620 | |
| 3543 | −__bpf_kfunc int bpf_probe_read_user_dynptr(struct bpf_dynptr *dptr, u64 off, | |
| 3621 | +__bpf_kfunc int bpf_probe_read_user_dynptr(const struct bpf_dynptr *dptr, u64 off, | |
| 3544 | 3622 | u64 size, const void __user *unsafe_ptr__ign) |
| 3545 | 3623 | { |
| 3546 | 3624 | return __bpf_dynptr_copy(dptr, off, size, (const void __force *)unsafe_ptr__ign, |
| 3547 | 3625 | copy_user_data_nofault, NULL); |
| 3548 | 3626 | } |
| 3549 | 3627 | |
| 3550 | −__bpf_kfunc int bpf_probe_read_kernel_dynptr(struct bpf_dynptr *dptr, u64 off, | |
| 3628 | +__bpf_kfunc int bpf_probe_read_kernel_dynptr(const struct bpf_dynptr *dptr, u64 off, | |
| 3551 | 3629 | u64 size, const void *unsafe_ptr__ign) |
| 3552 | 3630 | { |
| 3553 | 3631 | return __bpf_dynptr_copy(dptr, off, size, unsafe_ptr__ign, |
| 3554 | 3632 | copy_kernel_data_nofault, NULL); |
| 3555 | 3633 | } |
| 3556 | 3634 | |
| 3557 | −__bpf_kfunc int bpf_probe_read_user_str_dynptr(struct bpf_dynptr *dptr, u64 off, | |
| 3635 | +__bpf_kfunc int bpf_probe_read_user_str_dynptr(const struct bpf_dynptr *dptr, u64 off, | |
| 3558 | 3636 | u64 size, const void __user *unsafe_ptr__ign) |
| 3559 | 3637 | { |
| 3560 | 3638 | return __bpf_dynptr_copy_str(dptr, off, size, (const void __force *)unsafe_ptr__ign, |
| 3561 | 3639 | copy_user_str_nofault, NULL); |
| 3562 | 3640 | } |
| 3563 | 3641 | |
| 3564 | −__bpf_kfunc int bpf_probe_read_kernel_str_dynptr(struct bpf_dynptr *dptr, u64 off, | |
| 3642 | +__bpf_kfunc int bpf_probe_read_kernel_str_dynptr(const struct bpf_dynptr *dptr, u64 off, | |
| 3565 | 3643 | u64 size, const void *unsafe_ptr__ign) |
| 3566 | 3644 | { |
| 3567 | 3645 | return __bpf_dynptr_copy_str(dptr, off, size, unsafe_ptr__ign, |
| 3568 | 3646 | copy_kernel_str_nofault, NULL); |
| 3569 | 3647 | } |
| 3570 | 3648 | |
| 3571 | −__bpf_kfunc int bpf_copy_from_user_dynptr(struct bpf_dynptr *dptr, u64 off, | |
| 3649 | +__bpf_kfunc int bpf_copy_from_user_dynptr(const struct bpf_dynptr *dptr, u64 off, | |
| 3572 | 3650 | u64 size, const void __user *unsafe_ptr__ign) |
| 3573 | 3651 | { |
| 3574 | 3652 | return __bpf_dynptr_copy(dptr, off, size, (const void __force *)unsafe_ptr__ign, |
| 3575 | 3653 | copy_user_data_sleepable, NULL); |
| 3576 | 3654 | } |
| 3577 | 3655 | |
| 3578 | −__bpf_kfunc int bpf_copy_from_user_str_dynptr(struct bpf_dynptr *dptr, u64 off, | |
| 3656 | +__bpf_kfunc int bpf_copy_from_user_str_dynptr(const struct bpf_dynptr *dptr, u64 off, | |
| 3579 | 3657 | u64 size, const void __user *unsafe_ptr__ign) |
| 3580 | 3658 | { |
| 3581 | 3659 | return __bpf_dynptr_copy_str(dptr, off, size, (const void __force *)unsafe_ptr__ign, |
| 3582 | 3660 | copy_user_str_sleepable, NULL); |
| 3583 | 3661 | } |
| 3584 | 3662 | |
| 3585 | −__bpf_kfunc int bpf_copy_from_user_task_dynptr(struct bpf_dynptr *dptr, u64 off, | |
| 3663 | +__bpf_kfunc int bpf_copy_from_user_task_dynptr(const struct bpf_dynptr *dptr, u64 off, | |
| 3586 | 3664 | u64 size, const void __user *unsafe_ptr__ign, |
| 3587 | 3665 | struct task_struct *tsk) |
| 3588 | 3666 | { |
| @@ -3590,7 +3668,7 @@ __bpf_kfunc int bpf_copy_from_user_task_dynptr(struct bpf_dynptr *dptr, u64 off, · 1 unmodified line | ||
| 3590 | 3668 | copy_user_data_sleepable, tsk); |
| 3591 | 3669 | } |
| 3592 | 3670 | |
| 3593 | −__bpf_kfunc int bpf_copy_from_user_task_str_dynptr(struct bpf_dynptr *dptr, u64 off, | |
| 3671 | +__bpf_kfunc int bpf_copy_from_user_task_str_dynptr(const struct bpf_dynptr *dptr, u64 off, | |
| 3594 | 3672 | u64 size, const void __user *unsafe_ptr__ign, |
| 3595 | 3673 | struct task_struct *tsk) |
| 3596 | 3674 | { |
| @@ -3599,3 +3677,203 @@ __bpf_kfunc int bpf_copy_from_user_task_str_dynptr(struct bpf_dynptr *dptr, u64 · 2 unmodified lines | ||
| 3599 | 3677 | } |
| 3600 | 3678 | |
| 3601 | 3679 | __bpf_kfunc_end_defs(); |
| 3680 | + | |
| 3681 | +#if defined(CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS) && \ | |
| 3682 | + defined(CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS) | |
| 3683 | + | |
| 3684 | +static void bpf_tracing_multi_link_release(struct bpf_link *link) | |
| 3685 | +{ | |
| 3686 | + struct bpf_tracing_multi_link *tr_link = | |
| 3687 | + container_of(link, struct bpf_tracing_multi_link, link); | |
| 3688 | + | |
| 3689 | + WARN_ON_ONCE(bpf_trampoline_multi_detach(link->prog, tr_link)); | |
| 3690 | +} | |
| 3691 | + | |
| 3692 | +static void bpf_tracing_multi_link_dealloc(struct bpf_link *link) | |
| 3693 | +{ | |
| 3694 | + struct bpf_tracing_multi_link *tr_link = | |
| 3695 | + container_of(link, struct bpf_tracing_multi_link, link); | |
| 3696 | + | |
| 3697 | + kvfree(tr_link->fexits); | |
| 3698 | + kvfree(tr_link->cookies); | |
| 3699 | + kvfree(tr_link); | |
| 3700 | +} | |
| 3701 | + | |
| 3702 | +#ifdef CONFIG_PROC_FS | |
| 3703 | +static void bpf_tracing_multi_show_fdinfo(const struct bpf_link *link, | |
| 3704 | + struct seq_file *seq) | |
| 3705 | +{ | |
| 3706 | + struct bpf_tracing_multi_link *tr_link = | |
| 3707 | + container_of(link, struct bpf_tracing_multi_link, link); | |
| 3708 | + bool has_cookies = !!tr_link->cookies; | |
| 3709 | + | |
| 3710 | + seq_printf(seq, "attach_type:\t%u\n", tr_link->link.attach_type); | |
| 3711 | + seq_printf(seq, "cnt:\t%u\n", tr_link->nodes_cnt); | |
| 3712 | + | |
| 3713 | + seq_printf(seq, "%s\t %s\t %s\t %s\n", "obj-id", "btf-id", "cookie", "func"); | |
| 3714 | + for (int i = 0; i < tr_link->nodes_cnt; i++) { | |
| 3715 | + struct bpf_tracing_multi_node *mnode = &tr_link->nodes[i]; | |
| 3716 | + u32 btf_id, obj_id; | |
| 3717 | + | |
| 3718 | + bpf_trampoline_unpack_key(mnode->trampoline->key, &obj_id, &btf_id); | |
| 3719 | + seq_printf(seq, "%u\t %u\t %llu\t %pS\n", | |
| 3720 | + obj_id, btf_id, | |
| 3721 | + has_cookies ? tr_link->cookies[i] : 0, | |
| 3722 | + (void *) mnode->trampoline->ip); | |
| 3723 | + | |
| 3724 | + cond_resched(); | |
| 3725 | + } | |
| 3726 | +} | |
| 3727 | +#endif | |
| 3728 | + | |
| 3729 | +static const struct bpf_link_ops bpf_tracing_multi_link_lops = { | |
| 3730 | + .release = bpf_tracing_multi_link_release, | |
| 3731 | + .dealloc_deferred = bpf_tracing_multi_link_dealloc, | |
| 3732 | +#ifdef CONFIG_PROC_FS | |
| 3733 | + .show_fdinfo = bpf_tracing_multi_show_fdinfo, | |
| 3734 | +#endif | |
| 3735 | +}; | |
| 3736 | + | |
| 3737 | +static int ids_cmp_r(const void *pa, const void *pb, const void *priv __maybe_unused) | |
| 3738 | +{ | |
| 3739 | + u32 a = *(u32 *) pa; | |
| 3740 | + u32 b = *(u32 *) pb; | |
| 3741 | + | |
| 3742 | + return (a > b) - (a < b); | |
| 3743 | +} | |
| 3744 | + | |
| 3745 | +static void ids_swap_r(void *a, void *b, int size __maybe_unused, | |
| 3746 | + const void *priv __maybe_unused) | |
| 3747 | +{ | |
| 3748 | + u64 *cookie_a, *cookie_b, *cookies; | |
| 3749 | + u32 *id_a = a, *id_b = b, *ids; | |
| 3750 | + void **data = (void **) priv; | |
| 3751 | + | |
| 3752 | + ids = data[0]; | |
| 3753 | + cookies = data[1]; | |
| 3754 | + | |
| 3755 | + if (cookies) { | |
| 3756 | + cookie_a = cookies + (id_a - ids); | |
| 3757 | + cookie_b = cookies + (id_b - ids); | |
| 3758 | + swap(*cookie_a, *cookie_b); | |
| 3759 | + } | |
| 3760 | + swap(*id_a, *id_b); | |
| 3761 | +} | |
| 3762 | + | |
| 3763 | +static int check_dup_ids(u32 *ids, u64 *cookies, u32 cnt) | |
| 3764 | +{ | |
| 3765 | + void *data[2] = { ids, cookies }; | |
| 3766 | + int err = 0; | |
| 3767 | + | |
| 3768 | + /* | |
| 3769 | + * Sort ids array (together with cookies array if defined) | |
| 3770 | + * and check it for duplicates. The ids and cookies arrays | |
| 3771 | + * are left sorted. | |
| 3772 | + */ | |
| 3773 | + sort_r_nonatomic(ids, cnt, sizeof(ids[0]), ids_cmp_r, ids_swap_r, data); | |
| 3774 | + | |
| 3775 | + for (int i = 1; i < cnt; i++) { | |
| 3776 | + if (ids[i] == ids[i - 1]) { | |
| 3777 | + err = -EINVAL; | |
| 3778 | + break; | |
| 3779 | + } | |
| 3780 | + } | |
| 3781 | + return err; | |
| 3782 | +} | |
| 3783 | + | |
| 3784 | +int bpf_tracing_multi_attach(struct bpf_prog *prog, const union bpf_attr *attr) | |
| 3785 | +{ | |
| 3786 | + struct bpf_tracing_multi_link *link = NULL; | |
| 3787 | + struct bpf_tramp_node *fexits = NULL; | |
| 3788 | + struct bpf_link_primer link_primer; | |
| 3789 | + u32 cnt, *ids = NULL; | |
| 3790 | + u64 __user *ucookies; | |
| 3791 | + u64 *cookies = NULL; | |
| 3792 | + u32 __user *uids; | |
| 3793 | + int err; | |
| 3794 | + | |
| 3795 | + uids = u64_to_user_ptr(attr->link_create.tracing_multi.ids); | |
| 3796 | + cnt = attr->link_create.tracing_multi.cnt; | |
| 3797 | + | |
| 3798 | + if (!cnt || !uids) | |
| 3799 | + return -EINVAL; | |
| 3800 | + if (cnt > MAX_TRACING_MULTI_CNT) | |
| 3801 | + return -E2BIG; | |
| 3802 | + if (attr->link_create.flags || attr->link_create.target_fd) | |
| 3803 | + return -EINVAL; | |
| 3804 | + | |
| 3805 | + ids = kvmalloc_objs(*ids, cnt); | |
| 3806 | + if (!ids) | |
| 3807 | + return -ENOMEM; | |
| 3808 | + | |
| 3809 | + if (copy_from_user(ids, uids, cnt * sizeof(*ids))) { | |
| 3810 | + err = -EFAULT; | |
| 3811 | + goto error; | |
| 3812 | + } | |
| 3813 | + | |
| 3814 | + ucookies = u64_to_user_ptr(attr->link_create.tracing_multi.cookies); | |
| 3815 | + if (ucookies) { | |
| 3816 | + cookies = kvmalloc_objs(*cookies, cnt); | |
| 3817 | + if (!cookies) { | |
| 3818 | + err = -ENOMEM; | |
| 3819 | + goto error; | |
| 3820 | + } | |
| 3821 | + if (copy_from_user(cookies, ucookies, cnt * sizeof(*cookies))) { | |
| 3822 | + err = -EFAULT; | |
| 3823 | + goto error; | |
| 3824 | + } | |
| 3825 | + } | |
| 3826 | + | |
| 3827 | + err = check_dup_ids(ids, cookies, cnt); | |
| 3828 | + if (err) | |
| 3829 | + goto error; | |
| 3830 | + | |
| 3831 | + if (prog->expected_attach_type == BPF_TRACE_FSESSION_MULTI) { | |
| 3832 | + fexits = kvmalloc_objs(*fexits, cnt); | |
| 3833 | + if (!fexits) { | |
| 3834 | + err = -ENOMEM; | |
| 3835 | + goto error; | |
| 3836 | + } | |
| 3837 | + } | |
| 3838 | + | |
| 3839 | + link = kvzalloc_flex(*link, nodes, cnt); | |
| 3840 | + if (!link) { | |
| 3841 | + err = -ENOMEM; | |
| 3842 | + goto error; | |
| 3843 | + } | |
| 3844 | + | |
| 3845 | + bpf_link_init(&link->link, BPF_LINK_TYPE_TRACING_MULTI, | |
| 3846 | + &bpf_tracing_multi_link_lops, prog, prog->expected_attach_type); | |
| 3847 | + | |
| 3848 | + err = bpf_link_prime(&link->link, &link_primer); | |
| 3849 | + if (err) | |
| 3850 | + goto error; | |
| 3851 | + | |
| 3852 | + link->nodes_cnt = cnt; | |
| 3853 | + link->cookies = cookies; | |
| 3854 | + link->fexits = fexits; | |
| 3855 | + | |
| 3856 | + err = bpf_trampoline_multi_attach(prog, ids, link); | |
| 3857 | + kvfree(ids); | |
| 3858 | + if (err) { | |
| 3859 | + bpf_link_cleanup(&link_primer); | |
| 3860 | + return err; | |
| 3861 | + } | |
| 3862 | + return bpf_link_settle(&link_primer); | |
| 3863 | + | |
| 3864 | +error: | |
| 3865 | + kvfree(fexits); | |
| 3866 | + kvfree(cookies); | |
| 3867 | + kvfree(ids); | |
| 3868 | + kvfree(link); | |
| 3869 | + return err; | |
| 3870 | +} | |
| 3871 | + | |
| 3872 | +#else | |
| 3873 | + | |
| 3874 | +int bpf_tracing_multi_attach(struct bpf_prog *prog, const union bpf_attr *attr) | |
| 3875 | +{ | |
| 3876 | + return -EOPNOTSUPP; | |
| 3877 | +} | |
| 3878 | + | |
| 3879 | +#endif /* CONFIG_DYNAMIC_FTRACE_WITH_DIRECT_CALLS && CONFIG_HAVE_SINGLE_FTRACE_DIRECT_OPS */ | |
| @@ -1198,8 +1198,7 @@ ftrace_lookup_ip(struct ftrace_hash *hash, unsigned long ip) · 1197 unmodified lines | ||
| 1198 | 1198 | return __ftrace_lookup_ip(hash, ip); |
| 1199 | 1199 | } |
| 1200 | 1200 | |
| 1201 | −static void __add_hash_entry(struct ftrace_hash *hash, | |
| 1202 | − struct ftrace_func_entry *entry) | |
| 1201 | +void add_ftrace_hash_entry(struct ftrace_hash *hash, struct ftrace_func_entry *entry) | |
| 1203 | 1202 | { |
| 1204 | 1203 | struct hlist_head *hhd; |
| 1205 | 1204 | unsigned long key; |
| @@ -1221,7 +1220,7 @@ add_ftrace_hash_entry_direct(struct ftrace_hash *hash, unsigned long ip, unsigne · 15 unmodified lines | ||
| 1221 | 1220 | |
| 1222 | 1221 | entry->ip = ip; |
| 1223 | 1222 | entry->direct = direct; |
| 1224 | − __add_hash_entry(hash, entry); | |
| 1223 | + add_ftrace_hash_entry(hash, entry); | |
| 1225 | 1224 | |
| 1226 | 1225 | return entry; |
| 1227 | 1226 | } |
| @@ -1249,6 +1248,25 @@ remove_hash_entry(struct ftrace_hash *hash, · 21 unmodified lines | ||
| 1249 | 1248 | hash->count--; |
| 1250 | 1249 | } |
| 1251 | 1250 | |
| 1251 | +void ftrace_hash_remove(struct ftrace_hash *hash) | |
| 1252 | +{ | |
| 1253 | + struct ftrace_func_entry *entry; | |
| 1254 | + struct hlist_head *hhd; | |
| 1255 | + struct hlist_node *tn; | |
| 1256 | + int size; | |
| 1257 | + int i; | |
| 1258 | + | |
| 1259 | + if (!hash || !hash->count) | |
| 1260 | + return; | |
| 1261 | + size = 1 << hash->size_bits; | |
| 1262 | + for (i = 0; i < size; i++) { | |
| 1263 | + hhd = &hash->buckets[i]; | |
| 1264 | + hlist_for_each_entry_safe(entry, tn, hhd, hlist) | |
| 1265 | + remove_hash_entry(hash, entry); | |
| 1266 | + } | |
| 1267 | + FTRACE_WARN_ON(hash->count); | |
| 1268 | +} | |
| 1269 | + | |
| 1252 | 1270 | static void ftrace_hash_clear(struct ftrace_hash *hash) |
| 1253 | 1271 | { |
| 1254 | 1272 | struct hlist_head *hhd; |
| @@ -1458,7 +1476,7 @@ static struct ftrace_hash *__move_hash(struct ftrace_hash *src, int size) · 203 unmodified lines | ||
| 1458 | 1476 | hhd = &src->buckets[i]; |
| 1459 | 1477 | hlist_for_each_entry_safe(entry, tn, hhd, hlist) { |
| 1460 | 1478 | remove_hash_entry(src, entry); |
| 1461 | − __add_hash_entry(new_hash, entry); | |
| 1479 | + add_ftrace_hash_entry(new_hash, entry); | |
| 1462 | 1480 | } |
| 1463 | 1481 | } |
| 1464 | 1482 | return new_hash; |
| @@ -5341,7 +5359,7 @@ int ftrace_func_mapper_add_ip(struct ftrace_func_mapper *mapper, · 3876 unmodified lines | ||
| 5341 | 5359 | map->entry.ip = ip; |
| 5342 | 5360 | map->data = data; |
| 5343 | 5361 | |
| 5344 | − __add_hash_entry(&mapper->hash, &map->entry); | |
| 5362 | + add_ftrace_hash_entry(&mapper->hash, &map->entry); | |
| 5345 | 5363 | |
| 5346 | 5364 | return 0; |
| 5347 | 5365 | } |
| @@ -6288,11 +6306,16 @@ int modify_ftrace_direct(struct ftrace_ops *ops, unsigned long addr) · 940 unmodified lines | ||
| 6288 | 6306 | } |
| 6289 | 6307 | EXPORT_SYMBOL_GPL(modify_ftrace_direct); |
| 6290 | 6308 | |
| 6291 | −static unsigned long hash_count(struct ftrace_hash *hash) | |
| 6309 | +static inline unsigned long hash_count(struct ftrace_hash *hash) | |
| 6292 | 6310 | { |
| 6293 | 6311 | return hash ? hash->count : 0; |
| 6294 | 6312 | } |
| 6295 | 6313 | |
| 6314 | +unsigned long ftrace_hash_count(struct ftrace_hash *hash) | |
| 6315 | +{ | |
| 6316 | + return hash_count(hash); | |
| 6317 | +} | |
| 6318 | + | |
| 6296 | 6319 | /** |
| 6297 | 6320 | * hash_add - adds two struct ftrace_hash and returns the result |
| 6298 | 6321 | * @a: struct ftrace_hash object |
| @@ -1371,33 +1371,33 @@ static DECLARE_BITMAP(enabled_perf_exit_syscalls, NR_syscalls); · 1370 unmodified lines | ||
| 1371 | 1371 | static int sys_perf_refcount_enter; |
| 1372 | 1372 | static int sys_perf_refcount_exit; |
| 1373 | 1373 | |
| 1374 | −static int perf_call_bpf_enter(struct trace_event_call *call, struct pt_regs *regs, | |
| 1374 | +static int perf_call_bpf_enter(struct trace_event_call *call, | |
| 1375 | 1375 | struct syscall_metadata *sys_data, |
| 1376 | − struct syscall_trace_enter *rec) | |
| 1376 | + int syscall_nr, unsigned long *args) | |
| 1377 | 1377 | { |
| 1378 | 1378 | struct syscall_tp_t { |
| 1379 | 1379 | struct trace_entry ent; |
| 1380 | 1380 | int syscall_nr; |
| 1381 | 1381 | unsigned long args[SYSCALL_DEFINE_MAXARGS]; |
| 1382 | 1382 | } __aligned(8) param; |
| 1383 | + struct pt_regs regs = {}; | |
| 1383 | 1384 | int i; |
| 1384 | 1385 | |
| 1385 | 1386 | BUILD_BUG_ON(sizeof(param.ent) < sizeof(void *)); |
| 1386 | 1387 | |
| 1387 | − /* bpf prog requires 'regs' to be the first member in the ctx (a.k.a. ¶m) */ | |
| 1388 | − perf_fetch_caller_regs(regs); | |
| 1389 | − *(struct pt_regs **)¶m = regs; | |
| 1390 | − param.syscall_nr = rec->nr; | |
| 1388 | + /* bpf prog requires 'regs' to be the first member in the ctx */ | |
| 1389 | + perf_fetch_caller_regs(®s); | |
| 1390 | + *(struct pt_regs **)¶m = ®s; | |
| 1391 | + param.syscall_nr = syscall_nr; | |
| 1391 | 1392 | for (i = 0; i < sys_data->nb_args; i++) |
| 1392 | − param.args[i] = rec->args[i]; | |
| 1393 | − return trace_call_bpf(call, ¶m); | |
| 1393 | + param.args[i] = args[i]; | |
| 1394 | + return trace_call_bpf_faultable(call, ¶m); | |
| 1394 | 1395 | } |
| 1395 | 1396 | |
| 1396 | 1397 | static void perf_syscall_enter(void *ignore, struct pt_regs *regs, long id) |
| 1397 | 1398 | { |
| 1398 | 1399 | struct syscall_metadata *sys_data; |
| 1399 | 1400 | struct syscall_trace_enter *rec; |
| 1400 | − struct pt_regs *fake_regs; | |
| 1401 | 1401 | struct hlist_head *head; |
| 1402 | 1402 | unsigned long args[6]; |
| 1403 | 1403 | bool valid_prog_array; |
| @@ -1410,12 +1410,7 @@ static void perf_syscall_enter(void *ignore, struct pt_regs *regs, long id) · 6 unmodified lines | ||
| 1410 | 1410 | int size = 0; |
| 1411 | 1411 | int uargs = 0; |
| 1412 | 1412 | |
| 1413 | − /* | |
| 1414 | − * Syscall probe called with preemption enabled, but the ring | |
| 1415 | − * buffer and per-cpu data require preemption to be disabled. | |
| 1416 | − */ | |
| 1417 | 1413 | might_fault(); |
| 1418 | − guard(preempt_notrace)(); | |
| 1419 | 1414 | |
| 1420 | 1415 | syscall_nr = trace_get_syscall_nr(current, regs); |
| 1421 | 1416 | if (syscall_nr < 0 || syscall_nr >= NR_syscalls) |
| @@ -1429,6 +1424,26 @@ static void perf_syscall_enter(void *ignore, struct pt_regs *regs, long id) · 7 unmodified lines | ||
| 1429 | 1424 | |
| 1430 | 1425 | syscall_get_arguments(current, regs, args); |
| 1431 | 1426 | |
| 1427 | + /* | |
| 1428 | + * Run BPF program in faultable context before per-cpu buffer | |
| 1429 | + * allocation, allowing sleepable BPF programs to execute. | |
| 1430 | + */ | |
| 1431 | + valid_prog_array = bpf_prog_array_valid(sys_data->enter_event); | |
| 1432 | + if (valid_prog_array && | |
| 1433 | + !perf_call_bpf_enter(sys_data->enter_event, sys_data, | |
| 1434 | + syscall_nr, args)) | |
| 1435 | + return; | |
| 1436 | + | |
| 1437 | + /* | |
| 1438 | + * Per-cpu ring buffer and perf event list operations require | |
| 1439 | + * preemption to be disabled. | |
| 1440 | + */ | |
| 1441 | + guard(preempt_notrace)(); | |
| 1442 | + | |
| 1443 | + head = this_cpu_ptr(sys_data->enter_event->perf_events); | |
| 1444 | + if (hlist_empty(head)) | |
| 1445 | + return; | |
| 1446 | + | |
| 1432 | 1447 | /* Check if this syscall event faults in user space memory */ |
| 1433 | 1448 | mayfault = sys_data->user_mask != 0; |
| 1434 | 1449 | |
| @@ -1438,17 +1453,12 @@ static void perf_syscall_enter(void *ignore, struct pt_regs *regs, long id) · 3 unmodified lines | ||
| 1438 | 1453 | return; |
| 1439 | 1454 | } |
| 1440 | 1455 | |
| 1441 | − head = this_cpu_ptr(sys_data->enter_event->perf_events); | |
| 1442 | − valid_prog_array = bpf_prog_array_valid(sys_data->enter_event); | |
| 1443 | − if (!valid_prog_array && hlist_empty(head)) | |
| 1444 | − return; | |
| 1445 | − | |
| 1446 | 1456 | /* get the size after alignment with the u32 buffer size field */ |
| 1447 | 1457 | size += sizeof(unsigned long) * sys_data->nb_args + sizeof(*rec); |
| 1448 | 1458 | size = ALIGN(size + sizeof(u32), sizeof(u64)); |
| 1449 | 1459 | size -= sizeof(u32); |
| 1450 | 1460 | |
| 1451 | − rec = perf_trace_buf_alloc(size, &fake_regs, &rctx); | |
| 1461 | + rec = perf_trace_buf_alloc(size, NULL, &rctx); | |
| 1452 | 1462 | if (!rec) |
| 1453 | 1463 | return; |
| 1454 | 1464 | |
| @@ -1458,13 +1468,6 @@ static void perf_syscall_enter(void *ignore, struct pt_regs *regs, long id) · 3 unmodified lines | ||
| 1458 | 1468 | if (mayfault) |
| 1459 | 1469 | syscall_put_data(sys_data, rec, user_ptr, size, user_sizes, uargs); |
| 1460 | 1470 | |
| 1461 | − if ((valid_prog_array && | |
| 1462 | − !perf_call_bpf_enter(sys_data->enter_event, fake_regs, sys_data, rec)) || | |
| 1463 | − hlist_empty(head)) { | |
| 1464 | − perf_swevent_put_recursion_context(rctx); | |
| 1465 | − return; | |
| 1466 | − } | |
| 1467 | − | |
| 1468 | 1471 | perf_trace_buf_submit(rec, size, rctx, |
| 1469 | 1472 | sys_data->enter_event->event.type, 1, regs, |
| 1470 | 1473 | head, NULL); |
| @@ -1514,40 +1517,35 @@ static void perf_sysenter_disable(struct trace_event_call *call) · 43 unmodified lines | ||
| 1514 | 1517 | syscall_fault_buffer_disable(); |
| 1515 | 1518 | } |
| 1516 | 1519 | |
| 1517 | −static int perf_call_bpf_exit(struct trace_event_call *call, struct pt_regs *regs, | |
| 1518 | − struct syscall_trace_exit *rec) | |
| 1520 | +static int perf_call_bpf_exit(struct trace_event_call *call, | |
| 1521 | + int syscall_nr, long ret_val) | |
| 1519 | 1522 | { |
| 1520 | 1523 | struct syscall_tp_t { |
| 1521 | 1524 | struct trace_entry ent; |
| 1522 | 1525 | int syscall_nr; |
| 1523 | 1526 | unsigned long ret; |
| 1524 | 1527 | } __aligned(8) param; |
| 1525 | − | |
| 1526 | − /* bpf prog requires 'regs' to be the first member in the ctx (a.k.a. ¶m) */ | |
| 1527 | − perf_fetch_caller_regs(regs); | |
| 1528 | − *(struct pt_regs **)¶m = regs; | |
| 1529 | − param.syscall_nr = rec->nr; | |
| 1530 | − param.ret = rec->ret; | |
| 1531 | − return trace_call_bpf(call, ¶m); | |
| 1528 | + struct pt_regs regs = {}; | |
| 1529 | + | |
| 1530 | + /* bpf prog requires 'regs' to be the first member in the ctx */ | |
| 1531 | + perf_fetch_caller_regs(®s); | |
| 1532 | + *(struct pt_regs **)¶m = ®s; | |
| 1533 | + param.syscall_nr = syscall_nr; | |
| 1534 | + param.ret = ret_val; | |
| 1535 | + return trace_call_bpf_faultable(call, ¶m); | |
| 1532 | 1536 | } |
| 1533 | 1537 | |
| 1534 | 1538 | static void perf_syscall_exit(void *ignore, struct pt_regs *regs, long ret) |
| 1535 | 1539 | { |
| 1536 | 1540 | struct syscall_metadata *sys_data; |
| 1537 | 1541 | struct syscall_trace_exit *rec; |
| 1538 | − struct pt_regs *fake_regs; | |
| 1539 | 1542 | struct hlist_head *head; |
| 1540 | 1543 | bool valid_prog_array; |
| 1541 | 1544 | int syscall_nr; |
| 1542 | 1545 | int rctx; |
| 1543 | 1546 | int size; |
| 1544 | 1547 | |
| 1545 | − /* | |
| 1546 | − * Syscall probe called with preemption enabled, but the ring | |
| 1547 | − * buffer and per-cpu data require preemption to be disabled. | |
| 1548 | − */ | |
| 1549 | 1548 | might_fault(); |
| 1550 | − guard(preempt_notrace)(); | |
| 1551 | 1549 | |
| 1552 | 1550 | syscall_nr = trace_get_syscall_nr(current, regs); |
| 1553 | 1551 | if (syscall_nr < 0 || syscall_nr >= NR_syscalls) |
| @@ -1559,29 +1557,37 @@ static void perf_syscall_exit(void *ignore, struct pt_regs *regs, long ret) · 5 unmodified lines | ||
| 1559 | 1557 | if (!sys_data) |
| 1560 | 1558 | return; |
| 1561 | 1559 | |
| 1562 | − head = this_cpu_ptr(sys_data->exit_event->perf_events); | |
| 1560 | + /* | |
| 1561 | + * Run BPF program in faultable context before per-cpu buffer | |
| 1562 | + * allocation, allowing sleepable BPF programs to execute. | |
| 1563 | + */ | |
| 1563 | 1564 | valid_prog_array = bpf_prog_array_valid(sys_data->exit_event); |
| 1564 | − if (!valid_prog_array && hlist_empty(head)) | |
| 1565 | + if (valid_prog_array && | |
| 1566 | + !perf_call_bpf_exit(sys_data->exit_event, syscall_nr, | |
| 1567 | + syscall_get_return_value(current, regs))) | |
| 1568 | + return; | |
| 1569 | + | |
| 1570 | + /* | |
| 1571 | + * Per-cpu ring buffer and perf event list operations require | |
| 1572 | + * preemption to be disabled. | |
| 1573 | + */ | |
| 1574 | + guard(preempt_notrace)(); | |
| 1575 | + | |
| 1576 | + head = this_cpu_ptr(sys_data->exit_event->perf_events); | |
| 1577 | + if (hlist_empty(head)) | |
| 1565 | 1578 | return; |
| 1566 | 1579 | |
| 1567 | 1580 | /* We can probably do that at build time */ |
| 1568 | 1581 | size = ALIGN(sizeof(*rec) + sizeof(u32), sizeof(u64)); |
| 1569 | 1582 | size -= sizeof(u32); |
| 1570 | 1583 | |
| 1571 | − rec = perf_trace_buf_alloc(size, &fake_regs, &rctx); | |
| 1584 | + rec = perf_trace_buf_alloc(size, NULL, &rctx); | |
| 1572 | 1585 | if (!rec) |
| 1573 | 1586 | return; |
| 1574 | 1587 | |
| 1575 | 1588 | rec->nr = syscall_nr; |
| 1576 | 1589 | rec->ret = syscall_get_return_value(current, regs); |
| 1577 | 1590 | |
| 1578 | − if ((valid_prog_array && | |
| 1579 | − !perf_call_bpf_exit(sys_data->exit_event, fake_regs, rec)) || | |
| 1580 | − hlist_empty(head)) { | |
| 1581 | − perf_swevent_put_recursion_context(rctx); | |
| 1582 | − return; | |
| 1583 | − } | |
| 1584 | − | |
| 1585 | 1591 | perf_trace_buf_submit(rec, size, rctx, sys_data->exit_event->event.type, |
| 1586 | 1592 | 1, regs, head, NULL); |
| 1587 | 1593 | } |
| @@ -687,6 +687,108 @@ void *rhashtable_insert_slow(struct rhashtable *ht, const void *key, · 686 unmodified lines | ||
| 687 | 687 | } |
| 688 | 688 | EXPORT_SYMBOL_GPL(rhashtable_insert_slow); |
| 689 | 689 | |
| 690 | +/* Scan one element forward from prev_key's position in @tbl. | |
| 691 | + * Returns first rhash_head whose bucket > prev_key's bucket, or the | |
| 692 | + * element immediately after prev_key inside prev_key's bucket. | |
| 693 | + * Returns the first element if prev_key is NULL, NULL when @tbl is | |
| 694 | + * exhausted, or ERR_PTR(-ENOENT) if prev_key is not found in @tbl. | |
| 695 | + */ | |
| 696 | +static struct rhash_head *__rhashtable_next_in_table( | |
| 697 | + struct rhashtable *ht, struct bucket_table *tbl, | |
| 698 | + const void *prev_key) | |
| 699 | +{ | |
| 700 | + struct rhashtable_compare_arg arg = { .ht = ht, .key = prev_key }; | |
| 701 | + const struct rhashtable_params params = ht->p; | |
| 702 | + struct rhash_head *he; | |
| 703 | + unsigned int b = 0; | |
| 704 | + bool found = false; | |
| 705 | + | |
| 706 | + if (prev_key) { | |
| 707 | + b = rht_key_hashfn(ht, tbl, prev_key, params); | |
| 708 | + rht_for_each_rcu(he, tbl, b) { | |
| 709 | + bool match = params.obj_cmpfn | |
| 710 | + ? !params.obj_cmpfn(&arg, rht_obj(ht, he)) | |
| 711 | + : !rhashtable_compare(&arg, rht_obj(ht, he)); | |
| 712 | + if (found) { | |
| 713 | + if (match) | |
| 714 | + continue; | |
| 715 | + return he; | |
| 716 | + } | |
| 717 | + if (match) | |
| 718 | + found = true; | |
| 719 | + } | |
| 720 | + if (!found) | |
| 721 | + return ERR_PTR(-ENOENT); | |
| 722 | + b++; | |
| 723 | + } | |
| 724 | + | |
| 725 | + for (; b < tbl->size; b++) | |
| 726 | + rht_for_each_rcu(he, tbl, b) | |
| 727 | + return he; | |
| 728 | + return NULL; | |
| 729 | +} | |
| 730 | + | |
| 731 | +/** | |
| 732 | + * rhashtable_next_key - return next element after a given key | |
| 733 | + * @ht: hash table | |
| 734 | + * @prev_key: pointer to previous key, or NULL for the first element | |
| 735 | + * | |
| 736 | + * WARNING: this walk is highly unstable. Unlike rhashtable_walk_*(), | |
| 737 | + * it cannot detect a concurrent resize or rehash, so a full iteration | |
| 738 | + * is NOT guaranteed to terminate under adversarial or sustained | |
| 739 | + * rehashing. Callers MUST tolerate skipped and duplicated elements and | |
| 740 | + * SHOULD bound their loop externally. | |
| 741 | + * | |
| 742 | + * Returns the next element in best-effort iteration order, walking the | |
| 743 | + * @tbl chain (including any future_tbl in flight). Caller must hold RCU. | |
| 744 | + * | |
| 745 | + * Pass @prev_key == NULL to obtain the first element. To iterate, set | |
| 746 | + * @prev_key to the key of the previously returned element on each call, | |
| 747 | + * and stop when NULL is returned. | |
| 748 | + * | |
| 749 | + * Best-effort semantics: | |
| 750 | + * - Across the tbl->future_tbl chain, an element being migrated may | |
| 751 | + * transiently appear in both tables and be observed twice. | |
| 752 | + * - Concurrent inserts may or may not be observed. | |
| 753 | + * - Termination of a full iteration loop is NOT guaranteed under | |
| 754 | + * adversarial continuous rehash; callers MUST tolerate skips and | |
| 755 | + * repeats and SHOULD bound their loop externally. | |
| 756 | + * - Behavior on tables that contain duplicate keys is undefined: | |
| 757 | + * duplicates may be skipped, repeated, or trap the walk in a | |
| 758 | + * cycle. Callers requiring duplicate-key iteration must use | |
| 759 | + * rhashtable_walk_*() instead. | |
| 760 | + * - rhltable instances are not supported and return | |
| 761 | + * ERR_PTR(-EOPNOTSUPP). | |
| 762 | + * - If prev_key was concurrently deleted and is not present in any | |
| 763 | + * in-flight table, returns ERR_PTR(-ENOENT). | |
| 764 | + * | |
| 765 | + * Returns entry of the next element, or NULL when iteration is exhausted, | |
| 766 | + * or ERR_PTR(-ENOENT) if prev_key is not found, or | |
| 767 | + * ERR_PTR(-EOPNOTSUPP) if @ht is an rhltable. | |
| 768 | + */ | |
| 769 | +void *rhashtable_next_key(struct rhashtable *ht, const void *prev_key) | |
| 770 | +{ | |
| 771 | + struct bucket_table *tbl; | |
| 772 | + struct rhash_head *he; | |
| 773 | + | |
| 774 | + if (unlikely(ht->rhlist)) | |
| 775 | + return ERR_PTR(-EOPNOTSUPP); | |
| 776 | + | |
| 777 | + tbl = rht_dereference_rcu(ht->tbl, ht); | |
| 778 | + do { | |
| 779 | + he = __rhashtable_next_in_table(ht, tbl, prev_key); | |
| 780 | + if (!IS_ERR_OR_NULL(he)) | |
| 781 | + return rht_obj(ht, he); | |
| 782 | + if (!he) | |
| 783 | + prev_key = NULL; | |
| 784 | + /* See any new future_tbl attached during a rehash. */ | |
| 785 | + smp_rmb(); | |
| 786 | + tbl = rht_dereference_rcu(tbl->future_tbl, ht); | |
| 787 | + } while (tbl); | |
| 788 | + return he; /* NULL or -ENOENT */ | |
| 789 | +} | |
| 790 | +EXPORT_SYMBOL_GPL(rhashtable_next_key); | |
| 791 | + | |
| 690 | 792 | /** |
| 691 | 793 | * rhashtable_walk_enter - Initialise an iterator |
| 692 | 794 | * @ht: Table to walk over |
| @@ -560,8 +560,23 @@ static int bpf_fill_max_jmp_never_taken(struct bpf_test *self) · 559 unmodified lines | ||
| 560 | 560 | } |
| 561 | 561 | |
| 562 | 562 | /* ALU result computation used in tests */ |
| 563 | −static bool __bpf_alu_result(u64 *res, u64 v1, u64 v2, u8 op) | |
| 563 | +enum { F_ALU32 = 1, F_SIGNED = 2 }; | |
| 564 | + | |
| 565 | +static bool __bpf_alu_result(u64 *res, u64 v1, u64 v2, u8 op, u32 flags) | |
| 564 | 566 | { |
| 567 | + bool is_signed = flags & F_SIGNED; | |
| 568 | + | |
| 569 | + /* Narrow operands for ALU32 */ | |
| 570 | + if (flags & F_ALU32) { | |
| 571 | + if (is_signed) { | |
| 572 | + v1 = (u64)(s32)v1; | |
| 573 | + v2 = (u64)(s32)v2; | |
| 574 | + } else { | |
| 575 | + v1 = (u32)v1; | |
| 576 | + v2 = (u32)v2; | |
| 577 | + } | |
| 578 | + } | |
| 579 | + | |
| 565 | 580 | *res = 0; |
| 566 | 581 | switch (op) { |
| 567 | 582 | case BPF_MOV: |
| @@ -599,12 +614,28 @@ static bool __bpf_alu_result(u64 *res, u64 v1, u64 v2, u8 op) · 31 unmodified lines | ||
| 599 | 614 | case BPF_DIV: |
| 600 | 615 | if (v2 == 0) |
| 601 | 616 | return false; |
| 602 | − *res = div64_u64(v1, v2); | |
| 617 | + if (!is_signed) { | |
| 618 | + *res = div64_u64(v1, v2); | |
| 619 | + } else { | |
| 620 | + if ((s64)v2 == -1) /* Handled by verifier */ | |
| 621 | + return false; | |
| 622 | + *res = (u64)div64_s64(v1, v2); | |
| 623 | + } | |
| 603 | 624 | break; |
| 604 | 625 | case BPF_MOD: |
| 605 | 626 | if (v2 == 0) |
| 606 | 627 | return false; |
| 607 | − div64_u64_rem(v1, v2, res); | |
| 628 | + if (!is_signed) { | |
| 629 | + div64_u64_rem(v1, v2, res); | |
| 630 | + } else { | |
| 631 | + if ((s64)v2 == -1) | |
| 632 | + return false; | |
| 633 | + /* | |
| 634 | + * Avoid s64 % s64 which generates __moddi3 on | |
| 635 | + * 32-bit architectures. Use div64_s64 instead. | |
| 636 | + */ | |
| 637 | + *res = (u64)((s64)v1 - div64_s64(v1, v2) * (s64)v2); | |
| 638 | + } | |
| 608 | 639 | break; |
| 609 | 640 | } |
| 610 | 641 | return true; |
| @@ -612,7 +643,7 @@ static bool __bpf_alu_result(u64 *res, u64 v1, u64 v2, u8 op) · 1 unmodified line | ||
| 612 | 643 | |
| 613 | 644 | /* Test an ALU shift operation for all valid shift values */ |
| 614 | 645 | static int __bpf_fill_alu_shift(struct bpf_test *self, u8 op, |
| 615 | − u8 mode, bool alu32) | |
| 646 | + u8 mode, u32 flags) | |
| 616 | 647 | { |
| 617 | 648 | static const s64 regs[] = { |
| 618 | 649 | 0x0123456789abcdefLL, /* dword > 0, word < 0 */ |
| @@ -620,7 +651,7 @@ static int __bpf_fill_alu_shift(struct bpf_test *self, u8 op, · 1 unmodified line | ||
| 620 | 651 | 0xfedcba0198765432LL, /* dword < 0, word < 0 */ |
| 621 | 652 | 0x0123458967abcdefLL, /* dword > 0, word > 0 */ |
| 622 | 653 | }; |
| 623 | − int bits = alu32 ? 32 : 64; | |
| 654 | + int bits = (flags & F_ALU32) ? 32 : 64; | |
| 624 | 655 | int len = (2 + 7 * bits) * ARRAY_SIZE(regs) + 3; |
| 625 | 656 | struct bpf_insn *insn; |
| 626 | 657 | int imm, k; |
| @@ -643,7 +674,7 @@ static int __bpf_fill_alu_shift(struct bpf_test *self, u8 op, · 16 unmodified lines | ||
| 643 | 674 | /* Perform operation */ |
| 644 | 675 | insn[i++] = BPF_ALU64_REG(BPF_MOV, R1, R3); |
| 645 | 676 | insn[i++] = BPF_ALU64_IMM(BPF_MOV, R2, imm); |
| 646 | − if (alu32) { | |
| 677 | + if (flags & F_ALU32) { | |
| 647 | 678 | if (mode == BPF_K) |
| 648 | 679 | insn[i++] = BPF_ALU32_IMM(op, R1, imm); |
| 649 | 680 | else |
| @@ -653,14 +684,14 @@ static int __bpf_fill_alu_shift(struct bpf_test *self, u8 op, · 3 unmodified lines | ||
| 653 | 684 | reg = (s32)reg; |
| 654 | 685 | else |
| 655 | 686 | reg = (u32)reg; |
| 656 | − __bpf_alu_result(&val, reg, imm, op); | |
| 687 | + __bpf_alu_result(&val, reg, imm, op, 0); | |
| 657 | 688 | val = (u32)val; |
| 658 | 689 | } else { |
| 659 | 690 | if (mode == BPF_K) |
| 660 | 691 | insn[i++] = BPF_ALU64_IMM(op, R1, imm); |
| 661 | 692 | else |
| 662 | 693 | insn[i++] = BPF_ALU64_REG(op, R1, R2); |
| 663 | − __bpf_alu_result(&val, reg, imm, op); | |
| 694 | + __bpf_alu_result(&val, reg, imm, op, 0); | |
| 664 | 695 | } |
| 665 | 696 | |
| 666 | 697 | /* |
| @@ -688,62 +719,62 @@ static int __bpf_fill_alu_shift(struct bpf_test *self, u8 op, · 21 unmodified lines | ||
| 688 | 719 | |
| 689 | 720 | static int bpf_fill_alu64_lsh_imm(struct bpf_test *self) |
| 690 | 721 | { |
| 691 | − return __bpf_fill_alu_shift(self, BPF_LSH, BPF_K, false); | |
| 722 | + return __bpf_fill_alu_shift(self, BPF_LSH, BPF_K, 0); | |
| 692 | 723 | } |
| 693 | 724 | |
| 694 | 725 | static int bpf_fill_alu64_rsh_imm(struct bpf_test *self) |
| 695 | 726 | { |
| 696 | − return __bpf_fill_alu_shift(self, BPF_RSH, BPF_K, false); | |
| 727 | + return __bpf_fill_alu_shift(self, BPF_RSH, BPF_K, 0); | |
| 697 | 728 | } |
| 698 | 729 | |
| 699 | 730 | static int bpf_fill_alu64_arsh_imm(struct bpf_test *self) |
| 700 | 731 | { |
| 701 | − return __bpf_fill_alu_shift(self, BPF_ARSH, BPF_K, false); | |
| 732 | + return __bpf_fill_alu_shift(self, BPF_ARSH, BPF_K, 0); | |
| 702 | 733 | } |
| 703 | 734 | |
| 704 | 735 | static int bpf_fill_alu64_lsh_reg(struct bpf_test *self) |
| 705 | 736 | { |
| 706 | − return __bpf_fill_alu_shift(self, BPF_LSH, BPF_X, false); | |
| 737 | + return __bpf_fill_alu_shift(self, BPF_LSH, BPF_X, 0); | |
| 707 | 738 | } |
| 708 | 739 | |
| 709 | 740 | static int bpf_fill_alu64_rsh_reg(struct bpf_test *self) |
| 710 | 741 | { |
| 711 | − return __bpf_fill_alu_shift(self, BPF_RSH, BPF_X, false); | |
| 742 | + return __bpf_fill_alu_shift(self, BPF_RSH, BPF_X, 0); | |
| 712 | 743 | } |
| 713 | 744 | |
| 714 | 745 | static int bpf_fill_alu64_arsh_reg(struct bpf_test *self) |
| 715 | 746 | { |
| 716 | − return __bpf_fill_alu_shift(self, BPF_ARSH, BPF_X, false); | |
| 747 | + return __bpf_fill_alu_shift(self, BPF_ARSH, BPF_X, 0); | |
| 717 | 748 | } |
| 718 | 749 | |
| 719 | 750 | static int bpf_fill_alu32_lsh_imm(struct bpf_test *self) |
| 720 | 751 | { |
| 721 | − return __bpf_fill_alu_shift(self, BPF_LSH, BPF_K, true); | |
| 752 | + return __bpf_fill_alu_shift(self, BPF_LSH, BPF_K, F_ALU32); | |
| 722 | 753 | } |
| 723 | 754 | |
| 724 | 755 | static int bpf_fill_alu32_rsh_imm(struct bpf_test *self) |
| 725 | 756 | { |
| 726 | − return __bpf_fill_alu_shift(self, BPF_RSH, BPF_K, true); | |
| 757 | + return __bpf_fill_alu_shift(self, BPF_RSH, BPF_K, F_ALU32); | |
| 727 | 758 | } |
| 728 | 759 | |
| 729 | 760 | static int bpf_fill_alu32_arsh_imm(struct bpf_test *self) |
| 730 | 761 | { |
| 731 | − return __bpf_fill_alu_shift(self, BPF_ARSH, BPF_K, true); | |
| 762 | + return __bpf_fill_alu_shift(self, BPF_ARSH, BPF_K, F_ALU32); | |
| 732 | 763 | } |
| 733 | 764 | |
| 734 | 765 | static int bpf_fill_alu32_lsh_reg(struct bpf_test *self) |
| 735 | 766 | { |
| 736 | − return __bpf_fill_alu_shift(self, BPF_LSH, BPF_X, true); | |
| 767 | + return __bpf_fill_alu_shift(self, BPF_LSH, BPF_X, F_ALU32); | |
| 737 | 768 | } |
| 738 | 769 | |
| 739 | 770 | static int bpf_fill_alu32_rsh_reg(struct bpf_test *self) |
| 740 | 771 | { |
| 741 | − return __bpf_fill_alu_shift(self, BPF_RSH, BPF_X, true); | |
| 772 | + return __bpf_fill_alu_shift(self, BPF_RSH, BPF_X, F_ALU32); | |
| 742 | 773 | } |
| 743 | 774 | |
| 744 | 775 | static int bpf_fill_alu32_arsh_reg(struct bpf_test *self) |
| 745 | 776 | { |
| 746 | − return __bpf_fill_alu_shift(self, BPF_ARSH, BPF_X, true); | |
| 777 | + return __bpf_fill_alu_shift(self, BPF_ARSH, BPF_X, F_ALU32); | |
| 747 | 778 | } |
| 748 | 779 | |
| 749 | 780 | /* |
| @@ -751,9 +782,9 @@ static int bpf_fill_alu32_arsh_reg(struct bpf_test *self) · 1 unmodified line | ||
| 751 | 782 | * for the case when the source and destination are the same. |
| 752 | 783 | */ |
| 753 | 784 | static int __bpf_fill_alu_shift_same_reg(struct bpf_test *self, u8 op, |
| 754 | − bool alu32) | |
| 785 | + u32 flags) | |
| 755 | 786 | { |
| 756 | − int bits = alu32 ? 32 : 64; | |
| 787 | + int bits = (flags & F_ALU32) ? 32 : 64; | |
| 757 | 788 | int len = 3 + 6 * bits; |
| 758 | 789 | struct bpf_insn *insn; |
| 759 | 790 | int i = 0; |
| @@ -770,14 +801,14 @@ static int __bpf_fill_alu_shift_same_reg(struct bpf_test *self, u8 op, · 10 unmodified lines | ||
| 770 | 801 | |
| 771 | 802 | /* Perform operation */ |
| 772 | 803 | insn[i++] = BPF_ALU64_IMM(BPF_MOV, R1, val); |
| 773 | − if (alu32) | |
| 804 | + if (flags & F_ALU32) | |
| 774 | 805 | insn[i++] = BPF_ALU32_REG(op, R1, R1); |
| 775 | 806 | else |
| 776 | 807 | insn[i++] = BPF_ALU64_REG(op, R1, R1); |
| 777 | 808 | |
| 778 | 809 | /* Compute the reference result */ |
| 779 | − __bpf_alu_result(&res, val, val, op); | |
| 780 | − if (alu32) | |
| 810 | + __bpf_alu_result(&res, val, val, op, 0); | |
| 811 | + if (flags & F_ALU32) | |
| 781 | 812 | res = (u32)res; |
| 782 | 813 | i += __bpf_ld_imm64(&insn[i], R2, res); |
| 783 | 814 | |
| @@ -798,32 +829,32 @@ static int __bpf_fill_alu_shift_same_reg(struct bpf_test *self, u8 op, · 14 unmodified lines | ||
| 798 | 829 | |
| 799 | 830 | static int bpf_fill_alu64_lsh_same_reg(struct bpf_test *self) |
| 800 | 831 | { |
| 801 | − return __bpf_fill_alu_shift_same_reg(self, BPF_LSH, false); | |
| 832 | + return __bpf_fill_alu_shift_same_reg(self, BPF_LSH, 0); | |
| 802 | 833 | } |
| 803 | 834 | |
| 804 | 835 | static int bpf_fill_alu64_rsh_same_reg(struct bpf_test *self) |
| 805 | 836 | { |
| 806 | − return __bpf_fill_alu_shift_same_reg(self, BPF_RSH, false); | |
| 837 | + return __bpf_fill_alu_shift_same_reg(self, BPF_RSH, 0); | |
| 807 | 838 | } |
| 808 | 839 | |
| 809 | 840 | static int bpf_fill_alu64_arsh_same_reg(struct bpf_test *self) |
| 810 | 841 | { |
| 811 | − return __bpf_fill_alu_shift_same_reg(self, BPF_ARSH, false); | |
| 842 | + return __bpf_fill_alu_shift_same_reg(self, BPF_ARSH, 0); | |
| 812 | 843 | } |
| 813 | 844 | |
| 814 | 845 | static int bpf_fill_alu32_lsh_same_reg(struct bpf_test *self) |
| 815 | 846 | { |
| 816 | − return __bpf_fill_alu_shift_same_reg(self, BPF_LSH, true); | |
| 847 | + return __bpf_fill_alu_shift_same_reg(self, BPF_LSH, F_ALU32); | |
| 817 | 848 | } |
| 818 | 849 | |
| 819 | 850 | static int bpf_fill_alu32_rsh_same_reg(struct bpf_test *self) |
| 820 | 851 | { |
| 821 | − return __bpf_fill_alu_shift_same_reg(self, BPF_RSH, true); | |
| 852 | + return __bpf_fill_alu_shift_same_reg(self, BPF_RSH, F_ALU32); | |
| 822 | 853 | } |
| 823 | 854 | |
| 824 | 855 | static int bpf_fill_alu32_arsh_same_reg(struct bpf_test *self) |
| 825 | 856 | { |
| 826 | − return __bpf_fill_alu_shift_same_reg(self, BPF_ARSH, true); | |
| 857 | + return __bpf_fill_alu_shift_same_reg(self, BPF_ARSH, F_ALU32); | |
| 827 | 858 | } |
| 828 | 859 | |
| 829 | 860 | /* |
| @@ -936,17 +967,20 @@ static int __bpf_fill_pattern(struct bpf_test *self, void *arg, · 106 unmodified lines | ||
| 936 | 967 | static int __bpf_emit_alu64_imm(struct bpf_test *self, void *arg, |
| 937 | 968 | struct bpf_insn *insns, s64 dst, s64 imm) |
| 938 | 969 | { |
| 939 | − int op = *(int *)arg; | |
| 970 | + int *a = arg; | |
| 971 | + int op = a[0]; | |
| 972 | + u32 flags = a[1]; | |
| 973 | + s16 off = (flags & F_SIGNED) ? 1 : 0; | |
| 940 | 974 | int i = 0; |
| 941 | 975 | u64 res; |
| 942 | 976 | |
| 943 | 977 | if (!insns) |
| 944 | 978 | return 7; |
| 945 | 979 | |
| 946 | − if (__bpf_alu_result(&res, dst, (s32)imm, op)) { | |
| 980 | + if (__bpf_alu_result(&res, dst, (s32)imm, op, flags)) { | |
| 947 | 981 | i += __bpf_ld_imm64(&insns[i], R1, dst); |
| 948 | 982 | i += __bpf_ld_imm64(&insns[i], R3, res); |
| 949 | − insns[i++] = BPF_ALU64_IMM(op, R1, imm); | |
| 983 | + insns[i++] = BPF_ALU64_IMM_OFF(op, R1, imm, off); | |
| 950 | 984 | insns[i++] = BPF_JMP_REG(BPF_JEQ, R1, R3, 1); |
| 951 | 985 | insns[i++] = BPF_EXIT_INSN(); |
| 952 | 986 | } |
| @@ -957,17 +991,20 @@ static int __bpf_emit_alu64_imm(struct bpf_test *self, void *arg, · 4 unmodified lines | ||
| 957 | 991 | static int __bpf_emit_alu32_imm(struct bpf_test *self, void *arg, |
| 958 | 992 | struct bpf_insn *insns, s64 dst, s64 imm) |
| 959 | 993 | { |
| 960 | − int op = *(int *)arg; | |
| 994 | + int *a = arg; | |
| 995 | + int op = a[0]; | |
| 996 | + u32 flags = a[1]; | |
| 997 | + s16 off = (flags & F_SIGNED) ? 1 : 0; | |
| 961 | 998 | int i = 0; |
| 962 | 999 | u64 res; |
| 963 | 1000 | |
| 964 | 1001 | if (!insns) |
| 965 | 1002 | return 7; |
| 966 | 1003 | |
| 967 | − if (__bpf_alu_result(&res, (u32)dst, (u32)imm, op)) { | |
| 1004 | + if (__bpf_alu_result(&res, dst, (s32)imm, op, flags | F_ALU32)) { | |
| 968 | 1005 | i += __bpf_ld_imm64(&insns[i], R1, dst); |
| 969 | 1006 | i += __bpf_ld_imm64(&insns[i], R3, (u32)res); |
| 970 | − insns[i++] = BPF_ALU32_IMM(op, R1, imm); | |
| 1007 | + insns[i++] = BPF_ALU32_IMM_OFF(op, R1, imm, off); | |
| 971 | 1008 | insns[i++] = BPF_JMP_REG(BPF_JEQ, R1, R3, 1); |
| 972 | 1009 | insns[i++] = BPF_EXIT_INSN(); |
| 973 | 1010 | } |
| @@ -985,7 +1022,7 @@ static int __bpf_emit_alu64_reg(struct bpf_test *self, void *arg, · 11 unmodified lines | ||
| 985 | 1022 | if (!insns) |
| 986 | 1023 | return 9; |
| 987 | 1024 | |
| 988 | − if (__bpf_alu_result(&res, dst, src, op)) { | |
| 1025 | + if (__bpf_alu_result(&res, dst, src, op, 0)) { | |
| 989 | 1026 | i += __bpf_ld_imm64(&insns[i], R1, dst); |
| 990 | 1027 | i += __bpf_ld_imm64(&insns[i], R2, src); |
| 991 | 1028 | i += __bpf_ld_imm64(&insns[i], R3, res); |
| @@ -1007,7 +1044,7 @@ static int __bpf_emit_alu32_reg(struct bpf_test *self, void *arg, · 15 unmodified lines | ||
| 1007 | 1044 | if (!insns) |
| 1008 | 1045 | return 9; |
| 1009 | 1046 | |
| 1010 | − if (__bpf_alu_result(&res, (u32)dst, (u32)src, op)) { | |
| 1047 | + if (__bpf_alu_result(&res, (u32)dst, (u32)src, op, 0)) { | |
| 1011 | 1048 | i += __bpf_ld_imm64(&insns[i], R1, dst); |
| 1012 | 1049 | i += __bpf_ld_imm64(&insns[i], R2, src); |
| 1013 | 1050 | i += __bpf_ld_imm64(&insns[i], R3, (u32)res); |
| @@ -1019,16 +1056,20 @@ static int __bpf_emit_alu32_reg(struct bpf_test *self, void *arg, · 5 unmodified lines | ||
| 1019 | 1056 | return i; |
| 1020 | 1057 | } |
| 1021 | 1058 | |
| 1022 | −static int __bpf_fill_alu64_imm(struct bpf_test *self, int op) | |
| 1059 | +static int __bpf_fill_alu64_imm(struct bpf_test *self, int op, u32 flags) | |
| 1023 | 1060 | { |
| 1024 | − return __bpf_fill_pattern(self, &op, 64, 32, | |
| 1061 | + int arg[2] = {op, flags}; | |
| 1062 | + | |
| 1063 | + return __bpf_fill_pattern(self, &arg, 64, 32, | |
| 1025 | 1064 | PATTERN_BLOCK1, PATTERN_BLOCK2, |
| 1026 | 1065 | &__bpf_emit_alu64_imm); |
| 1027 | 1066 | } |
| 1028 | 1067 | |
| 1029 | −static int __bpf_fill_alu32_imm(struct bpf_test *self, int op) | |
| 1068 | +static int __bpf_fill_alu32_imm(struct bpf_test *self, int op, u32 flags) | |
| 1030 | 1069 | { |
| 1031 | − return __bpf_fill_pattern(self, &op, 64, 32, | |
| 1070 | + int arg[2] = {op, flags}; | |
| 1071 | + | |
| 1072 | + return __bpf_fill_pattern(self, &arg, 64, 32, | |
| 1032 | 1073 | PATTERN_BLOCK1, PATTERN_BLOCK2, |
| 1033 | 1074 | &__bpf_emit_alu32_imm); |
| 1034 | 1075 | } |
| @@ -1050,93 +1091,115 @@ static int __bpf_fill_alu32_reg(struct bpf_test *self, int op) · 15 unmodified lines | ||
| 1050 | 1091 | /* ALU64 immediate operations */ |
| 1051 | 1092 | static int bpf_fill_alu64_mov_imm(struct bpf_test *self) |
| 1052 | 1093 | { |
| 1053 | − return __bpf_fill_alu64_imm(self, BPF_MOV); | |
| 1094 | + return __bpf_fill_alu64_imm(self, BPF_MOV, 0); | |
| 1054 | 1095 | } |
| 1055 | 1096 | |
| 1056 | 1097 | static int bpf_fill_alu64_and_imm(struct bpf_test *self) |
| 1057 | 1098 | { |
| 1058 | − return __bpf_fill_alu64_imm(self, BPF_AND); | |
| 1099 | + return __bpf_fill_alu64_imm(self, BPF_AND, 0); | |
| 1059 | 1100 | } |
| 1060 | 1101 | |
| 1061 | 1102 | static int bpf_fill_alu64_or_imm(struct bpf_test *self) |
| 1062 | 1103 | { |
| 1063 | − return __bpf_fill_alu64_imm(self, BPF_OR); | |
| 1104 | + return __bpf_fill_alu64_imm(self, BPF_OR, 0); | |
| 1064 | 1105 | } |
| 1065 | 1106 | |
| 1066 | 1107 | static int bpf_fill_alu64_xor_imm(struct bpf_test *self) |
| 1067 | 1108 | { |
| 1068 | − return __bpf_fill_alu64_imm(self, BPF_XOR); | |
| 1109 | + return __bpf_fill_alu64_imm(self, BPF_XOR, 0); | |
| 1069 | 1110 | } |
| 1070 | 1111 | |
| 1071 | 1112 | static int bpf_fill_alu64_add_imm(struct bpf_test *self) |
| 1072 | 1113 | { |
| 1073 | − return __bpf_fill_alu64_imm(self, BPF_ADD); | |
| 1114 | + return __bpf_fill_alu64_imm(self, BPF_ADD, 0); | |
| 1074 | 1115 | } |
| 1075 | 1116 | |
| 1076 | 1117 | static int bpf_fill_alu64_sub_imm(struct bpf_test *self) |
| 1077 | 1118 | { |
| 1078 | − return __bpf_fill_alu64_imm(self, BPF_SUB); | |
| 1119 | + return __bpf_fill_alu64_imm(self, BPF_SUB, 0); | |
| 1079 | 1120 | } |
| 1080 | 1121 | |
| 1081 | 1122 | static int bpf_fill_alu64_mul_imm(struct bpf_test *self) |
| 1082 | 1123 | { |
| 1083 | − return __bpf_fill_alu64_imm(self, BPF_MUL); | |
| 1124 | + return __bpf_fill_alu64_imm(self, BPF_MUL, 0); | |
| 1084 | 1125 | } |
| 1085 | 1126 | |
| 1086 | 1127 | static int bpf_fill_alu64_div_imm(struct bpf_test *self) |
| 1087 | 1128 | { |
| 1088 | − return __bpf_fill_alu64_imm(self, BPF_DIV); | |
| 1129 | + return __bpf_fill_alu64_imm(self, BPF_DIV, 0); | |
| 1089 | 1130 | } |
| 1090 | 1131 | |
| 1091 | 1132 | static int bpf_fill_alu64_mod_imm(struct bpf_test *self) |
| 1092 | 1133 | { |
| 1093 | − return __bpf_fill_alu64_imm(self, BPF_MOD); | |
| 1134 | + return __bpf_fill_alu64_imm(self, BPF_MOD, 0); | |
| 1135 | +} | |
| 1136 | + | |
| 1137 | +/* Signed ALU64 immediate operations */ | |
| 1138 | +static int bpf_fill_alu64_sdiv_imm(struct bpf_test *self) | |
| 1139 | +{ | |
| 1140 | + return __bpf_fill_alu64_imm(self, BPF_DIV, F_SIGNED); | |
| 1141 | +} | |
| 1142 | + | |
| 1143 | +static int bpf_fill_alu64_smod_imm(struct bpf_test *self) | |
| 1144 | +{ | |
| 1145 | + return __bpf_fill_alu64_imm(self, BPF_MOD, F_SIGNED); | |
| 1146 | +} | |
| 1147 | + | |
| 1148 | +/* Signed ALU32 immediate operations */ | |
| 1149 | +static int bpf_fill_alu32_sdiv_imm(struct bpf_test *self) | |
| 1150 | +{ | |
| 1151 | + return __bpf_fill_alu32_imm(self, BPF_DIV, F_SIGNED); | |
| 1152 | +} | |
| 1153 | + | |
| 1154 | +static int bpf_fill_alu32_smod_imm(struct bpf_test *self) | |
| 1155 | +{ | |
| 1156 | + return __bpf_fill_alu32_imm(self, BPF_MOD, F_SIGNED); | |
| 1094 | 1157 | } |
| 1095 | 1158 | |
| 1096 | 1159 | /* ALU32 immediate operations */ |
| 1097 | 1160 | static int bpf_fill_alu32_mov_imm(struct bpf_test *self) |
| 1098 | 1161 | { |
| 1099 | − return __bpf_fill_alu32_imm(self, BPF_MOV); | |
| 1162 | + return __bpf_fill_alu32_imm(self, BPF_MOV, 0); | |
| 1100 | 1163 | } |
| 1101 | 1164 | |
| 1102 | 1165 | static int bpf_fill_alu32_and_imm(struct bpf_test *self) |
| 1103 | 1166 | { |
| 1104 | − return __bpf_fill_alu32_imm(self, BPF_AND); | |
| 1167 | + return __bpf_fill_alu32_imm(self, BPF_AND, 0); | |
| 1105 | 1168 | } |
| 1106 | 1169 | |
| 1107 | 1170 | static int bpf_fill_alu32_or_imm(struct bpf_test *self) |
| 1108 | 1171 | { |
| 1109 | − return __bpf_fill_alu32_imm(self, BPF_OR); | |
| 1172 | + return __bpf_fill_alu32_imm(self, BPF_OR, 0); | |
| 1110 | 1173 | } |
| 1111 | 1174 | |
| 1112 | 1175 | static int bpf_fill_alu32_xor_imm(struct bpf_test *self) |
| 1113 | 1176 | { |
| 1114 | − return __bpf_fill_alu32_imm(self, BPF_XOR); | |
| 1177 | + return __bpf_fill_alu32_imm(self, BPF_XOR, 0); | |
| 1115 | 1178 | } |
| 1116 | 1179 | |
| 1117 | 1180 | static int bpf_fill_alu32_add_imm(struct bpf_test *self) |
| 1118 | 1181 | { |
| 1119 | − return __bpf_fill_alu32_imm(self, BPF_ADD); | |
| 1182 | + return __bpf_fill_alu32_imm(self, BPF_ADD, 0); | |
| 1120 | 1183 | } |
| 1121 | 1184 | |
| 1122 | 1185 | static int bpf_fill_alu32_sub_imm(struct bpf_test *self) |
| 1123 | 1186 | { |
| 1124 | − return __bpf_fill_alu32_imm(self, BPF_SUB); | |
| 1187 | + return __bpf_fill_alu32_imm(self, BPF_SUB, 0); | |
| 1125 | 1188 | } |
| 1126 | 1189 | |
| 1127 | 1190 | static int bpf_fill_alu32_mul_imm(struct bpf_test *self) |
| 1128 | 1191 | { |
| 1129 | − return __bpf_fill_alu32_imm(self, BPF_MUL); | |
| 1192 | + return __bpf_fill_alu32_imm(self, BPF_MUL, 0); | |
| 1130 | 1193 | } |
| 1131 | 1194 | |
| 1132 | 1195 | static int bpf_fill_alu32_div_imm(struct bpf_test *self) |
| 1133 | 1196 | { |
| 1134 | − return __bpf_fill_alu32_imm(self, BPF_DIV); | |
| 1197 | + return __bpf_fill_alu32_imm(self, BPF_DIV, 0); | |
| 1135 | 1198 | } |
| 1136 | 1199 | |
| 1137 | 1200 | static int bpf_fill_alu32_mod_imm(struct bpf_test *self) |
| 1138 | 1201 | { |
| 1139 | − return __bpf_fill_alu32_imm(self, BPF_MOD); | |
| 1202 | + return __bpf_fill_alu32_imm(self, BPF_MOD, 0); | |
| 1140 | 1203 | } |
| 1141 | 1204 | |
| 1142 | 1205 | /* ALU64 register operations */ |
| @@ -1235,7 +1298,8 @@ static int bpf_fill_alu32_mod_reg(struct bpf_test *self) · 92 unmodified lines | ||
| 1235 | 1298 | * Test JITs that implement complex ALU operations as function |
| 1236 | 1299 | * calls, and must re-arrange operands for argument passing. |
| 1237 | 1300 | */ |
| 1238 | −static int __bpf_fill_alu_imm_regs(struct bpf_test *self, u8 op, bool alu32) | |
| 1301 | +static int __bpf_fill_alu_imm_regs(struct bpf_test *self, u8 op, | |
| 1302 | + u32 flags) | |
| 1239 | 1303 | { |
| 1240 | 1304 | int len = 2 + 10 * 10; |
| 1241 | 1305 | struct bpf_insn *insns; |
| @@ -1249,28 +1313,37 @@ static int __bpf_fill_alu_imm_regs(struct bpf_test *self, u8 op, bool alu32) · 7 unmodified lines | ||
| 1249 | 1313 | return -ENOMEM; |
| 1250 | 1314 | |
| 1251 | 1315 | /* Operand and result values according to operation */ |
| 1252 | − if (alu32) | |
| 1253 | − dst = 0x76543210U; | |
| 1254 | − else | |
| 1255 | − dst = 0x7edcba9876543210ULL; | |
| 1316 | + if (flags & F_SIGNED) { | |
| 1317 | + if (flags & F_ALU32) | |
| 1318 | + dst = -76543210; | |
| 1319 | + else | |
| 1320 | + dst = -7654321076543210LL; | |
| 1321 | + } else { | |
| 1322 | + if (flags & F_ALU32) | |
| 1323 | + dst = 0x76543210U; | |
| 1324 | + else | |
| 1325 | + dst = 0x7edcba9876543210ULL; | |
| 1326 | + } | |
| 1256 | 1327 | imm = 0x01234567U; |
| 1257 | 1328 | |
| 1258 | 1329 | if (op == BPF_LSH || op == BPF_RSH || op == BPF_ARSH) |
| 1259 | 1330 | imm &= 31; |
| 1260 | 1331 | |
| 1261 | − __bpf_alu_result(&res, dst, imm, op); | |
| 1332 | + __bpf_alu_result(&res, dst, imm, op, flags); | |
| 1262 | 1333 | |
| 1263 | − if (alu32) | |
| 1334 | + if (flags & F_ALU32) | |
| 1264 | 1335 | res = (u32)res; |
| 1265 | 1336 | |
| 1266 | 1337 | /* Check all operand registers */ |
| 1267 | 1338 | for (rd = R0; rd <= R9; rd++) { |
| 1268 | 1339 | i += __bpf_ld_imm64(&insns[i], rd, dst); |
| 1269 | 1340 | |
| 1270 | − if (alu32) | |
| 1271 | − insns[i++] = BPF_ALU32_IMM(op, rd, imm); | |
| 1341 | + s16 off = (flags & F_SIGNED) ? 1 : 0; | |
| 1342 | + | |
| 1343 | + if (flags & F_ALU32) | |
| 1344 | + insns[i++] = BPF_ALU32_IMM_OFF(op, rd, imm, off); | |
| 1272 | 1345 | else |
| 1273 | − insns[i++] = BPF_ALU64_IMM(op, rd, imm); | |
| 1346 | + insns[i++] = BPF_ALU64_IMM_OFF(op, rd, imm, off); | |
| 1274 | 1347 | |
| 1275 | 1348 | insns[i++] = BPF_JMP32_IMM(BPF_JEQ, rd, res, 2); |
| 1276 | 1349 | insns[i++] = BPF_MOV64_IMM(R0, __LINE__); |
| @@ -1295,123 +1368,145 @@ static int __bpf_fill_alu_imm_regs(struct bpf_test *self, u8 op, bool alu32) · 18 unmodified lines | ||
| 1295 | 1368 | /* ALU64 K registers */ |
| 1296 | 1369 | static int bpf_fill_alu64_mov_imm_regs(struct bpf_test *self) |
| 1297 | 1370 | { |
| 1298 | − return __bpf_fill_alu_imm_regs(self, BPF_MOV, false); | |
| 1371 | + return __bpf_fill_alu_imm_regs(self, BPF_MOV, 0); | |
| 1299 | 1372 | } |
| 1300 | 1373 | |
| 1301 | 1374 | static int bpf_fill_alu64_and_imm_regs(struct bpf_test *self) |
| 1302 | 1375 | { |
| 1303 | − return __bpf_fill_alu_imm_regs(self, BPF_AND, false); | |
| 1376 | + return __bpf_fill_alu_imm_regs(self, BPF_AND, 0); | |
| 1304 | 1377 | } |
| 1305 | 1378 | |
| 1306 | 1379 | static int bpf_fill_alu64_or_imm_regs(struct bpf_test *self) |
| 1307 | 1380 | { |
| 1308 | − return __bpf_fill_alu_imm_regs(self, BPF_OR, false); | |
| 1381 | + return __bpf_fill_alu_imm_regs(self, BPF_OR, 0); | |
| 1309 | 1382 | } |
| 1310 | 1383 | |
| 1311 | 1384 | static int bpf_fill_alu64_xor_imm_regs(struct bpf_test *self) |
| 1312 | 1385 | { |
| 1313 | − return __bpf_fill_alu_imm_regs(self, BPF_XOR, false); | |
| 1386 | + return __bpf_fill_alu_imm_regs(self, BPF_XOR, 0); | |
| 1314 | 1387 | } |
| 1315 | 1388 | |
| 1316 | 1389 | static int bpf_fill_alu64_lsh_imm_regs(struct bpf_test *self) |
| 1317 | 1390 | { |
| 1318 | − return __bpf_fill_alu_imm_regs(self, BPF_LSH, false); | |
| 1391 | + return __bpf_fill_alu_imm_regs(self, BPF_LSH, 0); | |
| 1319 | 1392 | } |
| 1320 | 1393 | |
| 1321 | 1394 | static int bpf_fill_alu64_rsh_imm_regs(struct bpf_test *self) |
| 1322 | 1395 | { |
| 1323 | − return __bpf_fill_alu_imm_regs(self, BPF_RSH, false); | |
| 1396 | + return __bpf_fill_alu_imm_regs(self, BPF_RSH, 0); | |
| 1324 | 1397 | } |
| 1325 | 1398 | |
| 1326 | 1399 | static int bpf_fill_alu64_arsh_imm_regs(struct bpf_test *self) |
| 1327 | 1400 | { |
| 1328 | − return __bpf_fill_alu_imm_regs(self, BPF_ARSH, false); | |
| 1401 | + return __bpf_fill_alu_imm_regs(self, BPF_ARSH, 0); | |
| 1329 | 1402 | } |
| 1330 | 1403 | |
| 1331 | 1404 | static int bpf_fill_alu64_add_imm_regs(struct bpf_test *self) |
| 1332 | 1405 | { |
| 1333 | − return __bpf_fill_alu_imm_regs(self, BPF_ADD, false); | |
| 1406 | + return __bpf_fill_alu_imm_regs(self, BPF_ADD, 0); | |
| 1334 | 1407 | } |
| 1335 | 1408 | |
| 1336 | 1409 | static int bpf_fill_alu64_sub_imm_regs(struct bpf_test *self) |
| 1337 | 1410 | { |
| 1338 | − return __bpf_fill_alu_imm_regs(self, BPF_SUB, false); | |
| 1411 | + return __bpf_fill_alu_imm_regs(self, BPF_SUB, 0); | |
| 1339 | 1412 | } |
| 1340 | 1413 | |
| 1341 | 1414 | static int bpf_fill_alu64_mul_imm_regs(struct bpf_test *self) |
| 1342 | 1415 | { |
| 1343 | − return __bpf_fill_alu_imm_regs(self, BPF_MUL, false); | |
| 1416 | + return __bpf_fill_alu_imm_regs(self, BPF_MUL, 0); | |
| 1344 | 1417 | } |
| 1345 | 1418 | |
| 1346 | 1419 | static int bpf_fill_alu64_div_imm_regs(struct bpf_test *self) |
| 1347 | 1420 | { |
| 1348 | − return __bpf_fill_alu_imm_regs(self, BPF_DIV, false); | |
| 1421 | + return __bpf_fill_alu_imm_regs(self, BPF_DIV, 0); | |
| 1349 | 1422 | } |
| 1350 | 1423 | |
| 1351 | 1424 | static int bpf_fill_alu64_mod_imm_regs(struct bpf_test *self) |
| 1352 | 1425 | { |
| 1353 | − return __bpf_fill_alu_imm_regs(self, BPF_MOD, false); | |
| 1426 | + return __bpf_fill_alu_imm_regs(self, BPF_MOD, 0); | |
| 1427 | +} | |
| 1428 | + | |
| 1429 | +/* Signed ALU64 K registers */ | |
| 1430 | +static int bpf_fill_alu64_sdiv_imm_regs(struct bpf_test *self) | |
| 1431 | +{ | |
| 1432 | + return __bpf_fill_alu_imm_regs(self, BPF_DIV, F_SIGNED); | |
| 1433 | +} | |
| 1434 | + | |
| 1435 | +static int bpf_fill_alu64_smod_imm_regs(struct bpf_test *self) | |
| 1436 | +{ | |
| 1437 | + return __bpf_fill_alu_imm_regs(self, BPF_MOD, F_SIGNED); | |
| 1354 | 1438 | } |
| 1355 | 1439 | |
| 1356 | 1440 | /* ALU32 K registers */ |
| 1357 | 1441 | static int bpf_fill_alu32_mov_imm_regs(struct bpf_test *self) |
| 1358 | 1442 | { |
| 1359 | − return __bpf_fill_alu_imm_regs(self, BPF_MOV, true); | |
| 1443 | + return __bpf_fill_alu_imm_regs(self, BPF_MOV, F_ALU32); | |
| 1360 | 1444 | } |
| 1361 | 1445 | |
| 1362 | 1446 | static int bpf_fill_alu32_and_imm_regs(struct bpf_test *self) |
| 1363 | 1447 | { |
| 1364 | − return __bpf_fill_alu_imm_regs(self, BPF_AND, true); | |
| 1448 | + return __bpf_fill_alu_imm_regs(self, BPF_AND, F_ALU32); | |
| 1365 | 1449 | } |
| 1366 | 1450 | |
| 1367 | 1451 | static int bpf_fill_alu32_or_imm_regs(struct bpf_test *self) |
| 1368 | 1452 | { |
| 1369 | − return __bpf_fill_alu_imm_regs(self, BPF_OR, true); | |
| 1453 | + return __bpf_fill_alu_imm_regs(self, BPF_OR, F_ALU32); | |
| 1370 | 1454 | } |
| 1371 | 1455 | |
| 1372 | 1456 | static int bpf_fill_alu32_xor_imm_regs(struct bpf_test *self) |
| 1373 | 1457 | { |
| 1374 | − return __bpf_fill_alu_imm_regs(self, BPF_XOR, true); | |
| 1458 | + return __bpf_fill_alu_imm_regs(self, BPF_XOR, F_ALU32); | |
| 1375 | 1459 | } |
| 1376 | 1460 | |
| 1377 | 1461 | static int bpf_fill_alu32_lsh_imm_regs(struct bpf_test *self) |
| 1378 | 1462 | { |
| 1379 | − return __bpf_fill_alu_imm_regs(self, BPF_LSH, true); | |
| 1463 | + return __bpf_fill_alu_imm_regs(self, BPF_LSH, F_ALU32); | |
| 1380 | 1464 | } |
| 1381 | 1465 | |
| 1382 | 1466 | static int bpf_fill_alu32_rsh_imm_regs(struct bpf_test *self) |
| 1383 | 1467 | { |
| 1384 | − return __bpf_fill_alu_imm_regs(self, BPF_RSH, true); | |
| 1468 | + return __bpf_fill_alu_imm_regs(self, BPF_RSH, F_ALU32); | |
| 1385 | 1469 | } |
| 1386 | 1470 | |
| 1387 | 1471 | static int bpf_fill_alu32_arsh_imm_regs(struct bpf_test *self) |
| 1388 | 1472 | { |
| 1389 | − return __bpf_fill_alu_imm_regs(self, BPF_ARSH, true); | |
| 1473 | + return __bpf_fill_alu_imm_regs(self, BPF_ARSH, F_ALU32); | |
| 1390 | 1474 | } |
| 1391 | 1475 | |
| 1392 | 1476 | static int bpf_fill_alu32_add_imm_regs(struct bpf_test *self) |
| 1393 | 1477 | { |
| 1394 | − return __bpf_fill_alu_imm_regs(self, BPF_ADD, true); | |
| 1478 | + return __bpf_fill_alu_imm_regs(self, BPF_ADD, F_ALU32); | |
| 1395 | 1479 | } |
| 1396 | 1480 | |
| 1397 | 1481 | static int bpf_fill_alu32_sub_imm_regs(struct bpf_test *self) |
| 1398 | 1482 | { |
| 1399 | − return __bpf_fill_alu_imm_regs(self, BPF_SUB, true); | |
| 1483 | + return __bpf_fill_alu_imm_regs(self, BPF_SUB, F_ALU32); | |
| 1400 | 1484 | } |
| 1401 | 1485 | |
| 1402 | 1486 | static int bpf_fill_alu32_mul_imm_regs(struct bpf_test *self) |
| 1403 | 1487 | { |
| 1404 | − return __bpf_fill_alu_imm_regs(self, BPF_MUL, true); | |
| 1488 | + return __bpf_fill_alu_imm_regs(self, BPF_MUL, F_ALU32); | |
| 1405 | 1489 | } |
| 1406 | 1490 | |
| 1407 | 1491 | static int bpf_fill_alu32_div_imm_regs(struct bpf_test *self) |
| 1408 | 1492 | { |
| 1409 | − return __bpf_fill_alu_imm_regs(self, BPF_DIV, true); | |
| 1493 | + return __bpf_fill_alu_imm_regs(self, BPF_DIV, F_ALU32); | |
| 1410 | 1494 | } |
| 1411 | 1495 | |
| 1412 | 1496 | static int bpf_fill_alu32_mod_imm_regs(struct bpf_test *self) |
| 1413 | 1497 | { |
| 1414 | − return __bpf_fill_alu_imm_regs(self, BPF_MOD, true); | |
| 1498 | + return __bpf_fill_alu_imm_regs(self, BPF_MOD, F_ALU32); | |
| 1499 | +} | |
| 1500 | + | |
| 1501 | +/* Signed ALU32 K registers */ | |
| 1502 | +static int bpf_fill_alu32_sdiv_imm_regs(struct bpf_test *self) | |
| 1503 | +{ | |
| 1504 | + return __bpf_fill_alu_imm_regs(self, BPF_DIV, F_ALU32 | F_SIGNED); | |
| 1505 | +} | |
| 1506 | + | |
| 1507 | +static int bpf_fill_alu32_smod_imm_regs(struct bpf_test *self) | |
| 1508 | +{ | |
| 1509 | + return __bpf_fill_alu_imm_regs(self, BPF_MOD, F_ALU32 | F_SIGNED); | |
| 1415 | 1510 | } |
| 1416 | 1511 | |
| 1417 | 1512 | /* |
| @@ -1442,8 +1537,8 @@ static int __bpf_fill_alu_reg_pairs(struct bpf_test *self, u8 op, bool alu32) · 24 unmodified lines | ||
| 1442 | 1537 | if (op == BPF_LSH || op == BPF_RSH || op == BPF_ARSH) |
| 1443 | 1538 | src &= 31; |
| 1444 | 1539 | |
| 1445 | − __bpf_alu_result(&res, dst, src, op); | |
| 1446 | − __bpf_alu_result(&same, src, src, op); | |
| 1540 | + __bpf_alu_result(&res, dst, src, op, 0); | |
| 1541 | + __bpf_alu_result(&same, src, src, op, 0); | |
| 1447 | 1542 | |
| 1448 | 1543 | if (alu32) { |
| 1449 | 1544 | res = (u32)res; |
| @@ -1626,7 +1721,7 @@ static int __bpf_emit_atomic64(struct bpf_test *self, void *arg, · 176 unmodified lines | ||
| 1626 | 1721 | res = src; |
| 1627 | 1722 | break; |
| 1628 | 1723 | default: |
| 1629 | − __bpf_alu_result(&res, dst, src, BPF_OP(op)); | |
| 1724 | + __bpf_alu_result(&res, dst, src, BPF_OP(op), 0); | |
| 1630 | 1725 | } |
| 1631 | 1726 | |
| 1632 | 1727 | keep = 0x0123456789abcdefULL; |
| @@ -1673,7 +1768,7 @@ static int __bpf_emit_atomic32(struct bpf_test *self, void *arg, · 40 unmodified lines | ||
| 1673 | 1768 | res = src; |
| 1674 | 1769 | break; |
| 1675 | 1770 | default: |
| 1676 | − __bpf_alu_result(&res, (u32)dst, (u32)src, BPF_OP(op)); | |
| 1771 | + __bpf_alu_result(&res, (u32)dst, (u32)src, BPF_OP(op), 0); | |
| 1677 | 1772 | } |
| 1678 | 1773 | |
| 1679 | 1774 | keep = 0x0123456789abcdefULL; |
| @@ -1939,7 +2034,7 @@ static int __bpf_fill_atomic_reg_pairs(struct bpf_test *self, u8 width, u8 op) · 259 unmodified lines | ||
| 1939 | 2034 | res = mem; |
| 1940 | 2035 | break; |
| 1941 | 2036 | default: |
| 1942 | − __bpf_alu_result(&res, mem, upd, BPF_OP(op)); | |
| 2037 | + __bpf_alu_result(&res, mem, upd, BPF_OP(op), 0); | |
| 1943 | 2038 | } |
| 1944 | 2039 | |
| 1945 | 2040 | /* Test all operand registers */ |
| @@ -12354,6 +12449,22 @@ static struct bpf_test tests[] = { · 10408 unmodified lines | ||
| 12354 | 12449 | { { 0, 1 } }, |
| 12355 | 12450 | .fill_helper = bpf_fill_alu64_mod_imm_regs, |
| 12356 | 12451 | }, |
| 12452 | + { | |
| 12453 | + "ALU64_SDIV_K: registers", | |
| 12454 | + { }, | |
| 12455 | + INTERNAL, | |
| 12456 | + { }, | |
| 12457 | + { { 0, 1 } }, | |
| 12458 | + .fill_helper = bpf_fill_alu64_sdiv_imm_regs, | |
| 12459 | + }, | |
| 12460 | + { | |
| 12461 | + "ALU64_SMOD_K: registers", | |
| 12462 | + { }, | |
| 12463 | + INTERNAL, | |
| 12464 | + { }, | |
| 12465 | + { { 0, 1 } }, | |
| 12466 | + .fill_helper = bpf_fill_alu64_smod_imm_regs, | |
| 12467 | + }, | |
| 12357 | 12468 | /* ALU32 K registers */ |
| 12358 | 12469 | { |
| 12359 | 12470 | "ALU32_MOV_K: registers", |
| @@ -12451,6 +12562,22 @@ static struct bpf_test tests[] = { · 91 unmodified lines | ||
| 12451 | 12562 | { { 0, 1 } }, |
| 12452 | 12563 | .fill_helper = bpf_fill_alu32_mod_imm_regs, |
| 12453 | 12564 | }, |
| 12565 | + { | |
| 12566 | + "ALU32_SDIV_K: registers", | |
| 12567 | + { }, | |
| 12568 | + INTERNAL, | |
| 12569 | + { }, | |
| 12570 | + { { 0, 1 } }, | |
| 12571 | + .fill_helper = bpf_fill_alu32_sdiv_imm_regs, | |
| 12572 | + }, | |
| 12573 | + { | |
| 12574 | + "ALU32_SMOD_K: registers", | |
| 12575 | + { }, | |
| 12576 | + INTERNAL, | |
| 12577 | + { }, | |
| 12578 | + { { 0, 1 } }, | |
| 12579 | + .fill_helper = bpf_fill_alu32_smod_imm_regs, | |
| 12580 | + }, | |
| 12454 | 12581 | /* ALU64 X register combinations */ |
| 12455 | 12582 | { |
| 12456 | 12583 | "ALU64_MOV_X: register combinations", |
| @@ -12881,6 +13008,24 @@ static struct bpf_test tests[] = { · 424 unmodified lines | ||
| 12881 | 13008 | .fill_helper = bpf_fill_alu64_mod_imm, |
| 12882 | 13009 | .nr_testruns = NR_PATTERN_RUNS, |
| 12883 | 13010 | }, |
| 13011 | + { | |
| 13012 | + "ALU64_SDIV_K: all immediate value magnitudes", | |
| 13013 | + { }, | |
| 13014 | + INTERNAL | FLAG_NO_DATA, | |
| 13015 | + { }, | |
| 13016 | + { { 0, 1 } }, | |
| 13017 | + .fill_helper = bpf_fill_alu64_sdiv_imm, | |
| 13018 | + .nr_testruns = NR_PATTERN_RUNS, | |
| 13019 | + }, | |
| 13020 | + { | |
| 13021 | + "ALU64_SMOD_K: all immediate value magnitudes", | |
| 13022 | + { }, | |
| 13023 | + INTERNAL | FLAG_NO_DATA, | |
| 13024 | + { }, | |
| 13025 | + { { 0, 1 } }, | |
| 13026 | + .fill_helper = bpf_fill_alu64_smod_imm, | |
| 13027 | + .nr_testruns = NR_PATTERN_RUNS, | |
| 13028 | + }, | |
| 12884 | 13029 | /* ALU32 immediate magnitudes */ |
| 12885 | 13030 | { |
| 12886 | 13031 | "ALU32_MOV_K: all immediate value magnitudes", |
| @@ -12963,6 +13108,24 @@ static struct bpf_test tests[] = { · 76 unmodified lines | ||
| 12963 | 13108 | .fill_helper = bpf_fill_alu32_mod_imm, |
| 12964 | 13109 | .nr_testruns = NR_PATTERN_RUNS, |
| 12965 | 13110 | }, |
| 13111 | + { | |
| 13112 | + "ALU32_SDIV_K: all immediate value magnitudes", | |
| 13113 | + { }, | |
| 13114 | + INTERNAL | FLAG_NO_DATA, | |
| 13115 | + { }, | |
| 13116 | + { { 0, 1 } }, | |
| 13117 | + .fill_helper = bpf_fill_alu32_sdiv_imm, | |
| 13118 | + .nr_testruns = NR_PATTERN_RUNS, | |
| 13119 | + }, | |
| 13120 | + { | |
| 13121 | + "ALU32_SMOD_K: all immediate value magnitudes", | |
| 13122 | + { }, | |
| 13123 | + INTERNAL | FLAG_NO_DATA, | |
| 13124 | + { }, | |
| 13125 | + { { 0, 1 } }, | |
| 13126 | + .fill_helper = bpf_fill_alu32_smod_imm, | |
| 13127 | + .nr_testruns = NR_PATTERN_RUNS, | |
| 13128 | + }, | |
| 12966 | 13129 | /* ALU64 register magnitudes */ |
| 12967 | 13130 | { |
| 12968 | 13131 | "ALU64_MOV_X: all register value magnitudes", |
| @@ -679,6 +679,78 @@ out: · 678 unmodified lines | ||
| 679 | 679 | return err; |
| 680 | 680 | } |
| 681 | 681 | |
| 682 | +static int __init test_rhashtable_next_key(void) | |
| 683 | +{ | |
| 684 | + struct rhashtable_params params = test_rht_params; | |
| 685 | + struct test_obj_val key_missing = { .id = 99999, .tid = 0 }; | |
| 686 | + struct test_obj_val *prev_key = NULL; | |
| 687 | + struct rhashtable ht; | |
| 688 | + struct test_obj *objs, *cur; | |
| 689 | + int i, count = 0, err; | |
| 690 | + int visited_keys[8] = { 0 }; | |
| 691 | + const int n = ARRAY_SIZE(visited_keys); | |
| 692 | + | |
| 693 | + params.nelem_hint = n; | |
| 694 | + | |
| 695 | + err = rhashtable_init(&ht, ¶ms); | |
| 696 | + if (err) | |
| 697 | + return err; | |
| 698 | + | |
| 699 | + objs = kcalloc(n, sizeof(*objs), GFP_KERNEL); | |
| 700 | + if (!objs) { | |
| 701 | + rhashtable_destroy(&ht); | |
| 702 | + return -ENOMEM; | |
| 703 | + } | |
| 704 | + | |
| 705 | + for (i = 0; i < n; i++) { | |
| 706 | + objs[i].value.id = i; | |
| 707 | + err = rhashtable_insert_fast(&ht, &objs[i].node, params); | |
| 708 | + if (err) | |
| 709 | + goto out; | |
| 710 | + } | |
| 711 | + | |
| 712 | + rcu_read_lock(); | |
| 713 | + | |
| 714 | + /* NULL prev_key: walk from the beginning, expect all n elements. */ | |
| 715 | + while ((cur = rhashtable_next_key(&ht, prev_key))) { | |
| 716 | + if (IS_ERR(cur)) { | |
| 717 | + err = -EINVAL; | |
| 718 | + goto unlock; | |
| 719 | + } | |
| 720 | + count++; | |
| 721 | + prev_key = &cur->value; | |
| 722 | + visited_keys[cur->value.id] = 1; | |
| 723 | + if (count > n) | |
| 724 | + break; | |
| 725 | + } | |
| 726 | + | |
| 727 | + if (count != n) { | |
| 728 | + err = -EINVAL; | |
| 729 | + goto unlock; | |
| 730 | + } | |
| 731 | + | |
| 732 | + for (i = 0; i < n; i++) { | |
| 733 | + if (!visited_keys[i]) { | |
| 734 | + err = -EINVAL; | |
| 735 | + goto unlock; | |
| 736 | + } | |
| 737 | + } | |
| 738 | + | |
| 739 | + /* Non-existing prev_key: must return ERR_PTR(-ENOENT). */ | |
| 740 | + cur = rhashtable_next_key(&ht, &key_missing); | |
| 741 | + if (!IS_ERR(cur) || PTR_ERR(cur) != -ENOENT) | |
| 742 | + err = -EINVAL; | |
| 743 | + | |
| 744 | +unlock: | |
| 745 | + rcu_read_unlock(); | |
| 746 | +out: | |
| 747 | + for (i = 0; i < n; i++) | |
| 748 | + rhashtable_remove_fast(&ht, &objs[i].node, params); | |
| 749 | + kfree(objs); | |
| 750 | + rhashtable_destroy(&ht); | |
| 751 | + return err; | |
| 752 | +} | |
| 753 | + | |
| 682 | 754 | static int __init test_rht_init(void) |
| 683 | 755 | { |
| 684 | 756 | unsigned int entries; |
| @@ -738,6 +810,9 @@ static int __init test_rht_init(void) · 53 unmodified lines | ||
| 738 | 810 | |
| 739 | 811 | test_insert_duplicates_run(); |
| 740 | 812 | |
| 813 | + pr_info("Testing rhashtable_next_key: %s\n", | |
| 814 | + test_rhashtable_next_key() == 0 ? "pass" : "FAIL"); | |
| 815 | + | |
| 741 | 816 | if (!tcount) |
| 742 | 817 | return 0; |
| 743 | 818 | |
| @@ -132,7 +132,7 @@ int bpf_struct_ops_test_run(struct bpf_prog *prog, const union bpf_attr *kattr, · 131 unmodified lines | ||
| 132 | 132 | const struct bpf_struct_ops *st_ops = &bpf_bpf_dummy_ops; |
| 133 | 133 | const struct btf_type *func_proto; |
| 134 | 134 | struct bpf_dummy_ops_test_args *args; |
| 135 | − struct bpf_tramp_links *tlinks = NULL; | |
| 135 | + struct bpf_tramp_nodes *tnodes = NULL; | |
| 136 | 136 | struct bpf_tramp_link *link = NULL; |
| 137 | 137 | void *image = NULL; |
| 138 | 138 | unsigned int op_idx; |
| @@ -158,8 +158,8 @@ int bpf_struct_ops_test_run(struct bpf_prog *prog, const union bpf_attr *kattr, · 19 unmodified lines | ||
| 158 | 158 | if (err) |
| 159 | 159 | goto out; |
| 160 | 160 | |
| 161 | − tlinks = kzalloc_objs(*tlinks, BPF_TRAMP_MAX); | |
| 162 | − if (!tlinks) { | |
| 161 | + tnodes = kzalloc_objs(*tnodes, BPF_TRAMP_MAX); | |
| 162 | + if (!tnodes) { | |
| 163 | 163 | err = -ENOMEM; |
| 164 | 164 | goto out; |
| 165 | 165 | } |
| @@ -171,11 +171,11 @@ int bpf_struct_ops_test_run(struct bpf_prog *prog, const union bpf_attr *kattr, · 5 unmodified lines | ||
| 171 | 171 | } |
| 172 | 172 | /* prog doesn't take the ownership of the reference from caller */ |
| 173 | 173 | bpf_prog_inc(prog); |
| 174 | − bpf_link_init(&link->link, BPF_LINK_TYPE_STRUCT_OPS, &bpf_struct_ops_link_lops, prog, | |
| 175 | − prog->expected_attach_type); | |
| 174 | + bpf_tramp_link_init(link, BPF_LINK_TYPE_STRUCT_OPS, &bpf_struct_ops_link_lops, | |
| 175 | + prog, prog->expected_attach_type, 0); | |
| 176 | 176 | |
| 177 | 177 | op_idx = prog->expected_attach_type; |
| 178 | − err = bpf_struct_ops_prepare_trampoline(tlinks, link, | |
| 178 | + err = bpf_struct_ops_prepare_trampoline(tnodes, &link->node, | |
| 179 | 179 | &st_ops->func_models[op_idx], |
| 180 | 180 | &dummy_ops_test_ret_function, |
| 181 | 181 | &image, &image_off, |
| @@ -198,7 +198,7 @@ out: · 16 unmodified lines | ||
| 198 | 198 | bpf_struct_ops_image_free(image); |
| 199 | 199 | if (link) |
| 200 | 200 | bpf_link_put(&link->link); |
| 201 | − kfree(tlinks); | |
| 201 | + kfree(tnodes); | |
| 202 | 202 | return err; |
| 203 | 203 | } |
| 204 | 204 | |
| @@ -702,6 +702,9 @@ int bpf_prog_test_run_tracing(struct bpf_prog *prog, · 701 unmodified lines | ||
| 702 | 702 | case BPF_TRACE_FENTRY: |
| 703 | 703 | case BPF_TRACE_FEXIT: |
| 704 | 704 | case BPF_TRACE_FSESSION: |
| 705 | + case BPF_TRACE_FENTRY_MULTI: | |
| 706 | + case BPF_TRACE_FEXIT_MULTI: | |
| 707 | + case BPF_TRACE_FSESSION_MULTI: | |
| 705 | 708 | if (bpf_fentry_test1(1) != 2 || |
| 706 | 709 | bpf_fentry_test2(2, 3) != 5 || |
| 707 | 710 | bpf_fentry_test3(4, 5, 6) != 15 || |
| @@ -747,14 +750,35 @@ static void · 39 unmodified lines | ||
| 747 | 750 | __bpf_prog_test_run_raw_tp(void *data) |
| 748 | 751 | { |
| 749 | 752 | struct bpf_raw_tp_test_run_info *info = data; |
| 753 | + struct srcu_ctr __percpu *scp = NULL; | |
| 750 | 754 | struct bpf_trace_run_ctx run_ctx = {}; |
| 751 | 755 | struct bpf_run_ctx *old_run_ctx; |
| 752 | 756 | |
| 753 | 757 | old_run_ctx = bpf_set_run_ctx(&run_ctx.run_ctx); |
| 754 | 758 | |
| 755 | − rcu_read_lock(); | |
| 759 | + if (info->prog->sleepable) { | |
| 760 | + scp = rcu_read_lock_tasks_trace(); | |
| 761 | + migrate_disable(); | |
| 762 | + } else { | |
| 763 | + rcu_read_lock(); | |
| 764 | + } | |
| 765 | + | |
| 766 | + if (unlikely(!bpf_prog_get_recursion_context(info->prog))) { | |
| 767 | + bpf_prog_inc_misses_counter(info->prog); | |
| 768 | + goto out; | |
| 769 | + } | |
| 770 | + | |
| 756 | 771 | info->retval = bpf_prog_run(info->prog, info->ctx); |
| 757 | − rcu_read_unlock(); | |
| 772 | + | |
| 773 | +out: | |
| 774 | + bpf_prog_put_recursion_context(info->prog); | |
| 775 | + | |
| 776 | + if (info->prog->sleepable) { | |
| 777 | + migrate_enable(); | |
| 778 | + rcu_read_unlock_tasks_trace(scp); | |
| 779 | + } else { | |
| 780 | + rcu_read_unlock(); | |
| 781 | + } | |
| 758 | 782 | |
| 759 | 783 | bpf_reset_run_ctx(old_run_ctx); |
| 760 | 784 | } |
| @@ -782,6 +806,13 @@ int bpf_prog_test_run_raw_tp(struct bpf_prog *prog, · 21 unmodified lines | ||
| 782 | 806 | if ((kattr->test.flags & BPF_F_TEST_RUN_ON_CPU) == 0 && cpu != 0) |
| 783 | 807 | return -EINVAL; |
| 784 | 808 | |
| 809 | + /* | |
| 810 | + * Sleepable programs cannot run with preemption disabled or in | |
| 811 | + * hardirq context (smp_call_function_single), reject the flag. | |
| 812 | + */ | |
| 813 | + if (prog->sleepable && (kattr->test.flags & BPF_F_TEST_RUN_ON_CPU)) | |
| 814 | + return -EINVAL; | |
| 815 | + | |
| 785 | 816 | if (ctx_size_in) { |
| 786 | 817 | info.ctx = memdup_user(ctx_in, ctx_size_in); |
| 787 | 818 | if (IS_ERR(info.ctx)) |
| @@ -790,24 +821,31 @@ int bpf_prog_test_run_raw_tp(struct bpf_prog *prog, · 2 unmodified lines | ||
| 790 | 821 | info.ctx = NULL; |
| 791 | 822 | } |
| 792 | 823 | |
| 824 | + info.retval = 0; | |
| 793 | 825 | info.prog = prog; |
| 794 | 826 | |
| 795 | − current_cpu = get_cpu(); | |
| 796 | − if ((kattr->test.flags & BPF_F_TEST_RUN_ON_CPU) == 0 || | |
| 797 | − cpu == current_cpu) { | |
| 827 | + if (prog->sleepable) { | |
| 798 | 828 | __bpf_prog_test_run_raw_tp(&info); |
| 799 | − } else if (cpu >= nr_cpu_ids || !cpu_online(cpu)) { | |
| 800 | − /* smp_call_function_single() also checks cpu_online() | |
| 801 | − * after csd_lock(). However, since cpu is from user | |
| 802 | − * space, let's do an extra quick check to filter out | |
| 803 | − * invalid value before smp_call_function_single(). | |
| 804 | − */ | |
| 805 | − err = -ENXIO; | |
| 806 | 829 | } else { |
| 807 | − err = smp_call_function_single(cpu, __bpf_prog_test_run_raw_tp, | |
| 808 | − &info, 1); | |
| 830 | + current_cpu = get_cpu(); | |
| 831 | + if ((kattr->test.flags & BPF_F_TEST_RUN_ON_CPU) == 0 || | |
| 832 | + cpu == current_cpu) { | |
| 833 | + __bpf_prog_test_run_raw_tp(&info); | |
| 834 | + } else if (cpu >= nr_cpu_ids || !cpu_online(cpu)) { | |
| 835 | + /* | |
| 836 | + * smp_call_function_single() also checks cpu_online() | |
| 837 | + * after csd_lock(). However, since cpu is from user | |
| 838 | + * space, let's do an extra quick check to filter out | |
| 839 | + * invalid value before smp_call_function_single(). | |
| 840 | + */ | |
| 841 | + err = -ENXIO; | |
| 842 | + } else { | |
| 843 | + err = smp_call_function_single(cpu, | |
| 844 | + __bpf_prog_test_run_raw_tp, | |
| 845 | + &info, 1); | |
| 846 | + } | |
| 847 | + put_cpu(); | |
| 809 | 848 | } |
| 810 | − put_cpu(); | |
| 811 | 849 | |
| 812 | 850 | if (!err && |
| 813 | 851 | copy_to_user(&uattr->test.retval, &info.retval, sizeof(u32))) |
| @@ -2654,6 +2654,37 @@ static void sk_msg_reset_curr(struct sk_msg *msg) · 2653 unmodified lines | ||
| 2654 | 2654 | } |
| 2655 | 2655 | } |
| 2656 | 2656 | |
| 2657 | +static bool sk_msg_elem_is_copy(const struct sk_msg *msg, u32 i) | |
| 2658 | +{ | |
| 2659 | + return test_bit(i, msg->sg.copy); | |
| 2660 | +} | |
| 2661 | + | |
| 2662 | +static void sk_msg_clear_elem_copy(struct sk_msg *msg, u32 i) | |
| 2663 | +{ | |
| 2664 | + __clear_bit(i, msg->sg.copy); | |
| 2665 | +} | |
| 2666 | + | |
| 2667 | +static void sk_msg_set_elem_copy(struct sk_msg *msg, u32 i, bool sg_copy) | |
| 2668 | +{ | |
| 2669 | + __assign_bit(i, msg->sg.copy, sg_copy); | |
| 2670 | +} | |
| 2671 | + | |
| 2672 | +static void sk_msg_clear_copy_range(struct sk_msg *msg, u32 start, u32 end) | |
| 2673 | +{ | |
| 2674 | + while (start != end) { | |
| 2675 | + sk_msg_clear_elem_copy(msg, start); | |
| 2676 | + sk_msg_iter_var_next(start); | |
| 2677 | + } | |
| 2678 | +} | |
| 2679 | + | |
| 2680 | +static void sk_msg_sg_move(struct sk_msg *msg, u32 dst, u32 src) | |
| 2681 | +{ | |
| 2682 | + msg->sg.data[dst] = msg->sg.data[src]; | |
| 2683 | + | |
| 2684 | + sk_msg_set_elem_copy(msg, dst, | |
| 2685 | + sk_msg_elem_is_copy(msg, src)); | |
| 2686 | +} | |
| 2687 | + | |
| 2657 | 2688 | static const struct bpf_func_proto bpf_msg_cork_bytes_proto = { |
| 2658 | 2689 | .func = bpf_msg_cork_bytes, |
| 2659 | 2690 | .gpl_only = false, |
| @@ -2692,7 +2723,7 @@ BPF_CALL_4(bpf_msg_pull_data, struct sk_msg *, msg, u32, start, · 32 unmodified lines | ||
| 2692 | 2723 | * account for the headroom. |
| 2693 | 2724 | */ |
| 2694 | 2725 | bytes_sg_total = start - offset + bytes; |
| 2695 | − if (!test_bit(i, msg->sg.copy) && bytes_sg_total <= len) | |
| 2726 | + if (!sk_msg_elem_is_copy(msg, i) && bytes_sg_total <= len) | |
| 2696 | 2727 | goto out; |
| 2697 | 2728 | |
| 2698 | 2729 | /* At this point we need to linearize multiple scatterlist |
| @@ -2733,13 +2764,13 @@ BPF_CALL_4(bpf_msg_pull_data, struct sk_msg *, msg, u32, start, · 34 unmodified lines | ||
| 2733 | 2764 | poffset += len; |
| 2734 | 2765 | sge->length = 0; |
| 2735 | 2766 | put_page(sg_page(sge)); |
| 2736 | − __clear_bit(i, msg->sg.copy); | |
| 2767 | + sk_msg_clear_elem_copy(msg, i); | |
| 2737 | 2768 | |
| 2738 | 2769 | sk_msg_iter_var_next(i); |
| 2739 | 2770 | } while (i != last_sge); |
| 2740 | 2771 | |
| 2741 | 2772 | sg_set_page(&msg->sg.data[first_sge], page, copy, 0); |
| 2742 | − __clear_bit(first_sge, msg->sg.copy); | |
| 2773 | + sk_msg_clear_elem_copy(msg, first_sge); | |
| 2743 | 2774 | |
| 2744 | 2775 | /* To repair sg ring we need to shift entries. If we only |
| 2745 | 2776 | * had a single entry though we can just replace it and |
| @@ -2749,8 +2780,14 @@ BPF_CALL_4(bpf_msg_pull_data, struct sk_msg *, msg, u32, start, · 3 unmodified lines | ||
| 2749 | 2780 | shift = last_sge > first_sge ? |
| 2750 | 2781 | last_sge - first_sge - 1 : |
| 2751 | 2782 | NR_MSG_FRAG_IDS - first_sge + last_sge - 1; |
| 2752 | − if (!shift) | |
| 2783 | + if (!shift) { | |
| 2784 | + sk_msg_clear_elem_copy(msg, msg->sg.end); | |
| 2753 | 2785 | goto out; |
| 2786 | + } | |
| 2787 | + | |
| 2788 | + i = first_sge; | |
| 2789 | + sk_msg_iter_var_next(i); | |
| 2790 | + sk_msg_clear_copy_range(msg, i, last_sge); | |
| 2754 | 2791 | |
| 2755 | 2792 | i = first_sge; |
| 2756 | 2793 | sk_msg_iter_var_next(i); |
| @@ -2764,18 +2801,18 @@ BPF_CALL_4(bpf_msg_pull_data, struct sk_msg *, msg, u32, start, · 7 unmodified lines | ||
| 2764 | 2801 | if (move_from == msg->sg.end) |
| 2765 | 2802 | break; |
| 2766 | 2803 | |
| 2767 | − msg->sg.data[i] = msg->sg.data[move_from]; | |
| 2768 | − sk_msg_sg_copy_assign(msg, i, msg, move_from); | |
| 2804 | + sk_msg_sg_move(msg, i, move_from); | |
| 2769 | 2805 | msg->sg.data[move_from].length = 0; |
| 2770 | 2806 | msg->sg.data[move_from].page_link = 0; |
| 2771 | 2807 | msg->sg.data[move_from].offset = 0; |
| 2772 | − __clear_bit(move_from, msg->sg.copy); | |
| 2808 | + sk_msg_clear_elem_copy(msg, move_from); | |
| 2773 | 2809 | sk_msg_iter_var_next(i); |
| 2774 | 2810 | } while (1); |
| 2775 | 2811 | |
| 2776 | 2812 | msg->sg.end = msg->sg.end - shift > msg->sg.end ? |
| 2777 | 2813 | msg->sg.end - shift + NR_MSG_FRAG_IDS : |
| 2778 | 2814 | msg->sg.end - shift; |
| 2815 | + sk_msg_clear_elem_copy(msg, msg->sg.end); | |
| 2779 | 2816 | out: |
| 2780 | 2817 | sk_msg_reset_curr(msg); |
| 2781 | 2818 | msg->data = sg_virt(&msg->sg.data[first_sge]) + start - offset; |
| @@ -2796,9 +2833,10 @@ static const struct bpf_func_proto bpf_msg_pull_data_proto = { · 14 unmodified lines | ||
| 2796 | 2833 | BPF_CALL_4(bpf_msg_push_data, struct sk_msg *, msg, u32, start, |
| 2797 | 2834 | u32, len, u64, flags) |
| 2798 | 2835 | { |
| 2836 | + bool sge_copy = false, nsge_copy = false, nnsge_copy = false; | |
| 2799 | 2837 | struct scatterlist sge, nsge, nnsge, rsge = {0}, *psge; |
| 2800 | 2838 | u32 new, i = 0, l = 0, space, copy = 0, offset = 0; |
| 2801 | − bool sge_copy, nsge_copy, nnsge_copy, rsge_copy = false; | |
| 2839 | + bool rsge_copy = false; | |
| 2802 | 2840 | u8 *raw, *to, *from; |
| 2803 | 2841 | struct page *page; |
| 2804 | 2842 | |
| @@ -2834,6 +2872,9 @@ BPF_CALL_4(bpf_msg_push_data, struct sk_msg *, msg, u32, start, · 29 unmodified lines | ||
| 2834 | 2872 | if (!space || (space == 1 && start != offset)) |
| 2835 | 2873 | copy = msg->sg.data[i].length; |
| 2836 | 2874 | |
| 2875 | + if (unlikely(copy + len < copy)) | |
| 2876 | + return -EINVAL; | |
| 2877 | + | |
| 2837 | 2878 | page = alloc_pages(__GFP_NOWARN | GFP_ATOMIC | __GFP_COMP, |
| 2838 | 2879 | get_order(copy + len)); |
| 2839 | 2880 | if (unlikely(!page)) |
| @@ -2871,7 +2912,7 @@ BPF_CALL_4(bpf_msg_push_data, struct sk_msg *, msg, u32, start, · 31 unmodified lines | ||
| 2871 | 2912 | sk_msg_iter_var_prev(i); |
| 2872 | 2913 | psge = sk_msg_elem(msg, i); |
| 2873 | 2914 | rsge = sk_msg_elem_cpy(msg, i); |
| 2874 | − rsge_copy = test_bit(i, msg->sg.copy); | |
| 2915 | + rsge_copy = sk_msg_elem_is_copy(msg, i); | |
| 2875 | 2916 | |
| 2876 | 2917 | psge->length = start - offset; |
| 2877 | 2918 | rsge.length -= psge->length; |
| @@ -2896,21 +2937,21 @@ BPF_CALL_4(bpf_msg_push_data, struct sk_msg *, msg, u32, start, · 18 unmodified lines | ||
| 2896 | 2937 | |
| 2897 | 2938 | /* Shift one or two slots as needed */ |
| 2898 | 2939 | sge = sk_msg_elem_cpy(msg, new); |
| 2899 | − sge_copy = test_bit(new, msg->sg.copy); | |
| 2900 | 2940 | sg_unmark_end(&sge); |
| 2941 | + sge_copy = sk_msg_elem_is_copy(msg, new); | |
| 2901 | 2942 | |
| 2902 | 2943 | nsge = sk_msg_elem_cpy(msg, i); |
| 2903 | − nsge_copy = test_bit(i, msg->sg.copy); | |
| 2944 | + nsge_copy = sk_msg_elem_is_copy(msg, i); | |
| 2904 | 2945 | if (rsge.length) { |
| 2905 | 2946 | sk_msg_iter_var_next(i); |
| 2906 | 2947 | nnsge = sk_msg_elem_cpy(msg, i); |
| 2907 | − nnsge_copy = test_bit(i, msg->sg.copy); | |
| 2948 | + nnsge_copy = sk_msg_elem_is_copy(msg, i); | |
| 2908 | 2949 | sk_msg_iter_next(msg, end); |
| 2909 | 2950 | } |
| 2910 | 2951 | |
| 2911 | 2952 | while (i != msg->sg.end) { |
| 2912 | 2953 | msg->sg.data[i] = sge; |
| 2913 | − __assign_bit(i, msg->sg.copy, sge_copy); | |
| 2954 | + sk_msg_set_elem_copy(msg, i, sge_copy); | |
| 2914 | 2955 | sge = nsge; |
| 2915 | 2956 | sge_copy = nsge_copy; |
| 2916 | 2957 | sk_msg_iter_var_next(i); |
| @@ -2918,10 +2959,10 @@ BPF_CALL_4(bpf_msg_push_data, struct sk_msg *, msg, u32, start, · 1 unmodified line | ||
| 2918 | 2959 | nsge = nnsge; |
| 2919 | 2960 | nsge_copy = nnsge_copy; |
| 2920 | 2961 | nnsge = sk_msg_elem_cpy(msg, i); |
| 2921 | − nnsge_copy = test_bit(i, msg->sg.copy); | |
| 2962 | + nnsge_copy = sk_msg_elem_is_copy(msg, i); | |
| 2922 | 2963 | } else { |
| 2923 | 2964 | nsge = sk_msg_elem_cpy(msg, i); |
| 2924 | − nsge_copy = test_bit(i, msg->sg.copy); | |
| 2965 | + nsge_copy = sk_msg_elem_is_copy(msg, i); | |
| 2925 | 2966 | } |
| 2926 | 2967 | } |
| 2927 | 2968 | |
| @@ -2929,14 +2970,15 @@ place_new: · 1 unmodified line | ||
| 2929 | 2970 | /* Place newly allocated data buffer */ |
| 2930 | 2971 | sk_mem_charge(msg->sk, len); |
| 2931 | 2972 | msg->sg.size += len; |
| 2932 | − __clear_bit(new, msg->sg.copy); | |
| 2973 | + sk_msg_clear_elem_copy(msg, new); | |
| 2933 | 2974 | sg_set_page(&msg->sg.data[new], page, len + copy, 0); |
| 2934 | 2975 | if (rsge.length) { |
| 2935 | 2976 | get_page(sg_page(&rsge)); |
| 2936 | 2977 | sk_msg_iter_var_next(new); |
| 2937 | 2978 | msg->sg.data[new] = rsge; |
| 2938 | − __assign_bit(new, msg->sg.copy, rsge_copy); | |
| 2979 | + sk_msg_set_elem_copy(msg, new, rsge_copy); | |
| 2939 | 2980 | } |
| 2981 | + sk_msg_clear_elem_copy(msg, msg->sg.end); | |
| 2940 | 2982 | |
| 2941 | 2983 | sk_msg_reset_curr(msg); |
| 2942 | 2984 | sk_msg_compute_data_pointers(msg); |
| @@ -2962,12 +3004,11 @@ static void sk_msg_shift_left(struct sk_msg *msg, int i) · 19 unmodified lines | ||
| 2962 | 3004 | do { |
| 2963 | 3005 | prev = i; |
| 2964 | 3006 | sk_msg_iter_var_next(i); |
| 2965 | − msg->sg.data[prev] = msg->sg.data[i]; | |
| 2966 | − sk_msg_sg_copy_assign(msg, prev, msg, i); | |
| 3007 | + sk_msg_sg_move(msg, prev, i); | |
| 2967 | 3008 | } while (i != msg->sg.end); |
| 2968 | 3009 | |
| 2969 | 3010 | sk_msg_iter_prev(msg, end); |
| 2970 | − __clear_bit(msg->sg.end, msg->sg.copy); | |
| 3011 | + sk_msg_clear_elem_copy(msg, msg->sg.end); | |
| 2971 | 3012 | } |
| 2972 | 3013 | |
| 2973 | 3014 | static void sk_msg_shift_right(struct sk_msg *msg, int i) |
| @@ -2977,28 +3018,29 @@ static void sk_msg_shift_right(struct sk_msg *msg, int i) · 3 unmodified lines | ||
| 2977 | 3018 | |
| 2978 | 3019 | sk_msg_iter_next(msg, end); |
| 2979 | 3020 | sge = sk_msg_elem_cpy(msg, i); |
| 2980 | − sge_copy = test_bit(i, msg->sg.copy); | |
| 3021 | + sge_copy = sk_msg_elem_is_copy(msg, i); | |
| 2981 | 3022 | sk_msg_iter_var_next(i); |
| 2982 | 3023 | tmp = sk_msg_elem_cpy(msg, i); |
| 2983 | − tmp_copy = test_bit(i, msg->sg.copy); | |
| 3024 | + tmp_copy = sk_msg_elem_is_copy(msg, i); | |
| 2984 | 3025 | |
| 2985 | 3026 | while (i != msg->sg.end) { |
| 2986 | 3027 | msg->sg.data[i] = sge; |
| 2987 | − __assign_bit(i, msg->sg.copy, sge_copy); | |
| 3028 | + sk_msg_set_elem_copy(msg, i, sge_copy); | |
| 2988 | 3029 | sk_msg_iter_var_next(i); |
| 2989 | 3030 | sge = tmp; |
| 2990 | 3031 | sge_copy = tmp_copy; |
| 2991 | 3032 | tmp = sk_msg_elem_cpy(msg, i); |
| 2992 | − tmp_copy = test_bit(i, msg->sg.copy); | |
| 3033 | + tmp_copy = sk_msg_elem_is_copy(msg, i); | |
| 2993 | 3034 | } |
| 3035 | + sk_msg_clear_elem_copy(msg, msg->sg.end); | |
| 2994 | 3036 | } |
| 2995 | 3037 | |
| 2996 | 3038 | BPF_CALL_4(bpf_msg_pop_data, struct sk_msg *, msg, u32, start, |
| 2997 | 3039 | u32, len, u64, flags) |
| 2998 | 3040 | { |
| 2999 | 3041 | u32 i = 0, l = 0, space, offset = 0; |
| 3000 | − u64 last = start + len; | |
| 3001 | − int pop; | |
| 3042 | + u64 last = (u64)start + len; | |
| 3043 | + u32 pop; | |
| 3002 | 3044 | |
| 3003 | 3045 | if (unlikely(flags)) |
| 3004 | 3046 | return -EINVAL; |
| @@ -3047,10 +3089,10 @@ BPF_CALL_4(bpf_msg_pop_data, struct sk_msg *, msg, u32, start, · 42 unmodified lines | ||
| 3047 | 3089 | */ |
| 3048 | 3090 | if (start != offset) { |
| 3049 | 3091 | struct scatterlist *nsge, *sge = sk_msg_elem(msg, i); |
| 3092 | + bool sge_copy = sk_msg_elem_is_copy(msg, i); | |
| 3050 | 3093 | int a = start - offset; |
| 3051 | 3094 | int b = sge->length - pop - a; |
| 3052 | − u32 sge_i = i; | |
| 3053 | − bool sge_copy = test_bit(i, msg->sg.copy); | |
| 3095 | + u32 sge_idx = i; | |
| 3054 | 3096 | |
| 3055 | 3097 | sk_msg_iter_var_next(i); |
| 3056 | 3098 | |
| @@ -3063,7 +3105,7 @@ BPF_CALL_4(bpf_msg_pop_data, struct sk_msg *, msg, u32, start, · 6 unmodified lines | ||
| 3063 | 3105 | sg_set_page(nsge, |
| 3064 | 3106 | sg_page(sge), |
| 3065 | 3107 | b, sge->offset + pop + a); |
| 3066 | − __assign_bit(i, msg->sg.copy, sge_copy); | |
| 3108 | + sk_msg_set_elem_copy(msg, i, sge_copy); | |
| 3067 | 3109 | } else { |
| 3068 | 3110 | struct page *page, *orig; |
| 3069 | 3111 | u8 *to, *from; |
| @@ -3080,7 +3122,7 @@ BPF_CALL_4(bpf_msg_pop_data, struct sk_msg *, msg, u32, start, · 10 unmodified lines | ||
| 3080 | 3122 | memcpy(to, from, a); |
| 3081 | 3123 | memcpy(to + a, from + a + pop, b); |
| 3082 | 3124 | sg_set_page(sge, page, a + b, 0); |
| 3083 | − __clear_bit(sge_i, msg->sg.copy); | |
| 3125 | + sk_msg_clear_elem_copy(msg, sge_idx); | |
| 3084 | 3126 | put_page(orig); |
| 3085 | 3127 | } |
| 3086 | 3128 | pop = 0; |
| @@ -5571,11 +5613,24 @@ static int sol_tcp_sockopt(struct sock *sk, int optname, · 2484 unmodified lines | ||
| 5571 | 5613 | KERNEL_SOCKPTR(optval), *optlen); |
| 5572 | 5614 | } |
| 5573 | 5615 | |
| 5616 | +static bool sk_allows_sol_ip_sockopt(struct sock *sk) | |
| 5617 | +{ | |
| 5618 | + switch (sk->sk_family) { | |
| 5619 | + case AF_INET: | |
| 5620 | + return true; | |
| 5621 | + case AF_INET6: | |
| 5622 | + /* Allow getting/setting sockopt for possible ipv4-mapped ipv6 socket. */ | |
| 5623 | + return sk->sk_type != SOCK_RAW && !ipv6_only_sock(sk); | |
| 5624 | + default: | |
| 5625 | + return false; | |
| 5626 | + } | |
| 5627 | +} | |
| 5628 | + | |
| 5574 | 5629 | static int sol_ip_sockopt(struct sock *sk, int optname, |
| 5575 | 5630 | char *optval, int *optlen, |
| 5576 | 5631 | bool getopt) |
| 5577 | 5632 | { |
| 5578 | − if (sk->sk_family != AF_INET) | |
| 5633 | + if (!sk_allows_sol_ip_sockopt(sk)) | |
| 5579 | 5634 | return -EINVAL; |
| 5580 | 5635 | |
| 5581 | 5636 | switch (optname) { |
| @@ -599,6 +599,7 @@ static int handle_gso_encap(struct sk_buff *skb, bool ipv4, int encap_len) · 598 unmodified lines | ||
| 599 | 599 | |
| 600 | 600 | int bpf_lwt_push_ip_encap(struct sk_buff *skb, void *hdr, u32 len, bool ingress) |
| 601 | 601 | { |
| 602 | + bool is_udp_tunnel; | |
| 602 | 603 | struct iphdr *iph; |
| 603 | 604 | bool ipv4; |
| 604 | 605 | int err; |
| @@ -612,10 +613,16 @@ int bpf_lwt_push_ip_encap(struct sk_buff *skb, void *hdr, u32 len, bool ingress) · 7 unmodified lines | ||
| 612 | 613 | ipv4 = true; |
| 613 | 614 | if (unlikely(len < iph->ihl * 4)) |
| 614 | 615 | return -EINVAL; |
| 616 | + is_udp_tunnel = iph->protocol == IPPROTO_UDP; | |
| 617 | + if (unlikely(is_udp_tunnel && len < iph->ihl * 4 + sizeof(struct udphdr))) | |
| 618 | + return -EINVAL; | |
| 615 | 619 | } else if (iph->version == 6) { |
| 616 | 620 | ipv4 = false; |
| 617 | 621 | if (unlikely(len < sizeof(struct ipv6hdr))) |
| 618 | 622 | return -EINVAL; |
| 623 | + is_udp_tunnel = ((struct ipv6hdr *)iph)->nexthdr == NEXTHDR_UDP; | |
| 624 | + if (unlikely(is_udp_tunnel && len < sizeof(struct ipv6hdr) + sizeof(struct udphdr))) | |
| 625 | + return -EINVAL; | |
| 619 | 626 | } else { |
| 620 | 627 | return -EINVAL; |
| 621 | 628 | } |
| @@ -637,6 +644,11 @@ int bpf_lwt_push_ip_encap(struct sk_buff *skb, void *hdr, u32 len, bool ingress) · 15 unmodified lines | ||
| 637 | 644 | if (ingress) |
| 638 | 645 | skb_postpush_rcsum(skb, iph, len); |
| 639 | 646 | skb_reset_network_header(skb); |
| 647 | + if (is_udp_tunnel) { | |
| 648 | + size_t iph_sz = ipv4 ? iph->ihl * 4 : sizeof(struct ipv6hdr); | |
| 649 | + | |
| 650 | + skb_set_transport_header(skb, skb_network_offset(skb) + iph_sz); | |
| 651 | + } | |
| 640 | 652 | memcpy(skb_network_header(skb), hdr, len); |
| 641 | 653 | bpf_compute_data_pointers(skb); |
| 642 | 654 | skb_clear_hash(skb); |
| @@ -50,7 +50,9 @@ static int udp_msg_wait_data(struct sock *sk, struct sk_psock *psock, · 49 unmodified lines | ||
| 50 | 50 | sk_set_bit(SOCKWQ_ASYNC_WAITDATA, sk); |
| 51 | 51 | ret = udp_msg_has_data(sk, psock); |
| 52 | 52 | if (!ret) { |
| 53 | + release_sock(sk); | |
| 53 | 54 | wait_woken(&wait, TASK_INTERRUPTIBLE, timeo); |
| 55 | + lock_sock(sk); | |
| 54 | 56 | ret = udp_msg_has_data(sk, psock); |
| 55 | 57 | } |
| 56 | 58 | sk_clear_bit(SOCKWQ_ASYNC_WAITDATA, sk); |
| @@ -79,6 +81,7 @@ static int udp_bpf_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, · 22 unmodified lines | ||
| 79 | 81 | goto out; |
| 80 | 82 | } |
| 81 | 83 | |
| 84 | + lock_sock(sk); | |
| 82 | 85 | msg_bytes_ready: |
| 83 | 86 | copied = sk_msg_recvmsg(sk, psock, msg, len, flags); |
| 84 | 87 | if (!copied) { |
| @@ -90,11 +93,17 @@ msg_bytes_ready: · 5 unmodified lines | ||
| 90 | 93 | if (data) { |
| 91 | 94 | if (psock_has_data(psock)) |
| 92 | 95 | goto msg_bytes_ready; |
| 96 | + | |
| 97 | + release_sock(sk); | |
| 98 | + | |
| 93 | 99 | ret = sk_udp_recvmsg(sk, msg, len, flags); |
| 94 | 100 | goto out; |
| 95 | 101 | } |
| 96 | 102 | copied = -EAGAIN; |
| 97 | 103 | } |
| 104 | + | |
| 105 | + release_sock(sk); | |
| 106 | + | |
| 98 | 107 | ret = copied; |
| 99 | 108 | out: |
| 100 | 109 | sk_psock_put(sk, psock); |
| @@ -11,6 +11,7 @@ INSTALL ?= install · 10 unmodified lines | ||
| 11 | 11 | CFLAGS += -Wall -O2 |
| 12 | 12 | CFLAGS += -D__EXPORTED_HEADERS__ -I$(srctree)/tools/include/uapi \ |
| 13 | 13 | -I$(srctree)/tools/include |
| 14 | +CFLAGS += $(EXTRA_CFLAGS) | |
| 14 | 15 | |
| 15 | 16 | # This will work when bpf is built in tools env. where srctree |
| 16 | 17 | # isn't set and when invoked from selftests build, where srctree |
| @@ -56,7 +56,7 @@ MAP COMMANDS · 55 unmodified lines | ||
| 56 | 56 | | | **cgroup_storage** | **reuseport_sockarray** | **percpu_cgroup_storage** |
| 57 | 57 | | | **queue** | **stack** | **sk_storage** | **struct_ops** | **ringbuf** | **inode_storage** |
| 58 | 58 | | | **task_storage** | **bloom_filter** | **user_ringbuf** | **cgrp_storage** | **arena** |
| 59 | −| | **insn_array** } | |
| 59 | +| | **insn_array** | **rhash** } | |
| 60 | 60 | |
| 61 | 61 | DESCRIPTION |
| 62 | 62 | =========== |
| @@ -47,7 +47,8 @@ $(LIBBPF_INTERNAL_HDRS): $(LIBBPF_HDRS_DIR)/%.h: $(BPF_DIR)/%.h | $(LIBBPF_HDRS_ · 46 unmodified lines | ||
| 47 | 47 | $(LIBBPF_BOOTSTRAP): $(wildcard $(BPF_DIR)/*.[ch] $(BPF_DIR)/Makefile) | $(LIBBPF_BOOTSTRAP_OUTPUT) |
| 48 | 48 | $(Q)$(MAKE) -C $(BPF_DIR) OUTPUT=$(LIBBPF_BOOTSTRAP_OUTPUT) \ |
| 49 | 49 | DESTDIR=$(LIBBPF_BOOTSTRAP_DESTDIR:/=) prefix= \ |
| 50 | − ARCH= CROSS_COMPILE= CC="$(HOSTCC)" LD="$(HOSTLD)" AR="$(HOSTAR)" $@ install_headers | |
| 50 | + ARCH= CROSS_COMPILE= CC="$(HOSTCC)" LD="$(HOSTLD)" AR="$(HOSTAR)" \ | |
| 51 | + CFLAGS="$(LIBBPF_BOOTSTRAP_CFLAGS)" EXTRA_CFLAGS= $@ install_headers | |
| 51 | 52 | |
| 52 | 53 | $(LIBBPF_BOOTSTRAP_INTERNAL_HDRS): $(LIBBPF_BOOTSTRAP_HDRS_DIR)/%.h: $(BPF_DIR)/%.h | $(LIBBPF_BOOTSTRAP_HDRS_DIR) |
| 53 | 54 | $(call QUIET_INSTALL, $@) |
| @@ -81,6 +82,13 @@ CFLAGS += -DPACKAGE='"bpftool"' -D__EXPORTED_HEADERS__ \ · 27 unmodified lines | ||
| 81 | 82 | ifneq ($(BPFTOOL_VERSION),) |
| 82 | 83 | CFLAGS += -DBPFTOOL_VERSION='"$(BPFTOOL_VERSION)"' |
| 83 | 84 | endif |
| 85 | + | |
| 86 | +# This must be done before appending EXTRA_CFLAGS to CFLAGS to avoid | |
| 87 | +# including flags that are not applicable to the host compiler. | |
| 88 | +HOST_CFLAGS := $(subst -I$(LIBBPF_INCLUDE),-I$(LIBBPF_BOOTSTRAP_INCLUDE),\ | |
| 89 | + $(subst $(CLANG_CROSS_FLAGS),,$(CFLAGS))) | |
| 90 | +HOST_CFLAGS += $(HOST_EXTRACFLAGS) | |
| 91 | + | |
| 84 | 92 | ifneq ($(EXTRA_CFLAGS),) |
| 85 | 93 | CFLAGS += $(EXTRA_CFLAGS) |
| 86 | 94 | endif |
| @@ -88,10 +96,11 @@ ifneq ($(EXTRA_LDFLAGS),) · 1 unmodified line | ||
| 88 | 96 | LDFLAGS += $(EXTRA_LDFLAGS) |
| 89 | 97 | endif |
| 90 | 98 | |
| 91 | −HOST_CFLAGS := $(subst -I$(LIBBPF_INCLUDE),-I$(LIBBPF_BOOTSTRAP_INCLUDE),\ | |
| 92 | − $(subst $(CLANG_CROSS_FLAGS),,$(CFLAGS))) | |
| 93 | 99 | HOST_LDFLAGS := $(LDFLAGS) |
| 94 | 100 | |
| 101 | +# Remove warnings for libbpf bootstrap build | |
| 102 | +LIBBPF_BOOTSTRAP_CFLAGS := $(filter-out -W -Wall -Wextra -Wformat -Wformat-signedness,$(HOST_CFLAGS)) | |
| 103 | + | |
| 95 | 104 | INSTALL ?= install |
| 96 | 105 | RM ?= rm -f |
| 97 | 106 | |
| @@ -106,6 +115,10 @@ ifneq ($(SKIP_CRYPTO),1) · 8 unmodified lines | ||
| 106 | 115 | CRYPTO_LIBS := -lcrypto |
| 107 | 116 | endif |
| 108 | 117 | |
| 118 | +ifeq ($(MAKECMDGOALS),bootstrap) | |
| 119 | +FEATURE_TESTS := libelf-zstd | |
| 120 | +FEATURE_DISPLAY := | |
| 121 | +else | |
| 109 | 122 | FEATURE_TESTS := clang-bpf-co-re |
| 110 | 123 | FEATURE_TESTS += llvm |
| 111 | 124 | FEATURE_TESTS += libcap |
| @@ -122,6 +135,7 @@ FEATURE_DISPLAY += libcap · 10 unmodified lines | ||
| 122 | 135 | FEATURE_DISPLAY += libbfd |
| 123 | 136 | FEATURE_DISPLAY += libbfd-liberty |
| 124 | 137 | FEATURE_DISPLAY += libbfd-liberty-z |
| 138 | +endif | |
| 125 | 139 | |
| 126 | 140 | check_feat := 1 |
| 127 | 141 | NON_CHECK_FEAT_TARGETS := clean uninstall doc doc-clean doc-install doc-uninstall |
| @@ -179,8 +179,7 @@ static int dump_btf_type(const struct btf *btf, __u32 id, · 178 unmodified lines | ||
| 179 | 179 | case BTF_KIND_STRUCT: |
| 180 | 180 | case BTF_KIND_UNION: { |
| 181 | 181 | const struct btf_member *m = (const void *)(t + 1); |
| 182 | − __u16 vlen = BTF_INFO_VLEN(t->info); | |
| 183 | − int i; | |
| 182 | + __u32 i, vlen = BTF_INFO_VLEN(t->info); | |
| 184 | 183 | |
| 185 | 184 | if (json_output) { |
| 186 | 185 | jsonw_uint_field(w, "size", t->size); |
| @@ -225,9 +224,8 @@ static int dump_btf_type(const struct btf *btf, __u32 id, · 38 unmodified lines | ||
| 225 | 224 | } |
| 226 | 225 | case BTF_KIND_ENUM: { |
| 227 | 226 | const struct btf_enum *v = (const void *)(t + 1); |
| 228 | − __u16 vlen = BTF_INFO_VLEN(t->info); | |
| 227 | + __u32 i, vlen = BTF_INFO_VLEN(t->info); | |
| 229 | 228 | const char *encoding; |
| 230 | − int i; | |
| 231 | 229 | |
| 232 | 230 | encoding = btf_kflag(t) ? "SIGNED" : "UNSIGNED"; |
| 233 | 231 | if (json_output) { |
| @@ -263,9 +261,8 @@ static int dump_btf_type(const struct btf *btf, __u32 id, · 29 unmodified lines | ||
| 263 | 261 | } |
| 264 | 262 | case BTF_KIND_ENUM64: { |
| 265 | 263 | const struct btf_enum64 *v = btf_enum64(t); |
| 266 | − __u16 vlen = btf_vlen(t); | |
| 264 | + __u32 i, vlen = btf_vlen(t); | |
| 267 | 265 | const char *encoding; |
| 268 | − int i; | |
| 269 | 266 | |
| 270 | 267 | encoding = btf_kflag(t) ? "SIGNED" : "UNSIGNED"; |
| 271 | 268 | if (json_output) { |
| @@ -325,8 +322,7 @@ static int dump_btf_type(const struct btf *btf, __u32 id, · 53 unmodified lines | ||
| 325 | 322 | } |
| 326 | 323 | case BTF_KIND_FUNC_PROTO: { |
| 327 | 324 | const struct btf_param *p = (const void *)(t + 1); |
| 328 | − __u16 vlen = BTF_INFO_VLEN(t->info); | |
| 329 | − int i; | |
| 325 | + __u32 i, vlen = BTF_INFO_VLEN(t->info); | |
| 330 | 326 | |
| 331 | 327 | if (json_output) { |
| 332 | 328 | jsonw_uint_field(w, "ret_type_id", t->type); |
| @@ -369,8 +365,7 @@ static int dump_btf_type(const struct btf *btf, __u32 id, · 36 unmodified lines | ||
| 369 | 365 | case BTF_KIND_DATASEC: { |
| 370 | 366 | const struct btf_var_secinfo *v = (const void *)(t + 1); |
| 371 | 367 | const struct btf_type *vt; |
| 372 | − __u16 vlen = BTF_INFO_VLEN(t->info); | |
| 373 | − int i; | |
| 368 | + __u32 i, vlen = BTF_INFO_VLEN(t->info); | |
| 374 | 369 | |
| 375 | 370 | if (json_output) { |
| 376 | 371 | jsonw_uint_field(w, "size", t->size); |
| @@ -675,7 +670,7 @@ static __u64 btf_name_hasher(__u64 hash, const struct btf *btf, __u32 name_off) · 298 unmodified lines | ||
| 675 | 670 | static __u64 btf_type_disambig_hash(const struct btf *btf, __u32 id, bool include_members) |
| 676 | 671 | { |
| 677 | 672 | const struct btf_type *t = btf__type_by_id(btf, id); |
| 678 | − int i; | |
| 673 | + __u32 i; | |
| 679 | 674 | size_t hash = 0; |
| 680 | 675 | |
| 681 | 676 | hash = btf_name_hasher(hash, btf, t->name_off); |
| @@ -150,7 +150,7 @@ static int btf_dumper_enum(const struct btf_dumper *d, · 149 unmodified lines | ||
| 150 | 150 | { |
| 151 | 151 | const struct btf_enum *enums = btf_enum(t); |
| 152 | 152 | __s64 value; |
| 153 | − __u16 i; | |
| 153 | + __u32 i; | |
| 154 | 154 | |
| 155 | 155 | switch (t->size) { |
| 156 | 156 | case 8: |
| @@ -189,7 +189,7 @@ static int btf_dumper_enum64(const struct btf_dumper *d, · 32 unmodified lines | ||
| 189 | 189 | const struct btf_enum64 *enums = btf_enum64(t); |
| 190 | 190 | __u32 val_lo32, val_hi32; |
| 191 | 191 | __u64 value; |
| 192 | − __u16 i; | |
| 192 | + __u32 i; | |
| 193 | 193 | |
| 194 | 194 | value = *(__u64 *)data; |
| 195 | 195 | val_lo32 = (__u32)value; |
| @@ -1399,7 +1399,7 @@ static int do_skeleton(int argc, char **argv) · 1398 unmodified lines | ||
| 1399 | 1399 | continue; |
| 1400 | 1400 | |
| 1401 | 1401 | if (use_loader) |
| 1402 | − printf("t\tint %s_fd;\n", ident); | |
| 1402 | + printf("\t\tint %s_fd;\n", ident); | |
| 1403 | 1403 | else |
| 1404 | 1404 | printf("\t\tstruct bpf_link *%s;\n", ident); |
| 1405 | 1405 | } |
| @@ -2094,7 +2094,8 @@ btfgen_mark_type(struct btfgen_info *info, unsigned int type_id, bool follow_poi · 688 unmodified lines | ||
| 2094 | 2094 | struct btf_type *cloned_type; |
| 2095 | 2095 | struct btf_param *param; |
| 2096 | 2096 | struct btf_array *array; |
| 2097 | − int err, i; | |
| 2097 | + __u32 i; | |
| 2098 | + int err; | |
| 2098 | 2099 | |
| 2099 | 2100 | if (type_id == 0) |
| 2100 | 2101 | return 0; |
| @@ -2229,7 +2230,8 @@ static int btfgen_mark_type_match(struct btfgen_info *info, __u32 type_id, bool · 128 unmodified lines | ||
| 2229 | 2230 | const struct btf_type *btf_type; |
| 2230 | 2231 | struct btf *btf = info->src_btf; |
| 2231 | 2232 | struct btf_type *cloned_type; |
| 2232 | − int i, err; | |
| 2233 | + int err; | |
| 2234 | + __u32 i; | |
| 2233 | 2235 | |
| 2234 | 2236 | if (type_id == 0) |
| 2235 | 2237 | return 0; |
| @@ -2249,7 +2251,7 @@ static int btfgen_mark_type_match(struct btfgen_info *info, __u32 type_id, bool · 13 unmodified lines | ||
| 2249 | 2251 | case BTF_KIND_STRUCT: |
| 2250 | 2252 | case BTF_KIND_UNION: { |
| 2251 | 2253 | struct btf_member *m = btf_members(btf_type); |
| 2252 | − __u16 vlen = btf_vlen(btf_type); | |
| 2254 | + __u32 vlen = btf_vlen(btf_type); | |
| 2253 | 2255 | |
| 2254 | 2256 | if (behind_ptr) |
| 2255 | 2257 | break; |
| @@ -2286,7 +2288,7 @@ static int btfgen_mark_type_match(struct btfgen_info *info, __u32 type_id, bool · 30 unmodified lines | ||
| 2286 | 2288 | break; |
| 2287 | 2289 | } |
| 2288 | 2290 | case BTF_KIND_FUNC_PROTO: { |
| 2289 | − __u16 vlen = btf_vlen(btf_type); | |
| 2291 | + __u32 vlen = btf_vlen(btf_type); | |
| 2290 | 2292 | struct btf_param *param; |
| 2291 | 2293 | |
| 2292 | 2294 | /* mark ret type */ |
| @@ -2492,8 +2494,9 @@ static struct btf *btfgen_get_btf(struct btfgen_info *info) · 199 unmodified lines | ||
| 2492 | 2494 | { |
| 2493 | 2495 | struct btf *btf_new = NULL; |
| 2494 | 2496 | unsigned int *ids = NULL; |
| 2495 | − unsigned int i, n = btf__type_cnt(info->marked_btf); | |
| 2497 | + unsigned int n = btf__type_cnt(info->marked_btf); | |
| 2496 | 2498 | int err = 0; |
| 2499 | + __u32 i; | |
| 2497 | 2500 | |
| 2498 | 2501 | btf_new = btf__new_empty(); |
| 2499 | 2502 | if (!btf_new) { |
| @@ -2523,8 +2526,7 @@ static struct btf *btfgen_get_btf(struct btfgen_info *info) · 23 unmodified lines | ||
| 2523 | 2526 | /* add members for struct and union */ |
| 2524 | 2527 | if (btf_is_composite(type)) { |
| 2525 | 2528 | struct btf_member *cloned_m, *m; |
| 2526 | − unsigned short vlen; | |
| 2527 | − int idx_src; | |
| 2529 | + __u32 vlen, idx_src; | |
| 2528 | 2530 | |
| 2529 | 2531 | name = btf__str_by_offset(info->src_btf, type->name_off); |
| 2530 | 2532 | |
| @@ -1478,7 +1478,7 @@ static int do_help(int argc, char **argv) · 1477 unmodified lines | ||
| 1478 | 1478 | " cgroup_storage | reuseport_sockarray | percpu_cgroup_storage |\n" |
| 1479 | 1479 | " queue | stack | sk_storage | struct_ops | ringbuf | inode_storage |\n" |
| 1480 | 1480 | " task_storage | bloom_filter | user_ringbuf | cgrp_storage | arena |\n" |
| 1481 | − " insn_array }\n" | |
| 1481 | + " insn_array | rhash }\n" | |
| 1482 | 1482 | " " HELP_SPEC_OPTIONS " |\n" |
| 1483 | 1483 | " {-f|--bpffs} | {-n|--nomount} }\n" |
| 1484 | 1484 | "", |
| @@ -603,14 +603,14 @@ static int query_flow_dissector(struct bpf_attach_info *attach_info) · 602 unmodified lines | ||
| 603 | 603 | &attach_flags, prog_ids, &prog_cnt); |
| 604 | 604 | close(fd); |
| 605 | 605 | if (err) { |
| 606 | − if (errno == EINVAL) { | |
| 606 | + if (err == -EINVAL) { | |
| 607 | 607 | /* Older kernel's don't support querying |
| 608 | 608 | * flow dissector programs. |
| 609 | 609 | */ |
| 610 | 610 | errno = 0; |
| 611 | 611 | return 0; |
| 612 | 612 | } |
| 613 | − p_err("can't query prog: %s", strerror(errno)); | |
| 613 | + p_err("can't query prog: %s", strerror(-err)); | |
| 614 | 614 | return -1; |
| 615 | 615 | } |
| 616 | 616 | |
| @@ -20,6 +20,8 @@ · 19 unmodified lines | ||
| 20 | 20 | # define __NR_bpf 6319 |
| 21 | 21 | # elif defined(__mips__) && defined(_ABI64) |
| 22 | 22 | # define __NR_bpf 5315 |
| 23 | +# elif defined(__loongarch__) | |
| 24 | +# define __NR_bpf 280 | |
| 23 | 25 | # else |
| 24 | 26 | # error __NR_bpf not defined. libbpf does not support your arch. |
| 25 | 27 | # endif |
| @@ -994,6 +994,7 @@ enum bpf_cmd { · 993 unmodified lines | ||
| 994 | 994 | BPF_PROG_STREAM_READ_BY_FD, |
| 995 | 995 | BPF_PROG_ASSOC_STRUCT_OPS, |
| 996 | 996 | __MAX_BPF_CMD, |
| 997 | + BPF_COMMON_ATTRS = 1 << 16, /* Indicate carrying syscall common attrs. */ | |
| 997 | 998 | }; |
| 998 | 999 | |
| 999 | 1000 | enum bpf_map_type { |
| @@ -1046,6 +1047,7 @@ enum bpf_map_type { · 46 unmodified lines | ||
| 1046 | 1047 | BPF_MAP_TYPE_CGRP_STORAGE, |
| 1047 | 1048 | BPF_MAP_TYPE_ARENA, |
| 1048 | 1049 | BPF_MAP_TYPE_INSN_ARRAY, |
| 1050 | + BPF_MAP_TYPE_RHASH, | |
| 1049 | 1051 | __MAX_BPF_MAP_TYPE |
| 1050 | 1052 | }; |
| 1051 | 1053 | |
| @@ -1154,6 +1156,9 @@ enum bpf_attach_type { · 102 unmodified lines | ||
| 1154 | 1156 | BPF_TRACE_KPROBE_SESSION, |
| 1155 | 1157 | BPF_TRACE_UPROBE_SESSION, |
| 1156 | 1158 | BPF_TRACE_FSESSION, |
| 1159 | + BPF_TRACE_FENTRY_MULTI, | |
| 1160 | + BPF_TRACE_FEXIT_MULTI, | |
| 1161 | + BPF_TRACE_FSESSION_MULTI, | |
| 1157 | 1162 | __MAX_BPF_ATTACH_TYPE |
| 1158 | 1163 | }; |
| 1159 | 1164 | |
| @@ -1178,6 +1183,7 @@ enum bpf_link_type { · 18 unmodified lines | ||
| 1178 | 1183 | BPF_LINK_TYPE_UPROBE_MULTI = 12, |
| 1179 | 1184 | BPF_LINK_TYPE_NETKIT = 13, |
| 1180 | 1185 | BPF_LINK_TYPE_SOCKMAP = 14, |
| 1186 | + BPF_LINK_TYPE_TRACING_MULTI = 15, | |
| 1181 | 1187 | __MAX_BPF_LINK_TYPE, |
| 1182 | 1188 | }; |
| 1183 | 1189 | |
| @@ -1321,7 +1327,11 @@ enum { · 137 unmodified lines | ||
| 1321 | 1327 | * BPF_TRACE_UPROBE_MULTI attach type to create return probe. |
| 1322 | 1328 | */ |
| 1323 | 1329 | enum { |
| 1324 | − BPF_F_UPROBE_MULTI_RETURN = (1U << 0) | |
| 1330 | + /* Get return uprobe. */ | |
| 1331 | + BPF_F_UPROBE_MULTI_RETURN = (1U << 0), | |
| 1332 | + | |
| 1333 | + /* Get path from provided path_fd. */ | |
| 1334 | + BPF_F_UPROBE_MULTI_PATH_FD = (1U << 1), | |
| 1325 | 1335 | }; |
| 1326 | 1336 | |
| 1327 | 1337 | /* link_create.netfilter.flags used in LINK_CREATE command for |
| @@ -1500,6 +1510,13 @@ struct bpf_stack_build_id { · 172 unmodified lines | ||
| 1500 | 1510 | }; |
| 1501 | 1511 | }; |
| 1502 | 1512 | |
| 1513 | +struct bpf_common_attr { | |
| 1514 | + __aligned_u64 log_buf; | |
| 1515 | + __u32 log_size; | |
| 1516 | + __u32 log_level; | |
| 1517 | + __u32 log_true_size; | |
| 1518 | +}; | |
| 1519 | + | |
| 1503 | 1520 | #define BPF_OBJ_NAME_LEN 16U |
| 1504 | 1521 | |
| 1505 | 1522 | enum { |
| @@ -1537,6 +1554,11 @@ union bpf_attr { · 31 unmodified lines | ||
| 1537 | 1554 | * |
| 1538 | 1555 | * BPF_MAP_TYPE_ARENA - contains the address where user space |
| 1539 | 1556 | * is going to mmap() the arena. It has to be page aligned. |
| 1557 | + * | |
| 1558 | + * BPF_MAP_TYPE_RHASH - initial table size hint | |
| 1559 | + * (nelem_hint). 0 = use rhashtable default. Must be | |
| 1560 | + * <= min(max_entries, U16_MAX). Upper 32 bits reserved, | |
| 1561 | + * must be zero. | |
| 1540 | 1562 | */ |
| 1541 | 1563 | __u64 map_extra; |
| 1542 | 1564 | |
| @@ -1846,6 +1868,7 @@ union bpf_attr { · 303 unmodified lines | ||
| 1846 | 1868 | __u32 cnt; |
| 1847 | 1869 | __u32 flags; |
| 1848 | 1870 | __u32 pid; |
| 1871 | + __u32 path_fd; | |
| 1849 | 1872 | } uprobe_multi; |
| 1850 | 1873 | struct { |
| 1851 | 1874 | union { |
| @@ -1861,6 +1884,11 @@ union bpf_attr { · 9 unmodified lines | ||
| 1861 | 1884 | }; |
| 1862 | 1885 | __u64 expected_revision; |
| 1863 | 1886 | } cgroup; |
| 1887 | + struct { | |
| 1888 | + __aligned_u64 ids; | |
| 1889 | + __aligned_u64 cookies; | |
| 1890 | + __u32 cnt; | |
| 1891 | + } tracing_multi; | |
| 1864 | 1892 | }; |
| 1865 | 1893 | } link_create; |
| 1866 | 1894 | |
| @@ -6698,6 +6726,7 @@ struct bpf_prog_info { · 4831 unmodified lines | ||
| 6698 | 6726 | __u32 verified_insns; |
| 6699 | 6727 | __u32 attach_btf_obj_id; |
| 6700 | 6728 | __u32 attach_btf_id; |
| 6729 | + __u32 :32; | |
| 6701 | 6730 | } __attribute__((aligned(8))); |
| 6702 | 6731 | |
| 6703 | 6732 | struct bpf_map_info { |
| @@ -6719,6 +6748,7 @@ struct bpf_map_info { · 15 unmodified lines | ||
| 6719 | 6748 | __u64 map_extra; |
| 6720 | 6749 | __aligned_u64 hash; |
| 6721 | 6750 | __u32 hash_size; |
| 6751 | + __u32 :32; | |
| 6722 | 6752 | } __attribute__((aligned(8))); |
| 6723 | 6753 | |
| 6724 | 6754 | struct bpf_btf_info { |
| @@ -7236,6 +7266,7 @@ enum { · 511 unmodified lines | ||
| 7236 | 7266 | TCP_BPF_SOCK_OPS_CB_FLAGS = 1008, /* Get or Set TCP sock ops flags */ |
| 7237 | 7267 | SK_BPF_CB_FLAGS = 1009, /* Get or set sock ops flags in socket */ |
| 7238 | 7268 | SK_BPF_BYPASS_PROT_MEM = 1010, /* Get or Set sk->sk_bypass_prot_mem */ |
| 7269 | + | |
| 7239 | 7270 | }; |
| 7240 | 7271 | |
| 7241 | 7272 | enum { |
| @@ -33,20 +33,22 @@ struct btf_header { · 32 unmodified lines | ||
| 33 | 33 | __u32 layout_len; /* length of layout section */ |
| 34 | 34 | }; |
| 35 | 35 | |
| 36 | −/* Max # of type identifier */ | |
| 37 | −#define BTF_MAX_TYPE 0x000fffff | |
| 38 | −/* Max offset into the string section */ | |
| 39 | −#define BTF_MAX_NAME_OFFSET 0x00ffffff | |
| 40 | −/* Max # of struct/union/enum members or func args */ | |
| 41 | −#define BTF_MAX_VLEN 0xffff | |
| 36 | +enum btf_max { | |
| 37 | + /* Max possible kind */ | |
| 38 | + BTF_MAX_KIND = 0x0000007f, | |
| 39 | + /* Max # of type identifier */ | |
| 40 | + BTF_MAX_TYPE = 0x000fffff, | |
| 41 | + /* Max offset into the string section */ | |
| 42 | + BTF_MAX_NAME_OFFSET = 0x00ffffff, | |
| 43 | + /* Max # of struct/union/enum members or func args */ | |
| 44 | + BTF_MAX_VLEN = 0x00ffffff, | |
| 45 | +}; | |
| 42 | 46 | |
| 43 | 47 | struct btf_type { |
| 44 | 48 | __u32 name_off; |
| 45 | 49 | /* "info" bits arrangement |
| 46 | − * bits 0-15: vlen (e.g. # of struct's members) | |
| 47 | − * bits 16-23: unused | |
| 48 | − * bits 24-28: kind (e.g. int, ptr, array...etc) | |
| 49 | − * bits 29-30: unused | |
| 50 | + * bits 0-23: vlen (e.g. # of struct's members) | |
| 51 | + * bits 24-30: kind (e.g. int, ptr, array...etc) | |
| 50 | 52 | * bit 31: kind_flag, currently used by |
| 51 | 53 | * struct, union, enum, fwd, enum64, |
| 52 | 54 | * decl_tag and type_tag |
| @@ -65,8 +67,8 @@ struct btf_type { · 12 unmodified lines | ||
| 65 | 67 | }; |
| 66 | 68 | }; |
| 67 | 69 | |
| 68 | −#define BTF_INFO_KIND(info) (((info) >> 24) & 0x1f) | |
| 69 | −#define BTF_INFO_VLEN(info) ((info) & 0xffff) | |
| 70 | +#define BTF_INFO_KIND(info) (((info) >> 24) & 0x7f) | |
| 71 | +#define BTF_INFO_VLEN(info) ((info) & 0xffffff) | |
| 70 | 72 | #define BTF_INFO_KFLAG(info) ((info) >> 31) |
| 71 | 73 | |
| 72 | 74 | enum { |
| @@ -49,6 +49,14 @@ man_dir_SQ = '$(subst ','\'',$(man_dir))' · 48 unmodified lines | ||
| 49 | 49 | export man_dir man_dir_SQ INSTALL |
| 50 | 50 | export DESTDIR DESTDIR_SQ |
| 51 | 51 | |
| 52 | +# Defer assigning EXTRA_CFLAGS to CFLAGS until after including | |
| 53 | +# tools/scripts/Makefile.include, as it may add flags to EXTRA_CFLAGS. | |
| 54 | +ifdef EXTRA_CFLAGS | |
| 55 | + CFLAGS := | |
| 56 | +else | |
| 57 | + CFLAGS := -g -O2 | |
| 58 | +endif | |
| 59 | + | |
| 52 | 60 | include $(srctree)/tools/scripts/Makefile.include |
| 53 | 61 | |
| 54 | 62 | # copy a bit from Linux kbuild |
| @@ -70,13 +78,6 @@ LIB_TARGET = libbpf.a libbpf.so.$(LIBBPF_VERSION) · 15 unmodified lines | ||
| 70 | 78 | LIB_FILE = libbpf.a libbpf.so* |
| 71 | 79 | PC_FILE = libbpf.pc |
| 72 | 80 | |
| 73 | −# Set compile option CFLAGS | |
| 74 | −ifdef EXTRA_CFLAGS | |
| 75 | − CFLAGS := $(EXTRA_CFLAGS) | |
| 76 | −else | |
| 77 | − CFLAGS := -g -O2 | |
| 78 | −endif | |
| 79 | − | |
| 80 | 81 | # Append required CFLAGS |
| 81 | 82 | override CFLAGS += -std=gnu89 |
| 82 | 83 | override CFLAGS += $(EXTRA_WARNINGS) -Wno-switch-enum |
| @@ -84,7 +85,7 @@ override CFLAGS += -Werror -Wall · 1 unmodified line | ||
| 84 | 85 | override CFLAGS += $(INCLUDES) |
| 85 | 86 | override CFLAGS += -fvisibility=hidden |
| 86 | 87 | override CFLAGS += -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 |
| 87 | −override CFLAGS += $(CLANG_CROSS_FLAGS) | |
| 88 | +override CFLAGS += $(EXTRA_CFLAGS) | |
| 88 | 89 | |
| 89 | 90 | # flags specific for shared library |
| 90 | 91 | SHLIB_FLAGS := -DSHARED -fPIC |
| @@ -59,6 +59,8 @@ · 58 unmodified lines | ||
| 59 | 59 | # define __NR_bpf 6319 |
| 60 | 60 | # elif defined(__mips__) && defined(_ABI64) |
| 61 | 61 | # define __NR_bpf 5315 |
| 62 | +# elif defined(__loongarch__) | |
| 63 | +# define __NR_bpf 280 | |
| 62 | 64 | # else |
| 63 | 65 | # error __NR_bpf not defined. libbpf does not support your arch. |
| 64 | 66 | # endif |
| @@ -69,6 +71,42 @@ static inline __u64 ptr_to_u64(const void *ptr) · 4 unmodified lines | ||
| 69 | 71 | return (__u64) (unsigned long) ptr; |
| 70 | 72 | } |
| 71 | 73 | |
| 74 | +static inline int sys_bpf_ext(enum bpf_cmd cmd, union bpf_attr *attr, | |
| 75 | + unsigned int size, | |
| 76 | + struct bpf_common_attr *attr_common, | |
| 77 | + unsigned int size_common) | |
| 78 | +{ | |
| 79 | + cmd = attr_common ? (cmd | BPF_COMMON_ATTRS) : (cmd & ~BPF_COMMON_ATTRS); | |
| 80 | + return syscall(__NR_bpf, cmd, attr, size, attr_common, size_common); | |
| 81 | +} | |
| 82 | + | |
| 83 | +static inline int sys_bpf_ext_fd(enum bpf_cmd cmd, union bpf_attr *attr, | |
| 84 | + unsigned int size, | |
| 85 | + struct bpf_common_attr *attr_common, | |
| 86 | + unsigned int size_common) | |
| 87 | +{ | |
| 88 | + int fd; | |
| 89 | + | |
| 90 | + fd = sys_bpf_ext(cmd, attr, size, attr_common, size_common); | |
| 91 | + return ensure_good_fd(fd); | |
| 92 | +} | |
| 93 | + | |
| 94 | +int probe_sys_bpf_ext(void) | |
| 95 | +{ | |
| 96 | + const size_t attr_sz = offsetofend(union bpf_attr, prog_token_fd); | |
| 97 | + union bpf_attr attr; | |
| 98 | + int fd; | |
| 99 | + | |
| 100 | + memset(&attr, 0, attr_sz); | |
| 101 | + fd = syscall(__NR_bpf, BPF_PROG_LOAD | BPF_COMMON_ATTRS, &attr, attr_sz, NULL, | |
| 102 | + sizeof(struct bpf_common_attr)); | |
| 103 | + if (fd >= 0) { | |
| 104 | + close(fd); | |
| 105 | + return -EINVAL; | |
| 106 | + } | |
| 107 | + return errno == EFAULT ? 1 : 0; | |
| 108 | +} | |
| 109 | + | |
| 72 | 110 | static inline int sys_bpf(enum bpf_cmd cmd, union bpf_attr *attr, |
| 73 | 111 | unsigned int size) |
| 74 | 112 | { |
| @@ -173,6 +211,9 @@ int bpf_map_create(enum bpf_map_type map_type, · 98 unmodified lines | ||
| 173 | 211 | const struct bpf_map_create_opts *opts) |
| 174 | 212 | { |
| 175 | 213 | const size_t attr_sz = offsetofend(union bpf_attr, excl_prog_hash_size); |
| 214 | + const size_t attr_common_sz = sizeof(struct bpf_common_attr); | |
| 215 | + struct bpf_common_attr attr_common; | |
| 216 | + struct bpf_log_opts *log_opts; | |
| 176 | 217 | union bpf_attr attr; |
| 177 | 218 | int fd; |
| 178 | 219 | |
| @@ -206,7 +247,21 @@ int bpf_map_create(enum bpf_map_type map_type, · 27 unmodified lines | ||
| 206 | 247 | attr.excl_prog_hash = ptr_to_u64(OPTS_GET(opts, excl_prog_hash, NULL)); |
| 207 | 248 | attr.excl_prog_hash_size = OPTS_GET(opts, excl_prog_hash_size, 0); |
| 208 | 249 | |
| 209 | − fd = sys_bpf_fd(BPF_MAP_CREATE, &attr, attr_sz); | |
| 250 | + log_opts = OPTS_GET(opts, log_opts, NULL); | |
| 251 | + if (!OPTS_VALID(log_opts, bpf_log_opts)) | |
| 252 | + return libbpf_err(-EINVAL); | |
| 253 | + | |
| 254 | + if (log_opts && feat_supported(NULL, FEAT_BPF_SYSCALL_COMMON_ATTRS)) { | |
| 255 | + memset(&attr_common, 0, attr_common_sz); | |
| 256 | + attr_common.log_buf = ptr_to_u64(OPTS_GET(log_opts, buf, NULL)); | |
| 257 | + attr_common.log_size = OPTS_GET(log_opts, size, 0); | |
| 258 | + attr_common.log_level = OPTS_GET(log_opts, level, 0); | |
| 259 | + fd = sys_bpf_ext_fd(BPF_MAP_CREATE, &attr, attr_sz, &attr_common, attr_common_sz); | |
| 260 | + OPTS_SET(log_opts, true_size, attr_common.log_true_size); | |
| 261 | + } else { | |
| 262 | + fd = sys_bpf_fd(BPF_MAP_CREATE, &attr, attr_sz); | |
| 263 | + OPTS_SET(log_opts, true_size, 0); | |
| 264 | + } | |
| 210 | 265 | return libbpf_err_errno(fd); |
| 211 | 266 | } |
| 212 | 267 | |
| @@ -787,9 +842,19 @@ int bpf_link_create(int prog_fd, int target_fd, · 574 unmodified lines | ||
| 787 | 842 | attr.link_create.uprobe_multi.ref_ctr_offsets = ptr_to_u64(OPTS_GET(opts, uprobe_multi.ref_ctr_offsets, 0)); |
| 788 | 843 | attr.link_create.uprobe_multi.cookies = ptr_to_u64(OPTS_GET(opts, uprobe_multi.cookies, 0)); |
| 789 | 844 | attr.link_create.uprobe_multi.pid = OPTS_GET(opts, uprobe_multi.pid, 0); |
| 845 | + attr.link_create.uprobe_multi.path_fd = OPTS_GET(opts, uprobe_multi.path_fd, 0); | |
| 790 | 846 | if (!OPTS_ZEROED(opts, uprobe_multi)) |
| 791 | 847 | return libbpf_err(-EINVAL); |
| 792 | 848 | break; |
| 849 | + case BPF_TRACE_FENTRY_MULTI: | |
| 850 | + case BPF_TRACE_FEXIT_MULTI: | |
| 851 | + case BPF_TRACE_FSESSION_MULTI: | |
| 852 | + attr.link_create.tracing_multi.ids = ptr_to_u64(OPTS_GET(opts, tracing_multi.ids, 0)); | |
| 853 | + attr.link_create.tracing_multi.cookies = ptr_to_u64(OPTS_GET(opts, tracing_multi.cookies, 0)); | |
| 854 | + attr.link_create.tracing_multi.cnt = OPTS_GET(opts, tracing_multi.cnt, 0); | |
| 855 | + if (!OPTS_ZEROED(opts, tracing_multi)) | |
| 856 | + return libbpf_err(-EINVAL); | |
| 857 | + break; | |
| 793 | 858 | case BPF_TRACE_RAW_TP: |
| 794 | 859 | case BPF_TRACE_FENTRY: |
| 795 | 860 | case BPF_TRACE_FEXIT: |
| @@ -37,6 +37,18 @@ extern "C" { · 36 unmodified lines | ||
| 37 | 37 | |
| 38 | 38 | LIBBPF_API int libbpf_set_memlock_rlim(size_t memlock_bytes); |
| 39 | 39 | |
| 40 | +struct bpf_log_opts { | |
| 41 | + size_t sz; /* size of this struct for forward/backward compatibility */ | |
| 42 | + | |
| 43 | + char *buf; | |
| 44 | + __u32 size; | |
| 45 | + __u32 level; | |
| 46 | + __u32 true_size; /* out parameter set by kernel */ | |
| 47 | + | |
| 48 | + size_t :0; | |
| 49 | +}; | |
| 50 | +#define bpf_log_opts__last_field true_size | |
| 51 | + | |
| 40 | 52 | struct bpf_map_create_opts { |
| 41 | 53 | size_t sz; /* size of this struct for forward/backward compatibility */ |
| 42 | 54 | |
| @@ -57,9 +69,12 @@ struct bpf_map_create_opts { · 14 unmodified lines | ||
| 57 | 69 | |
| 58 | 70 | const void *excl_prog_hash; |
| 59 | 71 | __u32 excl_prog_hash_size; |
| 72 | + | |
| 73 | + struct bpf_log_opts *log_opts; | |
| 74 | + | |
| 60 | 75 | size_t :0; |
| 61 | 76 | }; |
| 62 | −#define bpf_map_create_opts__last_field excl_prog_hash_size | |
| 77 | +#define bpf_map_create_opts__last_field log_opts | |
| 63 | 78 | |
| 64 | 79 | LIBBPF_API int bpf_map_create(enum bpf_map_type map_type, |
| 65 | 80 | const char *map_name, |
| @@ -429,6 +444,7 @@ struct bpf_link_create_opts { · 363 unmodified lines | ||
| 429 | 444 | const unsigned long *ref_ctr_offsets; |
| 430 | 445 | const __u64 *cookies; |
| 431 | 446 | __u32 pid; |
| 447 | + __u32 path_fd; | |
| 432 | 448 | } uprobe_multi; |
| 433 | 449 | struct { |
| 434 | 450 | __u64 cookie; |
| @@ -454,10 +470,15 @@ struct bpf_link_create_opts { · 19 unmodified lines | ||
| 454 | 470 | __u32 relative_id; |
| 455 | 471 | __u64 expected_revision; |
| 456 | 472 | } cgroup; |
| 473 | + struct { | |
| 474 | + const __u32 *ids; | |
| 475 | + const __u64 *cookies; | |
| 476 | + __u32 cnt; | |
| 477 | + } tracing_multi; | |
| 457 | 478 | }; |
| 458 | 479 | size_t :0; |
| 459 | 480 | }; |
| 460 | −#define bpf_link_create_opts__last_field uprobe_multi.pid | |
| 481 | +#define bpf_link_create_opts__last_field uprobe_multi.path_fd | |
| 461 | 482 | |
| 462 | 483 | LIBBPF_API int bpf_link_create(int prog_fd, int target_fd, |
| 463 | 484 | enum bpf_attach_type attach_type, |
| @@ -421,7 +421,7 @@ static int btf_type_size_unknown(const struct btf *btf, const struct btf_type *t · 420 unmodified lines | ||
| 421 | 421 | { |
| 422 | 422 | __u32 l_cnt = btf->hdr.layout_len / sizeof(struct btf_layout); |
| 423 | 423 | struct btf_layout *l = btf->layout; |
| 424 | − __u16 vlen = btf_vlen(t); | |
| 424 | + __u32 vlen = btf_vlen(t); | |
| 425 | 425 | __u32 kind = btf_kind(t); |
| 426 | 426 | |
| 427 | 427 | /* Fall back to base BTF if needed as they share layout information */ |
| @@ -454,7 +454,7 @@ static int btf_type_size_unknown(const struct btf *btf, const struct btf_type *t · 26 unmodified lines | ||
| 454 | 454 | static int btf_type_size(const struct btf *btf, const struct btf_type *t) |
| 455 | 455 | { |
| 456 | 456 | const int base_size = sizeof(struct btf_type); |
| 457 | − __u16 vlen = btf_vlen(t); | |
| 457 | + __u32 vlen = btf_vlen(t); | |
| 458 | 458 | |
| 459 | 459 | switch (btf_kind(t)) { |
| 460 | 460 | case BTF_KIND_FWD: |
| @@ -506,7 +506,7 @@ static int btf_bswap_type_rest(struct btf_type *t) · 45 unmodified lines | ||
| 506 | 506 | struct btf_array *a; |
| 507 | 507 | struct btf_param *p; |
| 508 | 508 | struct btf_enum *e; |
| 509 | − __u16 vlen = btf_vlen(t); | |
| 509 | + __u32 vlen = btf_vlen(t); | |
| 510 | 510 | int i; |
| 511 | 511 | |
| 512 | 512 | switch (btf_kind(t)) { |
| @@ -1007,7 +1007,7 @@ int btf__align_of(const struct btf *btf, __u32 id) · 494 unmodified lines | ||
| 1007 | 1007 | case BTF_KIND_STRUCT: |
| 1008 | 1008 | case BTF_KIND_UNION: { |
| 1009 | 1009 | const struct btf_member *m = btf_members(t); |
| 1010 | − __u16 vlen = btf_vlen(t); | |
| 1010 | + __u32 vlen = btf_vlen(t); | |
| 1011 | 1011 | int i, max_align = 1, align; |
| 1012 | 1012 | |
| 1013 | 1013 | for (i = 0; i < vlen; i++, m++) { |
| @@ -2121,9 +2121,12 @@ static void *btf_add_type_mem(struct btf *btf, size_t add_sz) · 1107 unmodified lines | ||
| 2121 | 2121 | btf->hdr.type_len, UINT_MAX, add_sz); |
| 2122 | 2122 | } |
| 2123 | 2123 | |
| 2124 | −static void btf_type_inc_vlen(struct btf_type *t) | |
| 2124 | +static int btf_type_inc_vlen(struct btf_type *t) | |
| 2125 | 2125 | { |
| 2126 | + if (btf_vlen(t) == BTF_MAX_VLEN) | |
| 2127 | + return -ENOSPC; | |
| 2126 | 2128 | t->info = btf_type_info(btf_kind(t), btf_vlen(t) + 1, btf_kflag(t)); |
| 2129 | + return 0; | |
| 2127 | 2130 | } |
| 2128 | 2131 | |
| 2129 | 2132 | static void btf_hdr_update_type_len(struct btf *btf, int new_len) |
| @@ -2652,6 +2655,8 @@ int btf__add_field(struct btf *btf, const char *name, int type_id, · 522 unmodified lines | ||
| 2652 | 2655 | t = btf_last_type(btf); |
| 2653 | 2656 | if (!btf_is_composite(t)) |
| 2654 | 2657 | return libbpf_err(-EINVAL); |
| 2658 | + if (btf_vlen(t) == BTF_MAX_VLEN) | |
| 2659 | + return libbpf_err(-ENOSPC); | |
| 2655 | 2660 | |
| 2656 | 2661 | if (validate_type_id(type_id)) |
| 2657 | 2662 | return libbpf_err(-EINVAL); |
| @@ -2686,6 +2691,7 @@ int btf__add_field(struct btf *btf, const char *name, int type_id, · 28 unmodified lines | ||
| 2686 | 2691 | |
| 2687 | 2692 | /* btf_add_type_mem can invalidate t pointer */ |
| 2688 | 2693 | t = btf_last_type(btf); |
| 2694 | + | |
| 2689 | 2695 | /* update parent type's vlen and kflag */ |
| 2690 | 2696 | t->info = btf_type_info(btf_kind(t), btf_vlen(t) + 1, is_bitfield || btf_kflag(t)); |
| 2691 | 2697 | |
| @@ -2796,7 +2802,9 @@ int btf__add_enum_value(struct btf *btf, const char *name, __s64 value) · 104 unmodified lines | ||
| 2796 | 2802 | |
| 2797 | 2803 | /* update parent type's vlen */ |
| 2798 | 2804 | t = btf_last_type(btf); |
| 2799 | − btf_type_inc_vlen(t); | |
| 2805 | + err = btf_type_inc_vlen(t); | |
| 2806 | + if (err) | |
| 2807 | + return libbpf_err(err); | |
| 2800 | 2808 | |
| 2801 | 2809 | /* if negative value, set signedness to signed */ |
| 2802 | 2810 | if (value < 0) |
| @@ -2873,7 +2881,9 @@ int btf__add_enum64_value(struct btf *btf, const char *name, __u64 value) · 70 unmodified lines | ||
| 2873 | 2881 | |
| 2874 | 2882 | /* update parent type's vlen */ |
| 2875 | 2883 | t = btf_last_type(btf); |
| 2876 | − btf_type_inc_vlen(t); | |
| 2884 | + err = btf_type_inc_vlen(t); | |
| 2885 | + if (err) | |
| 2886 | + return libbpf_err(err); | |
| 2877 | 2887 | |
| 2878 | 2888 | btf_hdr_update_type_len(btf, btf->hdr.type_len + sz); |
| 2879 | 2889 | return 0; |
| @@ -3115,7 +3125,9 @@ int btf__add_func_param(struct btf *btf, const char *name, int type_id) · 235 unmodified lines | ||
| 3115 | 3125 | |
| 3116 | 3126 | /* update parent type's vlen */ |
| 3117 | 3127 | t = btf_last_type(btf); |
| 3118 | − btf_type_inc_vlen(t); | |
| 3128 | + err = btf_type_inc_vlen(t); | |
| 3129 | + if (err) | |
| 3130 | + return libbpf_err(err); | |
| 3119 | 3131 | |
| 3120 | 3132 | btf_hdr_update_type_len(btf, btf->hdr.type_len + sz); |
| 3121 | 3133 | return 0; |
| @@ -3257,7 +3269,9 @@ int btf__add_datasec_var_info(struct btf *btf, int var_type_id, __u32 offset, __ · 135 unmodified lines | ||
| 3257 | 3269 | |
| 3258 | 3270 | /* update parent type's vlen */ |
| 3259 | 3271 | t = btf_last_type(btf); |
| 3260 | − btf_type_inc_vlen(t); | |
| 3272 | + err = btf_type_inc_vlen(t); | |
| 3273 | + if (err) | |
| 3274 | + return libbpf_err(err); | |
| 3261 | 3275 | |
| 3262 | 3276 | btf_hdr_update_type_len(btf, btf->hdr.type_len + sz); |
| 3263 | 3277 | return 0; |
| @@ -4311,7 +4325,7 @@ static long btf_hash_enum(struct btf_type *t) · 1047 unmodified lines | ||
| 4311 | 4325 | static bool btf_equal_enum_members(struct btf_type *t1, struct btf_type *t2) |
| 4312 | 4326 | { |
| 4313 | 4327 | const struct btf_enum *m1, *m2; |
| 4314 | − __u16 vlen; | |
| 4328 | + __u32 vlen; | |
| 4315 | 4329 | int i; |
| 4316 | 4330 | |
| 4317 | 4331 | vlen = btf_vlen(t1); |
| @@ -4329,7 +4343,7 @@ static bool btf_equal_enum_members(struct btf_type *t1, struct btf_type *t2) · 11 unmodified lines | ||
| 4329 | 4343 | static bool btf_equal_enum64_members(struct btf_type *t1, struct btf_type *t2) |
| 4330 | 4344 | { |
| 4331 | 4345 | const struct btf_enum64 *m1, *m2; |
| 4332 | − __u16 vlen; | |
| 4346 | + __u32 vlen; | |
| 4333 | 4347 | int i; |
| 4334 | 4348 | |
| 4335 | 4349 | vlen = btf_vlen(t1); |
| @@ -4406,7 +4420,7 @@ static long btf_hash_struct(struct btf_type *t) · 70 unmodified lines | ||
| 4406 | 4420 | static bool btf_shallow_equal_struct(struct btf_type *t1, struct btf_type *t2) |
| 4407 | 4421 | { |
| 4408 | 4422 | const struct btf_member *m1, *m2; |
| 4409 | − __u16 vlen; | |
| 4423 | + __u32 vlen; | |
| 4410 | 4424 | int i; |
| 4411 | 4425 | |
| 4412 | 4426 | if (!btf_equal_common(t1, t2)) |
| @@ -4482,7 +4496,7 @@ static bool btf_compat_array(struct btf_type *t1, struct btf_type *t2) · 69 unmodified lines | ||
| 4482 | 4496 | static long btf_hash_fnproto(struct btf_type *t) |
| 4483 | 4497 | { |
| 4484 | 4498 | const struct btf_param *member = btf_params(t); |
| 4485 | − __u16 vlen = btf_vlen(t); | |
| 4499 | + __u32 vlen = btf_vlen(t); | |
| 4486 | 4500 | long h = btf_hash_common(t); |
| 4487 | 4501 | int i; |
| 4488 | 4502 | |
| @@ -4504,7 +4518,7 @@ static long btf_hash_fnproto(struct btf_type *t) · 15 unmodified lines | ||
| 4504 | 4518 | static bool btf_equal_fnproto(struct btf_type *t1, struct btf_type *t2) |
| 4505 | 4519 | { |
| 4506 | 4520 | const struct btf_param *m1, *m2; |
| 4507 | − __u16 vlen; | |
| 4521 | + __u32 vlen; | |
| 4508 | 4522 | int i; |
| 4509 | 4523 | |
| 4510 | 4524 | if (!btf_equal_common(t1, t2)) |
| @@ -4530,7 +4544,7 @@ static bool btf_equal_fnproto(struct btf_type *t1, struct btf_type *t2) · 19 unmodified lines | ||
| 4530 | 4544 | static bool btf_compat_fnproto(struct btf_type *t1, struct btf_type *t2) |
| 4531 | 4545 | { |
| 4532 | 4546 | const struct btf_param *m1, *m2; |
| 4533 | − __u16 vlen; | |
| 4547 | + __u32 vlen; | |
| 4534 | 4548 | int i; |
| 4535 | 4549 | |
| 4536 | 4550 | /* skip return type ID */ |
| @@ -4578,12 +4592,14 @@ static int btf_dedup_prep(struct btf_dedup *d) · 41 unmodified lines | ||
| 4578 | 4592 | case BTF_KIND_RESTRICT: |
| 4579 | 4593 | case BTF_KIND_PTR: |
| 4580 | 4594 | case BTF_KIND_FWD: |
| 4581 | − case BTF_KIND_TYPEDEF: | |
| 4582 | 4595 | case BTF_KIND_FUNC: |
| 4583 | 4596 | case BTF_KIND_FLOAT: |
| 4584 | 4597 | case BTF_KIND_TYPE_TAG: |
| 4585 | 4598 | h = btf_hash_common(t); |
| 4586 | 4599 | break; |
| 4600 | + case BTF_KIND_TYPEDEF: | |
| 4601 | + h = btf_hash_typedef(t); | |
| 4602 | + break; | |
| 4587 | 4603 | case BTF_KIND_INT: |
| 4588 | 4604 | case BTF_KIND_DECL_TAG: |
| 4589 | 4605 | h = btf_hash_int_decl_tag(t); |
| @@ -5077,7 +5093,7 @@ static int btf_dedup_is_equiv(struct btf_dedup *d, __u32 cand_id, · 487 unmodified lines | ||
| 5077 | 5093 | case BTF_KIND_STRUCT: |
| 5078 | 5094 | case BTF_KIND_UNION: { |
| 5079 | 5095 | const struct btf_member *cand_m, *canon_m; |
| 5080 | − __u16 vlen; | |
| 5096 | + __u32 vlen; | |
| 5081 | 5097 | |
| 5082 | 5098 | if (!btf_shallow_equal_struct(cand_type, canon_type)) |
| 5083 | 5099 | return 0; |
| @@ -5105,7 +5121,7 @@ static int btf_dedup_is_equiv(struct btf_dedup *d, __u32 cand_id, · 21 unmodified lines | ||
| 5105 | 5121 | |
| 5106 | 5122 | case BTF_KIND_FUNC_PROTO: { |
| 5107 | 5123 | const struct btf_param *cand_p, *canon_p; |
| 5108 | − __u16 vlen; | |
| 5124 | + __u32 vlen; | |
| 5109 | 5125 | |
| 5110 | 5126 | if (!btf_compat_fnproto(cand_type, canon_type)) |
| 5111 | 5127 | return 0; |
| @@ -5439,7 +5455,7 @@ static int btf_dedup_ref_type(struct btf_dedup *d, __u32 type_id) · 327 unmodified lines | ||
| 5439 | 5455 | |
| 5440 | 5456 | case BTF_KIND_FUNC_PROTO: { |
| 5441 | 5457 | struct btf_param *param; |
| 5442 | − __u16 vlen; | |
| 5458 | + __u32 vlen; | |
| 5443 | 5459 | int i; |
| 5444 | 5460 | |
| 5445 | 5461 | ref_type_id = btf_dedup_ref_type(d, t->type); |
| @@ -435,7 +435,7 @@ static inline __u16 btf_kind(const struct btf_type *t) · 434 unmodified lines | ||
| 435 | 435 | return BTF_INFO_KIND(t->info); |
| 436 | 436 | } |
| 437 | 437 | |
| 438 | −static inline __u16 btf_vlen(const struct btf_type *t) | |
| 438 | +static inline __u32 btf_vlen(const struct btf_type *t) | |
| 439 | 439 | { |
| 440 | 440 | return BTF_INFO_VLEN(t->info); |
| 441 | 441 | } |
| @@ -316,7 +316,7 @@ static int btf_dump_mark_referenced(struct btf_dump *d) · 315 unmodified lines | ||
| 316 | 316 | { |
| 317 | 317 | int i, j, n = btf__type_cnt(d->btf); |
| 318 | 318 | const struct btf_type *t; |
| 319 | − __u16 vlen; | |
| 319 | + __u32 vlen; | |
| 320 | 320 | |
| 321 | 321 | for (i = d->last_id + 1; i < n; i++) { |
| 322 | 322 | t = btf__type_by_id(d->btf, i); |
| @@ -485,7 +485,7 @@ static int btf_dump_order_type(struct btf_dump *d, __u32 id, bool through_ptr) · 162 unmodified lines | ||
| 485 | 485 | */ |
| 486 | 486 | struct btf_dump_type_aux_state *tstate = &d->type_states[id]; |
| 487 | 487 | const struct btf_type *t; |
| 488 | − __u16 vlen; | |
| 488 | + __u32 vlen; | |
| 489 | 489 | int err, i; |
| 490 | 490 | |
| 491 | 491 | /* return true, letting typedefs know that it's ok to be emitted */ |
| @@ -798,7 +798,7 @@ static void btf_dump_emit_type(struct btf_dump *d, __u32 id, __u32 cont_id) · 306 unmodified lines | ||
| 798 | 798 | */ |
| 799 | 799 | if (top_level_def || t->name_off == 0) { |
| 800 | 800 | const struct btf_member *m = btf_members(t); |
| 801 | − __u16 vlen = btf_vlen(t); | |
| 801 | + __u32 vlen = btf_vlen(t); | |
| 802 | 802 | int i, new_cont_id; |
| 803 | 803 | |
| 804 | 804 | new_cont_id = t->name_off == 0 ? cont_id : id; |
| @@ -820,7 +820,7 @@ static void btf_dump_emit_type(struct btf_dump *d, __u32 id, __u32 cont_id) · 15 unmodified lines | ||
| 820 | 820 | break; |
| 821 | 821 | case BTF_KIND_FUNC_PROTO: { |
| 822 | 822 | const struct btf_param *p = btf_params(t); |
| 823 | − __u16 n = btf_vlen(t); | |
| 823 | + __u32 n = btf_vlen(t); | |
| 824 | 824 | int i; |
| 825 | 825 | |
| 826 | 826 | btf_dump_emit_type(d, t->type, cont_id); |
| @@ -839,7 +839,7 @@ static bool btf_is_struct_packed(const struct btf *btf, __u32 id, · 12 unmodified lines | ||
| 839 | 839 | { |
| 840 | 840 | const struct btf_member *m; |
| 841 | 841 | int max_align = 1, align, i, bit_sz; |
| 842 | − __u16 vlen; | |
| 842 | + __u32 vlen; | |
| 843 | 843 | |
| 844 | 844 | m = btf_members(t); |
| 845 | 845 | vlen = btf_vlen(t); |
| @@ -973,7 +973,7 @@ static void btf_dump_emit_struct_def(struct btf_dump *d, · 127 unmodified lines | ||
| 973 | 973 | bool is_struct = btf_is_struct(t); |
| 974 | 974 | bool packed, prev_bitfield = false; |
| 975 | 975 | int align, i, off = 0; |
| 976 | − __u16 vlen = btf_vlen(t); | |
| 976 | + __u32 vlen = btf_vlen(t); | |
| 977 | 977 | |
| 978 | 978 | align = btf__align_of(d->btf, id); |
| 979 | 979 | packed = is_struct ? btf_is_struct_packed(d->btf, id, t) : 0; |
| @@ -1064,7 +1064,7 @@ static void btf_dump_emit_enum_fwd(struct btf_dump *d, __u32 id, · 84 unmodified lines | ||
| 1064 | 1064 | |
| 1065 | 1065 | static void btf_dump_emit_enum32_val(struct btf_dump *d, |
| 1066 | 1066 | const struct btf_type *t, |
| 1067 | − int lvl, __u16 vlen) | |
| 1067 | + int lvl, __u32 vlen) | |
| 1068 | 1068 | { |
| 1069 | 1069 | const struct btf_enum *v = btf_enum(t); |
| 1070 | 1070 | bool is_signed = btf_kflag(t); |
| @@ -1089,7 +1089,7 @@ static void btf_dump_emit_enum32_val(struct btf_dump *d, · 18 unmodified lines | ||
| 1089 | 1089 | |
| 1090 | 1090 | static void btf_dump_emit_enum64_val(struct btf_dump *d, |
| 1091 | 1091 | const struct btf_type *t, |
| 1092 | − int lvl, __u16 vlen) | |
| 1092 | + int lvl, __u32 vlen) | |
| 1093 | 1093 | { |
| 1094 | 1094 | const struct btf_enum64 *v = btf_enum64(t); |
| 1095 | 1095 | bool is_signed = btf_kflag(t); |
| @@ -1122,7 +1122,7 @@ static void btf_dump_emit_enum_def(struct btf_dump *d, __u32 id, · 26 unmodified lines | ||
| 1122 | 1122 | const struct btf_type *t, |
| 1123 | 1123 | int lvl) |
| 1124 | 1124 | { |
| 1125 | − __u16 vlen = btf_vlen(t); | |
| 1125 | + __u32 vlen = btf_vlen(t); | |
| 1126 | 1126 | |
| 1127 | 1127 | btf_dump_printf(d, "enum%s%s", |
| 1128 | 1128 | t->name_off ? " " : "", |
| @@ -1542,7 +1542,7 @@ static void btf_dump_emit_type_chain(struct btf_dump *d, · 413 unmodified lines | ||
| 1542 | 1542 | } |
| 1543 | 1543 | case BTF_KIND_FUNC_PROTO: { |
| 1544 | 1544 | const struct btf_param *p = btf_params(t); |
| 1545 | − __u16 vlen = btf_vlen(t); | |
| 1545 | + __u32 vlen = btf_vlen(t); | |
| 1546 | 1546 | int i; |
| 1547 | 1547 | |
| 1548 | 1548 | /* |
| @@ -2159,7 +2159,7 @@ static int btf_dump_struct_data(struct btf_dump *d, · 610 unmodified lines | ||
| 2159 | 2159 | const void *data) |
| 2160 | 2160 | { |
| 2161 | 2161 | const struct btf_member *m = btf_members(t); |
| 2162 | − __u16 n = btf_vlen(t); | |
| 2162 | + __u32 n = btf_vlen(t); | |
| 2163 | 2163 | int i, err = 0; |
| 2164 | 2164 | |
| 2165 | 2165 | /* note that we increment depth before calling btf_dump_print() below; |
| @@ -2449,7 +2449,7 @@ static int btf_dump_type_data_check_zero(struct btf_dump *d, · 283 unmodified lines | ||
| 2449 | 2449 | case BTF_KIND_STRUCT: |
| 2450 | 2450 | case BTF_KIND_UNION: { |
| 2451 | 2451 | const struct btf_member *m = btf_members(t); |
| 2452 | − __u16 n = btf_vlen(t); | |
| 2452 | + __u32 n = btf_vlen(t); | |
| 2453 | 2453 | |
| 2454 | 2454 | /* if any struct/union member is non-zero, the struct/union |
| 2455 | 2455 | * is considered non-zero and dumped. |
| @@ -615,6 +615,11 @@ static int probe_kern_btf_layout(int token_fd) · 614 unmodified lines | ||
| 615 | 615 | (char *)layout, token_fd)); |
| 616 | 616 | } |
| 617 | 617 | |
| 618 | +static int probe_bpf_syscall_common_attrs(int token_fd) | |
| 619 | +{ | |
| 620 | + return probe_sys_bpf_ext(); | |
| 621 | +} | |
| 622 | + | |
| 618 | 623 | typedef int (*feature_probe_fn)(int /* token_fd */); |
| 619 | 624 | |
| 620 | 625 | static struct kern_feature_cache feature_cache; |
| @@ -699,6 +704,9 @@ static struct kern_feature_desc { · 78 unmodified lines | ||
| 699 | 704 | [FEAT_BTF_LAYOUT] = { |
| 700 | 705 | "kernel supports BTF layout", probe_kern_btf_layout, |
| 701 | 706 | }, |
| 707 | + [FEAT_BPF_SYSCALL_COMMON_ATTRS] = { | |
| 708 | + "BPF syscall common attributes support", probe_bpf_syscall_common_attrs, | |
| 709 | + }, | |
| 702 | 710 | }; |
| 703 | 711 | |
| 704 | 712 | bool feat_supported(struct kern_feature_cache *cache, enum kern_feature_id feat_id) |
| @@ -63,6 +63,7 @@ static int realloc_insn_buf(struct bpf_gen *gen, __u32 size) · 62 unmodified lines | ||
| 63 | 63 | gen->error = -ENOMEM; |
| 64 | 64 | free(gen->insn_start); |
| 65 | 65 | gen->insn_start = NULL; |
| 66 | + gen->insn_cur = NULL; | |
| 66 | 67 | return -ENOMEM; |
| 67 | 68 | } |
| 68 | 69 | gen->insn_start = insn_start; |
| @@ -86,6 +87,7 @@ static int realloc_data_buf(struct bpf_gen *gen, __u32 size) · 17 unmodified lines | ||
| 86 | 87 | gen->error = -ENOMEM; |
| 87 | 88 | free(gen->data_start); |
| 88 | 89 | gen->data_start = NULL; |
| 90 | + gen->data_cur = NULL; | |
| 89 | 91 | return -ENOMEM; |
| 90 | 92 | } |
| 91 | 93 | gen->data_start = data_start; |
| @@ -158,10 +160,16 @@ void bpf_gen__init(struct bpf_gen *gen, int log_level, int nr_progs, int nr_maps · 66 unmodified lines | ||
| 158 | 160 | |
| 159 | 161 | static int add_data(struct bpf_gen *gen, const void *data, __u32 size) |
| 160 | 162 | { |
| 161 | − __u32 size8 = roundup(size, 8); | |
| 162 | 163 | __u64 zero = 0; |
| 164 | + __u32 size8; | |
| 163 | 165 | void *prev; |
| 164 | 166 | |
| 167 | + if (size > INT32_MAX) { | |
| 168 | + gen->error = -ERANGE; | |
| 169 | + return 0; | |
| 170 | + } | |
| 171 | + size8 = roundup(size, 8); | |
| 172 | + | |
| 165 | 173 | if (realloc_data_buf(gen, size8)) |
| 166 | 174 | return 0; |
| 167 | 175 | prev = gen->data_cur; |
| @@ -293,7 +301,6 @@ static void emit_check_err(struct bpf_gen *gen) · 125 unmodified lines | ||
| 293 | 301 | emit(gen, BPF_JMP_IMM(BPF_JSLT, BPF_REG_7, 0, off)); |
| 294 | 302 | } else { |
| 295 | 303 | gen->error = -ERANGE; |
| 296 | − emit(gen, BPF_JMP_IMM(BPF_JA, 0, 0, -1)); | |
| 297 | 304 | } |
| 298 | 305 | } |
| 299 | 306 | |
| @@ -398,13 +405,12 @@ int bpf_gen__finish(struct bpf_gen *gen, int nr_progs, int nr_maps) · 98 unmodified lines | ||
| 398 | 405 | blob_fd_array_off(gen, i)); |
| 399 | 406 | emit(gen, BPF_MOV64_IMM(BPF_REG_0, 0)); |
| 400 | 407 | emit(gen, BPF_EXIT_INSN()); |
| 401 | − if (OPTS_GET(gen->opts, gen_hash, false)) | |
| 402 | − compute_sha_update_offsets(gen); | |
| 403 | − | |
| 404 | − pr_debug("gen: finish %s\n", errstr(gen->error)); | |
| 405 | 408 | if (!gen->error) { |
| 406 | 409 | struct gen_loader_opts *opts = gen->opts; |
| 407 | 410 | |
| 411 | + if (OPTS_GET(opts, gen_hash, false)) | |
| 412 | + compute_sha_update_offsets(gen); | |
| 413 | + | |
| 408 | 414 | opts->insns = gen->insn_start; |
| 409 | 415 | opts->insns_sz = gen->insn_cur - gen->insn_start; |
| 410 | 416 | opts->data = gen->data_start; |
| @@ -419,6 +425,7 @@ int bpf_gen__finish(struct bpf_gen *gen, int nr_progs, int nr_maps) · 8 unmodified lines | ||
| 419 | 425 | bpf_insn_bswap(insn++); |
| 420 | 426 | } |
| 421 | 427 | } |
| 428 | + pr_debug("gen: finish %s\n", errstr(gen->error)); | |
| 422 | 429 | return gen->error; |
| 423 | 430 | } |
| 424 | 431 | |
| @@ -545,13 +552,22 @@ void bpf_gen__map_create(struct bpf_gen *gen, · 120 unmodified lines | ||
| 545 | 552 | default: |
| 546 | 553 | break; |
| 547 | 554 | } |
| 548 | − /* conditionally update max_entries */ | |
| 549 | − if (map_idx >= 0) | |
| 555 | + | |
| 556 | + /* | |
| 557 | + * Conditionally update max_entries from the host-supplied loader | |
| 558 | + * ctx. This sizes the map at runtime, but for a signed loader | |
| 559 | + * (gen_hash) it would let an untrusted host re-dimension the | |
| 560 | + * program's maps after emit_signature_match(), outside what the | |
| 561 | + * signature attests to. Keep the signer-provided max_entries | |
| 562 | + * baked into the blob in that case. | |
| 563 | + */ | |
| 564 | + if (map_idx >= 0 && !OPTS_GET(gen->opts, gen_hash, false)) | |
| 550 | 565 | move_ctx2blob(gen, attr_field(map_create_attr, max_entries), 4, |
| 551 | 566 | sizeof(struct bpf_loader_ctx) + |
| 552 | 567 | sizeof(struct bpf_map_desc) * map_idx + |
| 553 | 568 | offsetof(struct bpf_map_desc, max_entries), |
| 554 | 569 | true /* check that max_entries != 0 */); |
| 570 | + | |
| 555 | 571 | /* emit MAP_CREATE command */ |
| 556 | 572 | emit_sys_bpf(gen, BPF_MAP_CREATE, map_create_attr, attr_size); |
| 557 | 573 | debug_ret(gen, "map_create %s idx %d type %d value_size %d value_btf_id %d", |
| @@ -585,6 +601,23 @@ static void emit_signature_match(struct bpf_gen *gen) · 27 unmodified lines | ||
| 585 | 601 | __s64 off; |
| 586 | 602 | int i; |
| 587 | 603 | |
| 604 | + /* | |
| 605 | + * Reject if the metadata map is not exclusive. Without exclusivity | |
| 606 | + * the cached map->sha[] verified above can be stale: another BPF | |
| 607 | + * program with map access could have mutated the contents between | |
| 608 | + * BPF_OBJ_GET_INFO_BY_FD and loader execution. | |
| 609 | + */ | |
| 610 | + emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_IDX, | |
| 611 | + 0, 0, 0, 0)); | |
| 612 | + emit(gen, BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, SHA256_DIGEST_LENGTH)); | |
| 613 | + off = -(gen->insn_cur - gen->insn_start - gen->cleanup_label) / 8 - 2; | |
| 614 | + if (is_simm16(off)) { | |
| 615 | + emit(gen, BPF_MOV64_IMM(BPF_REG_7, -EINVAL)); | |
| 616 | + emit(gen, BPF_JMP_IMM(BPF_JNE, BPF_REG_2, 1, off)); | |
| 617 | + } else { | |
| 618 | + gen->error = -ERANGE; | |
| 619 | + } | |
| 620 | + | |
| 588 | 621 | for (i = 0; i < SHA256_DWORD_SIZE; i++) { |
| 589 | 622 | emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_IDX, |
| 590 | 623 | 0, 0, 0, 0)); |
| @@ -1053,7 +1086,7 @@ void bpf_gen__prog_load(struct bpf_gen *gen, · 462 unmodified lines | ||
| 1053 | 1086 | prog_idx, prog_type, insns_off, insn_cnt, license_off); |
| 1054 | 1087 | |
| 1055 | 1088 | /* convert blob insns to target endianness */ |
| 1056 | − if (gen->swapped_endian) { | |
| 1089 | + if (gen->swapped_endian && !gen->error) { | |
| 1057 | 1090 | struct bpf_insn *insn = gen->data_start + insns_off; |
| 1058 | 1091 | int i; |
| 1059 | 1092 | |
| @@ -1091,7 +1124,7 @@ void bpf_gen__prog_load(struct bpf_gen *gen, · 31 unmodified lines | ||
| 1091 | 1124 | sizeof(struct bpf_core_relo)); |
| 1092 | 1125 | |
| 1093 | 1126 | /* convert all info blobs to target endianness */ |
| 1094 | − if (gen->swapped_endian) | |
| 1127 | + if (gen->swapped_endian && !gen->error) | |
| 1095 | 1128 | info_blob_bswap(gen, func_info, line_info, core_relos, load_attr); |
| 1096 | 1129 | |
| 1097 | 1130 | libbpf_strlcpy(attr.prog_name, prog_name, sizeof(attr.prog_name)); |
| @@ -1169,27 +1202,36 @@ void bpf_gen__map_update_elem(struct bpf_gen *gen, int map_idx, void *pvalue, · 71 unmodified lines | ||
| 1169 | 1202 | value = add_data(gen, pvalue, value_size); |
| 1170 | 1203 | key = add_data(gen, &zero, sizeof(zero)); |
| 1171 | 1204 | |
| 1172 | − /* if (map_desc[map_idx].initial_value) { | |
| 1205 | + /* | |
| 1206 | + * if (map_desc[map_idx].initial_value) { | |
| 1173 | 1207 | * if (ctx->flags & BPF_SKEL_KERNEL) |
| 1174 | 1208 | * bpf_probe_read_kernel(value, value_size, initial_value); |
| 1175 | 1209 | * else |
| 1176 | 1210 | * bpf_copy_from_user(value, value_size, initial_value); |
| 1177 | 1211 | * } |
| 1212 | + * | |
| 1213 | + * The runtime initial_value comes from the host-supplied loader | |
| 1214 | + * ctx and would overwrite the blob value after emit_signature_match() | |
| 1215 | + * has already validated map->sha[]. For a signed loader (gen_hash) | |
| 1216 | + * the attested blob value must be authoritative, so skip the override | |
| 1217 | + * and leave the hashed value in place. | |
| 1178 | 1218 | */ |
| 1179 | − emit(gen, BPF_LDX_MEM(BPF_DW, BPF_REG_3, BPF_REG_6, | |
| 1180 | − sizeof(struct bpf_loader_ctx) + | |
| 1181 | − sizeof(struct bpf_map_desc) * map_idx + | |
| 1182 | − offsetof(struct bpf_map_desc, initial_value))); | |
| 1183 | − emit(gen, BPF_JMP_IMM(BPF_JEQ, BPF_REG_3, 0, 8)); | |
| 1184 | − emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_IDX_VALUE, | |
| 1185 | − 0, 0, 0, value)); | |
| 1186 | − emit(gen, BPF_MOV64_IMM(BPF_REG_2, value_size)); | |
| 1187 | − emit(gen, BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_6, | |
| 1188 | − offsetof(struct bpf_loader_ctx, flags))); | |
| 1189 | − emit(gen, BPF_JMP_IMM(BPF_JSET, BPF_REG_0, BPF_SKEL_KERNEL, 2)); | |
| 1190 | − emit(gen, BPF_EMIT_CALL(BPF_FUNC_copy_from_user)); | |
| 1191 | − emit(gen, BPF_JMP_IMM(BPF_JA, 0, 0, 1)); | |
| 1192 | − emit(gen, BPF_EMIT_CALL(BPF_FUNC_probe_read_kernel)); | |
| 1219 | + if (!OPTS_GET(gen->opts, gen_hash, false)) { | |
| 1220 | + emit(gen, BPF_LDX_MEM(BPF_DW, BPF_REG_3, BPF_REG_6, | |
| 1221 | + sizeof(struct bpf_loader_ctx) + | |
| 1222 | + sizeof(struct bpf_map_desc) * map_idx + | |
| 1223 | + offsetof(struct bpf_map_desc, initial_value))); | |
| 1224 | + emit(gen, BPF_JMP_IMM(BPF_JEQ, BPF_REG_3, 0, 8)); | |
| 1225 | + emit2(gen, BPF_LD_IMM64_RAW_FULL(BPF_REG_1, BPF_PSEUDO_MAP_IDX_VALUE, | |
| 1226 | + 0, 0, 0, value)); | |
| 1227 | + emit(gen, BPF_MOV64_IMM(BPF_REG_2, value_size)); | |
| 1228 | + emit(gen, BPF_LDX_MEM(BPF_W, BPF_REG_0, BPF_REG_6, | |
| 1229 | + offsetof(struct bpf_loader_ctx, flags))); | |
| 1230 | + emit(gen, BPF_JMP_IMM(BPF_JSET, BPF_REG_0, BPF_SKEL_KERNEL, 2)); | |
| 1231 | + emit(gen, BPF_EMIT_CALL(BPF_FUNC_copy_from_user)); | |
| 1232 | + emit(gen, BPF_JMP_IMM(BPF_JA, 0, 0, 1)); | |
| 1233 | + emit(gen, BPF_EMIT_CALL(BPF_FUNC_probe_read_kernel)); | |
| 1234 | + } | |
| 1193 | 1235 | |
| 1194 | 1236 | map_update_attr = add_data(gen, &attr, attr_size); |
| 1195 | 1237 | pr_debug("gen: map_update_elem: idx %d, value: off %d size %d, attr: off %d size %d\n", |
| @@ -136,6 +136,9 @@ static const char * const attach_type_name[] = { · 135 unmodified lines | ||
| 136 | 136 | [BPF_NETKIT_PEER] = "netkit_peer", |
| 137 | 137 | [BPF_TRACE_KPROBE_SESSION] = "trace_kprobe_session", |
| 138 | 138 | [BPF_TRACE_UPROBE_SESSION] = "trace_uprobe_session", |
| 139 | + [BPF_TRACE_FENTRY_MULTI] = "trace_fentry_multi", | |
| 140 | + [BPF_TRACE_FEXIT_MULTI] = "trace_fexit_multi", | |
| 141 | + [BPF_TRACE_FSESSION_MULTI] = "trace_fsession_multi", | |
| 139 | 142 | }; |
| 140 | 143 | |
| 141 | 144 | static const char * const link_type_name[] = { |
| @@ -154,6 +157,7 @@ static const char * const link_type_name[] = { · 12 unmodified lines | ||
| 154 | 157 | [BPF_LINK_TYPE_UPROBE_MULTI] = "uprobe_multi", |
| 155 | 158 | [BPF_LINK_TYPE_NETKIT] = "netkit", |
| 156 | 159 | [BPF_LINK_TYPE_SOCKMAP] = "sockmap", |
| 160 | + [BPF_LINK_TYPE_TRACING_MULTI] = "tracing_multi", | |
| 157 | 161 | }; |
| 158 | 162 | |
| 159 | 163 | static const char * const map_type_name[] = { |
| @@ -192,6 +196,7 @@ static const char * const map_type_name[] = { · 32 unmodified lines | ||
| 192 | 196 | [BPF_MAP_TYPE_CGRP_STORAGE] = "cgrp_storage", |
| 193 | 197 | [BPF_MAP_TYPE_ARENA] = "arena", |
| 194 | 198 | [BPF_MAP_TYPE_INSN_ARRAY] = "insn_array", |
| 199 | + [BPF_MAP_TYPE_RHASH] = "rhash", | |
| 195 | 200 | }; |
| 196 | 201 | |
| 197 | 202 | static const char * const prog_type_name[] = { |
| @@ -7767,6 +7772,69 @@ static int bpf_object__sanitize_prog(struct bpf_object *obj, struct bpf_program · 7569 unmodified lines | ||
| 7767 | 7772 | static int libbpf_find_attach_btf_id(struct bpf_program *prog, const char *attach_name, |
| 7768 | 7773 | int *btf_obj_fd, int *btf_type_id); |
| 7769 | 7774 | |
| 7775 | +static inline bool is_tracing_multi(enum bpf_attach_type type) | |
| 7776 | +{ | |
| 7777 | + return type == BPF_TRACE_FENTRY_MULTI || type == BPF_TRACE_FEXIT_MULTI || | |
| 7778 | + type == BPF_TRACE_FSESSION_MULTI; | |
| 7779 | +} | |
| 7780 | + | |
| 7781 | +static const struct module_btf *find_attach_module(struct bpf_object *obj, const char *attach) | |
| 7782 | +{ | |
| 7783 | + const char *sep, *mod_name = NULL; | |
| 7784 | + int i, mod_len, err; | |
| 7785 | + | |
| 7786 | + /* | |
| 7787 | + * We expect attach string in the form of either | |
| 7788 | + * - function_pattern or | |
| 7789 | + * - <module>:function_pattern | |
| 7790 | + */ | |
| 7791 | + sep = strchr(attach, ':'); | |
| 7792 | + if (sep) { | |
| 7793 | + mod_name = attach; | |
| 7794 | + mod_len = sep - mod_name; | |
| 7795 | + } | |
| 7796 | + if (!mod_name) | |
| 7797 | + return NULL; | |
| 7798 | + | |
| 7799 | + err = load_module_btfs(obj); | |
| 7800 | + if (err) | |
| 7801 | + return NULL; | |
| 7802 | + | |
| 7803 | + for (i = 0; i < obj->btf_module_cnt; i++) { | |
| 7804 | + const struct module_btf *mod = &obj->btf_modules[i]; | |
| 7805 | + | |
| 7806 | + if (strncmp(mod->name, mod_name, mod_len) == 0 && mod->name[mod_len] == '\0') | |
| 7807 | + return mod; | |
| 7808 | + } | |
| 7809 | + return NULL; | |
| 7810 | +} | |
| 7811 | + | |
| 7812 | +static int tracing_multi_mod_fd(struct bpf_program *prog, int *btf_obj_fd) | |
| 7813 | +{ | |
| 7814 | + const char *attach_name, *sep; | |
| 7815 | + const struct module_btf *mod; | |
| 7816 | + | |
| 7817 | + *btf_obj_fd = 0; | |
| 7818 | + attach_name = strchr(prog->sec_name, '/'); | |
| 7819 | + | |
| 7820 | + /* Program with no details in spec, using kernel btf. */ | |
| 7821 | + if (!attach_name) | |
| 7822 | + return 0; | |
| 7823 | + | |
| 7824 | + /* Program with no module section, using kernel btf. */ | |
| 7825 | + sep = strchr(++attach_name, ':'); | |
| 7826 | + if (!sep) | |
| 7827 | + return 0; | |
| 7828 | + | |
| 7829 | + /* Program with module specified, get its btf fd. */ | |
| 7830 | + mod = find_attach_module(prog->obj, attach_name); | |
| 7831 | + if (!mod) | |
| 7832 | + return -EINVAL; | |
| 7833 | + | |
| 7834 | + *btf_obj_fd = mod->fd; | |
| 7835 | + return 0; | |
| 7836 | +} | |
| 7837 | + | |
| 7770 | 7838 | /* this is called as prog->sec_def->prog_prepare_load_fn for libbpf-supported sec_defs */ |
| 7771 | 7839 | static int libbpf_prepare_prog_load(struct bpf_program *prog, |
| 7772 | 7840 | struct bpf_prog_load_opts *opts, long cookie) |
| @@ -7830,6 +7898,18 @@ static int libbpf_prepare_prog_load(struct bpf_program *prog, · 57 unmodified lines | ||
| 7830 | 7898 | opts->attach_btf_obj_fd = btf_obj_fd; |
| 7831 | 7899 | opts->attach_btf_id = btf_type_id; |
| 7832 | 7900 | } |
| 7901 | + | |
| 7902 | + if (is_tracing_multi(prog->expected_attach_type)) { | |
| 7903 | + int err, btf_obj_fd = 0; | |
| 7904 | + | |
| 7905 | + err = tracing_multi_mod_fd(prog, &btf_obj_fd); | |
| 7906 | + if (err < 0) | |
| 7907 | + return err; | |
| 7908 | + | |
| 7909 | + prog->attach_btf_obj_fd = btf_obj_fd; | |
| 7910 | + opts->attach_btf_obj_fd = btf_obj_fd; | |
| 7911 | + } | |
| 7912 | + | |
| 7833 | 7913 | return 0; |
| 7834 | 7914 | } |
| 7835 | 7915 | |
| @@ -8936,13 +9016,10 @@ static void bpf_object_unpin(struct bpf_object *obj) · 1100 unmodified lines | ||
| 8936 | 9016 | bpf_map__unpin(&obj->maps[i], NULL); |
| 8937 | 9017 | } |
| 8938 | 9018 | |
| 8939 | −static void bpf_object_post_load_cleanup(struct bpf_object *obj) | |
| 9019 | +static void bpf_object_cleanup_btf(struct bpf_object *obj) | |
| 8940 | 9020 | { |
| 8941 | 9021 | int i; |
| 8942 | 9022 | |
| 8943 | − /* clean up fd_array */ | |
| 8944 | − zfree(&obj->fd_array); | |
| 8945 | − | |
| 8946 | 9023 | /* clean up module BTFs */ |
| 8947 | 9024 | for (i = 0; i < obj->btf_module_cnt; i++) { |
| 8948 | 9025 | close(obj->btf_modules[i].fd); |
| @@ -8950,6 +9027,8 @@ static void bpf_object_post_load_cleanup(struct bpf_object *obj) · 1 unmodified line | ||
| 8950 | 9027 | free(obj->btf_modules[i].name); |
| 8951 | 9028 | } |
| 8952 | 9029 | obj->btf_module_cnt = 0; |
| 9030 | + obj->btf_module_cap = 0; | |
| 9031 | + obj->btf_modules_loaded = false; | |
| 8953 | 9032 | zfree(&obj->btf_modules); |
| 8954 | 9033 | |
| 8955 | 9034 | /* clean up vmlinux BTF */ |
| @@ -8957,6 +9036,15 @@ static void bpf_object_post_load_cleanup(struct bpf_object *obj) · 1 unmodified line | ||
| 8957 | 9036 | obj->btf_vmlinux = NULL; |
| 8958 | 9037 | } |
| 8959 | 9038 | |
| 9039 | +static void bpf_object_post_load_cleanup(struct bpf_object *obj) | |
| 9040 | +{ | |
| 9041 | + /* clean up fd_array */ | |
| 9042 | + zfree(&obj->fd_array); | |
| 9043 | + | |
| 9044 | + /* clean up BTF */ | |
| 9045 | + bpf_object_cleanup_btf(obj); | |
| 9046 | +} | |
| 9047 | + | |
| 8960 | 9048 | static int bpf_object_prepare(struct bpf_object *obj, const char *target_btf_path) |
| 8961 | 9049 | { |
| 8962 | 9050 | int err; |
| @@ -9983,6 +10071,7 @@ static int attach_kprobe_session(const struct bpf_program *prog, long cookie, st · 1020 unmodified lines | ||
| 9983 | 10071 | static int attach_uprobe_multi(const struct bpf_program *prog, long cookie, struct bpf_link **link); |
| 9984 | 10072 | static int attach_lsm(const struct bpf_program *prog, long cookie, struct bpf_link **link); |
| 9985 | 10073 | static int attach_iter(const struct bpf_program *prog, long cookie, struct bpf_link **link); |
| 10074 | +static int attach_tracing_multi(const struct bpf_program *prog, long cookie, struct bpf_link **link); | |
| 9986 | 10075 | |
| 9987 | 10076 | static const struct bpf_sec_def section_defs[] = { |
| 9988 | 10077 | SEC_DEF("socket", SOCKET_FILTER, 0, SEC_NONE), |
| @@ -10018,11 +10107,16 @@ static const struct bpf_sec_def section_defs[] = { · 29 unmodified lines | ||
| 10018 | 10107 | SEC_DEF("netkit/peer", SCHED_CLS, BPF_NETKIT_PEER, SEC_NONE), |
| 10019 | 10108 | SEC_DEF("tracepoint+", TRACEPOINT, 0, SEC_NONE, attach_tp), |
| 10020 | 10109 | SEC_DEF("tp+", TRACEPOINT, 0, SEC_NONE, attach_tp), |
| 10110 | + SEC_DEF("tracepoint.s+", TRACEPOINT, 0, SEC_SLEEPABLE, attach_tp), | |
| 10111 | + SEC_DEF("tp.s+", TRACEPOINT, 0, SEC_SLEEPABLE, attach_tp), | |
| 10021 | 10112 | SEC_DEF("raw_tracepoint+", RAW_TRACEPOINT, 0, SEC_NONE, attach_raw_tp), |
| 10022 | 10113 | SEC_DEF("raw_tp+", RAW_TRACEPOINT, 0, SEC_NONE, attach_raw_tp), |
| 10114 | + SEC_DEF("raw_tracepoint.s+", RAW_TRACEPOINT, 0, SEC_SLEEPABLE, attach_raw_tp), | |
| 10115 | + SEC_DEF("raw_tp.s+", RAW_TRACEPOINT, 0, SEC_SLEEPABLE, attach_raw_tp), | |
| 10023 | 10116 | SEC_DEF("raw_tracepoint.w+", RAW_TRACEPOINT_WRITABLE, 0, SEC_NONE, attach_raw_tp), |
| 10024 | 10117 | SEC_DEF("raw_tp.w+", RAW_TRACEPOINT_WRITABLE, 0, SEC_NONE, attach_raw_tp), |
| 10025 | 10118 | SEC_DEF("tp_btf+", TRACING, BPF_TRACE_RAW_TP, SEC_ATTACH_BTF, attach_trace), |
| 10119 | + SEC_DEF("tp_btf.s+", TRACING, BPF_TRACE_RAW_TP, SEC_ATTACH_BTF | SEC_SLEEPABLE, attach_trace), | |
| 10026 | 10120 | SEC_DEF("fentry+", TRACING, BPF_TRACE_FENTRY, SEC_ATTACH_BTF, attach_trace), |
| 10027 | 10121 | SEC_DEF("fmod_ret+", TRACING, BPF_MODIFY_RETURN, SEC_ATTACH_BTF, attach_trace), |
| 10028 | 10122 | SEC_DEF("fexit+", TRACING, BPF_TRACE_FEXIT, SEC_ATTACH_BTF, attach_trace), |
| @@ -10031,6 +10125,12 @@ static const struct bpf_sec_def section_defs[] = { · 2 unmodified lines | ||
| 10031 | 10125 | SEC_DEF("fexit.s+", TRACING, BPF_TRACE_FEXIT, SEC_ATTACH_BTF | SEC_SLEEPABLE, attach_trace), |
| 10032 | 10126 | SEC_DEF("fsession+", TRACING, BPF_TRACE_FSESSION, SEC_ATTACH_BTF, attach_trace), |
| 10033 | 10127 | SEC_DEF("fsession.s+", TRACING, BPF_TRACE_FSESSION, SEC_ATTACH_BTF | SEC_SLEEPABLE, attach_trace), |
| 10128 | + SEC_DEF("fsession.multi+", TRACING, BPF_TRACE_FSESSION_MULTI, 0, attach_tracing_multi), | |
| 10129 | + SEC_DEF("fsession.multi.s+", TRACING, BPF_TRACE_FSESSION_MULTI, SEC_SLEEPABLE, attach_tracing_multi), | |
| 10130 | + SEC_DEF("fentry.multi+", TRACING, BPF_TRACE_FENTRY_MULTI, 0, attach_tracing_multi), | |
| 10131 | + SEC_DEF("fexit.multi+", TRACING, BPF_TRACE_FEXIT_MULTI, 0, attach_tracing_multi), | |
| 10132 | + SEC_DEF("fentry.multi.s+", TRACING, BPF_TRACE_FENTRY_MULTI, SEC_SLEEPABLE, attach_tracing_multi), | |
| 10133 | + SEC_DEF("fexit.multi.s+", TRACING, BPF_TRACE_FEXIT_MULTI, SEC_SLEEPABLE, attach_tracing_multi), | |
| 10034 | 10134 | SEC_DEF("freplace+", EXT, 0, SEC_ATTACH_BTF, attach_trace), |
| 10035 | 10135 | SEC_DEF("lsm+", LSM, BPF_LSM_MAC, SEC_ATTACH_BTF, attach_lsm), |
| 10036 | 10136 | SEC_DEF("lsm.s+", LSM, BPF_LSM_MAC, SEC_ATTACH_BTF | SEC_SLEEPABLE, attach_lsm), |
| @@ -12280,7 +12380,7 @@ error: · 2243 unmodified lines | ||
| 12280 | 12380 | static int attach_kprobe(const struct bpf_program *prog, long cookie, struct bpf_link **link) |
| 12281 | 12381 | { |
| 12282 | 12382 | DECLARE_LIBBPF_OPTS(bpf_kprobe_opts, opts); |
| 12283 | − unsigned long offset = 0; | |
| 12383 | + long offset = 0; | |
| 12284 | 12384 | const char *func_name; |
| 12285 | 12385 | char *func; |
| 12286 | 12386 | int n; |
| @@ -12302,6 +12402,13 @@ static int attach_kprobe(const struct bpf_program *prog, long cookie, struct bpf · 15 unmodified lines | ||
| 12302 | 12402 | pr_warn("kprobe name is invalid: %s\n", func_name); |
| 12303 | 12403 | return -EINVAL; |
| 12304 | 12404 | } |
| 12405 | + | |
| 12406 | + if (offset < 0) { | |
| 12407 | + free(func); | |
| 12408 | + pr_warn("kprobe offset must be a non-negative integer: %li\n", offset); | |
| 12409 | + return -EINVAL; | |
| 12410 | + } | |
| 12411 | + | |
| 12305 | 12412 | if (opts.retprobe && offset != 0) { |
| 12306 | 12413 | free(func); |
| 12307 | 12414 | pr_warn("kretprobes do not support offset specification\n"); |
| @@ -12425,6 +12532,279 @@ static int attach_uprobe_multi(const struct bpf_program *prog, long cookie, stru · 117 unmodified lines | ||
| 12425 | 12532 | return ret; |
| 12426 | 12533 | } |
| 12427 | 12534 | |
| 12535 | +#define MAX_BPF_FUNC_ARGS 12 | |
| 12536 | + | |
| 12537 | +static bool btf_type_is_modifier(const struct btf_type *t) | |
| 12538 | +{ | |
| 12539 | + switch (BTF_INFO_KIND(t->info)) { | |
| 12540 | + case BTF_KIND_TYPEDEF: | |
| 12541 | + case BTF_KIND_VOLATILE: | |
| 12542 | + case BTF_KIND_CONST: | |
| 12543 | + case BTF_KIND_RESTRICT: | |
| 12544 | + case BTF_KIND_TYPE_TAG: | |
| 12545 | + return true; | |
| 12546 | + default: | |
| 12547 | + return false; | |
| 12548 | + } | |
| 12549 | +} | |
| 12550 | + | |
| 12551 | +#define MAX_RESOLVE_DEPTH 32 | |
| 12552 | + | |
| 12553 | +static int btf_get_type_size(const struct btf *btf, __u32 type_id, | |
| 12554 | + const struct btf_type **ret_type) | |
| 12555 | +{ | |
| 12556 | + const struct btf_type *t; | |
| 12557 | + int i; | |
| 12558 | + | |
| 12559 | + *ret_type = btf__type_by_id(btf, 0); | |
| 12560 | + if (!type_id) | |
| 12561 | + return 0; | |
| 12562 | + t = btf__type_by_id(btf, type_id); | |
| 12563 | + for (i = 0; i < MAX_RESOLVE_DEPTH && t && btf_type_is_modifier(t); i++) | |
| 12564 | + t = btf__type_by_id(btf, t->type); | |
| 12565 | + if (!t || i == MAX_RESOLVE_DEPTH) | |
| 12566 | + return -EINVAL; | |
| 12567 | + *ret_type = t; | |
| 12568 | + if (btf_is_ptr(t)) | |
| 12569 | + return btf__pointer_size(btf); | |
| 12570 | + if (btf_is_int(t) || btf_is_any_enum(t) || btf_is_struct(t) || btf_is_union(t)) | |
| 12571 | + return t->size; | |
| 12572 | + return -EINVAL; | |
| 12573 | +} | |
| 12574 | + | |
| 12575 | +bool btf_type_is_traceable_func(const struct btf *btf, const struct btf_type *t) | |
| 12576 | +{ | |
| 12577 | + const struct btf_param *args; | |
| 12578 | + const struct btf_type *proto; | |
| 12579 | + __u32 i, nargs; | |
| 12580 | + int ret; | |
| 12581 | + | |
| 12582 | + if (!btf_is_func(t)) | |
| 12583 | + return false; | |
| 12584 | + proto = btf__type_by_id(btf, t->type); | |
| 12585 | + if (!proto || !btf_is_func_proto(proto)) | |
| 12586 | + return false; | |
| 12587 | + | |
| 12588 | + args = (const struct btf_param *)(proto + 1); | |
| 12589 | + nargs = btf_vlen(proto); | |
| 12590 | + if (nargs > MAX_BPF_FUNC_ARGS) | |
| 12591 | + return false; | |
| 12592 | + | |
| 12593 | + /* No support for struct return type. */ | |
| 12594 | + ret = btf_get_type_size(btf, proto->type, &t); | |
| 12595 | + if (ret < 0 || btf_is_struct(t) || btf_is_union(t)) | |
| 12596 | + return false; | |
| 12597 | + | |
| 12598 | + for (i = 0; i < nargs; i++) { | |
| 12599 | + /* No support for variable args. */ | |
| 12600 | + if (i == nargs - 1 && args[i].type == 0) | |
| 12601 | + return false; | |
| 12602 | + ret = btf_get_type_size(btf, args[i].type, &t); | |
| 12603 | + /* No support of struct argument size greater than 16 bytes. */ | |
| 12604 | + if (ret < 0 || ret > 16) | |
| 12605 | + return false; | |
| 12606 | + /* No support for void argument. */ | |
| 12607 | + if (ret == 0) | |
| 12608 | + return false; | |
| 12609 | + } | |
| 12610 | + | |
| 12611 | + return true; | |
| 12612 | +} | |
| 12613 | + | |
| 12614 | +static int | |
| 12615 | +collect_btf_func_ids_by_glob(const struct btf *btf, const char *pattern, __u32 **ids) | |
| 12616 | +{ | |
| 12617 | + __u32 type_id, nr_types = btf__type_cnt(btf); | |
| 12618 | + size_t cap = 0, cnt = 0; | |
| 12619 | + | |
| 12620 | + if (!pattern) | |
| 12621 | + return -EINVAL; | |
| 12622 | + | |
| 12623 | + for (type_id = 1; type_id < nr_types; type_id++) { | |
| 12624 | + const struct btf_type *t = btf__type_by_id(btf, type_id); | |
| 12625 | + const char *name; | |
| 12626 | + int err; | |
| 12627 | + | |
| 12628 | + if (btf_kind(t) != BTF_KIND_FUNC) | |
| 12629 | + continue; | |
| 12630 | + name = btf__name_by_offset(btf, t->name_off); | |
| 12631 | + if (!name) | |
| 12632 | + continue; | |
| 12633 | + | |
| 12634 | + if (!glob_match(name, pattern)) | |
| 12635 | + continue; | |
| 12636 | + if (!btf_type_is_traceable_func(btf, t)) | |
| 12637 | + continue; | |
| 12638 | + | |
| 12639 | + err = libbpf_ensure_mem((void **) ids, &cap, sizeof(**ids), cnt + 1); | |
| 12640 | + if (err) { | |
| 12641 | + free(*ids); | |
| 12642 | + return -ENOMEM; | |
| 12643 | + } | |
| 12644 | + (*ids)[cnt++] = type_id; | |
| 12645 | + } | |
| 12646 | + | |
| 12647 | + return cnt; | |
| 12648 | +} | |
| 12649 | + | |
| 12650 | +static int collect_func_ids_by_glob(const struct bpf_program *prog, const char *pattern, __u32 **ids) | |
| 12651 | +{ | |
| 12652 | + struct bpf_object *obj = prog->obj; | |
| 12653 | + const struct module_btf *mod; | |
| 12654 | + struct btf *btf = NULL; | |
| 12655 | + const char *sep; | |
| 12656 | + int err; | |
| 12657 | + | |
| 12658 | + err = bpf_object__load_vmlinux_btf(obj, true); | |
| 12659 | + if (err) | |
| 12660 | + return err; | |
| 12661 | + | |
| 12662 | + /* In case we have module specified, we will find its btf and use that. */ | |
| 12663 | + sep = strchr(pattern, ':'); | |
| 12664 | + if (sep) { | |
| 12665 | + mod = find_attach_module(obj, pattern); | |
| 12666 | + if (!mod) { | |
| 12667 | + err = -EINVAL; | |
| 12668 | + goto cleanup; | |
| 12669 | + } | |
| 12670 | + btf = mod->btf; | |
| 12671 | + pattern = sep + 1; | |
| 12672 | + } else { | |
| 12673 | + /* Program is loaded for kernel module. */ | |
| 12674 | + if (prog->attach_btf_obj_fd) { | |
| 12675 | + err = -EINVAL; | |
| 12676 | + goto cleanup; | |
| 12677 | + } | |
| 12678 | + btf = obj->btf_vmlinux; | |
| 12679 | + } | |
| 12680 | + | |
| 12681 | + err = collect_btf_func_ids_by_glob(btf, pattern, ids); | |
| 12682 | + | |
| 12683 | +cleanup: | |
| 12684 | + bpf_object_cleanup_btf(obj); | |
| 12685 | + return err; | |
| 12686 | +} | |
| 12687 | + | |
| 12688 | +struct bpf_link * | |
| 12689 | +bpf_program__attach_tracing_multi(const struct bpf_program *prog, const char *pattern, | |
| 12690 | + const struct bpf_tracing_multi_opts *opts) | |
| 12691 | +{ | |
| 12692 | + LIBBPF_OPTS(bpf_link_create_opts, lopts); | |
| 12693 | + int prog_fd, link_fd, err, cnt; | |
| 12694 | + __u32 *free_ids = NULL; | |
| 12695 | + struct bpf_link *link; | |
| 12696 | + const __u64 *cookies; | |
| 12697 | + const __u32 *ids; | |
| 12698 | + | |
| 12699 | + if (!OPTS_VALID(opts, bpf_tracing_multi_opts)) | |
| 12700 | + return libbpf_err_ptr(-EINVAL); | |
| 12701 | + | |
| 12702 | + prog_fd = bpf_program__fd(prog); | |
| 12703 | + if (prog_fd < 0) { | |
| 12704 | + pr_warn("prog '%s': can't attach BPF program without FD (was it loaded?)\n", | |
| 12705 | + prog->name); | |
| 12706 | + return libbpf_err_ptr(-EINVAL); | |
| 12707 | + } | |
| 12708 | + | |
| 12709 | + cnt = OPTS_GET(opts, cnt, 0); | |
| 12710 | + ids = OPTS_GET(opts, ids, NULL); | |
| 12711 | + cookies = OPTS_GET(opts, cookies, NULL); | |
| 12712 | + | |
| 12713 | + if (!!ids != !!cnt) | |
| 12714 | + return libbpf_err_ptr(-EINVAL); | |
| 12715 | + if (pattern && (ids || cookies)) | |
| 12716 | + return libbpf_err_ptr(-EINVAL); | |
| 12717 | + if (!pattern && !ids) | |
| 12718 | + return libbpf_err_ptr(-EINVAL); | |
| 12719 | + | |
| 12720 | + if (pattern) { | |
| 12721 | + cnt = collect_func_ids_by_glob(prog, pattern, &free_ids); | |
| 12722 | + if (cnt < 0) | |
| 12723 | + return libbpf_err_ptr(cnt); | |
| 12724 | + if (cnt == 0) | |
| 12725 | + return libbpf_err_ptr(-EINVAL); | |
| 12726 | + ids = (const __u32 *) free_ids; | |
| 12727 | + } | |
| 12728 | + | |
| 12729 | + lopts.tracing_multi.ids = ids; | |
| 12730 | + lopts.tracing_multi.cookies = cookies; | |
| 12731 | + lopts.tracing_multi.cnt = cnt; | |
| 12732 | + | |
| 12733 | + link = calloc(1, sizeof(*link)); | |
| 12734 | + if (!link) { | |
| 12735 | + err = -ENOMEM; | |
| 12736 | + goto error; | |
| 12737 | + } | |
| 12738 | + link->detach = &bpf_link__detach_fd; | |
| 12739 | + | |
| 12740 | + link_fd = bpf_link_create(prog_fd, 0, prog->expected_attach_type, &lopts); | |
| 12741 | + if (link_fd < 0) { | |
| 12742 | + err = -errno; | |
| 12743 | + pr_warn("prog '%s': failed to attach: %s\n", prog->name, errstr(err)); | |
| 12744 | + goto error; | |
| 12745 | + } | |
| 12746 | + link->fd = link_fd; | |
| 12747 | + free(free_ids); | |
| 12748 | + return link; | |
| 12749 | + | |
| 12750 | +error: | |
| 12751 | + free(link); | |
| 12752 | + free(free_ids); | |
| 12753 | + return libbpf_err_ptr(err); | |
| 12754 | +} | |
| 12755 | + | |
| 12756 | +static int attach_tracing_multi(const struct bpf_program *prog, long cookie, struct bpf_link **link) | |
| 12757 | +{ | |
| 12758 | + static const char *const prefixes[] = { | |
| 12759 | + "fentry.multi", | |
| 12760 | + "fexit.multi", | |
| 12761 | + "fsession.multi", | |
| 12762 | + "fentry.multi.s", | |
| 12763 | + "fexit.multi.s", | |
| 12764 | + "fsession.multi.s", | |
| 12765 | + }; | |
| 12766 | + const char *spec = NULL; | |
| 12767 | + char *pattern; | |
| 12768 | + size_t i; | |
| 12769 | + int n; | |
| 12770 | + | |
| 12771 | + *link = NULL; | |
| 12772 | + | |
| 12773 | + for (i = 0; i < ARRAY_SIZE(prefixes); i++) { | |
| 12774 | + size_t pfx_len; | |
| 12775 | + | |
| 12776 | + if (!str_has_pfx(prog->sec_name, prefixes[i])) | |
| 12777 | + continue; | |
| 12778 | + | |
| 12779 | + pfx_len = strlen(prefixes[i]); | |
| 12780 | + /* no auto-attach case of, e.g., SEC("fentry.multi") */ | |
| 12781 | + if (prog->sec_name[pfx_len] == '\0') | |
| 12782 | + return 0; | |
| 12783 | + | |
| 12784 | + if (prog->sec_name[pfx_len] != '/') | |
| 12785 | + continue; | |
| 12786 | + | |
| 12787 | + spec = prog->sec_name + pfx_len + 1; | |
| 12788 | + break; | |
| 12789 | + } | |
| 12790 | + | |
| 12791 | + if (!spec) { | |
| 12792 | + pr_warn("prog '%s': invalid section name '%s'\n", | |
| 12793 | + prog->name, prog->sec_name); | |
| 12794 | + return -EINVAL; | |
| 12795 | + } | |
| 12796 | + | |
| 12797 | + n = sscanf(spec, "%m[a-zA-Z0-9_.*?:]", &pattern); | |
| 12798 | + if (n < 1) { | |
| 12799 | + pr_warn("tracing multi pattern is invalid: %s\n", spec); | |
| 12800 | + return -EINVAL; | |
| 12801 | + } | |
| 12802 | + | |
| 12803 | + *link = bpf_program__attach_tracing_multi(prog, pattern, NULL); | |
| 12804 | + free(pattern); | |
| 12805 | + return libbpf_get_error(*link); | |
| 12806 | +} | |
| 12807 | + | |
| 12428 | 12808 | static inline int add_uprobe_event_legacy(const char *probe_name, bool retprobe, |
| 12429 | 12809 | const char *binary_path, size_t offset) |
| 12430 | 12810 | { |
| @@ -13145,25 +13525,61 @@ struct bpf_link *bpf_program__attach_tracepoint(const struct bpf_program *prog, · 714 unmodified lines | ||
| 13145 | 13525 | return bpf_program__attach_tracepoint_opts(prog, tp_category, tp_name, NULL); |
| 13146 | 13526 | } |
| 13147 | 13527 | |
| 13528 | +/* | |
| 13529 | + * Match section name against a prefix array. Returns pointer past | |
| 13530 | + * "prefix/" on match, empty string for bare sections (exact prefix | |
| 13531 | + * match), or NULL if no prefix matches. | |
| 13532 | + */ | |
| 13533 | +static const char *sec_name_match_prefix(const char *sec_name, | |
| 13534 | + const char *const *prefixes, | |
| 13535 | + size_t n) | |
| 13536 | +{ | |
| 13537 | + size_t i; | |
| 13538 | + | |
| 13539 | + for (i = 0; i < n; i++) { | |
| 13540 | + size_t pfx_len; | |
| 13541 | + | |
| 13542 | + if (!str_has_pfx(sec_name, prefixes[i])) | |
| 13543 | + continue; | |
| 13544 | + | |
| 13545 | + pfx_len = strlen(prefixes[i]); | |
| 13546 | + if (sec_name[pfx_len] == '\0') | |
| 13547 | + return sec_name + pfx_len; | |
| 13548 | + | |
| 13549 | + if (sec_name[pfx_len] != '/' || sec_name[pfx_len + 1] == '\0') | |
| 13550 | + continue; | |
| 13551 | + | |
| 13552 | + return sec_name + pfx_len + 1; | |
| 13553 | + } | |
| 13554 | + return NULL; | |
| 13555 | +} | |
| 13556 | + | |
| 13148 | 13557 | static int attach_tp(const struct bpf_program *prog, long cookie, struct bpf_link **link) |
| 13149 | 13558 | { |
| 13559 | + static const char *const prefixes[] = { | |
| 13560 | + "tp.s", | |
| 13561 | + "tp", | |
| 13562 | + "tracepoint.s", | |
| 13563 | + "tracepoint", | |
| 13564 | + }; | |
| 13150 | 13565 | char *sec_name, *tp_cat, *tp_name; |
| 13566 | + const char *match; | |
| 13151 | 13567 | |
| 13152 | 13568 | *link = NULL; |
| 13153 | 13569 | |
| 13154 | − /* no auto-attach for SEC("tp") or SEC("tracepoint") */ | |
| 13155 | − if (strcmp(prog->sec_name, "tp") == 0 || strcmp(prog->sec_name, "tracepoint") == 0) | |
| 13570 | + match = sec_name_match_prefix(prog->sec_name, prefixes, ARRAY_SIZE(prefixes)); | |
| 13571 | + if (!match) { | |
| 13572 | + pr_warn("prog '%s': invalid section name '%s'\n", prog->name, prog->sec_name); | |
| 13573 | + return -EINVAL; | |
| 13574 | + } | |
| 13575 | + if (!match[0]) /* bare section name no autoattach */ | |
| 13156 | 13576 | return 0; |
| 13157 | 13577 | |
| 13158 | 13578 | sec_name = strdup(prog->sec_name); |
| 13159 | 13579 | if (!sec_name) |
| 13160 | 13580 | return -ENOMEM; |
| 13161 | 13581 | |
| 13162 | − /* extract "tp/<category>/<name>" or "tracepoint/<category>/<name>" */ | |
| 13163 | − if (str_has_pfx(prog->sec_name, "tp/")) | |
| 13164 | − tp_cat = sec_name + sizeof("tp/") - 1; | |
| 13165 | − else | |
| 13166 | − tp_cat = sec_name + sizeof("tracepoint/") - 1; | |
| 13582 | + tp_cat = sec_name + (match - prog->sec_name); | |
| 13167 | 13583 | tp_name = strchr(tp_cat, '/'); |
| 13168 | 13584 | if (!tp_name) { |
| 13169 | 13585 | free(sec_name); |
| @@ -13227,37 +13643,22 @@ static int attach_raw_tp(const struct bpf_program *prog, long cookie, struct bpf · 57 unmodified lines | ||
| 13227 | 13643 | "raw_tracepoint", |
| 13228 | 13644 | "raw_tp.w", |
| 13229 | 13645 | "raw_tracepoint.w", |
| 13646 | + "raw_tp.s", | |
| 13647 | + "raw_tracepoint.s", | |
| 13230 | 13648 | }; |
| 13231 | − size_t i; | |
| 13232 | − const char *tp_name = NULL; | |
| 13649 | + const char *match; | |
| 13233 | 13650 | |
| 13234 | 13651 | *link = NULL; |
| 13235 | 13652 | |
| 13236 | − for (i = 0; i < ARRAY_SIZE(prefixes); i++) { | |
| 13237 | − size_t pfx_len; | |
| 13238 | − | |
| 13239 | − if (!str_has_pfx(prog->sec_name, prefixes[i])) | |
| 13240 | − continue; | |
| 13241 | − | |
| 13242 | − pfx_len = strlen(prefixes[i]); | |
| 13243 | − /* no auto-attach case of, e.g., SEC("raw_tp") */ | |
| 13244 | − if (prog->sec_name[pfx_len] == '\0') | |
| 13245 | − return 0; | |
| 13246 | − | |
| 13247 | − if (prog->sec_name[pfx_len] != '/') | |
| 13248 | − continue; | |
| 13249 | − | |
| 13250 | − tp_name = prog->sec_name + pfx_len + 1; | |
| 13251 | − break; | |
| 13252 | − } | |
| 13253 | − | |
| 13254 | − if (!tp_name) { | |
| 13255 | − pr_warn("prog '%s': invalid section name '%s'\n", | |
| 13256 | − prog->name, prog->sec_name); | |
| 13653 | + match = sec_name_match_prefix(prog->sec_name, prefixes, ARRAY_SIZE(prefixes)); | |
| 13654 | + if (!match) { | |
| 13655 | + pr_warn("prog '%s': invalid section name '%s'\n", prog->name, prog->sec_name); | |
| 13257 | 13656 | return -EINVAL; |
| 13258 | 13657 | } |
| 13658 | + if (!match[0]) | |
| 13659 | + return 0; | |
| 13259 | 13660 | |
| 13260 | − *link = bpf_program__attach_raw_tracepoint(prog, tp_name); | |
| 13661 | + *link = bpf_program__attach_raw_tracepoint(prog, match); | |
| 13261 | 13662 | return libbpf_get_error(*link); |
| 13262 | 13663 | } |
| 13263 | 13664 | |
| @@ -726,6 +726,21 @@ bpf_program__attach_ksyscall(const struct bpf_program *prog, · 725 unmodified lines | ||
| 726 | 726 | const char *syscall_name, |
| 727 | 727 | const struct bpf_ksyscall_opts *opts); |
| 728 | 728 | |
| 729 | +struct bpf_tracing_multi_opts { | |
| 730 | + /* size of this struct, for forward/backward compatibility */ | |
| 731 | + size_t sz; | |
| 732 | + const __u32 *ids; | |
| 733 | + const __u64 *cookies; | |
| 734 | + size_t cnt; | |
| 735 | + size_t :0; | |
| 736 | +}; | |
| 737 | + | |
| 738 | +#define bpf_tracing_multi_opts__last_field cnt | |
| 739 | + | |
| 740 | +LIBBPF_API struct bpf_link * | |
| 741 | +bpf_program__attach_tracing_multi(const struct bpf_program *prog, const char *pattern, | |
| 742 | + const struct bpf_tracing_multi_opts *opts); | |
| 743 | + | |
| 729 | 744 | struct bpf_uprobe_opts { |
| 730 | 745 | /* size of this struct, for forward/backward compatibility */ |
| 731 | 746 | size_t sz; |
| @@ -458,6 +458,7 @@ LIBBPF_1.7.0 { · 457 unmodified lines | ||
| 458 | 458 | |
| 459 | 459 | LIBBPF_1.8.0 { |
| 460 | 460 | global: |
| 461 | + bpf_program__attach_tracing_multi; | |
| 461 | 462 | bpf_program__clone; |
| 462 | 463 | btf__new_empty_opts; |
| 463 | 464 | } LIBBPF_1.7.0; |
| @@ -250,6 +250,7 @@ const struct btf_type *skip_mods_and_typedefs(const struct btf *btf, __u32 id, _ · 249 unmodified lines | ||
| 250 | 250 | const struct btf_header *btf_header(const struct btf *btf); |
| 251 | 251 | void btf_set_base_btf(struct btf *btf, const struct btf *base_btf); |
| 252 | 252 | int btf_relocate(struct btf *btf, const struct btf *base_btf, __u32 **id_map); |
| 253 | +bool btf_type_is_traceable_func(const struct btf *btf, const struct btf_type *t); | |
| 253 | 254 | |
| 254 | 255 | static inline enum btf_func_linkage btf_func_linkage(const struct btf_type *t) |
| 255 | 256 | { |
| @@ -398,6 +399,8 @@ enum kern_feature_id { · 142 unmodified lines | ||
| 398 | 399 | FEAT_UPROBE_SYSCALL, |
| 399 | 400 | /* Kernel supports BTF layout information */ |
| 400 | 401 | FEAT_BTF_LAYOUT, |
| 402 | + /* Kernel supports BPF syscall common attributes */ | |
| 403 | + FEAT_BPF_SYSCALL_COMMON_ATTRS, | |
| 401 | 404 | __FEAT_CNT, |
| 402 | 405 | }; |
| 403 | 406 | |
| @@ -768,4 +771,5 @@ int probe_fd(int fd); · 364 unmodified lines | ||
| 768 | 771 | #define SHA256_DWORD_SIZE SHA256_DIGEST_LENGTH / sizeof(__u64) |
| 769 | 772 | |
| 770 | 773 | void libbpf_sha256(const void *data, size_t len, __u8 out[SHA256_DIGEST_LENGTH]); |
| 774 | +int probe_sys_bpf_ext(void); | |
| 771 | 775 | #endif /* __LIBBPF_LIBBPF_INTERNAL_H */ |
| @@ -309,6 +309,9 @@ static int probe_map_create(enum bpf_map_type map_type) · 308 unmodified lines | ||
| 309 | 309 | value_size = sizeof(__u64); |
| 310 | 310 | opts.map_flags = BPF_F_NO_PREALLOC; |
| 311 | 311 | break; |
| 312 | + case BPF_MAP_TYPE_RHASH: | |
| 313 | + opts.map_flags = BPF_F_NO_PREALLOC; | |
| 314 | + break; | |
| 312 | 315 | case BPF_MAP_TYPE_CGROUP_STORAGE: |
| 313 | 316 | case BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE: |
| 314 | 317 | key_size = sizeof(struct bpf_cgroup_storage_key); |
| @@ -191,8 +191,8 @@ recur: · 190 unmodified lines | ||
| 191 | 191 | case BTF_KIND_FUNC_PROTO: { |
| 192 | 192 | struct btf_param *local_p = btf_params(local_type); |
| 193 | 193 | struct btf_param *targ_p = btf_params(targ_type); |
| 194 | − __u16 local_vlen = btf_vlen(local_type); | |
| 195 | − __u16 targ_vlen = btf_vlen(targ_type); | |
| 194 | + __u32 local_vlen = btf_vlen(local_type); | |
| 195 | + __u32 targ_vlen = btf_vlen(targ_type); | |
| 196 | 196 | int i, err; |
| 197 | 197 | |
| 198 | 198 | if (local_vlen != targ_vlen) |
| @@ -1457,8 +1457,8 @@ static bool bpf_core_names_match(const struct btf *local_btf, size_t local_name_ · 1258 unmodified lines | ||
| 1457 | 1457 | static int bpf_core_enums_match(const struct btf *local_btf, const struct btf_type *local_t, |
| 1458 | 1458 | const struct btf *targ_btf, const struct btf_type *targ_t) |
| 1459 | 1459 | { |
| 1460 | − __u16 local_vlen = btf_vlen(local_t); | |
| 1461 | − __u16 targ_vlen = btf_vlen(targ_t); | |
| 1460 | + __u32 local_vlen = btf_vlen(local_t); | |
| 1461 | + __u32 targ_vlen = btf_vlen(targ_t); | |
| 1462 | 1462 | int i, j; |
| 1463 | 1463 | |
| 1464 | 1464 | if (local_t->size != targ_t->size) |
| @@ -1498,8 +1498,8 @@ static int bpf_core_composites_match(const struct btf *local_btf, const struct b · 33 unmodified lines | ||
| 1498 | 1498 | bool behind_ptr, int level) |
| 1499 | 1499 | { |
| 1500 | 1500 | const struct btf_member *local_m = btf_members(local_t); |
| 1501 | − __u16 local_vlen = btf_vlen(local_t); | |
| 1502 | − __u16 targ_vlen = btf_vlen(targ_t); | |
| 1501 | + __u32 local_vlen = btf_vlen(local_t); | |
| 1502 | + __u32 targ_vlen = btf_vlen(targ_t); | |
| 1503 | 1503 | int i, j, err; |
| 1504 | 1504 | |
| 1505 | 1505 | if (local_vlen > targ_vlen) |
| @@ -1674,8 +1674,8 @@ recur: · 168 unmodified lines | ||
| 1674 | 1674 | case BTF_KIND_FUNC_PROTO: { |
| 1675 | 1675 | struct btf_param *local_p = btf_params(local_t); |
| 1676 | 1676 | struct btf_param *targ_p = btf_params(targ_t); |
| 1677 | − __u16 local_vlen = btf_vlen(local_t); | |
| 1678 | − __u16 targ_vlen = btf_vlen(targ_t); | |
| 1677 | + __u32 local_vlen = btf_vlen(local_t); | |
| 1678 | + __u32 targ_vlen = btf_vlen(targ_t); | |
| 1679 | 1679 | int i, err; |
| 1680 | 1680 | |
| 1681 | 1681 | if (local_k != targ_k) |
| @@ -243,7 +243,12 @@ static inline int skel_map_create(enum bpf_map_type map_type, · 242 unmodified lines | ||
| 243 | 243 | attr.excl_prog_hash = (unsigned long) excl_prog_hash; |
| 244 | 244 | attr.excl_prog_hash_size = excl_prog_hash_sz; |
| 245 | 245 | |
| 246 | +#ifdef __KERNEL__ | |
| 247 | + if (strscpy(attr.map_name, map_name) < 0) | |
| 248 | + return -EINVAL; | |
| 249 | +#else | |
| 246 | 250 | strncpy(attr.map_name, map_name, sizeof(attr.map_name)); |
| 251 | +#endif | |
| 247 | 252 | attr.key_size = key_size; |
| 248 | 253 | attr.value_size = value_size; |
| 249 | 254 | attr.max_entries = max_entries; |
| @@ -107,6 +107,41 @@ static void *strset_add_str_mem(struct strset *set, size_t add_sz) · 106 unmodified lines | ||
| 107 | 107 | set->strs_data_len, set->strs_data_max_len, add_sz); |
| 108 | 108 | } |
| 109 | 109 | |
| 110 | +static long strset_str_append(struct strset *set, const char *s) | |
| 111 | +{ | |
| 112 | + uintptr_t old_data = (uintptr_t)set->strs_data; | |
| 113 | + size_t old_data_len = set->strs_data_len; | |
| 114 | + uintptr_t old_s = (uintptr_t)s; | |
| 115 | + long len = strlen(s) + 1; | |
| 116 | + void *p; | |
| 117 | + | |
| 118 | + /* | |
| 119 | + * Hashmap keys are always offsets within set->strs_data, so to even | |
| 120 | + * look up some string from the "outside", we need to first append it | |
| 121 | + * at the end, so that it can be addressed with an offset. Luckily, | |
| 122 | + * until set->strs_data_len is incremented, that string is just a piece | |
| 123 | + * of garbage for the rest of the code, so no harm, no foul. On the | |
| 124 | + * other hand, if the string is unique, it's already appended and | |
| 125 | + * ready to be used, only a simple set->strs_data_len increment away. | |
| 126 | + */ | |
| 127 | + p = strset_add_str_mem(set, len); | |
| 128 | + if (!p) | |
| 129 | + return -ENOMEM; | |
| 130 | + | |
| 131 | + /* | |
| 132 | + * The set->strs_data might have reallocated and if 's' pointed | |
| 133 | + * to an internal string within the old buffer, then it became | |
| 134 | + * dangling and needs to be reconstructed before the copy. | |
| 135 | + */ | |
| 136 | + if (old_data && old_data != (uintptr_t)set->strs_data && | |
| 137 | + old_s >= old_data && old_s < old_data + old_data_len) | |
| 138 | + s = set->strs_data + (old_s - old_data); | |
| 139 | + | |
| 140 | + memcpy(p, s, len); | |
| 141 | + | |
| 142 | + return len; | |
| 143 | +} | |
| 144 | + | |
| 110 | 145 | /* Find string offset that corresponds to a given string *s*. |
| 111 | 146 | * Returns: |
| 112 | 147 | * - >0 offset into string data, if string is found; |
| @@ -116,16 +151,12 @@ static void *strset_add_str_mem(struct strset *set, size_t add_sz) · 3 unmodified lines | ||
| 116 | 151 | int strset__find_str(struct strset *set, const char *s) |
| 117 | 152 | { |
| 118 | 153 | long old_off, new_off, len; |
| 119 | − void *p; | |
| 120 | 154 | |
| 121 | − /* see strset__add_str() for why we do this */ | |
| 122 | − len = strlen(s) + 1; | |
| 123 | − p = strset_add_str_mem(set, len); | |
| 124 | − if (!p) | |
| 125 | − return -ENOMEM; | |
| 155 | + len = strset_str_append(set, s); | |
| 156 | + if (len < 0) | |
| 157 | + return len; | |
| 126 | 158 | |
| 127 | 159 | new_off = set->strs_data_len; |
| 128 | − memcpy(p, s, len); | |
| 129 | 160 | |
| 130 | 161 | if (hashmap__find(set->strs_hash, new_off, &old_off)) |
| 131 | 162 | return old_off; |
| @@ -142,24 +173,13 @@ int strset__find_str(struct strset *set, const char *s) · 10 unmodified lines | ||
| 142 | 173 | int strset__add_str(struct strset *set, const char *s) |
| 143 | 174 | { |
| 144 | 175 | long old_off, new_off, len; |
| 145 | − void *p; | |
| 146 | 176 | int err; |
| 147 | 177 | |
| 148 | − /* Hashmap keys are always offsets within set->strs_data, so to even | |
| 149 | − * look up some string from the "outside", we need to first append it | |
| 150 | − * at the end, so that it can be addressed with an offset. Luckily, | |
| 151 | − * until set->strs_data_len is incremented, that string is just a piece | |
| 152 | − * of garbage for the rest of the code, so no harm, no foul. On the | |
| 153 | − * other hand, if the string is unique, it's already appended and | |
| 154 | − * ready to be used, only a simple set->strs_data_len increment away. | |
| 155 | − */ | |
| 156 | − len = strlen(s) + 1; | |
| 157 | − p = strset_add_str_mem(set, len); | |
| 158 | − if (!p) | |
| 159 | − return -ENOMEM; | |
| 178 | + len = strset_str_append(set, s); | |
| 179 | + if (len < 0) | |
| 180 | + return len; | |
| 160 | 181 | |
| 161 | 182 | new_off = set->strs_data_len; |
| 162 | − memcpy(p, s, len); | |
| 163 | 183 | |
| 164 | 184 | /* Now attempt to add the string, but only if the string with the same |
| 165 | 185 | * contents doesn't exist already (HASHMAP_ADD strategy). If such |
| @@ -468,10 +468,10 @@ static int parse_elf_segs(Elf *elf, const char *path, struct elf_seg **segs, siz · 467 unmodified lines | ||
| 468 | 468 | |
| 469 | 469 | static int parse_vma_segs(int pid, const char *lib_path, struct elf_seg **segs, size_t *seg_cnt) |
| 470 | 470 | { |
| 471 | − char path[PATH_MAX], line[PATH_MAX], mode[16]; | |
| 471 | + char path[PATH_MAX], line[4096], mode[16]; | |
| 472 | 472 | size_t seg_start, seg_end, seg_off; |
| 473 | 473 | struct elf_seg *seg; |
| 474 | − int tmp_pid, i, err; | |
| 474 | + int tmp_pid, n, i, err; | |
| 475 | 475 | FILE *f; |
| 476 | 476 | |
| 477 | 477 | *seg_cnt = 0; |
| @@ -480,8 +480,13 @@ static int parse_vma_segs(int pid, const char *lib_path, struct elf_seg **segs, · 2 unmodified lines | ||
| 480 | 480 | * /proc/<pid>/root/<path>. They will be reported as just /<path> in |
| 481 | 481 | * /proc/<pid>/maps. |
| 482 | 482 | */ |
| 483 | − if (sscanf(lib_path, "/proc/%d/root%s", &tmp_pid, path) == 2 && pid == tmp_pid) | |
| 483 | + /* %n is not counted in sscanf() return value, so initialize it. */ | |
| 484 | + n = 0; | |
| 485 | + if (sscanf(lib_path, "/proc/%d/root%n", &tmp_pid, &n) == 1 && | |
| 486 | + n > 0 && pid == tmp_pid && lib_path[n] == '/') { | |
| 487 | + libbpf_strlcpy(path, lib_path + n, sizeof(path)); | |
| 484 | 488 | goto proceed; |
| 489 | + } | |
| 485 | 490 | |
| 486 | 491 | if (!realpath(lib_path, path)) { |
| 487 | 492 | pr_warn("usdt: failed to get absolute path of '%s' (err %s), using path as is...\n", |
| @@ -504,8 +509,11 @@ proceed: · 16 unmodified lines | ||
| 504 | 509 | * 7f5c6f5d1000-7f5c6f5d3000 rw-p 001c7000 08:04 21238613 /usr/lib64/libc-2.17.so |
| 505 | 510 | * 7f5c6f5d3000-7f5c6f5d8000 rw-p 00000000 00:00 0 |
| 506 | 511 | * 7f5c6f5d8000-7f5c6f5d9000 r-xp 00000000 103:01 362990598 /data/users/andriin/linux/tools/bpf/usdt/libhello_usdt.so |
| 512 | + * | |
| 513 | + * Some VMA names can be longer than the local buffer. Bound the | |
| 514 | + * writes, but still consume the rest of the line. | |
| 507 | 515 | */ |
| 508 | − while (fscanf(f, "%zx-%zx %s %zx %*s %*d%[^\n]\n", | |
| 516 | + while (fscanf(f, "%zx-%zx %15s %zx %*s %*d%4095[^\n]%*[^\n]\n", | |
| 509 | 517 | &seg_start, &seg_end, mode, &seg_off, line) == 5) { |
| 510 | 518 | void *tmp; |
| 511 | 519 | |
| @@ -21,7 +21,6 @@ test_lirc_mode2_user · 20 unmodified lines | ||
| 21 | 21 | flow_dissector_load |
| 22 | 22 | test_tcpnotify_user |
| 23 | 23 | test_libbpf |
| 24 | −xdping | |
| 25 | 24 | test_cpp |
| 26 | 25 | *.d |
| 27 | 26 | *.subskel.h |
| @@ -44,6 +44,12 @@ SKIP_LLVM ?= · 43 unmodified lines | ||
| 44 | 44 | SKIP_LIBBFD ?= |
| 45 | 45 | SKIP_CRYPTO ?= |
| 46 | 46 | |
| 47 | +# When BPF_STRICT_BUILD is 1, any BPF object, skeleton, test object, or | |
| 48 | +# benchmark compilation failure is fatal. Set to 0 to tolerate failures | |
| 49 | +# and continue building the remaining tests. | |
| 50 | +BPF_STRICT_BUILD ?= 1 | |
| 51 | +PERMISSIVE := $(filter 0,$(BPF_STRICT_BUILD)) | |
| 52 | + | |
| 47 | 53 | ifeq ($(srctree),) |
| 48 | 54 | srctree := $(patsubst %/,%,$(dir $(CURDIR))) |
| 49 | 55 | srctree := $(patsubst %/,%,$(dir $(srctree))) |
| @@ -51,19 +57,20 @@ srctree := $(patsubst %/,%,$(dir $(srctree))) · 1 unmodified line | ||
| 51 | 57 | srctree := $(patsubst %/,%,$(dir $(srctree))) |
| 52 | 58 | endif |
| 53 | 59 | |
| 54 | −CFLAGS += -g $(OPT_FLAGS) -rdynamic -std=gnu11 \ | |
| 60 | +COMMON_CFLAGS = -g $(OPT_FLAGS) -rdynamic -std=gnu11 \ | |
| 55 | 61 | -Wall -Werror -fno-omit-frame-pointer \ |
| 56 | 62 | -Wno-unused-but-set-variable \ |
| 57 | 63 | $(GENFLAGS) $(SAN_CFLAGS) $(LIBELF_CFLAGS) \ |
| 58 | 64 | -I$(CURDIR) -I$(INCLUDE_DIR) -I$(GENDIR) -I$(LIBDIR) \ |
| 59 | − -I$(TOOLSINCDIR) -I$(TOOLSARCHINCDIR) -I$(APIDIR) -I$(OUTPUT) | |
| 65 | + -I$(TOOLSINCDIR) -I$(TOOLSARCHINCDIR) -I$(APIDIR) -I$(OUTPUT) \ | |
| 66 | + -I$(CURDIR)/libarena/include | |
| 60 | 67 | LDFLAGS += $(SAN_LDFLAGS) |
| 61 | 68 | LDLIBS += $(LIBELF_LIBS) -lz -lrt -lpthread |
| 62 | 69 | |
| 63 | 70 | PCAP_CFLAGS := $(shell $(PKG_CONFIG) --cflags libpcap 2>/dev/null && echo "-DTRAFFIC_MONITOR=1") |
| 64 | 71 | PCAP_LIBS := $(shell $(PKG_CONFIG) --libs libpcap 2>/dev/null) |
| 65 | 72 | LDLIBS += $(PCAP_LIBS) |
| 66 | −CFLAGS += $(PCAP_CFLAGS) | |
| 73 | +CFLAGS += $(COMMON_CFLAGS) $(PCAP_CFLAGS) | |
| 67 | 74 | |
| 68 | 75 | # Some utility functions use LLVM libraries |
| 69 | 76 | jit_disasm_helpers.c-CFLAGS = $(LLVM_CFLAGS) |
| @@ -78,6 +85,12 @@ ifneq ($(shell $(CLANG) --target=bpf -mcpu=help 2>&1 | grep 'v4'),) · 8 unmodified lines | ||
| 78 | 85 | CLANG_CPUV4 := 1 |
| 79 | 86 | endif |
| 80 | 87 | |
| 88 | +# Check whether clang supports BPF address sanitizer (requires LLVM 22+) | |
| 89 | +CLANG_HAS_ARENA_ASAN := $(shell echo 'int x;' | \ | |
| 90 | + $(CLANG) --target=bpf -fsanitize=kernel-address \ | |
| 91 | + -mllvm -asan-shadow-addr-space=1 \ | |
| 92 | + -x c -c - -o /dev/null 2>/dev/null && echo 1) | |
| 93 | + | |
| 81 | 94 | # Order correspond to 'make run_tests' order |
| 82 | 95 | TEST_GEN_PROGS = test_verifier test_tag test_maps test_lru_map test_progs \ |
| 83 | 96 | test_sockmap \ |
| @@ -111,7 +124,6 @@ TEST_FILES = xsk_prereqs.sh $(wildcard progs/btf_dump_test_case_*.c) · 27 unmodified lines | ||
| 111 | 124 | # Order correspond to 'make run_tests' order |
| 112 | 125 | TEST_PROGS := test_kmod.sh \ |
| 113 | 126 | test_lirc_mode2.sh \ |
| 114 | − test_xdping.sh \ | |
| 115 | 127 | test_bpftool_build.sh \ |
| 116 | 128 | test_doc_build.sh \ |
| 117 | 129 | test_xsk.sh \ |
| @@ -134,7 +146,6 @@ TEST_GEN_PROGS_EXTENDED = \ · 16 unmodified lines | ||
| 134 | 146 | xdp_features \ |
| 135 | 147 | xdp_hw_metadata \ |
| 136 | 148 | xdp_synproxy \ |
| 137 | − xdping \ | |
| 138 | 149 | xskxceiver |
| 139 | 150 | |
| 140 | 151 | TEST_GEN_FILES += $(TEST_KMODS) liburandom_read.so urandom_read sign-file uprobe_multi |
| @@ -153,12 +164,13 @@ override define CLEAN · 12 unmodified lines | ||
| 153 | 164 | $(Q)$(RM) -r $(TEST_KMODS) |
| 154 | 165 | $(Q)$(RM) -r $(EXTRA_CLEAN) |
445 more lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 96 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 51 lines aren’t shown here.
This diff is large: its 63 lines aren’t shown here.
This diff is large: its 61 lines aren’t shown here.
This diff is large: its 85 lines aren’t shown here.
This diff is large: its 299 lines aren’t shown here.
This diff is large: its 70 lines aren’t shown here.
This diff is large: its 1125 lines aren’t shown here.
This diff is large: its 80 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 47 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 98 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 7 lines aren’t shown here.
Binary file not shown.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 93 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 45 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 85 lines aren’t shown here.
This diff is large: its 102 lines aren’t shown here.
This diff is large: its 82 lines aren’t shown here.
This diff is large: its 94 lines aren’t shown here.
This diff is large: its 84 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 139 lines aren’t shown here.
This diff is large: its 259 lines aren’t shown here.
This diff is large: its 53 lines aren’t shown here.
This diff is large: its 210 lines aren’t shown here.
This diff is large: its 670 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 969 lines aren’t shown here.
This diff is large: its 195 lines aren’t shown here.
This diff is large: its 554 lines aren’t shown here.
This diff is large: its 904 lines aren’t shown here.
This diff is large: its 53 lines aren’t shown here.
This diff is large: its 1048 lines aren’t shown here.
This diff is large: its 235 lines aren’t shown here.
This diff is large: its 74 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 125 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 76 lines aren’t shown here.
This diff is large: its 125 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 38 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 50 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 254 lines aren’t shown here.
This diff is large: its 94 lines aren’t shown here.
This diff is large: its 114 lines aren’t shown here.
This diff is large: its 244 lines aren’t shown here.
This diff is large: its 98 lines aren’t shown here.
This diff is large: its 163 lines aren’t shown here.
This diff is large: its 140 lines aren’t shown here.
This diff is large: its 196 lines aren’t shown here.
This diff is large: its 136 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 120 lines aren’t shown here.
This diff is large: its 184 lines aren’t shown here.
This diff is large: its 92 lines aren’t shown here.
This diff is large: its 1136 lines aren’t shown here.
This diff is large: its 143 lines aren’t shown here.
This diff is large: its 69 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 140 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 205 lines aren’t shown here.
This diff is large: its 70 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 200 lines aren’t shown here.
This diff is large: its 961 lines aren’t shown here.
This diff is large: its 122 lines aren’t shown here.
This diff is large: its 50 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 71 lines aren’t shown here.
This diff is large: its 119 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 70 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 15 lines aren’t shown here.
This diff is large: its 75 lines aren’t shown here.
This diff is large: its 69 lines aren’t shown here.
This diff is large: its 75 lines aren’t shown here.
This diff is large: its 71 lines aren’t shown here.
This diff is large: its 22 lines aren’t shown here.
This diff is large: its 25 lines aren’t shown here.
This diff is large: its 50 lines aren’t shown here.
This diff is large: its 81 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 45 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 44 lines aren’t shown here.
This diff is large: its 45 lines aren’t shown here.
This diff is large: its 193 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 119 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 82 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 81 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 51 lines aren’t shown here.
This diff is large: its 78 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 34 lines aren’t shown here.
This diff is large: its 41 lines aren’t shown here.
This diff is large: its 107 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 461 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 249 lines aren’t shown here.
This diff is large: its 37 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 274 lines aren’t shown here.
This diff is large: its 115 lines aren’t shown here.
This diff is large: its 167 lines aren’t shown here.
This diff is large: its 136 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 57 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 45 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 33 lines aren’t shown here.
This diff is large: its 146 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 27 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 60 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 12 lines aren’t shown here.
This diff is large: its 176 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 113 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 28 lines aren’t shown here.
This diff is large: its 50 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 93 lines aren’t shown here.
This diff is large: its 40 lines aren’t shown here.
This diff is large: its 26 lines aren’t shown here.
This diff is large: its 13 lines aren’t shown here.
This diff is large: its 215 lines aren’t shown here.
This diff is large: its 19 lines aren’t shown here.
This diff is large: its 42 lines aren’t shown here.
This diff is large: its 44 lines aren’t shown here.
This diff is large: its 66 lines aren’t shown here.
This diff is large: its 32 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 81 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 18 lines aren’t shown here.
This diff is large: its 175 lines aren’t shown here.
This diff is large: its 50 lines aren’t shown here.
This diff is large: its 98 lines aren’t shown here.
This diff is large: its 35 lines aren’t shown here.
This diff is large: its 39 lines aren’t shown here.
This diff is large: its 11 lines aren’t shown here.
This diff is large: its 24 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.
This diff is large: its 54 lines aren’t shown here.
This diff is large: its 58 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 108 lines aren’t shown here.
This diff is large: its 36 lines aren’t shown here.
This diff is large: its 50 lines aren’t shown here.
This diff is large: its 120 lines aren’t shown here.
This diff is large: its 108 lines aren’t shown here.
This diff is large: its 54 lines aren’t shown here.
This diff is large: its 448 lines aren’t shown here.
This diff is large: its 186 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 8 lines aren’t shown here.
This diff is large: its 45 lines aren’t shown here.
This diff is large: its 23 lines aren’t shown here.
This diff is large: its 77 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 21 lines aren’t shown here.
This diff is large: its 648 lines aren’t shown here.
This diff is large: its 184 lines aren’t shown here.
This diff is large: its 42 lines aren’t shown here.
This diff is large: its 93 lines aren’t shown here.
This diff is large: its 54 lines aren’t shown here.
This diff is large: its 20 lines aren’t shown here.
This diff is large: its 242 lines aren’t shown here.
This diff is large: its 49 lines aren’t shown here.
This diff is large: its 125 lines aren’t shown here.
This diff is large: its 16 lines aren’t shown here.
This diff is large: its 104 lines aren’t shown here.
This diff is large: its 29 lines aren’t shown here.
This diff is large: its 7 lines aren’t shown here.
This diff is large: its 30 lines aren’t shown here.
This diff is large: its 6 lines aren’t shown here.
This diff is large: its 10 lines aren’t shown here.
This diff is large: its 153 lines aren’t shown here.
This diff is large: its 31 lines aren’t shown here.
This diff is large: its 67 lines aren’t shown here.
This diff is large: its 9 lines aren’t shown here.
This diff is large: its 113 lines aren’t shown here.
This diff is large: its 255 lines aren’t shown here.
This diff is large: its 14 lines aren’t shown here.