Pull bpf updates from Daniel Borkmann: "Major changes: - Redesign the verifier error reporting: failures now carry source and instruction annotations along with the causal event history that led to them, making program rejections far easier to debug and repair (Kumar Kartikeya Dwivedi) - Add arena argument support to kfuncs and struct_ops through the new __arena and __arena__nullable suffixes (Tejun Heo, Puranjay Mohan, Kumar Kartikeya Dwivedi, Ihor Solodrai) - Signed BPF program loader rework to accommodate both BPF and security community needs where the kernel runs the signature verification at BPF_PROG_LOAD time before the LSM admission hook (Daniel Borkmann) - Add a set of ksock kfuncs which let BPF LSM and syscall programs create, connect and send on UDP sockets in order to emit telemetry data (Mahe Tardy) - Unify helper and kfunc call argument verification and classify kfunc arguments purely from BTF into a generated bpf_func_proto which is computed once at add-call time (Amery Hung) Other features and fixes: - Enable EXECMEM_ROX_CACHE for BPF allocations on x86 (Mike Rapoport) - Add bidirectional VLAN support to bpf_fib_lookup() through the new BPF_FIB_LOOKUP_VLAN and BPF_FIB_LOOKUP_VLAN_INPUT flags (Avinash Duduskar) - Infer zext_dst from static register liveness analysis to fix 32-bit zero-extension semantics, and remove the artificial limitations on pointer types eligible for spilling (Eduard Zingerman) - Inline the numeric open-coded iterator kfuncs so that bpf_for() loops no longer pay a kfunc call on every iteration (Puranjay Mohan) - Add an arena-based bitmap data structure to libarena along with serial and parallel selftests (Emil Tsalapatis) - Teach resolve_btfids to discover kfuncs from the kernel's BTF ID sets and to emit kfunc BTF decl tags, reducing the kernel build's dependency on pahole features (Ihor Solodrai) - Add BPF_F_ADJ_ROOM_DECAP_* flags to bpf_skb_adjust_room() so that tunnel decapsulation can update the GSO and encapsulation state of the skb (Nick Hudson) - Fix the ring buffer pending_pos walk and the available-data accounting on 32-bit position wrap (Israel Téllez García) - Add memory usage accounting for arena maps and fix an mmap_lock deadlock on arena lock failure (Jiayuan Chen) - Add tracing_multi link info support to the kernel UAPI and bpftool, and refactor the stack map code to run with preemption disabled (Jiri Olsa) - Support BPF_F_EGRESS in bpf_redirect_peer() to emit the skb in the egress direction of the target's peer device (Jordan Rife) - Add a KF_SPINLOCK_SAFE kfunc flag so that providers, in particular modules, can declare kfuncs safe to call under bpf_spin_lock instead of relying on the verifier's hard-coded allowlist (Kaitao Cheng) - Introduce global percpu data for BPF programs with libbpf probing and bpftool skeleton support, and stop exposing uninitialized kernel heap memory when copying per-CPU map values (Leon Hwang) - Add s390 JIT support for load-acquire and store-release instructions (Maxim Khmelevskii) - Fix a CFI mismatch in the task work callback and an arm64 KASAN false positive after bpf_throw() (Mykyta Yatsenko) - Reject writes through untrusted BTF pointers and bound the rdonly/rdwr_buf_size kfunc arguments (Nicholas Dudar) - Invalidate RCU pointers only after the final spin unlock and account for preempt and IRQ disabled regions as overlapping RCU protection (Ning Ding) - Support mixing bpf2bpf calls and tail calls on RV64, add signed operations and 32-bit atomics to the RV32 JIT, and add timed may_goto support (Pu Lehui, Kuan-Wei Chiu, Feng Jiang) - Fix a use-after-free on mm_struct in bpf_find_vma() for foreign tasks and an mmap_lock leak in the irq_work path (Sanghyun Park) - Populate mmap-able BPF array map memory lazily which makes mmap() O(1) instead of proportional to the map size (Song Liu) - Introduce a jit_required flag and reject programs with inlined helpers when no JIT is available, where the interpreter would otherwise jump into an invalid address (Tiezhu Yang) - Fix the x86 JIT per-CPU address resolution into an extended register where the REX prefix dropped the high destination register bit (Vineet Gupta) - Reject MEM_ALLOC BTF accesses past object bounds, arena frees below the arena base, and mixed arena and ordinary atomic paths (Yiyang Chen) - Fix the trampoline handling of 128-bit arguments and of return values larger than 8 bytes (Yonghong Song) - Ensure that any fault prone load is rewritten with exception table handling, and fix the arena load-acquire and atomic fetch handling in the x86, arm64, riscv and s390 JITs (Daniel Borkmann) - Many more fixes and cleanups across the verifier, arena, trampolines, sockmap, cgroup, ring buffer, x86/arm64/riscv/s390 JITs, libbpf, bpftool, resolve_btfids and selftests" * tag 'bpf-next-7.3' of git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next: (373 commits) selftests/bpf: Add tests for a store on a fault prone qdisc pointer selftests/bpf: Add tests for fault prone loads out of RCU pointers selftests/bpf: Add tests for pointer type merge at a shared load selftests/bpf: Remove duplicate copies of the arena spinlock qnodes selftests/bpf: Retry stat generation in cgroup_iter_memcg selftests/bpf: Test pseudo-function policy diagnostics bpf: Distinguish function references in policy diagnostics bpf: Preserve source attribution without source text selftests/bpf: Test kfunc argument diagnostics bpf: Correct kfunc argument diagnostics bpf: Use canonical stack argument names in diagnostics bpf: Preserve R0 lineage across helper calls selftests/bpf: Exercise negative optlen in cgroup getsockopt hook bpf: Reject negative optlen in cgroup getsockopt hook selftests/bpf: tc_tunnel - validate decap GSO and encapsulation state bpf: Clear decap state on skb_adjust_room shrink path bpf: Allow new DECAP flags and add guard rails bpf: Add BPF_F_ADJ_ROOM_DECAP_* flags for tunnel decapsulation bpf: Refactor masks for ADJ_ROOM flags and encap validation bpf: Name the enum for BPF_FUNC_skb_adjust_room flags ...
Showing the changes to include/uapi/linux (1 of 272 files). Show all
Unified · Split
| @@ -3038,8 +3038,34 @@ union bpf_attr { · 3037 unmodified lines | |||
| 3038 | * | 3038 | * |
| 3039 | * * **BPF_F_ADJ_ROOM_DECAP_L3_IPV4**, | 3039 | * * **BPF_F_ADJ_ROOM_DECAP_L3_IPV4**, |
| 3040 | * **BPF_F_ADJ_ROOM_DECAP_L3_IPV6**: | 3040 | * **BPF_F_ADJ_ROOM_DECAP_L3_IPV6**: |
| 3041 | * Indicate the new IP header version after decapsulating the outer | 3041 | * Indicate the new IP header version after decapsulating the |
| 3042 | * IP header. Used when the inner and outer IP versions are different. | 3042 | * outer IP header. Used when the inner and outer IP versions |
| 3043 | * are different. These flags only trigger a protocol change | ||
| 3044 | * without clearing any tunnel-specific GSO flags. | ||
| 3045 | * | ||
| 3046 | * * **BPF_F_ADJ_ROOM_DECAP_L4_GRE**: | ||
| 3047 | * Clear GRE tunnel GSO flags (SKB_GSO_GRE and SKB_GSO_GRE_CSUM) | ||
| 3048 | * when decapsulating a GRE tunnel. | ||
| 3049 | * | ||
| 3050 | * * **BPF_F_ADJ_ROOM_DECAP_L4_UDP**: | ||
| 3051 | * Clear UDP tunnel GSO flags (SKB_GSO_UDP_TUNNEL and | ||
| 3052 | * SKB_GSO_UDP_TUNNEL_CSUM) when decapsulating a UDP tunnel. | ||
| 3053 | * | ||
| 3054 | * * **BPF_F_ADJ_ROOM_DECAP_IPXIP4**: | ||
| 3055 | * Clear IPIP/SIT tunnel GSO flag (SKB_GSO_IPXIP4) when decapsulating | ||
| 3056 | * a tunnel with an outer IPv4 header (IPv4-in-IPv4 or IPv6-in-IPv4). | ||
| 3057 | * | ||
| 3058 | * * **BPF_F_ADJ_ROOM_DECAP_IPXIP6**: | ||
| 3059 | * Clear IPv6 encapsulation tunnel GSO flag (SKB_GSO_IPXIP6) when | ||
| 3060 | * decapsulating a tunnel with an outer IPv6 header (IPv6-in-IPv6 | ||
| 3061 | * or IPv4-in-IPv6). | ||
| 3062 | * | ||
| 3063 | * When using the decapsulation flags above, the skb->encapsulation | ||
| 3064 | * flag is automatically cleared if all tunnel-specific GSO flags | ||
| 3065 | * (SKB_GSO_UDP_TUNNEL, SKB_GSO_UDP_TUNNEL_CSUM, SKB_GSO_GRE, | ||
| 3066 | * SKB_GSO_GRE_CSUM, SKB_GSO_IPXIP4, SKB_GSO_IPXIP6) have been | ||
| 3067 | * removed from the packet. This handles cases where all tunnel | ||
| 3068 | * layers have been decapsulated. | ||
| 3043 | * | 3069 | * |
| 3044 | * A call to this helper is susceptible to change the underlying | 3070 | * A call to this helper is susceptible to change the underlying |
| 3045 | * packet buffer. Therefore, at load time, all checks on pointers | 3071 | * packet buffer. Therefore, at load time, all checks on pointers |
| @@ -3532,6 +3558,47 @@ union bpf_attr { · 486 unmodified lines | |||
| 3532 | * Use the mark present in *params*->mark for the fib lookup. | 3558 | * Use the mark present in *params*->mark for the fib lookup. |
| 3533 | * This option should not be used with BPF_FIB_LOOKUP_DIRECT, | 3559 | * This option should not be used with BPF_FIB_LOOKUP_DIRECT, |
| 3534 | * as it only has meaning for full lookups. | 3560 | * as it only has meaning for full lookups. |
| 3561 | * **BPF_FIB_LOOKUP_VLAN** | ||
| 3562 | * If the fib lookup resolves to a VLAN device whose | ||
| 3563 | * parent is a real (non-VLAN) device, set | ||
| 3564 | * *params*->h_vlan_proto and *params*->h_vlan_TCI from | ||
| 3565 | * the VLAN device and replace *params*->ifindex with the | ||
| 3566 | * parent's ifindex. *params*->h_vlan_TCI carries the VID | ||
| 3567 | * only, with PCP and DEI bits zero; a consumer wanting to | ||
| 3568 | * set egress priority writes PCP itself. *params*->smac is | ||
| 3569 | * the VLAN device's own address, which can differ from the | ||
| 3570 | * parent's. Only the immediate parent is resolved; if it | ||
| 3571 | * is itself a VLAN device (QinQ) or in another namespace, | ||
| 3572 | * the egress cannot be reduced to a physical device plus | ||
| 3573 | * one tag and the lookup returns | ||
| 3574 | * **BPF_FIB_LKUP_RET_VLAN_FAILURE** with *params*->ifindex | ||
| 3575 | * left at the input. To obtain the VLAN device's own | ||
| 3576 | * ifindex, repeat the lookup without | ||
| 3577 | * **BPF_FIB_LOOKUP_VLAN**, re-initializing *params* | ||
| 3578 | * first: output fields overwrite the inputs they share | ||
| 3579 | * storage with. The swap and the vlan fields | ||
| 3580 | * are written only on success; other output fields keep | ||
| 3581 | * the helper's existing behaviour, so a frag-needed result | ||
| 3582 | * still reports the route mtu in *params*->mtu_result. | ||
| 3583 | * This flag is only valid for XDP programs; tc programs | ||
| 3584 | * receive -EINVAL since they can redirect to the VLAN | ||
| 3585 | * device directly. | ||
| 3586 | * **BPF_FIB_LOOKUP_VLAN_INPUT** | ||
| 3587 | * Treat *params*->h_vlan_proto and *params*->h_vlan_TCI | ||
| 3588 | * as an input VLAN tag and run the lookup as if ingress | ||
| 3589 | * had happened on the VLAN subinterface carrying that tag | ||
| 3590 | * on *params*->ifindex. The VID is the low 12 bits of | ||
| 3591 | * *params*->h_vlan_TCI; *params*->h_vlan_proto must be | ||
| 3592 | * ETH_P_8021Q or ETH_P_8021AD in network byte order, else | ||
| 3593 | * **-EINVAL**. If *params*->ifindex is itself a VLAN | ||
| 3594 | * device, its inner (QinQ) subinterface is matched; for a | ||
| 3595 | * bond or team, pass the master's ifindex. An unmatched | ||
| 3596 | * tag, a down device, or one in another namespace returns | ||
| 3597 | * **BPF_FIB_LKUP_RET_NOT_FWDED**, mirroring real ingress. | ||
| 3598 | * A VID of 0 is looked up literally, so do not set this | ||
| 3599 | * flag for priority-tagged frames. Cannot be combined with | ||
| 3600 | * **BPF_FIB_LOOKUP_TBID** or **BPF_FIB_LOOKUP_OUTPUT** | ||
| 3601 | * (returns **-EINVAL**). | ||
| 3535 | * | 3602 | * |
| 3536 | * *ctx* is either **struct xdp_md** for XDP programs or | 3603 | * *ctx* is either **struct xdp_md** for XDP programs or |
| 3537 | * **struct sk_buff** tc cls_act programs. | 3604 | * **struct sk_buff** tc cls_act programs. |
| @@ -4694,6 +4761,7 @@ union bpf_attr { · 1156 unmodified lines | |||
| 4694 | * * **BPF_RB_RING_SIZE**: The size of ring buffer. | 4761 | * * **BPF_RB_RING_SIZE**: The size of ring buffer. |
| 4695 | * * **BPF_RB_CONS_POS**: Consumer position (can wrap around). | 4762 | * * **BPF_RB_CONS_POS**: Consumer position (can wrap around). |
| 4696 | * * **BPF_RB_PROD_POS**: Producer(s) position (can wrap around). | 4763 | * * **BPF_RB_PROD_POS**: Producer(s) position (can wrap around). |
| 4764 | * * **BPF_RB_OVERWRITE_POS**: Overwrite position (can wrap around). | ||
| 4697 | * | 4765 | * |
| 4698 | * Data returned is just a momentary snapshot of actual values | 4766 | * Data returned is just a momentary snapshot of actual values |
| 4699 | * and could be inaccurate, so this facility should be used to | 4767 | * and could be inaccurate, so this facility should be used to |
| @@ -5079,17 +5147,19 @@ union bpf_attr { · 379 unmodified lines | |||
| 5079 | * Description | 5147 | * Description |
| 5080 | * Redirect the packet to another net device of index *ifindex*. | 5148 | * Redirect the packet to another net device of index *ifindex*. |
| 5081 | * This helper is somewhat similar to **bpf_redirect**\ (), except | 5149 | * This helper is somewhat similar to **bpf_redirect**\ (), except |
| 5082 | * that the redirection happens to the *ifindex*' peer device and | 5150 | * that the redirection happens to the *ifindex*' peer device. If |
| 5083 | * the netns switch takes place from ingress to ingress without | 5151 | * *flags* is 0, the netns switch takes place from ingress to |
| 5084 | * going through the CPU's backlog queue. | 5152 | * ingress without going through the CPU's backlog queue. If the |
| 5153 | * **BPF_F_EGRESS** flag is provided then redirection happens in | ||
| 5154 | * the egress direction of the peer device. | ||
| 5085 | * | 5155 | * |
| 5086 | * *skb*\ **->mark** and *skb*\ **->tstamp** are not cleared during | 5156 | * *skb*\ **->mark** and *skb*\ **->tstamp** are not cleared during |
| 5087 | * the netns switch. | 5157 | * the netns switch. |
| 5088 | * | 5158 | * |
| 5089 | * The *flags* argument is reserved and must be 0. The helper is | 5159 | * If the *flags* argument is 0, the helper is currently only |
| 5090 | * currently only supported for tc BPF program types at the | 5160 | * supported for tc BPF program types at the ingress hook and for |
| 5091 | * ingress hook and for veth and netkit target device types. The | 5161 | * veth and netkit target device types. The peer device must reside |
| 5092 | * peer device must reside in a different network namespace. | 5162 | * in a different network namespace. |
| 5093 | * Return | 5163 | * Return |
| 5094 | * The helper returns **TC_ACT_REDIRECT** on success or | 5164 | * The helper returns **TC_ACT_REDIRECT** on success or |
| 5095 | * **TC_ACT_SHOT** on error. | 5165 | * **TC_ACT_SHOT** on error. |
| @@ -6239,7 +6309,7 @@ enum { · 1143 unmodified lines | |||
| 6239 | }; | 6309 | }; |
| 6240 | 6310 | ||
| 6241 | /* BPF_FUNC_skb_adjust_room flags. */ | 6311 | /* BPF_FUNC_skb_adjust_room flags. */ |
| 6242 | enum { | 6312 | enum bpf_adj_room_flags { |
| 6243 | BPF_F_ADJ_ROOM_FIXED_GSO = (1ULL << 0), | 6313 | BPF_F_ADJ_ROOM_FIXED_GSO = (1ULL << 0), |
| 6244 | BPF_F_ADJ_ROOM_ENCAP_L3_IPV4 = (1ULL << 1), | 6314 | BPF_F_ADJ_ROOM_ENCAP_L3_IPV4 = (1ULL << 1), |
| 6245 | BPF_F_ADJ_ROOM_ENCAP_L3_IPV6 = (1ULL << 2), | 6315 | BPF_F_ADJ_ROOM_ENCAP_L3_IPV6 = (1ULL << 2), |
| @@ -6249,6 +6319,10 @@ enum { · 3 unmodified lines | |||
| 6249 | BPF_F_ADJ_ROOM_ENCAP_L2_ETH = (1ULL << 6), | 6319 | BPF_F_ADJ_ROOM_ENCAP_L2_ETH = (1ULL << 6), |
| 6250 | BPF_F_ADJ_ROOM_DECAP_L3_IPV4 = (1ULL << 7), | 6320 | BPF_F_ADJ_ROOM_DECAP_L3_IPV4 = (1ULL << 7), |
| 6251 | BPF_F_ADJ_ROOM_DECAP_L3_IPV6 = (1ULL << 8), | 6321 | BPF_F_ADJ_ROOM_DECAP_L3_IPV6 = (1ULL << 8), |
| 6322 | BPF_F_ADJ_ROOM_DECAP_L4_GRE = (1ULL << 9), | ||
| 6323 | BPF_F_ADJ_ROOM_DECAP_L4_UDP = (1ULL << 10), | ||
| 6324 | BPF_F_ADJ_ROOM_DECAP_IPXIP4 = (1ULL << 11), | ||
| 6325 | BPF_F_ADJ_ROOM_DECAP_IPXIP6 = (1ULL << 12), | ||
| 6252 | }; | 6326 | }; |
| 6253 | 6327 | ||
| 6254 | enum { | 6328 | enum { |
| @@ -6336,9 +6410,10 @@ enum { · 81 unmodified lines | |||
| 6336 | /* Flags for bpf_redirect and bpf_redirect_map helpers */ | 6410 | /* Flags for bpf_redirect and bpf_redirect_map helpers */ |
| 6337 | enum { | 6411 | enum { |
| 6338 | BPF_F_INGRESS = (1ULL << 0), /* used for skb path */ | 6412 | BPF_F_INGRESS = (1ULL << 0), /* used for skb path */ |
| 6413 | BPF_F_EGRESS = (1ULL << 1), /* used for skb path */ | ||
| 6339 | BPF_F_BROADCAST = (1ULL << 3), /* used for XDP path */ | 6414 | BPF_F_BROADCAST = (1ULL << 3), /* used for XDP path */ |
| 6340 | BPF_F_EXCLUDE_INGRESS = (1ULL << 4), /* used for XDP path */ | 6415 | BPF_F_EXCLUDE_INGRESS = (1ULL << 4), /* used for XDP path */ |
| 6341 | #define BPF_F_REDIRECT_FLAGS (BPF_F_INGRESS | BPF_F_BROADCAST | BPF_F_EXCLUDE_INGRESS) | 6416 | #define BPF_F_REDIRECT_FLAGS (BPF_F_INGRESS | BPF_F_EGRESS | BPF_F_BROADCAST | BPF_F_EXCLUDE_INGRESS) |
| 6342 | }; | 6417 | }; |
| 6343 | 6418 | ||
| 6344 | #define __bpf_md_ptr(type, name) \ | 6419 | #define __bpf_md_ptr(type, name) \ |
| @@ -6839,6 +6914,15 @@ struct bpf_link_info { · 494 unmodified lines | |||
| 6839 | __u32 flags; | 6914 | __u32 flags; |
| 6840 | __u32 pid; | 6915 | __u32 pid; |
| 6841 | } uprobe_multi; | 6916 | } uprobe_multi; |
| 6917 | struct { | ||
| 6918 | __u32 attach_type; | ||
| 6919 | __u32 count; /* in/out: tracing_multi target count */ | ||
| 6920 | __u32 btf_obj_id; | ||
| 6921 | __u32 :32; | ||
| 6922 | __aligned_u64 ids; | ||
| 6923 | __aligned_u64 addrs; | ||
| 6924 | __aligned_u64 cookies; | ||
| 6925 | } tracing_multi; | ||
| 6842 | struct { | 6926 | struct { |
| 6843 | __u32 type; /* enum bpf_perf_event_type */ | 6927 | __u32 type; /* enum bpf_perf_event_type */ |
| 6844 | __u32 :32; | 6928 | __u32 :32; |
| @@ -7327,6 +7411,8 @@ enum { · 482 unmodified lines | |||
| 7327 | BPF_FIB_LOOKUP_TBID = (1U << 3), | 7411 | BPF_FIB_LOOKUP_TBID = (1U << 3), |
| 7328 | BPF_FIB_LOOKUP_SRC = (1U << 4), | 7412 | BPF_FIB_LOOKUP_SRC = (1U << 4), |
| 7329 | BPF_FIB_LOOKUP_MARK = (1U << 5), | 7413 | BPF_FIB_LOOKUP_MARK = (1U << 5), |
| 7414 | BPF_FIB_LOOKUP_VLAN = (1U << 6), | ||
| 7415 | BPF_FIB_LOOKUP_VLAN_INPUT = (1U << 7), | ||
| 7330 | }; | 7416 | }; |
| 7331 | 7417 | ||
| 7332 | enum { | 7418 | enum { |
| @@ -7340,6 +7426,7 @@ enum { · 7 unmodified lines | |||
| 7340 | BPF_FIB_LKUP_RET_NO_NEIGH, /* no neighbor entry for nh */ | 7426 | BPF_FIB_LKUP_RET_NO_NEIGH, /* no neighbor entry for nh */ |
| 7341 | BPF_FIB_LKUP_RET_FRAG_NEEDED, /* fragmentation required to fwd */ | 7427 | BPF_FIB_LKUP_RET_FRAG_NEEDED, /* fragmentation required to fwd */ |
| 7342 | BPF_FIB_LKUP_RET_NO_SRC_ADDR, /* failed to derive IP src addr */ | 7428 | BPF_FIB_LKUP_RET_NO_SRC_ADDR, /* failed to derive IP src addr */ |
| 7429 | BPF_FIB_LKUP_RET_VLAN_FAILURE, /* VLAN egress, parent unresolvable */ | ||
| 7343 | }; | 7430 | }; |
| 7344 | 7431 | ||
| 7345 | struct bpf_fib_lookup { | 7432 | struct bpf_fib_lookup { |
| @@ -7393,7 +7480,13 @@ struct bpf_fib_lookup { · 47 unmodified lines | |||
| 7393 | 7480 | ||
| 7394 | union { | 7481 | union { |
| 7395 | struct { | 7482 | struct { |
| 7396 | /* output */ | 7483 | /* |
| 7484 | * output with BPF_FIB_LOOKUP_VLAN: set from the | ||
| 7485 | * resolved egress VLAN device (see the flag); zeroed | ||
| 7486 | * on other successful lookups. input with | ||
| 7487 | * BPF_FIB_LOOKUP_VLAN_INPUT: the VLAN tag to scope | ||
| 7488 | * the lookup by. | ||
| 7489 | */ | ||
| 7397 | __be16 h_vlan_proto; | 7490 | __be16 h_vlan_proto; |
| 7398 | __be16 h_vlan_TCI; | 7491 | __be16 h_vlan_TCI; |
| 7399 | }; | 7492 | }; |