History

Merge tag 'net-next-5.10' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next

Pull networking updates from Jakub Kicinski: - Add redirect_neigh() BPF packet redirect helper, allowing to limit stack traversal in common container configs and improving TCP back-pressure. Daniel reports ~10Gbps => ~15Gbps single stream TCP performance gain. - Expand netlink policy support and improve policy export to user space. (Ge)netlink core performs request validation according to declared policies. Expand the expressiveness of those policies (min/max length and bitmasks). Allow dumping policies for particular commands. This is used for feature discovery by user space (instead of kernel version parsing or trial and error). - Support IGMPv3/MLDv2 multicast listener discovery protocols in bridge. - Allow more than 255 IPv4 multicast interfaces. - Add support for Type of Service (ToS) reflection in SYN/SYN-ACK packets of TCPv6. - In Multi-patch TCP (MPTCP) support concurrent transmission of data on multiple subflows in a load balancing scenario. Enhance advertising addresses via the RM_ADDR/ADD_ADDR options. - Support SMC-Dv2 version of SMC, which enables multi-subnet deployments. - Allow more calls to same peer in RxRPC. - Support two new Controller Area Network (CAN) protocols - CAN-FD and ISO 15765-2:2016. - Add xfrm/IPsec compat layer, solving the 32bit user space on 64bit kernel problem. - Add TC actions for implementing MPLS L2 VPNs. - Improve nexthop code - e.g. handle various corner cases when nexthop objects are removed from groups better, skip unnecessary notifications and make it easier to offload nexthops into HW by converting to a blocking notifier. - Support adding and consuming TCP header options by BPF programs, opening the doors for easy experimental and deployment-specific TCP option use. - Reorganize TCP congestion control (CC) initialization to simplify life of TCP CC implemented in BPF. - Add support for shipping BPF programs with the kernel and loading them early on boot via the User Mode Driver mechanism, hence reusing all the user space infra we have. - Support sleepable BPF programs, initially targeting LSM and tracing. - Add bpf_d_path() helper for returning full path for given 'struct path'. - Make bpf_tail_call compatible with bpf-to-bpf calls. - Allow BPF programs to call map_update_elem on sockmaps. - Add BPF Type Format (BTF) support for type and enum discovery, as well as support for using BTF within the kernel itself (current use is for pretty printing structures). - Support listing and getting information about bpf_links via the bpf syscall. - Enhance kernel interfaces around NIC firmware update. Allow specifying overwrite mask to control if settings etc. are reset during update; report expected max time operation may take to users; support firmware activation without machine reboot incl. limits of how much impact reset may have (e.g. dropping link or not). - Extend ethtool configuration interface to report IEEE-standard counters, to limit the need for per-vendor logic in user space. - Adopt or extend devlink use for debug, monitoring, fw update in many drivers (dsa loop, ice, ionic, sja1105, qed, mlxsw, mv88e6xxx, dpaa2-eth). - In mlxsw expose critical and emergency SFP module temperature alarms. Refactor port buffer handling to make the defaults more suitable and support setting these values explicitly via the DCBNL interface. - Add XDP support for Intel's igb driver. - Support offloading TC flower classification and filtering rules to mscc_ocelot switches. - Add PTP support for Marvell Octeontx2 and PP2.2 hardware, as well as fixed interval period pulse generator and one-step timestamping in dpaa-eth. - Add support for various auth offloads in WiFi APs, e.g. SAE (WPA3) offload. - Add Lynx PHY/PCS MDIO module, and convert various drivers which have this HW to use it. Convert mvpp2 to split PCS. - Support Marvell Prestera 98DX3255 24-port switch ASICs, as well as 7-port Mediatek MT7531 IP. - Add initial support for QCA6390 and IPQ6018 in ath11k WiFi driver, and wcn3680 support in wcn36xx. - Improve performance for packets which don't require much offloads on recent Mellanox NICs by 20% by making multiple packets share a descriptor entry. - Move chelsio inline crypto drivers (for TLS and IPsec) from the crypto subtree to drivers/net. Move MDIO drivers out of the phy directory. - Clean up a lot of W=1 warnings, reportedly the actively developed subsections of networking drivers should now build W=1 warning free. - Make sure drivers don't use in_interrupt() to dynamically adapt their code. Convert tasklets to use new tasklet_setup API (sadly this conversion is not yet complete). * tag 'net-next-5.10' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next: (2583 commits) Revert "bpfilter: Fix build error with CONFIG_BPFILTER_UMH" net, sockmap: Don't call bpf_prog_put() on NULL pointer bpf, selftest: Fix flaky tcp_hdr_options test when adding addr to lo bpf, sockmap: Add locking annotations to iterator netfilter: nftables: allow re-computing sctp CRC-32C in 'payload' statements net: fix pos incrementment in ipv6_route_seq_next net/smc: fix invalid return code in smcd_new_buf_create() net/smc: fix valid DMBE buffer sizes net/smc: fix use-after-free of delayed events bpfilter: Fix build error with CONFIG_BPFILTER_UMH cxgb4/ch_ipsec: Replace the module name to ch_ipsec from chcr net: sched: Fix suspicious RCU usage while accessing tcf_tunnel_info bpf: Fix register equivalence tracking. rxrpc: Fix loss of final ack on shutdown rxrpc: Fix bundle counting for exclusive connections netfilter: restore NF_INET_NUMHOOKS ibmveth: Identify ingress large send packets. ibmveth: Switch order of ibmveth_helper calls. cxgb4: handle 4-tuple PEDIT to NAT mode translation selftests: Add VRF route leaking tests ...

Linus Torvalds committed 6 years ago

Commit 9ff9b0d2 parents 840e5bb 105faa82301 files · +130475 −51396Browse files

Showing the changes to kernel (45 of 2301 files). Show all

Unified · Split

45 files
  1. kernel/Makefile +1 −1
  2. kernel/bpf/Makefile +3 −0
  3. kernel/bpf/arraymap.c +84 −18
  4. kernel/bpf/bpf_inode_storage.c +272 −0
  5. kernel/bpf/bpf_iter.c +60 −2
  6. kernel/bpf/bpf_local_storage.c +600 −0
  7. kernel/bpf/bpf_lsm.c +20 −1
  8. kernel/bpf/bpf_struct_ops.c +2 −4
  9. kernel/bpf/btf.c +1054 −167
  10. kernel/bpf/core.c +21 −8
  11. kernel/bpf/cpumap.c +6 −11
  12. kernel/bpf/devmap.c +9 −8
  13. kernel/bpf/hashtab.c +13 −9
  14. kernel/bpf/helpers.c +58 −0
  15. kernel/bpf/inode.c +113 −3
  16. kernel/bpf/lpm_trie.c +1 −0
  17. kernel/bpf/map_in_map.c +9 −15
  18. kernel/bpf/map_in_map.h +0 −2
  19. kernel/bpf/map_iter.c +15 −0
  20. kernel/bpf/percpu_freelist.c +96 −5
  21. kernel/bpf/percpu_freelist.h +1 −0
  22. kernel/bpf/preload/.gitignore +4 −0
  23. kernel/bpf/preload/Kconfig +26 −0
  24. kernel/bpf/preload/Makefile +25 −0
  25. kernel/bpf/preload/bpf_preload.h +16 −0
  26. kernel/bpf/preload/bpf_preload_kern.c +91 −0
  27. kernel/bpf/preload/bpf_preload_umd_blob.S +7 −0
  28. kernel/bpf/preload/iterators/.gitignore +2 −0
  29. kernel/bpf/preload/iterators/Makefile +57 −0
  30. kernel/bpf/preload/iterators/README +4 −0
  31. kernel/bpf/preload/iterators/bpf_preload_common.h +13 −0
  32. kernel/bpf/preload/iterators/iterators.bpf.c +114 −0
  33. kernel/bpf/preload/iterators/iterators.c +94 −0
  34. kernel/bpf/preload/iterators/iterators.skel.h +412 −0
  35. kernel/bpf/queue_stack_maps.c +2 −0
  36. kernel/bpf/reuseport_array.c +2 −1
  37. kernel/bpf/ringbuf.c +1 −0
  38. kernel/bpf/stackmap.c +3 −3
  39. kernel/bpf/syscall.c +278 −53
  40. kernel/bpf/task_iter.c +11 −4
  41. kernel/bpf/trampoline.c +51 −12
  42. kernel/bpf/verifier.c +1060 −328
  43. kernel/rcu/tasks.h +41 −12
  44. kernel/taskstats.c +7 −33
  45. kernel/trace/bpf_trace.c +165 −7
kernel/Makefile+1−1
@@ -12,7 +12,7 @@ obj-y = fork.o exec_domain.o panic.o \ · 11 unmodified lines
1212 notifier.o ksysfs.o cred.o reboot.o \
1313 async.o range.o smpboot.o ucount.o regset.o
1414
15−obj-$(CONFIG_BPFILTER) += usermode_driver.o
15+obj-$(CONFIG_USERMODE_DRIVER) += usermode_driver.o
1616 obj-$(CONFIG_MODULES) += kmod.o
1717 obj-$(CONFIG_MULTIUSER) += groups.o
1818
kernel/bpf/Makefile+3−0
@@ -5,6 +5,7 @@ CFLAGS_core.o += $(call cc-disable-warning, override-init) · 4 unmodified lines
55 obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o tnum.o bpf_iter.o map_iter.o task_iter.o prog_iter.o
66 obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o
77 obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o
8+obj-${CONFIG_BPF_LSM} += bpf_inode_storage.o
89 obj-$(CONFIG_BPF_SYSCALL) += disasm.o
910 obj-$(CONFIG_BPF_JIT) += trampoline.o
1011 obj-$(CONFIG_BPF_SYSCALL) += btf.o
@@ -12,6 +13,7 @@ obj-$(CONFIG_BPF_JIT) += dispatcher.o · 1 unmodified line
1213 ifeq ($(CONFIG_NET),y)
1314 obj-$(CONFIG_BPF_SYSCALL) += devmap.o
1415 obj-$(CONFIG_BPF_SYSCALL) += cpumap.o
16+obj-$(CONFIG_BPF_SYSCALL) += bpf_local_storage.o
1517 obj-$(CONFIG_BPF_SYSCALL) += offload.o
1618 obj-$(CONFIG_BPF_SYSCALL) += net_namespace.o
1719 endif
@@ -29,3 +31,4 @@ ifeq ($(CONFIG_BPF_JIT),y) · 11 unmodified lines
2931 obj-$(CONFIG_BPF_SYSCALL) += bpf_struct_ops.o
3032 obj-${CONFIG_BPF_LSM} += bpf_lsm.o
3133 endif
34+obj-$(CONFIG_BPF_PRELOAD) += preload/
kernel/bpf/arraymap.c+84−18
@@ -10,11 +10,13 @@ · 9 unmodified lines
1010 #include <linux/filter.h>
1111 #include <linux/perf_event.h>
1212 #include <uapi/linux/btf.h>
13+#include <linux/rcupdate_trace.h>
1314
1415 #include "map_in_map.h"
1516
1617 #define ARRAY_CREATE_FLAG_MASK \
17− (BPF_F_NUMA_NODE | BPF_F_MMAPABLE | BPF_F_ACCESS_MASK)
18+ (BPF_F_NUMA_NODE | BPF_F_MMAPABLE | BPF_F_ACCESS_MASK | \
19+ BPF_F_PRESERVE_ELEMS | BPF_F_INNER_MAP)
1820
1921 static void bpf_array_free_percpu(struct bpf_array *array)
2022 {
@@ -60,7 +62,11 @@ int array_map_alloc_check(union bpf_attr *attr) · 39 unmodified lines
6062 return -EINVAL;
6163
6264 if (attr->map_type != BPF_MAP_TYPE_ARRAY &&
63− attr->map_flags & BPF_F_MMAPABLE)
65+ attr->map_flags & (BPF_F_MMAPABLE | BPF_F_INNER_MAP))
66+ return -EINVAL;
67+
68+ if (attr->map_type != BPF_MAP_TYPE_PERF_EVENT_ARRAY &&
69+ attr->map_flags & BPF_F_PRESERVE_ELEMS)
6470 return -EINVAL;
6571
6672 if (attr->value_size > KMALLOC_MAX_SIZE)
@@ -208,7 +214,7 @@ static int array_map_direct_value_meta(const struct bpf_map *map, u64 imm, · 141 unmodified lines
208214 }
209215
210216 /* emit BPF instructions equivalent to C code of array_map_lookup_elem() */
211−static u32 array_map_gen_lookup(struct bpf_map *map, struct bpf_insn *insn_buf)
217+static int array_map_gen_lookup(struct bpf_map *map, struct bpf_insn *insn_buf)
212218 {
213219 struct bpf_array *array = container_of(map, struct bpf_array, map);
214220 struct bpf_insn *insn = insn_buf;
@@ -217,6 +223,9 @@ static u32 array_map_gen_lookup(struct bpf_map *map, struct bpf_insn *insn_buf) · 2 unmodified lines
217223 const int map_ptr = BPF_REG_1;
218224 const int index = BPF_REG_2;
219225
226+ if (map->map_flags & BPF_F_INNER_MAP)
227+ return -EOPNOTSUPP;
228+
220229 *insn++ = BPF_ALU64_IMM(BPF_ADD, map_ptr, offsetof(struct bpf_array, value));
221230 *insn++ = BPF_LDX_MEM(BPF_W, ret, index, 0);
222231 if (!map->bypass_spec_v1) {
@@ -487,6 +496,15 @@ static int array_map_mmap(struct bpf_map *map, struct vm_area_struct *vma) · 264 unmodified lines
487496 vma->vm_pgoff + pgoff);
488497 }
489498
499+static bool array_map_meta_equal(const struct bpf_map *meta0,
500+ const struct bpf_map *meta1)
501+{
502+ if (!bpf_map_meta_equal(meta0, meta1))
503+ return false;
504+ return meta0->map_flags & BPF_F_INNER_MAP ? true :
505+ meta0->max_entries == meta1->max_entries;
506+}
507+
490508 struct bpf_iter_seq_array_map_info {
491509 struct bpf_map *map;
492510 void *percpu_value_buf;
@@ -625,6 +643,7 @@ static const struct bpf_iter_seq_info iter_seq_info = { · 132 unmodified lines
625643
626644 static int array_map_btf_id;
627645 const struct bpf_map_ops array_map_ops = {
646+ .map_meta_equal = array_map_meta_equal,
628647 .map_alloc_check = array_map_alloc_check,
629648 .map_alloc = array_map_alloc,
630649 .map_free = array_map_free,
@@ -647,6 +666,7 @@ const struct bpf_map_ops array_map_ops = { · 16 unmodified lines
647666
648667 static int percpu_array_map_btf_id;
649668 const struct bpf_map_ops percpu_array_map_ops = {
669+ .map_meta_equal = bpf_map_meta_equal,
650670 .map_alloc_check = array_map_alloc_check,
651671 .map_alloc = array_map_alloc,
652672 .map_free = array_map_free,
@@ -888,6 +908,7 @@ static void prog_array_map_poke_run(struct bpf_map *map, u32 key, · 235 unmodified lines
888908 struct bpf_prog *old,
889909 struct bpf_prog *new)
890910 {
911+ u8 *old_addr, *new_addr, *old_bypass_addr;
891912 struct prog_poke_elem *elem;
892913 struct bpf_array_aux *aux;
893914
@@ -908,12 +929,13 @@ static void prog_array_map_poke_run(struct bpf_map *map, u32 key, · 14 unmodified lines
908929 * there could be danger of use after free otherwise.
909930 * 2) Initially when we start tracking aux, the program
910931 * is not JITed yet and also does not have a kallsyms
911− * entry. We skip these as poke->ip_stable is not
912− * active yet. The JIT will do the final fixup before
913− * setting it stable. The various poke->ip_stable are
914− * successively activated, so tail call updates can
915− * arrive from here while JIT is still finishing its
916− * final fixup for non-activated poke entries.
932+ * entry. We skip these as poke->tailcall_target_stable
933+ * is not active yet. The JIT will do the final fixup
934+ * before setting it stable. The various
935+ * poke->tailcall_target_stable are successively
936+ * activated, so tail call updates can arrive from here
937+ * while JIT is still finishing its final fixup for
938+ * non-activated poke entries.
917939 * 3) On program teardown, the program's kallsym entry gets
918940 * removed out of RCU callback, but we can only untrack
919941 * from sleepable context, therefore bpf_arch_text_poke()
@@ -930,7 +952,7 @@ static void prog_array_map_poke_run(struct bpf_map *map, u32 key, · 10 unmodified lines
930952 * 5) Any other error happening below from bpf_arch_text_poke()
931953 * is a unexpected bug.
932954 */
933− if (!READ_ONCE(poke->ip_stable))
955+ if (!READ_ONCE(poke->tailcall_target_stable))
934956 continue;
935957 if (poke->reason != BPF_POKE_REASON_TAIL_CALL)
936958 continue;
@@ -938,12 +960,39 @@ static void prog_array_map_poke_run(struct bpf_map *map, u32 key, · 1 unmodified line
938960 poke->tail_call.key != key)
939961 continue;
940962
941− ret = bpf_arch_text_poke(poke->ip, BPF_MOD_JUMP,
942− old ? (u8 *)old->bpf_func +
943− poke->adj_off : NULL,
944− new ? (u8 *)new->bpf_func +
945− poke->adj_off : NULL);
946− BUG_ON(ret < 0 && ret != -EINVAL);
963+ old_bypass_addr = old ? NULL : poke->bypass_addr;
964+ old_addr = old ? (u8 *)old->bpf_func + poke->adj_off : NULL;
965+ new_addr = new ? (u8 *)new->bpf_func + poke->adj_off : NULL;
966+
967+ if (new) {
968+ ret = bpf_arch_text_poke(poke->tailcall_target,
969+ BPF_MOD_JUMP,
970+ old_addr, new_addr);
971+ BUG_ON(ret < 0 && ret != -EINVAL);
972+ if (!old) {
973+ ret = bpf_arch_text_poke(poke->tailcall_bypass,
974+ BPF_MOD_JUMP,
975+ poke->bypass_addr,
976+ NULL);
977+ BUG_ON(ret < 0 && ret != -EINVAL);
978+ }
979+ } else {
980+ ret = bpf_arch_text_poke(poke->tailcall_bypass,
981+ BPF_MOD_JUMP,
982+ old_bypass_addr,
983+ poke->bypass_addr);
984+ BUG_ON(ret < 0 && ret != -EINVAL);
985+ /* let other CPUs finish the execution of program
986+ * so that it will not possible to expose them
987+ * to invalid nop, stack unwind, nop state
988+ */
989+ if (!ret)
990+ synchronize_rcu();
991+ ret = bpf_arch_text_poke(poke->tailcall_target,
992+ BPF_MOD_JUMP,
993+ old_addr, NULL);
994+ BUG_ON(ret < 0 && ret != -EINVAL);
995+ }
947996 }
948997 }
949998 }
@@ -1003,6 +1052,11 @@ static void prog_array_map_free(struct bpf_map *map) · 53 unmodified lines
10031052 fd_array_map_free(map);
10041053 }
10051054
1055+/* prog_array->aux->{type,jited} is a runtime binding.
1056+ * Doing static check alone in the verifier is not enough.
1057+ * Thus, prog_array_map cannot be used as an inner_map
1058+ * and map_meta_equal is not implemented.
1059+ */
10061060 static int prog_array_map_btf_id;
10071061 const struct bpf_map_ops prog_array_map_ops = {
10081062 .map_alloc_check = fd_array_map_alloc_check,
@@ -1090,6 +1144,9 @@ static void perf_event_fd_array_release(struct bpf_map *map, · 81 unmodified lines
10901144 struct bpf_event_entry *ee;
10911145 int i;
10921146
1147+ if (map->map_flags & BPF_F_PRESERVE_ELEMS)
1148+ return;
1149+
10931150 rcu_read_lock();
10941151 for (i = 0; i < array->map.max_entries; i++) {
10951152 ee = READ_ONCE(array->ptrs[i]);
@@ -1099,11 +1156,19 @@ static void perf_event_fd_array_release(struct bpf_map *map, · 3 unmodified lines
10991156 rcu_read_unlock();
11001157 }
11011158
1159+static void perf_event_fd_array_map_free(struct bpf_map *map)
1160+{
1161+ if (map->map_flags & BPF_F_PRESERVE_ELEMS)
1162+ bpf_fd_array_map_clear(map);
1163+ fd_array_map_free(map);
1164+}
1165+
11021166 static int perf_event_array_map_btf_id;
11031167 const struct bpf_map_ops perf_event_array_map_ops = {
1168+ .map_meta_equal = bpf_map_meta_equal,
11041169 .map_alloc_check = fd_array_map_alloc_check,
11051170 .map_alloc = array_map_alloc,
1106− .map_free = fd_array_map_free,
1171+ .map_free = perf_event_fd_array_map_free,
11071172 .map_get_next_key = array_map_get_next_key,
11081173 .map_lookup_elem = fd_array_map_lookup_elem,
11091174 .map_delete_elem = fd_array_map_delete_elem,
@@ -1137,6 +1202,7 @@ static void cgroup_fd_array_free(struct bpf_map *map) · 27 unmodified lines
11371202
11381203 static int cgroup_array_map_btf_id;
11391204 const struct bpf_map_ops cgroup_array_map_ops = {
1205+ .map_meta_equal = bpf_map_meta_equal,
11401206 .map_alloc_check = fd_array_map_alloc_check,
11411207 .map_alloc = array_map_alloc,
11421208 .map_free = cgroup_fd_array_free,
@@ -1190,7 +1256,7 @@ static void *array_of_map_lookup_elem(struct bpf_map *map, void *key) · 47 unmodified lines
11901256 return READ_ONCE(*inner_map);
11911257 }
11921258
1193−static u32 array_of_map_gen_lookup(struct bpf_map *map,
1259+static int array_of_map_gen_lookup(struct bpf_map *map,
11941260 struct bpf_insn *insn_buf)
11951261 {
11961262 struct bpf_array *array = container_of(map, struct bpf_array, map);
kernel/bpf/bpf_inode_storage.c+272−0
@@ -0,0 +1,272 @@
1+// SPDX-License-Identifier: GPL-2.0
2+/*
3+ * Copyright (c) 2019 Facebook
4+ * Copyright 2020 Google LLC.
5+ */
6+
7+#include <linux/rculist.h>
8+#include <linux/list.h>
9+#include <linux/hash.h>
10+#include <linux/types.h>
11+#include <linux/spinlock.h>
12+#include <linux/bpf.h>
13+#include <linux/bpf_local_storage.h>
14+#include <net/sock.h>
15+#include <uapi/linux/sock_diag.h>
16+#include <uapi/linux/btf.h>
17+#include <linux/bpf_lsm.h>
18+#include <linux/btf_ids.h>
19+#include <linux/fdtable.h>
20+
21+DEFINE_BPF_STORAGE_CACHE(inode_cache);
22+
23+static struct bpf_local_storage __rcu **
24+inode_storage_ptr(void *owner)
25+{
26+ struct inode *inode = owner;
27+ struct bpf_storage_blob *bsb;
28+
29+ bsb = bpf_inode(inode);
30+ if (!bsb)
31+ return NULL;
32+ return &bsb->storage;
33+}
34+
35+static struct bpf_local_storage_data *inode_storage_lookup(struct inode *inode,
36+ struct bpf_map *map,
37+ bool cacheit_lockit)
38+{
39+ struct bpf_local_storage *inode_storage;
40+ struct bpf_local_storage_map *smap;
41+ struct bpf_storage_blob *bsb;
42+
43+ bsb = bpf_inode(inode);
44+ if (!bsb)
45+ return NULL;
46+
47+ inode_storage = rcu_dereference(bsb->storage);
48+ if (!inode_storage)
49+ return NULL;
50+
51+ smap = (struct bpf_local_storage_map *)map;
52+ return bpf_local_storage_lookup(inode_storage, smap, cacheit_lockit);
53+}
54+
55+void bpf_inode_storage_free(struct inode *inode)
56+{
57+ struct bpf_local_storage_elem *selem;
58+ struct bpf_local_storage *local_storage;
59+ bool free_inode_storage = false;
60+ struct bpf_storage_blob *bsb;
61+ struct hlist_node *n;
62+
63+ bsb = bpf_inode(inode);
64+ if (!bsb)
65+ return;
66+
67+ rcu_read_lock();
68+
69+ local_storage = rcu_dereference(bsb->storage);
70+ if (!local_storage) {
71+ rcu_read_unlock();
72+ return;
73+ }
74+
75+ /* Netiher the bpf_prog nor the bpf-map's syscall
76+ * could be modifying the local_storage->list now.
77+ * Thus, no elem can be added-to or deleted-from the
78+ * local_storage->list by the bpf_prog or by the bpf-map's syscall.
79+ *
80+ * It is racing with bpf_local_storage_map_free() alone
81+ * when unlinking elem from the local_storage->list and
82+ * the map's bucket->list.
83+ */
84+ raw_spin_lock_bh(&local_storage->lock);
85+ hlist_for_each_entry_safe(selem, n, &local_storage->list, snode) {
86+ /* Always unlink from map before unlinking from
87+ * local_storage.
88+ */
89+ bpf_selem_unlink_map(selem);
90+ free_inode_storage = bpf_selem_unlink_storage_nolock(
91+ local_storage, selem, false);
92+ }
93+ raw_spin_unlock_bh(&local_storage->lock);
94+ rcu_read_unlock();
95+
96+ /* free_inoode_storage should always be true as long as
97+ * local_storage->list was non-empty.
98+ */
99+ if (free_inode_storage)
100+ kfree_rcu(local_storage, rcu);
101+}
102+
103+static void *bpf_fd_inode_storage_lookup_elem(struct bpf_map *map, void *key)
104+{
105+ struct bpf_local_storage_data *sdata;
106+ struct file *f;
107+ int fd;
108+
109+ fd = *(int *)key;
110+ f = fget_raw(fd);
111+ if (!f)
112+ return NULL;
113+
114+ sdata = inode_storage_lookup(f->f_inode, map, true);
115+ fput(f);
116+ return sdata ? sdata->data : NULL;
117+}
118+
119+static int bpf_fd_inode_storage_update_elem(struct bpf_map *map, void *key,
120+ void *value, u64 map_flags)
121+{
122+ struct bpf_local_storage_data *sdata;
123+ struct file *f;
124+ int fd;
125+
126+ fd = *(int *)key;
127+ f = fget_raw(fd);
128+ if (!f || !inode_storage_ptr(f->f_inode))
129+ return -EBADF;
130+
131+ sdata = bpf_local_storage_update(f->f_inode,
132+ (struct bpf_local_storage_map *)map,
133+ value, map_flags);
134+ fput(f);
135+ return PTR_ERR_OR_ZERO(sdata);
136+}
137+
138+static int inode_storage_delete(struct inode *inode, struct bpf_map *map)
139+{
140+ struct bpf_local_storage_data *sdata;
141+
142+ sdata = inode_storage_lookup(inode, map, false);
143+ if (!sdata)
144+ return -ENOENT;
145+
146+ bpf_selem_unlink(SELEM(sdata));
147+
148+ return 0;
149+}
150+
151+static int bpf_fd_inode_storage_delete_elem(struct bpf_map *map, void *key)
152+{
153+ struct file *f;
154+ int fd, err;
155+
156+ fd = *(int *)key;
157+ f = fget_raw(fd);
158+ if (!f)
159+ return -EBADF;
160+
161+ err = inode_storage_delete(f->f_inode, map);
162+ fput(f);
163+ return err;
164+}
165+
166+BPF_CALL_4(bpf_inode_storage_get, struct bpf_map *, map, struct inode *, inode,
167+ void *, value, u64, flags)
168+{
169+ struct bpf_local_storage_data *sdata;
170+
171+ if (flags & ~(BPF_LOCAL_STORAGE_GET_F_CREATE))
172+ return (unsigned long)NULL;
173+
174+ /* explicitly check that the inode_storage_ptr is not
175+ * NULL as inode_storage_lookup returns NULL in this case and
176+ * bpf_local_storage_update expects the owner to have a
177+ * valid storage pointer.
178+ */
179+ if (!inode_storage_ptr(inode))
180+ return (unsigned long)NULL;
181+
182+ sdata = inode_storage_lookup(inode, map, true);
183+ if (sdata)
184+ return (unsigned long)sdata->data;
185+
186+ /* This helper must only called from where the inode is gurranteed
187+ * to have a refcount and cannot be freed.
188+ */
189+ if (flags & BPF_LOCAL_STORAGE_GET_F_CREATE) {
190+ sdata = bpf_local_storage_update(
191+ inode, (struct bpf_local_storage_map *)map, value,
192+ BPF_NOEXIST);
193+ return IS_ERR(sdata) ? (unsigned long)NULL :
194+ (unsigned long)sdata->data;
195+ }
196+
197+ return (unsigned long)NULL;
198+}
199+
200+BPF_CALL_2(bpf_inode_storage_delete,
201+ struct bpf_map *, map, struct inode *, inode)
202+{
203+ /* This helper must only called from where the inode is gurranteed
204+ * to have a refcount and cannot be freed.
205+ */
206+ return inode_storage_delete(inode, map);
207+}
208+
209+static int notsupp_get_next_key(struct bpf_map *map, void *key,
210+ void *next_key)
211+{
212+ return -ENOTSUPP;
213+}
214+
215+static struct bpf_map *inode_storage_map_alloc(union bpf_attr *attr)
216+{
217+ struct bpf_local_storage_map *smap;
218+
219+ smap = bpf_local_storage_map_alloc(attr);
220+ if (IS_ERR(smap))
221+ return ERR_CAST(smap);
222+
223+ smap->cache_idx = bpf_local_storage_cache_idx_get(&inode_cache);
224+ return &smap->map;
225+}
226+
227+static void inode_storage_map_free(struct bpf_map *map)
228+{
229+ struct bpf_local_storage_map *smap;
230+
231+ smap = (struct bpf_local_storage_map *)map;
232+ bpf_local_storage_cache_idx_free(&inode_cache, smap->cache_idx);
233+ bpf_local_storage_map_free(smap);
234+}
235+
236+static int inode_storage_map_btf_id;
237+const struct bpf_map_ops inode_storage_map_ops = {
238+ .map_meta_equal = bpf_map_meta_equal,
239+ .map_alloc_check = bpf_local_storage_map_alloc_check,
240+ .map_alloc = inode_storage_map_alloc,
241+ .map_free = inode_storage_map_free,
242+ .map_get_next_key = notsupp_get_next_key,
243+ .map_lookup_elem = bpf_fd_inode_storage_lookup_elem,
244+ .map_update_elem = bpf_fd_inode_storage_update_elem,
245+ .map_delete_elem = bpf_fd_inode_storage_delete_elem,
246+ .map_check_btf = bpf_local_storage_map_check_btf,
247+ .map_btf_name = "bpf_local_storage_map",
248+ .map_btf_id = &inode_storage_map_btf_id,
249+ .map_owner_storage_ptr = inode_storage_ptr,
250+};
251+
252+BTF_ID_LIST_SINGLE(bpf_inode_storage_btf_ids, struct, inode)
253+
254+const struct bpf_func_proto bpf_inode_storage_get_proto = {
255+ .func = bpf_inode_storage_get,
256+ .gpl_only = false,
257+ .ret_type = RET_PTR_TO_MAP_VALUE_OR_NULL,
258+ .arg1_type = ARG_CONST_MAP_PTR,
259+ .arg2_type = ARG_PTR_TO_BTF_ID,
260+ .arg2_btf_id = &bpf_inode_storage_btf_ids[0],
261+ .arg3_type = ARG_PTR_TO_MAP_VALUE_OR_NULL,
262+ .arg4_type = ARG_ANYTHING,
263+};
264+
265+const struct bpf_func_proto bpf_inode_storage_delete_proto = {
266+ .func = bpf_inode_storage_delete,
267+ .gpl_only = false,
268+ .ret_type = RET_INTEGER,
269+ .arg1_type = ARG_CONST_MAP_PTR,
270+ .arg2_type = ARG_PTR_TO_BTF_ID,
271+ .arg2_btf_id = &bpf_inode_storage_btf_ids[0],
272+};
kernel/bpf/bpf_iter.c+60−2
@@ -88,8 +88,8 @@ static ssize_t bpf_seq_read(struct file *file, char __user *buf, size_t size, · 87 unmodified lines
8888 mutex_lock(&seq->lock);
8989
9090 if (!seq->buf) {
91− seq->size = PAGE_SIZE;
92− seq->buf = kmalloc(seq->size, GFP_KERNEL);
91+ seq->size = PAGE_SIZE << 3;
92+ seq->buf = kvmalloc(seq->size, GFP_KERNEL);
9393 if (!seq->buf) {
9494 err = -ENOMEM;
9595 goto done;
@@ -390,10 +390,68 @@ out_unlock: · 294 unmodified lines
390390 return ret;
391391 }
392392
393+static void bpf_iter_link_show_fdinfo(const struct bpf_link *link,
394+ struct seq_file *seq)
395+{
396+ struct bpf_iter_link *iter_link =
397+ container_of(link, struct bpf_iter_link, link);
398+ bpf_iter_show_fdinfo_t show_fdinfo;
399+
400+ seq_printf(seq,
401+ "target_name:\t%s\n",
402+ iter_link->tinfo->reg_info->target);
403+
404+ show_fdinfo = iter_link->tinfo->reg_info->show_fdinfo;
405+ if (show_fdinfo)
406+ show_fdinfo(&iter_link->aux, seq);
407+}
408+
409+static int bpf_iter_link_fill_link_info(const struct bpf_link *link,
410+ struct bpf_link_info *info)
411+{
412+ struct bpf_iter_link *iter_link =
413+ container_of(link, struct bpf_iter_link, link);
414+ char __user *ubuf = u64_to_user_ptr(info->iter.target_name);
415+ bpf_iter_fill_link_info_t fill_link_info;
416+ u32 ulen = info->iter.target_name_len;
417+ const char *target_name;
418+ u32 target_len;
419+
420+ if (!ulen ^ !ubuf)
421+ return -EINVAL;
422+
423+ target_name = iter_link->tinfo->reg_info->target;
424+ target_len = strlen(target_name);
425+ info->iter.target_name_len = target_len + 1;
426+
427+ if (ubuf) {
428+ if (ulen >= target_len + 1) {
429+ if (copy_to_user(ubuf, target_name, target_len + 1))
430+ return -EFAULT;
431+ } else {
432+ char zero = '\0';
433+
434+ if (copy_to_user(ubuf, target_name, ulen - 1))
435+ return -EFAULT;
436+ if (put_user(zero, ubuf + ulen - 1))
437+ return -EFAULT;
438+ return -ENOSPC;
439+ }
440+ }
441+
442+ fill_link_info = iter_link->tinfo->reg_info->fill_link_info;
443+ if (fill_link_info)
444+ return fill_link_info(&iter_link->aux, info);
445+
446+ return 0;
447+}
448+
393449 static const struct bpf_link_ops bpf_iter_link_lops = {
394450 .release = bpf_iter_link_release,
395451 .dealloc = bpf_iter_link_dealloc,
396452 .update_prog = bpf_iter_link_replace,
453+ .show_fdinfo = bpf_iter_link_show_fdinfo,
454+ .fill_link_info = bpf_iter_link_fill_link_info,
397455 };
398456
399457 bool bpf_link_is_iter(struct bpf_link *link)
kernel/bpf/bpf_local_storage.c+600−0
@@ -0,0 +1,600 @@
1+// SPDX-License-Identifier: GPL-2.0
2+/* Copyright (c) 2019 Facebook */
3+#include <linux/rculist.h>
4+#include <linux/list.h>
5+#include <linux/hash.h>
6+#include <linux/types.h>
7+#include <linux/spinlock.h>
8+#include <linux/bpf.h>
9+#include <linux/btf_ids.h>
10+#include <linux/bpf_local_storage.h>
11+#include <net/sock.h>
12+#include <uapi/linux/sock_diag.h>
13+#include <uapi/linux/btf.h>
14+
15+#define BPF_LOCAL_STORAGE_CREATE_FLAG_MASK (BPF_F_NO_PREALLOC | BPF_F_CLONE)
16+
17+static struct bpf_local_storage_map_bucket *
18+select_bucket(struct bpf_local_storage_map *smap,
19+ struct bpf_local_storage_elem *selem)
20+{
21+ return &smap->buckets[hash_ptr(selem, smap->bucket_log)];
22+}
23+
24+static int mem_charge(struct bpf_local_storage_map *smap, void *owner, u32 size)
25+{
26+ struct bpf_map *map = &smap->map;
27+
28+ if (!map->ops->map_local_storage_charge)
29+ return 0;
30+
31+ return map->ops->map_local_storage_charge(smap, owner, size);
32+}
33+
34+static void mem_uncharge(struct bpf_local_storage_map *smap, void *owner,
35+ u32 size)
36+{
37+ struct bpf_map *map = &smap->map;
38+
39+ if (map->ops->map_local_storage_uncharge)
40+ map->ops->map_local_storage_uncharge(smap, owner, size);
41+}
42+
43+static struct bpf_local_storage __rcu **
44+owner_storage(struct bpf_local_storage_map *smap, void *owner)
45+{
46+ struct bpf_map *map = &smap->map;
47+
48+ return map->ops->map_owner_storage_ptr(owner);
49+}
50+
51+static bool selem_linked_to_storage(const struct bpf_local_storage_elem *selem)
52+{
53+ return !hlist_unhashed(&selem->snode);
54+}
55+
56+static bool selem_linked_to_map(const struct bpf_local_storage_elem *selem)
57+{
58+ return !hlist_unhashed(&selem->map_node);
59+}
60+
61+struct bpf_local_storage_elem *
62+bpf_selem_alloc(struct bpf_local_storage_map *smap, void *owner,
63+ void *value, bool charge_mem)
64+{
65+ struct bpf_local_storage_elem *selem;
66+
67+ if (charge_mem && mem_charge(smap, owner, smap->elem_size))
68+ return NULL;
69+
70+ selem = kzalloc(smap->elem_size, GFP_ATOMIC | __GFP_NOWARN);
71+ if (selem) {
72+ if (value)
73+ memcpy(SDATA(selem)->data, value, smap->map.value_size);
74+ return selem;
75+ }
76+
77+ if (charge_mem)
78+ mem_uncharge(smap, owner, smap->elem_size);
79+
80+ return NULL;
81+}
82+
83+/* local_storage->lock must be held and selem->local_storage == local_storage.
84+ * The caller must ensure selem->smap is still valid to be
85+ * dereferenced for its smap->elem_size and smap->cache_idx.
86+ */
87+bool bpf_selem_unlink_storage_nolock(struct bpf_local_storage *local_storage,
88+ struct bpf_local_storage_elem *selem,
89+ bool uncharge_mem)
90+{
91+ struct bpf_local_storage_map *smap;
92+ bool free_local_storage;
93+ void *owner;
94+
95+ smap = rcu_dereference(SDATA(selem)->smap);
96+ owner = local_storage->owner;
97+
98+ /* All uncharging on the owner must be done first.
99+ * The owner may be freed once the last selem is unlinked
100+ * from local_storage.
101+ */
102+ if (uncharge_mem)
103+ mem_uncharge(smap, owner, smap->elem_size);
104+
105+ free_local_storage = hlist_is_singular_node(&selem->snode,
106+ &local_storage->list);
107+ if (free_local_storage) {
108+ mem_uncharge(smap, owner, sizeof(struct bpf_local_storage));
109+ local_storage->owner = NULL;
110+
111+ /* After this RCU_INIT, owner may be freed and cannot be used */
112+ RCU_INIT_POINTER(*owner_storage(smap, owner), NULL);
113+
114+ /* local_storage is not freed now. local_storage->lock is
115+ * still held and raw_spin_unlock_bh(&local_storage->lock)
116+ * will be done by the caller.
117+ *
118+ * Although the unlock will be done under
119+ * rcu_read_lock(), it is more intutivie to
120+ * read if kfree_rcu(local_storage, rcu) is done
121+ * after the raw_spin_unlock_bh(&local_storage->lock).
122+ *
123+ * Hence, a "bool free_local_storage" is returned
124+ * to the caller which then calls the kfree_rcu()
125+ * after unlock.
126+ */
127+ }
128+ hlist_del_init_rcu(&selem->snode);
129+ if (rcu_access_pointer(local_storage->cache[smap->cache_idx]) ==
130+ SDATA(selem))
131+ RCU_INIT_POINTER(local_storage->cache[smap->cache_idx], NULL);
132+
133+ kfree_rcu(selem, rcu);
134+
135+ return free_local_storage;
136+}
137+
138+static void __bpf_selem_unlink_storage(struct bpf_local_storage_elem *selem)
139+{
140+ struct bpf_local_storage *local_storage;
141+ bool free_local_storage = false;
142+
143+ if (unlikely(!selem_linked_to_storage(selem)))
144+ /* selem has already been unlinked from sk */
145+ return;
146+
147+ local_storage = rcu_dereference(selem->local_storage);
148+ raw_spin_lock_bh(&local_storage->lock);
149+ if (likely(selem_linked_to_storage(selem)))
150+ free_local_storage = bpf_selem_unlink_storage_nolock(
151+ local_storage, selem, true);
152+ raw_spin_unlock_bh(&local_storage->lock);
153+
154+ if (free_local_storage)
155+ kfree_rcu(local_storage, rcu);
156+}
157+
158+void bpf_selem_link_storage_nolock(struct bpf_local_storage *local_storage,
159+ struct bpf_local_storage_elem *selem)
160+{
161+ RCU_INIT_POINTER(selem->local_storage, local_storage);
162+ hlist_add_head_rcu(&selem->snode, &local_storage->list);
163+}
164+
165+void bpf_selem_unlink_map(struct bpf_local_storage_elem *selem)
166+{
167+ struct bpf_local_storage_map *smap;
168+ struct bpf_local_storage_map_bucket *b;
169+
170+ if (unlikely(!selem_linked_to_map(selem)))
171+ /* selem has already be unlinked from smap */
172+ return;
173+
174+ smap = rcu_dereference(SDATA(selem)->smap);
175+ b = select_bucket(smap, selem);
176+ raw_spin_lock_bh(&b->lock);
177+ if (likely(selem_linked_to_map(selem)))
178+ hlist_del_init_rcu(&selem->map_node);
179+ raw_spin_unlock_bh(&b->lock);
180+}
181+
182+void bpf_selem_link_map(struct bpf_local_storage_map *smap,
183+ struct bpf_local_storage_elem *selem)
184+{
185+ struct bpf_local_storage_map_bucket *b = select_bucket(smap, selem);
186+
187+ raw_spin_lock_bh(&b->lock);
188+ RCU_INIT_POINTER(SDATA(selem)->smap, smap);
189+ hlist_add_head_rcu(&selem->map_node, &b->list);
190+ raw_spin_unlock_bh(&b->lock);
191+}
192+
193+void bpf_selem_unlink(struct bpf_local_storage_elem *selem)
194+{
195+ /* Always unlink from map before unlinking from local_storage
196+ * because selem will be freed after successfully unlinked from
197+ * the local_storage.
198+ */
199+ bpf_selem_unlink_map(selem);
200+ __bpf_selem_unlink_storage(selem);
201+}
202+
203+struct bpf_local_storage_data *
204+bpf_local_storage_lookup(struct bpf_local_storage *local_storage,
205+ struct bpf_local_storage_map *smap,
206+ bool cacheit_lockit)
207+{
208+ struct bpf_local_storage_data *sdata;
209+ struct bpf_local_storage_elem *selem;
210+
211+ /* Fast path (cache hit) */
212+ sdata = rcu_dereference(local_storage->cache[smap->cache_idx]);
213+ if (sdata && rcu_access_pointer(sdata->smap) == smap)
214+ return sdata;
215+
216+ /* Slow path (cache miss) */
217+ hlist_for_each_entry_rcu(selem, &local_storage->list, snode)
218+ if (rcu_access_pointer(SDATA(selem)->smap) == smap)
219+ break;
220+
221+ if (!selem)
222+ return NULL;
223+
224+ sdata = SDATA(selem);
225+ if (cacheit_lockit) {
226+ /* spinlock is needed to avoid racing with the
227+ * parallel delete. Otherwise, publishing an already
228+ * deleted sdata to the cache will become a use-after-free
229+ * problem in the next bpf_local_storage_lookup().
230+ */
231+ raw_spin_lock_bh(&local_storage->lock);
232+ if (selem_linked_to_storage(selem))
233+ rcu_assign_pointer(local_storage->cache[smap->cache_idx],
234+ sdata);
235+ raw_spin_unlock_bh(&local_storage->lock);
236+ }
237+
238+ return sdata;
239+}
240+
241+static int check_flags(const struct bpf_local_storage_data *old_sdata,
242+ u64 map_flags)
243+{
244+ if (old_sdata && (map_flags & ~BPF_F_LOCK) == BPF_NOEXIST)
245+ /* elem already exists */
246+ return -EEXIST;
247+
248+ if (!old_sdata && (map_flags & ~BPF_F_LOCK) == BPF_EXIST)
249+ /* elem doesn't exist, cannot update it */
250+ return -ENOENT;
251+
252+ return 0;
253+}
254+
255+int bpf_local_storage_alloc(void *owner,
256+ struct bpf_local_storage_map *smap,
257+ struct bpf_local_storage_elem *first_selem)
258+{
259+ struct bpf_local_storage *prev_storage, *storage;
260+ struct bpf_local_storage **owner_storage_ptr;
261+ int err;
262+
263+ err = mem_charge(smap, owner, sizeof(*storage));
264+ if (err)
265+ return err;
266+
267+ storage = kzalloc(sizeof(*storage), GFP_ATOMIC | __GFP_NOWARN);
268+ if (!storage) {
269+ err = -ENOMEM;
270+ goto uncharge;
271+ }
272+
273+ INIT_HLIST_HEAD(&storage->list);
274+ raw_spin_lock_init(&storage->lock);
275+ storage->owner = owner;
276+
277+ bpf_selem_link_storage_nolock(storage, first_selem);
278+ bpf_selem_link_map(smap, first_selem);
279+
280+ owner_storage_ptr =
281+ (struct bpf_local_storage **)owner_storage(smap, owner);
282+ /* Publish storage to the owner.
283+ * Instead of using any lock of the kernel object (i.e. owner),
284+ * cmpxchg will work with any kernel object regardless what
285+ * the running context is, bh, irq...etc.
286+ *
287+ * From now on, the owner->storage pointer (e.g. sk->sk_bpf_storage)
288+ * is protected by the storage->lock. Hence, when freeing
289+ * the owner->storage, the storage->lock must be held before
290+ * setting owner->storage ptr to NULL.
291+ */
292+ prev_storage = cmpxchg(owner_storage_ptr, NULL, storage);
293+ if (unlikely(prev_storage)) {
294+ bpf_selem_unlink_map(first_selem);
295+ err = -EAGAIN;
296+ goto uncharge;
297+
298+ /* Note that even first_selem was linked to smap's
299+ * bucket->list, first_selem can be freed immediately
300+ * (instead of kfree_rcu) because
301+ * bpf_local_storage_map_free() does a
302+ * synchronize_rcu() before walking the bucket->list.
303+ * Hence, no one is accessing selem from the
304+ * bucket->list under rcu_read_lock().
305+ */
306+ }
307+
308+ return 0;
309+
310+uncharge:
311+ kfree(storage);
312+ mem_uncharge(smap, owner, sizeof(*storage));
313+ return err;
314+}
315+
316+/* sk cannot be going away because it is linking new elem
317+ * to sk->sk_bpf_storage. (i.e. sk->sk_refcnt cannot be 0).
318+ * Otherwise, it will become a leak (and other memory issues
319+ * during map destruction).
320+ */
321+struct bpf_local_storage_data *
322+bpf_local_storage_update(void *owner, struct bpf_local_storage_map *smap,
323+ void *value, u64 map_flags)
324+{
325+ struct bpf_local_storage_data *old_sdata = NULL;
326+ struct bpf_local_storage_elem *selem;
327+ struct bpf_local_storage *local_storage;
328+ int err;
329+
330+ /* BPF_EXIST and BPF_NOEXIST cannot be both set */
331+ if (unlikely((map_flags & ~BPF_F_LOCK) > BPF_EXIST) ||
332+ /* BPF_F_LOCK can only be used in a value with spin_lock */
333+ unlikely((map_flags & BPF_F_LOCK) &&
334+ !map_value_has_spin_lock(&smap->map)))
335+ return ERR_PTR(-EINVAL);
336+
337+ local_storage = rcu_dereference(*owner_storage(smap, owner));
338+ if (!local_storage || hlist_empty(&local_storage->list)) {
339+ /* Very first elem for the owner */
340+ err = check_flags(NULL, map_flags);
341+ if (err)
342+ return ERR_PTR(err);
343+
344+ selem = bpf_selem_alloc(smap, owner, value, true);
345+ if (!selem)
346+ return ERR_PTR(-ENOMEM);
347+
348+ err = bpf_local_storage_alloc(owner, smap, selem);
349+ if (err) {
350+ kfree(selem);
351+ mem_uncharge(smap, owner, smap->elem_size);
352+ return ERR_PTR(err);
353+ }
354+
355+ return SDATA(selem);
356+ }
357+
358+ if ((map_flags & BPF_F_LOCK) && !(map_flags & BPF_NOEXIST)) {
359+ /* Hoping to find an old_sdata to do inline update
360+ * such that it can avoid taking the local_storage->lock
361+ * and changing the lists.
362+ */
363+ old_sdata =
364+ bpf_local_storage_lookup(local_storage, smap, false);
365+ err = check_flags(old_sdata, map_flags);
366+ if (err)
367+ return ERR_PTR(err);
368+ if (old_sdata && selem_linked_to_storage(SELEM(old_sdata))) {
369+ copy_map_value_locked(&smap->map, old_sdata->data,
370+ value, false);
371+ return old_sdata;
372+ }
373+ }
374+
375+ raw_spin_lock_bh(&local_storage->lock);
376+
377+ /* Recheck local_storage->list under local_storage->lock */
378+ if (unlikely(hlist_empty(&local_storage->list))) {
379+ /* A parallel del is happening and local_storage is going
380+ * away. It has just been checked before, so very
381+ * unlikely. Return instead of retry to keep things
382+ * simple.
383+ */
384+ err = -EAGAIN;
385+ goto unlock_err;
386+ }
387+
388+ old_sdata = bpf_local_storage_lookup(local_storage, smap, false);
389+ err = check_flags(old_sdata, map_flags);
390+ if (err)
391+ goto unlock_err;
392+
393+ if (old_sdata && (map_flags & BPF_F_LOCK)) {
394+ copy_map_value_locked(&smap->map, old_sdata->data, value,
395+ false);
396+ selem = SELEM(old_sdata);
397+ goto unlock;
398+ }
399+
400+ /* local_storage->lock is held. Hence, we are sure
401+ * we can unlink and uncharge the old_sdata successfully
402+ * later. Hence, instead of charging the new selem now
403+ * and then uncharge the old selem later (which may cause
404+ * a potential but unnecessary charge failure), avoid taking
405+ * a charge at all here (the "!old_sdata" check) and the
406+ * old_sdata will not be uncharged later during
407+ * bpf_selem_unlink_storage_nolock().
408+ */
409+ selem = bpf_selem_alloc(smap, owner, value, !old_sdata);
410+ if (!selem) {
411+ err = -ENOMEM;
412+ goto unlock_err;
413+ }
414+
415+ /* First, link the new selem to the map */
416+ bpf_selem_link_map(smap, selem);
417+
418+ /* Second, link (and publish) the new selem to local_storage */
419+ bpf_selem_link_storage_nolock(local_storage, selem);
420+
421+ /* Third, remove old selem, SELEM(old_sdata) */
422+ if (old_sdata) {
423+ bpf_selem_unlink_map(SELEM(old_sdata));
424+ bpf_selem_unlink_storage_nolock(local_storage, SELEM(old_sdata),
425+ false);
426+ }
427+
428+unlock:
429+ raw_spin_unlock_bh(&local_storage->lock);
430+ return SDATA(selem);
431+
432+unlock_err:
433+ raw_spin_unlock_bh(&local_storage->lock);
434+ return ERR_PTR(err);
435+}
436+
437+u16 bpf_local_storage_cache_idx_get(struct bpf_local_storage_cache *cache)
438+{
439+ u64 min_usage = U64_MAX;
440+ u16 i, res = 0;
441+
442+ spin_lock(&cache->idx_lock);
443+
444+ for (i = 0; i < BPF_LOCAL_STORAGE_CACHE_SIZE; i++) {
445+ if (cache->idx_usage_counts[i] < min_usage) {
446+ min_usage = cache->idx_usage_counts[i];
447+ res = i;
448+
449+ /* Found a free cache_idx */
450+ if (!min_usage)
451+ break;
452+ }
453+ }
454+ cache->idx_usage_counts[res]++;
455+
456+ spin_unlock(&cache->idx_lock);
457+
458+ return res;
459+}
460+
461+void bpf_local_storage_cache_idx_free(struct bpf_local_storage_cache *cache,
462+ u16 idx)
463+{
464+ spin_lock(&cache->idx_lock);
465+ cache->idx_usage_counts[idx]--;
466+ spin_unlock(&cache->idx_lock);
467+}
468+
469+void bpf_local_storage_map_free(struct bpf_local_storage_map *smap)
470+{
471+ struct bpf_local_storage_elem *selem;
472+ struct bpf_local_storage_map_bucket *b;
473+ unsigned int i;
474+
475+ /* Note that this map might be concurrently cloned from
476+ * bpf_sk_storage_clone. Wait for any existing bpf_sk_storage_clone
477+ * RCU read section to finish before proceeding. New RCU
478+ * read sections should be prevented via bpf_map_inc_not_zero.
479+ */
480+ synchronize_rcu();
481+
482+ /* bpf prog and the userspace can no longer access this map
483+ * now. No new selem (of this map) can be added
484+ * to the owner->storage or to the map bucket's list.
485+ *
486+ * The elem of this map can be cleaned up here
487+ * or when the storage is freed e.g.
488+ * by bpf_sk_storage_free() during __sk_destruct().
489+ */
490+ for (i = 0; i < (1U << smap->bucket_log); i++) {
491+ b = &smap->buckets[i];
492+
493+ rcu_read_lock();
494+ /* No one is adding to b->list now */
495+ while ((selem = hlist_entry_safe(
496+ rcu_dereference_raw(hlist_first_rcu(&b->list)),
497+ struct bpf_local_storage_elem, map_node))) {
498+ bpf_selem_unlink(selem);
499+ cond_resched_rcu();
500+ }
501+ rcu_read_unlock();
502+ }
503+
504+ /* While freeing the storage we may still need to access the map.
505+ *
506+ * e.g. when bpf_sk_storage_free() has unlinked selem from the map
507+ * which then made the above while((selem = ...)) loop
508+ * exit immediately.
509+ *
510+ * However, while freeing the storage one still needs to access the
511+ * smap->elem_size to do the uncharging in
512+ * bpf_selem_unlink_storage_nolock().
513+ *
514+ * Hence, wait another rcu grace period for the storage to be freed.
515+ */
516+ synchronize_rcu();
517+
518+ kvfree(smap->buckets);
519+ kfree(smap);
520+}
521+
522+int bpf_local_storage_map_alloc_check(union bpf_attr *attr)
523+{
524+ if (attr->map_flags & ~BPF_LOCAL_STORAGE_CREATE_FLAG_MASK ||
525+ !(attr->map_flags & BPF_F_NO_PREALLOC) ||
526+ attr->max_entries ||
527+ attr->key_size != sizeof(int) || !attr->value_size ||
528+ /* Enforce BTF for userspace sk dumping */
529+ !attr->btf_key_type_id || !attr->btf_value_type_id)
530+ return -EINVAL;
531+
532+ if (!bpf_capable())
533+ return -EPERM;
534+
535+ if (attr->value_size > BPF_LOCAL_STORAGE_MAX_VALUE_SIZE)
536+ return -E2BIG;
537+
538+ return 0;
539+}
540+
541+struct bpf_local_storage_map *bpf_local_storage_map_alloc(union bpf_attr *attr)
542+{
543+ struct bpf_local_storage_map *smap;
544+ unsigned int i;
545+ u32 nbuckets;
546+ u64 cost;
547+ int ret;
548+
549+ smap = kzalloc(sizeof(*smap), GFP_USER | __GFP_NOWARN);
550+ if (!smap)
551+ return ERR_PTR(-ENOMEM);
552+ bpf_map_init_from_attr(&smap->map, attr);
553+
554+ nbuckets = roundup_pow_of_two(num_possible_cpus());
555+ /* Use at least 2 buckets, select_bucket() is undefined behavior with 1 bucket */
556+ nbuckets = max_t(u32, 2, nbuckets);
557+ smap->bucket_log = ilog2(nbuckets);
558+ cost = sizeof(*smap->buckets) * nbuckets + sizeof(*smap);
559+
560+ ret = bpf_map_charge_init(&smap->map.memory, cost);
561+ if (ret < 0) {
562+ kfree(smap);
563+ return ERR_PTR(ret);
564+ }
565+
566+ smap->buckets = kvcalloc(sizeof(*smap->buckets), nbuckets,
567+ GFP_USER | __GFP_NOWARN);
568+ if (!smap->buckets) {
569+ bpf_map_charge_finish(&smap->map.memory);
570+ kfree(smap);
571+ return ERR_PTR(-ENOMEM);
572+ }
573+
574+ for (i = 0; i < nbuckets; i++) {
575+ INIT_HLIST_HEAD(&smap->buckets[i].list);
576+ raw_spin_lock_init(&smap->buckets[i].lock);
577+ }
578+
579+ smap->elem_size =
580+ sizeof(struct bpf_local_storage_elem) + attr->value_size;
581+
582+ return smap;
583+}
584+
585+int bpf_local_storage_map_check_btf(const struct bpf_map *map,
586+ const struct btf *btf,
587+ const struct btf_type *key_type,
588+ const struct btf_type *value_type)
589+{
590+ u32 int_data;
591+
592+ if (BTF_INFO_KIND(key_type->info) != BTF_KIND_INT)
593+ return -EINVAL;
594+
595+ int_data = *(u32 *)(key_type + 1);
596+ if (BTF_INT_BITS(int_data) != 32 || BTF_INT_OFFSET(int_data))
597+ return -EINVAL;
598+
599+ return 0;
600+}
kernel/bpf/bpf_lsm.c+20−1
@@ -11,6 +11,8 @@ · 10 unmodified lines
1111 #include <linux/bpf_lsm.h>
1212 #include <linux/kallsyms.h>
1313 #include <linux/bpf_verifier.h>
14+#include <net/bpf_sk_storage.h>
15+#include <linux/bpf_local_storage.h>
1416
1517 /* For every LSM hook that allows attachment of BPF programs, declare a nop
1618 * function where a BPF program can be attached.
@@ -45,10 +47,27 @@ int bpf_lsm_verify_prog(struct bpf_verifier_log *vlog, · 28 unmodified lines
4547 return 0;
4648 }
4749
50+static const struct bpf_func_proto *
51+bpf_lsm_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
52+{
53+ switch (func_id) {
54+ case BPF_FUNC_inode_storage_get:
55+ return &bpf_inode_storage_get_proto;
56+ case BPF_FUNC_inode_storage_delete:
57+ return &bpf_inode_storage_delete_proto;
58+ case BPF_FUNC_sk_storage_get:
59+ return &bpf_sk_storage_get_proto;
60+ case BPF_FUNC_sk_storage_delete:
61+ return &bpf_sk_storage_delete_proto;
62+ default:
63+ return tracing_prog_func_proto(func_id, prog);
64+ }
65+}
66+
4867 const struct bpf_prog_ops lsm_prog_ops = {
4968 };
5069
5170 const struct bpf_verifier_ops lsm_verifier_ops = {
52− .get_func_proto = tracing_prog_func_proto,
71+ .get_func_proto = bpf_lsm_func_proto,
5372 .is_valid_access = btf_ctx_access,
5473 };
kernel/bpf/bpf_struct_ops.c+2−4
@@ -298,8 +298,7 @@ static int check_zero_holes(const struct btf_type *t, void *data) · 297 unmodified lines
298298 return -EINVAL;
299299
300300 mtype = btf_type_by_id(btf_vmlinux, member->type);
301− mtype = btf_resolve_size(btf_vmlinux, mtype, &msize,
302− NULL, NULL);
301+ mtype = btf_resolve_size(btf_vmlinux, mtype, &msize);
303302 if (IS_ERR(mtype))
304303 return PTR_ERR(mtype);
305304 prev_mend = moff + msize;
@@ -396,8 +395,7 @@ static int bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, · 90 unmodified lines
396395 u32 msize;
397396
398397 mtype = btf_type_by_id(btf_vmlinux, member->type);
399− mtype = btf_resolve_size(btf_vmlinux, mtype, &msize,
400− NULL, NULL);
398+ mtype = btf_resolve_size(btf_vmlinux, mtype, &msize);
401399 if (IS_ERR(mtype)) {
402400 err = PTR_ERR(mtype);
403401 goto reset_unlock;
kernel/bpf/btf.c+1054−167
@@ -21,6 +21,8 @@ · 20 unmodified lines
2121 #include <linux/btf_ids.h>
2222 #include <linux/skmsg.h>
2323 #include <linux/perf_event.h>
24+#include <linux/bsearch.h>
25+#include <linux/btf_ids.h>
2426 #include <net/sock.h>
2527
2628 /* BTF (BPF Type Format) is the meta data format which describes
@@ -186,11 +188,6 @@ · 159 unmodified lines
186188 i < btf_type_vlen(struct_type); \
187189 i++, member++)
188190
189−#define for_each_vsi(i, struct_type, member) \
190− for (i = 0, member = btf_type_var_secinfo(struct_type); \
191− i < btf_type_vlen(struct_type); \
192− i++, member++)
193−
194191 #define for_each_vsi_from(i, from, struct_type, member) \
195192 for (i = from, member = btf_type_var_secinfo(struct_type) + from; \
196193 i < btf_type_vlen(struct_type); \
@@ -282,6 +279,91 @@ static const char *btf_type_str(const struct btf_type *t) · 85 unmodified lines
282279 return btf_kind_str[BTF_INFO_KIND(t->info)];
283280 }
284281
282+/* Chunk size we use in safe copy of data to be shown. */
283+#define BTF_SHOW_OBJ_SAFE_SIZE 32
284+
285+/*
286+ * This is the maximum size of a base type value (equivalent to a
287+ * 128-bit int); if we are at the end of our safe buffer and have
288+ * less than 16 bytes space we can't be assured of being able
289+ * to copy the next type safely, so in such cases we will initiate
290+ * a new copy.
291+ */
292+#define BTF_SHOW_OBJ_BASE_TYPE_SIZE 16
293+
294+/* Type name size */
295+#define BTF_SHOW_NAME_SIZE 80
296+
297+/*
298+ * Common data to all BTF show operations. Private show functions can add
299+ * their own data to a structure containing a struct btf_show and consult it
300+ * in the show callback. See btf_type_show() below.
301+ *
302+ * One challenge with showing nested data is we want to skip 0-valued
303+ * data, but in order to figure out whether a nested object is all zeros
304+ * we need to walk through it. As a result, we need to make two passes
305+ * when handling structs, unions and arrays; the first path simply looks
306+ * for nonzero data, while the second actually does the display. The first
307+ * pass is signalled by show->state.depth_check being set, and if we
308+ * encounter a non-zero value we set show->state.depth_to_show to
309+ * the depth at which we encountered it. When we have completed the
310+ * first pass, we will know if anything needs to be displayed if
311+ * depth_to_show > depth. See btf_[struct,array]_show() for the
312+ * implementation of this.
313+ *
314+ * Another problem is we want to ensure the data for display is safe to
315+ * access. To support this, the anonymous "struct {} obj" tracks the data
316+ * object and our safe copy of it. We copy portions of the data needed
317+ * to the object "copy" buffer, but because its size is limited to
318+ * BTF_SHOW_OBJ_COPY_LEN bytes, multiple copies may be required as we
319+ * traverse larger objects for display.
320+ *
321+ * The various data type show functions all start with a call to
322+ * btf_show_start_type() which returns a pointer to the safe copy
323+ * of the data needed (or if BTF_SHOW_UNSAFE is specified, to the
324+ * raw data itself). btf_show_obj_safe() is responsible for
325+ * using copy_from_kernel_nofault() to update the safe data if necessary
326+ * as we traverse the object's data. skbuff-like semantics are
327+ * used:
328+ *
329+ * - obj.head points to the start of the toplevel object for display
330+ * - obj.size is the size of the toplevel object
331+ * - obj.data points to the current point in the original data at
332+ * which our safe data starts. obj.data will advance as we copy
333+ * portions of the data.
334+ *
335+ * In most cases a single copy will suffice, but larger data structures
336+ * such as "struct task_struct" will require many copies. The logic in
337+ * btf_show_obj_safe() handles the logic that determines if a new
338+ * copy_from_kernel_nofault() is needed.
339+ */
340+struct btf_show {
341+ u64 flags;
342+ void *target; /* target of show operation (seq file, buffer) */
343+ void (*showfn)(struct btf_show *show, const char *fmt, va_list args);
344+ const struct btf *btf;
345+ /* below are used during iteration */
346+ struct {
347+ u8 depth;
348+ u8 depth_to_show;
349+ u8 depth_check;
350+ u8 array_member:1,
351+ array_terminated:1;
352+ u16 array_encoding;
353+ u32 type_id;
354+ int status; /* non-zero for error */
355+ const struct btf_type *type;
356+ const struct btf_member *member;
357+ char name[BTF_SHOW_NAME_SIZE]; /* space for member name/type */
358+ } state;
359+ struct {
360+ u32 size;
361+ void *head;
362+ void *data;
363+ u8 safe[BTF_SHOW_OBJ_SAFE_SIZE];
364+ } obj;
365+};
366+
285367 struct btf_kind_operations {
286368 s32 (*check_meta)(struct btf_verifier_env *env,
287369 const struct btf_type *t,
@@ -298,9 +380,9 @@ struct btf_kind_operations { · 10 unmodified lines
298380 const struct btf_type *member_type);
299381 void (*log_details)(struct btf_verifier_env *env,
300382 const struct btf_type *t);
301− void (*seq_show)(const struct btf *btf, const struct btf_type *t,
383+ void (*show)(const struct btf *btf, const struct btf_type *t,
302384 u32 type_id, void *data, u8 bits_offsets,
303− struct seq_file *m);
385+ struct btf_show *show);
304386 };
305387
306388 static const struct btf_kind_operations * const kind_ops[NR_BTF_KINDS];
@@ -353,16 +435,6 @@ static bool btf_type_nosize_or_null(const struct btf_type *t) · 46 unmodified lines
353435 return !t || btf_type_nosize(t);
354436 }
355437
356−/* union is only a special case of struct:
357− * all its offsetof(member) == 0
358− */
359−static bool btf_type_is_struct(const struct btf_type *t)
360−{
361− u8 kind = BTF_INFO_KIND(t->info);
362−
363− return kind == BTF_KIND_STRUCT || kind == BTF_KIND_UNION;
364−}
365−
366438 static bool __btf_type_is_struct(const struct btf_type *t)
367439 {
368440 return BTF_INFO_KIND(t->info) == BTF_KIND_STRUCT;
@@ -373,11 +445,6 @@ static bool btf_type_is_array(const struct btf_type *t) · 4 unmodified lines
373445 return BTF_INFO_KIND(t->info) == BTF_KIND_ARRAY;
374446 }
375447
376−static bool btf_type_is_var(const struct btf_type *t)
377−{
378− return BTF_INFO_KIND(t->info) == BTF_KIND_VAR;
379−}
380−
381448 static bool btf_type_is_datasec(const struct btf_type *t)
382449 {
383450 return BTF_INFO_KIND(t->info) == BTF_KIND_DATASEC;
@@ -526,11 +593,6 @@ static const struct btf_var *btf_type_var(const struct btf_type *t) · 142 unmodified lines
526593 return (const struct btf_var *)(t + 1);
527594 }
528595
529−static const struct btf_var_secinfo *btf_type_var_secinfo(const struct btf_type *t)
530−{
531− return (const struct btf_var_secinfo *)(t + 1);
532−}
533−
534596 static const struct btf_kind_operations *btf_type_ops(const struct btf_type *t)
535597 {
536598 return kind_ops[BTF_INFO_KIND(t->info)];
@@ -677,6 +739,488 @@ bool btf_member_is_reg_int(const struct btf *btf, const struct btf_type *s, · 140 unmodified lines
677739 return true;
678740 }
679741
742+/* Similar to btf_type_skip_modifiers() but does not skip typedefs. */
743+static const struct btf_type *btf_type_skip_qualifiers(const struct btf *btf,
744+ u32 id)
745+{
746+ const struct btf_type *t = btf_type_by_id(btf, id);
747+
748+ while (btf_type_is_modifier(t) &&
749+ BTF_INFO_KIND(t->info) != BTF_KIND_TYPEDEF) {
750+ id = t->type;
751+ t = btf_type_by_id(btf, t->type);
752+ }
753+
754+ return t;
755+}
756+
757+#define BTF_SHOW_MAX_ITER 10
758+
759+#define BTF_KIND_BIT(kind) (1ULL << kind)
760+
761+/*
762+ * Populate show->state.name with type name information.
763+ * Format of type name is
764+ *
765+ * [.member_name = ] (type_name)
766+ */
767+static const char *btf_show_name(struct btf_show *show)
768+{
769+ /* BTF_MAX_ITER array suffixes "[]" */
770+ const char *array_suffixes = "[][][][][][][][][][]";
771+ const char *array_suffix = &array_suffixes[strlen(array_suffixes)];
772+ /* BTF_MAX_ITER pointer suffixes "*" */
773+ const char *ptr_suffixes = "**********";
774+ const char *ptr_suffix = &ptr_suffixes[strlen(ptr_suffixes)];
775+ const char *name = NULL, *prefix = "", *parens = "";
776+ const struct btf_member *m = show->state.member;
777+ const struct btf_type *t = show->state.type;
778+ const struct btf_array *array;
779+ u32 id = show->state.type_id;
780+ const char *member = NULL;
781+ bool show_member = false;
782+ u64 kinds = 0;
783+ int i;
784+
785+ show->state.name[0] = '\0';
786+
787+ /*
788+ * Don't show type name if we're showing an array member;
789+ * in that case we show the array type so don't need to repeat
790+ * ourselves for each member.
791+ */
792+ if (show->state.array_member)
793+ return "";
794+
795+ /* Retrieve member name, if any. */
796+ if (m) {
797+ member = btf_name_by_offset(show->btf, m->name_off);
798+ show_member = strlen(member) > 0;
799+ id = m->type;
800+ }
801+
802+ /*
803+ * Start with type_id, as we have resolved the struct btf_type *
804+ * via btf_modifier_show() past the parent typedef to the child
805+ * struct, int etc it is defined as. In such cases, the type_id
806+ * still represents the starting type while the struct btf_type *
807+ * in our show->state points at the resolved type of the typedef.
808+ */
809+ t = btf_type_by_id(show->btf, id);
810+ if (!t)
811+ return "";
812+
813+ /*
814+ * The goal here is to build up the right number of pointer and
815+ * array suffixes while ensuring the type name for a typedef
816+ * is represented. Along the way we accumulate a list of
817+ * BTF kinds we have encountered, since these will inform later
818+ * display; for example, pointer types will not require an
819+ * opening "{" for struct, we will just display the pointer value.
820+ *
821+ * We also want to accumulate the right number of pointer or array
822+ * indices in the format string while iterating until we get to
823+ * the typedef/pointee/array member target type.
824+ *
825+ * We start by pointing at the end of pointer and array suffix
826+ * strings; as we accumulate pointers and arrays we move the pointer
827+ * or array string backwards so it will show the expected number of
828+ * '*' or '[]' for the type. BTF_SHOW_MAX_ITER of nesting of pointers
829+ * and/or arrays and typedefs are supported as a precaution.
830+ *
831+ * We also want to get typedef name while proceeding to resolve
832+ * type it points to so that we can add parentheses if it is a
833+ * "typedef struct" etc.
834+ */
835+ for (i = 0; i < BTF_SHOW_MAX_ITER; i++) {
836+
837+ switch (BTF_INFO_KIND(t->info)) {
838+ case BTF_KIND_TYPEDEF:
839+ if (!name)
840+ name = btf_name_by_offset(show->btf,
841+ t->name_off);
842+ kinds |= BTF_KIND_BIT(BTF_KIND_TYPEDEF);
843+ id = t->type;
844+ break;
845+ case BTF_KIND_ARRAY:
846+ kinds |= BTF_KIND_BIT(BTF_KIND_ARRAY);
847+ parens = "[";
848+ if (!t)
849+ return "";
850+ array = btf_type_array(t);
851+ if (array_suffix > array_suffixes)
852+ array_suffix -= 2;
853+ id = array->type;
854+ break;
855+ case BTF_KIND_PTR:
856+ kinds |= BTF_KIND_BIT(BTF_KIND_PTR);
857+ if (ptr_suffix > ptr_suffixes)
858+ ptr_suffix -= 1;
859+ id = t->type;
860+ break;
861+ default:
862+ id = 0;
863+ break;
864+ }
865+ if (!id)
866+ break;
867+ t = btf_type_skip_qualifiers(show->btf, id);
868+ }
869+ /* We may not be able to represent this type; bail to be safe */
870+ if (i == BTF_SHOW_MAX_ITER)
871+ return "";
872+
873+ if (!name)
874+ name = btf_name_by_offset(show->btf, t->name_off);
875+
876+ switch (BTF_INFO_KIND(t->info)) {
877+ case BTF_KIND_STRUCT:
878+ case BTF_KIND_UNION:
879+ prefix = BTF_INFO_KIND(t->info) == BTF_KIND_STRUCT ?
880+ "struct" : "union";
881+ /* if it's an array of struct/union, parens is already set */
882+ if (!(kinds & (BTF_KIND_BIT(BTF_KIND_ARRAY))))
883+ parens = "{";
884+ break;
885+ case BTF_KIND_ENUM:
886+ prefix = "enum";
887+ break;
888+ default:
889+ break;
890+ }
891+
892+ /* pointer does not require parens */
893+ if (kinds & BTF_KIND_BIT(BTF_KIND_PTR))
894+ parens = "";
895+ /* typedef does not require struct/union/enum prefix */
896+ if (kinds & BTF_KIND_BIT(BTF_KIND_TYPEDEF))
897+ prefix = "";
898+
899+ if (!name)
900+ name = "";
901+
902+ /* Even if we don't want type name info, we want parentheses etc */
903+ if (show->flags & BTF_SHOW_NONAME)
904+ snprintf(show->state.name, sizeof(show->state.name), "%s",
905+ parens);
906+ else
907+ snprintf(show->state.name, sizeof(show->state.name),
908+ "%s%s%s(%s%s%s%s%s%s)%s",
909+ /* first 3 strings comprise ".member = " */
910+ show_member ? "." : "",
911+ show_member ? member : "",
912+ show_member ? " = " : "",
913+ /* ...next is our prefix (struct, enum, etc) */
914+ prefix,
915+ strlen(prefix) > 0 && strlen(name) > 0 ? " " : "",
916+ /* ...this is the type name itself */
917+ name,
918+ /* ...suffixed by the appropriate '*', '[]' suffixes */
919+ strlen(ptr_suffix) > 0 ? " " : "", ptr_suffix,
920+ array_suffix, parens);
921+
922+ return show->state.name;
923+}
924+
925+static const char *__btf_show_indent(struct btf_show *show)
926+{
927+ const char *indents = " ";
928+ const char *indent = &indents[strlen(indents)];
929+
930+ if ((indent - show->state.depth) >= indents)
931+ return indent - show->state.depth;
932+ return indents;
933+}
934+
935+static const char *btf_show_indent(struct btf_show *show)
936+{
937+ return show->flags & BTF_SHOW_COMPACT ? "" : __btf_show_indent(show);
938+}
939+
940+static const char *btf_show_newline(struct btf_show *show)
941+{
942+ return show->flags & BTF_SHOW_COMPACT ? "" : "\n";
943+}
944+
945+static const char *btf_show_delim(struct btf_show *show)
946+{
947+ if (show->state.depth == 0)
948+ return "";
949+
950+ if ((show->flags & BTF_SHOW_COMPACT) && show->state.type &&
951+ BTF_INFO_KIND(show->state.type->info) == BTF_KIND_UNION)
952+ return "|";
953+
954+ return ",";
955+}
956+
957+__printf(2, 3) static void btf_show(struct btf_show *show, const char *fmt, ...)
958+{
959+ va_list args;
960+
961+ if (!show->state.depth_check) {
962+ va_start(args, fmt);
963+ show->showfn(show, fmt, args);
964+ va_end(args);
965+ }
966+}
967+
968+/* Macros are used here as btf_show_type_value[s]() prepends and appends
969+ * format specifiers to the format specifier passed in; these do the work of
970+ * adding indentation, delimiters etc while the caller simply has to specify
971+ * the type value(s) in the format specifier + value(s).
972+ */
973+#define btf_show_type_value(show, fmt, value) \
974+ do { \
975+ if ((value) != 0 || (show->flags & BTF_SHOW_ZERO) || \
976+ show->state.depth == 0) { \
977+ btf_show(show, "%s%s" fmt "%s%s", \
978+ btf_show_indent(show), \
979+ btf_show_name(show), \
980+ value, btf_show_delim(show), \
981+ btf_show_newline(show)); \
982+ if (show->state.depth > show->state.depth_to_show) \
983+ show->state.depth_to_show = show->state.depth; \
984+ } \
985+ } while (0)
986+
987+#define btf_show_type_values(show, fmt, ...) \
988+ do { \
989+ btf_show(show, "%s%s" fmt "%s%s", btf_show_indent(show), \
990+ btf_show_name(show), \
991+ __VA_ARGS__, btf_show_delim(show), \
992+ btf_show_newline(show)); \
993+ if (show->state.depth > show->state.depth_to_show) \
994+ show->state.depth_to_show = show->state.depth; \
995+ } while (0)
996+
997+/* How much is left to copy to safe buffer after @data? */
998+static int btf_show_obj_size_left(struct btf_show *show, void *data)
999+{
1000+ return show->obj.head + show->obj.size - data;
1001+}
1002+
1003+/* Is object pointed to by @data of @size already copied to our safe buffer? */
1004+static bool btf_show_obj_is_safe(struct btf_show *show, void *data, int size)
1005+{
1006+ return data >= show->obj.data &&
1007+ (data + size) < (show->obj.data + BTF_SHOW_OBJ_SAFE_SIZE);
1008+}
1009+
1010+/*
1011+ * If object pointed to by @data of @size falls within our safe buffer, return
1012+ * the equivalent pointer to the same safe data. Assumes
1013+ * copy_from_kernel_nofault() has already happened and our safe buffer is
1014+ * populated.
1015+ */
1016+static void *__btf_show_obj_safe(struct btf_show *show, void *data, int size)
1017+{
1018+ if (btf_show_obj_is_safe(show, data, size))
1019+ return show->obj.safe + (data - show->obj.data);
1020+ return NULL;
1021+}
1022+
1023+/*
1024+ * Return a safe-to-access version of data pointed to by @data.
1025+ * We do this by copying the relevant amount of information
1026+ * to the struct btf_show obj.safe buffer using copy_from_kernel_nofault().
1027+ *
1028+ * If BTF_SHOW_UNSAFE is specified, just return data as-is; no
1029+ * safe copy is needed.
1030+ *
1031+ * Otherwise we need to determine if we have the required amount
1032+ * of data (determined by the @data pointer and the size of the
1033+ * largest base type we can encounter (represented by
1034+ * BTF_SHOW_OBJ_BASE_TYPE_SIZE). Having that much data ensures
1035+ * that we will be able to print some of the current object,
1036+ * and if more is needed a copy will be triggered.
1037+ * Some objects such as structs will not fit into the buffer;
1038+ * in such cases additional copies when we iterate over their
1039+ * members may be needed.
1040+ *
1041+ * btf_show_obj_safe() is used to return a safe buffer for
1042+ * btf_show_start_type(); this ensures that as we recurse into
1043+ * nested types we always have safe data for the given type.
1044+ * This approach is somewhat wasteful; it's possible for example
1045+ * that when iterating over a large union we'll end up copying the
1046+ * same data repeatedly, but the goal is safety not performance.
1047+ * We use stack data as opposed to per-CPU buffers because the
1048+ * iteration over a type can take some time, and preemption handling
1049+ * would greatly complicate use of the safe buffer.
1050+ */
1051+static void *btf_show_obj_safe(struct btf_show *show,
1052+ const struct btf_type *t,
1053+ void *data)
1054+{
1055+ const struct btf_type *rt;
1056+ int size_left, size;
1057+ void *safe = NULL;
1058+
1059+ if (show->flags & BTF_SHOW_UNSAFE)
1060+ return data;
1061+
1062+ rt = btf_resolve_size(show->btf, t, &size);
1063+ if (IS_ERR(rt)) {
1064+ show->state.status = PTR_ERR(rt);
1065+ return NULL;
1066+ }
1067+
1068+ /*
1069+ * Is this toplevel object? If so, set total object size and
1070+ * initialize pointers. Otherwise check if we still fall within
1071+ * our safe object data.
1072+ */
1073+ if (show->state.depth == 0) {
1074+ show->obj.size = size;
1075+ show->obj.head = data;
1076+ } else {
1077+ /*
1078+ * If the size of the current object is > our remaining
1079+ * safe buffer we _may_ need to do a new copy. However
1080+ * consider the case of a nested struct; it's size pushes
1081+ * us over the safe buffer limit, but showing any individual
1082+ * struct members does not. In such cases, we don't need
1083+ * to initiate a fresh copy yet; however we definitely need
1084+ * at least BTF_SHOW_OBJ_BASE_TYPE_SIZE bytes left
1085+ * in our buffer, regardless of the current object size.
1086+ * The logic here is that as we resolve types we will
1087+ * hit a base type at some point, and we need to be sure
1088+ * the next chunk of data is safely available to display
1089+ * that type info safely. We cannot rely on the size of
1090+ * the current object here because it may be much larger
1091+ * than our current buffer (e.g. task_struct is 8k).
1092+ * All we want to do here is ensure that we can print the
1093+ * next basic type, which we can if either
1094+ * - the current type size is within the safe buffer; or
1095+ * - at least BTF_SHOW_OBJ_BASE_TYPE_SIZE bytes are left in
1096+ * the safe buffer.
1097+ */
1098+ safe = __btf_show_obj_safe(show, data,
1099+ min(size,
1100+ BTF_SHOW_OBJ_BASE_TYPE_SIZE));
1101+ }
1102+
1103+ /*
1104+ * We need a new copy to our safe object, either because we haven't
1105+ * yet copied and are intializing safe data, or because the data
1106+ * we want falls outside the boundaries of the safe object.
1107+ */
1108+ if (!safe) {
1109+ size_left = btf_show_obj_size_left(show, data);
1110+ if (size_left > BTF_SHOW_OBJ_SAFE_SIZE)
1111+ size_left = BTF_SHOW_OBJ_SAFE_SIZE;
1112+ show->state.status = copy_from_kernel_nofault(show->obj.safe,
1113+ data, size_left);
1114+ if (!show->state.status) {
1115+ show->obj.data = data;
1116+ safe = show->obj.safe;
1117+ }
1118+ }
1119+
1120+ return safe;
1121+}
1122+
1123+/*
1124+ * Set the type we are starting to show and return a safe data pointer
1125+ * to be used for showing the associated data.
1126+ */
1127+static void *btf_show_start_type(struct btf_show *show,
1128+ const struct btf_type *t,
1129+ u32 type_id, void *data)
1130+{
1131+ show->state.type = t;
1132+ show->state.type_id = type_id;
1133+ show->state.name[0] = '\0';
1134+
1135+ return btf_show_obj_safe(show, t, data);
1136+}
1137+
1138+static void btf_show_end_type(struct btf_show *show)
1139+{
1140+ show->state.type = NULL;
1141+ show->state.type_id = 0;
1142+ show->state.name[0] = '\0';
1143+}
1144+
1145+static void *btf_show_start_aggr_type(struct btf_show *show,
1146+ const struct btf_type *t,
1147+ u32 type_id, void *data)
1148+{
1149+ void *safe_data = btf_show_start_type(show, t, type_id, data);
1150+
1151+ if (!safe_data)
1152+ return safe_data;
1153+
1154+ btf_show(show, "%s%s%s", btf_show_indent(show),
1155+ btf_show_name(show),
1156+ btf_show_newline(show));
1157+ show->state.depth++;
1158+ return safe_data;
1159+}
1160+
1161+static void btf_show_end_aggr_type(struct btf_show *show,
1162+ const char *suffix)
1163+{
1164+ show->state.depth--;
1165+ btf_show(show, "%s%s%s%s", btf_show_indent(show), suffix,
1166+ btf_show_delim(show), btf_show_newline(show));
1167+ btf_show_end_type(show);
1168+}
1169+
1170+static void btf_show_start_member(struct btf_show *show,
1171+ const struct btf_member *m)
1172+{
1173+ show->state.member = m;
1174+}
1175+
1176+static void btf_show_start_array_member(struct btf_show *show)
1177+{
1178+ show->state.array_member = 1;
1179+ btf_show_start_member(show, NULL);
1180+}
1181+
1182+static void btf_show_end_member(struct btf_show *show)
1183+{
1184+ show->state.member = NULL;
1185+}
1186+
1187+static void btf_show_end_array_member(struct btf_show *show)
1188+{
1189+ show->state.array_member = 0;
1190+ btf_show_end_member(show);
1191+}
1192+
1193+static void *btf_show_start_array_type(struct btf_show *show,
1194+ const struct btf_type *t,
1195+ u32 type_id,
1196+ u16 array_encoding,
1197+ void *data)
1198+{
1199+ show->state.array_encoding = array_encoding;
1200+ show->state.array_terminated = 0;
1201+ return btf_show_start_aggr_type(show, t, type_id, data);
1202+}
1203+
1204+static void btf_show_end_array_type(struct btf_show *show)
1205+{
1206+ show->state.array_encoding = 0;
1207+ show->state.array_terminated = 0;
1208+ btf_show_end_aggr_type(show, "]");
1209+}
1210+
1211+static void *btf_show_start_struct_type(struct btf_show *show,
1212+ const struct btf_type *t,
1213+ u32 type_id,
1214+ void *data)
1215+{
1216+ return btf_show_start_aggr_type(show, t, type_id, data);
1217+}
1218+
1219+static void btf_show_end_struct_type(struct btf_show *show)
1220+{
1221+ btf_show_end_aggr_type(show, "}");
1222+}
1223+
6801224 __printf(2, 3) static void __btf_verifier_log(struct bpf_verifier_log *log,
6811225 const char *fmt, ...)
6821226 {
@@ -1079,23 +1623,27 @@ static const struct resolve_vertex *env_stack_peak(struct btf_verifier_env *env) · 396 unmodified lines
10791623 * *type_size: (x * y * sizeof(u32)). Hence, *type_size always
10801624 * corresponds to the return type.
10811625 * *elem_type: u32
1626+ * *elem_id: id of u32
10821627 * *total_nelems: (x * y). Hence, individual elem size is
10831628 * (*type_size / *total_nelems)
1629+ * *type_id: id of type if it's changed within the function, 0 if not
10841630 *
10851631 * type: is not an array (e.g. const struct X)
10861632 * return type: type "struct X"
10871633 * *type_size: sizeof(struct X)
10881634 * *elem_type: same as return type ("struct X")
1635+ * *elem_id: 0
10891636 * *total_nelems: 1
1637+ * *type_id: id of type if it's changed within the function, 0 if not
10901638 */
1091−const struct btf_type *
1092−btf_resolve_size(const struct btf *btf, const struct btf_type *type,
1093− u32 *type_size, const struct btf_type **elem_type,
1094− u32 *total_nelems)
1639+static const struct btf_type *
1640+__btf_resolve_size(const struct btf *btf, const struct btf_type *type,
1641+ u32 *type_size, const struct btf_type **elem_type,
1642+ u32 *elem_id, u32 *total_nelems, u32 *type_id)
10951643 {
10961644 const struct btf_type *array_type = NULL;
1097− const struct btf_array *array;
1098− u32 i, size, nelems = 1;
1645+ const struct btf_array *array = NULL;
1646+ u32 i, size, nelems = 1, id = 0;
10991647
11001648 for (i = 0; i < MAX_RESOLVE_DEPTH; i++) {
11011649 switch (BTF_INFO_KIND(type->info)) {
@@ -1116,6 +1664,7 @@ btf_resolve_size(const struct btf *btf, const struct btf_type *type, · 14 unmodified lines
11161664 case BTF_KIND_VOLATILE:
11171665 case BTF_KIND_CONST:
11181666 case BTF_KIND_RESTRICT:
1667+ id = type->type;
11191668 type = btf_type_by_id(btf, type->type);
11201669 break;
11211670
@@ -1146,10 +1695,21 @@ resolved: · 24 unmodified lines
11461695 *total_nelems = nelems;
11471696 if (elem_type)
11481697 *elem_type = type;
1698+ if (elem_id)
1699+ *elem_id = array ? array->type : 0;
1700+ if (type_id && id)
1701+ *type_id = id;
11491702
11501703 return array_type ? : type;
11511704 }
11521705
1706+const struct btf_type *
1707+btf_resolve_size(const struct btf *btf, const struct btf_type *type,
1708+ u32 *type_size)
1709+{
1710+ return __btf_resolve_size(btf, type, type_size, NULL, NULL, NULL, NULL);
1711+}
1712+
11531713 /* The input param "type_id" must point to a needs_resolve type */
11541714 static const struct btf_type *btf_type_id_resolve(const struct btf *btf,
11551715 u32 *type_id)
@@ -1250,11 +1810,11 @@ static int btf_df_resolve(struct btf_verifier_env *env, · 94 unmodified lines
12501810 return -EINVAL;
12511811 }
12521812
1253−static void btf_df_seq_show(const struct btf *btf, const struct btf_type *t,
1254− u32 type_id, void *data, u8 bits_offsets,
1255− struct seq_file *m)
1813+static void btf_df_show(const struct btf *btf, const struct btf_type *t,
1814+ u32 type_id, void *data, u8 bits_offsets,
1815+ struct btf_show *show)
12561816 {
1257− seq_printf(m, "<unsupported kind:%u>", BTF_INFO_KIND(t->info));
1817+ btf_show(show, "<unsupported kind:%u>", BTF_INFO_KIND(t->info));
12581818 }
12591819
12601820 static int btf_int_check_member(struct btf_verifier_env *env,
@@ -1427,7 +1987,7 @@ static void btf_int_log(struct btf_verifier_env *env, · 166 unmodified lines
14271987 btf_int_encoding_str(BTF_INT_ENCODING(int_data)));
14281988 }
14291989
1430−static void btf_int128_print(struct seq_file *m, void *data)
1990+static void btf_int128_print(struct btf_show *show, void *data)
14311991 {
14321992 /* data points to a __int128 number.
14331993 * Suppose
@@ -1446,9 +2006,10 @@ static void btf_int128_print(struct seq_file *m, void *data) · 12 unmodified lines
14462006 lower_num = *(u64 *)data;
14472007 #endif
14482008 if (upper_num == 0)
1449− seq_printf(m, "0x%llx", lower_num);
2009+ btf_show_type_value(show, "0x%llx", lower_num);
14502010 else
1451− seq_printf(m, "0x%llx%016llx", upper_num, lower_num);
2011+ btf_show_type_values(show, "0x%llx%016llx", upper_num,
2012+ lower_num);
14522013 }
14532014
14542015 static void btf_int128_shift(u64 *print_num, u16 left_shift_bits,
@@ -1492,8 +2053,8 @@ static void btf_int128_shift(u64 *print_num, u16 left_shift_bits, · 37 unmodified lines
14922053 #endif
14932054 }
14942055
1495−static void btf_bitfield_seq_show(void *data, u8 bits_offset,
1496− u8 nr_bits, struct seq_file *m)
2056+static void btf_bitfield_show(void *data, u8 bits_offset,
2057+ u8 nr_bits, struct btf_show *show)
14972058 {
14982059 u16 left_shift_bits, right_shift_bits;
14992060 u8 nr_copy_bytes;
@@ -1513,14 +2074,14 @@ static void btf_bitfield_seq_show(void *data, u8 bits_offset, · 13 unmodified lines
15132074 right_shift_bits = BITS_PER_U128 - nr_bits;
15142075
15152076 btf_int128_shift(print_num, left_shift_bits, right_shift_bits);
1516− btf_int128_print(m, print_num);
2077+ btf_int128_print(show, print_num);
15172078 }
15182079
15192080
1520−static void btf_int_bits_seq_show(const struct btf *btf,
1521− const struct btf_type *t,
1522− void *data, u8 bits_offset,
1523− struct seq_file *m)
2081+static void btf_int_bits_show(const struct btf *btf,
2082+ const struct btf_type *t,
2083+ void *data, u8 bits_offset,
2084+ struct btf_show *show)
15242085 {
15252086 u32 int_data = btf_type_int(t);
15262087 u8 nr_bits = BTF_INT_BITS(int_data);
@@ -1533,55 +2094,77 @@ static void btf_int_bits_seq_show(const struct btf *btf, · 6 unmodified lines
15332094 total_bits_offset = bits_offset + BTF_INT_OFFSET(int_data);
15342095 data += BITS_ROUNDDOWN_BYTES(total_bits_offset);
15352096 bits_offset = BITS_PER_BYTE_MASKED(total_bits_offset);
1536− btf_bitfield_seq_show(data, bits_offset, nr_bits, m);
2097+ btf_bitfield_show(data, bits_offset, nr_bits, show);
15372098 }
15382099
1539−static void btf_int_seq_show(const struct btf *btf, const struct btf_type *t,
1540− u32 type_id, void *data, u8 bits_offset,
1541− struct seq_file *m)
2100+static void btf_int_show(const struct btf *btf, const struct btf_type *t,
2101+ u32 type_id, void *data, u8 bits_offset,
2102+ struct btf_show *show)
15422103 {
15432104 u32 int_data = btf_type_int(t);
15442105 u8 encoding = BTF_INT_ENCODING(int_data);
15452106 bool sign = encoding & BTF_INT_SIGNED;
15462107 u8 nr_bits = BTF_INT_BITS(int_data);
2108+ void *safe_data;
2109+
2110+ safe_data = btf_show_start_type(show, t, type_id, data);
2111+ if (!safe_data)
2112+ return;
15472113
15482114 if (bits_offset || BTF_INT_OFFSET(int_data) ||
15492115 BITS_PER_BYTE_MASKED(nr_bits)) {
1550− btf_int_bits_seq_show(btf, t, data, bits_offset, m);
1551− return;
2116+ btf_int_bits_show(btf, t, safe_data, bits_offset, show);
2117+ goto out;
15522118 }
15532119
15542120 switch (nr_bits) {
15552121 case 128:
1556− btf_int128_print(m, data);
2122+ btf_int128_print(show, safe_data);
15572123 break;
15582124 case 64:
15592125 if (sign)
1560− seq_printf(m, "%lld", *(s64 *)data);
2126+ btf_show_type_value(show, "%lld", *(s64 *)safe_data);
15612127 else
1562− seq_printf(m, "%llu", *(u64 *)data);
2128+ btf_show_type_value(show, "%llu", *(u64 *)safe_data);
15632129 break;
15642130 case 32:
15652131 if (sign)
1566− seq_printf(m, "%d", *(s32 *)data);
2132+ btf_show_type_value(show, "%d", *(s32 *)safe_data);
15672133 else
1568− seq_printf(m, "%u", *(u32 *)data);
2134+ btf_show_type_value(show, "%u", *(u32 *)safe_data);
15692135 break;
15702136 case 16:
15712137 if (sign)
1572− seq_printf(m, "%d", *(s16 *)data);
2138+ btf_show_type_value(show, "%d", *(s16 *)safe_data);
15732139 else
1574− seq_printf(m, "%u", *(u16 *)data);
2140+ btf_show_type_value(show, "%u", *(u16 *)safe_data);
15752141 break;
15762142 case 8:
2143+ if (show->state.array_encoding == BTF_INT_CHAR) {
2144+ /* check for null terminator */
2145+ if (show->state.array_terminated)
2146+ break;
2147+ if (*(char *)data == '\0') {
2148+ show->state.array_terminated = 1;
2149+ break;
2150+ }
2151+ if (isprint(*(char *)data)) {
2152+ btf_show_type_value(show, "'%c'",
2153+ *(char *)safe_data);
2154+ break;
2155+ }
2156+ }
15772157 if (sign)
1578− seq_printf(m, "%d", *(s8 *)data);
2158+ btf_show_type_value(show, "%d", *(s8 *)safe_data);
15792159 else
1580− seq_printf(m, "%u", *(u8 *)data);
2160+ btf_show_type_value(show, "%u", *(u8 *)safe_data);
15812161 break;
15822162 default:
1583− btf_int_bits_seq_show(btf, t, data, bits_offset, m);
2163+ btf_int_bits_show(btf, t, safe_data, bits_offset, show);
2164+ break;
15842165 }
2166+out:
2167+ btf_show_end_type(show);
15852168 }
15862169
15872170 static const struct btf_kind_operations int_ops = {
@@ -1590,7 +2173,7 @@ static const struct btf_kind_operations int_ops = { · 2 unmodified lines
15902173 .check_member = btf_int_check_member,
15912174 .check_kflag_member = btf_int_check_kflag_member,
15922175 .log_details = btf_int_log,
1593− .seq_show = btf_int_seq_show,
2176+ .show = btf_int_show,
15942177 };
15952178
15962179 static int btf_modifier_check_member(struct btf_verifier_env *env,
@@ -1854,34 +2437,44 @@ static int btf_ptr_resolve(struct btf_verifier_env *env, · 257 unmodified lines
18542437 return 0;
18552438 }
18562439
1857−static void btf_modifier_seq_show(const struct btf *btf,
1858− const struct btf_type *t,
1859− u32 type_id, void *data,
1860− u8 bits_offset, struct seq_file *m)
2440+static void btf_modifier_show(const struct btf *btf,
2441+ const struct btf_type *t,
2442+ u32 type_id, void *data,
2443+ u8 bits_offset, struct btf_show *show)
18612444 {
18622445 if (btf->resolved_ids)
18632446 t = btf_type_id_resolve(btf, &type_id);
18642447 else
18652448 t = btf_type_skip_modifiers(btf, type_id, NULL);
18662449
1867− btf_type_ops(t)->seq_show(btf, t, type_id, data, bits_offset, m);
2450+ btf_type_ops(t)->show(btf, t, type_id, data, bits_offset, show);
18682451 }
18692452
1870−static void btf_var_seq_show(const struct btf *btf, const struct btf_type *t,
1871− u32 type_id, void *data, u8 bits_offset,
1872− struct seq_file *m)
2453+static void btf_var_show(const struct btf *btf, const struct btf_type *t,
2454+ u32 type_id, void *data, u8 bits_offset,
2455+ struct btf_show *show)
18732456 {
18742457 t = btf_type_id_resolve(btf, &type_id);
18752458
1876− btf_type_ops(t)->seq_show(btf, t, type_id, data, bits_offset, m);
2459+ btf_type_ops(t)->show(btf, t, type_id, data, bits_offset, show);
18772460 }
18782461
1879−static void btf_ptr_seq_show(const struct btf *btf, const struct btf_type *t,
1880− u32 type_id, void *data, u8 bits_offset,
1881− struct seq_file *m)
2462+static void btf_ptr_show(const struct btf *btf, const struct btf_type *t,
2463+ u32 type_id, void *data, u8 bits_offset,
2464+ struct btf_show *show)
18822465 {
1883− /* It is a hashed value */
1884− seq_printf(m, "%p", *(void **)data);
2466+ void *safe_data;
2467+
2468+ safe_data = btf_show_start_type(show, t, type_id, data);
2469+ if (!safe_data)
2470+ return;
2471+
2472+ /* It is a hashed value unless BTF_SHOW_PTR_RAW is specified */
2473+ if (show->flags & BTF_SHOW_PTR_RAW)
2474+ btf_show_type_value(show, "0x%px", *(void **)safe_data);
2475+ else
2476+ btf_show_type_value(show, "0x%p", *(void **)safe_data);
2477+ btf_show_end_type(show);
18852478 }
18862479
18872480 static void btf_ref_type_log(struct btf_verifier_env *env,
@@ -1896,7 +2489,7 @@ static struct btf_kind_operations modifier_ops = { · 8 unmodified lines
18962489 .check_member = btf_modifier_check_member,
18972490 .check_kflag_member = btf_modifier_check_kflag_member,
18982491 .log_details = btf_ref_type_log,
1899− .seq_show = btf_modifier_seq_show,
2492+ .show = btf_modifier_show,
19002493 };
19012494
19022495 static struct btf_kind_operations ptr_ops = {
@@ -1905,7 +2498,7 @@ static struct btf_kind_operations ptr_ops = { · 2 unmodified lines
19052498 .check_member = btf_ptr_check_member,
19062499 .check_kflag_member = btf_generic_check_kflag_member,
19072500 .log_details = btf_ref_type_log,
1908− .seq_show = btf_ptr_seq_show,
2501+ .show = btf_ptr_show,
19092502 };
19102503
19112504 static s32 btf_fwd_check_meta(struct btf_verifier_env *env,
@@ -1946,7 +2539,7 @@ static struct btf_kind_operations fwd_ops = { · 34 unmodified lines
19462539 .check_member = btf_df_check_member,
19472540 .check_kflag_member = btf_df_check_kflag_member,
19482541 .log_details = btf_fwd_type_log,
1949− .seq_show = btf_df_seq_show,
2542+ .show = btf_df_show,
19502543 };
19512544
19522545 static int btf_array_check_member(struct btf_verifier_env *env,
@@ -2105,28 +2698,90 @@ static void btf_array_log(struct btf_verifier_env *env, · 152 unmodified lines
21052698 array->type, array->index_type, array->nelems);
21062699 }
21072700
2108−static void btf_array_seq_show(const struct btf *btf, const struct btf_type *t,
2109− u32 type_id, void *data, u8 bits_offset,
2110− struct seq_file *m)
2701+static void __btf_array_show(const struct btf *btf, const struct btf_type *t,
2702+ u32 type_id, void *data, u8 bits_offset,
2703+ struct btf_show *show)
21112704 {
21122705 const struct btf_array *array = btf_type_array(t);
21132706 const struct btf_kind_operations *elem_ops;
21142707 const struct btf_type *elem_type;
2115− u32 i, elem_size, elem_type_id;
2708+ u32 i, elem_size = 0, elem_type_id;
2709+ u16 encoding = 0;
21162710
21172711 elem_type_id = array->type;
2118− elem_type = btf_type_id_size(btf, &elem_type_id, &elem_size);
2712+ elem_type = btf_type_skip_modifiers(btf, elem_type_id, NULL);
2713+ if (elem_type && btf_type_has_size(elem_type))
2714+ elem_size = elem_type->size;
2715+
2716+ if (elem_type && btf_type_is_int(elem_type)) {
2717+ u32 int_type = btf_type_int(elem_type);
2718+
2719+ encoding = BTF_INT_ENCODING(int_type);
2720+
2721+ /*
2722+ * BTF_INT_CHAR encoding never seems to be set for
2723+ * char arrays, so if size is 1 and element is
2724+ * printable as a char, we'll do that.
2725+ */
2726+ if (elem_size == 1)
2727+ encoding = BTF_INT_CHAR;
2728+ }
2729+
2730+ if (!btf_show_start_array_type(show, t, type_id, encoding, data))
2731+ return;
2732+
2733+ if (!elem_type)
2734+ goto out;
21192735 elem_ops = btf_type_ops(elem_type);
2120− seq_puts(m, "[");
2736+
21212737 for (i = 0; i < array->nelems; i++) {
2122− if (i)
2123− seq_puts(m, ",");
21242738
2125− elem_ops->seq_show(btf, elem_type, elem_type_id, data,
2126− bits_offset, m);
2739+ btf_show_start_array_member(show);
2740+
2741+ elem_ops->show(btf, elem_type, elem_type_id, data,
2742+ bits_offset, show);
21272743 data += elem_size;
2744+
2745+ btf_show_end_array_member(show);
2746+
2747+ if (show->state.array_terminated)
2748+ break;
2749+ }
2750+out:
2751+ btf_show_end_array_type(show);
2752+}
2753+
2754+static void btf_array_show(const struct btf *btf, const struct btf_type *t,
2755+ u32 type_id, void *data, u8 bits_offset,
2756+ struct btf_show *show)
2757+{
2758+ const struct btf_member *m = show->state.member;
2759+
2760+ /*
2761+ * First check if any members would be shown (are non-zero).
2762+ * See comments above "struct btf_show" definition for more
2763+ * details on how this works at a high-level.
2764+ */
2765+ if (show->state.depth > 0 && !(show->flags & BTF_SHOW_ZERO)) {
2766+ if (!show->state.depth_check) {
2767+ show->state.depth_check = show->state.depth + 1;
2768+ show->state.depth_to_show = 0;
2769+ }
2770+ __btf_array_show(btf, t, type_id, data, bits_offset, show);
2771+ show->state.member = m;
2772+
2773+ if (show->state.depth_check != show->state.depth + 1)
2774+ return;
2775+ show->state.depth_check = 0;
2776+
2777+ if (show->state.depth_to_show <= show->state.depth)
2778+ return;
2779+ /*
2780+ * Reaching here indicates we have recursed and found
2781+ * non-zero array member(s).
2782+ */
21282783 }
2129− seq_puts(m, "]");
2784+ __btf_array_show(btf, t, type_id, data, bits_offset, show);
21302785 }
21312786
21322787 static struct btf_kind_operations array_ops = {
@@ -2135,7 +2790,7 @@ static struct btf_kind_operations array_ops = { · 2 unmodified lines
21352790 .check_member = btf_array_check_member,
21362791 .check_kflag_member = btf_generic_check_kflag_member,
21372792 .log_details = btf_array_log,
2138− .seq_show = btf_array_seq_show,
2793+ .show = btf_array_show,
21392794 };
21402795
21412796 static int btf_struct_check_member(struct btf_verifier_env *env,
@@ -2358,15 +3013,18 @@ int btf_find_spin_lock(const struct btf *btf, const struct btf_type *t) · 216 unmodified lines
23583013 return off;
23593014 }
23603015
2361−static void btf_struct_seq_show(const struct btf *btf, const struct btf_type *t,
2362− u32 type_id, void *data, u8 bits_offset,
2363− struct seq_file *m)
3016+static void __btf_struct_show(const struct btf *btf, const struct btf_type *t,
3017+ u32 type_id, void *data, u8 bits_offset,
3018+ struct btf_show *show)
23643019 {
2365− const char *seq = BTF_INFO_KIND(t->info) == BTF_KIND_UNION ? "|" : ",";
23663020 const struct btf_member *member;
3021+ void *safe_data;
23673022 u32 i;
23683023
2369− seq_puts(m, "{");
3024+ safe_data = btf_show_start_struct_type(show, t, type_id, data);
3025+ if (!safe_data)
3026+ return;
3027+
23703028 for_each_member(i, t, member) {
23713029 const struct btf_type *member_type = btf_type_by_id(btf,
23723030 member->type);
@@ -2375,23 +3033,65 @@ static void btf_struct_seq_show(const struct btf *btf, const struct btf_type *t, · 2 unmodified lines
23753033 u32 bytes_offset;
23763034 u8 bits8_offset;
23773035
2378− if (i)
2379− seq_puts(m, seq);
3036+ btf_show_start_member(show, member);
23803037
23813038 member_offset = btf_member_bit_offset(t, member);
23823039 bitfield_size = btf_member_bitfield_size(t, member);
23833040 bytes_offset = BITS_ROUNDDOWN_BYTES(member_offset);
23843041 bits8_offset = BITS_PER_BYTE_MASKED(member_offset);
23853042 if (bitfield_size) {
2386− btf_bitfield_seq_show(data + bytes_offset, bits8_offset,
2387− bitfield_size, m);
3043+ safe_data = btf_show_start_type(show, member_type,
3044+ member->type,
3045+ data + bytes_offset);
3046+ if (safe_data)
3047+ btf_bitfield_show(safe_data,
3048+ bits8_offset,
3049+ bitfield_size, show);
3050+ btf_show_end_type(show);
23883051 } else {
23893052 ops = btf_type_ops(member_type);
2390− ops->seq_show(btf, member_type, member->type,
2391− data + bytes_offset, bits8_offset, m);
3053+ ops->show(btf, member_type, member->type,
3054+ data + bytes_offset, bits8_offset, show);
23923055 }
3056+
3057+ btf_show_end_member(show);
23933058 }
2394− seq_puts(m, "}");
3059+
3060+ btf_show_end_struct_type(show);
3061+}
3062+
3063+static void btf_struct_show(const struct btf *btf, const struct btf_type *t,
3064+ u32 type_id, void *data, u8 bits_offset,
3065+ struct btf_show *show)
3066+{
3067+ const struct btf_member *m = show->state.member;
3068+
3069+ /*
3070+ * First check if any members would be shown (are non-zero).
3071+ * See comments above "struct btf_show" definition for more
3072+ * details on how this works at a high-level.
3073+ */
3074+ if (show->state.depth > 0 && !(show->flags & BTF_SHOW_ZERO)) {
3075+ if (!show->state.depth_check) {
3076+ show->state.depth_check = show->state.depth + 1;
3077+ show->state.depth_to_show = 0;
3078+ }
3079+ __btf_struct_show(btf, t, type_id, data, bits_offset, show);
3080+ /* Restore saved member data here */
3081+ show->state.member = m;
3082+ if (show->state.depth_check != show->state.depth + 1)
3083+ return;
3084+ show->state.depth_check = 0;
3085+
3086+ if (show->state.depth_to_show <= show->state.depth)
3087+ return;
3088+ /*
3089+ * Reaching here indicates we have recursed and found
3090+ * non-zero child values.
3091+ */
3092+ }
3093+
3094+ __btf_struct_show(btf, t, type_id, data, bits_offset, show);
23953095 }
23963096
23973097 static struct btf_kind_operations struct_ops = {
@@ -2400,7 +3100,7 @@ static struct btf_kind_operations struct_ops = { · 2 unmodified lines
24003100 .check_member = btf_struct_check_member,
24013101 .check_kflag_member = btf_generic_check_kflag_member,
24023102 .log_details = btf_struct_log,
2403− .seq_show = btf_struct_seq_show,
3103+ .show = btf_struct_show,
24043104 };
24053105
24063106 static int btf_enum_check_member(struct btf_verifier_env *env,
@@ -2531,24 +3231,35 @@ static void btf_enum_log(struct btf_verifier_env *env, · 124 unmodified lines
25313231 btf_verifier_log(env, "size=%u vlen=%u", t->size, btf_type_vlen(t));
25323232 }
25333233
2534−static void btf_enum_seq_show(const struct btf *btf, const struct btf_type *t,
2535− u32 type_id, void *data, u8 bits_offset,
2536− struct seq_file *m)
3234+static void btf_enum_show(const struct btf *btf, const struct btf_type *t,
3235+ u32 type_id, void *data, u8 bits_offset,
3236+ struct btf_show *show)
25373237 {
25383238 const struct btf_enum *enums = btf_type_enum(t);
25393239 u32 i, nr_enums = btf_type_vlen(t);
2540− int v = *(int *)data;
3240+ void *safe_data;
3241+ int v;
3242+
3243+ safe_data = btf_show_start_type(show, t, type_id, data);
3244+ if (!safe_data)
3245+ return;
3246+
3247+ v = *(int *)safe_data;
25413248
25423249 for (i = 0; i < nr_enums; i++) {
2543− if (v == enums[i].val) {
2544− seq_printf(m, "%s",
2545− __btf_name_by_offset(btf,
2546− enums[i].name_off));
2547− return;
2548− }
3250+ if (v != enums[i].val)
3251+ continue;
3252+
3253+ btf_show_type_value(show, "%s",
3254+ __btf_name_by_offset(btf,
3255+ enums[i].name_off));
3256+
3257+ btf_show_end_type(show);
3258+ return;
25493259 }
25503260
2551− seq_printf(m, "%d", v);
3261+ btf_show_type_value(show, "%d", v);
3262+ btf_show_end_type(show);
25523263 }
25533264
25543265 static struct btf_kind_operations enum_ops = {
@@ -2557,7 +3268,7 @@ static struct btf_kind_operations enum_ops = { · 2 unmodified lines
25573268 .check_member = btf_enum_check_member,
25583269 .check_kflag_member = btf_enum_check_kflag_member,
25593270 .log_details = btf_enum_log,
2560− .seq_show = btf_enum_seq_show,
3271+ .show = btf_enum_show,
25613272 };
25623273
25633274 static s32 btf_func_proto_check_meta(struct btf_verifier_env *env,
@@ -2644,7 +3355,7 @@ static struct btf_kind_operations func_proto_ops = { · 80 unmodified lines
26443355 .check_member = btf_df_check_member,
26453356 .check_kflag_member = btf_df_check_kflag_member,
26463357 .log_details = btf_func_proto_log,
2647− .seq_show = btf_df_seq_show,
3358+ .show = btf_df_show,
26483359 };
26493360
26503361 static s32 btf_func_check_meta(struct btf_verifier_env *env,
@@ -2678,7 +3389,7 @@ static struct btf_kind_operations func_ops = { · 27 unmodified lines
26783389 .check_member = btf_df_check_member,
26793390 .check_kflag_member = btf_df_check_kflag_member,
26803391 .log_details = btf_ref_type_log,
2681− .seq_show = btf_df_seq_show,
3392+ .show = btf_df_show,
26823393 };
26833394
26843395 static s32 btf_var_check_meta(struct btf_verifier_env *env,
@@ -2742,7 +3453,7 @@ static const struct btf_kind_operations var_ops = { · 57 unmodified lines
27423453 .check_member = btf_df_check_member,
27433454 .check_kflag_member = btf_df_check_kflag_member,
27443455 .log_details = btf_var_log,
2745− .seq_show = btf_var_seq_show,
3456+ .show = btf_var_show,
27463457 };
27473458
27483459 static s32 btf_datasec_check_meta(struct btf_verifier_env *env,
@@ -2868,24 +3579,28 @@ static void btf_datasec_log(struct btf_verifier_env *env, · 119 unmodified lines
28683579 btf_verifier_log(env, "size=%u vlen=%u", t->size, btf_type_vlen(t));
28693580 }
28703581
2871−static void btf_datasec_seq_show(const struct btf *btf,
2872− const struct btf_type *t, u32 type_id,
2873− void *data, u8 bits_offset,
2874− struct seq_file *m)
3582+static void btf_datasec_show(const struct btf *btf,
3583+ const struct btf_type *t, u32 type_id,
3584+ void *data, u8 bits_offset,
3585+ struct btf_show *show)
28753586 {
28763587 const struct btf_var_secinfo *vsi;
28773588 const struct btf_type *var;
28783589 u32 i;
28793590
2880− seq_printf(m, "section (\"%s\") = {", __btf_name_by_offset(btf, t->name_off));
3591+ if (!btf_show_start_type(show, t, type_id, data))
3592+ return;
3593+
3594+ btf_show_type_value(show, "section (\"%s\") = {",
3595+ __btf_name_by_offset(btf, t->name_off));
28813596 for_each_vsi(i, t, vsi) {
28823597 var = btf_type_by_id(btf, vsi->type);
28833598 if (i)
2884− seq_puts(m, ",");
2885− btf_type_ops(var)->seq_show(btf, var, vsi->type,
2886− data + vsi->offset, bits_offset, m);
3599+ btf_show(show, ",");
3600+ btf_type_ops(var)->show(btf, var, vsi->type,
3601+ data + vsi->offset, bits_offset, show);
28873602 }
2888− seq_puts(m, "}");
3603+ btf_show_end_type(show);
28893604 }
28903605
28913606 static const struct btf_kind_operations datasec_ops = {
@@ -2894,7 +3609,7 @@ static const struct btf_kind_operations datasec_ops = { · 2 unmodified lines
28943609 .check_member = btf_df_check_member,
28953610 .check_kflag_member = btf_df_check_kflag_member,
28963611 .log_details = btf_datasec_log,
2897− .seq_show = btf_datasec_seq_show,
3612+ .show = btf_datasec_show,
28983613 };
28993614
29003615 static int btf_func_proto_check(struct btf_verifier_env *env,
@@ -3688,7 +4403,7 @@ errout: · 787 unmodified lines
36884403
36894404 struct btf *bpf_prog_get_target_btf(const struct bpf_prog *prog)
36904405 {
3691− struct bpf_prog *tgt_prog = prog->aux->linked_prog;
4406+ struct bpf_prog *tgt_prog = prog->aux->dst_prog;
36924407
36934408 if (tgt_prog) {
36944409 return tgt_prog->aux->btf;
@@ -3715,7 +4430,7 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type, · 20 unmodified lines
37154430 struct bpf_insn_access_aux *info)
37164431 {
37174432 const struct btf_type *t = prog->aux->attach_func_proto;
3718− struct bpf_prog *tgt_prog = prog->aux->linked_prog;
4433+ struct bpf_prog *tgt_prog = prog->aux->dst_prog;
37194434 struct btf *btf = bpf_prog_get_target_btf(prog);
37204435 const char *tname = prog->aux->attach_func_name;
37214436 struct bpf_verifier_log *log = info->log;
@@ -3842,7 +4557,14 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type, · 120 unmodified lines
38424557
38434558 info->reg_type = PTR_TO_BTF_ID;
38444559 if (tgt_prog) {
3845− ret = btf_translate_to_vmlinux(log, btf, t, tgt_prog->type, arg);
4560+ enum bpf_prog_type tgt_type;
4561+
4562+ if (tgt_prog->type == BPF_PROG_TYPE_EXT)
4563+ tgt_type = tgt_prog->aux->saved_dst_prog_type;
4564+ else
4565+ tgt_type = tgt_prog->type;
4566+
4567+ ret = btf_translate_to_vmlinux(log, btf, t, tgt_type, arg);
38464568 if (ret > 0) {
38474569 info->btf_id = ret;
38484570 return true;
@@ -3870,16 +4592,22 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type, · 21 unmodified lines
38704592 return true;
38714593 }
38724594
3873−int btf_struct_access(struct bpf_verifier_log *log,
3874− const struct btf_type *t, int off, int size,
3875− enum bpf_access_type atype,
3876− u32 *next_btf_id)
4595+enum bpf_struct_walk_result {
4596+ /* < 0 error */
4597+ WALK_SCALAR = 0,
4598+ WALK_PTR,
4599+ WALK_STRUCT,
4600+};
4601+
4602+static int btf_struct_walk(struct bpf_verifier_log *log,
4603+ const struct btf_type *t, int off, int size,
4604+ u32 *next_btf_id)
38774605 {
38784606 u32 i, moff, mtrue_end, msize = 0, total_nelems = 0;
38794607 const struct btf_type *mtype, *elem_type = NULL;
38804608 const struct btf_member *member;
38814609 const char *tname, *mname;
3882− u32 vlen;
4610+ u32 vlen, elem_id, mid;
38834611
38844612 again:
38854613 tname = __btf_name_by_offset(btf_vmlinux, t->name_off);
@@ -3915,14 +4643,13 @@ again: · 29 unmodified lines
39154643 /* Only allow structure for now, can be relaxed for
39164644 * other types later.
39174645 */
3918− elem_type = btf_type_skip_modifiers(btf_vmlinux,
3919− array_elem->type, NULL);
3920− if (!btf_type_is_struct(elem_type))
4646+ t = btf_type_skip_modifiers(btf_vmlinux, array_elem->type,
4647+ NULL);
4648+ if (!btf_type_is_struct(t))
39214649 goto error;
39224650
3923− off = (off - moff) % elem_type->size;
3924− return btf_struct_access(log, elem_type, off, size, atype,
3925− next_btf_id);
4651+ off = (off - moff) % t->size;
4652+ goto again;
39264653
39274654 error:
39284655 bpf_log(log, "access beyond struct %s at off %u size %u\n",
@@ -3951,7 +4678,7 @@ error: · 22 unmodified lines
39514678 */
39524679 if (off <= moff &&
39534680 BITS_ROUNDUP_BYTES(end_bit) <= off + size)
3954− return SCALAR_VALUE;
4681+ return WALK_SCALAR;
39554682
39564683 /* off may be accessing a following member
39574684 *
@@ -3973,11 +4700,13 @@ error: · 15 unmodified lines
39734700 break;
39744701
39754702 /* type of the field */
4703+ mid = member->type;
39764704 mtype = btf_type_by_id(btf_vmlinux, member->type);
39774705 mname = __btf_name_by_offset(btf_vmlinux, member->name_off);
39784706
3979− mtype = btf_resolve_size(btf_vmlinux, mtype, &msize,
3980− &elem_type, &total_nelems);
4707+ mtype = __btf_resolve_size(btf_vmlinux, mtype, &msize,
4708+ &elem_type, &elem_id, &total_nelems,
4709+ &mid);
39814710 if (IS_ERR(mtype)) {
39824711 bpf_log(log, "field %s doesn't have size\n", mname);
39834712 return -EFAULT;
@@ -3991,7 +4720,7 @@ error: · 7 unmodified lines
39914720 if (btf_type_is_array(mtype)) {
39924721 u32 elem_idx;
39934722
3994− /* btf_resolve_size() above helps to
4723+ /* __btf_resolve_size() above helps to
39954724 * linearize a multi-dimensional array.
39964725 *
39974726 * The logic here is treating an array
@@ -4039,6 +4768,7 @@ error: · 41 unmodified lines
40394768 elem_idx = (off - moff) / msize;
40404769 moff += elem_idx * msize;
40414770 mtype = elem_type;
4771+ mid = elem_id;
40424772 }
40434773
40444774 /* the 'off' we're looking for is either equal to start
@@ -4048,6 +4778,12 @@ error: · 3 unmodified lines
40484778 /* our field must be inside that union or struct */
40494779 t = mtype;
40504780
4781+ /* return if the offset matches the member offset */
4782+ if (off == moff) {
4783+ *next_btf_id = mid;
4784+ return WALK_STRUCT;
4785+ }
4786+
40514787 /* adjust offset we're looking for */
40524788 off -= moff;
40534789 goto again;
@@ -4063,11 +4799,10 @@ error: · 9 unmodified lines
40634799 mname, moff, tname, off, size);
40644800 return -EACCES;
40654801 }
4066−
40674802 stype = btf_type_skip_modifiers(btf_vmlinux, mtype->type, &id);
40684803 if (btf_type_is_struct(stype)) {
40694804 *next_btf_id = id;
4070− return PTR_TO_BTF_ID;
4805+ return WALK_PTR;
40714806 }
40724807 }
40734808
@@ -4084,23 +4819,82 @@ error: · 10 unmodified lines
40844819 return -EACCES;
40854820 }
40864821
4087− return SCALAR_VALUE;
4822+ return WALK_SCALAR;
40884823 }
40894824 bpf_log(log, "struct %s doesn't have field at offset %d\n", tname, off);
40904825 return -EINVAL;
40914826 }
40924827
4093−int btf_resolve_helper_id(struct bpf_verifier_log *log,
4094− const struct bpf_func_proto *fn, int arg)
4828+int btf_struct_access(struct bpf_verifier_log *log,
4829+ const struct btf_type *t, int off, int size,
4830+ enum bpf_access_type atype __maybe_unused,
4831+ u32 *next_btf_id)
4832+{
4833+ int err;
4834+ u32 id;
4835+
4836+ do {
4837+ err = btf_struct_walk(log, t, off, size, &id);
4838+
4839+ switch (err) {
4840+ case WALK_PTR:
4841+ /* If we found the pointer or scalar on t+off,
4842+ * we're done.
4843+ */
4844+ *next_btf_id = id;
4845+ return PTR_TO_BTF_ID;
4846+ case WALK_SCALAR:
4847+ return SCALAR_VALUE;
4848+ case WALK_STRUCT:
4849+ /* We found nested struct, so continue the search
4850+ * by diving in it. At this point the offset is
4851+ * aligned with the new type, so set it to 0.
4852+ */
4853+ t = btf_type_by_id(btf_vmlinux, id);
4854+ off = 0;
4855+ break;
4856+ default:
4857+ /* It's either error or unknown return value..
4858+ * scream and leave.
4859+ */
4860+ if (WARN_ONCE(err > 0, "unknown btf_struct_walk return value"))
4861+ return -EINVAL;
4862+ return err;
4863+ }
4864+ } while (t);
4865+
4866+ return -EINVAL;
4867+}
4868+
4869+bool btf_struct_ids_match(struct bpf_verifier_log *log,
4870+ int off, u32 id, u32 need_type_id)
40954871 {
4096− int id;
4872+ const struct btf_type *type;
4873+ int err;
40974874
4098− if (fn->arg_type[arg] != ARG_PTR_TO_BTF_ID || !btf_vmlinux)
4099− return -EINVAL;
4100− id = fn->btf_id[arg];
4101− if (!id || id > btf_vmlinux->nr_types)
4102− return -EINVAL;
4103− return id;
4875+ /* Are we already done? */
4876+ if (need_type_id == id && off == 0)
4877+ return true;
4878+
4879+again:
4880+ type = btf_type_by_id(btf_vmlinux, id);
4881+ if (!type)
4882+ return false;
4883+ err = btf_struct_walk(log, type, off, 1, &id);
4884+ if (err != WALK_STRUCT)
4885+ return false;
4886+
4887+ /* We found nested struct object. If it matches
4888+ * the requested ID, we're done. Otherwise let's
4889+ * continue the search with offset 0 in the new
4890+ * type.
4891+ */
4892+ if (need_type_id != id) {
4893+ off = 0;
4894+ goto again;
4895+ }
4896+
4897+ return true;
41044898 }
41054899
41064900 static int __get_type_size(struct btf *btf, u32 btf_id,
@@ -4298,7 +5092,7 @@ static int btf_check_func_type_match(struct bpf_verifier_log *log, · 191 unmodified lines
42985092 }
42995093
43005094 /* Compare BTFs of given program with BTF of target program */
4301−int btf_check_type_match(struct bpf_verifier_env *env, struct bpf_prog *prog,
5095+int btf_check_type_match(struct bpf_verifier_log *log, const struct bpf_prog *prog,
43025096 struct btf *btf2, const struct btf_type *t2)
43035097 {
43045098 struct btf *btf1 = prog->aux->btf;
@@ -4306,7 +5100,7 @@ int btf_check_type_match(struct bpf_verifier_env *env, struct bpf_prog *prog, · 1 unmodified line
43065100 u32 btf_id = 0;
43075101
43085102 if (!prog->aux->func_info) {
4309− bpf_log(&env->log, "Program extension requires BTF\n");
5103+ bpf_log(log, "Program extension requires BTF\n");
43105104 return -EINVAL;
43115105 }
43125106
@@ -4318,7 +5112,7 @@ int btf_check_type_match(struct bpf_verifier_env *env, struct bpf_prog *prog, · 5 unmodified lines
43185112 if (!t1 || !btf_type_is_func(t1))
43195113 return -EFAULT;
43205114
4321− return btf_check_func_type_match(&env->log, btf1, t1, btf2, t2);
5115+ return btf_check_func_type_match(log, btf1, t1, btf2, t2);
43225116 }
43235117
43245118 /* Compare BTF of a function with given bpf_reg_state.
@@ -4469,7 +5263,7 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog, · 144 unmodified lines
44695263 return -EFAULT;
44705264 }
44715265 if (prog_type == BPF_PROG_TYPE_EXT)
4472− prog_type = prog->aux->linked_prog->type;
5266+ prog_type = prog->aux->dst_prog->type;
44735267
44745268 t = btf_type_by_id(btf, t->type);
44755269 if (!t || !btf_type_is_func_proto(t)) {
@@ -4516,12 +5310,93 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog, · 40 unmodified lines
45165310 return 0;
45175311 }
45185312
5313+static void btf_type_show(const struct btf *btf, u32 type_id, void *obj,
5314+ struct btf_show *show)
5315+{
5316+ const struct btf_type *t = btf_type_by_id(btf, type_id);
5317+
5318+ show->btf = btf;
5319+ memset(&show->state, 0, sizeof(show->state));
5320+ memset(&show->obj, 0, sizeof(show->obj));
5321+
5322+ btf_type_ops(t)->show(btf, t, type_id, obj, 0, show);
5323+}
5324+
5325+static void btf_seq_show(struct btf_show *show, const char *fmt,
5326+ va_list args)
5327+{
5328+ seq_vprintf((struct seq_file *)show->target, fmt, args);
5329+}
5330+
5331+int btf_type_seq_show_flags(const struct btf *btf, u32 type_id,
5332+ void *obj, struct seq_file *m, u64 flags)
5333+{
5334+ struct btf_show sseq;
5335+
5336+ sseq.target = m;
5337+ sseq.showfn = btf_seq_show;
5338+ sseq.flags = flags;
5339+
5340+ btf_type_show(btf, type_id, obj, &sseq);
5341+
5342+ return sseq.state.status;
5343+}
5344+
45195345 void btf_type_seq_show(const struct btf *btf, u32 type_id, void *obj,
45205346 struct seq_file *m)
45215347 {
4522− const struct btf_type *t = btf_type_by_id(btf, type_id);
5348+ (void) btf_type_seq_show_flags(btf, type_id, obj, m,
5349+ BTF_SHOW_NONAME | BTF_SHOW_COMPACT |
5350+ BTF_SHOW_ZERO | BTF_SHOW_UNSAFE);
5351+}
5352+
5353+struct btf_show_snprintf {
5354+ struct btf_show show;
5355+ int len_left; /* space left in string */
5356+ int len; /* length we would have written */
5357+};
5358+
5359+static void btf_snprintf_show(struct btf_show *show, const char *fmt,
5360+ va_list args)
5361+{
5362+ struct btf_show_snprintf *ssnprintf = (struct btf_show_snprintf *)show;
5363+ int len;
45235364
4524− btf_type_ops(t)->seq_show(btf, t, type_id, obj, 0, m);
5365+ len = vsnprintf(show->target, ssnprintf->len_left, fmt, args);
5366+
5367+ if (len < 0) {
5368+ ssnprintf->len_left = 0;
5369+ ssnprintf->len = len;
5370+ } else if (len > ssnprintf->len_left) {
5371+ /* no space, drive on to get length we would have written */
5372+ ssnprintf->len_left = 0;
5373+ ssnprintf->len += len;
5374+ } else {
5375+ ssnprintf->len_left -= len;
5376+ ssnprintf->len += len;
5377+ show->target += len;
5378+ }
5379+}
5380+
5381+int btf_type_snprintf_show(const struct btf *btf, u32 type_id, void *obj,
5382+ char *buf, int len, u64 flags)
5383+{
5384+ struct btf_show_snprintf ssnprintf;
5385+
5386+ ssnprintf.show.target = buf;
5387+ ssnprintf.show.flags = flags;
5388+ ssnprintf.show.showfn = btf_snprintf_show;
5389+ ssnprintf.len_left = len;
5390+ ssnprintf.len = 0;
5391+
5392+ btf_type_show(btf, type_id, obj, (struct btf_show *)&ssnprintf);
5393+
5394+ /* If we encontered an error, return it. */
5395+ if (ssnprintf.show.state.status)
5396+ return ssnprintf.show.state.status;
5397+
5398+ /* Otherwise return length we would have written */
5399+ return ssnprintf.len;
45255400 }
45265401
45275402 #ifdef CONFIG_PROC_FS
@@ -4661,3 +5536,15 @@ u32 btf_id(const struct btf *btf) · 133 unmodified lines
46615536 {
46625537 return btf->id;
46635538 }
5539+
5540+static int btf_id_cmp_func(const void *a, const void *b)
5541+{
5542+ const int *pa = a, *pb = b;
5543+
5544+ return *pa - *pb;
5545+}
5546+
5547+bool btf_id_set_contains(const struct btf_id_set *set, u32 id)
5548+{
5549+ return bsearch(&id, set->ids, set->cnt, sizeof(u32), btf_id_cmp_func) != NULL;
5550+}
kernel/bpf/core.c+21−8
@@ -98,6 +98,8 @@ struct bpf_prog *bpf_prog_alloc_no_stats(unsigned int size, gfp_t gfp_extra_flag · 97 unmodified lines
9898 fp->jit_requested = ebpf_jit_enabled();
9999
100100 INIT_LIST_HEAD_RCU(&fp->aux->ksym.lnode);
101+ mutex_init(&fp->aux->used_maps_mutex);
102+ mutex_init(&fp->aux->dst_mutex);
101103
102104 return fp;
103105 }
@@ -253,6 +255,8 @@ struct bpf_prog *bpf_prog_realloc(struct bpf_prog *fp_old, unsigned int size, · 149 unmodified lines
253255 void __bpf_prog_free(struct bpf_prog *fp)
254256 {
255257 if (fp->aux) {
258+ mutex_destroy(&fp->aux->used_maps_mutex);
259+ mutex_destroy(&fp->aux->dst_mutex);
256260 free_percpu(fp->aux->stats);
257261 kfree(fp->aux->poke_tab);
258262 kfree(fp->aux);
@@ -773,7 +777,8 @@ int bpf_jit_add_poke_descriptor(struct bpf_prog *prog, · 514 unmodified lines
773777
774778 if (size > poke_tab_max)
775779 return -ENOSPC;
776− if (poke->ip || poke->ip_stable || poke->adj_off)
780+ if (poke->tailcall_target || poke->tailcall_target_stable ||
781+ poke->tailcall_bypass || poke->adj_off || poke->bypass_addr)
777782 return -EINVAL;
778783
779784 switch (poke->reason) {
@@ -1747,8 +1752,9 @@ bool bpf_prog_array_compatible(struct bpf_array *array, · 967 unmodified lines
17471752 static int bpf_check_tail_call(const struct bpf_prog *fp)
17481753 {
17491754 struct bpf_prog_aux *aux = fp->aux;
1750− int i;
1755+ int i, ret = 0;
17511756
1757+ mutex_lock(&aux->used_maps_mutex);
17521758 for (i = 0; i < aux->used_map_cnt; i++) {
17531759 struct bpf_map *map = aux->used_maps[i];
17541760 struct bpf_array *array;
@@ -1757,11 +1763,15 @@ static int bpf_check_tail_call(const struct bpf_prog *fp) · 2 unmodified lines
17571763 continue;
17581764
17591765 array = container_of(map, struct bpf_array, map);
1760− if (!bpf_prog_array_compatible(array, fp))
1761− return -EINVAL;
1766+ if (!bpf_prog_array_compatible(array, fp)) {
1767+ ret = -EINVAL;
1768+ goto out;
1769+ }
17621770 }
17631771
1764− return 0;
1772+out:
1773+ mutex_unlock(&aux->used_maps_mutex);
1774+ return ret;
17651775 }
17661776
17671777 static void bpf_prog_select_func(struct bpf_prog *fp)
@@ -2130,7 +2140,8 @@ static void bpf_prog_free_deferred(struct work_struct *work) · 362 unmodified lines
21302140 if (aux->prog->has_callchain_buf)
21312141 put_callchain_buffers();
21322142 #endif
2133− bpf_trampoline_put(aux->trampoline);
2143+ if (aux->dst_trampoline)
2144+ bpf_trampoline_put(aux->dst_trampoline);
21342145 for (i = 0; i < aux->func_cnt; i++)
21352146 bpf_jit_free(aux->func[i]);
21362147 if (aux->func_cnt) {
@@ -2146,8 +2157,8 @@ void bpf_prog_free(struct bpf_prog *fp) · 9 unmodified lines
21462157 {
21472158 struct bpf_prog_aux *aux = fp->aux;
21482159
2149− if (aux->linked_prog)
2150− bpf_prog_put(aux->linked_prog);
2160+ if (aux->dst_prog)
2161+ bpf_prog_put(aux->dst_prog);
21512162 INIT_WORK(&aux->work, bpf_prog_free_deferred);
21522163 schedule_work(&aux->work);
21532164 }
@@ -2208,6 +2219,8 @@ const struct bpf_func_proto bpf_get_current_cgroup_id_proto __weak; · 54 unmodified lines
22082219 const struct bpf_func_proto bpf_get_current_ancestor_cgroup_id_proto __weak;
22092220 const struct bpf_func_proto bpf_get_local_storage_proto __weak;
22102221 const struct bpf_func_proto bpf_get_ns_current_pid_tgid_proto __weak;
2222+const struct bpf_func_proto bpf_snprintf_btf_proto __weak;
2223+const struct bpf_func_proto bpf_seq_printf_btf_proto __weak;
22112224
22122225 const struct bpf_func_proto * __weak bpf_get_trace_printk_proto(void)
22132226 {
kernel/bpf/cpumap.c+6−11
@@ -79,8 +79,6 @@ struct bpf_cpu_map { · 78 unmodified lines
7979
8080 static DEFINE_PER_CPU(struct list_head, cpu_map_flush_list);
8181
82−static int bq_flush_to_queue(struct xdp_bulk_queue *bq);
83−
8482 static struct bpf_map *cpu_map_alloc(union bpf_attr *attr)
8583 {
8684 u32 value_size = attr->value_size;
@@ -157,8 +155,7 @@ static void cpu_map_kthread_stop(struct work_struct *work) · 70 unmodified lines
157155 kthread_stop(rcpu->kthread);
158156 }
159157
160−static struct sk_buff *cpu_map_build_skb(struct bpf_cpu_map_entry *rcpu,
161− struct xdp_frame *xdpf,
158+static struct sk_buff *cpu_map_build_skb(struct xdp_frame *xdpf,
162159 struct sk_buff *skb)
163160 {
164161 unsigned int hard_start_headroom;
@@ -367,7 +364,7 @@ static int cpu_map_kthread_run(void *data) · 202 unmodified lines
367364 struct sk_buff *skb = skbs[i];
368365 int ret;
369366
370− skb = cpu_map_build_skb(rcpu, xdpf, skb);
367+ skb = cpu_map_build_skb(xdpf, skb);
371368 if (!skb) {
372369 xdp_return_frame(xdpf);
373370 continue;
@@ -658,6 +655,7 @@ static int cpu_map_get_next_key(struct bpf_map *map, void *key, void *next_key) · 284 unmodified lines
658655
659656 static int cpu_map_btf_id;
660657 const struct bpf_map_ops cpu_map_ops = {
658+ .map_meta_equal = bpf_map_meta_equal,
661659 .map_alloc = cpu_map_alloc,
662660 .map_free = cpu_map_free,
663661 .map_delete_elem = cpu_map_delete_elem,
@@ -669,7 +667,7 @@ const struct bpf_map_ops cpu_map_ops = { · 5 unmodified lines
669667 .map_btf_id = &cpu_map_btf_id,
670668 };
671669
672−static int bq_flush_to_queue(struct xdp_bulk_queue *bq)
670+static void bq_flush_to_queue(struct xdp_bulk_queue *bq)
673671 {
674672 struct bpf_cpu_map_entry *rcpu = bq->obj;
675673 unsigned int processed = 0, drops = 0;
@@ -678,7 +676,7 @@ static int bq_flush_to_queue(struct xdp_bulk_queue *bq) · 2 unmodified lines
678676 int i;
679677
680678 if (unlikely(!bq->count))
681− return 0;
679+ return;
682680
683681 q = rcpu->queue;
684682 spin_lock(&q->producer_lock);
@@ -701,13 +699,12 @@ static int bq_flush_to_queue(struct xdp_bulk_queue *bq) · 16 unmodified lines
701699
702700 /* Feedback loop via tracepoints */
703701 trace_xdp_cpumap_enqueue(rcpu->map_id, processed, drops, to_cpu);
704− return 0;
705702 }
706703
707704 /* Runs under RCU-read-side, plus in softirq under NAPI protection.
708705 * Thus, safe percpu variable access.
709706 */
710−static int bq_enqueue(struct bpf_cpu_map_entry *rcpu, struct xdp_frame *xdpf)
707+static void bq_enqueue(struct bpf_cpu_map_entry *rcpu, struct xdp_frame *xdpf)
711708 {
712709 struct list_head *flush_list = this_cpu_ptr(&cpu_map_flush_list);
713710 struct xdp_bulk_queue *bq = this_cpu_ptr(rcpu->bulkq);
@@ -728,8 +725,6 @@ static int bq_enqueue(struct bpf_cpu_map_entry *rcpu, struct xdp_frame *xdpf) · 14 unmodified lines
728725
729726 if (!bq->flush_node.prev)
730727 list_add(&bq->flush_node, flush_list);
731−
732− return 0;
733728 }
734729
735730 int cpu_map_enqueue(struct bpf_cpu_map_entry *rcpu, struct xdp_buff *xdp,
kernel/bpf/devmap.c+9−8
@@ -341,14 +341,14 @@ bool dev_map_can_have_prog(struct bpf_map *map) · 340 unmodified lines
341341 return false;
342342 }
343343
344−static int bq_xmit_all(struct xdp_dev_bulk_queue *bq, u32 flags)
344+static void bq_xmit_all(struct xdp_dev_bulk_queue *bq, u32 flags)
345345 {
346346 struct net_device *dev = bq->dev;
347347 int sent = 0, drops = 0, err = 0;
348348 int i;
349349
350350 if (unlikely(!bq->count))
351− return 0;
351+ return;
352352
353353 for (i = 0; i < bq->count; i++) {
354354 struct xdp_frame *xdpf = bq->q[i];
@@ -369,7 +369,7 @@ out: · 14 unmodified lines
369369 trace_xdp_devmap_xmit(bq->dev_rx, dev, sent, drops, err);
370370 bq->dev_rx = NULL;
371371 __list_del_clearprev(&bq->flush_node);
372− return 0;
372+ return;
373373 error:
374374 /* If ndo_xdp_xmit fails with an errno, no frames have been
375375 * xmit'ed and it's our responsibility to them free all.
@@ -421,8 +421,8 @@ struct bpf_dtab_netdev *__dev_map_lookup_elem(struct bpf_map *map, u32 key) · 45 unmodified lines
421421 /* Runs under RCU-read-side, plus in softirq under NAPI protection.
422422 * Thus, safe percpu variable access.
423423 */
424−static int bq_enqueue(struct net_device *dev, struct xdp_frame *xdpf,
425− struct net_device *dev_rx)
424+static void bq_enqueue(struct net_device *dev, struct xdp_frame *xdpf,
425+ struct net_device *dev_rx)
426426 {
427427 struct list_head *flush_list = this_cpu_ptr(&dev_flush_list);
428428 struct xdp_dev_bulk_queue *bq = this_cpu_ptr(dev->xdp_bulkq);
@@ -441,8 +441,6 @@ static int bq_enqueue(struct net_device *dev, struct xdp_frame *xdpf, · 12 unmodified lines
441441
442442 if (!bq->flush_node.prev)
443443 list_add(&bq->flush_node, flush_list);
444−
445− return 0;
446444 }
447445
448446 static inline int __xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp,
@@ -462,7 +460,8 @@ static inline int __xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, · 13 unmodified lines
462460 if (unlikely(!xdpf))
463461 return -EOVERFLOW;
464462
465− return bq_enqueue(dev, xdpf, dev_rx);
463+ bq_enqueue(dev, xdpf, dev_rx);
464+ return 0;
466465 }
467466
468467 static struct xdp_buff *dev_map_run_prog(struct net_device *dev,
@@ -751,6 +750,7 @@ static int dev_map_hash_update_elem(struct bpf_map *map, void *key, void *value, · 282 unmodified lines
751750
752751 static int dev_map_btf_id;
753752 const struct bpf_map_ops dev_map_ops = {
753+ .map_meta_equal = bpf_map_meta_equal,
754754 .map_alloc = dev_map_alloc,
755755 .map_free = dev_map_free,
756756 .map_get_next_key = dev_map_get_next_key,
@@ -764,6 +764,7 @@ const struct bpf_map_ops dev_map_ops = { · 7 unmodified lines
764764
765765 static int dev_map_hash_map_btf_id;
766766 const struct bpf_map_ops dev_map_hash_ops = {
767+ .map_meta_equal = bpf_map_meta_equal,
767768 .map_alloc = dev_map_alloc,
768769 .map_free = dev_map_free,
769770 .map_get_next_key = dev_map_hash_get_next_key,
kernel/bpf/hashtab.c+13−9
@@ -9,6 +9,7 @@ · 8 unmodified lines
99 #include <linux/rculist_nulls.h>
1010 #include <linux/random.h>
1111 #include <uapi/linux/btf.h>
12+#include <linux/rcupdate_trace.h>
1213 #include "percpu_freelist.h"
1314 #include "bpf_lru_list.h"
1415 #include "map_in_map.h"
@@ -577,8 +578,7 @@ static void *__htab_map_lookup_elem(struct bpf_map *map, void *key) · 562 unmodified lines
577578 struct htab_elem *l;
578579 u32 hash, key_size;
579580
580− /* Must be called with rcu_read_lock. */
581− WARN_ON_ONCE(!rcu_read_lock_held());
581+ WARN_ON_ONCE(!rcu_read_lock_held() && !rcu_read_lock_trace_held());
582582
583583 key_size = map->key_size;
584584
@@ -612,7 +612,7 @@ static void *htab_map_lookup_elem(struct bpf_map *map, void *key) · 27 unmodified lines
612612 * bpf_prog
613613 * __htab_map_lookup_elem
614614 */
615−static u32 htab_map_gen_lookup(struct bpf_map *map, struct bpf_insn *insn_buf)
615+static int htab_map_gen_lookup(struct bpf_map *map, struct bpf_insn *insn_buf)
616616 {
617617 struct bpf_insn *insn = insn_buf;
618618 const int ret = BPF_REG_0;
@@ -651,7 +651,7 @@ static void *htab_lru_map_lookup_elem_sys(struct bpf_map *map, void *key) · 32 unmodified lines
651651 return __htab_lru_map_lookup_elem(map, key, false);
652652 }
653653
654−static u32 htab_lru_map_gen_lookup(struct bpf_map *map,
654+static int htab_lru_map_gen_lookup(struct bpf_map *map,
655655 struct bpf_insn *insn_buf)
656656 {
657657 struct bpf_insn *insn = insn_buf;
@@ -941,7 +941,7 @@ static int htab_map_update_elem(struct bpf_map *map, void *key, void *value, · 283 unmodified lines
941941 /* unknown flags */
942942 return -EINVAL;
943943
944− WARN_ON_ONCE(!rcu_read_lock_held());
944+ WARN_ON_ONCE(!rcu_read_lock_held() && !rcu_read_lock_trace_held());
945945
946946 key_size = map->key_size;
947947
@@ -1032,7 +1032,7 @@ static int htab_lru_map_update_elem(struct bpf_map *map, void *key, void *value, · 84 unmodified lines
10321032 /* unknown flags */
10331033 return -EINVAL;
10341034
1035− WARN_ON_ONCE(!rcu_read_lock_held());
1035+ WARN_ON_ONCE(!rcu_read_lock_held() && !rcu_read_lock_trace_held());
10361036
10371037 key_size = map->key_size;
10381038
@@ -1220,7 +1220,7 @@ static int htab_map_delete_elem(struct bpf_map *map, void *key) · 181 unmodified lines
12201220 u32 hash, key_size;
12211221 int ret = -ENOENT;
12221222
1223− WARN_ON_ONCE(!rcu_read_lock_held());
1223+ WARN_ON_ONCE(!rcu_read_lock_held() && !rcu_read_lock_trace_held());
12241224
12251225 key_size = map->key_size;
12261226
@@ -1252,7 +1252,7 @@ static int htab_lru_map_delete_elem(struct bpf_map *map, void *key) · 25 unmodified lines
12521252 u32 hash, key_size;
12531253 int ret = -ENOENT;
12541254
1255− WARN_ON_ONCE(!rcu_read_lock_held());
1255+ WARN_ON_ONCE(!rcu_read_lock_held() && !rcu_read_lock_trace_held());
12561256
12571257 key_size = map->key_size;
12581258
@@ -1803,6 +1803,7 @@ static const struct bpf_iter_seq_info iter_seq_info = { · 544 unmodified lines
18031803
18041804 static int htab_map_btf_id;
18051805 const struct bpf_map_ops htab_map_ops = {
1806+ .map_meta_equal = bpf_map_meta_equal,
18061807 .map_alloc_check = htab_map_alloc_check,
18071808 .map_alloc = htab_map_alloc,
18081809 .map_free = htab_map_free,
@@ -1820,6 +1821,7 @@ const struct bpf_map_ops htab_map_ops = { · 11 unmodified lines
18201821
18211822 static int htab_lru_map_btf_id;
18221823 const struct bpf_map_ops htab_lru_map_ops = {
1824+ .map_meta_equal = bpf_map_meta_equal,
18231825 .map_alloc_check = htab_map_alloc_check,
18241826 .map_alloc = htab_map_alloc,
18251827 .map_free = htab_map_free,
@@ -1940,6 +1942,7 @@ static void htab_percpu_map_seq_show_elem(struct bpf_map *map, void *key, · 114 unmodified lines
19401942
19411943 static int htab_percpu_map_btf_id;
19421944 const struct bpf_map_ops htab_percpu_map_ops = {
1945+ .map_meta_equal = bpf_map_meta_equal,
19431946 .map_alloc_check = htab_map_alloc_check,
19441947 .map_alloc = htab_map_alloc,
19451948 .map_free = htab_map_free,
@@ -1956,6 +1959,7 @@ const struct bpf_map_ops htab_percpu_map_ops = { · 10 unmodified lines
19561959
19571960 static int htab_lru_percpu_map_btf_id;
19581961 const struct bpf_map_ops htab_lru_percpu_map_ops = {
1962+ .map_meta_equal = bpf_map_meta_equal,
19591963 .map_alloc_check = htab_map_alloc_check,
19601964 .map_alloc = htab_map_alloc,
19611965 .map_free = htab_map_free,
@@ -2066,7 +2070,7 @@ static void *htab_of_map_lookup_elem(struct bpf_map *map, void *key) · 104 unmodified lines
20662070 return READ_ONCE(*inner_map);
20672071 }
20682072
2069−static u32 htab_of_map_gen_lookup(struct bpf_map *map,
2073+static int htab_of_map_gen_lookup(struct bpf_map *map,
20702074 struct bpf_insn *insn_buf)
20712075 {
20722076 struct bpf_insn *insn = insn_buf;
kernel/bpf/helpers.c+58−0
@@ -601,6 +601,56 @@ const struct bpf_func_proto bpf_event_output_data_proto = { · 600 unmodified lines
601601 .arg5_type = ARG_CONST_SIZE_OR_ZERO,
602602 };
603603
604+BPF_CALL_3(bpf_copy_from_user, void *, dst, u32, size,
605+ const void __user *, user_ptr)
606+{
607+ int ret = copy_from_user(dst, user_ptr, size);
608+
609+ if (unlikely(ret)) {
610+ memset(dst, 0, size);
611+ ret = -EFAULT;
612+ }
613+
614+ return ret;
615+}
616+
617+const struct bpf_func_proto bpf_copy_from_user_proto = {
618+ .func = bpf_copy_from_user,
619+ .gpl_only = false,
620+ .ret_type = RET_INTEGER,
621+ .arg1_type = ARG_PTR_TO_UNINIT_MEM,
622+ .arg2_type = ARG_CONST_SIZE_OR_ZERO,
623+ .arg3_type = ARG_ANYTHING,
624+};
625+
626+BPF_CALL_2(bpf_per_cpu_ptr, const void *, ptr, u32, cpu)
627+{
628+ if (cpu >= nr_cpu_ids)
629+ return (unsigned long)NULL;
630+
631+ return (unsigned long)per_cpu_ptr((const void __percpu *)ptr, cpu);
632+}
633+
634+const struct bpf_func_proto bpf_per_cpu_ptr_proto = {
635+ .func = bpf_per_cpu_ptr,
636+ .gpl_only = false,
637+ .ret_type = RET_PTR_TO_MEM_OR_BTF_ID_OR_NULL,
638+ .arg1_type = ARG_PTR_TO_PERCPU_BTF_ID,
639+ .arg2_type = ARG_ANYTHING,
640+};
641+
642+BPF_CALL_1(bpf_this_cpu_ptr, const void *, percpu_ptr)
643+{
644+ return (unsigned long)this_cpu_ptr((const void __percpu *)percpu_ptr);
645+}
646+
647+const struct bpf_func_proto bpf_this_cpu_ptr_proto = {
648+ .func = bpf_this_cpu_ptr,
649+ .gpl_only = false,
650+ .ret_type = RET_PTR_TO_MEM_OR_BTF_ID,
651+ .arg1_type = ARG_PTR_TO_PERCPU_BTF_ID,
652+};
653+
604654 const struct bpf_func_proto bpf_get_current_task_proto __weak;
605655 const struct bpf_func_proto bpf_probe_read_user_proto __weak;
606656 const struct bpf_func_proto bpf_probe_read_user_str_proto __weak;
@@ -661,8 +711,16 @@ bpf_base_func_proto(enum bpf_func_id func_id) · 54 unmodified lines
661711 if (!perfmon_capable())
662712 return NULL;
663713 return bpf_get_trace_printk_proto();
714+ case BPF_FUNC_snprintf_btf:
715+ if (!perfmon_capable())
716+ return NULL;
717+ return &bpf_snprintf_btf_proto;
664718 case BPF_FUNC_jiffies64:
665719 return &bpf_jiffies64_proto;
720+ case BPF_FUNC_bpf_per_cpu_ptr:
721+ return &bpf_per_cpu_ptr_proto;
722+ case BPF_FUNC_bpf_this_cpu_ptr:
723+ return &bpf_this_cpu_ptr_proto;
666724 default:
667725 break;
668726 }
kernel/bpf/inode.c+113−3
@@ -20,6 +20,7 @@ · 19 unmodified lines
2020 #include <linux/filter.h>
2121 #include <linux/bpf.h>
2222 #include <linux/bpf_trace.h>
23+#include "preload/bpf_preload.h"
2324
2425 enum bpf_type {
2526 BPF_TYPE_UNSPEC = 0,
@@ -371,9 +372,10 @@ static struct dentry * · 345 unmodified lines
371372 bpf_lookup(struct inode *dir, struct dentry *dentry, unsigned flags)
372373 {
373374 /* Dots in names (e.g. "/sys/fs/bpf/foo.bar") are reserved for future
374− * extensions.
375+ * extensions. That allows popoulate_bpffs() create special files.
375376 */
376− if (strchr(dentry->d_name.name, '.'))
377+ if ((dir->i_mode & S_IALLUGO) &&
378+ strchr(dentry->d_name.name, '.'))
377379 return ERR_PTR(-EPERM);
378380
379381 return simple_lookup(dir, dentry, flags);
@@ -411,6 +413,27 @@ static const struct inode_operations bpf_dir_iops = { · 31 unmodified lines
411413 .unlink = simple_unlink,
412414 };
413415
416+/* pin iterator link into bpffs */
417+static int bpf_iter_link_pin_kernel(struct dentry *parent,
418+ const char *name, struct bpf_link *link)
419+{
420+ umode_t mode = S_IFREG | S_IRUSR;
421+ struct dentry *dentry;
422+ int ret;
423+
424+ inode_lock(parent->d_inode);
425+ dentry = lookup_one_len(name, parent, strlen(name));
426+ if (IS_ERR(dentry)) {
427+ inode_unlock(parent->d_inode);
428+ return PTR_ERR(dentry);
429+ }
430+ ret = bpf_mkobj_ops(dentry, mode, link, &bpf_link_iops,
431+ &bpf_iter_fops);
432+ dput(dentry);
433+ inode_unlock(parent->d_inode);
434+ return ret;
435+}
436+
414437 static int bpf_obj_do_pin(const char __user *pathname, void *raw,
415438 enum bpf_type type)
416439 {
@@ -640,6 +663,91 @@ static int bpf_parse_param(struct fs_context *fc, struct fs_parameter *param) · 223 unmodified lines
640663 return 0;
641664 }
642665
666+struct bpf_preload_ops *bpf_preload_ops;
667+EXPORT_SYMBOL_GPL(bpf_preload_ops);
668+
669+static bool bpf_preload_mod_get(void)
670+{
671+ /* If bpf_preload.ko wasn't loaded earlier then load it now.
672+ * When bpf_preload is built into vmlinux the module's __init
673+ * function will populate it.
674+ */
675+ if (!bpf_preload_ops) {
676+ request_module("bpf_preload");
677+ if (!bpf_preload_ops)
678+ return false;
679+ }
680+ /* And grab the reference, so the module doesn't disappear while the
681+ * kernel is interacting with the kernel module and its UMD.
682+ */
683+ if (!try_module_get(bpf_preload_ops->owner)) {
684+ pr_err("bpf_preload module get failed.\n");
685+ return false;
686+ }
687+ return true;
688+}
689+
690+static void bpf_preload_mod_put(void)
691+{
692+ if (bpf_preload_ops)
693+ /* now user can "rmmod bpf_preload" if necessary */
694+ module_put(bpf_preload_ops->owner);
695+}
696+
697+static DEFINE_MUTEX(bpf_preload_lock);
698+
699+static int populate_bpffs(struct dentry *parent)
700+{
701+ struct bpf_preload_info objs[BPF_PRELOAD_LINKS] = {};
702+ struct bpf_link *links[BPF_PRELOAD_LINKS] = {};
703+ int err = 0, i;
704+
705+ /* grab the mutex to make sure the kernel interactions with bpf_preload
706+ * UMD are serialized
707+ */
708+ mutex_lock(&bpf_preload_lock);
709+
710+ /* if bpf_preload.ko wasn't built into vmlinux then load it */
711+ if (!bpf_preload_mod_get())
712+ goto out;
713+
714+ if (!bpf_preload_ops->info.tgid) {
715+ /* preload() will start UMD that will load BPF iterator programs */
716+ err = bpf_preload_ops->preload(objs);
717+ if (err)
718+ goto out_put;
719+ for (i = 0; i < BPF_PRELOAD_LINKS; i++) {
720+ links[i] = bpf_link_by_id(objs[i].link_id);
721+ if (IS_ERR(links[i])) {
722+ err = PTR_ERR(links[i]);
723+ goto out_put;
724+ }
725+ }
726+ for (i = 0; i < BPF_PRELOAD_LINKS; i++) {
727+ err = bpf_iter_link_pin_kernel(parent,
728+ objs[i].link_name, links[i]);
729+ if (err)
730+ goto out_put;
731+ /* do not unlink successfully pinned links even
732+ * if later link fails to pin
733+ */
734+ links[i] = NULL;
735+ }
736+ /* finish() will tell UMD process to exit */
737+ err = bpf_preload_ops->finish();
738+ if (err)
739+ goto out_put;
740+ }
741+out_put:
742+ bpf_preload_mod_put();
743+out:
744+ mutex_unlock(&bpf_preload_lock);
745+ for (i = 0; i < BPF_PRELOAD_LINKS && err; i++)
746+ if (!IS_ERR_OR_NULL(links[i]))
747+ bpf_link_put(links[i]);
748+ return err;
749+}
750+
643751 static int bpf_fill_super(struct super_block *sb, struct fs_context *fc)
644752 {
645753 static const struct tree_descr bpf_rfiles[] = { { "" } };
@@ -656,8 +764,8 @@ static int bpf_fill_super(struct super_block *sb, struct fs_context *fc) · 10 unmodified lines
656764 inode = sb->s_root->d_inode;
657765 inode->i_op = &bpf_dir_iops;
658766 inode->i_mode &= ~S_IALLUGO;
767+ populate_bpffs(sb->s_root);
659768 inode->i_mode |= S_ISVTX | opts->mode;
660−
661769 return 0;
662770 }
663771
@@ -707,6 +815,8 @@ static int __init bpf_init(void) · 43 unmodified lines
707815 {
708816 int ret;
709817
818+ mutex_init(&bpf_preload_lock);
819+
710820 ret = sysfs_create_mount_point(fs_kobj, "bpf");
711821 if (ret)
712822 return ret;
kernel/bpf/lpm_trie.c+1−0
@@ -732,6 +732,7 @@ static int trie_check_btf(const struct bpf_map *map, · 731 unmodified lines
732732
733733 static int trie_map_btf_id;
734734 const struct bpf_map_ops trie_map_ops = {
735+ .map_meta_equal = bpf_map_meta_equal,
735736 .map_alloc = trie_alloc,
736737 .map_free = trie_free,
737738 .map_get_next_key = trie_get_next_key,
kernel/bpf/map_in_map.c+9−15
@@ -17,23 +17,17 @@ struct bpf_map *bpf_map_meta_alloc(int inner_map_ufd) · 16 unmodified lines
1717 if (IS_ERR(inner_map))
1818 return inner_map;
1919
20− /* prog_array->aux->{type,jited} is a runtime binding.
21− * Doing static check alone in the verifier is not enough.
22− */
23− if (inner_map->map_type == BPF_MAP_TYPE_PROG_ARRAY ||
24− inner_map->map_type == BPF_MAP_TYPE_CGROUP_STORAGE ||
25− inner_map->map_type == BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE ||
26− inner_map->map_type == BPF_MAP_TYPE_STRUCT_OPS) {
27− fdput(f);
28− return ERR_PTR(-ENOTSUPP);
29− }
30−
3120 /* Does not support >1 level map-in-map */
3221 if (inner_map->inner_map_meta) {
3322 fdput(f);
3423 return ERR_PTR(-EINVAL);
3524 }
3625
26+ if (!inner_map->ops->map_meta_equal) {
27+ fdput(f);
28+ return ERR_PTR(-ENOTSUPP);
29+ }
30+
3731 if (map_value_has_spin_lock(inner_map)) {
3832 fdput(f);
3933 return ERR_PTR(-ENOTSUPP);
@@ -81,15 +75,14 @@ bool bpf_map_meta_equal(const struct bpf_map *meta0, · 41 unmodified lines
8175 return meta0->map_type == meta1->map_type &&
8276 meta0->key_size == meta1->key_size &&
8377 meta0->value_size == meta1->value_size &&
84− meta0->map_flags == meta1->map_flags &&
85− meta0->max_entries == meta1->max_entries;
78+ meta0->map_flags == meta1->map_flags;
8679 }
8780
8881 void *bpf_map_fd_get_ptr(struct bpf_map *map,
8982 struct file *map_file /* not used */,
9083 int ufd)
9184 {
92− struct bpf_map *inner_map;
85+ struct bpf_map *inner_map, *inner_map_meta;
9386 struct fd f;
9487
9588 f = fdget(ufd);
@@ -97,7 +90,8 @@ void *bpf_map_fd_get_ptr(struct bpf_map *map, · 1 unmodified line
9790 if (IS_ERR(inner_map))
9891 return inner_map;
9992
100− if (bpf_map_meta_equal(map->inner_map_meta, inner_map))
93+ inner_map_meta = map->inner_map_meta;
94+ if (inner_map_meta->ops->map_meta_equal(inner_map_meta, inner_map))
10195 bpf_map_inc(inner_map);
10296 else
10397 inner_map = ERR_PTR(-EINVAL);
kernel/bpf/map_in_map.h+0−2
@@ -11,8 +11,6 @@ struct bpf_map; · 10 unmodified lines
1111
1212 struct bpf_map *bpf_map_meta_alloc(int inner_map_ufd);
1313 void bpf_map_meta_free(struct bpf_map *map_meta);
14−bool bpf_map_meta_equal(const struct bpf_map *meta0,
15− const struct bpf_map *meta1);
1614 void *bpf_map_fd_get_ptr(struct bpf_map *map, struct file *map_file,
1715 int ufd);
1816 void bpf_map_fd_put_ptr(void *ptr);
kernel/bpf/map_iter.c+15−0
@@ -149,6 +149,19 @@ static void bpf_iter_detach_map(struct bpf_iter_aux_info *aux) · 148 unmodified lines
149149 bpf_map_put_with_uref(aux->map);
150150 }
151151
152+void bpf_iter_map_show_fdinfo(const struct bpf_iter_aux_info *aux,
153+ struct seq_file *seq)
154+{
155+ seq_printf(seq, "map_id:\t%u\n", aux->map->id);
156+}
157+
158+int bpf_iter_map_fill_link_info(const struct bpf_iter_aux_info *aux,
159+ struct bpf_link_info *info)
160+{
161+ info->iter.map.map_id = aux->map->id;
162+ return 0;
163+}
164+
152165 DEFINE_BPF_ITER_FUNC(bpf_map_elem, struct bpf_iter_meta *meta,
153166 struct bpf_map *map, void *key, void *value)
154167
@@ -156,6 +169,8 @@ static const struct bpf_iter_reg bpf_map_elem_reg_info = { · 1 unmodified line
156169 .target = "bpf_map_elem",
157170 .attach_target = bpf_iter_attach_map,
158171 .detach_target = bpf_iter_detach_map,
172+ .show_fdinfo = bpf_iter_map_show_fdinfo,
173+ .fill_link_info = bpf_iter_map_fill_link_info,
159174 .ctx_arg_info_size = 2,
160175 .ctx_arg_info = {
161176 { offsetof(struct bpf_iter__bpf_map_elem, key),
kernel/bpf/percpu_freelist.c+96−5
@@ -17,6 +17,8 @@ int pcpu_freelist_init(struct pcpu_freelist *s) · 16 unmodified lines
1717 raw_spin_lock_init(&head->lock);
1818 head->first = NULL;
1919 }
20+ raw_spin_lock_init(&s->extralist.lock);
21+ s->extralist.first = NULL;
2022 return 0;
2123 }
2224
@@ -40,12 +42,50 @@ static inline void ___pcpu_freelist_push(struct pcpu_freelist_head *head, · 17 unmodified lines
4042 raw_spin_unlock(&head->lock);
4143 }
4244
45+static inline bool pcpu_freelist_try_push_extra(struct pcpu_freelist *s,
46+ struct pcpu_freelist_node *node)
47+{
48+ if (!raw_spin_trylock(&s->extralist.lock))
49+ return false;
50+
51+ pcpu_freelist_push_node(&s->extralist, node);
52+ raw_spin_unlock(&s->extralist.lock);
53+ return true;
54+}
55+
56+static inline void ___pcpu_freelist_push_nmi(struct pcpu_freelist *s,
57+ struct pcpu_freelist_node *node)
58+{
59+ int cpu, orig_cpu;
60+
61+ orig_cpu = cpu = raw_smp_processor_id();
62+ while (1) {
63+ struct pcpu_freelist_head *head;
64+
65+ head = per_cpu_ptr(s->freelist, cpu);
66+ if (raw_spin_trylock(&head->lock)) {
67+ pcpu_freelist_push_node(head, node);
68+ raw_spin_unlock(&head->lock);
69+ return;
70+ }
71+ cpu = cpumask_next(cpu, cpu_possible_mask);
72+ if (cpu >= nr_cpu_ids)
73+ cpu = 0;
74+
75+ /* cannot lock any per cpu lock, try extralist */
76+ if (cpu == orig_cpu &&
77+ pcpu_freelist_try_push_extra(s, node))
78+ return;
79+ }
80+}
81+
4382 void __pcpu_freelist_push(struct pcpu_freelist *s,
4483 struct pcpu_freelist_node *node)
4584 {
46− struct pcpu_freelist_head *head = this_cpu_ptr(s->freelist);
47−
48− ___pcpu_freelist_push(head, node);
85+ if (in_nmi())
86+ ___pcpu_freelist_push_nmi(s, node);
87+ else
88+ ___pcpu_freelist_push(this_cpu_ptr(s->freelist), node);
4989 }
5090
5191 void pcpu_freelist_push(struct pcpu_freelist *s,
@@ -81,7 +121,7 @@ again: · 29 unmodified lines
81121 }
82122 }
83123
84−struct pcpu_freelist_node *__pcpu_freelist_pop(struct pcpu_freelist *s)
124+static struct pcpu_freelist_node *___pcpu_freelist_pop(struct pcpu_freelist *s)
85125 {
86126 struct pcpu_freelist_head *head;
87127 struct pcpu_freelist_node *node;
@@ -102,8 +142,59 @@ struct pcpu_freelist_node *__pcpu_freelist_pop(struct pcpu_freelist *s) · 14 unmodified lines
102142 if (cpu >= nr_cpu_ids)
103143 cpu = 0;
104144 if (cpu == orig_cpu)
105− return NULL;
145+ break;
146+ }
147+
148+ /* per cpu lists are all empty, try extralist */
149+ raw_spin_lock(&s->extralist.lock);
150+ node = s->extralist.first;
151+ if (node)
152+ s->extralist.first = node->next;
153+ raw_spin_unlock(&s->extralist.lock);
154+ return node;
155+}
156+
157+static struct pcpu_freelist_node *
158+___pcpu_freelist_pop_nmi(struct pcpu_freelist *s)
159+{
160+ struct pcpu_freelist_head *head;
161+ struct pcpu_freelist_node *node;
162+ int orig_cpu, cpu;
163+
164+ orig_cpu = cpu = raw_smp_processor_id();
165+ while (1) {
166+ head = per_cpu_ptr(s->freelist, cpu);
167+ if (raw_spin_trylock(&head->lock)) {
168+ node = head->first;
169+ if (node) {
170+ head->first = node->next;
171+ raw_spin_unlock(&head->lock);
172+ return node;
173+ }
174+ raw_spin_unlock(&head->lock);
175+ }
176+ cpu = cpumask_next(cpu, cpu_possible_mask);
177+ if (cpu >= nr_cpu_ids)
178+ cpu = 0;
179+ if (cpu == orig_cpu)
180+ break;
106181 }
182+
183+ /* cannot pop from per cpu lists, try extralist */
184+ if (!raw_spin_trylock(&s->extralist.lock))
185+ return NULL;
186+ node = s->extralist.first;
187+ if (node)
188+ s->extralist.first = node->next;
189+ raw_spin_unlock(&s->extralist.lock);
190+ return node;
191+}
192+
193+struct pcpu_freelist_node *__pcpu_freelist_pop(struct pcpu_freelist *s)
194+{
195+ if (in_nmi())
196+ return ___pcpu_freelist_pop_nmi(s);
197+ return ___pcpu_freelist_pop(s);
107198 }
108199
109200 struct pcpu_freelist_node *pcpu_freelist_pop(struct pcpu_freelist *s)
kernel/bpf/percpu_freelist.h+1−0
@@ -13,6 +13,7 @@ struct pcpu_freelist_head { · 12 unmodified lines
1313
1414 struct pcpu_freelist {
1515 struct pcpu_freelist_head __percpu *freelist;
16+ struct pcpu_freelist_head extralist;
1617 };
1718
1819 struct pcpu_freelist_node {
kernel/bpf/preload/.gitignore+4−0
@@ -0,0 +1,4 @@
1+/FEATURE-DUMP.libbpf
2+/bpf_helper_defs.h
3+/feature
4+/bpf_preload_umd
kernel/bpf/preload/Kconfig+26−0
@@ -0,0 +1,26 @@
1+# SPDX-License-Identifier: GPL-2.0-only
2+config USERMODE_DRIVER
3+ bool
4+ default n
5+
6+menuconfig BPF_PRELOAD
7+ bool "Preload BPF file system with kernel specific program and map iterators"
8+ depends on BPF
9+ # The dependency on !COMPILE_TEST prevents it from being enabled
10+ # in allmodconfig or allyesconfig configurations
11+ depends on !COMPILE_TEST
12+ select USERMODE_DRIVER
13+ help
14+ This builds kernel module with several embedded BPF programs that are
15+ pinned into BPF FS mount point as human readable files that are
16+ useful in debugging and introspection of BPF programs and maps.
17+
18+if BPF_PRELOAD
19+config BPF_PRELOAD_UMD
20+ tristate "bpf_preload kernel module with user mode driver"
21+ depends on CC_CAN_LINK
22+ depends on m || CC_CAN_LINK_STATIC
23+ default m
24+ help
25+ This builds bpf_preload kernel module with embedded user mode driver.
26+endif
kernel/bpf/preload/Makefile+25−0
@@ -0,0 +1,25 @@
1+# SPDX-License-Identifier: GPL-2.0
2+
3+LIBBPF_SRCS = $(srctree)/tools/lib/bpf/
4+LIBBPF_A = $(obj)/libbpf.a
5+LIBBPF_OUT = $(abspath $(obj))
6+
7+$(LIBBPF_A):
8+ $(Q)$(MAKE) -C $(LIBBPF_SRCS) OUTPUT=$(LIBBPF_OUT)/ $(LIBBPF_OUT)/libbpf.a
9+
10+userccflags += -I $(srctree)/tools/include/ -I $(srctree)/tools/include/uapi \
11+ -I $(srctree)/tools/lib/ -Wno-unused-result
12+
13+userprogs := bpf_preload_umd
14+
15+clean-files := $(userprogs) bpf_helper_defs.h FEATURE-DUMP.libbpf staticobjs/ feature/
16+
17+bpf_preload_umd-objs := iterators/iterators.o
18+bpf_preload_umd-userldlibs := $(LIBBPF_A) -lelf -lz
19+
20+$(obj)/bpf_preload_umd: $(LIBBPF_A)
21+
22+$(obj)/bpf_preload_umd_blob.o: $(obj)/bpf_preload_umd
23+
24+obj-$(CONFIG_BPF_PRELOAD_UMD) += bpf_preload.o
25+bpf_preload-objs += bpf_preload_kern.o bpf_preload_umd_blob.o
kernel/bpf/preload/bpf_preload.h+16−0
@@ -0,0 +1,16 @@
1+/* SPDX-License-Identifier: GPL-2.0 */
2+#ifndef _BPF_PRELOAD_H
3+#define _BPF_PRELOAD_H
4+
5+#include <linux/usermode_driver.h>
6+#include "iterators/bpf_preload_common.h"
7+
8+struct bpf_preload_ops {
9+ struct umd_info info;
10+ int (*preload)(struct bpf_preload_info *);
11+ int (*finish)(void);
12+ struct module *owner;
13+};
14+extern struct bpf_preload_ops *bpf_preload_ops;
15+#define BPF_PRELOAD_LINKS 2
16+#endif
kernel/bpf/preload/bpf_preload_kern.c+91−0
@@ -0,0 +1,91 @@
1+// SPDX-License-Identifier: GPL-2.0
2+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
3+#include <linux/init.h>
4+#include <linux/module.h>
5+#include <linux/pid.h>
6+#include <linux/fs.h>
7+#include <linux/sched/signal.h>
8+#include "bpf_preload.h"
9+
10+extern char bpf_preload_umd_start;
11+extern char bpf_preload_umd_end;
12+
13+static int preload(struct bpf_preload_info *obj);
14+static int finish(void);
15+
16+static struct bpf_preload_ops umd_ops = {
17+ .info.driver_name = "bpf_preload",
18+ .preload = preload,
19+ .finish = finish,
20+ .owner = THIS_MODULE,
21+};
22+
23+static int preload(struct bpf_preload_info *obj)
24+{
25+ int magic = BPF_PRELOAD_START;
26+ loff_t pos = 0;
27+ int i, err;
28+ ssize_t n;
29+
30+ err = fork_usermode_driver(&umd_ops.info);
31+ if (err)
32+ return err;
33+
34+ /* send the start magic to let UMD proceed with loading BPF progs */
35+ n = kernel_write(umd_ops.info.pipe_to_umh,
36+ &magic, sizeof(magic), &pos);
37+ if (n != sizeof(magic))
38+ return -EPIPE;
39+
40+ /* receive bpf_link IDs and names from UMD */
41+ pos = 0;
42+ for (i = 0; i < BPF_PRELOAD_LINKS; i++) {
43+ n = kernel_read(umd_ops.info.pipe_from_umh,
44+ &obj[i], sizeof(*obj), &pos);
45+ if (n != sizeof(*obj))
46+ return -EPIPE;
47+ }
48+ return 0;
49+}
50+
51+static int finish(void)
52+{
53+ int magic = BPF_PRELOAD_END;
54+ struct pid *tgid;
55+ loff_t pos = 0;
56+ ssize_t n;
57+
58+ /* send the last magic to UMD. It will do a normal exit. */
59+ n = kernel_write(umd_ops.info.pipe_to_umh,
60+ &magic, sizeof(magic), &pos);
61+ if (n != sizeof(magic))
62+ return -EPIPE;
63+ tgid = umd_ops.info.tgid;
64+ wait_event(tgid->wait_pidfd, thread_group_exited(tgid));
65+ umd_ops.info.tgid = NULL;
66+ return 0;
67+}
68+
69+static int __init load_umd(void)
70+{
71+ int err;
72+
73+ err = umd_load_blob(&umd_ops.info, &bpf_preload_umd_start,
74+ &bpf_preload_umd_end - &bpf_preload_umd_start);
75+ if (err)
76+ return err;
77+ bpf_preload_ops = &umd_ops;
78+ return err;
79+}
80+
81+static void __exit fini_umd(void)
82+{
83+ bpf_preload_ops = NULL;
84+ /* kill UMD in case it's still there due to earlier error */
85+ kill_pid(umd_ops.info.tgid, SIGKILL, 1);
86+ umd_ops.info.tgid = NULL;
87+ umd_unload_blob(&umd_ops.info);
88+}
89+late_initcall(load_umd);
90+module_exit(fini_umd);
91+MODULE_LICENSE("GPL");
kernel/bpf/preload/bpf_preload_umd_blob.S+7−0
@@ -0,0 +1,7 @@
1+/* SPDX-License-Identifier: GPL-2.0 */
2+ .section .init.rodata, "a"
3+ .global bpf_preload_umd_start
4+bpf_preload_umd_start:
5+ .incbin "kernel/bpf/preload/bpf_preload_umd"
6+ .global bpf_preload_umd_end
7+bpf_preload_umd_end:
kernel/bpf/preload/iterators/.gitignore+2−0
@@ -0,0 +1,2 @@
1+# SPDX-License-Identifier: GPL-2.0-only
2+/.output
kernel/bpf/preload/iterators/Makefile+57−0
@@ -0,0 +1,57 @@
1+# SPDX-License-Identifier: GPL-2.0
2+OUTPUT := .output
3+CLANG ?= clang
4+LLC ?= llc
5+LLVM_STRIP ?= llvm-strip
6+DEFAULT_BPFTOOL := $(OUTPUT)/sbin/bpftool
7+BPFTOOL ?= $(DEFAULT_BPFTOOL)
8+LIBBPF_SRC := $(abspath ../../../../tools/lib/bpf)
9+BPFOBJ := $(OUTPUT)/libbpf.a
10+BPF_INCLUDE := $(OUTPUT)
11+INCLUDES := -I$(OUTPUT) -I$(BPF_INCLUDE) -I$(abspath ../../../../tools/lib) \
12+ -I$(abspath ../../../../tools/include/uapi)
13+CFLAGS := -g -Wall
14+
15+abs_out := $(abspath $(OUTPUT))
16+ifeq ($(V),1)
17+Q =
18+msg =
19+else
20+Q = @
21+msg = @printf ' %-8s %s%s\n' "$(1)" "$(notdir $(2))" "$(if $(3), $(3))";
22+MAKEFLAGS += --no-print-directory
23+submake_extras := feature_display=0
24+endif
25+
26+.DELETE_ON_ERROR:
27+
28+.PHONY: all clean
29+
30+all: iterators.skel.h
31+
32+clean:
33+ $(call msg,CLEAN)
34+ $(Q)rm -rf $(OUTPUT) iterators
35+
36+iterators.skel.h: $(OUTPUT)/iterators.bpf.o | $(BPFTOOL)
37+ $(call msg,GEN-SKEL,$@)
38+ $(Q)$(BPFTOOL) gen skeleton $< > $@
39+
40+
41+$(OUTPUT)/iterators.bpf.o: iterators.bpf.c $(BPFOBJ) | $(OUTPUT)
42+ $(call msg,BPF,$@)
43+ $(Q)$(CLANG) -g -O2 -target bpf $(INCLUDES) \
44+ -c $(filter %.c,$^) -o $@ && \
45+ $(LLVM_STRIP) -g $@
46+
47+$(OUTPUT):
48+ $(call msg,MKDIR,$@)
49+ $(Q)mkdir -p $(OUTPUT)
50+
51+$(BPFOBJ): $(wildcard $(LIBBPF_SRC)/*.[ch] $(LIBBPF_SRC)/Makefile) | $(OUTPUT)
52+ $(Q)$(MAKE) $(submake_extras) -C $(LIBBPF_SRC) \
53+ OUTPUT=$(abspath $(dir $@))/ $(abspath $@)
54+
55+$(DEFAULT_BPFTOOL):
56+ $(Q)$(MAKE) $(submake_extras) -C ../../../../tools/bpf/bpftool \
57+ prefix= OUTPUT=$(abs_out)/ DESTDIR=$(abs_out) install
kernel/bpf/preload/iterators/README+4−0
@@ -0,0 +1,4 @@
1+WARNING:
2+If you change "iterators.bpf.c" do "make -j" in this directory to rebuild "iterators.skel.h".
3+Make sure to have clang 10 installed.
4+See Documentation/bpf/bpf_devel_QA.rst
kernel/bpf/preload/iterators/bpf_preload_common.h+13−0
@@ -0,0 +1,13 @@
1+/* SPDX-License-Identifier: GPL-2.0 */
2+#ifndef _BPF_PRELOAD_COMMON_H
3+#define _BPF_PRELOAD_COMMON_H
4+
5+#define BPF_PRELOAD_START 0x5555
6+#define BPF_PRELOAD_END 0xAAAA
7+
8+struct bpf_preload_info {
9+ char link_name[16];
10+ int link_id;
11+};
12+
13+#endif
kernel/bpf/preload/iterators/iterators.bpf.c+114−0
@@ -0,0 +1,114 @@
1+// SPDX-License-Identifier: GPL-2.0
2+/* Copyright (c) 2020 Facebook */
3+#include <linux/bpf.h>
4+#include <bpf/bpf_helpers.h>
5+#include <bpf/bpf_tracing.h>
6+#include <bpf/bpf_core_read.h>
7+
8+#pragma clang attribute push (__attribute__((preserve_access_index)), apply_to = record)
9+struct seq_file;
10+struct bpf_iter_meta {
11+ struct seq_file *seq;
12+ __u64 session_id;
13+ __u64 seq_num;
14+};
15+
16+struct bpf_map {
17+ __u32 id;
18+ char name[16];
19+ __u32 max_entries;
20+};
21+
22+struct bpf_iter__bpf_map {
23+ struct bpf_iter_meta *meta;
24+ struct bpf_map *map;
25+};
26+
27+struct btf_type {
28+ __u32 name_off;
29+};
30+
31+struct btf_header {
32+ __u32 str_len;
33+};
34+
35+struct btf {
36+ const char *strings;
37+ struct btf_type **types;
38+ struct btf_header hdr;
39+};
40+
41+struct bpf_prog_aux {
42+ __u32 id;
43+ char name[16];
44+ const char *attach_func_name;
45+ struct bpf_prog *dst_prog;
46+ struct bpf_func_info *func_info;
47+ struct btf *btf;
48+};
49+
50+struct bpf_prog {
51+ struct bpf_prog_aux *aux;
52+};
53+
54+struct bpf_iter__bpf_prog {
55+ struct bpf_iter_meta *meta;
56+ struct bpf_prog *prog;
57+};
58+#pragma clang attribute pop
59+
60+static const char *get_name(struct btf *btf, long btf_id, const char *fallback)
61+{
62+ struct btf_type **types, *t;
63+ unsigned int name_off;
64+ const char *str;
65+
66+ if (!btf)
67+ return fallback;
68+ str = btf->strings;
69+ types = btf->types;
70+ bpf_probe_read_kernel(&t, sizeof(t), types + btf_id);
71+ name_off = BPF_CORE_READ(t, name_off);
72+ if (name_off >= btf->hdr.str_len)
73+ return fallback;
74+ return str + name_off;
75+}
76+
77+SEC("iter/bpf_map")
78+int dump_bpf_map(struct bpf_iter__bpf_map *ctx)
79+{
80+ struct seq_file *seq = ctx->meta->seq;
81+ __u64 seq_num = ctx->meta->seq_num;
82+ struct bpf_map *map = ctx->map;
83+
84+ if (!map)
85+ return 0;
86+
87+ if (seq_num == 0)
88+ BPF_SEQ_PRINTF(seq, " id name max_entries\n");
89+
90+ BPF_SEQ_PRINTF(seq, "%4u %-16s%6d\n", map->id, map->name, map->max_entries);
91+ return 0;
92+}
93+
94+SEC("iter/bpf_prog")
95+int dump_bpf_prog(struct bpf_iter__bpf_prog *ctx)
96+{
97+ struct seq_file *seq = ctx->meta->seq;
98+ __u64 seq_num = ctx->meta->seq_num;
99+ struct bpf_prog *prog = ctx->prog;
100+ struct bpf_prog_aux *aux;
101+
102+ if (!prog)
103+ return 0;
104+
105+ aux = prog->aux;
106+ if (seq_num == 0)
107+ BPF_SEQ_PRINTF(seq, " id name attached\n");
108+
109+ BPF_SEQ_PRINTF(seq, "%4u %-16s %s %s\n", aux->id,
110+ get_name(aux->btf, aux->func_info[0].type_id, aux->name),
111+ aux->attach_func_name, aux->dst_prog->aux->name);
112+ return 0;
113+}
114+char LICENSE[] SEC("license") = "GPL";
kernel/bpf/preload/iterators/iterators.c+94−0
@@ -0,0 +1,94 @@
1+// SPDX-License-Identifier: GPL-2.0
2+/* Copyright (c) 2020 Facebook */
3+#include <argp.h>
4+#include <stdio.h>
5+#include <stdlib.h>
6+#include <string.h>
7+#include <unistd.h>
8+#include <fcntl.h>
9+#include <sys/resource.h>
10+#include <bpf/libbpf.h>
11+#include <bpf/bpf.h>
12+#include <sys/mount.h>
13+#include "iterators.skel.h"
14+#include "bpf_preload_common.h"
15+
16+int to_kernel = -1;
17+int from_kernel = 0;
18+
19+static int send_link_to_kernel(struct bpf_link *link, const char *link_name)
20+{
21+ struct bpf_preload_info obj = {};
22+ struct bpf_link_info info = {};
23+ __u32 info_len = sizeof(info);
24+ int err;
25+
26+ err = bpf_obj_get_info_by_fd(bpf_link__fd(link), &info, &info_len);
27+ if (err)
28+ return err;
29+ obj.link_id = info.id;
30+ if (strlen(link_name) >= sizeof(obj.link_name))
31+ return -E2BIG;
32+ strcpy(obj.link_name, link_name);
33+ if (write(to_kernel, &obj, sizeof(obj)) != sizeof(obj))
34+ return -EPIPE;
35+ return 0;
36+}
37+
38+int main(int argc, char **argv)
39+{
40+ struct rlimit rlim = { RLIM_INFINITY, RLIM_INFINITY };
41+ struct iterators_bpf *skel;
42+ int err, magic;
43+ int debug_fd;
44+
45+ debug_fd = open("/dev/console", O_WRONLY | O_NOCTTY | O_CLOEXEC);
46+ if (debug_fd < 0)
47+ return 1;
48+ to_kernel = dup(1);
49+ close(1);
50+ dup(debug_fd);
51+ /* now stdin and stderr point to /dev/console */
52+
53+ read(from_kernel, &magic, sizeof(magic));
54+ if (magic != BPF_PRELOAD_START) {
55+ printf("bad start magic %d\n", magic);
56+ return 1;
57+ }
58+ setrlimit(RLIMIT_MEMLOCK, &rlim);
59+ /* libbpf opens BPF object and loads it into the kernel */
60+ skel = iterators_bpf__open_and_load();
61+ if (!skel) {
62+ /* iterators.skel.h is little endian.
63+ * libbpf doesn't support automatic little->big conversion
64+ * of BPF bytecode yet.
65+ * The program load will fail in such case.
66+ */
67+ printf("Failed load could be due to wrong endianness\n");
68+ return 1;
69+ }
70+ err = iterators_bpf__attach(skel);
71+ if (err)
72+ goto cleanup;
73+
74+ /* send two bpf_link IDs with names to the kernel */
75+ err = send_link_to_kernel(skel->links.dump_bpf_map, "maps.debug");
76+ if (err)
77+ goto cleanup;
78+ err = send_link_to_kernel(skel->links.dump_bpf_prog, "progs.debug");
79+ if (err)
80+ goto cleanup;
81+
82+ /* The kernel will proceed with pinnging the links in bpffs.
83+ * UMD will wait on read from pipe.
84+ */
85+ read(from_kernel, &magic, sizeof(magic));
86+ if (magic != BPF_PRELOAD_END) {
87+ printf("bad final magic %d\n", magic);
88+ err = -EINVAL;
89+ }
90+cleanup:
91+ iterators_bpf__destroy(skel);
92+
93+ return err != 0;
94+}
kernel/bpf/preload/iterators/iterators.skel.h+412−0
@@ -0,0 +1,412 @@
1+/* SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause) */
2+
3+/* THIS FILE IS AUTOGENERATED! */
4+#ifndef __ITERATORS_BPF_SKEL_H__
5+#define __ITERATORS_BPF_SKEL_H__
6+
7+#include <stdlib.h>
8+#include <bpf/libbpf.h>
9+
10+struct iterators_bpf {
11+ struct bpf_object_skeleton *skeleton;
12+ struct bpf_object *obj;
13+ struct {
14+ struct bpf_map *rodata;
15+ } maps;
16+ struct {
17+ struct bpf_program *dump_bpf_map;
18+ struct bpf_program *dump_bpf_prog;
19+ } progs;
20+ struct {
21+ struct bpf_link *dump_bpf_map;
22+ struct bpf_link *dump_bpf_prog;
23+ } links;
24+ struct iterators_bpf__rodata {
25+ char dump_bpf_map____fmt[35];
26+ char dump_bpf_map____fmt_1[14];
27+ char dump_bpf_prog____fmt[32];
28+ char dump_bpf_prog____fmt_2[17];
29+ } *rodata;
30+};
31+
32+static void
33+iterators_bpf__destroy(struct iterators_bpf *obj)
34+{
35+ if (!obj)
36+ return;
37+ if (obj->skeleton)
38+ bpf_object__destroy_skeleton(obj->skeleton);
39+ free(obj);
40+}
41+
42+static inline int
43+iterators_bpf__create_skeleton(struct iterators_bpf *obj);
44+
45+static inline struct iterators_bpf *
46+iterators_bpf__open_opts(const struct bpf_object_open_opts *opts)
47+{
48+ struct iterators_bpf *obj;
49+
50+ obj = (struct iterators_bpf *)calloc(1, sizeof(*obj));
51+ if (!obj)
52+ return NULL;
53+ if (iterators_bpf__create_skeleton(obj))
54+ goto err;
55+ if (bpf_object__open_skeleton(obj->skeleton, opts))
56+ goto err;
57+
58+ return obj;
59+err:
60+ iterators_bpf__destroy(obj);
61+ return NULL;
62+}
63+
64+static inline struct iterators_bpf *
65+iterators_bpf__open(void)
66+{
67+ return iterators_bpf__open_opts(NULL);
68+}
69+
70+static inline int
71+iterators_bpf__load(struct iterators_bpf *obj)
72+{
73+ return bpf_object__load_skeleton(obj->skeleton);
74+}
75+
76+static inline struct iterators_bpf *
77+iterators_bpf__open_and_load(void)
78+{
79+ struct iterators_bpf *obj;
80+
81+ obj = iterators_bpf__open();
82+ if (!obj)
83+ return NULL;
84+ if (iterators_bpf__load(obj)) {
85+ iterators_bpf__destroy(obj);
86+ return NULL;
87+ }
88+ return obj;
89+}
90+
91+static inline int
92+iterators_bpf__attach(struct iterators_bpf *obj)
93+{
94+ return bpf_object__attach_skeleton(obj->skeleton);
95+}
96+
97+static inline void
98+iterators_bpf__detach(struct iterators_bpf *obj)
99+{
100+ return bpf_object__detach_skeleton(obj->skeleton);
101+}
102+
103+static inline int
104+iterators_bpf__create_skeleton(struct iterators_bpf *obj)
105+{
106+ struct bpf_object_skeleton *s;
107+
108+ s = (struct bpf_object_skeleton *)calloc(1, sizeof(*s));
109+ if (!s)
110+ return -1;
111+ obj->skeleton = s;
112+
113+ s->sz = sizeof(*s);
114+ s->name = "iterators_bpf";
115+ s->obj = &obj->obj;
116+
117+ /* maps */
118+ s->map_cnt = 1;
119+ s->map_skel_sz = sizeof(*s->maps);
120+ s->maps = (struct bpf_map_skeleton *)calloc(s->map_cnt, s->map_skel_sz);
121+ if (!s->maps)
122+ goto err;
123+
124+ s->maps[0].name = "iterator.rodata";
125+ s->maps[0].map = &obj->maps.rodata;
126+ s->maps[0].mmaped = (void **)&obj->rodata;
127+
128+ /* programs */
129+ s->prog_cnt = 2;
130+ s->prog_skel_sz = sizeof(*s->progs);
131+ s->progs = (struct bpf_prog_skeleton *)calloc(s->prog_cnt, s->prog_skel_sz);
132+ if (!s->progs)
133+ goto err;
134+
135+ s->progs[0].name = "dump_bpf_map";
136+ s->progs[0].prog = &obj->progs.dump_bpf_map;
137+ s->progs[0].link = &obj->links.dump_bpf_map;
138+
139+ s->progs[1].name = "dump_bpf_prog";
140+ s->progs[1].prog = &obj->progs.dump_bpf_prog;
141+ s->progs[1].link = &obj->links.dump_bpf_prog;
142+
143+ s->data_sz = 7176;
144+ s->data = (void *)"\
145+\x7f\x45\x4c\x46\x02\x01\x01\0\0\0\0\0\0\0\0\0\x01\0\xf7\0\x01\0\0\0\0\0\0\0\0\
146+\0\0\0\0\0\0\0\0\0\0\0\x48\x18\0\0\0\0\0\0\0\0\0\0\x40\0\0\0\0\0\x40\0\x0f\0\
147+\x0e\0\x79\x12\0\0\0\0\0\0\x79\x26\0\0\0\0\0\0\x79\x17\x08\0\0\0\0\0\x15\x07\
148+\x1a\0\0\0\0\0\x79\x21\x10\0\0\0\0\0\x55\x01\x08\0\0\0\0\0\xbf\xa4\0\0\0\0\0\0\
149+\x07\x04\0\0\xe8\xff\xff\xff\xbf\x61\0\0\0\0\0\0\x18\x02\0\0\0\0\0\0\0\0\0\0\0\
150+\0\0\0\xb7\x03\0\0\x23\0\0\0\xb7\x05\0\0\0\0\0\0\x85\0\0\0\x7e\0\0\0\x61\x71\0\
151+\0\0\0\0\0\x7b\x1a\xe8\xff\0\0\0\0\xb7\x01\0\0\x04\0\0\0\xbf\x72\0\0\0\0\0\0\
152+\x0f\x12\0\0\0\0\0\0\x7b\x2a\xf0\xff\0\0\0\0\x61\x71\x14\0\0\0\0\0\x7b\x1a\xf8\
153+\xff\0\0\0\0\xbf\xa4\0\0\0\0\0\0\x07\x04\0\0\xe8\xff\xff\xff\xbf\x61\0\0\0\0\0\
154+\0\x18\x02\0\0\x23\0\0\0\0\0\0\0\0\0\0\0\xb7\x03\0\0\x0e\0\0\0\xb7\x05\0\0\x18\
155+\0\0\0\x85\0\0\0\x7e\0\0\0\xb7\0\0\0\0\0\0\0\x95\0\0\0\0\0\0\0\x79\x12\0\0\0\0\
156+\0\0\x79\x26\0\0\0\0\0\0\x79\x11\x08\0\0\0\0\0\x15\x01\x3b\0\0\0\0\0\x79\x17\0\
157+\0\0\0\0\0\x79\x21\x10\0\0\0\0\0\x55\x01\x08\0\0\0\0\0\xbf\xa4\0\0\0\0\0\0\x07\
158+\x04\0\0\xd0\xff\xff\xff\xbf\x61\0\0\0\0\0\0\x18\x02\0\0\x31\0\0\0\0\0\0\0\0\0\
159+\0\0\xb7\x03\0\0\x20\0\0\0\xb7\x05\0\0\0\0\0\0\x85\0\0\0\x7e\0\0\0\x7b\x6a\xc8\
160+\xff\0\0\0\0\x61\x71\0\0\0\0\0\0\x7b\x1a\xd0\xff\0\0\0\0\xb7\x03\0\0\x04\0\0\0\
161+\xbf\x79\0\0\0\0\0\0\x0f\x39\0\0\0\0\0\0\x79\x71\x28\0\0\0\0\0\x79\x78\x30\0\0\
162+\0\0\0\x15\x08\x18\0\0\0\0\0\xb7\x02\0\0\0\0\0\0\x0f\x21\0\0\0\0\0\0\x61\x11\
163+\x04\0\0\0\0\0\x79\x83\x08\0\0\0\0\0\x67\x01\0\0\x03\0\0\0\x0f\x13\0\0\0\0\0\0\
164+\x79\x86\0\0\0\0\0\0\xbf\xa1\0\0\0\0\0\0\x07\x01\0\0\xf8\xff\xff\xff\xb7\x02\0\
165+\0\x08\0\0\0\x85\0\0\0\x71\0\0\0\xb7\x01\0\0\0\0\0\0\x79\xa3\xf8\xff\0\0\0\0\
166+\x0f\x13\0\0\0\0\0\0\xbf\xa1\0\0\0\0\0\0\x07\x01\0\0\xf4\xff\xff\xff\xb7\x02\0\
167+\0\x04\0\0\0\x85\0\0\0\x71\0\0\0\xb7\x03\0\0\x04\0\0\0\x61\xa1\xf4\xff\0\0\0\0\
168+\x61\x82\x10\0\0\0\0\0\x3d\x21\x02\0\0\0\0\0\x0f\x16\0\0\0\0\0\0\xbf\x69\0\0\0\
169+\0\0\0\x7b\x9a\xd8\xff\0\0\0\0\x79\x71\x18\0\0\0\0\0\x7b\x1a\xe0\xff\0\0\0\0\
170+\x79\x71\x20\0\0\0\0\0\x79\x11\0\0\0\0\0\0\x0f\x31\0\0\0\0\0\0\x7b\x1a\xe8\xff\
171+\0\0\0\0\xbf\xa4\0\0\0\0\0\0\x07\x04\0\0\xd0\xff\xff\xff\x79\xa1\xc8\xff\0\0\0\
172+\0\x18\x02\0\0\x51\0\0\0\0\0\0\0\0\0\0\0\xb7\x03\0\0\x11\0\0\0\xb7\x05\0\0\x20\
173+\0\0\0\x85\0\0\0\x7e\0\0\0\xb7\0\0\0\0\0\0\0\x95\0\0\0\0\0\0\0\x20\x20\x69\x64\
174+\x20\x6e\x61\x6d\x65\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x6d\
175+\x61\x78\x5f\x65\x6e\x74\x72\x69\x65\x73\x0a\0\x25\x34\x75\x20\x25\x2d\x31\x36\
176+\x73\x25\x36\x64\x0a\0\x20\x20\x69\x64\x20\x6e\x61\x6d\x65\x20\x20\x20\x20\x20\
177+\x20\x20\x20\x20\x20\x20\x20\x20\x61\x74\x74\x61\x63\x68\x65\x64\x0a\0\x25\x34\
178+\x75\x20\x25\x2d\x31\x36\x73\x20\x25\x73\x20\x25\x73\x0a\0\x47\x50\x4c\0\x9f\
179+\xeb\x01\0\x18\0\0\0\0\0\0\0\x1c\x04\0\0\x1c\x04\0\0\x09\x05\0\0\0\0\0\0\0\0\0\
180+\x02\x02\0\0\0\x01\0\0\0\x02\0\0\x04\x10\0\0\0\x13\0\0\0\x03\0\0\0\0\0\0\0\x18\
181+\0\0\0\x04\0\0\0\x40\0\0\0\0\0\0\0\0\0\0\x02\x08\0\0\0\0\0\0\0\0\0\0\x02\x0d\0\
182+\0\0\0\0\0\0\x01\0\0\x0d\x06\0\0\0\x1c\0\0\0\x01\0\0\0\x20\0\0\0\0\0\0\x01\x04\
183+\0\0\0\x20\0\0\x01\x24\0\0\0\x01\0\0\x0c\x05\0\0\0\xaf\0\0\0\x03\0\0\x04\x18\0\
184+\0\0\xbd\0\0\0\x09\0\0\0\0\0\0\0\xc1\0\0\0\x0b\0\0\0\x40\0\0\0\xcc\0\0\0\x0b\0\
185+\0\0\x80\0\0\0\0\0\0\0\0\0\0\x02\x0a\0\0\0\xd4\0\0\0\0\0\0\x07\0\0\0\0\xdd\0\0\
186+\0\0\0\0\x08\x0c\0\0\0\xe3\0\0\0\0\0\0\x01\x08\0\0\0\x40\0\0\0\xa4\x01\0\0\x03\
187+\0\0\x04\x18\0\0\0\xac\x01\0\0\x0e\0\0\0\0\0\0\0\xaf\x01\0\0\x11\0\0\0\x20\0\0\
188+\0\xb4\x01\0\0\x0e\0\0\0\xa0\0\0\0\xc0\x01\0\0\0\0\0\x08\x0f\0\0\0\xc6\x01\0\0\
189+\0\0\0\x01\x04\0\0\0\x20\0\0\0\xd3\x01\0\0\0\0\0\x01\x01\0\0\0\x08\0\0\x01\0\0\
190+\0\0\0\0\0\x03\0\0\0\0\x10\0\0\0\x12\0\0\0\x10\0\0\0\xd8\x01\0\0\0\0\0\x01\x04\
191+\0\0\0\x20\0\0\0\0\0\0\0\0\0\0\x02\x14\0\0\0\x3c\x02\0\0\x02\0\0\x04\x10\0\0\0\
192+\x13\0\0\0\x03\0\0\0\0\0\0\0\x4f\x02\0\0\x15\0\0\0\x40\0\0\0\0\0\0\0\0\0\0\x02\
193+\x18\0\0\0\0\0\0\0\x01\0\0\x0d\x06\0\0\0\x1c\0\0\0\x13\0\0\0\x54\x02\0\0\x01\0\
194+\0\x0c\x16\0\0\0\xa0\x02\0\0\x01\0\0\x04\x08\0\0\0\xa9\x02\0\0\x19\0\0\0\0\0\0\
195+\0\0\0\0\0\0\0\0\x02\x1a\0\0\0\xfa\x02\0\0\x06\0\0\x04\x38\0\0\0\xac\x01\0\0\
196+\x0e\0\0\0\0\0\0\0\xaf\x01\0\0\x11\0\0\0\x20\0\0\0\x07\x03\0\0\x1b\0\0\0\xc0\0\
197+\0\0\x18\x03\0\0\x15\0\0\0\0\x01\0\0\x21\x03\0\0\x1d\0\0\0\x40\x01\0\0\x2b\x03\
198+\0\0\x1e\0\0\0\x80\x01\0\0\0\0\0\0\0\0\0\x02\x1c\0\0\0\0\0\0\0\0\0\0\x0a\x10\0\
199+\0\0\0\0\0\0\0\0\0\x02\x1f\0\0\0\0\0\0\0\0\0\0\x02\x20\0\0\0\x75\x03\0\0\x02\0\
200+\0\x04\x08\0\0\0\x83\x03\0\0\x0e\0\0\0\0\0\0\0\x8c\x03\0\0\x0e\0\0\0\x20\0\0\0\
201+\x2b\x03\0\0\x03\0\0\x04\x18\0\0\0\x96\x03\0\0\x1b\0\0\0\0\0\0\0\x9e\x03\0\0\
202+\x21\0\0\0\x40\0\0\0\xa4\x03\0\0\x23\0\0\0\x80\0\0\0\0\0\0\0\0\0\0\x02\x22\0\0\
203+\0\0\0\0\0\0\0\0\x02\x24\0\0\0\xa8\x03\0\0\x01\0\0\x04\x04\0\0\0\xb3\x03\0\0\
204+\x0e\0\0\0\0\0\0\0\x1c\x04\0\0\x01\0\0\x04\x04\0\0\0\x25\x04\0\0\x0e\0\0\0\0\0\
205+\0\0\0\0\0\0\0\0\0\x03\0\0\0\0\x1c\0\0\0\x12\0\0\0\x23\0\0\0\x9b\x04\0\0\0\0\0\
206+\x0e\x25\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x03\0\0\0\0\x1c\0\0\0\x12\0\0\0\x0e\0\0\0\
207+\xaf\x04\0\0\0\0\0\x0e\x27\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x03\0\0\0\0\x1c\0\0\0\
208+\x12\0\0\0\x20\0\0\0\xc5\x04\0\0\0\0\0\x0e\x29\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x03\
209+\0\0\0\0\x1c\0\0\0\x12\0\0\0\x11\0\0\0\xda\x04\0\0\0\0\0\x0e\x2b\0\0\0\0\0\0\0\
210+\0\0\0\0\0\0\0\x03\0\0\0\0\x10\0\0\0\x12\0\0\0\x04\0\0\0\xf1\x04\0\0\0\0\0\x0e\
211+\x2d\0\0\0\x01\0\0\0\xf9\x04\0\0\x04\0\0\x0f\0\0\0\0\x26\0\0\0\0\0\0\0\x23\0\0\
212+\0\x28\0\0\0\x23\0\0\0\x0e\0\0\0\x2a\0\0\0\x31\0\0\0\x20\0\0\0\x2c\0\0\0\x51\0\
213+\0\0\x11\0\0\0\x01\x05\0\0\x01\0\0\x0f\0\0\0\0\x2e\0\0\0\0\0\0\0\x04\0\0\0\0\
214+\x62\x70\x66\x5f\x69\x74\x65\x72\x5f\x5f\x62\x70\x66\x5f\x6d\x61\x70\0\x6d\x65\
215+\x74\x61\0\x6d\x61\x70\0\x63\x74\x78\0\x69\x6e\x74\0\x64\x75\x6d\x70\x5f\x62\
216+\x70\x66\x5f\x6d\x61\x70\0\x69\x74\x65\x72\x2f\x62\x70\x66\x5f\x6d\x61\x70\0\
217+\x30\x3a\x30\0\x2f\x68\x6f\x6d\x65\x2f\x61\x6c\x72\x75\x61\x2f\x62\x75\x69\x6c\
218+\x64\x2f\x6c\x69\x6e\x75\x78\x2f\x6b\x65\x72\x6e\x65\x6c\x2f\x62\x70\x66\x2f\
219+\x70\x72\x65\x6c\x6f\x61\x64\x2f\x69\x74\x65\x72\x61\x74\x6f\x72\x73\x2f\x69\
220+\x74\x65\x72\x61\x74\x6f\x72\x73\x2e\x62\x70\x66\x2e\x63\0\x09\x73\x74\x72\x75\
221+\x63\x74\x20\x73\x65\x71\x5f\x66\x69\x6c\x65\x20\x2a\x73\x65\x71\x20\x3d\x20\
222+\x63\x74\x78\x2d\x3e\x6d\x65\x74\x61\x2d\x3e\x73\x65\x71\x3b\0\x62\x70\x66\x5f\
223+\x69\x74\x65\x72\x5f\x6d\x65\x74\x61\0\x73\x65\x71\0\x73\x65\x73\x73\x69\x6f\
224+\x6e\x5f\x69\x64\0\x73\x65\x71\x5f\x6e\x75\x6d\0\x73\x65\x71\x5f\x66\x69\x6c\
225+\x65\0\x5f\x5f\x75\x36\x34\0\x6c\x6f\x6e\x67\x20\x6c\x6f\x6e\x67\x20\x75\x6e\
226+\x73\x69\x67\x6e\x65\x64\x20\x69\x6e\x74\0\x30\x3a\x31\0\x09\x73\x74\x72\x75\
227+\x63\x74\x20\x62\x70\x66\x5f\x6d\x61\x70\x20\x2a\x6d\x61\x70\x20\x3d\x20\x63\
228+\x74\x78\x2d\x3e\x6d\x61\x70\x3b\0\x09\x69\x66\x20\x28\x21\x6d\x61\x70\x29\0\
229+\x30\x3a\x32\0\x09\x5f\x5f\x75\x36\x34\x20\x73\x65\x71\x5f\x6e\x75\x6d\x20\x3d\
230+\x20\x63\x74\x78\x2d\x3e\x6d\x65\x74\x61\x2d\x3e\x73\x65\x71\x5f\x6e\x75\x6d\
231+\x3b\0\x09\x69\x66\x20\x28\x73\x65\x71\x5f\x6e\x75\x6d\x20\x3d\x3d\x20\x30\x29\
232+\0\x09\x09\x42\x50\x46\x5f\x53\x45\x51\x5f\x50\x52\x49\x4e\x54\x46\x28\x73\x65\
233+\x71\x2c\x20\x22\x20\x20\x69\x64\x20\x6e\x61\x6d\x65\x20\x20\x20\x20\x20\x20\
234+\x20\x20\x20\x20\x20\x20\x20\x6d\x61\x78\x5f\x65\x6e\x74\x72\x69\x65\x73\x5c\
235+\x6e\x22\x29\x3b\0\x62\x70\x66\x5f\x6d\x61\x70\0\x69\x64\0\x6e\x61\x6d\x65\0\
236+\x6d\x61\x78\x5f\x65\x6e\x74\x72\x69\x65\x73\0\x5f\x5f\x75\x33\x32\0\x75\x6e\
237+\x73\x69\x67\x6e\x65\x64\x20\x69\x6e\x74\0\x63\x68\x61\x72\0\x5f\x5f\x41\x52\
238+\x52\x41\x59\x5f\x53\x49\x5a\x45\x5f\x54\x59\x50\x45\x5f\x5f\0\x09\x42\x50\x46\
239+\x5f\x53\x45\x51\x5f\x50\x52\x49\x4e\x54\x46\x28\x73\x65\x71\x2c\x20\x22\x25\
240+\x34\x75\x20\x25\x2d\x31\x36\x73\x25\x36\x64\x5c\x6e\x22\x2c\x20\x6d\x61\x70\
241+\x2d\x3e\x69\x64\x2c\x20\x6d\x61\x70\x2d\x3e\x6e\x61\x6d\x65\x2c\x20\x6d\x61\
242+\x70\x2d\x3e\x6d\x61\x78\x5f\x65\x6e\x74\x72\x69\x65\x73\x29\x3b\0\x7d\0\x62\
243+\x70\x66\x5f\x69\x74\x65\x72\x5f\x5f\x62\x70\x66\x5f\x70\x72\x6f\x67\0\x70\x72\
244+\x6f\x67\0\x64\x75\x6d\x70\x5f\x62\x70\x66\x5f\x70\x72\x6f\x67\0\x69\x74\x65\
245+\x72\x2f\x62\x70\x66\x5f\x70\x72\x6f\x67\0\x09\x73\x74\x72\x75\x63\x74\x20\x62\
246+\x70\x66\x5f\x70\x72\x6f\x67\x20\x2a\x70\x72\x6f\x67\x20\x3d\x20\x63\x74\x78\
247+\x2d\x3e\x70\x72\x6f\x67\x3b\0\x09\x69\x66\x20\x28\x21\x70\x72\x6f\x67\x29\0\
248+\x62\x70\x66\x5f\x70\x72\x6f\x67\0\x61\x75\x78\0\x09\x61\x75\x78\x20\x3d\x20\
249+\x70\x72\x6f\x67\x2d\x3e\x61\x75\x78\x3b\0\x09\x09\x42\x50\x46\x5f\x53\x45\x51\
250+\x5f\x50\x52\x49\x4e\x54\x46\x28\x73\x65\x71\x2c\x20\x22\x20\x20\x69\x64\x20\
251+\x6e\x61\x6d\x65\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x61\x74\
252+\x74\x61\x63\x68\x65\x64\x5c\x6e\x22\x29\x3b\0\x62\x70\x66\x5f\x70\x72\x6f\x67\
253+\x5f\x61\x75\x78\0\x61\x74\x74\x61\x63\x68\x5f\x66\x75\x6e\x63\x5f\x6e\x61\x6d\
254+\x65\0\x64\x73\x74\x5f\x70\x72\x6f\x67\0\x66\x75\x6e\x63\x5f\x69\x6e\x66\x6f\0\
255+\x62\x74\x66\0\x09\x42\x50\x46\x5f\x53\x45\x51\x5f\x50\x52\x49\x4e\x54\x46\x28\
256+\x73\x65\x71\x2c\x20\x22\x25\x34\x75\x20\x25\x2d\x31\x36\x73\x20\x25\x73\x20\
257+\x25\x73\x5c\x6e\x22\x2c\x20\x61\x75\x78\x2d\x3e\x69\x64\x2c\0\x30\x3a\x34\0\
258+\x30\x3a\x35\0\x09\x69\x66\x20\x28\x21\x62\x74\x66\x29\0\x62\x70\x66\x5f\x66\
259+\x75\x6e\x63\x5f\x69\x6e\x66\x6f\0\x69\x6e\x73\x6e\x5f\x6f\x66\x66\0\x74\x79\
260+\x70\x65\x5f\x69\x64\0\x30\0\x73\x74\x72\x69\x6e\x67\x73\0\x74\x79\x70\x65\x73\
261+\0\x68\x64\x72\0\x62\x74\x66\x5f\x68\x65\x61\x64\x65\x72\0\x73\x74\x72\x5f\x6c\
262+\x65\x6e\0\x09\x74\x79\x70\x65\x73\x20\x3d\x20\x62\x74\x66\x2d\x3e\x74\x79\x70\
263+\x65\x73\x3b\0\x09\x62\x70\x66\x5f\x70\x72\x6f\x62\x65\x5f\x72\x65\x61\x64\x5f\
264+\x6b\x65\x72\x6e\x65\x6c\x28\x26\x74\x2c\x20\x73\x69\x7a\x65\x6f\x66\x28\x74\
265+\x29\x2c\x20\x74\x79\x70\x65\x73\x20\x2b\x20\x62\x74\x66\x5f\x69\x64\x29\x3b\0\
266+\x09\x73\x74\x72\x20\x3d\x20\x62\x74\x66\x2d\x3e\x73\x74\x72\x69\x6e\x67\x73\
267+\x3b\0\x62\x74\x66\x5f\x74\x79\x70\x65\0\x6e\x61\x6d\x65\x5f\x6f\x66\x66\0\x09\
268+\x6e\x61\x6d\x65\x5f\x6f\x66\x66\x20\x3d\x20\x42\x50\x46\x5f\x43\x4f\x52\x45\
269+\x5f\x52\x45\x41\x44\x28\x74\x2c\x20\x6e\x61\x6d\x65\x5f\x6f\x66\x66\x29\x3b\0\
270+\x30\x3a\x32\x3a\x30\0\x09\x69\x66\x20\x28\x6e\x61\x6d\x65\x5f\x6f\x66\x66\x20\
271+\x3e\x3d\x20\x62\x74\x66\x2d\x3e\x68\x64\x72\x2e\x73\x74\x72\x5f\x6c\x65\x6e\
272+\x29\0\x09\x72\x65\x74\x75\x72\x6e\x20\x73\x74\x72\x20\x2b\x20\x6e\x61\x6d\x65\
273+\x5f\x6f\x66\x66\x3b\0\x30\x3a\x33\0\x64\x75\x6d\x70\x5f\x62\x70\x66\x5f\x6d\
274+\x61\x70\x2e\x5f\x5f\x5f\x66\x6d\x74\0\x64\x75\x6d\x70\x5f\x62\x70\x66\x5f\x6d\
275+\x61\x70\x2e\x5f\x5f\x5f\x66\x6d\x74\x2e\x31\0\x64\x75\x6d\x70\x5f\x62\x70\x66\
276+\x5f\x70\x72\x6f\x67\x2e\x5f\x5f\x5f\x66\x6d\x74\0\x64\x75\x6d\x70\x5f\x62\x70\
277+\x66\x5f\x70\x72\x6f\x67\x2e\x5f\x5f\x5f\x66\x6d\x74\x2e\x32\0\x4c\x49\x43\x45\
278+\x4e\x53\x45\0\x2e\x72\x6f\x64\x61\x74\x61\0\x6c\x69\x63\x65\x6e\x73\x65\0\x9f\
279+\xeb\x01\0\x20\0\0\0\0\0\0\0\x24\0\0\0\x24\0\0\0\x44\x02\0\0\x68\x02\0\0\xa4\
280+\x01\0\0\x08\0\0\0\x31\0\0\0\x01\0\0\0\0\0\0\0\x07\0\0\0\x62\x02\0\0\x01\0\0\0\
281+\0\0\0\0\x17\0\0\0\x10\0\0\0\x31\0\0\0\x09\0\0\0\0\0\0\0\x42\0\0\0\x87\0\0\0\
282+\x1e\x40\x01\0\x08\0\0\0\x42\0\0\0\x87\0\0\0\x24\x40\x01\0\x10\0\0\0\x42\0\0\0\
283+\xfe\0\0\0\x1d\x48\x01\0\x18\0\0\0\x42\0\0\0\x1f\x01\0\0\x06\x50\x01\0\x20\0\0\
284+\0\x42\0\0\0\x2e\x01\0\0\x1d\x44\x01\0\x28\0\0\0\x42\0\0\0\x53\x01\0\0\x06\x5c\
285+\x01\0\x38\0\0\0\x42\0\0\0\x66\x01\0\0\x03\x60\x01\0\x70\0\0\0\x42\0\0\0\xec\
286+\x01\0\0\x02\x68\x01\0\xf0\0\0\0\x42\0\0\0\x3a\x02\0\0\x01\x70\x01\0\x62\x02\0\
287+\0\x1a\0\0\0\0\0\0\0\x42\0\0\0\x87\0\0\0\x1e\x84\x01\0\x08\0\0\0\x42\0\0\0\x87\
288+\0\0\0\x24\x84\x01\0\x10\0\0\0\x42\0\0\0\x70\x02\0\0\x1f\x8c\x01\0\x18\0\0\0\
289+\x42\0\0\0\x94\x02\0\0\x06\x98\x01\0\x20\0\0\0\x42\0\0\0\xad\x02\0\0\x0e\xa4\
290+\x01\0\x28\0\0\0\x42\0\0\0\x2e\x01\0\0\x1d\x88\x01\0\x30\0\0\0\x42\0\0\0\x53\
291+\x01\0\0\x06\xa8\x01\0\x40\0\0\0\x42\0\0\0\xbf\x02\0\0\x03\xac\x01\0\x80\0\0\0\
292+\x42\0\0\0\x2f\x03\0\0\x02\xb4\x01\0\xb8\0\0\0\x42\0\0\0\x6a\x03\0\0\x06\x08\
293+\x01\0\xd0\0\0\0\x42\0\0\0\0\0\0\0\0\0\0\0\xd8\0\0\0\x42\0\0\0\xbb\x03\0\0\x0f\
294+\x14\x01\0\xe0\0\0\0\x42\0\0\0\xd0\x03\0\0\x2d\x18\x01\0\xf0\0\0\0\x42\0\0\0\
295+\x07\x04\0\0\x0d\x10\x01\0\0\x01\0\0\x42\0\0\0\0\0\0\0\0\0\0\0\x08\x01\0\0\x42\
296+\0\0\0\xd0\x03\0\0\x02\x18\x01\0\x20\x01\0\0\x42\0\0\0\x2e\x04\0\0\x0d\x1c\x01\
297+\0\x38\x01\0\0\x42\0\0\0\0\0\0\0\0\0\0\0\x40\x01\0\0\x42\0\0\0\x2e\x04\0\0\x0d\
298+\x1c\x01\0\x58\x01\0\0\x42\0\0\0\x2e\x04\0\0\x0d\x1c\x01\0\x60\x01\0\0\x42\0\0\
299+\0\x5c\x04\0\0\x1b\x20\x01\0\x68\x01\0\0\x42\0\0\0\x5c\x04\0\0\x06\x20\x01\0\
300+\x70\x01\0\0\x42\0\0\0\x7f\x04\0\0\x0d\x28\x01\0\x78\x01\0\0\x42\0\0\0\0\0\0\0\
301+\0\0\0\0\x80\x01\0\0\x42\0\0\0\x2f\x03\0\0\x02\xb4\x01\0\xf8\x01\0\0\x42\0\0\0\
302+\x3a\x02\0\0\x01\xc4\x01\0\x10\0\0\0\x31\0\0\0\x07\0\0\0\0\0\0\0\x02\0\0\0\x3e\
303+\0\0\0\0\0\0\0\x08\0\0\0\x08\0\0\0\x3e\0\0\0\0\0\0\0\x10\0\0\0\x02\0\0\0\xfa\0\
304+\0\0\0\0\0\0\x20\0\0\0\x08\0\0\0\x2a\x01\0\0\0\0\0\0\x70\0\0\0\x0d\0\0\0\x3e\0\
305+\0\0\0\0\0\0\x80\0\0\0\x0d\0\0\0\xfa\0\0\0\0\0\0\0\xa0\0\0\0\x0d\0\0\0\x2a\x01\
306+\0\0\0\0\0\0\x62\x02\0\0\x12\0\0\0\0\0\0\0\x14\0\0\0\x3e\0\0\0\0\0\0\0\x08\0\0\
307+\0\x08\0\0\0\x3e\0\0\0\0\0\0\0\x10\0\0\0\x14\0\0\0\xfa\0\0\0\0\0\0\0\x20\0\0\0\
308+\x18\0\0\0\x3e\0\0\0\0\0\0\0\x28\0\0\0\x08\0\0\0\x2a\x01\0\0\0\0\0\0\x80\0\0\0\
309+\x1a\0\0\0\x3e\0\0\0\0\0\0\0\x90\0\0\0\x1a\0\0\0\xfa\0\0\0\0\0\0\0\xa8\0\0\0\
310+\x1a\0\0\0\x62\x03\0\0\0\0\0\0\xb0\0\0\0\x1a\0\0\0\x66\x03\0\0\0\0\0\0\xc0\0\0\
311+\0\x1f\0\0\0\x94\x03\0\0\0\0\0\0\xd8\0\0\0\x20\0\0\0\xfa\0\0\0\0\0\0\0\xf0\0\0\
312+\0\x20\0\0\0\x3e\0\0\0\0\0\0\0\x18\x01\0\0\x24\0\0\0\x3e\0\0\0\0\0\0\0\x50\x01\
313+\0\0\x1a\0\0\0\xfa\0\0\0\0\0\0\0\x60\x01\0\0\x20\0\0\0\x56\x04\0\0\0\0\0\0\x88\
314+\x01\0\0\x1a\0\0\0\x2a\x01\0\0\0\0\0\0\x98\x01\0\0\x1a\0\0\0\x97\x04\0\0\0\0\0\
315+\0\xa0\x01\0\0\x18\0\0\0\x3e\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\
316+\0\0\0\0\0\0\0\x91\0\0\0\x04\0\xf1\xff\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\xe6\0\0\
317+\0\0\0\x02\0\x70\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\xd8\0\0\0\0\0\x02\0\xf0\0\0\0\0\
318+\0\0\0\0\0\0\0\0\0\0\0\xdf\0\0\0\0\0\x03\0\x78\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\
319+\xd1\0\0\0\0\0\x03\0\x80\x01\0\0\0\0\0\0\0\0\0\0\0\0\0\0\xca\0\0\0\0\0\x03\0\
320+\xf8\x01\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x14\0\0\0\x01\0\x04\0\0\0\0\0\0\0\0\0\x23\
321+\0\0\0\0\0\0\0\x04\x01\0\0\x01\0\x04\0\x23\0\0\0\0\0\0\0\x0e\0\0\0\0\0\0\0\x28\
322+\0\0\0\x01\0\x04\0\x31\0\0\0\0\0\0\0\x20\0\0\0\0\0\0\0\xed\0\0\0\x01\0\x04\0\
323+\x51\0\0\0\0\0\0\0\x11\0\0\0\0\0\0\0\0\0\0\0\x03\0\x02\0\0\0\0\0\0\0\0\0\0\0\0\
324+\0\0\0\0\0\0\0\0\0\x03\0\x03\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x03\0\
325+\x04\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\xc2\0\0\0\x11\0\x05\0\0\0\0\0\0\0\0\0\
326+\x04\0\0\0\0\0\0\0\x3d\0\0\0\x12\0\x02\0\0\0\0\0\0\0\0\0\0\x01\0\0\0\0\0\0\x5b\
327+\0\0\0\x12\0\x03\0\0\0\0\0\0\0\0\0\x08\x02\0\0\0\0\0\0\x48\0\0\0\0\0\0\0\x01\0\
328+\0\0\x0d\0\0\0\xc8\0\0\0\0\0\0\0\x01\0\0\0\x0d\0\0\0\x50\0\0\0\0\0\0\0\x01\0\0\
329+\0\x0d\0\0\0\xd0\x01\0\0\0\0\0\0\x01\0\0\0\x0d\0\0\0\xf0\x03\0\0\0\0\0\0\x0a\0\
330+\0\0\x0d\0\0\0\xfc\x03\0\0\0\0\0\0\x0a\0\0\0\x0d\0\0\0\x08\x04\0\0\0\0\0\0\x0a\
331+\0\0\0\x0d\0\0\0\x14\x04\0\0\0\0\0\0\x0a\0\0\0\x0d\0\0\0\x2c\x04\0\0\0\0\0\0\0\
332+\0\0\0\x0e\0\0\0\x2c\0\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\x3c\0\0\0\0\0\0\0\0\0\0\0\
333+\x0c\0\0\0\x50\0\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\x60\0\0\0\0\0\0\0\0\0\0\0\x0b\0\
334+\0\0\x70\0\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\x80\0\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\
335+\x90\0\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\xa0\0\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\xb0\0\
336+\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\xc0\0\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\xd0\0\0\0\0\
337+\0\0\0\0\0\0\0\x0b\0\0\0\xe8\0\0\0\0\0\0\0\0\0\0\0\x0c\0\0\0\xf8\0\0\0\0\0\0\0\
338+\0\0\0\0\x0c\0\0\0\x08\x01\0\0\0\0\0\0\0\0\0\0\x0c\0\0\0\x18\x01\0\0\0\0\0\0\0\
339+\0\0\0\x0c\0\0\0\x28\x01\0\0\0\0\0\0\0\0\0\0\x0c\0\0\0\x38\x01\0\0\0\0\0\0\0\0\
340+\0\0\x0c\0\0\0\x48\x01\0\0\0\0\0\0\0\0\0\0\x0c\0\0\0\x58\x01\0\0\0\0\0\0\0\0\0\
341+\0\x0c\0\0\0\x68\x01\0\0\0\0\0\0\0\0\0\0\x0c\0\0\0\x78\x01\0\0\0\0\0\0\0\0\0\0\
342+\x0c\0\0\0\x88\x01\0\0\0\0\0\0\0\0\0\0\x0c\0\0\0\x98\x01\0\0\0\0\0\0\0\0\0\0\
343+\x0c\0\0\0\xa8\x01\0\0\0\0\0\0\0\0\0\0\x0c\0\0\0\xb8\x01\0\0\0\0\0\0\0\0\0\0\
344+\x0c\0\0\0\xc8\x01\0\0\0\0\0\0\0\0\0\0\x0c\0\0\0\xd8\x01\0\0\0\0\0\0\0\0\0\0\
345+\x0c\0\0\0\xe8\x01\0\0\0\0\0\0\0\0\0\0\x0c\0\0\0\xf8\x01\0\0\0\0\0\0\0\0\0\0\
346+\x0c\0\0\0\x08\x02\0\0\0\0\0\0\0\0\0\0\x0c\0\0\0\x18\x02\0\0\0\0\0\0\0\0\0\0\
347+\x0c\0\0\0\x28\x02\0\0\0\0\0\0\0\0\0\0\x0c\0\0\0\x38\x02\0\0\0\0\0\0\0\0\0\0\
348+\x0c\0\0\0\x48\x02\0\0\0\0\0\0\0\0\0\0\x0c\0\0\0\x58\x02\0\0\0\0\0\0\0\0\0\0\
349+\x0c\0\0\0\x68\x02\0\0\0\0\0\0\0\0\0\0\x0c\0\0\0\x78\x02\0\0\0\0\0\0\0\0\0\0\
350+\x0c\0\0\0\x94\x02\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\xa4\x02\0\0\0\0\0\0\0\0\0\0\
351+\x0b\0\0\0\xb4\x02\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\xc4\x02\0\0\0\0\0\0\0\0\0\0\
352+\x0b\0\0\0\xd4\x02\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\xe4\x02\0\0\0\0\0\0\0\0\0\0\
353+\x0b\0\0\0\xf4\x02\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\x0c\x03\0\0\0\0\0\0\0\0\0\0\
354+\x0c\0\0\0\x1c\x03\0\0\0\0\0\0\0\0\0\0\x0c\0\0\0\x2c\x03\0\0\0\0\0\0\0\0\0\0\
355+\x0c\0\0\0\x3c\x03\0\0\0\0\0\0\0\0\0\0\x0c\0\0\0\x4c\x03\0\0\0\0\0\0\0\0\0\0\
356+\x0c\0\0\0\x5c\x03\0\0\0\0\0\0\0\0\0\0\x0c\0\0\0\x6c\x03\0\0\0\0\0\0\0\0\0\0\
357+\x0c\0\0\0\x7c\x03\0\0\0\0\0\0\0\0\0\0\x0c\0\0\0\x8c\x03\0\0\0\0\0\0\0\0\0\0\
358+\x0c\0\0\0\x9c\x03\0\0\0\0\0\0\0\0\0\0\x0c\0\0\0\xac\x03\0\0\0\0\0\0\0\0\0\0\
359+\x0c\0\0\0\xbc\x03\0\0\0\0\0\0\0\0\0\0\x0c\0\0\0\xcc\x03\0\0\0\0\0\0\0\0\0\0\
360+\x0c\0\0\0\xdc\x03\0\0\0\0\0\0\0\0\0\0\x0c\0\0\0\xec\x03\0\0\0\0\0\0\0\0\0\0\
361+\x0c\0\0\0\xfc\x03\0\0\0\0\0\0\0\0\0\0\x0c\0\0\0\x0c\x04\0\0\0\0\0\0\0\0\0\0\
362+\x0c\0\0\0\x1c\x04\0\0\0\0\0\0\0\0\0\0\x0c\0\0\0\x4d\x4e\x40\x41\x42\x43\x4c\0\
363+\x2e\x74\x65\x78\x74\0\x2e\x72\x65\x6c\x2e\x42\x54\x46\x2e\x65\x78\x74\0\x64\
364+\x75\x6d\x70\x5f\x62\x70\x66\x5f\x6d\x61\x70\x2e\x5f\x5f\x5f\x66\x6d\x74\0\x64\
365+\x75\x6d\x70\x5f\x62\x70\x66\x5f\x70\x72\x6f\x67\x2e\x5f\x5f\x5f\x66\x6d\x74\0\
366+\x64\x75\x6d\x70\x5f\x62\x70\x66\x5f\x6d\x61\x70\0\x2e\x72\x65\x6c\x69\x74\x65\
367+\x72\x2f\x62\x70\x66\x5f\x6d\x61\x70\0\x64\x75\x6d\x70\x5f\x62\x70\x66\x5f\x70\
368+\x72\x6f\x67\0\x2e\x72\x65\x6c\x69\x74\x65\x72\x2f\x62\x70\x66\x5f\x70\x72\x6f\
369+\x67\0\x2e\x6c\x6c\x76\x6d\x5f\x61\x64\x64\x72\x73\x69\x67\0\x6c\x69\x63\x65\
370+\x6e\x73\x65\0\x69\x74\x65\x72\x61\x74\x6f\x72\x73\x2e\x62\x70\x66\x2e\x63\0\
371+\x2e\x73\x74\x72\x74\x61\x62\0\x2e\x73\x79\x6d\x74\x61\x62\0\x2e\x72\x6f\x64\
372+\x61\x74\x61\0\x2e\x72\x65\x6c\x2e\x42\x54\x46\0\x4c\x49\x43\x45\x4e\x53\x45\0\
373+\x4c\x42\x42\x31\x5f\x37\0\x4c\x42\x42\x31\x5f\x36\0\x4c\x42\x42\x30\x5f\x34\0\
374+\x4c\x42\x42\x31\x5f\x33\0\x4c\x42\x42\x30\x5f\x33\0\x64\x75\x6d\x70\x5f\x62\
375+\x70\x66\x5f\x70\x72\x6f\x67\x2e\x5f\x5f\x5f\x66\x6d\x74\x2e\x32\0\x64\x75\x6d\
376+\x70\x5f\x62\x70\x66\x5f\x6d\x61\x70\x2e\x5f\x5f\x5f\x66\x6d\x74\x2e\x31\0\0\0\
377+\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\
378+\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x01\0\0\0\x01\0\0\
379+\0\x06\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x40\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\
380+\0\0\0\0\x04\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x4e\0\0\0\x01\0\0\0\x06\0\0\0\0\0\0\
381+\0\0\0\0\0\0\0\0\0\x40\0\0\0\0\0\0\0\0\x01\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x08\0\0\
382+\0\0\0\0\0\0\0\0\0\0\0\0\0\x6d\0\0\0\x01\0\0\0\x06\0\0\0\0\0\0\0\0\0\0\0\0\0\0\
383+\0\x40\x01\0\0\0\0\0\0\x08\x02\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x08\0\0\0\0\0\0\0\0\
384+\0\0\0\0\0\0\0\xb1\0\0\0\x01\0\0\0\x02\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x48\x03\0\
385+\0\0\0\0\0\x62\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x01\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\
386+\x89\0\0\0\x01\0\0\0\x03\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\xaa\x03\0\0\0\0\0\0\x04\
387+\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x01\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\xbd\0\0\0\x01\
388+\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\xae\x03\0\0\0\0\0\0\x3d\x09\0\0\0\0\0\0\
389+\0\0\0\0\0\0\0\0\x01\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\x01\0\0\0\0\0\0\0\
390+\0\0\0\0\0\0\0\0\0\0\0\0\xeb\x0c\0\0\0\0\0\0\x2c\x04\0\0\0\0\0\0\0\0\0\0\0\0\0\
391+\0\x01\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\xa9\0\0\0\x02\0\0\0\0\0\0\0\0\0\0\0\0\0\0\
392+\0\0\0\0\0\x18\x11\0\0\0\0\0\0\x98\x01\0\0\0\0\0\0\x0e\0\0\0\x0e\0\0\0\x08\0\0\
393+\0\0\0\0\0\x18\0\0\0\0\0\0\0\x4a\0\0\0\x09\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\
394+\0\xb0\x12\0\0\0\0\0\0\x20\0\0\0\0\0\0\0\x08\0\0\0\x02\0\0\0\x08\0\0\0\0\0\0\0\
395+\x10\0\0\0\0\0\0\0\x69\0\0\0\x09\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\xd0\x12\
396+\0\0\0\0\0\0\x20\0\0\0\0\0\0\0\x08\0\0\0\x03\0\0\0\x08\0\0\0\0\0\0\0\x10\0\0\0\
397+\0\0\0\0\xb9\0\0\0\x09\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\xf0\x12\0\0\0\0\0\
398+\0\x50\0\0\0\0\0\0\0\x08\0\0\0\x06\0\0\0\x08\0\0\0\0\0\0\0\x10\0\0\0\0\0\0\0\
399+\x07\0\0\0\x09\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x40\x13\0\0\0\0\0\0\xe0\
400+\x03\0\0\0\0\0\0\x08\0\0\0\x07\0\0\0\x08\0\0\0\0\0\0\0\x10\0\0\0\0\0\0\0\x7b\0\
401+\0\0\x03\x4c\xff\x6f\0\0\0\x80\0\0\0\0\0\0\0\0\0\0\0\0\x20\x17\0\0\0\0\0\0\x07\
402+\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x01\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\xa1\0\0\0\x03\
403+\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x27\x17\0\0\0\0\0\0\x1a\x01\0\0\0\0\0\0\
404+\0\0\0\0\0\0\0\0\x01\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0";
405+
406+ return 0;
407+err:
408+ bpf_object__destroy_skeleton(s);
409+ return -1;
410+}
411+
412+#endif /* __ITERATORS_BPF_SKEL_H__ */
kernel/bpf/queue_stack_maps.c+2−0
@@ -257,6 +257,7 @@ static int queue_stack_map_get_next_key(struct bpf_map *map, void *key, · 256 unmodified lines
257257
258258 static int queue_map_btf_id;
259259 const struct bpf_map_ops queue_map_ops = {
260+ .map_meta_equal = bpf_map_meta_equal,
260261 .map_alloc_check = queue_stack_map_alloc_check,
261262 .map_alloc = queue_stack_map_alloc,
262263 .map_free = queue_stack_map_free,
@@ -273,6 +274,7 @@ const struct bpf_map_ops queue_map_ops = { · 10 unmodified lines
273274
274275 static int stack_map_btf_id;
275276 const struct bpf_map_ops stack_map_ops = {
277+ .map_meta_equal = bpf_map_meta_equal,
276278 .map_alloc_check = queue_stack_map_alloc_check,
277279 .map_alloc = queue_stack_map_alloc,
278280 .map_free = queue_stack_map_free,
kernel/bpf/reuseport_array.c+2−1
@@ -191,7 +191,7 @@ int bpf_fd_reuseport_array_lookup_elem(struct bpf_map *map, void *key, · 190 unmodified lines
191191 rcu_read_lock();
192192 sk = reuseport_array_lookup_elem(map, key);
193193 if (sk) {
194− *(u64 *)value = sock_gen_cookie(sk);
194+ *(u64 *)value = __sock_gen_cookie(sk);
195195 err = 0;
196196 } else {
197197 err = -ENOENT;
@@ -351,6 +351,7 @@ static int reuseport_array_get_next_key(struct bpf_map *map, void *key, · 153 unmodified lines
351351
352352 static int reuseport_array_map_btf_id;
353353 const struct bpf_map_ops reuseport_array_ops = {
354+ .map_meta_equal = bpf_map_meta_equal,
354355 .map_alloc_check = reuseport_array_alloc_check,
355356 .map_alloc = reuseport_array_alloc,
356357 .map_free = reuseport_array_free,
kernel/bpf/ringbuf.c+1−0
@@ -287,6 +287,7 @@ static __poll_t ringbuf_map_poll(struct bpf_map *map, struct file *filp, · 286 unmodified lines
287287
288288 static int ringbuf_map_btf_id;
289289 const struct bpf_map_ops ringbuf_map_ops = {
290+ .map_meta_equal = bpf_map_meta_equal,
290291 .map_alloc = ringbuf_map_alloc,
291292 .map_free = ringbuf_map_free,
292293 .map_mmap = ringbuf_map_mmap,
kernel/bpf/stackmap.c+3−3
@@ -665,18 +665,17 @@ BPF_CALL_4(bpf_get_task_stack, struct task_struct *, task, void *, buf, · 664 unmodified lines
665665 return __bpf_get_stack(regs, task, NULL, buf, size, flags);
666666 }
667667
668−BTF_ID_LIST(bpf_get_task_stack_btf_ids)
669−BTF_ID(struct, task_struct)
668+BTF_ID_LIST_SINGLE(bpf_get_task_stack_btf_ids, struct, task_struct)
670669
671670 const struct bpf_func_proto bpf_get_task_stack_proto = {
672671 .func = bpf_get_task_stack,
673672 .gpl_only = false,
674673 .ret_type = RET_INTEGER,
675674 .arg1_type = ARG_PTR_TO_BTF_ID,
675+ .arg1_btf_id = &bpf_get_task_stack_btf_ids[0],
676676 .arg2_type = ARG_PTR_TO_UNINIT_MEM,
677677 .arg3_type = ARG_CONST_SIZE_OR_ZERO,
678678 .arg4_type = ARG_ANYTHING,
679− .btf_id = bpf_get_task_stack_btf_ids,
680679 };
681680
682681 BPF_CALL_4(bpf_get_stack_pe, struct bpf_perf_event_data_kern *, ctx,
@@ -839,6 +838,7 @@ static void stack_map_free(struct bpf_map *map) · 156 unmodified lines
839838
840839 static int stack_trace_map_btf_id;
841840 const struct bpf_map_ops stack_trace_map_ops = {
841+ .map_meta_equal = bpf_map_meta_equal,
842842 .map_alloc = stack_map_alloc,
843843 .map_free = stack_map_free,
844844 .map_get_next_key = stack_map_get_next_key,
kernel/bpf/syscall.c+278−53
@@ -4,6 +4,7 @@ · 3 unmodified lines
44 #include <linux/bpf.h>
55 #include <linux/bpf_trace.h>
66 #include <linux/bpf_lirc.h>
7+#include <linux/bpf_verifier.h>
78 #include <linux/btf.h>
89 #include <linux/syscalls.h>
910 #include <linux/slab.h>
@@ -29,6 +30,7 @@ · 19 unmodified lines
2930 #include <linux/bpf_lsm.h>
3031 #include <linux/poll.h>
3132 #include <linux/bpf-netns.h>
33+#include <linux/rcupdate_trace.h>
3234
3335 #define IS_FD_ARRAY(map) ((map)->map_type == BPF_MAP_TYPE_PERF_EVENT_ARRAY || \
3436 (map)->map_type == BPF_MAP_TYPE_CGROUP_ARRAY || \
@@ -90,6 +92,7 @@ int bpf_check_uarg_tail_zero(void __user *uaddr, · 55 unmodified lines
9092 }
9193
9294 const struct bpf_map_ops bpf_map_offload_ops = {
95+ .map_meta_equal = bpf_map_meta_equal,
9396 .map_alloc = bpf_map_offload_map_alloc,
9497 .map_free = bpf_map_offload_map_free,
9598 .map_check_btf = map_check_no_btf,
@@ -157,10 +160,11 @@ static int bpf_map_update_value(struct bpf_map *map, struct fd f, void *key, · 61 unmodified lines
157160 if (bpf_map_is_dev_bound(map)) {
158161 return bpf_map_offload_update_elem(map, key, value, flags);
159162 } else if (map->map_type == BPF_MAP_TYPE_CPUMAP ||
160− map->map_type == BPF_MAP_TYPE_SOCKHASH ||
161− map->map_type == BPF_MAP_TYPE_SOCKMAP ||
162163 map->map_type == BPF_MAP_TYPE_STRUCT_OPS) {
163164 return map->ops->map_update_elem(map, key, value, flags);
165+ } else if (map->map_type == BPF_MAP_TYPE_SOCKHASH ||
166+ map->map_type == BPF_MAP_TYPE_SOCKMAP) {
167+ return sock_map_update_elem_sys(map, key, value, flags);
164168 } else if (IS_FD_PROG_ARRAY(map)) {
165169 return bpf_fd_array_map_update_elem(map, f.file, key, value,
166170 flags);
@@ -768,7 +772,8 @@ static int map_check_btf(struct bpf_map *map, const struct btf *btf, · 601 unmodified lines
768772 if (map->map_type != BPF_MAP_TYPE_HASH &&
769773 map->map_type != BPF_MAP_TYPE_ARRAY &&
770774 map->map_type != BPF_MAP_TYPE_CGROUP_STORAGE &&
771− map->map_type != BPF_MAP_TYPE_SK_STORAGE)
775+ map->map_type != BPF_MAP_TYPE_SK_STORAGE &&
776+ map->map_type != BPF_MAP_TYPE_INODE_STORAGE)
772777 return -ENOTSUPP;
773778 if (map->spin_lock_off + sizeof(struct bpf_spin_lock) >
774779 map->value_size) {
@@ -1728,10 +1733,14 @@ static void __bpf_prog_put_noref(struct bpf_prog *prog, bool deferred) · 953 unmodified lines
17281733 btf_put(prog->aux->btf);
17291734 bpf_prog_free_linfo(prog);
17301735
1731− if (deferred)
1732− call_rcu(&prog->aux->rcu, __bpf_prog_put_rcu);
1733− else
1736+ if (deferred) {
1737+ if (prog->aux->sleepable)
1738+ call_rcu_tasks_trace(&prog->aux->rcu, __bpf_prog_put_rcu);
1739+ else
1740+ call_rcu(&prog->aux->rcu, __bpf_prog_put_rcu);
1741+ } else {
17341742 __bpf_prog_put_rcu(&prog->aux->rcu);
1743+ }
17351744 }
17361745
17371746 static void __bpf_prog_put(struct bpf_prog *prog, bool do_idr_lock)
@@ -2101,6 +2110,7 @@ static int bpf_prog_load(union bpf_attr *attr, union bpf_attr __user *uattr) · 363 unmodified lines
21012110 if (attr->prog_flags & ~(BPF_F_STRICT_ALIGNMENT |
21022111 BPF_F_ANY_ALIGNMENT |
21032112 BPF_F_TEST_STATE_FREQ |
2113+ BPF_F_SLEEPABLE |
21042114 BPF_F_TEST_RND_HI32))
21052115 return -EINVAL;
21062116
@@ -2145,17 +2155,18 @@ static int bpf_prog_load(union bpf_attr *attr, union bpf_attr __user *uattr) · 38 unmodified lines
21452155 prog->expected_attach_type = attr->expected_attach_type;
21462156 prog->aux->attach_btf_id = attr->attach_btf_id;
21472157 if (attr->attach_prog_fd) {
2148− struct bpf_prog *tgt_prog;
2158+ struct bpf_prog *dst_prog;
21492159
2150− tgt_prog = bpf_prog_get(attr->attach_prog_fd);
2151− if (IS_ERR(tgt_prog)) {
2152− err = PTR_ERR(tgt_prog);
2160+ dst_prog = bpf_prog_get(attr->attach_prog_fd);
2161+ if (IS_ERR(dst_prog)) {
2162+ err = PTR_ERR(dst_prog);
21532163 goto free_prog_nouncharge;
21542164 }
2155− prog->aux->linked_prog = tgt_prog;
2165+ prog->aux->dst_prog = dst_prog;
21562166 }
21572167
21582168 prog->aux->offload_requested = !!attr->prog_ifindex;
2169+ prog->aux->sleepable = attr->prog_flags & BPF_F_SLEEPABLE;
21592170
21602171 err = security_bpf_prog_alloc(prog->aux);
21612172 if (err)
@@ -2488,11 +2499,23 @@ struct bpf_link *bpf_link_get_from_fd(u32 ufd) · 326 unmodified lines
24882499 struct bpf_tracing_link {
24892500 struct bpf_link link;
24902501 enum bpf_attach_type attach_type;
2502+ struct bpf_trampoline *trampoline;
2503+ struct bpf_prog *tgt_prog;
24912504 };
24922505
24932506 static void bpf_tracing_link_release(struct bpf_link *link)
24942507 {
2495− WARN_ON_ONCE(bpf_trampoline_unlink_prog(link->prog));
2508+ struct bpf_tracing_link *tr_link =
2509+ container_of(link, struct bpf_tracing_link, link);
2510+
2511+ WARN_ON_ONCE(bpf_trampoline_unlink_prog(link->prog,
2512+ tr_link->trampoline));
2513+
2514+ bpf_trampoline_put(tr_link->trampoline);
2515+
2516+ /* tgt_prog is NULL if target is a kernel function */
2517+ if (tr_link->tgt_prog)
2518+ bpf_prog_put(tr_link->tgt_prog);
24962519 }
24972520
24982521 static void bpf_tracing_link_dealloc(struct bpf_link *link)
@@ -2532,10 +2555,15 @@ static const struct bpf_link_ops bpf_tracing_link_lops = { · 33 unmodified lines
25322555 .fill_link_info = bpf_tracing_link_fill_link_info,
25332556 };
25342557
2535−static int bpf_tracing_prog_attach(struct bpf_prog *prog)
2558+static int bpf_tracing_prog_attach(struct bpf_prog *prog,
2559+ int tgt_prog_fd,
2560+ u32 btf_id)
25362561 {
25372562 struct bpf_link_primer link_primer;
2563+ struct bpf_prog *tgt_prog = NULL;
2564+ struct bpf_trampoline *tr = NULL;
25382565 struct bpf_tracing_link *link;
2566+ u64 key = 0;
25392567 int err;
25402568
25412569 switch (prog->type) {
@@ -2564,6 +2592,28 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog) · 22 unmodified lines
25642592 goto out_put_prog;
25652593 }
25662594
2595+ if (!!tgt_prog_fd != !!btf_id) {
2596+ err = -EINVAL;
2597+ goto out_put_prog;
2598+ }
2599+
2600+ if (tgt_prog_fd) {
2601+ /* For now we only allow new targets for BPF_PROG_TYPE_EXT */
2602+ if (prog->type != BPF_PROG_TYPE_EXT) {
2603+ err = -EINVAL;
2604+ goto out_put_prog;
2605+ }
2606+
2607+ tgt_prog = bpf_prog_get(tgt_prog_fd);
2608+ if (IS_ERR(tgt_prog)) {
2609+ err = PTR_ERR(tgt_prog);
2610+ tgt_prog = NULL;
2611+ goto out_put_prog;
2612+ }
2613+
2614+ key = bpf_trampoline_compute_key(tgt_prog, btf_id);
2615+ }
2616+
25672617 link = kzalloc(sizeof(*link), GFP_USER);
25682618 if (!link) {
25692619 err = -ENOMEM;
@@ -2573,20 +2623,100 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog) · 3 unmodified lines
25732623 &bpf_tracing_link_lops, prog);
25742624 link->attach_type = prog->expected_attach_type;
25752625
2576− err = bpf_link_prime(&link->link, &link_primer);
2577− if (err) {
2578− kfree(link);
2579− goto out_put_prog;
2626+ mutex_lock(&prog->aux->dst_mutex);
2627+
2628+ /* There are a few possible cases here:
2629+ *
2630+ * - if prog->aux->dst_trampoline is set, the program was just loaded
2631+ * and not yet attached to anything, so we can use the values stored
2632+ * in prog->aux
2633+ *
2634+ * - if prog->aux->dst_trampoline is NULL, the program has already been
2635+ * attached to a target and its initial target was cleared (below)
2636+ *
2637+ * - if tgt_prog != NULL, the caller specified tgt_prog_fd +
2638+ * target_btf_id using the link_create API.
2639+ *
2640+ * - if tgt_prog == NULL when this function was called using the old
2641+ * raw_tracepoint_open API, and we need a target from prog->aux
2642+ *
2643+ * The combination of no saved target in prog->aux, and no target
2644+ * specified on load is illegal, and we reject that here.
2645+ */
2646+ if (!prog->aux->dst_trampoline && !tgt_prog) {
2647+ err = -ENOENT;
2648+ goto out_unlock;
2649+ }
2650+
2651+ if (!prog->aux->dst_trampoline ||
2652+ (key && key != prog->aux->dst_trampoline->key)) {
2653+ /* If there is no saved target, or the specified target is
2654+ * different from the destination specified at load time, we
2655+ * need a new trampoline and a check for compatibility
2656+ */
2657+ struct bpf_attach_target_info tgt_info = {};
2658+
2659+ err = bpf_check_attach_target(NULL, prog, tgt_prog, btf_id,
2660+ &tgt_info);
2661+ if (err)
2662+ goto out_unlock;
2663+
2664+ tr = bpf_trampoline_get(key, &tgt_info);
2665+ if (!tr) {
2666+ err = -ENOMEM;
2667+ goto out_unlock;
2668+ }
2669+ } else {
2670+ /* The caller didn't specify a target, or the target was the
2671+ * same as the destination supplied during program load. This
2672+ * means we can reuse the trampoline and reference from program
2673+ * load time, and there is no need to allocate a new one. This
2674+ * can only happen once for any program, as the saved values in
2675+ * prog->aux are cleared below.
2676+ */
2677+ tr = prog->aux->dst_trampoline;
2678+ tgt_prog = prog->aux->dst_prog;
25802679 }
25812680
2582− err = bpf_trampoline_link_prog(prog);
2681+ err = bpf_link_prime(&link->link, &link_primer);
2682+ if (err)
2683+ goto out_unlock;
2684+
2685+ err = bpf_trampoline_link_prog(prog, tr);
25832686 if (err) {
25842687 bpf_link_cleanup(&link_primer);
2585− goto out_put_prog;
2688+ link = NULL;
2689+ goto out_unlock;
25862690 }
25872691
2692+ link->tgt_prog = tgt_prog;
2693+ link->trampoline = tr;
2694+
2695+ /* Always clear the trampoline and target prog from prog->aux to make
2696+ * sure the original attach destination is not kept alive after a
2697+ * program is (re-)attached to another target.
2698+ */
2699+ if (prog->aux->dst_prog &&
2700+ (tgt_prog_fd || tr != prog->aux->dst_trampoline))
2701+ /* got extra prog ref from syscall, or attaching to different prog */
2702+ bpf_prog_put(prog->aux->dst_prog);
2703+ if (prog->aux->dst_trampoline && tr != prog->aux->dst_trampoline)
2704+ /* we allocated a new trampoline, so free the old one */
2705+ bpf_trampoline_put(prog->aux->dst_trampoline);
2706+
2707+ prog->aux->dst_prog = NULL;
2708+ prog->aux->dst_trampoline = NULL;
2709+ mutex_unlock(&prog->aux->dst_mutex);
2710+
25882711 return bpf_link_settle(&link_primer);
2712+out_unlock:
2713+ if (tr && tr != prog->aux->dst_trampoline)
2714+ bpf_trampoline_put(tr);
2715+ mutex_unlock(&prog->aux->dst_mutex);
2716+ kfree(link);
25892717 out_put_prog:
2718+ if (tgt_prog_fd && tgt_prog)
2719+ bpf_prog_put(tgt_prog);
25902720 bpf_prog_put(prog);
25912721 return err;
25922722 }
@@ -2700,7 +2830,7 @@ static int bpf_raw_tracepoint_open(const union bpf_attr *attr) · 107 unmodified lines
27002830 tp_name = prog->aux->attach_func_name;
27012831 break;
27022832 }
2703− return bpf_tracing_prog_attach(prog);
2833+ return bpf_tracing_prog_attach(prog, 0, 0);
27042834 case BPF_PROG_TYPE_RAW_TRACEPOINT:
27052835 case BPF_PROG_TYPE_RAW_TRACEPOINT_WRITABLE:
27062836 if (strncpy_from_user(buf,
@@ -2969,7 +3099,7 @@ static int bpf_prog_query(const union bpf_attr *attr, · 262 unmodified lines
29693099 }
29703100 }
29713101
2972−#define BPF_PROG_TEST_RUN_LAST_FIELD test.ctx_out
3102+#define BPF_PROG_TEST_RUN_LAST_FIELD test.cpu
29733103
29743104 static int bpf_prog_test_run(const union bpf_attr *attr,
29753105 union bpf_attr __user *uattr)
@@ -3152,21 +3282,25 @@ static const struct bpf_map *bpf_map_from_imm(const struct bpf_prog *prog, · 176 unmodified lines
31523282 const struct bpf_map *map;
31533283 int i;
31543284
3285+ mutex_lock(&prog->aux->used_maps_mutex);
31553286 for (i = 0, *off = 0; i < prog->aux->used_map_cnt; i++) {
31563287 map = prog->aux->used_maps[i];
31573288 if (map == (void *)addr) {
31583289 *type = BPF_PSEUDO_MAP_FD;
3159− return map;
3290+ goto out;
31603291 }
31613292 if (!map->ops->map_direct_value_meta)
31623293 continue;
31633294 if (!map->ops->map_direct_value_meta(map, addr, off)) {
31643295 *type = BPF_PSEUDO_MAP_VALUE;
3165− return map;
3296+ goto out;
31663297 }
31673298 }
3299+ map = NULL;
31683300
3169− return NULL;
3301+out:
3302+ mutex_unlock(&prog->aux->used_maps_mutex);
3303+ return map;
31703304 }
31713305
31723306 static struct bpf_insn *bpf_insn_prepare_dump(const struct bpf_prog *prog,
@@ -3284,6 +3418,7 @@ static int bpf_prog_get_info_by_fd(struct file *file, · 111 unmodified lines
32843418 memcpy(info.tag, prog->tag, sizeof(prog->tag));
32853419 memcpy(info.name, prog->aux->name, sizeof(prog->aux->name));
32863420
3421+ mutex_lock(&prog->aux->used_maps_mutex);
32873422 ulen = info.nr_map_ids;
32883423 info.nr_map_ids = prog->aux->used_map_cnt;
32893424 ulen = min_t(u32, info.nr_map_ids, ulen);
@@ -3293,9 +3428,12 @@ static int bpf_prog_get_info_by_fd(struct file *file, · 3 unmodified lines
32933428
32943429 for (i = 0; i < ulen; i++)
32953430 if (put_user(prog->aux->used_maps[i]->id,
3296− &user_map_ids[i]))
3431+ &user_map_ids[i])) {
3432+ mutex_unlock(&prog->aux->used_maps_mutex);
32973433 return -EFAULT;
3434+ }
32983435 }
3436+ mutex_unlock(&prog->aux->used_maps_mutex);
32993437
33003438 err = set_info_rec_size(&info);
33013439 if (err)
@@ -3876,10 +4014,15 @@ err_put: · 574 unmodified lines
38764014
38774015 static int tracing_bpf_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)
38784016 {
3879− if (attr->link_create.attach_type == BPF_TRACE_ITER &&
3880− prog->expected_attach_type == BPF_TRACE_ITER)
3881− return bpf_iter_link_attach(attr, prog);
4017+ if (attr->link_create.attach_type != prog->expected_attach_type)
4018+ return -EINVAL;
38824019
4020+ if (prog->expected_attach_type == BPF_TRACE_ITER)
4021+ return bpf_iter_link_attach(attr, prog);
4022+ else if (prog->type == BPF_PROG_TYPE_EXT)
4023+ return bpf_tracing_prog_attach(prog,
4024+ attr->link_create.target_fd,
4025+ attr->link_create.target_btf_id);
38834026 return -EINVAL;
38844027 }
38854028
@@ -3893,18 +4036,25 @@ static int link_create(union bpf_attr *attr) · 7 unmodified lines
38934036 if (CHECK_ATTR(BPF_LINK_CREATE))
38944037 return -EINVAL;
38954038
3896− ptype = attach_type_to_prog_type(attr->link_create.attach_type);
3897− if (ptype == BPF_PROG_TYPE_UNSPEC)
3898− return -EINVAL;
3899−
3900− prog = bpf_prog_get_type(attr->link_create.prog_fd, ptype);
4039+ prog = bpf_prog_get(attr->link_create.prog_fd);
39014040 if (IS_ERR(prog))
39024041 return PTR_ERR(prog);
39034042
39044043 ret = bpf_prog_attach_check_attach_type(prog,
39054044 attr->link_create.attach_type);
39064045 if (ret)
3907− goto err_out;
4046+ goto out;
4047+
4048+ if (prog->type == BPF_PROG_TYPE_EXT) {
4049+ ret = tracing_bpf_link_attach(attr, prog);
4050+ goto out;
4051+ }
4052+
4053+ ptype = attach_type_to_prog_type(attr->link_create.attach_type);
4054+ if (ptype == BPF_PROG_TYPE_UNSPEC || ptype != prog->type) {
4055+ ret = -EINVAL;
4056+ goto out;
4057+ }
39084058
39094059 switch (ptype) {
39104060 case BPF_PROG_TYPE_CGROUP_SKB:
@@ -3932,7 +4082,7 @@ static int link_create(union bpf_attr *attr) · 21 unmodified lines
39324082 ret = -EINVAL;
39334083 }
39344084
3935−err_out:
4085+out:
39364086 if (ret < 0)
39374087 bpf_prog_put(prog);
39384088 return ret;
@@ -4014,40 +4164,50 @@ static int link_detach(union bpf_attr *attr) · 75 unmodified lines
40144164 return ret;
40154165 }
40164166
4017−static int bpf_link_inc_not_zero(struct bpf_link *link)
4167+static struct bpf_link *bpf_link_inc_not_zero(struct bpf_link *link)
40184168 {
4019− return atomic64_fetch_add_unless(&link->refcnt, 1, 0) ? 0 : -ENOENT;
4169+ return atomic64_fetch_add_unless(&link->refcnt, 1, 0) ? link : ERR_PTR(-ENOENT);
40204170 }
40214171
4022−#define BPF_LINK_GET_FD_BY_ID_LAST_FIELD link_id
4023−
4024−static int bpf_link_get_fd_by_id(const union bpf_attr *attr)
4172+struct bpf_link *bpf_link_by_id(u32 id)
40254173 {
40264174 struct bpf_link *link;
4027− u32 id = attr->link_id;
4028− int fd, err;
40294175
4030− if (CHECK_ATTR(BPF_LINK_GET_FD_BY_ID))
4031− return -EINVAL;
4032−
4033− if (!capable(CAP_SYS_ADMIN))
4034− return -EPERM;
4176+ if (!id)
4177+ return ERR_PTR(-ENOENT);
40354178
40364179 spin_lock_bh(&link_idr_lock);
4037− link = idr_find(&link_idr, id);
40384180 /* before link is "settled", ID is 0, pretend it doesn't exist yet */
4181+ link = idr_find(&link_idr, id);
40394182 if (link) {
40404183 if (link->id)
4041− err = bpf_link_inc_not_zero(link);
4184+ link = bpf_link_inc_not_zero(link);
40424185 else
4043− err = -EAGAIN;
4186+ link = ERR_PTR(-EAGAIN);
40444187 } else {
4045− err = -ENOENT;
4188+ link = ERR_PTR(-ENOENT);
40464189 }
40474190 spin_unlock_bh(&link_idr_lock);
4191+ return link;
4192+}
40484193
4049− if (err)
4050− return err;
4194+#define BPF_LINK_GET_FD_BY_ID_LAST_FIELD link_id
4195+
4196+static int bpf_link_get_fd_by_id(const union bpf_attr *attr)
4197+{
4198+ struct bpf_link *link;
4199+ u32 id = attr->link_id;
4200+ int fd;
4201+
4202+ if (CHECK_ATTR(BPF_LINK_GET_FD_BY_ID))
4203+ return -EINVAL;
4204+
4205+ if (!capable(CAP_SYS_ADMIN))
4206+ return -EPERM;
4207+
4208+ link = bpf_link_by_id(id);
4209+ if (IS_ERR(link))
4210+ return PTR_ERR(link);
40514211
40524212 fd = bpf_link_new_fd(link);
40534213 if (fd < 0)
@@ -4133,6 +4293,68 @@ static int bpf_iter_create(union bpf_attr *attr) · 79 unmodified lines
41334293 return err;
41344294 }
41354295
4296+#define BPF_PROG_BIND_MAP_LAST_FIELD prog_bind_map.flags
4297+
4298+static int bpf_prog_bind_map(union bpf_attr *attr)
4299+{
4300+ struct bpf_prog *prog;
4301+ struct bpf_map *map;
4302+ struct bpf_map **used_maps_old, **used_maps_new;
4303+ int i, ret = 0;
4304+
4305+ if (CHECK_ATTR(BPF_PROG_BIND_MAP))
4306+ return -EINVAL;
4307+
4308+ if (attr->prog_bind_map.flags)
4309+ return -EINVAL;
4310+
4311+ prog = bpf_prog_get(attr->prog_bind_map.prog_fd);
4312+ if (IS_ERR(prog))
4313+ return PTR_ERR(prog);
4314+
4315+ map = bpf_map_get(attr->prog_bind_map.map_fd);
4316+ if (IS_ERR(map)) {
4317+ ret = PTR_ERR(map);
4318+ goto out_prog_put;
4319+ }
4320+
4321+ mutex_lock(&prog->aux->used_maps_mutex);
4322+
4323+ used_maps_old = prog->aux->used_maps;
4324+
4325+ for (i = 0; i < prog->aux->used_map_cnt; i++)
4326+ if (used_maps_old[i] == map) {
4327+ bpf_map_put(map);
4328+ goto out_unlock;
4329+ }
4330+
4331+ used_maps_new = kmalloc_array(prog->aux->used_map_cnt + 1,
4332+ sizeof(used_maps_new[0]),
4333+ GFP_KERNEL);
4334+ if (!used_maps_new) {
4335+ ret = -ENOMEM;
4336+ goto out_unlock;
4337+ }
4338+
4339+ memcpy(used_maps_new, used_maps_old,
4340+ sizeof(used_maps_old[0]) * prog->aux->used_map_cnt);
4341+ used_maps_new[prog->aux->used_map_cnt] = map;
4342+
4343+ prog->aux->used_map_cnt++;
4344+ prog->aux->used_maps = used_maps_new;
4345+
4346+ kfree(used_maps_old);
4347+
4348+out_unlock:
4349+ mutex_unlock(&prog->aux->used_maps_mutex);
4350+
4351+ if (ret)
4352+ bpf_map_put(map);
4353+out_prog_put:
4354+ bpf_prog_put(prog);
4355+ return ret;
4356+}
4357+
41364358 SYSCALL_DEFINE3(bpf, int, cmd, union bpf_attr __user *, uattr, unsigned int, size)
41374359 {
41384360 union bpf_attr attr;
@@ -4266,6 +4488,9 @@ SYSCALL_DEFINE3(bpf, int, cmd, union bpf_attr __user *, uattr, unsigned int, siz · 127 unmodified lines
42664488 case BPF_LINK_DETACH:
42674489 err = link_detach(&attr);
42684490 break;
4491+ case BPF_PROG_BIND_MAP:
4492+ err = bpf_prog_bind_map(&attr);
4493+ break;
42694494 default:
42704495 err = -EINVAL;
42714496 break;
kernel/bpf/task_iter.c+11−4
@@ -22,7 +22,8 @@ struct bpf_iter_seq_task_info { · 21 unmodified lines
2222 };
2323
2424 static struct task_struct *task_seq_get_next(struct pid_namespace *ns,
25− u32 *tid)
25+ u32 *tid,
26+ bool skip_if_dup_files)
2627 {
2728 struct task_struct *task = NULL;
2829 struct pid *pid;
@@ -36,6 +37,12 @@ retry: · 7 unmodified lines
3637 if (!task) {
3738 ++*tid;
3839 goto retry;
40+ } else if (skip_if_dup_files && task->tgid != task->pid &&
41+ task->files == task->group_leader->files) {
42+ put_task_struct(task);
43+ task = NULL;
44+ ++*tid;
45+ goto retry;
3946 }
4047 }
4148 rcu_read_unlock();
@@ -48,7 +55,7 @@ static void *task_seq_start(struct seq_file *seq, loff_t *pos) · 6 unmodified lines
4855 struct bpf_iter_seq_task_info *info = seq->private;
4956 struct task_struct *task;
5057
51− task = task_seq_get_next(info->common.ns, &info->tid);
58+ task = task_seq_get_next(info->common.ns, &info->tid, false);
5259 if (!task)
5360 return NULL;
5461
@@ -65,7 +72,7 @@ static void *task_seq_next(struct seq_file *seq, void *v, loff_t *pos) · 10 unmodified lines
6572 ++*pos;
6673 ++info->tid;
6774 put_task_struct((struct task_struct *)v);
68− task = task_seq_get_next(info->common.ns, &info->tid);
75+ task = task_seq_get_next(info->common.ns, &info->tid, false);
6976 if (!task)
7077 return NULL;
7178
@@ -148,7 +155,7 @@ again: · 76 unmodified lines
148155 curr_files = *fstruct;
149156 curr_fd = info->fd;
150157 } else {
151− curr_task = task_seq_get_next(ns, &curr_tid);
158+ curr_task = task_seq_get_next(ns, &curr_tid, true);
152159 if (!curr_task)
153160 return NULL;
154161
kernel/bpf/trampoline.c+51−12
@@ -7,6 +7,8 @@ · 6 unmodified lines
77 #include <linux/rbtree_latch.h>
88 #include <linux/perf_event.h>
99 #include <linux/btf.h>
10+#include <linux/rcupdate_trace.h>
11+#include <linux/rcupdate_wait.h>
1012
1113 /* dummy _ops. The verifier will operate on target program's ops. */
1214 const struct bpf_verifier_ops bpf_extension_verifier_ops = {
@@ -63,7 +65,7 @@ static void bpf_trampoline_ksym_add(struct bpf_trampoline *tr) · 50 unmodified lines
6365 bpf_image_ksym_add(tr->image, ksym);
6466 }
6567
66−struct bpf_trampoline *bpf_trampoline_lookup(u64 key)
68+static struct bpf_trampoline *bpf_trampoline_lookup(u64 key)
6769 {
6870 struct bpf_trampoline *tr;
6971 struct hlist_head *head;
@@ -210,9 +212,12 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr) · 140 unmodified lines
210212 * updates to trampoline would change the code from underneath the
211213 * preempted task. Hence wait for tasks to voluntarily schedule or go
212214 * to userspace.
215+ * The same trampoline can hold both sleepable and non-sleepable progs.
216+ * synchronize_rcu_tasks_trace() is needed to make sure all sleepable
217+ * programs finish executing.
218+ * Wait for these two grace periods together.
213219 */
214−
215− synchronize_rcu_tasks();
220+ synchronize_rcu_mult(call_rcu_tasks, call_rcu_tasks_trace);
216221
217222 err = arch_prepare_bpf_trampoline(new_image, new_image + PAGE_SIZE / 2,
218223 &tr->func.model, flags, tprogs,
@@ -256,14 +261,12 @@ static enum bpf_tramp_prog_type bpf_attach_type_to_tramp(struct bpf_prog *prog) · 37 unmodified lines
256261 }
257262 }
258263
259−int bpf_trampoline_link_prog(struct bpf_prog *prog)
264+int bpf_trampoline_link_prog(struct bpf_prog *prog, struct bpf_trampoline *tr)
260265 {
261266 enum bpf_tramp_prog_type kind;
262− struct bpf_trampoline *tr;
263267 int err = 0;
264268 int cnt;
265269
266− tr = prog->aux->trampoline;
267270 kind = bpf_attach_type_to_tramp(prog);
268271 mutex_lock(&tr->mutex);
269272 if (tr->extension_prog) {
@@ -296,7 +299,7 @@ int bpf_trampoline_link_prog(struct bpf_prog *prog) · 26 unmodified lines
296299 }
297300 hlist_add_head(&prog->aux->tramp_hlist, &tr->progs_hlist[kind]);
298301 tr->progs_cnt[kind]++;
299− err = bpf_trampoline_update(prog->aux->trampoline);
302+ err = bpf_trampoline_update(tr);
300303 if (err) {
301304 hlist_del(&prog->aux->tramp_hlist);
302305 tr->progs_cnt[kind]--;
@@ -307,13 +310,11 @@ out: · 4 unmodified lines
307310 }
308311
309312 /* bpf_trampoline_unlink_prog() should never fail. */
310−int bpf_trampoline_unlink_prog(struct bpf_prog *prog)
313+int bpf_trampoline_unlink_prog(struct bpf_prog *prog, struct bpf_trampoline *tr)
311314 {
312315 enum bpf_tramp_prog_type kind;
313− struct bpf_trampoline *tr;
314316 int err;
315317
316− tr = prog->aux->trampoline;
317318 kind = bpf_attach_type_to_tramp(prog);
318319 mutex_lock(&tr->mutex);
319320 if (kind == BPF_TRAMP_REPLACE) {
@@ -325,12 +326,32 @@ int bpf_trampoline_unlink_prog(struct bpf_prog *prog) · 5 unmodified lines
325326 }
326327 hlist_del(&prog->aux->tramp_hlist);
327328 tr->progs_cnt[kind]--;
328− err = bpf_trampoline_update(prog->aux->trampoline);
329+ err = bpf_trampoline_update(tr);
329330 out:
330331 mutex_unlock(&tr->mutex);
331332 return err;
332333 }
333334
335+struct bpf_trampoline *bpf_trampoline_get(u64 key,
336+ struct bpf_attach_target_info *tgt_info)
337+{
338+ struct bpf_trampoline *tr;
339+
340+ tr = bpf_trampoline_lookup(key);
341+ if (!tr)
342+ return NULL;
343+
344+ mutex_lock(&tr->mutex);
345+ if (tr->func.addr)
346+ goto out;
347+
348+ memcpy(&tr->func.model, &tgt_info->fmodel, sizeof(tgt_info->fmodel));
349+ tr->func.addr = (void *)tgt_info->tgt_addr;
350+out:
351+ mutex_unlock(&tr->mutex);
352+ return tr;
353+}
354+
334355 void bpf_trampoline_put(struct bpf_trampoline *tr)
335356 {
336357 if (!tr)
@@ -344,7 +365,14 @@ void bpf_trampoline_put(struct bpf_trampoline *tr) · 7 unmodified lines
344365 if (WARN_ON_ONCE(!hlist_empty(&tr->progs_hlist[BPF_TRAMP_FEXIT])))
345366 goto out;
346367 bpf_image_ksym_del(&tr->ksym);
347− /* wait for tasks to get out of trampoline before freeing it */
368+ /* This code will be executed when all bpf progs (both sleepable and
369+ * non-sleepable) went through
370+ * bpf_prog_put()->call_rcu[_tasks_trace]()->bpf_prog_free_deferred().
371+ * Hence no need for another synchronize_rcu_tasks_trace() here,
372+ * but synchronize_rcu_tasks() is still needed, since trampoline
373+ * may not have had any sleepable programs and we need to wait
374+ * for tasks to get out of trampoline code before freeing it.
375+ */
348376 synchronize_rcu_tasks();
349377 bpf_jit_free_exec(tr->image);
350378 hlist_del(&tr->hlist);
@@ -394,6 +422,17 @@ void notrace __bpf_prog_exit(struct bpf_prog *prog, u64 start) · 43 unmodified lines
394422 rcu_read_unlock();
395423 }
396424
425+void notrace __bpf_prog_enter_sleepable(void)
426+{
427+ rcu_read_lock_trace();
428+ might_fault();
429+}
430+
431+void notrace __bpf_prog_exit_sleepable(void)
432+{
433+ rcu_read_unlock_trace();
434+}
435+
397436 int __weak
398437 arch_prepare_bpf_trampoline(void *image, void *image_end,
399438 const struct btf_func_model *m, u32 flags,
kernel/bpf/verifier.c+1060−328
@@ -21,6 +21,7 @@ · 20 unmodified lines
2121 #include <linux/ctype.h>
2222 #include <linux/error-injection.h>
2323 #include <linux/bpf_lsm.h>
24+#include <linux/btf_ids.h>
2425
2526 #include "disasm.h"
2627
@@ -238,6 +239,7 @@ struct bpf_call_arg_meta { · 211 unmodified lines
238239 int ref_obj_id;
239240 int func_id;
240241 u32 btf_id;
242+ u32 ret_btf_id;
241243 };
242244
243245 struct btf *btf_vmlinux;
@@ -435,6 +437,15 @@ static bool arg_type_may_be_refcounted(enum bpf_arg_type type) · 191 unmodified lines
435437 return type == ARG_PTR_TO_SOCK_COMMON;
436438 }
437439
440+static bool arg_type_may_be_null(enum bpf_arg_type type)
441+{
442+ return type == ARG_PTR_TO_MAP_VALUE_OR_NULL ||
443+ type == ARG_PTR_TO_MEM_OR_NULL ||
444+ type == ARG_PTR_TO_CTX_OR_NULL ||
445+ type == ARG_PTR_TO_SOCKET_OR_NULL ||
446+ type == ARG_PTR_TO_ALLOC_MEM_OR_NULL;
447+}
448+
438449 /* Determine whether the function releases some resources allocated by another
439450 * function call. The first reference type argument will be assumed to be
440451 * released by release_reference().
@@ -477,7 +488,12 @@ static bool is_acquire_function(enum bpf_func_id func_id, · 36 unmodified lines
477488 static bool is_ptr_cast_function(enum bpf_func_id func_id)
478489 {
479490 return func_id == BPF_FUNC_tcp_sock ||
480− func_id == BPF_FUNC_sk_fullsock;
491+ func_id == BPF_FUNC_sk_fullsock ||
492+ func_id == BPF_FUNC_skc_to_tcp_sock ||
493+ func_id == BPF_FUNC_skc_to_tcp6_sock ||
494+ func_id == BPF_FUNC_skc_to_udp6_sock ||
495+ func_id == BPF_FUNC_skc_to_tcp_timewait_sock ||
496+ func_id == BPF_FUNC_skc_to_tcp_request_sock;
481497 }
482498
483499 /* string representation of 'enum bpf_reg_type' */
@@ -503,6 +519,7 @@ static const char * const reg_type_str[] = { · 19 unmodified lines
503519 [PTR_TO_XDP_SOCK] = "xdp_sock",
504520 [PTR_TO_BTF_ID] = "ptr_",
505521 [PTR_TO_BTF_ID_OR_NULL] = "ptr_or_null_",
522+ [PTR_TO_PERCPU_BTF_ID] = "percpu_ptr_",
506523 [PTR_TO_MEM] = "mem",
507524 [PTR_TO_MEM_OR_NULL] = "mem_or_null",
508525 [PTR_TO_RDONLY_BUF] = "rdonly_buf",
@@ -569,7 +586,9 @@ static void print_verifier_state(struct bpf_verifier_env *env, · 60 unmodified lines
569586 /* reg->off should be 0 for SCALAR_VALUE */
570587 verbose(env, "%lld", reg->var_off.value + reg->off);
571588 } else {
572− if (t == PTR_TO_BTF_ID || t == PTR_TO_BTF_ID_OR_NULL)
589+ if (t == PTR_TO_BTF_ID ||
590+ t == PTR_TO_BTF_ID_OR_NULL ||
591+ t == PTR_TO_PERCPU_BTF_ID)
573592 verbose(env, "%s", kernel_type_name(reg->btf_id));
574593 verbose(env, "(id=%d", reg->id);
575594 if (reg_type_may_be_refcounted_or_null(t))
@@ -991,14 +1010,9 @@ static const int caller_saved[CALLER_SAVED_REGS] = { · 415 unmodified lines
9911010 static void __mark_reg_not_init(const struct bpf_verifier_env *env,
9921011 struct bpf_reg_state *reg);
9931012
994−/* Mark the unknown part of a register (variable offset or scalar value) as
995− * known to have the value @imm.
996− */
997−static void __mark_reg_known(struct bpf_reg_state *reg, u64 imm)
1013+/* This helper doesn't clear reg->id */
1014+static void ___mark_reg_known(struct bpf_reg_state *reg, u64 imm)
9981015 {
999− /* Clear id, off, and union(map_ptr, range) */
1000− memset(((u8 *)reg) + sizeof(reg->type), 0,
1001− offsetof(struct bpf_reg_state, var_off) - sizeof(reg->type));
10021016 reg->var_off = tnum_const(imm);
10031017 reg->smin_value = (s64)imm;
10041018 reg->smax_value = (s64)imm;
@@ -1011,6 +1025,17 @@ static void __mark_reg_known(struct bpf_reg_state *reg, u64 imm) · 6 unmodified lines
10111025 reg->u32_max_value = (u32)imm;
10121026 }
10131027
1028+/* Mark the unknown part of a register (variable offset or scalar value) as
1029+ * known to have the value @imm.
1030+ */
1031+static void __mark_reg_known(struct bpf_reg_state *reg, u64 imm)
1032+{
1033+ /* Clear id, off, and union(map_ptr, range) */
1034+ memset(((u8 *)reg) + sizeof(reg->type), 0,
1035+ offsetof(struct bpf_reg_state, var_off) - sizeof(reg->type));
1036+ ___mark_reg_known(reg, imm);
1037+}
1038+
10141039 static void __mark_reg32_known(struct bpf_reg_state *reg, u64 imm)
10151040 {
10161041 reg->var_off = tnum_const_subreg(reg->var_off, imm);
@@ -1489,6 +1514,13 @@ static int check_subprogs(struct bpf_verifier_env *env) · 472 unmodified lines
14891514 for (i = 0; i < insn_cnt; i++) {
14901515 u8 code = insn[i].code;
14911516
1517+ if (code == (BPF_JMP | BPF_CALL) &&
1518+ insn[i].imm == BPF_FUNC_tail_call &&
1519+ insn[i].src_reg != BPF_PSEUDO_CALL)
1520+ subprog[cur_subprog].has_tail_call = true;
1521+ if (BPF_CLASS(code) == BPF_LD &&
1522+ (BPF_MODE(code) == BPF_ABS || BPF_MODE(code) == BPF_IND))
1523+ subprog[cur_subprog].has_ld_abs = true;
14921524 if (BPF_CLASS(code) != BPF_JMP && BPF_CLASS(code) != BPF_JMP32)
14931525 goto next;
14941526 if (BPF_OP(code) == BPF_EXIT || BPF_OP(code) == BPF_CALL)
@@ -2183,6 +2215,7 @@ static bool is_spillable_regtype(enum bpf_reg_type type) · 688 unmodified lines
21832215 case PTR_TO_RDONLY_BUF_OR_NULL:
21842216 case PTR_TO_RDWR_BUF:
21852217 case PTR_TO_RDWR_BUF_OR_NULL:
2218+ case PTR_TO_PERCPU_BTF_ID:
21862219 return true;
21872220 default:
21882221 return false;
@@ -2200,6 +2233,20 @@ static bool register_is_const(struct bpf_reg_state *reg) · 11 unmodified lines
22002233 return reg->type == SCALAR_VALUE && tnum_is_const(reg->var_off);
22012234 }
22022235
2236+static bool __is_scalar_unbounded(struct bpf_reg_state *reg)
2237+{
2238+ return tnum_is_unknown(reg->var_off) &&
2239+ reg->smin_value == S64_MIN && reg->smax_value == S64_MAX &&
2240+ reg->umin_value == 0 && reg->umax_value == U64_MAX &&
2241+ reg->s32_min_value == S32_MIN && reg->s32_max_value == S32_MAX &&
2242+ reg->u32_min_value == 0 && reg->u32_max_value == U32_MAX;
2243+}
2244+
2245+static bool register_is_bounded(struct bpf_reg_state *reg)
2246+{
2247+ return reg->type == SCALAR_VALUE && !__is_scalar_unbounded(reg);
2248+}
2249+
22032250 static bool __is_pointer_value(bool allow_ptr_leaks,
22042251 const struct bpf_reg_state *reg)
22052252 {
@@ -2251,7 +2298,7 @@ static int check_stack_write(struct bpf_verifier_env *env, · 45 unmodified lines
22512298 if (value_regno >= 0)
22522299 reg = &cur->regs[value_regno];
22532300
2254− if (reg && size == BPF_REG_SIZE && register_is_const(reg) &&
2301+ if (reg && size == BPF_REG_SIZE && register_is_bounded(reg) &&
22552302 !register_is_null(reg) && env->bpf_capable) {
22562303 if (dst_reg != BPF_REG_FP) {
22572304 /* The backtracking logic can only recognize explicit
@@ -2625,11 +2672,18 @@ static int check_map_access(struct bpf_verifier_env *env, u32 regno, · 367 unmodified lines
26252672
26262673 #define MAX_PACKET_OFF 0xffff
26272674
2675+static enum bpf_prog_type resolve_prog_type(struct bpf_prog *prog)
2676+{
2677+ return prog->aux->dst_prog ? prog->aux->dst_prog->type : prog->type;
2678+}
2679+
26282680 static bool may_access_direct_pkt_data(struct bpf_verifier_env *env,
26292681 const struct bpf_call_arg_meta *meta,
26302682 enum bpf_access_type t)
26312683 {
2632− switch (env->prog->type) {
2684+ enum bpf_prog_type prog_type = resolve_prog_type(env->prog);
2685+
2686+ switch (prog_type) {
26332687 /* Program types only with direct read access go here! */
26342688 case BPF_PROG_TYPE_LWT_IN:
26352689 case BPF_PROG_TYPE_LWT_OUT:
@@ -2639,7 +2693,7 @@ static bool may_access_direct_pkt_data(struct bpf_verifier_env *env, · 3 unmodified lines
26392693 case BPF_PROG_TYPE_CGROUP_SKB:
26402694 if (t == BPF_WRITE)
26412695 return false;
2642− /* fallthrough */
2696+ fallthrough;
26432697
26442698 /* Program types with direct read + write access go here! */
26452699 case BPF_PROG_TYPE_SCHED_CLS:
@@ -2970,10 +3024,37 @@ static int check_max_stack_depth(struct bpf_verifier_env *env) · 324 unmodified lines
29703024 int depth = 0, frame = 0, idx = 0, i = 0, subprog_end;
29713025 struct bpf_subprog_info *subprog = env->subprog_info;
29723026 struct bpf_insn *insn = env->prog->insnsi;
3027+ bool tail_call_reachable = false;
29733028 int ret_insn[MAX_CALL_FRAMES];
29743029 int ret_prog[MAX_CALL_FRAMES];
3030+ int j;
29753031
29763032 process_func:
3033+ /* protect against potential stack overflow that might happen when
3034+ * bpf2bpf calls get combined with tailcalls. Limit the caller's stack
3035+ * depth for such case down to 256 so that the worst case scenario
3036+ * would result in 8k stack size (32 which is tailcall limit * 256 =
3037+ * 8k).
3038+ *
3039+ * To get the idea what might happen, see an example:
3040+ * func1 -> sub rsp, 128
3041+ * subfunc1 -> sub rsp, 256
3042+ * tailcall1 -> add rsp, 256
3043+ * func2 -> sub rsp, 192 (total stack size = 128 + 192 = 320)
3044+ * subfunc2 -> sub rsp, 64
3045+ * subfunc22 -> sub rsp, 128
3046+ * tailcall2 -> add rsp, 128
3047+ * func3 -> sub rsp, 32 (total stack size 128 + 192 + 64 + 32 = 416)
3048+ *
3049+ * tailcall will unwind the current stack frame but it will not get rid
3050+ * of caller's stack as shown on the example above.
3051+ */
3052+ if (idx && subprog[idx].has_tail_call && depth >= 256) {
3053+ verbose(env,
3054+ "tail_calls are not allowed when call stack of previous frames is %d bytes. Too large\n",
3055+ depth);
3056+ return -EACCES;
3057+ }
29773058 /* round up to 32-bytes, since this is granularity
29783059 * of interpreter stack size
29793060 */
@@ -3002,6 +3083,10 @@ continue_func: · 22 unmodified lines
30023083 i);
30033084 return -EFAULT;
30043085 }
3086+
3087+ if (subprog[idx].has_tail_call)
3088+ tail_call_reachable = true;
3089+
30053090 frame++;
30063091 if (frame >= MAX_CALL_FRAMES) {
30073092 verbose(env, "the call stack of %d frames is too deep !\n",
@@ -3010,6 +3095,15 @@ continue_func: · 2 unmodified lines
30103095 }
30113096 goto process_func;
30123097 }
3098+ /* if tail call got detected across bpf2bpf calls then mark each of the
3099+ * currently present subprog frames as tail call reachable subprogs;
3100+ * this info will be utilized by JIT so that we will be preserving the
3101+ * tail call counter throughout bpf2bpf calls combined with tailcalls
3102+ */
3103+ if (tail_call_reachable)
3104+ for (j = 0; j < frame; j++)
3105+ subprog[ret_prog[j]].tail_call_reachable = true;
3106+
30133107 /* end of for() loop means the last insn of the 'subprog'
30143108 * was reached. Doesn't matter whether it was JA or EXIT
30153109 */
@@ -3585,18 +3679,6 @@ static int check_stack_boundary(struct bpf_verifier_env *env, int regno, · 569 unmodified lines
35853679 struct bpf_func_state *state = func(env, reg);
35863680 int err, min_off, max_off, i, j, slot, spi;
35873681
3588− if (reg->type != PTR_TO_STACK) {
3589− /* Allow zero-byte read from NULL, regardless of pointer type */
3590− if (zero_size_allowed && access_size == 0 &&
3591− register_is_null(reg))
3592− return 0;
3593−
3594− verbose(env, "R%d type=%s expected=%s\n", regno,
3595− reg_type_str[reg->type],
3596− reg_type_str[PTR_TO_STACK]);
3597− return -EACCES;
3598− }
3599−
36003682 if (tnum_is_const(reg->var_off)) {
36013683 min_off = max_off = reg->var_off.value + reg->off;
36023684 err = __check_stack_boundary(env, regno, min_off, access_size,
@@ -3741,9 +3823,19 @@ static int check_helper_mem_access(struct bpf_verifier_env *env, int regno, · 138 unmodified lines
37413823 access_size, zero_size_allowed,
37423824 "rdwr",
37433825 &env->prog->aux->max_rdwr_access);
3744− default: /* scalar_value|ptr_to_stack or invalid ptr */
3826+ case PTR_TO_STACK:
37453827 return check_stack_boundary(env, regno, access_size,
37463828 zero_size_allowed, meta);
3829+ default: /* scalar_value or invalid ptr */
3830+ /* Allow zero-byte read from NULL, regardless of pointer type */
3831+ if (zero_size_allowed && access_size == 0 &&
3832+ register_is_null(reg))
3833+ return 0;
3834+
3835+ verbose(env, "R%d type=%s expected=%s\n", regno,
3836+ reg_type_str[reg->type],
3837+ reg_type_str[PTR_TO_STACK]);
3838+ return -EACCES;
37473839 }
37483840 }
37493841
@@ -3775,10 +3867,6 @@ static int process_spin_lock(struct bpf_verifier_env *env, int regno, · 25 unmodified lines
37753867 struct bpf_map *map = reg->map_ptr;
37763868 u64 val = reg->var_off.value;
37773869
3778− if (reg->type != PTR_TO_MAP_VALUE) {
3779− verbose(env, "R%d is not a pointer to map_value\n", regno);
3780− return -EINVAL;
3781− }
37823870 if (!is_const) {
37833871 verbose(env,
37843872 "R%d doesn't have constant offset. bpf_spin_lock has to be at the constant offset\n",
@@ -3845,12 +3933,6 @@ static bool arg_type_is_mem_size(enum bpf_arg_type type) · 60 unmodified lines
38453933 type == ARG_CONST_SIZE_OR_ZERO;
38463934 }
38473935
3848−static bool arg_type_is_alloc_mem_ptr(enum bpf_arg_type type)
3849−{
3850− return type == ARG_PTR_TO_ALLOC_MEM ||
3851− type == ARG_PTR_TO_ALLOC_MEM_OR_NULL;
3852−}
3853−
38543936 static bool arg_type_is_alloc_size(enum bpf_arg_type type)
38553937 {
38563938 return type == ARG_CONST_ALLOC_SIZE_OR_ZERO;
@@ -3872,14 +3954,194 @@ static int int_ptr_type_to_size(enum bpf_arg_type type) · 15 unmodified lines
38723954 return -EINVAL;
38733955 }
38743956
3957+static int resolve_map_arg_type(struct bpf_verifier_env *env,
3958+ const struct bpf_call_arg_meta *meta,
3959+ enum bpf_arg_type *arg_type)
3960+{
3961+ if (!meta->map_ptr) {
3962+ /* kernel subsystem misconfigured verifier */
3963+ verbose(env, "invalid map_ptr to access map->type\n");
3964+ return -EACCES;
3965+ }
3966+
3967+ switch (meta->map_ptr->map_type) {
3968+ case BPF_MAP_TYPE_SOCKMAP:
3969+ case BPF_MAP_TYPE_SOCKHASH:
3970+ if (*arg_type == ARG_PTR_TO_MAP_VALUE) {
3971+ *arg_type = ARG_PTR_TO_BTF_ID_SOCK_COMMON;
3972+ } else {
3973+ verbose(env, "invalid arg_type for sockmap/sockhash\n");
3974+ return -EINVAL;
3975+ }
3976+ break;
3977+
3978+ default:
3979+ break;
3980+ }
3981+ return 0;
3982+}
3983+
3984+struct bpf_reg_types {
3985+ const enum bpf_reg_type types[10];
3986+ u32 *btf_id;
3987+};
3988+
3989+static const struct bpf_reg_types map_key_value_types = {
3990+ .types = {
3991+ PTR_TO_STACK,
3992+ PTR_TO_PACKET,
3993+ PTR_TO_PACKET_META,
3994+ PTR_TO_MAP_VALUE,
3995+ },
3996+};
3997+
3998+static const struct bpf_reg_types sock_types = {
3999+ .types = {
4000+ PTR_TO_SOCK_COMMON,
4001+ PTR_TO_SOCKET,
4002+ PTR_TO_TCP_SOCK,
4003+ PTR_TO_XDP_SOCK,
4004+ },
4005+};
4006+
4007+#ifdef CONFIG_NET
4008+static const struct bpf_reg_types btf_id_sock_common_types = {
4009+ .types = {
4010+ PTR_TO_SOCK_COMMON,
4011+ PTR_TO_SOCKET,
4012+ PTR_TO_TCP_SOCK,
4013+ PTR_TO_XDP_SOCK,
4014+ PTR_TO_BTF_ID,
4015+ },
4016+ .btf_id = &btf_sock_ids[BTF_SOCK_TYPE_SOCK_COMMON],
4017+};
4018+#endif
4019+
4020+static const struct bpf_reg_types mem_types = {
4021+ .types = {
4022+ PTR_TO_STACK,
4023+ PTR_TO_PACKET,
4024+ PTR_TO_PACKET_META,
4025+ PTR_TO_MAP_VALUE,
4026+ PTR_TO_MEM,
4027+ PTR_TO_RDONLY_BUF,
4028+ PTR_TO_RDWR_BUF,
4029+ },
4030+};
4031+
4032+static const struct bpf_reg_types int_ptr_types = {
4033+ .types = {
4034+ PTR_TO_STACK,
4035+ PTR_TO_PACKET,
4036+ PTR_TO_PACKET_META,
4037+ PTR_TO_MAP_VALUE,
4038+ },
4039+};
4040+
4041+static const struct bpf_reg_types fullsock_types = { .types = { PTR_TO_SOCKET } };
4042+static const struct bpf_reg_types scalar_types = { .types = { SCALAR_VALUE } };
4043+static const struct bpf_reg_types context_types = { .types = { PTR_TO_CTX } };
4044+static const struct bpf_reg_types alloc_mem_types = { .types = { PTR_TO_MEM } };
4045+static const struct bpf_reg_types const_map_ptr_types = { .types = { CONST_PTR_TO_MAP } };
4046+static const struct bpf_reg_types btf_ptr_types = { .types = { PTR_TO_BTF_ID } };
4047+static const struct bpf_reg_types spin_lock_types = { .types = { PTR_TO_MAP_VALUE } };
4048+static const struct bpf_reg_types percpu_btf_ptr_types = { .types = { PTR_TO_PERCPU_BTF_ID } };
4049+
4050+static const struct bpf_reg_types *compatible_reg_types[__BPF_ARG_TYPE_MAX] = {
4051+ [ARG_PTR_TO_MAP_KEY] = &map_key_value_types,
4052+ [ARG_PTR_TO_MAP_VALUE] = &map_key_value_types,
4053+ [ARG_PTR_TO_UNINIT_MAP_VALUE] = &map_key_value_types,
4054+ [ARG_PTR_TO_MAP_VALUE_OR_NULL] = &map_key_value_types,
4055+ [ARG_CONST_SIZE] = &scalar_types,
4056+ [ARG_CONST_SIZE_OR_ZERO] = &scalar_types,
4057+ [ARG_CONST_ALLOC_SIZE_OR_ZERO] = &scalar_types,
4058+ [ARG_CONST_MAP_PTR] = &const_map_ptr_types,
4059+ [ARG_PTR_TO_CTX] = &context_types,
4060+ [ARG_PTR_TO_CTX_OR_NULL] = &context_types,
4061+ [ARG_PTR_TO_SOCK_COMMON] = &sock_types,
4062+#ifdef CONFIG_NET
4063+ [ARG_PTR_TO_BTF_ID_SOCK_COMMON] = &btf_id_sock_common_types,
4064+#endif
4065+ [ARG_PTR_TO_SOCKET] = &fullsock_types,
4066+ [ARG_PTR_TO_SOCKET_OR_NULL] = &fullsock_types,
4067+ [ARG_PTR_TO_BTF_ID] = &btf_ptr_types,
4068+ [ARG_PTR_TO_SPIN_LOCK] = &spin_lock_types,
4069+ [ARG_PTR_TO_MEM] = &mem_types,
4070+ [ARG_PTR_TO_MEM_OR_NULL] = &mem_types,
4071+ [ARG_PTR_TO_UNINIT_MEM] = &mem_types,
4072+ [ARG_PTR_TO_ALLOC_MEM] = &alloc_mem_types,
4073+ [ARG_PTR_TO_ALLOC_MEM_OR_NULL] = &alloc_mem_types,
4074+ [ARG_PTR_TO_INT] = &int_ptr_types,
4075+ [ARG_PTR_TO_LONG] = &int_ptr_types,
4076+ [ARG_PTR_TO_PERCPU_BTF_ID] = &percpu_btf_ptr_types,
4077+};
4078+
4079+static int check_reg_type(struct bpf_verifier_env *env, u32 regno,
4080+ enum bpf_arg_type arg_type,
4081+ const u32 *arg_btf_id)
4082+{
4083+ struct bpf_reg_state *regs = cur_regs(env), *reg = &regs[regno];
4084+ enum bpf_reg_type expected, type = reg->type;
4085+ const struct bpf_reg_types *compatible;
4086+ int i, j;
4087+
4088+ compatible = compatible_reg_types[arg_type];
4089+ if (!compatible) {
4090+ verbose(env, "verifier internal error: unsupported arg type %d\n", arg_type);
4091+ return -EFAULT;
4092+ }
4093+
4094+ for (i = 0; i < ARRAY_SIZE(compatible->types); i++) {
4095+ expected = compatible->types[i];
4096+ if (expected == NOT_INIT)
4097+ break;
4098+
4099+ if (type == expected)
4100+ goto found;
4101+ }
4102+
4103+ verbose(env, "R%d type=%s expected=", regno, reg_type_str[type]);
4104+ for (j = 0; j + 1 < i; j++)
4105+ verbose(env, "%s, ", reg_type_str[compatible->types[j]]);
4106+ verbose(env, "%s\n", reg_type_str[compatible->types[j]]);
4107+ return -EACCES;
4108+
4109+found:
4110+ if (type == PTR_TO_BTF_ID) {
4111+ if (!arg_btf_id) {
4112+ if (!compatible->btf_id) {
4113+ verbose(env, "verifier internal error: missing arg compatible BTF ID\n");
4114+ return -EFAULT;
4115+ }
4116+ arg_btf_id = compatible->btf_id;
4117+ }
4118+
4119+ if (!btf_struct_ids_match(&env->log, reg->off, reg->btf_id,
4120+ *arg_btf_id)) {
4121+ verbose(env, "R%d is of type %s but %s is expected\n",
4122+ regno, kernel_type_name(reg->btf_id),
4123+ kernel_type_name(*arg_btf_id));
4124+ return -EACCES;
4125+ }
4126+
4127+ if (!tnum_is_const(reg->var_off) || reg->var_off.value) {
4128+ verbose(env, "R%d is a pointer to in-kernel struct with non-zero offset\n",
4129+ regno);
4130+ return -EACCES;
4131+ }
4132+ }
4133+
4134+ return 0;
4135+}
4136+
38754137 static int check_func_arg(struct bpf_verifier_env *env, u32 arg,
38764138 struct bpf_call_arg_meta *meta,
38774139 const struct bpf_func_proto *fn)
38784140 {
38794141 u32 regno = BPF_REG_1 + arg;
38804142 struct bpf_reg_state *regs = cur_regs(env), *reg = &regs[regno];
3881− enum bpf_reg_type expected_type, type = reg->type;
38824143 enum bpf_arg_type arg_type = fn->arg_type[arg];
4144+ enum bpf_reg_type type = reg->type;
38834145 int err = 0;
38844146
38854147 if (arg_type == ARG_DONTCARE)
@@ -3904,120 +4166,32 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, · 18 unmodified lines
39044166 return -EACCES;
39054167 }
39064168
3907− if (arg_type == ARG_PTR_TO_MAP_KEY ||
3908− arg_type == ARG_PTR_TO_MAP_VALUE ||
4169+ if (arg_type == ARG_PTR_TO_MAP_VALUE ||
39094170 arg_type == ARG_PTR_TO_UNINIT_MAP_VALUE ||
39104171 arg_type == ARG_PTR_TO_MAP_VALUE_OR_NULL) {
3911− expected_type = PTR_TO_STACK;
3912− if (register_is_null(reg) &&
3913− arg_type == ARG_PTR_TO_MAP_VALUE_OR_NULL)
3914− /* final test in check_stack_boundary() */;
3915− else if (!type_is_pkt_pointer(type) &&
3916− type != PTR_TO_MAP_VALUE &&
3917− type != expected_type)
3918− goto err_type;
3919− } else if (arg_type == ARG_CONST_SIZE ||
3920− arg_type == ARG_CONST_SIZE_OR_ZERO ||
3921− arg_type == ARG_CONST_ALLOC_SIZE_OR_ZERO) {
3922− expected_type = SCALAR_VALUE;
3923− if (type != expected_type)
3924− goto err_type;
3925− } else if (arg_type == ARG_CONST_MAP_PTR) {
3926− expected_type = CONST_PTR_TO_MAP;
3927− if (type != expected_type)
3928− goto err_type;
3929− } else if (arg_type == ARG_PTR_TO_CTX ||
3930− arg_type == ARG_PTR_TO_CTX_OR_NULL) {
3931− expected_type = PTR_TO_CTX;
3932− if (!(register_is_null(reg) &&
3933− arg_type == ARG_PTR_TO_CTX_OR_NULL)) {
3934− if (type != expected_type)
3935− goto err_type;
3936− err = check_ctx_reg(env, reg, regno);
3937− if (err < 0)
3938− return err;
3939− }
3940− } else if (arg_type == ARG_PTR_TO_SOCK_COMMON) {
3941− expected_type = PTR_TO_SOCK_COMMON;
3942− /* Any sk pointer can be ARG_PTR_TO_SOCK_COMMON */
3943− if (!type_is_sk_pointer(type))
3944− goto err_type;
3945− if (reg->ref_obj_id) {
3946− if (meta->ref_obj_id) {
3947− verbose(env, "verifier internal error: more than one arg with ref_obj_id R%d %u %u\n",
3948− regno, reg->ref_obj_id,
3949− meta->ref_obj_id);
3950− return -EFAULT;
3951− }
3952− meta->ref_obj_id = reg->ref_obj_id;
3953− }
3954− } else if (arg_type == ARG_PTR_TO_SOCKET ||
3955− arg_type == ARG_PTR_TO_SOCKET_OR_NULL) {
3956− expected_type = PTR_TO_SOCKET;
3957− if (!(register_is_null(reg) &&
3958− arg_type == ARG_PTR_TO_SOCKET_OR_NULL)) {
3959− if (type != expected_type)
3960− goto err_type;
3961− }
3962− } else if (arg_type == ARG_PTR_TO_BTF_ID) {
3963− expected_type = PTR_TO_BTF_ID;
3964− if (type != expected_type)
3965− goto err_type;
3966− if (!fn->check_btf_id) {
3967− if (reg->btf_id != meta->btf_id) {
3968− verbose(env, "Helper has type %s got %s in R%d\n",
3969− kernel_type_name(meta->btf_id),
3970− kernel_type_name(reg->btf_id), regno);
3971−
3972− return -EACCES;
3973− }
3974− } else if (!fn->check_btf_id(reg->btf_id, arg)) {
3975− verbose(env, "Helper does not support %s in R%d\n",
3976− kernel_type_name(reg->btf_id), regno);
4172+ err = resolve_map_arg_type(env, meta, &arg_type);
4173+ if (err)
4174+ return err;
4175+ }
39774176
3978− return -EACCES;
3979− }
3980− if (!tnum_is_const(reg->var_off) || reg->var_off.value || reg->off) {
3981− verbose(env, "R%d is a pointer to in-kernel struct with non-zero offset\n",
3982− regno);
3983− return -EACCES;
3984− }
3985− } else if (arg_type == ARG_PTR_TO_SPIN_LOCK) {
3986− if (meta->func_id == BPF_FUNC_spin_lock) {
3987− if (process_spin_lock(env, regno, true))
3988− return -EACCES;
3989− } else if (meta->func_id == BPF_FUNC_spin_unlock) {
3990− if (process_spin_lock(env, regno, false))
3991− return -EACCES;
3992− } else {
3993− verbose(env, "verifier internal error\n");
3994− return -EFAULT;
3995− }
3996− } else if (arg_type_is_mem_ptr(arg_type)) {
3997− expected_type = PTR_TO_STACK;
3998− /* One exception here. In case function allows for NULL to be
3999− * passed in as argument, it's a SCALAR_VALUE type. Final test
4000− * happens during stack boundary checking.
4177+ if (register_is_null(reg) && arg_type_may_be_null(arg_type))
4178+ /* A NULL register has a SCALAR_VALUE type, so skip
4179+ * type checking.
40014180 */
4002− if (register_is_null(reg) &&
4003− (arg_type == ARG_PTR_TO_MEM_OR_NULL ||
4004− arg_type == ARG_PTR_TO_ALLOC_MEM_OR_NULL))
4005− /* final test in check_stack_boundary() */;
4006− else if (!type_is_pkt_pointer(type) &&
4007− type != PTR_TO_MAP_VALUE &&
4008− type != PTR_TO_MEM &&
4009− type != PTR_TO_RDONLY_BUF &&
4010− type != PTR_TO_RDWR_BUF &&
4011− type != expected_type)
4012− goto err_type;
4013− meta->raw_mode = arg_type == ARG_PTR_TO_UNINIT_MEM;
4014− } else if (arg_type_is_alloc_mem_ptr(arg_type)) {
4015− expected_type = PTR_TO_MEM;
4016− if (register_is_null(reg) &&
4017− arg_type == ARG_PTR_TO_ALLOC_MEM_OR_NULL)
4018− /* final test in check_stack_boundary() */;
4019− else if (type != expected_type)
4020− goto err_type;
4181+ goto skip_type_check;
4182+
4183+ err = check_reg_type(env, regno, arg_type, fn->arg_btf_id[arg]);
4184+ if (err)
4185+ return err;
4186+
4187+ if (type == PTR_TO_CTX) {
4188+ err = check_ctx_reg(env, reg, regno);
4189+ if (err < 0)
4190+ return err;
4191+ }
4192+
4193+skip_type_check:
4194+ if (reg->ref_obj_id) {
40214195 if (meta->ref_obj_id) {
40224196 verbose(env, "verifier internal error: more than one arg with ref_obj_id R%d %u %u\n",
40234197 regno, reg->ref_obj_id,
@@ -4025,15 +4199,6 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, · 1 unmodified line
40254199 return -EFAULT;
40264200 }
40274201 meta->ref_obj_id = reg->ref_obj_id;
4028− } else if (arg_type_is_int_ptr(arg_type)) {
4029− expected_type = PTR_TO_STACK;
4030− if (!type_is_pkt_pointer(type) &&
4031− type != PTR_TO_MAP_VALUE &&
4032− type != expected_type)
4033− goto err_type;
4034− } else {
4035− verbose(env, "unsupported arg_type %d\n", arg_type);
4036− return -EFAULT;
40374202 }
40384203
40394204 if (arg_type == ARG_CONST_MAP_PTR) {
@@ -4072,6 +4237,28 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, · 32 unmodified lines
40724237 err = check_helper_mem_access(env, regno,
40734238 meta->map_ptr->value_size, false,
40744239 meta);
4240+ } else if (arg_type == ARG_PTR_TO_PERCPU_BTF_ID) {
4241+ if (!reg->btf_id) {
4242+ verbose(env, "Helper has invalid btf_id in R%d\n", regno);
4243+ return -EACCES;
4244+ }
4245+ meta->ret_btf_id = reg->btf_id;
4246+ } else if (arg_type == ARG_PTR_TO_SPIN_LOCK) {
4247+ if (meta->func_id == BPF_FUNC_spin_lock) {
4248+ if (process_spin_lock(env, regno, true))
4249+ return -EACCES;
4250+ } else if (meta->func_id == BPF_FUNC_spin_unlock) {
4251+ if (process_spin_lock(env, regno, false))
4252+ return -EACCES;
4253+ } else {
4254+ verbose(env, "verifier internal error\n");
4255+ return -EFAULT;
4256+ }
4257+ } else if (arg_type_is_mem_ptr(arg_type)) {
4258+ /* The access to this pointer is only checked when we hit the
4259+ * next is_mem_size argument below.
4260+ */
4261+ meta->raw_mode = (arg_type == ARG_PTR_TO_UNINIT_MEM);
40754262 } else if (arg_type_is_mem_size(arg_type)) {
40764263 bool zero_size_allowed = (arg_type == ARG_CONST_SIZE_OR_ZERO);
40774264
@@ -4137,10 +4324,43 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, · 59 unmodified lines
41374324 }
41384325
41394326 return err;
4140−err_type:
4141− verbose(env, "R%d type=%s expected=%s\n", regno,
4142− reg_type_str[type], reg_type_str[expected_type]);
4143− return -EACCES;
4327+}
4328+
4329+static bool may_update_sockmap(struct bpf_verifier_env *env, int func_id)
4330+{
4331+ enum bpf_attach_type eatype = env->prog->expected_attach_type;
4332+ enum bpf_prog_type type = resolve_prog_type(env->prog);
4333+
4334+ if (func_id != BPF_FUNC_map_update_elem)
4335+ return false;
4336+
4337+ /* It's not possible to get access to a locked struct sock in these
4338+ * contexts, so updating is safe.
4339+ */
4340+ switch (type) {
4341+ case BPF_PROG_TYPE_TRACING:
4342+ if (eatype == BPF_TRACE_ITER)
4343+ return true;
4344+ break;
4345+ case BPF_PROG_TYPE_SOCKET_FILTER:
4346+ case BPF_PROG_TYPE_SCHED_CLS:
4347+ case BPF_PROG_TYPE_SCHED_ACT:
4348+ case BPF_PROG_TYPE_XDP:
4349+ case BPF_PROG_TYPE_SK_REUSEPORT:
4350+ case BPF_PROG_TYPE_FLOW_DISSECTOR:
4351+ case BPF_PROG_TYPE_SK_LOOKUP:
4352+ return true;
4353+ default:
4354+ break;
4355+ }
4356+
4357+ verbose(env, "cannot update sockmap in this context\n");
4358+ return false;
4359+}
4360+
4361+static bool allow_tail_call_in_subprogs(struct bpf_verifier_env *env)
4362+{
4363+ return env->prog->jit_requested && IS_ENABLED(CONFIG_X86_64);
41444364 }
41454365
41464366 static int check_map_func_compatibility(struct bpf_verifier_env *env,
@@ -4214,7 +4434,8 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, · 67 unmodified lines
42144434 func_id != BPF_FUNC_map_delete_elem &&
42154435 func_id != BPF_FUNC_msg_redirect_map &&
42164436 func_id != BPF_FUNC_sk_select_reuseport &&
4217− func_id != BPF_FUNC_map_lookup_elem)
4437+ func_id != BPF_FUNC_map_lookup_elem &&
4438+ !may_update_sockmap(env, func_id))
42184439 goto error;
42194440 break;
42204441 case BPF_MAP_TYPE_SOCKHASH:
@@ -4223,7 +4444,8 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, · 2 unmodified lines
42234444 func_id != BPF_FUNC_map_delete_elem &&
42244445 func_id != BPF_FUNC_msg_redirect_hash &&
42254446 func_id != BPF_FUNC_sk_select_reuseport &&
4226− func_id != BPF_FUNC_map_lookup_elem)
4447+ func_id != BPF_FUNC_map_lookup_elem &&
4448+ !may_update_sockmap(env, func_id))
42274449 goto error;
42284450 break;
42294451 case BPF_MAP_TYPE_REUSEPORT_SOCKARRAY:
@@ -4242,6 +4464,11 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, · 12 unmodified lines
42424464 func_id != BPF_FUNC_sk_storage_delete)
42434465 goto error;
42444466 break;
4467+ case BPF_MAP_TYPE_INODE_STORAGE:
4468+ if (func_id != BPF_FUNC_inode_storage_get &&
4469+ func_id != BPF_FUNC_inode_storage_delete)
4470+ goto error;
4471+ break;
42454472 default:
42464473 break;
42474474 }
@@ -4251,8 +4478,8 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, · 3 unmodified lines
42514478 case BPF_FUNC_tail_call:
42524479 if (map->map_type != BPF_MAP_TYPE_PROG_ARRAY)
42534480 goto error;
4254− if (env->subprog_cnt > 1) {
4255− verbose(env, "tail_calls are not allowed in programs with bpf-to-bpf calls\n");
4481+ if (env->subprog_cnt > 1 && !allow_tail_call_in_subprogs(env)) {
4482+ verbose(env, "tail_calls are not allowed in non-JITed programs with bpf-to-bpf calls\n");
42564483 return -EINVAL;
42574484 }
42584485 break;
@@ -4315,6 +4542,11 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, · 56 unmodified lines
43154542 if (map->map_type != BPF_MAP_TYPE_SK_STORAGE)
43164543 goto error;
43174544 break;
4545+ case BPF_FUNC_inode_storage_get:
4546+ case BPF_FUNC_inode_storage_delete:
4547+ if (map->map_type != BPF_MAP_TYPE_INODE_STORAGE)
4548+ goto error;
4549+ break;
43184550 default:
43194551 break;
43204552 }
@@ -4402,10 +4634,26 @@ static bool check_refcount_ok(const struct bpf_func_proto *fn, int func_id) · 81 unmodified lines
44024634 return count <= 1;
44034635 }
44044636
4637+static bool check_btf_id_ok(const struct bpf_func_proto *fn)
4638+{
4639+ int i;
4640+
4641+ for (i = 0; i < ARRAY_SIZE(fn->arg_type); i++) {
4642+ if (fn->arg_type[i] == ARG_PTR_TO_BTF_ID && !fn->arg_btf_id[i])
4643+ return false;
4644+
4645+ if (fn->arg_type[i] != ARG_PTR_TO_BTF_ID && fn->arg_btf_id[i])
4646+ return false;
4647+ }
4648+
4649+ return true;
4650+}
4651+
44054652 static int check_func_proto(const struct bpf_func_proto *fn, int func_id)
44064653 {
44074654 return check_raw_mode_ok(fn) &&
44084655 check_arg_pair_ok(fn) &&
4656+ check_btf_id_ok(fn) &&
44094657 check_refcount_ok(fn, func_id) ? 0 : -EINVAL;
44104658 }
44114659
@@ -4775,6 +5023,11 @@ static int check_helper_call(struct bpf_verifier_env *env, int func_id, int insn · 363 unmodified lines
47755023 return -EINVAL;
47765024 }
47775025
5026+ if (fn->allowed && !fn->allowed(env->prog)) {
5027+ verbose(env, "helper call is not allowed in probe\n");
5028+ return -EINVAL;
5029+ }
5030+
47785031 /* With LD_ABS/IND some JITs save/restore skb from r1. */
47795032 changes_data = bpf_helper_changes_pkt_data(fn->func);
47805033 if (changes_data && fn->arg1_type != ARG_PTR_TO_CTX) {
@@ -4796,11 +5049,6 @@ static int check_helper_call(struct bpf_verifier_env *env, int func_id, int insn · 15 unmodified lines
47965049 meta.func_id = func_id;
47975050 /* check args */
47985051 for (i = 0; i < 5; i++) {
4799− if (!fn->check_btf_id) {
4800− err = btf_resolve_helper_id(&env->log, fn, i);
4801− if (err > 0)
4802− meta.btf_id = err;
4803− }
48045052 err = check_func_arg(env, i, &meta, fn);
48055053 if (err)
48065054 return err;
@@ -4904,6 +5152,35 @@ static int check_helper_call(struct bpf_verifier_env *env, int func_id, int insn · 97 unmodified lines
49045152 regs[BPF_REG_0].type = PTR_TO_MEM_OR_NULL;
49055153 regs[BPF_REG_0].id = ++env->id_gen;
49065154 regs[BPF_REG_0].mem_size = meta.mem_size;
5155+ } else if (fn->ret_type == RET_PTR_TO_MEM_OR_BTF_ID_OR_NULL ||
5156+ fn->ret_type == RET_PTR_TO_MEM_OR_BTF_ID) {
5157+ const struct btf_type *t;
5158+
5159+ mark_reg_known_zero(env, regs, BPF_REG_0);
5160+ t = btf_type_skip_modifiers(btf_vmlinux, meta.ret_btf_id, NULL);
5161+ if (!btf_type_is_struct(t)) {
5162+ u32 tsize;
5163+ const struct btf_type *ret;
5164+ const char *tname;
5165+
5166+ /* resolve the type size of ksym. */
5167+ ret = btf_resolve_size(btf_vmlinux, t, &tsize);
5168+ if (IS_ERR(ret)) {
5169+ tname = btf_name_by_offset(btf_vmlinux, t->name_off);
5170+ verbose(env, "unable to resolve the size of type '%s': %ld\n",
5171+ tname, PTR_ERR(ret));
5172+ return -EINVAL;
5173+ }
5174+ regs[BPF_REG_0].type =
5175+ fn->ret_type == RET_PTR_TO_MEM_OR_BTF_ID ?
5176+ PTR_TO_MEM : PTR_TO_MEM_OR_NULL;
5177+ regs[BPF_REG_0].mem_size = tsize;
5178+ } else {
5179+ regs[BPF_REG_0].type =
5180+ fn->ret_type == RET_PTR_TO_MEM_OR_BTF_ID ?
5181+ PTR_TO_BTF_ID : PTR_TO_BTF_ID_OR_NULL;
5182+ regs[BPF_REG_0].btf_id = meta.ret_btf_id;
5183+ }
49075184 } else if (fn->ret_type == RET_PTR_TO_BTF_ID_OR_NULL) {
49085185 int ret_btf_id;
49095186
@@ -5219,6 +5496,10 @@ static int adjust_ptr_min_max_vals(struct bpf_verifier_env *env, · 309 unmodified lines
52195496 dst, reg_type_str[ptr_reg->type]);
52205497 return -EACCES;
52215498 case CONST_PTR_TO_MAP:
5499+ /* smin_val represents the known value */
5500+ if (known && smin_val == 0 && opcode == BPF_ADD)
5501+ break;
5502+ fallthrough;
52225503 case PTR_TO_PACKET_END:
52235504 case PTR_TO_SOCKET:
52245505 case PTR_TO_SOCKET_OR_NULL:
@@ -5634,8 +5915,7 @@ static void scalar_min_max_and(struct bpf_reg_state *dst_reg, · 409 unmodified lines
56345915 u64 umax_val = src_reg->umax_value;
56355916
56365917 if (src_known && dst_known) {
5637− __mark_reg_known(dst_reg, dst_reg->var_off.value &
5638− src_reg->var_off.value);
5918+ __mark_reg_known(dst_reg, dst_reg->var_off.value);
56395919 return;
56405920 }
56415921
@@ -5705,8 +5985,7 @@ static void scalar_min_max_or(struct bpf_reg_state *dst_reg, · 63 unmodified lines
57055985 u64 umin_val = src_reg->umin_value;
57065986
57075987 if (src_known && dst_known) {
5708− __mark_reg_known(dst_reg, dst_reg->var_off.value |
5709− src_reg->var_off.value);
5988+ __mark_reg_known(dst_reg, dst_reg->var_off.value);
57105989 return;
57115990 }
57125991
@@ -5732,11 +6011,72 @@ static void scalar_min_max_or(struct bpf_reg_state *dst_reg, · 19 unmodified lines
57326011 __update_reg_bounds(dst_reg);
57336012 }
57346013
5735−static void __scalar32_min_max_lsh(struct bpf_reg_state *dst_reg,
5736− u64 umin_val, u64 umax_val)
6014+static void scalar32_min_max_xor(struct bpf_reg_state *dst_reg,
6015+ struct bpf_reg_state *src_reg)
57376016 {
5738− /* We lose all sign bit information (except what we can pick
5739− * up from var_off)
6017+ bool src_known = tnum_subreg_is_const(src_reg->var_off);
6018+ bool dst_known = tnum_subreg_is_const(dst_reg->var_off);
6019+ struct tnum var32_off = tnum_subreg(dst_reg->var_off);
6020+ s32 smin_val = src_reg->s32_min_value;
6021+
6022+ /* Assuming scalar64_min_max_xor will be called so it is safe
6023+ * to skip updating register for known case.
6024+ */
6025+ if (src_known && dst_known)
6026+ return;
6027+
6028+ /* We get both minimum and maximum from the var32_off. */
6029+ dst_reg->u32_min_value = var32_off.value;
6030+ dst_reg->u32_max_value = var32_off.value | var32_off.mask;
6031+
6032+ if (dst_reg->s32_min_value >= 0 && smin_val >= 0) {
6033+ /* XORing two positive sign numbers gives a positive,
6034+ * so safe to cast u32 result into s32.
6035+ */
6036+ dst_reg->s32_min_value = dst_reg->u32_min_value;
6037+ dst_reg->s32_max_value = dst_reg->u32_max_value;
6038+ } else {
6039+ dst_reg->s32_min_value = S32_MIN;
6040+ dst_reg->s32_max_value = S32_MAX;
6041+ }
6042+}
6043+
6044+static void scalar_min_max_xor(struct bpf_reg_state *dst_reg,
6045+ struct bpf_reg_state *src_reg)
6046+{
6047+ bool src_known = tnum_is_const(src_reg->var_off);
6048+ bool dst_known = tnum_is_const(dst_reg->var_off);
6049+ s64 smin_val = src_reg->smin_value;
6050+
6051+ if (src_known && dst_known) {
6052+ /* dst_reg->var_off.value has been updated earlier */
6053+ __mark_reg_known(dst_reg, dst_reg->var_off.value);
6054+ return;
6055+ }
6056+
6057+ /* We get both minimum and maximum from the var_off. */
6058+ dst_reg->umin_value = dst_reg->var_off.value;
6059+ dst_reg->umax_value = dst_reg->var_off.value | dst_reg->var_off.mask;
6060+
6061+ if (dst_reg->smin_value >= 0 && smin_val >= 0) {
6062+ /* XORing two positive sign numbers gives a positive,
6063+ * so safe to cast u64 result into s64.
6064+ */
6065+ dst_reg->smin_value = dst_reg->umin_value;
6066+ dst_reg->smax_value = dst_reg->umax_value;
6067+ } else {
6068+ dst_reg->smin_value = S64_MIN;
6069+ dst_reg->smax_value = S64_MAX;
6070+ }
6071+
6072+ __update_reg_bounds(dst_reg);
6073+}
6074+
6075+static void __scalar32_min_max_lsh(struct bpf_reg_state *dst_reg,
6076+ u64 umin_val, u64 umax_val)
6077+{
6078+ /* We lose all sign bit information (except what we can pick
6079+ * up from var_off)
57406080 */
57416081 dst_reg->s32_min_value = S32_MIN;
57426082 dst_reg->s32_max_value = S32_MAX;
@@ -6040,6 +6380,11 @@ static int adjust_scalar_min_max_vals(struct bpf_verifier_env *env, · 297 unmodified lines
60406380 scalar32_min_max_or(dst_reg, &src_reg);
60416381 scalar_min_max_or(dst_reg, &src_reg);
60426382 break;
6383+ case BPF_XOR:
6384+ dst_reg->var_off = tnum_xor(dst_reg->var_off, src_reg.var_off);
6385+ scalar32_min_max_xor(dst_reg, &src_reg);
6386+ scalar_min_max_xor(dst_reg, &src_reg);
6387+ break;
60436388 case BPF_LSH:
60446389 if (umax_val >= insn_bitness) {
60456390 /* Shifts greater than 31 or 63 are undefined.
@@ -6111,6 +6456,11 @@ static int adjust_reg_min_max_vals(struct bpf_verifier_env *env, · 65 unmodified lines
61116456 src_reg = NULL;
61126457 if (dst_reg->type != SCALAR_VALUE)
61136458 ptr_reg = dst_reg;
6459+ else
6460+ /* Make sure ID is cleared otherwise dst_reg min/max could be
6461+ * incorrectly propagated into other registers by find_equal_scalars()
6462+ */
6463+ dst_reg->id = 0;
61146464 if (BPF_SRC(insn->code) == BPF_X) {
61156465 src_reg = &regs[insn->src_reg];
61166466 if (src_reg->type != SCALAR_VALUE) {
@@ -6244,6 +6594,12 @@ static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn) · 127 unmodified lines
62446594 /* case: R1 = R2
62456595 * copy register state to dest reg
62466596 */
6597+ if (src_reg->type == SCALAR_VALUE && !src_reg->id)
6598+ /* Assign src and dst registers the same ID
6599+ * that will be used by find_equal_scalars()
6600+ * to propagate min/max range.
6601+ */
6602+ src_reg->id = ++env->id_gen;
62476603 *dst_reg = *src_reg;
62486604 dst_reg->live |= REG_LIVE_WRITTEN;
62496605 dst_reg->subreg_def = DEF_NOT_SUBREG;
@@ -6256,6 +6612,11 @@ static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn) · 6 unmodified lines
62566612 return -EACCES;
62576613 } else if (src_reg->type == SCALAR_VALUE) {
62586614 *dst_reg = *src_reg;
6615+ /* Make sure ID is cleared otherwise
6616+ * dst_reg min/max could be incorrectly
6617+ * propagated into src_reg by find_equal_scalars()
6618+ */
6619+ dst_reg->id = 0;
62596620 dst_reg->live |= REG_LIVE_WRITTEN;
62606621 dst_reg->subreg_def = env->insn_idx + 1;
62616622 } else {
@@ -6646,14 +7007,18 @@ static void reg_set_min_max(struct bpf_reg_state *true_reg, · 384 unmodified lines
66467007 struct bpf_reg_state *reg =
66477008 opcode == BPF_JEQ ? true_reg : false_reg;
66487009
6649− /* For BPF_JEQ, if this is false we know nothing Jon Snow, but
6650− * if it is true we know the value for sure. Likewise for
6651− * BPF_JNE.
7010+ /* JEQ/JNE comparison doesn't change the register equivalence.
7011+ * r1 = r2;
7012+ * if (r1 == 42) goto label;
7013+ * ...
7014+ * label: // here both r1 and r2 are known to be 42.
7015+ *
7016+ * Hence when marking register as known preserve it's ID.
66527017 */
66537018 if (is_jmp32)
66547019 __mark_reg32_known(reg, val32);
66557020 else
6656− __mark_reg_known(reg, val);
7021+ ___mark_reg_known(reg, val);
66577022 break;
66587023 }
66597024 case BPF_JSET:
@@ -7044,6 +7409,30 @@ static bool try_match_pkt_pointers(const struct bpf_insn *insn, · 384 unmodified lines
70447409 return true;
70457410 }
70467411
7412+static void find_equal_scalars(struct bpf_verifier_state *vstate,
7413+ struct bpf_reg_state *known_reg)
7414+{
7415+ struct bpf_func_state *state;
7416+ struct bpf_reg_state *reg;
7417+ int i, j;
7418+
7419+ for (i = 0; i <= vstate->curframe; i++) {
7420+ state = vstate->frame[i];
7421+ for (j = 0; j < MAX_BPF_REG; j++) {
7422+ reg = &state->regs[j];
7423+ if (reg->type == SCALAR_VALUE && reg->id == known_reg->id)
7424+ *reg = *known_reg;
7425+ }
7426+
7427+ bpf_for_each_spilled_reg(j, state, reg) {
7428+ if (!reg)
7429+ continue;
7430+ if (reg->type == SCALAR_VALUE && reg->id == known_reg->id)
7431+ *reg = *known_reg;
7432+ }
7433+ }
7434+}
7435+
70477436 static int check_cond_jmp_op(struct bpf_verifier_env *env,
70487437 struct bpf_insn *insn, int *insn_idx)
70497438 {
@@ -7172,6 +7561,12 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env, · 122 unmodified lines
71727561 reg_combine_min_max(&other_branch_regs[insn->src_reg],
71737562 &other_branch_regs[insn->dst_reg],
71747563 src_reg, dst_reg, opcode);
7564+ if (src_reg->id &&
7565+ !WARN_ON_ONCE(src_reg->id != other_branch_regs[insn->src_reg].id)) {
7566+ find_equal_scalars(this_branch, src_reg);
7567+ find_equal_scalars(other_branch, &other_branch_regs[insn->src_reg]);
7568+ }
7569+
71757570 }
71767571 } else if (dst_reg->type == SCALAR_VALUE) {
71777572 reg_set_min_max(&other_branch_regs[insn->dst_reg],
@@ -7179,6 +7574,12 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env, · 1 unmodified line
71797574 opcode, is_jmp32);
71807575 }
71817576
7577+ if (dst_reg->type == SCALAR_VALUE && dst_reg->id &&
7578+ !WARN_ON_ONCE(dst_reg->id != other_branch_regs[insn->dst_reg].id)) {
7579+ find_equal_scalars(this_branch, dst_reg);
7580+ find_equal_scalars(other_branch, &other_branch_regs[insn->dst_reg]);
7581+ }
7582+
71827583 /* detect if R == 0 where R is returned from bpf_map_lookup_elem().
71837584 * NOTE: these optimizations below are related with pointer comparison
71847585 * which will never be JMP32.
@@ -7210,6 +7611,7 @@ static int check_ld_imm(struct bpf_verifier_env *env, struct bpf_insn *insn) · 25 unmodified lines
72107611 {
72117612 struct bpf_insn_aux_data *aux = cur_aux(env);
72127613 struct bpf_reg_state *regs = cur_regs(env);
7614+ struct bpf_reg_state *dst_reg;
72137615 struct bpf_map *map;
72147616 int err;
72157617
@@ -7226,25 +7628,45 @@ static int check_ld_imm(struct bpf_verifier_env *env, struct bpf_insn *insn) · 10 unmodified lines
72267628 if (err)
72277629 return err;
72287630
7631+ dst_reg = &regs[insn->dst_reg];
72297632 if (insn->src_reg == 0) {
72307633 u64 imm = ((u64)(insn + 1)->imm << 32) | (u32)insn->imm;
72317634
7232− regs[insn->dst_reg].type = SCALAR_VALUE;
7635+ dst_reg->type = SCALAR_VALUE;
72337636 __mark_reg_known(&regs[insn->dst_reg], imm);
72347637 return 0;
72357638 }
72367639
7640+ if (insn->src_reg == BPF_PSEUDO_BTF_ID) {
7641+ mark_reg_known_zero(env, regs, insn->dst_reg);
7642+
7643+ dst_reg->type = aux->btf_var.reg_type;
7644+ switch (dst_reg->type) {
7645+ case PTR_TO_MEM:
7646+ dst_reg->mem_size = aux->btf_var.mem_size;
7647+ break;
7648+ case PTR_TO_BTF_ID:
7649+ case PTR_TO_PERCPU_BTF_ID:
7650+ dst_reg->btf_id = aux->btf_var.btf_id;
7651+ break;
7652+ default:
7653+ verbose(env, "bpf verifier is misconfigured\n");
7654+ return -EFAULT;
7655+ }
7656+ return 0;
7657+ }
7658+
72377659 map = env->used_maps[aux->map_index];
72387660 mark_reg_known_zero(env, regs, insn->dst_reg);
7239− regs[insn->dst_reg].map_ptr = map;
7661+ dst_reg->map_ptr = map;
72407662
72417663 if (insn->src_reg == BPF_PSEUDO_MAP_VALUE) {
7242− regs[insn->dst_reg].type = PTR_TO_MAP_VALUE;
7243− regs[insn->dst_reg].off = aux->map_off;
7664+ dst_reg->type = PTR_TO_MAP_VALUE;
7665+ dst_reg->off = aux->map_off;
72447666 if (map_value_has_spin_lock(map))
7245− regs[insn->dst_reg].id = ++env->id_gen;
7667+ dst_reg->id = ++env->id_gen;
72467668 } else if (insn->src_reg == BPF_PSEUDO_MAP_FD) {
7247− regs[insn->dst_reg].type = CONST_PTR_TO_MAP;
7669+ dst_reg->type = CONST_PTR_TO_MAP;
72487670 } else {
72497671 verbose(env, "bpf verifier is misconfigured\n");
72507672 return -EINVAL;
@@ -7287,7 +7709,7 @@ static int check_ld_abs(struct bpf_verifier_env *env, struct bpf_insn *insn) · 36 unmodified lines
72877709 u8 mode = BPF_MODE(insn->code);
72887710 int i, err;
72897711
7290− if (!may_access_skb(env->prog->type)) {
7712+ if (!may_access_skb(resolve_prog_type(env->prog))) {
72917713 verbose(env, "BPF_LD_[ABS|IND] instructions not allowed for this program type\n");
72927714 return -EINVAL;
72937715 }
@@ -7297,18 +7719,6 @@ static int check_ld_abs(struct bpf_verifier_env *env, struct bpf_insn *insn) · 3 unmodified lines
72977719 return -EINVAL;
72987720 }
72997721
7300− if (env->subprog_cnt > 1) {
7301− /* when program has LD_ABS insn JITs and interpreter assume
7302− * that r1 == ctx == skb which is not the case for callees
7303− * that can have arbitrary arguments. It's problematic
7304− * for main prog as well since JITs would need to analyze
7305− * all functions in order to make proper register save/restore
7306− * decisions in the main prog. Hence disallow LD_ABS with calls
7307− */
7308− verbose(env, "BPF_LD_[ABS|IND] instructions cannot be mixed with bpf-to-bpf calls\n");
7309− return -EINVAL;
7310− }
7311−
73127722 if (insn->dst_reg != BPF_REG_0 || insn->off != 0 ||
73137723 BPF_SIZE(insn->code) == BPF_DW ||
73147724 (mode == BPF_ABS && insn->src_reg != BPF_REG_0)) {
@@ -7375,11 +7785,12 @@ static int check_return_code(struct bpf_verifier_env *env) · 60 unmodified lines
73757785 const struct bpf_prog *prog = env->prog;
73767786 struct bpf_reg_state *reg;
73777787 struct tnum range = tnum_range(0, 1);
7788+ enum bpf_prog_type prog_type = resolve_prog_type(env->prog);
73787789 int err;
73797790
73807791 /* LSM and struct_ops func-ptr's return type could be "void" */
7381− if ((env->prog->type == BPF_PROG_TYPE_STRUCT_OPS ||
7382− env->prog->type == BPF_PROG_TYPE_LSM) &&
7792+ if ((prog_type == BPF_PROG_TYPE_STRUCT_OPS ||
7793+ prog_type == BPF_PROG_TYPE_LSM) &&
73837794 !prog->aux->attach_func_proto->type)
73847795 return 0;
73857796
@@ -7398,7 +7809,7 @@ static int check_return_code(struct bpf_verifier_env *env) · 12 unmodified lines
73987809 return -EACCES;
73997810 }
74007811
7401− switch (env->prog->type) {
7812+ switch (prog_type) {
74027813 case BPF_PROG_TYPE_CGROUP_SOCK_ADDR:
74037814 if (env->prog->expected_attach_type == BPF_CGROUP_UDP4_RECVMSG ||
74047815 env->prog->expected_attach_type == BPF_CGROUP_UDP6_RECVMSG ||
@@ -7718,6 +8129,23 @@ err_free: · 313 unmodified lines
77188129 return ret;
77198130 }
77208131
8132+static int check_abnormal_return(struct bpf_verifier_env *env)
8133+{
8134+ int i;
8135+
8136+ for (i = 1; i < env->subprog_cnt; i++) {
8137+ if (env->subprog_info[i].has_ld_abs) {
8138+ verbose(env, "LD_ABS is not allowed in subprogs without BTF\n");
8139+ return -EINVAL;
8140+ }
8141+ if (env->subprog_info[i].has_tail_call) {
8142+ verbose(env, "tail_call is not allowed in subprogs without BTF\n");
8143+ return -EINVAL;
8144+ }
8145+ }
8146+ return 0;
8147+}
8148+
77218149 /* The minimum supported BTF func info size */
77228150 #define MIN_BPF_FUNCINFO_SIZE 8
77238151 #define MAX_FUNCINFO_REC_SIZE 252
@@ -7726,20 +8154,24 @@ static int check_btf_func(struct bpf_verifier_env *env, · 2 unmodified lines
77268154 const union bpf_attr *attr,
77278155 union bpf_attr __user *uattr)
77288156 {
8157+ const struct btf_type *type, *func_proto, *ret_type;
77298158 u32 i, nfuncs, urec_size, min_size;
77308159 u32 krec_size = sizeof(struct bpf_func_info);
77318160 struct bpf_func_info *krecord;
77328161 struct bpf_func_info_aux *info_aux = NULL;
7733− const struct btf_type *type;
77348162 struct bpf_prog *prog;
77358163 const struct btf *btf;
77368164 void __user *urecord;
77378165 u32 prev_offset = 0;
8166+ bool scalar_return;
77388167 int ret = -ENOMEM;
77398168
77408169 nfuncs = attr->func_info_cnt;
7741− if (!nfuncs)
8170+ if (!nfuncs) {
8171+ if (check_abnormal_return(env))
8172+ return -EINVAL;
77428173 return 0;
8174+ }
77438175
77448176 if (nfuncs != env->subprog_cnt) {
77458177 verbose(env, "number of funcs in func_info doesn't match number of subprogs\n");
@@ -7787,25 +8219,23 @@ static int check_btf_func(struct bpf_verifier_env *env, · 41 unmodified lines
77878219 }
77888220
77898221 /* check insn_off */
8222+ ret = -EINVAL;
77908223 if (i == 0) {
77918224 if (krecord[i].insn_off) {
77928225 verbose(env,
77938226 "nonzero insn_off %u for the first func info record",
77948227 krecord[i].insn_off);
7795− ret = -EINVAL;
77968228 goto err_free;
77978229 }
77988230 } else if (krecord[i].insn_off <= prev_offset) {
77998231 verbose(env,
78008232 "same or smaller insn offset (%u) than previous func info record (%u)",
78018233 krecord[i].insn_off, prev_offset);
7802− ret = -EINVAL;
78038234 goto err_free;
78048235 }
78058236
78068237 if (env->subprog_info[i].start != krecord[i].insn_off) {
78078238 verbose(env, "func_info BTF section doesn't match subprog layout in BPF program\n");
7808− ret = -EINVAL;
78098239 goto err_free;
78108240 }
78118241
@@ -7814,10 +8244,26 @@ static int check_btf_func(struct bpf_verifier_env *env, · 2 unmodified lines
78148244 if (!type || !btf_type_is_func(type)) {
78158245 verbose(env, "invalid type id %d in func info",
78168246 krecord[i].type_id);
7817− ret = -EINVAL;
78188247 goto err_free;
78198248 }
78208249 info_aux[i].linkage = BTF_INFO_VLEN(type->info);
8250+
8251+ func_proto = btf_type_by_id(btf, type->type);
8252+ if (unlikely(!func_proto || !btf_type_is_func_proto(func_proto)))
8253+ /* btf_func_check() already verified it during BTF load */
8254+ goto err_free;
8255+ ret_type = btf_type_skip_modifiers(btf, func_proto->type, NULL);
8256+ scalar_return =
8257+ btf_type_is_small_int(ret_type) || btf_type_is_enum(ret_type);
8258+ if (i && !scalar_return && env->subprog_info[i].has_ld_abs) {
8259+ verbose(env, "LD_ABS is only allowed in functions that return 'int'.\n");
8260+ goto err_free;
8261+ }
8262+ if (i && !scalar_return && env->subprog_info[i].has_tail_call) {
8263+ verbose(env, "tail_call is only allowed in functions that return 'int'.\n");
8264+ goto err_free;
8265+ }
8266+
78218267 prev_offset = krecord[i].insn_off;
78228268 urecord += urec_size;
78238269 }
@@ -7978,8 +8424,11 @@ static int check_btf_info(struct bpf_verifier_env *env, · 154 unmodified lines
79788424 struct btf *btf;
79798425 int err;
79808426
7981− if (!attr->func_info_cnt && !attr->line_info_cnt)
8427+ if (!attr->func_info_cnt && !attr->line_info_cnt) {
8428+ if (check_abnormal_return(env))
8429+ return -EINVAL;
79828430 return 0;
8431+ }
79838432
79848433 btf = btf_get_by_fd(attr->prog_btf_fd);
79858434 if (IS_ERR(btf))
@@ -9119,6 +9568,92 @@ process_bpf_exit: · 1133 unmodified lines
91199568 return 0;
91209569 }
91219570
9571+/* replace pseudo btf_id with kernel symbol address */
9572+static int check_pseudo_btf_id(struct bpf_verifier_env *env,
9573+ struct bpf_insn *insn,
9574+ struct bpf_insn_aux_data *aux)
9575+{
9576+ u32 datasec_id, type, id = insn->imm;
9577+ const struct btf_var_secinfo *vsi;
9578+ const struct btf_type *datasec;
9579+ const struct btf_type *t;
9580+ const char *sym_name;
9581+ bool percpu = false;
9582+ u64 addr;
9583+ int i;
9584+
9585+ if (!btf_vmlinux) {
9586+ verbose(env, "kernel is missing BTF, make sure CONFIG_DEBUG_INFO_BTF=y is specified in Kconfig.\n");
9587+ return -EINVAL;
9588+ }
9589+
9590+ if (insn[1].imm != 0) {
9591+ verbose(env, "reserved field (insn[1].imm) is used in pseudo_btf_id ldimm64 insn.\n");
9592+ return -EINVAL;
9593+ }
9594+
9595+ t = btf_type_by_id(btf_vmlinux, id);
9596+ if (!t) {
9597+ verbose(env, "ldimm64 insn specifies invalid btf_id %d.\n", id);
9598+ return -ENOENT;
9599+ }
9600+
9601+ if (!btf_type_is_var(t)) {
9602+ verbose(env, "pseudo btf_id %d in ldimm64 isn't KIND_VAR.\n",
9603+ id);
9604+ return -EINVAL;
9605+ }
9606+
9607+ sym_name = btf_name_by_offset(btf_vmlinux, t->name_off);
9608+ addr = kallsyms_lookup_name(sym_name);
9609+ if (!addr) {
9610+ verbose(env, "ldimm64 failed to find the address for kernel symbol '%s'.\n",
9611+ sym_name);
9612+ return -ENOENT;
9613+ }
9614+
9615+ datasec_id = btf_find_by_name_kind(btf_vmlinux, ".data..percpu",
9616+ BTF_KIND_DATASEC);
9617+ if (datasec_id > 0) {
9618+ datasec = btf_type_by_id(btf_vmlinux, datasec_id);
9619+ for_each_vsi(i, datasec, vsi) {
9620+ if (vsi->type == id) {
9621+ percpu = true;
9622+ break;
9623+ }
9624+ }
9625+ }
9626+
9627+ insn[0].imm = (u32)addr;
9628+ insn[1].imm = addr >> 32;
9629+
9630+ type = t->type;
9631+ t = btf_type_skip_modifiers(btf_vmlinux, type, NULL);
9632+ if (percpu) {
9633+ aux->btf_var.reg_type = PTR_TO_PERCPU_BTF_ID;
9634+ aux->btf_var.btf_id = type;
9635+ } else if (!btf_type_is_struct(t)) {
9636+ const struct btf_type *ret;
9637+ const char *tname;
9638+ u32 tsize;
9639+
9640+ /* resolve the type size of ksym. */
9641+ ret = btf_resolve_size(btf_vmlinux, t, &tsize);
9642+ if (IS_ERR(ret)) {
9643+ tname = btf_name_by_offset(btf_vmlinux, t->name_off);
9644+ verbose(env, "ldimm64 unable to resolve the size of type '%s': %ld\n",
9645+ tname, PTR_ERR(ret));
9646+ return -EINVAL;
9647+ }
9648+ aux->btf_var.reg_type = PTR_TO_MEM;
9649+ aux->btf_var.mem_size = tsize;
9650+ } else {
9651+ aux->btf_var.reg_type = PTR_TO_BTF_ID;
9652+ aux->btf_var.btf_id = type;
9653+ }
9654+ return 0;
9655+}
9656+
91229657 static int check_map_prealloc(struct bpf_map *map)
91239658 {
91249659 return (map->map_type != BPF_MAP_TYPE_HASH &&
@@ -9154,6 +9689,7 @@ static int check_map_prog_compatibility(struct bpf_verifier_env *env, · 29 unmodified lines
91549689 struct bpf_prog *prog)
91559690
91569691 {
9692+ enum bpf_prog_type prog_type = resolve_prog_type(prog);
91579693 /*
91589694 * Validate that trace type programs use preallocated hash maps.
91599695 *
@@ -9171,8 +9707,8 @@ static int check_map_prog_compatibility(struct bpf_verifier_env *env, · 11 unmodified lines
91719707 * now, but warnings are emitted so developers are made aware of
91729708 * the unsafety and can fix their programs before this is enforced.
91739709 */
9174− if (is_tracing_prog_type(prog->type) && !is_preallocated_map(map)) {
9175− if (prog->type == BPF_PROG_TYPE_PERF_EVENT) {
9710+ if (is_tracing_prog_type(prog_type) && !is_preallocated_map(map)) {
9711+ if (prog_type == BPF_PROG_TYPE_PERF_EVENT) {
91769712 verbose(env, "perf_event programs can only use preallocated hash map\n");
91779713 return -EINVAL;
91789714 }
@@ -9184,8 +9720,8 @@ static int check_map_prog_compatibility(struct bpf_verifier_env *env, · 5 unmodified lines
91849720 verbose(env, "trace type programs with run-time allocated hash maps are unsafe. Switch to preallocated hash maps.\n");
91859721 }
91869722
9187− if ((is_tracing_prog_type(prog->type) ||
9188− prog->type == BPF_PROG_TYPE_SOCKET_FILTER) &&
9723+ if ((is_tracing_prog_type(prog_type) ||
9724+ prog_type == BPF_PROG_TYPE_SOCKET_FILTER) &&
91899725 map_value_has_spin_lock(map)) {
91909726 verbose(env, "tracing progs cannot use bpf_spin_lock yet\n");
91919727 return -EINVAL;
@@ -9202,6 +9738,23 @@ static int check_map_prog_compatibility(struct bpf_verifier_env *env, · 10 unmodified lines
92029738 return -EINVAL;
92039739 }
92049740
9741+ if (prog->aux->sleepable)
9742+ switch (map->map_type) {
9743+ case BPF_MAP_TYPE_HASH:
9744+ case BPF_MAP_TYPE_LRU_HASH:
9745+ case BPF_MAP_TYPE_ARRAY:
9746+ if (!is_preallocated_map(map)) {
9747+ verbose(env,
9748+ "Sleepable programs can only use preallocated hash maps\n");
9749+ return -EINVAL;
9750+ }
9751+ break;
9752+ default:
9753+ verbose(env,
9754+ "Sleepable programs can only use array and hash maps\n");
9755+ return -EINVAL;
9756+ }
9757+
92059758 return 0;
92069759 }
92079760
@@ -9211,10 +9764,14 @@ static bool bpf_map_is_cgroup_storage(struct bpf_map *map) · 3 unmodified lines
92119764 map->map_type == BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE);
92129765 }
92139766
9214−/* look for pseudo eBPF instructions that access map FDs and
9215− * replace them with actual map pointers
9767+/* find and rewrite pseudo imm in ld_imm64 instructions:
9768+ *
9769+ * 1. if it accesses map FD, replace it with actual map pointer.
9770+ * 2. if it accesses btf_id of a VAR, replace it with pointer to the var.
9771+ *
9772+ * NOTE: btf_vmlinux is required for converting pseudo btf_id.
92169773 */
9217−static int replace_map_fd_with_map_ptr(struct bpf_verifier_env *env)
9774+static int resolve_pseudo_ldimm64(struct bpf_verifier_env *env)
92189775 {
92199776 struct bpf_insn *insn = env->prog->insnsi;
92209777 int insn_cnt = env->prog->len;
@@ -9255,6 +9812,14 @@ static int replace_map_fd_with_map_ptr(struct bpf_verifier_env *env) · 34 unmodified lines
92559812 /* valid generic load 64-bit imm */
92569813 goto next_insn;
92579814
9815+ if (insn[0].src_reg == BPF_PSEUDO_BTF_ID) {
9816+ aux = &env->insn_aux_data[i];
9817+ err = check_pseudo_btf_id(env, insn, aux);
9818+ if (err)
9819+ return err;
9820+ goto next_insn;
9821+ }
9822+
92589823 /* In final convert_pseudo_ld_imm64() step, this is
92599824 * converted into regular 64-bit imm load insn.
92609825 */
@@ -9436,6 +10001,18 @@ static void adjust_subprog_starts(struct bpf_verifier_env *env, u32 off, u32 len · 175 unmodified lines
943610001 }
943710002 }
943810003
10004+static void adjust_poke_descs(struct bpf_prog *prog, u32 len)
10005+{
10006+ struct bpf_jit_poke_descriptor *tab = prog->aux->poke_tab;
10007+ int i, sz = prog->aux->size_poke_tab;
10008+ struct bpf_jit_poke_descriptor *desc;
10009+
10010+ for (i = 0; i < sz; i++) {
10011+ desc = &tab[i];
10012+ desc->insn_idx += len - 1;
10013+ }
10014+}
10015+
943910016 static struct bpf_prog *bpf_patch_insn_data(struct bpf_verifier_env *env, u32 off,
944010017 const struct bpf_insn *patch, u32 len)
944110018 {
@@ -9452,6 +10029,7 @@ static struct bpf_prog *bpf_patch_insn_data(struct bpf_verifier_env *env, u32 of · 10 unmodified lines
945210029 if (adjust_insn_aux_data(env, new_prog, off, len))
945310030 return NULL;
945410031 adjust_subprog_starts(env, off, len);
10032+ adjust_poke_descs(new_prog, len);
945510033 return new_prog;
945610034 }
945710035
@@ -9897,7 +10475,7 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env) · 439 unmodified lines
989710475 insn->code = BPF_LDX | BPF_PROBE_MEM |
989810476 BPF_SIZE((insn)->code);
989910477 env->prog->aux->num_exentries++;
9900− } else if (env->prog->type != BPF_PROG_TYPE_STRUCT_OPS) {
10478+ } else if (resolve_prog_type(env->prog) != BPF_PROG_TYPE_STRUCT_OPS) {
990110479 verbose(env, "Writes through BTF pointers are not allowed\n");
990210480 return -EINVAL;
990310481 }
@@ -9982,6 +10560,7 @@ static int jit_subprogs(struct bpf_verifier_env *env) · 78 unmodified lines
998210560 {
998310561 struct bpf_prog *prog = env->prog, **func, *tmp;
998410562 int i, j, subprog_start, subprog_end = 0, len, subprog;
10563+ struct bpf_map *map_ptr;
998510564 struct bpf_insn *insn;
998610565 void *old_bpf_func;
998710566 int err, num_exentries;
@@ -10049,6 +10628,31 @@ static int jit_subprogs(struct bpf_verifier_env *env) · 61 unmodified lines
1004910628 func[i]->aux->btf = prog->aux->btf;
1005010629 func[i]->aux->func_info = prog->aux->func_info;
1005110630
10631+ for (j = 0; j < prog->aux->size_poke_tab; j++) {
10632+ u32 insn_idx = prog->aux->poke_tab[j].insn_idx;
10633+ int ret;
10634+
10635+ if (!(insn_idx >= subprog_start &&
10636+ insn_idx <= subprog_end))
10637+ continue;
10638+
10639+ ret = bpf_jit_add_poke_descriptor(func[i],
10640+ &prog->aux->poke_tab[j]);
10641+ if (ret < 0) {
10642+ verbose(env, "adding tail call poke descriptor failed\n");
10643+ goto out_free;
10644+ }
10645+
10646+ func[i]->insnsi[insn_idx - subprog_start].imm = ret + 1;
10647+
10648+ map_ptr = func[i]->aux->poke_tab[ret].tail_call.map;
10649+ ret = map_ptr->ops->map_poke_track(map_ptr, func[i]->aux);
10650+ if (ret < 0) {
10651+ verbose(env, "tracking tail call prog failed\n");
10652+ goto out_free;
10653+ }
10654+ }
10655+
1005210656 /* Use bpf_prog_F_tag to indicate functions in stack traces.
1005310657 * Long term would need debug info to populate names
1005410658 */
@@ -10067,6 +10671,7 @@ static int jit_subprogs(struct bpf_verifier_env *env) · 12 unmodified lines
1006710671 num_exentries++;
1006810672 }
1006910673 func[i]->aux->num_exentries = num_exentries;
10674+ func[i]->aux->tail_call_reachable = env->subprog_info[i].tail_call_reachable;
1007010675 func[i] = bpf_int_jit_compile(func[i]);
1007110676 if (!func[i]->jited) {
1007210677 err = -ENOTSUPP;
@@ -10074,6 +10679,19 @@ static int jit_subprogs(struct bpf_verifier_env *env) · 1 unmodified line
1007410679 }
1007510680 cond_resched();
1007610681 }
10682+
10683+ /* Untrack main program's aux structs so that during map_poke_run()
10684+ * we will not stumble upon the unfilled poke descriptors; each
10685+ * of the main program's poke descs got distributed across subprogs
10686+ * and got tracked onto map, so we are sure that none of them will
10687+ * be missed after the operation below
10688+ */
10689+ for (i = 0; i < prog->aux->size_poke_tab; i++) {
10690+ map_ptr = prog->aux->poke_tab[i].tail_call.map;
10691+
10692+ map_ptr->ops->map_poke_untrack(map_ptr, prog->aux);
10693+ }
10694+
1007710695 /* at this point all bpf functions were successfully JITed
1007810696 * now populate all bpf_calls with correct addresses and
1007910697 * run last pass of JIT
@@ -10142,9 +10760,16 @@ static int jit_subprogs(struct bpf_verifier_env *env) · 62 unmodified lines
1014210760 bpf_prog_free_unused_jited_linfo(prog);
1014310761 return 0;
1014410762 out_free:
10145− for (i = 0; i < env->subprog_cnt; i++)
10146− if (func[i])
10147− bpf_jit_free(func[i]);
10763+ for (i = 0; i < env->subprog_cnt; i++) {
10764+ if (!func[i])
10765+ continue;
10766+
10767+ for (j = 0; j < func[i]->aux->size_poke_tab; j++) {
10768+ map_ptr = func[i]->aux->poke_tab[j].tail_call.map;
10769+ map_ptr->ops->map_poke_untrack(map_ptr, func[i]->aux);
10770+ }
10771+ bpf_jit_free(func[i]);
10772+ }
1014810773 kfree(func);
1014910774 out_undo_insn:
1015010775 /* cleanup main prog to be interpreted */
@@ -10178,6 +10803,13 @@ static int fixup_call_args(struct bpf_verifier_env *env) · 27 unmodified lines
1017810803 return err;
1017910804 }
1018010805 #ifndef CONFIG_BPF_JIT_ALWAYS_ON
10806+ if (env->subprog_cnt > 1 && env->prog->aux->tail_call_reachable) {
10807+ /* When JIT fails the progs with bpf2bpf calls and tail_calls
10808+ * have to be rejected, since interpreter doesn't support them yet.
10809+ */
10810+ verbose(env, "tail_calls are not allowed in non-JITed programs with bpf-to-bpf calls\n");
10811+ return -EINVAL;
10812+ }
1018110813 for (i = 0; i < prog->len; i++, insn++) {
1018210814 if (insn->code != (BPF_JMP | BPF_CALL) ||
1018310815 insn->src_reg != BPF_PSEUDO_CALL)
@@ -10341,8 +10973,9 @@ static int fixup_bpf_calls(struct bpf_verifier_env *env) · 157 unmodified lines
1034110973 * the program array.
1034210974 */
1034310975 prog->cb_access = 1;
10344− env->prog->aux->stack_depth = MAX_BPF_STACK;
10345− env->prog->aux->max_pkt_offset = MAX_PACKET_OFF;
10976+ if (!allow_tail_call_in_subprogs(env))
10977+ prog->aux->stack_depth = MAX_BPF_STACK;
10978+ prog->aux->max_pkt_offset = MAX_PACKET_OFF;
1034610979
1034710980 /* mark bpf_tail_call as different opcode to avoid
1034810981 * conditional branch in the interpeter for every normal
@@ -10362,6 +10995,7 @@ static int fixup_bpf_calls(struct bpf_verifier_env *env) · 13 unmodified lines
1036210995 .reason = BPF_POKE_REASON_TAIL_CALL,
1036310996 .tail_call.map = BPF_MAP_PTR(aux->map_ptr_state),
1036410997 .tail_call.key = bpf_map_key_immediate(aux),
10998+ .insn_idx = i + delta,
1036510999 };
1036611000
1036711001 ret = bpf_jit_add_poke_descriptor(prog, &desc);
@@ -10427,7 +11061,9 @@ static int fixup_bpf_calls(struct bpf_verifier_env *env) · 59 unmodified lines
1042711061 if (insn->imm == BPF_FUNC_map_lookup_elem &&
1042811062 ops->map_gen_lookup) {
1042911063 cnt = ops->map_gen_lookup(map_ptr, insn_buf);
10430− if (cnt == 0 || cnt >= ARRAY_SIZE(insn_buf)) {
11064+ if (cnt == -EOPNOTSUPP)
11065+ goto patch_map_ops_generic;
11066+ if (cnt <= 0 || cnt >= ARRAY_SIZE(insn_buf)) {
1043111067 verbose(env, "bpf verifier is misconfigured\n");
1043211068 return -EINVAL;
1043311069 }
@@ -10457,7 +11093,7 @@ static int fixup_bpf_calls(struct bpf_verifier_env *env) · 23 unmodified lines
1045711093 (int (*)(struct bpf_map *map, void *value))NULL));
1045811094 BUILD_BUG_ON(!__same_type(ops->map_peek_elem,
1045911095 (int (*)(struct bpf_map *map, void *value))NULL));
10460−
11096+patch_map_ops_generic:
1046111097 switch (insn->imm) {
1046211098 case BPF_FUNC_map_lookup_elem:
1046311099 insn->imm = BPF_CAST_CALL(ops->map_lookup_elem) -
@@ -10810,59 +11446,79 @@ static int check_struct_ops_btf_id(struct bpf_verifier_env *env) · 346 unmodified lines
1081011446 }
1081111447 #define SECURITY_PREFIX "security_"
1081211448
10813−static int check_attach_modify_return(struct bpf_prog *prog, unsigned long addr)
11449+static int check_attach_modify_return(unsigned long addr, const char *func_name)
1081411450 {
1081511451 if (within_error_injection_list(addr) ||
10816− !strncmp(SECURITY_PREFIX, prog->aux->attach_func_name,
10817− sizeof(SECURITY_PREFIX) - 1))
11452+ !strncmp(SECURITY_PREFIX, func_name, sizeof(SECURITY_PREFIX) - 1))
1081811453 return 0;
1081911454
1082011455 return -EINVAL;
1082111456 }
1082211457
10823−static int check_attach_btf_id(struct bpf_verifier_env *env)
11458+/* non exhaustive list of sleepable bpf_lsm_*() functions */
11459+BTF_SET_START(btf_sleepable_lsm_hooks)
11460+#ifdef CONFIG_BPF_LSM
11461+BTF_ID(func, bpf_lsm_bprm_committed_creds)
11462+#else
11463+BTF_ID_UNUSED
11464+#endif
11465+BTF_SET_END(btf_sleepable_lsm_hooks)
11466+
11467+static int check_sleepable_lsm_hook(u32 btf_id)
11468+{
11469+ return btf_id_set_contains(&btf_sleepable_lsm_hooks, btf_id);
11470+}
11471+
11472+/* list of non-sleepable functions that are otherwise on
11473+ * ALLOW_ERROR_INJECTION list
11474+ */
11475+BTF_SET_START(btf_non_sleepable_error_inject)
11476+/* Three functions below can be called from sleepable and non-sleepable context.
11477+ * Assume non-sleepable from bpf safety point of view.
11478+ */
11479+BTF_ID(func, __add_to_page_cache_locked)
11480+BTF_ID(func, should_fail_alloc_page)
11481+BTF_ID(func, should_failslab)
11482+BTF_SET_END(btf_non_sleepable_error_inject)
11483+
11484+static int check_non_sleepable_error_inject(u32 btf_id)
11485+{
11486+ return btf_id_set_contains(&btf_non_sleepable_error_inject, btf_id);
11487+}
11488+
11489+int bpf_check_attach_target(struct bpf_verifier_log *log,
11490+ const struct bpf_prog *prog,
11491+ const struct bpf_prog *tgt_prog,
11492+ u32 btf_id,
11493+ struct bpf_attach_target_info *tgt_info)
1082411494 {
10825− struct bpf_prog *prog = env->prog;
1082611495 bool prog_extension = prog->type == BPF_PROG_TYPE_EXT;
10827− struct bpf_prog *tgt_prog = prog->aux->linked_prog;
10828− u32 btf_id = prog->aux->attach_btf_id;
1082911496 const char prefix[] = "btf_trace_";
10830− struct btf_func_model fmodel;
1083111497 int ret = 0, subprog = -1, i;
10832− struct bpf_trampoline *tr;
1083311498 const struct btf_type *t;
1083411499 bool conservative = true;
1083511500 const char *tname;
1083611501 struct btf *btf;
10837− long addr;
10838− u64 key;
10839−
10840− if (prog->type == BPF_PROG_TYPE_STRUCT_OPS)
10841− return check_struct_ops_btf_id(env);
10842−
10843− if (prog->type != BPF_PROG_TYPE_TRACING &&
10844− prog->type != BPF_PROG_TYPE_LSM &&
10845− !prog_extension)
10846− return 0;
11502+ long addr = 0;
1084711503
1084811504 if (!btf_id) {
10849− verbose(env, "Tracing programs must provide btf_id\n");
11505+ bpf_log(log, "Tracing programs must provide btf_id\n");
1085011506 return -EINVAL;
1085111507 }
10852− btf = bpf_prog_get_target_btf(prog);
11508+ btf = tgt_prog ? tgt_prog->aux->btf : btf_vmlinux;
1085311509 if (!btf) {
10854− verbose(env,
11510+ bpf_log(log,
1085511511 "FENTRY/FEXIT program can only be attached to another program annotated with BTF\n");
1085611512 return -EINVAL;
1085711513 }
1085811514 t = btf_type_by_id(btf, btf_id);
1085911515 if (!t) {
10860− verbose(env, "attach_btf_id %u is invalid\n", btf_id);
11516+ bpf_log(log, "attach_btf_id %u is invalid\n", btf_id);
1086111517 return -EINVAL;
1086211518 }
1086311519 tname = btf_name_by_offset(btf, t->name_off);
1086411520 if (!tname) {
10865− verbose(env, "attach_btf_id %u doesn't have a name\n", btf_id);
11521+ bpf_log(log, "attach_btf_id %u doesn't have a name\n", btf_id);
1086611522 return -EINVAL;
1086711523 }
1086811524 if (tgt_prog) {
@@ -10874,26 +11530,24 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) · 5 unmodified lines
1087411530 break;
1087511531 }
1087611532 if (subprog == -1) {
10877− verbose(env, "Subprog %s doesn't exist\n", tname);
11533+ bpf_log(log, "Subprog %s doesn't exist\n", tname);
1087811534 return -EINVAL;
1087911535 }
1088011536 conservative = aux->func_info_aux[subprog].unreliable;
1088111537 if (prog_extension) {
1088211538 if (conservative) {
10883− verbose(env,
11539+ bpf_log(log,
1088411540 "Cannot replace static functions\n");
1088511541 return -EINVAL;
1088611542 }
1088711543 if (!prog->jit_requested) {
10888− verbose(env,
11544+ bpf_log(log,
1088911545 "Extension programs should be JITed\n");
1089011546 return -EINVAL;
1089111547 }
10892− env->ops = bpf_verifier_ops[tgt_prog->type];
10893− prog->expected_attach_type = tgt_prog->expected_attach_type;
1089411548 }
1089511549 if (!tgt_prog->jited) {
10896− verbose(env, "Can attach to only JITed progs\n");
11550+ bpf_log(log, "Can attach to only JITed progs\n");
1089711551 return -EINVAL;
1089811552 }
1089911553 if (tgt_prog->type == prog->type) {
@@ -10901,7 +11555,7 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) · 1 unmodified line
1090111555 * Cannot attach program extension to another extension.
1090211556 * It's ok to attach fentry/fexit to extension program.
1090311557 */
10904− verbose(env, "Cannot recursively attach\n");
11558+ bpf_log(log, "Cannot recursively attach\n");
1090511559 return -EINVAL;
1090611560 }
1090711561 if (tgt_prog->type == BPF_PROG_TYPE_TRACING &&
@@ -10923,32 +11577,30 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) · 15 unmodified lines
1092311577 * reasonable stack size. Hence extending fentry is not
1092411578 * allowed.
1092511579 */
10926− verbose(env, "Cannot extend fentry/fexit\n");
11580+ bpf_log(log, "Cannot extend fentry/fexit\n");
1092711581 return -EINVAL;
1092811582 }
10929− key = ((u64)aux->id) << 32 | btf_id;
1093011583 } else {
1093111584 if (prog_extension) {
10932− verbose(env, "Cannot replace kernel functions\n");
11585+ bpf_log(log, "Cannot replace kernel functions\n");
1093311586 return -EINVAL;
1093411587 }
10935− key = btf_id;
1093611588 }
1093711589
1093811590 switch (prog->expected_attach_type) {
1093911591 case BPF_TRACE_RAW_TP:
1094011592 if (tgt_prog) {
10941− verbose(env,
11593+ bpf_log(log,
1094211594 "Only FENTRY/FEXIT progs are attachable to another BPF prog\n");
1094311595 return -EINVAL;
1094411596 }
1094511597 if (!btf_type_is_typedef(t)) {
10946− verbose(env, "attach_btf_id %u is not a typedef\n",
11598+ bpf_log(log, "attach_btf_id %u is not a typedef\n",
1094711599 btf_id);
1094811600 return -EINVAL;
1094911601 }
1095011602 if (strncmp(prefix, tname, sizeof(prefix) - 1)) {
10951− verbose(env, "attach_btf_id %u points to wrong type name %s\n",
11603+ bpf_log(log, "attach_btf_id %u points to wrong type name %s\n",
1095211604 btf_id, tname);
1095311605 return -EINVAL;
1095411606 }
@@ -10962,29 +11614,20 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) · 7 unmodified lines
1096211614 /* should never happen in valid vmlinux build */
1096311615 return -EINVAL;
1096411616
10965− /* remember two read only pointers that are valid for
10966− * the life time of the kernel
10967− */
10968− prog->aux->attach_func_name = tname;
10969− prog->aux->attach_func_proto = t;
10970− prog->aux->attach_btf_trace = true;
10971− return 0;
11617+ break;
1097211618 case BPF_TRACE_ITER:
1097311619 if (!btf_type_is_func(t)) {
10974− verbose(env, "attach_btf_id %u is not a function\n",
11620+ bpf_log(log, "attach_btf_id %u is not a function\n",
1097511621 btf_id);
1097611622 return -EINVAL;
1097711623 }
1097811624 t = btf_type_by_id(btf, t->type);
1097911625 if (!btf_type_is_func_proto(t))
1098011626 return -EINVAL;
10981− prog->aux->attach_func_name = tname;
10982− prog->aux->attach_func_proto = t;
10983− if (!bpf_iter_prog_supported(prog))
10984− return -EINVAL;
10985− ret = btf_distill_func_proto(&env->log, btf, t,
10986− tname, &fmodel);
10987− return ret;
11627+ ret = btf_distill_func_proto(log, btf, t, tname, &tgt_info->fmodel);
11628+ if (ret)
11629+ return ret;
11630+ break;
1098811631 default:
1098911632 if (!prog_extension)
1099011633 return -EINVAL;
@@ -10993,42 +11636,30 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) · 2 unmodified lines
1099311636 case BPF_LSM_MAC:
1099411637 case BPF_TRACE_FENTRY:
1099511638 case BPF_TRACE_FEXIT:
10996− prog->aux->attach_func_name = tname;
10997− if (prog->type == BPF_PROG_TYPE_LSM) {
10998− ret = bpf_lsm_verify_prog(&env->log, prog);
10999− if (ret < 0)
11000− return ret;
11001− }
11002−
1100311639 if (!btf_type_is_func(t)) {
11004− verbose(env, "attach_btf_id %u is not a function\n",
11640+ bpf_log(log, "attach_btf_id %u is not a function\n",
1100511641 btf_id);
1100611642 return -EINVAL;
1100711643 }
1100811644 if (prog_extension &&
11009− btf_check_type_match(env, prog, btf, t))
11645+ btf_check_type_match(log, prog, btf, t))
1101011646 return -EINVAL;
1101111647 t = btf_type_by_id(btf, t->type);
1101211648 if (!btf_type_is_func_proto(t))
1101311649 return -EINVAL;
11014− tr = bpf_trampoline_lookup(key);
11015− if (!tr)
11016− return -ENOMEM;
11017− /* t is either vmlinux type or another program's type */
11018− prog->aux->attach_func_proto = t;
11019− mutex_lock(&tr->mutex);
11020− if (tr->func.addr) {
11021− prog->aux->trampoline = tr;
11022− goto out;
11023− }
11024− if (tgt_prog && conservative) {
11025− prog->aux->attach_func_proto = NULL;
11650+
11651+ if ((prog->aux->saved_dst_prog_type || prog->aux->saved_dst_attach_type) &&
11652+ (!tgt_prog || prog->aux->saved_dst_prog_type != tgt_prog->type ||
11653+ prog->aux->saved_dst_attach_type != tgt_prog->expected_attach_type))
11654+ return -EINVAL;
11655+
11656+ if (tgt_prog && conservative)
1102611657 t = NULL;
11027− }
11028− ret = btf_distill_func_proto(&env->log, btf, t,
11029− tname, &tr->func.model);
11658+
11659+ ret = btf_distill_func_proto(log, btf, t, tname, &tgt_info->fmodel);
1103011660 if (ret < 0)
11031− goto out;
11661+ return ret;
11662+
1103211663 if (tgt_prog) {
1103311664 if (subprog == 0)
1103411665 addr = (long) tgt_prog->bpf_func;
@@ -11037,31 +11668,137 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) · 2 unmodified lines
1103711668 } else {
1103811669 addr = kallsyms_lookup_name(tname);
1103911670 if (!addr) {
11040− verbose(env,
11671+ bpf_log(log,
1104111672 "The address of function %s cannot be found\n",
1104211673 tname);
11043− ret = -ENOENT;
11044− goto out;
11674+ return -ENOENT;
1104511675 }
1104611676 }
1104711677
11048− if (prog->expected_attach_type == BPF_MODIFY_RETURN) {
11049− ret = check_attach_modify_return(prog, addr);
11050− if (ret)
11051− verbose(env, "%s() is not modifiable\n",
11052− prog->aux->attach_func_name);
11678+ if (prog->aux->sleepable) {
11679+ ret = -EINVAL;
11680+ switch (prog->type) {
11681+ case BPF_PROG_TYPE_TRACING:
11682+ /* fentry/fexit/fmod_ret progs can be sleepable only if they are
11683+ * attached to ALLOW_ERROR_INJECTION and are not in denylist.
11684+ */
11685+ if (!check_non_sleepable_error_inject(btf_id) &&
11686+ within_error_injection_list(addr))
11687+ ret = 0;
11688+ break;
11689+ case BPF_PROG_TYPE_LSM:
11690+ /* LSM progs check that they are attached to bpf_lsm_*() funcs.
11691+ * Only some of them are sleepable.
11692+ */
11693+ if (check_sleepable_lsm_hook(btf_id))
11694+ ret = 0;
11695+ break;
11696+ default:
11697+ break;
11698+ }
11699+ if (ret) {
11700+ bpf_log(log, "%s is not sleepable\n", tname);
11701+ return ret;
11702+ }
11703+ } else if (prog->expected_attach_type == BPF_MODIFY_RETURN) {
11704+ if (tgt_prog) {
11705+ bpf_log(log, "can't modify return codes of BPF programs\n");
11706+ return -EINVAL;
11707+ }
11708+ ret = check_attach_modify_return(addr, tname);
11709+ if (ret) {
11710+ bpf_log(log, "%s() is not modifiable\n", tname);
11711+ return ret;
11712+ }
1105311713 }
1105411714
11055− if (ret)
11056− goto out;
11057− tr->func.addr = (void *)addr;
11058− prog->aux->trampoline = tr;
11059−out:
11060− mutex_unlock(&tr->mutex);
11061− if (ret)
11062− bpf_trampoline_put(tr);
11715+ break;
11716+ }
11717+ tgt_info->tgt_addr = addr;
11718+ tgt_info->tgt_name = tname;
11719+ tgt_info->tgt_type = t;
11720+ return 0;
11721+}
11722+
11723+static int check_attach_btf_id(struct bpf_verifier_env *env)
11724+{
11725+ struct bpf_prog *prog = env->prog;
11726+ struct bpf_prog *tgt_prog = prog->aux->dst_prog;
11727+ struct bpf_attach_target_info tgt_info = {};
11728+ u32 btf_id = prog->aux->attach_btf_id;
11729+ struct bpf_trampoline *tr;
11730+ int ret;
11731+ u64 key;
11732+
11733+ if (prog->aux->sleepable && prog->type != BPF_PROG_TYPE_TRACING &&
11734+ prog->type != BPF_PROG_TYPE_LSM) {
11735+ verbose(env, "Only fentry/fexit/fmod_ret and lsm programs can be sleepable\n");
11736+ return -EINVAL;
11737+ }
11738+
11739+ if (prog->type == BPF_PROG_TYPE_STRUCT_OPS)
11740+ return check_struct_ops_btf_id(env);
11741+
11742+ if (prog->type != BPF_PROG_TYPE_TRACING &&
11743+ prog->type != BPF_PROG_TYPE_LSM &&
11744+ prog->type != BPF_PROG_TYPE_EXT)
11745+ return 0;
11746+
11747+ ret = bpf_check_attach_target(&env->log, prog, tgt_prog, btf_id, &tgt_info);
11748+ if (ret)
1106311749 return ret;
11750+
11751+ if (tgt_prog && prog->type == BPF_PROG_TYPE_EXT) {
11752+ /* to make freplace equivalent to their targets, they need to
11753+ * inherit env->ops and expected_attach_type for the rest of the
11754+ * verification
11755+ */
11756+ env->ops = bpf_verifier_ops[tgt_prog->type];
11757+ prog->expected_attach_type = tgt_prog->expected_attach_type;
11758+ }
11759+
11760+ /* store info about the attachment target that will be used later */
11761+ prog->aux->attach_func_proto = tgt_info.tgt_type;
11762+ prog->aux->attach_func_name = tgt_info.tgt_name;
11763+
11764+ if (tgt_prog) {
11765+ prog->aux->saved_dst_prog_type = tgt_prog->type;
11766+ prog->aux->saved_dst_attach_type = tgt_prog->expected_attach_type;
1106411767 }
11768+
11769+ if (prog->expected_attach_type == BPF_TRACE_RAW_TP) {
11770+ prog->aux->attach_btf_trace = true;
11771+ return 0;
11772+ } else if (prog->expected_attach_type == BPF_TRACE_ITER) {
11773+ if (!bpf_iter_prog_supported(prog))
11774+ return -EINVAL;
11775+ return 0;
11776+ }
11777+
11778+ if (prog->type == BPF_PROG_TYPE_LSM) {
11779+ ret = bpf_lsm_verify_prog(&env->log, prog);
11780+ if (ret < 0)
11781+ return ret;
11782+ }
11783+
11784+ key = bpf_trampoline_compute_key(tgt_prog, btf_id);
11785+ tr = bpf_trampoline_get(key, &tgt_info);
11786+ if (!tr)
11787+ return -ENOMEM;
11788+
11789+ prog->aux->dst_trampoline = tr;
11790+ return 0;
11791+}
11792+
11793+struct btf *bpf_get_btf_vmlinux(void)
11794+{
11795+ if (!btf_vmlinux && IS_ENABLED(CONFIG_DEBUG_INFO_BTF)) {
11796+ mutex_lock(&bpf_verifier_lock);
11797+ if (!btf_vmlinux)
11798+ btf_vmlinux = btf_parse_vmlinux();
11799+ mutex_unlock(&bpf_verifier_lock);
11800+ }
11801+ return btf_vmlinux;
1106511802 }
1106611803
1106711804 int bpf_check(struct bpf_prog **prog, union bpf_attr *attr,
@@ -11097,12 +11834,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, · 29 unmodified lines
1109711834 env->ops = bpf_verifier_ops[env->prog->type];
1109811835 is_priv = bpf_capable();
1109911836
11100− if (!btf_vmlinux && IS_ENABLED(CONFIG_DEBUG_INFO_BTF)) {
11101− mutex_lock(&bpf_verifier_lock);
11102− if (!btf_vmlinux)
11103− btf_vmlinux = btf_parse_vmlinux();
11104− mutex_unlock(&bpf_verifier_lock);
11105− }
11837+ bpf_get_btf_vmlinux();
1110611838
1110711839 /* grab the mutex to protect few globals used by verifier */
1110811840 if (!is_priv)
@@ -11145,10 +11877,6 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, · 36 unmodified lines
1114511877 if (is_priv)
1114611878 env->test_state_freq = attr->prog_flags & BPF_F_TEST_STATE_FREQ;
1114711879
11148− ret = replace_map_fd_with_map_ptr(env);
11149− if (ret < 0)
11150− goto skip_full_check;
11151−
1115211880 if (bpf_prog_is_dev_bound(env->prog->aux)) {
1115311881 ret = bpf_prog_offload_verifier_prep(env->prog);
1115411882 if (ret)
@@ -11174,6 +11902,10 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, · 19 unmodified lines
1117411902 if (ret)
1117511903 goto skip_full_check;
1117611904
11905+ ret = resolve_pseudo_ldimm64(env);
11906+ if (ret < 0)
11907+ goto skip_full_check;
11908+
1117711909 ret = check_cfg(env);
1117811910 if (ret < 0)
1117911911 goto skip_full_check;
kernel/rcu/tasks.h+41−12
@@ -28,6 +28,8 @@ typedef void (*postgp_func_t)(struct rcu_tasks *rtp); · 27 unmodified lines
2828 * @kthread_ptr: This flavor's grace-period/callback-invocation kthread.
2929 * @gp_func: This flavor's grace-period-wait function.
3030 * @gp_state: Grace period's most recent state transition (debugging).
31+ * @gp_sleep: Per-grace-period sleep to prevent CPU-bound looping.
32+ * @init_fract: Initial backoff sleep interval.
3133 * @gp_jiffies: Time of last @gp_state transition.
3234 * @gp_start: Most recent grace-period start in jiffies.
3335 * @n_gps: Number of grace periods completed since boot.
@@ -48,6 +50,8 @@ struct rcu_tasks { · 14 unmodified lines
4850 struct wait_queue_head cbs_wq;
4951 raw_spinlock_t cbs_lock;
5052 int gp_state;
53+ int gp_sleep;
54+ int init_fract;
5155 unsigned long gp_jiffies;
5256 unsigned long gp_start;
5357 unsigned long n_gps;
@@ -81,7 +85,7 @@ static struct rcu_tasks rt_name = \ · 27 unmodified lines
8185 DEFINE_STATIC_SRCU(tasks_rcu_exit_srcu);
8286
8387 /* Avoid IPIing CPUs early in the grace period. */
84−#define RCU_TASK_IPI_DELAY (HZ / 2)
88+#define RCU_TASK_IPI_DELAY (IS_ENABLED(CONFIG_TASKS_TRACE_RCU_READ_MB) ? HZ / 2 : 0)
8589 static int rcu_task_ipi_delay __read_mostly = RCU_TASK_IPI_DELAY;
8690 module_param(rcu_task_ipi_delay, int, 0644);
8791
@@ -231,7 +235,7 @@ static int __noreturn rcu_tasks_kthread(void *arg) · 143 unmodified lines
231235 cond_resched();
232236 }
233237 /* Paranoid sleep to keep this from entering a tight loop */
234− schedule_timeout_idle(HZ/10);
238+ schedule_timeout_idle(rtp->gp_sleep);
235239
236240 set_tasks_gp_state(rtp, RTGS_WAIT_CBS);
237241 }
@@ -329,8 +333,10 @@ static void rcu_tasks_wait_gp(struct rcu_tasks *rtp) · 91 unmodified lines
329333 */
330334 lastreport = jiffies;
331335
332− /* Start off with HZ/10 wait and slowly back off to 1 HZ wait. */
333− fract = 10;
336+ // Start off with initial wait and slowly back off to 1 HZ wait.
337+ fract = rtp->init_fract;
338+ if (fract > HZ)
339+ fract = HZ;
334340
335341 for (;;) {
336342 bool firstreport;
@@ -553,6 +559,8 @@ EXPORT_SYMBOL_GPL(rcu_barrier_tasks); · 216 unmodified lines
553559
554560 static int __init rcu_spawn_tasks_kthread(void)
555561 {
562+ rcu_tasks.gp_sleep = HZ / 10;
563+ rcu_tasks.init_fract = 10;
556564 rcu_tasks.pregp_func = rcu_tasks_pregp_step;
557565 rcu_tasks.pertask_func = rcu_tasks_pertask;
558566 rcu_tasks.postscan_func = rcu_tasks_postscan;
@@ -685,6 +693,7 @@ EXPORT_SYMBOL_GPL(rcu_barrier_tasks_rude); · 126 unmodified lines
685693
686694 static int __init rcu_spawn_tasks_rude_kthread(void)
687695 {
696+ rcu_tasks_rude.gp_sleep = HZ / 10;
688697 rcu_spawn_tasks_kthread_generic(&rcu_tasks_rude);
689698 return 0;
690699 }
@@ -745,9 +754,9 @@ static DEFINE_PER_CPU(bool, trc_ipi_to_cpu); · 54 unmodified lines
745754
746755 // The number of detections of task quiescent state relying on
747756 // heavyweight readers executing explicit memory barriers.
748−unsigned long n_heavy_reader_attempts;
749−unsigned long n_heavy_reader_updates;
750−unsigned long n_heavy_reader_ofl_updates;
757+static unsigned long n_heavy_reader_attempts;
758+static unsigned long n_heavy_reader_updates;
759+static unsigned long n_heavy_reader_ofl_updates;
751760
752761 void call_rcu_tasks_trace(struct rcu_head *rhp, rcu_callback_t func);
753762 DEFINE_RCU_TASKS(rcu_tasks_trace, rcu_tasks_wait_gp, call_rcu_tasks_trace,
@@ -821,6 +830,12 @@ static void trc_read_check_handler(void *t_in) · 67 unmodified lines
821830 WRITE_ONCE(t->trc_reader_checked, true);
822831 goto reset_ipi;
823832 }
833+ // If we are racing with an rcu_read_unlock_trace(), try again later.
834+ if (unlikely(t->trc_reader_nesting < 0)) {
835+ if (WARN_ON_ONCE(atomic_dec_and_test(&trc_n_readers_need_end)))
836+ wake_up(&trc_wait);
837+ goto reset_ipi;
838+ }
824839 WRITE_ONCE(t->trc_reader_checked, true);
825840
826841 // Get here if the task is in a read-side critical section. Set
@@ -911,7 +926,8 @@ static void trc_wait_for_one_reader(struct task_struct *t, · 84 unmodified lines
911926
912927 // If currently running, send an IPI, either way, add to list.
913928 trc_add_holdout(t, bhp);
914− if (task_curr(t) && time_after(jiffies, rcu_tasks_trace.gp_start + rcu_task_ipi_delay)) {
929+ if (task_curr(t) &&
930+ time_after(jiffies + 1, rcu_tasks_trace.gp_start + rcu_task_ipi_delay)) {
915931 // The task is currently running, so try IPIing it.
916932 cpu = task_cpu(t);
917933
@@ -1072,15 +1088,17 @@ static void rcu_tasks_trace_postgp(struct rcu_tasks *rtp) · 154 unmodified lines
10721088 if (ret)
10731089 break; // Count reached zero.
10741090 // Stall warning time, so make a list of the offenders.
1091+ rcu_read_lock();
10751092 for_each_process_thread(g, t)
10761093 if (READ_ONCE(t->trc_reader_special.b.need_qs))
10771094 trc_add_holdout(t, &holdouts);
1095+ rcu_read_unlock();
10781096 firstreport = true;
1079− list_for_each_entry_safe(t, g, &holdouts, trc_holdout_list)
1080− if (READ_ONCE(t->trc_reader_special.b.need_qs)) {
1097+ list_for_each_entry_safe(t, g, &holdouts, trc_holdout_list) {
1098+ if (READ_ONCE(t->trc_reader_special.b.need_qs))
10811099 show_stalled_task_trace(t, &firstreport);
1082− trc_del_holdout(t);
1083− }
1100+ trc_del_holdout(t); // Release task_struct reference.
1101+ }
10841102 if (firstreport)
10851103 pr_err("INFO: rcu_tasks_trace detected stalls? (Counter/taskslist mismatch?)\n");
10861104 show_stalled_ipi_trace();
@@ -1163,6 +1181,17 @@ EXPORT_SYMBOL_GPL(rcu_barrier_tasks_trace); · 76 unmodified lines
11631181
11641182 static int __init rcu_spawn_tasks_trace_kthread(void)
11651183 {
1184+ if (IS_ENABLED(CONFIG_TASKS_TRACE_RCU_READ_MB)) {
1185+ rcu_tasks_trace.gp_sleep = HZ / 10;
1186+ rcu_tasks_trace.init_fract = 10;
1187+ } else {
1188+ rcu_tasks_trace.gp_sleep = HZ / 200;
1189+ if (rcu_tasks_trace.gp_sleep <= 0)
1190+ rcu_tasks_trace.gp_sleep = 1;
1191+ rcu_tasks_trace.init_fract = HZ / 5;
1192+ if (rcu_tasks_trace.init_fract <= 0)
1193+ rcu_tasks_trace.init_fract = 1;
1194+ }
11661195 rcu_tasks_trace.pregp_func = rcu_tasks_trace_pregp_step;
11671196 rcu_tasks_trace.pertask_func = rcu_tasks_trace_pertask;
11681197 rcu_tasks_trace.postscan_func = rcu_tasks_trace_postscan;
kernel/taskstats.c+7−33
@@ -34,17 +34,13 @@ struct kmem_cache *taskstats_cache; · 33 unmodified lines
3434
3535 static struct genl_family family;
3636
37−static const struct nla_policy taskstats_cmd_get_policy[TASKSTATS_CMD_ATTR_MAX+1] = {
37+static const struct nla_policy taskstats_cmd_get_policy[] = {
3838 [TASKSTATS_CMD_ATTR_PID] = { .type = NLA_U32 },
3939 [TASKSTATS_CMD_ATTR_TGID] = { .type = NLA_U32 },
4040 [TASKSTATS_CMD_ATTR_REGISTER_CPUMASK] = { .type = NLA_STRING },
4141 [TASKSTATS_CMD_ATTR_DEREGISTER_CPUMASK] = { .type = NLA_STRING },};
4242
43−/*
44− * We have to use TASKSTATS_CMD_ATTR_MAX here, it is the maxattr in the family.
45− * Make sure they are always aligned.
46− */
47−static const struct nla_policy cgroupstats_cmd_get_policy[TASKSTATS_CMD_ATTR_MAX+1] = {
43+static const struct nla_policy cgroupstats_cmd_get_policy[] = {
4844 [CGROUPSTATS_CMD_ATTR_FD] = { .type = NLA_U32 },
4945 };
5046
@@ -649,47 +645,25 @@ static const struct genl_ops taskstats_ops[] = { · 598 unmodified lines
649645 .cmd = TASKSTATS_CMD_GET,
650646 .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP,
651647 .doit = taskstats_user_cmd,
652− /* policy enforced later */
653− .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_HASPOL,
648+ .policy = taskstats_cmd_get_policy,
649+ .maxattr = ARRAY_SIZE(taskstats_cmd_get_policy) - 1,
650+ .flags = GENL_ADMIN_PERM,
654651 },
655652 {
656653 .cmd = CGROUPSTATS_CMD_GET,
657654 .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP,
658655 .doit = cgroupstats_user_cmd,
659− /* policy enforced later */
660− .flags = GENL_CMD_CAP_HASPOL,
656+ .policy = cgroupstats_cmd_get_policy,
657+ .maxattr = ARRAY_SIZE(cgroupstats_cmd_get_policy) - 1,
661658 },
662659 };
663660
664−static int taskstats_pre_doit(const struct genl_ops *ops, struct sk_buff *skb,
665− struct genl_info *info)
666−{
667− const struct nla_policy *policy = NULL;
668−
669− switch (ops->cmd) {
670− case TASKSTATS_CMD_GET:
671− policy = taskstats_cmd_get_policy;
672− break;
673− case CGROUPSTATS_CMD_GET:
674− policy = cgroupstats_cmd_get_policy;
675− break;
676− default:
677− return -EINVAL;
678− }
679−
680− return nlmsg_validate_deprecated(info->nlhdr, GENL_HDRLEN,
681− TASKSTATS_CMD_ATTR_MAX, policy,
682− info->extack);
683−}
684−
685661 static struct genl_family family __ro_after_init = {
686662 .name = TASKSTATS_GENL_NAME,
687663 .version = TASKSTATS_GENL_VERSION,
688− .maxattr = TASKSTATS_CMD_ATTR_MAX,
689664 .module = THIS_MODULE,
690665 .ops = taskstats_ops,
691666 .n_ops = ARRAY_SIZE(taskstats_ops),
692− .pre_doit = taskstats_pre_doit,
693667 };
694668
695669 /* Needed early in initialization */
kernel/trace/bpf_trace.c+165−7
@@ -7,6 +7,7 @@ · 6 unmodified lines
77 #include <linux/slab.h>
88 #include <linux/bpf.h>
99 #include <linux/bpf_perf_event.h>
10+#include <linux/btf.h>
1011 #include <linux/filter.h>
1112 #include <linux/uaccess.h>
1213 #include <linux/ctype.h>
@@ -16,6 +17,9 @@ · 3 unmodified lines
1617 #include <linux/error-injection.h>
1718 #include <linux/btf_ids.h>
1819
20+#include <uapi/linux/bpf.h>
21+#include <uapi/linux/btf.h>
22+
1923 #include <asm/tlb.h>
2024
2125 #include "trace_probe.h"
@@ -67,6 +71,10 @@ static struct bpf_raw_event_map *bpf_get_raw_tracepoint_module(const char *name) · 45 unmodified lines
6771 u64 bpf_get_stackid(u64 r1, u64 r2, u64 r3, u64 r4, u64 r5);
6872 u64 bpf_get_stack(u64 r1, u64 r2, u64 r3, u64 r4, u64 r5);
6973
74+static int bpf_btf_printf_prepare(struct btf_ptr *ptr, u32 btf_ptr_size,
75+ u64 flags, const struct btf **btf,
76+ s32 *btf_id);
77+
7078 /**
7179 * trace_call_bpf - invoke BPF program
7280 * @call: tracepoint event
@@ -743,19 +751,18 @@ out: · 670 unmodified lines
743751 return err;
744752 }
745753
746−BTF_ID_LIST(bpf_seq_printf_btf_ids)
747−BTF_ID(struct, seq_file)
754+BTF_ID_LIST_SINGLE(btf_seq_file_ids, struct, seq_file)
748755
749756 static const struct bpf_func_proto bpf_seq_printf_proto = {
750757 .func = bpf_seq_printf,
751758 .gpl_only = true,
752759 .ret_type = RET_INTEGER,
753760 .arg1_type = ARG_PTR_TO_BTF_ID,
761+ .arg1_btf_id = &btf_seq_file_ids[0],
754762 .arg2_type = ARG_PTR_TO_MEM,
755763 .arg3_type = ARG_CONST_SIZE,
756764 .arg4_type = ARG_PTR_TO_MEM_OR_NULL,
757765 .arg5_type = ARG_CONST_SIZE_OR_ZERO,
758− .btf_id = bpf_seq_printf_btf_ids,
759766 };
760767
761768 BPF_CALL_3(bpf_seq_write, struct seq_file *, m, const void *, data, u32, len)
@@ -763,17 +770,39 @@ BPF_CALL_3(bpf_seq_write, struct seq_file *, m, const void *, data, u32, len) · 1 unmodified line
763770 return seq_write(m, data, len) ? -EOVERFLOW : 0;
764771 }
765772
766−BTF_ID_LIST(bpf_seq_write_btf_ids)
767−BTF_ID(struct, seq_file)
768−
769773 static const struct bpf_func_proto bpf_seq_write_proto = {
770774 .func = bpf_seq_write,
771775 .gpl_only = true,
772776 .ret_type = RET_INTEGER,
773777 .arg1_type = ARG_PTR_TO_BTF_ID,
778+ .arg1_btf_id = &btf_seq_file_ids[0],
774779 .arg2_type = ARG_PTR_TO_MEM,
775780 .arg3_type = ARG_CONST_SIZE_OR_ZERO,
776− .btf_id = bpf_seq_write_btf_ids,
781+};
782+
783+BPF_CALL_4(bpf_seq_printf_btf, struct seq_file *, m, struct btf_ptr *, ptr,
784+ u32, btf_ptr_size, u64, flags)
785+{
786+ const struct btf *btf;
787+ s32 btf_id;
788+ int ret;
789+
790+ ret = bpf_btf_printf_prepare(ptr, btf_ptr_size, flags, &btf, &btf_id);
791+ if (ret)
792+ return ret;
793+
794+ return btf_type_seq_show_flags(btf, btf_id, ptr->ptr, m, flags);
795+}
796+
797+static const struct bpf_func_proto bpf_seq_printf_btf_proto = {
798+ .func = bpf_seq_printf_btf,
799+ .gpl_only = true,
800+ .ret_type = RET_INTEGER,
801+ .arg1_type = ARG_PTR_TO_BTF_ID,
802+ .arg1_btf_id = &btf_seq_file_ids[0],
803+ .arg2_type = ARG_PTR_TO_MEM,
804+ .arg3_type = ARG_CONST_SIZE_OR_ZERO,
805+ .arg4_type = ARG_ANYTHING,
777806 };
778807
779808 static __always_inline int
@@ -1098,6 +1127,118 @@ static const struct bpf_func_proto bpf_send_signal_thread_proto = { · 318 unmodified lines
10981127 .arg1_type = ARG_ANYTHING,
10991128 };
11001129
1130+BPF_CALL_3(bpf_d_path, struct path *, path, char *, buf, u32, sz)
1131+{
1132+ long len;
1133+ char *p;
1134+
1135+ if (!sz)
1136+ return 0;
1137+
1138+ p = d_path(path, buf, sz);
1139+ if (IS_ERR(p)) {
1140+ len = PTR_ERR(p);
1141+ } else {
1142+ len = buf + sz - p;
1143+ memmove(buf, p, len);
1144+ }
1145+
1146+ return len;
1147+}
1148+
1149+BTF_SET_START(btf_allowlist_d_path)
1150+#ifdef CONFIG_SECURITY
1151+BTF_ID(func, security_file_permission)
1152+BTF_ID(func, security_inode_getattr)
1153+BTF_ID(func, security_file_open)
1154+#endif
1155+#ifdef CONFIG_SECURITY_PATH
1156+BTF_ID(func, security_path_truncate)
1157+#endif
1158+BTF_ID(func, vfs_truncate)
1159+BTF_ID(func, vfs_fallocate)
1160+BTF_ID(func, dentry_open)
1161+BTF_ID(func, vfs_getattr)
1162+BTF_ID(func, filp_close)
1163+BTF_SET_END(btf_allowlist_d_path)
1164+
1165+static bool bpf_d_path_allowed(const struct bpf_prog *prog)
1166+{
1167+ return btf_id_set_contains(&btf_allowlist_d_path, prog->aux->attach_btf_id);
1168+}
1169+
1170+BTF_ID_LIST_SINGLE(bpf_d_path_btf_ids, struct, path)
1171+
1172+static const struct bpf_func_proto bpf_d_path_proto = {
1173+ .func = bpf_d_path,
1174+ .gpl_only = false,
1175+ .ret_type = RET_INTEGER,
1176+ .arg1_type = ARG_PTR_TO_BTF_ID,
1177+ .arg1_btf_id = &bpf_d_path_btf_ids[0],
1178+ .arg2_type = ARG_PTR_TO_MEM,
1179+ .arg3_type = ARG_CONST_SIZE_OR_ZERO,
1180+ .allowed = bpf_d_path_allowed,
1181+};
1182+
1183+#define BTF_F_ALL (BTF_F_COMPACT | BTF_F_NONAME | \
1184+ BTF_F_PTR_RAW | BTF_F_ZERO)
1185+
1186+static int bpf_btf_printf_prepare(struct btf_ptr *ptr, u32 btf_ptr_size,
1187+ u64 flags, const struct btf **btf,
1188+ s32 *btf_id)
1189+{
1190+ const struct btf_type *t;
1191+
1192+ if (unlikely(flags & ~(BTF_F_ALL)))
1193+ return -EINVAL;
1194+
1195+ if (btf_ptr_size != sizeof(struct btf_ptr))
1196+ return -EINVAL;
1197+
1198+ *btf = bpf_get_btf_vmlinux();
1199+
1200+ if (IS_ERR_OR_NULL(*btf))
1201+ return PTR_ERR(*btf);
1202+
1203+ if (ptr->type_id > 0)
1204+ *btf_id = ptr->type_id;
1205+ else
1206+ return -EINVAL;
1207+
1208+ if (*btf_id > 0)
1209+ t = btf_type_by_id(*btf, *btf_id);
1210+ if (*btf_id <= 0 || !t)
1211+ return -ENOENT;
1212+
1213+ return 0;
1214+}
1215+
1216+BPF_CALL_5(bpf_snprintf_btf, char *, str, u32, str_size, struct btf_ptr *, ptr,
1217+ u32, btf_ptr_size, u64, flags)
1218+{
1219+ const struct btf *btf;
1220+ s32 btf_id;
1221+ int ret;
1222+
1223+ ret = bpf_btf_printf_prepare(ptr, btf_ptr_size, flags, &btf, &btf_id);
1224+ if (ret)
1225+ return ret;
1226+
1227+ return btf_type_snprintf_show(btf, btf_id, ptr->ptr, str, str_size,
1228+ flags);
1229+}
1230+
1231+const struct bpf_func_proto bpf_snprintf_btf_proto = {
1232+ .func = bpf_snprintf_btf,
1233+ .gpl_only = false,
1234+ .ret_type = RET_INTEGER,
1235+ .arg1_type = ARG_PTR_TO_MEM,
1236+ .arg2_type = ARG_CONST_SIZE,
1237+ .arg3_type = ARG_PTR_TO_MEM,
1238+ .arg4_type = ARG_CONST_SIZE,
1239+ .arg5_type = ARG_ANYTHING,
1240+};
1241+
11011242 const struct bpf_func_proto *
11021243 bpf_tracing_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
11031244 {
@@ -1182,6 +1323,14 @@ bpf_tracing_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) · 78 unmodified lines
11821323 return &bpf_jiffies64_proto;
11831324 case BPF_FUNC_get_task_stack:
11841325 return &bpf_get_task_stack_proto;
1326+ case BPF_FUNC_copy_from_user:
1327+ return prog->aux->sleepable ? &bpf_copy_from_user_proto : NULL;
1328+ case BPF_FUNC_snprintf_btf:
1329+ return &bpf_snprintf_btf_proto;
1330+ case BPF_FUNC_bpf_per_cpu_ptr:
1331+ return &bpf_per_cpu_ptr_proto;
1332+ case BPF_FUNC_bpf_this_cpu_ptr:
1333+ return &bpf_this_cpu_ptr_proto;
11851334 default:
11861335 return NULL;
11871336 }
@@ -1579,6 +1728,12 @@ tracing_prog_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) · 391 unmodified lines
15791728 return prog->expected_attach_type == BPF_TRACE_ITER ?
15801729 &bpf_seq_write_proto :
15811730 NULL;
1731+ case BPF_FUNC_seq_printf_btf:
1732+ return prog->expected_attach_type == BPF_TRACE_ITER ?
1733+ &bpf_seq_printf_btf_proto :
1734+ NULL;
1735+ case BPF_FUNC_d_path:
1736+ return &bpf_d_path_proto;
15821737 default:
15831738 return raw_tp_prog_func_proto(func_id, prog);
15841739 }
@@ -1625,6 +1780,9 @@ const struct bpf_verifier_ops raw_tracepoint_verifier_ops = { · 40 unmodified lines
16251780 };
16261781
16271782 const struct bpf_prog_ops raw_tracepoint_prog_ops = {
1783+#ifdef CONFIG_NET
1784+ .test_run = bpf_prog_test_run_raw_tp,
1785+#endif
16281786 };
16291787
16301788 const struct bpf_verifier_ops tracing_verifier_ops = {

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t