History

Merge tag 'net-next-5.10' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next

Pull networking updates from Jakub Kicinski: - Add redirect_neigh() BPF packet redirect helper, allowing to limit stack traversal in common container configs and improving TCP back-pressure. Daniel reports ~10Gbps => ~15Gbps single stream TCP performance gain. - Expand netlink policy support and improve policy export to user space. (Ge)netlink core performs request validation according to declared policies. Expand the expressiveness of those policies (min/max length and bitmasks). Allow dumping policies for particular commands. This is used for feature discovery by user space (instead of kernel version parsing or trial and error). - Support IGMPv3/MLDv2 multicast listener discovery protocols in bridge. - Allow more than 255 IPv4 multicast interfaces. - Add support for Type of Service (ToS) reflection in SYN/SYN-ACK packets of TCPv6. - In Multi-patch TCP (MPTCP) support concurrent transmission of data on multiple subflows in a load balancing scenario. Enhance advertising addresses via the RM_ADDR/ADD_ADDR options. - Support SMC-Dv2 version of SMC, which enables multi-subnet deployments. - Allow more calls to same peer in RxRPC. - Support two new Controller Area Network (CAN) protocols - CAN-FD and ISO 15765-2:2016. - Add xfrm/IPsec compat layer, solving the 32bit user space on 64bit kernel problem. - Add TC actions for implementing MPLS L2 VPNs. - Improve nexthop code - e.g. handle various corner cases when nexthop objects are removed from groups better, skip unnecessary notifications and make it easier to offload nexthops into HW by converting to a blocking notifier. - Support adding and consuming TCP header options by BPF programs, opening the doors for easy experimental and deployment-specific TCP option use. - Reorganize TCP congestion control (CC) initialization to simplify life of TCP CC implemented in BPF. - Add support for shipping BPF programs with the kernel and loading them early on boot via the User Mode Driver mechanism, hence reusing all the user space infra we have. - Support sleepable BPF programs, initially targeting LSM and tracing. - Add bpf_d_path() helper for returning full path for given 'struct path'. - Make bpf_tail_call compatible with bpf-to-bpf calls. - Allow BPF programs to call map_update_elem on sockmaps. - Add BPF Type Format (BTF) support for type and enum discovery, as well as support for using BTF within the kernel itself (current use is for pretty printing structures). - Support listing and getting information about bpf_links via the bpf syscall. - Enhance kernel interfaces around NIC firmware update. Allow specifying overwrite mask to control if settings etc. are reset during update; report expected max time operation may take to users; support firmware activation without machine reboot incl. limits of how much impact reset may have (e.g. dropping link or not). - Extend ethtool configuration interface to report IEEE-standard counters, to limit the need for per-vendor logic in user space. - Adopt or extend devlink use for debug, monitoring, fw update in many drivers (dsa loop, ice, ionic, sja1105, qed, mlxsw, mv88e6xxx, dpaa2-eth). - In mlxsw expose critical and emergency SFP module temperature alarms. Refactor port buffer handling to make the defaults more suitable and support setting these values explicitly via the DCBNL interface. - Add XDP support for Intel's igb driver. - Support offloading TC flower classification and filtering rules to mscc_ocelot switches. - Add PTP support for Marvell Octeontx2 and PP2.2 hardware, as well as fixed interval period pulse generator and one-step timestamping in dpaa-eth. - Add support for various auth offloads in WiFi APs, e.g. SAE (WPA3) offload. - Add Lynx PHY/PCS MDIO module, and convert various drivers which have this HW to use it. Convert mvpp2 to split PCS. - Support Marvell Prestera 98DX3255 24-port switch ASICs, as well as 7-port Mediatek MT7531 IP. - Add initial support for QCA6390 and IPQ6018 in ath11k WiFi driver, and wcn3680 support in wcn36xx. - Improve performance for packets which don't require much offloads on recent Mellanox NICs by 20% by making multiple packets share a descriptor entry. - Move chelsio inline crypto drivers (for TLS and IPsec) from the crypto subtree to drivers/net. Move MDIO drivers out of the phy directory. - Clean up a lot of W=1 warnings, reportedly the actively developed subsections of networking drivers should now build W=1 warning free. - Make sure drivers don't use in_interrupt() to dynamically adapt their code. Convert tasklets to use new tasklet_setup API (sadly this conversion is not yet complete). * tag 'net-next-5.10' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next: (2583 commits) Revert "bpfilter: Fix build error with CONFIG_BPFILTER_UMH" net, sockmap: Don't call bpf_prog_put() on NULL pointer bpf, selftest: Fix flaky tcp_hdr_options test when adding addr to lo bpf, sockmap: Add locking annotations to iterator netfilter: nftables: allow re-computing sctp CRC-32C in 'payload' statements net: fix pos incrementment in ipv6_route_seq_next net/smc: fix invalid return code in smcd_new_buf_create() net/smc: fix valid DMBE buffer sizes net/smc: fix use-after-free of delayed events bpfilter: Fix build error with CONFIG_BPFILTER_UMH cxgb4/ch_ipsec: Replace the module name to ch_ipsec from chcr net: sched: Fix suspicious RCU usage while accessing tcf_tunnel_info bpf: Fix register equivalence tracking. rxrpc: Fix loss of final ack on shutdown rxrpc: Fix bundle counting for exclusive connections netfilter: restore NF_INET_NUMHOOKS ibmveth: Identify ingress large send packets. ibmveth: Switch order of ibmveth_helper calls. cxgb4: handle 4-tuple PEDIT to NAT mode translation selftests: Add VRF route leaking tests ...

Linus Torvalds committed 6 years ago

Commit 9ff9b0d2 parents 840e5bb 105faa82301 files · +130475 −51396Browse files

Showing the changes to kernel (45 of 2301 files). Show all

Unified · Split

45 files
  1. kernel/Makefile +1 −1
  2. kernel/bpf/Makefile +3 −0
  3. kernel/bpf/arraymap.c +84 −18
  4. kernel/bpf/bpf_inode_storage.c +272 −0
  5. kernel/bpf/bpf_iter.c +60 −2
  6. kernel/bpf/bpf_local_storage.c +600 −0
  7. kernel/bpf/bpf_lsm.c +20 −1
  8. kernel/bpf/bpf_struct_ops.c +2 −4
  9. kernel/bpf/btf.c +1054 −167
  10. kernel/bpf/core.c +21 −8
  11. kernel/bpf/cpumap.c +6 −11
  12. kernel/bpf/devmap.c +9 −8
  13. kernel/bpf/hashtab.c +13 −9
  14. kernel/bpf/helpers.c +58 −0
  15. kernel/bpf/inode.c +113 −3
  16. kernel/bpf/lpm_trie.c +1 −0
  17. kernel/bpf/map_in_map.c +9 −15
  18. kernel/bpf/map_in_map.h +0 −2
  19. kernel/bpf/map_iter.c +15 −0
  20. kernel/bpf/percpu_freelist.c +96 −5
  21. kernel/bpf/percpu_freelist.h +1 −0
  22. kernel/bpf/preload/.gitignore +4 −0
  23. kernel/bpf/preload/Kconfig +26 −0
  24. kernel/bpf/preload/Makefile +25 −0
  25. kernel/bpf/preload/bpf_preload.h +16 −0
  26. kernel/bpf/preload/bpf_preload_kern.c +91 −0
  27. kernel/bpf/preload/bpf_preload_umd_blob.S +7 −0
  28. kernel/bpf/preload/iterators/.gitignore +2 −0
  29. kernel/bpf/preload/iterators/Makefile +57 −0
  30. kernel/bpf/preload/iterators/README +4 −0
  31. kernel/bpf/preload/iterators/bpf_preload_common.h +13 −0
  32. kernel/bpf/preload/iterators/iterators.bpf.c +114 −0
  33. kernel/bpf/preload/iterators/iterators.c +94 −0
  34. kernel/bpf/preload/iterators/iterators.skel.h +412 −0
  35. kernel/bpf/queue_stack_maps.c +2 −0
  36. kernel/bpf/reuseport_array.c +2 −1
  37. kernel/bpf/ringbuf.c +1 −0
  38. kernel/bpf/stackmap.c +3 −3
  39. kernel/bpf/syscall.c +278 −53
  40. kernel/bpf/task_iter.c +11 −4
  41. kernel/bpf/trampoline.c +51 −12
  42. kernel/bpf/verifier.c +1060 −328
  43. kernel/rcu/tasks.h +41 −12
  44. kernel/taskstats.c +7 −33
  45. kernel/trace/bpf_trace.c +165 −7
kernel/Makefile+1−1
@@ -12,7 +12,7 @@ obj-y = fork.o exec_domain.o panic.o \ · 11 unmodified lines
12 notifier.o ksysfs.o cred.o reboot.o \12 notifier.o ksysfs.o cred.o reboot.o \
13 async.o range.o smpboot.o ucount.o regset.o13 async.o range.o smpboot.o ucount.o regset.o
14 14
15obj-$(CONFIG_BPFILTER) += usermode_driver.o15obj-$(CONFIG_USERMODE_DRIVER) += usermode_driver.o
16obj-$(CONFIG_MODULES) += kmod.o16obj-$(CONFIG_MODULES) += kmod.o
17obj-$(CONFIG_MULTIUSER) += groups.o17obj-$(CONFIG_MULTIUSER) += groups.o
18 18
kernel/bpf/Makefile+3−0
@@ -5,6 +5,7 @@ CFLAGS_core.o += $(call cc-disable-warning, override-init) · 4 unmodified lines
5obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o tnum.o bpf_iter.o map_iter.o task_iter.o prog_iter.o5obj-$(CONFIG_BPF_SYSCALL) += syscall.o verifier.o inode.o helpers.o tnum.o bpf_iter.o map_iter.o task_iter.o prog_iter.o
6obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o6obj-$(CONFIG_BPF_SYSCALL) += hashtab.o arraymap.o percpu_freelist.o bpf_lru_list.o lpm_trie.o map_in_map.o
7obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o7obj-$(CONFIG_BPF_SYSCALL) += local_storage.o queue_stack_maps.o ringbuf.o
8obj-${CONFIG_BPF_LSM} += bpf_inode_storage.o
8obj-$(CONFIG_BPF_SYSCALL) += disasm.o9obj-$(CONFIG_BPF_SYSCALL) += disasm.o
9obj-$(CONFIG_BPF_JIT) += trampoline.o10obj-$(CONFIG_BPF_JIT) += trampoline.o
10obj-$(CONFIG_BPF_SYSCALL) += btf.o11obj-$(CONFIG_BPF_SYSCALL) += btf.o
@@ -12,6 +13,7 @@ obj-$(CONFIG_BPF_JIT) += dispatcher.o · 1 unmodified line
12ifeq ($(CONFIG_NET),y)13ifeq ($(CONFIG_NET),y)
13obj-$(CONFIG_BPF_SYSCALL) += devmap.o14obj-$(CONFIG_BPF_SYSCALL) += devmap.o
14obj-$(CONFIG_BPF_SYSCALL) += cpumap.o15obj-$(CONFIG_BPF_SYSCALL) += cpumap.o
16obj-$(CONFIG_BPF_SYSCALL) += bpf_local_storage.o
15obj-$(CONFIG_BPF_SYSCALL) += offload.o17obj-$(CONFIG_BPF_SYSCALL) += offload.o
16obj-$(CONFIG_BPF_SYSCALL) += net_namespace.o18obj-$(CONFIG_BPF_SYSCALL) += net_namespace.o
17endif19endif
@@ -29,3 +31,4 @@ ifeq ($(CONFIG_BPF_JIT),y) · 11 unmodified lines
29obj-$(CONFIG_BPF_SYSCALL) += bpf_struct_ops.o31obj-$(CONFIG_BPF_SYSCALL) += bpf_struct_ops.o
30obj-${CONFIG_BPF_LSM} += bpf_lsm.o32obj-${CONFIG_BPF_LSM} += bpf_lsm.o
31endif33endif
34obj-$(CONFIG_BPF_PRELOAD) += preload/
kernel/bpf/arraymap.c+84−18
@@ -10,11 +10,13 @@ · 9 unmodified lines
10#include <linux/filter.h>10#include <linux/filter.h>
11#include <linux/perf_event.h>11#include <linux/perf_event.h>
12#include <uapi/linux/btf.h>12#include <uapi/linux/btf.h>
13#include <linux/rcupdate_trace.h>
13 14
14#include "map_in_map.h"15#include "map_in_map.h"
15 16
16#define ARRAY_CREATE_FLAG_MASK \17#define ARRAY_CREATE_FLAG_MASK \
17 (BPF_F_NUMA_NODE | BPF_F_MMAPABLE | BPF_F_ACCESS_MASK)18 (BPF_F_NUMA_NODE | BPF_F_MMAPABLE | BPF_F_ACCESS_MASK | \
19 BPF_F_PRESERVE_ELEMS | BPF_F_INNER_MAP)
18 20
19static void bpf_array_free_percpu(struct bpf_array *array)21static void bpf_array_free_percpu(struct bpf_array *array)
20{22{
@@ -60,7 +62,11 @@ int array_map_alloc_check(union bpf_attr *attr) · 39 unmodified lines
60 return -EINVAL;62 return -EINVAL;
61 63
62 if (attr->map_type != BPF_MAP_TYPE_ARRAY &&64 if (attr->map_type != BPF_MAP_TYPE_ARRAY &&
63 attr->map_flags & BPF_F_MMAPABLE)65 attr->map_flags & (BPF_F_MMAPABLE | BPF_F_INNER_MAP))
66 return -EINVAL;
67
68 if (attr->map_type != BPF_MAP_TYPE_PERF_EVENT_ARRAY &&
69 attr->map_flags & BPF_F_PRESERVE_ELEMS)
64 return -EINVAL;70 return -EINVAL;
65 71
66 if (attr->value_size > KMALLOC_MAX_SIZE)72 if (attr->value_size > KMALLOC_MAX_SIZE)
@@ -208,7 +214,7 @@ static int array_map_direct_value_meta(const struct bpf_map *map, u64 imm, · 141 unmodified lines
208}214}
209 215
210/* emit BPF instructions equivalent to C code of array_map_lookup_elem() */216/* emit BPF instructions equivalent to C code of array_map_lookup_elem() */
211static u32 array_map_gen_lookup(struct bpf_map *map, struct bpf_insn *insn_buf)217static int array_map_gen_lookup(struct bpf_map *map, struct bpf_insn *insn_buf)
212{218{
213 struct bpf_array *array = container_of(map, struct bpf_array, map);219 struct bpf_array *array = container_of(map, struct bpf_array, map);
214 struct bpf_insn *insn = insn_buf;220 struct bpf_insn *insn = insn_buf;
@@ -217,6 +223,9 @@ static u32 array_map_gen_lookup(struct bpf_map *map, struct bpf_insn *insn_buf) · 2 unmodified lines
217 const int map_ptr = BPF_REG_1;223 const int map_ptr = BPF_REG_1;
218 const int index = BPF_REG_2;224 const int index = BPF_REG_2;
219 225
226 if (map->map_flags & BPF_F_INNER_MAP)
227 return -EOPNOTSUPP;
228
220 *insn++ = BPF_ALU64_IMM(BPF_ADD, map_ptr, offsetof(struct bpf_array, value));229 *insn++ = BPF_ALU64_IMM(BPF_ADD, map_ptr, offsetof(struct bpf_array, value));
221 *insn++ = BPF_LDX_MEM(BPF_W, ret, index, 0);230 *insn++ = BPF_LDX_MEM(BPF_W, ret, index, 0);
222 if (!map->bypass_spec_v1) {231 if (!map->bypass_spec_v1) {
@@ -487,6 +496,15 @@ static int array_map_mmap(struct bpf_map *map, struct vm_area_struct *vma) · 264 unmodified lines
487 vma->vm_pgoff + pgoff);496 vma->vm_pgoff + pgoff);
488}497}
489 498
499static bool array_map_meta_equal(const struct bpf_map *meta0,
500 const struct bpf_map *meta1)
501{
502 if (!bpf_map_meta_equal(meta0, meta1))
503 return false;
504 return meta0->map_flags & BPF_F_INNER_MAP ? true :
505 meta0->max_entries == meta1->max_entries;
506}
507
490struct bpf_iter_seq_array_map_info {508struct bpf_iter_seq_array_map_info {
491 struct bpf_map *map;509 struct bpf_map *map;
492 void *percpu_value_buf;510 void *percpu_value_buf;
@@ -625,6 +643,7 @@ static const struct bpf_iter_seq_info iter_seq_info = { · 132 unmodified lines
625 643
626static int array_map_btf_id;644static int array_map_btf_id;
627const struct bpf_map_ops array_map_ops = {645const struct bpf_map_ops array_map_ops = {
646 .map_meta_equal = array_map_meta_equal,
628 .map_alloc_check = array_map_alloc_check,647 .map_alloc_check = array_map_alloc_check,
629 .map_alloc = array_map_alloc,648 .map_alloc = array_map_alloc,
630 .map_free = array_map_free,649 .map_free = array_map_free,
@@ -647,6 +666,7 @@ const struct bpf_map_ops array_map_ops = { · 16 unmodified lines
647 666
648static int percpu_array_map_btf_id;667static int percpu_array_map_btf_id;
649const struct bpf_map_ops percpu_array_map_ops = {668const struct bpf_map_ops percpu_array_map_ops = {
669 .map_meta_equal = bpf_map_meta_equal,
650 .map_alloc_check = array_map_alloc_check,670 .map_alloc_check = array_map_alloc_check,
651 .map_alloc = array_map_alloc,671 .map_alloc = array_map_alloc,
652 .map_free = array_map_free,672 .map_free = array_map_free,
@@ -888,6 +908,7 @@ static void prog_array_map_poke_run(struct bpf_map *map, u32 key, · 235 unmodified lines
888 struct bpf_prog *old,908 struct bpf_prog *old,
889 struct bpf_prog *new)909 struct bpf_prog *new)
890{910{
911 u8 *old_addr, *new_addr, *old_bypass_addr;
891 struct prog_poke_elem *elem;912 struct prog_poke_elem *elem;
892 struct bpf_array_aux *aux;913 struct bpf_array_aux *aux;
893 914
@@ -908,12 +929,13 @@ static void prog_array_map_poke_run(struct bpf_map *map, u32 key, · 14 unmodified lines
908 * there could be danger of use after free otherwise.929 * there could be danger of use after free otherwise.
909 * 2) Initially when we start tracking aux, the program930 * 2) Initially when we start tracking aux, the program
910 * is not JITed yet and also does not have a kallsyms931 * is not JITed yet and also does not have a kallsyms
911 * entry. We skip these as poke->ip_stable is not932 * entry. We skip these as poke->tailcall_target_stable
912 * active yet. The JIT will do the final fixup before933 * is not active yet. The JIT will do the final fixup
913 * setting it stable. The various poke->ip_stable are934 * before setting it stable. The various
914 * successively activated, so tail call updates can935 * poke->tailcall_target_stable are successively
915 * arrive from here while JIT is still finishing its936 * activated, so tail call updates can arrive from here
916 * final fixup for non-activated poke entries.937 * while JIT is still finishing its final fixup for
938 * non-activated poke entries.
917 * 3) On program teardown, the program's kallsym entry gets939 * 3) On program teardown, the program's kallsym entry gets
918 * removed out of RCU callback, but we can only untrack940 * removed out of RCU callback, but we can only untrack
919 * from sleepable context, therefore bpf_arch_text_poke()941 * from sleepable context, therefore bpf_arch_text_poke()
@@ -930,7 +952,7 @@ static void prog_array_map_poke_run(struct bpf_map *map, u32 key, · 10 unmodified lines
930 * 5) Any other error happening below from bpf_arch_text_poke()952 * 5) Any other error happening below from bpf_arch_text_poke()
931 * is a unexpected bug.953 * is a unexpected bug.
932 */954 */
933 if (!READ_ONCE(poke->ip_stable))955 if (!READ_ONCE(poke->tailcall_target_stable))
934 continue;956 continue;
935 if (poke->reason != BPF_POKE_REASON_TAIL_CALL)957 if (poke->reason != BPF_POKE_REASON_TAIL_CALL)
936 continue;958 continue;
@@ -938,12 +960,39 @@ static void prog_array_map_poke_run(struct bpf_map *map, u32 key, · 1 unmodified line
938 poke->tail_call.key != key)960 poke->tail_call.key != key)
939 continue;961 continue;
940 962
941 ret = bpf_arch_text_poke(poke->ip, BPF_MOD_JUMP,963 old_bypass_addr = old ? NULL : poke->bypass_addr;
942 old ? (u8 *)old->bpf_func +964 old_addr = old ? (u8 *)old->bpf_func + poke->adj_off : NULL;
943 poke->adj_off : NULL,965 new_addr = new ? (u8 *)new->bpf_func + poke->adj_off : NULL;
944 new ? (u8 *)new->bpf_func +966
945 poke->adj_off : NULL);967 if (new) {
946 BUG_ON(ret < 0 && ret != -EINVAL);968 ret = bpf_arch_text_poke(poke->tailcall_target,
969 BPF_MOD_JUMP,
970 old_addr, new_addr);
971 BUG_ON(ret < 0 && ret != -EINVAL);
972 if (!old) {
973 ret = bpf_arch_text_poke(poke->tailcall_bypass,
974 BPF_MOD_JUMP,
975 poke->bypass_addr,
976 NULL);
977 BUG_ON(ret < 0 && ret != -EINVAL);
978 }
979 } else {
980 ret = bpf_arch_text_poke(poke->tailcall_bypass,
981 BPF_MOD_JUMP,
982 old_bypass_addr,
983 poke->bypass_addr);
984 BUG_ON(ret < 0 && ret != -EINVAL);
985 /* let other CPUs finish the execution of program
986 * so that it will not possible to expose them
987 * to invalid nop, stack unwind, nop state
988 */
989 if (!ret)
990 synchronize_rcu();
991 ret = bpf_arch_text_poke(poke->tailcall_target,
992 BPF_MOD_JUMP,
993 old_addr, NULL);
994 BUG_ON(ret < 0 && ret != -EINVAL);
995 }
947 }996 }
948 }997 }
949}998}
@@ -1003,6 +1052,11 @@ static void prog_array_map_free(struct bpf_map *map) · 53 unmodified lines
1003 fd_array_map_free(map);1052 fd_array_map_free(map);
1004}1053}
1005 1054
1055/* prog_array->aux->{type,jited} is a runtime binding.
1056 * Doing static check alone in the verifier is not enough.
1057 * Thus, prog_array_map cannot be used as an inner_map
1058 * and map_meta_equal is not implemented.
1059 */
1006static int prog_array_map_btf_id;1060static int prog_array_map_btf_id;
1007const struct bpf_map_ops prog_array_map_ops = {1061const struct bpf_map_ops prog_array_map_ops = {
1008 .map_alloc_check = fd_array_map_alloc_check,1062 .map_alloc_check = fd_array_map_alloc_check,
@@ -1090,6 +1144,9 @@ static void perf_event_fd_array_release(struct bpf_map *map, · 81 unmodified lines
1090 struct bpf_event_entry *ee;1144 struct bpf_event_entry *ee;
1091 int i;1145 int i;
1092 1146
1147 if (map->map_flags & BPF_F_PRESERVE_ELEMS)
1148 return;
1149
1093 rcu_read_lock();1150 rcu_read_lock();
1094 for (i = 0; i < array->map.max_entries; i++) {1151 for (i = 0; i < array->map.max_entries; i++) {
1095 ee = READ_ONCE(array->ptrs[i]);1152 ee = READ_ONCE(array->ptrs[i]);
@@ -1099,11 +1156,19 @@ static void perf_event_fd_array_release(struct bpf_map *map, · 3 unmodified lines
1099 rcu_read_unlock();1156 rcu_read_unlock();
1100}1157}
1101 1158
1159static void perf_event_fd_array_map_free(struct bpf_map *map)
1160{
1161 if (map->map_flags & BPF_F_PRESERVE_ELEMS)
1162 bpf_fd_array_map_clear(map);
1163 fd_array_map_free(map);
1164}
1165
1102static int perf_event_array_map_btf_id;1166static int perf_event_array_map_btf_id;
1103const struct bpf_map_ops perf_event_array_map_ops = {1167const struct bpf_map_ops perf_event_array_map_ops = {
1168 .map_meta_equal = bpf_map_meta_equal,
1104 .map_alloc_check = fd_array_map_alloc_check,1169 .map_alloc_check = fd_array_map_alloc_check,
1105 .map_alloc = array_map_alloc,1170 .map_alloc = array_map_alloc,
1106 .map_free = fd_array_map_free,1171 .map_free = perf_event_fd_array_map_free,
1107 .map_get_next_key = array_map_get_next_key,1172 .map_get_next_key = array_map_get_next_key,
1108 .map_lookup_elem = fd_array_map_lookup_elem,1173 .map_lookup_elem = fd_array_map_lookup_elem,
1109 .map_delete_elem = fd_array_map_delete_elem,1174 .map_delete_elem = fd_array_map_delete_elem,
@@ -1137,6 +1202,7 @@ static void cgroup_fd_array_free(struct bpf_map *map) · 27 unmodified lines
1137 1202
1138static int cgroup_array_map_btf_id;1203static int cgroup_array_map_btf_id;
1139const struct bpf_map_ops cgroup_array_map_ops = {1204const struct bpf_map_ops cgroup_array_map_ops = {
1205 .map_meta_equal = bpf_map_meta_equal,
1140 .map_alloc_check = fd_array_map_alloc_check,1206 .map_alloc_check = fd_array_map_alloc_check,
1141 .map_alloc = array_map_alloc,1207 .map_alloc = array_map_alloc,
1142 .map_free = cgroup_fd_array_free,1208 .map_free = cgroup_fd_array_free,
@@ -1190,7 +1256,7 @@ static void *array_of_map_lookup_elem(struct bpf_map *map, void *key) · 47 unmodified lines
1190 return READ_ONCE(*inner_map);1256 return READ_ONCE(*inner_map);
1191}1257}
1192 1258
1193static u32 array_of_map_gen_lookup(struct bpf_map *map,1259static int array_of_map_gen_lookup(struct bpf_map *map,
1194 struct bpf_insn *insn_buf)1260 struct bpf_insn *insn_buf)
1195{1261{
1196 struct bpf_array *array = container_of(map, struct bpf_array, map);1262 struct bpf_array *array = container_of(map, struct bpf_array, map);
kernel/bpf/bpf_inode_storage.c+272−0
@@ -0,0 +1,272 @@
1// SPDX-License-Identifier: GPL-2.0
2/*
3 * Copyright (c) 2019 Facebook
4 * Copyright 2020 Google LLC.
5 */
6
7#include <linux/rculist.h>
8#include <linux/list.h>
9#include <linux/hash.h>
10#include <linux/types.h>
11#include <linux/spinlock.h>
12#include <linux/bpf.h>
13#include <linux/bpf_local_storage.h>
14#include <net/sock.h>
15#include <uapi/linux/sock_diag.h>
16#include <uapi/linux/btf.h>
17#include <linux/bpf_lsm.h>
18#include <linux/btf_ids.h>
19#include <linux/fdtable.h>
20
21DEFINE_BPF_STORAGE_CACHE(inode_cache);
22
23static struct bpf_local_storage __rcu **
24inode_storage_ptr(void *owner)
25{
26 struct inode *inode = owner;
27 struct bpf_storage_blob *bsb;
28
29 bsb = bpf_inode(inode);
30 if (!bsb)
31 return NULL;
32 return &bsb->storage;
33}
34
35static struct bpf_local_storage_data *inode_storage_lookup(struct inode *inode,
36 struct bpf_map *map,
37 bool cacheit_lockit)
38{
39 struct bpf_local_storage *inode_storage;
40 struct bpf_local_storage_map *smap;
41 struct bpf_storage_blob *bsb;
42
43 bsb = bpf_inode(inode);
44 if (!bsb)
45 return NULL;
46
47 inode_storage = rcu_dereference(bsb->storage);
48 if (!inode_storage)
49 return NULL;
50
51 smap = (struct bpf_local_storage_map *)map;
52 return bpf_local_storage_lookup(inode_storage, smap, cacheit_lockit);
53}
54
55void bpf_inode_storage_free(struct inode *inode)
56{
57 struct bpf_local_storage_elem *selem;
58 struct bpf_local_storage *local_storage;
59 bool free_inode_storage = false;
60 struct bpf_storage_blob *bsb;
61 struct hlist_node *n;
62
63 bsb = bpf_inode(inode);
64 if (!bsb)
65 return;
66
67 rcu_read_lock();
68
69 local_storage = rcu_dereference(bsb->storage);
70 if (!local_storage) {
71 rcu_read_unlock();
72 return;
73 }
74
75 /* Netiher the bpf_prog nor the bpf-map's syscall
76 * could be modifying the local_storage->list now.
77 * Thus, no elem can be added-to or deleted-from the
78 * local_storage->list by the bpf_prog or by the bpf-map's syscall.
79 *
80 * It is racing with bpf_local_storage_map_free() alone
81 * when unlinking elem from the local_storage->list and
82 * the map's bucket->list.
83 */
84 raw_spin_lock_bh(&local_storage->lock);
85 hlist_for_each_entry_safe(selem, n, &local_storage->list, snode) {
86 /* Always unlink from map before unlinking from
87 * local_storage.
88 */
89 bpf_selem_unlink_map(selem);
90 free_inode_storage = bpf_selem_unlink_storage_nolock(
91 local_storage, selem, false);
92 }
93 raw_spin_unlock_bh(&local_storage->lock);
94 rcu_read_unlock();
95
96 /* free_inoode_storage should always be true as long as
97 * local_storage->list was non-empty.
98 */
99 if (free_inode_storage)
100 kfree_rcu(local_storage, rcu);
101}
102
103static void *bpf_fd_inode_storage_lookup_elem(struct bpf_map *map, void *key)
104{
105 struct bpf_local_storage_data *sdata;
106 struct file *f;
107 int fd;
108
109 fd = *(int *)key;
110 f = fget_raw(fd);
111 if (!f)
112 return NULL;
113
114 sdata = inode_storage_lookup(f->f_inode, map, true);
115 fput(f);
116 return sdata ? sdata->data : NULL;
117}
118
119static int bpf_fd_inode_storage_update_elem(struct bpf_map *map, void *key,
120 void *value, u64 map_flags)
121{
122 struct bpf_local_storage_data *sdata;
123 struct file *f;
124 int fd;
125
126 fd = *(int *)key;
127 f = fget_raw(fd);
128 if (!f || !inode_storage_ptr(f->f_inode))
129 return -EBADF;
130
131 sdata = bpf_local_storage_update(f->f_inode,
132 (struct bpf_local_storage_map *)map,
133 value, map_flags);
134 fput(f);
135 return PTR_ERR_OR_ZERO(sdata);
136}
137
138static int inode_storage_delete(struct inode *inode, struct bpf_map *map)
139{
140 struct bpf_local_storage_data *sdata;
141
142 sdata = inode_storage_lookup(inode, map, false);
143 if (!sdata)
144 return -ENOENT;
145
146 bpf_selem_unlink(SELEM(sdata));
147
148 return 0;
149}
150
151static int bpf_fd_inode_storage_delete_elem(struct bpf_map *map, void *key)
152{
153 struct file *f;
154 int fd, err;
155
156 fd = *(int *)key;
157 f = fget_raw(fd);
158 if (!f)
159 return -EBADF;
160
161 err = inode_storage_delete(f->f_inode, map);
162 fput(f);
163 return err;
164}
165
166BPF_CALL_4(bpf_inode_storage_get, struct bpf_map *, map, struct inode *, inode,
167 void *, value, u64, flags)
168{
169 struct bpf_local_storage_data *sdata;
170
171 if (flags & ~(BPF_LOCAL_STORAGE_GET_F_CREATE))
172 return (unsigned long)NULL;
173
174 /* explicitly check that the inode_storage_ptr is not
175 * NULL as inode_storage_lookup returns NULL in this case and
176 * bpf_local_storage_update expects the owner to have a
177 * valid storage pointer.
178 */
179 if (!inode_storage_ptr(inode))
180 return (unsigned long)NULL;
181
182 sdata = inode_storage_lookup(inode, map, true);
183 if (sdata)
184 return (unsigned long)sdata->data;
185
186 /* This helper must only called from where the inode is gurranteed
187 * to have a refcount and cannot be freed.
188 */
189 if (flags & BPF_LOCAL_STORAGE_GET_F_CREATE) {
190 sdata = bpf_local_storage_update(
191 inode, (struct bpf_local_storage_map *)map, value,
192 BPF_NOEXIST);
193 return IS_ERR(sdata) ? (unsigned long)NULL :
194 (unsigned long)sdata->data;
195 }
196
197 return (unsigned long)NULL;
198}
199
200BPF_CALL_2(bpf_inode_storage_delete,
201 struct bpf_map *, map, struct inode *, inode)
202{
203 /* This helper must only called from where the inode is gurranteed
204 * to have a refcount and cannot be freed.
205 */
206 return inode_storage_delete(inode, map);
207}
208
209static int notsupp_get_next_key(struct bpf_map *map, void *key,
210 void *next_key)
211{
212 return -ENOTSUPP;
213}
214
215static struct bpf_map *inode_storage_map_alloc(union bpf_attr *attr)
216{
217 struct bpf_local_storage_map *smap;
218
219 smap = bpf_local_storage_map_alloc(attr);
220 if (IS_ERR(smap))
221 return ERR_CAST(smap);
222
223 smap->cache_idx = bpf_local_storage_cache_idx_get(&inode_cache);
224 return &smap->map;
225}
226
227static void inode_storage_map_free(struct bpf_map *map)
228{
229 struct bpf_local_storage_map *smap;
230
231 smap = (struct bpf_local_storage_map *)map;
232 bpf_local_storage_cache_idx_free(&inode_cache, smap->cache_idx);
233 bpf_local_storage_map_free(smap);
234}
235
236static int inode_storage_map_btf_id;
237const struct bpf_map_ops inode_storage_map_ops = {
238 .map_meta_equal = bpf_map_meta_equal,
239 .map_alloc_check = bpf_local_storage_map_alloc_check,
240 .map_alloc = inode_storage_map_alloc,
241 .map_free = inode_storage_map_free,
242 .map_get_next_key = notsupp_get_next_key,
243 .map_lookup_elem = bpf_fd_inode_storage_lookup_elem,
244 .map_update_elem = bpf_fd_inode_storage_update_elem,
245 .map_delete_elem = bpf_fd_inode_storage_delete_elem,
246 .map_check_btf = bpf_local_storage_map_check_btf,
247 .map_btf_name = "bpf_local_storage_map",
248 .map_btf_id = &inode_storage_map_btf_id,
249 .map_owner_storage_ptr = inode_storage_ptr,
250};
251
252BTF_ID_LIST_SINGLE(bpf_inode_storage_btf_ids, struct, inode)
253
254const struct bpf_func_proto bpf_inode_storage_get_proto = {
255 .func = bpf_inode_storage_get,
256 .gpl_only = false,
257 .ret_type = RET_PTR_TO_MAP_VALUE_OR_NULL,
258 .arg1_type = ARG_CONST_MAP_PTR,
259 .arg2_type = ARG_PTR_TO_BTF_ID,
260 .arg2_btf_id = &bpf_inode_storage_btf_ids[0],
261 .arg3_type = ARG_PTR_TO_MAP_VALUE_OR_NULL,
262 .arg4_type = ARG_ANYTHING,
263};
264
265const struct bpf_func_proto bpf_inode_storage_delete_proto = {
266 .func = bpf_inode_storage_delete,
267 .gpl_only = false,
268 .ret_type = RET_INTEGER,
269 .arg1_type = ARG_CONST_MAP_PTR,
270 .arg2_type = ARG_PTR_TO_BTF_ID,
271 .arg2_btf_id = &bpf_inode_storage_btf_ids[0],
272};
kernel/bpf/bpf_iter.c+60−2
@@ -88,8 +88,8 @@ static ssize_t bpf_seq_read(struct file *file, char __user *buf, size_t size, · 87 unmodified lines
88 mutex_lock(&seq->lock);88 mutex_lock(&seq->lock);
89 89
90 if (!seq->buf) {90 if (!seq->buf) {
91 seq->size = PAGE_SIZE;91 seq->size = PAGE_SIZE << 3;
92 seq->buf = kmalloc(seq->size, GFP_KERNEL);92 seq->buf = kvmalloc(seq->size, GFP_KERNEL);
93 if (!seq->buf) {93 if (!seq->buf) {
94 err = -ENOMEM;94 err = -ENOMEM;
95 goto done;95 goto done;
@@ -390,10 +390,68 @@ out_unlock: · 294 unmodified lines
390 return ret;390 return ret;
391}391}
392 392
393static void bpf_iter_link_show_fdinfo(const struct bpf_link *link,
394 struct seq_file *seq)
395{
396 struct bpf_iter_link *iter_link =
397 container_of(link, struct bpf_iter_link, link);
398 bpf_iter_show_fdinfo_t show_fdinfo;
399
400 seq_printf(seq,
401 "target_name:\t%s\n",
402 iter_link->tinfo->reg_info->target);
403
404 show_fdinfo = iter_link->tinfo->reg_info->show_fdinfo;
405 if (show_fdinfo)
406 show_fdinfo(&iter_link->aux, seq);
407}
408
409static int bpf_iter_link_fill_link_info(const struct bpf_link *link,
410 struct bpf_link_info *info)
411{
412 struct bpf_iter_link *iter_link =
413 container_of(link, struct bpf_iter_link, link);
414 char __user *ubuf = u64_to_user_ptr(info->iter.target_name);
415 bpf_iter_fill_link_info_t fill_link_info;
416 u32 ulen = info->iter.target_name_len;
417 const char *target_name;
418 u32 target_len;
419
420 if (!ulen ^ !ubuf)
421 return -EINVAL;
422
423 target_name = iter_link->tinfo->reg_info->target;
424 target_len = strlen(target_name);
425 info->iter.target_name_len = target_len + 1;
426
427 if (ubuf) {
428 if (ulen >= target_len + 1) {
429 if (copy_to_user(ubuf, target_name, target_len + 1))
430 return -EFAULT;
431 } else {
432 char zero = '\0';
433
434 if (copy_to_user(ubuf, target_name, ulen - 1))
435 return -EFAULT;
436 if (put_user(zero, ubuf + ulen - 1))
437 return -EFAULT;
438 return -ENOSPC;
439 }
440 }
441
442 fill_link_info = iter_link->tinfo->reg_info->fill_link_info;
443 if (fill_link_info)
444 return fill_link_info(&iter_link->aux, info);
445
446 return 0;
447}
448
393static const struct bpf_link_ops bpf_iter_link_lops = {449static const struct bpf_link_ops bpf_iter_link_lops = {
394 .release = bpf_iter_link_release,450 .release = bpf_iter_link_release,
395 .dealloc = bpf_iter_link_dealloc,451 .dealloc = bpf_iter_link_dealloc,
396 .update_prog = bpf_iter_link_replace,452 .update_prog = bpf_iter_link_replace,
453 .show_fdinfo = bpf_iter_link_show_fdinfo,
454 .fill_link_info = bpf_iter_link_fill_link_info,
397};455};
398 456
399bool bpf_link_is_iter(struct bpf_link *link)457bool bpf_link_is_iter(struct bpf_link *link)
kernel/bpf/bpf_local_storage.c+600−0
@@ -0,0 +1,600 @@
1// SPDX-License-Identifier: GPL-2.0
2/* Copyright (c) 2019 Facebook */
3#include <linux/rculist.h>
4#include <linux/list.h>
5#include <linux/hash.h>
6#include <linux/types.h>
7#include <linux/spinlock.h>
8#include <linux/bpf.h>
9#include <linux/btf_ids.h>
10#include <linux/bpf_local_storage.h>
11#include <net/sock.h>
12#include <uapi/linux/sock_diag.h>
13#include <uapi/linux/btf.h>
14
15#define BPF_LOCAL_STORAGE_CREATE_FLAG_MASK (BPF_F_NO_PREALLOC | BPF_F_CLONE)
16
17static struct bpf_local_storage_map_bucket *
18select_bucket(struct bpf_local_storage_map *smap,
19 struct bpf_local_storage_elem *selem)
20{
21 return &smap->buckets[hash_ptr(selem, smap->bucket_log)];
22}
23
24static int mem_charge(struct bpf_local_storage_map *smap, void *owner, u32 size)
25{
26 struct bpf_map *map = &smap->map;
27
28 if (!map->ops->map_local_storage_charge)
29 return 0;
30
31 return map->ops->map_local_storage_charge(smap, owner, size);
32}
33
34static void mem_uncharge(struct bpf_local_storage_map *smap, void *owner,
35 u32 size)
36{
37 struct bpf_map *map = &smap->map;
38
39 if (map->ops->map_local_storage_uncharge)
40 map->ops->map_local_storage_uncharge(smap, owner, size);
41}
42
43static struct bpf_local_storage __rcu **
44owner_storage(struct bpf_local_storage_map *smap, void *owner)
45{
46 struct bpf_map *map = &smap->map;
47
48 return map->ops->map_owner_storage_ptr(owner);
49}
50
51static bool selem_linked_to_storage(const struct bpf_local_storage_elem *selem)
52{
53 return !hlist_unhashed(&selem->snode);
54}
55
56static bool selem_linked_to_map(const struct bpf_local_storage_elem *selem)
57{
58 return !hlist_unhashed(&selem->map_node);
59}
60
61struct bpf_local_storage_elem *
62bpf_selem_alloc(struct bpf_local_storage_map *smap, void *owner,
63 void *value, bool charge_mem)
64{
65 struct bpf_local_storage_elem *selem;
66
67 if (charge_mem && mem_charge(smap, owner, smap->elem_size))
68 return NULL;
69
70 selem = kzalloc(smap->elem_size, GFP_ATOMIC | __GFP_NOWARN);
71 if (selem) {
72 if (value)
73 memcpy(SDATA(selem)->data, value, smap->map.value_size);
74 return selem;
75 }
76
77 if (charge_mem)
78 mem_uncharge(smap, owner, smap->elem_size);
79
80 return NULL;
81}
82
83/* local_storage->lock must be held and selem->local_storage == local_storage.
84 * The caller must ensure selem->smap is still valid to be
85 * dereferenced for its smap->elem_size and smap->cache_idx.
86 */
87bool bpf_selem_unlink_storage_nolock(struct bpf_local_storage *local_storage,
88 struct bpf_local_storage_elem *selem,
89 bool uncharge_mem)
90{
91 struct bpf_local_storage_map *smap;
92 bool free_local_storage;
93 void *owner;
94
95 smap = rcu_dereference(SDATA(selem)->smap);
96 owner = local_storage->owner;
97
98 /* All uncharging on the owner must be done first.
99 * The owner may be freed once the last selem is unlinked
100 * from local_storage.
101 */
102 if (uncharge_mem)
103 mem_uncharge(smap, owner, smap->elem_size);
104
105 free_local_storage = hlist_is_singular_node(&selem->snode,
106 &local_storage->list);
107 if (free_local_storage) {
108 mem_uncharge(smap, owner, sizeof(struct bpf_local_storage));
109 local_storage->owner = NULL;
110
111 /* After this RCU_INIT, owner may be freed and cannot be used */
112 RCU_INIT_POINTER(*owner_storage(smap, owner), NULL);
113
114 /* local_storage is not freed now. local_storage->lock is
115 * still held and raw_spin_unlock_bh(&local_storage->lock)
116 * will be done by the caller.
117 *
118 * Although the unlock will be done under
119 * rcu_read_lock(), it is more intutivie to
120 * read if kfree_rcu(local_storage, rcu) is done
121 * after the raw_spin_unlock_bh(&local_storage->lock).
122 *
123 * Hence, a "bool free_local_storage" is returned
124 * to the caller which then calls the kfree_rcu()
125 * after unlock.
126 */
127 }
128 hlist_del_init_rcu(&selem->snode);
129 if (rcu_access_pointer(local_storage->cache[smap->cache_idx]) ==
130 SDATA(selem))
131 RCU_INIT_POINTER(local_storage->cache[smap->cache_idx], NULL);
132
133 kfree_rcu(selem, rcu);
134
135 return free_local_storage;
136}
137
138static void __bpf_selem_unlink_storage(struct bpf_local_storage_elem *selem)
139{
140 struct bpf_local_storage *local_storage;
141 bool free_local_storage = false;
142
143 if (unlikely(!selem_linked_to_storage(selem)))
144 /* selem has already been unlinked from sk */
145 return;
146
147 local_storage = rcu_dereference(selem->local_storage);
148 raw_spin_lock_bh(&local_storage->lock);
149 if (likely(selem_linked_to_storage(selem)))
150 free_local_storage = bpf_selem_unlink_storage_nolock(
151 local_storage, selem, true);
152 raw_spin_unlock_bh(&local_storage->lock);
153
154 if (free_local_storage)
155 kfree_rcu(local_storage, rcu);
156}
157
158void bpf_selem_link_storage_nolock(struct bpf_local_storage *local_storage,
159 struct bpf_local_storage_elem *selem)
160{
161 RCU_INIT_POINTER(selem->local_storage, local_storage);
162 hlist_add_head_rcu(&selem->snode, &local_storage->list);
163}
164
165void bpf_selem_unlink_map(struct bpf_local_storage_elem *selem)
166{
167 struct bpf_local_storage_map *smap;
168 struct bpf_local_storage_map_bucket *b;
169
170 if (unlikely(!selem_linked_to_map(selem)))
171 /* selem has already be unlinked from smap */
172 return;
173
174 smap = rcu_dereference(SDATA(selem)->smap);
175 b = select_bucket(smap, selem);
176 raw_spin_lock_bh(&b->lock);
177 if (likely(selem_linked_to_map(selem)))
178 hlist_del_init_rcu(&selem->map_node);
179 raw_spin_unlock_bh(&b->lock);
180}
181
182void bpf_selem_link_map(struct bpf_local_storage_map *smap,
183 struct bpf_local_storage_elem *selem)
184{
185 struct bpf_local_storage_map_bucket *b = select_bucket(smap, selem);
186
187 raw_spin_lock_bh(&b->lock);
188 RCU_INIT_POINTER(SDATA(selem)->smap, smap);
189 hlist_add_head_rcu(&selem->map_node, &b->list);
190 raw_spin_unlock_bh(&b->lock);
191}
192
193void bpf_selem_unlink(struct bpf_local_storage_elem *selem)
194{
195 /* Always unlink from map before unlinking from local_storage
196 * because selem will be freed after successfully unlinked from
197 * the local_storage.
198 */
199 bpf_selem_unlink_map(selem);
200 __bpf_selem_unlink_storage(selem);
201}
202
203struct bpf_local_storage_data *
204bpf_local_storage_lookup(struct bpf_local_storage *local_storage,
205 struct bpf_local_storage_map *smap,
206 bool cacheit_lockit)
207{
208 struct bpf_local_storage_data *sdata;
209 struct bpf_local_storage_elem *selem;
210
211 /* Fast path (cache hit) */
212 sdata = rcu_dereference(local_storage->cache[smap->cache_idx]);
213 if (sdata && rcu_access_pointer(sdata->smap) == smap)
214 return sdata;
215
216 /* Slow path (cache miss) */
217 hlist_for_each_entry_rcu(selem, &local_storage->list, snode)
218 if (rcu_access_pointer(SDATA(selem)->smap) == smap)
219 break;
220
221 if (!selem)
222 return NULL;
223
224 sdata = SDATA(selem);
225 if (cacheit_lockit) {
226 /* spinlock is needed to avoid racing with the
227 * parallel delete. Otherwise, publishing an already
228 * deleted sdata to the cache will become a use-after-free
229 * problem in the next bpf_local_storage_lookup().
230 */
231 raw_spin_lock_bh(&local_storage->lock);
232 if (selem_linked_to_storage(selem))
233 rcu_assign_pointer(local_storage->cache[smap->cache_idx],
234 sdata);
235 raw_spin_unlock_bh(&local_storage->lock);
236 }
237
238 return sdata;
239}
240
241static int check_flags(const struct bpf_local_storage_data *old_sdata,
242 u64 map_flags)
243{
244 if (old_sdata && (map_flags & ~BPF_F_LOCK) == BPF_NOEXIST)
245 /* elem already exists */
246 return -EEXIST;
247
248 if (!old_sdata && (map_flags & ~BPF_F_LOCK) == BPF_EXIST)
249 /* elem doesn't exist, cannot update it */
250 return -ENOENT;
251
252 return 0;
253}
254
255int bpf_local_storage_alloc(void *owner,
256 struct bpf_local_storage_map *smap,
257 struct bpf_local_storage_elem *first_selem)
258{
259 struct bpf_local_storage *prev_storage, *storage;
260 struct bpf_local_storage **owner_storage_ptr;
261 int err;
262
263 err = mem_charge(smap, owner, sizeof(*storage));
264 if (err)
265 return err;
266
267 storage = kzalloc(sizeof(*storage), GFP_ATOMIC | __GFP_NOWARN);
268 if (!storage) {
269 err = -ENOMEM;
270 goto uncharge;
271 }
272
273 INIT_HLIST_HEAD(&storage->list);
274 raw_spin_lock_init(&storage->lock);
275 storage->owner = owner;
276
277 bpf_selem_link_storage_nolock(storage, first_selem);
278 bpf_selem_link_map(smap, first_selem);
279
280 owner_storage_ptr =
281 (struct bpf_local_storage **)owner_storage(smap, owner);
282 /* Publish storage to the owner.
283 * Instead of using any lock of the kernel object (i.e. owner),
284 * cmpxchg will work with any kernel object regardless what
285 * the running context is, bh, irq...etc.
286 *
287 * From now on, the owner->storage pointer (e.g. sk->sk_bpf_storage)
288 * is protected by the storage->lock. Hence, when freeing
289 * the owner->storage, the storage->lock must be held before
290 * setting owner->storage ptr to NULL.
291 */
292 prev_storage = cmpxchg(owner_storage_ptr, NULL, storage);
293 if (unlikely(prev_storage)) {
294 bpf_selem_unlink_map(first_selem);
295 err = -EAGAIN;
296 goto uncharge;
297
298 /* Note that even first_selem was linked to smap's
299 * bucket->list, first_selem can be freed immediately
300 * (instead of kfree_rcu) because
301 * bpf_local_storage_map_free() does a
302 * synchronize_rcu() before walking the bucket->list.
303 * Hence, no one is accessing selem from the
304 * bucket->list under rcu_read_lock().
305 */
306 }
307
308 return 0;
309
310uncharge:
311 kfree(storage);
312 mem_uncharge(smap, owner, sizeof(*storage));
313 return err;
314}
315
316/* sk cannot be going away because it is linking new elem
317 * to sk->sk_bpf_storage. (i.e. sk->sk_refcnt cannot be 0).
318 * Otherwise, it will become a leak (and other memory issues
319 * during map destruction).
320 */
321struct bpf_local_storage_data *
322bpf_local_storage_update(void *owner, struct bpf_local_storage_map *smap,
323 void *value, u64 map_flags)
324{
325 struct bpf_local_storage_data *old_sdata = NULL;
326 struct bpf_local_storage_elem *selem;
327 struct bpf_local_storage *local_storage;
328 int err;
329
330 /* BPF_EXIST and BPF_NOEXIST cannot be both set */
331 if (unlikely((map_flags & ~BPF_F_LOCK) > BPF_EXIST) ||
332 /* BPF_F_LOCK can only be used in a value with spin_lock */
333 unlikely((map_flags & BPF_F_LOCK) &&
334 !map_value_has_spin_lock(&smap->map)))
335 return ERR_PTR(-EINVAL);
336
337 local_storage = rcu_dereference(*owner_storage(smap, owner));
338 if (!local_storage || hlist_empty(&local_storage->list)) {
339 /* Very first elem for the owner */
340 err = check_flags(NULL, map_flags);
341 if (err)
342 return ERR_PTR(err);
343
344 selem = bpf_selem_alloc(smap, owner, value, true);
345 if (!selem)
346 return ERR_PTR(-ENOMEM);
347
348 err = bpf_local_storage_alloc(owner, smap, selem);
349 if (err) {
350 kfree(selem);
351 mem_uncharge(smap, owner, smap->elem_size);
352 return ERR_PTR(err);
353 }
354
355 return SDATA(selem);
356 }
357
358 if ((map_flags & BPF_F_LOCK) && !(map_flags & BPF_NOEXIST)) {
359 /* Hoping to find an old_sdata to do inline update
360 * such that it can avoid taking the local_storage->lock
361 * and changing the lists.
362 */
363 old_sdata =
364 bpf_local_storage_lookup(local_storage, smap, false);
365 err = check_flags(old_sdata, map_flags);
366 if (err)
367 return ERR_PTR(err);
368 if (old_sdata && selem_linked_to_storage(SELEM(old_sdata))) {
369 copy_map_value_locked(&smap->map, old_sdata->data,
370 value, false);
371 return old_sdata;
372 }
373 }
374
375 raw_spin_lock_bh(&local_storage->lock);
376
377 /* Recheck local_storage->list under local_storage->lock */
378 if (unlikely(hlist_empty(&local_storage->list))) {
379 /* A parallel del is happening and local_storage is going
380 * away. It has just been checked before, so very
381 * unlikely. Return instead of retry to keep things
382 * simple.
383 */
384 err = -EAGAIN;
385 goto unlock_err;
386 }
387
388 old_sdata = bpf_local_storage_lookup(local_storage, smap, false);
389 err = check_flags(old_sdata, map_flags);
390 if (err)
391 goto unlock_err;
392
393 if (old_sdata && (map_flags & BPF_F_LOCK)) {
394 copy_map_value_locked(&smap->map, old_sdata->data, value,
395 false);
396 selem = SELEM(old_sdata);
397 goto unlock;
398 }
399
400 /* local_storage->lock is held. Hence, we are sure
401 * we can unlink and uncharge the old_sdata successfully
402 * later. Hence, instead of charging the new selem now
403 * and then uncharge the old selem later (which may cause
404 * a potential but unnecessary charge failure), avoid taking
405 * a charge at all here (the "!old_sdata" check) and the
406 * old_sdata will not be uncharged later during
407 * bpf_selem_unlink_storage_nolock().
408 */
409 selem = bpf_selem_alloc(smap, owner, value, !old_sdata);
410 if (!selem) {
411 err = -ENOMEM;
412 goto unlock_err;
413 }
414
415 /* First, link the new selem to the map */
416 bpf_selem_link_map(smap, selem);
417
418 /* Second, link (and publish) the new selem to local_storage */
419 bpf_selem_link_storage_nolock(local_storage, selem);
420
421 /* Third, remove old selem, SELEM(old_sdata) */
422 if (old_sdata) {
423 bpf_selem_unlink_map(SELEM(old_sdata));
424 bpf_selem_unlink_storage_nolock(local_storage, SELEM(old_sdata),
425 false);
426 }
427
428unlock:
429 raw_spin_unlock_bh(&local_storage->lock);
430 return SDATA(selem);
431
432unlock_err:
433 raw_spin_unlock_bh(&local_storage->lock);
434 return ERR_PTR(err);
435}
436
437u16 bpf_local_storage_cache_idx_get(struct bpf_local_storage_cache *cache)
438{
439 u64 min_usage = U64_MAX;
440 u16 i, res = 0;
441
442 spin_lock(&cache->idx_lock);
443
444 for (i = 0; i < BPF_LOCAL_STORAGE_CACHE_SIZE; i++) {
445 if (cache->idx_usage_counts[i] < min_usage) {
446 min_usage = cache->idx_usage_counts[i];
447 res = i;
448
449 /* Found a free cache_idx */
450 if (!min_usage)
451 break;
452 }
453 }
454 cache->idx_usage_counts[res]++;
455
456 spin_unlock(&cache->idx_lock);
457
458 return res;
459}
460
461void bpf_local_storage_cache_idx_free(struct bpf_local_storage_cache *cache,
462 u16 idx)
463{
464 spin_lock(&cache->idx_lock);
465 cache->idx_usage_counts[idx]--;
466 spin_unlock(&cache->idx_lock);
467}
468
469void bpf_local_storage_map_free(struct bpf_local_storage_map *smap)
470{
471 struct bpf_local_storage_elem *selem;
472 struct bpf_local_storage_map_bucket *b;
473 unsigned int i;
474
475 /* Note that this map might be concurrently cloned from
476 * bpf_sk_storage_clone. Wait for any existing bpf_sk_storage_clone
477 * RCU read section to finish before proceeding. New RCU
478 * read sections should be prevented via bpf_map_inc_not_zero.
479 */
480 synchronize_rcu();
481
482 /* bpf prog and the userspace can no longer access this map
483 * now. No new selem (of this map) can be added
484 * to the owner->storage or to the map bucket's list.
485 *
486 * The elem of this map can be cleaned up here
487 * or when the storage is freed e.g.
488 * by bpf_sk_storage_free() during __sk_destruct().
489 */
490 for (i = 0; i < (1U << smap->bucket_log); i++) {
491 b = &smap->buckets[i];
492
493 rcu_read_lock();
494 /* No one is adding to b->list now */
495 while ((selem = hlist_entry_safe(
496 rcu_dereference_raw(hlist_first_rcu(&b->list)),
497 struct bpf_local_storage_elem, map_node))) {
498 bpf_selem_unlink(selem);
499 cond_resched_rcu();
500 }
501 rcu_read_unlock();
502 }
503
504 /* While freeing the storage we may still need to access the map.
505 *
506 * e.g. when bpf_sk_storage_free() has unlinked selem from the map
507 * which then made the above while((selem = ...)) loop
508 * exit immediately.
509 *
510 * However, while freeing the storage one still needs to access the
511 * smap->elem_size to do the uncharging in
512 * bpf_selem_unlink_storage_nolock().
513 *
514 * Hence, wait another rcu grace period for the storage to be freed.
515 */
516 synchronize_rcu();
517
518 kvfree(smap->buckets);
519 kfree(smap);
520}
521
522int bpf_local_storage_map_alloc_check(union bpf_attr *attr)
523{
524 if (attr->map_flags & ~BPF_LOCAL_STORAGE_CREATE_FLAG_MASK ||
525 !(attr->map_flags & BPF_F_NO_PREALLOC) ||
526 attr->max_entries ||
527 attr->key_size != sizeof(int) || !attr->value_size ||
528 /* Enforce BTF for userspace sk dumping */
529 !attr->btf_key_type_id || !attr->btf_value_type_id)
530 return -EINVAL;
531
532 if (!bpf_capable())
533 return -EPERM;
534
535 if (attr->value_size > BPF_LOCAL_STORAGE_MAX_VALUE_SIZE)
536 return -E2BIG;
537
538 return 0;
539}
540
541struct bpf_local_storage_map *bpf_local_storage_map_alloc(union bpf_attr *attr)
542{
543 struct bpf_local_storage_map *smap;
544 unsigned int i;
545 u32 nbuckets;
546 u64 cost;
547 int ret;
548
549 smap = kzalloc(sizeof(*smap), GFP_USER | __GFP_NOWARN);
550 if (!smap)
551 return ERR_PTR(-ENOMEM);
552 bpf_map_init_from_attr(&smap->map, attr);
553
554 nbuckets = roundup_pow_of_two(num_possible_cpus());
555 /* Use at least 2 buckets, select_bucket() is undefined behavior with 1 bucket */
556 nbuckets = max_t(u32, 2, nbuckets);
557 smap->bucket_log = ilog2(nbuckets);
558 cost = sizeof(*smap->buckets) * nbuckets + sizeof(*smap);
559
560 ret = bpf_map_charge_init(&smap->map.memory, cost);
561 if (ret < 0) {
562 kfree(smap);
563 return ERR_PTR(ret);
564 }
565
566 smap->buckets = kvcalloc(sizeof(*smap->buckets), nbuckets,
567 GFP_USER | __GFP_NOWARN);
568 if (!smap->buckets) {
569 bpf_map_charge_finish(&smap->map.memory);
570 kfree(smap);
571 return ERR_PTR(-ENOMEM);
572 }
573
574 for (i = 0; i < nbuckets; i++) {
575 INIT_HLIST_HEAD(&smap->buckets[i].list);
576 raw_spin_lock_init(&smap->buckets[i].lock);
577 }
578
579 smap->elem_size =
580 sizeof(struct bpf_local_storage_elem) + attr->value_size;
581
582 return smap;
583}
584
585int bpf_local_storage_map_check_btf(const struct bpf_map *map,
586 const struct btf *btf,
587 const struct btf_type *key_type,
588 const struct btf_type *value_type)
589{
590 u32 int_data;
591
592 if (BTF_INFO_KIND(key_type->info) != BTF_KIND_INT)
593 return -EINVAL;
594
595 int_data = *(u32 *)(key_type + 1);
596 if (BTF_INT_BITS(int_data) != 32 || BTF_INT_OFFSET(int_data))
597 return -EINVAL;
598
599 return 0;
600}
kernel/bpf/bpf_lsm.c+20−1
@@ -11,6 +11,8 @@ · 10 unmodified lines
11#include <linux/bpf_lsm.h>11#include <linux/bpf_lsm.h>
12#include <linux/kallsyms.h>12#include <linux/kallsyms.h>
13#include <linux/bpf_verifier.h>13#include <linux/bpf_verifier.h>
14#include <net/bpf_sk_storage.h>
15#include <linux/bpf_local_storage.h>
14 16
15/* For every LSM hook that allows attachment of BPF programs, declare a nop17/* For every LSM hook that allows attachment of BPF programs, declare a nop
16 * function where a BPF program can be attached.18 * function where a BPF program can be attached.
@@ -45,10 +47,27 @@ int bpf_lsm_verify_prog(struct bpf_verifier_log *vlog, · 28 unmodified lines
45 return 0;47 return 0;
46}48}
47 49
50static const struct bpf_func_proto *
51bpf_lsm_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
52{
53 switch (func_id) {
54 case BPF_FUNC_inode_storage_get:
55 return &bpf_inode_storage_get_proto;
56 case BPF_FUNC_inode_storage_delete:
57 return &bpf_inode_storage_delete_proto;
58 case BPF_FUNC_sk_storage_get:
59 return &bpf_sk_storage_get_proto;
60 case BPF_FUNC_sk_storage_delete:
61 return &bpf_sk_storage_delete_proto;
62 default:
63 return tracing_prog_func_proto(func_id, prog);
64 }
65}
66
48const struct bpf_prog_ops lsm_prog_ops = {67const struct bpf_prog_ops lsm_prog_ops = {
49};68};
50 69
51const struct bpf_verifier_ops lsm_verifier_ops = {70const struct bpf_verifier_ops lsm_verifier_ops = {
52 .get_func_proto = tracing_prog_func_proto,71 .get_func_proto = bpf_lsm_func_proto,
53 .is_valid_access = btf_ctx_access,72 .is_valid_access = btf_ctx_access,
54};73};
kernel/bpf/bpf_struct_ops.c+2−4
@@ -298,8 +298,7 @@ static int check_zero_holes(const struct btf_type *t, void *data) · 297 unmodified lines
298 return -EINVAL;298 return -EINVAL;
299 299
300 mtype = btf_type_by_id(btf_vmlinux, member->type);300 mtype = btf_type_by_id(btf_vmlinux, member->type);
301 mtype = btf_resolve_size(btf_vmlinux, mtype, &msize,301 mtype = btf_resolve_size(btf_vmlinux, mtype, &msize);
302 NULL, NULL);
303 if (IS_ERR(mtype))302 if (IS_ERR(mtype))
304 return PTR_ERR(mtype);303 return PTR_ERR(mtype);
305 prev_mend = moff + msize;304 prev_mend = moff + msize;
@@ -396,8 +395,7 @@ static int bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key, · 90 unmodified lines
396 u32 msize;395 u32 msize;
397 396
398 mtype = btf_type_by_id(btf_vmlinux, member->type);397 mtype = btf_type_by_id(btf_vmlinux, member->type);
399 mtype = btf_resolve_size(btf_vmlinux, mtype, &msize,398 mtype = btf_resolve_size(btf_vmlinux, mtype, &msize);
400 NULL, NULL);
401 if (IS_ERR(mtype)) {399 if (IS_ERR(mtype)) {
402 err = PTR_ERR(mtype);400 err = PTR_ERR(mtype);
403 goto reset_unlock;401 goto reset_unlock;
kernel/bpf/btf.c+1054−167
@@ -21,6 +21,8 @@ · 20 unmodified lines
21#include <linux/btf_ids.h>21#include <linux/btf_ids.h>
22#include <linux/skmsg.h>22#include <linux/skmsg.h>
23#include <linux/perf_event.h>23#include <linux/perf_event.h>
24#include <linux/bsearch.h>
25#include <linux/btf_ids.h>
24#include <net/sock.h>26#include <net/sock.h>
25 27
26/* BTF (BPF Type Format) is the meta data format which describes28/* BTF (BPF Type Format) is the meta data format which describes
@@ -186,11 +188,6 @@ · 159 unmodified lines
186 i < btf_type_vlen(struct_type); \188 i < btf_type_vlen(struct_type); \
187 i++, member++)189 i++, member++)
188 190
189#define for_each_vsi(i, struct_type, member) \
190 for (i = 0, member = btf_type_var_secinfo(struct_type); \
191 i < btf_type_vlen(struct_type); \
192 i++, member++)
193
194#define for_each_vsi_from(i, from, struct_type, member) \191#define for_each_vsi_from(i, from, struct_type, member) \
195 for (i = from, member = btf_type_var_secinfo(struct_type) + from; \192 for (i = from, member = btf_type_var_secinfo(struct_type) + from; \
196 i < btf_type_vlen(struct_type); \193 i < btf_type_vlen(struct_type); \
@@ -282,6 +279,91 @@ static const char *btf_type_str(const struct btf_type *t) · 85 unmodified lines
282 return btf_kind_str[BTF_INFO_KIND(t->info)];279 return btf_kind_str[BTF_INFO_KIND(t->info)];
283}280}
284 281
282/* Chunk size we use in safe copy of data to be shown. */
283#define BTF_SHOW_OBJ_SAFE_SIZE 32
284
285/*
286 * This is the maximum size of a base type value (equivalent to a
287 * 128-bit int); if we are at the end of our safe buffer and have
288 * less than 16 bytes space we can't be assured of being able
289 * to copy the next type safely, so in such cases we will initiate
290 * a new copy.
291 */
292#define BTF_SHOW_OBJ_BASE_TYPE_SIZE 16
293
294/* Type name size */
295#define BTF_SHOW_NAME_SIZE 80
296
297/*
298 * Common data to all BTF show operations. Private show functions can add
299 * their own data to a structure containing a struct btf_show and consult it
300 * in the show callback. See btf_type_show() below.
301 *
302 * One challenge with showing nested data is we want to skip 0-valued
303 * data, but in order to figure out whether a nested object is all zeros
304 * we need to walk through it. As a result, we need to make two passes
305 * when handling structs, unions and arrays; the first path simply looks
306 * for nonzero data, while the second actually does the display. The first
307 * pass is signalled by show->state.depth_check being set, and if we
308 * encounter a non-zero value we set show->state.depth_to_show to
309 * the depth at which we encountered it. When we have completed the
310 * first pass, we will know if anything needs to be displayed if
311 * depth_to_show > depth. See btf_[struct,array]_show() for the
312 * implementation of this.
313 *
314 * Another problem is we want to ensure the data for display is safe to
315 * access. To support this, the anonymous "struct {} obj" tracks the data
316 * object and our safe copy of it. We copy portions of the data needed
317 * to the object "copy" buffer, but because its size is limited to
318 * BTF_SHOW_OBJ_COPY_LEN bytes, multiple copies may be required as we
319 * traverse larger objects for display.
320 *
321 * The various data type show functions all start with a call to
322 * btf_show_start_type() which returns a pointer to the safe copy
323 * of the data needed (or if BTF_SHOW_UNSAFE is specified, to the
324 * raw data itself). btf_show_obj_safe() is responsible for
325 * using copy_from_kernel_nofault() to update the safe data if necessary
326 * as we traverse the object's data. skbuff-like semantics are
327 * used:
328 *
329 * - obj.head points to the start of the toplevel object for display
330 * - obj.size is the size of the toplevel object
331 * - obj.data points to the current point in the original data at
332 * which our safe data starts. obj.data will advance as we copy
333 * portions of the data.
334 *
335 * In most cases a single copy will suffice, but larger data structures
336 * such as "struct task_struct" will require many copies. The logic in
337 * btf_show_obj_safe() handles the logic that determines if a new
338 * copy_from_kernel_nofault() is needed.
339 */
340struct btf_show {
341 u64 flags;
342 void *target; /* target of show operation (seq file, buffer) */
343 void (*showfn)(struct btf_show *show, const char *fmt, va_list args);
344 const struct btf *btf;
345 /* below are used during iteration */
346 struct {
347 u8 depth;
348 u8 depth_to_show;
349 u8 depth_check;
350 u8 array_member:1,
351 array_terminated:1;
352 u16 array_encoding;
353 u32 type_id;
354 int status; /* non-zero for error */
355 const struct btf_type *type;
356 const struct btf_member *member;
357 char name[BTF_SHOW_NAME_SIZE]; /* space for member name/type */
358 } state;
359 struct {
360 u32 size;
361 void *head;
362 void *data;
363 u8 safe[BTF_SHOW_OBJ_SAFE_SIZE];
364 } obj;
365};
366
285struct btf_kind_operations {367struct btf_kind_operations {
286 s32 (*check_meta)(struct btf_verifier_env *env,368 s32 (*check_meta)(struct btf_verifier_env *env,
287 const struct btf_type *t,369 const struct btf_type *t,
@@ -298,9 +380,9 @@ struct btf_kind_operations { · 10 unmodified lines
298 const struct btf_type *member_type);380 const struct btf_type *member_type);
299 void (*log_details)(struct btf_verifier_env *env,381 void (*log_details)(struct btf_verifier_env *env,
300 const struct btf_type *t);382 const struct btf_type *t);
301 void (*seq_show)(const struct btf *btf, const struct btf_type *t,383 void (*show)(const struct btf *btf, const struct btf_type *t,
302 u32 type_id, void *data, u8 bits_offsets,384 u32 type_id, void *data, u8 bits_offsets,
303 struct seq_file *m);385 struct btf_show *show);
304};386};
305 387
306static const struct btf_kind_operations * const kind_ops[NR_BTF_KINDS];388static const struct btf_kind_operations * const kind_ops[NR_BTF_KINDS];
@@ -353,16 +435,6 @@ static bool btf_type_nosize_or_null(const struct btf_type *t) · 46 unmodified lines
353 return !t || btf_type_nosize(t);435 return !t || btf_type_nosize(t);
354}436}
355 437
356/* union is only a special case of struct:
357 * all its offsetof(member) == 0
358 */
359static bool btf_type_is_struct(const struct btf_type *t)
360{
361 u8 kind = BTF_INFO_KIND(t->info);
362
363 return kind == BTF_KIND_STRUCT || kind == BTF_KIND_UNION;
364}
365
366static bool __btf_type_is_struct(const struct btf_type *t)438static bool __btf_type_is_struct(const struct btf_type *t)
367{439{
368 return BTF_INFO_KIND(t->info) == BTF_KIND_STRUCT;440 return BTF_INFO_KIND(t->info) == BTF_KIND_STRUCT;
@@ -373,11 +445,6 @@ static bool btf_type_is_array(const struct btf_type *t) · 4 unmodified lines
373 return BTF_INFO_KIND(t->info) == BTF_KIND_ARRAY;445 return BTF_INFO_KIND(t->info) == BTF_KIND_ARRAY;
374}446}
375 447
376static bool btf_type_is_var(const struct btf_type *t)
377{
378 return BTF_INFO_KIND(t->info) == BTF_KIND_VAR;
379}
380
381static bool btf_type_is_datasec(const struct btf_type *t)448static bool btf_type_is_datasec(const struct btf_type *t)
382{449{
383 return BTF_INFO_KIND(t->info) == BTF_KIND_DATASEC;450 return BTF_INFO_KIND(t->info) == BTF_KIND_DATASEC;
@@ -526,11 +593,6 @@ static const struct btf_var *btf_type_var(const struct btf_type *t) · 142 unmodified lines
526 return (const struct btf_var *)(t + 1);593 return (const struct btf_var *)(t + 1);
527}594}
528 595
529static const struct btf_var_secinfo *btf_type_var_secinfo(const struct btf_type *t)
530{
531 return (const struct btf_var_secinfo *)(t + 1);
532}
533
534static const struct btf_kind_operations *btf_type_ops(const struct btf_type *t)596static const struct btf_kind_operations *btf_type_ops(const struct btf_type *t)
535{597{
536 return kind_ops[BTF_INFO_KIND(t->info)];598 return kind_ops[BTF_INFO_KIND(t->info)];
@@ -677,6 +739,488 @@ bool btf_member_is_reg_int(const struct btf *btf, const struct btf_type *s, · 140 unmodified lines
677 return true;739 return true;
678}740}
679 741
742/* Similar to btf_type_skip_modifiers() but does not skip typedefs. */
743static const struct btf_type *btf_type_skip_qualifiers(const struct btf *btf,
744 u32 id)
745{
746 const struct btf_type *t = btf_type_by_id(btf, id);
747
748 while (btf_type_is_modifier(t) &&
749 BTF_INFO_KIND(t->info) != BTF_KIND_TYPEDEF) {
750 id = t->type;
751 t = btf_type_by_id(btf, t->type);
752 }
753
754 return t;
755}
756
757#define BTF_SHOW_MAX_ITER 10
758
759#define BTF_KIND_BIT(kind) (1ULL << kind)
760
761/*
762 * Populate show->state.name with type name information.
763 * Format of type name is
764 *
765 * [.member_name = ] (type_name)
766 */
767static const char *btf_show_name(struct btf_show *show)
768{
769 /* BTF_MAX_ITER array suffixes "[]" */
770 const char *array_suffixes = "[][][][][][][][][][]";
771 const char *array_suffix = &array_suffixes[strlen(array_suffixes)];
772 /* BTF_MAX_ITER pointer suffixes "*" */
773 const char *ptr_suffixes = "**********";
774 const char *ptr_suffix = &ptr_suffixes[strlen(ptr_suffixes)];
775 const char *name = NULL, *prefix = "", *parens = "";
776 const struct btf_member *m = show->state.member;
777 const struct btf_type *t = show->state.type;
778 const struct btf_array *array;
779 u32 id = show->state.type_id;
780 const char *member = NULL;
781 bool show_member = false;
782 u64 kinds = 0;
783 int i;
784
785 show->state.name[0] = '\0';
786
787 /*
788 * Don't show type name if we're showing an array member;
789 * in that case we show the array type so don't need to repeat
790 * ourselves for each member.
791 */
792 if (show->state.array_member)
793 return "";
794
795 /* Retrieve member name, if any. */
796 if (m) {
797 member = btf_name_by_offset(show->btf, m->name_off);
798 show_member = strlen(member) > 0;
799 id = m->type;
800 }
801
802 /*
803 * Start with type_id, as we have resolved the struct btf_type *
804 * via btf_modifier_show() past the parent typedef to the child
805 * struct, int etc it is defined as. In such cases, the type_id
806 * still represents the starting type while the struct btf_type *
807 * in our show->state points at the resolved type of the typedef.
808 */
809 t = btf_type_by_id(show->btf, id);
810 if (!t)
811 return "";
812
813 /*
814 * The goal here is to build up the right number of pointer and
815 * array suffixes while ensuring the type name for a typedef
816 * is represented. Along the way we accumulate a list of
817 * BTF kinds we have encountered, since these will inform later
818 * display; for example, pointer types will not require an
819 * opening "{" for struct, we will just display the pointer value.
820 *
821 * We also want to accumulate the right number of pointer or array
822 * indices in the format string while iterating until we get to
823 * the typedef/pointee/array member target type.
824 *
825 * We start by pointing at the end of pointer and array suffix
826 * strings; as we accumulate pointers and arrays we move the pointer
827 * or array string backwards so it will show the expected number of
828 * '*' or '[]' for the type. BTF_SHOW_MAX_ITER of nesting of pointers
829 * and/or arrays and typedefs are supported as a precaution.
830 *
831 * We also want to get typedef name while proceeding to resolve
832 * type it points to so that we can add parentheses if it is a
833 * "typedef struct" etc.
834 */
835 for (i = 0; i < BTF_SHOW_MAX_ITER; i++) {
836
837 switch (BTF_INFO_KIND(t->info)) {
838 case BTF_KIND_TYPEDEF:
839 if (!name)
840 name = btf_name_by_offset(show->btf,
841 t->name_off);
842 kinds |= BTF_KIND_BIT(BTF_KIND_TYPEDEF);
843 id = t->type;
844 break;
845 case BTF_KIND_ARRAY:
846 kinds |= BTF_KIND_BIT(BTF_KIND_ARRAY);
847 parens = "[";
848 if (!t)
849 return "";
850 array = btf_type_array(t);
851 if (array_suffix > array_suffixes)
852 array_suffix -= 2;
853 id = array->type;
854 break;
855 case BTF_KIND_PTR:
856 kinds |= BTF_KIND_BIT(BTF_KIND_PTR);
857 if (ptr_suffix > ptr_suffixes)
858 ptr_suffix -= 1;
859 id = t->type;
860 break;
861 default:
862 id = 0;
863 break;
864 }
865 if (!id)
866 break;
867 t = btf_type_skip_qualifiers(show->btf, id);
868 }
869 /* We may not be able to represent this type; bail to be safe */
870 if (i == BTF_SHOW_MAX_ITER)
871 return "";
872
873 if (!name)
874 name = btf_name_by_offset(show->btf, t->name_off);
875
876 switch (BTF_INFO_KIND(t->info)) {
877 case BTF_KIND_STRUCT:
878 case BTF_KIND_UNION:
879 prefix = BTF_INFO_KIND(t->info) == BTF_KIND_STRUCT ?
880 "struct" : "union";
881 /* if it's an array of struct/union, parens is already set */
882 if (!(kinds & (BTF_KIND_BIT(BTF_KIND_ARRAY))))
883 parens = "{";
884 break;
885 case BTF_KIND_ENUM:
886 prefix = "enum";
887 break;
888 default:
889 break;
890 }
891
892 /* pointer does not require parens */
893 if (kinds & BTF_KIND_BIT(BTF_KIND_PTR))
894 parens = "";
895 /* typedef does not require struct/union/enum prefix */
896 if (kinds & BTF_KIND_BIT(BTF_KIND_TYPEDEF))
897 prefix = "";
898
899 if (!name)
900 name = "";
901
902 /* Even if we don't want type name info, we want parentheses etc */
903 if (show->flags & BTF_SHOW_NONAME)
904 snprintf(show->state.name, sizeof(show->state.name), "%s",
905 parens);
906 else
907 snprintf(show->state.name, sizeof(show->state.name),
908 "%s%s%s(%s%s%s%s%s%s)%s",
909 /* first 3 strings comprise ".member = " */
910 show_member ? "." : "",
911 show_member ? member : "",
912 show_member ? " = " : "",
913 /* ...next is our prefix (struct, enum, etc) */
914 prefix,
915 strlen(prefix) > 0 && strlen(name) > 0 ? " " : "",
916 /* ...this is the type name itself */
917 name,
918 /* ...suffixed by the appropriate '*', '[]' suffixes */
919 strlen(ptr_suffix) > 0 ? " " : "", ptr_suffix,
920 array_suffix, parens);
921
922 return show->state.name;
923}
924
925static const char *__btf_show_indent(struct btf_show *show)
926{
927 const char *indents = " ";
928 const char *indent = &indents[strlen(indents)];
929
930 if ((indent - show->state.depth) >= indents)
931 return indent - show->state.depth;
932 return indents;
933}
934
935static const char *btf_show_indent(struct btf_show *show)
936{
937 return show->flags & BTF_SHOW_COMPACT ? "" : __btf_show_indent(show);
938}
939
940static const char *btf_show_newline(struct btf_show *show)
941{
942 return show->flags & BTF_SHOW_COMPACT ? "" : "\n";
943}
944
945static const char *btf_show_delim(struct btf_show *show)
946{
947 if (show->state.depth == 0)
948 return "";
949
950 if ((show->flags & BTF_SHOW_COMPACT) && show->state.type &&
951 BTF_INFO_KIND(show->state.type->info) == BTF_KIND_UNION)
952 return "|";
953
954 return ",";
955}
956
957__printf(2, 3) static void btf_show(struct btf_show *show, const char *fmt, ...)
958{
959 va_list args;
960
961 if (!show->state.depth_check) {
962 va_start(args, fmt);
963 show->showfn(show, fmt, args);
964 va_end(args);
965 }
966}
967
968/* Macros are used here as btf_show_type_value[s]() prepends and appends
969 * format specifiers to the format specifier passed in; these do the work of
970 * adding indentation, delimiters etc while the caller simply has to specify
971 * the type value(s) in the format specifier + value(s).
972 */
973#define btf_show_type_value(show, fmt, value) \
974 do { \
975 if ((value) != 0 || (show->flags & BTF_SHOW_ZERO) || \
976 show->state.depth == 0) { \
977 btf_show(show, "%s%s" fmt "%s%s", \
978 btf_show_indent(show), \
979 btf_show_name(show), \
980 value, btf_show_delim(show), \
981 btf_show_newline(show)); \
982 if (show->state.depth > show->state.depth_to_show) \
983 show->state.depth_to_show = show->state.depth; \
984 } \
985 } while (0)
986
987#define btf_show_type_values(show, fmt, ...) \
988 do { \
989 btf_show(show, "%s%s" fmt "%s%s", btf_show_indent(show), \
990 btf_show_name(show), \
991 __VA_ARGS__, btf_show_delim(show), \
992 btf_show_newline(show)); \
993 if (show->state.depth > show->state.depth_to_show) \
994 show->state.depth_to_show = show->state.depth; \
995 } while (0)
996
997/* How much is left to copy to safe buffer after @data? */
998static int btf_show_obj_size_left(struct btf_show *show, void *data)
999{
1000 return show->obj.head + show->obj.size - data;
1001}
1002
1003/* Is object pointed to by @data of @size already copied to our safe buffer? */
1004static bool btf_show_obj_is_safe(struct btf_show *show, void *data, int size)
1005{
1006 return data >= show->obj.data &&
1007 (data + size) < (show->obj.data + BTF_SHOW_OBJ_SAFE_SIZE);
1008}
1009
1010/*
1011 * If object pointed to by @data of @size falls within our safe buffer, return
1012 * the equivalent pointer to the same safe data. Assumes
1013 * copy_from_kernel_nofault() has already happened and our safe buffer is
1014 * populated.
1015 */
1016static void *__btf_show_obj_safe(struct btf_show *show, void *data, int size)
1017{
1018 if (btf_show_obj_is_safe(show, data, size))
1019 return show->obj.safe + (data - show->obj.data);
1020 return NULL;
1021}
1022
1023/*
1024 * Return a safe-to-access version of data pointed to by @data.
1025 * We do this by copying the relevant amount of information
1026 * to the struct btf_show obj.safe buffer using copy_from_kernel_nofault().
1027 *
1028 * If BTF_SHOW_UNSAFE is specified, just return data as-is; no
1029 * safe copy is needed.
1030 *
1031 * Otherwise we need to determine if we have the required amount
1032 * of data (determined by the @data pointer and the size of the
1033 * largest base type we can encounter (represented by
1034 * BTF_SHOW_OBJ_BASE_TYPE_SIZE). Having that much data ensures
1035 * that we will be able to print some of the current object,
1036 * and if more is needed a copy will be triggered.
1037 * Some objects such as structs will not fit into the buffer;
1038 * in such cases additional copies when we iterate over their
1039 * members may be needed.
1040 *
1041 * btf_show_obj_safe() is used to return a safe buffer for
1042 * btf_show_start_type(); this ensures that as we recurse into
1043 * nested types we always have safe data for the given type.
1044 * This approach is somewhat wasteful; it's possible for example
1045 * that when iterating over a large union we'll end up copying the
1046 * same data repeatedly, but the goal is safety not performance.
1047 * We use stack data as opposed to per-CPU buffers because the
1048 * iteration over a type can take some time, and preemption handling
1049 * would greatly complicate use of the safe buffer.
1050 */
1051static void *btf_show_obj_safe(struct btf_show *show,
1052 const struct btf_type *t,
1053 void *data)
1054{
1055 const struct btf_type *rt;
1056 int size_left, size;
1057 void *safe = NULL;
1058
1059 if (show->flags & BTF_SHOW_UNSAFE)
1060 return data;
1061
1062 rt = btf_resolve_size(show->btf, t, &size);
1063 if (IS_ERR(rt)) {
1064 show->state.status = PTR_ERR(rt);
1065 return NULL;
1066 }
1067
1068 /*
1069 * Is this toplevel object? If so, set total object size and
1070 * initialize pointers. Otherwise check if we still fall within
1071 * our safe object data.
1072 */
1073 if (show->state.depth == 0) {
1074 show->obj.size = size;
1075 show->obj.head = data;
1076 } else {
1077 /*
1078 * If the size of the current object is > our remaining
1079 * safe buffer we _may_ need to do a new copy. However
1080 * consider the case of a nested struct; it's size pushes
1081 * us over the safe buffer limit, but showing any individual
1082 * struct members does not. In such cases, we don't need
1083 * to initiate a fresh copy yet; however we definitely need
1084 * at least BTF_SHOW_OBJ_BASE_TYPE_SIZE bytes left
1085 * in our buffer, regardless of the current object size.
1086 * The logic here is that as we resolve types we will
1087 * hit a base type at some point, and we need to be sure
1088 * the next chunk of data is safely available to display
1089 * that type info safely. We cannot rely on the size of
1090 * the current object here because it may be much larger
1091 * than our current buffer (e.g. task_struct is 8k).
1092 * All we want to do here is ensure that we can print the
1093 * next basic type, which we can if either
1094 * - the current type size is within the safe buffer; or
1095 * - at least BTF_SHOW_OBJ_BASE_TYPE_SIZE bytes are left in
1096 * the safe buffer.
1097 */
1098 safe = __btf_show_obj_safe(show, data,
1099 min(size,
1100 BTF_SHOW_OBJ_BASE_TYPE_SIZE));
1101 }
1102
1103 /*
1104 * We need a new copy to our safe object, either because we haven't
1105 * yet copied and are intializing safe data, or because the data
1106 * we want falls outside the boundaries of the safe object.
1107 */
1108 if (!safe) {
1109 size_left = btf_show_obj_size_left(show, data);
1110 if (size_left > BTF_SHOW_OBJ_SAFE_SIZE)
1111 size_left = BTF_SHOW_OBJ_SAFE_SIZE;
1112 show->state.status = copy_from_kernel_nofault(show->obj.safe,
1113 data, size_left);
1114 if (!show->state.status) {
1115 show->obj.data = data;
1116 safe = show->obj.safe;
1117 }
1118 }
1119
1120 return safe;
1121}
1122
1123/*
1124 * Set the type we are starting to show and return a safe data pointer
1125 * to be used for showing the associated data.
1126 */
1127static void *btf_show_start_type(struct btf_show *show,
1128 const struct btf_type *t,
1129 u32 type_id, void *data)
1130{
1131 show->state.type = t;
1132 show->state.type_id = type_id;
1133 show->state.name[0] = '\0';
1134
1135 return btf_show_obj_safe(show, t, data);
1136}
1137
1138static void btf_show_end_type(struct btf_show *show)
1139{
1140 show->state.type = NULL;
1141 show->state.type_id = 0;
1142 show->state.name[0] = '\0';
1143}
1144
1145static void *btf_show_start_aggr_type(struct btf_show *show,
1146 const struct btf_type *t,
1147 u32 type_id, void *data)
1148{
1149 void *safe_data = btf_show_start_type(show, t, type_id, data);
1150
1151 if (!safe_data)
1152 return safe_data;
1153
1154 btf_show(show, "%s%s%s", btf_show_indent(show),
1155 btf_show_name(show),
1156 btf_show_newline(show));
1157 show->state.depth++;
1158 return safe_data;
1159}
1160
1161static void btf_show_end_aggr_type(struct btf_show *show,
1162 const char *suffix)
1163{
1164 show->state.depth--;
1165 btf_show(show, "%s%s%s%s", btf_show_indent(show), suffix,
1166 btf_show_delim(show), btf_show_newline(show));
1167 btf_show_end_type(show);
1168}
1169
1170static void btf_show_start_member(struct btf_show *show,
1171 const struct btf_member *m)
1172{
1173 show->state.member = m;
1174}
1175
1176static void btf_show_start_array_member(struct btf_show *show)
1177{
1178 show->state.array_member = 1;
1179 btf_show_start_member(show, NULL);
1180}
1181
1182static void btf_show_end_member(struct btf_show *show)
1183{
1184 show->state.member = NULL;
1185}
1186
1187static void btf_show_end_array_member(struct btf_show *show)
1188{
1189 show->state.array_member = 0;
1190 btf_show_end_member(show);
1191}
1192
1193static void *btf_show_start_array_type(struct btf_show *show,
1194 const struct btf_type *t,
1195 u32 type_id,
1196 u16 array_encoding,
1197 void *data)
1198{
1199 show->state.array_encoding = array_encoding;
1200 show->state.array_terminated = 0;
1201 return btf_show_start_aggr_type(show, t, type_id, data);
1202}
1203
1204static void btf_show_end_array_type(struct btf_show *show)
1205{
1206 show->state.array_encoding = 0;
1207 show->state.array_terminated = 0;
1208 btf_show_end_aggr_type(show, "]");
1209}
1210
1211static void *btf_show_start_struct_type(struct btf_show *show,
1212 const struct btf_type *t,
1213 u32 type_id,
1214 void *data)
1215{
1216 return btf_show_start_aggr_type(show, t, type_id, data);
1217}
1218
1219static void btf_show_end_struct_type(struct btf_show *show)
1220{
1221 btf_show_end_aggr_type(show, "}");
1222}
1223
680__printf(2, 3) static void __btf_verifier_log(struct bpf_verifier_log *log,1224__printf(2, 3) static void __btf_verifier_log(struct bpf_verifier_log *log,
681 const char *fmt, ...)1225 const char *fmt, ...)
682{1226{
@@ -1079,23 +1623,27 @@ static const struct resolve_vertex *env_stack_peak(struct btf_verifier_env *env) · 396 unmodified lines
1079 * *type_size: (x * y * sizeof(u32)). Hence, *type_size always1623 * *type_size: (x * y * sizeof(u32)). Hence, *type_size always
1080 * corresponds to the return type.1624 * corresponds to the return type.
1081 * *elem_type: u321625 * *elem_type: u32
1626 * *elem_id: id of u32
1082 * *total_nelems: (x * y). Hence, individual elem size is1627 * *total_nelems: (x * y). Hence, individual elem size is
1083 * (*type_size / *total_nelems)1628 * (*type_size / *total_nelems)
1629 * *type_id: id of type if it's changed within the function, 0 if not
1084 *1630 *
1085 * type: is not an array (e.g. const struct X)1631 * type: is not an array (e.g. const struct X)
1086 * return type: type "struct X"1632 * return type: type "struct X"
1087 * *type_size: sizeof(struct X)1633 * *type_size: sizeof(struct X)
1088 * *elem_type: same as return type ("struct X")1634 * *elem_type: same as return type ("struct X")
1635 * *elem_id: 0
1089 * *total_nelems: 11636 * *total_nelems: 1
1637 * *type_id: id of type if it's changed within the function, 0 if not
1090 */1638 */
1091const struct btf_type *1639static const struct btf_type *
1092btf_resolve_size(const struct btf *btf, const struct btf_type *type,1640__btf_resolve_size(const struct btf *btf, const struct btf_type *type,
1093 u32 *type_size, const struct btf_type **elem_type,1641 u32 *type_size, const struct btf_type **elem_type,
1094 u32 *total_nelems)1642 u32 *elem_id, u32 *total_nelems, u32 *type_id)
1095{1643{
1096 const struct btf_type *array_type = NULL;1644 const struct btf_type *array_type = NULL;
1097 const struct btf_array *array;1645 const struct btf_array *array = NULL;
1098 u32 i, size, nelems = 1;1646 u32 i, size, nelems = 1, id = 0;
1099 1647
1100 for (i = 0; i < MAX_RESOLVE_DEPTH; i++) {1648 for (i = 0; i < MAX_RESOLVE_DEPTH; i++) {
1101 switch (BTF_INFO_KIND(type->info)) {1649 switch (BTF_INFO_KIND(type->info)) {
@@ -1116,6 +1664,7 @@ btf_resolve_size(const struct btf *btf, const struct btf_type *type, · 14 unmodified lines
1116 case BTF_KIND_VOLATILE:1664 case BTF_KIND_VOLATILE:
1117 case BTF_KIND_CONST:1665 case BTF_KIND_CONST:
1118 case BTF_KIND_RESTRICT:1666 case BTF_KIND_RESTRICT:
1667 id = type->type;
1119 type = btf_type_by_id(btf, type->type);1668 type = btf_type_by_id(btf, type->type);
1120 break;1669 break;
1121 1670
@@ -1146,10 +1695,21 @@ resolved: · 24 unmodified lines
1146 *total_nelems = nelems;1695 *total_nelems = nelems;
1147 if (elem_type)1696 if (elem_type)
1148 *elem_type = type;1697 *elem_type = type;
1698 if (elem_id)
1699 *elem_id = array ? array->type : 0;
1700 if (type_id && id)
1701 *type_id = id;
1149 1702
1150 return array_type ? : type;1703 return array_type ? : type;
1151}1704}
1152 1705
1706const struct btf_type *
1707btf_resolve_size(const struct btf *btf, const struct btf_type *type,
1708 u32 *type_size)
1709{
1710 return __btf_resolve_size(btf, type, type_size, NULL, NULL, NULL, NULL);
1711}
1712
1153/* The input param "type_id" must point to a needs_resolve type */1713/* The input param "type_id" must point to a needs_resolve type */
1154static const struct btf_type *btf_type_id_resolve(const struct btf *btf,1714static const struct btf_type *btf_type_id_resolve(const struct btf *btf,
1155 u32 *type_id)1715 u32 *type_id)
@@ -1250,11 +1810,11 @@ static int btf_df_resolve(struct btf_verifier_env *env, · 94 unmodified lines
1250 return -EINVAL;1810 return -EINVAL;
1251}1811}
1252 1812
1253static void btf_df_seq_show(const struct btf *btf, const struct btf_type *t,1813static void btf_df_show(const struct btf *btf, const struct btf_type *t,
1254 u32 type_id, void *data, u8 bits_offsets,1814 u32 type_id, void *data, u8 bits_offsets,
1255 struct seq_file *m)1815 struct btf_show *show)
1256{1816{
1257 seq_printf(m, "<unsupported kind:%u>", BTF_INFO_KIND(t->info));1817 btf_show(show, "<unsupported kind:%u>", BTF_INFO_KIND(t->info));
1258}1818}
1259 1819
1260static int btf_int_check_member(struct btf_verifier_env *env,1820static int btf_int_check_member(struct btf_verifier_env *env,
@@ -1427,7 +1987,7 @@ static void btf_int_log(struct btf_verifier_env *env, · 166 unmodified lines
1427 btf_int_encoding_str(BTF_INT_ENCODING(int_data)));1987 btf_int_encoding_str(BTF_INT_ENCODING(int_data)));
1428}1988}
1429 1989
1430static void btf_int128_print(struct seq_file *m, void *data)1990static void btf_int128_print(struct btf_show *show, void *data)
1431{1991{
1432 /* data points to a __int128 number.1992 /* data points to a __int128 number.
1433 * Suppose1993 * Suppose
@@ -1446,9 +2006,10 @@ static void btf_int128_print(struct seq_file *m, void *data) · 12 unmodified lines
1446 lower_num = *(u64 *)data;2006 lower_num = *(u64 *)data;
1447#endif2007#endif
1448 if (upper_num == 0)2008 if (upper_num == 0)
1449 seq_printf(m, "0x%llx", lower_num);2009 btf_show_type_value(show, "0x%llx", lower_num);
1450 else2010 else
1451 seq_printf(m, "0x%llx%016llx", upper_num, lower_num);2011 btf_show_type_values(show, "0x%llx%016llx", upper_num,
2012 lower_num);
1452}2013}
1453 2014
1454static void btf_int128_shift(u64 *print_num, u16 left_shift_bits,2015static void btf_int128_shift(u64 *print_num, u16 left_shift_bits,
@@ -1492,8 +2053,8 @@ static void btf_int128_shift(u64 *print_num, u16 left_shift_bits, · 37 unmodified lines
1492#endif2053#endif
1493}2054}
1494 2055
1495static void btf_bitfield_seq_show(void *data, u8 bits_offset,2056static void btf_bitfield_show(void *data, u8 bits_offset,
1496 u8 nr_bits, struct seq_file *m)2057 u8 nr_bits, struct btf_show *show)
1497{2058{
1498 u16 left_shift_bits, right_shift_bits;2059 u16 left_shift_bits, right_shift_bits;
1499 u8 nr_copy_bytes;2060 u8 nr_copy_bytes;
@@ -1513,14 +2074,14 @@ static void btf_bitfield_seq_show(void *data, u8 bits_offset, · 13 unmodified lines
1513 right_shift_bits = BITS_PER_U128 - nr_bits;2074 right_shift_bits = BITS_PER_U128 - nr_bits;
1514 2075
1515 btf_int128_shift(print_num, left_shift_bits, right_shift_bits);2076 btf_int128_shift(print_num, left_shift_bits, right_shift_bits);
1516 btf_int128_print(m, print_num);2077 btf_int128_print(show, print_num);
1517}2078}
1518 2079
1519 2080
1520static void btf_int_bits_seq_show(const struct btf *btf,2081static void btf_int_bits_show(const struct btf *btf,
1521 const struct btf_type *t,2082 const struct btf_type *t,
1522 void *data, u8 bits_offset,2083 void *data, u8 bits_offset,
1523 struct seq_file *m)2084 struct btf_show *show)
1524{2085{
1525 u32 int_data = btf_type_int(t);2086 u32 int_data = btf_type_int(t);
1526 u8 nr_bits = BTF_INT_BITS(int_data);2087 u8 nr_bits = BTF_INT_BITS(int_data);
@@ -1533,55 +2094,77 @@ static void btf_int_bits_seq_show(const struct btf *btf, · 6 unmodified lines
1533 total_bits_offset = bits_offset + BTF_INT_OFFSET(int_data);2094 total_bits_offset = bits_offset + BTF_INT_OFFSET(int_data);
1534 data += BITS_ROUNDDOWN_BYTES(total_bits_offset);2095 data += BITS_ROUNDDOWN_BYTES(total_bits_offset);
1535 bits_offset = BITS_PER_BYTE_MASKED(total_bits_offset);2096 bits_offset = BITS_PER_BYTE_MASKED(total_bits_offset);
1536 btf_bitfield_seq_show(data, bits_offset, nr_bits, m);2097 btf_bitfield_show(data, bits_offset, nr_bits, show);
1537}2098}
1538 2099
1539static void btf_int_seq_show(const struct btf *btf, const struct btf_type *t,2100static void btf_int_show(const struct btf *btf, const struct btf_type *t,
1540 u32 type_id, void *data, u8 bits_offset,2101 u32 type_id, void *data, u8 bits_offset,
1541 struct seq_file *m)2102 struct btf_show *show)
1542{2103{
1543 u32 int_data = btf_type_int(t);2104 u32 int_data = btf_type_int(t);
1544 u8 encoding = BTF_INT_ENCODING(int_data);2105 u8 encoding = BTF_INT_ENCODING(int_data);
1545 bool sign = encoding & BTF_INT_SIGNED;2106 bool sign = encoding & BTF_INT_SIGNED;
1546 u8 nr_bits = BTF_INT_BITS(int_data);2107 u8 nr_bits = BTF_INT_BITS(int_data);
2108 void *safe_data;
2109
2110 safe_data = btf_show_start_type(show, t, type_id, data);
2111 if (!safe_data)
2112 return;
1547 2113
1548 if (bits_offset || BTF_INT_OFFSET(int_data) ||2114 if (bits_offset || BTF_INT_OFFSET(int_data) ||
1549 BITS_PER_BYTE_MASKED(nr_bits)) {2115 BITS_PER_BYTE_MASKED(nr_bits)) {
1550 btf_int_bits_seq_show(btf, t, data, bits_offset, m);2116 btf_int_bits_show(btf, t, safe_data, bits_offset, show);
1551 return;2117 goto out;
1552 }2118 }
1553 2119
1554 switch (nr_bits) {2120 switch (nr_bits) {
1555 case 128:2121 case 128:
1556 btf_int128_print(m, data);2122 btf_int128_print(show, safe_data);
1557 break;2123 break;
1558 case 64:2124 case 64:
1559 if (sign)2125 if (sign)
1560 seq_printf(m, "%lld", *(s64 *)data);2126 btf_show_type_value(show, "%lld", *(s64 *)safe_data);
1561 else2127 else
1562 seq_printf(m, "%llu", *(u64 *)data);2128 btf_show_type_value(show, "%llu", *(u64 *)safe_data);
1563 break;2129 break;
1564 case 32:2130 case 32:
1565 if (sign)2131 if (sign)
1566 seq_printf(m, "%d", *(s32 *)data);2132 btf_show_type_value(show, "%d", *(s32 *)safe_data);
1567 else2133 else
1568 seq_printf(m, "%u", *(u32 *)data);2134 btf_show_type_value(show, "%u", *(u32 *)safe_data);
1569 break;2135 break;
1570 case 16:2136 case 16:
1571 if (sign)2137 if (sign)
1572 seq_printf(m, "%d", *(s16 *)data);2138 btf_show_type_value(show, "%d", *(s16 *)safe_data);
1573 else2139 else
1574 seq_printf(m, "%u", *(u16 *)data);2140 btf_show_type_value(show, "%u", *(u16 *)safe_data);
1575 break;2141 break;
1576 case 8:2142 case 8:
2143 if (show->state.array_encoding == BTF_INT_CHAR) {
2144 /* check for null terminator */
2145 if (show->state.array_terminated)
2146 break;
2147 if (*(char *)data == '\0') {
2148 show->state.array_terminated = 1;
2149 break;
2150 }
2151 if (isprint(*(char *)data)) {
2152 btf_show_type_value(show, "'%c'",
2153 *(char *)safe_data);
2154 break;
2155 }
2156 }
1577 if (sign)2157 if (sign)
1578 seq_printf(m, "%d", *(s8 *)data);2158 btf_show_type_value(show, "%d", *(s8 *)safe_data);
1579 else2159 else
1580 seq_printf(m, "%u", *(u8 *)data);2160 btf_show_type_value(show, "%u", *(u8 *)safe_data);
1581 break;2161 break;
1582 default:2162 default:
1583 btf_int_bits_seq_show(btf, t, data, bits_offset, m);2163 btf_int_bits_show(btf, t, safe_data, bits_offset, show);
2164 break;
1584 }2165 }
2166out:
2167 btf_show_end_type(show);
1585}2168}
1586 2169
1587static const struct btf_kind_operations int_ops = {2170static const struct btf_kind_operations int_ops = {
@@ -1590,7 +2173,7 @@ static const struct btf_kind_operations int_ops = { · 2 unmodified lines
1590 .check_member = btf_int_check_member,2173 .check_member = btf_int_check_member,
1591 .check_kflag_member = btf_int_check_kflag_member,2174 .check_kflag_member = btf_int_check_kflag_member,
1592 .log_details = btf_int_log,2175 .log_details = btf_int_log,
1593 .seq_show = btf_int_seq_show,2176 .show = btf_int_show,
1594};2177};
1595 2178
1596static int btf_modifier_check_member(struct btf_verifier_env *env,2179static int btf_modifier_check_member(struct btf_verifier_env *env,
@@ -1854,34 +2437,44 @@ static int btf_ptr_resolve(struct btf_verifier_env *env, · 257 unmodified lines
1854 return 0;2437 return 0;
1855}2438}
1856 2439
1857static void btf_modifier_seq_show(const struct btf *btf,2440static void btf_modifier_show(const struct btf *btf,
1858 const struct btf_type *t,2441 const struct btf_type *t,
1859 u32 type_id, void *data,2442 u32 type_id, void *data,
1860 u8 bits_offset, struct seq_file *m)2443 u8 bits_offset, struct btf_show *show)
1861{2444{
1862 if (btf->resolved_ids)2445 if (btf->resolved_ids)
1863 t = btf_type_id_resolve(btf, &type_id);2446 t = btf_type_id_resolve(btf, &type_id);
1864 else2447 else
1865 t = btf_type_skip_modifiers(btf, type_id, NULL);2448 t = btf_type_skip_modifiers(btf, type_id, NULL);
1866 2449
1867 btf_type_ops(t)->seq_show(btf, t, type_id, data, bits_offset, m);2450 btf_type_ops(t)->show(btf, t, type_id, data, bits_offset, show);
1868}2451}
1869 2452
1870static void btf_var_seq_show(const struct btf *btf, const struct btf_type *t,2453static void btf_var_show(const struct btf *btf, const struct btf_type *t,
1871 u32 type_id, void *data, u8 bits_offset,2454 u32 type_id, void *data, u8 bits_offset,
1872 struct seq_file *m)2455 struct btf_show *show)
1873{2456{
1874 t = btf_type_id_resolve(btf, &type_id);2457 t = btf_type_id_resolve(btf, &type_id);
1875 2458
1876 btf_type_ops(t)->seq_show(btf, t, type_id, data, bits_offset, m);2459 btf_type_ops(t)->show(btf, t, type_id, data, bits_offset, show);
1877}2460}
1878 2461
1879static void btf_ptr_seq_show(const struct btf *btf, const struct btf_type *t,2462static void btf_ptr_show(const struct btf *btf, const struct btf_type *t,
1880 u32 type_id, void *data, u8 bits_offset,2463 u32 type_id, void *data, u8 bits_offset,
1881 struct seq_file *m)2464 struct btf_show *show)
1882{2465{
1883 /* It is a hashed value */2466 void *safe_data;
1884 seq_printf(m, "%p", *(void **)data);2467
2468 safe_data = btf_show_start_type(show, t, type_id, data);
2469 if (!safe_data)
2470 return;
2471
2472 /* It is a hashed value unless BTF_SHOW_PTR_RAW is specified */
2473 if (show->flags & BTF_SHOW_PTR_RAW)
2474 btf_show_type_value(show, "0x%px", *(void **)safe_data);
2475 else
2476 btf_show_type_value(show, "0x%p", *(void **)safe_data);
2477 btf_show_end_type(show);
1885}2478}
1886 2479
1887static void btf_ref_type_log(struct btf_verifier_env *env,2480static void btf_ref_type_log(struct btf_verifier_env *env,
@@ -1896,7 +2489,7 @@ static struct btf_kind_operations modifier_ops = { · 8 unmodified lines
1896 .check_member = btf_modifier_check_member,2489 .check_member = btf_modifier_check_member,
1897 .check_kflag_member = btf_modifier_check_kflag_member,2490 .check_kflag_member = btf_modifier_check_kflag_member,
1898 .log_details = btf_ref_type_log,2491 .log_details = btf_ref_type_log,
1899 .seq_show = btf_modifier_seq_show,2492 .show = btf_modifier_show,
1900};2493};
1901 2494
1902static struct btf_kind_operations ptr_ops = {2495static struct btf_kind_operations ptr_ops = {
@@ -1905,7 +2498,7 @@ static struct btf_kind_operations ptr_ops = { · 2 unmodified lines
1905 .check_member = btf_ptr_check_member,2498 .check_member = btf_ptr_check_member,
1906 .check_kflag_member = btf_generic_check_kflag_member,2499 .check_kflag_member = btf_generic_check_kflag_member,
1907 .log_details = btf_ref_type_log,2500 .log_details = btf_ref_type_log,
1908 .seq_show = btf_ptr_seq_show,2501 .show = btf_ptr_show,
1909};2502};
1910 2503
1911static s32 btf_fwd_check_meta(struct btf_verifier_env *env,2504static s32 btf_fwd_check_meta(struct btf_verifier_env *env,
@@ -1946,7 +2539,7 @@ static struct btf_kind_operations fwd_ops = { · 34 unmodified lines
1946 .check_member = btf_df_check_member,2539 .check_member = btf_df_check_member,
1947 .check_kflag_member = btf_df_check_kflag_member,2540 .check_kflag_member = btf_df_check_kflag_member,
1948 .log_details = btf_fwd_type_log,2541 .log_details = btf_fwd_type_log,
1949 .seq_show = btf_df_seq_show,2542 .show = btf_df_show,
1950};2543};
1951 2544
1952static int btf_array_check_member(struct btf_verifier_env *env,2545static int btf_array_check_member(struct btf_verifier_env *env,
@@ -2105,28 +2698,90 @@ static void btf_array_log(struct btf_verifier_env *env, · 152 unmodified lines
2105 array->type, array->index_type, array->nelems);2698 array->type, array->index_type, array->nelems);
2106}2699}
2107 2700
2108static void btf_array_seq_show(const struct btf *btf, const struct btf_type *t,2701static void __btf_array_show(const struct btf *btf, const struct btf_type *t,
2109 u32 type_id, void *data, u8 bits_offset,2702 u32 type_id, void *data, u8 bits_offset,
2110 struct seq_file *m)2703 struct btf_show *show)
2111{2704{
2112 const struct btf_array *array = btf_type_array(t);2705 const struct btf_array *array = btf_type_array(t);
2113 const struct btf_kind_operations *elem_ops;2706 const struct btf_kind_operations *elem_ops;
2114 const struct btf_type *elem_type;2707 const struct btf_type *elem_type;
2115 u32 i, elem_size, elem_type_id;2708 u32 i, elem_size = 0, elem_type_id;
2709 u16 encoding = 0;
2116 2710
2117 elem_type_id = array->type;2711 elem_type_id = array->type;
2118 elem_type = btf_type_id_size(btf, &elem_type_id, &elem_size);2712 elem_type = btf_type_skip_modifiers(btf, elem_type_id, NULL);
2713 if (elem_type && btf_type_has_size(elem_type))
2714 elem_size = elem_type->size;
2715
2716 if (elem_type && btf_type_is_int(elem_type)) {
2717 u32 int_type = btf_type_int(elem_type);
2718
2719 encoding = BTF_INT_ENCODING(int_type);
2720
2721 /*
2722 * BTF_INT_CHAR encoding never seems to be set for
2723 * char arrays, so if size is 1 and element is
2724 * printable as a char, we'll do that.
2725 */
2726 if (elem_size == 1)
2727 encoding = BTF_INT_CHAR;
2728 }
2729
2730 if (!btf_show_start_array_type(show, t, type_id, encoding, data))
2731 return;
2732
2733 if (!elem_type)
2734 goto out;
2119 elem_ops = btf_type_ops(elem_type);2735 elem_ops = btf_type_ops(elem_type);
2120 seq_puts(m, "[");2736
2121 for (i = 0; i < array->nelems; i++) {2737 for (i = 0; i < array->nelems; i++) {
2122 if (i)
2123 seq_puts(m, ",");
2124 2738
2125 elem_ops->seq_show(btf, elem_type, elem_type_id, data,2739 btf_show_start_array_member(show);
2126 bits_offset, m);2740
2741 elem_ops->show(btf, elem_type, elem_type_id, data,
2742 bits_offset, show);
2127 data += elem_size;2743 data += elem_size;
2744
2745 btf_show_end_array_member(show);
2746
2747 if (show->state.array_terminated)
2748 break;
2749 }
2750out:
2751 btf_show_end_array_type(show);
2752}
2753
2754static void btf_array_show(const struct btf *btf, const struct btf_type *t,
2755 u32 type_id, void *data, u8 bits_offset,
2756 struct btf_show *show)
2757{
2758 const struct btf_member *m = show->state.member;
2759
2760 /*
2761 * First check if any members would be shown (are non-zero).
2762 * See comments above "struct btf_show" definition for more
2763 * details on how this works at a high-level.
2764 */
2765 if (show->state.depth > 0 && !(show->flags & BTF_SHOW_ZERO)) {
2766 if (!show->state.depth_check) {
2767 show->state.depth_check = show->state.depth + 1;
2768 show->state.depth_to_show = 0;
2769 }
2770 __btf_array_show(btf, t, type_id, data, bits_offset, show);
2771 show->state.member = m;
2772
2773 if (show->state.depth_check != show->state.depth + 1)
2774 return;
2775 show->state.depth_check = 0;
2776
2777 if (show->state.depth_to_show <= show->state.depth)
2778 return;
2779 /*
2780 * Reaching here indicates we have recursed and found
2781 * non-zero array member(s).
2782 */
2128 }2783 }
2129 seq_puts(m, "]");2784 __btf_array_show(btf, t, type_id, data, bits_offset, show);
2130}2785}
2131 2786
2132static struct btf_kind_operations array_ops = {2787static struct btf_kind_operations array_ops = {
@@ -2135,7 +2790,7 @@ static struct btf_kind_operations array_ops = { · 2 unmodified lines
2135 .check_member = btf_array_check_member,2790 .check_member = btf_array_check_member,
2136 .check_kflag_member = btf_generic_check_kflag_member,2791 .check_kflag_member = btf_generic_check_kflag_member,
2137 .log_details = btf_array_log,2792 .log_details = btf_array_log,
2138 .seq_show = btf_array_seq_show,2793 .show = btf_array_show,
2139};2794};
2140 2795
2141static int btf_struct_check_member(struct btf_verifier_env *env,2796static int btf_struct_check_member(struct btf_verifier_env *env,
@@ -2358,15 +3013,18 @@ int btf_find_spin_lock(const struct btf *btf, const struct btf_type *t) · 216 unmodified lines
2358 return off;3013 return off;
2359}3014}
2360 3015
2361static void btf_struct_seq_show(const struct btf *btf, const struct btf_type *t,3016static void __btf_struct_show(const struct btf *btf, const struct btf_type *t,
2362 u32 type_id, void *data, u8 bits_offset,3017 u32 type_id, void *data, u8 bits_offset,
2363 struct seq_file *m)3018 struct btf_show *show)
2364{3019{
2365 const char *seq = BTF_INFO_KIND(t->info) == BTF_KIND_UNION ? "|" : ",";
2366 const struct btf_member *member;3020 const struct btf_member *member;
3021 void *safe_data;
2367 u32 i;3022 u32 i;
2368 3023
2369 seq_puts(m, "{");3024 safe_data = btf_show_start_struct_type(show, t, type_id, data);
3025 if (!safe_data)
3026 return;
3027
2370 for_each_member(i, t, member) {3028 for_each_member(i, t, member) {
2371 const struct btf_type *member_type = btf_type_by_id(btf,3029 const struct btf_type *member_type = btf_type_by_id(btf,
2372 member->type);3030 member->type);
@@ -2375,23 +3033,65 @@ static void btf_struct_seq_show(const struct btf *btf, const struct btf_type *t, · 2 unmodified lines
2375 u32 bytes_offset;3033 u32 bytes_offset;
2376 u8 bits8_offset;3034 u8 bits8_offset;
2377 3035
2378 if (i)3036 btf_show_start_member(show, member);
2379 seq_puts(m, seq);
2380 3037
2381 member_offset = btf_member_bit_offset(t, member);3038 member_offset = btf_member_bit_offset(t, member);
2382 bitfield_size = btf_member_bitfield_size(t, member);3039 bitfield_size = btf_member_bitfield_size(t, member);
2383 bytes_offset = BITS_ROUNDDOWN_BYTES(member_offset);3040 bytes_offset = BITS_ROUNDDOWN_BYTES(member_offset);
2384 bits8_offset = BITS_PER_BYTE_MASKED(member_offset);3041 bits8_offset = BITS_PER_BYTE_MASKED(member_offset);
2385 if (bitfield_size) {3042 if (bitfield_size) {
2386 btf_bitfield_seq_show(data + bytes_offset, bits8_offset,3043 safe_data = btf_show_start_type(show, member_type,
2387 bitfield_size, m);3044 member->type,
3045 data + bytes_offset);
3046 if (safe_data)
3047 btf_bitfield_show(safe_data,
3048 bits8_offset,
3049 bitfield_size, show);
3050 btf_show_end_type(show);
2388 } else {3051 } else {
2389 ops = btf_type_ops(member_type);3052 ops = btf_type_ops(member_type);
2390 ops->seq_show(btf, member_type, member->type,3053 ops->show(btf, member_type, member->type,
2391 data + bytes_offset, bits8_offset, m);3054 data + bytes_offset, bits8_offset, show);
2392 }3055 }
3056
3057 btf_show_end_member(show);
2393 }3058 }
2394 seq_puts(m, "}");3059
3060 btf_show_end_struct_type(show);
3061}
3062
3063static void btf_struct_show(const struct btf *btf, const struct btf_type *t,
3064 u32 type_id, void *data, u8 bits_offset,
3065 struct btf_show *show)
3066{
3067 const struct btf_member *m = show->state.member;
3068
3069 /*
3070 * First check if any members would be shown (are non-zero).
3071 * See comments above "struct btf_show" definition for more
3072 * details on how this works at a high-level.
3073 */
3074 if (show->state.depth > 0 && !(show->flags & BTF_SHOW_ZERO)) {
3075 if (!show->state.depth_check) {
3076 show->state.depth_check = show->state.depth + 1;
3077 show->state.depth_to_show = 0;
3078 }
3079 __btf_struct_show(btf, t, type_id, data, bits_offset, show);
3080 /* Restore saved member data here */
3081 show->state.member = m;
3082 if (show->state.depth_check != show->state.depth + 1)
3083 return;
3084 show->state.depth_check = 0;
3085
3086 if (show->state.depth_to_show <= show->state.depth)
3087 return;
3088 /*
3089 * Reaching here indicates we have recursed and found
3090 * non-zero child values.
3091 */
3092 }
3093
3094 __btf_struct_show(btf, t, type_id, data, bits_offset, show);
2395}3095}
2396 3096
2397static struct btf_kind_operations struct_ops = {3097static struct btf_kind_operations struct_ops = {
@@ -2400,7 +3100,7 @@ static struct btf_kind_operations struct_ops = { · 2 unmodified lines
2400 .check_member = btf_struct_check_member,3100 .check_member = btf_struct_check_member,
2401 .check_kflag_member = btf_generic_check_kflag_member,3101 .check_kflag_member = btf_generic_check_kflag_member,
2402 .log_details = btf_struct_log,3102 .log_details = btf_struct_log,
2403 .seq_show = btf_struct_seq_show,3103 .show = btf_struct_show,
2404};3104};
2405 3105
2406static int btf_enum_check_member(struct btf_verifier_env *env,3106static int btf_enum_check_member(struct btf_verifier_env *env,
@@ -2531,24 +3231,35 @@ static void btf_enum_log(struct btf_verifier_env *env, · 124 unmodified lines
2531 btf_verifier_log(env, "size=%u vlen=%u", t->size, btf_type_vlen(t));3231 btf_verifier_log(env, "size=%u vlen=%u", t->size, btf_type_vlen(t));
2532}3232}
2533 3233
2534static void btf_enum_seq_show(const struct btf *btf, const struct btf_type *t,3234static void btf_enum_show(const struct btf *btf, const struct btf_type *t,
2535 u32 type_id, void *data, u8 bits_offset,3235 u32 type_id, void *data, u8 bits_offset,
2536 struct seq_file *m)3236 struct btf_show *show)
2537{3237{
2538 const struct btf_enum *enums = btf_type_enum(t);3238 const struct btf_enum *enums = btf_type_enum(t);
2539 u32 i, nr_enums = btf_type_vlen(t);3239 u32 i, nr_enums = btf_type_vlen(t);
2540 int v = *(int *)data;3240 void *safe_data;
3241 int v;
3242
3243 safe_data = btf_show_start_type(show, t, type_id, data);
3244 if (!safe_data)
3245 return;
3246
3247 v = *(int *)safe_data;
2541 3248
2542 for (i = 0; i < nr_enums; i++) {3249 for (i = 0; i < nr_enums; i++) {
2543 if (v == enums[i].val) {3250 if (v != enums[i].val)
2544 seq_printf(m, "%s",3251 continue;
2545 __btf_name_by_offset(btf,3252
2546 enums[i].name_off));3253 btf_show_type_value(show, "%s",
2547 return;3254 __btf_name_by_offset(btf,
2548 }3255 enums[i].name_off));
3256
3257 btf_show_end_type(show);
3258 return;
2549 }3259 }
2550 3260
2551 seq_printf(m, "%d", v);3261 btf_show_type_value(show, "%d", v);
3262 btf_show_end_type(show);
2552}3263}
2553 3264
2554static struct btf_kind_operations enum_ops = {3265static struct btf_kind_operations enum_ops = {
@@ -2557,7 +3268,7 @@ static struct btf_kind_operations enum_ops = { · 2 unmodified lines
2557 .check_member = btf_enum_check_member,3268 .check_member = btf_enum_check_member,
2558 .check_kflag_member = btf_enum_check_kflag_member,3269 .check_kflag_member = btf_enum_check_kflag_member,
2559 .log_details = btf_enum_log,3270 .log_details = btf_enum_log,
2560 .seq_show = btf_enum_seq_show,3271 .show = btf_enum_show,
2561};3272};
2562 3273
2563static s32 btf_func_proto_check_meta(struct btf_verifier_env *env,3274static s32 btf_func_proto_check_meta(struct btf_verifier_env *env,
@@ -2644,7 +3355,7 @@ static struct btf_kind_operations func_proto_ops = { · 80 unmodified lines
2644 .check_member = btf_df_check_member,3355 .check_member = btf_df_check_member,
2645 .check_kflag_member = btf_df_check_kflag_member,3356 .check_kflag_member = btf_df_check_kflag_member,
2646 .log_details = btf_func_proto_log,3357 .log_details = btf_func_proto_log,
2647 .seq_show = btf_df_seq_show,3358 .show = btf_df_show,
2648};3359};
2649 3360
2650static s32 btf_func_check_meta(struct btf_verifier_env *env,3361static s32 btf_func_check_meta(struct btf_verifier_env *env,
@@ -2678,7 +3389,7 @@ static struct btf_kind_operations func_ops = { · 27 unmodified lines
2678 .check_member = btf_df_check_member,3389 .check_member = btf_df_check_member,
2679 .check_kflag_member = btf_df_check_kflag_member,3390 .check_kflag_member = btf_df_check_kflag_member,
2680 .log_details = btf_ref_type_log,3391 .log_details = btf_ref_type_log,
2681 .seq_show = btf_df_seq_show,3392 .show = btf_df_show,
2682};3393};
2683 3394
2684static s32 btf_var_check_meta(struct btf_verifier_env *env,3395static s32 btf_var_check_meta(struct btf_verifier_env *env,
@@ -2742,7 +3453,7 @@ static const struct btf_kind_operations var_ops = { · 57 unmodified lines
2742 .check_member = btf_df_check_member,3453 .check_member = btf_df_check_member,
2743 .check_kflag_member = btf_df_check_kflag_member,3454 .check_kflag_member = btf_df_check_kflag_member,
2744 .log_details = btf_var_log,3455 .log_details = btf_var_log,
2745 .seq_show = btf_var_seq_show,3456 .show = btf_var_show,
2746};3457};
2747 3458
2748static s32 btf_datasec_check_meta(struct btf_verifier_env *env,3459static s32 btf_datasec_check_meta(struct btf_verifier_env *env,
@@ -2868,24 +3579,28 @@ static void btf_datasec_log(struct btf_verifier_env *env, · 119 unmodified lines
2868 btf_verifier_log(env, "size=%u vlen=%u", t->size, btf_type_vlen(t));3579 btf_verifier_log(env, "size=%u vlen=%u", t->size, btf_type_vlen(t));
2869}3580}
2870 3581
2871static void btf_datasec_seq_show(const struct btf *btf,3582static void btf_datasec_show(const struct btf *btf,
2872 const struct btf_type *t, u32 type_id,3583 const struct btf_type *t, u32 type_id,
2873 void *data, u8 bits_offset,3584 void *data, u8 bits_offset,
2874 struct seq_file *m)3585 struct btf_show *show)
2875{3586{
2876 const struct btf_var_secinfo *vsi;3587 const struct btf_var_secinfo *vsi;
2877 const struct btf_type *var;3588 const struct btf_type *var;
2878 u32 i;3589 u32 i;
2879 3590
2880 seq_printf(m, "section (\"%s\") = {", __btf_name_by_offset(btf, t->name_off));3591 if (!btf_show_start_type(show, t, type_id, data))
3592 return;
3593
3594 btf_show_type_value(show, "section (\"%s\") = {",
3595 __btf_name_by_offset(btf, t->name_off));
2881 for_each_vsi(i, t, vsi) {3596 for_each_vsi(i, t, vsi) {
2882 var = btf_type_by_id(btf, vsi->type);3597 var = btf_type_by_id(btf, vsi->type);
2883 if (i)3598 if (i)
2884 seq_puts(m, ",");3599 btf_show(show, ",");
2885 btf_type_ops(var)->seq_show(btf, var, vsi->type,3600 btf_type_ops(var)->show(btf, var, vsi->type,
2886 data + vsi->offset, bits_offset, m);3601 data + vsi->offset, bits_offset, show);
2887 }3602 }
2888 seq_puts(m, "}");3603 btf_show_end_type(show);
2889}3604}
2890 3605
2891static const struct btf_kind_operations datasec_ops = {3606static const struct btf_kind_operations datasec_ops = {
@@ -2894,7 +3609,7 @@ static const struct btf_kind_operations datasec_ops = { · 2 unmodified lines
2894 .check_member = btf_df_check_member,3609 .check_member = btf_df_check_member,
2895 .check_kflag_member = btf_df_check_kflag_member,3610 .check_kflag_member = btf_df_check_kflag_member,
2896 .log_details = btf_datasec_log,3611 .log_details = btf_datasec_log,
2897 .seq_show = btf_datasec_seq_show,3612 .show = btf_datasec_show,
2898};3613};
2899 3614
2900static int btf_func_proto_check(struct btf_verifier_env *env,3615static int btf_func_proto_check(struct btf_verifier_env *env,
@@ -3688,7 +4403,7 @@ errout: · 787 unmodified lines
3688 4403
3689struct btf *bpf_prog_get_target_btf(const struct bpf_prog *prog)4404struct btf *bpf_prog_get_target_btf(const struct bpf_prog *prog)
3690{4405{
3691 struct bpf_prog *tgt_prog = prog->aux->linked_prog;4406 struct bpf_prog *tgt_prog = prog->aux->dst_prog;
3692 4407
3693 if (tgt_prog) {4408 if (tgt_prog) {
3694 return tgt_prog->aux->btf;4409 return tgt_prog->aux->btf;
@@ -3715,7 +4430,7 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type, · 20 unmodified lines
3715 struct bpf_insn_access_aux *info)4430 struct bpf_insn_access_aux *info)
3716{4431{
3717 const struct btf_type *t = prog->aux->attach_func_proto;4432 const struct btf_type *t = prog->aux->attach_func_proto;
3718 struct bpf_prog *tgt_prog = prog->aux->linked_prog;4433 struct bpf_prog *tgt_prog = prog->aux->dst_prog;
3719 struct btf *btf = bpf_prog_get_target_btf(prog);4434 struct btf *btf = bpf_prog_get_target_btf(prog);
3720 const char *tname = prog->aux->attach_func_name;4435 const char *tname = prog->aux->attach_func_name;
3721 struct bpf_verifier_log *log = info->log;4436 struct bpf_verifier_log *log = info->log;
@@ -3842,7 +4557,14 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type, · 120 unmodified lines
3842 4557
3843 info->reg_type = PTR_TO_BTF_ID;4558 info->reg_type = PTR_TO_BTF_ID;
3844 if (tgt_prog) {4559 if (tgt_prog) {
3845 ret = btf_translate_to_vmlinux(log, btf, t, tgt_prog->type, arg);4560 enum bpf_prog_type tgt_type;
4561
4562 if (tgt_prog->type == BPF_PROG_TYPE_EXT)
4563 tgt_type = tgt_prog->aux->saved_dst_prog_type;
4564 else
4565 tgt_type = tgt_prog->type;
4566
4567 ret = btf_translate_to_vmlinux(log, btf, t, tgt_type, arg);
3846 if (ret > 0) {4568 if (ret > 0) {
3847 info->btf_id = ret;4569 info->btf_id = ret;
3848 return true;4570 return true;
@@ -3870,16 +4592,22 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type, · 21 unmodified lines
3870 return true;4592 return true;
3871}4593}
3872 4594
3873int btf_struct_access(struct bpf_verifier_log *log,4595enum bpf_struct_walk_result {
3874 const struct btf_type *t, int off, int size,4596 /* < 0 error */
3875 enum bpf_access_type atype,4597 WALK_SCALAR = 0,
3876 u32 *next_btf_id)4598 WALK_PTR,
4599 WALK_STRUCT,
4600};
4601
4602static int btf_struct_walk(struct bpf_verifier_log *log,
4603 const struct btf_type *t, int off, int size,
4604 u32 *next_btf_id)
3877{4605{
3878 u32 i, moff, mtrue_end, msize = 0, total_nelems = 0;4606 u32 i, moff, mtrue_end, msize = 0, total_nelems = 0;
3879 const struct btf_type *mtype, *elem_type = NULL;4607 const struct btf_type *mtype, *elem_type = NULL;
3880 const struct btf_member *member;4608 const struct btf_member *member;
3881 const char *tname, *mname;4609 const char *tname, *mname;
3882 u32 vlen;4610 u32 vlen, elem_id, mid;
3883 4611
3884again:4612again:
3885 tname = __btf_name_by_offset(btf_vmlinux, t->name_off);4613 tname = __btf_name_by_offset(btf_vmlinux, t->name_off);
@@ -3915,14 +4643,13 @@ again: · 29 unmodified lines
3915 /* Only allow structure for now, can be relaxed for4643 /* Only allow structure for now, can be relaxed for
3916 * other types later.4644 * other types later.
3917 */4645 */
3918 elem_type = btf_type_skip_modifiers(btf_vmlinux,4646 t = btf_type_skip_modifiers(btf_vmlinux, array_elem->type,
3919 array_elem->type, NULL);4647 NULL);
3920 if (!btf_type_is_struct(elem_type))4648 if (!btf_type_is_struct(t))
3921 goto error;4649 goto error;
3922 4650
3923 off = (off - moff) % elem_type->size;4651 off = (off - moff) % t->size;
3924 return btf_struct_access(log, elem_type, off, size, atype,4652 goto again;
3925 next_btf_id);
3926 4653
3927error:4654error:
3928 bpf_log(log, "access beyond struct %s at off %u size %u\n",4655 bpf_log(log, "access beyond struct %s at off %u size %u\n",
@@ -3951,7 +4678,7 @@ error: · 22 unmodified lines
3951 */4678 */
3952 if (off <= moff &&4679 if (off <= moff &&
3953 BITS_ROUNDUP_BYTES(end_bit) <= off + size)4680 BITS_ROUNDUP_BYTES(end_bit) <= off + size)
3954 return SCALAR_VALUE;4681 return WALK_SCALAR;
3955 4682
3956 /* off may be accessing a following member4683 /* off may be accessing a following member
3957 *4684 *
@@ -3973,11 +4700,13 @@ error: · 15 unmodified lines
3973 break;4700 break;
3974 4701
3975 /* type of the field */4702 /* type of the field */
4703 mid = member->type;
3976 mtype = btf_type_by_id(btf_vmlinux, member->type);4704 mtype = btf_type_by_id(btf_vmlinux, member->type);
3977 mname = __btf_name_by_offset(btf_vmlinux, member->name_off);4705 mname = __btf_name_by_offset(btf_vmlinux, member->name_off);
3978 4706
3979 mtype = btf_resolve_size(btf_vmlinux, mtype, &msize,4707 mtype = __btf_resolve_size(btf_vmlinux, mtype, &msize,
3980 &elem_type, &total_nelems);4708 &elem_type, &elem_id, &total_nelems,
4709 &mid);
3981 if (IS_ERR(mtype)) {4710 if (IS_ERR(mtype)) {
3982 bpf_log(log, "field %s doesn't have size\n", mname);4711 bpf_log(log, "field %s doesn't have size\n", mname);
3983 return -EFAULT;4712 return -EFAULT;
@@ -3991,7 +4720,7 @@ error: · 7 unmodified lines
3991 if (btf_type_is_array(mtype)) {4720 if (btf_type_is_array(mtype)) {
3992 u32 elem_idx;4721 u32 elem_idx;
3993 4722
3994 /* btf_resolve_size() above helps to4723 /* __btf_resolve_size() above helps to
3995 * linearize a multi-dimensional array.4724 * linearize a multi-dimensional array.
3996 *4725 *
3997 * The logic here is treating an array4726 * The logic here is treating an array
@@ -4039,6 +4768,7 @@ error: · 41 unmodified lines
4039 elem_idx = (off - moff) / msize;4768 elem_idx = (off - moff) / msize;
4040 moff += elem_idx * msize;4769 moff += elem_idx * msize;
4041 mtype = elem_type;4770 mtype = elem_type;
4771 mid = elem_id;
4042 }4772 }
4043 4773
4044 /* the 'off' we're looking for is either equal to start4774 /* the 'off' we're looking for is either equal to start
@@ -4048,6 +4778,12 @@ error: · 3 unmodified lines
4048 /* our field must be inside that union or struct */4778 /* our field must be inside that union or struct */
4049 t = mtype;4779 t = mtype;
4050 4780
4781 /* return if the offset matches the member offset */
4782 if (off == moff) {
4783 *next_btf_id = mid;
4784 return WALK_STRUCT;
4785 }
4786
4051 /* adjust offset we're looking for */4787 /* adjust offset we're looking for */
4052 off -= moff;4788 off -= moff;
4053 goto again;4789 goto again;
@@ -4063,11 +4799,10 @@ error: · 9 unmodified lines
4063 mname, moff, tname, off, size);4799 mname, moff, tname, off, size);
4064 return -EACCES;4800 return -EACCES;
4065 }4801 }
4066
4067 stype = btf_type_skip_modifiers(btf_vmlinux, mtype->type, &id);4802 stype = btf_type_skip_modifiers(btf_vmlinux, mtype->type, &id);
4068 if (btf_type_is_struct(stype)) {4803 if (btf_type_is_struct(stype)) {
4069 *next_btf_id = id;4804 *next_btf_id = id;
4070 return PTR_TO_BTF_ID;4805 return WALK_PTR;
4071 }4806 }
4072 }4807 }
4073 4808
@@ -4084,23 +4819,82 @@ error: · 10 unmodified lines
4084 return -EACCES;4819 return -EACCES;
4085 }4820 }
4086 4821
4087 return SCALAR_VALUE;4822 return WALK_SCALAR;
4088 }4823 }
4089 bpf_log(log, "struct %s doesn't have field at offset %d\n", tname, off);4824 bpf_log(log, "struct %s doesn't have field at offset %d\n", tname, off);
4090 return -EINVAL;4825 return -EINVAL;
4091}4826}
4092 4827
4093int btf_resolve_helper_id(struct bpf_verifier_log *log,4828int btf_struct_access(struct bpf_verifier_log *log,
4094 const struct bpf_func_proto *fn, int arg)4829 const struct btf_type *t, int off, int size,
4830 enum bpf_access_type atype __maybe_unused,
4831 u32 *next_btf_id)
4832{
4833 int err;
4834 u32 id;
4835
4836 do {
4837 err = btf_struct_walk(log, t, off, size, &id);
4838
4839 switch (err) {
4840 case WALK_PTR:
4841 /* If we found the pointer or scalar on t+off,
4842 * we're done.
4843 */
4844 *next_btf_id = id;
4845 return PTR_TO_BTF_ID;
4846 case WALK_SCALAR:
4847 return SCALAR_VALUE;
4848 case WALK_STRUCT:
4849 /* We found nested struct, so continue the search
4850 * by diving in it. At this point the offset is
4851 * aligned with the new type, so set it to 0.
4852 */
4853 t = btf_type_by_id(btf_vmlinux, id);
4854 off = 0;
4855 break;
4856 default:
4857 /* It's either error or unknown return value..
4858 * scream and leave.
4859 */
4860 if (WARN_ONCE(err > 0, "unknown btf_struct_walk return value"))
4861 return -EINVAL;
4862 return err;
4863 }
4864 } while (t);
4865
4866 return -EINVAL;
4867}
4868
4869bool btf_struct_ids_match(struct bpf_verifier_log *log,
4870 int off, u32 id, u32 need_type_id)
4095{4871{
4096 int id;4872 const struct btf_type *type;
4873 int err;
4097 4874
4098 if (fn->arg_type[arg] != ARG_PTR_TO_BTF_ID || !btf_vmlinux)4875 /* Are we already done? */
4099 return -EINVAL;4876 if (need_type_id == id && off == 0)
4100 id = fn->btf_id[arg];4877 return true;
4101 if (!id || id > btf_vmlinux->nr_types)4878
4102 return -EINVAL;4879again:
4103 return id;4880 type = btf_type_by_id(btf_vmlinux, id);
4881 if (!type)
4882 return false;
4883 err = btf_struct_walk(log, type, off, 1, &id);
4884 if (err != WALK_STRUCT)
4885 return false;
4886
4887 /* We found nested struct object. If it matches
4888 * the requested ID, we're done. Otherwise let's
4889 * continue the search with offset 0 in the new
4890 * type.
4891 */
4892 if (need_type_id != id) {
4893 off = 0;
4894 goto again;
4895 }
4896
4897 return true;
4104}4898}
4105 4899
4106static int __get_type_size(struct btf *btf, u32 btf_id,4900static int __get_type_size(struct btf *btf, u32 btf_id,
@@ -4298,7 +5092,7 @@ static int btf_check_func_type_match(struct bpf_verifier_log *log, · 191 unmodified lines
4298}5092}
4299 5093
4300/* Compare BTFs of given program with BTF of target program */5094/* Compare BTFs of given program with BTF of target program */
4301int btf_check_type_match(struct bpf_verifier_env *env, struct bpf_prog *prog,5095int btf_check_type_match(struct bpf_verifier_log *log, const struct bpf_prog *prog,
4302 struct btf *btf2, const struct btf_type *t2)5096 struct btf *btf2, const struct btf_type *t2)
4303{5097{
4304 struct btf *btf1 = prog->aux->btf;5098 struct btf *btf1 = prog->aux->btf;
@@ -4306,7 +5100,7 @@ int btf_check_type_match(struct bpf_verifier_env *env, struct bpf_prog *prog, · 1 unmodified line
4306 u32 btf_id = 0;5100 u32 btf_id = 0;
4307 5101
4308 if (!prog->aux->func_info) {5102 if (!prog->aux->func_info) {
4309 bpf_log(&env->log, "Program extension requires BTF\n");5103 bpf_log(log, "Program extension requires BTF\n");
4310 return -EINVAL;5104 return -EINVAL;
4311 }5105 }
4312 5106
@@ -4318,7 +5112,7 @@ int btf_check_type_match(struct bpf_verifier_env *env, struct bpf_prog *prog, · 5 unmodified lines
4318 if (!t1 || !btf_type_is_func(t1))5112 if (!t1 || !btf_type_is_func(t1))
4319 return -EFAULT;5113 return -EFAULT;
4320 5114
4321 return btf_check_func_type_match(&env->log, btf1, t1, btf2, t2);5115 return btf_check_func_type_match(log, btf1, t1, btf2, t2);
4322}5116}
4323 5117
4324/* Compare BTF of a function with given bpf_reg_state.5118/* Compare BTF of a function with given bpf_reg_state.
@@ -4469,7 +5263,7 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog, · 144 unmodified lines
4469 return -EFAULT;5263 return -EFAULT;
4470 }5264 }
4471 if (prog_type == BPF_PROG_TYPE_EXT)5265 if (prog_type == BPF_PROG_TYPE_EXT)
4472 prog_type = prog->aux->linked_prog->type;5266 prog_type = prog->aux->dst_prog->type;
4473 5267
4474 t = btf_type_by_id(btf, t->type);5268 t = btf_type_by_id(btf, t->type);
4475 if (!t || !btf_type_is_func_proto(t)) {5269 if (!t || !btf_type_is_func_proto(t)) {
@@ -4516,12 +5310,93 @@ int btf_prepare_func_args(struct bpf_verifier_env *env, int subprog, · 40 unmodified lines
4516 return 0;5310 return 0;
4517}5311}
4518 5312
5313static void btf_type_show(const struct btf *btf, u32 type_id, void *obj,
5314 struct btf_show *show)
5315{
5316 const struct btf_type *t = btf_type_by_id(btf, type_id);
5317
5318 show->btf = btf;
5319 memset(&show->state, 0, sizeof(show->state));
5320 memset(&show->obj, 0, sizeof(show->obj));
5321
5322 btf_type_ops(t)->show(btf, t, type_id, obj, 0, show);
5323}
5324
5325static void btf_seq_show(struct btf_show *show, const char *fmt,
5326 va_list args)
5327{
5328 seq_vprintf((struct seq_file *)show->target, fmt, args);
5329}
5330
5331int btf_type_seq_show_flags(const struct btf *btf, u32 type_id,
5332 void *obj, struct seq_file *m, u64 flags)
5333{
5334 struct btf_show sseq;
5335
5336 sseq.target = m;
5337 sseq.showfn = btf_seq_show;
5338 sseq.flags = flags;
5339
5340 btf_type_show(btf, type_id, obj, &sseq);
5341
5342 return sseq.state.status;
5343}
5344
4519void btf_type_seq_show(const struct btf *btf, u32 type_id, void *obj,5345void btf_type_seq_show(const struct btf *btf, u32 type_id, void *obj,
4520 struct seq_file *m)5346 struct seq_file *m)
4521{5347{
4522 const struct btf_type *t = btf_type_by_id(btf, type_id);5348 (void) btf_type_seq_show_flags(btf, type_id, obj, m,
5349 BTF_SHOW_NONAME | BTF_SHOW_COMPACT |
5350 BTF_SHOW_ZERO | BTF_SHOW_UNSAFE);
5351}
5352
5353struct btf_show_snprintf {
5354 struct btf_show show;
5355 int len_left; /* space left in string */
5356 int len; /* length we would have written */
5357};
5358
5359static void btf_snprintf_show(struct btf_show *show, const char *fmt,
5360 va_list args)
5361{
5362 struct btf_show_snprintf *ssnprintf = (struct btf_show_snprintf *)show;
5363 int len;
4523 5364
4524 btf_type_ops(t)->seq_show(btf, t, type_id, obj, 0, m);5365 len = vsnprintf(show->target, ssnprintf->len_left, fmt, args);
5366
5367 if (len < 0) {
5368 ssnprintf->len_left = 0;
5369 ssnprintf->len = len;
5370 } else if (len > ssnprintf->len_left) {
5371 /* no space, drive on to get length we would have written */
5372 ssnprintf->len_left = 0;
5373 ssnprintf->len += len;
5374 } else {
5375 ssnprintf->len_left -= len;
5376 ssnprintf->len += len;
5377 show->target += len;
5378 }
5379}
5380
5381int btf_type_snprintf_show(const struct btf *btf, u32 type_id, void *obj,
5382 char *buf, int len, u64 flags)
5383{
5384 struct btf_show_snprintf ssnprintf;
5385
5386 ssnprintf.show.target = buf;
5387 ssnprintf.show.flags = flags;
5388 ssnprintf.show.showfn = btf_snprintf_show;
5389 ssnprintf.len_left = len;
5390 ssnprintf.len = 0;
5391
5392 btf_type_show(btf, type_id, obj, (struct btf_show *)&ssnprintf);
5393
5394 /* If we encontered an error, return it. */
5395 if (ssnprintf.show.state.status)
5396 return ssnprintf.show.state.status;
5397
5398 /* Otherwise return length we would have written */
5399 return ssnprintf.len;
4525}5400}
4526 5401
4527#ifdef CONFIG_PROC_FS5402#ifdef CONFIG_PROC_FS
@@ -4661,3 +5536,15 @@ u32 btf_id(const struct btf *btf) · 133 unmodified lines
4661{5536{
4662 return btf->id;5537 return btf->id;
4663}5538}
5539
5540static int btf_id_cmp_func(const void *a, const void *b)
5541{
5542 const int *pa = a, *pb = b;
5543
5544 return *pa - *pb;
5545}
5546
5547bool btf_id_set_contains(const struct btf_id_set *set, u32 id)
5548{
5549 return bsearch(&id, set->ids, set->cnt, sizeof(u32), btf_id_cmp_func) != NULL;
5550}
kernel/bpf/core.c+21−8
@@ -98,6 +98,8 @@ struct bpf_prog *bpf_prog_alloc_no_stats(unsigned int size, gfp_t gfp_extra_flag · 97 unmodified lines
98 fp->jit_requested = ebpf_jit_enabled();98 fp->jit_requested = ebpf_jit_enabled();
99 99
100 INIT_LIST_HEAD_RCU(&fp->aux->ksym.lnode);100 INIT_LIST_HEAD_RCU(&fp->aux->ksym.lnode);
101 mutex_init(&fp->aux->used_maps_mutex);
102 mutex_init(&fp->aux->dst_mutex);
101 103
102 return fp;104 return fp;
103}105}
@@ -253,6 +255,8 @@ struct bpf_prog *bpf_prog_realloc(struct bpf_prog *fp_old, unsigned int size, · 149 unmodified lines
253void __bpf_prog_free(struct bpf_prog *fp)255void __bpf_prog_free(struct bpf_prog *fp)
254{256{
255 if (fp->aux) {257 if (fp->aux) {
258 mutex_destroy(&fp->aux->used_maps_mutex);
259 mutex_destroy(&fp->aux->dst_mutex);
256 free_percpu(fp->aux->stats);260 free_percpu(fp->aux->stats);
257 kfree(fp->aux->poke_tab);261 kfree(fp->aux->poke_tab);
258 kfree(fp->aux);262 kfree(fp->aux);
@@ -773,7 +777,8 @@ int bpf_jit_add_poke_descriptor(struct bpf_prog *prog, · 514 unmodified lines
773 777
774 if (size > poke_tab_max)778 if (size > poke_tab_max)
775 return -ENOSPC;779 return -ENOSPC;
776 if (poke->ip || poke->ip_stable || poke->adj_off)780 if (poke->tailcall_target || poke->tailcall_target_stable ||
781 poke->tailcall_bypass || poke->adj_off || poke->bypass_addr)
777 return -EINVAL;782 return -EINVAL;
778 783
779 switch (poke->reason) {784 switch (poke->reason) {
@@ -1747,8 +1752,9 @@ bool bpf_prog_array_compatible(struct bpf_array *array, · 967 unmodified lines
1747static int bpf_check_tail_call(const struct bpf_prog *fp)1752static int bpf_check_tail_call(const struct bpf_prog *fp)
1748{1753{
1749 struct bpf_prog_aux *aux = fp->aux;1754 struct bpf_prog_aux *aux = fp->aux;
1750 int i;1755 int i, ret = 0;
1751 1756
1757 mutex_lock(&aux->used_maps_mutex);
1752 for (i = 0; i < aux->used_map_cnt; i++) {1758 for (i = 0; i < aux->used_map_cnt; i++) {
1753 struct bpf_map *map = aux->used_maps[i];1759 struct bpf_map *map = aux->used_maps[i];
1754 struct bpf_array *array;1760 struct bpf_array *array;
@@ -1757,11 +1763,15 @@ static int bpf_check_tail_call(const struct bpf_prog *fp) · 2 unmodified lines
1757 continue;1763 continue;
1758 1764
1759 array = container_of(map, struct bpf_array, map);1765 array = container_of(map, struct bpf_array, map);
1760 if (!bpf_prog_array_compatible(array, fp))1766 if (!bpf_prog_array_compatible(array, fp)) {
1761 return -EINVAL;1767 ret = -EINVAL;
1768 goto out;
1769 }
1762 }1770 }
1763 1771
1764 return 0;1772out:
1773 mutex_unlock(&aux->used_maps_mutex);
1774 return ret;
1765}1775}
1766 1776
1767static void bpf_prog_select_func(struct bpf_prog *fp)1777static void bpf_prog_select_func(struct bpf_prog *fp)
@@ -2130,7 +2140,8 @@ static void bpf_prog_free_deferred(struct work_struct *work) · 362 unmodified lines
2130 if (aux->prog->has_callchain_buf)2140 if (aux->prog->has_callchain_buf)
2131 put_callchain_buffers();2141 put_callchain_buffers();
2132#endif2142#endif
2133 bpf_trampoline_put(aux->trampoline);2143 if (aux->dst_trampoline)
2144 bpf_trampoline_put(aux->dst_trampoline);
2134 for (i = 0; i < aux->func_cnt; i++)2145 for (i = 0; i < aux->func_cnt; i++)
2135 bpf_jit_free(aux->func[i]);2146 bpf_jit_free(aux->func[i]);
2136 if (aux->func_cnt) {2147 if (aux->func_cnt) {
@@ -2146,8 +2157,8 @@ void bpf_prog_free(struct bpf_prog *fp) · 9 unmodified lines
2146{2157{
2147 struct bpf_prog_aux *aux = fp->aux;2158 struct bpf_prog_aux *aux = fp->aux;
2148 2159
2149 if (aux->linked_prog)2160 if (aux->dst_prog)
2150 bpf_prog_put(aux->linked_prog);2161 bpf_prog_put(aux->dst_prog);
2151 INIT_WORK(&aux->work, bpf_prog_free_deferred);2162 INIT_WORK(&aux->work, bpf_prog_free_deferred);
2152 schedule_work(&aux->work);2163 schedule_work(&aux->work);
2153}2164}
@@ -2208,6 +2219,8 @@ const struct bpf_func_proto bpf_get_current_cgroup_id_proto __weak; · 54 unmodified lines
2208const struct bpf_func_proto bpf_get_current_ancestor_cgroup_id_proto __weak;2219const struct bpf_func_proto bpf_get_current_ancestor_cgroup_id_proto __weak;
2209const struct bpf_func_proto bpf_get_local_storage_proto __weak;2220const struct bpf_func_proto bpf_get_local_storage_proto __weak;
2210const struct bpf_func_proto bpf_get_ns_current_pid_tgid_proto __weak;2221const struct bpf_func_proto bpf_get_ns_current_pid_tgid_proto __weak;
2222const struct bpf_func_proto bpf_snprintf_btf_proto __weak;
2223const struct bpf_func_proto bpf_seq_printf_btf_proto __weak;
2211 2224
2212const struct bpf_func_proto * __weak bpf_get_trace_printk_proto(void)2225const struct bpf_func_proto * __weak bpf_get_trace_printk_proto(void)
2213{2226{
kernel/bpf/cpumap.c+6−11
@@ -79,8 +79,6 @@ struct bpf_cpu_map { · 78 unmodified lines
79 79
80static DEFINE_PER_CPU(struct list_head, cpu_map_flush_list);80static DEFINE_PER_CPU(struct list_head, cpu_map_flush_list);
81 81
82static int bq_flush_to_queue(struct xdp_bulk_queue *bq);
83
84static struct bpf_map *cpu_map_alloc(union bpf_attr *attr)82static struct bpf_map *cpu_map_alloc(union bpf_attr *attr)
85{83{
86 u32 value_size = attr->value_size;84 u32 value_size = attr->value_size;
@@ -157,8 +155,7 @@ static void cpu_map_kthread_stop(struct work_struct *work) · 70 unmodified lines
157 kthread_stop(rcpu->kthread);155 kthread_stop(rcpu->kthread);
158}156}
159 157
160static struct sk_buff *cpu_map_build_skb(struct bpf_cpu_map_entry *rcpu,158static struct sk_buff *cpu_map_build_skb(struct xdp_frame *xdpf,
161 struct xdp_frame *xdpf,
162 struct sk_buff *skb)159 struct sk_buff *skb)
163{160{
164 unsigned int hard_start_headroom;161 unsigned int hard_start_headroom;
@@ -367,7 +364,7 @@ static int cpu_map_kthread_run(void *data) · 202 unmodified lines
367 struct sk_buff *skb = skbs[i];364 struct sk_buff *skb = skbs[i];
368 int ret;365 int ret;
369 366
370 skb = cpu_map_build_skb(rcpu, xdpf, skb);367 skb = cpu_map_build_skb(xdpf, skb);
371 if (!skb) {368 if (!skb) {
372 xdp_return_frame(xdpf);369 xdp_return_frame(xdpf);
373 continue;370 continue;
@@ -658,6 +655,7 @@ static int cpu_map_get_next_key(struct bpf_map *map, void *key, void *next_key) · 284 unmodified lines
658 655
659static int cpu_map_btf_id;656static int cpu_map_btf_id;
660const struct bpf_map_ops cpu_map_ops = {657const struct bpf_map_ops cpu_map_ops = {
658 .map_meta_equal = bpf_map_meta_equal,
661 .map_alloc = cpu_map_alloc,659 .map_alloc = cpu_map_alloc,
662 .map_free = cpu_map_free,660 .map_free = cpu_map_free,
663 .map_delete_elem = cpu_map_delete_elem,661 .map_delete_elem = cpu_map_delete_elem,
@@ -669,7 +667,7 @@ const struct bpf_map_ops cpu_map_ops = { · 5 unmodified lines
669 .map_btf_id = &cpu_map_btf_id,667 .map_btf_id = &cpu_map_btf_id,
670};668};
671 669
672static int bq_flush_to_queue(struct xdp_bulk_queue *bq)670static void bq_flush_to_queue(struct xdp_bulk_queue *bq)
673{671{
674 struct bpf_cpu_map_entry *rcpu = bq->obj;672 struct bpf_cpu_map_entry *rcpu = bq->obj;
675 unsigned int processed = 0, drops = 0;673 unsigned int processed = 0, drops = 0;
@@ -678,7 +676,7 @@ static int bq_flush_to_queue(struct xdp_bulk_queue *bq) · 2 unmodified lines
678 int i;676 int i;
679 677
680 if (unlikely(!bq->count))678 if (unlikely(!bq->count))
681 return 0;679 return;
682 680
683 q = rcpu->queue;681 q = rcpu->queue;
684 spin_lock(&q->producer_lock);682 spin_lock(&q->producer_lock);
@@ -701,13 +699,12 @@ static int bq_flush_to_queue(struct xdp_bulk_queue *bq) · 16 unmodified lines
701 699
702 /* Feedback loop via tracepoints */700 /* Feedback loop via tracepoints */
703 trace_xdp_cpumap_enqueue(rcpu->map_id, processed, drops, to_cpu);701 trace_xdp_cpumap_enqueue(rcpu->map_id, processed, drops, to_cpu);
704 return 0;
705}702}
706 703
707/* Runs under RCU-read-side, plus in softirq under NAPI protection.704/* Runs under RCU-read-side, plus in softirq under NAPI protection.
708 * Thus, safe percpu variable access.705 * Thus, safe percpu variable access.
709 */706 */
710static int bq_enqueue(struct bpf_cpu_map_entry *rcpu, struct xdp_frame *xdpf)707static void bq_enqueue(struct bpf_cpu_map_entry *rcpu, struct xdp_frame *xdpf)
711{708{
712 struct list_head *flush_list = this_cpu_ptr(&cpu_map_flush_list);709 struct list_head *flush_list = this_cpu_ptr(&cpu_map_flush_list);
713 struct xdp_bulk_queue *bq = this_cpu_ptr(rcpu->bulkq);710 struct xdp_bulk_queue *bq = this_cpu_ptr(rcpu->bulkq);
@@ -728,8 +725,6 @@ static int bq_enqueue(struct bpf_cpu_map_entry *rcpu, struct xdp_frame *xdpf) · 14 unmodified lines
728 725
729 if (!bq->flush_node.prev)726 if (!bq->flush_node.prev)
730 list_add(&bq->flush_node, flush_list);727 list_add(&bq->flush_node, flush_list);
731
732 return 0;
733}728}
734 729
735int cpu_map_enqueue(struct bpf_cpu_map_entry *rcpu, struct xdp_buff *xdp,730int cpu_map_enqueue(struct bpf_cpu_map_entry *rcpu, struct xdp_buff *xdp,
kernel/bpf/devmap.c+9−8
@@ -341,14 +341,14 @@ bool dev_map_can_have_prog(struct bpf_map *map) · 340 unmodified lines
341 return false;341 return false;
342}342}
343 343
344static int bq_xmit_all(struct xdp_dev_bulk_queue *bq, u32 flags)344static void bq_xmit_all(struct xdp_dev_bulk_queue *bq, u32 flags)
345{345{
346 struct net_device *dev = bq->dev;346 struct net_device *dev = bq->dev;
347 int sent = 0, drops = 0, err = 0;347 int sent = 0, drops = 0, err = 0;
348 int i;348 int i;
349 349
350 if (unlikely(!bq->count))350 if (unlikely(!bq->count))
351 return 0;351 return;
352 352
353 for (i = 0; i < bq->count; i++) {353 for (i = 0; i < bq->count; i++) {
354 struct xdp_frame *xdpf = bq->q[i];354 struct xdp_frame *xdpf = bq->q[i];
@@ -369,7 +369,7 @@ out: · 14 unmodified lines
369 trace_xdp_devmap_xmit(bq->dev_rx, dev, sent, drops, err);369 trace_xdp_devmap_xmit(bq->dev_rx, dev, sent, drops, err);
370 bq->dev_rx = NULL;370 bq->dev_rx = NULL;
371 __list_del_clearprev(&bq->flush_node);371 __list_del_clearprev(&bq->flush_node);
372 return 0;372 return;
373error:373error:
374 /* If ndo_xdp_xmit fails with an errno, no frames have been374 /* If ndo_xdp_xmit fails with an errno, no frames have been
375 * xmit'ed and it's our responsibility to them free all.375 * xmit'ed and it's our responsibility to them free all.
@@ -421,8 +421,8 @@ struct bpf_dtab_netdev *__dev_map_lookup_elem(struct bpf_map *map, u32 key) · 45 unmodified lines
421/* Runs under RCU-read-side, plus in softirq under NAPI protection.421/* Runs under RCU-read-side, plus in softirq under NAPI protection.
422 * Thus, safe percpu variable access.422 * Thus, safe percpu variable access.
423 */423 */
424static int bq_enqueue(struct net_device *dev, struct xdp_frame *xdpf,424static void bq_enqueue(struct net_device *dev, struct xdp_frame *xdpf,
425 struct net_device *dev_rx)425 struct net_device *dev_rx)
426{426{
427 struct list_head *flush_list = this_cpu_ptr(&dev_flush_list);427 struct list_head *flush_list = this_cpu_ptr(&dev_flush_list);
428 struct xdp_dev_bulk_queue *bq = this_cpu_ptr(dev->xdp_bulkq);428 struct xdp_dev_bulk_queue *bq = this_cpu_ptr(dev->xdp_bulkq);
@@ -441,8 +441,6 @@ static int bq_enqueue(struct net_device *dev, struct xdp_frame *xdpf, · 12 unmodified lines
441 441
442 if (!bq->flush_node.prev)442 if (!bq->flush_node.prev)
443 list_add(&bq->flush_node, flush_list);443 list_add(&bq->flush_node, flush_list);
444
445 return 0;
446}444}
447 445
448static inline int __xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp,446static inline int __xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp,
@@ -462,7 +460,8 @@ static inline int __xdp_enqueue(struct net_device *dev, struct xdp_buff *xdp, · 13 unmodified lines
462 if (unlikely(!xdpf))460 if (unlikely(!xdpf))
463 return -EOVERFLOW;461 return -EOVERFLOW;
464 462
465 return bq_enqueue(dev, xdpf, dev_rx);463 bq_enqueue(dev, xdpf, dev_rx);
464 return 0;
466}465}
467 466
468static struct xdp_buff *dev_map_run_prog(struct net_device *dev,467static struct xdp_buff *dev_map_run_prog(struct net_device *dev,
@@ -751,6 +750,7 @@ static int dev_map_hash_update_elem(struct bpf_map *map, void *key, void *value, · 282 unmodified lines
751 750
752static int dev_map_btf_id;751static int dev_map_btf_id;
753const struct bpf_map_ops dev_map_ops = {752const struct bpf_map_ops dev_map_ops = {
753 .map_meta_equal = bpf_map_meta_equal,
754 .map_alloc = dev_map_alloc,754 .map_alloc = dev_map_alloc,
755 .map_free = dev_map_free,755 .map_free = dev_map_free,
756 .map_get_next_key = dev_map_get_next_key,756 .map_get_next_key = dev_map_get_next_key,
@@ -764,6 +764,7 @@ const struct bpf_map_ops dev_map_ops = { · 7 unmodified lines
764 764
765static int dev_map_hash_map_btf_id;765static int dev_map_hash_map_btf_id;
766const struct bpf_map_ops dev_map_hash_ops = {766const struct bpf_map_ops dev_map_hash_ops = {
767 .map_meta_equal = bpf_map_meta_equal,
767 .map_alloc = dev_map_alloc,768 .map_alloc = dev_map_alloc,
768 .map_free = dev_map_free,769 .map_free = dev_map_free,
769 .map_get_next_key = dev_map_hash_get_next_key,770 .map_get_next_key = dev_map_hash_get_next_key,
kernel/bpf/hashtab.c+13−9
@@ -9,6 +9,7 @@ · 8 unmodified lines
9#include <linux/rculist_nulls.h>9#include <linux/rculist_nulls.h>
10#include <linux/random.h>10#include <linux/random.h>
11#include <uapi/linux/btf.h>11#include <uapi/linux/btf.h>
12#include <linux/rcupdate_trace.h>
12#include "percpu_freelist.h"13#include "percpu_freelist.h"
13#include "bpf_lru_list.h"14#include "bpf_lru_list.h"
14#include "map_in_map.h"15#include "map_in_map.h"
@@ -577,8 +578,7 @@ static void *__htab_map_lookup_elem(struct bpf_map *map, void *key) · 562 unmodified lines
577 struct htab_elem *l;578 struct htab_elem *l;
578 u32 hash, key_size;579 u32 hash, key_size;
579 580
580 /* Must be called with rcu_read_lock. */581 WARN_ON_ONCE(!rcu_read_lock_held() && !rcu_read_lock_trace_held());
581 WARN_ON_ONCE(!rcu_read_lock_held());
582 582
583 key_size = map->key_size;583 key_size = map->key_size;
584 584
@@ -612,7 +612,7 @@ static void *htab_map_lookup_elem(struct bpf_map *map, void *key) · 27 unmodified lines
612 * bpf_prog612 * bpf_prog
613 * __htab_map_lookup_elem613 * __htab_map_lookup_elem
614 */614 */
615static u32 htab_map_gen_lookup(struct bpf_map *map, struct bpf_insn *insn_buf)615static int htab_map_gen_lookup(struct bpf_map *map, struct bpf_insn *insn_buf)
616{616{
617 struct bpf_insn *insn = insn_buf;617 struct bpf_insn *insn = insn_buf;
618 const int ret = BPF_REG_0;618 const int ret = BPF_REG_0;
@@ -651,7 +651,7 @@ static void *htab_lru_map_lookup_elem_sys(struct bpf_map *map, void *key) · 32 unmodified lines
651 return __htab_lru_map_lookup_elem(map, key, false);651 return __htab_lru_map_lookup_elem(map, key, false);
652}652}
653 653
654static u32 htab_lru_map_gen_lookup(struct bpf_map *map,654static int htab_lru_map_gen_lookup(struct bpf_map *map,
655 struct bpf_insn *insn_buf)655 struct bpf_insn *insn_buf)
656{656{
657 struct bpf_insn *insn = insn_buf;657 struct bpf_insn *insn = insn_buf;
@@ -941,7 +941,7 @@ static int htab_map_update_elem(struct bpf_map *map, void *key, void *value, · 283 unmodified lines
941 /* unknown flags */941 /* unknown flags */
942 return -EINVAL;942 return -EINVAL;
943 943
944 WARN_ON_ONCE(!rcu_read_lock_held());944 WARN_ON_ONCE(!rcu_read_lock_held() && !rcu_read_lock_trace_held());
945 945
946 key_size = map->key_size;946 key_size = map->key_size;
947 947
@@ -1032,7 +1032,7 @@ static int htab_lru_map_update_elem(struct bpf_map *map, void *key, void *value, · 84 unmodified lines
1032 /* unknown flags */1032 /* unknown flags */
1033 return -EINVAL;1033 return -EINVAL;
1034 1034
1035 WARN_ON_ONCE(!rcu_read_lock_held());1035 WARN_ON_ONCE(!rcu_read_lock_held() && !rcu_read_lock_trace_held());
1036 1036
1037 key_size = map->key_size;1037 key_size = map->key_size;
1038 1038
@@ -1220,7 +1220,7 @@ static int htab_map_delete_elem(struct bpf_map *map, void *key) · 181 unmodified lines
1220 u32 hash, key_size;1220 u32 hash, key_size;
1221 int ret = -ENOENT;1221 int ret = -ENOENT;
1222 1222
1223 WARN_ON_ONCE(!rcu_read_lock_held());1223 WARN_ON_ONCE(!rcu_read_lock_held() && !rcu_read_lock_trace_held());
1224 1224
1225 key_size = map->key_size;1225 key_size = map->key_size;
1226 1226
@@ -1252,7 +1252,7 @@ static int htab_lru_map_delete_elem(struct bpf_map *map, void *key) · 25 unmodified lines
1252 u32 hash, key_size;1252 u32 hash, key_size;
1253 int ret = -ENOENT;1253 int ret = -ENOENT;
1254 1254
1255 WARN_ON_ONCE(!rcu_read_lock_held());1255 WARN_ON_ONCE(!rcu_read_lock_held() && !rcu_read_lock_trace_held());
1256 1256
1257 key_size = map->key_size;1257 key_size = map->key_size;
1258 1258
@@ -1803,6 +1803,7 @@ static const struct bpf_iter_seq_info iter_seq_info = { · 544 unmodified lines
1803 1803
1804static int htab_map_btf_id;1804static int htab_map_btf_id;
1805const struct bpf_map_ops htab_map_ops = {1805const struct bpf_map_ops htab_map_ops = {
1806 .map_meta_equal = bpf_map_meta_equal,
1806 .map_alloc_check = htab_map_alloc_check,1807 .map_alloc_check = htab_map_alloc_check,
1807 .map_alloc = htab_map_alloc,1808 .map_alloc = htab_map_alloc,
1808 .map_free = htab_map_free,1809 .map_free = htab_map_free,
@@ -1820,6 +1821,7 @@ const struct bpf_map_ops htab_map_ops = { · 11 unmodified lines
1820 1821
1821static int htab_lru_map_btf_id;1822static int htab_lru_map_btf_id;
1822const struct bpf_map_ops htab_lru_map_ops = {1823const struct bpf_map_ops htab_lru_map_ops = {
1824 .map_meta_equal = bpf_map_meta_equal,
1823 .map_alloc_check = htab_map_alloc_check,1825 .map_alloc_check = htab_map_alloc_check,
1824 .map_alloc = htab_map_alloc,1826 .map_alloc = htab_map_alloc,
1825 .map_free = htab_map_free,1827 .map_free = htab_map_free,
@@ -1940,6 +1942,7 @@ static void htab_percpu_map_seq_show_elem(struct bpf_map *map, void *key, · 114 unmodified lines
1940 1942
1941static int htab_percpu_map_btf_id;1943static int htab_percpu_map_btf_id;
1942const struct bpf_map_ops htab_percpu_map_ops = {1944const struct bpf_map_ops htab_percpu_map_ops = {
1945 .map_meta_equal = bpf_map_meta_equal,
1943 .map_alloc_check = htab_map_alloc_check,1946 .map_alloc_check = htab_map_alloc_check,
1944 .map_alloc = htab_map_alloc,1947 .map_alloc = htab_map_alloc,
1945 .map_free = htab_map_free,1948 .map_free = htab_map_free,
@@ -1956,6 +1959,7 @@ const struct bpf_map_ops htab_percpu_map_ops = { · 10 unmodified lines
1956 1959
1957static int htab_lru_percpu_map_btf_id;1960static int htab_lru_percpu_map_btf_id;
1958const struct bpf_map_ops htab_lru_percpu_map_ops = {1961const struct bpf_map_ops htab_lru_percpu_map_ops = {
1962 .map_meta_equal = bpf_map_meta_equal,
1959 .map_alloc_check = htab_map_alloc_check,1963 .map_alloc_check = htab_map_alloc_check,
1960 .map_alloc = htab_map_alloc,1964 .map_alloc = htab_map_alloc,
1961 .map_free = htab_map_free,1965 .map_free = htab_map_free,
@@ -2066,7 +2070,7 @@ static void *htab_of_map_lookup_elem(struct bpf_map *map, void *key) · 104 unmodified lines
2066 return READ_ONCE(*inner_map);2070 return READ_ONCE(*inner_map);
2067}2071}
2068 2072
2069static u32 htab_of_map_gen_lookup(struct bpf_map *map,2073static int htab_of_map_gen_lookup(struct bpf_map *map,
2070 struct bpf_insn *insn_buf)2074 struct bpf_insn *insn_buf)
2071{2075{
2072 struct bpf_insn *insn = insn_buf;2076 struct bpf_insn *insn = insn_buf;
kernel/bpf/helpers.c+58−0
@@ -601,6 +601,56 @@ const struct bpf_func_proto bpf_event_output_data_proto = { · 600 unmodified lines
601 .arg5_type = ARG_CONST_SIZE_OR_ZERO,601 .arg5_type = ARG_CONST_SIZE_OR_ZERO,
602};602};
603 603
604BPF_CALL_3(bpf_copy_from_user, void *, dst, u32, size,
605 const void __user *, user_ptr)
606{
607 int ret = copy_from_user(dst, user_ptr, size);
608
609 if (unlikely(ret)) {
610 memset(dst, 0, size);
611 ret = -EFAULT;
612 }
613
614 return ret;
615}
616
617const struct bpf_func_proto bpf_copy_from_user_proto = {
618 .func = bpf_copy_from_user,
619 .gpl_only = false,
620 .ret_type = RET_INTEGER,
621 .arg1_type = ARG_PTR_TO_UNINIT_MEM,
622 .arg2_type = ARG_CONST_SIZE_OR_ZERO,
623 .arg3_type = ARG_ANYTHING,
624};
625
626BPF_CALL_2(bpf_per_cpu_ptr, const void *, ptr, u32, cpu)
627{
628 if (cpu >= nr_cpu_ids)
629 return (unsigned long)NULL;
630
631 return (unsigned long)per_cpu_ptr((const void __percpu *)ptr, cpu);
632}
633
634const struct bpf_func_proto bpf_per_cpu_ptr_proto = {
635 .func = bpf_per_cpu_ptr,
636 .gpl_only = false,
637 .ret_type = RET_PTR_TO_MEM_OR_BTF_ID_OR_NULL,
638 .arg1_type = ARG_PTR_TO_PERCPU_BTF_ID,
639 .arg2_type = ARG_ANYTHING,
640};
641
642BPF_CALL_1(bpf_this_cpu_ptr, const void *, percpu_ptr)
643{
644 return (unsigned long)this_cpu_ptr((const void __percpu *)percpu_ptr);
645}
646
647const struct bpf_func_proto bpf_this_cpu_ptr_proto = {
648 .func = bpf_this_cpu_ptr,
649 .gpl_only = false,
650 .ret_type = RET_PTR_TO_MEM_OR_BTF_ID,
651 .arg1_type = ARG_PTR_TO_PERCPU_BTF_ID,
652};
653
604const struct bpf_func_proto bpf_get_current_task_proto __weak;654const struct bpf_func_proto bpf_get_current_task_proto __weak;
605const struct bpf_func_proto bpf_probe_read_user_proto __weak;655const struct bpf_func_proto bpf_probe_read_user_proto __weak;
606const struct bpf_func_proto bpf_probe_read_user_str_proto __weak;656const struct bpf_func_proto bpf_probe_read_user_str_proto __weak;
@@ -661,8 +711,16 @@ bpf_base_func_proto(enum bpf_func_id func_id) · 54 unmodified lines
661 if (!perfmon_capable())711 if (!perfmon_capable())
662 return NULL;712 return NULL;
663 return bpf_get_trace_printk_proto();713 return bpf_get_trace_printk_proto();
714 case BPF_FUNC_snprintf_btf:
715 if (!perfmon_capable())
716 return NULL;
717 return &bpf_snprintf_btf_proto;
664 case BPF_FUNC_jiffies64:718 case BPF_FUNC_jiffies64:
665 return &bpf_jiffies64_proto;719 return &bpf_jiffies64_proto;
720 case BPF_FUNC_bpf_per_cpu_ptr:
721 return &bpf_per_cpu_ptr_proto;
722 case BPF_FUNC_bpf_this_cpu_ptr:
723 return &bpf_this_cpu_ptr_proto;
666 default:724 default:
667 break;725 break;
668 }726 }
kernel/bpf/inode.c+113−3
@@ -20,6 +20,7 @@ · 19 unmodified lines
20#include <linux/filter.h>20#include <linux/filter.h>
21#include <linux/bpf.h>21#include <linux/bpf.h>
22#include <linux/bpf_trace.h>22#include <linux/bpf_trace.h>
23#include "preload/bpf_preload.h"
23 24
24enum bpf_type {25enum bpf_type {
25 BPF_TYPE_UNSPEC = 0,26 BPF_TYPE_UNSPEC = 0,
@@ -371,9 +372,10 @@ static struct dentry * · 345 unmodified lines
371bpf_lookup(struct inode *dir, struct dentry *dentry, unsigned flags)372bpf_lookup(struct inode *dir, struct dentry *dentry, unsigned flags)
372{373{
373 /* Dots in names (e.g. "/sys/fs/bpf/foo.bar") are reserved for future374 /* Dots in names (e.g. "/sys/fs/bpf/foo.bar") are reserved for future
374 * extensions.375 * extensions. That allows popoulate_bpffs() create special files.
375 */376 */
376 if (strchr(dentry->d_name.name, '.'))377 if ((dir->i_mode & S_IALLUGO) &&
378 strchr(dentry->d_name.name, '.'))
377 return ERR_PTR(-EPERM);379 return ERR_PTR(-EPERM);
378 380
379 return simple_lookup(dir, dentry, flags);381 return simple_lookup(dir, dentry, flags);
@@ -411,6 +413,27 @@ static const struct inode_operations bpf_dir_iops = { · 31 unmodified lines
411 .unlink = simple_unlink,413 .unlink = simple_unlink,
412};414};
413 415
416/* pin iterator link into bpffs */
417static int bpf_iter_link_pin_kernel(struct dentry *parent,
418 const char *name, struct bpf_link *link)
419{
420 umode_t mode = S_IFREG | S_IRUSR;
421 struct dentry *dentry;
422 int ret;
423
424 inode_lock(parent->d_inode);
425 dentry = lookup_one_len(name, parent, strlen(name));
426 if (IS_ERR(dentry)) {
427 inode_unlock(parent->d_inode);
428 return PTR_ERR(dentry);
429 }
430 ret = bpf_mkobj_ops(dentry, mode, link, &bpf_link_iops,
431 &bpf_iter_fops);
432 dput(dentry);
433 inode_unlock(parent->d_inode);
434 return ret;
435}
436
414static int bpf_obj_do_pin(const char __user *pathname, void *raw,437static int bpf_obj_do_pin(const char __user *pathname, void *raw,
415 enum bpf_type type)438 enum bpf_type type)
416{439{
@@ -640,6 +663,91 @@ static int bpf_parse_param(struct fs_context *fc, struct fs_parameter *param) · 223 unmodified lines
640 return 0;663 return 0;
641}664}
642 665
666struct bpf_preload_ops *bpf_preload_ops;
667EXPORT_SYMBOL_GPL(bpf_preload_ops);
668
669static bool bpf_preload_mod_get(void)
670{
671 /* If bpf_preload.ko wasn't loaded earlier then load it now.
672 * When bpf_preload is built into vmlinux the module's __init
673 * function will populate it.
674 */
675 if (!bpf_preload_ops) {
676 request_module("bpf_preload");
677 if (!bpf_preload_ops)
678 return false;
679 }
680 /* And grab the reference, so the module doesn't disappear while the
681 * kernel is interacting with the kernel module and its UMD.
682 */
683 if (!try_module_get(bpf_preload_ops->owner)) {
684 pr_err("bpf_preload module get failed.\n");
685 return false;
686 }
687 return true;
688}
689
690static void bpf_preload_mod_put(void)
691{
692 if (bpf_preload_ops)
693 /* now user can "rmmod bpf_preload" if necessary */
694 module_put(bpf_preload_ops->owner);
695}
696
697static DEFINE_MUTEX(bpf_preload_lock);
698
699static int populate_bpffs(struct dentry *parent)
700{
701 struct bpf_preload_info objs[BPF_PRELOAD_LINKS] = {};
702 struct bpf_link *links[BPF_PRELOAD_LINKS] = {};
703 int err = 0, i;
704
705 /* grab the mutex to make sure the kernel interactions with bpf_preload
706 * UMD are serialized
707 */
708 mutex_lock(&bpf_preload_lock);
709
710 /* if bpf_preload.ko wasn't built into vmlinux then load it */
711 if (!bpf_preload_mod_get())
712 goto out;
713
714 if (!bpf_preload_ops->info.tgid) {
715 /* preload() will start UMD that will load BPF iterator programs */
716 err = bpf_preload_ops->preload(objs);
717 if (err)
718 goto out_put;
719 for (i = 0; i < BPF_PRELOAD_LINKS; i++) {
720 links[i] = bpf_link_by_id(objs[i].link_id);
721 if (IS_ERR(links[i])) {
722 err = PTR_ERR(links[i]);
723 goto out_put;
724 }
725 }
726 for (i = 0; i < BPF_PRELOAD_LINKS; i++) {
727 err = bpf_iter_link_pin_kernel(parent,
728 objs[i].link_name, links[i]);
729 if (err)
730 goto out_put;
731 /* do not unlink successfully pinned links even
732 * if later link fails to pin
733 */
734 links[i] = NULL;
735 }
736 /* finish() will tell UMD process to exit */
737 err = bpf_preload_ops->finish();
738 if (err)
739 goto out_put;
740 }
741out_put:
742 bpf_preload_mod_put();
743out:
744 mutex_unlock(&bpf_preload_lock);
745 for (i = 0; i < BPF_PRELOAD_LINKS && err; i++)
746 if (!IS_ERR_OR_NULL(links[i]))
747 bpf_link_put(links[i]);
748 return err;
749}
750
643static int bpf_fill_super(struct super_block *sb, struct fs_context *fc)751static int bpf_fill_super(struct super_block *sb, struct fs_context *fc)
644{752{
645 static const struct tree_descr bpf_rfiles[] = { { "" } };753 static const struct tree_descr bpf_rfiles[] = { { "" } };
@@ -656,8 +764,8 @@ static int bpf_fill_super(struct super_block *sb, struct fs_context *fc) · 10 unmodified lines
656 inode = sb->s_root->d_inode;764 inode = sb->s_root->d_inode;
657 inode->i_op = &bpf_dir_iops;765 inode->i_op = &bpf_dir_iops;
658 inode->i_mode &= ~S_IALLUGO;766 inode->i_mode &= ~S_IALLUGO;
767 populate_bpffs(sb->s_root);
659 inode->i_mode |= S_ISVTX | opts->mode;768 inode->i_mode |= S_ISVTX | opts->mode;
660
661 return 0;769 return 0;
662}770}
663 771
@@ -707,6 +815,8 @@ static int __init bpf_init(void) · 43 unmodified lines
707{815{
708 int ret;816 int ret;
709 817
818 mutex_init(&bpf_preload_lock);
819
710 ret = sysfs_create_mount_point(fs_kobj, "bpf");820 ret = sysfs_create_mount_point(fs_kobj, "bpf");
711 if (ret)821 if (ret)
712 return ret;822 return ret;
kernel/bpf/lpm_trie.c+1−0
@@ -732,6 +732,7 @@ static int trie_check_btf(const struct bpf_map *map, · 731 unmodified lines
732 732
733static int trie_map_btf_id;733static int trie_map_btf_id;
734const struct bpf_map_ops trie_map_ops = {734const struct bpf_map_ops trie_map_ops = {
735 .map_meta_equal = bpf_map_meta_equal,
735 .map_alloc = trie_alloc,736 .map_alloc = trie_alloc,
736 .map_free = trie_free,737 .map_free = trie_free,
737 .map_get_next_key = trie_get_next_key,738 .map_get_next_key = trie_get_next_key,
kernel/bpf/map_in_map.c+9−15
@@ -17,23 +17,17 @@ struct bpf_map *bpf_map_meta_alloc(int inner_map_ufd) · 16 unmodified lines
17 if (IS_ERR(inner_map))17 if (IS_ERR(inner_map))
18 return inner_map;18 return inner_map;
19 19
20 /* prog_array->aux->{type,jited} is a runtime binding.
21 * Doing static check alone in the verifier is not enough.
22 */
23 if (inner_map->map_type == BPF_MAP_TYPE_PROG_ARRAY ||
24 inner_map->map_type == BPF_MAP_TYPE_CGROUP_STORAGE ||
25 inner_map->map_type == BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE ||
26 inner_map->map_type == BPF_MAP_TYPE_STRUCT_OPS) {
27 fdput(f);
28 return ERR_PTR(-ENOTSUPP);
29 }
30
31 /* Does not support >1 level map-in-map */20 /* Does not support >1 level map-in-map */
32 if (inner_map->inner_map_meta) {21 if (inner_map->inner_map_meta) {
33 fdput(f);22 fdput(f);
34 return ERR_PTR(-EINVAL);23 return ERR_PTR(-EINVAL);
35 }24 }
36 25
26 if (!inner_map->ops->map_meta_equal) {
27 fdput(f);
28 return ERR_PTR(-ENOTSUPP);
29 }
30
37 if (map_value_has_spin_lock(inner_map)) {31 if (map_value_has_spin_lock(inner_map)) {
38 fdput(f);32 fdput(f);
39 return ERR_PTR(-ENOTSUPP);33 return ERR_PTR(-ENOTSUPP);
@@ -81,15 +75,14 @@ bool bpf_map_meta_equal(const struct bpf_map *meta0, · 41 unmodified lines
81 return meta0->map_type == meta1->map_type &&75 return meta0->map_type == meta1->map_type &&
82 meta0->key_size == meta1->key_size &&76 meta0->key_size == meta1->key_size &&
83 meta0->value_size == meta1->value_size &&77 meta0->value_size == meta1->value_size &&
84 meta0->map_flags == meta1->map_flags &&78 meta0->map_flags == meta1->map_flags;
85 meta0->max_entries == meta1->max_entries;
86}79}
87 80
88void *bpf_map_fd_get_ptr(struct bpf_map *map,81void *bpf_map_fd_get_ptr(struct bpf_map *map,
89 struct file *map_file /* not used */,82 struct file *map_file /* not used */,
90 int ufd)83 int ufd)
91{84{
92 struct bpf_map *inner_map;85 struct bpf_map *inner_map, *inner_map_meta;
93 struct fd f;86 struct fd f;
94 87
95 f = fdget(ufd);88 f = fdget(ufd);
@@ -97,7 +90,8 @@ void *bpf_map_fd_get_ptr(struct bpf_map *map, · 1 unmodified line
97 if (IS_ERR(inner_map))90 if (IS_ERR(inner_map))
98 return inner_map;91 return inner_map;
99 92
100 if (bpf_map_meta_equal(map->inner_map_meta, inner_map))93 inner_map_meta = map->inner_map_meta;
94 if (inner_map_meta->ops->map_meta_equal(inner_map_meta, inner_map))
101 bpf_map_inc(inner_map);95 bpf_map_inc(inner_map);
102 else96 else
103 inner_map = ERR_PTR(-EINVAL);97 inner_map = ERR_PTR(-EINVAL);
kernel/bpf/map_in_map.h+0−2
@@ -11,8 +11,6 @@ struct bpf_map; · 10 unmodified lines
11 11
12struct bpf_map *bpf_map_meta_alloc(int inner_map_ufd);12struct bpf_map *bpf_map_meta_alloc(int inner_map_ufd);
13void bpf_map_meta_free(struct bpf_map *map_meta);13void bpf_map_meta_free(struct bpf_map *map_meta);
14bool bpf_map_meta_equal(const struct bpf_map *meta0,
15 const struct bpf_map *meta1);
16void *bpf_map_fd_get_ptr(struct bpf_map *map, struct file *map_file,14void *bpf_map_fd_get_ptr(struct bpf_map *map, struct file *map_file,
17 int ufd);15 int ufd);
18void bpf_map_fd_put_ptr(void *ptr);16void bpf_map_fd_put_ptr(void *ptr);
kernel/bpf/map_iter.c+15−0
@@ -149,6 +149,19 @@ static void bpf_iter_detach_map(struct bpf_iter_aux_info *aux) · 148 unmodified lines
149 bpf_map_put_with_uref(aux->map);149 bpf_map_put_with_uref(aux->map);
150}150}
151 151
152void bpf_iter_map_show_fdinfo(const struct bpf_iter_aux_info *aux,
153 struct seq_file *seq)
154{
155 seq_printf(seq, "map_id:\t%u\n", aux->map->id);
156}
157
158int bpf_iter_map_fill_link_info(const struct bpf_iter_aux_info *aux,
159 struct bpf_link_info *info)
160{
161 info->iter.map.map_id = aux->map->id;
162 return 0;
163}
164
152DEFINE_BPF_ITER_FUNC(bpf_map_elem, struct bpf_iter_meta *meta,165DEFINE_BPF_ITER_FUNC(bpf_map_elem, struct bpf_iter_meta *meta,
153 struct bpf_map *map, void *key, void *value)166 struct bpf_map *map, void *key, void *value)
154 167
@@ -156,6 +169,8 @@ static const struct bpf_iter_reg bpf_map_elem_reg_info = { · 1 unmodified line
156 .target = "bpf_map_elem",169 .target = "bpf_map_elem",
157 .attach_target = bpf_iter_attach_map,170 .attach_target = bpf_iter_attach_map,
158 .detach_target = bpf_iter_detach_map,171 .detach_target = bpf_iter_detach_map,
172 .show_fdinfo = bpf_iter_map_show_fdinfo,
173 .fill_link_info = bpf_iter_map_fill_link_info,
159 .ctx_arg_info_size = 2,174 .ctx_arg_info_size = 2,
160 .ctx_arg_info = {175 .ctx_arg_info = {
161 { offsetof(struct bpf_iter__bpf_map_elem, key),176 { offsetof(struct bpf_iter__bpf_map_elem, key),
kernel/bpf/percpu_freelist.c+96−5
@@ -17,6 +17,8 @@ int pcpu_freelist_init(struct pcpu_freelist *s) · 16 unmodified lines
17 raw_spin_lock_init(&head->lock);17 raw_spin_lock_init(&head->lock);
18 head->first = NULL;18 head->first = NULL;
19 }19 }
20 raw_spin_lock_init(&s->extralist.lock);
21 s->extralist.first = NULL;
20 return 0;22 return 0;
21}23}
22 24
@@ -40,12 +42,50 @@ static inline void ___pcpu_freelist_push(struct pcpu_freelist_head *head, · 17 unmodified lines
40 raw_spin_unlock(&head->lock);42 raw_spin_unlock(&head->lock);
41}43}
42 44
45static inline bool pcpu_freelist_try_push_extra(struct pcpu_freelist *s,
46 struct pcpu_freelist_node *node)
47{
48 if (!raw_spin_trylock(&s->extralist.lock))
49 return false;
50
51 pcpu_freelist_push_node(&s->extralist, node);
52 raw_spin_unlock(&s->extralist.lock);
53 return true;
54}
55
56static inline void ___pcpu_freelist_push_nmi(struct pcpu_freelist *s,
57 struct pcpu_freelist_node *node)
58{
59 int cpu, orig_cpu;
60
61 orig_cpu = cpu = raw_smp_processor_id();
62 while (1) {
63 struct pcpu_freelist_head *head;
64
65 head = per_cpu_ptr(s->freelist, cpu);
66 if (raw_spin_trylock(&head->lock)) {
67 pcpu_freelist_push_node(head, node);
68 raw_spin_unlock(&head->lock);
69 return;
70 }
71 cpu = cpumask_next(cpu, cpu_possible_mask);
72 if (cpu >= nr_cpu_ids)
73 cpu = 0;
74
75 /* cannot lock any per cpu lock, try extralist */
76 if (cpu == orig_cpu &&
77 pcpu_freelist_try_push_extra(s, node))
78 return;
79 }
80}
81
43void __pcpu_freelist_push(struct pcpu_freelist *s,82void __pcpu_freelist_push(struct pcpu_freelist *s,
44 struct pcpu_freelist_node *node)83 struct pcpu_freelist_node *node)
45{84{
46 struct pcpu_freelist_head *head = this_cpu_ptr(s->freelist);85 if (in_nmi())
47 86 ___pcpu_freelist_push_nmi(s, node);
48 ___pcpu_freelist_push(head, node);87 else
88 ___pcpu_freelist_push(this_cpu_ptr(s->freelist), node);
49}89}
50 90
51void pcpu_freelist_push(struct pcpu_freelist *s,91void pcpu_freelist_push(struct pcpu_freelist *s,
@@ -81,7 +121,7 @@ again: · 29 unmodified lines
81 }121 }
82}122}
83 123
84struct pcpu_freelist_node *__pcpu_freelist_pop(struct pcpu_freelist *s)124static struct pcpu_freelist_node *___pcpu_freelist_pop(struct pcpu_freelist *s)
85{125{
86 struct pcpu_freelist_head *head;126 struct pcpu_freelist_head *head;
87 struct pcpu_freelist_node *node;127 struct pcpu_freelist_node *node;
@@ -102,8 +142,59 @@ struct pcpu_freelist_node *__pcpu_freelist_pop(struct pcpu_freelist *s) · 14 unmodified lines
102 if (cpu >= nr_cpu_ids)142 if (cpu >= nr_cpu_ids)
103 cpu = 0;143 cpu = 0;
104 if (cpu == orig_cpu)144 if (cpu == orig_cpu)
105 return NULL;145 break;
146 }
147
148 /* per cpu lists are all empty, try extralist */
149 raw_spin_lock(&s->extralist.lock);
150 node = s->extralist.first;
151 if (node)
152 s->extralist.first = node->next;
153 raw_spin_unlock(&s->extralist.lock);
154 return node;
155}
156
157static struct pcpu_freelist_node *
158___pcpu_freelist_pop_nmi(struct pcpu_freelist *s)
159{
160 struct pcpu_freelist_head *head;
161 struct pcpu_freelist_node *node;
162 int orig_cpu, cpu;
163
164 orig_cpu = cpu = raw_smp_processor_id();
165 while (1) {
166 head = per_cpu_ptr(s->freelist, cpu);
167 if (raw_spin_trylock(&head->lock)) {
168 node = head->first;
169 if (node) {
170 head->first = node->next;
171 raw_spin_unlock(&head->lock);
172 return node;
173 }
174 raw_spin_unlock(&head->lock);
175 }
176 cpu = cpumask_next(cpu, cpu_possible_mask);
177 if (cpu >= nr_cpu_ids)
178 cpu = 0;
179 if (cpu == orig_cpu)
180 break;
106 }181 }
182
183 /* cannot pop from per cpu lists, try extralist */
184 if (!raw_spin_trylock(&s->extralist.lock))
185 return NULL;
186 node = s->extralist.first;
187 if (node)
188 s->extralist.first = node->next;
189 raw_spin_unlock(&s->extralist.lock);
190 return node;
191}
192
193struct pcpu_freelist_node *__pcpu_freelist_pop(struct pcpu_freelist *s)
194{
195 if (in_nmi())
196 return ___pcpu_freelist_pop_nmi(s);
197 return ___pcpu_freelist_pop(s);
107}198}
108 199
109struct pcpu_freelist_node *pcpu_freelist_pop(struct pcpu_freelist *s)200struct pcpu_freelist_node *pcpu_freelist_pop(struct pcpu_freelist *s)
kernel/bpf/percpu_freelist.h+1−0
@@ -13,6 +13,7 @@ struct pcpu_freelist_head { · 12 unmodified lines
13 13
14struct pcpu_freelist {14struct pcpu_freelist {
15 struct pcpu_freelist_head __percpu *freelist;15 struct pcpu_freelist_head __percpu *freelist;
16 struct pcpu_freelist_head extralist;
16};17};
17 18
18struct pcpu_freelist_node {19struct pcpu_freelist_node {
kernel/bpf/preload/.gitignore+4−0
@@ -0,0 +1,4 @@
1/FEATURE-DUMP.libbpf
2/bpf_helper_defs.h
3/feature
4/bpf_preload_umd
kernel/bpf/preload/Kconfig+26−0
@@ -0,0 +1,26 @@
1# SPDX-License-Identifier: GPL-2.0-only
2config USERMODE_DRIVER
3 bool
4 default n
5
6menuconfig BPF_PRELOAD
7 bool "Preload BPF file system with kernel specific program and map iterators"
8 depends on BPF
9 # The dependency on !COMPILE_TEST prevents it from being enabled
10 # in allmodconfig or allyesconfig configurations
11 depends on !COMPILE_TEST
12 select USERMODE_DRIVER
13 help
14 This builds kernel module with several embedded BPF programs that are
15 pinned into BPF FS mount point as human readable files that are
16 useful in debugging and introspection of BPF programs and maps.
17
18if BPF_PRELOAD
19config BPF_PRELOAD_UMD
20 tristate "bpf_preload kernel module with user mode driver"
21 depends on CC_CAN_LINK
22 depends on m || CC_CAN_LINK_STATIC
23 default m
24 help
25 This builds bpf_preload kernel module with embedded user mode driver.
26endif
kernel/bpf/preload/Makefile+25−0
@@ -0,0 +1,25 @@
1# SPDX-License-Identifier: GPL-2.0
2
3LIBBPF_SRCS = $(srctree)/tools/lib/bpf/
4LIBBPF_A = $(obj)/libbpf.a
5LIBBPF_OUT = $(abspath $(obj))
6
7$(LIBBPF_A):
8 $(Q)$(MAKE) -C $(LIBBPF_SRCS) OUTPUT=$(LIBBPF_OUT)/ $(LIBBPF_OUT)/libbpf.a
9
10userccflags += -I $(srctree)/tools/include/ -I $(srctree)/tools/include/uapi \
11 -I $(srctree)/tools/lib/ -Wno-unused-result
12
13userprogs := bpf_preload_umd
14
15clean-files := $(userprogs) bpf_helper_defs.h FEATURE-DUMP.libbpf staticobjs/ feature/
16
17bpf_preload_umd-objs := iterators/iterators.o
18bpf_preload_umd-userldlibs := $(LIBBPF_A) -lelf -lz
19
20$(obj)/bpf_preload_umd: $(LIBBPF_A)
21
22$(obj)/bpf_preload_umd_blob.o: $(obj)/bpf_preload_umd
23
24obj-$(CONFIG_BPF_PRELOAD_UMD) += bpf_preload.o
25bpf_preload-objs += bpf_preload_kern.o bpf_preload_umd_blob.o
kernel/bpf/preload/bpf_preload.h+16−0
@@ -0,0 +1,16 @@
1/* SPDX-License-Identifier: GPL-2.0 */
2#ifndef _BPF_PRELOAD_H
3#define _BPF_PRELOAD_H
4
5#include <linux/usermode_driver.h>
6#include "iterators/bpf_preload_common.h"
7
8struct bpf_preload_ops {
9 struct umd_info info;
10 int (*preload)(struct bpf_preload_info *);
11 int (*finish)(void);
12 struct module *owner;
13};
14extern struct bpf_preload_ops *bpf_preload_ops;
15#define BPF_PRELOAD_LINKS 2
16#endif
kernel/bpf/preload/bpf_preload_kern.c+91−0
@@ -0,0 +1,91 @@
1// SPDX-License-Identifier: GPL-2.0
2#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
3#include <linux/init.h>
4#include <linux/module.h>
5#include <linux/pid.h>
6#include <linux/fs.h>
7#include <linux/sched/signal.h>
8#include "bpf_preload.h"
9
10extern char bpf_preload_umd_start;
11extern char bpf_preload_umd_end;
12
13static int preload(struct bpf_preload_info *obj);
14static int finish(void);
15
16static struct bpf_preload_ops umd_ops = {
17 .info.driver_name = "bpf_preload",
18 .preload = preload,
19 .finish = finish,
20 .owner = THIS_MODULE,
21};
22
23static int preload(struct bpf_preload_info *obj)
24{
25 int magic = BPF_PRELOAD_START;
26 loff_t pos = 0;
27 int i, err;
28 ssize_t n;
29
30 err = fork_usermode_driver(&umd_ops.info);
31 if (err)
32 return err;
33
34 /* send the start magic to let UMD proceed with loading BPF progs */
35 n = kernel_write(umd_ops.info.pipe_to_umh,
36 &magic, sizeof(magic), &pos);
37 if (n != sizeof(magic))
38 return -EPIPE;
39
40 /* receive bpf_link IDs and names from UMD */
41 pos = 0;
42 for (i = 0; i < BPF_PRELOAD_LINKS; i++) {
43 n = kernel_read(umd_ops.info.pipe_from_umh,
44 &obj[i], sizeof(*obj), &pos);
45 if (n != sizeof(*obj))
46 return -EPIPE;
47 }
48 return 0;
49}
50
51static int finish(void)
52{
53 int magic = BPF_PRELOAD_END;
54 struct pid *tgid;
55 loff_t pos = 0;
56 ssize_t n;
57
58 /* send the last magic to UMD. It will do a normal exit. */
59 n = kernel_write(umd_ops.info.pipe_to_umh,
60 &magic, sizeof(magic), &pos);
61 if (n != sizeof(magic))
62 return -EPIPE;
63 tgid = umd_ops.info.tgid;
64 wait_event(tgid->wait_pidfd, thread_group_exited(tgid));
65 umd_ops.info.tgid = NULL;
66 return 0;
67}
68
69static int __init load_umd(void)
70{
71 int err;
72
73 err = umd_load_blob(&umd_ops.info, &bpf_preload_umd_start,
74 &bpf_preload_umd_end - &bpf_preload_umd_start);
75 if (err)
76 return err;
77 bpf_preload_ops = &umd_ops;
78 return err;
79}
80
81static void __exit fini_umd(void)
82{
83 bpf_preload_ops = NULL;
84 /* kill UMD in case it's still there due to earlier error */
85 kill_pid(umd_ops.info.tgid, SIGKILL, 1);
86 umd_ops.info.tgid = NULL;
87 umd_unload_blob(&umd_ops.info);
88}
89late_initcall(load_umd);
90module_exit(fini_umd);
91MODULE_LICENSE("GPL");
kernel/bpf/preload/bpf_preload_umd_blob.S+7−0
@@ -0,0 +1,7 @@
1/* SPDX-License-Identifier: GPL-2.0 */
2 .section .init.rodata, "a"
3 .global bpf_preload_umd_start
4bpf_preload_umd_start:
5 .incbin "kernel/bpf/preload/bpf_preload_umd"
6 .global bpf_preload_umd_end
7bpf_preload_umd_end:
kernel/bpf/preload/iterators/.gitignore+2−0
@@ -0,0 +1,2 @@
1# SPDX-License-Identifier: GPL-2.0-only
2/.output
kernel/bpf/preload/iterators/Makefile+57−0
@@ -0,0 +1,57 @@
1# SPDX-License-Identifier: GPL-2.0
2OUTPUT := .output
3CLANG ?= clang
4LLC ?= llc
5LLVM_STRIP ?= llvm-strip
6DEFAULT_BPFTOOL := $(OUTPUT)/sbin/bpftool
7BPFTOOL ?= $(DEFAULT_BPFTOOL)
8LIBBPF_SRC := $(abspath ../../../../tools/lib/bpf)
9BPFOBJ := $(OUTPUT)/libbpf.a
10BPF_INCLUDE := $(OUTPUT)
11INCLUDES := -I$(OUTPUT) -I$(BPF_INCLUDE) -I$(abspath ../../../../tools/lib) \
12 -I$(abspath ../../../../tools/include/uapi)
13CFLAGS := -g -Wall
14
15abs_out := $(abspath $(OUTPUT))
16ifeq ($(V),1)
17Q =
18msg =
19else
20Q = @
21msg = @printf ' %-8s %s%s\n' "$(1)" "$(notdir $(2))" "$(if $(3), $(3))";
22MAKEFLAGS += --no-print-directory
23submake_extras := feature_display=0
24endif
25
26.DELETE_ON_ERROR:
27
28.PHONY: all clean
29
30all: iterators.skel.h
31
32clean:
33 $(call msg,CLEAN)
34 $(Q)rm -rf $(OUTPUT) iterators
35
36iterators.skel.h: $(OUTPUT)/iterators.bpf.o | $(BPFTOOL)
37 $(call msg,GEN-SKEL,$@)
38 $(Q)$(BPFTOOL) gen skeleton $< > $@
39
40
41$(OUTPUT)/iterators.bpf.o: iterators.bpf.c $(BPFOBJ) | $(OUTPUT)
42 $(call msg,BPF,$@)
43 $(Q)$(CLANG) -g -O2 -target bpf $(INCLUDES) \
44 -c $(filter %.c,$^) -o $@ && \
45 $(LLVM_STRIP) -g $@
46
47$(OUTPUT):
48 $(call msg,MKDIR,$@)
49 $(Q)mkdir -p $(OUTPUT)
50
51$(BPFOBJ): $(wildcard $(LIBBPF_SRC)/*.[ch] $(LIBBPF_SRC)/Makefile) | $(OUTPUT)
52 $(Q)$(MAKE) $(submake_extras) -C $(LIBBPF_SRC) \
53 OUTPUT=$(abspath $(dir $@))/ $(abspath $@)
54
55$(DEFAULT_BPFTOOL):
56 $(Q)$(MAKE) $(submake_extras) -C ../../../../tools/bpf/bpftool \
57 prefix= OUTPUT=$(abs_out)/ DESTDIR=$(abs_out) install
kernel/bpf/preload/iterators/README+4−0
@@ -0,0 +1,4 @@
1WARNING:
2If you change "iterators.bpf.c" do "make -j" in this directory to rebuild "iterators.skel.h".
3Make sure to have clang 10 installed.
4See Documentation/bpf/bpf_devel_QA.rst
kernel/bpf/preload/iterators/bpf_preload_common.h+13−0
@@ -0,0 +1,13 @@
1/* SPDX-License-Identifier: GPL-2.0 */
2#ifndef _BPF_PRELOAD_COMMON_H
3#define _BPF_PRELOAD_COMMON_H
4
5#define BPF_PRELOAD_START 0x5555
6#define BPF_PRELOAD_END 0xAAAA
7
8struct bpf_preload_info {
9 char link_name[16];
10 int link_id;
11};
12
13#endif
kernel/bpf/preload/iterators/iterators.bpf.c+114−0
@@ -0,0 +1,114 @@
1// SPDX-License-Identifier: GPL-2.0
2/* Copyright (c) 2020 Facebook */
3#include <linux/bpf.h>
4#include <bpf/bpf_helpers.h>
5#include <bpf/bpf_tracing.h>
6#include <bpf/bpf_core_read.h>
7
8#pragma clang attribute push (__attribute__((preserve_access_index)), apply_to = record)
9struct seq_file;
10struct bpf_iter_meta {
11 struct seq_file *seq;
12 __u64 session_id;
13 __u64 seq_num;
14};
15
16struct bpf_map {
17 __u32 id;
18 char name[16];
19 __u32 max_entries;
20};
21
22struct bpf_iter__bpf_map {
23 struct bpf_iter_meta *meta;
24 struct bpf_map *map;
25};
26
27struct btf_type {
28 __u32 name_off;
29};
30
31struct btf_header {
32 __u32 str_len;
33};
34
35struct btf {
36 const char *strings;
37 struct btf_type **types;
38 struct btf_header hdr;
39};
40
41struct bpf_prog_aux {
42 __u32 id;
43 char name[16];
44 const char *attach_func_name;
45 struct bpf_prog *dst_prog;
46 struct bpf_func_info *func_info;
47 struct btf *btf;
48};
49
50struct bpf_prog {
51 struct bpf_prog_aux *aux;
52};
53
54struct bpf_iter__bpf_prog {
55 struct bpf_iter_meta *meta;
56 struct bpf_prog *prog;
57};
58#pragma clang attribute pop
59
60static const char *get_name(struct btf *btf, long btf_id, const char *fallback)
61{
62 struct btf_type **types, *t;
63 unsigned int name_off;
64 const char *str;
65
66 if (!btf)
67 return fallback;
68 str = btf->strings;
69 types = btf->types;
70 bpf_probe_read_kernel(&t, sizeof(t), types + btf_id);
71 name_off = BPF_CORE_READ(t, name_off);
72 if (name_off >= btf->hdr.str_len)
73 return fallback;
74 return str + name_off;
75}
76
77SEC("iter/bpf_map")
78int dump_bpf_map(struct bpf_iter__bpf_map *ctx)
79{
80 struct seq_file *seq = ctx->meta->seq;
81 __u64 seq_num = ctx->meta->seq_num;
82 struct bpf_map *map = ctx->map;
83
84 if (!map)
85 return 0;
86
87 if (seq_num == 0)
88 BPF_SEQ_PRINTF(seq, " id name max_entries\n");
89
90 BPF_SEQ_PRINTF(seq, "%4u %-16s%6d\n", map->id, map->name, map->max_entries);
91 return 0;
92}
93
94SEC("iter/bpf_prog")
95int dump_bpf_prog(struct bpf_iter__bpf_prog *ctx)
96{
97 struct seq_file *seq = ctx->meta->seq;
98 __u64 seq_num = ctx->meta->seq_num;
99 struct bpf_prog *prog = ctx->prog;
100 struct bpf_prog_aux *aux;
101
102 if (!prog)
103 return 0;
104
105 aux = prog->aux;
106 if (seq_num == 0)
107 BPF_SEQ_PRINTF(seq, " id name attached\n");
108
109 BPF_SEQ_PRINTF(seq, "%4u %-16s %s %s\n", aux->id,
110 get_name(aux->btf, aux->func_info[0].type_id, aux->name),
111 aux->attach_func_name, aux->dst_prog->aux->name);
112 return 0;
113}
114char LICENSE[] SEC("license") = "GPL";
kernel/bpf/preload/iterators/iterators.c+94−0
@@ -0,0 +1,94 @@
1// SPDX-License-Identifier: GPL-2.0
2/* Copyright (c) 2020 Facebook */
3#include <argp.h>
4#include <stdio.h>
5#include <stdlib.h>
6#include <string.h>
7#include <unistd.h>
8#include <fcntl.h>
9#include <sys/resource.h>
10#include <bpf/libbpf.h>
11#include <bpf/bpf.h>
12#include <sys/mount.h>
13#include "iterators.skel.h"
14#include "bpf_preload_common.h"
15
16int to_kernel = -1;
17int from_kernel = 0;
18
19static int send_link_to_kernel(struct bpf_link *link, const char *link_name)
20{
21 struct bpf_preload_info obj = {};
22 struct bpf_link_info info = {};
23 __u32 info_len = sizeof(info);
24 int err;
25
26 err = bpf_obj_get_info_by_fd(bpf_link__fd(link), &info, &info_len);
27 if (err)
28 return err;
29 obj.link_id = info.id;
30 if (strlen(link_name) >= sizeof(obj.link_name))
31 return -E2BIG;
32 strcpy(obj.link_name, link_name);
33 if (write(to_kernel, &obj, sizeof(obj)) != sizeof(obj))
34 return -EPIPE;
35 return 0;
36}
37
38int main(int argc, char **argv)
39{
40 struct rlimit rlim = { RLIM_INFINITY, RLIM_INFINITY };
41 struct iterators_bpf *skel;
42 int err, magic;
43 int debug_fd;
44
45 debug_fd = open("/dev/console", O_WRONLY | O_NOCTTY | O_CLOEXEC);
46 if (debug_fd < 0)
47 return 1;
48 to_kernel = dup(1);
49 close(1);
50 dup(debug_fd);
51 /* now stdin and stderr point to /dev/console */
52
53 read(from_kernel, &magic, sizeof(magic));
54 if (magic != BPF_PRELOAD_START) {
55 printf("bad start magic %d\n", magic);
56 return 1;
57 }
58 setrlimit(RLIMIT_MEMLOCK, &rlim);
59 /* libbpf opens BPF object and loads it into the kernel */
60 skel = iterators_bpf__open_and_load();
61 if (!skel) {
62 /* iterators.skel.h is little endian.
63 * libbpf doesn't support automatic little->big conversion
64 * of BPF bytecode yet.
65 * The program load will fail in such case.
66 */
67 printf("Failed load could be due to wrong endianness\n");
68 return 1;
69 }
70 err = iterators_bpf__attach(skel);
71 if (err)
72 goto cleanup;
73
74 /* send two bpf_link IDs with names to the kernel */
75 err = send_link_to_kernel(skel->links.dump_bpf_map, "maps.debug");
76 if (err)
77 goto cleanup;
78 err = send_link_to_kernel(skel->links.dump_bpf_prog, "progs.debug");
79 if (err)
80 goto cleanup;
81
82 /* The kernel will proceed with pinnging the links in bpffs.
83 * UMD will wait on read from pipe.
84 */
85 read(from_kernel, &magic, sizeof(magic));
86 if (magic != BPF_PRELOAD_END) {
87 printf("bad final magic %d\n", magic);
88 err = -EINVAL;
89 }
90cleanup:
91 iterators_bpf__destroy(skel);
92
93 return err != 0;
94}
kernel/bpf/preload/iterators/iterators.skel.h+412−0
@@ -0,0 +1,412 @@
1/* SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause) */
2
3/* THIS FILE IS AUTOGENERATED! */
4#ifndef __ITERATORS_BPF_SKEL_H__
5#define __ITERATORS_BPF_SKEL_H__
6
7#include <stdlib.h>
8#include <bpf/libbpf.h>
9
10struct iterators_bpf {
11 struct bpf_object_skeleton *skeleton;
12 struct bpf_object *obj;
13 struct {
14 struct bpf_map *rodata;
15 } maps;
16 struct {
17 struct bpf_program *dump_bpf_map;
18 struct bpf_program *dump_bpf_prog;
19 } progs;
20 struct {
21 struct bpf_link *dump_bpf_map;
22 struct bpf_link *dump_bpf_prog;
23 } links;
24 struct iterators_bpf__rodata {
25 char dump_bpf_map____fmt[35];
26 char dump_bpf_map____fmt_1[14];
27 char dump_bpf_prog____fmt[32];
28 char dump_bpf_prog____fmt_2[17];
29 } *rodata;
30};
31
32static void
33iterators_bpf__destroy(struct iterators_bpf *obj)
34{
35 if (!obj)
36 return;
37 if (obj->skeleton)
38 bpf_object__destroy_skeleton(obj->skeleton);
39 free(obj);
40}
41
42static inline int
43iterators_bpf__create_skeleton(struct iterators_bpf *obj);
44
45static inline struct iterators_bpf *
46iterators_bpf__open_opts(const struct bpf_object_open_opts *opts)
47{
48 struct iterators_bpf *obj;
49
50 obj = (struct iterators_bpf *)calloc(1, sizeof(*obj));
51 if (!obj)
52 return NULL;
53 if (iterators_bpf__create_skeleton(obj))
54 goto err;
55 if (bpf_object__open_skeleton(obj->skeleton, opts))
56 goto err;
57
58 return obj;
59err:
60 iterators_bpf__destroy(obj);
61 return NULL;
62}
63
64static inline struct iterators_bpf *
65iterators_bpf__open(void)
66{
67 return iterators_bpf__open_opts(NULL);
68}
69
70static inline int
71iterators_bpf__load(struct iterators_bpf *obj)
72{
73 return bpf_object__load_skeleton(obj->skeleton);
74}
75
76static inline struct iterators_bpf *
77iterators_bpf__open_and_load(void)
78{
79 struct iterators_bpf *obj;
80
81 obj = iterators_bpf__open();
82 if (!obj)
83 return NULL;
84 if (iterators_bpf__load(obj)) {
85 iterators_bpf__destroy(obj);
86 return NULL;
87 }
88 return obj;
89}
90
91static inline int
92iterators_bpf__attach(struct iterators_bpf *obj)
93{
94 return bpf_object__attach_skeleton(obj->skeleton);
95}
96
97static inline void
98iterators_bpf__detach(struct iterators_bpf *obj)
99{
100 return bpf_object__detach_skeleton(obj->skeleton);
101}
102
103static inline int
104iterators_bpf__create_skeleton(struct iterators_bpf *obj)
105{
106 struct bpf_object_skeleton *s;
107
108 s = (struct bpf_object_skeleton *)calloc(1, sizeof(*s));
109 if (!s)
110 return -1;
111 obj->skeleton = s;
112
113 s->sz = sizeof(*s);
114 s->name = "iterators_bpf";
115 s->obj = &obj->obj;
116
117 /* maps */
118 s->map_cnt = 1;
119 s->map_skel_sz = sizeof(*s->maps);
120 s->maps = (struct bpf_map_skeleton *)calloc(s->map_cnt, s->map_skel_sz);
121 if (!s->maps)
122 goto err;
123
124 s->maps[0].name = "iterator.rodata";
125 s->maps[0].map = &obj->maps.rodata;
126 s->maps[0].mmaped = (void **)&obj->rodata;
127
128 /* programs */
129 s->prog_cnt = 2;
130 s->prog_skel_sz = sizeof(*s->progs);
131 s->progs = (struct bpf_prog_skeleton *)calloc(s->prog_cnt, s->prog_skel_sz);
132 if (!s->progs)
133 goto err;
134
135 s->progs[0].name = "dump_bpf_map";
136 s->progs[0].prog = &obj->progs.dump_bpf_map;
137 s->progs[0].link = &obj->links.dump_bpf_map;
138
139 s->progs[1].name = "dump_bpf_prog";
140 s->progs[1].prog = &obj->progs.dump_bpf_prog;
141 s->progs[1].link = &obj->links.dump_bpf_prog;
142
143 s->data_sz = 7176;
144 s->data = (void *)"\
145\x7f\x45\x4c\x46\x02\x01\x01\0\0\0\0\0\0\0\0\0\x01\0\xf7\0\x01\0\0\0\0\0\0\0\0\
146\0\0\0\0\0\0\0\0\0\0\0\x48\x18\0\0\0\0\0\0\0\0\0\0\x40\0\0\0\0\0\x40\0\x0f\0\
147\x0e\0\x79\x12\0\0\0\0\0\0\x79\x26\0\0\0\0\0\0\x79\x17\x08\0\0\0\0\0\x15\x07\
148\x1a\0\0\0\0\0\x79\x21\x10\0\0\0\0\0\x55\x01\x08\0\0\0\0\0\xbf\xa4\0\0\0\0\0\0\
149\x07\x04\0\0\xe8\xff\xff\xff\xbf\x61\0\0\0\0\0\0\x18\x02\0\0\0\0\0\0\0\0\0\0\0\
150\0\0\0\xb7\x03\0\0\x23\0\0\0\xb7\x05\0\0\0\0\0\0\x85\0\0\0\x7e\0\0\0\x61\x71\0\
151\0\0\0\0\0\x7b\x1a\xe8\xff\0\0\0\0\xb7\x01\0\0\x04\0\0\0\xbf\x72\0\0\0\0\0\0\
152\x0f\x12\0\0\0\0\0\0\x7b\x2a\xf0\xff\0\0\0\0\x61\x71\x14\0\0\0\0\0\x7b\x1a\xf8\
153\xff\0\0\0\0\xbf\xa4\0\0\0\0\0\0\x07\x04\0\0\xe8\xff\xff\xff\xbf\x61\0\0\0\0\0\
154\0\x18\x02\0\0\x23\0\0\0\0\0\0\0\0\0\0\0\xb7\x03\0\0\x0e\0\0\0\xb7\x05\0\0\x18\
155\0\0\0\x85\0\0\0\x7e\0\0\0\xb7\0\0\0\0\0\0\0\x95\0\0\0\0\0\0\0\x79\x12\0\0\0\0\
156\0\0\x79\x26\0\0\0\0\0\0\x79\x11\x08\0\0\0\0\0\x15\x01\x3b\0\0\0\0\0\x79\x17\0\
157\0\0\0\0\0\x79\x21\x10\0\0\0\0\0\x55\x01\x08\0\0\0\0\0\xbf\xa4\0\0\0\0\0\0\x07\
158\x04\0\0\xd0\xff\xff\xff\xbf\x61\0\0\0\0\0\0\x18\x02\0\0\x31\0\0\0\0\0\0\0\0\0\
159\0\0\xb7\x03\0\0\x20\0\0\0\xb7\x05\0\0\0\0\0\0\x85\0\0\0\x7e\0\0\0\x7b\x6a\xc8\
160\xff\0\0\0\0\x61\x71\0\0\0\0\0\0\x7b\x1a\xd0\xff\0\0\0\0\xb7\x03\0\0\x04\0\0\0\
161\xbf\x79\0\0\0\0\0\0\x0f\x39\0\0\0\0\0\0\x79\x71\x28\0\0\0\0\0\x79\x78\x30\0\0\
162\0\0\0\x15\x08\x18\0\0\0\0\0\xb7\x02\0\0\0\0\0\0\x0f\x21\0\0\0\0\0\0\x61\x11\
163\x04\0\0\0\0\0\x79\x83\x08\0\0\0\0\0\x67\x01\0\0\x03\0\0\0\x0f\x13\0\0\0\0\0\0\
164\x79\x86\0\0\0\0\0\0\xbf\xa1\0\0\0\0\0\0\x07\x01\0\0\xf8\xff\xff\xff\xb7\x02\0\
165\0\x08\0\0\0\x85\0\0\0\x71\0\0\0\xb7\x01\0\0\0\0\0\0\x79\xa3\xf8\xff\0\0\0\0\
166\x0f\x13\0\0\0\0\0\0\xbf\xa1\0\0\0\0\0\0\x07\x01\0\0\xf4\xff\xff\xff\xb7\x02\0\
167\0\x04\0\0\0\x85\0\0\0\x71\0\0\0\xb7\x03\0\0\x04\0\0\0\x61\xa1\xf4\xff\0\0\0\0\
168\x61\x82\x10\0\0\0\0\0\x3d\x21\x02\0\0\0\0\0\x0f\x16\0\0\0\0\0\0\xbf\x69\0\0\0\
169\0\0\0\x7b\x9a\xd8\xff\0\0\0\0\x79\x71\x18\0\0\0\0\0\x7b\x1a\xe0\xff\0\0\0\0\
170\x79\x71\x20\0\0\0\0\0\x79\x11\0\0\0\0\0\0\x0f\x31\0\0\0\0\0\0\x7b\x1a\xe8\xff\
171\0\0\0\0\xbf\xa4\0\0\0\0\0\0\x07\x04\0\0\xd0\xff\xff\xff\x79\xa1\xc8\xff\0\0\0\
172\0\x18\x02\0\0\x51\0\0\0\0\0\0\0\0\0\0\0\xb7\x03\0\0\x11\0\0\0\xb7\x05\0\0\x20\
173\0\0\0\x85\0\0\0\x7e\0\0\0\xb7\0\0\0\0\0\0\0\x95\0\0\0\0\0\0\0\x20\x20\x69\x64\
174\x20\x6e\x61\x6d\x65\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x6d\
175\x61\x78\x5f\x65\x6e\x74\x72\x69\x65\x73\x0a\0\x25\x34\x75\x20\x25\x2d\x31\x36\
176\x73\x25\x36\x64\x0a\0\x20\x20\x69\x64\x20\x6e\x61\x6d\x65\x20\x20\x20\x20\x20\
177\x20\x20\x20\x20\x20\x20\x20\x20\x61\x74\x74\x61\x63\x68\x65\x64\x0a\0\x25\x34\
178\x75\x20\x25\x2d\x31\x36\x73\x20\x25\x73\x20\x25\x73\x0a\0\x47\x50\x4c\0\x9f\
179\xeb\x01\0\x18\0\0\0\0\0\0\0\x1c\x04\0\0\x1c\x04\0\0\x09\x05\0\0\0\0\0\0\0\0\0\
180\x02\x02\0\0\0\x01\0\0\0\x02\0\0\x04\x10\0\0\0\x13\0\0\0\x03\0\0\0\0\0\0\0\x18\
181\0\0\0\x04\0\0\0\x40\0\0\0\0\0\0\0\0\0\0\x02\x08\0\0\0\0\0\0\0\0\0\0\x02\x0d\0\
182\0\0\0\0\0\0\x01\0\0\x0d\x06\0\0\0\x1c\0\0\0\x01\0\0\0\x20\0\0\0\0\0\0\x01\x04\
183\0\0\0\x20\0\0\x01\x24\0\0\0\x01\0\0\x0c\x05\0\0\0\xaf\0\0\0\x03\0\0\x04\x18\0\
184\0\0\xbd\0\0\0\x09\0\0\0\0\0\0\0\xc1\0\0\0\x0b\0\0\0\x40\0\0\0\xcc\0\0\0\x0b\0\
185\0\0\x80\0\0\0\0\0\0\0\0\0\0\x02\x0a\0\0\0\xd4\0\0\0\0\0\0\x07\0\0\0\0\xdd\0\0\
186\0\0\0\0\x08\x0c\0\0\0\xe3\0\0\0\0\0\0\x01\x08\0\0\0\x40\0\0\0\xa4\x01\0\0\x03\
187\0\0\x04\x18\0\0\0\xac\x01\0\0\x0e\0\0\0\0\0\0\0\xaf\x01\0\0\x11\0\0\0\x20\0\0\
188\0\xb4\x01\0\0\x0e\0\0\0\xa0\0\0\0\xc0\x01\0\0\0\0\0\x08\x0f\0\0\0\xc6\x01\0\0\
189\0\0\0\x01\x04\0\0\0\x20\0\0\0\xd3\x01\0\0\0\0\0\x01\x01\0\0\0\x08\0\0\x01\0\0\
190\0\0\0\0\0\x03\0\0\0\0\x10\0\0\0\x12\0\0\0\x10\0\0\0\xd8\x01\0\0\0\0\0\x01\x04\
191\0\0\0\x20\0\0\0\0\0\0\0\0\0\0\x02\x14\0\0\0\x3c\x02\0\0\x02\0\0\x04\x10\0\0\0\
192\x13\0\0\0\x03\0\0\0\0\0\0\0\x4f\x02\0\0\x15\0\0\0\x40\0\0\0\0\0\0\0\0\0\0\x02\
193\x18\0\0\0\0\0\0\0\x01\0\0\x0d\x06\0\0\0\x1c\0\0\0\x13\0\0\0\x54\x02\0\0\x01\0\
194\0\x0c\x16\0\0\0\xa0\x02\0\0\x01\0\0\x04\x08\0\0\0\xa9\x02\0\0\x19\0\0\0\0\0\0\
195\0\0\0\0\0\0\0\0\x02\x1a\0\0\0\xfa\x02\0\0\x06\0\0\x04\x38\0\0\0\xac\x01\0\0\
196\x0e\0\0\0\0\0\0\0\xaf\x01\0\0\x11\0\0\0\x20\0\0\0\x07\x03\0\0\x1b\0\0\0\xc0\0\
197\0\0\x18\x03\0\0\x15\0\0\0\0\x01\0\0\x21\x03\0\0\x1d\0\0\0\x40\x01\0\0\x2b\x03\
198\0\0\x1e\0\0\0\x80\x01\0\0\0\0\0\0\0\0\0\x02\x1c\0\0\0\0\0\0\0\0\0\0\x0a\x10\0\
199\0\0\0\0\0\0\0\0\0\x02\x1f\0\0\0\0\0\0\0\0\0\0\x02\x20\0\0\0\x75\x03\0\0\x02\0\
200\0\x04\x08\0\0\0\x83\x03\0\0\x0e\0\0\0\0\0\0\0\x8c\x03\0\0\x0e\0\0\0\x20\0\0\0\
201\x2b\x03\0\0\x03\0\0\x04\x18\0\0\0\x96\x03\0\0\x1b\0\0\0\0\0\0\0\x9e\x03\0\0\
202\x21\0\0\0\x40\0\0\0\xa4\x03\0\0\x23\0\0\0\x80\0\0\0\0\0\0\0\0\0\0\x02\x22\0\0\
203\0\0\0\0\0\0\0\0\x02\x24\0\0\0\xa8\x03\0\0\x01\0\0\x04\x04\0\0\0\xb3\x03\0\0\
204\x0e\0\0\0\0\0\0\0\x1c\x04\0\0\x01\0\0\x04\x04\0\0\0\x25\x04\0\0\x0e\0\0\0\0\0\
205\0\0\0\0\0\0\0\0\0\x03\0\0\0\0\x1c\0\0\0\x12\0\0\0\x23\0\0\0\x9b\x04\0\0\0\0\0\
206\x0e\x25\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x03\0\0\0\0\x1c\0\0\0\x12\0\0\0\x0e\0\0\0\
207\xaf\x04\0\0\0\0\0\x0e\x27\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x03\0\0\0\0\x1c\0\0\0\
208\x12\0\0\0\x20\0\0\0\xc5\x04\0\0\0\0\0\x0e\x29\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x03\
209\0\0\0\0\x1c\0\0\0\x12\0\0\0\x11\0\0\0\xda\x04\0\0\0\0\0\x0e\x2b\0\0\0\0\0\0\0\
210\0\0\0\0\0\0\0\x03\0\0\0\0\x10\0\0\0\x12\0\0\0\x04\0\0\0\xf1\x04\0\0\0\0\0\x0e\
211\x2d\0\0\0\x01\0\0\0\xf9\x04\0\0\x04\0\0\x0f\0\0\0\0\x26\0\0\0\0\0\0\0\x23\0\0\
212\0\x28\0\0\0\x23\0\0\0\x0e\0\0\0\x2a\0\0\0\x31\0\0\0\x20\0\0\0\x2c\0\0\0\x51\0\
213\0\0\x11\0\0\0\x01\x05\0\0\x01\0\0\x0f\0\0\0\0\x2e\0\0\0\0\0\0\0\x04\0\0\0\0\
214\x62\x70\x66\x5f\x69\x74\x65\x72\x5f\x5f\x62\x70\x66\x5f\x6d\x61\x70\0\x6d\x65\
215\x74\x61\0\x6d\x61\x70\0\x63\x74\x78\0\x69\x6e\x74\0\x64\x75\x6d\x70\x5f\x62\
216\x70\x66\x5f\x6d\x61\x70\0\x69\x74\x65\x72\x2f\x62\x70\x66\x5f\x6d\x61\x70\0\
217\x30\x3a\x30\0\x2f\x68\x6f\x6d\x65\x2f\x61\x6c\x72\x75\x61\x2f\x62\x75\x69\x6c\
218\x64\x2f\x6c\x69\x6e\x75\x78\x2f\x6b\x65\x72\x6e\x65\x6c\x2f\x62\x70\x66\x2f\
219\x70\x72\x65\x6c\x6f\x61\x64\x2f\x69\x74\x65\x72\x61\x74\x6f\x72\x73\x2f\x69\
220\x74\x65\x72\x61\x74\x6f\x72\x73\x2e\x62\x70\x66\x2e\x63\0\x09\x73\x74\x72\x75\
221\x63\x74\x20\x73\x65\x71\x5f\x66\x69\x6c\x65\x20\x2a\x73\x65\x71\x20\x3d\x20\
222\x63\x74\x78\x2d\x3e\x6d\x65\x74\x61\x2d\x3e\x73\x65\x71\x3b\0\x62\x70\x66\x5f\
223\x69\x74\x65\x72\x5f\x6d\x65\x74\x61\0\x73\x65\x71\0\x73\x65\x73\x73\x69\x6f\
224\x6e\x5f\x69\x64\0\x73\x65\x71\x5f\x6e\x75\x6d\0\x73\x65\x71\x5f\x66\x69\x6c\
225\x65\0\x5f\x5f\x75\x36\x34\0\x6c\x6f\x6e\x67\x20\x6c\x6f\x6e\x67\x20\x75\x6e\
226\x73\x69\x67\x6e\x65\x64\x20\x69\x6e\x74\0\x30\x3a\x31\0\x09\x73\x74\x72\x75\
227\x63\x74\x20\x62\x70\x66\x5f\x6d\x61\x70\x20\x2a\x6d\x61\x70\x20\x3d\x20\x63\
228\x74\x78\x2d\x3e\x6d\x61\x70\x3b\0\x09\x69\x66\x20\x28\x21\x6d\x61\x70\x29\0\
229\x30\x3a\x32\0\x09\x5f\x5f\x75\x36\x34\x20\x73\x65\x71\x5f\x6e\x75\x6d\x20\x3d\
230\x20\x63\x74\x78\x2d\x3e\x6d\x65\x74\x61\x2d\x3e\x73\x65\x71\x5f\x6e\x75\x6d\
231\x3b\0\x09\x69\x66\x20\x28\x73\x65\x71\x5f\x6e\x75\x6d\x20\x3d\x3d\x20\x30\x29\
232\0\x09\x09\x42\x50\x46\x5f\x53\x45\x51\x5f\x50\x52\x49\x4e\x54\x46\x28\x73\x65\
233\x71\x2c\x20\x22\x20\x20\x69\x64\x20\x6e\x61\x6d\x65\x20\x20\x20\x20\x20\x20\
234\x20\x20\x20\x20\x20\x20\x20\x6d\x61\x78\x5f\x65\x6e\x74\x72\x69\x65\x73\x5c\
235\x6e\x22\x29\x3b\0\x62\x70\x66\x5f\x6d\x61\x70\0\x69\x64\0\x6e\x61\x6d\x65\0\
236\x6d\x61\x78\x5f\x65\x6e\x74\x72\x69\x65\x73\0\x5f\x5f\x75\x33\x32\0\x75\x6e\
237\x73\x69\x67\x6e\x65\x64\x20\x69\x6e\x74\0\x63\x68\x61\x72\0\x5f\x5f\x41\x52\
238\x52\x41\x59\x5f\x53\x49\x5a\x45\x5f\x54\x59\x50\x45\x5f\x5f\0\x09\x42\x50\x46\
239\x5f\x53\x45\x51\x5f\x50\x52\x49\x4e\x54\x46\x28\x73\x65\x71\x2c\x20\x22\x25\
240\x34\x75\x20\x25\x2d\x31\x36\x73\x25\x36\x64\x5c\x6e\x22\x2c\x20\x6d\x61\x70\
241\x2d\x3e\x69\x64\x2c\x20\x6d\x61\x70\x2d\x3e\x6e\x61\x6d\x65\x2c\x20\x6d\x61\
242\x70\x2d\x3e\x6d\x61\x78\x5f\x65\x6e\x74\x72\x69\x65\x73\x29\x3b\0\x7d\0\x62\
243\x70\x66\x5f\x69\x74\x65\x72\x5f\x5f\x62\x70\x66\x5f\x70\x72\x6f\x67\0\x70\x72\
244\x6f\x67\0\x64\x75\x6d\x70\x5f\x62\x70\x66\x5f\x70\x72\x6f\x67\0\x69\x74\x65\
245\x72\x2f\x62\x70\x66\x5f\x70\x72\x6f\x67\0\x09\x73\x74\x72\x75\x63\x74\x20\x62\
246\x70\x66\x5f\x70\x72\x6f\x67\x20\x2a\x70\x72\x6f\x67\x20\x3d\x20\x63\x74\x78\
247\x2d\x3e\x70\x72\x6f\x67\x3b\0\x09\x69\x66\x20\x28\x21\x70\x72\x6f\x67\x29\0\
248\x62\x70\x66\x5f\x70\x72\x6f\x67\0\x61\x75\x78\0\x09\x61\x75\x78\x20\x3d\x20\
249\x70\x72\x6f\x67\x2d\x3e\x61\x75\x78\x3b\0\x09\x09\x42\x50\x46\x5f\x53\x45\x51\
250\x5f\x50\x52\x49\x4e\x54\x46\x28\x73\x65\x71\x2c\x20\x22\x20\x20\x69\x64\x20\
251\x6e\x61\x6d\x65\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x20\x61\x74\
252\x74\x61\x63\x68\x65\x64\x5c\x6e\x22\x29\x3b\0\x62\x70\x66\x5f\x70\x72\x6f\x67\
253\x5f\x61\x75\x78\0\x61\x74\x74\x61\x63\x68\x5f\x66\x75\x6e\x63\x5f\x6e\x61\x6d\
254\x65\0\x64\x73\x74\x5f\x70\x72\x6f\x67\0\x66\x75\x6e\x63\x5f\x69\x6e\x66\x6f\0\
255\x62\x74\x66\0\x09\x42\x50\x46\x5f\x53\x45\x51\x5f\x50\x52\x49\x4e\x54\x46\x28\
256\x73\x65\x71\x2c\x20\x22\x25\x34\x75\x20\x25\x2d\x31\x36\x73\x20\x25\x73\x20\
257\x25\x73\x5c\x6e\x22\x2c\x20\x61\x75\x78\x2d\x3e\x69\x64\x2c\0\x30\x3a\x34\0\
258\x30\x3a\x35\0\x09\x69\x66\x20\x28\x21\x62\x74\x66\x29\0\x62\x70\x66\x5f\x66\
259\x75\x6e\x63\x5f\x69\x6e\x66\x6f\0\x69\x6e\x73\x6e\x5f\x6f\x66\x66\0\x74\x79\
260\x70\x65\x5f\x69\x64\0\x30\0\x73\x74\x72\x69\x6e\x67\x73\0\x74\x79\x70\x65\x73\
261\0\x68\x64\x72\0\x62\x74\x66\x5f\x68\x65\x61\x64\x65\x72\0\x73\x74\x72\x5f\x6c\
262\x65\x6e\0\x09\x74\x79\x70\x65\x73\x20\x3d\x20\x62\x74\x66\x2d\x3e\x74\x79\x70\
263\x65\x73\x3b\0\x09\x62\x70\x66\x5f\x70\x72\x6f\x62\x65\x5f\x72\x65\x61\x64\x5f\
264\x6b\x65\x72\x6e\x65\x6c\x28\x26\x74\x2c\x20\x73\x69\x7a\x65\x6f\x66\x28\x74\
265\x29\x2c\x20\x74\x79\x70\x65\x73\x20\x2b\x20\x62\x74\x66\x5f\x69\x64\x29\x3b\0\
266\x09\x73\x74\x72\x20\x3d\x20\x62\x74\x66\x2d\x3e\x73\x74\x72\x69\x6e\x67\x73\
267\x3b\0\x62\x74\x66\x5f\x74\x79\x70\x65\0\x6e\x61\x6d\x65\x5f\x6f\x66\x66\0\x09\
268\x6e\x61\x6d\x65\x5f\x6f\x66\x66\x20\x3d\x20\x42\x50\x46\x5f\x43\x4f\x52\x45\
269\x5f\x52\x45\x41\x44\x28\x74\x2c\x20\x6e\x61\x6d\x65\x5f\x6f\x66\x66\x29\x3b\0\
270\x30\x3a\x32\x3a\x30\0\x09\x69\x66\x20\x28\x6e\x61\x6d\x65\x5f\x6f\x66\x66\x20\
271\x3e\x3d\x20\x62\x74\x66\x2d\x3e\x68\x64\x72\x2e\x73\x74\x72\x5f\x6c\x65\x6e\
272\x29\0\x09\x72\x65\x74\x75\x72\x6e\x20\x73\x74\x72\x20\x2b\x20\x6e\x61\x6d\x65\
273\x5f\x6f\x66\x66\x3b\0\x30\x3a\x33\0\x64\x75\x6d\x70\x5f\x62\x70\x66\x5f\x6d\
274\x61\x70\x2e\x5f\x5f\x5f\x66\x6d\x74\0\x64\x75\x6d\x70\x5f\x62\x70\x66\x5f\x6d\
275\x61\x70\x2e\x5f\x5f\x5f\x66\x6d\x74\x2e\x31\0\x64\x75\x6d\x70\x5f\x62\x70\x66\
276\x5f\x70\x72\x6f\x67\x2e\x5f\x5f\x5f\x66\x6d\x74\0\x64\x75\x6d\x70\x5f\x62\x70\
277\x66\x5f\x70\x72\x6f\x67\x2e\x5f\x5f\x5f\x66\x6d\x74\x2e\x32\0\x4c\x49\x43\x45\
278\x4e\x53\x45\0\x2e\x72\x6f\x64\x61\x74\x61\0\x6c\x69\x63\x65\x6e\x73\x65\0\x9f\
279\xeb\x01\0\x20\0\0\0\0\0\0\0\x24\0\0\0\x24\0\0\0\x44\x02\0\0\x68\x02\0\0\xa4\
280\x01\0\0\x08\0\0\0\x31\0\0\0\x01\0\0\0\0\0\0\0\x07\0\0\0\x62\x02\0\0\x01\0\0\0\
281\0\0\0\0\x17\0\0\0\x10\0\0\0\x31\0\0\0\x09\0\0\0\0\0\0\0\x42\0\0\0\x87\0\0\0\
282\x1e\x40\x01\0\x08\0\0\0\x42\0\0\0\x87\0\0\0\x24\x40\x01\0\x10\0\0\0\x42\0\0\0\
283\xfe\0\0\0\x1d\x48\x01\0\x18\0\0\0\x42\0\0\0\x1f\x01\0\0\x06\x50\x01\0\x20\0\0\
284\0\x42\0\0\0\x2e\x01\0\0\x1d\x44\x01\0\x28\0\0\0\x42\0\0\0\x53\x01\0\0\x06\x5c\
285\x01\0\x38\0\0\0\x42\0\0\0\x66\x01\0\0\x03\x60\x01\0\x70\0\0\0\x42\0\0\0\xec\
286\x01\0\0\x02\x68\x01\0\xf0\0\0\0\x42\0\0\0\x3a\x02\0\0\x01\x70\x01\0\x62\x02\0\
287\0\x1a\0\0\0\0\0\0\0\x42\0\0\0\x87\0\0\0\x1e\x84\x01\0\x08\0\0\0\x42\0\0\0\x87\
288\0\0\0\x24\x84\x01\0\x10\0\0\0\x42\0\0\0\x70\x02\0\0\x1f\x8c\x01\0\x18\0\0\0\
289\x42\0\0\0\x94\x02\0\0\x06\x98\x01\0\x20\0\0\0\x42\0\0\0\xad\x02\0\0\x0e\xa4\
290\x01\0\x28\0\0\0\x42\0\0\0\x2e\x01\0\0\x1d\x88\x01\0\x30\0\0\0\x42\0\0\0\x53\
291\x01\0\0\x06\xa8\x01\0\x40\0\0\0\x42\0\0\0\xbf\x02\0\0\x03\xac\x01\0\x80\0\0\0\
292\x42\0\0\0\x2f\x03\0\0\x02\xb4\x01\0\xb8\0\0\0\x42\0\0\0\x6a\x03\0\0\x06\x08\
293\x01\0\xd0\0\0\0\x42\0\0\0\0\0\0\0\0\0\0\0\xd8\0\0\0\x42\0\0\0\xbb\x03\0\0\x0f\
294\x14\x01\0\xe0\0\0\0\x42\0\0\0\xd0\x03\0\0\x2d\x18\x01\0\xf0\0\0\0\x42\0\0\0\
295\x07\x04\0\0\x0d\x10\x01\0\0\x01\0\0\x42\0\0\0\0\0\0\0\0\0\0\0\x08\x01\0\0\x42\
296\0\0\0\xd0\x03\0\0\x02\x18\x01\0\x20\x01\0\0\x42\0\0\0\x2e\x04\0\0\x0d\x1c\x01\
297\0\x38\x01\0\0\x42\0\0\0\0\0\0\0\0\0\0\0\x40\x01\0\0\x42\0\0\0\x2e\x04\0\0\x0d\
298\x1c\x01\0\x58\x01\0\0\x42\0\0\0\x2e\x04\0\0\x0d\x1c\x01\0\x60\x01\0\0\x42\0\0\
299\0\x5c\x04\0\0\x1b\x20\x01\0\x68\x01\0\0\x42\0\0\0\x5c\x04\0\0\x06\x20\x01\0\
300\x70\x01\0\0\x42\0\0\0\x7f\x04\0\0\x0d\x28\x01\0\x78\x01\0\0\x42\0\0\0\0\0\0\0\
301\0\0\0\0\x80\x01\0\0\x42\0\0\0\x2f\x03\0\0\x02\xb4\x01\0\xf8\x01\0\0\x42\0\0\0\
302\x3a\x02\0\0\x01\xc4\x01\0\x10\0\0\0\x31\0\0\0\x07\0\0\0\0\0\0\0\x02\0\0\0\x3e\
303\0\0\0\0\0\0\0\x08\0\0\0\x08\0\0\0\x3e\0\0\0\0\0\0\0\x10\0\0\0\x02\0\0\0\xfa\0\
304\0\0\0\0\0\0\x20\0\0\0\x08\0\0\0\x2a\x01\0\0\0\0\0\0\x70\0\0\0\x0d\0\0\0\x3e\0\
305\0\0\0\0\0\0\x80\0\0\0\x0d\0\0\0\xfa\0\0\0\0\0\0\0\xa0\0\0\0\x0d\0\0\0\x2a\x01\
306\0\0\0\0\0\0\x62\x02\0\0\x12\0\0\0\0\0\0\0\x14\0\0\0\x3e\0\0\0\0\0\0\0\x08\0\0\
307\0\x08\0\0\0\x3e\0\0\0\0\0\0\0\x10\0\0\0\x14\0\0\0\xfa\0\0\0\0\0\0\0\x20\0\0\0\
308\x18\0\0\0\x3e\0\0\0\0\0\0\0\x28\0\0\0\x08\0\0\0\x2a\x01\0\0\0\0\0\0\x80\0\0\0\
309\x1a\0\0\0\x3e\0\0\0\0\0\0\0\x90\0\0\0\x1a\0\0\0\xfa\0\0\0\0\0\0\0\xa8\0\0\0\
310\x1a\0\0\0\x62\x03\0\0\0\0\0\0\xb0\0\0\0\x1a\0\0\0\x66\x03\0\0\0\0\0\0\xc0\0\0\
311\0\x1f\0\0\0\x94\x03\0\0\0\0\0\0\xd8\0\0\0\x20\0\0\0\xfa\0\0\0\0\0\0\0\xf0\0\0\
312\0\x20\0\0\0\x3e\0\0\0\0\0\0\0\x18\x01\0\0\x24\0\0\0\x3e\0\0\0\0\0\0\0\x50\x01\
313\0\0\x1a\0\0\0\xfa\0\0\0\0\0\0\0\x60\x01\0\0\x20\0\0\0\x56\x04\0\0\0\0\0\0\x88\
314\x01\0\0\x1a\0\0\0\x2a\x01\0\0\0\0\0\0\x98\x01\0\0\x1a\0\0\0\x97\x04\0\0\0\0\0\
315\0\xa0\x01\0\0\x18\0\0\0\x3e\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\
316\0\0\0\0\0\0\0\x91\0\0\0\x04\0\xf1\xff\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\xe6\0\0\
317\0\0\0\x02\0\x70\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\xd8\0\0\0\0\0\x02\0\xf0\0\0\0\0\
318\0\0\0\0\0\0\0\0\0\0\0\xdf\0\0\0\0\0\x03\0\x78\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\
319\xd1\0\0\0\0\0\x03\0\x80\x01\0\0\0\0\0\0\0\0\0\0\0\0\0\0\xca\0\0\0\0\0\x03\0\
320\xf8\x01\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x14\0\0\0\x01\0\x04\0\0\0\0\0\0\0\0\0\x23\
321\0\0\0\0\0\0\0\x04\x01\0\0\x01\0\x04\0\x23\0\0\0\0\0\0\0\x0e\0\0\0\0\0\0\0\x28\
322\0\0\0\x01\0\x04\0\x31\0\0\0\0\0\0\0\x20\0\0\0\0\0\0\0\xed\0\0\0\x01\0\x04\0\
323\x51\0\0\0\0\0\0\0\x11\0\0\0\0\0\0\0\0\0\0\0\x03\0\x02\0\0\0\0\0\0\0\0\0\0\0\0\
324\0\0\0\0\0\0\0\0\0\x03\0\x03\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x03\0\
325\x04\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\xc2\0\0\0\x11\0\x05\0\0\0\0\0\0\0\0\0\
326\x04\0\0\0\0\0\0\0\x3d\0\0\0\x12\0\x02\0\0\0\0\0\0\0\0\0\0\x01\0\0\0\0\0\0\x5b\
327\0\0\0\x12\0\x03\0\0\0\0\0\0\0\0\0\x08\x02\0\0\0\0\0\0\x48\0\0\0\0\0\0\0\x01\0\
328\0\0\x0d\0\0\0\xc8\0\0\0\0\0\0\0\x01\0\0\0\x0d\0\0\0\x50\0\0\0\0\0\0\0\x01\0\0\
329\0\x0d\0\0\0\xd0\x01\0\0\0\0\0\0\x01\0\0\0\x0d\0\0\0\xf0\x03\0\0\0\0\0\0\x0a\0\
330\0\0\x0d\0\0\0\xfc\x03\0\0\0\0\0\0\x0a\0\0\0\x0d\0\0\0\x08\x04\0\0\0\0\0\0\x0a\
331\0\0\0\x0d\0\0\0\x14\x04\0\0\0\0\0\0\x0a\0\0\0\x0d\0\0\0\x2c\x04\0\0\0\0\0\0\0\
332\0\0\0\x0e\0\0\0\x2c\0\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\x3c\0\0\0\0\0\0\0\0\0\0\0\
333\x0c\0\0\0\x50\0\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\x60\0\0\0\0\0\0\0\0\0\0\0\x0b\0\
334\0\0\x70\0\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\x80\0\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\
335\x90\0\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\xa0\0\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\xb0\0\
336\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\xc0\0\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\xd0\0\0\0\0\
337\0\0\0\0\0\0\0\x0b\0\0\0\xe8\0\0\0\0\0\0\0\0\0\0\0\x0c\0\0\0\xf8\0\0\0\0\0\0\0\
338\0\0\0\0\x0c\0\0\0\x08\x01\0\0\0\0\0\0\0\0\0\0\x0c\0\0\0\x18\x01\0\0\0\0\0\0\0\
339\0\0\0\x0c\0\0\0\x28\x01\0\0\0\0\0\0\0\0\0\0\x0c\0\0\0\x38\x01\0\0\0\0\0\0\0\0\
340\0\0\x0c\0\0\0\x48\x01\0\0\0\0\0\0\0\0\0\0\x0c\0\0\0\x58\x01\0\0\0\0\0\0\0\0\0\
341\0\x0c\0\0\0\x68\x01\0\0\0\0\0\0\0\0\0\0\x0c\0\0\0\x78\x01\0\0\0\0\0\0\0\0\0\0\
342\x0c\0\0\0\x88\x01\0\0\0\0\0\0\0\0\0\0\x0c\0\0\0\x98\x01\0\0\0\0\0\0\0\0\0\0\
343\x0c\0\0\0\xa8\x01\0\0\0\0\0\0\0\0\0\0\x0c\0\0\0\xb8\x01\0\0\0\0\0\0\0\0\0\0\
344\x0c\0\0\0\xc8\x01\0\0\0\0\0\0\0\0\0\0\x0c\0\0\0\xd8\x01\0\0\0\0\0\0\0\0\0\0\
345\x0c\0\0\0\xe8\x01\0\0\0\0\0\0\0\0\0\0\x0c\0\0\0\xf8\x01\0\0\0\0\0\0\0\0\0\0\
346\x0c\0\0\0\x08\x02\0\0\0\0\0\0\0\0\0\0\x0c\0\0\0\x18\x02\0\0\0\0\0\0\0\0\0\0\
347\x0c\0\0\0\x28\x02\0\0\0\0\0\0\0\0\0\0\x0c\0\0\0\x38\x02\0\0\0\0\0\0\0\0\0\0\
348\x0c\0\0\0\x48\x02\0\0\0\0\0\0\0\0\0\0\x0c\0\0\0\x58\x02\0\0\0\0\0\0\0\0\0\0\
349\x0c\0\0\0\x68\x02\0\0\0\0\0\0\0\0\0\0\x0c\0\0\0\x78\x02\0\0\0\0\0\0\0\0\0\0\
350\x0c\0\0\0\x94\x02\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\xa4\x02\0\0\0\0\0\0\0\0\0\0\
351\x0b\0\0\0\xb4\x02\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\xc4\x02\0\0\0\0\0\0\0\0\0\0\
352\x0b\0\0\0\xd4\x02\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\xe4\x02\0\0\0\0\0\0\0\0\0\0\
353\x0b\0\0\0\xf4\x02\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\x0c\x03\0\0\0\0\0\0\0\0\0\0\
354\x0c\0\0\0\x1c\x03\0\0\0\0\0\0\0\0\0\0\x0c\0\0\0\x2c\x03\0\0\0\0\0\0\0\0\0\0\
355\x0c\0\0\0\x3c\x03\0\0\0\0\0\0\0\0\0\0\x0c\0\0\0\x4c\x03\0\0\0\0\0\0\0\0\0\0\
356\x0c\0\0\0\x5c\x03\0\0\0\0\0\0\0\0\0\0\x0c\0\0\0\x6c\x03\0\0\0\0\0\0\0\0\0\0\
357\x0c\0\0\0\x7c\x03\0\0\0\0\0\0\0\0\0\0\x0c\0\0\0\x8c\x03\0\0\0\0\0\0\0\0\0\0\
358\x0c\0\0\0\x9c\x03\0\0\0\0\0\0\0\0\0\0\x0c\0\0\0\xac\x03\0\0\0\0\0\0\0\0\0\0\
359\x0c\0\0\0\xbc\x03\0\0\0\0\0\0\0\0\0\0\x0c\0\0\0\xcc\x03\0\0\0\0\0\0\0\0\0\0\
360\x0c\0\0\0\xdc\x03\0\0\0\0\0\0\0\0\0\0\x0c\0\0\0\xec\x03\0\0\0\0\0\0\0\0\0\0\
361\x0c\0\0\0\xfc\x03\0\0\0\0\0\0\0\0\0\0\x0c\0\0\0\x0c\x04\0\0\0\0\0\0\0\0\0\0\
362\x0c\0\0\0\x1c\x04\0\0\0\0\0\0\0\0\0\0\x0c\0\0\0\x4d\x4e\x40\x41\x42\x43\x4c\0\
363\x2e\x74\x65\x78\x74\0\x2e\x72\x65\x6c\x2e\x42\x54\x46\x2e\x65\x78\x74\0\x64\
364\x75\x6d\x70\x5f\x62\x70\x66\x5f\x6d\x61\x70\x2e\x5f\x5f\x5f\x66\x6d\x74\0\x64\
365\x75\x6d\x70\x5f\x62\x70\x66\x5f\x70\x72\x6f\x67\x2e\x5f\x5f\x5f\x66\x6d\x74\0\
366\x64\x75\x6d\x70\x5f\x62\x70\x66\x5f\x6d\x61\x70\0\x2e\x72\x65\x6c\x69\x74\x65\
367\x72\x2f\x62\x70\x66\x5f\x6d\x61\x70\0\x64\x75\x6d\x70\x5f\x62\x70\x66\x5f\x70\
368\x72\x6f\x67\0\x2e\x72\x65\x6c\x69\x74\x65\x72\x2f\x62\x70\x66\x5f\x70\x72\x6f\
369\x67\0\x2e\x6c\x6c\x76\x6d\x5f\x61\x64\x64\x72\x73\x69\x67\0\x6c\x69\x63\x65\
370\x6e\x73\x65\0\x69\x74\x65\x72\x61\x74\x6f\x72\x73\x2e\x62\x70\x66\x2e\x63\0\
371\x2e\x73\x74\x72\x74\x61\x62\0\x2e\x73\x79\x6d\x74\x61\x62\0\x2e\x72\x6f\x64\
372\x61\x74\x61\0\x2e\x72\x65\x6c\x2e\x42\x54\x46\0\x4c\x49\x43\x45\x4e\x53\x45\0\
373\x4c\x42\x42\x31\x5f\x37\0\x4c\x42\x42\x31\x5f\x36\0\x4c\x42\x42\x30\x5f\x34\0\
374\x4c\x42\x42\x31\x5f\x33\0\x4c\x42\x42\x30\x5f\x33\0\x64\x75\x6d\x70\x5f\x62\
375\x70\x66\x5f\x70\x72\x6f\x67\x2e\x5f\x5f\x5f\x66\x6d\x74\x2e\x32\0\x64\x75\x6d\
376\x70\x5f\x62\x70\x66\x5f\x6d\x61\x70\x2e\x5f\x5f\x5f\x66\x6d\x74\x2e\x31\0\0\0\
377\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\
378\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x01\0\0\0\x01\0\0\
379\0\x06\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x40\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\
380\0\0\0\0\x04\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x4e\0\0\0\x01\0\0\0\x06\0\0\0\0\0\0\
381\0\0\0\0\0\0\0\0\0\x40\0\0\0\0\0\0\0\0\x01\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x08\0\0\
382\0\0\0\0\0\0\0\0\0\0\0\0\0\x6d\0\0\0\x01\0\0\0\x06\0\0\0\0\0\0\0\0\0\0\0\0\0\0\
383\0\x40\x01\0\0\0\0\0\0\x08\x02\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x08\0\0\0\0\0\0\0\0\
384\0\0\0\0\0\0\0\xb1\0\0\0\x01\0\0\0\x02\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x48\x03\0\
385\0\0\0\0\0\x62\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x01\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\
386\x89\0\0\0\x01\0\0\0\x03\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\xaa\x03\0\0\0\0\0\0\x04\
387\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x01\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\xbd\0\0\0\x01\
388\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\xae\x03\0\0\0\0\0\0\x3d\x09\0\0\0\0\0\0\
389\0\0\0\0\0\0\0\0\x01\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x0b\0\0\0\x01\0\0\0\0\0\0\0\
390\0\0\0\0\0\0\0\0\0\0\0\0\xeb\x0c\0\0\0\0\0\0\x2c\x04\0\0\0\0\0\0\0\0\0\0\0\0\0\
391\0\x01\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\xa9\0\0\0\x02\0\0\0\0\0\0\0\0\0\0\0\0\0\0\
392\0\0\0\0\0\x18\x11\0\0\0\0\0\0\x98\x01\0\0\0\0\0\0\x0e\0\0\0\x0e\0\0\0\x08\0\0\
393\0\0\0\0\0\x18\0\0\0\0\0\0\0\x4a\0\0\0\x09\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\
394\0\xb0\x12\0\0\0\0\0\0\x20\0\0\0\0\0\0\0\x08\0\0\0\x02\0\0\0\x08\0\0\0\0\0\0\0\
395\x10\0\0\0\0\0\0\0\x69\0\0\0\x09\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\xd0\x12\
396\0\0\0\0\0\0\x20\0\0\0\0\0\0\0\x08\0\0\0\x03\0\0\0\x08\0\0\0\0\0\0\0\x10\0\0\0\
397\0\0\0\0\xb9\0\0\0\x09\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\xf0\x12\0\0\0\0\0\
398\0\x50\0\0\0\0\0\0\0\x08\0\0\0\x06\0\0\0\x08\0\0\0\0\0\0\0\x10\0\0\0\0\0\0\0\
399\x07\0\0\0\x09\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x40\x13\0\0\0\0\0\0\xe0\
400\x03\0\0\0\0\0\0\x08\0\0\0\x07\0\0\0\x08\0\0\0\0\0\0\0\x10\0\0\0\0\0\0\0\x7b\0\
401\0\0\x03\x4c\xff\x6f\0\0\0\x80\0\0\0\0\0\0\0\0\0\0\0\0\x20\x17\0\0\0\0\0\0\x07\
402\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x01\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\xa1\0\0\0\x03\
403\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\x27\x17\0\0\0\0\0\0\x1a\x01\0\0\0\0\0\0\
404\0\0\0\0\0\0\0\0\x01\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0";
405
406 return 0;
407err:
408 bpf_object__destroy_skeleton(s);
409 return -1;
410}
411
412#endif /* __ITERATORS_BPF_SKEL_H__ */
kernel/bpf/queue_stack_maps.c+2−0
@@ -257,6 +257,7 @@ static int queue_stack_map_get_next_key(struct bpf_map *map, void *key, · 256 unmodified lines
257 257
258static int queue_map_btf_id;258static int queue_map_btf_id;
259const struct bpf_map_ops queue_map_ops = {259const struct bpf_map_ops queue_map_ops = {
260 .map_meta_equal = bpf_map_meta_equal,
260 .map_alloc_check = queue_stack_map_alloc_check,261 .map_alloc_check = queue_stack_map_alloc_check,
261 .map_alloc = queue_stack_map_alloc,262 .map_alloc = queue_stack_map_alloc,
262 .map_free = queue_stack_map_free,263 .map_free = queue_stack_map_free,
@@ -273,6 +274,7 @@ const struct bpf_map_ops queue_map_ops = { · 10 unmodified lines
273 274
274static int stack_map_btf_id;275static int stack_map_btf_id;
275const struct bpf_map_ops stack_map_ops = {276const struct bpf_map_ops stack_map_ops = {
277 .map_meta_equal = bpf_map_meta_equal,
276 .map_alloc_check = queue_stack_map_alloc_check,278 .map_alloc_check = queue_stack_map_alloc_check,
277 .map_alloc = queue_stack_map_alloc,279 .map_alloc = queue_stack_map_alloc,
278 .map_free = queue_stack_map_free,280 .map_free = queue_stack_map_free,
kernel/bpf/reuseport_array.c+2−1
@@ -191,7 +191,7 @@ int bpf_fd_reuseport_array_lookup_elem(struct bpf_map *map, void *key, · 190 unmodified lines
191 rcu_read_lock();191 rcu_read_lock();
192 sk = reuseport_array_lookup_elem(map, key);192 sk = reuseport_array_lookup_elem(map, key);
193 if (sk) {193 if (sk) {
194 *(u64 *)value = sock_gen_cookie(sk);194 *(u64 *)value = __sock_gen_cookie(sk);
195 err = 0;195 err = 0;
196 } else {196 } else {
197 err = -ENOENT;197 err = -ENOENT;
@@ -351,6 +351,7 @@ static int reuseport_array_get_next_key(struct bpf_map *map, void *key, · 153 unmodified lines
351 351
352static int reuseport_array_map_btf_id;352static int reuseport_array_map_btf_id;
353const struct bpf_map_ops reuseport_array_ops = {353const struct bpf_map_ops reuseport_array_ops = {
354 .map_meta_equal = bpf_map_meta_equal,
354 .map_alloc_check = reuseport_array_alloc_check,355 .map_alloc_check = reuseport_array_alloc_check,
355 .map_alloc = reuseport_array_alloc,356 .map_alloc = reuseport_array_alloc,
356 .map_free = reuseport_array_free,357 .map_free = reuseport_array_free,
kernel/bpf/ringbuf.c+1−0
@@ -287,6 +287,7 @@ static __poll_t ringbuf_map_poll(struct bpf_map *map, struct file *filp, · 286 unmodified lines
287 287
288static int ringbuf_map_btf_id;288static int ringbuf_map_btf_id;
289const struct bpf_map_ops ringbuf_map_ops = {289const struct bpf_map_ops ringbuf_map_ops = {
290 .map_meta_equal = bpf_map_meta_equal,
290 .map_alloc = ringbuf_map_alloc,291 .map_alloc = ringbuf_map_alloc,
291 .map_free = ringbuf_map_free,292 .map_free = ringbuf_map_free,
292 .map_mmap = ringbuf_map_mmap,293 .map_mmap = ringbuf_map_mmap,
kernel/bpf/stackmap.c+3−3
@@ -665,18 +665,17 @@ BPF_CALL_4(bpf_get_task_stack, struct task_struct *, task, void *, buf, · 664 unmodified lines
665 return __bpf_get_stack(regs, task, NULL, buf, size, flags);665 return __bpf_get_stack(regs, task, NULL, buf, size, flags);
666}666}
667 667
668BTF_ID_LIST(bpf_get_task_stack_btf_ids)668BTF_ID_LIST_SINGLE(bpf_get_task_stack_btf_ids, struct, task_struct)
669BTF_ID(struct, task_struct)
670 669
671const struct bpf_func_proto bpf_get_task_stack_proto = {670const struct bpf_func_proto bpf_get_task_stack_proto = {
672 .func = bpf_get_task_stack,671 .func = bpf_get_task_stack,
673 .gpl_only = false,672 .gpl_only = false,
674 .ret_type = RET_INTEGER,673 .ret_type = RET_INTEGER,
675 .arg1_type = ARG_PTR_TO_BTF_ID,674 .arg1_type = ARG_PTR_TO_BTF_ID,
675 .arg1_btf_id = &bpf_get_task_stack_btf_ids[0],
676 .arg2_type = ARG_PTR_TO_UNINIT_MEM,676 .arg2_type = ARG_PTR_TO_UNINIT_MEM,
677 .arg3_type = ARG_CONST_SIZE_OR_ZERO,677 .arg3_type = ARG_CONST_SIZE_OR_ZERO,
678 .arg4_type = ARG_ANYTHING,678 .arg4_type = ARG_ANYTHING,
679 .btf_id = bpf_get_task_stack_btf_ids,
680};679};
681 680
682BPF_CALL_4(bpf_get_stack_pe, struct bpf_perf_event_data_kern *, ctx,681BPF_CALL_4(bpf_get_stack_pe, struct bpf_perf_event_data_kern *, ctx,
@@ -839,6 +838,7 @@ static void stack_map_free(struct bpf_map *map) · 156 unmodified lines
839 838
840static int stack_trace_map_btf_id;839static int stack_trace_map_btf_id;
841const struct bpf_map_ops stack_trace_map_ops = {840const struct bpf_map_ops stack_trace_map_ops = {
841 .map_meta_equal = bpf_map_meta_equal,
842 .map_alloc = stack_map_alloc,842 .map_alloc = stack_map_alloc,
843 .map_free = stack_map_free,843 .map_free = stack_map_free,
844 .map_get_next_key = stack_map_get_next_key,844 .map_get_next_key = stack_map_get_next_key,
kernel/bpf/syscall.c+278−53
@@ -4,6 +4,7 @@ · 3 unmodified lines
4#include <linux/bpf.h>4#include <linux/bpf.h>
5#include <linux/bpf_trace.h>5#include <linux/bpf_trace.h>
6#include <linux/bpf_lirc.h>6#include <linux/bpf_lirc.h>
7#include <linux/bpf_verifier.h>
7#include <linux/btf.h>8#include <linux/btf.h>
8#include <linux/syscalls.h>9#include <linux/syscalls.h>
9#include <linux/slab.h>10#include <linux/slab.h>
@@ -29,6 +30,7 @@ · 19 unmodified lines
29#include <linux/bpf_lsm.h>30#include <linux/bpf_lsm.h>
30#include <linux/poll.h>31#include <linux/poll.h>
31#include <linux/bpf-netns.h>32#include <linux/bpf-netns.h>
33#include <linux/rcupdate_trace.h>
32 34
33#define IS_FD_ARRAY(map) ((map)->map_type == BPF_MAP_TYPE_PERF_EVENT_ARRAY || \35#define IS_FD_ARRAY(map) ((map)->map_type == BPF_MAP_TYPE_PERF_EVENT_ARRAY || \
34 (map)->map_type == BPF_MAP_TYPE_CGROUP_ARRAY || \36 (map)->map_type == BPF_MAP_TYPE_CGROUP_ARRAY || \
@@ -90,6 +92,7 @@ int bpf_check_uarg_tail_zero(void __user *uaddr, · 55 unmodified lines
90}92}
91 93
92const struct bpf_map_ops bpf_map_offload_ops = {94const struct bpf_map_ops bpf_map_offload_ops = {
95 .map_meta_equal = bpf_map_meta_equal,
93 .map_alloc = bpf_map_offload_map_alloc,96 .map_alloc = bpf_map_offload_map_alloc,
94 .map_free = bpf_map_offload_map_free,97 .map_free = bpf_map_offload_map_free,
95 .map_check_btf = map_check_no_btf,98 .map_check_btf = map_check_no_btf,
@@ -157,10 +160,11 @@ static int bpf_map_update_value(struct bpf_map *map, struct fd f, void *key, · 61 unmodified lines
157 if (bpf_map_is_dev_bound(map)) {160 if (bpf_map_is_dev_bound(map)) {
158 return bpf_map_offload_update_elem(map, key, value, flags);161 return bpf_map_offload_update_elem(map, key, value, flags);
159 } else if (map->map_type == BPF_MAP_TYPE_CPUMAP ||162 } else if (map->map_type == BPF_MAP_TYPE_CPUMAP ||
160 map->map_type == BPF_MAP_TYPE_SOCKHASH ||
161 map->map_type == BPF_MAP_TYPE_SOCKMAP ||
162 map->map_type == BPF_MAP_TYPE_STRUCT_OPS) {163 map->map_type == BPF_MAP_TYPE_STRUCT_OPS) {
163 return map->ops->map_update_elem(map, key, value, flags);164 return map->ops->map_update_elem(map, key, value, flags);
165 } else if (map->map_type == BPF_MAP_TYPE_SOCKHASH ||
166 map->map_type == BPF_MAP_TYPE_SOCKMAP) {
167 return sock_map_update_elem_sys(map, key, value, flags);
164 } else if (IS_FD_PROG_ARRAY(map)) {168 } else if (IS_FD_PROG_ARRAY(map)) {
165 return bpf_fd_array_map_update_elem(map, f.file, key, value,169 return bpf_fd_array_map_update_elem(map, f.file, key, value,
166 flags);170 flags);
@@ -768,7 +772,8 @@ static int map_check_btf(struct bpf_map *map, const struct btf *btf, · 601 unmodified lines
768 if (map->map_type != BPF_MAP_TYPE_HASH &&772 if (map->map_type != BPF_MAP_TYPE_HASH &&
769 map->map_type != BPF_MAP_TYPE_ARRAY &&773 map->map_type != BPF_MAP_TYPE_ARRAY &&
770 map->map_type != BPF_MAP_TYPE_CGROUP_STORAGE &&774 map->map_type != BPF_MAP_TYPE_CGROUP_STORAGE &&
771 map->map_type != BPF_MAP_TYPE_SK_STORAGE)775 map->map_type != BPF_MAP_TYPE_SK_STORAGE &&
776 map->map_type != BPF_MAP_TYPE_INODE_STORAGE)
772 return -ENOTSUPP;777 return -ENOTSUPP;
773 if (map->spin_lock_off + sizeof(struct bpf_spin_lock) >778 if (map->spin_lock_off + sizeof(struct bpf_spin_lock) >
774 map->value_size) {779 map->value_size) {
@@ -1728,10 +1733,14 @@ static void __bpf_prog_put_noref(struct bpf_prog *prog, bool deferred) · 953 unmodified lines
1728 btf_put(prog->aux->btf);1733 btf_put(prog->aux->btf);
1729 bpf_prog_free_linfo(prog);1734 bpf_prog_free_linfo(prog);
1730 1735
1731 if (deferred)1736 if (deferred) {
1732 call_rcu(&prog->aux->rcu, __bpf_prog_put_rcu);1737 if (prog->aux->sleepable)
1733 else1738 call_rcu_tasks_trace(&prog->aux->rcu, __bpf_prog_put_rcu);
1739 else
1740 call_rcu(&prog->aux->rcu, __bpf_prog_put_rcu);
1741 } else {
1734 __bpf_prog_put_rcu(&prog->aux->rcu);1742 __bpf_prog_put_rcu(&prog->aux->rcu);
1743 }
1735}1744}
1736 1745
1737static void __bpf_prog_put(struct bpf_prog *prog, bool do_idr_lock)1746static void __bpf_prog_put(struct bpf_prog *prog, bool do_idr_lock)
@@ -2101,6 +2110,7 @@ static int bpf_prog_load(union bpf_attr *attr, union bpf_attr __user *uattr) · 363 unmodified lines
2101 if (attr->prog_flags & ~(BPF_F_STRICT_ALIGNMENT |2110 if (attr->prog_flags & ~(BPF_F_STRICT_ALIGNMENT |
2102 BPF_F_ANY_ALIGNMENT |2111 BPF_F_ANY_ALIGNMENT |
2103 BPF_F_TEST_STATE_FREQ |2112 BPF_F_TEST_STATE_FREQ |
2113 BPF_F_SLEEPABLE |
2104 BPF_F_TEST_RND_HI32))2114 BPF_F_TEST_RND_HI32))
2105 return -EINVAL;2115 return -EINVAL;
2106 2116
@@ -2145,17 +2155,18 @@ static int bpf_prog_load(union bpf_attr *attr, union bpf_attr __user *uattr) · 38 unmodified lines
2145 prog->expected_attach_type = attr->expected_attach_type;2155 prog->expected_attach_type = attr->expected_attach_type;
2146 prog->aux->attach_btf_id = attr->attach_btf_id;2156 prog->aux->attach_btf_id = attr->attach_btf_id;
2147 if (attr->attach_prog_fd) {2157 if (attr->attach_prog_fd) {
2148 struct bpf_prog *tgt_prog;2158 struct bpf_prog *dst_prog;
2149 2159
2150 tgt_prog = bpf_prog_get(attr->attach_prog_fd);2160 dst_prog = bpf_prog_get(attr->attach_prog_fd);
2151 if (IS_ERR(tgt_prog)) {2161 if (IS_ERR(dst_prog)) {
2152 err = PTR_ERR(tgt_prog);2162 err = PTR_ERR(dst_prog);
2153 goto free_prog_nouncharge;2163 goto free_prog_nouncharge;
2154 }2164 }
2155 prog->aux->linked_prog = tgt_prog;2165 prog->aux->dst_prog = dst_prog;
2156 }2166 }
2157 2167
2158 prog->aux->offload_requested = !!attr->prog_ifindex;2168 prog->aux->offload_requested = !!attr->prog_ifindex;
2169 prog->aux->sleepable = attr->prog_flags & BPF_F_SLEEPABLE;
2159 2170
2160 err = security_bpf_prog_alloc(prog->aux);2171 err = security_bpf_prog_alloc(prog->aux);
2161 if (err)2172 if (err)
@@ -2488,11 +2499,23 @@ struct bpf_link *bpf_link_get_from_fd(u32 ufd) · 326 unmodified lines
2488struct bpf_tracing_link {2499struct bpf_tracing_link {
2489 struct bpf_link link;2500 struct bpf_link link;
2490 enum bpf_attach_type attach_type;2501 enum bpf_attach_type attach_type;
2502 struct bpf_trampoline *trampoline;
2503 struct bpf_prog *tgt_prog;
2491};2504};
2492 2505
2493static void bpf_tracing_link_release(struct bpf_link *link)2506static void bpf_tracing_link_release(struct bpf_link *link)
2494{2507{
2495 WARN_ON_ONCE(bpf_trampoline_unlink_prog(link->prog));2508 struct bpf_tracing_link *tr_link =
2509 container_of(link, struct bpf_tracing_link, link);
2510
2511 WARN_ON_ONCE(bpf_trampoline_unlink_prog(link->prog,
2512 tr_link->trampoline));
2513
2514 bpf_trampoline_put(tr_link->trampoline);
2515
2516 /* tgt_prog is NULL if target is a kernel function */
2517 if (tr_link->tgt_prog)
2518 bpf_prog_put(tr_link->tgt_prog);
2496}2519}
2497 2520
2498static void bpf_tracing_link_dealloc(struct bpf_link *link)2521static void bpf_tracing_link_dealloc(struct bpf_link *link)
@@ -2532,10 +2555,15 @@ static const struct bpf_link_ops bpf_tracing_link_lops = { · 33 unmodified lines
2532 .fill_link_info = bpf_tracing_link_fill_link_info,2555 .fill_link_info = bpf_tracing_link_fill_link_info,
2533};2556};
2534 2557
2535static int bpf_tracing_prog_attach(struct bpf_prog *prog)2558static int bpf_tracing_prog_attach(struct bpf_prog *prog,
2559 int tgt_prog_fd,
2560 u32 btf_id)
2536{2561{
2537 struct bpf_link_primer link_primer;2562 struct bpf_link_primer link_primer;
2563 struct bpf_prog *tgt_prog = NULL;
2564 struct bpf_trampoline *tr = NULL;
2538 struct bpf_tracing_link *link;2565 struct bpf_tracing_link *link;
2566 u64 key = 0;
2539 int err;2567 int err;
2540 2568
2541 switch (prog->type) {2569 switch (prog->type) {
@@ -2564,6 +2592,28 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog) · 22 unmodified lines
2564 goto out_put_prog;2592 goto out_put_prog;
2565 }2593 }
2566 2594
2595 if (!!tgt_prog_fd != !!btf_id) {
2596 err = -EINVAL;
2597 goto out_put_prog;
2598 }
2599
2600 if (tgt_prog_fd) {
2601 /* For now we only allow new targets for BPF_PROG_TYPE_EXT */
2602 if (prog->type != BPF_PROG_TYPE_EXT) {
2603 err = -EINVAL;
2604 goto out_put_prog;
2605 }
2606
2607 tgt_prog = bpf_prog_get(tgt_prog_fd);
2608 if (IS_ERR(tgt_prog)) {
2609 err = PTR_ERR(tgt_prog);
2610 tgt_prog = NULL;
2611 goto out_put_prog;
2612 }
2613
2614 key = bpf_trampoline_compute_key(tgt_prog, btf_id);
2615 }
2616
2567 link = kzalloc(sizeof(*link), GFP_USER);2617 link = kzalloc(sizeof(*link), GFP_USER);
2568 if (!link) {2618 if (!link) {
2569 err = -ENOMEM;2619 err = -ENOMEM;
@@ -2573,20 +2623,100 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog) · 3 unmodified lines
2573 &bpf_tracing_link_lops, prog);2623 &bpf_tracing_link_lops, prog);
2574 link->attach_type = prog->expected_attach_type;2624 link->attach_type = prog->expected_attach_type;
2575 2625
2576 err = bpf_link_prime(&link->link, &link_primer);2626 mutex_lock(&prog->aux->dst_mutex);
2577 if (err) {2627
2578 kfree(link);2628 /* There are a few possible cases here:
2579 goto out_put_prog;2629 *
2630 * - if prog->aux->dst_trampoline is set, the program was just loaded
2631 * and not yet attached to anything, so we can use the values stored
2632 * in prog->aux
2633 *
2634 * - if prog->aux->dst_trampoline is NULL, the program has already been
2635 * attached to a target and its initial target was cleared (below)
2636 *
2637 * - if tgt_prog != NULL, the caller specified tgt_prog_fd +
2638 * target_btf_id using the link_create API.
2639 *
2640 * - if tgt_prog == NULL when this function was called using the old
2641 * raw_tracepoint_open API, and we need a target from prog->aux
2642 *
2643 * The combination of no saved target in prog->aux, and no target
2644 * specified on load is illegal, and we reject that here.
2645 */
2646 if (!prog->aux->dst_trampoline && !tgt_prog) {
2647 err = -ENOENT;
2648 goto out_unlock;
2649 }
2650
2651 if (!prog->aux->dst_trampoline ||
2652 (key && key != prog->aux->dst_trampoline->key)) {
2653 /* If there is no saved target, or the specified target is
2654 * different from the destination specified at load time, we
2655 * need a new trampoline and a check for compatibility
2656 */
2657 struct bpf_attach_target_info tgt_info = {};
2658
2659 err = bpf_check_attach_target(NULL, prog, tgt_prog, btf_id,
2660 &tgt_info);
2661 if (err)
2662 goto out_unlock;
2663
2664 tr = bpf_trampoline_get(key, &tgt_info);
2665 if (!tr) {
2666 err = -ENOMEM;
2667 goto out_unlock;
2668 }
2669 } else {
2670 /* The caller didn't specify a target, or the target was the
2671 * same as the destination supplied during program load. This
2672 * means we can reuse the trampoline and reference from program
2673 * load time, and there is no need to allocate a new one. This
2674 * can only happen once for any program, as the saved values in
2675 * prog->aux are cleared below.
2676 */
2677 tr = prog->aux->dst_trampoline;
2678 tgt_prog = prog->aux->dst_prog;
2580 }2679 }
2581 2680
2582 err = bpf_trampoline_link_prog(prog);2681 err = bpf_link_prime(&link->link, &link_primer);
2682 if (err)
2683 goto out_unlock;
2684
2685 err = bpf_trampoline_link_prog(prog, tr);
2583 if (err) {2686 if (err) {
2584 bpf_link_cleanup(&link_primer);2687 bpf_link_cleanup(&link_primer);
2585 goto out_put_prog;2688 link = NULL;
2689 goto out_unlock;
2586 }2690 }
2587 2691
2692 link->tgt_prog = tgt_prog;
2693 link->trampoline = tr;
2694
2695 /* Always clear the trampoline and target prog from prog->aux to make
2696 * sure the original attach destination is not kept alive after a
2697 * program is (re-)attached to another target.
2698 */
2699 if (prog->aux->dst_prog &&
2700 (tgt_prog_fd || tr != prog->aux->dst_trampoline))
2701 /* got extra prog ref from syscall, or attaching to different prog */
2702 bpf_prog_put(prog->aux->dst_prog);
2703 if (prog->aux->dst_trampoline && tr != prog->aux->dst_trampoline)
2704 /* we allocated a new trampoline, so free the old one */
2705 bpf_trampoline_put(prog->aux->dst_trampoline);
2706
2707 prog->aux->dst_prog = NULL;
2708 prog->aux->dst_trampoline = NULL;
2709 mutex_unlock(&prog->aux->dst_mutex);
2710
2588 return bpf_link_settle(&link_primer);2711 return bpf_link_settle(&link_primer);
2712out_unlock:
2713 if (tr && tr != prog->aux->dst_trampoline)
2714 bpf_trampoline_put(tr);
2715 mutex_unlock(&prog->aux->dst_mutex);
2716 kfree(link);
2589out_put_prog:2717out_put_prog:
2718 if (tgt_prog_fd && tgt_prog)
2719 bpf_prog_put(tgt_prog);
2590 bpf_prog_put(prog);2720 bpf_prog_put(prog);
2591 return err;2721 return err;
2592}2722}
@@ -2700,7 +2830,7 @@ static int bpf_raw_tracepoint_open(const union bpf_attr *attr) · 107 unmodified lines
2700 tp_name = prog->aux->attach_func_name;2830 tp_name = prog->aux->attach_func_name;
2701 break;2831 break;
2702 }2832 }
2703 return bpf_tracing_prog_attach(prog);2833 return bpf_tracing_prog_attach(prog, 0, 0);
2704 case BPF_PROG_TYPE_RAW_TRACEPOINT:2834 case BPF_PROG_TYPE_RAW_TRACEPOINT:
2705 case BPF_PROG_TYPE_RAW_TRACEPOINT_WRITABLE:2835 case BPF_PROG_TYPE_RAW_TRACEPOINT_WRITABLE:
2706 if (strncpy_from_user(buf,2836 if (strncpy_from_user(buf,
@@ -2969,7 +3099,7 @@ static int bpf_prog_query(const union bpf_attr *attr, · 262 unmodified lines
2969 }3099 }
2970}3100}
2971 3101
2972#define BPF_PROG_TEST_RUN_LAST_FIELD test.ctx_out3102#define BPF_PROG_TEST_RUN_LAST_FIELD test.cpu
2973 3103
2974static int bpf_prog_test_run(const union bpf_attr *attr,3104static int bpf_prog_test_run(const union bpf_attr *attr,
2975 union bpf_attr __user *uattr)3105 union bpf_attr __user *uattr)
@@ -3152,21 +3282,25 @@ static const struct bpf_map *bpf_map_from_imm(const struct bpf_prog *prog, · 176 unmodified lines
3152 const struct bpf_map *map;3282 const struct bpf_map *map;
3153 int i;3283 int i;
3154 3284
3285 mutex_lock(&prog->aux->used_maps_mutex);
3155 for (i = 0, *off = 0; i < prog->aux->used_map_cnt; i++) {3286 for (i = 0, *off = 0; i < prog->aux->used_map_cnt; i++) {
3156 map = prog->aux->used_maps[i];3287 map = prog->aux->used_maps[i];
3157 if (map == (void *)addr) {3288 if (map == (void *)addr) {
3158 *type = BPF_PSEUDO_MAP_FD;3289 *type = BPF_PSEUDO_MAP_FD;
3159 return map;3290 goto out;
3160 }3291 }
3161 if (!map->ops->map_direct_value_meta)3292 if (!map->ops->map_direct_value_meta)
3162 continue;3293 continue;
3163 if (!map->ops->map_direct_value_meta(map, addr, off)) {3294 if (!map->ops->map_direct_value_meta(map, addr, off)) {
3164 *type = BPF_PSEUDO_MAP_VALUE;3295 *type = BPF_PSEUDO_MAP_VALUE;
3165 return map;3296 goto out;
3166 }3297 }
3167 }3298 }
3299 map = NULL;
3168 3300
3169 return NULL;3301out:
3302 mutex_unlock(&prog->aux->used_maps_mutex);
3303 return map;
3170}3304}
3171 3305
3172static struct bpf_insn *bpf_insn_prepare_dump(const struct bpf_prog *prog,3306static struct bpf_insn *bpf_insn_prepare_dump(const struct bpf_prog *prog,
@@ -3284,6 +3418,7 @@ static int bpf_prog_get_info_by_fd(struct file *file, · 111 unmodified lines
3284 memcpy(info.tag, prog->tag, sizeof(prog->tag));3418 memcpy(info.tag, prog->tag, sizeof(prog->tag));
3285 memcpy(info.name, prog->aux->name, sizeof(prog->aux->name));3419 memcpy(info.name, prog->aux->name, sizeof(prog->aux->name));
3286 3420
3421 mutex_lock(&prog->aux->used_maps_mutex);
3287 ulen = info.nr_map_ids;3422 ulen = info.nr_map_ids;
3288 info.nr_map_ids = prog->aux->used_map_cnt;3423 info.nr_map_ids = prog->aux->used_map_cnt;
3289 ulen = min_t(u32, info.nr_map_ids, ulen);3424 ulen = min_t(u32, info.nr_map_ids, ulen);
@@ -3293,9 +3428,12 @@ static int bpf_prog_get_info_by_fd(struct file *file, · 3 unmodified lines
3293 3428
3294 for (i = 0; i < ulen; i++)3429 for (i = 0; i < ulen; i++)
3295 if (put_user(prog->aux->used_maps[i]->id,3430 if (put_user(prog->aux->used_maps[i]->id,
3296 &user_map_ids[i]))3431 &user_map_ids[i])) {
3432 mutex_unlock(&prog->aux->used_maps_mutex);
3297 return -EFAULT;3433 return -EFAULT;
3434 }
3298 }3435 }
3436 mutex_unlock(&prog->aux->used_maps_mutex);
3299 3437
3300 err = set_info_rec_size(&info);3438 err = set_info_rec_size(&info);
3301 if (err)3439 if (err)
@@ -3876,10 +4014,15 @@ err_put: · 574 unmodified lines
3876 4014
3877static int tracing_bpf_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)4015static int tracing_bpf_link_attach(const union bpf_attr *attr, struct bpf_prog *prog)
3878{4016{
3879 if (attr->link_create.attach_type == BPF_TRACE_ITER &&4017 if (attr->link_create.attach_type != prog->expected_attach_type)
3880 prog->expected_attach_type == BPF_TRACE_ITER)4018 return -EINVAL;
3881 return bpf_iter_link_attach(attr, prog);
3882 4019
4020 if (prog->expected_attach_type == BPF_TRACE_ITER)
4021 return bpf_iter_link_attach(attr, prog);
4022 else if (prog->type == BPF_PROG_TYPE_EXT)
4023 return bpf_tracing_prog_attach(prog,
4024 attr->link_create.target_fd,
4025 attr->link_create.target_btf_id);
3883 return -EINVAL;4026 return -EINVAL;
3884}4027}
3885 4028
@@ -3893,18 +4036,25 @@ static int link_create(union bpf_attr *attr) · 7 unmodified lines
3893 if (CHECK_ATTR(BPF_LINK_CREATE))4036 if (CHECK_ATTR(BPF_LINK_CREATE))
3894 return -EINVAL;4037 return -EINVAL;
3895 4038
3896 ptype = attach_type_to_prog_type(attr->link_create.attach_type);4039 prog = bpf_prog_get(attr->link_create.prog_fd);
3897 if (ptype == BPF_PROG_TYPE_UNSPEC)
3898 return -EINVAL;
3899
3900 prog = bpf_prog_get_type(attr->link_create.prog_fd, ptype);
3901 if (IS_ERR(prog))4040 if (IS_ERR(prog))
3902 return PTR_ERR(prog);4041 return PTR_ERR(prog);
3903 4042
3904 ret = bpf_prog_attach_check_attach_type(prog,4043 ret = bpf_prog_attach_check_attach_type(prog,
3905 attr->link_create.attach_type);4044 attr->link_create.attach_type);
3906 if (ret)4045 if (ret)
3907 goto err_out;4046 goto out;
4047
4048 if (prog->type == BPF_PROG_TYPE_EXT) {
4049 ret = tracing_bpf_link_attach(attr, prog);
4050 goto out;
4051 }
4052
4053 ptype = attach_type_to_prog_type(attr->link_create.attach_type);
4054 if (ptype == BPF_PROG_TYPE_UNSPEC || ptype != prog->type) {
4055 ret = -EINVAL;
4056 goto out;
4057 }
3908 4058
3909 switch (ptype) {4059 switch (ptype) {
3910 case BPF_PROG_TYPE_CGROUP_SKB:4060 case BPF_PROG_TYPE_CGROUP_SKB:
@@ -3932,7 +4082,7 @@ static int link_create(union bpf_attr *attr) · 21 unmodified lines
3932 ret = -EINVAL;4082 ret = -EINVAL;
3933 }4083 }
3934 4084
3935err_out:4085out:
3936 if (ret < 0)4086 if (ret < 0)
3937 bpf_prog_put(prog);4087 bpf_prog_put(prog);
3938 return ret;4088 return ret;
@@ -4014,40 +4164,50 @@ static int link_detach(union bpf_attr *attr) · 75 unmodified lines
4014 return ret;4164 return ret;
4015}4165}
4016 4166
4017static int bpf_link_inc_not_zero(struct bpf_link *link)4167static struct bpf_link *bpf_link_inc_not_zero(struct bpf_link *link)
4018{4168{
4019 return atomic64_fetch_add_unless(&link->refcnt, 1, 0) ? 0 : -ENOENT;4169 return atomic64_fetch_add_unless(&link->refcnt, 1, 0) ? link : ERR_PTR(-ENOENT);
4020}4170}
4021 4171
4022#define BPF_LINK_GET_FD_BY_ID_LAST_FIELD link_id4172struct bpf_link *bpf_link_by_id(u32 id)
4023
4024static int bpf_link_get_fd_by_id(const union bpf_attr *attr)
4025{4173{
4026 struct bpf_link *link;4174 struct bpf_link *link;
4027 u32 id = attr->link_id;
4028 int fd, err;
4029 4175
4030 if (CHECK_ATTR(BPF_LINK_GET_FD_BY_ID))4176 if (!id)
4031 return -EINVAL;4177 return ERR_PTR(-ENOENT);
4032
4033 if (!capable(CAP_SYS_ADMIN))
4034 return -EPERM;
4035 4178
4036 spin_lock_bh(&link_idr_lock);4179 spin_lock_bh(&link_idr_lock);
4037 link = idr_find(&link_idr, id);
4038 /* before link is "settled", ID is 0, pretend it doesn't exist yet */4180 /* before link is "settled", ID is 0, pretend it doesn't exist yet */
4181 link = idr_find(&link_idr, id);
4039 if (link) {4182 if (link) {
4040 if (link->id)4183 if (link->id)
4041 err = bpf_link_inc_not_zero(link);4184 link = bpf_link_inc_not_zero(link);
4042 else4185 else
4043 err = -EAGAIN;4186 link = ERR_PTR(-EAGAIN);
4044 } else {4187 } else {
4045 err = -ENOENT;4188 link = ERR_PTR(-ENOENT);
4046 }4189 }
4047 spin_unlock_bh(&link_idr_lock);4190 spin_unlock_bh(&link_idr_lock);
4191 return link;
4192}
4048 4193
4049 if (err)4194#define BPF_LINK_GET_FD_BY_ID_LAST_FIELD link_id
4050 return err;4195
4196static int bpf_link_get_fd_by_id(const union bpf_attr *attr)
4197{
4198 struct bpf_link *link;
4199 u32 id = attr->link_id;
4200 int fd;
4201
4202 if (CHECK_ATTR(BPF_LINK_GET_FD_BY_ID))
4203 return -EINVAL;
4204
4205 if (!capable(CAP_SYS_ADMIN))
4206 return -EPERM;
4207
4208 link = bpf_link_by_id(id);
4209 if (IS_ERR(link))
4210 return PTR_ERR(link);
4051 4211
4052 fd = bpf_link_new_fd(link);4212 fd = bpf_link_new_fd(link);
4053 if (fd < 0)4213 if (fd < 0)
@@ -4133,6 +4293,68 @@ static int bpf_iter_create(union bpf_attr *attr) · 79 unmodified lines
4133 return err;4293 return err;
4134}4294}
4135 4295
4296#define BPF_PROG_BIND_MAP_LAST_FIELD prog_bind_map.flags
4297
4298static int bpf_prog_bind_map(union bpf_attr *attr)
4299{
4300 struct bpf_prog *prog;
4301 struct bpf_map *map;
4302 struct bpf_map **used_maps_old, **used_maps_new;
4303 int i, ret = 0;
4304
4305 if (CHECK_ATTR(BPF_PROG_BIND_MAP))
4306 return -EINVAL;
4307
4308 if (attr->prog_bind_map.flags)
4309 return -EINVAL;
4310
4311 prog = bpf_prog_get(attr->prog_bind_map.prog_fd);
4312 if (IS_ERR(prog))
4313 return PTR_ERR(prog);
4314
4315 map = bpf_map_get(attr->prog_bind_map.map_fd);
4316 if (IS_ERR(map)) {
4317 ret = PTR_ERR(map);
4318 goto out_prog_put;
4319 }
4320
4321 mutex_lock(&prog->aux->used_maps_mutex);
4322
4323 used_maps_old = prog->aux->used_maps;
4324
4325 for (i = 0; i < prog->aux->used_map_cnt; i++)
4326 if (used_maps_old[i] == map) {
4327 bpf_map_put(map);
4328 goto out_unlock;
4329 }
4330
4331 used_maps_new = kmalloc_array(prog->aux->used_map_cnt + 1,
4332 sizeof(used_maps_new[0]),
4333 GFP_KERNEL);
4334 if (!used_maps_new) {
4335 ret = -ENOMEM;
4336 goto out_unlock;
4337 }
4338
4339 memcpy(used_maps_new, used_maps_old,
4340 sizeof(used_maps_old[0]) * prog->aux->used_map_cnt);
4341 used_maps_new[prog->aux->used_map_cnt] = map;
4342
4343 prog->aux->used_map_cnt++;
4344 prog->aux->used_maps = used_maps_new;
4345
4346 kfree(used_maps_old);
4347
4348out_unlock:
4349 mutex_unlock(&prog->aux->used_maps_mutex);
4350
4351 if (ret)
4352 bpf_map_put(map);
4353out_prog_put:
4354 bpf_prog_put(prog);
4355 return ret;
4356}
4357
4136SYSCALL_DEFINE3(bpf, int, cmd, union bpf_attr __user *, uattr, unsigned int, size)4358SYSCALL_DEFINE3(bpf, int, cmd, union bpf_attr __user *, uattr, unsigned int, size)
4137{4359{
4138 union bpf_attr attr;4360 union bpf_attr attr;
@@ -4266,6 +4488,9 @@ SYSCALL_DEFINE3(bpf, int, cmd, union bpf_attr __user *, uattr, unsigned int, siz · 127 unmodified lines
4266 case BPF_LINK_DETACH:4488 case BPF_LINK_DETACH:
4267 err = link_detach(&attr);4489 err = link_detach(&attr);
4268 break;4490 break;
4491 case BPF_PROG_BIND_MAP:
4492 err = bpf_prog_bind_map(&attr);
4493 break;
4269 default:4494 default:
4270 err = -EINVAL;4495 err = -EINVAL;
4271 break;4496 break;
kernel/bpf/task_iter.c+11−4
@@ -22,7 +22,8 @@ struct bpf_iter_seq_task_info { · 21 unmodified lines
22};22};
23 23
24static struct task_struct *task_seq_get_next(struct pid_namespace *ns,24static struct task_struct *task_seq_get_next(struct pid_namespace *ns,
25 u32 *tid)25 u32 *tid,
26 bool skip_if_dup_files)
26{27{
27 struct task_struct *task = NULL;28 struct task_struct *task = NULL;
28 struct pid *pid;29 struct pid *pid;
@@ -36,6 +37,12 @@ retry: · 7 unmodified lines
36 if (!task) {37 if (!task) {
37 ++*tid;38 ++*tid;
38 goto retry;39 goto retry;
40 } else if (skip_if_dup_files && task->tgid != task->pid &&
41 task->files == task->group_leader->files) {
42 put_task_struct(task);
43 task = NULL;
44 ++*tid;
45 goto retry;
39 }46 }
40 }47 }
41 rcu_read_unlock();48 rcu_read_unlock();
@@ -48,7 +55,7 @@ static void *task_seq_start(struct seq_file *seq, loff_t *pos) · 6 unmodified lines
48 struct bpf_iter_seq_task_info *info = seq->private;55 struct bpf_iter_seq_task_info *info = seq->private;
49 struct task_struct *task;56 struct task_struct *task;
50 57
51 task = task_seq_get_next(info->common.ns, &info->tid);58 task = task_seq_get_next(info->common.ns, &info->tid, false);
52 if (!task)59 if (!task)
53 return NULL;60 return NULL;
54 61
@@ -65,7 +72,7 @@ static void *task_seq_next(struct seq_file *seq, void *v, loff_t *pos) · 10 unmodified lines
65 ++*pos;72 ++*pos;
66 ++info->tid;73 ++info->tid;
67 put_task_struct((struct task_struct *)v);74 put_task_struct((struct task_struct *)v);
68 task = task_seq_get_next(info->common.ns, &info->tid);75 task = task_seq_get_next(info->common.ns, &info->tid, false);
69 if (!task)76 if (!task)
70 return NULL;77 return NULL;
71 78
@@ -148,7 +155,7 @@ again: · 76 unmodified lines
148 curr_files = *fstruct;155 curr_files = *fstruct;
149 curr_fd = info->fd;156 curr_fd = info->fd;
150 } else {157 } else {
151 curr_task = task_seq_get_next(ns, &curr_tid);158 curr_task = task_seq_get_next(ns, &curr_tid, true);
152 if (!curr_task)159 if (!curr_task)
153 return NULL;160 return NULL;
154 161
kernel/bpf/trampoline.c+51−12
@@ -7,6 +7,8 @@ · 6 unmodified lines
7#include <linux/rbtree_latch.h>7#include <linux/rbtree_latch.h>
8#include <linux/perf_event.h>8#include <linux/perf_event.h>
9#include <linux/btf.h>9#include <linux/btf.h>
10#include <linux/rcupdate_trace.h>
11#include <linux/rcupdate_wait.h>
10 12
11/* dummy _ops. The verifier will operate on target program's ops. */13/* dummy _ops. The verifier will operate on target program's ops. */
12const struct bpf_verifier_ops bpf_extension_verifier_ops = {14const struct bpf_verifier_ops bpf_extension_verifier_ops = {
@@ -63,7 +65,7 @@ static void bpf_trampoline_ksym_add(struct bpf_trampoline *tr) · 50 unmodified lines
63 bpf_image_ksym_add(tr->image, ksym);65 bpf_image_ksym_add(tr->image, ksym);
64}66}
65 67
66struct bpf_trampoline *bpf_trampoline_lookup(u64 key)68static struct bpf_trampoline *bpf_trampoline_lookup(u64 key)
67{69{
68 struct bpf_trampoline *tr;70 struct bpf_trampoline *tr;
69 struct hlist_head *head;71 struct hlist_head *head;
@@ -210,9 +212,12 @@ static int bpf_trampoline_update(struct bpf_trampoline *tr) · 140 unmodified lines
210 * updates to trampoline would change the code from underneath the212 * updates to trampoline would change the code from underneath the
211 * preempted task. Hence wait for tasks to voluntarily schedule or go213 * preempted task. Hence wait for tasks to voluntarily schedule or go
212 * to userspace.214 * to userspace.
215 * The same trampoline can hold both sleepable and non-sleepable progs.
216 * synchronize_rcu_tasks_trace() is needed to make sure all sleepable
217 * programs finish executing.
218 * Wait for these two grace periods together.
213 */219 */
214 220 synchronize_rcu_mult(call_rcu_tasks, call_rcu_tasks_trace);
215 synchronize_rcu_tasks();
216 221
217 err = arch_prepare_bpf_trampoline(new_image, new_image + PAGE_SIZE / 2,222 err = arch_prepare_bpf_trampoline(new_image, new_image + PAGE_SIZE / 2,
218 &tr->func.model, flags, tprogs,223 &tr->func.model, flags, tprogs,
@@ -256,14 +261,12 @@ static enum bpf_tramp_prog_type bpf_attach_type_to_tramp(struct bpf_prog *prog) · 37 unmodified lines
256 }261 }
257}262}
258 263
259int bpf_trampoline_link_prog(struct bpf_prog *prog)264int bpf_trampoline_link_prog(struct bpf_prog *prog, struct bpf_trampoline *tr)
260{265{
261 enum bpf_tramp_prog_type kind;266 enum bpf_tramp_prog_type kind;
262 struct bpf_trampoline *tr;
263 int err = 0;267 int err = 0;
264 int cnt;268 int cnt;
265 269
266 tr = prog->aux->trampoline;
267 kind = bpf_attach_type_to_tramp(prog);270 kind = bpf_attach_type_to_tramp(prog);
268 mutex_lock(&tr->mutex);271 mutex_lock(&tr->mutex);
269 if (tr->extension_prog) {272 if (tr->extension_prog) {
@@ -296,7 +299,7 @@ int bpf_trampoline_link_prog(struct bpf_prog *prog) · 26 unmodified lines
296 }299 }
297 hlist_add_head(&prog->aux->tramp_hlist, &tr->progs_hlist[kind]);300 hlist_add_head(&prog->aux->tramp_hlist, &tr->progs_hlist[kind]);
298 tr->progs_cnt[kind]++;301 tr->progs_cnt[kind]++;
299 err = bpf_trampoline_update(prog->aux->trampoline);302 err = bpf_trampoline_update(tr);
300 if (err) {303 if (err) {
301 hlist_del(&prog->aux->tramp_hlist);304 hlist_del(&prog->aux->tramp_hlist);
302 tr->progs_cnt[kind]--;305 tr->progs_cnt[kind]--;
@@ -307,13 +310,11 @@ out: · 4 unmodified lines
307}310}
308 311
309/* bpf_trampoline_unlink_prog() should never fail. */312/* bpf_trampoline_unlink_prog() should never fail. */
310int bpf_trampoline_unlink_prog(struct bpf_prog *prog)313int bpf_trampoline_unlink_prog(struct bpf_prog *prog, struct bpf_trampoline *tr)
311{314{
312 enum bpf_tramp_prog_type kind;315 enum bpf_tramp_prog_type kind;
313 struct bpf_trampoline *tr;
314 int err;316 int err;
315 317
316 tr = prog->aux->trampoline;
317 kind = bpf_attach_type_to_tramp(prog);318 kind = bpf_attach_type_to_tramp(prog);
318 mutex_lock(&tr->mutex);319 mutex_lock(&tr->mutex);
319 if (kind == BPF_TRAMP_REPLACE) {320 if (kind == BPF_TRAMP_REPLACE) {
@@ -325,12 +326,32 @@ int bpf_trampoline_unlink_prog(struct bpf_prog *prog) · 5 unmodified lines
325 }326 }
326 hlist_del(&prog->aux->tramp_hlist);327 hlist_del(&prog->aux->tramp_hlist);
327 tr->progs_cnt[kind]--;328 tr->progs_cnt[kind]--;
328 err = bpf_trampoline_update(prog->aux->trampoline);329 err = bpf_trampoline_update(tr);
329out:330out:
330 mutex_unlock(&tr->mutex);331 mutex_unlock(&tr->mutex);
331 return err;332 return err;
332}333}
333 334
335struct bpf_trampoline *bpf_trampoline_get(u64 key,
336 struct bpf_attach_target_info *tgt_info)
337{
338 struct bpf_trampoline *tr;
339
340 tr = bpf_trampoline_lookup(key);
341 if (!tr)
342 return NULL;
343
344 mutex_lock(&tr->mutex);
345 if (tr->func.addr)
346 goto out;
347
348 memcpy(&tr->func.model, &tgt_info->fmodel, sizeof(tgt_info->fmodel));
349 tr->func.addr = (void *)tgt_info->tgt_addr;
350out:
351 mutex_unlock(&tr->mutex);
352 return tr;
353}
354
334void bpf_trampoline_put(struct bpf_trampoline *tr)355void bpf_trampoline_put(struct bpf_trampoline *tr)
335{356{
336 if (!tr)357 if (!tr)
@@ -344,7 +365,14 @@ void bpf_trampoline_put(struct bpf_trampoline *tr) · 7 unmodified lines
344 if (WARN_ON_ONCE(!hlist_empty(&tr->progs_hlist[BPF_TRAMP_FEXIT])))365 if (WARN_ON_ONCE(!hlist_empty(&tr->progs_hlist[BPF_TRAMP_FEXIT])))
345 goto out;366 goto out;
346 bpf_image_ksym_del(&tr->ksym);367 bpf_image_ksym_del(&tr->ksym);
347 /* wait for tasks to get out of trampoline before freeing it */368 /* This code will be executed when all bpf progs (both sleepable and
369 * non-sleepable) went through
370 * bpf_prog_put()->call_rcu[_tasks_trace]()->bpf_prog_free_deferred().
371 * Hence no need for another synchronize_rcu_tasks_trace() here,
372 * but synchronize_rcu_tasks() is still needed, since trampoline
373 * may not have had any sleepable programs and we need to wait
374 * for tasks to get out of trampoline code before freeing it.
375 */
348 synchronize_rcu_tasks();376 synchronize_rcu_tasks();
349 bpf_jit_free_exec(tr->image);377 bpf_jit_free_exec(tr->image);
350 hlist_del(&tr->hlist);378 hlist_del(&tr->hlist);
@@ -394,6 +422,17 @@ void notrace __bpf_prog_exit(struct bpf_prog *prog, u64 start) · 43 unmodified lines
394 rcu_read_unlock();422 rcu_read_unlock();
395}423}
396 424
425void notrace __bpf_prog_enter_sleepable(void)
426{
427 rcu_read_lock_trace();
428 might_fault();
429}
430
431void notrace __bpf_prog_exit_sleepable(void)
432{
433 rcu_read_unlock_trace();
434}
435
397int __weak436int __weak
398arch_prepare_bpf_trampoline(void *image, void *image_end,437arch_prepare_bpf_trampoline(void *image, void *image_end,
399 const struct btf_func_model *m, u32 flags,438 const struct btf_func_model *m, u32 flags,
kernel/bpf/verifier.c+1060−328
@@ -21,6 +21,7 @@ · 20 unmodified lines
21#include <linux/ctype.h>21#include <linux/ctype.h>
22#include <linux/error-injection.h>22#include <linux/error-injection.h>
23#include <linux/bpf_lsm.h>23#include <linux/bpf_lsm.h>
24#include <linux/btf_ids.h>
24 25
25#include "disasm.h"26#include "disasm.h"
26 27
@@ -238,6 +239,7 @@ struct bpf_call_arg_meta { · 211 unmodified lines
238 int ref_obj_id;239 int ref_obj_id;
239 int func_id;240 int func_id;
240 u32 btf_id;241 u32 btf_id;
242 u32 ret_btf_id;
241};243};
242 244
243struct btf *btf_vmlinux;245struct btf *btf_vmlinux;
@@ -435,6 +437,15 @@ static bool arg_type_may_be_refcounted(enum bpf_arg_type type) · 191 unmodified lines
435 return type == ARG_PTR_TO_SOCK_COMMON;437 return type == ARG_PTR_TO_SOCK_COMMON;
436}438}
437 439
440static bool arg_type_may_be_null(enum bpf_arg_type type)
441{
442 return type == ARG_PTR_TO_MAP_VALUE_OR_NULL ||
443 type == ARG_PTR_TO_MEM_OR_NULL ||
444 type == ARG_PTR_TO_CTX_OR_NULL ||
445 type == ARG_PTR_TO_SOCKET_OR_NULL ||
446 type == ARG_PTR_TO_ALLOC_MEM_OR_NULL;
447}
448
438/* Determine whether the function releases some resources allocated by another449/* Determine whether the function releases some resources allocated by another
439 * function call. The first reference type argument will be assumed to be450 * function call. The first reference type argument will be assumed to be
440 * released by release_reference().451 * released by release_reference().
@@ -477,7 +488,12 @@ static bool is_acquire_function(enum bpf_func_id func_id, · 36 unmodified lines
477static bool is_ptr_cast_function(enum bpf_func_id func_id)488static bool is_ptr_cast_function(enum bpf_func_id func_id)
478{489{
479 return func_id == BPF_FUNC_tcp_sock ||490 return func_id == BPF_FUNC_tcp_sock ||
480 func_id == BPF_FUNC_sk_fullsock;491 func_id == BPF_FUNC_sk_fullsock ||
492 func_id == BPF_FUNC_skc_to_tcp_sock ||
493 func_id == BPF_FUNC_skc_to_tcp6_sock ||
494 func_id == BPF_FUNC_skc_to_udp6_sock ||
495 func_id == BPF_FUNC_skc_to_tcp_timewait_sock ||
496 func_id == BPF_FUNC_skc_to_tcp_request_sock;
481}497}
482 498
483/* string representation of 'enum bpf_reg_type' */499/* string representation of 'enum bpf_reg_type' */
@@ -503,6 +519,7 @@ static const char * const reg_type_str[] = { · 19 unmodified lines
503 [PTR_TO_XDP_SOCK] = "xdp_sock",519 [PTR_TO_XDP_SOCK] = "xdp_sock",
504 [PTR_TO_BTF_ID] = "ptr_",520 [PTR_TO_BTF_ID] = "ptr_",
505 [PTR_TO_BTF_ID_OR_NULL] = "ptr_or_null_",521 [PTR_TO_BTF_ID_OR_NULL] = "ptr_or_null_",
522 [PTR_TO_PERCPU_BTF_ID] = "percpu_ptr_",
506 [PTR_TO_MEM] = "mem",523 [PTR_TO_MEM] = "mem",
507 [PTR_TO_MEM_OR_NULL] = "mem_or_null",524 [PTR_TO_MEM_OR_NULL] = "mem_or_null",
508 [PTR_TO_RDONLY_BUF] = "rdonly_buf",525 [PTR_TO_RDONLY_BUF] = "rdonly_buf",
@@ -569,7 +586,9 @@ static void print_verifier_state(struct bpf_verifier_env *env, · 60 unmodified lines
569 /* reg->off should be 0 for SCALAR_VALUE */586 /* reg->off should be 0 for SCALAR_VALUE */
570 verbose(env, "%lld", reg->var_off.value + reg->off);587 verbose(env, "%lld", reg->var_off.value + reg->off);
571 } else {588 } else {
572 if (t == PTR_TO_BTF_ID || t == PTR_TO_BTF_ID_OR_NULL)589 if (t == PTR_TO_BTF_ID ||
590 t == PTR_TO_BTF_ID_OR_NULL ||
591 t == PTR_TO_PERCPU_BTF_ID)
573 verbose(env, "%s", kernel_type_name(reg->btf_id));592 verbose(env, "%s", kernel_type_name(reg->btf_id));
574 verbose(env, "(id=%d", reg->id);593 verbose(env, "(id=%d", reg->id);
575 if (reg_type_may_be_refcounted_or_null(t))594 if (reg_type_may_be_refcounted_or_null(t))
@@ -991,14 +1010,9 @@ static const int caller_saved[CALLER_SAVED_REGS] = { · 415 unmodified lines
991static void __mark_reg_not_init(const struct bpf_verifier_env *env,1010static void __mark_reg_not_init(const struct bpf_verifier_env *env,
992 struct bpf_reg_state *reg);1011 struct bpf_reg_state *reg);
993 1012
994/* Mark the unknown part of a register (variable offset or scalar value) as1013/* This helper doesn't clear reg->id */
995 * known to have the value @imm.1014static void ___mark_reg_known(struct bpf_reg_state *reg, u64 imm)
996 */
997static void __mark_reg_known(struct bpf_reg_state *reg, u64 imm)
998{1015{
999 /* Clear id, off, and union(map_ptr, range) */
1000 memset(((u8 *)reg) + sizeof(reg->type), 0,
1001 offsetof(struct bpf_reg_state, var_off) - sizeof(reg->type));
1002 reg->var_off = tnum_const(imm);1016 reg->var_off = tnum_const(imm);
1003 reg->smin_value = (s64)imm;1017 reg->smin_value = (s64)imm;
1004 reg->smax_value = (s64)imm;1018 reg->smax_value = (s64)imm;
@@ -1011,6 +1025,17 @@ static void __mark_reg_known(struct bpf_reg_state *reg, u64 imm) · 6 unmodified lines
1011 reg->u32_max_value = (u32)imm;1025 reg->u32_max_value = (u32)imm;
1012}1026}
1013 1027
1028/* Mark the unknown part of a register (variable offset or scalar value) as
1029 * known to have the value @imm.
1030 */
1031static void __mark_reg_known(struct bpf_reg_state *reg, u64 imm)
1032{
1033 /* Clear id, off, and union(map_ptr, range) */
1034 memset(((u8 *)reg) + sizeof(reg->type), 0,
1035 offsetof(struct bpf_reg_state, var_off) - sizeof(reg->type));
1036 ___mark_reg_known(reg, imm);
1037}
1038
1014static void __mark_reg32_known(struct bpf_reg_state *reg, u64 imm)1039static void __mark_reg32_known(struct bpf_reg_state *reg, u64 imm)
1015{1040{
1016 reg->var_off = tnum_const_subreg(reg->var_off, imm);1041 reg->var_off = tnum_const_subreg(reg->var_off, imm);
@@ -1489,6 +1514,13 @@ static int check_subprogs(struct bpf_verifier_env *env) · 472 unmodified lines
1489 for (i = 0; i < insn_cnt; i++) {1514 for (i = 0; i < insn_cnt; i++) {
1490 u8 code = insn[i].code;1515 u8 code = insn[i].code;
1491 1516
1517 if (code == (BPF_JMP | BPF_CALL) &&
1518 insn[i].imm == BPF_FUNC_tail_call &&
1519 insn[i].src_reg != BPF_PSEUDO_CALL)
1520 subprog[cur_subprog].has_tail_call = true;
1521 if (BPF_CLASS(code) == BPF_LD &&
1522 (BPF_MODE(code) == BPF_ABS || BPF_MODE(code) == BPF_IND))
1523 subprog[cur_subprog].has_ld_abs = true;
1492 if (BPF_CLASS(code) != BPF_JMP && BPF_CLASS(code) != BPF_JMP32)1524 if (BPF_CLASS(code) != BPF_JMP && BPF_CLASS(code) != BPF_JMP32)
1493 goto next;1525 goto next;
1494 if (BPF_OP(code) == BPF_EXIT || BPF_OP(code) == BPF_CALL)1526 if (BPF_OP(code) == BPF_EXIT || BPF_OP(code) == BPF_CALL)
@@ -2183,6 +2215,7 @@ static bool is_spillable_regtype(enum bpf_reg_type type) · 688 unmodified lines
2183 case PTR_TO_RDONLY_BUF_OR_NULL:2215 case PTR_TO_RDONLY_BUF_OR_NULL:
2184 case PTR_TO_RDWR_BUF:2216 case PTR_TO_RDWR_BUF:
2185 case PTR_TO_RDWR_BUF_OR_NULL:2217 case PTR_TO_RDWR_BUF_OR_NULL:
2218 case PTR_TO_PERCPU_BTF_ID:
2186 return true;2219 return true;
2187 default:2220 default:
2188 return false;2221 return false;
@@ -2200,6 +2233,20 @@ static bool register_is_const(struct bpf_reg_state *reg) · 11 unmodified lines
2200 return reg->type == SCALAR_VALUE && tnum_is_const(reg->var_off);2233 return reg->type == SCALAR_VALUE && tnum_is_const(reg->var_off);
2201}2234}
2202 2235
2236static bool __is_scalar_unbounded(struct bpf_reg_state *reg)
2237{
2238 return tnum_is_unknown(reg->var_off) &&
2239 reg->smin_value == S64_MIN && reg->smax_value == S64_MAX &&
2240 reg->umin_value == 0 && reg->umax_value == U64_MAX &&
2241 reg->s32_min_value == S32_MIN && reg->s32_max_value == S32_MAX &&
2242 reg->u32_min_value == 0 && reg->u32_max_value == U32_MAX;
2243}
2244
2245static bool register_is_bounded(struct bpf_reg_state *reg)
2246{
2247 return reg->type == SCALAR_VALUE && !__is_scalar_unbounded(reg);
2248}
2249
2203static bool __is_pointer_value(bool allow_ptr_leaks,2250static bool __is_pointer_value(bool allow_ptr_leaks,
2204 const struct bpf_reg_state *reg)2251 const struct bpf_reg_state *reg)
2205{2252{
@@ -2251,7 +2298,7 @@ static int check_stack_write(struct bpf_verifier_env *env, · 45 unmodified lines
2251 if (value_regno >= 0)2298 if (value_regno >= 0)
2252 reg = &cur->regs[value_regno];2299 reg = &cur->regs[value_regno];
2253 2300
2254 if (reg && size == BPF_REG_SIZE && register_is_const(reg) &&2301 if (reg && size == BPF_REG_SIZE && register_is_bounded(reg) &&
2255 !register_is_null(reg) && env->bpf_capable) {2302 !register_is_null(reg) && env->bpf_capable) {
2256 if (dst_reg != BPF_REG_FP) {2303 if (dst_reg != BPF_REG_FP) {
2257 /* The backtracking logic can only recognize explicit2304 /* The backtracking logic can only recognize explicit
@@ -2625,11 +2672,18 @@ static int check_map_access(struct bpf_verifier_env *env, u32 regno, · 367 unmodified lines
2625 2672
2626#define MAX_PACKET_OFF 0xffff2673#define MAX_PACKET_OFF 0xffff
2627 2674
2675static enum bpf_prog_type resolve_prog_type(struct bpf_prog *prog)
2676{
2677 return prog->aux->dst_prog ? prog->aux->dst_prog->type : prog->type;
2678}
2679
2628static bool may_access_direct_pkt_data(struct bpf_verifier_env *env,2680static bool may_access_direct_pkt_data(struct bpf_verifier_env *env,
2629 const struct bpf_call_arg_meta *meta,2681 const struct bpf_call_arg_meta *meta,
2630 enum bpf_access_type t)2682 enum bpf_access_type t)
2631{2683{
2632 switch (env->prog->type) {2684 enum bpf_prog_type prog_type = resolve_prog_type(env->prog);
2685
2686 switch (prog_type) {
2633 /* Program types only with direct read access go here! */2687 /* Program types only with direct read access go here! */
2634 case BPF_PROG_TYPE_LWT_IN:2688 case BPF_PROG_TYPE_LWT_IN:
2635 case BPF_PROG_TYPE_LWT_OUT:2689 case BPF_PROG_TYPE_LWT_OUT:
@@ -2639,7 +2693,7 @@ static bool may_access_direct_pkt_data(struct bpf_verifier_env *env, · 3 unmodified lines
2639 case BPF_PROG_TYPE_CGROUP_SKB:2693 case BPF_PROG_TYPE_CGROUP_SKB:
2640 if (t == BPF_WRITE)2694 if (t == BPF_WRITE)
2641 return false;2695 return false;
2642 /* fallthrough */2696 fallthrough;
2643 2697
2644 /* Program types with direct read + write access go here! */2698 /* Program types with direct read + write access go here! */
2645 case BPF_PROG_TYPE_SCHED_CLS:2699 case BPF_PROG_TYPE_SCHED_CLS:
@@ -2970,10 +3024,37 @@ static int check_max_stack_depth(struct bpf_verifier_env *env) · 324 unmodified lines
2970 int depth = 0, frame = 0, idx = 0, i = 0, subprog_end;3024 int depth = 0, frame = 0, idx = 0, i = 0, subprog_end;
2971 struct bpf_subprog_info *subprog = env->subprog_info;3025 struct bpf_subprog_info *subprog = env->subprog_info;
2972 struct bpf_insn *insn = env->prog->insnsi;3026 struct bpf_insn *insn = env->prog->insnsi;
3027 bool tail_call_reachable = false;
2973 int ret_insn[MAX_CALL_FRAMES];3028 int ret_insn[MAX_CALL_FRAMES];
2974 int ret_prog[MAX_CALL_FRAMES];3029 int ret_prog[MAX_CALL_FRAMES];
3030 int j;
2975 3031
2976process_func:3032process_func:
3033 /* protect against potential stack overflow that might happen when
3034 * bpf2bpf calls get combined with tailcalls. Limit the caller's stack
3035 * depth for such case down to 256 so that the worst case scenario
3036 * would result in 8k stack size (32 which is tailcall limit * 256 =
3037 * 8k).
3038 *
3039 * To get the idea what might happen, see an example:
3040 * func1 -> sub rsp, 128
3041 * subfunc1 -> sub rsp, 256
3042 * tailcall1 -> add rsp, 256
3043 * func2 -> sub rsp, 192 (total stack size = 128 + 192 = 320)
3044 * subfunc2 -> sub rsp, 64
3045 * subfunc22 -> sub rsp, 128
3046 * tailcall2 -> add rsp, 128
3047 * func3 -> sub rsp, 32 (total stack size 128 + 192 + 64 + 32 = 416)
3048 *
3049 * tailcall will unwind the current stack frame but it will not get rid
3050 * of caller's stack as shown on the example above.
3051 */
3052 if (idx && subprog[idx].has_tail_call && depth >= 256) {
3053 verbose(env,
3054 "tail_calls are not allowed when call stack of previous frames is %d bytes. Too large\n",
3055 depth);
3056 return -EACCES;
3057 }
2977 /* round up to 32-bytes, since this is granularity3058 /* round up to 32-bytes, since this is granularity
2978 * of interpreter stack size3059 * of interpreter stack size
2979 */3060 */
@@ -3002,6 +3083,10 @@ continue_func: · 22 unmodified lines
3002 i);3083 i);
3003 return -EFAULT;3084 return -EFAULT;
3004 }3085 }
3086
3087 if (subprog[idx].has_tail_call)
3088 tail_call_reachable = true;
3089
3005 frame++;3090 frame++;
3006 if (frame >= MAX_CALL_FRAMES) {3091 if (frame >= MAX_CALL_FRAMES) {
3007 verbose(env, "the call stack of %d frames is too deep !\n",3092 verbose(env, "the call stack of %d frames is too deep !\n",
@@ -3010,6 +3095,15 @@ continue_func: · 2 unmodified lines
3010 }3095 }
3011 goto process_func;3096 goto process_func;
3012 }3097 }
3098 /* if tail call got detected across bpf2bpf calls then mark each of the
3099 * currently present subprog frames as tail call reachable subprogs;
3100 * this info will be utilized by JIT so that we will be preserving the
3101 * tail call counter throughout bpf2bpf calls combined with tailcalls
3102 */
3103 if (tail_call_reachable)
3104 for (j = 0; j < frame; j++)
3105 subprog[ret_prog[j]].tail_call_reachable = true;
3106
3013 /* end of for() loop means the last insn of the 'subprog'3107 /* end of for() loop means the last insn of the 'subprog'
3014 * was reached. Doesn't matter whether it was JA or EXIT3108 * was reached. Doesn't matter whether it was JA or EXIT
3015 */3109 */
@@ -3585,18 +3679,6 @@ static int check_stack_boundary(struct bpf_verifier_env *env, int regno, · 569 unmodified lines
3585 struct bpf_func_state *state = func(env, reg);3679 struct bpf_func_state *state = func(env, reg);
3586 int err, min_off, max_off, i, j, slot, spi;3680 int err, min_off, max_off, i, j, slot, spi;
3587 3681
3588 if (reg->type != PTR_TO_STACK) {
3589 /* Allow zero-byte read from NULL, regardless of pointer type */
3590 if (zero_size_allowed && access_size == 0 &&
3591 register_is_null(reg))
3592 return 0;
3593
3594 verbose(env, "R%d type=%s expected=%s\n", regno,
3595 reg_type_str[reg->type],
3596 reg_type_str[PTR_TO_STACK]);
3597 return -EACCES;
3598 }
3599
3600 if (tnum_is_const(reg->var_off)) {3682 if (tnum_is_const(reg->var_off)) {
3601 min_off = max_off = reg->var_off.value + reg->off;3683 min_off = max_off = reg->var_off.value + reg->off;
3602 err = __check_stack_boundary(env, regno, min_off, access_size,3684 err = __check_stack_boundary(env, regno, min_off, access_size,
@@ -3741,9 +3823,19 @@ static int check_helper_mem_access(struct bpf_verifier_env *env, int regno, · 138 unmodified lines
3741 access_size, zero_size_allowed,3823 access_size, zero_size_allowed,
3742 "rdwr",3824 "rdwr",
3743 &env->prog->aux->max_rdwr_access);3825 &env->prog->aux->max_rdwr_access);
3744 default: /* scalar_value|ptr_to_stack or invalid ptr */3826 case PTR_TO_STACK:
3745 return check_stack_boundary(env, regno, access_size,3827 return check_stack_boundary(env, regno, access_size,
3746 zero_size_allowed, meta);3828 zero_size_allowed, meta);
3829 default: /* scalar_value or invalid ptr */
3830 /* Allow zero-byte read from NULL, regardless of pointer type */
3831 if (zero_size_allowed && access_size == 0 &&
3832 register_is_null(reg))
3833 return 0;
3834
3835 verbose(env, "R%d type=%s expected=%s\n", regno,
3836 reg_type_str[reg->type],
3837 reg_type_str[PTR_TO_STACK]);
3838 return -EACCES;
3747 }3839 }
3748}3840}
3749 3841
@@ -3775,10 +3867,6 @@ static int process_spin_lock(struct bpf_verifier_env *env, int regno, · 25 unmodified lines
3775 struct bpf_map *map = reg->map_ptr;3867 struct bpf_map *map = reg->map_ptr;
3776 u64 val = reg->var_off.value;3868 u64 val = reg->var_off.value;
3777 3869
3778 if (reg->type != PTR_TO_MAP_VALUE) {
3779 verbose(env, "R%d is not a pointer to map_value\n", regno);
3780 return -EINVAL;
3781 }
3782 if (!is_const) {3870 if (!is_const) {
3783 verbose(env,3871 verbose(env,
3784 "R%d doesn't have constant offset. bpf_spin_lock has to be at the constant offset\n",3872 "R%d doesn't have constant offset. bpf_spin_lock has to be at the constant offset\n",
@@ -3845,12 +3933,6 @@ static bool arg_type_is_mem_size(enum bpf_arg_type type) · 60 unmodified lines
3845 type == ARG_CONST_SIZE_OR_ZERO;3933 type == ARG_CONST_SIZE_OR_ZERO;
3846}3934}
3847 3935
3848static bool arg_type_is_alloc_mem_ptr(enum bpf_arg_type type)
3849{
3850 return type == ARG_PTR_TO_ALLOC_MEM ||
3851 type == ARG_PTR_TO_ALLOC_MEM_OR_NULL;
3852}
3853
3854static bool arg_type_is_alloc_size(enum bpf_arg_type type)3936static bool arg_type_is_alloc_size(enum bpf_arg_type type)
3855{3937{
3856 return type == ARG_CONST_ALLOC_SIZE_OR_ZERO;3938 return type == ARG_CONST_ALLOC_SIZE_OR_ZERO;
@@ -3872,14 +3954,194 @@ static int int_ptr_type_to_size(enum bpf_arg_type type) · 15 unmodified lines
3872 return -EINVAL;3954 return -EINVAL;
3873}3955}
3874 3956
3957static int resolve_map_arg_type(struct bpf_verifier_env *env,
3958 const struct bpf_call_arg_meta *meta,
3959 enum bpf_arg_type *arg_type)
3960{
3961 if (!meta->map_ptr) {
3962 /* kernel subsystem misconfigured verifier */
3963 verbose(env, "invalid map_ptr to access map->type\n");
3964 return -EACCES;
3965 }
3966
3967 switch (meta->map_ptr->map_type) {
3968 case BPF_MAP_TYPE_SOCKMAP:
3969 case BPF_MAP_TYPE_SOCKHASH:
3970 if (*arg_type == ARG_PTR_TO_MAP_VALUE) {
3971 *arg_type = ARG_PTR_TO_BTF_ID_SOCK_COMMON;
3972 } else {
3973 verbose(env, "invalid arg_type for sockmap/sockhash\n");
3974 return -EINVAL;
3975 }
3976 break;
3977
3978 default:
3979 break;
3980 }
3981 return 0;
3982}
3983
3984struct bpf_reg_types {
3985 const enum bpf_reg_type types[10];
3986 u32 *btf_id;
3987};
3988
3989static const struct bpf_reg_types map_key_value_types = {
3990 .types = {
3991 PTR_TO_STACK,
3992 PTR_TO_PACKET,
3993 PTR_TO_PACKET_META,
3994 PTR_TO_MAP_VALUE,
3995 },
3996};
3997
3998static const struct bpf_reg_types sock_types = {
3999 .types = {
4000 PTR_TO_SOCK_COMMON,
4001 PTR_TO_SOCKET,
4002 PTR_TO_TCP_SOCK,
4003 PTR_TO_XDP_SOCK,
4004 },
4005};
4006
4007#ifdef CONFIG_NET
4008static const struct bpf_reg_types btf_id_sock_common_types = {
4009 .types = {
4010 PTR_TO_SOCK_COMMON,
4011 PTR_TO_SOCKET,
4012 PTR_TO_TCP_SOCK,
4013 PTR_TO_XDP_SOCK,
4014 PTR_TO_BTF_ID,
4015 },
4016 .btf_id = &btf_sock_ids[BTF_SOCK_TYPE_SOCK_COMMON],
4017};
4018#endif
4019
4020static const struct bpf_reg_types mem_types = {
4021 .types = {
4022 PTR_TO_STACK,
4023 PTR_TO_PACKET,
4024 PTR_TO_PACKET_META,
4025 PTR_TO_MAP_VALUE,
4026 PTR_TO_MEM,
4027 PTR_TO_RDONLY_BUF,
4028 PTR_TO_RDWR_BUF,
4029 },
4030};
4031
4032static const struct bpf_reg_types int_ptr_types = {
4033 .types = {
4034 PTR_TO_STACK,
4035 PTR_TO_PACKET,
4036 PTR_TO_PACKET_META,
4037 PTR_TO_MAP_VALUE,
4038 },
4039};
4040
4041static const struct bpf_reg_types fullsock_types = { .types = { PTR_TO_SOCKET } };
4042static const struct bpf_reg_types scalar_types = { .types = { SCALAR_VALUE } };
4043static const struct bpf_reg_types context_types = { .types = { PTR_TO_CTX } };
4044static const struct bpf_reg_types alloc_mem_types = { .types = { PTR_TO_MEM } };
4045static const struct bpf_reg_types const_map_ptr_types = { .types = { CONST_PTR_TO_MAP } };
4046static const struct bpf_reg_types btf_ptr_types = { .types = { PTR_TO_BTF_ID } };
4047static const struct bpf_reg_types spin_lock_types = { .types = { PTR_TO_MAP_VALUE } };
4048static const struct bpf_reg_types percpu_btf_ptr_types = { .types = { PTR_TO_PERCPU_BTF_ID } };
4049
4050static const struct bpf_reg_types *compatible_reg_types[__BPF_ARG_TYPE_MAX] = {
4051 [ARG_PTR_TO_MAP_KEY] = &map_key_value_types,
4052 [ARG_PTR_TO_MAP_VALUE] = &map_key_value_types,
4053 [ARG_PTR_TO_UNINIT_MAP_VALUE] = &map_key_value_types,
4054 [ARG_PTR_TO_MAP_VALUE_OR_NULL] = &map_key_value_types,
4055 [ARG_CONST_SIZE] = &scalar_types,
4056 [ARG_CONST_SIZE_OR_ZERO] = &scalar_types,
4057 [ARG_CONST_ALLOC_SIZE_OR_ZERO] = &scalar_types,
4058 [ARG_CONST_MAP_PTR] = &const_map_ptr_types,
4059 [ARG_PTR_TO_CTX] = &context_types,
4060 [ARG_PTR_TO_CTX_OR_NULL] = &context_types,
4061 [ARG_PTR_TO_SOCK_COMMON] = &sock_types,
4062#ifdef CONFIG_NET
4063 [ARG_PTR_TO_BTF_ID_SOCK_COMMON] = &btf_id_sock_common_types,
4064#endif
4065 [ARG_PTR_TO_SOCKET] = &fullsock_types,
4066 [ARG_PTR_TO_SOCKET_OR_NULL] = &fullsock_types,
4067 [ARG_PTR_TO_BTF_ID] = &btf_ptr_types,
4068 [ARG_PTR_TO_SPIN_LOCK] = &spin_lock_types,
4069 [ARG_PTR_TO_MEM] = &mem_types,
4070 [ARG_PTR_TO_MEM_OR_NULL] = &mem_types,
4071 [ARG_PTR_TO_UNINIT_MEM] = &mem_types,
4072 [ARG_PTR_TO_ALLOC_MEM] = &alloc_mem_types,
4073 [ARG_PTR_TO_ALLOC_MEM_OR_NULL] = &alloc_mem_types,
4074 [ARG_PTR_TO_INT] = &int_ptr_types,
4075 [ARG_PTR_TO_LONG] = &int_ptr_types,
4076 [ARG_PTR_TO_PERCPU_BTF_ID] = &percpu_btf_ptr_types,
4077};
4078
4079static int check_reg_type(struct bpf_verifier_env *env, u32 regno,
4080 enum bpf_arg_type arg_type,
4081 const u32 *arg_btf_id)
4082{
4083 struct bpf_reg_state *regs = cur_regs(env), *reg = &regs[regno];
4084 enum bpf_reg_type expected, type = reg->type;
4085 const struct bpf_reg_types *compatible;
4086 int i, j;
4087
4088 compatible = compatible_reg_types[arg_type];
4089 if (!compatible) {
4090 verbose(env, "verifier internal error: unsupported arg type %d\n", arg_type);
4091 return -EFAULT;
4092 }
4093
4094 for (i = 0; i < ARRAY_SIZE(compatible->types); i++) {
4095 expected = compatible->types[i];
4096 if (expected == NOT_INIT)
4097 break;
4098
4099 if (type == expected)
4100 goto found;
4101 }
4102
4103 verbose(env, "R%d type=%s expected=", regno, reg_type_str[type]);
4104 for (j = 0; j + 1 < i; j++)
4105 verbose(env, "%s, ", reg_type_str[compatible->types[j]]);
4106 verbose(env, "%s\n", reg_type_str[compatible->types[j]]);
4107 return -EACCES;
4108
4109found:
4110 if (type == PTR_TO_BTF_ID) {
4111 if (!arg_btf_id) {
4112 if (!compatible->btf_id) {
4113 verbose(env, "verifier internal error: missing arg compatible BTF ID\n");
4114 return -EFAULT;
4115 }
4116 arg_btf_id = compatible->btf_id;
4117 }
4118
4119 if (!btf_struct_ids_match(&env->log, reg->off, reg->btf_id,
4120 *arg_btf_id)) {
4121 verbose(env, "R%d is of type %s but %s is expected\n",
4122 regno, kernel_type_name(reg->btf_id),
4123 kernel_type_name(*arg_btf_id));
4124 return -EACCES;
4125 }
4126
4127 if (!tnum_is_const(reg->var_off) || reg->var_off.value) {
4128 verbose(env, "R%d is a pointer to in-kernel struct with non-zero offset\n",
4129 regno);
4130 return -EACCES;
4131 }
4132 }
4133
4134 return 0;
4135}
4136
3875static int check_func_arg(struct bpf_verifier_env *env, u32 arg,4137static int check_func_arg(struct bpf_verifier_env *env, u32 arg,
3876 struct bpf_call_arg_meta *meta,4138 struct bpf_call_arg_meta *meta,
3877 const struct bpf_func_proto *fn)4139 const struct bpf_func_proto *fn)
3878{4140{
3879 u32 regno = BPF_REG_1 + arg;4141 u32 regno = BPF_REG_1 + arg;
3880 struct bpf_reg_state *regs = cur_regs(env), *reg = &regs[regno];4142 struct bpf_reg_state *regs = cur_regs(env), *reg = &regs[regno];
3881 enum bpf_reg_type expected_type, type = reg->type;
3882 enum bpf_arg_type arg_type = fn->arg_type[arg];4143 enum bpf_arg_type arg_type = fn->arg_type[arg];
4144 enum bpf_reg_type type = reg->type;
3883 int err = 0;4145 int err = 0;
3884 4146
3885 if (arg_type == ARG_DONTCARE)4147 if (arg_type == ARG_DONTCARE)
@@ -3904,120 +4166,32 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, · 18 unmodified lines
3904 return -EACCES;4166 return -EACCES;
3905 }4167 }
3906 4168
3907 if (arg_type == ARG_PTR_TO_MAP_KEY ||4169 if (arg_type == ARG_PTR_TO_MAP_VALUE ||
3908 arg_type == ARG_PTR_TO_MAP_VALUE ||
3909 arg_type == ARG_PTR_TO_UNINIT_MAP_VALUE ||4170 arg_type == ARG_PTR_TO_UNINIT_MAP_VALUE ||
3910 arg_type == ARG_PTR_TO_MAP_VALUE_OR_NULL) {4171 arg_type == ARG_PTR_TO_MAP_VALUE_OR_NULL) {
3911 expected_type = PTR_TO_STACK;4172 err = resolve_map_arg_type(env, meta, &arg_type);
3912 if (register_is_null(reg) &&4173 if (err)
3913 arg_type == ARG_PTR_TO_MAP_VALUE_OR_NULL)4174 return err;
3914 /* final test in check_stack_boundary() */;4175 }
3915 else if (!type_is_pkt_pointer(type) &&
3916 type != PTR_TO_MAP_VALUE &&
3917 type != expected_type)
3918 goto err_type;
3919 } else if (arg_type == ARG_CONST_SIZE ||
3920 arg_type == ARG_CONST_SIZE_OR_ZERO ||
3921 arg_type == ARG_CONST_ALLOC_SIZE_OR_ZERO) {
3922 expected_type = SCALAR_VALUE;
3923 if (type != expected_type)
3924 goto err_type;
3925 } else if (arg_type == ARG_CONST_MAP_PTR) {
3926 expected_type = CONST_PTR_TO_MAP;
3927 if (type != expected_type)
3928 goto err_type;
3929 } else if (arg_type == ARG_PTR_TO_CTX ||
3930 arg_type == ARG_PTR_TO_CTX_OR_NULL) {
3931 expected_type = PTR_TO_CTX;
3932 if (!(register_is_null(reg) &&
3933 arg_type == ARG_PTR_TO_CTX_OR_NULL)) {
3934 if (type != expected_type)
3935 goto err_type;
3936 err = check_ctx_reg(env, reg, regno);
3937 if (err < 0)
3938 return err;
3939 }
3940 } else if (arg_type == ARG_PTR_TO_SOCK_COMMON) {
3941 expected_type = PTR_TO_SOCK_COMMON;
3942 /* Any sk pointer can be ARG_PTR_TO_SOCK_COMMON */
3943 if (!type_is_sk_pointer(type))
3944 goto err_type;
3945 if (reg->ref_obj_id) {
3946 if (meta->ref_obj_id) {
3947 verbose(env, "verifier internal error: more than one arg with ref_obj_id R%d %u %u\n",
3948 regno, reg->ref_obj_id,
3949 meta->ref_obj_id);
3950 return -EFAULT;
3951 }
3952 meta->ref_obj_id = reg->ref_obj_id;
3953 }
3954 } else if (arg_type == ARG_PTR_TO_SOCKET ||
3955 arg_type == ARG_PTR_TO_SOCKET_OR_NULL) {
3956 expected_type = PTR_TO_SOCKET;
3957 if (!(register_is_null(reg) &&
3958 arg_type == ARG_PTR_TO_SOCKET_OR_NULL)) {
3959 if (type != expected_type)
3960 goto err_type;
3961 }
3962 } else if (arg_type == ARG_PTR_TO_BTF_ID) {
3963 expected_type = PTR_TO_BTF_ID;
3964 if (type != expected_type)
3965 goto err_type;
3966 if (!fn->check_btf_id) {
3967 if (reg->btf_id != meta->btf_id) {
3968 verbose(env, "Helper has type %s got %s in R%d\n",
3969 kernel_type_name(meta->btf_id),
3970 kernel_type_name(reg->btf_id), regno);
3971
3972 return -EACCES;
3973 }
3974 } else if (!fn->check_btf_id(reg->btf_id, arg)) {
3975 verbose(env, "Helper does not support %s in R%d\n",
3976 kernel_type_name(reg->btf_id), regno);
3977 4176
3978 return -EACCES;4177 if (register_is_null(reg) && arg_type_may_be_null(arg_type))
3979 }4178 /* A NULL register has a SCALAR_VALUE type, so skip
3980 if (!tnum_is_const(reg->var_off) || reg->var_off.value || reg->off) {4179 * type checking.
3981 verbose(env, "R%d is a pointer to in-kernel struct with non-zero offset\n",
3982 regno);
3983 return -EACCES;
3984 }
3985 } else if (arg_type == ARG_PTR_TO_SPIN_LOCK) {
3986 if (meta->func_id == BPF_FUNC_spin_lock) {
3987 if (process_spin_lock(env, regno, true))
3988 return -EACCES;
3989 } else if (meta->func_id == BPF_FUNC_spin_unlock) {
3990 if (process_spin_lock(env, regno, false))
3991 return -EACCES;
3992 } else {
3993 verbose(env, "verifier internal error\n");
3994 return -EFAULT;
3995 }
3996 } else if (arg_type_is_mem_ptr(arg_type)) {
3997 expected_type = PTR_TO_STACK;
3998 /* One exception here. In case function allows for NULL to be
3999 * passed in as argument, it's a SCALAR_VALUE type. Final test
4000 * happens during stack boundary checking.
4001 */4180 */
4002 if (register_is_null(reg) &&4181 goto skip_type_check;
4003 (arg_type == ARG_PTR_TO_MEM_OR_NULL ||4182
4004 arg_type == ARG_PTR_TO_ALLOC_MEM_OR_NULL))4183 err = check_reg_type(env, regno, arg_type, fn->arg_btf_id[arg]);
4005 /* final test in check_stack_boundary() */;4184 if (err)
4006 else if (!type_is_pkt_pointer(type) &&4185 return err;
4007 type != PTR_TO_MAP_VALUE &&4186
4008 type != PTR_TO_MEM &&4187 if (type == PTR_TO_CTX) {
4009 type != PTR_TO_RDONLY_BUF &&4188 err = check_ctx_reg(env, reg, regno);
4010 type != PTR_TO_RDWR_BUF &&4189 if (err < 0)
4011 type != expected_type)4190 return err;
4012 goto err_type;4191 }
4013 meta->raw_mode = arg_type == ARG_PTR_TO_UNINIT_MEM;4192
4014 } else if (arg_type_is_alloc_mem_ptr(arg_type)) {4193skip_type_check:
4015 expected_type = PTR_TO_MEM;4194 if (reg->ref_obj_id) {
4016 if (register_is_null(reg) &&
4017 arg_type == ARG_PTR_TO_ALLOC_MEM_OR_NULL)
4018 /* final test in check_stack_boundary() */;
4019 else if (type != expected_type)
4020 goto err_type;
4021 if (meta->ref_obj_id) {4195 if (meta->ref_obj_id) {
4022 verbose(env, "verifier internal error: more than one arg with ref_obj_id R%d %u %u\n",4196 verbose(env, "verifier internal error: more than one arg with ref_obj_id R%d %u %u\n",
4023 regno, reg->ref_obj_id,4197 regno, reg->ref_obj_id,
@@ -4025,15 +4199,6 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, · 1 unmodified line
4025 return -EFAULT;4199 return -EFAULT;
4026 }4200 }
4027 meta->ref_obj_id = reg->ref_obj_id;4201 meta->ref_obj_id = reg->ref_obj_id;
4028 } else if (arg_type_is_int_ptr(arg_type)) {
4029 expected_type = PTR_TO_STACK;
4030 if (!type_is_pkt_pointer(type) &&
4031 type != PTR_TO_MAP_VALUE &&
4032 type != expected_type)
4033 goto err_type;
4034 } else {
4035 verbose(env, "unsupported arg_type %d\n", arg_type);
4036 return -EFAULT;
4037 }4202 }
4038 4203
4039 if (arg_type == ARG_CONST_MAP_PTR) {4204 if (arg_type == ARG_CONST_MAP_PTR) {
@@ -4072,6 +4237,28 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, · 32 unmodified lines
4072 err = check_helper_mem_access(env, regno,4237 err = check_helper_mem_access(env, regno,
4073 meta->map_ptr->value_size, false,4238 meta->map_ptr->value_size, false,
4074 meta);4239 meta);
4240 } else if (arg_type == ARG_PTR_TO_PERCPU_BTF_ID) {
4241 if (!reg->btf_id) {
4242 verbose(env, "Helper has invalid btf_id in R%d\n", regno);
4243 return -EACCES;
4244 }
4245 meta->ret_btf_id = reg->btf_id;
4246 } else if (arg_type == ARG_PTR_TO_SPIN_LOCK) {
4247 if (meta->func_id == BPF_FUNC_spin_lock) {
4248 if (process_spin_lock(env, regno, true))
4249 return -EACCES;
4250 } else if (meta->func_id == BPF_FUNC_spin_unlock) {
4251 if (process_spin_lock(env, regno, false))
4252 return -EACCES;
4253 } else {
4254 verbose(env, "verifier internal error\n");
4255 return -EFAULT;
4256 }
4257 } else if (arg_type_is_mem_ptr(arg_type)) {
4258 /* The access to this pointer is only checked when we hit the
4259 * next is_mem_size argument below.
4260 */
4261 meta->raw_mode = (arg_type == ARG_PTR_TO_UNINIT_MEM);
4075 } else if (arg_type_is_mem_size(arg_type)) {4262 } else if (arg_type_is_mem_size(arg_type)) {
4076 bool zero_size_allowed = (arg_type == ARG_CONST_SIZE_OR_ZERO);4263 bool zero_size_allowed = (arg_type == ARG_CONST_SIZE_OR_ZERO);
4077 4264
@@ -4137,10 +4324,43 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, · 59 unmodified lines
4137 }4324 }
4138 4325
4139 return err;4326 return err;
4140err_type:4327}
4141 verbose(env, "R%d type=%s expected=%s\n", regno,4328
4142 reg_type_str[type], reg_type_str[expected_type]);4329static bool may_update_sockmap(struct bpf_verifier_env *env, int func_id)
4143 return -EACCES;4330{
4331 enum bpf_attach_type eatype = env->prog->expected_attach_type;
4332 enum bpf_prog_type type = resolve_prog_type(env->prog);
4333
4334 if (func_id != BPF_FUNC_map_update_elem)
4335 return false;
4336
4337 /* It's not possible to get access to a locked struct sock in these
4338 * contexts, so updating is safe.
4339 */
4340 switch (type) {
4341 case BPF_PROG_TYPE_TRACING:
4342 if (eatype == BPF_TRACE_ITER)
4343 return true;
4344 break;
4345 case BPF_PROG_TYPE_SOCKET_FILTER:
4346 case BPF_PROG_TYPE_SCHED_CLS:
4347 case BPF_PROG_TYPE_SCHED_ACT:
4348 case BPF_PROG_TYPE_XDP:
4349 case BPF_PROG_TYPE_SK_REUSEPORT:
4350 case BPF_PROG_TYPE_FLOW_DISSECTOR:
4351 case BPF_PROG_TYPE_SK_LOOKUP:
4352 return true;
4353 default:
4354 break;
4355 }
4356
4357 verbose(env, "cannot update sockmap in this context\n");
4358 return false;
4359}
4360
4361static bool allow_tail_call_in_subprogs(struct bpf_verifier_env *env)
4362{
4363 return env->prog->jit_requested && IS_ENABLED(CONFIG_X86_64);
4144}4364}
4145 4365
4146static int check_map_func_compatibility(struct bpf_verifier_env *env,4366static int check_map_func_compatibility(struct bpf_verifier_env *env,
@@ -4214,7 +4434,8 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, · 67 unmodified lines
4214 func_id != BPF_FUNC_map_delete_elem &&4434 func_id != BPF_FUNC_map_delete_elem &&
4215 func_id != BPF_FUNC_msg_redirect_map &&4435 func_id != BPF_FUNC_msg_redirect_map &&
4216 func_id != BPF_FUNC_sk_select_reuseport &&4436 func_id != BPF_FUNC_sk_select_reuseport &&
4217 func_id != BPF_FUNC_map_lookup_elem)4437 func_id != BPF_FUNC_map_lookup_elem &&
4438 !may_update_sockmap(env, func_id))
4218 goto error;4439 goto error;
4219 break;4440 break;
4220 case BPF_MAP_TYPE_SOCKHASH:4441 case BPF_MAP_TYPE_SOCKHASH:
@@ -4223,7 +4444,8 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, · 2 unmodified lines
4223 func_id != BPF_FUNC_map_delete_elem &&4444 func_id != BPF_FUNC_map_delete_elem &&
4224 func_id != BPF_FUNC_msg_redirect_hash &&4445 func_id != BPF_FUNC_msg_redirect_hash &&
4225 func_id != BPF_FUNC_sk_select_reuseport &&4446 func_id != BPF_FUNC_sk_select_reuseport &&
4226 func_id != BPF_FUNC_map_lookup_elem)4447 func_id != BPF_FUNC_map_lookup_elem &&
4448 !may_update_sockmap(env, func_id))
4227 goto error;4449 goto error;
4228 break;4450 break;
4229 case BPF_MAP_TYPE_REUSEPORT_SOCKARRAY:4451 case BPF_MAP_TYPE_REUSEPORT_SOCKARRAY:
@@ -4242,6 +4464,11 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, · 12 unmodified lines
4242 func_id != BPF_FUNC_sk_storage_delete)4464 func_id != BPF_FUNC_sk_storage_delete)
4243 goto error;4465 goto error;
4244 break;4466 break;
4467 case BPF_MAP_TYPE_INODE_STORAGE:
4468 if (func_id != BPF_FUNC_inode_storage_get &&
4469 func_id != BPF_FUNC_inode_storage_delete)
4470 goto error;
4471 break;
4245 default:4472 default:
4246 break;4473 break;
4247 }4474 }
@@ -4251,8 +4478,8 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, · 3 unmodified lines
4251 case BPF_FUNC_tail_call:4478 case BPF_FUNC_tail_call:
4252 if (map->map_type != BPF_MAP_TYPE_PROG_ARRAY)4479 if (map->map_type != BPF_MAP_TYPE_PROG_ARRAY)
4253 goto error;4480 goto error;
4254 if (env->subprog_cnt > 1) {4481 if (env->subprog_cnt > 1 && !allow_tail_call_in_subprogs(env)) {
4255 verbose(env, "tail_calls are not allowed in programs with bpf-to-bpf calls\n");4482 verbose(env, "tail_calls are not allowed in non-JITed programs with bpf-to-bpf calls\n");
4256 return -EINVAL;4483 return -EINVAL;
4257 }4484 }
4258 break;4485 break;
@@ -4315,6 +4542,11 @@ static int check_map_func_compatibility(struct bpf_verifier_env *env, · 56 unmodified lines
4315 if (map->map_type != BPF_MAP_TYPE_SK_STORAGE)4542 if (map->map_type != BPF_MAP_TYPE_SK_STORAGE)
4316 goto error;4543 goto error;
4317 break;4544 break;
4545 case BPF_FUNC_inode_storage_get:
4546 case BPF_FUNC_inode_storage_delete:
4547 if (map->map_type != BPF_MAP_TYPE_INODE_STORAGE)
4548 goto error;
4549 break;
4318 default:4550 default:
4319 break;4551 break;
4320 }4552 }
@@ -4402,10 +4634,26 @@ static bool check_refcount_ok(const struct bpf_func_proto *fn, int func_id) · 81 unmodified lines
4402 return count <= 1;4634 return count <= 1;
4403}4635}
4404 4636
4637static bool check_btf_id_ok(const struct bpf_func_proto *fn)
4638{
4639 int i;
4640
4641 for (i = 0; i < ARRAY_SIZE(fn->arg_type); i++) {
4642 if (fn->arg_type[i] == ARG_PTR_TO_BTF_ID && !fn->arg_btf_id[i])
4643 return false;
4644
4645 if (fn->arg_type[i] != ARG_PTR_TO_BTF_ID && fn->arg_btf_id[i])
4646 return false;
4647 }
4648
4649 return true;
4650}
4651
4405static int check_func_proto(const struct bpf_func_proto *fn, int func_id)4652static int check_func_proto(const struct bpf_func_proto *fn, int func_id)
4406{4653{
4407 return check_raw_mode_ok(fn) &&4654 return check_raw_mode_ok(fn) &&
4408 check_arg_pair_ok(fn) &&4655 check_arg_pair_ok(fn) &&
4656 check_btf_id_ok(fn) &&
4409 check_refcount_ok(fn, func_id) ? 0 : -EINVAL;4657 check_refcount_ok(fn, func_id) ? 0 : -EINVAL;
4410}4658}
4411 4659
@@ -4775,6 +5023,11 @@ static int check_helper_call(struct bpf_verifier_env *env, int func_id, int insn · 363 unmodified lines
4775 return -EINVAL;5023 return -EINVAL;
4776 }5024 }
4777 5025
5026 if (fn->allowed && !fn->allowed(env->prog)) {
5027 verbose(env, "helper call is not allowed in probe\n");
5028 return -EINVAL;
5029 }
5030
4778 /* With LD_ABS/IND some JITs save/restore skb from r1. */5031 /* With LD_ABS/IND some JITs save/restore skb from r1. */
4779 changes_data = bpf_helper_changes_pkt_data(fn->func);5032 changes_data = bpf_helper_changes_pkt_data(fn->func);
4780 if (changes_data && fn->arg1_type != ARG_PTR_TO_CTX) {5033 if (changes_data && fn->arg1_type != ARG_PTR_TO_CTX) {
@@ -4796,11 +5049,6 @@ static int check_helper_call(struct bpf_verifier_env *env, int func_id, int insn · 15 unmodified lines
4796 meta.func_id = func_id;5049 meta.func_id = func_id;
4797 /* check args */5050 /* check args */
4798 for (i = 0; i < 5; i++) {5051 for (i = 0; i < 5; i++) {
4799 if (!fn->check_btf_id) {
4800 err = btf_resolve_helper_id(&env->log, fn, i);
4801 if (err > 0)
4802 meta.btf_id = err;
4803 }
4804 err = check_func_arg(env, i, &meta, fn);5052 err = check_func_arg(env, i, &meta, fn);
4805 if (err)5053 if (err)
4806 return err;5054 return err;
@@ -4904,6 +5152,35 @@ static int check_helper_call(struct bpf_verifier_env *env, int func_id, int insn · 97 unmodified lines
4904 regs[BPF_REG_0].type = PTR_TO_MEM_OR_NULL;5152 regs[BPF_REG_0].type = PTR_TO_MEM_OR_NULL;
4905 regs[BPF_REG_0].id = ++env->id_gen;5153 regs[BPF_REG_0].id = ++env->id_gen;
4906 regs[BPF_REG_0].mem_size = meta.mem_size;5154 regs[BPF_REG_0].mem_size = meta.mem_size;
5155 } else if (fn->ret_type == RET_PTR_TO_MEM_OR_BTF_ID_OR_NULL ||
5156 fn->ret_type == RET_PTR_TO_MEM_OR_BTF_ID) {
5157 const struct btf_type *t;
5158
5159 mark_reg_known_zero(env, regs, BPF_REG_0);
5160 t = btf_type_skip_modifiers(btf_vmlinux, meta.ret_btf_id, NULL);
5161 if (!btf_type_is_struct(t)) {
5162 u32 tsize;
5163 const struct btf_type *ret;
5164 const char *tname;
5165
5166 /* resolve the type size of ksym. */
5167 ret = btf_resolve_size(btf_vmlinux, t, &tsize);
5168 if (IS_ERR(ret)) {
5169 tname = btf_name_by_offset(btf_vmlinux, t->name_off);
5170 verbose(env, "unable to resolve the size of type '%s': %ld\n",
5171 tname, PTR_ERR(ret));
5172 return -EINVAL;
5173 }
5174 regs[BPF_REG_0].type =
5175 fn->ret_type == RET_PTR_TO_MEM_OR_BTF_ID ?
5176 PTR_TO_MEM : PTR_TO_MEM_OR_NULL;
5177 regs[BPF_REG_0].mem_size = tsize;
5178 } else {
5179 regs[BPF_REG_0].type =
5180 fn->ret_type == RET_PTR_TO_MEM_OR_BTF_ID ?
5181 PTR_TO_BTF_ID : PTR_TO_BTF_ID_OR_NULL;
5182 regs[BPF_REG_0].btf_id = meta.ret_btf_id;
5183 }
4907 } else if (fn->ret_type == RET_PTR_TO_BTF_ID_OR_NULL) {5184 } else if (fn->ret_type == RET_PTR_TO_BTF_ID_OR_NULL) {
4908 int ret_btf_id;5185 int ret_btf_id;
4909 5186
@@ -5219,6 +5496,10 @@ static int adjust_ptr_min_max_vals(struct bpf_verifier_env *env, · 309 unmodified lines
5219 dst, reg_type_str[ptr_reg->type]);5496 dst, reg_type_str[ptr_reg->type]);
5220 return -EACCES;5497 return -EACCES;
5221 case CONST_PTR_TO_MAP:5498 case CONST_PTR_TO_MAP:
5499 /* smin_val represents the known value */
5500 if (known && smin_val == 0 && opcode == BPF_ADD)
5501 break;
5502 fallthrough;
5222 case PTR_TO_PACKET_END:5503 case PTR_TO_PACKET_END:
5223 case PTR_TO_SOCKET:5504 case PTR_TO_SOCKET:
5224 case PTR_TO_SOCKET_OR_NULL:5505 case PTR_TO_SOCKET_OR_NULL:
@@ -5634,8 +5915,7 @@ static void scalar_min_max_and(struct bpf_reg_state *dst_reg, · 409 unmodified lines
5634 u64 umax_val = src_reg->umax_value;5915 u64 umax_val = src_reg->umax_value;
5635 5916
5636 if (src_known && dst_known) {5917 if (src_known && dst_known) {
5637 __mark_reg_known(dst_reg, dst_reg->var_off.value &5918 __mark_reg_known(dst_reg, dst_reg->var_off.value);
5638 src_reg->var_off.value);
5639 return;5919 return;
5640 }5920 }
5641 5921
@@ -5705,8 +5985,7 @@ static void scalar_min_max_or(struct bpf_reg_state *dst_reg, · 63 unmodified lines
5705 u64 umin_val = src_reg->umin_value;5985 u64 umin_val = src_reg->umin_value;
5706 5986
5707 if (src_known && dst_known) {5987 if (src_known && dst_known) {
5708 __mark_reg_known(dst_reg, dst_reg->var_off.value |5988 __mark_reg_known(dst_reg, dst_reg->var_off.value);
5709 src_reg->var_off.value);
5710 return;5989 return;
5711 }5990 }
5712 5991
@@ -5732,11 +6011,72 @@ static void scalar_min_max_or(struct bpf_reg_state *dst_reg, · 19 unmodified lines
5732 __update_reg_bounds(dst_reg);6011 __update_reg_bounds(dst_reg);
5733}6012}
5734 6013
5735static void __scalar32_min_max_lsh(struct bpf_reg_state *dst_reg,6014static void scalar32_min_max_xor(struct bpf_reg_state *dst_reg,
5736 u64 umin_val, u64 umax_val)6015 struct bpf_reg_state *src_reg)
5737{6016{
5738 /* We lose all sign bit information (except what we can pick6017 bool src_known = tnum_subreg_is_const(src_reg->var_off);
5739 * up from var_off)6018 bool dst_known = tnum_subreg_is_const(dst_reg->var_off);
6019 struct tnum var32_off = tnum_subreg(dst_reg->var_off);
6020 s32 smin_val = src_reg->s32_min_value;
6021
6022 /* Assuming scalar64_min_max_xor will be called so it is safe
6023 * to skip updating register for known case.
6024 */
6025 if (src_known && dst_known)
6026 return;
6027
6028 /* We get both minimum and maximum from the var32_off. */
6029 dst_reg->u32_min_value = var32_off.value;
6030 dst_reg->u32_max_value = var32_off.value | var32_off.mask;
6031
6032 if (dst_reg->s32_min_value >= 0 && smin_val >= 0) {
6033 /* XORing two positive sign numbers gives a positive,
6034 * so safe to cast u32 result into s32.
6035 */
6036 dst_reg->s32_min_value = dst_reg->u32_min_value;
6037 dst_reg->s32_max_value = dst_reg->u32_max_value;
6038 } else {
6039 dst_reg->s32_min_value = S32_MIN;
6040 dst_reg->s32_max_value = S32_MAX;
6041 }
6042}
6043
6044static void scalar_min_max_xor(struct bpf_reg_state *dst_reg,
6045 struct bpf_reg_state *src_reg)
6046{
6047 bool src_known = tnum_is_const(src_reg->var_off);
6048 bool dst_known = tnum_is_const(dst_reg->var_off);
6049 s64 smin_val = src_reg->smin_value;
6050
6051 if (src_known && dst_known) {
6052 /* dst_reg->var_off.value has been updated earlier */
6053 __mark_reg_known(dst_reg, dst_reg->var_off.value);
6054 return;
6055 }
6056
6057 /* We get both minimum and maximum from the var_off. */
6058 dst_reg->umin_value = dst_reg->var_off.value;
6059 dst_reg->umax_value = dst_reg->var_off.value | dst_reg->var_off.mask;
6060
6061 if (dst_reg->smin_value >= 0 && smin_val >= 0) {
6062 /* XORing two positive sign numbers gives a positive,
6063 * so safe to cast u64 result into s64.
6064 */
6065 dst_reg->smin_value = dst_reg->umin_value;
6066 dst_reg->smax_value = dst_reg->umax_value;
6067 } else {
6068 dst_reg->smin_value = S64_MIN;
6069 dst_reg->smax_value = S64_MAX;
6070 }
6071
6072 __update_reg_bounds(dst_reg);
6073}
6074
6075static void __scalar32_min_max_lsh(struct bpf_reg_state *dst_reg,
6076 u64 umin_val, u64 umax_val)
6077{
6078 /* We lose all sign bit information (except what we can pick
6079 * up from var_off)
5740 */6080 */
5741 dst_reg->s32_min_value = S32_MIN;6081 dst_reg->s32_min_value = S32_MIN;
5742 dst_reg->s32_max_value = S32_MAX;6082 dst_reg->s32_max_value = S32_MAX;
@@ -6040,6 +6380,11 @@ static int adjust_scalar_min_max_vals(struct bpf_verifier_env *env, · 297 unmodified lines
6040 scalar32_min_max_or(dst_reg, &src_reg);6380 scalar32_min_max_or(dst_reg, &src_reg);
6041 scalar_min_max_or(dst_reg, &src_reg);6381 scalar_min_max_or(dst_reg, &src_reg);
6042 break;6382 break;
6383 case BPF_XOR:
6384 dst_reg->var_off = tnum_xor(dst_reg->var_off, src_reg.var_off);
6385 scalar32_min_max_xor(dst_reg, &src_reg);
6386 scalar_min_max_xor(dst_reg, &src_reg);
6387 break;
6043 case BPF_LSH:6388 case BPF_LSH:
6044 if (umax_val >= insn_bitness) {6389 if (umax_val >= insn_bitness) {
6045 /* Shifts greater than 31 or 63 are undefined.6390 /* Shifts greater than 31 or 63 are undefined.
@@ -6111,6 +6456,11 @@ static int adjust_reg_min_max_vals(struct bpf_verifier_env *env, · 65 unmodified lines
6111 src_reg = NULL;6456 src_reg = NULL;
6112 if (dst_reg->type != SCALAR_VALUE)6457 if (dst_reg->type != SCALAR_VALUE)
6113 ptr_reg = dst_reg;6458 ptr_reg = dst_reg;
6459 else
6460 /* Make sure ID is cleared otherwise dst_reg min/max could be
6461 * incorrectly propagated into other registers by find_equal_scalars()
6462 */
6463 dst_reg->id = 0;
6114 if (BPF_SRC(insn->code) == BPF_X) {6464 if (BPF_SRC(insn->code) == BPF_X) {
6115 src_reg = &regs[insn->src_reg];6465 src_reg = &regs[insn->src_reg];
6116 if (src_reg->type != SCALAR_VALUE) {6466 if (src_reg->type != SCALAR_VALUE) {
@@ -6244,6 +6594,12 @@ static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn) · 127 unmodified lines
6244 /* case: R1 = R26594 /* case: R1 = R2
6245 * copy register state to dest reg6595 * copy register state to dest reg
6246 */6596 */
6597 if (src_reg->type == SCALAR_VALUE && !src_reg->id)
6598 /* Assign src and dst registers the same ID
6599 * that will be used by find_equal_scalars()
6600 * to propagate min/max range.
6601 */
6602 src_reg->id = ++env->id_gen;
6247 *dst_reg = *src_reg;6603 *dst_reg = *src_reg;
6248 dst_reg->live |= REG_LIVE_WRITTEN;6604 dst_reg->live |= REG_LIVE_WRITTEN;
6249 dst_reg->subreg_def = DEF_NOT_SUBREG;6605 dst_reg->subreg_def = DEF_NOT_SUBREG;
@@ -6256,6 +6612,11 @@ static int check_alu_op(struct bpf_verifier_env *env, struct bpf_insn *insn) · 6 unmodified lines
6256 return -EACCES;6612 return -EACCES;
6257 } else if (src_reg->type == SCALAR_VALUE) {6613 } else if (src_reg->type == SCALAR_VALUE) {
6258 *dst_reg = *src_reg;6614 *dst_reg = *src_reg;
6615 /* Make sure ID is cleared otherwise
6616 * dst_reg min/max could be incorrectly
6617 * propagated into src_reg by find_equal_scalars()
6618 */
6619 dst_reg->id = 0;
6259 dst_reg->live |= REG_LIVE_WRITTEN;6620 dst_reg->live |= REG_LIVE_WRITTEN;
6260 dst_reg->subreg_def = env->insn_idx + 1;6621 dst_reg->subreg_def = env->insn_idx + 1;
6261 } else {6622 } else {
@@ -6646,14 +7007,18 @@ static void reg_set_min_max(struct bpf_reg_state *true_reg, · 384 unmodified lines
6646 struct bpf_reg_state *reg =7007 struct bpf_reg_state *reg =
6647 opcode == BPF_JEQ ? true_reg : false_reg;7008 opcode == BPF_JEQ ? true_reg : false_reg;
6648 7009
6649 /* For BPF_JEQ, if this is false we know nothing Jon Snow, but7010 /* JEQ/JNE comparison doesn't change the register equivalence.
6650 * if it is true we know the value for sure. Likewise for7011 * r1 = r2;
6651 * BPF_JNE.7012 * if (r1 == 42) goto label;
7013 * ...
7014 * label: // here both r1 and r2 are known to be 42.
7015 *
7016 * Hence when marking register as known preserve it's ID.
6652 */7017 */
6653 if (is_jmp32)7018 if (is_jmp32)
6654 __mark_reg32_known(reg, val32);7019 __mark_reg32_known(reg, val32);
6655 else7020 else
6656 __mark_reg_known(reg, val);7021 ___mark_reg_known(reg, val);
6657 break;7022 break;
6658 }7023 }
6659 case BPF_JSET:7024 case BPF_JSET:
@@ -7044,6 +7409,30 @@ static bool try_match_pkt_pointers(const struct bpf_insn *insn, · 384 unmodified lines
7044 return true;7409 return true;
7045}7410}
7046 7411
7412static void find_equal_scalars(struct bpf_verifier_state *vstate,
7413 struct bpf_reg_state *known_reg)
7414{
7415 struct bpf_func_state *state;
7416 struct bpf_reg_state *reg;
7417 int i, j;
7418
7419 for (i = 0; i <= vstate->curframe; i++) {
7420 state = vstate->frame[i];
7421 for (j = 0; j < MAX_BPF_REG; j++) {
7422 reg = &state->regs[j];
7423 if (reg->type == SCALAR_VALUE && reg->id == known_reg->id)
7424 *reg = *known_reg;
7425 }
7426
7427 bpf_for_each_spilled_reg(j, state, reg) {
7428 if (!reg)
7429 continue;
7430 if (reg->type == SCALAR_VALUE && reg->id == known_reg->id)
7431 *reg = *known_reg;
7432 }
7433 }
7434}
7435
7047static int check_cond_jmp_op(struct bpf_verifier_env *env,7436static int check_cond_jmp_op(struct bpf_verifier_env *env,
7048 struct bpf_insn *insn, int *insn_idx)7437 struct bpf_insn *insn, int *insn_idx)
7049{7438{
@@ -7172,6 +7561,12 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env, · 122 unmodified lines
7172 reg_combine_min_max(&other_branch_regs[insn->src_reg],7561 reg_combine_min_max(&other_branch_regs[insn->src_reg],
7173 &other_branch_regs[insn->dst_reg],7562 &other_branch_regs[insn->dst_reg],
7174 src_reg, dst_reg, opcode);7563 src_reg, dst_reg, opcode);
7564 if (src_reg->id &&
7565 !WARN_ON_ONCE(src_reg->id != other_branch_regs[insn->src_reg].id)) {
7566 find_equal_scalars(this_branch, src_reg);
7567 find_equal_scalars(other_branch, &other_branch_regs[insn->src_reg]);
7568 }
7569
7175 }7570 }
7176 } else if (dst_reg->type == SCALAR_VALUE) {7571 } else if (dst_reg->type == SCALAR_VALUE) {
7177 reg_set_min_max(&other_branch_regs[insn->dst_reg],7572 reg_set_min_max(&other_branch_regs[insn->dst_reg],
@@ -7179,6 +7574,12 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env, · 1 unmodified line
7179 opcode, is_jmp32);7574 opcode, is_jmp32);
7180 }7575 }
7181 7576
7577 if (dst_reg->type == SCALAR_VALUE && dst_reg->id &&
7578 !WARN_ON_ONCE(dst_reg->id != other_branch_regs[insn->dst_reg].id)) {
7579 find_equal_scalars(this_branch, dst_reg);
7580 find_equal_scalars(other_branch, &other_branch_regs[insn->dst_reg]);
7581 }
7582
7182 /* detect if R == 0 where R is returned from bpf_map_lookup_elem().7583 /* detect if R == 0 where R is returned from bpf_map_lookup_elem().
7183 * NOTE: these optimizations below are related with pointer comparison7584 * NOTE: these optimizations below are related with pointer comparison
7184 * which will never be JMP32.7585 * which will never be JMP32.
@@ -7210,6 +7611,7 @@ static int check_ld_imm(struct bpf_verifier_env *env, struct bpf_insn *insn) · 25 unmodified lines
7210{7611{
7211 struct bpf_insn_aux_data *aux = cur_aux(env);7612 struct bpf_insn_aux_data *aux = cur_aux(env);
7212 struct bpf_reg_state *regs = cur_regs(env);7613 struct bpf_reg_state *regs = cur_regs(env);
7614 struct bpf_reg_state *dst_reg;
7213 struct bpf_map *map;7615 struct bpf_map *map;
7214 int err;7616 int err;
7215 7617
@@ -7226,25 +7628,45 @@ static int check_ld_imm(struct bpf_verifier_env *env, struct bpf_insn *insn) · 10 unmodified lines
7226 if (err)7628 if (err)
7227 return err;7629 return err;
7228 7630
7631 dst_reg = &regs[insn->dst_reg];
7229 if (insn->src_reg == 0) {7632 if (insn->src_reg == 0) {
7230 u64 imm = ((u64)(insn + 1)->imm << 32) | (u32)insn->imm;7633 u64 imm = ((u64)(insn + 1)->imm << 32) | (u32)insn->imm;
7231 7634
7232 regs[insn->dst_reg].type = SCALAR_VALUE;7635 dst_reg->type = SCALAR_VALUE;
7233 __mark_reg_known(&regs[insn->dst_reg], imm);7636 __mark_reg_known(&regs[insn->dst_reg], imm);
7234 return 0;7637 return 0;
7235 }7638 }
7236 7639
7640 if (insn->src_reg == BPF_PSEUDO_BTF_ID) {
7641 mark_reg_known_zero(env, regs, insn->dst_reg);
7642
7643 dst_reg->type = aux->btf_var.reg_type;
7644 switch (dst_reg->type) {
7645 case PTR_TO_MEM:
7646 dst_reg->mem_size = aux->btf_var.mem_size;
7647 break;
7648 case PTR_TO_BTF_ID:
7649 case PTR_TO_PERCPU_BTF_ID:
7650 dst_reg->btf_id = aux->btf_var.btf_id;
7651 break;
7652 default:
7653 verbose(env, "bpf verifier is misconfigured\n");
7654 return -EFAULT;
7655 }
7656 return 0;
7657 }
7658
7237 map = env->used_maps[aux->map_index];7659 map = env->used_maps[aux->map_index];
7238 mark_reg_known_zero(env, regs, insn->dst_reg);7660 mark_reg_known_zero(env, regs, insn->dst_reg);
7239 regs[insn->dst_reg].map_ptr = map;7661 dst_reg->map_ptr = map;
7240 7662
7241 if (insn->src_reg == BPF_PSEUDO_MAP_VALUE) {7663 if (insn->src_reg == BPF_PSEUDO_MAP_VALUE) {
7242 regs[insn->dst_reg].type = PTR_TO_MAP_VALUE;7664 dst_reg->type = PTR_TO_MAP_VALUE;
7243 regs[insn->dst_reg].off = aux->map_off;7665 dst_reg->off = aux->map_off;
7244 if (map_value_has_spin_lock(map))7666 if (map_value_has_spin_lock(map))
7245 regs[insn->dst_reg].id = ++env->id_gen;7667 dst_reg->id = ++env->id_gen;
7246 } else if (insn->src_reg == BPF_PSEUDO_MAP_FD) {7668 } else if (insn->src_reg == BPF_PSEUDO_MAP_FD) {
7247 regs[insn->dst_reg].type = CONST_PTR_TO_MAP;7669 dst_reg->type = CONST_PTR_TO_MAP;
7248 } else {7670 } else {
7249 verbose(env, "bpf verifier is misconfigured\n");7671 verbose(env, "bpf verifier is misconfigured\n");
7250 return -EINVAL;7672 return -EINVAL;
@@ -7287,7 +7709,7 @@ static int check_ld_abs(struct bpf_verifier_env *env, struct bpf_insn *insn) · 36 unmodified lines
7287 u8 mode = BPF_MODE(insn->code);7709 u8 mode = BPF_MODE(insn->code);
7288 int i, err;7710 int i, err;
7289 7711
7290 if (!may_access_skb(env->prog->type)) {7712 if (!may_access_skb(resolve_prog_type(env->prog))) {
7291 verbose(env, "BPF_LD_[ABS|IND] instructions not allowed for this program type\n");7713 verbose(env, "BPF_LD_[ABS|IND] instructions not allowed for this program type\n");
7292 return -EINVAL;7714 return -EINVAL;
7293 }7715 }
@@ -7297,18 +7719,6 @@ static int check_ld_abs(struct bpf_verifier_env *env, struct bpf_insn *insn) · 3 unmodified lines
7297 return -EINVAL;7719 return -EINVAL;
7298 }7720 }
7299 7721
7300 if (env->subprog_cnt > 1) {
7301 /* when program has LD_ABS insn JITs and interpreter assume
7302 * that r1 == ctx == skb which is not the case for callees
7303 * that can have arbitrary arguments. It's problematic
7304 * for main prog as well since JITs would need to analyze
7305 * all functions in order to make proper register save/restore
7306 * decisions in the main prog. Hence disallow LD_ABS with calls
7307 */
7308 verbose(env, "BPF_LD_[ABS|IND] instructions cannot be mixed with bpf-to-bpf calls\n");
7309 return -EINVAL;
7310 }
7311
7312 if (insn->dst_reg != BPF_REG_0 || insn->off != 0 ||7722 if (insn->dst_reg != BPF_REG_0 || insn->off != 0 ||
7313 BPF_SIZE(insn->code) == BPF_DW ||7723 BPF_SIZE(insn->code) == BPF_DW ||
7314 (mode == BPF_ABS && insn->src_reg != BPF_REG_0)) {7724 (mode == BPF_ABS && insn->src_reg != BPF_REG_0)) {
@@ -7375,11 +7785,12 @@ static int check_return_code(struct bpf_verifier_env *env) · 60 unmodified lines
7375 const struct bpf_prog *prog = env->prog;7785 const struct bpf_prog *prog = env->prog;
7376 struct bpf_reg_state *reg;7786 struct bpf_reg_state *reg;
7377 struct tnum range = tnum_range(0, 1);7787 struct tnum range = tnum_range(0, 1);
7788 enum bpf_prog_type prog_type = resolve_prog_type(env->prog);
7378 int err;7789 int err;
7379 7790
7380 /* LSM and struct_ops func-ptr's return type could be "void" */7791 /* LSM and struct_ops func-ptr's return type could be "void" */
7381 if ((env->prog->type == BPF_PROG_TYPE_STRUCT_OPS ||7792 if ((prog_type == BPF_PROG_TYPE_STRUCT_OPS ||
7382 env->prog->type == BPF_PROG_TYPE_LSM) &&7793 prog_type == BPF_PROG_TYPE_LSM) &&
7383 !prog->aux->attach_func_proto->type)7794 !prog->aux->attach_func_proto->type)
7384 return 0;7795 return 0;
7385 7796
@@ -7398,7 +7809,7 @@ static int check_return_code(struct bpf_verifier_env *env) · 12 unmodified lines
7398 return -EACCES;7809 return -EACCES;
7399 }7810 }
7400 7811
7401 switch (env->prog->type) {7812 switch (prog_type) {
7402 case BPF_PROG_TYPE_CGROUP_SOCK_ADDR:7813 case BPF_PROG_TYPE_CGROUP_SOCK_ADDR:
7403 if (env->prog->expected_attach_type == BPF_CGROUP_UDP4_RECVMSG ||7814 if (env->prog->expected_attach_type == BPF_CGROUP_UDP4_RECVMSG ||
7404 env->prog->expected_attach_type == BPF_CGROUP_UDP6_RECVMSG ||7815 env->prog->expected_attach_type == BPF_CGROUP_UDP6_RECVMSG ||
@@ -7718,6 +8129,23 @@ err_free: · 313 unmodified lines
7718 return ret;8129 return ret;
7719}8130}
7720 8131
8132static int check_abnormal_return(struct bpf_verifier_env *env)
8133{
8134 int i;
8135
8136 for (i = 1; i < env->subprog_cnt; i++) {
8137 if (env->subprog_info[i].has_ld_abs) {
8138 verbose(env, "LD_ABS is not allowed in subprogs without BTF\n");
8139 return -EINVAL;
8140 }
8141 if (env->subprog_info[i].has_tail_call) {
8142 verbose(env, "tail_call is not allowed in subprogs without BTF\n");
8143 return -EINVAL;
8144 }
8145 }
8146 return 0;
8147}
8148
7721/* The minimum supported BTF func info size */8149/* The minimum supported BTF func info size */
7722#define MIN_BPF_FUNCINFO_SIZE 88150#define MIN_BPF_FUNCINFO_SIZE 8
7723#define MAX_FUNCINFO_REC_SIZE 2528151#define MAX_FUNCINFO_REC_SIZE 252
@@ -7726,20 +8154,24 @@ static int check_btf_func(struct bpf_verifier_env *env, · 2 unmodified lines
7726 const union bpf_attr *attr,8154 const union bpf_attr *attr,
7727 union bpf_attr __user *uattr)8155 union bpf_attr __user *uattr)
7728{8156{
8157 const struct btf_type *type, *func_proto, *ret_type;
7729 u32 i, nfuncs, urec_size, min_size;8158 u32 i, nfuncs, urec_size, min_size;
7730 u32 krec_size = sizeof(struct bpf_func_info);8159 u32 krec_size = sizeof(struct bpf_func_info);
7731 struct bpf_func_info *krecord;8160 struct bpf_func_info *krecord;
7732 struct bpf_func_info_aux *info_aux = NULL;8161 struct bpf_func_info_aux *info_aux = NULL;
7733 const struct btf_type *type;
7734 struct bpf_prog *prog;8162 struct bpf_prog *prog;
7735 const struct btf *btf;8163 const struct btf *btf;
7736 void __user *urecord;8164 void __user *urecord;
7737 u32 prev_offset = 0;8165 u32 prev_offset = 0;
8166 bool scalar_return;
7738 int ret = -ENOMEM;8167 int ret = -ENOMEM;
7739 8168
7740 nfuncs = attr->func_info_cnt;8169 nfuncs = attr->func_info_cnt;
7741 if (!nfuncs)8170 if (!nfuncs) {
8171 if (check_abnormal_return(env))
8172 return -EINVAL;
7742 return 0;8173 return 0;
8174 }
7743 8175
7744 if (nfuncs != env->subprog_cnt) {8176 if (nfuncs != env->subprog_cnt) {
7745 verbose(env, "number of funcs in func_info doesn't match number of subprogs\n");8177 verbose(env, "number of funcs in func_info doesn't match number of subprogs\n");
@@ -7787,25 +8219,23 @@ static int check_btf_func(struct bpf_verifier_env *env, · 41 unmodified lines
7787 }8219 }
7788 8220
7789 /* check insn_off */8221 /* check insn_off */
8222 ret = -EINVAL;
7790 if (i == 0) {8223 if (i == 0) {
7791 if (krecord[i].insn_off) {8224 if (krecord[i].insn_off) {
7792 verbose(env,8225 verbose(env,
7793 "nonzero insn_off %u for the first func info record",8226 "nonzero insn_off %u for the first func info record",
7794 krecord[i].insn_off);8227 krecord[i].insn_off);
7795 ret = -EINVAL;
7796 goto err_free;8228 goto err_free;
7797 }8229 }
7798 } else if (krecord[i].insn_off <= prev_offset) {8230 } else if (krecord[i].insn_off <= prev_offset) {
7799 verbose(env,8231 verbose(env,
7800 "same or smaller insn offset (%u) than previous func info record (%u)",8232 "same or smaller insn offset (%u) than previous func info record (%u)",
7801 krecord[i].insn_off, prev_offset);8233 krecord[i].insn_off, prev_offset);
7802 ret = -EINVAL;
7803 goto err_free;8234 goto err_free;
7804 }8235 }
7805 8236
7806 if (env->subprog_info[i].start != krecord[i].insn_off) {8237 if (env->subprog_info[i].start != krecord[i].insn_off) {
7807 verbose(env, "func_info BTF section doesn't match subprog layout in BPF program\n");8238 verbose(env, "func_info BTF section doesn't match subprog layout in BPF program\n");
7808 ret = -EINVAL;
7809 goto err_free;8239 goto err_free;
7810 }8240 }
7811 8241
@@ -7814,10 +8244,26 @@ static int check_btf_func(struct bpf_verifier_env *env, · 2 unmodified lines
7814 if (!type || !btf_type_is_func(type)) {8244 if (!type || !btf_type_is_func(type)) {
7815 verbose(env, "invalid type id %d in func info",8245 verbose(env, "invalid type id %d in func info",
7816 krecord[i].type_id);8246 krecord[i].type_id);
7817 ret = -EINVAL;
7818 goto err_free;8247 goto err_free;
7819 }8248 }
7820 info_aux[i].linkage = BTF_INFO_VLEN(type->info);8249 info_aux[i].linkage = BTF_INFO_VLEN(type->info);
8250
8251 func_proto = btf_type_by_id(btf, type->type);
8252 if (unlikely(!func_proto || !btf_type_is_func_proto(func_proto)))
8253 /* btf_func_check() already verified it during BTF load */
8254 goto err_free;
8255 ret_type = btf_type_skip_modifiers(btf, func_proto->type, NULL);
8256 scalar_return =
8257 btf_type_is_small_int(ret_type) || btf_type_is_enum(ret_type);
8258 if (i && !scalar_return && env->subprog_info[i].has_ld_abs) {
8259 verbose(env, "LD_ABS is only allowed in functions that return 'int'.\n");
8260 goto err_free;
8261 }
8262 if (i && !scalar_return && env->subprog_info[i].has_tail_call) {
8263 verbose(env, "tail_call is only allowed in functions that return 'int'.\n");
8264 goto err_free;
8265 }
8266
7821 prev_offset = krecord[i].insn_off;8267 prev_offset = krecord[i].insn_off;
7822 urecord += urec_size;8268 urecord += urec_size;
7823 }8269 }
@@ -7978,8 +8424,11 @@ static int check_btf_info(struct bpf_verifier_env *env, · 154 unmodified lines
7978 struct btf *btf;8424 struct btf *btf;
7979 int err;8425 int err;
7980 8426
7981 if (!attr->func_info_cnt && !attr->line_info_cnt)8427 if (!attr->func_info_cnt && !attr->line_info_cnt) {
8428 if (check_abnormal_return(env))
8429 return -EINVAL;
7982 return 0;8430 return 0;
8431 }
7983 8432
7984 btf = btf_get_by_fd(attr->prog_btf_fd);8433 btf = btf_get_by_fd(attr->prog_btf_fd);
7985 if (IS_ERR(btf))8434 if (IS_ERR(btf))
@@ -9119,6 +9568,92 @@ process_bpf_exit: · 1133 unmodified lines
9119 return 0;9568 return 0;
9120}9569}
9121 9570
9571/* replace pseudo btf_id with kernel symbol address */
9572static int check_pseudo_btf_id(struct bpf_verifier_env *env,
9573 struct bpf_insn *insn,
9574 struct bpf_insn_aux_data *aux)
9575{
9576 u32 datasec_id, type, id = insn->imm;
9577 const struct btf_var_secinfo *vsi;
9578 const struct btf_type *datasec;
9579 const struct btf_type *t;
9580 const char *sym_name;
9581 bool percpu = false;
9582 u64 addr;
9583 int i;
9584
9585 if (!btf_vmlinux) {
9586 verbose(env, "kernel is missing BTF, make sure CONFIG_DEBUG_INFO_BTF=y is specified in Kconfig.\n");
9587 return -EINVAL;
9588 }
9589
9590 if (insn[1].imm != 0) {
9591 verbose(env, "reserved field (insn[1].imm) is used in pseudo_btf_id ldimm64 insn.\n");
9592 return -EINVAL;
9593 }
9594
9595 t = btf_type_by_id(btf_vmlinux, id);
9596 if (!t) {
9597 verbose(env, "ldimm64 insn specifies invalid btf_id %d.\n", id);
9598 return -ENOENT;
9599 }
9600
9601 if (!btf_type_is_var(t)) {
9602 verbose(env, "pseudo btf_id %d in ldimm64 isn't KIND_VAR.\n",
9603 id);
9604 return -EINVAL;
9605 }
9606
9607 sym_name = btf_name_by_offset(btf_vmlinux, t->name_off);
9608 addr = kallsyms_lookup_name(sym_name);
9609 if (!addr) {
9610 verbose(env, "ldimm64 failed to find the address for kernel symbol '%s'.\n",
9611 sym_name);
9612 return -ENOENT;
9613 }
9614
9615 datasec_id = btf_find_by_name_kind(btf_vmlinux, ".data..percpu",
9616 BTF_KIND_DATASEC);
9617 if (datasec_id > 0) {
9618 datasec = btf_type_by_id(btf_vmlinux, datasec_id);
9619 for_each_vsi(i, datasec, vsi) {
9620 if (vsi->type == id) {
9621 percpu = true;
9622 break;
9623 }
9624 }
9625 }
9626
9627 insn[0].imm = (u32)addr;
9628 insn[1].imm = addr >> 32;
9629
9630 type = t->type;
9631 t = btf_type_skip_modifiers(btf_vmlinux, type, NULL);
9632 if (percpu) {
9633 aux->btf_var.reg_type = PTR_TO_PERCPU_BTF_ID;
9634 aux->btf_var.btf_id = type;
9635 } else if (!btf_type_is_struct(t)) {
9636 const struct btf_type *ret;
9637 const char *tname;
9638 u32 tsize;
9639
9640 /* resolve the type size of ksym. */
9641 ret = btf_resolve_size(btf_vmlinux, t, &tsize);
9642 if (IS_ERR(ret)) {
9643 tname = btf_name_by_offset(btf_vmlinux, t->name_off);
9644 verbose(env, "ldimm64 unable to resolve the size of type '%s': %ld\n",
9645 tname, PTR_ERR(ret));
9646 return -EINVAL;
9647 }
9648 aux->btf_var.reg_type = PTR_TO_MEM;
9649 aux->btf_var.mem_size = tsize;
9650 } else {
9651 aux->btf_var.reg_type = PTR_TO_BTF_ID;
9652 aux->btf_var.btf_id = type;
9653 }
9654 return 0;
9655}
9656
9122static int check_map_prealloc(struct bpf_map *map)9657static int check_map_prealloc(struct bpf_map *map)
9123{9658{
9124 return (map->map_type != BPF_MAP_TYPE_HASH &&9659 return (map->map_type != BPF_MAP_TYPE_HASH &&
@@ -9154,6 +9689,7 @@ static int check_map_prog_compatibility(struct bpf_verifier_env *env, · 29 unmodified lines
9154 struct bpf_prog *prog)9689 struct bpf_prog *prog)
9155 9690
9156{9691{
9692 enum bpf_prog_type prog_type = resolve_prog_type(prog);
9157 /*9693 /*
9158 * Validate that trace type programs use preallocated hash maps.9694 * Validate that trace type programs use preallocated hash maps.
9159 *9695 *
@@ -9171,8 +9707,8 @@ static int check_map_prog_compatibility(struct bpf_verifier_env *env, · 11 unmodified lines
9171 * now, but warnings are emitted so developers are made aware of9707 * now, but warnings are emitted so developers are made aware of
9172 * the unsafety and can fix their programs before this is enforced.9708 * the unsafety and can fix their programs before this is enforced.
9173 */9709 */
9174 if (is_tracing_prog_type(prog->type) && !is_preallocated_map(map)) {9710 if (is_tracing_prog_type(prog_type) && !is_preallocated_map(map)) {
9175 if (prog->type == BPF_PROG_TYPE_PERF_EVENT) {9711 if (prog_type == BPF_PROG_TYPE_PERF_EVENT) {
9176 verbose(env, "perf_event programs can only use preallocated hash map\n");9712 verbose(env, "perf_event programs can only use preallocated hash map\n");
9177 return -EINVAL;9713 return -EINVAL;
9178 }9714 }
@@ -9184,8 +9720,8 @@ static int check_map_prog_compatibility(struct bpf_verifier_env *env, · 5 unmodified lines
9184 verbose(env, "trace type programs with run-time allocated hash maps are unsafe. Switch to preallocated hash maps.\n");9720 verbose(env, "trace type programs with run-time allocated hash maps are unsafe. Switch to preallocated hash maps.\n");
9185 }9721 }
9186 9722
9187 if ((is_tracing_prog_type(prog->type) ||9723 if ((is_tracing_prog_type(prog_type) ||
9188 prog->type == BPF_PROG_TYPE_SOCKET_FILTER) &&9724 prog_type == BPF_PROG_TYPE_SOCKET_FILTER) &&
9189 map_value_has_spin_lock(map)) {9725 map_value_has_spin_lock(map)) {
9190 verbose(env, "tracing progs cannot use bpf_spin_lock yet\n");9726 verbose(env, "tracing progs cannot use bpf_spin_lock yet\n");
9191 return -EINVAL;9727 return -EINVAL;
@@ -9202,6 +9738,23 @@ static int check_map_prog_compatibility(struct bpf_verifier_env *env, · 10 unmodified lines
9202 return -EINVAL;9738 return -EINVAL;
9203 }9739 }
9204 9740
9741 if (prog->aux->sleepable)
9742 switch (map->map_type) {
9743 case BPF_MAP_TYPE_HASH:
9744 case BPF_MAP_TYPE_LRU_HASH:
9745 case BPF_MAP_TYPE_ARRAY:
9746 if (!is_preallocated_map(map)) {
9747 verbose(env,
9748 "Sleepable programs can only use preallocated hash maps\n");
9749 return -EINVAL;
9750 }
9751 break;
9752 default:
9753 verbose(env,
9754 "Sleepable programs can only use array and hash maps\n");
9755 return -EINVAL;
9756 }
9757
9205 return 0;9758 return 0;
9206}9759}
9207 9760
@@ -9211,10 +9764,14 @@ static bool bpf_map_is_cgroup_storage(struct bpf_map *map) · 3 unmodified lines
9211 map->map_type == BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE);9764 map->map_type == BPF_MAP_TYPE_PERCPU_CGROUP_STORAGE);
9212}9765}
9213 9766
9214/* look for pseudo eBPF instructions that access map FDs and9767/* find and rewrite pseudo imm in ld_imm64 instructions:
9215 * replace them with actual map pointers9768 *
9769 * 1. if it accesses map FD, replace it with actual map pointer.
9770 * 2. if it accesses btf_id of a VAR, replace it with pointer to the var.
9771 *
9772 * NOTE: btf_vmlinux is required for converting pseudo btf_id.
9216 */9773 */
9217static int replace_map_fd_with_map_ptr(struct bpf_verifier_env *env)9774static int resolve_pseudo_ldimm64(struct bpf_verifier_env *env)
9218{9775{
9219 struct bpf_insn *insn = env->prog->insnsi;9776 struct bpf_insn *insn = env->prog->insnsi;
9220 int insn_cnt = env->prog->len;9777 int insn_cnt = env->prog->len;
@@ -9255,6 +9812,14 @@ static int replace_map_fd_with_map_ptr(struct bpf_verifier_env *env) · 34 unmodified lines
9255 /* valid generic load 64-bit imm */9812 /* valid generic load 64-bit imm */
9256 goto next_insn;9813 goto next_insn;
9257 9814
9815 if (insn[0].src_reg == BPF_PSEUDO_BTF_ID) {
9816 aux = &env->insn_aux_data[i];
9817 err = check_pseudo_btf_id(env, insn, aux);
9818 if (err)
9819 return err;
9820 goto next_insn;
9821 }
9822
9258 /* In final convert_pseudo_ld_imm64() step, this is9823 /* In final convert_pseudo_ld_imm64() step, this is
9259 * converted into regular 64-bit imm load insn.9824 * converted into regular 64-bit imm load insn.
9260 */9825 */
@@ -9436,6 +10001,18 @@ static void adjust_subprog_starts(struct bpf_verifier_env *env, u32 off, u32 len · 175 unmodified lines
9436 }10001 }
9437}10002}
9438 10003
10004static void adjust_poke_descs(struct bpf_prog *prog, u32 len)
10005{
10006 struct bpf_jit_poke_descriptor *tab = prog->aux->poke_tab;
10007 int i, sz = prog->aux->size_poke_tab;
10008 struct bpf_jit_poke_descriptor *desc;
10009
10010 for (i = 0; i < sz; i++) {
10011 desc = &tab[i];
10012 desc->insn_idx += len - 1;
10013 }
10014}
10015
9439static struct bpf_prog *bpf_patch_insn_data(struct bpf_verifier_env *env, u32 off,10016static struct bpf_prog *bpf_patch_insn_data(struct bpf_verifier_env *env, u32 off,
9440 const struct bpf_insn *patch, u32 len)10017 const struct bpf_insn *patch, u32 len)
9441{10018{
@@ -9452,6 +10029,7 @@ static struct bpf_prog *bpf_patch_insn_data(struct bpf_verifier_env *env, u32 of · 10 unmodified lines
9452 if (adjust_insn_aux_data(env, new_prog, off, len))10029 if (adjust_insn_aux_data(env, new_prog, off, len))
9453 return NULL;10030 return NULL;
9454 adjust_subprog_starts(env, off, len);10031 adjust_subprog_starts(env, off, len);
10032 adjust_poke_descs(new_prog, len);
9455 return new_prog;10033 return new_prog;
9456}10034}
9457 10035
@@ -9897,7 +10475,7 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env) · 439 unmodified lines
9897 insn->code = BPF_LDX | BPF_PROBE_MEM |10475 insn->code = BPF_LDX | BPF_PROBE_MEM |
9898 BPF_SIZE((insn)->code);10476 BPF_SIZE((insn)->code);
9899 env->prog->aux->num_exentries++;10477 env->prog->aux->num_exentries++;
9900 } else if (env->prog->type != BPF_PROG_TYPE_STRUCT_OPS) {10478 } else if (resolve_prog_type(env->prog) != BPF_PROG_TYPE_STRUCT_OPS) {
9901 verbose(env, "Writes through BTF pointers are not allowed\n");10479 verbose(env, "Writes through BTF pointers are not allowed\n");
9902 return -EINVAL;10480 return -EINVAL;
9903 }10481 }
@@ -9982,6 +10560,7 @@ static int jit_subprogs(struct bpf_verifier_env *env) · 78 unmodified lines
9982{10560{
9983 struct bpf_prog *prog = env->prog, **func, *tmp;10561 struct bpf_prog *prog = env->prog, **func, *tmp;
9984 int i, j, subprog_start, subprog_end = 0, len, subprog;10562 int i, j, subprog_start, subprog_end = 0, len, subprog;
10563 struct bpf_map *map_ptr;
9985 struct bpf_insn *insn;10564 struct bpf_insn *insn;
9986 void *old_bpf_func;10565 void *old_bpf_func;
9987 int err, num_exentries;10566 int err, num_exentries;
@@ -10049,6 +10628,31 @@ static int jit_subprogs(struct bpf_verifier_env *env) · 61 unmodified lines
10049 func[i]->aux->btf = prog->aux->btf;10628 func[i]->aux->btf = prog->aux->btf;
10050 func[i]->aux->func_info = prog->aux->func_info;10629 func[i]->aux->func_info = prog->aux->func_info;
10051 10630
10631 for (j = 0; j < prog->aux->size_poke_tab; j++) {
10632 u32 insn_idx = prog->aux->poke_tab[j].insn_idx;
10633 int ret;
10634
10635 if (!(insn_idx >= subprog_start &&
10636 insn_idx <= subprog_end))
10637 continue;
10638
10639 ret = bpf_jit_add_poke_descriptor(func[i],
10640 &prog->aux->poke_tab[j]);
10641 if (ret < 0) {
10642 verbose(env, "adding tail call poke descriptor failed\n");
10643 goto out_free;
10644 }
10645
10646 func[i]->insnsi[insn_idx - subprog_start].imm = ret + 1;
10647
10648 map_ptr = func[i]->aux->poke_tab[ret].tail_call.map;
10649 ret = map_ptr->ops->map_poke_track(map_ptr, func[i]->aux);
10650 if (ret < 0) {
10651 verbose(env, "tracking tail call prog failed\n");
10652 goto out_free;
10653 }
10654 }
10655
10052 /* Use bpf_prog_F_tag to indicate functions in stack traces.10656 /* Use bpf_prog_F_tag to indicate functions in stack traces.
10053 * Long term would need debug info to populate names10657 * Long term would need debug info to populate names
10054 */10658 */
@@ -10067,6 +10671,7 @@ static int jit_subprogs(struct bpf_verifier_env *env) · 12 unmodified lines
10067 num_exentries++;10671 num_exentries++;
10068 }10672 }
10069 func[i]->aux->num_exentries = num_exentries;10673 func[i]->aux->num_exentries = num_exentries;
10674 func[i]->aux->tail_call_reachable = env->subprog_info[i].tail_call_reachable;
10070 func[i] = bpf_int_jit_compile(func[i]);10675 func[i] = bpf_int_jit_compile(func[i]);
10071 if (!func[i]->jited) {10676 if (!func[i]->jited) {
10072 err = -ENOTSUPP;10677 err = -ENOTSUPP;
@@ -10074,6 +10679,19 @@ static int jit_subprogs(struct bpf_verifier_env *env) · 1 unmodified line
10074 }10679 }
10075 cond_resched();10680 cond_resched();
10076 }10681 }
10682
10683 /* Untrack main program's aux structs so that during map_poke_run()
10684 * we will not stumble upon the unfilled poke descriptors; each
10685 * of the main program's poke descs got distributed across subprogs
10686 * and got tracked onto map, so we are sure that none of them will
10687 * be missed after the operation below
10688 */
10689 for (i = 0; i < prog->aux->size_poke_tab; i++) {
10690 map_ptr = prog->aux->poke_tab[i].tail_call.map;
10691
10692 map_ptr->ops->map_poke_untrack(map_ptr, prog->aux);
10693 }
10694
10077 /* at this point all bpf functions were successfully JITed10695 /* at this point all bpf functions were successfully JITed
10078 * now populate all bpf_calls with correct addresses and10696 * now populate all bpf_calls with correct addresses and
10079 * run last pass of JIT10697 * run last pass of JIT
@@ -10142,9 +10760,16 @@ static int jit_subprogs(struct bpf_verifier_env *env) · 62 unmodified lines
10142 bpf_prog_free_unused_jited_linfo(prog);10760 bpf_prog_free_unused_jited_linfo(prog);
10143 return 0;10761 return 0;
10144out_free:10762out_free:
10145 for (i = 0; i < env->subprog_cnt; i++)10763 for (i = 0; i < env->subprog_cnt; i++) {
10146 if (func[i])10764 if (!func[i])
10147 bpf_jit_free(func[i]);10765 continue;
10766
10767 for (j = 0; j < func[i]->aux->size_poke_tab; j++) {
10768 map_ptr = func[i]->aux->poke_tab[j].tail_call.map;
10769 map_ptr->ops->map_poke_untrack(map_ptr, func[i]->aux);
10770 }
10771 bpf_jit_free(func[i]);
10772 }
10148 kfree(func);10773 kfree(func);
10149out_undo_insn:10774out_undo_insn:
10150 /* cleanup main prog to be interpreted */10775 /* cleanup main prog to be interpreted */
@@ -10178,6 +10803,13 @@ static int fixup_call_args(struct bpf_verifier_env *env) · 27 unmodified lines
10178 return err;10803 return err;
10179 }10804 }
10180#ifndef CONFIG_BPF_JIT_ALWAYS_ON10805#ifndef CONFIG_BPF_JIT_ALWAYS_ON
10806 if (env->subprog_cnt > 1 && env->prog->aux->tail_call_reachable) {
10807 /* When JIT fails the progs with bpf2bpf calls and tail_calls
10808 * have to be rejected, since interpreter doesn't support them yet.
10809 */
10810 verbose(env, "tail_calls are not allowed in non-JITed programs with bpf-to-bpf calls\n");
10811 return -EINVAL;
10812 }
10181 for (i = 0; i < prog->len; i++, insn++) {10813 for (i = 0; i < prog->len; i++, insn++) {
10182 if (insn->code != (BPF_JMP | BPF_CALL) ||10814 if (insn->code != (BPF_JMP | BPF_CALL) ||
10183 insn->src_reg != BPF_PSEUDO_CALL)10815 insn->src_reg != BPF_PSEUDO_CALL)
@@ -10341,8 +10973,9 @@ static int fixup_bpf_calls(struct bpf_verifier_env *env) · 157 unmodified lines
10341 * the program array.10973 * the program array.
10342 */10974 */
10343 prog->cb_access = 1;10975 prog->cb_access = 1;
10344 env->prog->aux->stack_depth = MAX_BPF_STACK;10976 if (!allow_tail_call_in_subprogs(env))
10345 env->prog->aux->max_pkt_offset = MAX_PACKET_OFF;10977 prog->aux->stack_depth = MAX_BPF_STACK;
10978 prog->aux->max_pkt_offset = MAX_PACKET_OFF;
10346 10979
10347 /* mark bpf_tail_call as different opcode to avoid10980 /* mark bpf_tail_call as different opcode to avoid
10348 * conditional branch in the interpeter for every normal10981 * conditional branch in the interpeter for every normal
@@ -10362,6 +10995,7 @@ static int fixup_bpf_calls(struct bpf_verifier_env *env) · 13 unmodified lines
10362 .reason = BPF_POKE_REASON_TAIL_CALL,10995 .reason = BPF_POKE_REASON_TAIL_CALL,
10363 .tail_call.map = BPF_MAP_PTR(aux->map_ptr_state),10996 .tail_call.map = BPF_MAP_PTR(aux->map_ptr_state),
10364 .tail_call.key = bpf_map_key_immediate(aux),10997 .tail_call.key = bpf_map_key_immediate(aux),
10998 .insn_idx = i + delta,
10365 };10999 };
10366 11000
10367 ret = bpf_jit_add_poke_descriptor(prog, &desc);11001 ret = bpf_jit_add_poke_descriptor(prog, &desc);
@@ -10427,7 +11061,9 @@ static int fixup_bpf_calls(struct bpf_verifier_env *env) · 59 unmodified lines
10427 if (insn->imm == BPF_FUNC_map_lookup_elem &&11061 if (insn->imm == BPF_FUNC_map_lookup_elem &&
10428 ops->map_gen_lookup) {11062 ops->map_gen_lookup) {
10429 cnt = ops->map_gen_lookup(map_ptr, insn_buf);11063 cnt = ops->map_gen_lookup(map_ptr, insn_buf);
10430 if (cnt == 0 || cnt >= ARRAY_SIZE(insn_buf)) {11064 if (cnt == -EOPNOTSUPP)
11065 goto patch_map_ops_generic;
11066 if (cnt <= 0 || cnt >= ARRAY_SIZE(insn_buf)) {
10431 verbose(env, "bpf verifier is misconfigured\n");11067 verbose(env, "bpf verifier is misconfigured\n");
10432 return -EINVAL;11068 return -EINVAL;
10433 }11069 }
@@ -10457,7 +11093,7 @@ static int fixup_bpf_calls(struct bpf_verifier_env *env) · 23 unmodified lines
10457 (int (*)(struct bpf_map *map, void *value))NULL));11093 (int (*)(struct bpf_map *map, void *value))NULL));
10458 BUILD_BUG_ON(!__same_type(ops->map_peek_elem,11094 BUILD_BUG_ON(!__same_type(ops->map_peek_elem,
10459 (int (*)(struct bpf_map *map, void *value))NULL));11095 (int (*)(struct bpf_map *map, void *value))NULL));
10460 11096patch_map_ops_generic:
10461 switch (insn->imm) {11097 switch (insn->imm) {
10462 case BPF_FUNC_map_lookup_elem:11098 case BPF_FUNC_map_lookup_elem:
10463 insn->imm = BPF_CAST_CALL(ops->map_lookup_elem) -11099 insn->imm = BPF_CAST_CALL(ops->map_lookup_elem) -
@@ -10810,59 +11446,79 @@ static int check_struct_ops_btf_id(struct bpf_verifier_env *env) · 346 unmodified lines
10810}11446}
10811#define SECURITY_PREFIX "security_"11447#define SECURITY_PREFIX "security_"
10812 11448
10813static int check_attach_modify_return(struct bpf_prog *prog, unsigned long addr)11449static int check_attach_modify_return(unsigned long addr, const char *func_name)
10814{11450{
10815 if (within_error_injection_list(addr) ||11451 if (within_error_injection_list(addr) ||
10816 !strncmp(SECURITY_PREFIX, prog->aux->attach_func_name,11452 !strncmp(SECURITY_PREFIX, func_name, sizeof(SECURITY_PREFIX) - 1))
10817 sizeof(SECURITY_PREFIX) - 1))
10818 return 0;11453 return 0;
10819 11454
10820 return -EINVAL;11455 return -EINVAL;
10821}11456}
10822 11457
10823static int check_attach_btf_id(struct bpf_verifier_env *env)11458/* non exhaustive list of sleepable bpf_lsm_*() functions */
11459BTF_SET_START(btf_sleepable_lsm_hooks)
11460#ifdef CONFIG_BPF_LSM
11461BTF_ID(func, bpf_lsm_bprm_committed_creds)
11462#else
11463BTF_ID_UNUSED
11464#endif
11465BTF_SET_END(btf_sleepable_lsm_hooks)
11466
11467static int check_sleepable_lsm_hook(u32 btf_id)
11468{
11469 return btf_id_set_contains(&btf_sleepable_lsm_hooks, btf_id);
11470}
11471
11472/* list of non-sleepable functions that are otherwise on
11473 * ALLOW_ERROR_INJECTION list
11474 */
11475BTF_SET_START(btf_non_sleepable_error_inject)
11476/* Three functions below can be called from sleepable and non-sleepable context.
11477 * Assume non-sleepable from bpf safety point of view.
11478 */
11479BTF_ID(func, __add_to_page_cache_locked)
11480BTF_ID(func, should_fail_alloc_page)
11481BTF_ID(func, should_failslab)
11482BTF_SET_END(btf_non_sleepable_error_inject)
11483
11484static int check_non_sleepable_error_inject(u32 btf_id)
11485{
11486 return btf_id_set_contains(&btf_non_sleepable_error_inject, btf_id);
11487}
11488
11489int bpf_check_attach_target(struct bpf_verifier_log *log,
11490 const struct bpf_prog *prog,
11491 const struct bpf_prog *tgt_prog,
11492 u32 btf_id,
11493 struct bpf_attach_target_info *tgt_info)
10824{11494{
10825 struct bpf_prog *prog = env->prog;
10826 bool prog_extension = prog->type == BPF_PROG_TYPE_EXT;11495 bool prog_extension = prog->type == BPF_PROG_TYPE_EXT;
10827 struct bpf_prog *tgt_prog = prog->aux->linked_prog;
10828 u32 btf_id = prog->aux->attach_btf_id;
10829 const char prefix[] = "btf_trace_";11496 const char prefix[] = "btf_trace_";
10830 struct btf_func_model fmodel;
10831 int ret = 0, subprog = -1, i;11497 int ret = 0, subprog = -1, i;
10832 struct bpf_trampoline *tr;
10833 const struct btf_type *t;11498 const struct btf_type *t;
10834 bool conservative = true;11499 bool conservative = true;
10835 const char *tname;11500 const char *tname;
10836 struct btf *btf;11501 struct btf *btf;
10837 long addr;11502 long addr = 0;
10838 u64 key;
10839
10840 if (prog->type == BPF_PROG_TYPE_STRUCT_OPS)
10841 return check_struct_ops_btf_id(env);
10842
10843 if (prog->type != BPF_PROG_TYPE_TRACING &&
10844 prog->type != BPF_PROG_TYPE_LSM &&
10845 !prog_extension)
10846 return 0;
10847 11503
10848 if (!btf_id) {11504 if (!btf_id) {
10849 verbose(env, "Tracing programs must provide btf_id\n");11505 bpf_log(log, "Tracing programs must provide btf_id\n");
10850 return -EINVAL;11506 return -EINVAL;
10851 }11507 }
10852 btf = bpf_prog_get_target_btf(prog);11508 btf = tgt_prog ? tgt_prog->aux->btf : btf_vmlinux;
10853 if (!btf) {11509 if (!btf) {
10854 verbose(env,11510 bpf_log(log,
10855 "FENTRY/FEXIT program can only be attached to another program annotated with BTF\n");11511 "FENTRY/FEXIT program can only be attached to another program annotated with BTF\n");
10856 return -EINVAL;11512 return -EINVAL;
10857 }11513 }
10858 t = btf_type_by_id(btf, btf_id);11514 t = btf_type_by_id(btf, btf_id);
10859 if (!t) {11515 if (!t) {
10860 verbose(env, "attach_btf_id %u is invalid\n", btf_id);11516 bpf_log(log, "attach_btf_id %u is invalid\n", btf_id);
10861 return -EINVAL;11517 return -EINVAL;
10862 }11518 }
10863 tname = btf_name_by_offset(btf, t->name_off);11519 tname = btf_name_by_offset(btf, t->name_off);
10864 if (!tname) {11520 if (!tname) {
10865 verbose(env, "attach_btf_id %u doesn't have a name\n", btf_id);11521 bpf_log(log, "attach_btf_id %u doesn't have a name\n", btf_id);
10866 return -EINVAL;11522 return -EINVAL;
10867 }11523 }
10868 if (tgt_prog) {11524 if (tgt_prog) {
@@ -10874,26 +11530,24 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) · 5 unmodified lines
10874 break;11530 break;
10875 }11531 }
10876 if (subprog == -1) {11532 if (subprog == -1) {
10877 verbose(env, "Subprog %s doesn't exist\n", tname);11533 bpf_log(log, "Subprog %s doesn't exist\n", tname);
10878 return -EINVAL;11534 return -EINVAL;
10879 }11535 }
10880 conservative = aux->func_info_aux[subprog].unreliable;11536 conservative = aux->func_info_aux[subprog].unreliable;
10881 if (prog_extension) {11537 if (prog_extension) {
10882 if (conservative) {11538 if (conservative) {
10883 verbose(env,11539 bpf_log(log,
10884 "Cannot replace static functions\n");11540 "Cannot replace static functions\n");
10885 return -EINVAL;11541 return -EINVAL;
10886 }11542 }
10887 if (!prog->jit_requested) {11543 if (!prog->jit_requested) {
10888 verbose(env,11544 bpf_log(log,
10889 "Extension programs should be JITed\n");11545 "Extension programs should be JITed\n");
10890 return -EINVAL;11546 return -EINVAL;
10891 }11547 }
10892 env->ops = bpf_verifier_ops[tgt_prog->type];
10893 prog->expected_attach_type = tgt_prog->expected_attach_type;
10894 }11548 }
10895 if (!tgt_prog->jited) {11549 if (!tgt_prog->jited) {
10896 verbose(env, "Can attach to only JITed progs\n");11550 bpf_log(log, "Can attach to only JITed progs\n");
10897 return -EINVAL;11551 return -EINVAL;
10898 }11552 }
10899 if (tgt_prog->type == prog->type) {11553 if (tgt_prog->type == prog->type) {
@@ -10901,7 +11555,7 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) · 1 unmodified line
10901 * Cannot attach program extension to another extension.11555 * Cannot attach program extension to another extension.
10902 * It's ok to attach fentry/fexit to extension program.11556 * It's ok to attach fentry/fexit to extension program.
10903 */11557 */
10904 verbose(env, "Cannot recursively attach\n");11558 bpf_log(log, "Cannot recursively attach\n");
10905 return -EINVAL;11559 return -EINVAL;
10906 }11560 }
10907 if (tgt_prog->type == BPF_PROG_TYPE_TRACING &&11561 if (tgt_prog->type == BPF_PROG_TYPE_TRACING &&
@@ -10923,32 +11577,30 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) · 15 unmodified lines
10923 * reasonable stack size. Hence extending fentry is not11577 * reasonable stack size. Hence extending fentry is not
10924 * allowed.11578 * allowed.
10925 */11579 */
10926 verbose(env, "Cannot extend fentry/fexit\n");11580 bpf_log(log, "Cannot extend fentry/fexit\n");
10927 return -EINVAL;11581 return -EINVAL;
10928 }11582 }
10929 key = ((u64)aux->id) << 32 | btf_id;
10930 } else {11583 } else {
10931 if (prog_extension) {11584 if (prog_extension) {
10932 verbose(env, "Cannot replace kernel functions\n");11585 bpf_log(log, "Cannot replace kernel functions\n");
10933 return -EINVAL;11586 return -EINVAL;
10934 }11587 }
10935 key = btf_id;
10936 }11588 }
10937 11589
10938 switch (prog->expected_attach_type) {11590 switch (prog->expected_attach_type) {
10939 case BPF_TRACE_RAW_TP:11591 case BPF_TRACE_RAW_TP:
10940 if (tgt_prog) {11592 if (tgt_prog) {
10941 verbose(env,11593 bpf_log(log,
10942 "Only FENTRY/FEXIT progs are attachable to another BPF prog\n");11594 "Only FENTRY/FEXIT progs are attachable to another BPF prog\n");
10943 return -EINVAL;11595 return -EINVAL;
10944 }11596 }
10945 if (!btf_type_is_typedef(t)) {11597 if (!btf_type_is_typedef(t)) {
10946 verbose(env, "attach_btf_id %u is not a typedef\n",11598 bpf_log(log, "attach_btf_id %u is not a typedef\n",
10947 btf_id);11599 btf_id);
10948 return -EINVAL;11600 return -EINVAL;
10949 }11601 }
10950 if (strncmp(prefix, tname, sizeof(prefix) - 1)) {11602 if (strncmp(prefix, tname, sizeof(prefix) - 1)) {
10951 verbose(env, "attach_btf_id %u points to wrong type name %s\n",11603 bpf_log(log, "attach_btf_id %u points to wrong type name %s\n",
10952 btf_id, tname);11604 btf_id, tname);
10953 return -EINVAL;11605 return -EINVAL;
10954 }11606 }
@@ -10962,29 +11614,20 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) · 7 unmodified lines
10962 /* should never happen in valid vmlinux build */11614 /* should never happen in valid vmlinux build */
10963 return -EINVAL;11615 return -EINVAL;
10964 11616
10965 /* remember two read only pointers that are valid for11617 break;
10966 * the life time of the kernel
10967 */
10968 prog->aux->attach_func_name = tname;
10969 prog->aux->attach_func_proto = t;
10970 prog->aux->attach_btf_trace = true;
10971 return 0;
10972 case BPF_TRACE_ITER:11618 case BPF_TRACE_ITER:
10973 if (!btf_type_is_func(t)) {11619 if (!btf_type_is_func(t)) {
10974 verbose(env, "attach_btf_id %u is not a function\n",11620 bpf_log(log, "attach_btf_id %u is not a function\n",
10975 btf_id);11621 btf_id);
10976 return -EINVAL;11622 return -EINVAL;
10977 }11623 }
10978 t = btf_type_by_id(btf, t->type);11624 t = btf_type_by_id(btf, t->type);
10979 if (!btf_type_is_func_proto(t))11625 if (!btf_type_is_func_proto(t))
10980 return -EINVAL;11626 return -EINVAL;
10981 prog->aux->attach_func_name = tname;11627 ret = btf_distill_func_proto(log, btf, t, tname, &tgt_info->fmodel);
10982 prog->aux->attach_func_proto = t;11628 if (ret)
10983 if (!bpf_iter_prog_supported(prog))11629 return ret;
10984 return -EINVAL;11630 break;
10985 ret = btf_distill_func_proto(&env->log, btf, t,
10986 tname, &fmodel);
10987 return ret;
10988 default:11631 default:
10989 if (!prog_extension)11632 if (!prog_extension)
10990 return -EINVAL;11633 return -EINVAL;
@@ -10993,42 +11636,30 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) · 2 unmodified lines
10993 case BPF_LSM_MAC:11636 case BPF_LSM_MAC:
10994 case BPF_TRACE_FENTRY:11637 case BPF_TRACE_FENTRY:
10995 case BPF_TRACE_FEXIT:11638 case BPF_TRACE_FEXIT:
10996 prog->aux->attach_func_name = tname;
10997 if (prog->type == BPF_PROG_TYPE_LSM) {
10998 ret = bpf_lsm_verify_prog(&env->log, prog);
10999 if (ret < 0)
11000 return ret;
11001 }
11002
11003 if (!btf_type_is_func(t)) {11639 if (!btf_type_is_func(t)) {
11004 verbose(env, "attach_btf_id %u is not a function\n",11640 bpf_log(log, "attach_btf_id %u is not a function\n",
11005 btf_id);11641 btf_id);
11006 return -EINVAL;11642 return -EINVAL;
11007 }11643 }
11008 if (prog_extension &&11644 if (prog_extension &&
11009 btf_check_type_match(env, prog, btf, t))11645 btf_check_type_match(log, prog, btf, t))
11010 return -EINVAL;11646 return -EINVAL;
11011 t = btf_type_by_id(btf, t->type);11647 t = btf_type_by_id(btf, t->type);
11012 if (!btf_type_is_func_proto(t))11648 if (!btf_type_is_func_proto(t))
11013 return -EINVAL;11649 return -EINVAL;
11014 tr = bpf_trampoline_lookup(key);11650
11015 if (!tr)11651 if ((prog->aux->saved_dst_prog_type || prog->aux->saved_dst_attach_type) &&
11016 return -ENOMEM;11652 (!tgt_prog || prog->aux->saved_dst_prog_type != tgt_prog->type ||
11017 /* t is either vmlinux type or another program's type */11653 prog->aux->saved_dst_attach_type != tgt_prog->expected_attach_type))
11018 prog->aux->attach_func_proto = t;11654 return -EINVAL;
11019 mutex_lock(&tr->mutex);11655
11020 if (tr->func.addr) {11656 if (tgt_prog && conservative)
11021 prog->aux->trampoline = tr;
11022 goto out;
11023 }
11024 if (tgt_prog && conservative) {
11025 prog->aux->attach_func_proto = NULL;
11026 t = NULL;11657 t = NULL;
11027 }11658
11028 ret = btf_distill_func_proto(&env->log, btf, t,11659 ret = btf_distill_func_proto(log, btf, t, tname, &tgt_info->fmodel);
11029 tname, &tr->func.model);
11030 if (ret < 0)11660 if (ret < 0)
11031 goto out;11661 return ret;
11662
11032 if (tgt_prog) {11663 if (tgt_prog) {
11033 if (subprog == 0)11664 if (subprog == 0)
11034 addr = (long) tgt_prog->bpf_func;11665 addr = (long) tgt_prog->bpf_func;
@@ -11037,31 +11668,137 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) · 2 unmodified lines
11037 } else {11668 } else {
11038 addr = kallsyms_lookup_name(tname);11669 addr = kallsyms_lookup_name(tname);
11039 if (!addr) {11670 if (!addr) {
11040 verbose(env,11671 bpf_log(log,
11041 "The address of function %s cannot be found\n",11672 "The address of function %s cannot be found\n",
11042 tname);11673 tname);
11043 ret = -ENOENT;11674 return -ENOENT;
11044 goto out;
11045 }11675 }
11046 }11676 }
11047 11677
11048 if (prog->expected_attach_type == BPF_MODIFY_RETURN) {11678 if (prog->aux->sleepable) {
11049 ret = check_attach_modify_return(prog, addr);11679 ret = -EINVAL;
11050 if (ret)11680 switch (prog->type) {
11051 verbose(env, "%s() is not modifiable\n",11681 case BPF_PROG_TYPE_TRACING:
11052 prog->aux->attach_func_name);11682 /* fentry/fexit/fmod_ret progs can be sleepable only if they are
11683 * attached to ALLOW_ERROR_INJECTION and are not in denylist.
11684 */
11685 if (!check_non_sleepable_error_inject(btf_id) &&
11686 within_error_injection_list(addr))
11687 ret = 0;
11688 break;
11689 case BPF_PROG_TYPE_LSM:
11690 /* LSM progs check that they are attached to bpf_lsm_*() funcs.
11691 * Only some of them are sleepable.
11692 */
11693 if (check_sleepable_lsm_hook(btf_id))
11694 ret = 0;
11695 break;
11696 default:
11697 break;
11698 }
11699 if (ret) {
11700 bpf_log(log, "%s is not sleepable\n", tname);
11701 return ret;
11702 }
11703 } else if (prog->expected_attach_type == BPF_MODIFY_RETURN) {
11704 if (tgt_prog) {
11705 bpf_log(log, "can't modify return codes of BPF programs\n");
11706 return -EINVAL;
11707 }
11708 ret = check_attach_modify_return(addr, tname);
11709 if (ret) {
11710 bpf_log(log, "%s() is not modifiable\n", tname);
11711 return ret;
11712 }
11053 }11713 }
11054 11714
11055 if (ret)11715 break;
11056 goto out;11716 }
11057 tr->func.addr = (void *)addr;11717 tgt_info->tgt_addr = addr;
11058 prog->aux->trampoline = tr;11718 tgt_info->tgt_name = tname;
11059out:11719 tgt_info->tgt_type = t;
11060 mutex_unlock(&tr->mutex);11720 return 0;
11061 if (ret)11721}
11062 bpf_trampoline_put(tr);11722
11723static int check_attach_btf_id(struct bpf_verifier_env *env)
11724{
11725 struct bpf_prog *prog = env->prog;
11726 struct bpf_prog *tgt_prog = prog->aux->dst_prog;
11727 struct bpf_attach_target_info tgt_info = {};
11728 u32 btf_id = prog->aux->attach_btf_id;
11729 struct bpf_trampoline *tr;
11730 int ret;
11731 u64 key;
11732
11733 if (prog->aux->sleepable && prog->type != BPF_PROG_TYPE_TRACING &&
11734 prog->type != BPF_PROG_TYPE_LSM) {
11735 verbose(env, "Only fentry/fexit/fmod_ret and lsm programs can be sleepable\n");
11736 return -EINVAL;
11737 }
11738
11739 if (prog->type == BPF_PROG_TYPE_STRUCT_OPS)
11740 return check_struct_ops_btf_id(env);
11741
11742 if (prog->type != BPF_PROG_TYPE_TRACING &&
11743 prog->type != BPF_PROG_TYPE_LSM &&
11744 prog->type != BPF_PROG_TYPE_EXT)
11745 return 0;
11746
11747 ret = bpf_check_attach_target(&env->log, prog, tgt_prog, btf_id, &tgt_info);
11748 if (ret)
11063 return ret;11749 return ret;
11750
11751 if (tgt_prog && prog->type == BPF_PROG_TYPE_EXT) {
11752 /* to make freplace equivalent to their targets, they need to
11753 * inherit env->ops and expected_attach_type for the rest of the
11754 * verification
11755 */
11756 env->ops = bpf_verifier_ops[tgt_prog->type];
11757 prog->expected_attach_type = tgt_prog->expected_attach_type;
11758 }
11759
11760 /* store info about the attachment target that will be used later */
11761 prog->aux->attach_func_proto = tgt_info.tgt_type;
11762 prog->aux->attach_func_name = tgt_info.tgt_name;
11763
11764 if (tgt_prog) {
11765 prog->aux->saved_dst_prog_type = tgt_prog->type;
11766 prog->aux->saved_dst_attach_type = tgt_prog->expected_attach_type;
11064 }11767 }
11768
11769 if (prog->expected_attach_type == BPF_TRACE_RAW_TP) {
11770 prog->aux->attach_btf_trace = true;
11771 return 0;
11772 } else if (prog->expected_attach_type == BPF_TRACE_ITER) {
11773 if (!bpf_iter_prog_supported(prog))
11774 return -EINVAL;
11775 return 0;
11776 }
11777
11778 if (prog->type == BPF_PROG_TYPE_LSM) {
11779 ret = bpf_lsm_verify_prog(&env->log, prog);
11780 if (ret < 0)
11781 return ret;
11782 }
11783
11784 key = bpf_trampoline_compute_key(tgt_prog, btf_id);
11785 tr = bpf_trampoline_get(key, &tgt_info);
11786 if (!tr)
11787 return -ENOMEM;
11788
11789 prog->aux->dst_trampoline = tr;
11790 return 0;
11791}
11792
11793struct btf *bpf_get_btf_vmlinux(void)
11794{
11795 if (!btf_vmlinux && IS_ENABLED(CONFIG_DEBUG_INFO_BTF)) {
11796 mutex_lock(&bpf_verifier_lock);
11797 if (!btf_vmlinux)
11798 btf_vmlinux = btf_parse_vmlinux();
11799 mutex_unlock(&bpf_verifier_lock);
11800 }
11801 return btf_vmlinux;
11065}11802}
11066 11803
11067int bpf_check(struct bpf_prog **prog, union bpf_attr *attr,11804int bpf_check(struct bpf_prog **prog, union bpf_attr *attr,
@@ -11097,12 +11834,7 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, · 29 unmodified lines
11097 env->ops = bpf_verifier_ops[env->prog->type];11834 env->ops = bpf_verifier_ops[env->prog->type];
11098 is_priv = bpf_capable();11835 is_priv = bpf_capable();
11099 11836
11100 if (!btf_vmlinux && IS_ENABLED(CONFIG_DEBUG_INFO_BTF)) {11837 bpf_get_btf_vmlinux();
11101 mutex_lock(&bpf_verifier_lock);
11102 if (!btf_vmlinux)
11103 btf_vmlinux = btf_parse_vmlinux();
11104 mutex_unlock(&bpf_verifier_lock);
11105 }
11106 11838
11107 /* grab the mutex to protect few globals used by verifier */11839 /* grab the mutex to protect few globals used by verifier */
11108 if (!is_priv)11840 if (!is_priv)
@@ -11145,10 +11877,6 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, · 36 unmodified lines
11145 if (is_priv)11877 if (is_priv)
11146 env->test_state_freq = attr->prog_flags & BPF_F_TEST_STATE_FREQ;11878 env->test_state_freq = attr->prog_flags & BPF_F_TEST_STATE_FREQ;
11147 11879
11148 ret = replace_map_fd_with_map_ptr(env);
11149 if (ret < 0)
11150 goto skip_full_check;
11151
11152 if (bpf_prog_is_dev_bound(env->prog->aux)) {11880 if (bpf_prog_is_dev_bound(env->prog->aux)) {
11153 ret = bpf_prog_offload_verifier_prep(env->prog);11881 ret = bpf_prog_offload_verifier_prep(env->prog);
11154 if (ret)11882 if (ret)
@@ -11174,6 +11902,10 @@ int bpf_check(struct bpf_prog **prog, union bpf_attr *attr, · 19 unmodified lines
11174 if (ret)11902 if (ret)
11175 goto skip_full_check;11903 goto skip_full_check;
11176 11904
11905 ret = resolve_pseudo_ldimm64(env);
11906 if (ret < 0)
11907 goto skip_full_check;
11908
11177 ret = check_cfg(env);11909 ret = check_cfg(env);
11178 if (ret < 0)11910 if (ret < 0)
11179 goto skip_full_check;11911 goto skip_full_check;
kernel/rcu/tasks.h+41−12
@@ -28,6 +28,8 @@ typedef void (*postgp_func_t)(struct rcu_tasks *rtp); · 27 unmodified lines
28 * @kthread_ptr: This flavor's grace-period/callback-invocation kthread.28 * @kthread_ptr: This flavor's grace-period/callback-invocation kthread.
29 * @gp_func: This flavor's grace-period-wait function.29 * @gp_func: This flavor's grace-period-wait function.
30 * @gp_state: Grace period's most recent state transition (debugging).30 * @gp_state: Grace period's most recent state transition (debugging).
31 * @gp_sleep: Per-grace-period sleep to prevent CPU-bound looping.
32 * @init_fract: Initial backoff sleep interval.
31 * @gp_jiffies: Time of last @gp_state transition.33 * @gp_jiffies: Time of last @gp_state transition.
32 * @gp_start: Most recent grace-period start in jiffies.34 * @gp_start: Most recent grace-period start in jiffies.
33 * @n_gps: Number of grace periods completed since boot.35 * @n_gps: Number of grace periods completed since boot.
@@ -48,6 +50,8 @@ struct rcu_tasks { · 14 unmodified lines
48 struct wait_queue_head cbs_wq;50 struct wait_queue_head cbs_wq;
49 raw_spinlock_t cbs_lock;51 raw_spinlock_t cbs_lock;
50 int gp_state;52 int gp_state;
53 int gp_sleep;
54 int init_fract;
51 unsigned long gp_jiffies;55 unsigned long gp_jiffies;
52 unsigned long gp_start;56 unsigned long gp_start;
53 unsigned long n_gps;57 unsigned long n_gps;
@@ -81,7 +85,7 @@ static struct rcu_tasks rt_name = \ · 27 unmodified lines
81DEFINE_STATIC_SRCU(tasks_rcu_exit_srcu);85DEFINE_STATIC_SRCU(tasks_rcu_exit_srcu);
82 86
83/* Avoid IPIing CPUs early in the grace period. */87/* Avoid IPIing CPUs early in the grace period. */
84#define RCU_TASK_IPI_DELAY (HZ / 2)88#define RCU_TASK_IPI_DELAY (IS_ENABLED(CONFIG_TASKS_TRACE_RCU_READ_MB) ? HZ / 2 : 0)
85static int rcu_task_ipi_delay __read_mostly = RCU_TASK_IPI_DELAY;89static int rcu_task_ipi_delay __read_mostly = RCU_TASK_IPI_DELAY;
86module_param(rcu_task_ipi_delay, int, 0644);90module_param(rcu_task_ipi_delay, int, 0644);
87 91
@@ -231,7 +235,7 @@ static int __noreturn rcu_tasks_kthread(void *arg) · 143 unmodified lines
231 cond_resched();235 cond_resched();
232 }236 }
233 /* Paranoid sleep to keep this from entering a tight loop */237 /* Paranoid sleep to keep this from entering a tight loop */
234 schedule_timeout_idle(HZ/10);238 schedule_timeout_idle(rtp->gp_sleep);
235 239
236 set_tasks_gp_state(rtp, RTGS_WAIT_CBS);240 set_tasks_gp_state(rtp, RTGS_WAIT_CBS);
237 }241 }
@@ -329,8 +333,10 @@ static void rcu_tasks_wait_gp(struct rcu_tasks *rtp) · 91 unmodified lines
329 */333 */
330 lastreport = jiffies;334 lastreport = jiffies;
331 335
332 /* Start off with HZ/10 wait and slowly back off to 1 HZ wait. */336 // Start off with initial wait and slowly back off to 1 HZ wait.
333 fract = 10;337 fract = rtp->init_fract;
338 if (fract > HZ)
339 fract = HZ;
334 340
335 for (;;) {341 for (;;) {
336 bool firstreport;342 bool firstreport;
@@ -553,6 +559,8 @@ EXPORT_SYMBOL_GPL(rcu_barrier_tasks); · 216 unmodified lines
553 559
554static int __init rcu_spawn_tasks_kthread(void)560static int __init rcu_spawn_tasks_kthread(void)
555{561{
562 rcu_tasks.gp_sleep = HZ / 10;
563 rcu_tasks.init_fract = 10;
556 rcu_tasks.pregp_func = rcu_tasks_pregp_step;564 rcu_tasks.pregp_func = rcu_tasks_pregp_step;
557 rcu_tasks.pertask_func = rcu_tasks_pertask;565 rcu_tasks.pertask_func = rcu_tasks_pertask;
558 rcu_tasks.postscan_func = rcu_tasks_postscan;566 rcu_tasks.postscan_func = rcu_tasks_postscan;
@@ -685,6 +693,7 @@ EXPORT_SYMBOL_GPL(rcu_barrier_tasks_rude); · 126 unmodified lines
685 693
686static int __init rcu_spawn_tasks_rude_kthread(void)694static int __init rcu_spawn_tasks_rude_kthread(void)
687{695{
696 rcu_tasks_rude.gp_sleep = HZ / 10;
688 rcu_spawn_tasks_kthread_generic(&rcu_tasks_rude);697 rcu_spawn_tasks_kthread_generic(&rcu_tasks_rude);
689 return 0;698 return 0;
690}699}
@@ -745,9 +754,9 @@ static DEFINE_PER_CPU(bool, trc_ipi_to_cpu); · 54 unmodified lines
745 754
746// The number of detections of task quiescent state relying on755// The number of detections of task quiescent state relying on
747// heavyweight readers executing explicit memory barriers.756// heavyweight readers executing explicit memory barriers.
748unsigned long n_heavy_reader_attempts;757static unsigned long n_heavy_reader_attempts;
749unsigned long n_heavy_reader_updates;758static unsigned long n_heavy_reader_updates;
750unsigned long n_heavy_reader_ofl_updates;759static unsigned long n_heavy_reader_ofl_updates;
751 760
752void call_rcu_tasks_trace(struct rcu_head *rhp, rcu_callback_t func);761void call_rcu_tasks_trace(struct rcu_head *rhp, rcu_callback_t func);
753DEFINE_RCU_TASKS(rcu_tasks_trace, rcu_tasks_wait_gp, call_rcu_tasks_trace,762DEFINE_RCU_TASKS(rcu_tasks_trace, rcu_tasks_wait_gp, call_rcu_tasks_trace,
@@ -821,6 +830,12 @@ static void trc_read_check_handler(void *t_in) · 67 unmodified lines
821 WRITE_ONCE(t->trc_reader_checked, true);830 WRITE_ONCE(t->trc_reader_checked, true);
822 goto reset_ipi;831 goto reset_ipi;
823 }832 }
833 // If we are racing with an rcu_read_unlock_trace(), try again later.
834 if (unlikely(t->trc_reader_nesting < 0)) {
835 if (WARN_ON_ONCE(atomic_dec_and_test(&trc_n_readers_need_end)))
836 wake_up(&trc_wait);
837 goto reset_ipi;
838 }
824 WRITE_ONCE(t->trc_reader_checked, true);839 WRITE_ONCE(t->trc_reader_checked, true);
825 840
826 // Get here if the task is in a read-side critical section. Set841 // Get here if the task is in a read-side critical section. Set
@@ -911,7 +926,8 @@ static void trc_wait_for_one_reader(struct task_struct *t, · 84 unmodified lines
911 926
912 // If currently running, send an IPI, either way, add to list.927 // If currently running, send an IPI, either way, add to list.
913 trc_add_holdout(t, bhp);928 trc_add_holdout(t, bhp);
914 if (task_curr(t) && time_after(jiffies, rcu_tasks_trace.gp_start + rcu_task_ipi_delay)) {929 if (task_curr(t) &&
930 time_after(jiffies + 1, rcu_tasks_trace.gp_start + rcu_task_ipi_delay)) {
915 // The task is currently running, so try IPIing it.931 // The task is currently running, so try IPIing it.
916 cpu = task_cpu(t);932 cpu = task_cpu(t);
917 933
@@ -1072,15 +1088,17 @@ static void rcu_tasks_trace_postgp(struct rcu_tasks *rtp) · 154 unmodified lines
1072 if (ret)1088 if (ret)
1073 break; // Count reached zero.1089 break; // Count reached zero.
1074 // Stall warning time, so make a list of the offenders.1090 // Stall warning time, so make a list of the offenders.
1091 rcu_read_lock();
1075 for_each_process_thread(g, t)1092 for_each_process_thread(g, t)
1076 if (READ_ONCE(t->trc_reader_special.b.need_qs))1093 if (READ_ONCE(t->trc_reader_special.b.need_qs))
1077 trc_add_holdout(t, &holdouts);1094 trc_add_holdout(t, &holdouts);
1095 rcu_read_unlock();
1078 firstreport = true;1096 firstreport = true;
1079 list_for_each_entry_safe(t, g, &holdouts, trc_holdout_list)1097 list_for_each_entry_safe(t, g, &holdouts, trc_holdout_list) {
1080 if (READ_ONCE(t->trc_reader_special.b.need_qs)) {1098 if (READ_ONCE(t->trc_reader_special.b.need_qs))
1081 show_stalled_task_trace(t, &firstreport);1099 show_stalled_task_trace(t, &firstreport);
1082 trc_del_holdout(t);1100 trc_del_holdout(t); // Release task_struct reference.
1083 }1101 }
1084 if (firstreport)1102 if (firstreport)
1085 pr_err("INFO: rcu_tasks_trace detected stalls? (Counter/taskslist mismatch?)\n");1103 pr_err("INFO: rcu_tasks_trace detected stalls? (Counter/taskslist mismatch?)\n");
1086 show_stalled_ipi_trace();1104 show_stalled_ipi_trace();
@@ -1163,6 +1181,17 @@ EXPORT_SYMBOL_GPL(rcu_barrier_tasks_trace); · 76 unmodified lines
1163 1181
1164static int __init rcu_spawn_tasks_trace_kthread(void)1182static int __init rcu_spawn_tasks_trace_kthread(void)
1165{1183{
1184 if (IS_ENABLED(CONFIG_TASKS_TRACE_RCU_READ_MB)) {
1185 rcu_tasks_trace.gp_sleep = HZ / 10;
1186 rcu_tasks_trace.init_fract = 10;
1187 } else {
1188 rcu_tasks_trace.gp_sleep = HZ / 200;
1189 if (rcu_tasks_trace.gp_sleep <= 0)
1190 rcu_tasks_trace.gp_sleep = 1;
1191 rcu_tasks_trace.init_fract = HZ / 5;
1192 if (rcu_tasks_trace.init_fract <= 0)
1193 rcu_tasks_trace.init_fract = 1;
1194 }
1166 rcu_tasks_trace.pregp_func = rcu_tasks_trace_pregp_step;1195 rcu_tasks_trace.pregp_func = rcu_tasks_trace_pregp_step;
1167 rcu_tasks_trace.pertask_func = rcu_tasks_trace_pertask;1196 rcu_tasks_trace.pertask_func = rcu_tasks_trace_pertask;
1168 rcu_tasks_trace.postscan_func = rcu_tasks_trace_postscan;1197 rcu_tasks_trace.postscan_func = rcu_tasks_trace_postscan;
kernel/taskstats.c+7−33
@@ -34,17 +34,13 @@ struct kmem_cache *taskstats_cache; · 33 unmodified lines
34 34
35static struct genl_family family;35static struct genl_family family;
36 36
37static const struct nla_policy taskstats_cmd_get_policy[TASKSTATS_CMD_ATTR_MAX+1] = {37static const struct nla_policy taskstats_cmd_get_policy[] = {
38 [TASKSTATS_CMD_ATTR_PID] = { .type = NLA_U32 },38 [TASKSTATS_CMD_ATTR_PID] = { .type = NLA_U32 },
39 [TASKSTATS_CMD_ATTR_TGID] = { .type = NLA_U32 },39 [TASKSTATS_CMD_ATTR_TGID] = { .type = NLA_U32 },
40 [TASKSTATS_CMD_ATTR_REGISTER_CPUMASK] = { .type = NLA_STRING },40 [TASKSTATS_CMD_ATTR_REGISTER_CPUMASK] = { .type = NLA_STRING },
41 [TASKSTATS_CMD_ATTR_DEREGISTER_CPUMASK] = { .type = NLA_STRING },};41 [TASKSTATS_CMD_ATTR_DEREGISTER_CPUMASK] = { .type = NLA_STRING },};
42 42
43/*43static const struct nla_policy cgroupstats_cmd_get_policy[] = {
44 * We have to use TASKSTATS_CMD_ATTR_MAX here, it is the maxattr in the family.
45 * Make sure they are always aligned.
46 */
47static const struct nla_policy cgroupstats_cmd_get_policy[TASKSTATS_CMD_ATTR_MAX+1] = {
48 [CGROUPSTATS_CMD_ATTR_FD] = { .type = NLA_U32 },44 [CGROUPSTATS_CMD_ATTR_FD] = { .type = NLA_U32 },
49};45};
50 46
@@ -649,47 +645,25 @@ static const struct genl_ops taskstats_ops[] = { · 598 unmodified lines
649 .cmd = TASKSTATS_CMD_GET,645 .cmd = TASKSTATS_CMD_GET,
650 .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP,646 .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP,
651 .doit = taskstats_user_cmd,647 .doit = taskstats_user_cmd,
652 /* policy enforced later */648 .policy = taskstats_cmd_get_policy,
653 .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_HASPOL,649 .maxattr = ARRAY_SIZE(taskstats_cmd_get_policy) - 1,
650 .flags = GENL_ADMIN_PERM,
654 },651 },
655 {652 {
656 .cmd = CGROUPSTATS_CMD_GET,653 .cmd = CGROUPSTATS_CMD_GET,
657 .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP,654 .validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP,
658 .doit = cgroupstats_user_cmd,655 .doit = cgroupstats_user_cmd,
659 /* policy enforced later */656 .policy = cgroupstats_cmd_get_policy,
660 .flags = GENL_CMD_CAP_HASPOL,657 .maxattr = ARRAY_SIZE(cgroupstats_cmd_get_policy) - 1,
661 },658 },
662};659};
663 660
664static int taskstats_pre_doit(const struct genl_ops *ops, struct sk_buff *skb,
665 struct genl_info *info)
666{
667 const struct nla_policy *policy = NULL;
668
669 switch (ops->cmd) {
670 case TASKSTATS_CMD_GET:
671 policy = taskstats_cmd_get_policy;
672 break;
673 case CGROUPSTATS_CMD_GET:
674 policy = cgroupstats_cmd_get_policy;
675 break;
676 default:
677 return -EINVAL;
678 }
679
680 return nlmsg_validate_deprecated(info->nlhdr, GENL_HDRLEN,
681 TASKSTATS_CMD_ATTR_MAX, policy,
682 info->extack);
683}
684
685static struct genl_family family __ro_after_init = {661static struct genl_family family __ro_after_init = {
686 .name = TASKSTATS_GENL_NAME,662 .name = TASKSTATS_GENL_NAME,
687 .version = TASKSTATS_GENL_VERSION,663 .version = TASKSTATS_GENL_VERSION,
688 .maxattr = TASKSTATS_CMD_ATTR_MAX,
689 .module = THIS_MODULE,664 .module = THIS_MODULE,
690 .ops = taskstats_ops,665 .ops = taskstats_ops,
691 .n_ops = ARRAY_SIZE(taskstats_ops),666 .n_ops = ARRAY_SIZE(taskstats_ops),
692 .pre_doit = taskstats_pre_doit,
693};667};
694 668
695/* Needed early in initialization */669/* Needed early in initialization */
kernel/trace/bpf_trace.c+165−7
@@ -7,6 +7,7 @@ · 6 unmodified lines
7#include <linux/slab.h>7#include <linux/slab.h>
8#include <linux/bpf.h>8#include <linux/bpf.h>
9#include <linux/bpf_perf_event.h>9#include <linux/bpf_perf_event.h>
10#include <linux/btf.h>
10#include <linux/filter.h>11#include <linux/filter.h>
11#include <linux/uaccess.h>12#include <linux/uaccess.h>
12#include <linux/ctype.h>13#include <linux/ctype.h>
@@ -16,6 +17,9 @@ · 3 unmodified lines
16#include <linux/error-injection.h>17#include <linux/error-injection.h>
17#include <linux/btf_ids.h>18#include <linux/btf_ids.h>
18 19
20#include <uapi/linux/bpf.h>
21#include <uapi/linux/btf.h>
22
19#include <asm/tlb.h>23#include <asm/tlb.h>
20 24
21#include "trace_probe.h"25#include "trace_probe.h"
@@ -67,6 +71,10 @@ static struct bpf_raw_event_map *bpf_get_raw_tracepoint_module(const char *name) · 45 unmodified lines
67u64 bpf_get_stackid(u64 r1, u64 r2, u64 r3, u64 r4, u64 r5);71u64 bpf_get_stackid(u64 r1, u64 r2, u64 r3, u64 r4, u64 r5);
68u64 bpf_get_stack(u64 r1, u64 r2, u64 r3, u64 r4, u64 r5);72u64 bpf_get_stack(u64 r1, u64 r2, u64 r3, u64 r4, u64 r5);
69 73
74static int bpf_btf_printf_prepare(struct btf_ptr *ptr, u32 btf_ptr_size,
75 u64 flags, const struct btf **btf,
76 s32 *btf_id);
77
70/**78/**
71 * trace_call_bpf - invoke BPF program79 * trace_call_bpf - invoke BPF program
72 * @call: tracepoint event80 * @call: tracepoint event
@@ -743,19 +751,18 @@ out: · 670 unmodified lines
743 return err;751 return err;
744}752}
745 753
746BTF_ID_LIST(bpf_seq_printf_btf_ids)754BTF_ID_LIST_SINGLE(btf_seq_file_ids, struct, seq_file)
747BTF_ID(struct, seq_file)
748 755
749static const struct bpf_func_proto bpf_seq_printf_proto = {756static const struct bpf_func_proto bpf_seq_printf_proto = {
750 .func = bpf_seq_printf,757 .func = bpf_seq_printf,
751 .gpl_only = true,758 .gpl_only = true,
752 .ret_type = RET_INTEGER,759 .ret_type = RET_INTEGER,
753 .arg1_type = ARG_PTR_TO_BTF_ID,760 .arg1_type = ARG_PTR_TO_BTF_ID,
761 .arg1_btf_id = &btf_seq_file_ids[0],
754 .arg2_type = ARG_PTR_TO_MEM,762 .arg2_type = ARG_PTR_TO_MEM,
755 .arg3_type = ARG_CONST_SIZE,763 .arg3_type = ARG_CONST_SIZE,
756 .arg4_type = ARG_PTR_TO_MEM_OR_NULL,764 .arg4_type = ARG_PTR_TO_MEM_OR_NULL,
757 .arg5_type = ARG_CONST_SIZE_OR_ZERO,765 .arg5_type = ARG_CONST_SIZE_OR_ZERO,
758 .btf_id = bpf_seq_printf_btf_ids,
759};766};
760 767
761BPF_CALL_3(bpf_seq_write, struct seq_file *, m, const void *, data, u32, len)768BPF_CALL_3(bpf_seq_write, struct seq_file *, m, const void *, data, u32, len)
@@ -763,17 +770,39 @@ BPF_CALL_3(bpf_seq_write, struct seq_file *, m, const void *, data, u32, len) · 1 unmodified line
763 return seq_write(m, data, len) ? -EOVERFLOW : 0;770 return seq_write(m, data, len) ? -EOVERFLOW : 0;
764}771}
765 772
766BTF_ID_LIST(bpf_seq_write_btf_ids)
767BTF_ID(struct, seq_file)
768
769static const struct bpf_func_proto bpf_seq_write_proto = {773static const struct bpf_func_proto bpf_seq_write_proto = {
770 .func = bpf_seq_write,774 .func = bpf_seq_write,
771 .gpl_only = true,775 .gpl_only = true,
772 .ret_type = RET_INTEGER,776 .ret_type = RET_INTEGER,
773 .arg1_type = ARG_PTR_TO_BTF_ID,777 .arg1_type = ARG_PTR_TO_BTF_ID,
778 .arg1_btf_id = &btf_seq_file_ids[0],
774 .arg2_type = ARG_PTR_TO_MEM,779 .arg2_type = ARG_PTR_TO_MEM,
775 .arg3_type = ARG_CONST_SIZE_OR_ZERO,780 .arg3_type = ARG_CONST_SIZE_OR_ZERO,
776 .btf_id = bpf_seq_write_btf_ids,781};
782
783BPF_CALL_4(bpf_seq_printf_btf, struct seq_file *, m, struct btf_ptr *, ptr,
784 u32, btf_ptr_size, u64, flags)
785{
786 const struct btf *btf;
787 s32 btf_id;
788 int ret;
789
790 ret = bpf_btf_printf_prepare(ptr, btf_ptr_size, flags, &btf, &btf_id);
791 if (ret)
792 return ret;
793
794 return btf_type_seq_show_flags(btf, btf_id, ptr->ptr, m, flags);
795}
796
797static const struct bpf_func_proto bpf_seq_printf_btf_proto = {
798 .func = bpf_seq_printf_btf,
799 .gpl_only = true,
800 .ret_type = RET_INTEGER,
801 .arg1_type = ARG_PTR_TO_BTF_ID,
802 .arg1_btf_id = &btf_seq_file_ids[0],
803 .arg2_type = ARG_PTR_TO_MEM,
804 .arg3_type = ARG_CONST_SIZE_OR_ZERO,
805 .arg4_type = ARG_ANYTHING,
777};806};
778 807
779static __always_inline int808static __always_inline int
@@ -1098,6 +1127,118 @@ static const struct bpf_func_proto bpf_send_signal_thread_proto = { · 318 unmodified lines
1098 .arg1_type = ARG_ANYTHING,1127 .arg1_type = ARG_ANYTHING,
1099};1128};
1100 1129
1130BPF_CALL_3(bpf_d_path, struct path *, path, char *, buf, u32, sz)
1131{
1132 long len;
1133 char *p;
1134
1135 if (!sz)
1136 return 0;
1137
1138 p = d_path(path, buf, sz);
1139 if (IS_ERR(p)) {
1140 len = PTR_ERR(p);
1141 } else {
1142 len = buf + sz - p;
1143 memmove(buf, p, len);
1144 }
1145
1146 return len;
1147}
1148
1149BTF_SET_START(btf_allowlist_d_path)
1150#ifdef CONFIG_SECURITY
1151BTF_ID(func, security_file_permission)
1152BTF_ID(func, security_inode_getattr)
1153BTF_ID(func, security_file_open)
1154#endif
1155#ifdef CONFIG_SECURITY_PATH
1156BTF_ID(func, security_path_truncate)
1157#endif
1158BTF_ID(func, vfs_truncate)
1159BTF_ID(func, vfs_fallocate)
1160BTF_ID(func, dentry_open)
1161BTF_ID(func, vfs_getattr)
1162BTF_ID(func, filp_close)
1163BTF_SET_END(btf_allowlist_d_path)
1164
1165static bool bpf_d_path_allowed(const struct bpf_prog *prog)
1166{
1167 return btf_id_set_contains(&btf_allowlist_d_path, prog->aux->attach_btf_id);
1168}
1169
1170BTF_ID_LIST_SINGLE(bpf_d_path_btf_ids, struct, path)
1171
1172static const struct bpf_func_proto bpf_d_path_proto = {
1173 .func = bpf_d_path,
1174 .gpl_only = false,
1175 .ret_type = RET_INTEGER,
1176 .arg1_type = ARG_PTR_TO_BTF_ID,
1177 .arg1_btf_id = &bpf_d_path_btf_ids[0],
1178 .arg2_type = ARG_PTR_TO_MEM,
1179 .arg3_type = ARG_CONST_SIZE_OR_ZERO,
1180 .allowed = bpf_d_path_allowed,
1181};
1182
1183#define BTF_F_ALL (BTF_F_COMPACT | BTF_F_NONAME | \
1184 BTF_F_PTR_RAW | BTF_F_ZERO)
1185
1186static int bpf_btf_printf_prepare(struct btf_ptr *ptr, u32 btf_ptr_size,
1187 u64 flags, const struct btf **btf,
1188 s32 *btf_id)
1189{
1190 const struct btf_type *t;
1191
1192 if (unlikely(flags & ~(BTF_F_ALL)))
1193 return -EINVAL;
1194
1195 if (btf_ptr_size != sizeof(struct btf_ptr))
1196 return -EINVAL;
1197
1198 *btf = bpf_get_btf_vmlinux();
1199
1200 if (IS_ERR_OR_NULL(*btf))
1201 return PTR_ERR(*btf);
1202
1203 if (ptr->type_id > 0)
1204 *btf_id = ptr->type_id;
1205 else
1206 return -EINVAL;
1207
1208 if (*btf_id > 0)
1209 t = btf_type_by_id(*btf, *btf_id);
1210 if (*btf_id <= 0 || !t)
1211 return -ENOENT;
1212
1213 return 0;
1214}
1215
1216BPF_CALL_5(bpf_snprintf_btf, char *, str, u32, str_size, struct btf_ptr *, ptr,
1217 u32, btf_ptr_size, u64, flags)
1218{
1219 const struct btf *btf;
1220 s32 btf_id;
1221 int ret;
1222
1223 ret = bpf_btf_printf_prepare(ptr, btf_ptr_size, flags, &btf, &btf_id);
1224 if (ret)
1225 return ret;
1226
1227 return btf_type_snprintf_show(btf, btf_id, ptr->ptr, str, str_size,
1228 flags);
1229}
1230
1231const struct bpf_func_proto bpf_snprintf_btf_proto = {
1232 .func = bpf_snprintf_btf,
1233 .gpl_only = false,
1234 .ret_type = RET_INTEGER,
1235 .arg1_type = ARG_PTR_TO_MEM,
1236 .arg2_type = ARG_CONST_SIZE,
1237 .arg3_type = ARG_PTR_TO_MEM,
1238 .arg4_type = ARG_CONST_SIZE,
1239 .arg5_type = ARG_ANYTHING,
1240};
1241
1101const struct bpf_func_proto *1242const struct bpf_func_proto *
1102bpf_tracing_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)1243bpf_tracing_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog)
1103{1244{
@@ -1182,6 +1323,14 @@ bpf_tracing_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) · 78 unmodified lines
1182 return &bpf_jiffies64_proto;1323 return &bpf_jiffies64_proto;
1183 case BPF_FUNC_get_task_stack:1324 case BPF_FUNC_get_task_stack:
1184 return &bpf_get_task_stack_proto;1325 return &bpf_get_task_stack_proto;
1326 case BPF_FUNC_copy_from_user:
1327 return prog->aux->sleepable ? &bpf_copy_from_user_proto : NULL;
1328 case BPF_FUNC_snprintf_btf:
1329 return &bpf_snprintf_btf_proto;
1330 case BPF_FUNC_bpf_per_cpu_ptr:
1331 return &bpf_per_cpu_ptr_proto;
1332 case BPF_FUNC_bpf_this_cpu_ptr:
1333 return &bpf_this_cpu_ptr_proto;
1185 default:1334 default:
1186 return NULL;1335 return NULL;
1187 }1336 }
@@ -1579,6 +1728,12 @@ tracing_prog_func_proto(enum bpf_func_id func_id, const struct bpf_prog *prog) · 391 unmodified lines
1579 return prog->expected_attach_type == BPF_TRACE_ITER ?1728 return prog->expected_attach_type == BPF_TRACE_ITER ?
1580 &bpf_seq_write_proto :1729 &bpf_seq_write_proto :
1581 NULL;1730 NULL;
1731 case BPF_FUNC_seq_printf_btf:
1732 return prog->expected_attach_type == BPF_TRACE_ITER ?
1733 &bpf_seq_printf_btf_proto :
1734 NULL;
1735 case BPF_FUNC_d_path:
1736 return &bpf_d_path_proto;
1582 default:1737 default:
1583 return raw_tp_prog_func_proto(func_id, prog);1738 return raw_tp_prog_func_proto(func_id, prog);
1584 }1739 }
@@ -1625,6 +1780,9 @@ const struct bpf_verifier_ops raw_tracepoint_verifier_ops = { · 40 unmodified lines
1625};1780};
1626 1781
1627const struct bpf_prog_ops raw_tracepoint_prog_ops = {1782const struct bpf_prog_ops raw_tracepoint_prog_ops = {
1783#ifdef CONFIG_NET
1784 .test_run = bpf_prog_test_run_raw_tp,
1785#endif
1628};1786};
1629 1787
1630const struct bpf_verifier_ops tracing_verifier_ops = {1788const struct bpf_verifier_ops tracing_verifier_ops = {

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t