== The problem == There is a use-case when all processes inside a cgroup should use one single IP address on a host that has multiple IP configured. Those processes should use the IP for both ingress and egress, for TCP and UDP traffic. So TCP/UDP servers should be bound to that IP to accept incoming connections on it, and TCP/UDP clients should make outgoing connections from that IP. It should not require changing application code since it's often not possible. Currently it's solved by intercepting glibc wrappers around syscalls such as `bind(2)` and `connect(2)`. It's done by a shared library that is preloaded for every process in a cgroup so that whenever TCP/UDP server calls `bind(2)`, the library replaces IP in sockaddr before passing arguments to syscall. When application calls `connect(2)` the library transparently binds the local end of connection to that IP (`bind(2)` with `IP_BIND_ADDRESS_NO_PORT` to avoid performance penalty). Shared library approach is fragile though, e.g.: * some applications clear env vars (incl. `LD_PRELOAD`); * `/etc/ld.so.preload` doesn't help since some applications are linked with option `-z nodefaultlib`; * other applications don't use glibc and there is nothing to intercept. == The solution == The patch provides much more reliable in-kernel solution for the 1st part of the problem: binding TCP/UDP servers on desired IP. It does not depend on application environment and implementation details (whether glibc is used or not). It adds new eBPF program type `BPF_PROG_TYPE_CGROUP_SOCK_ADDR` and attach types `BPF_CGROUP_INET4_BIND` and `BPF_CGROUP_INET6_BIND` (similar to already existing `BPF_CGROUP_INET_SOCK_CREATE`). The new program type is intended to be used with sockets (`struct sock`) in a cgroup and provided by user `struct sockaddr`. Pointers to both of them are parts of the context passed to programs of newly added types. The new attach types provides hooks in `bind(2)` system call for both IPv4 and IPv6 so that one can write a program to override IP addresses and ports user program tries to bind to and apply such a program for whole cgroup. == Implementation notes == [1] Separate attach types for `AF_INET` and `AF_INET6` are added intentionally to prevent reading/writing to offsets that don't make sense for corresponding socket family. E.g. if user passes `sockaddr_in` it doesn't make sense to read from / write to `user_ip6[]` context fields. [2] The write access to `struct bpf_sock_addr_kern` is implemented using special field as an additional "register". There are just two registers in `sock_addr_convert_ctx_access`: `src` with value to write and `dst` with pointer to context that can't be changed not to break later instructions. But the fields, allowed to write to, are not available directly and to access them address of corresponding pointer has to be loaded first. To get additional register the 1st not used by `src` and `dst` one is taken, its content is saved to `bpf_sock_addr_kern.tmp_reg`, then the register is used to load address of pointer field, and finally the register's content is restored from the temporary field after writing `src` value.
Showing the changes to include/uapi/linux (1 of 10 files). Show all
Unified · Split
| @@ -136,6 +136,7 @@ enum bpf_prog_type { · 135 unmodified lines | |||
| 136 | BPF_PROG_TYPE_CGROUP_DEVICE, | 136 | BPF_PROG_TYPE_CGROUP_DEVICE, |
| 137 | BPF_PROG_TYPE_SK_MSG, | 137 | BPF_PROG_TYPE_SK_MSG, |
| 138 | BPF_PROG_TYPE_RAW_TRACEPOINT, | 138 | BPF_PROG_TYPE_RAW_TRACEPOINT, |
| 139 | BPF_PROG_TYPE_CGROUP_SOCK_ADDR, | ||
| 139 | }; | 140 | }; |
| 140 | 141 | ||
| 141 | enum bpf_attach_type { | 142 | enum bpf_attach_type { |
| @@ -147,6 +148,8 @@ enum bpf_attach_type { · 5 unmodified lines | |||
| 147 | BPF_SK_SKB_STREAM_VERDICT, | 148 | BPF_SK_SKB_STREAM_VERDICT, |
| 148 | BPF_CGROUP_DEVICE, | 149 | BPF_CGROUP_DEVICE, |
| 149 | BPF_SK_MSG_VERDICT, | 150 | BPF_SK_MSG_VERDICT, |
| 151 | BPF_CGROUP_INET4_BIND, | ||
| 152 | BPF_CGROUP_INET6_BIND, | ||
| 150 | __MAX_BPF_ATTACH_TYPE | 153 | __MAX_BPF_ATTACH_TYPE |
| 151 | }; | 154 | }; |
| 152 | 155 | ||
| @@ -1010,6 +1013,26 @@ struct bpf_map_info { · 857 unmodified lines | |||
| 1010 | __u64 netns_ino; | 1013 | __u64 netns_ino; |
| 1011 | } __attribute__((aligned(8))); | 1014 | } __attribute__((aligned(8))); |
| 1012 | 1015 | ||
| 1016 | /* User bpf_sock_addr struct to access socket fields and sockaddr struct passed | ||
| 1017 | * by user and intended to be used by socket (e.g. to bind to, depends on | ||
| 1018 | * attach attach type). | ||
| 1019 | */ | ||
| 1020 | struct bpf_sock_addr { | ||
| 1021 | __u32 user_family; /* Allows 4-byte read, but no write. */ | ||
| 1022 | __u32 user_ip4; /* Allows 1,2,4-byte read and 4-byte write. | ||
| 1023 | * Stored in network byte order. | ||
| 1024 | */ | ||
| 1025 | __u32 user_ip6[4]; /* Allows 1,2,4-byte read an 4-byte write. | ||
| 1026 | * Stored in network byte order. | ||
| 1027 | */ | ||
| 1028 | __u32 user_port; /* Allows 4-byte read and write. | ||
| 1029 | * Stored in network byte order | ||
| 1030 | */ | ||
| 1031 | __u32 family; /* Allows 4-byte read, but no write */ | ||
| 1032 | __u32 type; /* Allows 4-byte read, but no write */ | ||
| 1033 | __u32 protocol; /* Allows 4-byte read, but no write */ | ||
| 1034 | }; | ||
| 1035 | |||
| 1013 | /* User bpf_sock_ops struct to access socket values and specify request ops | 1036 | /* User bpf_sock_ops struct to access socket values and specify request ops |
| 1014 | * and their replies. | 1037 | * and their replies. |
| 1015 | * Some of this fields are in network (bigendian) byte order and may need | 1038 | * Some of this fields are in network (bigendian) byte order and may need |