Add an efficient ingress to ingress netns switch that can be used out of tc BPF programs in order to redirect traffic from host ns ingress into a container veth device ingress without having to go via CPU backlog queue [0]. For local containers this can also be utilized and path via CPU backlog queue only needs to be taken once, not twice. On a high level this borrows from ipvlan which does similar switch in __netif_receive_skb_core() and then iterates via another_round. This helps to reduce latency for mentioned use cases. Pod to remote pod with redirect(), TCP_RR [1]: # percpu_netperf 10.217.1.33 RT_LATENCY: 122.450 (per CPU: 122.666 122.401 122.333 122.401 ) MEAN_LATENCY: 121.210 (per CPU: 121.100 121.260 121.320 121.160 ) STDDEV_LATENCY: 120.040 (per CPU: 119.420 119.910 125.460 115.370 ) MIN_LATENCY: 46.500 (per CPU: 47.000 47.000 47.000 45.000 ) P50_LATENCY: 118.500 (per CPU: 118.000 119.000 118.000 119.000 ) P90_LATENCY: 127.500 (per CPU: 127.000 128.000 127.000 128.000 ) P99_LATENCY: 130.750 (per CPU: 131.000 131.000 129.000 132.000 ) TRANSACTION_RATE: 32666.400 (per CPU: 8152.200 8169.842 8174.439 8169.897 ) Pod to remote pod with redirect_peer(), TCP_RR: # percpu_netperf 10.217.1.33 RT_LATENCY: 44.449 (per CPU: 43.767 43.127 45.279 45.622 ) MEAN_LATENCY: 45.065 (per CPU: 44.030 45.530 45.190 45.510 ) STDDEV_LATENCY: 84.823 (per CPU: 66.770 97.290 84.380 90.850 ) MIN_LATENCY: 33.500 (per CPU: 33.000 33.000 34.000 34.000 ) P50_LATENCY: 43.250 (per CPU: 43.000 43.000 43.000 44.000 ) P90_LATENCY: 46.750 (per CPU: 46.000 47.000 47.000 47.000 ) P99_LATENCY: 52.750 (per CPU: 51.000 54.000 53.000 53.000 ) TRANSACTION_RATE: 90039.500 (per CPU: 22848.186 23187.089 22085.077 21919.130 ) [0] https://linuxplumbersconf.org/event/7/contributions/674/attachments/568/1002/plumbers_2020_cilium_load_balancer.pdf [1] https://github.com/borkmann/netperf_scripts/blob/master/percpu_netperf
Showing the changes to include/uapi/linux (1 of 6 files). Show all
Unified · Split
| @@ -3719,6 +3719,22 @@ union bpf_attr { · 3718 unmodified lines | |||
| 3719 | * never return NULL. | 3719 | * never return NULL. |
| 3720 | * Return | 3720 | * Return |
| 3721 | * A pointer pointing to the kernel percpu variable on this cpu. | 3721 | * A pointer pointing to the kernel percpu variable on this cpu. |
| 3722 | * | ||
| 3723 | * long bpf_redirect_peer(u32 ifindex, u64 flags) | ||
| 3724 | * Description | ||
| 3725 | * Redirect the packet to another net device of index *ifindex*. | ||
| 3726 | * This helper is somewhat similar to **bpf_redirect**\ (), except | ||
| 3727 | * that the redirection happens to the *ifindex*' peer device and | ||
| 3728 | * the netns switch takes place from ingress to ingress without | ||
| 3729 | * going through the CPU's backlog queue. | ||
| 3730 | * | ||
| 3731 | * The *flags* argument is reserved and must be 0. The helper is | ||
| 3732 | * currently only supported for tc BPF program types at the ingress | ||
| 3733 | * hook and for veth device types. The peer device must reside in a | ||
| 3734 | * different network namespace. | ||
| 3735 | * Return | ||
| 3736 | * The helper returns **TC_ACT_REDIRECT** on success or | ||
| 3737 | * **TC_ACT_SHOT** on error. | ||
| 3722 | */ | 3738 | */ |
| 3723 | #define __BPF_FUNC_MAPPER(FN) \ | 3739 | #define __BPF_FUNC_MAPPER(FN) \ |
| 3724 | FN(unspec), \ | 3740 | FN(unspec), \ |
| @@ -3876,6 +3892,7 @@ union bpf_attr { · 151 unmodified lines | |||
| 3876 | FN(redirect_neigh), \ | 3892 | FN(redirect_neigh), \ |
| 3877 | FN(bpf_per_cpu_ptr), \ | 3893 | FN(bpf_per_cpu_ptr), \ |
| 3878 | FN(bpf_this_cpu_ptr), \ | 3894 | FN(bpf_this_cpu_ptr), \ |
| 3895 | FN(redirect_peer), \ | ||
| 3879 | /* */ | 3896 | /* */ |
| 3880 | 3897 | ||
| 3881 | /* integer value in 'imm' field of BPF_CALL instruction selects which helper | 3898 | /* integer value in 'imm' field of BPF_CALL instruction selects which helper |