Pull networking fixes from Jakub Kicinski: "Including fixes from CAN, netfilter, wireguard and IPsec. I'd like to highlight [ lowlight? - Linus ] Florian W stepping down as a netfilter maintainer due to constant stream of bug reports. Not sure what we can do but IIUC this is not the first such case. Current release - regressions: - rxrpc: fix use of page_frag_alloc_align(), it changed semantics and we added a new caller in a different subtree - xfrm: allow UDP encapsulation only in offload modes Current release - new code bugs: - tcp: fix refcnt handling in __inet_hash_connect() - Revert "net: Re-use and set mono_delivery_time bit for userspace tstamp packets", conflicted with some expectations in BPF uAPI Previous releases - regressions: - ipv4: raw: fix sending packets from raw sockets via IPsec tunnels - devlink: fix devlink's parallel command processing - veth: do not manipulate GRO when using XDP - esp: fix bad handling of pages from page_pool Previous releases - always broken: - report RCU QS for busy network kthreads (with Paul McK's blessing) - tcp/rds: fix use-after-free on netns with kernel TCP reqsk - virt: vmxnet3: fix missing reserved tailroom with XDP Misc: - couple of build fixes for Documentation" * tag 'net-6.9-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net: (59 commits) selftests: forwarding: Fix ping failure due to short timeout MAINTAINERS: step down as netfilter maintainer netfilter: nf_tables: Fix a memory leak in nf_tables_updchain net: dsa: mt7530: fix handling of all link-local frames net: dsa: mt7530: fix link-local frames that ingress vlan filtering ports bpf: report RCU QS in cpumap kthread net: report RCU QS on threaded NAPI repolling rcu: add a helper to report consolidated flavor QS ionic: update documentation for XDP support lib/bitmap: Fix bitmap_scatter() and bitmap_gather() kernel doc netfilter: nf_tables: do not compare internal table flags on updates netfilter: nft_set_pipapo: release elements in clone only from destroy path octeontx2-af: Use separate handlers for interrupts octeontx2-pf: Send UP messages to VF only when VF is up. octeontx2-pf: Use default max_active works instead of one octeontx2-pf: Wait till detach_resources msg is complete octeontx2: Detect the mbox up or down message via register devlink: fix port new reply cmd type tcp: Clear req->syncookie in reqsk_alloc(). net/bnx2x: Prevent access to a freed page in page_pool ...
Unified · Split
| @@ -99,6 +99,12 @@ Minimal SR-IOV support is currently offered and can be enabled by setting · 98 unmodified lines | |||
| 99 | the sysfs 'sriov_numvfs' value, if supported by your particular firmware | 99 | the sysfs 'sriov_numvfs' value, if supported by your particular firmware |
| 100 | configuration. | 100 | configuration. |
| 101 | 101 | ||
| 102 | XDP | ||
| 103 | --- | ||
| 104 | |||
| 105 | Support for XDP includes the basics, plus Jumbo frames, Redirect and | ||
| 106 | ndo_xmit. There is no current support for zero-copy sockets or HW offload. | ||
| 107 | |||
| 102 | Statistics | 108 | Statistics |
| 103 | ========== | 109 | ========== |
| 104 | 110 | ||
| @@ -138,6 +144,12 @@ Driver port specific:: · 33 unmodified lines | |||
| 138 | rx_csum_none: 0 | 144 | rx_csum_none: 0 |
| 139 | rx_csum_complete: 3 | 145 | rx_csum_complete: 3 |
| 140 | rx_csum_error: 0 | 146 | rx_csum_error: 0 |
| 147 | xdp_drop: 0 | ||
| 148 | xdp_aborted: 0 | ||
| 149 | xdp_pass: 0 | ||
| 150 | xdp_tx: 0 | ||
| 151 | xdp_redirect: 0 | ||
| 152 | xdp_frames: 0 | ||
| 141 | 153 | ||
| 142 | Driver queue specific:: | 154 | Driver queue specific:: |
| 143 | 155 | ||
| @@ -149,9 +161,12 @@ Driver queue specific:: · 5 unmodified lines | |||
| 149 | tx_0_frags: 0 | 161 | tx_0_frags: 0 |
| 150 | tx_0_tso: 0 | 162 | tx_0_tso: 0 |
| 151 | tx_0_tso_bytes: 0 | 163 | tx_0_tso_bytes: 0 |
| 164 | tx_0_hwstamp_valid: 0 | ||
| 165 | tx_0_hwstamp_invalid: 0 | ||
| 152 | tx_0_csum_none: 3 | 166 | tx_0_csum_none: 3 |
| 153 | tx_0_csum: 0 | 167 | tx_0_csum: 0 |
| 154 | tx_0_vlan_inserted: 0 | 168 | tx_0_vlan_inserted: 0 |
| 169 | tx_0_xdp_frames: 0 | ||
| 155 | rx_0_pkts: 2 | 170 | rx_0_pkts: 2 |
| 156 | rx_0_bytes: 120 | 171 | rx_0_bytes: 120 |
| 157 | rx_0_dma_map_err: 0 | 172 | rx_0_dma_map_err: 0 |
| @@ -159,8 +174,15 @@ Driver queue specific:: · 1 unmodified line | |||
| 159 | rx_0_csum_none: 0 | 174 | rx_0_csum_none: 0 |
| 160 | rx_0_csum_complete: 0 | 175 | rx_0_csum_complete: 0 |
| 161 | rx_0_csum_error: 0 | 176 | rx_0_csum_error: 0 |
| 177 | rx_0_hwstamp_valid: 0 | ||
| 178 | rx_0_hwstamp_invalid: 0 | ||
| 162 | rx_0_dropped: 0 | 179 | rx_0_dropped: 0 |
| 163 | rx_0_vlan_stripped: 0 | 180 | rx_0_vlan_stripped: 0 |
| 181 | rx_0_xdp_drop: 0 | ||
| 182 | rx_0_xdp_aborted: 0 | ||
| 183 | rx_0_xdp_pass: 0 | ||
| 184 | rx_0_xdp_tx: 0 | ||
| 185 | rx_0_xdp_redirect: 0 | ||
| 164 | 186 | ||
| 165 | Firmware port specific:: | 187 | Firmware port specific:: |
| 166 | 188 | ||
| @@ -87,35 +87,35 @@ all using the same instance under "priv->mdev". · 86 unmodified lines | |||
| 87 | 87 | ||
| 88 | Observability | 88 | Observability |
| 89 | ============= | 89 | ============= |
| 90 | The relation between PF, irq, napi, and queue can be observed via netlink spec: | 90 | The relation between PF, irq, napi, and queue can be observed via netlink spec:: |
| 91 | 91 | ||
| 92 | $ ./tools/net/ynl/cli.py --spec Documentation/netlink/specs/netdev.yaml --dump queue-get --json='{"ifindex": 13}' | 92 | $ ./tools/net/ynl/cli.py --spec Documentation/netlink/specs/netdev.yaml --dump queue-get --json='{"ifindex": 13}' |
| 93 | [{'id': 0, 'ifindex': 13, 'napi-id': 539, 'type': 'rx'}, | 93 | [{'id': 0, 'ifindex': 13, 'napi-id': 539, 'type': 'rx'}, |
| 94 | {'id': 1, 'ifindex': 13, 'napi-id': 540, 'type': 'rx'}, | 94 | {'id': 1, 'ifindex': 13, 'napi-id': 540, 'type': 'rx'}, |
| 95 | {'id': 2, 'ifindex': 13, 'napi-id': 541, 'type': 'rx'}, | 95 | {'id': 2, 'ifindex': 13, 'napi-id': 541, 'type': 'rx'}, |
| 96 | {'id': 3, 'ifindex': 13, 'napi-id': 542, 'type': 'rx'}, | 96 | {'id': 3, 'ifindex': 13, 'napi-id': 542, 'type': 'rx'}, |
| 97 | {'id': 4, 'ifindex': 13, 'napi-id': 543, 'type': 'rx'}, | 97 | {'id': 4, 'ifindex': 13, 'napi-id': 543, 'type': 'rx'}, |
| 98 | {'id': 0, 'ifindex': 13, 'napi-id': 539, 'type': 'tx'}, | 98 | {'id': 0, 'ifindex': 13, 'napi-id': 539, 'type': 'tx'}, |
| 99 | {'id': 1, 'ifindex': 13, 'napi-id': 540, 'type': 'tx'}, | 99 | {'id': 1, 'ifindex': 13, 'napi-id': 540, 'type': 'tx'}, |
| 100 | {'id': 2, 'ifindex': 13, 'napi-id': 541, 'type': 'tx'}, | 100 | {'id': 2, 'ifindex': 13, 'napi-id': 541, 'type': 'tx'}, |
| 101 | {'id': 3, 'ifindex': 13, 'napi-id': 542, 'type': 'tx'}, | 101 | {'id': 3, 'ifindex': 13, 'napi-id': 542, 'type': 'tx'}, |
| 102 | {'id': 4, 'ifindex': 13, 'napi-id': 543, 'type': 'tx'}] | 102 | {'id': 4, 'ifindex': 13, 'napi-id': 543, 'type': 'tx'}] |
| 103 | 103 | ||
| 104 | $ ./tools/net/ynl/cli.py --spec Documentation/netlink/specs/netdev.yaml --dump napi-get --json='{"ifindex": 13}' | 104 | $ ./tools/net/ynl/cli.py --spec Documentation/netlink/specs/netdev.yaml --dump napi-get --json='{"ifindex": 13}' |
| 105 | [{'id': 543, 'ifindex': 13, 'irq': 42}, | 105 | [{'id': 543, 'ifindex': 13, 'irq': 42}, |
| 106 | {'id': 542, 'ifindex': 13, 'irq': 41}, | 106 | {'id': 542, 'ifindex': 13, 'irq': 41}, |
| 107 | {'id': 541, 'ifindex': 13, 'irq': 40}, | 107 | {'id': 541, 'ifindex': 13, 'irq': 40}, |
| 108 | {'id': 540, 'ifindex': 13, 'irq': 39}, | 108 | {'id': 540, 'ifindex': 13, 'irq': 39}, |
| 109 | {'id': 539, 'ifindex': 13, 'irq': 36}] | 109 | {'id': 539, 'ifindex': 13, 'irq': 36}] |
| 110 | 110 | ||
| 111 | Here you can clearly observe our channels distribution policy: | 111 | Here you can clearly observe our channels distribution policy:: |
| 112 | 112 | ||
| 113 | $ ls /proc/irq/{36,39,40,41,42}/mlx5* -d -1 | 113 | $ ls /proc/irq/{36,39,40,41,42}/mlx5* -d -1 |
| 114 | /proc/irq/36/mlx5_comp1@pci:0000:08:00.0 | 114 | /proc/irq/36/mlx5_comp1@pci:0000:08:00.0 |
| 115 | /proc/irq/39/mlx5_comp1@pci:0000:09:00.0 | 115 | /proc/irq/39/mlx5_comp1@pci:0000:09:00.0 |
| 116 | /proc/irq/40/mlx5_comp2@pci:0000:08:00.0 | 116 | /proc/irq/40/mlx5_comp2@pci:0000:08:00.0 |
| 117 | /proc/irq/41/mlx5_comp2@pci:0000:09:00.0 | 117 | /proc/irq/41/mlx5_comp2@pci:0000:09:00.0 |
| 118 | /proc/irq/42/mlx5_comp3@pci:0000:08:00.0 | 118 | /proc/irq/42/mlx5_comp3@pci:0000:08:00.0 |
| 119 | 119 | ||
| 120 | Steering | 120 | Steering |
| 121 | ======== | 121 | ======== |
| @@ -13,7 +13,7 @@ struct_dev_ifalias* ifalias · 12 unmodified lines | |||
| 13 | unsigned_long mem_end | 13 | unsigned_long mem_end |
| 14 | unsigned_long mem_start | 14 | unsigned_long mem_start |
| 15 | unsigned_long base_addr | 15 | unsigned_long base_addr |
| 16 | unsigned_long state | 16 | unsigned_long state read_mostly read_mostly netif_running(dev) |
| 17 | struct_list_head dev_list | 17 | struct_list_head dev_list |
| 18 | struct_list_head napi_list | 18 | struct_list_head napi_list |
| 19 | struct_list_head unreg_list | 19 | struct_list_head unreg_list |
| @@ -15237,7 +15237,6 @@ F: drivers/net/ethernet/neterion/ · 15236 unmodified lines | |||
| 15237 | NETFILTER | 15237 | NETFILTER |
| 15238 | M: Pablo Neira Ayuso <pablo@netfilter.org> | 15238 | M: Pablo Neira Ayuso <pablo@netfilter.org> |
| 15239 | M: Jozsef Kadlecsik <kadlec@netfilter.org> | 15239 | M: Jozsef Kadlecsik <kadlec@netfilter.org> |
| 15240 | M: Florian Westphal <fw@strlen.de> | ||
| 15241 | L: netfilter-devel@vger.kernel.org | 15240 | L: netfilter-devel@vger.kernel.org |
| 15242 | L: coreteam@netfilter.org | 15241 | L: coreteam@netfilter.org |
| 15243 | S: Maintained | 15242 | S: Maintained |
| @@ -370,8 +370,8 @@ static const struct kvaser_pciefd_irq_mask kvaser_pciefd_sf2_irq_mask = { · 369 unmodified lines | |||
| 370 | 370 | ||
| 371 | static const struct kvaser_pciefd_irq_mask kvaser_pciefd_xilinx_irq_mask = { | 371 | static const struct kvaser_pciefd_irq_mask kvaser_pciefd_xilinx_irq_mask = { |
| 372 | .kcan_rx0 = BIT(4), | 372 | .kcan_rx0 = BIT(4), |
| 373 | .kcan_tx = { BIT(16), BIT(17), BIT(18), BIT(19) }, | 373 | .kcan_tx = { BIT(16), BIT(17), BIT(18), BIT(19), BIT(20), BIT(21), BIT(22), BIT(23) }, |
| 374 | .all = GENMASK(19, 16) | BIT(4), | 374 | .all = GENMASK(23, 16) | BIT(4), |
| 375 | }; | 375 | }; |
| 376 | 376 | ||
| 377 | static const struct kvaser_pciefd_dev_ops kvaser_pciefd_altera_dev_ops = { | 377 | static const struct kvaser_pciefd_dev_ops kvaser_pciefd_altera_dev_ops = { |
| @@ -950,20 +950,56 @@ static void mt7530_setup_port5(struct dsa_switch *ds, phy_interface_t interface) · 949 unmodified lines | |||
| 950 | mutex_unlock(&priv->reg_mutex); | 950 | mutex_unlock(&priv->reg_mutex); |
| 951 | } | 951 | } |
| 952 | 952 | ||
| 953 | /* On page 205, section "8.6.3 Frame filtering" of the active standard, IEEE Std | ||
| 954 | * 802.1Q™-2022, it is stated that frames with 01:80:C2:00:00:00-0F as MAC DA | ||
| 955 | * must only be propagated to C-VLAN and MAC Bridge components. That means | ||
| 956 | * VLAN-aware and VLAN-unaware bridges. On the switch designs with CPU ports, | ||
| 957 | * these frames are supposed to be processed by the CPU (software). So we make | ||
| 958 | * the switch only forward them to the CPU port. And if received from a CPU | ||
| 959 | * port, forward to a single port. The software is responsible of making the | ||
| 960 | * switch conform to the latter by setting a single port as destination port on | ||
| 961 | * the special tag. | ||
| 962 | * | ||
| 963 | * This switch intellectual property cannot conform to this part of the standard | ||
| 964 | * fully. Whilst the REV_UN frame tag covers the remaining :04-0D and :0F MAC | ||
| 965 | * DAs, it also includes :22-FF which the scope of propagation is not supposed | ||
| 966 | * to be restricted for these MAC DAs. | ||
| 967 | */ | ||
| 953 | static void | 968 | static void |
| 954 | mt753x_trap_frames(struct mt7530_priv *priv) | 969 | mt753x_trap_frames(struct mt7530_priv *priv) |
| 955 | { | 970 | { |
| 956 | /* Trap BPDUs to the CPU port(s) */ | 971 | /* Trap 802.1X PAE frames and BPDUs to the CPU port(s) and egress them |
| 957 | mt7530_rmw(priv, MT753X_BPC, MT753X_BPDU_PORT_FW_MASK, | 972 | * VLAN-untagged. |
| 973 | */ | ||
| 974 | mt7530_rmw(priv, MT753X_BPC, MT753X_PAE_EG_TAG_MASK | | ||
| 975 | MT753X_PAE_PORT_FW_MASK | MT753X_BPDU_EG_TAG_MASK | | ||
| 976 | MT753X_BPDU_PORT_FW_MASK, | ||
| 977 | MT753X_PAE_EG_TAG(MT7530_VLAN_EG_UNTAGGED) | | ||
| 978 | MT753X_PAE_PORT_FW(MT753X_BPDU_CPU_ONLY) | | ||
| 979 | MT753X_BPDU_EG_TAG(MT7530_VLAN_EG_UNTAGGED) | | ||
| 958 | MT753X_BPDU_CPU_ONLY); | 980 | MT753X_BPDU_CPU_ONLY); |
| 959 | 981 | ||
| 960 | /* Trap 802.1X PAE frames to the CPU port(s) */ | 982 | /* Trap frames with :01 and :02 MAC DAs to the CPU port(s) and egress |
| 961 | mt7530_rmw(priv, MT753X_BPC, MT753X_PAE_PORT_FW_MASK, | 983 | * them VLAN-untagged. |
| 962 | MT753X_PAE_PORT_FW(MT753X_BPDU_CPU_ONLY)); | 984 | */ |
| 985 | mt7530_rmw(priv, MT753X_RGAC1, MT753X_R02_EG_TAG_MASK | | ||
| 986 | MT753X_R02_PORT_FW_MASK | MT753X_R01_EG_TAG_MASK | | ||
| 987 | MT753X_R01_PORT_FW_MASK, | ||
| 988 | MT753X_R02_EG_TAG(MT7530_VLAN_EG_UNTAGGED) | | ||
| 989 | MT753X_R02_PORT_FW(MT753X_BPDU_CPU_ONLY) | | ||
| 990 | MT753X_R01_EG_TAG(MT7530_VLAN_EG_UNTAGGED) | | ||
| 991 | MT753X_BPDU_CPU_ONLY); | ||
| 963 | 992 | ||
| 964 | /* Trap LLDP frames with :0E MAC DA to the CPU port(s) */ | 993 | /* Trap frames with :03 and :0E MAC DAs to the CPU port(s) and egress |
| 965 | mt7530_rmw(priv, MT753X_RGAC2, MT753X_R0E_PORT_FW_MASK, | 994 | * them VLAN-untagged. |
| 966 | MT753X_R0E_PORT_FW(MT753X_BPDU_CPU_ONLY)); | 995 | */ |
| 996 | mt7530_rmw(priv, MT753X_RGAC2, MT753X_R0E_EG_TAG_MASK | | ||
| 997 | MT753X_R0E_PORT_FW_MASK | MT753X_R03_EG_TAG_MASK | | ||
| 998 | MT753X_R03_PORT_FW_MASK, | ||
| 999 | MT753X_R0E_EG_TAG(MT7530_VLAN_EG_UNTAGGED) | | ||
| 1000 | MT753X_R0E_PORT_FW(MT753X_BPDU_CPU_ONLY) | | ||
| 1001 | MT753X_R03_EG_TAG(MT7530_VLAN_EG_UNTAGGED) | | ||
| 1002 | MT753X_BPDU_CPU_ONLY); | ||
| 967 | } | 1003 | } |
| 968 | 1004 | ||
| 969 | static void | 1005 | static void |
| @@ -2192,22 +2228,16 @@ mt7530_setup(struct dsa_switch *ds) · 1222 unmodified lines | |||
| 2192 | } | 2228 | } |
| 2193 | } | 2229 | } |
| 2194 | 2230 | ||
| 2195 | /* Disable LEDs before reset to prevent the MT7530 sampling a | ||
| 2196 | * potentially incorrect HT_XTAL_FSEL value. | ||
| 2197 | */ | ||
| 2198 | mt7530_write(priv, MT7530_LED_EN, 0); | ||
| 2199 | usleep_range(1000, 1100); | ||
| 2200 | |||
| 2201 | /* Reset whole chip through gpio pin or memory-mapped registers for | 2231 | /* Reset whole chip through gpio pin or memory-mapped registers for |
| 2202 | * different type of hardware | 2232 | * different type of hardware |
| 2203 | */ | 2233 | */ |
| 2204 | if (priv->mcm) { | 2234 | if (priv->mcm) { |
| 2205 | reset_control_assert(priv->rstc); | 2235 | reset_control_assert(priv->rstc); |
| 2206 | usleep_range(1000, 1100); | 2236 | usleep_range(5000, 5100); |
| 2207 | reset_control_deassert(priv->rstc); | 2237 | reset_control_deassert(priv->rstc); |
| 2208 | } else { | 2238 | } else { |
| 2209 | gpiod_set_value_cansleep(priv->reset, 0); | 2239 | gpiod_set_value_cansleep(priv->reset, 0); |
| 2210 | usleep_range(1000, 1100); | 2240 | usleep_range(5000, 5100); |
| 2211 | gpiod_set_value_cansleep(priv->reset, 1); | 2241 | gpiod_set_value_cansleep(priv->reset, 1); |
| 2212 | } | 2242 | } |
| 2213 | 2243 | ||
| @@ -2420,11 +2450,11 @@ mt7531_setup(struct dsa_switch *ds) · 206 unmodified lines | |||
| 2420 | */ | 2450 | */ |
| 2421 | if (priv->mcm) { | 2451 | if (priv->mcm) { |
| 2422 | reset_control_assert(priv->rstc); | 2452 | reset_control_assert(priv->rstc); |
| 2423 | usleep_range(1000, 1100); | 2453 | usleep_range(5000, 5100); |
| 2424 | reset_control_deassert(priv->rstc); | 2454 | reset_control_deassert(priv->rstc); |
| 2425 | } else { | 2455 | } else { |
| 2426 | gpiod_set_value_cansleep(priv->reset, 0); | 2456 | gpiod_set_value_cansleep(priv->reset, 0); |
| 2427 | usleep_range(1000, 1100); | 2457 | usleep_range(5000, 5100); |
| 2428 | gpiod_set_value_cansleep(priv->reset, 1); | 2458 | gpiod_set_value_cansleep(priv->reset, 1); |
| 2429 | } | 2459 | } |
| 2430 | 2460 | ||
| @@ -65,14 +65,33 @@ enum mt753x_id { · 64 unmodified lines | |||
| 65 | 65 | ||
| 66 | /* Registers for BPDU and PAE frame control*/ | 66 | /* Registers for BPDU and PAE frame control*/ |
| 67 | #define MT753X_BPC 0x24 | 67 | #define MT753X_BPC 0x24 |
| 68 | #define MT753X_BPDU_PORT_FW_MASK GENMASK(2, 0) | 68 | #define MT753X_PAE_EG_TAG_MASK GENMASK(24, 22) |
| 69 | #define MT753X_PAE_EG_TAG(x) FIELD_PREP(MT753X_PAE_EG_TAG_MASK, x) | ||
| 69 | #define MT753X_PAE_PORT_FW_MASK GENMASK(18, 16) | 70 | #define MT753X_PAE_PORT_FW_MASK GENMASK(18, 16) |
| 70 | #define MT753X_PAE_PORT_FW(x) FIELD_PREP(MT753X_PAE_PORT_FW_MASK, x) | 71 | #define MT753X_PAE_PORT_FW(x) FIELD_PREP(MT753X_PAE_PORT_FW_MASK, x) |
| 72 | #define MT753X_BPDU_EG_TAG_MASK GENMASK(8, 6) | ||
| 73 | #define MT753X_BPDU_EG_TAG(x) FIELD_PREP(MT753X_BPDU_EG_TAG_MASK, x) | ||
| 74 | #define MT753X_BPDU_PORT_FW_MASK GENMASK(2, 0) | ||
| 75 | |||
| 76 | /* Register for :01 and :02 MAC DA frame control */ | ||
| 77 | #define MT753X_RGAC1 0x28 | ||
| 78 | #define MT753X_R02_EG_TAG_MASK GENMASK(24, 22) | ||
| 79 | #define MT753X_R02_EG_TAG(x) FIELD_PREP(MT753X_R02_EG_TAG_MASK, x) | ||
| 80 | #define MT753X_R02_PORT_FW_MASK GENMASK(18, 16) | ||
| 81 | #define MT753X_R02_PORT_FW(x) FIELD_PREP(MT753X_R02_PORT_FW_MASK, x) | ||
| 82 | #define MT753X_R01_EG_TAG_MASK GENMASK(8, 6) | ||
| 83 | #define MT753X_R01_EG_TAG(x) FIELD_PREP(MT753X_R01_EG_TAG_MASK, x) | ||
| 84 | #define MT753X_R01_PORT_FW_MASK GENMASK(2, 0) | ||
| 71 | 85 | ||
| 72 | /* Register for :03 and :0E MAC DA frame control */ | 86 | /* Register for :03 and :0E MAC DA frame control */ |
| 73 | #define MT753X_RGAC2 0x2c | 87 | #define MT753X_RGAC2 0x2c |
| 88 | #define MT753X_R0E_EG_TAG_MASK GENMASK(24, 22) | ||
| 89 | #define MT753X_R0E_EG_TAG(x) FIELD_PREP(MT753X_R0E_EG_TAG_MASK, x) | ||
| 74 | #define MT753X_R0E_PORT_FW_MASK GENMASK(18, 16) | 90 | #define MT753X_R0E_PORT_FW_MASK GENMASK(18, 16) |
| 75 | #define MT753X_R0E_PORT_FW(x) FIELD_PREP(MT753X_R0E_PORT_FW_MASK, x) | 91 | #define MT753X_R0E_PORT_FW(x) FIELD_PREP(MT753X_R0E_PORT_FW_MASK, x) |
| 92 | #define MT753X_R03_EG_TAG_MASK GENMASK(8, 6) | ||
| 93 | #define MT753X_R03_EG_TAG(x) FIELD_PREP(MT753X_R03_EG_TAG_MASK, x) | ||
| 94 | #define MT753X_R03_PORT_FW_MASK GENMASK(2, 0) | ||
| 76 | 95 | ||
| 77 | enum mt753x_bpdu_port_fw { | 96 | enum mt753x_bpdu_port_fw { |
| 78 | MT753X_BPDU_FOLLOW_MFC, | 97 | MT753X_BPDU_FOLLOW_MFC, |
| @@ -253,6 +272,7 @@ enum mt7530_port_mode { · 174 unmodified lines | |||
| 253 | enum mt7530_vlan_port_eg_tag { | 272 | enum mt7530_vlan_port_eg_tag { |
| 254 | MT7530_VLAN_EG_DISABLED = 0, | 273 | MT7530_VLAN_EG_DISABLED = 0, |
| 255 | MT7530_VLAN_EG_CONSISTENT = 1, | 274 | MT7530_VLAN_EG_CONSISTENT = 1, |
| 275 | MT7530_VLAN_EG_UNTAGGED = 4, | ||
| 256 | }; | 276 | }; |
| 257 | 277 | ||
| 258 | enum mt7530_vlan_port_attr { | 278 | enum mt7530_vlan_port_attr { |
| @@ -1002,9 +1002,6 @@ static inline void bnx2x_set_fw_mac_addr(__le16 *fw_hi, __le16 *fw_mid, · 1001 unmodified lines | |||
| 1002 | static inline void bnx2x_free_rx_mem_pool(struct bnx2x *bp, | 1002 | static inline void bnx2x_free_rx_mem_pool(struct bnx2x *bp, |
| 1003 | struct bnx2x_alloc_pool *pool) | 1003 | struct bnx2x_alloc_pool *pool) |
| 1004 | { | 1004 | { |
| 1005 | if (!pool->page) | ||
| 1006 | return; | ||
| 1007 | |||
| 1008 | put_page(pool->page); | 1005 | put_page(pool->page); |
| 1009 | 1006 | ||
| 1010 | pool->page = NULL; | 1007 | pool->page = NULL; |
| @@ -1015,6 +1012,9 @@ static inline void bnx2x_free_rx_sge_range(struct bnx2x *bp, · 4 unmodified lines | |||
| 1015 | { | 1012 | { |
| 1016 | int i; | 1013 | int i; |
| 1017 | 1014 | ||
| 1015 | if (!fp->page_pool.page) | ||
| 1016 | return; | ||
| 1017 | |||
| 1018 | if (fp->mode == TPA_MODE_DISABLED) | 1018 | if (fp->mode == TPA_MODE_DISABLED) |
| 1019 | return; | 1019 | return; |
| 1020 | 1020 | ||
| @@ -1338,7 +1338,7 @@ static irqreturn_t cgx_fwi_event_handler(int irq, void *data) · 1337 unmodified lines | |||
| 1338 | 1338 | ||
| 1339 | /* Release thread waiting for completion */ | 1339 | /* Release thread waiting for completion */ |
| 1340 | lmac->cmd_pend = false; | 1340 | lmac->cmd_pend = false; |
| 1341 | wake_up_interruptible(&lmac->wq_cmd_cmplt); | 1341 | wake_up(&lmac->wq_cmd_cmplt); |
| 1342 | break; | 1342 | break; |
| 1343 | case CGX_EVT_ASYNC: | 1343 | case CGX_EVT_ASYNC: |
| 1344 | if (cgx_event_is_linkevent(event)) | 1344 | if (cgx_event_is_linkevent(event)) |
| @@ -214,11 +214,12 @@ int otx2_mbox_busy_poll_for_rsp(struct otx2_mbox *mbox, int devid) · 213 unmodified lines | |||
| 214 | } | 214 | } |
| 215 | EXPORT_SYMBOL(otx2_mbox_busy_poll_for_rsp); | 215 | EXPORT_SYMBOL(otx2_mbox_busy_poll_for_rsp); |
| 216 | 216 | ||
| 217 | void otx2_mbox_msg_send(struct otx2_mbox *mbox, int devid) | 217 | static void otx2_mbox_msg_send_data(struct otx2_mbox *mbox, int devid, u64 data) |
| 218 | { | 218 | { |
| 219 | struct otx2_mbox_dev *mdev = &mbox->dev[devid]; | 219 | struct otx2_mbox_dev *mdev = &mbox->dev[devid]; |
| 220 | struct mbox_hdr *tx_hdr, *rx_hdr; | 220 | struct mbox_hdr *tx_hdr, *rx_hdr; |
| 221 | void *hw_mbase = mdev->hwbase; | 221 | void *hw_mbase = mdev->hwbase; |
| 222 | u64 intr_val; | ||
| 222 | 223 | ||
| 223 | tx_hdr = hw_mbase + mbox->tx_start; | 224 | tx_hdr = hw_mbase + mbox->tx_start; |
| 224 | rx_hdr = hw_mbase + mbox->rx_start; | 225 | rx_hdr = hw_mbase + mbox->rx_start; |
| @@ -254,14 +255,52 @@ void otx2_mbox_msg_send(struct otx2_mbox *mbox, int devid) · 29 unmodified lines | |||
| 254 | 255 | ||
| 255 | spin_unlock(&mdev->mbox_lock); | 256 | spin_unlock(&mdev->mbox_lock); |
| 256 | 257 | ||
| 258 | /* Check if interrupt pending */ | ||
| 259 | intr_val = readq((void __iomem *)mbox->reg_base + | ||
| 260 | (mbox->trigger | (devid << mbox->tr_shift))); | ||
| 261 | |||
| 262 | intr_val |= data; | ||
| 257 | /* The interrupt should be fired after num_msgs is written | 263 | /* The interrupt should be fired after num_msgs is written |
| 258 | * to the shared memory | 264 | * to the shared memory |
| 259 | */ | 265 | */ |
| 260 | writeq(1, (void __iomem *)mbox->reg_base + | 266 | writeq(intr_val, (void __iomem *)mbox->reg_base + |
| 261 | (mbox->trigger | (devid << mbox->tr_shift))); | 267 | (mbox->trigger | (devid << mbox->tr_shift))); |
| 262 | } | 268 | } |
| 269 | |||
| 270 | void otx2_mbox_msg_send(struct otx2_mbox *mbox, int devid) | ||
| 271 | { | ||
| 272 | otx2_mbox_msg_send_data(mbox, devid, MBOX_DOWN_MSG); | ||
| 273 | } | ||
| 263 | EXPORT_SYMBOL(otx2_mbox_msg_send); | 274 | EXPORT_SYMBOL(otx2_mbox_msg_send); |
| 264 | 275 | ||
| 276 | void otx2_mbox_msg_send_up(struct otx2_mbox *mbox, int devid) | ||
| 277 | { | ||
| 278 | otx2_mbox_msg_send_data(mbox, devid, MBOX_UP_MSG); | ||
| 279 | } | ||
| 280 | EXPORT_SYMBOL(otx2_mbox_msg_send_up); | ||
| 281 | |||
| 282 | bool otx2_mbox_wait_for_zero(struct otx2_mbox *mbox, int devid) | ||
| 283 | { | ||
| 284 | u64 data; | ||
| 285 | |||
| 286 | data = readq((void __iomem *)mbox->reg_base + | ||
| 287 | (mbox->trigger | (devid << mbox->tr_shift))); | ||
| 288 | |||
| 289 | /* If data is non-zero wait for ~1ms and return to caller | ||
| 290 | * whether data has changed to zero or not after the wait. | ||
| 291 | */ | ||
| 292 | if (!data) | ||
| 293 | return true; | ||
| 294 | |||
| 295 | usleep_range(950, 1000); | ||
| 296 | |||
| 297 | data = readq((void __iomem *)mbox->reg_base + | ||
| 298 | (mbox->trigger | (devid << mbox->tr_shift))); | ||
| 299 | |||
| 300 | return data == 0; | ||
| 301 | } | ||
| 302 | EXPORT_SYMBOL(otx2_mbox_wait_for_zero); | ||
| 303 | |||
| 265 | struct mbox_msghdr *otx2_mbox_alloc_msg_rsp(struct otx2_mbox *mbox, int devid, | 304 | struct mbox_msghdr *otx2_mbox_alloc_msg_rsp(struct otx2_mbox *mbox, int devid, |
| 266 | int size, int size_rsp) | 305 | int size, int size_rsp) |
| 267 | { | 306 | { |
| @@ -16,6 +16,9 @@ · 15 unmodified lines | |||
| 16 | 16 | ||
| 17 | #define MBOX_SIZE SZ_64K | 17 | #define MBOX_SIZE SZ_64K |
| 18 | 18 | ||
| 19 | #define MBOX_DOWN_MSG 1 | ||
| 20 | #define MBOX_UP_MSG 2 | ||
| 21 | |||
| 19 | /* AF/PF: PF initiated, PF/VF VF initiated */ | 22 | /* AF/PF: PF initiated, PF/VF VF initiated */ |
| 20 | #define MBOX_DOWN_RX_START 0 | 23 | #define MBOX_DOWN_RX_START 0 |
| 21 | #define MBOX_DOWN_RX_SIZE (46 * SZ_1K) | 24 | #define MBOX_DOWN_RX_SIZE (46 * SZ_1K) |
| @@ -101,6 +104,7 @@ int otx2_mbox_regions_init(struct otx2_mbox *mbox, void __force **hwbase, · 79 unmodified lines | |||
| 101 | struct pci_dev *pdev, void __force *reg_base, | 104 | struct pci_dev *pdev, void __force *reg_base, |
| 102 | int direction, int ndevs, unsigned long *bmap); | 105 | int direction, int ndevs, unsigned long *bmap); |
| 103 | void otx2_mbox_msg_send(struct otx2_mbox *mbox, int devid); | 106 | void otx2_mbox_msg_send(struct otx2_mbox *mbox, int devid); |
| 107 | void otx2_mbox_msg_send_up(struct otx2_mbox *mbox, int devid); | ||
| 104 | int otx2_mbox_wait_for_rsp(struct otx2_mbox *mbox, int devid); | 108 | int otx2_mbox_wait_for_rsp(struct otx2_mbox *mbox, int devid); |
| 105 | int otx2_mbox_busy_poll_for_rsp(struct otx2_mbox *mbox, int devid); | 109 | int otx2_mbox_busy_poll_for_rsp(struct otx2_mbox *mbox, int devid); |
| 106 | struct mbox_msghdr *otx2_mbox_alloc_msg_rsp(struct otx2_mbox *mbox, int devid, | 110 | struct mbox_msghdr *otx2_mbox_alloc_msg_rsp(struct otx2_mbox *mbox, int devid, |
| @@ -118,6 +122,8 @@ static inline struct mbox_msghdr *otx2_mbox_alloc_msg(struct otx2_mbox *mbox, · 11 unmodified lines | |||
| 118 | return otx2_mbox_alloc_msg_rsp(mbox, devid, size, 0); | 122 | return otx2_mbox_alloc_msg_rsp(mbox, devid, size, 0); |
| 119 | } | 123 | } |
| 120 | 124 | ||
| 125 | bool otx2_mbox_wait_for_zero(struct otx2_mbox *mbox, int devid); | ||
| 126 | |||
| 121 | /* Mailbox message types */ | 127 | /* Mailbox message types */ |
| 122 | #define MBOX_MSG_MASK 0xFFFF | 128 | #define MBOX_MSG_MASK 0xFFFF |
| 123 | #define MBOX_MSG_INVALID 0xFFFE | 129 | #define MBOX_MSG_INVALID 0xFFFE |
| @@ -121,13 +121,17 @@ int mcs_add_intr_wq_entry(struct mcs *mcs, struct mcs_intr_event *event) · 120 unmodified lines | |||
| 121 | static int mcs_notify_pfvf(struct mcs_intr_event *event, struct rvu *rvu) | 121 | static int mcs_notify_pfvf(struct mcs_intr_event *event, struct rvu *rvu) |
| 122 | { | 122 | { |
| 123 | struct mcs_intr_info *req; | 123 | struct mcs_intr_info *req; |
| 124 | int err, pf; | 124 | int pf; |
| 125 | 125 | ||
| 126 | pf = rvu_get_pf(event->pcifunc); | 126 | pf = rvu_get_pf(event->pcifunc); |
| 127 | 127 | ||
| 128 | mutex_lock(&rvu->mbox_lock); | ||
| 129 | |||
| 128 | req = otx2_mbox_alloc_msg_mcs_intr_notify(rvu, pf); | 130 | req = otx2_mbox_alloc_msg_mcs_intr_notify(rvu, pf); |
| 129 | if (!req) | 131 | if (!req) { |
| 132 | mutex_unlock(&rvu->mbox_lock); | ||
| 130 | return -ENOMEM; | 133 | return -ENOMEM; |
| 134 | } | ||
| 131 | 135 | ||
| 132 | req->mcs_id = event->mcs_id; | 136 | req->mcs_id = event->mcs_id; |
| 133 | req->intr_mask = event->intr_mask; | 137 | req->intr_mask = event->intr_mask; |
| @@ -135,10 +139,11 @@ static int mcs_notify_pfvf(struct mcs_intr_event *event, struct rvu *rvu) · 1 unmodified line | |||
| 135 | req->hdr.pcifunc = event->pcifunc; | 139 | req->hdr.pcifunc = event->pcifunc; |
| 136 | req->lmac_id = event->lmac_id; | 140 | req->lmac_id = event->lmac_id; |
| 137 | 141 | ||
| 138 | otx2_mbox_msg_send(&rvu->afpf_wq_info.mbox_up, pf); | 142 | otx2_mbox_wait_for_zero(&rvu->afpf_wq_info.mbox_up, pf); |
| 139 | err = otx2_mbox_wait_for_rsp(&rvu->afpf_wq_info.mbox_up, pf); | 143 | |
| 140 | if (err) | 144 | otx2_mbox_msg_send_up(&rvu->afpf_wq_info.mbox_up, pf); |
| 141 | dev_warn(rvu->dev, "MCS notification to pf %d failed\n", pf); | 145 | |
| 146 | mutex_unlock(&rvu->mbox_lock); | ||
| 142 | 147 | ||
| 143 | return 0; | 148 | return 0; |
| 144 | } | 149 | } |
| @@ -2119,7 +2119,7 @@ bad_message: · 2118 unmodified lines | |||
| 2119 | } | 2119 | } |
| 2120 | } | 2120 | } |
| 2121 | 2121 | ||
| 2122 | static void __rvu_mbox_handler(struct rvu_work *mwork, int type) | 2122 | static void __rvu_mbox_handler(struct rvu_work *mwork, int type, bool poll) |
| 2123 | { | 2123 | { |
| 2124 | struct rvu *rvu = mwork->rvu; | 2124 | struct rvu *rvu = mwork->rvu; |
| 2125 | int offset, err, id, devid; | 2125 | int offset, err, id, devid; |
| @@ -2186,6 +2186,9 @@ static void __rvu_mbox_handler(struct rvu_work *mwork, int type) · 60 unmodified lines | |||
| 2186 | } | 2186 | } |
| 2187 | mw->mbox_wrk[devid].num_msgs = 0; | 2187 | mw->mbox_wrk[devid].num_msgs = 0; |
| 2188 | 2188 | ||
| 2189 | if (poll) | ||
| 2190 | otx2_mbox_wait_for_zero(mbox, devid); | ||
| 2191 | |||
| 2189 | /* Send mbox responses to VF/PF */ | 2192 | /* Send mbox responses to VF/PF */ |
| 2190 | otx2_mbox_msg_send(mbox, devid); | 2193 | otx2_mbox_msg_send(mbox, devid); |
| 2191 | } | 2194 | } |
| @@ -2193,15 +2196,18 @@ static void __rvu_mbox_handler(struct rvu_work *mwork, int type) · 1 unmodified line | |||
| 2193 | static inline void rvu_afpf_mbox_handler(struct work_struct *work) | 2196 | static inline void rvu_afpf_mbox_handler(struct work_struct *work) |
| 2194 | { | 2197 | { |
| 2195 | struct rvu_work *mwork = container_of(work, struct rvu_work, work); | 2198 | struct rvu_work *mwork = container_of(work, struct rvu_work, work); |
| 2199 | struct rvu *rvu = mwork->rvu; | ||
| 2196 | 2200 | ||
| 2197 | __rvu_mbox_handler(mwork, TYPE_AFPF); | 2201 | mutex_lock(&rvu->mbox_lock); |
| 2202 | __rvu_mbox_handler(mwork, TYPE_AFPF, true); | ||
| 2203 | mutex_unlock(&rvu->mbox_lock); | ||
| 2198 | } | 2204 | } |
| 2199 | 2205 | ||
| 2200 | static inline void rvu_afvf_mbox_handler(struct work_struct *work) | 2206 | static inline void rvu_afvf_mbox_handler(struct work_struct *work) |
| 2201 | { | 2207 | { |
| 2202 | struct rvu_work *mwork = container_of(work, struct rvu_work, work); | 2208 | struct rvu_work *mwork = container_of(work, struct rvu_work, work); |
| 2203 | 2209 | ||
| 2204 | __rvu_mbox_handler(mwork, TYPE_AFVF); | 2210 | __rvu_mbox_handler(mwork, TYPE_AFVF, false); |
| 2205 | } | 2211 | } |
| 2206 | 2212 | ||
| 2207 | static void __rvu_mbox_up_handler(struct rvu_work *mwork, int type) | 2213 | static void __rvu_mbox_up_handler(struct rvu_work *mwork, int type) |
| @@ -2376,6 +2382,8 @@ static int rvu_mbox_init(struct rvu *rvu, struct mbox_wq_info *mw, · 168 unmodified lines | |||
| 2376 | } | 2382 | } |
| 2377 | } | 2383 | } |
| 2378 | 2384 | ||
| 2385 | mutex_init(&rvu->mbox_lock); | ||
| 2386 | |||
| 2379 | mbox_regions = kcalloc(num, sizeof(void *), GFP_KERNEL); | 2387 | mbox_regions = kcalloc(num, sizeof(void *), GFP_KERNEL); |
| 2380 | if (!mbox_regions) { | 2388 | if (!mbox_regions) { |
| 2381 | err = -ENOMEM; | 2389 | err = -ENOMEM; |
| @@ -2525,10 +2533,9 @@ static void rvu_queue_work(struct mbox_wq_info *mw, int first, · 143 unmodified lines | |||
| 2525 | } | 2533 | } |
| 2526 | } | 2534 | } |
| 2527 | 2535 | ||
| 2528 | static irqreturn_t rvu_mbox_intr_handler(int irq, void *rvu_irq) | 2536 | static irqreturn_t rvu_mbox_pf_intr_handler(int irq, void *rvu_irq) |
| 2529 | { | 2537 | { |
| 2530 | struct rvu *rvu = (struct rvu *)rvu_irq; | 2538 | struct rvu *rvu = (struct rvu *)rvu_irq; |
| 2531 | int vfs = rvu->vfs; | ||
| 2532 | u64 intr; | 2539 | u64 intr; |
| 2533 | 2540 | ||
| 2534 | intr = rvu_read64(rvu, BLKADDR_RVUM, RVU_AF_PFAF_MBOX_INT); | 2541 | intr = rvu_read64(rvu, BLKADDR_RVUM, RVU_AF_PFAF_MBOX_INT); |
| @@ -2542,6 +2549,18 @@ static irqreturn_t rvu_mbox_intr_handler(int irq, void *rvu_irq) · 7 unmodified lines | |||
| 2542 | 2549 | ||
| 2543 | rvu_queue_work(&rvu->afpf_wq_info, 0, rvu->hw->total_pfs, intr); | 2550 | rvu_queue_work(&rvu->afpf_wq_info, 0, rvu->hw->total_pfs, intr); |
| 2544 | 2551 | ||
| 2552 | return IRQ_HANDLED; | ||
| 2553 | } | ||
| 2554 | |||
| 2555 | static irqreturn_t rvu_mbox_intr_handler(int irq, void *rvu_irq) | ||
| 2556 | { | ||
| 2557 | struct rvu *rvu = (struct rvu *)rvu_irq; | ||
| 2558 | int vfs = rvu->vfs; | ||
| 2559 | u64 intr; | ||
| 2560 | |||
| 2561 | /* Sync with mbox memory region */ | ||
| 2562 | rmb(); | ||
| 2563 | |||
| 2545 | /* Handle VF interrupts */ | 2564 | /* Handle VF interrupts */ |
| 2546 | if (vfs > 64) { | 2565 | if (vfs > 64) { |
| 2547 | intr = rvupf_read64(rvu, RVU_PF_VFPF_MBOX_INTX(1)); | 2566 | intr = rvupf_read64(rvu, RVU_PF_VFPF_MBOX_INTX(1)); |
| @@ -2886,7 +2905,7 @@ static int rvu_register_interrupts(struct rvu *rvu) · 338 unmodified lines | |||
| 2886 | /* Register mailbox interrupt handler */ | 2905 | /* Register mailbox interrupt handler */ |
| 2887 | sprintf(&rvu->irq_name[RVU_AF_INT_VEC_MBOX * NAME_SIZE], "RVUAF Mbox"); | 2906 | sprintf(&rvu->irq_name[RVU_AF_INT_VEC_MBOX * NAME_SIZE], "RVUAF Mbox"); |
| 2888 | ret = request_irq(pci_irq_vector(rvu->pdev, RVU_AF_INT_VEC_MBOX), | 2907 | ret = request_irq(pci_irq_vector(rvu->pdev, RVU_AF_INT_VEC_MBOX), |
| 2889 | rvu_mbox_intr_handler, 0, | 2908 | rvu_mbox_pf_intr_handler, 0, |
| 2890 | &rvu->irq_name[RVU_AF_INT_VEC_MBOX * NAME_SIZE], rvu); | 2909 | &rvu->irq_name[RVU_AF_INT_VEC_MBOX * NAME_SIZE], rvu); |
| 2891 | if (ret) { | 2910 | if (ret) { |
| 2892 | dev_err(rvu->dev, | 2911 | dev_err(rvu->dev, |
| @@ -591,6 +591,8 @@ struct rvu { · 590 unmodified lines | |||
| 591 | spinlock_t mcs_intrq_lock; | 591 | spinlock_t mcs_intrq_lock; |
| 592 | /* CPT interrupt lock */ | 592 | /* CPT interrupt lock */ |
| 593 | spinlock_t cpt_intr_lock; | 593 | spinlock_t cpt_intr_lock; |
| 594 | |||
| 595 | struct mutex mbox_lock; /* Serialize mbox up and down msgs */ | ||
| 594 | }; | 596 | }; |
| 595 | 597 | ||
| 596 | static inline void rvu_write64(struct rvu *rvu, u64 block, u64 offset, u64 val) | 598 | static inline void rvu_write64(struct rvu *rvu, u64 block, u64 offset, u64 val) |
| @@ -232,7 +232,7 @@ static void cgx_notify_pfs(struct cgx_link_event *event, struct rvu *rvu) · 231 unmodified lines | |||
| 232 | struct cgx_link_user_info *linfo; | 232 | struct cgx_link_user_info *linfo; |
| 233 | struct cgx_link_info_msg *msg; | 233 | struct cgx_link_info_msg *msg; |
| 234 | unsigned long pfmap; | 234 | unsigned long pfmap; |
| 235 | int err, pfid; | 235 | int pfid; |
| 236 | 236 | ||
| 237 | linfo = &event->link_uinfo; | 237 | linfo = &event->link_uinfo; |
| 238 | pfmap = cgxlmac_to_pfmap(rvu, event->cgx_id, event->lmac_id); | 238 | pfmap = cgxlmac_to_pfmap(rvu, event->cgx_id, event->lmac_id); |
| @@ -255,16 +255,22 @@ static void cgx_notify_pfs(struct cgx_link_event *event, struct rvu *rvu) · 16 unmodified lines | |||
| 255 | continue; | 255 | continue; |
| 256 | } | 256 | } |
| 257 | 257 | ||
| 258 | mutex_lock(&rvu->mbox_lock); | ||
| 259 | |||
| 258 | /* Send mbox message to PF */ | 260 | /* Send mbox message to PF */ |
| 259 | msg = otx2_mbox_alloc_msg_cgx_link_event(rvu, pfid); | 261 | msg = otx2_mbox_alloc_msg_cgx_link_event(rvu, pfid); |
| 260 | if (!msg) | 262 | if (!msg) { |
| 263 | mutex_unlock(&rvu->mbox_lock); | ||
| 261 | continue; | 264 | continue; |
| 265 | } | ||
| 266 | |||
| 262 | msg->link_info = *linfo; | 267 | msg->link_info = *linfo; |
| 263 | otx2_mbox_msg_send(&rvu->afpf_wq_info.mbox_up, pfid); | 268 | |
| 264 | err = otx2_mbox_wait_for_rsp(&rvu->afpf_wq_info.mbox_up, pfid); | 269 | otx2_mbox_wait_for_zero(&rvu->afpf_wq_info.mbox_up, pfid); |
| 265 | if (err) | 270 | |
| 266 | dev_warn(rvu->dev, "notification to pf %d failed\n", | 271 | otx2_mbox_msg_send_up(&rvu->afpf_wq_info.mbox_up, pfid); |
| 267 | pfid); | 272 | |
| 273 | mutex_unlock(&rvu->mbox_lock); | ||
| 268 | } while (pfmap); | 274 | } while (pfmap); |
| 269 | } | 275 | } |
| 270 | 276 | ||
| @@ -1592,7 +1592,7 @@ int otx2_detach_resources(struct mbox *mbox) · 1591 unmodified lines | |||
| 1592 | detach->partial = false; | 1592 | detach->partial = false; |
| 1593 | 1593 | ||
| 1594 | /* Send detach request to AF */ | 1594 | /* Send detach request to AF */ |
| 1595 | otx2_mbox_msg_send(&mbox->mbox, 0); | 1595 | otx2_sync_mbox_msg(mbox); |
| 1596 | mutex_unlock(&mbox->lock); | 1596 | mutex_unlock(&mbox->lock); |
| 1597 | return 0; | 1597 | return 0; |
| 1598 | } | 1598 | } |
| @@ -815,7 +815,7 @@ static inline int otx2_sync_mbox_up_msg(struct mbox *mbox, int devid) · 814 unmodified lines | |||
| 815 | 815 | ||
| 816 | if (!otx2_mbox_nonempty(&mbox->mbox_up, devid)) | 816 | if (!otx2_mbox_nonempty(&mbox->mbox_up, devid)) |
| 817 | return 0; | 817 | return 0; |
| 818 | otx2_mbox_msg_send(&mbox->mbox_up, devid); | 818 | otx2_mbox_msg_send_up(&mbox->mbox_up, devid); |
| 819 | err = otx2_mbox_wait_for_rsp(&mbox->mbox_up, devid); | 819 | err = otx2_mbox_wait_for_rsp(&mbox->mbox_up, devid); |
| 820 | if (err) | 820 | if (err) |
| 821 | return err; | 821 | return err; |
| @@ -292,8 +292,8 @@ static int otx2_pf_flr_init(struct otx2_nic *pf, int num_vfs) · 291 unmodified lines | |||
| 292 | return 0; | 292 | return 0; |
| 293 | } | 293 | } |
| 294 | 294 | ||
| 295 | static void otx2_queue_work(struct mbox *mw, struct workqueue_struct *mbox_wq, | 295 | static void otx2_queue_vf_work(struct mbox *mw, struct workqueue_struct *mbox_wq, |
| 296 | int first, int mdevs, u64 intr, int type) | 296 | int first, int mdevs, u64 intr) |
| 297 | { | 297 | { |
| 298 | struct otx2_mbox_dev *mdev; | 298 | struct otx2_mbox_dev *mdev; |
| 299 | struct otx2_mbox *mbox; | 299 | struct otx2_mbox *mbox; |
| @@ -307,40 +307,26 @@ static void otx2_queue_work(struct mbox *mw, struct workqueue_struct *mbox_wq, · 7 unmodified lines | |||
| 307 | 307 | ||
| 308 | mbox = &mw->mbox; | 308 | mbox = &mw->mbox; |
| 309 | mdev = &mbox->dev[i]; | 309 | mdev = &mbox->dev[i]; |
| 310 | if (type == TYPE_PFAF) | ||
| 311 | otx2_sync_mbox_bbuf(mbox, i); | ||
| 312 | hdr = mdev->mbase + mbox->rx_start; | 310 | hdr = mdev->mbase + mbox->rx_start; |
| 313 | /* The hdr->num_msgs is set to zero immediately in the interrupt | 311 | /* The hdr->num_msgs is set to zero immediately in the interrupt |
| 314 | * handler to ensure that it holds a correct value next time | 312 | * handler to ensure that it holds a correct value next time |
| 315 | * when the interrupt handler is called. | 313 | * when the interrupt handler is called. pf->mw[i].num_msgs |
| 316 | * pf->mbox.num_msgs holds the data for use in pfaf_mbox_handler | 314 | * holds the data for use in otx2_pfvf_mbox_handler and |
| 317 | * pf>mbox.up_num_msgs holds the data for use in | 315 | * pf->mw[i].up_num_msgs holds the data for use in |
| 318 | * pfaf_mbox_up_handler. | 316 | * otx2_pfvf_mbox_up_handler. |
| 319 | */ | 317 | */ |
| 320 | if (hdr->num_msgs) { | 318 | if (hdr->num_msgs) { |
| 321 | mw[i].num_msgs = hdr->num_msgs; | 319 | mw[i].num_msgs = hdr->num_msgs; |
| 322 | hdr->num_msgs = 0; | 320 | hdr->num_msgs = 0; |
| 323 | if (type == TYPE_PFAF) | ||
| 324 | memset(mbox->hwbase + mbox->rx_start, 0, | ||
| 325 | ALIGN(sizeof(struct mbox_hdr), | ||
| 326 | sizeof(u64))); | ||
| 327 | |||
| 328 | queue_work(mbox_wq, &mw[i].mbox_wrk); | 321 | queue_work(mbox_wq, &mw[i].mbox_wrk); |
| 329 | } | 322 | } |
| 330 | 323 | ||
| 331 | mbox = &mw->mbox_up; | 324 | mbox = &mw->mbox_up; |
| 332 | mdev = &mbox->dev[i]; | 325 | mdev = &mbox->dev[i]; |
| 333 | if (type == TYPE_PFAF) | ||
| 334 | otx2_sync_mbox_bbuf(mbox, i); | ||
| 335 | hdr = mdev->mbase + mbox->rx_start; | 326 | hdr = mdev->mbase + mbox->rx_start; |
| 336 | if (hdr->num_msgs) { | 327 | if (hdr->num_msgs) { |
| 337 | mw[i].up_num_msgs = hdr->num_msgs; | 328 | mw[i].up_num_msgs = hdr->num_msgs; |
| 338 | hdr->num_msgs = 0; | 329 | hdr->num_msgs = 0; |
| 339 | if (type == TYPE_PFAF) | ||
| 340 | memset(mbox->hwbase + mbox->rx_start, 0, | ||
| 341 | ALIGN(sizeof(struct mbox_hdr), | ||
| 342 | sizeof(u64))); | ||
| 343 | |||
| 344 | queue_work(mbox_wq, &mw[i].mbox_up_wrk); | 330 | queue_work(mbox_wq, &mw[i].mbox_up_wrk); |
| 345 | } | 331 | } |
| 346 | } | 332 | } |
| @@ -356,8 +342,10 @@ static void otx2_forward_msg_pfvf(struct otx2_mbox_dev *mdev, · 9 unmodified lines | |||
| 356 | /* Msgs are already copied, trigger VF's mbox irq */ | 342 | /* Msgs are already copied, trigger VF's mbox irq */ |
| 357 | smp_wmb(); | 343 | smp_wmb(); |
| 358 | 344 | ||
| 345 | otx2_mbox_wait_for_zero(pfvf_mbox, devid); | ||
| 346 | |||
| 359 | offset = pfvf_mbox->trigger | (devid << pfvf_mbox->tr_shift); | 347 | offset = pfvf_mbox->trigger | (devid << pfvf_mbox->tr_shift); |
| 360 | writeq(1, (void __iomem *)pfvf_mbox->reg_base + offset); | 348 | writeq(MBOX_DOWN_MSG, (void __iomem *)pfvf_mbox->reg_base + offset); |
| 361 | 349 | ||
| 362 | /* Restore VF's mbox bounce buffer region address */ | 350 | /* Restore VF's mbox bounce buffer region address */ |
| 363 | src_mdev->mbase = bbuf_base; | 351 | src_mdev->mbase = bbuf_base; |
| @@ -547,7 +535,7 @@ static void otx2_pfvf_mbox_up_handler(struct work_struct *work) · 183 unmodified lines | |||
| 547 | end: | 535 | end: |
| 548 | offset = mbox->rx_start + msg->next_msgoff; | 536 | offset = mbox->rx_start + msg->next_msgoff; |
| 549 | if (mdev->msgs_acked == (vf_mbox->up_num_msgs - 1)) | 537 | if (mdev->msgs_acked == (vf_mbox->up_num_msgs - 1)) |
| 550 | __otx2_mbox_reset(mbox, 0); | 538 | __otx2_mbox_reset(mbox, vf_idx); |
| 551 | mdev->msgs_acked++; | 539 | mdev->msgs_acked++; |
| 552 | } | 540 | } |
| 553 | } | 541 | } |
| @@ -564,8 +552,7 @@ static irqreturn_t otx2_pfvf_mbox_intr_handler(int irq, void *pf_irq) · 10 unmodified lines | |||
| 564 | if (vfs > 64) { | 552 | if (vfs > 64) { |
| 565 | intr = otx2_read64(pf, RVU_PF_VFPF_MBOX_INTX(1)); | 553 | intr = otx2_read64(pf, RVU_PF_VFPF_MBOX_INTX(1)); |
| 566 | otx2_write64(pf, RVU_PF_VFPF_MBOX_INTX(1), intr); | 554 | otx2_write64(pf, RVU_PF_VFPF_MBOX_INTX(1), intr); |
| 567 | otx2_queue_work(mbox, pf->mbox_pfvf_wq, 64, vfs, intr, | 555 | otx2_queue_vf_work(mbox, pf->mbox_pfvf_wq, 64, vfs, intr); |
| 568 | TYPE_PFVF); | ||
| 569 | if (intr) | 556 | if (intr) |
| 570 | trace_otx2_msg_interrupt(mbox->mbox.pdev, "VF(s) to PF", intr); | 557 | trace_otx2_msg_interrupt(mbox->mbox.pdev, "VF(s) to PF", intr); |
| 571 | vfs = 64; | 558 | vfs = 64; |
| @@ -574,7 +561,7 @@ static irqreturn_t otx2_pfvf_mbox_intr_handler(int irq, void *pf_irq) · 2 unmodified lines | |||
| 574 | intr = otx2_read64(pf, RVU_PF_VFPF_MBOX_INTX(0)); | 561 | intr = otx2_read64(pf, RVU_PF_VFPF_MBOX_INTX(0)); |
| 575 | otx2_write64(pf, RVU_PF_VFPF_MBOX_INTX(0), intr); | 562 | otx2_write64(pf, RVU_PF_VFPF_MBOX_INTX(0), intr); |
| 576 | 563 | ||
| 577 | otx2_queue_work(mbox, pf->mbox_pfvf_wq, 0, vfs, intr, TYPE_PFVF); | 564 | otx2_queue_vf_work(mbox, pf->mbox_pfvf_wq, 0, vfs, intr); |
| 578 | 565 | ||
| 579 | if (intr) | 566 | if (intr) |
| 580 | trace_otx2_msg_interrupt(mbox->mbox.pdev, "VF(s) to PF", intr); | 567 | trace_otx2_msg_interrupt(mbox->mbox.pdev, "VF(s) to PF", intr); |
| @@ -597,8 +584,9 @@ static int otx2_pfvf_mbox_init(struct otx2_nic *pf, int numvfs) · 16 unmodified lines | |||
| 597 | if (!pf->mbox_pfvf) | 584 | if (!pf->mbox_pfvf) |
| 598 | return -ENOMEM; | 585 | return -ENOMEM; |
| 599 | 586 | ||
| 600 | pf->mbox_pfvf_wq = alloc_ordered_workqueue("otx2_pfvf_mailbox", | 587 | pf->mbox_pfvf_wq = alloc_workqueue("otx2_pfvf_mailbox", |
| 601 | WQ_HIGHPRI | WQ_MEM_RECLAIM); | 588 | WQ_UNBOUND | WQ_HIGHPRI | |
| 589 | WQ_MEM_RECLAIM, 0); | ||
| 602 | if (!pf->mbox_pfvf_wq) | 590 | if (!pf->mbox_pfvf_wq) |
| 603 | return -ENOMEM; | 591 | return -ENOMEM; |
| 604 | 592 | ||
| @@ -821,20 +809,22 @@ static void otx2_pfaf_mbox_handler(struct work_struct *work) · 216 unmodified lines | |||
| 821 | struct mbox *af_mbox; | 809 | struct mbox *af_mbox; |
| 822 | struct otx2_nic *pf; | 810 | struct otx2_nic *pf; |
| 823 | int offset, id; | 811 | int offset, id; |
| 812 | u16 num_msgs; | ||
| 824 | 813 | ||
| 825 | af_mbox = container_of(work, struct mbox, mbox_wrk); | 814 | af_mbox = container_of(work, struct mbox, mbox_wrk); |
| 826 | mbox = &af_mbox->mbox; | 815 | mbox = &af_mbox->mbox; |
| 827 | mdev = &mbox->dev[0]; | 816 | mdev = &mbox->dev[0]; |
| 828 | rsp_hdr = (struct mbox_hdr *)(mdev->mbase + mbox->rx_start); | 817 | rsp_hdr = (struct mbox_hdr *)(mdev->mbase + mbox->rx_start); |
| 818 | num_msgs = rsp_hdr->num_msgs; | ||
| 829 | 819 | ||
| 830 | offset = mbox->rx_start + ALIGN(sizeof(*rsp_hdr), MBOX_MSG_ALIGN); | 820 | offset = mbox->rx_start + ALIGN(sizeof(*rsp_hdr), MBOX_MSG_ALIGN); |
| 831 | pf = af_mbox->pfvf; | 821 | pf = af_mbox->pfvf; |
| 832 | 822 | ||
| 833 | for (id = 0; id < af_mbox->num_msgs; id++) { | 823 | for (id = 0; id < num_msgs; id++) { |
| 834 | msg = (struct mbox_msghdr *)(mdev->mbase + offset); | 824 | msg = (struct mbox_msghdr *)(mdev->mbase + offset); |
| 835 | otx2_process_pfaf_mbox_msg(pf, msg); | 825 | otx2_process_pfaf_mbox_msg(pf, msg); |
| 836 | offset = mbox->rx_start + msg->next_msgoff; | 826 | offset = mbox->rx_start + msg->next_msgoff; |
| 837 | if (mdev->msgs_acked == (af_mbox->num_msgs - 1)) | 827 | if (mdev->msgs_acked == (num_msgs - 1)) |
| 838 | __otx2_mbox_reset(mbox, 0); | 828 | __otx2_mbox_reset(mbox, 0); |
| 839 | mdev->msgs_acked++; | 829 | mdev->msgs_acked++; |
| 840 | } | 830 | } |
| @@ -945,12 +935,14 @@ static void otx2_pfaf_mbox_up_handler(struct work_struct *work) · 104 unmodified lines | |||
| 945 | int offset, id, devid = 0; | 935 | int offset, id, devid = 0; |
| 946 | struct mbox_hdr *rsp_hdr; | 936 | struct mbox_hdr *rsp_hdr; |
| 947 | struct mbox_msghdr *msg; | 937 | struct mbox_msghdr *msg; |
| 938 | u16 num_msgs; | ||
| 948 | 939 | ||
| 949 | rsp_hdr = (struct mbox_hdr *)(mdev->mbase + mbox->rx_start); | 940 | rsp_hdr = (struct mbox_hdr *)(mdev->mbase + mbox->rx_start); |
| 941 | num_msgs = rsp_hdr->num_msgs; | ||
| 950 | 942 | ||
| 951 | offset = mbox->rx_start + ALIGN(sizeof(*rsp_hdr), MBOX_MSG_ALIGN); | 943 | offset = mbox->rx_start + ALIGN(sizeof(*rsp_hdr), MBOX_MSG_ALIGN); |
| 952 | 944 | ||
| 953 | for (id = 0; id < af_mbox->up_num_msgs; id++) { | 945 | for (id = 0; id < num_msgs; id++) { |
| 954 | msg = (struct mbox_msghdr *)(mdev->mbase + offset); | 946 | msg = (struct mbox_msghdr *)(mdev->mbase + offset); |
| 955 | 947 | ||
| 956 | devid = msg->pcifunc & RVU_PFVF_FUNC_MASK; | 948 | devid = msg->pcifunc & RVU_PFVF_FUNC_MASK; |
| @@ -959,10 +951,11 @@ static void otx2_pfaf_mbox_up_handler(struct work_struct *work) · 2 unmodified lines | |||
| 959 | otx2_process_mbox_msg_up(pf, msg); | 951 | otx2_process_mbox_msg_up(pf, msg); |
| 960 | offset = mbox->rx_start + msg->next_msgoff; | 952 | offset = mbox->rx_start + msg->next_msgoff; |
| 961 | } | 953 | } |
| 962 | if (devid) { | 954 | /* Forward to VF iff VFs are really present */ |
| 955 | if (devid && pci_num_vf(pf->pdev)) { | ||
| 963 | otx2_forward_vf_mbox_msgs(pf, &pf->mbox.mbox_up, | 956 | otx2_forward_vf_mbox_msgs(pf, &pf->mbox.mbox_up, |
| 964 | MBOX_DIR_PFVF_UP, devid - 1, | 957 | MBOX_DIR_PFVF_UP, devid - 1, |
| 965 | af_mbox->up_num_msgs); | 958 | num_msgs); |
| 966 | return; | 959 | return; |
| 967 | } | 960 | } |
| 968 | 961 | ||
| @@ -972,16 +965,49 @@ static void otx2_pfaf_mbox_up_handler(struct work_struct *work) · 3 unmodified lines | |||
| 972 | static irqreturn_t otx2_pfaf_mbox_intr_handler(int irq, void *pf_irq) | 965 | static irqreturn_t otx2_pfaf_mbox_intr_handler(int irq, void *pf_irq) |
| 973 | { | 966 | { |
| 974 | struct otx2_nic *pf = (struct otx2_nic *)pf_irq; | 967 | struct otx2_nic *pf = (struct otx2_nic *)pf_irq; |
| 975 | struct mbox *mbox; | 968 | struct mbox *mw = &pf->mbox; |
| 969 | struct otx2_mbox_dev *mdev; | ||
| 970 | struct otx2_mbox *mbox; | ||
| 971 | struct mbox_hdr *hdr; | ||
| 972 | u64 mbox_data; | ||
| 976 | 973 | ||
| 977 | /* Clear the IRQ */ | 974 | /* Clear the IRQ */ |
| 978 | otx2_write64(pf, RVU_PF_INT, BIT_ULL(0)); | 975 | otx2_write64(pf, RVU_PF_INT, BIT_ULL(0)); |
| 979 | 976 | ||
| 980 | mbox = &pf->mbox; | ||
| 981 | 977 | ||
| 982 | trace_otx2_msg_interrupt(mbox->mbox.pdev, "AF to PF", BIT_ULL(0)); | 978 | mbox_data = otx2_read64(pf, RVU_PF_PFAF_MBOX0); |
| 979 | |||
| 980 | if (mbox_data & MBOX_UP_MSG) { | ||
| 981 | mbox_data &= ~MBOX_UP_MSG; | ||
| 982 | otx2_write64(pf, RVU_PF_PFAF_MBOX0, mbox_data); | ||
| 983 | |||
| 984 | mbox = &mw->mbox_up; | ||
| 985 | mdev = &mbox->dev[0]; | ||
| 986 | otx2_sync_mbox_bbuf(mbox, 0); | ||
| 987 | |||
| 988 | hdr = (struct mbox_hdr *)(mdev->mbase + mbox->rx_start); | ||
| 989 | if (hdr->num_msgs) | ||
| 990 | queue_work(pf->mbox_wq, &mw->mbox_up_wrk); | ||
| 991 | |||
| 992 | trace_otx2_msg_interrupt(pf->pdev, "UP message from AF to PF", | ||
| 993 | BIT_ULL(0)); | ||
| 994 | } | ||
| 995 | |||
| 996 | if (mbox_data & MBOX_DOWN_MSG) { | ||
| 997 | mbox_data &= ~MBOX_DOWN_MSG; | ||
| 998 | otx2_write64(pf, RVU_PF_PFAF_MBOX0, mbox_data); | ||
| 999 | |||
| 1000 | mbox = &mw->mbox; | ||
| 1001 | mdev = &mbox->dev[0]; | ||
| 1002 | otx2_sync_mbox_bbuf(mbox, 0); | ||
| 1003 | |||
| 1004 | hdr = (struct mbox_hdr *)(mdev->mbase + mbox->rx_start); | ||
| 1005 | if (hdr->num_msgs) | ||
| 1006 | queue_work(pf->mbox_wq, &mw->mbox_wrk); | ||
| 983 | 1007 | ||
| 984 | otx2_queue_work(mbox, pf->mbox_wq, 0, 1, 1, TYPE_PFAF); | 1008 | trace_otx2_msg_interrupt(pf->pdev, "DOWN reply from AF to PF", |
| 1009 | BIT_ULL(0)); | ||
| 1010 | } | ||
| 985 | 1011 | ||
| 986 | return IRQ_HANDLED; | 1012 | return IRQ_HANDLED; |
| 987 | } | 1013 | } |
| @@ -3087,6 +3113,7 @@ static void otx2_vf_link_event_task(struct work_struct *work) · 2099 unmodified lines | |||
| 3087 | struct otx2_vf_config *config; | 3113 | struct otx2_vf_config *config; |
| 3088 | struct cgx_link_info_msg *req; | 3114 | struct cgx_link_info_msg *req; |
| 3089 | struct mbox_msghdr *msghdr; | 3115 | struct mbox_msghdr *msghdr; |
| 3116 | struct delayed_work *dwork; | ||
| 3090 | struct otx2_nic *pf; | 3117 | struct otx2_nic *pf; |
| 3091 | int vf_idx; | 3118 | int vf_idx; |
| 3092 | 3119 | ||
| @@ -3095,10 +3122,24 @@ static void otx2_vf_link_event_task(struct work_struct *work) · 2 unmodified lines | |||
| 3095 | vf_idx = config - config->pf->vf_configs; | 3122 | vf_idx = config - config->pf->vf_configs; |
| 3096 | pf = config->pf; | 3123 | pf = config->pf; |
| 3097 | 3124 | ||
| 3125 | if (config->intf_down) | ||
| 3126 | return; | ||
| 3127 | |||
| 3128 | mutex_lock(&pf->mbox.lock); | ||
| 3129 | |||
| 3130 | dwork = &config->link_event_work; | ||
| 3131 | |||
| 3132 | if (!otx2_mbox_wait_for_zero(&pf->mbox_pfvf[0].mbox_up, vf_idx)) { | ||
| 3133 | schedule_delayed_work(dwork, msecs_to_jiffies(100)); | ||
| 3134 | mutex_unlock(&pf->mbox.lock); | ||
| 3135 | return; | ||
| 3136 | } | ||
| 3137 | |||
| 3098 | msghdr = otx2_mbox_alloc_msg_rsp(&pf->mbox_pfvf[0].mbox_up, vf_idx, | 3138 | msghdr = otx2_mbox_alloc_msg_rsp(&pf->mbox_pfvf[0].mbox_up, vf_idx, |
| 3099 | sizeof(*req), sizeof(struct msg_rsp)); | 3139 | sizeof(*req), sizeof(struct msg_rsp)); |
| 3100 | if (!msghdr) { | 3140 | if (!msghdr) { |
| 3101 | dev_err(pf->dev, "Failed to create VF%d link event\n", vf_idx); | 3141 | dev_err(pf->dev, "Failed to create VF%d link event\n", vf_idx); |
| 3142 | mutex_unlock(&pf->mbox.lock); | ||
| 3102 | return; | 3143 | return; |
| 3103 | } | 3144 | } |
| 3104 | 3145 | ||
| @@ -3107,7 +3148,11 @@ static void otx2_vf_link_event_task(struct work_struct *work) · 2 unmodified lines | |||
| 3107 | req->hdr.sig = OTX2_MBOX_REQ_SIG; | 3148 | req->hdr.sig = OTX2_MBOX_REQ_SIG; |
| 3108 | memcpy(&req->link_info, &pf->linfo, sizeof(req->link_info)); | 3149 | memcpy(&req->link_info, &pf->linfo, sizeof(req->link_info)); |
| 3109 | 3150 | ||
| 3151 | otx2_mbox_wait_for_zero(&pf->mbox_pfvf[0].mbox_up, vf_idx); | ||
| 3152 | |||
| 3110 | otx2_sync_mbox_up_msg(&pf->mbox_pfvf[0], vf_idx); | 3153 | otx2_sync_mbox_up_msg(&pf->mbox_pfvf[0], vf_idx); |
| 3154 | |||
| 3155 | mutex_unlock(&pf->mbox.lock); | ||
| 3111 | } | 3156 | } |
| 3112 | 3157 | ||
| 3113 | static int otx2_sriov_enable(struct pci_dev *pdev, int numvfs) | 3158 | static int otx2_sriov_enable(struct pci_dev *pdev, int numvfs) |
| @@ -89,16 +89,20 @@ static void otx2vf_vfaf_mbox_handler(struct work_struct *work) · 88 unmodified lines | |||
| 89 | struct otx2_mbox *mbox; | 89 | struct otx2_mbox *mbox; |
| 90 | struct mbox *af_mbox; | 90 | struct mbox *af_mbox; |
| 91 | int offset, id; | 91 | int offset, id; |
| 92 | u16 num_msgs; | ||
| 92 | 93 | ||
| 93 | af_mbox = container_of(work, struct mbox, mbox_wrk); | 94 | af_mbox = container_of(work, struct mbox, mbox_wrk); |
| 94 | mbox = &af_mbox->mbox; | 95 | mbox = &af_mbox->mbox; |
| 95 | mdev = &mbox->dev[0]; | 96 | mdev = &mbox->dev[0]; |
| 96 | rsp_hdr = (struct mbox_hdr *)(mdev->mbase + mbox->rx_start); | 97 | rsp_hdr = (struct mbox_hdr *)(mdev->mbase + mbox->rx_start); |
| 97 | if (af_mbox->num_msgs == 0) | 98 | num_msgs = rsp_hdr->num_msgs; |
| 99 | |||
| 100 | if (num_msgs == 0) | ||
| 98 | return; | 101 | return; |
| 102 | |||
| 99 | offset = mbox->rx_start + ALIGN(sizeof(*rsp_hdr), MBOX_MSG_ALIGN); | 103 | offset = mbox->rx_start + ALIGN(sizeof(*rsp_hdr), MBOX_MSG_ALIGN); |
| 100 | 104 | ||
| 101 | for (id = 0; id < af_mbox->num_msgs; id++) { | 105 | for (id = 0; id < num_msgs; id++) { |
| 102 | msg = (struct mbox_msghdr *)(mdev->mbase + offset); | 106 | msg = (struct mbox_msghdr *)(mdev->mbase + offset); |
| 103 | otx2vf_process_vfaf_mbox_msg(af_mbox->pfvf, msg); | 107 | otx2vf_process_vfaf_mbox_msg(af_mbox->pfvf, msg); |
| 104 | offset = mbox->rx_start + msg->next_msgoff; | 108 | offset = mbox->rx_start + msg->next_msgoff; |
| @@ -151,6 +155,7 @@ static void otx2vf_vfaf_mbox_up_handler(struct work_struct *work) · 46 unmodified lines | |||
| 151 | struct mbox *vf_mbox; | 155 | struct mbox *vf_mbox; |
| 152 | struct otx2_nic *vf; | 156 | struct otx2_nic *vf; |
| 153 | int offset, id; | 157 | int offset, id; |
| 158 | u16 num_msgs; | ||
| 154 | 159 | ||
| 155 | vf_mbox = container_of(work, struct mbox, mbox_up_wrk); | 160 | vf_mbox = container_of(work, struct mbox, mbox_up_wrk); |
| 156 | vf = vf_mbox->pfvf; | 161 | vf = vf_mbox->pfvf; |
| @@ -158,12 +163,14 @@ static void otx2vf_vfaf_mbox_up_handler(struct work_struct *work) · 1 unmodified line | |||
| 158 | mdev = &mbox->dev[0]; | 163 | mdev = &mbox->dev[0]; |
| 159 | 164 | ||
| 160 | rsp_hdr = (struct mbox_hdr *)(mdev->mbase + mbox->rx_start); | 165 | rsp_hdr = (struct mbox_hdr *)(mdev->mbase + mbox->rx_start); |
| 161 | if (vf_mbox->up_num_msgs == 0) | 166 | num_msgs = rsp_hdr->num_msgs; |
| 167 | |||
| 168 | if (num_msgs == 0) | ||
| 162 | return; | 169 | return; |
| 163 | 170 | ||
| 164 | offset = mbox->rx_start + ALIGN(sizeof(*rsp_hdr), MBOX_MSG_ALIGN); | 171 | offset = mbox->rx_start + ALIGN(sizeof(*rsp_hdr), MBOX_MSG_ALIGN); |
| 165 | 172 | ||
| 166 | for (id = 0; id < vf_mbox->up_num_msgs; id++) { | 173 | for (id = 0; id < num_msgs; id++) { |
| 167 | msg = (struct mbox_msghdr *)(mdev->mbase + offset); | 174 | msg = (struct mbox_msghdr *)(mdev->mbase + offset); |
| 168 | otx2vf_process_mbox_msg_up(vf, msg); | 175 | otx2vf_process_mbox_msg_up(vf, msg); |
| 169 | offset = mbox->rx_start + msg->next_msgoff; | 176 | offset = mbox->rx_start + msg->next_msgoff; |
| @@ -178,40 +185,48 @@ static irqreturn_t otx2vf_vfaf_mbox_intr_handler(int irq, void *vf_irq) · 8 unmodified lines | |||
| 178 | struct otx2_mbox_dev *mdev; | 185 | struct otx2_mbox_dev *mdev; |
| 179 | struct otx2_mbox *mbox; | 186 | struct otx2_mbox *mbox; |
| 180 | struct mbox_hdr *hdr; | 187 | struct mbox_hdr *hdr; |
| 188 | u64 mbox_data; | ||
| 181 | 189 | ||
| 182 | /* Clear the IRQ */ | 190 | /* Clear the IRQ */ |
| 183 | otx2_write64(vf, RVU_VF_INT, BIT_ULL(0)); | 191 | otx2_write64(vf, RVU_VF_INT, BIT_ULL(0)); |
| 184 | 192 | ||
| 193 | mbox_data = otx2_read64(vf, RVU_VF_VFPF_MBOX0); | ||
| 194 | |||
| 185 | /* Read latest mbox data */ | 195 | /* Read latest mbox data */ |
| 186 | smp_rmb(); | 196 | smp_rmb(); |
| 187 | 197 | ||
| 188 | /* Check for PF => VF response messages */ | 198 | if (mbox_data & MBOX_DOWN_MSG) { |
| 189 | mbox = &vf->mbox.mbox; | 199 | mbox_data &= ~MBOX_DOWN_MSG; |
| 190 | mdev = &mbox->dev[0]; | 200 | otx2_write64(vf, RVU_VF_VFPF_MBOX0, mbox_data); |
| 191 | otx2_sync_mbox_bbuf(mbox, 0); | 201 | |
| 202 | /* Check for PF => VF response messages */ | ||
| 203 | mbox = &vf->mbox.mbox; | ||
| 204 | mdev = &mbox->dev[0]; | ||
| 205 | otx2_sync_mbox_bbuf(mbox, 0); | ||
| 192 | 206 | ||
| 193 | trace_otx2_msg_interrupt(mbox->pdev, "PF to VF", BIT_ULL(0)); | 207 | hdr = (struct mbox_hdr *)(mdev->mbase + mbox->rx_start); |
| 208 | if (hdr->num_msgs) | ||
| 209 | queue_work(vf->mbox_wq, &vf->mbox.mbox_wrk); | ||
| 194 | 210 | ||
| 195 | hdr = (struct mbox_hdr *)(mdev->mbase + mbox->rx_start); | 211 | trace_otx2_msg_interrupt(mbox->pdev, "DOWN reply from PF to VF", |
| 196 | if (hdr->num_msgs) { | 212 | BIT_ULL(0)); |
| 197 | vf->mbox.num_msgs = hdr->num_msgs; | ||
| 198 | hdr->num_msgs = 0; | ||
| 199 | memset(mbox->hwbase + mbox->rx_start, 0, | ||
| 200 | ALIGN(sizeof(struct mbox_hdr), sizeof(u64))); | ||
| 201 | queue_work(vf->mbox_wq, &vf->mbox.mbox_wrk); | ||
| 202 | } | 213 | } |
| 203 | /* Check for PF => VF notification messages */ | ||
| 204 | mbox = &vf->mbox.mbox_up; | ||
| 205 | mdev = &mbox->dev[0]; | ||
| 206 | otx2_sync_mbox_bbuf(mbox, 0); | ||
| 207 | 214 | ||
| 208 | hdr = (struct mbox_hdr *)(mdev->mbase + mbox->rx_start); | 215 | if (mbox_data & MBOX_UP_MSG) { |
| 209 | if (hdr->num_msgs) { | 216 | mbox_data &= ~MBOX_UP_MSG; |
| 210 | vf->mbox.up_num_msgs = hdr->num_msgs; | 217 | otx2_write64(vf, RVU_VF_VFPF_MBOX0, mbox_data); |
| 211 | hdr->num_msgs = 0; | 218 | |
| 212 | memset(mbox->hwbase + mbox->rx_start, 0, | 219 | /* Check for PF => VF notification messages */ |
| 213 | ALIGN(sizeof(struct mbox_hdr), sizeof(u64))); | 220 | mbox = &vf->mbox.mbox_up; |
| 214 | queue_work(vf->mbox_wq, &vf->mbox.mbox_up_wrk); | 221 | mdev = &mbox->dev[0]; |
| 222 | otx2_sync_mbox_bbuf(mbox, 0); | ||
| 223 | |||
| 224 | hdr = (struct mbox_hdr *)(mdev->mbase + mbox->rx_start); | ||
| 225 | if (hdr->num_msgs) | ||
| 226 | queue_work(vf->mbox_wq, &vf->mbox.mbox_up_wrk); | ||
| 227 | |||
| 228 | trace_otx2_msg_interrupt(mbox->pdev, "UP message from PF to VF", | ||
| 229 | BIT_ULL(0)); | ||
| 215 | } | 230 | } |
| 216 | 231 | ||
| 217 | return IRQ_HANDLED; | 232 | return IRQ_HANDLED; |
| @@ -760,8 +775,8 @@ static void otx2vf_remove(struct pci_dev *pdev) · 542 unmodified lines | |||
| 760 | otx2_mcam_flow_del(vf); | 775 | otx2_mcam_flow_del(vf); |
| 761 | otx2_shutdown_tc(vf); | 776 | otx2_shutdown_tc(vf); |
| 762 | otx2_shutdown_qos(vf); | 777 | otx2_shutdown_qos(vf); |
| 763 | otx2vf_disable_mbox_intr(vf); | ||
| 764 | otx2_detach_resources(&vf->mbox); | 778 | otx2_detach_resources(&vf->mbox); |
| 779 | otx2vf_disable_mbox_intr(vf); | ||
| 765 | free_percpu(vf->hw.lmt_info); | 780 | free_percpu(vf->hw.lmt_info); |
| 766 | if (test_bit(CN10K_LMTST, &vf->hw.cap_flag)) | 781 | if (test_bit(CN10K_LMTST, &vf->hw.cap_flag)) |
| 767 | qmem_free(vf->dev, vf->dync_lmt); | 782 | qmem_free(vf->dev, vf->dync_lmt); |
| @@ -677,8 +677,7 @@ static int mtk_mac_finish(struct phylink_config *config, unsigned int mode, · 676 unmodified lines | |||
| 677 | mcr_cur = mtk_r32(mac->hw, MTK_MAC_MCR(mac->id)); | 677 | mcr_cur = mtk_r32(mac->hw, MTK_MAC_MCR(mac->id)); |
| 678 | mcr_new = mcr_cur; | 678 | mcr_new = mcr_cur; |
| 679 | mcr_new |= MAC_MCR_IPG_CFG | MAC_MCR_FORCE_MODE | | 679 | mcr_new |= MAC_MCR_IPG_CFG | MAC_MCR_FORCE_MODE | |
| 680 | MAC_MCR_BACKOFF_EN | MAC_MCR_BACKPR_EN | MAC_MCR_FORCE_LINK | | 680 | MAC_MCR_BACKOFF_EN | MAC_MCR_BACKPR_EN | MAC_MCR_RX_FIFO_CLR_DIS; |
| 681 | MAC_MCR_RX_FIFO_CLR_DIS; | ||
| 682 | 681 | ||
| 683 | /* Only update control register when needed! */ | 682 | /* Only update control register when needed! */ |
| 684 | if (mcr_new != mcr_cur) | 683 | if (mcr_new != mcr_cur) |
| @@ -694,7 +693,7 @@ static void mtk_mac_link_down(struct phylink_config *config, unsigned int mode, · 9 unmodified lines | |||
| 694 | phylink_config); | 693 | phylink_config); |
| 695 | u32 mcr = mtk_r32(mac->hw, MTK_MAC_MCR(mac->id)); | 694 | u32 mcr = mtk_r32(mac->hw, MTK_MAC_MCR(mac->id)); |
| 696 | 695 | ||
| 697 | mcr &= ~(MAC_MCR_TX_EN | MAC_MCR_RX_EN); | 696 | mcr &= ~(MAC_MCR_TX_EN | MAC_MCR_RX_EN | MAC_MCR_FORCE_LINK); |
| 698 | mtk_w32(mac->hw, mcr, MTK_MAC_MCR(mac->id)); | 697 | mtk_w32(mac->hw, mcr, MTK_MAC_MCR(mac->id)); |
| 699 | } | 698 | } |
| 700 | 699 | ||
| @@ -803,7 +802,7 @@ static void mtk_mac_link_up(struct phylink_config *config, · 102 unmodified lines | |||
| 803 | if (rx_pause) | 802 | if (rx_pause) |
| 804 | mcr |= MAC_MCR_FORCE_RX_FC; | 803 | mcr |= MAC_MCR_FORCE_RX_FC; |
| 805 | 804 | ||
| 806 | mcr |= MAC_MCR_TX_EN | MAC_MCR_RX_EN; | 805 | mcr |= MAC_MCR_TX_EN | MAC_MCR_RX_EN | MAC_MCR_FORCE_LINK; |
| 807 | mtk_w32(mac->hw, mcr, MTK_MAC_MCR(mac->id)); | 806 | mtk_w32(mac->hw, mcr, MTK_MAC_MCR(mac->id)); |
| 808 | } | 807 | } |
| 809 | 808 | ||
| @@ -994,7 +994,7 @@ void mtk_ppe_start(struct mtk_ppe *ppe) · 993 unmodified lines | |||
| 994 | MTK_PPE_KEEPALIVE_DISABLE) | | 994 | MTK_PPE_KEEPALIVE_DISABLE) | |
| 995 | FIELD_PREP(MTK_PPE_TB_CFG_HASH_MODE, 1) | | 995 | FIELD_PREP(MTK_PPE_TB_CFG_HASH_MODE, 1) | |
| 996 | FIELD_PREP(MTK_PPE_TB_CFG_SCAN_MODE, | 996 | FIELD_PREP(MTK_PPE_TB_CFG_SCAN_MODE, |
| 997 | MTK_PPE_SCAN_MODE_KEEPALIVE_AGE) | | 997 | MTK_PPE_SCAN_MODE_CHECK_AGE) | |
| 998 | FIELD_PREP(MTK_PPE_TB_CFG_ENTRY_NUM, | 998 | FIELD_PREP(MTK_PPE_TB_CFG_ENTRY_NUM, |
| 999 | MTK_PPE_ENTRIES_SHIFT); | 999 | MTK_PPE_ENTRIES_SHIFT); |
| 1000 | if (mtk_is_netsys_v2_or_greater(ppe->eth)) | 1000 | if (mtk_is_netsys_v2_or_greater(ppe->eth)) |
| @@ -1090,17 +1090,21 @@ int mtk_ppe_stop(struct mtk_ppe *ppe) · 89 unmodified lines | |||
| 1090 | 1090 | ||
| 1091 | mtk_ppe_cache_enable(ppe, false); | 1091 | mtk_ppe_cache_enable(ppe, false); |
| 1092 | 1092 | ||
| 1093 | /* disable offload engine */ | ||
| 1094 | ppe_clear(ppe, MTK_PPE_GLO_CFG, MTK_PPE_GLO_CFG_EN); | ||
| 1095 | ppe_w32(ppe, MTK_PPE_FLOW_CFG, 0); | ||
| 1096 | |||
| 1097 | /* disable aging */ | 1093 | /* disable aging */ |
| 1098 | val = MTK_PPE_TB_CFG_AGE_NON_L4 | | 1094 | val = MTK_PPE_TB_CFG_AGE_NON_L4 | |
| 1099 | MTK_PPE_TB_CFG_AGE_UNBIND | | 1095 | MTK_PPE_TB_CFG_AGE_UNBIND | |
| 1100 | MTK_PPE_TB_CFG_AGE_TCP | | 1096 | MTK_PPE_TB_CFG_AGE_TCP | |
| 1101 | MTK_PPE_TB_CFG_AGE_UDP | | 1097 | MTK_PPE_TB_CFG_AGE_UDP | |
| 1102 | MTK_PPE_TB_CFG_AGE_TCP_FIN; | 1098 | MTK_PPE_TB_CFG_AGE_TCP_FIN | |
| 1099 | MTK_PPE_TB_CFG_SCAN_MODE; | ||
| 1103 | ppe_clear(ppe, MTK_PPE_TB_CFG, val); | 1100 | ppe_clear(ppe, MTK_PPE_TB_CFG, val); |
| 1104 | 1101 | ||
| 1105 | return mtk_ppe_wait_busy(ppe); | 1102 | if (mtk_ppe_wait_busy(ppe)) |
| 1103 | return -ETIMEDOUT; | ||
| 1104 | |||
| 1105 | /* disable offload engine */ | ||
| 1106 | ppe_clear(ppe, MTK_PPE_GLO_CFG, MTK_PPE_GLO_CFG_EN); | ||
| 1107 | ppe_w32(ppe, MTK_PPE_FLOW_CFG, 0); | ||
| 1108 | |||
| 1109 | return 0; | ||
| 1106 | } | 1110 | } |
| @@ -571,7 +571,7 @@ static int txgbe_clock_register(struct txgbe *txgbe) · 570 unmodified lines | |||
| 571 | char clk_name[32]; | 571 | char clk_name[32]; |
| 572 | struct clk *clk; | 572 | struct clk *clk; |
| 573 | 573 | ||
| 574 | snprintf(clk_name, sizeof(clk_name), "i2c_designware.%d", | 574 | snprintf(clk_name, sizeof(clk_name), "i2c_dw.%d", |
| 575 | pci_dev_id(pdev)); | 575 | pci_dev_id(pdev)); |
| 576 | 576 | ||
| 577 | clk = clk_register_fixed_rate(NULL, clk_name, NULL, 0, 156250000); | 577 | clk = clk_register_fixed_rate(NULL, clk_name, NULL, 0, 156250000); |
| @@ -2831,8 +2831,8 @@ EXPORT_SYMBOL(genphy_resume); · 2830 unmodified lines | |||
| 2831 | int genphy_loopback(struct phy_device *phydev, bool enable) | 2831 | int genphy_loopback(struct phy_device *phydev, bool enable) |
| 2832 | { | 2832 | { |
| 2833 | if (enable) { | 2833 | if (enable) { |
| 2834 | u16 val, ctl = BMCR_LOOPBACK; | 2834 | u16 ctl = BMCR_LOOPBACK; |
| 2835 | int ret; | 2835 | int ret, val; |
| 2836 | 2836 | ||
| 2837 | ctl |= mii_bmcr_encode_fixed(phydev->speed, phydev->duplex); | 2837 | ctl |= mii_bmcr_encode_fixed(phydev->speed, phydev->duplex); |
| 2838 | 2838 | ||
| @@ -1464,8 +1464,6 @@ static netdev_features_t veth_fix_features(struct net_device *dev, · 1463 unmodified lines | |||
| 1464 | if (peer_priv->_xdp_prog) | 1464 | if (peer_priv->_xdp_prog) |
| 1465 | features &= ~NETIF_F_GSO_SOFTWARE; | 1465 | features &= ~NETIF_F_GSO_SOFTWARE; |
| 1466 | } | 1466 | } |
| 1467 | if (priv->_xdp_prog) | ||
| 1468 | features |= NETIF_F_GRO; | ||
| 1469 | 1467 | ||
| 1470 | return features; | 1468 | return features; |
| 1471 | } | 1469 | } |
| @@ -1569,14 +1567,6 @@ static int veth_xdp_set(struct net_device *dev, struct bpf_prog *prog, · 97 unmodified lines | |||
| 1569 | } | 1567 | } |
| 1570 | 1568 | ||
| 1571 | if (!old_prog) { | 1569 | if (!old_prog) { |
| 1572 | if (!veth_gro_requested(dev)) { | ||
| 1573 | /* user-space did not require GRO, but adding | ||
| 1574 | * XDP is supposed to get GRO working | ||
| 1575 | */ | ||
| 1576 | dev->features |= NETIF_F_GRO; | ||
| 1577 | netdev_features_change(dev); | ||
| 1578 | } | ||
| 1579 | |||
| 1580 | peer->hw_features &= ~NETIF_F_GSO_SOFTWARE; | 1570 | peer->hw_features &= ~NETIF_F_GSO_SOFTWARE; |
| 1581 | peer->max_mtu = max_mtu; | 1571 | peer->max_mtu = max_mtu; |
| 1582 | } | 1572 | } |
| @@ -1592,14 +1582,6 @@ static int veth_xdp_set(struct net_device *dev, struct bpf_prog *prog, · 9 unmodified lines | |||
| 1592 | if (dev->flags & IFF_UP) | 1582 | if (dev->flags & IFF_UP) |
| 1593 | veth_disable_xdp(dev); | 1583 | veth_disable_xdp(dev); |
| 1594 | 1584 | ||
| 1595 | /* if user-space did not require GRO, since adding XDP | ||
| 1596 | * enabled it, clear it now | ||
| 1597 | */ | ||
| 1598 | if (!veth_gro_requested(dev)) { | ||
| 1599 | dev->features &= ~NETIF_F_GRO; | ||
| 1600 | netdev_features_change(dev); | ||
| 1601 | } | ||
| 1602 | |||
| 1603 | if (peer) { | 1585 | if (peer) { |
| 1604 | peer->hw_features |= NETIF_F_GSO_SOFTWARE; | 1586 | peer->hw_features |= NETIF_F_GSO_SOFTWARE; |
| 1605 | peer->max_mtu = ETH_MAX_MTU; | 1587 | peer->max_mtu = ETH_MAX_MTU; |
| @@ -382,12 +382,12 @@ vmxnet3_process_xdp(struct vmxnet3_adapter *adapter, · 381 unmodified lines | |||
| 382 | page = rbi->page; | 382 | page = rbi->page; |
| 383 | dma_sync_single_for_cpu(&adapter->pdev->dev, | 383 | dma_sync_single_for_cpu(&adapter->pdev->dev, |
| 384 | page_pool_get_dma_addr(page) + | 384 | page_pool_get_dma_addr(page) + |
| 385 | rq->page_pool->p.offset, rcd->len, | 385 | rq->page_pool->p.offset, rbi->len, |
| 386 | page_pool_get_dma_dir(rq->page_pool)); | 386 | page_pool_get_dma_dir(rq->page_pool)); |
| 387 | 387 | ||
| 388 | xdp_init_buff(&xdp, rbi->len, &rq->xdp_rxq); | 388 | xdp_init_buff(&xdp, PAGE_SIZE, &rq->xdp_rxq); |
| 389 | xdp_prepare_buff(&xdp, page_address(page), rq->page_pool->p.offset, | 389 | xdp_prepare_buff(&xdp, page_address(page), rq->page_pool->p.offset, |
| 390 | rcd->len, false); | 390 | rbi->len, false); |
| 391 | xdp_buff_clear_frags_flag(&xdp); | 391 | xdp_buff_clear_frags_flag(&xdp); |
| 392 | 392 | ||
| 393 | xdp_prog = rcu_dereference(rq->adapter->xdp_bpf_prog); | 393 | xdp_prog = rcu_dereference(rq->adapter->xdp_bpf_prog); |
| @@ -780,7 +780,7 @@ static const struct of_device_id qmc_hdlc_id_table[] = { · 779 unmodified lines | |||
| 780 | { .compatible = "fsl,qmc-hdlc" }, | 780 | { .compatible = "fsl,qmc-hdlc" }, |
| 781 | {} /* sentinel */ | 781 | {} /* sentinel */ |
| 782 | }; | 782 | }; |
| 783 | MODULE_DEVICE_TABLE(of, qmc_hdlc_driver); | 783 | MODULE_DEVICE_TABLE(of, qmc_hdlc_id_table); |
| 784 | 784 | ||
| 785 | static struct platform_driver qmc_hdlc_driver = { | 785 | static struct platform_driver qmc_hdlc_driver = { |
| 786 | .driver = { | 786 | .driver = { |
| @@ -237,7 +237,6 @@ static const struct net_device_ops netdev_ops = { · 236 unmodified lines | |||
| 237 | .ndo_open = wg_open, | 237 | .ndo_open = wg_open, |
| 238 | .ndo_stop = wg_stop, | 238 | .ndo_stop = wg_stop, |
| 239 | .ndo_start_xmit = wg_xmit, | 239 | .ndo_start_xmit = wg_xmit, |
| 240 | .ndo_get_stats64 = dev_get_tstats64 | ||
| 241 | }; | 240 | }; |
| 242 | 241 | ||
| 243 | static void wg_destruct(struct net_device *dev) | 242 | static void wg_destruct(struct net_device *dev) |
| @@ -262,7 +261,6 @@ static void wg_destruct(struct net_device *dev) · 18 unmodified lines | |||
| 262 | rcu_barrier(); /* Wait for all the peers to be actually freed. */ | 261 | rcu_barrier(); /* Wait for all the peers to be actually freed. */ |
| 263 | wg_ratelimiter_uninit(); | 262 | wg_ratelimiter_uninit(); |
| 264 | memzero_explicit(&wg->static_identity, sizeof(wg->static_identity)); | 263 | memzero_explicit(&wg->static_identity, sizeof(wg->static_identity)); |
| 265 | free_percpu(dev->tstats); | ||
| 266 | kvfree(wg->index_hashtable); | 264 | kvfree(wg->index_hashtable); |
| 267 | kvfree(wg->peer_hashtable); | 265 | kvfree(wg->peer_hashtable); |
| 268 | mutex_unlock(&wg->device_update_lock); | 266 | mutex_unlock(&wg->device_update_lock); |
| @@ -297,6 +295,7 @@ static void wg_setup(struct net_device *dev) · 28 unmodified lines | |||
| 297 | dev->hw_enc_features |= WG_NETDEV_FEATURES; | 295 | dev->hw_enc_features |= WG_NETDEV_FEATURES; |
| 298 | dev->mtu = ETH_DATA_LEN - overhead; | 296 | dev->mtu = ETH_DATA_LEN - overhead; |
| 299 | dev->max_mtu = round_down(INT_MAX, MESSAGE_PADDING_MULTIPLE) - overhead; | 297 | dev->max_mtu = round_down(INT_MAX, MESSAGE_PADDING_MULTIPLE) - overhead; |
| 298 | dev->pcpu_stat_type = NETDEV_PCPU_STAT_TSTATS; | ||
| 300 | 299 | ||
| 301 | SET_NETDEV_DEVTYPE(dev, &device_type); | 300 | SET_NETDEV_DEVTYPE(dev, &device_type); |
| 302 | 301 | ||
| @@ -331,14 +330,10 @@ static int wg_newlink(struct net *src_net, struct net_device *dev, · 28 unmodified lines | |||
| 331 | if (!wg->index_hashtable) | 330 | if (!wg->index_hashtable) |
| 332 | goto err_free_peer_hashtable; | 331 | goto err_free_peer_hashtable; |
| 333 | 332 | ||
| 334 | dev->tstats = netdev_alloc_pcpu_stats(struct pcpu_sw_netstats); | ||
| 335 | if (!dev->tstats) | ||
| 336 | goto err_free_index_hashtable; | ||
| 337 | |||
| 338 | wg->handshake_receive_wq = alloc_workqueue("wg-kex-%s", | 333 | wg->handshake_receive_wq = alloc_workqueue("wg-kex-%s", |
| 339 | WQ_CPU_INTENSIVE | WQ_FREEZABLE, 0, dev->name); | 334 | WQ_CPU_INTENSIVE | WQ_FREEZABLE, 0, dev->name); |
| 340 | if (!wg->handshake_receive_wq) | 335 | if (!wg->handshake_receive_wq) |
| 341 | goto err_free_tstats; | 336 | goto err_free_index_hashtable; |
| 342 | 337 | ||
| 343 | wg->handshake_send_wq = alloc_workqueue("wg-kex-%s", | 338 | wg->handshake_send_wq = alloc_workqueue("wg-kex-%s", |
| 344 | WQ_UNBOUND | WQ_FREEZABLE, 0, dev->name); | 339 | WQ_UNBOUND | WQ_FREEZABLE, 0, dev->name); |
| @@ -397,8 +392,6 @@ err_destroy_handshake_send: · 52 unmodified lines | |||
| 397 | destroy_workqueue(wg->handshake_send_wq); | 392 | destroy_workqueue(wg->handshake_send_wq); |
| 398 | err_destroy_handshake_receive: | 393 | err_destroy_handshake_receive: |
| 399 | destroy_workqueue(wg->handshake_receive_wq); | 394 | destroy_workqueue(wg->handshake_receive_wq); |
| 400 | err_free_tstats: | ||
| 401 | free_percpu(dev->tstats); | ||
| 402 | err_free_index_hashtable: | 395 | err_free_index_hashtable: |
| 403 | kvfree(wg->index_hashtable); | 396 | kvfree(wg->index_hashtable); |
| 404 | err_free_peer_hashtable: | 397 | err_free_peer_hashtable: |
| @@ -164,8 +164,8 @@ get_peer(struct wg_peer *peer, struct sk_buff *skb, struct dump_ctx *ctx) · 163 unmodified lines | |||
| 164 | if (!allowedips_node) | 164 | if (!allowedips_node) |
| 165 | goto no_allowedips; | 165 | goto no_allowedips; |
| 166 | if (!ctx->allowedips_seq) | 166 | if (!ctx->allowedips_seq) |
| 167 | ctx->allowedips_seq = peer->device->peer_allowedips.seq; | 167 | ctx->allowedips_seq = ctx->wg->peer_allowedips.seq; |
| 168 | else if (ctx->allowedips_seq != peer->device->peer_allowedips.seq) | 168 | else if (ctx->allowedips_seq != ctx->wg->peer_allowedips.seq) |
| 169 | goto no_allowedips; | 169 | goto no_allowedips; |
| 170 | 170 | ||
| 171 | allowedips_nest = nla_nest_start(skb, WGPEER_A_ALLOWEDIPS); | 171 | allowedips_nest = nla_nest_start(skb, WGPEER_A_ALLOWEDIPS); |
| @@ -255,17 +255,17 @@ static int wg_get_device_dump(struct sk_buff *skb, struct netlink_callback *cb) · 83 unmodified lines | |||
| 255 | if (!peers_nest) | 255 | if (!peers_nest) |
| 256 | goto out; | 256 | goto out; |
| 257 | ret = 0; | 257 | ret = 0; |
| 258 | /* If the last cursor was removed via list_del_init in peer_remove, then | 258 | lockdep_assert_held(&wg->device_update_lock); |
| 259 | /* If the last cursor was removed in peer_remove or peer_remove_all, then | ||
| 259 | * we just treat this the same as there being no more peers left. The | 260 | * we just treat this the same as there being no more peers left. The |
| 260 | * reason is that seq_nr should indicate to userspace that this isn't a | 261 | * reason is that seq_nr should indicate to userspace that this isn't a |
| 261 | * coherent dump anyway, so they'll try again. | 262 | * coherent dump anyway, so they'll try again. |
| 262 | */ | 263 | */ |
| 263 | if (list_empty(&wg->peer_list) || | 264 | if (list_empty(&wg->peer_list) || |
| 264 | (ctx->next_peer && list_empty(&ctx->next_peer->peer_list))) { | 265 | (ctx->next_peer && ctx->next_peer->is_dead)) { |
| 265 | nla_nest_cancel(skb, peers_nest); | 266 | nla_nest_cancel(skb, peers_nest); |
| 266 | goto out; | 267 | goto out; |
| 267 | } | 268 | } |
| 268 | lockdep_assert_held(&wg->device_update_lock); | ||
| 269 | peer = list_prepare_entry(ctx->next_peer, &wg->peer_list, peer_list); | 269 | peer = list_prepare_entry(ctx->next_peer, &wg->peer_list, peer_list); |
| 270 | list_for_each_entry_continue(peer, &wg->peer_list, peer_list) { | 270 | list_for_each_entry_continue(peer, &wg->peer_list, peer_list) { |
| 271 | if (get_peer(peer, skb, ctx)) { | 271 | if (get_peer(peer, skb, ctx)) { |
| @@ -251,7 +251,7 @@ static bool decrypt_packet(struct sk_buff *skb, struct noise_keypair *keypair) · 250 unmodified lines | |||
| 251 | 251 | ||
| 252 | if (unlikely(!READ_ONCE(keypair->receiving.is_valid) || | 252 | if (unlikely(!READ_ONCE(keypair->receiving.is_valid) || |
| 253 | wg_birthdate_has_expired(keypair->receiving.birthdate, REJECT_AFTER_TIME) || | 253 | wg_birthdate_has_expired(keypair->receiving.birthdate, REJECT_AFTER_TIME) || |
| 254 | keypair->receiving_counter.counter >= REJECT_AFTER_MESSAGES)) { | 254 | READ_ONCE(keypair->receiving_counter.counter) >= REJECT_AFTER_MESSAGES)) { |
| 255 | WRITE_ONCE(keypair->receiving.is_valid, false); | 255 | WRITE_ONCE(keypair->receiving.is_valid, false); |
| 256 | return false; | 256 | return false; |
| 257 | } | 257 | } |
| @@ -318,7 +318,7 @@ static bool counter_validate(struct noise_replay_counter *counter, u64 their_cou · 60 unmodified lines | |||
| 318 | for (i = 1; i <= top; ++i) | 318 | for (i = 1; i <= top; ++i) |
| 319 | counter->backtrack[(i + index_current) & | 319 | counter->backtrack[(i + index_current) & |
| 320 | ((COUNTER_BITS_TOTAL / BITS_PER_LONG) - 1)] = 0; | 320 | ((COUNTER_BITS_TOTAL / BITS_PER_LONG) - 1)] = 0; |
| 321 | counter->counter = their_counter; | 321 | WRITE_ONCE(counter->counter, their_counter); |
| 322 | } | 322 | } |
| 323 | 323 | ||
| 324 | index &= (COUNTER_BITS_TOTAL / BITS_PER_LONG) - 1; | 324 | index &= (COUNTER_BITS_TOTAL / BITS_PER_LONG) - 1; |
| @@ -463,7 +463,7 @@ int wg_packet_rx_poll(struct napi_struct *napi, int budget) · 138 unmodified lines | |||
| 463 | net_dbg_ratelimited("%s: Packet has invalid nonce %llu (max %llu)\n", | 463 | net_dbg_ratelimited("%s: Packet has invalid nonce %llu (max %llu)\n", |
| 464 | peer->device->dev->name, | 464 | peer->device->dev->name, |
| 465 | PACKET_CB(skb)->nonce, | 465 | PACKET_CB(skb)->nonce, |
| 466 | keypair->receiving_counter.counter); | 466 | READ_ONCE(keypair->receiving_counter.counter)); |
| 467 | goto next; | 467 | goto next; |
| 468 | } | 468 | } |
| 469 | 469 | ||
| @@ -991,7 +991,7 @@ struct qman_portal { · 990 unmodified lines | |||
| 991 | /* linked-list of CSCN handlers. */ | 991 | /* linked-list of CSCN handlers. */ |
| 992 | struct list_head cgr_cbs; | 992 | struct list_head cgr_cbs; |
| 993 | /* list lock */ | 993 | /* list lock */ |
| 994 | spinlock_t cgr_lock; | 994 | raw_spinlock_t cgr_lock; |
| 995 | struct work_struct congestion_work; | 995 | struct work_struct congestion_work; |
| 996 | struct work_struct mr_work; | 996 | struct work_struct mr_work; |
| 997 | char irqname[MAX_IRQNAME]; | 997 | char irqname[MAX_IRQNAME]; |
| @@ -1281,7 +1281,7 @@ static int qman_create_portal(struct qman_portal *portal, · 283 unmodified lines | |||
| 1281 | /* if the given mask is NULL, assume all CGRs can be seen */ | 1281 | /* if the given mask is NULL, assume all CGRs can be seen */ |
| 1282 | qman_cgrs_fill(&portal->cgrs[0]); | 1282 | qman_cgrs_fill(&portal->cgrs[0]); |
| 1283 | INIT_LIST_HEAD(&portal->cgr_cbs); | 1283 | INIT_LIST_HEAD(&portal->cgr_cbs); |
| 1284 | spin_lock_init(&portal->cgr_lock); | 1284 | raw_spin_lock_init(&portal->cgr_lock); |
| 1285 | INIT_WORK(&portal->congestion_work, qm_congestion_task); | 1285 | INIT_WORK(&portal->congestion_work, qm_congestion_task); |
| 1286 | INIT_WORK(&portal->mr_work, qm_mr_process_task); | 1286 | INIT_WORK(&portal->mr_work, qm_mr_process_task); |
| 1287 | portal->bits = 0; | 1287 | portal->bits = 0; |
| @@ -1456,11 +1456,14 @@ static void qm_congestion_task(struct work_struct *work) · 168 unmodified lines | |||
| 1456 | union qm_mc_result *mcr; | 1456 | union qm_mc_result *mcr; |
| 1457 | struct qman_cgr *cgr; | 1457 | struct qman_cgr *cgr; |
| 1458 | 1458 | ||
| 1459 | spin_lock(&p->cgr_lock); | 1459 | /* |
| 1460 | * FIXME: QM_MCR_TIMEOUT is 10ms, which is too long for a raw spinlock! | ||
| 1461 | */ | ||
| 1462 | raw_spin_lock_irq(&p->cgr_lock); | ||
| 1460 | qm_mc_start(&p->p); | 1463 | qm_mc_start(&p->p); |
| 1461 | qm_mc_commit(&p->p, QM_MCC_VERB_QUERYCONGESTION); | 1464 | qm_mc_commit(&p->p, QM_MCC_VERB_QUERYCONGESTION); |
| 1462 | if (!qm_mc_result_timeout(&p->p, &mcr)) { | 1465 | if (!qm_mc_result_timeout(&p->p, &mcr)) { |
| 1463 | spin_unlock(&p->cgr_lock); | 1466 | raw_spin_unlock_irq(&p->cgr_lock); |
| 1464 | dev_crit(p->config->dev, "QUERYCONGESTION timeout\n"); | 1467 | dev_crit(p->config->dev, "QUERYCONGESTION timeout\n"); |
| 1465 | qman_p_irqsource_add(p, QM_PIRQ_CSCI); | 1468 | qman_p_irqsource_add(p, QM_PIRQ_CSCI); |
| 1466 | return; | 1469 | return; |
| @@ -1476,7 +1479,7 @@ static void qm_congestion_task(struct work_struct *work) · 9 unmodified lines | |||
| 1476 | list_for_each_entry(cgr, &p->cgr_cbs, node) | 1479 | list_for_each_entry(cgr, &p->cgr_cbs, node) |
| 1477 | if (cgr->cb && qman_cgrs_get(&c, cgr->cgrid)) | 1480 | if (cgr->cb && qman_cgrs_get(&c, cgr->cgrid)) |
| 1478 | cgr->cb(p, cgr, qman_cgrs_get(&rr, cgr->cgrid)); | 1481 | cgr->cb(p, cgr, qman_cgrs_get(&rr, cgr->cgrid)); |
| 1479 | spin_unlock(&p->cgr_lock); | 1482 | raw_spin_unlock_irq(&p->cgr_lock); |
| 1480 | qman_p_irqsource_add(p, QM_PIRQ_CSCI); | 1483 | qman_p_irqsource_add(p, QM_PIRQ_CSCI); |
| 1481 | } | 1484 | } |
| 1482 | 1485 | ||
| @@ -2440,7 +2443,7 @@ int qman_create_cgr(struct qman_cgr *cgr, u32 flags, · 957 unmodified lines | |||
| 2440 | preempt_enable(); | 2443 | preempt_enable(); |
| 2441 | 2444 | ||
| 2442 | cgr->chan = p->config->channel; | 2445 | cgr->chan = p->config->channel; |
| 2443 | spin_lock(&p->cgr_lock); | 2446 | raw_spin_lock_irq(&p->cgr_lock); |
| 2444 | 2447 | ||
| 2445 | if (opts) { | 2448 | if (opts) { |
| 2446 | struct qm_mcc_initcgr local_opts = *opts; | 2449 | struct qm_mcc_initcgr local_opts = *opts; |
| @@ -2477,7 +2480,7 @@ int qman_create_cgr(struct qman_cgr *cgr, u32 flags, · 30 unmodified lines | |||
| 2477 | qman_cgrs_get(&p->cgrs[1], cgr->cgrid)) | 2480 | qman_cgrs_get(&p->cgrs[1], cgr->cgrid)) |
| 2478 | cgr->cb(p, cgr, 1); | 2481 | cgr->cb(p, cgr, 1); |
| 2479 | out: | 2482 | out: |
| 2480 | spin_unlock(&p->cgr_lock); | 2483 | raw_spin_unlock_irq(&p->cgr_lock); |
| 2481 | put_affine_portal(); | 2484 | put_affine_portal(); |
| 2482 | return ret; | 2485 | return ret; |
| 2483 | } | 2486 | } |
| @@ -2512,7 +2515,7 @@ int qman_delete_cgr(struct qman_cgr *cgr) · 28 unmodified lines | |||
| 2512 | return -EINVAL; | 2515 | return -EINVAL; |
| 2513 | 2516 | ||
| 2514 | memset(&local_opts, 0, sizeof(struct qm_mcc_initcgr)); | 2517 | memset(&local_opts, 0, sizeof(struct qm_mcc_initcgr)); |
| 2515 | spin_lock_irqsave(&p->cgr_lock, irqflags); | 2518 | raw_spin_lock_irqsave(&p->cgr_lock, irqflags); |
| 2516 | list_del(&cgr->node); | 2519 | list_del(&cgr->node); |
| 2517 | /* | 2520 | /* |
| 2518 | * If there are no other CGR objects for this CGRID in the list, | 2521 | * If there are no other CGR objects for this CGRID in the list, |
| @@ -2537,7 +2540,7 @@ int qman_delete_cgr(struct qman_cgr *cgr) · 18 unmodified lines | |||
| 2537 | /* add back to the list */ | 2540 | /* add back to the list */ |
| 2538 | list_add(&cgr->node, &p->cgr_cbs); | 2541 | list_add(&cgr->node, &p->cgr_cbs); |
| 2539 | release_lock: | 2542 | release_lock: |
| 2540 | spin_unlock_irqrestore(&p->cgr_lock, irqflags); | 2543 | raw_spin_unlock_irqrestore(&p->cgr_lock, irqflags); |
| 2541 | put_affine_portal(); | 2544 | put_affine_portal(); |
| 2542 | return ret; | 2545 | return ret; |
| 2543 | } | 2546 | } |
| @@ -2577,9 +2580,9 @@ static int qman_update_cgr(struct qman_cgr *cgr, struct qm_mcc_initcgr *opts) · 33 unmodified lines | |||
| 2577 | if (!p) | 2580 | if (!p) |
| 2578 | return -EINVAL; | 2581 | return -EINVAL; |
| 2579 | 2582 | ||
| 2580 | spin_lock_irqsave(&p->cgr_lock, irqflags); | 2583 | raw_spin_lock_irqsave(&p->cgr_lock, irqflags); |
| 2581 | ret = qm_modify_cgr(cgr, 0, opts); | 2584 | ret = qm_modify_cgr(cgr, 0, opts); |
| 2582 | spin_unlock_irqrestore(&p->cgr_lock, irqflags); | 2585 | raw_spin_unlock_irqrestore(&p->cgr_lock, irqflags); |
| 2583 | put_affine_portal(); | 2586 | put_affine_portal(); |
| 2584 | return ret; | 2587 | return ret; |
| 2585 | } | 2588 | } |
| @@ -522,17 +522,18 @@ static inline void bitmap_replace(unsigned long *dst, · 521 unmodified lines | |||
| 522 | * | 522 | * |
| 523 | * (Bits 0, 1, 2, 3, 4, 5 are copied to the bits 0, 1, 4, 8, 9, 12) | 523 | * (Bits 0, 1, 2, 3, 4, 5 are copied to the bits 0, 1, 4, 8, 9, 12) |
| 524 | * | 524 | * |
| 525 | * A more 'visual' description of the operation: | 525 | * A more 'visual' description of the operation:: |
| 526 | * src: 0000000001011010 | 526 | * |
| 527 | * |||||| | 527 | * src: 0000000001011010 |
| 528 | * +------+||||| | 528 | * |||||| |
| 529 | * | +----+|||| | 529 | * +------+||||| |
| 530 | * | |+----+||| | 530 | * | +----+|||| |
| 531 | * | || +-+|| | 531 | * | |+----+||| |
| 532 | * | || | || | 532 | * | || +-+|| |
| 533 | * mask: ...v..vv...v..vv | 533 | * | || | || |
| 534 | * ...0..11...0..10 | 534 | * mask: ...v..vv...v..vv |
| 535 | * dst: 0000001100000010 | 535 | * ...0..11...0..10 |
| 536 | * dst: 0000001100000010 | ||
| 536 | * | 537 | * |
| 537 | * A relationship exists between bitmap_scatter() and bitmap_gather(). | 538 | * A relationship exists between bitmap_scatter() and bitmap_gather(). |
| 538 | * bitmap_gather() can be seen as the 'reverse' bitmap_scatter() operation. | 539 | * bitmap_gather() can be seen as the 'reverse' bitmap_scatter() operation. |
| @@ -568,16 +569,17 @@ static inline void bitmap_scatter(unsigned long *dst, const unsigned long *src, · 29 unmodified lines | |||
| 568 | * | 569 | * |
| 569 | * (Bits 0, 1, 4, 8, 9, 12 are copied to the bits 0, 1, 2, 3, 4, 5) | 570 | * (Bits 0, 1, 4, 8, 9, 12 are copied to the bits 0, 1, 2, 3, 4, 5) |
| 570 | * | 571 | * |
| 571 | * A more 'visual' description of the operation: | 572 | * A more 'visual' description of the operation:: |
| 572 | * mask: ...v..vv...v..vv | 573 | * |
| 573 | * src: 0000001100000010 | 574 | * mask: ...v..vv...v..vv |
| 574 | * ^ ^^ ^ 0 | 575 | * src: 0000001100000010 |
| 575 | * | || | 10 | 576 | * ^ ^^ ^ 0 |
| 576 | * | || > 010 | 577 | * | || | 10 |
| 577 | * | |+--> 1010 | 578 | * | || > 010 |
| 578 | * | +--> 11010 | 579 | * | |+--> 1010 |
| 579 | * +----> 011010 | 580 | * | +--> 11010 |
| 580 | * dst: 0000000000011010 | 581 | * +----> 011010 |
| 582 | * dst: 0000000000011010 | ||
| 581 | * | 583 | * |
| 582 | * A relationship exists between bitmap_gather() and bitmap_scatter(). See | 584 | * A relationship exists between bitmap_gather() and bitmap_scatter(). See |
| 583 | * bitmap_scatter() for the bitmap scatter detailed operations. | 585 | * bitmap_scatter() for the bitmap scatter detailed operations. |
| @@ -2072,6 +2072,7 @@ struct net_device { · 2071 unmodified lines | |||
| 2072 | struct pcpu_sw_netstats __percpu *tstats; | 2072 | struct pcpu_sw_netstats __percpu *tstats; |
| 2073 | struct pcpu_dstats __percpu *dstats; | 2073 | struct pcpu_dstats __percpu *dstats; |
| 2074 | }; | 2074 | }; |
| 2075 | unsigned long state; | ||
| 2075 | unsigned int flags; | 2076 | unsigned int flags; |
| 2076 | unsigned short hard_header_len; | 2077 | unsigned short hard_header_len; |
| 2077 | netdev_features_t features; | 2078 | netdev_features_t features; |
| @@ -2117,7 +2118,6 @@ struct net_device { · 39 unmodified lines | |||
| 2117 | * part of the usual set specified in Space.c. | 2118 | * part of the usual set specified in Space.c. |
| 2118 | */ | 2119 | */ |
| 2119 | 2120 | ||
| 2120 | unsigned long state; | ||
| 2121 | 2121 | ||
| 2122 | struct list_head dev_list; | 2122 | struct list_head dev_list; |
| 2123 | struct list_head napi_list; | 2123 | struct list_head napi_list; |
| @@ -247,6 +247,37 @@ do { \ · 246 unmodified lines | |||
| 247 | cond_resched(); \ | 247 | cond_resched(); \ |
| 248 | } while (0) | 248 | } while (0) |
| 249 | 249 | ||
| 250 | /** | ||
| 251 | * rcu_softirq_qs_periodic - Report RCU and RCU-Tasks quiescent states | ||
| 252 | * @old_ts: jiffies at start of processing. | ||
| 253 | * | ||
| 254 | * This helper is for long-running softirq handlers, such as NAPI threads in | ||
| 255 | * networking. The caller should initialize the variable passed in as @old_ts | ||
| 256 | * at the beginning of the softirq handler. When invoked frequently, this macro | ||
| 257 | * will invoke rcu_softirq_qs() every 100 milliseconds thereafter, which will | ||
| 258 | * provide both RCU and RCU-Tasks quiescent states. Note that this macro | ||
| 259 | * modifies its old_ts argument. | ||
| 260 | * | ||
| 261 | * Because regions of code that have disabled softirq act as RCU read-side | ||
| 262 | * critical sections, this macro should be invoked with softirq (and | ||
| 263 | * preemption) enabled. | ||
| 264 | * | ||
| 265 | * The macro is not needed when CONFIG_PREEMPT_RT is defined. RT kernels would | ||
| 266 | * have more chance to invoke schedule() calls and provide necessary quiescent | ||
| 267 | * states. As a contrast, calling cond_resched() only won't achieve the same | ||
| 268 | * effect because cond_resched() does not provide RCU-Tasks quiescent states. | ||
| 269 | */ | ||
| 270 | #define rcu_softirq_qs_periodic(old_ts) \ | ||
| 271 | do { \ | ||
| 272 | if (!IS_ENABLED(CONFIG_PREEMPT_RT) && \ | ||
| 273 | time_after(jiffies, (old_ts) + HZ / 10)) { \ | ||
| 274 | preempt_disable(); \ | ||
| 275 | rcu_softirq_qs(); \ | ||
| 276 | preempt_enable(); \ | ||
| 277 | (old_ts) = jiffies; \ | ||
| 278 | } \ | ||
| 279 | } while (0) | ||
| 280 | |||
| 250 | /* | 281 | /* |
| 251 | * Infrastructure to implement the synchronize_() primitives in | 282 | * Infrastructure to implement the synchronize_() primitives in |
| 252 | * TREE_RCU and rcu_barrier_() primitives in TINY_RCU. | 283 | * TREE_RCU and rcu_barrier_() primitives in TINY_RCU. |
| @@ -822,9 +822,9 @@ typedef unsigned char *sk_buff_data_t; · 821 unmodified lines | |||
| 822 | * @decrypted: Decrypted SKB | 822 | * @decrypted: Decrypted SKB |
| 823 | * @slow_gro: state present at GRO time, slower prepare step required | 823 | * @slow_gro: state present at GRO time, slower prepare step required |
| 824 | * @mono_delivery_time: When set, skb->tstamp has the | 824 | * @mono_delivery_time: When set, skb->tstamp has the |
| 825 | * delivery_time in mono clock base (i.e., EDT) or a clock base chosen | 825 | * delivery_time in mono clock base (i.e. EDT). Otherwise, the |
| 826 | * by SO_TXTIME. If zero, skb->tstamp has the (rcv) timestamp at | 826 | * skb->tstamp has the (rcv) timestamp at ingress and |
| 827 | * ingress. | 827 | * delivery_time at egress. |
| 828 | * @napi_id: id of the NAPI struct this skb came from | 828 | * @napi_id: id of the NAPI struct this skb came from |
| 829 | * @sender_cpu: (aka @napi_id) source CPU in XPS | 829 | * @sender_cpu: (aka @napi_id) source CPU in XPS |
| 830 | * @alloc_cpu: CPU which did the skb allocation. | 830 | * @alloc_cpu: CPU which did the skb allocation. |
| @@ -3523,6 +3523,16 @@ int skb_cow_data_for_xdp(struct page_pool *pool, struct sk_buff **pskb, · 2692 unmodified lines | |||
| 3523 | struct bpf_prog *prog); | 3523 | struct bpf_prog *prog); |
| 3524 | bool napi_pp_put_page(struct page *page, bool napi_safe); | 3524 | bool napi_pp_put_page(struct page *page, bool napi_safe); |
| 3525 | 3525 | ||
| 3526 | static inline void | ||
| 3527 | skb_page_unref(const struct sk_buff *skb, struct page *page, bool napi_safe) | ||
| 3528 | { | ||
| 3529 | #ifdef CONFIG_PAGE_POOL | ||
| 3530 | if (skb->pp_recycle && napi_pp_put_page(page, napi_safe)) | ||
| 3531 | return; | ||
| 3532 | #endif | ||
| 3533 | put_page(page); | ||
| 3534 | } | ||
| 3535 | |||
| 3526 | static inline void | 3536 | static inline void |
| 3527 | napi_frag_unref(skb_frag_t *frag, bool recycle, bool napi_safe) | 3537 | napi_frag_unref(skb_frag_t *frag, bool recycle, bool napi_safe) |
| 3528 | { | 3538 | { |
| @@ -422,13 +422,6 @@ extern long __sys_recvmsg_sock(struct socket *sock, struct msghdr *msg, · 421 unmodified lines | |||
| 422 | struct user_msghdr __user *umsg, | 422 | struct user_msghdr __user *umsg, |
| 423 | struct sockaddr __user *uaddr, | 423 | struct sockaddr __user *uaddr, |
| 424 | unsigned int flags); | 424 | unsigned int flags); |
| 425 | extern int sendmsg_copy_msghdr(struct msghdr *msg, | ||
| 426 | struct user_msghdr __user *umsg, unsigned flags, | ||
| 427 | struct iovec **iov); | ||
| 428 | extern int recvmsg_copy_msghdr(struct msghdr *msg, | ||
| 429 | struct user_msghdr __user *umsg, unsigned flags, | ||
| 430 | struct sockaddr __user **uaddr, | ||
| 431 | struct iovec **iov); | ||
| 432 | extern int __copy_msghdr(struct msghdr *kmsg, | 425 | extern int __copy_msghdr(struct msghdr *kmsg, |
| 433 | struct user_msghdr *umsg, | 426 | struct user_msghdr *umsg, |
| 434 | struct sockaddr __user **save_addr); | 427 | struct sockaddr __user **save_addr); |
| @@ -61,7 +61,11 @@ struct request_sock { · 60 unmodified lines | |||
| 61 | struct request_sock *dl_next; | 61 | struct request_sock *dl_next; |
| 62 | u16 mss; | 62 | u16 mss; |
| 63 | u8 num_retrans; /* number of retransmits */ | 63 | u8 num_retrans; /* number of retransmits */ |
| 64 | u8 syncookie:1; /* syncookie: encode tcpopts in timestamp */ | 64 | u8 syncookie:1; /* True if |
| 65 | * 1) tcpopts needs to be encoded in | ||
| 66 | * TS of SYN+ACK | ||
| 67 | * 2) ACK is validated by BPF kfunc. | ||
| 68 | */ | ||
| 65 | u8 num_timeout:7; /* number of timeouts */ | 69 | u8 num_timeout:7; /* number of timeouts */ |
| 66 | u32 ts_recent; | 70 | u32 ts_recent; |
| 67 | struct timer_list rsk_timer; | 71 | struct timer_list rsk_timer; |
| @@ -144,6 +148,7 @@ reqsk_alloc(const struct request_sock_ops *ops, struct sock *sk_listener, · 76 unmodified lines | |||
| 144 | sk_node_init(&req_to_sk(req)->sk_node); | 148 | sk_node_init(&req_to_sk(req)->sk_node); |
| 145 | sk_tx_queue_clear(req_to_sk(req)); | 149 | sk_tx_queue_clear(req_to_sk(req)); |
| 146 | req->saved_syn = NULL; | 150 | req->saved_syn = NULL; |
| 151 | req->syncookie = 0; | ||
| 147 | req->timeout = 0; | 152 | req->timeout = 0; |
| 148 | req->num_timeout = 0; | 153 | req->num_timeout = 0; |
| 149 | req->num_retrans = 0; | 154 | req->num_retrans = 0; |
| @@ -263,6 +263,7 @@ static int cpu_map_bpf_prog_run(struct bpf_cpu_map_entry *rcpu, void **frames, · 262 unmodified lines | |||
| 263 | static int cpu_map_kthread_run(void *data) | 263 | static int cpu_map_kthread_run(void *data) |
| 264 | { | 264 | { |
| 265 | struct bpf_cpu_map_entry *rcpu = data; | 265 | struct bpf_cpu_map_entry *rcpu = data; |
| 266 | unsigned long last_qs = jiffies; | ||
| 266 | 267 | ||
| 267 | complete(&rcpu->kthread_running); | 268 | complete(&rcpu->kthread_running); |
| 268 | set_current_state(TASK_INTERRUPTIBLE); | 269 | set_current_state(TASK_INTERRUPTIBLE); |
| @@ -288,10 +289,12 @@ static int cpu_map_kthread_run(void *data) · 19 unmodified lines | |||
| 288 | if (__ptr_ring_empty(rcpu->queue)) { | 289 | if (__ptr_ring_empty(rcpu->queue)) { |
| 289 | schedule(); | 290 | schedule(); |
| 290 | sched = 1; | 291 | sched = 1; |
| 292 | last_qs = jiffies; | ||
| 291 | } else { | 293 | } else { |
| 292 | __set_current_state(TASK_RUNNING); | 294 | __set_current_state(TASK_RUNNING); |
| 293 | } | 295 | } |
| 294 | } else { | 296 | } else { |
| 297 | rcu_softirq_qs_periodic(last_qs); | ||
| 295 | sched = cond_resched(); | 298 | sched = cond_resched(); |
| 296 | } | 299 | } |
| 297 | 300 | ||
| @@ -2245,7 +2245,7 @@ void dev_queue_xmit_nit(struct sk_buff *skb, struct net_device *dev) · 2244 unmodified lines | |||
| 2245 | rcu_read_lock(); | 2245 | rcu_read_lock(); |
| 2246 | again: | 2246 | again: |
| 2247 | list_for_each_entry_rcu(ptype, ptype_list, list) { | 2247 | list_for_each_entry_rcu(ptype, ptype_list, list) { |
| 2248 | if (ptype->ignore_outgoing) | 2248 | if (READ_ONCE(ptype->ignore_outgoing)) |
| 2249 | continue; | 2249 | continue; |
| 2250 | 2250 | ||
| 2251 | /* Never send packets back to the socket | 2251 | /* Never send packets back to the socket |
| @@ -6743,6 +6743,8 @@ static int napi_threaded_poll(void *data) · 4491 unmodified lines | |||
| 6743 | void *have; | 6743 | void *have; |
| 6744 | 6744 | ||
| 6745 | while (!napi_thread_wait(napi)) { | 6745 | while (!napi_thread_wait(napi)) { |
| 6746 | unsigned long last_qs = jiffies; | ||
| 6747 | |||
| 6746 | for (;;) { | 6748 | for (;;) { |
| 6747 | bool repoll = false; | 6749 | bool repoll = false; |
| 6748 | 6750 | ||
| @@ -6767,6 +6769,7 @@ static int napi_threaded_poll(void *data) · 18 unmodified lines | |||
| 6767 | if (!repoll) | 6769 | if (!repoll) |
| 6768 | break; | 6770 | break; |
| 6769 | 6771 | ||
| 6772 | rcu_softirq_qs_periodic(last_qs); | ||
| 6770 | cond_resched(); | 6773 | cond_resched(); |
| 6771 | } | 6774 | } |
| 6772 | } | 6775 | } |
| @@ -11665,11 +11668,12 @@ static void __init net_dev_struct_check(void) · 4892 unmodified lines | |||
| 11665 | 11668 | ||
| 11666 | /* TXRX read-mostly hotpath */ | 11669 | /* TXRX read-mostly hotpath */ |
| 11667 | CACHELINE_ASSERT_GROUP_MEMBER(struct net_device, net_device_read_txrx, lstats); | 11670 | CACHELINE_ASSERT_GROUP_MEMBER(struct net_device, net_device_read_txrx, lstats); |
| 11671 | CACHELINE_ASSERT_GROUP_MEMBER(struct net_device, net_device_read_txrx, state); | ||
| 11668 | CACHELINE_ASSERT_GROUP_MEMBER(struct net_device, net_device_read_txrx, flags); | 11672 | CACHELINE_ASSERT_GROUP_MEMBER(struct net_device, net_device_read_txrx, flags); |
| 11669 | CACHELINE_ASSERT_GROUP_MEMBER(struct net_device, net_device_read_txrx, hard_header_len); | 11673 | CACHELINE_ASSERT_GROUP_MEMBER(struct net_device, net_device_read_txrx, hard_header_len); |
| 11670 | CACHELINE_ASSERT_GROUP_MEMBER(struct net_device, net_device_read_txrx, features); | 11674 | CACHELINE_ASSERT_GROUP_MEMBER(struct net_device, net_device_read_txrx, features); |
| 11671 | CACHELINE_ASSERT_GROUP_MEMBER(struct net_device, net_device_read_txrx, ip6_ptr); | 11675 | CACHELINE_ASSERT_GROUP_MEMBER(struct net_device, net_device_read_txrx, ip6_ptr); |
| 11672 | CACHELINE_ASSERT_GROUP_SIZE(struct net_device, net_device_read_txrx, 38); | 11676 | CACHELINE_ASSERT_GROUP_SIZE(struct net_device, net_device_read_txrx, 46); |
| 11673 | 11677 | ||
| 11674 | /* RX read-mostly hotpath */ | 11678 | /* RX read-mostly hotpath */ |
| 11675 | CACHELINE_ASSERT_GROUP_MEMBER(struct net_device, net_device_read_rx, ptype_specific); | 11679 | CACHELINE_ASSERT_GROUP_MEMBER(struct net_device, net_device_read_rx, ptype_specific); |
| @@ -193,12 +193,13 @@ devlink_get_from_attrs_lock(struct net *net, struct nlattr **attrs, · 192 unmodified lines | |||
| 193 | devname = nla_data(attrs[DEVLINK_ATTR_DEV_NAME]); | 193 | devname = nla_data(attrs[DEVLINK_ATTR_DEV_NAME]); |
| 194 | 194 | ||
| 195 | devlinks_xa_for_each_registered_get(net, index, devlink) { | 195 | devlinks_xa_for_each_registered_get(net, index, devlink) { |
| 196 | devl_dev_lock(devlink, dev_lock); | 196 | if (strcmp(devlink->dev->bus->name, busname) == 0 && |
| 197 | if (devl_is_registered(devlink) && | 197 | strcmp(dev_name(devlink->dev), devname) == 0) { |
| 198 | strcmp(devlink->dev->bus->name, busname) == 0 && | 198 | devl_dev_lock(devlink, dev_lock); |
| 199 | strcmp(dev_name(devlink->dev), devname) == 0) | 199 | if (devl_is_registered(devlink)) |
| 200 | return devlink; | 200 | return devlink; |
| 201 | devl_dev_unlock(devlink, dev_lock); | 201 | devl_dev_unlock(devlink, dev_lock); |
| 202 | } | ||
| 202 | devlink_put(devlink); | 203 | devlink_put(devlink); |
| 203 | } | 204 | } |
| 204 | 205 | ||
| @@ -889,7 +889,7 @@ int devlink_nl_port_new_doit(struct sk_buff *skb, struct genl_info *info) · 888 unmodified lines | |||
| 889 | err = -ENOMEM; | 889 | err = -ENOMEM; |
| 890 | goto err_out_port_del; | 890 | goto err_out_port_del; |
| 891 | } | 891 | } |
| 892 | err = devlink_nl_port_fill(msg, devlink_port, DEVLINK_CMD_NEW, | 892 | err = devlink_nl_port_fill(msg, devlink_port, DEVLINK_CMD_PORT_NEW, |
| 893 | info->snd_portid, info->snd_seq, 0, NULL); | 893 | info->snd_portid, info->snd_seq, 0, NULL); |
| 894 | if (WARN_ON_ONCE(err)) | 894 | if (WARN_ON_ONCE(err)) |
| 895 | goto err_out_msg_free; | 895 | goto err_out_msg_free; |
| @@ -228,6 +228,10 @@ struct hsr_node *hsr_get_node(struct hsr_port *port, struct list_head *node_db, · 227 unmodified lines | |||
| 228 | */ | 228 | */ |
| 229 | if (ethhdr->h_proto == htons(ETH_P_PRP) || | 229 | if (ethhdr->h_proto == htons(ETH_P_PRP) || |
| 230 | ethhdr->h_proto == htons(ETH_P_HSR)) { | 230 | ethhdr->h_proto == htons(ETH_P_HSR)) { |
| 231 | /* Check if skb contains hsr_ethhdr */ | ||
| 232 | if (skb->mac_len < sizeof(struct hsr_ethhdr)) | ||
| 233 | return NULL; | ||
| 234 | |||
| 231 | /* Use the existing sequence_nr from the tag as starting point | 235 | /* Use the existing sequence_nr from the tag as starting point |
| 232 | * for filtering duplicate frames. | 236 | * for filtering duplicate frames. |
| 233 | */ | 237 | */ |
| @@ -148,14 +148,21 @@ static struct notifier_block hsr_nb = { · 147 unmodified lines | |||
| 148 | 148 | ||
| 149 | static int __init hsr_init(void) | 149 | static int __init hsr_init(void) |
| 150 | { | 150 | { |
| 151 | int res; | 151 | int err; |
| 152 | 152 | ||
| 153 | BUILD_BUG_ON(sizeof(struct hsr_tag) != HSR_HLEN); | 153 | BUILD_BUG_ON(sizeof(struct hsr_tag) != HSR_HLEN); |
| 154 | 154 | ||
| 155 | register_netdevice_notifier(&hsr_nb); | 155 | err = register_netdevice_notifier(&hsr_nb); |
| 156 | res = hsr_netlink_init(); | 156 | if (err) |
| 157 | return err; | ||
| 158 | |||
| 159 | err = hsr_netlink_init(); | ||
| 160 | if (err) { | ||
| 161 | unregister_netdevice_notifier(&hsr_nb); | ||
| 162 | return err; | ||
| 163 | } | ||
| 157 | 164 | ||
| 158 | return res; | 165 | return 0; |
| 159 | } | 166 | } |
| 160 | 167 | ||
| 161 | static void __exit hsr_exit(void) | 168 | static void __exit hsr_exit(void) |
| @@ -95,7 +95,7 @@ static inline struct scatterlist *esp_req_sg(struct crypto_aead *aead, · 94 unmodified lines | |||
| 95 | __alignof__(struct scatterlist)); | 95 | __alignof__(struct scatterlist)); |
| 96 | } | 96 | } |
| 97 | 97 | ||
| 98 | static void esp_ssg_unref(struct xfrm_state *x, void *tmp) | 98 | static void esp_ssg_unref(struct xfrm_state *x, void *tmp, struct sk_buff *skb) |
| 99 | { | 99 | { |
| 100 | struct crypto_aead *aead = x->data; | 100 | struct crypto_aead *aead = x->data; |
| 101 | int extralen = 0; | 101 | int extralen = 0; |
| @@ -114,7 +114,7 @@ static void esp_ssg_unref(struct xfrm_state *x, void *tmp) · 12 unmodified lines | |||
| 114 | */ | 114 | */ |
| 115 | if (req->src != req->dst) | 115 | if (req->src != req->dst) |
| 116 | for (sg = sg_next(req->src); sg; sg = sg_next(sg)) | 116 | for (sg = sg_next(req->src); sg; sg = sg_next(sg)) |
| 117 | put_page(sg_page(sg)); | 117 | skb_page_unref(skb, sg_page(sg), false); |
| 118 | } | 118 | } |
| 119 | 119 | ||
| 120 | #ifdef CONFIG_INET_ESPINTCP | 120 | #ifdef CONFIG_INET_ESPINTCP |
| @@ -260,7 +260,7 @@ static void esp_output_done(void *data, int err) · 139 unmodified lines | |||
| 260 | } | 260 | } |
| 261 | 261 | ||
| 262 | tmp = ESP_SKB_CB(skb)->tmp; | 262 | tmp = ESP_SKB_CB(skb)->tmp; |
| 263 | esp_ssg_unref(x, tmp); | 263 | esp_ssg_unref(x, tmp, skb); |
| 264 | kfree(tmp); | 264 | kfree(tmp); |
| 265 | 265 | ||
| 266 | if (xo && (xo->flags & XFRM_DEV_RESUME)) { | 266 | if (xo && (xo->flags & XFRM_DEV_RESUME)) { |
| @@ -639,7 +639,7 @@ int esp_output_tail(struct xfrm_state *x, struct sk_buff *skb, struct esp_info * · 372 unmodified lines | |||
| 639 | } | 639 | } |
| 640 | 640 | ||
| 641 | if (sg != dsg) | 641 | if (sg != dsg) |
| 642 | esp_ssg_unref(x, tmp); | 642 | esp_ssg_unref(x, tmp, skb); |
| 643 | 643 | ||
| 644 | if (!err && x->encap && x->encap->encap_type == TCP_ENCAP_ESPINTCP) | 644 | if (!err && x->encap && x->encap->encap_type == TCP_ENCAP_ESPINTCP) |
| 645 | err = esp_output_tail_tcp(x, skb); | 645 | err = esp_output_tail_tcp(x, skb); |
| @@ -1135,7 +1135,7 @@ error: · 1134 unmodified lines | |||
| 1135 | sock_prot_inuse_add(net, sk->sk_prot, -1); | 1135 | sock_prot_inuse_add(net, sk->sk_prot, -1); |
| 1136 | 1136 | ||
| 1137 | spin_lock(lock); | 1137 | spin_lock(lock); |
| 1138 | sk_nulls_del_node_init_rcu(sk); | 1138 | __sk_nulls_del_node_init_rcu(sk); |
| 1139 | spin_unlock(lock); | 1139 | spin_unlock(lock); |
| 1140 | 1140 | ||
| 1141 | sk->sk_hash = 0; | 1141 | sk->sk_hash = 0; |
| @@ -263,12 +263,12 @@ void __inet_twsk_schedule(struct inet_timewait_sock *tw, int timeo, bool rearm) · 262 unmodified lines | |||
| 263 | } | 263 | } |
| 264 | EXPORT_SYMBOL_GPL(__inet_twsk_schedule); | 264 | EXPORT_SYMBOL_GPL(__inet_twsk_schedule); |
| 265 | 265 | ||
| 266 | /* Remove all non full sockets (TIME_WAIT and NEW_SYN_RECV) for dead netns */ | ||
| 266 | void inet_twsk_purge(struct inet_hashinfo *hashinfo, int family) | 267 | void inet_twsk_purge(struct inet_hashinfo *hashinfo, int family) |
| 267 | { | 268 | { |
| 268 | struct inet_timewait_sock *tw; | ||
| 269 | struct sock *sk; | ||
| 270 | struct hlist_nulls_node *node; | 269 | struct hlist_nulls_node *node; |
| 271 | unsigned int slot; | 270 | unsigned int slot; |
| 271 | struct sock *sk; | ||
| 272 | 272 | ||
| 273 | for (slot = 0; slot <= hashinfo->ehash_mask; slot++) { | 273 | for (slot = 0; slot <= hashinfo->ehash_mask; slot++) { |
| 274 | struct inet_ehash_bucket *head = &hashinfo->ehash[slot]; | 274 | struct inet_ehash_bucket *head = &hashinfo->ehash[slot]; |
| @@ -277,38 +277,35 @@ restart_rcu: · 2 unmodified lines | |||
| 277 | rcu_read_lock(); | 277 | rcu_read_lock(); |
| 278 | restart: | 278 | restart: |
| 279 | sk_nulls_for_each_rcu(sk, node, &head->chain) { | 279 | sk_nulls_for_each_rcu(sk, node, &head->chain) { |
| 280 | if (sk->sk_state != TCP_TIME_WAIT) { | 280 | int state = inet_sk_state_load(sk); |
| 281 | /* A kernel listener socket might not hold refcnt for net, | ||
| 282 | * so reqsk_timer_handler() could be fired after net is | ||
| 283 | * freed. Userspace listener and reqsk never exist here. | ||
| 284 | */ | ||
| 285 | if (unlikely(sk->sk_state == TCP_NEW_SYN_RECV && | ||
| 286 | hashinfo->pernet)) { | ||
| 287 | struct request_sock *req = inet_reqsk(sk); | ||
| 288 | |||
| 289 | inet_csk_reqsk_queue_drop_and_put(req->rsk_listener, req); | ||
| 290 | } | ||
| 291 | 281 | ||
| 282 | if ((1 << state) & ~(TCPF_TIME_WAIT | | ||
| 283 | TCPF_NEW_SYN_RECV)) | ||
| 292 | continue; | 284 | continue; |
| 293 | } | ||
| 294 | 285 | ||
| 295 | tw = inet_twsk(sk); | 286 | if (sk->sk_family != family || |
| 296 | if ((tw->tw_family != family) || | 287 | refcount_read(&sock_net(sk)->ns.count)) |
| 297 | refcount_read(&twsk_net(tw)->ns.count)) | ||
| 298 | continue; | 288 | continue; |
| 299 | 289 | ||
| 300 | if (unlikely(!refcount_inc_not_zero(&tw->tw_refcnt))) | 290 | if (unlikely(!refcount_inc_not_zero(&sk->sk_refcnt))) |
| 301 | continue; | 291 | continue; |
| 302 | 292 | ||
| 303 | if (unlikely((tw->tw_family != family) || | 293 | if (unlikely(sk->sk_family != family || |
| 304 | refcount_read(&twsk_net(tw)->ns.count))) { | 294 | refcount_read(&sock_net(sk)->ns.count))) { |
| 305 | inet_twsk_put(tw); | 295 | sock_gen_put(sk); |
| 306 | goto restart; | 296 | goto restart; |
| 307 | } | 297 | } |
| 308 | 298 | ||
| 309 | rcu_read_unlock(); | 299 | rcu_read_unlock(); |
| 310 | local_bh_disable(); | 300 | local_bh_disable(); |
| 311 | inet_twsk_deschedule_put(tw); | 301 | if (state == TCP_TIME_WAIT) { |
| 302 | inet_twsk_deschedule_put(inet_twsk(sk)); | ||
| 303 | } else { | ||
| 304 | struct request_sock *req = inet_reqsk(sk); | ||
| 305 | |||
| 306 | inet_csk_reqsk_queue_drop_and_put(req->rsk_listener, | ||
| 307 | req); | ||
| 308 | } | ||
| 312 | local_bh_enable(); | 309 | local_bh_enable(); |
| 313 | goto restart_rcu; | 310 | goto restart_rcu; |
| 314 | } | 311 | } |
| @@ -1458,7 +1458,6 @@ struct sk_buff *__ip_make_skb(struct sock *sk, · 1457 unmodified lines | |||
| 1458 | skb->priority = (cork->tos != -1) ? cork->priority: READ_ONCE(sk->sk_priority); | 1458 | skb->priority = (cork->tos != -1) ? cork->priority: READ_ONCE(sk->sk_priority); |
| 1459 | skb->mark = cork->mark; | 1459 | skb->mark = cork->mark; |
| 1460 | skb->tstamp = cork->transmit_time; | 1460 | skb->tstamp = cork->transmit_time; |
| 1461 | skb->mono_delivery_time = !!skb->tstamp; | ||
| 1462 | /* | 1461 | /* |
| 1463 | * Steal rt from cork.dst to avoid a pair of atomic_inc/atomic_dec | 1462 | * Steal rt from cork.dst to avoid a pair of atomic_inc/atomic_dec |
| 1464 | * on dst refcount | 1463 | * on dst refcount |
| @@ -357,10 +357,10 @@ static int raw_send_hdrinc(struct sock *sk, struct flowi4 *fl4, · 356 unmodified lines | |||
| 357 | goto error; | 357 | goto error; |
| 358 | skb_reserve(skb, hlen); | 358 | skb_reserve(skb, hlen); |
| 359 | 359 | ||
| 360 | skb->protocol = htons(ETH_P_IP); | ||
| 360 | skb->priority = READ_ONCE(sk->sk_priority); | 361 | skb->priority = READ_ONCE(sk->sk_priority); |
| 361 | skb->mark = sockc->mark; | 362 | skb->mark = sockc->mark; |
| 362 | skb->tstamp = sockc->transmit_time; | 363 | skb->tstamp = sockc->transmit_time; |
| 363 | skb->mono_delivery_time = !!skb->tstamp; | ||
| 364 | skb_dst_set(skb, &rt->dst); | 364 | skb_dst_set(skb, &rt->dst); |
| 365 | *rtp = NULL; | 365 | *rtp = NULL; |
| 366 | 366 | ||
| @@ -474,6 +474,9 @@ struct sock *cookie_v4_check(struct sock *sk, struct sk_buff *skb) · 473 unmodified lines | |||
| 474 | ireq->wscale_ok, &rcv_wscale, | 474 | ireq->wscale_ok, &rcv_wscale, |
| 475 | dst_metric(&rt->dst, RTAX_INITRWND)); | 475 | dst_metric(&rt->dst, RTAX_INITRWND)); |
| 476 | 476 | ||
| 477 | /* req->syncookie is set true only if ACK is validated | ||
| 478 | * by BPF kfunc, then, rcv_wscale is already configured. | ||
| 479 | */ | ||
| 477 | if (!req->syncookie) | 480 | if (!req->syncookie) |
| 478 | ireq->rcv_wscale = rcv_wscale; | 481 | ireq->rcv_wscale = rcv_wscale; |
| 479 | ireq->ecn_ok &= cookie_ecn_ok(net, &rt->dst); | 482 | ireq->ecn_ok &= cookie_ecn_ok(net, &rt->dst); |
| @@ -398,10 +398,6 @@ void tcp_twsk_purge(struct list_head *net_exit_list, int family) · 397 unmodified lines | |||
| 398 | /* Even if tw_refcount == 1, we must clean up kernel reqsk */ | 398 | /* Even if tw_refcount == 1, we must clean up kernel reqsk */ |
| 399 | inet_twsk_purge(net->ipv4.tcp_death_row.hashinfo, family); | 399 | inet_twsk_purge(net->ipv4.tcp_death_row.hashinfo, family); |
| 400 | } else if (!purged_once) { | 400 | } else if (!purged_once) { |
| 401 | /* The last refcount is decremented in tcp_sk_exit_batch() */ | ||
| 402 | if (refcount_read(&net->ipv4.tcp_death_row.tw_refcount) == 1) | ||
| 403 | continue; | ||
| 404 | |||
| 405 | inet_twsk_purge(&tcp_hashinfo, family); | 401 | inet_twsk_purge(&tcp_hashinfo, family); |
| 406 | purged_once = true; | 402 | purged_once = true; |
| 407 | } | 403 | } |
| @@ -112,7 +112,7 @@ static inline struct scatterlist *esp_req_sg(struct crypto_aead *aead, · 111 unmodified lines | |||
| 112 | __alignof__(struct scatterlist)); | 112 | __alignof__(struct scatterlist)); |
| 113 | } | 113 | } |
| 114 | 114 | ||
| 115 | static void esp_ssg_unref(struct xfrm_state *x, void *tmp) | 115 | static void esp_ssg_unref(struct xfrm_state *x, void *tmp, struct sk_buff *skb) |
| 116 | { | 116 | { |
| 117 | struct crypto_aead *aead = x->data; | 117 | struct crypto_aead *aead = x->data; |
| 118 | int extralen = 0; | 118 | int extralen = 0; |
| @@ -131,7 +131,7 @@ static void esp_ssg_unref(struct xfrm_state *x, void *tmp) · 12 unmodified lines | |||
| 131 | */ | 131 | */ |
| 132 | if (req->src != req->dst) | 132 | if (req->src != req->dst) |
| 133 | for (sg = sg_next(req->src); sg; sg = sg_next(sg)) | 133 | for (sg = sg_next(req->src); sg; sg = sg_next(sg)) |
| 134 | put_page(sg_page(sg)); | 134 | skb_page_unref(skb, sg_page(sg), false); |
| 135 | } | 135 | } |
| 136 | 136 | ||
| 137 | #ifdef CONFIG_INET6_ESPINTCP | 137 | #ifdef CONFIG_INET6_ESPINTCP |
| @@ -294,7 +294,7 @@ static void esp_output_done(void *data, int err) · 156 unmodified lines | |||
| 294 | } | 294 | } |
| 295 | 295 | ||
| 296 | tmp = ESP_SKB_CB(skb)->tmp; | 296 | tmp = ESP_SKB_CB(skb)->tmp; |
| 297 | esp_ssg_unref(x, tmp); | 297 | esp_ssg_unref(x, tmp, skb); |
| 298 | kfree(tmp); | 298 | kfree(tmp); |
| 299 | 299 | ||
| 300 | esp_output_encap_csum(skb); | 300 | esp_output_encap_csum(skb); |
| @@ -677,7 +677,7 @@ int esp6_output_tail(struct xfrm_state *x, struct sk_buff *skb, struct esp_info · 376 unmodified lines | |||
| 677 | } | 677 | } |
| 678 | 678 | ||
| 679 | if (sg != dsg) | 679 | if (sg != dsg) |
| 680 | esp_ssg_unref(x, tmp); | 680 | esp_ssg_unref(x, tmp, skb); |
| 681 | 681 | ||
| 682 | if (!err && x->encap && x->encap->encap_type == TCP_ENCAP_ESPINTCP) | 682 | if (!err && x->encap && x->encap->encap_type == TCP_ENCAP_ESPINTCP) |
| 683 | err = esp_output_tail_tcp(x, skb); | 683 | err = esp_output_tail_tcp(x, skb); |
| @@ -1925,7 +1925,7 @@ struct sk_buff *__ip6_make_skb(struct sock *sk, · 1924 unmodified lines | |||
| 1925 | skb->priority = READ_ONCE(sk->sk_priority); | 1925 | skb->priority = READ_ONCE(sk->sk_priority); |
| 1926 | skb->mark = cork->base.mark; | 1926 | skb->mark = cork->base.mark; |
| 1927 | skb->tstamp = cork->base.transmit_time; | 1927 | skb->tstamp = cork->base.transmit_time; |
| 1928 | skb->mono_delivery_time = !!skb->tstamp; | 1928 | |
| 1929 | ip6_cork_steal_dst(skb, cork); | 1929 | ip6_cork_steal_dst(skb, cork); |
| 1930 | IP6_INC_STATS(net, rt->rt6i_idev, IPSTATS_MIB_OUTREQUESTS); | 1930 | IP6_INC_STATS(net, rt->rt6i_idev, IPSTATS_MIB_OUTREQUESTS); |
| 1931 | if (proto == IPPROTO_ICMPV6) { | 1931 | if (proto == IPPROTO_ICMPV6) { |
| @@ -622,7 +622,7 @@ static int rawv6_send_hdrinc(struct sock *sk, struct msghdr *msg, int length, · 621 unmodified lines | |||
| 622 | skb->priority = READ_ONCE(sk->sk_priority); | 622 | skb->priority = READ_ONCE(sk->sk_priority); |
| 623 | skb->mark = sockc->mark; | 623 | skb->mark = sockc->mark; |
| 624 | skb->tstamp = sockc->transmit_time; | 624 | skb->tstamp = sockc->transmit_time; |
| 625 | skb->mono_delivery_time = !!skb->tstamp; | 625 | |
| 626 | skb_put(skb, length); | 626 | skb_put(skb, length); |
| 627 | skb_reset_network_header(skb); | 627 | skb_reset_network_header(skb); |
| 628 | iph = ipv6_hdr(skb); | 628 | iph = ipv6_hdr(skb); |
| @@ -258,6 +258,9 @@ struct sock *cookie_v6_check(struct sock *sk, struct sk_buff *skb) · 257 unmodified lines | |||
| 258 | ireq->wscale_ok, &rcv_wscale, | 258 | ireq->wscale_ok, &rcv_wscale, |
| 259 | dst_metric(dst, RTAX_INITRWND)); | 259 | dst_metric(dst, RTAX_INITRWND)); |
| 260 | 260 | ||
| 261 | /* req->syncookie is set true only if ACK is validated | ||
| 262 | * by BPF kfunc, then, rcv_wscale is already configured. | ||
| 263 | */ | ||
| 261 | if (!req->syncookie) | 264 | if (!req->syncookie) |
| 262 | ireq->rcv_wscale = rcv_wscale; | 265 | ireq->rcv_wscale = rcv_wscale; |
| 263 | ireq->ecn_ok &= cookie_ecn_ok(net, dst); | 266 | ireq->ecn_ok &= cookie_ecn_ok(net, dst); |
| @@ -1213,7 +1213,7 @@ static int nf_tables_updtable(struct nft_ctx *ctx) · 1212 unmodified lines | |||
| 1213 | if (flags & ~NFT_TABLE_F_MASK) | 1213 | if (flags & ~NFT_TABLE_F_MASK) |
| 1214 | return -EOPNOTSUPP; | 1214 | return -EOPNOTSUPP; |
| 1215 | 1215 | ||
| 1216 | if (flags == ctx->table->flags) | 1216 | if (flags == (ctx->table->flags & NFT_TABLE_F_MASK)) |
| 1217 | return 0; | 1217 | return 0; |
| 1218 | 1218 | ||
| 1219 | if ((nft_table_has_owner(ctx->table) && | 1219 | if ((nft_table_has_owner(ctx->table) && |
| @@ -2631,19 +2631,6 @@ static int nf_tables_updchain(struct nft_ctx *ctx, u8 genmask, u8 policy, · 1411 unmodified lines | |||
| 2631 | } | 2631 | } |
| 2632 | } | 2632 | } |
| 2633 | 2633 | ||
| 2634 | if (nla[NFTA_CHAIN_COUNTERS]) { | ||
| 2635 | if (!nft_is_base_chain(chain)) { | ||
| 2636 | err = -EOPNOTSUPP; | ||
| 2637 | goto err_hooks; | ||
| 2638 | } | ||
| 2639 | |||
| 2640 | stats = nft_stats_alloc(nla[NFTA_CHAIN_COUNTERS]); | ||
| 2641 | if (IS_ERR(stats)) { | ||
| 2642 | err = PTR_ERR(stats); | ||
| 2643 | goto err_hooks; | ||
| 2644 | } | ||
| 2645 | } | ||
| 2646 | |||
| 2647 | if (!(table->flags & NFT_TABLE_F_DORMANT) && | 2634 | if (!(table->flags & NFT_TABLE_F_DORMANT) && |
| 2648 | nft_is_base_chain(chain) && | 2635 | nft_is_base_chain(chain) && |
| 2649 | !list_empty(&hook.list)) { | 2636 | !list_empty(&hook.list)) { |
| @@ -2658,6 +2645,20 @@ static int nf_tables_updchain(struct nft_ctx *ctx, u8 genmask, u8 policy, · 8 unmodified lines | |||
| 2658 | } | 2645 | } |
| 2659 | 2646 | ||
| 2660 | unregister = true; | 2647 | unregister = true; |
| 2648 | |||
| 2649 | if (nla[NFTA_CHAIN_COUNTERS]) { | ||
| 2650 | if (!nft_is_base_chain(chain)) { | ||
| 2651 | err = -EOPNOTSUPP; | ||
| 2652 | goto err_hooks; | ||
| 2653 | } | ||
| 2654 | |||
| 2655 | stats = nft_stats_alloc(nla[NFTA_CHAIN_COUNTERS]); | ||
| 2656 | if (IS_ERR(stats)) { | ||
| 2657 | err = PTR_ERR(stats); | ||
| 2658 | goto err_hooks; | ||
| 2659 | } | ||
| 2660 | } | ||
| 2661 | |||
| 2661 | err = -ENOMEM; | 2662 | err = -ENOMEM; |
| 2662 | trans = nft_trans_alloc(ctx, NFT_MSG_NEWCHAIN, | 2663 | trans = nft_trans_alloc(ctx, NFT_MSG_NEWCHAIN, |
| 2663 | sizeof(struct nft_trans_chain)); | 2664 | sizeof(struct nft_trans_chain)); |
| @@ -2329,8 +2329,6 @@ static void nft_pipapo_destroy(const struct nft_ctx *ctx, · 2328 unmodified lines | |||
| 2329 | if (m) { | 2329 | if (m) { |
| 2330 | rcu_barrier(); | 2330 | rcu_barrier(); |
| 2331 | 2331 | ||
| 2332 | nft_set_pipapo_match_destroy(ctx, set, m); | ||
| 2333 | |||
| 2334 | for_each_possible_cpu(cpu) | 2332 | for_each_possible_cpu(cpu) |
| 2335 | pipapo_free_scratch(m, cpu); | 2333 | pipapo_free_scratch(m, cpu); |
| 2336 | free_percpu(m->scratch); | 2334 | free_percpu(m->scratch); |
| @@ -2342,8 +2340,7 @@ static void nft_pipapo_destroy(const struct nft_ctx *ctx, · 5 unmodified lines | |||
| 2342 | if (priv->clone) { | 2340 | if (priv->clone) { |
| 2343 | m = priv->clone; | 2341 | m = priv->clone; |
| 2344 | 2342 | ||
| 2345 | if (priv->dirty) | 2343 | nft_set_pipapo_match_destroy(ctx, set, m); |
| 2346 | nft_set_pipapo_match_destroy(ctx, set, m); | ||
| 2347 | 2344 | ||
| 2348 | for_each_possible_cpu(cpu) | 2345 | for_each_possible_cpu(cpu) |
| 2349 | pipapo_free_scratch(priv->clone, cpu); | 2346 | pipapo_free_scratch(priv->clone, cpu); |
| @@ -2057,7 +2057,7 @@ retry: · 2056 unmodified lines | |||
| 2057 | skb->priority = READ_ONCE(sk->sk_priority); | 2057 | skb->priority = READ_ONCE(sk->sk_priority); |
| 2058 | skb->mark = READ_ONCE(sk->sk_mark); | 2058 | skb->mark = READ_ONCE(sk->sk_mark); |
| 2059 | skb->tstamp = sockc.transmit_time; | 2059 | skb->tstamp = sockc.transmit_time; |
| 2060 | skb->mono_delivery_time = !!skb->tstamp; | 2060 | |
| 2061 | skb_setup_tx_timestamp(skb, sockc.tsflags); | 2061 | skb_setup_tx_timestamp(skb, sockc.tsflags); |
| 2062 | 2062 | ||
| 2063 | if (unlikely(extra_len == 4)) | 2063 | if (unlikely(extra_len == 4)) |
| @@ -2586,7 +2586,6 @@ static int tpacket_fill_skb(struct packet_sock *po, struct sk_buff *skb, · 522 unmodified lines | |||
| 2586 | skb->priority = READ_ONCE(po->sk.sk_priority); | 2586 | skb->priority = READ_ONCE(po->sk.sk_priority); |
| 2587 | skb->mark = READ_ONCE(po->sk.sk_mark); | 2587 | skb->mark = READ_ONCE(po->sk.sk_mark); |
| 2588 | skb->tstamp = sockc->transmit_time; | 2588 | skb->tstamp = sockc->transmit_time; |
| 2589 | skb->mono_delivery_time = !!skb->tstamp; | ||
| 2590 | skb_setup_tx_timestamp(skb, sockc->tsflags); | 2589 | skb_setup_tx_timestamp(skb, sockc->tsflags); |
| 2591 | skb_zcopy_set_nouarg(skb, ph.raw); | 2590 | skb_zcopy_set_nouarg(skb, ph.raw); |
| 2592 | 2591 | ||
| @@ -3065,7 +3064,6 @@ static int packet_snd(struct socket *sock, struct msghdr *msg, size_t len) · 472 unmodified lines | |||
| 3065 | skb->priority = READ_ONCE(sk->sk_priority); | 3064 | skb->priority = READ_ONCE(sk->sk_priority); |
| 3066 | skb->mark = sockc.mark; | 3065 | skb->mark = sockc.mark; |
| 3067 | skb->tstamp = sockc.transmit_time; | 3066 | skb->tstamp = sockc.transmit_time; |
| 3068 | skb->mono_delivery_time = !!skb->tstamp; | ||
| 3069 | 3067 | ||
| 3070 | if (unlikely(extra_len == 4)) | 3068 | if (unlikely(extra_len == 4)) |
| 3071 | skb->no_fcs = 1; | 3069 | skb->no_fcs = 1; |
| @@ -4000,7 +3998,7 @@ packet_setsockopt(struct socket *sock, int level, int optname, sockptr_t optval, · 928 unmodified lines | |||
| 4000 | if (val < 0 || val > 1) | 3998 | if (val < 0 || val > 1) |
| 4001 | return -EINVAL; | 3999 | return -EINVAL; |
| 4002 | 4000 | ||
| 4003 | po->prot_hook.ignore_outgoing = !!val; | 4001 | WRITE_ONCE(po->prot_hook.ignore_outgoing, !!val); |
| 4004 | return 0; | 4002 | return 0; |
| 4005 | } | 4003 | } |
| 4006 | case PACKET_TX_HAS_OFF: | 4004 | case PACKET_TX_HAS_OFF: |
| @@ -4134,7 +4132,7 @@ static int packet_getsockopt(struct socket *sock, int level, int optname, · 127 unmodified lines | |||
| 4134 | 0); | 4132 | 0); |
| 4135 | break; | 4133 | break; |
| 4136 | case PACKET_IGNORE_OUTGOING: | 4134 | case PACKET_IGNORE_OUTGOING: |
| 4137 | val = po->prot_hook.ignore_outgoing; | 4135 | val = READ_ONCE(po->prot_hook.ignore_outgoing); |
| 4138 | break; | 4136 | break; |
| 4139 | case PACKET_ROLLOVER_STATS: | 4137 | case PACKET_ROLLOVER_STATS: |
| 4140 | if (!po->rollover) | 4138 | if (!po->rollover) |
| @@ -103,13 +103,12 @@ EXPORT_SYMBOL_GPL(rds_send_path_reset); · 102 unmodified lines | |||
| 103 | 103 | ||
| 104 | static int acquire_in_xmit(struct rds_conn_path *cp) | 104 | static int acquire_in_xmit(struct rds_conn_path *cp) |
| 105 | { | 105 | { |
| 106 | return test_and_set_bit(RDS_IN_XMIT, &cp->cp_flags) == 0; | 106 | return test_and_set_bit_lock(RDS_IN_XMIT, &cp->cp_flags) == 0; |
| 107 | } | 107 | } |
| 108 | 108 | ||
| 109 | static void release_in_xmit(struct rds_conn_path *cp) | 109 | static void release_in_xmit(struct rds_conn_path *cp) |
| 110 | { | 110 | { |
| 111 | clear_bit(RDS_IN_XMIT, &cp->cp_flags); | 111 | clear_bit_unlock(RDS_IN_XMIT, &cp->cp_flags); |
| 112 | smp_mb__after_atomic(); | ||
| 113 | /* | 112 | /* |
| 114 | * We don't use wait_on_bit()/wake_up_bit() because our waking is in a | 113 | * We don't use wait_on_bit()/wake_up_bit() because our waking is in a |
| 115 | * hot path and finding waiters is very rare. We don't want to walk | 114 | * hot path and finding waiters is very rare. We don't want to walk |
| @@ -349,8 +349,8 @@ reload: · 348 unmodified lines | |||
| 349 | */ | 349 | */ |
| 350 | remain = more ? INT_MAX : msg_data_left(msg); | 350 | remain = more ? INT_MAX : msg_data_left(msg); |
| 351 | txb = call->conn->security->alloc_txbuf(call, remain, sk->sk_allocation); | 351 | txb = call->conn->security->alloc_txbuf(call, remain, sk->sk_allocation); |
| 352 | if (IS_ERR(txb)) { | 352 | if (!txb) { |
| 353 | ret = PTR_ERR(txb); | 353 | ret = -ENOMEM; |
| 354 | goto maybe_error; | 354 | goto maybe_error; |
| 355 | } | 355 | } |
| 356 | } | 356 | } |
| @@ -33,8 +33,8 @@ struct rxrpc_txbuf *rxrpc_alloc_data_txbuf(struct rxrpc_call *call, size_t data_ · 32 unmodified lines | |||
| 33 | total = hoff + sizeof(*whdr) + data_size; | 33 | total = hoff + sizeof(*whdr) + data_size; |
| 34 | 34 | ||
| 35 | mutex_lock(&call->conn->tx_data_alloc_lock); | 35 | mutex_lock(&call->conn->tx_data_alloc_lock); |
| 36 | buf = page_frag_alloc_align(&call->conn->tx_data_alloc, total, gfp, | 36 | buf = __page_frag_alloc_align(&call->conn->tx_data_alloc, total, gfp, |
| 37 | ~(data_align - 1) & ~(L1_CACHE_BYTES - 1)); | 37 | ~(data_align - 1) & ~(L1_CACHE_BYTES - 1)); |
| 38 | mutex_unlock(&call->conn->tx_data_alloc_lock); | 38 | mutex_unlock(&call->conn->tx_data_alloc_lock); |
| 39 | if (!buf) { | 39 | if (!buf) { |
| 40 | kfree(txb); | 40 | kfree(txb); |
| @@ -10,6 +10,7 @@ · 9 unmodified lines | |||
| 10 | */ | 10 | */ |
| 11 | 11 | ||
| 12 | #include <linux/jhash.h> | 12 | #include <linux/jhash.h> |
| 13 | #include <linux/module.h> | ||
| 13 | #include <linux/sizes.h> | 14 | #include <linux/sizes.h> |
| 14 | #include <linux/vmalloc.h> | 15 | #include <linux/vmalloc.h> |
| 15 | #include <net/pkt_cls.h> | 16 | #include <net/pkt_cls.h> |
| @@ -563,6 +564,7 @@ static struct Qdisc_ops fq_pie_qdisc_ops __read_mostly = { · 547 unmodified lines | |||
| 563 | .dump_stats = fq_pie_dump_stats, | 564 | .dump_stats = fq_pie_dump_stats, |
| 564 | .owner = THIS_MODULE, | 565 | .owner = THIS_MODULE, |
| 565 | }; | 566 | }; |
| 567 | MODULE_ALIAS_NET_SCH("fq_pie"); | ||
| 566 | 568 | ||
| 567 | static int __init fq_pie_module_init(void) | 569 | static int __init fq_pie_module_init(void) |
| 568 | { | 570 | { |
| @@ -997,7 +997,8 @@ static const struct nla_policy entry_policy[TCA_TAPRIO_SCHED_ENTRY_MAX + 1] = { · 996 unmodified lines | |||
| 997 | }; | 997 | }; |
| 998 | 998 | ||
| 999 | static const struct nla_policy taprio_tc_policy[TCA_TAPRIO_TC_ENTRY_MAX + 1] = { | 999 | static const struct nla_policy taprio_tc_policy[TCA_TAPRIO_TC_ENTRY_MAX + 1] = { |
| 1000 | [TCA_TAPRIO_TC_ENTRY_INDEX] = { .type = NLA_U32 }, | 1000 | [TCA_TAPRIO_TC_ENTRY_INDEX] = NLA_POLICY_MAX(NLA_U32, |
| 1001 | TC_QOPT_MAX_QUEUE), | ||
| 1001 | [TCA_TAPRIO_TC_ENTRY_MAX_SDU] = { .type = NLA_U32 }, | 1002 | [TCA_TAPRIO_TC_ENTRY_MAX_SDU] = { .type = NLA_U32 }, |
| 1002 | [TCA_TAPRIO_TC_ENTRY_FP] = NLA_POLICY_RANGE(NLA_U32, | 1003 | [TCA_TAPRIO_TC_ENTRY_FP] = NLA_POLICY_RANGE(NLA_U32, |
| 1003 | TC_FP_EXPRESS, | 1004 | TC_FP_EXPRESS, |
| @@ -2600,9 +2600,9 @@ out: · 2599 unmodified lines | |||
| 2600 | return err; | 2600 | return err; |
| 2601 | } | 2601 | } |
| 2602 | 2602 | ||
| 2603 | int sendmsg_copy_msghdr(struct msghdr *msg, | 2603 | static int sendmsg_copy_msghdr(struct msghdr *msg, |
| 2604 | struct user_msghdr __user *umsg, unsigned flags, | 2604 | struct user_msghdr __user *umsg, unsigned flags, |
| 2605 | struct iovec **iov) | 2605 | struct iovec **iov) |
| 2606 | { | 2606 | { |
| 2607 | int err; | 2607 | int err; |
| 2608 | 2608 | ||
| @@ -2753,10 +2753,10 @@ SYSCALL_DEFINE4(sendmmsg, int, fd, struct mmsghdr __user *, mmsg, · 144 unmodified lines | |||
| 2753 | return __sys_sendmmsg(fd, mmsg, vlen, flags, true); | 2753 | return __sys_sendmmsg(fd, mmsg, vlen, flags, true); |
| 2754 | } | 2754 | } |
| 2755 | 2755 | ||
| 2756 | int recvmsg_copy_msghdr(struct msghdr *msg, | 2756 | static int recvmsg_copy_msghdr(struct msghdr *msg, |
| 2757 | struct user_msghdr __user *umsg, unsigned flags, | 2757 | struct user_msghdr __user *umsg, unsigned flags, |
| 2758 | struct sockaddr __user **uaddr, | 2758 | struct sockaddr __user **uaddr, |
| 2759 | struct iovec **iov) | 2759 | struct iovec **iov) |
| 2760 | { | 2760 | { |
| 2761 | ssize_t err; | 2761 | ssize_t err; |
| 2762 | 2762 | ||
| @@ -407,7 +407,8 @@ bool xfrm_dev_offload_ok(struct sk_buff *skb, struct xfrm_state *x) · 406 unmodified lines | |||
| 407 | struct xfrm_dst *xdst = (struct xfrm_dst *)dst; | 407 | struct xfrm_dst *xdst = (struct xfrm_dst *)dst; |
| 408 | struct net_device *dev = x->xso.dev; | 408 | struct net_device *dev = x->xso.dev; |
| 409 | 409 | ||
| 410 | if (!x->type_offload) | 410 | if (!x->type_offload || |
| 411 | (x->xso.type == XFRM_DEV_OFFLOAD_UNSPECIFIED && x->encap)) | ||
| 411 | return false; | 412 | return false; |
| 412 | 413 | ||
| 413 | if (x->xso.type == XFRM_DEV_OFFLOAD_PACKET || | 414 | if (x->xso.type == XFRM_DEV_OFFLOAD_PACKET || |
| @@ -20,6 +20,7 @@ CFLAGS_ethtool:=$(call get_hdr_inc,_LINUX_ETHTOOL_NETLINK_H_,ethtool_netlink.h) · 19 unmodified lines | |||
| 20 | CFLAGS_handshake:=$(call get_hdr_inc,_LINUX_HANDSHAKE_H,handshake.h) | 20 | CFLAGS_handshake:=$(call get_hdr_inc,_LINUX_HANDSHAKE_H,handshake.h) |
| 21 | CFLAGS_mptcp_pm:=$(call get_hdr_inc,_LINUX_MPTCP_PM_H,mptcp_pm.h) | 21 | CFLAGS_mptcp_pm:=$(call get_hdr_inc,_LINUX_MPTCP_PM_H,mptcp_pm.h) |
| 22 | CFLAGS_netdev:=$(call get_hdr_inc,_LINUX_NETDEV_H,netdev.h) | 22 | CFLAGS_netdev:=$(call get_hdr_inc,_LINUX_NETDEV_H,netdev.h) |
| 23 | CFLAGS_nlctrl:=$(call get_hdr_inc,__LINUX_GENERIC_NETLINK_H,genetlink.h) | ||
| 23 | CFLAGS_nfsd:=$(call get_hdr_inc,_LINUX_NFSD_NETLINK_H,nfsd_netlink.h) | 24 | CFLAGS_nfsd:=$(call get_hdr_inc,_LINUX_NFSD_NETLINK_H,nfsd_netlink.h) |
| 24 | CFLAGS_ovs_datapath:=$(call get_hdr_inc,__LINUX_OPENVSWITCH_H,openvswitch.h) | 25 | CFLAGS_ovs_datapath:=$(call get_hdr_inc,__LINUX_OPENVSWITCH_H,openvswitch.h) |
| 25 | CFLAGS_ovs_flow:=$(call get_hdr_inc,__LINUX_OPENVSWITCH_H,openvswitch.h) | 26 | CFLAGS_ovs_flow:=$(call get_hdr_inc,__LINUX_OPENVSWITCH_H,openvswitch.h) |
| @@ -422,13 +422,6 @@ extern long __sys_recvmsg_sock(struct socket *sock, struct msghdr *msg, · 421 unmodified lines | |||
| 422 | struct user_msghdr __user *umsg, | 422 | struct user_msghdr __user *umsg, |
| 423 | struct sockaddr __user *uaddr, | 423 | struct sockaddr __user *uaddr, |
| 424 | unsigned int flags); | 424 | unsigned int flags); |
| 425 | extern int sendmsg_copy_msghdr(struct msghdr *msg, | ||
| 426 | struct user_msghdr __user *umsg, unsigned flags, | ||
| 427 | struct iovec **iov); | ||
| 428 | extern int recvmsg_copy_msghdr(struct msghdr *msg, | ||
| 429 | struct user_msghdr __user *umsg, unsigned flags, | ||
| 430 | struct sockaddr __user **uaddr, | ||
| 431 | struct iovec **iov); | ||
| 432 | extern int __copy_msghdr(struct msghdr *kmsg, | 425 | extern int __copy_msghdr(struct msghdr *kmsg, |
| 433 | struct user_msghdr *umsg, | 426 | struct user_msghdr *umsg, |
| 434 | struct sockaddr __user **save_addr); | 427 | struct sockaddr __user **save_addr); |
| @@ -354,7 +354,7 @@ __ping_ipv4() · 353 unmodified lines | |||
| 354 | 354 | ||
| 355 | # Send 100 packets and verify that at least 100 packets hit the rule, | 355 | # Send 100 packets and verify that at least 100 packets hit the rule, |
| 356 | # to overcome ARP noise. | 356 | # to overcome ARP noise. |
| 357 | PING_COUNT=100 PING_TIMEOUT=11 ping_do $dev $dst_ip | 357 | PING_COUNT=100 PING_TIMEOUT=20 ping_do $dev $dst_ip |
| 358 | check_err $? "Ping failed" | 358 | check_err $? "Ping failed" |
| 359 | 359 | ||
| 360 | tc_check_at_least_x_packets "dev $rp1 egress" 101 10 100 | 360 | tc_check_at_least_x_packets "dev $rp1 egress" 101 10 100 |
| @@ -410,7 +410,7 @@ __ping_ipv6() · 49 unmodified lines | |||
| 410 | 410 | ||
| 411 | # Send 100 packets and verify that at least 100 packets hit the rule, | 411 | # Send 100 packets and verify that at least 100 packets hit the rule, |
| 412 | # to overcome neighbor discovery noise. | 412 | # to overcome neighbor discovery noise. |
| 413 | PING_COUNT=100 PING_TIMEOUT=11 ping6_do $dev $dst_ip | 413 | PING_COUNT=100 PING_TIMEOUT=20 ping6_do $dev $dst_ip |
| 414 | check_err $? "Ping failed" | 414 | check_err $? "Ping failed" |
| 415 | 415 | ||
| 416 | tc_check_at_least_x_packets "dev $rp1 egress" 101 100 | 416 | tc_check_at_least_x_packets "dev $rp1 egress" 101 100 |
| @@ -457,7 +457,7 @@ __ping_ipv4() · 456 unmodified lines | |||
| 457 | 457 | ||
| 458 | # Send 100 packets and verify that at least 100 packets hit the rule, | 458 | # Send 100 packets and verify that at least 100 packets hit the rule, |
| 459 | # to overcome ARP noise. | 459 | # to overcome ARP noise. |
| 460 | PING_COUNT=100 PING_TIMEOUT=11 ping_do $dev $dst_ip | 460 | PING_COUNT=100 PING_TIMEOUT=20 ping_do $dev $dst_ip |
| 461 | check_err $? "Ping failed" | 461 | check_err $? "Ping failed" |
| 462 | 462 | ||
| 463 | tc_check_at_least_x_packets "dev $rp1 egress" 101 10 100 | 463 | tc_check_at_least_x_packets "dev $rp1 egress" 101 10 100 |
| @@ -522,7 +522,7 @@ __ping_ipv6() · 58 unmodified lines | |||
| 522 | 522 | ||
| 523 | # Send 100 packets and verify that at least 100 packets hit the rule, | 523 | # Send 100 packets and verify that at least 100 packets hit the rule, |
| 524 | # to overcome neighbor discovery noise. | 524 | # to overcome neighbor discovery noise. |
| 525 | PING_COUNT=100 PING_TIMEOUT=11 ping6_do $dev $dst_ip | 525 | PING_COUNT=100 PING_TIMEOUT=20 ping6_do $dev $dst_ip |
| 526 | check_err $? "Ping failed" | 526 | check_err $? "Ping failed" |
| 527 | 527 | ||
| 528 | tc_check_at_least_x_packets "dev $rp1 egress" 101 100 | 528 | tc_check_at_least_x_packets "dev $rp1 egress" 101 100 |
| @@ -217,6 +217,7 @@ for family in 4 6; do · 216 unmodified lines | |||
| 217 | cleanup | 217 | cleanup |
| 218 | 218 | ||
| 219 | create_ns | 219 | create_ns |
| 220 | ip netns exec $NS_DST ethtool -K veth$DST generic-receive-offload on | ||
| 220 | ip netns exec $NS_DST ethtool -K veth$DST rx-gro-list on | 221 | ip netns exec $NS_DST ethtool -K veth$DST rx-gro-list on |
| 221 | run_test "GRO frag list" $BM_NET$DST 1 0 | 222 | run_test "GRO frag list" $BM_NET$DST 1 0 |
| 222 | cleanup | 223 | cleanup |
| @@ -227,6 +228,7 @@ for family in 4 6; do · 4 unmodified lines | |||
| 227 | # use NAT to circumvent GRO FWD check | 228 | # use NAT to circumvent GRO FWD check |
| 228 | create_ns | 229 | create_ns |
| 229 | ip -n $NS_DST addr add dev veth$DST $BM_NET$DST_NAT/$SUFFIX | 230 | ip -n $NS_DST addr add dev veth$DST $BM_NET$DST_NAT/$SUFFIX |
| 231 | ip netns exec $NS_DST ethtool -K veth$DST generic-receive-offload on | ||
| 230 | ip netns exec $NS_DST ethtool -K veth$DST rx-udp-gro-forwarding on | 232 | ip netns exec $NS_DST ethtool -K veth$DST rx-udp-gro-forwarding on |
| 231 | ip netns exec $NS_DST $IPT -t nat -I PREROUTING -d $BM_NET$DST_NAT \ | 233 | ip netns exec $NS_DST $IPT -t nat -I PREROUTING -d $BM_NET$DST_NAT \ |
| 232 | -j DNAT --to-destination $BM_NET$DST | 234 | -j DNAT --to-destination $BM_NET$DST |
| @@ -240,6 +242,7 @@ for family in 4 6; do · 7 unmodified lines | |||
| 240 | cleanup | 242 | cleanup |
| 241 | 243 | ||
| 242 | create_vxlan_pair | 244 | create_vxlan_pair |
| 245 | ip netns exec $NS_DST ethtool -K veth$DST generic-receive-offload on | ||
| 243 | ip netns exec $NS_DST ethtool -K veth$DST rx-gro-list on | 246 | ip netns exec $NS_DST ethtool -K veth$DST rx-gro-list on |
| 244 | run_test "GRO frag list over UDP tunnel" $OL_NET$DST 1 1 | 247 | run_test "GRO frag list over UDP tunnel" $OL_NET$DST 1 1 |
| 245 | cleanup | 248 | cleanup |
| @@ -247,6 +250,7 @@ for family in 4 6; do · 1 unmodified line | |||
| 247 | # use NAT to circumvent GRO FWD check | 250 | # use NAT to circumvent GRO FWD check |
| 248 | create_vxlan_pair | 251 | create_vxlan_pair |
| 249 | ip -n $NS_DST addr add dev $VXDEV$DST $OL_NET$DST_NAT/$SUFFIX | 252 | ip -n $NS_DST addr add dev $VXDEV$DST $OL_NET$DST_NAT/$SUFFIX |
| 253 | ip netns exec $NS_DST ethtool -K veth$DST generic-receive-offload on | ||
| 250 | ip netns exec $NS_DST ethtool -K veth$DST rx-udp-gro-forwarding on | 254 | ip netns exec $NS_DST ethtool -K veth$DST rx-udp-gro-forwarding on |
| 251 | ip netns exec $NS_DST $IPT -t nat -I PREROUTING -d $OL_NET$DST_NAT \ | 255 | ip netns exec $NS_DST $IPT -t nat -I PREROUTING -d $OL_NET$DST_NAT \ |
| 252 | -j DNAT --to-destination $OL_NET$DST | 256 | -j DNAT --to-destination $OL_NET$DST |
| @@ -249,9 +249,9 @@ cleanup · 248 unmodified lines | |||
| 249 | create_ns | 249 | create_ns |
| 250 | ip -n $NS_DST link set dev veth$DST up | 250 | ip -n $NS_DST link set dev veth$DST up |
| 251 | ip -n $NS_DST link set dev veth$DST xdp object ${BPF_FILE} section xdp | 251 | ip -n $NS_DST link set dev veth$DST xdp object ${BPF_FILE} section xdp |
| 252 | chk_gro_flag "gro vs xdp while down - gro flag on" $DST on | 252 | chk_gro_flag "gro vs xdp while down - gro flag off" $DST off |
| 253 | ip -n $NS_DST link set dev veth$DST down | 253 | ip -n $NS_DST link set dev veth$DST down |
| 254 | chk_gro_flag " - after down" $DST on | 254 | chk_gro_flag " - after down" $DST off |
| 255 | ip -n $NS_DST link set dev veth$DST xdp off | 255 | ip -n $NS_DST link set dev veth$DST xdp off |
| 256 | chk_gro_flag " - after xdp off" $DST off | 256 | chk_gro_flag " - after xdp off" $DST off |
| 257 | ip -n $NS_DST link set dev veth$DST up | 257 | ip -n $NS_DST link set dev veth$DST up |
| @@ -260,6 +260,21 @@ ip -n $NS_SRC link set dev veth$SRC xdp object ${BPF_FILE} section xdp · 2 unmodified lines | |||
| 260 | chk_gro_flag " - after peer xdp" $DST off | 260 | chk_gro_flag " - after peer xdp" $DST off |
| 261 | cleanup | 261 | cleanup |
| 262 | 262 | ||
| 263 | create_ns | ||
| 264 | ip -n $NS_DST link set dev veth$DST up | ||
| 265 | ip -n $NS_DST link set dev veth$DST xdp object ${BPF_FILE} section xdp | ||
| 266 | ip netns exec $NS_DST ethtool -K veth$DST generic-receive-offload on | ||
| 267 | chk_gro_flag "gro vs xdp while down - gro flag on" $DST on | ||
| 268 | ip -n $NS_DST link set dev veth$DST down | ||
| 269 | chk_gro_flag " - after down" $DST on | ||
| 270 | ip -n $NS_DST link set dev veth$DST xdp off | ||
| 271 | chk_gro_flag " - after xdp off" $DST on | ||
| 272 | ip -n $NS_DST link set dev veth$DST up | ||
| 273 | chk_gro_flag " - after up" $DST on | ||
| 274 | ip -n $NS_SRC link set dev veth$SRC xdp object ${BPF_FILE} section xdp | ||
| 275 | chk_gro_flag " - after peer xdp" $DST on | ||
| 276 | cleanup | ||
| 277 | |||
| 263 | create_ns | 278 | create_ns |
| 264 | chk_channels "default channels" $DST 1 1 | 279 | chk_channels "default channels" $DST 1 1 |
| 265 | 280 | ||
| @@ -327,11 +342,14 @@ if [ $CPUS -gt 2 ]; then · 61 unmodified lines | |||
| 327 | fi | 342 | fi |
| 328 | 343 | ||
| 329 | ip -n $NS_DST link set dev veth$DST xdp object ${BPF_FILE} section xdp 2>/dev/null | 344 | ip -n $NS_DST link set dev veth$DST xdp object ${BPF_FILE} section xdp 2>/dev/null |
| 330 | chk_gro_flag "with xdp attached - gro flag" $DST on | 345 | chk_gro_flag "with xdp attached - gro flag" $DST off |
| 331 | chk_gro_flag " - peer gro flag" $SRC off | 346 | chk_gro_flag " - peer gro flag" $SRC off |
| 332 | chk_tso_flag " - tso flag" $SRC off | 347 | chk_tso_flag " - tso flag" $SRC off |
| 333 | chk_tso_flag " - peer tso flag" $DST on | 348 | chk_tso_flag " - peer tso flag" $DST on |
| 334 | ip netns exec $NS_DST ethtool -K veth$DST rx-udp-gro-forwarding on | 349 | ip netns exec $NS_DST ethtool -K veth$DST rx-udp-gro-forwarding on |
| 350 | chk_gro " - no aggregation" 10 | ||
| 351 | ip netns exec $NS_DST ethtool -K veth$DST generic-receive-offload on | ||
| 352 | chk_gro_flag " - gro flag with GRO on" $DST on | ||
| 335 | chk_gro " - aggregation" 1 | 353 | chk_gro " - aggregation" 1 |
| 336 | 354 | ||
| 337 | 355 | ||
| @@ -3,6 +3,7 @@ CONFIG_ARCH_RV32I=y · 2 unmodified lines | |||
| 3 | CONFIG_MMU=y | 3 | CONFIG_MMU=y |
| 4 | CONFIG_FPU=y | 4 | CONFIG_FPU=y |
| 5 | CONFIG_SOC_VIRT=y | 5 | CONFIG_SOC_VIRT=y |
| 6 | CONFIG_RISCV_ISA_FALLBACK=y | ||
| 6 | CONFIG_SERIAL_8250=y | 7 | CONFIG_SERIAL_8250=y |
| 7 | CONFIG_SERIAL_8250_CONSOLE=y | 8 | CONFIG_SERIAL_8250_CONSOLE=y |
| 8 | CONFIG_SERIAL_OF_PLATFORM=y | 9 | CONFIG_SERIAL_OF_PLATFORM=y |
| @@ -2,6 +2,7 @@ CONFIG_ARCH_RV64I=y · 1 unmodified line | |||
| 2 | CONFIG_MMU=y | 2 | CONFIG_MMU=y |
| 3 | CONFIG_FPU=y | 3 | CONFIG_FPU=y |
| 4 | CONFIG_SOC_VIRT=y | 4 | CONFIG_SOC_VIRT=y |
| 5 | CONFIG_RISCV_ISA_FALLBACK=y | ||
| 5 | CONFIG_SERIAL_8250=y | 6 | CONFIG_SERIAL_8250=y |
| 6 | CONFIG_SERIAL_8250_CONSOLE=y | 7 | CONFIG_SERIAL_8250_CONSOLE=y |
| 7 | CONFIG_SERIAL_OF_PLATFORM=y | 8 | CONFIG_SERIAL_OF_PLATFORM=y |