Saeed Mahameed says: ==================== mlx5 tc flow handling for concurrent execution (Part 3) This series includes updates to mlx5 ethernet and core driver: Vlad submits part 3 of 3 part series to allow TC flow handling for concurrent execution. Vlad says: ========== Structure mlx5e_neigh_hash_entry code that uses it are refactored in following ways: - Extend neigh_hash_entry with rcu and modify its users to always take reference to the structure when using it (neigh_hash_entry has already had atomic reference counter which was only used when scheduling neigh update on workqueue from atomic context of neigh update netevent). - Always use mlx5e_neigh_update_table->encap_lock when modifying neigh update hash table and list. Originally, this lock was only used to synchronize with netevent handler function, which is called from bh context and cannot use rtnl lock for synchronization. Use rcu read lock instead of encap_lock to lookup nhe in atomic context of netevent even handler function. Convert encap_lock to mutex to allow creating new neigh hash entries while holding it, which is safe to do because the lock is no longer used in atomic context. - Rcu-ify mlx5e_neigh_hash_entry->encap_list by changing operations on encap list to their rcu counterparts and extending encap structure with rcu_head to free the encap instances after rcu grace period. This allows fast traversal of list of encaps attached to nhe under rcu read lock protection. - Take encap_table_lock when accessing encap entries in neigh update and neigh stats update code to protect from concurrent encap entry insertion or removal. This approach leads to potential race condition when neigh update and neigh stats update code can access encap and flow entries that are not fully initialized or are being destroyed, or neigh can change state without updating encaps that are created concurrently. Prevent these issues by following changes in flow and encap initialization: - Extend mlx5e_tc_flow with 'init_done' completion. Modify neigh update to wait for both encap and flow completions to prevent concurrent access to a structure that is being initialized by tc. - Skip structures that failed during initialization: encaps with encap_id<0 and flows that don't have OFFLOADED flag set. - To ensure that no new flows are added to encap when it is being accessed by neigh update or neigh stats update, take encap_table_lock mutex. - To prevent concurrent deletion by tc, ensure that neigh update and neigh stats update hold references to encap and flow instances while using them. With changes presented in this patch set it is now safe to execute tc concurrently with neigh update and neigh stats update. However, these two workqueue tasks modify same flow "tmp_list" field to store flows with reference taken in temporary list to release the references after update operation finishes and should not be executed concurrently with each other. Last 3 patches of this series provide 3 new mlx5 trace points to track mlx5 tc requests and mlx5 neigh updates. ====================
Unified · Split
| @@ -12,6 +12,7 @@ Contents · 11 unmodified lines | |||
| 12 | - `Enabling the driver and kconfig options`_ | 12 | - `Enabling the driver and kconfig options`_ |
| 13 | - `Devlink info`_ | 13 | - `Devlink info`_ |
| 14 | - `Devlink health reporters`_ | 14 | - `Devlink health reporters`_ |
| 15 | - `mlx5 tracepoints`_ | ||
| 15 | 16 | ||
| 16 | Enabling the driver and kconfig options | 17 | Enabling the driver and kconfig options |
| 17 | ================================================ | 18 | ================================================ |
| @@ -219,3 +220,48 @@ User commands examples: · 201 unmodified lines | |||
| 219 | $ devlink health dump show pci/0000:82:00.1 reporter fw_fatal | 220 | $ devlink health dump show pci/0000:82:00.1 reporter fw_fatal |
| 220 | 221 | ||
| 221 | NOTE: This command can run only on PF. | 222 | NOTE: This command can run only on PF. |
| 223 | |||
| 224 | mlx5 tracepoints | ||
| 225 | ================ | ||
| 226 | |||
| 227 | mlx5 driver provides internal trace points for tracking and debugging using | ||
| 228 | kernel tracepoints interfaces (refer to Documentation/trace/ftrase.rst). | ||
| 229 | |||
| 230 | For the list of support mlx5 events check /sys/kernel/debug/tracing/events/mlx5/ | ||
| 231 | |||
| 232 | tc and eswitch offloads tracepoints: | ||
| 233 | |||
| 234 | - mlx5e_configure_flower: trace flower filter actions and cookies offloaded to mlx5:: | ||
| 235 | |||
| 236 | $ echo mlx5:mlx5e_configure_flower >> /sys/kernel/debug/tracing/set_event | ||
| 237 | $ cat /sys/kernel/debug/tracing/trace | ||
| 238 | ... | ||
| 239 | tc-6535 [019] ...1 2672.404466: mlx5e_configure_flower: cookie=0000000067874a55 actions= REDIRECT | ||
| 240 | |||
| 241 | - mlx5e_delete_flower: trace flower filter actions and cookies deleted from mlx5:: | ||
| 242 | |||
| 243 | $ echo mlx5:mlx5e_delete_flower >> /sys/kernel/debug/tracing/set_event | ||
| 244 | $ cat /sys/kernel/debug/tracing/trace | ||
| 245 | ... | ||
| 246 | tc-6569 [010] .N.1 2686.379075: mlx5e_delete_flower: cookie=0000000067874a55 actions= NULL | ||
| 247 | |||
| 248 | - mlx5e_stats_flower: trace flower stats request:: | ||
| 249 | |||
| 250 | $ echo mlx5:mlx5e_stats_flower >> /sys/kernel/debug/tracing/set_event | ||
| 251 | $ cat /sys/kernel/debug/tracing/trace | ||
| 252 | ... | ||
| 253 | tc-6546 [010] ...1 2679.704889: mlx5e_stats_flower: cookie=0000000060eb3d6a bytes=0 packets=0 lastused=4295560217 | ||
| 254 | |||
| 255 | - mlx5e_tc_update_neigh_used_value: trace tunnel rule neigh update value offloaded to mlx5:: | ||
| 256 | |||
| 257 | $ echo mlx5:mlx5e_tc_update_neigh_used_value >> /sys/kernel/debug/tracing/set_event | ||
| 258 | $ cat /sys/kernel/debug/tracing/trace | ||
| 259 | ... | ||
| 260 | kworker/u48:4-8806 [009] ...1 55117.882428: mlx5e_tc_update_neigh_used_value: netdev: ens1f0 IPv4: 1.1.1.10 IPv6: ::ffff:1.1.1.10 neigh_used=1 | ||
| 261 | |||
| 262 | - mlx5e_rep_neigh_update: trace neigh update tasks scheduled due to neigh state change events:: | ||
| 263 | |||
| 264 | $ echo mlx5:mlx5e_rep_neigh_update >> /sys/kernel/debug/tracing/set_event | ||
| 265 | $ cat /sys/kernel/debug/tracing/trace | ||
| 266 | ... | ||
| 267 | kworker/u48:7-2221 [009] ...1 1475.387435: mlx5e_rep_neigh_update: netdev: ens1f0 MAC: 24:8a:07:9a:17:9a IPv4: 1.1.1.10 IPv6: ::ffff:1.1.1.10 neigh_connected=1 | ||
| @@ -35,7 +35,7 @@ mlx5_core-$(CONFIG_MLX5_EN_RXNFC) += en_fs_ethtool.o · 34 unmodified lines | |||
| 35 | mlx5_core-$(CONFIG_MLX5_CORE_EN_DCB) += en_dcbnl.o en/port_buffer.o | 35 | mlx5_core-$(CONFIG_MLX5_CORE_EN_DCB) += en_dcbnl.o en/port_buffer.o |
| 36 | mlx5_core-$(CONFIG_MLX5_ESWITCH) += en_rep.o en_tc.o en/tc_tun.o lib/port_tun.o lag_mp.o \ | 36 | mlx5_core-$(CONFIG_MLX5_ESWITCH) += en_rep.o en_tc.o en/tc_tun.o lib/port_tun.o lag_mp.o \ |
| 37 | lib/geneve.o en/tc_tun_vxlan.o en/tc_tun_gre.o \ | 37 | lib/geneve.o en/tc_tun_vxlan.o en/tc_tun_gre.o \ |
| 38 | en/tc_tun_geneve.o | 38 | en/tc_tun_geneve.o diag/en_tc_tracepoint.o |
| 39 | 39 | ||
| 40 | # | 40 | # |
| 41 | # Core extra | 41 | # Core extra |
| @@ -0,0 +1,54 @@ | |||
| 1 | /* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */ | ||
| 2 | /* Copyright (c) 2019 Mellanox Technologies. */ | ||
| 3 | |||
| 4 | #undef TRACE_SYSTEM | ||
| 5 | #define TRACE_SYSTEM mlx5 | ||
| 6 | |||
| 7 | #if !defined(_MLX5_EN_REP_TP_) || defined(TRACE_HEADER_MULTI_READ) | ||
| 8 | #define _MLX5_EN_REP_TP_ | ||
| 9 | |||
| 10 | #include <linux/tracepoint.h> | ||
| 11 | #include <linux/trace_seq.h> | ||
| 12 | #include "en_rep.h" | ||
| 13 | |||
| 14 | TRACE_EVENT(mlx5e_rep_neigh_update, | ||
| 15 | TP_PROTO(const struct mlx5e_neigh_hash_entry *nhe, const u8 *ha, | ||
| 16 | bool neigh_connected), | ||
| 17 | TP_ARGS(nhe, ha, neigh_connected), | ||
| 18 | TP_STRUCT__entry(__string(devname, nhe->m_neigh.dev->name) | ||
| 19 | __array(u8, ha, ETH_ALEN) | ||
| 20 | __array(u8, v4, 4) | ||
| 21 | __array(u8, v6, 16) | ||
| 22 | __field(bool, neigh_connected) | ||
| 23 | ), | ||
| 24 | TP_fast_assign(const struct mlx5e_neigh *mn = &nhe->m_neigh; | ||
| 25 | struct in6_addr *pin6; | ||
| 26 | __be32 *p32; | ||
| 27 | |||
| 28 | __assign_str(devname, mn->dev->name); | ||
| 29 | __entry->neigh_connected = neigh_connected; | ||
| 30 | memcpy(__entry->ha, ha, ETH_ALEN); | ||
| 31 | |||
| 32 | p32 = (__be32 *)__entry->v4; | ||
| 33 | pin6 = (struct in6_addr *)__entry->v6; | ||
| 34 | if (mn->family == AF_INET) { | ||
| 35 | *p32 = mn->dst_ip.v4; | ||
| 36 | ipv6_addr_set_v4mapped(*p32, pin6); | ||
| 37 | } else if (mn->family == AF_INET6) { | ||
| 38 | *pin6 = mn->dst_ip.v6; | ||
| 39 | } | ||
| 40 | ), | ||
| 41 | TP_printk("netdev: %s MAC: %pM IPv4: %pI4 IPv6: %pI6c neigh_connected=%d\n", | ||
| 42 | __get_str(devname), __entry->ha, | ||
| 43 | __entry->v4, __entry->v6, __entry->neigh_connected | ||
| 44 | ) | ||
| 45 | ); | ||
| 46 | |||
| 47 | #endif /* _MLX5_EN_REP_TP_ */ | ||
| 48 | |||
| 49 | /* This part must be outside protection */ | ||
| 50 | #undef TRACE_INCLUDE_PATH | ||
| 51 | #define TRACE_INCLUDE_PATH ./diag | ||
| 52 | #undef TRACE_INCLUDE_FILE | ||
| 53 | #define TRACE_INCLUDE_FILE en_rep_tracepoint | ||
| 54 | #include <trace/define_trace.h> | ||
| @@ -0,0 +1,58 @@ | |||
| 1 | // SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB | ||
| 2 | /* Copyright (c) 2019 Mellanox Technologies. */ | ||
| 3 | |||
| 4 | #define CREATE_TRACE_POINTS | ||
| 5 | #include "en_tc_tracepoint.h" | ||
| 6 | |||
| 7 | void put_ids_to_array(int *ids, | ||
| 8 | const struct flow_action_entry *entries, | ||
| 9 | unsigned int num) | ||
| 10 | { | ||
| 11 | unsigned int i; | ||
| 12 | |||
| 13 | for (i = 0; i < num; i++) | ||
| 14 | ids[i] = entries[i].id; | ||
| 15 | } | ||
| 16 | |||
| 17 | #define NAME_SIZE 16 | ||
| 18 | |||
| 19 | static const char FLOWACT2STR[NUM_FLOW_ACTIONS][NAME_SIZE] = { | ||
| 20 | [FLOW_ACTION_ACCEPT] = "ACCEPT", | ||
| 21 | [FLOW_ACTION_DROP] = "DROP", | ||
| 22 | [FLOW_ACTION_TRAP] = "TRAP", | ||
| 23 | [FLOW_ACTION_GOTO] = "GOTO", | ||
| 24 | [FLOW_ACTION_REDIRECT] = "REDIRECT", | ||
| 25 | [FLOW_ACTION_MIRRED] = "MIRRED", | ||
| 26 | [FLOW_ACTION_VLAN_PUSH] = "VLAN_PUSH", | ||
| 27 | [FLOW_ACTION_VLAN_POP] = "VLAN_POP", | ||
| 28 | [FLOW_ACTION_VLAN_MANGLE] = "VLAN_MANGLE", | ||
| 29 | [FLOW_ACTION_TUNNEL_ENCAP] = "TUNNEL_ENCAP", | ||
| 30 | [FLOW_ACTION_TUNNEL_DECAP] = "TUNNEL_DECAP", | ||
| 31 | [FLOW_ACTION_MANGLE] = "MANGLE", | ||
| 32 | [FLOW_ACTION_ADD] = "ADD", | ||
| 33 | [FLOW_ACTION_CSUM] = "CSUM", | ||
| 34 | [FLOW_ACTION_MARK] = "MARK", | ||
| 35 | [FLOW_ACTION_WAKE] = "WAKE", | ||
| 36 | [FLOW_ACTION_QUEUE] = "QUEUE", | ||
| 37 | [FLOW_ACTION_SAMPLE] = "SAMPLE", | ||
| 38 | [FLOW_ACTION_POLICE] = "POLICE", | ||
| 39 | [FLOW_ACTION_CT] = "CT", | ||
| 40 | }; | ||
| 41 | |||
| 42 | const char *parse_action(struct trace_seq *p, | ||
| 43 | int *ids, | ||
| 44 | unsigned int num) | ||
| 45 | { | ||
| 46 | const char *ret = trace_seq_buffer_ptr(p); | ||
| 47 | unsigned int i; | ||
| 48 | |||
| 49 | for (i = 0; i < num; i++) { | ||
| 50 | if (ids[i] < NUM_FLOW_ACTIONS) | ||
| 51 | trace_seq_printf(p, "%s ", FLOWACT2STR[ids[i]]); | ||
| 52 | else | ||
| 53 | trace_seq_printf(p, "UNKNOWN "); | ||
| 54 | } | ||
| 55 | |||
| 56 | trace_seq_putc(p, 0); | ||
| 57 | return ret; | ||
| 58 | } | ||
| @@ -0,0 +1,114 @@ | |||
| 1 | /* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */ | ||
| 2 | /* Copyright (c) 2019 Mellanox Technologies. */ | ||
| 3 | |||
| 4 | #undef TRACE_SYSTEM | ||
| 5 | #define TRACE_SYSTEM mlx5 | ||
| 6 | |||
| 7 | #if !defined(_MLX5_TC_TP_) || defined(TRACE_HEADER_MULTI_READ) | ||
| 8 | #define _MLX5_TC_TP_ | ||
| 9 | |||
| 10 | #include <linux/tracepoint.h> | ||
| 11 | #include <linux/trace_seq.h> | ||
| 12 | #include <net/flow_offload.h> | ||
| 13 | #include "en_rep.h" | ||
| 14 | |||
| 15 | #define __parse_action(ids, num) parse_action(p, ids, num) | ||
| 16 | |||
| 17 | void put_ids_to_array(int *ids, | ||
| 18 | const struct flow_action_entry *entries, | ||
| 19 | unsigned int num); | ||
| 20 | |||
| 21 | const char *parse_action(struct trace_seq *p, | ||
| 22 | int *ids, | ||
| 23 | unsigned int num); | ||
| 24 | |||
| 25 | DECLARE_EVENT_CLASS(mlx5e_flower_template, | ||
| 26 | TP_PROTO(const struct flow_cls_offload *f), | ||
| 27 | TP_ARGS(f), | ||
| 28 | TP_STRUCT__entry(__field(void *, cookie) | ||
| 29 | __field(unsigned int, num) | ||
| 30 | __dynamic_array(int, ids, f->rule ? | ||
| 31 | f->rule->action.num_entries : 0) | ||
| 32 | ), | ||
| 33 | TP_fast_assign(__entry->cookie = (void *)f->cookie; | ||
| 34 | __entry->num = (f->rule ? | ||
| 35 | f->rule->action.num_entries : 0); | ||
| 36 | if (__entry->num) | ||
| 37 | put_ids_to_array(__get_dynamic_array(ids), | ||
| 38 | f->rule->action.entries, | ||
| 39 | f->rule->action.num_entries); | ||
| 40 | ), | ||
| 41 | TP_printk("cookie=%p actions= %s\n", | ||
| 42 | __entry->cookie, __entry->num ? | ||
| 43 | __parse_action(__get_dynamic_array(ids), | ||
| 44 | __entry->num) : "NULL" | ||
| 45 | ) | ||
| 46 | ); | ||
| 47 | |||
| 48 | DEFINE_EVENT(mlx5e_flower_template, mlx5e_configure_flower, | ||
| 49 | TP_PROTO(const struct flow_cls_offload *f), | ||
| 50 | TP_ARGS(f) | ||
| 51 | ); | ||
| 52 | |||
| 53 | DEFINE_EVENT(mlx5e_flower_template, mlx5e_delete_flower, | ||
| 54 | TP_PROTO(const struct flow_cls_offload *f), | ||
| 55 | TP_ARGS(f) | ||
| 56 | ); | ||
| 57 | |||
| 58 | TRACE_EVENT(mlx5e_stats_flower, | ||
| 59 | TP_PROTO(const struct flow_cls_offload *f), | ||
| 60 | TP_ARGS(f), | ||
| 61 | TP_STRUCT__entry(__field(void *, cookie) | ||
| 62 | __field(u64, bytes) | ||
| 63 | __field(u64, packets) | ||
| 64 | __field(u64, lastused) | ||
| 65 | ), | ||
| 66 | TP_fast_assign(__entry->cookie = (void *)f->cookie; | ||
| 67 | __entry->bytes = f->stats.bytes; | ||
| 68 | __entry->packets = f->stats.pkts; | ||
| 69 | __entry->lastused = f->stats.lastused; | ||
| 70 | ), | ||
| 71 | TP_printk("cookie=%p bytes=%llu packets=%llu lastused=%llu\n", | ||
| 72 | __entry->cookie, __entry->bytes, | ||
| 73 | __entry->packets, __entry->lastused | ||
| 74 | ) | ||
| 75 | ); | ||
| 76 | |||
| 77 | TRACE_EVENT(mlx5e_tc_update_neigh_used_value, | ||
| 78 | TP_PROTO(const struct mlx5e_neigh_hash_entry *nhe, bool neigh_used), | ||
| 79 | TP_ARGS(nhe, neigh_used), | ||
| 80 | TP_STRUCT__entry(__string(devname, nhe->m_neigh.dev->name) | ||
| 81 | __array(u8, v4, 4) | ||
| 82 | __array(u8, v6, 16) | ||
| 83 | __field(bool, neigh_used) | ||
| 84 | ), | ||
| 85 | TP_fast_assign(const struct mlx5e_neigh *mn = &nhe->m_neigh; | ||
| 86 | struct in6_addr *pin6; | ||
| 87 | __be32 *p32; | ||
| 88 | |||
| 89 | __assign_str(devname, mn->dev->name); | ||
| 90 | __entry->neigh_used = neigh_used; | ||
| 91 | |||
| 92 | p32 = (__be32 *)__entry->v4; | ||
| 93 | pin6 = (struct in6_addr *)__entry->v6; | ||
| 94 | if (mn->family == AF_INET) { | ||
| 95 | *p32 = mn->dst_ip.v4; | ||
| 96 | ipv6_addr_set_v4mapped(*p32, pin6); | ||
| 97 | } else if (mn->family == AF_INET6) { | ||
| 98 | *pin6 = mn->dst_ip.v6; | ||
| 99 | } | ||
| 100 | ), | ||
| 101 | TP_printk("netdev: %s IPv4: %pI4 IPv6: %pI6c neigh_used=%d\n", | ||
| 102 | __get_str(devname), __entry->v4, __entry->v6, | ||
| 103 | __entry->neigh_used | ||
| 104 | ) | ||
| 105 | ); | ||
| 106 | |||
| 107 | #endif /* _MLX5_TC_TP_ */ | ||
| 108 | |||
| 109 | /* This part must be outside protection */ | ||
| 110 | #undef TRACE_INCLUDE_PATH | ||
| 111 | #define TRACE_INCLUDE_PATH ./diag | ||
| 112 | #undef TRACE_INCLUDE_FILE | ||
| 113 | #define TRACE_INCLUDE_FILE en_tc_tracepoint | ||
| 114 | #include <trace/define_trace.h> | ||
| @@ -46,6 +46,8 @@ · 45 unmodified lines | |||
| 46 | #include "en/tc_tun.h" | 46 | #include "en/tc_tun.h" |
| 47 | #include "fs_core.h" | 47 | #include "fs_core.h" |
| 48 | #include "lib/port_tun.h" | 48 | #include "lib/port_tun.h" |
| 49 | #define CREATE_TRACE_POINTS | ||
| 50 | #include "diag/en_rep_tracepoint.h" | ||
| 49 | 51 | ||
| 50 | #define MLX5E_REP_PARAMS_DEF_LOG_SQ_SIZE \ | 52 | #define MLX5E_REP_PARAMS_DEF_LOG_SQ_SIZE \ |
| 51 | max(0x7, MLX5E_PARAMS_MINIMUM_LOG_SQ_SIZE) | 53 | max(0x7, MLX5E_PARAMS_MINIMUM_LOG_SQ_SIZE) |
| @@ -524,47 +526,97 @@ void mlx5e_rep_queue_neigh_stats_work(struct mlx5e_priv *priv) · 472 unmodified lines | |||
| 524 | neigh_update->min_interval); | 526 | neigh_update->min_interval); |
| 525 | } | 527 | } |
| 526 | 528 | ||
| 529 | static bool mlx5e_rep_neigh_entry_hold(struct mlx5e_neigh_hash_entry *nhe) | ||
| 530 | { | ||
| 531 | return refcount_inc_not_zero(&nhe->refcnt); | ||
| 532 | } | ||
| 533 | |||
| 534 | static void mlx5e_rep_neigh_entry_remove(struct mlx5e_neigh_hash_entry *nhe); | ||
| 535 | |||
| 536 | static void mlx5e_rep_neigh_entry_release(struct mlx5e_neigh_hash_entry *nhe) | ||
| 537 | { | ||
| 538 | if (refcount_dec_and_test(&nhe->refcnt)) { | ||
| 539 | mlx5e_rep_neigh_entry_remove(nhe); | ||
| 540 | kfree_rcu(nhe, rcu); | ||
| 541 | } | ||
| 542 | } | ||
| 543 | |||
| 544 | static struct mlx5e_neigh_hash_entry * | ||
| 545 | mlx5e_get_next_nhe(struct mlx5e_rep_priv *rpriv, | ||
| 546 | struct mlx5e_neigh_hash_entry *nhe) | ||
| 547 | { | ||
| 548 | struct mlx5e_neigh_hash_entry *next = NULL; | ||
| 549 | |||
| 550 | rcu_read_lock(); | ||
| 551 | |||
| 552 | for (next = nhe ? | ||
| 553 | list_next_or_null_rcu(&rpriv->neigh_update.neigh_list, | ||
| 554 | &nhe->neigh_list, | ||
| 555 | struct mlx5e_neigh_hash_entry, | ||
| 556 | neigh_list) : | ||
| 557 | list_first_or_null_rcu(&rpriv->neigh_update.neigh_list, | ||
| 558 | struct mlx5e_neigh_hash_entry, | ||
| 559 | neigh_list); | ||
| 560 | next; | ||
| 561 | next = list_next_or_null_rcu(&rpriv->neigh_update.neigh_list, | ||
| 562 | &next->neigh_list, | ||
| 563 | struct mlx5e_neigh_hash_entry, | ||
| 564 | neigh_list)) | ||
| 565 | if (mlx5e_rep_neigh_entry_hold(next)) | ||
| 566 | break; | ||
| 567 | |||
| 568 | rcu_read_unlock(); | ||
| 569 | |||
| 570 | if (nhe) | ||
| 571 | mlx5e_rep_neigh_entry_release(nhe); | ||
| 572 | |||
| 573 | return next; | ||
| 574 | } | ||
| 575 | |||
| 527 | static void mlx5e_rep_neigh_stats_work(struct work_struct *work) | 576 | static void mlx5e_rep_neigh_stats_work(struct work_struct *work) |
| 528 | { | 577 | { |
| 529 | struct mlx5e_rep_priv *rpriv = container_of(work, struct mlx5e_rep_priv, | 578 | struct mlx5e_rep_priv *rpriv = container_of(work, struct mlx5e_rep_priv, |
| 530 | neigh_update.neigh_stats_work.work); | 579 | neigh_update.neigh_stats_work.work); |
| 531 | struct net_device *netdev = rpriv->netdev; | 580 | struct net_device *netdev = rpriv->netdev; |
| 532 | struct mlx5e_priv *priv = netdev_priv(netdev); | 581 | struct mlx5e_priv *priv = netdev_priv(netdev); |
| 533 | struct mlx5e_neigh_hash_entry *nhe; | 582 | struct mlx5e_neigh_hash_entry *nhe = NULL; |
| 534 | 583 | ||
| 535 | rtnl_lock(); | 584 | rtnl_lock(); |
| 536 | if (!list_empty(&rpriv->neigh_update.neigh_list)) | 585 | if (!list_empty(&rpriv->neigh_update.neigh_list)) |
| 537 | mlx5e_rep_queue_neigh_stats_work(priv); | 586 | mlx5e_rep_queue_neigh_stats_work(priv); |
| 538 | 587 | ||
| 539 | list_for_each_entry(nhe, &rpriv->neigh_update.neigh_list, neigh_list) | 588 | while ((nhe = mlx5e_get_next_nhe(rpriv, nhe)) != NULL) |
| 540 | mlx5e_tc_update_neigh_used_value(nhe); | 589 | mlx5e_tc_update_neigh_used_value(nhe); |
| 541 | 590 | ||
| 542 | rtnl_unlock(); | 591 | rtnl_unlock(); |
| 543 | } | 592 | } |
| 544 | 593 | ||
| 545 | static void mlx5e_rep_neigh_entry_hold(struct mlx5e_neigh_hash_entry *nhe) | ||
| 546 | { | ||
| 547 | refcount_inc(&nhe->refcnt); | ||
| 548 | } | ||
| 549 | |||
| 550 | static void mlx5e_rep_neigh_entry_release(struct mlx5e_neigh_hash_entry *nhe) | ||
| 551 | { | ||
| 552 | if (refcount_dec_and_test(&nhe->refcnt)) | ||
| 553 | kfree(nhe); | ||
| 554 | } | ||
| 555 | |||
| 556 | static void mlx5e_rep_update_flows(struct mlx5e_priv *priv, | 594 | static void mlx5e_rep_update_flows(struct mlx5e_priv *priv, |
| 557 | struct mlx5e_encap_entry *e, | 595 | struct mlx5e_encap_entry *e, |
| 558 | bool neigh_connected, | 596 | bool neigh_connected, |
| 559 | unsigned char ha[ETH_ALEN]) | 597 | unsigned char ha[ETH_ALEN]) |
| 560 | { | 598 | { |
| 561 | struct ethhdr *eth = (struct ethhdr *)e->encap_header; | 599 | struct ethhdr *eth = (struct ethhdr *)e->encap_header; |
| 600 | struct mlx5_eswitch *esw = priv->mdev->priv.eswitch; | ||
| 601 | bool encap_connected; | ||
| 602 | LIST_HEAD(flow_list); | ||
| 562 | 603 | ||
| 563 | ASSERT_RTNL(); | 604 | ASSERT_RTNL(); |
| 564 | 605 | ||
| 606 | /* wait for encap to be fully initialized */ | ||
| 607 | wait_for_completion(&e->res_ready); | ||
| 608 | |||
| 609 | mutex_lock(&esw->offloads.encap_tbl_lock); | ||
| 610 | encap_connected = !!(e->flags & MLX5_ENCAP_ENTRY_VALID); | ||
| 611 | if (e->compl_result || (encap_connected == neigh_connected && | ||
| 612 | ether_addr_equal(e->h_dest, ha))) | ||
| 613 | goto unlock; | ||
| 614 | |||
| 615 | mlx5e_take_all_encap_flows(e, &flow_list); | ||
| 616 | |||
| 565 | if ((e->flags & MLX5_ENCAP_ENTRY_VALID) && | 617 | if ((e->flags & MLX5_ENCAP_ENTRY_VALID) && |
| 566 | (!neigh_connected || !ether_addr_equal(e->h_dest, ha))) | 618 | (!neigh_connected || !ether_addr_equal(e->h_dest, ha))) |
| 567 | mlx5e_tc_encap_flows_del(priv, e); | 619 | mlx5e_tc_encap_flows_del(priv, e, &flow_list); |
| 568 | 620 | ||
| 569 | if (neigh_connected && !(e->flags & MLX5_ENCAP_ENTRY_VALID)) { | 621 | if (neigh_connected && !(e->flags & MLX5_ENCAP_ENTRY_VALID)) { |
| 570 | ether_addr_copy(e->h_dest, ha); | 622 | ether_addr_copy(e->h_dest, ha); |
| @@ -574,8 +626,11 @@ static void mlx5e_rep_update_flows(struct mlx5e_priv *priv, · 3 unmodified lines | |||
| 574 | */ | 626 | */ |
| 575 | ether_addr_copy(eth->h_source, e->route_dev->dev_addr); | 627 | ether_addr_copy(eth->h_source, e->route_dev->dev_addr); |
| 576 | 628 | ||
| 577 | mlx5e_tc_encap_flows_add(priv, e); | 629 | mlx5e_tc_encap_flows_add(priv, e, &flow_list); |
| 578 | } | 630 | } |
| 631 | unlock: | ||
| 632 | mutex_unlock(&esw->offloads.encap_tbl_lock); | ||
| 633 | mlx5e_put_encap_flow_list(priv, &flow_list); | ||
| 579 | } | 634 | } |
| 580 | 635 | ||
| 581 | static void mlx5e_rep_neigh_update(struct work_struct *work) | 636 | static void mlx5e_rep_neigh_update(struct work_struct *work) |
| @@ -587,7 +642,6 @@ static void mlx5e_rep_neigh_update(struct work_struct *work) · 5 unmodified lines | |||
| 587 | unsigned char ha[ETH_ALEN]; | 642 | unsigned char ha[ETH_ALEN]; |
| 588 | struct mlx5e_priv *priv; | 643 | struct mlx5e_priv *priv; |
| 589 | bool neigh_connected; | 644 | bool neigh_connected; |
| 590 | bool encap_connected; | ||
| 591 | u8 nud_state, dead; | 645 | u8 nud_state, dead; |
| 592 | 646 | ||
| 593 | rtnl_lock(); | 647 | rtnl_lock(); |
| @@ -605,17 +659,14 @@ static void mlx5e_rep_neigh_update(struct work_struct *work) · 11 unmodified lines | |||
| 605 | 659 | ||
| 606 | neigh_connected = (nud_state & NUD_VALID) && !dead; | 660 | neigh_connected = (nud_state & NUD_VALID) && !dead; |
| 607 | 661 | ||
| 662 | trace_mlx5e_rep_neigh_update(nhe, ha, neigh_connected); | ||
| 663 | |||
| 608 | list_for_each_entry(e, &nhe->encap_list, encap_list) { | 664 | list_for_each_entry(e, &nhe->encap_list, encap_list) { |
| 609 | if (!mlx5e_encap_take(e)) | 665 | if (!mlx5e_encap_take(e)) |
| 610 | continue; | 666 | continue; |
| 611 | 667 | ||
| 612 | encap_connected = !!(e->flags & MLX5_ENCAP_ENTRY_VALID); | ||
| 613 | priv = netdev_priv(e->out_dev); | 668 | priv = netdev_priv(e->out_dev); |
| 614 | 669 | mlx5e_rep_update_flows(priv, e, neigh_connected, ha); | |
| 615 | if (encap_connected != neigh_connected || | ||
| 616 | !ether_addr_equal(e->h_dest, ha)) | ||
| 617 | mlx5e_rep_update_flows(priv, e, neigh_connected, ha); | ||
| 618 | |||
| 619 | mlx5e_encap_put(priv, e); | 670 | mlx5e_encap_put(priv, e); |
| 620 | } | 671 | } |
| 621 | mlx5e_rep_neigh_entry_release(nhe); | 672 | mlx5e_rep_neigh_entry_release(nhe); |
| @@ -821,6 +872,28 @@ static int mlx5e_nic_rep_netdevice_event(struct notifier_block *nb, · 199 unmodified lines | |||
| 821 | return NOTIFY_OK; | 872 | return NOTIFY_OK; |
| 822 | } | 873 | } |
| 823 | 874 | ||
| 875 | static void | ||
| 876 | mlx5e_rep_queue_neigh_update_work(struct mlx5e_priv *priv, | ||
| 877 | struct mlx5e_neigh_hash_entry *nhe, | ||
| 878 | struct neighbour *n) | ||
| 879 | { | ||
| 880 | /* Take a reference to ensure the neighbour and mlx5 encap | ||
| 881 | * entry won't be destructed until we drop the reference in | ||
| 882 | * delayed work. | ||
| 883 | */ | ||
| 884 | neigh_hold(n); | ||
| 885 | |||
| 886 | /* This assignment is valid as long as the the neigh reference | ||
| 887 | * is taken | ||
| 888 | */ | ||
| 889 | nhe->n = n; | ||
| 890 | |||
| 891 | if (!queue_work(priv->wq, &nhe->neigh_update_work)) { | ||
| 892 | mlx5e_rep_neigh_entry_release(nhe); | ||
| 893 | neigh_release(n); | ||
| 894 | } | ||
| 895 | } | ||
| 896 | |||
| 824 | static struct mlx5e_neigh_hash_entry * | 897 | static struct mlx5e_neigh_hash_entry * |
| 825 | mlx5e_rep_neigh_entry_lookup(struct mlx5e_priv *priv, | 898 | mlx5e_rep_neigh_entry_lookup(struct mlx5e_priv *priv, |
| 826 | struct mlx5e_neigh *m_neigh); | 899 | struct mlx5e_neigh *m_neigh); |
| @@ -853,34 +926,13 @@ static int mlx5e_rep_netevent_event(struct notifier_block *nb, · 26 unmodified lines | |||
| 853 | m_neigh.family = n->ops->family; | 926 | m_neigh.family = n->ops->family; |
| 854 | memcpy(&m_neigh.dst_ip, n->primary_key, n->tbl->key_len); | 927 | memcpy(&m_neigh.dst_ip, n->primary_key, n->tbl->key_len); |
| 855 | 928 | ||
| 856 | /* We are in atomic context and can't take RTNL mutex, so use | 929 | rcu_read_lock(); |
| 857 | * spin_lock_bh to lookup the neigh table. bh is used since | ||
| 858 | * netevent can be called from a softirq context. | ||
| 859 | */ | ||
| 860 | spin_lock_bh(&neigh_update->encap_lock); | ||
| 861 | nhe = mlx5e_rep_neigh_entry_lookup(priv, &m_neigh); | 930 | nhe = mlx5e_rep_neigh_entry_lookup(priv, &m_neigh); |
| 862 | if (!nhe) { | 931 | rcu_read_unlock(); |
| 863 | spin_unlock_bh(&neigh_update->encap_lock); | 932 | if (!nhe) |
| 864 | return NOTIFY_DONE; | 933 | return NOTIFY_DONE; |
| 865 | } | ||
| 866 | |||
| 867 | /* This assignment is valid as long as the the neigh reference | ||
| 868 | * is taken | ||
| 869 | */ | ||
| 870 | nhe->n = n; | ||
| 871 | |||
| 872 | /* Take a reference to ensure the neighbour and mlx5 encap | ||
| 873 | * entry won't be destructed until we drop the reference in | ||
| 874 | * delayed work. | ||
| 875 | */ | ||
| 876 | neigh_hold(n); | ||
| 877 | mlx5e_rep_neigh_entry_hold(nhe); | ||
| 878 | 934 | ||
| 879 | if (!queue_work(priv->wq, &nhe->neigh_update_work)) { | 935 | mlx5e_rep_queue_neigh_update_work(priv, nhe, n); |
| 880 | mlx5e_rep_neigh_entry_release(nhe); | ||
| 881 | neigh_release(n); | ||
| 882 | } | ||
| 883 | spin_unlock_bh(&neigh_update->encap_lock); | ||
| 884 | break; | 936 | break; |
| 885 | 937 | ||
| 886 | case NETEVENT_DELAY_PROBE_TIME_UPDATE: | 938 | case NETEVENT_DELAY_PROBE_TIME_UPDATE: |
| @@ -897,19 +949,15 @@ static int mlx5e_rep_netevent_event(struct notifier_block *nb, · 10 unmodified lines | |||
| 897 | #endif | 949 | #endif |
| 898 | return NOTIFY_DONE; | 950 | return NOTIFY_DONE; |
| 899 | 951 | ||
| 900 | /* We are in atomic context and can't take RTNL mutex, | 952 | rcu_read_lock(); |
| 901 | * so use spin_lock_bh to walk the neigh list and look for | 953 | list_for_each_entry_rcu(nhe, &neigh_update->neigh_list, |
| 902 | * the relevant device. bh is used since netevent can be | 954 | neigh_list) { |
| 903 | * called from a softirq context. | ||
| 904 | */ | ||
| 905 | spin_lock_bh(&neigh_update->encap_lock); | ||
| 906 | list_for_each_entry(nhe, &neigh_update->neigh_list, neigh_list) { | ||
| 907 | if (p->dev == nhe->m_neigh.dev) { | 955 | if (p->dev == nhe->m_neigh.dev) { |
| 908 | found = true; | 956 | found = true; |
| 909 | break; | 957 | break; |
| 910 | } | 958 | } |
| 911 | } | 959 | } |
| 912 | spin_unlock_bh(&neigh_update->encap_lock); | 960 | rcu_read_unlock(); |
| 913 | if (!found) | 961 | if (!found) |
| 914 | return NOTIFY_DONE; | 962 | return NOTIFY_DONE; |
| 915 | 963 | ||
| @@ -940,7 +988,7 @@ static int mlx5e_rep_neigh_init(struct mlx5e_rep_priv *rpriv) · 24 unmodified lines | |||
| 940 | return err; | 988 | return err; |
| 941 | 989 | ||
| 942 | INIT_LIST_HEAD(&neigh_update->neigh_list); | 990 | INIT_LIST_HEAD(&neigh_update->neigh_list); |
| 943 | spin_lock_init(&neigh_update->encap_lock); | 991 | mutex_init(&neigh_update->encap_lock); |
| 944 | INIT_DELAYED_WORK(&neigh_update->neigh_stats_work, | 992 | INIT_DELAYED_WORK(&neigh_update->neigh_stats_work, |
| 945 | mlx5e_rep_neigh_stats_work); | 993 | mlx5e_rep_neigh_stats_work); |
| 946 | mlx5e_rep_neigh_update_init_interval(rpriv); | 994 | mlx5e_rep_neigh_update_init_interval(rpriv); |
| @@ -967,6 +1015,7 @@ static void mlx5e_rep_neigh_cleanup(struct mlx5e_rep_priv *rpriv) · 20 unmodified lines | |||
| 967 | 1015 | ||
| 968 | cancel_delayed_work_sync(&rpriv->neigh_update.neigh_stats_work); | 1016 | cancel_delayed_work_sync(&rpriv->neigh_update.neigh_stats_work); |
| 969 | 1017 | ||
| 1018 | mutex_destroy(&neigh_update->encap_lock); | ||
| 970 | rhashtable_destroy(&neigh_update->neigh_ht); | 1019 | rhashtable_destroy(&neigh_update->neigh_ht); |
| 971 | } | 1020 | } |
| 972 | 1021 | ||
| @@ -982,28 +1031,27 @@ static int mlx5e_rep_neigh_entry_insert(struct mlx5e_priv *priv, · 9 unmodified lines | |||
| 982 | if (err) | 1031 | if (err) |
| 983 | return err; | 1032 | return err; |
| 984 | 1033 | ||
| 985 | list_add(&nhe->neigh_list, &rpriv->neigh_update.neigh_list); | 1034 | list_add_rcu(&nhe->neigh_list, &rpriv->neigh_update.neigh_list); |
| 986 | 1035 | ||
| 987 | return err; | 1036 | return err; |
| 988 | } | 1037 | } |
| 989 | 1038 | ||
| 990 | static void mlx5e_rep_neigh_entry_remove(struct mlx5e_priv *priv, | 1039 | static void mlx5e_rep_neigh_entry_remove(struct mlx5e_neigh_hash_entry *nhe) |
| 991 | struct mlx5e_neigh_hash_entry *nhe) | ||
| 992 | { | 1040 | { |
| 993 | struct mlx5e_rep_priv *rpriv = priv->ppriv; | 1041 | struct mlx5e_rep_priv *rpriv = nhe->priv->ppriv; |
| 994 | 1042 | ||
| 995 | spin_lock_bh(&rpriv->neigh_update.encap_lock); | 1043 | mutex_lock(&rpriv->neigh_update.encap_lock); |
| 996 | 1044 | ||
| 997 | list_del(&nhe->neigh_list); | 1045 | list_del_rcu(&nhe->neigh_list); |
| 998 | 1046 | ||
| 999 | rhashtable_remove_fast(&rpriv->neigh_update.neigh_ht, | 1047 | rhashtable_remove_fast(&rpriv->neigh_update.neigh_ht, |
| 1000 | &nhe->rhash_node, | 1048 | &nhe->rhash_node, |
| 1001 | mlx5e_neigh_ht_params); | 1049 | mlx5e_neigh_ht_params); |
| 1002 | spin_unlock_bh(&rpriv->neigh_update.encap_lock); | 1050 | mutex_unlock(&rpriv->neigh_update.encap_lock); |
| 1003 | } | 1051 | } |
| 1004 | 1052 | ||
| 1005 | /* This function must only be called under RTNL lock or under the | 1053 | /* This function must only be called under the representor's encap_lock or |
| 1006 | * representor's encap_lock in case RTNL mutex can't be held. | 1054 | * inside rcu read lock section. |
| 1007 | */ | 1055 | */ |
| 1008 | static struct mlx5e_neigh_hash_entry * | 1056 | static struct mlx5e_neigh_hash_entry * |
| 1009 | mlx5e_rep_neigh_entry_lookup(struct mlx5e_priv *priv, | 1057 | mlx5e_rep_neigh_entry_lookup(struct mlx5e_priv *priv, |
| @@ -1011,9 +1059,11 @@ mlx5e_rep_neigh_entry_lookup(struct mlx5e_priv *priv, · 1 unmodified line | |||
| 1011 | { | 1059 | { |
| 1012 | struct mlx5e_rep_priv *rpriv = priv->ppriv; | 1060 | struct mlx5e_rep_priv *rpriv = priv->ppriv; |
| 1013 | struct mlx5e_neigh_update_table *neigh_update = &rpriv->neigh_update; | 1061 | struct mlx5e_neigh_update_table *neigh_update = &rpriv->neigh_update; |
| 1062 | struct mlx5e_neigh_hash_entry *nhe; | ||
| 1014 | 1063 | ||
| 1015 | return rhashtable_lookup_fast(&neigh_update->neigh_ht, m_neigh, | 1064 | nhe = rhashtable_lookup_fast(&neigh_update->neigh_ht, m_neigh, |
| 1016 | mlx5e_neigh_ht_params); | 1065 | mlx5e_neigh_ht_params); |
| 1066 | return nhe && mlx5e_rep_neigh_entry_hold(nhe) ? nhe : NULL; | ||
| 1017 | } | 1067 | } |
| 1018 | 1068 | ||
| 1019 | static int mlx5e_rep_neigh_entry_create(struct mlx5e_priv *priv, | 1069 | static int mlx5e_rep_neigh_entry_create(struct mlx5e_priv *priv, |
| @@ -1026,8 +1076,10 @@ static int mlx5e_rep_neigh_entry_create(struct mlx5e_priv *priv, · 6 unmodified lines | |||
| 1026 | if (!*nhe) | 1076 | if (!*nhe) |
| 1027 | return -ENOMEM; | 1077 | return -ENOMEM; |
| 1028 | 1078 | ||
| 1079 | (*nhe)->priv = priv; | ||
| 1029 | memcpy(&(*nhe)->m_neigh, &e->m_neigh, sizeof(e->m_neigh)); | 1080 | memcpy(&(*nhe)->m_neigh, &e->m_neigh, sizeof(e->m_neigh)); |
| 1030 | INIT_WORK(&(*nhe)->neigh_update_work, mlx5e_rep_neigh_update); | 1081 | INIT_WORK(&(*nhe)->neigh_update_work, mlx5e_rep_neigh_update); |
| 1082 | spin_lock_init(&(*nhe)->encap_list_lock); | ||
| 1031 | INIT_LIST_HEAD(&(*nhe)->encap_list); | 1083 | INIT_LIST_HEAD(&(*nhe)->encap_list); |
| 1032 | refcount_set(&(*nhe)->refcnt, 1); | 1084 | refcount_set(&(*nhe)->refcnt, 1); |
| 1033 | 1085 | ||
| @@ -1041,19 +1093,6 @@ out_free: · 7 unmodified lines | |||
| 1041 | return err; | 1093 | return err; |
| 1042 | } | 1094 | } |
| 1043 | 1095 | ||
| 1044 | static void mlx5e_rep_neigh_entry_destroy(struct mlx5e_priv *priv, | ||
| 1045 | struct mlx5e_neigh_hash_entry *nhe) | ||
| 1046 | { | ||
| 1047 | /* The neigh hash entry must be removed from the hash table regardless | ||
| 1048 | * of the reference count value, so it won't be found by the next | ||
| 1049 | * neigh notification call. The neigh hash entry reference count is | ||
| 1050 | * incremented only during creation and neigh notification calls and | ||
| 1051 | * protects from freeing the nhe struct. | ||
| 1052 | */ | ||
| 1053 | mlx5e_rep_neigh_entry_remove(priv, nhe); | ||
| 1054 | mlx5e_rep_neigh_entry_release(nhe); | ||
| 1055 | } | ||
| 1056 | |||
| 1057 | int mlx5e_rep_encap_entry_attach(struct mlx5e_priv *priv, | 1096 | int mlx5e_rep_encap_entry_attach(struct mlx5e_priv *priv, |
| 1058 | struct mlx5e_encap_entry *e) | 1097 | struct mlx5e_encap_entry *e) |
| 1059 | { | 1098 | { |
| @@ -1066,16 +1105,26 @@ int mlx5e_rep_encap_entry_attach(struct mlx5e_priv *priv, · 6 unmodified lines | |||
| 1066 | err = mlx5_tun_entropy_refcount_inc(tun_entropy, e->reformat_type); | 1105 | err = mlx5_tun_entropy_refcount_inc(tun_entropy, e->reformat_type); |
| 1067 | if (err) | 1106 | if (err) |
| 1068 | return err; | 1107 | return err; |
| 1108 | |||
| 1109 | mutex_lock(&rpriv->neigh_update.encap_lock); | ||
| 1069 | nhe = mlx5e_rep_neigh_entry_lookup(priv, &e->m_neigh); | 1110 | nhe = mlx5e_rep_neigh_entry_lookup(priv, &e->m_neigh); |
| 1070 | if (!nhe) { | 1111 | if (!nhe) { |
| 1071 | err = mlx5e_rep_neigh_entry_create(priv, e, &nhe); | 1112 | err = mlx5e_rep_neigh_entry_create(priv, e, &nhe); |
| 1072 | if (err) { | 1113 | if (err) { |
| 1114 | mutex_unlock(&rpriv->neigh_update.encap_lock); | ||
| 1073 | mlx5_tun_entropy_refcount_dec(tun_entropy, | 1115 | mlx5_tun_entropy_refcount_dec(tun_entropy, |
| 1074 | e->reformat_type); | 1116 | e->reformat_type); |
| 1075 | return err; | 1117 | return err; |
| 1076 | } | 1118 | } |
| 1077 | } | 1119 | } |
| 1078 | list_add(&e->encap_list, &nhe->encap_list); | 1120 | |
| 1121 | e->nhe = nhe; | ||
| 1122 | spin_lock(&nhe->encap_list_lock); | ||
| 1123 | list_add_rcu(&e->encap_list, &nhe->encap_list); | ||
| 1124 | spin_unlock(&nhe->encap_list_lock); | ||
| 1125 | |||
| 1126 | mutex_unlock(&rpriv->neigh_update.encap_lock); | ||
| 1127 | |||
| 1079 | return 0; | 1128 | return 0; |
| 1080 | } | 1129 | } |
| 1081 | 1130 | ||
| @@ -1085,13 +1134,16 @@ void mlx5e_rep_encap_entry_detach(struct mlx5e_priv *priv, · 3 unmodified lines | |||
| 1085 | struct mlx5e_rep_priv *rpriv = priv->ppriv; | 1134 | struct mlx5e_rep_priv *rpriv = priv->ppriv; |
| 1086 | struct mlx5_rep_uplink_priv *uplink_priv = &rpriv->uplink_priv; | 1135 | struct mlx5_rep_uplink_priv *uplink_priv = &rpriv->uplink_priv; |
| 1087 | struct mlx5_tun_entropy *tun_entropy = &uplink_priv->tun_entropy; | 1136 | struct mlx5_tun_entropy *tun_entropy = &uplink_priv->tun_entropy; |
| 1088 | struct mlx5e_neigh_hash_entry *nhe; | ||
| 1089 | 1137 | ||
| 1090 | list_del(&e->encap_list); | 1138 | if (!e->nhe) |
| 1091 | nhe = mlx5e_rep_neigh_entry_lookup(priv, &e->m_neigh); | 1139 | return; |
| 1140 | |||
| 1141 | spin_lock(&e->nhe->encap_list_lock); | ||
| 1142 | list_del_rcu(&e->encap_list); | ||
| 1143 | spin_unlock(&e->nhe->encap_list_lock); | ||
| 1092 | 1144 | ||
| 1093 | if (list_empty(&nhe->encap_list)) | 1145 | mlx5e_rep_neigh_entry_release(e->nhe); |
| 1094 | mlx5e_rep_neigh_entry_destroy(priv, nhe); | 1146 | e->nhe = NULL; |
| 1095 | mlx5_tun_entropy_refcount_dec(tun_entropy, e->reformat_type); | 1147 | mlx5_tun_entropy_refcount_dec(tun_entropy, e->reformat_type); |
| 1096 | } | 1148 | } |
| 1097 | 1149 | ||
| @@ -35,6 +35,7 @@ · 34 unmodified lines | |||
| 35 | 35 | ||
| 36 | #include <net/ip_tunnels.h> | 36 | #include <net/ip_tunnels.h> |
| 37 | #include <linux/rhashtable.h> | 37 | #include <linux/rhashtable.h> |
| 38 | #include <linux/mutex.h> | ||
| 38 | #include "eswitch.h" | 39 | #include "eswitch.h" |
| 39 | #include "en.h" | 40 | #include "en.h" |
| 40 | #include "lib/port_tun.h" | 41 | #include "lib/port_tun.h" |
| @@ -48,7 +49,7 @@ struct mlx5e_neigh_update_table { · 7 unmodified lines | |||
| 48 | */ | 49 | */ |
| 49 | struct list_head neigh_list; | 50 | struct list_head neigh_list; |
| 50 | /* protect lookup/remove operations */ | 51 | /* protect lookup/remove operations */ |
| 51 | spinlock_t encap_lock; | 52 | struct mutex encap_lock; |
| 52 | struct notifier_block netevent_nb; | 53 | struct notifier_block netevent_nb; |
| 53 | struct delayed_work neigh_stats_work; | 54 | struct delayed_work neigh_stats_work; |
| 54 | unsigned long min_interval; /* jiffies */ | 55 | unsigned long min_interval; /* jiffies */ |
| @@ -110,6 +111,7 @@ struct mlx5e_neigh { · 55 unmodified lines | |||
| 110 | struct mlx5e_neigh_hash_entry { | 111 | struct mlx5e_neigh_hash_entry { |
| 111 | struct rhash_head rhash_node; | 112 | struct rhash_head rhash_node; |
| 112 | struct mlx5e_neigh m_neigh; | 113 | struct mlx5e_neigh m_neigh; |
| 114 | struct mlx5e_priv *priv; | ||
| 113 | 115 | ||
| 114 | /* Save the neigh hash entry in a list on the representor in | 116 | /* Save the neigh hash entry in a list on the representor in |
| 115 | * addition to the hash table. In order to iterate easily over the | 117 | * addition to the hash table. In order to iterate easily over the |
| @@ -117,6 +119,8 @@ struct mlx5e_neigh_hash_entry { · 1 unmodified line | |||
| 117 | */ | 119 | */ |
| 118 | struct list_head neigh_list; | 120 | struct list_head neigh_list; |
| 119 | 121 | ||
| 122 | /* protects encap list */ | ||
| 123 | spinlock_t encap_list_lock; | ||
| 120 | /* encap list sharing the same neigh */ | 124 | /* encap list sharing the same neigh */ |
| 121 | struct list_head encap_list; | 125 | struct list_head encap_list; |
| 122 | 126 | ||
| @@ -137,6 +141,8 @@ struct mlx5e_neigh_hash_entry { · 14 unmodified lines | |||
| 137 | * 'used' value and avoid neigh deleting by the kernel. | 141 | * 'used' value and avoid neigh deleting by the kernel. |
| 138 | */ | 142 | */ |
| 139 | unsigned long reported_lastuse; | 143 | unsigned long reported_lastuse; |
| 144 | |||
| 145 | struct rcu_head rcu; | ||
| 140 | }; | 146 | }; |
| 141 | 147 | ||
| 142 | enum { | 148 | enum { |
| @@ -145,6 +151,8 @@ enum { · 2 unmodified lines | |||
| 145 | }; | 151 | }; |
| 146 | 152 | ||
| 147 | struct mlx5e_encap_entry { | 153 | struct mlx5e_encap_entry { |
| 154 | /* attached neigh hash entry */ | ||
| 155 | struct mlx5e_neigh_hash_entry *nhe; | ||
| 148 | /* neigh hash entry list of encaps sharing the same neigh */ | 156 | /* neigh hash entry list of encaps sharing the same neigh */ |
| 149 | struct list_head encap_list; | 157 | struct list_head encap_list; |
| 150 | struct mlx5e_neigh m_neigh; | 158 | struct mlx5e_neigh m_neigh; |
| @@ -167,6 +175,7 @@ struct mlx5e_encap_entry { · 16 unmodified lines | |||
| 167 | refcount_t refcnt; | 175 | refcount_t refcnt; |
| 168 | struct completion res_ready; | 176 | struct completion res_ready; |
| 169 | int compl_result; | 177 | int compl_result; |
| 178 | struct rcu_head rcu; | ||
| 170 | }; | 179 | }; |
| 171 | 180 | ||
| 172 | struct mlx5e_rep_sq { | 181 | struct mlx5e_rep_sq { |
| @@ -56,6 +56,7 @@ · 55 unmodified lines | |||
| 56 | #include "en/tc_tun.h" | 56 | #include "en/tc_tun.h" |
| 57 | #include "lib/devcom.h" | 57 | #include "lib/devcom.h" |
| 58 | #include "lib/geneve.h" | 58 | #include "lib/geneve.h" |
| 59 | #include "diag/en_tc_tracepoint.h" | ||
| 59 | 60 | ||
| 60 | struct mlx5_nic_flow_attr { | 61 | struct mlx5_nic_flow_attr { |
| 61 | u32 action; | 62 | u32 action; |
| @@ -126,8 +127,11 @@ struct mlx5e_tc_flow { · 64 unmodified lines | |||
| 126 | struct list_head hairpin; /* flows sharing the same hairpin */ | 127 | struct list_head hairpin; /* flows sharing the same hairpin */ |
| 127 | struct list_head peer; /* flows with peer flow */ | 128 | struct list_head peer; /* flows with peer flow */ |
| 128 | struct list_head unready; /* flows not ready to be offloaded (e.g due to missing route) */ | 129 | struct list_head unready; /* flows not ready to be offloaded (e.g due to missing route) */ |
| 130 | int tmp_efi_index; | ||
| 131 | struct list_head tmp_list; /* temporary flow list used by neigh update */ | ||
| 129 | refcount_t refcnt; | 132 | refcount_t refcnt; |
| 130 | struct rcu_head rcu_head; | 133 | struct rcu_head rcu_head; |
| 134 | struct completion init_done; | ||
| 131 | union { | 135 | union { |
| 132 | struct mlx5_esw_flow_attr esw_attr[0]; | 136 | struct mlx5_esw_flow_attr esw_attr[0]; |
| 133 | struct mlx5_nic_flow_attr nic_attr[0]; | 137 | struct mlx5_nic_flow_attr nic_attr[0]; |
| @@ -1290,11 +1294,11 @@ static void mlx5e_tc_del_fdb_flow(struct mlx5e_priv *priv, · 1156 unmodified lines | |||
| 1290 | } | 1294 | } |
| 1291 | 1295 | ||
| 1292 | void mlx5e_tc_encap_flows_add(struct mlx5e_priv *priv, | 1296 | void mlx5e_tc_encap_flows_add(struct mlx5e_priv *priv, |
| 1293 | struct mlx5e_encap_entry *e) | 1297 | struct mlx5e_encap_entry *e, |
| 1298 | struct list_head *flow_list) | ||
| 1294 | { | 1299 | { |
| 1295 | struct mlx5_eswitch *esw = priv->mdev->priv.eswitch; | 1300 | struct mlx5_eswitch *esw = priv->mdev->priv.eswitch; |
| 1296 | struct mlx5_esw_flow_attr slow_attr, *esw_attr; | 1301 | struct mlx5_esw_flow_attr slow_attr, *esw_attr; |
| 1297 | struct encap_flow_item *efi, *tmp; | ||
| 1298 | struct mlx5_flow_handle *rule; | 1302 | struct mlx5_flow_handle *rule; |
| 1299 | struct mlx5_flow_spec *spec; | 1303 | struct mlx5_flow_spec *spec; |
| 1300 | struct mlx5e_tc_flow *flow; | 1304 | struct mlx5e_tc_flow *flow; |
| @@ -1313,19 +1317,17 @@ void mlx5e_tc_encap_flows_add(struct mlx5e_priv *priv, · 12 unmodified lines | |||
| 1313 | e->flags |= MLX5_ENCAP_ENTRY_VALID; | 1317 | e->flags |= MLX5_ENCAP_ENTRY_VALID; |
| 1314 | mlx5e_rep_queue_neigh_stats_work(priv); | 1318 | mlx5e_rep_queue_neigh_stats_work(priv); |
| 1315 | 1319 | ||
| 1316 | list_for_each_entry_safe(efi, tmp, &e->flows, list) { | 1320 | list_for_each_entry(flow, flow_list, tmp_list) { |
| 1317 | bool all_flow_encaps_valid = true; | 1321 | bool all_flow_encaps_valid = true; |
| 1318 | int i; | 1322 | int i; |
| 1319 | 1323 | ||
| 1320 | flow = container_of(efi, struct mlx5e_tc_flow, encaps[efi->index]); | 1324 | if (!mlx5e_is_offloaded_flow(flow)) |
| 1321 | if (IS_ERR(mlx5e_flow_get(flow))) | ||
| 1322 | continue; | 1325 | continue; |
| 1323 | |||
| 1324 | esw_attr = flow->esw_attr; | 1326 | esw_attr = flow->esw_attr; |
| 1325 | spec = &esw_attr->parse_attr->spec; | 1327 | spec = &esw_attr->parse_attr->spec; |
| 1326 | 1328 | ||
| 1327 | esw_attr->dests[efi->index].encap_id = e->encap_id; | 1329 | esw_attr->dests[flow->tmp_efi_index].encap_id = e->encap_id; |
| 1328 | esw_attr->dests[efi->index].flags |= MLX5_ESW_DEST_ENCAP_VALID; | 1330 | esw_attr->dests[flow->tmp_efi_index].flags |= MLX5_ESW_DEST_ENCAP_VALID; |
| 1329 | /* Flow can be associated with multiple encap entries. | 1331 | /* Flow can be associated with multiple encap entries. |
| 1330 | * Before offloading the flow verify that all of them have | 1332 | * Before offloading the flow verify that all of them have |
| 1331 | * a valid neighbour. | 1333 | * a valid neighbour. |
| @@ -1340,63 +1342,55 @@ void mlx5e_tc_encap_flows_add(struct mlx5e_priv *priv, · 8 unmodified lines | |||
| 1340 | } | 1342 | } |
| 1341 | /* Do not offload flows with unresolved neighbors */ | 1343 | /* Do not offload flows with unresolved neighbors */ |
| 1342 | if (!all_flow_encaps_valid) | 1344 | if (!all_flow_encaps_valid) |
| 1343 | goto loop_cont; | 1345 | continue; |
| 1344 | /* update from slow path rule to encap rule */ | 1346 | /* update from slow path rule to encap rule */ |
| 1345 | rule = mlx5e_tc_offload_fdb_rules(esw, flow, spec, esw_attr); | 1347 | rule = mlx5e_tc_offload_fdb_rules(esw, flow, spec, esw_attr); |
| 1346 | if (IS_ERR(rule)) { | 1348 | if (IS_ERR(rule)) { |
| 1347 | err = PTR_ERR(rule); | 1349 | err = PTR_ERR(rule); |
| 1348 | mlx5_core_warn(priv->mdev, "Failed to update cached encapsulation flow, %d\n", | 1350 | mlx5_core_warn(priv->mdev, "Failed to update cached encapsulation flow, %d\n", |
| 1349 | err); | 1351 | err); |
| 1350 | goto loop_cont; | 1352 | continue; |
| 1351 | } | 1353 | } |
| 1352 | 1354 | ||
| 1353 | mlx5e_tc_unoffload_from_slow_path(esw, flow, &slow_attr); | 1355 | mlx5e_tc_unoffload_from_slow_path(esw, flow, &slow_attr); |
| 1354 | flow->rule[0] = rule; | 1356 | flow->rule[0] = rule; |
| 1355 | /* was unset when slow path rule removed */ | 1357 | /* was unset when slow path rule removed */ |
| 1356 | flow_flag_set(flow, OFFLOADED); | 1358 | flow_flag_set(flow, OFFLOADED); |
| 1357 | |||
| 1358 | loop_cont: | ||
| 1359 | mlx5e_flow_put(priv, flow); | ||
| 1360 | } | 1359 | } |
| 1361 | } | 1360 | } |
| 1362 | 1361 | ||
| 1363 | void mlx5e_tc_encap_flows_del(struct mlx5e_priv *priv, | 1362 | void mlx5e_tc_encap_flows_del(struct mlx5e_priv *priv, |
| 1364 | struct mlx5e_encap_entry *e) | 1363 | struct mlx5e_encap_entry *e, |
| 1364 | struct list_head *flow_list) | ||
| 1365 | { | 1365 | { |
| 1366 | struct mlx5_eswitch *esw = priv->mdev->priv.eswitch; | 1366 | struct mlx5_eswitch *esw = priv->mdev->priv.eswitch; |
| 1367 | struct mlx5_esw_flow_attr slow_attr; | 1367 | struct mlx5_esw_flow_attr slow_attr; |
| 1368 | struct encap_flow_item *efi, *tmp; | ||
| 1369 | struct mlx5_flow_handle *rule; | 1368 | struct mlx5_flow_handle *rule; |
| 1370 | struct mlx5_flow_spec *spec; | 1369 | struct mlx5_flow_spec *spec; |
| 1371 | struct mlx5e_tc_flow *flow; | 1370 | struct mlx5e_tc_flow *flow; |
| 1372 | int err; | 1371 | int err; |
| 1373 | 1372 | ||
| 1374 | list_for_each_entry_safe(efi, tmp, &e->flows, list) { | 1373 | list_for_each_entry(flow, flow_list, tmp_list) { |
| 1375 | flow = container_of(efi, struct mlx5e_tc_flow, encaps[efi->index]); | 1374 | if (!mlx5e_is_offloaded_flow(flow)) |
| 1376 | if (IS_ERR(mlx5e_flow_get(flow))) | ||
| 1377 | continue; | 1375 | continue; |
| 1378 | |||
| 1379 | spec = &flow->esw_attr->parse_attr->spec; | 1376 | spec = &flow->esw_attr->parse_attr->spec; |
| 1380 | 1377 | ||
| 1381 | /* update from encap rule to slow path rule */ | 1378 | /* update from encap rule to slow path rule */ |
| 1382 | rule = mlx5e_tc_offload_to_slow_path(esw, flow, spec, &slow_attr); | 1379 | rule = mlx5e_tc_offload_to_slow_path(esw, flow, spec, &slow_attr); |
| 1383 | /* mark the flow's encap dest as non-valid */ | 1380 | /* mark the flow's encap dest as non-valid */ |
| 1384 | flow->esw_attr->dests[efi->index].flags &= ~MLX5_ESW_DEST_ENCAP_VALID; | 1381 | flow->esw_attr->dests[flow->tmp_efi_index].flags &= ~MLX5_ESW_DEST_ENCAP_VALID; |
| 1385 | 1382 | ||
| 1386 | if (IS_ERR(rule)) { | 1383 | if (IS_ERR(rule)) { |
| 1387 | err = PTR_ERR(rule); | 1384 | err = PTR_ERR(rule); |
| 1388 | mlx5_core_warn(priv->mdev, "Failed to update slow path (encap) flow, %d\n", | 1385 | mlx5_core_warn(priv->mdev, "Failed to update slow path (encap) flow, %d\n", |
| 1389 | err); | 1386 | err); |
| 1390 | goto loop_cont; | 1387 | continue; |
| 1391 | } | 1388 | } |
| 1392 | 1389 | ||
| 1393 | mlx5e_tc_unoffload_fdb_rules(esw, flow, flow->esw_attr); | 1390 | mlx5e_tc_unoffload_fdb_rules(esw, flow, flow->esw_attr); |
| 1394 | flow->rule[0] = rule; | 1391 | flow->rule[0] = rule; |
| 1395 | /* was unset when fast path rule removed */ | 1392 | /* was unset when fast path rule removed */ |
| 1396 | flow_flag_set(flow, OFFLOADED); | 1393 | flow_flag_set(flow, OFFLOADED); |
| 1397 | |||
| 1398 | loop_cont: | ||
| 1399 | mlx5e_flow_put(priv, flow); | ||
| 1400 | } | 1394 | } |
| 1401 | 1395 | ||
| 1402 | /* we know that the encap is valid */ | 1396 | /* we know that the encap is valid */ |
| @@ -1412,11 +1406,84 @@ static struct mlx5_fc *mlx5e_tc_get_counter(struct mlx5e_tc_flow *flow) · 9 unmodified lines | |||
| 1412 | return flow->nic_attr->counter; | 1406 | return flow->nic_attr->counter; |
| 1413 | } | 1407 | } |
| 1414 | 1408 | ||
| 1409 | /* Takes reference to all flows attached to encap and adds the flows to | ||
| 1410 | * flow_list using 'tmp_list' list_head in mlx5e_tc_flow. | ||
| 1411 | */ | ||
| 1412 | void mlx5e_take_all_encap_flows(struct mlx5e_encap_entry *e, struct list_head *flow_list) | ||
| 1413 | { | ||
| 1414 | struct encap_flow_item *efi; | ||
| 1415 | struct mlx5e_tc_flow *flow; | ||
| 1416 | |||
| 1417 | list_for_each_entry(efi, &e->flows, list) { | ||
| 1418 | flow = container_of(efi, struct mlx5e_tc_flow, encaps[efi->index]); | ||
| 1419 | if (IS_ERR(mlx5e_flow_get(flow))) | ||
| 1420 | continue; | ||
| 1421 | wait_for_completion(&flow->init_done); | ||
| 1422 | |||
| 1423 | flow->tmp_efi_index = efi->index; | ||
| 1424 | list_add(&flow->tmp_list, flow_list); | ||
| 1425 | } | ||
| 1426 | } | ||
| 1427 | |||
| 1428 | /* Iterate over tmp_list of flows attached to flow_list head. */ | ||
| 1429 | void mlx5e_put_encap_flow_list(struct mlx5e_priv *priv, struct list_head *flow_list) | ||
| 1430 | { | ||
| 1431 | struct mlx5e_tc_flow *flow, *tmp; | ||
| 1432 | |||
| 1433 | list_for_each_entry_safe(flow, tmp, flow_list, tmp_list) | ||
| 1434 | mlx5e_flow_put(priv, flow); | ||
| 1435 | } | ||
| 1436 | |||
| 1437 | static struct mlx5e_encap_entry * | ||
| 1438 | mlx5e_get_next_valid_encap(struct mlx5e_neigh_hash_entry *nhe, | ||
| 1439 | struct mlx5e_encap_entry *e) | ||
| 1440 | { | ||
| 1441 | struct mlx5e_encap_entry *next = NULL; | ||
| 1442 | |||
| 1443 | retry: | ||
| 1444 | rcu_read_lock(); | ||
| 1445 | |||
| 1446 | /* find encap with non-zero reference counter value */ | ||
| 1447 | for (next = e ? | ||
| 1448 | list_next_or_null_rcu(&nhe->encap_list, | ||
| 1449 | &e->encap_list, | ||
| 1450 | struct mlx5e_encap_entry, | ||
| 1451 | encap_list) : | ||
| 1452 | list_first_or_null_rcu(&nhe->encap_list, | ||
| 1453 | struct mlx5e_encap_entry, | ||
| 1454 | encap_list); | ||
| 1455 | next; | ||
| 1456 | next = list_next_or_null_rcu(&nhe->encap_list, | ||
| 1457 | &next->encap_list, | ||
| 1458 | struct mlx5e_encap_entry, | ||
| 1459 | encap_list)) | ||
| 1460 | if (mlx5e_encap_take(next)) | ||
| 1461 | break; | ||
| 1462 | |||
| 1463 | rcu_read_unlock(); | ||
| 1464 | |||
| 1465 | /* release starting encap */ | ||
| 1466 | if (e) | ||
| 1467 | mlx5e_encap_put(netdev_priv(e->out_dev), e); | ||
| 1468 | if (!next) | ||
| 1469 | return next; | ||
| 1470 | |||
| 1471 | /* wait for encap to be fully initialized */ | ||
| 1472 | wait_for_completion(&next->res_ready); | ||
| 1473 | /* continue searching if encap entry is not in valid state after completion */ | ||
| 1474 | if (!(next->flags & MLX5_ENCAP_ENTRY_VALID)) { | ||
| 1475 | e = next; | ||
| 1476 | goto retry; | ||
| 1477 | } | ||
| 1478 | |||
| 1479 | return next; | ||
| 1480 | } | ||
| 1481 | |||
| 1415 | void mlx5e_tc_update_neigh_used_value(struct mlx5e_neigh_hash_entry *nhe) | 1482 | void mlx5e_tc_update_neigh_used_value(struct mlx5e_neigh_hash_entry *nhe) |
| 1416 | { | 1483 | { |
| 1417 | struct mlx5e_neigh *m_neigh = &nhe->m_neigh; | 1484 | struct mlx5e_neigh *m_neigh = &nhe->m_neigh; |
| 1485 | struct mlx5e_encap_entry *e = NULL; | ||
| 1418 | struct mlx5e_tc_flow *flow; | 1486 | struct mlx5e_tc_flow *flow; |
| 1419 | struct mlx5e_encap_entry *e; | ||
| 1420 | struct mlx5_fc *counter; | 1487 | struct mlx5_fc *counter; |
| 1421 | struct neigh_table *tbl; | 1488 | struct neigh_table *tbl; |
| 1422 | bool neigh_used = false; | 1489 | bool neigh_used = false; |
| @@ -1432,37 +1499,45 @@ void mlx5e_tc_update_neigh_used_value(struct mlx5e_neigh_hash_entry *nhe) · 9 unmodified lines | |||
| 1432 | else | 1499 | else |
| 1433 | return; | 1500 | return; |
| 1434 | 1501 | ||
| 1435 | list_for_each_entry(e, &nhe->encap_list, encap_list) { | 1502 | /* mlx5e_get_next_valid_encap() releases previous encap before returning |
| 1503 | * next one. | ||
| 1504 | */ | ||
| 1505 | while ((e = mlx5e_get_next_valid_encap(nhe, e)) != NULL) { | ||
| 1506 | struct mlx5e_priv *priv = netdev_priv(e->out_dev); | ||
| 1436 | struct encap_flow_item *efi, *tmp; | 1507 | struct encap_flow_item *efi, *tmp; |
| 1508 | struct mlx5_eswitch *esw; | ||
| 1509 | LIST_HEAD(flow_list); | ||
| 1437 | 1510 | ||
| 1438 | if (!(e->flags & MLX5_ENCAP_ENTRY_VALID) || | 1511 | esw = priv->mdev->priv.eswitch; |
| 1439 | !mlx5e_encap_take(e)) | 1512 | mutex_lock(&esw->offloads.encap_tbl_lock); |
| 1440 | continue; | ||
| 1441 | |||
| 1442 | list_for_each_entry_safe(efi, tmp, &e->flows, list) { | 1513 | list_for_each_entry_safe(efi, tmp, &e->flows, list) { |
| 1443 | flow = container_of(efi, struct mlx5e_tc_flow, | 1514 | flow = container_of(efi, struct mlx5e_tc_flow, |
| 1444 | encaps[efi->index]); | 1515 | encaps[efi->index]); |
| 1445 | if (IS_ERR(mlx5e_flow_get(flow))) | 1516 | if (IS_ERR(mlx5e_flow_get(flow))) |
| 1446 | continue; | 1517 | continue; |
| 1518 | list_add(&flow->tmp_list, &flow_list); | ||
| 1447 | 1519 | ||
| 1448 | if (mlx5e_is_offloaded_flow(flow)) { | 1520 | if (mlx5e_is_offloaded_flow(flow)) { |
| 1449 | counter = mlx5e_tc_get_counter(flow); | 1521 | counter = mlx5e_tc_get_counter(flow); |
| 1450 | lastuse = mlx5_fc_query_lastuse(counter); | 1522 | lastuse = mlx5_fc_query_lastuse(counter); |
| 1451 | if (time_after((unsigned long)lastuse, nhe->reported_lastuse)) { | 1523 | if (time_after((unsigned long)lastuse, nhe->reported_lastuse)) { |
| 1452 | mlx5e_flow_put(netdev_priv(e->out_dev), flow); | ||
| 1453 | neigh_used = true; | 1524 | neigh_used = true; |
| 1454 | break; | 1525 | break; |
| 1455 | } | 1526 | } |
| 1456 | } | 1527 | } |
| 1457 | |||
| 1458 | mlx5e_flow_put(netdev_priv(e->out_dev), flow); | ||
| 1459 | } | 1528 | } |
| 1529 | mutex_unlock(&esw->offloads.encap_tbl_lock); | ||
| 1460 | 1530 | ||
| 1461 | mlx5e_encap_put(netdev_priv(e->out_dev), e); | 1531 | mlx5e_put_encap_flow_list(priv, &flow_list); |
| 1462 | if (neigh_used) | 1532 | if (neigh_used) { |
| 1533 | /* release current encap before breaking the loop */ | ||
| 1534 | mlx5e_encap_put(priv, e); | ||
| 1463 | break; | 1535 | break; |
| 1536 | } | ||
| 1464 | } | 1537 | } |
| 1465 | 1538 | ||
| 1539 | trace_mlx5e_tc_update_neigh_used_value(nhe, neigh_used); | ||
| 1540 | |||
| 1466 | if (neigh_used) { | 1541 | if (neigh_used) { |
| 1467 | nhe->reported_lastuse = jiffies; | 1542 | nhe->reported_lastuse = jiffies; |
| 1468 | 1543 | ||
| @@ -1490,7 +1565,7 @@ static void mlx5e_encap_dealloc(struct mlx5e_priv *priv, struct mlx5e_encap_entr · 21 unmodified lines | |||
| 1490 | } | 1565 | } |
| 1491 | 1566 | ||
| 1492 | kfree(e->encap_header); | 1567 | kfree(e->encap_header); |
| 1493 | kfree(e); | 1568 | kfree_rcu(e, rcu); |
| 1494 | } | 1569 | } |
| 1495 | 1570 | ||
| 1496 | void mlx5e_encap_put(struct mlx5e_priv *priv, struct mlx5e_encap_entry *e) | 1571 | void mlx5e_encap_put(struct mlx5e_priv *priv, struct mlx5e_encap_entry *e) |
| @@ -3426,6 +3501,7 @@ mlx5e_alloc_flow(struct mlx5e_priv *priv, int attr_size, · 1929 unmodified lines | |||
| 3426 | INIT_LIST_HEAD(&flow->mod_hdr); | 3501 | INIT_LIST_HEAD(&flow->mod_hdr); |
| 3427 | INIT_LIST_HEAD(&flow->hairpin); | 3502 | INIT_LIST_HEAD(&flow->hairpin); |
| 3428 | refcount_set(&flow->refcnt, 1); | 3503 | refcount_set(&flow->refcnt, 1); |
| 3504 | init_completion(&flow->init_done); | ||
| 3429 | 3505 | ||
| 3430 | *__flow = flow; | 3506 | *__flow = flow; |
| 3431 | *__parse_attr = parse_attr; | 3507 | *__parse_attr = parse_attr; |
| @@ -3498,6 +3574,7 @@ __mlx5e_add_fdb_flow(struct mlx5e_priv *priv, · 66 unmodified lines | |||
| 3498 | goto err_free; | 3574 | goto err_free; |
| 3499 | 3575 | ||
| 3500 | err = mlx5e_tc_add_fdb_flow(priv, flow, extack); | 3576 | err = mlx5e_tc_add_fdb_flow(priv, flow, extack); |
| 3577 | complete_all(&flow->init_done); | ||
| 3501 | if (err) { | 3578 | if (err) { |
| 3502 | if (!(err == -ENETUNREACH && mlx5_lag_is_multipath(in_mdev))) | 3579 | if (!(err == -ENETUNREACH && mlx5_lag_is_multipath(in_mdev))) |
| 3503 | goto err_free; | 3580 | goto err_free; |
| @@ -3695,6 +3772,7 @@ int mlx5e_configure_flower(struct net_device *dev, struct mlx5e_priv *priv, · 191 unmodified lines | |||
| 3695 | goto out; | 3772 | goto out; |
| 3696 | } | 3773 | } |
| 3697 | 3774 | ||
| 3775 | trace_mlx5e_configure_flower(f); | ||
| 3698 | err = mlx5e_tc_add_flow(priv, f, flags, dev, &flow); | 3776 | err = mlx5e_tc_add_flow(priv, f, flags, dev, &flow); |
| 3699 | if (err) | 3777 | if (err) |
| 3700 | goto out; | 3778 | goto out; |
| @@ -3744,6 +3822,7 @@ int mlx5e_delete_flower(struct net_device *dev, struct mlx5e_priv *priv, · 43 unmodified lines | |||
| 3744 | rhashtable_remove_fast(tc_ht, &flow->node, tc_ht_params); | 3822 | rhashtable_remove_fast(tc_ht, &flow->node, tc_ht_params); |
| 3745 | rcu_read_unlock(); | 3823 | rcu_read_unlock(); |
| 3746 | 3824 | ||
| 3825 | trace_mlx5e_delete_flower(f); | ||
| 3747 | mlx5e_flow_put(priv, flow); | 3826 | mlx5e_flow_put(priv, flow); |
| 3748 | 3827 | ||
| 3749 | return 0; | 3828 | return 0; |
| @@ -3813,6 +3892,7 @@ no_peer_counter: · 63 unmodified lines | |||
| 3813 | mlx5_devcom_release_peer_data(devcom, MLX5_DEVCOM_ESW_OFFLOADS); | 3892 | mlx5_devcom_release_peer_data(devcom, MLX5_DEVCOM_ESW_OFFLOADS); |
| 3814 | out: | 3893 | out: |
| 3815 | flow_stats_update(&f->stats, bytes, packets, lastuse); | 3894 | flow_stats_update(&f->stats, bytes, packets, lastuse); |
| 3895 | trace_mlx5e_stats_flower(f); | ||
| 3816 | errout: | 3896 | errout: |
| 3817 | mlx5e_flow_put(priv, flow); | 3897 | mlx5e_flow_put(priv, flow); |
| 3818 | return err; | 3898 | return err; |
| @@ -72,12 +72,17 @@ void mlx5e_tc_stats_matchall(struct mlx5e_priv *priv, · 71 unmodified lines | |||
| 72 | 72 | ||
| 73 | struct mlx5e_encap_entry; | 73 | struct mlx5e_encap_entry; |
| 74 | void mlx5e_tc_encap_flows_add(struct mlx5e_priv *priv, | 74 | void mlx5e_tc_encap_flows_add(struct mlx5e_priv *priv, |
| 75 | struct mlx5e_encap_entry *e); | 75 | struct mlx5e_encap_entry *e, |
| 76 | struct list_head *flow_list); | ||
| 76 | void mlx5e_tc_encap_flows_del(struct mlx5e_priv *priv, | 77 | void mlx5e_tc_encap_flows_del(struct mlx5e_priv *priv, |
| 77 | struct mlx5e_encap_entry *e); | 78 | struct mlx5e_encap_entry *e, |
| 79 | struct list_head *flow_list); | ||
| 78 | bool mlx5e_encap_take(struct mlx5e_encap_entry *e); | 80 | bool mlx5e_encap_take(struct mlx5e_encap_entry *e); |
| 79 | void mlx5e_encap_put(struct mlx5e_priv *priv, struct mlx5e_encap_entry *e); | 81 | void mlx5e_encap_put(struct mlx5e_priv *priv, struct mlx5e_encap_entry *e); |
| 80 | 82 | ||
| 83 | void mlx5e_take_all_encap_flows(struct mlx5e_encap_entry *e, struct list_head *flow_list); | ||
| 84 | void mlx5e_put_encap_flow_list(struct mlx5e_priv *priv, struct list_head *flow_list); | ||
| 85 | |||
| 81 | struct mlx5e_neigh_hash_entry; | 86 | struct mlx5e_neigh_hash_entry; |
| 82 | void mlx5e_tc_update_neigh_used_value(struct mlx5e_neigh_hash_entry *nhe); | 87 | void mlx5e_tc_update_neigh_used_value(struct mlx5e_neigh_hash_entry *nhe); |
| 83 | 88 | ||
| @@ -138,6 +138,7 @@ enum flow_action_id { · 137 unmodified lines | |||
| 138 | FLOW_ACTION_MPLS_PUSH, | 138 | FLOW_ACTION_MPLS_PUSH, |
| 139 | FLOW_ACTION_MPLS_POP, | 139 | FLOW_ACTION_MPLS_POP, |
| 140 | FLOW_ACTION_MPLS_MANGLE, | 140 | FLOW_ACTION_MPLS_MANGLE, |
| 141 | NUM_FLOW_ACTIONS, | ||
| 141 | }; | 142 | }; |
| 142 | 143 | ||
| 143 | /* This is mirroring enum pedit_header_type definition for easy mapping between | 144 | /* This is mirroring enum pedit_header_type definition for easy mapping between |