Merge tag 'mlx5-updates-2019-08-21' of git://git.kernel.org/pub/scm/linux/kernel/git/saeed/linux

Saeed Mahameed says: ==================== mlx5 tc flow handling for concurrent execution (Part 3) This series includes updates to mlx5 ethernet and core driver: Vlad submits part 3 of 3 part series to allow TC flow handling for concurrent execution. Vlad says: ========== Structure mlx5e_neigh_hash_entry code that uses it are refactored in following ways: - Extend neigh_hash_entry with rcu and modify its users to always take reference to the structure when using it (neigh_hash_entry has already had atomic reference counter which was only used when scheduling neigh update on workqueue from atomic context of neigh update netevent). - Always use mlx5e_neigh_update_table->encap_lock when modifying neigh update hash table and list. Originally, this lock was only used to synchronize with netevent handler function, which is called from bh context and cannot use rtnl lock for synchronization. Use rcu read lock instead of encap_lock to lookup nhe in atomic context of netevent even handler function. Convert encap_lock to mutex to allow creating new neigh hash entries while holding it, which is safe to do because the lock is no longer used in atomic context. - Rcu-ify mlx5e_neigh_hash_entry->encap_list by changing operations on encap list to their rcu counterparts and extending encap structure with rcu_head to free the encap instances after rcu grace period. This allows fast traversal of list of encaps attached to nhe under rcu read lock protection. - Take encap_table_lock when accessing encap entries in neigh update and neigh stats update code to protect from concurrent encap entry insertion or removal. This approach leads to potential race condition when neigh update and neigh stats update code can access encap and flow entries that are not fully initialized or are being destroyed, or neigh can change state without updating encaps that are created concurrently. Prevent these issues by following changes in flow and encap initialization: - Extend mlx5e_tc_flow with 'init_done' completion. Modify neigh update to wait for both encap and flow completions to prevent concurrent access to a structure that is being initialized by tc. - Skip structures that failed during initialization: encaps with encap_id<0 and flows that don't have OFFLOADED flag set. - To ensure that no new flows are added to encap when it is being accessed by neigh update or neigh stats update, take encap_table_lock mutex. - To prevent concurrent deletion by tc, ensure that neigh update and neigh stats update hold references to encap and flow instances while using them. With changes presented in this patch set it is now safe to execute tc concurrently with neigh update and neigh stats update. However, these two workqueue tasks modify same flow "tmp_list" field to store flows with reference taken in temporary list to release the references after update operation finishes and should not be executed concurrently with each other. Last 3 patches of this series provide 3 new mlx5 trace points to track mlx5 tc requests and mlx5 neigh updates. ====================

David S. Miller committed 7 years ago

Commit fed07ef2 parents 2b9b5e7 597088210 files · +545 −126Browse files

Unified · Split

10 files

Documentation/networking/device_drivers/mellanox/mlx5.rst +46 −0
drivers/net/ethernet/mellanox/mlx5/core/Makefile +1 −1
drivers/net/ethernet/mellanox/mlx5/core/diag/en_rep_tracepoint.h +54 −0
drivers/net/ethernet/mellanox/mlx5/core/diag/en_tc_tracepoint.c +58 −0
drivers/net/ethernet/mellanox/mlx5/core/diag/en_tc_tracepoint.h +114 −0
drivers/net/ethernet/mellanox/mlx5/core/en_rep.c +138 −86
drivers/net/ethernet/mellanox/mlx5/core/en_rep.h +10 −1
drivers/net/ethernet/mellanox/mlx5/core/en_tc.c +116 −36
drivers/net/ethernet/mellanox/mlx5/core/en_tc.h +7 −2
include/net/flow_offload.h +1 −0

Documentation/networking/device_drivers/mellanox/mlx5.rst+46−0

@@ -12,6 +12,7 @@ Contents · 11 unmodified lines
12	12	- `Enabling the driver and kconfig options`_
13	13	- `Devlink info`_
14	14	- `Devlink health reporters`_
	15	+- `mlx5 tracepoints`_
15	16
16	17	Enabling the driver and kconfig options
17	18	================================================
@@ -219,3 +220,48 @@ User commands examples: · 201 unmodified lines
219	220	$ devlink health dump show pci/0000:82:00.1 reporter fw_fatal
220	221
221	222	NOTE: This command can run only on PF.
	223	+
	224	+mlx5 tracepoints
	225	+================
	226	+
	227	+mlx5 driver provides internal trace points for tracking and debugging using
	228	+kernel tracepoints interfaces (refer to Documentation/trace/ftrase.rst).
	229	+
	230	+For the list of support mlx5 events check /sys/kernel/debug/tracing/events/mlx5/
	231	+
	232	+tc and eswitch offloads tracepoints:
	233	+
	234	+- mlx5e_configure_flower: trace flower filter actions and cookies offloaded to mlx5::
	235	+
	236	+ $ echo mlx5:mlx5e_configure_flower >> /sys/kernel/debug/tracing/set_event
	237	+ $ cat /sys/kernel/debug/tracing/trace
	238	+ ...
	239	+ tc-6535 [019] ...1 2672.404466: mlx5e_configure_flower: cookie=0000000067874a55 actions= REDIRECT
	240	+
	241	+- mlx5e_delete_flower: trace flower filter actions and cookies deleted from mlx5::
	242	+
	243	+ $ echo mlx5:mlx5e_delete_flower >> /sys/kernel/debug/tracing/set_event
	244	+ $ cat /sys/kernel/debug/tracing/trace
	245	+ ...
	246	+ tc-6569 [010] .N.1 2686.379075: mlx5e_delete_flower: cookie=0000000067874a55 actions= NULL
	247	+
	248	+- mlx5e_stats_flower: trace flower stats request::
	249	+
	250	+ $ echo mlx5:mlx5e_stats_flower >> /sys/kernel/debug/tracing/set_event
	251	+ $ cat /sys/kernel/debug/tracing/trace
	252	+ ...
	253	+ tc-6546 [010] ...1 2679.704889: mlx5e_stats_flower: cookie=0000000060eb3d6a bytes=0 packets=0 lastused=4295560217
	254	+
	255	+- mlx5e_tc_update_neigh_used_value: trace tunnel rule neigh update value offloaded to mlx5::
	256	+
	257	+ $ echo mlx5:mlx5e_tc_update_neigh_used_value >> /sys/kernel/debug/tracing/set_event
	258	+ $ cat /sys/kernel/debug/tracing/trace
	259	+ ...
	260	+ kworker/u48:4-8806 [009] ...1 55117.882428: mlx5e_tc_update_neigh_used_value: netdev: ens1f0 IPv4: 1.1.1.10 IPv6: ::ffff:1.1.1.10 neigh_used=1
	261	+
	262	+- mlx5e_rep_neigh_update: trace neigh update tasks scheduled due to neigh state change events::
	263	+
	264	+ $ echo mlx5:mlx5e_rep_neigh_update >> /sys/kernel/debug/tracing/set_event
	265	+ $ cat /sys/kernel/debug/tracing/trace
	266	+ ...
	267	+ kworker/u48:7-2221 [009] ...1 1475.387435: mlx5e_rep_neigh_update: netdev: ens1f0 MAC: 24:8a:07:9a:17:9a IPv4: 1.1.1.10 IPv6: ::ffff:1.1.1.10 neigh_connected=1

drivers/net/ethernet/mellanox/mlx5/core/Makefile+1−1

@@ -35,7 +35,7 @@ mlx5_core-$(CONFIG_MLX5_EN_RXNFC) += en_fs_ethtool.o · 34 unmodified lines
35	35	mlx5_core-$(CONFIG_MLX5_CORE_EN_DCB) += en_dcbnl.o en/port_buffer.o
36	36	mlx5_core-$(CONFIG_MLX5_ESWITCH) += en_rep.o en_tc.o en/tc_tun.o lib/port_tun.o lag_mp.o \
37	37	lib/geneve.o en/tc_tun_vxlan.o en/tc_tun_gre.o \
38		− en/tc_tun_geneve.o
	38	+ en/tc_tun_geneve.o diag/en_tc_tracepoint.o
39	39
40	40	#
41	41	# Core extra

drivers/net/ethernet/mellanox/mlx5/core/diag/en_rep_tracepoint.h+54−0

@@ -0,0 +1,54 @@
	1	+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
	2	+/* Copyright (c) 2019 Mellanox Technologies. */
	3	+
	4	+#undef TRACE_SYSTEM
	5	+#define TRACE_SYSTEM mlx5
	6	+
	7	+#if !defined(_MLX5_EN_REP_TP_) \|\| defined(TRACE_HEADER_MULTI_READ)
	8	+#define _MLX5_EN_REP_TP_
	9	+
	10	+#include <linux/tracepoint.h>
	11	+#include <linux/trace_seq.h>
	12	+#include "en_rep.h"
	13	+
	14	+TRACE_EVENT(mlx5e_rep_neigh_update,
	15	+ TP_PROTO(const struct mlx5e_neigh_hash_entry nhe, const u8 ha,
	16	+ bool neigh_connected),
	17	+ TP_ARGS(nhe, ha, neigh_connected),
	18	+ TP_STRUCT__entry(__string(devname, nhe->m_neigh.dev->name)
	19	+ __array(u8, ha, ETH_ALEN)
	20	+ __array(u8, v4, 4)
	21	+ __array(u8, v6, 16)
	22	+ __field(bool, neigh_connected)
	23	+ ),
	24	+ TP_fast_assign(const struct mlx5e_neigh *mn = &nhe->m_neigh;
	25	+ struct in6_addr *pin6;
	26	+ __be32 *p32;
	27	+
	28	+ __assign_str(devname, mn->dev->name);
	29	+ __entry->neigh_connected = neigh_connected;
	30	+ memcpy(__entry->ha, ha, ETH_ALEN);
	31	+
	32	+ p32 = (__be32 *)__entry->v4;
	33	+ pin6 = (struct in6_addr *)__entry->v6;
	34	+ if (mn->family == AF_INET) {
	35	+ *p32 = mn->dst_ip.v4;
	36	+ ipv6_addr_set_v4mapped(*p32, pin6);
	37	+ } else if (mn->family == AF_INET6) {
	38	+ *pin6 = mn->dst_ip.v6;
	39	+ }
	40	+ ),
	41	+ TP_printk("netdev: %s MAC: %pM IPv4: %pI4 IPv6: %pI6c neigh_connected=%d\n",
	42	+ __get_str(devname), __entry->ha,
	43	+ __entry->v4, __entry->v6, __entry->neigh_connected
	44	+ )
	45	+);
	46	+
	47	+#endif /* _MLX5_EN_REP_TP_ */
	48	+
	49	+/* This part must be outside protection */
	50	+#undef TRACE_INCLUDE_PATH
	51	+#define TRACE_INCLUDE_PATH ./diag
	52	+#undef TRACE_INCLUDE_FILE
	53	+#define TRACE_INCLUDE_FILE en_rep_tracepoint
	54	+#include <trace/define_trace.h>

drivers/net/ethernet/mellanox/mlx5/core/diag/en_tc_tracepoint.c+58−0

@@ -0,0 +1,58 @@
	1	+// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
	2	+/* Copyright (c) 2019 Mellanox Technologies. */
	3	+
	4	+#define CREATE_TRACE_POINTS
	5	+#include "en_tc_tracepoint.h"
	6	+
	7	+void put_ids_to_array(int *ids,
	8	+ const struct flow_action_entry *entries,
	9	+ unsigned int num)
	10	+{
	11	+ unsigned int i;
	12	+
	13	+ for (i = 0; i < num; i++)
	14	+ ids[i] = entries[i].id;
	15	+}
	16	+
	17	+#define NAME_SIZE 16
	18	+
	19	+static const char FLOWACT2STR[NUM_FLOW_ACTIONS][NAME_SIZE] = {
	20	+ [FLOW_ACTION_ACCEPT] = "ACCEPT",
	21	+ [FLOW_ACTION_DROP] = "DROP",
	22	+ [FLOW_ACTION_TRAP] = "TRAP",
	23	+ [FLOW_ACTION_GOTO] = "GOTO",
	24	+ [FLOW_ACTION_REDIRECT] = "REDIRECT",
	25	+ [FLOW_ACTION_MIRRED] = "MIRRED",
	26	+ [FLOW_ACTION_VLAN_PUSH] = "VLAN_PUSH",
	27	+ [FLOW_ACTION_VLAN_POP] = "VLAN_POP",
	28	+ [FLOW_ACTION_VLAN_MANGLE] = "VLAN_MANGLE",
	29	+ [FLOW_ACTION_TUNNEL_ENCAP] = "TUNNEL_ENCAP",
	30	+ [FLOW_ACTION_TUNNEL_DECAP] = "TUNNEL_DECAP",
	31	+ [FLOW_ACTION_MANGLE] = "MANGLE",
	32	+ [FLOW_ACTION_ADD] = "ADD",
	33	+ [FLOW_ACTION_CSUM] = "CSUM",
	34	+ [FLOW_ACTION_MARK] = "MARK",
	35	+ [FLOW_ACTION_WAKE] = "WAKE",
	36	+ [FLOW_ACTION_QUEUE] = "QUEUE",
	37	+ [FLOW_ACTION_SAMPLE] = "SAMPLE",
	38	+ [FLOW_ACTION_POLICE] = "POLICE",
	39	+ [FLOW_ACTION_CT] = "CT",
	40	+};
	41	+
	42	+const char parse_action(struct trace_seq p,
	43	+ int *ids,
	44	+ unsigned int num)
	45	+{
	46	+ const char *ret = trace_seq_buffer_ptr(p);
	47	+ unsigned int i;
	48	+
	49	+ for (i = 0; i < num; i++) {
	50	+ if (ids[i] < NUM_FLOW_ACTIONS)
	51	+ trace_seq_printf(p, "%s ", FLOWACT2STR[ids[i]]);
	52	+ else
	53	+ trace_seq_printf(p, "UNKNOWN ");
	54	+ }
	55	+
	56	+ trace_seq_putc(p, 0);
	57	+ return ret;
	58	+}

drivers/net/ethernet/mellanox/mlx5/core/diag/en_tc_tracepoint.h+114−0

@@ -0,0 +1,114 @@
	1	+/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
	2	+/* Copyright (c) 2019 Mellanox Technologies. */
	3	+
	4	+#undef TRACE_SYSTEM
	5	+#define TRACE_SYSTEM mlx5
	6	+
	7	+#if !defined(_MLX5_TC_TP_) \|\| defined(TRACE_HEADER_MULTI_READ)
	8	+#define _MLX5_TC_TP_
	9	+
	10	+#include <linux/tracepoint.h>
	11	+#include <linux/trace_seq.h>
	12	+#include <net/flow_offload.h>
	13	+#include "en_rep.h"
	14	+
	15	+#define __parse_action(ids, num) parse_action(p, ids, num)
	16	+
	17	+void put_ids_to_array(int *ids,
	18	+ const struct flow_action_entry *entries,
	19	+ unsigned int num);
	20	+
	21	+const char parse_action(struct trace_seq p,
	22	+ int *ids,
	23	+ unsigned int num);
	24	+
	25	+DECLARE_EVENT_CLASS(mlx5e_flower_template,
	26	+ TP_PROTO(const struct flow_cls_offload *f),
	27	+ TP_ARGS(f),
	28	+ TP_STRUCT__entry(__field(void *, cookie)
	29	+ __field(unsigned int, num)
	30	+ __dynamic_array(int, ids, f->rule ?
	31	+ f->rule->action.num_entries : 0)
	32	+ ),
	33	+ TP_fast_assign(__entry->cookie = (void *)f->cookie;
	34	+ __entry->num = (f->rule ?
	35	+ f->rule->action.num_entries : 0);
	36	+ if (__entry->num)
	37	+ put_ids_to_array(__get_dynamic_array(ids),
	38	+ f->rule->action.entries,
	39	+ f->rule->action.num_entries);
	40	+ ),
	41	+ TP_printk("cookie=%p actions= %s\n",
	42	+ __entry->cookie, __entry->num ?
	43	+ __parse_action(__get_dynamic_array(ids),
	44	+ __entry->num) : "NULL"
	45	+ )
	46	+);
	47	+
	48	+DEFINE_EVENT(mlx5e_flower_template, mlx5e_configure_flower,
	49	+ TP_PROTO(const struct flow_cls_offload *f),
	50	+ TP_ARGS(f)
	51	+ );
	52	+
	53	+DEFINE_EVENT(mlx5e_flower_template, mlx5e_delete_flower,
	54	+ TP_PROTO(const struct flow_cls_offload *f),
	55	+ TP_ARGS(f)
	56	+ );
	57	+
	58	+TRACE_EVENT(mlx5e_stats_flower,
	59	+ TP_PROTO(const struct flow_cls_offload *f),
	60	+ TP_ARGS(f),
	61	+ TP_STRUCT__entry(__field(void *, cookie)
	62	+ __field(u64, bytes)
	63	+ __field(u64, packets)
	64	+ __field(u64, lastused)
	65	+ ),
	66	+ TP_fast_assign(__entry->cookie = (void *)f->cookie;
	67	+ __entry->bytes = f->stats.bytes;
	68	+ __entry->packets = f->stats.pkts;
	69	+ __entry->lastused = f->stats.lastused;
	70	+ ),
	71	+ TP_printk("cookie=%p bytes=%llu packets=%llu lastused=%llu\n",
	72	+ __entry->cookie, __entry->bytes,
	73	+ __entry->packets, __entry->lastused
	74	+ )
	75	+);
	76	+
	77	+TRACE_EVENT(mlx5e_tc_update_neigh_used_value,
	78	+ TP_PROTO(const struct mlx5e_neigh_hash_entry *nhe, bool neigh_used),
	79	+ TP_ARGS(nhe, neigh_used),
	80	+ TP_STRUCT__entry(__string(devname, nhe->m_neigh.dev->name)
	81	+ __array(u8, v4, 4)
	82	+ __array(u8, v6, 16)
	83	+ __field(bool, neigh_used)
	84	+ ),
	85	+ TP_fast_assign(const struct mlx5e_neigh *mn = &nhe->m_neigh;
	86	+ struct in6_addr *pin6;
	87	+ __be32 *p32;
	88	+
	89	+ __assign_str(devname, mn->dev->name);
	90	+ __entry->neigh_used = neigh_used;
	91	+
	92	+ p32 = (__be32 *)__entry->v4;
	93	+ pin6 = (struct in6_addr *)__entry->v6;
	94	+ if (mn->family == AF_INET) {
	95	+ *p32 = mn->dst_ip.v4;
	96	+ ipv6_addr_set_v4mapped(*p32, pin6);
	97	+ } else if (mn->family == AF_INET6) {
	98	+ *pin6 = mn->dst_ip.v6;
	99	+ }
	100	+ ),
	101	+ TP_printk("netdev: %s IPv4: %pI4 IPv6: %pI6c neigh_used=%d\n",
	102	+ __get_str(devname), __entry->v4, __entry->v6,
	103	+ __entry->neigh_used
	104	+ )
	105	+);
	106	+
	107	+#endif /* _MLX5_TC_TP_ */
	108	+
	109	+/* This part must be outside protection */
	110	+#undef TRACE_INCLUDE_PATH
	111	+#define TRACE_INCLUDE_PATH ./diag
	112	+#undef TRACE_INCLUDE_FILE
	113	+#define TRACE_INCLUDE_FILE en_tc_tracepoint
	114	+#include <trace/define_trace.h>

drivers/net/ethernet/mellanox/mlx5/core/en_rep.c+138−86

@@ -46,6 +46,8 @@ · 45 unmodified lines
46	46	#include "en/tc_tun.h"
47	47	#include "fs_core.h"
48	48	#include "lib/port_tun.h"
	49	+#define CREATE_TRACE_POINTS
	50	+#include "diag/en_rep_tracepoint.h"
49	51
50	52	#define MLX5E_REP_PARAMS_DEF_LOG_SQ_SIZE \
51	53	max(0x7, MLX5E_PARAMS_MINIMUM_LOG_SQ_SIZE)
@@ -524,47 +526,97 @@ void mlx5e_rep_queue_neigh_stats_work(struct mlx5e_priv *priv) · 472 unmodified lines
524	526	neigh_update->min_interval);
525	527	}
526	528
	529	+static bool mlx5e_rep_neigh_entry_hold(struct mlx5e_neigh_hash_entry *nhe)
	530	+{
	531	+ return refcount_inc_not_zero(&nhe->refcnt);
	532	+}
	533	+
	534	+static void mlx5e_rep_neigh_entry_remove(struct mlx5e_neigh_hash_entry *nhe);
	535	+
	536	+static void mlx5e_rep_neigh_entry_release(struct mlx5e_neigh_hash_entry *nhe)
	537	+{
	538	+ if (refcount_dec_and_test(&nhe->refcnt)) {
	539	+ mlx5e_rep_neigh_entry_remove(nhe);
	540	+ kfree_rcu(nhe, rcu);
	541	+ }
	542	+}
	543	+
	544	+static struct mlx5e_neigh_hash_entry *
	545	+mlx5e_get_next_nhe(struct mlx5e_rep_priv *rpriv,
	546	+ struct mlx5e_neigh_hash_entry *nhe)
	547	+{
	548	+ struct mlx5e_neigh_hash_entry *next = NULL;
	549	+
	550	+ rcu_read_lock();
	551	+
	552	+ for (next = nhe ?
	553	+ list_next_or_null_rcu(&rpriv->neigh_update.neigh_list,
	554	+ &nhe->neigh_list,
	555	+ struct mlx5e_neigh_hash_entry,
	556	+ neigh_list) :
	557	+ list_first_or_null_rcu(&rpriv->neigh_update.neigh_list,
	558	+ struct mlx5e_neigh_hash_entry,
	559	+ neigh_list);
	560	+ next;
	561	+ next = list_next_or_null_rcu(&rpriv->neigh_update.neigh_list,
	562	+ &next->neigh_list,
	563	+ struct mlx5e_neigh_hash_entry,
	564	+ neigh_list))
	565	+ if (mlx5e_rep_neigh_entry_hold(next))
	566	+ break;
	567	+
	568	+ rcu_read_unlock();
	569	+
	570	+ if (nhe)
	571	+ mlx5e_rep_neigh_entry_release(nhe);
	572	+
	573	+ return next;
	574	+}
	575	+
527	576	static void mlx5e_rep_neigh_stats_work(struct work_struct *work)
528	577	{
529	578	struct mlx5e_rep_priv *rpriv = container_of(work, struct mlx5e_rep_priv,
530	579	neigh_update.neigh_stats_work.work);
531	580	struct net_device *netdev = rpriv->netdev;
532	581	struct mlx5e_priv *priv = netdev_priv(netdev);
533		− struct mlx5e_neigh_hash_entry *nhe;
	582	+ struct mlx5e_neigh_hash_entry *nhe = NULL;
534	583
535	584	rtnl_lock();
536	585	if (!list_empty(&rpriv->neigh_update.neigh_list))
537	586	mlx5e_rep_queue_neigh_stats_work(priv);
538	587
539		− list_for_each_entry(nhe, &rpriv->neigh_update.neigh_list, neigh_list)
	588	+ while ((nhe = mlx5e_get_next_nhe(rpriv, nhe)) != NULL)
540	589	mlx5e_tc_update_neigh_used_value(nhe);
541	590
542	591	rtnl_unlock();
543	592	}
544	593
545		−static void mlx5e_rep_neigh_entry_hold(struct mlx5e_neigh_hash_entry *nhe)
546		−{
547		− refcount_inc(&nhe->refcnt);
548		−}
549		−
550		−static void mlx5e_rep_neigh_entry_release(struct mlx5e_neigh_hash_entry *nhe)
551		−{
552		− if (refcount_dec_and_test(&nhe->refcnt))
553		− kfree(nhe);
554		−}
555		−
556	594	static void mlx5e_rep_update_flows(struct mlx5e_priv *priv,
557	595	struct mlx5e_encap_entry *e,
558	596	bool neigh_connected,
559	597	unsigned char ha[ETH_ALEN])
560	598	{
561	599	struct ethhdr eth = (struct ethhdr )e->encap_header;
	600	+ struct mlx5_eswitch *esw = priv->mdev->priv.eswitch;
	601	+ bool encap_connected;
	602	+ LIST_HEAD(flow_list);
562	603
563	604	ASSERT_RTNL();
564	605
	606	+ /* wait for encap to be fully initialized */
	607	+ wait_for_completion(&e->res_ready);
	608	+
	609	+ mutex_lock(&esw->offloads.encap_tbl_lock);
	610	+ encap_connected = !!(e->flags & MLX5_ENCAP_ENTRY_VALID);
	611	+ if (e->compl_result \|\| (encap_connected == neigh_connected &&
	612	+ ether_addr_equal(e->h_dest, ha)))
	613	+ goto unlock;
	614	+
	615	+ mlx5e_take_all_encap_flows(e, &flow_list);
	616	+
565	617	if ((e->flags & MLX5_ENCAP_ENTRY_VALID) &&
566	618	(!neigh_connected \|\| !ether_addr_equal(e->h_dest, ha)))
567		− mlx5e_tc_encap_flows_del(priv, e);
	619	+ mlx5e_tc_encap_flows_del(priv, e, &flow_list);
568	620
569	621	if (neigh_connected && !(e->flags & MLX5_ENCAP_ENTRY_VALID)) {
570	622	ether_addr_copy(e->h_dest, ha);
@@ -574,8 +626,11 @@ static void mlx5e_rep_update_flows(struct mlx5e_priv *priv, · 3 unmodified lines
574	626	*/
575	627	ether_addr_copy(eth->h_source, e->route_dev->dev_addr);
576	628
577		− mlx5e_tc_encap_flows_add(priv, e);
	629	+ mlx5e_tc_encap_flows_add(priv, e, &flow_list);
578	630	}
	631	+unlock:
	632	+ mutex_unlock(&esw->offloads.encap_tbl_lock);
	633	+ mlx5e_put_encap_flow_list(priv, &flow_list);
579	634	}
580	635
581	636	static void mlx5e_rep_neigh_update(struct work_struct *work)
@@ -587,7 +642,6 @@ static void mlx5e_rep_neigh_update(struct work_struct *work) · 5 unmodified lines
587	642	unsigned char ha[ETH_ALEN];
588	643	struct mlx5e_priv *priv;
589	644	bool neigh_connected;
590		− bool encap_connected;
591	645	u8 nud_state, dead;
592	646
593	647	rtnl_lock();
@@ -605,17 +659,14 @@ static void mlx5e_rep_neigh_update(struct work_struct *work) · 11 unmodified lines
605	659
606	660	neigh_connected = (nud_state & NUD_VALID) && !dead;
607	661
	662	+ trace_mlx5e_rep_neigh_update(nhe, ha, neigh_connected);
	663	+
608	664	list_for_each_entry(e, &nhe->encap_list, encap_list) {
609	665	if (!mlx5e_encap_take(e))
610	666	continue;
611	667
612		− encap_connected = !!(e->flags & MLX5_ENCAP_ENTRY_VALID);
613	668	priv = netdev_priv(e->out_dev);
614		−
615		− if (encap_connected != neigh_connected \|\|
616		− !ether_addr_equal(e->h_dest, ha))
617		− mlx5e_rep_update_flows(priv, e, neigh_connected, ha);
618		−
	669	+ mlx5e_rep_update_flows(priv, e, neigh_connected, ha);
619	670	mlx5e_encap_put(priv, e);
620	671	}
621	672	mlx5e_rep_neigh_entry_release(nhe);
@@ -821,6 +872,28 @@ static int mlx5e_nic_rep_netdevice_event(struct notifier_block *nb, · 199 unmodified lines
821	872	return NOTIFY_OK;
822	873	}
823	874
	875	+static void
	876	+mlx5e_rep_queue_neigh_update_work(struct mlx5e_priv *priv,
	877	+ struct mlx5e_neigh_hash_entry *nhe,
	878	+ struct neighbour *n)
	879	+{
	880	+ /* Take a reference to ensure the neighbour and mlx5 encap
	881	+ * entry won't be destructed until we drop the reference in
	882	+ * delayed work.
	883	+ */
	884	+ neigh_hold(n);
	885	+
	886	+ /* This assignment is valid as long as the the neigh reference
	887	+ * is taken
	888	+ */
	889	+ nhe->n = n;
	890	+
	891	+ if (!queue_work(priv->wq, &nhe->neigh_update_work)) {
	892	+ mlx5e_rep_neigh_entry_release(nhe);
	893	+ neigh_release(n);
	894	+ }
	895	+}
	896	+
824	897	static struct mlx5e_neigh_hash_entry *
825	898	mlx5e_rep_neigh_entry_lookup(struct mlx5e_priv *priv,
826	899	struct mlx5e_neigh *m_neigh);
@@ -853,34 +926,13 @@ static int mlx5e_rep_netevent_event(struct notifier_block *nb, · 26 unmodified lines
853	926	m_neigh.family = n->ops->family;
854	927	memcpy(&m_neigh.dst_ip, n->primary_key, n->tbl->key_len);
855	928
856		− /* We are in atomic context and can't take RTNL mutex, so use
857		− * spin_lock_bh to lookup the neigh table. bh is used since
858		− * netevent can be called from a softirq context.
859		− */
860		− spin_lock_bh(&neigh_update->encap_lock);
	929	+ rcu_read_lock();
861	930	nhe = mlx5e_rep_neigh_entry_lookup(priv, &m_neigh);
862		− if (!nhe) {
863		− spin_unlock_bh(&neigh_update->encap_lock);
	931	+ rcu_read_unlock();
	932	+ if (!nhe)
864	933	return NOTIFY_DONE;
865		− }
866		−
867		− /* This assignment is valid as long as the the neigh reference
868		− * is taken
869		− */
870		− nhe->n = n;
871		−
872		− /* Take a reference to ensure the neighbour and mlx5 encap
873		− * entry won't be destructed until we drop the reference in
874		− * delayed work.
875		− */
876		− neigh_hold(n);
877		− mlx5e_rep_neigh_entry_hold(nhe);
878	934
879		− if (!queue_work(priv->wq, &nhe->neigh_update_work)) {
880		− mlx5e_rep_neigh_entry_release(nhe);
881		− neigh_release(n);
882		− }
883		− spin_unlock_bh(&neigh_update->encap_lock);
	935	+ mlx5e_rep_queue_neigh_update_work(priv, nhe, n);
884	936	break;
885	937
886	938	case NETEVENT_DELAY_PROBE_TIME_UPDATE:
@@ -897,19 +949,15 @@ static int mlx5e_rep_netevent_event(struct notifier_block *nb, · 10 unmodified lines
897	949	#endif
898	950	return NOTIFY_DONE;
899	951
900		− /* We are in atomic context and can't take RTNL mutex,
901		− * so use spin_lock_bh to walk the neigh list and look for
902		− * the relevant device. bh is used since netevent can be
903		− * called from a softirq context.
904		− */
905		− spin_lock_bh(&neigh_update->encap_lock);
906		− list_for_each_entry(nhe, &neigh_update->neigh_list, neigh_list) {
	952	+ rcu_read_lock();
	953	+ list_for_each_entry_rcu(nhe, &neigh_update->neigh_list,
	954	+ neigh_list) {
907	955	if (p->dev == nhe->m_neigh.dev) {
908	956	found = true;
909	957	break;
910	958	}
911	959	}
912		− spin_unlock_bh(&neigh_update->encap_lock);
	960	+ rcu_read_unlock();
913	961	if (!found)
914	962	return NOTIFY_DONE;
915	963
@@ -940,7 +988,7 @@ static int mlx5e_rep_neigh_init(struct mlx5e_rep_priv *rpriv) · 24 unmodified lines
940	988	return err;
941	989
942	990	INIT_LIST_HEAD(&neigh_update->neigh_list);
943		− spin_lock_init(&neigh_update->encap_lock);
	991	+ mutex_init(&neigh_update->encap_lock);
944	992	INIT_DELAYED_WORK(&neigh_update->neigh_stats_work,
945	993	mlx5e_rep_neigh_stats_work);
946	994	mlx5e_rep_neigh_update_init_interval(rpriv);
@@ -967,6 +1015,7 @@ static void mlx5e_rep_neigh_cleanup(struct mlx5e_rep_priv *rpriv) · 20 unmodified lines
967	1015
968	1016	cancel_delayed_work_sync(&rpriv->neigh_update.neigh_stats_work);
969	1017
	1018	+ mutex_destroy(&neigh_update->encap_lock);
970	1019	rhashtable_destroy(&neigh_update->neigh_ht);
971	1020	}
972	1021
@@ -982,28 +1031,27 @@ static int mlx5e_rep_neigh_entry_insert(struct mlx5e_priv *priv, · 9 unmodified lines
982	1031	if (err)
983	1032	return err;
984	1033
985		− list_add(&nhe->neigh_list, &rpriv->neigh_update.neigh_list);
	1034	+ list_add_rcu(&nhe->neigh_list, &rpriv->neigh_update.neigh_list);
986	1035
987	1036	return err;
988	1037	}
989	1038
990		−static void mlx5e_rep_neigh_entry_remove(struct mlx5e_priv *priv,
991		− struct mlx5e_neigh_hash_entry *nhe)
	1039	+static void mlx5e_rep_neigh_entry_remove(struct mlx5e_neigh_hash_entry *nhe)
992	1040	{
993		− struct mlx5e_rep_priv *rpriv = priv->ppriv;
	1041	+ struct mlx5e_rep_priv *rpriv = nhe->priv->ppriv;
994	1042
995		− spin_lock_bh(&rpriv->neigh_update.encap_lock);
	1043	+ mutex_lock(&rpriv->neigh_update.encap_lock);
996	1044
997		− list_del(&nhe->neigh_list);
	1045	+ list_del_rcu(&nhe->neigh_list);
998	1046
999	1047	rhashtable_remove_fast(&rpriv->neigh_update.neigh_ht,
1000	1048	&nhe->rhash_node,
1001	1049	mlx5e_neigh_ht_params);
1002		− spin_unlock_bh(&rpriv->neigh_update.encap_lock);
	1050	+ mutex_unlock(&rpriv->neigh_update.encap_lock);
1003	1051	}
1004	1052
1005		−/* This function must only be called under RTNL lock or under the
1006		− * representor's encap_lock in case RTNL mutex can't be held.
	1053	+/* This function must only be called under the representor's encap_lock or
	1054	+ * inside rcu read lock section.
1007	1055	*/
1008	1056	static struct mlx5e_neigh_hash_entry *
1009	1057	mlx5e_rep_neigh_entry_lookup(struct mlx5e_priv *priv,
@@ -1011,9 +1059,11 @@ mlx5e_rep_neigh_entry_lookup(struct mlx5e_priv *priv, · 1 unmodified line
1011	1059	{
1012	1060	struct mlx5e_rep_priv *rpriv = priv->ppriv;
1013	1061	struct mlx5e_neigh_update_table *neigh_update = &rpriv->neigh_update;
	1062	+ struct mlx5e_neigh_hash_entry *nhe;
1014	1063
1015		− return rhashtable_lookup_fast(&neigh_update->neigh_ht, m_neigh,
1016		− mlx5e_neigh_ht_params);
	1064	+ nhe = rhashtable_lookup_fast(&neigh_update->neigh_ht, m_neigh,
	1065	+ mlx5e_neigh_ht_params);
	1066	+ return nhe && mlx5e_rep_neigh_entry_hold(nhe) ? nhe : NULL;
1017	1067	}
1018	1068
1019	1069	static int mlx5e_rep_neigh_entry_create(struct mlx5e_priv *priv,
@@ -1026,8 +1076,10 @@ static int mlx5e_rep_neigh_entry_create(struct mlx5e_priv *priv, · 6 unmodified lines
1026	1076	if (!*nhe)
1027	1077	return -ENOMEM;
1028	1078
	1079	+ (*nhe)->priv = priv;
1029	1080	memcpy(&(*nhe)->m_neigh, &e->m_neigh, sizeof(e->m_neigh));
1030	1081	INIT_WORK(&(*nhe)->neigh_update_work, mlx5e_rep_neigh_update);
	1082	+ spin_lock_init(&(*nhe)->encap_list_lock);
1031	1083	INIT_LIST_HEAD(&(*nhe)->encap_list);
1032	1084	refcount_set(&(*nhe)->refcnt, 1);
1033	1085
@@ -1041,19 +1093,6 @@ out_free: · 7 unmodified lines
1041	1093	return err;
1042	1094	}
1043	1095
1044		−static void mlx5e_rep_neigh_entry_destroy(struct mlx5e_priv *priv,
1045		− struct mlx5e_neigh_hash_entry *nhe)
1046		−{
1047		− /* The neigh hash entry must be removed from the hash table regardless
1048		− * of the reference count value, so it won't be found by the next
1049		− * neigh notification call. The neigh hash entry reference count is
1050		− * incremented only during creation and neigh notification calls and
1051		− * protects from freeing the nhe struct.
1052		− */
1053		− mlx5e_rep_neigh_entry_remove(priv, nhe);
1054		− mlx5e_rep_neigh_entry_release(nhe);
1055		−}
1056		−
1057	1096	int mlx5e_rep_encap_entry_attach(struct mlx5e_priv *priv,
1058	1097	struct mlx5e_encap_entry *e)
1059	1098	{
@@ -1066,16 +1105,26 @@ int mlx5e_rep_encap_entry_attach(struct mlx5e_priv *priv, · 6 unmodified lines
1066	1105	err = mlx5_tun_entropy_refcount_inc(tun_entropy, e->reformat_type);
1067	1106	if (err)
1068	1107	return err;
	1108	+
	1109	+ mutex_lock(&rpriv->neigh_update.encap_lock);
1069	1110	nhe = mlx5e_rep_neigh_entry_lookup(priv, &e->m_neigh);
1070	1111	if (!nhe) {
1071	1112	err = mlx5e_rep_neigh_entry_create(priv, e, &nhe);
1072	1113	if (err) {
	1114	+ mutex_unlock(&rpriv->neigh_update.encap_lock);
1073	1115	mlx5_tun_entropy_refcount_dec(tun_entropy,
1074	1116	e->reformat_type);
1075	1117	return err;
1076	1118	}
1077	1119	}
1078		− list_add(&e->encap_list, &nhe->encap_list);
	1120	+
	1121	+ e->nhe = nhe;
	1122	+ spin_lock(&nhe->encap_list_lock);
	1123	+ list_add_rcu(&e->encap_list, &nhe->encap_list);
	1124	+ spin_unlock(&nhe->encap_list_lock);
	1125	+
	1126	+ mutex_unlock(&rpriv->neigh_update.encap_lock);
	1127	+
1079	1128	return 0;
1080	1129	}
1081	1130
@@ -1085,13 +1134,16 @@ void mlx5e_rep_encap_entry_detach(struct mlx5e_priv *priv, · 3 unmodified lines
1085	1134	struct mlx5e_rep_priv *rpriv = priv->ppriv;
1086	1135	struct mlx5_rep_uplink_priv *uplink_priv = &rpriv->uplink_priv;
1087	1136	struct mlx5_tun_entropy *tun_entropy = &uplink_priv->tun_entropy;
1088		− struct mlx5e_neigh_hash_entry *nhe;
1089	1137
1090		− list_del(&e->encap_list);
1091		− nhe = mlx5e_rep_neigh_entry_lookup(priv, &e->m_neigh);
	1138	+ if (!e->nhe)
	1139	+ return;
	1140	+
	1141	+ spin_lock(&e->nhe->encap_list_lock);
	1142	+ list_del_rcu(&e->encap_list);
	1143	+ spin_unlock(&e->nhe->encap_list_lock);
1092	1144
1093		− if (list_empty(&nhe->encap_list))
1094		− mlx5e_rep_neigh_entry_destroy(priv, nhe);
	1145	+ mlx5e_rep_neigh_entry_release(e->nhe);
	1146	+ e->nhe = NULL;
1095	1147	mlx5_tun_entropy_refcount_dec(tun_entropy, e->reformat_type);
1096	1148	}
1097	1149

drivers/net/ethernet/mellanox/mlx5/core/en_rep.h+10−1

@@ -35,6 +35,7 @@ · 34 unmodified lines
35	35
36	36	#include <net/ip_tunnels.h>
37	37	#include <linux/rhashtable.h>
	38	+#include <linux/mutex.h>
38	39	#include "eswitch.h"
39	40	#include "en.h"
40	41	#include "lib/port_tun.h"
@@ -48,7 +49,7 @@ struct mlx5e_neigh_update_table { · 7 unmodified lines
48	49	*/
49	50	struct list_head neigh_list;
50	51	/* protect lookup/remove operations */
51		− spinlock_t encap_lock;
	52	+ struct mutex encap_lock;
52	53	struct notifier_block netevent_nb;
53	54	struct delayed_work neigh_stats_work;
54	55	unsigned long min_interval; /* jiffies */
@@ -110,6 +111,7 @@ struct mlx5e_neigh { · 55 unmodified lines
110	111	struct mlx5e_neigh_hash_entry {
111	112	struct rhash_head rhash_node;
112	113	struct mlx5e_neigh m_neigh;
	114	+ struct mlx5e_priv *priv;
113	115
114	116	/* Save the neigh hash entry in a list on the representor in
115	117	* addition to the hash table. In order to iterate easily over the
@@ -117,6 +119,8 @@ struct mlx5e_neigh_hash_entry { · 1 unmodified line
117	119	*/
118	120	struct list_head neigh_list;
119	121
	122	+ /* protects encap list */
	123	+ spinlock_t encap_list_lock;
120	124	/* encap list sharing the same neigh */
121	125	struct list_head encap_list;
122	126
@@ -137,6 +141,8 @@ struct mlx5e_neigh_hash_entry { · 14 unmodified lines
137	141	* 'used' value and avoid neigh deleting by the kernel.
138	142	*/
139	143	unsigned long reported_lastuse;
	144	+
	145	+ struct rcu_head rcu;
140	146	};
141	147
142	148	enum {
@@ -145,6 +151,8 @@ enum { · 2 unmodified lines
145	151	};
146	152
147	153	struct mlx5e_encap_entry {
	154	+ /* attached neigh hash entry */
	155	+ struct mlx5e_neigh_hash_entry *nhe;
148	156	/* neigh hash entry list of encaps sharing the same neigh */
149	157	struct list_head encap_list;
150	158	struct mlx5e_neigh m_neigh;
@@ -167,6 +175,7 @@ struct mlx5e_encap_entry { · 16 unmodified lines
167	175	refcount_t refcnt;
168	176	struct completion res_ready;
169	177	int compl_result;
	178	+ struct rcu_head rcu;
170	179	};
171	180
172	181	struct mlx5e_rep_sq {

drivers/net/ethernet/mellanox/mlx5/core/en_tc.c+116−36

@@ -56,6 +56,7 @@ · 55 unmodified lines
56	56	#include "en/tc_tun.h"
57	57	#include "lib/devcom.h"
58	58	#include "lib/geneve.h"
	59	+#include "diag/en_tc_tracepoint.h"
59	60
60	61	struct mlx5_nic_flow_attr {
61	62	u32 action;
@@ -126,8 +127,11 @@ struct mlx5e_tc_flow { · 64 unmodified lines
126	127	struct list_head hairpin; /* flows sharing the same hairpin */
127	128	struct list_head peer; /* flows with peer flow */
128	129	struct list_head unready; /* flows not ready to be offloaded (e.g due to missing route) */
	130	+ int tmp_efi_index;
	131	+ struct list_head tmp_list; /* temporary flow list used by neigh update */
129	132	refcount_t refcnt;
130	133	struct rcu_head rcu_head;
	134	+ struct completion init_done;
131	135	union {
132	136	struct mlx5_esw_flow_attr esw_attr[0];
133	137	struct mlx5_nic_flow_attr nic_attr[0];
@@ -1290,11 +1294,11 @@ static void mlx5e_tc_del_fdb_flow(struct mlx5e_priv *priv, · 1156 unmodified lines
1290	1294	}
1291	1295
1292	1296	void mlx5e_tc_encap_flows_add(struct mlx5e_priv *priv,
1293		− struct mlx5e_encap_entry *e)
	1297	+ struct mlx5e_encap_entry *e,
	1298	+ struct list_head *flow_list)
1294	1299	{
1295	1300	struct mlx5_eswitch *esw = priv->mdev->priv.eswitch;
1296	1301	struct mlx5_esw_flow_attr slow_attr, *esw_attr;
1297		− struct encap_flow_item efi, tmp;
1298	1302	struct mlx5_flow_handle *rule;
1299	1303	struct mlx5_flow_spec *spec;
1300	1304	struct mlx5e_tc_flow *flow;
@@ -1313,19 +1317,17 @@ void mlx5e_tc_encap_flows_add(struct mlx5e_priv *priv, · 12 unmodified lines
1313	1317	e->flags \|= MLX5_ENCAP_ENTRY_VALID;
1314	1318	mlx5e_rep_queue_neigh_stats_work(priv);
1315	1319
1316		− list_for_each_entry_safe(efi, tmp, &e->flows, list) {
	1320	+ list_for_each_entry(flow, flow_list, tmp_list) {
1317	1321	bool all_flow_encaps_valid = true;
1318	1322	int i;
1319	1323
1320		− flow = container_of(efi, struct mlx5e_tc_flow, encaps[efi->index]);
1321		− if (IS_ERR(mlx5e_flow_get(flow)))
	1324	+ if (!mlx5e_is_offloaded_flow(flow))
1322	1325	continue;
1323		−
1324	1326	esw_attr = flow->esw_attr;
1325	1327	spec = &esw_attr->parse_attr->spec;
1326	1328
1327		− esw_attr->dests[efi->index].encap_id = e->encap_id;
1328		− esw_attr->dests[efi->index].flags \|= MLX5_ESW_DEST_ENCAP_VALID;
	1329	+ esw_attr->dests[flow->tmp_efi_index].encap_id = e->encap_id;
	1330	+ esw_attr->dests[flow->tmp_efi_index].flags \|= MLX5_ESW_DEST_ENCAP_VALID;
1329	1331	/* Flow can be associated with multiple encap entries.
1330	1332	* Before offloading the flow verify that all of them have
1331	1333	* a valid neighbour.
@@ -1340,63 +1342,55 @@ void mlx5e_tc_encap_flows_add(struct mlx5e_priv *priv, · 8 unmodified lines
1340	1342	}
1341	1343	/* Do not offload flows with unresolved neighbors */
1342	1344	if (!all_flow_encaps_valid)
1343		− goto loop_cont;
	1345	+ continue;
1344	1346	/* update from slow path rule to encap rule */
1345	1347	rule = mlx5e_tc_offload_fdb_rules(esw, flow, spec, esw_attr);
1346	1348	if (IS_ERR(rule)) {
1347	1349	err = PTR_ERR(rule);
1348	1350	mlx5_core_warn(priv->mdev, "Failed to update cached encapsulation flow, %d\n",
1349	1351	err);
1350		− goto loop_cont;
	1352	+ continue;
1351	1353	}
1352	1354
1353	1355	mlx5e_tc_unoffload_from_slow_path(esw, flow, &slow_attr);
1354	1356	flow->rule[0] = rule;
1355	1357	/* was unset when slow path rule removed */
1356	1358	flow_flag_set(flow, OFFLOADED);
1357		−
1358		−loop_cont:
1359		− mlx5e_flow_put(priv, flow);
1360	1359	}
1361	1360	}
1362	1361
1363	1362	void mlx5e_tc_encap_flows_del(struct mlx5e_priv *priv,
1364		− struct mlx5e_encap_entry *e)
	1363	+ struct mlx5e_encap_entry *e,
	1364	+ struct list_head *flow_list)
1365	1365	{
1366	1366	struct mlx5_eswitch *esw = priv->mdev->priv.eswitch;
1367	1367	struct mlx5_esw_flow_attr slow_attr;
1368		− struct encap_flow_item efi, tmp;
1369	1368	struct mlx5_flow_handle *rule;
1370	1369	struct mlx5_flow_spec *spec;
1371	1370	struct mlx5e_tc_flow *flow;
1372	1371	int err;
1373	1372
1374		− list_for_each_entry_safe(efi, tmp, &e->flows, list) {
1375		− flow = container_of(efi, struct mlx5e_tc_flow, encaps[efi->index]);
1376		− if (IS_ERR(mlx5e_flow_get(flow)))
	1373	+ list_for_each_entry(flow, flow_list, tmp_list) {
	1374	+ if (!mlx5e_is_offloaded_flow(flow))
1377	1375	continue;
1378		−
1379	1376	spec = &flow->esw_attr->parse_attr->spec;
1380	1377
1381	1378	/* update from encap rule to slow path rule */
1382	1379	rule = mlx5e_tc_offload_to_slow_path(esw, flow, spec, &slow_attr);
1383	1380	/* mark the flow's encap dest as non-valid */
1384		− flow->esw_attr->dests[efi->index].flags &= ~MLX5_ESW_DEST_ENCAP_VALID;
	1381	+ flow->esw_attr->dests[flow->tmp_efi_index].flags &= ~MLX5_ESW_DEST_ENCAP_VALID;
1385	1382
1386	1383	if (IS_ERR(rule)) {
1387	1384	err = PTR_ERR(rule);
1388	1385	mlx5_core_warn(priv->mdev, "Failed to update slow path (encap) flow, %d\n",
1389	1386	err);
1390		− goto loop_cont;
	1387	+ continue;
1391	1388	}
1392	1389
1393	1390	mlx5e_tc_unoffload_fdb_rules(esw, flow, flow->esw_attr);
1394	1391	flow->rule[0] = rule;
1395	1392	/* was unset when fast path rule removed */
1396	1393	flow_flag_set(flow, OFFLOADED);
1397		−
1398		−loop_cont:
1399		− mlx5e_flow_put(priv, flow);
1400	1394	}
1401	1395
1402	1396	/* we know that the encap is valid */
@@ -1412,11 +1406,84 @@ static struct mlx5_fc mlx5e_tc_get_counter(struct mlx5e_tc_flow flow) · 9 unmodified lines
1412	1406	return flow->nic_attr->counter;
1413	1407	}
1414	1408
	1409	+/* Takes reference to all flows attached to encap and adds the flows to
	1410	+ * flow_list using 'tmp_list' list_head in mlx5e_tc_flow.
	1411	+ */
	1412	+void mlx5e_take_all_encap_flows(struct mlx5e_encap_entry e, struct list_head flow_list)
	1413	+{
	1414	+ struct encap_flow_item *efi;
	1415	+ struct mlx5e_tc_flow *flow;
	1416	+
	1417	+ list_for_each_entry(efi, &e->flows, list) {
	1418	+ flow = container_of(efi, struct mlx5e_tc_flow, encaps[efi->index]);
	1419	+ if (IS_ERR(mlx5e_flow_get(flow)))
	1420	+ continue;
	1421	+ wait_for_completion(&flow->init_done);
	1422	+
	1423	+ flow->tmp_efi_index = efi->index;
	1424	+ list_add(&flow->tmp_list, flow_list);
	1425	+ }
	1426	+}
	1427	+
	1428	+/* Iterate over tmp_list of flows attached to flow_list head. */
	1429	+void mlx5e_put_encap_flow_list(struct mlx5e_priv priv, struct list_head flow_list)
	1430	+{
	1431	+ struct mlx5e_tc_flow flow, tmp;
	1432	+
	1433	+ list_for_each_entry_safe(flow, tmp, flow_list, tmp_list)
	1434	+ mlx5e_flow_put(priv, flow);
	1435	+}
	1436	+
	1437	+static struct mlx5e_encap_entry *
	1438	+mlx5e_get_next_valid_encap(struct mlx5e_neigh_hash_entry *nhe,
	1439	+ struct mlx5e_encap_entry *e)
	1440	+{
	1441	+ struct mlx5e_encap_entry *next = NULL;
	1442	+
	1443	+retry:
	1444	+ rcu_read_lock();
	1445	+
	1446	+ /* find encap with non-zero reference counter value */
	1447	+ for (next = e ?
	1448	+ list_next_or_null_rcu(&nhe->encap_list,
	1449	+ &e->encap_list,
	1450	+ struct mlx5e_encap_entry,
	1451	+ encap_list) :
	1452	+ list_first_or_null_rcu(&nhe->encap_list,
	1453	+ struct mlx5e_encap_entry,
	1454	+ encap_list);
	1455	+ next;
	1456	+ next = list_next_or_null_rcu(&nhe->encap_list,
	1457	+ &next->encap_list,
	1458	+ struct mlx5e_encap_entry,
	1459	+ encap_list))
	1460	+ if (mlx5e_encap_take(next))
	1461	+ break;
	1462	+
	1463	+ rcu_read_unlock();
	1464	+
	1465	+ /* release starting encap */
	1466	+ if (e)
	1467	+ mlx5e_encap_put(netdev_priv(e->out_dev), e);
	1468	+ if (!next)
	1469	+ return next;
	1470	+
	1471	+ /* wait for encap to be fully initialized */
	1472	+ wait_for_completion(&next->res_ready);
	1473	+ /* continue searching if encap entry is not in valid state after completion */
	1474	+ if (!(next->flags & MLX5_ENCAP_ENTRY_VALID)) {
	1475	+ e = next;
	1476	+ goto retry;
	1477	+ }
	1478	+
	1479	+ return next;
	1480	+}
	1481	+
1415	1482	void mlx5e_tc_update_neigh_used_value(struct mlx5e_neigh_hash_entry *nhe)
1416	1483	{
1417	1484	struct mlx5e_neigh *m_neigh = &nhe->m_neigh;
	1485	+ struct mlx5e_encap_entry *e = NULL;
1418	1486	struct mlx5e_tc_flow *flow;
1419		− struct mlx5e_encap_entry *e;
1420	1487	struct mlx5_fc *counter;
1421	1488	struct neigh_table *tbl;
1422	1489	bool neigh_used = false;
@@ -1432,37 +1499,45 @@ void mlx5e_tc_update_neigh_used_value(struct mlx5e_neigh_hash_entry *nhe) · 9 unmodified lines
1432	1499	else
1433	1500	return;
1434	1501
1435		− list_for_each_entry(e, &nhe->encap_list, encap_list) {
	1502	+ /* mlx5e_get_next_valid_encap() releases previous encap before returning
	1503	+ * next one.
	1504	+ */
	1505	+ while ((e = mlx5e_get_next_valid_encap(nhe, e)) != NULL) {
	1506	+ struct mlx5e_priv *priv = netdev_priv(e->out_dev);
1436	1507	struct encap_flow_item efi, tmp;
	1508	+ struct mlx5_eswitch *esw;
	1509	+ LIST_HEAD(flow_list);
1437	1510
1438		− if (!(e->flags & MLX5_ENCAP_ENTRY_VALID) \|\|
1439		− !mlx5e_encap_take(e))
1440		− continue;
1441		−
	1511	+ esw = priv->mdev->priv.eswitch;
	1512	+ mutex_lock(&esw->offloads.encap_tbl_lock);
1442	1513	list_for_each_entry_safe(efi, tmp, &e->flows, list) {
1443	1514	flow = container_of(efi, struct mlx5e_tc_flow,
1444	1515	encaps[efi->index]);
1445	1516	if (IS_ERR(mlx5e_flow_get(flow)))
1446	1517	continue;
	1518	+ list_add(&flow->tmp_list, &flow_list);
1447	1519
1448	1520	if (mlx5e_is_offloaded_flow(flow)) {
1449	1521	counter = mlx5e_tc_get_counter(flow);
1450	1522	lastuse = mlx5_fc_query_lastuse(counter);
1451	1523	if (time_after((unsigned long)lastuse, nhe->reported_lastuse)) {
1452		− mlx5e_flow_put(netdev_priv(e->out_dev), flow);
1453	1524	neigh_used = true;
1454	1525	break;
1455	1526	}
1456	1527	}
1457		−
1458		− mlx5e_flow_put(netdev_priv(e->out_dev), flow);
1459	1528	}
	1529	+ mutex_unlock(&esw->offloads.encap_tbl_lock);
1460	1530
1461		− mlx5e_encap_put(netdev_priv(e->out_dev), e);
1462		− if (neigh_used)
	1531	+ mlx5e_put_encap_flow_list(priv, &flow_list);
	1532	+ if (neigh_used) {
	1533	+ /* release current encap before breaking the loop */
	1534	+ mlx5e_encap_put(priv, e);
1463	1535	break;
	1536	+ }
1464	1537	}
1465	1538
	1539	+ trace_mlx5e_tc_update_neigh_used_value(nhe, neigh_used);
	1540	+
1466	1541	if (neigh_used) {
1467	1542	nhe->reported_lastuse = jiffies;
1468	1543
@@ -1490,7 +1565,7 @@ static void mlx5e_encap_dealloc(struct mlx5e_priv *priv, struct mlx5e_encap_entr · 21 unmodified lines
1490	1565	}
1491	1566
1492	1567	kfree(e->encap_header);
1493		− kfree(e);
	1568	+ kfree_rcu(e, rcu);
1494	1569	}
1495	1570
1496	1571	void mlx5e_encap_put(struct mlx5e_priv priv, struct mlx5e_encap_entry e)
@@ -3426,6 +3501,7 @@ mlx5e_alloc_flow(struct mlx5e_priv *priv, int attr_size, · 1929 unmodified lines
3426	3501	INIT_LIST_HEAD(&flow->mod_hdr);
3427	3502	INIT_LIST_HEAD(&flow->hairpin);
3428	3503	refcount_set(&flow->refcnt, 1);
	3504	+ init_completion(&flow->init_done);
3429	3505
3430	3506	*__flow = flow;
3431	3507	*__parse_attr = parse_attr;
@@ -3498,6 +3574,7 @@ __mlx5e_add_fdb_flow(struct mlx5e_priv *priv, · 66 unmodified lines
3498	3574	goto err_free;
3499	3575
3500	3576	err = mlx5e_tc_add_fdb_flow(priv, flow, extack);
	3577	+ complete_all(&flow->init_done);
3501	3578	if (err) {
3502	3579	if (!(err == -ENETUNREACH && mlx5_lag_is_multipath(in_mdev)))
3503	3580	goto err_free;
@@ -3695,6 +3772,7 @@ int mlx5e_configure_flower(struct net_device dev, struct mlx5e_priv priv, · 191 unmodified lines
3695	3772	goto out;
3696	3773	}
3697	3774
	3775	+ trace_mlx5e_configure_flower(f);
3698	3776	err = mlx5e_tc_add_flow(priv, f, flags, dev, &flow);
3699	3777	if (err)
3700	3778	goto out;
@@ -3744,6 +3822,7 @@ int mlx5e_delete_flower(struct net_device dev, struct mlx5e_priv priv, · 43 unmodified lines
3744	3822	rhashtable_remove_fast(tc_ht, &flow->node, tc_ht_params);
3745	3823	rcu_read_unlock();
3746	3824
	3825	+ trace_mlx5e_delete_flower(f);
3747	3826	mlx5e_flow_put(priv, flow);
3748	3827
3749	3828	return 0;
@@ -3813,6 +3892,7 @@ no_peer_counter: · 63 unmodified lines
3813	3892	mlx5_devcom_release_peer_data(devcom, MLX5_DEVCOM_ESW_OFFLOADS);
3814	3893	out:
3815	3894	flow_stats_update(&f->stats, bytes, packets, lastuse);
	3895	+ trace_mlx5e_stats_flower(f);
3816	3896	errout:
3817	3897	mlx5e_flow_put(priv, flow);
3818	3898	return err;

drivers/net/ethernet/mellanox/mlx5/core/en_tc.h+7−2

@@ -72,12 +72,17 @@ void mlx5e_tc_stats_matchall(struct mlx5e_priv *priv, · 71 unmodified lines
72	72
73	73	struct mlx5e_encap_entry;
74	74	void mlx5e_tc_encap_flows_add(struct mlx5e_priv *priv,
75		− struct mlx5e_encap_entry *e);
	75	+ struct mlx5e_encap_entry *e,
	76	+ struct list_head *flow_list);
76	77	void mlx5e_tc_encap_flows_del(struct mlx5e_priv *priv,
77		− struct mlx5e_encap_entry *e);
	78	+ struct mlx5e_encap_entry *e,
	79	+ struct list_head *flow_list);
78	80	bool mlx5e_encap_take(struct mlx5e_encap_entry *e);
79	81	void mlx5e_encap_put(struct mlx5e_priv priv, struct mlx5e_encap_entry e);
80	82
	83	+void mlx5e_take_all_encap_flows(struct mlx5e_encap_entry e, struct list_head flow_list);
	84	+void mlx5e_put_encap_flow_list(struct mlx5e_priv priv, struct list_head flow_list);
	85	+
81	86	struct mlx5e_neigh_hash_entry;
82	87	void mlx5e_tc_update_neigh_used_value(struct mlx5e_neigh_hash_entry *nhe);
83	88

include/net/flow_offload.h+1−0

@@ -138,6 +138,7 @@ enum flow_action_id { · 137 unmodified lines
138	138	FLOW_ACTION_MPLS_PUSH,
139	139	FLOW_ACTION_MPLS_POP,
140	140	FLOW_ACTION_MPLS_MANGLE,
	141	+ NUM_FLOW_ACTIONS,
141	142	};
142	143
143	144	/* This is mirroring enum pedit_header_type definition for easy mapping between

Merge tag 'mlx5-updates-2019-08-21' of git://git.kernel.org/pub/scm/linux/kernel/git/saeed/linux

Keyboard shortcuts

General

Issues and pull requests

Issues, pull requests and Inbox

Repository code