History

net: dev: Add hardware stats support

Offloading switch device drivers may be able to collect statistics of the traffic taking place in the HW datapath that pertains to a certain soft netdevice, such as VLAN. Add the necessary infrastructure to allow exposing these statistics to the offloaded netdevice in question. The API was shaped by the following considerations: - Collection of HW statistics is not free: there may be a finite number of counters, and the act of counting may have a performance impact. It is therefore necessary to allow toggling whether HW counting should be done for any particular SW netdevice. - As the drivers are loaded and removed, a particular device may get offloaded and unoffloaded again. At the same time, the statistics values need to stay monotonic (modulo the eventual 64-bit wraparound), increasing only to reflect traffic measured in the device. To that end, the netdevice keeps around a lazily-allocated copy of struct rtnl_link_stats64. Device drivers then contribute to the values kept therein at various points. Even as the driver goes away, the struct stays around to maintain the statistics values. - Different HW devices may be able to count different things. The motivation behind this patch in particular is exposure of HW counters on Nvidia Spectrum switches, where the only practical approach to counting traffic on offloaded soft netdevices currently is to use router interface counters, and count L3 traffic. Correspondingly that is the statistics suite added in this patch. Other devices may be able to measure different kinds of traffic, and for that reason, the APIs are built to allow uniform access to different statistics suites. - Because soft netdevices and offloading drivers are only loosely bound, a netdevice uses a notifier chain to communicate with the drivers. Several new notifiers, NETDEV_OFFLOAD_XSTATS_*, have been added to carry messages to the offloading drivers. - Devices can have various conditions for when a particular counter is available. As the device is configured and reconfigured, the device offload may become or cease being suitable for counter binding. A netdevice can use a notifier type NETDEV_OFFLOAD_XSTATS_REPORT_USED to ping offloading drivers and determine whether anyone currently implements a given statistics suite. This information can then be propagated to user space. When the driver decides to unoffload a netdevice, it can use a newly-added function, netdev_offload_xstats_report_delta(), to record outstanding collected statistics, before destroying the HW counter. This patch adds a helper, call_netdevice_notifiers_info_robust(), for dispatching a notifier with the possibility of unwind when one of the consumers bails. Given the wish to eventually get rid of the global notifier block altogether, this helper only invokes the per-netns notifier block.

Petr Machata committed 4 years ago

Commit 9309f971 parent 216e6903 files · +323 −1Browse files

Unified · Split

3 files
  1. include/linux/netdevice.h +42 −0
  2. include/uapi/linux/if_link.h +15 −0
  3. net/core/dev.c +266 −1
include/linux/netdevice.h+42−0
@@ -1950,6 +1950,7 @@ enum netdev_ml_priv_type { · 1949 unmodified lines
19501950 * @watchdog_dev_tracker: refcount tracker used by watchdog.
19511951 * @dev_registered_tracker: tracker for reference held while
19521952 * registered
1953+ * @offload_xstats_l3: L3 HW stats for this netdevice.
19531954 *
19541955 * FIXME: cleanup struct net_device such that network protocol info
19551956 * moves out.
@@ -2287,6 +2288,7 @@ struct net_device { · 331 unmodified lines
22872288 netdevice_tracker linkwatch_dev_tracker;
22882289 netdevice_tracker watchdog_dev_tracker;
22892290 netdevice_tracker dev_registered_tracker;
2291+ struct rtnl_hw_stats64 *offload_xstats_l3;
22902292 };
22912293 #define to_net_dev(d) container_of(d, struct net_device, dev)
22922294
@@ -2727,6 +2729,10 @@ enum netdev_cmd { · 434 unmodified lines
27272729 NETDEV_CVLAN_FILTER_DROP_INFO,
27282730 NETDEV_SVLAN_FILTER_PUSH_INFO,
27292731 NETDEV_SVLAN_FILTER_DROP_INFO,
2732+ NETDEV_OFFLOAD_XSTATS_ENABLE,
2733+ NETDEV_OFFLOAD_XSTATS_DISABLE,
2734+ NETDEV_OFFLOAD_XSTATS_REPORT_USED,
2735+ NETDEV_OFFLOAD_XSTATS_REPORT_DELTA,
27302736 };
27312737 const char *netdev_cmd_to_name(enum netdev_cmd cmd);
27322738
@@ -2777,6 +2783,42 @@ struct netdev_notifier_pre_changeaddr_info { · 44 unmodified lines
27772783 const unsigned char *dev_addr;
27782784 };
27792785
2786+enum netdev_offload_xstats_type {
2787+ NETDEV_OFFLOAD_XSTATS_TYPE_L3 = 1,
2788+};
2789+
2790+struct netdev_notifier_offload_xstats_info {
2791+ struct netdev_notifier_info info; /* must be first */
2792+ enum netdev_offload_xstats_type type;
2793+
2794+ union {
2795+ /* NETDEV_OFFLOAD_XSTATS_REPORT_DELTA */
2796+ struct netdev_notifier_offload_xstats_rd *report_delta;
2797+ /* NETDEV_OFFLOAD_XSTATS_REPORT_USED */
2798+ struct netdev_notifier_offload_xstats_ru *report_used;
2799+ };
2800+};
2801+
2802+int netdev_offload_xstats_enable(struct net_device *dev,
2803+ enum netdev_offload_xstats_type type,
2804+ struct netlink_ext_ack *extack);
2805+int netdev_offload_xstats_disable(struct net_device *dev,
2806+ enum netdev_offload_xstats_type type);
2807+bool netdev_offload_xstats_enabled(const struct net_device *dev,
2808+ enum netdev_offload_xstats_type type);
2809+int netdev_offload_xstats_get(struct net_device *dev,
2810+ enum netdev_offload_xstats_type type,
2811+ struct rtnl_hw_stats64 *stats, bool *used,
2812+ struct netlink_ext_ack *extack);
2813+void
2814+netdev_offload_xstats_report_delta(struct netdev_notifier_offload_xstats_rd *rd,
2815+ const struct rtnl_hw_stats64 *stats);
2816+void
2817+netdev_offload_xstats_report_used(struct netdev_notifier_offload_xstats_ru *ru);
2818+void netdev_offload_xstats_push_delta(struct net_device *dev,
2819+ enum netdev_offload_xstats_type type,
2820+ const struct rtnl_hw_stats64 *stats);
2821+
27802822 static inline void netdev_notifier_info_init(struct netdev_notifier_info *info,
27812823 struct net_device *dev)
27822824 {
include/uapi/linux/if_link.h+15−0
@@ -245,6 +245,21 @@ struct rtnl_link_stats64 { · 244 unmodified lines
245245 __u64 rx_nohandler;
246246 };
247247
248+/* Subset of link stats useful for in-HW collection. Meaning of the fields is as
249+ * for struct rtnl_link_stats64.
250+ */
251+struct rtnl_hw_stats64 {
252+ __u64 rx_packets;
253+ __u64 tx_packets;
254+ __u64 rx_bytes;
255+ __u64 tx_bytes;
256+ __u64 rx_errors;
257+ __u64 tx_errors;
258+ __u64 rx_dropped;
259+ __u64 tx_dropped;
260+ __u64 multicast;
261+};
262+
248263 /* The struct should be in sync with struct ifmap */
249264 struct rtnl_link_ifmap {
250265 __u64 mem_start;
net/core/dev.c+266−1
@@ -1622,7 +1622,8 @@ const char *netdev_cmd_to_name(enum netdev_cmd cmd) · 1621 unmodified lines
16221622 N(UDP_TUNNEL_DROP_INFO) N(CHANGE_TX_QUEUE_LEN)
16231623 N(CVLAN_FILTER_PUSH_INFO) N(CVLAN_FILTER_DROP_INFO)
16241624 N(SVLAN_FILTER_PUSH_INFO) N(SVLAN_FILTER_DROP_INFO)
1625− N(PRE_CHANGEADDR)
1625+ N(PRE_CHANGEADDR) N(OFFLOAD_XSTATS_ENABLE) N(OFFLOAD_XSTATS_DISABLE)
1626+ N(OFFLOAD_XSTATS_REPORT_USED) N(OFFLOAD_XSTATS_REPORT_DELTA)
16261627 }
16271628 #undef N
16281629 return "UNKNOWN_NETDEV_EVENT";
@@ -1939,6 +1940,32 @@ static int call_netdevice_notifiers_info(unsigned long val, · 310 unmodified lines
19391940 return raw_notifier_call_chain(&netdev_chain, val, info);
19401941 }
19411942
1943+/**
1944+ * call_netdevice_notifiers_info_robust - call per-netns notifier blocks
1945+ * for and rollback on error
1946+ * @val_up: value passed unmodified to notifier function
1947+ * @val_down: value passed unmodified to the notifier function when
1948+ * recovering from an error on @val_up
1949+ * @info: notifier information data
1950+ *
1951+ * Call all per-netns network notifier blocks, but not notifier blocks on
1952+ * the global notifier chain. Parameters and return value are as for
1953+ * raw_notifier_call_chain_robust().
1954+ */
1955+
1956+static int
1957+call_netdevice_notifiers_info_robust(unsigned long val_up,
1958+ unsigned long val_down,
1959+ struct netdev_notifier_info *info)
1960+{
1961+ struct net *net = dev_net(info->dev);
1962+
1963+ ASSERT_RTNL();
1964+
1965+ return raw_notifier_call_chain_robust(&net->netdev_chain,
1966+ val_up, val_down, info);
1967+}
1968+
19421969 static int call_netdevice_notifiers_extack(unsigned long val,
19431970 struct net_device *dev,
19441971 struct netlink_ext_ack *extack)
@@ -7728,6 +7755,242 @@ void netdev_bonding_info_change(struct net_device *dev, · 5783 unmodified lines
77287755 }
77297756 EXPORT_SYMBOL(netdev_bonding_info_change);
77307757
7758+static int netdev_offload_xstats_enable_l3(struct net_device *dev,
7759+ struct netlink_ext_ack *extack)
7760+{
7761+ struct netdev_notifier_offload_xstats_info info = {
7762+ .info.dev = dev,
7763+ .info.extack = extack,
7764+ .type = NETDEV_OFFLOAD_XSTATS_TYPE_L3,
7765+ };
7766+ int err;
7767+ int rc;
7768+
7769+ dev->offload_xstats_l3 = kzalloc(sizeof(*dev->offload_xstats_l3),
7770+ GFP_KERNEL);
7771+ if (!dev->offload_xstats_l3)
7772+ return -ENOMEM;
7773+
7774+ rc = call_netdevice_notifiers_info_robust(NETDEV_OFFLOAD_XSTATS_ENABLE,
7775+ NETDEV_OFFLOAD_XSTATS_DISABLE,
7776+ &info.info);
7777+ err = notifier_to_errno(rc);
7778+ if (err)
7779+ goto free_stats;
7780+
7781+ return 0;
7782+
7783+free_stats:
7784+ kfree(dev->offload_xstats_l3);
7785+ dev->offload_xstats_l3 = NULL;
7786+ return err;
7787+}
7788+
7789+int netdev_offload_xstats_enable(struct net_device *dev,
7790+ enum netdev_offload_xstats_type type,
7791+ struct netlink_ext_ack *extack)
7792+{
7793+ ASSERT_RTNL();
7794+
7795+ if (netdev_offload_xstats_enabled(dev, type))
7796+ return -EALREADY;
7797+
7798+ switch (type) {
7799+ case NETDEV_OFFLOAD_XSTATS_TYPE_L3:
7800+ return netdev_offload_xstats_enable_l3(dev, extack);
7801+ }
7802+
7803+ WARN_ON(1);
7804+ return -EINVAL;
7805+}
7806+EXPORT_SYMBOL(netdev_offload_xstats_enable);
7807+
7808+static void netdev_offload_xstats_disable_l3(struct net_device *dev)
7809+{
7810+ struct netdev_notifier_offload_xstats_info info = {
7811+ .info.dev = dev,
7812+ .type = NETDEV_OFFLOAD_XSTATS_TYPE_L3,
7813+ };
7814+
7815+ call_netdevice_notifiers_info(NETDEV_OFFLOAD_XSTATS_DISABLE,
7816+ &info.info);
7817+ kfree(dev->offload_xstats_l3);
7818+ dev->offload_xstats_l3 = NULL;
7819+}
7820+
7821+int netdev_offload_xstats_disable(struct net_device *dev,
7822+ enum netdev_offload_xstats_type type)
7823+{
7824+ ASSERT_RTNL();
7825+
7826+ if (!netdev_offload_xstats_enabled(dev, type))
7827+ return -EALREADY;
7828+
7829+ switch (type) {
7830+ case NETDEV_OFFLOAD_XSTATS_TYPE_L3:
7831+ netdev_offload_xstats_disable_l3(dev);
7832+ return 0;
7833+ }
7834+
7835+ WARN_ON(1);
7836+ return -EINVAL;
7837+}
7838+EXPORT_SYMBOL(netdev_offload_xstats_disable);
7839+
7840+static void netdev_offload_xstats_disable_all(struct net_device *dev)
7841+{
7842+ netdev_offload_xstats_disable(dev, NETDEV_OFFLOAD_XSTATS_TYPE_L3);
7843+}
7844+
7845+static struct rtnl_hw_stats64 *
7846+netdev_offload_xstats_get_ptr(const struct net_device *dev,
7847+ enum netdev_offload_xstats_type type)
7848+{
7849+ switch (type) {
7850+ case NETDEV_OFFLOAD_XSTATS_TYPE_L3:
7851+ return dev->offload_xstats_l3;
7852+ }
7853+
7854+ WARN_ON(1);
7855+ return NULL;
7856+}
7857+
7858+bool netdev_offload_xstats_enabled(const struct net_device *dev,
7859+ enum netdev_offload_xstats_type type)
7860+{
7861+ ASSERT_RTNL();
7862+
7863+ return netdev_offload_xstats_get_ptr(dev, type);
7864+}
7865+EXPORT_SYMBOL(netdev_offload_xstats_enabled);
7866+
7867+struct netdev_notifier_offload_xstats_ru {
7868+ bool used;
7869+};
7870+
7871+struct netdev_notifier_offload_xstats_rd {
7872+ struct rtnl_hw_stats64 stats;
7873+ bool used;
7874+};
7875+
7876+static void netdev_hw_stats64_add(struct rtnl_hw_stats64 *dest,
7877+ const struct rtnl_hw_stats64 *src)
7878+{
7879+ dest->rx_packets += src->rx_packets;
7880+ dest->tx_packets += src->tx_packets;
7881+ dest->rx_bytes += src->rx_bytes;
7882+ dest->tx_bytes += src->tx_bytes;
7883+ dest->rx_errors += src->rx_errors;
7884+ dest->tx_errors += src->tx_errors;
7885+ dest->rx_dropped += src->rx_dropped;
7886+ dest->tx_dropped += src->tx_dropped;
7887+ dest->multicast += src->multicast;
7888+}
7889+
7890+static int netdev_offload_xstats_get_used(struct net_device *dev,
7891+ enum netdev_offload_xstats_type type,
7892+ bool *p_used,
7893+ struct netlink_ext_ack *extack)
7894+{
7895+ struct netdev_notifier_offload_xstats_ru report_used = {};
7896+ struct netdev_notifier_offload_xstats_info info = {
7897+ .info.dev = dev,
7898+ .info.extack = extack,
7899+ .type = type,
7900+ .report_used = &report_used,
7901+ };
7902+ int rc;
7903+
7904+ WARN_ON(!netdev_offload_xstats_enabled(dev, type));
7905+ rc = call_netdevice_notifiers_info(NETDEV_OFFLOAD_XSTATS_REPORT_USED,
7906+ &info.info);
7907+ *p_used = report_used.used;
7908+ return notifier_to_errno(rc);
7909+}
7910+
7911+static int netdev_offload_xstats_get_stats(struct net_device *dev,
7912+ enum netdev_offload_xstats_type type,
7913+ struct rtnl_hw_stats64 *p_stats,
7914+ bool *p_used,
7915+ struct netlink_ext_ack *extack)
7916+{
7917+ struct netdev_notifier_offload_xstats_rd report_delta = {};
7918+ struct netdev_notifier_offload_xstats_info info = {
7919+ .info.dev = dev,
7920+ .info.extack = extack,
7921+ .type = type,
7922+ .report_delta = &report_delta,
7923+ };
7924+ struct rtnl_hw_stats64 *stats;
7925+ int rc;
7926+
7927+ stats = netdev_offload_xstats_get_ptr(dev, type);
7928+ if (WARN_ON(!stats))
7929+ return -EINVAL;
7930+
7931+ rc = call_netdevice_notifiers_info(NETDEV_OFFLOAD_XSTATS_REPORT_DELTA,
7932+ &info.info);
7933+
7934+ /* Cache whatever we got, even if there was an error, otherwise the
7935+ * successful stats retrievals would get lost.
7936+ */
7937+ netdev_hw_stats64_add(stats, &report_delta.stats);
7938+
7939+ if (p_stats)
7940+ *p_stats = *stats;
7941+ *p_used = report_delta.used;
7942+
7943+ return notifier_to_errno(rc);
7944+}
7945+
7946+int netdev_offload_xstats_get(struct net_device *dev,
7947+ enum netdev_offload_xstats_type type,
7948+ struct rtnl_hw_stats64 *p_stats, bool *p_used,
7949+ struct netlink_ext_ack *extack)
7950+{
7951+ ASSERT_RTNL();
7952+
7953+ if (p_stats)
7954+ return netdev_offload_xstats_get_stats(dev, type, p_stats,
7955+ p_used, extack);
7956+ else
7957+ return netdev_offload_xstats_get_used(dev, type, p_used,
7958+ extack);
7959+}
7960+EXPORT_SYMBOL(netdev_offload_xstats_get);
7961+
7962+void
7963+netdev_offload_xstats_report_delta(struct netdev_notifier_offload_xstats_rd *report_delta,
7964+ const struct rtnl_hw_stats64 *stats)
7965+{
7966+ report_delta->used = true;
7967+ netdev_hw_stats64_add(&report_delta->stats, stats);
7968+}
7969+EXPORT_SYMBOL(netdev_offload_xstats_report_delta);
7970+
7971+void
7972+netdev_offload_xstats_report_used(struct netdev_notifier_offload_xstats_ru *report_used)
7973+{
7974+ report_used->used = true;
7975+}
7976+EXPORT_SYMBOL(netdev_offload_xstats_report_used);
7977+
7978+void netdev_offload_xstats_push_delta(struct net_device *dev,
7979+ enum netdev_offload_xstats_type type,
7980+ const struct rtnl_hw_stats64 *p_stats)
7981+{
7982+ struct rtnl_hw_stats64 *stats;
7983+
7984+ ASSERT_RTNL();
7985+
7986+ stats = netdev_offload_xstats_get_ptr(dev, type);
7987+ if (WARN_ON(!stats))
7988+ return;
7989+
7990+ netdev_hw_stats64_add(stats, p_stats);
7991+}
7992+EXPORT_SYMBOL(netdev_offload_xstats_push_delta);
7993+
77317994 /**
77327995 * netdev_get_xmit_slave - Get the xmit slave of master device
77337996 * @dev: device
@@ -10417,6 +10680,8 @@ void unregister_netdevice_many(struct list_head *head) · 2683 unmodified lines
1041710680
1041810681 dev_xdp_uninstall(dev);
1041910682
10683+ netdev_offload_xstats_disable_all(dev);
10684+
1042010685 /* Notify protocols, that we are about to destroy
1042110686 * this device. They should clean all the things.
1042210687 */