History

net: dev: Add hardware stats support

Offloading switch device drivers may be able to collect statistics of the traffic taking place in the HW datapath that pertains to a certain soft netdevice, such as VLAN. Add the necessary infrastructure to allow exposing these statistics to the offloaded netdevice in question. The API was shaped by the following considerations: - Collection of HW statistics is not free: there may be a finite number of counters, and the act of counting may have a performance impact. It is therefore necessary to allow toggling whether HW counting should be done for any particular SW netdevice. - As the drivers are loaded and removed, a particular device may get offloaded and unoffloaded again. At the same time, the statistics values need to stay monotonic (modulo the eventual 64-bit wraparound), increasing only to reflect traffic measured in the device. To that end, the netdevice keeps around a lazily-allocated copy of struct rtnl_link_stats64. Device drivers then contribute to the values kept therein at various points. Even as the driver goes away, the struct stays around to maintain the statistics values. - Different HW devices may be able to count different things. The motivation behind this patch in particular is exposure of HW counters on Nvidia Spectrum switches, where the only practical approach to counting traffic on offloaded soft netdevices currently is to use router interface counters, and count L3 traffic. Correspondingly that is the statistics suite added in this patch. Other devices may be able to measure different kinds of traffic, and for that reason, the APIs are built to allow uniform access to different statistics suites. - Because soft netdevices and offloading drivers are only loosely bound, a netdevice uses a notifier chain to communicate with the drivers. Several new notifiers, NETDEV_OFFLOAD_XSTATS_*, have been added to carry messages to the offloading drivers. - Devices can have various conditions for when a particular counter is available. As the device is configured and reconfigured, the device offload may become or cease being suitable for counter binding. A netdevice can use a notifier type NETDEV_OFFLOAD_XSTATS_REPORT_USED to ping offloading drivers and determine whether anyone currently implements a given statistics suite. This information can then be propagated to user space. When the driver decides to unoffload a netdevice, it can use a newly-added function, netdev_offload_xstats_report_delta(), to record outstanding collected statistics, before destroying the HW counter. This patch adds a helper, call_netdevice_notifiers_info_robust(), for dispatching a notifier with the possibility of unwind when one of the consumers bails. Given the wish to eventually get rid of the global notifier block altogether, this helper only invokes the per-netns notifier block.

Petr Machata committed 4 years ago

Commit 9309f971 parent 216e6903 files · +323 −1Browse files

Unified · Split

3 files
  1. include/linux/netdevice.h +42 −0
  2. include/uapi/linux/if_link.h +15 −0
  3. net/core/dev.c +266 −1
include/linux/netdevice.h+42−0
@@ -1950,6 +1950,7 @@ enum netdev_ml_priv_type { · 1949 unmodified lines
1950 * @watchdog_dev_tracker: refcount tracker used by watchdog.1950 * @watchdog_dev_tracker: refcount tracker used by watchdog.
1951 * @dev_registered_tracker: tracker for reference held while1951 * @dev_registered_tracker: tracker for reference held while
1952 * registered1952 * registered
1953 * @offload_xstats_l3: L3 HW stats for this netdevice.
1953 *1954 *
1954 * FIXME: cleanup struct net_device such that network protocol info1955 * FIXME: cleanup struct net_device such that network protocol info
1955 * moves out.1956 * moves out.
@@ -2287,6 +2288,7 @@ struct net_device { · 331 unmodified lines
2287 netdevice_tracker linkwatch_dev_tracker;2288 netdevice_tracker linkwatch_dev_tracker;
2288 netdevice_tracker watchdog_dev_tracker;2289 netdevice_tracker watchdog_dev_tracker;
2289 netdevice_tracker dev_registered_tracker;2290 netdevice_tracker dev_registered_tracker;
2291 struct rtnl_hw_stats64 *offload_xstats_l3;
2290};2292};
2291#define to_net_dev(d) container_of(d, struct net_device, dev)2293#define to_net_dev(d) container_of(d, struct net_device, dev)
2292 2294
@@ -2727,6 +2729,10 @@ enum netdev_cmd { · 434 unmodified lines
2727 NETDEV_CVLAN_FILTER_DROP_INFO,2729 NETDEV_CVLAN_FILTER_DROP_INFO,
2728 NETDEV_SVLAN_FILTER_PUSH_INFO,2730 NETDEV_SVLAN_FILTER_PUSH_INFO,
2729 NETDEV_SVLAN_FILTER_DROP_INFO,2731 NETDEV_SVLAN_FILTER_DROP_INFO,
2732 NETDEV_OFFLOAD_XSTATS_ENABLE,
2733 NETDEV_OFFLOAD_XSTATS_DISABLE,
2734 NETDEV_OFFLOAD_XSTATS_REPORT_USED,
2735 NETDEV_OFFLOAD_XSTATS_REPORT_DELTA,
2730};2736};
2731const char *netdev_cmd_to_name(enum netdev_cmd cmd);2737const char *netdev_cmd_to_name(enum netdev_cmd cmd);
2732 2738
@@ -2777,6 +2783,42 @@ struct netdev_notifier_pre_changeaddr_info { · 44 unmodified lines
2777 const unsigned char *dev_addr;2783 const unsigned char *dev_addr;
2778};2784};
2779 2785
2786enum netdev_offload_xstats_type {
2787 NETDEV_OFFLOAD_XSTATS_TYPE_L3 = 1,
2788};
2789
2790struct netdev_notifier_offload_xstats_info {
2791 struct netdev_notifier_info info; /* must be first */
2792 enum netdev_offload_xstats_type type;
2793
2794 union {
2795 /* NETDEV_OFFLOAD_XSTATS_REPORT_DELTA */
2796 struct netdev_notifier_offload_xstats_rd *report_delta;
2797 /* NETDEV_OFFLOAD_XSTATS_REPORT_USED */
2798 struct netdev_notifier_offload_xstats_ru *report_used;
2799 };
2800};
2801
2802int netdev_offload_xstats_enable(struct net_device *dev,
2803 enum netdev_offload_xstats_type type,
2804 struct netlink_ext_ack *extack);
2805int netdev_offload_xstats_disable(struct net_device *dev,
2806 enum netdev_offload_xstats_type type);
2807bool netdev_offload_xstats_enabled(const struct net_device *dev,
2808 enum netdev_offload_xstats_type type);
2809int netdev_offload_xstats_get(struct net_device *dev,
2810 enum netdev_offload_xstats_type type,
2811 struct rtnl_hw_stats64 *stats, bool *used,
2812 struct netlink_ext_ack *extack);
2813void
2814netdev_offload_xstats_report_delta(struct netdev_notifier_offload_xstats_rd *rd,
2815 const struct rtnl_hw_stats64 *stats);
2816void
2817netdev_offload_xstats_report_used(struct netdev_notifier_offload_xstats_ru *ru);
2818void netdev_offload_xstats_push_delta(struct net_device *dev,
2819 enum netdev_offload_xstats_type type,
2820 const struct rtnl_hw_stats64 *stats);
2821
2780static inline void netdev_notifier_info_init(struct netdev_notifier_info *info,2822static inline void netdev_notifier_info_init(struct netdev_notifier_info *info,
2781 struct net_device *dev)2823 struct net_device *dev)
2782{2824{
include/uapi/linux/if_link.h+15−0
@@ -245,6 +245,21 @@ struct rtnl_link_stats64 { · 244 unmodified lines
245 __u64 rx_nohandler;245 __u64 rx_nohandler;
246};246};
247 247
248/* Subset of link stats useful for in-HW collection. Meaning of the fields is as
249 * for struct rtnl_link_stats64.
250 */
251struct rtnl_hw_stats64 {
252 __u64 rx_packets;
253 __u64 tx_packets;
254 __u64 rx_bytes;
255 __u64 tx_bytes;
256 __u64 rx_errors;
257 __u64 tx_errors;
258 __u64 rx_dropped;
259 __u64 tx_dropped;
260 __u64 multicast;
261};
262
248/* The struct should be in sync with struct ifmap */263/* The struct should be in sync with struct ifmap */
249struct rtnl_link_ifmap {264struct rtnl_link_ifmap {
250 __u64 mem_start;265 __u64 mem_start;
net/core/dev.c+266−1
@@ -1622,7 +1622,8 @@ const char *netdev_cmd_to_name(enum netdev_cmd cmd) · 1621 unmodified lines
1622 N(UDP_TUNNEL_DROP_INFO) N(CHANGE_TX_QUEUE_LEN)1622 N(UDP_TUNNEL_DROP_INFO) N(CHANGE_TX_QUEUE_LEN)
1623 N(CVLAN_FILTER_PUSH_INFO) N(CVLAN_FILTER_DROP_INFO)1623 N(CVLAN_FILTER_PUSH_INFO) N(CVLAN_FILTER_DROP_INFO)
1624 N(SVLAN_FILTER_PUSH_INFO) N(SVLAN_FILTER_DROP_INFO)1624 N(SVLAN_FILTER_PUSH_INFO) N(SVLAN_FILTER_DROP_INFO)
1625 N(PRE_CHANGEADDR)1625 N(PRE_CHANGEADDR) N(OFFLOAD_XSTATS_ENABLE) N(OFFLOAD_XSTATS_DISABLE)
1626 N(OFFLOAD_XSTATS_REPORT_USED) N(OFFLOAD_XSTATS_REPORT_DELTA)
1626 }1627 }
1627#undef N1628#undef N
1628 return "UNKNOWN_NETDEV_EVENT";1629 return "UNKNOWN_NETDEV_EVENT";
@@ -1939,6 +1940,32 @@ static int call_netdevice_notifiers_info(unsigned long val, · 310 unmodified lines
1939 return raw_notifier_call_chain(&netdev_chain, val, info);1940 return raw_notifier_call_chain(&netdev_chain, val, info);
1940}1941}
1941 1942
1943/**
1944 * call_netdevice_notifiers_info_robust - call per-netns notifier blocks
1945 * for and rollback on error
1946 * @val_up: value passed unmodified to notifier function
1947 * @val_down: value passed unmodified to the notifier function when
1948 * recovering from an error on @val_up
1949 * @info: notifier information data
1950 *
1951 * Call all per-netns network notifier blocks, but not notifier blocks on
1952 * the global notifier chain. Parameters and return value are as for
1953 * raw_notifier_call_chain_robust().
1954 */
1955
1956static int
1957call_netdevice_notifiers_info_robust(unsigned long val_up,
1958 unsigned long val_down,
1959 struct netdev_notifier_info *info)
1960{
1961 struct net *net = dev_net(info->dev);
1962
1963 ASSERT_RTNL();
1964
1965 return raw_notifier_call_chain_robust(&net->netdev_chain,
1966 val_up, val_down, info);
1967}
1968
1942static int call_netdevice_notifiers_extack(unsigned long val,1969static int call_netdevice_notifiers_extack(unsigned long val,
1943 struct net_device *dev,1970 struct net_device *dev,
1944 struct netlink_ext_ack *extack)1971 struct netlink_ext_ack *extack)
@@ -7728,6 +7755,242 @@ void netdev_bonding_info_change(struct net_device *dev, · 5783 unmodified lines
7728}7755}
7729EXPORT_SYMBOL(netdev_bonding_info_change);7756EXPORT_SYMBOL(netdev_bonding_info_change);
7730 7757
7758static int netdev_offload_xstats_enable_l3(struct net_device *dev,
7759 struct netlink_ext_ack *extack)
7760{
7761 struct netdev_notifier_offload_xstats_info info = {
7762 .info.dev = dev,
7763 .info.extack = extack,
7764 .type = NETDEV_OFFLOAD_XSTATS_TYPE_L3,
7765 };
7766 int err;
7767 int rc;
7768
7769 dev->offload_xstats_l3 = kzalloc(sizeof(*dev->offload_xstats_l3),
7770 GFP_KERNEL);
7771 if (!dev->offload_xstats_l3)
7772 return -ENOMEM;
7773
7774 rc = call_netdevice_notifiers_info_robust(NETDEV_OFFLOAD_XSTATS_ENABLE,
7775 NETDEV_OFFLOAD_XSTATS_DISABLE,
7776 &info.info);
7777 err = notifier_to_errno(rc);
7778 if (err)
7779 goto free_stats;
7780
7781 return 0;
7782
7783free_stats:
7784 kfree(dev->offload_xstats_l3);
7785 dev->offload_xstats_l3 = NULL;
7786 return err;
7787}
7788
7789int netdev_offload_xstats_enable(struct net_device *dev,
7790 enum netdev_offload_xstats_type type,
7791 struct netlink_ext_ack *extack)
7792{
7793 ASSERT_RTNL();
7794
7795 if (netdev_offload_xstats_enabled(dev, type))
7796 return -EALREADY;
7797
7798 switch (type) {
7799 case NETDEV_OFFLOAD_XSTATS_TYPE_L3:
7800 return netdev_offload_xstats_enable_l3(dev, extack);
7801 }
7802
7803 WARN_ON(1);
7804 return -EINVAL;
7805}
7806EXPORT_SYMBOL(netdev_offload_xstats_enable);
7807
7808static void netdev_offload_xstats_disable_l3(struct net_device *dev)
7809{
7810 struct netdev_notifier_offload_xstats_info info = {
7811 .info.dev = dev,
7812 .type = NETDEV_OFFLOAD_XSTATS_TYPE_L3,
7813 };
7814
7815 call_netdevice_notifiers_info(NETDEV_OFFLOAD_XSTATS_DISABLE,
7816 &info.info);
7817 kfree(dev->offload_xstats_l3);
7818 dev->offload_xstats_l3 = NULL;
7819}
7820
7821int netdev_offload_xstats_disable(struct net_device *dev,
7822 enum netdev_offload_xstats_type type)
7823{
7824 ASSERT_RTNL();
7825
7826 if (!netdev_offload_xstats_enabled(dev, type))
7827 return -EALREADY;
7828
7829 switch (type) {
7830 case NETDEV_OFFLOAD_XSTATS_TYPE_L3:
7831 netdev_offload_xstats_disable_l3(dev);
7832 return 0;
7833 }
7834
7835 WARN_ON(1);
7836 return -EINVAL;
7837}
7838EXPORT_SYMBOL(netdev_offload_xstats_disable);
7839
7840static void netdev_offload_xstats_disable_all(struct net_device *dev)
7841{
7842 netdev_offload_xstats_disable(dev, NETDEV_OFFLOAD_XSTATS_TYPE_L3);
7843}
7844
7845static struct rtnl_hw_stats64 *
7846netdev_offload_xstats_get_ptr(const struct net_device *dev,
7847 enum netdev_offload_xstats_type type)
7848{
7849 switch (type) {
7850 case NETDEV_OFFLOAD_XSTATS_TYPE_L3:
7851 return dev->offload_xstats_l3;
7852 }
7853
7854 WARN_ON(1);
7855 return NULL;
7856}
7857
7858bool netdev_offload_xstats_enabled(const struct net_device *dev,
7859 enum netdev_offload_xstats_type type)
7860{
7861 ASSERT_RTNL();
7862
7863 return netdev_offload_xstats_get_ptr(dev, type);
7864}
7865EXPORT_SYMBOL(netdev_offload_xstats_enabled);
7866
7867struct netdev_notifier_offload_xstats_ru {
7868 bool used;
7869};
7870
7871struct netdev_notifier_offload_xstats_rd {
7872 struct rtnl_hw_stats64 stats;
7873 bool used;
7874};
7875
7876static void netdev_hw_stats64_add(struct rtnl_hw_stats64 *dest,
7877 const struct rtnl_hw_stats64 *src)
7878{
7879 dest->rx_packets += src->rx_packets;
7880 dest->tx_packets += src->tx_packets;
7881 dest->rx_bytes += src->rx_bytes;
7882 dest->tx_bytes += src->tx_bytes;
7883 dest->rx_errors += src->rx_errors;
7884 dest->tx_errors += src->tx_errors;
7885 dest->rx_dropped += src->rx_dropped;
7886 dest->tx_dropped += src->tx_dropped;
7887 dest->multicast += src->multicast;
7888}
7889
7890static int netdev_offload_xstats_get_used(struct net_device *dev,
7891 enum netdev_offload_xstats_type type,
7892 bool *p_used,
7893 struct netlink_ext_ack *extack)
7894{
7895 struct netdev_notifier_offload_xstats_ru report_used = {};
7896 struct netdev_notifier_offload_xstats_info info = {
7897 .info.dev = dev,
7898 .info.extack = extack,
7899 .type = type,
7900 .report_used = &report_used,
7901 };
7902 int rc;
7903
7904 WARN_ON(!netdev_offload_xstats_enabled(dev, type));
7905 rc = call_netdevice_notifiers_info(NETDEV_OFFLOAD_XSTATS_REPORT_USED,
7906 &info.info);
7907 *p_used = report_used.used;
7908 return notifier_to_errno(rc);
7909}
7910
7911static int netdev_offload_xstats_get_stats(struct net_device *dev,
7912 enum netdev_offload_xstats_type type,
7913 struct rtnl_hw_stats64 *p_stats,
7914 bool *p_used,
7915 struct netlink_ext_ack *extack)
7916{
7917 struct netdev_notifier_offload_xstats_rd report_delta = {};
7918 struct netdev_notifier_offload_xstats_info info = {
7919 .info.dev = dev,
7920 .info.extack = extack,
7921 .type = type,
7922 .report_delta = &report_delta,
7923 };
7924 struct rtnl_hw_stats64 *stats;
7925 int rc;
7926
7927 stats = netdev_offload_xstats_get_ptr(dev, type);
7928 if (WARN_ON(!stats))
7929 return -EINVAL;
7930
7931 rc = call_netdevice_notifiers_info(NETDEV_OFFLOAD_XSTATS_REPORT_DELTA,
7932 &info.info);
7933
7934 /* Cache whatever we got, even if there was an error, otherwise the
7935 * successful stats retrievals would get lost.
7936 */
7937 netdev_hw_stats64_add(stats, &report_delta.stats);
7938
7939 if (p_stats)
7940 *p_stats = *stats;
7941 *p_used = report_delta.used;
7942
7943 return notifier_to_errno(rc);
7944}
7945
7946int netdev_offload_xstats_get(struct net_device *dev,
7947 enum netdev_offload_xstats_type type,
7948 struct rtnl_hw_stats64 *p_stats, bool *p_used,
7949 struct netlink_ext_ack *extack)
7950{
7951 ASSERT_RTNL();
7952
7953 if (p_stats)
7954 return netdev_offload_xstats_get_stats(dev, type, p_stats,
7955 p_used, extack);
7956 else
7957 return netdev_offload_xstats_get_used(dev, type, p_used,
7958 extack);
7959}
7960EXPORT_SYMBOL(netdev_offload_xstats_get);
7961
7962void
7963netdev_offload_xstats_report_delta(struct netdev_notifier_offload_xstats_rd *report_delta,
7964 const struct rtnl_hw_stats64 *stats)
7965{
7966 report_delta->used = true;
7967 netdev_hw_stats64_add(&report_delta->stats, stats);
7968}
7969EXPORT_SYMBOL(netdev_offload_xstats_report_delta);
7970
7971void
7972netdev_offload_xstats_report_used(struct netdev_notifier_offload_xstats_ru *report_used)
7973{
7974 report_used->used = true;
7975}
7976EXPORT_SYMBOL(netdev_offload_xstats_report_used);
7977
7978void netdev_offload_xstats_push_delta(struct net_device *dev,
7979 enum netdev_offload_xstats_type type,
7980 const struct rtnl_hw_stats64 *p_stats)
7981{
7982 struct rtnl_hw_stats64 *stats;
7983
7984 ASSERT_RTNL();
7985
7986 stats = netdev_offload_xstats_get_ptr(dev, type);
7987 if (WARN_ON(!stats))
7988 return;
7989
7990 netdev_hw_stats64_add(stats, p_stats);
7991}
7992EXPORT_SYMBOL(netdev_offload_xstats_push_delta);
7993
7731/**7994/**
7732 * netdev_get_xmit_slave - Get the xmit slave of master device7995 * netdev_get_xmit_slave - Get the xmit slave of master device
7733 * @dev: device7996 * @dev: device
@@ -10417,6 +10680,8 @@ void unregister_netdevice_many(struct list_head *head) · 2683 unmodified lines
10417 10680
10418 dev_xdp_uninstall(dev);10681 dev_xdp_uninstall(dev);
10419 10682
10683 netdev_offload_xstats_disable_all(dev);
10684
10420 /* Notify protocols, that we are about to destroy10685 /* Notify protocols, that we are about to destroy
10421 * this device. They should clean all the things.10686 * this device. They should clean all the things.
10422 */10687 */