.. SPDX-License-Identifier: GPL-2.0 =============Devlink DPIPE============= Background========== While performing the hardware offloading process, much of the hardwarespecifics cannot be presented. These details are useful for debugging, and``devlink-dpipe`` provides a standardized way to provide visibility into theoffloading process. For example, the routing longest prefix match (LPM) algorithm used by theLinux kernel may differ from the hardware implementation. The pipeline debugAPI (DPIPE) is aimed at providing the user visibility into the ASIC'spipeline in a generic way. The hardware offload process is expected to be done in a way that the usershould not be able to distinguish between the hardware vs. softwareimplementation. In this process, hardware specifics are neglected. Inreality those details can have lots of meaning and should be exposed in somestandard way. This problem is made even more complex when one wishes to offload thecontrol path of the whole networking stack to a switch ASIC. Due todifferences in the hardware and software models some processes cannot berepresented correctly. One example is the kernel's LPM algorithm which in many cases differsgreatly to the hardware implementation. The configuration API is the same,but one cannot rely on the Forward Information Base (FIB) to look like theLevel Path Compression trie (LPC-trie) in hardware. In many situations trying to analyze systems failure solely based on thekernel's dump may not be enough. By combining this data with complementaryinformation about the underlying hardware, this debugging can be madeeasier; additionally, the information can be useful when debuggingperformance issues. Overview======== The ``devlink-dpipe`` interface closes this gap. The hardware's pipeline ismodeled as a graph of match/action tables. Each table represents a specifichardware block. This model is not new, first being used by the P4 language. Traditionally it has been used as an alternative model for hardwareconfiguration, but the ``devlink-dpipe`` interface uses it for visibilitypurposes as a standard complementary tool. The system's view from``devlink-dpipe`` should change according to the changes done by thestandard configuration tools. For example, it’s quite common to  implement Access Control Lists (ACL)using Ternary Content Addressable Memory (TCAM). The TCAM memory can bedivided into TCAM regions. Complex TC filters can have multiple rules withdifferent priorities and different lookup keys. On the other hand hardwareTCAM regions have a predefined lookup key. Offloading the TC filter rulesusing TCAM engine can result in multiple TCAM regions being interconnectedin a chain (which may affect the data path latency). In response to a new TCfilter new tables should be created describing those regions. Model===== The ``DPIPE`` model introduces several objects:   * headers  * tables  * entries A ``header`` describes packet formats and provides names for fields withinthe packet. A ``table`` describes hardware blocks. An ``entry`` describesthe actual content of a specific table. The hardware pipeline is not port specific, but rather describes the wholeASIC. Thus it is tied to the top of the ``devlink`` infrastructure. Drivers can register and unregister tables at run time, in order to supportdynamic behavior. This dynamic behavior is mandatory for describing hardwareblocks like TCAM regions which can be allocated and freed dynamically. ``devlink-dpipe`` generally is not intended for configuration. The exceptionis hardware counting for a specific table. The following commands are used to obtain the ``dpipe`` objects fromuserspace:   * ``table_get``: Receive a table's description.  * ``headers_get``: Receive a device's supported headers.  * ``entries_get``: Receive a table's current entries.  * ``counters_set``: Enable or disable counters on a table. Table----- The driver should implement the following operations for each table:   * ``matches_dump``: Dump the supported matches.  * ``actions_dump``: Dump the supported actions.  * ``entries_dump``: Dump the actual content of the table.  * ``counters_set_update``: Synchronize hardware with counters enabled or    disabled. Header/Field------------ In a similar way to P4 headers and fields are used to describe a table'sbehavior. There is a slight difference between the standard protocol headersand specific ASIC metadata. The protocol headers should be declared in the``devlink`` core API. On the other hand ASIC meta data is driver specificand should be defined in the driver. Additionally, each driver-specificdevlink documentation file should document the driver-specific ``dpipe``headers it implements. The headers and fields are identified by enumeration. In order to provide further visibility some ASIC metadata fields could bemapped to kernel objects. For example, internal router interface indexes canbe directly mapped to the net device ifindex. FIB table indexes used bydifferent Virtual Routing and Forwarding (VRF) tables can be mapped tointernal routing table indexes. Match----- Matches are kept primitive and close to hardware operation. Match types likeLPM are not supported due to the fact that this is exactly a process we wishto describe in full detail. Example of matches:   * ``field_exact``: Exact match on a specific field.  * ``field_exact_mask``: Exact match on a specific field after masking.  * ``field_range``: Match on a specific range. The id's of the header and the field should be specified in order toidentify the specific field. Furthermore, the header index should bespecified in order to distinguish multiple headers of the same type in apacket (tunneling). Action------ Similar to match, the actions are kept primitive and close to hardwareoperation. For example:   * ``field_modify``: Modify the field value.  * ``field_inc``: Increment the field value.  * ``push_header``: Add a header.  * ``pop_header``: Remove a header. Entry----- Entries of a specific table can be dumped on demand. Each eentry isidentified with an index and its properties are described by a list ofmatch/action values and specific counter. By dumping the tables content theinteractions between tables can be resolved. Abstraction Example=================== The following is an example of the abstraction model of the L3 part ofMellanox Spectrum ASIC. The blocks are described in the order they appear inthe pipeline. The table sizes in the following examples are not realhardware sizes and are provided for demonstration purposes. LPM--- The LPM algorithm can be implemented as a list of hash tables. Each hashtable contains routes with the same prefix length. The root of the list is/32, and in case of a miss the hardware will continue to the next hashtable. The depth of the search will affect the data path latency. In case of a hit the entry contains information about the next stage of thepipeline which resolves the MAC address. The next stage can be either localhost table for directly connected routes, or adjacency table for next-hops.The ``meta.lpm_prefix`` field is used to connect two LPM tables. .. code::     table lpm_prefix_16 {      size: 4096,      counters_enabled: true,      match: { meta.vr_id: exact,               ipv4.dst_addr: exact_mask,               ipv6.dst_addr: exact_mask,               meta.lpm_prefix: exact },      action: { meta.adj_index: set,                meta.adj_group_size: set,                meta.rif_port: set,                meta.lpm_prefix: set },    } Local Host---------- In the case of local routes the LPM lookup already resolves the egressrouter interface (RIF), yet the exact MAC address is not known. The localhost table is a hash table combining the output interface id withdestination IP address as a key. The result is the MAC address. .. code::     table local_host {      size: 4096,      counters_enabled: true,      match: { meta.rif_port: exact,               ipv4.dst_addr: exact},      action: { ethernet.daddr: set }    } Adjacency--------- In case of remote routes this table does the ECMP. The LPM lookup results inECMP group size and index that serves as a global offset into this table.Concurrently a hash of the packet is generated. Based on the ECMP group sizeand the packet's hash a local offset is generated. Multiple LPM entries canpoint to the same adjacency group. .. code::     table adjacency {      size: 4096,      counters_enabled: true,      match: { meta.adj_index: exact,               meta.adj_group_size: exact,               meta.packet_hash_index: exact },      action: { ethernet.daddr: set,                meta.erif: set }    } ERIF---- In case the egress RIF and destination MAC have been resolved by previoustables this table does multiple operations like TTL decrease and MTU check.Then the decision of forward/drop is taken and the port L3 statistics areupdated based on the packet's type (broadcast, unicast, multicast). .. code::     table erif {      size: 800,      counters_enabled: true,      match: { meta.rif_port: exact,               meta.is_l3_unicast: exact,               meta.is_l3_broadcast: exact,               meta.is_l3_multicast, exact },      action: { meta.l3_drop: set,                meta.l3_forward: set }    }