.. SPDX-License-Identifier: GPL-2.0.. include:: <isonum.txt> =====================================================User Interface for Resource Control feature (resctrl)===================================================== :Copyright: |copy| 2016 Intel Corporation:Authors: - Fenghua Yu <fenghua.yu@intel.com> - Tony Luck <tony.luck@intel.com> - Vikas Shivappa <vikas.shivappa@intel.com> Intel refers to this feature as Intel Resource Director Technology(Intel(R) RDT).AMD refers to this feature as AMD Platform Quality of Service(AMD QoS). This feature is enabled by the CONFIG_X86_CPU_RESCTRL and the x86 /proc/cpuinfoflag bits: =============================================================== ================================RDT (Resource Director Technology) Allocation "rdt_a"CAT (Cache Allocation Technology) "cat_l3", "cat_l2"CDP (Code and Data Prioritization) "cdp_l3", "cdp_l2"CQM (Cache QoS Monitoring) "cqm_llc", "cqm_occup_llc"MBM (Memory Bandwidth Monitoring) "cqm_mbm_total", "cqm_mbm_local"MBA (Memory Bandwidth Allocation) "mba"SMBA (Slow Memory Bandwidth Allocation) ""BMEC (Bandwidth Monitoring Event Configuration) ""ABMC (Assignable Bandwidth Monitoring Counters) ""SDCIAE (Smart Data Cache Injection Allocation Enforcement) ""=============================================================== ================================ Historically, new features were made visible by default in /proc/cpuinfo. Thisresulted in the feature flags becoming hard to parse by humans. Adding a newflag to /proc/cpuinfo should be avoided if user space can obtain informationabout the feature from resctrl's info directory. To use the feature mount the file system:: # mount -t resctrl resctrl [-o cdp[,cdpl2][,mba_MBps][,debug]] /sys/fs/resctrl mount options are: "cdp": Enable code/data prioritization in L3 cache allocations."cdpl2": Enable code/data prioritization in L2 cache allocations."mba_MBps": Enable the MBA Software Controller(mba_sc) to specify MBA bandwidth in MiBps"debug": Make debug files accessible. Available debug files are annotated with "Available only with debug option". L2 and L3 CDP are controlled separately. RDT features are orthogonal. A particular system may support onlymonitoring, only control, or both monitoring and control. Cachepseudo-locking is a unique way of using cache control to "pin" or"lock" data in the cache. Details can be found in"Cache Pseudo-Locking". The mount succeeds if either of allocation or monitoring is present, butonly those files and directories supported by the system will be created.For more details on the behavior of the interface during monitoringand allocation, see the "Resource alloc and monitor groups" section. Info directory============== The 'info' directory contains information about the enabledresources. Each resource has its own subdirectory. The subdirectorynames reflect the resource names. Most of the files in the resource's subdirectory are read-only, anddescribe properties of the resource. Resources that support globalconfiguration options also include writable files that can be usedto modify those settings. Each subdirectory contains the following files with respect toallocation: Cache resource(L3/L2) subdirectory contains the following filesrelated to allocation: "num_closids": The number of CLOSIDs which are valid for this resource. The kernel uses the smallest number of CLOSIDs of all enabled resources as limit."cbm_mask": The bitmask which is valid for this resource. This mask is equivalent to 100%."min_cbm_bits": The minimum number of consecutive bits which must be set when writing a mask. "shareable_bits": Bitmask of shareable resource with other executing entities (e.g. I/O). Applies to all instances of this resource. User can use this when setting up exclusive cache partitions. Note that some platforms support devices that have their own settings for cache use which can over-ride these bits. When "io_alloc" is enabled, a portion of each cache instance can be configured for shared use between hardware and software. "bit_usage" should be used to see which portions of each cache instance is configured for hardware use via "io_alloc" feature because every cache instance can have its "io_alloc" bitmask configured independently via "io_alloc_cbm". "bit_usage": Annotated capacity bitmasks showing how all instances of the resource are used. The legend is: "0": Corresponding region is unused. When the system's resources have been allocated and a "0" is found in "bit_usage" it is a sign that resources are wasted. "H": Corresponding region is used by hardware only but available for software use. If a resource has bits set in "shareable_bits" or "io_alloc_cbm" but not all of these bits appear in the resource groups' schemata then the bits appearing in "shareable_bits" or "io_alloc_cbm" but no resource group will be marked as "H". "X": Corresponding region is available for sharing and used by hardware and software. These are the bits that appear in "shareable_bits" or "io_alloc_cbm" as well as a resource group's allocation. "S": Corresponding region is used by software and available for sharing. "E": Corresponding region is used exclusively by one resource group. No sharing allowed. "P": Corresponding region is pseudo-locked. No sharing allowed."sparse_masks": Indicates if non-contiguous 1s value in CBM is supported. "0": Only contiguous 1s value in CBM is supported. "1": Non-contiguous 1s value in CBM is supported. "io_alloc": "io_alloc" enables system software to configure the portion of the cache allocated for I/O traffic. File may only exist if the system supports this feature on some of its cache resources. "disabled": Resource supports "io_alloc" but the feature is disabled. Portions of cache used for allocation of I/O traffic cannot be configured. "enabled": Portions of cache used for allocation of I/O traffic can be configured using "io_alloc_cbm". "not supported": Support not available for this resource. The feature can be modified by writing to the interface, for example: To enable:: # echo 1 > /sys/fs/resctrl/info/L3/io_alloc To disable:: # echo 0 > /sys/fs/resctrl/info/L3/io_alloc The underlying implementation may reduce resources available to general (CPU) cache allocation. See architecture specific notes below. Depending on usage requirements the feature can be enabled or disabled. On AMD systems, io_alloc feature is supported by the L3 Smart Data Cache Injection Allocation Enforcement (SDCIAE). The CLOSID for io_alloc is the highest CLOSID supported by the resource. When io_alloc is enabled, the highest CLOSID is dedicated to io_alloc and no longer available for general (CPU) cache allocation. When CDP is enabled, io_alloc routes I/O traffic using the highest CLOSID allocated for the instruction cache (CDP_CODE), making this CLOSID no longer available for general (CPU) cache allocation for both the CDP_CODE and CDP_DATA resources. "io_alloc_cbm": Capacity bitmasks that describe the portions of cache instances to which I/O traffic from supported I/O devices are routed when "io_alloc" is enabled. CBMs are displayed in the following format: <cache_id0>=<cbm>;<cache_id1>=<cbm>;... Example:: # cat /sys/fs/resctrl/info/L3/io_alloc_cbm 0=ffff;1=ffff CBMs can be configured by writing to the interface. Example:: # echo 1=ff > /sys/fs/resctrl/info/L3/io_alloc_cbm # cat /sys/fs/resctrl/info/L3/io_alloc_cbm 0=ffff;1=00ff # echo "0=ff;1=f" > /sys/fs/resctrl/info/L3/io_alloc_cbm # cat /sys/fs/resctrl/info/L3/io_alloc_cbm 0=00ff;1=000f An ID of "*" configures all domains with the provided CBM. Example on a system that does not require a minimum number of consecutive bits in the mask:: # echo "*=0" > /sys/fs/resctrl/info/L3/io_alloc_cbm # cat /sys/fs/resctrl/info/L3/io_alloc_cbm 0=0;1=0 When CDP is enabled "io_alloc_cbm" associated with the CDP_DATA and CDP_CODE resources may reflect the same values. For example, values read from and written to /sys/fs/resctrl/info/L3DATA/io_alloc_cbm may be reflected by /sys/fs/resctrl/info/L3CODE/io_alloc_cbm and vice versa. Memory bandwidth(MB) subdirectory contains the following fileswith respect to allocation: "min_bandwidth": The minimum memory bandwidth percentage which user can request. "bandwidth_gran": The granularity in which the memory bandwidth percentage is allocated. The allocated b/w percentage is rounded off to the next control step available on the hardware. The available bandwidth control steps are: min_bandwidth + N * bandwidth_gran. "delay_linear": Indicates if the delay scale is linear or non-linear. This field is purely informational only. "thread_throttle_mode": Indicator on Intel systems of how tasks running on threads of a physical core are throttled in cases where they request different memory bandwidth percentages: "max": the smallest percentage is applied to all threads "per-thread": bandwidth percentages are directly applied to the threads running on the core If L3 monitoring is available there will be an "L3_MON" directorywith the following files: "num_rmids": The number of RMIDs supported by hardware for L3 monitoring events. "mon_features": Lists the monitoring events if monitoring is enabled for the resource. Example:: # cat /sys/fs/resctrl/info/L3_MON/mon_features llc_occupancy mbm_total_bytes mbm_local_bytes If the system supports Bandwidth Monitoring Event Configuration (BMEC), then the bandwidth events will be configurable. The output will be:: # cat /sys/fs/resctrl/info/L3_MON/mon_features llc_occupancy mbm_total_bytes mbm_total_bytes_config mbm_local_bytes mbm_local_bytes_config "mbm_total_bytes_config", "mbm_local_bytes_config": Read/write files containing the configuration for the mbm_total_bytes and mbm_local_bytes events, respectively, when the Bandwidth Monitoring Event Configuration (BMEC) feature is supported. The event configuration settings are domain specific and affect all the CPUs in the domain. When either event configuration is changed, the bandwidth counters for all RMIDs of both events (mbm_total_bytes as well as mbm_local_bytes) are cleared for that domain. The next read for every RMID will report "Unavailable" and subsequent reads will report the valid value. Following are the types of events supported: ==== ======================================================== Bits Description ==== ======================================================== 6 Dirty Victims from the QOS domain to all types of memory 5 Reads to slow memory in the non-local NUMA domain 4 Reads to slow memory in the local NUMA domain 3 Non-temporal writes to non-local NUMA domain 2 Non-temporal writes to local NUMA domain 1 Reads to memory in the non-local NUMA domain 0 Reads to memory in the local NUMA domain ==== ======================================================== By default, the mbm_total_bytes configuration is set to 0x7f to count all the event types and the mbm_local_bytes configuration is set to 0x15 to count all the local memory events. Examples: * To view the current configuration:: :: # cat /sys/fs/resctrl/info/L3_MON/mbm_total_bytes_config 0=0x7f;1=0x7f;2=0x7f;3=0x7f # cat /sys/fs/resctrl/info/L3_MON/mbm_local_bytes_config 0=0x15;1=0x15;3=0x15;4=0x15 * To change the mbm_total_bytes to count only reads on domain 0, the bits 0, 1, 4 and 5 needs to be set, which is 110011b in binary (in hexadecimal 0x33): :: # echo "0=0x33" > /sys/fs/resctrl/info/L3_MON/mbm_total_bytes_config # cat /sys/fs/resctrl/info/L3_MON/mbm_total_bytes_config 0=0x33;1=0x7f;2=0x7f;3=0x7f * To change the mbm_local_bytes to count all the slow memory reads on domain 0 and 1, the bits 4 and 5 needs to be set, which is 110000b in binary (in hexadecimal 0x30): :: # echo "0=0x30;1=0x30" > /sys/fs/resctrl/info/L3_MON/mbm_local_bytes_config # cat /sys/fs/resctrl/info/L3_MON/mbm_local_bytes_config 0=0x30;1=0x30;3=0x15;4=0x15 "mbm_assign_mode": The supported counter assignment modes. The enclosed brackets indicate which mode is enabled. The MBM events associated with counters may reset when "mbm_assign_mode" is changed. :: # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode [mbm_event] default "mbm_event": mbm_event mode allows users to assign a hardware counter to an RMID, event pair and monitor the bandwidth usage as long as it is assigned. The hardware continues to track the assigned counter until it is explicitly unassigned by the user. Each event within a resctrl group can be assigned independently. In this mode, a monitoring event can only accumulate data while it is backed by a hardware counter. Use "mbm_L3_assignments" found in each CTRL_MON and MON group to specify which of the events should have a counter assigned. The number of counters available is described in the "num_mbm_cntrs" file. Changing the mode may cause all counters on the resource to reset. Moving to mbm_event counter assignment mode requires users to assign the counters to the events. Otherwise, the MBM event counters will return 'Unassigned' when read. The mode is beneficial for AMD platforms that support more CTRL_MON and MON groups than available hardware counters. By default, this feature is enabled on AMD platforms with the ABMC (Assignable Bandwidth Monitoring Counters) capability, ensuring counters remain assigned even when the corresponding RMID is not actively used by any processor. "default": In default mode, resctrl assumes there is a hardware counter for each event within every CTRL_MON and MON group. On AMD platforms, it is recommended to use the mbm_event mode, if supported, to prevent reset of MBM events between reads resulting from hardware re-allocating counters. This can result in misleading values or display "Unavailable" if no counter is assigned to the event. * To enable "mbm_event" counter assignment mode: :: # echo "mbm_event" > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode * To enable "default" monitoring mode: :: # echo "default" > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode "num_mbm_cntrs": The maximum number of counters (total of available and assigned counters) in each domain when the system supports mbm_event mode. For example, on a system with maximum of 32 memory bandwidth monitoring counters in each of its L3 domains: :: # cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs 0=32;1=32 "available_mbm_cntrs": The number of counters available for assignment in each domain when mbm_event mode is enabled on the system. For example, on a system with 30 available [hardware] assignable counters in each of its L3 domains: :: # cat /sys/fs/resctrl/info/L3_MON/available_mbm_cntrs 0=30;1=30 "event_configs": Directory that exists when "mbm_event" counter assignment mode is supported. Contains a sub-directory for each MBM event that can be assigned to a counter. Two MBM events are supported by default: mbm_local_bytes and mbm_total_bytes. Each MBM event's sub-directory contains a file named "event_filter" that is used to view and (if writable) modify which memory transactions the MBM event is configured with. The file is accessible only when "mbm_event" counter assignment mode is enabled. List of memory transaction types supported: ========================== ======================================================== Name Description ========================== ======================================================== dirty_victim_writes_all Dirty Victims from the QOS domain to all types of memory remote_reads_slow_memory Reads to slow memory in the non-local NUMA domain local_reads_slow_memory Reads to slow memory in the local NUMA domain remote_non_temporal_writes Non-temporal writes to non-local NUMA domain local_non_temporal_writes Non-temporal writes to local NUMA domain remote_reads Reads to memory in the non-local NUMA domain local_reads Reads to memory in the local NUMA domain ========================== ======================================================== For example:: # cat /sys/fs/resctrl/info/L3_MON/event_configs/mbm_total_bytes/event_filter local_reads,remote_reads,local_non_temporal_writes,remote_non_temporal_writes, local_reads_slow_memory,remote_reads_slow_memory,dirty_victim_writes_all # cat /sys/fs/resctrl/info/L3_MON/event_configs/mbm_local_bytes/event_filter local_reads,local_non_temporal_writes,local_reads_slow_memory The memory transactions the MBM event is configured with can be changed if "event_filter" is writable. For example:: # echo "local_reads, local_non_temporal_writes" > /sys/fs/resctrl/info/L3_MON/event_configs/mbm_total_bytes/event_filter # cat /sys/fs/resctrl/info/L3_MON/event_configs/mbm_total_bytes/event_filter local_reads,local_non_temporal_writes "mbm_assign_on_mkdir": Exists when "mbm_event" counter assignment mode is supported. Accessible only when "mbm_event" counter assignment mode is enabled. Determines if a counter will automatically be assigned to an RMID, MBM event pair when its associated monitor group is created via mkdir. Enabled by default on boot, also when switched from "default" mode to "mbm_event" counter assignment mode. Users can disable this capability by writing to the interface. "0": Auto assignment is disabled. "1": Auto assignment is enabled. Automatic counter assignment is done with best effort. If auto assignment is enabled but there are not enough available counters then monitor group creation could succeed while one or more events belonging to the group may not have a counter assigned in all domains. Consult mbm_L3_assignments for counter assignment states of the new groups. Example:: # echo 0 > /sys/fs/resctrl/info/L3_MON/mbm_assign_on_mkdir # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_on_mkdir 0 "max_threshold_occupancy": Read/write file provides the largest value (in bytes) at which a previously used LLC_occupancy counter can be considered for reuse. If telemetry monitoring is available there will be a "PERF_PKG_MON" directorywith the following files:Lines 1–500 of 1992. Load more
.. SPDX-License-Identifier: GPL-2.0 .. include:: <isonum.txt> ===================================================== User Interface for Resource Control feature (resctrl) ===================================================== :Copyright: |copy| 2016 Intel Corporation :Authors: - Fenghua Yu <fenghua.yu@intel.com> - Tony Luck <tony.luck@intel.com> - Vikas Shivappa <vikas.shivappa@intel.com> Intel refers to this feature as Intel Resource Director Technology(Intel(R) RDT). AMD refers to this feature as AMD Platform Quality of Service(AMD QoS). This feature is enabled by the CONFIG_X86_CPU_RESCTRL and the x86 /proc/cpuinfo flag bits: =============================================================== ================================ RDT (Resource Director Technology) Allocation "rdt_a" CAT (Cache Allocation Technology) "cat_l3", "cat_l2" CDP (Code and Data Prioritization) "cdp_l3", "cdp_l2" CQM (Cache QoS Monitoring) "cqm_llc", "cqm_occup_llc" MBM (Memory Bandwidth Monitoring) "cqm_mbm_total", "cqm_mbm_local" MBA (Memory Bandwidth Allocation) "mba" SMBA (Slow Memory Bandwidth Allocation) "" BMEC (Bandwidth Monitoring Event Configuration) "" ABMC (Assignable Bandwidth Monitoring Counters) "" SDCIAE (Smart Data Cache Injection Allocation Enforcement) "" =============================================================== ================================ Historically, new features were made visible by default in /proc/cpuinfo. This resulted in the feature flags becoming hard to parse by humans. Adding a new flag to /proc/cpuinfo should be avoided if user space can obtain information about the feature from resctrl's info directory. To use the feature mount the file system:: # mount -t resctrl resctrl [-o cdp[,cdpl2][,mba_MBps][,debug]] /sys/fs/resctrl mount options are: "cdp": Enable code/data prioritization in L3 cache allocations. "cdpl2": Enable code/data prioritization in L2 cache allocations. "mba_MBps": Enable the MBA Software Controller(mba_sc) to specify MBA bandwidth in MiBps "debug": Make debug files accessible. Available debug files are annotated with "Available only with debug option". L2 and L3 CDP are controlled separately. RDT features are orthogonal. A particular system may support only monitoring, only control, or both monitoring and control. Cache pseudo-locking is a unique way of using cache control to "pin" or "lock" data in the cache. Details can be found in "Cache Pseudo-Locking". The mount succeeds if either of allocation or monitoring is present, but only those files and directories supported by the system will be created. For more details on the behavior of the interface during monitoring and allocation, see the "Resource alloc and monitor groups" section. Info directory ============== The 'info' directory contains information about the enabled resources. Each resource has its own subdirectory. The subdirectory names reflect the resource names. Most of the files in the resource's subdirectory are read-only, and describe properties of the resource. Resources that support global configuration options also include writable files that can be used to modify those settings. Each subdirectory contains the following files with respect to allocation: Cache resource(L3/L2) subdirectory contains the following files related to allocation: "num_closids": The number of CLOSIDs which are valid for this resource. The kernel uses the smallest number of CLOSIDs of all enabled resources as limit. "cbm_mask": The bitmask which is valid for this resource. This mask is equivalent to 100%. "min_cbm_bits": The minimum number of consecutive bits which must be set when writing a mask. "shareable_bits": Bitmask of shareable resource with other executing entities (e.g. I/O). Applies to all instances of this resource. User can use this when setting up exclusive cache partitions. Note that some platforms support devices that have their own settings for cache use which can over-ride these bits. When "io_alloc" is enabled, a portion of each cache instance can be configured for shared use between hardware and software. "bit_usage" should be used to see which portions of each cache instance is configured for hardware use via "io_alloc" feature because every cache instance can have its "io_alloc" bitmask configured independently via "io_alloc_cbm". "bit_usage": Annotated capacity bitmasks showing how all instances of the resource are used. The legend is: "0": Corresponding region is unused. When the system's resources have been allocated and a "0" is found in "bit_usage" it is a sign that resources are wasted. "H": Corresponding region is used by hardware only but available for software use. If a resource has bits set in "shareable_bits" or "io_alloc_cbm" but not all of these bits appear in the resource groups' schemata then the bits appearing in "shareable_bits" or "io_alloc_cbm" but no resource group will be marked as "H". "X": Corresponding region is available for sharing and used by hardware and software. These are the bits that appear in "shareable_bits" or "io_alloc_cbm" as well as a resource group's allocation. "S": Corresponding region is used by software and available for sharing. "E": Corresponding region is used exclusively by one resource group. No sharing allowed. "P": Corresponding region is pseudo-locked. No sharing allowed. "sparse_masks": Indicates if non-contiguous 1s value in CBM is supported. "0": Only contiguous 1s value in CBM is supported. "1": Non-contiguous 1s value in CBM is supported. "io_alloc": "io_alloc" enables system software to configure the portion of the cache allocated for I/O traffic. File may only exist if the system supports this feature on some of its cache resources. "disabled": Resource supports "io_alloc" but the feature is disabled. Portions of cache used for allocation of I/O traffic cannot be configured. "enabled": Portions of cache used for allocation of I/O traffic can be configured using "io_alloc_cbm". "not supported": Support not available for this resource. The feature can be modified by writing to the interface, for example: To enable:: # echo 1 > /sys/fs/resctrl/info/L3/io_alloc To disable:: # echo 0 > /sys/fs/resctrl/info/L3/io_alloc The underlying implementation may reduce resources available to general (CPU) cache allocation. See architecture specific notes below. Depending on usage requirements the feature can be enabled or disabled. On AMD systems, io_alloc feature is supported by the L3 Smart Data Cache Injection Allocation Enforcement (SDCIAE). The CLOSID for io_alloc is the highest CLOSID supported by the resource. When io_alloc is enabled, the highest CLOSID is dedicated to io_alloc and no longer available for general (CPU) cache allocation. When CDP is enabled, io_alloc routes I/O traffic using the highest CLOSID allocated for the instruction cache (CDP_CODE), making this CLOSID no longer available for general (CPU) cache allocation for both the CDP_CODE and CDP_DATA resources. "io_alloc_cbm": Capacity bitmasks that describe the portions of cache instances to which I/O traffic from supported I/O devices are routed when "io_alloc" is enabled. CBMs are displayed in the following format: <cache_id0>=<cbm>;<cache_id1>=<cbm>;... Example:: # cat /sys/fs/resctrl/info/L3/io_alloc_cbm 0=ffff;1=ffff CBMs can be configured by writing to the interface. Example:: # echo 1=ff > /sys/fs/resctrl/info/L3/io_alloc_cbm # cat /sys/fs/resctrl/info/L3/io_alloc_cbm 0=ffff;1=00ff # echo "0=ff;1=f" > /sys/fs/resctrl/info/L3/io_alloc_cbm # cat /sys/fs/resctrl/info/L3/io_alloc_cbm 0=00ff;1=000f An ID of "*" configures all domains with the provided CBM. Example on a system that does not require a minimum number of consecutive bits in the mask:: # echo "*=0" > /sys/fs/resctrl/info/L3/io_alloc_cbm # cat /sys/fs/resctrl/info/L3/io_alloc_cbm 0=0;1=0 When CDP is enabled "io_alloc_cbm" associated with the CDP_DATA and CDP_CODE resources may reflect the same values. For example, values read from and written to /sys/fs/resctrl/info/L3DATA/io_alloc_cbm may be reflected by /sys/fs/resctrl/info/L3CODE/io_alloc_cbm and vice versa. Memory bandwidth(MB) subdirectory contains the following files with respect to allocation: "min_bandwidth": The minimum memory bandwidth percentage which user can request. "bandwidth_gran": The granularity in which the memory bandwidth percentage is allocated. The allocated b/w percentage is rounded off to the next control step available on the hardware. The available bandwidth control steps are: min_bandwidth + N * bandwidth_gran. "delay_linear": Indicates if the delay scale is linear or non-linear. This field is purely informational only. "thread_throttle_mode": Indicator on Intel systems of how tasks running on threads of a physical core are throttled in cases where they request different memory bandwidth percentages: "max": the smallest percentage is applied to all threads "per-thread": bandwidth percentages are directly applied to the threads running on the core If L3 monitoring is available there will be an "L3_MON" directory with the following files: "num_rmids": The number of RMIDs supported by hardware for L3 monitoring events. "mon_features": Lists the monitoring events if monitoring is enabled for the resource. Example:: # cat /sys/fs/resctrl/info/L3_MON/mon_features llc_occupancy mbm_total_bytes mbm_local_bytes If the system supports Bandwidth Monitoring Event Configuration (BMEC), then the bandwidth events will be configurable. The output will be:: # cat /sys/fs/resctrl/info/L3_MON/mon_features llc_occupancy mbm_total_bytes mbm_total_bytes_config mbm_local_bytes mbm_local_bytes_config "mbm_total_bytes_config", "mbm_local_bytes_config": Read/write files containing the configuration for the mbm_total_bytes and mbm_local_bytes events, respectively, when the Bandwidth Monitoring Event Configuration (BMEC) feature is supported. The event configuration settings are domain specific and affect all the CPUs in the domain. When either event configuration is changed, the bandwidth counters for all RMIDs of both events (mbm_total_bytes as well as mbm_local_bytes) are cleared for that domain. The next read for every RMID will report "Unavailable" and subsequent reads will report the valid value. Following are the types of events supported: ==== ======================================================== Bits Description ==== ======================================================== 6 Dirty Victims from the QOS domain to all types of memory 5 Reads to slow memory in the non-local NUMA domain 4 Reads to slow memory in the local NUMA domain 3 Non-temporal writes to non-local NUMA domain 2 Non-temporal writes to local NUMA domain 1 Reads to memory in the non-local NUMA domain 0 Reads to memory in the local NUMA domain ==== ======================================================== By default, the mbm_total_bytes configuration is set to 0x7f to count all the event types and the mbm_local_bytes configuration is set to 0x15 to count all the local memory events. Examples: * To view the current configuration:: :: # cat /sys/fs/resctrl/info/L3_MON/mbm_total_bytes_config 0=0x7f;1=0x7f;2=0x7f;3=0x7f # cat /sys/fs/resctrl/info/L3_MON/mbm_local_bytes_config 0=0x15;1=0x15;3=0x15;4=0x15 * To change the mbm_total_bytes to count only reads on domain 0, the bits 0, 1, 4 and 5 needs to be set, which is 110011b in binary (in hexadecimal 0x33): :: # echo "0=0x33" > /sys/fs/resctrl/info/L3_MON/mbm_total_bytes_config # cat /sys/fs/resctrl/info/L3_MON/mbm_total_bytes_config 0=0x33;1=0x7f;2=0x7f;3=0x7f * To change the mbm_local_bytes to count all the slow memory reads on domain 0 and 1, the bits 4 and 5 needs to be set, which is 110000b in binary (in hexadecimal 0x30): :: # echo "0=0x30;1=0x30" > /sys/fs/resctrl/info/L3_MON/mbm_local_bytes_config # cat /sys/fs/resctrl/info/L3_MON/mbm_local_bytes_config 0=0x30;1=0x30;3=0x15;4=0x15 "mbm_assign_mode": The supported counter assignment modes. The enclosed brackets indicate which mode is enabled. The MBM events associated with counters may reset when "mbm_assign_mode" is changed. :: # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode [mbm_event] default "mbm_event": mbm_event mode allows users to assign a hardware counter to an RMID, event pair and monitor the bandwidth usage as long as it is assigned. The hardware continues to track the assigned counter until it is explicitly unassigned by the user. Each event within a resctrl group can be assigned independently. In this mode, a monitoring event can only accumulate data while it is backed by a hardware counter. Use "mbm_L3_assignments" found in each CTRL_MON and MON group to specify which of the events should have a counter assigned. The number of counters available is described in the "num_mbm_cntrs" file. Changing the mode may cause all counters on the resource to reset. Moving to mbm_event counter assignment mode requires users to assign the counters to the events. Otherwise, the MBM event counters will return 'Unassigned' when read. The mode is beneficial for AMD platforms that support more CTRL_MON and MON groups than available hardware counters. By default, this feature is enabled on AMD platforms with the ABMC (Assignable Bandwidth Monitoring Counters) capability, ensuring counters remain assigned even when the corresponding RMID is not actively used by any processor. "default": In default mode, resctrl assumes there is a hardware counter for each event within every CTRL_MON and MON group. On AMD platforms, it is recommended to use the mbm_event mode, if supported, to prevent reset of MBM events between reads resulting from hardware re-allocating counters. This can result in misleading values or display "Unavailable" if no counter is assigned to the event. * To enable "mbm_event" counter assignment mode: :: # echo "mbm_event" > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode * To enable "default" monitoring mode: :: # echo "default" > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode "num_mbm_cntrs": The maximum number of counters (total of available and assigned counters) in each domain when the system supports mbm_event mode. For example, on a system with maximum of 32 memory bandwidth monitoring counters in each of its L3 domains: :: # cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs 0=32;1=32 "available_mbm_cntrs": The number of counters available for assignment in each domain when mbm_event mode is enabled on the system. For example, on a system with 30 available [hardware] assignable counters in each of its L3 domains: :: # cat /sys/fs/resctrl/info/L3_MON/available_mbm_cntrs 0=30;1=30 "event_configs": Directory that exists when "mbm_event" counter assignment mode is supported. Contains a sub-directory for each MBM event that can be assigned to a counter. Two MBM events are supported by default: mbm_local_bytes and mbm_total_bytes. Each MBM event's sub-directory contains a file named "event_filter" that is used to view and (if writable) modify which memory transactions the MBM event is configured with. The file is accessible only when "mbm_event" counter assignment mode is enabled. List of memory transaction types supported: ========================== ======================================================== Name Description ========================== ======================================================== dirty_victim_writes_all Dirty Victims from the QOS domain to all types of memory remote_reads_slow_memory Reads to slow memory in the non-local NUMA domain local_reads_slow_memory Reads to slow memory in the local NUMA domain remote_non_temporal_writes Non-temporal writes to non-local NUMA domain local_non_temporal_writes Non-temporal writes to local NUMA domain remote_reads Reads to memory in the non-local NUMA domain local_reads Reads to memory in the local NUMA domain ========================== ======================================================== For example:: # cat /sys/fs/resctrl/info/L3_MON/event_configs/mbm_total_bytes/event_filter local_reads,remote_reads,local_non_temporal_writes,remote_non_temporal_writes, local_reads_slow_memory,remote_reads_slow_memory,dirty_victim_writes_all # cat /sys/fs/resctrl/info/L3_MON/event_configs/mbm_local_bytes/event_filter local_reads,local_non_temporal_writes,local_reads_slow_memory The memory transactions the MBM event is configured with can be changed if "event_filter" is writable. For example:: # echo "local_reads, local_non_temporal_writes" > /sys/fs/resctrl/info/L3_MON/event_configs/mbm_total_bytes/event_filter # cat /sys/fs/resctrl/info/L3_MON/event_configs/mbm_total_bytes/event_filter local_reads,local_non_temporal_writes "mbm_assign_on_mkdir": Exists when "mbm_event" counter assignment mode is supported. Accessible only when "mbm_event" counter assignment mode is enabled. Determines if a counter will automatically be assigned to an RMID, MBM event pair when its associated monitor group is created via mkdir. Enabled by default on boot, also when switched from "default" mode to "mbm_event" counter assignment mode. Users can disable this capability by writing to the interface. "0": Auto assignment is disabled. "1": Auto assignment is enabled. Automatic counter assignment is done with best effort. If auto assignment is enabled but there are not enough available counters then monitor group creation could succeed while one or more events belonging to the group may not have a counter assigned in all domains. Consult mbm_L3_assignments for counter assignment states of the new groups. Example:: # echo 0 > /sys/fs/resctrl/info/L3_MON/mbm_assign_on_mkdir # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_on_mkdir 0 "max_threshold_occupancy": Read/write file provides the largest value (in bytes) at which a previously used LLC_occupancy counter can be considered for reuse. If telemetry monitoring is available there will be a "PERF_PKG_MON" directory with the following files: