| 1 | |
| 2 | .. include:: <isonum.txt>
|
| 3 |
|
| 4 | =====================================================
|
| 5 | User Interface for Resource Control feature (resctrl)
|
| 6 | =====================================================
|
| 7 |
|
| 8 | :Copyright: |copy| 2016 Intel Corporation
|
| 9 | :Authors: - Fenghua Yu <[email protected]>
|
| 10 | - Tony Luck <[email protected]>
|
| 11 | - Vikas Shivappa <[email protected]>
|
| 12 |
|
| 13 |
|
| 14 | Intel refers to this feature as Intel Resource Director Technology(Intel(R) RDT).
|
| 15 | AMD refers to this feature as AMD Platform Quality of Service(AMD QoS).
|
| 16 |
|
| 17 | This feature is enabled by the CONFIG_X86_CPU_RESCTRL and the x86 /proc/cpuinfo
|
| 18 | flag bits:
|
| 19 |
|
| 20 | =============================================================== ================================
|
| 21 | RDT (Resource Director Technology) Allocation "rdt_a"
|
| 22 | CAT (Cache Allocation Technology) "cat_l3", "cat_l2"
|
| 23 | CDP (Code and Data Prioritization) "cdp_l3", "cdp_l2"
|
| 24 | CQM (Cache QoS Monitoring) "cqm_llc", "cqm_occup_llc"
|
| 25 | MBM (Memory Bandwidth Monitoring) "cqm_mbm_total", "cqm_mbm_local"
|
| 26 | MBA (Memory Bandwidth Allocation) "mba"
|
| 27 | SMBA (Slow Memory Bandwidth Allocation) ""
|
| 28 | BMEC (Bandwidth Monitoring Event Configuration) ""
|
| 29 | ABMC (Assignable Bandwidth Monitoring Counters) ""
|
| 30 | SDCIAE (Smart Data Cache Injection Allocation Enforcement) ""
|
| 31 | =============================================================== ================================
|
| 32 |
|
| 33 | Historically, new features were made visible by default in /proc/cpuinfo. This
|
| 34 | resulted in the feature flags becoming hard to parse by humans. Adding a new
|
| 35 | flag to /proc/cpuinfo should be avoided if user space can obtain information
|
| 36 | about the feature from resctrl's info directory.
|
| 37 |
|
| 38 | To use the feature mount the file system::
|
| 39 |
|
| 40 | # mount -t resctrl resctrl [-o cdp[,cdpl2][,mba_MBps][,debug]] /sys/fs/resctrl
|
| 41 |
|
| 42 | mount options are:
|
| 43 |
|
| 44 | "cdp":
|
| 45 | Enable code/data prioritization in L3 cache allocations.
|
| 46 | "cdpl2":
|
| 47 | Enable code/data prioritization in L2 cache allocations.
|
| 48 | "mba_MBps":
|
| 49 | Enable the MBA Software Controller(mba_sc) to specify MBA
|
| 50 | bandwidth in MiBps
|
| 51 | "debug":
|
| 52 | Make debug files accessible. Available debug files are annotated with
|
| 53 | "Available only with debug option".
|
| 54 |
|
| 55 | L2 and L3 CDP are controlled separately.
|
| 56 |
|
| 57 | RDT features are orthogonal. A particular system may support only
|
| 58 | monitoring, only control, or both monitoring and control. Cache
|
| 59 | pseudo-locking is a unique way of using cache control to "pin" or
|
| 60 | "lock" data in the cache. Details can be found in
|
| 61 | "Cache Pseudo-Locking".
|
| 62 |
|
| 63 |
|
| 64 | The mount succeeds if either of allocation or monitoring is present, but
|
| 65 | only those files and directories supported by the system will be created.
|
| 66 | For more details on the behavior of the interface during monitoring
|
| 67 | and allocation, see the "Resource alloc and monitor groups" section.
|
| 68 |
|
| 69 | Info directory
|
| 70 | ==============
|
| 71 |
|
| 72 | The 'info' directory contains information about the enabled
|
| 73 | resources. Each resource has its own subdirectory. The subdirectory
|
| 74 | names reflect the resource names.
|
| 75 |
|
| 76 | Most of the files in the resource's subdirectory are read-only, and
|
| 77 | describe properties of the resource. Resources that support global
|
| 78 | configuration options also include writable files that can be used
|
| 79 | to modify those settings.
|
| 80 |
|
| 81 | Each subdirectory contains the following files with respect to
|
| 82 | allocation:
|
| 83 |
|
| 84 | Cache resource(L3/L2) subdirectory contains the following files
|
| 85 | related to allocation:
|
| 86 |
|
| 87 | "num_closids":
|
| 88 | The number of CLOSIDs which are valid for this
|
| 89 | resource. The kernel uses the smallest number of
|
| 90 | CLOSIDs of all enabled resources as limit.
|
| 91 | "cbm_mask":
|
| 92 | The bitmask which is valid for this resource.
|
| 93 | This mask is equivalent to 100%.
|
| 94 | "min_cbm_bits":
|
| 95 | The minimum number of consecutive bits which
|
| 96 | must be set when writing a mask.
|
| 97 |
|
| 98 | "shareable_bits":
|
| 99 | Bitmask of shareable resource with other executing entities
|
| 100 | (e.g. I/O). Applies to all instances of this resource. User
|
| 101 | can use this when setting up exclusive cache partitions.
|
| 102 | Note that some platforms support devices that have their
|
| 103 | own settings for cache use which can over-ride these bits.
|
| 104 |
|
| 105 | When "io_alloc" is enabled, a portion of each cache instance can
|
| 106 | be configured for shared use between hardware and software.
|
| 107 | "bit_usage" should be used to see which portions of each cache
|
| 108 | instance is configured for hardware use via "io_alloc" feature
|
| 109 | because every cache instance can have its "io_alloc" bitmask
|
| 110 | configured independently via "io_alloc_cbm".
|
| 111 |
|
| 112 | "bit_usage":
|
| 113 | Annotated capacity bitmasks showing how all
|
| 114 | instances of the resource are used. The legend is:
|
| 115 |
|
| 116 | "0":
|
| 117 | Corresponding region is unused. When the system's
|
| 118 | resources have been allocated and a "0" is found
|
| 119 | in "bit_usage" it is a sign that resources are
|
| 120 | wasted.
|
| 121 |
|
| 122 | "H":
|
| 123 | Corresponding region is used by hardware only
|
| 124 | but available for software use. If a resource
|
| 125 | has bits set in "shareable_bits" or "io_alloc_cbm"
|
| 126 | but not all of these bits appear in the resource
|
| 127 | groups' schemata then the bits appearing in
|
| 128 | "shareable_bits" or "io_alloc_cbm" but no
|
| 129 | resource group will be marked as "H".
|
| 130 | "X":
|
| 131 | Corresponding region is available for sharing and
|
| 132 | used by hardware and software. These are the bits
|
| 133 | that appear in "shareable_bits" or "io_alloc_cbm"
|
| 134 | as well as a resource group's allocation.
|
| 135 | "S":
|
| 136 | Corresponding region is used by software
|
| 137 | and available for sharing.
|
| 138 | "E":
|
| 139 | Corresponding region is used exclusively by
|
| 140 | one resource group. No sharing allowed.
|
| 141 | "P":
|
| 142 | Corresponding region is pseudo-locked. No
|
| 143 | sharing allowed.
|
| 144 | "sparse_masks":
|
| 145 | Indicates if non-contiguous 1s value in CBM is supported.
|
| 146 |
|
| 147 | "0":
|
| 148 | Only contiguous 1s value in CBM is supported.
|
| 149 | "1":
|
| 150 | Non-contiguous 1s value in CBM is supported.
|
| 151 |
|
| 152 | "io_alloc":
|
| 153 | "io_alloc" enables system software to configure the portion of
|
| 154 | the cache allocated for I/O traffic. File may only exist if the
|
| 155 | system supports this feature on some of its cache resources.
|
| 156 |
|
| 157 | "disabled":
|
| 158 | Resource supports "io_alloc" but the feature is disabled.
|
| 159 | Portions of cache used for allocation of I/O traffic cannot
|
| 160 | be configured.
|
| 161 | "enabled":
|
| 162 | Portions of cache used for allocation of I/O traffic
|
| 163 | can be configured using "io_alloc_cbm".
|
| 164 | "not supported":
|
| 165 | Support not available for this resource.
|
| 166 |
|
| 167 | The feature can be modified by writing to the interface, for example:
|
| 168 |
|
| 169 | To enable::
|
| 170 |
|
| 171 | # echo 1 > /sys/fs/resctrl/info/L3/io_alloc
|
| 172 |
|
| 173 | To disable::
|
| 174 |
|
| 175 | # echo 0 > /sys/fs/resctrl/info/L3/io_alloc
|
| 176 |
|
| 177 | The underlying implementation may reduce resources available to
|
| 178 | general (CPU) cache allocation. See architecture specific notes
|
| 179 | below. Depending on usage requirements the feature can be enabled
|
| 180 | or disabled.
|
| 181 |
|
| 182 | On AMD systems, io_alloc feature is supported by the L3 Smart
|
| 183 | Data Cache Injection Allocation Enforcement (SDCIAE). The CLOSID for
|
| 184 | io_alloc is the highest CLOSID supported by the resource. When
|
| 185 | io_alloc is enabled, the highest CLOSID is dedicated to io_alloc and
|
| 186 | no longer available for general (CPU) cache allocation. When CDP is
|
| 187 | enabled, io_alloc routes I/O traffic using the highest CLOSID allocated
|
| 188 | for the instruction cache (CDP_CODE), making this CLOSID no longer
|
| 189 | available for general (CPU) cache allocation for both the CDP_CODE
|
| 190 | and CDP_DATA resources.
|
| 191 |
|
| 192 | "io_alloc_cbm":
|
| 193 | Capacity bitmasks that describe the portions of cache instances to
|
| 194 | which I/O traffic from supported I/O devices are routed when "io_alloc"
|
| 195 | is enabled.
|
| 196 |
|
| 197 | CBMs are displayed in the following format:
|
| 198 |
|
| 199 | <cache_id0>=<cbm>;<cache_id1>=<cbm>;...
|
| 200 |
|
| 201 | Example::
|
| 202 |
|
| 203 | # cat /sys/fs/resctrl/info/L3/io_alloc_cbm
|
| 204 | 0=ffff;1=ffff
|
| 205 |
|
| 206 | CBMs can be configured by writing to the interface.
|
| 207 |
|
| 208 | Example::
|
| 209 |
|
| 210 | # echo 1=ff > /sys/fs/resctrl/info/L3/io_alloc_cbm
|
| 211 | # cat /sys/fs/resctrl/info/L3/io_alloc_cbm
|
| 212 | 0=ffff;1=00ff
|
| 213 |
|
| 214 | # echo "0=ff;1=f" > /sys/fs/resctrl/info/L3/io_alloc_cbm
|
| 215 | # cat /sys/fs/resctrl/info/L3/io_alloc_cbm
|
| 216 | 0=00ff;1=000f
|
| 217 |
|
| 218 | An ID of "*" configures all domains with the provided CBM.
|
| 219 |
|
| 220 | Example on a system that does not require a minimum number of consecutive bits in the mask::
|
| 221 |
|
| 222 | # echo "*=0" > /sys/fs/resctrl/info/L3/io_alloc_cbm
|
| 223 | # cat /sys/fs/resctrl/info/L3/io_alloc_cbm
|
| 224 | 0=0;1=0
|
| 225 |
|
| 226 | When CDP is enabled "io_alloc_cbm" associated with the CDP_DATA and CDP_CODE
|
| 227 | resources may reflect the same values. For example, values read from and
|
| 228 | written to /sys/fs/resctrl/info/L3DATA/io_alloc_cbm may be reflected by
|
| 229 | /sys/fs/resctrl/info/L3CODE/io_alloc_cbm and vice versa.
|
| 230 |
|
| 231 | Memory bandwidth(MB) subdirectory contains the following files
|
| 232 | with respect to allocation:
|
| 233 |
|
| 234 | "min_bandwidth":
|
| 235 | The minimum memory bandwidth percentage which
|
| 236 | user can request.
|
| 237 |
|
| 238 | "bandwidth_gran":
|
| 239 | The approximate granularity in which the memory bandwidth
|
| 240 | percentage is allocated. The allocated bandwidth percentage is
|
| 241 | rounded up or down to the closest control step available on the
|
| 242 | hardware. The available hardware steps are no larger than this
|
| 243 | value.
|
| 244 |
|
| 245 | "delay_linear":
|
| 246 | Indicates if the delay scale is linear or
|
| 247 | non-linear. This field is purely informational
|
| 248 | only.
|
| 249 |
|
| 250 | "thread_throttle_mode":
|
| 251 | Indicator on Intel systems of how tasks running on threads
|
| 252 | of a physical core are throttled in cases where they
|
| 253 | request different memory bandwidth percentages:
|
| 254 |
|
| 255 | "max":
|
| 256 | the smallest percentage is applied
|
| 257 | to all threads
|
| 258 | "per-thread":
|
| 259 | bandwidth percentages are directly applied to
|
| 260 | the threads running on the core
|
| 261 |
|
| 262 | If L3 monitoring is available there will be an "L3_MON" directory
|
| 263 | with the following files:
|
| 264 |
|
| 265 | "num_rmids":
|
| 266 | The number of RMIDs supported by hardware for
|
| 267 | L3 monitoring events.
|
| 268 |
|
| 269 | "mon_features":
|
| 270 | Lists the monitoring events if
|
| 271 | monitoring is enabled for the resource.
|
| 272 | Example::
|
| 273 |
|
| 274 | # cat /sys/fs/resctrl/info/L3_MON/mon_features
|
| 275 | llc_occupancy
|
| 276 | mbm_total_bytes
|
| 277 | mbm_local_bytes
|
| 278 |
|
| 279 | If the system supports Bandwidth Monitoring Event
|
| 280 | Configuration (BMEC), then the bandwidth events will
|
| 281 | be configurable. The output will be::
|
| 282 |
|
| 283 | # cat /sys/fs/resctrl/info/L3_MON/mon_features
|
| 284 | llc_occupancy
|
| 285 | mbm_total_bytes
|
| 286 | mbm_total_bytes_config
|
| 287 | mbm_local_bytes
|
| 288 | mbm_local_bytes_config
|
| 289 |
|
| 290 | "mbm_total_bytes_config", "mbm_local_bytes_config":
|
| 291 | Read/write files containing the configuration for the mbm_total_bytes
|
| 292 | and mbm_local_bytes events, respectively, when the Bandwidth
|
| 293 | Monitoring Event Configuration (BMEC) feature is supported.
|
| 294 | The event configuration settings are domain specific and affect
|
| 295 | all the CPUs in the domain. When either event configuration is
|
| 296 | changed, the bandwidth counters for all RMIDs of both events
|
| 297 | (mbm_total_bytes as well as mbm_local_bytes) are cleared for that
|
| 298 | domain. The next read for every RMID will report "Unavailable"
|
| 299 | and subsequent reads will report the valid value.
|
| 300 |
|
| 301 | Following are the types of events supported:
|
| 302 |
|
| 303 | ==== ========================================================
|
| 304 | Bits Description
|
| 305 | ==== ========================================================
|
| 306 | 6 Dirty Victims from the QOS domain to all types of memory
|
| 307 | 5 Reads to slow memory in the non-local NUMA domain
|
| 308 | 4 Reads to slow memory in the local NUMA domain
|
| 309 | 3 Non-temporal writes to non-local NUMA domain
|
| 310 | 2 Non-temporal writes to local NUMA domain
|
| 311 | 1 Reads to memory in the non-local NUMA domain
|
| 312 | 0 Reads to memory in the local NUMA domain
|
| 313 | ==== ========================================================
|
| 314 |
|
| 315 | By default, the mbm_total_bytes configuration is set to 0x7f to count
|
| 316 | all the event types and the mbm_local_bytes configuration is set to
|
| 317 | 0x15 to count all the local memory events.
|
| 318 |
|
| 319 | Examples:
|
| 320 |
|
| 321 | * To view the current configuration::
|
| 322 | ::
|
| 323 |
|
| 324 | # cat /sys/fs/resctrl/info/L3_MON/mbm_total_bytes_config
|
| 325 | 0=0x7f;1=0x7f;2=0x7f;3=0x7f
|
| 326 |
|
| 327 | # cat /sys/fs/resctrl/info/L3_MON/mbm_local_bytes_config
|
| 328 | 0=0x15;1=0x15;3=0x15;4=0x15
|
| 329 |
|
| 330 | * To change the mbm_total_bytes to count only reads on domain 0,
|
| 331 | the bits 0, 1, 4 and 5 needs to be set, which is 110011b in binary
|
| 332 | (in hexadecimal 0x33):
|
| 333 | ::
|
| 334 |
|
| 335 | # echo "0=0x33" > /sys/fs/resctrl/info/L3_MON/mbm_total_bytes_config
|
| 336 |
|
| 337 | # cat /sys/fs/resctrl/info/L3_MON/mbm_total_bytes_config
|
| 338 | 0=0x33;1=0x7f;2=0x7f;3=0x7f
|
| 339 |
|
| 340 | * To change the mbm_local_bytes to count all the slow memory reads on
|
| 341 | domain 0 and 1, the bits 4 and 5 needs to be set, which is 110000b
|
| 342 | in binary (in hexadecimal 0x30):
|
| 343 | ::
|
| 344 |
|
| 345 | # echo "0=0x30;1=0x30" > /sys/fs/resctrl/info/L3_MON/mbm_local_bytes_config
|
| 346 |
|
| 347 | # cat /sys/fs/resctrl/info/L3_MON/mbm_local_bytes_config
|
| 348 | 0=0x30;1=0x30;3=0x15;4=0x15
|
| 349 |
|
| 350 | "mbm_assign_mode":
|
| 351 | The supported counter assignment modes. The enclosed brackets indicate which mode
|
| 352 | is enabled. The MBM events associated with counters may reset when "mbm_assign_mode"
|
| 353 | is changed.
|
| 354 | ::
|
| 355 |
|
| 356 | # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
|
| 357 | [mbm_event]
|
| 358 | default
|
| 359 |
|
| 360 | "mbm_event":
|
| 361 |
|
| 362 | mbm_event mode allows users to assign a hardware counter to an RMID, event
|
| 363 | pair and monitor the bandwidth usage as long as it is assigned. The hardware
|
| 364 | continues to track the assigned counter until it is explicitly unassigned by
|
| 365 | the user. Each event within a resctrl group can be assigned independently.
|
| 366 |
|
| 367 | In this mode, a monitoring event can only accumulate data while it is backed
|
| 368 | by a hardware counter. Use "mbm_L3_assignments" found in each CTRL_MON and MON
|
| 369 | group to specify which of the events should have a counter assigned. The number
|
| 370 | of counters available is described in the "num_mbm_cntrs" file. Changing the
|
| 371 | mode may cause all counters on the resource to reset.
|
| 372 |
|
| 373 | Moving to mbm_event counter assignment mode requires users to assign the counters
|
| 374 | to the events. Otherwise, the MBM event counters will return 'Unassigned' when read.
|
| 375 |
|
| 376 | The mode is beneficial for AMD platforms that support more CTRL_MON
|
| 377 | and MON groups than available hardware counters. By default, this
|
| 378 | feature is enabled on AMD platforms with the ABMC (Assignable Bandwidth
|
| 379 | Monitoring Counters) capability, ensuring counters remain assigned even
|
| 380 | when the corresponding RMID is not actively used by any processor.
|
| 381 |
|
| 382 | "default":
|
| 383 |
|
| 384 | In default mode, resctrl assumes there is a hardware counter for each
|
| 385 | event within every CTRL_MON and MON group. On AMD platforms, it is
|
| 386 | recommended to use the mbm_event mode, if supported, to prevent reset of MBM
|
| 387 | events between reads resulting from hardware re-allocating counters. This can
|
| 388 | result in misleading values or display "Unavailable" if no counter is assigned
|
| 389 | to the event.
|
| 390 |
|
| 391 | * To enable "mbm_event" counter assignment mode:
|
| 392 | ::
|
| 393 |
|
| 394 | # echo "mbm_event" > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
|
| 395 |
|
| 396 | * To enable "default" monitoring mode:
|
| 397 | ::
|
| 398 |
|
| 399 | # echo "default" > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
|
| 400 |
|
| 401 | "num_mbm_cntrs":
|
| 402 | The maximum number of counters (total of available and assigned counters) in
|
| 403 | each domain when the system supports mbm_event mode.
|
| 404 |
|
| 405 | For example, on a system with maximum of 32 memory bandwidth monitoring
|
| 406 | counters in each of its L3 domains:
|
| 407 | ::
|
| 408 |
|
| 409 | # cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs
|
| 410 | 0=32;1=32
|
| 411 |
|
| 412 | "available_mbm_cntrs":
|
| 413 | The number of counters available for assignment in each domain when mbm_event
|
| 414 | mode is enabled on the system.
|
| 415 |
|
| 416 | For example, on a system with 30 available [hardware] assignable counters
|
| 417 | in each of its L3 domains:
|
| 418 | ::
|
| 419 |
|
| 420 | # cat /sys/fs/resctrl/info/L3_MON/available_mbm_cntrs
|
| 421 | 0=30;1=30
|
| 422 |
|
| 423 | "event_configs":
|
| 424 | Directory that exists when "mbm_event" counter assignment mode is supported.
|
| 425 | Contains a sub-directory for each MBM event that can be assigned to a counter.
|
| 426 |
|
| 427 | Two MBM events are supported by default: mbm_local_bytes and mbm_total_bytes.
|
| 428 | Each MBM event's sub-directory contains a file named "event_filter" that is
|
| 429 | used to view and (if writable) modify which memory transactions the MBM event
|
| 430 | is configured with. The file is accessible only when "mbm_event" counter
|
| 431 | assignment mode is enabled.
|
| 432 |
|
| 433 | List of memory transaction types supported:
|
| 434 |
|
| 435 | ========================== ========================================================
|
| 436 | Name Description
|
| 437 | ========================== ========================================================
|
| 438 | dirty_victim_writes_all Dirty Victims from the QOS domain to all types of memory
|
| 439 | remote_reads_slow_memory Reads to slow memory in the non-local NUMA domain
|
| 440 | local_reads_slow_memory Reads to slow memory in the local NUMA domain
|
| 441 | remote_non_temporal_writes Non-temporal writes to non-local NUMA domain
|
| 442 | local_non_temporal_writes Non-temporal writes to local NUMA domain
|
| 443 | remote_reads Reads to memory in the non-local NUMA domain
|
| 444 | local_reads Reads to memory in the local NUMA domain
|
| 445 | ========================== ========================================================
|
| 446 |
|
| 447 | For example::
|
| 448 |
|
| 449 | # cat /sys/fs/resctrl/info/L3_MON/event_configs/mbm_total_bytes/event_filter
|
| 450 | local_reads,remote_reads,local_non_temporal_writes,remote_non_temporal_writes,
|
| 451 | local_reads_slow_memory,remote_reads_slow_memory,dirty_victim_writes_all
|
| 452 |
|
| 453 | # cat /sys/fs/resctrl/info/L3_MON/event_configs/mbm_local_bytes/event_filter
|
| 454 | local_reads,local_non_temporal_writes,local_reads_slow_memory
|
| 455 |
|
| 456 | The memory transactions the MBM event is configured with can be changed
|
| 457 | if "event_filter" is writable.
|
| 458 |
|
| 459 | For example::
|
| 460 |
|
| 461 | # echo "local_reads, local_non_temporal_writes" >
|
| 462 | /sys/fs/resctrl/info/L3_MON/event_configs/mbm_total_bytes/event_filter
|
| 463 |
|
| 464 | # cat /sys/fs/resctrl/info/L3_MON/event_configs/mbm_total_bytes/event_filter
|
| 465 | local_reads,local_non_temporal_writes
|
| 466 |
|
| 467 | "mbm_assign_on_mkdir":
|
| 468 | Exists when "mbm_event" counter assignment mode is supported. Accessible
|
| 469 | only when "mbm_event" counter assignment mode is enabled.
|
| 470 |
|
| 471 | Determines if a counter will automatically be assigned to an RMID, MBM event
|
| 472 | pair when its associated monitor group is created via mkdir. Enabled by default
|
| 473 | on boot, also when switched from "default" mode to "mbm_event" counter assignment
|
| 474 | mode. Users can disable this capability by writing to the interface.
|
| 475 |
|
| 476 | "0":
|
| 477 | Auto assignment is disabled.
|
| 478 | "1":
|
| 479 | Auto assignment is enabled.
|
| 480 |
|
| 481 | Automatic counter assignment is done with best effort. If auto
|
| 482 | assignment is enabled but there are not enough available counters then
|
| 483 | monitor group creation could succeed while one or more events belonging
|
| 484 | to the group may not have a counter assigned in all domains. Consult
|
| 485 | mbm_L3_assignments for counter assignment states of the new groups.
|
| 486 |
|
| 487 | Example::
|
| 488 |
|
| 489 | # echo 0 > /sys/fs/resctrl/info/L3_MON/mbm_assign_on_mkdir
|
| 490 | # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_on_mkdir
|
| 491 | 0
|
| 492 |
|
| 493 | "max_threshold_occupancy":
|
| 494 | Read/write file provides the largest value (in
|
| 495 | bytes) at which a previously used LLC_occupancy
|
| 496 | counter can be considered for reuse.
|
| 497 |
|
| 498 | If telemetry monitoring is available there will be a "PERF_PKG_MON" directory
|
| 499 | with the following files:
|
| 500 |
|
| 501 | "num_rmids":
|
| 502 | The number of RMIDs for telemetry monitoring events.
|
| 503 |
|
| 504 | On Intel resctrl will not enable telemetry events if the number of
|
| 505 | RMIDs that can be tracked concurrently is lower than the total number
|
| 506 | of RMIDs supported. Telemetry events can be force-enabled with the
|
| 507 | "rdt=" kernel parameter, but this may reduce the number of
|
| 508 | monitoring groups that can be created.
|
| 509 |
|
| 510 | "mon_features":
|
| 511 | Lists the telemetry monitoring events that are enabled on this system.
|
| 512 |
|
| 513 | The upper bound for how many "CTRL_MON" + "MON" can be created
|
| 514 | is the smaller of the L3_MON and PERF_PKG_MON "num_rmids" values.
|
| 515 |
|
| 516 | Finally, in the top level of the "info" directory there is a file
|
| 517 | named "last_cmd_status". This is reset with every "command" issued
|
| 518 | via the file system (making new directories or writing to any of the
|
| 519 | control files). If the command was successful, it will read as "ok".
|
| 520 | If the command failed, it will provide more information that can be
|
| 521 | conveyed in the error returns from file operations. E.g.
|
| 522 | ::
|
| 523 |
|
| 524 | # echo L3:0=f7 > schemata
|
| 525 | bash: echo: write error: Invalid argument
|
| 526 | # cat info/last_cmd_status
|
| 527 | mask f7 has non-consecutive 1-bits
|
| 528 |
|
| 529 | Resource alloc and monitor groups
|
| 530 | =================================
|
| 531 |
|
| 532 | Resource groups are represented as directories in the resctrl file
|
| 533 | system. The default group is the root directory which, immediately
|
| 534 | after mounting, owns all the tasks and cpus in the system and can make
|
| 535 | full use of all resources.
|
| 536 |
|
| 537 | On a system with RDT control features additional directories can be
|
| 538 | created in the root directory that specify different amounts of each
|
| 539 | resource (see "schemata" below). The root and these additional top level
|
| 540 | directories are referred to as "CTRL_MON" groups below.
|
| 541 |
|
| 542 | On a system with RDT monitoring the root directory and other top level
|
| 543 | directories contain a directory named "mon_groups" in which additional
|
| 544 | directories can be created to monitor subsets of tasks in the CTRL_MON
|
| 545 | group that is their ancestor. These are called "MON" groups in the rest
|
| 546 | of this document.
|
| 547 |
|
| 548 | Removing a directory will move all tasks and cpus owned by the group it
|
| 549 | represents to the parent. Removing one of the created CTRL_MON groups
|
| 550 | will automatically remove all MON groups below it.
|
| 551 |
|
| 552 | Moving MON group directories to a new parent CTRL_MON group is supported
|
| 553 | for the purpose of changing the resource allocations of a MON group
|
| 554 | without impacting its monitoring data or assigned tasks. This operation
|
| 555 | is not allowed for MON groups which monitor CPUs. No other move
|
| 556 | operation is currently allowed other than simply renaming a CTRL_MON or
|
| 557 | MON group.
|
| 558 |
|
| 559 | All groups contain the following files:
|
| 560 |
|
| 561 | "tasks":
|
| 562 | Reading this file shows the list of all tasks that belong to
|
| 563 | this group. Writing a task id to the file will add a task to the
|
| 564 | group. Multiple tasks can be added by separating the task ids
|
| 565 | with commas. Tasks will be assigned sequentially. Multiple
|
| 566 | failures are not supported. A single failure encountered while
|
| 567 | attempting to assign a task will cause the operation to abort and
|
| 568 | already added tasks before the failure will remain in the group.
|
| 569 | Failures will be logged to /sys/fs/resctrl/info/last_cmd_status.
|
| 570 |
|
| 571 | If the group is a CTRL_MON group the task is removed from
|
| 572 | whichever previous CTRL_MON group owned the task and also from
|
| 573 | any MON group that owned the task. If the group is a MON group,
|
| 574 | then the task must already belong to the CTRL_MON parent of this
|
| 575 | group. The task is removed from any previous MON group.
|
| 576 |
|
| 577 | When writing to this file, a task id of 0 is interpreted as the
|
| 578 | task id of the currently running task. On reading the file, a task
|
| 579 | id of 0 will never be shown and there is no representation of the
|
| 580 | idle tasks. Instead, a CPU's idle task is always considered as a
|
| 581 | member of the group owning the CPU.
|
| 582 |
|
| 583 | "cpus":
|
| 584 | Reading this file shows a bitmask of the logical CPUs owned by
|
| 585 | this group. Writing a mask to this file will add and remove
|
| 586 | CPUs to/from this group. As with the tasks file a hierarchy is
|
| 587 | maintained where MON groups may only include CPUs owned by the
|
| 588 | parent CTRL_MON group.
|
| 589 | When the resource group is in pseudo-locked mode this file will
|
| 590 | only be readable, reflecting the CPUs associated with the
|
| 591 | pseudo-locked region.
|
| 592 |
|
| 593 |
|
| 594 | "cpus_list":
|
| 595 | Just like "cpus", only using ranges of CPUs instead of bitmasks.
|
| 596 |
|
| 597 |
|
| 598 | When control is enabled all CTRL_MON groups will also contain:
|
| 599 |
|
| 600 | "schemata":
|
| 601 | A list of all the resources available to this group.
|
| 602 | Each resource has its own line and format - see below for details.
|
| 603 |
|
| 604 | "size":
|
| 605 | Mirrors the display of the "schemata" file to display the size in
|
| 606 | bytes of each allocation instead of the bits representing the
|
| 607 | allocation.
|
| 608 |
|
| 609 | "mode":
|
| 610 | The "mode" of the resource group dictates the sharing of its
|
| 611 | allocations. A "shareable" resource group allows sharing of its
|
| 612 | allocations while an "exclusive" resource group does not. A
|
| 613 | cache pseudo-locked region is created by first writing
|
| 614 | "pseudo-locksetup" to the "mode" file before writing the cache
|
| 615 | pseudo-locked region's schemata to the resource group's "schemata"
|
| 616 | file. On successful pseudo-locked region creation the mode will
|
| 617 | automatically change to "pseudo-locked".
|
| 618 |
|
| 619 | "ctrl_hw_id":
|
| 620 | Available only with debug option. The identifier used by hardware
|
| 621 | for the control group. On x86 this is the CLOSID.
|
| 622 |
|
| 623 | When monitoring is enabled all MON groups will also contain:
|
| 624 |
|
| 625 | "mon_data":
|
| 626 | This contains directories for each monitor domain.
|
| 627 |
|
| 628 | If L3 monitoring is enabled, there will be a "mon_L3_XX" directory for
|
| 629 | each instance of an L3 cache. Each directory contains files for the enabled
|
| 630 | L3 events (e.g. "llc_occupancy", "mbm_total_bytes", and "mbm_local_bytes").
|
| 631 |
|
| 632 | If telemetry monitoring is enabled, there will be a "mon_PERF_PKG_YY"
|
| 633 | directory for each physical processor package. Each directory contains
|
| 634 | files for the enabled telemetry events (e.g. "core_energy". "activity",
|
| 635 | "uops_retired", etc.)
|
| 636 |
|
| 637 | The info/`*`/mon_features files provide the full list of enabled
|
| 638 | event/file names.
|
| 639 |
|
| 640 | "core energy" reports a floating point number for the energy (in Joules)
|
| 641 | consumed by cores (registers, arithmetic units, TLB and L1/L2 caches)
|
| 642 | during execution of instructions summed across all logical CPUs on a
|
| 643 | package for the current monitoring group.
|
| 644 |
|
| 645 | "activity" also reports a floating point value (in nanofarads). This provides
|
| 646 | an estimate of work done independent of the frequency that the CPUs used
|
| 647 | for execution.
|
| 648 |
|
| 649 | Note that "core energy" and "activity" only measure energy/activity in the
|
| 650 | "core" of the CPU (arithmetic units, TLB, L1 and L2 caches, etc.). They
|
| 651 | do not include L3 cache, memory, I/O devices etc.
|
| 652 |
|
| 653 | All other events report decimal integer values.
|
| 654 |
|
| 655 | In a MON group these files provide a read out of the current value of
|
| 656 | the event for all tasks in the group. In CTRL_MON groups these files
|
| 657 | provide the sum for all tasks in the CTRL_MON group and all tasks in
|
| 658 | MON groups. Please see example section for more details on usage.
|
| 659 |
|
| 660 | On systems with Sub-NUMA Cluster (SNC) enabled there are extra
|
| 661 | directories for each node (located within the "mon_L3_XX" directory
|
| 662 | for the L3 cache they occupy). These are named "mon_sub_L3_YY"
|
| 663 | where "YY" is the node number.
|
| 664 |
|
| 665 | When the 'mbm_event' counter assignment mode is enabled, reading
|
| 666 | an MBM event of a MON group returns 'Unassigned' if no hardware
|
| 667 | counter is assigned to it. For CTRL_MON groups, 'Unassigned' is
|
| 668 | returned if the MBM event does not have an assigned counter in the
|
| 669 | CTRL_MON group nor in any of its associated MON groups.
|
| 670 |
|
| 671 | "mon_hw_id":
|
| 672 | Available only with debug option. The identifier used by hardware
|
| 673 | for the monitor group. On x86 this is the RMID.
|
| 674 |
|
| 675 | When monitoring is enabled all MON groups may also contain:
|
| 676 |
|
| 677 | "mbm_L3_assignments":
|
| 678 | Exists when "mbm_event" counter assignment mode is supported and lists the
|
| 679 | counter assignment states of the group.
|
| 680 |
|
| 681 | The assignment list is displayed in the following format:
|
| 682 |
|
| 683 | <Event>:<Domain ID>=<Assignment state>;<Domain ID>=<Assignment state>
|
| 684 |
|
| 685 | Event: A valid MBM event in the
|
| 686 | /sys/fs/resctrl/info/L3_MON/event_configs directory.
|
| 687 |
|
| 688 | Domain ID: A valid domain ID. When writing, '*' applies the changes
|
| 689 | to all the domains.
|
| 690 |
|
| 691 | Assignment states:
|
| 692 |
|
| 693 | _ : No counter assigned.
|
| 694 |
|
| 695 | e : Counter assigned exclusively.
|
| 696 |
|
| 697 | Example:
|
| 698 |
|
| 699 | To display the counter assignment states for the default group.
|
| 700 | ::
|
| 701 |
|
| 702 | # cd /sys/fs/resctrl
|
| 703 | # cat /sys/fs/resctrl/mbm_L3_assignments
|
| 704 | mbm_total_bytes:0=e;1=e
|
| 705 | mbm_local_bytes:0=e;1=e
|
| 706 |
|
| 707 | Assignments can be modified by writing to the interface.
|
| 708 |
|
| 709 | Examples:
|
| 710 |
|
| 711 | To unassign the counter associated with the mbm_total_bytes event on domain 0:
|
| 712 | ::
|
| 713 |
|
| 714 | # echo "mbm_total_bytes:0=_" > /sys/fs/resctrl/mbm_L3_assignments
|
| 715 | # cat /sys/fs/resctrl/mbm_L3_assignments
|
| 716 | mbm_total_bytes:0=_;1=e
|
| 717 | mbm_local_bytes:0=e;1=e
|
| 718 |
|
| 719 | To unassign the counter associated with the mbm_total_bytes event on all the domains:
|
| 720 | ::
|
| 721 |
|
| 722 | # echo "mbm_total_bytes:*=_" > /sys/fs/resctrl/mbm_L3_assignments
|
| 723 | # cat /sys/fs/resctrl/mbm_L3_assignments
|
| 724 | mbm_total_bytes:0=_;1=_
|
| 725 | mbm_local_bytes:0=e;1=e
|
| 726 |
|
| 727 | To assign a counter associated with the mbm_total_bytes event on all domains in
|
| 728 | exclusive mode:
|
| 729 | ::
|
| 730 |
|
| 731 | # echo "mbm_total_bytes:*=e" > /sys/fs/resctrl/mbm_L3_assignments
|
| 732 | # cat /sys/fs/resctrl/mbm_L3_assignments
|
| 733 | mbm_total_bytes:0=e;1=e
|
| 734 | mbm_local_bytes:0=e;1=e
|
| 735 |
|
| 736 | When the "mba_MBps" mount option is used all CTRL_MON groups will also contain:
|
| 737 |
|
| 738 | "mba_MBps_event":
|
| 739 | Reading this file shows which memory bandwidth event is used
|
| 740 | as input to the software feedback loop that keeps memory bandwidth
|
| 741 | below the value specified in the schemata file. Writing the
|
| 742 | name of one of the supported memory bandwidth events found in
|
| 743 | /sys/fs/resctrl/info/L3_MON/mon_features changes the input
|
| 744 | event.
|
| 745 |
|
| 746 | Resource allocation rules
|
| 747 | -------------------------
|
| 748 |
|
| 749 | When a task is running the following rules define which resources are
|
| 750 | available to it:
|
| 751 |
|
| 752 | 1) If the task is a member of a non-default group, then the schemata
|
| 753 | for that group is used.
|
| 754 |
|
| 755 | 2) Else if the task belongs to the default group, but is running on a
|
| 756 | CPU that is assigned to some specific group, then the schemata for the
|
| 757 | CPU's group is used.
|
| 758 |
|
| 759 | 3) Otherwise the schemata for the default group is used.
|
| 760 |
|
| 761 | Resource monitoring rules
|
| 762 | -------------------------
|
| 763 | 1) If a task is a member of a MON group, or non-default CTRL_MON group
|
| 764 | then RDT events for the task will be reported in that group.
|
| 765 |
|
| 766 | 2) If a task is a member of the default CTRL_MON group, but is running
|
| 767 | on a CPU that is assigned to some specific group, then the RDT events
|
| 768 | for the task will be reported in that group.
|
| 769 |
|
| 770 | 3) Otherwise RDT events for the task will be reported in the root level
|
| 771 | "mon_data" group.
|
| 772 |
|
| 773 |
|
| 774 | Notes on cache occupancy monitoring and control
|
| 775 | ===============================================
|
| 776 | When moving a task from one group to another you should remember that
|
| 777 | this only affects *new* cache allocations by the task. E.g. you may have
|
| 778 | a task in a monitor group showing 3 MB of cache occupancy. If you move
|
| 779 | to a new group and immediately check the occupancy of the old and new
|
| 780 | groups you will likely see that the old group is still showing 3 MB and
|
| 781 | the new group zero. When the task accesses locations still in cache from
|
| 782 | before the move, the h/w does not update any counters. On a busy system
|
| 783 | you will likely see the occupancy in the old group go down as cache lines
|
| 784 | are evicted and re-used while the occupancy in the new group rises as
|
| 785 | the task accesses memory and loads into the cache are counted based on
|
| 786 | membership in the new group.
|
| 787 |
|
| 788 | The same applies to cache allocation control. Moving a task to a group
|
| 789 | with a smaller cache partition will not evict any cache lines. The
|
| 790 | process may continue to use them from the old partition.
|
| 791 |
|
| 792 | Hardware uses CLOSid(Class of service ID) and an RMID(Resource monitoring ID)
|
| 793 | to identify a control group and a monitoring group respectively. Each of
|
| 794 | the resource groups are mapped to these IDs based on the kind of group. The
|
| 795 | number of CLOSid and RMID are limited by the hardware and hence the creation of
|
| 796 | a "CTRL_MON" directory may fail if we run out of either CLOSID or RMID
|
| 797 | and creation of "MON" group may fail if we run out of RMIDs.
|
| 798 |
|
| 799 | max_threshold_occupancy - generic concepts
|
| 800 | ------------------------------------------
|
| 801 |
|
| 802 | Note that an RMID once freed may not be immediately available for use as
|
| 803 | the RMID is still tagged the cache lines of the previous user of RMID.
|
| 804 | Hence such RMIDs are placed on limbo list and checked back if the cache
|
| 805 | occupancy has gone down. If there is a time when system has a lot of
|
| 806 | limbo RMIDs but which are not ready to be used, user may see an -EBUSY
|
| 807 | during mkdir.
|
| 808 |
|
| 809 | max_threshold_occupancy is a user configurable value to determine the
|
| 810 | occupancy at which an RMID can be freed.
|
| 811 |
|
| 812 | The mon_llc_occupancy_limbo tracepoint gives the precise occupancy in bytes
|
| 813 | for a subset of RMID that are not immediately available for allocation.
|
| 814 | This can't be relied on to produce output every second, it may be necessary
|
| 815 | to attempt to create an empty monitor group to force an update. Output may
|
| 816 | only be produced if creation of a control or monitor group fails.
|
| 817 |
|
| 818 | Schemata files - general concepts
|
| 819 | ---------------------------------
|
| 820 | Each line in the file describes one resource. The line starts with
|
| 821 | the name of the resource, followed by specific values to be applied
|
| 822 | in each of the instances of that resource on the system.
|
| 823 |
|
| 824 | Cache IDs
|
| 825 | ---------
|
| 826 | On current generation systems there is one L3 cache per socket and L2
|
| 827 | caches are generally just shared by the hyperthreads on a core, but this
|
| 828 | isn't an architectural requirement. We could have multiple separate L3
|
| 829 | caches on a socket, multiple cores could share an L2 cache. So instead
|
| 830 | of using "socket" or "core" to define the set of logical cpus sharing
|
| 831 | a resource we use a "Cache ID". At a given cache level this will be a
|
| 832 | unique number across the whole system (but it isn't guaranteed to be a
|
| 833 | contiguous sequence, there may be gaps). To find the ID for each logical
|
| 834 | CPU look in /sys/devices/system/cpu/cpu*/cache/index*/id
|
| 835 |
|
| 836 | Cache Bit Masks (CBM)
|
| 837 | ---------------------
|
| 838 | For cache resources we describe the portion of the cache that is available
|
| 839 | for allocation using a bitmask. The maximum value of the mask is defined
|
| 840 | by each cpu model (and may be different for different cache levels). It
|
| 841 | is found using CPUID, but is also provided in the "info" directory of
|
| 842 | the resctrl file system in "info/{resource}/cbm_mask". Some Intel hardware
|
| 843 | requires that these masks have all the '1' bits in a contiguous block. So
|
| 844 | 0x3, 0x6 and 0xC are legal 4-bit masks with two bits set, but 0x5, 0x9
|
| 845 | and 0xA are not. Check /sys/fs/resctrl/info/{resource}/sparse_masks
|
| 846 | if non-contiguous 1s value is supported. On a system with a 20-bit mask
|
| 847 | each bit represents 5% of the capacity of the cache. You could partition
|
| 848 | the cache into four equal parts with masks: 0x1f, 0x3e0, 0x7c00, 0xf8000.
|
| 849 |
|
| 850 | Notes on Sub-NUMA Cluster mode
|
| 851 | ==============================
|
| 852 | When SNC mode is enabled, Linux may load balance tasks between Sub-NUMA
|
| 853 | nodes much more readily than between regular NUMA nodes since the CPUs
|
| 854 | on Sub-NUMA nodes share the same L3 cache and the system may report
|
| 855 | the NUMA distance between Sub-NUMA nodes with a lower value than used
|
| 856 | for regular NUMA nodes.
|
| 857 |
|
| 858 | The top-level monitoring files in each "mon_L3_XX" directory provide
|
| 859 | the sum of data across all SNC nodes sharing an L3 cache instance.
|
| 860 | Users who bind tasks to the CPUs of a specific Sub-NUMA node can read
|
| 861 | the "llc_occupancy", "mbm_total_bytes", and "mbm_local_bytes" in the
|
| 862 | "mon_sub_L3_YY" directories to get node local data.
|
| 863 |
|
| 864 | Memory bandwidth allocation is still performed at the L3 cache
|
| 865 | level. I.e. throttling controls are applied to all SNC nodes.
|
| 866 |
|
| 867 | L3 cache allocation bitmaps also apply to all SNC nodes. But note that
|
| 868 | the amount of L3 cache represented by each bit is divided by the number
|
| 869 | of SNC nodes per L3 cache. E.g. with a 100MB cache on a system with 10-bit
|
| 870 | allocation masks each bit normally represents 10MB. With SNC mode enabled
|
| 871 | with two SNC nodes per L3 cache, each bit only represents 5MB.
|
| 872 |
|
| 873 | Memory bandwidth Allocation and monitoring
|
| 874 | ==========================================
|
| 875 |
|
| 876 | For Memory bandwidth resource, by default the user controls the resource
|
| 877 | by indicating the percentage of total memory bandwidth.
|
| 878 |
|
| 879 | The minimum bandwidth percentage value for each cpu model is predefined
|
| 880 | and can be looked up through "info/MB/min_bandwidth". The bandwidth
|
| 881 | granularity that is allocated is also dependent on the cpu model and can
|
| 882 | be looked up at "info/MB/bandwidth_gran". The available bandwidth
|
| 883 | control steps are, approximately, min_bw + N * bw_gran. The steps may
|
| 884 | appear irregular due to rounding to an exact percentage: bw_gran is the
|
| 885 | maximum interval between the percentage values corresponding to any two
|
| 886 | adjacent steps in the hardware.
|
| 887 |
|
| 888 | The bandwidth throttling is a core specific mechanism on some of Intel
|
| 889 | SKUs. Using a high bandwidth and a low bandwidth setting on two threads
|
| 890 | sharing a core may result in both threads being throttled to use the
|
| 891 | low bandwidth (see "thread_throttle_mode").
|
| 892 |
|
| 893 | The fact that Memory bandwidth allocation(MBA) may be a core
|
| 894 | specific mechanism where as memory bandwidth monitoring(MBM) is done at
|
| 895 | the package level may lead to confusion when users try to apply control
|
| 896 | via the MBA and then monitor the bandwidth to see if the controls are
|
| 897 | effective. Below are such scenarios:
|
| 898 |
|
| 899 | 1. User may *not* see increase in actual bandwidth when percentage
|
| 900 | values are increased:
|
| 901 |
|
| 902 | This can occur when aggregate L2 external bandwidth is more than L3
|
| 903 | external bandwidth. Consider an SKL SKU with 24 cores on a package and
|
| 904 | where L2 external is 10GBps (hence aggregate L2 external bandwidth is
|
| 905 | 240GBps) and L3 external bandwidth is 100GBps. Now a workload with '20
|
| 906 | threads, having 50% bandwidth, each consuming 5GBps' consumes the max L3
|
| 907 | bandwidth of 100GBps although the percentage value specified is only 50%
|
| 908 | << 100%. Hence increasing the bandwidth percentage will not yield any
|
| 909 | more bandwidth. This is because although the L2 external bandwidth still
|
| 910 | has capacity, the L3 external bandwidth is fully used. Also note that
|
| 911 | this would be dependent on number of cores the benchmark is run on.
|
| 912 |
|
| 913 | 2. Same bandwidth percentage may mean different actual bandwidth
|
| 914 | depending on # of threads:
|
| 915 |
|
| 916 | For the same SKU in #1, a 'single thread, with 10% bandwidth' and '4
|
| 917 | thread, with 10% bandwidth' can consume up to 10GBps and 40GBps although
|
| 918 | they have same percentage bandwidth of 10%. This is simply because as
|
| 919 | threads start using more cores in an rdtgroup, the actual bandwidth may
|
| 920 | increase or vary although user specified bandwidth percentage is same.
|
| 921 |
|
| 922 | In order to mitigate this and make the interface more user friendly,
|
| 923 | resctrl added support for specifying the bandwidth in MiBps as well. The
|
| 924 | kernel underneath would use a software feedback mechanism or a "Software
|
| 925 | Controller(mba_sc)" which reads the actual bandwidth using MBM counters
|
| 926 | and adjust the memory bandwidth percentages to ensure::
|
| 927 |
|
| 928 | "actual bandwidth < user specified bandwidth".
|
| 929 |
|
| 930 | By default, the schemata would take the bandwidth percentage values
|
| 931 | where as user can switch to the "MBA software controller" mode using
|
| 932 | a mount option 'mba_MBps'. The schemata format is specified in the below
|
| 933 | sections.
|
| 934 |
|
| 935 | L3 schemata file details (code and data prioritization disabled)
|
| 936 | ----------------------------------------------------------------
|
| 937 | With CDP disabled the L3 schemata format is::
|
| 938 |
|
| 939 | L3:<cache_id0>=<cbm>;<cache_id1>=<cbm>;...
|
| 940 |
|
| 941 | L3 schemata file details (CDP enabled via mount option to resctrl)
|
| 942 | ------------------------------------------------------------------
|
| 943 | When CDP is enabled L3 control is split into two separate resources
|
| 944 | so you can specify independent masks for code and data like this::
|
| 945 |
|
| 946 | L3DATA:<cache_id0>=<cbm>;<cache_id1>=<cbm>;...
|
| 947 | L3CODE:<cache_id0>=<cbm>;<cache_id1>=<cbm>;...
|
| 948 |
|
| 949 | L2 schemata file details
|
| 950 | ------------------------
|
| 951 | CDP is supported at L2 using the 'cdpl2' mount option. The schemata
|
| 952 | format is either::
|
| 953 |
|
| 954 | L2:<cache_id0>=<cbm>;<cache_id1>=<cbm>;...
|
| 955 |
|
| 956 | or
|
| 957 |
|
| 958 | L2DATA:<cache_id0>=<cbm>;<cache_id1>=<cbm>;...
|
| 959 | L2CODE:<cache_id0>=<cbm>;<cache_id1>=<cbm>;...
|
| 960 |
|
| 961 |
|
| 962 | Memory bandwidth Allocation (default mode)
|
| 963 | ------------------------------------------
|
| 964 |
|
| 965 | Memory b/w domain is L3 cache.
|
| 966 | ::
|
| 967 |
|
| 968 | MB:<cache_id0>=bandwidth0;<cache_id1>=bandwidth1;...
|
| 969 |
|
| 970 | Memory bandwidth Allocation specified in MiBps
|
| 971 | ----------------------------------------------
|
| 972 |
|
| 973 | Memory bandwidth domain is L3 cache.
|
| 974 | ::
|
| 975 |
|
| 976 | MB:<cache_id0>=bw_MiBps0;<cache_id1>=bw_MiBps1;...
|
| 977 |
|
| 978 | Slow Memory Bandwidth Allocation (SMBA)
|
| 979 | ---------------------------------------
|
| 980 | AMD hardware supports Slow Memory Bandwidth Allocation (SMBA).
|
| 981 | CXL.memory is the only supported "slow" memory device. With the
|
| 982 | support of SMBA, the hardware enables bandwidth allocation on
|
| 983 | the slow memory devices. If there are multiple such devices in
|
| 984 | the system, the throttling logic groups all the slow sources
|
| 985 | together and applies the limit on them as a whole.
|
| 986 |
|
| 987 | The presence of SMBA (with CXL.memory) is independent of slow memory
|
| 988 | devices presence. If there are no such devices on the system, then
|
| 989 | configuring SMBA will have no impact on the performance of the system.
|
| 990 |
|
| 991 | The bandwidth domain for slow memory is L3 cache. Its schemata file
|
| 992 | is formatted as:
|
| 993 | ::
|
| 994 |
|
| 995 | SMBA:<cache_id0>=bandwidth0;<cache_id1>=bandwidth1;...
|
| 996 |
|
| 997 | Reading/writing the schemata file
|
| 998 | ---------------------------------
|
| 999 | Reading the schemata file will show the state of all resources
|
| 1000 | on all domains. When writing you only need to specify those values
|
| 1001 | which you wish to change. E.g.
|
| 1002 | ::
|
| 1003 |
|
| 1004 | # cat schemata
|
| 1005 | L3DATA:0=fffff;1=fffff;2=fffff;3=fffff
|
| 1006 | L3CODE:0=fffff;1=fffff;2=fffff;3=fffff
|
| 1007 | # echo "L3DATA:2=3c0;" > schemata
|
| 1008 | # cat schemata
|
| 1009 | L3DATA:0=fffff;1=fffff;2=3c0;3=fffff
|
| 1010 | L3CODE:0=fffff;1=fffff;2=fffff;3=fffff
|
| 1011 |
|
| 1012 | Reading/writing the schemata file (on AMD systems)
|
| 1013 | --------------------------------------------------
|
| 1014 | Reading the schemata file will show the current bandwidth limit on all
|
| 1015 | domains. The allocated resources are in multiples of one eighth GB/s.
|
| 1016 | When writing to the file, you need to specify what cache id you wish to
|
| 1017 | configure the bandwidth limit.
|
| 1018 |
|
| 1019 | For example, to allocate 2GB/s limit on the first cache id:
|
| 1020 |
|
| 1021 | ::
|
| 1022 |
|
| 1023 | # cat schemata
|
| 1024 | MB:0=2048;1=2048;2=2048;3=2048
|
| 1025 | L3:0=ffff;1=ffff;2=ffff;3=ffff
|
| 1026 |
|
| 1027 | # echo "MB:1=16" > schemata
|
| 1028 | # cat schemata
|
| 1029 | MB:0=2048;1= 16;2=2048;3=2048
|
| 1030 | L3:0=ffff;1=ffff;2=ffff;3=ffff
|
| 1031 |
|
| 1032 | Reading/writing the schemata file (on AMD systems) with SMBA feature
|
| 1033 | --------------------------------------------------------------------
|
| 1034 | Reading and writing the schemata file is the same as without SMBA in
|
| 1035 | above section.
|
| 1036 |
|
| 1037 | For example, to allocate 8GB/s limit on the first cache id:
|
| 1038 |
|
| 1039 | ::
|
| 1040 |
|
| 1041 | # cat schemata
|
| 1042 | SMBA:0=2048;1=2048;2=2048;3=2048
|
| 1043 | MB:0=2048;1=2048;2=2048;3=2048
|
| 1044 | L3:0=ffff;1=ffff;2=ffff;3=ffff
|
| 1045 |
|
| 1046 | # echo "SMBA:1=64" > schemata
|
| 1047 | # cat schemata
|
| 1048 | SMBA:0=2048;1= 64;2=2048;3=2048
|
| 1049 | MB:0=2048;1=2048;2=2048;3=2048
|
| 1050 | L3:0=ffff;1=ffff;2=ffff;3=ffff
|
| 1051 |
|
| 1052 | Cache Pseudo-Locking
|
| 1053 | ====================
|
| 1054 | CAT enables a user to specify the amount of cache space that an
|
| 1055 | application can fill. Cache pseudo-locking builds on the fact that a
|
| 1056 | CPU can still read and write data pre-allocated outside its current
|
| 1057 | allocated area on a cache hit. With cache pseudo-locking, data can be
|
| 1058 | preloaded into a reserved portion of cache that no application can
|
| 1059 | fill, and from that point on will only serve cache hits. The cache
|
| 1060 | pseudo-locked memory is made accessible to user space where an
|
| 1061 | application can map it into its virtual address space and thus have
|
| 1062 | a region of memory with reduced average read latency.
|
| 1063 |
|
| 1064 | The creation of a cache pseudo-locked region is triggered by a request
|
| 1065 | from the user to do so that is accompanied by a schemata of the region
|
| 1066 | to be pseudo-locked. The cache pseudo-locked region is created as follows:
|
| 1067 |
|
| 1068 | - Create a CAT allocation CLOSNEW with a CBM matching the schemata
|
| 1069 | from the user of the cache region that will contain the pseudo-locked
|
| 1070 | memory. This region must not overlap with any current CAT allocation/CLOS
|
| 1071 | on the system and no future overlap with this cache region is allowed
|
| 1072 | while the pseudo-locked region exists.
|
| 1073 | - Create a contiguous region of memory of the same size as the cache
|
| 1074 | region.
|
| 1075 | - Flush the cache, disable hardware prefetchers, disable preemption.
|
| 1076 | - Make CLOSNEW the active CLOS and touch the allocated memory to load
|
| 1077 | it into the cache.
|
| 1078 | - Set the previous CLOS as active.
|
| 1079 | - At this point the closid CLOSNEW can be released - the cache
|
| 1080 | pseudo-locked region is protected as long as its CBM does not appear in
|
| 1081 | any CAT allocation. Even though the cache pseudo-locked region will from
|
| 1082 | this point on not appear in any CBM of any CLOS an application running with
|
| 1083 | any CLOS will be able to access the memory in the pseudo-locked region since
|
| 1084 | the region continues to serve cache hits.
|
| 1085 | - The contiguous region of memory loaded into the cache is exposed to
|
| 1086 | user-space as a character device.
|
| 1087 |
|
| 1088 | Cache pseudo-locking increases the probability that data will remain
|
| 1089 | in the cache via carefully configuring the CAT feature and controlling
|
| 1090 | application behavior. There is no guarantee that data is placed in
|
| 1091 | cache. Instructions like INVD, WBINVD, CLFLUSH, etc. can still evict
|
| 1092 | “locked” data from cache. Power management C-states may shrink or
|
| 1093 | power off cache. Deeper C-states will automatically be restricted on
|
| 1094 | pseudo-locked region creation.
|
| 1095 |
|
| 1096 | It is required that an application using a pseudo-locked region runs
|
| 1097 | with affinity to the cores (or a subset of the cores) associated
|
| 1098 | with the cache on which the pseudo-locked region resides. A sanity check
|
| 1099 | within the code will not allow an application to map pseudo-locked memory
|
| 1100 | unless it runs with affinity to cores associated with the cache on which the
|
| 1101 | pseudo-locked region resides. The sanity check is only done during the
|
| 1102 | initial mmap() handling, there is no enforcement afterwards and the
|
| 1103 | application self needs to ensure it remains affine to the correct cores.
|
| 1104 |
|
| 1105 | Pseudo-locking is accomplished in two stages:
|
| 1106 |
|
| 1107 | 1) During the first stage the system administrator allocates a portion
|
| 1108 | of cache that should be dedicated to pseudo-locking. At this time an
|
| 1109 | equivalent portion of memory is allocated, loaded into allocated
|
| 1110 | cache portion, and exposed as a character device.
|
| 1111 | 2) During the second stage a user-space application maps (mmap()) the
|
| 1112 | pseudo-locked memory into its address space.
|
| 1113 |
|
| 1114 | Cache Pseudo-Locking Interface
|
| 1115 | ------------------------------
|
| 1116 | A pseudo-locked region is created using the resctrl interface as follows:
|
| 1117 |
|
| 1118 | 1) Create a new resource group by creating a new directory in /sys/fs/resctrl.
|
| 1119 | 2) Change the new resource group's mode to "pseudo-locksetup" by writing
|
| 1120 | "pseudo-locksetup" to the "mode" file.
|
| 1121 | 3) Write the schemata of the pseudo-locked region to the "schemata" file. All
|
| 1122 | bits within the schemata should be "unused" according to the "bit_usage"
|
| 1123 | file.
|
| 1124 |
|
| 1125 | On successful pseudo-locked region creation the "mode" file will contain
|
| 1126 | "pseudo-locked" and a new character device with the same name as the resource
|
| 1127 | group will exist in /dev/pseudo_lock. This character device can be mmap()'ed
|
| 1128 | by user space in order to obtain access to the pseudo-locked memory region.
|
| 1129 |
|
| 1130 | An example of cache pseudo-locked region creation and usage can be found below.
|
| 1131 |
|
| 1132 | Cache Pseudo-Locking Debugging Interface
|
| 1133 | ----------------------------------------
|
| 1134 | The pseudo-locking debugging interface is enabled by default (if
|
| 1135 | CONFIG_DEBUG_FS is enabled) and can be found in /sys/kernel/debug/resctrl.
|
| 1136 |
|
| 1137 | There is no explicit way for the kernel to test if a provided memory
|
| 1138 | location is present in the cache. The pseudo-locking debugging interface uses
|
| 1139 | the tracing infrastructure to provide two ways to measure cache residency of
|
| 1140 | the pseudo-locked region:
|
| 1141 |
|
| 1142 | 1) Memory access latency using the pseudo_lock_mem_latency tracepoint. Data
|
| 1143 | from these measurements are best visualized using a hist trigger (see
|
| 1144 | example below). In this test the pseudo-locked region is traversed at
|
| 1145 | a stride of 32 bytes while hardware prefetchers and preemption
|
| 1146 | are disabled. This also provides a substitute visualization of cache
|
| 1147 | hits and misses.
|
| 1148 | 2) Cache hit and miss measurements using model specific precision counters if
|
| 1149 | available. Depending on the levels of cache on the system the pseudo_lock_l2
|
| 1150 | and pseudo_lock_l3 tracepoints are available.
|
| 1151 |
|
| 1152 | When a pseudo-locked region is created a new debugfs directory is created for
|
| 1153 | it in debugfs as /sys/kernel/debug/resctrl/<newdir>. A single
|
| 1154 | write-only file, pseudo_lock_measure, is present in this directory. The
|
| 1155 | measurement of the pseudo-locked region depends on the number written to this
|
| 1156 | debugfs file:
|
| 1157 |
|
| 1158 | 1:
|
| 1159 | writing "1" to the pseudo_lock_measure file will trigger the latency
|
| 1160 | measurement captured in the pseudo_lock_mem_latency tracepoint. See
|
| 1161 | example below.
|
| 1162 | 2:
|
| 1163 | writing "2" to the pseudo_lock_measure file will trigger the L2 cache
|
| 1164 | residency (cache hits and misses) measurement captured in the
|
| 1165 | pseudo_lock_l2 tracepoint. See example below.
|
| 1166 | 3:
|
| 1167 | writing "3" to the pseudo_lock_measure file will trigger the L3 cache
|
| 1168 | residency (cache hits and misses) measurement captured in the
|
| 1169 | pseudo_lock_l3 tracepoint.
|
| 1170 |
|
| 1171 | All measurements are recorded with the tracing infrastructure. This requires
|
| 1172 | the relevant tracepoints to be enabled before the measurement is triggered.
|
| 1173 |
|
| 1174 | Example of latency debugging interface
|
| 1175 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
| 1176 | In this example a pseudo-locked region named "newlock" was created. Here is
|
| 1177 | how we can measure the latency in cycles of reading from this region and
|
| 1178 | visualize this data with a histogram that is available if CONFIG_HIST_TRIGGERS
|
| 1179 | is set::
|
| 1180 |
|
| 1181 | # :> /sys/kernel/tracing/trace
|
| 1182 | # echo 'hist:keys=latency' > /sys/kernel/tracing/events/resctrl/pseudo_lock_mem_latency/trigger
|
| 1183 | # echo 1 > /sys/kernel/tracing/events/resctrl/pseudo_lock_mem_latency/enable
|
| 1184 | # echo 1 > /sys/kernel/debug/resctrl/newlock/pseudo_lock_measure
|
| 1185 | # echo 0 > /sys/kernel/tracing/events/resctrl/pseudo_lock_mem_latency/enable
|
| 1186 | # cat /sys/kernel/tracing/events/resctrl/pseudo_lock_mem_latency/hist
|
| 1187 |
|
| 1188 | # event histogram
|
| 1189 | #
|
| 1190 | # trigger info: hist:keys=latency:vals=hitcount:sort=hitcount:size=2048 [active]
|
| 1191 | #
|
| 1192 |
|
| 1193 | { latency: 456 } hitcount: 1
|
| 1194 | { latency: 50 } hitcount: 83
|
| 1195 | { latency: 36 } hitcount: 96
|
| 1196 | { latency: 44 } hitcount: 174
|
| 1197 | { latency: 48 } hitcount: 195
|
| 1198 | { latency: 46 } hitcount: 262
|
| 1199 | { latency: 42 } hitcount: 693
|
| 1200 | { latency: 40 } hitcount: 3204
|
| 1201 | { latency: 38 } hitcount: 3484
|
| 1202 |
|
| 1203 | Totals:
|
| 1204 | Hits: 8192
|
| 1205 | Entries: 9
|
| 1206 | Dropped: 0
|
| 1207 |
|
| 1208 | Example of cache hits/misses debugging
|
| 1209 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
| 1210 | In this example a pseudo-locked region named "newlock" was created on the L2
|
| 1211 | cache of a platform. Here is how we can obtain details of the cache hits
|
| 1212 | and misses using the platform's precision counters.
|
| 1213 | ::
|
| 1214 |
|
| 1215 | # :> /sys/kernel/tracing/trace
|
| 1216 | # echo 1 > /sys/kernel/tracing/events/resctrl/pseudo_lock_l2/enable
|
| 1217 | # echo 2 > /sys/kernel/debug/resctrl/newlock/pseudo_lock_measure
|
| 1218 | # echo 0 > /sys/kernel/tracing/events/resctrl/pseudo_lock_l2/enable
|
| 1219 | # cat /sys/kernel/tracing/trace
|
| 1220 |
|
| 1221 | # tracer: nop
|
| 1222 | #
|
| 1223 | # _-----=> irqs-off
|
| 1224 | # / _----=> need-resched
|
| 1225 | # | / _---=> hardirq/softirq
|
| 1226 | # || / _--=> preempt-depth
|
| 1227 | # ||| / delay
|
| 1228 | # TASK-PID CPU# |||| TIMESTAMP FUNCTION
|
| 1229 | # | | | |||| | |
|
| 1230 | pseudo_lock_mea-1672 [002] .... 3132.860500: pseudo_lock_l2: hits=4097 miss=0
|
| 1231 |
|
| 1232 |
|
| 1233 | Examples for RDT allocation usage
|
| 1234 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
| 1235 |
|
| 1236 | 1) Example 1
|
| 1237 |
|
| 1238 | On a two socket machine (one L3 cache per socket) with just four bits
|
| 1239 | for cache bit masks, minimum b/w of 10% with a memory bandwidth
|
| 1240 | granularity of 10%.
|
| 1241 | ::
|
| 1242 |
|
| 1243 | # mount -t resctrl resctrl /sys/fs/resctrl
|
| 1244 | # cd /sys/fs/resctrl
|
| 1245 | # mkdir p0 p1
|
| 1246 | # echo "L3:0=3;1=c\nMB:0=50;1=50" > /sys/fs/resctrl/p0/schemata
|
| 1247 | # echo "L3:0=3;1=3\nMB:0=50;1=50" > /sys/fs/resctrl/p1/schemata
|
| 1248 |
|
| 1249 | The default resource group is unmodified, so we have access to all parts
|
| 1250 | of all caches (its schemata file reads "L3:0=f;1=f").
|
| 1251 |
|
| 1252 | Tasks that are under the control of group "p0" may only allocate from the
|
| 1253 | "lower" 50% on cache ID 0, and the "upper" 50% of cache ID 1.
|
| 1254 | Tasks in group "p1" use the "lower" 50% of cache on both sockets.
|
| 1255 |
|
| 1256 | Similarly, tasks that are under the control of group "p0" may use a
|
| 1257 | maximum memory b/w of 50% on socket0 and 50% on socket 1.
|
| 1258 | Tasks in group "p1" may also use 50% memory b/w on both sockets.
|
| 1259 | Note that unlike cache masks, memory b/w cannot specify whether these
|
| 1260 | allocations can overlap or not. The allocations specifies the maximum
|
| 1261 | b/w that the group may be able to use and the system admin can configure
|
| 1262 | the b/w accordingly.
|
| 1263 |
|
| 1264 | If resctrl is using the software controller (mba_sc) then user can enter the
|
| 1265 | max b/w in MB rather than the percentage values.
|
| 1266 | ::
|
| 1267 |
|
| 1268 | # echo "L3:0=3;1=c\nMB:0=1024;1=500" > /sys/fs/resctrl/p0/schemata
|
| 1269 | # echo "L3:0=3;1=3\nMB:0=1024;1=500" > /sys/fs/resctrl/p1/schemata
|
| 1270 |
|
| 1271 | In the above example the tasks in "p1" and "p0" on socket 0 would use a max b/w
|
| 1272 | of 1024MB where as on socket 1 they would use 500MB.
|
| 1273 |
|
| 1274 | 2) Example 2
|
| 1275 |
|
| 1276 | Again two sockets, but this time with a more realistic 20-bit mask.
|
| 1277 |
|
| 1278 | Two real time tasks pid=1234 running on processor 0 and pid=5678 running on
|
| 1279 | processor 1 on socket 0 on a 2-socket and dual core machine. To avoid noisy
|
| 1280 | neighbors, each of the two real-time tasks exclusively occupies one quarter
|
| 1281 | of L3 cache on socket 0.
|
| 1282 | ::
|
| 1283 |
|
| 1284 | # mount -t resctrl resctrl /sys/fs/resctrl
|
| 1285 | # cd /sys/fs/resctrl
|
| 1286 |
|
| 1287 | First we reset the schemata for the default group so that the "upper"
|
| 1288 | 50% of the L3 cache on socket 0 and 50% of memory b/w cannot be used by
|
| 1289 | ordinary tasks::
|
| 1290 |
|
| 1291 | # echo "L3:0=3ff;1=fffff\nMB:0=50;1=100" > schemata
|
| 1292 |
|
| 1293 | Next we make a resource group for our first real time task and give
|
| 1294 | it access to the "top" 25% of the cache on socket 0.
|
| 1295 | ::
|
| 1296 |
|
| 1297 | # mkdir p0
|
| 1298 | # echo "L3:0=f8000;1=fffff" > p0/schemata
|
| 1299 |
|
| 1300 | Finally we move our first real time task into this resource group. We
|
| 1301 | also use taskset(1) to ensure the task always runs on a dedicated CPU
|
| 1302 | on socket 0. Most uses of resource groups will also constrain which
|
| 1303 | processors tasks run on.
|
| 1304 | ::
|
| 1305 |
|
| 1306 | # echo 1234 > p0/tasks
|
| 1307 | # taskset -cp 1 1234
|
| 1308 |
|
| 1309 | Ditto for the second real time task (with the remaining 25% of cache)::
|
| 1310 |
|
| 1311 | # mkdir p1
|
| 1312 | # echo "L3:0=7c00;1=fffff" > p1/schemata
|
| 1313 | # echo 5678 > p1/tasks
|
| 1314 | # taskset -cp 2 5678
|
| 1315 |
|
| 1316 | For the same 2 socket system with memory b/w resource and CAT L3 the
|
| 1317 | schemata would look like(Assume min_bandwidth 10 and bandwidth_gran is
|
| 1318 | 10):
|
| 1319 |
|
| 1320 | For our first real time task this would request 20% memory b/w on socket 0.
|
| 1321 | ::
|
| 1322 |
|
| 1323 | # echo -e "L3:0=f8000;1=fffff\nMB:0=20;1=100" > p0/schemata
|
| 1324 |
|
| 1325 | For our second real time task this would request an other 20% memory b/w
|
| 1326 | on socket 0.
|
| 1327 | ::
|
| 1328 |
|
| 1329 | # echo -e "L3:0=f8000;1=fffff\nMB:0=20;1=100" > p0/schemata
|
| 1330 |
|
| 1331 | 3) Example 3
|
| 1332 |
|
| 1333 | A single socket system which has real-time tasks running on core 4-7 and
|
| 1334 | non real-time workload assigned to core 0-3. The real-time tasks share text
|
| 1335 | and data, so a per task association is not required and due to interaction
|
| 1336 | with the kernel it's desired that the kernel on these cores shares L3 with
|
| 1337 | the tasks.
|
| 1338 | ::
|
| 1339 |
|
| 1340 | # mount -t resctrl resctrl /sys/fs/resctrl
|
| 1341 | # cd /sys/fs/resctrl
|
| 1342 |
|
| 1343 | First we reset the schemata for the default group so that the "upper"
|
| 1344 | 50% of the L3 cache on socket 0, and 50% of memory bandwidth on socket 0
|
| 1345 | cannot be used by ordinary tasks::
|
| 1346 |
|
| 1347 | # echo "L3:0=3ff\nMB:0=50" > schemata
|
| 1348 |
|
| 1349 | Next we make a resource group for our real time cores and give it access
|
| 1350 | to the "top" 50% of the cache on socket 0 and 50% of memory bandwidth on
|
| 1351 | socket 0.
|
| 1352 | ::
|
| 1353 |
|
| 1354 | # mkdir p0
|
| 1355 | # echo "L3:0=ffc00\nMB:0=50" > p0/schemata
|
| 1356 |
|
| 1357 | Finally we move core 4-7 over to the new group and make sure that the
|
| 1358 | kernel and the tasks running there get 50% of the cache. They should
|
| 1359 | also get 50% of memory bandwidth assuming that the cores 4-7 are SMT
|
| 1360 | siblings and only the real time threads are scheduled on the cores 4-7.
|
| 1361 | ::
|
| 1362 |
|
| 1363 | # echo F0 > p0/cpus
|
| 1364 |
|
| 1365 | 4) Example 4
|
| 1366 |
|
| 1367 | The resource groups in previous examples were all in the default "shareable"
|
| 1368 | mode allowing sharing of their cache allocations. If one resource group
|
| 1369 | configures a cache allocation then nothing prevents another resource group
|
| 1370 | to overlap with that allocation.
|
| 1371 |
|
| 1372 | In this example a new exclusive resource group will be created on a L2 CAT
|
| 1373 | system with two L2 cache instances that can be configured with an 8-bit
|
| 1374 | capacity bitmask. The new exclusive resource group will be configured to use
|
| 1375 | 25% of each cache instance.
|
| 1376 | ::
|
| 1377 |
|
| 1378 | # mount -t resctrl resctrl /sys/fs/resctrl/
|
| 1379 | # cd /sys/fs/resctrl
|
| 1380 |
|
| 1381 | First, we observe that the default group is configured to allocate to all L2
|
| 1382 | cache::
|
| 1383 |
|
| 1384 | # cat schemata
|
| 1385 | L2:0=ff;1=ff
|
| 1386 |
|
| 1387 | We could attempt to create the new resource group at this point, but it will
|
| 1388 | fail because of the overlap with the schemata of the default group::
|
| 1389 |
|
| 1390 | # mkdir p0
|
| 1391 | # echo 'L2:0=0x3;1=0x3' > p0/schemata
|
| 1392 | # cat p0/mode
|
| 1393 | shareable
|
| 1394 | # echo exclusive > p0/mode
|
| 1395 | -sh: echo: write error: Invalid argument
|
| 1396 | # cat info/last_cmd_status
|
| 1397 | schemata overlaps
|
| 1398 |
|
| 1399 | To ensure that there is no overlap with another resource group the default
|
| 1400 | resource group's schemata has to change, making it possible for the new
|
| 1401 | resource group to become exclusive.
|
| 1402 | ::
|
| 1403 |
|
| 1404 | # echo 'L2:0=0xfc;1=0xfc' > schemata
|
| 1405 | # echo exclusive > p0/mode
|
| 1406 | # grep . p0/*
|
| 1407 | p0/cpus:0
|
| 1408 | p0/mode:exclusive
|
| 1409 | p0/schemata:L2:0=03;1=03
|
| 1410 | p0/size:L2:0=262144;1=262144
|
| 1411 |
|
| 1412 | A new resource group will on creation not overlap with an exclusive resource
|
| 1413 | group::
|
| 1414 |
|
| 1415 | # mkdir p1
|
| 1416 | # grep . p1/*
|
| 1417 | p1/cpus:0
|
| 1418 | p1/mode:shareable
|
| 1419 | p1/schemata:L2:0=fc;1=fc
|
| 1420 | p1/size:L2:0=786432;1=786432
|
| 1421 |
|
| 1422 | The bit_usage will reflect how the cache is used::
|
| 1423 |
|
| 1424 | # cat info/L2/bit_usage
|
| 1425 | 0=SSSSSSEE;1=SSSSSSEE
|
| 1426 |
|
| 1427 | A resource group cannot be forced to overlap with an exclusive resource group::
|
| 1428 |
|
| 1429 | # echo 'L2:0=0x1;1=0x1' > p1/schemata
|
| 1430 | -sh: echo: write error: Invalid argument
|
| 1431 | # cat info/last_cmd_status
|
| 1432 | overlaps with exclusive group
|
| 1433 |
|
| 1434 | Example of Cache Pseudo-Locking
|
| 1435 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
| 1436 | Lock portion of L2 cache from cache id 1 using CBM 0x3. Pseudo-locked
|
| 1437 | region is exposed at /dev/pseudo_lock/newlock that can be provided to
|
| 1438 | application for argument to mmap().
|
| 1439 | ::
|
| 1440 |
|
| 1441 | # mount -t resctrl resctrl /sys/fs/resctrl/
|
| 1442 | # cd /sys/fs/resctrl
|
| 1443 |
|
| 1444 | Ensure that there are bits available that can be pseudo-locked, since only
|
| 1445 | unused bits can be pseudo-locked the bits to be pseudo-locked needs to be
|
| 1446 | removed from the default resource group's schemata::
|
| 1447 |
|
| 1448 | # cat info/L2/bit_usage
|
| 1449 | 0=SSSSSSSS;1=SSSSSSSS
|
| 1450 | # echo 'L2:1=0xfc' > schemata
|
| 1451 | # cat info/L2/bit_usage
|
| 1452 | 0=SSSSSSSS;1=SSSSSS00
|
| 1453 |
|
| 1454 | Create a new resource group that will be associated with the pseudo-locked
|
| 1455 | region, indicate that it will be used for a pseudo-locked region, and
|
| 1456 | configure the requested pseudo-locked region capacity bitmask::
|
| 1457 |
|
| 1458 | # mkdir newlock
|
| 1459 | # echo pseudo-locksetup > newlock/mode
|
| 1460 | # echo 'L2:1=0x3' > newlock/schemata
|
| 1461 |
|
| 1462 | On success the resource group's mode will change to pseudo-locked, the
|
| 1463 | bit_usage will reflect the pseudo-locked region, and the character device
|
| 1464 | exposing the pseudo-locked region will exist::
|
| 1465 |
|
| 1466 | # cat newlock/mode
|
| 1467 | pseudo-locked
|
| 1468 | # cat info/L2/bit_usage
|
| 1469 | 0=SSSSSSSS;1=SSSSSSPP
|
| 1470 | # ls -l /dev/pseudo_lock/newlock
|
| 1471 | crw------- 1 root root 243, 0 Apr 3 05:01 /dev/pseudo_lock/newlock
|
| 1472 |
|
| 1473 | ::
|
| 1474 |
|
| 1475 | /*
|
| 1476 | * Example code to access one page of pseudo-locked cache region
|
| 1477 | * from user space.
|
| 1478 | */
|
| 1479 | #define _GNU_SOURCE
|
| 1480 | #include <fcntl.h>
|
| 1481 | #include <sched.h>
|
| 1482 | #include <stdio.h>
|
| 1483 | #include <stdlib.h>
|
| 1484 | #include <unistd.h>
|
| 1485 | #include <sys/mman.h>
|
| 1486 |
|
| 1487 | /*
|
| 1488 | * It is required that the application runs with affinity to only
|
| 1489 | * cores associated with the pseudo-locked region. Here the cpu
|
| 1490 | * is hardcoded for convenience of example.
|
| 1491 | */
|
| 1492 | static int cpuid = 2;
|
| 1493 |
|
| 1494 | int main(int argc, char *argv[])
|
| 1495 | {
|
| 1496 | cpu_set_t cpuset;
|
| 1497 | long page_size;
|
| 1498 | void *mapping;
|
| 1499 | int dev_fd;
|
| 1500 | int ret;
|
| 1501 |
|
| 1502 | page_size = sysconf(_SC_PAGESIZE);
|
| 1503 |
|
| 1504 | CPU_ZERO(&cpuset);
|
| 1505 | CPU_SET(cpuid, &cpuset);
|
| 1506 | ret = sched_setaffinity(0, sizeof(cpuset), &cpuset);
|
| 1507 | if (ret < 0) {
|
| 1508 | perror("sched_setaffinity");
|
| 1509 | exit(EXIT_FAILURE);
|
| 1510 | }
|
| 1511 |
|
| 1512 | dev_fd = open("/dev/pseudo_lock/newlock", O_RDWR);
|
| 1513 | if (dev_fd < 0) {
|
| 1514 | perror("open");
|
| 1515 | exit(EXIT_FAILURE);
|
| 1516 | }
|
| 1517 |
|
| 1518 | mapping = mmap(0, page_size, PROT_READ | PROT_WRITE, MAP_SHARED,
|
| 1519 | dev_fd, 0);
|
| 1520 | if (mapping == MAP_FAILED) {
|
| 1521 | perror("mmap");
|
| 1522 | close(dev_fd);
|
| 1523 | exit(EXIT_FAILURE);
|
| 1524 | }
|
| 1525 |
|
| 1526 | /* Application interacts with pseudo-locked memory @mapping */
|
| 1527 |
|
| 1528 | ret = munmap(mapping, page_size);
|
| 1529 | if (ret < 0) {
|
| 1530 | perror("munmap");
|
| 1531 | close(dev_fd);
|
| 1532 | exit(EXIT_FAILURE);
|
| 1533 | }
|
| 1534 |
|
| 1535 | close(dev_fd);
|
| 1536 | exit(EXIT_SUCCESS);
|
| 1537 | }
|
| 1538 |
|
| 1539 | Locking between applications
|
| 1540 | ----------------------------
|
| 1541 |
|
| 1542 | Certain operations on the resctrl filesystem, composed of read/writes
|
| 1543 | to/from multiple files, must be atomic.
|
| 1544 |
|
| 1545 | As an example, the allocation of an exclusive reservation of L3 cache
|
| 1546 | involves:
|
| 1547 |
|
| 1548 | 1. Read the cbmmasks from each directory or the per-resource "bit_usage"
|
| 1549 | 2. Find a contiguous set of bits in the global CBM bitmask that is clear
|
| 1550 | in any of the directory cbmmasks
|
| 1551 | 3. Create a new directory
|
| 1552 | 4. Set the bits found in step 2 to the new directory "schemata" file
|
| 1553 |
|
| 1554 | If two applications attempt to allocate space concurrently then they can
|
| 1555 | end up allocating the same bits so the reservations are shared instead of
|
| 1556 | exclusive.
|
| 1557 |
|
| 1558 | To coordinate atomic operations on the resctrlfs and to avoid the problem
|
| 1559 | above, the following locking procedure is recommended:
|
| 1560 |
|
| 1561 | Locking is based on flock, which is available in libc and also as a shell
|
| 1562 | script command
|
| 1563 |
|
| 1564 | Write lock:
|
| 1565 |
|
| 1566 | A) Take flock(LOCK_EX) on /sys/fs/resctrl
|
| 1567 | B) Read/write the directory structure.
|
| 1568 | C) funlock
|
| 1569 |
|
| 1570 | Read lock:
|
| 1571 |
|
| 1572 | A) Take flock(LOCK_SH) on /sys/fs/resctrl
|
| 1573 | B) If success read the directory structure.
|
| 1574 | C) funlock
|
| 1575 |
|
| 1576 | Example with bash::
|
| 1577 |
|
| 1578 | # Atomically read directory structure
|
| 1579 | $ flock -s /sys/fs/resctrl/ find /sys/fs/resctrl
|
| 1580 |
|
| 1581 | # Read directory contents and create new subdirectory
|
| 1582 |
|
| 1583 | $ cat create-dir.sh
|
| 1584 | find /sys/fs/resctrl/ > output.txt
|
| 1585 | mask = function-of(output.txt)
|
| 1586 | mkdir /sys/fs/resctrl/newres/
|
| 1587 | echo mask > /sys/fs/resctrl/newres/schemata
|
| 1588 |
|
| 1589 | $ flock /sys/fs/resctrl/ ./create-dir.sh
|
| 1590 |
|
| 1591 | Example with C::
|
| 1592 |
|
| 1593 | /*
|
| 1594 | * Example code do take advisory locks
|
| 1595 | * before accessing resctrl filesystem
|
| 1596 | */
|
| 1597 | #include <sys/file.h>
|
| 1598 | #include <stdlib.h>
|
| 1599 |
|
| 1600 | void resctrl_take_shared_lock(int fd)
|
| 1601 | {
|
| 1602 | int ret;
|
| 1603 |
|
| 1604 | /* take shared lock on resctrl filesystem */
|
| 1605 | ret = flock(fd, LOCK_SH);
|
| 1606 | if (ret) {
|
| 1607 | perror("flock");
|
| 1608 | exit(-1);
|
| 1609 | }
|
| 1610 | }
|
| 1611 |
|
| 1612 | void resctrl_take_exclusive_lock(int fd)
|
| 1613 | {
|
| 1614 | int ret;
|
| 1615 |
|
| 1616 | /* release lock on resctrl filesystem */
|
| 1617 | ret = flock(fd, LOCK_EX);
|
| 1618 | if (ret) {
|
| 1619 | perror("flock");
|
| 1620 | exit(-1);
|
| 1621 | }
|
| 1622 | }
|
| 1623 |
|
| 1624 | void resctrl_release_lock(int fd)
|
| 1625 | {
|
| 1626 | int ret;
|
| 1627 |
|
| 1628 | /* take shared lock on resctrl filesystem */
|
| 1629 | ret = flock(fd, LOCK_UN);
|
| 1630 | if (ret) {
|
| 1631 | perror("flock");
|
| 1632 | exit(-1);
|
| 1633 | }
|
| 1634 | }
|
| 1635 |
|
| 1636 | void main(void)
|
| 1637 | {
|
| 1638 | int fd, ret;
|
| 1639 |
|
| 1640 | fd = open("/sys/fs/resctrl", O_DIRECTORY);
|
| 1641 | if (fd == -1) {
|
| 1642 | perror("open");
|
| 1643 | exit(-1);
|
| 1644 | }
|
| 1645 | resctrl_take_shared_lock(fd);
|
| 1646 | /* code to read directory contents */
|
| 1647 | resctrl_release_lock(fd);
|
| 1648 |
|
| 1649 | resctrl_take_exclusive_lock(fd);
|
| 1650 | /* code to read and write directory contents */
|
| 1651 | resctrl_release_lock(fd);
|
| 1652 | }
|
| 1653 |
|
| 1654 | Examples for RDT Monitoring along with allocation usage
|
| 1655 | =======================================================
|
| 1656 | Reading monitored data
|
| 1657 | ----------------------
|
| 1658 | Reading an event file (for ex: mon_data/mon_L3_00/llc_occupancy) would
|
| 1659 | show the current snapshot of LLC occupancy of the corresponding MON
|
| 1660 | group or CTRL_MON group.
|
| 1661 |
|
| 1662 |
|
| 1663 | Example 1 (Monitor CTRL_MON group and subset of tasks in CTRL_MON group)
|
| 1664 | ------------------------------------------------------------------------
|
| 1665 | On a two socket machine (one L3 cache per socket) with just four bits
|
| 1666 | for cache bit masks::
|
| 1667 |
|
| 1668 | # mount -t resctrl resctrl /sys/fs/resctrl
|
| 1669 | # cd /sys/fs/resctrl
|
| 1670 | # mkdir p0 p1
|
| 1671 | # echo "L3:0=3;1=c" > /sys/fs/resctrl/p0/schemata
|
| 1672 | # echo "L3:0=3;1=3" > /sys/fs/resctrl/p1/schemata
|
| 1673 | # echo 5678 > p1/tasks
|
| 1674 | # echo 5679 > p1/tasks
|
| 1675 |
|
| 1676 | The default resource group is unmodified, so we have access to all parts
|
| 1677 | of all caches (its schemata file reads "L3:0=f;1=f").
|
| 1678 |
|
| 1679 | Tasks that are under the control of group "p0" may only allocate from the
|
| 1680 | "lower" 50% on cache ID 0, and the "upper" 50% of cache ID 1.
|
| 1681 | Tasks in group "p1" use the "lower" 50% of cache on both sockets.
|
| 1682 |
|
| 1683 | Create monitor groups and assign a subset of tasks to each monitor group.
|
| 1684 | ::
|
| 1685 |
|
| 1686 | # cd /sys/fs/resctrl/p1/mon_groups
|
| 1687 | # mkdir m11 m12
|
| 1688 | # echo 5678 > m11/tasks
|
| 1689 | # echo 5679 > m12/tasks
|
| 1690 |
|
| 1691 | fetch data (data shown in bytes)
|
| 1692 | ::
|
| 1693 |
|
| 1694 | # cat m11/mon_data/mon_L3_00/llc_occupancy
|
| 1695 | 16234000
|
| 1696 | # cat m11/mon_data/mon_L3_01/llc_occupancy
|
| 1697 | 14789000
|
| 1698 | # cat m12/mon_data/mon_L3_00/llc_occupancy
|
| 1699 | 16789000
|
| 1700 |
|
| 1701 | The parent ctrl_mon group shows the aggregated data.
|
| 1702 | ::
|
| 1703 |
|
| 1704 | # cat /sys/fs/resctrl/p1/mon_data/mon_l3_00/llc_occupancy
|
| 1705 | 31234000
|
| 1706 |
|
| 1707 | Example 2 (Monitor a task from its creation)
|
| 1708 | --------------------------------------------
|
| 1709 | On a two socket machine (one L3 cache per socket)::
|
| 1710 |
|
| 1711 | # mount -t resctrl resctrl /sys/fs/resctrl
|
| 1712 | # cd /sys/fs/resctrl
|
| 1713 | # mkdir p0 p1
|
| 1714 |
|
| 1715 | An RMID is allocated to the group once its created and hence the <cmd>
|
| 1716 | below is monitored from its creation.
|
| 1717 | ::
|
| 1718 |
|
| 1719 | # echo $$ > /sys/fs/resctrl/p1/tasks
|
| 1720 | # <cmd>
|
| 1721 |
|
| 1722 | Fetch the data::
|
| 1723 |
|
| 1724 | # cat /sys/fs/resctrl/p1/mon_data/mon_l3_00/llc_occupancy
|
| 1725 | 31789000
|
| 1726 |
|
| 1727 | Example 3 (Monitor without CAT support or before creating CAT groups)
|
| 1728 | ---------------------------------------------------------------------
|
| 1729 |
|
| 1730 | Assume a system like HSW has only CQM and no CAT support. In this case
|
| 1731 | the resctrl will still mount but cannot create CTRL_MON directories.
|
| 1732 | But user can create different MON groups within the root group thereby
|
| 1733 | able to monitor all tasks including kernel threads.
|
| 1734 |
|
| 1735 | This can also be used to profile jobs cache size footprint before being
|
| 1736 | able to allocate them to different allocation groups.
|
| 1737 | ::
|
| 1738 |
|
| 1739 | # mount -t resctrl resctrl /sys/fs/resctrl
|
| 1740 | # cd /sys/fs/resctrl
|
| 1741 | # mkdir mon_groups/m01
|
| 1742 | # mkdir mon_groups/m02
|
| 1743 |
|
| 1744 | # echo 3478 > /sys/fs/resctrl/mon_groups/m01/tasks
|
| 1745 | # echo 2467 > /sys/fs/resctrl/mon_groups/m02/tasks
|
| 1746 |
|
| 1747 | Monitor the groups separately and also get per domain data. From the
|
| 1748 | below its apparent that the tasks are mostly doing work on
|
| 1749 | domain(socket) 0.
|
| 1750 | ::
|
| 1751 |
|
| 1752 | # cat /sys/fs/resctrl/mon_groups/m01/mon_L3_00/llc_occupancy
|
| 1753 | 31234000
|
| 1754 | # cat /sys/fs/resctrl/mon_groups/m01/mon_L3_01/llc_occupancy
|
| 1755 | 34555
|
| 1756 | # cat /sys/fs/resctrl/mon_groups/m02/mon_L3_00/llc_occupancy
|
| 1757 | 31234000
|
| 1758 | # cat /sys/fs/resctrl/mon_groups/m02/mon_L3_01/llc_occupancy
|
| 1759 | 32789
|
| 1760 |
|
| 1761 |
|
| 1762 | Example 4 (Monitor real time tasks)
|
| 1763 | -----------------------------------
|
| 1764 |
|
| 1765 | A single socket system which has real time tasks running on cores 4-7
|
| 1766 | and non real time tasks on other cpus. We want to monitor the cache
|
| 1767 | occupancy of the real time threads on these cores.
|
| 1768 | ::
|
| 1769 |
|
| 1770 | # mount -t resctrl resctrl /sys/fs/resctrl
|
| 1771 | # cd /sys/fs/resctrl
|
| 1772 | # mkdir p1
|
| 1773 |
|
| 1774 | Move the cpus 4-7 over to p1::
|
| 1775 |
|
| 1776 | # echo f0 > p1/cpus
|
| 1777 |
|
| 1778 | View the llc occupancy snapshot::
|
| 1779 |
|
| 1780 | # cat /sys/fs/resctrl/p1/mon_data/mon_L3_00/llc_occupancy
|
| 1781 | 11234000
|
| 1782 |
|
| 1783 |
|
| 1784 | Examples on working with mbm_assign_mode
|
| 1785 | ========================================
|
| 1786 |
|
| 1787 | a. Check if MBM counter assignment mode is supported.
|
| 1788 | ::
|
| 1789 |
|
| 1790 | # mount -t resctrl resctrl /sys/fs/resctrl/
|
| 1791 |
|
| 1792 | # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
|
| 1793 | [mbm_event]
|
| 1794 | default
|
| 1795 |
|
| 1796 | The "mbm_event" mode is detected and enabled.
|
| 1797 |
|
| 1798 | b. Check how many assignable counters are supported.
|
| 1799 | ::
|
| 1800 |
|
| 1801 | # cat /sys/fs/resctrl/info/L3_MON/num_mbm_cntrs
|
| 1802 | 0=32;1=32
|
| 1803 |
|
| 1804 | c. Check how many assignable counters are available for assignment in each domain.
|
| 1805 | ::
|
| 1806 |
|
| 1807 | # cat /sys/fs/resctrl/info/L3_MON/available_mbm_cntrs
|
| 1808 | 0=30;1=30
|
| 1809 |
|
| 1810 | d. To list the default group's assign states.
|
| 1811 | ::
|
| 1812 |
|
| 1813 | # cat /sys/fs/resctrl/mbm_L3_assignments
|
| 1814 | mbm_total_bytes:0=e;1=e
|
| 1815 | mbm_local_bytes:0=e;1=e
|
| 1816 |
|
| 1817 | e. To unassign the counter associated with the mbm_total_bytes event on domain 0.
|
| 1818 | ::
|
| 1819 |
|
| 1820 | # echo "mbm_total_bytes:0=_" > /sys/fs/resctrl/mbm_L3_assignments
|
| 1821 | # cat /sys/fs/resctrl/mbm_L3_assignments
|
| 1822 | mbm_total_bytes:0=_;1=e
|
| 1823 | mbm_local_bytes:0=e;1=e
|
| 1824 |
|
| 1825 | f. To unassign the counter associated with the mbm_total_bytes event on all domains.
|
| 1826 | ::
|
| 1827 |
|
| 1828 | # echo "mbm_total_bytes:*=_" > /sys/fs/resctrl/mbm_L3_assignments
|
| 1829 | # cat /sys/fs/resctrl/mbm_L3_assignment
|
| 1830 | mbm_total_bytes:0=_;1=_
|
| 1831 | mbm_local_bytes:0=e;1=e
|
| 1832 |
|
| 1833 | g. To assign a counter associated with the mbm_total_bytes event on all domains in
|
| 1834 | exclusive mode.
|
| 1835 | ::
|
| 1836 |
|
| 1837 | # echo "mbm_total_bytes:*=e" > /sys/fs/resctrl/mbm_L3_assignments
|
| 1838 | # cat /sys/fs/resctrl/mbm_L3_assignments
|
| 1839 | mbm_total_bytes:0=e;1=e
|
| 1840 | mbm_local_bytes:0=e;1=e
|
| 1841 |
|
| 1842 | h. Read the events mbm_total_bytes and mbm_local_bytes of the default group. There is
|
| 1843 | no change in reading the events with the assignment.
|
| 1844 | ::
|
| 1845 |
|
| 1846 | # cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_total_bytes
|
| 1847 | 779247936
|
| 1848 | # cat /sys/fs/resctrl/mon_data/mon_L3_01/mbm_total_bytes
|
| 1849 | 562324232
|
| 1850 | # cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_local_bytes
|
| 1851 | 212122123
|
| 1852 | # cat /sys/fs/resctrl/mon_data/mon_L3_01/mbm_local_bytes
|
| 1853 | 121212144
|
| 1854 |
|
| 1855 | i. Check the event configurations.
|
| 1856 | ::
|
| 1857 |
|
| 1858 | # cat /sys/fs/resctrl/info/L3_MON/event_configs/mbm_total_bytes/event_filter
|
| 1859 | local_reads,remote_reads,local_non_temporal_writes,remote_non_temporal_writes,
|
| 1860 | local_reads_slow_memory,remote_reads_slow_memory,dirty_victim_writes_all
|
| 1861 |
|
| 1862 | # cat /sys/fs/resctrl/info/L3_MON/event_configs/mbm_local_bytes/event_filter
|
| 1863 | local_reads,local_non_temporal_writes,local_reads_slow_memory
|
| 1864 |
|
| 1865 | j. Change the event configuration for mbm_local_bytes.
|
| 1866 | ::
|
| 1867 |
|
| 1868 | # echo "local_reads, local_non_temporal_writes, local_reads_slow_memory, remote_reads" >
|
| 1869 | /sys/fs/resctrl/info/L3_MON/event_configs/mbm_local_bytes/event_filter
|
| 1870 |
|
| 1871 | # cat /sys/fs/resctrl/info/L3_MON/event_configs/mbm_local_bytes/event_filter
|
| 1872 | local_reads,local_non_temporal_writes,local_reads_slow_memory,remote_reads
|
| 1873 |
|
| 1874 | k. Now read the local events again. The first read may come back with "Unavailable"
|
| 1875 | status. The subsequent read of mbm_local_bytes will display the current value.
|
| 1876 | ::
|
| 1877 |
|
| 1878 | # cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_local_bytes
|
| 1879 | Unavailable
|
| 1880 | # cat /sys/fs/resctrl/mon_data/mon_L3_00/mbm_local_bytes
|
| 1881 | 2252323
|
| 1882 | # cat /sys/fs/resctrl/mon_data/mon_L3_01/mbm_local_bytes
|
| 1883 | Unavailable
|
| 1884 | # cat /sys/fs/resctrl/mon_data/mon_L3_01/mbm_local_bytes
|
| 1885 | 1566565
|
| 1886 |
|
| 1887 | l. Users have the option to go back to 'default' mbm_assign_mode if required. This can be
|
| 1888 | done using the following command. Note that switching the mbm_assign_mode may reset all
|
| 1889 | the MBM counters (and thus all MBM events) of all the resctrl groups.
|
| 1890 | ::
|
| 1891 |
|
| 1892 | # echo "default" > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
|
| 1893 | # cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode
|
| 1894 | mbm_event
|
| 1895 | [default]
|
| 1896 |
|
| 1897 | m. Unmount the resctrl filesystem.
|
| 1898 | ::
|
| 1899 |
|
| 1900 | # umount /sys/fs/resctrl/
|
| 1901 |
|
| 1902 | Intel RDT Errata
|
| 1903 | ================
|
| 1904 |
|
| 1905 | Intel MBM Counters May Report System Memory Bandwidth Incorrectly
|
| 1906 | -----------------------------------------------------------------
|
| 1907 |
|
| 1908 | Errata SKX99 for Skylake server and BDF102 for Broadwell server.
|
| 1909 |
|
| 1910 | Problem: Intel Memory Bandwidth Monitoring (MBM) counters track metrics
|
| 1911 | according to the assigned Resource Monitor ID (RMID) for that logical
|
| 1912 | core. The IA32_QM_CTR register (MSR 0xC8E), used to report these
|
| 1913 | metrics, may report incorrect system bandwidth for certain RMID values.
|
| 1914 |
|
| 1915 | Implication: Due to the errata, system memory bandwidth may not match
|
| 1916 | what is reported.
|
| 1917 |
|
| 1918 | Workaround: MBM total and local readings are corrected according to the
|
| 1919 | following correction factor table:
|
| 1920 |
|
| 1921 | +---------------+---------------+---------------+-----------------+
|
| 1922 | |core count |rmid count |rmid threshold |correction factor|
|
| 1923 | +---------------+---------------+---------------+-----------------+
|
| 1924 | |1 |8 |0 |1.000000 |
|
| 1925 | +---------------+---------------+---------------+-----------------+
|
| 1926 | |2 |16 |0 |1.000000 |
|
| 1927 | +---------------+---------------+---------------+-----------------+
|
| 1928 | |3 |24 |15 |0.969650 |
|
| 1929 | +---------------+---------------+---------------+-----------------+
|
| 1930 | |4 |32 |0 |1.000000 |
|
| 1931 | +---------------+---------------+---------------+-----------------+
|
| 1932 | |6 |48 |31 |0.969650 |
|
| 1933 | +---------------+---------------+---------------+-----------------+
|
| 1934 | |7 |56 |47 |1.142857 |
|
| 1935 | +---------------+---------------+---------------+-----------------+
|
| 1936 | |8 |64 |0 |1.000000 |
|
| 1937 | +---------------+---------------+---------------+-----------------+
|
| 1938 | |9 |72 |63 |1.185115 |
|
| 1939 | +---------------+---------------+---------------+-----------------+
|
| 1940 | |10 |80 |63 |1.066553 |
|
| 1941 | +---------------+---------------+---------------+-----------------+
|
| 1942 | |11 |88 |79 |1.454545 |
|
| 1943 | +---------------+---------------+---------------+-----------------+
|
| 1944 | |12 |96 |0 |1.000000 |
|
| 1945 | +---------------+---------------+---------------+-----------------+
|
| 1946 | |13 |104 |95 |1.230769 |
|
| 1947 | +---------------+---------------+---------------+-----------------+
|
| 1948 | |14 |112 |95 |1.142857 |
|
| 1949 | +---------------+---------------+---------------+-----------------+
|
| 1950 | |15 |120 |95 |1.066667 |
|
| 1951 | +---------------+---------------+---------------+-----------------+
|
| 1952 | |16 |128 |0 |1.000000 |
|
| 1953 | +---------------+---------------+---------------+-----------------+
|
| 1954 | |17 |136 |127 |1.254863 |
|
| 1955 | +---------------+---------------+---------------+-----------------+
|
| 1956 | |18 |144 |127 |1.185255 |
|
| 1957 | +---------------+---------------+---------------+-----------------+
|
| 1958 | |19 |152 |0 |1.000000 |
|
| 1959 | +---------------+---------------+---------------+-----------------+
|
| 1960 | |20 |160 |127 |1.066667 |
|
| 1961 | +---------------+---------------+---------------+-----------------+
|
| 1962 | |21 |168 |0 |1.000000 |
|
| 1963 | +---------------+---------------+---------------+-----------------+
|
| 1964 | |22 |176 |159 |1.454334 |
|
| 1965 | +---------------+---------------+---------------+-----------------+
|
| 1966 | |23 |184 |0 |1.000000 |
|
| 1967 | +---------------+---------------+---------------+-----------------+
|
| 1968 | |24 |192 |127 |0.969744 |
|
| 1969 | +---------------+---------------+---------------+-----------------+
|
| 1970 | |25 |200 |191 |1.280246 |
|
| 1971 | +---------------+---------------+---------------+-----------------+
|
| 1972 | |26 |208 |191 |1.230921 |
|
| 1973 | +---------------+---------------+---------------+-----------------+
|
| 1974 | |27 |216 |0 |1.000000 |
|
| 1975 | +---------------+---------------+---------------+-----------------+
|
| 1976 | |28 |224 |191 |1.143118 |
|
| 1977 | +---------------+---------------+---------------+-----------------+
|
| 1978 |
|
| 1979 | If rmid > rmid threshold, MBM total and local values should be multiplied
|
| 1980 | by the correction factor.
|
| 1981 |
|
| 1982 | See:
|
| 1983 |
|
| 1984 | 1. Erratum SKX99 in Intel Xeon Processor Scalable Family Specification Update:
|
| 1985 | http://web.archive.org/web/20200716124958/https://www.intel.com/content/www/us/en/processors/xeon/scalable/xeon-scalable-spec-update.html
|
| 1986 |
|
| 1987 | 2. Erratum BDF102 in Intel Xeon E5-2600 v4 Processor Product Family Specification Update:
|
| 1988 | http://web.archive.org/web/20191125200531/https://www.intel.com/content/dam/www/public/us/en/documents/specification-updates/xeon-e5-v4-spec-update.pdf
|
| 1989 |
|
| 1990 | 3. The errata in Intel Resource Director Technology (Intel RDT) on 2nd Generation Intel Xeon Scalable Processors Reference Manual:
|
| 1991 | https://software.intel.com/content/www/us/en/develop/articles/intel-resource-director-technology-rdt-reference-manual.html
|
| 1992 |
|
| 1993 | for further information.
|