| 1 | |
| 2 |
|
| 3 | ====================
|
| 4 | mlx5 devlink support
|
| 5 | ====================
|
| 6 |
|
| 7 | This document describes the devlink features implemented by the ``mlx5``
|
| 8 | device driver.
|
| 9 |
|
| 10 | Parameters
|
| 11 | ==========
|
| 12 |
|
| 13 | .. list-table:: Generic parameters implemented
|
| 14 |
|
| 15 | * - Name
|
| 16 | - Mode
|
| 17 | - Validation
|
| 18 | - Notes
|
| 19 | * - ``enable_roce``
|
| 20 | - driverinit
|
| 21 | - Boolean
|
| 22 | - If the device supports RoCE disablement, RoCE enablement state controls
|
| 23 | device support for RoCE capability. Otherwise, the control occurs in the
|
| 24 | driver stack. When RoCE is disabled at the driver level, only raw
|
| 25 | ethernet QPs are supported.
|
| 26 | * - ``io_eq_size``
|
| 27 | - driverinit
|
| 28 | - The range is between 64 and 4096.
|
| 29 | -
|
| 30 | * - ``event_eq_size``
|
| 31 | - driverinit
|
| 32 | - The range is between 64 and 4096.
|
| 33 | -
|
| 34 | * - ``max_macs``
|
| 35 | - driverinit
|
| 36 | - The range is between 1 and 2^31. Only power of 2 values are supported.
|
| 37 | -
|
| 38 | * - ``enable_sriov``
|
| 39 | - permanent
|
| 40 | - Boolean
|
| 41 | - Applies to each physical function (PF) independently, if the device
|
| 42 | supports it. Otherwise, it applies symmetrically to all PFs.
|
| 43 | * - ``total_vfs``
|
| 44 | - permanent
|
| 45 | - The range is between 1 and a device-specific max.
|
| 46 | - Applies to each physical function (PF) independently, if the device
|
| 47 | supports it. Otherwise, it applies symmetrically to all PFs.
|
| 48 | * - ``max_sfs``
|
| 49 | - permanent
|
| 50 | - The range is between 0 and a device-specific max.
|
| 51 | - Applies to each physical function (PF) independently.
|
| 52 |
|
| 53 | Note: permanent parameters such as ``enable_sriov``, ``total_vfs`` and ``max_sfs``
|
| 54 | require FW reset to take effect
|
| 55 |
|
| 56 | .. code-block:: bash
|
| 57 |
|
| 58 | # setup parameters
|
| 59 | devlink dev param set pci/0000:01:00.0 name enable_sriov value true cmode permanent
|
| 60 | devlink dev param set pci/0000:01:00.0 name total_vfs value 8 cmode permanent
|
| 61 |
|
| 62 | # Fw reset
|
| 63 | devlink dev reload pci/0000:01:00.0 action fw_activate
|
| 64 |
|
| 65 | # for PCI related config such as sriov PCI reset/rescan is required:
|
| 66 | echo 1 >/sys/bus/pci/devices/0000:01:00.0/remove
|
| 67 | echo 1 >/sys/bus/pci/rescan
|
| 68 | grep ^ /sys/bus/pci/devices/0000:01:00.0/sriov_*
|
| 69 |
|
| 70 | * - ``num_doorbells``
|
| 71 | - driverinit
|
| 72 | - This controls the number of channel doorbells used by the netdev. In all
|
| 73 | cases, an additional doorbell is allocated and used for non-channel
|
| 74 | communication (e.g. for PTP, HWS, etc.). Supported values are:
|
| 75 |
|
| 76 | - 0: No channel-specific doorbells, use the global one for everything.
|
| 77 | - [1, max_num_channels]: Spread netdev channels equally across these
|
| 78 | doorbells.
|
| 79 |
|
| 80 | The ``mlx5`` driver also implements the following driver-specific
|
| 81 | parameters.
|
| 82 |
|
| 83 | .. list-table:: Driver-specific parameters implemented
|
| 84 | :widths: 5 5 5 85
|
| 85 |
|
| 86 | * - Name
|
| 87 | - Type
|
| 88 | - Mode
|
| 89 | - Description
|
| 90 | * - ``flow_steering_mode``
|
| 91 | - string
|
| 92 | - runtime
|
| 93 | - Controls the flow steering mode of the driver
|
| 94 |
|
| 95 | * ``dmfs`` Device managed flow steering. In DMFS mode, the HW
|
| 96 | steering entities are created and managed through firmware.
|
| 97 | * ``smfs`` Software managed flow steering. In SMFS mode, the HW
|
| 98 | steering entities are created and manage through the driver without
|
| 99 | firmware intervention.
|
| 100 | * ``hmfs`` Hardware managed flow steering. In HMFS mode, the driver
|
| 101 | is configuring steering rules directly to the HW using Work Queues with
|
| 102 | a special new type of WQE (Work Queue Element).
|
| 103 |
|
| 104 | SMFS mode is faster and provides better rule insertion rate compared to
|
| 105 | default DMFS mode.
|
| 106 | * - ``fdb_large_groups``
|
| 107 | - u32
|
| 108 | - driverinit
|
| 109 | - Control the number of large groups (size > 1) in the FDB table.
|
| 110 |
|
| 111 | * The default value is 15, and the range is between 1 and 1024.
|
| 112 | * - ``esw_multiport``
|
| 113 | - Boolean
|
| 114 | - runtime
|
| 115 | - Control MultiPort E-Switch shared fdb mode.
|
| 116 |
|
| 117 | An experimental mode where a single E-Switch is used and all the vports
|
| 118 | and physical ports on the NIC are connected to it.
|
| 119 |
|
| 120 | An example is to send traffic from a VF that is created on PF0 to an
|
| 121 | uplink that is natively associated with the uplink of PF1
|
| 122 |
|
| 123 | Note: Future devices, ConnectX-8 and onward, will eventually have this
|
| 124 | as the default to allow forwarding between all NIC ports in a single
|
| 125 | E-switch environment and the dual E-switch mode will likely get
|
| 126 | deprecated.
|
| 127 |
|
| 128 | Default: disabled
|
| 129 | * - ``esw_port_metadata``
|
| 130 | - Boolean
|
| 131 | - runtime
|
| 132 | - When applicable, disabling eswitch metadata can increase packet rate up
|
| 133 | to 20% depending on the use case and packet sizes.
|
| 134 |
|
| 135 | Eswitch port metadata state controls whether to internally tag packets
|
| 136 | with metadata. Metadata tagging must be enabled for multi-port RoCE,
|
| 137 | failover between representors and stacked devices. By default metadata is
|
| 138 | enabled on the supported devices in E-switch. Metadata is applicable only
|
| 139 | for E-switch in switchdev mode and users may disable it when NONE of the
|
| 140 | below use cases will be in use:
|
| 141 | 1. HCA is in Dual/multi-port RoCE mode.
|
| 142 | 2. VF/SF representor bonding (Usually used for Live migration)
|
| 143 | 3. Stacked devices
|
| 144 |
|
| 145 | When metadata is disabled, the above use cases will fail to initialize if
|
| 146 | users try to enable them.
|
| 147 |
|
| 148 | Note: Setting this parameter does not take effect immediately. Setting
|
| 149 | must happen in legacy mode and eswitch port metadata takes effect after
|
| 150 | enabling switchdev mode.
|
| 151 | * - ``hairpin_num_queues``
|
| 152 | - u32
|
| 153 | - driverinit
|
| 154 | - We refer to a TC NIC rule that involves forwarding as "hairpin".
|
| 155 | Hairpin queues are mlx5 hardware specific implementation for hardware
|
| 156 | forwarding of such packets.
|
| 157 |
|
| 158 | Control the number of hairpin queues.
|
| 159 | * - ``hairpin_queue_size``
|
| 160 | - u32
|
| 161 | - driverinit
|
| 162 | - Control the size (in packets) of the hairpin queues.
|
| 163 | * - ``pcie_cong_inbound_high``
|
| 164 | - u16
|
| 165 | - driverinit
|
| 166 | - High threshold configuration for PCIe congestion events. The firmware
|
| 167 | will send an event once device side inbound PCIe traffic went
|
| 168 | above the configured high threshold for a long enough period (at least
|
| 169 | 200ms).
|
| 170 |
|
| 171 | See pci_bw_inbound_high ethtool stat.
|
| 172 |
|
| 173 | Units are 0.01 %. Accepted values are in range [0, 10000].
|
| 174 | pcie_cong_inbound_low < pcie_cong_inbound_high.
|
| 175 | Default value: 9000 (Corresponds to 90%).
|
| 176 | * - ``pcie_cong_inbound_low``
|
| 177 | - u16
|
| 178 | - driverinit
|
| 179 | - Low threshold configuration for PCIe congestion events. The firmware
|
| 180 | will send an event once device side inbound PCIe traffic went
|
| 181 | below the configured low threshold, only after having been previously in
|
| 182 | a congested state.
|
| 183 |
|
| 184 | See pci_bw_inbound_low ethtool stat.
|
| 185 |
|
| 186 | Units are 0.01 %. Accepted values are in range [0, 10000].
|
| 187 | pcie_cong_inbound_low < pcie_cong_inbound_high.
|
| 188 | Default value: 7500.
|
| 189 | * - ``pcie_cong_outbound_high``
|
| 190 | - u16
|
| 191 | - driverinit
|
| 192 | - High threshold configuration for PCIe congestion events. The firmware
|
| 193 | will send an event once device side outbound PCIe traffic went
|
| 194 | above the configured high threshold for a long enough period (at least
|
| 195 | 200ms).
|
| 196 |
|
| 197 | See pci_bw_outbound_high ethtool stat.
|
| 198 |
|
| 199 | Units are 0.01 %. Accepted values are in range [0, 10000].
|
| 200 | pcie_cong_outbound_low < pcie_cong_outbound_high.
|
| 201 | Default value: 9000 (Corresponds to 90%).
|
| 202 | * - ``pcie_cong_outbound_low``
|
| 203 | - u16
|
| 204 | - driverinit
|
| 205 | - Low threshold configuration for PCIe congestion events. The firmware
|
| 206 | will send an event once device side outbound PCIe traffic went
|
| 207 | below the configured low threshold, only after having been previously in
|
| 208 | a congested state.
|
| 209 |
|
| 210 | See pci_bw_outbound_low ethtool stat.
|
| 211 |
|
| 212 | Units are 0.01 %. Accepted values are in range [0, 10000].
|
| 213 | pcie_cong_outbound_low < pcie_cong_outbound_high.
|
| 214 | Default value: 7500.
|
| 215 |
|
| 216 | * - ``cqe_compress_type``
|
| 217 | - string
|
| 218 | - permanent
|
| 219 | - Configure which mechanism/algorithm should be used by the NIC that will
|
| 220 | affect the rate (aggressiveness) of compressed CQEs depending on PCIe bus
|
| 221 | conditions and other internal NIC factors. This mode affects all queues
|
| 222 | that enable compression.
|
| 223 | * ``balanced`` : Merges fewer CQEs, resulting in a moderate compression ratio but maintaining a balance between bandwidth savings and performance
|
| 224 | * ``aggressive`` : Merges more CQEs into a single entry, achieving a higher compression rate and maximizing performance, particularly under high traffic loads
|
| 225 |
|
| 226 | * - ``swp_l4_csum_mode``
|
| 227 | - string
|
| 228 | - permanent
|
| 229 | - Configure how the L4 checksum is calculated by the device when using
|
| 230 | Software Parser (SWP) hints for header locations.
|
| 231 |
|
| 232 | * ``default`` : Use the device's default checksum calculation
|
| 233 | mode. The driver will discover during init whether or
|
| 234 | full_csum or l4_only is in use. Setting this value explicitly
|
| 235 | from userspace is not allowed, but some firmware versions may
|
| 236 | return this value on param read.
|
| 237 | * ``full_csum`` : Calculate full checksum including the pseudo-header
|
| 238 | * ``l4_only`` : Calculate L4-only checksum, excluding the pseudo-header
|
| 239 |
|
| 240 | The ``mlx5`` driver supports reloading via ``DEVLINK_CMD_RELOAD``
|
| 241 |
|
| 242 | Info versions
|
| 243 | =============
|
| 244 |
|
| 245 | The ``mlx5`` driver reports the following versions
|
| 246 |
|
| 247 | .. list-table:: devlink info versions implemented
|
| 248 | :widths: 5 5 90
|
| 249 |
|
| 250 | * - Name
|
| 251 | - Type
|
| 252 | - Description
|
| 253 | * - ``fw.psid``
|
| 254 | - fixed
|
| 255 | - Used to represent the board id of the device.
|
| 256 | * - ``fw.version``
|
| 257 | - stored, running
|
| 258 | - Three digit major.minor.subminor firmware version number.
|
| 259 |
|
| 260 | Health reporters
|
| 261 | ================
|
| 262 |
|
| 263 | tx reporter
|
| 264 | -----------
|
| 265 | The tx reporter is responsible for reporting and recovering of the following three error scenarios:
|
| 266 |
|
| 267 | - tx timeout
|
| 268 | Report on kernel tx timeout detection.
|
| 269 | Recover by searching lost interrupts.
|
| 270 | - tx error completion
|
| 271 | Report on error tx completion.
|
| 272 | Recover by flushing the tx queue and reset it.
|
| 273 | - tx PTP port timestamping CQ unhealthy
|
| 274 | Report too many CQEs never delivered on port ts CQ.
|
| 275 | Recover by flushing and re-creating all PTP channels.
|
| 276 |
|
| 277 | tx reporter also support on demand diagnose callback, on which it provides
|
| 278 | real time information of its send queues status.
|
| 279 |
|
| 280 | User commands examples:
|
| 281 |
|
| 282 | - Diagnose send queues status::
|
| 283 |
|
| 284 | $ devlink health diagnose pci/0000:82:00.0 reporter tx
|
| 285 |
|
| 286 | .. note::
|
| 287 | This command has valid output only when interface is up, otherwise the command has empty output.
|
| 288 |
|
| 289 | - Show number of tx errors indicated, number of recover flows ended successfully,
|
| 290 | is autorecover enabled and graceful period from last recover::
|
| 291 |
|
| 292 | $ devlink health show pci/0000:82:00.0 reporter tx
|
| 293 |
|
| 294 | rx reporter
|
| 295 | -----------
|
| 296 | The rx reporter is responsible for reporting and recovering of the following two error scenarios:
|
| 297 |
|
| 298 | - rx queues' initialization (population) timeout
|
| 299 | Population of rx queues' descriptors on ring initialization is done
|
| 300 | in napi context via triggering an irq. In case of a failure to get
|
| 301 | the minimum amount of descriptors, a timeout would occur, and
|
| 302 | descriptors could be recovered by polling the EQ (Event Queue).
|
| 303 | - rx completions with errors (reported by HW on interrupt context)
|
| 304 | Report on rx completion error.
|
| 305 | Recover (if needed) by flushing the related queue and reset it.
|
| 306 |
|
| 307 | rx reporter also supports on demand diagnose callback, on which it
|
| 308 | provides real time information of its receive queues' status.
|
| 309 |
|
| 310 | - Diagnose rx queues' status and corresponding completion queue::
|
| 311 |
|
| 312 | $ devlink health diagnose pci/0000:82:00.0 reporter rx
|
| 313 |
|
| 314 | .. note::
|
| 315 | This command has valid output only when interface is up. Otherwise, the command has empty output.
|
| 316 |
|
| 317 | - Show number of rx errors indicated, number of recover flows ended successfully,
|
| 318 | is autorecover enabled, and graceful period from last recover::
|
| 319 |
|
| 320 | $ devlink health show pci/0000:82:00.0 reporter rx
|
| 321 |
|
| 322 | fw reporter
|
| 323 | -----------
|
| 324 | The fw reporter implements `diagnose` and `dump` callbacks.
|
| 325 | It follows symptoms of fw error such as fw syndrome by triggering
|
| 326 | fw core dump and storing it into the dump buffer.
|
| 327 | The fw reporter diagnose command can be triggered any time by the user to check
|
| 328 | current fw status.
|
| 329 |
|
| 330 | User commands examples:
|
| 331 |
|
| 332 | - Check fw heath status::
|
| 333 |
|
| 334 | $ devlink health diagnose pci/0000:82:00.0 reporter fw
|
| 335 |
|
| 336 | - Read FW core dump if already stored or trigger new one::
|
| 337 |
|
| 338 | $ devlink health dump show pci/0000:82:00.0 reporter fw
|
| 339 |
|
| 340 | .. note::
|
| 341 | This command can run only on the PF which has fw tracer ownership,
|
| 342 | running it on other PF or any VF will return "Operation not permitted".
|
| 343 |
|
| 344 | fw fatal reporter
|
| 345 | -----------------
|
| 346 | The fw fatal reporter implements `dump` and `recover` callbacks.
|
| 347 | It follows fatal errors indications by CR-space dump and recover flow.
|
| 348 | The CR-space dump uses vsc interface which is valid even if the FW command
|
| 349 | interface is not functional, which is the case in most FW fatal errors.
|
| 350 | The recover function runs recover flow which reloads the driver and triggers fw
|
| 351 | reset if needed.
|
| 352 | On firmware error, the health buffer is dumped into the dmesg. The log
|
| 353 | level is derived from the error's severity (given in health buffer).
|
| 354 |
|
| 355 | User commands examples:
|
| 356 |
|
| 357 | - Run fw recover flow manually::
|
| 358 |
|
| 359 | $ devlink health recover pci/0000:82:00.0 reporter fw_fatal
|
| 360 |
|
| 361 | - Read FW CR-space dump if already stored or trigger new one::
|
| 362 |
|
| 363 | $ devlink health dump show pci/0000:82:00.1 reporter fw_fatal
|
| 364 |
|
| 365 | .. note::
|
| 366 | This command can run only on PF.
|
| 367 |
|
| 368 | vnic reporter
|
| 369 | -------------
|
| 370 | The vnic reporter implements only the `diagnose` callback.
|
| 371 | It is responsible for querying the vnic diagnostic counters from fw and displaying
|
| 372 | them in realtime.
|
| 373 |
|
| 374 | Description of the vnic counters:
|
| 375 |
|
| 376 | - total_error_queues
|
| 377 | number of queues in an error state due to
|
| 378 | an async error or errored command.
|
| 379 | - send_queue_priority_update_flow
|
| 380 | number of QP/SQ priority/SL update events.
|
| 381 | - cq_overrun
|
| 382 | number of times CQ entered an error state due to an overflow.
|
| 383 | - async_eq_overrun
|
| 384 | number of times an EQ mapped to async events was overrun.
|
| 385 | - comp_eq_overrun
|
| 386 | number of times an EQ mapped to completion events was
|
| 387 | overrun.
|
| 388 | - quota_exceeded_command
|
| 389 | number of commands issued and failed due to quota exceeded.
|
| 390 | - invalid_command
|
| 391 | number of commands issued and failed dues to any reason other than quota
|
| 392 | exceeded.
|
| 393 | - nic_receive_steering_discard
|
| 394 | number of packets that completed RX flow
|
| 395 | steering but were discarded due to a mismatch in flow table.
|
| 396 | - generated_pkt_steering_fail
|
| 397 | number of packets generated by the VNIC experiencing unexpected steering
|
| 398 | failure (at any point in steering flow).
|
| 399 | - handled_pkt_steering_fail
|
| 400 | number of packets handled by the VNIC experiencing unexpected steering
|
| 401 | failure (at any point in steering flow owned by the VNIC, including the FDB
|
| 402 | for the eswitch owner).
|
| 403 | - icm_consumption
|
| 404 | amount of Interconnect Host Memory (ICM) consumed by the vnic in
|
| 405 | granularity of 4KB. ICM is host memory allocated by SW upon HCA request
|
| 406 | and is used for storing data structures that control HCA operation.
|
| 407 | - bar_uar_access
|
| 408 | number of WRITE or READ access operations to the UAR on the PCIe BAR.
|
| 409 | - odp_local_triggered_page_fault
|
| 410 | number of locally-triggered page-faults due to ODP.
|
| 411 | - odp_remote_triggered_page_fault
|
| 412 | number of remotly-triggered page-faults due to ODP.
|
| 413 |
|
| 414 | User commands examples:
|
| 415 |
|
| 416 | - Diagnose PF/VF vnic counters::
|
| 417 |
|
| 418 | $ devlink health diagnose pci/0000:82:00.1 reporter vnic
|
| 419 |
|
| 420 | - Diagnose representor vnic counters (performed by supplying devlink port of the
|
| 421 | representor, which can be obtained via devlink port command)::
|
| 422 |
|
| 423 | $ devlink health diagnose pci/0000:82:00.1/65537 reporter vnic
|
| 424 |
|
| 425 | .. note::
|
| 426 | This command can run over all interfaces such as PF/VF and representor ports.
|
| 427 |
|
| 428 | Rates
|
| 429 | =====
|
| 430 |
|
| 431 | mlx5 devices can limit transmission of individual VFs or a group of them via
|
| 432 | the devlink-rate API in switchdev mode.
|
| 433 |
|
| 434 | User commands examples:
|
| 435 |
|
| 436 | - Print the existing rates::
|
| 437 |
|
| 438 | $ devlink port function rate show
|
| 439 |
|
| 440 | - Set a max tx limit on traffic from VF0::
|
| 441 |
|
| 442 | $ devlink port function rate set pci/0000:82:00.0/1 tx_max 10Gbit
|
| 443 |
|
| 444 | - Create a rate group with a max tx limit and add two VFs to it::
|
| 445 |
|
| 446 | $ devlink port function rate add pci/0000:82:00.0/group1 tx_max 10Gbit
|
| 447 | $ devlink port function rate set pci/0000:82:00.0/1 parent group1
|
| 448 | $ devlink port function rate set pci/0000:82:00.0/2 parent group1
|
| 449 |
|
| 450 | - Same scenario, with a min guarantee of 20% of the bandwidth for the first VF::
|
| 451 |
|
| 452 | $ devlink port function rate add pci/0000:82:00.0/group1 tx_max 10Gbit
|
| 453 | $ devlink port function rate set pci/0000:82:00.0/1 parent group1 tx_share 2Gbit
|
| 454 | $ devlink port function rate set pci/0000:82:00.0/2 parent group1
|
| 455 |
|
| 456 | - Cross-device scheduling::
|
| 457 |
|
| 458 | $ devlink port function rate add pci/0000:82:00.0/group1 tx_max 10Gbit
|
| 459 | $ devlink port function rate set pci/0000:82:00.1/32769 parent pci/0000:82:00.0/group1
|