| 1 | |
| 2 | .. include:: <isonum.txt>
|
| 3 |
|
| 4 | ===========================================================
|
| 5 | The PCI Express Advanced Error Reporting Driver Guide HOWTO
|
| 6 | ===========================================================
|
| 7 |
|
| 8 | :Authors: - T. Long Nguyen <[email protected]>
|
| 9 | - Yanmin Zhang <[email protected]>
|
| 10 |
|
| 11 | :Copyright: |copy| 2006 Intel Corporation
|
| 12 |
|
| 13 | Overview
|
| 14 | ===========
|
| 15 |
|
| 16 | About this guide
|
| 17 | ----------------
|
| 18 |
|
| 19 | This guide describes the basics of the PCI Express (PCIe) Advanced Error
|
| 20 | Reporting (AER) driver and provides information on how to use it, as
|
| 21 | well as how to enable the drivers of Endpoint devices to conform with
|
| 22 | the PCIe AER driver.
|
| 23 |
|
| 24 |
|
| 25 | What is the PCIe AER Driver?
|
| 26 | ----------------------------
|
| 27 |
|
| 28 | PCIe error signaling can occur on the PCIe link itself
|
| 29 | or on behalf of transactions initiated on the link. PCIe
|
| 30 | defines two error reporting paradigms: the baseline capability and
|
| 31 | the Advanced Error Reporting capability. The baseline capability is
|
| 32 | required of all PCIe components providing a minimum defined
|
| 33 | set of error reporting requirements. Advanced Error Reporting
|
| 34 | capability is implemented with a PCIe Advanced Error Reporting
|
| 35 | extended capability structure providing more robust error reporting.
|
| 36 |
|
| 37 | The PCIe AER driver provides the infrastructure to support PCIe Advanced
|
| 38 | Error Reporting capability. The PCIe AER driver provides three basic
|
| 39 | functions:
|
| 40 |
|
| 41 | - Gathers the comprehensive error information if errors occurred.
|
| 42 | - Reports error to the users.
|
| 43 | - Performs error recovery actions.
|
| 44 |
|
| 45 | The AER driver only attaches to Root Ports and RCECs that support the PCIe
|
| 46 | AER capability.
|
| 47 |
|
| 48 |
|
| 49 | User Guide
|
| 50 | ==========
|
| 51 |
|
| 52 | Include the PCIe AER Root Driver into the Linux Kernel
|
| 53 | ------------------------------------------------------
|
| 54 |
|
| 55 | The PCIe AER driver is a Root Port service driver attached
|
| 56 | via the PCIe Port Bus driver. If a user wants to use it, the driver
|
| 57 | must be compiled. It is enabled with CONFIG_PCIEAER, which
|
| 58 | depends on CONFIG_PCIEPORTBUS.
|
| 59 |
|
| 60 | Load PCIe AER Root Driver
|
| 61 | -------------------------
|
| 62 |
|
| 63 | Some systems have AER support in firmware. Enabling Linux AER support at
|
| 64 | the same time the firmware handles AER would result in unpredictable
|
| 65 | behavior. Therefore, Linux does not handle AER events unless the firmware
|
| 66 | grants AER control to the OS via the ACPI _OSC method. See the PCI Firmware
|
| 67 | Specification for details regarding _OSC usage.
|
| 68 |
|
| 69 | AER error output
|
| 70 | ----------------
|
| 71 |
|
| 72 | When a PCIe AER error is captured, an error message will be output to
|
| 73 | console. If it's a correctable error, it is output as a warning message.
|
| 74 | Otherwise, it is printed as an error. So users could choose different
|
| 75 | log level to filter out correctable error messages.
|
| 76 |
|
| 77 | Below shows an example::
|
| 78 |
|
| 79 | 0000:50:00.0: PCIe Bus Error: severity=Uncorrectable (Fatal), type=Transaction Layer, (Requester ID)
|
| 80 | 0000:50:00.0: device [8086:0329] error status/mask=00100000/00000000
|
| 81 | 0000:50:00.0: [20] UnsupReq (First)
|
| 82 | 0000:50:00.0: TLP Header: 0x04000001 0x00200a03 0x05010000 0x00050100
|
| 83 |
|
| 84 | In the example, 'Requester ID' means the ID of the device that sent
|
| 85 | the error message to the Root Port. Please refer to PCIe specs for other
|
| 86 | fields.
|
| 87 |
|
| 88 | The 'TLP Header' is the prefix/header of the TLP that caused the error
|
| 89 | in raw hex format. To decode the TLP Header into human-readable form
|
| 90 | one may use tlp-tool:
|
| 91 |
|
| 92 | https://github.com/mmpg-x86/tlp-tool
|
| 93 |
|
| 94 | Example usage::
|
| 95 |
|
| 96 | curl -L https://git.kernel.org/linus/2ca1c94ce0b6 | rtlp-tool --aer
|
| 97 |
|
| 98 | AER Ratelimits
|
| 99 | --------------
|
| 100 |
|
| 101 | Since error messages can be generated for each transaction, we may see
|
| 102 | large volumes of errors reported. To prevent spammy devices from flooding
|
| 103 | the console/stalling execution, messages are throttled by device and error
|
| 104 | type (correctable vs. non-fatal uncorrectable). Fatal errors, including
|
| 105 | DPC errors, are not ratelimited.
|
| 106 |
|
| 107 | AER uses the default ratelimit of DEFAULT_RATELIMIT_BURST (10 events) over
|
| 108 | DEFAULT_RATELIMIT_INTERVAL (5 seconds).
|
| 109 |
|
| 110 | Ratelimits are exposed in the form of sysfs attributes and configurable.
|
| 111 | See Documentation/ABI/testing/sysfs-bus-pci-devices-aer.
|
| 112 |
|
| 113 | AER Statistics / Counters
|
| 114 | -------------------------
|
| 115 |
|
| 116 | When PCIe AER errors are captured, the counters / statistics are also exposed
|
| 117 | in the form of sysfs attributes which are documented at
|
| 118 | Documentation/ABI/testing/sysfs-bus-pci-devices-aer.
|
| 119 |
|
| 120 | Developer Guide
|
| 121 | ===============
|
| 122 |
|
| 123 | To enable error recovery, a software driver must provide callbacks.
|
| 124 |
|
| 125 | To support AER better, developers need to understand how AER works.
|
| 126 |
|
| 127 | PCIe errors are classified into two types: correctable errors
|
| 128 | and uncorrectable errors. This classification is based on the impact
|
| 129 | of those errors, which may result in degraded performance or function
|
| 130 | failure.
|
| 131 |
|
| 132 | Correctable errors pose no impacts on the functionality of the
|
| 133 | interface. The PCIe protocol can recover without any software
|
| 134 | intervention or any loss of data. These errors are detected and
|
| 135 | corrected by hardware.
|
| 136 |
|
| 137 | Unlike correctable errors, uncorrectable
|
| 138 | errors impact functionality of the interface. Uncorrectable errors
|
| 139 | can cause a particular transaction or a particular PCIe link
|
| 140 | to be unreliable. Depending on those error conditions, uncorrectable
|
| 141 | errors are further classified into non-fatal errors and fatal errors.
|
| 142 | Non-fatal errors cause the particular transaction to be unreliable,
|
| 143 | but the PCIe link itself is fully functional. Fatal errors, on
|
| 144 | the other hand, cause the link to be unreliable.
|
| 145 |
|
| 146 | When PCIe error reporting is enabled, a device will automatically send an
|
| 147 | error message to the Root Port above it when it captures
|
| 148 | an error. The Root Port, upon receiving an error reporting message,
|
| 149 | internally processes and logs the error message in its AER
|
| 150 | Capability structure. Error information being logged includes storing
|
| 151 | the error reporting agent's Requester ID into the Error Source
|
| 152 | Identification Registers and setting the error bits of the Root Error
|
| 153 | Status Register accordingly. If AER error reporting is enabled in the Root
|
| 154 | Error Command Register, the Root Port generates an interrupt when an
|
| 155 | error is detected.
|
| 156 |
|
| 157 | Note that the errors as described above are related to the PCIe
|
| 158 | hierarchy and links. These errors do not include any device specific
|
| 159 | errors because device specific errors will still get sent directly to
|
| 160 | the device driver.
|
| 161 |
|
| 162 | Provide callbacks
|
| 163 | -----------------
|
| 164 |
|
| 165 | PCI error-recovery callbacks
|
| 166 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
| 167 |
|
| 168 | The PCIe AER Root driver uses error callbacks to coordinate
|
| 169 | with downstream device drivers associated with a hierarchy in question
|
| 170 | when performing error recovery actions.
|
| 171 |
|
| 172 | Data struct pci_driver has a pointer, err_handler, to point to
|
| 173 | pci_error_handlers who consists of a couple of callback function
|
| 174 | pointers. The AER driver follows the rules defined in
|
| 175 | pci-error-recovery.rst except PCIe-specific parts (see
|
| 176 | below). Please refer to pci-error-recovery.rst for detailed
|
| 177 | definitions of the callbacks.
|
| 178 |
|
| 179 | The sections below specify when to call the error callback functions.
|
| 180 |
|
| 181 | Correctable errors
|
| 182 | ~~~~~~~~~~~~~~~~~~
|
| 183 |
|
| 184 | Correctable errors pose no impacts on the functionality of
|
| 185 | the interface. The PCIe protocol can recover without any
|
| 186 | software intervention or any loss of data. These errors do not
|
| 187 | require any recovery actions. The AER driver clears the device's
|
| 188 | correctable error status register accordingly and logs these errors.
|
| 189 |
|
| 190 | Uncorrectable (non-fatal and fatal) errors
|
| 191 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
| 192 |
|
| 193 | The AER driver performs a Secondary Bus Reset to recover from
|
| 194 | uncorrectable errors. The reset is applied at the port above
|
| 195 | the originating device: If the originating device is an Endpoint,
|
| 196 | only the Endpoint is reset. If on the other hand the originating
|
| 197 | device has subordinate devices, those are all affected by the
|
| 198 | reset as well.
|
| 199 |
|
| 200 | If the originating device is a Root Complex Integrated Endpoint,
|
| 201 | there's no port above where a Secondary Bus Reset could be applied.
|
| 202 | In this case, the AER driver instead applies a Function Level Reset.
|
| 203 |
|
| 204 | If an error message indicates a non-fatal error, performing a reset
|
| 205 | at upstream is not required. The AER driver calls error_detected(dev,
|
| 206 | pci_channel_io_normal) to all drivers associated within a hierarchy in
|
| 207 | question. For example::
|
| 208 |
|
| 209 | Endpoint <==> Downstream Port B <==> Upstream Port A <==> Root Port
|
| 210 |
|
| 211 | If Upstream Port A captures an AER error, the hierarchy consists of
|
| 212 | Downstream Port B and Endpoint.
|
| 213 |
|
| 214 | A driver may return PCI_ERS_RESULT_CAN_RECOVER,
|
| 215 | PCI_ERS_RESULT_DISCONNECT, or PCI_ERS_RESULT_NEED_RESET, depending on
|
| 216 | whether it can recover without a reset, considers the device unrecoverable
|
| 217 | or needs a reset for recovery. If all affected drivers agree that they can
|
| 218 | recover without a reset, it is skipped. Should one driver request a reset,
|
| 219 | it overrides all other drivers.
|
| 220 |
|
| 221 | If an error message indicates a fatal error, kernel will broadcast
|
| 222 | error_detected(dev, pci_channel_io_frozen) to all drivers within
|
| 223 | a hierarchy in question. Then, performing a reset at upstream is
|
| 224 | necessary. If error_detected returns PCI_ERS_RESULT_CAN_RECOVER
|
| 225 | to indicate that recovery without a reset is possible, the error
|
| 226 | handling goes to mmio_enabled, but afterwards a reset is still
|
| 227 | performed.
|
| 228 |
|
| 229 | In other words, for non-fatal errors, drivers may opt in to a reset.
|
| 230 | But for fatal errors, they cannot opt out of a reset, based on the
|
| 231 | assumption that the link is unreliable.
|
| 232 |
|
| 233 | Frequently Asked Questions
|
| 234 | --------------------------
|
| 235 |
|
| 236 | Q:
|
| 237 | What happens if a PCIe device driver does not provide an
|
| 238 | error recovery handler (pci_driver->err_handler is equal to NULL)?
|
| 239 |
|
| 240 | A:
|
| 241 | The devices attached with the driver won't be recovered.
|
| 242 | The kernel will print out informational messages to identify
|
| 243 | unrecoverable devices.
|
| 244 |
|
| 245 |
|
| 246 | Software error injection
|
| 247 | ========================
|
| 248 |
|
| 249 | Debugging PCIe AER error recovery code is quite difficult because it
|
| 250 | is hard to trigger real hardware errors. Software based error
|
| 251 | injection can be used to fake various kinds of PCIe errors.
|
| 252 |
|
| 253 | First you should enable PCIe AER software error injection in kernel
|
| 254 | configuration, that is, following item should be in your .config.
|
| 255 |
|
| 256 | CONFIG_PCIEAER_INJECT=y or CONFIG_PCIEAER_INJECT=m
|
| 257 |
|
| 258 | After reboot with new kernel or insert the module, a device file named
|
| 259 | /dev/aer_inject should be created.
|
| 260 |
|
| 261 | Then, you need a user space tool named aer-inject, which can be gotten
|
| 262 | from:
|
| 263 |
|
| 264 | https://github.com/intel/aer-inject.git
|
| 265 |
|
| 266 | More information about aer-inject can be found in the document in
|
| 267 | its source code.
|