projects/linux·blob master · refs

1087 lines · 38.3 KiB

1.. SPDX-License-Identifier: GPL-2.0
2.. include:: <isonum.txt>
3
4==================================================
5Reliability, Availability and Serviceability (RAS)
6==================================================
7
8This documents different aspects of the RAS functionality present in the
9kernel.
10
11RAS concepts
12************
13
14Reliability, Availability and Serviceability (RAS) is a concept used on
15servers meant to measure their robustness.
16
17Reliability
18 is the probability that a system will produce correct outputs.
19
20 * Generally measured as Mean Time Between Failures (MTBF)
21 * Enhanced by features that help to avoid, detect and repair hardware faults
22
23Availability
24 is the probability that a system is operational at a given time
25
26 * Generally measured as a percentage of downtime per a period of time
27 * Often uses mechanisms to detect and correct hardware faults in
28 runtime;
29
30Serviceability (or maintainability)
31 is the simplicity and speed with which a system can be repaired or
32 maintained
33
34 * Generally measured on Mean Time Between Repair (MTBR)
35
36Improving RAS
37-------------
38
39In order to reduce systems downtime, a system should be capable of detecting
40hardware errors, and, when possible correcting them in runtime. It should
41also provide mechanisms to detect hardware degradation, in order to warn
42the system administrator to take the action of replacing a component before
43it causes data loss or system downtime.
44
45Among the monitoring measures, the most usual ones include:
46
47* CPU – detect errors at instruction execution and at L1/L2/L3 caches;
48* Memory – add error correction logic (ECC) to detect and correct errors;
49* I/O – add CRC checksums for transferred data;
50* Storage – RAID, journal file systems, checksums,
51 Self-Monitoring, Analysis and Reporting Technology (SMART).
52
53By monitoring the number of occurrences of error detections, it is possible
54to identify if the probability of hardware errors is increasing, and, on such
55case, do a preventive maintenance to replace a degraded component while
56those errors are correctable.
57
58Types of errors
59---------------
60
61Most mechanisms used on modern systems use technologies like Hamming
62Codes that allow error correction when the number of errors on a bit packet
63is below a threshold. If the number of errors is above, those mechanisms
64can indicate with a high degree of confidence that an error happened, but
65they can't correct.
66
67Also, sometimes an error occur on a component that it is not used. For
68example, a part of the memory that it is not currently allocated.
69
70That defines some categories of errors:
71
72* **Correctable Error (CE)** - the error detection mechanism detected and
73 corrected the error. Such errors are usually not fatal, although some
74 Kernel mechanisms allow the system administrator to consider them as fatal.
75
76* **Uncorrected Error (UE)** - the amount of errors happened above the error
77 correction threshold, and the system was unable to auto-correct.
78
79* **Fatal Error** - when an UE error happens on a critical component of the
80 system (for example, a piece of the Kernel got corrupted by an UE), the
81 only reliable way to avoid data corruption is to hang or reboot the machine.
82
83* **Non-fatal Error** - when an UE error happens on an unused component,
84 like a CPU in power down state or an unused memory bank, the system may
85 still run, eventually replacing the affected hardware by a hot spare,
86 if available.
87
88 Also, when an error happens on a userspace process, it is also possible to
89 kill such process and let userspace restart it.
90
91The mechanism for handling non-fatal errors is usually complex and may
92require the help of some userspace application, in order to apply the
93policy desired by the system administrator.
94
95Identifying a bad hardware component
96------------------------------------
97
98Just detecting a hardware flaw is usually not enough, as the system needs
99to pinpoint to the minimal replaceable unit (MRU) that should be exchanged
100to make the hardware reliable again.
101
102So, it requires not only error logging facilities, but also mechanisms that
103will translate the error message to the silkscreen or component label for
104the MRU.
105
106Typically, it is very complex for memory, as modern CPUs interlace memory
107from different memory modules, in order to provide a better performance. The
108DMI BIOS usually have a list of memory module labels, with can be obtained
109using the ``dmidecode`` tool. For example, on a desktop machine, it shows::
110
111 Memory Device
112 Total Width: 64 bits
113 Data Width: 64 bits
114 Size: 16384 MB
115 Form Factor: SODIMM
116 Set: None
117 Locator: ChannelA-DIMM0
118 Bank Locator: BANK 0
119 Type: DDR4
120 Type Detail: Synchronous
121 Speed: 2133 MHz
122 Rank: 2
123 Configured Clock Speed: 2133 MHz
124
125On the above example, a DDR4 SO-DIMM memory module is located at the
126system's memory labeled as "BANK 0", as given by the *bank locator* field.
127Please notice that, on such system, the *total width* is equal to the
128*data width*. It means that such memory module doesn't have error
129detection/correction mechanisms.
130
131Unfortunately, not all systems use the same field to specify the memory
132bank. On this example, from an older server, ``dmidecode`` shows::
133
134 Memory Device
135 Array Handle: 0x1000
136 Error Information Handle: Not Provided
137 Total Width: 72 bits
138 Data Width: 64 bits
139 Size: 8192 MB
140 Form Factor: DIMM
141 Set: 1
142 Locator: DIMM_A1
143 Bank Locator: Not Specified
144 Type: DDR3
145 Type Detail: Synchronous Registered (Buffered)
146 Speed: 1600 MHz
147 Rank: 2
148 Configured Clock Speed: 1600 MHz
149
150There, the DDR3 RDIMM memory module is located at the system's memory labeled
151as "DIMM_A1", as given by the *locator* field. Please notice that this
152memory module has 64 bits of *data width* and 72 bits of *total width*. So,
153it has 8 extra bits to be used by error detection and correction mechanisms.
154Such kind of memory is called Error-correcting code memory (ECC memory).
155
156To make things even worse, it is not uncommon that systems with different
157labels on their system's board to use exactly the same BIOS, meaning that
158the labels provided by the BIOS won't match the real ones.
159
160ECC memory
161----------
162
163As mentioned in the previous section, ECC memory has extra bits to be
164used for error correction. In the above example, a memory module has
16564 bits of *data width*, and 72 bits of *total width*. The extra 8
166bits which are used for the error detection and correction mechanisms
167are referred to as the *syndrome*\ [#f1]_\ [#f2]_.
168
169So, when the cpu requests the memory controller to write a word with
170*data width*, the memory controller calculates the *syndrome* in real time,
171using Hamming code, or some other error correction code, like SECDED+,
172producing a code with *total width* size. Such code is then written
173on the memory modules.
174
175At read, the *total width* bits code is converted back, using the same
176ECC code used on write, producing a word with *data width* and a *syndrome*.
177The word with *data width* is sent to the CPU, even when errors happen.
178
179The memory controller also looks at the *syndrome* in order to check if
180there was an error, and if the ECC code was able to fix such error.
181If the error was corrected, a Corrected Error (CE) happened. If not, an
182Uncorrected Error (UE) happened.
183
184The information about the CE/UE errors is stored on some special registers
185at the memory controller and can be accessed by reading such registers,
186either by BIOS, by some special CPUs or by Linux EDAC driver. On x86 64
187bit CPUs, such errors can also be retrieved via the Machine Check
188Architecture (MCA)\ [#f3]_.
189
190.. [#f1] Please notice that several memory controllers allow operation on a
191 mode called "Lock-Step", where it groups two memory modules together,
192 doing 128-bit reads/writes. That gives 16 bits for error correction, with
193 significantly improves the error correction mechanism, at the expense
194 that, when an error happens, there's no way to know what memory module is
195 to blame. So, it has to blame both memory modules.
196
197.. [#f2] Some memory controllers also allow using memory in mirror mode.
198 On such mode, the same data is written to two memory modules. At read,
199 the system checks both memory modules, in order to check if both provide
200 identical data. On such configuration, when an error happens, there's no
201 way to know what memory module is to blame. So, it has to blame both
202 memory modules (or 4 memory modules, if the system is also on Lock-step
203 mode).
204
205.. [#f3] For more details about the Machine Check Architecture (MCA),
206 please read Documentation/arch/x86/x86_64/machinecheck.rst at the Kernel tree.
207
208EDAC - Error Detection And Correction
209*************************************
210
211.. note::
212
213 "bluesmoke" was the name for this device driver subsystem when it
214 was "out-of-tree" and maintained at http://bluesmoke.sourceforge.net.
215 That site is mostly archaic now and can be used only for historical
216 purposes.
217
218 When the subsystem was pushed upstream for the first time, on
219 Kernel 2.6.16, it was renamed to ``EDAC``.
220
221Purpose
222-------
223
224The ``edac`` kernel module's goal is to detect and report hardware errors
225that occur within the computer system running under linux.
226
227Memory
228------
229
230Memory Correctable Errors (CE) and Uncorrectable Errors (UE) are the
231primary errors being harvested. These types of errors are harvested by
232the ``edac_mc`` device.
233
234Detecting CE events, then harvesting those events and reporting them,
235**can** but must not necessarily be a predictor of future UE events. With
236CE events only, the system can and will continue to operate as no data
237has been damaged yet.
238
239However, preventive maintenance and proactive part replacement of memory
240modules exhibiting CEs can reduce the likelihood of the dreaded UE events
241and system panics.
242
243Other hardware elements
244-----------------------
245
246A new feature for EDAC, the ``edac_device`` class of device, was added in
247the 2.6.23 version of the kernel.
248
249This new device type allows for non-memory type of ECC hardware detectors
250to have their states harvested and presented to userspace via the sysfs
251interface.
252
253Some architectures have ECC detectors for L1, L2 and L3 caches,
254along with DMA engines, fabric switches, main data path switches,
255interconnections, and various other hardware data paths. If the hardware
256reports it, then an edac_device device probably can be constructed to
257harvest and present that to userspace.
258
259
260PCI bus scanning
261----------------
262
263In addition, PCI devices are scanned for PCI Bus Parity and SERR Errors
264in order to determine if errors are occurring during data transfers.
265
266The presence of PCI Parity errors must be examined with a grain of salt.
267There are several add-in adapters that do **not** follow the PCI specification
268with regards to Parity generation and reporting. The specification says
269the vendor should tie the parity status bits to 0 if they do not intend
270to generate parity. Some vendors do not do this, and thus the parity bit
271can "float" giving false positives.
272
273There is a PCI device attribute located in sysfs that is checked by
274the EDAC PCI scanning code. If that attribute is set, PCI parity/error
275scanning is skipped for that device. The attribute is::
276
277 broken_parity_status
278
279and is located in ``/sys/devices/pci<XXX>/0000:XX:YY.Z`` directories for
280PCI devices.
281
282
283Versioning
284----------
285
286EDAC is composed of a "core" module (``edac_core.ko``) and several Memory
287Controller (MC) driver modules. On a given system, the CORE is loaded
288and one MC driver will be loaded. Both the CORE and the MC driver (or
289``edac_device`` driver) have individual versions that reflect current
290release level of their respective modules.
291
292Thus, to "report" on what version a system is running, one must report
293both the CORE's and the MC driver's versions.
294
295
296Loading
297-------
298
299If ``edac`` was statically linked with the kernel then no loading
300is necessary. If ``edac`` was built as modules then simply modprobe
301the ``edac`` pieces that you need. You should be able to modprobe
302hardware-specific modules and have the dependencies load the necessary
303core modules.
304
305Example::
306
307 $ modprobe amd76x_edac
308
309loads both the ``amd76x_edac.ko`` memory controller module and the
310``edac_mc.ko`` core module.
311
312
313Sysfs interface
314---------------
315
316EDAC presents a ``sysfs`` interface for control and reporting purposes. It
317lives in the /sys/devices/system/edac directory.
318
319Within this directory there currently reside 2 components:
320
321 ======= ==============================
322 mc memory controller(s) system
323 pci PCI control and status system
324 ======= ==============================
325
326
327
328Memory Controller (mc) Model
329----------------------------
330
331Each ``mc`` device controls a set of memory modules [#f4]_. These modules
332are laid out in a Chip-Select Row (``csrowX``) and Channel table (``chX``).
333There can be multiple csrows and multiple channels.
334
335.. [#f4] Nowadays, the term DIMM (Dual In-line Memory Module) is widely
336 used to refer to a memory module, although there are other memory
337 packaging alternatives, like SO-DIMM, SIMM, etc. The UEFI
338 specification (Version 2.7) defines a memory module in the Common
339 Platform Error Record (CPER) section to be an SMBIOS Memory Device
340 (Type 17). Along this document, and inside the EDAC subsystem, the term
341 "dimm" is used for all memory modules, even when they use a
342 different kind of packaging.
343
344Memory controllers allow for several csrows, with 8 csrows being a
345typical value. Yet, the actual number of csrows depends on the layout of
346a given motherboard, memory controller and memory module characteristics.
347
348Dual channels allow for dual data length (e. g. 128 bits, on 64 bit systems)
349data transfers to/from the CPU from/to memory. Some newer chipsets allow
350for more than 2 channels, like Fully Buffered DIMMs (FB-DIMMs) memory
351controllers. The following example will assume 2 channels:
352
353 +------------+-----------------------+
354 | CS Rows | Channels |
355 +------------+-----------+-----------+
356 | | ``ch0`` | ``ch1`` |
357 +============+===========+===========+
358 | |**DIMM_A0**|**DIMM_B0**|
359 +------------+-----------+-----------+
360 | ``csrow0`` | rank0 | rank0 |
361 +------------+-----------+-----------+
362 | ``csrow1`` | rank1 | rank1 |
363 +------------+-----------+-----------+
364 | |**DIMM_A1**|**DIMM_B1**|
365 +------------+-----------+-----------+
366 | ``csrow2`` | rank0 | rank0 |
367 +------------+-----------+-----------+
368 | ``csrow3`` | rank1 | rank1 |
369 +------------+-----------+-----------+
370
371In the above example, there are 4 physical slots on the motherboard
372for memory DIMMs:
373
374 +---------+---------+
375 | DIMM_A0 | DIMM_B0 |
376 +---------+---------+
377 | DIMM_A1 | DIMM_B1 |
378 +---------+---------+
379
380Labels for these slots are usually silk-screened on the motherboard.
381Slots labeled ``A`` are channel 0 in this example. Slots labeled ``B`` are
382channel 1. Notice that there are two csrows possible on a physical DIMM.
383These csrows are allocated their csrow assignment based on the slot into
384which the memory DIMM is placed. Thus, when 1 DIMM is placed in each
385Channel, the csrows cross both DIMMs.
386
387Memory DIMMs come single or dual "ranked". A rank is a populated csrow.
388In the example above 2 dual ranked DIMMs are similarly placed. Thus,
389both csrow0 and csrow1 are populated. On the other hand, when 2 single
390ranked DIMMs are placed in slots DIMM_A0 and DIMM_B0, then they will
391have just one csrow (csrow0) and csrow1 will be empty. The pattern
392repeats itself for csrow2 and csrow3. Also note that some memory
393controllers don't have any logic to identify the memory module, see
394``rankX`` directories below.
395
396The representation of the above is reflected in the directory
397tree in EDAC's sysfs interface. Starting in directory
398``/sys/devices/system/edac/mc``, each memory controller will be
399represented by its own ``mcX`` directory, where ``X`` is the
400index of the MC::
401
402 ..../edac/mc/
403 |
404 |->mc0
405 |->mc1
406 |->mc2
407 ....
408
409Within each of the ``mcX`` directory are several EDAC control and
410attribute files.
411
412``mcX`` directories
413-------------------
414
415In ``mcX`` directories are EDAC control and attribute files for
416this ``X`` instance of the memory controllers.
417
418For a description of the sysfs API, please see:
419
420 Documentation/ABI/testing/sysfs-devices-edac
421
422
423``dimmX`` or ``rankX`` directories
424----------------------------------
425
426The recommended way to use the EDAC subsystem is to look at the information
427provided by the ``dimmX`` or ``rankX`` directories [#f5]_.
428
429A typical EDAC system has the following structure under
430``/sys/devices/system/edac/``\ [#f6]_::
431
432 /sys/devices/system/edac/
433 ├── mc
434 │   ├── mc0
435 │   │   ├── ce_count
436 │   │   ├── ce_noinfo_count
437 │   │   ├── dimm0
438 │   │   │   ├── dimm_ce_count
439 │   │   │   ├── dimm_dev_type
440 │   │   │   ├── dimm_edac_mode
441 │   │   │   ├── dimm_label
442 │   │   │   ├── dimm_location
443 │   │   │   ├── dimm_mem_type
444 │   │   │   ├── dimm_ue_count
445 │   │   │   ├── size
446 │   │   │   └── uevent
447 │   │   ├── max_location
448 │   │   ├── mc_name
449 │   │   ├── reset_counters
450 │   │   ├── seconds_since_reset
451 │   │   ├── size_mb
452 │   │   ├── ue_count
453 │   │   ├── ue_noinfo_count
454 │   │   └── uevent
455 │   ├── mc1
456 │   │   ├── ce_count
457 │   │   ├── ce_noinfo_count
458 │   │   ├── dimm0
459 │   │   │   ├── dimm_ce_count
460 │   │   │   ├── dimm_dev_type
461 │   │   │   ├── dimm_edac_mode
462 │   │   │   ├── dimm_label
463 │   │   │   ├── dimm_location
464 │   │   │   ├── dimm_mem_type
465 │   │   │   ├── dimm_ue_count
466 │   │   │   ├── size
467 │   │   │   └── uevent
468 │   │   ├── max_location
469 │   │   ├── mc_name
470 │   │   ├── reset_counters
471 │   │   ├── seconds_since_reset
472 │   │   ├── size_mb
473 │   │   ├── ue_count
474 │   │   ├── ue_noinfo_count
475 │   │   └── uevent
476 │   └── uevent
477 └── uevent
478
479In the ``dimmX`` directories are EDAC control and attribute files for
480this ``X`` memory module:
481
482- ``size`` - Total memory managed by this csrow attribute file
483
484 This attribute file displays, in count of megabytes, the memory
485 that this csrow contains.
486
487- ``dimm_ue_count`` - Uncorrectable Errors count attribute file
488
489 This attribute file displays the total count of uncorrectable
490 errors that have occurred on this DIMM. If panic_on_ue is set
491 this counter will not have a chance to increment, since EDAC
492 will panic the system.
493
494- ``dimm_ce_count`` - Correctable Errors count attribute file
495
496 This attribute file displays the total count of correctable
497 errors that have occurred on this DIMM. This count is very
498 important to examine. CEs provide early indications that a
499 DIMM is beginning to fail. This count field should be
500 monitored for non-zero values and report such information
501 to the system administrator.
502
503- ``dimm_dev_type`` - Device type attribute file
504
505 This attribute file will display what type of DRAM device is
506 being utilized on this DIMM.
507 Examples:
508
509 - x1
510 - x2
511 - x4
512 - x8
513
514- ``dimm_edac_mode`` - EDAC Mode of operation attribute file
515
516 This attribute file will display what type of Error detection
517 and correction is being utilized.
518
519- ``dimm_label`` - memory module label control file
520
521 This control file allows this DIMM to have a label assigned
522 to it. With this label in the module, when errors occur
523 the output can provide the DIMM label in the system log.
524 This becomes vital for panic events to isolate the
525 cause of the UE event.
526
527 DIMM Labels must be assigned after booting, with information
528 that correctly identifies the physical slot with its
529 silk screen label. This information is currently very
530 motherboard specific and determination of this information
531 must occur in userland at this time.
532
533- ``dimm_location`` - location of the memory module
534
535 The location can have up to 3 levels, and describe how the
536 memory controller identifies the location of a memory module.
537 Depending on the type of memory and memory controller, it
538 can be:
539
540 - *csrow* and *channel* - used when the memory controller
541 doesn't identify a single DIMM - e. g. in ``rankX`` dir;
542 - *branch*, *channel*, *slot* - typically used on FB-DIMM memory
543 controllers;
544 - *channel*, *slot* - used on Nehalem and newer Intel drivers.
545
546- ``dimm_mem_type`` - Memory Type attribute file
547
548 This attribute file will display what type of memory is currently
549 on this csrow. Normally, either buffered or unbuffered memory.
550 Examples:
551
552 - Registered-DDR
553 - Unbuffered-DDR
554
555.. [#f5] On some systems, the memory controller doesn't have any logic
556 to identify the memory module. On such systems, the directory is called ``rankX``.
557 On modern Intel memory controllers, the memory controller identifies the
558 memory modules directly. On such systems, the directory is called ``dimmX``.
559
560.. [#f6] There are also some ``power`` directories and ``subsystem``
561 symlinks inside the sysfs mapping that are automatically created by
562 the sysfs subsystem. Currently, they serve no purpose.
563
564
565System Logging
566--------------
567
568If logging for UEs and CEs is enabled, then system logs will contain
569information indicating that errors have been detected::
570
571 EDAC MC0: CE page 0x283, offset 0xce0, grain 8, syndrome 0x6ec3, row 0, channel 1 "DIMM_B1": amd76x_edac
572 EDAC MC0: CE page 0x1e5, offset 0xfb0, grain 8, syndrome 0xb741, row 0, channel 1 "DIMM_B1": amd76x_edac
573
574
575The structure of the message is:
576
577 +---------------------------------------+-------------+
578 | Content | Example |
579 +=======================================+=============+
580 | The memory controller | MC0 |
581 +---------------------------------------+-------------+
582 | Error type | CE |
583 +---------------------------------------+-------------+
584 | Memory page | 0x283 |
585 +---------------------------------------+-------------+
586 | Offset in the page | 0xce0 |
587 +---------------------------------------+-------------+
588 | The byte granularity | grain 8 |
589 | or resolution of the error | |
590 +---------------------------------------+-------------+
591 | The error syndrome | 0xb741 |
592 +---------------------------------------+-------------+
593 | Memory row | row 0 |
594 +---------------------------------------+-------------+
595 | Memory channel | channel 1 |
596 +---------------------------------------+-------------+
597 | DIMM label, if set prior | DIMM B1 |
598 +---------------------------------------+-------------+
599 | And then an optional, driver-specific | |
600 | message that may have additional | |
601 | information. | |
602 +---------------------------------------+-------------+
603
604Both UEs and CEs with no info will lack all but memory controller, error
605type, a notice of "no info" and then an optional, driver-specific error
606message.
607
608
609PCI Bus Parity Detection
610------------------------
611
612On Header Type 00 devices, the primary status is looked at for any
613parity error regardless of whether parity is enabled on the device or
614not. (The spec indicates parity is generated in some cases). On Header
615Type 01 bridges, the secondary status register is also looked at to see
616if parity occurred on the bus on the other side of the bridge.
617
618
619Sysfs configuration
620-------------------
621
622Under ``/sys/devices/system/edac/pci`` are control and attribute files as
623follows:
624
625
626- ``check_pci_errors`` - Enable/Disable PCI Parity checking control file
627
628 This control file enables or disables the PCI Bus Parity scanning
629 operation. Writing a 1 to this file enables the scanning. Writing
630 a 0 to this file disables the scanning.
631
632 Enable::
633
634 echo "1" >/sys/devices/system/edac/pci/check_pci_errors
635
636 Disable::
637
638 echo "0" >/sys/devices/system/edac/pci/check_pci_errors
639
640
641- ``pci_parity_count`` - Parity Count
642
643 This attribute file will display the number of parity errors that
644 have been detected.
645
646
647Module parameters
648-----------------
649
650- ``edac_mc_panic_on_ue`` - Panic on UE control file
651
652 An uncorrectable error will cause a machine panic. This is usually
653 desirable. It is a bad idea to continue when an uncorrectable error
654 occurs - it is indeterminate what was uncorrected and the operating
655 system context might be so mangled that continuing will lead to further
656 corruption. If the kernel has MCE configured, then EDAC will never
657 notice the UE.
658
659 LOAD TIME::
660
661 module/kernel parameter: edac_mc_panic_on_ue=[0|1]
662
663 RUN TIME::
664
665 echo "1" > /sys/module/edac_core/parameters/edac_mc_panic_on_ue
666
667
668- ``edac_mc_log_ue`` - Log UE control file
669
670
671 Generate kernel messages describing uncorrectable errors. These errors
672 are reported through the system message log system. UE statistics
673 will be accumulated even when UE logging is disabled.
674
675 LOAD TIME::
676
677 module/kernel parameter: edac_mc_log_ue=[0|1]
678
679 RUN TIME::
680
681 echo "1" > /sys/module/edac_core/parameters/edac_mc_log_ue
682
683
684- ``edac_mc_log_ce`` - Log CE control file
685
686
687 Generate kernel messages describing correctable errors. These
688 errors are reported through the system message log system.
689 CE statistics will be accumulated even when CE logging is disabled.
690
691 LOAD TIME::
692
693 module/kernel parameter: edac_mc_log_ce=[0|1]
694
695 RUN TIME::
696
697 echo "1" > /sys/module/edac_core/parameters/edac_mc_log_ce
698
699
700- ``edac_mc_poll_msec`` - Polling period control file
701
702
703 The time period, in milliseconds, for polling for error information.
704 Too small a value wastes resources. Too large a value might delay
705 necessary handling of errors and might loose valuable information for
706 locating the error. 1000 milliseconds (once each second) is the current
707 default. Systems which require all the bandwidth they can get, may
708 increase this.
709
710 LOAD TIME::
711
712 module/kernel parameter: edac_mc_poll_msec=[0|1]
713
714 RUN TIME::
715
716 echo "1000" > /sys/module/edac_core/parameters/edac_mc_poll_msec
717
718
719- ``panic_on_pci_parity`` - Panic on PCI PARITY Error
720
721
722 This control file enables or disables panicking when a parity
723 error has been detected.
724
725
726 module/kernel parameter::
727
728 edac_pci_panic_on_pe=[0|1]
729
730 Enable::
731
732 echo "1" > /sys/module/edac_core/parameters/edac_pci_panic_on_pe
733
734 Disable::
735
736 echo "0" > /sys/module/edac_core/parameters/edac_pci_panic_on_pe
737
738
739
740EDAC device type
741----------------
742
743In the header file, edac_pci.h, there is a series of edac_device structures
744and APIs for the EDAC_DEVICE.
745
746User space access to an edac_device is through the sysfs interface.
747
748At the location ``/sys/devices/system/edac`` (sysfs) new edac_device devices
749will appear.
750
751There is a three level tree beneath the above ``edac`` directory. For example,
752the ``test_device_edac`` device (found at the http://bluesmoke.sourceforget.net
753website) installs itself as::
754
755 /sys/devices/system/edac/test-instance
756
757in this directory are various controls, a symlink and one or more ``instance``
758directories.
759
760The standard default controls are:
761
762 ============== =======================================================
763 log_ce boolean to log CE events
764 log_ue boolean to log UE events
765 panic_on_ue boolean to ``panic`` the system if an UE is encountered
766 (default off, can be set true via startup script)
767 poll_msec time period between POLL cycles for events
768 ============== =======================================================
769
770The test_device_edac device adds at least one of its own custom control:
771
772 ============== ==================================================
773 test_bits which in the current test driver does nothing but
774 show how it is installed. A ported driver can
775 add one or more such controls and/or attributes
776 for specific uses.
777 One out-of-tree driver uses controls here to allow
778 for ERROR INJECTION operations to hardware
779 injection registers
780 ============== ==================================================
781
782The symlink points to the 'struct dev' that is registered for this edac_device.
783
784Instances
785---------
786
787One or more instance directories are present. For the ``test_device_edac``
788case:
789
790 +----------------+
791 | test-instance0 |
792 +----------------+
793
794
795In this directory there are two default counter attributes, which are totals of
796counter in deeper subdirectories.
797
798 ============== ====================================
799 ce_count total of CE events of subdirectories
800 ue_count total of UE events of subdirectories
801 ============== ====================================
802
803Blocks
804------
805
806At the lowest directory level is the ``block`` directory. There can be 0, 1
807or more blocks specified in each instance:
808
809 +-------------+
810 | test-block0 |
811 +-------------+
812
813In this directory the default attributes are:
814
815 ============== ================================================
816 ce_count which is counter of CE events for this ``block``
817 of hardware being monitored
818 ue_count which is counter of UE events for this ``block``
819 of hardware being monitored
820 ============== ================================================
821
822
823The ``test_device_edac`` device adds 4 attributes and 1 control:
824
825 ================== ====================================================
826 test-block-bits-0 for every POLL cycle this counter
827 is incremented
828 test-block-bits-1 every 10 cycles, this counter is bumped once,
829 and test-block-bits-0 is set to 0
830 test-block-bits-2 every 100 cycles, this counter is bumped once,
831 and test-block-bits-1 is set to 0
832 test-block-bits-3 every 1000 cycles, this counter is bumped once,
833 and test-block-bits-2 is set to 0
834 ================== ====================================================
835
836
837 ================== ====================================================
838 reset-counters writing ANY thing to this control will
839 reset all the above counters.
840 ================== ====================================================
841
842
843Use of the ``test_device_edac`` driver should enable any others to create their own
844unique drivers for their hardware systems.
845
846The ``test_device_edac`` sample driver is located at the
847http://bluesmoke.sourceforge.net project site for EDAC.
848
849
850Usage of EDAC APIs on Nehalem and newer Intel CPUs
851--------------------------------------------------
852
853On older Intel architectures, the memory controller was part of the North
854Bridge chipset. Nehalem, Sandy Bridge, Ivy Bridge, Haswell, Sky Lake and
855newer Intel architectures integrated an enhanced version of the memory
856controller (MC) inside the CPUs.
857
858This chapter will cover the differences of the enhanced memory controllers
859found on newer Intel CPUs, such as ``i7core_edac``, ``sb_edac`` and
860``sbx_edac`` drivers.
861
862.. note::
863
864 The Xeon E7 processor families use a separate chip for the memory
865 controller, called Intel Scalable Memory Buffer. This section doesn't
866 apply for such families.
867
8681) There is one Memory Controller per Quick Patch Interconnect
869 (QPI). At the driver, the term "socket" means one QPI. This is
870 associated with a physical CPU socket.
871
872 Each MC have 3 physical read channels, 3 physical write channels and
873 3 logic channels. The driver currently sees it as just 3 channels.
874 Each channel can have up to 3 DIMMs.
875
876 The minimum known unity is DIMMs. There are no information about csrows.
877 As EDAC API maps the minimum unity is csrows, the driver sequentially
878 maps channel/DIMM into different csrows.
879
880 For example, supposing the following layout::
881
882 Ch0 phy rd0, wr0 (0x063f4031): 2 ranks, UDIMMs
883 dimm 0 1024 Mb offset: 0, bank: 8, rank: 1, row: 0x4000, col: 0x400
884 dimm 1 1024 Mb offset: 4, bank: 8, rank: 1, row: 0x4000, col: 0x400
885 Ch1 phy rd1, wr1 (0x063f4031): 2 ranks, UDIMMs
886 dimm 0 1024 Mb offset: 0, bank: 8, rank: 1, row: 0x4000, col: 0x400
887 Ch2 phy rd3, wr3 (0x063f4031): 2 ranks, UDIMMs
888 dimm 0 1024 Mb offset: 0, bank: 8, rank: 1, row: 0x4000, col: 0x400
889
890 The driver will map it as::
891
892 csrow0: channel 0, dimm0
893 csrow1: channel 0, dimm1
894 csrow2: channel 1, dimm0
895 csrow3: channel 2, dimm0
896
897 exports one DIMM per csrow.
898
899 Each QPI is exported as a different memory controller.
900
9012) The MC has the ability to inject errors to test drivers. The drivers
902 implement this functionality via some error injection nodes:
903
904 For injecting a memory error, there are some sysfs nodes, under
905 ``/sys/devices/system/edac/mc/mc?/``:
906
907 - ``inject_addrmatch/*``:
908 Controls the error injection mask register. It is possible to specify
909 several characteristics of the address to match an error code::
910
911 dimm = the affected dimm. Numbers are relative to a channel;
912 rank = the memory rank;
913 channel = the channel that will generate an error;
914 bank = the affected bank;
915 page = the page address;
916 column (or col) = the address column.
917
918 each of the above values can be set to "any" to match any valid value.
919
920 At driver init, all values are set to any.
921
922 For example, to generate an error at rank 1 of dimm 2, for any channel,
923 any bank, any page, any column::
924
925 echo 2 >/sys/devices/system/edac/mc/mc0/inject_addrmatch/dimm
926 echo 1 >/sys/devices/system/edac/mc/mc0/inject_addrmatch/rank
927
928 To return to the default behaviour of matching any, you can do::
929
930 echo any >/sys/devices/system/edac/mc/mc0/inject_addrmatch/dimm
931 echo any >/sys/devices/system/edac/mc/mc0/inject_addrmatch/rank
932
933 - ``inject_eccmask``:
934 specifies what bits will have troubles,
935
936 - ``inject_section``:
937 specifies what ECC cache section will get the error::
938
939 3 for both
940 2 for the highest
941 1 for the lowest
942
943 - ``inject_type``:
944 specifies the type of error, being a combination of the following bits::
945
946 bit 0 - repeat
947 bit 1 - ecc
948 bit 2 - parity
949
950 - ``inject_enable``:
951 starts the error generation when something different than 0 is written.
952
953 All inject vars can be read. root permission is needed for write.
954
955 Datasheet states that the error will only be generated after a write on an
956 address that matches inject_addrmatch. It seems, however, that reading will
957 also produce an error.
958
959 For example, the following code will generate an error for any write access
960 at socket 0, on any DIMM/address on channel 2::
961
962 echo 2 >/sys/devices/system/edac/mc/mc0/inject_addrmatch/channel
963 echo 2 >/sys/devices/system/edac/mc/mc0/inject_type
964 echo 64 >/sys/devices/system/edac/mc/mc0/inject_eccmask
965 echo 3 >/sys/devices/system/edac/mc/mc0/inject_section
966 echo 1 >/sys/devices/system/edac/mc/mc0/inject_enable
967 dd if=/dev/mem of=/dev/null seek=16k bs=4k count=1 >& /dev/null
968
969 For socket 1, it is needed to replace "mc0" by "mc1" at the above
970 commands.
971
972 The generated error message will look like::
973
974 EDAC MC0: UE row 0, channel-a= 0 channel-b= 0 labels "-": NON_FATAL (addr = 0x0075b980, socket=0, Dimm=0, Channel=2, syndrome=0x00000040, count=1, Err=8c0000400001009f:4000080482 (read error: read ECC error))
975
9763) Corrected Error memory register counters
977
978 Those newer MCs have some registers to count memory errors. The driver
979 uses those registers to report Corrected Errors on devices with Registered
980 DIMMs.
981
982 However, those counters don't work with Unregistered DIMM. As the chipset
983 offers some counters that also work with UDIMMs (but with a worse level of
984 granularity than the default ones), the driver exposes those registers for
985 UDIMM memories.
986
987 They can be read by looking at the contents of ``all_channel_counts/``::
988
989 $ for i in /sys/devices/system/edac/mc/mc0/all_channel_counts/*; do echo $i; cat $i; done
990 /sys/devices/system/edac/mc/mc0/all_channel_counts/udimm0
991 0
992 /sys/devices/system/edac/mc/mc0/all_channel_counts/udimm1
993 0
994 /sys/devices/system/edac/mc/mc0/all_channel_counts/udimm2
995 0
996
997 What happens here is that errors on different csrows, but at the same
998 dimm number will increment the same counter.
999 So, in this memory mapping::
1000
1001 csrow0: channel 0, dimm0
1002 csrow1: channel 0, dimm1
1003 csrow2: channel 1, dimm0
1004 csrow3: channel 2, dimm0
1005
1006 The hardware will increment udimm0 for an error at the first dimm at either
1007 csrow0, csrow2 or csrow3;
1008
1009 The hardware will increment udimm1 for an error at the second dimm at either
1010 csrow0, csrow2 or csrow3;
1011
1012 The hardware will increment udimm2 for an error at the third dimm at either
1013 csrow0, csrow2 or csrow3;
1014
10154) Standard error counters
1016
1017 The standard error counters are generated when an mcelog error is received
1018 by the driver. Since, with UDIMM, this is counted by software, it is
1019 possible that some errors could be lost. With RDIMM's, they display the
1020 contents of the registers
1021
1022Reference documents used on ``amd64_edac``
1023------------------------------------------
1024
1025``amd64_edac`` module is based on the following documents
1026(available from http://support.amd.com/en-us/search/tech-docs):
1027
10281. :Title: BIOS and Kernel Developer's Guide for AMD Athlon 64 and AMD
1029 Opteron Processors
1030 :AMD publication #: 26094
1031 :Revision: 3.26
1032 :Link: http://support.amd.com/TechDocs/26094.PDF
1033
10342. :Title: BIOS and Kernel Developer's Guide for AMD NPT Family 0Fh
1035 Processors
1036 :AMD publication #: 32559
1037 :Revision: 3.00
1038 :Issue Date: May 2006
1039 :Link: http://support.amd.com/TechDocs/32559.pdf
1040
10413. :Title: BIOS and Kernel Developer's Guide (BKDG) For AMD Family 10h
1042 Processors
1043 :AMD publication #: 31116
1044 :Revision: 3.00
1045 :Issue Date: September 07, 2007
1046 :Link: http://support.amd.com/TechDocs/31116.pdf
1047
10484. :Title: BIOS and Kernel Developer's Guide (BKDG) for AMD Family 15h
1049 Models 30h-3Fh Processors
1050 :AMD publication #: 49125
1051 :Revision: 3.06
1052 :Issue Date: 2/12/2015 (latest release)
1053 :Link: http://support.amd.com/TechDocs/49125_15h_Models_30h-3Fh_BKDG.pdf
1054
10555. :Title: BIOS and Kernel Developer's Guide (BKDG) for AMD Family 15h
1056 Models 60h-6Fh Processors
1057 :AMD publication #: 50742
1058 :Revision: 3.01
1059 :Issue Date: 7/23/2015 (latest release)
1060 :Link: http://support.amd.com/TechDocs/50742_15h_Models_60h-6Fh_BKDG.pdf
1061
10626. :Title: BIOS and Kernel Developer's Guide (BKDG) for AMD Family 16h
1063 Models 00h-0Fh Processors
1064 :AMD publication #: 48751
1065 :Revision: 3.03
1066 :Issue Date: 2/23/2015 (latest release)
1067 :Link: http://support.amd.com/TechDocs/48751_16h_bkdg.pdf
1068
1069Credits
1070=======
1071
1072* Written by Doug Thompson <[email protected]>
1073
1074 - 7 Dec 2005
1075 - 17 Jul 2007 Updated
1076
1077* |copy| Mauro Carvalho Chehab
1078
1079 - 05 Aug 2009 Nehalem interface
1080 - 26 Oct 2016 Converted to ReST and cleanups at the Nehalem section
1081
1082* EDAC authors/maintainers:
1083
1084 - Doug Thompson, Dave Jiang, Dave Peterson et al,
1085 - Mauro Carvalho Chehab
1086 - Borislav Petkov
1087 - original author: Thayne Harbaugh