| 1 | Tainted kernels
|
| 2 | ---------------
|
| 3 |
|
| 4 | The kernel will mark itself as 'tainted' when something occurs that might be
|
| 5 | relevant later when investigating problems. Don't worry too much about this,
|
| 6 | most of the time it's not a problem to run a tainted kernel; the information is
|
| 7 | mainly of interest once someone wants to investigate some problem, as its real
|
| 8 | cause might be the event that got the kernel tainted. That's why bug reports
|
| 9 | from tainted kernels will often be ignored by developers, hence try to reproduce
|
| 10 | problems with an untainted kernel.
|
| 11 |
|
| 12 | Note the kernel will remain tainted even after you undo what caused the taint
|
| 13 | (i.e. unload a proprietary kernel module), to indicate the kernel remains not
|
| 14 | trustworthy. That's also why the kernel will print the tainted state when it
|
| 15 | notices an internal problem (a 'kernel bug'), a recoverable error
|
| 16 | ('kernel oops') or a non-recoverable error ('kernel panic') and writes debug
|
| 17 | information about this to the logs ``dmesg`` outputs. It's also possible to
|
| 18 | check the tainted state at runtime through a file in ``/proc/``.
|
| 19 |
|
| 20 |
|
| 21 | Tainted flag in bugs, oops or panics messages
|
| 22 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
| 23 |
|
| 24 | You find the tainted state near the top in a line starting with 'CPU:'; if or
|
| 25 | why the kernel was tainted is shown after the Process ID ('PID:') and a shortened
|
| 26 | name of the command ('Comm:') that triggered the event::
|
| 27 |
|
| 28 | BUG: unable to handle kernel NULL pointer dereference at 0000000000000000
|
| 29 | Oops: 0002 [#1] SMP PTI
|
| 30 | CPU: 0 PID: 4424 Comm: insmod Tainted: P W O 4.20.0-0.rc6.fc30 #1
|
| 31 | Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
|
| 32 | RIP: 0010:my_oops_init+0x13/0x1000 [kpanic]
|
| 33 | [...]
|
| 34 |
|
| 35 | You'll find a 'Not tainted: ' there if the kernel was not tainted at the
|
| 36 | time of the event; if it was, then it will print 'Tainted: ' and characters
|
| 37 | either letters or blanks. In the example above it looks like this::
|
| 38 |
|
| 39 | Tainted: P W O
|
| 40 |
|
| 41 | The meaning of those characters is explained in the table below. In this case
|
| 42 | the kernel got tainted earlier because a proprietary Module (``P``) was loaded,
|
| 43 | a warning occurred (``W``), and an externally-built module was loaded (``O``).
|
| 44 | To decode other letters use the table below.
|
| 45 |
|
| 46 |
|
| 47 | Decoding tainted state at runtime
|
| 48 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
| 49 |
|
| 50 | At runtime, you can query the tainted state by reading
|
| 51 | ``cat /proc/sys/kernel/tainted``. If that returns ``0``, the kernel is not
|
| 52 | tainted; any other number indicates the reasons why it is. The easiest way to
|
| 53 | decode that number is the script ``tools/debugging/kernel-chktaint``, which your
|
| 54 | distribution might ship as part of a package called ``linux-tools`` or
|
| 55 | ``kernel-tools``; if it doesn't, you can download the script from
|
| 56 | `git.kernel.org <https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/plain/tools/debugging/kernel-chktaint>`_
|
| 57 | and execute it with ``sh kernel-chktaint``, which would print something like
|
| 58 | this on the machine that had the statements in the logs that were quoted earlier::
|
| 59 |
|
| 60 | Kernel is Tainted for following reasons:
|
| 61 | * Proprietary module was loaded (#0)
|
| 62 | * Kernel issued warning (#9)
|
| 63 | * Externally-built ('out-of-tree') module was loaded (#12)
|
| 64 | See Documentation/admin-guide/tainted-kernels.rst in the Linux kernel or
|
| 65 | https://www.kernel.org/doc/html/latest/admin-guide/tainted-kernels.html for
|
| 66 | a more details explanation of the various taint flags.
|
| 67 | Raw taint value as int/string: 4609/'P W O '
|
| 68 |
|
| 69 | You can try to decode the number yourself. That's easy if there was only one
|
| 70 | reason that got your kernel tainted, as in this case you can find the number
|
| 71 | with the table below. If there were multiple reasons you need to decode the
|
| 72 | number, as it is a bitfield, where each bit indicates the absence or presence of
|
| 73 | a particular type of taint. It's best to leave that to the aforementioned
|
| 74 | script, but if you need something quick you can use this shell command to check
|
| 75 | which bits are set::
|
| 76 |
|
| 77 | $ for i in $(seq 20); do echo $(($i-1)) $(($(cat /proc/sys/kernel/tainted)>>($i-1)&1));done
|
| 78 |
|
| 79 | Table for decoding tainted state
|
| 80 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
| 81 |
|
| 82 | === === ====== ========================================================
|
| 83 | Bit Log Number Reason that got the kernel tainted
|
| 84 | === === ====== ========================================================
|
| 85 | 0 G/P 1 proprietary module was loaded
|
| 86 | 1 _/F 2 module was force loaded
|
| 87 | 2 _/S 4 kernel running on an out of specification system
|
| 88 | 3 _/R 8 module was force unloaded
|
| 89 | 4 _/M 16 processor reported a Machine Check Exception (MCE)
|
| 90 | 5 _/B 32 bad page referenced or some unexpected page flags
|
| 91 | 6 _/U 64 taint requested by userspace application
|
| 92 | 7 _/D 128 kernel died recently, i.e. there was an OOPS or BUG
|
| 93 | 8 _/A 256 ACPI table overridden by user
|
| 94 | 9 _/W 512 kernel issued warning
|
| 95 | 10 _/C 1024 staging driver was loaded
|
| 96 | 11 _/I 2048 workaround for bug in platform firmware applied
|
| 97 | 12 _/O 4096 externally-built ("out-of-tree") module was loaded
|
| 98 | 13 _/E 8192 unsigned module was loaded
|
| 99 | 14 _/L 16384 soft lockup occurred
|
| 100 | 15 _/K 32768 kernel has been live patched
|
| 101 | 16 _/X 65536 auxiliary taint, defined for and used by distros
|
| 102 | 17 _/T 131072 kernel was built with the struct randomization plugin
|
| 103 | 18 _/N 262144 an in-kernel test has been run
|
| 104 | 19 _/J 524288 userspace used a mutating debug operation in fwctl
|
| 105 | === === ====== ========================================================
|
| 106 |
|
| 107 | Note: The character ``_`` is representing a blank in this table to make reading
|
| 108 | easier.
|
| 109 |
|
| 110 | More detailed explanation for tainting
|
| 111 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
| 112 |
|
| 113 | 0) ``G`` if all modules loaded have a GPL or compatible license, ``P`` if
|
| 114 | any proprietary module has been loaded. Modules without a
|
| 115 | MODULE_LICENSE or with a MODULE_LICENSE that is not recognised by
|
| 116 | insmod as GPL compatible are assumed to be proprietary.
|
| 117 |
|
| 118 | 1) ``F`` if any module was force loaded by ``insmod -f``, ``' '`` if all
|
| 119 | modules were loaded normally.
|
| 120 |
|
| 121 | 2) ``S`` if the kernel is running on a processor or system that is out of
|
| 122 | specification: hardware has been put into an unsupported configuration,
|
| 123 | therefore proper execution cannot be guaranteed.
|
| 124 | Kernel will be tainted if, for example:
|
| 125 |
|
| 126 | - on x86: PAE is forced through forcepae on intel CPUs (such as Pentium M)
|
| 127 | which do not report PAE but may have a functional implementation, an SMP
|
| 128 | kernel is running on non officially capable SMP Athlon CPUs, MSRs are
|
| 129 | being poked at from userspace.
|
| 130 | - on arm: kernel running on certain CPUs (such as Keystone 2) without
|
| 131 | having certain kernel features enabled.
|
| 132 | - on arm64: there are mismatched hardware features between CPUs, the
|
| 133 | bootloader has booted CPUs in different modes.
|
| 134 | - certain drivers are being used on non supported architectures (such as
|
| 135 | scsi/snic on something else than x86_64, scsi/ips on non
|
| 136 | x86/x86_64/itanium, have broken firmware settings for the
|
| 137 | irqchip/irq-gic on arm64 ...).
|
| 138 | - x86/x86_64: Microcode late loading is dangerous and will result in
|
| 139 | tainting the kernel. It requires that all CPUs rendezvous to make sure
|
| 140 | the update happens when the system is as quiescent as possible. However,
|
| 141 | a higher priority MCE/SMI/NMI can move control flow away from that
|
| 142 | rendezvous and interrupt the update, which can be detrimental to the
|
| 143 | machine.
|
| 144 |
|
| 145 | 3) ``R`` if a module was force unloaded by ``rmmod -f``, ``' '`` if all
|
| 146 | modules were unloaded normally.
|
| 147 |
|
| 148 | 4) ``M`` if any processor has reported a Machine Check Exception,
|
| 149 | ``' '`` if no Machine Check Exceptions have occurred.
|
| 150 |
|
| 151 | 5) ``B`` If a page-release function has found a bad page reference or some
|
| 152 | unexpected page flags. This indicates a hardware problem or a kernel bug;
|
| 153 | there should be other information in the log indicating why this tainting
|
| 154 | occurred.
|
| 155 |
|
| 156 | 6) ``U`` if a user or user application specifically requested that the
|
| 157 | Tainted flag be set, ``' '`` otherwise.
|
| 158 |
|
| 159 | 7) ``D`` if the kernel has died recently, i.e. there was an OOPS or BUG.
|
| 160 |
|
| 161 | 8) ``A`` if an ACPI table has been overridden.
|
| 162 |
|
| 163 | 9) ``W`` if a warning has previously been issued by the kernel.
|
| 164 | (Though some warnings may set more specific taint flags.)
|
| 165 |
|
| 166 | 10) ``C`` if a staging driver has been loaded.
|
| 167 |
|
| 168 | 11) ``I`` if the kernel is working around a severe bug in the platform
|
| 169 | firmware (BIOS or similar).
|
| 170 |
|
| 171 | 12) ``O`` if an externally-built ("out-of-tree") module has been loaded.
|
| 172 |
|
| 173 | 13) ``E`` if an unsigned module has been loaded in a kernel supporting
|
| 174 | module signature.
|
| 175 |
|
| 176 | 14) ``L`` if a soft lockup has previously occurred on the system.
|
| 177 |
|
| 178 | 15) ``K`` if the kernel has been live patched.
|
| 179 |
|
| 180 | 16) ``X`` Auxiliary taint, defined for and used by Linux distributors.
|
| 181 |
|
| 182 | 17) ``T`` Kernel was build with the randstruct plugin, which can intentionally
|
| 183 | produce extremely unusual kernel structure layouts (even performance
|
| 184 | pathological ones), which is important to know when debugging. Set at
|
| 185 | build time.
|
| 186 |
|
| 187 | 18) ``N`` if an in-kernel test, such as a KUnit test, has been run.
|
| 188 |
|
| 189 | 19) ``J`` if userspace opened /dev/fwctl/* and performed a FWTCL_RPC_DEBUG_WRITE
|
| 190 | to use the devices debugging features. Device debugging features could
|
| 191 | cause the device to malfunction in undefined ways.
|