| 1 | |
| 2 |
|
| 3 | ==========================
|
| 4 | The Linux Microcode Loader
|
| 5 | ==========================
|
| 6 |
|
| 7 | :Authors: - Fenghua Yu <[email protected]>
|
| 8 | - Borislav Petkov <[email protected]>
|
| 9 | - Ashok Raj <[email protected]>
|
| 10 |
|
| 11 | The kernel has a x86 microcode loading facility which is supposed to
|
| 12 | provide microcode loading methods in the OS. Potential use cases are
|
| 13 | updating the microcode on platforms beyond the OEM End-Of-Life support,
|
| 14 | and updating the microcode on long-running systems without rebooting.
|
| 15 |
|
| 16 | The loader supports three loading methods:
|
| 17 |
|
| 18 | Early load microcode
|
| 19 | ====================
|
| 20 |
|
| 21 | The kernel can update microcode very early during boot. Loading
|
| 22 | microcode early can fix CPU issues before they are observed during
|
| 23 | kernel boot time.
|
| 24 |
|
| 25 | The microcode is stored in an initrd file. During boot, it is read from
|
| 26 | it and loaded into the CPU cores.
|
| 27 |
|
| 28 | The format of the combined initrd image is microcode in (uncompressed)
|
| 29 | cpio format followed by the (possibly compressed) initrd image. The
|
| 30 | loader parses the combined initrd image during boot.
|
| 31 |
|
| 32 | The microcode files in cpio name space are:
|
| 33 |
|
| 34 | on Intel:
|
| 35 | kernel/x86/microcode/GenuineIntel.bin
|
| 36 | on AMD :
|
| 37 | kernel/x86/microcode/AuthenticAMD.bin
|
| 38 |
|
| 39 | During BSP (BootStrapping Processor) boot (pre-SMP), the kernel
|
| 40 | scans the microcode file in the initrd. If microcode matching the
|
| 41 | CPU is found, it will be applied in the BSP and later on in all APs
|
| 42 | (Application Processors).
|
| 43 |
|
| 44 | The loader also saves the matching microcode for the CPU in memory.
|
| 45 | Thus, the cached microcode patch is applied when CPUs resume from a
|
| 46 | sleep state.
|
| 47 |
|
| 48 | Here's a crude example how to prepare an initrd with microcode (this is
|
| 49 | normally done automatically by the distribution, when recreating the
|
| 50 | initrd, so you don't really have to do it yourself. It is documented
|
| 51 | here for future reference only).
|
| 52 | ::
|
| 53 |
|
| 54 | #!/bin/bash
|
| 55 |
|
| 56 | if [ -z "$1" ]; then
|
| 57 | echo "You need to supply an initrd file"
|
| 58 | exit 1
|
| 59 | fi
|
| 60 |
|
| 61 | INITRD="$1"
|
| 62 |
|
| 63 | DSTDIR=kernel/x86/microcode
|
| 64 | TMPDIR=/tmp/initrd
|
| 65 |
|
| 66 | rm -rf $TMPDIR
|
| 67 |
|
| 68 | mkdir $TMPDIR
|
| 69 | cd $TMPDIR
|
| 70 | mkdir -p $DSTDIR
|
| 71 |
|
| 72 | if [ -d /lib/firmware/amd-ucode ]; then
|
| 73 | cat /lib/firmware/amd-ucode/microcode_amd*.bin > $DSTDIR/AuthenticAMD.bin
|
| 74 | fi
|
| 75 |
|
| 76 | if [ -d /lib/firmware/intel-ucode ]; then
|
| 77 | cat /lib/firmware/intel-ucode/* > $DSTDIR/GenuineIntel.bin
|
| 78 | fi
|
| 79 |
|
| 80 | find . | cpio -o -H newc >../ucode.cpio
|
| 81 | cd ..
|
| 82 | mv $INITRD $INITRD.orig
|
| 83 | cat ucode.cpio $INITRD.orig > $INITRD
|
| 84 |
|
| 85 | rm -rf $TMPDIR
|
| 86 |
|
| 87 |
|
| 88 | The system needs to have the microcode packages installed into
|
| 89 | /lib/firmware or you need to fixup the paths above if yours are
|
| 90 | somewhere else and/or you've downloaded them directly from the processor
|
| 91 | vendor's site.
|
| 92 |
|
| 93 | Late loading
|
| 94 | ============
|
| 95 |
|
| 96 | You simply install the microcode packages your distro supplies and
|
| 97 | run::
|
| 98 |
|
| 99 | # echo 1 > /sys/devices/system/cpu/microcode/reload
|
| 100 |
|
| 101 | as root.
|
| 102 |
|
| 103 | The loading mechanism looks for microcode blobs in
|
| 104 | /lib/firmware/{intel-ucode,amd-ucode}. The default distro installation
|
| 105 | packages already put them there.
|
| 106 |
|
| 107 | Since kernel 5.19, late loading is not enabled by default.
|
| 108 |
|
| 109 | The /dev/cpu/microcode method has been removed in 5.19.
|
| 110 |
|
| 111 | Why is late loading dangerous?
|
| 112 | ==============================
|
| 113 |
|
| 114 | Synchronizing all CPUs
|
| 115 | ----------------------
|
| 116 |
|
| 117 | The microcode engine which receives the microcode update is shared
|
| 118 | between the two logical threads in a SMT system. Therefore, when
|
| 119 | the update is executed on one SMT thread of the core, the sibling
|
| 120 | "automatically" gets the update.
|
| 121 |
|
| 122 | Since the microcode can "simulate" MSRs too, while the microcode update
|
| 123 | is in progress, those simulated MSRs transiently cease to exist. This
|
| 124 | can result in unpredictable results if the SMT sibling thread happens to
|
| 125 | be in the middle of an access to such an MSR. The usual observation is
|
| 126 | that such MSR accesses cause #GPs to be raised to signal that former are
|
| 127 | not present.
|
| 128 |
|
| 129 | The disappearing MSRs are just one common issue which is being observed.
|
| 130 | Any other instruction that's being patched and gets concurrently
|
| 131 | executed by the other SMT sibling, can also result in similar,
|
| 132 | unpredictable behavior.
|
| 133 |
|
| 134 | To eliminate this case, a stop_machine()-based CPU synchronization was
|
| 135 | introduced as a way to guarantee that all logical CPUs will not execute
|
| 136 | any code but just wait in a spin loop, polling an atomic variable.
|
| 137 |
|
| 138 | While this took care of device or external interrupts, IPIs including
|
| 139 | LVT ones, such as CMCI etc, it cannot address other special interrupts
|
| 140 | that can't be shut off. Those are Machine Check (#MC), System Management
|
| 141 | (#SMI) and Non-Maskable interrupts (#NMI).
|
| 142 |
|
| 143 | Machine Checks
|
| 144 | --------------
|
| 145 |
|
| 146 | Machine Checks (#MC) are non-maskable. There are two kinds of MCEs.
|
| 147 | Fatal un-recoverable MCEs and recoverable MCEs. While un-recoverable
|
| 148 | errors are fatal, recoverable errors can also happen in kernel context
|
| 149 | are also treated as fatal by the kernel.
|
| 150 |
|
| 151 | On certain Intel machines, MCEs are also broadcast to all threads in a
|
| 152 | system. If one thread is in the middle of executing WRMSR, a MCE will be
|
| 153 | taken at the end of the flow. Either way, they will wait for the thread
|
| 154 | performing the wrmsr(0x79) to rendezvous in the MCE handler and shutdown
|
| 155 | eventually if any of the threads in the system fail to check in to the
|
| 156 | MCE rendezvous.
|
| 157 |
|
| 158 | To be paranoid and get predictable behavior, the OS can choose to set
|
| 159 | MCG_STATUS.MCIP. Since MCEs can be at most one in a system, if an
|
| 160 | MCE was signaled, the above condition will promote to a system reset
|
| 161 | automatically. OS can turn off MCIP at the end of the update for that
|
| 162 | core.
|
| 163 |
|
| 164 | System Management Interrupt
|
| 165 | ---------------------------
|
| 166 |
|
| 167 | SMIs are also broadcast to all CPUs in the platform. Microcode update
|
| 168 | requests exclusive access to the core before writing to MSR 0x79. So if
|
| 169 | it does happen such that, one thread is in WRMSR flow, and the 2nd got
|
| 170 | an SMI, that thread will be stopped in the first instruction in the SMI
|
| 171 | handler.
|
| 172 |
|
| 173 | Since the secondary thread is stopped in the first instruction in SMI,
|
| 174 | there is very little chance that it would be in the middle of executing
|
| 175 | an instruction being patched. Plus OS has no way to stop SMIs from
|
| 176 | happening.
|
| 177 |
|
| 178 | Non-Maskable Interrupts
|
| 179 | -----------------------
|
| 180 |
|
| 181 | When thread0 of a core is doing the microcode update, if thread1 is
|
| 182 | pulled into NMI, that can cause unpredictable behavior due to the
|
| 183 | reasons above.
|
| 184 |
|
| 185 | OS can choose a variety of methods to avoid running into this situation.
|
| 186 |
|
| 187 |
|
| 188 | Is the microcode suitable for late loading?
|
| 189 | -------------------------------------------
|
| 190 |
|
| 191 | Late loading is done when the system is fully operational and running
|
| 192 | real workloads. Late loading behavior depends on what the base patch on
|
| 193 | the CPU is before upgrading to the new patch.
|
| 194 |
|
| 195 | This is true for Intel CPUs.
|
| 196 |
|
| 197 | Consider, for example, a CPU has patch level 1 and the update is to
|
| 198 | patch level 3.
|
| 199 |
|
| 200 | Between patch1 and patch3, patch2 might have deprecated a software-visible
|
| 201 | feature.
|
| 202 |
|
| 203 | This is unacceptable if software is even potentially using that feature.
|
| 204 | For instance, say MSR_X is no longer available after an update,
|
| 205 | accessing that MSR will cause a #GP fault.
|
| 206 |
|
| 207 | Basically there is no way to declare a new microcode update suitable
|
| 208 | for late-loading. This is another one of the problems that caused late
|
| 209 | loading to be not enabled by default.
|
| 210 |
|
| 211 | Builtin microcode
|
| 212 | =================
|
| 213 |
|
| 214 | The loader supports also loading of a builtin microcode supplied through
|
| 215 | the regular builtin firmware method CONFIG_EXTRA_FIRMWARE. Only 64-bit is
|
| 216 | currently supported.
|
| 217 |
|
| 218 | Here's an example::
|
| 219 |
|
| 220 | CONFIG_EXTRA_FIRMWARE="intel-ucode/06-3a-09 amd-ucode/microcode_amd_fam15h.bin"
|
| 221 | CONFIG_EXTRA_FIRMWARE_DIR="/lib/firmware"
|
| 222 |
|
| 223 | This basically means, you have the following tree structure locally::
|
| 224 |
|
| 225 | /lib/firmware/
|
| 226 | |-- amd-ucode
|
| 227 | ...
|
| 228 | | |-- microcode_amd_fam15h.bin
|
| 229 | ...
|
| 230 | |-- intel-ucode
|
| 231 | ...
|
| 232 | | |-- 06-3a-09
|
| 233 | ...
|
| 234 |
|
| 235 | so that the build system can find those files and integrate them into
|
| 236 | the final kernel image. The early loader finds them and applies them.
|
| 237 |
|
| 238 | Needless to say, this method is not the most flexible one because it
|
| 239 | requires rebuilding the kernel each time updated microcode from the CPU
|
| 240 | vendor is available.
|