| 1 | |
| 2 |
|
| 3 | =================
|
| 4 | KVM Lock Overview
|
| 5 | =================
|
| 6 |
|
| 7 | 1. Acquisition Orders
|
| 8 | ---------------------
|
| 9 |
|
| 10 | The acquisition orders for mutexes are as follows:
|
| 11 |
|
| 12 | - cpus_read_lock() is taken outside kvm_lock
|
| 13 |
|
| 14 | - kvm_usage_lock is taken outside cpus_read_lock()
|
| 15 |
|
| 16 | - kvm->lock is taken outside vcpu->mutex
|
| 17 |
|
| 18 | - kvm->lock is taken outside kvm->slots_lock and kvm->irq_lock
|
| 19 |
|
| 20 | - vcpu->mutex is taken outside kvm->slots_lock and kvm->slots_arch_lock
|
| 21 |
|
| 22 | - kvm->slots_lock is taken outside kvm->irq_lock, though acquiring
|
| 23 | them together is quite rare.
|
| 24 |
|
| 25 | - kvm->mn_active_invalidate_count ensures that pairs of
|
| 26 | invalidate_range_start() and invalidate_range_end() callbacks
|
| 27 | use the same memslots array. kvm->slots_lock and kvm->slots_arch_lock
|
| 28 | are taken on the waiting side when modifying memslots, so MMU notifiers
|
| 29 | must not take either kvm->slots_lock or kvm->slots_arch_lock.
|
| 30 |
|
| 31 | cpus_read_lock() vs kvm_lock:
|
| 32 |
|
| 33 | - Taking cpus_read_lock() outside of kvm_lock is problematic, despite that
|
| 34 | being the official ordering, as it is quite easy to unknowingly trigger
|
| 35 | cpus_read_lock() while holding kvm_lock. Use caution when walking vm_list,
|
| 36 | e.g. avoid complex operations when possible.
|
| 37 |
|
| 38 | For SRCU:
|
| 39 |
|
| 40 | - ``synchronize_srcu(&kvm->srcu)`` is called inside critical sections
|
| 41 | for kvm->lock, vcpu->mutex and kvm->slots_lock. These locks _cannot_
|
| 42 | be taken inside a kvm->srcu read-side critical section; that is, the
|
| 43 | following is broken::
|
| 44 |
|
| 45 | srcu_read_lock(&kvm->srcu);
|
| 46 | mutex_lock(&kvm->slots_lock);
|
| 47 |
|
| 48 | - kvm->slots_arch_lock instead is released before the call to
|
| 49 | ``synchronize_srcu()``. It _can_ therefore be taken inside a
|
| 50 | kvm->srcu read-side critical section, for example while processing
|
| 51 | a vmexit.
|
| 52 |
|
| 53 | On x86:
|
| 54 |
|
| 55 | - vcpu->mutex is taken outside kvm->arch.hyperv.hv_lock and kvm->arch.xen.xen_lock
|
| 56 |
|
| 57 | - kvm->arch.mmu_lock is an rwlock; critical sections for
|
| 58 | kvm->arch.tdp_mmu_pages_lock and kvm->arch.mmu_unsync_pages_lock must
|
| 59 | also take kvm->arch.mmu_lock
|
| 60 |
|
| 61 | Everything else is a leaf: no other lock is taken inside the critical
|
| 62 | sections.
|
| 63 |
|
| 64 | 2. Exception
|
| 65 | ------------
|
| 66 |
|
| 67 | Fast page fault:
|
| 68 |
|
| 69 | Fast page fault is the fast path which fixes the guest page fault out of
|
| 70 | the mmu-lock on x86. Currently, the page fault can be fast in one of the
|
| 71 | following two cases:
|
| 72 |
|
| 73 | 1. Access Tracking: The SPTE is not present, but it is marked for access
|
| 74 | tracking. That means we need to restore the saved R/X bits. This is
|
| 75 | described in more detail later below.
|
| 76 |
|
| 77 | 2. Write-Protection: The SPTE is present and the fault is caused by
|
| 78 | write-protect. That means we just need to change the W bit of the spte.
|
| 79 |
|
| 80 | What we use to avoid all the races is the Host-writable bit and MMU-writable bit
|
| 81 | on the spte:
|
| 82 |
|
| 83 | - Host-writable means the gfn is writable in the host kernel page tables and in
|
| 84 | its KVM memslot.
|
| 85 | - MMU-writable means the gfn is writable in the guest's mmu and it is not
|
| 86 | write-protected by shadow page write-protection.
|
| 87 |
|
| 88 | On fast page fault path, we will use cmpxchg to atomically set the spte W
|
| 89 | bit if spte.HOST_WRITEABLE = 1 and spte.WRITE_PROTECT = 1, to restore the saved
|
| 90 | R/X bits if for an access-traced spte, or both. This is safe because whenever
|
| 91 | changing these bits can be detected by cmpxchg.
|
| 92 |
|
| 93 | But we need carefully check these cases:
|
| 94 |
|
| 95 | 1) The mapping from gfn to pfn
|
| 96 |
|
| 97 | The mapping from gfn to pfn may be changed since we can only ensure the pfn
|
| 98 | is not changed during cmpxchg. This is a ABA problem, for example, below case
|
| 99 | will happen:
|
| 100 |
|
| 101 | +------------------------------------------------------------------------+
|
| 102 | | At the beginning:: |
|
| 103 | | |
|
| 104 | | gpte = gfn1 |
|
| 105 | | gfn1 is mapped to pfn1 on host |
|
| 106 | | spte is the shadow page table entry corresponding with gpte and |
|
| 107 | | spte = pfn1 |
|
| 108 | +------------------------------------------------------------------------+
|
| 109 | | On fast page fault path: |
|
| 110 | +------------------------------------+-----------------------------------+
|
| 111 | | CPU 0: | CPU 1: |
|
| 112 | +------------------------------------+-----------------------------------+
|
| 113 | | :: | |
|
| 114 | | | |
|
| 115 | | old_spte = *spte; | |
|
| 116 | +------------------------------------+-----------------------------------+
|
| 117 | | | pfn1 is swapped out:: |
|
| 118 | | | |
|
| 119 | | | spte = 0; |
|
| 120 | | | |
|
| 121 | | | pfn1 is re-alloced for gfn2. |
|
| 122 | | | |
|
| 123 | | | gpte is changed to point to |
|
| 124 | | | gfn2 by the guest:: |
|
| 125 | | | |
|
| 126 | | | spte = pfn1; |
|
| 127 | +------------------------------------+-----------------------------------+
|
| 128 | | :: |
|
| 129 | | |
|
| 130 | | if (cmpxchg(spte, old_spte, old_spte+W) |
|
| 131 | | mark_page_dirty(vcpu->kvm, gfn1) |
|
| 132 | | OOPS!!! |
|
| 133 | +------------------------------------------------------------------------+
|
| 134 |
|
| 135 | We dirty-log for gfn1, that means gfn2 is lost in dirty-bitmap.
|
| 136 |
|
| 137 | For direct sp, we can easily avoid it since the spte of direct sp is fixed
|
| 138 | to gfn. For indirect sp, we disabled fast page fault for simplicity.
|
| 139 |
|
| 140 | A solution for indirect sp could be to pin the gfn before the cmpxchg. After
|
| 141 | the pinning:
|
| 142 |
|
| 143 | - We have held the refcount of pfn; that means the pfn can not be freed and
|
| 144 | be reused for another gfn.
|
| 145 | - The pfn is writable and therefore it cannot be shared between different gfns
|
| 146 | by KSM.
|
| 147 |
|
| 148 | Then, we can ensure the dirty bitmaps is correctly set for a gfn.
|
| 149 |
|
| 150 | 2) Dirty bit tracking
|
| 151 |
|
| 152 | In the original code, the spte can be fast updated (non-atomically) if the
|
| 153 | spte is read-only and the Accessed bit has already been set since the
|
| 154 | Accessed bit and Dirty bit can not be lost.
|
| 155 |
|
| 156 | But it is not true after fast page fault since the spte can be marked
|
| 157 | writable between reading spte and updating spte. Like below case:
|
| 158 |
|
| 159 | +-------------------------------------------------------------------------+
|
| 160 | | At the beginning:: |
|
| 161 | | |
|
| 162 | | spte.W = 0 |
|
| 163 | | spte.Accessed = 1 |
|
| 164 | +-------------------------------------+-----------------------------------+
|
| 165 | | CPU 0: | CPU 1: |
|
| 166 | +-------------------------------------+-----------------------------------+
|
| 167 | | In mmu_spte_update():: | |
|
| 168 | | | |
|
| 169 | | old_spte = *spte; | |
|
| 170 | | | |
|
| 171 | | | |
|
| 172 | | /* 'if' condition is satisfied. */ | |
|
| 173 | | if (old_spte.Accessed == 1 && | |
|
| 174 | | old_spte.W == 0) | |
|
| 175 | | spte = new_spte; | |
|
| 176 | +-------------------------------------+-----------------------------------+
|
| 177 | | | on fast page fault path:: |
|
| 178 | | | |
|
| 179 | | | spte.W = 1 |
|
| 180 | | | |
|
| 181 | | | memory write on the spte:: |
|
| 182 | | | |
|
| 183 | | | spte.Dirty = 1 |
|
| 184 | +-------------------------------------+-----------------------------------+
|
| 185 | | :: | |
|
| 186 | | | |
|
| 187 | | else | |
|
| 188 | | old_spte = xchg(spte, new_spte);| |
|
| 189 | | if (old_spte.Accessed && | |
|
| 190 | | !new_spte.Accessed) | |
|
| 191 | | flush = true; | |
|
| 192 | | if (old_spte.Dirty && | |
|
| 193 | | !new_spte.Dirty) | |
|
| 194 | | flush = true; | |
|
| 195 | | OOPS!!! | |
|
| 196 | +-------------------------------------+-----------------------------------+
|
| 197 |
|
| 198 | The Dirty bit is lost in this case.
|
| 199 |
|
| 200 | In order to avoid this kind of issue, we always treat the spte as "volatile"
|
| 201 | if it can be updated out of mmu-lock [see spte_needs_atomic_update()]; it means
|
| 202 | the spte is always atomically updated in this case.
|
| 203 |
|
| 204 | 3) flush tlbs due to spte updated
|
| 205 |
|
| 206 | If the spte is updated from writable to read-only, we should flush all TLBs,
|
| 207 | otherwise rmap_write_protect will find a read-only spte, even though the
|
| 208 | writable spte might be cached on a CPU's TLB.
|
| 209 |
|
| 210 | As mentioned before, the spte can be updated to writable out of mmu-lock on
|
| 211 | fast page fault path. In order to easily audit the path, we see if TLBs needing
|
| 212 | to be flushed caused this reason in mmu_spte_update() since this is a common
|
| 213 | function to update spte (present -> present).
|
| 214 |
|
| 215 | Since the spte is "volatile" if it can be updated out of mmu-lock, we always
|
| 216 | atomically update the spte and the race caused by fast page fault can be avoided.
|
| 217 | See the comments in spte_needs_atomic_update() and mmu_spte_update().
|
| 218 |
|
| 219 | Lockless Access Tracking:
|
| 220 |
|
| 221 | This is used for Intel CPUs that are using EPT but do not support the EPT A/D
|
| 222 | bits. In this case, PTEs are tagged as A/D disabled (using ignored bits), and
|
| 223 | when the KVM MMU notifier is called to track accesses to a page (via
|
| 224 | kvm_mmu_notifier_clear_flush_young), it marks the PTE not-present in hardware
|
| 225 | by clearing the RWX bits in the PTE and storing the original R & X bits in more
|
| 226 | unused/ignored bits. When the VM tries to access the page later on, a fault is
|
| 227 | generated and the fast page fault mechanism described above is used to
|
| 228 | atomically restore the PTE to a Present state. The W bit is not saved when the
|
| 229 | PTE is marked for access tracking and during restoration to the Present state,
|
| 230 | the W bit is set depending on whether or not it was a write access. If it
|
| 231 | wasn't, then the W bit will remain clear until a write access happens, at which
|
| 232 | time it will be set using the Dirty tracking mechanism described above.
|
| 233 |
|
| 234 | 3. Reference
|
| 235 | ------------
|
| 236 |
|
| 237 | ``kvm_lock``
|
| 238 | ^^^^^^^^^^^^
|
| 239 |
|
| 240 | :Type: mutex
|
| 241 | :Arch: any
|
| 242 | :Protects: - vm_list
|
| 243 |
|
| 244 | ``kvm_usage_lock``
|
| 245 | ^^^^^^^^^^^^^^^^^^
|
| 246 |
|
| 247 | :Type: mutex
|
| 248 | :Arch: any
|
| 249 | :Protects: - kvm_usage_count
|
| 250 | - hardware virtualization enable/disable
|
| 251 | :Comment: Exists to allow taking cpus_read_lock() while kvm_usage_count is
|
| 252 | protected, which simplifies the virtualization enabling logic.
|
| 253 |
|
| 254 | ``kvm->mn_invalidate_lock``
|
| 255 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 256 |
|
| 257 | :Type: spinlock_t
|
| 258 | :Arch: any
|
| 259 | :Protects: mn_active_invalidate_count, mn_memslots_update_rcuwait
|
| 260 |
|
| 261 | ``kvm_arch::tsc_write_lock``
|
| 262 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 263 |
|
| 264 | :Type: raw_spinlock_t
|
| 265 | :Arch: x86
|
| 266 | :Protects: - kvm_arch::{last_tsc_write,last_tsc_nsec,last_tsc_offset}
|
| 267 | - tsc offset in vmcb
|
| 268 | :Comment: 'raw' because updating the tsc offsets must not be preempted.
|
| 269 |
|
| 270 | ``kvm->mmu_lock``
|
| 271 | ^^^^^^^^^^^^^^^^^
|
| 272 | :Type: spinlock_t or rwlock_t
|
| 273 | :Arch: any
|
| 274 | :Protects: -shadow page/shadow tlb entry
|
| 275 | :Comment: it is a spinlock since it is used in mmu notifier.
|
| 276 |
|
| 277 | ``kvm->srcu``
|
| 278 | ^^^^^^^^^^^^^
|
| 279 | :Type: srcu lock
|
| 280 | :Arch: any
|
| 281 | :Protects: - kvm->memslots
|
| 282 | - kvm->buses
|
| 283 | :Comment: The srcu read lock must be held while accessing memslots (e.g.
|
| 284 | when using gfn_to_* functions) and while accessing in-kernel
|
| 285 | MMIO/PIO address->device structure mapping (kvm->buses).
|
| 286 | The srcu index can be stored in kvm_vcpu->srcu_idx per vcpu
|
| 287 | if it is needed by multiple functions.
|
| 288 |
|
| 289 | ``kvm->slots_arch_lock``
|
| 290 | ^^^^^^^^^^^^^^^^^^^^^^^^
|
| 291 | :Type: mutex
|
| 292 | :Arch: any (only needed on x86 though)
|
| 293 | :Protects: any arch-specific fields of memslots that have to be modified
|
| 294 | in a ``kvm->srcu`` read-side critical section.
|
| 295 | :Comment: must be held before reading the pointer to the current memslots,
|
| 296 | until after all changes to the memslots are complete
|
| 297 |
|
| 298 | ``wakeup_vcpus_on_cpu_lock``
|
| 299 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 300 | :Type: spinlock_t
|
| 301 | :Arch: x86
|
| 302 | :Protects: wakeup_vcpus_on_cpu
|
| 303 | :Comment: This is a per-CPU lock and it is used for VT-d posted-interrupts.
|
| 304 | When VT-d posted-interrupts are supported and the VM has assigned
|
| 305 | devices, we put the blocked vCPU on the list blocked_vcpu_on_cpu
|
| 306 | protected by blocked_vcpu_on_cpu_lock. When VT-d hardware issues
|
| 307 | wakeup notification event since external interrupts from the
|
| 308 | assigned devices happens, we will find the vCPU on the list to
|
| 309 | wakeup.
|
| 310 |
|
| 311 | ``vendor_module_lock``
|
| 312 | ^^^^^^^^^^^^^^^^^^^^^^
|
| 313 | :Type: mutex
|
| 314 | :Arch: x86
|
| 315 | :Protects: loading a vendor module (kvm_amd or kvm_intel)
|
| 316 | :Comment: Exists because using kvm_lock leads to deadlock. kvm_lock is taken
|
| 317 | in notifiers, e.g. __kvmclock_cpufreq_notifier(), that may be invoked while
|
| 318 | cpu_hotplug_lock is held, e.g. from cpufreq_boost_trigger_state(), and many
|
| 319 | operations need to take cpu_hotplug_lock when loading a vendor module, e.g.
|
| 320 | updating static calls.
|