| 1 | Entry/exit handling for exceptions, interrupts, syscalls and KVM
|
| 2 | ================================================================
|
| 3 |
|
| 4 | All transitions between execution domains require state updates which are
|
| 5 | subject to strict ordering constraints. State updates are required for the
|
| 6 | following:
|
| 7 |
|
| 8 | * Lockdep
|
| 9 | * RCU / Context tracking
|
| 10 | * Preemption counter
|
| 11 | * Tracing
|
| 12 | * Time accounting
|
| 13 |
|
| 14 | The update order depends on the transition type and is explained below in
|
| 15 | the transition type sections: `Syscalls`_, `KVM`_, `Interrupts and regular
|
| 16 | exceptions`_, `NMI and NMI-like exceptions`_.
|
| 17 |
|
| 18 | Non-instrumentable code - noinstr
|
| 19 | ---------------------------------
|
| 20 |
|
| 21 | Most instrumentation facilities depend on RCU, so instrumentation is prohibited
|
| 22 | for entry code before RCU starts watching and exit code after RCU stops
|
| 23 | watching. In addition, many architectures must save and restore register state,
|
| 24 | which means that (for example) a breakpoint in the breakpoint entry code would
|
| 25 | overwrite the debug registers of the initial breakpoint.
|
| 26 |
|
| 27 | Such code must be marked with the 'noinstr' attribute, placing that code into a
|
| 28 | special section inaccessible to instrumentation and debug facilities. Some
|
| 29 | functions are partially instrumentable, which is handled by marking them
|
| 30 | noinstr and using instrumentation_begin() and instrumentation_end() to flag the
|
| 31 | instrumentable ranges of code:
|
| 32 |
|
| 33 | .. code-block:: c
|
| 34 |
|
| 35 | noinstr void entry(void)
|
| 36 | {
|
| 37 | handle_entry(); // <-- must be 'noinstr' or '__always_inline'
|
| 38 | ...
|
| 39 |
|
| 40 | instrumentation_begin();
|
| 41 | handle_context(); // <-- instrumentable code
|
| 42 | instrumentation_end();
|
| 43 |
|
| 44 | ...
|
| 45 | handle_exit(); // <-- must be 'noinstr' or '__always_inline'
|
| 46 | }
|
| 47 |
|
| 48 | This allows verification of the 'noinstr' restrictions via objtool on
|
| 49 | supported architectures.
|
| 50 |
|
| 51 | Invoking non-instrumentable functions from instrumentable context has no
|
| 52 | restrictions and is useful to protect e.g. state switching which would
|
| 53 | cause malfunction if instrumented.
|
| 54 |
|
| 55 | All non-instrumentable entry/exit code sections before and after the RCU
|
| 56 | state transitions must run with interrupts disabled.
|
| 57 |
|
| 58 | Syscalls
|
| 59 | --------
|
| 60 |
|
| 61 | Syscall-entry code starts in assembly code and calls out into low-level C
|
| 62 | code after establishing low-level architecture-specific state and stack
|
| 63 | frames. This low-level C code must not be instrumented. The recommended
|
| 64 | syscall handling function invoked from low-level assembly code looks like
|
| 65 | this:
|
| 66 |
|
| 67 | .. code-block:: c
|
| 68 |
|
| 69 | noinstr void syscall(struct pt_regs *regs, long nr)
|
| 70 | {
|
| 71 | arch_syscall_enter(regs);
|
| 72 | result_reg(regs) = -ENOSYS;
|
| 73 | if (syscall_enter_from_user_mode_randomize_stack(regs, &nr)) {
|
| 74 | instrumentation_begin();
|
| 75 | if (valid(nr)
|
| 76 | result_reg(regs) = invoke_syscall(regs, nr);
|
| 77 | instrumentation_end();
|
| 78 | }
|
| 79 | syscall_exit_to_user_mode(regs);
|
| 80 | }
|
| 81 |
|
| 82 | This is the most resilent variant as it has always a guaranteed valid
|
| 83 | return code. The alternative variant is:
|
| 84 |
|
| 85 | .. code-block:: c
|
| 86 |
|
| 87 | noinstr void syscall(struct pt_regs *regs, long nr)
|
| 88 | {
|
| 89 | arch_syscall_enter(regs);
|
| 90 | if (syscall_enter_from_user_mode_randomize_stack(regs, &nr)) {
|
| 91 | instrumentation_begin();
|
| 92 | if (valid(nr)
|
| 93 | result_reg(regs) = invoke_syscall(regs, nr);
|
| 94 | else
|
| 95 | result_reg(regs) = -ENOSYS;
|
| 96 | instrumentation_end();
|
| 97 | }
|
| 98 | syscall_exit_to_user_mode(regs);
|
| 99 | }
|
| 100 |
|
| 101 | That works for most situations except when a probe/BPF attached to the
|
| 102 | syscall tracepoint sets an invalid syscall number e.g. -1 and also modifies
|
| 103 | the result register. So this variant will obviously overwrite the modified
|
| 104 | result with -ENOSYS.
|
| 105 |
|
| 106 | syscall_enter_from_user_mode_randomize_stack() first invokes
|
| 107 | enter_from_user_mode_randomize_stack() which establishes state in the
|
| 108 | following order:
|
| 109 |
|
| 110 | * Lockdep
|
| 111 | * RCU / Context tracking
|
| 112 | * Tracing
|
| 113 | * Apply stack randomization
|
| 114 |
|
| 115 | and then invokes the various entry work functions like ptrace, seccomp, audit,
|
| 116 | syscall tracing, etc. After all that is done, the instrumentable invoke_syscall
|
| 117 | function can be invoked. The instrumentable code section then ends, after which
|
| 118 | syscall_exit_to_user_mode() is invoked.
|
| 119 |
|
| 120 | syscall_exit_to_user_mode() handles all work which needs to be done before
|
| 121 | returning to user space like tracing, audit, signals, task work etc. After
|
| 122 | that it invokes exit_to_user_mode() which again handles the state
|
| 123 | transition in the reverse order:
|
| 124 |
|
| 125 | * Tracing
|
| 126 | * RCU / Context tracking
|
| 127 | * Lockdep
|
| 128 |
|
| 129 | syscall_enter_from_user_mode_randomize_stack() and
|
| 130 | syscall_exit_to_user_mode() are also available as fine grained subfunctions
|
| 131 | in cases where the architecture code has to do extra work between the
|
| 132 | various steps. In such cases it has to ensure that
|
| 133 | enter_from_user_mode_randomize_stack() is called first on entry and
|
| 134 | exit_to_user_mode() is called last on exit.
|
| 135 |
|
| 136 | Do not nest syscalls. Nested syscalls will cause RCU and/or context tracking
|
| 137 | to print a warning.
|
| 138 |
|
| 139 | KVM
|
| 140 | ---
|
| 141 |
|
| 142 | Entering or exiting guest mode is very similar to syscalls. From the host
|
| 143 | kernel point of view the CPU goes off into user space when entering the
|
| 144 | guest and returns to the kernel on exit.
|
| 145 |
|
| 146 | guest_state_enter_irqoff() is a KVM-specific variant of exit_to_user_mode()
|
| 147 | and guest_state_exit_irqoff() is the KVM variant of enter_from_user_mode().
|
| 148 | The state operations have the same ordering.
|
| 149 |
|
| 150 | Task work handling is done separately for guest at the boundary of the
|
| 151 | vcpu_run() loop via xfer_to_guest_mode_handle_work() which is a subset of
|
| 152 | the work handled on return to user space.
|
| 153 |
|
| 154 | Do not nest KVM entry/exit transitions because doing so is nonsensical.
|
| 155 |
|
| 156 | Interrupts and regular exceptions
|
| 157 | ---------------------------------
|
| 158 |
|
| 159 | Interrupts entry and exit handling is slightly more complex than syscalls
|
| 160 | and KVM transitions.
|
| 161 |
|
| 162 | If an interrupt is raised while the CPU executes in user space, the entry
|
| 163 | and exit handling is exactly the same as for syscalls.
|
| 164 |
|
| 165 | If the interrupt is raised while the CPU executes in kernel space the entry and
|
| 166 | exit handling is slightly different. RCU state is only updated when the
|
| 167 | interrupt is raised in the context of the CPU's idle task. Otherwise, RCU will
|
| 168 | already be watching. Lockdep and tracing have to be updated unconditionally.
|
| 169 |
|
| 170 | irqentry_enter() and irqentry_exit() provide the implementation for this.
|
| 171 |
|
| 172 | The architecture-specific part looks similar to syscall handling:
|
| 173 |
|
| 174 | .. code-block:: c
|
| 175 |
|
| 176 | noinstr void interrupt(struct pt_regs *regs, int nr)
|
| 177 | {
|
| 178 | arch_interrupt_enter(regs);
|
| 179 | state = irqentry_enter(regs);
|
| 180 |
|
| 181 | instrumentation_begin();
|
| 182 |
|
| 183 | irq_enter_rcu();
|
| 184 | invoke_irq_handler(regs, nr);
|
| 185 | irq_exit_rcu();
|
| 186 |
|
| 187 | instrumentation_end();
|
| 188 |
|
| 189 | irqentry_exit(regs, state);
|
| 190 | }
|
| 191 |
|
| 192 | Note that the invocation of the actual interrupt handler is within a
|
| 193 | irq_enter_rcu() and irq_exit_rcu() pair.
|
| 194 |
|
| 195 | irq_enter_rcu() updates the preemption count which makes in_hardirq()
|
| 196 | return true, handles NOHZ tick state and interrupt time accounting. This
|
| 197 | means that up to the point where irq_enter_rcu() is invoked in_hardirq()
|
| 198 | returns false.
|
| 199 |
|
| 200 | irq_exit_rcu() handles interrupt time accounting, undoes the preemption
|
| 201 | count update and eventually handles soft interrupts and NOHZ tick state.
|
| 202 |
|
| 203 | In theory, the preemption count could be updated in irqentry_enter(). In
|
| 204 | practice, deferring this update to irq_enter_rcu() allows the preemption-count
|
| 205 | code to be traced, while also maintaining symmetry with irq_exit_rcu() and
|
| 206 | irqentry_exit(), which are described in the next paragraph. The only downside
|
| 207 | is that the early entry code up to irq_enter_rcu() must be aware that the
|
| 208 | preemption count has not yet been updated with the HARDIRQ_OFFSET state.
|
| 209 |
|
| 210 | Note that irq_exit_rcu() must remove HARDIRQ_OFFSET from the preemption count
|
| 211 | before it handles soft interrupts, whose handlers must run in BH context rather
|
| 212 | than irq-disabled context. In addition, irqentry_exit() might schedule, which
|
| 213 | also requires that HARDIRQ_OFFSET has been removed from the preemption count.
|
| 214 |
|
| 215 | Even though interrupt handlers are expected to run with local interrupts
|
| 216 | disabled, interrupt nesting is common from an entry/exit perspective. For
|
| 217 | example, softirq handling happens within an irqentry_{enter,exit}() block with
|
| 218 | local interrupts enabled. Also, although uncommon, nothing prevents an
|
| 219 | interrupt handler from re-enabling interrupts.
|
| 220 |
|
| 221 | Interrupt entry/exit code doesn't strictly need to handle reentrancy, since it
|
| 222 | runs with local interrupts disabled. But NMIs can happen anytime, and a lot of
|
| 223 | the entry code is shared between the two.
|
| 224 |
|
| 225 | NMI and NMI-like exceptions
|
| 226 | ---------------------------
|
| 227 |
|
| 228 | NMIs and NMI-like exceptions (machine checks, double faults, debug
|
| 229 | interrupts, etc.) can hit any context and must be extra careful with
|
| 230 | the state.
|
| 231 |
|
| 232 | State changes for debug exceptions and machine-check exceptions depend on
|
| 233 | whether these exceptions happened in user-space (breakpoints or watchpoints) or
|
| 234 | in kernel mode (code patching). From user-space, they are treated like
|
| 235 | interrupts, while from kernel mode they are treated like NMIs.
|
| 236 |
|
| 237 | NMIs and other NMI-like exceptions handle state transitions without
|
| 238 | distinguishing between user-mode and kernel-mode origin.
|
| 239 |
|
| 240 | The state update on entry is handled in irqentry_nmi_enter() which updates
|
| 241 | state in the following order:
|
| 242 |
|
| 243 | * Preemption counter
|
| 244 | * Lockdep
|
| 245 | * RCU / Context tracking
|
| 246 | * Tracing
|
| 247 |
|
| 248 | The exit counterpart irqentry_nmi_exit() does the reverse operation in the
|
| 249 | reverse order.
|
| 250 |
|
| 251 | Note that the update of the preemption counter has to be the first
|
| 252 | operation on enter and the last operation on exit. The reason is that both
|
| 253 | lockdep and RCU rely on in_nmi() returning true in this case. The
|
| 254 | preemption count modification in the NMI entry/exit case must not be
|
| 255 | traced.
|
| 256 |
|
| 257 | Architecture-specific code looks like this:
|
| 258 |
|
| 259 | .. code-block:: c
|
| 260 |
|
| 261 | noinstr void nmi(struct pt_regs *regs)
|
| 262 | {
|
| 263 | arch_nmi_enter(regs);
|
| 264 | state = irqentry_nmi_enter(regs);
|
| 265 |
|
| 266 | instrumentation_begin();
|
| 267 | nmi_handler(regs);
|
| 268 | instrumentation_end();
|
| 269 |
|
| 270 | irqentry_nmi_exit(regs);
|
| 271 | }
|
| 272 |
|
| 273 | and for e.g. a debug exception it can look like this:
|
| 274 |
|
| 275 | .. code-block:: c
|
| 276 |
|
| 277 | noinstr void debug(struct pt_regs *regs)
|
| 278 | {
|
| 279 | arch_nmi_enter(regs);
|
| 280 |
|
| 281 | debug_regs = save_debug_regs();
|
| 282 |
|
| 283 | if (user_mode(regs)) {
|
| 284 | state = irqentry_enter(regs);
|
| 285 |
|
| 286 | instrumentation_begin();
|
| 287 | user_mode_debug_handler(regs, debug_regs);
|
| 288 | instrumentation_end();
|
| 289 |
|
| 290 | irqentry_exit(regs, state);
|
| 291 | } else {
|
| 292 | state = irqentry_nmi_enter(regs);
|
| 293 |
|
| 294 | instrumentation_begin();
|
| 295 | kernel_mode_debug_handler(regs, debug_regs);
|
| 296 | instrumentation_end();
|
| 297 |
|
| 298 | irqentry_nmi_exit(regs, state);
|
| 299 | }
|
| 300 | }
|
| 301 |
|
| 302 | There is no combined irqentry_nmi_if_kernel() function available as the
|
| 303 | above cannot be handled in an exception-agnostic way.
|
| 304 |
|
| 305 | NMIs can happen in any context. For example, an NMI-like exception triggered
|
| 306 | while handling an NMI. So NMI entry code has to be reentrant and state updates
|
| 307 | need to handle nesting.
|