| 1 | |
| 2 |
|
| 3 | ===========================
|
| 4 | How realtime kernels differ
|
| 5 | ===========================
|
| 6 |
|
| 7 | :Author: Sebastian Andrzej Siewior <[email protected]>
|
| 8 |
|
| 9 | Preface
|
| 10 | =======
|
| 11 |
|
| 12 | With forced-threaded interrupts and sleeping spin locks, code paths that
|
| 13 | previously caused long scheduling latencies have been made preemptible and
|
| 14 | moved into process context. This allows the scheduler to manage them more
|
| 15 | effectively and respond to higher-priority tasks with reduced latency.
|
| 16 |
|
| 17 | The following chapters provide an overview of key differences between a
|
| 18 | PREEMPT_RT kernel and a standard, non-PREEMPT_RT kernel.
|
| 19 |
|
| 20 | Locking
|
| 21 | =======
|
| 22 |
|
| 23 | Spinning locks such as spinlock_t are used to provide synchronization for data
|
| 24 | structures accessed from both interrupt context and process context. For this
|
| 25 | reason, locking functions are also available with the _irq() or _irqsave()
|
| 26 | suffixes, which disable interrupts before acquiring the lock. This ensures that
|
| 27 | the lock can be safely acquired in process context when interrupts are enabled.
|
| 28 |
|
| 29 | However, on a PREEMPT_RT system, interrupts are forced-threaded and no longer
|
| 30 | run in hard IRQ context. As a result, there is no need to disable interrupts as
|
| 31 | part of the locking procedure when using spinlock_t.
|
| 32 |
|
| 33 | For low-level core components such as interrupt handling, the scheduler, or the
|
| 34 | timer subsystem the kernel uses raw_spinlock_t. This lock type preserves
|
| 35 | traditional semantics: it disables preemption and, when used with _irq() or
|
| 36 | _irqsave(), also disables interrupts. This ensures proper synchronization in
|
| 37 | critical sections that must remain non-preemptible or with interrupts disabled.
|
| 38 |
|
| 39 | Execution context
|
| 40 | =================
|
| 41 |
|
| 42 | Interrupt handling in a PREEMPT_RT system is invoked in process context through
|
| 43 | the use of threaded interrupts. Other parts of the kernel also shift their
|
| 44 | execution into threaded context by different mechanisms. The goal is to keep
|
| 45 | execution paths preemptible, allowing the scheduler to interrupt them when a
|
| 46 | higher-priority task needs to run.
|
| 47 |
|
| 48 | Below is an overview of the kernel subsystems involved in this transition to
|
| 49 | threaded, preemptible execution.
|
| 50 |
|
| 51 | Interrupt handling
|
| 52 | ------------------
|
| 53 |
|
| 54 | All interrupts are forced-threaded in a PREEMPT_RT system. The exceptions are
|
| 55 | interrupts that are requested with the IRQF_NO_THREAD, IRQF_PERCPU, or
|
| 56 | IRQF_ONESHOT flags.
|
| 57 |
|
| 58 | The IRQF_ONESHOT flag is used together with threaded interrupts, meaning those
|
| 59 | registered using request_threaded_irq() and providing only a threaded handler.
|
| 60 | Its purpose is to keep the interrupt line masked until the threaded handler has
|
| 61 | completed.
|
| 62 |
|
| 63 | If a primary handler is also provided in this case, it is essential that the
|
| 64 | handler does not acquire any sleeping locks, as it will not be threaded. The
|
| 65 | handler should be minimal and must avoid introducing delays, such as
|
| 66 | busy-waiting on hardware registers.
|
| 67 |
|
| 68 |
|
| 69 | Soft interrupts, bottom half handling
|
| 70 | -------------------------------------
|
| 71 |
|
| 72 | Soft interrupts are raised by the interrupt handler and are executed after the
|
| 73 | handler returns. Since they run in thread context, they can be preempted by
|
| 74 | other threads. Do not assume that softirq context runs with preemption
|
| 75 | disabled. This means you must not rely on mechanisms like local_bh_disable() in
|
| 76 | process context to protect per-CPU variables. Because softirq handlers are
|
| 77 | preemptible under PREEMPT_RT, this approach does not provide reliable
|
| 78 | synchronization.
|
| 79 |
|
| 80 | If this kind of protection is required for performance reasons, consider using
|
| 81 | local_lock_nested_bh(). On non-PREEMPT_RT kernels, this allows lockdep to
|
| 82 | verify that bottom halves are disabled. On PREEMPT_RT systems, it adds the
|
| 83 | necessary locking to ensure proper protection.
|
| 84 |
|
| 85 | Using local_lock_nested_bh() also makes the locking scope explicit and easier
|
| 86 | for readers and maintainers to understand.
|
| 87 |
|
| 88 |
|
| 89 | per-CPU variables
|
| 90 | -----------------
|
| 91 |
|
| 92 | Protecting access to per-CPU variables solely by using preempt_disable() should
|
| 93 | be avoided, especially if the critical section has unbounded runtime or may
|
| 94 | call APIs that can sleep.
|
| 95 |
|
| 96 | If using a spinlock_t is considered too costly for performance reasons,
|
| 97 | consider using local_lock_t. On non-PREEMPT_RT configurations, this introduces
|
| 98 | no runtime overhead when lockdep is disabled. With lockdep enabled, it verifies
|
| 99 | that the lock is only acquired in process context and never from softirq or
|
| 100 | hard IRQ context.
|
| 101 |
|
| 102 | On a PREEMPT_RT kernel, local_lock_t is implemented using a per-CPU spinlock_t,
|
| 103 | which provides safe local protection for per-CPU data while keeping the system
|
| 104 | preemptible.
|
| 105 |
|
| 106 | Because spinlock_t on PREEMPT_RT does not disable preemption, it cannot be used
|
| 107 | to protect per-CPU data by relying on implicit preemption disabling. If this
|
| 108 | inherited preemption disabling is essential and if local_lock_t cannot be used
|
| 109 | due to performance constraints, brevity of the code, or abstraction boundaries
|
| 110 | within an API then preempt_disable_nested() may be a suitable alternative. On
|
| 111 | non-PREEMPT_RT kernels, it verifies with lockdep that preemption is already
|
| 112 | disabled. On PREEMPT_RT, it explicitly disables preemption.
|
| 113 |
|
| 114 | Timers
|
| 115 | ------
|
| 116 |
|
| 117 | By default, an hrtimer is executed in hard interrupt context. The exception is
|
| 118 | timers initialized with the HRTIMER_MODE_SOFT flag, which are executed in
|
| 119 | softirq context.
|
| 120 |
|
| 121 | On a PREEMPT_RT kernel, this behavior is reversed: hrtimers are executed in
|
| 122 | softirq context by default, typically within the ktimers thread. This thread
|
| 123 | runs at the lowest real-time priority, ensuring it executes before any
|
| 124 | SCHED_OTHER tasks but does not interfere with higher-priority real-time
|
| 125 | threads. To explicitly request execution in hard interrupt context on
|
| 126 | PREEMPT_RT, the timer must be marked with the HRTIMER_MODE_HARD flag.
|
| 127 |
|
| 128 | Userland sleepers usually deploy a hrtimer to guarantee a precise wakeup
|
| 129 | time. The timer is initialized with hrtimer_setup_sleeper_on_stack(), which
|
| 130 | distinguishes between real-time and regular tasks. The hrtimer of a task
|
| 131 | without a real-time priority is handled in soft interrupt context, but for
|
| 132 | real-time priorities HRTIMER_MODE_HARD is used. This ensures that real-time
|
| 133 | tasks are woken up as soon as possible while ordinary tasks cannot block the
|
| 134 | CPU with a thundering herd of wakeups.
|
| 135 |
|
| 136 | Memory allocation
|
| 137 | -----------------
|
| 138 |
|
| 139 | The memory allocation APIs, such as kmalloc() and alloc_pages(), require a
|
| 140 | gfp_t flag to indicate the allocation context. On non-PREEMPT_RT kernels, it is
|
| 141 | necessary to use GFP_ATOMIC when allocating memory from interrupt context or
|
| 142 | from sections where preemption is disabled. This is because the allocator must
|
| 143 | not sleep in these contexts waiting for memory to become available.
|
| 144 |
|
| 145 | However, this approach does not work on PREEMPT_RT kernels. The memory
|
| 146 | allocator in PREEMPT_RT uses sleeping locks internally, which cannot be
|
| 147 | acquired when preemption is disabled. Fortunately, this is generally not a
|
| 148 | problem, because PREEMPT_RT moves most contexts that would traditionally run
|
| 149 | with preemption or interrupts disabled into threaded context, where sleeping is
|
| 150 | allowed.
|
| 151 |
|
| 152 | What remains problematic is code that explicitly disables preemption or
|
| 153 | interrupts. In such cases, memory allocation must be performed outside the
|
| 154 | critical section.
|
| 155 |
|
| 156 | This restriction also applies to memory deallocation routines such as kfree()
|
| 157 | and free_pages(), which may also involve internal locking and must not be
|
| 158 | called from non-preemptible contexts.
|
| 159 |
|
| 160 | IRQ work
|
| 161 | --------
|
| 162 |
|
| 163 | The irq_work API provides a mechanism to schedule a callback in interrupt
|
| 164 | context. It is designed for use in contexts where traditional scheduling is not
|
| 165 | possible, such as from within NMI handlers or from inside the scheduler, where
|
| 166 | using a workqueue would be unsafe.
|
| 167 |
|
| 168 | On non-PREEMPT_RT systems, all irq_work items are executed immediately in
|
| 169 | interrupt context. Items marked with IRQ_WORK_LAZY are deferred until the next
|
| 170 | timer tick but are still executed in interrupt context.
|
| 171 |
|
| 172 | On PREEMPT_RT systems, the execution model changes. Because irq_work callbacks
|
| 173 | may acquire sleeping locks or have unbounded execution time, they are handled
|
| 174 | in thread context by a per-CPU irq_work kernel thread. This thread runs at the
|
| 175 | lowest real-time priority, ensuring it executes before any SCHED_OTHER tasks
|
| 176 | but does not interfere with higher-priority real-time threads.
|
| 177 |
|
| 178 | The exception are work items marked with IRQ_WORK_HARD_IRQ, which are still
|
| 179 | executed in hard interrupt context. Lazy items (IRQ_WORK_LAZY) continue to be
|
| 180 | deferred until the next timer tick and are also executed by the irq_work/
|
| 181 | thread.
|
| 182 |
|
| 183 | RCU callbacks
|
| 184 | -------------
|
| 185 |
|
| 186 | RCU callbacks are invoked by default in softirq context. Their execution is
|
| 187 | important because, depending on the use case, they either free memory or ensure
|
| 188 | progress in state transitions. Running these callbacks as part of the softirq
|
| 189 | chain can lead to undesired situations, such as contention for CPU resources
|
| 190 | with other SCHED_OTHER tasks when executed within ksoftirqd.
|
| 191 |
|
| 192 | To avoid running callbacks in softirq context, the RCU subsystem provides a
|
| 193 | mechanism to execute them in process context instead. This behavior can be
|
| 194 | enabled by setting the boot command-line parameter rcutree.use_softirq=0. This
|
| 195 | setting is enforced in kernels configured with PREEMPT_RT.
|
| 196 |
|
| 197 | Spin until ready
|
| 198 | ================
|
| 199 |
|
| 200 | The "spin until ready" pattern involves repeatedly checking (spinning on) the
|
| 201 | state of a data structure until it becomes available. This pattern assumes that
|
| 202 | preemption, soft interrupts, or interrupts are disabled. If the data structure
|
| 203 | is marked busy, it is presumed to be in use by another CPU, and spinning should
|
| 204 | eventually succeed as that CPU makes progress.
|
| 205 |
|
| 206 | Some examples are hrtimer_cancel() or timer_delete_sync(). These functions
|
| 207 | cancel timers that execute with interrupts or soft interrupts disabled. If a
|
| 208 | thread attempts to cancel a timer and finds it active, spinning until the
|
| 209 | callback completes is safe because the callback can only run on another CPU and
|
| 210 | will eventually finish.
|
| 211 |
|
| 212 | On PREEMPT_RT kernels, however, timer callbacks run in thread context. This
|
| 213 | introduces a challenge: a higher-priority thread attempting to cancel the timer
|
| 214 | may preempt the timer callback thread. Since the scheduler cannot migrate the
|
| 215 | callback thread to another CPU due to affinity constraints, spinning can result
|
| 216 | in livelock even on multiprocessor systems.
|
| 217 |
|
| 218 | To avoid this, both the canceling and callback sides must use a handshake
|
| 219 | mechanism that supports priority inheritance. This allows the canceling thread
|
| 220 | to suspend until the callback completes, ensuring forward progress without
|
| 221 | risking livelock.
|
| 222 |
|
| 223 | In order to solve the problem at the API level, the sequence locks were extended
|
| 224 | to allow a proper handover between the spinning reader and the maybe
|
| 225 | blocked writer.
|
| 226 |
|
| 227 | Sequence locks
|
| 228 | --------------
|
| 229 |
|
| 230 | Sequence counters and sequential locks are documented in
|
| 231 | Documentation/locking/seqlock.rst.
|
| 232 |
|
| 233 | The interface has been extended to ensure proper preemption states for the
|
| 234 | writer and spinning reader contexts. This is achieved by embedding the writer
|
| 235 | serialization lock directly into the sequence counter type, resulting in
|
| 236 | composite types such as seqcount_spinlock_t or seqcount_mutex_t.
|
| 237 |
|
| 238 | These composite types allow readers to detect an ongoing write and actively
|
| 239 | boost the writer’s priority to help it complete its update instead of spinning
|
| 240 | and waiting for its completion.
|
| 241 |
|
| 242 | If the plain seqcount_t is used, extra care must be taken to synchronize the
|
| 243 | reader with the writer during updates. The writer must ensure its update is
|
| 244 | serialized and non-preemptible relative to the reader. This cannot be achieved
|
| 245 | using a regular spinlock_t because spinlock_t on PREEMPT_RT does not disable
|
| 246 | preemption. In such cases, using seqcount_spinlock_t is the preferred solution.
|
| 247 |
|
| 248 | However, if there is no spinning involved i.e., if the reader only needs to
|
| 249 | detect whether a write has started and not serialize against it then using
|
| 250 | seqcount_t is reasonable.
|