| 1 | ==========================================
|
| 2 | Reducing OS jitter due to per-cpu kthreads
|
| 3 | ==========================================
|
| 4 |
|
| 5 | This document lists per-CPU kthreads in the Linux kernel and presents
|
| 6 | options to control their OS jitter. Note that non-per-CPU kthreads are
|
| 7 | not listed here. To reduce OS jitter from non-per-CPU kthreads, bind
|
| 8 | them to a "housekeeping" CPU dedicated to such work.
|
| 9 |
|
| 10 | References
|
| 11 | ==========
|
| 12 |
|
| 13 | - Documentation/core-api/irq/irq-affinity.rst: Binding interrupts to sets of CPUs.
|
| 14 |
|
| 15 | - Documentation/admin-guide/cgroup-v1: Using cgroups to bind tasks to sets of CPUs.
|
| 16 |
|
| 17 | - man taskset: Using the taskset command to bind tasks to sets
|
| 18 | of CPUs.
|
| 19 |
|
| 20 | - man sched_setaffinity: Using the sched_setaffinity() system
|
| 21 | call to bind tasks to sets of CPUs.
|
| 22 |
|
| 23 | - /sys/devices/system/cpu/cpuN/online: Control CPU N's hotplug state,
|
| 24 | writing "0" to offline and "1" to online.
|
| 25 |
|
| 26 | - In order to locate kernel-generated OS jitter on CPU N:
|
| 27 |
|
| 28 | cd /sys/kernel/tracing
|
| 29 | echo 1 > max_graph_depth # Increase the "1" for more detail
|
| 30 | echo function_graph > current_tracer
|
| 31 | # run workload
|
| 32 | cat per_cpu/cpuN/trace
|
| 33 |
|
| 34 | kthreads
|
| 35 | ========
|
| 36 |
|
| 37 | Name:
|
| 38 | ehca_comp/%u
|
| 39 |
|
| 40 | Purpose:
|
| 41 | Periodically process Infiniband-related work.
|
| 42 |
|
| 43 | To reduce its OS jitter, do any of the following:
|
| 44 |
|
| 45 | 1. Don't use eHCA Infiniband hardware, instead choosing hardware
|
| 46 | that does not require per-CPU kthreads. This will prevent these
|
| 47 | kthreads from being created in the first place. (This will
|
| 48 | work for most people, as this hardware, though important, is
|
| 49 | relatively old and is produced in relatively low unit volumes.)
|
| 50 | 2. Do all eHCA-Infiniband-related work on other CPUs, including
|
| 51 | interrupts.
|
| 52 | 3. Rework the eHCA driver so that its per-CPU kthreads are
|
| 53 | provisioned only on selected CPUs.
|
| 54 |
|
| 55 |
|
| 56 | Name:
|
| 57 | irq/%d-%s
|
| 58 |
|
| 59 | Purpose:
|
| 60 | Handle threaded interrupts.
|
| 61 |
|
| 62 | To reduce its OS jitter, do the following:
|
| 63 |
|
| 64 | 1. Use irq affinity to force the irq threads to execute on
|
| 65 | some other CPU.
|
| 66 |
|
| 67 | Name:
|
| 68 | kcmtpd_ctr_%d
|
| 69 |
|
| 70 | Purpose:
|
| 71 | Handle Bluetooth work.
|
| 72 |
|
| 73 | To reduce its OS jitter, do one of the following:
|
| 74 |
|
| 75 | 1. Don't use Bluetooth, in which case these kthreads won't be
|
| 76 | created in the first place.
|
| 77 | 2. Use irq affinity to force Bluetooth-related interrupts to
|
| 78 | occur on some other CPU and furthermore initiate all
|
| 79 | Bluetooth activity on some other CPU.
|
| 80 |
|
| 81 | Name:
|
| 82 | ksoftirqd/%u
|
| 83 |
|
| 84 | Purpose:
|
| 85 | Execute softirq handlers when threaded or when under heavy load.
|
| 86 |
|
| 87 | To reduce its OS jitter, each softirq vector must be handled
|
| 88 | separately as follows:
|
| 89 |
|
| 90 | TIMER_SOFTIRQ
|
| 91 | -------------
|
| 92 |
|
| 93 | Do all of the following:
|
| 94 |
|
| 95 | 1. To the extent possible, keep the CPU out of the kernel when it
|
| 96 | is non-idle, for example, by avoiding system calls and by forcing
|
| 97 | both kernel threads and interrupts to execute elsewhere.
|
| 98 | 2. Build with CONFIG_HOTPLUG_CPU=y. After boot completes, force
|
| 99 | the CPU offline, then bring it back online. This forces
|
| 100 | recurring timers to migrate elsewhere. If you are concerned
|
| 101 | with multiple CPUs, force them all offline before bringing the
|
| 102 | first one back online. Once you have onlined the CPUs in question,
|
| 103 | do not offline any other CPUs, because doing so could force the
|
| 104 | timer back onto one of the CPUs in question.
|
| 105 |
|
| 106 | NET_TX_SOFTIRQ and NET_RX_SOFTIRQ
|
| 107 | ---------------------------------
|
| 108 |
|
| 109 | Do all of the following:
|
| 110 |
|
| 111 | 1. Force networking interrupts onto other CPUs.
|
| 112 | 2. Initiate any network I/O on other CPUs.
|
| 113 | 3. Once your application has started, prevent CPU-hotplug operations
|
| 114 | from being initiated from tasks that might run on the CPU to
|
| 115 | be de-jittered. (It is OK to force this CPU offline and then
|
| 116 | bring it back online before you start your application.)
|
| 117 |
|
| 118 | BLOCK_SOFTIRQ
|
| 119 | -------------
|
| 120 |
|
| 121 | Do all of the following:
|
| 122 |
|
| 123 | 1. Force block-device interrupts onto some other CPU.
|
| 124 | 2. Initiate any block I/O on other CPUs.
|
| 125 | 3. Once your application has started, prevent CPU-hotplug operations
|
| 126 | from being initiated from tasks that might run on the CPU to
|
| 127 | be de-jittered. (It is OK to force this CPU offline and then
|
| 128 | bring it back online before you start your application.)
|
| 129 |
|
| 130 | IRQ_POLL_SOFTIRQ
|
| 131 | ----------------
|
| 132 |
|
| 133 | Do all of the following:
|
| 134 |
|
| 135 | 1. Force block-device interrupts onto some other CPU.
|
| 136 | 2. Initiate any block I/O and block-I/O polling on other CPUs.
|
| 137 | 3. Once your application has started, prevent CPU-hotplug operations
|
| 138 | from being initiated from tasks that might run on the CPU to
|
| 139 | be de-jittered. (It is OK to force this CPU offline and then
|
| 140 | bring it back online before you start your application.)
|
| 141 |
|
| 142 | TASKLET_SOFTIRQ
|
| 143 | ---------------
|
| 144 |
|
| 145 | Do one or more of the following:
|
| 146 |
|
| 147 | 1. Avoid use of drivers that use tasklets. (Such drivers will contain
|
| 148 | calls to things like tasklet_schedule().)
|
| 149 | 2. Convert all drivers that you must use from tasklets to workqueues.
|
| 150 | 3. Force interrupts for drivers using tasklets onto other CPUs,
|
| 151 | and also do I/O involving these drivers on other CPUs.
|
| 152 |
|
| 153 | SCHED_SOFTIRQ
|
| 154 | -------------
|
| 155 |
|
| 156 | Do all of the following:
|
| 157 |
|
| 158 | 1. Avoid sending scheduler IPIs to the CPU to be de-jittered,
|
| 159 | for example, ensure that at most one runnable kthread is present
|
| 160 | on that CPU. If a thread that expects to run on the de-jittered
|
| 161 | CPU awakens, the scheduler will send an IPI that can result in
|
| 162 | a subsequent SCHED_SOFTIRQ.
|
| 163 | 2. CONFIG_NO_HZ_FULL=y and ensure that the CPU to be de-jittered
|
| 164 | is marked as an adaptive-ticks CPU using the "nohz_full="
|
| 165 | boot parameter. This reduces the number of scheduler-clock
|
| 166 | interrupts that the de-jittered CPU receives, minimizing its
|
| 167 | chances of being selected to do the load balancing work that
|
| 168 | runs in SCHED_SOFTIRQ context.
|
| 169 | 3. To the extent possible, keep the CPU out of the kernel when it
|
| 170 | is non-idle, for example, by avoiding system calls and by
|
| 171 | forcing both kernel threads and interrupts to execute elsewhere.
|
| 172 | This further reduces the number of scheduler-clock interrupts
|
| 173 | received by the de-jittered CPU.
|
| 174 |
|
| 175 | HRTIMER_SOFTIRQ
|
| 176 | ---------------
|
| 177 |
|
| 178 | Do all of the following:
|
| 179 |
|
| 180 | 1. To the extent possible, keep the CPU out of the kernel when it
|
| 181 | is non-idle. For example, avoid system calls and force both
|
| 182 | kernel threads and interrupts to execute elsewhere.
|
| 183 | 2. Build with CONFIG_HOTPLUG_CPU=y. Once boot completes, force the
|
| 184 | CPU offline, then bring it back online. This forces recurring
|
| 185 | timers to migrate elsewhere. If you are concerned with multiple
|
| 186 | CPUs, force them all offline before bringing the first one
|
| 187 | back online. Once you have onlined the CPUs in question, do not
|
| 188 | offline any other CPUs, because doing so could force the timer
|
| 189 | back onto one of the CPUs in question.
|
| 190 |
|
| 191 | RCU_SOFTIRQ
|
| 192 | -----------
|
| 193 |
|
| 194 | Do at least one of the following:
|
| 195 |
|
| 196 | 1. Offload callbacks and keep the CPU in either dyntick-idle or
|
| 197 | adaptive-ticks state by doing all of the following:
|
| 198 |
|
| 199 | a. CONFIG_NO_HZ_FULL=y and ensure that the CPU to be
|
| 200 | de-jittered is marked as an adaptive-ticks CPU using the
|
| 201 | "nohz_full=" boot parameter. Bind the rcuo kthreads to
|
| 202 | housekeeping CPUs, which can tolerate OS jitter.
|
| 203 | b. To the extent possible, keep the CPU out of the kernel
|
| 204 | when it is non-idle, for example, by avoiding system
|
| 205 | calls and by forcing both kernel threads and interrupts
|
| 206 | to execute elsewhere.
|
| 207 |
|
| 208 | 2. Enable RCU to do its processing remotely via dyntick-idle by
|
| 209 | doing all of the following:
|
| 210 |
|
| 211 | a. Build with CONFIG_NO_HZ=y.
|
| 212 | b. Ensure that the CPU goes idle frequently, allowing other
|
| 213 | CPUs to detect that it has passed through an RCU quiescent
|
| 214 | state. If the kernel is built with CONFIG_NO_HZ_FULL=y,
|
| 215 | userspace execution also allows other CPUs to detect that
|
| 216 | the CPU in question has passed through a quiescent state.
|
| 217 | c. To the extent possible, keep the CPU out of the kernel
|
| 218 | when it is non-idle, for example, by avoiding system
|
| 219 | calls and by forcing both kernel threads and interrupts
|
| 220 | to execute elsewhere.
|
| 221 |
|
| 222 | Name:
|
| 223 | kworker/%u:%d%s (cpu, id, priority)
|
| 224 |
|
| 225 | Purpose:
|
| 226 | Execute workqueue requests
|
| 227 |
|
| 228 | To reduce its OS jitter, do any of the following:
|
| 229 |
|
| 230 | 1. Run your workload at a real-time priority, which will allow
|
| 231 | preempting the kworker daemons.
|
| 232 | 2. A given workqueue can be made visible in the sysfs filesystem
|
| 233 | by passing the WQ_SYSFS to that workqueue's alloc_workqueue().
|
| 234 | Such a workqueue can be confined to a given subset of the
|
| 235 | CPUs using the ``/sys/devices/virtual/workqueue/*/cpumask`` sysfs
|
| 236 | files. The set of WQ_SYSFS workqueues can be displayed using
|
| 237 | "ls /sys/devices/virtual/workqueue". That said, the workqueues
|
| 238 | maintainer would like to caution people against indiscriminately
|
| 239 | sprinkling WQ_SYSFS across all the workqueues. The reason for
|
| 240 | caution is that it is easy to add WQ_SYSFS, but because sysfs is
|
| 241 | part of the formal user/kernel API, it can be nearly impossible
|
| 242 | to remove it, even if its addition was a mistake.
|
| 243 | 3. Do any of the following needed to avoid jitter that your
|
| 244 | application cannot tolerate:
|
| 245 |
|
| 246 | a. Avoid using oprofile, thus avoiding OS jitter from
|
| 247 | wq_sync_buffer().
|
| 248 | b. Limit your CPU frequency so that a CPU-frequency
|
| 249 | governor is not required, possibly enlisting the aid of
|
| 250 | special heatsinks or other cooling technologies. If done
|
| 251 | correctly, and if you CPU architecture permits, you should
|
| 252 | be able to build your kernel with CONFIG_CPU_FREQ=n to
|
| 253 | avoid the CPU-frequency governor periodically running
|
| 254 | on each CPU, including cs_dbs_timer() and od_dbs_timer().
|
| 255 |
|
| 256 | WARNING: Please check your CPU specifications to
|
| 257 | make sure that this is safe on your particular system.
|
| 258 | c. As of v3.18, Christoph Lameter's on-demand vmstat workers
|
| 259 | commit prevents OS jitter due to vmstat_update() on
|
| 260 | CONFIG_SMP=y systems. Before v3.18, is not possible
|
| 261 | to entirely get rid of the OS jitter, but you can
|
| 262 | decrease its frequency by writing a large value to
|
| 263 | /proc/sys/vm/stat_interval. The default value is HZ,
|
| 264 | for an interval of one second. Of course, larger values
|
| 265 | will make your virtual-memory statistics update more
|
| 266 | slowly. Of course, you can also run your workload at
|
| 267 | a real-time priority, thus preempting vmstat_update(),
|
| 268 | but if your workload is CPU-bound, this is a bad idea.
|
| 269 | However, there is an RFC patch from Christoph Lameter
|
| 270 | (based on an earlier one from Gilad Ben-Yossef) that
|
| 271 | reduces or even eliminates vmstat overhead for some
|
| 272 | workloads at https://lore.kernel.org/r/00000140e9dfd6bd-40db3d4f-c1be-434f-8132-7820f81bb586-000000@email.amazonses.com.
|
| 273 | d. If running on high-end powerpc servers, build with
|
| 274 | CONFIG_PPC_RTAS_DAEMON=n. This prevents the RTAS
|
| 275 | daemon from running on each CPU every second or so.
|
| 276 | (This will require editing Kconfig files and will defeat
|
| 277 | this platform's RAS functionality.) This avoids jitter
|
| 278 | due to the rtas_event_scan() function.
|
| 279 | WARNING: Please check your CPU specifications to
|
| 280 | make sure that this is safe on your particular system.
|
| 281 | e. If running on PowerMAC, build your kernel with
|
| 282 | CONFIG_PMAC_RACKMETER=n to disable the CPU-meter,
|
| 283 | avoiding OS jitter from rackmeter_do_timer().
|
| 284 |
|
| 285 | Name:
|
| 286 | rcuc/%u
|
| 287 |
|
| 288 | Purpose:
|
| 289 | Execute RCU callbacks in CONFIG_RCU_BOOST=y kernels.
|
| 290 |
|
| 291 | To reduce its OS jitter, do at least one of the following:
|
| 292 |
|
| 293 | 1. Build the kernel with CONFIG_PREEMPT=n. This prevents these
|
| 294 | kthreads from being created in the first place, and also obviates
|
| 295 | the need for RCU priority boosting. This approach is feasible
|
| 296 | for workloads that do not require high degrees of responsiveness.
|
| 297 | 2. Build the kernel with CONFIG_RCU_BOOST=n. This prevents these
|
| 298 | kthreads from being created in the first place. This approach
|
| 299 | is feasible only if your workload never requires RCU priority
|
| 300 | boosting, for example, if you ensure frequent idle time on all
|
| 301 | CPUs that might execute within the kernel.
|
| 302 | 3. Build with CONFIG_RCU_NOCB_CPU=y and boot with the rcu_nocbs=
|
| 303 | boot parameter offloading RCU callbacks from all CPUs susceptible
|
| 304 | to OS jitter. This approach prevents the rcuc/%u kthreads from
|
| 305 | having any work to do, so that they are never awakened.
|
| 306 | 4. Ensure that the CPU never enters the kernel, and, in particular,
|
| 307 | avoid initiating any CPU hotplug operations on this CPU. This is
|
| 308 | another way of preventing any callbacks from being queued on the
|
| 309 | CPU, again preventing the rcuc/%u kthreads from having any work
|
| 310 | to do.
|
| 311 |
|
| 312 | Name:
|
| 313 | rcuop/%d, rcuos/%d, and rcuog/%d
|
| 314 |
|
| 315 | Purpose:
|
| 316 | Offload RCU callbacks from the corresponding CPU.
|
| 317 |
|
| 318 | To reduce its OS jitter, do at least one of the following:
|
| 319 |
|
| 320 | 1. Use affinity, cgroups, or other mechanism to force these kthreads
|
| 321 | to execute on some other CPU.
|
| 322 | 2. Build with CONFIG_RCU_NOCB_CPU=n, which will prevent these
|
| 323 | kthreads from being created in the first place. However, please
|
| 324 | note that this will not eliminate OS jitter, but will instead
|
| 325 | shift it to RCU_SOFTIRQ.
|