| 1 | =========
|
| 2 | Workqueue
|
| 3 | =========
|
| 4 |
|
| 5 | :Date: September, 2010
|
| 6 | :Author: Tejun Heo <[email protected]>
|
| 7 | :Author: Florian Mickler <[email protected]>
|
| 8 |
|
| 9 |
|
| 10 | Introduction
|
| 11 | ============
|
| 12 |
|
| 13 | There are many cases where an asynchronous process execution context
|
| 14 | is needed and the workqueue (wq) API is the most commonly used
|
| 15 | mechanism for such cases.
|
| 16 |
|
| 17 | When such an asynchronous execution context is needed, a work item
|
| 18 | describing which function to execute is put on a queue. An
|
| 19 | independent thread serves as the asynchronous execution context. The
|
| 20 | queue is called workqueue and the thread is called worker.
|
| 21 |
|
| 22 | While there are work items on the workqueue the worker executes the
|
| 23 | functions associated with the work items one after the other. When
|
| 24 | there is no work item left on the workqueue the worker becomes idle.
|
| 25 | When a new work item gets queued, the worker begins executing again.
|
| 26 |
|
| 27 |
|
| 28 | Why Concurrency Managed Workqueue?
|
| 29 | ==================================
|
| 30 |
|
| 31 | In the original wq implementation, a multi threaded (MT) wq had one
|
| 32 | worker thread per CPU and a single threaded (ST) wq had one worker
|
| 33 | thread system-wide. A single MT wq needed to keep around the same
|
| 34 | number of workers as the number of CPUs. The kernel grew a lot of MT
|
| 35 | wq users over the years and with the number of CPU cores continuously
|
| 36 | rising, some systems saturated the default 32k PID space just booting
|
| 37 | up.
|
| 38 |
|
| 39 | Although MT wq wasted a lot of resource, the level of concurrency
|
| 40 | provided was unsatisfactory. The limitation was common to both ST and
|
| 41 | MT wq albeit less severe on MT. Each wq maintained its own separate
|
| 42 | worker pool. An MT wq could provide only one execution context per CPU
|
| 43 | while an ST wq one for the whole system. Work items had to compete for
|
| 44 | those very limited execution contexts leading to various problems
|
| 45 | including proneness to deadlocks around the single execution context.
|
| 46 |
|
| 47 | The tension between the provided level of concurrency and resource
|
| 48 | usage also forced its users to make unnecessary tradeoffs like libata
|
| 49 | choosing to use ST wq for polling PIOs and accepting an unnecessary
|
| 50 | limitation that no two polling PIOs can progress at the same time. As
|
| 51 | MT wq don't provide much better concurrency, users which require
|
| 52 | higher level of concurrency, like async or fscache, had to implement
|
| 53 | their own thread pool.
|
| 54 |
|
| 55 | Concurrency Managed Workqueue (cmwq) is a reimplementation of wq with
|
| 56 | focus on the following goals.
|
| 57 |
|
| 58 | * Maintain compatibility with the original workqueue API.
|
| 59 |
|
| 60 | * Use per-CPU unified worker pools shared by all wq to provide
|
| 61 | flexible level of concurrency on demand without wasting a lot of
|
| 62 | resource.
|
| 63 |
|
| 64 | * Automatically regulate worker pool and level of concurrency so that
|
| 65 | the API users don't need to worry about such details.
|
| 66 |
|
| 67 |
|
| 68 | The Design
|
| 69 | ==========
|
| 70 |
|
| 71 | In order to ease the asynchronous execution of functions a new
|
| 72 | abstraction, the work item, is introduced.
|
| 73 |
|
| 74 | A work item is a simple struct that holds a pointer to the function
|
| 75 | that is to be executed asynchronously. Whenever a driver or subsystem
|
| 76 | wants a function to be executed asynchronously it has to set up a work
|
| 77 | item pointing to that function and queue that work item on a
|
| 78 | workqueue.
|
| 79 |
|
| 80 | A work item can be executed in either a thread or the BH (softirq) context.
|
| 81 |
|
| 82 | For threaded workqueues, special purpose threads, called [k]workers, execute
|
| 83 | the functions off of the queue, one after the other. If no work is queued,
|
| 84 | the worker threads become idle. These worker threads are managed in
|
| 85 | worker-pools.
|
| 86 |
|
| 87 | The cmwq design differentiates between the user-facing workqueues that
|
| 88 | subsystems and drivers queue work items on and the backend mechanism
|
| 89 | which manages worker-pools and processes the queued work items.
|
| 90 |
|
| 91 | There are two worker-pools, one for normal work items and the other
|
| 92 | for high priority ones, for each possible CPU and some extra
|
| 93 | worker-pools to serve work items queued on unbound workqueues - the
|
| 94 | number of these backing pools is dynamic.
|
| 95 |
|
| 96 | BH workqueues use the same framework. However, as there can only be one
|
| 97 | concurrent execution context, there's no need to worry about concurrency.
|
| 98 | Each per-CPU BH worker pool contains only one pseudo worker which represents
|
| 99 | the BH execution context. A BH workqueue can be considered a convenience
|
| 100 | interface to softirq.
|
| 101 |
|
| 102 | Subsystems and drivers can create and queue work items through special
|
| 103 | workqueue API functions as they see fit. They can influence some
|
| 104 | aspects of the way the work items are executed by setting flags on the
|
| 105 | workqueue they are putting the work item on. These flags include
|
| 106 | things like CPU locality, concurrency limits, priority and more. To
|
| 107 | get a detailed overview refer to the API description of
|
| 108 | ``alloc_workqueue()`` below.
|
| 109 |
|
| 110 | When a work item is queued to a workqueue, the target worker-pool is
|
| 111 | determined according to the queue parameters and workqueue attributes
|
| 112 | and appended on the shared worklist of the worker-pool. For example,
|
| 113 | unless specifically overridden, a work item of a bound workqueue will
|
| 114 | be queued on the worklist of either normal or highpri worker-pool that
|
| 115 | is associated to the CPU the issuer is running on.
|
| 116 |
|
| 117 | For any thread pool implementation, managing the concurrency level
|
| 118 | (how many execution contexts are active) is an important issue. cmwq
|
| 119 | tries to keep the concurrency at a minimal but sufficient level.
|
| 120 | Minimal to save resources and sufficient in that the system is used at
|
| 121 | its full capacity.
|
| 122 |
|
| 123 | Each worker-pool bound to an actual CPU implements concurrency
|
| 124 | management by hooking into the scheduler. The worker-pool is notified
|
| 125 | whenever an active worker wakes up or sleeps and keeps track of the
|
| 126 | number of the currently runnable workers. Generally, work items are
|
| 127 | not expected to hog a CPU and consume many cycles. That means
|
| 128 | maintaining just enough concurrency to prevent work processing from
|
| 129 | stalling should be optimal. As long as there are one or more runnable
|
| 130 | workers on the CPU, the worker-pool doesn't start execution of a new
|
| 131 | work, but, when the last running worker goes to sleep, it immediately
|
| 132 | schedules a new worker so that the CPU doesn't sit idle while there
|
| 133 | are pending work items. This allows using a minimal number of workers
|
| 134 | without losing execution bandwidth.
|
| 135 |
|
| 136 | Keeping idle workers around doesn't cost other than the memory space
|
| 137 | for kthreads, so cmwq holds onto idle ones for a while before killing
|
| 138 | them.
|
| 139 |
|
| 140 | For unbound workqueues, the number of backing pools is dynamic.
|
| 141 | Unbound workqueue can be assigned custom attributes using
|
| 142 | ``apply_workqueue_attrs()`` and workqueue will automatically create
|
| 143 | backing worker pools matching the attributes. The responsibility of
|
| 144 | regulating concurrency level is on the users. There is also a flag to
|
| 145 | mark a bound wq to ignore the concurrency management. Please refer to
|
| 146 | the API section for details.
|
| 147 |
|
| 148 | Forward progress guarantee relies on that workers can be created when
|
| 149 | more execution contexts are necessary, which in turn is guaranteed
|
| 150 | through the use of rescue workers. All work items which might be used
|
| 151 | on code paths that handle memory reclaim are required to be queued on
|
| 152 | wq's that have a rescue-worker reserved for execution under memory
|
| 153 | pressure. Else it is possible that the worker-pool deadlocks waiting
|
| 154 | for execution contexts to free up.
|
| 155 |
|
| 156 |
|
| 157 | Application Programming Interface (API)
|
| 158 | =======================================
|
| 159 |
|
| 160 | ``alloc_workqueue()`` allocates a wq. The original
|
| 161 | ``create_*workqueue()`` functions are deprecated and scheduled for
|
| 162 | removal. ``alloc_workqueue()`` takes three arguments - ``@name``,
|
| 163 | ``@flags`` and ``@max_active``. ``@name`` is the name of the wq and
|
| 164 | also used as the name of the rescuer thread if there is one.
|
| 165 |
|
| 166 | A wq no longer manages execution resources but serves as a domain for
|
| 167 | forward progress guarantee, flush and work item attributes. ``@flags``
|
| 168 | and ``@max_active`` control how work items are assigned execution
|
| 169 | resources, scheduled and executed.
|
| 170 |
|
| 171 |
|
| 172 | ``flags``
|
| 173 | ---------
|
| 174 |
|
| 175 | ``WQ_BH``
|
| 176 | BH workqueues can be considered a convenience interface to softirq. BH
|
| 177 | workqueues are always per-CPU and all BH work items are executed in the
|
| 178 | queueing CPU's softirq context in the queueing order.
|
| 179 |
|
| 180 | All BH workqueues must have 0 ``max_active`` and ``WQ_HIGHPRI`` is the
|
| 181 | only allowed additional flag.
|
| 182 |
|
| 183 | BH work items cannot sleep. All other features such as delayed queueing,
|
| 184 | flushing and canceling are supported.
|
| 185 |
|
| 186 | ``WQ_PERCPU``
|
| 187 | Work items queued to a per-cpu wq are bound to a specific CPU.
|
| 188 | This flag is the right choice when cpu locality is important.
|
| 189 |
|
| 190 | This flag is the complement of ``WQ_UNBOUND``.
|
| 191 |
|
| 192 | ``WQ_UNBOUND``
|
| 193 | Work items queued to an unbound wq are served by the special
|
| 194 | worker-pools which host workers which are not bound to any
|
| 195 | specific CPU. This makes the wq behave as a simple execution
|
| 196 | context provider without concurrency management. The unbound
|
| 197 | worker-pools try to start execution of work items as soon as
|
| 198 | possible. Unbound wq sacrifices locality but is useful for
|
| 199 | the following cases.
|
| 200 |
|
| 201 | * Wide fluctuation in the concurrency level requirement is
|
| 202 | expected and using bound wq may end up creating large number
|
| 203 | of mostly unused workers across different CPUs as the issuer
|
| 204 | hops through different CPUs.
|
| 205 |
|
| 206 | * Long running CPU intensive workloads which can be better
|
| 207 | managed by the system scheduler.
|
| 208 |
|
| 209 | ``WQ_FREEZABLE``
|
| 210 | A freezable wq participates in the freeze phase of the system
|
| 211 | suspend operations. Work items on the wq are drained and no
|
| 212 | new work item starts execution until thawed.
|
| 213 |
|
| 214 | ``WQ_MEM_RECLAIM``
|
| 215 | All wq which might be used in the memory reclaim paths **MUST**
|
| 216 | have this flag set. The wq is guaranteed to have at least one
|
| 217 | execution context regardless of memory pressure.
|
| 218 |
|
| 219 | ``WQ_HIGHPRI``
|
| 220 | Work items of a highpri wq are queued to the highpri
|
| 221 | worker-pool of the target cpu. Highpri worker-pools are
|
| 222 | served by worker threads with elevated nice level.
|
| 223 |
|
| 224 | Note that normal and highpri worker-pools don't interact with
|
| 225 | each other. Each maintains its separate pool of workers and
|
| 226 | implements concurrency management among its workers.
|
| 227 |
|
| 228 | ``WQ_CPU_INTENSIVE``
|
| 229 | Work items of a CPU intensive wq do not contribute to the
|
| 230 | concurrency level. In other words, runnable CPU intensive
|
| 231 | work items will not prevent other work items in the same
|
| 232 | worker-pool from starting execution. This is useful for bound
|
| 233 | work items which are expected to hog CPU cycles so that their
|
| 234 | execution is regulated by the system scheduler.
|
| 235 |
|
| 236 | Although CPU intensive work items don't contribute to the
|
| 237 | concurrency level, start of their executions is still
|
| 238 | regulated by the concurrency management and runnable
|
| 239 | non-CPU-intensive work items can delay execution of CPU
|
| 240 | intensive work items.
|
| 241 |
|
| 242 | This flag is meaningless for unbound wq.
|
| 243 |
|
| 244 |
|
| 245 | ``max_active``
|
| 246 | --------------
|
| 247 |
|
| 248 | ``@max_active`` determines the maximum number of execution contexts per
|
| 249 | CPU which can be assigned to the work items of a wq. For example, with
|
| 250 | ``@max_active`` of 16, at most 16 work items of the wq can be executing
|
| 251 | at the same time per CPU. This is always a per-CPU attribute, even for
|
| 252 | unbound workqueues.
|
| 253 |
|
| 254 | The maximum limit for ``@max_active`` is 2048 and the default value used
|
| 255 | when 0 is specified is 1024. These values are chosen sufficiently high
|
| 256 | such that they are not the limiting factor while providing protection in
|
| 257 | runaway cases.
|
| 258 |
|
| 259 | The number of active work items of a wq is usually regulated by the
|
| 260 | users of the wq, more specifically, by how many work items the users
|
| 261 | may queue at the same time. Unless there is a specific need for
|
| 262 | throttling the number of active work items, specifying '0' is
|
| 263 | recommended.
|
| 264 |
|
| 265 | Some users depend on strict execution ordering where only one work item
|
| 266 | is in flight at any given time and the work items are processed in
|
| 267 | queueing order. While the combination of ``@max_active`` of 1 and
|
| 268 | ``WQ_UNBOUND`` used to achieve this behavior, this is no longer the
|
| 269 | case. Use alloc_ordered_workqueue() instead.
|
| 270 |
|
| 271 |
|
| 272 | Example Execution Scenarios
|
| 273 | ===========================
|
| 274 |
|
| 275 | The following example execution scenarios try to illustrate how cmwq
|
| 276 | behave under different configurations.
|
| 277 |
|
| 278 | Work items w0, w1, w2 are queued to a bound wq q0 on the same CPU.
|
| 279 | w0 burns CPU for 5ms then sleeps for 10ms then burns CPU for 5ms
|
| 280 | again before finishing. w1 and w2 burn CPU for 5ms then sleep for
|
| 281 | 10ms.
|
| 282 |
|
| 283 | Ignoring all other tasks, works and processing overhead, and assuming
|
| 284 | simple FIFO scheduling, the following is one highly simplified version
|
| 285 | of possible sequences of events with the original wq. ::
|
| 286 |
|
| 287 | TIME IN MSECS EVENT
|
| 288 | 0 w0 starts and burns CPU
|
| 289 | 5 w0 sleeps
|
| 290 | 15 w0 wakes up and burns CPU
|
| 291 | 20 w0 finishes
|
| 292 | 20 w1 starts and burns CPU
|
| 293 | 25 w1 sleeps
|
| 294 | 35 w1 wakes up and finishes
|
| 295 | 35 w2 starts and burns CPU
|
| 296 | 40 w2 sleeps
|
| 297 | 50 w2 wakes up and finishes
|
| 298 |
|
| 299 | And with cmwq with ``@max_active`` >= 3, ::
|
| 300 |
|
| 301 | TIME IN MSECS EVENT
|
| 302 | 0 w0 starts and burns CPU
|
| 303 | 5 w0 sleeps
|
| 304 | 5 w1 starts and burns CPU
|
| 305 | 10 w1 sleeps
|
| 306 | 10 w2 starts and burns CPU
|
| 307 | 15 w2 sleeps
|
| 308 | 15 w0 wakes up and burns CPU
|
| 309 | 20 w0 finishes
|
| 310 | 20 w1 wakes up and finishes
|
| 311 | 25 w2 wakes up and finishes
|
| 312 |
|
| 313 | If ``@max_active`` == 2, ::
|
| 314 |
|
| 315 | TIME IN MSECS EVENT
|
| 316 | 0 w0 starts and burns CPU
|
| 317 | 5 w0 sleeps
|
| 318 | 5 w1 starts and burns CPU
|
| 319 | 10 w1 sleeps
|
| 320 | 15 w0 wakes up and burns CPU
|
| 321 | 20 w0 finishes
|
| 322 | 20 w1 wakes up and finishes
|
| 323 | 20 w2 starts and burns CPU
|
| 324 | 25 w2 sleeps
|
| 325 | 35 w2 wakes up and finishes
|
| 326 |
|
| 327 | Now, let's assume w1 and w2 are queued to a different wq q1 which has
|
| 328 | ``WQ_CPU_INTENSIVE`` set, ::
|
| 329 |
|
| 330 | TIME IN MSECS EVENT
|
| 331 | 0 w0 starts and burns CPU
|
| 332 | 5 w0 sleeps
|
| 333 | 5 w1 and w2 start and burn CPU
|
| 334 | 10 w1 sleeps
|
| 335 | 15 w2 sleeps
|
| 336 | 15 w0 wakes up and burns CPU
|
| 337 | 20 w0 finishes
|
| 338 | 20 w1 wakes up and finishes
|
| 339 | 25 w2 wakes up and finishes
|
| 340 |
|
| 341 |
|
| 342 | Guidelines
|
| 343 | ==========
|
| 344 |
|
| 345 | * Do not forget to use ``WQ_MEM_RECLAIM`` if a wq may process work
|
| 346 | items which are used during memory reclaim. Each wq with
|
| 347 | ``WQ_MEM_RECLAIM`` set has an execution context reserved for it. If
|
| 348 | there is dependency among multiple work items used during memory
|
| 349 | reclaim, they should be queued to separate wq each with
|
| 350 | ``WQ_MEM_RECLAIM``.
|
| 351 |
|
| 352 | * Unless strict ordering is required, there is no need to use ST wq.
|
| 353 |
|
| 354 | * Unless there is a specific need, using 0 for @max_active is
|
| 355 | recommended. In most use cases, concurrency level usually stays
|
| 356 | well under the default limit.
|
| 357 |
|
| 358 | * A wq serves as a domain for forward progress guarantee
|
| 359 | (``WQ_MEM_RECLAIM``), flush and work item attributes. Work items
|
| 360 | which are not involved in memory reclaim and don't need to be
|
| 361 | flushed as a part of a group of work items, and don't require any
|
| 362 | special attribute, can use one of the system wq. There is no
|
| 363 | difference in execution characteristics between using a dedicated wq
|
| 364 | and a system wq.
|
| 365 |
|
| 366 | Note: If something may generate more than @max_active outstanding
|
| 367 | work items (do stress test your producers), it may saturate a system
|
| 368 | wq and potentially lead to deadlock. It should utilize its own
|
| 369 | dedicated workqueue rather than the system wq.
|
| 370 |
|
| 371 | * Unless work items are expected to consume a huge amount of CPU
|
| 372 | cycles, using a bound wq is usually beneficial due to the increased
|
| 373 | level of locality in wq operations and work item execution.
|
| 374 |
|
| 375 |
|
| 376 | Affinity Scopes
|
| 377 | ===============
|
| 378 |
|
| 379 | An unbound workqueue groups CPUs according to its affinity scope to improve
|
| 380 | cache locality. For example, if a workqueue is using the default affinity
|
| 381 | scope of "cache_shard", it will group CPUs into sub-LLC shards. A work item
|
| 382 | queued on the workqueue will be assigned to a worker on one of the CPUs
|
| 383 | within the same shard as the issuing CPU.
|
| 384 | Once started, the worker may or may not be allowed to move outside the scope
|
| 385 | depending on the ``affinity_strict`` setting of the scope.
|
| 386 |
|
| 387 | Workqueue currently supports the following affinity scopes.
|
| 388 |
|
| 389 | ``default``
|
| 390 | Use the scope in module parameter ``workqueue.default_affinity_scope``
|
| 391 | which is always set to one of the scopes below.
|
| 392 |
|
| 393 | ``cpu``
|
| 394 | CPUs are not grouped. A work item issued on one CPU is processed by a
|
| 395 | worker on the same CPU. This makes unbound workqueues behave as per-cpu
|
| 396 | workqueues without concurrency management.
|
| 397 |
|
| 398 | ``smt``
|
| 399 | CPUs are grouped according to SMT boundaries. This usually means that the
|
| 400 | logical threads of each physical CPU core are grouped together.
|
| 401 |
|
| 402 | ``cache``
|
| 403 | CPUs are grouped according to cache boundaries. Which specific cache
|
| 404 | boundary is used is determined by the arch code. L3 is used in a lot of
|
| 405 | cases.
|
| 406 |
|
| 407 | ``cache_shard``
|
| 408 | CPUs are grouped into sub-LLC shards of at most ``wq_cache_shard_size``
|
| 409 | cores (default 8, tunable via the ``workqueue.cache_shard_size`` boot
|
| 410 | parameter). Shards are always split on core (SMT group) boundaries.
|
| 411 | This is the default affinity scope.
|
| 412 |
|
| 413 | ``numa``
|
| 414 | CPUs are grouped according to NUMA boundaries.
|
| 415 |
|
| 416 | ``system``
|
| 417 | All CPUs are put in the same group. Workqueue makes no effort to process a
|
| 418 | work item on a CPU close to the issuing CPU.
|
| 419 |
|
| 420 | The default affinity scope can be changed with the module parameter
|
| 421 | ``workqueue.default_affinity_scope`` and a specific workqueue's affinity
|
| 422 | scope can be changed using ``apply_workqueue_attrs()``.
|
| 423 |
|
| 424 | If ``WQ_SYSFS`` is set, the workqueue will have the following affinity scope
|
| 425 | related interface files under its ``/sys/devices/virtual/workqueue/WQ_NAME/``
|
| 426 | directory.
|
| 427 |
|
| 428 | ``affinity_scope``
|
| 429 | Read to see the current affinity scope. Write to change.
|
| 430 |
|
| 431 | When default is the current scope, reading this file will also show the
|
| 432 | current effective scope in parentheses, for example, ``default (cache)``.
|
| 433 |
|
| 434 | ``affinity_strict``
|
| 435 | 0 by default indicating that affinity scopes are not strict. When a work
|
| 436 | item starts execution, workqueue makes a best-effort attempt to ensure
|
| 437 | that the worker is inside its affinity scope, which is called
|
| 438 | repatriation. Once started, the scheduler is free to move the worker
|
| 439 | anywhere in the system as it sees fit. This enables benefiting from scope
|
| 440 | locality while still being able to utilize other CPUs if necessary and
|
| 441 | available.
|
| 442 |
|
| 443 | If set to 1, all workers of the scope are guaranteed always to be in the
|
| 444 | scope. This may be useful when crossing affinity scopes has other
|
| 445 | implications, for example, in terms of power consumption or workload
|
| 446 | isolation. Strict NUMA scope can also be used to match the workqueue
|
| 447 | behavior of older kernels.
|
| 448 |
|
| 449 |
|
| 450 | Affinity Scopes and Performance
|
| 451 | ===============================
|
| 452 |
|
| 453 | It'd be ideal if an unbound workqueue's behavior is optimal for vast
|
| 454 | majority of use cases without further tuning. Unfortunately, in the current
|
| 455 | kernel, there exists a pronounced trade-off between locality and utilization
|
| 456 | necessitating explicit configurations when workqueues are heavily used.
|
| 457 |
|
| 458 | Higher locality leads to higher efficiency where more work is performed for
|
| 459 | the same number of consumed CPU cycles. However, higher locality may also
|
| 460 | cause lower overall system utilization if the work items are not spread
|
| 461 | enough across the affinity scopes by the issuers. The following performance
|
| 462 | testing with dm-crypt clearly illustrates this trade-off.
|
| 463 |
|
| 464 | The tests are run on a CPU with 12-cores/24-threads split across four L3
|
| 465 | caches (AMD Ryzen 9 3900x). CPU clock boost is turned off for consistency.
|
| 466 | ``/dev/dm-0`` is a dm-crypt device created on NVME SSD (Samsung 990 PRO) and
|
| 467 | opened with ``cryptsetup`` with default settings.
|
| 468 |
|
| 469 |
|
| 470 | Scenario 1: Enough issuers and work spread across the machine
|
| 471 | -------------------------------------------------------------
|
| 472 |
|
| 473 | The command used: ::
|
| 474 |
|
| 475 | $ fio --filename=/dev/dm-0 --direct=1 --rw=randrw --bs=32k --ioengine=libaio \
|
| 476 | --iodepth=64 --runtime=60 --numjobs=24 --time_based --group_reporting \
|
| 477 | --name=iops-test-job --verify=sha512
|
| 478 |
|
| 479 | There are 24 issuers, each issuing 64 IOs concurrently. ``--verify=sha512``
|
| 480 | makes ``fio`` generate and read back the content each time which makes
|
| 481 | execution locality matter between the issuer and ``kcryptd``. The following
|
| 482 | are the read bandwidths and CPU utilizations depending on different affinity
|
| 483 | scope settings on ``kcryptd`` measured over five runs. Bandwidths are in
|
| 484 | MiBps, and CPU util in percents.
|
| 485 |
|
| 486 | .. list-table::
|
| 487 | :widths: 16 20 20
|
| 488 | :header-rows: 1
|
| 489 |
|
| 490 | * - Affinity
|
| 491 | - Bandwidth (MiBps)
|
| 492 | - CPU util (%)
|
| 493 |
|
| 494 | * - system
|
| 495 | - 1159.40 ±1.34
|
| 496 | - 99.31 ±0.02
|
| 497 |
|
| 498 | * - cache
|
| 499 | - 1166.40 ±0.89
|
| 500 | - 99.34 ±0.01
|
| 501 |
|
| 502 | * - cache (strict)
|
| 503 | - 1166.00 ±0.71
|
| 504 | - 99.35 ±0.01
|
| 505 |
|
| 506 | With enough issuers spread across the system, there is no downside to
|
| 507 | "cache", strict or otherwise. All three configurations saturate the whole
|
| 508 | machine but the cache-affine ones outperform by 0.6% thanks to improved
|
| 509 | locality.
|
| 510 |
|
| 511 |
|
| 512 | Scenario 2: Fewer issuers, enough work for saturation
|
| 513 | -----------------------------------------------------
|
| 514 |
|
| 515 | The command used: ::
|
| 516 |
|
| 517 | $ fio --filename=/dev/dm-0 --direct=1 --rw=randrw --bs=32k \
|
| 518 | --ioengine=libaio --iodepth=64 --runtime=60 --numjobs=8 \
|
| 519 | --time_based --group_reporting --name=iops-test-job --verify=sha512
|
| 520 |
|
| 521 | The only difference from the previous scenario is ``--numjobs=8``. There are
|
| 522 | a third of the issuers but is still enough total work to saturate the
|
| 523 | system.
|
| 524 |
|
| 525 | .. list-table::
|
| 526 | :widths: 16 20 20
|
| 527 | :header-rows: 1
|
| 528 |
|
| 529 | * - Affinity
|
| 530 | - Bandwidth (MiBps)
|
| 531 | - CPU util (%)
|
| 532 |
|
| 533 | * - system
|
| 534 | - 1155.40 ±0.89
|
| 535 | - 97.41 ±0.05
|
| 536 |
|
| 537 | * - cache
|
| 538 | - 1154.40 ±1.14
|
| 539 | - 96.15 ±0.09
|
| 540 |
|
| 541 | * - cache (strict)
|
| 542 | - 1112.00 ±4.64
|
| 543 | - 93.26 ±0.35
|
| 544 |
|
| 545 | This is more than enough work to saturate the system. Both "system" and
|
| 546 | "cache" are nearly saturating the machine but not fully. "cache" is using
|
| 547 | less CPU but the better efficiency puts it at the same bandwidth as
|
| 548 | "system".
|
| 549 |
|
| 550 | Eight issuers moving around over four L3 cache scope still allow "cache
|
| 551 | (strict)" to mostly saturate the machine but the loss of work conservation
|
| 552 | is now starting to hurt with 3.7% bandwidth loss.
|
| 553 |
|
| 554 |
|
| 555 | Scenario 3: Even fewer issuers, not enough work to saturate
|
| 556 | -----------------------------------------------------------
|
| 557 |
|
| 558 | The command used: ::
|
| 559 |
|
| 560 | $ fio --filename=/dev/dm-0 --direct=1 --rw=randrw --bs=32k \
|
| 561 | --ioengine=libaio --iodepth=64 --runtime=60 --numjobs=4 \
|
| 562 | --time_based --group_reporting --name=iops-test-job --verify=sha512
|
| 563 |
|
| 564 | Again, the only difference is ``--numjobs=4``. With the number of issuers
|
| 565 | reduced to four, there now isn't enough work to saturate the whole system
|
| 566 | and the bandwidth becomes dependent on completion latencies.
|
| 567 |
|
| 568 | .. list-table::
|
| 569 | :widths: 16 20 20
|
| 570 | :header-rows: 1
|
| 571 |
|
| 572 | * - Affinity
|
| 573 | - Bandwidth (MiBps)
|
| 574 | - CPU util (%)
|
| 575 |
|
| 576 | * - system
|
| 577 | - 993.60 ±1.82
|
| 578 | - 75.49 ±0.06
|
| 579 |
|
| 580 | * - cache
|
| 581 | - 973.40 ±1.52
|
| 582 | - 74.90 ±0.07
|
| 583 |
|
| 584 | * - cache (strict)
|
| 585 | - 828.20 ±4.49
|
| 586 | - 66.84 ±0.29
|
| 587 |
|
| 588 | Now, the tradeoff between locality and utilization is clearer. "cache" shows
|
| 589 | 2% bandwidth loss compared to "system" and "cache (struct)" whopping 20%.
|
| 590 |
|
| 591 |
|
| 592 | Conclusion and Recommendations
|
| 593 | ------------------------------
|
| 594 |
|
| 595 | In the above experiments, the efficiency advantage of the "cache" affinity
|
| 596 | scope over "system" is, while consistent and noticeable, small. However, the
|
| 597 | impact is dependent on the distances between the scopes and may be more
|
| 598 | pronounced in processors with more complex topologies.
|
| 599 |
|
| 600 | While the loss of work-conservation in certain scenarios hurts, it is a lot
|
| 601 | better than "cache (strict)" and maximizing workqueue utilization is
|
| 602 | unlikely to be the common case anyway. As such, "cache" is the default
|
| 603 | affinity scope for unbound pools.
|
| 604 |
|
| 605 | * As there is no one option which is great for most cases, workqueue usages
|
| 606 | that may consume a significant amount of CPU are recommended to configure
|
| 607 | the workqueues using ``apply_workqueue_attrs()`` and/or enable
|
| 608 | ``WQ_SYSFS``.
|
| 609 |
|
| 610 | * An unbound workqueue with strict "cpu" affinity scope behaves the same as
|
| 611 | ``WQ_CPU_INTENSIVE`` per-cpu workqueue. There is no real advanage to the
|
| 612 | latter and an unbound workqueue provides a lot more flexibility.
|
| 613 |
|
| 614 | * Affinity scopes are introduced in Linux v6.5. To emulate the previous
|
| 615 | behavior, use strict "numa" affinity scope.
|
| 616 |
|
| 617 | * The loss of work-conservation in non-strict affinity scopes is likely
|
| 618 | originating from the scheduler. There is no theoretical reason why the
|
| 619 | kernel wouldn't be able to do the right thing and maintain
|
| 620 | work-conservation in most cases. As such, it is possible that future
|
| 621 | scheduler improvements may make most of these tunables unnecessary.
|
| 622 |
|
| 623 |
|
| 624 | Examining Configuration
|
| 625 | =======================
|
| 626 |
|
| 627 | Use tools/workqueue/wq_dump.py to examine unbound CPU affinity
|
| 628 | configuration, worker pools and how workqueues map to the pools: ::
|
| 629 |
|
| 630 | $ tools/workqueue/wq_dump.py
|
| 631 | Affinity Scopes
|
| 632 | ===============
|
| 633 | wq_unbound_cpumask=0000000f
|
| 634 |
|
| 635 | CPU
|
| 636 | nr_pods 4
|
| 637 | pod_cpus [0]=00000001 [1]=00000002 [2]=00000004 [3]=00000008
|
| 638 | pod_node [0]=0 [1]=0 [2]=1 [3]=1
|
| 639 | cpu_pod [0]=0 [1]=1 [2]=2 [3]=3
|
| 640 |
|
| 641 | SMT
|
| 642 | nr_pods 4
|
| 643 | pod_cpus [0]=00000001 [1]=00000002 [2]=00000004 [3]=00000008
|
| 644 | pod_node [0]=0 [1]=0 [2]=1 [3]=1
|
| 645 | cpu_pod [0]=0 [1]=1 [2]=2 [3]=3
|
| 646 |
|
| 647 | CACHE (default)
|
| 648 | nr_pods 2
|
| 649 | pod_cpus [0]=00000003 [1]=0000000c
|
| 650 | pod_node [0]=0 [1]=1
|
| 651 | cpu_pod [0]=0 [1]=0 [2]=1 [3]=1
|
| 652 |
|
| 653 | NUMA
|
| 654 | nr_pods 2
|
| 655 | pod_cpus [0]=00000003 [1]=0000000c
|
| 656 | pod_node [0]=0 [1]=1
|
| 657 | cpu_pod [0]=0 [1]=0 [2]=1 [3]=1
|
| 658 |
|
| 659 | SYSTEM
|
| 660 | nr_pods 1
|
| 661 | pod_cpus [0]=0000000f
|
| 662 | pod_node [0]=-1
|
| 663 | cpu_pod [0]=0 [1]=0 [2]=0 [3]=0
|
| 664 |
|
| 665 | Worker Pools
|
| 666 | ============
|
| 667 | pool[00] ref= 1 nice= 0 idle/workers= 4/ 4 cpu= 0
|
| 668 | pool[01] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 0
|
| 669 | pool[02] ref= 1 nice= 0 idle/workers= 4/ 4 cpu= 1
|
| 670 | pool[03] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 1
|
| 671 | pool[04] ref= 1 nice= 0 idle/workers= 4/ 4 cpu= 2
|
| 672 | pool[05] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 2
|
| 673 | pool[06] ref= 1 nice= 0 idle/workers= 3/ 3 cpu= 3
|
| 674 | pool[07] ref= 1 nice=-20 idle/workers= 2/ 2 cpu= 3
|
| 675 | pool[08] ref=42 nice= 0 idle/workers= 6/ 6 cpus=0000000f
|
| 676 | pool[09] ref=28 nice= 0 idle/workers= 3/ 3 cpus=00000003
|
| 677 | pool[10] ref=28 nice= 0 idle/workers= 17/ 17 cpus=0000000c
|
| 678 | pool[11] ref= 1 nice=-20 idle/workers= 1/ 1 cpus=0000000f
|
| 679 | pool[12] ref= 2 nice=-20 idle/workers= 1/ 1 cpus=00000003
|
| 680 | pool[13] ref= 2 nice=-20 idle/workers= 1/ 1 cpus=0000000c
|
| 681 |
|
| 682 | Workqueue CPU -> pool
|
| 683 | =====================
|
| 684 | [ workqueue \ CPU 0 1 2 3 dfl]
|
| 685 | events percpu 0 2 4 6
|
| 686 | events_highpri percpu 1 3 5 7
|
| 687 | events_long percpu 0 2 4 6
|
| 688 | events_unbound unbound 9 9 10 10 8
|
| 689 | events_freezable percpu 0 2 4 6
|
| 690 | events_power_efficient percpu 0 2 4 6
|
| 691 | events_freezable_pwr_ef percpu 0 2 4 6
|
| 692 | rcu_gp percpu 0 2 4 6
|
| 693 | rcu_par_gp percpu 0 2 4 6
|
| 694 | slub_flushwq percpu 0 2 4 6
|
| 695 | netns ordered 8 8 8 8 8
|
| 696 | ...
|
| 697 |
|
| 698 | See the command's help message for more info.
|
| 699 |
|
| 700 |
|
| 701 | Monitoring
|
| 702 | ==========
|
| 703 |
|
| 704 | Use tools/workqueue/wq_monitor.py to monitor workqueue operations: ::
|
| 705 |
|
| 706 | $ tools/workqueue/wq_monitor.py events
|
| 707 | total infl CPUtime CPUhog CMW/RPR mayday rescued
|
| 708 | events 18545 0 6.1 0 5 - -
|
| 709 | events_highpri 8 0 0.0 0 0 - -
|
| 710 | events_long 3 0 0.0 0 0 - -
|
| 711 | events_unbound 38306 0 0.1 - 7 - -
|
| 712 | events_freezable 0 0 0.0 0 0 - -
|
| 713 | events_power_efficient 29598 0 0.2 0 0 - -
|
| 714 | events_freezable_pwr_ef 10 0 0.0 0 0 - -
|
| 715 | sock_diag_events 0 0 0.0 0 0 - -
|
| 716 |
|
| 717 | total infl CPUtime CPUhog CMW/RPR mayday rescued
|
| 718 | events 18548 0 6.1 0 5 - -
|
| 719 | events_highpri 8 0 0.0 0 0 - -
|
| 720 | events_long 3 0 0.0 0 0 - -
|
| 721 | events_unbound 38322 0 0.1 - 7 - -
|
| 722 | events_freezable 0 0 0.0 0 0 - -
|
| 723 | events_power_efficient 29603 0 0.2 0 0 - -
|
| 724 | events_freezable_pwr_ef 10 0 0.0 0 0 - -
|
| 725 | sock_diag_events 0 0 0.0 0 0 - -
|
| 726 |
|
| 727 | ...
|
| 728 |
|
| 729 | See the command's help message for more info.
|
| 730 |
|
| 731 |
|
| 732 | Debugging
|
| 733 | =========
|
| 734 |
|
| 735 | Because the work functions are executed by generic worker threads
|
| 736 | there are a few tricks needed to shed some light on misbehaving
|
| 737 | workqueue users.
|
| 738 |
|
| 739 | Worker threads show up in the process list as: ::
|
| 740 |
|
| 741 | root 5671 0.0 0.0 0 0 ? S 12:07 0:00 [kworker/0:1]
|
| 742 | root 5672 0.0 0.0 0 0 ? S 12:07 0:00 [kworker/1:2]
|
| 743 | root 5673 0.0 0.0 0 0 ? S 12:12 0:00 [kworker/0:0]
|
| 744 | root 5674 0.0 0.0 0 0 ? S 12:13 0:00 [kworker/1:0]
|
| 745 |
|
| 746 | If kworkers are going crazy (using too much cpu), there are two types
|
| 747 | of possible problems:
|
| 748 |
|
| 749 | 1. Something being scheduled in rapid succession
|
| 750 | 2. A single work item that consumes lots of cpu cycles
|
| 751 |
|
| 752 | The first one can be tracked using tracing: ::
|
| 753 |
|
| 754 | $ echo workqueue:workqueue_queue_work > /sys/kernel/tracing/set_event
|
| 755 | $ cat /sys/kernel/tracing/trace_pipe > out.txt
|
| 756 | (wait a few secs)
|
| 757 | ^C
|
| 758 |
|
| 759 | If something is busy looping on work queueing, it would be dominating
|
| 760 | the output and the offender can be determined with the work item
|
| 761 | function.
|
| 762 |
|
| 763 | For the second type of problems it should be possible to just check
|
| 764 | the stack trace of the offending worker thread. ::
|
| 765 |
|
| 766 | $ cat /proc/THE_OFFENDING_KWORKER/stack
|
| 767 |
|
| 768 | The work item's function should be trivially visible in the stack
|
| 769 | trace.
|
| 770 |
|
| 771 |
|
| 772 | Non-reentrance Conditions
|
| 773 | =========================
|
| 774 |
|
| 775 | Workqueue guarantees that a work item cannot be re-entrant if the following
|
| 776 | conditions hold after a work item gets queued:
|
| 777 |
|
| 778 | 1. The work function hasn't been changed.
|
| 779 | 2. No one queues the work item to another workqueue.
|
| 780 | 3. The work item hasn't been reinitiated.
|
| 781 |
|
| 782 | In other words, if the above conditions hold, the work item is guaranteed to be
|
| 783 | executed by at most one worker system-wide at any given time.
|
| 784 |
|
| 785 | Note that requeuing the work item (to the same queue) in the self function
|
| 786 | doesn't break these conditions, so it's safe to do. Otherwise, caution is
|
| 787 | required when breaking the conditions inside a work function.
|
| 788 |
|
| 789 |
|
| 790 | Kernel Inline Documentations Reference
|
| 791 | ======================================
|
| 792 |
|
| 793 | .. kernel-doc:: include/linux/workqueue.h
|
| 794 |
|
| 795 | .. kernel-doc:: kernel/workqueue.c
|