mm: expedite the grace period when a CPU has many deferred frees

2ce8df46 · Linus Torvalds <[email protected]> · 2026-10-12 11:16:29 -07:00

A folio whose free is deferred waits one to two grace periods on a
per-CPU list, where the allocator does not see it.  The memory on the
lists is the rate of deferred frees times that wait, which was 20 to
25 ms on an 8-CPU laptop (on the first form of the series, below).

With only hot folios marked, what gets deferred is a set of hot folios
leaving the page cache together: a file that was being read is
truncated or dropped with fadvise(DONTNEED), or memory pressure
reclaims what was hot a moment ago.  Kiryl Shutsemau ran 16 processes
that read a shared 1G range and drop it with fadvise(DONTNEED) in a
loop, on a 2G guest with 64 CPUs: with only the marking and no
expedite work, a process next to them was OOM-killed in 5 of 8 runs.

How much waits at a given rate was measured on the first form of the
series, which deferred every folio removed from a page cache.  There,
four processes that fallocate() and hole-punch 64 MiB of a memfd in a
loop freed 23.7 GiB/s and kept 565 MiB waiting on average, 1.2 GiB at
the peak.  With this patch on that form, Kiryl measured 67M waiting on
average for four such loops on tmpfs, on 8 CPUs.  With only hot folios
marked those loads defer nothing: shmem folios are never marked.

Count the pages on each CPU's pending list.  When a free takes the
count above RCU_FREE_EXPEDITE pages, queue a work item, the expedite
work, that takes the list, calls synchronize_rcu_expedited() and frees
it.  With this patch only that work waits for the grace period: the
task that frees does not, and neither does reclaim.  (The next patch
has reclaim wait for one as a last resort.)

The batch whose grace period is already in flight stays with its
callback, so nothing changes for the existing work.

With the whole series, Kiryl's load, run by us in a 2G guest with 64
CPUs on a Threadripper 3970X, three runs of 60 seconds: 50 million
pages are deferred in a run and the expedite work runs 5750-5828 times
(the pgfree_rcu_kick event of the last patch); 9-14 MiB wait on
average and about 1 GiB at the peak, and the process next to the
readers is not killed.  The readers get through 49.6-50.4 GiB/s where
a kernel without the series gets through 59.9-64.7, about 20% less;
what that is spent on has not been profiled.

Every run of the expedite work is a synchronize_rcu_expedited(), which
sends IPIs to other CPUs, and a task that reads a file and drops it in
a loop causes them at the rate above.  What that costs CPUs that are
isolated or run nohz_full was not measured.  The work sleeps for its
grace period on the workqueue that also frees the batches, which is
WQ_MEM_RECLAIM and so has one rescuer: when the work items run on it,
under memory pressure, an expedite work in synchronize_rcu_expedited()
can hold up the frees queued behind it.

The limit, 1024 pages, is a guess.  The allocator still does not see
what waits: up to the limit on each CPU's pending list, the batch whose
grace period is in flight, what the expedite work has taken until its
grace period ends, and whatever is freed onto the pending list while
the work waits to run, which nothing bounds.  The next patch is about
that.

The work runs on the CPU of the task that queued it, and with lazy
preemption it gets that CPU only at the next tick.  On the first form
of the series, having the page cache removal loops reschedule when
asked shortened that.  Kiryl found that none of the loads he measured
with hot-only marking needed it, and whether mm should act on a lazy
reschedule request is a question for the scheduler maintainers.  It is
not in this series.

Assisted-by: LLM
Signed-off-by: Linus Torvalds <[email protected]>

Commit

2ce8df4627fef977bd1bcee299a35e335b1f2d9f

Parents

5028a42b

Changes

1 files changed · +58 −9