| 1 | ===================
|
| 2 | this_cpu operations
|
| 3 | ===================
|
| 4 |
|
| 5 | :Author: Christoph Lameter, August 4th, 2014
|
| 6 | :Author: Pranith Kumar, Aug 2nd, 2014
|
| 7 |
|
| 8 | this_cpu operations are a way of optimizing access to per cpu
|
| 9 | variables associated with the *currently* executing processor. This is
|
| 10 | done through the use of segment registers (or a dedicated register where
|
| 11 | the cpu permanently stored the beginning of the per cpu area for a
|
| 12 | specific processor).
|
| 13 |
|
| 14 | this_cpu operations add a per cpu variable offset to the processor
|
| 15 | specific per cpu base and encode that operation in the instruction
|
| 16 | operating on the per cpu variable.
|
| 17 |
|
| 18 | This means that there are no atomicity issues between the calculation of
|
| 19 | the offset and the operation on the data. Therefore it is not
|
| 20 | necessary to disable preemption or interrupts to ensure that the
|
| 21 | processor is not changed between the calculation of the address and
|
| 22 | the operation on the data.
|
| 23 |
|
| 24 | Read-modify-write operations are of particular interest. Frequently
|
| 25 | processors have special lower latency instructions that can operate
|
| 26 | without the typical synchronization overhead, but still provide some
|
| 27 | sort of relaxed atomicity guarantees. The x86, for example, can execute
|
| 28 | RMW (Read Modify Write) instructions like inc/dec/cmpxchg without the
|
| 29 | lock prefix and the associated latency penalty.
|
| 30 |
|
| 31 | Access to the variable without the lock prefix is not synchronized but
|
| 32 | synchronization is not necessary since we are dealing with per cpu
|
| 33 | data specific to the currently executing processor. Only the current
|
| 34 | processor should be accessing that variable and therefore there are no
|
| 35 | concurrency issues with other processors in the system.
|
| 36 |
|
| 37 | Please note that accesses by remote processors to a per cpu area are
|
| 38 | exceptional situations and may impact performance and/or correctness
|
| 39 | (remote write operations) of local RMW operations via this_cpu_*.
|
| 40 |
|
| 41 | The main use of the this_cpu operations has been to optimize counter
|
| 42 | operations.
|
| 43 |
|
| 44 | The following this_cpu() operations with implied preemption protection
|
| 45 | are defined. These operations can be used without worrying about
|
| 46 | preemption and interrupts::
|
| 47 |
|
| 48 | this_cpu_read(pcp)
|
| 49 | this_cpu_write(pcp, val)
|
| 50 | this_cpu_add(pcp, val)
|
| 51 | this_cpu_and(pcp, val)
|
| 52 | this_cpu_or(pcp, val)
|
| 53 | this_cpu_add_return(pcp, val)
|
| 54 | this_cpu_xchg(pcp, nval)
|
| 55 | this_cpu_cmpxchg(pcp, oval, nval)
|
| 56 | this_cpu_sub(pcp, val)
|
| 57 | this_cpu_inc(pcp)
|
| 58 | this_cpu_dec(pcp)
|
| 59 | this_cpu_sub_return(pcp, val)
|
| 60 | this_cpu_inc_return(pcp)
|
| 61 | this_cpu_dec_return(pcp)
|
| 62 |
|
| 63 |
|
| 64 | Inner working of this_cpu operations
|
| 65 | ------------------------------------
|
| 66 |
|
| 67 | On x86 the fs: or the gs: segment registers contain the base of the
|
| 68 | per cpu area. It is then possible to simply use the segment override
|
| 69 | to relocate a per cpu relative address to the proper per cpu area for
|
| 70 | the processor. So the relocation to the per cpu base is encoded in the
|
| 71 | instruction via a segment register prefix.
|
| 72 |
|
| 73 | For example::
|
| 74 |
|
| 75 | DEFINE_PER_CPU(int, x);
|
| 76 | int z;
|
| 77 |
|
| 78 | z = this_cpu_read(x);
|
| 79 |
|
| 80 | results in a single instruction::
|
| 81 |
|
| 82 | mov ax, gs:[x]
|
| 83 |
|
| 84 | instead of a sequence of calculation of the address and then a fetch
|
| 85 | from that address which occurs with the per cpu operations. Before
|
| 86 | this_cpu_ops such sequence also required preempt disable/enable to
|
| 87 | prevent the kernel from moving the thread to a different processor
|
| 88 | while the calculation is performed.
|
| 89 |
|
| 90 | Consider the following this_cpu operation::
|
| 91 |
|
| 92 | this_cpu_inc(x)
|
| 93 |
|
| 94 | The above results in the following single instruction (no lock prefix!)::
|
| 95 |
|
| 96 | inc gs:[x]
|
| 97 |
|
| 98 | instead of the following operations required if there is no segment
|
| 99 | register::
|
| 100 |
|
| 101 | int *y;
|
| 102 | int cpu;
|
| 103 |
|
| 104 | cpu = get_cpu();
|
| 105 | y = per_cpu_ptr(&x, cpu);
|
| 106 | (*y)++;
|
| 107 | put_cpu();
|
| 108 |
|
| 109 | Note that these operations can only be used on per cpu data that is
|
| 110 | reserved for a specific processor. Without disabling preemption in the
|
| 111 | surrounding code this_cpu_inc() will only guarantee that one of the
|
| 112 | per cpu counters is correctly incremented. However, there is no
|
| 113 | guarantee that the OS will not move the process directly before or
|
| 114 | after the this_cpu instruction is executed. In general this means that
|
| 115 | the value of the individual counters for each processor are
|
| 116 | meaningless. The sum of all the per cpu counters is the only value
|
| 117 | that is of interest.
|
| 118 |
|
| 119 | Per cpu variables are used for performance reasons. Bouncing cache
|
| 120 | lines can be avoided if multiple processors concurrently go through
|
| 121 | the same code paths. Since each processor has its own per cpu
|
| 122 | variables no concurrent cache line updates take place. The price that
|
| 123 | has to be paid for this optimization is the need to add up the per cpu
|
| 124 | counters when the value of a counter is needed.
|
| 125 |
|
| 126 |
|
| 127 | Special operations
|
| 128 | ------------------
|
| 129 |
|
| 130 | ::
|
| 131 |
|
| 132 | y = this_cpu_ptr(&x)
|
| 133 |
|
| 134 | Takes the offset of a per cpu variable (&x !) and returns the address
|
| 135 | of the per cpu variable that belongs to the currently executing
|
| 136 | processor. this_cpu_ptr avoids multiple steps that the common
|
| 137 | get_cpu/put_cpu sequence requires. No processor number is
|
| 138 | available. Instead, the offset of the local per cpu area is simply
|
| 139 | added to the per cpu offset.
|
| 140 |
|
| 141 | Note that this operation can only be used in code segments where
|
| 142 | smp_processor_id() may be used, for example, where preemption has been
|
| 143 | disabled. The pointer is then used to access local per cpu data in a
|
| 144 | critical section. When preemption is re-enabled this pointer is usually
|
| 145 | no longer useful since it may no longer point to per cpu data of the
|
| 146 | current processor.
|
| 147 |
|
| 148 | The special cases where it makes sense to obtain a per-CPU pointer in
|
| 149 | preemptible code are addressed by raw_cpu_ptr(), but such use cases need
|
| 150 | to handle cases where two different CPUs are accessing the same per cpu
|
| 151 | variable, which might well be that of a third CPU. These use cases are
|
| 152 | typically performance optimizations. For example, SRCU implements a pair
|
| 153 | of counters as a pair of per-CPU variables, and srcu_read_lock_nmisafe()
|
| 154 | uses raw_cpu_ptr() to get a pointer to some CPU's counter, and uses
|
| 155 | atomic_long_inc() to handle migration between the raw_cpu_ptr() and
|
| 156 | the atomic_long_inc().
|
| 157 |
|
| 158 | Per cpu variables and offsets
|
| 159 | -----------------------------
|
| 160 |
|
| 161 | Per cpu variables have *offsets* to the beginning of the per cpu
|
| 162 | area. They do not have addresses although they look like that in the
|
| 163 | code. Offsets cannot be directly dereferenced. The offset must be
|
| 164 | added to a base pointer of a per cpu area of a processor in order to
|
| 165 | form a valid address.
|
| 166 |
|
| 167 | Therefore the use of x or &x outside of the context of per cpu
|
| 168 | operations is invalid and will generally be treated like a NULL
|
| 169 | pointer dereference.
|
| 170 |
|
| 171 | ::
|
| 172 |
|
| 173 | DEFINE_PER_CPU(int, x);
|
| 174 |
|
| 175 | In the context of per cpu operations the above implies that x is a per
|
| 176 | cpu variable. Most this_cpu operations take a cpu variable.
|
| 177 |
|
| 178 | ::
|
| 179 |
|
| 180 | int __percpu *p = &x;
|
| 181 |
|
| 182 | &x and hence p is the *offset* of a per cpu variable. this_cpu_ptr()
|
| 183 | takes the offset of a per cpu variable which makes this look a bit
|
| 184 | strange.
|
| 185 |
|
| 186 |
|
| 187 | Operations on a field of a per cpu structure
|
| 188 | --------------------------------------------
|
| 189 |
|
| 190 | Let's say we have a percpu structure::
|
| 191 |
|
| 192 | struct s {
|
| 193 | int n,m;
|
| 194 | };
|
| 195 |
|
| 196 | DEFINE_PER_CPU(struct s, p);
|
| 197 |
|
| 198 |
|
| 199 | Operations on these fields are straightforward::
|
| 200 |
|
| 201 | this_cpu_inc(p.m)
|
| 202 |
|
| 203 | z = this_cpu_cmpxchg(p.m, 0, 1);
|
| 204 |
|
| 205 |
|
| 206 | If we have an offset to struct s::
|
| 207 |
|
| 208 | struct s __percpu *ps = &p;
|
| 209 |
|
| 210 | this_cpu_dec(ps->m);
|
| 211 |
|
| 212 | z = this_cpu_inc_return(ps->n);
|
| 213 |
|
| 214 |
|
| 215 | The calculation of the pointer may require the use of this_cpu_ptr()
|
| 216 | if we do not make use of this_cpu ops later to manipulate fields::
|
| 217 |
|
| 218 | struct s *pp;
|
| 219 |
|
| 220 | pp = this_cpu_ptr(&p);
|
| 221 |
|
| 222 | pp->m--;
|
| 223 |
|
| 224 | z = pp->n++;
|
| 225 |
|
| 226 |
|
| 227 | Variants of this_cpu ops
|
| 228 | ------------------------
|
| 229 |
|
| 230 | this_cpu ops are interrupt safe. Some architectures do not support
|
| 231 | these per cpu local operations. In that case the operation must be
|
| 232 | replaced by code that disables interrupts, then does the operations
|
| 233 | that are guaranteed to be atomic and then re-enable interrupts. Doing
|
| 234 | so is expensive. If there are other reasons why the scheduler cannot
|
| 235 | change the processor we are executing on then there is no reason to
|
| 236 | disable interrupts. For that purpose the following __this_cpu operations
|
| 237 | are provided.
|
| 238 |
|
| 239 | These operations have no guarantee against concurrent interrupts or
|
| 240 | preemption. If a per cpu variable is not used in an interrupt context
|
| 241 | and the scheduler cannot preempt, then they are safe. If any interrupts
|
| 242 | still occur while an operation is in progress and if the interrupt too
|
| 243 | modifies the variable, then RMW actions can not be guaranteed to be
|
| 244 | safe::
|
| 245 |
|
| 246 | __this_cpu_read(pcp)
|
| 247 | __this_cpu_write(pcp, val)
|
| 248 | __this_cpu_add(pcp, val)
|
| 249 | __this_cpu_and(pcp, val)
|
| 250 | __this_cpu_or(pcp, val)
|
| 251 | __this_cpu_add_return(pcp, val)
|
| 252 | __this_cpu_xchg(pcp, nval)
|
| 253 | __this_cpu_cmpxchg(pcp, oval, nval)
|
| 254 | __this_cpu_sub(pcp, val)
|
| 255 | __this_cpu_inc(pcp)
|
| 256 | __this_cpu_dec(pcp)
|
| 257 | __this_cpu_sub_return(pcp, val)
|
| 258 | __this_cpu_inc_return(pcp)
|
| 259 | __this_cpu_dec_return(pcp)
|
| 260 |
|
| 261 |
|
| 262 | Will increment x and will not fall-back to code that disables
|
| 263 | interrupts on platforms that cannot accomplish atomicity through
|
| 264 | address relocation and a Read-Modify-Write operation in the same
|
| 265 | instruction.
|
| 266 |
|
| 267 |
|
| 268 | &this_cpu_ptr(pp)->n vs this_cpu_ptr(&pp->n)
|
| 269 | --------------------------------------------
|
| 270 |
|
| 271 | The first operation takes the offset and forms an address and then
|
| 272 | adds the offset of the n field. This may result in two add
|
| 273 | instructions emitted by the compiler.
|
| 274 |
|
| 275 | The second one first adds the two offsets and then does the
|
| 276 | relocation. IMHO the second form looks cleaner and has an easier time
|
| 277 | with (). The second form also is consistent with the way
|
| 278 | this_cpu_read() and friends are used.
|
| 279 |
|
| 280 |
|
| 281 | Remote access to per cpu data
|
| 282 | ------------------------------
|
| 283 |
|
| 284 | Per cpu data structures are designed to be used by one cpu exclusively.
|
| 285 | If you use the variables as intended, this_cpu_ops() are guaranteed to
|
| 286 | be "atomic" as no other CPU has access to these data structures.
|
| 287 |
|
| 288 | There are special cases where you might need to access per cpu data
|
| 289 | structures remotely. It is usually safe to do a remote read access
|
| 290 | and that is frequently done to summarize counters. Remote write access
|
| 291 | something which could be problematic because this_cpu ops do not
|
| 292 | have lock semantics. A remote write may interfere with a this_cpu
|
| 293 | RMW operation.
|
| 294 |
|
| 295 | Remote write accesses to percpu data structures are highly discouraged
|
| 296 | unless absolutely necessary. Please consider using an IPI to wake up
|
| 297 | the remote CPU and perform the update to its per cpu area.
|
| 298 |
|
| 299 | To access per-cpu data structure remotely, typically the per_cpu_ptr()
|
| 300 | function is used::
|
| 301 |
|
| 302 |
|
| 303 | DEFINE_PER_CPU(struct data, datap);
|
| 304 |
|
| 305 | struct data *p = per_cpu_ptr(&datap, cpu);
|
| 306 |
|
| 307 | This makes it explicit that we are getting ready to access a percpu
|
| 308 | area remotely.
|
| 309 |
|
| 310 | You can also do the following to convert the datap offset to an address::
|
| 311 |
|
| 312 | struct data *p = this_cpu_ptr(&datap);
|
| 313 |
|
| 314 | but, passing of pointers calculated via this_cpu_ptr to other cpus is
|
| 315 | unusual and should be avoided.
|
| 316 |
|
| 317 | Remote access are typically only for reading the status of another cpus
|
| 318 | per cpu data. Write accesses can cause unique problems due to the
|
| 319 | relaxed synchronization requirements for this_cpu operations.
|
| 320 |
|
| 321 | One example that illustrates some concerns with write operations is
|
| 322 | the following scenario that occurs because two per cpu variables
|
| 323 | share a cache-line but the relaxed synchronization is applied to
|
| 324 | only one process updating the cache-line.
|
| 325 |
|
| 326 | Consider the following example::
|
| 327 |
|
| 328 |
|
| 329 | struct test {
|
| 330 | atomic_t a;
|
| 331 | int b;
|
| 332 | };
|
| 333 |
|
| 334 | DEFINE_PER_CPU(struct test, onecacheline);
|
| 335 |
|
| 336 | There is some concern about what would happen if the field 'a' is updated
|
| 337 | remotely from one processor and the local processor would use this_cpu ops
|
| 338 | to update field b. Care should be taken that such simultaneous accesses to
|
| 339 | data within the same cache line are avoided. Also costly synchronization
|
| 340 | may be necessary. IPIs are generally recommended in such scenarios instead
|
| 341 | of a remote write to the per cpu area of another processor.
|
| 342 |
|
| 343 | Even in cases where the remote writes are rare, please bear in
|
| 344 | mind that a remote write will evict the cache line from the processor
|
| 345 | that most likely will access it. If the processor wakes up and finds a
|
| 346 | missing local cache line of a per cpu area, its performance and hence
|
| 347 | the wake up times will be affected.
|