| 1 | |
| 2 |
|
| 3 | ================
|
| 4 | Perf ring buffer
|
| 5 | ================
|
| 6 |
|
| 7 | |
| 8 |
|
| 9 | 1. Introduction
|
| 10 |
|
| 11 | 2. Ring buffer implementation
|
| 12 | 2.1 Basic algorithm
|
| 13 | 2.2 Ring buffer for different tracing modes
|
| 14 | 2.2.1 Default mode
|
| 15 | 2.2.2 Per-thread mode
|
| 16 | 2.2.3 Per-CPU mode
|
| 17 | 2.2.4 System wide mode
|
| 18 | 2.3 Accessing buffer
|
| 19 | 2.3.1 Producer-consumer model
|
| 20 | 2.3.2 Properties of the ring buffers
|
| 21 | 2.3.3 Writing samples into buffer
|
| 22 | 2.3.4 Reading samples from buffer
|
| 23 | 2.3.5 Memory synchronization
|
| 24 |
|
| 25 | 3. The mechanism of AUX ring buffer
|
| 26 | 3.1 The relationship between AUX and regular ring buffers
|
| 27 | 3.2 AUX events
|
| 28 | 3.3 Snapshot mode
|
| 29 |
|
| 30 |
|
| 31 | 1. Introduction
|
| 32 | ===============
|
| 33 |
|
| 34 | The ring buffer is a fundamental mechanism for data transfer. perf uses
|
| 35 | ring buffers to transfer event data from kernel to user space, another
|
| 36 | kind of ring buffer which is so called auxiliary (AUX) ring buffer also
|
| 37 | plays an important role for hardware tracing with Intel PT, Arm
|
| 38 | CoreSight, etc.
|
| 39 |
|
| 40 | The ring buffer implementation is critical but it's also a very
|
| 41 | challenging work. On the one hand, the kernel and perf tool in the user
|
| 42 | space use the ring buffer to exchange data and stores data into data
|
| 43 | file, thus the ring buffer needs to transfer data with high throughput;
|
| 44 | on the other hand, the ring buffer management should avoid significant
|
| 45 | overload to distract profiling results.
|
| 46 |
|
| 47 | This documentation dives into the details for perf ring buffer with two
|
| 48 | parts: firstly it explains the perf ring buffer implementation, then the
|
| 49 | second part discusses the AUX ring buffer mechanism.
|
| 50 |
|
| 51 | 2. Ring buffer implementation
|
| 52 | =============================
|
| 53 |
|
| 54 | 2.1 Basic algorithm
|
| 55 | -------------------
|
| 56 |
|
| 57 | That said, a typical ring buffer is managed by a head pointer and a tail
|
| 58 | pointer; the head pointer is manipulated by a writer and the tail
|
| 59 | pointer is updated by a reader respectively.
|
| 60 |
|
| 61 | ::
|
| 62 |
|
| 63 | +---------------------------+
|
| 64 | | | |***|***|***| | |
|
| 65 | +---------------------------+
|
| 66 | `-> Tail `-> Head
|
| 67 |
|
| 68 | * : the data is filled by the writer.
|
| 69 |
|
| 70 | Figure 1. Ring buffer
|
| 71 |
|
| 72 | Perf uses the same way to manage its ring buffer. In the implementation
|
| 73 | there are two key data structures held together in a set of consecutive
|
| 74 | pages, the control structure and then the ring buffer itself. The page
|
| 75 | with the control structure in is known as the "user page". Being held
|
| 76 | in continuous virtual addresses simplifies locating the ring buffer
|
| 77 | address, it is in the pages after the page with the user page.
|
| 78 |
|
| 79 | The control structure is named as ``perf_event_mmap_page``, it contains a
|
| 80 | head pointer ``data_head`` and a tail pointer ``data_tail``. When the
|
| 81 | kernel starts to fill records into the ring buffer, it updates the head
|
| 82 | pointer to reserve the memory so later it can safely store events into
|
| 83 | the buffer. On the other side, when the user page is a writable mapping,
|
| 84 | the perf tool has the permission to update the tail pointer after consuming
|
| 85 | data from the ring buffer. Yet another case is for the user page's
|
| 86 | read-only mapping, which is to be addressed in the section
|
| 87 | :ref:`writing_samples_into_buffer`.
|
| 88 |
|
| 89 | ::
|
| 90 |
|
| 91 | user page ring buffer
|
| 92 | +---------+---------+ +---------------------------------------+
|
| 93 | |data_head|data_tail|...| | |***|***|***|***|***| | | |
|
| 94 | +---------+---------+ +---------------------------------------+
|
| 95 | ` `----------------^ ^
|
| 96 | `----------------------------------------------|
|
| 97 |
|
| 98 | * : the data is filled by the writer.
|
| 99 |
|
| 100 | Figure 2. Perf ring buffer
|
| 101 |
|
| 102 | When using the ``perf record`` tool, we can specify the ring buffer size
|
| 103 | with option ``-m`` or ``--mmap-pages=``, the given size will be rounded up
|
| 104 | to a power of two that is a multiple of a page size. Though the kernel
|
| 105 | allocates at once for all memory pages, it's deferred to map the pages
|
| 106 | to VMA area until the perf tool accesses the buffer from the user space.
|
| 107 | In other words, at the first time accesses the buffer's page from user
|
| 108 | space in the perf tool, a data abort exception for page fault is taken
|
| 109 | and the kernel uses this occasion to map the page into process VMA
|
| 110 | (see ``perf_mmap_fault()``), thus the perf tool can continue to access
|
| 111 | the page after returning from the exception.
|
| 112 |
|
| 113 | 2.2 Ring buffer for different tracing modes
|
| 114 | -------------------------------------------
|
| 115 |
|
| 116 | The perf profiles programs with different modes: default mode, per thread
|
| 117 | mode, per cpu mode, and system wide mode. This section describes these
|
| 118 | modes and how the ring buffer meets requirements for them. At last we
|
| 119 | will review the race conditions caused by these modes.
|
| 120 |
|
| 121 | 2.2.1 Default mode
|
| 122 | ^^^^^^^^^^^^^^^^^^
|
| 123 |
|
| 124 | Usually we execute ``perf record`` command followed by a profiling program
|
| 125 | name, like below command::
|
| 126 |
|
| 127 | perf record test_program
|
| 128 |
|
| 129 | This command doesn't specify any options for CPU and thread modes, the
|
| 130 | perf tool applies the default mode on the perf event. It maps all the
|
| 131 | CPUs in the system and the profiled program's PID on the perf event, and
|
| 132 | it enables inheritance mode on the event so that child tasks inherits
|
| 133 | the events. As a result, the perf event is attributed as::
|
| 134 |
|
| 135 | evsel::cpus::map[] = { 0 .. _SC_NPROCESSORS_ONLN-1 }
|
| 136 | evsel::threads::map[] = { pid }
|
| 137 | evsel::attr::inherit = 1
|
| 138 |
|
| 139 | These attributions finally will be reflected on the deployment of ring
|
| 140 | buffers. As shown below, the perf tool allocates individual ring buffer
|
| 141 | for each CPU, but it only enables events for the profiled program rather
|
| 142 | than for all threads in the system. The *T1* thread represents the
|
| 143 | thread context of the 'test_program', whereas *T2* and *T3* are irrelevant
|
| 144 | threads in the system. The perf samples are exclusively collected for
|
| 145 | the *T1* thread and stored in the ring buffer associated with the CPU on
|
| 146 | which the *T1* thread is running.
|
| 147 |
|
| 148 | ::
|
| 149 |
|
| 150 | T1 T2 T1
|
| 151 | +----+ +-----------+ +----+
|
| 152 | CPU0 |xxxx| |xxxxxxxxxxx| |xxxx|
|
| 153 | +----+--------------+-----------+----------+----+-------->
|
| 154 | | |
|
| 155 | v v
|
| 156 | +-----------------------------------------------------+
|
| 157 | | Ring buffer 0 |
|
| 158 | +-----------------------------------------------------+
|
| 159 |
|
| 160 | T1
|
| 161 | +-----+
|
| 162 | CPU1 |xxxxx|
|
| 163 | -----+-----+--------------------------------------------->
|
| 164 | |
|
| 165 | v
|
| 166 | +-----------------------------------------------------+
|
| 167 | | Ring buffer 1 |
|
| 168 | +-----------------------------------------------------+
|
| 169 |
|
| 170 | T1 T3
|
| 171 | +----+ +-------+
|
| 172 | CPU2 |xxxx| |xxxxxxx|
|
| 173 | --------------------------+----+--------+-------+-------->
|
| 174 | |
|
| 175 | v
|
| 176 | +-----------------------------------------------------+
|
| 177 | | Ring buffer 2 |
|
| 178 | +-----------------------------------------------------+
|
| 179 |
|
| 180 | T1
|
| 181 | +--------------+
|
| 182 | CPU3 |xxxxxxxxxxxxxx|
|
| 183 | -----------+--------------+------------------------------>
|
| 184 | |
|
| 185 | v
|
| 186 | +-----------------------------------------------------+
|
| 187 | | Ring buffer 3 |
|
| 188 | +-----------------------------------------------------+
|
| 189 |
|
| 190 | T1: Thread 1; T2: Thread 2; T3: Thread 3
|
| 191 | x: Thread is in running state
|
| 192 |
|
| 193 | Figure 3. Ring buffer for default mode
|
| 194 |
|
| 195 | 2.2.2 Per-thread mode
|
| 196 | ^^^^^^^^^^^^^^^^^^^^^
|
| 197 |
|
| 198 | By specifying option ``--per-thread`` in perf command, e.g.
|
| 199 |
|
| 200 | ::
|
| 201 |
|
| 202 | perf record --per-thread test_program
|
| 203 |
|
| 204 | The perf event doesn't map to any CPUs and is only bound to the
|
| 205 | profiled process, thus, the perf event's attributions are::
|
| 206 |
|
| 207 | evsel::cpus::map[0] = { -1 }
|
| 208 | evsel::threads::map[] = { pid }
|
| 209 | evsel::attr::inherit = 0
|
| 210 |
|
| 211 | In this mode, a single ring buffer is allocated for the profiled thread;
|
| 212 | if the thread is scheduled on a CPU, the events on that CPU will be
|
| 213 | enabled; and if the thread is scheduled out from the CPU, the events on
|
| 214 | the CPU will be disabled. When the thread is migrated from one CPU to
|
| 215 | another, the events are to be disabled on the previous CPU and enabled
|
| 216 | on the next CPU correspondingly.
|
| 217 |
|
| 218 | ::
|
| 219 |
|
| 220 | T1 T2 T1
|
| 221 | +----+ +-----------+ +----+
|
| 222 | CPU0 |xxxx| |xxxxxxxxxxx| |xxxx|
|
| 223 | +----+--------------+-----------+----------+----+-------->
|
| 224 | | |
|
| 225 | | T1 |
|
| 226 | | +-----+ |
|
| 227 | CPU1 | |xxxxx| |
|
| 228 | --|--+-----+----------------------------------|---------->
|
| 229 | | | |
|
| 230 | | | T1 T3 |
|
| 231 | | | +----+ +---+ |
|
| 232 | CPU2 | | |xxxx| |xxx| |
|
| 233 | --|-----|-----------------+----+--------+---+-|---------->
|
| 234 | | | | |
|
| 235 | | | T1 | |
|
| 236 | | | +--------------+ | |
|
| 237 | CPU3 | | |xxxxxxxxxxxxxx| | |
|
| 238 | --|-----|--+--------------+-|-----------------|---------->
|
| 239 | | | | | |
|
| 240 | v v v v v
|
| 241 | +-----------------------------------------------------+
|
| 242 | | Ring buffer |
|
| 243 | +-----------------------------------------------------+
|
| 244 |
|
| 245 | T1: Thread 1
|
| 246 | x: Thread is in running state
|
| 247 |
|
| 248 | Figure 4. Ring buffer for per-thread mode
|
| 249 |
|
| 250 | When perf runs in per-thread mode, a ring buffer is allocated for the
|
| 251 | profiled thread *T1*. The ring buffer is dedicated for thread *T1*, if the
|
| 252 | thread *T1* is running, the perf events will be recorded into the ring
|
| 253 | buffer; when the thread is sleeping, all associated events will be
|
| 254 | disabled, thus no trace data will be recorded into the ring buffer.
|
| 255 |
|
| 256 | 2.2.3 Per-CPU mode
|
| 257 | ^^^^^^^^^^^^^^^^^^
|
| 258 |
|
| 259 | The option ``-C`` is used to collect samples on the list of CPUs, for
|
| 260 | example the below perf command receives option ``-C 0,2``::
|
| 261 |
|
| 262 | perf record -C 0,2 test_program
|
| 263 |
|
| 264 | It maps the perf event to CPUs 0 and 2, and the event is not associated to any
|
| 265 | PID. Thus the perf event attributions are set as::
|
| 266 |
|
| 267 | evsel::cpus::map[0] = { 0, 2 }
|
| 268 | evsel::threads::map[] = { -1 }
|
| 269 | evsel::attr::inherit = 0
|
| 270 |
|
| 271 | This results in the session of ``perf record`` will sample all threads on CPU0
|
| 272 | and CPU2, and be terminated until test_program exits. Even there have tasks
|
| 273 | running on CPU1 and CPU3, since the ring buffer is absent for them, any
|
| 274 | activities on these two CPUs will be ignored. A usage case is to combine the
|
| 275 | options for per-thread mode and per-CPU mode, e.g. the options ``–C 0,2`` and
|
| 276 | ``––per–thread`` are specified together, the samples are recorded only when
|
| 277 | the profiled thread is scheduled on any of the listed CPUs.
|
| 278 |
|
| 279 | ::
|
| 280 |
|
| 281 | T1 T2 T1
|
| 282 | +----+ +-----------+ +----+
|
| 283 | CPU0 |xxxx| |xxxxxxxxxxx| |xxxx|
|
| 284 | +----+--------------+-----------+----------+----+-------->
|
| 285 | | | |
|
| 286 | v v v
|
| 287 | +-----------------------------------------------------+
|
| 288 | | Ring buffer 0 |
|
| 289 | +-----------------------------------------------------+
|
| 290 |
|
| 291 | T1
|
| 292 | +-----+
|
| 293 | CPU1 |xxxxx|
|
| 294 | -----+-----+--------------------------------------------->
|
| 295 |
|
| 296 | T1 T3
|
| 297 | +----+ +-------+
|
| 298 | CPU2 |xxxx| |xxxxxxx|
|
| 299 | --------------------------+----+--------+-------+-------->
|
| 300 | | |
|
| 301 | v v
|
| 302 | +-----------------------------------------------------+
|
| 303 | | Ring buffer 1 |
|
| 304 | +-----------------------------------------------------+
|
| 305 |
|
| 306 | T1
|
| 307 | +--------------+
|
| 308 | CPU3 |xxxxxxxxxxxxxx|
|
| 309 | -----------+--------------+------------------------------>
|
| 310 |
|
| 311 | T1: Thread 1; T2: Thread 2; T3: Thread 3
|
| 312 | x: Thread is in running state
|
| 313 |
|
| 314 | Figure 5. Ring buffer for per-CPU mode
|
| 315 |
|
| 316 | 2.2.4 System wide mode
|
| 317 | ^^^^^^^^^^^^^^^^^^^^^^
|
| 318 |
|
| 319 | By using option ``–a`` or ``––all–cpus``, perf collects samples on all CPUs
|
| 320 | for all tasks, we call it as the system wide mode, the command is::
|
| 321 |
|
| 322 | perf record -a test_program
|
| 323 |
|
| 324 | Similar to the per-CPU mode, the perf event doesn't bind to any PID, and
|
| 325 | it maps to all CPUs in the system::
|
| 326 |
|
| 327 | evsel::cpus::map[] = { 0 .. _SC_NPROCESSORS_ONLN-1 }
|
| 328 | evsel::threads::map[] = { -1 }
|
| 329 | evsel::attr::inherit = 0
|
| 330 |
|
| 331 | In the system wide mode, every CPU has its own ring buffer, all threads
|
| 332 | are monitored during the running state and the samples are recorded into
|
| 333 | the ring buffer belonging to the CPU which the events occurred on.
|
| 334 |
|
| 335 | ::
|
| 336 |
|
| 337 | T1 T2 T1
|
| 338 | +----+ +-----------+ +----+
|
| 339 | CPU0 |xxxx| |xxxxxxxxxxx| |xxxx|
|
| 340 | +----+--------------+-----------+----------+----+-------->
|
| 341 | | | |
|
| 342 | v v v
|
| 343 | +-----------------------------------------------------+
|
| 344 | | Ring buffer 0 |
|
| 345 | +-----------------------------------------------------+
|
| 346 |
|
| 347 | T1
|
| 348 | +-----+
|
| 349 | CPU1 |xxxxx|
|
| 350 | -----+-----+--------------------------------------------->
|
| 351 | |
|
| 352 | v
|
| 353 | +-----------------------------------------------------+
|
| 354 | | Ring buffer 1 |
|
| 355 | +-----------------------------------------------------+
|
| 356 |
|
| 357 | T1 T3
|
| 358 | +----+ +-------+
|
| 359 | CPU2 |xxxx| |xxxxxxx|
|
| 360 | --------------------------+----+--------+-------+-------->
|
| 361 | | |
|
| 362 | v v
|
| 363 | +-----------------------------------------------------+
|
| 364 | | Ring buffer 2 |
|
| 365 | +-----------------------------------------------------+
|
| 366 |
|
| 367 | T1
|
| 368 | +--------------+
|
| 369 | CPU3 |xxxxxxxxxxxxxx|
|
| 370 | -----------+--------------+------------------------------>
|
| 371 | |
|
| 372 | v
|
| 373 | +-----------------------------------------------------+
|
| 374 | | Ring buffer 3 |
|
| 375 | +-----------------------------------------------------+
|
| 376 |
|
| 377 | T1: Thread 1; T2: Thread 2; T3: Thread 3
|
| 378 | x: Thread is in running state
|
| 379 |
|
| 380 | Figure 6. Ring buffer for system wide mode
|
| 381 |
|
| 382 | 2.3 Accessing buffer
|
| 383 | --------------------
|
| 384 |
|
| 385 | Based on the understanding of how the ring buffer is allocated in
|
| 386 | various modes, this section explains access the ring buffer.
|
| 387 |
|
| 388 | 2.3.1 Producer-consumer model
|
| 389 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 390 |
|
| 391 | In the Linux kernel, the PMU events can produce samples which are stored
|
| 392 | into the ring buffer; the perf command in user space consumes the
|
| 393 | samples by reading out data from the ring buffer and finally saves the
|
| 394 | data into the file for post analysis. It’s a typical producer-consumer
|
| 395 | model for using the ring buffer.
|
| 396 |
|
| 397 | The perf process polls on the PMU events and sleeps when no events are
|
| 398 | incoming. To prevent frequent exchanges between the kernel and user
|
| 399 | space, the kernel event core layer introduces a watermark, which is
|
| 400 | stored in the ``perf_buffer::watermark``. When a sample is recorded into
|
| 401 | the ring buffer, and if the used buffer exceeds the watermark, the
|
| 402 | kernel wakes up the perf process to read samples from the ring buffer.
|
| 403 |
|
| 404 | ::
|
| 405 |
|
| 406 | Perf
|
| 407 | / | Read samples
|
| 408 | Polling / `--------------| Ring buffer
|
| 409 | v v ;---------------------v
|
| 410 | +----------------+ +---------+---------+ +-------------------+
|
| 411 | |Event wait queue| |data_head|data_tail| |***|***| | |***|
|
| 412 | +----------------+ +---------+---------+ +-------------------+
|
| 413 | ^ ^ `------------------------^
|
| 414 | | Wake up tasks | Store samples
|
| 415 | +-----------------------------+
|
| 416 | | Kernel event core layer |
|
| 417 | +-----------------------------+
|
| 418 |
|
| 419 | * : the data is filled by the writer.
|
| 420 |
|
| 421 | Figure 7. Writing and reading the ring buffer
|
| 422 |
|
| 423 | When the kernel event core layer notifies the user space, because
|
| 424 | multiple events might share the same ring buffer for recording samples,
|
| 425 | the core layer iterates every event associated with the ring buffer and
|
| 426 | wakes up tasks waiting on the event. This is fulfilled by the kernel
|
| 427 | function ``ring_buffer_wakeup()``.
|
| 428 |
|
| 429 | After the perf process is woken up, it starts to check the ring buffers
|
| 430 | one by one, if it finds any ring buffer containing samples it will read
|
| 431 | out the samples for statistics or saving into the data file. Given the
|
| 432 | perf process is able to run on any CPU, this leads to the ring buffer
|
| 433 | potentially being accessed from multiple CPUs simultaneously, which
|
| 434 | causes race conditions. The race condition handling is described in the
|
| 435 | section :ref:`memory_synchronization`.
|
| 436 |
|
| 437 | 2.3.2 Properties of the ring buffers
|
| 438 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 439 |
|
| 440 | Linux kernel supports two write directions for the ring buffer: forward and
|
| 441 | backward. The forward writing saves samples from the beginning of the ring
|
| 442 | buffer, the backward writing stores data from the end of the ring buffer with
|
| 443 | the reversed direction. The perf tool determines the writing direction.
|
| 444 |
|
| 445 | Additionally, the tool can map buffers in either read-write mode or read-only
|
| 446 | mode to the user space.
|
| 447 |
|
| 448 | The ring buffer in the read-write mode is mapped with the property
|
| 449 | ``PROT_READ | PROT_WRITE``. With the write permission, the perf tool
|
| 450 | updates the ``data_tail`` to indicate the data start position. Combining
|
| 451 | with the head pointer ``data_head``, which works as the end position of
|
| 452 | the current data, the perf tool can easily know where read out the data
|
| 453 | from.
|
| 454 |
|
| 455 | Alternatively, in the read-only mode, only the kernel keeps to update
|
| 456 | the ``data_head`` while the user space cannot access the ``data_tail`` due
|
| 457 | to the mapping property ``PROT_READ``.
|
| 458 |
|
| 459 | As a result, the matrix below illustrates the various combinations of
|
| 460 | direction and mapping characteristics. The perf tool employs two of these
|
| 461 | combinations to support buffer types: the non-overwrite buffer and the
|
| 462 | overwritable buffer.
|
| 463 |
|
| 464 | .. list-table::
|
| 465 | :widths: 1 1 1
|
| 466 | :header-rows: 1
|
| 467 |
|
| 468 | * - Mapping mode
|
| 469 | - Forward
|
| 470 | - Backward
|
| 471 | * - read-write
|
| 472 | - Non-overwrite ring buffer
|
| 473 | - Not used
|
| 474 | * - read-only
|
| 475 | - Not used
|
| 476 | - Overwritable ring buffer
|
| 477 |
|
| 478 | The non-overwrite ring buffer uses the read-write mapping with forward
|
| 479 | writing. It starts to save data from the beginning of the ring buffer
|
| 480 | and wrap around when overflow, which is used with the read-write mode in
|
| 481 | the normal ring buffer. When the consumer doesn't keep up with the
|
| 482 | producer, it would lose some data, the kernel keeps how many records it
|
| 483 | lost and generates the ``PERF_RECORD_LOST`` records in the next time
|
| 484 | when it finds a space in the ring buffer.
|
| 485 |
|
| 486 | The overwritable ring buffer uses the backward writing with the
|
| 487 | read-only mode. It saves the data from the end of the ring buffer and
|
| 488 | the ``data_head`` keeps the position of current data, the perf always
|
| 489 | knows where it starts to read and until the end of the ring buffer, thus
|
| 490 | it don't need the ``data_tail``. In this mode, it will not generate the
|
| 491 | ``PERF_RECORD_LOST`` records.
|
| 492 |
|
| 493 | .. _writing_samples_into_buffer:
|
| 494 |
|
| 495 | 2.3.3 Writing samples into buffer
|
| 496 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 497 |
|
| 498 | When a sample is taken and saved into the ring buffer, the kernel
|
| 499 | prepares sample fields based on the sample type; then it prepares the
|
| 500 | info for writing ring buffer which is stored in the structure
|
| 501 | ``perf_output_handle``. In the end, the kernel outputs the sample into
|
| 502 | the ring buffer and updates the head pointer in the user page so the
|
| 503 | perf tool can see the latest value.
|
| 504 |
|
| 505 | The structure ``perf_output_handle`` serves as a temporary context for
|
| 506 | tracking the information related to the buffer. The advantages of it is
|
| 507 | that it enables concurrent writing to the buffer by different events.
|
| 508 | For example, a software event and a hardware PMU event both are enabled
|
| 509 | for profiling, two instances of ``perf_output_handle`` serve as separate
|
| 510 | contexts for the software event and the hardware event respectively.
|
| 511 | This allows each event to reserve its own memory space for populating
|
| 512 | the record data.
|
| 513 |
|
| 514 | 2.3.4 Reading samples from buffer
|
| 515 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 516 |
|
| 517 | In the user space, the perf tool utilizes the ``perf_event_mmap_page``
|
| 518 | structure to handle the head and tail of the buffer. It also uses
|
| 519 | ``perf_mmap`` structure to keep track of a context for the ring buffer, this
|
| 520 | context includes information about the buffer's starting and ending
|
| 521 | addresses. Additionally, the mask value can be utilized to compute the
|
| 522 | circular buffer pointer even for an overflow.
|
| 523 |
|
| 524 | Similar to the kernel, the perf tool in the user space first reads out
|
| 525 | the recorded data from the ring buffer, and then updates the buffer's
|
| 526 | tail pointer ``perf_event_mmap_page::data_tail``.
|
| 527 |
|
| 528 | .. _memory_synchronization:
|
| 529 |
|
| 530 | 2.3.5 Memory synchronization
|
| 531 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 532 |
|
| 533 | The modern CPUs with relaxed memory model cannot promise the memory
|
| 534 | ordering, this means it’s possible to access the ring buffer and the
|
| 535 | ``perf_event_mmap_page`` structure out of order. To assure the specific
|
| 536 | sequence for memory accessing perf ring buffer, memory barriers are
|
| 537 | used to assure the data dependency. The rationale for the memory
|
| 538 | synchronization is as below::
|
| 539 |
|
| 540 | Kernel User space
|
| 541 |
|
| 542 | if (LOAD ->data_tail) { LOAD ->data_head
|
| 543 | (A) smp_rmb() (C)
|
| 544 | STORE $data LOAD $data
|
| 545 | smp_wmb() (B) smp_mb() (D)
|
| 546 | STORE ->data_head STORE ->data_tail
|
| 547 | }
|
| 548 |
|
| 549 | The comments in tools/include/linux/ring_buffer.h gives nice description
|
| 550 | for why and how to use memory barriers, here we will just provide an
|
| 551 | alternative explanation:
|
| 552 |
|
| 553 | (A) is a control dependency so that CPU assures order between checking
|
| 554 | pointer ``perf_event_mmap_page::data_tail`` and filling sample into ring
|
| 555 | buffer;
|
| 556 |
|
| 557 | (D) pairs with (A). (D) separates the ring buffer data reading from
|
| 558 | writing the pointer ``data_tail``, perf tool first consumes samples and then
|
| 559 | tells the kernel that the data chunk has been released. Since a reading
|
| 560 | operation is followed by a writing operation, thus (D) is a full memory
|
| 561 | barrier.
|
| 562 |
|
| 563 | (B) is a writing barrier in the middle of two writing operations, which
|
| 564 | makes sure that recording a sample must be prior to updating the head
|
| 565 | pointer.
|
| 566 |
|
| 567 | (C) pairs with (B). (C) is a read memory barrier to ensure the head
|
| 568 | pointer is fetched before reading samples.
|
| 569 |
|
| 570 | To implement the above algorithm, the ``perf_output_put_handle()`` function
|
| 571 | in the kernel and two helpers ``ring_buffer_read_head()`` and
|
| 572 | ``ring_buffer_write_tail()`` in the user space are introduced, they rely
|
| 573 | on memory barriers as described above to ensure the data dependency.
|
| 574 |
|
| 575 | Some architectures support one-way permeable barrier with load-acquire
|
| 576 | and store-release operations, these barriers are more relaxed with less
|
| 577 | performance penalty, so (C) and (D) can be optimized to use barriers
|
| 578 | ``smp_load_acquire()`` and ``smp_store_release()`` respectively.
|
| 579 |
|
| 580 | If an architecture doesn’t support load-acquire and store-release in its
|
| 581 | memory model, it will roll back to the old fashion of memory barrier
|
| 582 | operations. In this case, ``smp_load_acquire()`` encapsulates
|
| 583 | ``READ_ONCE()`` + ``smp_mb()``, since ``smp_mb()`` is costly,
|
| 584 | ``ring_buffer_read_head()`` doesn't invoke ``smp_load_acquire()`` and it uses
|
| 585 | the barriers ``READ_ONCE()`` + ``smp_rmb()`` instead.
|
| 586 |
|
| 587 | 3. The mechanism of AUX ring buffer
|
| 588 | ===================================
|
| 589 |
|
| 590 | In this chapter, we will explain the implementation of the AUX ring
|
| 591 | buffer. In the first part it will discuss the connection between the
|
| 592 | AUX ring buffer and the regular ring buffer, then the second part will
|
| 593 | examine how the AUX ring buffer co-works with the regular ring buffer,
|
| 594 | as well as the additional features introduced by the AUX ring buffer for
|
| 595 | the sampling mechanism.
|
| 596 |
|
| 597 | 3.1 The relationship between AUX and regular ring buffers
|
| 598 | ---------------------------------------------------------
|
| 599 |
|
| 600 | Generally, the AUX ring buffer is an auxiliary for the regular ring
|
| 601 | buffer. The regular ring buffer is primarily used to store the event
|
| 602 | samples and every event format complies with the definition in the
|
| 603 | union ``perf_event``; the AUX ring buffer is for recording the hardware
|
| 604 | trace data and the trace data format is hardware IP dependent.
|
| 605 |
|
| 606 | The general use and advantage of the AUX ring buffer is that it is
|
| 607 | written directly by hardware rather than by the kernel. For example,
|
| 608 | regular profile samples that write to the regular ring buffer cause an
|
| 609 | interrupt. Tracing execution requires a high number of samples and
|
| 610 | using interrupts would be overwhelming for the regular ring buffer
|
| 611 | mechanism. Having an AUX buffer allows for a region of memory more
|
| 612 | decoupled from the kernel and written to directly by hardware tracing.
|
| 613 |
|
| 614 | The AUX ring buffer reuses the same algorithm with the regular ring
|
| 615 | buffer for the buffer management. The control structure
|
| 616 | ``perf_event_mmap_page`` extends the new fields ``aux_head`` and ``aux_tail``
|
| 617 | for the head and tail pointers of the AUX ring buffer.
|
| 618 |
|
| 619 | During the initialisation phase, besides the mmap()-ed regular ring
|
| 620 | buffer, the perf tool invokes a second syscall in the
|
| 621 | ``auxtrace_mmap__mmap()`` function for the mmap of the AUX buffer with
|
| 622 | non-zero file offset; ``rb_alloc_aux()`` in the kernel allocates pages
|
| 623 | correspondingly, these pages will be deferred to map into VMA when
|
| 624 | handling the page fault, which is the same lazy mechanism with the
|
| 625 | regular ring buffer.
|
| 626 |
|
| 627 | AUX events and AUX trace data are two different things. Let's see an
|
| 628 | example::
|
| 629 |
|
| 630 | perf record -a -e cycles -e cs_etm// -- sleep 2
|
| 631 |
|
| 632 | The above command enables two events: one is the event *cycles* from PMU
|
| 633 | and another is the AUX event *cs_etm* from Arm CoreSight, both are saved
|
| 634 | into the regular ring buffer while the CoreSight's AUX trace data is
|
| 635 | stored in the AUX ring buffer.
|
| 636 |
|
| 637 | As a result, we can see the regular ring buffer and the AUX ring buffer
|
| 638 | are allocated in pairs. The perf in default mode allocates the regular
|
| 639 | ring buffer and the AUX ring buffer per CPU-wise, which is the same as
|
| 640 | the system wide mode, however, the default mode records samples only for
|
| 641 | the profiled program, whereas the latter mode profiles for all programs
|
| 642 | in the system. For per-thread mode, the perf tool allocates only one
|
| 643 | regular ring buffer and one AUX ring buffer for the whole session. For
|
| 644 | the per-CPU mode, the perf allocates two kinds of ring buffers for
|
| 645 | selected CPUs specified by the option ``-C``.
|
| 646 |
|
| 647 | The below figure demonstrates the buffers' layout in the system wide
|
| 648 | mode; if there are any activities on one CPU, the AUX event samples and
|
| 649 | the hardware trace data will be recorded into the dedicated buffers for
|
| 650 | the CPU.
|
| 651 |
|
| 652 | ::
|
| 653 |
|
| 654 | T1 T2 T1
|
| 655 | +----+ +-----------+ +----+
|
| 656 | CPU0 |xxxx| |xxxxxxxxxxx| |xxxx|
|
| 657 | +----+--------------+-----------+----------+----+-------->
|
| 658 | | | |
|
| 659 | v v v
|
| 660 | +-----------------------------------------------------+
|
| 661 | | Ring buffer 0 |
|
| 662 | +-----------------------------------------------------+
|
| 663 | | | |
|
| 664 | v v v
|
| 665 | +-----------------------------------------------------+
|
| 666 | | AUX Ring buffer 0 |
|
| 667 | +-----------------------------------------------------+
|
| 668 |
|
| 669 | T1
|
| 670 | +-----+
|
| 671 | CPU1 |xxxxx|
|
| 672 | -----+-----+--------------------------------------------->
|
| 673 | |
|
| 674 | v
|
| 675 | +-----------------------------------------------------+
|
| 676 | | Ring buffer 1 |
|
| 677 | +-----------------------------------------------------+
|
| 678 | |
|
| 679 | v
|
| 680 | +-----------------------------------------------------+
|
| 681 | | AUX Ring buffer 1 |
|
| 682 | +-----------------------------------------------------+
|
| 683 |
|
| 684 | T1 T3
|
| 685 | +----+ +-------+
|
| 686 | CPU2 |xxxx| |xxxxxxx|
|
| 687 | --------------------------+----+--------+-------+-------->
|
| 688 | | |
|
| 689 | v v
|
| 690 | +-----------------------------------------------------+
|
| 691 | | Ring buffer 2 |
|
| 692 | +-----------------------------------------------------+
|
| 693 | | |
|
| 694 | v v
|
| 695 | +-----------------------------------------------------+
|
| 696 | | AUX Ring buffer 2 |
|
| 697 | +-----------------------------------------------------+
|
| 698 |
|
| 699 | T1
|
| 700 | +--------------+
|
| 701 | CPU3 |xxxxxxxxxxxxxx|
|
| 702 | -----------+--------------+------------------------------>
|
| 703 | |
|
| 704 | v
|
| 705 | +-----------------------------------------------------+
|
| 706 | | Ring buffer 3 |
|
| 707 | +-----------------------------------------------------+
|
| 708 | |
|
| 709 | v
|
| 710 | +-----------------------------------------------------+
|
| 711 | | AUX Ring buffer 3 |
|
| 712 | +-----------------------------------------------------+
|
| 713 |
|
| 714 | T1: Thread 1; T2: Thread 2; T3: Thread 3
|
| 715 | x: Thread is in running state
|
| 716 |
|
| 717 | Figure 8. AUX ring buffer for system wide mode
|
| 718 |
|
| 719 | 3.2 AUX events
|
| 720 | --------------
|
| 721 |
|
| 722 | Similar to ``perf_output_begin()`` and ``perf_output_end()``'s working for the
|
| 723 | regular ring buffer, ``perf_aux_output_begin()`` and ``perf_aux_output_end()``
|
| 724 | serve for the AUX ring buffer for processing the hardware trace data.
|
| 725 |
|
| 726 | Once the hardware trace data is stored into the AUX ring buffer, the PMU
|
| 727 | driver will stop hardware tracing by calling the ``pmu::stop()`` callback.
|
| 728 | Similar to the regular ring buffer, the AUX ring buffer needs to apply
|
| 729 | the memory synchronization mechanism as discussed in the section
|
| 730 | :ref:`memory_synchronization`. Since the AUX ring buffer is managed by the
|
| 731 | PMU driver, the barrier (B), which is a writing barrier to ensure the trace
|
| 732 | data is externally visible prior to updating the head pointer, is asked
|
| 733 | to be implemented in the PMU driver.
|
| 734 |
|
| 735 | Then ``pmu::stop()`` can safely call the ``perf_aux_output_end()`` function to
|
| 736 | finish two things:
|
| 737 |
|
| 738 | - It fills an event ``PERF_RECORD_AUX`` into the regular ring buffer, this
|
| 739 | event delivers the information of the start address and data size for a
|
| 740 | chunk of hardware trace data has been stored into the AUX ring buffer;
|
| 741 |
|
| 742 | - Since the hardware trace driver has stored new trace data into the AUX
|
| 743 | ring buffer, the argument *size* indicates how many bytes have been
|
| 744 | consumed by the hardware tracing, thus ``perf_aux_output_end()`` updates the
|
| 745 | header pointer ``perf_buffer::aux_head`` to reflect the latest buffer usage.
|
| 746 |
|
| 747 | At the end, the PMU driver will restart hardware tracing. During this
|
| 748 | temporary suspending period, it will lose hardware trace data, which
|
| 749 | will introduce a discontinuity during decoding phase.
|
| 750 |
|
| 751 | The event ``PERF_RECORD_AUX`` presents an AUX event which is handled in the
|
| 752 | kernel, but it lacks the information for saving the AUX trace data in
|
| 753 | the perf file. When the perf tool copies the trace data from AUX ring
|
| 754 | buffer to the perf data file, it synthesizes a ``PERF_RECORD_AUXTRACE``
|
| 755 | event which is not a kernel ABI, it's defined by the perf tool to describe
|
| 756 | which portion of data in the AUX ring buffer is saved. Afterwards, the perf
|
| 757 | tool reads out the AUX trace data from the perf file based on the
|
| 758 | ``PERF_RECORD_AUXTRACE`` events, and the ``PERF_RECORD_AUX`` event is used to
|
| 759 | decode a chunk of data by correlating with time order.
|
| 760 |
|
| 761 | 3.3 Snapshot mode
|
| 762 | -----------------
|
| 763 |
|
| 764 | Perf supports snapshot mode for AUX ring buffer, in this mode, users
|
| 765 | only record AUX trace data at a specific time point which users are
|
| 766 | interested in. E.g. below gives an example of how to take snapshots
|
| 767 | with 1 second interval with Arm CoreSight::
|
| 768 |
|
| 769 | perf record -e cs_etm//u -S -a program &
|
| 770 | PERFPID=$!
|
| 771 | while true; do
|
| 772 | kill -USR2 $PERFPID
|
| 773 | sleep 1
|
| 774 | done
|
| 775 |
|
| 776 | The main flow for snapshot mode is:
|
| 777 |
|
| 778 | - Before a snapshot is taken, the AUX ring buffer acts in free run mode.
|
| 779 | During free run mode the perf doesn't record any of the AUX events and
|
| 780 | trace data;
|
| 781 |
|
| 782 | - Once the perf tool receives the *USR2* signal, it triggers the callback
|
| 783 | function ``auxtrace_record::snapshot_start()`` to deactivate hardware
|
| 784 | tracing. The kernel driver then populates the AUX ring buffer with the
|
| 785 | hardware trace data, and the event ``PERF_RECORD_AUX`` is stored in the
|
| 786 | regular ring buffer;
|
| 787 |
|
| 788 | - Then perf tool takes a snapshot, ``record__read_auxtrace_snapshot()``
|
| 789 | reads out the hardware trace data from the AUX ring buffer and saves it
|
| 790 | into perf data file;
|
| 791 |
|
| 792 | - After the snapshot is finished, ``auxtrace_record::snapshot_finish()``
|
| 793 | restarts the PMU event for AUX tracing.
|
| 794 |
|
| 795 | The perf only accesses the head pointer ``perf_event_mmap_page::aux_head``
|
| 796 | in snapshot mode and doesn’t touch tail pointer ``aux_tail``, this is
|
| 797 | because the AUX ring buffer can overflow in free run mode, the tail
|
| 798 | pointer is useless in this case. Alternatively, the callback
|
| 799 | ``auxtrace_record::find_snapshot()`` is introduced for making the decision
|
| 800 | of whether the AUX ring buffer has been wrapped around or not, at the
|
| 801 | end it fixes up the AUX buffer's head which are used to calculate the
|
| 802 | trace data size.
|
| 803 |
|
| 804 | As we know, the buffers' deployment can be per-thread mode, per-CPU
|
| 805 | mode, or system wide mode, and the snapshot can be applied to any of
|
| 806 | these modes. Below is an example of taking snapshot with system wide
|
| 807 | mode.
|
| 808 |
|
| 809 | ::
|
| 810 |
|
| 811 | Snapshot is taken
|
| 812 | |
|
| 813 | v
|
| 814 | +------------------------+
|
| 815 | | AUX Ring buffer 0 | <- aux_head
|
| 816 | +------------------------+
|
| 817 | v
|
| 818 | +--------------------------------+
|
| 819 | | AUX Ring buffer 1 | <- aux_head
|
| 820 | +--------------------------------+
|
| 821 | v
|
| 822 | +--------------------------------------------+
|
| 823 | | AUX Ring buffer 2 | <- aux_head
|
| 824 | +--------------------------------------------+
|
| 825 | v
|
| 826 | +---------------------------------------+
|
| 827 | | AUX Ring buffer 3 | <- aux_head
|
| 828 | +---------------------------------------+
|
| 829 |
|
| 830 | Figure 9. Snapshot with system wide mode
|