| 1 | =============
|
| 2 | Ring Buffer
|
| 3 | =============
|
| 4 |
|
| 5 | To handle communication between user space and kernel space, AMD GPUs use a
|
| 6 | ring buffer design to feed the engines (GFX, Compute, SDMA, UVD, VCE, VCN, VPE,
|
| 7 | etc.). See the figure below that illustrates how this communication works:
|
| 8 |
|
| 9 | .. kernel-figure:: ring_buffers.svg
|
| 10 |
|
| 11 | Ring buffers in the amdgpu work as a producer-consumer model, where userspace
|
| 12 | acts as the producer, constantly filling the ring buffer with GPU commands to
|
| 13 | be executed. Meanwhile, the GPU retrieves the information from the ring, parses
|
| 14 | it, and distributes the specific set of instructions between the different
|
| 15 | amdgpu blocks.
|
| 16 |
|
| 17 | Notice from the diagram that the ring has a Read Pointer (rptr), which
|
| 18 | indicates where the engine is currently reading packets from the ring, and a
|
| 19 | Write Pointer (wptr), which indicates how many packets software has added to
|
| 20 | the ring. When the rptr and wptr are equal, the ring is idle. When software
|
| 21 | adds packets to the ring, it updates the wptr, this causes the engine to start
|
| 22 | fetching and processing packets. As the engine processes packets, the rptr gets
|
| 23 | updates until the rptr catches up to the wptr and they are equal again.
|
| 24 |
|
| 25 | Usually, ring buffers in the driver have a limited size (search for occurrences
|
| 26 | of `amdgpu_ring_init()`). One of the reasons for the small ring buffer size is
|
| 27 | that CP (Command Processor) is capable of following addresses inserted into the
|
| 28 | ring; this is illustrated in the image by the reference to the IB (Indirect
|
| 29 | Buffer). The IB gives userspace the possibility to have an area in memory that
|
| 30 | CP can read and feed the hardware with extra instructions.
|
| 31 |
|
| 32 | All ASICs pre-GFX11 use what is called a kernel queue, which means
|
| 33 | the ring is allocated in kernel space and has some restrictions, such as not
|
| 34 | being able to be :ref:`preempted directly by the scheduler<amdgpu-mes>`. GFX11
|
| 35 | and newer support kernel queues, but also provide a new mechanism named
|
| 36 | :ref:`user queues<amdgpu-userq>`, where the queue is moved to the user space
|
| 37 | and can be mapped and unmapped via the scheduler. In practice, both queues
|
| 38 | insert user-space-generated GPU commands from different jobs into the requested
|
| 39 | component ring.
|
| 40 |
|
| 41 | Enforce Isolation
|
| 42 | =================
|
| 43 |
|
| 44 | .. note:: After reading this section, you might want to check the
|
| 45 | :ref:`Process Isolation<amdgpu-process-isolation>` page for more details.
|
| 46 |
|
| 47 | Before examining the Enforce Isolation mechanism in the ring buffer context, it
|
| 48 | is helpful to briefly discuss how instructions from the ring buffer are
|
| 49 | processed in the graphics pipeline. Let’s expand on this topic by checking the
|
| 50 | diagram below that illustrates the graphics pipeline:
|
| 51 |
|
| 52 | .. kernel-figure:: gfx_pipeline_seq.svg
|
| 53 |
|
| 54 | In terms of executing instructions, the GFX pipeline follows the sequence:
|
| 55 | Shader Export (SX), Geometry Engine (GE), Shader Process or Input (SPI), Scan
|
| 56 | Converter (SC), Primitive Assembler (PA), and cache manipulation (which may
|
| 57 | vary across ASICs). Another common way to describe the pipeline is to use Pixel
|
| 58 | Shader (PS), raster, and Vertex Shader (VS) to symbolize the two shader stages.
|
| 59 | Now, with this pipeline in mind, let's assume that Job B causes a hang issue,
|
| 60 | but Job C's instruction might already be executing, leading developers to
|
| 61 | incorrectly identify Job C as the problematic one. This problem can be
|
| 62 | mitigated on multiple levels; the diagram below illustrates how to minimize
|
| 63 | part of this problem:
|
| 64 |
|
| 65 | .. kernel-figure:: no_enforce_isolation.svg
|
| 66 |
|
| 67 | Note from the diagram that there is no guarantee of order or a clear separation
|
| 68 | between instructions, which is not a problem most of the time, and is also good
|
| 69 | for performance. Furthermore, notice some circles between jobs in the diagram
|
| 70 | that represent a **fence wait** used to avoid overlapping work in the ring. At
|
| 71 | the end of the fence, a cache flush occurs, ensuring that when the next job
|
| 72 | starts, it begins in a clean state and, if issues arise, the developer can
|
| 73 | pinpoint the problematic process more precisely.
|
| 74 |
|
| 75 | To increase the level of isolation between jobs, there is the "Enforce
|
| 76 | Isolation" method described in the picture below:
|
| 77 |
|
| 78 | .. kernel-figure:: enforce_isolation.svg
|
| 79 |
|
| 80 | As shown in the diagram, enforcing isolation introduces ordering between
|
| 81 | submissions, since the access to GFX/Compute is serialized, think about it as
|
| 82 | single process at a time mode for gfx/compute. Notice that this approach has a
|
| 83 | significant performance impact, as it allows only one job to submit commands at
|
| 84 | a time. However, this option can help pinpoint the job that caused the problem.
|
| 85 | Although enforcing isolation improves the situation, it does not fully resolve
|
| 86 | the issue of precisely pinpointing bad jobs, since isolation might mask the
|
| 87 | problem. In summary, identifying which job caused the issue may not be precise,
|
| 88 | but enforcing isolation might help with the debugging.
|
| 89 |
|
| 90 | Ring Operations
|
| 91 | ===============
|
| 92 |
|
| 93 | .. kernel-doc:: drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c
|
| 94 | :internal:
|
| 95 |
|