| 1 | |
| 2 |
|
| 3 | ===================
|
| 4 | Userland interfaces
|
| 5 | ===================
|
| 6 |
|
| 7 | The DRM core exports several interfaces to applications, generally
|
| 8 | intended to be used through corresponding libdrm wrapper functions. In
|
| 9 | addition, drivers export device-specific interfaces for use by userspace
|
| 10 | drivers & device-aware applications through ioctls and sysfs files.
|
| 11 |
|
| 12 | External interfaces include: memory mapping, context management, DMA
|
| 13 | operations, AGP management, vblank control, fence management, memory
|
| 14 | management, and output management.
|
| 15 |
|
| 16 | Cover generic ioctls and sysfs layout here. We only need high-level
|
| 17 | info, since man pages should cover the rest.
|
| 18 |
|
| 19 | .. contents::
|
| 20 |
|
| 21 | libdrm Device Lookup
|
| 22 | ====================
|
| 23 |
|
| 24 | .. kernel-doc:: drivers/gpu/drm/drm_ioctl.c
|
| 25 | :doc: getunique and setversion story
|
| 26 |
|
| 27 |
|
| 28 | .. _drm_primary_node:
|
| 29 |
|
| 30 | Primary Nodes, DRM Master and Authentication
|
| 31 | ============================================
|
| 32 |
|
| 33 | .. kernel-doc:: drivers/gpu/drm/drm_auth.c
|
| 34 | :doc: master and authentication
|
| 35 |
|
| 36 | .. kernel-doc:: drivers/gpu/drm/drm_auth.c
|
| 37 | :export:
|
| 38 |
|
| 39 | .. kernel-doc:: include/drm/drm_auth.h
|
| 40 | :internal:
|
| 41 |
|
| 42 |
|
| 43 | .. _drm_leasing:
|
| 44 |
|
| 45 | DRM Display Resource Leasing
|
| 46 | ============================
|
| 47 |
|
| 48 | .. kernel-doc:: drivers/gpu/drm/drm_lease.c
|
| 49 | :doc: drm leasing
|
| 50 |
|
| 51 | Open-Source Userspace Requirements
|
| 52 | ==================================
|
| 53 |
|
| 54 | The DRM subsystem has stricter requirements than most other kernel subsystems on
|
| 55 | what the userspace side for new uAPI needs to look like. This section here
|
| 56 | explains what exactly those requirements are, and why they exist.
|
| 57 |
|
| 58 | The short summary is that any addition of DRM uAPI requires corresponding
|
| 59 | open-sourced userspace patches, and those patches must be reviewed and ready for
|
| 60 | merging into a suitable and canonical upstream project.
|
| 61 |
|
| 62 | GFX devices (both display and render/GPU side) are really complex bits of
|
| 63 | hardware, with userspace and kernel by necessity having to work together really
|
| 64 | closely. The interfaces, for rendering and modesetting, must be extremely wide
|
| 65 | and flexible, and therefore it is almost always impossible to precisely define
|
| 66 | them for every possible corner case. This in turn makes it really practically
|
| 67 | infeasible to differentiate between behaviour that's required by userspace, and
|
| 68 | which must not be changed to avoid regressions, and behaviour which is only an
|
| 69 | accidental artifact of the current implementation.
|
| 70 |
|
| 71 | Without access to the full source code of all userspace users that means it
|
| 72 | becomes impossible to change the implementation details, since userspace could
|
| 73 | depend upon the accidental behaviour of the current implementation in minute
|
| 74 | details. And debugging such regressions without access to source code is pretty
|
| 75 | much impossible. As a consequence this means:
|
| 76 |
|
| 77 | - The Linux kernel's "no regression" policy holds in practice only for
|
| 78 | open-source userspace of the DRM subsystem. DRM developers are perfectly fine
|
| 79 | if closed-source blob drivers in userspace use the same uAPI as the open
|
| 80 | drivers, but they must do so in the exact same way as the open drivers.
|
| 81 | Creative (ab)use of the interfaces will, and in the past routinely has, lead
|
| 82 | to breakage.
|
| 83 |
|
| 84 | - Any new userspace interface must have an open-source implementation as
|
| 85 | demonstration vehicle.
|
| 86 |
|
| 87 | The other reason for requiring open-source userspace is uAPI review. Since the
|
| 88 | kernel and userspace parts of a GFX stack must work together so closely, code
|
| 89 | review can only assess whether a new interface achieves its goals by looking at
|
| 90 | both sides. Making sure that the interface indeed covers the use-case fully
|
| 91 | leads to a few additional requirements:
|
| 92 |
|
| 93 | - The open-source userspace must not be a toy/test application, but the real
|
| 94 | thing. Specifically it needs to handle all the usual error and corner cases.
|
| 95 | These are often the places where new uAPI falls apart and hence essential to
|
| 96 | assess the fitness of a proposed interface.
|
| 97 |
|
| 98 | - The userspace side must be fully reviewed and tested to the standards of that
|
| 99 | userspace project. For e.g. mesa this means piglit testcases and review on the
|
| 100 | mailing list. This is again to ensure that the new interface actually gets the
|
| 101 | job done. The userspace-side reviewer should also provide an Acked-by on the
|
| 102 | kernel uAPI patch indicating that they believe the proposed uAPI is sound and
|
| 103 | sufficiently documented and validated for userspace's consumption.
|
| 104 |
|
| 105 | - The userspace patches must be against the canonical upstream, not some vendor
|
| 106 | fork. This is to make sure that no one cheats on the review and testing
|
| 107 | requirements by doing a quick fork.
|
| 108 |
|
| 109 | - The kernel patch can only be merged after all the above requirements are met,
|
| 110 | but it **must** be merged to either drm-next or drm-misc-next **before** the
|
| 111 | userspace patches land. uAPI always flows from the kernel, doing things the
|
| 112 | other way round risks divergence of the uAPI definitions and header files.
|
| 113 |
|
| 114 | These are fairly steep requirements, but have grown out from years of shared
|
| 115 | pain and experience with uAPI added hastily, and almost always regretted about
|
| 116 | just as fast. GFX devices change really fast, requiring a paradigm shift and
|
| 117 | entire new set of uAPI interfaces every few years at least. Together with the
|
| 118 | Linux kernel's guarantee to keep existing userspace running for 10+ years this
|
| 119 | is already rather painful for the DRM subsystem, with multiple different uAPIs
|
| 120 | for the same thing co-existing. If we add a few more complete mistakes into the
|
| 121 | mix every year it would be entirely unmanageable.
|
| 122 |
|
| 123 | The DRM subsystem has however no concern with independent closed-source
|
| 124 | userspace implementations. To officialize that position, the DRM uAPI headers
|
| 125 | are covered by the MIT license.
|
| 126 |
|
| 127 | .. _drm_render_node:
|
| 128 |
|
| 129 | Render nodes
|
| 130 | ============
|
| 131 |
|
| 132 | DRM core provides multiple character-devices for user-space to use.
|
| 133 | Depending on which device is opened, user-space can perform a different
|
| 134 | set of operations (mainly ioctls). The primary node is always created
|
| 135 | and called card<num>. Additionally, a currently unused control node,
|
| 136 | called controlD<num> is also created. The primary node provides all
|
| 137 | legacy operations and historically was the only interface used by
|
| 138 | userspace. With KMS, the control node was introduced. However, the
|
| 139 | planned KMS control interface has never been written and so the control
|
| 140 | node stays unused to date.
|
| 141 |
|
| 142 | With the increased use of offscreen renderers and GPGPU applications,
|
| 143 | clients no longer require running compositors or graphics servers to
|
| 144 | make use of a GPU. But the DRM API required unprivileged clients to
|
| 145 | authenticate to a DRM-Master prior to getting GPU access. To avoid this
|
| 146 | step and to grant clients GPU access without authenticating, render
|
| 147 | nodes were introduced. Render nodes solely serve render clients, that
|
| 148 | is, no modesetting or privileged ioctls can be issued on render nodes.
|
| 149 | Only non-global rendering commands are allowed. If a driver supports
|
| 150 | render nodes, it must advertise it via the DRIVER_RENDER DRM driver
|
| 151 | capability. If not supported, the primary node must be used for render
|
| 152 | clients together with the legacy drmAuth authentication procedure.
|
| 153 |
|
| 154 | If a driver advertises render node support, DRM core will create a
|
| 155 | separate render node called renderD<num>. There will be one render node
|
| 156 | per device. No ioctls except PRIME-related ioctls will be allowed on
|
| 157 | this node. Especially GEM_OPEN will be explicitly prohibited. For a
|
| 158 | complete list of driver-independent ioctls that can be used on render
|
| 159 | nodes, see the ioctls marked DRM_RENDER_ALLOW in drm_ioctl.c Render
|
| 160 | nodes are designed to avoid the buffer-leaks, which occur if clients
|
| 161 | guess the flink names or mmap offsets on the legacy interface.
|
| 162 | Additionally to this basic interface, drivers must mark their
|
| 163 | driver-dependent render-only ioctls as DRM_RENDER_ALLOW so render
|
| 164 | clients can use them. Driver authors must be careful not to allow any
|
| 165 | privileged ioctls on render nodes.
|
| 166 |
|
| 167 | With render nodes, user-space can now control access to the render node
|
| 168 | via basic file-system access-modes. A running graphics server which
|
| 169 | authenticates clients on the privileged primary/legacy node is no longer
|
| 170 | required. Instead, a client can open the render node and is immediately
|
| 171 | granted GPU access. Communication between clients (or servers) is done
|
| 172 | via PRIME. FLINK from render node to legacy node is not supported. New
|
| 173 | clients must not use the insecure FLINK interface.
|
| 174 |
|
| 175 | Besides dropping all modeset/global ioctls, render nodes also drop the
|
| 176 | DRM-Master concept. There is no reason to associate render clients with
|
| 177 | a DRM-Master as they are independent of any graphics server. Besides,
|
| 178 | they must work without any running master, anyway. Drivers must be able
|
| 179 | to run without a master object if they support render nodes. If, on the
|
| 180 | other hand, a driver requires shared state between clients which is
|
| 181 | visible to user-space and accessible beyond open-file boundaries, they
|
| 182 | cannot support render nodes.
|
| 183 |
|
| 184 | Device Hot-Unplug
|
| 185 | =================
|
| 186 |
|
| 187 | .. note::
|
| 188 | The following is the plan. Implementation is not there yet
|
| 189 | (2020 May).
|
| 190 |
|
| 191 | Graphics devices (display and/or render) may be connected via USB (e.g.
|
| 192 | display adapters or docking stations) or Thunderbolt (e.g. eGPU). An end
|
| 193 | user is able to hot-unplug this kind of devices while they are being
|
| 194 | used, and expects that the very least the machine does not crash. Any
|
| 195 | damage from hot-unplugging a DRM device needs to be limited as much as
|
| 196 | possible and userspace must be given the chance to handle it if it wants
|
| 197 | to. Ideally, unplugging a DRM device still lets a desktop continue to
|
| 198 | run, but that is going to need explicit support throughout the whole
|
| 199 | graphics stack: from kernel and userspace drivers, through display
|
| 200 | servers, via window system protocols, and in applications and libraries.
|
| 201 |
|
| 202 | Other scenarios that should lead to the same are: unrecoverable GPU
|
| 203 | crash, PCI device disappearing off the bus, or forced unbind of a driver
|
| 204 | from the physical device.
|
| 205 |
|
| 206 | In other words, from userspace perspective everything needs to keep on
|
| 207 | working more or less, until userspace stops using the disappeared DRM
|
| 208 | device and closes it completely. Userspace will learn of the device
|
| 209 | disappearance from the device removed uevent, ioctls returning ENODEV
|
| 210 | (or driver-specific ioctls returning driver-specific things), or open()
|
| 211 | returning ENXIO.
|
| 212 |
|
| 213 | Only after userspace has closed all relevant DRM device and dmabuf file
|
| 214 | descriptors and removed all mmaps, the DRM driver can tear down its
|
| 215 | instance for the device that no longer exists. If the same physical
|
| 216 | device somehow comes back in the mean time, it shall be a new DRM
|
| 217 | device.
|
| 218 |
|
| 219 | Similar to PIDs, chardev minor numbers are not recycled immediately. A
|
| 220 | new DRM device always picks the next free minor number compared to the
|
| 221 | previous one allocated, and wraps around when minor numbers are
|
| 222 | exhausted.
|
| 223 |
|
| 224 | The goal raises at least the following requirements for the kernel and
|
| 225 | drivers.
|
| 226 |
|
| 227 | Requirements for KMS UAPI
|
| 228 | -------------------------
|
| 229 |
|
| 230 | - KMS connectors must change their status to disconnected.
|
| 231 |
|
| 232 | - Legacy modesets and pageflips, and atomic commits, both real and
|
| 233 | TEST_ONLY, and any other ioctls either fail with ENODEV or fake
|
| 234 | success.
|
| 235 |
|
| 236 | - Pending non-blocking KMS operations deliver the DRM events userspace
|
| 237 | is expecting. This applies also to ioctls that faked success.
|
| 238 |
|
| 239 | - open() on a device node whose underlying device has disappeared will
|
| 240 | fail with ENXIO.
|
| 241 |
|
| 242 | - Attempting to create a DRM lease on a disappeared DRM device will
|
| 243 | fail with ENODEV. Existing DRM leases remain and work as listed
|
| 244 | above.
|
| 245 |
|
| 246 | Requirements for Render and Cross-Device UAPI
|
| 247 | ---------------------------------------------
|
| 248 |
|
| 249 | - All GPU jobs that can no longer run must have their fences
|
| 250 | force-signalled to avoid inflicting hangs on userspace.
|
| 251 | The associated error code is ENODEV.
|
| 252 |
|
| 253 | - Some userspace APIs already define what should happen when the device
|
| 254 | disappears (OpenGL, GL ES: `GL_KHR_robustness`_; `Vulkan`_:
|
| 255 | VK_ERROR_DEVICE_LOST; etc.). DRM drivers are free to implement this
|
| 256 | behaviour the way they see best, e.g. returning failures in
|
| 257 | driver-specific ioctls and handling those in userspace drivers, or
|
| 258 | rely on uevents, and so on.
|
| 259 |
|
| 260 | - dmabuf which point to memory that has disappeared will either fail to
|
| 261 | import with ENODEV or continue to be successfully imported if it would
|
| 262 | have succeeded before the disappearance. See also about memory maps
|
| 263 | below for already imported dmabufs.
|
| 264 |
|
| 265 | - Attempting to import a dmabuf to a disappeared device will either fail
|
| 266 | with ENODEV or succeed if it would have succeeded without the
|
| 267 | disappearance.
|
| 268 |
|
| 269 | - open() on a device node whose underlying device has disappeared will
|
| 270 | fail with ENXIO.
|
| 271 |
|
| 272 | .. _GL_KHR_robustness: https://www.khronos.org/registry/OpenGL/extensions/KHR/KHR_robustness.txt
|
| 273 | .. _Vulkan: https://www.khronos.org/vulkan/
|
| 274 |
|
| 275 | Requirements for Memory Maps
|
| 276 | ----------------------------
|
| 277 |
|
| 278 | Memory maps have further requirements that apply to both existing maps
|
| 279 | and maps created after the device has disappeared. If the underlying
|
| 280 | memory disappears, the map is created or modified such that reads and
|
| 281 | writes will still complete successfully but the result is undefined.
|
| 282 | This applies to both userspace mmap()'d memory and memory pointed to by
|
| 283 | dmabuf which might be mapped to other devices (cross-device dmabuf
|
| 284 | imports).
|
| 285 |
|
| 286 | Raising SIGBUS is not an option, because userspace cannot realistically
|
| 287 | handle it. Signal handlers are global, which makes them extremely
|
| 288 | difficult to use correctly from libraries like those that Mesa produces.
|
| 289 | Signal handlers are not composable, you can't have different handlers
|
| 290 | for GPU1 and GPU2 from different vendors, and a third handler for
|
| 291 | mmapped regular files. Threads cause additional pain with signal
|
| 292 | handling as well.
|
| 293 |
|
| 294 | Device reset
|
| 295 | ============
|
| 296 |
|
| 297 | The GPU stack is really complex and is prone to errors, from hardware bugs,
|
| 298 | faulty applications and everything in between the many layers. Some errors
|
| 299 | require resetting the device in order to make the device usable again. This
|
| 300 | section describes the expectations for DRM and usermode drivers when a
|
| 301 | device resets and how to propagate the reset status.
|
| 302 |
|
| 303 | Device resets can not be disabled without tainting the kernel, which can lead to
|
| 304 | hanging the entire kernel through shrinkers/mmu_notifiers. Userspace role in
|
| 305 | device resets is to propagate the message to the application and apply any
|
| 306 | special policy for blocking guilty applications, if any. Corollary is that
|
| 307 | debugging a hung GPU context require hardware support to be able to preempt such
|
| 308 | a GPU context while it's stopped.
|
| 309 |
|
| 310 | Kernel Mode Driver
|
| 311 | ------------------
|
| 312 |
|
| 313 | The KMD is responsible for checking if the device needs a reset, and to perform
|
| 314 | it as needed. Usually a hang is detected when a job gets stuck executing.
|
| 315 |
|
| 316 | Propagation of errors to userspace has proven to be tricky since it goes in
|
| 317 | the opposite direction of the usual flow of commands. Because of this vendor
|
| 318 | independent error handling was added to the &dma_fence object, this way drivers
|
| 319 | can add an error code to their fences before signaling them. See function
|
| 320 | dma_fence_set_error() on how to do this and for examples of error codes to use.
|
| 321 |
|
| 322 | The DRM scheduler also allows setting error codes on all pending fences when
|
| 323 | hardware submissions are restarted after an reset. Error codes are also
|
| 324 | forwarded from the hardware fence to the scheduler fence to bubble up errors
|
| 325 | to the higher levels of the stack and eventually userspace.
|
| 326 |
|
| 327 | Fence errors can be queried by userspace through the generic SYNC_IOC_FILE_INFO
|
| 328 | IOCTL as well as through driver specific interfaces.
|
| 329 |
|
| 330 | Additional to setting fence errors drivers should also keep track of resets per
|
| 331 | context, the DRM scheduler provides the drm_sched_entity_error() function as
|
| 332 | helper for this use case. After a reset, KMD should reject new command
|
| 333 | submissions for affected contexts.
|
| 334 |
|
| 335 | User Mode Driver
|
| 336 | ----------------
|
| 337 |
|
| 338 | After command submission, UMD should check if the submission was accepted or
|
| 339 | rejected. After a reset, KMD should reject submissions, and UMD can issue an
|
| 340 | ioctl to the KMD to check the reset status, and this can be checked more often
|
| 341 | if the UMD requires it. After detecting a reset, UMD will then proceed to report
|
| 342 | it to the application using the appropriate API error code, as explained in the
|
| 343 | section below about robustness.
|
| 344 |
|
| 345 | Robustness
|
| 346 | ----------
|
| 347 |
|
| 348 | The only way to try to keep a graphical API context working after a reset is if
|
| 349 | it complies with the robustness aspects of the graphical API that it is using.
|
| 350 |
|
| 351 | Graphical APIs provide ways to applications to deal with device resets. However,
|
| 352 | there is no guarantee that the app will use such features correctly, and a
|
| 353 | userspace that doesn't support robust interfaces (like a non-robust
|
| 354 | OpenGL context or API without any robustness support like libva) leave the
|
| 355 | robustness handling entirely to the userspace driver. There is no strong
|
| 356 | community consensus on what the userspace driver should do in that case,
|
| 357 | since all reasonable approaches have some clear downsides.
|
| 358 |
|
| 359 | OpenGL
|
| 360 | ~~~~~~
|
| 361 |
|
| 362 | Apps using OpenGL should use the available robust interfaces, like the
|
| 363 | extension ``GL_ARB_robustness`` (or ``GL_EXT_robustness`` for OpenGL ES). This
|
| 364 | interface tells if a reset has happened, and if so, all the context state is
|
| 365 | considered lost and the app proceeds by creating new ones. There's no consensus
|
| 366 | on what to do to if robustness is not in use.
|
| 367 |
|
| 368 | Vulkan
|
| 369 | ~~~~~~
|
| 370 |
|
| 371 | Apps using Vulkan should check for ``VK_ERROR_DEVICE_LOST`` for submissions.
|
| 372 | This error code means, among other things, that a device reset has happened and
|
| 373 | it needs to recreate the contexts to keep going.
|
| 374 |
|
| 375 | Reporting causes of resets
|
| 376 | --------------------------
|
| 377 |
|
| 378 | Apart from propagating the reset through the stack so apps can recover, it's
|
| 379 | really useful for driver developers to learn more about what caused the reset in
|
| 380 | the first place. For this, drivers can make use of devcoredump to store relevant
|
| 381 | information about the reset and send device wedged event with ``none`` recovery
|
| 382 | method (as explained in "Device Wedging" chapter) to notify userspace, so this
|
| 383 | information can be collected and added to user bug reports.
|
| 384 |
|
| 385 | Device Wedging
|
| 386 | ==============
|
| 387 |
|
| 388 | Drivers can optionally make use of device wedged event (implemented as
|
| 389 | drm_dev_wedged_event() in DRM subsystem), which notifies userspace of 'wedged'
|
| 390 | (hanged/unusable) state of the DRM device through a uevent. This is useful
|
| 391 | especially in cases where the device is no longer operating as expected and has
|
| 392 | become unrecoverable from driver context. Purpose of this implementation is to
|
| 393 | provide drivers a generic way to recover the device with the help of userspace
|
| 394 | intervention, without taking any drastic measures (like resetting or
|
| 395 | re-enumerating the full bus, on which the underlying physical device is sitting)
|
| 396 | in the driver.
|
| 397 |
|
| 398 | A 'wedged' device is basically a device that is declared dead by the driver
|
| 399 | after exhausting all possible attempts to recover it from driver context. The
|
| 400 | uevent is the notification that is sent to userspace along with a hint about
|
| 401 | what could possibly be attempted to recover the device from userspace and bring
|
| 402 | it back to usable state. Different drivers may have different ideas of a
|
| 403 | 'wedged' device depending on hardware implementation of the underlying physical
|
| 404 | device, and hence the vendor agnostic nature of the event. It is up to the
|
| 405 | drivers to decide when they see the need for device recovery and how they want
|
| 406 | to recover from the available methods.
|
| 407 |
|
| 408 | Driver prerequisites
|
| 409 | --------------------
|
| 410 |
|
| 411 | The driver, before opting for recovery, needs to make sure that the 'wedged'
|
| 412 | device doesn't harm the system as a whole by taking care of the prerequisites.
|
| 413 | Necessary actions must include disabling DMA to system memory as well as any
|
| 414 | communication channels with other devices. Further, the driver must ensure
|
| 415 | that all dma_fences are signalled and any device state that the core kernel
|
| 416 | might depend on is cleaned up. All existing mmaps should be invalidated and
|
| 417 | page faults should be redirected to a dummy page. Once the event is sent, the
|
| 418 | device must be kept in 'wedged' state until the recovery is performed. New
|
| 419 | accesses to the device (IOCTLs) should be rejected, preferably with an error
|
| 420 | code that resembles the type of failure the device has encountered. This will
|
| 421 | signify the reason for wedging, which can be reported to the application if
|
| 422 | needed.
|
| 423 |
|
| 424 | Recovery
|
| 425 | --------
|
| 426 |
|
| 427 | Current implementation defines four recovery methods, out of which, drivers
|
| 428 | can use any one, multiple or none. Method(s) of choice will be sent in the
|
| 429 | uevent environment as ``WEDGED=<method1>[,..,<methodN>]`` in order of less to
|
| 430 | more side-effects. See the section `Vendor Specific Recovery`_
|
| 431 | for ``WEDGED=vendor-specific``. If driver is unsure about recovery or
|
| 432 | method is unknown, ``WEDGED=unknown`` will be sent instead.
|
| 433 |
|
| 434 | Userspace consumers can parse this event and attempt recovery as per the
|
| 435 | following expectations.
|
| 436 |
|
| 437 | =============== ========================================
|
| 438 | Recovery method Consumer expectations
|
| 439 | =============== ========================================
|
| 440 | none optional telemetry collection
|
| 441 | rebind unbind + bind driver
|
| 442 | bus-reset unbind + bus reset/re-enumeration + bind
|
| 443 | vendor-specific vendor specific recovery method
|
| 444 | unknown consumer policy
|
| 445 | =============== ========================================
|
| 446 |
|
| 447 | No Recovery
|
| 448 | -----------
|
| 449 |
|
| 450 | Here ``WEDGED=none`` signifies that no recovery is expected from the consumer
|
| 451 | but it can still try to gather telemetry information (devcoredump, syslog) for
|
| 452 | debug purpose in order to root cause the hang. This is useful because the first
|
| 453 | hang is usually the most critical one which can result in consequential hangs
|
| 454 | or complete wedging.
|
| 455 |
|
| 456 | Vendor Specific Recovery
|
| 457 | ------------------------
|
| 458 |
|
| 459 | When ``WEDGED=vendor-specific`` is sent, it indicates that the device requires
|
| 460 | a recovery procedure specific to the hardware vendor and is not one of the
|
| 461 | standardized approaches.
|
| 462 |
|
| 463 | ``WEDGED=vendor-specific`` may be used to indicate different cases within a
|
| 464 | single vendor driver, each requiring a distinct recovery procedure.
|
| 465 | In such scenarios, the vendor driver must provide comprehensive documentation
|
| 466 | that describes each case, include additional hints to identify specific case and
|
| 467 | outline the corresponding recovery procedure. The documentation includes:
|
| 468 |
|
| 469 | Case - A list of all cases that sends the ``WEDGED=vendor-specific`` recovery method.
|
| 470 |
|
| 471 | Hints - Additional Information to assist the userspace consumer in identifying and
|
| 472 | differentiating between different cases. This can be exposed through sysfs, debugfs,
|
| 473 | traces, dmesg etc.
|
| 474 |
|
| 475 | Recovery Procedure - Clear instructions and guidance for recovering each case.
|
| 476 | This may include userspace scripts, tools needed for the recovery procedure.
|
| 477 |
|
| 478 | It is the responsibility of the admin/userspace consumer to identify the case and
|
| 479 | verify additional identification hints before attempting a recovery procedure.
|
| 480 |
|
| 481 | Example: If the device uses the Xe driver, then userspace consumer should refer to
|
| 482 | :ref:`Xe Device Wedging <xe-device-wedging>` for the detailed documentation.
|
| 483 |
|
| 484 | Task information
|
| 485 | ----------------
|
| 486 |
|
| 487 | The information about which application (if any) was involved in the device
|
| 488 | wedging is useful for userspace if they want to notify the user about what
|
| 489 | happened (e.g. the compositor display a message to the user "The <task name>
|
| 490 | caused a graphical error and the system recovered") or to implement policies
|
| 491 | (e.g. the daemon may "ban" an task that keeps resetting the device). If the task
|
| 492 | information is available, the uevent will display as ``PID=<pid>`` and
|
| 493 | ``TASK=<task name>``. Otherwise, ``PID`` and ``TASK`` will not appear in the
|
| 494 | event string.
|
| 495 |
|
| 496 | The reliability of this information is driver and hardware specific, and should
|
| 497 | be taken with a caution regarding it's precision. To have a big picture of what
|
| 498 | really happened, the devcoredump file provides much more detailed information
|
| 499 | about the device state and about the event.
|
| 500 |
|
| 501 | Consumer prerequisites
|
| 502 | ----------------------
|
| 503 |
|
| 504 | It is the responsibility of the consumer to make sure that the device or its
|
| 505 | resources are not in use by any process before attempting recovery. With IOCTLs
|
| 506 | erroring out, all device memory should be unmapped and file descriptors should
|
| 507 | be closed to prevent leaks or undefined behaviour. The idea here is to clear the
|
| 508 | device of all user context beforehand and set the stage for a clean recovery.
|
| 509 |
|
| 510 | For ``WEDGED=vendor-specific`` recovery method, it is the responsibility of the
|
| 511 | consumer to check the driver documentation and the usecase before attempting
|
| 512 | a recovery.
|
| 513 |
|
| 514 | Example - rebind
|
| 515 | ----------------
|
| 516 |
|
| 517 | Udev rule::
|
| 518 |
|
| 519 | SUBSYSTEM=="drm", ENV{WEDGED}=="rebind", DEVPATH=="*/drm/card[0-9]",
|
| 520 | RUN+="/path/to/rebind.sh $env{DEVPATH}"
|
| 521 |
|
| 522 | Recovery script::
|
| 523 |
|
| 524 | #!/bin/sh
|
| 525 |
|
| 526 | DEVPATH=$(readlink -f /sys/$1/device)
|
| 527 | DEVICE=$(basename $DEVPATH)
|
| 528 | DRIVER=$(readlink -f $DEVPATH/driver)
|
| 529 |
|
| 530 | echo -n $DEVICE > $DRIVER/unbind
|
| 531 | echo -n $DEVICE > $DRIVER/bind
|
| 532 |
|
| 533 | Customization
|
| 534 | -------------
|
| 535 |
|
| 536 | Although basic recovery is possible with a simple script, consumers can define
|
| 537 | custom policies around recovery. For example, if the driver supports multiple
|
| 538 | recovery methods, consumers can opt for the suitable one depending on scenarios
|
| 539 | like repeat offences or vendor specific failures. Consumers can also choose to
|
| 540 | have the device available for debugging or telemetry collection and base their
|
| 541 | recovery decision on the findings. This is useful especially when the driver is
|
| 542 | unsure about recovery or method is unknown.
|
| 543 |
|
| 544 | .. _drm_driver_ioctl:
|
| 545 |
|
| 546 | IOCTL Support on Device Nodes
|
| 547 | =============================
|
| 548 |
|
| 549 | .. kernel-doc:: drivers/gpu/drm/drm_ioctl.c
|
| 550 | :doc: driver specific ioctls
|
| 551 |
|
| 552 | Recommended IOCTL Return Values
|
| 553 | -------------------------------
|
| 554 |
|
| 555 | In theory a driver's IOCTL callback is only allowed to return very few error
|
| 556 | codes. In practice it's good to abuse a few more. This section documents common
|
| 557 | practice within the DRM subsystem:
|
| 558 |
|
| 559 | ENOENT:
|
| 560 | Strictly this should only be used when a file doesn't exist e.g. when
|
| 561 | calling the open() syscall. We reuse that to signal any kind of object
|
| 562 | lookup failure, e.g. for unknown GEM buffer object handles, unknown KMS
|
| 563 | object handles and similar cases.
|
| 564 |
|
| 565 | ENOSPC:
|
| 566 | Some drivers use this to differentiate "out of kernel memory" from "out
|
| 567 | of VRAM". Sometimes also applies to other limited gpu resources used for
|
| 568 | rendering (e.g. when you have a special limited compression buffer).
|
| 569 | Sometimes resource allocation/reservation issues in command submission
|
| 570 | IOCTLs are also signalled through EDEADLK.
|
| 571 |
|
| 572 | Simply running out of kernel/system memory is signalled through ENOMEM.
|
| 573 |
|
| 574 | EPERM/EACCES:
|
| 575 | Returned for an operation that is valid, but needs more privileges.
|
| 576 | E.g. root-only or much more common, DRM master-only operations return
|
| 577 | this when called by unprivileged clients. There's no clear
|
| 578 | difference between EACCES and EPERM.
|
| 579 |
|
| 580 | ENODEV:
|
| 581 | The device is not present anymore or is not yet fully initialized.
|
| 582 |
|
| 583 | EOPNOTSUPP:
|
| 584 | Feature (like PRIME, modesetting, GEM) is not supported by the driver.
|
| 585 |
|
| 586 | ENXIO:
|
| 587 | Remote failure, either a hardware transaction (like i2c), but also used
|
| 588 | when the exporting driver of a shared dma-buf or fence doesn't support a
|
| 589 | feature needed.
|
| 590 |
|
| 591 | EINTR:
|
| 592 | DRM drivers assume that userspace restarts all IOCTLs. Any DRM IOCTL can
|
| 593 | return EINTR and in such a case should be restarted with the IOCTL
|
| 594 | parameters left unchanged.
|
| 595 |
|
| 596 | EIO:
|
| 597 | The GPU died and couldn't be resurrected through a reset. Modesetting
|
| 598 | hardware failures are signalled through the "link status" connector
|
| 599 | property.
|
| 600 |
|
| 601 | EINVAL:
|
| 602 | Catch-all for anything that is an invalid argument combination which
|
| 603 | cannot work.
|
| 604 |
|
| 605 | IOCTL also use other error codes like ETIME, EFAULT, EBUSY, ENOTTY but their
|
| 606 | usage is in line with the common meanings. The above list tries to just document
|
| 607 | DRM specific patterns. Note that ENOTTY has the slightly unintuitive meaning of
|
| 608 | "this IOCTL does not exist", and is used exactly as such in DRM.
|
| 609 |
|
| 610 | .. kernel-doc:: include/drm/drm_ioctl.h
|
| 611 | :internal:
|
| 612 |
|
| 613 | .. kernel-doc:: drivers/gpu/drm/drm_ioctl.c
|
| 614 | :export:
|
| 615 |
|
| 616 | .. kernel-doc:: drivers/gpu/drm/drm_ioc32.c
|
| 617 | :export:
|
| 618 |
|
| 619 | Testing and validation
|
| 620 | ======================
|
| 621 |
|
| 622 | Testing Requirements for userspace API
|
| 623 | --------------------------------------
|
| 624 |
|
| 625 | New cross-driver userspace interface extensions, like new IOCTL, new KMS
|
| 626 | properties, new files in sysfs or anything else that constitutes an API change
|
| 627 | should have driver-agnostic testcases in IGT for that feature, if such a test
|
| 628 | can be reasonably made using IGT for the target hardware.
|
| 629 |
|
| 630 | Validating changes with IGT
|
| 631 | ---------------------------
|
| 632 |
|
| 633 | There's a collection of tests that aims to cover the whole functionality of
|
| 634 | DRM drivers and that can be used to check that changes to DRM drivers or the
|
| 635 | core don't regress existing functionality. This test suite is called IGT and
|
| 636 | its code and instructions to build and run can be found in
|
| 637 | https://gitlab.freedesktop.org/drm/igt-gpu-tools/.
|
| 638 |
|
| 639 | Using VKMS to test DRM API
|
| 640 | --------------------------
|
| 641 |
|
| 642 | VKMS is a software-only model of a KMS driver that is useful for testing
|
| 643 | and for running compositors. VKMS aims to enable a virtual display without
|
| 644 | the need for a hardware display capability. These characteristics made VKMS
|
| 645 | a perfect tool for validating the DRM core behavior and also support the
|
| 646 | compositor developer. VKMS makes it possible to test DRM functions in a
|
| 647 | virtual machine without display, simplifying the validation of some of the
|
| 648 | core changes.
|
| 649 |
|
| 650 | To Validate changes in DRM API with VKMS, start setting the kernel: make
|
| 651 | sure to enable VKMS module; compile the kernel with the VKMS enabled and
|
| 652 | install it in the target machine. VKMS can be run in a Virtual Machine
|
| 653 | (QEMU, virtme or similar). It's recommended the use of KVM with the minimum
|
| 654 | of 1GB of RAM and four cores.
|
| 655 |
|
| 656 | It's possible to run the IGT-tests in a VM in two ways:
|
| 657 |
|
| 658 | 1. Use IGT inside a VM
|
| 659 | 2. Use IGT from the host machine and write the results in a shared directory.
|
| 660 |
|
| 661 | Following is an example of using a VM with a shared directory with
|
| 662 | the host machine to run igt-tests. This example uses virtme::
|
| 663 |
|
| 664 | $ virtme-run --rwdir /path/for/shared_dir --kdir=path/for/kernel/directory --mods=auto
|
| 665 |
|
| 666 | Run the igt-tests in the guest machine. This example runs the 'kms_flip'
|
| 667 | tests::
|
| 668 |
|
| 669 | $ /path/for/igt-gpu-tools/scripts/run-tests.sh -p -s -t "kms_flip.*" -v
|
| 670 |
|
| 671 | In this example, instead of building the igt_runner, Piglit is used
|
| 672 | (-p option). It creates an HTML summary of the test results and saves
|
| 673 | them in the folder "igt-gpu-tools/results". It executes only the igt-tests
|
| 674 | matching the -t option.
|
| 675 |
|
| 676 | Display CRC Support
|
| 677 | -------------------
|
| 678 |
|
| 679 | .. kernel-doc:: drivers/gpu/drm/drm_debugfs_crc.c
|
| 680 | :doc: CRC ABI
|
| 681 |
|
| 682 | .. kernel-doc:: drivers/gpu/drm/drm_debugfs_crc.c
|
| 683 | :export:
|
| 684 |
|
| 685 | Debugfs Support
|
| 686 | ---------------
|
| 687 |
|
| 688 | .. kernel-doc:: include/drm/drm_debugfs.h
|
| 689 | :internal:
|
| 690 |
|
| 691 | .. kernel-doc:: drivers/gpu/drm/drm_debugfs.c
|
| 692 | :export:
|
| 693 |
|
| 694 | Sysfs Support
|
| 695 | =============
|
| 696 |
|
| 697 | .. kernel-doc:: drivers/gpu/drm/drm_sysfs.c
|
| 698 | :doc: overview
|
| 699 |
|
| 700 | .. kernel-doc:: drivers/gpu/drm/drm_sysfs.c
|
| 701 | :export:
|
| 702 |
|
| 703 |
|
| 704 | VBlank event handling
|
| 705 | =====================
|
| 706 |
|
| 707 | The DRM core exposes two vertical blank related ioctls:
|
| 708 |
|
| 709 | :c:macro:`DRM_IOCTL_WAIT_VBLANK`
|
| 710 | This takes a struct drm_wait_vblank structure as its argument, and
|
| 711 | it is used to block or request a signal when a specified vblank
|
| 712 | event occurs.
|
| 713 |
|
| 714 | :c:macro:`DRM_IOCTL_MODESET_CTL`
|
| 715 | This was only used for user-mode-settind drivers around modesetting
|
| 716 | changes to allow the kernel to update the vblank interrupt after
|
| 717 | mode setting, since on many devices the vertical blank counter is
|
| 718 | reset to 0 at some point during modeset. Modern drivers should not
|
| 719 | call this any more since with kernel mode setting it is a no-op.
|
| 720 |
|
| 721 | Userspace API Structures
|
| 722 | ========================
|
| 723 |
|
| 724 | .. kernel-doc:: include/uapi/drm/drm_mode.h
|
| 725 | :doc: overview
|
| 726 |
|
| 727 | .. _crtc_index:
|
| 728 |
|
| 729 | CRTC index
|
| 730 | ----------
|
| 731 |
|
| 732 | CRTC's have both an object ID and an index, and they are not the same thing.
|
| 733 | The index is used in cases where a densely packed identifier for a CRTC is
|
| 734 | needed, for instance a bitmask of CRTC's. The member possible_crtcs of struct
|
| 735 | drm_mode_get_plane is an example.
|
| 736 |
|
| 737 | :c:macro:`DRM_IOCTL_MODE_GETRESOURCES` populates a structure with an array of
|
| 738 | CRTC ID's, and the CRTC index is its position in this array.
|
| 739 |
|
| 740 | .. kernel-doc:: include/uapi/drm/drm.h
|
| 741 | :internal:
|
| 742 |
|
| 743 | .. kernel-doc:: include/uapi/drm/drm_mode.h
|
| 744 | :internal:
|
| 745 |
|
| 746 |
|
| 747 | dma-buf interoperability
|
| 748 | ========================
|
| 749 |
|
| 750 | Please see Documentation/userspace-api/dma-buf-alloc-exchange.rst for
|
| 751 | information on how dma-buf is integrated and exposed within DRM.
|
| 752 |
|
| 753 |
|
| 754 | Trace events
|
| 755 | ============
|
| 756 |
|
| 757 | See Documentation/trace/tracepoints.rst for information about using
|
| 758 | Linux Kernel Tracepoints.
|
| 759 | In the DRM subsystem, some events are considered stable uAPI to avoid
|
| 760 | breaking tools (e.g.: GPUVis, umr) relying on them. Stable means that fields
|
| 761 | cannot be removed, nor their formatting updated. Adding new fields is
|
| 762 | possible, under the normal uAPI requirements.
|
| 763 |
|
| 764 | Stable uAPI events
|
| 765 | ------------------
|
| 766 |
|
| 767 | From ``drivers/gpu/drm/scheduler/gpu_scheduler_trace.h``
|
| 768 |
|
| 769 | .. kernel-doc:: drivers/gpu/drm/scheduler/gpu_scheduler_trace.h
|
| 770 | :doc: uAPI trace events
|