| 1 | |
| 2 | .. c:namespace:: V4L
|
| 3 |
|
| 4 | .. _stateless_decoder:
|
| 5 |
|
| 6 | **************************************************
|
| 7 | Memory-to-memory Stateless Video Decoder Interface
|
| 8 | **************************************************
|
| 9 |
|
| 10 | A stateless decoder is a decoder that works without retaining any kind of state
|
| 11 | between processed frames. This means that each frame is decoded independently
|
| 12 | of any previous and future frames, and that the client is responsible for
|
| 13 | maintaining the decoding state and providing it to the decoder with each
|
| 14 | decoding request. This is in contrast to the stateful video decoder interface,
|
| 15 | where the hardware and driver maintain the decoding state and all the client
|
| 16 | has to do is to provide the raw encoded stream and dequeue decoded frames in
|
| 17 | display order.
|
| 18 |
|
| 19 | This section describes how user-space ("the client") is expected to communicate
|
| 20 | with stateless decoders in order to successfully decode an encoded stream.
|
| 21 | Compared to stateful codecs, the decoder/client sequence is simpler, but the
|
| 22 | cost of this simplicity is extra complexity in the client which is responsible
|
| 23 | for maintaining a consistent decoding state.
|
| 24 |
|
| 25 | Stateless decoders make use of the :ref:`media-request-api`. A stateless
|
| 26 | decoder must expose the ``V4L2_BUF_CAP_SUPPORTS_REQUESTS`` capability on its
|
| 27 | ``OUTPUT`` queue when :c:func:`VIDIOC_REQBUFS` or :c:func:`VIDIOC_CREATE_BUFS`
|
| 28 | are invoked.
|
| 29 |
|
| 30 | Depending on the encoded formats supported by the decoder, a single decoded
|
| 31 | frame may be the result of several decode requests (for instance, H.264 streams
|
| 32 | with multiple slices per frame). Decoders that support such formats must also
|
| 33 | expose the ``V4L2_BUF_CAP_SUPPORTS_M2M_HOLD_CAPTURE_BUF`` capability on their
|
| 34 | ``OUTPUT`` queue.
|
| 35 |
|
| 36 | Querying capabilities
|
| 37 | =====================
|
| 38 |
|
| 39 | 1. To enumerate the set of coded formats supported by the decoder, the client
|
| 40 | calls :c:func:`VIDIOC_ENUM_FMT` on the ``OUTPUT`` queue.
|
| 41 |
|
| 42 | * The driver must always return the full set of supported ``OUTPUT`` formats,
|
| 43 | irrespective of the format currently set on the ``CAPTURE`` queue.
|
| 44 |
|
| 45 | * Simultaneously, the driver must restrain the set of values returned by
|
| 46 | codec-specific capability controls (such as H.264 profiles) to the set
|
| 47 | actually supported by the hardware.
|
| 48 |
|
| 49 | 2. To enumerate the set of supported raw formats, the client calls
|
| 50 | :c:func:`VIDIOC_ENUM_FMT` on the ``CAPTURE`` queue.
|
| 51 |
|
| 52 | * The driver must return only the formats supported for the format currently
|
| 53 | active on the ``OUTPUT`` queue.
|
| 54 |
|
| 55 | * Depending on the currently set ``OUTPUT`` format, the set of supported raw
|
| 56 | formats may depend on the value of some codec-dependent controls.
|
| 57 | The client is responsible for making sure that these controls are set
|
| 58 | before querying the ``CAPTURE`` queue. Failure to do so will result in the
|
| 59 | default values for these controls being used, and a returned set of formats
|
| 60 | that may not be usable for the media the client is trying to decode.
|
| 61 |
|
| 62 | 3. The client may use :c:func:`VIDIOC_ENUM_FRAMESIZES` to detect supported
|
| 63 | resolutions for a given format, passing desired pixel format in
|
| 64 | :c:type:`v4l2_frmsizeenum`'s ``pixel_format``.
|
| 65 |
|
| 66 | 4. Supported profiles and levels for the current ``OUTPUT`` format, if
|
| 67 | applicable, may be queried using their respective controls via
|
| 68 | :c:func:`VIDIOC_QUERYCTRL`.
|
| 69 |
|
| 70 | Initialization
|
| 71 | ==============
|
| 72 |
|
| 73 | 1. Set the coded format on the ``OUTPUT`` queue via :c:func:`VIDIOC_S_FMT`.
|
| 74 |
|
| 75 | * **Required fields:**
|
| 76 |
|
| 77 | ``type``
|
| 78 | a ``V4L2_BUF_TYPE_*`` enum appropriate for ``OUTPUT``.
|
| 79 |
|
| 80 | ``pixelformat``
|
| 81 | a coded pixel format.
|
| 82 |
|
| 83 | ``width``, ``height``
|
| 84 | coded width and height parsed from the stream.
|
| 85 |
|
| 86 | other fields
|
| 87 | follow standard semantics.
|
| 88 |
|
| 89 | .. note::
|
| 90 |
|
| 91 | Changing the ``OUTPUT`` format may change the currently set ``CAPTURE``
|
| 92 | format. The driver will derive a new ``CAPTURE`` format from the
|
| 93 | ``OUTPUT`` format being set, including resolution, colorimetry
|
| 94 | parameters, etc. If the client needs a specific ``CAPTURE`` format,
|
| 95 | it must adjust it afterwards.
|
| 96 |
|
| 97 | 2. Call :c:func:`VIDIOC_S_EXT_CTRLS` to set all the controls (parsed headers,
|
| 98 | etc.) required by the ``OUTPUT`` format to enumerate the ``CAPTURE`` formats.
|
| 99 |
|
| 100 | 3. Call :c:func:`VIDIOC_G_FMT` for ``CAPTURE`` queue to get the format for the
|
| 101 | destination buffers parsed/decoded from the bytestream.
|
| 102 |
|
| 103 | * **Required fields:**
|
| 104 |
|
| 105 | ``type``
|
| 106 | a ``V4L2_BUF_TYPE_*`` enum appropriate for ``CAPTURE``.
|
| 107 |
|
| 108 | * **Returned fields:**
|
| 109 |
|
| 110 | ``width``, ``height``
|
| 111 | frame buffer resolution for the decoded frames.
|
| 112 |
|
| 113 | ``pixelformat``
|
| 114 | pixel format for decoded frames.
|
| 115 |
|
| 116 | ``num_planes`` (for _MPLANE ``type`` only)
|
| 117 | number of planes for pixelformat.
|
| 118 |
|
| 119 | ``sizeimage``, ``bytesperline``
|
| 120 | as per standard semantics; matching frame buffer format.
|
| 121 |
|
| 122 | .. note::
|
| 123 |
|
| 124 | The value of ``pixelformat`` may be any pixel format supported for the
|
| 125 | ``OUTPUT`` format, based on the hardware capabilities. It is suggested
|
| 126 | that the driver chooses the preferred/optimal format for the current
|
| 127 | configuration. For example, a YUV format may be preferred over an RGB
|
| 128 | format, if an additional conversion step would be required for RGB.
|
| 129 |
|
| 130 | 4. *[optional]* Enumerate ``CAPTURE`` formats via :c:func:`VIDIOC_ENUM_FMT` on
|
| 131 | the ``CAPTURE`` queue. The client may use this ioctl to discover which
|
| 132 | alternative raw formats are supported for the current ``OUTPUT`` format and
|
| 133 | select one of them via :c:func:`VIDIOC_S_FMT`.
|
| 134 |
|
| 135 | .. note::
|
| 136 |
|
| 137 | The driver will return only formats supported for the currently selected
|
| 138 | ``OUTPUT`` format and currently set controls, even if more formats may be
|
| 139 | supported by the decoder in general.
|
| 140 |
|
| 141 | For example, a decoder may support YUV and RGB formats for
|
| 142 | resolutions 1920x1088 and lower, but only YUV for higher resolutions (due
|
| 143 | to hardware limitations). After setting a resolution of 1920x1088 or lower
|
| 144 | as the ``OUTPUT`` format, :c:func:`VIDIOC_ENUM_FMT` may return a set of
|
| 145 | YUV and RGB pixel formats, but after setting a resolution higher than
|
| 146 | 1920x1088, the driver will not return RGB pixel formats, since they are
|
| 147 | unsupported for this resolution.
|
| 148 |
|
| 149 | 5. *[optional]* Choose a different ``CAPTURE`` format than suggested via
|
| 150 | :c:func:`VIDIOC_S_FMT` on ``CAPTURE`` queue. It is possible for the client to
|
| 151 | choose a different format than selected/suggested by the driver in
|
| 152 | :c:func:`VIDIOC_G_FMT`.
|
| 153 |
|
| 154 | * **Required fields:**
|
| 155 |
|
| 156 | ``type``
|
| 157 | a ``V4L2_BUF_TYPE_*`` enum appropriate for ``CAPTURE``.
|
| 158 |
|
| 159 | ``pixelformat``
|
| 160 | a raw pixel format.
|
| 161 |
|
| 162 | ``width``, ``height``
|
| 163 | frame buffer resolution of the decoded stream; typically unchanged from
|
| 164 | what was returned with :c:func:`VIDIOC_G_FMT`, but it may be different
|
| 165 | if the hardware supports composition and/or scaling.
|
| 166 |
|
| 167 | After performing this step, the client must perform step 3 again in order
|
| 168 | to obtain up-to-date information about the buffers size and layout.
|
| 169 |
|
| 170 | 6. Allocate source (bytestream) buffers via :c:func:`VIDIOC_REQBUFS` on
|
| 171 | ``OUTPUT`` queue.
|
| 172 |
|
| 173 | * **Required fields:**
|
| 174 |
|
| 175 | ``count``
|
| 176 | requested number of buffers to allocate; greater than zero.
|
| 177 |
|
| 178 | ``type``
|
| 179 | a ``V4L2_BUF_TYPE_*`` enum appropriate for ``OUTPUT``.
|
| 180 |
|
| 181 | ``memory``
|
| 182 | follows standard semantics.
|
| 183 |
|
| 184 | * **Returned fields:**
|
| 185 |
|
| 186 | ``count``
|
| 187 | actual number of buffers allocated.
|
| 188 |
|
| 189 | * If required, the driver will adjust ``count`` to be equal or bigger to the
|
| 190 | minimum of required number of ``OUTPUT`` buffers for the given format and
|
| 191 | requested count. The client must check this value after the ioctl returns
|
| 192 | to get the actual number of buffers allocated.
|
| 193 |
|
| 194 | 7. Allocate destination (raw format) buffers via :c:func:`VIDIOC_REQBUFS` on the
|
| 195 | ``CAPTURE`` queue.
|
| 196 |
|
| 197 | * **Required fields:**
|
| 198 |
|
| 199 | ``count``
|
| 200 | requested number of buffers to allocate; greater than zero. The client
|
| 201 | is responsible for deducing the minimum number of buffers required
|
| 202 | for the stream to be properly decoded (taking e.g. reference frames
|
| 203 | into account) and pass an equal or bigger number.
|
| 204 |
|
| 205 | ``type``
|
| 206 | a ``V4L2_BUF_TYPE_*`` enum appropriate for ``CAPTURE``.
|
| 207 |
|
| 208 | ``memory``
|
| 209 | follows standard semantics. ``V4L2_MEMORY_USERPTR`` is not supported
|
| 210 | for ``CAPTURE`` buffers.
|
| 211 |
|
| 212 | * **Returned fields:**
|
| 213 |
|
| 214 | ``count``
|
| 215 | adjusted to allocated number of buffers, in case the codec requires
|
| 216 | more buffers than requested.
|
| 217 |
|
| 218 | * The driver must adjust count to the minimum of required number of
|
| 219 | ``CAPTURE`` buffers for the current format, stream configuration and
|
| 220 | requested count. The client must check this value after the ioctl
|
| 221 | returns to get the number of buffers allocated.
|
| 222 |
|
| 223 | 8. Allocate requests (likely one per ``OUTPUT`` buffer) via
|
| 224 | :c:func:`MEDIA_IOC_REQUEST_ALLOC` on the media device.
|
| 225 |
|
| 226 | 9. Start streaming on both ``OUTPUT`` and ``CAPTURE`` queues via
|
| 227 | :c:func:`VIDIOC_STREAMON`.
|
| 228 |
|
| 229 | Decoding
|
| 230 | ========
|
| 231 |
|
| 232 | For each frame, the client is responsible for submitting at least one request to
|
| 233 | which the following is attached:
|
| 234 |
|
| 235 | * The amount of encoded data expected by the codec for its current
|
| 236 | configuration, as a buffer submitted to the ``OUTPUT`` queue. Typically, this
|
| 237 | corresponds to one frame worth of encoded data, but some formats may allow (or
|
| 238 | require) different amounts per unit.
|
| 239 | * All the metadata needed to decode the submitted encoded data, in the form of
|
| 240 | controls relevant to the format being decoded.
|
| 241 |
|
| 242 | The amount of data and contents of the source ``OUTPUT`` buffer, as well as the
|
| 243 | controls that must be set on the request, depend on the active coded pixel
|
| 244 | format and might be affected by codec-specific extended controls, as stated in
|
| 245 | documentation of each format.
|
| 246 |
|
| 247 | If there is a possibility that the decoded frame will require one or more
|
| 248 | decode requests after the current one in order to be produced, then the client
|
| 249 | must set the ``V4L2_BUF_FLAG_M2M_HOLD_CAPTURE_BUF`` flag on the ``OUTPUT``
|
| 250 | buffer. This will result in the (potentially partially) decoded ``CAPTURE``
|
| 251 | buffer not being made available for dequeueing, and reused for the next decode
|
| 252 | request if the timestamp of the next ``OUTPUT`` buffer has not changed.
|
| 253 |
|
| 254 | A typical frame would thus be decoded using the following sequence:
|
| 255 |
|
| 256 | 1. Queue an ``OUTPUT`` buffer containing one unit of encoded bytestream data for
|
| 257 | the decoding request, using :c:func:`VIDIOC_QBUF`.
|
| 258 |
|
| 259 | * **Required fields:**
|
| 260 |
|
| 261 | ``index``
|
| 262 | index of the buffer being queued.
|
| 263 |
|
| 264 | ``type``
|
| 265 | type of the buffer.
|
| 266 |
|
| 267 | ``bytesused``
|
| 268 | number of bytes taken by the encoded data frame in the buffer.
|
| 269 |
|
| 270 | ``flags``
|
| 271 | the ``V4L2_BUF_FLAG_REQUEST_FD`` flag must be set. Additionally, if
|
| 272 | we are not sure that the current decode request is the last one needed
|
| 273 | to produce a fully decoded frame, then
|
| 274 | ``V4L2_BUF_FLAG_M2M_HOLD_CAPTURE_BUF`` must also be set.
|
| 275 |
|
| 276 | ``request_fd``
|
| 277 | must be set to the file descriptor of the decoding request.
|
| 278 |
|
| 279 | ``timestamp``
|
| 280 | must be set to a unique value per frame. This value will be propagated
|
| 281 | into the decoded frame's buffer and can also be used to use this frame
|
| 282 | as the reference of another. If using multiple decode requests per
|
| 283 | frame, then the timestamps of all the ``OUTPUT`` buffers for a given
|
| 284 | frame must be identical. If the timestamp changes, then the currently
|
| 285 | held ``CAPTURE`` buffer will be made available for dequeuing and the
|
| 286 | current request will work on a new ``CAPTURE`` buffer.
|
| 287 |
|
| 288 | 2. Set the codec-specific controls for the decoding request, using
|
| 289 | :c:func:`VIDIOC_S_EXT_CTRLS`.
|
| 290 |
|
| 291 | * **Required fields:**
|
| 292 |
|
| 293 | ``which``
|
| 294 | must be ``V4L2_CTRL_WHICH_REQUEST_VAL``.
|
| 295 |
|
| 296 | ``request_fd``
|
| 297 | must be set to the file descriptor of the decoding request.
|
| 298 |
|
| 299 | other fields
|
| 300 | other fields are set as usual when setting controls. The ``controls``
|
| 301 | array must contain all the codec-specific controls required to decode
|
| 302 | a frame.
|
| 303 |
|
| 304 | .. note::
|
| 305 |
|
| 306 | It is possible to specify the controls in different invocations of
|
| 307 | :c:func:`VIDIOC_S_EXT_CTRLS`, or to overwrite a previously set control, as
|
| 308 | long as ``request_fd`` and ``which`` are properly set. The controls state
|
| 309 | at the moment of request submission is the one that will be considered.
|
| 310 |
|
| 311 | .. note::
|
| 312 |
|
| 313 | The order in which steps 1 and 2 take place is interchangeable.
|
| 314 |
|
| 315 | 3. Submit the request by invoking :c:func:`MEDIA_REQUEST_IOC_QUEUE` on the
|
| 316 | request FD.
|
| 317 |
|
| 318 | If the request is submitted without an ``OUTPUT`` buffer, or if some of the
|
| 319 | required controls are missing from the request, then
|
| 320 | :c:func:`MEDIA_REQUEST_IOC_QUEUE` will return ``-ENOENT``. If more than one
|
| 321 | ``OUTPUT`` buffer is queued, then it will return ``-EINVAL``.
|
| 322 | :c:func:`MEDIA_REQUEST_IOC_QUEUE` returning non-zero means that no
|
| 323 | ``CAPTURE`` buffer will be produced for this request.
|
| 324 |
|
| 325 | ``CAPTURE`` buffers must not be part of the request, and are queued
|
| 326 | independently. They are returned in decode order (i.e. the same order as coded
|
| 327 | frames were submitted to the ``OUTPUT`` queue).
|
| 328 |
|
| 329 | Runtime decoding errors are signaled by the dequeued ``CAPTURE`` buffers
|
| 330 | carrying the ``V4L2_BUF_FLAG_ERROR`` flag. If a decoded reference frame has an
|
| 331 | error, then all following decoded frames that refer to it also have the
|
| 332 | ``V4L2_BUF_FLAG_ERROR`` flag set, although the decoder will still try to
|
| 333 | produce (likely corrupted) frames.
|
| 334 |
|
| 335 | Buffer management while decoding
|
| 336 | ================================
|
| 337 | Contrary to stateful decoders, a stateless decoder does not perform any kind of
|
| 338 | buffer management: it only guarantees that dequeued ``CAPTURE`` buffers can be
|
| 339 | used by the client for as long as they are not queued again. "Used" here
|
| 340 | encompasses using the buffer for compositing or display.
|
| 341 |
|
| 342 | A dequeued capture buffer can also be used as the reference frame of another
|
| 343 | buffer.
|
| 344 |
|
| 345 | A frame is specified as reference by converting its timestamp into nanoseconds,
|
| 346 | and storing it into the relevant member of a codec-dependent control structure.
|
| 347 | The :c:func:`v4l2_timeval_to_ns` function must be used to perform that
|
| 348 | conversion. The timestamp of a frame can be used to reference it as soon as all
|
| 349 | its units of encoded data are successfully submitted to the ``OUTPUT`` queue.
|
| 350 |
|
| 351 | A decoded buffer containing a reference frame must not be reused as a decoding
|
| 352 | target until all the frames referencing it have been decoded. The safest way to
|
| 353 | achieve this is to refrain from queueing a reference buffer until all the
|
| 354 | decoded frames referencing it have been dequeued. However, if the driver can
|
| 355 | guarantee that buffers queued to the ``CAPTURE`` queue are processed in queued
|
| 356 | order, then user-space can take advantage of this guarantee and queue a
|
| 357 | reference buffer when the following conditions are met:
|
| 358 |
|
| 359 | 1. All the requests for frames affected by the reference frame have been
|
| 360 | queued, and
|
| 361 |
|
| 362 | 2. A sufficient number of ``CAPTURE`` buffers to cover all the decoded
|
| 363 | referencing frames have been queued.
|
| 364 |
|
| 365 | When queuing a decoding request, the driver will increase the reference count of
|
| 366 | all the resources associated with reference frames. This means that the client
|
| 367 | can e.g. close the DMABUF file descriptors of reference frame buffers if it
|
| 368 | won't need them afterwards.
|
| 369 |
|
| 370 | Seeking
|
| 371 | =======
|
| 372 | In order to seek, the client just needs to submit requests using input buffers
|
| 373 | corresponding to the new stream position. It must however be aware that
|
| 374 | resolution may have changed and follow the dynamic resolution change sequence in
|
| 375 | that case. Also depending on the codec used, picture parameters (e.g. SPS/PPS
|
| 376 | for H.264) may have changed and the client is responsible for making sure that a
|
| 377 | valid state is sent to the decoder.
|
| 378 |
|
| 379 | The client is then free to ignore any returned ``CAPTURE`` buffer that comes
|
| 380 | from the pre-seek position.
|
| 381 |
|
| 382 | Pausing
|
| 383 | =======
|
| 384 |
|
| 385 | In order to pause, the client can just cease queuing buffers onto the ``OUTPUT``
|
| 386 | queue. Without source bytestream data, there is no data to process and the codec
|
| 387 | will remain idle.
|
| 388 |
|
| 389 | Dynamic resolution change
|
| 390 | =========================
|
| 391 |
|
| 392 | If the client detects a resolution change in the stream, it will need to perform
|
| 393 | the initialization sequence again with the new resolution:
|
| 394 |
|
| 395 | 1. If the last submitted request resulted in a ``CAPTURE`` buffer being
|
| 396 | held by the use of the ``V4L2_BUF_FLAG_M2M_HOLD_CAPTURE_BUF`` flag, then the
|
| 397 | last frame is not available on the ``CAPTURE`` queue. In this case, a
|
| 398 | ``V4L2_DEC_CMD_FLUSH`` command shall be sent. This will make the driver
|
| 399 | dequeue the held ``CAPTURE`` buffer.
|
| 400 |
|
| 401 | 2. Wait until all submitted requests have completed and dequeue the
|
| 402 | corresponding output buffers.
|
| 403 |
|
| 404 | 3. Call :c:func:`VIDIOC_STREAMOFF` on both the ``OUTPUT`` and ``CAPTURE``
|
| 405 | queues.
|
| 406 |
|
| 407 | 4. Free all ``CAPTURE`` buffers by calling :c:func:`VIDIOC_REQBUFS` on the
|
| 408 | ``CAPTURE`` queue with a buffer count of zero.
|
| 409 |
|
| 410 | 5. Perform the initialization sequence again (minus the allocation of
|
| 411 | ``OUTPUT`` buffers), with the new resolution set on the ``OUTPUT`` queue.
|
| 412 | Note that due to resolution constraints, a different format may need to be
|
| 413 | picked on the ``CAPTURE`` queue.
|
| 414 |
|
| 415 | Drain
|
| 416 | =====
|
| 417 |
|
| 418 | If the last submitted request resulted in a ``CAPTURE`` buffer being
|
| 419 | held by the use of the ``V4L2_BUF_FLAG_M2M_HOLD_CAPTURE_BUF`` flag, then the
|
| 420 | last frame is not available on the ``CAPTURE`` queue. In this case, a
|
| 421 | ``V4L2_DEC_CMD_FLUSH`` command shall be sent. This will make the driver
|
| 422 | dequeue the held ``CAPTURE`` buffer.
|
| 423 |
|
| 424 | After that, in order to drain the stream on a stateless decoder, the client
|
| 425 | just needs to wait until all the submitted requests are completed.
|