| 1 | |
| 2 | .. c:namespace:: V4L
|
| 3 |
|
| 4 | .. _decoder:
|
| 5 |
|
| 6 | *************************************************
|
| 7 | Memory-to-Memory Stateful Video Decoder Interface
|
| 8 | *************************************************
|
| 9 |
|
| 10 | A stateful video decoder takes complete chunks of the bytestream (e.g. Annex-B
|
| 11 | H.264/HEVC stream, raw VP8/9 stream) and decodes them into raw video frames in
|
| 12 | display order. The decoder is expected not to require any additional information
|
| 13 | from the client to process these buffers.
|
| 14 |
|
| 15 | Performing software parsing, processing etc. of the stream in the driver in
|
| 16 | order to support this interface is strongly discouraged. In case such
|
| 17 | operations are needed, use of the Stateless Video Decoder Interface (in
|
| 18 | development) is strongly advised.
|
| 19 |
|
| 20 | Conventions and Notations Used in This Document
|
| 21 | ===============================================
|
| 22 |
|
| 23 | 1. The general V4L2 API rules apply if not specified in this document
|
| 24 | otherwise.
|
| 25 |
|
| 26 | 2. The meaning of words "must", "may", "should", etc. is as per `RFC
|
| 27 | 2119 <https://tools.ietf.org/html/rfc2119>`_.
|
| 28 |
|
| 29 | 3. All steps not marked "optional" are required.
|
| 30 |
|
| 31 | 4. :c:func:`VIDIOC_G_EXT_CTRLS` and :c:func:`VIDIOC_S_EXT_CTRLS` may be used
|
| 32 | interchangeably with :c:func:`VIDIOC_G_CTRL` and :c:func:`VIDIOC_S_CTRL`,
|
| 33 | unless specified otherwise.
|
| 34 |
|
| 35 | 5. Single-planar API (see :ref:`planar-apis`) and applicable structures may be
|
| 36 | used interchangeably with multi-planar API, unless specified otherwise,
|
| 37 | depending on decoder capabilities and following the general V4L2 guidelines.
|
| 38 |
|
| 39 | 6. i = [a..b]: sequence of integers from a to b, inclusive, i.e. i =
|
| 40 | [0..2]: i = 0, 1, 2.
|
| 41 |
|
| 42 | 7. Given an ``OUTPUT`` buffer A, then A' represents a buffer on the ``CAPTURE``
|
| 43 | queue containing data that resulted from processing buffer A.
|
| 44 |
|
| 45 | |
| 46 |
|
| 47 | Glossary
|
| 48 | ========
|
| 49 |
|
| 50 | CAPTURE
|
| 51 | the destination buffer queue; for decoders, the queue of buffers containing
|
| 52 | decoded frames; for encoders, the queue of buffers containing an encoded
|
| 53 | bytestream; ``V4L2_BUF_TYPE_VIDEO_CAPTURE`` or
|
| 54 | ``V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE``; data is captured from the hardware
|
| 55 | into ``CAPTURE`` buffers.
|
| 56 |
|
| 57 | client
|
| 58 | the application communicating with the decoder or encoder implementing
|
| 59 | this interface.
|
| 60 |
|
| 61 | coded format
|
| 62 | encoded/compressed video bytestream format (e.g. H.264, VP8, etc.); see
|
| 63 | also: raw format.
|
| 64 |
|
| 65 | coded height
|
| 66 | height for given coded resolution.
|
| 67 |
|
| 68 | coded resolution
|
| 69 | stream resolution in pixels aligned to codec and hardware requirements;
|
| 70 | typically visible resolution rounded up to full macroblocks;
|
| 71 | see also: visible resolution.
|
| 72 |
|
| 73 | coded width
|
| 74 | width for given coded resolution.
|
| 75 |
|
| 76 | coding tree unit
|
| 77 | processing unit of the HEVC codec (corresponds to macroblock units in
|
| 78 | H.264, VP8, VP9),
|
| 79 | can use block structures of up to 64×64 pixels.
|
| 80 | Good at sub-partitioning the picture into variable sized structures.
|
| 81 |
|
| 82 | decode order
|
| 83 | the order in which frames are decoded; may differ from display order if the
|
| 84 | coded format includes a feature of frame reordering; for decoders,
|
| 85 | ``OUTPUT`` buffers must be queued by the client in decode order; for
|
| 86 | encoders ``CAPTURE`` buffers must be returned by the encoder in decode order.
|
| 87 |
|
| 88 | destination
|
| 89 | data resulting from the decode process; see ``CAPTURE``.
|
| 90 |
|
| 91 | display order
|
| 92 | the order in which frames must be displayed; for encoders, ``OUTPUT``
|
| 93 | buffers must be queued by the client in display order; for decoders,
|
| 94 | ``CAPTURE`` buffers must be returned by the decoder in display order.
|
| 95 |
|
| 96 | DPB
|
| 97 | Decoded Picture Buffer; an H.264/HEVC term for a buffer that stores a decoded
|
| 98 | raw frame available for reference in further decoding steps.
|
| 99 |
|
| 100 | EOS
|
| 101 | end of stream.
|
| 102 |
|
| 103 | IDR
|
| 104 | Instantaneous Decoder Refresh; a type of a keyframe in an H.264/HEVC-encoded
|
| 105 | stream, which clears the list of earlier reference frames (DPBs).
|
| 106 |
|
| 107 | keyframe
|
| 108 | an encoded frame that does not reference frames decoded earlier, i.e.
|
| 109 | can be decoded fully on its own.
|
| 110 |
|
| 111 | macroblock
|
| 112 | a processing unit in image and video compression formats based on linear
|
| 113 | block transforms (e.g. H.264, VP8, VP9); codec-specific, but for most of
|
| 114 | popular codecs the size is 16x16 samples (pixels). The HEVC codec uses a
|
| 115 | slightly more flexible processing unit called coding tree unit (CTU).
|
| 116 |
|
| 117 | OUTPUT
|
| 118 | the source buffer queue; for decoders, the queue of buffers containing
|
| 119 | an encoded bytestream; for encoders, the queue of buffers containing raw
|
| 120 | frames; ``V4L2_BUF_TYPE_VIDEO_OUTPUT`` or
|
| 121 | ``V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE``; the hardware is fed with data
|
| 122 | from ``OUTPUT`` buffers.
|
| 123 |
|
| 124 | PPS
|
| 125 | Picture Parameter Set; a type of metadata entity in an H.264/HEVC bytestream.
|
| 126 |
|
| 127 | raw format
|
| 128 | uncompressed format containing raw pixel data (e.g. YUV, RGB formats).
|
| 129 |
|
| 130 | resume point
|
| 131 | a point in the bytestream from which decoding may start/continue, without
|
| 132 | any previous state/data present, e.g.: a keyframe (VP8/VP9) or
|
| 133 | SPS/PPS/IDR sequence (H.264/HEVC); a resume point is required to start decode
|
| 134 | of a new stream, or to resume decoding after a seek.
|
| 135 |
|
| 136 | source
|
| 137 | data fed to the decoder or encoder; see ``OUTPUT``.
|
| 138 |
|
| 139 | source height
|
| 140 | height in pixels for given source resolution; relevant to encoders only.
|
| 141 |
|
| 142 | source resolution
|
| 143 | resolution in pixels of source frames being source to the encoder and
|
| 144 | subject to further cropping to the bounds of visible resolution; relevant to
|
| 145 | encoders only.
|
| 146 |
|
| 147 | source width
|
| 148 | width in pixels for given source resolution; relevant to encoders only.
|
| 149 |
|
| 150 | SPS
|
| 151 | Sequence Parameter Set; a type of metadata entity in an H.264/HEVC bytestream.
|
| 152 |
|
| 153 | stream metadata
|
| 154 | additional (non-visual) information contained inside encoded bytestream;
|
| 155 | for example: coded resolution, visible resolution, codec profile.
|
| 156 |
|
| 157 | visible height
|
| 158 | height for given visible resolution; display height.
|
| 159 |
|
| 160 | visible resolution
|
| 161 | stream resolution of the visible picture, in pixels, to be used for
|
| 162 | display purposes; must be smaller or equal to coded resolution;
|
| 163 | display resolution.
|
| 164 |
|
| 165 | visible width
|
| 166 | width for given visible resolution; display width.
|
| 167 |
|
| 168 | State Machine
|
| 169 | =============
|
| 170 |
|
| 171 | .. kernel-render:: DOT
|
| 172 | :alt: DOT digraph of decoder state machine
|
| 173 | :caption: Decoder State Machine
|
| 174 |
|
| 175 | digraph decoder_state_machine {
|
| 176 | node [shape = doublecircle, label="Decoding"] Decoding;
|
| 177 |
|
| 178 | node [shape = circle, label="Initialization"] Initialization;
|
| 179 | node [shape = circle, label="Capture\nsetup"] CaptureSetup;
|
| 180 | node [shape = circle, label="Dynamic\nResolution\nChange"] ResChange;
|
| 181 | node [shape = circle, label="Stopped"] Stopped;
|
| 182 | node [shape = circle, label="Drain"] Drain;
|
| 183 | node [shape = circle, label="Seek"] Seek;
|
| 184 | node [shape = circle, label="End of Stream"] EoS;
|
| 185 |
|
| 186 | node [shape = point]; qi
|
| 187 | qi -> Initialization [ label = "open()" ];
|
| 188 |
|
| 189 | Initialization -> CaptureSetup [ label = "CAPTURE\nformat\nestablished" ];
|
| 190 |
|
| 191 | CaptureSetup -> Stopped [ label = "CAPTURE\nbuffers\nready" ];
|
| 192 |
|
| 193 | Decoding -> ResChange [ label = "Stream\nresolution\nchange" ];
|
| 194 | Decoding -> Drain [ label = "V4L2_DEC_CMD_STOP" ];
|
| 195 | Decoding -> EoS [ label = "EoS mark\nin the stream" ];
|
| 196 | Decoding -> Seek [ label = "VIDIOC_STREAMOFF(OUTPUT)" ];
|
| 197 | Decoding -> Stopped [ label = "VIDIOC_STREAMOFF(CAPTURE)" ];
|
| 198 | Decoding -> Decoding;
|
| 199 |
|
| 200 | ResChange -> CaptureSetup [ label = "CAPTURE\nformat\nestablished" ];
|
| 201 | ResChange -> Seek [ label = "VIDIOC_STREAMOFF(OUTPUT)" ];
|
| 202 |
|
| 203 | EoS -> Drain [ label = "Implicit\ndrain" ];
|
| 204 |
|
| 205 | Drain -> Stopped [ label = "All CAPTURE\nbuffers dequeued\nor\nVIDIOC_STREAMOFF(CAPTURE)" ];
|
| 206 | Drain -> Seek [ label = "VIDIOC_STREAMOFF(OUTPUT)" ];
|
| 207 |
|
| 208 | Seek -> Decoding [ label = "VIDIOC_STREAMON(OUTPUT)" ];
|
| 209 | Seek -> Initialization [ label = "VIDIOC_REQBUFS(OUTPUT, 0)" ];
|
| 210 |
|
| 211 | Stopped -> Decoding [ label = "V4L2_DEC_CMD_START\nor\nVIDIOC_STREAMON(CAPTURE)" ];
|
| 212 | Stopped -> Seek [ label = "VIDIOC_STREAMOFF(OUTPUT)" ];
|
| 213 | }
|
| 214 |
|
| 215 | Querying Capabilities
|
| 216 | =====================
|
| 217 |
|
| 218 | 1. To enumerate the set of coded formats supported by the decoder, the
|
| 219 | client may call :c:func:`VIDIOC_ENUM_FMT` on ``OUTPUT``.
|
| 220 |
|
| 221 | * The full set of supported formats will be returned, regardless of the
|
| 222 | format set on ``CAPTURE``.
|
| 223 | * Check the flags field of :c:type:`v4l2_fmtdesc` for more information
|
| 224 | about the decoder's capabilities with respect to each coded format.
|
| 225 | In particular whether or not the decoder has a full-fledged bytestream
|
| 226 | parser and if the decoder supports dynamic resolution changes.
|
| 227 |
|
| 228 | 2. To enumerate the set of supported raw formats, the client may call
|
| 229 | :c:func:`VIDIOC_ENUM_FMT` on ``CAPTURE``.
|
| 230 |
|
| 231 | * Only the formats supported for the format currently active on ``OUTPUT``
|
| 232 | will be returned.
|
| 233 |
|
| 234 | * In order to enumerate raw formats supported by a given coded format,
|
| 235 | the client must first set that coded format on ``OUTPUT`` and then
|
| 236 | enumerate formats on ``CAPTURE``.
|
| 237 |
|
| 238 | 3. The client may use :c:func:`VIDIOC_ENUM_FRAMESIZES` to detect supported
|
| 239 | resolutions for a given format, passing desired pixel format in
|
| 240 | :c:type:`v4l2_frmsizeenum` ``pixel_format``.
|
| 241 |
|
| 242 | * Values returned by :c:func:`VIDIOC_ENUM_FRAMESIZES` for a coded pixel
|
| 243 | format will include all possible coded resolutions supported by the
|
| 244 | decoder for given coded pixel format.
|
| 245 |
|
| 246 | * Values returned by :c:func:`VIDIOC_ENUM_FRAMESIZES` for a raw pixel format
|
| 247 | will include all possible frame buffer resolutions supported by the
|
| 248 | decoder for given raw pixel format and the coded format currently set on
|
| 249 | ``OUTPUT``.
|
| 250 |
|
| 251 | 4. Supported profiles and levels for the coded format currently set on
|
| 252 | ``OUTPUT``, if applicable, may be queried using their respective controls
|
| 253 | via :c:func:`VIDIOC_QUERYCTRL`.
|
| 254 |
|
| 255 | Initialization
|
| 256 | ==============
|
| 257 |
|
| 258 | 1. Set the coded format on ``OUTPUT`` via :c:func:`VIDIOC_S_FMT`.
|
| 259 |
|
| 260 | * **Required fields:**
|
| 261 |
|
| 262 | ``type``
|
| 263 | a ``V4L2_BUF_TYPE_*`` enum appropriate for ``OUTPUT``.
|
| 264 |
|
| 265 | ``pixelformat``
|
| 266 | a coded pixel format.
|
| 267 |
|
| 268 | ``width``, ``height``
|
| 269 | coded resolution of the stream; required only if it cannot be parsed
|
| 270 | from the stream for the given coded format; otherwise the decoder will
|
| 271 | use this resolution as a placeholder resolution that will likely change
|
| 272 | as soon as it can parse the actual coded resolution from the stream.
|
| 273 |
|
| 274 | ``sizeimage``
|
| 275 | desired size of ``OUTPUT`` buffers; the decoder may adjust it to
|
| 276 | match hardware requirements.
|
| 277 |
|
| 278 | other fields
|
| 279 | follow standard semantics.
|
| 280 |
|
| 281 | * **Returned fields:**
|
| 282 |
|
| 283 | ``sizeimage``
|
| 284 | adjusted size of ``OUTPUT`` buffers.
|
| 285 |
|
| 286 | * The ``CAPTURE`` format will be updated with an appropriate frame buffer
|
| 287 | resolution instantly based on the width and height returned by
|
| 288 | :c:func:`VIDIOC_S_FMT`.
|
| 289 | However, for coded formats that include stream resolution information,
|
| 290 | after the decoder is done parsing the information from the stream, it will
|
| 291 | update the ``CAPTURE`` format with new values and signal a source change
|
| 292 | event, regardless of whether they match the values set by the client or
|
| 293 | not.
|
| 294 |
|
| 295 | .. important::
|
| 296 |
|
| 297 | Changing the ``OUTPUT`` format may change the currently set ``CAPTURE``
|
| 298 | format. How the new ``CAPTURE`` format is determined is up to the decoder
|
| 299 | and the client must ensure it matches its needs afterwards.
|
| 300 |
|
| 301 | 2. Allocate source (bytestream) buffers via :c:func:`VIDIOC_REQBUFS` on
|
| 302 | ``OUTPUT``.
|
| 303 |
|
| 304 | * **Required fields:**
|
| 305 |
|
| 306 | ``count``
|
| 307 | requested number of buffers to allocate; greater than zero.
|
| 308 |
|
| 309 | ``type``
|
| 310 | a ``V4L2_BUF_TYPE_*`` enum appropriate for ``OUTPUT``.
|
| 311 |
|
| 312 | ``memory``
|
| 313 | follows standard semantics.
|
| 314 |
|
| 315 | * **Returned fields:**
|
| 316 |
|
| 317 | ``count``
|
| 318 | the actual number of buffers allocated.
|
| 319 |
|
| 320 | .. warning::
|
| 321 |
|
| 322 | The actual number of allocated buffers may differ from the ``count``
|
| 323 | given. The client must check the updated value of ``count`` after the
|
| 324 | call returns.
|
| 325 |
|
| 326 | Alternatively, :c:func:`VIDIOC_CREATE_BUFS` on the ``OUTPUT`` queue can be
|
| 327 | used to have more control over buffer allocation.
|
| 328 |
|
| 329 | * **Required fields:**
|
| 330 |
|
| 331 | ``count``
|
| 332 | requested number of buffers to allocate; greater than zero.
|
| 333 |
|
| 334 | ``type``
|
| 335 | a ``V4L2_BUF_TYPE_*`` enum appropriate for ``OUTPUT``.
|
| 336 |
|
| 337 | ``memory``
|
| 338 | follows standard semantics.
|
| 339 |
|
| 340 | ``format``
|
| 341 | follows standard semantics.
|
| 342 |
|
| 343 | * **Returned fields:**
|
| 344 |
|
| 345 | ``count``
|
| 346 | adjusted to the number of allocated buffers.
|
| 347 |
|
| 348 | .. warning::
|
| 349 |
|
| 350 | The actual number of allocated buffers may differ from the ``count``
|
| 351 | given. The client must check the updated value of ``count`` after the
|
| 352 | call returns.
|
| 353 |
|
| 354 | 3. Start streaming on the ``OUTPUT`` queue via :c:func:`VIDIOC_STREAMON`.
|
| 355 |
|
| 356 | 4. **This step only applies to coded formats that contain resolution information
|
| 357 | in the stream.** Continue queuing/dequeuing bytestream buffers to/from the
|
| 358 | ``OUTPUT`` queue via :c:func:`VIDIOC_QBUF` and :c:func:`VIDIOC_DQBUF`. The
|
| 359 | buffers will be processed and returned to the client in order, until
|
| 360 | required metadata to configure the ``CAPTURE`` queue are found. This is
|
| 361 | indicated by the decoder sending a ``V4L2_EVENT_SOURCE_CHANGE`` event with
|
| 362 | ``changes`` set to ``V4L2_EVENT_SRC_CH_RESOLUTION``.
|
| 363 |
|
| 364 | * It is not an error if the first buffer does not contain enough data for
|
| 365 | this to occur. Processing of the buffers will continue as long as more
|
| 366 | data is needed.
|
| 367 |
|
| 368 | * If data in a buffer that triggers the event is required to decode the
|
| 369 | first frame, it will not be returned to the client, until the
|
| 370 | initialization sequence completes and the frame is decoded.
|
| 371 |
|
| 372 | * If the client has not set the coded resolution of the stream on its own,
|
| 373 | calling :c:func:`VIDIOC_G_FMT`, :c:func:`VIDIOC_S_FMT`,
|
| 374 | :c:func:`VIDIOC_TRY_FMT` or :c:func:`VIDIOC_REQBUFS` on the ``CAPTURE``
|
| 375 | queue will not return the real values for the stream until a
|
| 376 | ``V4L2_EVENT_SOURCE_CHANGE`` event with ``changes`` set to
|
| 377 | ``V4L2_EVENT_SRC_CH_RESOLUTION`` is signaled.
|
| 378 |
|
| 379 | .. important::
|
| 380 |
|
| 381 | Any client query issued after the decoder queues the event will return
|
| 382 | values applying to the just parsed stream, including queue formats,
|
| 383 | selection rectangles and controls.
|
| 384 |
|
| 385 | .. note::
|
| 386 |
|
| 387 | A client capable of acquiring stream parameters from the bytestream on
|
| 388 | its own may attempt to set the width and height of the ``OUTPUT`` format
|
| 389 | to non-zero values matching the coded size of the stream, skip this step
|
| 390 | and continue with the `Capture Setup` sequence. However, it must not
|
| 391 | rely on any driver queries regarding stream parameters, such as
|
| 392 | selection rectangles and controls, since the decoder has not parsed them
|
| 393 | from the stream yet. If the values configured by the client do not match
|
| 394 | those parsed by the decoder, a `Dynamic Resolution Change` will be
|
| 395 | triggered to reconfigure them.
|
| 396 |
|
| 397 | .. note::
|
| 398 |
|
| 399 | No decoded frames are produced during this phase.
|
| 400 |
|
| 401 | 5. Continue with the `Capture Setup` sequence.
|
| 402 |
|
| 403 | Capture Setup
|
| 404 | =============
|
| 405 |
|
| 406 | 1. Call :c:func:`VIDIOC_G_FMT` on the ``CAPTURE`` queue to get format for the
|
| 407 | destination buffers parsed/decoded from the bytestream.
|
| 408 |
|
| 409 | * **Required fields:**
|
| 410 |
|
| 411 | ``type``
|
| 412 | a ``V4L2_BUF_TYPE_*`` enum appropriate for ``CAPTURE``.
|
| 413 |
|
| 414 | * **Returned fields:**
|
| 415 |
|
| 416 | ``width``, ``height``
|
| 417 | frame buffer resolution for the decoded frames.
|
| 418 |
|
| 419 | ``pixelformat``
|
| 420 | pixel format for decoded frames.
|
| 421 |
|
| 422 | ``num_planes`` (for _MPLANE ``type`` only)
|
| 423 | number of planes for pixelformat.
|
| 424 |
|
| 425 | ``sizeimage``, ``bytesperline``
|
| 426 | as per standard semantics; matching frame buffer format.
|
| 427 |
|
| 428 | .. note::
|
| 429 |
|
| 430 | The value of ``pixelformat`` may be any pixel format supported by the
|
| 431 | decoder for the current stream. The decoder should choose a
|
| 432 | preferred/optimal format for the default configuration. For example, a
|
| 433 | YUV format may be preferred over an RGB format if an additional
|
| 434 | conversion step would be required for the latter.
|
| 435 |
|
| 436 | 2. **Optional.** Acquire the visible resolution via
|
| 437 | :c:func:`VIDIOC_G_SELECTION`.
|
| 438 |
|
| 439 | * **Required fields:**
|
| 440 |
|
| 441 | ``type``
|
| 442 | a ``V4L2_BUF_TYPE_*`` enum appropriate for ``CAPTURE``.
|
| 443 |
|
| 444 | ``target``
|
| 445 | set to ``V4L2_SEL_TGT_COMPOSE``.
|
| 446 |
|
| 447 | * **Returned fields:**
|
| 448 |
|
| 449 | ``r.left``, ``r.top``, ``r.width``, ``r.height``
|
| 450 | the visible rectangle; it must fit within the frame buffer resolution
|
| 451 | returned by :c:func:`VIDIOC_G_FMT` on ``CAPTURE``.
|
| 452 |
|
| 453 | * The following selection targets are supported on ``CAPTURE``:
|
| 454 |
|
| 455 | ``V4L2_SEL_TGT_CROP_BOUNDS``
|
| 456 | corresponds to the coded resolution of the stream.
|
| 457 |
|
| 458 | ``V4L2_SEL_TGT_CROP_DEFAULT``
|
| 459 | the rectangle covering the part of the ``CAPTURE`` buffer that
|
| 460 | contains meaningful picture data (visible area); width and height
|
| 461 | will be equal to the visible resolution of the stream.
|
| 462 |
|
| 463 | ``V4L2_SEL_TGT_CROP``
|
| 464 | the rectangle within the coded resolution to be output to
|
| 465 | ``CAPTURE``; defaults to ``V4L2_SEL_TGT_CROP_DEFAULT``; read-only on
|
| 466 | hardware without additional compose/scaling capabilities.
|
| 467 |
|
| 468 | ``V4L2_SEL_TGT_COMPOSE_BOUNDS``
|
| 469 | the maximum rectangle within a ``CAPTURE`` buffer, which the cropped
|
| 470 | frame can be composed into; equal to ``V4L2_SEL_TGT_CROP`` if the
|
| 471 | hardware does not support compose/scaling.
|
| 472 |
|
| 473 | ``V4L2_SEL_TGT_COMPOSE_DEFAULT``
|
| 474 | equal to ``V4L2_SEL_TGT_CROP``.
|
| 475 |
|
| 476 | ``V4L2_SEL_TGT_COMPOSE``
|
| 477 | the rectangle inside a ``CAPTURE`` buffer into which the cropped
|
| 478 | frame is written; defaults to ``V4L2_SEL_TGT_COMPOSE_DEFAULT``;
|
| 479 | read-only on hardware without additional compose/scaling capabilities.
|
| 480 |
|
| 481 | ``V4L2_SEL_TGT_COMPOSE_PADDED``
|
| 482 | the rectangle inside a ``CAPTURE`` buffer which is overwritten by the
|
| 483 | hardware; equal to ``V4L2_SEL_TGT_COMPOSE`` if the hardware does not
|
| 484 | write padding pixels.
|
| 485 |
|
| 486 | .. warning::
|
| 487 |
|
| 488 | The values are guaranteed to be meaningful only after the decoder
|
| 489 | successfully parses the stream metadata. The client must not rely on the
|
| 490 | query before that happens.
|
| 491 |
|
| 492 | 3. **Optional.** Enumerate ``CAPTURE`` formats via :c:func:`VIDIOC_ENUM_FMT` on
|
| 493 | the ``CAPTURE`` queue. Once the stream information is parsed and known, the
|
| 494 | client may use this ioctl to discover which raw formats are supported for
|
| 495 | given stream and select one of them via :c:func:`VIDIOC_S_FMT`.
|
| 496 |
|
| 497 | .. important::
|
| 498 |
|
| 499 | The decoder will return only formats supported for the currently
|
| 500 | established coded format, as per the ``OUTPUT`` format and/or stream
|
| 501 | metadata parsed in this initialization sequence, even if more formats
|
| 502 | may be supported by the decoder in general. In other words, the set
|
| 503 | returned will be a subset of the initial query mentioned in the
|
| 504 | `Querying Capabilities` section.
|
| 505 |
|
| 506 | For example, a decoder may support YUV and RGB formats for resolutions
|
| 507 | 1920x1088 and lower, but only YUV for higher resolutions (due to
|
| 508 | hardware limitations). After parsing a resolution of 1920x1088 or lower,
|
| 509 | :c:func:`VIDIOC_ENUM_FMT` may return a set of YUV and RGB pixel formats,
|
| 510 | but after parsing resolution higher than 1920x1088, the decoder will not
|
| 511 | return RGB, unsupported for this resolution.
|
| 512 |
|
| 513 | However, subsequent resolution change event triggered after
|
| 514 | discovering a resolution change within the same stream may switch
|
| 515 | the stream into a lower resolution and :c:func:`VIDIOC_ENUM_FMT`
|
| 516 | would return RGB formats again in that case.
|
| 517 |
|
| 518 | 4. **Optional.** Set the ``CAPTURE`` format via :c:func:`VIDIOC_S_FMT` on the
|
| 519 | ``CAPTURE`` queue. The client may choose a different format than
|
| 520 | selected/suggested by the decoder in :c:func:`VIDIOC_G_FMT`.
|
| 521 |
|
| 522 | * **Required fields:**
|
| 523 |
|
| 524 | ``type``
|
| 525 | a ``V4L2_BUF_TYPE_*`` enum appropriate for ``CAPTURE``.
|
| 526 |
|
| 527 | ``pixelformat``
|
| 528 | a raw pixel format.
|
| 529 |
|
| 530 | ``width``, ``height``
|
| 531 | frame buffer resolution of the decoded stream; typically unchanged from
|
| 532 | what was returned with :c:func:`VIDIOC_G_FMT`, but it may be different
|
| 533 | if the hardware supports composition and/or scaling.
|
| 534 |
|
| 535 | * Setting the ``CAPTURE`` format will reset the compose selection rectangles
|
| 536 | to their default values, based on the new resolution, as described in the
|
| 537 | previous step.
|
| 538 |
|
| 539 | 5. **Optional.** Set the compose rectangle via :c:func:`VIDIOC_S_SELECTION` on
|
| 540 | the ``CAPTURE`` queue if it is desired and if the decoder has compose and/or
|
| 541 | scaling capabilities.
|
| 542 |
|
| 543 | * **Required fields:**
|
| 544 |
|
| 545 | ``type``
|
| 546 | a ``V4L2_BUF_TYPE_*`` enum appropriate for ``CAPTURE``.
|
| 547 |
|
| 548 | ``target``
|
| 549 | set to ``V4L2_SEL_TGT_COMPOSE``.
|
| 550 |
|
| 551 | ``r.left``, ``r.top``, ``r.width``, ``r.height``
|
| 552 | the rectangle inside a ``CAPTURE`` buffer into which the cropped
|
| 553 | frame is written; defaults to ``V4L2_SEL_TGT_COMPOSE_DEFAULT``;
|
| 554 | read-only on hardware without additional compose/scaling capabilities.
|
| 555 |
|
| 556 | * **Returned fields:**
|
| 557 |
|
| 558 | ``r.left``, ``r.top``, ``r.width``, ``r.height``
|
| 559 | the visible rectangle; it must fit within the frame buffer resolution
|
| 560 | returned by :c:func:`VIDIOC_G_FMT` on ``CAPTURE``.
|
| 561 |
|
| 562 | .. warning::
|
| 563 |
|
| 564 | The decoder may adjust the compose rectangle to the nearest
|
| 565 | supported one to meet codec and hardware requirements. The client needs
|
| 566 | to check the adjusted rectangle returned by :c:func:`VIDIOC_S_SELECTION`.
|
| 567 |
|
| 568 | 6. If all the following conditions are met, the client may resume the decoding
|
| 569 | instantly:
|
| 570 |
|
| 571 | * ``sizeimage`` of the new format (determined in previous steps) is less
|
| 572 | than or equal to the size of currently allocated buffers,
|
| 573 |
|
| 574 | * the number of buffers currently allocated is greater than or equal to the
|
| 575 | minimum number of buffers acquired in previous steps. To fulfill this
|
| 576 | requirement, the client may use :c:func:`VIDIOC_CREATE_BUFS` to add new
|
| 577 | buffers.
|
| 578 |
|
| 579 | In that case, the remaining steps do not apply and the client may resume
|
| 580 | the decoding by one of the following actions:
|
| 581 |
|
| 582 | * if the ``CAPTURE`` queue is streaming, call :c:func:`VIDIOC_DECODER_CMD`
|
| 583 | with the ``V4L2_DEC_CMD_START`` command,
|
| 584 |
|
| 585 | * if the ``CAPTURE`` queue is not streaming, call :c:func:`VIDIOC_STREAMON`
|
| 586 | on the ``CAPTURE`` queue.
|
| 587 |
|
| 588 | However, if the client intends to change the buffer set, to lower
|
| 589 | memory usage or for any other reasons, it may be achieved by following
|
| 590 | the steps below.
|
| 591 |
|
| 592 | 7. **If the** ``CAPTURE`` **queue is streaming,** keep queuing and dequeuing
|
| 593 | buffers on the ``CAPTURE`` queue until a buffer marked with the
|
| 594 | ``V4L2_BUF_FLAG_LAST`` flag is dequeued.
|
| 595 |
|
| 596 | 8. **If the** ``CAPTURE`` **queue is streaming,** call :c:func:`VIDIOC_STREAMOFF`
|
| 597 | on the ``CAPTURE`` queue to stop streaming.
|
| 598 |
|
| 599 | .. warning::
|
| 600 |
|
| 601 | The ``OUTPUT`` queue must remain streaming. Calling
|
| 602 | :c:func:`VIDIOC_STREAMOFF` on it would abort the sequence and trigger a
|
| 603 | seek.
|
| 604 |
|
| 605 | 9. **If the** ``CAPTURE`` **queue has buffers allocated,** free the ``CAPTURE``
|
| 606 | buffers using :c:func:`VIDIOC_REQBUFS`.
|
| 607 |
|
| 608 | * **Required fields:**
|
| 609 |
|
| 610 | ``count``
|
| 611 | set to 0.
|
| 612 |
|
| 613 | ``type``
|
| 614 | a ``V4L2_BUF_TYPE_*`` enum appropriate for ``CAPTURE``.
|
| 615 |
|
| 616 | ``memory``
|
| 617 | follows standard semantics.
|
| 618 |
|
| 619 | 10. Allocate ``CAPTURE`` buffers via :c:func:`VIDIOC_REQBUFS` on the
|
| 620 | ``CAPTURE`` queue.
|
| 621 |
|
| 622 | * **Required fields:**
|
| 623 |
|
| 624 | ``count``
|
| 625 | requested number of buffers to allocate; greater than zero.
|
| 626 |
|
| 627 | ``type``
|
| 628 | a ``V4L2_BUF_TYPE_*`` enum appropriate for ``CAPTURE``.
|
| 629 |
|
| 630 | ``memory``
|
| 631 | follows standard semantics.
|
| 632 |
|
| 633 | * **Returned fields:**
|
| 634 |
|
| 635 | ``count``
|
| 636 | actual number of buffers allocated.
|
| 637 |
|
| 638 | .. warning::
|
| 639 |
|
| 640 | The actual number of allocated buffers may differ from the ``count``
|
| 641 | given. The client must check the updated value of ``count`` after the
|
| 642 | call returns.
|
| 643 |
|
| 644 | .. note::
|
| 645 |
|
| 646 | To allocate more than the minimum number of buffers (for pipeline
|
| 647 | depth), the client may query the ``V4L2_CID_MIN_BUFFERS_FOR_CAPTURE``
|
| 648 | control to get the minimum number of buffers required, and pass the
|
| 649 | obtained value plus the number of additional buffers needed in the
|
| 650 | ``count`` field to :c:func:`VIDIOC_REQBUFS`.
|
| 651 |
|
| 652 | Alternatively, :c:func:`VIDIOC_CREATE_BUFS` on the ``CAPTURE`` queue can be
|
| 653 | used to have more control over buffer allocation. For example, by
|
| 654 | allocating buffers larger than the current ``CAPTURE`` format, future
|
| 655 | resolution changes can be accommodated.
|
| 656 |
|
| 657 | * **Required fields:**
|
| 658 |
|
| 659 | ``count``
|
| 660 | requested number of buffers to allocate; greater than zero.
|
| 661 |
|
| 662 | ``type``
|
| 663 | a ``V4L2_BUF_TYPE_*`` enum appropriate for ``CAPTURE``.
|
| 664 |
|
| 665 | ``memory``
|
| 666 | follows standard semantics.
|
| 667 |
|
| 668 | ``format``
|
| 669 | a format representing the maximum framebuffer resolution to be
|
| 670 | accommodated by newly allocated buffers.
|
| 671 |
|
| 672 | * **Returned fields:**
|
| 673 |
|
| 674 | ``count``
|
| 675 | adjusted to the number of allocated buffers.
|
| 676 |
|
| 677 | .. warning::
|
| 678 |
|
| 679 | The actual number of allocated buffers may differ from the ``count``
|
| 680 | given. The client must check the updated value of ``count`` after the
|
| 681 | call returns.
|
| 682 |
|
| 683 | .. note::
|
| 684 |
|
| 685 | To allocate buffers for a format different than parsed from the stream
|
| 686 | metadata, the client must proceed as follows, before the metadata
|
| 687 | parsing is initiated:
|
| 688 |
|
| 689 | * set width and height of the ``OUTPUT`` format to desired coded resolution to
|
| 690 | let the decoder configure the ``CAPTURE`` format appropriately,
|
| 691 |
|
| 692 | * query the ``CAPTURE`` format using :c:func:`VIDIOC_G_FMT` and save it
|
| 693 | until this step.
|
| 694 |
|
| 695 | The format obtained in the query may be then used with
|
| 696 | :c:func:`VIDIOC_CREATE_BUFS` in this step to allocate the buffers.
|
| 697 |
|
| 698 | 11. Call :c:func:`VIDIOC_STREAMON` on the ``CAPTURE`` queue to start decoding
|
| 699 | frames.
|
| 700 |
|
| 701 | Decoding
|
| 702 | ========
|
| 703 |
|
| 704 | This state is reached after the `Capture Setup` sequence finishes successfully.
|
| 705 | In this state, the client queues and dequeues buffers to both queues via
|
| 706 | :c:func:`VIDIOC_QBUF` and :c:func:`VIDIOC_DQBUF`, following the standard
|
| 707 | semantics.
|
| 708 |
|
| 709 | The content of the source ``OUTPUT`` buffers depends on the active coded pixel
|
| 710 | format and may be affected by codec-specific extended controls, as stated in
|
| 711 | the documentation of each format.
|
| 712 |
|
| 713 | Both queues operate independently, following the standard behavior of V4L2
|
| 714 | buffer queues and memory-to-memory devices. In addition, the order of decoded
|
| 715 | frames dequeued from the ``CAPTURE`` queue may differ from the order of queuing
|
| 716 | coded frames to the ``OUTPUT`` queue, due to properties of the selected coded
|
| 717 | format, e.g. frame reordering.
|
| 718 |
|
| 719 | The client must not assume any direct relationship between ``CAPTURE``
|
| 720 | and ``OUTPUT`` buffers and any specific timing of buffers becoming
|
| 721 | available to dequeue. Specifically:
|
| 722 |
|
| 723 | * a buffer queued to ``OUTPUT`` may result in no buffers being produced
|
| 724 | on ``CAPTURE`` (e.g. if it does not contain encoded data, or if only
|
| 725 | metadata syntax structures are present in it),
|
| 726 |
|
| 727 | * a buffer queued to ``OUTPUT`` may result in more than one buffer produced
|
| 728 | on ``CAPTURE`` (if the encoded data contained more than one frame, or if
|
| 729 | returning a decoded frame allowed the decoder to return a frame that
|
| 730 | preceded it in decode, but succeeded it in the display order),
|
| 731 |
|
| 732 | * a buffer queued to ``OUTPUT`` may result in a buffer being produced on
|
| 733 | ``CAPTURE`` later into decode process, and/or after processing further
|
| 734 | ``OUTPUT`` buffers, or be returned out of order, e.g. if display
|
| 735 | reordering is used,
|
| 736 |
|
| 737 | * buffers may become available on the ``CAPTURE`` queue without additional
|
| 738 | buffers queued to ``OUTPUT`` (e.g. during drain or ``EOS``), because of the
|
| 739 | ``OUTPUT`` buffers queued in the past whose decoding results are only
|
| 740 | available at later time, due to specifics of the decoding process.
|
| 741 |
|
| 742 | .. note::
|
| 743 |
|
| 744 | To allow matching decoded ``CAPTURE`` buffers with ``OUTPUT`` buffers they
|
| 745 | originated from, the client can set the ``timestamp`` field of the
|
| 746 | :c:type:`v4l2_buffer` struct when queuing an ``OUTPUT`` buffer. The
|
| 747 | ``CAPTURE`` buffer(s), which resulted from decoding that ``OUTPUT`` buffer
|
| 748 | will have their ``timestamp`` field set to the same value when dequeued.
|
| 749 |
|
| 750 | In addition to the straightforward case of one ``OUTPUT`` buffer producing
|
| 751 | one ``CAPTURE`` buffer, the following cases are defined:
|
| 752 |
|
| 753 | * one ``OUTPUT`` buffer generates multiple ``CAPTURE`` buffers: the same
|
| 754 | ``OUTPUT`` timestamp will be copied to multiple ``CAPTURE`` buffers.
|
| 755 |
|
| 756 | * multiple ``OUTPUT`` buffers generate one ``CAPTURE`` buffer: timestamp of
|
| 757 | the ``OUTPUT`` buffer queued first will be copied.
|
| 758 |
|
| 759 | * the decoding order differs from the display order (i.e. the ``CAPTURE``
|
| 760 | buffers are out-of-order compared to the ``OUTPUT`` buffers): ``CAPTURE``
|
| 761 | timestamps will not retain the order of ``OUTPUT`` timestamps.
|
| 762 |
|
| 763 | .. note::
|
| 764 |
|
| 765 | The backing memory of ``CAPTURE`` buffers that are used as reference frames
|
| 766 | by the stream may be read by the hardware even after they are dequeued.
|
| 767 | Consequently, the client should avoid writing into this memory while the
|
| 768 | ``CAPTURE`` queue is streaming. Failure to observe this may result in
|
| 769 | corruption of decoded frames.
|
| 770 |
|
| 771 | Similarly, when using a memory type other than ``V4L2_MEMORY_MMAP``, the
|
| 772 | client should make sure that each ``CAPTURE`` buffer is always queued with
|
| 773 | the same backing memory for as long as the ``CAPTURE`` queue is streaming.
|
| 774 | The reason for this is that V4L2 buffer indices can be used by drivers to
|
| 775 | identify frames. Thus, if the backing memory of a reference frame is
|
| 776 | submitted under a different buffer ID, the driver may misidentify it and
|
| 777 | decode a new frame into it while it is still in use, resulting in corruption
|
| 778 | of the following frames.
|
| 779 |
|
| 780 | During the decoding, the decoder may initiate one of the special sequences, as
|
| 781 | listed below. The sequences will result in the decoder returning all the
|
| 782 | ``CAPTURE`` buffers that originated from all the ``OUTPUT`` buffers processed
|
| 783 | before the sequence started. Last of the buffers will have the
|
| 784 | ``V4L2_BUF_FLAG_LAST`` flag set. To determine the sequence to follow, the client
|
| 785 | must check if there is any pending event and:
|
| 786 |
|
| 787 | * if a ``V4L2_EVENT_SOURCE_CHANGE`` event with ``changes`` set to
|
| 788 | ``V4L2_EVENT_SRC_CH_RESOLUTION`` is pending, the `Dynamic Resolution
|
| 789 | Change` sequence needs to be followed,
|
| 790 |
|
| 791 | * if a ``V4L2_EVENT_EOS`` event is pending, the `End of Stream` sequence needs
|
| 792 | to be followed.
|
| 793 |
|
| 794 | Some of the sequences can be intermixed with each other and need to be handled
|
| 795 | as they happen. The exact operation is documented for each sequence.
|
| 796 |
|
| 797 | Should a decoding error occur, it will be reported to the client with the level
|
| 798 | of details depending on the decoder capabilities. Specifically:
|
| 799 |
|
| 800 | * the CAPTURE buffer that contains the results of the failed decode operation
|
| 801 | will be returned with the V4L2_BUF_FLAG_ERROR flag set,
|
| 802 |
|
| 803 | * if the decoder is able to precisely report the OUTPUT buffer that triggered
|
| 804 | the error, such buffer will be returned with the V4L2_BUF_FLAG_ERROR flag
|
| 805 | set.
|
| 806 |
|
| 807 | In case of a fatal failure that does not allow the decoding to continue, any
|
| 808 | further operations on corresponding decoder file handle will return the -EIO
|
| 809 | error code. The client may close the file handle and open a new one, or
|
| 810 | alternatively reinitialize the instance by stopping streaming on both queues,
|
| 811 | releasing all buffers and performing the Initialization sequence again.
|
| 812 |
|
| 813 | Seek
|
| 814 | ====
|
| 815 |
|
| 816 | Seek is controlled by the ``OUTPUT`` queue, as it is the source of coded data.
|
| 817 | The seek does not require any specific operation on the ``CAPTURE`` queue, but
|
| 818 | it may be affected as per normal decoder operation.
|
| 819 |
|
| 820 | 1. Stop the ``OUTPUT`` queue to begin the seek sequence via
|
| 821 | :c:func:`VIDIOC_STREAMOFF`.
|
| 822 |
|
| 823 | * **Required fields:**
|
| 824 |
|
| 825 | ``type``
|
| 826 | a ``V4L2_BUF_TYPE_*`` enum appropriate for ``OUTPUT``.
|
| 827 |
|
| 828 | * The decoder will drop all the pending ``OUTPUT`` buffers and they must be
|
| 829 | treated as returned to the client (following standard semantics).
|
| 830 |
|
| 831 | 2. Restart the ``OUTPUT`` queue via :c:func:`VIDIOC_STREAMON`.
|
| 832 |
|
| 833 | * **Required fields:**
|
| 834 |
|
| 835 | ``type``
|
| 836 | a ``V4L2_BUF_TYPE_*`` enum appropriate for ``OUTPUT``.
|
| 837 |
|
| 838 | * The decoder will start accepting new source bytestream buffers after the
|
| 839 | call returns.
|
| 840 |
|
| 841 | 3. Start queuing buffers containing coded data after the seek to the ``OUTPUT``
|
| 842 | queue until a suitable resume point is found.
|
| 843 |
|
| 844 | .. note::
|
| 845 |
|
| 846 | There is no requirement to begin queuing coded data starting exactly
|
| 847 | from a resume point (e.g. SPS or a keyframe). Any queued ``OUTPUT``
|
| 848 | buffers will be processed and returned to the client until a suitable
|
| 849 | resume point is found. While looking for a resume point, the decoder
|
| 850 | should not produce any decoded frames into ``CAPTURE`` buffers.
|
| 851 |
|
| 852 | Some hardware is known to mishandle seeks to a non-resume point. Such an
|
| 853 | operation may result in an unspecified number of corrupted decoded frames
|
| 854 | being made available on the ``CAPTURE`` queue. Drivers must ensure that
|
| 855 | no fatal decoding errors or crashes occur, and implement any necessary
|
| 856 | handling and workarounds for hardware issues related to seek operations.
|
| 857 |
|
| 858 | .. warning::
|
| 859 |
|
| 860 | In case of the H.264/HEVC codec, the client must take care not to seek
|
| 861 | over a change of SPS/PPS. Even though the target frame could be a
|
| 862 | keyframe, the stale SPS/PPS inside decoder state would lead to undefined
|
| 863 | results when decoding. Although the decoder must handle that case without
|
| 864 | a crash or a fatal decode error, the client must not expect a sensible
|
| 865 | decode output.
|
| 866 |
|
| 867 | If the hardware can detect such corrupted decoded frames, then
|
| 868 | corresponding buffers will be returned to the client with the
|
| 869 | V4L2_BUF_FLAG_ERROR set. See the `Decoding` section for further
|
| 870 | description of decode error reporting.
|
| 871 |
|
| 872 | 4. After a resume point is found, the decoder will start returning ``CAPTURE``
|
| 873 | buffers containing decoded frames.
|
| 874 |
|
| 875 | .. important::
|
| 876 |
|
| 877 | A seek may result in the `Dynamic Resolution Change` sequence being
|
| 878 | initiated, due to the seek target having decoding parameters different from
|
| 879 | the part of the stream decoded before the seek. The sequence must be handled
|
| 880 | as per normal decoder operation.
|
| 881 |
|
| 882 | .. warning::
|
| 883 |
|
| 884 | It is not specified when the ``CAPTURE`` queue starts producing buffers
|
| 885 | containing decoded data from the ``OUTPUT`` buffers queued after the seek,
|
| 886 | as it operates independently from the ``OUTPUT`` queue.
|
| 887 |
|
| 888 | The decoder may return a number of remaining ``CAPTURE`` buffers containing
|
| 889 | decoded frames originating from the ``OUTPUT`` buffers queued before the
|
| 890 | seek sequence is performed.
|
| 891 |
|
| 892 | The ``VIDIOC_STREAMOFF`` operation discards any remaining queued
|
| 893 | ``OUTPUT`` buffers, which means that not all of the ``OUTPUT`` buffers
|
| 894 | queued before the seek sequence may have matching ``CAPTURE`` buffers
|
| 895 | produced. For example, given the sequence of operations on the
|
| 896 | ``OUTPUT`` queue:
|
| 897 |
|
| 898 | QBUF(A), QBUF(B), STREAMOFF(), STREAMON(), QBUF(G), QBUF(H),
|
| 899 |
|
| 900 | any of the following results on the ``CAPTURE`` queue is allowed:
|
| 901 |
|
| 902 | {A', B', G', H'}, {A', G', H'}, {G', H'}.
|
| 903 |
|
| 904 | To determine the CAPTURE buffer containing the first decoded frame after the
|
| 905 | seek, the client may observe the timestamps to match the CAPTURE and OUTPUT
|
| 906 | buffers or use V4L2_DEC_CMD_STOP and V4L2_DEC_CMD_START to drain the
|
| 907 | decoder.
|
| 908 |
|
| 909 | .. note::
|
| 910 |
|
| 911 | To achieve instantaneous seek, the client may restart streaming on the
|
| 912 | ``CAPTURE`` queue too to discard decoded, but not yet dequeued buffers.
|
| 913 |
|
| 914 | Dynamic Resolution Change
|
| 915 | =========================
|
| 916 |
|
| 917 | Streams that include resolution metadata in the bytestream may require switching
|
| 918 | to a different resolution during the decoding.
|
| 919 |
|
| 920 | .. note::
|
| 921 |
|
| 922 | Not all decoders can detect resolution changes. Those that do set the
|
| 923 | ``V4L2_FMT_FLAG_DYN_RESOLUTION`` flag for the coded format when
|
| 924 | :c:func:`VIDIOC_ENUM_FMT` is called.
|
| 925 |
|
| 926 | The sequence starts when the decoder detects a coded frame with one or more of
|
| 927 | the following parameters different from those previously established (and
|
| 928 | reflected by corresponding queries):
|
| 929 |
|
| 930 | * coded resolution (``OUTPUT`` width and height),
|
| 931 |
|
| 932 | * visible resolution (selection rectangles),
|
| 933 |
|
| 934 | * the minimum number of buffers needed for decoding,
|
| 935 |
|
| 936 | * bit-depth of the bitstream has been changed,
|
| 937 |
|
| 938 | * colorspace of the bitstream has been changed, but it doesn't require
|
| 939 | buffer reallocation.
|
| 940 |
|
| 941 | Whenever that happens, the decoder must proceed as follows:
|
| 942 |
|
| 943 | 1. After encountering a resolution change in the stream, the decoder sends a
|
| 944 | ``V4L2_EVENT_SOURCE_CHANGE`` event with ``changes`` set to
|
| 945 | ``V4L2_EVENT_SRC_CH_RESOLUTION``.
|
| 946 |
|
| 947 | .. important::
|
| 948 |
|
| 949 | Any client query issued after the decoder queues the event will return
|
| 950 | values applying to the stream after the resolution change, including
|
| 951 | queue formats, selection rectangles and controls.
|
| 952 |
|
| 953 | 2. The decoder will then process and decode all remaining buffers from before
|
| 954 | the resolution change point.
|
| 955 |
|
| 956 | * The last buffer from before the change must be marked with the
|
| 957 | ``V4L2_BUF_FLAG_LAST`` flag, similarly to the `Drain` sequence above.
|
| 958 |
|
| 959 | .. warning::
|
| 960 |
|
| 961 | The last buffer may be empty (with :c:type:`v4l2_buffer` ``bytesused``
|
| 962 | = 0) and in that case it must be ignored by the client, as it does not
|
| 963 | contain a decoded frame.
|
| 964 |
|
| 965 | .. note::
|
| 966 |
|
| 967 | Any attempt to dequeue more ``CAPTURE`` buffers beyond the buffer marked
|
| 968 | with ``V4L2_BUF_FLAG_LAST`` will result in a -EPIPE error from
|
| 969 | :c:func:`VIDIOC_DQBUF`.
|
| 970 |
|
| 971 | The client must continue the sequence as described below to continue the
|
| 972 | decoding process.
|
| 973 |
|
| 974 | 1. Dequeue the source change event.
|
| 975 |
|
| 976 | .. important::
|
| 977 |
|
| 978 | A source change triggers an implicit decoder drain, similar to the
|
| 979 | explicit `Drain` sequence. The decoder is stopped after it completes.
|
| 980 | The decoding process must be resumed with either a pair of calls to
|
| 981 | :c:func:`VIDIOC_STREAMOFF` and :c:func:`VIDIOC_STREAMON` on the
|
| 982 | ``CAPTURE`` queue, or a call to :c:func:`VIDIOC_DECODER_CMD` with the
|
| 983 | ``V4L2_DEC_CMD_START`` command.
|
| 984 |
|
| 985 | 2. Continue with the `Capture Setup` sequence.
|
| 986 |
|
| 987 | .. note::
|
| 988 |
|
| 989 | During the resolution change sequence, the ``OUTPUT`` queue must remain
|
| 990 | streaming. Calling :c:func:`VIDIOC_STREAMOFF` on the ``OUTPUT`` queue would
|
| 991 | abort the sequence and initiate a seek.
|
| 992 |
|
| 993 | In principle, the ``OUTPUT`` queue operates separately from the ``CAPTURE``
|
| 994 | queue and this remains true for the duration of the entire resolution change
|
| 995 | sequence as well.
|
| 996 |
|
| 997 | The client should, for best performance and simplicity, keep queuing/dequeuing
|
| 998 | buffers to/from the ``OUTPUT`` queue even while processing this sequence.
|
| 999 |
|
| 1000 | Drain
|
| 1001 | =====
|
| 1002 |
|
| 1003 | To ensure that all queued ``OUTPUT`` buffers have been processed and related
|
| 1004 | ``CAPTURE`` buffers are given to the client, the client must follow the drain
|
| 1005 | sequence described below. After the drain sequence ends, the client has
|
| 1006 | received all decoded frames for all ``OUTPUT`` buffers queued before the
|
| 1007 | sequence was started.
|
| 1008 |
|
| 1009 | 1. Begin drain by issuing :c:func:`VIDIOC_DECODER_CMD`.
|
| 1010 |
|
| 1011 | * **Required fields:**
|
| 1012 |
|
| 1013 | ``cmd``
|
| 1014 | set to ``V4L2_DEC_CMD_STOP``.
|
| 1015 |
|
| 1016 | ``flags``
|
| 1017 | set to 0.
|
| 1018 |
|
| 1019 | ``pts``
|
| 1020 | set to 0.
|
| 1021 |
|
| 1022 | .. warning::
|
| 1023 |
|
| 1024 | The sequence can be only initiated if both ``OUTPUT`` and ``CAPTURE``
|
| 1025 | queues are streaming. For compatibility reasons, the call to
|
| 1026 | :c:func:`VIDIOC_DECODER_CMD` will not fail even if any of the queues is
|
| 1027 | not streaming, but at the same time it will not initiate the `Drain`
|
| 1028 | sequence and so the steps described below would not be applicable.
|
| 1029 |
|
| 1030 | 2. Any ``OUTPUT`` buffers queued by the client before the
|
| 1031 | :c:func:`VIDIOC_DECODER_CMD` was issued will be processed and decoded as
|
| 1032 | normal. The client must continue to handle both queues independently,
|
| 1033 | similarly to normal decode operation. This includes:
|
| 1034 |
|
| 1035 | * handling any operations triggered as a result of processing those buffers,
|
| 1036 | such as the `Dynamic Resolution Change` sequence, before continuing with
|
| 1037 | the drain sequence,
|
| 1038 |
|
| 1039 | * queuing and dequeuing ``CAPTURE`` buffers, until a buffer marked with the
|
| 1040 | ``V4L2_BUF_FLAG_LAST`` flag is dequeued,
|
| 1041 |
|
| 1042 | .. warning::
|
| 1043 |
|
| 1044 | The last buffer may be empty (with :c:type:`v4l2_buffer`
|
| 1045 | ``bytesused`` = 0) and in that case it must be ignored by the client,
|
| 1046 | as it does not contain a decoded frame.
|
| 1047 |
|
| 1048 | .. note::
|
| 1049 |
|
| 1050 | Any attempt to dequeue more ``CAPTURE`` buffers beyond the buffer
|
| 1051 | marked with ``V4L2_BUF_FLAG_LAST`` will result in a -EPIPE error from
|
| 1052 | :c:func:`VIDIOC_DQBUF`.
|
| 1053 |
|
| 1054 | * dequeuing processed ``OUTPUT`` buffers, until all the buffers queued
|
| 1055 | before the ``V4L2_DEC_CMD_STOP`` command are dequeued,
|
| 1056 |
|
| 1057 | * dequeuing the ``V4L2_EVENT_EOS`` event, if the client subscribed to it.
|
| 1058 |
|
| 1059 | .. note::
|
| 1060 |
|
| 1061 | For backwards compatibility, the decoder will signal a ``V4L2_EVENT_EOS``
|
| 1062 | event when the last frame has been decoded and all frames are ready to be
|
| 1063 | dequeued. It is a deprecated behavior and the client must not rely on it.
|
| 1064 | The ``V4L2_BUF_FLAG_LAST`` buffer flag should be used instead.
|
| 1065 |
|
| 1066 | 3. Once all the ``OUTPUT`` buffers queued before the ``V4L2_DEC_CMD_STOP`` call
|
| 1067 | are dequeued and the last ``CAPTURE`` buffer is dequeued, the decoder is
|
| 1068 | stopped and it will accept, but not process, any newly queued ``OUTPUT``
|
| 1069 | buffers until the client issues any of the following operations:
|
| 1070 |
|
| 1071 | * ``V4L2_DEC_CMD_START`` - the decoder will not be reset and will resume
|
| 1072 | operation normally, with all the state from before the drain,
|
| 1073 |
|
| 1074 | * a pair of :c:func:`VIDIOC_STREAMOFF` and :c:func:`VIDIOC_STREAMON` on the
|
| 1075 | ``CAPTURE`` queue - the decoder will resume the operation normally,
|
| 1076 | however any ``CAPTURE`` buffers still in the queue will be returned to the
|
| 1077 | client,
|
| 1078 |
|
| 1079 | * a pair of :c:func:`VIDIOC_STREAMOFF` and :c:func:`VIDIOC_STREAMON` on the
|
| 1080 | ``OUTPUT`` queue - any pending source buffers will be returned to the
|
| 1081 | client and the `Seek` sequence will be triggered.
|
| 1082 |
|
| 1083 | .. note::
|
| 1084 |
|
| 1085 | Once the drain sequence is initiated, the client needs to drive it to
|
| 1086 | completion, as described by the steps above, unless it aborts the process by
|
| 1087 | issuing :c:func:`VIDIOC_STREAMOFF` on any of the ``OUTPUT`` or ``CAPTURE``
|
| 1088 | queues. The client is not allowed to issue ``V4L2_DEC_CMD_START`` or
|
| 1089 | ``V4L2_DEC_CMD_STOP`` again while the drain sequence is in progress and they
|
| 1090 | will fail with -EBUSY error code if attempted.
|
| 1091 |
|
| 1092 | Although not mandatory, the availability of decoder commands may be queried
|
| 1093 | using :c:func:`VIDIOC_TRY_DECODER_CMD`.
|
| 1094 |
|
| 1095 | End of Stream
|
| 1096 | =============
|
| 1097 |
|
| 1098 | If the decoder encounters an end of stream marking in the stream, the decoder
|
| 1099 | will initiate the `Drain` sequence, which the client must handle as described
|
| 1100 | above, skipping the initial :c:func:`VIDIOC_DECODER_CMD`.
|
| 1101 |
|
| 1102 | Commit Points
|
| 1103 | =============
|
| 1104 |
|
| 1105 | Setting formats and allocating buffers trigger changes in the behavior of the
|
| 1106 | decoder.
|
| 1107 |
|
| 1108 | 1. Setting the format on the ``OUTPUT`` queue may change the set of formats
|
| 1109 | supported/advertised on the ``CAPTURE`` queue. In particular, it also means
|
| 1110 | that the ``CAPTURE`` format may be reset and the client must not rely on the
|
| 1111 | previously set format being preserved.
|
| 1112 |
|
| 1113 | 2. Enumerating formats on the ``CAPTURE`` queue always returns only formats
|
| 1114 | supported for the current ``OUTPUT`` format.
|
| 1115 |
|
| 1116 | 3. Setting the format on the ``CAPTURE`` queue does not change the list of
|
| 1117 | formats available on the ``OUTPUT`` queue. An attempt to set a ``CAPTURE``
|
| 1118 | format that is not supported for the currently selected ``OUTPUT`` format
|
| 1119 | will result in the decoder adjusting the requested ``CAPTURE`` format to a
|
| 1120 | supported one.
|
| 1121 |
|
| 1122 | 4. Enumerating formats on the ``OUTPUT`` queue always returns the full set of
|
| 1123 | supported coded formats, irrespectively of the current ``CAPTURE`` format.
|
| 1124 |
|
| 1125 | 5. While buffers are allocated on any of the ``OUTPUT`` or ``CAPTURE`` queues,
|
| 1126 | the client must not change the format on the ``OUTPUT`` queue. Drivers will
|
| 1127 | return the -EBUSY error code for any such format change attempt.
|
| 1128 |
|
| 1129 | To summarize, setting formats and allocation must always start with the
|
| 1130 | ``OUTPUT`` queue and the ``OUTPUT`` queue is the master that governs the
|
| 1131 | set of supported formats for the ``CAPTURE`` queue.
|