| 1 | |
| 2 | .. c:namespace:: V4L
|
| 3 |
|
| 4 | .. _encoder:
|
| 5 |
|
| 6 | *************************************************
|
| 7 | Memory-to-Memory Stateful Video Encoder Interface
|
| 8 | *************************************************
|
| 9 |
|
| 10 | A stateful video encoder takes raw video frames in display order and encodes
|
| 11 | them into a bytestream. It generates complete chunks of the bytestream, including
|
| 12 | all metadata, headers, etc. The resulting bytestream does not require any
|
| 13 | further post-processing by the client.
|
| 14 |
|
| 15 | Performing software stream processing, header generation etc. in the driver
|
| 16 | in order to support this interface is strongly discouraged. In case such
|
| 17 | operations are needed, use of the Stateless Video Encoder Interface (in
|
| 18 | development) is strongly advised.
|
| 19 |
|
| 20 | Conventions and Notations Used in This Document
|
| 21 | ===============================================
|
| 22 |
|
| 23 | 1. The general V4L2 API rules apply if not specified in this document
|
| 24 | otherwise.
|
| 25 |
|
| 26 | 2. The meaning of words "must", "may", "should", etc. is as per `RFC
|
| 27 | 2119 <https://tools.ietf.org/html/rfc2119>`_.
|
| 28 |
|
| 29 | 3. All steps not marked "optional" are required.
|
| 30 |
|
| 31 | 4. :c:func:`VIDIOC_G_EXT_CTRLS` and :c:func:`VIDIOC_S_EXT_CTRLS` may be used
|
| 32 | interchangeably with :c:func:`VIDIOC_G_CTRL` and :c:func:`VIDIOC_S_CTRL`,
|
| 33 | unless specified otherwise.
|
| 34 |
|
| 35 | 5. Single-planar API (see :ref:`planar-apis`) and applicable structures may be
|
| 36 | used interchangeably with multi-planar API, unless specified otherwise,
|
| 37 | depending on encoder capabilities and following the general V4L2 guidelines.
|
| 38 |
|
| 39 | 6. i = [a..b]: sequence of integers from a to b, inclusive, i.e. i =
|
| 40 | [0..2]: i = 0, 1, 2.
|
| 41 |
|
| 42 | 7. Given an ``OUTPUT`` buffer A, then A' represents a buffer on the ``CAPTURE``
|
| 43 | queue containing data that resulted from processing buffer A.
|
| 44 |
|
| 45 | Glossary
|
| 46 | ========
|
| 47 |
|
| 48 | Refer to :ref:`decoder-glossary`.
|
| 49 |
|
| 50 | State Machine
|
| 51 | =============
|
| 52 |
|
| 53 | .. kernel-render:: DOT
|
| 54 | :alt: DOT digraph of encoder state machine
|
| 55 | :caption: Encoder State Machine
|
| 56 |
|
| 57 | digraph encoder_state_machine {
|
| 58 | node [shape = doublecircle, label="Encoding"] Encoding;
|
| 59 |
|
| 60 | node [shape = circle, label="Initialization"] Initialization;
|
| 61 | node [shape = circle, label="Stopped"] Stopped;
|
| 62 | node [shape = circle, label="Drain"] Drain;
|
| 63 | node [shape = circle, label="Reset"] Reset;
|
| 64 |
|
| 65 | node [shape = point]; qi
|
| 66 | qi -> Initialization [ label = "open()" ];
|
| 67 |
|
| 68 | Initialization -> Encoding [ label = "Both queues streaming" ];
|
| 69 |
|
| 70 | Encoding -> Drain [ label = "V4L2_ENC_CMD_STOP" ];
|
| 71 | Encoding -> Reset [ label = "VIDIOC_STREAMOFF(CAPTURE)" ];
|
| 72 | Encoding -> Stopped [ label = "VIDIOC_STREAMOFF(OUTPUT)" ];
|
| 73 | Encoding -> Encoding;
|
| 74 |
|
| 75 | Drain -> Stopped [ label = "All CAPTURE\nbuffers dequeued\nor\nVIDIOC_STREAMOFF(OUTPUT)" ];
|
| 76 | Drain -> Reset [ label = "VIDIOC_STREAMOFF(CAPTURE)" ];
|
| 77 |
|
| 78 | Reset -> Encoding [ label = "VIDIOC_STREAMON(CAPTURE)" ];
|
| 79 | Reset -> Initialization [ label = "VIDIOC_REQBUFS(OUTPUT, 0)" ];
|
| 80 |
|
| 81 | Stopped -> Encoding [ label = "V4L2_ENC_CMD_START\nor\nVIDIOC_STREAMON(OUTPUT)" ];
|
| 82 | Stopped -> Reset [ label = "VIDIOC_STREAMOFF(CAPTURE)" ];
|
| 83 | }
|
| 84 |
|
| 85 | Querying Capabilities
|
| 86 | =====================
|
| 87 |
|
| 88 | 1. To enumerate the set of coded formats supported by the encoder, the
|
| 89 | client may call :c:func:`VIDIOC_ENUM_FMT` on ``CAPTURE``.
|
| 90 |
|
| 91 | * The full set of supported formats will be returned, regardless of the
|
| 92 | format set on ``OUTPUT``.
|
| 93 |
|
| 94 | 2. To enumerate the set of supported raw formats, the client may call
|
| 95 | :c:func:`VIDIOC_ENUM_FMT` on ``OUTPUT``.
|
| 96 |
|
| 97 | * Only the formats supported for the format currently active on ``CAPTURE``
|
| 98 | will be returned.
|
| 99 |
|
| 100 | * In order to enumerate raw formats supported by a given coded format,
|
| 101 | the client must first set that coded format on ``CAPTURE`` and then
|
| 102 | enumerate the formats on ``OUTPUT``.
|
| 103 |
|
| 104 | 3. The client may use :c:func:`VIDIOC_ENUM_FRAMESIZES` to detect supported
|
| 105 | resolutions for a given format, passing the desired pixel format in
|
| 106 | :c:type:`v4l2_frmsizeenum` ``pixel_format``.
|
| 107 |
|
| 108 | * Values returned by :c:func:`VIDIOC_ENUM_FRAMESIZES` for a coded pixel
|
| 109 | format will include all possible coded resolutions supported by the
|
| 110 | encoder for the given coded pixel format.
|
| 111 |
|
| 112 | * Values returned by :c:func:`VIDIOC_ENUM_FRAMESIZES` for a raw pixel format
|
| 113 | will include all possible frame buffer resolutions supported by the
|
| 114 | encoder for the given raw pixel format and coded format currently set on
|
| 115 | ``CAPTURE``.
|
| 116 |
|
| 117 | 4. The client may use :c:func:`VIDIOC_ENUM_FRAMEINTERVALS` to detect supported
|
| 118 | frame intervals for a given format and resolution, passing the desired pixel
|
| 119 | format in :c:type:`v4l2_frmivalenum` ``pixel_format`` and the resolution
|
| 120 | in :c:type:`v4l2_frmivalenum` ``width`` and :c:type:`v4l2_frmivalenum`
|
| 121 | ``height``.
|
| 122 |
|
| 123 | * Values returned by :c:func:`VIDIOC_ENUM_FRAMEINTERVALS` for a coded pixel
|
| 124 | format and coded resolution will include all possible frame intervals
|
| 125 | supported by the encoder for the given coded pixel format and resolution.
|
| 126 |
|
| 127 | * Values returned by :c:func:`VIDIOC_ENUM_FRAMEINTERVALS` for a raw pixel
|
| 128 | format and resolution will include all possible frame intervals supported
|
| 129 | by the encoder for the given raw pixel format and resolution and for the
|
| 130 | coded format, coded resolution and coded frame interval currently set on
|
| 131 | ``CAPTURE``.
|
| 132 |
|
| 133 | * Support for :c:func:`VIDIOC_ENUM_FRAMEINTERVALS` is optional. If it is
|
| 134 | not implemented, then there are no special restrictions other than the
|
| 135 | limits of the codec itself.
|
| 136 |
|
| 137 | 5. Supported profiles and levels for the coded format currently set on
|
| 138 | ``CAPTURE``, if applicable, may be queried using their respective controls
|
| 139 | via :c:func:`VIDIOC_QUERYCTRL`.
|
| 140 |
|
| 141 | 6. Any additional encoder capabilities may be discovered by querying
|
| 142 | their respective controls.
|
| 143 |
|
| 144 | Initialization
|
| 145 | ==============
|
| 146 |
|
| 147 | 1. Set the coded format on the ``CAPTURE`` queue via :c:func:`VIDIOC_S_FMT`.
|
| 148 |
|
| 149 | * **Required fields:**
|
| 150 |
|
| 151 | ``type``
|
| 152 | a ``V4L2_BUF_TYPE_*`` enum appropriate for ``CAPTURE``.
|
| 153 |
|
| 154 | ``pixelformat``
|
| 155 | the coded format to be produced.
|
| 156 |
|
| 157 | ``sizeimage``
|
| 158 | desired size of ``CAPTURE`` buffers; the encoder may adjust it to
|
| 159 | match hardware requirements.
|
| 160 |
|
| 161 | ``width``, ``height``
|
| 162 | ignored (read-only).
|
| 163 |
|
| 164 | other fields
|
| 165 | follow standard semantics.
|
| 166 |
|
| 167 | * **Returned fields:**
|
| 168 |
|
| 169 | ``sizeimage``
|
| 170 | adjusted size of ``CAPTURE`` buffers.
|
| 171 |
|
| 172 | ``width``, ``height``
|
| 173 | the coded size selected by the encoder based on current state, e.g.
|
| 174 | ``OUTPUT`` format, selection rectangles, etc. (read-only).
|
| 175 |
|
| 176 | .. important::
|
| 177 |
|
| 178 | Changing the ``CAPTURE`` format may change the currently set ``OUTPUT``
|
| 179 | format. How the new ``OUTPUT`` format is determined is up to the encoder
|
| 180 | and the client must ensure it matches its needs afterwards.
|
| 181 |
|
| 182 | 2. **Optional.** Enumerate supported ``OUTPUT`` formats (raw formats for
|
| 183 | source) for the selected coded format via :c:func:`VIDIOC_ENUM_FMT`.
|
| 184 |
|
| 185 | * **Required fields:**
|
| 186 |
|
| 187 | ``type``
|
| 188 | a ``V4L2_BUF_TYPE_*`` enum appropriate for ``OUTPUT``.
|
| 189 |
|
| 190 | other fields
|
| 191 | follow standard semantics.
|
| 192 |
|
| 193 | * **Returned fields:**
|
| 194 |
|
| 195 | ``pixelformat``
|
| 196 | raw format supported for the coded format currently selected on
|
| 197 | the ``CAPTURE`` queue.
|
| 198 |
|
| 199 | other fields
|
| 200 | follow standard semantics.
|
| 201 |
|
| 202 | 3. Set the raw source format on the ``OUTPUT`` queue via
|
| 203 | :c:func:`VIDIOC_S_FMT`.
|
| 204 |
|
| 205 | * **Required fields:**
|
| 206 |
|
| 207 | ``type``
|
| 208 | a ``V4L2_BUF_TYPE_*`` enum appropriate for ``OUTPUT``.
|
| 209 |
|
| 210 | ``pixelformat``
|
| 211 | raw format of the source.
|
| 212 |
|
| 213 | ``width``, ``height``
|
| 214 | source resolution.
|
| 215 |
|
| 216 | other fields
|
| 217 | follow standard semantics.
|
| 218 |
|
| 219 | * **Returned fields:**
|
| 220 |
|
| 221 | ``width``, ``height``
|
| 222 | may be adjusted to match encoder minimums, maximums and alignment
|
| 223 | requirements, as required by the currently selected formats, as
|
| 224 | reported by :c:func:`VIDIOC_ENUM_FRAMESIZES`.
|
| 225 |
|
| 226 | other fields
|
| 227 | follow standard semantics.
|
| 228 |
|
| 229 | * Setting the ``OUTPUT`` format will reset the selection rectangles to their
|
| 230 | default values, based on the new resolution, as described in the next
|
| 231 | step.
|
| 232 |
|
| 233 | 4. Set the raw frame interval on the ``OUTPUT`` queue via
|
| 234 | :c:func:`VIDIOC_S_PARM`. This also sets the coded frame interval on the
|
| 235 | ``CAPTURE`` queue to the same value.
|
| 236 |
|
| 237 | * **Required fields:**
|
| 238 |
|
| 239 | ``type``
|
| 240 | a ``V4L2_BUF_TYPE_*`` enum appropriate for ``OUTPUT``.
|
| 241 |
|
| 242 | ``parm.output``
|
| 243 | set all fields except ``parm.output.timeperframe`` to 0.
|
| 244 |
|
| 245 | ``parm.output.timeperframe``
|
| 246 | the desired frame interval; the encoder may adjust it to
|
| 247 | match hardware requirements.
|
| 248 |
|
| 249 | * **Returned fields:**
|
| 250 |
|
| 251 | ``parm.output.timeperframe``
|
| 252 | the adjusted frame interval.
|
| 253 |
|
| 254 | .. important::
|
| 255 |
|
| 256 | Changing the ``OUTPUT`` frame interval *also* sets the framerate that
|
| 257 | the encoder uses to encode the video. So setting the frame interval
|
| 258 | to 1/24 (or 24 frames per second) will produce a coded video stream
|
| 259 | that can be played back at that speed. The frame interval for the
|
| 260 | ``OUTPUT`` queue is just a hint, the application may provide raw
|
| 261 | frames at a different rate. It can be used by the driver to help
|
| 262 | schedule multiple encoders running in parallel.
|
| 263 |
|
| 264 | In the next step the ``CAPTURE`` frame interval can optionally be
|
| 265 | changed to a different value. This is useful for off-line encoding
|
| 266 | were the coded frame interval can be different from the rate at
|
| 267 | which raw frames are supplied.
|
| 268 |
|
| 269 | .. important::
|
| 270 |
|
| 271 | ``timeperframe`` deals with *frames*, not fields. So for interlaced
|
| 272 | formats this is the time per two fields, since a frame consists of
|
| 273 | a top and a bottom field.
|
| 274 |
|
| 275 | .. note::
|
| 276 |
|
| 277 | It is due to historical reasons that changing the ``OUTPUT`` frame
|
| 278 | interval also changes the coded frame interval on the ``CAPTURE``
|
| 279 | queue. Ideally these would be independent settings, but that would
|
| 280 | break the existing API.
|
| 281 |
|
| 282 | 5. **Optional** Set the coded frame interval on the ``CAPTURE`` queue via
|
| 283 | :c:func:`VIDIOC_S_PARM`. This is only necessary if the coded frame
|
| 284 | interval is different from the raw frame interval, which is typically
|
| 285 | the case for off-line encoding. Support for this feature is signalled
|
| 286 | by the :ref:`V4L2_FMT_FLAG_ENC_CAP_FRAME_INTERVAL <fmtdesc-flags>` format flag.
|
| 287 |
|
| 288 | * **Required fields:**
|
| 289 |
|
| 290 | ``type``
|
| 291 | a ``V4L2_BUF_TYPE_*`` enum appropriate for ``CAPTURE``.
|
| 292 |
|
| 293 | ``parm.capture``
|
| 294 | set all fields except ``parm.capture.timeperframe`` to 0.
|
| 295 |
|
| 296 | ``parm.capture.timeperframe``
|
| 297 | the desired coded frame interval; the encoder may adjust it to
|
| 298 | match hardware requirements.
|
| 299 |
|
| 300 | * **Returned fields:**
|
| 301 |
|
| 302 | ``parm.capture.timeperframe``
|
| 303 | the adjusted frame interval.
|
| 304 |
|
| 305 | .. important::
|
| 306 |
|
| 307 | Changing the ``CAPTURE`` frame interval sets the framerate for the
|
| 308 | coded video. It does *not* set the rate at which buffers arrive on the
|
| 309 | ``CAPTURE`` queue, that depends on how fast the encoder is and how
|
| 310 | fast raw frames are queued on the ``OUTPUT`` queue.
|
| 311 |
|
| 312 | .. important::
|
| 313 |
|
| 314 | ``timeperframe`` deals with *frames*, not fields. So for interlaced
|
| 315 | formats this is the time per two fields, since a frame consists of
|
| 316 | a top and a bottom field.
|
| 317 |
|
| 318 | .. note::
|
| 319 |
|
| 320 | Not all drivers support this functionality, in that case just set
|
| 321 | the desired coded frame interval for the ``OUTPUT`` queue.
|
| 322 |
|
| 323 | However, drivers that can schedule multiple encoders based on the
|
| 324 | ``OUTPUT`` frame interval must support this optional feature.
|
| 325 |
|
| 326 | 6. **Optional.** Set the visible resolution for the stream metadata via
|
| 327 | :c:func:`VIDIOC_S_SELECTION` on the ``OUTPUT`` queue if it is desired
|
| 328 | to be different than the full OUTPUT resolution.
|
| 329 |
|
| 330 | * **Required fields:**
|
| 331 |
|
| 332 | ``type``
|
| 333 | a ``V4L2_BUF_TYPE_*`` enum appropriate for ``OUTPUT``.
|
| 334 |
|
| 335 | ``target``
|
| 336 | set to ``V4L2_SEL_TGT_CROP``.
|
| 337 |
|
| 338 | ``r.left``, ``r.top``, ``r.width``, ``r.height``
|
| 339 | visible rectangle; this must fit within the `V4L2_SEL_TGT_CROP_BOUNDS`
|
| 340 | rectangle and may be subject to adjustment to match codec and
|
| 341 | hardware constraints.
|
| 342 |
|
| 343 | * **Returned fields:**
|
| 344 |
|
| 345 | ``r.left``, ``r.top``, ``r.width``, ``r.height``
|
| 346 | visible rectangle adjusted by the encoder.
|
| 347 |
|
| 348 | * The following selection targets are supported on ``OUTPUT``:
|
| 349 |
|
| 350 | ``V4L2_SEL_TGT_CROP_BOUNDS``
|
| 351 | equal to the full source frame, matching the active ``OUTPUT``
|
| 352 | format.
|
| 353 |
|
| 354 | ``V4L2_SEL_TGT_CROP_DEFAULT``
|
| 355 | equal to ``V4L2_SEL_TGT_CROP_BOUNDS``.
|
| 356 |
|
| 357 | ``V4L2_SEL_TGT_CROP``
|
| 358 | rectangle within the source buffer to be encoded into the
|
| 359 | ``CAPTURE`` stream; defaults to ``V4L2_SEL_TGT_CROP_DEFAULT``.
|
| 360 |
|
| 361 | .. note::
|
| 362 |
|
| 363 | A common use case for this selection target is encoding a source
|
| 364 | video with a resolution that is not a multiple of a macroblock,
|
| 365 | e.g. the common 1920x1080 resolution may require the source
|
| 366 | buffers to be aligned to 1920x1088 for codecs with 16x16 macroblock
|
| 367 | size. To avoid encoding the padding, the client needs to explicitly
|
| 368 | configure this selection target to 1920x1080.
|
| 369 |
|
| 370 | .. warning::
|
| 371 |
|
| 372 | The encoder may adjust the crop/compose rectangles to the nearest
|
| 373 | supported ones to meet codec and hardware requirements. The client needs
|
| 374 | to check the adjusted rectangle returned by :c:func:`VIDIOC_S_SELECTION`.
|
| 375 |
|
| 376 | 7. Allocate buffers for both ``OUTPUT`` and ``CAPTURE`` via
|
| 377 | :c:func:`VIDIOC_REQBUFS`. This may be performed in any order.
|
| 378 |
|
| 379 | * **Required fields:**
|
| 380 |
|
| 381 | ``count``
|
| 382 | requested number of buffers to allocate; greater than zero.
|
| 383 |
|
| 384 | ``type``
|
| 385 | a ``V4L2_BUF_TYPE_*`` enum appropriate for ``OUTPUT`` or
|
| 386 | ``CAPTURE``.
|
| 387 |
|
| 388 | other fields
|
| 389 | follow standard semantics.
|
| 390 |
|
| 391 | * **Returned fields:**
|
| 392 |
|
| 393 | ``count``
|
| 394 | actual number of buffers allocated.
|
| 395 |
|
| 396 | .. warning::
|
| 397 |
|
| 398 | The actual number of allocated buffers may differ from the ``count``
|
| 399 | given. The client must check the updated value of ``count`` after the
|
| 400 | call returns.
|
| 401 |
|
| 402 | .. note::
|
| 403 |
|
| 404 | To allocate more than the minimum number of OUTPUT buffers (for pipeline
|
| 405 | depth), the client may query the ``V4L2_CID_MIN_BUFFERS_FOR_OUTPUT``
|
| 406 | control to get the minimum number of buffers required, and pass the
|
| 407 | obtained value plus the number of additional buffers needed in the
|
| 408 | ``count`` field to :c:func:`VIDIOC_REQBUFS`.
|
| 409 |
|
| 410 | Alternatively, :c:func:`VIDIOC_CREATE_BUFS` can be used to have more
|
| 411 | control over buffer allocation.
|
| 412 |
|
| 413 | * **Required fields:**
|
| 414 |
|
| 415 | ``count``
|
| 416 | requested number of buffers to allocate; greater than zero.
|
| 417 |
|
| 418 | ``type``
|
| 419 | a ``V4L2_BUF_TYPE_*`` enum appropriate for ``OUTPUT``.
|
| 420 |
|
| 421 | other fields
|
| 422 | follow standard semantics.
|
| 423 |
|
| 424 | * **Returned fields:**
|
| 425 |
|
| 426 | ``count``
|
| 427 | adjusted to the number of allocated buffers.
|
| 428 |
|
| 429 | 8. Begin streaming on both ``OUTPUT`` and ``CAPTURE`` queues via
|
| 430 | :c:func:`VIDIOC_STREAMON`. This may be performed in any order. The actual
|
| 431 | encoding process starts when both queues start streaming.
|
| 432 |
|
| 433 | .. note::
|
| 434 |
|
| 435 | If the client stops the ``CAPTURE`` queue during the encode process and then
|
| 436 | restarts it again, the encoder will begin generating a stream independent
|
| 437 | from the stream generated before the stop. The exact constraints depend
|
| 438 | on the coded format, but may include the following implications:
|
| 439 |
|
| 440 | * encoded frames produced after the restart must not reference any
|
| 441 | frames produced before the stop, e.g. no long term references for
|
| 442 | H.264/HEVC,
|
| 443 |
|
| 444 | * any headers that must be included in a standalone stream must be
|
| 445 | produced again, e.g. SPS and PPS for H.264/HEVC.
|
| 446 |
|
| 447 | Encoding
|
| 448 | ========
|
| 449 |
|
| 450 | This state is reached after the `Initialization` sequence finishes
|
| 451 | successfully. In this state, the client queues and dequeues buffers to both
|
| 452 | queues via :c:func:`VIDIOC_QBUF` and :c:func:`VIDIOC_DQBUF`, following the
|
| 453 | standard semantics.
|
| 454 |
|
| 455 | The content of encoded ``CAPTURE`` buffers depends on the active coded pixel
|
| 456 | format and may be affected by codec-specific extended controls, as stated
|
| 457 | in the documentation of each format.
|
| 458 |
|
| 459 | Both queues operate independently, following standard behavior of V4L2 buffer
|
| 460 | queues and memory-to-memory devices. In addition, the order of encoded frames
|
| 461 | dequeued from the ``CAPTURE`` queue may differ from the order of queuing raw
|
| 462 | frames to the ``OUTPUT`` queue, due to properties of the selected coded format,
|
| 463 | e.g. frame reordering.
|
| 464 |
|
| 465 | The client must not assume any direct relationship between ``CAPTURE`` and
|
| 466 | ``OUTPUT`` buffers and any specific timing of buffers becoming
|
| 467 | available to dequeue. Specifically:
|
| 468 |
|
| 469 | * a buffer queued to ``OUTPUT`` may result in more than one buffer produced on
|
| 470 | ``CAPTURE`` (for example, if returning an encoded frame allowed the encoder
|
| 471 | to return a frame that preceded it in display, but succeeded it in the decode
|
| 472 | order; however, there may be other reasons for this as well),
|
| 473 |
|
| 474 | * a buffer queued to ``OUTPUT`` may result in a buffer being produced on
|
| 475 | ``CAPTURE`` later into encode process, and/or after processing further
|
| 476 | ``OUTPUT`` buffers, or be returned out of order, e.g. if display
|
| 477 | reordering is used,
|
| 478 |
|
| 479 | * buffers may become available on the ``CAPTURE`` queue without additional
|
| 480 | buffers queued to ``OUTPUT`` (e.g. during drain or ``EOS``), because of the
|
| 481 | ``OUTPUT`` buffers queued in the past whose encoding results are only
|
| 482 | available at later time, due to specifics of the encoding process,
|
| 483 |
|
| 484 | * buffers queued to ``OUTPUT`` may not become available to dequeue instantly
|
| 485 | after being encoded into a corresponding ``CAPTURE`` buffer, e.g. if the
|
| 486 | encoder needs to use the frame as a reference for encoding further frames.
|
| 487 |
|
| 488 | .. note::
|
| 489 |
|
| 490 | To allow matching encoded ``CAPTURE`` buffers with ``OUTPUT`` buffers they
|
| 491 | originated from, the client can set the ``timestamp`` field of the
|
| 492 | :c:type:`v4l2_buffer` struct when queuing an ``OUTPUT`` buffer. The
|
| 493 | ``CAPTURE`` buffer(s), which resulted from encoding that ``OUTPUT`` buffer
|
| 494 | will have their ``timestamp`` field set to the same value when dequeued.
|
| 495 |
|
| 496 | In addition to the straightforward case of one ``OUTPUT`` buffer producing
|
| 497 | one ``CAPTURE`` buffer, the following cases are defined:
|
| 498 |
|
| 499 | * one ``OUTPUT`` buffer generates multiple ``CAPTURE`` buffers: the same
|
| 500 | ``OUTPUT`` timestamp will be copied to multiple ``CAPTURE`` buffers,
|
| 501 |
|
| 502 | * the encoding order differs from the presentation order (i.e. the
|
| 503 | ``CAPTURE`` buffers are out-of-order compared to the ``OUTPUT`` buffers):
|
| 504 | ``CAPTURE`` timestamps will not retain the order of ``OUTPUT`` timestamps.
|
| 505 |
|
| 506 | .. note::
|
| 507 |
|
| 508 | To let the client distinguish between frame types (keyframes, intermediate
|
| 509 | frames; the exact list of types depends on the coded format), the
|
| 510 | ``CAPTURE`` buffers will have corresponding flag bits set in their
|
| 511 | :c:type:`v4l2_buffer` struct when dequeued. See the documentation of
|
| 512 | :c:type:`v4l2_buffer` and each coded pixel format for exact list of flags
|
| 513 | and their meanings.
|
| 514 |
|
| 515 | Should an encoding error occur, it will be reported to the client with the level
|
| 516 | of details depending on the encoder capabilities. Specifically:
|
| 517 |
|
| 518 | * the ``CAPTURE`` buffer (if any) that contains the results of the failed encode
|
| 519 | operation will be returned with the ``V4L2_BUF_FLAG_ERROR`` flag set,
|
| 520 |
|
| 521 | * if the encoder is able to precisely report the ``OUTPUT`` buffer(s) that triggered
|
| 522 | the error, such buffer(s) will be returned with the ``V4L2_BUF_FLAG_ERROR`` flag
|
| 523 | set.
|
| 524 |
|
| 525 | .. note::
|
| 526 |
|
| 527 | If a ``CAPTURE`` buffer is too small then it is just returned with the
|
| 528 | ``V4L2_BUF_FLAG_ERROR`` flag set. More work is needed to detect that this
|
| 529 | error occurred because the buffer was too small, and to provide support to
|
| 530 | free existing buffers that were too small.
|
| 531 |
|
| 532 | In case of a fatal failure that does not allow the encoding to continue, any
|
| 533 | further operations on corresponding encoder file handle will return the -EIO
|
| 534 | error code. The client may close the file handle and open a new one, or
|
| 535 | alternatively reinitialize the instance by stopping streaming on both queues,
|
| 536 | releasing all buffers and performing the Initialization sequence again.
|
| 537 |
|
| 538 | Encoding Parameter Changes
|
| 539 | ==========================
|
| 540 |
|
| 541 | The client is allowed to use :c:func:`VIDIOC_S_CTRL` to change encoder
|
| 542 | parameters at any time. The availability of parameters is encoder-specific
|
| 543 | and the client must query the encoder to find the set of available controls.
|
| 544 |
|
| 545 | The ability to change each parameter during encoding is encoder-specific, as
|
| 546 | per the standard semantics of the V4L2 control interface. The client may
|
| 547 | attempt to set a control during encoding and if the operation fails with the
|
| 548 | -EBUSY error code, the ``CAPTURE`` queue needs to be stopped for the
|
| 549 | configuration change to be allowed. To do this, it may follow the `Drain`
|
| 550 | sequence to avoid losing the already queued/encoded frames.
|
| 551 |
|
| 552 | The timing of parameter updates is encoder-specific, as per the standard
|
| 553 | semantics of the V4L2 control interface. If the client needs to apply the
|
| 554 | parameters exactly at specific frame, using the Request API
|
| 555 | (:ref:`media-request-api`) should be considered, if supported by the encoder.
|
| 556 |
|
| 557 | Drain
|
| 558 | =====
|
| 559 |
|
| 560 | To ensure that all the queued ``OUTPUT`` buffers have been processed and the
|
| 561 | related ``CAPTURE`` buffers are given to the client, the client must follow the
|
| 562 | drain sequence described below. After the drain sequence ends, the client has
|
| 563 | received all encoded frames for all ``OUTPUT`` buffers queued before the
|
| 564 | sequence was started.
|
| 565 |
|
| 566 | 1. Begin the drain sequence by issuing :c:func:`VIDIOC_ENCODER_CMD`.
|
| 567 |
|
| 568 | * **Required fields:**
|
| 569 |
|
| 570 | ``cmd``
|
| 571 | set to ``V4L2_ENC_CMD_STOP``.
|
| 572 |
|
| 573 | ``flags``
|
| 574 | set to 0.
|
| 575 |
|
| 576 | ``pts``
|
| 577 | set to 0.
|
| 578 |
|
| 579 | .. warning::
|
| 580 |
|
| 581 | The sequence can be only initiated if both ``OUTPUT`` and ``CAPTURE``
|
| 582 | queues are streaming. For compatibility reasons, the call to
|
| 583 | :c:func:`VIDIOC_ENCODER_CMD` will not fail even if any of the queues is
|
| 584 | not streaming, but at the same time it will not initiate the `Drain`
|
| 585 | sequence and so the steps described below would not be applicable.
|
| 586 |
|
| 587 | 2. Any ``OUTPUT`` buffers queued by the client before the
|
| 588 | :c:func:`VIDIOC_ENCODER_CMD` was issued will be processed and encoded as
|
| 589 | normal. The client must continue to handle both queues independently,
|
| 590 | similarly to normal encode operation. This includes:
|
| 591 |
|
| 592 | * queuing and dequeuing ``CAPTURE`` buffers, until a buffer marked with the
|
| 593 | ``V4L2_BUF_FLAG_LAST`` flag is dequeued,
|
| 594 |
|
| 595 | .. warning::
|
| 596 |
|
| 597 | The last buffer may be empty (with :c:type:`v4l2_buffer`
|
| 598 | ``bytesused`` = 0) and in that case it must be ignored by the client,
|
| 599 | as it does not contain an encoded frame.
|
| 600 |
|
| 601 | .. note::
|
| 602 |
|
| 603 | Any attempt to dequeue more ``CAPTURE`` buffers beyond the buffer
|
| 604 | marked with ``V4L2_BUF_FLAG_LAST`` will result in a -EPIPE error from
|
| 605 | :c:func:`VIDIOC_DQBUF`.
|
| 606 |
|
| 607 | * dequeuing processed ``OUTPUT`` buffers, until all the buffers queued
|
| 608 | before the ``V4L2_ENC_CMD_STOP`` command are dequeued,
|
| 609 |
|
| 610 | * dequeuing the ``V4L2_EVENT_EOS`` event, if the client subscribes to it.
|
| 611 |
|
| 612 | .. note::
|
| 613 |
|
| 614 | For backwards compatibility, the encoder will signal a ``V4L2_EVENT_EOS``
|
| 615 | event when the last frame has been encoded and all frames are ready to be
|
| 616 | dequeued. It is deprecated behavior and the client must not rely on it.
|
| 617 | The ``V4L2_BUF_FLAG_LAST`` buffer flag should be used instead.
|
| 618 |
|
| 619 | 3. Once all ``OUTPUT`` buffers queued before the ``V4L2_ENC_CMD_STOP`` call are
|
| 620 | dequeued and the last ``CAPTURE`` buffer is dequeued, the encoder is stopped
|
| 621 | and it will accept, but not process any newly queued ``OUTPUT`` buffers
|
| 622 | until the client issues any of the following operations:
|
| 623 |
|
| 624 | * ``V4L2_ENC_CMD_START`` - the encoder will not be reset and will resume
|
| 625 | operation normally, with all the state from before the drain,
|
| 626 |
|
| 627 | * a pair of :c:func:`VIDIOC_STREAMOFF` and :c:func:`VIDIOC_STREAMON` on the
|
| 628 | ``CAPTURE`` queue - the encoder will be reset (see the `Reset` sequence)
|
| 629 | and then resume encoding,
|
| 630 |
|
| 631 | * a pair of :c:func:`VIDIOC_STREAMOFF` and :c:func:`VIDIOC_STREAMON` on the
|
| 632 | ``OUTPUT`` queue - the encoder will resume operation normally, however any
|
| 633 | source frames queued to the ``OUTPUT`` queue between ``V4L2_ENC_CMD_STOP``
|
| 634 | and :c:func:`VIDIOC_STREAMOFF` will be discarded.
|
| 635 |
|
| 636 | .. note::
|
| 637 |
|
| 638 | Once the drain sequence is initiated, the client needs to drive it to
|
| 639 | completion, as described by the steps above, unless it aborts the process by
|
| 640 | issuing :c:func:`VIDIOC_STREAMOFF` on any of the ``OUTPUT`` or ``CAPTURE``
|
| 641 | queues. The client is not allowed to issue ``V4L2_ENC_CMD_START`` or
|
| 642 | ``V4L2_ENC_CMD_STOP`` again while the drain sequence is in progress and they
|
| 643 | will fail with -EBUSY error code if attempted.
|
| 644 |
|
| 645 | For reference, handling of various corner cases is described below:
|
| 646 |
|
| 647 | * In case of no buffer in the ``OUTPUT`` queue at the time the
|
| 648 | ``V4L2_ENC_CMD_STOP`` command was issued, the drain sequence completes
|
| 649 | immediately and the encoder returns an empty ``CAPTURE`` buffer with the
|
| 650 | ``V4L2_BUF_FLAG_LAST`` flag set.
|
| 651 |
|
| 652 | * In case of no buffer in the ``CAPTURE`` queue at the time the drain
|
| 653 | sequence completes, the next time the client queues a ``CAPTURE`` buffer
|
| 654 | it is returned at once as an empty buffer with the ``V4L2_BUF_FLAG_LAST``
|
| 655 | flag set.
|
| 656 |
|
| 657 | * If :c:func:`VIDIOC_STREAMOFF` is called on the ``CAPTURE`` queue in the
|
| 658 | middle of the drain sequence, the drain sequence is canceled and all
|
| 659 | ``CAPTURE`` buffers are implicitly returned to the client.
|
| 660 |
|
| 661 | * If :c:func:`VIDIOC_STREAMOFF` is called on the ``OUTPUT`` queue in the
|
| 662 | middle of the drain sequence, the drain sequence completes immediately and
|
| 663 | next ``CAPTURE`` buffer will be returned empty with the
|
| 664 | ``V4L2_BUF_FLAG_LAST`` flag set.
|
| 665 |
|
| 666 | Although not mandatory, the availability of encoder commands may be queried
|
| 667 | using :c:func:`VIDIOC_TRY_ENCODER_CMD`.
|
| 668 |
|
| 669 | Reset
|
| 670 | =====
|
| 671 |
|
| 672 | The client may want to request the encoder to reinitialize the encoding, so
|
| 673 | that the following stream data becomes independent from the stream data
|
| 674 | generated before. Depending on the coded format, that may imply that:
|
| 675 |
|
| 676 | * encoded frames produced after the restart must not reference any frames
|
| 677 | produced before the stop, e.g. no long term references for H.264/HEVC,
|
| 678 |
|
| 679 | * any headers that must be included in a standalone stream must be produced
|
| 680 | again, e.g. SPS and PPS for H.264/HEVC.
|
| 681 |
|
| 682 | This can be achieved by performing the reset sequence.
|
| 683 |
|
| 684 | 1. Perform the `Drain` sequence to ensure all the in-flight encoding finishes
|
| 685 | and respective buffers are dequeued.
|
| 686 |
|
| 687 | 2. Stop streaming on the ``CAPTURE`` queue via :c:func:`VIDIOC_STREAMOFF`. This
|
| 688 | will return all currently queued ``CAPTURE`` buffers to the client, without
|
| 689 | valid frame data.
|
| 690 |
|
| 691 | 3. Start streaming on the ``CAPTURE`` queue via :c:func:`VIDIOC_STREAMON` and
|
| 692 | continue with regular encoding sequence. The encoded frames produced into
|
| 693 | ``CAPTURE`` buffers from now on will contain a standalone stream that can be
|
| 694 | decoded without the need for frames encoded before the reset sequence,
|
| 695 | starting at the first ``OUTPUT`` buffer queued after issuing the
|
| 696 | `V4L2_ENC_CMD_STOP` of the `Drain` sequence.
|
| 697 |
|
| 698 | This sequence may be also used to change encoding parameters for encoders
|
| 699 | without the ability to change the parameters on the fly.
|
| 700 |
|
| 701 | Commit Points
|
| 702 | =============
|
| 703 |
|
| 704 | Setting formats and allocating buffers triggers changes in the behavior of the
|
| 705 | encoder.
|
| 706 |
|
| 707 | 1. Setting the format on the ``CAPTURE`` queue may change the set of formats
|
| 708 | supported/advertised on the ``OUTPUT`` queue. In particular, it also means
|
| 709 | that the ``OUTPUT`` format may be reset and the client must not rely on the
|
| 710 | previously set format being preserved.
|
| 711 |
|
| 712 | 2. Enumerating formats on the ``OUTPUT`` queue always returns only formats
|
| 713 | supported for the current ``CAPTURE`` format.
|
| 714 |
|
| 715 | 3. Setting the format on the ``OUTPUT`` queue does not change the list of
|
| 716 | formats available on the ``CAPTURE`` queue. An attempt to set the ``OUTPUT``
|
| 717 | format that is not supported for the currently selected ``CAPTURE`` format
|
| 718 | will result in the encoder adjusting the requested ``OUTPUT`` format to a
|
| 719 | supported one.
|
| 720 |
|
| 721 | 4. Enumerating formats on the ``CAPTURE`` queue always returns the full set of
|
| 722 | supported coded formats, irrespective of the current ``OUTPUT`` format.
|
| 723 |
|
| 724 | 5. While buffers are allocated on any of the ``OUTPUT`` or ``CAPTURE`` queues,
|
| 725 | the client must not change the format on the ``CAPTURE`` queue. Drivers will
|
| 726 | return the -EBUSY error code for any such format change attempt.
|
| 727 |
|
| 728 | To summarize, setting formats and allocation must always start with the
|
| 729 | ``CAPTURE`` queue and the ``CAPTURE`` queue is the master that governs the
|
| 730 | set of supported formats for the ``OUTPUT`` queue.
|