693 lines · 26.7 KB · Text
RawDownload
.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later.. c:namespace:: V4L .. _subdev: ********************Sub-device Interface******************** The complex nature of V4L2 devices, where hardware is often made ofseveral integrated circuits that need to interact with each other in acontrolled way, leads to complex V4L2 drivers. The drivers usuallyreflect the hardware model in software, and model the different hardwarecomponents as software blocks called sub-devices. V4L2 sub-devices are usually kernel-only objects. If the V4L2 driverimplements the media device API, they will automatically inherit frommedia entities. Applications will be able to enumerate the sub-devicesand discover the hardware topology using the media entities, pads andlinks enumeration API. In addition to make sub-devices discoverable, drivers can also choose tomake them directly configurable by applications. When both thesub-device driver and the V4L2 device driver support this, sub-deviceswill feature a character device node on which ioctls can be called to -  query, read and write sub-devices controls -  subscribe and unsubscribe to events and retrieve them -  negotiate image formats on individual pads -  inspect and modify internal data routing between pads of the same entity Sub-device character device nodes, conventionally named``/dev/v4l-subdev*``, use major number 81. Drivers may opt to limit the sub-device character devices to only exposeoperations that do not modify the device state. In such a case the sub-devicesare referred to as ``read-only`` in the rest of this documentation, and therelated restrictions are documented in individual ioctls.  Controls======== Most V4L2 controls are implemented by sub-device hardware. Driversusually merge all controls and expose them through video device nodes.Applications can control all sub-devices through a single interface. Complex devices sometimes implement the same control in different piecesof hardware. This situation is common in embedded platforms, where bothsensors and image processing hardware implement identical functions,such as contrast adjustment, white balance or faulty pixels correction.As the V4L2 controls API doesn't support several identical controls in asingle device, all but one of the identical controls are hidden. Applications can access those hidden controls through the sub-devicenode with the V4L2 control API described in :ref:`control`. The ioctlsbehave identically as when issued on V4L2 device nodes, with theexception that they deal only with controls implemented in thesub-device. Depending on the driver, those controls might also be exposed throughone (or several) V4L2 device nodes.  Events====== V4L2 sub-devices can notify applications of events as described in:ref:`event`. The API behaves identically as when used on V4L2 devicenodes, with the exception that it only deals with events generated bythe sub-device. Depending on the driver, those events might also bereported on one (or several) V4L2 device nodes.  .. _pad-level-formats: Pad-level Formats================= .. warning::     Pad-level formats are only applicable to very complex devices that    need to expose low-level format configuration to user space. Generic    V4L2 applications do *not* need to use the API described in this    section. .. note::     For the purpose of this section, the term *format* means the    combination of media bus data format, frame width and frame height. Image formats are typically negotiated on video capture and outputdevices using the format and:ref:`selection <VIDIOC_SUBDEV_G_SELECTION>` ioctls. The driver isresponsible for configuring every block in the video pipeline accordingto the requested format at the pipeline input and/or output. For complex devices, such as often found in embedded systems, identicalimage sizes at the output of a pipeline can be achieved using differenthardware configurations. One such example is shown on:ref:`pipeline-scaling`, where image scaling can be performed on boththe video sensor and the host image processing hardware.  .. _pipeline-scaling: .. kernel-figure:: pipeline.dot    :alt:   pipeline.dot    :align: center     Image Format Negotiation on Pipelines     High quality and high speed pipeline configuration   The sensor scaler is usually of less quality than the host scaler, butscaling on the sensor is required to achieve higher frame rates.Depending on the use case (quality vs. speed), the pipeline must beconfigured differently. Applications need to configure the formats atevery point in the pipeline explicitly. Drivers that implement the :ref:`media API <media-controller-intro>`can expose pad-level image format configuration to applications. Whenthey do, applications can use the:ref:`VIDIOC_SUBDEV_G_FMT <VIDIOC_SUBDEV_G_FMT>` and:ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls. tonegotiate formats on a per-pad basis. Applications are responsible for configuring coherent parameters on thewhole pipeline and making sure that connected pads have compatibleformats. The pipeline is checked for formats mismatch at:ref:`VIDIOC_STREAMON <VIDIOC_STREAMON>` time, and an ``EPIPE`` errorcode is then returned if the configuration is invalid. Pad-level image format configuration support can be tested by callingthe :ref:`VIDIOC_SUBDEV_G_FMT` ioctl on pad0. If the driver returns an ``EINVAL`` error code pad-level formatconfiguration is not supported by the sub-device.  Format Negotiation------------------ Acceptable formats on pads can (and usually do) depend on a number ofexternal parameters, such as formats on other pads, active links, oreven controls. Finding a combination of formats on all pads in a videopipeline, acceptable to both application and driver, can't rely onformats enumeration only. A format negotiation mechanism is required. Central to the format negotiation mechanism are the get/set formatoperations. When called with the ``which`` argument set to:ref:`V4L2_SUBDEV_FORMAT_TRY <VIDIOC_SUBDEV_G_FMT>`, the:ref:`VIDIOC_SUBDEV_G_FMT <VIDIOC_SUBDEV_G_FMT>` and:ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls operate ona set of formats parameters that are not connected to the hardwareconfiguration. Modifying those 'try' formats leaves the device stateuntouched (this applies to both the software state stored in the driverand the hardware state stored in the device itself). While not kept as part of the device state, try formats are stored inthe sub-device file handles. A:ref:`VIDIOC_SUBDEV_G_FMT <VIDIOC_SUBDEV_G_FMT>` call will returnthe last try format set *on the same sub-device file handle*. Severalapplications querying the same sub-device at the same time will thus notinteract with each other. To find out whether a particular format is supported by the device,applications use the:ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctl. Driversverify and, if needed, change the requested ``format`` based on devicerequirements and return the possibly modified value. Applications canthen choose to try a different format or accept the returned value andcontinue. Formats returned by the driver during a negotiation iteration areguaranteed to be supported by the device. In particular, driversguarantee that a returned format will not be further changed if passedto an :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` call as-is(as long as external parameters, such as formats on other pads or links'configuration are not changed). Drivers automatically propagate formats inside sub-devices. When a tryor active format is set on a pad, corresponding formats on other pads ofthe same sub-device can be modified by the driver. Drivers are free tomodify formats as required by the device. However, they should complywith the following rules when possible: -  Formats should be propagated from sink pads to source pads. Modifying   a format on a source pad should not modify the format on any sink   pad. -  Sub-devices that scale frames using variable scaling factors should   reset the scale factors to default values when sink pads formats are   modified. If the 1:1 scaling ratio is supported, this means that   source pads formats should be reset to the sink pads formats. Formats are not propagated across links, as that would involvepropagating them from one sub-device file handle to another.Applications must then take care to configure both ends of every linkexplicitly with compatible formats. Identical formats on the two ends ofa link are guaranteed to be compatible. Drivers are free to acceptdifferent formats matching device requirements as being compatible. :ref:`sample-pipeline-config` shows a sample configuration sequencefor the pipeline described in :ref:`pipeline-scaling` (table columnslist entity names and pad numbers).  .. raw:: latex     \begingroup    \scriptsize    \setlength{\tabcolsep}{2pt} .. tabularcolumns:: |p{2.0cm}|p{2.1cm}|p{2.1cm}|p{2.1cm}|p{2.1cm}|p{2.1cm}|p{2.1cm}| .. _sample-pipeline-config: .. flat-table:: Sample Pipeline Configuration    :header-rows:  1    :stub-columns: 0    :widths: 5 5 5 5 5 5 5     * -      - Sensor/0         format      - Frontend/0         format      - Frontend/1         format      - Scaler/0         format      - Scaler/0         compose selection rectangle      - Scaler/1         format    * - Initial state      - 2048x1536         SGRBG8_1X8      - (default)      - (default)      - (default)      - (default)      - (default)    * - Configure frontend sink format      - 2048x1536         SGRBG8_1X8      - *2048x1536*         *SGRBG8_1X8*      - *2046x1534*         *SGRBG8_1X8*      - (default)      - (default)      - (default)    * - Configure scaler sink format      - 2048x1536         SGRBG8_1X8      - 2048x1536         SGRBG8_1X8      - 2046x1534         SGRBG8_1X8      - *2046x1534*         *SGRBG8_1X8*      - *0,0/2046x1534*      - *2046x1534*         *SGRBG8_1X8*    * - Configure scaler sink compose selection      - 2048x1536         SGRBG8_1X8      - 2048x1536         SGRBG8_1X8      - 2046x1534         SGRBG8_1X8      - 2046x1534         SGRBG8_1X8      - *0,0/1280x960*      - *1280x960*         *SGRBG8_1X8* .. raw:: latex     \endgroup 1. Initial state. The sensor source pad format is set to its native 3MP   size and V4L2_MBUS_FMT_SGRBG8_1X8 media bus code. Formats on the   host frontend and scaler sink and source pads have the default   values, as well as the compose rectangle on the scaler's sink pad. 2. The application configures the frontend sink pad format's size to   2048x1536 and its media bus code to V4L2_MBUS_FMT_SGRBG_1X8. The   driver propagates the format to the frontend source pad. 3. The application configures the scaler sink pad format's size to   2046x1534 and the media bus code to V4L2_MBUS_FMT_SGRBG_1X8 to   match the frontend source size and media bus code. The media bus code   on the sink pad is set to V4L2_MBUS_FMT_SGRBG_1X8. The driver   propagates the size to the compose selection rectangle on the   scaler's sink pad, and the format to the scaler source pad. 4. The application configures the size of the compose selection   rectangle of the scaler's sink pad 1280x960. The driver propagates   the size to the scaler's source pad format. When satisfied with the try results, applications can set the activeformats by setting the ``which`` argument to``V4L2_SUBDEV_FORMAT_ACTIVE``. Active formats are changed exactly as tryformats by drivers. To avoid modifying the hardware state during formatnegotiation, applications should negotiate try formats first and thenmodify the active settings using the try formats returned during thelast negotiation iteration. This guarantees that the active format willbe applied as-is by the driver without being modified.  .. _v4l2-subdev-selections: Selections: cropping, scaling and composition--------------------------------------------- Many sub-devices support cropping frames on their input or output pads(or possible even on both). Cropping is used to select the area ofinterest in an image, typically on an image sensor or a video decoder.It can also be used as part of digital zoom implementations to selectthe area of the image that will be scaled up. Crop settings are defined by a crop rectangle and represented in astruct :c:type:`v4l2_rect` by the coordinates of the topleft corner and the rectangle size. Both the coordinates and sizes areexpressed in pixels. As for pad formats, drivers store try and active rectangles for theselection targets :ref:`v4l2-selections-common`. On sink pads, cropping is applied relative to the current pad format.The pad format represents the image size as received by the sub-devicefrom the previous block in the pipeline, and the crop rectanglerepresents the sub-image that will be transmitted further inside thesub-device for processing. The scaling operation changes the size of the image by scaling it to newdimensions. The scaling ratio isn't specified explicitly, but is impliedfrom the original and scaled image sizes. Both sizes are represented bystruct :c:type:`v4l2_rect`. Scaling support is optional. When supported by a subdev, the croprectangle on the subdev's sink pad is scaled to the size configuredusing the:ref:`VIDIOC_SUBDEV_S_SELECTION <VIDIOC_SUBDEV_G_SELECTION>` IOCTLusing ``V4L2_SEL_TGT_COMPOSE`` selection target on the same pad. If thesubdev supports scaling but not composing, the top and left values arenot used and must always be set to zero. On source pads, cropping is similar to sink pads, with the exceptionthat the source size from which the cropping is performed, is theCOMPOSE rectangle on the sink pad. In both sink and source pads, thecrop rectangle must be entirely contained inside the source image sizefor the crop operation. The drivers should always use the closest possible rectangle the userrequests on all selection targets, unless specifically told otherwise.``V4L2_SEL_FLAG_GE`` and ``V4L2_SEL_FLAG_LE`` flags may be used to roundthe image size either up or down. :ref:`v4l2-selection-flags`  Types of selection targets--------------------------  Actual targets^^^^^^^^^^^^^^ Actual targets (without a postfix) reflect the actual hardwareconfiguration at any point of time. There is a BOUNDS targetcorresponding to every actual target.  BOUNDS targets^^^^^^^^^^^^^^ BOUNDS targets is the smallest rectangle that contains all valid actualrectangles. It may not be possible to set the actual rectangle as largeas the BOUNDS rectangle, however. This may be because e.g. a sensor'spixel array is not rectangular but cross-shaped or round. The maximumsize may also be smaller than the BOUNDS rectangle.  .. _format-propagation: Order of configuration and format propagation--------------------------------------------- Inside subdevs, the order of image processing steps will always be fromthe sink pad towards the source pad. This is also reflected in the orderin which the configuration must be performed by the user: the changesmade will be propagated to any subsequent stages. If this behaviour isnot desired, the user must set ``V4L2_SEL_FLAG_KEEP_CONFIG`` flag. Thisflag causes no propagation of the changes are allowed in anycircumstances. This may also cause the accessed rectangle to be adjustedby the driver, depending on the properties of the underlying hardware. The coordinates to a step always refer to the actual size of theprevious step. The exception to this rule is the sink composerectangle, which refers to the sink compose bounds rectangle --- if itis supported by the hardware. 1. Sink pad format. The user configures the sink pad format. This format   defines the parameters of the image the entity receives through the   pad for further processing. 2. Sink pad actual crop selection. The sink pad crop defines the crop   performed to the sink pad format. 3. Sink pad actual compose selection. The size of the sink pad compose   rectangle defines the scaling ratio compared to the size of the sink   pad crop rectangle. The location of the compose rectangle specifies   the location of the actual sink compose rectangle in the sink compose   bounds rectangle. 4. Source pad actual crop selection. Crop on the source pad defines crop   performed to the image in the sink compose bounds rectangle. 5. Source pad format. The source pad format defines the output pixel   format of the subdev, as well as the other parameters with the   exception of the image width and height. Width and height are defined   by the size of the source pad actual crop selection. Accessing any of the above rectangles not supported by the subdev willreturn ``EINVAL``. Any rectangle referring to a previous unsupportedrectangle coordinates will instead refer to the previous supportedrectangle. For example, if sink crop is not supported, the composeselection will refer to the sink pad format dimensions instead.  .. _subdev-image-processing-crop: .. kernel-figure:: subdev-image-processing-crop.svg    :alt:   subdev-image-processing-crop.svg    :align: center     Image processing in subdevs: simple crop example In the above example, the subdev supports cropping on its sink pad. Toconfigure it, the user sets the media bus format on the subdev's sinkpad. Now the actual crop rectangle can be set on the sink pad --- thelocation and size of this rectangle reflect the location and size of arectangle to be cropped from the sink format. The size of the sink croprectangle will also be the size of the format of the subdev's sourcepad.  .. _subdev-image-processing-scaling-multi-source: .. kernel-figure:: subdev-image-processing-scaling-multi-source.svg    :alt:   subdev-image-processing-scaling-multi-source.svg    :align: center     Image processing in subdevs: scaling with multiple sources In this example, the subdev is capable of first cropping, then scalingand finally cropping for two source pads individually from the resultingscaled image. The location of the scaled image in the cropped image isignored in sink compose target. Both of the locations of the source croprectangles refer to the sink scaling rectangle, independently croppingan area at location specified by the source crop rectangle from it.  .. _subdev-image-processing-full: .. kernel-figure:: subdev-image-processing-full.svg    :alt:    subdev-image-processing-full.svg    :align:  center     Image processing in subdevs: scaling and composition with multiple sinks and sources The subdev driver supports two sink pads and two source pads. The imagesfrom both of the sink pads are individually cropped, then scaled andfurther composed on the composition bounds rectangle. From that, two

Lines 1–500 of 693.

Keyboard shortcuts

General

Show keyboard shortcuts
?
Focus the current search
/

Issues and pull requests

Open filters
f

Issues, pull requests and Inbox

Select next item
j↓
Select previous item
k↑
Open selected item
Enter
Dismiss filters or clear selection
Esc

Repository code

Go to file
t