Commit | Line | Data |
---|---|---|
333c8b97 LP |
1 | <title>Sub-device Interface</title> |
2 | ||
3 | <note> | |
4 | <title>Experimental</title> | |
5 | <para>This is an <link linkend="experimental">experimental</link> | |
6 | interface and may change in the future.</para> | |
7 | </note> | |
8 | ||
9 | <para>The complex nature of V4L2 devices, where hardware is often made of | |
10 | several integrated circuits that need to interact with each other in a | |
11 | controlled way, leads to complex V4L2 drivers. The drivers usually reflect | |
12 | the hardware model in software, and model the different hardware components | |
13 | as software blocks called sub-devices.</para> | |
14 | ||
15 | <para>V4L2 sub-devices are usually kernel-only objects. If the V4L2 driver | |
16 | implements the media device API, they will automatically inherit from media | |
17 | entities. Applications will be able to enumerate the sub-devices and discover | |
18 | the hardware topology using the media entities, pads and links enumeration | |
19 | API.</para> | |
20 | ||
21 | <para>In addition to make sub-devices discoverable, drivers can also choose | |
22 | to make them directly configurable by applications. When both the sub-device | |
23 | driver and the V4L2 device driver support this, sub-devices will feature a | |
24 | character device node on which ioctls can be called to | |
25 | <itemizedlist> | |
665bf368 HV |
26 | <listitem><para>query, read and write sub-devices controls</para></listitem> |
27 | <listitem><para>subscribe and unsubscribe to events and retrieve them</para></listitem> | |
28 | <listitem><para>negotiate image formats on individual pads</para></listitem> | |
333c8b97 LP |
29 | </itemizedlist> |
30 | </para> | |
31 | ||
32 | <para>Sub-device character device nodes, conventionally named | |
33 | <filename>/dev/v4l-subdev*</filename>, use major number 81.</para> | |
34 | ||
35 | <section> | |
36 | <title>Controls</title> | |
37 | <para>Most V4L2 controls are implemented by sub-device hardware. Drivers | |
38 | usually merge all controls and expose them through video device nodes. | |
39 | Applications can control all sub-devices through a single interface.</para> | |
40 | ||
41 | <para>Complex devices sometimes implement the same control in different | |
42 | pieces of hardware. This situation is common in embedded platforms, where | |
43 | both sensors and image processing hardware implement identical functions, | |
44 | such as contrast adjustment, white balance or faulty pixels correction. As | |
45 | the V4L2 controls API doesn't support several identical controls in a single | |
46 | device, all but one of the identical controls are hidden.</para> | |
47 | ||
48 | <para>Applications can access those hidden controls through the sub-device | |
49 | node with the V4L2 control API described in <xref linkend="control" />. The | |
50 | ioctls behave identically as when issued on V4L2 device nodes, with the | |
51 | exception that they deal only with controls implemented in the sub-device. | |
52 | </para> | |
53 | ||
54 | <para>Depending on the driver, those controls might also be exposed through | |
55 | one (or several) V4L2 device nodes.</para> | |
56 | </section> | |
57 | ||
58 | <section> | |
59 | <title>Events</title> | |
60 | <para>V4L2 sub-devices can notify applications of events as described in | |
61 | <xref linkend="event" />. The API behaves identically as when used on V4L2 | |
62 | device nodes, with the exception that it only deals with events generated by | |
63 | the sub-device. Depending on the driver, those events might also be reported | |
64 | on one (or several) V4L2 device nodes.</para> | |
65 | </section> | |
66 | ||
67 | <section id="pad-level-formats"> | |
68 | <title>Pad-level Formats</title> | |
69 | ||
665bf368 | 70 | <warning><para>Pad-level formats are only applicable to very complex device that |
333c8b97 LP |
71 | need to expose low-level format configuration to user space. Generic V4L2 |
72 | applications do <emphasis>not</emphasis> need to use the API described in | |
665bf368 | 73 | this section.</para></warning> |
333c8b97 | 74 | |
665bf368 | 75 | <note><para>For the purpose of this section, the term |
333c8b97 | 76 | <wordasword>format</wordasword> means the combination of media bus data |
665bf368 | 77 | format, frame width and frame height.</para></note> |
333c8b97 | 78 | |
955f645a SA |
79 | <para>Image formats are typically negotiated on video capture and |
80 | output devices using the format and <link | |
81 | linkend="vidioc-subdev-g-selection">selection</link> ioctls. The | |
82 | driver is responsible for configuring every block in the video | |
83 | pipeline according to the requested format at the pipeline input | |
84 | and/or output.</para> | |
333c8b97 LP |
85 | |
86 | <para>For complex devices, such as often found in embedded systems, | |
87 | identical image sizes at the output of a pipeline can be achieved using | |
665bf368 HV |
88 | different hardware configurations. One such example is shown on |
89 | <xref linkend="pipeline-scaling" />, where | |
333c8b97 LP |
90 | image scaling can be performed on both the video sensor and the host image |
91 | processing hardware.</para> | |
92 | ||
93 | <figure id="pipeline-scaling"> | |
25985edc | 94 | <title>Image Format Negotiation on Pipelines</title> |
333c8b97 LP |
95 | <mediaobject> |
96 | <imageobject> | |
97 | <imagedata fileref="pipeline.pdf" format="PS" /> | |
98 | </imageobject> | |
99 | <imageobject> | |
100 | <imagedata fileref="pipeline.png" format="PNG" /> | |
101 | </imageobject> | |
102 | <textobject> | |
103 | <phrase>High quality and high speed pipeline configuration</phrase> | |
104 | </textobject> | |
105 | </mediaobject> | |
106 | </figure> | |
107 | ||
108 | <para>The sensor scaler is usually of less quality than the host scaler, but | |
109 | scaling on the sensor is required to achieve higher frame rates. Depending | |
110 | on the use case (quality vs. speed), the pipeline must be configured | |
111 | differently. Applications need to configure the formats at every point in | |
112 | the pipeline explicitly.</para> | |
113 | ||
114 | <para>Drivers that implement the <link linkend="media-controller-intro">media | |
115 | API</link> can expose pad-level image format configuration to applications. | |
116 | When they do, applications can use the &VIDIOC-SUBDEV-G-FMT; and | |
117 | &VIDIOC-SUBDEV-S-FMT; ioctls. to negotiate formats on a per-pad basis.</para> | |
118 | ||
119 | <para>Applications are responsible for configuring coherent parameters on | |
120 | the whole pipeline and making sure that connected pads have compatible | |
121 | formats. The pipeline is checked for formats mismatch at &VIDIOC-STREAMON; | |
122 | time, and an &EPIPE; is then returned if the configuration is | |
123 | invalid.</para> | |
124 | ||
125 | <para>Pad-level image format configuration support can be tested by calling | |
126 | the &VIDIOC-SUBDEV-G-FMT; ioctl on pad 0. If the driver returns an &EINVAL; | |
127 | pad-level format configuration is not supported by the sub-device.</para> | |
128 | ||
129 | <section> | |
130 | <title>Format Negotiation</title> | |
131 | ||
132 | <para>Acceptable formats on pads can (and usually do) depend on a number | |
133 | of external parameters, such as formats on other pads, active links, or | |
134 | even controls. Finding a combination of formats on all pads in a video | |
135 | pipeline, acceptable to both application and driver, can't rely on formats | |
136 | enumeration only. A format negotiation mechanism is required.</para> | |
137 | ||
138 | <para>Central to the format negotiation mechanism are the get/set format | |
139 | operations. When called with the <structfield>which</structfield> argument | |
140 | set to <constant>V4L2_SUBDEV_FORMAT_TRY</constant>, the | |
141 | &VIDIOC-SUBDEV-G-FMT; and &VIDIOC-SUBDEV-S-FMT; ioctls operate on a set of | |
142 | formats parameters that are not connected to the hardware configuration. | |
143 | Modifying those 'try' formats leaves the device state untouched (this | |
144 | applies to both the software state stored in the driver and the hardware | |
145 | state stored in the device itself).</para> | |
146 | ||
147 | <para>While not kept as part of the device state, try formats are stored | |
148 | in the sub-device file handles. A &VIDIOC-SUBDEV-G-FMT; call will return | |
149 | the last try format set <emphasis>on the same sub-device file | |
150 | handle</emphasis>. Several applications querying the same sub-device at | |
151 | the same time will thus not interact with each other.</para> | |
152 | ||
153 | <para>To find out whether a particular format is supported by the device, | |
154 | applications use the &VIDIOC-SUBDEV-S-FMT; ioctl. Drivers verify and, if | |
155 | needed, change the requested <structfield>format</structfield> based on | |
156 | device requirements and return the possibly modified value. Applications | |
157 | can then choose to try a different format or accept the returned value and | |
158 | continue.</para> | |
159 | ||
160 | <para>Formats returned by the driver during a negotiation iteration are | |
161 | guaranteed to be supported by the device. In particular, drivers guarantee | |
162 | that a returned format will not be further changed if passed to an | |
163 | &VIDIOC-SUBDEV-S-FMT; call as-is (as long as external parameters, such as | |
164 | formats on other pads or links' configuration are not changed).</para> | |
165 | ||
166 | <para>Drivers automatically propagate formats inside sub-devices. When a | |
167 | try or active format is set on a pad, corresponding formats on other pads | |
168 | of the same sub-device can be modified by the driver. Drivers are free to | |
169 | modify formats as required by the device. However, they should comply with | |
170 | the following rules when possible: | |
171 | <itemizedlist> | |
665bf368 | 172 | <listitem><para>Formats should be propagated from sink pads to source pads. |
333c8b97 | 173 | Modifying a format on a source pad should not modify the format on any |
665bf368 HV |
174 | sink pad.</para></listitem> |
175 | <listitem><para>Sub-devices that scale frames using variable scaling factors | |
333c8b97 LP |
176 | should reset the scale factors to default values when sink pads formats |
177 | are modified. If the 1:1 scaling ratio is supported, this means that | |
665bf368 | 178 | source pads formats should be reset to the sink pads formats.</para></listitem> |
333c8b97 LP |
179 | </itemizedlist> |
180 | </para> | |
181 | ||
182 | <para>Formats are not propagated across links, as that would involve | |
183 | propagating them from one sub-device file handle to another. Applications | |
184 | must then take care to configure both ends of every link explicitly with | |
185 | compatible formats. Identical formats on the two ends of a link are | |
186 | guaranteed to be compatible. Drivers are free to accept different formats | |
187 | matching device requirements as being compatible.</para> | |
188 | ||
665bf368 | 189 | <para><xref linkend="sample-pipeline-config" /> |
333c8b97 | 190 | shows a sample configuration sequence for the pipeline described in |
665bf368 | 191 | <xref linkend="pipeline-scaling" /> (table |
333c8b97 LP |
192 | columns list entity names and pad numbers).</para> |
193 | ||
194 | <table pgwide="0" frame="none" id="sample-pipeline-config"> | |
195 | <title>Sample Pipeline Configuration</title> | |
196 | <tgroup cols="3"> | |
197 | <colspec colname="what"/> | |
198 | <colspec colname="sensor-0" /> | |
199 | <colspec colname="frontend-0" /> | |
200 | <colspec colname="frontend-1" /> | |
201 | <colspec colname="scaler-0" /> | |
202 | <colspec colname="scaler-1" /> | |
203 | <thead> | |
204 | <row> | |
205 | <entry></entry> | |
206 | <entry>Sensor/0</entry> | |
207 | <entry>Frontend/0</entry> | |
208 | <entry>Frontend/1</entry> | |
209 | <entry>Scaler/0</entry> | |
210 | <entry>Scaler/1</entry> | |
211 | </row> | |
212 | </thead> | |
213 | <tbody valign="top"> | |
214 | <row> | |
215 | <entry>Initial state</entry> | |
216 | <entry>2048x1536</entry> | |
217 | <entry>-</entry> | |
218 | <entry>-</entry> | |
219 | <entry>-</entry> | |
220 | <entry>-</entry> | |
221 | </row> | |
222 | <row> | |
223 | <entry>Configure frontend input</entry> | |
224 | <entry>2048x1536</entry> | |
225 | <entry><emphasis>2048x1536</emphasis></entry> | |
226 | <entry><emphasis>2046x1534</emphasis></entry> | |
227 | <entry>-</entry> | |
228 | <entry>-</entry> | |
229 | </row> | |
230 | <row> | |
231 | <entry>Configure scaler input</entry> | |
232 | <entry>2048x1536</entry> | |
233 | <entry>2048x1536</entry> | |
234 | <entry>2046x1534</entry> | |
235 | <entry><emphasis>2046x1534</emphasis></entry> | |
236 | <entry><emphasis>2046x1534</emphasis></entry> | |
237 | </row> | |
238 | <row> | |
239 | <entry>Configure scaler output</entry> | |
240 | <entry>2048x1536</entry> | |
241 | <entry>2048x1536</entry> | |
242 | <entry>2046x1534</entry> | |
243 | <entry>2046x1534</entry> | |
244 | <entry><emphasis>1280x960</emphasis></entry> | |
245 | </row> | |
246 | </tbody> | |
247 | </tgroup> | |
248 | </table> | |
249 | ||
250 | <para> | |
251 | <orderedlist> | |
665bf368 | 252 | <listitem><para>Initial state. The sensor output is set to its native 3MP |
333c8b97 | 253 | resolution. Resolutions on the host frontend and scaler input and output |
665bf368 HV |
254 | pads are undefined.</para></listitem> |
255 | <listitem><para>The application configures the frontend input pad resolution to | |
333c8b97 LP |
256 | 2048x1536. The driver propagates the format to the frontend output pad. |
257 | Note that the propagated output format can be different, as in this case, | |
258 | than the input format, as the hardware might need to crop pixels (for | |
665bf368 HV |
259 | instance when converting a Bayer filter pattern to RGB or YUV).</para></listitem> |
260 | <listitem><para>The application configures the scaler input pad resolution to | |
333c8b97 | 261 | 2046x1534 to match the frontend output resolution. The driver propagates |
665bf368 HV |
262 | the format to the scaler output pad.</para></listitem> |
263 | <listitem><para>The application configures the scaler output pad resolution to | |
264 | 1280x960.</para></listitem> | |
333c8b97 LP |
265 | </orderedlist> |
266 | </para> | |
267 | ||
268 | <para>When satisfied with the try results, applications can set the active | |
269 | formats by setting the <structfield>which</structfield> argument to | |
446b792c | 270 | <constant>V4L2_SUBDEV_FORMAT_ACTIVE</constant>. Active formats are changed |
333c8b97 LP |
271 | exactly as try formats by drivers. To avoid modifying the hardware state |
272 | during format negotiation, applications should negotiate try formats first | |
273 | and then modify the active settings using the try formats returned during | |
274 | the last negotiation iteration. This guarantees that the active format | |
275 | will be applied as-is by the driver without being modified. | |
276 | </para> | |
277 | </section> | |
278 | ||
64b9ce83 | 279 | <section id="v4l2-subdev-selections"> |
955f645a | 280 | <title>Selections: cropping, scaling and composition</title> |
f6a5cb1b AK |
281 | |
282 | <para>Many sub-devices support cropping frames on their input or output | |
283 | pads (or possible even on both). Cropping is used to select the area of | |
955f645a | 284 | interest in an image, typically on an image sensor or a video decoder. It can |
f6a5cb1b AK |
285 | also be used as part of digital zoom implementations to select the area of |
286 | the image that will be scaled up.</para> | |
287 | ||
288 | <para>Crop settings are defined by a crop rectangle and represented in a | |
289 | &v4l2-rect; by the coordinates of the top left corner and the rectangle | |
290 | size. Both the coordinates and sizes are expressed in pixels.</para> | |
291 | ||
64b9ce83 SA |
292 | <para>As for pad formats, drivers store try and active |
293 | rectangles for the selection targets <xref | |
294 | linkend="v4l2-selections-common" />.</para> | |
955f645a SA |
295 | |
296 | <para>On sink pads, cropping is applied relative to the | |
297 | current pad format. The pad format represents the image size as | |
298 | received by the sub-device from the previous block in the | |
299 | pipeline, and the crop rectangle represents the sub-image that | |
300 | will be transmitted further inside the sub-device for | |
301 | processing.</para> | |
302 | ||
303 | <para>The scaling operation changes the size of the image by | |
304 | scaling it to new dimensions. The scaling ratio isn't specified | |
305 | explicitly, but is implied from the original and scaled image | |
306 | sizes. Both sizes are represented by &v4l2-rect;.</para> | |
307 | ||
308 | <para>Scaling support is optional. When supported by a subdev, | |
309 | the crop rectangle on the subdev's sink pad is scaled to the | |
310 | size configured using the &VIDIOC-SUBDEV-S-SELECTION; IOCTL | |
64b9ce83 | 311 | using <constant>V4L2_SEL_TGT_COMPOSE</constant> |
955f645a SA |
312 | selection target on the same pad. If the subdev supports scaling |
313 | but not composing, the top and left values are not used and must | |
314 | always be set to zero.</para> | |
315 | ||
316 | <para>On source pads, cropping is similar to sink pads, with the | |
317 | exception that the source size from which the cropping is | |
318 | performed, is the COMPOSE rectangle on the sink pad. In both | |
319 | sink and source pads, the crop rectangle must be entirely | |
320 | contained inside the source image size for the crop | |
321 | operation.</para> | |
322 | ||
323 | <para>The drivers should always use the closest possible | |
324 | rectangle the user requests on all selection targets, unless | |
325 | specifically told otherwise. | |
326 | <constant>V4L2_SUBDEV_SEL_FLAG_SIZE_GE</constant> and | |
327 | <constant>V4L2_SUBDEV_SEL_FLAG_SIZE_LE</constant> flags may be | |
328 | used to round the image size either up or down. <xref | |
329 | linkend="v4l2-subdev-selection-flags"></xref></para> | |
330 | </section> | |
331 | ||
332 | <section> | |
333 | <title>Types of selection targets</title> | |
334 | ||
335 | <section> | |
1ec0ed08 | 336 | <title>Actual targets</title> |
955f645a | 337 | |
1ec0ed08 SA |
338 | <para>Actual targets (without a postfix) reflect the actual |
339 | hardware configuration at any point of time. There is a BOUNDS | |
340 | target corresponding to every actual target.</para> | |
955f645a SA |
341 | </section> |
342 | ||
343 | <section> | |
344 | <title>BOUNDS targets</title> | |
345 | ||
1ec0ed08 SA |
346 | <para>BOUNDS targets is the smallest rectangle that contains all |
347 | valid actual rectangles. It may not be possible to set the actual | |
348 | rectangle as large as the BOUNDS rectangle, however. This may be | |
349 | because e.g. a sensor's pixel array is not rectangular but | |
350 | cross-shaped or round. The maximum size may also be smaller than the | |
351 | BOUNDS rectangle.</para> | |
955f645a | 352 | </section> |
f6a5cb1b | 353 | |
955f645a SA |
354 | </section> |
355 | ||
356 | <section> | |
357 | <title>Order of configuration and format propagation</title> | |
358 | ||
359 | <para>Inside subdevs, the order of image processing steps will | |
360 | always be from the sink pad towards the source pad. This is also | |
361 | reflected in the order in which the configuration must be | |
362 | performed by the user: the changes made will be propagated to | |
363 | any subsequent stages. If this behaviour is not desired, the | |
364 | user must set | |
64b9ce83 | 365 | <constant>V4L2_SEL_FLAG_KEEP_CONFIG</constant> flag. This |
955f645a SA |
366 | flag causes no propagation of the changes are allowed in any |
367 | circumstances. This may also cause the accessed rectangle to be | |
368 | adjusted by the driver, depending on the properties of the | |
369 | underlying hardware.</para> | |
370 | ||
371 | <para>The coordinates to a step always refer to the actual size | |
372 | of the previous step. The exception to this rule is the source | |
373 | compose rectangle, which refers to the sink compose bounds | |
374 | rectangle --- if it is supported by the hardware.</para> | |
375 | ||
376 | <orderedlist> | |
377 | <listitem>Sink pad format. The user configures the sink pad | |
378 | format. This format defines the parameters of the image the | |
379 | entity receives through the pad for further processing.</listitem> | |
380 | ||
381 | <listitem>Sink pad actual crop selection. The sink pad crop | |
382 | defines the crop performed to the sink pad format.</listitem> | |
383 | ||
384 | <listitem>Sink pad actual compose selection. The size of the | |
385 | sink pad compose rectangle defines the scaling ratio compared | |
386 | to the size of the sink pad crop rectangle. The location of | |
387 | the compose rectangle specifies the location of the actual | |
388 | sink compose rectangle in the sink compose bounds | |
389 | rectangle.</listitem> | |
390 | ||
391 | <listitem>Source pad actual crop selection. Crop on the source | |
392 | pad defines crop performed to the image in the sink compose | |
393 | bounds rectangle.</listitem> | |
394 | ||
395 | <listitem>Source pad format. The source pad format defines the | |
396 | output pixel format of the subdev, as well as the other | |
397 | parameters with the exception of the image width and height. | |
398 | Width and height are defined by the size of the source pad | |
399 | actual crop selection.</listitem> | |
400 | </orderedlist> | |
401 | ||
402 | <para>Accessing any of the above rectangles not supported by the | |
403 | subdev will return <constant>EINVAL</constant>. Any rectangle | |
404 | referring to a previous unsupported rectangle coordinates will | |
405 | instead refer to the previous supported rectangle. For example, | |
406 | if sink crop is not supported, the compose selection will refer | |
407 | to the sink pad format dimensions instead.</para> | |
408 | ||
409 | <figure id="subdev-image-processing-crop"> | |
410 | <title>Image processing in subdevs: simple crop example</title> | |
411 | <mediaobject> | |
412 | <imageobject> | |
413 | <imagedata fileref="subdev-image-processing-crop.svg" | |
414 | format="SVG" scale="200" /> | |
415 | </imageobject> | |
416 | </mediaobject> | |
417 | </figure> | |
418 | ||
419 | <para>In the above example, the subdev supports cropping on its | |
420 | sink pad. To configure it, the user sets the media bus format on | |
421 | the subdev's sink pad. Now the actual crop rectangle can be set | |
422 | on the sink pad --- the location and size of this rectangle | |
423 | reflect the location and size of a rectangle to be cropped from | |
424 | the sink format. The size of the sink crop rectangle will also | |
425 | be the size of the format of the subdev's source pad.</para> | |
426 | ||
427 | <figure id="subdev-image-processing-scaling-multi-source"> | |
428 | <title>Image processing in subdevs: scaling with multiple sources</title> | |
429 | <mediaobject> | |
430 | <imageobject> | |
431 | <imagedata fileref="subdev-image-processing-scaling-multi-source.svg" | |
432 | format="SVG" scale="200" /> | |
433 | </imageobject> | |
434 | </mediaobject> | |
435 | </figure> | |
436 | ||
437 | <para>In this example, the subdev is capable of first cropping, | |
438 | then scaling and finally cropping for two source pads | |
439 | individually from the resulting scaled image. The location of | |
440 | the scaled image in the cropped image is ignored in sink compose | |
441 | target. Both of the locations of the source crop rectangles | |
442 | refer to the sink scaling rectangle, independently cropping an | |
443 | area at location specified by the source crop rectangle from | |
444 | it.</para> | |
445 | ||
446 | <figure id="subdev-image-processing-full"> | |
447 | <title>Image processing in subdevs: scaling and composition | |
448 | with multiple sinks and sources</title> | |
449 | <mediaobject> | |
450 | <imageobject> | |
451 | <imagedata fileref="subdev-image-processing-full.svg" | |
452 | format="SVG" scale="200" /> | |
453 | </imageobject> | |
454 | </mediaobject> | |
455 | </figure> | |
456 | ||
457 | <para>The subdev driver supports two sink pads and two source | |
458 | pads. The images from both of the sink pads are individually | |
459 | cropped, then scaled and further composed on the composition | |
460 | bounds rectangle. From that, two independent streams are cropped | |
461 | and sent out of the subdev from the source pads.</para> | |
f6a5cb1b AK |
462 | |
463 | </section> | |
955f645a | 464 | |
333c8b97 LP |
465 | </section> |
466 | ||
467 | &sub-subdev-formats; |