| Message ID | afe51f5f-03dd-4092-9ec0-297afb1453c7@mentor.com (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
On Wed, Jan 11, 2017 at 7:22 PM, Steve Longerbeam <steve_longerbeam@mentor.com> wrote: > Hi Tim, > > > On 01/11/2017 03:14 PM, Tim Harvey wrote: >> >> >> <snip> >> >> Hi Steve, >> >> I took a stab at testing this today on a gw51xx which has an adv7180 >> hooked up as follows: >> - i2c3@0x20 >> - 8bit data bus from DAT12 to DAT19, HSYNC, VSYNC, PIXCLK on CSI0 pads >> (CSI0_IPU1) >> - PWRDWN# on MX6QDL_PAD_CSI0_DATA_EN__GPIO5_IO20 >> - IRQ# on MX6QDL_PAD_CSI0_DAT5__GPIO5_IO23 >> - all three analog inputs available to off-board connector >> >> My patch to the imx6qdl-gw51xx dtsi is: > > > As long as you used the patch to imx6qdl-sabreauto.dtsti that adds > the adv7180 support as a guide, you should be ok here. yes - fairly straightforward > >> <snip> >> >> >> >> On an IMX6Q I'm getting the following when the adv7180 module loads: >> [ 12.862477] adv7180 2-0020: chip found @ 0x20 (21a8000.i2c) >> [ 12.907767] imx-media: Registered subdev adv7180 2-0020 >> [ 12.907793] imx-media soc:media@0: Entity type for entity adv7180 >> 2-0020 was not initialized! >> [ 12.907867] imx-media: imx_media_create_link: adv7180 2-0020:0 -> >> ipu1_csi0_mux:1 >> >> Is the warning that adv7180 was not initialized expected and or an issue? > > > Yeah it's still a bug in the adv7180 driver, needs fixing. > ok - ignoring >> >> Now that your driver is hooking into the current media framework, I'm >> not at all clear on how to link and configure the media entities. > > > It's all documented at Documentation/media/v4l-drivers/imx.rst. > Follow the SabreAuto pipeline setup example. > ah yes... it helps to read your patches! You did a great job on the documentation. Regarding the The ipu1_csi0_mux/ipu2_csi1_mux entities which have 1 source and 2 sinks (which makes sense for a mux) how do you know which sink pad you should use (in your adv7180 example you use the 2nd sink pad vs the first)? As my hardware is the same as the SabreAuto except that my adv7180 is on i2c-2@0x20 I follow your example from Documentation/media/v4l-drivers/imx.rst: # Setup links media-ctl -l '"adv7180 2-0020":0 -> "ipu1_csi0_mux":1[1]' media-ctl -l '"ipu1_csi0_mux":2 -> "ipu1_csi0":0[1]' media-ctl -l '"ipu1_csi0":1 -> "ipu1_smfc0":0[1]' media-ctl -l '"ipu1_smfc0":1 -> "ipu1_ic_prpvf":0[1]' media-ctl -l '"ipu1_ic_prpvf":1 -> "camif0":0[1]' media-ctl -l '"camif0":1 -> "camif0 devnode":0[1]' # Configure pads media-ctl -V "\"adv7180 2-0020\":0 [fmt:UYVY2X8/720x480]" media-ctl -V "\"ipu1_csi0_mux\":1 [fmt:UYVY2X8/720x480]" media-ctl -V "\"ipu1_csi0_mux\":2 [fmt:UYVY2X8/720x480]" media-ctl -V "\"ipu1_csi0\":0 [fmt:UYVY2X8/720x480]" media-ctl -V "\"ipu1_csi0\":1 [fmt:UYVY2X8/720x480]" media-ctl -V "\"ipu1_smfc0\":0 [fmt:UYVY2X8/720x480]" media-ctl -V "\"ipu1_smfc0\":1 [fmt:UYVY2X8/720x480]" media-ctl -V "\"ipu1_ic_prpvf\":0 [fmt:UYVY2X8/720x480]" # pad field types for camif can be any format prpvf supports export outputfmt="UYVY2X8/720x480" media-ctl -V "\"ipu1_ic_prpvf\":1 [fmt:$outputfmt]" media-ctl -V "\"camif0\":0 [fmt:$outputfmt]" media-ctl -V "\"camif0\":1 [fmt:$outputfmt]" # select AIN1 v4l2-ctl -d0 -i0 Video input set to 0 (ADV7180 Composite on Ain1: ok) v4l2-ctl -d0 --set-fmt-video=width=720,height=480,pixelformat=UYVY # capture a single raw frame v4l2-ctl -d0 --stream-mmap --stream-to=/x.raw --stream-count=1 [ 2092.056394] camif0: pipeline_set_stream failed with -32 VIDIOC_STREAMON: failed: Broken pipe Enabling debug in drivers/media/media-entity.c I see: [ 38.870087] imx-media soc:media@0: link validation failed for "ipu1_smfc0":1 -> "ipu1_ic_prpvf":0, error -32 Looking at ipu1_smfc0 and ipu1_ic_prpvf with media-ctl I see: - entity 12: ipu1_ic_prpvf (2 pads, 8 links) type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev3 pad0: Sink [fmt:UYVY2X8/720x480 field:alternate] <- "ipu1_csi0":1 [] <- "ipu1_csi1":1 [] <- "ipu1_smfc0":1 [ENABLED] <- "ipu1_smfc1":1 [] pad1: Source [fmt:UYVY2X8/720x480 field:none] -> "camif0":0 [ENABLED] -> "camif1":0 [] -> "ipu1_ic_pp0":0 [] -> "ipu1_ic_pp1":0 [] - entity 45: ipu1_smfc0 (2 pads, 5 links) type V4L2 subdev subtype Unknown flags 0 device node name /dev/v4l-subdev14 pad0: Sink [fmt:UYVY2X8/720x480] <- "ipu1_csi0":1 [ENABLED] pad1: Source [fmt:UYVY2X8/720x480] -> "ipu1_ic_prpvf":0 [ENABLED] -> "ipu1_ic_pp0":0 [] -> "camif0":0 [] -> "camif1":0 [] Any ideas what is going wrong here? Seems like its perhaps a field type mismatch. Is my outputfmt incorrect perhaps? I likely have misunderstood the pad type comments in your documentation. > > >> <snip> >> >> >> >> Additionally I've found that on an IMX6S/IMX6DL we crash while >> registering the media-ic subdev's: <snip> > > Yep, I only have quad boards here so I haven't gotten around to > testing on S/DL. > > But it looks like I forgot to clear out the csi subdev pointer array before > passing it to imx_media_of_parse(). I think that might explain the OOPS > above. Try this patch: > > diff --git a/drivers/staging/media/imx/imx-media-dev.c > b/drivers/staging/media/imx/imx-media-dev.c > index 357654d..0cf2d61 100644 > --- a/drivers/staging/media/imx/imx-media-dev.c > +++ b/drivers/staging/media/imx/imx-media-dev.c > @@ -379,7 +379,7 @@ static int imx_media_probe(struct platform_device *pdev) > { > struct device *dev = &pdev->dev; > struct device_node *node = dev->of_node; > - struct imx_media_subdev *csi[4]; > + struct imx_media_subdev *csi[4] = {0}; > struct imx_media_dev *imxmd; > int ret; > This does resolves the crash on S/DL. I do notice that the ipu1_csi*_mux entities on the S/DL have 3 more sink pads compared to the D/Q which is from the additional ports defined in the GPR nodes you add for mipi_vc1/vc2/vc3. Are there really 3 more MIPI virtual channels on the S/DL vs the D/Q? I get the same results on the S/DL as I do on D/Q as long as I adjust the links to compensate for these additional sinks: media-ctl -l '"adv7180 2-0020":0 -> "ipu1_csi0_mux":4[1]' # pad4 media-ctl -l '"ipu1_csi0_mux":5 -> "ipu1_csi0":0[1]' # pad5 ... This means link configuration must differ depending on S/DL vs D/Q which is a bummer but I suppose this is the harsh reality as for boards that use the EIM pads for IPU's they also will be using IPU2 for IMX6D/Q and IPU1 for IMX6S/DL. Tim -- To unsubscribe from this list: send the line "unsubscribe linux-media" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi Tim, On 01/12/2017 01:13 PM, Tim Harvey wrote: > >>> Now that your driver is hooking into the current media framework, I'm >>> not at all clear on how to link and configure the media entities. >> >> It's all documented at Documentation/media/v4l-drivers/imx.rst. >> Follow the SabreAuto pipeline setup example. >> > ah yes... it helps to read your patches! You did a great job on the > documentation. > > Regarding the The ipu1_csi0_mux/ipu2_csi1_mux entities which have 1 > source and 2 sinks (which makes sense for a mux) how do you know which > sink pad you should use (in your adv7180 example you use the 2nd sink > pad vs the first)? The adv7180 can only go to the parallel input pad (ipu1_csi0_mux:1 on quad). The other input pads select from the mipi csi-2 receiver virtual channels. Have you generated a dot graph? It makes it much easier to visualize: # media-ctl --print-dot > graph.dot then on your host: % dot -Tpng graph.dot > graph.png > > As my hardware is the same as the SabreAuto except that my adv7180 is > on i2c-2@0x20 I follow your example from > Documentation/media/v4l-drivers/imx.rst: > > # Setup links > media-ctl -l '"adv7180 2-0020":0 -> "ipu1_csi0_mux":1[1]' > media-ctl -l '"ipu1_csi0_mux":2 -> "ipu1_csi0":0[1]' > media-ctl -l '"ipu1_csi0":1 -> "ipu1_smfc0":0[1]' > media-ctl -l '"ipu1_smfc0":1 -> "ipu1_ic_prpvf":0[1]' > media-ctl -l '"ipu1_ic_prpvf":1 -> "camif0":0[1]' > media-ctl -l '"camif0":1 -> "camif0 devnode":0[1]' > > # Configure pads > media-ctl -V "\"adv7180 2-0020\":0 [fmt:UYVY2X8/720x480]" > media-ctl -V "\"ipu1_csi0_mux\":1 [fmt:UYVY2X8/720x480]" > media-ctl -V "\"ipu1_csi0_mux\":2 [fmt:UYVY2X8/720x480]" > media-ctl -V "\"ipu1_csi0\":0 [fmt:UYVY2X8/720x480]" > media-ctl -V "\"ipu1_csi0\":1 [fmt:UYVY2X8/720x480]" > media-ctl -V "\"ipu1_smfc0\":0 [fmt:UYVY2X8/720x480]" > media-ctl -V "\"ipu1_smfc0\":1 [fmt:UYVY2X8/720x480]" > media-ctl -V "\"ipu1_ic_prpvf\":0 [fmt:UYVY2X8/720x480]" > # pad field types for camif can be any format prpvf supports > export outputfmt="UYVY2X8/720x480" > media-ctl -V "\"ipu1_ic_prpvf\":1 [fmt:$outputfmt]" > media-ctl -V "\"camif0\":0 [fmt:$outputfmt]" > media-ctl -V "\"camif0\":1 [fmt:$outputfmt]" > > # select AIN1 > v4l2-ctl -d0 -i0 > Video input set to 0 (ADV7180 Composite on Ain1: ok) > v4l2-ctl -d0 --set-fmt-video=width=720,height=480,pixelformat=UYVY > # capture a single raw frame > v4l2-ctl -d0 --stream-mmap --stream-to=/x.raw --stream-count=1 > [ 2092.056394] camif0: pipeline_set_stream failed with -32 > VIDIOC_STREAMON: failed: Broken pipe > > Enabling debug in drivers/media/media-entity.c I see: > [ 38.870087] imx-media soc:media@0: link validation failed for > "ipu1_smfc0":1 -> "ipu1_ic_prpvf":0, error -32 > > Looking at ipu1_smfc0 and ipu1_ic_prpvf with media-ctl I see: > - entity 12: ipu1_ic_prpvf (2 pads, 8 links) > type V4L2 subdev subtype Unknown flags 0 > device node name /dev/v4l-subdev3 > pad0: Sink > [fmt:UYVY2X8/720x480 field:alternate] > <- "ipu1_csi0":1 [] > <- "ipu1_csi1":1 [] > <- "ipu1_smfc0":1 [ENABLED] > <- "ipu1_smfc1":1 [] > pad1: Source > [fmt:UYVY2X8/720x480 field:none] > -> "camif0":0 [ENABLED] > -> "camif1":0 [] > -> "ipu1_ic_pp0":0 [] > -> "ipu1_ic_pp1":0 [] > > - entity 45: ipu1_smfc0 (2 pads, 5 links) > type V4L2 subdev subtype Unknown flags 0 > device node name /dev/v4l-subdev14 > pad0: Sink > [fmt:UYVY2X8/720x480] > <- "ipu1_csi0":1 [ENABLED] > pad1: Source > [fmt:UYVY2X8/720x480] > -> "ipu1_ic_prpvf":0 [ENABLED] > -> "ipu1_ic_pp0":0 [] > -> "camif0":0 [] > -> "camif1":0 [] > > Any ideas what is going wrong here? Seems like its perhaps a field > type mismatch. Yes, exactly, you'll need to set the field types on every pad in your pipeline. > Is my outputfmt incorrect perhaps? I likely have > misunderstood the pad type comments in your documentation. Attached is an update doc (from branch imx-media-staging-md-v7 on my fork). I recently upgraded my v4l-utils package and media-ctl now supports specifying the field type in the pad format strings. If you don't have the latest v4l-utils, it's fairly straightforward to cross-build. > >> >>> <snip> >>> >>> >>> >>> Additionally I've found that on an IMX6S/IMX6DL we crash while >>> registering the media-ic subdev's: > <snip> >> Yep, I only have quad boards here so I haven't gotten around to >> testing on S/DL. >> >> But it looks like I forgot to clear out the csi subdev pointer array before >> passing it to imx_media_of_parse(). I think that might explain the OOPS >> above. Try this patch: >> >> diff --git a/drivers/staging/media/imx/imx-media-dev.c >> b/drivers/staging/media/imx/imx-media-dev.c >> index 357654d..0cf2d61 100644 >> --- a/drivers/staging/media/imx/imx-media-dev.c >> +++ b/drivers/staging/media/imx/imx-media-dev.c >> @@ -379,7 +379,7 @@ static int imx_media_probe(struct platform_device *pdev) >> { >> struct device *dev = &pdev->dev; >> struct device_node *node = dev->of_node; >> - struct imx_media_subdev *csi[4]; >> + struct imx_media_subdev *csi[4] = {0}; >> struct imx_media_dev *imxmd; >> int ret; >> > This does resolves the crash on S/DL. Cool thanks for verifying, I've applied this to the imx-media-staging-md-v7 branch. > > I do notice that the ipu1_csi*_mux entities on the S/DL have 3 more > sink pads compared to the D/Q which is from the additional ports > defined in the GPR nodes you add for mipi_vc1/vc2/vc3. Are there > really 3 more MIPI virtual channels on the S/DL vs the D/Q? Well, same number of virtual channels on quad vs S/DL. It's just that the video mux on S/DL can select from all 4 virtual channels, whereas quad's mux'es only select either vc0 or vc3. > > I get the same results on the S/DL as I do on D/Q as long as I adjust > the links to compensate for these additional sinks: > media-ctl -l '"adv7180 2-0020":0 -> "ipu1_csi0_mux":4[1]' # pad4 > media-ctl -l '"ipu1_csi0_mux":5 -> "ipu1_csi0":0[1]' # pad5 > ... > > This means link configuration must differ depending on S/DL vs D/Q > which is a bummer but I suppose this is the harsh reality as for > boards that use the EIM pads for IPU's they also will be using IPU2 > for IMX6D/Q and IPU1 for IMX6S/DL. yeah, the links necessarily must be different between quad and S/DL. Steve i.MX Video Capture Driver ========================= Introduction ------------ The Freescale i.MX5/6 contains an Image Processing Unit (IPU), which handles the flow of image frames to and from capture devices and display devices. For image capture, the IPU contains the following internal subunits: - Image DMA Controller (IDMAC) - Camera Serial Interface (CSI) - Image Converter (IC) - Sensor Multi-FIFO Controller (SMFC) - Image Rotator (IRT) - Video De-Interlace Controller (VDIC) The IDMAC is the DMA controller for transfer of image frames to and from memory. Various dedicated DMA channels exist for both video capture and display paths. The CSI is the frontend capture unit that interfaces directly with capture sensors over Parallel, BT.656/1120, and MIPI CSI-2 busses. The IC handles color-space conversion, resizing, and rotation operations. There are three independent "tasks" within the IC that can carry out conversions concurrently: pre-processing encoding, pre-processing preview, and post-processing. The SMFC is composed of four independent channels that each can transfer captured frames from sensors directly to memory concurrently. The IRT carries out 90 and 270 degree image rotation operations. The VDIC handles the conversion of interlaced video to progressive, with support for different motion compensation modes (low, medium, and high motion). The deinterlaced output frames from the VDIC can be sent to the IC pre-process preview task for further conversions. In addition to the IPU internal subunits, there are also two units outside the IPU that are also involved in video capture on i.MX: - MIPI CSI-2 Receiver for camera sensors with the MIPI CSI-2 bus interface. This is a Synopsys DesignWare core. - A video multiplexer for selecting among multiple sensor inputs to send to a CSI. For more info, refer to the latest versions of the i.MX5/6 reference manuals listed under References. Features -------- Some of the features of this driver include: - Many different pipelines can be configured via media controller API, that correspond to the hardware video capture pipelines supported in the i.MX. - Supports parallel, BT.565, and MIPI CSI-2 interfaces. - Up to four concurrent sensor acquisitions, by configuring each sensor's pipeline using independent entities. This is currently demonstrated with the SabreSD and SabreLite reference boards with independent OV5642 and MIPI CSI-2 OV5640 sensor modules. - Scaling, color-space conversion, and image rotation via IC task subdevs. - Many pixel formats supported (RGB, packed and planar YUV, partial planar YUV). - The IC pre-process preview subdev supports motion compensated de-interlacing using the VDIC, with three motion compensation modes: low, medium, and high motion. The mode is specified with a custom control. Pipelines are defined that allow sending frames to the preview subdev directly from the CSI or from the SMFC. - Includes a Frame Interval Monitor (FIM) that can correct vertical sync problems with the ADV718x video decoders. See below for a description of the FIM. Capture Pipelines ----------------- The following describe the various use-cases supported by the pipelines. The links shown do not include the frontend sensor, video mux, or mipi csi-2 receiver links. This depends on the type of sensor interface (parallel or mipi csi-2). So in all cases, these pipelines begin with: sensor -> ipu_csi_mux -> ipu_csi -> ... for parallel sensors, or: sensor -> imx-mipi-csi2 -> (ipu_csi_mux) -> ipu_csi -> ... for mipi csi-2 sensors. The imx-mipi-csi2 receiver may need to route to the video mux (ipu_csi_mux) before sending to the CSI, depending on the mipi csi-2 virtual channel, hence ipu_csi_mux is shown in parenthesis. Unprocessed Video Capture: -------------------------- Send frames directly from sensor to camera interface, with no conversions: -> ipu_smfc -> camif Note the ipu_smfc can do pixel reordering within the same colorspace. For example, its sink pad can take UYVY2X8, but its source pad can output YUYV2X8. IC Direct Conversions: ---------------------- This pipeline uses the preprocess encode entity to route frames directly from the CSI to the IC (bypassing the SMFC), to carry out scaling up to 1024x1024 resolution, CSC, and image rotation: -> ipu_ic_prpenc -> camif This can be a useful capture pipeline for heavily loaded memory bus traffic environments, since it has minimal IDMAC channel usage. Post-Processing Conversions: ---------------------------- This pipeline routes frames from the SMFC to the post-processing entity. In addition to CSC and rotation, this entity supports tiling which allows scaled output beyond the 1024x1024 limitation of the IC (up to 4096x4096 scaling output is supported): -> ipu_smfc -> ipu_ic_pp -> camif Motion Compensated De-interlace: -------------------------------- This pipeline routes frames from the SMFC to the preprocess preview entity to support motion-compensated de-interlacing using the VDIC, scaling up to 1024x1024, and CSC: -> ipu_smfc -> ipu_ic_prpvf -> camif This pipeline also carries out the same conversions as above, but routes frames directly from the CSI to the IC preprocess preview entity for minimal memory bandwidth usage (note: this pipeline only works in "high motion" mode): -> ipu_ic_prpvf -> camif This pipeline takes the motion-compensated de-interlaced frames and sends them to the post-processor, to support motion-compensated de-interlacing, scaling up to 4096x4096, CSC, and rotation: -> (ipu_smfc) -> ipu_ic_prpvf -> ipu_ic_pp -> camif Usage Notes ----------- Many of the subdevs require information from the active sensor in the current pipeline when configuring pad formats. Therefore the media links should be established before configuring the media pad formats. Similarly, the capture v4l2 interface subdev inherits controls from the active subdevs in the current pipeline at link-setup time. Therefore the capture links should be the last links established in order for capture to "see" and inherit all possible controls. The following are usage notes for Sabre- reference platforms: SabreLite with OV5642 and OV5640 -------------------------------- This platform requires the OmniVision OV5642 module with a parallel camera interface, and the OV5640 module with a MIPI CSI-2 interface. Both modules are available from Boundary Devices: https://boundarydevices.com/products/nit6x_5mp https://boundarydevices.com/product/nit6x_5mp_mipi Note that if only one camera module is available, the other sensor node can be disabled in the device tree. The OV5642 module is connected to the parallel bus input on the i.MX internal video mux to IPU1 CSI0. It's i2c bus connects to i2c bus 2. The MIPI CSI-2 OV5640 module is connected to the i.MX internal MIPI CSI-2 receiver, and the four virtual channel outputs from the receiver are routed as follows: vc0 to the IPU1 CSI0 mux, vc1 directly to IPU1 CSI1, vc2 directly to IPU2 CSI0, and vc3 to the IPU2 CSI1 mux. The OV5640 is also connected to i2c bus 2 on the SabreLite, therefore the OV5642 and OV5640 must not share the same i2c slave address. The following basic example configures unprocessed video capture pipelines for both sensors. The OV5642 is routed to camif0 (usually /dev/video0), and the OV5640 (transmitting on mipi csi-2 virtual channel 1) is routed to camif1 (usually /dev/video1). Both sensors are configured to output 640x480, the OV5642 outputs YUYV2X8, the OV5640 UYVY2X8: .. code-block:: none # Setup links for OV5642 media-ctl -l '"ov5642 1-0042":0 -> "ipu1_csi0_mux":1[1]' media-ctl -l '"ipu1_csi0_mux":2 -> "ipu1_csi0":0[1]' media-ctl -l '"ipu1_csi0":1 -> "ipu1_smfc0":0[1]' media-ctl -l '"ipu1_smfc0":1 -> "camif0":0[1]' media-ctl -l '"camif0":1 -> "camif0 devnode":0[1]' # Setup links for OV5640 media-ctl -l '"ov5640_mipi 1-0040":0 -> "imx-mipi-csi2":0[1]' media-ctl -l '"imx-mipi-csi2":2 -> "ipu1_csi1":0[1]' media-ctl -l '"ipu1_csi1":1 -> "ipu1_smfc1":0[1]' media-ctl -l '"ipu1_smfc1":1 -> "camif1":0[1]' media-ctl -l '"camif1":1 -> "camif1 devnode":0[1]' # Configure pads for OV5642 pipeline media-ctl -V "\"ov5642 1-0042\":0 [fmt:YUYV2X8/640x480 field:none]" media-ctl -V "\"ipu1_csi0_mux\":1 [fmt:YUYV2X8/640x480 field:none]" media-ctl -V "\"ipu1_csi0_mux\":2 [fmt:YUYV2X8/640x480 field:none]" media-ctl -V "\"ipu1_csi0\":0 [fmt:YUYV2X8/640x480 field:none]" media-ctl -V "\"ipu1_csi0\":1 [fmt:YUYV2X8/640x480 field:none]" media-ctl -V "\"ipu1_smfc0\":0 [fmt:YUYV2X8/640x480 field:none]" media-ctl -V "\"ipu1_smfc0\":1 [fmt:UYVY2X8/640x480 field:none]" media-ctl -V "\"camif0\":0 [fmt:UYVY2X8/640x480 field:none]" media-ctl -V "\"camif0\":1 [fmt:UYVY2X8/640x480 field:none]" # Configure pads for OV5640 pipeline media-ctl -V "\"ov5640_mipi 1-0040\":0 [fmt:UYVY2X8/640x480 field:none]" media-ctl -V "\"imx-mipi-csi2\":0 [fmt:UYVY2X8/640x480 field:none]" media-ctl -V "\"imx-mipi-csi2\":2 [fmt:UYVY2X8/640x480 field:none]" media-ctl -V "\"ipu1_csi1\":0 [fmt:UYVY2X8/640x480 field:none]" media-ctl -V "\"ipu1_csi1\":1 [fmt:UYVY2X8/640x480 field:none]" media-ctl -V "\"ipu1_smfc1\":0 [fmt:UYVY2X8/640x480 field:none]" media-ctl -V "\"ipu1_smfc1\":1 [fmt:UYVY2X8/640x480 field:none]" media-ctl -V "\"camif1\":0 [fmt:UYVY2X8/640x480 field:none]" media-ctl -V "\"camif1\":1 [fmt:UYVY2X8/640x480 field:none]" Streaming can then begin independently on device nodes /dev/video0 and /dev/video1. SabreAuto with ADV7180 decoder ------------------------------ On the SabreAuto, an on-board ADV7180 SD decoder is connected to the parallel bus input on the internal video mux to IPU1 CSI0. The following example configures a pipeline to capture from the ADV7180 video decoder, assuming NTSC 720x480 input signals, with Motion Compensated de-interlacing. Pad field types assume the adv7180 outputs "alternate". $outputfmt can be any format supported by the ipu1_ic_prpvf entity at its output pad: .. code-block:: none # Setup links media-ctl -l '"adv7180 4-0021":0 -> "ipu1_csi0_mux":1[1]' media-ctl -l '"ipu1_csi0_mux":2 -> "ipu1_csi0":0[1]' media-ctl -l '"ipu1_csi0":1 -> "ipu1_smfc0":0[1]' media-ctl -l '"ipu1_smfc0":1 -> "ipu1_ic_prpvf":0[1]' media-ctl -l '"ipu1_ic_prpvf":1 -> "camif0":0[1]' media-ctl -l '"camif0":1 -> "camif0 devnode":0[1]' # Configure pads media-ctl -V "\"adv7180 4-0021\":0 [fmt:UYVY2X8/720x480]" media-ctl -V "\"ipu1_csi0_mux\":1 [fmt:UYVY2X8/720x480 field:alternate]" media-ctl -V "\"ipu1_csi0_mux\":2 [fmt:UYVY2X8/720x480 field:alternate]" media-ctl -V "\"ipu1_csi0\":0 [fmt:UYVY2X8/720x480 field:alternate]" media-ctl -V "\"ipu1_csi0\":1 [fmt:UYVY2X8/720x480 field:alternate]" media-ctl -V "\"ipu1_smfc0\":0 [fmt:UYVY2X8/720x480 field:alternate]" media-ctl -V "\"ipu1_smfc0\":1 [fmt:UYVY2X8/720x480 field:alternate]" media-ctl -V "\"ipu1_ic_prpvf\":0 [fmt:UYVY2X8/720x480 field:alternate]" media-ctl -V "\"ipu1_ic_prpvf\":1 [fmt:$outputfmt field:none]" media-ctl -V "\"camif0\":0 [fmt:$outputfmt field:none]" media-ctl -V "\"camif0\":1 [fmt:$outputfmt field:none]" Streaming can then begin on /dev/video0. This platform accepts Composite Video analog inputs to the ADV7180 on Ain1 (connector J42) and Ain3 (connector J43). To switch to Ain1: .. code-block:: none # v4l2-ctl -i0 To switch to Ain3: .. code-block:: none # v4l2-ctl -i1 Frame Interval Monitor ---------------------- The adv718x decoders can occasionally send corrupt fields during NTSC/PAL signal re-sync (too little or too many video lines). When this happens, the IPU triggers a mechanism to re-establish vertical sync by adding 1 dummy line every frame, which causes a rolling effect from image to image, and can last a long time before a stable image is recovered. Or sometimes the mechanism doesn't work at all, causing a permanent split image (one frame contains lines from two consecutive captured images). From experiment it was found that during image rolling, the frame intervals (elapsed time between two EOF's) drop below the nominal value for the current standard, by about one frame time (60 usec), and remain at that value until rolling stops. While the reason for this observation isn't known (the IPU dummy line mechanism should show an increase in the intervals by 1 line time every frame, not a fixed value), we can use it to detect the corrupt fields using a frame interval monitor. If the FIM detects a bad frame interval, a subdev event is sent. In response, userland can issue a streaming restart to correct the rolling/split image. The FIM is implemented in the imx-csi entity, and the entities that have direct connections to the CSI call into the FIM to monitor the frame intervals: ipu_smfc, ipu_ic_prpenc, and ipu_prpvf (when configured with a direct link from ipu_csi). Userland can register with the FIM event notifications on the imx-csi subdev device node (V4L2_EVENT_IMX_FRAME_INTERVAL). The imx-csi entity includes custom controls to tweak some dials for FIM. If one of these controls is changed during streaming, the FIM will be reset and will continue at the new settings. - V4L2_CID_IMX_FIM_ENABLE Enable/disable the FIM. - V4L2_CID_IMX_FIM_NUM How many frame interval errors to average before comparing against the nominal frame interval reported by the sensor. This can reduce noise from interrupt latency. - V4L2_CID_IMX_FIM_TOLERANCE_MIN If the averaged intervals fall outside nominal by this amount, in microseconds, streaming will be restarted. - V4L2_CID_IMX_FIM_TOLERANCE_MAX If any interval errors are higher than this value, those error samples are discarded and do not enter into the average. This can be used to discard really high interval errors that might be due to very high system load, causing excessive interrupt latencies. - V4L2_CID_IMX_FIM_NUM_SKIP How many frames to skip after a FIM reset or stream restart before FIM begins to average intervals. It has been found that there can be a few bad frame intervals after stream restart which are not attributed to adv718x sending a corrupt field, so this is used to skip those frames to prevent unnecessary restarts. SabreSD with MIPI CSI-2 OV5640 ------------------------------ Similarly to SabreLite, the SabreSD supports a parallel interface OV5642 module on IPU1 CSI0, and a MIPI CSI-2 OV5640 module. The OV5642 connects to i2c bus 1 and the OV5640 to i2c bus 2. The device tree for SabreSD includes OF graphs for both the parallel OV5642 and the MIPI CSI-2 OV5640, but as of this writing only the MIPI CSI-2 OV5640 has been tested, so the OV5642 node is currently disabled. The OV5640 module connects to MIPI connector J5 (sorry I don't have the compatible module part number or URL). The following example configures a post-processing pipeline to capture from the OV5640. $sensorfmt can be any format supported by the OV5640. $outputfmt can be any format supported by the ipu1_ic_pp1 entity at its output pad: .. code-block:: none # Setup links media-ctl -l '"ov5640_mipi 1-003c":0 -> "imx-mipi-csi2":0[1]' media-ctl -l '"imx-mipi-csi2":2 -> "ipu1_csi1":0[1]' media-ctl -l '"ipu1_csi1":1 -> "ipu1_smfc1":0[1]' media-ctl -l '"ipu1_smfc1":1 -> "ipu1_ic_pp1":0[1]' media-ctl -l '"ipu1_ic_pp1":1 -> "camif0":0[1]' media-ctl -l '"camif0":1 -> "camif0 devnode":0[1]' # Configure pads media-ctl -V "\"ov5640_mipi 1-003c\":0 [fmt:$sensorfmt field:none]" media-ctl -V "\"imx-mipi-csi2\":0 [fmt:$sensorfmt field:none]" media-ctl -V "\"imx-mipi-csi2\":2 [fmt:$sensorfmt field:none]" media-ctl -V "\"ipu1_csi1\":0 [fmt:$sensorfmt field:none]" media-ctl -V "\"ipu1_csi1\":1 [fmt:$sensorfmt field:none]" media-ctl -V "\"ipu1_smfc1\":0 [fmt:$sensorfmt field:none]" media-ctl -V "\"ipu1_smfc1\":1 [fmt:$sensorfmt field:none]" media-ctl -V "\"ipu1_ic_pp1\":0 [fmt:$sensorfmt field:none]" media-ctl -V "\"ipu1_ic_pp1\":1 [fmt:$outputfmt field:none]" media-ctl -V "\"camif0\":0 [fmt:$outputfmt field:none]" media-ctl -V "\"camif0\":1 [fmt:$outputfmt field:none]" Streaming can then begin on /dev/video0. Known Issues ------------ 1. When using 90 or 270 degree rotation control at capture resolutions near the IC resizer limit of 1024x1024, and combined with planar pixel formats (YUV420, YUV422p), frame capture will often fail with no end-of-frame interrupts from the IDMAC channel. To work around this, use lower resolution and/or packed formats (YUYV, RGB3, etc.) when 90 or 270 rotations are needed. File list --------- drivers/staging/media/imx/ include/media/imx.h include/uapi/media/imx.h References ---------- [1] "i.MX 6Dual/6Quad Applications Processor Reference Manual" [2] "i.MX 6Solo/6DualLite Applications Processor Reference Manual" Author ------ Steve Longerbeam <steve_longerbeam@mentor.com> Copyright (C) 2012-2016 Mentor Graphics Inc.
On Thu, Jan 12, 2017 at 2:32 PM, Steve Longerbeam <slongerbeam@gmail.com> wrote: > Hi Tim, > > > On 01/12/2017 01:13 PM, Tim Harvey wrote: >> >> >>>> Now that your driver is hooking into the current media framework, I'm >>>> not at all clear on how to link and configure the media entities. >>> >>> >>> It's all documented at Documentation/media/v4l-drivers/imx.rst. >>> Follow the SabreAuto pipeline setup example. >>> >> ah yes... it helps to read your patches! You did a great job on the >> documentation. >> >> Regarding the The ipu1_csi0_mux/ipu2_csi1_mux entities which have 1 >> source and 2 sinks (which makes sense for a mux) how do you know which >> sink pad you should use (in your adv7180 example you use the 2nd sink >> pad vs the first)? > > > The adv7180 can only go to the parallel input pad (ipu1_csi0_mux:1 > on quad). The other input pads select from the mipi csi-2 receiver virtual > channels. right - my question was how does the user know which pad is which. I see that the imx6q.dtsi makes it clear that port0 (sink1) is the mipi port and port1 is the parallel port (sink2). Do you know how a user would determine this from runtime information (maybe something via media-ctl or /sys/class/media that I haven't yet found) or perhaps this is to be taken care of by documentation or referring to the dts? > > Have you generated a dot graph? It makes it much easier to > visualize: > > # media-ctl --print-dot > graph.dot > > then on your host: > > % dot -Tpng graph.dot > graph.png > Yes - that makes it much easier to understand the possible links. I notice 'media-ctl --print-topology' shows link and pad type fields, but 'media-ctl --print-dot' does not include the pad type field (maybe newer versions do or will in the future) Do you know how one goes about determining the possible format types possible for each pad? > > >> >> As my hardware is the same as the SabreAuto except that my adv7180 is >> on i2c-2@0x20 I follow your example from >> Documentation/media/v4l-drivers/imx.rst: >> >> # Setup links >> media-ctl -l '"adv7180 2-0020":0 -> "ipu1_csi0_mux":1[1]' >> media-ctl -l '"ipu1_csi0_mux":2 -> "ipu1_csi0":0[1]' >> media-ctl -l '"ipu1_csi0":1 -> "ipu1_smfc0":0[1]' >> media-ctl -l '"ipu1_smfc0":1 -> "ipu1_ic_prpvf":0[1]' >> media-ctl -l '"ipu1_ic_prpvf":1 -> "camif0":0[1]' >> media-ctl -l '"camif0":1 -> "camif0 devnode":0[1]' >> >> # Configure pads >> media-ctl -V "\"adv7180 2-0020\":0 [fmt:UYVY2X8/720x480]" >> media-ctl -V "\"ipu1_csi0_mux\":1 [fmt:UYVY2X8/720x480]" >> media-ctl -V "\"ipu1_csi0_mux\":2 [fmt:UYVY2X8/720x480]" >> media-ctl -V "\"ipu1_csi0\":0 [fmt:UYVY2X8/720x480]" >> media-ctl -V "\"ipu1_csi0\":1 [fmt:UYVY2X8/720x480]" >> media-ctl -V "\"ipu1_smfc0\":0 [fmt:UYVY2X8/720x480]" >> media-ctl -V "\"ipu1_smfc0\":1 [fmt:UYVY2X8/720x480]" >> media-ctl -V "\"ipu1_ic_prpvf\":0 [fmt:UYVY2X8/720x480]" >> # pad field types for camif can be any format prpvf supports >> export outputfmt="UYVY2X8/720x480" >> media-ctl -V "\"ipu1_ic_prpvf\":1 [fmt:$outputfmt]" >> media-ctl -V "\"camif0\":0 [fmt:$outputfmt]" >> media-ctl -V "\"camif0\":1 [fmt:$outputfmt]" >> >> # select AIN1 >> v4l2-ctl -d0 -i0 >> Video input set to 0 (ADV7180 Composite on Ain1: ok) >> v4l2-ctl -d0 --set-fmt-video=width=720,height=480,pixelformat=UYVY >> # capture a single raw frame >> v4l2-ctl -d0 --stream-mmap --stream-to=/x.raw --stream-count=1 >> [ 2092.056394] camif0: pipeline_set_stream failed with -32 >> VIDIOC_STREAMON: failed: Broken pipe >> >> Enabling debug in drivers/media/media-entity.c I see: >> [ 38.870087] imx-media soc:media@0: link validation failed for >> "ipu1_smfc0":1 -> "ipu1_ic_prpvf":0, error -32 >> >> Looking at ipu1_smfc0 and ipu1_ic_prpvf with media-ctl I see: >> - entity 12: ipu1_ic_prpvf (2 pads, 8 links) >> type V4L2 subdev subtype Unknown flags 0 >> device node name /dev/v4l-subdev3 >> pad0: Sink >> [fmt:UYVY2X8/720x480 field:alternate] >> <- "ipu1_csi0":1 [] >> <- "ipu1_csi1":1 [] >> <- "ipu1_smfc0":1 [ENABLED] >> <- "ipu1_smfc1":1 [] >> pad1: Source >> [fmt:UYVY2X8/720x480 field:none] >> -> "camif0":0 [ENABLED] >> -> "camif1":0 [] >> -> "ipu1_ic_pp0":0 [] >> -> "ipu1_ic_pp1":0 [] >> >> - entity 45: ipu1_smfc0 (2 pads, 5 links) >> type V4L2 subdev subtype Unknown flags 0 >> device node name /dev/v4l-subdev14 >> pad0: Sink >> [fmt:UYVY2X8/720x480] >> <- "ipu1_csi0":1 [ENABLED] >> pad1: Source >> [fmt:UYVY2X8/720x480] >> -> "ipu1_ic_prpvf":0 [ENABLED] >> -> "ipu1_ic_pp0":0 [] >> -> "camif0":0 [] >> -> "camif1":0 [] >> >> Any ideas what is going wrong here? Seems like its perhaps a field >> type mismatch. > > > Yes, exactly, you'll need to set the field types on every pad in your > pipeline. > >> Is my outputfmt incorrect perhaps? I likely have >> misunderstood the pad type comments in your documentation. > > > Attached is an update doc (from branch imx-media-staging-md-v7 on my fork). > I recently upgraded my v4l-utils package and media-ctl now supports > specifying > the field type in the pad format strings. If you don't have the latest > v4l-utils, it's > fairly straightforward to cross-build. > Your updated imx.rst makes it very clear - I was misunderstanding the pervious version and comments as it skipped setting the pad field types. The v4l-utils-1.10 from Ubuntu 16.04 allows setting field types with the links so I didn't need to build a newer one and this did resolve my issue. >> >>> >>>> <snip> >>>> >>>> >>>> >>>> Additionally I've found that on an IMX6S/IMX6DL we crash while >>>> registering the media-ic subdev's: >> >> <snip> >>> >>> Yep, I only have quad boards here so I haven't gotten around to >>> testing on S/DL. >>> >>> But it looks like I forgot to clear out the csi subdev pointer array >>> before >>> passing it to imx_media_of_parse(). I think that might explain the OOPS >>> above. Try this patch: >>> >>> diff --git a/drivers/staging/media/imx/imx-media-dev.c >>> b/drivers/staging/media/imx/imx-media-dev.c >>> index 357654d..0cf2d61 100644 >>> --- a/drivers/staging/media/imx/imx-media-dev.c >>> +++ b/drivers/staging/media/imx/imx-media-dev.c >>> @@ -379,7 +379,7 @@ static int imx_media_probe(struct platform_device >>> *pdev) >>> { >>> struct device *dev = &pdev->dev; >>> struct device_node *node = dev->of_node; >>> - struct imx_media_subdev *csi[4]; >>> + struct imx_media_subdev *csi[4] = {0}; >>> struct imx_media_dev *imxmd; >>> int ret; >>> >> This does resolves the crash on S/DL. > > I now have dts patches ready for the following which have an on-board ADV7180 SD capture on: arch/arm/boot/dts/imx6dl-gw52xx.dts arch/arm/boot/dts/imx6dl-gw53xx.dts arch/arm/boot/dts/imx6dl-gw54xx.dts arch/arm/boot/dts/imx6q-gw52xx.dts arch/arm/boot/dts/imx6q-gw53xx.dts arch/arm/boot/dts/imx6q-gw54xx.dts arch/arm/boot/dts/imx6qdl-gw51xx.dtsi arch/arm/boot/dts/imx6qdl-gw553x.dtsi So for the above which I've tested with the 'sensor -> ipu_csi_mux -> ipu_csi -> ipu_smfc -> ipu_ic_prpvf -> camif' pipeline Tested-by: Tim Harvey <tharvey@gateworks.com> Thanks for all your continued effort on these drivers! Tim -- To unsubscribe from this list: send the line "unsubscribe linux-media" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
diff --git a/drivers/staging/media/imx/imx-media-dev.c b/drivers/staging/media/imx/imx-media-dev.c index 357654d..0cf2d61 100644 --- a/drivers/staging/media/imx/imx-media-dev.c +++ b/drivers/staging/media/imx/imx-media-dev.c @@ -379,7 +379,7 @@ static int imx_media_probe(struct platform_device *pdev) { struct device *dev = &pdev->dev; struct device_node *node = dev->of_node; - struct imx_media_subdev *csi[4]; + struct imx_media_subdev *csi[4] = {0}; struct imx_media_dev *imxmd; int ret;