| Message ID | cover.8ec406bf8f4f097e9dc909d5aac466556822f592.1555487650.git-series.maxime.ripard@bootlin.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | drm: Split out the formats API and move it to a common place | expand |
Hi, On Wed, 2019-04-17 at 09:54 +0200, Maxime Ripard wrote: > Hi, > > DRM comes with an extensive format support to retrieve the various > parameters associated with a given format (such as the subsampling, or the > bits per pixel), as well as some helpers and utilities to ease the driver > development. > > v4l2, on the other side, doesn't provide such facilities, leaving each > driver reimplement a subset of the formats parameters for the one supported > by that particular driver. This leads to a lot of duplication and > boilerplate code in the v4l2 drivers. > > This series tries to address this by moving the DRM format API into lib and > turning it into a more generic API. In order to do this, we've needed to do > some preliminary changes on the DRM drivers, then moved the API and finally > converted a v4l2 driver to give an example of how such library could be > used. > > Let me know what you think, > Maxime Here is one general suggestion: calling the new library and functions and formats "pixel format" instead of "image format" since this is the most widely-used terminology (see the pixel format guide) and also the one that (personally) feels most accurate. What do you think? Cheers, Paul > Changes from RFC: > - Rebased on next > - Fixed the various formats mapping > - Added tags > - Did most of the formats functions as inline functions > - Used a consistent prefix for all the utilities functions > - Fixed the compilation breakages, and did a run of allmodconfig for arm, > arm64 and x86_64 > - Fixed out of array bounds warnings in the image_format_info_block_* > functions > - Added License and copyright headers on missing files > > Maxime Ripard (20): > drm: Remove users of drm_format_num_planes > drm: Remove users of drm_format_(horz|vert)_chroma_subsampling > drm/fourcc: Pass the format_info pointer to drm_format_plane_cpp > drm/fourcc: Pass the format_info pointer to drm_format_plane_width/height > drm: Replace instances of drm_format_info by drm_get_format_info > lib: Add video format information library > drm/fb: Move from drm_format_info to image_format_info > drm/malidp: Convert to generic image format library > drm/client: Convert to generic image format library > drm/exynos: Convert to generic image format library > drm/i915: Convert to generic image format library > drm/ipuv3: Convert to generic image format library > drm/msm: Convert to generic image format library > drm/omap: Convert to generic image format library > drm/rockchip: Convert to generic image format library > drm/tegra: Convert to generic image format library > drm/fourcc: Remove old DRM format API > lib: image-formats: Add v4l2 formats support > lib: image-formats: Add more functions > media: sun6i: Convert to the image format API > > drivers/gpu/drm/Kconfig | 2 +- > drivers/gpu/drm/amd/amdgpu/amdgpu_fb.c | 4 +- > drivers/gpu/drm/arm/malidp_drv.c | 3 +- > drivers/gpu/drm/arm/malidp_hw.c | 3 +- > drivers/gpu/drm/arm/malidp_mw.c | 2 +- > drivers/gpu/drm/arm/malidp_planes.c | 20 +- > drivers/gpu/drm/armada/armada_fb.c | 3 +- > drivers/gpu/drm/armada/armada_overlay.c | 2 +- > drivers/gpu/drm/armada/armada_plane.c | 2 +- > drivers/gpu/drm/atmel-hlcdc/atmel_hlcdc_plane.c | 11 +- > drivers/gpu/drm/bochs/bochs.h | 4 +- > drivers/gpu/drm/bochs/bochs_hw.c | 2 +- > drivers/gpu/drm/drm_client.c | 7 +- > drivers/gpu/drm/drm_fb_cma_helper.c | 4 +- > drivers/gpu/drm/drm_fb_helper.c | 14 +- > drivers/gpu/drm/drm_format_helper.c | 4 +- > drivers/gpu/drm/drm_fourcc.c | 345 +----- > drivers/gpu/drm/drm_framebuffer.c | 10 +- > drivers/gpu/drm/drm_gem_framebuffer_helper.c | 4 +- > drivers/gpu/drm/exynos/exynos_drm_fb.c | 2 +- > drivers/gpu/drm/exynos/exynos_drm_ipp.c | 2 +- > drivers/gpu/drm/exynos/exynos_drm_ipp.h | 4 +- > drivers/gpu/drm/exynos/exynos_drm_scaler.c | 3 +- > drivers/gpu/drm/gma500/framebuffer.c | 4 +- > drivers/gpu/drm/i915/i915_drv.h | 6 +- > drivers/gpu/drm/i915/intel_display.c | 14 +- > drivers/gpu/drm/i915/intel_pm.c | 6 +- > drivers/gpu/drm/i915/intel_sprite.c | 3 +- > drivers/gpu/drm/imx/ipuv3-plane.c | 19 +- > drivers/gpu/drm/mediatek/mtk_drm_fb.c | 8 +- > drivers/gpu/drm/meson/meson_overlay.c | 14 +- > drivers/gpu/drm/msm/disp/dpu1/dpu_formats.c | 11 +- > drivers/gpu/drm/msm/disp/dpu1/dpu_plane.c | 10 +- > drivers/gpu/drm/msm/disp/mdp5/mdp5_crtc.c | 4 +- > drivers/gpu/drm/msm/disp/mdp5/mdp5_plane.c | 25 +- > drivers/gpu/drm/msm/disp/mdp5/mdp5_smp.c | 8 +- > drivers/gpu/drm/msm/msm_fb.c | 18 +- > drivers/gpu/drm/omapdrm/dss/dispc.c | 9 +- > drivers/gpu/drm/omapdrm/omap_fb.c | 15 +- > drivers/gpu/drm/radeon/radeon_fb.c | 4 +- > drivers/gpu/drm/rockchip/rockchip_drm_fb.c | 17 +- > drivers/gpu/drm/rockchip/rockchip_drm_vop.c | 14 +- > drivers/gpu/drm/selftests/Makefile | 3 +- > drivers/gpu/drm/selftests/drm_modeset_selftests.h | 3 +- > drivers/gpu/drm/selftests/test-drm_format.c | 280 +---- > drivers/gpu/drm/selftests/test-drm_modeset_common.h | 3 +- > drivers/gpu/drm/stm/ltdc.c | 2 +- > drivers/gpu/drm/sun4i/sun4i_backend.c | 6 +- > drivers/gpu/drm/sun4i/sun4i_frontend.c | 18 +- > drivers/gpu/drm/sun4i/sun8i_vi_layer.c | 4 +- > drivers/gpu/drm/sun4i/sun8i_vi_scaler.c | 6 +- > drivers/gpu/drm/sun4i/sun8i_vi_scaler.h | 5 +- > drivers/gpu/drm/tegra/fb.c | 14 +- > drivers/gpu/drm/tegra/plane.c | 4 +- > drivers/gpu/drm/vc4/vc4_plane.c | 15 +- > drivers/gpu/drm/zte/zx_plane.c | 6 +- > drivers/gpu/ipu-v3/ipu-pre.c | 3 +- > drivers/gpu/ipu-v3/ipu-prg.c | 3 +- > drivers/media/platform/sunxi/sun6i-csi/sun6i_csi.c | 89 +- > drivers/media/platform/sunxi/sun6i-csi/sun6i_csi.h | 49 +- > drivers/media/platform/sunxi/sun6i-csi/sun6i_csi_reg.h | 6 +- > drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c | 11 +- > include/drm/drm_fourcc.h | 219 +--- > include/drm/drm_framebuffer.h | 4 +- > include/drm/drm_mode_config.h | 5 +- > include/linux/image-formats.h | 438 ++++++- > lib/Kconfig | 7 +- > lib/Makefile | 3 +- > lib/image-formats-selftests.c | 325 +++++- > lib/image-formats.c | 717 ++++++++++- > 70 files changed, 1740 insertions(+), 1174 deletions(-) > delete mode 100644 drivers/gpu/drm/selftests/test-drm_format.c > create mode 100644 include/linux/image-formats.h > create mode 100644 lib/image-formats-selftests.c > create mode 100644 lib/image-formats.c > > base-commit: de3c659c83ce88d495f2a36397071fdd0a337345
Hi, On Wed, 2019-04-17 at 09:54 +0200, Maxime Ripard wrote: > Hi, > > DRM comes with an extensive format support to retrieve the various > parameters associated with a given format (such as the subsampling, or the > bits per pixel), as well as some helpers and utilities to ease the driver > development. > > v4l2, on the other side, doesn't provide such facilities, leaving each > driver reimplement a subset of the formats parameters for the one supported > by that particular driver. This leads to a lot of duplication and > boilerplate code in the v4l2 drivers. > > This series tries to address this by moving the DRM format API into lib and > turning it into a more generic API. In order to do this, we've needed to do > some preliminary changes on the DRM drivers, then moved the API and finally > converted a v4l2 driver to give an example of how such library could be > used. > > Let me know what you think, Another thing to consider would be using unsigned ints for the plane index so we can avoid < 0 checks all over the place. What do you think? Cheers, Paul > Maxime > > Changes from RFC: > - Rebased on next > - Fixed the various formats mapping > - Added tags > - Did most of the formats functions as inline functions > - Used a consistent prefix for all the utilities functions > - Fixed the compilation breakages, and did a run of allmodconfig for arm, > arm64 and x86_64 > - Fixed out of array bounds warnings in the image_format_info_block_* > functions > - Added License and copyright headers on missing files > > Maxime Ripard (20): > drm: Remove users of drm_format_num_planes > drm: Remove users of drm_format_(horz|vert)_chroma_subsampling > drm/fourcc: Pass the format_info pointer to drm_format_plane_cpp > drm/fourcc: Pass the format_info pointer to drm_format_plane_width/height > drm: Replace instances of drm_format_info by drm_get_format_info > lib: Add video format information library > drm/fb: Move from drm_format_info to image_format_info > drm/malidp: Convert to generic image format library > drm/client: Convert to generic image format library > drm/exynos: Convert to generic image format library > drm/i915: Convert to generic image format library > drm/ipuv3: Convert to generic image format library > drm/msm: Convert to generic image format library > drm/omap: Convert to generic image format library > drm/rockchip: Convert to generic image format library > drm/tegra: Convert to generic image format library > drm/fourcc: Remove old DRM format API > lib: image-formats: Add v4l2 formats support > lib: image-formats: Add more functions > media: sun6i: Convert to the image format API > > drivers/gpu/drm/Kconfig | 2 +- > drivers/gpu/drm/amd/amdgpu/amdgpu_fb.c | 4 +- > drivers/gpu/drm/arm/malidp_drv.c | 3 +- > drivers/gpu/drm/arm/malidp_hw.c | 3 +- > drivers/gpu/drm/arm/malidp_mw.c | 2 +- > drivers/gpu/drm/arm/malidp_planes.c | 20 +- > drivers/gpu/drm/armada/armada_fb.c | 3 +- > drivers/gpu/drm/armada/armada_overlay.c | 2 +- > drivers/gpu/drm/armada/armada_plane.c | 2 +- > drivers/gpu/drm/atmel-hlcdc/atmel_hlcdc_plane.c | 11 +- > drivers/gpu/drm/bochs/bochs.h | 4 +- > drivers/gpu/drm/bochs/bochs_hw.c | 2 +- > drivers/gpu/drm/drm_client.c | 7 +- > drivers/gpu/drm/drm_fb_cma_helper.c | 4 +- > drivers/gpu/drm/drm_fb_helper.c | 14 +- > drivers/gpu/drm/drm_format_helper.c | 4 +- > drivers/gpu/drm/drm_fourcc.c | 345 +----- > drivers/gpu/drm/drm_framebuffer.c | 10 +- > drivers/gpu/drm/drm_gem_framebuffer_helper.c | 4 +- > drivers/gpu/drm/exynos/exynos_drm_fb.c | 2 +- > drivers/gpu/drm/exynos/exynos_drm_ipp.c | 2 +- > drivers/gpu/drm/exynos/exynos_drm_ipp.h | 4 +- > drivers/gpu/drm/exynos/exynos_drm_scaler.c | 3 +- > drivers/gpu/drm/gma500/framebuffer.c | 4 +- > drivers/gpu/drm/i915/i915_drv.h | 6 +- > drivers/gpu/drm/i915/intel_display.c | 14 +- > drivers/gpu/drm/i915/intel_pm.c | 6 +- > drivers/gpu/drm/i915/intel_sprite.c | 3 +- > drivers/gpu/drm/imx/ipuv3-plane.c | 19 +- > drivers/gpu/drm/mediatek/mtk_drm_fb.c | 8 +- > drivers/gpu/drm/meson/meson_overlay.c | 14 +- > drivers/gpu/drm/msm/disp/dpu1/dpu_formats.c | 11 +- > drivers/gpu/drm/msm/disp/dpu1/dpu_plane.c | 10 +- > drivers/gpu/drm/msm/disp/mdp5/mdp5_crtc.c | 4 +- > drivers/gpu/drm/msm/disp/mdp5/mdp5_plane.c | 25 +- > drivers/gpu/drm/msm/disp/mdp5/mdp5_smp.c | 8 +- > drivers/gpu/drm/msm/msm_fb.c | 18 +- > drivers/gpu/drm/omapdrm/dss/dispc.c | 9 +- > drivers/gpu/drm/omapdrm/omap_fb.c | 15 +- > drivers/gpu/drm/radeon/radeon_fb.c | 4 +- > drivers/gpu/drm/rockchip/rockchip_drm_fb.c | 17 +- > drivers/gpu/drm/rockchip/rockchip_drm_vop.c | 14 +- > drivers/gpu/drm/selftests/Makefile | 3 +- > drivers/gpu/drm/selftests/drm_modeset_selftests.h | 3 +- > drivers/gpu/drm/selftests/test-drm_format.c | 280 +---- > drivers/gpu/drm/selftests/test-drm_modeset_common.h | 3 +- > drivers/gpu/drm/stm/ltdc.c | 2 +- > drivers/gpu/drm/sun4i/sun4i_backend.c | 6 +- > drivers/gpu/drm/sun4i/sun4i_frontend.c | 18 +- > drivers/gpu/drm/sun4i/sun8i_vi_layer.c | 4 +- > drivers/gpu/drm/sun4i/sun8i_vi_scaler.c | 6 +- > drivers/gpu/drm/sun4i/sun8i_vi_scaler.h | 5 +- > drivers/gpu/drm/tegra/fb.c | 14 +- > drivers/gpu/drm/tegra/plane.c | 4 +- > drivers/gpu/drm/vc4/vc4_plane.c | 15 +- > drivers/gpu/drm/zte/zx_plane.c | 6 +- > drivers/gpu/ipu-v3/ipu-pre.c | 3 +- > drivers/gpu/ipu-v3/ipu-prg.c | 3 +- > drivers/media/platform/sunxi/sun6i-csi/sun6i_csi.c | 89 +- > drivers/media/platform/sunxi/sun6i-csi/sun6i_csi.h | 49 +- > drivers/media/platform/sunxi/sun6i-csi/sun6i_csi_reg.h | 6 +- > drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c | 11 +- > include/drm/drm_fourcc.h | 219 +--- > include/drm/drm_framebuffer.h | 4 +- > include/drm/drm_mode_config.h | 5 +- > include/linux/image-formats.h | 438 ++++++- > lib/Kconfig | 7 +- > lib/Makefile | 3 +- > lib/image-formats-selftests.c | 325 +++++- > lib/image-formats.c | 717 ++++++++++- > 70 files changed, 1740 insertions(+), 1174 deletions(-) > delete mode 100644 drivers/gpu/drm/selftests/test-drm_format.c > create mode 100644 include/linux/image-formats.h > create mode 100644 lib/image-formats-selftests.c > create mode 100644 lib/image-formats.c > > base-commit: de3c659c83ce88d495f2a36397071fdd0a337345
On Wed, Apr 17, 2019 at 09:54:26AM +0200, Maxime Ripard wrote: > Hi, > > DRM comes with an extensive format support to retrieve the various > parameters associated with a given format (such as the subsampling, or the > bits per pixel), as well as some helpers and utilities to ease the driver > development. > > v4l2, on the other side, doesn't provide such facilities, leaving each > driver reimplement a subset of the formats parameters for the one supported > by that particular driver. This leads to a lot of duplication and > boilerplate code in the v4l2 drivers. > > This series tries to address this by moving the DRM format API into lib and > turning it into a more generic API. In order to do this, we've needed to do > some preliminary changes on the DRM drivers, then moved the API and finally > converted a v4l2 driver to give an example of how such library could be > used. > > Let me know what you think, > Maxime > > Changes from RFC: > - Rebased on next > - Fixed the various formats mapping > - Added tags > - Did most of the formats functions as inline functions > - Used a consistent prefix for all the utilities functions > - Fixed the compilation breakages, and did a run of allmodconfig for arm, > arm64 and x86_64 > - Fixed out of array bounds warnings in the image_format_info_block_* > functions > - Added License and copyright headers on missing files > > Maxime Ripard (20): > drm: Remove users of drm_format_num_planes > drm: Remove users of drm_format_(horz|vert)_chroma_subsampling > drm/fourcc: Pass the format_info pointer to drm_format_plane_cpp > drm/fourcc: Pass the format_info pointer to drm_format_plane_width/height > drm: Replace instances of drm_format_info by drm_get_format_info > lib: Add video format information library > drm/fb: Move from drm_format_info to image_format_info > drm/malidp: Convert to generic image format library > drm/client: Convert to generic image format library > drm/exynos: Convert to generic image format library > drm/i915: Convert to generic image format library > drm/ipuv3: Convert to generic image format library > drm/msm: Convert to generic image format library > drm/omap: Convert to generic image format library > drm/rockchip: Convert to generic image format library > drm/tegra: Convert to generic image format library > drm/fourcc: Remove old DRM format API > lib: image-formats: Add v4l2 formats support > lib: image-formats: Add more functions > media: sun6i: Convert to the image format API In the interest of making myself unpopular: Why move this out of drm? We do have a bunch of other such shared helpers already (mostly in drivers/video) for dt videomode and hdmi infoframes, and I'm not super sure that's going better than keeping it maintained in drm. Plus the uapi is already that you include drm_fourcc.h to get at these, dropping the drm prefix isn't going to help I think. Of course we'd need to make it a separate drm_formats.ko (so that v4l can use it without dragging in all of drm), and we need to add some fields for converting to v4l fourcc codes (which are different). But that should be all possible. And I don't think the drm_ prefix in v4l code is a problem, it's actually a feature: It makes it really clear that these are the drm fourcc codes, as allocated in drm_fourcc.h, plus their modifiers, and all that. That allocation authority is also already baked into various khr/ext standards, too. Ofc it'll perpetuate the joke that drm has no meaning anymore as an abbrevation, but oh welp :-) -Daniel > > drivers/gpu/drm/Kconfig | 2 +- > drivers/gpu/drm/amd/amdgpu/amdgpu_fb.c | 4 +- > drivers/gpu/drm/arm/malidp_drv.c | 3 +- > drivers/gpu/drm/arm/malidp_hw.c | 3 +- > drivers/gpu/drm/arm/malidp_mw.c | 2 +- > drivers/gpu/drm/arm/malidp_planes.c | 20 +- > drivers/gpu/drm/armada/armada_fb.c | 3 +- > drivers/gpu/drm/armada/armada_overlay.c | 2 +- > drivers/gpu/drm/armada/armada_plane.c | 2 +- > drivers/gpu/drm/atmel-hlcdc/atmel_hlcdc_plane.c | 11 +- > drivers/gpu/drm/bochs/bochs.h | 4 +- > drivers/gpu/drm/bochs/bochs_hw.c | 2 +- > drivers/gpu/drm/drm_client.c | 7 +- > drivers/gpu/drm/drm_fb_cma_helper.c | 4 +- > drivers/gpu/drm/drm_fb_helper.c | 14 +- > drivers/gpu/drm/drm_format_helper.c | 4 +- > drivers/gpu/drm/drm_fourcc.c | 345 +----- > drivers/gpu/drm/drm_framebuffer.c | 10 +- > drivers/gpu/drm/drm_gem_framebuffer_helper.c | 4 +- > drivers/gpu/drm/exynos/exynos_drm_fb.c | 2 +- > drivers/gpu/drm/exynos/exynos_drm_ipp.c | 2 +- > drivers/gpu/drm/exynos/exynos_drm_ipp.h | 4 +- > drivers/gpu/drm/exynos/exynos_drm_scaler.c | 3 +- > drivers/gpu/drm/gma500/framebuffer.c | 4 +- > drivers/gpu/drm/i915/i915_drv.h | 6 +- > drivers/gpu/drm/i915/intel_display.c | 14 +- > drivers/gpu/drm/i915/intel_pm.c | 6 +- > drivers/gpu/drm/i915/intel_sprite.c | 3 +- > drivers/gpu/drm/imx/ipuv3-plane.c | 19 +- > drivers/gpu/drm/mediatek/mtk_drm_fb.c | 8 +- > drivers/gpu/drm/meson/meson_overlay.c | 14 +- > drivers/gpu/drm/msm/disp/dpu1/dpu_formats.c | 11 +- > drivers/gpu/drm/msm/disp/dpu1/dpu_plane.c | 10 +- > drivers/gpu/drm/msm/disp/mdp5/mdp5_crtc.c | 4 +- > drivers/gpu/drm/msm/disp/mdp5/mdp5_plane.c | 25 +- > drivers/gpu/drm/msm/disp/mdp5/mdp5_smp.c | 8 +- > drivers/gpu/drm/msm/msm_fb.c | 18 +- > drivers/gpu/drm/omapdrm/dss/dispc.c | 9 +- > drivers/gpu/drm/omapdrm/omap_fb.c | 15 +- > drivers/gpu/drm/radeon/radeon_fb.c | 4 +- > drivers/gpu/drm/rockchip/rockchip_drm_fb.c | 17 +- > drivers/gpu/drm/rockchip/rockchip_drm_vop.c | 14 +- > drivers/gpu/drm/selftests/Makefile | 3 +- > drivers/gpu/drm/selftests/drm_modeset_selftests.h | 3 +- > drivers/gpu/drm/selftests/test-drm_format.c | 280 +---- > drivers/gpu/drm/selftests/test-drm_modeset_common.h | 3 +- > drivers/gpu/drm/stm/ltdc.c | 2 +- > drivers/gpu/drm/sun4i/sun4i_backend.c | 6 +- > drivers/gpu/drm/sun4i/sun4i_frontend.c | 18 +- > drivers/gpu/drm/sun4i/sun8i_vi_layer.c | 4 +- > drivers/gpu/drm/sun4i/sun8i_vi_scaler.c | 6 +- > drivers/gpu/drm/sun4i/sun8i_vi_scaler.h | 5 +- > drivers/gpu/drm/tegra/fb.c | 14 +- > drivers/gpu/drm/tegra/plane.c | 4 +- > drivers/gpu/drm/vc4/vc4_plane.c | 15 +- > drivers/gpu/drm/zte/zx_plane.c | 6 +- > drivers/gpu/ipu-v3/ipu-pre.c | 3 +- > drivers/gpu/ipu-v3/ipu-prg.c | 3 +- > drivers/media/platform/sunxi/sun6i-csi/sun6i_csi.c | 89 +- > drivers/media/platform/sunxi/sun6i-csi/sun6i_csi.h | 49 +- > drivers/media/platform/sunxi/sun6i-csi/sun6i_csi_reg.h | 6 +- > drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c | 11 +- > include/drm/drm_fourcc.h | 219 +--- > include/drm/drm_framebuffer.h | 4 +- > include/drm/drm_mode_config.h | 5 +- > include/linux/image-formats.h | 438 ++++++- > lib/Kconfig | 7 +- > lib/Makefile | 3 +- > lib/image-formats-selftests.c | 325 +++++- > lib/image-formats.c | 717 ++++++++++- > 70 files changed, 1740 insertions(+), 1174 deletions(-) > delete mode 100644 drivers/gpu/drm/selftests/test-drm_format.c > create mode 100644 include/linux/image-formats.h > create mode 100644 lib/image-formats-selftests.c > create mode 100644 lib/image-formats.c > > base-commit: de3c659c83ce88d495f2a36397071fdd0a337345 > -- > git-series 0.9.1 > _______________________________________________ > dri-devel mailing list > dri-devel@lists.freedesktop.org > https://lists.freedesktop.org/mailman/listinfo/dri-devel
Hi Daniel, On Wed, Apr 17, 2019 at 05:41:21PM +0200, Daniel Vetter wrote: > On Wed, Apr 17, 2019 at 09:54:26AM +0200, Maxime Ripard wrote: > > Hi, > > > > DRM comes with an extensive format support to retrieve the various > > parameters associated with a given format (such as the subsampling, or the > > bits per pixel), as well as some helpers and utilities to ease the driver > > development. > > > > v4l2, on the other side, doesn't provide such facilities, leaving each > > driver reimplement a subset of the formats parameters for the one supported > > by that particular driver. This leads to a lot of duplication and > > boilerplate code in the v4l2 drivers. > > > > This series tries to address this by moving the DRM format API into lib and > > turning it into a more generic API. In order to do this, we've needed to do > > some preliminary changes on the DRM drivers, then moved the API and finally > > converted a v4l2 driver to give an example of how such library could be > > used. > > > > Let me know what you think, > > Maxime > > > > Changes from RFC: > > - Rebased on next > > - Fixed the various formats mapping > > - Added tags > > - Did most of the formats functions as inline functions > > - Used a consistent prefix for all the utilities functions > > - Fixed the compilation breakages, and did a run of allmodconfig for arm, > > arm64 and x86_64 > > - Fixed out of array bounds warnings in the image_format_info_block_* > > functions > > - Added License and copyright headers on missing files > > > > Maxime Ripard (20): > > drm: Remove users of drm_format_num_planes > > drm: Remove users of drm_format_(horz|vert)_chroma_subsampling > > drm/fourcc: Pass the format_info pointer to drm_format_plane_cpp > > drm/fourcc: Pass the format_info pointer to drm_format_plane_width/height > > drm: Replace instances of drm_format_info by drm_get_format_info > > lib: Add video format information library > > drm/fb: Move from drm_format_info to image_format_info > > drm/malidp: Convert to generic image format library > > drm/client: Convert to generic image format library > > drm/exynos: Convert to generic image format library > > drm/i915: Convert to generic image format library > > drm/ipuv3: Convert to generic image format library > > drm/msm: Convert to generic image format library > > drm/omap: Convert to generic image format library > > drm/rockchip: Convert to generic image format library > > drm/tegra: Convert to generic image format library > > drm/fourcc: Remove old DRM format API > > lib: image-formats: Add v4l2 formats support > > lib: image-formats: Add more functions > > media: sun6i: Convert to the image format API > > In the interest of making myself unpopular: Why move this out of drm? > > We do have a bunch of other such shared helpers already (mostly in > drivers/video) for dt videomode and hdmi infoframes, and I'm not super > sure that's going better than keeping it maintained in drm. > > Plus the uapi is already that you include drm_fourcc.h to get at these, > dropping the drm prefix isn't going to help I think. > > Of course we'd need to make it a separate drm_formats.ko (so that v4l can > use it without dragging in all of drm), and we need to add some fields for > converting to v4l fourcc codes (which are different). But that should be > all possible. And I don't think the drm_ prefix in v4l code is a problem, > it's actually a feature: It makes it really clear that these are the drm > fourcc codes, as allocated in drm_fourcc.h, plus their modifiers, and all > that. That allocation authority is also already baked into various khr/ext > standards, too. The way I see it, there's a fundamental difference between the UAPI and the kernel. I don't suggest we change anything about the UAPI: the drm formats should keep their prefix, drm_fourcc.h can remain that authority, it's all fine. Internally however, the long term goal is to share the fourcc's between DRM and V4L2 for the same formats. It basically means that of course v4l2 should be using the DRM fourcc when a format exists in DRM and not v4l2, but also that DRM should use v4l2 fourcc when the format exists in v4l2 but not DRM, and that is far more likely, given the already extensive v4l2 formats support. And given how the current state is a mess in this regard, I'm not too optimistic about keeping the formats in their relevant frameworks. Having a shared library, governed by both, will make this far easier, since it will be easy to discover the formats that are already supported by the other subsystem. If we want to keep the current library in DRM, we have two options then: - Support all the v4l2 formats in the DRM library, which is essentially what I'm doing in the last patches. However, that would require to have the v4l2 developpers also reviewing stuff there. And given how busy they are, I cannot really see how that would work. - Develop the same library from within v4l2. That is really a poor solution, since the information would be greatly duplicated between the two, and in terms of maintainance, code, and binary size that would be duplicated too. Having it shared allows to easily share, and discover formats from the other subsystem, and to have a single, unique place where this is centralized. Maxime -- Maxime Ripard, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
On Thu, Apr 18, 2019 at 8:22 AM Maxime Ripard <maxime.ripard@bootlin.com> wrote: > > Hi Daniel, > > On Wed, Apr 17, 2019 at 05:41:21PM +0200, Daniel Vetter wrote: > > On Wed, Apr 17, 2019 at 09:54:26AM +0200, Maxime Ripard wrote: > > > Hi, > > > > > > DRM comes with an extensive format support to retrieve the various > > > parameters associated with a given format (such as the subsampling, or the > > > bits per pixel), as well as some helpers and utilities to ease the driver > > > development. > > > > > > v4l2, on the other side, doesn't provide such facilities, leaving each > > > driver reimplement a subset of the formats parameters for the one supported > > > by that particular driver. This leads to a lot of duplication and > > > boilerplate code in the v4l2 drivers. > > > > > > This series tries to address this by moving the DRM format API into lib and > > > turning it into a more generic API. In order to do this, we've needed to do > > > some preliminary changes on the DRM drivers, then moved the API and finally > > > converted a v4l2 driver to give an example of how such library could be > > > used. > > > > > > Let me know what you think, > > > Maxime > > > > > > Changes from RFC: > > > - Rebased on next > > > - Fixed the various formats mapping > > > - Added tags > > > - Did most of the formats functions as inline functions > > > - Used a consistent prefix for all the utilities functions > > > - Fixed the compilation breakages, and did a run of allmodconfig for arm, > > > arm64 and x86_64 > > > - Fixed out of array bounds warnings in the image_format_info_block_* > > > functions > > > - Added License and copyright headers on missing files > > > > > > Maxime Ripard (20): > > > drm: Remove users of drm_format_num_planes > > > drm: Remove users of drm_format_(horz|vert)_chroma_subsampling > > > drm/fourcc: Pass the format_info pointer to drm_format_plane_cpp > > > drm/fourcc: Pass the format_info pointer to drm_format_plane_width/height > > > drm: Replace instances of drm_format_info by drm_get_format_info > > > lib: Add video format information library > > > drm/fb: Move from drm_format_info to image_format_info > > > drm/malidp: Convert to generic image format library > > > drm/client: Convert to generic image format library > > > drm/exynos: Convert to generic image format library > > > drm/i915: Convert to generic image format library > > > drm/ipuv3: Convert to generic image format library > > > drm/msm: Convert to generic image format library > > > drm/omap: Convert to generic image format library > > > drm/rockchip: Convert to generic image format library > > > drm/tegra: Convert to generic image format library > > > drm/fourcc: Remove old DRM format API > > > lib: image-formats: Add v4l2 formats support > > > lib: image-formats: Add more functions > > > media: sun6i: Convert to the image format API > > > > In the interest of making myself unpopular: Why move this out of drm? > > > > We do have a bunch of other such shared helpers already (mostly in > > drivers/video) for dt videomode and hdmi infoframes, and I'm not super > > sure that's going better than keeping it maintained in drm. > > > > Plus the uapi is already that you include drm_fourcc.h to get at these, > > dropping the drm prefix isn't going to help I think. > > > > Of course we'd need to make it a separate drm_formats.ko (so that v4l can > > use it without dragging in all of drm), and we need to add some fields for > > converting to v4l fourcc codes (which are different). But that should be > > all possible. And I don't think the drm_ prefix in v4l code is a problem, > > it's actually a feature: It makes it really clear that these are the drm > > fourcc codes, as allocated in drm_fourcc.h, plus their modifiers, and all > > that. That allocation authority is also already baked into various khr/ext > > standards, too. > > The way I see it, there's a fundamental difference between the UAPI > and the kernel. I don't suggest we change anything about the UAPI: the > drm formats should keep their prefix, drm_fourcc.h can remain that > authority, it's all fine. > > Internally however, the long term goal is to share the fourcc's > between DRM and V4L2 for the same formats. It basically means that of > course v4l2 should be using the DRM fourcc when a format exists in DRM > and not v4l2, but also that DRM should use v4l2 fourcc when the format > exists in v4l2 but not DRM, and that is far more likely, given the > already extensive v4l2 formats support. Uh no. We did look at v4l fourcc extensively when deciding upon a drm format identifier space. And a lot of people pushed for the "fourcc is a standard", when really it's totally not. v4l tends to conflate pixel format with stuff that we tend to encode in modifiers a lot more. There's a bunch of reasons we can't just use v4l, and they're as valid as ever: - We overlap badly in some areas, so even if fourcc codes match, we can't use them and need a duplicated DRM_FOURCC code. - v4l encodes some metadata into the fourcc that we encode elsewhere, e.g. offset for planar yuv formats, or tiling mode - drm fourcc code doesn't actually define the drm_format_info uniquely, drivers can override that (that's an explicit design intent of modifiers, to allow drivers to add another plane for e.g. compression information). You'd need to pull that driver knowledge into your format library. Iow there's no way we can easily adopt v4l fourcc, except if we do something like a new addfb flag. > And given how the current state is a mess in this regard, I'm not too > optimistic about keeping the formats in their relevant frameworks. > > Having a shared library, governed by both, will make this far easier, > since it will be easy to discover the formats that are already > supported by the other subsystem. I think a compat library that (tries to, best effort) convert between v4l and drm fourcc would make sense. Somewhere in drivers/video, next to the conversion functions for videomode <-> drm_display_mode perhaps. That should be useful for drivers. Unifying the formats themselves, and all the associated metadata is imo a no-go, and was a pretty conscious decision when we implemented drm_fourcc a few years back. > If we want to keep the current library in DRM, we have two options > then: > > - Support all the v4l2 formats in the DRM library, which is > essentially what I'm doing in the last patches. However, that > would require to have the v4l2 developpers also reviewing stuff > there. And given how busy they are, I cannot really see how that > would work. Well, if we end up with a common library then yes we need cross review. There's no way around that. Doesn't matter where exactly that library is in the filesystem tree, and adding a special MAINTAINERS entry for anything related to fourcc (both drm and v4l) to make sure they get cross-posted is easy. No file renaming needed. > - Develop the same library from within v4l2. That is really a poor > solution, since the information would be greatly duplicated > between the two, and in terms of maintainance, code, and binary > size that would be duplicated too. It's essentially what we decided to do for drm years back. > Having it shared allows to easily share, and discover formats from the > other subsystem, and to have a single, unique place where this is > centralized. What I think could work as middle ground: - Put drm_format stuff into a separate .ko - Add a MAINTAINERS entry to make sure all things fourcc are cross posted to both drm and v4l lists. Easy on the drm side, since it's all separate files. Not sure it's so convenient for v4l uapi. - Add a conversion library that tries to best-effort map between drm and v4l formats. On the drm side that most likely means you need offsets for planes, and modifiers too (since those are implied in some v4l fourcc). Emphasis on "best effort" i.e. only support as much as the drivers that use this library need. - Add drm_fourcc as-needed by these drivers that want to use a unified format space. Forcing this unification on everyone otoh is imo way too much. -Daniel -- Daniel Vetter Software Engineer, Intel Corporation +41 (0) 79 365 57 48 - http://blog.ffwll.ch
On Thu, Apr 18, 2019 at 09:52:10AM +0200, Daniel Vetter wrote: > On Thu, Apr 18, 2019 at 8:22 AM Maxime Ripard <maxime.ripard@bootlin.com> wrote: > > On Wed, Apr 17, 2019 at 05:41:21PM +0200, Daniel Vetter wrote: > > > On Wed, Apr 17, 2019 at 09:54:26AM +0200, Maxime Ripard wrote: > > > > Hi, > > > > > > > > DRM comes with an extensive format support to retrieve the various > > > > parameters associated with a given format (such as the subsampling, or the > > > > bits per pixel), as well as some helpers and utilities to ease the driver > > > > development. > > > > > > > > v4l2, on the other side, doesn't provide such facilities, leaving each > > > > driver reimplement a subset of the formats parameters for the one supported > > > > by that particular driver. This leads to a lot of duplication and > > > > boilerplate code in the v4l2 drivers. > > > > > > > > This series tries to address this by moving the DRM format API into lib and > > > > turning it into a more generic API. In order to do this, we've needed to do > > > > some preliminary changes on the DRM drivers, then moved the API and finally > > > > converted a v4l2 driver to give an example of how such library could be > > > > used. > > > > > > > > Let me know what you think, > > > > Maxime > > > > > > > > Changes from RFC: > > > > - Rebased on next > > > > - Fixed the various formats mapping > > > > - Added tags > > > > - Did most of the formats functions as inline functions > > > > - Used a consistent prefix for all the utilities functions > > > > - Fixed the compilation breakages, and did a run of allmodconfig for arm, > > > > arm64 and x86_64 > > > > - Fixed out of array bounds warnings in the image_format_info_block_* > > > > functions > > > > - Added License and copyright headers on missing files > > > > > > > > Maxime Ripard (20): > > > > drm: Remove users of drm_format_num_planes > > > > drm: Remove users of drm_format_(horz|vert)_chroma_subsampling > > > > drm/fourcc: Pass the format_info pointer to drm_format_plane_cpp > > > > drm/fourcc: Pass the format_info pointer to drm_format_plane_width/height > > > > drm: Replace instances of drm_format_info by drm_get_format_info > > > > lib: Add video format information library > > > > drm/fb: Move from drm_format_info to image_format_info > > > > drm/malidp: Convert to generic image format library > > > > drm/client: Convert to generic image format library > > > > drm/exynos: Convert to generic image format library > > > > drm/i915: Convert to generic image format library > > > > drm/ipuv3: Convert to generic image format library > > > > drm/msm: Convert to generic image format library > > > > drm/omap: Convert to generic image format library > > > > drm/rockchip: Convert to generic image format library > > > > drm/tegra: Convert to generic image format library > > > > drm/fourcc: Remove old DRM format API > > > > lib: image-formats: Add v4l2 formats support > > > > lib: image-formats: Add more functions > > > > media: sun6i: Convert to the image format API > > > > > > In the interest of making myself unpopular: Why move this out of drm? > > > > > > We do have a bunch of other such shared helpers already (mostly in > > > drivers/video) for dt videomode and hdmi infoframes, and I'm not super > > > sure that's going better than keeping it maintained in drm. > > > > > > Plus the uapi is already that you include drm_fourcc.h to get at these, > > > dropping the drm prefix isn't going to help I think. > > > > > > Of course we'd need to make it a separate drm_formats.ko (so that v4l can > > > use it without dragging in all of drm), and we need to add some fields for > > > converting to v4l fourcc codes (which are different). But that should be > > > all possible. And I don't think the drm_ prefix in v4l code is a problem, > > > it's actually a feature: It makes it really clear that these are the drm > > > fourcc codes, as allocated in drm_fourcc.h, plus their modifiers, and all > > > that. That allocation authority is also already baked into various khr/ext > > > standards, too. > > > > The way I see it, there's a fundamental difference between the UAPI > > and the kernel. I don't suggest we change anything about the UAPI: the > > drm formats should keep their prefix, drm_fourcc.h can remain that > > authority, it's all fine. > > > > Internally however, the long term goal is to share the fourcc's > > between DRM and V4L2 for the same formats. It basically means that of > > course v4l2 should be using the DRM fourcc when a format exists in DRM > > and not v4l2, but also that DRM should use v4l2 fourcc when the format > > exists in v4l2 but not DRM, and that is far more likely, given the > > already extensive v4l2 formats support. > > Uh no. We did look at v4l fourcc extensively when deciding upon a drm > format identifier space. No to what exactly? > And a lot of people pushed for the "fourcc is a standard", when > really it's totally not. Even if it's not a standard, having consistency would be a good thing. And you said yourself that DRM fourcc is now pretty much an authority for the fourcc, so it definitely looks like a standard to me. > v4l tends to conflate pixel format with stuff that we tend to encode > in modifiers a lot more. Boris is working on adding the modifiers concept to v4l2, so we're converging here, and we can totally have a layer in v4l2 to convert between old v4l2 "format+modifiers" formats, and DRM style formats. > There's a bunch of reasons we can't just use v4l, and they're as > valid as ever: > > - We overlap badly in some areas, so even if fourcc codes match, we > can't use them and need a duplicated DRM_FOURCC code. Do yo have an example of one of those areas? > - v4l encodes some metadata into the fourcc that we encode elsewhere, > e.g. offset for planar yuv formats, or tiling mode As I was saying, this changes on the v4l2 side, and converging to what DRM is doing. > - drm fourcc code doesn't actually define the drm_format_info > uniquely, drivers can override that (that's an explicit design > intent of modifiers, to allow drivers to add another plane for > e.g. compression information). You'd need to pull that driver > knowledge into your format library. I'm not sure how my patches are changing anything here. This is litterally the same code, with the functions renamed. If drivers want to override that, then yeah, fine, we can let them do that. Just like any helper this just provides a default that covers most of the cases. > Iow there's no way we can easily adopt v4l fourcc, except if we do > something like a new addfb flag. For the formats that would be described as a modifier, sure. For all the others (that are not yet supported by DRM), then I don't really see why not. > > And given how the current state is a mess in this regard, I'm not too > > optimistic about keeping the formats in their relevant frameworks. > > > > Having a shared library, governed by both, will make this far easier, > > since it will be easy to discover the formats that are already > > supported by the other subsystem. > > I think a compat library that (tries to, best effort) convert between > v4l and drm fourcc would make sense. Somewhere in drivers/video, next > to the conversion functions for videomode <-> drm_display_mode > perhaps. That should be useful for drivers. That's not really what this series is about though. That series is about sharing the (image|pixels) formats database across everyone so that everyone can benefit from it. > Unifying the formats themselves, and all the associated metadata is > imo a no-go, and was a pretty conscious decision when we implemented > drm_fourcc a few years back. > > > If we want to keep the current library in DRM, we have two options > > then: > > > > - Support all the v4l2 formats in the DRM library, which is > > essentially what I'm doing in the last patches. However, that > > would require to have the v4l2 developpers also reviewing stuff > > there. And given how busy they are, I cannot really see how that > > would work. > > Well, if we end up with a common library then yes we need cross > review. There's no way around that. Doesn't matter where exactly that > library is in the filesystem tree, and adding a special MAINTAINERS > entry for anything related to fourcc (both drm and v4l) to make sure > they get cross-posted is easy. No file renaming needed. That would create some governing issues as well. For example, if you ever have a patch from one fourcc addition (that might or might not be covered by v4l2), will you wait for any v4l2 developper to review it? If it's shared code, then it should be shared, and every client framework put on an equal footing. > > - Develop the same library from within v4l2. That is really a poor > > solution, since the information would be greatly duplicated > > between the two, and in terms of maintainance, code, and binary > > size that would be duplicated too. > > It's essentially what we decided to do for drm years back. And it was probably the right solution back then, but I'm really not convinced it's still the right thing to do today. > > Having it shared allows to easily share, and discover formats from the > > other subsystem, and to have a single, unique place where this is > > centralized. > > What I think could work as middle ground: > - Put drm_format stuff into a separate .ko > - Add a MAINTAINERS entry to make sure all things fourcc are cross > posted to both drm and v4l lists. Easy on the drm side, since it's all > separate files. Not sure it's so convenient for v4l uapi. > - Add a conversion library that tries to best-effort map between drm > and v4l formats. On the drm side that most likely means you need > offsets for planes, and modifiers too (since those are implied in some > v4l fourcc). Emphasis on "best effort" i.e. only support as much as > the drivers that use this library need. > - Add drm_fourcc as-needed by these drivers that want to use a unified > format space. > > Forcing this unification on everyone otoh is imo way too much. v4l2 is starting to converge with DRM, and we're using the DRM API pretty much untouched for that library, so I'm not really sure how anyone is hurt by that unification. Maxime -- Maxime Ripard, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
On Thu, Apr 18, 2019 at 11:02 AM Maxime Ripard <maxime.ripard@bootlin.com> wrote: > > On Thu, Apr 18, 2019 at 09:52:10AM +0200, Daniel Vetter wrote: > > On Thu, Apr 18, 2019 at 8:22 AM Maxime Ripard <maxime.ripard@bootlin.com> wrote: > > > On Wed, Apr 17, 2019 at 05:41:21PM +0200, Daniel Vetter wrote: > > > > On Wed, Apr 17, 2019 at 09:54:26AM +0200, Maxime Ripard wrote: > > > > > Hi, > > > > > > > > > > DRM comes with an extensive format support to retrieve the various > > > > > parameters associated with a given format (such as the subsampling, or the > > > > > bits per pixel), as well as some helpers and utilities to ease the driver > > > > > development. > > > > > > > > > > v4l2, on the other side, doesn't provide such facilities, leaving each > > > > > driver reimplement a subset of the formats parameters for the one supported > > > > > by that particular driver. This leads to a lot of duplication and > > > > > boilerplate code in the v4l2 drivers. > > > > > > > > > > This series tries to address this by moving the DRM format API into lib and > > > > > turning it into a more generic API. In order to do this, we've needed to do > > > > > some preliminary changes on the DRM drivers, then moved the API and finally > > > > > converted a v4l2 driver to give an example of how such library could be > > > > > used. > > > > > > > > > > Let me know what you think, > > > > > Maxime > > > > > > > > > > Changes from RFC: > > > > > - Rebased on next > > > > > - Fixed the various formats mapping > > > > > - Added tags > > > > > - Did most of the formats functions as inline functions > > > > > - Used a consistent prefix for all the utilities functions > > > > > - Fixed the compilation breakages, and did a run of allmodconfig for arm, > > > > > arm64 and x86_64 > > > > > - Fixed out of array bounds warnings in the image_format_info_block_* > > > > > functions > > > > > - Added License and copyright headers on missing files > > > > > > > > > > Maxime Ripard (20): > > > > > drm: Remove users of drm_format_num_planes > > > > > drm: Remove users of drm_format_(horz|vert)_chroma_subsampling > > > > > drm/fourcc: Pass the format_info pointer to drm_format_plane_cpp > > > > > drm/fourcc: Pass the format_info pointer to drm_format_plane_width/height > > > > > drm: Replace instances of drm_format_info by drm_get_format_info > > > > > lib: Add video format information library > > > > > drm/fb: Move from drm_format_info to image_format_info > > > > > drm/malidp: Convert to generic image format library > > > > > drm/client: Convert to generic image format library > > > > > drm/exynos: Convert to generic image format library > > > > > drm/i915: Convert to generic image format library > > > > > drm/ipuv3: Convert to generic image format library > > > > > drm/msm: Convert to generic image format library > > > > > drm/omap: Convert to generic image format library > > > > > drm/rockchip: Convert to generic image format library > > > > > drm/tegra: Convert to generic image format library > > > > > drm/fourcc: Remove old DRM format API > > > > > lib: image-formats: Add v4l2 formats support > > > > > lib: image-formats: Add more functions > > > > > media: sun6i: Convert to the image format API > > > > > > > > In the interest of making myself unpopular: Why move this out of drm? > > > > > > > > We do have a bunch of other such shared helpers already (mostly in > > > > drivers/video) for dt videomode and hdmi infoframes, and I'm not super > > > > sure that's going better than keeping it maintained in drm. > > > > > > > > Plus the uapi is already that you include drm_fourcc.h to get at these, > > > > dropping the drm prefix isn't going to help I think. > > > > > > > > Of course we'd need to make it a separate drm_formats.ko (so that v4l can > > > > use it without dragging in all of drm), and we need to add some fields for > > > > converting to v4l fourcc codes (which are different). But that should be > > > > all possible. And I don't think the drm_ prefix in v4l code is a problem, > > > > it's actually a feature: It makes it really clear that these are the drm > > > > fourcc codes, as allocated in drm_fourcc.h, plus their modifiers, and all > > > > that. That allocation authority is also already baked into various khr/ext > > > > standards, too. > > > > > > The way I see it, there's a fundamental difference between the UAPI > > > and the kernel. I don't suggest we change anything about the UAPI: the > > > drm formats should keep their prefix, drm_fourcc.h can remain that > > > authority, it's all fine. > > > > > > Internally however, the long term goal is to share the fourcc's > > > between DRM and V4L2 for the same formats. It basically means that of > > > course v4l2 should be using the DRM fourcc when a format exists in DRM > > > and not v4l2, but also that DRM should use v4l2 fourcc when the format > > > exists in v4l2 but not DRM, and that is far more likely, given the > > > already extensive v4l2 formats support. > > > > Uh no. We did look at v4l fourcc extensively when deciding upon a drm > > format identifier space. > > No to what exactly? > > > And a lot of people pushed for the "fourcc is a standard", when > > really it's totally not. > > Even if it's not a standard, having consistency would be a good thing. > > And you said yourself that DRM fourcc is now pretty much an authority > for the fourcc, so it definitely looks like a standard to me. drm fourcc is the authority for drm fourcc codes. Not for any of the others (and there's lots of them). Now it's used in a bunch of other places (khr standards, dri protocols in wayland/X11), but they're still only drm fourcc. There is no overall fourcc standard. > > v4l tends to conflate pixel format with stuff that we tend to encode > > in modifiers a lot more. > > Boris is working on adding the modifiers concept to v4l2, so we're > converging here, and we can totally have a layer in v4l2 to convert > between old v4l2 "format+modifiers" formats, and DRM style formats. > > > There's a bunch of reasons we can't just use v4l, and they're as > > valid as ever: > > > > - We overlap badly in some areas, so even if fourcc codes match, we > > can't use them and need a duplicated DRM_FOURCC code. > > Do yo have an example of one of those areas? I think the rgb stuff was one of the big reasons to not reuse any existing fourcc standard (whether v4l, or another one from e.g. video container formats). We had initial patches to reuse the fourcc that existed, but the end result was really inconsistent, so we ditch that and rolled out a new set of entirely drm specific fourcc codes for rgba. > > - v4l encodes some metadata into the fourcc that we encode elsewhere, > > e.g. offset for planar yuv formats, or tiling mode > > As I was saying, this changes on the v4l2 side, and converging to > what DRM is doing. > > > - drm fourcc code doesn't actually define the drm_format_info > > uniquely, drivers can override that (that's an explicit design > > intent of modifiers, to allow drivers to add another plane for > > e.g. compression information). You'd need to pull that driver > > knowledge into your format library. > > I'm not sure how my patches are changing anything here. This is > litterally the same code, with the functions renamed. > > If drivers want to override that, then yeah, fine, we can let them do > that. Just like any helper this just provides a default that covers > most of the cases. > > > Iow there's no way we can easily adopt v4l fourcc, except if we do > > something like a new addfb flag. > > For the formats that would be described as a modifier, sure. For all > the others (that are not yet supported by DRM), then I don't really > see why not. See above, we tried that initially, didn't pass the ocd tests when reviewing. Maybe the situation is better outside of rbgx/a formats, and I think we do at least try to use the same fourcc codes there when there already is one. But then the details occasionally look different. > > > And given how the current state is a mess in this regard, I'm not too > > > optimistic about keeping the formats in their relevant frameworks. > > > > > > Having a shared library, governed by both, will make this far easier, > > > since it will be easy to discover the formats that are already > > > supported by the other subsystem. > > > > I think a compat library that (tries to, best effort) convert between > > v4l and drm fourcc would make sense. Somewhere in drivers/video, next > > to the conversion functions for videomode <-> drm_display_mode > > perhaps. That should be useful for drivers. > > That's not really what this series is about though. That series is > about sharing the (image|pixels) formats database across everyone so > that everyone can benefit from it. Yeah I know. I still think leaving the drm fourcc with the drm prefix would be good, since there's really no standard here. > > Unifying the formats themselves, and all the associated metadata is > > imo a no-go, and was a pretty conscious decision when we implemented > > drm_fourcc a few years back. > > > > > If we want to keep the current library in DRM, we have two options > > > then: > > > > > > - Support all the v4l2 formats in the DRM library, which is > > > essentially what I'm doing in the last patches. However, that > > > would require to have the v4l2 developpers also reviewing stuff > > > there. And given how busy they are, I cannot really see how that > > > would work. > > > > Well, if we end up with a common library then yes we need cross > > review. There's no way around that. Doesn't matter where exactly that > > library is in the filesystem tree, and adding a special MAINTAINERS > > entry for anything related to fourcc (both drm and v4l) to make sure > > they get cross-posted is easy. No file renaming needed. > > That would create some governing issues as well. For example, if you > ever have a patch from one fourcc addition (that might or might not be > covered by v4l2), will you wait for any v4l2 developper to review it? None of this is fixed by code renaming or locations. Either way we need to figure that out. And yes part of the reasons for not moving this out of drm is that I'm not a fan of boutique trees for small stuff. If sharing means we need to split the drm_fourcc code and library out of drm trees, I'm not sure that's a good idea. We're already super liberal with merging anything through driver trees with acks, and integrating them quickly into drm-next. This would still be workable if v4l sends regular pull requests to drm-next (every 1-2 weeks, like the other big gpu trees do). If we can only sync up once per merge window with a shared boutique tree for formats only, life is going to be painful. > If it's shared code, then it should be shared, and every client > framework put on an equal footing. > > > > - Develop the same library from within v4l2. That is really a poor > > > solution, since the information would be greatly duplicated > > > between the two, and in terms of maintainance, code, and binary > > > size that would be duplicated too. > > > > It's essentially what we decided to do for drm years back. > > And it was probably the right solution back then, but I'm really not > convinced it's still the right thing to do today. > > > > Having it shared allows to easily share, and discover formats from the > > > other subsystem, and to have a single, unique place where this is > > > centralized. > > > > What I think could work as middle ground: > > - Put drm_format stuff into a separate .ko > > - Add a MAINTAINERS entry to make sure all things fourcc are cross > > posted to both drm and v4l lists. Easy on the drm side, since it's all > > separate files. Not sure it's so convenient for v4l uapi. > > - Add a conversion library that tries to best-effort map between drm > > and v4l formats. On the drm side that most likely means you need > > offsets for planes, and modifiers too (since those are implied in some > > v4l fourcc). Emphasis on "best effort" i.e. only support as much as > > the drivers that use this library need. > > - Add drm_fourcc as-needed by these drivers that want to use a unified > > format space. > > > > Forcing this unification on everyone otoh is imo way too much. > > v4l2 is starting to converge with DRM, and we're using the DRM API > pretty much untouched for that library, so I'm not really sure how > anyone is hurt by that unification. It might make sense to regularly pull v4l updates into drm-next then anyway. That would also remove the need to have the format library somewhere else. -Daniel
Hi, On Thu, 2019-04-18 at 11:02 +0200, Maxime Ripard wrote: > On Thu, Apr 18, 2019 at 09:52:10AM +0200, Daniel Vetter wrote: > > And a lot of people pushed for the "fourcc is a standard", when > > really it's totally not. > > Even if it's not a standard, having consistency would be a good thing. > > And you said yourself that DRM fourcc is now pretty much an authority > for the fourcc, so it definitely looks like a standard to me. I think trying to make the V4L2 and DRM fourccs converge is a lost cause, as it has already significantly diverged. Even if we coordinate an effort to introduce new formats with the same fourcc on both sides, I don't see what good that would be since the formats we have now are still plagued by the inconsistency. I think we always need an explicit translation step from either v4l2 or drm to the internal representation and back, without ever assuming that formats might be compatible because they share the same fourcc. It looks like so far, V4L2 pixel formats describe a DRM pixel format + modifier. I think Boris (CCed) is working to change that by allowing to pass a DRM modifier through V4L2. With that, we'd be in a situation where some formats are described by the v4l2 pixfmt alone and some formats are also described a modifier (but I looked at it from a distance so might have misunderstod). That feels better since it avoids the combinatory explosion from describing each format + modifier individually. What do you think? Cheers, Paul > > v4l tends to conflate pixel format with stuff that we tend to encode > > in modifiers a lot more. > > Boris is working on adding the modifiers concept to v4l2, so we're > converging here, and we can totally have a layer in v4l2 to convert > between old v4l2 "format+modifiers" formats, and DRM style formats. > > > There's a bunch of reasons we can't just use v4l, and they're as > > valid as ever: > > > > - We overlap badly in some areas, so even if fourcc codes match, we > > can't use them and need a duplicated DRM_FOURCC code. > > Do yo have an example of one of those areas? > > > - v4l encodes some metadata into the fourcc that we encode elsewhere, > > e.g. offset for planar yuv formats, or tiling mode > > As I was saying, this changes on the v4l2 side, and converging to > what DRM is doing. > > > - drm fourcc code doesn't actually define the drm_format_info > > uniquely, drivers can override that (that's an explicit design > > intent of modifiers, to allow drivers to add another plane for > > e.g. compression information). You'd need to pull that driver > > knowledge into your format library. > > I'm not sure how my patches are changing anything here. This is > litterally the same code, with the functions renamed. > > If drivers want to override that, then yeah, fine, we can let them do > that. Just like any helper this just provides a default that covers > most of the cases. > > > Iow there's no way we can easily adopt v4l fourcc, except if we do > > something like a new addfb flag. > > For the formats that would be described as a modifier, sure. For all > the others (that are not yet supported by DRM), then I don't really > see why not. > > > > And given how the current state is a mess in this regard, I'm not too > > > optimistic about keeping the formats in their relevant frameworks. > > > > > > Having a shared library, governed by both, will make this far easier, > > > since it will be easy to discover the formats that are already > > > supported by the other subsystem. > > > > I think a compat library that (tries to, best effort) convert between > > v4l and drm fourcc would make sense. Somewhere in drivers/video, next > > to the conversion functions for videomode <-> drm_display_mode > > perhaps. That should be useful for drivers. > > That's not really what this series is about though. That series is > about sharing the (image|pixels) formats database across everyone so > that everyone can benefit from it. > > > Unifying the formats themselves, and all the associated metadata is > > imo a no-go, and was a pretty conscious decision when we implemented > > drm_fourcc a few years back. > > > > > If we want to keep the current library in DRM, we have two options > > > then: > > > > > > - Support all the v4l2 formats in the DRM library, which is > > > essentially what I'm doing in the last patches. However, that > > > would require to have the v4l2 developpers also reviewing stuff > > > there. And given how busy they are, I cannot really see how that > > > would work. > > > > Well, if we end up with a common library then yes we need cross > > review. There's no way around that. Doesn't matter where exactly that > > library is in the filesystem tree, and adding a special MAINTAINERS > > entry for anything related to fourcc (both drm and v4l) to make sure > > they get cross-posted is easy. No file renaming needed. > > That would create some governing issues as well. For example, if you > ever have a patch from one fourcc addition (that might or might not be > covered by v4l2), will you wait for any v4l2 developper to review it? > > If it's shared code, then it should be shared, and every client > framework put on an equal footing. > > > > - Develop the same library from within v4l2. That is really a poor > > > solution, since the information would be greatly duplicated > > > between the two, and in terms of maintainance, code, and binary > > > size that would be duplicated too. > > > > It's essentially what we decided to do for drm years back. > > And it was probably the right solution back then, but I'm really not > convinced it's still the right thing to do today. > > > > Having it shared allows to easily share, and discover formats from the > > > other subsystem, and to have a single, unique place where this is > > > centralized. > > > > What I think could work as middle ground: > > - Put drm_format stuff into a separate .ko > > - Add a MAINTAINERS entry to make sure all things fourcc are cross > > posted to both drm and v4l lists. Easy on the drm side, since it's all > > separate files. Not sure it's so convenient for v4l uapi. > > - Add a conversion library that tries to best-effort map between drm > > and v4l formats. On the drm side that most likely means you need > > offsets for planes, and modifiers too (since those are implied in some > > v4l fourcc). Emphasis on "best effort" i.e. only support as much as > > the drivers that use this library need. > > - Add drm_fourcc as-needed by these drivers that want to use a unified > > format space. > > > > Forcing this unification on everyone otoh is imo way too much. > > v4l2 is starting to converge with DRM, and we're using the DRM API > pretty much untouched for that library, so I'm not really sure how > anyone is hurt by that unification. > > Maxime > > -- > Maxime Ripard, Bootlin > Embedded Linux and Kernel engineering > https://bootlin.com
On Thu, Apr 18, 2019 at 12:07:44PM +0200, Daniel Vetter wrote: > On Thu, Apr 18, 2019 at 11:02 AM Maxime Ripard > <maxime.ripard@bootlin.com> wrote: > > > > On Thu, Apr 18, 2019 at 09:52:10AM +0200, Daniel Vetter wrote: > > > On Thu, Apr 18, 2019 at 8:22 AM Maxime Ripard <maxime.ripard@bootlin.com> wrote: > > > > On Wed, Apr 17, 2019 at 05:41:21PM +0200, Daniel Vetter wrote: > > > > > On Wed, Apr 17, 2019 at 09:54:26AM +0200, Maxime Ripard wrote: > > > > > > Hi, > > > > > > > > > > > > DRM comes with an extensive format support to retrieve the various > > > > > > parameters associated with a given format (such as the subsampling, or the > > > > > > bits per pixel), as well as some helpers and utilities to ease the driver > > > > > > development. > > > > > > > > > > > > v4l2, on the other side, doesn't provide such facilities, leaving each > > > > > > driver reimplement a subset of the formats parameters for the one supported > > > > > > by that particular driver. This leads to a lot of duplication and > > > > > > boilerplate code in the v4l2 drivers. > > > > > > > > > > > > This series tries to address this by moving the DRM format API into lib and > > > > > > turning it into a more generic API. In order to do this, we've needed to do > > > > > > some preliminary changes on the DRM drivers, then moved the API and finally > > > > > > converted a v4l2 driver to give an example of how such library could be > > > > > > used. > > > > > > > > > > > > Let me know what you think, > > > > > > Maxime > > > > > > > > > > > > Changes from RFC: > > > > > > - Rebased on next > > > > > > - Fixed the various formats mapping > > > > > > - Added tags > > > > > > - Did most of the formats functions as inline functions > > > > > > - Used a consistent prefix for all the utilities functions > > > > > > - Fixed the compilation breakages, and did a run of allmodconfig for arm, > > > > > > arm64 and x86_64 > > > > > > - Fixed out of array bounds warnings in the image_format_info_block_* > > > > > > functions > > > > > > - Added License and copyright headers on missing files > > > > > > > > > > > > Maxime Ripard (20): > > > > > > drm: Remove users of drm_format_num_planes > > > > > > drm: Remove users of drm_format_(horz|vert)_chroma_subsampling > > > > > > drm/fourcc: Pass the format_info pointer to drm_format_plane_cpp > > > > > > drm/fourcc: Pass the format_info pointer to drm_format_plane_width/height > > > > > > drm: Replace instances of drm_format_info by drm_get_format_info > > > > > > lib: Add video format information library > > > > > > drm/fb: Move from drm_format_info to image_format_info > > > > > > drm/malidp: Convert to generic image format library > > > > > > drm/client: Convert to generic image format library > > > > > > drm/exynos: Convert to generic image format library > > > > > > drm/i915: Convert to generic image format library > > > > > > drm/ipuv3: Convert to generic image format library > > > > > > drm/msm: Convert to generic image format library > > > > > > drm/omap: Convert to generic image format library > > > > > > drm/rockchip: Convert to generic image format library > > > > > > drm/tegra: Convert to generic image format library > > > > > > drm/fourcc: Remove old DRM format API > > > > > > lib: image-formats: Add v4l2 formats support > > > > > > lib: image-formats: Add more functions > > > > > > media: sun6i: Convert to the image format API > > > > > > > > > > In the interest of making myself unpopular: Why move this out of drm? > > > > > > > > > > We do have a bunch of other such shared helpers already (mostly in > > > > > drivers/video) for dt videomode and hdmi infoframes, and I'm not super > > > > > sure that's going better than keeping it maintained in drm. > > > > > > > > > > Plus the uapi is already that you include drm_fourcc.h to get at these, > > > > > dropping the drm prefix isn't going to help I think. > > > > > > > > > > Of course we'd need to make it a separate drm_formats.ko (so that v4l can > > > > > use it without dragging in all of drm), and we need to add some fields for > > > > > converting to v4l fourcc codes (which are different). But that should be > > > > > all possible. And I don't think the drm_ prefix in v4l code is a problem, > > > > > it's actually a feature: It makes it really clear that these are the drm > > > > > fourcc codes, as allocated in drm_fourcc.h, plus their modifiers, and all > > > > > that. That allocation authority is also already baked into various khr/ext > > > > > standards, too. > > > > > > > > The way I see it, there's a fundamental difference between the UAPI > > > > and the kernel. I don't suggest we change anything about the UAPI: the > > > > drm formats should keep their prefix, drm_fourcc.h can remain that > > > > authority, it's all fine. > > > > > > > > Internally however, the long term goal is to share the fourcc's > > > > between DRM and V4L2 for the same formats. It basically means that of > > > > course v4l2 should be using the DRM fourcc when a format exists in DRM > > > > and not v4l2, but also that DRM should use v4l2 fourcc when the format > > > > exists in v4l2 but not DRM, and that is far more likely, given the > > > > already extensive v4l2 formats support. > > > > > > Uh no. We did look at v4l fourcc extensively when deciding upon a drm > > > format identifier space. > > > > No to what exactly? > > > > > And a lot of people pushed for the "fourcc is a standard", when > > > really it's totally not. > > > > Even if it's not a standard, having consistency would be a good thing. > > > > And you said yourself that DRM fourcc is now pretty much an authority > > for the fourcc, so it definitely looks like a standard to me. > > drm fourcc is the authority for drm fourcc codes. Not for any of the > others (and there's lots of them). Now it's used in a bunch of other > places (khr standards, dri protocols in wayland/X11), but they're > still only drm fourcc. There is no overall fourcc standard. Sounds like a de-facto standard to me ¯\_(ツ)_/¯ But even then, whether we decide to converge the fourcc or not, that's still the long term goal. Short term, that series doesn't do any of it, it just makes it easier if we ever want to go down that road. > > > v4l tends to conflate pixel format with stuff that we tend to encode > > > in modifiers a lot more. > > > > Boris is working on adding the modifiers concept to v4l2, so we're > > converging here, and we can totally have a layer in v4l2 to convert > > between old v4l2 "format+modifiers" formats, and DRM style formats. > > > > > There's a bunch of reasons we can't just use v4l, and they're as > > > valid as ever: > > > > > > - We overlap badly in some areas, so even if fourcc codes match, we > > > can't use them and need a duplicated DRM_FOURCC code. > > > > Do yo have an example of one of those areas? > > I think the rgb stuff was one of the big reasons to not reuse any > existing fourcc standard (whether v4l, or another one from e.g. video > container formats). We had initial patches to reuse the fourcc that > existed, but the end result was really inconsistent, so we ditch that > and rolled out a new set of entirely drm specific fourcc codes for > rgba. Ok, so let's ditch that plan and focus on the rest > > > > And given how the current state is a mess in this regard, I'm not too > > > > optimistic about keeping the formats in their relevant frameworks. > > > > > > > > Having a shared library, governed by both, will make this far easier, > > > > since it will be easy to discover the formats that are already > > > > supported by the other subsystem. > > > > > > I think a compat library that (tries to, best effort) convert between > > > v4l and drm fourcc would make sense. Somewhere in drivers/video, next > > > to the conversion functions for videomode <-> drm_display_mode > > > perhaps. That should be useful for drivers. > > > > That's not really what this series is about though. That series is > > about sharing the (image|pixels) formats database across everyone so > > that everyone can benefit from it. > > Yeah I know. I still think leaving the drm fourcc with the drm prefix > would be good, since there's really no standard here. > > > > Unifying the formats themselves, and all the associated metadata is > > > imo a no-go, and was a pretty conscious decision when we implemented > > > drm_fourcc a few years back. > > > > > > > If we want to keep the current library in DRM, we have two options > > > > then: > > > > > > > > - Support all the v4l2 formats in the DRM library, which is > > > > essentially what I'm doing in the last patches. However, that > > > > would require to have the v4l2 developpers also reviewing stuff > > > > there. And given how busy they are, I cannot really see how that > > > > would work. > > > > > > Well, if we end up with a common library then yes we need cross > > > review. There's no way around that. Doesn't matter where exactly that > > > library is in the filesystem tree, and adding a special MAINTAINERS > > > entry for anything related to fourcc (both drm and v4l) to make sure > > > they get cross-posted is easy. No file renaming needed. > > > > That would create some governing issues as well. For example, if you > > ever have a patch from one fourcc addition (that might or might not be > > covered by v4l2), will you wait for any v4l2 developper to review it? > > None of this is fixed by code renaming or locations. Either way we > need to figure that out. > > And yes part of the reasons for not moving this out of drm is that I'm > not a fan of boutique trees for small stuff. If sharing means we need > to split the drm_fourcc code and library out of drm trees, I'm not > sure that's a good idea. We're already super liberal with merging > anything through driver trees with acks, and integrating them quickly > into drm-next. This would still be workable if v4l sends regular pull > requests to drm-next (every 1-2 weeks, like the other big gpu trees > do). If we can only sync up once per merge window with a shared > boutique tree for formats only, life is going to be painful. I don't get why we want to turn DRM into some kind of a black hole that would pull everything. We don't have to, really. And at the same time it carries the message that v4l2 is less important than DRM for some reason, which I'm really not comfortable with. And I don't really get why you're against this in the first place. When you have some code in a single driver that would benefit more driver, you create a helper and move it into the core. In this case, we have some code used by a framework that more framework could use, and we move it to a core-er place. How is that different? Maxime -- Maxime Ripard, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
On Thu, Apr 18, 2019 at 2:01 PM Maxime Ripard <maxime.ripard@bootlin.com> wrote: > On Thu, Apr 18, 2019 at 12:07:44PM +0200, Daniel Vetter wrote: > > On Thu, Apr 18, 2019 at 11:02 AM Maxime Ripard > > <maxime.ripard@bootlin.com> wrote: > > > > > > On Thu, Apr 18, 2019 at 09:52:10AM +0200, Daniel Vetter wrote: > > > > On Thu, Apr 18, 2019 at 8:22 AM Maxime Ripard <maxime.ripard@bootlin.com> wrote: > > > > > On Wed, Apr 17, 2019 at 05:41:21PM +0200, Daniel Vetter wrote: > > > > > > On Wed, Apr 17, 2019 at 09:54:26AM +0200, Maxime Ripard wrote: > > > > > > > Hi, > > > > > > > > > > > > > > DRM comes with an extensive format support to retrieve the various > > > > > > > parameters associated with a given format (such as the subsampling, or the > > > > > > > bits per pixel), as well as some helpers and utilities to ease the driver > > > > > > > development. > > > > > > > > > > > > > > v4l2, on the other side, doesn't provide such facilities, leaving each > > > > > > > driver reimplement a subset of the formats parameters for the one supported > > > > > > > by that particular driver. This leads to a lot of duplication and > > > > > > > boilerplate code in the v4l2 drivers. > > > > > > > > > > > > > > This series tries to address this by moving the DRM format API into lib and > > > > > > > turning it into a more generic API. In order to do this, we've needed to do > > > > > > > some preliminary changes on the DRM drivers, then moved the API and finally > > > > > > > converted a v4l2 driver to give an example of how such library could be > > > > > > > used. > > > > > > > > > > > > > > Let me know what you think, > > > > > > > Maxime > > > > > > > > > > > > > > Changes from RFC: > > > > > > > - Rebased on next > > > > > > > - Fixed the various formats mapping > > > > > > > - Added tags > > > > > > > - Did most of the formats functions as inline functions > > > > > > > - Used a consistent prefix for all the utilities functions > > > > > > > - Fixed the compilation breakages, and did a run of allmodconfig for arm, > > > > > > > arm64 and x86_64 > > > > > > > - Fixed out of array bounds warnings in the image_format_info_block_* > > > > > > > functions > > > > > > > - Added License and copyright headers on missing files > > > > > > > > > > > > > > Maxime Ripard (20): > > > > > > > drm: Remove users of drm_format_num_planes > > > > > > > drm: Remove users of drm_format_(horz|vert)_chroma_subsampling > > > > > > > drm/fourcc: Pass the format_info pointer to drm_format_plane_cpp > > > > > > > drm/fourcc: Pass the format_info pointer to drm_format_plane_width/height > > > > > > > drm: Replace instances of drm_format_info by drm_get_format_info > > > > > > > lib: Add video format information library > > > > > > > drm/fb: Move from drm_format_info to image_format_info > > > > > > > drm/malidp: Convert to generic image format library > > > > > > > drm/client: Convert to generic image format library > > > > > > > drm/exynos: Convert to generic image format library > > > > > > > drm/i915: Convert to generic image format library > > > > > > > drm/ipuv3: Convert to generic image format library > > > > > > > drm/msm: Convert to generic image format library > > > > > > > drm/omap: Convert to generic image format library > > > > > > > drm/rockchip: Convert to generic image format library > > > > > > > drm/tegra: Convert to generic image format library > > > > > > > drm/fourcc: Remove old DRM format API > > > > > > > lib: image-formats: Add v4l2 formats support > > > > > > > lib: image-formats: Add more functions > > > > > > > media: sun6i: Convert to the image format API > > > > > > > > > > > > In the interest of making myself unpopular: Why move this out of drm? > > > > > > > > > > > > We do have a bunch of other such shared helpers already (mostly in > > > > > > drivers/video) for dt videomode and hdmi infoframes, and I'm not super > > > > > > sure that's going better than keeping it maintained in drm. > > > > > > > > > > > > Plus the uapi is already that you include drm_fourcc.h to get at these, > > > > > > dropping the drm prefix isn't going to help I think. > > > > > > > > > > > > Of course we'd need to make it a separate drm_formats.ko (so that v4l can > > > > > > use it without dragging in all of drm), and we need to add some fields for > > > > > > converting to v4l fourcc codes (which are different). But that should be > > > > > > all possible. And I don't think the drm_ prefix in v4l code is a problem, > > > > > > it's actually a feature: It makes it really clear that these are the drm > > > > > > fourcc codes, as allocated in drm_fourcc.h, plus their modifiers, and all > > > > > > that. That allocation authority is also already baked into various khr/ext > > > > > > standards, too. > > > > > > > > > > The way I see it, there's a fundamental difference between the UAPI > > > > > and the kernel. I don't suggest we change anything about the UAPI: the > > > > > drm formats should keep their prefix, drm_fourcc.h can remain that > > > > > authority, it's all fine. > > > > > > > > > > Internally however, the long term goal is to share the fourcc's > > > > > between DRM and V4L2 for the same formats. It basically means that of > > > > > course v4l2 should be using the DRM fourcc when a format exists in DRM > > > > > and not v4l2, but also that DRM should use v4l2 fourcc when the format > > > > > exists in v4l2 but not DRM, and that is far more likely, given the > > > > > already extensive v4l2 formats support. > > > > > > > > Uh no. We did look at v4l fourcc extensively when deciding upon a drm > > > > format identifier space. > > > > > > No to what exactly? > > > > > > > And a lot of people pushed for the "fourcc is a standard", when > > > > really it's totally not. > > > > > > Even if it's not a standard, having consistency would be a good thing. > > > > > > And you said yourself that DRM fourcc is now pretty much an authority > > > for the fourcc, so it definitely looks like a standard to me. > > > > drm fourcc is the authority for drm fourcc codes. Not for any of the > > others (and there's lots of them). Now it's used in a bunch of other > > places (khr standards, dri protocols in wayland/X11), but they're > > still only drm fourcc. There is no overall fourcc standard. > > Sounds like a de-facto standard to me ¯\_(ツ)_/¯ > > But even then, whether we decide to converge the fourcc or not, that's > still the long term goal. Short term, that series doesn't do any of > it, it just makes it easier if we ever want to go down that road. > > > > > v4l tends to conflate pixel format with stuff that we tend to encode > > > > in modifiers a lot more. > > > > > > Boris is working on adding the modifiers concept to v4l2, so we're > > > converging here, and we can totally have a layer in v4l2 to convert > > > between old v4l2 "format+modifiers" formats, and DRM style formats. > > > > > > > There's a bunch of reasons we can't just use v4l, and they're as > > > > valid as ever: > > > > > > > > - We overlap badly in some areas, so even if fourcc codes match, we > > > > can't use them and need a duplicated DRM_FOURCC code. > > > > > > Do yo have an example of one of those areas? > > > > I think the rgb stuff was one of the big reasons to not reuse any > > existing fourcc standard (whether v4l, or another one from e.g. video > > container formats). We had initial patches to reuse the fourcc that > > existed, but the end result was really inconsistent, so we ditch that > > and rolled out a new set of entirely drm specific fourcc codes for > > rgba. > > Ok, so let's ditch that plan and focus on the rest > > > > > > And given how the current state is a mess in this regard, I'm not too > > > > > optimistic about keeping the formats in their relevant frameworks. > > > > > > > > > > Having a shared library, governed by both, will make this far easier, > > > > > since it will be easy to discover the formats that are already > > > > > supported by the other subsystem. > > > > > > > > I think a compat library that (tries to, best effort) convert between > > > > v4l and drm fourcc would make sense. Somewhere in drivers/video, next > > > > to the conversion functions for videomode <-> drm_display_mode > > > > perhaps. That should be useful for drivers. > > > > > > That's not really what this series is about though. That series is > > > about sharing the (image|pixels) formats database across everyone so > > > that everyone can benefit from it. > > > > Yeah I know. I still think leaving the drm fourcc with the drm prefix > > would be good, since there's really no standard here. > > > > > > Unifying the formats themselves, and all the associated metadata is > > > > imo a no-go, and was a pretty conscious decision when we implemented > > > > drm_fourcc a few years back. > > > > > > > > > If we want to keep the current library in DRM, we have two options > > > > > then: > > > > > > > > > > - Support all the v4l2 formats in the DRM library, which is > > > > > essentially what I'm doing in the last patches. However, that > > > > > would require to have the v4l2 developpers also reviewing stuff > > > > > there. And given how busy they are, I cannot really see how that > > > > > would work. > > > > > > > > Well, if we end up with a common library then yes we need cross > > > > review. There's no way around that. Doesn't matter where exactly that > > > > library is in the filesystem tree, and adding a special MAINTAINERS > > > > entry for anything related to fourcc (both drm and v4l) to make sure > > > > they get cross-posted is easy. No file renaming needed. > > > > > > That would create some governing issues as well. For example, if you > > > ever have a patch from one fourcc addition (that might or might not be > > > covered by v4l2), will you wait for any v4l2 developper to review it? > > > > None of this is fixed by code renaming or locations. Either way we > > need to figure that out. > > > > And yes part of the reasons for not moving this out of drm is that I'm > > not a fan of boutique trees for small stuff. If sharing means we need > > to split the drm_fourcc code and library out of drm trees, I'm not > > sure that's a good idea. We're already super liberal with merging > > anything through driver trees with acks, and integrating them quickly > > into drm-next. This would still be workable if v4l sends regular pull > > requests to drm-next (every 1-2 weeks, like the other big gpu trees > > do). If we can only sync up once per merge window with a shared > > boutique tree for formats only, life is going to be painful. > > I don't get why we want to turn DRM into some kind of a black hole > that would pull everything. We don't have to, really. And at the same > time it carries the message that v4l2 is less important than DRM for > some reason, which I'm really not comfortable with. Make another tree somewhere that pulls in trees more often than every merge window, and I'm happy. It's the coordination effort of lots of trees that creates the black hole, not the other way round. Yes topic trees work, but if topic trees are persistent something with the organization of trees is wrong and needs to change. This very much looks like we'll end up with a perpetual topic branch for format stuff between drm and v4l. The other shared stuff (like hdmi infoframes) seems to change a lot less often, so the occasional patch hasn't been a pain. But drm_fourcc related stuff sees a lot of work, both in adding new formats and in refactoring the library to keep up with all the new use-cases. And yes I think an overall gfx-like-stuff.git tree for drm and v4l and the few other bits really makes tons of sense. Not as a tree where people commit, but as the 2nd-level integration tree (like drm.git right now for gpu stuff). > And I don't really get why you're against this in the first > place. When you have some code in a single driver that would benefit > more driver, you create a helper and move it into the core. It's a feature that drm doesn't share that much code with other parts of the kernel, it makes backporting the gfx stack to lts kernels a lot easier. Until someone fixes the upstream kernel release model to no longer need large scale gpu driver backports, we need to keep that. > In this case, we have some code used by a framework that more > framework could use, and we move it to a core-er place. How is that > different? Imo core sharing for code sharing's sake is overrated. If we already have drm and v4l tightly integrated as a community, then code sharing becomes a lot easier, and a lot more reasonable to do. Plus we can then just stuff code int drivers/gpu or drivers/video (or merge these two because really it's all the same). But my understanding is that integrating more tightly with the drm folks is a very contreversial topic in v4l, and until that's resolved I don't see a huge need or benefit in sharing tons of code. And the format stuff is a lot more central to kms than e.g. the infoframe helpers. Au contraire, I think forcing this has a lot of potential for needless fights between drm and v4l. Hence my suggestion to try a minimal format conversion library between the drm format world and the v4l format wolrd, and see how that goes. That contains a lot less risk than going all in right from the start. -Daniel
On Thu, Apr 18, 2019 at 02:32:14PM +0200, Daniel Vetter wrote: > > > > > Unifying the formats themselves, and all the associated metadata is > > > > > imo a no-go, and was a pretty conscious decision when we implemented > > > > > drm_fourcc a few years back. > > > > > > > > > > > If we want to keep the current library in DRM, we have two options > > > > > > then: > > > > > > > > > > > > - Support all the v4l2 formats in the DRM library, which is > > > > > > essentially what I'm doing in the last patches. However, that > > > > > > would require to have the v4l2 developpers also reviewing stuff > > > > > > there. And given how busy they are, I cannot really see how that > > > > > > would work. > > > > > > > > > > Well, if we end up with a common library then yes we need cross > > > > > review. There's no way around that. Doesn't matter where exactly that > > > > > library is in the filesystem tree, and adding a special MAINTAINERS > > > > > entry for anything related to fourcc (both drm and v4l) to make sure > > > > > they get cross-posted is easy. No file renaming needed. > > > > > > > > That would create some governing issues as well. For example, if you > > > > ever have a patch from one fourcc addition (that might or might not be > > > > covered by v4l2), will you wait for any v4l2 developper to review it? > > > > > > None of this is fixed by code renaming or locations. Either way we > > > need to figure that out. > > > > > > And yes part of the reasons for not moving this out of drm is that I'm > > > not a fan of boutique trees for small stuff. If sharing means we need > > > to split the drm_fourcc code and library out of drm trees, I'm not > > > sure that's a good idea. We're already super liberal with merging > > > anything through driver trees with acks, and integrating them quickly > > > into drm-next. This would still be workable if v4l sends regular pull > > > requests to drm-next (every 1-2 weeks, like the other big gpu trees > > > do). If we can only sync up once per merge window with a shared > > > boutique tree for formats only, life is going to be painful. > > > > I don't get why we want to turn DRM into some kind of a black hole > > that would pull everything. We don't have to, really. And at the same > > time it carries the message that v4l2 is less important than DRM for > > some reason, which I'm really not comfortable with. > > Make another tree somewhere that pulls in trees more often than every > merge window, and I'm happy. It's the coordination effort of lots of > trees that creates the black hole, not the other way round. Yes topic > trees work, but if topic trees are persistent something with the > organization of trees is wrong and needs to change. This very much > looks like we'll end up with a perpetual topic branch for format stuff > between drm and v4l. Well, if v4l2 sends a PR to DRM every 1 or 2 weeks, that definitely looks like a topic branch to me. And on a far more frequent basis than when we will merge a format description. > The other shared stuff (like hdmi infoframes) seems to change a lot > less often, so the occasional patch hasn't been a pain. But drm_fourcc > related stuff sees a lot of work, both in adding new formats and in > refactoring the library to keep up with all the new use-cases. When was the last time a refactoring that changed the API happened? Most of the changes will be new format descriptions, and I guess that would only concern a single tree. And really, we're doing this all the time, so I'm not sure what the big deal is here. I feel like there's something that you don't really like about this, but you're not saying this out loud. Sure, the exact process needs to be figured out, and everyone needs to agree upon that process. But that's pretty much it, and it's nothing out of the ordinary. > And yes I think an overall gfx-like-stuff.git tree for drm and v4l and > the few other bits really makes tons of sense. Not as a tree where > people commit, but as the 2nd-level integration tree (like drm.git > right now for gpu stuff). > > > And I don't really get why you're against this in the first > > place. When you have some code in a single driver that would benefit > > more driver, you create a helper and move it into the core. > > It's a feature that drm doesn't share that much code with other parts > of the kernel, it makes backporting the gfx stack to lts kernels a lot > easier. Until someone fixes the upstream kernel release model to no > longer need large scale gpu driver backports, we need to keep that. > > > In this case, we have some code used by a framework that more > > framework could use, and we move it to a core-er place. How is that > > different? > > Imo core sharing for code sharing's sake is overrated. If we already > have drm and v4l tightly integrated as a community, then code sharing > becomes a lot easier, and a lot more reasonable to do. At least Laurent, Boris, Ezequiel, Gustavo and I have been working on v4l2, so I'm not sure how not integrated we are. > Plus we can then just stuff code int drivers/gpu or drivers/video > (or merge these two because really it's all the same). But my > understanding is that integrating more tightly with the drm folks is > a very contreversial topic in v4l So, I sent the RFC expecting that kind of feedback. Hans replied mainly to that patch https://patchwork.freedesktop.org/patch/293043/ " If we are creating a common library then I think we should change that rule to: "unless they are in use by a DRM or V4L2 driver". And when new formats are added, and they exists already for DRM or V4L2, then we should use the same fourcc for the other subsystem. I.e. if pixelformat V4L2_PIX_FMT_FOO was already defined, then add a: #define DRM_FORMAT_FOO V4L2_PIX_FMT_FOO rather than creating a new fourcc. We could even start looking at redoing the whole scheme in a unified way, but that's something for the (far) future. This is already a big step forward. " So, not controversial at all. > and until that's resolved I don't see a huge need or benefit in > sharing tons of code. That's mostly tons of data though. The code is pretty small and trivial. > And the format stuff is a lot more central to kms than e.g. the > infoframe helpers. > > Au contraire, I think forcing this has a lot of potential for > needless fights between drm and v4l. We're all reasonable, so I'm not sure why we would need to fight here. > Hence my suggestion to try a minimal format conversion library > between the drm format world and the v4l format wolrd, and see how > that goes. That contains a lot less risk than going all in right > from the start. And it's really not about getting access to the DRM fourcc. It's about getting access to DRM's format description, so I'm not really sure what there is to convert, we just want a lookup. Maxime -- Maxime Ripard, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Hi Paul, On Thu, Apr 18, 2019 at 01:49:54PM +0200, Paul Kocialkowski wrote: > On Thu, 2019-04-18 at 11:02 +0200, Maxime Ripard wrote: > > On Thu, Apr 18, 2019 at 09:52:10AM +0200, Daniel Vetter wrote: > >> And a lot of people pushed for the "fourcc is a standard", when > >> really it's totally not. > > > > Even if it's not a standard, having consistency would be a good thing. > > > > And you said yourself that DRM fourcc is now pretty much an authority > > for the fourcc, so it definitely looks like a standard to me. > > I think trying to make the V4L2 and DRM fourccs converge is a lost > cause, as it has already significantly diverged. Even if we coordinate > an effort to introduce new formats with the same fourcc on both sides, > I don't see what good that would be since the formats we have now are > still plagued by the inconsistency. > > I think we always need an explicit translation step from either v4l2 or > drm to the internal representation and back, without ever assuming that > formats might be compatible because they share the same fourcc. I don't agree. APIs evolve, and while we can't switch from one set of 4CCs to another in existing APIs, we could in new APIs. Boris is working on new ioctls to handle formats in V4L2, and while 4CC unification could be impopular from a userspace developers point of view there, I don't think we have ruled it out completely. The move to the request API is also an area where a common set of 4CCs could be used, as it will depart from the existing V4L2 ioctls. To summarize my opinion, we're not there yet, but I wouldn't rule it out completely for the future. > It looks like so far, V4L2 pixel formats describe a DRM pixel format + > modifier. DRM modifiers are mostly about tiling and compression, and we hardly support these in V4L2. What are the modifiers you think are hardcoded in 4CCs in V4L2 ? > I think Boris (CCed) is working to change that by allowing to > pass a DRM modifier through V4L2. With that, we'd be in a situation > where some formats are described by the v4l2 pixfmt alone and some > formats are also described a modifier (but I looked at it from a > distance so might have misunderstod). That feels better since it avoids > the combinatory explosion from describing each format + modifier > individually. > > What do you think? > > >> v4l tends to conflate pixel format with stuff that we tend to encode > >> in modifiers a lot more. > > > > Boris is working on adding the modifiers concept to v4l2, so we're > > converging here, and we can totally have a layer in v4l2 to convert > > between old v4l2 "format+modifiers" formats, and DRM style formats. > > > >> There's a bunch of reasons we can't just use v4l, and they're as > >> valid as ever: > >> > >> - We overlap badly in some areas, so even if fourcc codes match, we > >> can't use them and need a duplicated DRM_FOURCC code. > > > > Do yo have an example of one of those areas? > > > >> - v4l encodes some metadata into the fourcc that we encode elsewhere, > >> e.g. offset for planar yuv formats, or tiling mode > > > > As I was saying, this changes on the v4l2 side, and converging to > > what DRM is doing. > > > >> - drm fourcc code doesn't actually define the drm_format_info > >> uniquely, drivers can override that (that's an explicit design > >> intent of modifiers, to allow drivers to add another plane for > >> e.g. compression information). You'd need to pull that driver > >> knowledge into your format library. > > > > I'm not sure how my patches are changing anything here. This is > > litterally the same code, with the functions renamed. > > > > If drivers want to override that, then yeah, fine, we can let them do > > that. Just like any helper this just provides a default that covers > > most of the cases. > > > >> Iow there's no way we can easily adopt v4l fourcc, except if we do > >> something like a new addfb flag. > > > > For the formats that would be described as a modifier, sure. For all > > the others (that are not yet supported by DRM), then I don't really > > see why not. > > > >>> And given how the current state is a mess in this regard, I'm not too > >>> optimistic about keeping the formats in their relevant frameworks. > >>> > >>> Having a shared library, governed by both, will make this far easier, > >>> since it will be easy to discover the formats that are already > >>> supported by the other subsystem. > >> > >> I think a compat library that (tries to, best effort) convert between > >> v4l and drm fourcc would make sense. Somewhere in drivers/video, next > >> to the conversion functions for videomode <-> drm_display_mode > >> perhaps. That should be useful for drivers. > > > > That's not really what this series is about though. That series is > > about sharing the (image|pixels) formats database across everyone so > > that everyone can benefit from it. > > > >> Unifying the formats themselves, and all the associated metadata is > >> imo a no-go, and was a pretty conscious decision when we implemented > >> drm_fourcc a few years back. > >> > >>> If we want to keep the current library in DRM, we have two options > >>> then: > >>> > >>> - Support all the v4l2 formats in the DRM library, which is > >>> essentially what I'm doing in the last patches. However, that > >>> would require to have the v4l2 developpers also reviewing stuff > >>> there. And given how busy they are, I cannot really see how that > >>> would work. > >> > >> Well, if we end up with a common library then yes we need cross > >> review. There's no way around that. Doesn't matter where exactly that > >> library is in the filesystem tree, and adding a special MAINTAINERS > >> entry for anything related to fourcc (both drm and v4l) to make sure > >> they get cross-posted is easy. No file renaming needed. > > > > That would create some governing issues as well. For example, if you > > ever have a patch from one fourcc addition (that might or might not be > > covered by v4l2), will you wait for any v4l2 developper to review it? > > > > If it's shared code, then it should be shared, and every client > > framework put on an equal footing. > > > >>> - Develop the same library from within v4l2. That is really a poor > >>> solution, since the information would be greatly duplicated > >>> between the two, and in terms of maintainance, code, and binary > >>> size that would be duplicated too. > >> > >> It's essentially what we decided to do for drm years back. > > > > And it was probably the right solution back then, but I'm really not > > convinced it's still the right thing to do today. > > > >>> Having it shared allows to easily share, and discover formats from the > >>> other subsystem, and to have a single, unique place where this is > >>> centralized. > >> > >> What I think could work as middle ground: > >> - Put drm_format stuff into a separate .ko > >> - Add a MAINTAINERS entry to make sure all things fourcc are cross > >> posted to both drm and v4l lists. Easy on the drm side, since it's all > >> separate files. Not sure it's so convenient for v4l uapi. > >> - Add a conversion library that tries to best-effort map between drm > >> and v4l formats. On the drm side that most likely means you need > >> offsets for planes, and modifiers too (since those are implied in some > >> v4l fourcc). Emphasis on "best effort" i.e. only support as much as > >> the drivers that use this library need. > >> - Add drm_fourcc as-needed by these drivers that want to use a unified > >> format space. > >> > >> Forcing this unification on everyone otoh is imo way too much. > > > > v4l2 is starting to converge with DRM, and we're using the DRM API > > pretty much untouched for that library, so I'm not really sure how > > anyone is hurt by that unification.
Hi Daniel, On Thu, Apr 18, 2019 at 12:07:44PM +0200, Daniel Vetter wrote: > On Thu, Apr 18, 2019 at 11:02 AM Maxime Ripard wrote: > > On Thu, Apr 18, 2019 at 09:52:10AM +0200, Daniel Vetter wrote: > > > On Thu, Apr 18, 2019 at 8:22 AM Maxime Ripard wrote: > > > > On Wed, Apr 17, 2019 at 05:41:21PM +0200, Daniel Vetter wrote: > > > > > On Wed, Apr 17, 2019 at 09:54:26AM +0200, Maxime Ripard wrote: > > > > > > Hi, > > > > > > > > > > > > DRM comes with an extensive format support to retrieve the various > > > > > > parameters associated with a given format (such as the subsampling, or the > > > > > > bits per pixel), as well as some helpers and utilities to ease the driver > > > > > > development. > > > > > > > > > > > > v4l2, on the other side, doesn't provide such facilities, leaving each > > > > > > driver reimplement a subset of the formats parameters for the one supported > > > > > > by that particular driver. This leads to a lot of duplication and > > > > > > boilerplate code in the v4l2 drivers. > > > > > > > > > > > > This series tries to address this by moving the DRM format API into lib and > > > > > > turning it into a more generic API. In order to do this, we've needed to do > > > > > > some preliminary changes on the DRM drivers, then moved the API and finally > > > > > > converted a v4l2 driver to give an example of how such library could be > > > > > > used. > > > > > > > > > > > > Let me know what you think, > > > > > > Maxime > > > > > > > > > > > > Changes from RFC: > > > > > > - Rebased on next > > > > > > - Fixed the various formats mapping > > > > > > - Added tags > > > > > > - Did most of the formats functions as inline functions > > > > > > - Used a consistent prefix for all the utilities functions > > > > > > - Fixed the compilation breakages, and did a run of allmodconfig for arm, > > > > > > arm64 and x86_64 > > > > > > - Fixed out of array bounds warnings in the image_format_info_block_* > > > > > > functions > > > > > > - Added License and copyright headers on missing files > > > > > > > > > > > > Maxime Ripard (20): > > > > > > drm: Remove users of drm_format_num_planes > > > > > > drm: Remove users of drm_format_(horz|vert)_chroma_subsampling > > > > > > drm/fourcc: Pass the format_info pointer to drm_format_plane_cpp > > > > > > drm/fourcc: Pass the format_info pointer to drm_format_plane_width/height > > > > > > drm: Replace instances of drm_format_info by drm_get_format_info > > > > > > lib: Add video format information library > > > > > > drm/fb: Move from drm_format_info to image_format_info > > > > > > drm/malidp: Convert to generic image format library > > > > > > drm/client: Convert to generic image format library > > > > > > drm/exynos: Convert to generic image format library > > > > > > drm/i915: Convert to generic image format library > > > > > > drm/ipuv3: Convert to generic image format library > > > > > > drm/msm: Convert to generic image format library > > > > > > drm/omap: Convert to generic image format library > > > > > > drm/rockchip: Convert to generic image format library > > > > > > drm/tegra: Convert to generic image format library > > > > > > drm/fourcc: Remove old DRM format API > > > > > > lib: image-formats: Add v4l2 formats support > > > > > > lib: image-formats: Add more functions > > > > > > media: sun6i: Convert to the image format API > > > > > > > > > > In the interest of making myself unpopular: Why move this out of drm? > > > > > > > > > > We do have a bunch of other such shared helpers already (mostly in > > > > > drivers/video) for dt videomode and hdmi infoframes, and I'm not super > > > > > sure that's going better than keeping it maintained in drm. That's a bit of a different situation as both DRM and FBDEV address the same features (display output), and FBDEV is deprecared and replaced by DRM. I'm not against maintaining a 4CC library in DRM (adding a third-party maintainer would likely create more problems than it would solve), but that doesn't mean the library has to live in drivers/gpu/, nor that it needs to have the drm_ prefix. I would actually advocate to make it live in a neutral directory, with a neutral prefix (kudos to anyone who can propose a nice naming convention that would establish a new shared ground for image/video-related Linux APIs), and maintained through the DRM tree (possibly with extra entries in MAINTAINERS to ensure it reaches all the related folks). > > > > > Plus the uapi is already that you include drm_fourcc.h to get at these, > > > > > dropping the drm prefix isn't going to help I think. > > > > > > > > > > Of course we'd need to make it a separate drm_formats.ko (so that v4l can > > > > > use it without dragging in all of drm), and we need to add some fields for > > > > > converting to v4l fourcc codes (which are different). But that should be > > > > > all possible. And I don't think the drm_ prefix in v4l code is a problem, > > > > > it's actually a feature: It makes it really clear that these are the drm > > > > > fourcc codes, as allocated in drm_fourcc.h, plus their modifiers, and all > > > > > that. That allocation authority is also already baked into various khr/ext > > > > > standards, too. There's one thing that V4L2 has and DRM hasn't for 4CCs: good documentation. Look at https://linuxtv.org/downloads/v4l-dvb-apis-new/uapi/v4l/pixfmt-packed-rgb.html or https://linuxtv.org/downloads/v4l-dvb-apis-new/uapi/v4l/yuv-formats.html for instance. It's painful to write, painful to read, but defines the 4CCs very clearly without ambiguity. I wouldn't be surprised if different drivers used the same DRM 4CC for different formats due to the lack of detailed documentation. Moving to a shared library for 4CCs should also address the documentation side, and any new format added to the kernel, whether from the V4L2 side or the DRM side, would be required to provide detailed documentation. That would be a great improvement for DRM 4CC handling. > > > > The way I see it, there's a fundamental difference between the UAPI > > > > and the kernel. I don't suggest we change anything about the UAPI: the > > > > drm formats should keep their prefix, drm_fourcc.h can remain that > > > > authority, it's all fine. > > > > > > > > Internally however, the long term goal is to share the fourcc's > > > > between DRM and V4L2 for the same formats. It basically means that of > > > > course v4l2 should be using the DRM fourcc when a format exists in DRM > > > > and not v4l2, but also that DRM should use v4l2 fourcc when the format > > > > exists in v4l2 but not DRM, and that is far more likely, given the > > > > already extensive v4l2 formats support. > > > > > > Uh no. We did look at v4l fourcc extensively when deciding upon a drm > > > format identifier space. > > > > No to what exactly? > > > > > And a lot of people pushed for the "fourcc is a standard", when > > > really it's totally not. > > > > Even if it's not a standard, having consistency would be a good thing. > > > > And you said yourself that DRM fourcc is now pretty much an authority > > for the fourcc, so it definitely looks like a standard to me. > > drm fourcc is the authority for drm fourcc codes. Not for any of the > others (and there's lots of them). Now it's used in a bunch of other > places (khr standards, dri protocols in wayland/X11), but they're > still only drm fourcc. There is no overall fourcc standard. > > > > v4l tends to conflate pixel format with stuff that we tend to encode > > > in modifiers a lot more. > > > > Boris is working on adding the modifiers concept to v4l2, so we're > > converging here, and we can totally have a layer in v4l2 to convert > > between old v4l2 "format+modifiers" formats, and DRM style formats. > > > > > There's a bunch of reasons we can't just use v4l, and they're as > > > valid as ever: > > > > > > - We overlap badly in some areas, so even if fourcc codes match, we > > > can't use them and need a duplicated DRM_FOURCC code. > > > > Do yo have an example of one of those areas? > > I think the rgb stuff was one of the big reasons to not reuse any > existing fourcc standard (whether v4l, or another one from e.g. video > container formats). We had initial patches to reuse the fourcc that > existed, but the end result was really inconsistent, so we ditch that > and rolled out a new set of entirely drm specific fourcc codes for > rgba. Could you give one or a couple of examples of V4L2 4CCs that are not OCD-compatible ? :-) > > > - v4l encodes some metadata into the fourcc that we encode elsewhere, > > > e.g. offset for planar yuv formats, or tiling mode > > > > As I was saying, this changes on the v4l2 side, and converging to > > what DRM is doing. > > > > > - drm fourcc code doesn't actually define the drm_format_info > > > uniquely, drivers can override that (that's an explicit design > > > intent of modifiers, to allow drivers to add another plane for > > > e.g. compression information). You'd need to pull that driver > > > knowledge into your format library. That's a mistake in my opinion. We tried that in V4L2 to store metadata in a separate plane, and had to go another route eventually as it created a very bad mess. There's a tendency in both subsystems to look at the other as a bit of a retarded relative, and that's a shame, we have lots to learn from each other's mistakes. That wouldn't be too difficult if people started talking to each other. A semi-related comment: DRM also carries other mistakes of its own, I'm thinking about DRM_FORMAT_BIG_ENDIAN in particular > > I'm not sure how my patches are changing anything here. This is > > litterally the same code, with the functions renamed. > > > > If drivers want to override that, then yeah, fine, we can let them do > > that. Just like any helper this just provides a default that covers > > most of the cases. > > > > > Iow there's no way we can easily adopt v4l fourcc, except if we do > > > something like a new addfb flag. > > > > For the formats that would be described as a modifier, sure. For all > > the others (that are not yet supported by DRM), then I don't really > > see why not. > > See above, we tried that initially, didn't pass the ocd tests when > reviewing. Maybe the situation is better outside of rbgx/a formats, > and I think we do at least try to use the same fourcc codes there when > there already is one. But then the details occasionally look > different. > > > > > And given how the current state is a mess in this regard, I'm not too > > > > optimistic about keeping the formats in their relevant frameworks. > > > > > > > > Having a shared library, governed by both, will make this far easier, > > > > since it will be easy to discover the formats that are already > > > > supported by the other subsystem. > > > > > > I think a compat library that (tries to, best effort) convert between > > > v4l and drm fourcc would make sense. Somewhere in drivers/video, next > > > to the conversion functions for videomode <-> drm_display_mode > > > perhaps. That should be useful for drivers. > > > > That's not really what this series is about though. That series is > > about sharing the (image|pixels) formats database across everyone so > > that everyone can benefit from it. > > Yeah I know. I still think leaving the drm fourcc with the drm prefix > would be good, since there's really no standard here. > > > > Unifying the formats themselves, and all the associated metadata is > > > imo a no-go, and was a pretty conscious decision when we implemented > > > drm_fourcc a few years back. > > > > > > > If we want to keep the current library in DRM, we have two options > > > > then: > > > > > > > > - Support all the v4l2 formats in the DRM library, which is > > > > essentially what I'm doing in the last patches. However, that > > > > would require to have the v4l2 developpers also reviewing stuff > > > > there. And given how busy they are, I cannot really see how that > > > > would work. > > > > > > Well, if we end up with a common library then yes we need cross > > > review. There's no way around that. Doesn't matter where exactly that > > > library is in the filesystem tree, and adding a special MAINTAINERS > > > entry for anything related to fourcc (both drm and v4l) to make sure > > > they get cross-posted is easy. No file renaming needed. > > > > That would create some governing issues as well. For example, if you > > ever have a patch from one fourcc addition (that might or might not be > > covered by v4l2), will you wait for any v4l2 developper to review it? > > None of this is fixed by code renaming or locations. Either way we > need to figure that out. > > And yes part of the reasons for not moving this out of drm is that I'm > not a fan of boutique trees for small stuff. If sharing means we need > to split the drm_fourcc code and library out of drm trees, I'm not > sure that's a good idea. We're already super liberal with merging > anything through driver trees with acks, and integrating them quickly > into drm-next. This would still be workable if v4l sends regular pull > requests to drm-next (every 1-2 weeks, like the other big gpu trees > do). If we can only sync up once per merge window with a shared > boutique tree for formats only, life is going to be painful. That should be solved by the proposal above, maintaining the shared library in the DRM tree. We would need to be careful there, and ideally maintain that in a separate branch that could be merged in both DRM and V4L2 without having to merge most of the other subsystem's pending changes at the same time, but I think it's doable without any big issue. > > If it's shared code, then it should be shared, and every client > > framework put on an equal footing. > > > > > > - Develop the same library from within v4l2. That is really a poor > > > > solution, since the information would be greatly duplicated > > > > between the two, and in terms of maintainance, code, and binary > > > > size that would be duplicated too. > > > > > > It's essentially what we decided to do for drm years back. > > > > And it was probably the right solution back then, but I'm really not > > convinced it's still the right thing to do today. > > > > > > Having it shared allows to easily share, and discover formats from the > > > > other subsystem, and to have a single, unique place where this is > > > > centralized. > > > > > > What I think could work as middle ground: > > > - Put drm_format stuff into a separate .ko > > > - Add a MAINTAINERS entry to make sure all things fourcc are cross > > > posted to both drm and v4l lists. Easy on the drm side, since it's all > > > separate files. Not sure it's so convenient for v4l uapi. > > > - Add a conversion library that tries to best-effort map between drm > > > and v4l formats. On the drm side that most likely means you need > > > offsets for planes, and modifiers too (since those are implied in some > > > v4l fourcc). Emphasis on "best effort" i.e. only support as much as > > > the drivers that use this library need. > > > - Add drm_fourcc as-needed by these drivers that want to use a unified > > > format space. > > > > > > Forcing this unification on everyone otoh is imo way too much. > > > > v4l2 is starting to converge with DRM, and we're using the DRM API > > pretty much untouched for that library, so I'm not really sure how > > anyone is hurt by that unification. > > It might make sense to regularly pull v4l updates into drm-next then > anyway. That would also remove the need to have the format library > somewhere else. Are you saying it should the live in V4L2 ? ;-)
Hi Maxime, On Thu, Apr 18, 2019 at 10:56:30PM +0200, Maxime Ripard wrote: > On Thu, Apr 18, 2019 at 02:32:14PM +0200, Daniel Vetter wrote: > >>>>> Unifying the formats themselves, and all the associated metadata is > >>>>> imo a no-go, and was a pretty conscious decision when we implemented > >>>>> drm_fourcc a few years back. > >>>>> > >>>>>> If we want to keep the current library in DRM, we have two options > >>>>>> then: > >>>>>> > >>>>>> - Support all the v4l2 formats in the DRM library, which is > >>>>>> essentially what I'm doing in the last patches. However, that > >>>>>> would require to have the v4l2 developpers also reviewing stuff > >>>>>> there. And given how busy they are, I cannot really see how that > >>>>>> would work. > >>>>> > >>>>> Well, if we end up with a common library then yes we need cross > >>>>> review. There's no way around that. Doesn't matter where exactly that > >>>>> library is in the filesystem tree, and adding a special MAINTAINERS > >>>>> entry for anything related to fourcc (both drm and v4l) to make sure > >>>>> they get cross-posted is easy. No file renaming needed. > >>>> > >>>> That would create some governing issues as well. For example, if you > >>>> ever have a patch from one fourcc addition (that might or might not be > >>>> covered by v4l2), will you wait for any v4l2 developper to review it? > >>> > >>> None of this is fixed by code renaming or locations. Either way we > >>> need to figure that out. > >>> > >>> And yes part of the reasons for not moving this out of drm is that I'm > >>> not a fan of boutique trees for small stuff. If sharing means we need > >>> to split the drm_fourcc code and library out of drm trees, I'm not > >>> sure that's a good idea. We're already super liberal with merging > >>> anything through driver trees with acks, and integrating them quickly > >>> into drm-next. This would still be workable if v4l sends regular pull > >>> requests to drm-next (every 1-2 weeks, like the other big gpu trees > >>> do). If we can only sync up once per merge window with a shared > >>> boutique tree for formats only, life is going to be painful. > >> > >> I don't get why we want to turn DRM into some kind of a black hole > >> that would pull everything. We don't have to, really. And at the same > >> time it carries the message that v4l2 is less important than DRM for > >> some reason, which I'm really not comfortable with. > > > > Make another tree somewhere that pulls in trees more often than every > > merge window, and I'm happy. It's the coordination effort of lots of > > trees that creates the black hole, not the other way round. Yes topic > > trees work, but if topic trees are persistent something with the > > organization of trees is wrong and needs to change. This very much > > looks like we'll end up with a perpetual topic branch for format stuff > > between drm and v4l. > > Well, if v4l2 sends a PR to DRM every 1 or 2 weeks, that definitely > looks like a topic branch to me. And on a far more frequent basis than > when we will merge a format description. > > > The other shared stuff (like hdmi infoframes) seems to change a lot > > less often, so the occasional patch hasn't been a pain. But drm_fourcc > > related stuff sees a lot of work, both in adding new formats and in > > refactoring the library to keep up with all the new use-cases. > > When was the last time a refactoring that changed the API happened? > > Most of the changes will be new format descriptions, and I guess that > would only concern a single tree. > > And really, we're doing this all the time, so I'm not sure what the > big deal is here. > > I feel like there's something that you don't really like about this, > but you're not saying this out loud. > > Sure, the exact process needs to be figured out, and everyone needs to > agree upon that process. But that's pretty much it, and it's nothing > out of the ordinary. > > > And yes I think an overall gfx-like-stuff.git tree for drm and v4l and > > the few other bits really makes tons of sense. Not as a tree where > > people commit, but as the 2nd-level integration tree (like drm.git > > right now for gpu stuff). > > > >> And I don't really get why you're against this in the first > >> place. When you have some code in a single driver that would benefit > >> more driver, you create a helper and move it into the core. > > > > It's a feature that drm doesn't share that much code with other parts > > of the kernel, it makes backporting the gfx stack to lts kernels a lot > > easier. Until someone fixes the upstream kernel release model to no > > longer need large scale gpu driver backports, we need to keep that. > > > >> In this case, we have some code used by a framework that more > >> framework could use, and we move it to a core-er place. How is that > >> different? > > > > Imo core sharing for code sharing's sake is overrated. If we already > > have drm and v4l tightly integrated as a community, then code sharing > > becomes a lot easier, and a lot more reasonable to do. > > At least Laurent, Boris, Ezequiel, Gustavo and I have been working on > v4l2, so I'm not sure how not integrated we are. Let's also mention that we'll need more code sharing as we move forward. We already have chips that have two drivers, one in V4L2 and one in DRM (I'm thinking about the adv7511 bridge driver for instance). I know SoC vendors that are not happy about this state of things as they have IP cores that can be used in both camera and display pipelines. It's a long standing issue, it won't be solve today, but we'll need to get to it eventually. Sharing a 4CC library would be a great step forward in my opinion, in order to get the two subsystems to collaborate better. > > Plus we can then just stuff code int drivers/gpu or drivers/video > > (or merge these two because really it's all the same). But my > > understanding is that integrating more tightly with the drm folks is > > a very contreversial topic in v4l > > So, I sent the RFC expecting that kind of feedback. > > Hans replied mainly to that patch https://patchwork.freedesktop.org/patch/293043/ > > " > If we are creating a common library then I think we should change that rule > to: "unless they are in use by a DRM or V4L2 driver". And when new formats are > added, and they exists already for DRM or V4L2, then we should use the same > fourcc for the other subsystem. > > I.e. if pixelformat V4L2_PIX_FMT_FOO was already defined, then add a: > > #define DRM_FORMAT_FOO V4L2_PIX_FMT_FOO > > rather than creating a new fourcc. I would like to point out here that we have two different items we call 4CC, the macro name (FOO in your example here), and the numerical value. If a numerical value already exists in DRM or V4L2 for a given pixel format, it should be used, period. The macro name, however, could be changed, as many of them carry historical mistakes. > We could even start looking at redoing the whole scheme in a unified way, but > that's something for the (far) future. > > This is already a big step forward. > " > > So, not controversial at all. > > > and until that's resolved I don't see a huge need or benefit in > > sharing tons of code. > > That's mostly tons of data though. The code is pretty small and > trivial. > > > And the format stuff is a lot more central to kms than e.g. the > > infoframe helpers. > > > > Au contraire, I think forcing this has a lot of potential for > > needless fights between drm and v4l. > > We're all reasonable, so I'm not sure why we would need to fight here. > > > Hence my suggestion to try a minimal format conversion library > > between the drm format world and the v4l format wolrd, and see how > > that goes. That contains a lot less risk than going all in right > > from the start. > > And it's really not about getting access to the DRM fourcc. It's about > getting access to DRM's format description, so I'm not really sure > what there is to convert, we just want a lookup.
On Sun, Apr 21, 2019 at 01:59:04AM +0300, Laurent Pinchart wrote: > Hi Daniel, > > On Thu, Apr 18, 2019 at 12:07:44PM +0200, Daniel Vetter wrote: > > On Thu, Apr 18, 2019 at 11:02 AM Maxime Ripard wrote: > > > On Thu, Apr 18, 2019 at 09:52:10AM +0200, Daniel Vetter wrote: > > > > On Thu, Apr 18, 2019 at 8:22 AM Maxime Ripard wrote: > > > > > On Wed, Apr 17, 2019 at 05:41:21PM +0200, Daniel Vetter wrote: > > > > > > On Wed, Apr 17, 2019 at 09:54:26AM +0200, Maxime Ripard wrote: > > > > > > > Hi, > > > > > > > > > > > > > > DRM comes with an extensive format support to retrieve the various > > > > > > > parameters associated with a given format (such as the subsampling, or the > > > > > > > bits per pixel), as well as some helpers and utilities to ease the driver > > > > > > > development. > > > > > > > > > > > > > > v4l2, on the other side, doesn't provide such facilities, leaving each > > > > > > > driver reimplement a subset of the formats parameters for the one supported > > > > > > > by that particular driver. This leads to a lot of duplication and > > > > > > > boilerplate code in the v4l2 drivers. > > > > > > > > > > > > > > This series tries to address this by moving the DRM format API into lib and > > > > > > > turning it into a more generic API. In order to do this, we've needed to do > > > > > > > some preliminary changes on the DRM drivers, then moved the API and finally > > > > > > > converted a v4l2 driver to give an example of how such library could be > > > > > > > used. > > > > > > > > > > > > > > Let me know what you think, > > > > > > > Maxime > > > > > > > > > > > > > > Changes from RFC: > > > > > > > - Rebased on next > > > > > > > - Fixed the various formats mapping > > > > > > > - Added tags > > > > > > > - Did most of the formats functions as inline functions > > > > > > > - Used a consistent prefix for all the utilities functions > > > > > > > - Fixed the compilation breakages, and did a run of allmodconfig for arm, > > > > > > > arm64 and x86_64 > > > > > > > - Fixed out of array bounds warnings in the image_format_info_block_* > > > > > > > functions > > > > > > > - Added License and copyright headers on missing files > > > > > > > > > > > > > > Maxime Ripard (20): > > > > > > > drm: Remove users of drm_format_num_planes > > > > > > > drm: Remove users of drm_format_(horz|vert)_chroma_subsampling > > > > > > > drm/fourcc: Pass the format_info pointer to drm_format_plane_cpp > > > > > > > drm/fourcc: Pass the format_info pointer to drm_format_plane_width/height > > > > > > > drm: Replace instances of drm_format_info by drm_get_format_info > > > > > > > lib: Add video format information library > > > > > > > drm/fb: Move from drm_format_info to image_format_info > > > > > > > drm/malidp: Convert to generic image format library > > > > > > > drm/client: Convert to generic image format library > > > > > > > drm/exynos: Convert to generic image format library > > > > > > > drm/i915: Convert to generic image format library > > > > > > > drm/ipuv3: Convert to generic image format library > > > > > > > drm/msm: Convert to generic image format library > > > > > > > drm/omap: Convert to generic image format library > > > > > > > drm/rockchip: Convert to generic image format library > > > > > > > drm/tegra: Convert to generic image format library > > > > > > > drm/fourcc: Remove old DRM format API > > > > > > > lib: image-formats: Add v4l2 formats support > > > > > > > lib: image-formats: Add more functions > > > > > > > media: sun6i: Convert to the image format API > > > > > > > > > > > > In the interest of making myself unpopular: Why move this out of drm? > > > > > > > > > > > > We do have a bunch of other such shared helpers already (mostly in > > > > > > drivers/video) for dt videomode and hdmi infoframes, and I'm not super > > > > > > sure that's going better than keeping it maintained in drm. > > That's a bit of a different situation as both DRM and FBDEV address the > same features (display output), and FBDEV is deprecared and replaced by > DRM. > > I'm not against maintaining a 4CC library in DRM (adding a third-party > maintainer would likely create more problems than it would solve), but > that doesn't mean the library has to live in drivers/gpu/, nor that it > needs to have the drm_ prefix. I would actually advocate to make it live > in a neutral directory, with a neutral prefix (kudos to anyone who can > propose a nice naming convention that would establish a new shared > ground for image/video-related Linux APIs), and maintained through the > DRM tree (possibly with extra entries in MAINTAINERS to ensure it > reaches all the related folks). > > > > > > > Plus the uapi is already that you include drm_fourcc.h to get at these, > > > > > > dropping the drm prefix isn't going to help I think. > > > > > > > > > > > > Of course we'd need to make it a separate drm_formats.ko (so that v4l can > > > > > > use it without dragging in all of drm), and we need to add some fields for > > > > > > converting to v4l fourcc codes (which are different). But that should be > > > > > > all possible. And I don't think the drm_ prefix in v4l code is a problem, > > > > > > it's actually a feature: It makes it really clear that these are the drm > > > > > > fourcc codes, as allocated in drm_fourcc.h, plus their modifiers, and all > > > > > > that. That allocation authority is also already baked into various khr/ext > > > > > > standards, too. > > There's one thing that V4L2 has and DRM hasn't for 4CCs: good > documentation. Look at > https://linuxtv.org/downloads/v4l-dvb-apis-new/uapi/v4l/pixfmt-packed-rgb.html > or > https://linuxtv.org/downloads/v4l-dvb-apis-new/uapi/v4l/yuv-formats.html > for instance. It's painful to write, painful to read, but defines the > 4CCs very clearly without ambiguity. I wouldn't be surprised if > different drivers used the same DRM 4CC for different formats due to the > lack of detailed documentation. Moving to a shared library for 4CCs > should also address the documentation side, and any new format added to > the kernel, whether from the V4L2 side or the DRM side, would be > required to provide detailed documentation. That would be a great > improvement for DRM 4CC handling. > > > > > > The way I see it, there's a fundamental difference between the UAPI > > > > > and the kernel. I don't suggest we change anything about the UAPI: the > > > > > drm formats should keep their prefix, drm_fourcc.h can remain that > > > > > authority, it's all fine. > > > > > > > > > > Internally however, the long term goal is to share the fourcc's > > > > > between DRM and V4L2 for the same formats. It basically means that of > > > > > course v4l2 should be using the DRM fourcc when a format exists in DRM > > > > > and not v4l2, but also that DRM should use v4l2 fourcc when the format > > > > > exists in v4l2 but not DRM, and that is far more likely, given the > > > > > already extensive v4l2 formats support. > > > > > > > > Uh no. We did look at v4l fourcc extensively when deciding upon a drm > > > > format identifier space. > > > > > > No to what exactly? > > > > > > > And a lot of people pushed for the "fourcc is a standard", when > > > > really it's totally not. > > > > > > Even if it's not a standard, having consistency would be a good thing. > > > > > > And you said yourself that DRM fourcc is now pretty much an authority > > > for the fourcc, so it definitely looks like a standard to me. > > > > drm fourcc is the authority for drm fourcc codes. Not for any of the > > others (and there's lots of them). Now it's used in a bunch of other > > places (khr standards, dri protocols in wayland/X11), but they're > > still only drm fourcc. There is no overall fourcc standard. > > > > > > v4l tends to conflate pixel format with stuff that we tend to encode > > > > in modifiers a lot more. > > > > > > Boris is working on adding the modifiers concept to v4l2, so we're > > > converging here, and we can totally have a layer in v4l2 to convert > > > between old v4l2 "format+modifiers" formats, and DRM style formats. > > > > > > > There's a bunch of reasons we can't just use v4l, and they're as > > > > valid as ever: > > > > > > > > - We overlap badly in some areas, so even if fourcc codes match, we > > > > can't use them and need a duplicated DRM_FOURCC code. > > > > > > Do yo have an example of one of those areas? > > > > I think the rgb stuff was one of the big reasons to not reuse any > > existing fourcc standard (whether v4l, or another one from e.g. video > > container formats). We had initial patches to reuse the fourcc that > > existed, but the end result was really inconsistent, so we ditch that > > and rolled out a new set of entirely drm specific fourcc codes for > > rgba. > > Could you give one or a couple of examples of V4L2 4CCs that are not > OCD-compatible ? :-) > > > > > - v4l encodes some metadata into the fourcc that we encode elsewhere, > > > > e.g. offset for planar yuv formats, or tiling mode > > > > > > As I was saying, this changes on the v4l2 side, and converging to > > > what DRM is doing. > > > > > > > - drm fourcc code doesn't actually define the drm_format_info > > > > uniquely, drivers can override that (that's an explicit design > > > > intent of modifiers, to allow drivers to add another plane for > > > > e.g. compression information). You'd need to pull that driver > > > > knowledge into your format library. > > That's a mistake in my opinion. We tried that in V4L2 to store metadata > in a separate plane, and had to go another route eventually as it > created a very bad mess. Just quick clarification in the middle here: This is how the hw works. It's not metadata that sw ever touches (in general, testcases to make sure we display these correctly excepted). There has been some talking to add maybe a bit more mixed metadata, for fast-clear colors (which isn't used by any display engine afaik yet). That would generally be written by the cpu (in the gl stack), but again read by the hw (loaded as indirect state packet most likely, or something like that). So again hw specific layout, because the hw needs to read it. Pure metadata only of interest for the cpu/sw stack has been shot down completely on the drm side too. > There's a tendency in both subsystems to look at the other as a bit of a > retarded relative, and that's a shame, we have lots to learn from each > other's mistakes. That wouldn't be too difficult if people started > talking to each other. > > A semi-related comment: DRM also carries other mistakes of its own, I'm > thinking about DRM_FORMAT_BIG_ENDIAN in particular Yeah that one is hilarios, but in practice big endian is dead, except for a very few server chips, and there I think Gerd's work mostly fixed up that mess. > > > I'm not sure how my patches are changing anything here. This is > > > litterally the same code, with the functions renamed. > > > > > > If drivers want to override that, then yeah, fine, we can let them do > > > that. Just like any helper this just provides a default that covers > > > most of the cases. > > > > > > > Iow there's no way we can easily adopt v4l fourcc, except if we do > > > > something like a new addfb flag. > > > > > > For the formats that would be described as a modifier, sure. For all > > > the others (that are not yet supported by DRM), then I don't really > > > see why not. > > > > See above, we tried that initially, didn't pass the ocd tests when > > reviewing. Maybe the situation is better outside of rbgx/a formats, > > and I think we do at least try to use the same fourcc codes there when > > there already is one. But then the details occasionally look > > different. > > > > > > > And given how the current state is a mess in this regard, I'm not too > > > > > optimistic about keeping the formats in their relevant frameworks. > > > > > > > > > > Having a shared library, governed by both, will make this far easier, > > > > > since it will be easy to discover the formats that are already > > > > > supported by the other subsystem. > > > > > > > > I think a compat library that (tries to, best effort) convert between > > > > v4l and drm fourcc would make sense. Somewhere in drivers/video, next > > > > to the conversion functions for videomode <-> drm_display_mode > > > > perhaps. That should be useful for drivers. > > > > > > That's not really what this series is about though. That series is > > > about sharing the (image|pixels) formats database across everyone so > > > that everyone can benefit from it. > > > > Yeah I know. I still think leaving the drm fourcc with the drm prefix > > would be good, since there's really no standard here. > > > > > > Unifying the formats themselves, and all the associated metadata is > > > > imo a no-go, and was a pretty conscious decision when we implemented > > > > drm_fourcc a few years back. > > > > > > > > > If we want to keep the current library in DRM, we have two options > > > > > then: > > > > > > > > > > - Support all the v4l2 formats in the DRM library, which is > > > > > essentially what I'm doing in the last patches. However, that > > > > > would require to have the v4l2 developpers also reviewing stuff > > > > > there. And given how busy they are, I cannot really see how that > > > > > would work. > > > > > > > > Well, if we end up with a common library then yes we need cross > > > > review. There's no way around that. Doesn't matter where exactly that > > > > library is in the filesystem tree, and adding a special MAINTAINERS > > > > entry for anything related to fourcc (both drm and v4l) to make sure > > > > they get cross-posted is easy. No file renaming needed. > > > > > > That would create some governing issues as well. For example, if you > > > ever have a patch from one fourcc addition (that might or might not be > > > covered by v4l2), will you wait for any v4l2 developper to review it? > > > > None of this is fixed by code renaming or locations. Either way we > > need to figure that out. > > > > And yes part of the reasons for not moving this out of drm is that I'm > > not a fan of boutique trees for small stuff. If sharing means we need > > to split the drm_fourcc code and library out of drm trees, I'm not > > sure that's a good idea. We're already super liberal with merging > > anything through driver trees with acks, and integrating them quickly > > into drm-next. This would still be workable if v4l sends regular pull > > requests to drm-next (every 1-2 weeks, like the other big gpu trees > > do). If we can only sync up once per merge window with a shared > > boutique tree for formats only, life is going to be painful. > > That should be solved by the proposal above, maintaining the shared > library in the DRM tree. We would need to be careful there, and ideally > maintain that in a separate branch that could be merged in both DRM and > V4L2 without having to merge most of the other subsystem's pending > changes at the same time, but I think it's doable without any big issue. > > > > If it's shared code, then it should be shared, and every client > > > framework put on an equal footing. > > > > > > > > - Develop the same library from within v4l2. That is really a poor > > > > > solution, since the information would be greatly duplicated > > > > > between the two, and in terms of maintainance, code, and binary > > > > > size that would be duplicated too. > > > > > > > > It's essentially what we decided to do for drm years back. > > > > > > And it was probably the right solution back then, but I'm really not > > > convinced it's still the right thing to do today. > > > > > > > > Having it shared allows to easily share, and discover formats from the > > > > > other subsystem, and to have a single, unique place where this is > > > > > centralized. > > > > > > > > What I think could work as middle ground: > > > > - Put drm_format stuff into a separate .ko > > > > - Add a MAINTAINERS entry to make sure all things fourcc are cross > > > > posted to both drm and v4l lists. Easy on the drm side, since it's all > > > > separate files. Not sure it's so convenient for v4l uapi. > > > > - Add a conversion library that tries to best-effort map between drm > > > > and v4l formats. On the drm side that most likely means you need > > > > offsets for planes, and modifiers too (since those are implied in some > > > > v4l fourcc). Emphasis on "best effort" i.e. only support as much as > > > > the drivers that use this library need. > > > > - Add drm_fourcc as-needed by these drivers that want to use a unified > > > > format space. > > > > > > > > Forcing this unification on everyone otoh is imo way too much. > > > > > > v4l2 is starting to converge with DRM, and we're using the DRM API > > > pretty much untouched for that library, so I'm not really sure how > > > anyone is hurt by that unification. > > > > It might make sense to regularly pull v4l updates into drm-next then > > anyway. That would also remove the need to have the format library > > somewhere else. > > Are you saying it should the live in V4L2 ? ;-) Maybe a few clarifications on how the drm shared core thing usually works, and why I'm a sticker here. Bottom reply since I'm not sure where to put it: - Refactorings usually go in through drm-misc (at least since a few years). - Small patches go in through the relevant driver tree (which is often drm-misc, but not always), with an Ack from drm maintainers. - No topic branches, everyone just pushes patches where it's most convenient. We get away with this mess because everyone sends regular pull requests to drm, where the entire history is baked in and others can backmerge/fast forward/rebase. Worst case you wait one month (around the merge window, when drm-next is closed for features), but usually it's just 1-2 weeks. Plus we tend to have fairly big trees, with good chances that the next patch series lands in the same tree again and no work at all is needed. If we start regularly sharing lots of code with v4l (which seems to be the long term goal here), then I think we need something equally convenient for all that. We're not going to be able to teach some complicated topic branch scheme to the 50+ submaintainers/committers we have in drm - a lot much more basic stuff already takes lots of work to get it to stick. If the proposal is "to be careful" and "maintain it in a separate branch", I'm not in favour because I think that just wouldn't work. -Daniel
On Sun, Apr 21, 2019 at 01:40:45AM +0300, Laurent Pinchart wrote: > Hi Paul, > > On Thu, Apr 18, 2019 at 01:49:54PM +0200, Paul Kocialkowski wrote: > > On Thu, 2019-04-18 at 11:02 +0200, Maxime Ripard wrote: > > > On Thu, Apr 18, 2019 at 09:52:10AM +0200, Daniel Vetter wrote: > > >> And a lot of people pushed for the "fourcc is a standard", when > > >> really it's totally not. > > > > > > Even if it's not a standard, having consistency would be a good thing. > > > > > > And you said yourself that DRM fourcc is now pretty much an authority > > > for the fourcc, so it definitely looks like a standard to me. > > > > I think trying to make the V4L2 and DRM fourccs converge is a lost > > cause, as it has already significantly diverged. Even if we coordinate > > an effort to introduce new formats with the same fourcc on both sides, > > I don't see what good that would be since the formats we have now are > > still plagued by the inconsistency. > > > > I think we always need an explicit translation step from either v4l2 or > > drm to the internal representation and back, without ever assuming that > > formats might be compatible because they share the same fourcc. > > I don't agree. APIs evolve, and while we can't switch from one set of > 4CCs to another in existing APIs, we could in new APIs. Boris is working > on new ioctls to handle formats in V4L2, and while 4CC unification could > be impopular from a userspace developers point of view there, I don't > think we have ruled it out completely. The move to the request API is > also an area where a common set of 4CCs could be used, as it will depart > from the existing V4L2 ioctls. To summarize my opinion, we're not there > yet, but I wouldn't rule it out completely for the future. > > > It looks like so far, V4L2 pixel formats describe a DRM pixel format + > > modifier. > > DRM modifiers are mostly about tiling and compression, and we hardly > support these in V4L2. What are the modifiers you think are hardcoded in > 4CCs in V4L2 ? Hm maybe it was a drm one that didn't come from v4l or anywhere else really, but the NV12MT one is nv12 + some tiling. I think we managed to uapi-bend that one into shape in at least drm. -Daniel > > I think Boris (CCed) is working to change that by allowing to > > pass a DRM modifier through V4L2. With that, we'd be in a situation > > where some formats are described by the v4l2 pixfmt alone and some > > formats are also described a modifier (but I looked at it from a > > distance so might have misunderstod). That feels better since it avoids > > the combinatory explosion from describing each format + modifier > > individually. > > > > What do you think? > > > > >> v4l tends to conflate pixel format with stuff that we tend to encode > > >> in modifiers a lot more. > > > > > > Boris is working on adding the modifiers concept to v4l2, so we're > > > converging here, and we can totally have a layer in v4l2 to convert > > > between old v4l2 "format+modifiers" formats, and DRM style formats. > > > > > >> There's a bunch of reasons we can't just use v4l, and they're as > > >> valid as ever: > > >> > > >> - We overlap badly in some areas, so even if fourcc codes match, we > > >> can't use them and need a duplicated DRM_FOURCC code. > > > > > > Do yo have an example of one of those areas? > > > > > >> - v4l encodes some metadata into the fourcc that we encode elsewhere, > > >> e.g. offset for planar yuv formats, or tiling mode > > > > > > As I was saying, this changes on the v4l2 side, and converging to > > > what DRM is doing. > > > > > >> - drm fourcc code doesn't actually define the drm_format_info > > >> uniquely, drivers can override that (that's an explicit design > > >> intent of modifiers, to allow drivers to add another plane for > > >> e.g. compression information). You'd need to pull that driver > > >> knowledge into your format library. > > > > > > I'm not sure how my patches are changing anything here. This is > > > litterally the same code, with the functions renamed. > > > > > > If drivers want to override that, then yeah, fine, we can let them do > > > that. Just like any helper this just provides a default that covers > > > most of the cases. > > > > > >> Iow there's no way we can easily adopt v4l fourcc, except if we do > > >> something like a new addfb flag. > > > > > > For the formats that would be described as a modifier, sure. For all > > > the others (that are not yet supported by DRM), then I don't really > > > see why not. > > > > > >>> And given how the current state is a mess in this regard, I'm not too > > >>> optimistic about keeping the formats in their relevant frameworks. > > >>> > > >>> Having a shared library, governed by both, will make this far easier, > > >>> since it will be easy to discover the formats that are already > > >>> supported by the other subsystem. > > >> > > >> I think a compat library that (tries to, best effort) convert between > > >> v4l and drm fourcc would make sense. Somewhere in drivers/video, next > > >> to the conversion functions for videomode <-> drm_display_mode > > >> perhaps. That should be useful for drivers. > > > > > > That's not really what this series is about though. That series is > > > about sharing the (image|pixels) formats database across everyone so > > > that everyone can benefit from it. > > > > > >> Unifying the formats themselves, and all the associated metadata is > > >> imo a no-go, and was a pretty conscious decision when we implemented > > >> drm_fourcc a few years back. > > >> > > >>> If we want to keep the current library in DRM, we have two options > > >>> then: > > >>> > > >>> - Support all the v4l2 formats in the DRM library, which is > > >>> essentially what I'm doing in the last patches. However, that > > >>> would require to have the v4l2 developpers also reviewing stuff > > >>> there. And given how busy they are, I cannot really see how that > > >>> would work. > > >> > > >> Well, if we end up with a common library then yes we need cross > > >> review. There's no way around that. Doesn't matter where exactly that > > >> library is in the filesystem tree, and adding a special MAINTAINERS > > >> entry for anything related to fourcc (both drm and v4l) to make sure > > >> they get cross-posted is easy. No file renaming needed. > > > > > > That would create some governing issues as well. For example, if you > > > ever have a patch from one fourcc addition (that might or might not be > > > covered by v4l2), will you wait for any v4l2 developper to review it? > > > > > > If it's shared code, then it should be shared, and every client > > > framework put on an equal footing. > > > > > >>> - Develop the same library from within v4l2. That is really a poor > > >>> solution, since the information would be greatly duplicated > > >>> between the two, and in terms of maintainance, code, and binary > > >>> size that would be duplicated too. > > >> > > >> It's essentially what we decided to do for drm years back. > > > > > > And it was probably the right solution back then, but I'm really not > > > convinced it's still the right thing to do today. > > > > > >>> Having it shared allows to easily share, and discover formats from the > > >>> other subsystem, and to have a single, unique place where this is > > >>> centralized. > > >> > > >> What I think could work as middle ground: > > >> - Put drm_format stuff into a separate .ko > > >> - Add a MAINTAINERS entry to make sure all things fourcc are cross > > >> posted to both drm and v4l lists. Easy on the drm side, since it's all > > >> separate files. Not sure it's so convenient for v4l uapi. > > >> - Add a conversion library that tries to best-effort map between drm > > >> and v4l formats. On the drm side that most likely means you need > > >> offsets for planes, and modifiers too (since those are implied in some > > >> v4l fourcc). Emphasis on "best effort" i.e. only support as much as > > >> the drivers that use this library need. > > >> - Add drm_fourcc as-needed by these drivers that want to use a unified > > >> format space. > > >> > > >> Forcing this unification on everyone otoh is imo way too much. > > > > > > v4l2 is starting to converge with DRM, and we're using the DRM API > > > pretty much untouched for that library, so I'm not really sure how > > > anyone is hurt by that unification. > > -- > Regards, > > Laurent Pinchart
Hi, On Tue, 23 Apr 2019 at 08:26, Daniel Vetter <daniel@ffwll.ch> wrote: > On Sun, Apr 21, 2019 at 01:59:04AM +0300, Laurent Pinchart wrote: > > > > > - drm fourcc code doesn't actually define the drm_format_info > > > > > uniquely, drivers can override that (that's an explicit design > > > > > intent of modifiers, to allow drivers to add another plane for > > > > > e.g. compression information). You'd need to pull that driver > > > > > knowledge into your format library. > > > > That's a mistake in my opinion. We tried that in V4L2 to store metadata > > in a separate plane, and had to go another route eventually as it > > created a very bad mess. > > Just quick clarification in the middle here: This is how the hw works. > It's not metadata that sw ever touches (in general, testcases to make sure > we display these correctly excepted). > > There has been some talking to add maybe a bit more mixed metadata, for > fast-clear colors (which isn't used by any display engine afaik yet). That > would generally be written by the cpu (in the gl stack), but again read by > the hw (loaded as indirect state packet most likely, or something like > that). So again hw specific layout, because the hw needs to read it. > > Pure metadata only of interest for the cpu/sw stack has been shot down > completely on the drm side too. Totally. Let's take DRM_FORMAT_XRGB8888 + I915_FORMAT_MOD_Y_TILED as an example. Here, there is one colour plane which is laid out in a documented tiled format, containing normal XRGB8888 pixels once you do the maths to get the correct pixel location. So that's fine. I915_FORMAT_MOD_Y_TILED_CCS has a base colour plane as above, but adds an auxiliary plane which has a few bits describing the state of every (differently-sized) tile. Before reading the tile from the colour plane, you look at the corresponding location in the auxiliary plane: if you read 0x55 from the auxiliary plane, then the entire cacheline is the value of the first pixel, i.e. a solid fill. Hardware takes advantage of this to only write out the first pixel: if you try to read the colour plane as Y_TILED then for solid-filled regions, only the first pixel of every tile will show correctly, and the rest will be garbage. The auxiliary plane has its own layout and placement requirements, so we need to carry around an offset and a stride for the auxiliary data. We already have this for multiple planes; stuffing it into the base plane would require us to reinvent the same for auxiliary data within a single plane. I understand at least one of the Tegra colour-compression layouts (for Tegra 1xx?) is similar to this. It would be good to understand what you had in mind when you said that using multiple planes created a mess. I haven't touched media encode/decode units at a low level for quite a while (hooray for gst-v4l2!), but I remember that they often used padding areas around the buffer for scratch space - maybe motion vectors or similar? That case is quite different to something like CCS, since the data is only meaningful to the media engine and must be ignored (but preserved) by everyone else. Using multiple planes in that case isn't appropriate, since it's very specific to how that hardware unit deals with that buffer, instead of something that every consumer needs to understand in order to use it. Cheers, Daniel
Hi, On Tue, 2019-04-23 at 09:30 +0200, Daniel Vetter wrote: > On Sun, Apr 21, 2019 at 01:40:45AM +0300, Laurent Pinchart wrote: > > Hi Paul, > > > > On Thu, Apr 18, 2019 at 01:49:54PM +0200, Paul Kocialkowski wrote: > > > On Thu, 2019-04-18 at 11:02 +0200, Maxime Ripard wrote: > > > > On Thu, Apr 18, 2019 at 09:52:10AM +0200, Daniel Vetter wrote: > > > > > And a lot of people pushed for the "fourcc is a standard", when > > > > > really it's totally not. > > > > > > > > Even if it's not a standard, having consistency would be a good thing. > > > > > > > > And you said yourself that DRM fourcc is now pretty much an authority > > > > for the fourcc, so it definitely looks like a standard to me. > > > > > > I think trying to make the V4L2 and DRM fourccs converge is a lost > > > cause, as it has already significantly diverged. Even if we coordinate > > > an effort to introduce new formats with the same fourcc on both sides, > > > I don't see what good that would be since the formats we have now are > > > still plagued by the inconsistency. > > > > > > I think we always need an explicit translation step from either v4l2 or > > > drm to the internal representation and back, without ever assuming that > > > formats might be compatible because they share the same fourcc. > > > > I don't agree. APIs evolve, and while we can't switch from one set of > > 4CCs to another in existing APIs, we could in new APIs. Boris is working > > on new ioctls to handle formats in V4L2, and while 4CC unification could > > be impopular from a userspace developers point of view there, I don't > > think we have ruled it out completely. The move to the request API is > > also an area where a common set of 4CCs could be used, as it will depart > > from the existing V4L2 ioctls. To summarize my opinion, we're not there > > yet, but I wouldn't rule it out completely for the future. > > > > > It looks like so far, V4L2 pixel formats describe a DRM pixel format + > > > modifier. > > > > DRM modifiers are mostly about tiling and compression, and we hardly > > support these in V4L2. What are the modifiers you think are hardcoded in > > 4CCs in V4L2 ? > > Hm maybe it was a drm one that didn't come from v4l or anywhere else > really, but the NV12MT one is nv12 + some tiling. I think we managed to > uapi-bend that one into shape in at least drm. The one I had in mind is V4L2_PIX_FMT_SUNXI_TILED_NV12 which translates to DRM_FORMAT_NV12 + DRM_FORMAT_MOD_ALLWINNER_TILED. Seems to be a pretty similar case to the Mediatek one indeed. In our cause, that's because the video decoding engine produces its destination buffers in a specific tiled format, that the display engine can take in directly. Cheers, Paul > -Daniel > > > > I think Boris (CCed) is working to change that by allowing to > > > pass a DRM modifier through V4L2. With that, we'd be in a situation > > > where some formats are described by the v4l2 pixfmt alone and some > > > formats are also described a modifier (but I looked at it from a > > > distance so might have misunderstod). That feels better since it avoids > > > the combinatory explosion from describing each format + modifier > > > individually. > > > > > > What do you think? > > > > > > > > v4l tends to conflate pixel format with stuff that we tend to encode > > > > > in modifiers a lot more. > > > > > > > > Boris is working on adding the modifiers concept to v4l2, so we're > > > > converging here, and we can totally have a layer in v4l2 to convert > > > > between old v4l2 "format+modifiers" formats, and DRM style formats. > > > > > > > > > There's a bunch of reasons we can't just use v4l, and they're as > > > > > valid as ever: > > > > > > > > > > - We overlap badly in some areas, so even if fourcc codes match, we > > > > > can't use them and need a duplicated DRM_FOURCC code. > > > > > > > > Do yo have an example of one of those areas? > > > > > > > > > - v4l encodes some metadata into the fourcc that we encode elsewhere, > > > > > e.g. offset for planar yuv formats, or tiling mode > > > > > > > > As I was saying, this changes on the v4l2 side, and converging to > > > > what DRM is doing. > > > > > > > > > - drm fourcc code doesn't actually define the drm_format_info > > > > > uniquely, drivers can override that (that's an explicit design > > > > > intent of modifiers, to allow drivers to add another plane for > > > > > e.g. compression information). You'd need to pull that driver > > > > > knowledge into your format library. > > > > > > > > I'm not sure how my patches are changing anything here. This is > > > > litterally the same code, with the functions renamed. > > > > > > > > If drivers want to override that, then yeah, fine, we can let them do > > > > that. Just like any helper this just provides a default that covers > > > > most of the cases. > > > > > > > > > Iow there's no way we can easily adopt v4l fourcc, except if we do > > > > > something like a new addfb flag. > > > > > > > > For the formats that would be described as a modifier, sure. For all > > > > the others (that are not yet supported by DRM), then I don't really > > > > see why not. > > > > > > > > > > And given how the current state is a mess in this regard, I'm not too > > > > > > optimistic about keeping the formats in their relevant frameworks. > > > > > > > > > > > > Having a shared library, governed by both, will make this far easier, > > > > > > since it will be easy to discover the formats that are already > > > > > > supported by the other subsystem. > > > > > > > > > > I think a compat library that (tries to, best effort) convert between > > > > > v4l and drm fourcc would make sense. Somewhere in drivers/video, next > > > > > to the conversion functions for videomode <-> drm_display_mode > > > > > perhaps. That should be useful for drivers. > > > > > > > > That's not really what this series is about though. That series is > > > > about sharing the (image|pixels) formats database across everyone so > > > > that everyone can benefit from it. > > > > > > > > > Unifying the formats themselves, and all the associated metadata is > > > > > imo a no-go, and was a pretty conscious decision when we implemented > > > > > drm_fourcc a few years back. > > > > > > > > > > > If we want to keep the current library in DRM, we have two options > > > > > > then: > > > > > > > > > > > > - Support all the v4l2 formats in the DRM library, which is > > > > > > essentially what I'm doing in the last patches. However, that > > > > > > would require to have the v4l2 developpers also reviewing stuff > > > > > > there. And given how busy they are, I cannot really see how that > > > > > > would work. > > > > > > > > > > Well, if we end up with a common library then yes we need cross > > > > > review. There's no way around that. Doesn't matter where exactly that > > > > > library is in the filesystem tree, and adding a special MAINTAINERS > > > > > entry for anything related to fourcc (both drm and v4l) to make sure > > > > > they get cross-posted is easy. No file renaming needed. > > > > > > > > That would create some governing issues as well. For example, if you > > > > ever have a patch from one fourcc addition (that might or might not be > > > > covered by v4l2), will you wait for any v4l2 developper to review it? > > > > > > > > If it's shared code, then it should be shared, and every client > > > > framework put on an equal footing. > > > > > > > > > > - Develop the same library from within v4l2. That is really a poor > > > > > > solution, since the information would be greatly duplicated > > > > > > between the two, and in terms of maintainance, code, and binary > > > > > > size that would be duplicated too. > > > > > > > > > > It's essentially what we decided to do for drm years back. > > > > > > > > And it was probably the right solution back then, but I'm really not > > > > convinced it's still the right thing to do today. > > > > > > > > > > Having it shared allows to easily share, and discover formats from the > > > > > > other subsystem, and to have a single, unique place where this is > > > > > > centralized. > > > > > > > > > > What I think could work as middle ground: > > > > > - Put drm_format stuff into a separate .ko > > > > > - Add a MAINTAINERS entry to make sure all things fourcc are cross > > > > > posted to both drm and v4l lists. Easy on the drm side, since it's all > > > > > separate files. Not sure it's so convenient for v4l uapi. > > > > > - Add a conversion library that tries to best-effort map between drm > > > > > and v4l formats. On the drm side that most likely means you need > > > > > offsets for planes, and modifiers too (since those are implied in some > > > > > v4l fourcc). Emphasis on "best effort" i.e. only support as much as > > > > > the drivers that use this library need. > > > > > - Add drm_fourcc as-needed by these drivers that want to use a unified > > > > > format space. > > > > > > > > > > Forcing this unification on everyone otoh is imo way too much. > > > > > > > > v4l2 is starting to converge with DRM, and we're using the DRM API > > > > pretty much untouched for that library, so I'm not really sure how > > > > anyone is hurt by that unification. > > > > -- > > Regards, > > > > Laurent Pinchart
Le mardi 23 avril 2019 à 14:33 +0200, Paul Kocialkowski a écrit : > Hi, > > On Tue, 2019-04-23 at 09:30 +0200, Daniel Vetter wrote: > > On Sun, Apr 21, 2019 at 01:40:45AM +0300, Laurent Pinchart wrote: > > > Hi Paul, > > > > > > On Thu, Apr 18, 2019 at 01:49:54PM +0200, Paul Kocialkowski wrote: > > > > On Thu, 2019-04-18 at 11:02 +0200, Maxime Ripard wrote: > > > > > On Thu, Apr 18, 2019 at 09:52:10AM +0200, Daniel Vetter wrote: > > > > > > And a lot of people pushed for the "fourcc is a standard", when > > > > > > really it's totally not. > > > > > > > > > > Even if it's not a standard, having consistency would be a good thing. > > > > > > > > > > And you said yourself that DRM fourcc is now pretty much an authority > > > > > for the fourcc, so it definitely looks like a standard to me. > > > > > > > > I think trying to make the V4L2 and DRM fourccs converge is a lost > > > > cause, as it has already significantly diverged. Even if we coordinate > > > > an effort to introduce new formats with the same fourcc on both sides, > > > > I don't see what good that would be since the formats we have now are > > > > still plagued by the inconsistency. > > > > > > > > I think we always need an explicit translation step from either v4l2 or > > > > drm to the internal representation and back, without ever assuming that > > > > formats might be compatible because they share the same fourcc. > > > > > > I don't agree. APIs evolve, and while we can't switch from one set of > > > 4CCs to another in existing APIs, we could in new APIs. Boris is working > > > on new ioctls to handle formats in V4L2, and while 4CC unification could > > > be impopular from a userspace developers point of view there, I don't > > > think we have ruled it out completely. The move to the request API is > > > also an area where a common set of 4CCs could be used, as it will depart > > > from the existing V4L2 ioctls. To summarize my opinion, we're not there > > > yet, but I wouldn't rule it out completely for the future. > > > > > > > It looks like so far, V4L2 pixel formats describe a DRM pixel format + > > > > modifier. > > > > > > DRM modifiers are mostly about tiling and compression, and we hardly > > > support these in V4L2. What are the modifiers you think are hardcoded in > > > 4CCs in V4L2 ? > > > > Hm maybe it was a drm one that didn't come from v4l or anywhere else > > really, but the NV12MT one is nv12 + some tiling. I think we managed to > > uapi-bend that one into shape in at least drm. > > The one I had in mind is V4L2_PIX_FMT_SUNXI_TILED_NV12 which translates > to DRM_FORMAT_NV12 + DRM_FORMAT_MOD_ALLWINNER_TILED. Seems to be a > pretty similar case to the Mediatek one indeed. > > In our cause, that's because the video decoding engine produces its > destination buffers in a specific tiled format, that the display engine > can take in directly. We also have the Samsung tiling (Z pattern) as mentioned here, but also linear 16x16 tile placement (also from Samsung ?) and I believe Amlogic CODEC patches is bringing another tiling (unavoidable on older Meson8, with 64bytes swaps). All these should be expressed as NV12 + mod in DRM space. What is very often not enabled, but affect the performance on mainline media drivers is the ARM frame buffer compression. I know that RK chips have support for this, and that you can't achieve the maximum throughput without that. This one is not documented anywhere, but I understand that there is multiple variants that HW vendor licence. Though, in general, each SoC are likely running a single variant, so a single mod would make sense. So all this to say that V4L2 equally needs supports for these. What I understood through DRM API is that a buffer allocated for let's say NV12 + mod, is compatible with linear NV12. That could be used to simplify some code, but at the same time, a common API that deals with the padding and alignment of each format + mod independently would do that same as long as this is not variable depending on which target HW uses that same format. I think DRM + mod reduce the amount of dedicated formats that needs to be added, and simplify the documentation of each formats. I was looking at the Amlogic Axi configurations on Amlogic S905x recently, and for each well known format, there is a bitmask that let you do arbitrary swapping of bits, so effectively if we start exposing all these with V4L2 style, the list would become very long and hard to maintained. > > Cheers, > > Paul > > > -Daniel > > > > > > I think Boris (CCed) is working to change that by allowing to > > > > pass a DRM modifier through V4L2. With that, we'd be in a situation > > > > where some formats are described by the v4l2 pixfmt alone and some > > > > formats are also described a modifier (but I looked at it from a > > > > distance so might have misunderstod). That feels better since it avoids > > > > the combinatory explosion from describing each format + modifier > > > > individually. > > > > > > > > What do you think? > > > > > > > > > > v4l tends to conflate pixel format with stuff that we tend to encode > > > > > > in modifiers a lot more. > > > > > > > > > > Boris is working on adding the modifiers concept to v4l2, so we're > > > > > converging here, and we can totally have a layer in v4l2 to convert > > > > > between old v4l2 "format+modifiers" formats, and DRM style formats. > > > > > > > > > > > There's a bunch of reasons we can't just use v4l, and they're as > > > > > > valid as ever: > > > > > > > > > > > > - We overlap badly in some areas, so even if fourcc codes match, we > > > > > > can't use them and need a duplicated DRM_FOURCC code. > > > > > > > > > > Do yo have an example of one of those areas? > > > > > > > > > > > - v4l encodes some metadata into the fourcc that we encode elsewhere, > > > > > > e.g. offset for planar yuv formats, or tiling mode > > > > > > > > > > As I was saying, this changes on the v4l2 side, and converging to > > > > > what DRM is doing. > > > > > > > > > > > - drm fourcc code doesn't actually define the drm_format_info > > > > > > uniquely, drivers can override that (that's an explicit design > > > > > > intent of modifiers, to allow drivers to add another plane for > > > > > > e.g. compression information). You'd need to pull that driver > > > > > > knowledge into your format library. > > > > > > > > > > I'm not sure how my patches are changing anything here. This is > > > > > litterally the same code, with the functions renamed. > > > > > > > > > > If drivers want to override that, then yeah, fine, we can let them do > > > > > that. Just like any helper this just provides a default that covers > > > > > most of the cases. > > > > > > > > > > > Iow there's no way we can easily adopt v4l fourcc, except if we do > > > > > > something like a new addfb flag. > > > > > > > > > > For the formats that would be described as a modifier, sure. For all > > > > > the others (that are not yet supported by DRM), then I don't really > > > > > see why not. > > > > > > > > > > > > And given how the current state is a mess in this regard, I'm not too > > > > > > > optimistic about keeping the formats in their relevant frameworks. > > > > > > > > > > > > > > Having a shared library, governed by both, will make this far easier, > > > > > > > since it will be easy to discover the formats that are already > > > > > > > supported by the other subsystem. > > > > > > > > > > > > I think a compat library that (tries to, best effort) convert between > > > > > > v4l and drm fourcc would make sense. Somewhere in drivers/video, next > > > > > > to the conversion functions for videomode <-> drm_display_mode > > > > > > perhaps. That should be useful for drivers. > > > > > > > > > > That's not really what this series is about though. That series is > > > > > about sharing the (image|pixels) formats database across everyone so > > > > > that everyone can benefit from it. > > > > > > > > > > > Unifying the formats themselves, and all the associated metadata is > > > > > > imo a no-go, and was a pretty conscious decision when we implemented > > > > > > drm_fourcc a few years back. > > > > > > > > > > > > > If we want to keep the current library in DRM, we have two options > > > > > > > then: > > > > > > > > > > > > > > - Support all the v4l2 formats in the DRM library, which is > > > > > > > essentially what I'm doing in the last patches. However, that > > > > > > > would require to have the v4l2 developpers also reviewing stuff > > > > > > > there. And given how busy they are, I cannot really see how that > > > > > > > would work. > > > > > > > > > > > > Well, if we end up with a common library then yes we need cross > > > > > > review. There's no way around that. Doesn't matter where exactly that > > > > > > library is in the filesystem tree, and adding a special MAINTAINERS > > > > > > entry for anything related to fourcc (both drm and v4l) to make sure > > > > > > they get cross-posted is easy. No file renaming needed. > > > > > > > > > > That would create some governing issues as well. For example, if you > > > > > ever have a patch from one fourcc addition (that might or might not be > > > > > covered by v4l2), will you wait for any v4l2 developper to review it? > > > > > > > > > > If it's shared code, then it should be shared, and every client > > > > > framework put on an equal footing. > > > > > > > > > > > > - Develop the same library from within v4l2. That is really a poor > > > > > > > solution, since the information would be greatly duplicated > > > > > > > between the two, and in terms of maintainance, code, and binary > > > > > > > size that would be duplicated too. > > > > > > > > > > > > It's essentially what we decided to do for drm years back. > > > > > > > > > > And it was probably the right solution back then, but I'm really not > > > > > convinced it's still the right thing to do today. > > > > > > > > > > > > Having it shared allows to easily share, and discover formats from the > > > > > > > other subsystem, and to have a single, unique place where this is > > > > > > > centralized. > > > > > > > > > > > > What I think could work as middle ground: > > > > > > - Put drm_format stuff into a separate .ko > > > > > > - Add a MAINTAINERS entry to make sure all things fourcc are cross > > > > > > posted to both drm and v4l lists. Easy on the drm side, since it's all > > > > > > separate files. Not sure it's so convenient for v4l uapi. > > > > > > - Add a conversion library that tries to best-effort map between drm > > > > > > and v4l formats. On the drm side that most likely means you need > > > > > > offsets for planes, and modifiers too (since those are implied in some > > > > > > v4l fourcc). Emphasis on "best effort" i.e. only support as much as > > > > > > the drivers that use this library need. > > > > > > - Add drm_fourcc as-needed by these drivers that want to use a unified > > > > > > format space. > > > > > > > > > > > > Forcing this unification on everyone otoh is imo way too much. > > > > > > > > > > v4l2 is starting to converge with DRM, and we're using the DRM API > > > > > pretty much untouched for that library, so I'm not really sure how > > > > > anyone is hurt by that unification. > > > > > > -- > > > Regards, > > > > > > Laurent Pinchart
Hi, On Tue, 2019-04-23 at 10:28 -0400, Nicolas Dufresne wrote: > Le mardi 23 avril 2019 à 14:33 +0200, Paul Kocialkowski a écrit : > > Hi, > > > > On Tue, 2019-04-23 at 09:30 +0200, Daniel Vetter wrote: > > > On Sun, Apr 21, 2019 at 01:40:45AM +0300, Laurent Pinchart wrote: > > > > Hi Paul, > > > > > > > > On Thu, Apr 18, 2019 at 01:49:54PM +0200, Paul Kocialkowski wrote: > > > > > On Thu, 2019-04-18 at 11:02 +0200, Maxime Ripard wrote: > > > > > > On Thu, Apr 18, 2019 at 09:52:10AM +0200, Daniel Vetter wrote: > > > > > > > And a lot of people pushed for the "fourcc is a standard", when > > > > > > > really it's totally not. > > > > > > > > > > > > Even if it's not a standard, having consistency would be a good thing. > > > > > > > > > > > > And you said yourself that DRM fourcc is now pretty much an authority > > > > > > for the fourcc, so it definitely looks like a standard to me. > > > > > > > > > > I think trying to make the V4L2 and DRM fourccs converge is a lost > > > > > cause, as it has already significantly diverged. Even if we coordinate > > > > > an effort to introduce new formats with the same fourcc on both sides, > > > > > I don't see what good that would be since the formats we have now are > > > > > still plagued by the inconsistency. > > > > > > > > > > I think we always need an explicit translation step from either v4l2 or > > > > > drm to the internal representation and back, without ever assuming that > > > > > formats might be compatible because they share the same fourcc. > > > > > > > > I don't agree. APIs evolve, and while we can't switch from one set of > > > > 4CCs to another in existing APIs, we could in new APIs. Boris is working > > > > on new ioctls to handle formats in V4L2, and while 4CC unification could > > > > be impopular from a userspace developers point of view there, I don't > > > > think we have ruled it out completely. The move to the request API is > > > > also an area where a common set of 4CCs could be used, as it will depart > > > > from the existing V4L2 ioctls. To summarize my opinion, we're not there > > > > yet, but I wouldn't rule it out completely for the future. > > > > > > > > > It looks like so far, V4L2 pixel formats describe a DRM pixel format + > > > > > modifier. > > > > > > > > DRM modifiers are mostly about tiling and compression, and we hardly > > > > support these in V4L2. What are the modifiers you think are hardcoded in > > > > 4CCs in V4L2 ? > > > > > > Hm maybe it was a drm one that didn't come from v4l or anywhere else > > > really, but the NV12MT one is nv12 + some tiling. I think we managed to > > > uapi-bend that one into shape in at least drm. > > > > The one I had in mind is V4L2_PIX_FMT_SUNXI_TILED_NV12 which translates > > to DRM_FORMAT_NV12 + DRM_FORMAT_MOD_ALLWINNER_TILED. Seems to be a > > pretty similar case to the Mediatek one indeed. > > > > In our cause, that's because the video decoding engine produces its > > destination buffers in a specific tiled format, that the display engine > > can take in directly. > > We also have the Samsung tiling (Z pattern) as mentioned here, but also > linear 16x16 tile placement (also from Samsung ?) and I believe Amlogic > CODEC patches is bringing another tiling (unavoidable on older Meson8, > with 64bytes swaps). All these should be expressed as NV12 + mod in DRM > space. > > What is very often not enabled, but affect the performance on mainline > media drivers is the ARM frame buffer compression. I know that RK chips > have support for this, and that you can't achieve the maximum > throughput without that. This one is not documented anywhere, but I > understand that there is multiple variants that HW vendor licence. > Though, in general, each SoC are likely running a single variant, so a > single mod would make sense. > > So all this to say that V4L2 equally needs supports for these. Yes, I definitely agree with that. > What I understood through DRM API is that a buffer allocated for let's say > NV12 + mod, is compatible with linear NV12. That could be used to > simplify some code, but at the same time, a common API that deals with > the padding and alignment of each format + mod independently would do > that same as long as this is not variable depending on which target HW > uses that same format. > > I think DRM + mod reduce the amount of dedicated formats that needs to > be added, and simplify the documentation of each formats. I was looking > at the Amlogic Axi configurations on Amlogic S905x recently, and for > each well known format, there is a bitmask that let you do arbitrary > swapping of bits, so effectively if we start exposing all these with > V4L2 style, the list would become very long and hard to maintained. The number of possible combinations is indeed much too high to have them exposed as single V4L2 formats. Not to mention the VC4 tiling modifiers where the column height is passed as an argument to the modifier (so there is a modifier base and a variable part to it). So I definitely agree that having a way to represent modifiers in V4L2 would be quite relevant and useful. Cheers, Paul > > > -Daniel > > > > > > > > I think Boris (CCed) is working to change that by allowing to > > > > > pass a DRM modifier through V4L2. With that, we'd be in a situation > > > > > where some formats are described by the v4l2 pixfmt alone and some > > > > > formats are also described a modifier (but I looked at it from a > > > > > distance so might have misunderstod). That feels better since it avoids > > > > > the combinatory explosion from describing each format + modifier > > > > > individually. > > > > > > > > > > What do you think? > > > > > > > > > > > > v4l tends to conflate pixel format with stuff that we tend to encode > > > > > > > in modifiers a lot more. > > > > > > > > > > > > Boris is working on adding the modifiers concept to v4l2, so we're > > > > > > converging here, and we can totally have a layer in v4l2 to convert > > > > > > between old v4l2 "format+modifiers" formats, and DRM style formats. > > > > > > > > > > > > > There's a bunch of reasons we can't just use v4l, and they're as > > > > > > > valid as ever: > > > > > > > > > > > > > > - We overlap badly in some areas, so even if fourcc codes match, we > > > > > > > can't use them and need a duplicated DRM_FOURCC code. > > > > > > > > > > > > Do yo have an example of one of those areas? > > > > > > > > > > > > > - v4l encodes some metadata into the fourcc that we encode elsewhere, > > > > > > > e.g. offset for planar yuv formats, or tiling mode > > > > > > > > > > > > As I was saying, this changes on the v4l2 side, and converging to > > > > > > what DRM is doing. > > > > > > > > > > > > > - drm fourcc code doesn't actually define the drm_format_info > > > > > > > uniquely, drivers can override that (that's an explicit design > > > > > > > intent of modifiers, to allow drivers to add another plane for > > > > > > > e.g. compression information). You'd need to pull that driver > > > > > > > knowledge into your format library. > > > > > > > > > > > > I'm not sure how my patches are changing anything here. This is > > > > > > litterally the same code, with the functions renamed. > > > > > > > > > > > > If drivers want to override that, then yeah, fine, we can let them do > > > > > > that. Just like any helper this just provides a default that covers > > > > > > most of the cases. > > > > > > > > > > > > > Iow there's no way we can easily adopt v4l fourcc, except if we do > > > > > > > something like a new addfb flag. > > > > > > > > > > > > For the formats that would be described as a modifier, sure. For all > > > > > > the others (that are not yet supported by DRM), then I don't really > > > > > > see why not. > > > > > > > > > > > > > > And given how the current state is a mess in this regard, I'm not too > > > > > > > > optimistic about keeping the formats in their relevant frameworks. > > > > > > > > > > > > > > > > Having a shared library, governed by both, will make this far easier, > > > > > > > > since it will be easy to discover the formats that are already > > > > > > > > supported by the other subsystem. > > > > > > > > > > > > > > I think a compat library that (tries to, best effort) convert between > > > > > > > v4l and drm fourcc would make sense. Somewhere in drivers/video, next > > > > > > > to the conversion functions for videomode <-> drm_display_mode > > > > > > > perhaps. That should be useful for drivers. > > > > > > > > > > > > That's not really what this series is about though. That series is > > > > > > about sharing the (image|pixels) formats database across everyone so > > > > > > that everyone can benefit from it. > > > > > > > > > > > > > Unifying the formats themselves, and all the associated metadata is > > > > > > > imo a no-go, and was a pretty conscious decision when we implemented > > > > > > > drm_fourcc a few years back. > > > > > > > > > > > > > > > If we want to keep the current library in DRM, we have two options > > > > > > > > then: > > > > > > > > > > > > > > > > - Support all the v4l2 formats in the DRM library, which is > > > > > > > > essentially what I'm doing in the last patches. However, that > > > > > > > > would require to have the v4l2 developpers also reviewing stuff > > > > > > > > there. And given how busy they are, I cannot really see how that > > > > > > > > would work. > > > > > > > > > > > > > > Well, if we end up with a common library then yes we need cross > > > > > > > review. There's no way around that. Doesn't matter where exactly that > > > > > > > library is in the filesystem tree, and adding a special MAINTAINERS > > > > > > > entry for anything related to fourcc (both drm and v4l) to make sure > > > > > > > they get cross-posted is easy. No file renaming needed. > > > > > > > > > > > > That would create some governing issues as well. For example, if you > > > > > > ever have a patch from one fourcc addition (that might or might not be > > > > > > covered by v4l2), will you wait for any v4l2 developper to review it? > > > > > > > > > > > > If it's shared code, then it should be shared, and every client > > > > > > framework put on an equal footing. > > > > > > > > > > > > > > - Develop the same library from within v4l2. That is really a poor > > > > > > > > solution, since the information would be greatly duplicated > > > > > > > > between the two, and in terms of maintainance, code, and binary > > > > > > > > size that would be duplicated too. > > > > > > > > > > > > > > It's essentially what we decided to do for drm years back. > > > > > > > > > > > > And it was probably the right solution back then, but I'm really not > > > > > > convinced it's still the right thing to do today. > > > > > > > > > > > > > > Having it shared allows to easily share, and discover formats from the > > > > > > > > other subsystem, and to have a single, unique place where this is > > > > > > > > centralized. > > > > > > > > > > > > > > What I think could work as middle ground: > > > > > > > - Put drm_format stuff into a separate .ko > > > > > > > - Add a MAINTAINERS entry to make sure all things fourcc are cross > > > > > > > posted to both drm and v4l lists. Easy on the drm side, since it's all > > > > > > > separate files. Not sure it's so convenient for v4l uapi. > > > > > > > - Add a conversion library that tries to best-effort map between drm > > > > > > > and v4l formats. On the drm side that most likely means you need > > > > > > > offsets for planes, and modifiers too (since those are implied in some > > > > > > > v4l fourcc). Emphasis on "best effort" i.e. only support as much as > > > > > > > the drivers that use this library need. > > > > > > > - Add drm_fourcc as-needed by these drivers that want to use a unified > > > > > > > format space. > > > > > > > > > > > > > > Forcing this unification on everyone otoh is imo way too much. > > > > > > > > > > > > v4l2 is starting to converge with DRM, and we're using the DRM API > > > > > > pretty much untouched for that library, so I'm not really sure how > > > > > > anyone is hurt by that unification. > > > > > > > > -- > > > > Regards, > > > > > > > > Laurent Pinchart
On Tue, Apr 23, 2019 at 4:28 PM Nicolas Dufresne <nicolas@ndufresne.ca> wrote: > > Le mardi 23 avril 2019 à 14:33 +0200, Paul Kocialkowski a écrit : > > Hi, > > > > On Tue, 2019-04-23 at 09:30 +0200, Daniel Vetter wrote: > > > On Sun, Apr 21, 2019 at 01:40:45AM +0300, Laurent Pinchart wrote: > > > > Hi Paul, > > > > > > > > On Thu, Apr 18, 2019 at 01:49:54PM +0200, Paul Kocialkowski wrote: > > > > > On Thu, 2019-04-18 at 11:02 +0200, Maxime Ripard wrote: > > > > > > On Thu, Apr 18, 2019 at 09:52:10AM +0200, Daniel Vetter wrote: > > > > > > > And a lot of people pushed for the "fourcc is a standard", when > > > > > > > really it's totally not. > > > > > > > > > > > > Even if it's not a standard, having consistency would be a good thing. > > > > > > > > > > > > And you said yourself that DRM fourcc is now pretty much an authority > > > > > > for the fourcc, so it definitely looks like a standard to me. > > > > > > > > > > I think trying to make the V4L2 and DRM fourccs converge is a lost > > > > > cause, as it has already significantly diverged. Even if we coordinate > > > > > an effort to introduce new formats with the same fourcc on both sides, > > > > > I don't see what good that would be since the formats we have now are > > > > > still plagued by the inconsistency. > > > > > > > > > > I think we always need an explicit translation step from either v4l2 or > > > > > drm to the internal representation and back, without ever assuming that > > > > > formats might be compatible because they share the same fourcc. > > > > > > > > I don't agree. APIs evolve, and while we can't switch from one set of > > > > 4CCs to another in existing APIs, we could in new APIs. Boris is working > > > > on new ioctls to handle formats in V4L2, and while 4CC unification could > > > > be impopular from a userspace developers point of view there, I don't > > > > think we have ruled it out completely. The move to the request API is > > > > also an area where a common set of 4CCs could be used, as it will depart > > > > from the existing V4L2 ioctls. To summarize my opinion, we're not there > > > > yet, but I wouldn't rule it out completely for the future. > > > > > > > > > It looks like so far, V4L2 pixel formats describe a DRM pixel format + > > > > > modifier. > > > > > > > > DRM modifiers are mostly about tiling and compression, and we hardly > > > > support these in V4L2. What are the modifiers you think are hardcoded in > > > > 4CCs in V4L2 ? > > > > > > Hm maybe it was a drm one that didn't come from v4l or anywhere else > > > really, but the NV12MT one is nv12 + some tiling. I think we managed to > > > uapi-bend that one into shape in at least drm. > > > > The one I had in mind is V4L2_PIX_FMT_SUNXI_TILED_NV12 which translates > > to DRM_FORMAT_NV12 + DRM_FORMAT_MOD_ALLWINNER_TILED. Seems to be a > > pretty similar case to the Mediatek one indeed. > > > > In our cause, that's because the video decoding engine produces its > > destination buffers in a specific tiled format, that the display engine > > can take in directly. > > We also have the Samsung tiling (Z pattern) as mentioned here, but also > linear 16x16 tile placement (also from Samsung ?) and I believe Amlogic > CODEC patches is bringing another tiling (unavoidable on older Meson8, > with 64bytes swaps). All these should be expressed as NV12 + mod in DRM > space. > > What is very often not enabled, but affect the performance on mainline > media drivers is the ARM frame buffer compression. I know that RK chips > have support for this, and that you can't achieve the maximum > throughput without that. This one is not documented anywhere, but I > understand that there is multiple variants that HW vendor licence. > Though, in general, each SoC are likely running a single variant, so a > single mod would make sense. We have AFBC modifiers now in drm_fourcc.h, jointly developed by display engineers from ARM and mali gpu reverse engineer people doing the panfrost driver. So should be covered. > So all this to say that V4L2 equally needs supports for these. What I > understood through DRM API is that a buffer allocated for let's say > NV12 + mod, is compatible with linear NV12. That could be used to > simplify some code, but at the same time, a common API that deals with > the padding and alignment of each format + mod independently would do > that same as long as this is not variable depending on which target HW > uses that same format. Not sure why you mean with NV12 + mod is compatible with linear NV12. In general fourcc + modifier != fourcc = linear modifier, size, number of planes, alignment constraints and everything else can be changed by a modifier (and there's examples for all of these, maybe not yet in all cases for NV12, but I think NV12 + AFBC modifiers gives some pretty interesting results). In generally you need to think of the (drm fourcc, modifier) as the pair identified the pixel format, each part individually is fairly meanigless. We have lots of modifiers where the exact tiling mode/layout changes depending upon the fourcc code. The only exception is legacy interfaces, i.e. when you create a framebuffer with only the drm fourcc and not a modifier. In that case you get driver specific behaviour, but modifier aware drivers tend to change that into a specific (fourcc, modifier) pair again (at least i915.ko, and it's what I recommend). Oh and we have some legacy modifiers that depend upon the target hw, but it's very much not recommended (we did it in i915 to make things easier on really old platforms, on some of them we don't even know the exact tiling mode since it's not documented nor correctly reverse-engineered). Another fun case are some of the recent non-byte-aligned formats, for which you need to have a modifier to be able to use them with anything - there's not really a real linear layout for them, they just serve as an index/parameter into the modifier space. > I think DRM + mod reduce the amount of dedicated formats that needs to > be added, and simplify the documentation of each formats. I was looking > at the Amlogic Axi configurations on Amlogic S905x recently, and for > each well known format, there is a bitmask that let you do arbitrary > swapping of bits, so effectively if we start exposing all these with > V4L2 style, the list would become very long and hard to maintained. See above, modifiers aren't really simple ... -Daniel > > > > Cheers, > > > > Paul > > > > > -Daniel > > > > > > > > I think Boris (CCed) is working to change that by allowing to > > > > > pass a DRM modifier through V4L2. With that, we'd be in a situation > > > > > where some formats are described by the v4l2 pixfmt alone and some > > > > > formats are also described a modifier (but I looked at it from a > > > > > distance so might have misunderstod). That feels better since it avoids > > > > > the combinatory explosion from describing each format + modifier > > > > > individually. > > > > > > > > > > What do you think? > > > > > > > > > > > > v4l tends to conflate pixel format with stuff that we tend to encode > > > > > > > in modifiers a lot more. > > > > > > > > > > > > Boris is working on adding the modifiers concept to v4l2, so we're > > > > > > converging here, and we can totally have a layer in v4l2 to convert > > > > > > between old v4l2 "format+modifiers" formats, and DRM style formats. > > > > > > > > > > > > > There's a bunch of reasons we can't just use v4l, and they're as > > > > > > > valid as ever: > > > > > > > > > > > > > > - We overlap badly in some areas, so even if fourcc codes match, we > > > > > > > can't use them and need a duplicated DRM_FOURCC code. > > > > > > > > > > > > Do yo have an example of one of those areas? > > > > > > > > > > > > > - v4l encodes some metadata into the fourcc that we encode elsewhere, > > > > > > > e.g. offset for planar yuv formats, or tiling mode > > > > > > > > > > > > As I was saying, this changes on the v4l2 side, and converging to > > > > > > what DRM is doing. > > > > > > > > > > > > > - drm fourcc code doesn't actually define the drm_format_info > > > > > > > uniquely, drivers can override that (that's an explicit design > > > > > > > intent of modifiers, to allow drivers to add another plane for > > > > > > > e.g. compression information). You'd need to pull that driver > > > > > > > knowledge into your format library. > > > > > > > > > > > > I'm not sure how my patches are changing anything here. This is > > > > > > litterally the same code, with the functions renamed. > > > > > > > > > > > > If drivers want to override that, then yeah, fine, we can let them do > > > > > > that. Just like any helper this just provides a default that covers > > > > > > most of the cases. > > > > > > > > > > > > > Iow there's no way we can easily adopt v4l fourcc, except if we do > > > > > > > something like a new addfb flag. > > > > > > > > > > > > For the formats that would be described as a modifier, sure. For all > > > > > > the others (that are not yet supported by DRM), then I don't really > > > > > > see why not. > > > > > > > > > > > > > > And given how the current state is a mess in this regard, I'm not too > > > > > > > > optimistic about keeping the formats in their relevant frameworks. > > > > > > > > > > > > > > > > Having a shared library, governed by both, will make this far easier, > > > > > > > > since it will be easy to discover the formats that are already > > > > > > > > supported by the other subsystem. > > > > > > > > > > > > > > I think a compat library that (tries to, best effort) convert between > > > > > > > v4l and drm fourcc would make sense. Somewhere in drivers/video, next > > > > > > > to the conversion functions for videomode <-> drm_display_mode > > > > > > > perhaps. That should be useful for drivers. > > > > > > > > > > > > That's not really what this series is about though. That series is > > > > > > about sharing the (image|pixels) formats database across everyone so > > > > > > that everyone can benefit from it. > > > > > > > > > > > > > Unifying the formats themselves, and all the associated metadata is > > > > > > > imo a no-go, and was a pretty conscious decision when we implemented > > > > > > > drm_fourcc a few years back. > > > > > > > > > > > > > > > If we want to keep the current library in DRM, we have two options > > > > > > > > then: > > > > > > > > > > > > > > > > - Support all the v4l2 formats in the DRM library, which is > > > > > > > > essentially what I'm doing in the last patches. However, that > > > > > > > > would require to have the v4l2 developpers also reviewing stuff > > > > > > > > there. And given how busy they are, I cannot really see how that > > > > > > > > would work. > > > > > > > > > > > > > > Well, if we end up with a common library then yes we need cross > > > > > > > review. There's no way around that. Doesn't matter where exactly that > > > > > > > library is in the filesystem tree, and adding a special MAINTAINERS > > > > > > > entry for anything related to fourcc (both drm and v4l) to make sure > > > > > > > they get cross-posted is easy. No file renaming needed. > > > > > > > > > > > > That would create some governing issues as well. For example, if you > > > > > > ever have a patch from one fourcc addition (that might or might not be > > > > > > covered by v4l2), will you wait for any v4l2 developper to review it? > > > > > > > > > > > > If it's shared code, then it should be shared, and every client > > > > > > framework put on an equal footing. > > > > > > > > > > > > > > - Develop the same library from within v4l2. That is really a poor > > > > > > > > solution, since the information would be greatly duplicated > > > > > > > > between the two, and in terms of maintainance, code, and binary > > > > > > > > size that would be duplicated too. > > > > > > > > > > > > > > It's essentially what we decided to do for drm years back. > > > > > > > > > > > > And it was probably the right solution back then, but I'm really not > > > > > > convinced it's still the right thing to do today. > > > > > > > > > > > > > > Having it shared allows to easily share, and discover formats from the > > > > > > > > other subsystem, and to have a single, unique place where this is > > > > > > > > centralized. > > > > > > > > > > > > > > What I think could work as middle ground: > > > > > > > - Put drm_format stuff into a separate .ko > > > > > > > - Add a MAINTAINERS entry to make sure all things fourcc are cross > > > > > > > posted to both drm and v4l lists. Easy on the drm side, since it's all > > > > > > > separate files. Not sure it's so convenient for v4l uapi. > > > > > > > - Add a conversion library that tries to best-effort map between drm > > > > > > > and v4l formats. On the drm side that most likely means you need > > > > > > > offsets for planes, and modifiers too (since those are implied in some > > > > > > > v4l fourcc). Emphasis on "best effort" i.e. only support as much as > > > > > > > the drivers that use this library need. > > > > > > > - Add drm_fourcc as-needed by these drivers that want to use a unified > > > > > > > format space. > > > > > > > > > > > > > > Forcing this unification on everyone otoh is imo way too much. > > > > > > > > > > > > v4l2 is starting to converge with DRM, and we're using the DRM API > > > > > > pretty much untouched for that library, so I'm not really sure how > > > > > > anyone is hurt by that unification. > > > > > > > > -- > > > > Regards, > > > > > > > > Laurent Pinchart
Hi Daniel, On Tue, Apr 23, 2019 at 09:25:54AM +0200, Daniel Vetter wrote: > On Sun, Apr 21, 2019 at 01:59:04AM +0300, Laurent Pinchart wrote: > > On Thu, Apr 18, 2019 at 12:07:44PM +0200, Daniel Vetter wrote: > >> On Thu, Apr 18, 2019 at 11:02 AM Maxime Ripard wrote: > >>> On Thu, Apr 18, 2019 at 09:52:10AM +0200, Daniel Vetter wrote: > >>>> On Thu, Apr 18, 2019 at 8:22 AM Maxime Ripard wrote: > >>>>> On Wed, Apr 17, 2019 at 05:41:21PM +0200, Daniel Vetter wrote: > >>>>>> On Wed, Apr 17, 2019 at 09:54:26AM +0200, Maxime Ripard wrote: > >>>>>>> Hi, > >>>>>>> > >>>>>>> DRM comes with an extensive format support to retrieve the various > >>>>>>> parameters associated with a given format (such as the subsampling, or the > >>>>>>> bits per pixel), as well as some helpers and utilities to ease the driver > >>>>>>> development. > >>>>>>> > >>>>>>> v4l2, on the other side, doesn't provide such facilities, leaving each > >>>>>>> driver reimplement a subset of the formats parameters for the one supported > >>>>>>> by that particular driver. This leads to a lot of duplication and > >>>>>>> boilerplate code in the v4l2 drivers. > >>>>>>> > >>>>>>> This series tries to address this by moving the DRM format API into lib and > >>>>>>> turning it into a more generic API. In order to do this, we've needed to do > >>>>>>> some preliminary changes on the DRM drivers, then moved the API and finally > >>>>>>> converted a v4l2 driver to give an example of how such library could be > >>>>>>> used. > >>>>>>> > >>>>>>> Let me know what you think, > >>>>>>> Maxime > >>>>>>> > >>>>>>> Changes from RFC: > >>>>>>> - Rebased on next > >>>>>>> - Fixed the various formats mapping > >>>>>>> - Added tags > >>>>>>> - Did most of the formats functions as inline functions > >>>>>>> - Used a consistent prefix for all the utilities functions > >>>>>>> - Fixed the compilation breakages, and did a run of allmodconfig for arm, > >>>>>>> arm64 and x86_64 > >>>>>>> - Fixed out of array bounds warnings in the image_format_info_block_* > >>>>>>> functions > >>>>>>> - Added License and copyright headers on missing files > >>>>>>> > >>>>>>> Maxime Ripard (20): > >>>>>>> drm: Remove users of drm_format_num_planes > >>>>>>> drm: Remove users of drm_format_(horz|vert)_chroma_subsampling > >>>>>>> drm/fourcc: Pass the format_info pointer to drm_format_plane_cpp > >>>>>>> drm/fourcc: Pass the format_info pointer to drm_format_plane_width/height > >>>>>>> drm: Replace instances of drm_format_info by drm_get_format_info > >>>>>>> lib: Add video format information library > >>>>>>> drm/fb: Move from drm_format_info to image_format_info > >>>>>>> drm/malidp: Convert to generic image format library > >>>>>>> drm/client: Convert to generic image format library > >>>>>>> drm/exynos: Convert to generic image format library > >>>>>>> drm/i915: Convert to generic image format library > >>>>>>> drm/ipuv3: Convert to generic image format library > >>>>>>> drm/msm: Convert to generic image format library > >>>>>>> drm/omap: Convert to generic image format library > >>>>>>> drm/rockchip: Convert to generic image format library > >>>>>>> drm/tegra: Convert to generic image format library > >>>>>>> drm/fourcc: Remove old DRM format API > >>>>>>> lib: image-formats: Add v4l2 formats support > >>>>>>> lib: image-formats: Add more functions > >>>>>>> media: sun6i: Convert to the image format API > >>>>>> > >>>>>> In the interest of making myself unpopular: Why move this out of drm? > >>>>>> > >>>>>> We do have a bunch of other such shared helpers already (mostly in > >>>>>> drivers/video) for dt videomode and hdmi infoframes, and I'm not super > >>>>>> sure that's going better than keeping it maintained in drm. > > > > That's a bit of a different situation as both DRM and FBDEV address the > > same features (display output), and FBDEV is deprecared and replaced by > > DRM. > > > > I'm not against maintaining a 4CC library in DRM (adding a third-party > > maintainer would likely create more problems than it would solve), but > > that doesn't mean the library has to live in drivers/gpu/, nor that it > > needs to have the drm_ prefix. I would actually advocate to make it live > > in a neutral directory, with a neutral prefix (kudos to anyone who can > > propose a nice naming convention that would establish a new shared > > ground for image/video-related Linux APIs), and maintained through the > > DRM tree (possibly with extra entries in MAINTAINERS to ensure it > > reaches all the related folks). > > > >>>>>> Plus the uapi is already that you include drm_fourcc.h to get at these, > >>>>>> dropping the drm prefix isn't going to help I think. > >>>>>> > >>>>>> Of course we'd need to make it a separate drm_formats.ko (so that v4l can > >>>>>> use it without dragging in all of drm), and we need to add some fields for > >>>>>> converting to v4l fourcc codes (which are different). But that should be > >>>>>> all possible. And I don't think the drm_ prefix in v4l code is a problem, > >>>>>> it's actually a feature: It makes it really clear that these are the drm > >>>>>> fourcc codes, as allocated in drm_fourcc.h, plus their modifiers, and all > >>>>>> that. That allocation authority is also already baked into various khr/ext > >>>>>> standards, too. > > > > There's one thing that V4L2 has and DRM hasn't for 4CCs: good > > documentation. Look at > > https://linuxtv.org/downloads/v4l-dvb-apis-new/uapi/v4l/pixfmt-packed-rgb.html > > or > > https://linuxtv.org/downloads/v4l-dvb-apis-new/uapi/v4l/yuv-formats.html > > for instance. It's painful to write, painful to read, but defines the > > 4CCs very clearly without ambiguity. I wouldn't be surprised if > > different drivers used the same DRM 4CC for different formats due to the > > lack of detailed documentation. Moving to a shared library for 4CCs > > should also address the documentation side, and any new format added to > > the kernel, whether from the V4L2 side or the DRM side, would be > > required to provide detailed documentation. That would be a great > > improvement for DRM 4CC handling. > > > >>>>> The way I see it, there's a fundamental difference between the UAPI > >>>>> and the kernel. I don't suggest we change anything about the UAPI: the > >>>>> drm formats should keep their prefix, drm_fourcc.h can remain that > >>>>> authority, it's all fine. > >>>>> > >>>>> Internally however, the long term goal is to share the fourcc's > >>>>> between DRM and V4L2 for the same formats. It basically means that of > >>>>> course v4l2 should be using the DRM fourcc when a format exists in DRM > >>>>> and not v4l2, but also that DRM should use v4l2 fourcc when the format > >>>>> exists in v4l2 but not DRM, and that is far more likely, given the > >>>>> already extensive v4l2 formats support. > >>>> > >>>> Uh no. We did look at v4l fourcc extensively when deciding upon a drm > >>>> format identifier space. > >>> > >>> No to what exactly? > >>> > >>>> And a lot of people pushed for the "fourcc is a standard", when > >>>> really it's totally not. > >>> > >>> Even if it's not a standard, having consistency would be a good thing. > >>> > >>> And you said yourself that DRM fourcc is now pretty much an authority > >>> for the fourcc, so it definitely looks like a standard to me. > >> > >> drm fourcc is the authority for drm fourcc codes. Not for any of the > >> others (and there's lots of them). Now it's used in a bunch of other > >> places (khr standards, dri protocols in wayland/X11), but they're > >> still only drm fourcc. There is no overall fourcc standard. > >> > >>>> v4l tends to conflate pixel format with stuff that we tend to encode > >>>> in modifiers a lot more. > >>> > >>> Boris is working on adding the modifiers concept to v4l2, so we're > >>> converging here, and we can totally have a layer in v4l2 to convert > >>> between old v4l2 "format+modifiers" formats, and DRM style formats. > >>> > >>>> There's a bunch of reasons we can't just use v4l, and they're as > >>>> valid as ever: > >>>> > >>>> - We overlap badly in some areas, so even if fourcc codes match, we > >>>> can't use them and need a duplicated DRM_FOURCC code. > >>> > >>> Do yo have an example of one of those areas? > >> > >> I think the rgb stuff was one of the big reasons to not reuse any > >> existing fourcc standard (whether v4l, or another one from e.g. video > >> container formats). We had initial patches to reuse the fourcc that > >> existed, but the end result was really inconsistent, so we ditch that > >> and rolled out a new set of entirely drm specific fourcc codes for > >> rgba. > > > > Could you give one or a couple of examples of V4L2 4CCs that are not > > OCD-compatible ? :-) > > > >>>> - v4l encodes some metadata into the fourcc that we encode elsewhere, > >>>> e.g. offset for planar yuv formats, or tiling mode > >>> > >>> As I was saying, this changes on the v4l2 side, and converging to > >>> what DRM is doing. > >>> > >>>> - drm fourcc code doesn't actually define the drm_format_info > >>>> uniquely, drivers can override that (that's an explicit design > >>>> intent of modifiers, to allow drivers to add another plane for > >>>> e.g. compression information). You'd need to pull that driver > >>>> knowledge into your format library. > > > > That's a mistake in my opinion. We tried that in V4L2 to store metadata > > in a separate plane, and had to go another route eventually as it > > created a very bad mess. > > Just quick clarification in the middle here: This is how the hw works. The hardware takes parameters from a buffer, but it doesn't mandate how that buffer is exposed to userspace :-) Using an extra plane is one option, but other APIs are possible. > It's not metadata that sw ever touches (in general, testcases to make sure > we display these correctly excepted). > > There has been some talking to add maybe a bit more mixed metadata, for > fast-clear colors (which isn't used by any display engine afaik yet). That What are fast-clear colors ? > would generally be written by the cpu (in the gl stack), but again read by > the hw (loaded as indirect state packet most likely, or something like > that). So again hw specific layout, because the hw needs to read it. > > Pure metadata only of interest for the cpu/sw stack has been shot down > completely on the drm side too. > > > There's a tendency in both subsystems to look at the other as a bit of a > > retarded relative, and that's a shame, we have lots to learn from each > > other's mistakes. That wouldn't be too difficult if people started > > talking to each other. > > > > A semi-related comment: DRM also carries other mistakes of its own, I'm > > thinking about DRM_FORMAT_BIG_ENDIAN in particular > > Yeah that one is hilarios, but in practice big endian is dead, except for > a very few server chips, and there I think Gerd's work mostly fixed up > that mess. > > >>> I'm not sure how my patches are changing anything here. This is > >>> litterally the same code, with the functions renamed. > >>> > >>> If drivers want to override that, then yeah, fine, we can let them do > >>> that. Just like any helper this just provides a default that covers > >>> most of the cases. > >>> > >>>> Iow there's no way we can easily adopt v4l fourcc, except if we do > >>>> something like a new addfb flag. > >>> > >>> For the formats that would be described as a modifier, sure. For all > >>> the others (that are not yet supported by DRM), then I don't really > >>> see why not. > >> > >> See above, we tried that initially, didn't pass the ocd tests when > >> reviewing. Maybe the situation is better outside of rbgx/a formats, > >> and I think we do at least try to use the same fourcc codes there when > >> there already is one. But then the details occasionally look > >> different. > >> > >>>>> And given how the current state is a mess in this regard, I'm not too > >>>>> optimistic about keeping the formats in their relevant frameworks. > >>>>> > >>>>> Having a shared library, governed by both, will make this far easier, > >>>>> since it will be easy to discover the formats that are already > >>>>> supported by the other subsystem. > >>>> > >>>> I think a compat library that (tries to, best effort) convert between > >>>> v4l and drm fourcc would make sense. Somewhere in drivers/video, next > >>>> to the conversion functions for videomode <-> drm_display_mode > >>>> perhaps. That should be useful for drivers. > >>> > >>> That's not really what this series is about though. That series is > >>> about sharing the (image|pixels) formats database across everyone so > >>> that everyone can benefit from it. > >> > >> Yeah I know. I still think leaving the drm fourcc with the drm prefix > >> would be good, since there's really no standard here. > >> > >>>> Unifying the formats themselves, and all the associated metadata is > >>>> imo a no-go, and was a pretty conscious decision when we implemented > >>>> drm_fourcc a few years back. > >>>> > >>>>> If we want to keep the current library in DRM, we have two options > >>>>> then: > >>>>> > >>>>> - Support all the v4l2 formats in the DRM library, which is > >>>>> essentially what I'm doing in the last patches. However, that > >>>>> would require to have the v4l2 developpers also reviewing stuff > >>>>> there. And given how busy they are, I cannot really see how that > >>>>> would work. > >>>> > >>>> Well, if we end up with a common library then yes we need cross > >>>> review. There's no way around that. Doesn't matter where exactly that > >>>> library is in the filesystem tree, and adding a special MAINTAINERS > >>>> entry for anything related to fourcc (both drm and v4l) to make sure > >>>> they get cross-posted is easy. No file renaming needed. > >>> > >>> That would create some governing issues as well. For example, if you > >>> ever have a patch from one fourcc addition (that might or might not be > >>> covered by v4l2), will you wait for any v4l2 developper to review it? > >> > >> None of this is fixed by code renaming or locations. Either way we > >> need to figure that out. > >> > >> And yes part of the reasons for not moving this out of drm is that I'm > >> not a fan of boutique trees for small stuff. If sharing means we need > >> to split the drm_fourcc code and library out of drm trees, I'm not > >> sure that's a good idea. We're already super liberal with merging > >> anything through driver trees with acks, and integrating them quickly > >> into drm-next. This would still be workable if v4l sends regular pull > >> requests to drm-next (every 1-2 weeks, like the other big gpu trees > >> do). If we can only sync up once per merge window with a shared > >> boutique tree for formats only, life is going to be painful. > > > > That should be solved by the proposal above, maintaining the shared > > library in the DRM tree. We would need to be careful there, and ideally > > maintain that in a separate branch that could be merged in both DRM and > > V4L2 without having to merge most of the other subsystem's pending > > changes at the same time, but I think it's doable without any big issue. > > > >>> If it's shared code, then it should be shared, and every client > >>> framework put on an equal footing. > >>> > >>>>> - Develop the same library from within v4l2. That is really a poor > >>>>> solution, since the information would be greatly duplicated > >>>>> between the two, and in terms of maintainance, code, and binary > >>>>> size that would be duplicated too. > >>>> > >>>> It's essentially what we decided to do for drm years back. > >>> > >>> And it was probably the right solution back then, but I'm really not > >>> convinced it's still the right thing to do today. > >>> > >>>>> Having it shared allows to easily share, and discover formats from the > >>>>> other subsystem, and to have a single, unique place where this is > >>>>> centralized. > >>>> > >>>> What I think could work as middle ground: > >>>> - Put drm_format stuff into a separate .ko > >>>> - Add a MAINTAINERS entry to make sure all things fourcc are cross > >>>> posted to both drm and v4l lists. Easy on the drm side, since it's all > >>>> separate files. Not sure it's so convenient for v4l uapi. > >>>> - Add a conversion library that tries to best-effort map between drm > >>>> and v4l formats. On the drm side that most likely means you need > >>>> offsets for planes, and modifiers too (since those are implied in some > >>>> v4l fourcc). Emphasis on "best effort" i.e. only support as much as > >>>> the drivers that use this library need. > >>>> - Add drm_fourcc as-needed by these drivers that want to use a unified > >>>> format space. > >>>> > >>>> Forcing this unification on everyone otoh is imo way too much. > >>> > >>> v4l2 is starting to converge with DRM, and we're using the DRM API > >>> pretty much untouched for that library, so I'm not really sure how > >>> anyone is hurt by that unification. > >> > >> It might make sense to regularly pull v4l updates into drm-next then > >> anyway. That would also remove the need to have the format library > >> somewhere else. > > > > Are you saying it should the live in V4L2 ? ;-) > > Maybe a few clarifications on how the drm shared core thing usually works, > and why I'm a sticker here. Bottom reply since I'm not sure where to put > it: > > - Refactorings usually go in through drm-misc (at least since a few > years). > > - Small patches go in through the relevant driver tree (which is often > drm-misc, but not always), with an Ack from drm maintainers. > > - No topic branches, everyone just pushes patches where it's most > convenient. > > We get away with this mess because everyone sends regular pull requests to > drm, where the entire history is baked in and others can backmerge/fast > forward/rebase. Worst case you wait one month (around the merge window, > when drm-next is closed for features), but usually it's just 1-2 weeks. > Plus we tend to have fairly big trees, with good chances that the next > patch series lands in the same tree again and no work at all is needed. > > If we start regularly sharing lots of code with v4l (which seems to be the > long term goal here), then I think we need something equally convenient > for all that. > > We're not going to be able to teach some complicated topic branch scheme > to the 50+ submaintainers/committers we have in drm - a lot much more > basic stuff already takes lots of work to get it to stick. If the proposal > is "to be careful" and "maintain it in a separate branch", I'm not in > favour because I think that just wouldn't work. Why not ? It can be a fast-moving branch that gets merged in drm-misc as often as you want (even at every commit if that's desired). We're dealing with a limited amount of code here, and there's no more reason that V4L2 should pull in drm-misc to get 4CC updates than DRM should pull V4L2 for the same. I have no objection against a 4CC branch being officially maintained under the DRM umbrella, but I think the code should live elsewhere than drivers/gpu/drm/, have a neutral prefix, and not require pulling an entire subsystem in.
Hi Daniel, On Tue, Apr 23, 2019 at 09:59:37AM +0100, Daniel Stone wrote: > On Tue, 23 Apr 2019 at 08:26, Daniel Vetter <daniel@ffwll.ch> wrote: > > On Sun, Apr 21, 2019 at 01:59:04AM +0300, Laurent Pinchart wrote: > >>>>> - drm fourcc code doesn't actually define the drm_format_info > >>>>> uniquely, drivers can override that (that's an explicit design > >>>>> intent of modifiers, to allow drivers to add another plane for > >>>>> e.g. compression information). You'd need to pull that driver > >>>>> knowledge into your format library. > >> > >> That's a mistake in my opinion. We tried that in V4L2 to store metadata > >> in a separate plane, and had to go another route eventually as it > >> created a very bad mess. > > > > Just quick clarification in the middle here: This is how the hw works. > > It's not metadata that sw ever touches (in general, testcases to make sure > > we display these correctly excepted). > > > > There has been some talking to add maybe a bit more mixed metadata, for > > fast-clear colors (which isn't used by any display engine afaik yet). That > > would generally be written by the cpu (in the gl stack), but again read by > > the hw (loaded as indirect state packet most likely, or something like > > that). So again hw specific layout, because the hw needs to read it. > > > > Pure metadata only of interest for the cpu/sw stack has been shot down > > completely on the drm side too. > > Totally. Let's take DRM_FORMAT_XRGB8888 + I915_FORMAT_MOD_Y_TILED as > an example. Here, there is one colour plane which is laid out in a > documented tiled format, containing normal XRGB8888 pixels once you do > the maths to get the correct pixel location. So that's fine. > > I915_FORMAT_MOD_Y_TILED_CCS has a base colour plane as above, but adds > an auxiliary plane which has a few bits describing the state of every > (differently-sized) tile. Before reading the tile from the colour > plane, you look at the corresponding location in the auxiliary plane: > if you read 0x55 from the auxiliary plane, then the entire cacheline > is the value of the first pixel, i.e. a solid fill. Hardware takes > advantage of this to only write out the first pixel: if you try to > read the colour plane as Y_TILED then for solid-filled regions, only > the first pixel of every tile will show correctly, and the rest will > be garbage. > > The auxiliary plane has its own layout and placement requirements, so > we need to carry around an offset and a stride for the auxiliary data. > We already have this for multiple planes; stuffing it into the base > plane would require us to reinvent the same for auxiliary data within > a single plane. Looks like we have different kinds of metadata to consider. On the V4L2 side metadata usually refers to the context in which a frame was captured, it doesn't tell how to interpret the contents of the pixels. In the case you just described, the metadata is part of the frame contents. I agree that this is a proper use case for storing such metadata in a plane. What I wouldn't like to see being stored in a plane is for instance gamma tables or similar data that configures the processing pipeline in the display engine (I know we have an API for gamma tables, this is just an example). > I understand at least one of the Tegra colour-compression layouts (for > Tegra 1xx?) is similar to this. > > It would be good to understand what you had in mind when you said that > using multiple planes created a mess. I haven't touched media > encode/decode units at a low level for quite a while (hooray for > gst-v4l2!), but I remember that they often used padding areas around > the buffer for scratch space - maybe motion vectors or similar? That > case is quite different to something like CCS, since the data is only > meaningful to the media engine and must be ignored (but preserved) by > everyone else. Using multiple planes in that case isn't appropriate, > since it's very specific to how that hardware unit deals with that > buffer, instead of something that every consumer needs to understand > in order to use it. With metadata unrelated to the pixel content, using a separate plane in the same buffer resulted in an explosion of the number of combinations that we would need to support, and ultimately led to a very ill-defined API. We decided to convey metadata related to the frame capture context (e.g. what exposure time was used for the frame) and processing pipeline configuration data in different buffers than the frame itself.
Hi Laurent, On Tue, 23 Apr 2019 at 16:54, Laurent Pinchart <laurent.pinchart@ideasonboard.com> wrote: > On Tue, Apr 23, 2019 at 09:59:37AM +0100, Daniel Stone wrote: > > On Tue, 23 Apr 2019 at 08:26, Daniel Vetter <daniel@ffwll.ch> wrote: > > Totally. Let's take DRM_FORMAT_XRGB8888 + I915_FORMAT_MOD_Y_TILED as > > an example. [... details ...] > > Looks like we have different kinds of metadata to consider. On the V4L2 > side metadata usually refers to the context in which a frame was > captured, it doesn't tell how to interpret the contents of the pixels. > In the case you just described, the metadata is part of the frame > contents. I agree that this is a proper use case for storing such > metadata in a plane. What I wouldn't like to see being stored in a plane > is for instance gamma tables or similar data that configures the > processing pipeline in the display engine (I know we have an API for > gamma tables, this is just an example). > > > It would be good to understand what you had in mind when you said that > > using multiple planes created a mess. I haven't touched media > > encode/decode units at a low level for quite a while (hooray for > > gst-v4l2!), but I remember that they often used padding areas around > > the buffer for scratch space - maybe motion vectors or similar? That > > case is quite different to something like CCS, since the data is only > > meaningful to the media engine and must be ignored (but preserved) by > > everyone else. Using multiple planes in that case isn't appropriate, > > since it's very specific to how that hardware unit deals with that > > buffer, instead of something that every consumer needs to understand > > in order to use it. > > With metadata unrelated to the pixel content, using a separate plane in > the same buffer resulted in an explosion of the number of combinations > that we would need to support, and ultimately led to a very ill-defined > API. We decided to convey metadata related to the frame capture context > (e.g. what exposure time was used for the frame) and processing pipeline > configuration data in different buffers than the frame itself. Yeah, that makes sense. It's not really that different from what happens with GPUs either: the final colour buffer the display controller gets from a game is the product of a _lot_ of other work which is invisible to the display controller, including things like depth and stencil buffers, command buffers, etc etc. Those are closely related to the frame production, but totally irrelevant for exchanging the colour buffer with other subsystems. I think we should look at the metadata buffers you're describing in the same way. Perhaps each V4L2 buffer could have driver-private auxiliary buffer storage, or perhaps it's something you need to separately expose to userspace as auxiliary data which must be preserved but ignored. But modifiers are really only about what you need when exchanging image colour buffers between subsystems, not anything else. You're pretty close with gamma tables as well; for HDR and other kinds of complex colour management, we need to carry a fair bit of auxiliary information in order to display the image correctly. These have quite different uses though: normally the colour buffer is produced by the hardware and the primaries/whitepoints/etc are produced by software, with the colour-management details remaining static across the life of a swapchain even as you flip between multiple buffers. Given that, it doesn't really make sense to try to stuff them into the same storage. Cheers, Daniel
Hi, Le mardi 23 avril 2019 à 17:02 +0100, Daniel Stone a écrit : > Hi Laurent, > > On Tue, 23 Apr 2019 at 16:54, Laurent Pinchart > <laurent.pinchart@ideasonboard.com> wrote: > > On Tue, Apr 23, 2019 at 09:59:37AM +0100, Daniel Stone wrote: > > > On Tue, 23 Apr 2019 at 08:26, Daniel Vetter <daniel@ffwll.ch> wrote: > > > Totally. Let's take DRM_FORMAT_XRGB8888 + I915_FORMAT_MOD_Y_TILED as > > > an example. [... details ...] > > > > Looks like we have different kinds of metadata to consider. On the V4L2 > > side metadata usually refers to the context in which a frame was > > captured, it doesn't tell how to interpret the contents of the pixels. > > In the case you just described, the metadata is part of the frame > > contents. I agree that this is a proper use case for storing such > > metadata in a plane. What I wouldn't like to see being stored in a plane > > is for instance gamma tables or similar data that configures the > > processing pipeline in the display engine (I know we have an API for > > gamma tables, this is just an example). > > > > > It would be good to understand what you had in mind when you said that > > > using multiple planes created a mess. I haven't touched media > > > encode/decode units at a low level for quite a while (hooray for > > > gst-v4l2!), but I remember that they often used padding areas around > > > the buffer for scratch space - maybe motion vectors or similar? That > > > case is quite different to something like CCS, since the data is only > > > meaningful to the media engine and must be ignored (but preserved) by > > > everyone else. Using multiple planes in that case isn't appropriate, > > > since it's very specific to how that hardware unit deals with that > > > buffer, instead of something that every consumer needs to understand > > > in order to use it. > > > > With metadata unrelated to the pixel content, using a separate plane in > > the same buffer resulted in an explosion of the number of combinations > > that we would need to support, and ultimately led to a very ill-defined > > API. We decided to convey metadata related to the frame capture context > > (e.g. what exposure time was used for the frame) and processing pipeline > > configuration data in different buffers than the frame itself. > > Yeah, that makes sense. It's not really that different from what > happens with GPUs either: the final colour buffer the display > controller gets from a game is the product of a _lot_ of other work > which is invisible to the display controller, including things like > depth and stencil buffers, command buffers, etc etc. Those are closely > related to the frame production, but totally irrelevant for exchanging > the colour buffer with other subsystems. > > I think we should look at the metadata buffers you're describing in > the same way. Perhaps each V4L2 buffer could have driver-private > auxiliary buffer storage, or perhaps it's something you need to > separately expose to userspace as auxiliary data which must be > preserved but ignored. But modifiers are really only about what you > need when exchanging image colour buffers between subsystems, not > anything else. > > You're pretty close with gamma tables as well; for HDR and other kinds > of complex colour management, we need to carry a fair bit of auxiliary > information in order to display the image correctly. These have quite > different uses though: normally the colour buffer is produced by the > hardware and the primaries/whitepoints/etc are produced by software, > with the colour-management details remaining static across the life of > a swapchain even as you flip between multiple buffers. Given that, it > doesn't really make sense to try to stuff them into the same storage. I agree that we need to keep things minimal and clearly distinguish between what constitutes the description of the pixel buffer in terms of how it's laid out in memory and information about how the data should be interpreted. And there's indeed a fair share of things to consider there. Adding to the list above, I'm also thinking of the YUV colorspace information which must be passed along when displaying a buffer. But that's essentially not required to have a common description of buffers unified accross subsystems. Thinking about that, it would be interesting to have a common base structure for buffers used in v4l2 and drm. Ideally, that could be shared when doing dma-buf to avoid having userspace describe buffers and memory each time a buffer is imported into another subsystem. This could also help us solve the ambiguity related to the YUV M-suffixed formats. Another idea could be having common ioctls to get a unified buffer description from userspace and e.g. mmap on a per-plane basis (with virtual mappings like DRM does). What do you think? Cheers, Paul
Hi, Le mardi 23 avril 2019 à 18:45 +0300, Laurent Pinchart a écrit : > Hi Daniel, > > On Tue, Apr 23, 2019 at 09:25:54AM +0200, Daniel Vetter wrote: > > On Sun, Apr 21, 2019 at 01:59:04AM +0300, Laurent Pinchart wrote: > > > On Thu, Apr 18, 2019 at 12:07:44PM +0200, Daniel Vetter wrote: > > > > On Thu, Apr 18, 2019 at 11:02 AM Maxime Ripard wrote: > > > > > On Thu, Apr 18, 2019 at 09:52:10AM +0200, Daniel Vetter wrote: > > > > > > On Thu, Apr 18, 2019 at 8:22 AM Maxime Ripard wrote: > > > > > > > On Wed, Apr 17, 2019 at 05:41:21PM +0200, Daniel Vetter wrote: > > > > > > > > On Wed, Apr 17, 2019 at 09:54:26AM +0200, Maxime Ripard wrote: > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > DRM comes with an extensive format support to retrieve the various > > > > > > > > > parameters associated with a given format (such as the subsampling, or the > > > > > > > > > bits per pixel), as well as some helpers and utilities to ease the driver > > > > > > > > > development. > > > > > > > > > > > > > > > > > > v4l2, on the other side, doesn't provide such facilities, leaving each > > > > > > > > > driver reimplement a subset of the formats parameters for the one supported > > > > > > > > > by that particular driver. This leads to a lot of duplication and > > > > > > > > > boilerplate code in the v4l2 drivers. > > > > > > > > > > > > > > > > > > This series tries to address this by moving the DRM format API into lib and > > > > > > > > > turning it into a more generic API. In order to do this, we've needed to do > > > > > > > > > some preliminary changes on the DRM drivers, then moved the API and finally > > > > > > > > > converted a v4l2 driver to give an example of how such library could be > > > > > > > > > used. > > > > > > > > > > > > > > > > > > Let me know what you think, > > > > > > > > > Maxime > > > > > > > > > > > > > > > > > > Changes from RFC: > > > > > > > > > - Rebased on next > > > > > > > > > - Fixed the various formats mapping > > > > > > > > > - Added tags > > > > > > > > > - Did most of the formats functions as inline functions > > > > > > > > > - Used a consistent prefix for all the utilities functions > > > > > > > > > - Fixed the compilation breakages, and did a run of allmodconfig for arm, > > > > > > > > > arm64 and x86_64 > > > > > > > > > - Fixed out of array bounds warnings in the image_format_info_block_* > > > > > > > > > functions > > > > > > > > > - Added License and copyright headers on missing files > > > > > > > > > > > > > > > > > > Maxime Ripard (20): > > > > > > > > > drm: Remove users of drm_format_num_planes > > > > > > > > > drm: Remove users of drm_format_(horz|vert)_chroma_subsampling > > > > > > > > > drm/fourcc: Pass the format_info pointer to drm_format_plane_cpp > > > > > > > > > drm/fourcc: Pass the format_info pointer to drm_format_plane_width/height > > > > > > > > > drm: Replace instances of drm_format_info by drm_get_format_info > > > > > > > > > lib: Add video format information library > > > > > > > > > drm/fb: Move from drm_format_info to image_format_info > > > > > > > > > drm/malidp: Convert to generic image format library > > > > > > > > > drm/client: Convert to generic image format library > > > > > > > > > drm/exynos: Convert to generic image format library > > > > > > > > > drm/i915: Convert to generic image format library > > > > > > > > > drm/ipuv3: Convert to generic image format library > > > > > > > > > drm/msm: Convert to generic image format library > > > > > > > > > drm/omap: Convert to generic image format library > > > > > > > > > drm/rockchip: Convert to generic image format library > > > > > > > > > drm/tegra: Convert to generic image format library > > > > > > > > > drm/fourcc: Remove old DRM format API > > > > > > > > > lib: image-formats: Add v4l2 formats support > > > > > > > > > lib: image-formats: Add more functions > > > > > > > > > media: sun6i: Convert to the image format API > > > > > > > > > > > > > > > > In the interest of making myself unpopular: Why move this out of drm? > > > > > > > > > > > > > > > > We do have a bunch of other such shared helpers already (mostly in > > > > > > > > drivers/video) for dt videomode and hdmi infoframes, and I'm not super > > > > > > > > sure that's going better than keeping it maintained in drm. > > > > > > That's a bit of a different situation as both DRM and FBDEV address the > > > same features (display output), and FBDEV is deprecared and replaced by > > > DRM. > > > > > > I'm not against maintaining a 4CC library in DRM (adding a third-party > > > maintainer would likely create more problems than it would solve), but > > > that doesn't mean the library has to live in drivers/gpu/, nor that it > > > needs to have the drm_ prefix. I would actually advocate to make it live > > > in a neutral directory, with a neutral prefix (kudos to anyone who can > > > propose a nice naming convention that would establish a new shared > > > ground for image/video-related Linux APIs), and maintained through the > > > DRM tree (possibly with extra entries in MAINTAINERS to ensure it > > > reaches all the related folks). > > > > > > > > > > > Plus the uapi is already that you include drm_fourcc.h to get at these, > > > > > > > > dropping the drm prefix isn't going to help I think. > > > > > > > > > > > > > > > > Of course we'd need to make it a separate drm_formats.ko (so that v4l can > > > > > > > > use it without dragging in all of drm), and we need to add some fields for > > > > > > > > converting to v4l fourcc codes (which are different). But that should be > > > > > > > > all possible. And I don't think the drm_ prefix in v4l code is a problem, > > > > > > > > it's actually a feature: It makes it really clear that these are the drm > > > > > > > > fourcc codes, as allocated in drm_fourcc.h, plus their modifiers, and all > > > > > > > > that. That allocation authority is also already baked into various khr/ext > > > > > > > > standards, too. > > > > > > There's one thing that V4L2 has and DRM hasn't for 4CCs: good > > > documentation. Look at > > > https://linuxtv.org/downloads/v4l-dvb-apis-new/uapi/v4l/pixfmt-packed-rgb.html > > > or > > > https://linuxtv.org/downloads/v4l-dvb-apis-new/uapi/v4l/yuv-formats.html > > > for instance. It's painful to write, painful to read, but defines the > > > 4CCs very clearly without ambiguity. I wouldn't be surprised if > > > different drivers used the same DRM 4CC for different formats due to the > > > lack of detailed documentation. Moving to a shared library for 4CCs > > > should also address the documentation side, and any new format added to > > > the kernel, whether from the V4L2 side or the DRM side, would be > > > required to provide detailed documentation. That would be a great > > > improvement for DRM 4CC handling. > > > > > > > > > > The way I see it, there's a fundamental difference between the UAPI > > > > > > > and the kernel. I don't suggest we change anything about the UAPI: the > > > > > > > drm formats should keep their prefix, drm_fourcc.h can remain that > > > > > > > authority, it's all fine. > > > > > > > > > > > > > > Internally however, the long term goal is to share the fourcc's > > > > > > > between DRM and V4L2 for the same formats. It basically means that of > > > > > > > course v4l2 should be using the DRM fourcc when a format exists in DRM > > > > > > > and not v4l2, but also that DRM should use v4l2 fourcc when the format > > > > > > > exists in v4l2 but not DRM, and that is far more likely, given the > > > > > > > already extensive v4l2 formats support. > > > > > > > > > > > > Uh no. We did look at v4l fourcc extensively when deciding upon a drm > > > > > > format identifier space. > > > > > > > > > > No to what exactly? > > > > > > > > > > > And a lot of people pushed for the "fourcc is a standard", when > > > > > > really it's totally not. > > > > > > > > > > Even if it's not a standard, having consistency would be a good thing. > > > > > > > > > > And you said yourself that DRM fourcc is now pretty much an authority > > > > > for the fourcc, so it definitely looks like a standard to me. > > > > > > > > drm fourcc is the authority for drm fourcc codes. Not for any of the > > > > others (and there's lots of them). Now it's used in a bunch of other > > > > places (khr standards, dri protocols in wayland/X11), but they're > > > > still only drm fourcc. There is no overall fourcc standard. > > > > > > > > > > v4l tends to conflate pixel format with stuff that we tend to encode > > > > > > in modifiers a lot more. > > > > > > > > > > Boris is working on adding the modifiers concept to v4l2, so we're > > > > > converging here, and we can totally have a layer in v4l2 to convert > > > > > between old v4l2 "format+modifiers" formats, and DRM style formats. > > > > > > > > > > > There's a bunch of reasons we can't just use v4l, and they're as > > > > > > valid as ever: > > > > > > > > > > > > - We overlap badly in some areas, so even if fourcc codes match, we > > > > > > can't use them and need a duplicated DRM_FOURCC code. > > > > > > > > > > Do yo have an example of one of those areas? > > > > > > > > I think the rgb stuff was one of the big reasons to not reuse any > > > > existing fourcc standard (whether v4l, or another one from e.g. video > > > > container formats). We had initial patches to reuse the fourcc that > > > > existed, but the end result was really inconsistent, so we ditch that > > > > and rolled out a new set of entirely drm specific fourcc codes for > > > > rgba. > > > > > > Could you give one or a couple of examples of V4L2 4CCs that are not > > > OCD-compatible ? :-) > > > > > > > > > - v4l encodes some metadata into the fourcc that we encode elsewhere, > > > > > > e.g. offset for planar yuv formats, or tiling mode > > > > > > > > > > As I was saying, this changes on the v4l2 side, and converging to > > > > > what DRM is doing. > > > > > > > > > > > - drm fourcc code doesn't actually define the drm_format_info > > > > > > uniquely, drivers can override that (that's an explicit design > > > > > > intent of modifiers, to allow drivers to add another plane for > > > > > > e.g. compression information). You'd need to pull that driver > > > > > > knowledge into your format library. > > > > > > That's a mistake in my opinion. We tried that in V4L2 to store metadata > > > in a separate plane, and had to go another route eventually as it > > > created a very bad mess. > > > > Just quick clarification in the middle here: This is how the hw works. > > The hardware takes parameters from a buffer, but it doesn't mandate how > that buffer is exposed to userspace :-) Using an extra plane is one > option, but other APIs are possible. > > > It's not metadata that sw ever touches (in general, testcases to make sure > > we display these correctly excepted). > > > > There has been some talking to add maybe a bit more mixed metadata, for > > fast-clear colors (which isn't used by any display engine afaik yet). That > > What are fast-clear colors ? > > > would generally be written by the cpu (in the gl stack), but again read by > > the hw (loaded as indirect state packet most likely, or something like > > that). So again hw specific layout, because the hw needs to read it. > > > > Pure metadata only of interest for the cpu/sw stack has been shot down > > completely on the drm side too. > > > > > There's a tendency in both subsystems to look at the other as a bit of a > > > retarded relative, and that's a shame, we have lots to learn from each > > > other's mistakes. That wouldn't be too difficult if people started > > > talking to each other. > > > > > > A semi-related comment: DRM also carries other mistakes of its own, I'm > > > thinking about DRM_FORMAT_BIG_ENDIAN in particular > > > > Yeah that one is hilarios, but in practice big endian is dead, except for > > a very few server chips, and there I think Gerd's work mostly fixed up > > that mess. > > > > > > > I'm not sure how my patches are changing anything here. This is > > > > > litterally the same code, with the functions renamed. > > > > > > > > > > If drivers want to override that, then yeah, fine, we can let them do > > > > > that. Just like any helper this just provides a default that covers > > > > > most of the cases. > > > > > > > > > > > Iow there's no way we can easily adopt v4l fourcc, except if we do > > > > > > something like a new addfb flag. > > > > > > > > > > For the formats that would be described as a modifier, sure. For all > > > > > the others (that are not yet supported by DRM), then I don't really > > > > > see why not. > > > > > > > > See above, we tried that initially, didn't pass the ocd tests when > > > > reviewing. Maybe the situation is better outside of rbgx/a formats, > > > > and I think we do at least try to use the same fourcc codes there when > > > > there already is one. But then the details occasionally look > > > > different. > > > > > > > > > > > And given how the current state is a mess in this regard, I'm not too > > > > > > > optimistic about keeping the formats in their relevant frameworks. > > > > > > > > > > > > > > Having a shared library, governed by both, will make this far easier, > > > > > > > since it will be easy to discover the formats that are already > > > > > > > supported by the other subsystem. > > > > > > > > > > > > I think a compat library that (tries to, best effort) convert between > > > > > > v4l and drm fourcc would make sense. Somewhere in drivers/video, next > > > > > > to the conversion functions for videomode <-> drm_display_mode > > > > > > perhaps. That should be useful for drivers. > > > > > > > > > > That's not really what this series is about though. That series is > > > > > about sharing the (image|pixels) formats database across everyone so > > > > > that everyone can benefit from it. > > > > > > > > Yeah I know. I still think leaving the drm fourcc with the drm prefix > > > > would be good, since there's really no standard here. > > > > > > > > > > Unifying the formats themselves, and all the associated metadata is > > > > > > imo a no-go, and was a pretty conscious decision when we implemented > > > > > > drm_fourcc a few years back. > > > > > > > > > > > > > If we want to keep the current library in DRM, we have two options > > > > > > > then: > > > > > > > > > > > > > > - Support all the v4l2 formats in the DRM library, which is > > > > > > > essentially what I'm doing in the last patches. However, that > > > > > > > would require to have the v4l2 developpers also reviewing stuff > > > > > > > there. And given how busy they are, I cannot really see how that > > > > > > > would work. > > > > > > > > > > > > Well, if we end up with a common library then yes we need cross > > > > > > review. There's no way around that. Doesn't matter where exactly that > > > > > > library is in the filesystem tree, and adding a special MAINTAINERS > > > > > > entry for anything related to fourcc (both drm and v4l) to make sure > > > > > > they get cross-posted is easy. No file renaming needed. > > > > > > > > > > That would create some governing issues as well. For example, if you > > > > > ever have a patch from one fourcc addition (that might or might not be > > > > > covered by v4l2), will you wait for any v4l2 developper to review it? > > > > > > > > None of this is fixed by code renaming or locations. Either way we > > > > need to figure that out. > > > > > > > > And yes part of the reasons for not moving this out of drm is that I'm > > > > not a fan of boutique trees for small stuff. If sharing means we need > > > > to split the drm_fourcc code and library out of drm trees, I'm not > > > > sure that's a good idea. We're already super liberal with merging > > > > anything through driver trees with acks, and integrating them quickly > > > > into drm-next. This would still be workable if v4l sends regular pull > > > > requests to drm-next (every 1-2 weeks, like the other big gpu trees > > > > do). If we can only sync up once per merge window with a shared > > > > boutique tree for formats only, life is going to be painful. > > > > > > That should be solved by the proposal above, maintaining the shared > > > library in the DRM tree. We would need to be careful there, and ideally > > > maintain that in a separate branch that could be merged in both DRM and > > > V4L2 without having to merge most of the other subsystem's pending > > > changes at the same time, but I think it's doable without any big issue. > > > > > > > > If it's shared code, then it should be shared, and every client > > > > > framework put on an equal footing. > > > > > > > > > > > > - Develop the same library from within v4l2. That is really a poor > > > > > > > solution, since the information would be greatly duplicated > > > > > > > between the two, and in terms of maintainance, code, and binary > > > > > > > size that would be duplicated too. > > > > > > > > > > > > It's essentially what we decided to do for drm years back. > > > > > > > > > > And it was probably the right solution back then, but I'm really not > > > > > convinced it's still the right thing to do today. > > > > > > > > > > > > Having it shared allows to easily share, and discover formats from the > > > > > > > other subsystem, and to have a single, unique place where this is > > > > > > > centralized. > > > > > > > > > > > > What I think could work as middle ground: > > > > > > - Put drm_format stuff into a separate .ko > > > > > > - Add a MAINTAINERS entry to make sure all things fourcc are cross > > > > > > posted to both drm and v4l lists. Easy on the drm side, since it's all > > > > > > separate files. Not sure it's so convenient for v4l uapi. > > > > > > - Add a conversion library that tries to best-effort map between drm > > > > > > and v4l formats. On the drm side that most likely means you need > > > > > > offsets for planes, and modifiers too (since those are implied in some > > > > > > v4l fourcc). Emphasis on "best effort" i.e. only support as much as > > > > > > the drivers that use this library need. > > > > > > - Add drm_fourcc as-needed by these drivers that want to use a unified > > > > > > format space. > > > > > > > > > > > > Forcing this unification on everyone otoh is imo way too much. > > > > > > > > > > v4l2 is starting to converge with DRM, and we're using the DRM API > > > > > pretty much untouched for that library, so I'm not really sure how > > > > > anyone is hurt by that unification. > > > > > > > > It might make sense to regularly pull v4l updates into drm-next then > > > > anyway. That would also remove the need to have the format library > > > > somewhere else. > > > > > > Are you saying it should the live in V4L2 ? ;-) > > > > Maybe a few clarifications on how the drm shared core thing usually works, > > and why I'm a sticker here. Bottom reply since I'm not sure where to put > > it: > > > > - Refactorings usually go in through drm-misc (at least since a few > > years). > > > > - Small patches go in through the relevant driver tree (which is often > > drm-misc, but not always), with an Ack from drm maintainers. > > > > - No topic branches, everyone just pushes patches where it's most > > convenient. > > > > We get away with this mess because everyone sends regular pull requests to > > drm, where the entire history is baked in and others can backmerge/fast > > forward/rebase. Worst case you wait one month (around the merge window, > > when drm-next is closed for features), but usually it's just 1-2 weeks. > > Plus we tend to have fairly big trees, with good chances that the next > > patch series lands in the same tree again and no work at all is needed. > > > > If we start regularly sharing lots of code with v4l (which seems to be the > > long term goal here), then I think we need something equally convenient > > for all that. > > > > We're not going to be able to teach some complicated topic branch scheme > > to the 50+ submaintainers/committers we have in drm - a lot much more > > basic stuff already takes lots of work to get it to stick. If the proposal > > is "to be careful" and "maintain it in a separate branch", I'm not in > > favour because I think that just wouldn't work. > > Why not ? It can be a fast-moving branch that gets merged in drm-misc > as often as you want (even at every commit if that's desired). We're > dealing with a limited amount of code here, and there's no more reason > that V4L2 should pull in drm-misc to get 4CC updates than DRM should > pull V4L2 for the same. I have no objection against a 4CC branch being > officially maintained under the DRM umbrella, but I think the code > should live elsewhere than drivers/gpu/drm/, have a neutral prefix, and > not require pulling an entire subsystem in. I think that having a subsystem-neutral tree is what Maxime had in mind initially. The way I see it, the idea would indeed be to keep it low- traffic and avoid having any serious core code in there. Basically just stick to format helpers and ideally later buffer description structures and helpers. I also like the idea of having a few shared ioctls to retrieve a buffer memory layout information from a dma-buf fd and mmap-ing on a per-plane basis, but that's rather unlikely to evolve much once it's there. Cheers, Paul
Hi, Le dimanche 21 avril 2019 à 01:40 +0300, Laurent Pinchart a écrit : > Hi Paul, > > On Thu, Apr 18, 2019 at 01:49:54PM +0200, Paul Kocialkowski wrote: > > On Thu, 2019-04-18 at 11:02 +0200, Maxime Ripard wrote: > > > On Thu, Apr 18, 2019 at 09:52:10AM +0200, Daniel Vetter wrote: > > > > And a lot of people pushed for the "fourcc is a standard", when > > > > really it's totally not. > > > > > > Even if it's not a standard, having consistency would be a good thing. > > > > > > And you said yourself that DRM fourcc is now pretty much an authority > > > for the fourcc, so it definitely looks like a standard to me. > > > > I think trying to make the V4L2 and DRM fourccs converge is a lost > > cause, as it has already significantly diverged. Even if we coordinate > > an effort to introduce new formats with the same fourcc on both sides, > > I don't see what good that would be since the formats we have now are > > still plagued by the inconsistency. > > > > I think we always need an explicit translation step from either v4l2 or > > drm to the internal representation and back, without ever assuming that > > formats might be compatible because they share the same fourcc. > > I don't agree. APIs evolve, and while we can't switch from one set of > 4CCs to another in existing APIs, we could in new APIs. Boris is working > on new ioctls to handle formats in V4L2, and while 4CC unification could > be impopular from a userspace developers point of view there, I don't > think we have ruled it out completely. The move to the request API is > also an area where a common set of 4CCs could be used, as it will depart > from the existing V4L2 ioctls. To summarize my opinion, we're not there > yet, but I wouldn't rule it out completely for the future. Well, I don't see how we could maintain backward compatibility with some DRM and V4L2 fourccs that are compatible and some that aren't. Since both descriptions have diverged already, one would need explicit checking of whether the format at hand is a compatible one or not before passing-it along as-is to the other subsystem or going through a format conversion step (in userspace, duplicating the information). So it feels like it kind of defeats the purpose. If we're going to use a unified 4CC representation in the future, I think we should do it by using the new formats that this proposal is introducing instead of subsystem-specific formats. At which point I believe we will need an internal conversion step between that format and what the subsystem uses internally. Or do it the other way round and use the unified format all around the subsystem, with a legacy layer for the previous subsystem-specific format. Cheers, Paul
Le mardi 23 avril 2019 à 17:09 +0200, Daniel Vetter a écrit : > On Tue, Apr 23, 2019 at 4:28 PM Nicolas Dufresne <nicolas@ndufresne.ca> wrote: > > Le mardi 23 avril 2019 à 14:33 +0200, Paul Kocialkowski a écrit : > > > Hi, > > > > > > On Tue, 2019-04-23 at 09:30 +0200, Daniel Vetter wrote: > > > > On Sun, Apr 21, 2019 at 01:40:45AM +0300, Laurent Pinchart wrote: > > > > > Hi Paul, > > > > > > > > > > On Thu, Apr 18, 2019 at 01:49:54PM +0200, Paul Kocialkowski wrote: > > > > > > On Thu, 2019-04-18 at 11:02 +0200, Maxime Ripard wrote: > > > > > > > On Thu, Apr 18, 2019 at 09:52:10AM +0200, Daniel Vetter wrote: > > > > > > > > And a lot of people pushed for the "fourcc is a standard", when > > > > > > > > really it's totally not. > > > > > > > > > > > > > > Even if it's not a standard, having consistency would be a good thing. > > > > > > > > > > > > > > And you said yourself that DRM fourcc is now pretty much an authority > > > > > > > for the fourcc, so it definitely looks like a standard to me. > > > > > > > > > > > > I think trying to make the V4L2 and DRM fourccs converge is a lost > > > > > > cause, as it has already significantly diverged. Even if we coordinate > > > > > > an effort to introduce new formats with the same fourcc on both sides, > > > > > > I don't see what good that would be since the formats we have now are > > > > > > still plagued by the inconsistency. > > > > > > > > > > > > I think we always need an explicit translation step from either v4l2 or > > > > > > drm to the internal representation and back, without ever assuming that > > > > > > formats might be compatible because they share the same fourcc. > > > > > > > > > > I don't agree. APIs evolve, and while we can't switch from one set of > > > > > 4CCs to another in existing APIs, we could in new APIs. Boris is working > > > > > on new ioctls to handle formats in V4L2, and while 4CC unification could > > > > > be impopular from a userspace developers point of view there, I don't > > > > > think we have ruled it out completely. The move to the request API is > > > > > also an area where a common set of 4CCs could be used, as it will depart > > > > > from the existing V4L2 ioctls. To summarize my opinion, we're not there > > > > > yet, but I wouldn't rule it out completely for the future. > > > > > > > > > > > It looks like so far, V4L2 pixel formats describe a DRM pixel format + > > > > > > modifier. > > > > > > > > > > DRM modifiers are mostly about tiling and compression, and we hardly > > > > > support these in V4L2. What are the modifiers you think are hardcoded in > > > > > 4CCs in V4L2 ? > > > > > > > > Hm maybe it was a drm one that didn't come from v4l or anywhere else > > > > really, but the NV12MT one is nv12 + some tiling. I think we managed to > > > > uapi-bend that one into shape in at least drm. > > > > > > The one I had in mind is V4L2_PIX_FMT_SUNXI_TILED_NV12 which translates > > > to DRM_FORMAT_NV12 + DRM_FORMAT_MOD_ALLWINNER_TILED. Seems to be a > > > pretty similar case to the Mediatek one indeed. > > > > > > In our cause, that's because the video decoding engine produces its > > > destination buffers in a specific tiled format, that the display engine > > > can take in directly. > > > > We also have the Samsung tiling (Z pattern) as mentioned here, but also > > linear 16x16 tile placement (also from Samsung ?) and I believe Amlogic > > CODEC patches is bringing another tiling (unavoidable on older Meson8, > > with 64bytes swaps). All these should be expressed as NV12 + mod in DRM > > space. > > > > What is very often not enabled, but affect the performance on mainline > > media drivers is the ARM frame buffer compression. I know that RK chips > > have support for this, and that you can't achieve the maximum > > throughput without that. This one is not documented anywhere, but I > > understand that there is multiple variants that HW vendor licence. > > Though, in general, each SoC are likely running a single variant, so a > > single mod would make sense. > > We have AFBC modifiers now in drm_fourcc.h, jointly developed by > display engineers from ARM and mali gpu reverse engineer people doing > the panfrost driver. So should be covered. > > > So all this to say that V4L2 equally needs supports for these. What I > > understood through DRM API is that a buffer allocated for let's say > > NV12 + mod, is compatible with linear NV12. That could be used to > > simplify some code, but at the same time, a common API that deals with > > the padding and alignment of each format + mod independently would do > > that same as long as this is not variable depending on which target HW > > uses that same format. > > Not sure why you mean with NV12 + mod is compatible with linear NV12. > In general fourcc + modifier != fourcc = linear modifier, size, number > of planes, alignment constraints and everything else can be changed by > a modifier (and there's examples for all of these, maybe not yet in > all cases for NV12, but I think NV12 + AFBC modifiers gives some > pretty interesting results). In generally you need to think of the > (drm fourcc, modifier) as the pair identified the pixel format, each > part individually is fairly meanigless. We have lots of modifiers > where the exact tiling mode/layout changes depending upon the fourcc > code. I only meant that the NV12 + mod have the same number of planes, and should be large enough to store a linear NV12 equivalent. Not that it would render correctly (even though I found it useful to be able to render them when I needed to reverse it). > > The only exception is legacy interfaces, i.e. when you create a > framebuffer with only the drm fourcc and not a modifier. In that case > you get driver specific behaviour, but modifier aware drivers tend to > change that into a specific (fourcc, modifier) pair again (at least > i915.ko, and it's what I recommend). > > Oh and we have some legacy modifiers that depend upon the target hw, > but it's very much not recommended (we did it in i915 to make things > easier on really old platforms, on some of them we don't even know the > exact tiling mode since it's not documented nor correctly > reverse-engineered). > > Another fun case are some of the recent non-byte-aligned formats, for > which you need to have a modifier to be able to use them with anything > - there's not really a real linear layout for them, they just serve as > an index/parameter into the modifier space. > > > I think DRM + mod reduce the amount of dedicated formats that needs to > > be added, and simplify the documentation of each formats. I was looking > > at the Amlogic Axi configurations on Amlogic S905x recently, and for > > each well known format, there is a bitmask that let you do arbitrary > > swapping of bits, so effectively if we start exposing all these with > > V4L2 style, the list would become very long and hard to maintained. > > See above, modifiers aren't really simple ... > -Daniel > > > > Cheers, > > > > > > Paul > > > > > > > -Daniel > > > > > > > > > > I think Boris (CCed) is working to change that by allowing to > > > > > > pass a DRM modifier through V4L2. With that, we'd be in a situation > > > > > > where some formats are described by the v4l2 pixfmt alone and some > > > > > > formats are also described a modifier (but I looked at it from a > > > > > > distance so might have misunderstod). That feels better since it avoids > > > > > > the combinatory explosion from describing each format + modifier > > > > > > individually. > > > > > > > > > > > > What do you think? > > > > > > > > > > > > > > v4l tends to conflate pixel format with stuff that we tend to encode > > > > > > > > in modifiers a lot more. > > > > > > > > > > > > > > Boris is working on adding the modifiers concept to v4l2, so we're > > > > > > > converging here, and we can totally have a layer in v4l2 to convert > > > > > > > between old v4l2 "format+modifiers" formats, and DRM style formats. > > > > > > > > > > > > > > > There's a bunch of reasons we can't just use v4l, and they're as > > > > > > > > valid as ever: > > > > > > > > > > > > > > > > - We overlap badly in some areas, so even if fourcc codes match, we > > > > > > > > can't use them and need a duplicated DRM_FOURCC code. > > > > > > > > > > > > > > Do yo have an example of one of those areas? > > > > > > > > > > > > > > > - v4l encodes some metadata into the fourcc that we encode elsewhere, > > > > > > > > e.g. offset for planar yuv formats, or tiling mode > > > > > > > > > > > > > > As I was saying, this changes on the v4l2 side, and converging to > > > > > > > what DRM is doing. > > > > > > > > > > > > > > > - drm fourcc code doesn't actually define the drm_format_info > > > > > > > > uniquely, drivers can override that (that's an explicit design > > > > > > > > intent of modifiers, to allow drivers to add another plane for > > > > > > > > e.g. compression information). You'd need to pull that driver > > > > > > > > knowledge into your format library. > > > > > > > > > > > > > > I'm not sure how my patches are changing anything here. This is > > > > > > > litterally the same code, with the functions renamed. > > > > > > > > > > > > > > If drivers want to override that, then yeah, fine, we can let them do > > > > > > > that. Just like any helper this just provides a default that covers > > > > > > > most of the cases. > > > > > > > > > > > > > > > Iow there's no way we can easily adopt v4l fourcc, except if we do > > > > > > > > something like a new addfb flag. > > > > > > > > > > > > > > For the formats that would be described as a modifier, sure. For all > > > > > > > the others (that are not yet supported by DRM), then I don't really > > > > > > > see why not. > > > > > > > > > > > > > > > > And given how the current state is a mess in this regard, I'm not too > > > > > > > > > optimistic about keeping the formats in their relevant frameworks. > > > > > > > > > > > > > > > > > > Having a shared library, governed by both, will make this far easier, > > > > > > > > > since it will be easy to discover the formats that are already > > > > > > > > > supported by the other subsystem. > > > > > > > > > > > > > > > > I think a compat library that (tries to, best effort) convert between > > > > > > > > v4l and drm fourcc would make sense. Somewhere in drivers/video, next > > > > > > > > to the conversion functions for videomode <-> drm_display_mode > > > > > > > > perhaps. That should be useful for drivers. > > > > > > > > > > > > > > That's not really what this series is about though. That series is > > > > > > > about sharing the (image|pixels) formats database across everyone so > > > > > > > that everyone can benefit from it. > > > > > > > > > > > > > > > Unifying the formats themselves, and all the associated metadata is > > > > > > > > imo a no-go, and was a pretty conscious decision when we implemented > > > > > > > > drm_fourcc a few years back. > > > > > > > > > > > > > > > > > If we want to keep the current library in DRM, we have two options > > > > > > > > > then: > > > > > > > > > > > > > > > > > > - Support all the v4l2 formats in the DRM library, which is > > > > > > > > > essentially what I'm doing in the last patches. However, that > > > > > > > > > would require to have the v4l2 developpers also reviewing stuff > > > > > > > > > there. And given how busy they are, I cannot really see how that > > > > > > > > > would work. > > > > > > > > > > > > > > > > Well, if we end up with a common library then yes we need cross > > > > > > > > review. There's no way around that. Doesn't matter where exactly that > > > > > > > > library is in the filesystem tree, and adding a special MAINTAINERS > > > > > > > > entry for anything related to fourcc (both drm and v4l) to make sure > > > > > > > > they get cross-posted is easy. No file renaming needed. > > > > > > > > > > > > > > That would create some governing issues as well. For example, if you > > > > > > > ever have a patch from one fourcc addition (that might or might not be > > > > > > > covered by v4l2), will you wait for any v4l2 developper to review it? > > > > > > > > > > > > > > If it's shared code, then it should be shared, and every client > > > > > > > framework put on an equal footing. > > > > > > > > > > > > > > > > - Develop the same library from within v4l2. That is really a poor > > > > > > > > > solution, since the information would be greatly duplicated > > > > > > > > > between the two, and in terms of maintainance, code, and binary > > > > > > > > > size that would be duplicated too. > > > > > > > > > > > > > > > > It's essentially what we decided to do for drm years back. > > > > > > > > > > > > > > And it was probably the right solution back then, but I'm really not > > > > > > > convinced it's still the right thing to do today. > > > > > > > > > > > > > > > > Having it shared allows to easily share, and discover formats from the > > > > > > > > > other subsystem, and to have a single, unique place where this is > > > > > > > > > centralized. > > > > > > > > > > > > > > > > What I think could work as middle ground: > > > > > > > > - Put drm_format stuff into a separate .ko > > > > > > > > - Add a MAINTAINERS entry to make sure all things fourcc are cross > > > > > > > > posted to both drm and v4l lists. Easy on the drm side, since it's all > > > > > > > > separate files. Not sure it's so convenient for v4l uapi. > > > > > > > > - Add a conversion library that tries to best-effort map between drm > > > > > > > > and v4l formats. On the drm side that most likely means you need > > > > > > > > offsets for planes, and modifiers too (since those are implied in some > > > > > > > > v4l fourcc). Emphasis on "best effort" i.e. only support as much as > > > > > > > > the drivers that use this library need. > > > > > > > > - Add drm_fourcc as-needed by these drivers that want to use a unified > > > > > > > > format space. > > > > > > > > > > > > > > > > Forcing this unification on everyone otoh is imo way too much. > > > > > > > > > > > > > > v4l2 is starting to converge with DRM, and we're using the DRM API > > > > > > > pretty much untouched for that library, so I'm not really sure how > > > > > > > anyone is hurt by that unification. > > > > > > > > > > -- > > > > > Regards, > > > > > > > > > > Laurent Pinchart > >
On Tue, Apr 23, 2019 at 7:17 PM Nicolas Dufresne <nicolas@ndufresne.ca> wrote: > > Le mardi 23 avril 2019 à 17:09 +0200, Daniel Vetter a écrit : > > On Tue, Apr 23, 2019 at 4:28 PM Nicolas Dufresne <nicolas@ndufresne.ca> wrote: > > > Le mardi 23 avril 2019 à 14:33 +0200, Paul Kocialkowski a écrit : > > > > Hi, > > > > > > > > On Tue, 2019-04-23 at 09:30 +0200, Daniel Vetter wrote: > > > > > On Sun, Apr 21, 2019 at 01:40:45AM +0300, Laurent Pinchart wrote: > > > > > > Hi Paul, > > > > > > > > > > > > On Thu, Apr 18, 2019 at 01:49:54PM +0200, Paul Kocialkowski wrote: > > > > > > > On Thu, 2019-04-18 at 11:02 +0200, Maxime Ripard wrote: > > > > > > > > On Thu, Apr 18, 2019 at 09:52:10AM +0200, Daniel Vetter wrote: > > > > > > > > > And a lot of people pushed for the "fourcc is a standard", when > > > > > > > > > really it's totally not. > > > > > > > > > > > > > > > > Even if it's not a standard, having consistency would be a good thing. > > > > > > > > > > > > > > > > And you said yourself that DRM fourcc is now pretty much an authority > > > > > > > > for the fourcc, so it definitely looks like a standard to me. > > > > > > > > > > > > > > I think trying to make the V4L2 and DRM fourccs converge is a lost > > > > > > > cause, as it has already significantly diverged. Even if we coordinate > > > > > > > an effort to introduce new formats with the same fourcc on both sides, > > > > > > > I don't see what good that would be since the formats we have now are > > > > > > > still plagued by the inconsistency. > > > > > > > > > > > > > > I think we always need an explicit translation step from either v4l2 or > > > > > > > drm to the internal representation and back, without ever assuming that > > > > > > > formats might be compatible because they share the same fourcc. > > > > > > > > > > > > I don't agree. APIs evolve, and while we can't switch from one set of > > > > > > 4CCs to another in existing APIs, we could in new APIs. Boris is working > > > > > > on new ioctls to handle formats in V4L2, and while 4CC unification could > > > > > > be impopular from a userspace developers point of view there, I don't > > > > > > think we have ruled it out completely. The move to the request API is > > > > > > also an area where a common set of 4CCs could be used, as it will depart > > > > > > from the existing V4L2 ioctls. To summarize my opinion, we're not there > > > > > > yet, but I wouldn't rule it out completely for the future. > > > > > > > > > > > > > It looks like so far, V4L2 pixel formats describe a DRM pixel format + > > > > > > > modifier. > > > > > > > > > > > > DRM modifiers are mostly about tiling and compression, and we hardly > > > > > > support these in V4L2. What are the modifiers you think are hardcoded in > > > > > > 4CCs in V4L2 ? > > > > > > > > > > Hm maybe it was a drm one that didn't come from v4l or anywhere else > > > > > really, but the NV12MT one is nv12 + some tiling. I think we managed to > > > > > uapi-bend that one into shape in at least drm. > > > > > > > > The one I had in mind is V4L2_PIX_FMT_SUNXI_TILED_NV12 which translates > > > > to DRM_FORMAT_NV12 + DRM_FORMAT_MOD_ALLWINNER_TILED. Seems to be a > > > > pretty similar case to the Mediatek one indeed. > > > > > > > > In our cause, that's because the video decoding engine produces its > > > > destination buffers in a specific tiled format, that the display engine > > > > can take in directly. > > > > > > We also have the Samsung tiling (Z pattern) as mentioned here, but also > > > linear 16x16 tile placement (also from Samsung ?) and I believe Amlogic > > > CODEC patches is bringing another tiling (unavoidable on older Meson8, > > > with 64bytes swaps). All these should be expressed as NV12 + mod in DRM > > > space. > > > > > > What is very often not enabled, but affect the performance on mainline > > > media drivers is the ARM frame buffer compression. I know that RK chips > > > have support for this, and that you can't achieve the maximum > > > throughput without that. This one is not documented anywhere, but I > > > understand that there is multiple variants that HW vendor licence. > > > Though, in general, each SoC are likely running a single variant, so a > > > single mod would make sense. > > > > We have AFBC modifiers now in drm_fourcc.h, jointly developed by > > display engineers from ARM and mali gpu reverse engineer people doing > > the panfrost driver. So should be covered. > > > > > So all this to say that V4L2 equally needs supports for these. What I > > > understood through DRM API is that a buffer allocated for let's say > > > NV12 + mod, is compatible with linear NV12. That could be used to > > > simplify some code, but at the same time, a common API that deals with > > > the padding and alignment of each format + mod independently would do > > > that same as long as this is not variable depending on which target HW > > > uses that same format. > > > > Not sure why you mean with NV12 + mod is compatible with linear NV12. > > In general fourcc + modifier != fourcc = linear modifier, size, number > > of planes, alignment constraints and everything else can be changed by > > a modifier (and there's examples for all of these, maybe not yet in > > all cases for NV12, but I think NV12 + AFBC modifiers gives some > > pretty interesting results). In generally you need to think of the > > (drm fourcc, modifier) as the pair identified the pixel format, each > > part individually is fairly meanigless. We have lots of modifiers > > where the exact tiling mode/layout changes depending upon the fourcc > > code. > > I only meant that the NV12 + mod have the same number of planes, and > should be large enough to store a linear NV12 equivalent. Not that it > would render correctly (even though I found it useful to be able to > render them when I needed to reverse it). It might be this assumption still holds for nv12, but we've definitely broken it for other fourcc. You can't rely on this at least being universally true. -Daniel
On Tue, Apr 23, 2019 at 5:45 PM Laurent Pinchart <laurent.pinchart@ideasonboard.com> wrote: > > Hi Daniel, > > On Tue, Apr 23, 2019 at 09:25:54AM +0200, Daniel Vetter wrote: > > On Sun, Apr 21, 2019 at 01:59:04AM +0300, Laurent Pinchart wrote: > > > On Thu, Apr 18, 2019 at 12:07:44PM +0200, Daniel Vetter wrote: > > >> On Thu, Apr 18, 2019 at 11:02 AM Maxime Ripard wrote: > > >>> On Thu, Apr 18, 2019 at 09:52:10AM +0200, Daniel Vetter wrote: > > >>>> On Thu, Apr 18, 2019 at 8:22 AM Maxime Ripard wrote: > > >>>>> On Wed, Apr 17, 2019 at 05:41:21PM +0200, Daniel Vetter wrote: > > >>>>>> On Wed, Apr 17, 2019 at 09:54:26AM +0200, Maxime Ripard wrote: > > >>>>>>> Hi, > > >>>>>>> > > >>>>>>> DRM comes with an extensive format support to retrieve the various > > >>>>>>> parameters associated with a given format (such as the subsampling, or the > > >>>>>>> bits per pixel), as well as some helpers and utilities to ease the driver > > >>>>>>> development. > > >>>>>>> > > >>>>>>> v4l2, on the other side, doesn't provide such facilities, leaving each > > >>>>>>> driver reimplement a subset of the formats parameters for the one supported > > >>>>>>> by that particular driver. This leads to a lot of duplication and > > >>>>>>> boilerplate code in the v4l2 drivers. > > >>>>>>> > > >>>>>>> This series tries to address this by moving the DRM format API into lib and > > >>>>>>> turning it into a more generic API. In order to do this, we've needed to do > > >>>>>>> some preliminary changes on the DRM drivers, then moved the API and finally > > >>>>>>> converted a v4l2 driver to give an example of how such library could be > > >>>>>>> used. > > >>>>>>> > > >>>>>>> Let me know what you think, > > >>>>>>> Maxime > > >>>>>>> > > >>>>>>> Changes from RFC: > > >>>>>>> - Rebased on next > > >>>>>>> - Fixed the various formats mapping > > >>>>>>> - Added tags > > >>>>>>> - Did most of the formats functions as inline functions > > >>>>>>> - Used a consistent prefix for all the utilities functions > > >>>>>>> - Fixed the compilation breakages, and did a run of allmodconfig for arm, > > >>>>>>> arm64 and x86_64 > > >>>>>>> - Fixed out of array bounds warnings in the image_format_info_block_* > > >>>>>>> functions > > >>>>>>> - Added License and copyright headers on missing files > > >>>>>>> > > >>>>>>> Maxime Ripard (20): > > >>>>>>> drm: Remove users of drm_format_num_planes > > >>>>>>> drm: Remove users of drm_format_(horz|vert)_chroma_subsampling > > >>>>>>> drm/fourcc: Pass the format_info pointer to drm_format_plane_cpp > > >>>>>>> drm/fourcc: Pass the format_info pointer to drm_format_plane_width/height > > >>>>>>> drm: Replace instances of drm_format_info by drm_get_format_info > > >>>>>>> lib: Add video format information library > > >>>>>>> drm/fb: Move from drm_format_info to image_format_info > > >>>>>>> drm/malidp: Convert to generic image format library > > >>>>>>> drm/client: Convert to generic image format library > > >>>>>>> drm/exynos: Convert to generic image format library > > >>>>>>> drm/i915: Convert to generic image format library > > >>>>>>> drm/ipuv3: Convert to generic image format library > > >>>>>>> drm/msm: Convert to generic image format library > > >>>>>>> drm/omap: Convert to generic image format library > > >>>>>>> drm/rockchip: Convert to generic image format library > > >>>>>>> drm/tegra: Convert to generic image format library > > >>>>>>> drm/fourcc: Remove old DRM format API > > >>>>>>> lib: image-formats: Add v4l2 formats support > > >>>>>>> lib: image-formats: Add more functions > > >>>>>>> media: sun6i: Convert to the image format API > > >>>>>> > > >>>>>> In the interest of making myself unpopular: Why move this out of drm? > > >>>>>> > > >>>>>> We do have a bunch of other such shared helpers already (mostly in > > >>>>>> drivers/video) for dt videomode and hdmi infoframes, and I'm not super > > >>>>>> sure that's going better than keeping it maintained in drm. > > > > > > That's a bit of a different situation as both DRM and FBDEV address the > > > same features (display output), and FBDEV is deprecared and replaced by > > > DRM. > > > > > > I'm not against maintaining a 4CC library in DRM (adding a third-party > > > maintainer would likely create more problems than it would solve), but > > > that doesn't mean the library has to live in drivers/gpu/, nor that it > > > needs to have the drm_ prefix. I would actually advocate to make it live > > > in a neutral directory, with a neutral prefix (kudos to anyone who can > > > propose a nice naming convention that would establish a new shared > > > ground for image/video-related Linux APIs), and maintained through the > > > DRM tree (possibly with extra entries in MAINTAINERS to ensure it > > > reaches all the related folks). > > > > > >>>>>> Plus the uapi is already that you include drm_fourcc.h to get at these, > > >>>>>> dropping the drm prefix isn't going to help I think. > > >>>>>> > > >>>>>> Of course we'd need to make it a separate drm_formats.ko (so that v4l can > > >>>>>> use it without dragging in all of drm), and we need to add some fields for > > >>>>>> converting to v4l fourcc codes (which are different). But that should be > > >>>>>> all possible. And I don't think the drm_ prefix in v4l code is a problem, > > >>>>>> it's actually a feature: It makes it really clear that these are the drm > > >>>>>> fourcc codes, as allocated in drm_fourcc.h, plus their modifiers, and all > > >>>>>> that. That allocation authority is also already baked into various khr/ext > > >>>>>> standards, too. > > > > > > There's one thing that V4L2 has and DRM hasn't for 4CCs: good > > > documentation. Look at > > > https://linuxtv.org/downloads/v4l-dvb-apis-new/uapi/v4l/pixfmt-packed-rgb.html > > > or > > > https://linuxtv.org/downloads/v4l-dvb-apis-new/uapi/v4l/yuv-formats.html > > > for instance. It's painful to write, painful to read, but defines the > > > 4CCs very clearly without ambiguity. I wouldn't be surprised if > > > different drivers used the same DRM 4CC for different formats due to the > > > lack of detailed documentation. Moving to a shared library for 4CCs > > > should also address the documentation side, and any new format added to > > > the kernel, whether from the V4L2 side or the DRM side, would be > > > required to provide detailed documentation. That would be a great > > > improvement for DRM 4CC handling. > > > > > >>>>> The way I see it, there's a fundamental difference between the UAPI > > >>>>> and the kernel. I don't suggest we change anything about the UAPI: the > > >>>>> drm formats should keep their prefix, drm_fourcc.h can remain that > > >>>>> authority, it's all fine. > > >>>>> > > >>>>> Internally however, the long term goal is to share the fourcc's > > >>>>> between DRM and V4L2 for the same formats. It basically means that of > > >>>>> course v4l2 should be using the DRM fourcc when a format exists in DRM > > >>>>> and not v4l2, but also that DRM should use v4l2 fourcc when the format > > >>>>> exists in v4l2 but not DRM, and that is far more likely, given the > > >>>>> already extensive v4l2 formats support. > > >>>> > > >>>> Uh no. We did look at v4l fourcc extensively when deciding upon a drm > > >>>> format identifier space. > > >>> > > >>> No to what exactly? > > >>> > > >>>> And a lot of people pushed for the "fourcc is a standard", when > > >>>> really it's totally not. > > >>> > > >>> Even if it's not a standard, having consistency would be a good thing. > > >>> > > >>> And you said yourself that DRM fourcc is now pretty much an authority > > >>> for the fourcc, so it definitely looks like a standard to me. > > >> > > >> drm fourcc is the authority for drm fourcc codes. Not for any of the > > >> others (and there's lots of them). Now it's used in a bunch of other > > >> places (khr standards, dri protocols in wayland/X11), but they're > > >> still only drm fourcc. There is no overall fourcc standard. > > >> > > >>>> v4l tends to conflate pixel format with stuff that we tend to encode > > >>>> in modifiers a lot more. > > >>> > > >>> Boris is working on adding the modifiers concept to v4l2, so we're > > >>> converging here, and we can totally have a layer in v4l2 to convert > > >>> between old v4l2 "format+modifiers" formats, and DRM style formats. > > >>> > > >>>> There's a bunch of reasons we can't just use v4l, and they're as > > >>>> valid as ever: > > >>>> > > >>>> - We overlap badly in some areas, so even if fourcc codes match, we > > >>>> can't use them and need a duplicated DRM_FOURCC code. > > >>> > > >>> Do yo have an example of one of those areas? > > >> > > >> I think the rgb stuff was one of the big reasons to not reuse any > > >> existing fourcc standard (whether v4l, or another one from e.g. video > > >> container formats). We had initial patches to reuse the fourcc that > > >> existed, but the end result was really inconsistent, so we ditch that > > >> and rolled out a new set of entirely drm specific fourcc codes for > > >> rgba. > > > > > > Could you give one or a couple of examples of V4L2 4CCs that are not > > > OCD-compatible ? :-) > > > > > >>>> - v4l encodes some metadata into the fourcc that we encode elsewhere, > > >>>> e.g. offset for planar yuv formats, or tiling mode > > >>> > > >>> As I was saying, this changes on the v4l2 side, and converging to > > >>> what DRM is doing. > > >>> > > >>>> - drm fourcc code doesn't actually define the drm_format_info > > >>>> uniquely, drivers can override that (that's an explicit design > > >>>> intent of modifiers, to allow drivers to add another plane for > > >>>> e.g. compression information). You'd need to pull that driver > > >>>> knowledge into your format library. > > > > > > That's a mistake in my opinion. We tried that in V4L2 to store metadata > > > in a separate plane, and had to go another route eventually as it > > > created a very bad mess. > > > > Just quick clarification in the middle here: This is how the hw works. > > The hardware takes parameters from a buffer, but it doesn't mandate how > that buffer is exposed to userspace :-) Using an extra plane is one > option, but other APIs are possible. I think Daniel Stone explains fairly well why some of our additional metadata is included as a plane, while a lot of the other metadata involved in rendering/compute the framebuffer isn't. Not really anything to add here. > > It's not metadata that sw ever touches (in general, testcases to make sure > > we display these correctly excepted). > > > > There has been some talking to add maybe a bit more mixed metadata, for > > fast-clear colors (which isn't used by any display engine afaik yet). That > > What are fast-clear colors ? hw offers enormous amounts of tricks to make glClear O(1), or at least close enough. glClear is usually what's done at the start of every frame, and clears the entire framebuffer to a uniform color. This is achieved usually through 3 pieces: - actual framebuffer plane with the pixel data - a 2nd plane that (usually, but there's lots of tricks here) contains a bit of metadata per cacheline/block/whatever in the framebuffer to indicate how/if those pixels are compressed, or whether they are still the uniform color supplied through glClear. For actual O(1) this metadata is hierarchical, so that a glClear really only sets the top-level metadata to "all subordinate blocks still have are the clear color". hw tends to have some pretty strong opinions on where it's going to look for that 2nd plane. - curiously on most hw the actual clear color is supplied separately (and our plan is to just stuff it into a 3rd plane) > > would generally be written by the cpu (in the gl stack), but again read by > > the hw (loaded as indirect state packet most likely, or something like > > that). So again hw specific layout, because the hw needs to read it. > > > > Pure metadata only of interest for the cpu/sw stack has been shot down > > completely on the drm side too. > > > > > There's a tendency in both subsystems to look at the other as a bit of a > > > retarded relative, and that's a shame, we have lots to learn from each > > > other's mistakes. That wouldn't be too difficult if people started > > > talking to each other. > > > > > > A semi-related comment: DRM also carries other mistakes of its own, I'm > > > thinking about DRM_FORMAT_BIG_ENDIAN in particular > > > > Yeah that one is hilarios, but in practice big endian is dead, except for > > a very few server chips, and there I think Gerd's work mostly fixed up > > that mess. > > > > >>> I'm not sure how my patches are changing anything here. This is > > >>> litterally the same code, with the functions renamed. > > >>> > > >>> If drivers want to override that, then yeah, fine, we can let them do > > >>> that. Just like any helper this just provides a default that covers > > >>> most of the cases. > > >>> > > >>>> Iow there's no way we can easily adopt v4l fourcc, except if we do > > >>>> something like a new addfb flag. > > >>> > > >>> For the formats that would be described as a modifier, sure. For all > > >>> the others (that are not yet supported by DRM), then I don't really > > >>> see why not. > > >> > > >> See above, we tried that initially, didn't pass the ocd tests when > > >> reviewing. Maybe the situation is better outside of rbgx/a formats, > > >> and I think we do at least try to use the same fourcc codes there when > > >> there already is one. But then the details occasionally look > > >> different. > > >> > > >>>>> And given how the current state is a mess in this regard, I'm not too > > >>>>> optimistic about keeping the formats in their relevant frameworks. > > >>>>> > > >>>>> Having a shared library, governed by both, will make this far easier, > > >>>>> since it will be easy to discover the formats that are already > > >>>>> supported by the other subsystem. > > >>>> > > >>>> I think a compat library that (tries to, best effort) convert between > > >>>> v4l and drm fourcc would make sense. Somewhere in drivers/video, next > > >>>> to the conversion functions for videomode <-> drm_display_mode > > >>>> perhaps. That should be useful for drivers. > > >>> > > >>> That's not really what this series is about though. That series is > > >>> about sharing the (image|pixels) formats database across everyone so > > >>> that everyone can benefit from it. > > >> > > >> Yeah I know. I still think leaving the drm fourcc with the drm prefix > > >> would be good, since there's really no standard here. > > >> > > >>>> Unifying the formats themselves, and all the associated metadata is > > >>>> imo a no-go, and was a pretty conscious decision when we implemented > > >>>> drm_fourcc a few years back. > > >>>> > > >>>>> If we want to keep the current library in DRM, we have two options > > >>>>> then: > > >>>>> > > >>>>> - Support all the v4l2 formats in the DRM library, which is > > >>>>> essentially what I'm doing in the last patches. However, that > > >>>>> would require to have the v4l2 developpers also reviewing stuff > > >>>>> there. And given how busy they are, I cannot really see how that > > >>>>> would work. > > >>>> > > >>>> Well, if we end up with a common library then yes we need cross > > >>>> review. There's no way around that. Doesn't matter where exactly that > > >>>> library is in the filesystem tree, and adding a special MAINTAINERS > > >>>> entry for anything related to fourcc (both drm and v4l) to make sure > > >>>> they get cross-posted is easy. No file renaming needed. > > >>> > > >>> That would create some governing issues as well. For example, if you > > >>> ever have a patch from one fourcc addition (that might or might not be > > >>> covered by v4l2), will you wait for any v4l2 developper to review it? > > >> > > >> None of this is fixed by code renaming or locations. Either way we > > >> need to figure that out. > > >> > > >> And yes part of the reasons for not moving this out of drm is that I'm > > >> not a fan of boutique trees for small stuff. If sharing means we need > > >> to split the drm_fourcc code and library out of drm trees, I'm not > > >> sure that's a good idea. We're already super liberal with merging > > >> anything through driver trees with acks, and integrating them quickly > > >> into drm-next. This would still be workable if v4l sends regular pull > > >> requests to drm-next (every 1-2 weeks, like the other big gpu trees > > >> do). If we can only sync up once per merge window with a shared > > >> boutique tree for formats only, life is going to be painful. > > > > > > That should be solved by the proposal above, maintaining the shared > > > library in the DRM tree. We would need to be careful there, and ideally > > > maintain that in a separate branch that could be merged in both DRM and > > > V4L2 without having to merge most of the other subsystem's pending > > > changes at the same time, but I think it's doable without any big issue. > > > > > >>> If it's shared code, then it should be shared, and every client > > >>> framework put on an equal footing. > > >>> > > >>>>> - Develop the same library from within v4l2. That is really a poor > > >>>>> solution, since the information would be greatly duplicated > > >>>>> between the two, and in terms of maintainance, code, and binary > > >>>>> size that would be duplicated too. > > >>>> > > >>>> It's essentially what we decided to do for drm years back. > > >>> > > >>> And it was probably the right solution back then, but I'm really not > > >>> convinced it's still the right thing to do today. > > >>> > > >>>>> Having it shared allows to easily share, and discover formats from the > > >>>>> other subsystem, and to have a single, unique place where this is > > >>>>> centralized. > > >>>> > > >>>> What I think could work as middle ground: > > >>>> - Put drm_format stuff into a separate .ko > > >>>> - Add a MAINTAINERS entry to make sure all things fourcc are cross > > >>>> posted to both drm and v4l lists. Easy on the drm side, since it's all > > >>>> separate files. Not sure it's so convenient for v4l uapi. > > >>>> - Add a conversion library that tries to best-effort map between drm > > >>>> and v4l formats. On the drm side that most likely means you need > > >>>> offsets for planes, and modifiers too (since those are implied in some > > >>>> v4l fourcc). Emphasis on "best effort" i.e. only support as much as > > >>>> the drivers that use this library need. > > >>>> - Add drm_fourcc as-needed by these drivers that want to use a unified > > >>>> format space. > > >>>> > > >>>> Forcing this unification on everyone otoh is imo way too much. > > >>> > > >>> v4l2 is starting to converge with DRM, and we're using the DRM API > > >>> pretty much untouched for that library, so I'm not really sure how > > >>> anyone is hurt by that unification. > > >> > > >> It might make sense to regularly pull v4l updates into drm-next then > > >> anyway. That would also remove the need to have the format library > > >> somewhere else. > > > > > > Are you saying it should the live in V4L2 ? ;-) > > > > Maybe a few clarifications on how the drm shared core thing usually works, > > and why I'm a sticker here. Bottom reply since I'm not sure where to put > > it: > > > > - Refactorings usually go in through drm-misc (at least since a few > > years). > > > > - Small patches go in through the relevant driver tree (which is often > > drm-misc, but not always), with an Ack from drm maintainers. > > > > - No topic branches, everyone just pushes patches where it's most > > convenient. > > > > We get away with this mess because everyone sends regular pull requests to > > drm, where the entire history is baked in and others can backmerge/fast > > forward/rebase. Worst case you wait one month (around the merge window, > > when drm-next is closed for features), but usually it's just 1-2 weeks. > > Plus we tend to have fairly big trees, with good chances that the next > > patch series lands in the same tree again and no work at all is needed. > > > > If we start regularly sharing lots of code with v4l (which seems to be the > > long term goal here), then I think we need something equally convenient > > for all that. > > > > We're not going to be able to teach some complicated topic branch scheme > > to the 50+ submaintainers/committers we have in drm - a lot much more > > basic stuff already takes lots of work to get it to stick. If the proposal > > is "to be careful" and "maintain it in a separate branch", I'm not in > > favour because I think that just wouldn't work. > > Why not ? It can be a fast-moving branch that gets merged in drm-misc > as often as you want (even at every commit if that's desired). We're > dealing with a limited amount of code here, and there's no more reason > that V4L2 should pull in drm-misc to get 4CC updates than DRM should > pull V4L2 for the same. I have no objection against a 4CC branch being > officially maintained under the DRM umbrella, but I think the code > should live elsewhere than drivers/gpu/drm/, have a neutral prefix, and > not require pulling an entire subsystem in. I think small boutique trees are a problem themselves, not a solution. So if you're creating a new small boutique tree to fix a problem, you then have 2. Yes, assuming sufficient expenditure of energy it can be made to work, but I'd prefer to make my own life as easy and lazy as possible :-) And I think I've been fairly successful at that within drivers/gpu at least. Imo the proper fix is to merge v4l and drm, at a process/maintainer level. That would solve both the original issue and the 2ndary one of the permanent topic branch. -Daniel
Hi Maxime, Apologies for the late reply, most if this thread happened when I was on vacation, and I missed that I should reply to it. Thank you for reminding me. Please let me know if I should reply to other mails in the discussion of this patch series. On 4/18/19 10:56 PM, Maxime Ripard wrote: > On Thu, Apr 18, 2019 at 02:32:14PM +0200, Daniel Vetter wrote: >>>>>> Unifying the formats themselves, and all the associated metadata is >>>>>> imo a no-go, and was a pretty conscious decision when we implemented >>>>>> drm_fourcc a few years back. >>>>>> >>>>>>> If we want to keep the current library in DRM, we have two options >>>>>>> then: >>>>>>> >>>>>>> - Support all the v4l2 formats in the DRM library, which is >>>>>>> essentially what I'm doing in the last patches. However, that >>>>>>> would require to have the v4l2 developpers also reviewing stuff >>>>>>> there. And given how busy they are, I cannot really see how that >>>>>>> would work. >>>>>> >>>>>> Well, if we end up with a common library then yes we need cross >>>>>> review. There's no way around that. Doesn't matter where exactly that >>>>>> library is in the filesystem tree, and adding a special MAINTAINERS >>>>>> entry for anything related to fourcc (both drm and v4l) to make sure >>>>>> they get cross-posted is easy. No file renaming needed. >>>>> >>>>> That would create some governing issues as well. For example, if you >>>>> ever have a patch from one fourcc addition (that might or might not be >>>>> covered by v4l2), will you wait for any v4l2 developper to review it? >>>> >>>> None of this is fixed by code renaming or locations. Either way we >>>> need to figure that out. >>>> >>>> And yes part of the reasons for not moving this out of drm is that I'm >>>> not a fan of boutique trees for small stuff. If sharing means we need >>>> to split the drm_fourcc code and library out of drm trees, I'm not >>>> sure that's a good idea. We're already super liberal with merging >>>> anything through driver trees with acks, and integrating them quickly >>>> into drm-next. This would still be workable if v4l sends regular pull >>>> requests to drm-next (every 1-2 weeks, like the other big gpu trees >>>> do). If we can only sync up once per merge window with a shared >>>> boutique tree for formats only, life is going to be painful. >>> >>> I don't get why we want to turn DRM into some kind of a black hole >>> that would pull everything. We don't have to, really. And at the same >>> time it carries the message that v4l2 is less important than DRM for >>> some reason, which I'm really not comfortable with. >> >> Make another tree somewhere that pulls in trees more often than every >> merge window, and I'm happy. It's the coordination effort of lots of >> trees that creates the black hole, not the other way round. Yes topic >> trees work, but if topic trees are persistent something with the >> organization of trees is wrong and needs to change. This very much >> looks like we'll end up with a perpetual topic branch for format stuff >> between drm and v4l. > > Well, if v4l2 sends a PR to DRM every 1 or 2 weeks, that definitely > looks like a topic branch to me. And on a far more frequent basis than > when we will merge a format description. > >> The other shared stuff (like hdmi infoframes) seems to change a lot >> less often, so the occasional patch hasn't been a pain. But drm_fourcc >> related stuff sees a lot of work, both in adding new formats and in >> refactoring the library to keep up with all the new use-cases. > > When was the last time a refactoring that changed the API happened? > > Most of the changes will be new format descriptions, and I guess that > would only concern a single tree. > > And really, we're doing this all the time, so I'm not sure what the > big deal is here. > > I feel like there's something that you don't really like about this, > but you're not saying this out loud. > > Sure, the exact process needs to be figured out, and everyone needs to > agree upon that process. But that's pretty much it, and it's nothing > out of the ordinary. > >> And yes I think an overall gfx-like-stuff.git tree for drm and v4l and >> the few other bits really makes tons of sense. Not as a tree where >> people commit, but as the 2nd-level integration tree (like drm.git >> right now for gpu stuff). >> >>> And I don't really get why you're against this in the first >>> place. When you have some code in a single driver that would benefit >>> more driver, you create a helper and move it into the core. >> >> It's a feature that drm doesn't share that much code with other parts >> of the kernel, it makes backporting the gfx stack to lts kernels a lot >> easier. Until someone fixes the upstream kernel release model to no >> longer need large scale gpu driver backports, we need to keep that. >> >>> In this case, we have some code used by a framework that more >>> framework could use, and we move it to a core-er place. How is that >>> different? >> >> Imo core sharing for code sharing's sake is overrated. If we already >> have drm and v4l tightly integrated as a community, then code sharing >> becomes a lot easier, and a lot more reasonable to do. > > At least Laurent, Boris, Ezequiel, Gustavo and I have been working on > v4l2, so I'm not sure how not integrated we are. > >> Plus we can then just stuff code int drivers/gpu or drivers/video >> (or merge these two because really it's all the same). But my >> understanding is that integrating more tightly with the drm folks is >> a very contreversial topic in v4l > > So, I sent the RFC expecting that kind of feedback. > > Hans replied mainly to that patch https://patchwork.freedesktop.org/patch/293043/ > > " > If we are creating a common library then I think we should change that rule > to: "unless they are in use by a DRM or V4L2 driver". And when new formats are > added, and they exists already for DRM or V4L2, then we should use the same > fourcc for the other subsystem. > > I.e. if pixelformat V4L2_PIX_FMT_FOO was already defined, then add a: > > #define DRM_FORMAT_FOO V4L2_PIX_FMT_FOO > > rather than creating a new fourcc. > > We could even start looking at redoing the whole scheme in a unified way, but > that's something for the (far) future. > > This is already a big step forward. > " > > So, not controversial at all. > >> and until that's resolved I don't see a huge need or benefit in >> sharing tons of code. > > That's mostly tons of data though. The code is pretty small and > trivial. > >> And the format stuff is a lot more central to kms than e.g. the >> infoframe helpers. >> >> Au contraire, I think forcing this has a lot of potential for >> needless fights between drm and v4l. > > We're all reasonable, so I'm not sure why we would need to fight here. > >> Hence my suggestion to try a minimal format conversion library >> between the drm format world and the v4l format wolrd, and see how >> that goes. That contains a lot less risk than going all in right >> from the start. > > And it's really not about getting access to the DRM fourcc. It's about > getting access to DRM's format description, so I'm not really sure > what there is to convert, we just want a lookup. I see this patch series as a first step towards further integration between drm/v4l (at least in this area). That said, the V4L API has some serious limitations at the moment when it comes to describing these formats, and until that is fixed there is no point in doing more than what this series does. I certainly don't expect a lot of patches, and I am happy for drm to maintain this for now. Work has started on improving how V4L2 handles formats (1) but this will take quite some time before this is ready. In part because we are working on things that have a higher prio (HW codec support). But the intention is to become much closer to how drm handles this to simplify format matching between the two subsystems. Once that's in place (next year?) it will become much more interesting to look at further integration between drm and v4l. Note that that is just my personal opinion. Others may very well disagree. So in my view this series is a first solid step towards further integration. And initially it should be fine to have the most common formats available in this new lib. That should solve most of the current problems. Regards, Hans (1) https://www.mail-archive.com/linux-media@vger.kernel.org/msg146075.html
Hi Paul, On Tue, Apr 23, 2019 at 06:54:49PM +0200, Paul Kocialkowski wrote: > Le dimanche 21 avril 2019 à 01:40 +0300, Laurent Pinchart a écrit : > > On Thu, Apr 18, 2019 at 01:49:54PM +0200, Paul Kocialkowski wrote: > >> On Thu, 2019-04-18 at 11:02 +0200, Maxime Ripard wrote: > >>> On Thu, Apr 18, 2019 at 09:52:10AM +0200, Daniel Vetter wrote: > >>>> And a lot of people pushed for the "fourcc is a standard", when > >>>> really it's totally not. > >>> > >>> Even if it's not a standard, having consistency would be a good thing. > >>> > >>> And you said yourself that DRM fourcc is now pretty much an authority > >>> for the fourcc, so it definitely looks like a standard to me. > >> > >> I think trying to make the V4L2 and DRM fourccs converge is a lost > >> cause, as it has already significantly diverged. Even if we coordinate > >> an effort to introduce new formats with the same fourcc on both sides, > >> I don't see what good that would be since the formats we have now are > >> still plagued by the inconsistency. > >> > >> I think we always need an explicit translation step from either v4l2 or > >> drm to the internal representation and back, without ever assuming that > >> formats might be compatible because they share the same fourcc. > > > > I don't agree. APIs evolve, and while we can't switch from one set of > > 4CCs to another in existing APIs, we could in new APIs. Boris is working > > on new ioctls to handle formats in V4L2, and while 4CC unification could > > be impopular from a userspace developers point of view there, I don't > > think we have ruled it out completely. The move to the request API is > > also an area where a common set of 4CCs could be used, as it will depart > > from the existing V4L2 ioctls. To summarize my opinion, we're not there > > yet, but I wouldn't rule it out completely for the future. > > Well, I don't see how we could maintain backward compatibility with > some DRM and V4L2 fourccs that are compatible and some that aren't. > Since both descriptions have diverged already, one would need explicit > checking of whether the format at hand is a compatible one or not > before passing-it along as-is to the other subsystem or going through a > format conversion step (in userspace, duplicating the information). > So it feels like it kind of defeats the purpose. Yes and no. If a V4L2-only application needs to deal with two sets of 4CCs for old and new APIs, it's a burden. However, if an application that shares buffers between V4L2 and DRM/KMS can use a single set of 4CCs, that's a win, as those applications already need to convert. In the end this shifts the problem from one set of applications to another, but as soon as the new APIs get old enough to mandate their usage (which would roughly be when the corresponding kernel ships in all supported versions of all major distros), then applications will be able to drop the conversion code, and everybody will be happier. Short term pain for long term gain. > If we're going to use a unified 4CC representation in the future, I > think we should do it by using the new formats that this proposal is > introducing instead of subsystem-specific formats. At which point I > believe we will need an internal conversion step between that format > and what the subsystem uses internally. Or do it the other way round > and use the unified format all around the subsystem, with a legacy > layer for the previous subsystem-specific format. I agree with you. I would like new APIs to use unified 4CCs (probably coming from DRM), and have an internal conversion from and to subsystem-specific 4CCs where needed. That's quite a lot of work of course, and won't happen overnight.
Hi Daniel, On Tue, Apr 23, 2019 at 09:18:52PM +0200, Daniel Vetter wrote: > On Tue, Apr 23, 2019 at 5:45 PM Laurent Pinchart wrote: > > On Tue, Apr 23, 2019 at 09:25:54AM +0200, Daniel Vetter wrote: > >> On Sun, Apr 21, 2019 at 01:59:04AM +0300, Laurent Pinchart wrote: > >>> On Thu, Apr 18, 2019 at 12:07:44PM +0200, Daniel Vetter wrote: > >>>> On Thu, Apr 18, 2019 at 11:02 AM Maxime Ripard wrote: > >>>>> On Thu, Apr 18, 2019 at 09:52:10AM +0200, Daniel Vetter wrote: > >>>>>> On Thu, Apr 18, 2019 at 8:22 AM Maxime Ripard wrote: > >>>>>>> On Wed, Apr 17, 2019 at 05:41:21PM +0200, Daniel Vetter wrote: > >>>>>>>> On Wed, Apr 17, 2019 at 09:54:26AM +0200, Maxime Ripard wrote: > >>>>>>>>> Hi, > >>>>>>>>> > >>>>>>>>> DRM comes with an extensive format support to retrieve the various > >>>>>>>>> parameters associated with a given format (such as the subsampling, or the > >>>>>>>>> bits per pixel), as well as some helpers and utilities to ease the driver > >>>>>>>>> development. > >>>>>>>>> > >>>>>>>>> v4l2, on the other side, doesn't provide such facilities, leaving each > >>>>>>>>> driver reimplement a subset of the formats parameters for the one supported > >>>>>>>>> by that particular driver. This leads to a lot of duplication and > >>>>>>>>> boilerplate code in the v4l2 drivers. > >>>>>>>>> > >>>>>>>>> This series tries to address this by moving the DRM format API into lib and > >>>>>>>>> turning it into a more generic API. In order to do this, we've needed to do > >>>>>>>>> some preliminary changes on the DRM drivers, then moved the API and finally > >>>>>>>>> converted a v4l2 driver to give an example of how such library could be > >>>>>>>>> used. > >>>>>>>>> > >>>>>>>>> Let me know what you think, > >>>>>>>>> Maxime > >>>>>>>>> > >>>>>>>>> Changes from RFC: > >>>>>>>>> - Rebased on next > >>>>>>>>> - Fixed the various formats mapping > >>>>>>>>> - Added tags > >>>>>>>>> - Did most of the formats functions as inline functions > >>>>>>>>> - Used a consistent prefix for all the utilities functions > >>>>>>>>> - Fixed the compilation breakages, and did a run of allmodconfig for arm, > >>>>>>>>> arm64 and x86_64 > >>>>>>>>> - Fixed out of array bounds warnings in the image_format_info_block_* > >>>>>>>>> functions > >>>>>>>>> - Added License and copyright headers on missing files > >>>>>>>>> > >>>>>>>>> Maxime Ripard (20): > >>>>>>>>> drm: Remove users of drm_format_num_planes > >>>>>>>>> drm: Remove users of drm_format_(horz|vert)_chroma_subsampling > >>>>>>>>> drm/fourcc: Pass the format_info pointer to drm_format_plane_cpp > >>>>>>>>> drm/fourcc: Pass the format_info pointer to drm_format_plane_width/height > >>>>>>>>> drm: Replace instances of drm_format_info by drm_get_format_info > >>>>>>>>> lib: Add video format information library > >>>>>>>>> drm/fb: Move from drm_format_info to image_format_info > >>>>>>>>> drm/malidp: Convert to generic image format library > >>>>>>>>> drm/client: Convert to generic image format library > >>>>>>>>> drm/exynos: Convert to generic image format library > >>>>>>>>> drm/i915: Convert to generic image format library > >>>>>>>>> drm/ipuv3: Convert to generic image format library > >>>>>>>>> drm/msm: Convert to generic image format library > >>>>>>>>> drm/omap: Convert to generic image format library > >>>>>>>>> drm/rockchip: Convert to generic image format library > >>>>>>>>> drm/tegra: Convert to generic image format library > >>>>>>>>> drm/fourcc: Remove old DRM format API > >>>>>>>>> lib: image-formats: Add v4l2 formats support > >>>>>>>>> lib: image-formats: Add more functions > >>>>>>>>> media: sun6i: Convert to the image format API > >>>>>>>> > >>>>>>>> In the interest of making myself unpopular: Why move this out of drm? > >>>>>>>> > >>>>>>>> We do have a bunch of other such shared helpers already (mostly in > >>>>>>>> drivers/video) for dt videomode and hdmi infoframes, and I'm not super > >>>>>>>> sure that's going better than keeping it maintained in drm. > >>> > >>> That's a bit of a different situation as both DRM and FBDEV address the > >>> same features (display output), and FBDEV is deprecared and replaced by > >>> DRM. > >>> > >>> I'm not against maintaining a 4CC library in DRM (adding a third-party > >>> maintainer would likely create more problems than it would solve), but > >>> that doesn't mean the library has to live in drivers/gpu/, nor that it > >>> needs to have the drm_ prefix. I would actually advocate to make it live > >>> in a neutral directory, with a neutral prefix (kudos to anyone who can > >>> propose a nice naming convention that would establish a new shared > >>> ground for image/video-related Linux APIs), and maintained through the > >>> DRM tree (possibly with extra entries in MAINTAINERS to ensure it > >>> reaches all the related folks). > >>> > >>>>>>>> Plus the uapi is already that you include drm_fourcc.h to get at these, > >>>>>>>> dropping the drm prefix isn't going to help I think. > >>>>>>>> > >>>>>>>> Of course we'd need to make it a separate drm_formats.ko (so that v4l can > >>>>>>>> use it without dragging in all of drm), and we need to add some fields for > >>>>>>>> converting to v4l fourcc codes (which are different). But that should be > >>>>>>>> all possible. And I don't think the drm_ prefix in v4l code is a problem, > >>>>>>>> it's actually a feature: It makes it really clear that these are the drm > >>>>>>>> fourcc codes, as allocated in drm_fourcc.h, plus their modifiers, and all > >>>>>>>> that. That allocation authority is also already baked into various khr/ext > >>>>>>>> standards, too. > >>> > >>> There's one thing that V4L2 has and DRM hasn't for 4CCs: good > >>> documentation. Look at > >>> https://linuxtv.org/downloads/v4l-dvb-apis-new/uapi/v4l/pixfmt-packed-rgb.html > >>> or > >>> https://linuxtv.org/downloads/v4l-dvb-apis-new/uapi/v4l/yuv-formats.html > >>> for instance. It's painful to write, painful to read, but defines the > >>> 4CCs very clearly without ambiguity. I wouldn't be surprised if > >>> different drivers used the same DRM 4CC for different formats due to the > >>> lack of detailed documentation. Moving to a shared library for 4CCs > >>> should also address the documentation side, and any new format added to > >>> the kernel, whether from the V4L2 side or the DRM side, would be > >>> required to provide detailed documentation. That would be a great > >>> improvement for DRM 4CC handling. > >>> > >>>>>>> The way I see it, there's a fundamental difference between the UAPI > >>>>>>> and the kernel. I don't suggest we change anything about the UAPI: the > >>>>>>> drm formats should keep their prefix, drm_fourcc.h can remain that > >>>>>>> authority, it's all fine. > >>>>>>> > >>>>>>> Internally however, the long term goal is to share the fourcc's > >>>>>>> between DRM and V4L2 for the same formats. It basically means that of > >>>>>>> course v4l2 should be using the DRM fourcc when a format exists in DRM > >>>>>>> and not v4l2, but also that DRM should use v4l2 fourcc when the format > >>>>>>> exists in v4l2 but not DRM, and that is far more likely, given the > >>>>>>> already extensive v4l2 formats support. > >>>>>> > >>>>>> Uh no. We did look at v4l fourcc extensively when deciding upon a drm > >>>>>> format identifier space. > >>>>> > >>>>> No to what exactly? > >>>>> > >>>>>> And a lot of people pushed for the "fourcc is a standard", when > >>>>>> really it's totally not. > >>>>> > >>>>> Even if it's not a standard, having consistency would be a good thing. > >>>>> > >>>>> And you said yourself that DRM fourcc is now pretty much an authority > >>>>> for the fourcc, so it definitely looks like a standard to me. > >>>> > >>>> drm fourcc is the authority for drm fourcc codes. Not for any of the > >>>> others (and there's lots of them). Now it's used in a bunch of other > >>>> places (khr standards, dri protocols in wayland/X11), but they're > >>>> still only drm fourcc. There is no overall fourcc standard. > >>>> > >>>>>> v4l tends to conflate pixel format with stuff that we tend to encode > >>>>>> in modifiers a lot more. > >>>>> > >>>>> Boris is working on adding the modifiers concept to v4l2, so we're > >>>>> converging here, and we can totally have a layer in v4l2 to convert > >>>>> between old v4l2 "format+modifiers" formats, and DRM style formats. > >>>>> > >>>>>> There's a bunch of reasons we can't just use v4l, and they're as > >>>>>> valid as ever: > >>>>>> > >>>>>> - We overlap badly in some areas, so even if fourcc codes match, we > >>>>>> can't use them and need a duplicated DRM_FOURCC code. > >>>>> > >>>>> Do yo have an example of one of those areas? > >>>> > >>>> I think the rgb stuff was one of the big reasons to not reuse any > >>>> existing fourcc standard (whether v4l, or another one from e.g. video > >>>> container formats). We had initial patches to reuse the fourcc that > >>>> existed, but the end result was really inconsistent, so we ditch that > >>>> and rolled out a new set of entirely drm specific fourcc codes for > >>>> rgba. > >>> > >>> Could you give one or a couple of examples of V4L2 4CCs that are not > >>> OCD-compatible ? :-) > >>> > >>>>>> - v4l encodes some metadata into the fourcc that we encode elsewhere, > >>>>>> e.g. offset for planar yuv formats, or tiling mode > >>>>> > >>>>> As I was saying, this changes on the v4l2 side, and converging to > >>>>> what DRM is doing. > >>>>> > >>>>>> - drm fourcc code doesn't actually define the drm_format_info > >>>>>> uniquely, drivers can override that (that's an explicit design > >>>>>> intent of modifiers, to allow drivers to add another plane for > >>>>>> e.g. compression information). You'd need to pull that driver > >>>>>> knowledge into your format library. > >>> > >>> That's a mistake in my opinion. We tried that in V4L2 to store metadata > >>> in a separate plane, and had to go another route eventually as it > >>> created a very bad mess. > >> > >> Just quick clarification in the middle here: This is how the hw works. > > > > The hardware takes parameters from a buffer, but it doesn't mandate how > > that buffer is exposed to userspace :-) Using an extra plane is one > > option, but other APIs are possible. > > I think Daniel Stone explains fairly well why some of our additional > metadata is included as a plane, while a lot of the other metadata > involved in rendering/compute the framebuffer isn't. Not really > anything to add here. > > >> It's not metadata that sw ever touches (in general, testcases to make sure > >> we display these correctly excepted). > >> > >> There has been some talking to add maybe a bit more mixed metadata, for > >> fast-clear colors (which isn't used by any display engine afaik yet). That > > > > What are fast-clear colors ? > > hw offers enormous amounts of tricks to make glClear O(1), or at least > close enough. glClear is usually what's done at the start of every > frame, and clears the entire framebuffer to a uniform color. This is > achieved usually through 3 pieces: > - actual framebuffer plane with the pixel data > - a 2nd plane that (usually, but there's lots of tricks here) contains > a bit of metadata per cacheline/block/whatever in the framebuffer to > indicate how/if those pixels are compressed, or whether they are still > the uniform color supplied through glClear. For actual O(1) this > metadata is hierarchical, so that a glClear really only sets the > top-level metadata to "all subordinate blocks still have are the clear > color". hw tends to have some pretty strong opinions on where it's > going to look for that 2nd plane. > - curiously on most hw the actual clear color is supplied separately > (and our plan is to just stuff it into a 3rd plane) Just to clarify, is this needed for display engines ? Does the GPU render to a 3 planes buffer with glClear()-related data in planes 2 and 3, with the buffer then being passed to the display engine that knows how to interpret this, or are those extra planes only needed for GPU rendering ? > >> would generally be written by the cpu (in the gl stack), but again read by > >> the hw (loaded as indirect state packet most likely, or something like > >> that). So again hw specific layout, because the hw needs to read it. > >> > >> Pure metadata only of interest for the cpu/sw stack has been shot down > >> completely on the drm side too. > >> > >>> There's a tendency in both subsystems to look at the other as a bit of a > >>> retarded relative, and that's a shame, we have lots to learn from each > >>> other's mistakes. That wouldn't be too difficult if people started > >>> talking to each other. > >>> > >>> A semi-related comment: DRM also carries other mistakes of its own, I'm > >>> thinking about DRM_FORMAT_BIG_ENDIAN in particular > >> > >> Yeah that one is hilarios, but in practice big endian is dead, except for > >> a very few server chips, and there I think Gerd's work mostly fixed up > >> that mess. > >> > >>>>> I'm not sure how my patches are changing anything here. This is > >>>>> litterally the same code, with the functions renamed. > >>>>> > >>>>> If drivers want to override that, then yeah, fine, we can let them do > >>>>> that. Just like any helper this just provides a default that covers > >>>>> most of the cases. > >>>>> > >>>>>> Iow there's no way we can easily adopt v4l fourcc, except if we do > >>>>>> something like a new addfb flag. > >>>>> > >>>>> For the formats that would be described as a modifier, sure. For all > >>>>> the others (that are not yet supported by DRM), then I don't really > >>>>> see why not. > >>>> > >>>> See above, we tried that initially, didn't pass the ocd tests when > >>>> reviewing. Maybe the situation is better outside of rbgx/a formats, > >>>> and I think we do at least try to use the same fourcc codes there when > >>>> there already is one. But then the details occasionally look > >>>> different. > >>>> > >>>>>>> And given how the current state is a mess in this regard, I'm not too > >>>>>>> optimistic about keeping the formats in their relevant frameworks. > >>>>>>> > >>>>>>> Having a shared library, governed by both, will make this far easier, > >>>>>>> since it will be easy to discover the formats that are already > >>>>>>> supported by the other subsystem. > >>>>>> > >>>>>> I think a compat library that (tries to, best effort) convert between > >>>>>> v4l and drm fourcc would make sense. Somewhere in drivers/video, next > >>>>>> to the conversion functions for videomode <-> drm_display_mode > >>>>>> perhaps. That should be useful for drivers. > >>>>> > >>>>> That's not really what this series is about though. That series is > >>>>> about sharing the (image|pixels) formats database across everyone so > >>>>> that everyone can benefit from it. > >>>> > >>>> Yeah I know. I still think leaving the drm fourcc with the drm prefix > >>>> would be good, since there's really no standard here. > >>>> > >>>>>> Unifying the formats themselves, and all the associated metadata is > >>>>>> imo a no-go, and was a pretty conscious decision when we implemented > >>>>>> drm_fourcc a few years back. > >>>>>> > >>>>>>> If we want to keep the current library in DRM, we have two options > >>>>>>> then: > >>>>>>> > >>>>>>> - Support all the v4l2 formats in the DRM library, which is > >>>>>>> essentially what I'm doing in the last patches. However, that > >>>>>>> would require to have the v4l2 developpers also reviewing stuff > >>>>>>> there. And given how busy they are, I cannot really see how that > >>>>>>> would work. > >>>>>> > >>>>>> Well, if we end up with a common library then yes we need cross > >>>>>> review. There's no way around that. Doesn't matter where exactly that > >>>>>> library is in the filesystem tree, and adding a special MAINTAINERS > >>>>>> entry for anything related to fourcc (both drm and v4l) to make sure > >>>>>> they get cross-posted is easy. No file renaming needed. > >>>>> > >>>>> That would create some governing issues as well. For example, if you > >>>>> ever have a patch from one fourcc addition (that might or might not be > >>>>> covered by v4l2), will you wait for any v4l2 developper to review it? > >>>> > >>>> None of this is fixed by code renaming or locations. Either way we > >>>> need to figure that out. > >>>> > >>>> And yes part of the reasons for not moving this out of drm is that I'm > >>>> not a fan of boutique trees for small stuff. If sharing means we need > >>>> to split the drm_fourcc code and library out of drm trees, I'm not > >>>> sure that's a good idea. We're already super liberal with merging > >>>> anything through driver trees with acks, and integrating them quickly > >>>> into drm-next. This would still be workable if v4l sends regular pull > >>>> requests to drm-next (every 1-2 weeks, like the other big gpu trees > >>>> do). If we can only sync up once per merge window with a shared > >>>> boutique tree for formats only, life is going to be painful. > >>> > >>> That should be solved by the proposal above, maintaining the shared > >>> library in the DRM tree. We would need to be careful there, and ideally > >>> maintain that in a separate branch that could be merged in both DRM and > >>> V4L2 without having to merge most of the other subsystem's pending > >>> changes at the same time, but I think it's doable without any big issue. > >>> > >>>>> If it's shared code, then it should be shared, and every client > >>>>> framework put on an equal footing. > >>>>> > >>>>>>> - Develop the same library from within v4l2. That is really a poor > >>>>>>> solution, since the information would be greatly duplicated > >>>>>>> between the two, and in terms of maintainance, code, and binary > >>>>>>> size that would be duplicated too. > >>>>>> > >>>>>> It's essentially what we decided to do for drm years back. > >>>>> > >>>>> And it was probably the right solution back then, but I'm really not > >>>>> convinced it's still the right thing to do today. > >>>>> > >>>>>>> Having it shared allows to easily share, and discover formats from the > >>>>>>> other subsystem, and to have a single, unique place where this is > >>>>>>> centralized. > >>>>>> > >>>>>> What I think could work as middle ground: > >>>>>> - Put drm_format stuff into a separate .ko > >>>>>> - Add a MAINTAINERS entry to make sure all things fourcc are cross > >>>>>> posted to both drm and v4l lists. Easy on the drm side, since it's all > >>>>>> separate files. Not sure it's so convenient for v4l uapi. > >>>>>> - Add a conversion library that tries to best-effort map between drm > >>>>>> and v4l formats. On the drm side that most likely means you need > >>>>>> offsets for planes, and modifiers too (since those are implied in some > >>>>>> v4l fourcc). Emphasis on "best effort" i.e. only support as much as > >>>>>> the drivers that use this library need. > >>>>>> - Add drm_fourcc as-needed by these drivers that want to use a unified > >>>>>> format space. > >>>>>> > >>>>>> Forcing this unification on everyone otoh is imo way too much. > >>>>> > >>>>> v4l2 is starting to converge with DRM, and we're using the DRM API > >>>>> pretty much untouched for that library, so I'm not really sure how > >>>>> anyone is hurt by that unification. > >>>> > >>>> It might make sense to regularly pull v4l updates into drm-next then > >>>> anyway. That would also remove the need to have the format library > >>>> somewhere else. > >>> > >>> Are you saying it should the live in V4L2 ? ;-) > >> > >> Maybe a few clarifications on how the drm shared core thing usually works, > >> and why I'm a sticker here. Bottom reply since I'm not sure where to put > >> it: > >> > >> - Refactorings usually go in through drm-misc (at least since a few > >> years). > >> > >> - Small patches go in through the relevant driver tree (which is often > >> drm-misc, but not always), with an Ack from drm maintainers. > >> > >> - No topic branches, everyone just pushes patches where it's most > >> convenient. > >> > >> We get away with this mess because everyone sends regular pull requests to > >> drm, where the entire history is baked in and others can backmerge/fast > >> forward/rebase. Worst case you wait one month (around the merge window, > >> when drm-next is closed for features), but usually it's just 1-2 weeks. > >> Plus we tend to have fairly big trees, with good chances that the next > >> patch series lands in the same tree again and no work at all is needed. > >> > >> If we start regularly sharing lots of code with v4l (which seems to be the > >> long term goal here), then I think we need something equally convenient > >> for all that. > >> > >> We're not going to be able to teach some complicated topic branch scheme > >> to the 50+ submaintainers/committers we have in drm - a lot much more > >> basic stuff already takes lots of work to get it to stick. If the proposal > >> is "to be careful" and "maintain it in a separate branch", I'm not in > >> favour because I think that just wouldn't work. > > > > Why not ? It can be a fast-moving branch that gets merged in drm-misc > > as often as you want (even at every commit if that's desired). We're > > dealing with a limited amount of code here, and there's no more reason > > that V4L2 should pull in drm-misc to get 4CC updates than DRM should > > pull V4L2 for the same. I have no objection against a 4CC branch being > > officially maintained under the DRM umbrella, but I think the code > > should live elsewhere than drivers/gpu/drm/, have a neutral prefix, and > > not require pulling an entire subsystem in. > > I think small boutique trees are a problem themselves, not a solution. > So if you're creating a new small boutique tree to fix a problem, you > then have 2. Yes, assuming sufficient expenditure of energy it can be > made to work, but I'd prefer to make my own life as easy and lazy as > possible :-) And I think I've been fairly successful at that within > drivers/gpu at least. > > Imo the proper fix is to merge v4l and drm, at a process/maintainer > level. That would solve both the original issue and the 2ndary one of > the permanent topic branch. Proposals are welcome ;-)
On Sat, May 11, 2019 at 10:26:32PM +0300, Laurent Pinchart wrote: > On Tue, Apr 23, 2019 at 09:18:52PM +0200, Daniel Vetter wrote: > > On Tue, Apr 23, 2019 at 5:45 PM Laurent Pinchart wrote: > > > On Tue, Apr 23, 2019 at 09:25:54AM +0200, Daniel Vetter wrote: > > >> On Sun, Apr 21, 2019 at 01:59:04AM +0300, Laurent Pinchart wrote: > > >>> On Thu, Apr 18, 2019 at 12:07:44PM +0200, Daniel Vetter wrote: > > >>>> On Thu, Apr 18, 2019 at 11:02 AM Maxime Ripard wrote: > > >>>>> On Thu, Apr 18, 2019 at 09:52:10AM +0200, Daniel Vetter wrote: > > >>>>>> On Thu, Apr 18, 2019 at 8:22 AM Maxime Ripard wrote: > > >>>>>>> On Wed, Apr 17, 2019 at 05:41:21PM +0200, Daniel Vetter wrote: > > >>>>>>>> On Wed, Apr 17, 2019 at 09:54:26AM +0200, Maxime Ripard wrote: > > >>>>>>>>> Hi, > > >>>>>>>>> > > >>>>>>>>> DRM comes with an extensive format support to retrieve the various > > >>>>>>>>> parameters associated with a given format (such as the subsampling, or the > > >>>>>>>>> bits per pixel), as well as some helpers and utilities to ease the driver > > >>>>>>>>> development. > > >>>>>>>>> > > >>>>>>>>> v4l2, on the other side, doesn't provide such facilities, leaving each > > >>>>>>>>> driver reimplement a subset of the formats parameters for the one supported > > >>>>>>>>> by that particular driver. This leads to a lot of duplication and > > >>>>>>>>> boilerplate code in the v4l2 drivers. > > >>>>>>>>> > > >>>>>>>>> This series tries to address this by moving the DRM format API into lib and > > >>>>>>>>> turning it into a more generic API. In order to do this, we've needed to do > > >>>>>>>>> some preliminary changes on the DRM drivers, then moved the API and finally > > >>>>>>>>> converted a v4l2 driver to give an example of how such library could be > > >>>>>>>>> used. > > >>>>>>>>> > > >>>>>>>>> Let me know what you think, > > >>>>>>>>> Maxime > > >>>>>>>>> > > >>>>>>>>> Changes from RFC: > > >>>>>>>>> - Rebased on next > > >>>>>>>>> - Fixed the various formats mapping > > >>>>>>>>> - Added tags > > >>>>>>>>> - Did most of the formats functions as inline functions > > >>>>>>>>> - Used a consistent prefix for all the utilities functions > > >>>>>>>>> - Fixed the compilation breakages, and did a run of allmodconfig for arm, > > >>>>>>>>> arm64 and x86_64 > > >>>>>>>>> - Fixed out of array bounds warnings in the image_format_info_block_* > > >>>>>>>>> functions > > >>>>>>>>> - Added License and copyright headers on missing files > > >>>>>>>>> > > >>>>>>>>> Maxime Ripard (20): > > >>>>>>>>> drm: Remove users of drm_format_num_planes > > >>>>>>>>> drm: Remove users of drm_format_(horz|vert)_chroma_subsampling > > >>>>>>>>> drm/fourcc: Pass the format_info pointer to drm_format_plane_cpp > > >>>>>>>>> drm/fourcc: Pass the format_info pointer to drm_format_plane_width/height > > >>>>>>>>> drm: Replace instances of drm_format_info by drm_get_format_info > > >>>>>>>>> lib: Add video format information library > > >>>>>>>>> drm/fb: Move from drm_format_info to image_format_info > > >>>>>>>>> drm/malidp: Convert to generic image format library > > >>>>>>>>> drm/client: Convert to generic image format library > > >>>>>>>>> drm/exynos: Convert to generic image format library > > >>>>>>>>> drm/i915: Convert to generic image format library > > >>>>>>>>> drm/ipuv3: Convert to generic image format library > > >>>>>>>>> drm/msm: Convert to generic image format library > > >>>>>>>>> drm/omap: Convert to generic image format library > > >>>>>>>>> drm/rockchip: Convert to generic image format library > > >>>>>>>>> drm/tegra: Convert to generic image format library > > >>>>>>>>> drm/fourcc: Remove old DRM format API > > >>>>>>>>> lib: image-formats: Add v4l2 formats support > > >>>>>>>>> lib: image-formats: Add more functions > > >>>>>>>>> media: sun6i: Convert to the image format API > > >>>>>>>> > > >>>>>>>> In the interest of making myself unpopular: Why move this out of drm? > > >>>>>>>> > > >>>>>>>> We do have a bunch of other such shared helpers already (mostly in > > >>>>>>>> drivers/video) for dt videomode and hdmi infoframes, and I'm not super > > >>>>>>>> sure that's going better than keeping it maintained in drm. > > >>> > > >>> That's a bit of a different situation as both DRM and FBDEV address the > > >>> same features (display output), and FBDEV is deprecared and replaced by > > >>> DRM. > > >>> > > >>> I'm not against maintaining a 4CC library in DRM (adding a third-party > > >>> maintainer would likely create more problems than it would solve), but > > >>> that doesn't mean the library has to live in drivers/gpu/, nor that it > > >>> needs to have the drm_ prefix. I would actually advocate to make it live > > >>> in a neutral directory, with a neutral prefix (kudos to anyone who can > > >>> propose a nice naming convention that would establish a new shared > > >>> ground for image/video-related Linux APIs), and maintained through the > > >>> DRM tree (possibly with extra entries in MAINTAINERS to ensure it > > >>> reaches all the related folks). > > >>> > > >>>>>>>> Plus the uapi is already that you include drm_fourcc.h to get at these, > > >>>>>>>> dropping the drm prefix isn't going to help I think. > > >>>>>>>> > > >>>>>>>> Of course we'd need to make it a separate drm_formats.ko (so that v4l can > > >>>>>>>> use it without dragging in all of drm), and we need to add some fields for > > >>>>>>>> converting to v4l fourcc codes (which are different). But that should be > > >>>>>>>> all possible. And I don't think the drm_ prefix in v4l code is a problem, > > >>>>>>>> it's actually a feature: It makes it really clear that these are the drm > > >>>>>>>> fourcc codes, as allocated in drm_fourcc.h, plus their modifiers, and all > > >>>>>>>> that. That allocation authority is also already baked into various khr/ext > > >>>>>>>> standards, too. > > >>> > > >>> There's one thing that V4L2 has and DRM hasn't for 4CCs: good > > >>> documentation. Look at > > >>> https://linuxtv.org/downloads/v4l-dvb-apis-new/uapi/v4l/pixfmt-packed-rgb.html > > >>> or > > >>> https://linuxtv.org/downloads/v4l-dvb-apis-new/uapi/v4l/yuv-formats.html > > >>> for instance. It's painful to write, painful to read, but defines the > > >>> 4CCs very clearly without ambiguity. I wouldn't be surprised if > > >>> different drivers used the same DRM 4CC for different formats due to the > > >>> lack of detailed documentation. Moving to a shared library for 4CCs > > >>> should also address the documentation side, and any new format added to > > >>> the kernel, whether from the V4L2 side or the DRM side, would be > > >>> required to provide detailed documentation. That would be a great > > >>> improvement for DRM 4CC handling. > > >>> > > >>>>>>> The way I see it, there's a fundamental difference between the UAPI > > >>>>>>> and the kernel. I don't suggest we change anything about the UAPI: the > > >>>>>>> drm formats should keep their prefix, drm_fourcc.h can remain that > > >>>>>>> authority, it's all fine. > > >>>>>>> > > >>>>>>> Internally however, the long term goal is to share the fourcc's > > >>>>>>> between DRM and V4L2 for the same formats. It basically means that of > > >>>>>>> course v4l2 should be using the DRM fourcc when a format exists in DRM > > >>>>>>> and not v4l2, but also that DRM should use v4l2 fourcc when the format > > >>>>>>> exists in v4l2 but not DRM, and that is far more likely, given the > > >>>>>>> already extensive v4l2 formats support. > > >>>>>> > > >>>>>> Uh no. We did look at v4l fourcc extensively when deciding upon a drm > > >>>>>> format identifier space. > > >>>>> > > >>>>> No to what exactly? > > >>>>> > > >>>>>> And a lot of people pushed for the "fourcc is a standard", when > > >>>>>> really it's totally not. > > >>>>> > > >>>>> Even if it's not a standard, having consistency would be a good thing. > > >>>>> > > >>>>> And you said yourself that DRM fourcc is now pretty much an authority > > >>>>> for the fourcc, so it definitely looks like a standard to me. > > >>>> > > >>>> drm fourcc is the authority for drm fourcc codes. Not for any of the > > >>>> others (and there's lots of them). Now it's used in a bunch of other > > >>>> places (khr standards, dri protocols in wayland/X11), but they're > > >>>> still only drm fourcc. There is no overall fourcc standard. > > >>>> > > >>>>>> v4l tends to conflate pixel format with stuff that we tend to encode > > >>>>>> in modifiers a lot more. > > >>>>> > > >>>>> Boris is working on adding the modifiers concept to v4l2, so we're > > >>>>> converging here, and we can totally have a layer in v4l2 to convert > > >>>>> between old v4l2 "format+modifiers" formats, and DRM style formats. > > >>>>> > > >>>>>> There's a bunch of reasons we can't just use v4l, and they're as > > >>>>>> valid as ever: > > >>>>>> > > >>>>>> - We overlap badly in some areas, so even if fourcc codes match, we > > >>>>>> can't use them and need a duplicated DRM_FOURCC code. > > >>>>> > > >>>>> Do yo have an example of one of those areas? > > >>>> > > >>>> I think the rgb stuff was one of the big reasons to not reuse any > > >>>> existing fourcc standard (whether v4l, or another one from e.g. video > > >>>> container formats). We had initial patches to reuse the fourcc that > > >>>> existed, but the end result was really inconsistent, so we ditch that > > >>>> and rolled out a new set of entirely drm specific fourcc codes for > > >>>> rgba. > > >>> > > >>> Could you give one or a couple of examples of V4L2 4CCs that are not > > >>> OCD-compatible ? :-) > > >>> > > >>>>>> - v4l encodes some metadata into the fourcc that we encode elsewhere, > > >>>>>> e.g. offset for planar yuv formats, or tiling mode > > >>>>> > > >>>>> As I was saying, this changes on the v4l2 side, and converging to > > >>>>> what DRM is doing. > > >>>>> > > >>>>>> - drm fourcc code doesn't actually define the drm_format_info > > >>>>>> uniquely, drivers can override that (that's an explicit design > > >>>>>> intent of modifiers, to allow drivers to add another plane for > > >>>>>> e.g. compression information). You'd need to pull that driver > > >>>>>> knowledge into your format library. > > >>> > > >>> That's a mistake in my opinion. We tried that in V4L2 to store metadata > > >>> in a separate plane, and had to go another route eventually as it > > >>> created a very bad mess. > > >> > > >> Just quick clarification in the middle here: This is how the hw works. > > > > > > The hardware takes parameters from a buffer, but it doesn't mandate how > > > that buffer is exposed to userspace :-) Using an extra plane is one > > > option, but other APIs are possible. > > > > I think Daniel Stone explains fairly well why some of our additional > > metadata is included as a plane, while a lot of the other metadata > > involved in rendering/compute the framebuffer isn't. Not really > > anything to add here. > > > > >> It's not metadata that sw ever touches (in general, testcases to make sure > > >> we display these correctly excepted). > > >> > > >> There has been some talking to add maybe a bit more mixed metadata, for > > >> fast-clear colors (which isn't used by any display engine afaik yet). That > > > > > > What are fast-clear colors ? > > > > hw offers enormous amounts of tricks to make glClear O(1), or at least > > close enough. glClear is usually what's done at the start of every > > frame, and clears the entire framebuffer to a uniform color. This is > > achieved usually through 3 pieces: > > - actual framebuffer plane with the pixel data > > - a 2nd plane that (usually, but there's lots of tricks here) contains > > a bit of metadata per cacheline/block/whatever in the framebuffer to > > indicate how/if those pixels are compressed, or whether they are still > > the uniform color supplied through glClear. For actual O(1) this > > metadata is hierarchical, so that a glClear really only sets the > > top-level metadata to "all subordinate blocks still have are the clear > > color". hw tends to have some pretty strong opinions on where it's > > going to look for that 2nd plane. > > - curiously on most hw the actual clear color is supplied separately > > (and our plan is to just stuff it into a 3rd plane) > > Just to clarify, is this needed for display engines ? Does the GPU > render to a 3 planes buffer with glClear()-related data in planes 2 and > 3, with the buffer then being passed to the display engine that knows > how to interpret this, or are those extra planes only needed for GPU > rendering ? I haven't yet seen a display engine that can scan out fast-cleared buffers like these. Everyone can scan out compressed buffers, i.e. rgb + 2nd auxialliary buffer with some compression metadata. But not yet with the 3rd plane. So not relevant for display. But it is very much relevant for uapi/drm_fourcc.h, because we'll need it for userspace interop between different apis (vk, gl, ...) and different processes (client, compositor). And in these standard extension texts we've officially made the kernel's copy of that file the official registry. So these fourcc+modifier codes will end up at least in that namespace (but maybe not in the same library the kernel uses internally). > > >> would generally be written by the cpu (in the gl stack), but again read by > > >> the hw (loaded as indirect state packet most likely, or something like > > >> that). So again hw specific layout, because the hw needs to read it. > > >> > > >> Pure metadata only of interest for the cpu/sw stack has been shot down > > >> completely on the drm side too. > > >> > > >>> There's a tendency in both subsystems to look at the other as a bit of a > > >>> retarded relative, and that's a shame, we have lots to learn from each > > >>> other's mistakes. That wouldn't be too difficult if people started > > >>> talking to each other. > > >>> > > >>> A semi-related comment: DRM also carries other mistakes of its own, I'm > > >>> thinking about DRM_FORMAT_BIG_ENDIAN in particular > > >> > > >> Yeah that one is hilarios, but in practice big endian is dead, except for > > >> a very few server chips, and there I think Gerd's work mostly fixed up > > >> that mess. > > >> > > >>>>> I'm not sure how my patches are changing anything here. This is > > >>>>> litterally the same code, with the functions renamed. > > >>>>> > > >>>>> If drivers want to override that, then yeah, fine, we can let them do > > >>>>> that. Just like any helper this just provides a default that covers > > >>>>> most of the cases. > > >>>>> > > >>>>>> Iow there's no way we can easily adopt v4l fourcc, except if we do > > >>>>>> something like a new addfb flag. > > >>>>> > > >>>>> For the formats that would be described as a modifier, sure. For all > > >>>>> the others (that are not yet supported by DRM), then I don't really > > >>>>> see why not. > > >>>> > > >>>> See above, we tried that initially, didn't pass the ocd tests when > > >>>> reviewing. Maybe the situation is better outside of rbgx/a formats, > > >>>> and I think we do at least try to use the same fourcc codes there when > > >>>> there already is one. But then the details occasionally look > > >>>> different. > > >>>> > > >>>>>>> And given how the current state is a mess in this regard, I'm not too > > >>>>>>> optimistic about keeping the formats in their relevant frameworks. > > >>>>>>> > > >>>>>>> Having a shared library, governed by both, will make this far easier, > > >>>>>>> since it will be easy to discover the formats that are already > > >>>>>>> supported by the other subsystem. > > >>>>>> > > >>>>>> I think a compat library that (tries to, best effort) convert between > > >>>>>> v4l and drm fourcc would make sense. Somewhere in drivers/video, next > > >>>>>> to the conversion functions for videomode <-> drm_display_mode > > >>>>>> perhaps. That should be useful for drivers. > > >>>>> > > >>>>> That's not really what this series is about though. That series is > > >>>>> about sharing the (image|pixels) formats database across everyone so > > >>>>> that everyone can benefit from it. > > >>>> > > >>>> Yeah I know. I still think leaving the drm fourcc with the drm prefix > > >>>> would be good, since there's really no standard here. > > >>>> > > >>>>>> Unifying the formats themselves, and all the associated metadata is > > >>>>>> imo a no-go, and was a pretty conscious decision when we implemented > > >>>>>> drm_fourcc a few years back. > > >>>>>> > > >>>>>>> If we want to keep the current library in DRM, we have two options > > >>>>>>> then: > > >>>>>>> > > >>>>>>> - Support all the v4l2 formats in the DRM library, which is > > >>>>>>> essentially what I'm doing in the last patches. However, that > > >>>>>>> would require to have the v4l2 developpers also reviewing stuff > > >>>>>>> there. And given how busy they are, I cannot really see how that > > >>>>>>> would work. > > >>>>>> > > >>>>>> Well, if we end up with a common library then yes we need cross > > >>>>>> review. There's no way around that. Doesn't matter where exactly that > > >>>>>> library is in the filesystem tree, and adding a special MAINTAINERS > > >>>>>> entry for anything related to fourcc (both drm and v4l) to make sure > > >>>>>> they get cross-posted is easy. No file renaming needed. > > >>>>> > > >>>>> That would create some governing issues as well. For example, if you > > >>>>> ever have a patch from one fourcc addition (that might or might not be > > >>>>> covered by v4l2), will you wait for any v4l2 developper to review it? > > >>>> > > >>>> None of this is fixed by code renaming or locations. Either way we > > >>>> need to figure that out. > > >>>> > > >>>> And yes part of the reasons for not moving this out of drm is that I'm > > >>>> not a fan of boutique trees for small stuff. If sharing means we need > > >>>> to split the drm_fourcc code and library out of drm trees, I'm not > > >>>> sure that's a good idea. We're already super liberal with merging > > >>>> anything through driver trees with acks, and integrating them quickly > > >>>> into drm-next. This would still be workable if v4l sends regular pull > > >>>> requests to drm-next (every 1-2 weeks, like the other big gpu trees > > >>>> do). If we can only sync up once per merge window with a shared > > >>>> boutique tree for formats only, life is going to be painful. > > >>> > > >>> That should be solved by the proposal above, maintaining the shared > > >>> library in the DRM tree. We would need to be careful there, and ideally > > >>> maintain that in a separate branch that could be merged in both DRM and > > >>> V4L2 without having to merge most of the other subsystem's pending > > >>> changes at the same time, but I think it's doable without any big issue. > > >>> > > >>>>> If it's shared code, then it should be shared, and every client > > >>>>> framework put on an equal footing. > > >>>>> > > >>>>>>> - Develop the same library from within v4l2. That is really a poor > > >>>>>>> solution, since the information would be greatly duplicated > > >>>>>>> between the two, and in terms of maintainance, code, and binary > > >>>>>>> size that would be duplicated too. > > >>>>>> > > >>>>>> It's essentially what we decided to do for drm years back. > > >>>>> > > >>>>> And it was probably the right solution back then, but I'm really not > > >>>>> convinced it's still the right thing to do today. > > >>>>> > > >>>>>>> Having it shared allows to easily share, and discover formats from the > > >>>>>>> other subsystem, and to have a single, unique place where this is > > >>>>>>> centralized. > > >>>>>> > > >>>>>> What I think could work as middle ground: > > >>>>>> - Put drm_format stuff into a separate .ko > > >>>>>> - Add a MAINTAINERS entry to make sure all things fourcc are cross > > >>>>>> posted to both drm and v4l lists. Easy on the drm side, since it's all > > >>>>>> separate files. Not sure it's so convenient for v4l uapi. > > >>>>>> - Add a conversion library that tries to best-effort map between drm > > >>>>>> and v4l formats. On the drm side that most likely means you need > > >>>>>> offsets for planes, and modifiers too (since those are implied in some > > >>>>>> v4l fourcc). Emphasis on "best effort" i.e. only support as much as > > >>>>>> the drivers that use this library need. > > >>>>>> - Add drm_fourcc as-needed by these drivers that want to use a unified > > >>>>>> format space. > > >>>>>> > > >>>>>> Forcing this unification on everyone otoh is imo way too much. > > >>>>> > > >>>>> v4l2 is starting to converge with DRM, and we're using the DRM API > > >>>>> pretty much untouched for that library, so I'm not really sure how > > >>>>> anyone is hurt by that unification. > > >>>> > > >>>> It might make sense to regularly pull v4l updates into drm-next then > > >>>> anyway. That would also remove the need to have the format library > > >>>> somewhere else. > > >>> > > >>> Are you saying it should the live in V4L2 ? ;-) > > >> > > >> Maybe a few clarifications on how the drm shared core thing usually works, > > >> and why I'm a sticker here. Bottom reply since I'm not sure where to put > > >> it: > > >> > > >> - Refactorings usually go in through drm-misc (at least since a few > > >> years). > > >> > > >> - Small patches go in through the relevant driver tree (which is often > > >> drm-misc, but not always), with an Ack from drm maintainers. > > >> > > >> - No topic branches, everyone just pushes patches where it's most > > >> convenient. > > >> > > >> We get away with this mess because everyone sends regular pull requests to > > >> drm, where the entire history is baked in and others can backmerge/fast > > >> forward/rebase. Worst case you wait one month (around the merge window, > > >> when drm-next is closed for features), but usually it's just 1-2 weeks. > > >> Plus we tend to have fairly big trees, with good chances that the next > > >> patch series lands in the same tree again and no work at all is needed. > > >> > > >> If we start regularly sharing lots of code with v4l (which seems to be the > > >> long term goal here), then I think we need something equally convenient > > >> for all that. > > >> > > >> We're not going to be able to teach some complicated topic branch scheme > > >> to the 50+ submaintainers/committers we have in drm - a lot much more > > >> basic stuff already takes lots of work to get it to stick. If the proposal > > >> is "to be careful" and "maintain it in a separate branch", I'm not in > > >> favour because I think that just wouldn't work. > > > > > > Why not ? It can be a fast-moving branch that gets merged in drm-misc > > > as often as you want (even at every commit if that's desired). We're > > > dealing with a limited amount of code here, and there's no more reason > > > that V4L2 should pull in drm-misc to get 4CC updates than DRM should > > > pull V4L2 for the same. I have no objection against a 4CC branch being > > > officially maintained under the DRM umbrella, but I think the code > > > should live elsewhere than drivers/gpu/drm/, have a neutral prefix, and > > > not require pulling an entire subsystem in. > > > > I think small boutique trees are a problem themselves, not a solution. > > So if you're creating a new small boutique tree to fix a problem, you > > then have 2. Yes, assuming sufficient expenditure of energy it can be > > made to work, but I'd prefer to make my own life as easy and lazy as > > possible :-) And I think I've been fairly successful at that within > > drivers/gpu at least. > > > > Imo the proper fix is to merge v4l and drm, at a process/maintainer > > level. That would solve both the original issue and the 2ndary one of > > the permanent topic branch. > > Proposals are welcome ;-) I'm already somewhat unpopular at LPC/lkml/kernel-at-large, I don't want to make this worse. That's why I don't want to officially push for anything here myself, nor be in any other way involved in an effort to figure out v4l governance and maintainership rules. What I think is required for efficient collaboration with drm (no matter how we implement that in the details once we're ready for that step) is some kind of group maintainership model. Doesn't need to be as extreme as drm-misc, but I think at least something like the soc tree (while it was still a bit more limited as arm-soc). Otherwise the impendence mismatch between how drm rolls and how v4l rolls is probably way too much. From my understanding v4l is working differently. Also, through sheer inertia of size, I don't think it'll be easier to align drm with the v4l model. So that option is not realistic. -Daniel
Em Mon, 13 May 2019 16:57:19 +0200 Daniel Vetter <daniel@ffwll.ch> escreveu: > > > I think small boutique trees are a problem themselves, not a solution. > > > So if you're creating a new small boutique tree to fix a problem, you > > > then have 2. Yes, assuming sufficient expenditure of energy it can be > > > made to work, but I'd prefer to make my own life as easy and lazy as > > > possible :-) And I think I've been fairly successful at that within > > > drivers/gpu at least. > > > > > > Imo the proper fix is to merge v4l and drm, at a process/maintainer > > > level. That would solve both the original issue and the 2ndary one of > > > the permanent topic branch. > > > > Proposals are welcome ;-) > > I'm already somewhat unpopular at LPC/lkml/kernel-at-large, I don't want > to make this worse. That's why I don't want to officially push for > anything here myself, nor be in any other way involved in an effort to > figure out v4l governance and maintainership rules. > > What I think is required for efficient collaboration with drm (no matter > how we implement that in the details once we're ready for that step) is > some kind of group maintainership model. Doesn't need to be as extreme as > drm-misc, but I think at least something like the soc tree (while it was > still a bit more limited as arm-soc). Otherwise the impendence mismatch > between how drm rolls and how v4l rolls is probably way too much. From my > understanding v4l is working differently. > > Also, through sheer inertia of size, I don't think it'll be easier to > align drm with the v4l model. So that option is not realistic. I don't think it is realistic trying to align V4L2 model to every single other subsystems we use. We have a lot of alignment with alsa, with even increased during this merge window due to some drivers on media sharing media controller resources with usb-audio. We also have lots of alignment with i2c, as we use a lot of stuff there, and from time to time they need to add new features due to our needs. The same is true also for input and for other subsystems and arch/sub-arch trees. That doesn't mean at all that everybody should use the same maintainership model. Each subsystem should use whatever suits best. That's said, after following this thread for a while, I'm starting to convince that it wouldn't even be realistic to have a common fourcc API for both subsystems. The internal requirements from each subsystem are different, and, as it was already pointed in this thread, that's basically why DRM ended by having their own way of doing that. Yet, it would make sense to have at least a single nomenclature for both systems to use, but it could be a simple as what we already do with other common resources, like: Documentation/ioctl/ioctl-number.txt If we keep the fourcc codes there sorted, if one subsystem would add a conflicting code, it would be easy to notice at linux-next. Thanks, Mauro
Hi, DRM comes with an extensive format support to retrieve the various parameters associated with a given format (such as the subsampling, or the bits per pixel), as well as some helpers and utilities to ease the driver development. v4l2, on the other side, doesn't provide such facilities, leaving each driver reimplement a subset of the formats parameters for the one supported by that particular driver. This leads to a lot of duplication and boilerplate code in the v4l2 drivers. This series tries to address this by moving the DRM format API into lib and turning it into a more generic API. In order to do this, we've needed to do some preliminary changes on the DRM drivers, then moved the API and finally converted a v4l2 driver to give an example of how such library could be used. Let me know what you think, Maxime Changes from RFC: - Rebased on next - Fixed the various formats mapping - Added tags - Did most of the formats functions as inline functions - Used a consistent prefix for all the utilities functions - Fixed the compilation breakages, and did a run of allmodconfig for arm, arm64 and x86_64 - Fixed out of array bounds warnings in the image_format_info_block_* functions - Added License and copyright headers on missing files Maxime Ripard (20): drm: Remove users of drm_format_num_planes drm: Remove users of drm_format_(horz|vert)_chroma_subsampling drm/fourcc: Pass the format_info pointer to drm_format_plane_cpp drm/fourcc: Pass the format_info pointer to drm_format_plane_width/height drm: Replace instances of drm_format_info by drm_get_format_info lib: Add video format information library drm/fb: Move from drm_format_info to image_format_info drm/malidp: Convert to generic image format library drm/client: Convert to generic image format library drm/exynos: Convert to generic image format library drm/i915: Convert to generic image format library drm/ipuv3: Convert to generic image format library drm/msm: Convert to generic image format library drm/omap: Convert to generic image format library drm/rockchip: Convert to generic image format library drm/tegra: Convert to generic image format library drm/fourcc: Remove old DRM format API lib: image-formats: Add v4l2 formats support lib: image-formats: Add more functions media: sun6i: Convert to the image format API drivers/gpu/drm/Kconfig | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_fb.c | 4 +- drivers/gpu/drm/arm/malidp_drv.c | 3 +- drivers/gpu/drm/arm/malidp_hw.c | 3 +- drivers/gpu/drm/arm/malidp_mw.c | 2 +- drivers/gpu/drm/arm/malidp_planes.c | 20 +- drivers/gpu/drm/armada/armada_fb.c | 3 +- drivers/gpu/drm/armada/armada_overlay.c | 2 +- drivers/gpu/drm/armada/armada_plane.c | 2 +- drivers/gpu/drm/atmel-hlcdc/atmel_hlcdc_plane.c | 11 +- drivers/gpu/drm/bochs/bochs.h | 4 +- drivers/gpu/drm/bochs/bochs_hw.c | 2 +- drivers/gpu/drm/drm_client.c | 7 +- drivers/gpu/drm/drm_fb_cma_helper.c | 4 +- drivers/gpu/drm/drm_fb_helper.c | 14 +- drivers/gpu/drm/drm_format_helper.c | 4 +- drivers/gpu/drm/drm_fourcc.c | 345 +----- drivers/gpu/drm/drm_framebuffer.c | 10 +- drivers/gpu/drm/drm_gem_framebuffer_helper.c | 4 +- drivers/gpu/drm/exynos/exynos_drm_fb.c | 2 +- drivers/gpu/drm/exynos/exynos_drm_ipp.c | 2 +- drivers/gpu/drm/exynos/exynos_drm_ipp.h | 4 +- drivers/gpu/drm/exynos/exynos_drm_scaler.c | 3 +- drivers/gpu/drm/gma500/framebuffer.c | 4 +- drivers/gpu/drm/i915/i915_drv.h | 6 +- drivers/gpu/drm/i915/intel_display.c | 14 +- drivers/gpu/drm/i915/intel_pm.c | 6 +- drivers/gpu/drm/i915/intel_sprite.c | 3 +- drivers/gpu/drm/imx/ipuv3-plane.c | 19 +- drivers/gpu/drm/mediatek/mtk_drm_fb.c | 8 +- drivers/gpu/drm/meson/meson_overlay.c | 14 +- drivers/gpu/drm/msm/disp/dpu1/dpu_formats.c | 11 +- drivers/gpu/drm/msm/disp/dpu1/dpu_plane.c | 10 +- drivers/gpu/drm/msm/disp/mdp5/mdp5_crtc.c | 4 +- drivers/gpu/drm/msm/disp/mdp5/mdp5_plane.c | 25 +- drivers/gpu/drm/msm/disp/mdp5/mdp5_smp.c | 8 +- drivers/gpu/drm/msm/msm_fb.c | 18 +- drivers/gpu/drm/omapdrm/dss/dispc.c | 9 +- drivers/gpu/drm/omapdrm/omap_fb.c | 15 +- drivers/gpu/drm/radeon/radeon_fb.c | 4 +- drivers/gpu/drm/rockchip/rockchip_drm_fb.c | 17 +- drivers/gpu/drm/rockchip/rockchip_drm_vop.c | 14 +- drivers/gpu/drm/selftests/Makefile | 3 +- drivers/gpu/drm/selftests/drm_modeset_selftests.h | 3 +- drivers/gpu/drm/selftests/test-drm_format.c | 280 +---- drivers/gpu/drm/selftests/test-drm_modeset_common.h | 3 +- drivers/gpu/drm/stm/ltdc.c | 2 +- drivers/gpu/drm/sun4i/sun4i_backend.c | 6 +- drivers/gpu/drm/sun4i/sun4i_frontend.c | 18 +- drivers/gpu/drm/sun4i/sun8i_vi_layer.c | 4 +- drivers/gpu/drm/sun4i/sun8i_vi_scaler.c | 6 +- drivers/gpu/drm/sun4i/sun8i_vi_scaler.h | 5 +- drivers/gpu/drm/tegra/fb.c | 14 +- drivers/gpu/drm/tegra/plane.c | 4 +- drivers/gpu/drm/vc4/vc4_plane.c | 15 +- drivers/gpu/drm/zte/zx_plane.c | 6 +- drivers/gpu/ipu-v3/ipu-pre.c | 3 +- drivers/gpu/ipu-v3/ipu-prg.c | 3 +- drivers/media/platform/sunxi/sun6i-csi/sun6i_csi.c | 89 +- drivers/media/platform/sunxi/sun6i-csi/sun6i_csi.h | 49 +- drivers/media/platform/sunxi/sun6i-csi/sun6i_csi_reg.h | 6 +- drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c | 11 +- include/drm/drm_fourcc.h | 219 +--- include/drm/drm_framebuffer.h | 4 +- include/drm/drm_mode_config.h | 5 +- include/linux/image-formats.h | 438 ++++++- lib/Kconfig | 7 +- lib/Makefile | 3 +- lib/image-formats-selftests.c | 325 +++++- lib/image-formats.c | 717 ++++++++++- 70 files changed, 1740 insertions(+), 1174 deletions(-) delete mode 100644 drivers/gpu/drm/selftests/test-drm_format.c create mode 100644 include/linux/image-formats.h create mode 100644 lib/image-formats-selftests.c create mode 100644 lib/image-formats.c base-commit: de3c659c83ce88d495f2a36397071fdd0a337345