| Message ID | 20211202041627.291625-1-aford173@gmail.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | arm64: imx8mm: Enable Hantro VPUs | expand |
On Wed, Dec 1, 2021 at 10:17 PM Adam Ford <aford173@gmail.com> wrote: > > The i.MX8M has two Hantro video decoders, called G1 and G2 which appear > to be related to the video decoders used on the i.MX8MQ, but because of > how the Mini handles the power domains, the VPU driver does not need to > handle all the functions, so a new compatible flag is required. > > V3 is rebased from git://linuxtv.org/hverkuil/media_tree.git for-v5.17c > This branch has support for VP9. > > I set cma=512M, but this may not be enough memory as some tests appeard to run out of memory > > V3 of this series has several changes: > > Update imx8m_vpu_hw to add missing 'reg' reference names for G2 and include references to VP9 > Update device tree to remove IMX8MQ_VPU_RESET, remove some duplicate vpu clock parenting > Fix missing reg-names from vpu_g2 node. > Apply patch [1] to manage the power domains powering down. > [1] - https://lore.kernel.org/linux-arm-kernel/20211016210547.171717-1-marex@denx.de/t/ > > With the above, the following Fluster scores are produced: > > G1: > ./fluster.py run -dGStreamer-H.264-V4L2SL-Gst1.0 > Ran 90/135 tests successfully in 74.406 secs > > ./fluster.py run -d GStreamer-VP8-V4L2SL-Gst1.0 > Ran 55/61 tests successfully in 8.080 secs > > G2: > ./fluster.py run -d GStreamer-VP9-V4L2SL-Gst1.0 > Ran 127/303 tests successfully in 203.873 secs > > Fluster and G-Streamer were both built from their respective git repos using their respective master/main branches. > I should note, that both interrupts appear to be triggering. # cat /proc/interrupts |grep codec 57: 13442 0 0 0 GICv3 39 Level 38300000.video-codec 58: 7815 0 0 0 GICv3 40 Level 38310000.video-codec > > Adam Ford (2): > media: hantro: Add support for i.MX8M Mini > arm64: dts: imx8mm: Enable VPU-G1 and VPU-G2 > > arch/arm64/boot/dts/freescale/imx8mm.dtsi | 41 +++++++++++++ > drivers/staging/media/hantro/hantro_drv.c | 2 + > drivers/staging/media/hantro/hantro_hw.h | 2 + > drivers/staging/media/hantro/imx8m_vpu_hw.c | 65 +++++++++++++++++++++ > 4 files changed, 110 insertions(+) > > -- > 2.32.0 >
On Thu, Dec 2, 2021 at 4:29 AM Adam Ford <aford173@gmail.com> wrote: > > On Wed, Dec 1, 2021 at 10:17 PM Adam Ford <aford173@gmail.com> wrote: > > > > The i.MX8M has two Hantro video decoders, called G1 and G2 which appear > > to be related to the video decoders used on the i.MX8MQ, but because of > > how the Mini handles the power domains, the VPU driver does not need to > > handle all the functions, so a new compatible flag is required. > > > > V3 is rebased from git://linuxtv.org/hverkuil/media_tree.git for-v5.17c > > This branch has support for VP9. > > > > I set cma=512M, but this may not be enough memory as some tests appeard to run out of memory > > > > V3 of this series has several changes: > > > > Update imx8m_vpu_hw to add missing 'reg' reference names for G2 and include references to VP9 > > Update device tree to remove IMX8MQ_VPU_RESET, remove some duplicate vpu clock parenting > > Fix missing reg-names from vpu_g2 node. > > Apply patch [1] to manage the power domains powering down. > > [1] - https://lore.kernel.org/linux-arm-kernel/20211016210547.171717-1-marex@denx.de/t/ > > > > With the above, the following Fluster scores are produced: > > > > G1: > > ./fluster.py run -dGStreamer-H.264-V4L2SL-Gst1.0 > > Ran 90/135 tests successfully in 74.406 secs > > > > ./fluster.py run -d GStreamer-VP8-V4L2SL-Gst1.0 > > Ran 55/61 tests successfully in 8.080 secs > > > > G2: > > ./fluster.py run -d GStreamer-VP9-V4L2SL-Gst1.0 > > Ran 127/303 tests successfully in 203.873 secs > > > > Fluster and G-Streamer were both built from their respective git repos using their respective master/main branches. > > > > I should note, that both interrupts appear to be triggering. > > # cat /proc/interrupts |grep codec > 57: 13442 0 0 0 GICv3 39 Level > 38300000.video-codec > 58: 7815 0 0 0 GICv3 40 Level > 38310000.video-codec > Adam, On another thread you had let me know that you also removed the reset from the pgc_vpumix power domain which does appear to resolve the hang: diff --git a/arch/arm64/boot/dts/freescale/imx8mm.dtsi b/arch/arm64/boot/dts/freescale/imx8mm.dtsi index eb9dcd9d1a31..31710af544dc 100644 --- a/arch/arm64/boot/dts/freescale/imx8mm.dtsi +++ b/arch/arm64/boot/dts/freescale/imx8mm.dtsi @@ -681,7 +681,6 @@ clocks = <&clk IMX8MM_CLK_VPU_DEC_ROOT>; assigned-clocks = <&clk IMX8MM_CLK_VPU_BUS>; assigned-clock-parents = <&clk IMX8MM_SYS_PLL1_800M>; - resets = <&src IMX8MQ_RESET_VPU_RESET>; }; pgc_vpu_g1: power-domain@7 { That would make such a patch have a 'Fixes commit d39d4bb15310 ("arm64: dts: imx8mm: add GPC node")' but of course that vpu domain isn't active until your series so I'm not sure if we should send this separate or squash it with "arm64: dts: imx8mm: Enable VPU-G1 and VPU-G2". I'm also not clear if removing the reset requires some further discussion with Lucas. Best regards, Tim
On Thu, Dec 2, 2021 at 12:54 PM Tim Harvey <tharvey@gateworks.com> wrote: > > On Thu, Dec 2, 2021 at 4:29 AM Adam Ford <aford173@gmail.com> wrote: > > > > On Wed, Dec 1, 2021 at 10:17 PM Adam Ford <aford173@gmail.com> wrote: > > > > > > The i.MX8M has two Hantro video decoders, called G1 and G2 which appear > > > to be related to the video decoders used on the i.MX8MQ, but because of > > > how the Mini handles the power domains, the VPU driver does not need to > > > handle all the functions, so a new compatible flag is required. > > > > > > V3 is rebased from git://linuxtv.org/hverkuil/media_tree.git for-v5.17c > > > This branch has support for VP9. > > > > > > I set cma=512M, but this may not be enough memory as some tests appeard to run out of memory > > > > > > V3 of this series has several changes: > > > > > > Update imx8m_vpu_hw to add missing 'reg' reference names for G2 and include references to VP9 > > > Update device tree to remove IMX8MQ_VPU_RESET, remove some duplicate vpu clock parenting > > > Fix missing reg-names from vpu_g2 node. > > > Apply patch [1] to manage the power domains powering down. > > > [1] - https://lore.kernel.org/linux-arm-kernel/20211016210547.171717-1-marex@denx.de/t/ > > > > > > With the above, the following Fluster scores are produced: > > > > > > G1: > > > ./fluster.py run -dGStreamer-H.264-V4L2SL-Gst1.0 > > > Ran 90/135 tests successfully in 74.406 secs > > > > > > ./fluster.py run -d GStreamer-VP8-V4L2SL-Gst1.0 > > > Ran 55/61 tests successfully in 8.080 secs > > > > > > G2: > > > ./fluster.py run -d GStreamer-VP9-V4L2SL-Gst1.0 > > > Ran 127/303 tests successfully in 203.873 secs > > > > > > Fluster and G-Streamer were both built from their respective git repos using their respective master/main branches. > > > > > > > I should note, that both interrupts appear to be triggering. > > > > # cat /proc/interrupts |grep codec > > 57: 13442 0 0 0 GICv3 39 Level > > 38300000.video-codec > > 58: 7815 0 0 0 GICv3 40 Level > > 38310000.video-codec > > > > Adam, > > On another thread you had let me know that you also removed the reset > from the pgc_vpumix power domain which does appear to resolve the > hang: > > diff --git a/arch/arm64/boot/dts/freescale/imx8mm.dtsi > b/arch/arm64/boot/dts/freescale/imx8mm.dtsi > index eb9dcd9d1a31..31710af544dc 100644 > --- a/arch/arm64/boot/dts/freescale/imx8mm.dtsi > +++ b/arch/arm64/boot/dts/freescale/imx8mm.dtsi > @@ -681,7 +681,6 @@ > clocks = <&clk > IMX8MM_CLK_VPU_DEC_ROOT>; > assigned-clocks = > <&clk IMX8MM_CLK_VPU_BUS>; > assigned-clock-parents > = <&clk IMX8MM_SYS_PLL1_800M>; > - resets = <&src > IMX8MQ_RESET_VPU_RESET>; > }; > > pgc_vpu_g1: power-domain@7 { > > That would make such a patch have a 'Fixes commit d39d4bb15310 > ("arm64: dts: imx8mm: add GPC node")' but of course that vpu domain > isn't active until your series so I'm not sure if we should send this > separate or squash it with "arm64: dts: imx8mm: Enable VPU-G1 and > VPU-G2". I'm also not clear if removing the reset requires some > further discussion with Lucas. Unless there is objection from Lucas, I'll likely make it the first patch in the series marking it with a fixes tag so it gets backported, then the rest of the series would be adding the bindings, update the driver and adding the G1 and G2 nodes. adam > > Best regards, > > Tim
On Thu, Dec 2, 2021 at 1:00 PM Adam Ford <aford173@gmail.com> wrote: > > On Thu, Dec 2, 2021 at 12:54 PM Tim Harvey <tharvey@gateworks.com> wrote: > > > > On Thu, Dec 2, 2021 at 4:29 AM Adam Ford <aford173@gmail.com> wrote: > > > > > > On Wed, Dec 1, 2021 at 10:17 PM Adam Ford <aford173@gmail.com> wrote: > > > > > > > > The i.MX8M has two Hantro video decoders, called G1 and G2 which appear > > > > to be related to the video decoders used on the i.MX8MQ, but because of > > > > how the Mini handles the power domains, the VPU driver does not need to > > > > handle all the functions, so a new compatible flag is required. > > > > > > > > V3 is rebased from git://linuxtv.org/hverkuil/media_tree.git for-v5.17c > > > > This branch has support for VP9. > > > > > > > > I set cma=512M, but this may not be enough memory as some tests appeard to run out of memory > > > > > > > > V3 of this series has several changes: > > > > > > > > Update imx8m_vpu_hw to add missing 'reg' reference names for G2 and include references to VP9 > > > > Update device tree to remove IMX8MQ_VPU_RESET, remove some duplicate vpu clock parenting > > > > Fix missing reg-names from vpu_g2 node. > > > > Apply patch [1] to manage the power domains powering down. > > > > [1] - https://lore.kernel.org/linux-arm-kernel/20211016210547.171717-1-marex@denx.de/t/ > > > > > > > > With the above, the following Fluster scores are produced: > > > > > > > > G1: > > > > ./fluster.py run -dGStreamer-H.264-V4L2SL-Gst1.0 > > > > Ran 90/135 tests successfully in 74.406 secs > > > > > > > > ./fluster.py run -d GStreamer-VP8-V4L2SL-Gst1.0 > > > > Ran 55/61 tests successfully in 8.080 secs > > > > > > > > G2: > > > > ./fluster.py run -d GStreamer-VP9-V4L2SL-Gst1.0 > > > > Ran 127/303 tests successfully in 203.873 secs > > > > > > > > Fluster and G-Streamer were both built from their respective git repos using their respective master/main branches. > > > > > > > > > > I should note, that both interrupts appear to be triggering. > > > > > > # cat /proc/interrupts |grep codec > > > 57: 13442 0 0 0 GICv3 39 Level > > > 38300000.video-codec > > > 58: 7815 0 0 0 GICv3 40 Level > > > 38310000.video-codec > > > > > > > Adam, > > > > On another thread you had let me know that you also removed the reset > > from the pgc_vpumix power domain which does appear to resolve the > > hang: > > > > diff --git a/arch/arm64/boot/dts/freescale/imx8mm.dtsi > > b/arch/arm64/boot/dts/freescale/imx8mm.dtsi > > index eb9dcd9d1a31..31710af544dc 100644 > > --- a/arch/arm64/boot/dts/freescale/imx8mm.dtsi > > +++ b/arch/arm64/boot/dts/freescale/imx8mm.dtsi > > @@ -681,7 +681,6 @@ > > clocks = <&clk > > IMX8MM_CLK_VPU_DEC_ROOT>; > > assigned-clocks = > > <&clk IMX8MM_CLK_VPU_BUS>; > > assigned-clock-parents > > = <&clk IMX8MM_SYS_PLL1_800M>; > > - resets = <&src > > IMX8MQ_RESET_VPU_RESET>; > > }; > > > > pgc_vpu_g1: power-domain@7 { > > > > That would make such a patch have a 'Fixes commit d39d4bb15310 > > ("arm64: dts: imx8mm: add GPC node")' but of course that vpu domain > > isn't active until your series so I'm not sure if we should send this > > separate or squash it with "arm64: dts: imx8mm: Enable VPU-G1 and > > VPU-G2". I'm also not clear if removing the reset requires some > > further discussion with Lucas. > > Unless there is objection from Lucas, I'll likely make it the first > patch in the series marking it with a fixes tag so it gets backported, > then the rest of the series would be adding the bindings, update the > driver and adding the G1 and G2 nodes. > Adam, I've also gotten decode+display working for vp8/h264 using this series and gstreamer-1.19.3 (although I have to use software colorspace conversion) # source: vp8 software encode on x86 gst-launch-1.0 -v videotestsrc ! vp8enc ! rtpvp8pay ! udpsink host=172.24.33.15 port=9001 # sink: vp8 hardware decode on imx8mm gst-launch-1.0 udpsrc port=9001 caps = 'application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)VP8, payload=(int)96, ssrc=(uint)3363940374, timestamp-offset=(uint)3739685909, seqnum-offset=(uint)28161, a-framerate=(string)30' ! rtpvp8depay ! v4l2slvp8dec ! videoconvert ! kmssink # source: h264 software encode on x86 gst-launch-1.0 -v videotestsrc ! video/x-raw,width=800,height=480 ! x264enc ! rtph264pay ! udpsink host=172.24.33.15 port=9001 # sink: h264 hardware decode on imx8mm gst-launch-1.0 udpsrc port=9001 caps = 'application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, packetization-mode=(string)1, profile-level-id=(string)64001f, sprop-parameter-sets=(string)"Z2QAH6zZQMg9sBagwCC0oAAAAwAgAAAHkeMGMsA\=\,aOvssiw\=", payload=(int)96, ssrc=(uint)2753453329, timestamp-offset=(uint)3593065282, seqnum-offset=(uint)12297, a-framerate=(string)30' ! rtph264depay ! v4l2slh264dec ! videoconvert ! kmssink # source: vp9 software encode on x86 gst-launch-1.0 -v videotestsrc ! video/x-raw,width=800,height=480 ! vp9enc ! rtpvp9pay ! udpsink host=172.24.33.15 port=9001 # sink: vp9 hardware decode on imx8mm gst-launch-1.0 udpsrc port=9001 caps = 'application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)VP9, payload=(int)96, ssrc=(uint)2246741422, timestamp-offset=(uint)3441735424, seqnum-offset=(uint)30250, a-framerate=(string)30' ! rtpvp9depay ! v4l2slvp9dec ! fakesink ^^^ this fails with no-negotiated Note I have to use videoconvert because v4l2slvp8dev src is NV12/YUY2/NV12_32L32 and from testing only BGRx appears compatible with kmssink (even though gst-inspect kmssink says it can sink NV12/YUY2). With the 800x480 resolution of my display the CPU overhead of software colorspcae conversion with videoconvert only about 9% I haven't yet gotten vp9 decode+display working yet as 'rtpvp9depay ! v4l2slvp9dec ! fakesink' does not negotiate and it might be because my vp9enc source is on an old gstreamer 1.16. When you post the next series please add: Tested-By: Tim Harvey <tharvey@gateworks.com>
Le mercredi 01 décembre 2021 à 22:16 -0600, Adam Ford a écrit : > The i.MX8M has two Hantro video decoders, called G1 and G2 which appear > to be related to the video decoders used on the i.MX8MQ, but because of > how the Mini handles the power domains, the VPU driver does not need to > handle all the functions, so a new compatible flag is required. > > V3 is rebased from git://linuxtv.org/hverkuil/media_tree.git for-v5.17c > This branch has support for VP9. > > I set cma=512M, but this may not be enough memory as some tests appeard to run out of memory Consider passing -j1 to fluster and using patience, this will spare some memory. You can also increase the timeout (like -t 60), that often helps when CMA is involved (due to software conversion happening). Thanks for the serious testing, this is really appreciated that your submission really reflects the state. > > V3 of this series has several changes: > > Update imx8m_vpu_hw to add missing 'reg' reference names for G2 and include references to VP9 > Update device tree to remove IMX8MQ_VPU_RESET, remove some duplicate vpu clock parenting > Fix missing reg-names from vpu_g2 node. > Apply patch [1] to manage the power domains powering down. > [1] - https://lore.kernel.org/linux-arm-kernel/20211016210547.171717-1-marex@denx.de/t/ > > With the above, the following Fluster scores are produced: > > G1: > ./fluster.py run -dGStreamer-H.264-V4L2SL-Gst1.0 > Ran 90/135 tests successfully in 74.406 secs This matches my results on imx8mq. > > ./fluster.py run -d GStreamer-VP8-V4L2SL-Gst1.0 > Ran 55/61 tests successfully in 8.080 secs Idem. > > G2: > ./fluster.py run -d GStreamer-VP9-V4L2SL-Gst1.0 > Ran 127/303 tests successfully in 203.873 secs Benjamin reports 129, it could be flux. This is still a bit early, hence the low score. But most streams do playback normally. > > Fluster and G-Streamer were both built from their respective git repos using their respective master/main branches. That gave me an idea, I'll try and print the GStreamer version/commit inside fluster along with fluster own commit (ideally in the short summary). > > > Adam Ford (2): > media: hantro: Add support for i.MX8M Mini > arm64: dts: imx8mm: Enable VPU-G1 and VPU-G2 > > arch/arm64/boot/dts/freescale/imx8mm.dtsi | 41 +++++++++++++ > drivers/staging/media/hantro/hantro_drv.c | 2 + > drivers/staging/media/hantro/hantro_hw.h | 2 + > drivers/staging/media/hantro/imx8m_vpu_hw.c | 65 +++++++++++++++++++++ > 4 files changed, 110 insertions(+) >
Am Donnerstag, dem 02.12.2021 um 10:54 -0800 schrieb Tim Harvey: > On Thu, Dec 2, 2021 at 4:29 AM Adam Ford <aford173@gmail.com> wrote: > > > > On Wed, Dec 1, 2021 at 10:17 PM Adam Ford <aford173@gmail.com> wrote: > > > > > > The i.MX8M has two Hantro video decoders, called G1 and G2 which appear > > > to be related to the video decoders used on the i.MX8MQ, but because of > > > how the Mini handles the power domains, the VPU driver does not need to > > > handle all the functions, so a new compatible flag is required. > > > > > > V3 is rebased from git://linuxtv.org/hverkuil/media_tree.git for-v5.17c > > > This branch has support for VP9. > > > > > > I set cma=512M, but this may not be enough memory as some tests appeard to run out of memory > > > > > > V3 of this series has several changes: > > > > > > Update imx8m_vpu_hw to add missing 'reg' reference names for G2 and include references to VP9 > > > Update device tree to remove IMX8MQ_VPU_RESET, remove some duplicate vpu clock parenting > > > Fix missing reg-names from vpu_g2 node. > > > Apply patch [1] to manage the power domains powering down. > > > [1] - https://lore.kernel.org/linux-arm-kernel/20211016210547.171717-1-marex@denx.de/t/ > > > > > > With the above, the following Fluster scores are produced: > > > > > > G1: > > > ./fluster.py run -dGStreamer-H.264-V4L2SL-Gst1.0 > > > Ran 90/135 tests successfully in 74.406 secs > > > > > > ./fluster.py run -d GStreamer-VP8-V4L2SL-Gst1.0 > > > Ran 55/61 tests successfully in 8.080 secs > > > > > > G2: > > > ./fluster.py run -d GStreamer-VP9-V4L2SL-Gst1.0 > > > Ran 127/303 tests successfully in 203.873 secs > > > > > > Fluster and G-Streamer were both built from their respective git repos using their respective master/main branches. > > > > > > > I should note, that both interrupts appear to be triggering. > > > > # cat /proc/interrupts |grep codec > > 57: 13442 0 0 0 GICv3 39 Level > > 38300000.video-codec > > 58: 7815 0 0 0 GICv3 40 Level > > 38310000.video-codec > > > > Adam, > > On another thread you had let me know that you also removed the reset > from the pgc_vpumix power domain which does appear to resolve the > hang: > > diff --git a/arch/arm64/boot/dts/freescale/imx8mm.dtsi > b/arch/arm64/boot/dts/freescale/imx8mm.dtsi > index eb9dcd9d1a31..31710af544dc 100644 > --- a/arch/arm64/boot/dts/freescale/imx8mm.dtsi > +++ b/arch/arm64/boot/dts/freescale/imx8mm.dtsi > @@ -681,7 +681,6 @@ > clocks = <&clk > IMX8MM_CLK_VPU_DEC_ROOT>; > assigned-clocks = > <&clk IMX8MM_CLK_VPU_BUS>; > assigned-clock-parents > = <&clk IMX8MM_SYS_PLL1_800M>; > - resets = <&src > IMX8MQ_RESET_VPU_RESET>; > }; > > pgc_vpu_g1: power-domain@7 { > > That would make such a patch have a 'Fixes commit d39d4bb15310 > ("arm64: dts: imx8mm: add GPC node")' but of course that vpu domain > isn't active until your series so I'm not sure if we should send this > separate or squash it with "arm64: dts: imx8mm: Enable VPU-G1 and > VPU-G2". I'm also not clear if removing the reset requires some > further discussion with Lucas. > I'm fine with removing the reset when it fixes things. In normal operation the reset should already be triggered by the GPC itself via a hardware mechanism. We know that this doesn't work for the GPU reset on the i.MX8MM, so we have the ability for the driver to handle it by poking the SRC explicitly. Adding the reset to the VPU DT description wasn't done because I know that we need it, but more of a "I know that things are broken with the GPU domain, so better be safe than sorry with the VPU domain". My line of thought clearly was that it may not be needed, but it may prevent some issues in the long run. If it is _causing_ issues however, there is no need to discuss anything, just get rid of it. Regards, Lucas