| Message ID | 20240807082444.21280-1-yunfei.dong@mediatek.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | media: mediatek: vcodec: fix v4l2_ctrl_request_complete fail | expand |
Hi Yunfei, Le mercredi 07 août 2024 à 16:24 +0800, Yunfei Dong a écrit : > v4l2_m2m_buf_done is called in lat work queue, v4l2_ctrl_request_complete > is called in core queue. The request status of output queue will be set to > MEDIA_REQUEST_STATE_COMPLETE when v4l2_m2m_buf_done is called, leading to > output queue request complete fail. Must move v4l2_ctrl_request_complete > in front of v4l2_m2m_buf_done. Sebastian and I have analyzed further the issue and the description here does not seem to match. What happens is that in Stateless decoding, you have to set header controls out-of-request to negotiate the format at first. With VP9 notably, the header control is the only control there is. Chromium will optimize out this and only attach a bitstream buffer to the request. So when the buffer is mark to done, it is the last object in the request, which implicitly mark the request as complete. When v4l2_ctrl_request_complete() is later called, the control code detect that there is no controls in the request. It then creates an empty control, but attaching an object to a completed request is not allowed. > > Patch 1 setting request complete before buffer done > Patch 2 change flush decode order when stream off > Patch 3 flush decoder before stream off > Patch 4 using input information to get vb2 buffer > Patch 5 store source vb2 buffer > Patch 6 replace v4l2_m2m_next_src_buf with v4l2_m2m_src_buf_remove > Patch 7 remove media request checking I will give a some testing soon. Can you clarify on if the LAT and the CORE still runs in parallel after this change ? Nicolas > > --- > compared with v3: > - fix flush decoder issue when userspace stream off capture queue firstly > - fluster test result same with v3 > > compared with v2: > - add patch 5/6/7 to fix decode again issue > - add fluster test result with mt8195 platform(same with no changed): > 1> ./fluster.py run -d GStreamer-VP8-V4L2SL-Gst1.0 -j1 -t 90 > VP8-TEST-VECTORS 59/61 > 2> ./fluster.py run -d GStreamer-VP9-V4L2SL-Gst1.0 -j1 -t 90 > VP9-TEST-VECTORS 276/305 > 3> ./fluster.py run -d GStreamer-AV1-V4L2SL-Gst1.0 -j1 -t 90 > AV1-TEST-VECTORS 237/239 > 4> ./fluster.py run -d GStreamer-H.264-V4L2SL-Gst1.0 -j1 -t 90 > JVT-AVC_V1 95/135 > 5> ./fluster.py run -d GStreamer-H.265-V4L2SL-Gst1.0 -j1 -t 90 > JCT-VC-HEVC_V1 142/147 > > compared with v1: > - add patch 2/3/4 to fix timing issue. > --- > Yunfei Dong (7): > media: mediatek: vcodec: setting request complete before buffer done > media: mediatek: vcodec: change flush decode order when stream off > media: mediatek: vcodec: flush decoder before stream off > media: mediatek: vcodec: using input information to get vb2 buffer > media: mediatek: vcodec: store source vb2 buffer > media: mediatek: vcodec: replace v4l2_m2m_next_src_buf with > v4l2_m2m_src_buf_remove > media: mediatek: vcodec: remove media request checking > > .../mediatek/vcodec/decoder/mtk_vcodec_dec.c | 44 ++++++++--------- > .../vcodec/decoder/mtk_vcodec_dec_drv.h | 4 +- > .../vcodec/decoder/mtk_vcodec_dec_stateless.c | 48 ++++++++++++++----- > .../vcodec/decoder/vdec/vdec_av1_req_lat_if.c | 18 +++---- > .../decoder/vdec/vdec_h264_req_multi_if.c | 4 +- > .../decoder/vdec/vdec_hevc_req_multi_if.c | 4 +- > .../vcodec/decoder/vdec/vdec_vp9_req_lat_if.c | 19 ++++---- > .../mediatek/vcodec/decoder/vdec_msg_queue.h | 4 +- > 8 files changed, 85 insertions(+), 60 deletions(-) >
Hi Nicolas, Thanks for your advice. On Wed, 2024-08-07 at 09:08 -0400, Nicolas Dufresne wrote: > > > Hi Yunfei, > > Le mercredi 07 août 2024 à 16:24 +0800, Yunfei Dong a écrit : > > v4l2_m2m_buf_done is called in lat work queue, > > v4l2_ctrl_request_complete > > is called in core queue. The request status of output queue will be > > set to > > MEDIA_REQUEST_STATE_COMPLETE when v4l2_m2m_buf_done is called, > > leading to > > output queue request complete fail. Must move > > v4l2_ctrl_request_complete > > in front of v4l2_m2m_buf_done. > > Sebastian and I have analyzed further the issue and the description > here does > not seem to match. What happens is that in Stateless decoding, you > have to set > header controls out-of-request to negotiate the format at first. > > With VP9 notably, the header control is the only control there is. > Chromium will > optimize out this and only attach a bitstream buffer to the request. > So when the > buffer is mark to done, it is the last object in the request, which > implicitly > mark the request as complete. > > When v4l2_ctrl_request_complete() is later called, the control code > detect that > there is no controls in the request. It then creates an empty > control, but > attaching an object to a completed request is not allowed. > Whether I can write the commit message as below: "User space will attach the syntaxes and bit-stream buffer to a media request for stateless decoding, and the syntax controls are the only v4l2 control request. The request will be marked to complete status when the buffer is set to done, then request object will be cleaned from media request. When v4l2_ctrl_request_complete() is later called, the control request detect that there is no controls in the request object. It then creates an empty control request object, but attaching an object to a completed request is not allowed." > > Patch 1 setting request complete before buffer done > > Patch 2 change flush decode order when stream off > > Patch 3 flush decoder before stream off > > Patch 4 using input information to get vb2 buffer > > Patch 5 store source vb2 buffer > > Patch 6 replace v4l2_m2m_next_src_buf with v4l2_m2m_src_buf_remove > > Patch 7 remove media request checking > > I will give a some testing soon. Can you clarify on if the LAT and > the CORE Have you already helped to do the fluster test? > still runs in parallel after this change ? > Yes, the driver can work in parallel. Could you please help to review other patches? > Nicolas > Best Regards, Yunfei Dong > > > > --- > > compared with v3: > > - fix flush decoder issue when userspace stream off capture queue > > firstly > > - fluster test result same with v3 > > > > compared with v2: > > - add patch 5/6/7 to fix decode again issue > > - add fluster test result with mt8195 platform(same with no > > changed): > > 1> ./fluster.py run -d GStreamer-VP8-V4L2SL-Gst1.0 -j1 -t 90 > > VP8-TEST-VECTORS 59/61 > > 2> ./fluster.py run -d GStreamer-VP9-V4L2SL-Gst1.0 -j1 -t 90 > > VP9-TEST-VECTORS 276/305 > > 3> ./fluster.py run -d GStreamer-AV1-V4L2SL-Gst1.0 -j1 -t 90 > > AV1-TEST-VECTORS 237/239 > > 4> ./fluster.py run -d GStreamer-H.264-V4L2SL-Gst1.0 -j1 -t 90 > > JVT-AVC_V1 95/135 > > 5> ./fluster.py run -d GStreamer-H.265-V4L2SL-Gst1.0 -j1 -t 90 > > JCT-VC-HEVC_V1 142/147 > > > > compared with v1: > > - add patch 2/3/4 to fix timing issue. > > --- > > Yunfei Dong (7): > > media: mediatek: vcodec: setting request complete before buffer > > done > > media: mediatek: vcodec: change flush decode order when stream > > off > > media: mediatek: vcodec: flush decoder before stream off > > media: mediatek: vcodec: using input information to get vb2 > > buffer > > media: mediatek: vcodec: store source vb2 buffer > > media: mediatek: vcodec: replace v4l2_m2m_next_src_buf with > > v4l2_m2m_src_buf_remove > > media: mediatek: vcodec: remove media request checking > > > > .../mediatek/vcodec/decoder/mtk_vcodec_dec.c | 44 ++++++++------- > > -- > > .../vcodec/decoder/mtk_vcodec_dec_drv.h | 4 +- > > .../vcodec/decoder/mtk_vcodec_dec_stateless.c | 48 ++++++++++++++- > > ---- > > .../vcodec/decoder/vdec/vdec_av1_req_lat_if.c | 18 +++---- > > .../decoder/vdec/vdec_h264_req_multi_if.c | 4 +- > > .../decoder/vdec/vdec_hevc_req_multi_if.c | 4 +- > > .../vcodec/decoder/vdec/vdec_vp9_req_lat_if.c | 19 ++++---- > > .../mediatek/vcodec/decoder/vdec_msg_queue.h | 4 +- > > 8 files changed, 85 insertions(+), 60 deletions(-) > > > >
Hey Yunfei, given this new series by Hans: https://patchwork.linuxtv.org/project/linux-media/list/?series=13427 we might actually be able to find a more performant solution of the problem, I'll work on that a bit and give you feedback. Regards, Sebastian On 07.08.2024 16:24, Yunfei Dong wrote: >v4l2_m2m_buf_done is called in lat work queue, v4l2_ctrl_request_complete >is called in core queue. The request status of output queue will be set to >MEDIA_REQUEST_STATE_COMPLETE when v4l2_m2m_buf_done is called, leading to >output queue request complete fail. Must move v4l2_ctrl_request_complete >in front of v4l2_m2m_buf_done. > >Patch 1 setting request complete before buffer done >Patch 2 change flush decode order when stream off >Patch 3 flush decoder before stream off >Patch 4 using input information to get vb2 buffer >Patch 5 store source vb2 buffer >Patch 6 replace v4l2_m2m_next_src_buf with v4l2_m2m_src_buf_remove >Patch 7 remove media request checking > >--- >compared with v3: >- fix flush decoder issue when userspace stream off capture queue firstly >- fluster test result same with v3 > >compared with v2: >- add patch 5/6/7 to fix decode again issue >- add fluster test result with mt8195 platform(same with no changed): > 1> ./fluster.py run -d GStreamer-VP8-V4L2SL-Gst1.0 -j1 -t 90 > VP8-TEST-VECTORS 59/61 > 2> ./fluster.py run -d GStreamer-VP9-V4L2SL-Gst1.0 -j1 -t 90 > VP9-TEST-VECTORS 276/305 > 3> ./fluster.py run -d GStreamer-AV1-V4L2SL-Gst1.0 -j1 -t 90 > AV1-TEST-VECTORS 237/239 > 4> ./fluster.py run -d GStreamer-H.264-V4L2SL-Gst1.0 -j1 -t 90 > JVT-AVC_V1 95/135 > 5> ./fluster.py run -d GStreamer-H.265-V4L2SL-Gst1.0 -j1 -t 90 > JCT-VC-HEVC_V1 142/147 > >compared with v1: >- add patch 2/3/4 to fix timing issue. >--- >Yunfei Dong (7): > media: mediatek: vcodec: setting request complete before buffer done > media: mediatek: vcodec: change flush decode order when stream off > media: mediatek: vcodec: flush decoder before stream off > media: mediatek: vcodec: using input information to get vb2 buffer > media: mediatek: vcodec: store source vb2 buffer > media: mediatek: vcodec: replace v4l2_m2m_next_src_buf with > v4l2_m2m_src_buf_remove > media: mediatek: vcodec: remove media request checking > > .../mediatek/vcodec/decoder/mtk_vcodec_dec.c | 44 ++++++++--------- > .../vcodec/decoder/mtk_vcodec_dec_drv.h | 4 +- > .../vcodec/decoder/mtk_vcodec_dec_stateless.c | 48 ++++++++++++++----- > .../vcodec/decoder/vdec/vdec_av1_req_lat_if.c | 18 +++---- > .../decoder/vdec/vdec_h264_req_multi_if.c | 4 +- > .../decoder/vdec/vdec_hevc_req_multi_if.c | 4 +- > .../vcodec/decoder/vdec/vdec_vp9_req_lat_if.c | 19 ++++---- > .../mediatek/vcodec/decoder/vdec_msg_queue.h | 4 +- > 8 files changed, 85 insertions(+), 60 deletions(-) > >-- >2.46.0 > >
Hi Sebastian, Thanks for your suggestion. On Fri, 2024-08-23 at 17:45 +0200, Sebastian Fricke wrote: > Hey Yunfei, > > given this new series by Hans: > https://urldefense.com/v3/__https://patchwork.linuxtv.org/project/linux-media/list/?series=13427__;!!CTRNKA9wMg0ARbw!h4__bSC_RX1erZCq5wvZbjRQrmS9ecXytwjcdLc9sgjBgj3buUC8JrUG78rhtEsXAN6vmZEhM3tWumFQ50R5NcR_1hbqLyYlSg$ > > > we might actually be able to find a more performant solution of the > problem, I'll work on that a bit and give you feedback. > I already do so many test for different project with these patches, whether it's possible to merge these patch firstly, then to call hans's changes to do stress test. Best Regards, Yunfei Dong > Regards, > Sebastian > > On 07.08.2024 16:24, Yunfei Dong wrote: > > v4l2_m2m_buf_done is called in lat work queue, > > v4l2_ctrl_request_complete > > is called in core queue. The request status of output queue will be > > set to > > MEDIA_REQUEST_STATE_COMPLETE when v4l2_m2m_buf_done is called, > > leading to > > output queue request complete fail. Must move > > v4l2_ctrl_request_complete > > in front of v4l2_m2m_buf_done. > > > > Patch 1 setting request complete before buffer done > > Patch 2 change flush decode order when stream off > > Patch 3 flush decoder before stream off > > Patch 4 using input information to get vb2 buffer > > Patch 5 store source vb2 buffer > > Patch 6 replace v4l2_m2m_next_src_buf with v4l2_m2m_src_buf_remove > > Patch 7 remove media request checking > > > > --- > > compared with v3: > > - fix flush decoder issue when userspace stream off capture queue > > firstly > > - fluster test result same with v3 > > > > compared with v2: > > - add patch 5/6/7 to fix decode again issue > > - add fluster test result with mt8195 platform(same with no > > changed): > > 1> ./fluster.py run -d GStreamer-VP8-V4L2SL-Gst1.0 -j1 -t 90 > > VP8-TEST-VECTORS 59/61 > > 2> ./fluster.py run -d GStreamer-VP9-V4L2SL-Gst1.0 -j1 -t 90 > > VP9-TEST-VECTORS 276/305 > > 3> ./fluster.py run -d GStreamer-AV1-V4L2SL-Gst1.0 -j1 -t 90 > > AV1-TEST-VECTORS 237/239 > > 4> ./fluster.py run -d GStreamer-H.264-V4L2SL-Gst1.0 -j1 -t 90 > > JVT-AVC_V1 95/135 > > 5> ./fluster.py run -d GStreamer-H.265-V4L2SL-Gst1.0 -j1 -t 90 > > JCT-VC-HEVC_V1 142/147 > > > > compared with v1: > > - add patch 2/3/4 to fix timing issue. > > --- > > Yunfei Dong (7): > > media: mediatek: vcodec: setting request complete before buffer > > done > > media: mediatek: vcodec: change flush decode order when stream off > > media: mediatek: vcodec: flush decoder before stream off > > media: mediatek: vcodec: using input information to get vb2 buffer > > media: mediatek: vcodec: store source vb2 buffer > > media: mediatek: vcodec: replace v4l2_m2m_next_src_buf with > > v4l2_m2m_src_buf_remove > > media: mediatek: vcodec: remove media request checking > > > > .../mediatek/vcodec/decoder/mtk_vcodec_dec.c | 44 ++++++++------ > > --- > > .../vcodec/decoder/mtk_vcodec_dec_drv.h | 4 +- > > .../vcodec/decoder/mtk_vcodec_dec_stateless.c | 48 ++++++++++++++ > > ----- > > .../vcodec/decoder/vdec/vdec_av1_req_lat_if.c | 18 +++---- > > .../decoder/vdec/vdec_h264_req_multi_if.c | 4 +- > > .../decoder/vdec/vdec_hevc_req_multi_if.c | 4 +- > > .../vcodec/decoder/vdec/vdec_vp9_req_lat_if.c | 19 ++++---- > > .../mediatek/vcodec/decoder/vdec_msg_queue.h | 4 +- > > 8 files changed, 85 insertions(+), 60 deletions(-) > > > > -- > > 2.46.0 > > > >
v4l2_m2m_buf_done is called in lat work queue, v4l2_ctrl_request_complete is called in core queue. The request status of output queue will be set to MEDIA_REQUEST_STATE_COMPLETE when v4l2_m2m_buf_done is called, leading to output queue request complete fail. Must move v4l2_ctrl_request_complete in front of v4l2_m2m_buf_done. Patch 1 setting request complete before buffer done Patch 2 change flush decode order when stream off Patch 3 flush decoder before stream off Patch 4 using input information to get vb2 buffer Patch 5 store source vb2 buffer Patch 6 replace v4l2_m2m_next_src_buf with v4l2_m2m_src_buf_remove Patch 7 remove media request checking --- compared with v3: - fix flush decoder issue when userspace stream off capture queue firstly - fluster test result same with v3 compared with v2: - add patch 5/6/7 to fix decode again issue - add fluster test result with mt8195 platform(same with no changed): 1> ./fluster.py run -d GStreamer-VP8-V4L2SL-Gst1.0 -j1 -t 90 VP8-TEST-VECTORS 59/61 2> ./fluster.py run -d GStreamer-VP9-V4L2SL-Gst1.0 -j1 -t 90 VP9-TEST-VECTORS 276/305 3> ./fluster.py run -d GStreamer-AV1-V4L2SL-Gst1.0 -j1 -t 90 AV1-TEST-VECTORS 237/239 4> ./fluster.py run -d GStreamer-H.264-V4L2SL-Gst1.0 -j1 -t 90 JVT-AVC_V1 95/135 5> ./fluster.py run -d GStreamer-H.265-V4L2SL-Gst1.0 -j1 -t 90 JCT-VC-HEVC_V1 142/147 compared with v1: - add patch 2/3/4 to fix timing issue. --- Yunfei Dong (7): media: mediatek: vcodec: setting request complete before buffer done media: mediatek: vcodec: change flush decode order when stream off media: mediatek: vcodec: flush decoder before stream off media: mediatek: vcodec: using input information to get vb2 buffer media: mediatek: vcodec: store source vb2 buffer media: mediatek: vcodec: replace v4l2_m2m_next_src_buf with v4l2_m2m_src_buf_remove media: mediatek: vcodec: remove media request checking .../mediatek/vcodec/decoder/mtk_vcodec_dec.c | 44 ++++++++--------- .../vcodec/decoder/mtk_vcodec_dec_drv.h | 4 +- .../vcodec/decoder/mtk_vcodec_dec_stateless.c | 48 ++++++++++++++----- .../vcodec/decoder/vdec/vdec_av1_req_lat_if.c | 18 +++---- .../decoder/vdec/vdec_h264_req_multi_if.c | 4 +- .../decoder/vdec/vdec_hevc_req_multi_if.c | 4 +- .../vcodec/decoder/vdec/vdec_vp9_req_lat_if.c | 19 ++++---- .../mediatek/vcodec/decoder/vdec_msg_queue.h | 4 +- 8 files changed, 85 insertions(+), 60 deletions(-)