| Message ID | 1690265540-25999-1-git-send-email-shengjiu.wang@nxp.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | Add audio support in v4l2 framework | expand |
Hi all, On 25/07/2023 08:12, Shengjiu Wang wrote: > Audio signal processing has the requirement for memory to > memory similar as Video. > > This patch is to add this support in v4l2 framework, defined > new buffer type V4L2_BUF_TYPE_AUDIO_CAPTURE and > V4L2_BUF_TYPE_AUDIO_OUTPUT, defined new format v4l2_audio_format > for audio case usage. > > The created audio device is named "/dev/audioX". > > And add memory to memory support for two kinds of i.MX ASRC > module Before I spend time on this: are the audio maintainers OK with doing this in V4L2? I do want to have a clear statement on this as it is not something I can decide. Regards, Hans > > changes in v2: > - decouple the implementation in v4l2 and ALSA > - implement the memory to memory driver as a platfrom driver > and move it to driver/media > - move fsl_asrc_common.h to include/sound folder > > Shengjiu Wang (7): > ASoC: fsl_asrc: define functions for memory to memory usage > ASoC: fsl_easrc: define functions for memory to memory usage > ASoC: fsl_asrc: move fsl_asrc_common.h to include/sound > media: v4l2: Add audio capture and output support > media: imx: fsl_asrc: Add memory to memory driver > ASoC: fsl_asrc: register m2m platform device > ASoC: fsl_easrc: register m2m platform device > > .../media/common/videobuf2/videobuf2-v4l2.c | 4 + > drivers/media/platform/nxp/Kconfig | 12 + > drivers/media/platform/nxp/Makefile | 1 + > drivers/media/platform/nxp/fsl_asrc_m2m.c | 962 ++++++++++++++++++ > drivers/media/v4l2-core/v4l2-dev.c | 17 + > drivers/media/v4l2-core/v4l2-ioctl.c | 52 + > include/media/v4l2-dev.h | 2 + > include/media/v4l2-ioctl.h | 34 + > .../fsl => include/sound}/fsl_asrc_common.h | 48 + > include/uapi/linux/videodev2.h | 19 + > sound/soc/fsl/fsl_asrc.c | 150 +++ > sound/soc/fsl/fsl_asrc.h | 4 +- > sound/soc/fsl/fsl_asrc_dma.c | 2 +- > sound/soc/fsl/fsl_easrc.c | 227 +++++ > sound/soc/fsl/fsl_easrc.h | 8 +- > 15 files changed, 1539 insertions(+), 3 deletions(-) > create mode 100644 drivers/media/platform/nxp/fsl_asrc_m2m.c > rename {sound/soc/fsl => include/sound}/fsl_asrc_common.h (63%) >
On Wed, 02 Aug 2023 09:32:37 +0200, Hans Verkuil wrote: > > Hi all, > > On 25/07/2023 08:12, Shengjiu Wang wrote: > > Audio signal processing has the requirement for memory to > > memory similar as Video. > > > > This patch is to add this support in v4l2 framework, defined > > new buffer type V4L2_BUF_TYPE_AUDIO_CAPTURE and > > V4L2_BUF_TYPE_AUDIO_OUTPUT, defined new format v4l2_audio_format > > for audio case usage. > > > > The created audio device is named "/dev/audioX". > > > > And add memory to memory support for two kinds of i.MX ASRC > > module > > Before I spend time on this: are the audio maintainers OK with doing > this in V4L2? > > I do want to have a clear statement on this as it is not something I > can decide. Well, I personally don't mind to have some audio capability in v4l2 layer. But, the only uncertain thing for now is whether this is a must-have or not. IIRC, the implementation in the sound driver side was never done just because there was no similar implementation? If so, and if the extension to the v4l2 core layer is needed, shouldn't it be more considered for the possible other route? thanks, Takashi
On Wed, Aug 2, 2023 at 7:22 PM Takashi Iwai <tiwai@suse.de> wrote: > > On Wed, 02 Aug 2023 09:32:37 +0200, > Hans Verkuil wrote: > > > > Hi all, > > > > On 25/07/2023 08:12, Shengjiu Wang wrote: > > > Audio signal processing has the requirement for memory to > > > memory similar as Video. > > > > > > This patch is to add this support in v4l2 framework, defined > > > new buffer type V4L2_BUF_TYPE_AUDIO_CAPTURE and > > > V4L2_BUF_TYPE_AUDIO_OUTPUT, defined new format v4l2_audio_format > > > for audio case usage. > > > > > > The created audio device is named "/dev/audioX". > > > > > > And add memory to memory support for two kinds of i.MX ASRC > > > module > > > > Before I spend time on this: are the audio maintainers OK with doing > > this in V4L2? > > > > I do want to have a clear statement on this as it is not something I > > can decide. > > Well, I personally don't mind to have some audio capability in v4l2 > layer. But, the only uncertain thing for now is whether this is a > must-have or not. > Thanks, I am also not sure about this. I am also confused that why there is no m2m implementation for audio in the kernel. Audio also has similar decoder encoder post-processing as video. > > IIRC, the implementation in the sound driver side was never done just > because there was no similar implementation? If so, and if the > extension to the v4l2 core layer is needed, shouldn't it be more > considered for the possible other route? > Actually I'd like someone could point me to the other route. I'd like to try. The reason why I select to extend v4l2 for such audio usage is that v4l2 looks best for this audio m2m implementation. v4l2 is designed for m2m usage. if we need implement another 'route', I don't think it can do better that v4l2. I appreciate that someone can share his ideas or doable solutions. And please don't ignore my request, ignore my patch. Best regards Wang shengjiu
On Wed, 02 Aug 2023 14:02:29 +0200, Shengjiu Wang wrote: > > On Wed, Aug 2, 2023 at 7:22 PM Takashi Iwai <tiwai@suse.de> wrote: > > > > On Wed, 02 Aug 2023 09:32:37 +0200, > > Hans Verkuil wrote: > > > > > > Hi all, > > > > > > On 25/07/2023 08:12, Shengjiu Wang wrote: > > > > Audio signal processing has the requirement for memory to > > > > memory similar as Video. > > > > > > > > This patch is to add this support in v4l2 framework, defined > > > > new buffer type V4L2_BUF_TYPE_AUDIO_CAPTURE and > > > > V4L2_BUF_TYPE_AUDIO_OUTPUT, defined new format v4l2_audio_format > > > > for audio case usage. > > > > > > > > The created audio device is named "/dev/audioX". > > > > > > > > And add memory to memory support for two kinds of i.MX ASRC > > > > module > > > > > > Before I spend time on this: are the audio maintainers OK with doing > > > this in V4L2? > > > > > > I do want to have a clear statement on this as it is not something I > > > can decide. > > > > Well, I personally don't mind to have some audio capability in v4l2 > > layer. But, the only uncertain thing for now is whether this is a > > must-have or not. > > > > Thanks, I am also not sure about this. I am also confused that why > there is no m2m implementation for audio in the kernel. Audio also > has similar decoder encoder post-processing as video. > > > > > IIRC, the implementation in the sound driver side was never done just > > because there was no similar implementation? If so, and if the > > extension to the v4l2 core layer is needed, shouldn't it be more > > considered for the possible other route? > > > > Actually I'd like someone could point me to the other route. I'd like to > try. > > The reason why I select to extend v4l2 for such audio usage is that v4l2 > looks best for this audio m2m implementation. v4l2 is designed for m2m > usage. if we need implement another 'route', I don't think it can do better > that v4l2. > > I appreciate that someone can share his ideas or doable solutions. > And please don't ignore my request, ignore my patch. Can you explain a bit more details of your demand? At least, a "big picture" showing how your hardware is implemented and what is exactly necessary would be helpful for understanding the problem. thanks, Takashi
On Wed, Aug 02, 2023 at 08:02:29PM +0800, Shengjiu Wang wrote: > On Wed, Aug 2, 2023 at 7:22 PM Takashi Iwai <tiwai@suse.de> wrote: > > Well, I personally don't mind to have some audio capability in v4l2 > > layer. But, the only uncertain thing for now is whether this is a > > must-have or not. > Thanks, I am also not sure about this. I am also confused that why > there is no m2m implementation for audio in the kernel. Audio also > has similar decoder encoder post-processing as video. This is the thing where we've been trying to persuade people to work on replacing DPCM with full componentisation for about a decade now but nobody's had time other than Morimoto-san who's been chipping away at making everything component based for a good chunk of that time. One trick is that we don't just want this to work for things that are memory to memory, we also want things where there's a direct interconnect that bypasses memory for off-SoC case. > The reason why I select to extend v4l2 for such audio usage is that v4l2 > looks best for this audio m2m implementation. v4l2 is designed for m2m > usage. if we need implement another 'route', I don't think it can do better > that v4l2. > I appreciate that someone can share his ideas or doable solutions. > And please don't ignore my request, ignore my patch. There's a bunch of presentations Lars-Peter did at ELC some considerable time ago about this.
On 02/08/2023 14:02, Shengjiu Wang wrote: > On Wed, Aug 2, 2023 at 7:22 PM Takashi Iwai <tiwai@suse.de> wrote: >> >> On Wed, 02 Aug 2023 09:32:37 +0200, >> Hans Verkuil wrote: >>> >>> Hi all, >>> >>> On 25/07/2023 08:12, Shengjiu Wang wrote: >>>> Audio signal processing has the requirement for memory to >>>> memory similar as Video. >>>> >>>> This patch is to add this support in v4l2 framework, defined >>>> new buffer type V4L2_BUF_TYPE_AUDIO_CAPTURE and >>>> V4L2_BUF_TYPE_AUDIO_OUTPUT, defined new format v4l2_audio_format >>>> for audio case usage. >>>> >>>> The created audio device is named "/dev/audioX". >>>> >>>> And add memory to memory support for two kinds of i.MX ASRC >>>> module >>> >>> Before I spend time on this: are the audio maintainers OK with doing >>> this in V4L2? >>> >>> I do want to have a clear statement on this as it is not something I >>> can decide. >> >> Well, I personally don't mind to have some audio capability in v4l2 >> layer. But, the only uncertain thing for now is whether this is a >> must-have or not. >> > > Thanks, I am also not sure about this. I am also confused that why > there is no m2m implementation for audio in the kernel. Audio also > has similar decoder encoder post-processing as video. > >> >> IIRC, the implementation in the sound driver side was never done just >> because there was no similar implementation? If so, and if the >> extension to the v4l2 core layer is needed, shouldn't it be more >> considered for the possible other route? >> > > Actually I'd like someone could point me to the other route. I'd like to > try. > > The reason why I select to extend v4l2 for such audio usage is that v4l2 > looks best for this audio m2m implementation. v4l2 is designed for m2m > usage. if we need implement another 'route', I don't think it can do better > that v4l2. > > I appreciate that someone can share his ideas or doable solutions. > And please don't ignore my request, ignore my patch. To give a bit more background: if it is decided to use the v4l API for this (and I have no objection to this from my side since API/framework-wise it is a good fit for this), then there are a number of things that need to be done to get this into the media subsystem: - documentation for the new uAPI - add support for this to v4l2-ctl - add v4l2-compliance tests for the new device - highly desirable: have a virtual driver (similar to vim2m) that supports this: it could be as simple as just copy input to output. This helps regression testing. - it might need media controller support as well. TBD. None of this is particularly complex, but taken all together it is a fair amount of work that also needs a lot of review time from our side. I want to add one more option to the mix: drivers/media/core/v4l2-mem2mem.c is the main m2m framework, but it relies heavily on the videobuf2 framework for the capture and output queues. The core vb2 implementation in drivers/media/common/videobuf2/videobuf2-core.c is independent of V4L2 and can be used by other subsystems (in our case, it is also used by the DVB API). It is a possibility to create an alsa version of v4l2-mem2mem.c that uses the core vb2 code with an ALSA uAPI on top. So in drivers/media/common/videobuf2/ you would have a videobuf2-alsa.c besides the already existing videobuf2-v4l2.c and -dvb.c. Perhaps parts of v4l2-mem2mem.c can be reused as well in that case, but I am not sure if it is worth the effort. I suspect copying it to an alsa-mem2mem.c and adapting it for alsa is easiest if you want to go that way. Regards, Hans
On Wed, Aug 2, 2023 at 8:08 PM Takashi Iwai <tiwai@suse.de> wrote: > > On Wed, 02 Aug 2023 14:02:29 +0200, > Shengjiu Wang wrote: > > > > On Wed, Aug 2, 2023 at 7:22 PM Takashi Iwai <tiwai@suse.de> wrote: > > > > > > On Wed, 02 Aug 2023 09:32:37 +0200, > > > Hans Verkuil wrote: > > > > > > > > Hi all, > > > > > > > > On 25/07/2023 08:12, Shengjiu Wang wrote: > > > > > Audio signal processing has the requirement for memory to > > > > > memory similar as Video. > > > > > > > > > > This patch is to add this support in v4l2 framework, defined > > > > > new buffer type V4L2_BUF_TYPE_AUDIO_CAPTURE and > > > > > V4L2_BUF_TYPE_AUDIO_OUTPUT, defined new format v4l2_audio_format > > > > > for audio case usage. > > > > > > > > > > The created audio device is named "/dev/audioX". > > > > > > > > > > And add memory to memory support for two kinds of i.MX ASRC > > > > > module > > > > > > > > Before I spend time on this: are the audio maintainers OK with doing > > > > this in V4L2? > > > > > > > > I do want to have a clear statement on this as it is not something I > > > > can decide. > > > > > > Well, I personally don't mind to have some audio capability in v4l2 > > > layer. But, the only uncertain thing for now is whether this is a > > > must-have or not. > > > > > > > Thanks, I am also not sure about this. I am also confused that why > > there is no m2m implementation for audio in the kernel. Audio also > > has similar decoder encoder post-processing as video. > > > > > > > > IIRC, the implementation in the sound driver side was never done just > > > because there was no similar implementation? If so, and if the > > > extension to the v4l2 core layer is needed, shouldn't it be more > > > considered for the possible other route? > > > > > > > Actually I'd like someone could point me to the other route. I'd like to > > try. > > > > The reason why I select to extend v4l2 for such audio usage is that v4l2 > > looks best for this audio m2m implementation. v4l2 is designed for m2m > > usage. if we need implement another 'route', I don't think it can do better > > that v4l2. > > > > I appreciate that someone can share his ideas or doable solutions. > > And please don't ignore my request, ignore my patch. > > Can you explain a bit more details of your demand? > At least, a "big picture" showing how your hardware is implemented and > what is exactly necessary would be helpful for understanding the > problem. > We have the hardware IP: ASRC, asynchronous sample rate converter. Currently the ASRC in ALSA is to connect to another I2S device as a sound card. But we'd like to the ASRC can be used by user space directly that user space application can get the output after conversion from ASRC. The ASRC can be integrated into a multimedia framework (gstreamer) as a plugin. best regards wang shengjiu
On Wed, Aug 02, 2023 at 10:41:43PM +0800, Shengjiu Wang wrote: > Currently the ASRC in ALSA is to connect to another I2S device as > a sound card. But we'd like to the ASRC can be used by user space directly > that user space application can get the output after conversion from ASRC. That sort of use case would be handled via DPCM at the minute, though persuading it to connect two front ends together might be fun (which is the sort of reason why we want to push digital information down into DAPM and make everything a component).
On Thu, Aug 3, 2023 at 1:28 AM Mark Brown <broonie@kernel.org> wrote: > > On Wed, Aug 02, 2023 at 10:41:43PM +0800, Shengjiu Wang wrote: > > > Currently the ASRC in ALSA is to connect to another I2S device as > > a sound card. But we'd like to the ASRC can be used by user space directly > > that user space application can get the output after conversion from ASRC. > > That sort of use case would be handled via DPCM at the minute, though > persuading it to connect two front ends together might be fun (which is > the sort of reason why we want to push digital information down into > DAPM and make everything a component). Thanks. ASRC M2M case needs to run as fast as possible, no sync clock control. If use sound card to handle ASRC M2M case, the user application should be aplay/arecord, then we need to consider xrun issue, buffer may timeout, sync between aplay and arecord, these should't be considered by pure memory to memory operation. DPCM may achitect all the audio things in components and sound card, it is good. but for the M2M case, it is complcated. not sure it is doable. best regards wang shengjiu
On Wed, Aug 2, 2023 at 8:28 PM Hans Verkuil <hverkuil@xs4all.nl> wrote: > > On 02/08/2023 14:02, Shengjiu Wang wrote: > > On Wed, Aug 2, 2023 at 7:22 PM Takashi Iwai <tiwai@suse.de> wrote: > >> > >> On Wed, 02 Aug 2023 09:32:37 +0200, > >> Hans Verkuil wrote: > >>> > >>> Hi all, > >>> > >>> On 25/07/2023 08:12, Shengjiu Wang wrote: > >>>> Audio signal processing has the requirement for memory to > >>>> memory similar as Video. > >>>> > >>>> This patch is to add this support in v4l2 framework, defined > >>>> new buffer type V4L2_BUF_TYPE_AUDIO_CAPTURE and > >>>> V4L2_BUF_TYPE_AUDIO_OUTPUT, defined new format v4l2_audio_format > >>>> for audio case usage. > >>>> > >>>> The created audio device is named "/dev/audioX". > >>>> > >>>> And add memory to memory support for two kinds of i.MX ASRC > >>>> module > >>> > >>> Before I spend time on this: are the audio maintainers OK with doing > >>> this in V4L2? > >>> > >>> I do want to have a clear statement on this as it is not something I > >>> can decide. > >> > >> Well, I personally don't mind to have some audio capability in v4l2 > >> layer. But, the only uncertain thing for now is whether this is a > >> must-have or not. > >> > > > > Thanks, I am also not sure about this. I am also confused that why > > there is no m2m implementation for audio in the kernel. Audio also > > has similar decoder encoder post-processing as video. > > > >> > >> IIRC, the implementation in the sound driver side was never done just > >> because there was no similar implementation? If so, and if the > >> extension to the v4l2 core layer is needed, shouldn't it be more > >> considered for the possible other route? > >> > > > > Actually I'd like someone could point me to the other route. I'd like to > > try. > > > > The reason why I select to extend v4l2 for such audio usage is that v4l2 > > looks best for this audio m2m implementation. v4l2 is designed for m2m > > usage. if we need implement another 'route', I don't think it can do better > > that v4l2. > > > > I appreciate that someone can share his ideas or doable solutions. > > And please don't ignore my request, ignore my patch. > > To give a bit more background: if it is decided to use the v4l API for this > (and I have no objection to this from my side since API/framework-wise it is a > good fit for this), then there are a number of things that need to be done to > get this into the media subsystem: > > - documentation for the new uAPI > - add support for this to v4l2-ctl > - add v4l2-compliance tests for the new device > - highly desirable: have a virtual driver (similar to vim2m) that supports this: > it could be as simple as just copy input to output. This helps regression > testing. > - it might need media controller support as well. TBD. > > None of this is particularly complex, but taken all together it is a fair > amount of work that also needs a lot of review time from our side. > > I want to add one more option to the mix: drivers/media/core/v4l2-mem2mem.c is > the main m2m framework, but it relies heavily on the videobuf2 framework for > the capture and output queues. > > The core vb2 implementation in drivers/media/common/videobuf2/videobuf2-core.c > is independent of V4L2 and can be used by other subsystems (in our case, it is > also used by the DVB API). It is a possibility to create an alsa version of > v4l2-mem2mem.c that uses the core vb2 code with an ALSA uAPI on top. > > So in drivers/media/common/videobuf2/ you would have a videobuf2-alsa.c besides > the already existing videobuf2-v4l2.c and -dvb.c. > > Perhaps parts of v4l2-mem2mem.c can be reused as well in that case, but I am > not sure if it is worth the effort. I suspect copying it to an alsa-mem2mem.c > and adapting it for alsa is easiest if you want to go that way. > Thanks. Does this means that videobuf2-v4l2.c and v4l2-mem2mem.c are dedicate for video device? if audio want to use v4l2 framework, need to create videobuf2-alsa.c and alsa-mem2mem.c, but it may cause a lot of function duplicate. Best regards Wang Shengjiu
On 04/08/2023 14:19, Shengjiu Wang wrote: > On Wed, Aug 2, 2023 at 8:28 PM Hans Verkuil <hverkuil@xs4all.nl> wrote: >> >> On 02/08/2023 14:02, Shengjiu Wang wrote: >>> On Wed, Aug 2, 2023 at 7:22 PM Takashi Iwai <tiwai@suse.de> wrote: >>>> >>>> On Wed, 02 Aug 2023 09:32:37 +0200, >>>> Hans Verkuil wrote: >>>>> >>>>> Hi all, >>>>> >>>>> On 25/07/2023 08:12, Shengjiu Wang wrote: >>>>>> Audio signal processing has the requirement for memory to >>>>>> memory similar as Video. >>>>>> >>>>>> This patch is to add this support in v4l2 framework, defined >>>>>> new buffer type V4L2_BUF_TYPE_AUDIO_CAPTURE and >>>>>> V4L2_BUF_TYPE_AUDIO_OUTPUT, defined new format v4l2_audio_format >>>>>> for audio case usage. >>>>>> >>>>>> The created audio device is named "/dev/audioX". >>>>>> >>>>>> And add memory to memory support for two kinds of i.MX ASRC >>>>>> module >>>>> >>>>> Before I spend time on this: are the audio maintainers OK with doing >>>>> this in V4L2? >>>>> >>>>> I do want to have a clear statement on this as it is not something I >>>>> can decide. >>>> >>>> Well, I personally don't mind to have some audio capability in v4l2 >>>> layer. But, the only uncertain thing for now is whether this is a >>>> must-have or not. >>>> >>> >>> Thanks, I am also not sure about this. I am also confused that why >>> there is no m2m implementation for audio in the kernel. Audio also >>> has similar decoder encoder post-processing as video. >>> >>>> >>>> IIRC, the implementation in the sound driver side was never done just >>>> because there was no similar implementation? If so, and if the >>>> extension to the v4l2 core layer is needed, shouldn't it be more >>>> considered for the possible other route? >>>> >>> >>> Actually I'd like someone could point me to the other route. I'd like to >>> try. >>> >>> The reason why I select to extend v4l2 for such audio usage is that v4l2 >>> looks best for this audio m2m implementation. v4l2 is designed for m2m >>> usage. if we need implement another 'route', I don't think it can do better >>> that v4l2. >>> >>> I appreciate that someone can share his ideas or doable solutions. >>> And please don't ignore my request, ignore my patch. >> >> To give a bit more background: if it is decided to use the v4l API for this >> (and I have no objection to this from my side since API/framework-wise it is a >> good fit for this), then there are a number of things that need to be done to >> get this into the media subsystem: >> >> - documentation for the new uAPI >> - add support for this to v4l2-ctl >> - add v4l2-compliance tests for the new device >> - highly desirable: have a virtual driver (similar to vim2m) that supports this: >> it could be as simple as just copy input to output. This helps regression >> testing. >> - it might need media controller support as well. TBD. >> >> None of this is particularly complex, but taken all together it is a fair >> amount of work that also needs a lot of review time from our side. >> >> I want to add one more option to the mix: drivers/media/core/v4l2-mem2mem.c is >> the main m2m framework, but it relies heavily on the videobuf2 framework for >> the capture and output queues. >> >> The core vb2 implementation in drivers/media/common/videobuf2/videobuf2-core.c >> is independent of V4L2 and can be used by other subsystems (in our case, it is >> also used by the DVB API). It is a possibility to create an alsa version of >> v4l2-mem2mem.c that uses the core vb2 code with an ALSA uAPI on top. >> >> So in drivers/media/common/videobuf2/ you would have a videobuf2-alsa.c besides >> the already existing videobuf2-v4l2.c and -dvb.c. >> >> Perhaps parts of v4l2-mem2mem.c can be reused as well in that case, but I am >> not sure if it is worth the effort. I suspect copying it to an alsa-mem2mem.c >> and adapting it for alsa is easiest if you want to go that way. >> > > Thanks. > > Does this means that videobuf2-v4l2.c and v4l2-mem2mem.c are dedicate > for video device? if audio want to use v4l2 framework, need to create > videobuf2-alsa.c and alsa-mem2mem.c, but it may cause a lot of function > duplicate. The videobuf2-v4l2.c sits on top of videobuf2-core.c and provides the V4L2 uAPI for the streaming functionality. If you don't want to use the V4L2 uAPI for this, then you would need a videobuf2-alsa.c that provides a (possibly new) ALSA uAPI. Whether that makes sense is something I cannot decide. v4l2-mem2mem.c uses videobuf2-v4l2.c, so if you need a ALSA version, then you probably need to create an alsa-mem2mem.c (possibly some functionality can be shared). It's just a third option, and it can be useful if there is a strong desire to keep the uAPI for this functionality entirely within the ALSA subsystem, but you want to reuse the streaming I/O functionality that the videobuf2 core provides. If the decision is that it is fine to use the V4L2 uAPI for this type of audio functionality through a /dev/v4l-audioX device, then just ignore this option and use V4L2. Regards, Hans
Hi Mark, Takashi On Thu, Aug 3, 2023 at 9:11 PM Shengjiu Wang <shengjiu.wang@gmail.com> wrote: > > On Thu, Aug 3, 2023 at 1:28 AM Mark Brown <broonie@kernel.org> wrote: > > > > On Wed, Aug 02, 2023 at 10:41:43PM +0800, Shengjiu Wang wrote: > > > > > Currently the ASRC in ALSA is to connect to another I2S device as > > > a sound card. But we'd like to the ASRC can be used by user space directly > > > that user space application can get the output after conversion from ASRC. > > > > That sort of use case would be handled via DPCM at the minute, though > > persuading it to connect two front ends together might be fun (which is > > the sort of reason why we want to push digital information down into > > DAPM and make everything a component). > > Thanks. > > ASRC M2M case needs to run as fast as possible, no sync clock control. > If use sound card to handle ASRC M2M case, the user application > should be aplay/arecord, then we need to consider xrun issue, buffer > may timeout, sync between aplay and arecord, these should't be > considered by pure memory to memory operation. > > DPCM may achitect all the audio things in components and sound > card, it is good. but for the M2M case, it is complcated. not sure > it is doable. > Beside the concern in previous mail, DPCM needs to separate ASRC to be two substreams (playback and capture). But the ASRC needs the sample rate & format of input and output first then start conversion. If the playback controls the rate & format of input, capture substream controls the rate & format of output, as a result one substream needs to get information(dma buffer address, size... rate, format) from another substream, then start both substreams in the last substream. How to synchronize these two substreams is a problem. One stream can be released but another stream doesn't know . So I don't think it is a good idea to use DPCM for pure M2M case. So can I persuade you to consider the V4L2 solution? Best regards Wang Shengjiu
On Fri, Aug 11, 2023 at 7:05 PM Shengjiu Wang <shengjiu.wang@gmail.com> wrote: > > Hi Mark, Takashi > > On Thu, Aug 3, 2023 at 9:11 PM Shengjiu Wang <shengjiu.wang@gmail.com> wrote: > > > > On Thu, Aug 3, 2023 at 1:28 AM Mark Brown <broonie@kernel.org> wrote: > > > > > > On Wed, Aug 02, 2023 at 10:41:43PM +0800, Shengjiu Wang wrote: > > > > > > > Currently the ASRC in ALSA is to connect to another I2S device as > > > > a sound card. But we'd like to the ASRC can be used by user space directly > > > > that user space application can get the output after conversion from ASRC. > > > > > > That sort of use case would be handled via DPCM at the minute, though > > > persuading it to connect two front ends together might be fun (which is > > > the sort of reason why we want to push digital information down into > > > DAPM and make everything a component). > > > > Thanks. > > > > ASRC M2M case needs to run as fast as possible, no sync clock control. > > If use sound card to handle ASRC M2M case, the user application > > should be aplay/arecord, then we need to consider xrun issue, buffer > > may timeout, sync between aplay and arecord, these should't be > > considered by pure memory to memory operation. > > > > DPCM may achitect all the audio things in components and sound > > card, it is good. but for the M2M case, it is complcated. not sure > > it is doable. > > > > Beside the concern in previous mail, > > DPCM needs to separate ASRC to be two substreams (playback and capture). > > But the ASRC needs the sample rate & format of input and output first > then start conversion. > > If the playback controls the rate & format of input, capture substream > controls the rate & format of output, as a result > one substream needs to get information(dma buffer address, size... > rate, format) from another substream, then start both substreams in the > last substream. How to synchronize these two substreams is a problem. > One stream can be released but another stream doesn't know . > > So I don't think it is a good idea to use DPCM for pure M2M case. > > So can I persuade you to consider the V4L2 solution? > Just a summary: Basic M2M conversion can work with DPCM, I have tried with some workaround to make it work. But there are several issues: 1. Need to create sound cards. ASRC module support multi instances, then need to create multi sound cards for each instance. 2. The ASRC is an entirety but with DPCM we need to separate input port and output port to playback substream and capture stream. Synchronous between playback substream and capture substream is a problem. How to start them and stop them at the same time. 3. How to handle the xrun issue. pause or resume. which are brought by ALSA. So shall we make the decision that we can go to the V4L2 solution? Best regards Wang Shengjiu
On Wed, 23 Aug 2023 16:33:19 +0200, Shengjiu Wang wrote: > > On Fri, Aug 11, 2023 at 7:05 PM Shengjiu Wang <shengjiu.wang@gmail.com> wrote: > > > > Hi Mark, Takashi > > > > On Thu, Aug 3, 2023 at 9:11 PM Shengjiu Wang <shengjiu.wang@gmail.com> wrote: > > > > > > On Thu, Aug 3, 2023 at 1:28 AM Mark Brown <broonie@kernel.org> wrote: > > > > > > > > On Wed, Aug 02, 2023 at 10:41:43PM +0800, Shengjiu Wang wrote: > > > > > > > > > Currently the ASRC in ALSA is to connect to another I2S device as > > > > > a sound card. But we'd like to the ASRC can be used by user space directly > > > > > that user space application can get the output after conversion from ASRC. > > > > > > > > That sort of use case would be handled via DPCM at the minute, though > > > > persuading it to connect two front ends together might be fun (which is > > > > the sort of reason why we want to push digital information down into > > > > DAPM and make everything a component). > > > > > > Thanks. > > > > > > ASRC M2M case needs to run as fast as possible, no sync clock control. > > > If use sound card to handle ASRC M2M case, the user application > > > should be aplay/arecord, then we need to consider xrun issue, buffer > > > may timeout, sync between aplay and arecord, these should't be > > > considered by pure memory to memory operation. > > > > > > DPCM may achitect all the audio things in components and sound > > > card, it is good. but for the M2M case, it is complcated. not sure > > > it is doable. > > > > > > > Beside the concern in previous mail, > > > > DPCM needs to separate ASRC to be two substreams (playback and capture). > > > > But the ASRC needs the sample rate & format of input and output first > > then start conversion. > > > > If the playback controls the rate & format of input, capture substream > > controls the rate & format of output, as a result > > one substream needs to get information(dma buffer address, size... > > rate, format) from another substream, then start both substreams in the > > last substream. How to synchronize these two substreams is a problem. > > One stream can be released but another stream doesn't know . > > > > So I don't think it is a good idea to use DPCM for pure M2M case. > > > > So can I persuade you to consider the V4L2 solution? > > > > Just a summary: > > Basic M2M conversion can work with DPCM, I have tried with some > workaround to make it work. > > But there are several issues: > 1. Need to create sound cards. ASRC module support multi instances, then > need to create multi sound cards for each instance. Hm, why can't it be multiple PCM instances instead? > 2. The ASRC is an entirety but with DPCM we need to separate input port and > output port to playback substream and capture stream. Synchronous between > playback substream and capture substream is a problem. > How to start them and stop them at the same time. This could be done by enforcing the full duplex and linking the both PCM streams, I suppose. > 3. How to handle the xrun issue. pause or resume. which are brought by ALSA. Doesn't V4L2 handle the overrun/underrun at all? Also, no resume support? Pause and resume are optional in ALSA frame work, you don't need to implement them unless you want/need. > So shall we make the decision that we can go to the V4L2 solution? Honestly speaking, I don't mind much whether it's implemented in V2L4 or not -- at least for the kernel part, we can reorganize / refactor things internally. But, the biggest remaining question to me is whether this user-space interface is the most suitable one. Is it well defined, usable and maintained for the audio applications? Or is it meant to be a stop-gap for a specific use case? thanks, Takashi
On Thu, Aug 24, 2023 at 07:03:09PM +0200, Takashi Iwai wrote: > Shengjiu Wang wrote: > > But there are several issues: > > 1. Need to create sound cards. ASRC module support multi instances, then > > need to create multi sound cards for each instance. > Hm, why can't it be multiple PCM instances instead? I'm having a hard time following this one too. > > 2. The ASRC is an entirety but with DPCM we need to separate input port and > > output port to playback substream and capture stream. Synchronous between > > playback substream and capture substream is a problem. > > How to start them and stop them at the same time. > This could be done by enforcing the full duplex and linking the both > PCM streams, I suppose. > > So shall we make the decision that we can go to the V4L2 solution? > Honestly speaking, I don't mind much whether it's implemented in V2L4 > or not -- at least for the kernel part, we can reorganize / refactor > things internally. But, the biggest remaining question to me is > whether this user-space interface is the most suitable one. Is it > well defined, usable and maintained for the audio applications? Or > is it meant to be a stop-gap for a specific use case? I'm having a really hard time summoning much enthusiasm for using v4l here, it feels like this is heading down the same bodge route as DPCM but directly as ABI so even harder to fix properly. That said all the ALSA APIs are really intended to be used in real time and this sounds like a non real time application? I don't fully understand what the actual use case is here.
On Fri, Aug 25, 2023 at 4:21 AM Mark Brown <broonie@kernel.org> wrote: > > On Thu, Aug 24, 2023 at 07:03:09PM +0200, Takashi Iwai wrote: > > Shengjiu Wang wrote: > > > > But there are several issues: > > > 1. Need to create sound cards. ASRC module support multi instances, then > > > need to create multi sound cards for each instance. > > > Hm, why can't it be multiple PCM instances instead? > > I'm having a hard time following this one too. > > > > 2. The ASRC is an entirety but with DPCM we need to separate input port and > > > output port to playback substream and capture stream. Synchronous between > > > playback substream and capture substream is a problem. > > > How to start them and stop them at the same time. > > > This could be done by enforcing the full duplex and linking the both > > PCM streams, I suppose. > > > > So shall we make the decision that we can go to the V4L2 solution? > > > Honestly speaking, I don't mind much whether it's implemented in V2L4 > > or not -- at least for the kernel part, we can reorganize / refactor > > things internally. But, the biggest remaining question to me is > > whether this user-space interface is the most suitable one. Is it > > well defined, usable and maintained for the audio applications? Or > > is it meant to be a stop-gap for a specific use case? > > I'm having a really hard time summoning much enthusiasm for using v4l > here, it feels like this is heading down the same bodge route as DPCM > but directly as ABI so even harder to fix properly. That said all the > ALSA APIs are really intended to be used in real time and this sounds > like a non real time application? I don't fully understand what the > actual use case is here. Thanks for your reply. This asrc memory to memory (memory ->asrc->memory) case is a non real time use case. User fills the input buffer to the asrc module, after conversion, then asrc sends back the output buffer to user. So it is not a traditional ALSA playback and capture case. I think it is not good to create sound card for it, it is not a sound card actually. It is a specific use case, there is no reference in current kernel. v4l2 memory to memory is the closed implementation, v4l2 current support video, image, radio, tuner, touch devices, so it is not complicated to add support for this specific audio case. Maybe you can go through these patches first. Because we had implemented the "memory -> asrc ->i2s device-> codec" use case in ALSA. Now the "memory->asrc->memory" needs to reuse the code in asrc driver, so the first 3 patches is for refining the code to make it can be shared by the "memory->asrc->memory" driver. The main change is in the v4l2 side, A /dev/vl42-audio will be created, user applications only use the ioctl of v4l2 framework. Best regards Wang Shengjiu
On Fri, 25 Aug 2023 05:46:43 +0200, Shengjiu Wang wrote: > > On Fri, Aug 25, 2023 at 4:21 AM Mark Brown <broonie@kernel.org> wrote: > > > > On Thu, Aug 24, 2023 at 07:03:09PM +0200, Takashi Iwai wrote: > > > Shengjiu Wang wrote: > > > > > > But there are several issues: > > > > 1. Need to create sound cards. ASRC module support multi instances, then > > > > need to create multi sound cards for each instance. > > > > > Hm, why can't it be multiple PCM instances instead? > > > > I'm having a hard time following this one too. > > > > > > 2. The ASRC is an entirety but with DPCM we need to separate input port and > > > > output port to playback substream and capture stream. Synchronous between > > > > playback substream and capture substream is a problem. > > > > How to start them and stop them at the same time. > > > > > This could be done by enforcing the full duplex and linking the both > > > PCM streams, I suppose. > > > > > > So shall we make the decision that we can go to the V4L2 solution? > > > > > Honestly speaking, I don't mind much whether it's implemented in V2L4 > > > or not -- at least for the kernel part, we can reorganize / refactor > > > things internally. But, the biggest remaining question to me is > > > whether this user-space interface is the most suitable one. Is it > > > well defined, usable and maintained for the audio applications? Or > > > is it meant to be a stop-gap for a specific use case? > > > > I'm having a really hard time summoning much enthusiasm for using v4l > > here, it feels like this is heading down the same bodge route as DPCM > > but directly as ABI so even harder to fix properly. That said all the > > ALSA APIs are really intended to be used in real time and this sounds > > like a non real time application? I don't fully understand what the > > actual use case is here. > > Thanks for your reply. > > This asrc memory to memory (memory ->asrc->memory) case is a non > real time use case. > > User fills the input buffer to the asrc module, after conversion, then asrc > sends back the output buffer to user. So it is not a traditional ALSA playback > and capture case. I think it is not good to create sound card for it, it is > not a sound card actually. > > It is a specific use case, there is no reference in current kernel. > v4l2 memory to memory is the closed implementation, v4l2 current > support video, image, radio, tuner, touch devices, so it is not > complicated to add support for this specific audio case. > > Maybe you can go through these patches first. Because we > had implemented the "memory -> asrc ->i2s device-> codec" > use case in ALSA. Now the "memory->asrc->memory" needs > to reuse the code in asrc driver, so the first 3 patches is for refining > the code to make it can be shared by the "memory->asrc->memory" > driver. > > The main change is in the v4l2 side, A /dev/vl42-audio will be created, > user applications only use the ioctl of v4l2 framework. Ah, now I'm slowly understanding. So, what you want is to have an interface to perform the batch conversion of a data stream from an input to an output? And ALSA PCM interface doesn't fit fully for that purpose because the data handling is batched and it's not like a normal PCM streaming? Basically the whole M2M arguments are rather subtle. Those are implementation details that can be resolved in several different ways in the kernel side. But the design of the operation is the crucial point. Maybe we can consider implementing a similar feature in ALSA API, too. But it's too far-stretched for now. So, if v4l2 interface provides the requested feature (the batched audio stream conversion), it's OK to ride on it. thanks, Takashi
On 25/08/2023 15:54, Takashi Iwai wrote: > On Fri, 25 Aug 2023 05:46:43 +0200, > Shengjiu Wang wrote: >> >> On Fri, Aug 25, 2023 at 4:21 AM Mark Brown <broonie@kernel.org> wrote: >>> >>> On Thu, Aug 24, 2023 at 07:03:09PM +0200, Takashi Iwai wrote: >>>> Shengjiu Wang wrote: >>> >>>>> But there are several issues: >>>>> 1. Need to create sound cards. ASRC module support multi instances, then >>>>> need to create multi sound cards for each instance. >>> >>>> Hm, why can't it be multiple PCM instances instead? >>> >>> I'm having a hard time following this one too. >>> >>>>> 2. The ASRC is an entirety but with DPCM we need to separate input port and >>>>> output port to playback substream and capture stream. Synchronous between >>>>> playback substream and capture substream is a problem. >>>>> How to start them and stop them at the same time. >>> >>>> This could be done by enforcing the full duplex and linking the both >>>> PCM streams, I suppose. >>> >>>>> So shall we make the decision that we can go to the V4L2 solution? >>> >>>> Honestly speaking, I don't mind much whether it's implemented in V2L4 >>>> or not -- at least for the kernel part, we can reorganize / refactor >>>> things internally. But, the biggest remaining question to me is >>>> whether this user-space interface is the most suitable one. Is it >>>> well defined, usable and maintained for the audio applications? Or >>>> is it meant to be a stop-gap for a specific use case? >>> >>> I'm having a really hard time summoning much enthusiasm for using v4l >>> here, it feels like this is heading down the same bodge route as DPCM >>> but directly as ABI so even harder to fix properly. That said all the >>> ALSA APIs are really intended to be used in real time and this sounds >>> like a non real time application? I don't fully understand what the >>> actual use case is here. >> >> Thanks for your reply. >> >> This asrc memory to memory (memory ->asrc->memory) case is a non >> real time use case. >> >> User fills the input buffer to the asrc module, after conversion, then asrc >> sends back the output buffer to user. So it is not a traditional ALSA playback >> and capture case. I think it is not good to create sound card for it, it is >> not a sound card actually. >> >> It is a specific use case, there is no reference in current kernel. >> v4l2 memory to memory is the closed implementation, v4l2 current >> support video, image, radio, tuner, touch devices, so it is not >> complicated to add support for this specific audio case. >> >> Maybe you can go through these patches first. Because we >> had implemented the "memory -> asrc ->i2s device-> codec" >> use case in ALSA. Now the "memory->asrc->memory" needs >> to reuse the code in asrc driver, so the first 3 patches is for refining >> the code to make it can be shared by the "memory->asrc->memory" >> driver. >> >> The main change is in the v4l2 side, A /dev/vl42-audio will be created, >> user applications only use the ioctl of v4l2 framework. > > Ah, now I'm slowly understanding. So, what you want is to have an > interface to perform the batch conversion of a data stream from an > input to an output? And ALSA PCM interface doesn't fit fully for that > purpose because the data handling is batched and it's not like a > normal PCM streaming? > > Basically the whole M2M arguments are rather subtle. Those are > implementation details that can be resolved in several different ways > in the kernel side. But the design of the operation is the crucial > point. > > Maybe we can consider implementing a similar feature in ALSA API, too. > But it's too far-stretched for now. > > So, if v4l2 interface provides the requested feature (the batched > audio stream conversion), it's OK to ride on it. The V4L2 M2M interface is simple: you open a video device and then you can pass data to the hardware, it processes it and you get the processed data back. The hardware just processes the data as fast as it can. Each time you open the video device a new instance is created, and each instance can pass jobs to the hardware. Currently it is used for video scalers, deinterlacers, colorspace converters and codecs, but in the end it is just data in, data out with some job scheduling (fifo) towards the hardware. So supporting audio using the same core m2m framework wouldn't be a big deal. We'd probably make a /dev/v4l-audio device for that. It doesn't come for free: it is a new API, so besides adding support for it, it also needs to be documented, we would need compliance tests, and very likely I would want a new virtual driver for this (vim2m.c would be a good template). Regards, Hans
On Fri, Aug 25, 2023 at 10:15 PM Hans Verkuil <hverkuil@xs4all.nl> wrote: > > On 25/08/2023 15:54, Takashi Iwai wrote: > > On Fri, 25 Aug 2023 05:46:43 +0200, > > Shengjiu Wang wrote: > >> > >> On Fri, Aug 25, 2023 at 4:21 AM Mark Brown <broonie@kernel.org> wrote: > >>> > >>> On Thu, Aug 24, 2023 at 07:03:09PM +0200, Takashi Iwai wrote: > >>>> Shengjiu Wang wrote: > >>> > >>>>> But there are several issues: > >>>>> 1. Need to create sound cards. ASRC module support multi instances, then > >>>>> need to create multi sound cards for each instance. > >>> > >>>> Hm, why can't it be multiple PCM instances instead? > >>> > >>> I'm having a hard time following this one too. > >>> > >>>>> 2. The ASRC is an entirety but with DPCM we need to separate input port and > >>>>> output port to playback substream and capture stream. Synchronous between > >>>>> playback substream and capture substream is a problem. > >>>>> How to start them and stop them at the same time. > >>> > >>>> This could be done by enforcing the full duplex and linking the both > >>>> PCM streams, I suppose. > >>> > >>>>> So shall we make the decision that we can go to the V4L2 solution? > >>> > >>>> Honestly speaking, I don't mind much whether it's implemented in V2L4 > >>>> or not -- at least for the kernel part, we can reorganize / refactor > >>>> things internally. But, the biggest remaining question to me is > >>>> whether this user-space interface is the most suitable one. Is it > >>>> well defined, usable and maintained for the audio applications? Or > >>>> is it meant to be a stop-gap for a specific use case? > >>> > >>> I'm having a really hard time summoning much enthusiasm for using v4l > >>> here, it feels like this is heading down the same bodge route as DPCM > >>> but directly as ABI so even harder to fix properly. That said all the > >>> ALSA APIs are really intended to be used in real time and this sounds > >>> like a non real time application? I don't fully understand what the > >>> actual use case is here. > >> > >> Thanks for your reply. > >> > >> This asrc memory to memory (memory ->asrc->memory) case is a non > >> real time use case. > >> > >> User fills the input buffer to the asrc module, after conversion, then asrc > >> sends back the output buffer to user. So it is not a traditional ALSA playback > >> and capture case. I think it is not good to create sound card for it, it is > >> not a sound card actually. > >> > >> It is a specific use case, there is no reference in current kernel. > >> v4l2 memory to memory is the closed implementation, v4l2 current > >> support video, image, radio, tuner, touch devices, so it is not > >> complicated to add support for this specific audio case. > >> > >> Maybe you can go through these patches first. Because we > >> had implemented the "memory -> asrc ->i2s device-> codec" > >> use case in ALSA. Now the "memory->asrc->memory" needs > >> to reuse the code in asrc driver, so the first 3 patches is for refining > >> the code to make it can be shared by the "memory->asrc->memory" > >> driver. > >> > >> The main change is in the v4l2 side, A /dev/vl42-audio will be created, > >> user applications only use the ioctl of v4l2 framework. > > > > Ah, now I'm slowly understanding. So, what you want is to have an > > interface to perform the batch conversion of a data stream from an > > input to an output? And ALSA PCM interface doesn't fit fully for that > > purpose because the data handling is batched and it's not like a > > normal PCM streaming? > > > > Basically the whole M2M arguments are rather subtle. Those are > > implementation details that can be resolved in several different ways > > in the kernel side. But the design of the operation is the crucial > > point. > > > > Maybe we can consider implementing a similar feature in ALSA API, too. > > But it's too far-stretched for now. > > > > So, if v4l2 interface provides the requested feature (the batched > > audio stream conversion), it's OK to ride on it. > > The V4L2 M2M interface is simple: you open a video device and then you can > pass data to the hardware, it processes it and you get the processed data back. > > The hardware just processes the data as fast as it can. Each time you open > the video device a new instance is created, and each instance can pass jobs > to the hardware. > > Currently it is used for video scalers, deinterlacers, colorspace converters and > codecs, but in the end it is just data in, data out with some job scheduling (fifo) > towards the hardware. So supporting audio using the same core m2m framework wouldn't > be a big deal. We'd probably make a /dev/v4l-audio device for that. > > It doesn't come for free: it is a new API, so besides adding support for it, it > also needs to be documented, we would need compliance tests, and very likely I > would want a new virtual driver for this (vim2m.c would be a good template). > Thanks all. I will try to pass the compliance test. Should the virtual driver be added now? Best regards Wang Shengiu
Hi, Le jeudi 24 août 2023 à 19:03 +0200, Takashi Iwai a écrit : > > 3. How to handle the xrun issue. pause or resume. which are brought by ALSA. > > Doesn't V4L2 handle the overrun/underrun at all? Also, no resume > support? just a 2 cents comment. All our video m2m are job based. When there is no job available they stop and resume when there is more work in queues. As there is no time constraints coming from the hardware, there is also no API to know that there has been a period of time without anything being executed (under utilization). Only overrun can be detected by application, each chunk of work is in its own v4l2_buffer and the application will run out of buffer in that case, and will have to wait for free space in the queue. Understand though that the larger the queue, the large the latency. There is currently no way to submit job ahead of the data (unlike DRM subsystem). I have slight impression that all this seems rather inefficient for high rate / small buffer. I'd suggest creating a dummy benchmark driver to verify that the overhead isn't just too much for an audio use case. Nicolas
Audio signal processing has the requirement for memory to memory similar as Video. This patch is to add this support in v4l2 framework, defined new buffer type V4L2_BUF_TYPE_AUDIO_CAPTURE and V4L2_BUF_TYPE_AUDIO_OUTPUT, defined new format v4l2_audio_format for audio case usage. The created audio device is named "/dev/audioX". And add memory to memory support for two kinds of i.MX ASRC module changes in v2: - decouple the implementation in v4l2 and ALSA - implement the memory to memory driver as a platfrom driver and move it to driver/media - move fsl_asrc_common.h to include/sound folder Shengjiu Wang (7): ASoC: fsl_asrc: define functions for memory to memory usage ASoC: fsl_easrc: define functions for memory to memory usage ASoC: fsl_asrc: move fsl_asrc_common.h to include/sound media: v4l2: Add audio capture and output support media: imx: fsl_asrc: Add memory to memory driver ASoC: fsl_asrc: register m2m platform device ASoC: fsl_easrc: register m2m platform device .../media/common/videobuf2/videobuf2-v4l2.c | 4 + drivers/media/platform/nxp/Kconfig | 12 + drivers/media/platform/nxp/Makefile | 1 + drivers/media/platform/nxp/fsl_asrc_m2m.c | 962 ++++++++++++++++++ drivers/media/v4l2-core/v4l2-dev.c | 17 + drivers/media/v4l2-core/v4l2-ioctl.c | 52 + include/media/v4l2-dev.h | 2 + include/media/v4l2-ioctl.h | 34 + .../fsl => include/sound}/fsl_asrc_common.h | 48 + include/uapi/linux/videodev2.h | 19 + sound/soc/fsl/fsl_asrc.c | 150 +++ sound/soc/fsl/fsl_asrc.h | 4 +- sound/soc/fsl/fsl_asrc_dma.c | 2 +- sound/soc/fsl/fsl_easrc.c | 227 +++++ sound/soc/fsl/fsl_easrc.h | 8 +- 15 files changed, 1539 insertions(+), 3 deletions(-) create mode 100644 drivers/media/platform/nxp/fsl_asrc_m2m.c rename {sound/soc/fsl => include/sound}/fsl_asrc_common.h (63%)