Message ID | 20250409110731.3752332-1-cezary.rojewski@intel.com (mailing list archive) |
---|---|
Headers | show |
Series | ALSA/ASoC: USB Audio Offload | expand |
On Wed, Apr 09, 2025 at 01:07:15PM +0200, Cezary Rojewski wrote: > Note: this series is based on Mark's broonie/for-next. The xHCI > dependency is missing so it won't compile. Goal of this RFC is > discussing the direction of sound/usb changes. > > Note #2: Why exclude xHCI? Current form of xhci-sideband.c [1] does not > fare well with Intel's hardware design for USB Audio Offload feature. > The production shape for usb/xHCI subjects is being discussed with > Mathias. Once we're ready, I'll share the rest. How is that going to work with the patches that add xhci offload that have already been reviewed on the list? > Note #3: this series does _NOT_ aim to block QCOM's equivalent series > [2]. The team does acknowledge that we came the "table" late. At the > same time, we're prepared to help QCOM switch to the presented sound/usb > approach if that would benefit the framework and its users as a whole. > Make it part of this very series if need be. I don't understand, I'm just about to take the QCOM patches now. They have been on the list and reviewed for years now, why hasn't this work happened before now? Please work to make your changes on top of that patch series. So what do you expect me to do here? confused, greg k-h
On 2025-04-09 2:10 PM, Greg KH wrote: > On Wed, Apr 09, 2025 at 01:07:15PM +0200, Cezary Rojewski wrote: >> Note: this series is based on Mark's broonie/for-next. The xHCI >> dependency is missing so it won't compile. Goal of this RFC is >> discussing the direction of sound/usb changes. >> >> Note #2: Why exclude xHCI? Current form of xhci-sideband.c [1] does not >> fare well with Intel's hardware design for USB Audio Offload feature. >> The production shape for usb/xHCI subjects is being discussed with >> Mathias. Once we're ready, I'll share the rest. > > How is that going to work with the patches that add xhci offload that > have already been reviewed on the list? > >> Note #3: this series does _NOT_ aim to block QCOM's equivalent series >> [2]. The team does acknowledge that we came the "table" late. At the >> same time, we're prepared to help QCOM switch to the presented sound/usb >> approach if that would benefit the framework and its users as a whole. >> Make it part of this very series if need be. > > I don't understand, I'm just about to take the QCOM patches now. They > have been on the list and reviewed for years now, why hasn't this work > happened before now? Please work to make your changes on top of that > patch series. > > So what do you expect me to do here? > > confused, Hi Greg, I wanted the effort to be transparent, that's it. Given the size of the task, I've decided to get the feedback earlier. I believe asking sound-maintainers now for their take on sound/usb is better than coming here much later and shooting everything at once. If the proposed approach to sound/usb is not welcomed by the sound maintainers, obviously xhci-sideband additions that will arrive here later will be impacted two. In regard to the sound changes alone, there is no intention to blow up the parallel solution. TLDR: Work present here will be placed on top of the QCOM patches, no question about it. xhci-sideband.c will receive a number of changes to cope with Intel's design but nothing the list haven't seen before. Kind regards, Czarek
On Wed, 09 Apr 2025 15:06:18 +0200, Cezary Rojewski wrote: > > On 2025-04-09 2:10 PM, Greg KH wrote: > > On Wed, Apr 09, 2025 at 01:07:15PM +0200, Cezary Rojewski wrote: > >> Note: this series is based on Mark's broonie/for-next. The xHCI > >> dependency is missing so it won't compile. Goal of this RFC is > >> discussing the direction of sound/usb changes. > >> > >> Note #2: Why exclude xHCI? Current form of xhci-sideband.c [1] does not > >> fare well with Intel's hardware design for USB Audio Offload feature. > >> The production shape for usb/xHCI subjects is being discussed with > >> Mathias. Once we're ready, I'll share the rest. > > > > How is that going to work with the patches that add xhci offload that > > have already been reviewed on the list? > > > >> Note #3: this series does _NOT_ aim to block QCOM's equivalent series > >> [2]. The team does acknowledge that we came the "table" late. At the > >> same time, we're prepared to help QCOM switch to the presented sound/usb > >> approach if that would benefit the framework and its users as a whole. > >> Make it part of this very series if need be. > > > > I don't understand, I'm just about to take the QCOM patches now. They > > have been on the list and reviewed for years now, why hasn't this work > > happened before now? Please work to make your changes on top of that > > patch series. > > > > So what do you expect me to do here? > > > > confused, > > Hi Greg, > > I wanted the effort to be transparent, that's it. Given the size of > the task, I've decided to get the feedback earlier. I believe asking > sound-maintainers now for their take on sound/usb is better than > coming here much later and shooting everything at once. If the > proposed approach to sound/usb is not welcomed by the sound > maintainers, obviously xhci-sideband additions that will arrive here > later will be impacted two. In regard to the sound changes alone, > there is no intention to blow up the parallel solution. > > TLDR: > Work present here will be placed on top of the QCOM patches, no > question about it. xhci-sideband.c will receive a number of changes > to cope with Intel's design but nothing the list haven't seen before. Thanks for the patches, but they appeared obviously too late to the game. Qcom patches are close to the merge, and I'd love to make things proceeded finally after many years struggle at first. The basic question, though again, is how the offloading is managed -- to which card or device is applied. The same question was applied to qcom implementation and their solution was to manage via the control API. The rest are rather opaque to users, and it allows us to even switch the complete implementation later to a different way like what you suggested, too. But it's only when the user-space will keep working as is. Takashi
On Wed, 09 Apr 2025 13:07:15 +0200, Cezary Rojewski wrote: > > Note: this series is based on Mark's broonie/for-next. The xHCI > dependency is missing so it won't compile. Goal of this RFC is > discussing the direction of sound/usb changes. > > Note #2: Why exclude xHCI? Current form of xhci-sideband.c [1] does not > fare well with Intel's hardware design for USB Audio Offload feature. > The production shape for usb/xHCI subjects is being discussed with > Mathias. Once we're ready, I'll share the rest. > > Note #3: this series does _NOT_ aim to block QCOM's equivalent series > [2]. The team does acknowledge that we came the "table" late. At the > same time, we're prepared to help QCOM switch to the presented sound/usb > approach if that would benefit the framework and its users as a whole. > Make it part of this very series if need be. > > > > Apart from changes to sound/usb, the patchset contains exemplary > ASoC-based, offload-aware USB driver and a small sound card on the > avs-driver side to show how card/dai_link initialization looks like. > > In short, USB Audio Offload functionality lowers CPU usage when > streaming PCM over a USB Audio-Class device. It has been first > introduced in xHCI 1.2 release and is described in section 7.9 [3]. > Once hardware is prepared with hw_params(), all the USB endpoints > operations e.g.: start, stop, submitting URBs, are performed > internally, by the hardware and AudioDSP firmware. Software driver > shall not intervene. > > Startup flow from: > > usb_pcm_open() > usb_pcm_hw_params() > snd_usb_endpoint_open() > usb_pcm_prepare() > usb_set_interface() > snd_usb_endpoint_start() > usb_pcm_trigger(cmd: START/STOP etc.) > > reduced to: > > usb_pcm_open() > usb_pcm_hw_params() > snd_usb_endpoint_open() > usb_pcm_prepare() > usb_set_interface() Hmm, how can it be? The start of EP at prepare stage is done only conditionally for non-lowlatency or implicit-feedback mode, for example. > Handlers such as ack(), sync_stop(), pointer(), delay() are not used > here too. The AudioDSP driver will handle pointer(), rest is > firmware/hardware responsibility. > > > There's a hefty number of limitations, most importantly: > > 1) typically 2 USB devices tops, rest go the classic (non-offload) path > 2) AUDIOSTREAMING interfaces only, MIDI not. While not an interface > type, Media (sound/usb/media) unsupported either > 3) simple PCM, UAC_FORMAT_TYPE_I_PCM only > > Current patchset shows this in form of 'udev->audsb_capable' field. > True if xHCI sideband reasource has been assigned to the device. The > filtering is code is not part of the patchset. > > Important to highlight, 1) means both, offload-aware and offload-unaware > drivers could be utilized simultaneously on the system in runtime. > Opportunistically few devices would be controlled by the offload-aware > driver, whereas everything else by the offload-unaware one. This > differs from existing HDAudio Controller driver situation where either > classic, snd_hda_intel driver takes complete control -or- the > offload-aware snd_soc_avs driver. > In short, once all Audio Sideband resources are depleted, classic > sound/usb/card.c driver manages whatever comes next: > > snd_usb_audio (offload un-aware, sound/usb/card.c) > snd_soc_usb_codec (offload aware, sound/soc/codecs/usb.c) > > > The design goals: > - make ASoC first class citizen of sound/usb > - re-use code found in sound/usb, mimic HDAudio integration in ASoC: > small sound/soc/codecs/hda.c driver leveraging power of entire > sound/pci/hda/ > - no shared control over a USB device, either snd_usb_audio or > its ASoC equivalent takes control of the device > > To do that, major tasks are identified: > > a) On ASoC side 'struct snd_card' is part of 'struct snd_soc_card' and > is managed by the framework. Similar situation with 'struct snd_pcm' > and rtd->pcm. To keep the teardown path sane, drop card->private_free() > and pcm->private_free() usage. Well, this is a generic problem of ASoC framework. I believe this should be better handled in ASoC core side at first. e.g. the card object could be created at the very first step of the snd_soc_card creation, too (but without the actual slot assignment or device creation). > b) To initialize ASoC components/DAI properly, PCM capabilities should be > known up-front. To do that, existing USB card probe() has to be split. > From one-stage to two-stage process: > > - look ahead and parse usb_interface descriptors for PCM endpoints > but do not create any PCMs (sound devices) yet > - create all PCMs based on obtained ->pcm_list and follow with > MIDI/mixers/media > > Such approach allows to feed DAIs proper data even when a valid > sound-card pointer is not yet present - the initialization occurs before > snd_soc_bind_card() is called. This one is another thing that is needed to adjust for ASoC framework. But when snd_card object is available, this can be resolved automatically, too? e.g. snd_pcm object or such can be created at that point. The actual device registration is done anyway later via snd_device_register() call. thanks, Takashi > Point a) is scaled for all three "domains" of sound/usb: chip, stream > and quirks. That's why there total of 6 commits doing that job. First > implement, then switch. Everything that follows is, in my opinion, > self-explanatory. No need to repeat commit messages. > > > [1]: https://lore.kernel.org/all/20250319005141.312805-2-quic_wcheng@quicinc.com/ > [2]: https://lore.kernel.org/all/20250319005141.312805-1-quic_wcheng@quicinc.com/ > [3]: https://www.intel.com/content/dam/www/public/us/en/documents/technical-specifications/extensible-host-controler-interface-usb-xhci.pdf > > > Cezary Rojewski (15): > ALSA: usb: Move media-filters to the media code > ALSA: usb: Drop private_free() usage for card and pcms > ALSA: usb: Relocate the usbaudio header file > ALSA: usb: Implement two-stage quirk applying mechanism > ALSA: usb: Implement two-stage stream creation mechanism > ALSA: usb: Implement two-stage chip probing mechanism > ALSA: usb: Switch to the two-stage chip probing > ALSA: usb: Switch to the two-stage stream creation > ALSA: usb: Switch to the two-stage quirk applying > ALSA: usb: Export PCM operations > ALSA: usb: Export usb_interface driver operations > ALSA: usb: Export card-naming procedure > ALSA: usb: Add getters to obtain endpoint information > ASoC: codecs: Add USB-Audio driver > ASoC: Intel: avs: Add USB machine board > > include/linux/usb.h | 15 + > include/linux/usb/ch9.h | 11 + > sound/usb/usbaudio.h => include/sound/usb.h | 53 +- > include/sound/usb_offload.h | 46 + > sound/soc/codecs/Kconfig | 6 + > sound/soc/codecs/Makefile | 2 + > sound/soc/codecs/usb.c | 441 +++++++++ > sound/soc/intel/avs/boards/Kconfig | 8 + > sound/soc/intel/avs/boards/Makefile | 2 + > sound/soc/intel/avs/boards/usb.c | 115 +++ > sound/usb/caiaq/device.h | 2 +- > sound/usb/card.c | 946 +++++++++++++------- > sound/usb/card.h | 4 +- > sound/usb/clock.c | 2 +- > sound/usb/endpoint.c | 2 +- > sound/usb/format.c | 2 +- > sound/usb/helper.c | 2 +- > sound/usb/implicit.c | 2 +- > sound/usb/media.c | 8 +- > sound/usb/midi.c | 2 +- > sound/usb/midi.h | 2 + > sound/usb/midi2.c | 2 +- > sound/usb/midi2.h | 1 + > sound/usb/misc/ua101.c | 2 +- > sound/usb/mixer.c | 2 +- > sound/usb/mixer_quirks.c | 2 +- > sound/usb/mixer_s1810c.c | 2 +- > sound/usb/mixer_scarlett.c | 2 +- > sound/usb/mixer_scarlett2.c | 2 +- > sound/usb/mixer_us16x08.c | 2 +- > sound/usb/pcm.c | 161 +++- > sound/usb/power.c | 2 +- > sound/usb/proc.c | 2 +- > sound/usb/quirks.c | 203 +++-- > sound/usb/quirks.h | 6 + > sound/usb/stream.c | 217 +++-- > sound/usb/stream.h | 2 + > sound/usb/usx2y/us122l.c | 2 +- > sound/usb/usx2y/usX2Yhwdep.c | 1 + > sound/usb/usx2y/usbusx2y.h | 2 +- > sound/usb/validate.c | 2 +- > 41 files changed, 1749 insertions(+), 541 deletions(-) > rename sound/usb/usbaudio.h => include/sound/usb.h (82%) > create mode 100644 include/sound/usb_offload.h > create mode 100644 sound/soc/codecs/usb.c > create mode 100644 sound/soc/intel/avs/boards/usb.c > > -- > 2.25.1 >
On 2025-04-10 12:24 PM, Takashi Iwai wrote: > On Wed, 09 Apr 2025 13:07:15 +0200, > Cezary Rojewski wrote: >> >> Note: this series is based on Mark's broonie/for-next. The xHCI >> dependency is missing so it won't compile. Goal of this RFC is >> discussing the direction of sound/usb changes. >> >> Note #2: Why exclude xHCI? Current form of xhci-sideband.c [1] does not >> fare well with Intel's hardware design for USB Audio Offload feature. >> The production shape for usb/xHCI subjects is being discussed with >> Mathias. Once we're ready, I'll share the rest. >> >> Note #3: this series does _NOT_ aim to block QCOM's equivalent series >> [2]. The team does acknowledge that we came the "table" late. At the >> same time, we're prepared to help QCOM switch to the presented sound/usb >> approach if that would benefit the framework and its users as a whole. >> Make it part of this very series if need be. >> >> >> >> Apart from changes to sound/usb, the patchset contains exemplary >> ASoC-based, offload-aware USB driver and a small sound card on the >> avs-driver side to show how card/dai_link initialization looks like. >> >> In short, USB Audio Offload functionality lowers CPU usage when >> streaming PCM over a USB Audio-Class device. It has been first >> introduced in xHCI 1.2 release and is described in section 7.9 [3]. >> Once hardware is prepared with hw_params(), all the USB endpoints >> operations e.g.: start, stop, submitting URBs, are performed >> internally, by the hardware and AudioDSP firmware. Software driver >> shall not intervene. >> >> Startup flow from: >> >> usb_pcm_open() >> usb_pcm_hw_params() >> snd_usb_endpoint_open() >> usb_pcm_prepare() >> usb_set_interface() >> snd_usb_endpoint_start() >> usb_pcm_trigger(cmd: START/STOP etc.) >> >> reduced to: >> >> usb_pcm_open() >> usb_pcm_hw_params() >> snd_usb_endpoint_open() >> usb_pcm_prepare() >> usb_set_interface() > > Hmm, how can it be? The start of EP at prepare stage is done only > conditionally for non-lowlatency or implicit-feedback mode, for > example. On Intel architecture the AudioDSP utilizes internal channel to communicate with xHCI and perform all the transfer operations. In essence, once SET_INTERFACE TRB is done, the usb-driver is not supposed to do anything. The avs-driver (sound driver) would be the trigger here - sends the start/stop/etc. requests to the AudioDSP firmware, DSP does the rest. By trigger I mean SET_PIPELINE_STATE IPC and avs_path_xxx() which are utilized for any transfer-type (sound/soc/intel/avs/pcm.c). >> Handlers such as ack(), sync_stop(), pointer(), delay() are not used >> here too. The AudioDSP driver will handle pointer(), rest is >> firmware/hardware responsibility. >> >> >> There's a hefty number of limitations, most importantly: >> >> 1) typically 2 USB devices tops, rest go the classic (non-offload) path >> 2) AUDIOSTREAMING interfaces only, MIDI not. While not an interface >> type, Media (sound/usb/media) unsupported either >> 3) simple PCM, UAC_FORMAT_TYPE_I_PCM only >> >> Current patchset shows this in form of 'udev->audsb_capable' field. >> True if xHCI sideband reasource has been assigned to the device. The >> filtering is code is not part of the patchset. >> >> Important to highlight, 1) means both, offload-aware and offload-unaware >> drivers could be utilized simultaneously on the system in runtime. >> Opportunistically few devices would be controlled by the offload-aware >> driver, whereas everything else by the offload-unaware one. This >> differs from existing HDAudio Controller driver situation where either >> classic, snd_hda_intel driver takes complete control -or- the >> offload-aware snd_soc_avs driver. >> In short, once all Audio Sideband resources are depleted, classic >> sound/usb/card.c driver manages whatever comes next: >> >> snd_usb_audio (offload un-aware, sound/usb/card.c) >> snd_soc_usb_codec (offload aware, sound/soc/codecs/usb.c) >> >> >> The design goals: >> - make ASoC first class citizen of sound/usb >> - re-use code found in sound/usb, mimic HDAudio integration in ASoC: >> small sound/soc/codecs/hda.c driver leveraging power of entire >> sound/pci/hda/ >> - no shared control over a USB device, either snd_usb_audio or >> its ASoC equivalent takes control of the device >> >> To do that, major tasks are identified: >> >> a) On ASoC side 'struct snd_card' is part of 'struct snd_soc_card' and >> is managed by the framework. Similar situation with 'struct snd_pcm' >> and rtd->pcm. To keep the teardown path sane, drop card->private_free() >> and pcm->private_free() usage. > > Well, this is a generic problem of ASoC framework. > I believe this should be better handled in ASoC core side at first. > e.g. the card object could be created at the very first step of the > snd_soc_card creation, too (but without the actual slot assignment or > device creation). Well, in my opinion the card's tailing private context (extra size), private_free() and all that comes with them in sound/core brings unnecessary complexity to the ALSA framework. devm_xxx() is enough. Polluting snd_card and sound/core/init.c with driver-specifics causes the teardown procedures on the ALSA framework side harder to read/maintain. >> b) To initialize ASoC components/DAI properly, PCM capabilities should be >> known up-front. To do that, existing USB card probe() has to be split. >> From one-stage to two-stage process: >> >> - look ahead and parse usb_interface descriptors for PCM endpoints >> but do not create any PCMs (sound devices) yet >> - create all PCMs based on obtained ->pcm_list and follow with >> MIDI/mixers/media >> >> Such approach allows to feed DAIs proper data even when a valid >> sound-card pointer is not yet present - the initialization occurs before >> snd_soc_bind_card() is called. > > This one is another thing that is needed to adjust for ASoC > framework. But when snd_card object is available, this can be > resolved automatically, too? e.g. snd_pcm object or such can be > created at that point. The actual device registration is done anyway > later via snd_device_register() call. While I'm up for upgrading either framework in any area necessary to have proper UAO support in ASoC, not sure whether it's a good idea to make it a requirement for the feature. This will enlarge the series - well, guess it will be separate series (dependency) entirely. >> Point a) is scaled for all three "domains" of sound/usb: chip, stream >> and quirks. That's why there total of 6 commits doing that job. First >> implement, then switch. Everything that follows is, in my opinion, >> self-explanatory. No need to repeat commit messages. >> >> >> [1]: https://lore.kernel.org/all/20250319005141.312805-2-quic_wcheng@quicinc.com/ >> [2]: https://lore.kernel.org/all/20250319005141.312805-1-quic_wcheng@quicinc.com/ >> [3]: https://www.intel.com/content/dam/www/public/us/en/documents/technical-specifications/extensible-host-controler-interface-usb-xhci.pdf
On Wed, Apr 09, 2025 at 01:07:15PM +0200, Cezary Rojewski wrote: > Note: this series is based on Mark's broonie/for-next. The xHCI > dependency is missing so it won't compile. Goal of this RFC is > discussing the direction of sound/usb changes. Any specific reason you aren't cc: linux-usb@vger.kernel.org on USB stuff like this? thanks, greg k-h
On 2025-04-11 4:04 PM, Greg KH wrote: > On Wed, Apr 09, 2025 at 01:07:15PM +0200, Cezary Rojewski wrote: >> Note: this series is based on Mark's broonie/for-next. The xHCI >> dependency is missing so it won't compile. Goal of this RFC is >> discussing the direction of sound/usb changes. > > Any specific reason you aren't cc: linux-usb@vger.kernel.org on USB > stuff like this? Actually I was expecting the exact opposite feedback had I included linux-usb, given lack of drivers/usb changes. Lesson learned. However, given the current feedback, for the next revision I plan to rebase onto your tree and include the xHCI changes. While I'm familiar with sound/ practices, I clearly was not with drivers/usb/ ones. The initial offload-aware vs offload-unaware USB audio-device filtering stuff was done by xHCI. Thanks to Mathias now I'm aware there is no place for such specific class code in drivers/usb/host. I'll try to get the usb changes adjusted as quickly as possible so that the list can see the entire thing. Kind regards, Czarek
On Fri, 11 Apr 2025 11:39:55 +0200, Cezary Rojewski wrote: > > On 2025-04-10 12:24 PM, Takashi Iwai wrote: > > On Wed, 09 Apr 2025 13:07:15 +0200, > > Cezary Rojewski wrote: > >> > >> Note: this series is based on Mark's broonie/for-next. The xHCI > >> dependency is missing so it won't compile. Goal of this RFC is > >> discussing the direction of sound/usb changes. > >> > >> Note #2: Why exclude xHCI? Current form of xhci-sideband.c [1] does not > >> fare well with Intel's hardware design for USB Audio Offload feature. > >> The production shape for usb/xHCI subjects is being discussed with > >> Mathias. Once we're ready, I'll share the rest. > >> > >> Note #3: this series does _NOT_ aim to block QCOM's equivalent series > >> [2]. The team does acknowledge that we came the "table" late. At the > >> same time, we're prepared to help QCOM switch to the presented sound/usb > >> approach if that would benefit the framework and its users as a whole. > >> Make it part of this very series if need be. > >> > >> > >> > >> Apart from changes to sound/usb, the patchset contains exemplary > >> ASoC-based, offload-aware USB driver and a small sound card on the > >> avs-driver side to show how card/dai_link initialization looks like. > >> > >> In short, USB Audio Offload functionality lowers CPU usage when > >> streaming PCM over a USB Audio-Class device. It has been first > >> introduced in xHCI 1.2 release and is described in section 7.9 [3]. > >> Once hardware is prepared with hw_params(), all the USB endpoints > >> operations e.g.: start, stop, submitting URBs, are performed > >> internally, by the hardware and AudioDSP firmware. Software driver > >> shall not intervene. > >> > >> Startup flow from: > >> > >> usb_pcm_open() > >> usb_pcm_hw_params() > >> snd_usb_endpoint_open() > >> usb_pcm_prepare() > >> usb_set_interface() > >> snd_usb_endpoint_start() > >> usb_pcm_trigger(cmd: START/STOP etc.) > >> > >> reduced to: > >> > >> usb_pcm_open() > >> usb_pcm_hw_params() > >> snd_usb_endpoint_open() > >> usb_pcm_prepare() > >> usb_set_interface() > > > > Hmm, how can it be? The start of EP at prepare stage is done only > > conditionally for non-lowlatency or implicit-feedback mode, for > > example. > > > On Intel architecture the AudioDSP utilizes internal channel to > communicate with xHCI and perform all the transfer operations. In > essence, once SET_INTERFACE TRB is done, the usb-driver is not > supposed to do anything. The avs-driver (sound driver) would be the > trigger here - sends the start/stop/etc. requests to the AudioDSP > firmware, DSP does the rest. > > By trigger I mean SET_PIPELINE_STATE IPC and avs_path_xxx() which are > utilized for any transfer-type (sound/soc/intel/avs/pcm.c). But does the DSP handles the different ways for the implicit feedback mode, the low-latency playback and else? A specific mode like the implicit feedback mode is mandatory for many devices. Similarly, the low-latency mode is essential for some application setups (e.g. pipewire or JACK prefers), while it's not always applicable depending on the PCM parameters (such as the free-wheeling mode). AFAIK, the qcom offloading doesn't support those fully, hence the configuration must be dynamic for them. > >> Handlers such as ack(), sync_stop(), pointer(), delay() are not used > >> here too. The AudioDSP driver will handle pointer(), rest is > >> firmware/hardware responsibility. > >> > >> > >> There's a hefty number of limitations, most importantly: > >> > >> 1) typically 2 USB devices tops, rest go the classic (non-offload) path > >> 2) AUDIOSTREAMING interfaces only, MIDI not. While not an interface > >> type, Media (sound/usb/media) unsupported either > >> 3) simple PCM, UAC_FORMAT_TYPE_I_PCM only > >> > >> Current patchset shows this in form of 'udev->audsb_capable' field. > >> True if xHCI sideband reasource has been assigned to the device. The > >> filtering is code is not part of the patchset. > >> > >> Important to highlight, 1) means both, offload-aware and offload-unaware > >> drivers could be utilized simultaneously on the system in runtime. > >> Opportunistically few devices would be controlled by the offload-aware > >> driver, whereas everything else by the offload-unaware one. This > >> differs from existing HDAudio Controller driver situation where either > >> classic, snd_hda_intel driver takes complete control -or- the > >> offload-aware snd_soc_avs driver. > >> In short, once all Audio Sideband resources are depleted, classic > >> sound/usb/card.c driver manages whatever comes next: > >> > >> snd_usb_audio (offload un-aware, sound/usb/card.c) > >> snd_soc_usb_codec (offload aware, sound/soc/codecs/usb.c) > >> > >> > >> The design goals: > >> - make ASoC first class citizen of sound/usb > >> - re-use code found in sound/usb, mimic HDAudio integration in ASoC: > >> small sound/soc/codecs/hda.c driver leveraging power of entire > >> sound/pci/hda/ > >> - no shared control over a USB device, either snd_usb_audio or > >> its ASoC equivalent takes control of the device > >> > >> To do that, major tasks are identified: > >> > >> a) On ASoC side 'struct snd_card' is part of 'struct snd_soc_card' and > >> is managed by the framework. Similar situation with 'struct snd_pcm' > >> and rtd->pcm. To keep the teardown path sane, drop card->private_free() > >> and pcm->private_free() usage. > > > > Well, this is a generic problem of ASoC framework. > > I believe this should be better handled in ASoC core side at first. > > e.g. the card object could be created at the very first step of the > > snd_soc_card creation, too (but without the actual slot assignment or > > device creation). > > > Well, in my opinion the card's tailing private context (extra size), > private_free() and all that comes with them in sound/core brings > unnecessary complexity to the ALSA framework. devm_xxx() is > enough. Polluting snd_card and sound/core/init.c with driver-specifics > causes the teardown procedures on the ALSA framework side harder to > read/maintain. Well, I'd say it depends, and pretty much a matter of taste. The resources managed by the card free should be the last thing that is tied with the card object itself. So, yes, it can be devm, but this wouldn't make things easier; the devm is merely a serialized release, after all. > >> b) To initialize ASoC components/DAI properly, PCM capabilities should be > >> known up-front. To do that, existing USB card probe() has to be split. > >> From one-stage to two-stage process: > >> > >> - look ahead and parse usb_interface descriptors for PCM endpoints > >> but do not create any PCMs (sound devices) yet > >> - create all PCMs based on obtained ->pcm_list and follow with > >> MIDI/mixers/media > >> > >> Such approach allows to feed DAIs proper data even when a valid > >> sound-card pointer is not yet present - the initialization occurs before > >> snd_soc_bind_card() is called. > > This one is another thing that is needed to adjust for ASoC > > framework. But when snd_card object is available, this can be > > resolved automatically, too? e.g. snd_pcm object or such can be > > created at that point. The actual device registration is done anyway > > later via snd_device_register() call. > > > While I'm up for upgrading either framework in any area necessary to > have proper UAO support in ASoC, not sure whether it's a good idea to > make it a requirement for the feature. This will enlarge the series - > well, guess it will be separate series (dependency) entirely. The series contains lots of hackish API exposure, and I really would like to avoid that if possible. In the case of HD-audio, the whole stuff was moved to its own bus at first for adapting to ASoC hd-audio ext implementation, but for USB-audio, I don't see the need for that yet. And, if the concern is about the snd_card object lifecycle and ASoC binding, it can be improved in the lower level at first. thanks, Takashi
On 2025-04-15 6:15 PM, Takashi Iwai wrote: > On Fri, 11 Apr 2025 11:39:55 +0200, > Cezary Rojewski wrote: >> >> On 2025-04-10 12:24 PM, Takashi Iwai wrote: >>> On Wed, 09 Apr 2025 13:07:15 +0200, >>> Cezary Rojewski wrote: ... >>>> Apart from changes to sound/usb, the patchset contains exemplary >>>> ASoC-based, offload-aware USB driver and a small sound card on the >>>> avs-driver side to show how card/dai_link initialization looks like. >>>> >>>> In short, USB Audio Offload functionality lowers CPU usage when >>>> streaming PCM over a USB Audio-Class device. It has been first >>>> introduced in xHCI 1.2 release and is described in section 7.9 [3]. >>>> Once hardware is prepared with hw_params(), all the USB endpoints >>>> operations e.g.: start, stop, submitting URBs, are performed >>>> internally, by the hardware and AudioDSP firmware. Software driver >>>> shall not intervene. >>>> >>>> Startup flow from: >>>> >>>> usb_pcm_open() >>>> usb_pcm_hw_params() >>>> snd_usb_endpoint_open() >>>> usb_pcm_prepare() >>>> usb_set_interface() >>>> snd_usb_endpoint_start() >>>> usb_pcm_trigger(cmd: START/STOP etc.) >>>> >>>> reduced to: >>>> >>>> usb_pcm_open() >>>> usb_pcm_hw_params() >>>> snd_usb_endpoint_open() >>>> usb_pcm_prepare() >>>> usb_set_interface() >>> >>> Hmm, how can it be? The start of EP at prepare stage is done only >>> conditionally for non-lowlatency or implicit-feedback mode, for >>> example. >> >> >> On Intel architecture the AudioDSP utilizes internal channel to >> communicate with xHCI and perform all the transfer operations. In >> essence, once SET_INTERFACE TRB is done, the usb-driver is not >> supposed to do anything. The avs-driver (sound driver) would be the >> trigger here - sends the start/stop/etc. requests to the AudioDSP >> firmware, DSP does the rest. >> >> By trigger I mean SET_PIPELINE_STATE IPC and avs_path_xxx() which are >> utilized for any transfer-type (sound/soc/intel/avs/pcm.c). > > But does the DSP handles the different ways for the implicit feedback > mode, the low-latency playback and else? A specific mode like the > implicit feedback mode is mandatory for many devices. Similarly, the > low-latency mode is essential for some application setups > (e.g. pipewire or JACK prefers), while it's not always applicable > depending on the PCM parameters (such as the free-wheeling mode). > > AFAIK, the qcom offloading doesn't support those fully, hence the > configuration must be dynamic for them. The hw design is unfortunately not simple: xHCI <> SIO <> ALH ^ USB side ^Audio DSP side It's an internal channel between so-called Audio Sideband located on xHCI controller side, goes through Scalable-IO and links with Audio Link Hub that's acts as 'front-end' for DSP. Every block has their own requirements and restrictions. And thus the number of available 'streams' in such configuration is very limited - typically to just 6. These are not even bi-direction streams but an internal map: IDs {1, 2} for outputs, IDs {3, 4, 5, 6} for inputs. Due to such small number of available streams, there is pressure to filter USB devices. In regard to endpoint types, only DATA and FEEDBACK are supported, IMPLICIT_FEEDBACK is not. Such device should fail offload-candidate filtering and be serviced by snd_usb_audio driver instead. The official xHCI spec and its section regarding Audio Sideband is tailored for "claim entire device for offload or ignore" approach: - scan the USB device descriptors - verify if it's an offload-candidate - count number of DATA + FEEDBACK endpoints - ask xHCI to _reserve_ Audio Sideband resources up-front. This means the resources are reserved long before sound-devices are visible from userspace, yet alone opened for streaming - pcm->open/close() just ask xHCI to perform SET_ASSIGNMENT(reserved_resources_here) TRBs to prepare/close the internal channel for runtime operations, that's it There is no intention to allow for switching the ownership between offload-aware and offload-unaware. Either entire USB device and all its endpoints have Audio Sideband resources pre-allocated or no offload is present for the device at all. ... >>>> The design goals: >>>> - make ASoC first class citizen of sound/usb >>>> - re-use code found in sound/usb, mimic HDAudio integration in ASoC: >>>> small sound/soc/codecs/hda.c driver leveraging power of entire >>>> sound/pci/hda/ >>>> - no shared control over a USB device, either snd_usb_audio or >>>> its ASoC equivalent takes control of the device >>>> >>>> To do that, major tasks are identified: >>>> >>>> a) On ASoC side 'struct snd_card' is part of 'struct snd_soc_card' and >>>> is managed by the framework. Similar situation with 'struct snd_pcm' >>>> and rtd->pcm. To keep the teardown path sane, drop card->private_free() >>>> and pcm->private_free() usage. >>> >>> Well, this is a generic problem of ASoC framework. >>> I believe this should be better handled in ASoC core side at first. >>> e.g. the card object could be created at the very first step of the >>> snd_soc_card creation, too (but without the actual slot assignment or >>> device creation). >> >> >> Well, in my opinion the card's tailing private context (extra size), >> private_free() and all that comes with them in sound/core brings >> unnecessary complexity to the ALSA framework. devm_xxx() is >> enough. Polluting snd_card and sound/core/init.c with driver-specifics >> causes the teardown procedures on the ALSA framework side harder to >> read/maintain. > > Well, I'd say it depends, and pretty much a matter of taste. The > resources managed by the card free should be the last thing that is > tied with the card object itself. So, yes, it can be devm, but this > wouldn't make things easier; the devm is merely a serialized release, > after all. > >>>> b) To initialize ASoC components/DAI properly, PCM capabilities should be >>>> known up-front. To do that, existing USB card probe() has to be split. >>>> From one-stage to two-stage process: >>>> >>>> - look ahead and parse usb_interface descriptors for PCM endpoints >>>> but do not create any PCMs (sound devices) yet >>>> - create all PCMs based on obtained ->pcm_list and follow with >>>> MIDI/mixers/media >>>> >>>> Such approach allows to feed DAIs proper data even when a valid >>>> sound-card pointer is not yet present - the initialization occurs before >>>> snd_soc_bind_card() is called. >>> This one is another thing that is needed to adjust for ASoC >>> framework. But when snd_card object is available, this can be >>> resolved automatically, too? e.g. snd_pcm object or such can be >>> created at that point. The actual device registration is done anyway >>> later via snd_device_register() call. >> >> >> While I'm up for upgrading either framework in any area necessary to >> have proper UAO support in ASoC, not sure whether it's a good idea to >> make it a requirement for the feature. This will enlarge the series - >> well, guess it will be separate series (dependency) entirely. > > The series contains lots of hackish API exposure, and I really would > like to avoid that if possible. In the case of HD-audio, the whole > stuff was moved to its own bus at first for adapting to ASoC hd-audio > ext implementation, but for USB-audio, I don't see the need for that > yet. And, if the concern is about the snd_card object lifecycle and > ASoC binding, it can be improved in the lower level at first. I tried hard to avoid hackish stuff, perhaps not hard enough :) Would it be possible to list the areas which you believe need to be look at? This is frankly the feedback which I'm most interested in. Mathias gave me a number of these, e.g.: do not access udev->slot_id in sound/, avoid manipulating hcd/controller in sound/ and such. In regard to the HDAudio point, I see clear benefits by having HDAudio and USB aligned in the approach on ASoC side. It's a path that's known, works and is well tested. In regard to the snd_soc_card vs snd_card, in my opinion the refactor of card initialization is a large task. In the past I did similar change for the snd_soc_component [1] but the card is an entirely different beast. That's why I'm suggesting incremental approach - update ASoC initialization as a next step. [1]: https://lore.kernel.org/all/20200731144146.6678-1-cezary.rojewski@intel.com/ Kind regards, Czarek