mbox series

[RFC,00/15] ALSA/ASoC: USB Audio Offload

Message ID 20250409110731.3752332-1-cezary.rojewski@intel.com (mailing list archive)
Headers show
Series ALSA/ASoC: USB Audio Offload | expand

Message

Cezary Rojewski April 9, 2025, 11:07 a.m. UTC
Note: this series is based on Mark's broonie/for-next. The xHCI
dependency is missing so it won't compile.  Goal of this RFC is
discussing the direction of sound/usb changes.

Note #2: Why exclude xHCI?  Current form of xhci-sideband.c [1] does not
fare well with Intel's hardware design for USB Audio Offload feature.
The production shape for usb/xHCI subjects is being discussed with
Mathias.  Once we're ready, I'll share the rest.

Note #3: this series does _NOT_ aim to block QCOM's equivalent series
[2].  The team does acknowledge that we came the "table" late.  At the
same time, we're prepared to help QCOM switch to the presented sound/usb
approach if that would benefit the framework and its users as a whole.
Make it part of this very series if need be.



Apart from changes to sound/usb, the patchset contains exemplary
ASoC-based, offload-aware USB driver and a small sound card on the
avs-driver side to show how card/dai_link initialization looks like.

In short, USB Audio Offload functionality lowers CPU usage when
streaming PCM over a USB Audio-Class device.  It has been first
introduced in xHCI 1.2 release and is described in section 7.9 [3].
Once hardware is prepared with hw_params(), all the USB endpoints
operations e.g.: start, stop, submitting URBs, are performed
internally, by the hardware and AudioDSP firmware.  Software driver
shall not intervene.

Startup flow from:

	usb_pcm_open()
	usb_pcm_hw_params()
		snd_usb_endpoint_open()
	usb_pcm_prepare()
		usb_set_interface()
		snd_usb_endpoint_start()
	usb_pcm_trigger(cmd: START/STOP etc.)

reduced to:

	usb_pcm_open()
	usb_pcm_hw_params()
		snd_usb_endpoint_open()
	usb_pcm_prepare()
		usb_set_interface()

Handlers such as ack(), sync_stop(), pointer(), delay() are not used
here too.  The AudioDSP driver will handle pointer(), rest is
firmware/hardware responsibility.


There's a hefty number of limitations, most importantly:

1) typically 2 USB devices tops, rest go the classic (non-offload) path
2) AUDIOSTREAMING interfaces only, MIDI not.  While not an interface
   type, Media (sound/usb/media) unsupported either
3) simple PCM, UAC_FORMAT_TYPE_I_PCM only

Current patchset shows this in form of 'udev->audsb_capable' field.
True if xHCI sideband reasource has been assigned to the device.  The
filtering is code is not part of the patchset.

Important to highlight, 1) means both, offload-aware and offload-unaware
drivers could be utilized simultaneously on the system in runtime.
Opportunistically few devices would be controlled by the offload-aware
driver, whereas everything else by the offload-unaware one.  This
differs from existing HDAudio Controller driver situation where either
classic, snd_hda_intel driver takes complete control -or- the
offload-aware snd_soc_avs driver.
In short, once all Audio Sideband resources are depleted, classic
sound/usb/card.c driver manages whatever comes next:

	snd_usb_audio (offload un-aware, sound/usb/card.c)
	snd_soc_usb_codec (offload aware, sound/soc/codecs/usb.c)


The design goals:
	- make ASoC first class citizen of sound/usb
	- re-use code found in sound/usb, mimic HDAudio integration in ASoC:
	  small sound/soc/codecs/hda.c driver leveraging power of entire
	  sound/pci/hda/
	- no shared control over a USB device, either snd_usb_audio or
	  its ASoC equivalent takes control of the device

To do that, major tasks are identified:

a) On ASoC side 'struct snd_card' is part of 'struct snd_soc_card' and
is managed by the framework.  Similar situation with 'struct snd_pcm'
and rtd->pcm.  To keep the teardown path sane, drop card->private_free()
and pcm->private_free() usage.

b) To initialize ASoC components/DAI properly, PCM capabilities should be
known up-front.  To do that, existing USB card probe() has to be split.
From one-stage to two-stage process:

	- look ahead and parse usb_interface descriptors for PCM endpoints
	  but do not create any PCMs (sound devices) yet
	- create all PCMs based on obtained ->pcm_list and follow with
	  MIDI/mixers/media

Such approach allows to feed DAIs proper data even when a valid
sound-card pointer is not yet present - the initialization occurs before
snd_soc_bind_card() is called.

Point a) is scaled for all three "domains" of sound/usb: chip, stream
and quirks.  That's why there total of 6 commits doing that job.  First
implement, then switch.  Everything that follows is, in my opinion,
self-explanatory.  No need to repeat commit messages.


[1]: https://lore.kernel.org/all/20250319005141.312805-2-quic_wcheng@quicinc.com/
[2]: https://lore.kernel.org/all/20250319005141.312805-1-quic_wcheng@quicinc.com/
[3]: https://www.intel.com/content/dam/www/public/us/en/documents/technical-specifications/extensible-host-controler-interface-usb-xhci.pdf


Cezary Rojewski (15):
  ALSA: usb: Move media-filters to the media code
  ALSA: usb: Drop private_free() usage for card and pcms
  ALSA: usb: Relocate the usbaudio header file
  ALSA: usb: Implement two-stage quirk applying mechanism
  ALSA: usb: Implement two-stage stream creation mechanism
  ALSA: usb: Implement two-stage chip probing mechanism
  ALSA: usb: Switch to the two-stage chip probing
  ALSA: usb: Switch to the two-stage stream creation
  ALSA: usb: Switch to the two-stage quirk applying
  ALSA: usb: Export PCM operations
  ALSA: usb: Export usb_interface driver operations
  ALSA: usb: Export card-naming procedure
  ALSA: usb: Add getters to obtain endpoint information
  ASoC: codecs: Add USB-Audio driver
  ASoC: Intel: avs: Add USB machine board

 include/linux/usb.h                         |  15 +
 include/linux/usb/ch9.h                     |  11 +
 sound/usb/usbaudio.h => include/sound/usb.h |  53 +-
 include/sound/usb_offload.h                 |  46 +
 sound/soc/codecs/Kconfig                    |   6 +
 sound/soc/codecs/Makefile                   |   2 +
 sound/soc/codecs/usb.c                      | 441 +++++++++
 sound/soc/intel/avs/boards/Kconfig          |   8 +
 sound/soc/intel/avs/boards/Makefile         |   2 +
 sound/soc/intel/avs/boards/usb.c            | 115 +++
 sound/usb/caiaq/device.h                    |   2 +-
 sound/usb/card.c                            | 946 +++++++++++++-------
 sound/usb/card.h                            |   4 +-
 sound/usb/clock.c                           |   2 +-
 sound/usb/endpoint.c                        |   2 +-
 sound/usb/format.c                          |   2 +-
 sound/usb/helper.c                          |   2 +-
 sound/usb/implicit.c                        |   2 +-
 sound/usb/media.c                           |   8 +-
 sound/usb/midi.c                            |   2 +-
 sound/usb/midi.h                            |   2 +
 sound/usb/midi2.c                           |   2 +-
 sound/usb/midi2.h                           |   1 +
 sound/usb/misc/ua101.c                      |   2 +-
 sound/usb/mixer.c                           |   2 +-
 sound/usb/mixer_quirks.c                    |   2 +-
 sound/usb/mixer_s1810c.c                    |   2 +-
 sound/usb/mixer_scarlett.c                  |   2 +-
 sound/usb/mixer_scarlett2.c                 |   2 +-
 sound/usb/mixer_us16x08.c                   |   2 +-
 sound/usb/pcm.c                             | 161 +++-
 sound/usb/power.c                           |   2 +-
 sound/usb/proc.c                            |   2 +-
 sound/usb/quirks.c                          | 203 +++--
 sound/usb/quirks.h                          |   6 +
 sound/usb/stream.c                          | 217 +++--
 sound/usb/stream.h                          |   2 +
 sound/usb/usx2y/us122l.c                    |   2 +-
 sound/usb/usx2y/usX2Yhwdep.c                |   1 +
 sound/usb/usx2y/usbusx2y.h                  |   2 +-
 sound/usb/validate.c                        |   2 +-
 41 files changed, 1749 insertions(+), 541 deletions(-)
 rename sound/usb/usbaudio.h => include/sound/usb.h (82%)
 create mode 100644 include/sound/usb_offload.h
 create mode 100644 sound/soc/codecs/usb.c
 create mode 100644 sound/soc/intel/avs/boards/usb.c

Comments

Greg Kroah-Hartman April 9, 2025, 12:10 p.m. UTC | #1
On Wed, Apr 09, 2025 at 01:07:15PM +0200, Cezary Rojewski wrote:
> Note: this series is based on Mark's broonie/for-next. The xHCI
> dependency is missing so it won't compile.  Goal of this RFC is
> discussing the direction of sound/usb changes.
> 
> Note #2: Why exclude xHCI?  Current form of xhci-sideband.c [1] does not
> fare well with Intel's hardware design for USB Audio Offload feature.
> The production shape for usb/xHCI subjects is being discussed with
> Mathias.  Once we're ready, I'll share the rest.

How is that going to work with the patches that add xhci offload that
have already been reviewed on the list?

> Note #3: this series does _NOT_ aim to block QCOM's equivalent series
> [2].  The team does acknowledge that we came the "table" late.  At the
> same time, we're prepared to help QCOM switch to the presented sound/usb
> approach if that would benefit the framework and its users as a whole.
> Make it part of this very series if need be.

I don't understand, I'm just about to take the QCOM patches now.  They
have been on the list and reviewed for years now, why hasn't this work
happened before now?  Please work to make your changes on top of that
patch series.

So what do you expect me to do here?

confused,

greg k-h
Cezary Rojewski April 9, 2025, 1:06 p.m. UTC | #2
On 2025-04-09 2:10 PM, Greg KH wrote:
> On Wed, Apr 09, 2025 at 01:07:15PM +0200, Cezary Rojewski wrote:
>> Note: this series is based on Mark's broonie/for-next. The xHCI
>> dependency is missing so it won't compile.  Goal of this RFC is
>> discussing the direction of sound/usb changes.
>>
>> Note #2: Why exclude xHCI?  Current form of xhci-sideband.c [1] does not
>> fare well with Intel's hardware design for USB Audio Offload feature.
>> The production shape for usb/xHCI subjects is being discussed with
>> Mathias.  Once we're ready, I'll share the rest.
> 
> How is that going to work with the patches that add xhci offload that
> have already been reviewed on the list?
> 
>> Note #3: this series does _NOT_ aim to block QCOM's equivalent series
>> [2].  The team does acknowledge that we came the "table" late.  At the
>> same time, we're prepared to help QCOM switch to the presented sound/usb
>> approach if that would benefit the framework and its users as a whole.
>> Make it part of this very series if need be.
> 
> I don't understand, I'm just about to take the QCOM patches now.  They
> have been on the list and reviewed for years now, why hasn't this work
> happened before now?  Please work to make your changes on top of that
> patch series.
> 
> So what do you expect me to do here?
> 
> confused,

Hi Greg,

I wanted the effort to be transparent, that's it.  Given the size of the 
task, I've decided to get the feedback earlier.  I believe asking 
sound-maintainers now for their take on sound/usb is better than coming 
here much later and shooting everything at once.  If the proposed 
approach to sound/usb is not welcomed by the sound maintainers, 
obviously xhci-sideband additions that will arrive here later will be 
impacted two.  In regard to the sound changes alone, there is no 
intention to blow up the parallel solution.

TLDR:
Work present here will be placed on top of the QCOM patches, no question 
about it.  xhci-sideband.c will receive a number of changes to cope with 
Intel's design but nothing the list haven't seen before.


Kind regards,
Czarek
Takashi Iwai April 10, 2025, 10:10 a.m. UTC | #3
On Wed, 09 Apr 2025 15:06:18 +0200,
Cezary Rojewski wrote:
> 
> On 2025-04-09 2:10 PM, Greg KH wrote:
> > On Wed, Apr 09, 2025 at 01:07:15PM +0200, Cezary Rojewski wrote:
> >> Note: this series is based on Mark's broonie/for-next. The xHCI
> >> dependency is missing so it won't compile.  Goal of this RFC is
> >> discussing the direction of sound/usb changes.
> >> 
> >> Note #2: Why exclude xHCI?  Current form of xhci-sideband.c [1] does not
> >> fare well with Intel's hardware design for USB Audio Offload feature.
> >> The production shape for usb/xHCI subjects is being discussed with
> >> Mathias.  Once we're ready, I'll share the rest.
> > 
> > How is that going to work with the patches that add xhci offload that
> > have already been reviewed on the list?
> > 
> >> Note #3: this series does _NOT_ aim to block QCOM's equivalent series
> >> [2].  The team does acknowledge that we came the "table" late.  At the
> >> same time, we're prepared to help QCOM switch to the presented sound/usb
> >> approach if that would benefit the framework and its users as a whole.
> >> Make it part of this very series if need be.
> > 
> > I don't understand, I'm just about to take the QCOM patches now.  They
> > have been on the list and reviewed for years now, why hasn't this work
> > happened before now?  Please work to make your changes on top of that
> > patch series.
> > 
> > So what do you expect me to do here?
> > 
> > confused,
> 
> Hi Greg,
> 
> I wanted the effort to be transparent, that's it.  Given the size of
> the task, I've decided to get the feedback earlier.  I believe asking
> sound-maintainers now for their take on sound/usb is better than
> coming here much later and shooting everything at once.  If the
> proposed approach to sound/usb is not welcomed by the sound
> maintainers, obviously xhci-sideband additions that will arrive here
> later will be impacted two.  In regard to the sound changes alone,
> there is no intention to blow up the parallel solution.
> 
> TLDR:
> Work present here will be placed on top of the QCOM patches, no
> question about it.  xhci-sideband.c will receive a number of changes
> to cope with Intel's design but nothing the list haven't seen before.

Thanks for the patches, but they appeared obviously too late to the
game.  Qcom patches are close to the merge, and I'd love to make
things proceeded finally after many years struggle at first.

The basic question, though again, is how the offloading is managed --
to which card or device is applied.  The same question was applied to
qcom implementation and their solution was to manage via the control
API.  The rest are rather opaque to users, and it allows us to even
switch the complete implementation later to a different way like what
you suggested, too.  But it's only when the user-space will keep
working as is.


Takashi
Takashi Iwai April 10, 2025, 10:24 a.m. UTC | #4
On Wed, 09 Apr 2025 13:07:15 +0200,
Cezary Rojewski wrote:
> 
> Note: this series is based on Mark's broonie/for-next. The xHCI
> dependency is missing so it won't compile.  Goal of this RFC is
> discussing the direction of sound/usb changes.
> 
> Note #2: Why exclude xHCI?  Current form of xhci-sideband.c [1] does not
> fare well with Intel's hardware design for USB Audio Offload feature.
> The production shape for usb/xHCI subjects is being discussed with
> Mathias.  Once we're ready, I'll share the rest.
> 
> Note #3: this series does _NOT_ aim to block QCOM's equivalent series
> [2].  The team does acknowledge that we came the "table" late.  At the
> same time, we're prepared to help QCOM switch to the presented sound/usb
> approach if that would benefit the framework and its users as a whole.
> Make it part of this very series if need be.
> 
> 
> 
> Apart from changes to sound/usb, the patchset contains exemplary
> ASoC-based, offload-aware USB driver and a small sound card on the
> avs-driver side to show how card/dai_link initialization looks like.
> 
> In short, USB Audio Offload functionality lowers CPU usage when
> streaming PCM over a USB Audio-Class device.  It has been first
> introduced in xHCI 1.2 release and is described in section 7.9 [3].
> Once hardware is prepared with hw_params(), all the USB endpoints
> operations e.g.: start, stop, submitting URBs, are performed
> internally, by the hardware and AudioDSP firmware.  Software driver
> shall not intervene.
> 
> Startup flow from:
> 
> 	usb_pcm_open()
> 	usb_pcm_hw_params()
> 		snd_usb_endpoint_open()
> 	usb_pcm_prepare()
> 		usb_set_interface()
> 		snd_usb_endpoint_start()
> 	usb_pcm_trigger(cmd: START/STOP etc.)
> 
> reduced to:
> 
> 	usb_pcm_open()
> 	usb_pcm_hw_params()
> 		snd_usb_endpoint_open()
> 	usb_pcm_prepare()
> 		usb_set_interface()

Hmm, how can it be?  The start of EP at prepare stage is done only
conditionally for non-lowlatency or implicit-feedback mode, for
example.

> Handlers such as ack(), sync_stop(), pointer(), delay() are not used
> here too.  The AudioDSP driver will handle pointer(), rest is
> firmware/hardware responsibility.
> 
> 
> There's a hefty number of limitations, most importantly:
> 
> 1) typically 2 USB devices tops, rest go the classic (non-offload) path
> 2) AUDIOSTREAMING interfaces only, MIDI not.  While not an interface
>    type, Media (sound/usb/media) unsupported either
> 3) simple PCM, UAC_FORMAT_TYPE_I_PCM only
> 
> Current patchset shows this in form of 'udev->audsb_capable' field.
> True if xHCI sideband reasource has been assigned to the device.  The
> filtering is code is not part of the patchset.
> 
> Important to highlight, 1) means both, offload-aware and offload-unaware
> drivers could be utilized simultaneously on the system in runtime.
> Opportunistically few devices would be controlled by the offload-aware
> driver, whereas everything else by the offload-unaware one.  This
> differs from existing HDAudio Controller driver situation where either
> classic, snd_hda_intel driver takes complete control -or- the
> offload-aware snd_soc_avs driver.
> In short, once all Audio Sideband resources are depleted, classic
> sound/usb/card.c driver manages whatever comes next:
> 
> 	snd_usb_audio (offload un-aware, sound/usb/card.c)
> 	snd_soc_usb_codec (offload aware, sound/soc/codecs/usb.c)
> 
> 
> The design goals:
> 	- make ASoC first class citizen of sound/usb
> 	- re-use code found in sound/usb, mimic HDAudio integration in ASoC:
> 	  small sound/soc/codecs/hda.c driver leveraging power of entire
> 	  sound/pci/hda/
> 	- no shared control over a USB device, either snd_usb_audio or
> 	  its ASoC equivalent takes control of the device
> 
> To do that, major tasks are identified:
> 
> a) On ASoC side 'struct snd_card' is part of 'struct snd_soc_card' and
> is managed by the framework.  Similar situation with 'struct snd_pcm'
> and rtd->pcm.  To keep the teardown path sane, drop card->private_free()
> and pcm->private_free() usage.

Well, this is a generic problem of ASoC framework.
I believe this should be better handled in ASoC core side at first.
e.g. the card object could be created at the very first step of the
snd_soc_card creation, too (but without the actual slot assignment or
device creation).

> b) To initialize ASoC components/DAI properly, PCM capabilities should be
> known up-front.  To do that, existing USB card probe() has to be split.
> From one-stage to two-stage process:
> 
> 	- look ahead and parse usb_interface descriptors for PCM endpoints
> 	  but do not create any PCMs (sound devices) yet
> 	- create all PCMs based on obtained ->pcm_list and follow with
> 	  MIDI/mixers/media
> 
> Such approach allows to feed DAIs proper data even when a valid
> sound-card pointer is not yet present - the initialization occurs before
> snd_soc_bind_card() is called.
 
This one is another thing that is needed to adjust for ASoC
framework.  But when snd_card object is available, this can be
resolved automatically, too?  e.g. snd_pcm object or such can be
created at that point.  The actual device registration is done anyway
later via snd_device_register() call.


thanks,

Takashi

> Point a) is scaled for all three "domains" of sound/usb: chip, stream
> and quirks.  That's why there total of 6 commits doing that job.  First
> implement, then switch.  Everything that follows is, in my opinion,
> self-explanatory.  No need to repeat commit messages.
> 
> 
> [1]: https://lore.kernel.org/all/20250319005141.312805-2-quic_wcheng@quicinc.com/
> [2]: https://lore.kernel.org/all/20250319005141.312805-1-quic_wcheng@quicinc.com/
> [3]: https://www.intel.com/content/dam/www/public/us/en/documents/technical-specifications/extensible-host-controler-interface-usb-xhci.pdf
> 
> 
> Cezary Rojewski (15):
>   ALSA: usb: Move media-filters to the media code
>   ALSA: usb: Drop private_free() usage for card and pcms
>   ALSA: usb: Relocate the usbaudio header file
>   ALSA: usb: Implement two-stage quirk applying mechanism
>   ALSA: usb: Implement two-stage stream creation mechanism
>   ALSA: usb: Implement two-stage chip probing mechanism
>   ALSA: usb: Switch to the two-stage chip probing
>   ALSA: usb: Switch to the two-stage stream creation
>   ALSA: usb: Switch to the two-stage quirk applying
>   ALSA: usb: Export PCM operations
>   ALSA: usb: Export usb_interface driver operations
>   ALSA: usb: Export card-naming procedure
>   ALSA: usb: Add getters to obtain endpoint information
>   ASoC: codecs: Add USB-Audio driver
>   ASoC: Intel: avs: Add USB machine board
> 
>  include/linux/usb.h                         |  15 +
>  include/linux/usb/ch9.h                     |  11 +
>  sound/usb/usbaudio.h => include/sound/usb.h |  53 +-
>  include/sound/usb_offload.h                 |  46 +
>  sound/soc/codecs/Kconfig                    |   6 +
>  sound/soc/codecs/Makefile                   |   2 +
>  sound/soc/codecs/usb.c                      | 441 +++++++++
>  sound/soc/intel/avs/boards/Kconfig          |   8 +
>  sound/soc/intel/avs/boards/Makefile         |   2 +
>  sound/soc/intel/avs/boards/usb.c            | 115 +++
>  sound/usb/caiaq/device.h                    |   2 +-
>  sound/usb/card.c                            | 946 +++++++++++++-------
>  sound/usb/card.h                            |   4 +-
>  sound/usb/clock.c                           |   2 +-
>  sound/usb/endpoint.c                        |   2 +-
>  sound/usb/format.c                          |   2 +-
>  sound/usb/helper.c                          |   2 +-
>  sound/usb/implicit.c                        |   2 +-
>  sound/usb/media.c                           |   8 +-
>  sound/usb/midi.c                            |   2 +-
>  sound/usb/midi.h                            |   2 +
>  sound/usb/midi2.c                           |   2 +-
>  sound/usb/midi2.h                           |   1 +
>  sound/usb/misc/ua101.c                      |   2 +-
>  sound/usb/mixer.c                           |   2 +-
>  sound/usb/mixer_quirks.c                    |   2 +-
>  sound/usb/mixer_s1810c.c                    |   2 +-
>  sound/usb/mixer_scarlett.c                  |   2 +-
>  sound/usb/mixer_scarlett2.c                 |   2 +-
>  sound/usb/mixer_us16x08.c                   |   2 +-
>  sound/usb/pcm.c                             | 161 +++-
>  sound/usb/power.c                           |   2 +-
>  sound/usb/proc.c                            |   2 +-
>  sound/usb/quirks.c                          | 203 +++--
>  sound/usb/quirks.h                          |   6 +
>  sound/usb/stream.c                          | 217 +++--
>  sound/usb/stream.h                          |   2 +
>  sound/usb/usx2y/us122l.c                    |   2 +-
>  sound/usb/usx2y/usX2Yhwdep.c                |   1 +
>  sound/usb/usx2y/usbusx2y.h                  |   2 +-
>  sound/usb/validate.c                        |   2 +-
>  41 files changed, 1749 insertions(+), 541 deletions(-)
>  rename sound/usb/usbaudio.h => include/sound/usb.h (82%)
>  create mode 100644 include/sound/usb_offload.h
>  create mode 100644 sound/soc/codecs/usb.c
>  create mode 100644 sound/soc/intel/avs/boards/usb.c
> 
> -- 
> 2.25.1
>
Cezary Rojewski April 11, 2025, 9:39 a.m. UTC | #5
On 2025-04-10 12:24 PM, Takashi Iwai wrote:
> On Wed, 09 Apr 2025 13:07:15 +0200,
> Cezary Rojewski wrote:
>>
>> Note: this series is based on Mark's broonie/for-next. The xHCI
>> dependency is missing so it won't compile.  Goal of this RFC is
>> discussing the direction of sound/usb changes.
>>
>> Note #2: Why exclude xHCI?  Current form of xhci-sideband.c [1] does not
>> fare well with Intel's hardware design for USB Audio Offload feature.
>> The production shape for usb/xHCI subjects is being discussed with
>> Mathias.  Once we're ready, I'll share the rest.
>>
>> Note #3: this series does _NOT_ aim to block QCOM's equivalent series
>> [2].  The team does acknowledge that we came the "table" late.  At the
>> same time, we're prepared to help QCOM switch to the presented sound/usb
>> approach if that would benefit the framework and its users as a whole.
>> Make it part of this very series if need be.
>>
>>
>>
>> Apart from changes to sound/usb, the patchset contains exemplary
>> ASoC-based, offload-aware USB driver and a small sound card on the
>> avs-driver side to show how card/dai_link initialization looks like.
>>
>> In short, USB Audio Offload functionality lowers CPU usage when
>> streaming PCM over a USB Audio-Class device.  It has been first
>> introduced in xHCI 1.2 release and is described in section 7.9 [3].
>> Once hardware is prepared with hw_params(), all the USB endpoints
>> operations e.g.: start, stop, submitting URBs, are performed
>> internally, by the hardware and AudioDSP firmware.  Software driver
>> shall not intervene.
>>
>> Startup flow from:
>>
>> 	usb_pcm_open()
>> 	usb_pcm_hw_params()
>> 		snd_usb_endpoint_open()
>> 	usb_pcm_prepare()
>> 		usb_set_interface()
>> 		snd_usb_endpoint_start()
>> 	usb_pcm_trigger(cmd: START/STOP etc.)
>>
>> reduced to:
>>
>> 	usb_pcm_open()
>> 	usb_pcm_hw_params()
>> 		snd_usb_endpoint_open()
>> 	usb_pcm_prepare()
>> 		usb_set_interface()
> 
> Hmm, how can it be?  The start of EP at prepare stage is done only
> conditionally for non-lowlatency or implicit-feedback mode, for
> example.


On Intel architecture the AudioDSP utilizes internal channel to 
communicate with xHCI and perform all the transfer operations.  In 
essence, once SET_INTERFACE TRB is done, the usb-driver is not supposed 
to do anything.  The avs-driver (sound driver) would be the trigger here 
- sends the start/stop/etc. requests to the AudioDSP firmware, DSP does 
the rest.

By trigger I mean SET_PIPELINE_STATE IPC and avs_path_xxx() which are 
utilized for any transfer-type (sound/soc/intel/avs/pcm.c).

>> Handlers such as ack(), sync_stop(), pointer(), delay() are not used
>> here too.  The AudioDSP driver will handle pointer(), rest is
>> firmware/hardware responsibility.
>>
>>
>> There's a hefty number of limitations, most importantly:
>>
>> 1) typically 2 USB devices tops, rest go the classic (non-offload) path
>> 2) AUDIOSTREAMING interfaces only, MIDI not.  While not an interface
>>     type, Media (sound/usb/media) unsupported either
>> 3) simple PCM, UAC_FORMAT_TYPE_I_PCM only
>>
>> Current patchset shows this in form of 'udev->audsb_capable' field.
>> True if xHCI sideband reasource has been assigned to the device.  The
>> filtering is code is not part of the patchset.
>>
>> Important to highlight, 1) means both, offload-aware and offload-unaware
>> drivers could be utilized simultaneously on the system in runtime.
>> Opportunistically few devices would be controlled by the offload-aware
>> driver, whereas everything else by the offload-unaware one.  This
>> differs from existing HDAudio Controller driver situation where either
>> classic, snd_hda_intel driver takes complete control -or- the
>> offload-aware snd_soc_avs driver.
>> In short, once all Audio Sideband resources are depleted, classic
>> sound/usb/card.c driver manages whatever comes next:
>>
>> 	snd_usb_audio (offload un-aware, sound/usb/card.c)
>> 	snd_soc_usb_codec (offload aware, sound/soc/codecs/usb.c)
>>
>>
>> The design goals:
>> 	- make ASoC first class citizen of sound/usb
>> 	- re-use code found in sound/usb, mimic HDAudio integration in ASoC:
>> 	  small sound/soc/codecs/hda.c driver leveraging power of entire
>> 	  sound/pci/hda/
>> 	- no shared control over a USB device, either snd_usb_audio or
>> 	  its ASoC equivalent takes control of the device
>>
>> To do that, major tasks are identified:
>>
>> a) On ASoC side 'struct snd_card' is part of 'struct snd_soc_card' and
>> is managed by the framework.  Similar situation with 'struct snd_pcm'
>> and rtd->pcm.  To keep the teardown path sane, drop card->private_free()
>> and pcm->private_free() usage.
> 
> Well, this is a generic problem of ASoC framework.
> I believe this should be better handled in ASoC core side at first.
> e.g. the card object could be created at the very first step of the
> snd_soc_card creation, too (but without the actual slot assignment or
> device creation).


Well, in my opinion the card's tailing private context (extra size), 
private_free() and all that comes with them in sound/core brings 
unnecessary complexity to the ALSA framework.  devm_xxx() is enough. 
Polluting snd_card and sound/core/init.c with driver-specifics causes 
the teardown procedures on the ALSA framework side harder to read/maintain.

>> b) To initialize ASoC components/DAI properly, PCM capabilities should be
>> known up-front.  To do that, existing USB card probe() has to be split.
>>  From one-stage to two-stage process:
>>
>> 	- look ahead and parse usb_interface descriptors for PCM endpoints
>> 	  but do not create any PCMs (sound devices) yet
>> 	- create all PCMs based on obtained ->pcm_list and follow with
>> 	  MIDI/mixers/media
>>
>> Such approach allows to feed DAIs proper data even when a valid
>> sound-card pointer is not yet present - the initialization occurs before
>> snd_soc_bind_card() is called.
>   
> This one is another thing that is needed to adjust for ASoC
> framework.  But when snd_card object is available, this can be
> resolved automatically, too?  e.g. snd_pcm object or such can be
> created at that point.  The actual device registration is done anyway
> later via snd_device_register() call.


While I'm up for upgrading either framework in any area necessary to 
have proper UAO support in ASoC, not sure whether it's a good idea to 
make it a requirement for the feature.  This will enlarge the series - 
well, guess it will be separate series (dependency) entirely.

>> Point a) is scaled for all three "domains" of sound/usb: chip, stream
>> and quirks.  That's why there total of 6 commits doing that job.  First
>> implement, then switch.  Everything that follows is, in my opinion,
>> self-explanatory.  No need to repeat commit messages.
>>
>>
>> [1]: https://lore.kernel.org/all/20250319005141.312805-2-quic_wcheng@quicinc.com/
>> [2]: https://lore.kernel.org/all/20250319005141.312805-1-quic_wcheng@quicinc.com/
>> [3]: https://www.intel.com/content/dam/www/public/us/en/documents/technical-specifications/extensible-host-controler-interface-usb-xhci.pdf
Greg Kroah-Hartman April 11, 2025, 2:04 p.m. UTC | #6
On Wed, Apr 09, 2025 at 01:07:15PM +0200, Cezary Rojewski wrote:
> Note: this series is based on Mark's broonie/for-next. The xHCI
> dependency is missing so it won't compile.  Goal of this RFC is
> discussing the direction of sound/usb changes.

Any specific reason you aren't cc: linux-usb@vger.kernel.org on USB
stuff like this?

thanks,

greg k-h
Cezary Rojewski April 11, 2025, 4:51 p.m. UTC | #7
On 2025-04-11 4:04 PM, Greg KH wrote:
> On Wed, Apr 09, 2025 at 01:07:15PM +0200, Cezary Rojewski wrote:
>> Note: this series is based on Mark's broonie/for-next. The xHCI
>> dependency is missing so it won't compile.  Goal of this RFC is
>> discussing the direction of sound/usb changes.
> 
> Any specific reason you aren't cc: linux-usb@vger.kernel.org on USB
> stuff like this?

Actually I was expecting the exact opposite feedback had I included 
linux-usb, given lack of drivers/usb changes.  Lesson learned.

However, given the current feedback, for the next revision I plan to 
rebase onto your tree and include the xHCI changes.  While I'm familiar 
with sound/ practices, I clearly was not with drivers/usb/ ones.  The 
initial offload-aware vs offload-unaware USB audio-device filtering 
stuff was done by xHCI.  Thanks to Mathias now I'm aware there is no 
place for such specific class code in drivers/usb/host.

I'll try to get the usb changes adjusted as quickly as possible so that 
the list can see the entire thing.

Kind regards,
Czarek
Takashi Iwai April 15, 2025, 4:15 p.m. UTC | #8
On Fri, 11 Apr 2025 11:39:55 +0200,
Cezary Rojewski wrote:
> 
> On 2025-04-10 12:24 PM, Takashi Iwai wrote:
> > On Wed, 09 Apr 2025 13:07:15 +0200,
> > Cezary Rojewski wrote:
> >> 
> >> Note: this series is based on Mark's broonie/for-next. The xHCI
> >> dependency is missing so it won't compile.  Goal of this RFC is
> >> discussing the direction of sound/usb changes.
> >> 
> >> Note #2: Why exclude xHCI?  Current form of xhci-sideband.c [1] does not
> >> fare well with Intel's hardware design for USB Audio Offload feature.
> >> The production shape for usb/xHCI subjects is being discussed with
> >> Mathias.  Once we're ready, I'll share the rest.
> >> 
> >> Note #3: this series does _NOT_ aim to block QCOM's equivalent series
> >> [2].  The team does acknowledge that we came the "table" late.  At the
> >> same time, we're prepared to help QCOM switch to the presented sound/usb
> >> approach if that would benefit the framework and its users as a whole.
> >> Make it part of this very series if need be.
> >> 
> >> 
> >> 
> >> Apart from changes to sound/usb, the patchset contains exemplary
> >> ASoC-based, offload-aware USB driver and a small sound card on the
> >> avs-driver side to show how card/dai_link initialization looks like.
> >> 
> >> In short, USB Audio Offload functionality lowers CPU usage when
> >> streaming PCM over a USB Audio-Class device.  It has been first
> >> introduced in xHCI 1.2 release and is described in section 7.9 [3].
> >> Once hardware is prepared with hw_params(), all the USB endpoints
> >> operations e.g.: start, stop, submitting URBs, are performed
> >> internally, by the hardware and AudioDSP firmware.  Software driver
> >> shall not intervene.
> >> 
> >> Startup flow from:
> >> 
> >> 	usb_pcm_open()
> >> 	usb_pcm_hw_params()
> >> 		snd_usb_endpoint_open()
> >> 	usb_pcm_prepare()
> >> 		usb_set_interface()
> >> 		snd_usb_endpoint_start()
> >> 	usb_pcm_trigger(cmd: START/STOP etc.)
> >> 
> >> reduced to:
> >> 
> >> 	usb_pcm_open()
> >> 	usb_pcm_hw_params()
> >> 		snd_usb_endpoint_open()
> >> 	usb_pcm_prepare()
> >> 		usb_set_interface()
> > 
> > Hmm, how can it be?  The start of EP at prepare stage is done only
> > conditionally for non-lowlatency or implicit-feedback mode, for
> > example.
> 
> 
> On Intel architecture the AudioDSP utilizes internal channel to
> communicate with xHCI and perform all the transfer operations.  In
> essence, once SET_INTERFACE TRB is done, the usb-driver is not
> supposed to do anything.  The avs-driver (sound driver) would be the
> trigger here - sends the start/stop/etc. requests to the AudioDSP
> firmware, DSP does the rest.
> 
> By trigger I mean SET_PIPELINE_STATE IPC and avs_path_xxx() which are
> utilized for any transfer-type (sound/soc/intel/avs/pcm.c).

But does the DSP handles the different ways for the implicit feedback
mode, the low-latency playback and else?  A specific mode like the
implicit feedback mode is mandatory for many devices.  Similarly, the
low-latency mode is essential for some application setups
(e.g. pipewire or JACK prefers), while it's not always applicable
depending on the PCM parameters (such as the free-wheeling mode).

AFAIK, the qcom offloading doesn't support those fully, hence the
configuration must be dynamic for them.


> >> Handlers such as ack(), sync_stop(), pointer(), delay() are not used
> >> here too.  The AudioDSP driver will handle pointer(), rest is
> >> firmware/hardware responsibility.
> >> 
> >> 
> >> There's a hefty number of limitations, most importantly:
> >> 
> >> 1) typically 2 USB devices tops, rest go the classic (non-offload) path
> >> 2) AUDIOSTREAMING interfaces only, MIDI not.  While not an interface
> >>     type, Media (sound/usb/media) unsupported either
> >> 3) simple PCM, UAC_FORMAT_TYPE_I_PCM only
> >> 
> >> Current patchset shows this in form of 'udev->audsb_capable' field.
> >> True if xHCI sideband reasource has been assigned to the device.  The
> >> filtering is code is not part of the patchset.
> >> 
> >> Important to highlight, 1) means both, offload-aware and offload-unaware
> >> drivers could be utilized simultaneously on the system in runtime.
> >> Opportunistically few devices would be controlled by the offload-aware
> >> driver, whereas everything else by the offload-unaware one.  This
> >> differs from existing HDAudio Controller driver situation where either
> >> classic, snd_hda_intel driver takes complete control -or- the
> >> offload-aware snd_soc_avs driver.
> >> In short, once all Audio Sideband resources are depleted, classic
> >> sound/usb/card.c driver manages whatever comes next:
> >> 
> >> 	snd_usb_audio (offload un-aware, sound/usb/card.c)
> >> 	snd_soc_usb_codec (offload aware, sound/soc/codecs/usb.c)
> >> 
> >> 
> >> The design goals:
> >> 	- make ASoC first class citizen of sound/usb
> >> 	- re-use code found in sound/usb, mimic HDAudio integration in ASoC:
> >> 	  small sound/soc/codecs/hda.c driver leveraging power of entire
> >> 	  sound/pci/hda/
> >> 	- no shared control over a USB device, either snd_usb_audio or
> >> 	  its ASoC equivalent takes control of the device
> >> 
> >> To do that, major tasks are identified:
> >> 
> >> a) On ASoC side 'struct snd_card' is part of 'struct snd_soc_card' and
> >> is managed by the framework.  Similar situation with 'struct snd_pcm'
> >> and rtd->pcm.  To keep the teardown path sane, drop card->private_free()
> >> and pcm->private_free() usage.
> > 
> > Well, this is a generic problem of ASoC framework.
> > I believe this should be better handled in ASoC core side at first.
> > e.g. the card object could be created at the very first step of the
> > snd_soc_card creation, too (but without the actual slot assignment or
> > device creation).
> 
> 
> Well, in my opinion the card's tailing private context (extra size),
> private_free() and all that comes with them in sound/core brings
> unnecessary complexity to the ALSA framework.  devm_xxx() is
> enough. Polluting snd_card and sound/core/init.c with driver-specifics
> causes the teardown procedures on the ALSA framework side harder to
> read/maintain.

Well, I'd say it depends, and pretty much a matter of taste.  The
resources managed by the card free should be the last thing that is
tied with the card object itself.  So, yes, it can be devm, but this
wouldn't make things easier; the devm is merely a serialized release,
after all.

> >> b) To initialize ASoC components/DAI properly, PCM capabilities should be
> >> known up-front.  To do that, existing USB card probe() has to be split.
> >>  From one-stage to two-stage process:
> >> 
> >> 	- look ahead and parse usb_interface descriptors for PCM endpoints
> >> 	  but do not create any PCMs (sound devices) yet
> >> 	- create all PCMs based on obtained ->pcm_list and follow with
> >> 	  MIDI/mixers/media
> >> 
> >> Such approach allows to feed DAIs proper data even when a valid
> >> sound-card pointer is not yet present - the initialization occurs before
> >> snd_soc_bind_card() is called.
> >   This one is another thing that is needed to adjust for ASoC
> > framework.  But when snd_card object is available, this can be
> > resolved automatically, too?  e.g. snd_pcm object or such can be
> > created at that point.  The actual device registration is done anyway
> > later via snd_device_register() call.
> 
> 
> While I'm up for upgrading either framework in any area necessary to
> have proper UAO support in ASoC, not sure whether it's a good idea to
> make it a requirement for the feature.  This will enlarge the series -
> well, guess it will be separate series (dependency) entirely.

The series contains lots of hackish API exposure, and I really would
like to avoid that if possible.  In the case of HD-audio, the whole
stuff was moved to its own bus at first for adapting to ASoC hd-audio
ext implementation, but for USB-audio, I don't see the need for that
yet.  And, if the concern is about the snd_card object lifecycle and
ASoC binding, it can be improved in the lower level at first.


thanks,

Takashi
Cezary Rojewski April 17, 2025, 10:15 a.m. UTC | #9
On 2025-04-15 6:15 PM, Takashi Iwai wrote:
> On Fri, 11 Apr 2025 11:39:55 +0200,
> Cezary Rojewski wrote:
>>
>> On 2025-04-10 12:24 PM, Takashi Iwai wrote:
>>> On Wed, 09 Apr 2025 13:07:15 +0200,
>>> Cezary Rojewski wrote:

...

>>>> Apart from changes to sound/usb, the patchset contains exemplary
>>>> ASoC-based, offload-aware USB driver and a small sound card on the
>>>> avs-driver side to show how card/dai_link initialization looks like.
>>>>
>>>> In short, USB Audio Offload functionality lowers CPU usage when
>>>> streaming PCM over a USB Audio-Class device.  It has been first
>>>> introduced in xHCI 1.2 release and is described in section 7.9 [3].
>>>> Once hardware is prepared with hw_params(), all the USB endpoints
>>>> operations e.g.: start, stop, submitting URBs, are performed
>>>> internally, by the hardware and AudioDSP firmware.  Software driver
>>>> shall not intervene.
>>>>
>>>> Startup flow from:
>>>>
>>>> 	usb_pcm_open()
>>>> 	usb_pcm_hw_params()
>>>> 		snd_usb_endpoint_open()
>>>> 	usb_pcm_prepare()
>>>> 		usb_set_interface()
>>>> 		snd_usb_endpoint_start()
>>>> 	usb_pcm_trigger(cmd: START/STOP etc.)
>>>>
>>>> reduced to:
>>>>
>>>> 	usb_pcm_open()
>>>> 	usb_pcm_hw_params()
>>>> 		snd_usb_endpoint_open()
>>>> 	usb_pcm_prepare()
>>>> 		usb_set_interface()
>>>
>>> Hmm, how can it be?  The start of EP at prepare stage is done only
>>> conditionally for non-lowlatency or implicit-feedback mode, for
>>> example.
>>
>>
>> On Intel architecture the AudioDSP utilizes internal channel to
>> communicate with xHCI and perform all the transfer operations.  In
>> essence, once SET_INTERFACE TRB is done, the usb-driver is not
>> supposed to do anything.  The avs-driver (sound driver) would be the
>> trigger here - sends the start/stop/etc. requests to the AudioDSP
>> firmware, DSP does the rest.
>>
>> By trigger I mean SET_PIPELINE_STATE IPC and avs_path_xxx() which are
>> utilized for any transfer-type (sound/soc/intel/avs/pcm.c).
> 
> But does the DSP handles the different ways for the implicit feedback
> mode, the low-latency playback and else?  A specific mode like the
> implicit feedback mode is mandatory for many devices.  Similarly, the
> low-latency mode is essential for some application setups
> (e.g. pipewire or JACK prefers), while it's not always applicable
> depending on the PCM parameters (such as the free-wheeling mode).
> 
> AFAIK, the qcom offloading doesn't support those fully, hence the
> configuration must be dynamic for them.
The hw design is unfortunately not simple:

xHCI <> SIO <> ALH
^ USB side	^Audio DSP side

It's an internal channel between so-called Audio Sideband located on 
xHCI controller side, goes through Scalable-IO and links with Audio Link 
Hub that's acts as 'front-end' for DSP.  Every block has their own 
requirements and restrictions.

And thus the number of available 'streams' in such configuration is very 
limited - typically to just 6.  These are not even bi-direction streams 
but an internal map: IDs {1, 2} for outputs, IDs {3, 4, 5, 6} for 
inputs.  Due to such small number of available streams, there is 
pressure to filter USB devices.  In regard to endpoint types, only DATA 
and FEEDBACK are supported, IMPLICIT_FEEDBACK is not.  Such device 
should fail offload-candidate filtering and be serviced by snd_usb_audio 
driver instead.

The official xHCI spec and its section regarding Audio Sideband is 
tailored for "claim entire device for offload or ignore" approach:

- scan the USB device descriptors
- verify if it's an offload-candidate
- count number of DATA + FEEDBACK endpoints
- ask xHCI to _reserve_ Audio Sideband resources up-front. This means 
the resources are reserved long before sound-devices are visible from 
userspace, yet alone opened for streaming

- pcm->open/close() just ask xHCI to perform 
SET_ASSIGNMENT(reserved_resources_here) TRBs to prepare/close the 
internal channel for runtime operations, that's it

There is no intention to allow for switching the ownership between 
offload-aware and offload-unaware.  Either entire USB device and all its 
endpoints have Audio Sideband resources pre-allocated or no offload is 
present for the device at all.

...

>>>> The design goals:
>>>> 	- make ASoC first class citizen of sound/usb
>>>> 	- re-use code found in sound/usb, mimic HDAudio integration in ASoC:
>>>> 	  small sound/soc/codecs/hda.c driver leveraging power of entire
>>>> 	  sound/pci/hda/
>>>> 	- no shared control over a USB device, either snd_usb_audio or
>>>> 	  its ASoC equivalent takes control of the device
>>>>
>>>> To do that, major tasks are identified:
>>>>
>>>> a) On ASoC side 'struct snd_card' is part of 'struct snd_soc_card' and
>>>> is managed by the framework.  Similar situation with 'struct snd_pcm'
>>>> and rtd->pcm.  To keep the teardown path sane, drop card->private_free()
>>>> and pcm->private_free() usage.
>>>
>>> Well, this is a generic problem of ASoC framework.
>>> I believe this should be better handled in ASoC core side at first.
>>> e.g. the card object could be created at the very first step of the
>>> snd_soc_card creation, too (but without the actual slot assignment or
>>> device creation).
>>
>>
>> Well, in my opinion the card's tailing private context (extra size),
>> private_free() and all that comes with them in sound/core brings
>> unnecessary complexity to the ALSA framework.  devm_xxx() is
>> enough. Polluting snd_card and sound/core/init.c with driver-specifics
>> causes the teardown procedures on the ALSA framework side harder to
>> read/maintain.
> 
> Well, I'd say it depends, and pretty much a matter of taste.  The
> resources managed by the card free should be the last thing that is
> tied with the card object itself.  So, yes, it can be devm, but this
> wouldn't make things easier; the devm is merely a serialized release,
> after all.
> 
>>>> b) To initialize ASoC components/DAI properly, PCM capabilities should be
>>>> known up-front.  To do that, existing USB card probe() has to be split.
>>>>   From one-stage to two-stage process:
>>>>
>>>> 	- look ahead and parse usb_interface descriptors for PCM endpoints
>>>> 	  but do not create any PCMs (sound devices) yet
>>>> 	- create all PCMs based on obtained ->pcm_list and follow with
>>>> 	  MIDI/mixers/media
>>>>
>>>> Such approach allows to feed DAIs proper data even when a valid
>>>> sound-card pointer is not yet present - the initialization occurs before
>>>> snd_soc_bind_card() is called.
>>>    This one is another thing that is needed to adjust for ASoC
>>> framework.  But when snd_card object is available, this can be
>>> resolved automatically, too?  e.g. snd_pcm object or such can be
>>> created at that point.  The actual device registration is done anyway
>>> later via snd_device_register() call.
>>
>>
>> While I'm up for upgrading either framework in any area necessary to
>> have proper UAO support in ASoC, not sure whether it's a good idea to
>> make it a requirement for the feature.  This will enlarge the series -
>> well, guess it will be separate series (dependency) entirely.
> 
> The series contains lots of hackish API exposure, and I really would
> like to avoid that if possible.  In the case of HD-audio, the whole
> stuff was moved to its own bus at first for adapting to ASoC hd-audio
> ext implementation, but for USB-audio, I don't see the need for that
> yet.  And, if the concern is about the snd_card object lifecycle and
> ASoC binding, it can be improved in the lower level at first.

I tried hard to avoid hackish stuff, perhaps not hard enough :)  Would 
it be possible to list the areas which you believe need to be look at? 
This is frankly the feedback which I'm most interested in.  Mathias gave 
me a number of these, e.g.: do not access udev->slot_id in sound/, avoid 
manipulating hcd/controller in sound/ and such.

In regard to the HDAudio point, I see clear benefits by having HDAudio 
and USB aligned in the approach on ASoC side. It's a path that's known, 
works and is well tested.

In regard to the snd_soc_card vs snd_card, in my opinion the refactor of 
card initialization is a large task.  In the past I did similar change 
for the snd_soc_component [1] but the card is an entirely different 
beast.  That's why I'm suggesting incremental approach - update ASoC 
initialization as a next step.


[1]: 
https://lore.kernel.org/all/20200731144146.6678-1-cezary.rojewski@intel.com/

Kind regards,
Czarek