| Message ID | 20240425055759.1181165-1-quic_hprem@quicinc.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | wifi: ath12k: Introduce device group abstraction | expand |
On 4/24/2024 10:57 PM, Harshitha Prem wrote: > To support multi-link operation, multiple devices with different bands say > 2 GHz or 5 GHz or 6 GHz can be combined together as a group and provide > an abstraction to mac80211. > > Device group abstraction - when there are multiple devices that are > connected by any means of communication interface between them, then these > devices can be combined together as a single group using a group id to form > a group abstraction. In ath12k driver, this abstraction would be named as > ath12k_hw_group (ag). > > Please find below illustration of device group abstraction with two > devices. > > Grouping of multiple devices (in future) > +------------------------------------------------------------------------+ > | +-------------------------------------+ +-------------------+ | > | | +-----------+ | | +-----------+ | | +-----------+ | | > | | | ar (2GHz) | | | | ar (5GHz) | | | | ar (6GHz) | | | > | | +-----------+ | | +-----------+ | | +-----------+ | | > | | ath12k_base (ab) | | ath12k_base (ab) | | > | | (Dual band device) | | | | > | +-------------------------------------+ +-------------------+ | > | ath12k_hw_group (ag) based on group id | > +------------------------------------------------------------------------+ > > Say for example, device 1 has two radios (2 GHz and 5 GHz band) and > device 2 has one radio (6 GHz). > > In existing code - > device 1 will have two hardware abstractions hw1 (2 GHz) and hw2 > (5 GHz) will be registered separately to mac80211 as phy0 and phy1 > respectively. Similarly, device 2 will register its hw (6GHz) as > phy2 to mac80211. > > In future, with multi-link abstraction > > combination 1 - Different group id for device1 and device 2 > Device 1 will create a single hardware abstraction hw1 > (2 GHz and 5 GHz) and will be registered to mac80211 as > phy0. similarly, device 2 will register its hardware > (6 GHz) to mac80211 as phy1. > > combination 2 - Same group id for device1 and device 2 > Both device details are combined together as a group, say > group1, with single hardware abstraction of radios 2 GHz, > 5 GHz and 6 GHz band details and will be registered to > mac80211 as phy0. > > Add base infrastructure changes to add device grouping abstraction with > a single device. > > This patch series brings the base code changes with following order: > 1. Refactor existing code which would facilitate in introducing > device group abstraction. > 2. Create a device group abstraction during device probe. > 3. Start the device group only after QMI firmware ready event is > received for all the devices that are combined in the group. > 4. Move the hardware abstractions (ath12k_hw - ah) from device > (ath12k_base - ab) to device group abstraction (ag) as it would > ease in having different combinations of group abstraction that > can be registered to mac80211. > > > Depends-on: > [PATCH v2 0/3] wifi: ath12k: Refactor the hardware recovery > procedures. > Link - https://lore.kernel.org/ath12k/87a5ljt0p9.fsf@kernel.org/T/ current dependency link (v3): https://lore.kernel.org/ath12k/20240425090307.3233434-1-quic_periyasa@quicinc.com I verified this series cleanly applies on top of that > > v2: > - Rebased to ToT > > Karthikeyan Periyasamy (8): > wifi: ath12k: Refactor core start api > wifi: ath12k: Add helpers to get or set ath12k_hw > wifi: ath12k: Add ath12k_get_num_hw api > wifi: ath12k: Introduce QMI firmware ready flag > wifi: ath12k: move ATH12K_FLAG_REGISTERED flag set to mac_register api > wifi: ath12k: Introduce device group abstraction > wifi: ath12k: refactor core start based on hardware group > wifi: ath12k: move ath12k_hw from per soc to group > > drivers/net/wireless/ath/ath12k/core.c | 428 +++++++++++++++++++++---- > drivers/net/wireless/ath/ath12k/core.h | 88 ++++- > drivers/net/wireless/ath/ath12k/mac.c | 108 +++++-- > drivers/net/wireless/ath/ath12k/mac.h | 9 +- > drivers/net/wireless/ath/ath12k/pci.c | 1 + > drivers/net/wireless/ath/ath12k/qmi.c | 10 +- > 6 files changed, 527 insertions(+), 117 deletions(-) > > > base-commit: 326f8f68f28b0b831233acfabffb486a5b0f4717 > prerequisite-patch-id: fa330433b950da57175cc84c5e97c2def1d24959 > prerequisite-patch-id: 75254d9efeb5eb6e3b2027155be94334c85a76b9 > prerequisite-patch-id: edd3d755bafc868bae646b54d279c2a8ba66acd1
Harshitha Prem <quic_hprem@quicinc.com> writes: > To support multi-link operation, multiple devices with different bands say > 2 GHz or 5 GHz or 6 GHz can be combined together as a group and provide > an abstraction to mac80211. > > Device group abstraction - when there are multiple devices that are > connected by any means of communication interface between them, then these > devices can be combined together as a single group using a group id to form > a group abstraction. In ath12k driver, this abstraction would be named as > ath12k_hw_group (ag). > > Please find below illustration of device group abstraction with two > devices. > > Grouping of multiple devices (in future) > +------------------------------------------------------------------------+ > | +-------------------------------------+ +-------------------+ | > | | +-----------+ | | +-----------+ | | +-----------+ | | > | | | ar (2GHz) | | | | ar (5GHz) | | | | ar (6GHz) | | | > | | +-----------+ | | +-----------+ | | +-----------+ | | > | | ath12k_base (ab) | | ath12k_base (ab) | | > | | (Dual band device) | | | | > | +-------------------------------------+ +-------------------+ | > | ath12k_hw_group (ag) based on group id | > +------------------------------------------------------------------------+ > > Say for example, device 1 has two radios (2 GHz and 5 GHz band) and > device 2 has one radio (6 GHz). > > In existing code - > device 1 will have two hardware abstractions hw1 (2 GHz) and hw2 > (5 GHz) will be registered separately to mac80211 as phy0 and phy1 > respectively. Similarly, device 2 will register its hw (6GHz) as > phy2 to mac80211. > > In future, with multi-link abstraction > > combination 1 - Different group id for device1 and device 2 > Device 1 will create a single hardware abstraction hw1 > (2 GHz and 5 GHz) and will be registered to mac80211 as > phy0. similarly, device 2 will register its hardware > (6 GHz) to mac80211 as phy1. > > combination 2 - Same group id for device1 and device 2 > Both device details are combined together as a group, say > group1, with single hardware abstraction of radios 2 GHz, > 5 GHz and 6 GHz band details and will be registered to > mac80211 as phy0. > > Add base infrastructure changes to add device grouping abstraction with > a single device. > > This patch series brings the base code changes with following order: > 1. Refactor existing code which would facilitate in introducing > device group abstraction. > 2. Create a device group abstraction during device probe. > 3. Start the device group only after QMI firmware ready event is > received for all the devices that are combined in the group. > 4. Move the hardware abstractions (ath12k_hw - ah) from device > (ath12k_base - ab) to device group abstraction (ag) as it would > ease in having different combinations of group abstraction that > can be registered to mac80211. > > > Depends-on: > [PATCH v2 0/3] wifi: ath12k: Refactor the hardware recovery > procedures. > Link - https://lore.kernel.org/ath12k/87a5ljt0p9.fsf@kernel.org/T/ > > v2: > - Rebased to ToT > > Karthikeyan Periyasamy (8): > wifi: ath12k: Refactor core start api > wifi: ath12k: Add helpers to get or set ath12k_hw > wifi: ath12k: Add ath12k_get_num_hw api > wifi: ath12k: Introduce QMI firmware ready flag > wifi: ath12k: move ATH12K_FLAG_REGISTERED flag set to mac_register api > wifi: ath12k: Introduce device group abstraction > wifi: ath12k: refactor core start based on hardware group > wifi: ath12k: move ath12k_hw from per soc to group I see a deadlock warning in master-pending branch (tag ath-pending-202404291731) and based on manual bisect between patchsets it seems to come from this patchset. Do note that I didn't look at the patchset otherwise. Here's the warning I see during rmmod with WCN7850 and I see it every time: [ 147.211487] ====================================================== [ 147.211547] WARNING: possible circular locking dependency detected [ 147.211599] 6.9.0-rc5-wt-ath+ #1403 Not tainted [ 147.211646] ------------------------------------------------------ [ 147.211695] rmmod/1975 is trying to acquire lock: [ 147.211741] ffff888105c7f158 ((wq_completion)ath12k_qmi_driver_event){+.+.}-{0:0}, at: touch_wq_lockdep_map+0x46/0xf0 [ 147.211815] #012[ 147.211815] but task is already holding lock: [ 147.211864] ffff88810db290a8 (&ag->mutex_lock){+.+.}-{3:3}, at: ath12k_core_hw_group_destroy.part.0+0x26/0x290 [ath12k] [ 147.212003] #012[ 147.212003] which lock already depends on the new lock.#012[ 147.212003] [ 147.212069] #012[ 147.212069] the existing dependency chain (in reverse order) is: [ 147.212135] #012[ 147.212135] -> #2 (&ag->mutex_lock){+.+.}-{3:3}: [ 147.212189] __lock_acquire+0xd43/0x1dd0 [ 147.212238] lock_acquire+0x1b0/0x560 [ 147.212280] __mutex_lock+0x154/0x1430 [ 147.212327] mutex_lock_nested+0x16/0x20 [ 147.212369] ath12k_core_qmi_firmware_ready+0x9d/0x400 [ath12k] [ 147.212436] ath12k_qmi_driver_event_work+0x4e9/0x6e0 [ath12k] [ 147.212507] process_one_work+0x8a4/0x1980 [ 147.212551] worker_thread+0x715/0x1270 [ 147.212594] kthread+0x2fa/0x3f0 [ 147.212636] ret_from_fork+0x31/0x70 [ 147.212679] ret_from_fork_asm+0x11/0x20 [ 147.212754] #012[ 147.212754] -> #1 ((work_completion)(&ab->qmi.event_work)){+.+.}-{0:0}: [ 147.212828] __lock_acquire+0xd43/0x1dd0 [ 147.212885] lock_acquire+0x1b0/0x560 [ 147.213000] process_one_work+0x82d/0x1980 [ 147.213055] worker_thread+0x715/0x1270 [ 147.213779] kthread+0x2fa/0x3f0 [ 147.214529] ret_from_fork+0x31/0x70 [ 147.215291] ret_from_fork_asm+0x11/0x20 [ 147.216044] #012[ 147.216044] -> #0 ((wq_completion)ath12k_qmi_driver_event){+.+.}-{0:0}: [ 147.217433] check_prev_add+0x1bd/0x2330 [ 147.218174] validate_chain+0xf4e/0x1cf0 [ 147.218853] __lock_acquire+0xd43/0x1dd0 [ 147.219589] lock_acquire+0x1b0/0x560 [ 147.220324] touch_wq_lockdep_map+0x66/0xf0 [ 147.221055] __flush_workqueue+0xeb/0x1120 [ 147.221731] drain_workqueue+0xf5/0x320 [ 147.222460] destroy_workqueue+0xb2/0x920 [ 147.223187] ath12k_qmi_deinit_service+0x5a/0x1f0 [ath12k] [ 147.223876] ath12k_core_hw_group_destroy.part.0+0x1f8/0x290 [ath12k] [ 147.224606] ath12k_core_deinit+0x37/0x50 [ath12k] [ 147.225314] ath12k_pci_remove+0xad/0x1b0 [ath12k] [ 147.226640] pci_device_remove+0x9b/0x1b0 [ 147.227326] device_remove+0xbf/0x150 [ 147.227986] device_release_driver_internal+0x3c3/0x580 [ 147.228619] driver_detach+0xc4/0x190 [ 147.229290] bus_remove_driver+0x130/0x2a0 [ 147.229900] driver_unregister+0x68/0x90 [ 147.230576] pci_unregister_driver+0x24/0x240 [ 147.231240] ath12k_pci_exit+0x10/0x20 [ath12k] [ 147.231866] __do_sys_delete_module+0x32c/0x580 [ 147.232523] __x64_sys_delete_module+0x4f/0x70 [ 147.233168] x64_sys_call+0x51b/0x9e0 [ 147.233756] do_syscall_64+0x65/0x130 [ 147.234396] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 147.235023] #012[ 147.235023] other info that might help us debug this:#012[ 147.235023] [ 147.236688] Chain exists of:#012[ 147.236688] (wq_completion)ath12k_qmi_driver_event --> (work_completion)(&ab->qmi.event_work) --> &ag->mutex_lock#012[ 147.236688] [ 147.238414] Possible unsafe locking scenario:#012[ 147.238414] [ 147.239520] CPU0 CPU1 [ 147.240114] ---- ---- [ 147.240633] lock(&ag->mutex_lock); [ 147.241212] lock((work_completion)(&ab->qmi.event_work)); [ 147.241746] lock(&ag->mutex_lock); [ 147.242341] lock((wq_completion)ath12k_qmi_driver_event); [ 147.242876] #012[ 147.242876] *** DEADLOCK ***#012[ 147.242876] [ 147.244460] 2 locks held by rmmod/1975: [ 147.245040] #0: ffff88810d7191b8 (&dev->mutex){....}-{3:3}, at: device_release_driver_internal+0x9d/0x580 [ 147.245597] #1: ffff88810db290a8 (&ag->mutex_lock){+.+.}-{3:3}, at: ath12k_core_hw_group_destroy.part.0+0x26/0x290 [ath12k] [ 147.246230] #012[ 147.246230] stack backtrace: [ 147.247338] CPU: 5 PID: 1975 Comm: rmmod Not tainted 6.9.0-rc5-wt-ath+ #1403 [ 147.247900] Hardware name: Intel(R) Client Systems NUC8i7HVK/NUC8i7HVB, BIOS HNKBLi70.86A.0067.2021.0528.1339 05/28/2021 [ 147.248545] Call Trace: [ 147.249181] <TASK> [ 147.249757] dump_stack_lvl+0x7d/0xe0 [ 147.250401] dump_stack+0x10/0x20 [ 147.251031] print_circular_bug+0x2e8/0x480 [ 147.251606] check_noncircular+0x2f2/0x3d0 [ 147.252235] ? print_circular_bug+0x480/0x480 [ 147.252808] ? validate_chain+0x15e/0x1cf0 [ 147.253435] ? __kasan_check_read+0x11/0x20 [ 147.254065] ? mark_lock+0xe6/0x1470 [ 147.254636] ? alloc_chain_hlocks+0x4cc/0x790 [ 147.255265] check_prev_add+0x1bd/0x2330 [ 147.255839] ? __kasan_check_read+0x11/0x20 [ 147.256474] validate_chain+0xf4e/0x1cf0 [ 147.257104] ? check_prev_add+0x2330/0x2330 [ 147.257673] __lock_acquire+0xd43/0x1dd0 [ 147.258303] lock_acquire+0x1b0/0x560 [ 147.258875] ? touch_wq_lockdep_map+0x46/0xf0 [ 147.259502] ? lock_sync+0x1a0/0x1a0 [ 147.260131] ? __lock_acquired+0x208/0x810 [ 147.260699] ? lockdep_init_map_type+0x1a3/0x850 [ 147.261326] ? lockdep_init_map_type+0x1a3/0x850 [ 147.261892] ? touch_wq_lockdep_map+0x46/0xf0 [ 147.262513] touch_wq_lockdep_map+0x66/0xf0 [ 147.263126] ? touch_wq_lockdep_map+0x46/0xf0 [ 147.263684] __flush_workqueue+0xeb/0x1120 [ 147.264301] ? drain_workqueue+0xae/0x320 [ 147.264862] ? drain_workqueue+0xae/0x320 [ 147.265471] ? __this_cpu_preempt_check+0x13/0x20 [ 147.266086] ? wq_update_node_max_active+0x540/0x540 [ 147.266640] ? destroy_workqueue+0xaa/0x920 [ 147.267249] ? __this_cpu_preempt_check+0x13/0x20 [ 147.267806] ? bit_wait_timeout+0x160/0x160 [ 147.268420] ? __kasan_check_write+0x14/0x20 [ 147.269031] drain_workqueue+0xf5/0x320 [ 147.269586] destroy_workqueue+0xb2/0x920 [ 147.270202] ath12k_qmi_deinit_service+0x5a/0x1f0 [ath12k] [ 147.270785] ? debugfs_remove+0x52/0x60 [ 147.271400] ath12k_core_hw_group_destroy.part.0+0x1f8/0x290 [ath12k] [ 147.272046] ath12k_core_deinit+0x37/0x50 [ath12k] [ 147.272628] ath12k_pci_remove+0xad/0x1b0 [ath12k] [ 147.273271] pci_device_remove+0x9b/0x1b0 [ 147.273829] device_remove+0xbf/0x150 [ 147.274446] device_release_driver_internal+0x3c3/0x580 [ 147.275052] ? __kasan_check_read+0x11/0x20 [ 147.275592] driver_detach+0xc4/0x190 [ 147.276189] bus_remove_driver+0x130/0x2a0 [ 147.276728] driver_unregister+0x68/0x90 [ 147.277325] pci_unregister_driver+0x24/0x240 [ 147.277876] ? find_module_all+0x13e/0x1e0 [ 147.278469] ath12k_pci_exit+0x10/0x20 [ath12k] [ 147.279087] __do_sys_delete_module+0x32c/0x580 [ 147.279626] ? __kasan_slab_free+0x102/0x170 [ 147.280209] ? module_flags+0x2f0/0x2f0 [ 147.280741] ? kmem_cache_free+0xed/0x3e0 [ 147.281334] ? __fput+0x40c/0xa60 [ 147.281870] ? __fput+0x40c/0xa60 [ 147.282462] ? debug_smp_processor_id+0x17/0x20 [ 147.283045] __x64_sys_delete_module+0x4f/0x70 [ 147.283558] x64_sys_call+0x51b/0x9e0 [ 147.284125] do_syscall_64+0x65/0x130 [ 147.284613] entry_SYSCALL_64_after_hwframe+0x4b/0x53 [ 147.285159] RIP: 0033:0x7f2df2adcc8b [ 147.286954] Code: 73 01 c3 48 8b 0d 05 c2 0c 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa b8 b0 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d d5 c1 0c 00 f7 d8 64 89 01 48 [ 147.288033] RSP: 002b:00007ffdc52ca298 EFLAGS: 00000206 ORIG_RAX: 00000000000000b0 [ 147.288563] RAX: ffffffffffffffda RBX: 000055d10d0237e0 RCX: 00007f2df2adcc8b [ 147.289155] RDX: 000000000000000a RSI: 0000000000000800 RDI: 000055d10d023848 [ 147.289691] RBP: 00007ffdc52ca2f8 R08: 0000000000000000 R09: 0000000000000000 [ 147.290292] R10: 00007f2df2b58ac0 R11: 0000000000000206 R12: 00007ffdc52ca4d0 [ 147.290830] R13: 00007ffdc52caebf R14: 000055d10d0222a0 R15: 000055d10d0237e0 [ 147.291444] </TASK>
On 4/30/2024 5:24 PM, Kalle Valo wrote: > 47] WARNING: possible circular locking dependency detected > [ 147.211599] 6.9.0-rc5-wt-ath+ #1403 Not tainted Thank you, Kalle, for sharing the full logs. Unfortunately, I did not see this warning in my testbed with QCN9274. Seems like during rmmod case, ath12k_core_soc_destroy() is invoked with ag->mutex lock which in-turn invokes ath12k_qmi_deinit_service() where cancel_work_sync for qmi events are called. So, if ATH12K_QMI_EVENT_FW_READY was in queue during rmmod, it would hit a deadlock as in core start it requires ag->mutex lock. But, this scenario is quite unlikely to occur where rmmod and FW_READY event are going to be in parallel. Nevertheless, it has to be addressed. I will check further on this warning and see how to address this. Thanks, Harshitha
On 5/1/2024 8:34 AM, Harshitha Prem wrote: > > > On 4/30/2024 5:24 PM, Kalle Valo wrote: >> 47] WARNING: possible circular locking dependency detected >> [ 147.211599] 6.9.0-rc5-wt-ath+ #1403 Not tainted > > Thank you, Kalle, for sharing the full logs. Unfortunately, I did not > see this warning in my testbed with QCN9274. I didn't see it in my testing either, which caused me to look at my .config and notice that I didn't have any of the lock debugging enabled. Have fixed that now.
To support multi-link operation, multiple devices with different bands say 2 GHz or 5 GHz or 6 GHz can be combined together as a group and provide an abstraction to mac80211. Device group abstraction - when there are multiple devices that are connected by any means of communication interface between them, then these devices can be combined together as a single group using a group id to form a group abstraction. In ath12k driver, this abstraction would be named as ath12k_hw_group (ag). Please find below illustration of device group abstraction with two devices. Grouping of multiple devices (in future) +------------------------------------------------------------------------+ | +-------------------------------------+ +-------------------+ | | | +-----------+ | | +-----------+ | | +-----------+ | | | | | ar (2GHz) | | | | ar (5GHz) | | | | ar (6GHz) | | | | | +-----------+ | | +-----------+ | | +-----------+ | | | | ath12k_base (ab) | | ath12k_base (ab) | | | | (Dual band device) | | | | | +-------------------------------------+ +-------------------+ | | ath12k_hw_group (ag) based on group id | +------------------------------------------------------------------------+ Say for example, device 1 has two radios (2 GHz and 5 GHz band) and device 2 has one radio (6 GHz). In existing code - device 1 will have two hardware abstractions hw1 (2 GHz) and hw2 (5 GHz) will be registered separately to mac80211 as phy0 and phy1 respectively. Similarly, device 2 will register its hw (6GHz) as phy2 to mac80211. In future, with multi-link abstraction combination 1 - Different group id for device1 and device 2 Device 1 will create a single hardware abstraction hw1 (2 GHz and 5 GHz) and will be registered to mac80211 as phy0. similarly, device 2 will register its hardware (6 GHz) to mac80211 as phy1. combination 2 - Same group id for device1 and device 2 Both device details are combined together as a group, say group1, with single hardware abstraction of radios 2 GHz, 5 GHz and 6 GHz band details and will be registered to mac80211 as phy0. Add base infrastructure changes to add device grouping abstraction with a single device. This patch series brings the base code changes with following order: 1. Refactor existing code which would facilitate in introducing device group abstraction. 2. Create a device group abstraction during device probe. 3. Start the device group only after QMI firmware ready event is received for all the devices that are combined in the group. 4. Move the hardware abstractions (ath12k_hw - ah) from device (ath12k_base - ab) to device group abstraction (ag) as it would ease in having different combinations of group abstraction that can be registered to mac80211. Depends-on: [PATCH v2 0/3] wifi: ath12k: Refactor the hardware recovery procedures. Link - https://lore.kernel.org/ath12k/87a5ljt0p9.fsf@kernel.org/T/ v2: - Rebased to ToT Karthikeyan Periyasamy (8): wifi: ath12k: Refactor core start api wifi: ath12k: Add helpers to get or set ath12k_hw wifi: ath12k: Add ath12k_get_num_hw api wifi: ath12k: Introduce QMI firmware ready flag wifi: ath12k: move ATH12K_FLAG_REGISTERED flag set to mac_register api wifi: ath12k: Introduce device group abstraction wifi: ath12k: refactor core start based on hardware group wifi: ath12k: move ath12k_hw from per soc to group drivers/net/wireless/ath/ath12k/core.c | 428 +++++++++++++++++++++---- drivers/net/wireless/ath/ath12k/core.h | 88 ++++- drivers/net/wireless/ath/ath12k/mac.c | 108 +++++-- drivers/net/wireless/ath/ath12k/mac.h | 9 +- drivers/net/wireless/ath/ath12k/pci.c | 1 + drivers/net/wireless/ath/ath12k/qmi.c | 10 +- 6 files changed, 527 insertions(+), 117 deletions(-) base-commit: 326f8f68f28b0b831233acfabffb486a5b0f4717 prerequisite-patch-id: fa330433b950da57175cc84c5e97c2def1d24959 prerequisite-patch-id: 75254d9efeb5eb6e3b2027155be94334c85a76b9 prerequisite-patch-id: edd3d755bafc868bae646b54d279c2a8ba66acd1