| Message ID | 2f83cea3-1760-1557-c0ff-0d40ab20f9e8@schmut.com (mailing list archive) |
|---|---|
| State | Not Applicable |
| Headers | show |
On Tue, Oct 17, 2017 at 09:35:39PM +0200, Mario Theodoridis wrote: > On 16.10.2017 05:37, James Cameron wrote: > >On Sun, Oct 15, 2017 at 06:21:36PM +0200, Mario Theodoridis wrote: > >>Thanks for the pointers, James. > >> > >>On 12.10.2017 23:24, James Cameron wrote: > >>>There's a good chance this problem has been fixed already. You > >>>are using a v4.4 kernel with many patches applied by Ubuntu. Here, we > >>>are more concerned with the latest kernels, and v4.4 is quite old. > >>> > >>>Please test some of the later kernels, see > >>>https://wiki.ubuntu.com/Kernel/MainlineBuilds > >>> > >>>In particular, test v4.13 or v4.14-rc4. > >> > >>I'm having a hard time with that, because the virtualbox-dkms build fails > >>with the 4.13 kernel, and virtualbox unfortunately is essential. > > > >Is virtualbox essential for reproducing the problem, or essential for > >your general use? > > It is essential for general use, like Internet connectivity. Okay, good, that means we can ignore virtualbox, and leave that to you. Please test v4.13 or v4.14-rc5, ignoring virtualbox for the time being. > >If the former, then that's interesting. > > > >If the latter, then you might instead test the v4.13 or v14-rc4 > >kernels for only the problem, and then revert to an older kernel after > >testing. > > > >Either way, to use virtualbox-dkms with a later kernel you may be able > >to upgrade just the virtualbox packages from a later Ubuntu release. > > > >See https://packages.ubuntu.com/virtualbox-dkms and > >https://packages.ubuntu.com/virtualbox for the later versions available. > > > >Purpose of the test can be to help isolate the cause, not only to > >solve your problem. > > Thanks for the info. > > > > >[...] > >You might also try with later firmware package. > >See https://packages.ubuntu.com/linux-firmware > > > >You might also test with booting installation media in live-mode, > >ignoring the internal disk. > > Ok, that was completely off the radar. Updating linux-firmware may run different firmware on the wireless card, and the change in behaviour may fix the problem. A gamble. A test with later installation media is useful, because you can verify problems with different kernels and wireless firmware without change to configuration. You might try Ubuntu 17.10 Artful ISO. > I ended up going the other way. I still had a 4.4.0-79-generic kernel and > booted that. It does not have this problem. > After checking out > git://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/xenial > i tried to find the culprit but was not able to trace the back trace to a > potential null pointer or some such. I got stuck at > iwl_mvm_send_cmd_pdu_status not finding a reference to iwl_mvm_disable_txq > from there. > > I did got the following diff though > > git diff Ubuntu-4.4.0-79.100 Ubuntu-4.4.0-93.116 -- > drivers/net/wireless/iwlwifi/ drivers/net/wireless/mac80211_hwsim.c > > wifi.patch > > I don't know whether this came from upstream or was ubuntu sourced. Upstream. You found your problem was introduced in an Ubuntu kernel, in the update from -79 to -93. This contained Ubuntu backports of two stable kernel patches, which are also upstream patches; 8fbcfeb8a9cc ("mac80211_hwsim: Replace bogus hrtimer clockid") from v4.4.69 50ea05efaf3b ("mac80211: pass block ack session timeout to to driver") from v4.4.77 git log Ubuntu-4.4.0-79.100..Ubuntu-4.4.0-93.116 -- \ drivers/net/wireless/iwlwifi/ drivers/net/wireless/mac80211_hwsim.c git remote add stable \ git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git git fetch stable git log v4.4.68..v4.4.92 -- \ drivers/net/wireless/iwlwifi/ drivers/net/wireless/mac80211_hwsim.c > This fixed the issue for now, but now i'm stuck on that kernel :( Yes. Here in upstream, we would run the latest kernel v4.13 and work to fix that. Trouble you had with virtualbox packages would be eventually solvable, but aren't really a problem with the kernel itself. So your next step may be to report an Ubuntu bug, and say that -79 worked fine, and -93 did not. > While i'm perfectly comfortable with user land C, i have no kernel > experience (clue stick links definitely welcome). You might verify the above patches caused the problem by doing a bisection between -79 and -93. https://wiki.ubuntu.com/Kernel/KernelBisection Or by reverting only those patches. Then report to Ubuntu which patch caused the problem. > [...] Hope that helps.
Sorry for skipping the list one the last one. On 19.10.2017 22:59, James Cameron wrote: > On Thu, Oct 19, 2017 at 08:56:46AM +0200, Mario Theodoridis wrote: >> On 18/10/17 23:33, James Cameron wrote: >> >> For your interest, kernel v4.4.93 in stable series just released has >> changes in relevant files. >> >> [1]https://lwn.net/Articles/736770/ >> >> Thanks James, >> >> after looking into bisection last night, i found that just before i wanted to >> test out the 4.4.0-82 kernel, i found 3 stack traces in my syslog. :( >> >> I guess, i'm dealing with race conditions now. But it seems the 79 kernel still >> crashes wifi a lot less than later ones. >> >> How do i get line numbers into these traces? As the 4.4.0-79 kernel was sometimes crapping out, too, i decided to try to test the latest kernel instead of bisecting after all. This took a while because virtualbox was being a bitch. virtualbox-5.0 doesn't bode well with virtualbox-dkms-51, so i ended up rebuilding virtualbox-5.1 to prevent dependency hell. The vb-dkms package doesn't do 4.14, so i ended up going with the 4.13 kernel that comes with artful. This one pretty quickly loads my syslog with new error stacks. I haven't tested actual behavior yet, but the logs don't look so hot. I ran another wireless-info (attached) and appended some of the syslog stuff to it. > Several methods, though by far the most common seems to be personal > experience with offsets. > > When you don't have that personal experience, the methods are; > > 1. using GDB against the .o file, > > 2. using binutils objdump to disassemble .o file or vmlinuz, > > 3. using GCC to generate assembly listings, > > See https://wiki.ubuntu.com/Kernel/KernelDebuggingTricks right down > the end of page for the GDB method. I have gotten around to that part, yet, as i was busy with the above, but it seems later versions have issues, too.
Summary: WARN_ON(iwl_mvm_is_dqa_supported(mvm)) in iwl_mvm_rx_tx_cmd_single with v4.13, but code is since changed. On Tue, Oct 24, 2017 at 09:56:31PM +0200, Mario Theodoridis wrote: > Sorry for skipping the list one the last one. Sorry, that was my fault. It was a private message you replied to. > On 19.10.2017 22:59, James Cameron wrote: > >On Thu, Oct 19, 2017 at 08:56:46AM +0200, Mario Theodoridis wrote: > >>On 18/10/17 23:33, James Cameron wrote: > >> > >> For your interest, kernel v4.4.93 in stable series just released has > >> changes in relevant files. > >> > >> https://lwn.net/Articles/736770/ > >> > >>Thanks James, > >> > >>after looking into bisection last night, i found that just before > >>i wanted to test out the 4.4.0-82 kernel, i found 3 stack traces > >>in my syslog. :( > >> > >>I guess, i'm dealing with race conditions now. But it seems the 79 > >>kernel still crashes wifi a lot less than later ones. > >> > >>How do i get line numbers into these traces? > > As the 4.4.0-79 kernel was sometimes crapping out, too, i decided to > try to test the latest kernel instead of bisecting after all. This > took a while because virtualbox was being a bitch. virtualbox-5.0 > doesn't bode well with virtualbox-dkms-51, so i ended up rebuilding > virtualbox-5.1 to prevent dependency hell. The vb-dkms package > doesn't do 4.14, so i ended up going with the 4.13 kernel that comes > with artful. You didn't say virtualbox was essential for reproducing the problem, so I'm continuing to exclude it from thought. If it is essential for reproducing, then you might contact them. Please do make sure you can exclude virtualbox as a cause. > This one pretty quickly loads my syslog with new error stacks. I > haven't tested actual behavior yet, but the logs don't look so hot. Do connections frequently keep dying as before? > I ran another wireless-info (attached) and appended some of the > syslog stuff to it. Thanks, you identified a line of code and cause; a WARN_ON in iwl_mvm_rx_tx_cmd_single; case TX_STATUS_FAIL_DEST_PS: /* In DQA, the FW should have stopped the queue and not * return this status */ WARN_ON(iwl_mvm_is_dqa_supported(mvm)); info->flags |= IEEE80211_TX_STAT_TX_FILTERED; break; But it is only a warning. If connections aren't dying, it may not be important to you. Please check you are using the most recent linux-firmware? > >Several methods, though by far the most common seems to be personal > >experience with offsets. > > > >When you don't have that personal experience, the methods are; > > > >1. using GDB against the .o file, > > > >2. using binutils objdump to disassemble .o file or vmlinuz, > > > >3. using GCC to generate assembly listings, > > > >See https://wiki.ubuntu.com/Kernel/KernelDebuggingTricks right down > >the end of page for the GDB method. > > I have gotten around to that part, yet, as i was busy with the > above, but it seems later versions have issues, too. However, you're still testing old source code. Several changes made since are worth testing, please either cherry-pick the patches or test a 4.14 rc kernel, and without involving dkms or virtualbox. Or, if new firmware fixes the problem, go with that instead. > -- > Mit freundlichen Grüßen/Best regards > > Mario Theodoridis > > ########## wireless info START ########## > [...]
On 24/10/17 23:01, James Cameron wrote: > Summary: WARN_ON(iwl_mvm_is_dqa_supported(mvm)) in > iwl_mvm_rx_tx_cmd_single with v4.13, but code is since changed. > > On Tue, Oct 24, 2017 at 09:56:31PM +0200, Mario Theodoridis wrote: >> Sorry for skipping the list one the last one. > > Sorry, that was my fault. It was a private message you replied to. > >> On 19.10.2017 22:59, James Cameron wrote: >> [...] > > You didn't say virtualbox was essential for reproducing the problem, > so I'm continuing to exclude it from thought. If it is essential for > reproducing, then you might contact them. > > Please do make sure you can exclude virtualbox as a cause. Let me clarify the virtualbox thing. The machine in question is a VM host. It hosts several machines, one of which is my mail server, and another (openbsd) which acts as a gateway to the internet for all machines. If i run this machine without virtualbox, then my entire network topology is off-line. While one could argue, that this is bad design, the alternative would be to use openbsd as a virtual host, but i haven't seen many tutorials on that. I also would like to run just one machine 24/7 to keep a tap on the electricity consumption. This machine also bridges several interfaces and acts as a hotspot for my wlan. So i don't know whether virtualbox is responsible, but not running virtualbox is simply not an option. >> This one pretty quickly loads my syslog with new error stacks. I >> haven't tested actual behavior yet, but the logs don't look so hot. > > Do connections frequently keep dying as before? > >> I ran another wireless-info (attached) and appended some of the >> syslog stuff to it. > > Thanks, you identified a line of code and cause; a WARN_ON in > iwl_mvm_rx_tx_cmd_single; > > case TX_STATUS_FAIL_DEST_PS: > /* In DQA, the FW should have stopped the queue and not > * return this status > */ > WARN_ON(iwl_mvm_is_dqa_supported(mvm)); > info->flags |= IEEE80211_TX_STAT_TX_FILTERED; > break; > > But it is only a warning. If connections aren't dying, it may not be > important to you. > > Please check you are using the most recent linux-firmware? I'm running ii linux-firmware 1.169 all from artful. No difference to the xenial version. > >>> Several methods, though by far the most common seems to be personal >>> experience with offsets. >>> >>> When you don't have that personal experience, the methods are; >>> >>> 1. using GDB against the .o file, >>> >>> 2. using binutils objdump to disassemble .o file or vmlinuz, >>> >>> 3. using GCC to generate assembly listings, >>> >>> See https://wiki.ubuntu.com/Kernel/KernelDebuggingTricks right down >>> the end of page for the GDB method. >> >> I have gotten around to that part, yet, as i was busy with the >> above, but it seems later versions have issues, too. > > However, you're still testing old source code. > > Several changes made since are worth testing, please either > cherry-pick the patches or test a 4.14 rc kernel, and without > involving dkms or virtualbox. Then i'd have to patch those files so they build for 4.14 first. I've seen patches, but still need to figure out how to get them applied in the build process.
On Wed, Oct 25, 2017 at 09:08:17AM +0200, Mario Theodoridis wrote: > On 24/10/17 23:01, James Cameron wrote: > >Summary: WARN_ON(iwl_mvm_is_dqa_supported(mvm)) in > >iwl_mvm_rx_tx_cmd_single with v4.13, but code is since changed. > > > >On Tue, Oct 24, 2017 at 09:56:31PM +0200, Mario Theodoridis wrote: > >>Sorry for skipping the list one the last one. > > > >Sorry, that was my fault. It was a private message you replied to. > > > >>On 19.10.2017 22:59, James Cameron wrote: > >>[...] > > > >You didn't say virtualbox was essential for reproducing the problem, > >so I'm continuing to exclude it from thought. If it is essential for > >reproducing, then you might contact them. > > > >Please do make sure you can exclude virtualbox as a cause. > > Let me clarify the virtualbox thing. The machine in question is a VM host. > It hosts several machines, one of which is my mail server, and another > (openbsd) which acts as a gateway to the internet for all machines. > If i run this machine without virtualbox, then my entire network topology is > off-line. While one could argue, that this is bad design, the alternative > would be to use openbsd as a virtual host, but i haven't seen many tutorials > on that. I also would like to run just one machine 24/7 to keep a tap on the > electricity consumption. > > This machine also bridges several interfaces and acts as a hotspot for my > wlan. > > So i don't know whether virtualbox is responsible, but not running > virtualbox is simply not an option. Thanks. I don't have a machine with the same wireless device, so I can't hope to reproduce the problem or test fixes. I do have a slightly later wireless device which uses the same driver, but I'm not confident it would reproduce the problem, because (a) I've not seen the same stack traces, (b) the WARN_ON relates to device response coded in firmware, and my wireless device may use different firmware, and (c) it isn't clear to me what you did to enable the problem. You do have a machine, and you might do tests without virtualbox, but as you say, this is not an option for you. > >>This one pretty quickly loads my syslog with new error stacks. I > >>haven't tested actual behavior yet, but the logs don't look so hot. > > > >Do connections frequently keep dying as before? > > > >>I ran another wireless-info (attached) and appended some of the > >>syslog stuff to it. > > > >Thanks, you identified a line of code and cause; a WARN_ON in > >iwl_mvm_rx_tx_cmd_single; > > > > case TX_STATUS_FAIL_DEST_PS: > > /* In DQA, the FW should have stopped the queue and not > > * return this status > > */ > > WARN_ON(iwl_mvm_is_dqa_supported(mvm)); > > info->flags |= IEEE80211_TX_STAT_TX_FILTERED; > > break; > > > >But it is only a warning. If connections aren't dying, it may not be > >important to you. > > > >Please check you are using the most recent linux-firmware? > > I'm running > ii linux-firmware 1.169 all > from artful. > No difference to the xenial version. Good, thanks. > >>>Several methods, though by far the most common seems to be personal > >>>experience with offsets. > >>> > >>>When you don't have that personal experience, the methods are; > >>> > >>>1. using GDB against the .o file, > >>> > >>>2. using binutils objdump to disassemble .o file or vmlinuz, > >>> > >>>3. using GCC to generate assembly listings, > >>> > >>>See https://wiki.ubuntu.com/Kernel/KernelDebuggingTricks right down > >>>the end of page for the GDB method. > >> > >>I have gotten around to that part, yet, as i was busy with the > >>above, but it seems later versions have issues, too. > > > >However, you're still testing old source code. > > > >Several changes made since are worth testing, please either > >cherry-pick the patches or test a 4.14 rc kernel, and without > >involving dkms or virtualbox. > > Then i'd have to patch those files so they build for 4.14 first. > I've seen patches, but still need to figure out how to get them > applied in the build process. It may be more efficient to wait for your dkms packagers to catch up so that the v4.14-rc6 or v4.14 kernel will work with your package configuration. > -- > Mit freundlichen Grüßen/Best Regards > > Mario Theodoridis
Hi James, On 24.10.2017 23:01, James Cameron wrote: > But it is only a warning. If connections aren't dying, it may not be > important to you. Regarding whether wifi hangs, it's usually takes a while to get going and then disappears. Sunday night i ended up rebooting into the 4.4-79 kernel because the 4.13 just got too ridiculous. I.e. Wlan off, wlan on no longer worked. > Please check you are using the most recent linux-firmware? Just in case i haven't answered that it's at 1.169 >>> Several methods, though by far the most common seems to be personal >>> experience with offsets. >>> >>> When you don't have that personal experience, the methods are; >>> >>> 1. using GDB against the .o file, >>> >>> 2. using binutils objdump to disassemble .o file or vmlinuz, >>> >>> 3. using GCC to generate assembly listings, >>> >>> See https://wiki.ubuntu.com/Kernel/KernelDebuggingTricks right down >>> the end of page for the GDB method. >> >> I have gotten around to that part, yet, as i was busy with the >> above, but it seems later versions have issues, too. > > However, you're still testing old source code. > > Several changes made since are worth testing, please either > cherry-pick the patches or test a 4.14 rc kernel, and without > involving dkms or virtualbox. > > Or, if new firmware fixes the problem, go with that instead. I just managed to patch the 5.1.30 dkms package so i wouldn't need to update to virtualbox-5.2. Here are the results with the 4.14-rc7 kernel. As last time i appended what fills my syslog now.
On 31.10.2017 20:25, Mario Theodoridis wrote: > Hi James, > > > On 24.10.2017 23:01, James Cameron wrote: >> But it is only a warning. If connections aren't dying, it may not be >> important to you. > > Regarding whether wifi hangs, it's usually takes a while to get going > and then disappears. Sunday night i ended up rebooting into the 4.4-79 > kernel because the 4.13 just got too ridiculous. > I.e. Wlan off, wlan on no longer worked. > > >> Please check you are using the most recent linux-firmware? > > Just in case i haven't answered that it's at 1.169 > > >>>> Several methods, though by far the most common seems to be personal >>>> experience with offsets. >>>> >>>> When you don't have that personal experience, the methods are; >>>> >>>> 1. using GDB against the .o file, >>>> >>>> 2. using binutils objdump to disassemble .o file or vmlinuz, >>>> >>>> 3. using GCC to generate assembly listings, >>>> >>>> See https://wiki.ubuntu.com/Kernel/KernelDebuggingTricks right down >>>> the end of page for the GDB method. >>> >>> I have gotten around to that part, yet, as i was busy with the >>> above, but it seems later versions have issues, too. >> >> However, you're still testing old source code. >> >> Several changes made since are worth testing, please either >> cherry-pick the patches or test a 4.14 rc kernel, and without >> involving dkms or virtualbox. >> >> Or, if new firmware fixes the problem, go with that instead. > > I just managed to patch the 5.1.30 dkms package so i wouldn't need to > update to virtualbox-5.2. > > > Here are the results with the 4.14-rc7 kernel. > As last time i appended what fills my syslog now. Yes, maybe i ought to attach them ;)
diff --git a/drivers/net/wireless/iwlwifi/dvm/mac80211.c b/drivers/net/wireless/iwlwifi/dvm/mac80211.c index b3ad34e..1eb1a82 100644 --- a/drivers/net/wireless/iwlwifi/dvm/mac80211.c +++ b/drivers/net/wireless/iwlwifi/dvm/mac80211.c @@ -729,12 +729,15 @@ static inline bool iwl_enable_tx_ampdu(const struct iwl_cfg *cfg) static int iwlagn_mac_ampdu_action(struct ieee80211_hw *hw, struct ieee80211_vif *vif, - enum ieee80211_ampdu_mlme_action action, - struct ieee80211_sta *sta, u16 tid, u16 *ssn, - u8 buf_size, bool amsdu) + struct ieee80211_ampdu_params *params) { struct iwl_priv *priv = IWL_MAC80211_GET_DVM(hw); int ret = -EINVAL; + struct ieee80211_sta *sta = params->sta; + enum ieee80211_ampdu_mlme_action action = params->action; + u16 tid = params->tid; + u16 *ssn = ¶ms->ssn; + u8 buf_size = params->buf_size; struct iwl_station_priv *sta_priv = (void *) sta->drv_priv; IWL_DEBUG_HT(priv, "A-MPDU action on addr %pM tid %d\n", diff --git a/drivers/net/wireless/iwlwifi/mvm/mac80211.c b/drivers/net/wireless/iwlwifi/mvm/mac80211.c index ce12717..1a8ea77 100644 --- a/drivers/net/wireless/iwlwifi/mvm/mac80211.c +++ b/drivers/net/wireless/iwlwifi/mvm/mac80211.c @@ -826,13 +826,16 @@ iwl_mvm_ampdu_check_trigger(struct iwl_mvm *mvm, struct ieee80211_vif *vif, static int iwl_mvm_mac_ampdu_action(struct ieee80211_hw *hw, struct ieee80211_vif *vif, - enum ieee80211_ampdu_mlme_action action, - struct ieee80211_sta *sta, u16 tid, - u16 *ssn, u8 buf_size, bool amsdu) + struct ieee80211_ampdu_params *params) { struct iwl_mvm *mvm = IWL_MAC80211_GET_MVM(hw); int ret; bool tx_agg_ref = false; + struct ieee80211_sta *sta = params->sta; + enum ieee80211_ampdu_mlme_action action = params->action; + u16 tid = params->tid; + u16 *ssn = ¶ms->ssn; + u8 buf_size = params->buf_size; IWL_DEBUG_HT(mvm, "A-MPDU action on addr %pM tid %d: action %d\n", sta->addr, tid, action); diff --git a/drivers/net/wireless/mac80211_hwsim.c b/drivers/net/wireless/mac80211_hwsim.c index 0cd9512..019d716 100644 --- a/drivers/net/wireless/mac80211_hwsim.c +++ b/drivers/net/wireless/mac80211_hwsim.c @@ -1817,10 +1817,12 @@ static int mac80211_hwsim_testmode_cmd(struct ieee80211_hw *hw, static int mac80211_hwsim_ampdu_action(struct ieee80211_hw *hw, struct ieee80211_vif *vif, - enum ieee80211_ampdu_mlme_action action, - struct ieee80211_sta *sta, u16 tid, u16 *ssn, - u8 buf_size, bool amsdu) + struct ieee80211_ampdu_params *params) { + struct ieee80211_sta *sta = params->sta; + enum ieee80211_ampdu_mlme_action action = params->action; + u16 tid = params->tid; + switch (action) { case IEEE80211_AMPDU_TX_START: ieee80211_start_tx_ba_cb_irqsafe(vif, sta->addr, tid); @@ -2537,7 +2539,7 @@ static int mac80211_hwsim_new_radio(struct genl_info *info, tasklet_hrtimer_init(&data->beacon_timer, mac80211_hwsim_beacon, - CLOCK_MONOTONIC_RAW, HRTIMER_MODE_ABS); + CLOCK_MONOTONIC, HRTIMER_MODE_ABS); spin_lock_bh(&hwsim_radio_lock); list_add_tail(&data->list, &hwsim_radios);