| Message ID | 20201219214034.21123-1-ljp@linux.ibm.com (mailing list archive) |
|---|---|
| State | Superseded |
| Delegated to: | Netdev Maintainers |
| Headers | show |
| Series | [net] ibmvnic: continue fatal error reset after passive init | expand |
| Context | Check | Description |
|---|---|---|
| netdev/cover_letter | success | Link |
| netdev/fixes_present | success | Link |
| netdev/patch_count | success | Link |
| netdev/tree_selection | success | Clearly marked for net |
| netdev/subject_prefix | success | Link |
| netdev/cc_maintainers | warning | 9 maintainers not CCed: paulus@samba.org kuba@kernel.org drt@linux.ibm.com benh@kernel.crashing.org mpe@ellerman.id.au sukadev@linux.ibm.com linuxppc-dev@lists.ozlabs.org davem@davemloft.net julietk@linux.vnet.ibm.com |
| netdev/source_inline | success | Was 0 now: 0 |
| netdev/verify_signedoff | success | Link |
| netdev/module_param | success | Was 0 now: 0 |
| netdev/build_32bit | success | Errors and warnings before: 0 this patch: 0 |
| netdev/kdoc | success | Errors and warnings before: 19 this patch: 19 |
| netdev/verify_fixes | success | Link |
| netdev/checkpatch | success | total: 0 errors, 0 warnings, 0 checks, 9 lines checked |
| netdev/build_allmodconfig_warn | success | Errors and warnings before: 0 this patch: 0 |
| netdev/header_inline | success | Link |
| netdev/stable | success | Stable not CCed |
On Sat, 19 Dec 2020 15:40:34 -0600 Lijun Pan wrote: > Commit f9c6cea0b385 ("ibmvnic: Skip fatal error reset after passive init") > says "If the passive > CRQ initialization occurs before the FATAL reset task is processed, > the FATAL error reset task would try to access a CRQ message queue > that was freed, causing an oops. The problem may be most likely to > occur during DLPAR add vNIC with a non-default MTU, because the DLPAR > process will automatically issue a change MTU request. > Fix this by not processing fatal error reset if CRQ is passively > initialized after client-driven CRQ initialization fails." > > Even with this commit, we still see similar kernel crashes. In order > to completely solve this problem, we'd better continue the fatal error > reset, capture the kernel crash, and try to fix it from that end. This basically reverts the quoted fix. Does the quoted fix make things worse? Otherwise we should leave the code be until proper fix is found.
On Tue, Dec 22, 2020 at 8:48 PM Jakub Kicinski <kuba@kernel.org> wrote: > > On Sat, 19 Dec 2020 15:40:34 -0600 Lijun Pan wrote: > > Commit f9c6cea0b385 ("ibmvnic: Skip fatal error reset after passive init") > > says "If the passive > > CRQ initialization occurs before the FATAL reset task is processed, > > the FATAL error reset task would try to access a CRQ message queue > > that was freed, causing an oops. The problem may be most likely to > > occur during DLPAR add vNIC with a non-default MTU, because the DLPAR > > process will automatically issue a change MTU request. > > Fix this by not processing fatal error reset if CRQ is passively > > initialized after client-driven CRQ initialization fails." > > > > Even with this commit, we still see similar kernel crashes. In order > > to completely solve this problem, we'd better continue the fatal error > > reset, capture the kernel crash, and try to fix it from that end. > > This basically reverts the quoted fix. Does the quoted fix make things > worse? Otherwise we should leave the code be until proper fix is found. Yes, I think the quoted commit makes things worse. It skips the specific reset condition, but that does not fix the problem it claims to fix. The effective fix is upstream SHA 0e435befaea4 and a0faaa27c716. So I think reverting it to the original "else" condition is the right thing to do.
On Wed, 23 Dec 2020 02:21:09 -0600 Lijun Pan wrote: > On Tue, Dec 22, 2020 at 8:48 PM Jakub Kicinski <kuba@kernel.org> wrote: > > On Sat, 19 Dec 2020 15:40:34 -0600 Lijun Pan wrote: > > > Commit f9c6cea0b385 ("ibmvnic: Skip fatal error reset after passive init") > > > says "If the passive > > > CRQ initialization occurs before the FATAL reset task is processed, > > > the FATAL error reset task would try to access a CRQ message queue > > > that was freed, causing an oops. The problem may be most likely to > > > occur during DLPAR add vNIC with a non-default MTU, because the DLPAR > > > process will automatically issue a change MTU request. > > > Fix this by not processing fatal error reset if CRQ is passively > > > initialized after client-driven CRQ initialization fails." > > > > > > Even with this commit, we still see similar kernel crashes. In order > > > to completely solve this problem, we'd better continue the fatal error > > > reset, capture the kernel crash, and try to fix it from that end. > > > > This basically reverts the quoted fix. Does the quoted fix make things > > worse? Otherwise we should leave the code be until proper fix is found. > > Yes, I think the quoted commit makes things worse. It skips the specific > reset condition, but that does not fix the problem it claims to fix. Okay, let's make sure the commit message explains how it makes things worse. > The effective fix is upstream SHA 0e435befaea4 and a0faaa27c716. So I > think reverting it to the original "else" condition is the right thing to do. Hm. So the problem is fixed? But the commit message says "we still see similar kernel crashes", that's present tense suggesting that crashes are seen on current net/master. Are you saying that's not the case and after 0e435befaea4 and a0faaa27c716 there are no more crashes?
On Wed, Dec 23, 2020 at 10:50 AM Jakub Kicinski <kuba@kernel.org> wrote: > > On Wed, 23 Dec 2020 02:21:09 -0600 Lijun Pan wrote: > > On Tue, Dec 22, 2020 at 8:48 PM Jakub Kicinski <kuba@kernel.org> wrote: > > > On Sat, 19 Dec 2020 15:40:34 -0600 Lijun Pan wrote: > > > > Commit f9c6cea0b385 ("ibmvnic: Skip fatal error reset after passive init") > > > > says "If the passive > > > > CRQ initialization occurs before the FATAL reset task is processed, > > > > the FATAL error reset task would try to access a CRQ message queue > > > > that was freed, causing an oops. The problem may be most likely to > > > > occur during DLPAR add vNIC with a non-default MTU, because the DLPAR > > > > process will automatically issue a change MTU request. > > > > Fix this by not processing fatal error reset if CRQ is passively > > > > initialized after client-driven CRQ initialization fails." > > > > > > > > Even with this commit, we still see similar kernel crashes. In order > > > > to completely solve this problem, we'd better continue the fatal error > > > > reset, capture the kernel crash, and try to fix it from that end. > > > > > > This basically reverts the quoted fix. Does the quoted fix make things > > > worse? Otherwise we should leave the code be until proper fix is found. > > > > Yes, I think the quoted commit makes things worse. It skips the specific > > reset condition, but that does not fix the problem it claims to fix. > > Okay, let's make sure the commit message explains how it makes things > worse. I will reword the commit message. > > > The effective fix is upstream SHA 0e435befaea4 and a0faaa27c716. So I > > think reverting it to the original "else" condition is the right thing to do. > > Hm. So the problem is fixed? But the commit message says "we still see > similar kernel crashes", that's present tense suggesting that crashes > are seen on current net/master. Are you saying that's not the case and > after 0e435befaea4 and a0faaa27c716 there are no more crashes? This patch was formed before I submitted 0e435befaea4 and a0faaa27c716, so I used the wording "we still see similar kernel crashes". I will modify the commit message before I submit v2 of this patch. After 0e435befaea4 and a0faaa27c716, I don't see any crashes as described in this quoted commit even without this quoted commit. That's why I am sure this quoted commit does not fix the described problem and I want to revert it.
On Wed, 23 Dec 2020 14:10:32 -0600 Lijun Pan wrote: > On Wed, Dec 23, 2020 at 10:50 AM Jakub Kicinski <kuba@kernel.org> wrote: > > > > On Wed, 23 Dec 2020 02:21:09 -0600 Lijun Pan wrote: > > > On Tue, Dec 22, 2020 at 8:48 PM Jakub Kicinski <kuba@kernel.org> wrote: > > > > On Sat, 19 Dec 2020 15:40:34 -0600 Lijun Pan wrote: > > > > > Commit f9c6cea0b385 ("ibmvnic: Skip fatal error reset after passive init") > > > > > says "If the passive > > > > > CRQ initialization occurs before the FATAL reset task is processed, > > > > > the FATAL error reset task would try to access a CRQ message queue > > > > > that was freed, causing an oops. The problem may be most likely to > > > > > occur during DLPAR add vNIC with a non-default MTU, because the DLPAR > > > > > process will automatically issue a change MTU request. > > > > > Fix this by not processing fatal error reset if CRQ is passively > > > > > initialized after client-driven CRQ initialization fails." > > > > > > > > > > Even with this commit, we still see similar kernel crashes. In order > > > > > to completely solve this problem, we'd better continue the fatal error > > > > > reset, capture the kernel crash, and try to fix it from that end. > > > > > > > > This basically reverts the quoted fix. Does the quoted fix make things > > > > worse? Otherwise we should leave the code be until proper fix is found. > > > > > > Yes, I think the quoted commit makes things worse. It skips the specific > > > reset condition, but that does not fix the problem it claims to fix. > > > > Okay, let's make sure the commit message explains how it makes things > > worse. > > I will reword the commit message. > > > > The effective fix is upstream SHA 0e435befaea4 and a0faaa27c716. So I > > > think reverting it to the original "else" condition is the right thing to do. > > > > Hm. So the problem is fixed? But the commit message says "we still see > > similar kernel crashes", that's present tense suggesting that crashes > > are seen on current net/master. Are you saying that's not the case and > > after 0e435befaea4 and a0faaa27c716 there are no more crashes? > > This patch was formed before I submitted 0e435befaea4 and a0faaa27c716, so > I used the wording "we still see similar kernel crashes". I will modify > the commit message before I submit v2 of this patch. > After 0e435befaea4 and a0faaa27c716, I don't see any crashes as described > in this quoted commit even without this quoted commit. > That's why I am sure this quoted commit does not fix the described problem > and I want to revert it. I see, that explains it!
diff --git a/drivers/net/ethernet/ibm/ibmvnic.c b/drivers/net/ethernet/ibm/ibmvnic.c index b370c88a43f1..237a36040689 100644 --- a/drivers/net/ethernet/ibm/ibmvnic.c +++ b/drivers/net/ethernet/ibm/ibmvnic.c @@ -2342,8 +2342,7 @@ static void __ibmvnic_reset(struct work_struct *work) set_current_state(TASK_UNINTERRUPTIBLE); schedule_timeout(60 * HZ); } - } else if (!(rwi->reset_reason == VNIC_RESET_FATAL && - adapter->from_passive_init)) { + } else { rc = do_reset(adapter, rwi, reset_state); } kfree(rwi);
Commit f9c6cea0b385 ("ibmvnic: Skip fatal error reset after passive init") says "If the passive CRQ initialization occurs before the FATAL reset task is processed, the FATAL error reset task would try to access a CRQ message queue that was freed, causing an oops. The problem may be most likely to occur during DLPAR add vNIC with a non-default MTU, because the DLPAR process will automatically issue a change MTU request. Fix this by not processing fatal error reset if CRQ is passively initialized after client-driven CRQ initialization fails." Even with this commit, we still see similar kernel crashes. In order to completely solve this problem, we'd better continue the fatal error reset, capture the kernel crash, and try to fix it from that end. Fixes: f9c6cea0b385 ("ibmvnic: Skip fatal error reset after passive init") Signed-off-by: Lijun Pan <ljp@linux.ibm.com> --- drivers/net/ethernet/ibm/ibmvnic.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)