| Message ID | 20200103150722.20313-2-jschoenh@amazon.de (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
| Series | x86/mce: Various fixes and cleanups for MCE handling | expand |
On Fri, Jan 03, 2020 at 04:07:17PM +0100, Jan H. Schönherr wrote: > Commit fa92c5869426 ("x86, mce: Support memory error recovery for both > UCNA and Deferred error in machine_check_poll") added handling of UCNA > and Deferred errors by adding them to the ring for SRAO errors. > > Later, commit fd4cf79fcc4b ("x86/mce: Remove the MCE ring for Action > Optional errors") switched storage from the SRAO ring to the unified > pool that is still in use today. In order to only act on the intended > errors, a filter for MCE_AO_SEVERITY is used -- effectively removing > handling of UCNA/Deferred errors again. > > Extend the severity filter to include UCNA/Deferred errors again. > Also, generalize the naming of the notifier from SRAO to UC to capture > the extended scope. > > Note, that this change may cause a message like the following to appear, > as the same address may be reported as SRAO and as UCNA: > > Memory failure: 0x5fe3284: already hardware poisoned > > Technically, this is a return to previous behavior. > > Fixes: fd4cf79fcc4b ("x86/mce: Remove the MCE ring for Action Optional errors") > Signed-off-by: Jan H. Schönherr <jschoenh@amazon.de> Tony, ACK? Also, do you want it in stable@ so that it gets backported? > diff --git a/arch/x86/kernel/cpu/mce/core.c b/arch/x86/kernel/cpu/mce/core.c > index 8994fe7751a4..16134ce587fd 100644 > --- a/arch/x86/kernel/cpu/mce/core.c > +++ b/arch/x86/kernel/cpu/mce/core.c > @@ -156,10 +156,8 @@ void mce_log(struct mce *m) > } > EXPORT_SYMBOL_GPL(mce_log); > > -static struct notifier_block mce_srao_nb; > - > /* > - * We run the default notifier if we have only the SRAO, the first and the > + * We run the default notifier if we have only the UC, the first and the > * default notifier registered. I.e., the mandatory NUM_DEFAULT_NOTIFIERS > * notifiers registered on the chain. > */ > @@ -580,26 +578,29 @@ static struct notifier_block first_nb = { > .priority = MCE_PRIO_FIRST, > }; > > -static int srao_decode_notifier(struct notifier_block *nb, unsigned long val, > - void *data) > +static int uc_decode_notifier(struct notifier_block *nb, unsigned long val, > + void *data) > { > struct mce *mce = (struct mce *)data; > unsigned long pfn; > > - if (!mce) > + if (!mce || !mce_usable_address(mce)) > return NOTIFY_DONE; > > - if (mce_usable_address(mce) && (mce->severity == MCE_AO_SEVERITY)) { > - pfn = mce->addr >> PAGE_SHIFT; > - if (!memory_failure(pfn, 0)) > - set_mce_nospec(pfn); > - } > + if (mce->severity != MCE_AO_SEVERITY && > + mce->severity != MCE_DEFERRED_SEVERITY) > + return NOTIFY_DONE; > + > + pfn = mce->addr >> PAGE_SHIFT; > + if (!memory_failure(pfn, 0)) > + set_mce_nospec(pfn); I'm wondering if in the memory_failure error() case, we should hand it down to the remaining notifiers. Which also begs the question in light of this clumsy notifier counting: How about we have the default notifier *unconditionally* print the MCE? I.e., if the error has reached it, it would print it. If not and some other notifier consumed it, it will get handled differently. This way we won't need any special counting of notifiers and special reg/unreg of notifiers etc. IOW, the logic would be: If something consumes the error, then it doesn't get printed. Notifier does NOTIFY_STOP. If nothing consumes it or something looks at it and decides that it should still get printed, then the last catch-all notifier callback does that. Thoughts?
On Fri, Jan 10, 2020 at 10:50:04AM +0100, Borislav Petkov wrote: > On Fri, Jan 03, 2020 at 04:07:17PM +0100, Jan H. Schönherr wrote: > > Commit fa92c5869426 ("x86, mce: Support memory error recovery for both > > UCNA and Deferred error in machine_check_poll") added handling of UCNA > > and Deferred errors by adding them to the ring for SRAO errors. > > > > Later, commit fd4cf79fcc4b ("x86/mce: Remove the MCE ring for Action > > Optional errors") switched storage from the SRAO ring to the unified > > pool that is still in use today. In order to only act on the intended > > errors, a filter for MCE_AO_SEVERITY is used -- effectively removing > > handling of UCNA/Deferred errors again. > > > > Extend the severity filter to include UCNA/Deferred errors again. > > Also, generalize the naming of the notifier from SRAO to UC to capture > > the extended scope. > > > > Note, that this change may cause a message like the following to appear, > > as the same address may be reported as SRAO and as UCNA: > > > > Memory failure: 0x5fe3284: already hardware poisoned > > > > Technically, this is a return to previous behavior. > > > > Fixes: fd4cf79fcc4b ("x86/mce: Remove the MCE ring for Action Optional errors") > > Signed-off-by: Jan H. Schönherr <jschoenh@amazon.de> > > Tony, ACK? Acked-by: Tony Luck <tony.luck@intel.com> > Also, do you want it in stable@ so that it gets backported? That's a tricky question. We have changing behavior (UCNA pages offlined, then a few kernel versions stopped doing this, now we are going to start doing it again. But is it really a _BUG_ that needs backporting to stable? I'm leaning towards "no it isn't". But could perhaps be convinced to change my mind if somebody has a good reason for wanting it there. Is there something to put in the tags to stop this being autoselected for backport because it has a Fixes: tag? > I'm wondering if in the memory_failure error() case, we should hand it > down to the remaining notifiers. > > Which also begs the question in light of this clumsy notifier counting: > > How about we have the default notifier *unconditionally* print the MCE? > I.e., if the error has reached it, it would print it. If not and some > other notifier consumed it, it will get handled differently. > > This way we won't need any special counting of notifiers and special > reg/unreg of notifiers etc. > > IOW, the logic would be: > > If something consumes the error, then it doesn't get printed. Notifier > does NOTIFY_STOP. > > If nothing consumes it or something looks at it and decides that it > should still get printed, then the last catch-all notifier callback does > that. I totally agree that counting notifiers is clumsy. Also less than ideal is the concept that any notifier on the chain can declare: "I fixed it" and prevent any other notifiers from even seeing it. Well the concept is good, but it is overused. I think we may do better with a field in the "struct mce" that is being passed to each where notifiers can wiggle some bits (semantics to be defined later) which can tell subsequent notifiers what sort of actions have been taken. E.g. the SRAO/UCNA notifier can say "I took this page offline" the dev_mcelog one can say "I think I handed to a process that has /dev/mcelog open" EDAC drivers can say "I decoded the address and printed something" CEC can say: "I silently counted this corrected error", or "error exceeded threshold and I took the page offline". The default notifier can print to console if nobody set a bit to say that the error had been somehow logged. -Tony
On Fri, Jan 10, 2020 at 10:45:33AM -0800, Luck, Tony wrote: > That's a tricky question. We have changing behavior (UCNA pages offlined, > then a few kernel versions stopped doing this, now we are going to start > doing it again. But is it really a _BUG_ that needs backporting to stable? > I'm leaning towards "no it isn't". I think the same, so let's not. We can always backport it later, if it turns out that it is needed. > But could perhaps be convinced to change my mind if somebody has a > good reason for wanting it there. Yah. > Is there something to put in the tags to stop this being autoselected > for backport because it has a Fixes: tag? I don't think so. What we could do is write the "Fixes:" tag in free text in the commit message so that tools don't pick up on it. I'll reply to the other thing below in a separate mail. Thx.
On Fri, Jan 10, 2020 at 10:45:33AM -0800, Luck, Tony wrote: > I totally agree that counting notifiers is clumsy. Also less than > ideal is the concept that any notifier on the chain can declare: > "I fixed it" > and prevent any other notifiers from even seeing it. Well the concept > is good, but it is overused. But why can't we use it? Don't get me wrong: I'm simply following my KISS approach to do the simplest scheme required. So, do you see a use case where the whole error handling chain would need more sophisticated handling? > I think we may do better with a field in the "struct mce" that is being > passed to each where notifiers can wiggle some bits (semantics to be > defined later) which can tell subsequent notifiers what sort of actions > have been taken. > E.g. the SRAO/UCNA notifier can say "I took this page offline" > the dev_mcelog one can say "I think I handed to a process that has /dev/mcelog open" > EDAC drivers can say "I decoded the address and printed something" > CEC can say: "I silently counted this corrected error", or "error exceeded > threshold and I took the page offline". > > The default notifier can print to console if nobody set a bit to say > that the error had been somehow logged. That idea is good and I'll gladly take patches for it so if you wanna do it... Thx.
diff --git a/arch/x86/include/asm/mce.h b/arch/x86/include/asm/mce.h index dc2d4b206ab7..c8ff6f6750ef 100644 --- a/arch/x86/include/asm/mce.h +++ b/arch/x86/include/asm/mce.h @@ -144,7 +144,7 @@ struct mce_log_buffer { enum mce_notifier_prios { MCE_PRIO_FIRST = INT_MAX, - MCE_PRIO_SRAO = INT_MAX - 1, + MCE_PRIO_UC = INT_MAX - 1, MCE_PRIO_EXTLOG = INT_MAX - 2, MCE_PRIO_NFIT = INT_MAX - 3, MCE_PRIO_EDAC = INT_MAX - 4, diff --git a/arch/x86/kernel/cpu/mce/core.c b/arch/x86/kernel/cpu/mce/core.c index 8994fe7751a4..16134ce587fd 100644 --- a/arch/x86/kernel/cpu/mce/core.c +++ b/arch/x86/kernel/cpu/mce/core.c @@ -156,10 +156,8 @@ void mce_log(struct mce *m) } EXPORT_SYMBOL_GPL(mce_log); -static struct notifier_block mce_srao_nb; - /* - * We run the default notifier if we have only the SRAO, the first and the + * We run the default notifier if we have only the UC, the first and the * default notifier registered. I.e., the mandatory NUM_DEFAULT_NOTIFIERS * notifiers registered on the chain. */ @@ -580,26 +578,29 @@ static struct notifier_block first_nb = { .priority = MCE_PRIO_FIRST, }; -static int srao_decode_notifier(struct notifier_block *nb, unsigned long val, - void *data) +static int uc_decode_notifier(struct notifier_block *nb, unsigned long val, + void *data) { struct mce *mce = (struct mce *)data; unsigned long pfn; - if (!mce) + if (!mce || !mce_usable_address(mce)) return NOTIFY_DONE; - if (mce_usable_address(mce) && (mce->severity == MCE_AO_SEVERITY)) { - pfn = mce->addr >> PAGE_SHIFT; - if (!memory_failure(pfn, 0)) - set_mce_nospec(pfn); - } + if (mce->severity != MCE_AO_SEVERITY && + mce->severity != MCE_DEFERRED_SEVERITY) + return NOTIFY_DONE; + + pfn = mce->addr >> PAGE_SHIFT; + if (!memory_failure(pfn, 0)) + set_mce_nospec(pfn); return NOTIFY_OK; } -static struct notifier_block mce_srao_nb = { - .notifier_call = srao_decode_notifier, - .priority = MCE_PRIO_SRAO, + +static struct notifier_block mce_uc_nb = { + .notifier_call = uc_decode_notifier, + .priority = MCE_PRIO_UC, }; static int mce_default_notifier(struct notifier_block *nb, unsigned long val, @@ -1970,7 +1971,7 @@ int __init mcheck_init(void) { mcheck_intel_therm_init(); mce_register_decode_chain(&first_nb); - mce_register_decode_chain(&mce_srao_nb); + mce_register_decode_chain(&mce_uc_nb); mce_register_decode_chain(&mce_default_nb); mcheck_vendor_init_severity();
Commit fa92c5869426 ("x86, mce: Support memory error recovery for both UCNA and Deferred error in machine_check_poll") added handling of UCNA and Deferred errors by adding them to the ring for SRAO errors. Later, commit fd4cf79fcc4b ("x86/mce: Remove the MCE ring for Action Optional errors") switched storage from the SRAO ring to the unified pool that is still in use today. In order to only act on the intended errors, a filter for MCE_AO_SEVERITY is used -- effectively removing handling of UCNA/Deferred errors again. Extend the severity filter to include UCNA/Deferred errors again. Also, generalize the naming of the notifier from SRAO to UC to capture the extended scope. Note, that this change may cause a message like the following to appear, as the same address may be reported as SRAO and as UCNA: Memory failure: 0x5fe3284: already hardware poisoned Technically, this is a return to previous behavior. Fixes: fd4cf79fcc4b ("x86/mce: Remove the MCE ring for Action Optional errors") Signed-off-by: Jan H. Schönherr <jschoenh@amazon.de> --- Changes v1->v2: - rename notifier from SRAO to UC (as requested by Tony) - extend commit message (per remark from Tony) - don't mention Linux versions (per remark from Boris) There was some discussion on v1, whether the SRAO/UC notifier does the right thing or not. While it seems to be correct as is for Intel (per Tony), there were some concerns for AMD (per Yazen). Hence, there is a new patch 5 in this series, which disables the notifier on AMD. --- arch/x86/include/asm/mce.h | 2 +- arch/x86/kernel/cpu/mce/core.c | 31 ++++++++++++++++--------------- 2 files changed, 17 insertions(+), 16 deletions(-)