diff mbox

[3/3] ghes_edac: add platform check to enable ghes_edac

Message ID 20170718060007.GB8736@nazgul.tnic (mailing list archive)
State Not Applicable, archived
Headers show

Commit Message

Borislav Petkov July 18, 2017, 6 a.m. UTC
On Mon, Jul 17, 2017 at 03:59:12PM -0600, Toshi Kani wrote:
> The ghes_edac driver was introduced in 2013 [1], but it has not
> been enabled by any distro yet.  This driver obtains error info
> from firmware interfaces, which are not properly implemented on
> many platforms, as the driver always emits the messages below:
> 
>  This EDAC driver relies on BIOS to enumerate memory and get error reports.
>  Unfortunately, not all BIOSes reflect the memory layout correctly
>  So, the end result of using this driver varies from vendor to vendor
>  If you find incorrect reports, please contact your hardware vendor
>  to correct its BIOS.
> 
> To get out from this situation, add a platform type check to
> selectively enable the driver on the platforms that are known to
> have proper firmware implementation.  Platform vendors can add
> their platforms to the list when they support ghes_edac.

So maintaining whitelists for things has always been a PITA and we
should try to avoid it, if possible. (We can always do it if nothing
saner comes along.)

Now, below is a dirty patch converting ghes_edac to a normal module.
On systems where we have GHES, the firmware generally disables the
detection of the presence of ECC hardware, thus preventing the platform
EDAC driver from loading.

Let me clarify: I have an AMD HP box which, when GHES is enabled in
the BIOS, says that ECC is disabled in the memory controller and the
amd64_edac driver doesn't load for that memory controller.

And I think we should try this first: have the firmware disable
detection methods so that the platform drivers don't load.

Then, ghes_edac can be a simple module and no other driver would attempt
loading.

The question is: does the platform do this disabling now?

Tony, I'm looking at sb_edac and there we don't do something like that
or maybe I'm missing it.

Hmmm.

---
From: Borislav Petkov <bp@suse.de>
Date: Thu, 29 Jun 2017 10:28:32 +0200
Subject: [PATCH] WIP

Not-Signed-off-by: Borislav Petkov <bp@suse.de>
---
 drivers/acpi/apei/ghes.c |  32 ++++++-----
 drivers/edac/Kconfig     |   4 +-
 drivers/edac/edac_mc.h   |   3 ++
 drivers/edac/ghes_edac.c | 137 ++++++++++++++++++++++++-----------------------
 include/acpi/ghes.h      |  27 +---------
 5 files changed, 98 insertions(+), 105 deletions(-)

Comments

Borislav Petkov July 18, 2017, 8:08 a.m. UTC | #1
On Tue, Jul 18, 2017 at 08:00:07AM +0200, Borislav Petkov wrote:
> And I think we should try this first: have the firmware disable
> detection methods so that the platform drivers don't load.

Btw, in looking at this more, what about the firmware-first thing?

I.e., the firmware-first detection with apei_osc_setup() at the end of
ghes_init().

Can we make ghes_edac loading dependent on that? I mean, that was *the*
predicate for exactly that - to have the firmware look at the errors
first. No need for platform whitelisting and so on.

I'd still decouple ghes_edac loading from ghes_probe() even though
loading the platform driver should've been done *after* the
firmware-first detection regardless.

So what we could do is make ghes_edac a normal module and have the
relevant x86 EDAC modules query FF mode and if enabled, fail loading.

Hmmm?

My gut feeling tells me I'm on the right track here but who knows...

Thx.
Kani, Toshi July 18, 2017, 7:58 p.m. UTC | #2
T24gVHVlLCAyMDE3LTA3LTE4IGF0IDA4OjAwICswMjAwLCBCb3Jpc2xhdiBQZXRrb3Ygd3JvdGU6
DQo+IE9uIE1vbiwgSnVsIDE3LCAyMDE3IGF0IDAzOjU5OjEyUE0gLTA2MDAsIFRvc2hpIEthbmkg
d3JvdGU6DQo+ID4gVGhlIGdoZXNfZWRhYyBkcml2ZXIgd2FzIGludHJvZHVjZWQgaW4gMjAxMyBb
MV0sIGJ1dCBpdCBoYXMgbm90DQo+ID4gYmVlbiBlbmFibGVkIGJ5IGFueSBkaXN0cm8geWV0LsKg
wqBUaGlzIGRyaXZlciBvYnRhaW5zIGVycm9yIGluZm8NCj4gPiBmcm9tIGZpcm13YXJlIGludGVy
ZmFjZXMsIHdoaWNoIGFyZSBub3QgcHJvcGVybHkgaW1wbGVtZW50ZWQgb24NCj4gPiBtYW55IHBs
YXRmb3JtcywgYXMgdGhlIGRyaXZlciBhbHdheXMgZW1pdHMgdGhlIG1lc3NhZ2VzIGJlbG93Og0K
PiA+IA0KPiA+IMKgVGhpcyBFREFDIGRyaXZlciByZWxpZXMgb24gQklPUyB0byBlbnVtZXJhdGUg
bWVtb3J5IGFuZCBnZXQgZXJyb3INCj4gPiByZXBvcnRzLiDCoFVuZm9ydHVuYXRlbHksIG5vdCBh
bGwgQklPU2VzIHJlZmxlY3QgdGhlIG1lbW9yeSBsYXlvdXQNCj4gPiBjb3JyZWN0bHkgwqBTbywg
dGhlIGVuZCByZXN1bHQgb2YgdXNpbmcgdGhpcyBkcml2ZXIgdmFyaWVzIGZyb20NCj4gPiB2ZW5k
b3IgdG8gdmVuZG9yIMKgSWYgeW91IGZpbmQgaW5jb3JyZWN0IHJlcG9ydHMsIHBsZWFzZSBjb250
YWN0DQo+ID4geW91ciBoYXJkd2FyZSB2ZW5kb3IgwqB0byBjb3JyZWN0IGl0cyBCSU9TLg0KPiA+
IA0KPiA+IFRvIGdldCBvdXQgZnJvbSB0aGlzIHNpdHVhdGlvbiwgYWRkIGEgcGxhdGZvcm0gdHlw
ZSBjaGVjayB0bw0KPiA+IHNlbGVjdGl2ZWx5IGVuYWJsZSB0aGUgZHJpdmVyIG9uIHRoZSBwbGF0
Zm9ybXMgdGhhdCBhcmUga25vd24gdG8NCj4gPiBoYXZlIHByb3BlciBmaXJtd2FyZSBpbXBsZW1l
bnRhdGlvbi7CoMKgUGxhdGZvcm0gdmVuZG9ycyBjYW4gYWRkDQo+ID4gdGhlaXIgcGxhdGZvcm1z
IHRvIHRoZSBsaXN0IHdoZW4gdGhleSBzdXBwb3J0IGdoZXNfZWRhYy4NCj4gDQo+IFNvIG1haW50
YWluaW5nIHdoaXRlbGlzdHMgZm9yIHRoaW5ncyBoYXMgYWx3YXlzIGJlZW4gYSBQSVRBIGFuZCB3
ZQ0KPiBzaG91bGQgdHJ5IHRvIGF2b2lkIGl0LCBpZiBwb3NzaWJsZS4gKFdlIGNhbiBhbHdheXMg
ZG8gaXQgaWYgbm90aGluZw0KPiBzYW5lciBjb21lcyBhbG9uZy4pDQoNCkFncmVlZC4NCg0KPiBO
b3csIGJlbG93IGlzIGEgZGlydHkgcGF0Y2ggY29udmVydGluZyBnaGVzX2VkYWMgdG8gYSBub3Jt
YWwgbW9kdWxlLg0KPiBPbiBzeXN0ZW1zIHdoZXJlIHdlIGhhdmUgR0hFUywgdGhlIGZpcm13YXJl
IGdlbmVyYWxseSBkaXNhYmxlcyB0aGUNCj4gZGV0ZWN0aW9uIG9mIHRoZSBwcmVzZW5jZSBvZiBF
Q0MgaGFyZHdhcmUsIHRodXMgcHJldmVudGluZyB0aGUNCj4gcGxhdGZvcm0gRURBQyBkcml2ZXIg
ZnJvbSBsb2FkaW5nLg0KDQpJIGhhdmUgSFBFIEhhc3dlbGwgYW5kIFNreWxha2UgdGVzdCBzeXN0
ZW1zIHdpdGggR0hFUywgYnV0IHRoZXkgZG8gbm90DQpoaWRlIElNQ3MgZnJvbSB0aGUgT1MuICBT
bywgdGhlIHNiX2VkYWMgYW5kIHNreF9lZGFjIGRyaXZlcnMgZ2V0DQphdHRhY2hlZCBvbiB0aGVz
ZSBzeXN0ZW1zIHdoZW4gZ2hlc19lZGFjIGlzIGRpc2FibGVkLg0KDQo+IExldCBtZSBjbGFyaWZ5
OiBJIGhhdmUgYW4gQU1EIEhQIGJveCB3aGljaCwgd2hlbiBHSEVTIGlzIGVuYWJsZWQgaW4NCj4g
dGhlIEJJT1MsIHNheXMgdGhhdCBFQ0MgaXMgZGlzYWJsZWQgaW4gdGhlIG1lbW9yeSBjb250cm9s
bGVyIGFuZCB0aGUNCj4gYW1kNjRfZWRhYyBkcml2ZXIgZG9lc24ndCBsb2FkIGZvciB0aGF0IG1l
bW9yeSBjb250cm9sbGVyLg0KDQpIbW0uLi4gd2hhdCdzIHRoZSBwbGF0Zm9ybSBuYW1lIG9mIHRo
aXMgYm94PyAgSSBjYW4gbG9vayBpbnRvIHRoaXMgY2FzZQ0KaWYgeW91IG5lZWQuDQoNCj4gQW5k
IEkgdGhpbmsgd2Ugc2hvdWxkIHRyeSB0aGlzIGZpcnN0OiBoYXZlIHRoZSBmaXJtd2FyZSBkaXNh
YmxlDQo+IGRldGVjdGlvbiBtZXRob2RzIHNvIHRoYXQgdGhlIHBsYXRmb3JtIGRyaXZlcnMgZG9u
J3QgbG9hZC4NCg0KSSBkbyBub3QgdGhpbmsgd2UgY2FuIHJlbHkgb24gdGhpcyBtZXRob2QuDQoN
Cj4gVGhlbiwgZ2hlc19lZGFjIGNhbiBiZSBhIHNpbXBsZSBtb2R1bGUgYW5kIG5vIG90aGVyIGRy
aXZlciB3b3VsZA0KPiBhdHRlbXB0IGxvYWRpbmcuDQoNCkkgbGlrZSB0aGUgdXNlIG9mIG5vdGlm
aWVyIGNoYWluLCB3aGljaCBpcyBtdWNoIGNsZWFuZXIuDQoNCj4gVGhlIHF1ZXN0aW9uIGlzOiBk
b2VzIHRoZSBwbGF0Zm9ybSBkbyB0aGlzIGRpc2FibGluZyBub3c/DQoNClVuZm9ydHVuYXRlbHks
IHRoYXQgaXMgbm90IHRoZSBjYXNlIHRvZGF5LiAgVGhlIElNQ3MgY2Fubm90IGJlIGhpZGRlbg0K
d2l0aCB0aGUgRGV2aWNlIEhpZGUgcmVnaXN0ZXJzIGZvciBTa3lsYWtlIGF0IGxlYXN0Lg0KDQpU
aGFua3MsDQotVG9zaGkNCg0K
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Mauro Carvalho Chehab July 18, 2017, 9:15 p.m. UTC | #3
Em Tue, 18 Jul 2017 19:58:54 +0000
"Kani, Toshimitsu" <toshi.kani@hpe.com> escreveu:

> On Tue, 2017-07-18 at 08:00 +0200, Borislav Petkov wrote:
> > On Mon, Jul 17, 2017 at 03:59:12PM -0600, Toshi Kani wrote:  
> > > The ghes_edac driver was introduced in 2013 [1], but it has not
> > > been enabled by any distro yet.  This driver obtains error info
> > > from firmware interfaces, which are not properly implemented on
> > > many platforms, as the driver always emits the messages below:
> > > 
> > >  This EDAC driver relies on BIOS to enumerate memory and get error
> > > reports.  Unfortunately, not all BIOSes reflect the memory layout
> > > correctly  So, the end result of using this driver varies from
> > > vendor to vendor  If you find incorrect reports, please contact
> > > your hardware vendor  to correct its BIOS.
> > > 
> > > To get out from this situation, add a platform type check to
> > > selectively enable the driver on the platforms that are known to
> > > have proper firmware implementation.  Platform vendors can add
> > > their platforms to the list when they support ghes_edac.  
> > 
> > So maintaining whitelists for things has always been a PITA and we
> > should try to avoid it, if possible. (We can always do it if nothing
> > saner comes along.)  
> 
> Agreed.
> 
> > Now, below is a dirty patch converting ghes_edac to a normal module.
> > On systems where we have GHES, the firmware generally disables the
> > detection of the presence of ECC hardware, thus preventing the
> > platform EDAC driver from loading.  
> 
> I have HPE Haswell and Skylake test systems with GHES, but they do not
> hide IMCs from the OS.  So, the sb_edac and skx_edac drivers get
> attached on these systems when ghes_edac is disabled.
> 
> > Let me clarify: I have an AMD HP box which, when GHES is enabled in
> > the BIOS, says that ECC is disabled in the memory controller and the
> > amd64_edac driver doesn't load for that memory controller.  
> 
> Hmm... what's the platform name of this box?  I can look into this case
> if you need.
> 
> > And I think we should try this first: have the firmware disable
> > detection methods so that the platform drivers don't load.  
> 
> I do not think we can rely on this method.
> 
> > Then, ghes_edac can be a simple module and no other driver would
> > attempt loading.  
> 
> I like the use of notifier chain, which is much cleaner.
> 
> > The question is: does the platform do this disabling now?  
> 
> Unfortunately, that is not the case today.  The IMCs cannot be hidden
> with the Device Hide registers for Skylake at least.

We had a similar discussion several years ago when I wrote this driver.
On that time, I talked with Red Hat, HP, Dell, Intel people and with
some customers with large clusters.

The way it is, ghes_edac is a poor man's driver. What it hopefully
provide is a detection that an error happened, without really telling
the user what component should be replaced.

Ok, on machines with their own error reporting mechanism (like
HP servers), a sys admin can look on some proprietary software
(or bios), in order to identify what happened.

Yet, BIOS doesn't provide any glue about what's the memory architecture,
as it maps memory as if it was a single DIMM memory:

(from ghes_edac_register)

	layers[0].type = EDAC_MC_LAYER_ALL_MEM;
	layers[0].size = num_dimm;
	layers[0].is_virt_csrow = true;

So, even on systems where the BIOS actually knows how the memory
cards are wired, it will mask the memory controller data.

Now, the EDAC driver can also be used to identify what
channels are used. That helps the sys admin to know if the
memories are connected in a way that it will be using multiple
channels, or not, helping to setup the machine to obtain
the maximum possible performance.

So, for example, on my Intel-based HP server, I can check
such info with:

$ ras-mc-ctl --mainboard
ras-mc-ctl: mainboard: HP model ProLiant ML350 Gen9
$ ras-mc-ctl --layout
       +-----------------------------------------------------------------------+
       |                mc0                |                mc1                |
       | channel0  | channel1  | channel2  | channel0  | channel1  | channel2  |
-------+-----------------------------------------------------------------------+
slot2: |     0 MB  |     0 MB  |     0 MB  |     0 MB  |     0 MB  |     0 MB  |
slot1: |     0 MB  |     0 MB  |     0 MB  |     0 MB  |     0 MB  |     0 MB  |
slot0: |  16384 MB  |     0 MB  |  16384 MB  |  16384 MB  |     0 MB  |  16384 MB  |
-------+---------------------------------------------------------------------------+

So, I know that both CPUs will be connected to my memories, and,
on both, it is using 2 channels.

If I was using the ghes driver, that information would be hidden.

So, due to all problems with ghes, it is enabled only if there are no
better solution, e. g. on systems where there's no way to talk directly
to the hardware (like on E7 Xeon machines, where the memory controller
is actually on a separate chip that are controlled only by the BIOS).

Thanks,
Mauro
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Kani, Toshi July 18, 2017, 9:20 p.m. UTC | #4
On Tue, 2017-07-18 at 10:08 +0200, Borislav Petkov wrote:
> On Tue, Jul 18, 2017 at 08:00:07AM +0200, Borislav Petkov wrote:

> > And I think we should try this first: have the firmware disable

> > detection methods so that the platform drivers don't load.

> 

> Btw, in looking at this more, what about the firmware-first thing?

> 

> I.e., the firmware-first detection with apei_osc_setup() at the end

> of ghes_init().

> 

> Can we make ghes_edac loading dependent on that? I mean, that was

> *the* predicate for exactly that - to have the firmware look at the

> errors first. No need for platform whitelisting and so on.


I agree that 'osc_sb_apei_support_acked' should be checked when
enabling ghes_edac.  I do not know the details of existing issues, but
it sounds unlikely that this will address all of them since bugs can be
everywhere.  For instance, ghes_edac relies on DMI/SMBIOS info, unlike
other EDAC drivers, which can be buggy regardless of this _OSC info.

> I'd still decouple ghes_edac loading from ghes_probe() even though

> loading the platform driver should've been done *after* the

> firmware-first detection regardless.

> 

> So what we could do is make ghes_edac a normal module and have the

> relevant x86 EDAC modules query FF mode and if enabled, fail loading.


I agree that making ghes_edac as a normal module is a good thing, but I
do not think it's going to solve this issue.

Thanks,
-Toshi
Tony Luck July 18, 2017, 10:13 p.m. UTC | #5
> The question is: does the platform do this disabling now?

>

> Tony, I'm looking at sb_edac and there we don't do something like that

> or maybe I'm missing it.


Historically we've had complaints that sb_edac won't load that have been
tracked to BIOS hiding one of the (many) PCI devices that it needs.  But
device hiding is orthogonal to providing GHES error records.  A BIOS might
do that, but I don't know that anyone intentionally does so.

-Tony
Borislav Petkov July 19, 2017, 5:52 a.m. UTC | #6
On Tue, Jul 18, 2017 at 09:20:44PM +0000, Kani, Toshimitsu wrote:
> I agree that 'osc_sb_apei_support_acked' should be checked when
> enabling ghes_edac.  I do not know the details of existing issues, but
> it sounds unlikely that this will address all of them since bugs can be
> everywhere.

No, see below.

> For instance, ghes_edac relies on DMI/SMBIOS info, unlike
> other EDAC drivers, which can be buggy regardless of this _OSC info.

That's the problem with firmware. You can't really fix it and it is
buggy as hell.

> I agree that making ghes_edac as a normal module is a good thing, but I
> do not think it's going to solve this issue.

Of course it will - if the firmware says it wants to look at the errors
first, then it gets to do so. This is the whole handling of hardware
errors in the firmware deal. I admit, sometimes it makes sense because
the firmware has the most intimate knowledge of the platform and, in
a perfect world, we won't ever need to have platform-specific EDAC
drivers.

But, we don't live in a perfect world. And the vendor execution of the
whole firmware-error-handling deal is an abomination at best.

So, if we realize that the firmware is buggy, we can use a platform list
to blacklist it (^hint hint^) and have a parameter to disable ghes_edac
from loading.

But we'll deal with that when we get to cross that bridge. Right now,
I'd like to do the loading spec-conform and not fiddle with white-,
black-, or any-other-color lists.
Borislav Petkov July 19, 2017, 5:55 a.m. UTC | #7
On Tue, Jul 18, 2017 at 07:58:54PM +0000, Kani, Toshimitsu wrote:
> I have HPE Haswell and Skylake test systems with GHES, but they do not
> hide IMCs from the OS.  So, the sb_edac and skx_edac drivers get
> attached on these systems when ghes_edac is disabled.

That's how it is supposed to work. The platform drivers are the
fallback, practically.

But this is the important piece of info I was looking for - having GHES
enabled in the firmware does not prevent the platform drivers from
loading. But I think we have a better solution, the FF thing.

> Hmm... what's the platform name of this box?  I can look into this case
> if you need.

You can but that's not addressing the issue as a whole so it'll be a
waste of time.
Borislav Petkov July 19, 2017, 5:58 a.m. UTC | #8
On Tue, Jul 18, 2017 at 06:15:45PM -0300, Mauro Carvalho Chehab wrote:
> The way it is, ghes_edac is a poor man's driver. What it hopefully
> provide is a detection that an error happened, without really telling
> the user what component should be replaced.

I beg to differ. From the UEFI spec:

"The module number of the memory error location. (NODE, CARD, and MODULE
should provide the information necessary to identify the failing FRU)."

So this tuple is sufficient to pinpoint the DIMM, IIUC.

Which means, ghes_edac can have a single layer of DIMMs without channels.
Borislav Petkov July 19, 2017, 6:01 a.m. UTC | #9
On Tue, Jul 18, 2017 at 10:13:42PM +0000, Luck, Tony wrote:
> Historically we've had complaints that sb_edac won't load that have been
> tracked to BIOS hiding one of the (many) PCI devices that it needs.  But
> device hiding is orthogonal to providing GHES error records.  A BIOS might
> do that, but I don't know that anyone intentionally does so.

Yeah, the hiding-devices path doesn't look like the optimal one. I think we
should look at the firmware-first setting and load ghes if FF is being done by
the firmware.
Tony Luck July 19, 2017, 3:14 p.m. UTC | #10
> "The module number of the memory error location. (NODE, CARD, and MODULE

> should provide the information necessary to identify the failing FRU)."

>

> So this tuple is sufficient to pinpoint the DIMM, IIUC.

>

> Which means, ghes_edac can have a single layer of DIMMs without channels.


The tricky part is that you have to rely on SMBIOS/DMI to know what DIMMs are
on the system when the driver initializes so you can populate /sys/.*/edac

Later when GHES gives you a NODE/CARD/MODULE) in an error record.  You need
to match these up. But SMBIOS only gave you two strings "Locator" and "Bank
Locator" which have no defined syntax. You are at the mercy of the BIOS writer
to put in something parseable. Some writers used zero based counts, others are
Fortran fans and use one-based. Still other use letters.  About the one guarantee
is that they will make almost no effort to match the silkscreen labels on the motherboard
itself.

E.g. my Broadwell-EX has things like:

        Locator: CHANNEL D DIMM 1
        Bank Locator: Memriser8

Channel is A,B,C,D. DIMM is 0, 1, 2. Memriser is {1..8} so this manages to use all
three counting options!

-Tony
Borislav Petkov July 19, 2017, 3:57 p.m. UTC | #11
On Wed, Jul 19, 2017 at 03:14:32PM +0000, Luck, Tony wrote:
> Later when GHES gives you a NODE/CARD/MODULE) in an error record.  You need
> to match these up. But SMBIOS only gave you two strings "Locator" and "Bank
> Locator" which have no defined syntax. You are at the mercy of the BIOS writer
> to put in something parseable.

Well, at some point it is only so much we can do, right?

I mean, if FW says it wants to do firmware-first and we go and adhere
to that, it should be expected that said FW vendor marks the silkscreen
labels and DMI data accordingly.

I mean, it is time for FW to put its money where its mouth is, no?

How else would you do this?

Firmware First but the kernel does the figuring out which DIMMs are
where. So FW can't have the cake and eat it too.

:-)
Mauro Carvalho Chehab July 19, 2017, 4:02 p.m. UTC | #12
Tony/Aris,

I got yesterday an HP ML350 G9, equipped with Sandy Bridge EP CPUs (E5-2640v4).
I'm running Kernel 4.11 there.

AFAIKT, Sandy Bridge EP has 4 channels per memory controller, right?
That would match the number of memory slots on this machine (24 slots).

Yet, EDAC is only identifying 3 channels per CPU:

 $  ras-mc-ctl --layout
       +-----------------------------------------------------------------------+
       |                mc0                |                mc1                |
       | channel0  | channel1  | channel2  | channel0  | channel1  | channel2  |
-------+-----------------------------------------------------------------------+
slot2: |     0 MB  |     0 MB  |     0 MB  |     0 MB  |     0 MB  |     0 MB  |
slot1: |     0 MB  |     0 MB  |     0 MB  |     0 MB  |     0 MB  |     0 MB  |
slot0: |  16384 MB  |     0 MB  |  16384 MB  |  16384 MB  |     0 MB  |  16384 MB  |
-------+---------------------------------------------------------------------------+

So, it seems that either the BIOS is hidden the other channel or
there's something wrong with SandyBridge EP support at sb_edac driver.

Thanks,
Mauro
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Kani, Toshi July 19, 2017, 4:10 p.m. UTC | #13
On Wed, 2017-07-19 at 07:52 +0200, Borislav Petkov wrote:
> On Tue, Jul 18, 2017 at 09:20:44PM +0000, Kani, Toshimitsu wrote:

> > I agree that 'osc_sb_apei_support_acked' should be checked when

> > enabling ghes_edac.  I do not know the details of existing issues,

> > but it sounds unlikely that this will address all of them since

> > bugs can be everywhere.

> 

> No, see below.

> 

> > For instance, ghes_edac relies on DMI/SMBIOS info, unlike

> > other EDAC drivers, which can be buggy regardless of this _OSC

> > info.

> 

> That's the problem with firmware. You can't really fix it and it is

> buggy as hell.


Right, and that's what I was told as an issue for ghes_edac.  This is
why this patch introduces a white-list to preclude all buggy firmwares
that are unknown to us...

> > I agree that making ghes_edac as a normal module is a good thing,

> > but I do not think it's going to solve this issue.

> 

> Of course it will - if the firmware says it wants to look at the

> errors first, then it gets to do so. This is the whole handling of

> hardware errors in the firmware deal. I admit, sometimes it makes

> sense because the firmware has the most intimate knowledge of the

> platform and, in a perfect world, we won't ever need to have

> platform-specific EDAC drivers.

>

> But, we don't live in a perfect world. And the vendor execution of

> the whole firmware-error-handling deal is an abomination at best.

> 

> So, if we realize that the firmware is buggy, we can use a platform

> list to blacklist it (^hint hint^) and have a parameter to disable

> ghes_edac from loading.


Setting blacklist needs us to enable ghes_edac and find all buggy
firmwares to date.  I think this is too disturbing for people who are
happily using regular edac drivers today even though their platforms
have GHES.

> But we'll deal with that when we get to cross that bridge. Right now,

> I'd like to do the loading spec-conform and not fiddle with white-,

> black-, or any-other-color lists.


I do prefer to avoid any white / black listing.  But I do not see how
it solves the buggy DMI/SMBIOS info as an example of firmware bugs we
may have to deal with.

Thanks,
-Toshi
Borislav Petkov July 19, 2017, 4:22 p.m. UTC | #14
On Wed, Jul 19, 2017 at 04:10:07PM +0000, Kani, Toshimitsu wrote:
> I do prefer to avoid any white / black listing.  But I do not see how
> it solves the buggy DMI/SMBIOS info as an example of firmware bugs we
> may have to deal with.

So how do you want to deal with this?

Maintain an evergrowing whitelist of platforms which are OK and then the
moment a new platform comes along, you send a patch to add it to that
whitelist?

I'm sure you can see the problems with that approach.
Kani, Toshi July 19, 2017, 4:40 p.m. UTC | #15
On Tue, 2017-07-18 at 18:15 -0300, Mauro Carvalho Chehab wrote:
> Em Tue, 18 Jul 2017 19:58:54 +0000

 :
> We had a similar discussion several years ago when I wrote this

> driver. On that time, I talked with Red Hat, HP, Dell, Intel people

> and with some customers with large clusters.

> 

> The way it is, ghes_edac is a poor man's driver. What it hopefully

> provide is a detection that an error happened, without really telling

> the user what component should be replaced.


"poor man's driver" is a bit misleading, but yes, firmware-first
platforms have RAS features built-into the platforms, and they do not
need intelligence in EDAC drivers, which may conflict with the
platform's RAS features.  I cannot speak for other vendors, but HPE
platforms log errors and provide FRU info.  ghes_edac allows to report
errors to OS management tools like rasdaemon in addition to platform-
specific managements.

> Ok, on machines with their own error reporting mechanism (like

> HP servers), a sys admin can look on some proprietary software

> (or bios), in order to identify what happened.

> 

> Yet, BIOS doesn't provide any glue about what's the memory

> architecture, as it maps memory as if it was a single DIMM memory:

> 

> (from ghes_edac_register)

> 

> 	layers[0].type = EDAC_MC_LAYER_ALL_MEM;

> 	layers[0].size = num_dimm;

> 	layers[0].is_virt_csrow = true;

> 

> So, even on systems where the BIOS actually knows how the memory

> cards are wired, it will mask the memory controller data.

> 

> Now, the EDAC driver can also be used to identify what

> channels are used. That helps the sys admin to know if the

> memories are connected in a way that it will be using multiple

> channels, or not, helping to setup the machine to obtain

> the maximum possible performance.

> 

> So, for example, on my Intel-based HP server, I can check

> such info with:

> 

> $ ras-mc-ctl --mainboard

> ras-mc-ctl: mainboard: HP model ProLiant ML350 Gen9

> $ ras-mc-ctl --layout

>        +-------------------------------------------------------------

> ----------+

>        |                mc0                |                mc1      

>           |

>        | channel0  | channel1  | channel2  | channel0  | channel1  |

> channel2  |

> -------+-------------------------------------------------------------

> ----------+

> slot2: |     0 MB  |     0 MB  |     0 MB  |     0 MB  |     0

> MB  |     0 MB  |

> slot1: |     0 MB  |     0 MB  |     0 MB  |     0 MB  |     0

> MB  |     0 MB  |

> slot0: |  16384 MB  |     0 MB  |  16384 MB  |  16384 MB  |     0

> MB  |  16384 MB  |

> -------+-------------------------------------------------------------

> --------------+

> 

> So, I know that both CPUs will be connected to my memories, and,

> on both, it is using 2 channels.

> 

> If I was using the ghes driver, that information would be hidden.

> 

> So, due to all problems with ghes, it is enabled only if there are no

> better solution, e. g. on systems where there's no way to talk

> directly to the hardware (like on E7 Xeon machines, where the memory

> controller is actually on a separate chip that are controlled only by

> the BIOS).


Thanks for the info!  That's very helpful.  I will check to see if
ghes_edac provides enough info that we need.
-Toshi
Kani, Toshi July 19, 2017, 4:56 p.m. UTC | #16
On Wed, 2017-07-19 at 18:22 +0200, Borislav Petkov wrote:
> On Wed, Jul 19, 2017 at 04:10:07PM +0000, Kani, Toshimitsu wrote:

> > I do prefer to avoid any white / black listing.  But I do not see

> > how it solves the buggy DMI/SMBIOS info as an example of firmware

> > bugs we may have to deal with.

> 

> So how do you want to deal with this?

> 

> Maintain an evergrowing whitelist of platforms which are OK and then

> the moment a new platform comes along, you send a patch to add it to

> that whitelist?

> 

> I'm sure you can see the problems with that approach.


Since ghes_edac has not been used for a long time, I have a feeling
that not so many vendors want to use it.  In the case of HPE, we do not
need to update with each platform since "HPE" "Server" will cover all
platforms we need.

Thanks,
-Toshi
Tony Luck July 19, 2017, 6:06 p.m. UTC | #17
Pj4gTGF0ZXIgd2hlbiBHSEVTIGdpdmVzIHlvdSBhIE5PREUvQ0FSRC9NT0RVTEUpIGluIGFuIGVy
cm9yIHJlY29yZC4gIFlvdSBuZWVkDQo+PiB0byBtYXRjaCB0aGVzZSB1cC4gQnV0IFNNQklPUyBv
bmx5IGdhdmUgeW91IHR3byBzdHJpbmdzICJMb2NhdG9yIiBhbmQgIkJhbmsNCj4+IExvY2F0b3Ii
IHdoaWNoIGhhdmUgbm8gZGVmaW5lZCBzeW50YXguIFlvdSBhcmUgYXQgdGhlIG1lcmN5IG9mIHRo
ZSBCSU9TIHdyaXRlcg0KPj4gdG8gcHV0IGluIHNvbWV0aGluZyBwYXJzZWFibGUuDQo+DQo+IFdl
bGwsIGF0IHNvbWUgcG9pbnQgaXQgaXMgb25seSBzbyBtdWNoIHdlIGNhbiBkbywgcmlnaHQ/DQo+
DQo+IEkgbWVhbiwgaWYgRlcgc2F5cyBpdCB3YW50cyB0byBkbyBmaXJtd2FyZS1maXJzdCBhbmQg
d2UgZ28gYW5kIGFkaGVyZQ0KPiB0byB0aGF0LCBpdCBzaG91bGQgYmUgZXhwZWN0ZWQgdGhhdCBz
YWlkIEZXIHZlbmRvciBtYXJrcyB0aGUgc2lsa3NjcmVlbg0KPiBsYWJlbHMgYW5kIERNSSBkYXRh
IGFjY29yZGluZ2x5Lg0KPg0KPiBJIG1lYW4sIGl0IGlzIHRpbWUgZm9yIEZXIHRvIHB1dCBpdHMg
bW9uZXkgd2hlcmUgaXRzIG1vdXRoIGlzLCBubz8NCj4NCj4gSG93IGVsc2Ugd291bGQgeW91IGRv
IHRoaXM/DQoNCkJ5IHRoaW5raW5nIGEgYml0IG1vcmUgYW5kIHJlYWxpemluZyB0aGF0IHdoYXQg
SSB3cm90ZSB1cCBhYm92ZSBtaXNzZXMgdGhhdA0KYXQgYnl0ZSBvZmZzZXQgNzggaW4gdGhlIFVF
RkkgbWVtb3J5IGVycm9yIHNlY3Rpb24gdGhlcmUgaXMgIk1vZHVsZSBIYW5kbGUiDQp3aGljaCB0
ZWxscyB5b3Ugd2hpY2ggU01CSU9TIGVudHJ5IHRvIHVzZS4NCg0KU28gdGhpcyBzaG91bGQgd29y
ayBqdXN0IGZpbmUgKGFzIGxvbmcgYXMgQklPUyBmaWxscyBvdXQgYWxsIHRoZXNlIGZpZWxkcyAu
Li4NCnRoZXJlJ3MgYSAiVmFsaWRhdGlvbiBCaXRzIiBtYXNrIGF0IHRoZSBzdGFydCBvZiB0aGUg
ZXJyb3Igc3RydWN0dXJlIHRoYXQgc2F5cw0Kd2hpY2ggZmllbGRzIGhhdmUgYmVlbiBwb3B1bGF0
ZWQpLg0KDQotVG9ueQ0K
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Aristeu Rozanski July 19, 2017, 6:55 p.m. UTC | #18
On Wed, Jul 19, 2017 at 06:22:04PM +0200, Borislav Petkov wrote:
> On Wed, Jul 19, 2017 at 04:10:07PM +0000, Kani, Toshimitsu wrote:
> > I do prefer to avoid any white / black listing.  But I do not see how
> > it solves the buggy DMI/SMBIOS info as an example of firmware bugs we
> > may have to deal with.
> 
> So how do you want to deal with this?
> 
> Maintain an evergrowing whitelist of platforms which are OK and then the
> moment a new platform comes along, you send a patch to add it to that
> whitelist?

That would also need to keep an eye on versions. A newer version of BIOS
on a whitelisted platform might be broken.
Tony Luck July 19, 2017, 8:06 p.m. UTC | #19
> So, it seems that either the BIOS is hidden the other channel or
> there's something wrong with SandyBridge EP support at sb_edac driver.

Can you send me the out of "lspci -xxxx" (run as root)?

-Tony
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Kani, Toshi July 19, 2017, 8:13 p.m. UTC | #20
T24gV2VkLCAyMDE3LTA3LTE5IGF0IDE0OjU1IC0wNDAwLCBBcmlzdGV1IFJvemFuc2tpIHdyb3Rl
Og0KPiBPbiBXZWQsIEp1bCAxOSwgMjAxNyBhdCAwNjoyMjowNFBNICswMjAwLCBCb3Jpc2xhdiBQ
ZXRrb3Ygd3JvdGU6DQo+ID4gT24gV2VkLCBKdWwgMTksIDIwMTcgYXQgMDQ6MTA6MDdQTSArMDAw
MCwgS2FuaSwgVG9zaGltaXRzdSB3cm90ZToNCj4gPiA+IEkgZG8gcHJlZmVyIHRvIGF2b2lkIGFu
eSB3aGl0ZSAvIGJsYWNrIGxpc3RpbmcuwqDCoEJ1dCBJIGRvIG5vdCBzZWUNCj4gPiA+IGhvdw0K
PiA+ID4gaXQgc29sdmVzIHRoZSBidWdneSBETUkvU01CSU9TIGluZm8gYXMgYW4gZXhhbXBsZSBv
ZiBmaXJtd2FyZQ0KPiA+ID4gYnVncyB3ZQ0KPiA+ID4gbWF5IGhhdmUgdG8gZGVhbCB3aXRoLg0K
PiA+IA0KPiA+IFNvIGhvdyBkbyB5b3Ugd2FudCB0byBkZWFsIHdpdGggdGhpcz8NCj4gPiANCj4g
PiBNYWludGFpbiBhbiBldmVyZ3Jvd2luZyB3aGl0ZWxpc3Qgb2YgcGxhdGZvcm1zIHdoaWNoIGFy
ZSBPSyBhbmQNCj4gPiB0aGVuIHRoZSBtb21lbnQgYSBuZXcgcGxhdGZvcm0gY29tZXMgYWxvbmcs
IHlvdSBzZW5kIGEgcGF0Y2ggdG8gYWRkDQo+ID4gaXQgdG8gdGhhdCB3aGl0ZWxpc3Q/DQo+IA0K
PiBUaGF0IHdvdWxkIGFsc28gbmVlZCB0byBrZWVwIGFuIGV5ZSBvbiB2ZXJzaW9ucy4gQSBuZXdl
ciB2ZXJzaW9uIG9mDQo+IEJJT1Mgb24gYSB3aGl0ZWxpc3RlZCBwbGF0Zm9ybSBtaWdodCBiZSBi
cm9rZW4uDQoNClJpZ2h0LiAgSSB0aGluayBhIHF1ZXN0aW9uIGNvbWVzIHRvIHdobyBicm9rZSBh
IHJ1bm5pbmcgc3lzdGVtIC0tIE9TDQp1cGRhdGUgb3IgQklPUyB1cGRhdGUuICBUaGlzIHdoaXRl
bGlzdCBhdHRlbXB0cyB0byBwcm90ZWN0IHRoZSBmb3JtZXINCmNhc2UgYnkgbm90IGludHJvZHVj
aW5nIGdoZXNfZWRhYyBvbiBhcmJpdHJhcnkgcGxhdGZvcm1zLiAgVGhlIGxhdHRlcg0KY2FzZSBz
aG91bGQgYmUgdmVuZG9yJ3MgcmVzcG9uc2liaWxpdHkuDQoNClRoYW5rcywNCi1Ub3NoaQ0K
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Borislav Petkov July 20, 2017, 4:16 a.m. UTC | #21
On Wed, Jul 19, 2017 at 04:56:17PM +0000, Kani, Toshimitsu wrote:
> Since ghes_edac has not been used for a long time, I have a feeling
> that not so many vendors want to use it.  In the case of HPE, we do not
> need to update with each platform since "HPE" "Server" will cover all
> platforms we need.

Does the apei_osc_setup() detection with the uuid work on HP systems?
Borislav Petkov July 20, 2017, 4:19 a.m. UTC | #22
On Wed, Jul 19, 2017 at 02:55:08PM -0400, Aristeu Rozanski wrote:
> That would also need to keep an eye on versions. A newer version of BIOS
> on a whitelisted platform might be broken.

Yeah, that would be a nasty, back-stabbing SNAFU.

So I'm thinking of adding a bunch of FW_ERR sanity checks to that whole
ghes_edac and ghes init code to hopefully catch issues during platform
validation. I.e., early enough for them to get fixed.

But that's the same problem as with UEFI - vendors need to try to boot
Linux on their platforms early enough.
Borislav Petkov July 20, 2017, 4:33 a.m. UTC | #23
On Wed, Jul 19, 2017 at 04:40:25PM +0000, Kani, Toshimitsu wrote:
>  ghes_edac allows to report errors to OS management tools like
> rasdaemon in addition to platform- specific managements.

So ghes_edac *is* a poor man's driver in the sense that it doesn't do
anything fancy but repeat like a parrot data it has gotten from the
firmware and shoving it into the EDAC counters. At least that's the
intention. Nothing more. All the action stuff like error detection and
recovery should be done by the firmware.

But considering how SNAFU'd firmware is, I wouldn't expect any great RAS
functionality there. Of course, I'd be delighted to be proven wrong.
Kani, Toshi July 20, 2017, 2:42 p.m. UTC | #24
On Thu, 2017-07-20 at 06:16 +0200, Borislav Petkov wrote:
> On Wed, Jul 19, 2017 at 04:56:17PM +0000, Kani, Toshimitsu wrote:

> > Since ghes_edac has not been used for a long time, I have a feeling

> > that not so many vendors want to use it.  In the case of HPE, we do

> > not need to update with each platform since "HPE" "Server" will

> > cover all platforms we need.

> 

> Does the apei_osc_setup() detection with the uuid work on HP systems?


Yes, the following message is shown on HP systems.  Please note that
WHEA is a Windows-defined interface.

"GHES: APEI firmware first mode is enabled by APEI bit and WHEA _OSC."

Thanks,
-Toshi
Borislav Petkov July 20, 2017, 3:04 p.m. UTC | #25
On Thu, Jul 20, 2017 at 02:42:25PM +0000, Kani, Toshimitsu wrote:
> Yes, the following message is shown on HP systems.  Please note that
> WHEA is a Windows-defined interface.

Ok, so let's couple ghes_edac loading to that and see how far we could
go. I guess we should add checks for that to the major x86 EDAC drivers
to not load and this way ghes_edac will be the only driver loading.

Tony, how does that sound?
Tony Luck July 20, 2017, 4:55 p.m. UTC | #26
Pj4gWWVzLCB0aGUgZm9sbG93aW5nIG1lc3NhZ2UgaXMgc2hvd24gb24gSFAgc3lzdGVtcy4gIFBs
ZWFzZSBub3RlIHRoYXQNCj4+IFdIRUEgaXMgYSBXaW5kb3dzLWRlZmluZWQgaW50ZXJmYWNlLg0K
Pg0KPiBPaywgc28gbGV0J3MgY291cGxlIGdoZXNfZWRhYyBsb2FkaW5nIHRvIHRoYXQgYW5kIHNl
ZSBob3cgZmFyIHdlIGNvdWxkDQo+IGdvLiBJIGd1ZXNzIHdlIHNob3VsZCBhZGQgY2hlY2tzIGZv
ciB0aGF0IHRvIHRoZSBtYWpvciB4ODYgRURBQyBkcml2ZXJzDQo+IHRvIG5vdCBsb2FkIGFuZCB0
aGlzIHdheSBnaGVzX2VkYWMgd2lsbCBiZSB0aGUgb25seSBkcml2ZXIgbG9hZGluZy4NCj4NCj4g
VG9ueSwgaG93IGRvZXMgdGhhdCBzb3VuZD8NCg0KQWRkIGEgbW9kdWxlIHBhcmFtZXRlciB0byB0
aG9zZSBlZGFjIGRyaXZlcnMgdGhhdCBjYW4gb3ZlcnJpZGUgdGhlIGNoZWNrDQphbmQgbGV0IHRo
ZW0gbG9hZCBhbnl3YXkuICBJJ20gbm90IHBhcmFub2lkLCBJIGp1c3QgYXNzdW1lIHRoYXQgdGhl
cmUgaXMgYSBCSU9TDQpvdXQgdGhlcmUgdGhhdCBzZXRzIHRoZSBPU0MvV0hFQSBiaXRzLCBidXQg
aXNuJ3QgZ2VuZXJhdGluZyB1c2VmdWwgR0hFUyBsb2dzLg0KDQotVG9ueQ0K
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Borislav Petkov July 20, 2017, 5:05 p.m. UTC | #27
On Thu, Jul 20, 2017 at 04:55:59PM +0000, Luck, Tony wrote:
> Add a module parameter to those edac drivers that can override the check
> and let them load anyway.  I'm not paranoid, I just assume that there is a BIOS
> out there that sets the OSC/WHEA bits, but isn't generating useful GHES logs.

Or add that parameter to edac_core.ko and let it control which EDAC
driver gets loaded? Something like

edac=ignore_ghes

or so. And then the other EDAC drivers query it.
Tony Luck July 20, 2017, 5:10 p.m. UTC | #28
PiBPciBhZGQgdGhhdCBwYXJhbWV0ZXIgdG8gZWRhY19jb3JlLmtvIGFuZCBsZXQgaXQgY29udHJv
bCB3aGljaCBFREFDDQo+IGRyaXZlciBnZXRzIGxvYWRlZD8gU29tZXRoaW5nIGxpa2UNCj4NCj4g
ZWRhYz1pZ25vcmVfZ2hlcw0KPg0KPiBvciBzby4gQW5kIHRoZW4gdGhlIG90aGVyIEVEQUMgZHJp
dmVycyBxdWVyeSBpdC4NCg0KU3VyZSAuLi4gb25lIGNlbnRyYWwgcGxhY2UgaXMgYmV0dGVyIHRo
YW4gYWRkaW5nIGNvZGUgdG8gZWFjaA0KZHJpdmVyLg0KDQotVG9ueQ0K
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Mauro Carvalho Chehab July 20, 2017, 6:16 p.m. UTC | #29
Em Thu, 20 Jul 2017 19:05:04 +0200
Borislav Petkov <bp@alien8.de> escreveu:

> On Thu, Jul 20, 2017 at 04:55:59PM +0000, Luck, Tony wrote:
> > Add a module parameter to those edac drivers that can override the check
> > and let them load anyway.  I'm not paranoid, I just assume that there is a BIOS
> > out there that sets the OSC/WHEA bits, but isn't generating useful GHES logs.  
> 
> Or add that parameter to edac_core.ko and let it control which EDAC
> driver gets loaded? Something like
> 
> edac=ignore_ghes
> 
> or so. And then the other EDAC drivers query it.

Works for me.

Thanks,
Mauro
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Kani, Toshi July 20, 2017, 7:50 p.m. UTC | #30
On Thu, 2017-07-20 at 06:33 +0200, Borislav Petkov wrote:
> On Wed, Jul 19, 2017 at 04:40:25PM +0000, Kani, Toshimitsu wrote:

> >  ghes_edac allows to report errors to OS management tools like

> > rasdaemon in addition to platform- specific managements.

> 

> So ghes_edac *is* a poor man's driver in the sense that it doesn't do

> anything fancy but repeat like a parrot data it has gotten from the

> firmware and shoving it into the EDAC counters. At least that's the

> intention. Nothing more. 


Right for ghes_edac.

> All the action stuff like error detection and recovery should be done

> by the firmware.


GHES / firmware-first still requires OS recovery actions when an error
cannot be corrected by the platform.  They are handled by ghes_proc(),
and ghes_edac remains its error-reporting wrapper.

> But considering how SNAFU'd firmware is, I wouldn't expect any great

> RAS functionality there. Of course, I'd be delighted to be proven

> wrong.


Firmware has better knowledge about the platform and can provide better
RAS when implemented properly.  I agree that user experiences may vary
on platforms.

Thanks,
-Toshi
Mauro Carvalho Chehab July 20, 2017, 8:15 p.m. UTC | #31
Em Thu, 20 Jul 2017 19:50:03 +0000
"Kani, Toshimitsu" <toshi.kani@hpe.com> escreveu:

> On Thu, 2017-07-20 at 06:33 +0200, Borislav Petkov wrote:
> > On Wed, Jul 19, 2017 at 04:40:25PM +0000, Kani, Toshimitsu wrote:  
> > >  ghes_edac allows to report errors to OS management tools like
> > > rasdaemon in addition to platform- specific managements.  
> > 
> > So ghes_edac *is* a poor man's driver in the sense that it doesn't do
> > anything fancy but repeat like a parrot data it has gotten from the
> > firmware and shoving it into the EDAC counters. At least that's the
> > intention. Nothing more.   
> 
> Right for ghes_edac.
> 
> > All the action stuff like error detection and recovery should be done
> > by the firmware.  
> 
> GHES / firmware-first still requires OS recovery actions when an error
> cannot be corrected by the platform.  They are handled by ghes_proc(),
> and ghes_edac remains its error-reporting wrapper.
> 
> > But considering how SNAFU'd firmware is, I wouldn't expect any great
> > RAS functionality there. Of course, I'd be delighted to be proven
> > wrong.  
> 
> Firmware has better knowledge about the platform and can provide better
> RAS when implemented properly.  I agree that user experiences may vary
> on platforms.

It may have a better knowledge, when the vendor ships different BIOS
for platforms with different motherboard silkscreens, but a lot of
vendors just use the same BIOS on different models, with the same
information at "Locator" and "Bank Locator" data at DMI tables,
that don't match what's printed at the board's silkscreen.

So, GHES ends by exposing wrong data. Also, such BIOS fail
to properly expose such knowledge to drivers/userspace.

On the discussions I had with HP, back in 2012, the idea was to try 
to have some sort of way for the GHES driver to query the BIOS
on a reliable way, in order to get its layout, in a way
that tools like ras-mc-ctl would properly report the memory
configuration (with --layout) and the motherboard silkscreen
labels (with --print-labels). Unfortunately, at least on that
time, the discussions with HP didn't proceed.

Thanks,
Mauro
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Kani, Toshi July 20, 2017, 9:07 p.m. UTC | #32
On Thu, 2017-07-20 at 17:15 -0300, Mauro Carvalho Chehab wrote:
> Em Thu, 20 Jul 2017 19:50:03 +0000

> "Kani, Toshimitsu" <toshi.kani@hpe.com> escreveu:

 :
> > Firmware has better knowledge about the platform and can provide

> > better RAS when implemented properly.  I agree that user

> > experiences may vary on platforms.

> 

> It may have a better knowledge, when the vendor ships different BIOS

> for platforms with different motherboard silkscreens, but a lot of

> vendors just use the same BIOS on different models, with the same

> information at "Locator" and "Bank Locator" data at DMI tables,

> that don't match what's printed at the board's silkscreen.

> 

> So, GHES ends by exposing wrong data. Also, such BIOS fail

> to properly expose such knowledge to drivers/userspace.


I see.  Yeah, I can see such problems could be overlooked since normal
tests run just fine even if there is a mismatch in such info...

> On the discussions I had with HP, back in 2012, the idea was to try 

> to have some sort of way for the GHES driver to query the BIOS

> on a reliable way, in order to get its layout, in a way

> that tools like ras-mc-ctl would properly report the memory

> configuration (with --layout) and the motherboard silkscreen

> labels (with --print-labels). Unfortunately, at least on that

> time, the discussions with HP didn't proceed.


Thanks for the info.  I hope we can enable it this time around.
-Toshi
Tony Luck July 20, 2017, 9:15 p.m. UTC | #33
On Wed, Jul 19, 2017 at 01:02:45PM -0300, Mauro Carvalho Chehab wrote:
> Tony/Aris,
> 
> I got yesterday an HP ML350 G9, equipped with Sandy Bridge EP CPUs (E5-2640v4).
> I'm running Kernel 4.11 there.
> 
> AFAIKT, Sandy Bridge EP has 4 channels per memory controller, right?
> That would match the number of memory slots on this machine (24 slots).
> 
> Yet, EDAC is only identifying 3 channels per CPU:
> 
>  $  ras-mc-ctl --layout
>        +-----------------------------------------------------------------------+
>        |                mc0                |                mc1                |
>        | channel0  | channel1  | channel2  | channel0  | channel1  | channel2  |
> -------+-----------------------------------------------------------------------+
> slot2: |     0 MB  |     0 MB  |     0 MB  |     0 MB  |     0 MB  |     0 MB  |
> slot1: |     0 MB  |     0 MB  |     0 MB  |     0 MB  |     0 MB  |     0 MB  |
> slot0: |  16384 MB  |     0 MB  |  16384 MB  |  16384 MB  |     0 MB  |  16384 MB  |
> -------+---------------------------------------------------------------------------+
> 
> So, it seems that either the BIOS is hidden the other channel or
> there's something wrong with SandyBridge EP support at sb_edac driver.

Does lspci show all four of these devices?

include/linux/pci_ids.h:#define PCI_DEVICE_ID_INTEL_SBRIDGE_IMC_TAD0    0x3caa  /* 15.2 */
include/linux/pci_ids.h:#define PCI_DEVICE_ID_INTEL_SBRIDGE_IMC_TAD1    0x3cab  /* 15.3 */
include/linux/pci_ids.h:#define PCI_DEVICE_ID_INTEL_SBRIDGE_IMC_TAD2    0x3cac  /* 15.4 */
include/linux/pci_ids.h:#define PCI_DEVICE_ID_INTEL_SBRIDGE_IMC_TAD3    0x3cad  /* 15.5 */

There should be two of each (one on bus 7f, the other on bus ff).

-Tony
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Mauro Carvalho Chehab July 21, 2017, midnight UTC | #34
Em Thu, 20 Jul 2017 14:15:54 -0700
"Luck, Tony" <tony.luck@intel.com> escreveu:

> On Wed, Jul 19, 2017 at 01:02:45PM -0300, Mauro Carvalho Chehab wrote:
> > Tony/Aris,
> > 
> > I got yesterday an HP ML350 G9, equipped with Sandy Bridge EP CPUs (E5-2640v4).
> > I'm running Kernel 4.11 there.
> > 
> > AFAIKT, Sandy Bridge EP has 4 channels per memory controller, right?
> > That would match the number of memory slots on this machine (24 slots).
> > 
> > Yet, EDAC is only identifying 3 channels per CPU:
> > 
> >  $  ras-mc-ctl --layout
> >        +-----------------------------------------------------------------------+
> >        |                mc0                |                mc1                |
> >        | channel0  | channel1  | channel2  | channel0  | channel1  | channel2  |
> > -------+-----------------------------------------------------------------------+
> > slot2: |     0 MB  |     0 MB  |     0 MB  |     0 MB  |     0 MB  |     0 MB  |
> > slot1: |     0 MB  |     0 MB  |     0 MB  |     0 MB  |     0 MB  |     0 MB  |
> > slot0: |  16384 MB  |     0 MB  |  16384 MB  |  16384 MB  |     0 MB  |  16384 MB  |
> > -------+---------------------------------------------------------------------------+
> > 
> > So, it seems that either the BIOS is hidden the other channel or
> > there's something wrong with SandyBridge EP support at sb_edac driver.  
> 
> Does lspci show all four of these devices?
> 
> include/linux/pci_ids.h:#define PCI_DEVICE_ID_INTEL_SBRIDGE_IMC_TAD0    0x3caa  /* 15.2 */
> include/linux/pci_ids.h:#define PCI_DEVICE_ID_INTEL_SBRIDGE_IMC_TAD1    0x3cab  /* 15.3 */
> include/linux/pci_ids.h:#define PCI_DEVICE_ID_INTEL_SBRIDGE_IMC_TAD2    0x3cac  /* 15.4 */
> include/linux/pci_ids.h:#define PCI_DEVICE_ID_INTEL_SBRIDGE_IMC_TAD3    0x3cad  /* 15.5 */
> 
> There should be two of each (one on bus 7f, the other on bus ff).

It is getting all 4 TAD devices (Broadwell). This is what I'm getting
(from the PCI IDs that it is supposed to be on Broadwell):

00:05.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Map/VTd_Misc/System Management [8086:6f28] (rev 01)
7f:0f.4 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent [8086:6ffc] (rev 01)
7f:0f.5 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent [8086:6ffd] (rev 01)
7f:12.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Home Agent 0 [8086:6fa0] (rev 01)
7f:13.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Target Address/Thermal/RAS [8086:6fa8] (rev 01)
7f:13.1 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Target Address/Thermal/RAS [8086:6f71] (rev 01)
7f:13.2 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder [8086:6faa] (rev 01)
7f:13.3 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder [8086:6fab] (rev 01)
7f:13.4 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder [8086:6fac] (rev 01)
7f:13.5 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder [8086:6fad] (rev 01)
7f:13.7 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Global Broadcast [8086:6faf] (rev 01)
7f:16.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Target Address/Thermal/RAS [8086:6f68] (rev 01)
80:05.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Map/VTd_Misc/System Management [8086:6f28] (rev 01)
ff:0f.4 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent [8086:6ffc] (rev 01)
ff:0f.5 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent [8086:6ffd] (rev 01)
ff:12.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Home Agent 0 [8086:6fa0] (rev 01)
ff:13.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Target Address/Thermal/RAS [8086:6fa8] (rev 01)
ff:13.1 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Target Address/Thermal/RAS [8086:6f71] (rev 01)
ff:13.2 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder [8086:6faa] (rev 01)
ff:13.3 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder [8086:6fab] (rev 01)
ff:13.4 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder [8086:6fac] (rev 01)
ff:13.5 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder [8086:6fad] (rev 01)
ff:13.7 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Global Broadcast [8086:6faf] (rev 01)
ff:16.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Target Address/Thermal/RAS [8086:6f68] (rev 01)


Thanks,
Mauro
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Borislav Petkov July 21, 2017, 1:34 p.m. UTC | #35
On Thu, Jul 20, 2017 at 07:50:03PM +0000, Kani, Toshimitsu wrote:
> GHES / firmware-first still requires OS recovery actions when an error
> cannot be corrected by the platform.  They are handled by ghes_proc(),
> and ghes_edac remains its error-reporting wrapper.

I mean all the recovery actions the firmware does because it gets to see
the error first. Otherwise, Firmware First is the the dumbest repeater
layer in the history of layers.

> Firmware has better knowledge about the platform and can provide better
> RAS when implemented properly.

s/when/if/
Mauro Carvalho Chehab July 21, 2017, 1:40 p.m. UTC | #36
Em Fri, 21 Jul 2017 15:34:41 +0200
Borislav Petkov <bp@alien8.de> escreveu:

> On Thu, Jul 20, 2017 at 07:50:03PM +0000, Kani, Toshimitsu wrote:
> > GHES / firmware-first still requires OS recovery actions when an error
> > cannot be corrected by the platform.  They are handled by ghes_proc(),
> > and ghes_edac remains its error-reporting wrapper. 

What happens when the error can be corrected? Does it still report it to
userspace, or just silently hide the error?

If I remember well about a past discussion with some vendor, I was told
that the firmware can hide some errors from being reported. Is it
still the case?


Thanks,
Mauro
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Borislav Petkov July 21, 2017, 1:47 p.m. UTC | #37
On Fri, Jul 21, 2017 at 10:40:01AM -0300, Mauro Carvalho Chehab wrote:
> What happens when the error can be corrected? Does it still report it to
> userspace, or just silently hide the error?
> 
> If I remember well about a past discussion with some vendor, I was told
> that the firmware can hide some errors from being reported. Is it
> still the case?

I've heard the same thing but I have no idea what they're actually
doing. But it would make sense because the intention is not to worry
users unnecessarily if it can hide the error and if there are no adverse
consequences from it.
Kani, Toshi July 21, 2017, 3:08 p.m. UTC | #38
On Fri, 2017-07-21 at 15:47 +0200, Borislav Petkov wrote:
> On Fri, Jul 21, 2017 at 10:40:01AM -0300, Mauro Carvalho Chehab

> wrote:

> > What happens when the error can be corrected? Does it still report

> > it to userspace, or just silently hide the error?

> > 

> > If I remember well about a past discussion with some vendor, I was

> > told that the firmware can hide some errors from being reported. Is

> > it still the case?

> 

> I've heard the same thing but I have no idea what they're actually

> doing. But it would make sense because the intention is not to worry

> users unnecessarily if it can hide the error and if there are no

> adverse consequences from it.


Yes, that is correct.  Corrected errors are reported to the OS when
they exceeded the platform's threshold.

Thanks,
-Toshi
Borislav Petkov July 21, 2017, 3:13 p.m. UTC | #39
On Fri, Jul 21, 2017 at 03:08:41PM +0000, Kani, Toshimitsu wrote:
> Yes, that is correct.  Corrected errors are reported to the OS when
> they exceeded the platform's threshold.

Are those thresholds user-configurable?

If not, what are you telling users who want to see *every* corrected
error for measuring DIMM wear and so on...?
Kani, Toshi July 21, 2017, 3:34 p.m. UTC | #40
T24gRnJpLCAyMDE3LTA3LTIxIGF0IDE3OjEzICswMjAwLCBCb3Jpc2xhdiBQZXRrb3Ygd3JvdGU6
DQo+IE9uIEZyaSwgSnVsIDIxLCAyMDE3IGF0IDAzOjA4OjQxUE0gKzAwMDAsIEthbmksIFRvc2hp
bWl0c3Ugd3JvdGU6DQo+ID4gWWVzLCB0aGF0IGlzIGNvcnJlY3QuwqDCoENvcnJlY3RlZCBlcnJv
cnMgYXJlIHJlcG9ydGVkIHRvIHRoZSBPUyB3aGVuDQo+ID4gdGhleSBleGNlZWRlZCB0aGUgcGxh
dGZvcm0ncyB0aHJlc2hvbGQuDQo+IA0KPiBBcmUgdGhvc2UgdGhyZXNob2xkcyB1c2VyLWNvbmZp
Z3VyYWJsZT8NCg0KSSBzdXBwb3NlIGl0J2QgZGVwZW5kIG9uIHZlbmRvcnMsIGJ1dCBJIGRvIG5v
dCB0aGluayB1c2VycyBjYW4gZG8gaXQNCnByb3Blcmx5IHVubGVzcyB0aGV5IGhhdmUgZGVwdGgg
a25vd2xlZGdlIGFib3V0IHRoZSBoYXJkd2FyZS4NCg0KPiBJZiBub3QsIHdoYXQgYXJlIHlvdSB0
ZWxsaW5nIHVzZXJzIHdobyB3YW50IHRvIHNlZSAqZXZlcnkqIGNvcnJlY3RlZA0KPiBlcnJvciBm
b3IgbWVhc3VyaW5nIERJTU0gd2VhciBhbmQgc28gb24uLi4/DQoNCkNvcnJlY3RlZCBlcnJvcnMg
YXJlIG5vcm1hbCBhbmQgZXhwZWN0ZWQgdG8gb2NjdXIgb24gaGVhbHRoeSBoYXJkd2FyZS4gDQpU
aGV5IGRvIG5vdCBuZWVkIHVzZXIncyBhdHRlbnRpb24gdW50aWwgdGhleSByZXBlYXRlZGx5IG9j
Y3VycmVkIGF0IGENCnNhbWUgcGxhY2UuDQoNClRoYW5rcywNCi1Ub3NoaQ0K
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Mauro Carvalho Chehab July 21, 2017, 3:44 p.m. UTC | #41
Em Fri, 21 Jul 2017 15:34:50 +0000
"Kani, Toshimitsu" <toshi.kani@hpe.com> escreveu:

> On Fri, 2017-07-21 at 17:13 +0200, Borislav Petkov wrote:
> > On Fri, Jul 21, 2017 at 03:08:41PM +0000, Kani, Toshimitsu wrote:  
> > > Yes, that is correct.  Corrected errors are reported to the OS when
> > > they exceeded the platform's threshold.  
> > 
> > Are those thresholds user-configurable?  
> 
> I suppose it'd depend on vendors, but I do not think users can do it
> properly unless they have depth knowledge about the hardware.
> 
> > If not, what are you telling users who want to see *every* corrected
> > error for measuring DIMM wear and so on...?  
> 
> Corrected errors are normal and expected to occur on healthy hardware. 
> They do not need user's attention until they repeatedly occurred at a
> same place.

Yes, they're expected to happen. Still, some sys admins have their own
measurements about what's "normal" for their scenario, and want
to monitor every single corrected error, running their own
algorithm to warn if the number of corrected errors is above their
"normal" rate.

Thanks,
Mauro
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Borislav Petkov July 21, 2017, 3:53 p.m. UTC | #42
On Fri, Jul 21, 2017 at 03:34:50PM +0000, Kani, Toshimitsu wrote:
> I suppose it'd depend on vendors, but I do not think users can do it
> properly unless they have depth knowledge about the hardware.

I'm talking about a menu in the BIOS where you can set the thresholding
levels on the system. Does your BIOS have that?

> Corrected errors are normal and expected to occur on healthy hardware. 
> They do not need user's attention until they repeatedly occurred at a
> same place.

Apparently, you haven't been on enough maintanance calls, trying to calm
down the customer about the hardware error he sees in his logs...
Kani, Toshi July 21, 2017, 4:32 p.m. UTC | #43
On Fri, 2017-07-21 at 17:53 +0200, Borislav Petkov wrote:
> On Fri, Jul 21, 2017 at 03:34:50PM +0000, Kani, Toshimitsu wrote:

> > I suppose it'd depend on vendors, but I do not think users can do

> > it properly unless they have depth knowledge about the hardware.

> 

> I'm talking about a menu in the BIOS where you can set the

> thresholding levels on the system. Does your BIOS have that?


No, we don't offer such settings.

> > Corrected errors are normal and expected to occur on healthy

> > hardware. They do not need user's attention until they repeatedly

> > occurred at a same place.

> 

> Apparently, you haven't been on enough maintanance calls, trying to

> calm down the customer about the hardware error he sees in his

> logs...


Actually, that's why.  Reporting all corrected errors make users
worried, call support, and asking to replace healthy hardware...

Thanks,
-Toshi
Kani, Toshi July 21, 2017, 4:40 p.m. UTC | #44
On Fri, 2017-07-21 at 12:44 -0300, Mauro Carvalho Chehab wrote:
> Em Fri, 21 Jul 2017 15:34:50 +0000

> "Kani, Toshimitsu" <toshi.kani@hpe.com> escreveu:

> 

> > On Fri, 2017-07-21 at 17:13 +0200, Borislav Petkov wrote:

> > > On Fri, Jul 21, 2017 at 03:08:41PM +0000, Kani, Toshimitsu

> > > wrote:  

> > > > Yes, that is correct.  Corrected errors are reported to the OS

> > > > when they exceeded the platform's threshold.  

> > > 

> > > Are those thresholds user-configurable?  

> > 

> > I suppose it'd depend on vendors, but I do not think users can do

> > it properly unless they have depth knowledge about the hardware.

> > 

> > > If not, what are you telling users who want to see *every*

> > > corrected error for measuring DIMM wear and so on...?  

> > 

> > Corrected errors are normal and expected to occur on healthy

> > hardware.  They do not need user's attention until they repeatedly

> > occurred at a same place.

> 

> Yes, they're expected to happen. Still, some sys admins have their

> own measurements about what's "normal" for their scenario, and want

> to monitor every single corrected error, running their own

> algorithm to warn if the number of corrected errors is above their

> "normal" rate.


I suppose these admins had to do it because their platforms reported
all corrected errors.  It addresses such administrators' burden.

Thanks,
-Toshi
Tony Luck July 21, 2017, 4:53 p.m. UTC | #45
Hmmm so the BIOS isn't hiding any devices.

Can you read the MTR registers from each of those target address decoders?

for i in a b d c
do
	for j in 0 4 8
	do
		setpci -d 8086:6fa$i 0x8$j.L
	done
done

bit 14 is the IS_DIMM_PRESENT one.  So you should see values like 001c5050 for populated slots. I see 000f000c for empty slots.

-Tony

[If you send me "lspci -xxxx" output I can check why the driver isn't reporting the 4th channel]
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Mauro Carvalho Chehab July 21, 2017, 5:01 p.m. UTC | #46
Em Fri, 21 Jul 2017 16:40:20 +0000
"Kani, Toshimitsu" <toshi.kani@hpe.com> escreveu:

> On Fri, 2017-07-21 at 12:44 -0300, Mauro Carvalho Chehab wrote:
> > Em Fri, 21 Jul 2017 15:34:50 +0000
> > "Kani, Toshimitsu" <toshi.kani@hpe.com> escreveu:
> >   
> > > On Fri, 2017-07-21 at 17:13 +0200, Borislav Petkov wrote:  
> > > > On Fri, Jul 21, 2017 at 03:08:41PM +0000, Kani, Toshimitsu
> > > > wrote:    
> > > > > Yes, that is correct.  Corrected errors are reported to the OS
> > > > > when they exceeded the platform's threshold.    
> > > > 
> > > > Are those thresholds user-configurable?    
> > > 
> > > I suppose it'd depend on vendors, but I do not think users can do
> > > it properly unless they have depth knowledge about the hardware.
> > >   
> > > > If not, what are you telling users who want to see *every*
> > > > corrected error for measuring DIMM wear and so on...?    
> > > 
> > > Corrected errors are normal and expected to occur on healthy
> > > hardware.  They do not need user's attention until they repeatedly
> > > occurred at a same place.  
> > 
> > Yes, they're expected to happen. Still, some sys admins have their
> > own measurements about what's "normal" for their scenario, and want
> > to monitor every single corrected error, running their own
> > algorithm to warn if the number of corrected errors is above their
> > "normal" rate.  
> 
> I suppose these admins had to do it because their platforms reported
> all corrected errors.  It addresses such administrators' burden.

I see the value of having a threshold in BIOS, provided that it is
well documented, and whose value can be adjusted, if needed.

One of the things I wanted to implement in ras-daemon were an
algorithm that would be doing such threshold in software.
The problem is that it would require field experience. So,
I talked with a few vendors, to see if they could help doing
it, but, on that time, none rised their hands :-)

The thing with a BIOS threshold is that the user has no way to
audit the algorithm. So, when BIOS start reporting such errors,
it may be already too late: the systems may be in the verge of 
losing data (or some data was already lost).

That's critical on cluster systems with thousands of machines:
while the impact of disabling a cluster node to do some maintainance
is marginal, the impact of an uncorrected error on a single
machine may compromise weeks of expensive processing.

That's why some users prefer to monitor every single corrected
error, and compare with the probability distribution they
know that the risk of uncorrected errors is acceptable.

Thanks,
Mauro
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Kani, Toshi July 21, 2017, 5:21 p.m. UTC | #47
T24gRnJpLCAyMDE3LTA3LTIxIGF0IDE0OjAxIC0wMzAwLCBNYXVybyBDYXJ2YWxobyBDaGVoYWIg
d3JvdGU6DQo+IEVtIEZyaSwgMjEgSnVsIDIwMTcgMTY6NDA6MjAgKzAwMDANCj4gIkthbmksIFRv
c2hpbWl0c3UiIDx0b3NoaS5rYW5pQGhwZS5jb20+IGVzY3JldmV1Og0KPiANCj4gPiBPbiBGcmks
IDIwMTctMDctMjEgYXQgMTI6NDQgLTAzMDAsIE1hdXJvIENhcnZhbGhvIENoZWhhYiB3cm90ZToN
Cj4gPiA+IEVtIEZyaSwgMjEgSnVsIDIwMTcgMTU6MzQ6NTAgKzAwMDANCj4gPiA+ICJLYW5pLCBU
b3NoaW1pdHN1IiA8dG9zaGkua2FuaUBocGUuY29tPiBlc2NyZXZldToNCj4gPiA+IMKgwqANCj4g
PiA+ID4gT24gRnJpLCAyMDE3LTA3LTIxIGF0IDE3OjEzICswMjAwLCBCb3Jpc2xhdiBQZXRrb3Yg
d3JvdGU6wqDCoA0KPiA+ID4gPiA+IE9uIEZyaSwgSnVsIDIxLCAyMDE3IGF0IDAzOjA4OjQxUE0g
KzAwMDAsIEthbmksIFRvc2hpbWl0c3UNCj4gPiA+ID4gPiB3cm90ZTrCoMKgwqDCoA0KPiA+ID4g
PiA+ID4gWWVzLCB0aGF0IGlzIGNvcnJlY3QuwqDCoENvcnJlY3RlZCBlcnJvcnMgYXJlIHJlcG9y
dGVkIHRvIHRoZQ0KPiA+ID4gPiA+ID4gT1Mgd2hlbiB0aGV5IGV4Y2VlZGVkIHRoZSBwbGF0Zm9y
bSdzIHRocmVzaG9sZC7CoMKgwqDCoA0KPiA+ID4gPiA+IA0KPiA+ID4gPiA+IEFyZSB0aG9zZSB0
aHJlc2hvbGRzIHVzZXItY29uZmlndXJhYmxlP8KgwqDCoMKgDQo+ID4gPiA+IA0KPiA+ID4gPiBJ
IHN1cHBvc2UgaXQnZCBkZXBlbmQgb24gdmVuZG9ycywgYnV0IEkgZG8gbm90IHRoaW5rIHVzZXJz
IGNhbg0KPiA+ID4gPiBkbyBpdCBwcm9wZXJseSB1bmxlc3MgdGhleSBoYXZlIGRlcHRoIGtub3ds
ZWRnZSBhYm91dCB0aGUNCj4gPiA+ID4gaGFyZHdhcmUuDQo+ID4gPiA+IMKgwqANCj4gPiA+ID4g
PiBJZiBub3QsIHdoYXQgYXJlIHlvdSB0ZWxsaW5nIHVzZXJzIHdobyB3YW50IHRvIHNlZSAqZXZl
cnkqDQo+ID4gPiA+ID4gY29ycmVjdGVkIGVycm9yIGZvciBtZWFzdXJpbmcgRElNTSB3ZWFyIGFu
ZCBzbyBvbi4uLj/CoMKgwqDCoA0KPiA+ID4gPiANCj4gPiA+ID4gQ29ycmVjdGVkIGVycm9ycyBh
cmUgbm9ybWFsIGFuZCBleHBlY3RlZCB0byBvY2N1ciBvbiBoZWFsdGh5DQo+ID4gPiA+IGhhcmR3
YXJlLsKgwqBUaGV5IGRvIG5vdCBuZWVkIHVzZXIncyBhdHRlbnRpb24gdW50aWwgdGhleQ0KPiA+
ID4gPiByZXBlYXRlZGx5IG9jY3VycmVkIGF0IGEgc2FtZSBwbGFjZS7CoMKgDQo+ID4gPiANCj4g
PiA+IFllcywgdGhleSdyZSBleHBlY3RlZCB0byBoYXBwZW4uIFN0aWxsLCBzb21lIHN5cyBhZG1p
bnMgaGF2ZQ0KPiA+ID4gdGhlaXIgb3duIG1lYXN1cmVtZW50cyBhYm91dCB3aGF0J3MgIm5vcm1h
bCIgZm9yIHRoZWlyIHNjZW5hcmlvLA0KPiA+ID4gYW5kIHdhbnQgdG8gbW9uaXRvciBldmVyeSBz
aW5nbGUgY29ycmVjdGVkIGVycm9yLCBydW5uaW5nIHRoZWlyDQo+ID4gPiBvd24gYWxnb3JpdGht
IHRvIHdhcm4gaWYgdGhlIG51bWJlciBvZiBjb3JyZWN0ZWQgZXJyb3JzIGlzIGFib3ZlDQo+ID4g
PiB0aGVpciAibm9ybWFsIiByYXRlLsKgwqANCj4gPiANCj4gPiBJIHN1cHBvc2UgdGhlc2UgYWRt
aW5zIGhhZCB0byBkbyBpdCBiZWNhdXNlIHRoZWlyIHBsYXRmb3Jtcw0KPiA+IHJlcG9ydGVkIGFs
bCBjb3JyZWN0ZWQgZXJyb3JzLsKgwqBJdCBhZGRyZXNzZXMgc3VjaCBhZG1pbmlzdHJhdG9ycycN
Cj4gPiBidXJkZW4uDQo+IA0KPiBJIHNlZSB0aGUgdmFsdWUgb2YgaGF2aW5nIGEgdGhyZXNob2xk
IGluIEJJT1MsIHByb3ZpZGVkIHRoYXQgaXQgaXMNCj4gd2VsbCBkb2N1bWVudGVkLCBhbmQgd2hv
c2UgdmFsdWUgY2FuIGJlIGFkanVzdGVkLCBpZiBuZWVkZWQuDQo+IA0KPiBPbmUgb2YgdGhlIHRo
aW5ncyBJIHdhbnRlZCB0byBpbXBsZW1lbnQgaW4gcmFzLWRhZW1vbiB3ZXJlIGFuDQo+IGFsZ29y
aXRobSB0aGF0IHdvdWxkIGJlIGRvaW5nIHN1Y2ggdGhyZXNob2xkIGluIHNvZnR3YXJlLg0KPiBU
aGUgcHJvYmxlbSBpcyB0aGF0IGl0IHdvdWxkIHJlcXVpcmUgZmllbGQgZXhwZXJpZW5jZS4gU28s
DQo+IEkgdGFsa2VkIHdpdGggYSBmZXcgdmVuZG9ycywgdG8gc2VlIGlmIHRoZXkgY291bGQgaGVs
cCBkb2luZw0KPiBpdCwgYnV0LCBvbiB0aGF0IHRpbWUsIG5vbmUgcmlzZWQgdGhlaXIgaGFuZHMg
Oi0pDQoNCkkgdGhpbmsgaXQnZCBiZSB2ZXJ5IGhhcmQgdG8ga2VlcCBpdCB1cCB0byBkYXRlLg0K
DQo+IFRoZSB0aGluZyB3aXRoIGEgQklPUyB0aHJlc2hvbGQgaXMgdGhhdCB0aGUgdXNlciBoYXMg
bm8gd2F5IHRvDQo+IGF1ZGl0IHRoZSBhbGdvcml0aG0uIFNvLCB3aGVuIEJJT1Mgc3RhcnQgcmVw
b3J0aW5nIHN1Y2ggZXJyb3JzLA0KPiBpdCBtYXkgYmUgYWxyZWFkeSB0b28gbGF0ZTogdGhlIHN5
c3RlbXMgbWF5IGJlIGluIHRoZSB2ZXJnZSBvZsKgDQo+IGxvc2luZyBkYXRhIChvciBzb21lIGRh
dGEgd2FzIGFscmVhZHkgbG9zdCkuDQo+IA0KPiBUaGF0J3MgY3JpdGljYWwgb24gY2x1c3RlciBz
eXN0ZW1zIHdpdGggdGhvdXNhbmRzIG9mIG1hY2hpbmVzOg0KPiB3aGlsZSB0aGUgaW1wYWN0IG9m
IGRpc2FibGluZyBhIGNsdXN0ZXIgbm9kZSB0byBkbyBzb21lIG1haW50YWluYW5jZQ0KPiBpcyBt
YXJnaW5hbCwgdGhlIGltcGFjdCBvZiBhbiB1bmNvcnJlY3RlZCBlcnJvciBvbiBhIHNpbmdsZQ0K
PiBtYWNoaW5lIG1heSBjb21wcm9taXNlIHdlZWtzIG9mIGV4cGVuc2l2ZSBwcm9jZXNzaW5nLg0K
PiANCj4gVGhhdCdzIHdoeSBzb21lIHVzZXJzIHByZWZlciB0byBtb25pdG9yIGV2ZXJ5IHNpbmds
ZSBjb3JyZWN0ZWQNCj4gZXJyb3IsIGFuZCBjb21wYXJlIHdpdGggdGhlIHByb2JhYmlsaXR5IGRp
c3RyaWJ1dGlvbiB0aGV5DQo+IGtub3cgdGhhdCB0aGUgcmlzayBvZiB1bmNvcnJlY3RlZCBlcnJv
cnMgaXMgYWNjZXB0YWJsZS4NCg0KUmlnaHQsIEkgZG8gbm90IHRoaW5rIGFsbCBwbGF0Zm9ybXMg
bmVlZCB0byBiZSBmaXJtd2FyZS1maXJzdC4gIEkgZG8NCm5vdCB3YW50IHRvIHRhbGsgbGlrZSBh
IHNhbGUncyBwZXJzb24sIGJ1dCB3ZSBhbHNvIG9mZmVyIGxvd2VyLWNvc3QNCnBsYXRmb3JtcyB0
aGF0IGRvIG5vdCBjb21lIHdpdGggYnVpbHQtaW4gUkFTLiAgVXNlcnMgY2FuIGNob29zZSBhIHJp
Z2h0DQptb2RlbCBmb3IgdGhlaXIgbmVlZHMuDQoNClRoYW5rcywNCi1Ub3NoaQ0K
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Borislav Petkov July 21, 2017, 5:23 p.m. UTC | #48
On Fri, Jul 21, 2017 at 02:01:31PM -0300, Mauro Carvalho Chehab wrote:
> I see the value of having a threshold in BIOS, provided that it is
> well documented, and whose value can be adjusted, if needed.
> 
> One of the things I wanted to implement in ras-daemon were an
> algorithm that would be doing such threshold in software.

We have that now in the kernel: drivers/ras/cec.c

We did it exactly for that purpose - not upsetting users unnecessarily.

> The thing with a BIOS threshold is that the user has no way to
> audit the algorithm. So, when BIOS start reporting such errors,
> it may be already too late: the systems may be in the verge of 
> losing data (or some data was already lost).

Not only that: thresholds depend on the DIMM types which means, BIOS
must know what DIMM types are in there which I doubt. So exposing that
to configuration instead of "deciding" for people would be better.

> That's critical on cluster systems with thousands of machines:
> while the impact of disabling a cluster node to do some maintainance
> is marginal, the impact of an uncorrected error on a single
> machine may compromise weeks of expensive processing.
> 
> That's why some users prefer to monitor every single corrected
> error, and compare with the probability distribution they
> know that the risk of uncorrected errors is acceptable.

Yap, you need to have stuff like that configurable - BIOS can't predict
all possible use cases.
Kani, Toshi July 21, 2017, 6:38 p.m. UTC | #49
On Fri, 2017-07-21 at 19:23 +0200, Borislav Petkov wrote:
 :
> Not only that: thresholds depend on the DIMM types which means,

BIOS
> must know what DIMM types are in there which I doubt. 


BIOS knows DIMM model from the SPD data.

> So exposing that to configuration instead of "deciding" for people

> would be better.


Enterprise platforms have very different model (I do not say it's
better for everyone from the cost perspective).  Typically, such
platform vendors work with DIMM vendors directly to come with their
supported DIMMs with own part numbers, which are certified for the
platforms with extensive validation testings.

Thanks,
-Toshi
Borislav Petkov July 22, 2017, 6:28 a.m. UTC | #50
On Fri, Jul 21, 2017 at 06:38:52PM +0000, Kani, Toshimitsu wrote:
> Enterprise platforms have very different model (I do not say it's
> better for everyone from the cost perspective).  Typically, such

But you do tell your customers that the error counts they see are not
really what *actually* happens, right?
Kani, Toshi July 24, 2017, 2:49 p.m. UTC | #51
T24gU2F0LCAyMDE3LTA3LTIyIGF0IDA4OjI4ICswMjAwLCBCb3Jpc2xhdiBQZXRrb3Ygd3JvdGU6
DQo+IE9uIEZyaSwgSnVsIDIxLCAyMDE3IGF0IDA2OjM4OjUyUE0gKzAwMDAsIEthbmksIFRvc2hp
bWl0c3Ugd3JvdGU6DQo+ID4gRW50ZXJwcmlzZSBwbGF0Zm9ybXMgaGF2ZSB2ZXJ5IGRpZmZlcmVu
dCBtb2RlbCAoSSBkbyBub3Qgc2F5IGl0J3MNCj4gPiBiZXR0ZXIgZm9yIGV2ZXJ5b25lIGZyb20g
dGhlIGNvc3QgcGVyc3BlY3RpdmUpLsKgwqBUeXBpY2FsbHksIHN1Y2gNCj4gDQo+IEJ1dCB5b3Ug
ZG8gdGVsbCB5b3VyIGN1c3RvbWVycyB0aGF0IHRoZSBlcnJvciBjb3VudHMgdGhleSBzZWUgYXJl
IG5vdA0KPiByZWFsbHkgd2hhdCAqYWN0dWFsbHkqIGhhcHBlbnMsIHJpZ2h0Pw0KDQpXZSBkbyBu
b3QgdGVsbCB0aGUgZXJyb3IgY291bnRzIHRvIGN1c3RvbWVycy4gIFdlIHRlbGwgY3VzdG9tZXJz
IHdoZW4NCnRoZXkgbmVlZCBhdHRlbnRpb24gYW5kIGhhdmUgYWN0aW9uYWJsZSBpdGVtcywgYW5k
IHdlIHByb3ZpZGUgc3VwcG9ydA0KZm9yIHRoYXQuICBTdXBwb3J0IGdldHMgYWxsIGluZm8gbmVj
ZXNzYXJ5Lg0KDQpUaGVyZSBhcmUgbXVsdGlwbGUgbW9kZWxzIGZvciBtdWx0aXBsZSB0eXBlcyBv
ZiBjdXN0b21lcnMuICBJIGFtIG5vdA0Kc2F5aW5nIG9uZSBtb2RlbCBpcyBiZXR0ZXIgdGhhbiB0
aGUgb3RoZXIuDQoNClRoYW5rcywNCi1Ub3NoaQ0K
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Borislav Petkov July 24, 2017, 3:04 p.m. UTC | #52
On Mon, Jul 24, 2017 at 02:49:30PM +0000, Kani, Toshimitsu wrote:
> We do not tell the error counts to customers.

Please read what I said: do you tell your customers that the error
counts they're seeing (or are *not* seeing) is bogus because the BIOS is
hiding them? Not the *actual* numbers!

> We tell customers when they need attention and have actionable items,
> and we provide support for that. Support gets all info necessary.

Ok, good to know. I'll make sure to bounce such issues to you guys in
the future.
Kani, Toshi July 24, 2017, 3:25 p.m. UTC | #53
On Mon, 2017-07-24 at 17:04 +0200, Borislav Petkov wrote:
> On Mon, Jul 24, 2017 at 02:49:30PM +0000, Kani, Toshimitsu wrote:

> > We do not tell the error counts to customers.

> 

> Please read what I said: do you tell your customers that the error

> counts they're seeing (or are *not* seeing) is bogus because the BIOS

> is hiding them? Not the *actual* numbers!


Customers do not see error counts.  I do not think it's bogus.

This model is basically the same as your car.  You do not see error
counts or periodical normal errors from all kinds of controllers in the
car while you are driving.  You get an attention lamp lit when you need
to bring it to a car dealer.

> > We tell customers when they need attention and have actionable

> > items, and we provide support for that. Support gets all info

> > necessary.

> 

> Ok, good to know. I'll make sure to bounce such issues to you guys in

> the future.


We've been providing this model for many years now.  I am just trying
to enable OS error reporting with ghes_edac.

Thanks,
-Toshi
Borislav Petkov July 24, 2017, 3:37 p.m. UTC | #54
On Mon, Jul 24, 2017 at 03:25:34PM +0000, Kani, Toshimitsu wrote:
> Customers do not see error counts.  I do not think it's bogus.

Not showing the real error error counts but something contrived is the
definition of bogus numbers. But you're not showing anything - only when
some thresholds are being hit.

> This model is basically the same as your car.  You do not see error

Oh jeez, we're talking about cars now.

> We've been providing this model for many years now.

Dude, relax, I'm only trying to point out to you that there are
customers who want to see *every* error and thus track how their
hardware behaves. And that for those customers it is probably worth
considering exposing that info and providing a switch to disable that
dumbing of the RAS functionality in the BIOS so that people can decide
for themselves. That's all.

I'm not questioning your model - I'm just saying that it could be
improved for certain customers. Do me a favor and this time *actually*
*read* my reply.

> I am just trying to enable OS error reporting with ghes_edac.

I know, you don't have to state the obvious constantly.
Kani, Toshi July 24, 2017, 3:56 p.m. UTC | #55
On Mon, 2017-07-24 at 17:37 +0200, Borislav Petkov wrote:
> On Mon, Jul 24, 2017 at 03:25:34PM +0000, Kani, Toshimitsu wrote:

 :
> 

> > We've been providing this model for many years now.

> 

> Dude, relax, I'm only trying to point out to you that there are

> customers who want to see *every* error and thus track how their

> hardware behaves. And that for those customers it is probably worth

> considering exposing that info and providing a switch to disable that

> dumbing of the RAS functionality in the BIOS so that people can

> decide for themselves. That's all.


Yes, Mauro has already pointed this out.  As I replied to him, we do
have a separate series of platforms that do not have built-in RAS, and
report all errors.  Such customers can simply choose them.  They do not
need to pay for built-in RAS.

The model w/ built-in RAS provides warranty & full support.  As I said,
it's a different model. 

Thanks,
-Toshi
Mauro Carvalho Chehab July 24, 2017, 4:04 p.m. UTC | #56
Em Mon, 24 Jul 2017 17:37:16 +0200
Borislav Petkov <bp@alien8.de> escreveu:

> > Customers do not see error counts.  I do not think it's bogus.

> > I am just trying to enable OS error reporting with ghes_edac.  
> 
> I know, you don't have to state the obvious constantly.

The problem I see is that, currently, on users that have EDAC
already enabled, the users gets the errors directly from the
hardware.

If the Kernel force those users to use ghes_edac by default,
they they won't see the error counts anymore, but, instead,
hardware reports that the memories need to be replaced.

Well, if such users are handling thresholds themselves, they
won't see those errors anymore, as the errors will be masked.

That's a regression.

So, the right solution would be to keep hardware first, but
providing a modprobe parameter to let them switch to software
first.

Thanks,
Mauro
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Borislav Petkov July 24, 2017, 4:37 p.m. UTC | #57
On Mon, Jul 24, 2017 at 03:56:27PM +0000, Kani, Toshimitsu wrote:
> Yes, Mauro has already pointed this out.  As I replied to him, we do
> have a separate series of platforms that do not have built-in RAS, and

So this whitelist entry

+static struct acpi_oemlist oemlist[] = {
+       {"HPE   ", "Server  ", 0, ACPI_SIG_FADT, all_versions},
+       { } /* End */
+};

looks like it'll match every HP server platform not only the ones with
built-in RAS.
Borislav Petkov July 24, 2017, 4:44 p.m. UTC | #58
On Mon, Jul 24, 2017 at 01:04:13PM -0300, Mauro Carvalho Chehab wrote:
> If the Kernel force those users to use ghes_edac by default,
> they they won't see the error counts anymore, but, instead,
> hardware reports that the memories need to be replaced.

This is exactly why I'm trying to load ghes_edac only on those platforms
which would really want it.

> So, the right solution would be to keep hardware first, but
> providing a modprobe parameter to let them switch to software
> first.

That's exactly the issue: if we make it spec-conform and adhere to FF
setting, then it'll be clean. BUT(!), we will force ghes_edac on those
platforms which potentially are using the platform-specific drivers
until now. Not good.

If we do the whitelisting, then we're stuck with maintaining a yucky
whitelist and have to keep updating ghes_edac with it.

So we're basically between a rock and a hard place.

If I had to choose *right* *now*, I'd probably lean slightly towards the
whitelist as it won't break existing users.

A big grumpfy-grumbly hmmm. :-\
Kani, Toshi July 24, 2017, 5:44 p.m. UTC | #59
On Mon, 2017-07-24 at 18:37 +0200, Borislav Petkov wrote:
> On Mon, Jul 24, 2017 at 03:56:27PM +0000, Kani, Toshimitsu wrote:

> > Yes, Mauro has already pointed this out.  As I replied to him, we

> > do have a separate series of platforms that do not have built-in

> > RAS, and

> 

> So this whitelist entry

> 

> +static struct acpi_oemlist oemlist[] = {

> +       {"HPE   ", "Server  ", 0, ACPI_SIG_FADT, all_versions},

> +       { } /* End */

> +};

> 

> looks like it'll match every HP server platform not only the ones

> with built-in RAS.


I assumed our platforms w/o build-in RAS do not implement GHES, but I
will check for sure.  Also, all our previous/current platforms have
"HP".

Thanks,
-Toshi
Borislav Petkov July 24, 2017, 5:50 p.m. UTC | #60
On July 24, 2017 8:44:03 PM GMT+03:00, "Kani, Toshimitsu" <toshi.kani@hpe.com> wrote:
>I assumed our platforms w/o build-in RAS do not implement GHES,

If we make it a normal module, it will be decoupled from GHES and it will rely only on the whitelist to load.
Kani, Toshi July 24, 2017, 5:54 p.m. UTC | #61
On Mon, 2017-07-24 at 20:50 +0300, Boris Petkov wrote:
> On July 24, 2017 8:44:03 PM GMT+03:00, "Kani, Toshimitsu" <toshi.kani

> @hpe.com> wrote:

> > I assumed our platforms w/o build-in RAS do not implement GHES,

> 

> If we make it a normal module, it will be decoupled from GHES and it

> will rely only on the whitelist to load. 


Umm... I was under impression that we are adding the OSC bit check in
addition to the current GHES filtering.

Thanks,
-Toshi
Mauro Carvalho Chehab July 24, 2017, 5:56 p.m. UTC | #62
Em Mon, 24 Jul 2017 15:56:27 +0000
"Kani, Toshimitsu" <toshi.kani@hpe.com> escreveu:

> On Mon, 2017-07-24 at 17:37 +0200, Borislav Petkov wrote:
> > On Mon, Jul 24, 2017 at 03:25:34PM +0000, Kani, Toshimitsu wrote:  
>  :
> >   
> > > We've been providing this model for many years now.  
> > 
> > Dude, relax, I'm only trying to point out to you that there are
> > customers who want to see *every* error and thus track how their
> > hardware behaves. And that for those customers it is probably worth
> > considering exposing that info and providing a switch to disable that
> > dumbing of the RAS functionality in the BIOS so that people can
> > decide for themselves. That's all.  
> 
> Yes, Mauro has already pointed this out.  As I replied to him, we do
> have a separate series of platforms that do not have built-in RAS, and
> report all errors.  Such customers can simply choose them.  They do not
> need to pay for built-in RAS.

That's probably too late for me as I received a new HP machine
we bought just last week, but for the next time I would need to
get a new hardware, what would be the non-RAS equivalent to
a ML 350 G9 tower-mounted machine with two Xeon v4 CPUs and iLO?

Regards,
Mauro

> 
> The model w/ built-in RAS provides warranty & full support.  As I said,
> it's a different model. 
> 
> Thanks,
> -Toshi



Thanks,
Mauro
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Mauro Carvalho Chehab July 24, 2017, 6:10 p.m. UTC | #63
Em Mon, 24 Jul 2017 18:44:00 +0200
Borislav Petkov <bp@alien8.de> escreveu:

> On Mon, Jul 24, 2017 at 01:04:13PM -0300, Mauro Carvalho Chehab wrote:
> > If the Kernel force those users to use ghes_edac by default,
> > they they won't see the error counts anymore, but, instead,
> > hardware reports that the memories need to be replaced.  
> 
> This is exactly why I'm trying to load ghes_edac only on those platforms
> which would really want it.
> 
> > So, the right solution would be to keep hardware first, but
> > providing a modprobe parameter to let them switch to software
> > first.  
> 
> That's exactly the issue: if we make it spec-conform and adhere to FF
> setting, then it'll be clean. BUT(!), we will force ghes_edac on those
> platforms which potentially are using the platform-specific drivers
> until now. Not good.
> 
> If we do the whitelisting, then we're stuck with maintaining a yucky
> whitelist and have to keep updating ghes_edac with it.

Yeah, having a whitelist is a maintainership's burden, but, on
the other hand, I suspect that there aren't many systems that
implement FF, have a reliable BIOS mapping of MB's silkscreen
and doesn't filters out corrected errors using some sort of
undocumented mechanism.

So, I guess it is doable.

Another alternative, with, IMO, is better would be to add a parameter like:

	edac=FF - firmware first;
	edac=hw	 - hardware first;
	edac=auto - honors FF if set in BIOS. Otherwise, hardware first.

In order to avoid regressions, and to avoid the need of a whitelist,
I would keep "edac=hw" as default.

Thanks,
Mauro
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Kani, Toshi July 24, 2017, 6:12 p.m. UTC | #64
T24gTW9uLCAyMDE3LTA3LTI0IGF0IDE0OjU2IC0wMzAwLCBNYXVybyBDYXJ2YWxobyBDaGVoYWIg
d3JvdGU6DQo+IEVtIE1vbiwgMjQgSnVsIDIwMTcgMTU6NTY6MjcgKzAwMDANCiA6DQo+IFRoYXQn
cyBwcm9iYWJseSB0b28gbGF0ZSBmb3IgbWUgYXMgSSByZWNlaXZlZCBhIG5ldyBIUCBtYWNoaW5l
DQo+IHdlIGJvdWdodCBqdXN0IGxhc3Qgd2VlaywgYnV0IGZvciB0aGUgbmV4dCB0aW1lIEkgd291
bGQgbmVlZCB0bw0KPiBnZXQgYSBuZXcgaGFyZHdhcmUsIHdoYXQgd291bGQgYmUgdGhlIG5vbi1S
QVMgZXF1aXZhbGVudCB0bw0KPiBhIE1MIDM1MCBHOSB0b3dlci1tb3VudGVkIG1hY2hpbmUgd2l0
aCB0d28gWGVvbiB2NCBDUFVzIGFuZCBpTE8/DQoNClN1Y2ggc2VydmVycyBhcmUgY2FsbGVkICJI
UEUgQ2xvdWRsaW5lIi4gIEJ1dCBJIHRoaW5rIHRoZXkgYXJlIGFsbA0KcmFjay1tb3VudGVkLCBu
b3QgdG93ZXItbW91bnRlZCBtYWNoaW5lcy4gIEhQIEluYy4gKHdoaWNoIGlzIG5vdyBhDQpzZXBh
cmF0ZSBjb21wYW55IGZvciBjb25zdW1lci1vcmllbnRlZCBwcm9kdWN0cykgcHJvYmFibHkgaGFz
IHN1Y2gNCm1hY2hpbmUuDQoNClRoYW5rcywNCi1Ub3NoaQ0K
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Borislav Petkov July 24, 2017, 6:18 p.m. UTC | #65
On Mon, Jul 24, 2017 at 05:54:52PM +0000, Kani, Toshimitsu wrote:
> Umm... I was under impression that we are adding the OSC bit check in
> addition to the current GHES filtering.

Read the parallel subthread again.
Borislav Petkov July 24, 2017, 6:30 p.m. UTC | #66
(Sending to your other mail address because there's some temporary resolution
 issue:

msmtp: recipient address mchehab@s-opensource.com not accepted by the server
msmtp: server message: 451 4.3.0 <mchehab@s-opensource.com>: Temporary lookup failure
msmtp: could not send mail (account alien8.de from /home/boris/.msmtprc)

Maybe the problem is on my end.)

On Mon, Jul 24, 2017 at 03:10:13PM -0300, Mauro Carvalho Chehab wrote:
> Yeah, having a whitelist is a maintainership's burden, but, on
> the other hand, I suspect that there aren't many systems that
> implement FF, have a reliable BIOS mapping of MB's silkscreen
> and doesn't filters out corrected errors using some sort of
> undocumented mechanism.
> 
> So, I guess it is doable.

Right, let's hope.

> Another alternative, with, IMO, is better would be to add a parameter like:
> 
> 	edac=FF - firmware first;
> 	edac=hw	 - hardware first;
> 	edac=auto - honors FF if set in BIOS. Otherwise, hardware first.

Or maybe edac=try_FF or so. But yeah, I guess we'll need something to
tell the EDAC core to try FF first.

> In order to avoid regressions, and to avoid the need of a whitelist,
> I would keep "edac=hw" as default.

So I don't want to break existing users and thus make only explicitly
known platforms load ghes_edac. In the current case, the HPE machines.
All the rest will simply use the platform drivers and nothing will
change for them.

Later we'll probably need to revisit this decision but right now and
with all things considered, the whitelist seems - as ugly as it is - the
most workable solution for all the different use cases and machines...
Kani, Toshi July 25, 2017, 11 p.m. UTC | #67
T24gTW9uLCAyMDE3LTA3LTI0IGF0IDIwOjMwICswMjAwLCBCb3Jpc2xhdiBQZXRrb3Ygd3JvdGU6
DQogOg0KPiANCj4gU28gSSBkb24ndCB3YW50IHRvIGJyZWFrIGV4aXN0aW5nIHVzZXJzIGFuZCB0
aHVzIG1ha2Ugb25seSBleHBsaWNpdGx5DQo+IGtub3duIHBsYXRmb3JtcyBsb2FkIGdoZXNfZWRh
Yy4gSW4gdGhlIGN1cnJlbnQgY2FzZSwgdGhlIEhQRQ0KPiBtYWNoaW5lcy4gIEFsbCB0aGUgcmVz
dCB3aWxsIHNpbXBseSB1c2UgdGhlIHBsYXRmb3JtIGRyaXZlcnMgYW5kDQo+IG5vdGhpbmcgd2ls
bCBjaGFuZ2UgZm9yIHRoZW0uDQo+IA0KPiBMYXRlciB3ZSdsbCBwcm9iYWJseSBuZWVkIHRvIHJl
dmlzaXQgdGhpcyBkZWNpc2lvbiBidXQgcmlnaHQgbm93IGFuZA0KPiB3aXRoIGFsbCB0aGluZ3Mg
Y29uc2lkZXJlZCwgdGhlIHdoaXRlbGlzdCBzZWVtcyAtIGFzIHVnbHkgYXMgaXQgaXMgLQ0KPiB0
aGUgbW9zdCB3b3JrYWJsZSBzb2x1dGlvbiBmb3IgYWxsIHRoZSBkaWZmZXJlbnQgdXNlIGNhc2Vz
IGFuZA0KPiBtYWNoaW5lcy4uLg0KDQpBZ3JlZWQuICBJIHdpbGwgdmVyaWZ5IE9FTUlEIGluZm8g
b2Ygb3VyIG90aGVyIHBsYXRmb3JtcywgYW5kIGFkZCBBUEVJDQpPU0MgY2hlY2sgYmVmb3JlIGNh
bGxpbmcgZ2hlc19lZGFjX3JlZ2lzdGVyKCkuDQoNClRoYW5rcywNCi1Ub3NoaQ0K
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
diff mbox

Patch

diff --git a/drivers/acpi/apei/ghes.c b/drivers/acpi/apei/ghes.c
index d661d452b238..37cd698cacd2 100644
--- a/drivers/acpi/apei/ghes.c
+++ b/drivers/acpi/apei/ghes.c
@@ -140,6 +140,20 @@  static atomic_t ghes_estatus_cache_alloced;
 
 static int ghes_panic_timeout __read_mostly = 30;
 
+static ATOMIC_NOTIFIER_HEAD(ghes_edac_chain);
+
+void ghes_register_edac_chain(struct notifier_block *nb)
+{
+	atomic_notifier_chain_register(&ghes_edac_chain, nb);
+}
+EXPORT_SYMBOL_GPL(ghes_register_edac_chain);
+
+void ghes_unregister_edac_chain(struct notifier_block *nb)
+{
+	atomic_notifier_chain_unregister(&ghes_edac_chain, nb);
+}
+EXPORT_SYMBOL_GPL(ghes_unregister_edac_chain);
+
 static int ghes_ioremap_init(void)
 {
 	ghes_ioremap_area = __get_vm_area(PAGE_SIZE * GHES_IOREMAP_PAGES,
@@ -461,11 +475,11 @@  static void ghes_handle_memory_failure(struct acpi_hest_generic_data *gdata, int
 static void ghes_do_proc(struct ghes *ghes,
 			 const struct acpi_hest_generic_status *estatus)
 {
-	int sev, sec_sev;
 	struct acpi_hest_generic_data *gdata;
 	guid_t *sec_type;
 	guid_t *fru_id = &NULL_UUID_LE;
 	char *fru_text = "";
+	int sev, sec_sev;
 
 	sev = ghes_severity(estatus->error_severity);
 	apei_estatus_for_each_section(estatus, gdata) {
@@ -480,7 +494,8 @@  static void ghes_do_proc(struct ghes *ghes,
 		if (guid_equal(sec_type, &CPER_SEC_PLATFORM_MEM)) {
 			struct cper_sec_mem_err *mem_err = acpi_hest_get_payload(gdata);
 
-			ghes_edac_report_mem_error(ghes, sev, mem_err);
+
+			atomic_notifier_call_chain(&ghes_edac_chain, sev, &mem_err);
 
 			arch_apei_report_mem_error(sev, mem_err);
 			ghes_handle_memory_failure(gdata, sev);
@@ -1139,10 +1154,6 @@  static int ghes_probe(struct platform_device *ghes_dev)
 		goto err;
 	}
 
-	rc = ghes_edac_register(ghes, &ghes_dev->dev);
-	if (rc < 0)
-		goto err;
-
 	switch (generic->notify.type) {
 	case ACPI_HEST_NOTIFY_POLLED:
 		setup_deferrable_timer(&ghes->timer, ghes_poll_func,
@@ -1155,13 +1166,13 @@  static int ghes_probe(struct platform_device *ghes_dev)
 		if (rc) {
 			pr_err(GHES_PFX "Failed to map GSI to IRQ for generic hardware error source: %d\n",
 			       generic->header.source_id);
-			goto err_edac_unreg;
+			goto err;
 		}
 		rc = request_irq(ghes->irq, ghes_irq_func, 0, "GHES IRQ", ghes);
 		if (rc) {
 			pr_err(GHES_PFX "Failed to register IRQ for generic hardware error source: %d\n",
 			       generic->header.source_id);
-			goto err_edac_unreg;
+			goto err;
 		}
 		break;
 
@@ -1190,8 +1201,7 @@  static int ghes_probe(struct platform_device *ghes_dev)
 	ghes_proc(ghes);
 
 	return 0;
-err_edac_unreg:
-	ghes_edac_unregister(ghes);
+
 err:
 	if (ghes) {
 		ghes_fini(ghes);
@@ -1241,8 +1251,6 @@  static int ghes_remove(struct platform_device *ghes_dev)
 
 	ghes_fini(ghes);
 
-	ghes_edac_unregister(ghes);
-
 	kfree(ghes);
 
 	platform_set_drvdata(ghes_dev, NULL);
diff --git a/drivers/edac/Kconfig b/drivers/edac/Kconfig
index 96afb2aeed18..fdd8278ca89a 100644
--- a/drivers/edac/Kconfig
+++ b/drivers/edac/Kconfig
@@ -53,8 +53,8 @@  config EDAC_DECODE_MCE
 	  has been initialized.
 
 config EDAC_GHES
-	bool "Output ACPI APEI/GHES BIOS detected errors via EDAC"
-	depends on ACPI_APEI_GHES && (EDAC=y)
+	tristate "Output ACPI APEI/GHES BIOS detected errors via EDAC"
+	depends on ACPI_APEI_GHES
 	help
 	  Not all machines support hardware-driven error report. Some of those
 	  provide a BIOS-driven error report mechanism via ACPI, using the
diff --git a/drivers/edac/edac_mc.h b/drivers/edac/edac_mc.h
index 5357800e418d..6d46f30dc657 100644
--- a/drivers/edac/edac_mc.h
+++ b/drivers/edac/edac_mc.h
@@ -60,6 +60,9 @@ 
 #define edac_pci_printk(ctl, level, fmt, arg...) \
 	printk(level "EDAC PCI%d: " fmt, ctl->pci_idx, ##arg)
 
+#define edac_pr_err(fmt, arg...)	edac_printk(KERN_ERR, "", fmt, ##arg)
+#define edac_pr_info(fmt, arg...)	edac_printk(KERN_INFO, "", fmt, ##arg)
+
 /* prefixes for edac_printk() and edac_mc_printk() */
 #define EDAC_MC "MC"
 #define EDAC_PCI "PCI"
diff --git a/drivers/edac/ghes_edac.c b/drivers/edac/ghes_edac.c
index 4e61a6229dd2..20fafc55eb2d 100644
--- a/drivers/edac/ghes_edac.c
+++ b/drivers/edac/ghes_edac.c
@@ -5,6 +5,9 @@ 
  * License version 2.
  *
  * Copyright (c) 2013 by Mauro Carvalho Chehab
+ *	     (c) 2017 Borislav Petkov
+ *
+ * Borislav Petkov: turn it into a proper module.
  *
  * Red Hat Inc. http://www.redhat.com
  */
@@ -17,7 +20,14 @@ 
 #include "edac_module.h"
 #include <ras/ras_event.h>
 
-#define GHES_EDAC_REVISION " Ver: 1.0.0"
+#define GHES_EDAC_REVISION " Ver: 2.0.0"
+
+/*
+ * Hand it into EDAC's core so that we have a device to operate on.
+ */
+static struct device dummy_dev;
+
+struct ghes_edac_pvt *ghes_pvt;
 
 struct ghes_edac_pvt {
 	struct list_head list;
@@ -30,11 +40,6 @@  struct ghes_edac_pvt {
 	char msg[80];
 };
 
-static LIST_HEAD(ghes_reglist);
-static DEFINE_MUTEX(ghes_edac_lock);
-static int ghes_edac_mc_num;
-
-
 /* Memory Device - Type 17 of SMBIOS spec */
 struct memdev_dmi_entry {
 	u8 type;
@@ -165,24 +170,21 @@  static void ghes_edac_dmidecode(const struct dmi_header *dh, void *arg)
 	}
 }
 
-void ghes_edac_report_mem_error(struct ghes *ghes, int sev,
-				struct cper_sec_mem_err *mem_err)
+static int report_mem_error(struct notifier_block *nb, unsigned long sev, void *data)
 {
+	struct cper_sec_mem_err *mem_err = data;
 	enum hw_event_mc_err_type type;
 	struct edac_raw_error_desc *e;
 	struct mem_ctl_info *mci;
-	struct ghes_edac_pvt *pvt = NULL;
-	char *p;
+	struct ghes_edac_pvt *pvt = ghes_pvt;
 	u8 grain_bits;
+	char *p;
 
-	list_for_each_entry(pvt, &ghes_reglist, list) {
-		if (ghes == pvt->ghes)
-			break;
-	}
 	if (!pvt) {
-		pr_err("Internal error: Can't find EDAC structure\n");
-		return;
+		edac_pr_err("Internal error: Can't find EDAC structure\n");
+		return NOTIFY_DONE;
 	}
+
 	mci = pvt->mci;
 	e = &mci->error_desc;
 
@@ -402,23 +404,40 @@  void ghes_edac_report_mem_error(struct ghes *ghes, int sev,
 
 	/* Report the error via EDAC API */
 	edac_raw_mc_handle_error(type, mci, e);
+
+	return NOTIFY_DONE;
 }
-EXPORT_SYMBOL_GPL(ghes_edac_report_mem_error);
 
-int ghes_edac_register(struct ghes *ghes, struct device *dev)
+static struct notifier_block ghes_nb = {
+	.notifier_call  = report_mem_error,
+};
+
+static const char * const fake_msg =
+"This EDAC driver relies on BIOS to enumerate memory and get error reports.\n"
+"Unfortunately, not all BIOSes reflect the memory layout correctly.\n"
+"So, the end result of using this driver varies from vendor to vendor.\n"
+"If you find incorrect reports, please contact your hardware vendor\n"
+"to correct its BIOS.";
+
+static const char * const super_crap_msg =
+"This system has a very crappy BIOS: It doesn't even list the DIMMS.\n"
+"Its SMBIOS info is wrong. It is doubtful that the error report would\n"
+"work on such system. Use this driver with caution.";
+
+static int __init ghes_edac_register(void)
 {
+	struct ghes_edac_pvt *pvt = ghes_pvt;
 	bool fake = false;
 	int rc, num_dimm = 0;
 	struct mem_ctl_info *mci;
 	struct edac_mc_layer layers[1];
-	struct ghes_edac_pvt *pvt;
 	struct ghes_edac_dimm_fill dimm_fill;
 
 	/* Get the number of DIMMs */
 	dmi_walk(ghes_edac_count_dimms, &num_dimm);
 
 	/* Check if we've got a bogus BIOS */
-	if (num_dimm == 0) {
+	if (!num_dimm) {
 		fake = true;
 		num_dimm = 1;
 	}
@@ -431,21 +450,17 @@  int ghes_edac_register(struct ghes *ghes, struct device *dev)
 	 * We need to serialize edac_mc_alloc() and edac_mc_add_mc(),
 	 * to avoid duplicated memory controller numbers
 	 */
-	mutex_lock(&ghes_edac_lock);
-	mci = edac_mc_alloc(ghes_edac_mc_num, ARRAY_SIZE(layers), layers,
-			    sizeof(*pvt));
+	mci = edac_mc_alloc(1, ARRAY_SIZE(layers), layers, sizeof(*pvt));
 	if (!mci) {
-		pr_info("Can't allocate memory for EDAC data\n");
-		mutex_unlock(&ghes_edac_lock);
+		edac_pr_err("Can't allocate memory for EDAC data\n");
 		return -ENOMEM;
 	}
 
 	pvt = mci->pvt_info;
 	memset(pvt, 0, sizeof(*pvt));
-	list_add_tail(&pvt->list, &ghes_reglist);
-	pvt->ghes = ghes;
 	pvt->mci  = mci;
-	mci->pdev = dev;
+
+	mci->pdev = &dummy_dev;
 
 	mci->mtype_cap = MEM_FLAG_EMPTY;
 	mci->edac_ctl_cap = EDAC_FLAG_NONE;
@@ -455,21 +470,12 @@  int ghes_edac_register(struct ghes *ghes, struct device *dev)
 	mci->ctl_name = "ghes_edac";
 	mci->dev_name = "ghes";
 
-	if (!ghes_edac_mc_num) {
-		if (!fake) {
-			pr_info("This EDAC driver relies on BIOS to enumerate memory and get error reports.\n");
-			pr_info("Unfortunately, not all BIOSes reflect the memory layout correctly.\n");
-			pr_info("So, the end result of using this driver varies from vendor to vendor.\n");
-			pr_info("If you find incorrect reports, please contact your hardware vendor\n");
-			pr_info("to correct its BIOS.\n");
-			pr_info("This system has %d DIMM sockets.\n",
-				num_dimm);
-		} else {
-			pr_info("This system has a very crappy BIOS: It doesn't even list the DIMMS.\n");
-			pr_info("Its SMBIOS info is wrong. It is doubtful that the error report would\n");
-			pr_info("work on such system. Use this driver with caution\n");
-		}
-	}
+	if (!fake)
+		edac_pr_info("%s\n", fake_msg);
+	else
+		edac_pr_info("%s\n", super_crap_msg);
+
+	edac_pr_info("This system has %d DIMM sockets.\n", num_dimm);
 
 	if (!fake) {
 		/*
@@ -478,13 +484,11 @@  int ghes_edac_register(struct ghes *ghes, struct device *dev)
 		 * Keep it in blank for the other memory controllers, as
 		 * there's no reliable way to properly credit each DIMM to
 		 * the memory controller, as different BIOSes fill the
-		 * DMI bank location fields on different ways
+		 * DMI bank location fields in different ways.
 		 */
-		if (!ghes_edac_mc_num) {
-			dimm_fill.count = 0;
-			dimm_fill.mci = mci;
-			dmi_walk(ghes_edac_dmidecode, &dimm_fill);
-		}
+		dimm_fill.count = 0;
+		dimm_fill.mci = mci;
+		dmi_walk(ghes_edac_dmidecode, &dimm_fill);
 	} else {
 		struct dimm_info *dimm = EDAC_DIMM_PTR(mci->layers, mci->dimms,
 						       mci->n_layers, 0, 0, 0);
@@ -498,30 +502,31 @@  int ghes_edac_register(struct ghes *ghes, struct device *dev)
 
 	rc = edac_mc_add_mc(mci);
 	if (rc < 0) {
-		pr_info("Can't register at EDAC core\n");
+		edac_pr_err("Can't register with EDAC core\n");
 		edac_mc_free(mci);
-		mutex_unlock(&ghes_edac_lock);
 		return -ENODEV;
 	}
 
-	ghes_edac_mc_num++;
-	mutex_unlock(&ghes_edac_lock);
+	ghes_register_edac_chain(&ghes_nb);
+
 	return 0;
 }
-EXPORT_SYMBOL_GPL(ghes_edac_register);
+module_init(ghes_edac_register);
 
-void ghes_edac_unregister(struct ghes *ghes)
+static void __exit ghes_edac_unregister(void)
 {
 	struct mem_ctl_info *mci;
-	struct ghes_edac_pvt *pvt, *tmp;
-
-	list_for_each_entry_safe(pvt, tmp, &ghes_reglist, list) {
-		if (ghes == pvt->ghes) {
-			mci = pvt->mci;
-			edac_mc_del_mc(mci->pdev);
-			edac_mc_free(mci);
-			list_del(&pvt->list);
-		}
-	}
+
+	ghes_unregister_edac_chain(&ghes_nb);
+
+	mci = find_mci_by_dev(&dummy_dev);
+	WARN_ON(!mci);
+
+	edac_mc_del_mc(mci->pdev);
+	edac_mc_free(mci);
+
 }
-EXPORT_SYMBOL_GPL(ghes_edac_unregister);
+module_exit(ghes_edac_unregister);
+
+MODULE_LICENSE("GPL");
+MODULE_DESCRIPTION("GHES error decoding module - " GHES_EDAC_REVISION);
diff --git a/include/acpi/ghes.h b/include/acpi/ghes.h
index 9f26e01186ae..c02b8eb91bd6 100644
--- a/include/acpi/ghes.h
+++ b/include/acpi/ghes.h
@@ -51,31 +51,8 @@  enum {
 	GHES_SEV_PANIC = 0x3,
 };
 
-/* From drivers/edac/ghes_edac.c */
-
-#ifdef CONFIG_EDAC_GHES
-void ghes_edac_report_mem_error(struct ghes *ghes, int sev,
-				struct cper_sec_mem_err *mem_err);
-
-int ghes_edac_register(struct ghes *ghes, struct device *dev);
-
-void ghes_edac_unregister(struct ghes *ghes);
-
-#else
-static inline void ghes_edac_report_mem_error(struct ghes *ghes, int sev,
-				       struct cper_sec_mem_err *mem_err)
-{
-}
-
-static inline int ghes_edac_register(struct ghes *ghes, struct device *dev)
-{
-	return 0;
-}
-
-static inline void ghes_edac_unregister(struct ghes *ghes)
-{
-}
-#endif
+void ghes_register_edac_chain(struct notifier_block *nb);
+void ghes_unregister_edac_chain(struct notifier_block *nb);
 
 static inline int acpi_hest_get_version(struct acpi_hest_generic_data *gdata)
 {