| Message ID | 20170718060007.GB8736@nazgul.tnic (mailing list archive) |
|---|---|
| State | Not Applicable, archived |
| Headers | show |
On Tue, Jul 18, 2017 at 08:00:07AM +0200, Borislav Petkov wrote: > And I think we should try this first: have the firmware disable > detection methods so that the platform drivers don't load. Btw, in looking at this more, what about the firmware-first thing? I.e., the firmware-first detection with apei_osc_setup() at the end of ghes_init(). Can we make ghes_edac loading dependent on that? I mean, that was *the* predicate for exactly that - to have the firmware look at the errors first. No need for platform whitelisting and so on. I'd still decouple ghes_edac loading from ghes_probe() even though loading the platform driver should've been done *after* the firmware-first detection regardless. So what we could do is make ghes_edac a normal module and have the relevant x86 EDAC modules query FF mode and if enabled, fail loading. Hmmm? My gut feeling tells me I'm on the right track here but who knows... Thx.
T24gVHVlLCAyMDE3LTA3LTE4IGF0IDA4OjAwICswMjAwLCBCb3Jpc2xhdiBQZXRrb3Ygd3JvdGU6 DQo+IE9uIE1vbiwgSnVsIDE3LCAyMDE3IGF0IDAzOjU5OjEyUE0gLTA2MDAsIFRvc2hpIEthbmkg d3JvdGU6DQo+ID4gVGhlIGdoZXNfZWRhYyBkcml2ZXIgd2FzIGludHJvZHVjZWQgaW4gMjAxMyBb MV0sIGJ1dCBpdCBoYXMgbm90DQo+ID4gYmVlbiBlbmFibGVkIGJ5IGFueSBkaXN0cm8geWV0LsKg wqBUaGlzIGRyaXZlciBvYnRhaW5zIGVycm9yIGluZm8NCj4gPiBmcm9tIGZpcm13YXJlIGludGVy ZmFjZXMsIHdoaWNoIGFyZSBub3QgcHJvcGVybHkgaW1wbGVtZW50ZWQgb24NCj4gPiBtYW55IHBs YXRmb3JtcywgYXMgdGhlIGRyaXZlciBhbHdheXMgZW1pdHMgdGhlIG1lc3NhZ2VzIGJlbG93Og0K PiA+IA0KPiA+IMKgVGhpcyBFREFDIGRyaXZlciByZWxpZXMgb24gQklPUyB0byBlbnVtZXJhdGUg bWVtb3J5IGFuZCBnZXQgZXJyb3INCj4gPiByZXBvcnRzLiDCoFVuZm9ydHVuYXRlbHksIG5vdCBh bGwgQklPU2VzIHJlZmxlY3QgdGhlIG1lbW9yeSBsYXlvdXQNCj4gPiBjb3JyZWN0bHkgwqBTbywg dGhlIGVuZCByZXN1bHQgb2YgdXNpbmcgdGhpcyBkcml2ZXIgdmFyaWVzIGZyb20NCj4gPiB2ZW5k b3IgdG8gdmVuZG9yIMKgSWYgeW91IGZpbmQgaW5jb3JyZWN0IHJlcG9ydHMsIHBsZWFzZSBjb250 YWN0DQo+ID4geW91ciBoYXJkd2FyZSB2ZW5kb3IgwqB0byBjb3JyZWN0IGl0cyBCSU9TLg0KPiA+ IA0KPiA+IFRvIGdldCBvdXQgZnJvbSB0aGlzIHNpdHVhdGlvbiwgYWRkIGEgcGxhdGZvcm0gdHlw ZSBjaGVjayB0bw0KPiA+IHNlbGVjdGl2ZWx5IGVuYWJsZSB0aGUgZHJpdmVyIG9uIHRoZSBwbGF0 Zm9ybXMgdGhhdCBhcmUga25vd24gdG8NCj4gPiBoYXZlIHByb3BlciBmaXJtd2FyZSBpbXBsZW1l bnRhdGlvbi7CoMKgUGxhdGZvcm0gdmVuZG9ycyBjYW4gYWRkDQo+ID4gdGhlaXIgcGxhdGZvcm1z IHRvIHRoZSBsaXN0IHdoZW4gdGhleSBzdXBwb3J0IGdoZXNfZWRhYy4NCj4gDQo+IFNvIG1haW50 YWluaW5nIHdoaXRlbGlzdHMgZm9yIHRoaW5ncyBoYXMgYWx3YXlzIGJlZW4gYSBQSVRBIGFuZCB3 ZQ0KPiBzaG91bGQgdHJ5IHRvIGF2b2lkIGl0LCBpZiBwb3NzaWJsZS4gKFdlIGNhbiBhbHdheXMg ZG8gaXQgaWYgbm90aGluZw0KPiBzYW5lciBjb21lcyBhbG9uZy4pDQoNCkFncmVlZC4NCg0KPiBO b3csIGJlbG93IGlzIGEgZGlydHkgcGF0Y2ggY29udmVydGluZyBnaGVzX2VkYWMgdG8gYSBub3Jt YWwgbW9kdWxlLg0KPiBPbiBzeXN0ZW1zIHdoZXJlIHdlIGhhdmUgR0hFUywgdGhlIGZpcm13YXJl IGdlbmVyYWxseSBkaXNhYmxlcyB0aGUNCj4gZGV0ZWN0aW9uIG9mIHRoZSBwcmVzZW5jZSBvZiBF Q0MgaGFyZHdhcmUsIHRodXMgcHJldmVudGluZyB0aGUNCj4gcGxhdGZvcm0gRURBQyBkcml2ZXIg ZnJvbSBsb2FkaW5nLg0KDQpJIGhhdmUgSFBFIEhhc3dlbGwgYW5kIFNreWxha2UgdGVzdCBzeXN0 ZW1zIHdpdGggR0hFUywgYnV0IHRoZXkgZG8gbm90DQpoaWRlIElNQ3MgZnJvbSB0aGUgT1MuICBT bywgdGhlIHNiX2VkYWMgYW5kIHNreF9lZGFjIGRyaXZlcnMgZ2V0DQphdHRhY2hlZCBvbiB0aGVz ZSBzeXN0ZW1zIHdoZW4gZ2hlc19lZGFjIGlzIGRpc2FibGVkLg0KDQo+IExldCBtZSBjbGFyaWZ5 OiBJIGhhdmUgYW4gQU1EIEhQIGJveCB3aGljaCwgd2hlbiBHSEVTIGlzIGVuYWJsZWQgaW4NCj4g dGhlIEJJT1MsIHNheXMgdGhhdCBFQ0MgaXMgZGlzYWJsZWQgaW4gdGhlIG1lbW9yeSBjb250cm9s bGVyIGFuZCB0aGUNCj4gYW1kNjRfZWRhYyBkcml2ZXIgZG9lc24ndCBsb2FkIGZvciB0aGF0IG1l bW9yeSBjb250cm9sbGVyLg0KDQpIbW0uLi4gd2hhdCdzIHRoZSBwbGF0Zm9ybSBuYW1lIG9mIHRo aXMgYm94PyAgSSBjYW4gbG9vayBpbnRvIHRoaXMgY2FzZQ0KaWYgeW91IG5lZWQuDQoNCj4gQW5k IEkgdGhpbmsgd2Ugc2hvdWxkIHRyeSB0aGlzIGZpcnN0OiBoYXZlIHRoZSBmaXJtd2FyZSBkaXNh YmxlDQo+IGRldGVjdGlvbiBtZXRob2RzIHNvIHRoYXQgdGhlIHBsYXRmb3JtIGRyaXZlcnMgZG9u J3QgbG9hZC4NCg0KSSBkbyBub3QgdGhpbmsgd2UgY2FuIHJlbHkgb24gdGhpcyBtZXRob2QuDQoN Cj4gVGhlbiwgZ2hlc19lZGFjIGNhbiBiZSBhIHNpbXBsZSBtb2R1bGUgYW5kIG5vIG90aGVyIGRy aXZlciB3b3VsZA0KPiBhdHRlbXB0IGxvYWRpbmcuDQoNCkkgbGlrZSB0aGUgdXNlIG9mIG5vdGlm aWVyIGNoYWluLCB3aGljaCBpcyBtdWNoIGNsZWFuZXIuDQoNCj4gVGhlIHF1ZXN0aW9uIGlzOiBk b2VzIHRoZSBwbGF0Zm9ybSBkbyB0aGlzIGRpc2FibGluZyBub3c/DQoNClVuZm9ydHVuYXRlbHks IHRoYXQgaXMgbm90IHRoZSBjYXNlIHRvZGF5LiAgVGhlIElNQ3MgY2Fubm90IGJlIGhpZGRlbg0K d2l0aCB0aGUgRGV2aWNlIEhpZGUgcmVnaXN0ZXJzIGZvciBTa3lsYWtlIGF0IGxlYXN0Lg0KDQpU aGFua3MsDQotVG9zaGkNCg0K -- To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Em Tue, 18 Jul 2017 19:58:54 +0000 "Kani, Toshimitsu" <toshi.kani@hpe.com> escreveu: > On Tue, 2017-07-18 at 08:00 +0200, Borislav Petkov wrote: > > On Mon, Jul 17, 2017 at 03:59:12PM -0600, Toshi Kani wrote: > > > The ghes_edac driver was introduced in 2013 [1], but it has not > > > been enabled by any distro yet. This driver obtains error info > > > from firmware interfaces, which are not properly implemented on > > > many platforms, as the driver always emits the messages below: > > > > > > This EDAC driver relies on BIOS to enumerate memory and get error > > > reports. Unfortunately, not all BIOSes reflect the memory layout > > > correctly So, the end result of using this driver varies from > > > vendor to vendor If you find incorrect reports, please contact > > > your hardware vendor to correct its BIOS. > > > > > > To get out from this situation, add a platform type check to > > > selectively enable the driver on the platforms that are known to > > > have proper firmware implementation. Platform vendors can add > > > their platforms to the list when they support ghes_edac. > > > > So maintaining whitelists for things has always been a PITA and we > > should try to avoid it, if possible. (We can always do it if nothing > > saner comes along.) > > Agreed. > > > Now, below is a dirty patch converting ghes_edac to a normal module. > > On systems where we have GHES, the firmware generally disables the > > detection of the presence of ECC hardware, thus preventing the > > platform EDAC driver from loading. > > I have HPE Haswell and Skylake test systems with GHES, but they do not > hide IMCs from the OS. So, the sb_edac and skx_edac drivers get > attached on these systems when ghes_edac is disabled. > > > Let me clarify: I have an AMD HP box which, when GHES is enabled in > > the BIOS, says that ECC is disabled in the memory controller and the > > amd64_edac driver doesn't load for that memory controller. > > Hmm... what's the platform name of this box? I can look into this case > if you need. > > > And I think we should try this first: have the firmware disable > > detection methods so that the platform drivers don't load. > > I do not think we can rely on this method. > > > Then, ghes_edac can be a simple module and no other driver would > > attempt loading. > > I like the use of notifier chain, which is much cleaner. > > > The question is: does the platform do this disabling now? > > Unfortunately, that is not the case today. The IMCs cannot be hidden > with the Device Hide registers for Skylake at least. We had a similar discussion several years ago when I wrote this driver. On that time, I talked with Red Hat, HP, Dell, Intel people and with some customers with large clusters. The way it is, ghes_edac is a poor man's driver. What it hopefully provide is a detection that an error happened, without really telling the user what component should be replaced. Ok, on machines with their own error reporting mechanism (like HP servers), a sys admin can look on some proprietary software (or bios), in order to identify what happened. Yet, BIOS doesn't provide any glue about what's the memory architecture, as it maps memory as if it was a single DIMM memory: (from ghes_edac_register) layers[0].type = EDAC_MC_LAYER_ALL_MEM; layers[0].size = num_dimm; layers[0].is_virt_csrow = true; So, even on systems where the BIOS actually knows how the memory cards are wired, it will mask the memory controller data. Now, the EDAC driver can also be used to identify what channels are used. That helps the sys admin to know if the memories are connected in a way that it will be using multiple channels, or not, helping to setup the machine to obtain the maximum possible performance. So, for example, on my Intel-based HP server, I can check such info with: $ ras-mc-ctl --mainboard ras-mc-ctl: mainboard: HP model ProLiant ML350 Gen9 $ ras-mc-ctl --layout +-----------------------------------------------------------------------+ | mc0 | mc1 | | channel0 | channel1 | channel2 | channel0 | channel1 | channel2 | -------+-----------------------------------------------------------------------+ slot2: | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | slot1: | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | slot0: | 16384 MB | 0 MB | 16384 MB | 16384 MB | 0 MB | 16384 MB | -------+---------------------------------------------------------------------------+ So, I know that both CPUs will be connected to my memories, and, on both, it is using 2 channels. If I was using the ghes driver, that information would be hidden. So, due to all problems with ghes, it is enabled only if there are no better solution, e. g. on systems where there's no way to talk directly to the hardware (like on E7 Xeon machines, where the memory controller is actually on a separate chip that are controlled only by the BIOS). Thanks, Mauro -- To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Tue, 2017-07-18 at 10:08 +0200, Borislav Petkov wrote: > On Tue, Jul 18, 2017 at 08:00:07AM +0200, Borislav Petkov wrote: > > And I think we should try this first: have the firmware disable > > detection methods so that the platform drivers don't load. > > Btw, in looking at this more, what about the firmware-first thing? > > I.e., the firmware-first detection with apei_osc_setup() at the end > of ghes_init(). > > Can we make ghes_edac loading dependent on that? I mean, that was > *the* predicate for exactly that - to have the firmware look at the > errors first. No need for platform whitelisting and so on. I agree that 'osc_sb_apei_support_acked' should be checked when enabling ghes_edac. I do not know the details of existing issues, but it sounds unlikely that this will address all of them since bugs can be everywhere. For instance, ghes_edac relies on DMI/SMBIOS info, unlike other EDAC drivers, which can be buggy regardless of this _OSC info. > I'd still decouple ghes_edac loading from ghes_probe() even though > loading the platform driver should've been done *after* the > firmware-first detection regardless. > > So what we could do is make ghes_edac a normal module and have the > relevant x86 EDAC modules query FF mode and if enabled, fail loading. I agree that making ghes_edac as a normal module is a good thing, but I do not think it's going to solve this issue. Thanks, -Toshi
> The question is: does the platform do this disabling now? > > Tony, I'm looking at sb_edac and there we don't do something like that > or maybe I'm missing it. Historically we've had complaints that sb_edac won't load that have been tracked to BIOS hiding one of the (many) PCI devices that it needs. But device hiding is orthogonal to providing GHES error records. A BIOS might do that, but I don't know that anyone intentionally does so. -Tony
On Tue, Jul 18, 2017 at 09:20:44PM +0000, Kani, Toshimitsu wrote: > I agree that 'osc_sb_apei_support_acked' should be checked when > enabling ghes_edac. I do not know the details of existing issues, but > it sounds unlikely that this will address all of them since bugs can be > everywhere. No, see below. > For instance, ghes_edac relies on DMI/SMBIOS info, unlike > other EDAC drivers, which can be buggy regardless of this _OSC info. That's the problem with firmware. You can't really fix it and it is buggy as hell. > I agree that making ghes_edac as a normal module is a good thing, but I > do not think it's going to solve this issue. Of course it will - if the firmware says it wants to look at the errors first, then it gets to do so. This is the whole handling of hardware errors in the firmware deal. I admit, sometimes it makes sense because the firmware has the most intimate knowledge of the platform and, in a perfect world, we won't ever need to have platform-specific EDAC drivers. But, we don't live in a perfect world. And the vendor execution of the whole firmware-error-handling deal is an abomination at best. So, if we realize that the firmware is buggy, we can use a platform list to blacklist it (^hint hint^) and have a parameter to disable ghes_edac from loading. But we'll deal with that when we get to cross that bridge. Right now, I'd like to do the loading spec-conform and not fiddle with white-, black-, or any-other-color lists.
On Tue, Jul 18, 2017 at 07:58:54PM +0000, Kani, Toshimitsu wrote: > I have HPE Haswell and Skylake test systems with GHES, but they do not > hide IMCs from the OS. So, the sb_edac and skx_edac drivers get > attached on these systems when ghes_edac is disabled. That's how it is supposed to work. The platform drivers are the fallback, practically. But this is the important piece of info I was looking for - having GHES enabled in the firmware does not prevent the platform drivers from loading. But I think we have a better solution, the FF thing. > Hmm... what's the platform name of this box? I can look into this case > if you need. You can but that's not addressing the issue as a whole so it'll be a waste of time.
On Tue, Jul 18, 2017 at 06:15:45PM -0300, Mauro Carvalho Chehab wrote: > The way it is, ghes_edac is a poor man's driver. What it hopefully > provide is a detection that an error happened, without really telling > the user what component should be replaced. I beg to differ. From the UEFI spec: "The module number of the memory error location. (NODE, CARD, and MODULE should provide the information necessary to identify the failing FRU)." So this tuple is sufficient to pinpoint the DIMM, IIUC. Which means, ghes_edac can have a single layer of DIMMs without channels.
On Tue, Jul 18, 2017 at 10:13:42PM +0000, Luck, Tony wrote: > Historically we've had complaints that sb_edac won't load that have been > tracked to BIOS hiding one of the (many) PCI devices that it needs. But > device hiding is orthogonal to providing GHES error records. A BIOS might > do that, but I don't know that anyone intentionally does so. Yeah, the hiding-devices path doesn't look like the optimal one. I think we should look at the firmware-first setting and load ghes if FF is being done by the firmware.
> "The module number of the memory error location. (NODE, CARD, and MODULE > should provide the information necessary to identify the failing FRU)." > > So this tuple is sufficient to pinpoint the DIMM, IIUC. > > Which means, ghes_edac can have a single layer of DIMMs without channels. The tricky part is that you have to rely on SMBIOS/DMI to know what DIMMs are on the system when the driver initializes so you can populate /sys/.*/edac Later when GHES gives you a NODE/CARD/MODULE) in an error record. You need to match these up. But SMBIOS only gave you two strings "Locator" and "Bank Locator" which have no defined syntax. You are at the mercy of the BIOS writer to put in something parseable. Some writers used zero based counts, others are Fortran fans and use one-based. Still other use letters. About the one guarantee is that they will make almost no effort to match the silkscreen labels on the motherboard itself. E.g. my Broadwell-EX has things like: Locator: CHANNEL D DIMM 1 Bank Locator: Memriser8 Channel is A,B,C,D. DIMM is 0, 1, 2. Memriser is {1..8} so this manages to use all three counting options! -Tony
On Wed, Jul 19, 2017 at 03:14:32PM +0000, Luck, Tony wrote: > Later when GHES gives you a NODE/CARD/MODULE) in an error record. You need > to match these up. But SMBIOS only gave you two strings "Locator" and "Bank > Locator" which have no defined syntax. You are at the mercy of the BIOS writer > to put in something parseable. Well, at some point it is only so much we can do, right? I mean, if FW says it wants to do firmware-first and we go and adhere to that, it should be expected that said FW vendor marks the silkscreen labels and DMI data accordingly. I mean, it is time for FW to put its money where its mouth is, no? How else would you do this? Firmware First but the kernel does the figuring out which DIMMs are where. So FW can't have the cake and eat it too. :-)
Tony/Aris,
I got yesterday an HP ML350 G9, equipped with Sandy Bridge EP CPUs (E5-2640v4).
I'm running Kernel 4.11 there.
AFAIKT, Sandy Bridge EP has 4 channels per memory controller, right?
That would match the number of memory slots on this machine (24 slots).
Yet, EDAC is only identifying 3 channels per CPU:
$ ras-mc-ctl --layout
+-----------------------------------------------------------------------+
| mc0 | mc1 |
| channel0 | channel1 | channel2 | channel0 | channel1 | channel2 |
-------+-----------------------------------------------------------------------+
slot2: | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB |
slot1: | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB |
slot0: | 16384 MB | 0 MB | 16384 MB | 16384 MB | 0 MB | 16384 MB |
-------+---------------------------------------------------------------------------+
So, it seems that either the BIOS is hidden the other channel or
there's something wrong with SandyBridge EP support at sb_edac driver.
Thanks,
Mauro
--
To unsubscribe from this list: send the line "unsubscribe linux-acpi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
On Wed, 2017-07-19 at 07:52 +0200, Borislav Petkov wrote: > On Tue, Jul 18, 2017 at 09:20:44PM +0000, Kani, Toshimitsu wrote: > > I agree that 'osc_sb_apei_support_acked' should be checked when > > enabling ghes_edac. I do not know the details of existing issues, > > but it sounds unlikely that this will address all of them since > > bugs can be everywhere. > > No, see below. > > > For instance, ghes_edac relies on DMI/SMBIOS info, unlike > > other EDAC drivers, which can be buggy regardless of this _OSC > > info. > > That's the problem with firmware. You can't really fix it and it is > buggy as hell. Right, and that's what I was told as an issue for ghes_edac. This is why this patch introduces a white-list to preclude all buggy firmwares that are unknown to us... > > I agree that making ghes_edac as a normal module is a good thing, > > but I do not think it's going to solve this issue. > > Of course it will - if the firmware says it wants to look at the > errors first, then it gets to do so. This is the whole handling of > hardware errors in the firmware deal. I admit, sometimes it makes > sense because the firmware has the most intimate knowledge of the > platform and, in a perfect world, we won't ever need to have > platform-specific EDAC drivers. > > But, we don't live in a perfect world. And the vendor execution of > the whole firmware-error-handling deal is an abomination at best. > > So, if we realize that the firmware is buggy, we can use a platform > list to blacklist it (^hint hint^) and have a parameter to disable > ghes_edac from loading. Setting blacklist needs us to enable ghes_edac and find all buggy firmwares to date. I think this is too disturbing for people who are happily using regular edac drivers today even though their platforms have GHES. > But we'll deal with that when we get to cross that bridge. Right now, > I'd like to do the loading spec-conform and not fiddle with white-, > black-, or any-other-color lists. I do prefer to avoid any white / black listing. But I do not see how it solves the buggy DMI/SMBIOS info as an example of firmware bugs we may have to deal with. Thanks, -Toshi
On Wed, Jul 19, 2017 at 04:10:07PM +0000, Kani, Toshimitsu wrote: > I do prefer to avoid any white / black listing. But I do not see how > it solves the buggy DMI/SMBIOS info as an example of firmware bugs we > may have to deal with. So how do you want to deal with this? Maintain an evergrowing whitelist of platforms which are OK and then the moment a new platform comes along, you send a patch to add it to that whitelist? I'm sure you can see the problems with that approach.
On Tue, 2017-07-18 at 18:15 -0300, Mauro Carvalho Chehab wrote: > Em Tue, 18 Jul 2017 19:58:54 +0000 : > We had a similar discussion several years ago when I wrote this > driver. On that time, I talked with Red Hat, HP, Dell, Intel people > and with some customers with large clusters. > > The way it is, ghes_edac is a poor man's driver. What it hopefully > provide is a detection that an error happened, without really telling > the user what component should be replaced. "poor man's driver" is a bit misleading, but yes, firmware-first platforms have RAS features built-into the platforms, and they do not need intelligence in EDAC drivers, which may conflict with the platform's RAS features. I cannot speak for other vendors, but HPE platforms log errors and provide FRU info. ghes_edac allows to report errors to OS management tools like rasdaemon in addition to platform- specific managements. > Ok, on machines with their own error reporting mechanism (like > HP servers), a sys admin can look on some proprietary software > (or bios), in order to identify what happened. > > Yet, BIOS doesn't provide any glue about what's the memory > architecture, as it maps memory as if it was a single DIMM memory: > > (from ghes_edac_register) > > layers[0].type = EDAC_MC_LAYER_ALL_MEM; > layers[0].size = num_dimm; > layers[0].is_virt_csrow = true; > > So, even on systems where the BIOS actually knows how the memory > cards are wired, it will mask the memory controller data. > > Now, the EDAC driver can also be used to identify what > channels are used. That helps the sys admin to know if the > memories are connected in a way that it will be using multiple > channels, or not, helping to setup the machine to obtain > the maximum possible performance. > > So, for example, on my Intel-based HP server, I can check > such info with: > > $ ras-mc-ctl --mainboard > ras-mc-ctl: mainboard: HP model ProLiant ML350 Gen9 > $ ras-mc-ctl --layout > +------------------------------------------------------------- > ----------+ > | mc0 | mc1 > | > | channel0 | channel1 | channel2 | channel0 | channel1 | > channel2 | > -------+------------------------------------------------------------- > ----------+ > slot2: | 0 MB | 0 MB | 0 MB | 0 MB | 0 > MB | 0 MB | > slot1: | 0 MB | 0 MB | 0 MB | 0 MB | 0 > MB | 0 MB | > slot0: | 16384 MB | 0 MB | 16384 MB | 16384 MB | 0 > MB | 16384 MB | > -------+------------------------------------------------------------- > --------------+ > > So, I know that both CPUs will be connected to my memories, and, > on both, it is using 2 channels. > > If I was using the ghes driver, that information would be hidden. > > So, due to all problems with ghes, it is enabled only if there are no > better solution, e. g. on systems where there's no way to talk > directly to the hardware (like on E7 Xeon machines, where the memory > controller is actually on a separate chip that are controlled only by > the BIOS). Thanks for the info! That's very helpful. I will check to see if ghes_edac provides enough info that we need. -Toshi
On Wed, 2017-07-19 at 18:22 +0200, Borislav Petkov wrote: > On Wed, Jul 19, 2017 at 04:10:07PM +0000, Kani, Toshimitsu wrote: > > I do prefer to avoid any white / black listing. But I do not see > > how it solves the buggy DMI/SMBIOS info as an example of firmware > > bugs we may have to deal with. > > So how do you want to deal with this? > > Maintain an evergrowing whitelist of platforms which are OK and then > the moment a new platform comes along, you send a patch to add it to > that whitelist? > > I'm sure you can see the problems with that approach. Since ghes_edac has not been used for a long time, I have a feeling that not so many vendors want to use it. In the case of HPE, we do not need to update with each platform since "HPE" "Server" will cover all platforms we need. Thanks, -Toshi
Pj4gTGF0ZXIgd2hlbiBHSEVTIGdpdmVzIHlvdSBhIE5PREUvQ0FSRC9NT0RVTEUpIGluIGFuIGVy cm9yIHJlY29yZC4gIFlvdSBuZWVkDQo+PiB0byBtYXRjaCB0aGVzZSB1cC4gQnV0IFNNQklPUyBv bmx5IGdhdmUgeW91IHR3byBzdHJpbmdzICJMb2NhdG9yIiBhbmQgIkJhbmsNCj4+IExvY2F0b3Ii IHdoaWNoIGhhdmUgbm8gZGVmaW5lZCBzeW50YXguIFlvdSBhcmUgYXQgdGhlIG1lcmN5IG9mIHRo ZSBCSU9TIHdyaXRlcg0KPj4gdG8gcHV0IGluIHNvbWV0aGluZyBwYXJzZWFibGUuDQo+DQo+IFdl bGwsIGF0IHNvbWUgcG9pbnQgaXQgaXMgb25seSBzbyBtdWNoIHdlIGNhbiBkbywgcmlnaHQ/DQo+ DQo+IEkgbWVhbiwgaWYgRlcgc2F5cyBpdCB3YW50cyB0byBkbyBmaXJtd2FyZS1maXJzdCBhbmQg d2UgZ28gYW5kIGFkaGVyZQ0KPiB0byB0aGF0LCBpdCBzaG91bGQgYmUgZXhwZWN0ZWQgdGhhdCBz YWlkIEZXIHZlbmRvciBtYXJrcyB0aGUgc2lsa3NjcmVlbg0KPiBsYWJlbHMgYW5kIERNSSBkYXRh IGFjY29yZGluZ2x5Lg0KPg0KPiBJIG1lYW4sIGl0IGlzIHRpbWUgZm9yIEZXIHRvIHB1dCBpdHMg bW9uZXkgd2hlcmUgaXRzIG1vdXRoIGlzLCBubz8NCj4NCj4gSG93IGVsc2Ugd291bGQgeW91IGRv IHRoaXM/DQoNCkJ5IHRoaW5raW5nIGEgYml0IG1vcmUgYW5kIHJlYWxpemluZyB0aGF0IHdoYXQg SSB3cm90ZSB1cCBhYm92ZSBtaXNzZXMgdGhhdA0KYXQgYnl0ZSBvZmZzZXQgNzggaW4gdGhlIFVF RkkgbWVtb3J5IGVycm9yIHNlY3Rpb24gdGhlcmUgaXMgIk1vZHVsZSBIYW5kbGUiDQp3aGljaCB0 ZWxscyB5b3Ugd2hpY2ggU01CSU9TIGVudHJ5IHRvIHVzZS4NCg0KU28gdGhpcyBzaG91bGQgd29y ayBqdXN0IGZpbmUgKGFzIGxvbmcgYXMgQklPUyBmaWxscyBvdXQgYWxsIHRoZXNlIGZpZWxkcyAu Li4NCnRoZXJlJ3MgYSAiVmFsaWRhdGlvbiBCaXRzIiBtYXNrIGF0IHRoZSBzdGFydCBvZiB0aGUg ZXJyb3Igc3RydWN0dXJlIHRoYXQgc2F5cw0Kd2hpY2ggZmllbGRzIGhhdmUgYmVlbiBwb3B1bGF0 ZWQpLg0KDQotVG9ueQ0K -- To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Wed, Jul 19, 2017 at 06:22:04PM +0200, Borislav Petkov wrote: > On Wed, Jul 19, 2017 at 04:10:07PM +0000, Kani, Toshimitsu wrote: > > I do prefer to avoid any white / black listing. But I do not see how > > it solves the buggy DMI/SMBIOS info as an example of firmware bugs we > > may have to deal with. > > So how do you want to deal with this? > > Maintain an evergrowing whitelist of platforms which are OK and then the > moment a new platform comes along, you send a patch to add it to that > whitelist? That would also need to keep an eye on versions. A newer version of BIOS on a whitelisted platform might be broken.
> So, it seems that either the BIOS is hidden the other channel or > there's something wrong with SandyBridge EP support at sb_edac driver. Can you send me the out of "lspci -xxxx" (run as root)? -Tony -- To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
T24gV2VkLCAyMDE3LTA3LTE5IGF0IDE0OjU1IC0wNDAwLCBBcmlzdGV1IFJvemFuc2tpIHdyb3Rl Og0KPiBPbiBXZWQsIEp1bCAxOSwgMjAxNyBhdCAwNjoyMjowNFBNICswMjAwLCBCb3Jpc2xhdiBQ ZXRrb3Ygd3JvdGU6DQo+ID4gT24gV2VkLCBKdWwgMTksIDIwMTcgYXQgMDQ6MTA6MDdQTSArMDAw MCwgS2FuaSwgVG9zaGltaXRzdSB3cm90ZToNCj4gPiA+IEkgZG8gcHJlZmVyIHRvIGF2b2lkIGFu eSB3aGl0ZSAvIGJsYWNrIGxpc3RpbmcuwqDCoEJ1dCBJIGRvIG5vdCBzZWUNCj4gPiA+IGhvdw0K PiA+ID4gaXQgc29sdmVzIHRoZSBidWdneSBETUkvU01CSU9TIGluZm8gYXMgYW4gZXhhbXBsZSBv ZiBmaXJtd2FyZQ0KPiA+ID4gYnVncyB3ZQ0KPiA+ID4gbWF5IGhhdmUgdG8gZGVhbCB3aXRoLg0K PiA+IA0KPiA+IFNvIGhvdyBkbyB5b3Ugd2FudCB0byBkZWFsIHdpdGggdGhpcz8NCj4gPiANCj4g PiBNYWludGFpbiBhbiBldmVyZ3Jvd2luZyB3aGl0ZWxpc3Qgb2YgcGxhdGZvcm1zIHdoaWNoIGFy ZSBPSyBhbmQNCj4gPiB0aGVuIHRoZSBtb21lbnQgYSBuZXcgcGxhdGZvcm0gY29tZXMgYWxvbmcs IHlvdSBzZW5kIGEgcGF0Y2ggdG8gYWRkDQo+ID4gaXQgdG8gdGhhdCB3aGl0ZWxpc3Q/DQo+IA0K PiBUaGF0IHdvdWxkIGFsc28gbmVlZCB0byBrZWVwIGFuIGV5ZSBvbiB2ZXJzaW9ucy4gQSBuZXdl ciB2ZXJzaW9uIG9mDQo+IEJJT1Mgb24gYSB3aGl0ZWxpc3RlZCBwbGF0Zm9ybSBtaWdodCBiZSBi cm9rZW4uDQoNClJpZ2h0LiAgSSB0aGluayBhIHF1ZXN0aW9uIGNvbWVzIHRvIHdobyBicm9rZSBh IHJ1bm5pbmcgc3lzdGVtIC0tIE9TDQp1cGRhdGUgb3IgQklPUyB1cGRhdGUuICBUaGlzIHdoaXRl bGlzdCBhdHRlbXB0cyB0byBwcm90ZWN0IHRoZSBmb3JtZXINCmNhc2UgYnkgbm90IGludHJvZHVj aW5nIGdoZXNfZWRhYyBvbiBhcmJpdHJhcnkgcGxhdGZvcm1zLiAgVGhlIGxhdHRlcg0KY2FzZSBz aG91bGQgYmUgdmVuZG9yJ3MgcmVzcG9uc2liaWxpdHkuDQoNClRoYW5rcywNCi1Ub3NoaQ0K -- To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Wed, Jul 19, 2017 at 04:56:17PM +0000, Kani, Toshimitsu wrote: > Since ghes_edac has not been used for a long time, I have a feeling > that not so many vendors want to use it. In the case of HPE, we do not > need to update with each platform since "HPE" "Server" will cover all > platforms we need. Does the apei_osc_setup() detection with the uuid work on HP systems?
On Wed, Jul 19, 2017 at 02:55:08PM -0400, Aristeu Rozanski wrote: > That would also need to keep an eye on versions. A newer version of BIOS > on a whitelisted platform might be broken. Yeah, that would be a nasty, back-stabbing SNAFU. So I'm thinking of adding a bunch of FW_ERR sanity checks to that whole ghes_edac and ghes init code to hopefully catch issues during platform validation. I.e., early enough for them to get fixed. But that's the same problem as with UEFI - vendors need to try to boot Linux on their platforms early enough.
On Wed, Jul 19, 2017 at 04:40:25PM +0000, Kani, Toshimitsu wrote: > ghes_edac allows to report errors to OS management tools like > rasdaemon in addition to platform- specific managements. So ghes_edac *is* a poor man's driver in the sense that it doesn't do anything fancy but repeat like a parrot data it has gotten from the firmware and shoving it into the EDAC counters. At least that's the intention. Nothing more. All the action stuff like error detection and recovery should be done by the firmware. But considering how SNAFU'd firmware is, I wouldn't expect any great RAS functionality there. Of course, I'd be delighted to be proven wrong.
On Thu, 2017-07-20 at 06:16 +0200, Borislav Petkov wrote: > On Wed, Jul 19, 2017 at 04:56:17PM +0000, Kani, Toshimitsu wrote: > > Since ghes_edac has not been used for a long time, I have a feeling > > that not so many vendors want to use it. In the case of HPE, we do > > not need to update with each platform since "HPE" "Server" will > > cover all platforms we need. > > Does the apei_osc_setup() detection with the uuid work on HP systems? Yes, the following message is shown on HP systems. Please note that WHEA is a Windows-defined interface. "GHES: APEI firmware first mode is enabled by APEI bit and WHEA _OSC." Thanks, -Toshi
On Thu, Jul 20, 2017 at 02:42:25PM +0000, Kani, Toshimitsu wrote: > Yes, the following message is shown on HP systems. Please note that > WHEA is a Windows-defined interface. Ok, so let's couple ghes_edac loading to that and see how far we could go. I guess we should add checks for that to the major x86 EDAC drivers to not load and this way ghes_edac will be the only driver loading. Tony, how does that sound?
Pj4gWWVzLCB0aGUgZm9sbG93aW5nIG1lc3NhZ2UgaXMgc2hvd24gb24gSFAgc3lzdGVtcy4gIFBs ZWFzZSBub3RlIHRoYXQNCj4+IFdIRUEgaXMgYSBXaW5kb3dzLWRlZmluZWQgaW50ZXJmYWNlLg0K Pg0KPiBPaywgc28gbGV0J3MgY291cGxlIGdoZXNfZWRhYyBsb2FkaW5nIHRvIHRoYXQgYW5kIHNl ZSBob3cgZmFyIHdlIGNvdWxkDQo+IGdvLiBJIGd1ZXNzIHdlIHNob3VsZCBhZGQgY2hlY2tzIGZv ciB0aGF0IHRvIHRoZSBtYWpvciB4ODYgRURBQyBkcml2ZXJzDQo+IHRvIG5vdCBsb2FkIGFuZCB0 aGlzIHdheSBnaGVzX2VkYWMgd2lsbCBiZSB0aGUgb25seSBkcml2ZXIgbG9hZGluZy4NCj4NCj4g VG9ueSwgaG93IGRvZXMgdGhhdCBzb3VuZD8NCg0KQWRkIGEgbW9kdWxlIHBhcmFtZXRlciB0byB0 aG9zZSBlZGFjIGRyaXZlcnMgdGhhdCBjYW4gb3ZlcnJpZGUgdGhlIGNoZWNrDQphbmQgbGV0IHRo ZW0gbG9hZCBhbnl3YXkuICBJJ20gbm90IHBhcmFub2lkLCBJIGp1c3QgYXNzdW1lIHRoYXQgdGhl cmUgaXMgYSBCSU9TDQpvdXQgdGhlcmUgdGhhdCBzZXRzIHRoZSBPU0MvV0hFQSBiaXRzLCBidXQg aXNuJ3QgZ2VuZXJhdGluZyB1c2VmdWwgR0hFUyBsb2dzLg0KDQotVG9ueQ0K -- To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, Jul 20, 2017 at 04:55:59PM +0000, Luck, Tony wrote: > Add a module parameter to those edac drivers that can override the check > and let them load anyway. I'm not paranoid, I just assume that there is a BIOS > out there that sets the OSC/WHEA bits, but isn't generating useful GHES logs. Or add that parameter to edac_core.ko and let it control which EDAC driver gets loaded? Something like edac=ignore_ghes or so. And then the other EDAC drivers query it.
PiBPciBhZGQgdGhhdCBwYXJhbWV0ZXIgdG8gZWRhY19jb3JlLmtvIGFuZCBsZXQgaXQgY29udHJv bCB3aGljaCBFREFDDQo+IGRyaXZlciBnZXRzIGxvYWRlZD8gU29tZXRoaW5nIGxpa2UNCj4NCj4g ZWRhYz1pZ25vcmVfZ2hlcw0KPg0KPiBvciBzby4gQW5kIHRoZW4gdGhlIG90aGVyIEVEQUMgZHJp dmVycyBxdWVyeSBpdC4NCg0KU3VyZSAuLi4gb25lIGNlbnRyYWwgcGxhY2UgaXMgYmV0dGVyIHRo YW4gYWRkaW5nIGNvZGUgdG8gZWFjaA0KZHJpdmVyLg0KDQotVG9ueQ0K -- To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Em Thu, 20 Jul 2017 19:05:04 +0200 Borislav Petkov <bp@alien8.de> escreveu: > On Thu, Jul 20, 2017 at 04:55:59PM +0000, Luck, Tony wrote: > > Add a module parameter to those edac drivers that can override the check > > and let them load anyway. I'm not paranoid, I just assume that there is a BIOS > > out there that sets the OSC/WHEA bits, but isn't generating useful GHES logs. > > Or add that parameter to edac_core.ko and let it control which EDAC > driver gets loaded? Something like > > edac=ignore_ghes > > or so. And then the other EDAC drivers query it. Works for me. Thanks, Mauro -- To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, 2017-07-20 at 06:33 +0200, Borislav Petkov wrote: > On Wed, Jul 19, 2017 at 04:40:25PM +0000, Kani, Toshimitsu wrote: > > ghes_edac allows to report errors to OS management tools like > > rasdaemon in addition to platform- specific managements. > > So ghes_edac *is* a poor man's driver in the sense that it doesn't do > anything fancy but repeat like a parrot data it has gotten from the > firmware and shoving it into the EDAC counters. At least that's the > intention. Nothing more. Right for ghes_edac. > All the action stuff like error detection and recovery should be done > by the firmware. GHES / firmware-first still requires OS recovery actions when an error cannot be corrected by the platform. They are handled by ghes_proc(), and ghes_edac remains its error-reporting wrapper. > But considering how SNAFU'd firmware is, I wouldn't expect any great > RAS functionality there. Of course, I'd be delighted to be proven > wrong. Firmware has better knowledge about the platform and can provide better RAS when implemented properly. I agree that user experiences may vary on platforms. Thanks, -Toshi
Em Thu, 20 Jul 2017 19:50:03 +0000 "Kani, Toshimitsu" <toshi.kani@hpe.com> escreveu: > On Thu, 2017-07-20 at 06:33 +0200, Borislav Petkov wrote: > > On Wed, Jul 19, 2017 at 04:40:25PM +0000, Kani, Toshimitsu wrote: > > > ghes_edac allows to report errors to OS management tools like > > > rasdaemon in addition to platform- specific managements. > > > > So ghes_edac *is* a poor man's driver in the sense that it doesn't do > > anything fancy but repeat like a parrot data it has gotten from the > > firmware and shoving it into the EDAC counters. At least that's the > > intention. Nothing more. > > Right for ghes_edac. > > > All the action stuff like error detection and recovery should be done > > by the firmware. > > GHES / firmware-first still requires OS recovery actions when an error > cannot be corrected by the platform. They are handled by ghes_proc(), > and ghes_edac remains its error-reporting wrapper. > > > But considering how SNAFU'd firmware is, I wouldn't expect any great > > RAS functionality there. Of course, I'd be delighted to be proven > > wrong. > > Firmware has better knowledge about the platform and can provide better > RAS when implemented properly. I agree that user experiences may vary > on platforms. It may have a better knowledge, when the vendor ships different BIOS for platforms with different motherboard silkscreens, but a lot of vendors just use the same BIOS on different models, with the same information at "Locator" and "Bank Locator" data at DMI tables, that don't match what's printed at the board's silkscreen. So, GHES ends by exposing wrong data. Also, such BIOS fail to properly expose such knowledge to drivers/userspace. On the discussions I had with HP, back in 2012, the idea was to try to have some sort of way for the GHES driver to query the BIOS on a reliable way, in order to get its layout, in a way that tools like ras-mc-ctl would properly report the memory configuration (with --layout) and the motherboard silkscreen labels (with --print-labels). Unfortunately, at least on that time, the discussions with HP didn't proceed. Thanks, Mauro -- To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, 2017-07-20 at 17:15 -0300, Mauro Carvalho Chehab wrote: > Em Thu, 20 Jul 2017 19:50:03 +0000 > "Kani, Toshimitsu" <toshi.kani@hpe.com> escreveu: : > > Firmware has better knowledge about the platform and can provide > > better RAS when implemented properly. I agree that user > > experiences may vary on platforms. > > It may have a better knowledge, when the vendor ships different BIOS > for platforms with different motherboard silkscreens, but a lot of > vendors just use the same BIOS on different models, with the same > information at "Locator" and "Bank Locator" data at DMI tables, > that don't match what's printed at the board's silkscreen. > > So, GHES ends by exposing wrong data. Also, such BIOS fail > to properly expose such knowledge to drivers/userspace. I see. Yeah, I can see such problems could be overlooked since normal tests run just fine even if there is a mismatch in such info... > On the discussions I had with HP, back in 2012, the idea was to try > to have some sort of way for the GHES driver to query the BIOS > on a reliable way, in order to get its layout, in a way > that tools like ras-mc-ctl would properly report the memory > configuration (with --layout) and the motherboard silkscreen > labels (with --print-labels). Unfortunately, at least on that > time, the discussions with HP didn't proceed. Thanks for the info. I hope we can enable it this time around. -Toshi
On Wed, Jul 19, 2017 at 01:02:45PM -0300, Mauro Carvalho Chehab wrote: > Tony/Aris, > > I got yesterday an HP ML350 G9, equipped with Sandy Bridge EP CPUs (E5-2640v4). > I'm running Kernel 4.11 there. > > AFAIKT, Sandy Bridge EP has 4 channels per memory controller, right? > That would match the number of memory slots on this machine (24 slots). > > Yet, EDAC is only identifying 3 channels per CPU: > > $ ras-mc-ctl --layout > +-----------------------------------------------------------------------+ > | mc0 | mc1 | > | channel0 | channel1 | channel2 | channel0 | channel1 | channel2 | > -------+-----------------------------------------------------------------------+ > slot2: | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | > slot1: | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | > slot0: | 16384 MB | 0 MB | 16384 MB | 16384 MB | 0 MB | 16384 MB | > -------+---------------------------------------------------------------------------+ > > So, it seems that either the BIOS is hidden the other channel or > there's something wrong with SandyBridge EP support at sb_edac driver. Does lspci show all four of these devices? include/linux/pci_ids.h:#define PCI_DEVICE_ID_INTEL_SBRIDGE_IMC_TAD0 0x3caa /* 15.2 */ include/linux/pci_ids.h:#define PCI_DEVICE_ID_INTEL_SBRIDGE_IMC_TAD1 0x3cab /* 15.3 */ include/linux/pci_ids.h:#define PCI_DEVICE_ID_INTEL_SBRIDGE_IMC_TAD2 0x3cac /* 15.4 */ include/linux/pci_ids.h:#define PCI_DEVICE_ID_INTEL_SBRIDGE_IMC_TAD3 0x3cad /* 15.5 */ There should be two of each (one on bus 7f, the other on bus ff). -Tony -- To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Em Thu, 20 Jul 2017 14:15:54 -0700 "Luck, Tony" <tony.luck@intel.com> escreveu: > On Wed, Jul 19, 2017 at 01:02:45PM -0300, Mauro Carvalho Chehab wrote: > > Tony/Aris, > > > > I got yesterday an HP ML350 G9, equipped with Sandy Bridge EP CPUs (E5-2640v4). > > I'm running Kernel 4.11 there. > > > > AFAIKT, Sandy Bridge EP has 4 channels per memory controller, right? > > That would match the number of memory slots on this machine (24 slots). > > > > Yet, EDAC is only identifying 3 channels per CPU: > > > > $ ras-mc-ctl --layout > > +-----------------------------------------------------------------------+ > > | mc0 | mc1 | > > | channel0 | channel1 | channel2 | channel0 | channel1 | channel2 | > > -------+-----------------------------------------------------------------------+ > > slot2: | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | > > slot1: | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | 0 MB | > > slot0: | 16384 MB | 0 MB | 16384 MB | 16384 MB | 0 MB | 16384 MB | > > -------+---------------------------------------------------------------------------+ > > > > So, it seems that either the BIOS is hidden the other channel or > > there's something wrong with SandyBridge EP support at sb_edac driver. > > Does lspci show all four of these devices? > > include/linux/pci_ids.h:#define PCI_DEVICE_ID_INTEL_SBRIDGE_IMC_TAD0 0x3caa /* 15.2 */ > include/linux/pci_ids.h:#define PCI_DEVICE_ID_INTEL_SBRIDGE_IMC_TAD1 0x3cab /* 15.3 */ > include/linux/pci_ids.h:#define PCI_DEVICE_ID_INTEL_SBRIDGE_IMC_TAD2 0x3cac /* 15.4 */ > include/linux/pci_ids.h:#define PCI_DEVICE_ID_INTEL_SBRIDGE_IMC_TAD3 0x3cad /* 15.5 */ > > There should be two of each (one on bus 7f, the other on bus ff). It is getting all 4 TAD devices (Broadwell). This is what I'm getting (from the PCI IDs that it is supposed to be on Broadwell): 00:05.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Map/VTd_Misc/System Management [8086:6f28] (rev 01) 7f:0f.4 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent [8086:6ffc] (rev 01) 7f:0f.5 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent [8086:6ffd] (rev 01) 7f:12.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Home Agent 0 [8086:6fa0] (rev 01) 7f:13.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Target Address/Thermal/RAS [8086:6fa8] (rev 01) 7f:13.1 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Target Address/Thermal/RAS [8086:6f71] (rev 01) 7f:13.2 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder [8086:6faa] (rev 01) 7f:13.3 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder [8086:6fab] (rev 01) 7f:13.4 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder [8086:6fac] (rev 01) 7f:13.5 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder [8086:6fad] (rev 01) 7f:13.7 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Global Broadcast [8086:6faf] (rev 01) 7f:16.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Target Address/Thermal/RAS [8086:6f68] (rev 01) 80:05.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Map/VTd_Misc/System Management [8086:6f28] (rev 01) ff:0f.4 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent [8086:6ffc] (rev 01) ff:0f.5 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Caching Agent [8086:6ffd] (rev 01) ff:12.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Home Agent 0 [8086:6fa0] (rev 01) ff:13.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Target Address/Thermal/RAS [8086:6fa8] (rev 01) ff:13.1 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Target Address/Thermal/RAS [8086:6f71] (rev 01) ff:13.2 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder [8086:6faa] (rev 01) ff:13.3 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder [8086:6fab] (rev 01) ff:13.4 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder [8086:6fac] (rev 01) ff:13.5 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Memory Controller 0 - Channel Target Address Decoder [8086:6fad] (rev 01) ff:13.7 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D DDRIO Global Broadcast [8086:6faf] (rev 01) ff:16.0 System peripheral [0880]: Intel Corporation Xeon E7 v4/Xeon E5 v4/Xeon E3 v4/Xeon D Target Address/Thermal/RAS [8086:6f68] (rev 01) Thanks, Mauro -- To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, Jul 20, 2017 at 07:50:03PM +0000, Kani, Toshimitsu wrote: > GHES / firmware-first still requires OS recovery actions when an error > cannot be corrected by the platform. They are handled by ghes_proc(), > and ghes_edac remains its error-reporting wrapper. I mean all the recovery actions the firmware does because it gets to see the error first. Otherwise, Firmware First is the the dumbest repeater layer in the history of layers. > Firmware has better knowledge about the platform and can provide better > RAS when implemented properly. s/when/if/
Em Fri, 21 Jul 2017 15:34:41 +0200 Borislav Petkov <bp@alien8.de> escreveu: > On Thu, Jul 20, 2017 at 07:50:03PM +0000, Kani, Toshimitsu wrote: > > GHES / firmware-first still requires OS recovery actions when an error > > cannot be corrected by the platform. They are handled by ghes_proc(), > > and ghes_edac remains its error-reporting wrapper. What happens when the error can be corrected? Does it still report it to userspace, or just silently hide the error? If I remember well about a past discussion with some vendor, I was told that the firmware can hide some errors from being reported. Is it still the case? Thanks, Mauro -- To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Fri, Jul 21, 2017 at 10:40:01AM -0300, Mauro Carvalho Chehab wrote: > What happens when the error can be corrected? Does it still report it to > userspace, or just silently hide the error? > > If I remember well about a past discussion with some vendor, I was told > that the firmware can hide some errors from being reported. Is it > still the case? I've heard the same thing but I have no idea what they're actually doing. But it would make sense because the intention is not to worry users unnecessarily if it can hide the error and if there are no adverse consequences from it.
On Fri, 2017-07-21 at 15:47 +0200, Borislav Petkov wrote: > On Fri, Jul 21, 2017 at 10:40:01AM -0300, Mauro Carvalho Chehab > wrote: > > What happens when the error can be corrected? Does it still report > > it to userspace, or just silently hide the error? > > > > If I remember well about a past discussion with some vendor, I was > > told that the firmware can hide some errors from being reported. Is > > it still the case? > > I've heard the same thing but I have no idea what they're actually > doing. But it would make sense because the intention is not to worry > users unnecessarily if it can hide the error and if there are no > adverse consequences from it. Yes, that is correct. Corrected errors are reported to the OS when they exceeded the platform's threshold. Thanks, -Toshi
On Fri, Jul 21, 2017 at 03:08:41PM +0000, Kani, Toshimitsu wrote: > Yes, that is correct. Corrected errors are reported to the OS when > they exceeded the platform's threshold. Are those thresholds user-configurable? If not, what are you telling users who want to see *every* corrected error for measuring DIMM wear and so on...?
T24gRnJpLCAyMDE3LTA3LTIxIGF0IDE3OjEzICswMjAwLCBCb3Jpc2xhdiBQZXRrb3Ygd3JvdGU6 DQo+IE9uIEZyaSwgSnVsIDIxLCAyMDE3IGF0IDAzOjA4OjQxUE0gKzAwMDAsIEthbmksIFRvc2hp bWl0c3Ugd3JvdGU6DQo+ID4gWWVzLCB0aGF0IGlzIGNvcnJlY3QuwqDCoENvcnJlY3RlZCBlcnJv cnMgYXJlIHJlcG9ydGVkIHRvIHRoZSBPUyB3aGVuDQo+ID4gdGhleSBleGNlZWRlZCB0aGUgcGxh dGZvcm0ncyB0aHJlc2hvbGQuDQo+IA0KPiBBcmUgdGhvc2UgdGhyZXNob2xkcyB1c2VyLWNvbmZp Z3VyYWJsZT8NCg0KSSBzdXBwb3NlIGl0J2QgZGVwZW5kIG9uIHZlbmRvcnMsIGJ1dCBJIGRvIG5v dCB0aGluayB1c2VycyBjYW4gZG8gaXQNCnByb3Blcmx5IHVubGVzcyB0aGV5IGhhdmUgZGVwdGgg a25vd2xlZGdlIGFib3V0IHRoZSBoYXJkd2FyZS4NCg0KPiBJZiBub3QsIHdoYXQgYXJlIHlvdSB0 ZWxsaW5nIHVzZXJzIHdobyB3YW50IHRvIHNlZSAqZXZlcnkqIGNvcnJlY3RlZA0KPiBlcnJvciBm b3IgbWVhc3VyaW5nIERJTU0gd2VhciBhbmQgc28gb24uLi4/DQoNCkNvcnJlY3RlZCBlcnJvcnMg YXJlIG5vcm1hbCBhbmQgZXhwZWN0ZWQgdG8gb2NjdXIgb24gaGVhbHRoeSBoYXJkd2FyZS4gDQpU aGV5IGRvIG5vdCBuZWVkIHVzZXIncyBhdHRlbnRpb24gdW50aWwgdGhleSByZXBlYXRlZGx5IG9j Y3VycmVkIGF0IGENCnNhbWUgcGxhY2UuDQoNClRoYW5rcywNCi1Ub3NoaQ0K -- To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Em Fri, 21 Jul 2017 15:34:50 +0000 "Kani, Toshimitsu" <toshi.kani@hpe.com> escreveu: > On Fri, 2017-07-21 at 17:13 +0200, Borislav Petkov wrote: > > On Fri, Jul 21, 2017 at 03:08:41PM +0000, Kani, Toshimitsu wrote: > > > Yes, that is correct. Corrected errors are reported to the OS when > > > they exceeded the platform's threshold. > > > > Are those thresholds user-configurable? > > I suppose it'd depend on vendors, but I do not think users can do it > properly unless they have depth knowledge about the hardware. > > > If not, what are you telling users who want to see *every* corrected > > error for measuring DIMM wear and so on...? > > Corrected errors are normal and expected to occur on healthy hardware. > They do not need user's attention until they repeatedly occurred at a > same place. Yes, they're expected to happen. Still, some sys admins have their own measurements about what's "normal" for their scenario, and want to monitor every single corrected error, running their own algorithm to warn if the number of corrected errors is above their "normal" rate. Thanks, Mauro -- To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Fri, Jul 21, 2017 at 03:34:50PM +0000, Kani, Toshimitsu wrote: > I suppose it'd depend on vendors, but I do not think users can do it > properly unless they have depth knowledge about the hardware. I'm talking about a menu in the BIOS where you can set the thresholding levels on the system. Does your BIOS have that? > Corrected errors are normal and expected to occur on healthy hardware. > They do not need user's attention until they repeatedly occurred at a > same place. Apparently, you haven't been on enough maintanance calls, trying to calm down the customer about the hardware error he sees in his logs...
On Fri, 2017-07-21 at 17:53 +0200, Borislav Petkov wrote: > On Fri, Jul 21, 2017 at 03:34:50PM +0000, Kani, Toshimitsu wrote: > > I suppose it'd depend on vendors, but I do not think users can do > > it properly unless they have depth knowledge about the hardware. > > I'm talking about a menu in the BIOS where you can set the > thresholding levels on the system. Does your BIOS have that? No, we don't offer such settings. > > Corrected errors are normal and expected to occur on healthy > > hardware. They do not need user's attention until they repeatedly > > occurred at a same place. > > Apparently, you haven't been on enough maintanance calls, trying to > calm down the customer about the hardware error he sees in his > logs... Actually, that's why. Reporting all corrected errors make users worried, call support, and asking to replace healthy hardware... Thanks, -Toshi
On Fri, 2017-07-21 at 12:44 -0300, Mauro Carvalho Chehab wrote: > Em Fri, 21 Jul 2017 15:34:50 +0000 > "Kani, Toshimitsu" <toshi.kani@hpe.com> escreveu: > > > On Fri, 2017-07-21 at 17:13 +0200, Borislav Petkov wrote: > > > On Fri, Jul 21, 2017 at 03:08:41PM +0000, Kani, Toshimitsu > > > wrote: > > > > Yes, that is correct. Corrected errors are reported to the OS > > > > when they exceeded the platform's threshold. > > > > > > Are those thresholds user-configurable? > > > > I suppose it'd depend on vendors, but I do not think users can do > > it properly unless they have depth knowledge about the hardware. > > > > > If not, what are you telling users who want to see *every* > > > corrected error for measuring DIMM wear and so on...? > > > > Corrected errors are normal and expected to occur on healthy > > hardware. They do not need user's attention until they repeatedly > > occurred at a same place. > > Yes, they're expected to happen. Still, some sys admins have their > own measurements about what's "normal" for their scenario, and want > to monitor every single corrected error, running their own > algorithm to warn if the number of corrected errors is above their > "normal" rate. I suppose these admins had to do it because their platforms reported all corrected errors. It addresses such administrators' burden. Thanks, -Toshi
Hmmm so the BIOS isn't hiding any devices. Can you read the MTR registers from each of those target address decoders? for i in a b d c do for j in 0 4 8 do setpci -d 8086:6fa$i 0x8$j.L done done bit 14 is the IS_DIMM_PRESENT one. So you should see values like 001c5050 for populated slots. I see 000f000c for empty slots. -Tony [If you send me "lspci -xxxx" output I can check why the driver isn't reporting the 4th channel] -- To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Em Fri, 21 Jul 2017 16:40:20 +0000 "Kani, Toshimitsu" <toshi.kani@hpe.com> escreveu: > On Fri, 2017-07-21 at 12:44 -0300, Mauro Carvalho Chehab wrote: > > Em Fri, 21 Jul 2017 15:34:50 +0000 > > "Kani, Toshimitsu" <toshi.kani@hpe.com> escreveu: > > > > > On Fri, 2017-07-21 at 17:13 +0200, Borislav Petkov wrote: > > > > On Fri, Jul 21, 2017 at 03:08:41PM +0000, Kani, Toshimitsu > > > > wrote: > > > > > Yes, that is correct. Corrected errors are reported to the OS > > > > > when they exceeded the platform's threshold. > > > > > > > > Are those thresholds user-configurable? > > > > > > I suppose it'd depend on vendors, but I do not think users can do > > > it properly unless they have depth knowledge about the hardware. > > > > > > > If not, what are you telling users who want to see *every* > > > > corrected error for measuring DIMM wear and so on...? > > > > > > Corrected errors are normal and expected to occur on healthy > > > hardware. They do not need user's attention until they repeatedly > > > occurred at a same place. > > > > Yes, they're expected to happen. Still, some sys admins have their > > own measurements about what's "normal" for their scenario, and want > > to monitor every single corrected error, running their own > > algorithm to warn if the number of corrected errors is above their > > "normal" rate. > > I suppose these admins had to do it because their platforms reported > all corrected errors. It addresses such administrators' burden. I see the value of having a threshold in BIOS, provided that it is well documented, and whose value can be adjusted, if needed. One of the things I wanted to implement in ras-daemon were an algorithm that would be doing such threshold in software. The problem is that it would require field experience. So, I talked with a few vendors, to see if they could help doing it, but, on that time, none rised their hands :-) The thing with a BIOS threshold is that the user has no way to audit the algorithm. So, when BIOS start reporting such errors, it may be already too late: the systems may be in the verge of losing data (or some data was already lost). That's critical on cluster systems with thousands of machines: while the impact of disabling a cluster node to do some maintainance is marginal, the impact of an uncorrected error on a single machine may compromise weeks of expensive processing. That's why some users prefer to monitor every single corrected error, and compare with the probability distribution they know that the risk of uncorrected errors is acceptable. Thanks, Mauro -- To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
T24gRnJpLCAyMDE3LTA3LTIxIGF0IDE0OjAxIC0wMzAwLCBNYXVybyBDYXJ2YWxobyBDaGVoYWIg d3JvdGU6DQo+IEVtIEZyaSwgMjEgSnVsIDIwMTcgMTY6NDA6MjAgKzAwMDANCj4gIkthbmksIFRv c2hpbWl0c3UiIDx0b3NoaS5rYW5pQGhwZS5jb20+IGVzY3JldmV1Og0KPiANCj4gPiBPbiBGcmks IDIwMTctMDctMjEgYXQgMTI6NDQgLTAzMDAsIE1hdXJvIENhcnZhbGhvIENoZWhhYiB3cm90ZToN Cj4gPiA+IEVtIEZyaSwgMjEgSnVsIDIwMTcgMTU6MzQ6NTAgKzAwMDANCj4gPiA+ICJLYW5pLCBU b3NoaW1pdHN1IiA8dG9zaGkua2FuaUBocGUuY29tPiBlc2NyZXZldToNCj4gPiA+IMKgwqANCj4g PiA+ID4gT24gRnJpLCAyMDE3LTA3LTIxIGF0IDE3OjEzICswMjAwLCBCb3Jpc2xhdiBQZXRrb3Yg d3JvdGU6wqDCoA0KPiA+ID4gPiA+IE9uIEZyaSwgSnVsIDIxLCAyMDE3IGF0IDAzOjA4OjQxUE0g KzAwMDAsIEthbmksIFRvc2hpbWl0c3UNCj4gPiA+ID4gPiB3cm90ZTrCoMKgwqDCoA0KPiA+ID4g PiA+ID4gWWVzLCB0aGF0IGlzIGNvcnJlY3QuwqDCoENvcnJlY3RlZCBlcnJvcnMgYXJlIHJlcG9y dGVkIHRvIHRoZQ0KPiA+ID4gPiA+ID4gT1Mgd2hlbiB0aGV5IGV4Y2VlZGVkIHRoZSBwbGF0Zm9y bSdzIHRocmVzaG9sZC7CoMKgwqDCoA0KPiA+ID4gPiA+IA0KPiA+ID4gPiA+IEFyZSB0aG9zZSB0 aHJlc2hvbGRzIHVzZXItY29uZmlndXJhYmxlP8KgwqDCoMKgDQo+ID4gPiA+IA0KPiA+ID4gPiBJ IHN1cHBvc2UgaXQnZCBkZXBlbmQgb24gdmVuZG9ycywgYnV0IEkgZG8gbm90IHRoaW5rIHVzZXJz IGNhbg0KPiA+ID4gPiBkbyBpdCBwcm9wZXJseSB1bmxlc3MgdGhleSBoYXZlIGRlcHRoIGtub3ds ZWRnZSBhYm91dCB0aGUNCj4gPiA+ID4gaGFyZHdhcmUuDQo+ID4gPiA+IMKgwqANCj4gPiA+ID4g PiBJZiBub3QsIHdoYXQgYXJlIHlvdSB0ZWxsaW5nIHVzZXJzIHdobyB3YW50IHRvIHNlZSAqZXZl cnkqDQo+ID4gPiA+ID4gY29ycmVjdGVkIGVycm9yIGZvciBtZWFzdXJpbmcgRElNTSB3ZWFyIGFu ZCBzbyBvbi4uLj/CoMKgwqDCoA0KPiA+ID4gPiANCj4gPiA+ID4gQ29ycmVjdGVkIGVycm9ycyBh cmUgbm9ybWFsIGFuZCBleHBlY3RlZCB0byBvY2N1ciBvbiBoZWFsdGh5DQo+ID4gPiA+IGhhcmR3 YXJlLsKgwqBUaGV5IGRvIG5vdCBuZWVkIHVzZXIncyBhdHRlbnRpb24gdW50aWwgdGhleQ0KPiA+ ID4gPiByZXBlYXRlZGx5IG9jY3VycmVkIGF0IGEgc2FtZSBwbGFjZS7CoMKgDQo+ID4gPiANCj4g PiA+IFllcywgdGhleSdyZSBleHBlY3RlZCB0byBoYXBwZW4uIFN0aWxsLCBzb21lIHN5cyBhZG1p bnMgaGF2ZQ0KPiA+ID4gdGhlaXIgb3duIG1lYXN1cmVtZW50cyBhYm91dCB3aGF0J3MgIm5vcm1h bCIgZm9yIHRoZWlyIHNjZW5hcmlvLA0KPiA+ID4gYW5kIHdhbnQgdG8gbW9uaXRvciBldmVyeSBz aW5nbGUgY29ycmVjdGVkIGVycm9yLCBydW5uaW5nIHRoZWlyDQo+ID4gPiBvd24gYWxnb3JpdGht IHRvIHdhcm4gaWYgdGhlIG51bWJlciBvZiBjb3JyZWN0ZWQgZXJyb3JzIGlzIGFib3ZlDQo+ID4g PiB0aGVpciAibm9ybWFsIiByYXRlLsKgwqANCj4gPiANCj4gPiBJIHN1cHBvc2UgdGhlc2UgYWRt aW5zIGhhZCB0byBkbyBpdCBiZWNhdXNlIHRoZWlyIHBsYXRmb3Jtcw0KPiA+IHJlcG9ydGVkIGFs bCBjb3JyZWN0ZWQgZXJyb3JzLsKgwqBJdCBhZGRyZXNzZXMgc3VjaCBhZG1pbmlzdHJhdG9ycycN Cj4gPiBidXJkZW4uDQo+IA0KPiBJIHNlZSB0aGUgdmFsdWUgb2YgaGF2aW5nIGEgdGhyZXNob2xk IGluIEJJT1MsIHByb3ZpZGVkIHRoYXQgaXQgaXMNCj4gd2VsbCBkb2N1bWVudGVkLCBhbmQgd2hv c2UgdmFsdWUgY2FuIGJlIGFkanVzdGVkLCBpZiBuZWVkZWQuDQo+IA0KPiBPbmUgb2YgdGhlIHRo aW5ncyBJIHdhbnRlZCB0byBpbXBsZW1lbnQgaW4gcmFzLWRhZW1vbiB3ZXJlIGFuDQo+IGFsZ29y aXRobSB0aGF0IHdvdWxkIGJlIGRvaW5nIHN1Y2ggdGhyZXNob2xkIGluIHNvZnR3YXJlLg0KPiBU aGUgcHJvYmxlbSBpcyB0aGF0IGl0IHdvdWxkIHJlcXVpcmUgZmllbGQgZXhwZXJpZW5jZS4gU28s DQo+IEkgdGFsa2VkIHdpdGggYSBmZXcgdmVuZG9ycywgdG8gc2VlIGlmIHRoZXkgY291bGQgaGVs cCBkb2luZw0KPiBpdCwgYnV0LCBvbiB0aGF0IHRpbWUsIG5vbmUgcmlzZWQgdGhlaXIgaGFuZHMg Oi0pDQoNCkkgdGhpbmsgaXQnZCBiZSB2ZXJ5IGhhcmQgdG8ga2VlcCBpdCB1cCB0byBkYXRlLg0K DQo+IFRoZSB0aGluZyB3aXRoIGEgQklPUyB0aHJlc2hvbGQgaXMgdGhhdCB0aGUgdXNlciBoYXMg bm8gd2F5IHRvDQo+IGF1ZGl0IHRoZSBhbGdvcml0aG0uIFNvLCB3aGVuIEJJT1Mgc3RhcnQgcmVw b3J0aW5nIHN1Y2ggZXJyb3JzLA0KPiBpdCBtYXkgYmUgYWxyZWFkeSB0b28gbGF0ZTogdGhlIHN5 c3RlbXMgbWF5IGJlIGluIHRoZSB2ZXJnZSBvZsKgDQo+IGxvc2luZyBkYXRhIChvciBzb21lIGRh dGEgd2FzIGFscmVhZHkgbG9zdCkuDQo+IA0KPiBUaGF0J3MgY3JpdGljYWwgb24gY2x1c3RlciBz eXN0ZW1zIHdpdGggdGhvdXNhbmRzIG9mIG1hY2hpbmVzOg0KPiB3aGlsZSB0aGUgaW1wYWN0IG9m IGRpc2FibGluZyBhIGNsdXN0ZXIgbm9kZSB0byBkbyBzb21lIG1haW50YWluYW5jZQ0KPiBpcyBt YXJnaW5hbCwgdGhlIGltcGFjdCBvZiBhbiB1bmNvcnJlY3RlZCBlcnJvciBvbiBhIHNpbmdsZQ0K PiBtYWNoaW5lIG1heSBjb21wcm9taXNlIHdlZWtzIG9mIGV4cGVuc2l2ZSBwcm9jZXNzaW5nLg0K PiANCj4gVGhhdCdzIHdoeSBzb21lIHVzZXJzIHByZWZlciB0byBtb25pdG9yIGV2ZXJ5IHNpbmds ZSBjb3JyZWN0ZWQNCj4gZXJyb3IsIGFuZCBjb21wYXJlIHdpdGggdGhlIHByb2JhYmlsaXR5IGRp c3RyaWJ1dGlvbiB0aGV5DQo+IGtub3cgdGhhdCB0aGUgcmlzayBvZiB1bmNvcnJlY3RlZCBlcnJv cnMgaXMgYWNjZXB0YWJsZS4NCg0KUmlnaHQsIEkgZG8gbm90IHRoaW5rIGFsbCBwbGF0Zm9ybXMg bmVlZCB0byBiZSBmaXJtd2FyZS1maXJzdC4gIEkgZG8NCm5vdCB3YW50IHRvIHRhbGsgbGlrZSBh IHNhbGUncyBwZXJzb24sIGJ1dCB3ZSBhbHNvIG9mZmVyIGxvd2VyLWNvc3QNCnBsYXRmb3JtcyB0 aGF0IGRvIG5vdCBjb21lIHdpdGggYnVpbHQtaW4gUkFTLiAgVXNlcnMgY2FuIGNob29zZSBhIHJp Z2h0DQptb2RlbCBmb3IgdGhlaXIgbmVlZHMuDQoNClRoYW5rcywNCi1Ub3NoaQ0K -- To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Fri, Jul 21, 2017 at 02:01:31PM -0300, Mauro Carvalho Chehab wrote: > I see the value of having a threshold in BIOS, provided that it is > well documented, and whose value can be adjusted, if needed. > > One of the things I wanted to implement in ras-daemon were an > algorithm that would be doing such threshold in software. We have that now in the kernel: drivers/ras/cec.c We did it exactly for that purpose - not upsetting users unnecessarily. > The thing with a BIOS threshold is that the user has no way to > audit the algorithm. So, when BIOS start reporting such errors, > it may be already too late: the systems may be in the verge of > losing data (or some data was already lost). Not only that: thresholds depend on the DIMM types which means, BIOS must know what DIMM types are in there which I doubt. So exposing that to configuration instead of "deciding" for people would be better. > That's critical on cluster systems with thousands of machines: > while the impact of disabling a cluster node to do some maintainance > is marginal, the impact of an uncorrected error on a single > machine may compromise weeks of expensive processing. > > That's why some users prefer to monitor every single corrected > error, and compare with the probability distribution they > know that the risk of uncorrected errors is acceptable. Yap, you need to have stuff like that configurable - BIOS can't predict all possible use cases.
On Fri, 2017-07-21 at 19:23 +0200, Borislav Petkov wrote: : > Not only that: thresholds depend on the DIMM types which means, BIOS > must know what DIMM types are in there which I doubt. BIOS knows DIMM model from the SPD data. > So exposing that to configuration instead of "deciding" for people > would be better. Enterprise platforms have very different model (I do not say it's better for everyone from the cost perspective). Typically, such platform vendors work with DIMM vendors directly to come with their supported DIMMs with own part numbers, which are certified for the platforms with extensive validation testings. Thanks, -Toshi
On Fri, Jul 21, 2017 at 06:38:52PM +0000, Kani, Toshimitsu wrote: > Enterprise platforms have very different model (I do not say it's > better for everyone from the cost perspective). Typically, such But you do tell your customers that the error counts they see are not really what *actually* happens, right?
T24gU2F0LCAyMDE3LTA3LTIyIGF0IDA4OjI4ICswMjAwLCBCb3Jpc2xhdiBQZXRrb3Ygd3JvdGU6 DQo+IE9uIEZyaSwgSnVsIDIxLCAyMDE3IGF0IDA2OjM4OjUyUE0gKzAwMDAsIEthbmksIFRvc2hp bWl0c3Ugd3JvdGU6DQo+ID4gRW50ZXJwcmlzZSBwbGF0Zm9ybXMgaGF2ZSB2ZXJ5IGRpZmZlcmVu dCBtb2RlbCAoSSBkbyBub3Qgc2F5IGl0J3MNCj4gPiBiZXR0ZXIgZm9yIGV2ZXJ5b25lIGZyb20g dGhlIGNvc3QgcGVyc3BlY3RpdmUpLsKgwqBUeXBpY2FsbHksIHN1Y2gNCj4gDQo+IEJ1dCB5b3Ug ZG8gdGVsbCB5b3VyIGN1c3RvbWVycyB0aGF0IHRoZSBlcnJvciBjb3VudHMgdGhleSBzZWUgYXJl IG5vdA0KPiByZWFsbHkgd2hhdCAqYWN0dWFsbHkqIGhhcHBlbnMsIHJpZ2h0Pw0KDQpXZSBkbyBu b3QgdGVsbCB0aGUgZXJyb3IgY291bnRzIHRvIGN1c3RvbWVycy4gIFdlIHRlbGwgY3VzdG9tZXJz IHdoZW4NCnRoZXkgbmVlZCBhdHRlbnRpb24gYW5kIGhhdmUgYWN0aW9uYWJsZSBpdGVtcywgYW5k IHdlIHByb3ZpZGUgc3VwcG9ydA0KZm9yIHRoYXQuICBTdXBwb3J0IGdldHMgYWxsIGluZm8gbmVj ZXNzYXJ5Lg0KDQpUaGVyZSBhcmUgbXVsdGlwbGUgbW9kZWxzIGZvciBtdWx0aXBsZSB0eXBlcyBv ZiBjdXN0b21lcnMuICBJIGFtIG5vdA0Kc2F5aW5nIG9uZSBtb2RlbCBpcyBiZXR0ZXIgdGhhbiB0 aGUgb3RoZXIuDQoNClRoYW5rcywNCi1Ub3NoaQ0K -- To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Mon, Jul 24, 2017 at 02:49:30PM +0000, Kani, Toshimitsu wrote: > We do not tell the error counts to customers. Please read what I said: do you tell your customers that the error counts they're seeing (or are *not* seeing) is bogus because the BIOS is hiding them? Not the *actual* numbers! > We tell customers when they need attention and have actionable items, > and we provide support for that. Support gets all info necessary. Ok, good to know. I'll make sure to bounce such issues to you guys in the future.
On Mon, 2017-07-24 at 17:04 +0200, Borislav Petkov wrote: > On Mon, Jul 24, 2017 at 02:49:30PM +0000, Kani, Toshimitsu wrote: > > We do not tell the error counts to customers. > > Please read what I said: do you tell your customers that the error > counts they're seeing (or are *not* seeing) is bogus because the BIOS > is hiding them? Not the *actual* numbers! Customers do not see error counts. I do not think it's bogus. This model is basically the same as your car. You do not see error counts or periodical normal errors from all kinds of controllers in the car while you are driving. You get an attention lamp lit when you need to bring it to a car dealer. > > We tell customers when they need attention and have actionable > > items, and we provide support for that. Support gets all info > > necessary. > > Ok, good to know. I'll make sure to bounce such issues to you guys in > the future. We've been providing this model for many years now. I am just trying to enable OS error reporting with ghes_edac. Thanks, -Toshi
On Mon, Jul 24, 2017 at 03:25:34PM +0000, Kani, Toshimitsu wrote: > Customers do not see error counts. I do not think it's bogus. Not showing the real error error counts but something contrived is the definition of bogus numbers. But you're not showing anything - only when some thresholds are being hit. > This model is basically the same as your car. You do not see error Oh jeez, we're talking about cars now. > We've been providing this model for many years now. Dude, relax, I'm only trying to point out to you that there are customers who want to see *every* error and thus track how their hardware behaves. And that for those customers it is probably worth considering exposing that info and providing a switch to disable that dumbing of the RAS functionality in the BIOS so that people can decide for themselves. That's all. I'm not questioning your model - I'm just saying that it could be improved for certain customers. Do me a favor and this time *actually* *read* my reply. > I am just trying to enable OS error reporting with ghes_edac. I know, you don't have to state the obvious constantly.
On Mon, 2017-07-24 at 17:37 +0200, Borislav Petkov wrote: > On Mon, Jul 24, 2017 at 03:25:34PM +0000, Kani, Toshimitsu wrote: : > > > We've been providing this model for many years now. > > Dude, relax, I'm only trying to point out to you that there are > customers who want to see *every* error and thus track how their > hardware behaves. And that for those customers it is probably worth > considering exposing that info and providing a switch to disable that > dumbing of the RAS functionality in the BIOS so that people can > decide for themselves. That's all. Yes, Mauro has already pointed this out. As I replied to him, we do have a separate series of platforms that do not have built-in RAS, and report all errors. Such customers can simply choose them. They do not need to pay for built-in RAS. The model w/ built-in RAS provides warranty & full support. As I said, it's a different model. Thanks, -Toshi
Em Mon, 24 Jul 2017 17:37:16 +0200 Borislav Petkov <bp@alien8.de> escreveu: > > Customers do not see error counts. I do not think it's bogus. > > I am just trying to enable OS error reporting with ghes_edac. > > I know, you don't have to state the obvious constantly. The problem I see is that, currently, on users that have EDAC already enabled, the users gets the errors directly from the hardware. If the Kernel force those users to use ghes_edac by default, they they won't see the error counts anymore, but, instead, hardware reports that the memories need to be replaced. Well, if such users are handling thresholds themselves, they won't see those errors anymore, as the errors will be masked. That's a regression. So, the right solution would be to keep hardware first, but providing a modprobe parameter to let them switch to software first. Thanks, Mauro -- To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Mon, Jul 24, 2017 at 03:56:27PM +0000, Kani, Toshimitsu wrote: > Yes, Mauro has already pointed this out. As I replied to him, we do > have a separate series of platforms that do not have built-in RAS, and So this whitelist entry +static struct acpi_oemlist oemlist[] = { + {"HPE ", "Server ", 0, ACPI_SIG_FADT, all_versions}, + { } /* End */ +}; looks like it'll match every HP server platform not only the ones with built-in RAS.
On Mon, Jul 24, 2017 at 01:04:13PM -0300, Mauro Carvalho Chehab wrote: > If the Kernel force those users to use ghes_edac by default, > they they won't see the error counts anymore, but, instead, > hardware reports that the memories need to be replaced. This is exactly why I'm trying to load ghes_edac only on those platforms which would really want it. > So, the right solution would be to keep hardware first, but > providing a modprobe parameter to let them switch to software > first. That's exactly the issue: if we make it spec-conform and adhere to FF setting, then it'll be clean. BUT(!), we will force ghes_edac on those platforms which potentially are using the platform-specific drivers until now. Not good. If we do the whitelisting, then we're stuck with maintaining a yucky whitelist and have to keep updating ghes_edac with it. So we're basically between a rock and a hard place. If I had to choose *right* *now*, I'd probably lean slightly towards the whitelist as it won't break existing users. A big grumpfy-grumbly hmmm. :-\
On Mon, 2017-07-24 at 18:37 +0200, Borislav Petkov wrote: > On Mon, Jul 24, 2017 at 03:56:27PM +0000, Kani, Toshimitsu wrote: > > Yes, Mauro has already pointed this out. As I replied to him, we > > do have a separate series of platforms that do not have built-in > > RAS, and > > So this whitelist entry > > +static struct acpi_oemlist oemlist[] = { > + {"HPE ", "Server ", 0, ACPI_SIG_FADT, all_versions}, > + { } /* End */ > +}; > > looks like it'll match every HP server platform not only the ones > with built-in RAS. I assumed our platforms w/o build-in RAS do not implement GHES, but I will check for sure. Also, all our previous/current platforms have "HP". Thanks, -Toshi
On July 24, 2017 8:44:03 PM GMT+03:00, "Kani, Toshimitsu" <toshi.kani@hpe.com> wrote:
>I assumed our platforms w/o build-in RAS do not implement GHES,
If we make it a normal module, it will be decoupled from GHES and it will rely only on the whitelist to load.
On Mon, 2017-07-24 at 20:50 +0300, Boris Petkov wrote: > On July 24, 2017 8:44:03 PM GMT+03:00, "Kani, Toshimitsu" <toshi.kani > @hpe.com> wrote: > > I assumed our platforms w/o build-in RAS do not implement GHES, > > If we make it a normal module, it will be decoupled from GHES and it > will rely only on the whitelist to load. Umm... I was under impression that we are adding the OSC bit check in addition to the current GHES filtering. Thanks, -Toshi
Em Mon, 24 Jul 2017 15:56:27 +0000 "Kani, Toshimitsu" <toshi.kani@hpe.com> escreveu: > On Mon, 2017-07-24 at 17:37 +0200, Borislav Petkov wrote: > > On Mon, Jul 24, 2017 at 03:25:34PM +0000, Kani, Toshimitsu wrote: > : > > > > > We've been providing this model for many years now. > > > > Dude, relax, I'm only trying to point out to you that there are > > customers who want to see *every* error and thus track how their > > hardware behaves. And that for those customers it is probably worth > > considering exposing that info and providing a switch to disable that > > dumbing of the RAS functionality in the BIOS so that people can > > decide for themselves. That's all. > > Yes, Mauro has already pointed this out. As I replied to him, we do > have a separate series of platforms that do not have built-in RAS, and > report all errors. Such customers can simply choose them. They do not > need to pay for built-in RAS. That's probably too late for me as I received a new HP machine we bought just last week, but for the next time I would need to get a new hardware, what would be the non-RAS equivalent to a ML 350 G9 tower-mounted machine with two Xeon v4 CPUs and iLO? Regards, Mauro > > The model w/ built-in RAS provides warranty & full support. As I said, > it's a different model. > > Thanks, > -Toshi Thanks, Mauro -- To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Em Mon, 24 Jul 2017 18:44:00 +0200 Borislav Petkov <bp@alien8.de> escreveu: > On Mon, Jul 24, 2017 at 01:04:13PM -0300, Mauro Carvalho Chehab wrote: > > If the Kernel force those users to use ghes_edac by default, > > they they won't see the error counts anymore, but, instead, > > hardware reports that the memories need to be replaced. > > This is exactly why I'm trying to load ghes_edac only on those platforms > which would really want it. > > > So, the right solution would be to keep hardware first, but > > providing a modprobe parameter to let them switch to software > > first. > > That's exactly the issue: if we make it spec-conform and adhere to FF > setting, then it'll be clean. BUT(!), we will force ghes_edac on those > platforms which potentially are using the platform-specific drivers > until now. Not good. > > If we do the whitelisting, then we're stuck with maintaining a yucky > whitelist and have to keep updating ghes_edac with it. Yeah, having a whitelist is a maintainership's burden, but, on the other hand, I suspect that there aren't many systems that implement FF, have a reliable BIOS mapping of MB's silkscreen and doesn't filters out corrected errors using some sort of undocumented mechanism. So, I guess it is doable. Another alternative, with, IMO, is better would be to add a parameter like: edac=FF - firmware first; edac=hw - hardware first; edac=auto - honors FF if set in BIOS. Otherwise, hardware first. In order to avoid regressions, and to avoid the need of a whitelist, I would keep "edac=hw" as default. Thanks, Mauro -- To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
T24gTW9uLCAyMDE3LTA3LTI0IGF0IDE0OjU2IC0wMzAwLCBNYXVybyBDYXJ2YWxobyBDaGVoYWIg d3JvdGU6DQo+IEVtIE1vbiwgMjQgSnVsIDIwMTcgMTU6NTY6MjcgKzAwMDANCiA6DQo+IFRoYXQn cyBwcm9iYWJseSB0b28gbGF0ZSBmb3IgbWUgYXMgSSByZWNlaXZlZCBhIG5ldyBIUCBtYWNoaW5l DQo+IHdlIGJvdWdodCBqdXN0IGxhc3Qgd2VlaywgYnV0IGZvciB0aGUgbmV4dCB0aW1lIEkgd291 bGQgbmVlZCB0bw0KPiBnZXQgYSBuZXcgaGFyZHdhcmUsIHdoYXQgd291bGQgYmUgdGhlIG5vbi1S QVMgZXF1aXZhbGVudCB0bw0KPiBhIE1MIDM1MCBHOSB0b3dlci1tb3VudGVkIG1hY2hpbmUgd2l0 aCB0d28gWGVvbiB2NCBDUFVzIGFuZCBpTE8/DQoNClN1Y2ggc2VydmVycyBhcmUgY2FsbGVkICJI UEUgQ2xvdWRsaW5lIi4gIEJ1dCBJIHRoaW5rIHRoZXkgYXJlIGFsbA0KcmFjay1tb3VudGVkLCBu b3QgdG93ZXItbW91bnRlZCBtYWNoaW5lcy4gIEhQIEluYy4gKHdoaWNoIGlzIG5vdyBhDQpzZXBh cmF0ZSBjb21wYW55IGZvciBjb25zdW1lci1vcmllbnRlZCBwcm9kdWN0cykgcHJvYmFibHkgaGFz IHN1Y2gNCm1hY2hpbmUuDQoNClRoYW5rcywNCi1Ub3NoaQ0K -- To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
On Mon, Jul 24, 2017 at 05:54:52PM +0000, Kani, Toshimitsu wrote: > Umm... I was under impression that we are adding the OSC bit check in > addition to the current GHES filtering. Read the parallel subthread again.
(Sending to your other mail address because there's some temporary resolution issue: msmtp: recipient address mchehab@s-opensource.com not accepted by the server msmtp: server message: 451 4.3.0 <mchehab@s-opensource.com>: Temporary lookup failure msmtp: could not send mail (account alien8.de from /home/boris/.msmtprc) Maybe the problem is on my end.) On Mon, Jul 24, 2017 at 03:10:13PM -0300, Mauro Carvalho Chehab wrote: > Yeah, having a whitelist is a maintainership's burden, but, on > the other hand, I suspect that there aren't many systems that > implement FF, have a reliable BIOS mapping of MB's silkscreen > and doesn't filters out corrected errors using some sort of > undocumented mechanism. > > So, I guess it is doable. Right, let's hope. > Another alternative, with, IMO, is better would be to add a parameter like: > > edac=FF - firmware first; > edac=hw - hardware first; > edac=auto - honors FF if set in BIOS. Otherwise, hardware first. Or maybe edac=try_FF or so. But yeah, I guess we'll need something to tell the EDAC core to try FF first. > In order to avoid regressions, and to avoid the need of a whitelist, > I would keep "edac=hw" as default. So I don't want to break existing users and thus make only explicitly known platforms load ghes_edac. In the current case, the HPE machines. All the rest will simply use the platform drivers and nothing will change for them. Later we'll probably need to revisit this decision but right now and with all things considered, the whitelist seems - as ugly as it is - the most workable solution for all the different use cases and machines...
T24gTW9uLCAyMDE3LTA3LTI0IGF0IDIwOjMwICswMjAwLCBCb3Jpc2xhdiBQZXRrb3Ygd3JvdGU6 DQogOg0KPiANCj4gU28gSSBkb24ndCB3YW50IHRvIGJyZWFrIGV4aXN0aW5nIHVzZXJzIGFuZCB0 aHVzIG1ha2Ugb25seSBleHBsaWNpdGx5DQo+IGtub3duIHBsYXRmb3JtcyBsb2FkIGdoZXNfZWRh Yy4gSW4gdGhlIGN1cnJlbnQgY2FzZSwgdGhlIEhQRQ0KPiBtYWNoaW5lcy4gIEFsbCB0aGUgcmVz dCB3aWxsIHNpbXBseSB1c2UgdGhlIHBsYXRmb3JtIGRyaXZlcnMgYW5kDQo+IG5vdGhpbmcgd2ls bCBjaGFuZ2UgZm9yIHRoZW0uDQo+IA0KPiBMYXRlciB3ZSdsbCBwcm9iYWJseSBuZWVkIHRvIHJl dmlzaXQgdGhpcyBkZWNpc2lvbiBidXQgcmlnaHQgbm93IGFuZA0KPiB3aXRoIGFsbCB0aGluZ3Mg Y29uc2lkZXJlZCwgdGhlIHdoaXRlbGlzdCBzZWVtcyAtIGFzIHVnbHkgYXMgaXQgaXMgLQ0KPiB0 aGUgbW9zdCB3b3JrYWJsZSBzb2x1dGlvbiBmb3IgYWxsIHRoZSBkaWZmZXJlbnQgdXNlIGNhc2Vz IGFuZA0KPiBtYWNoaW5lcy4uLg0KDQpBZ3JlZWQuICBJIHdpbGwgdmVyaWZ5IE9FTUlEIGluZm8g b2Ygb3VyIG90aGVyIHBsYXRmb3JtcywgYW5kIGFkZCBBUEVJDQpPU0MgY2hlY2sgYmVmb3JlIGNh bGxpbmcgZ2hlc19lZGFjX3JlZ2lzdGVyKCkuDQoNClRoYW5rcywNCi1Ub3NoaQ0K -- To unsubscribe from this list: send the line "unsubscribe linux-acpi" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
diff --git a/drivers/acpi/apei/ghes.c b/drivers/acpi/apei/ghes.c index d661d452b238..37cd698cacd2 100644 --- a/drivers/acpi/apei/ghes.c +++ b/drivers/acpi/apei/ghes.c @@ -140,6 +140,20 @@ static atomic_t ghes_estatus_cache_alloced; static int ghes_panic_timeout __read_mostly = 30; +static ATOMIC_NOTIFIER_HEAD(ghes_edac_chain); + +void ghes_register_edac_chain(struct notifier_block *nb) +{ + atomic_notifier_chain_register(&ghes_edac_chain, nb); +} +EXPORT_SYMBOL_GPL(ghes_register_edac_chain); + +void ghes_unregister_edac_chain(struct notifier_block *nb) +{ + atomic_notifier_chain_unregister(&ghes_edac_chain, nb); +} +EXPORT_SYMBOL_GPL(ghes_unregister_edac_chain); + static int ghes_ioremap_init(void) { ghes_ioremap_area = __get_vm_area(PAGE_SIZE * GHES_IOREMAP_PAGES, @@ -461,11 +475,11 @@ static void ghes_handle_memory_failure(struct acpi_hest_generic_data *gdata, int static void ghes_do_proc(struct ghes *ghes, const struct acpi_hest_generic_status *estatus) { - int sev, sec_sev; struct acpi_hest_generic_data *gdata; guid_t *sec_type; guid_t *fru_id = &NULL_UUID_LE; char *fru_text = ""; + int sev, sec_sev; sev = ghes_severity(estatus->error_severity); apei_estatus_for_each_section(estatus, gdata) { @@ -480,7 +494,8 @@ static void ghes_do_proc(struct ghes *ghes, if (guid_equal(sec_type, &CPER_SEC_PLATFORM_MEM)) { struct cper_sec_mem_err *mem_err = acpi_hest_get_payload(gdata); - ghes_edac_report_mem_error(ghes, sev, mem_err); + + atomic_notifier_call_chain(&ghes_edac_chain, sev, &mem_err); arch_apei_report_mem_error(sev, mem_err); ghes_handle_memory_failure(gdata, sev); @@ -1139,10 +1154,6 @@ static int ghes_probe(struct platform_device *ghes_dev) goto err; } - rc = ghes_edac_register(ghes, &ghes_dev->dev); - if (rc < 0) - goto err; - switch (generic->notify.type) { case ACPI_HEST_NOTIFY_POLLED: setup_deferrable_timer(&ghes->timer, ghes_poll_func, @@ -1155,13 +1166,13 @@ static int ghes_probe(struct platform_device *ghes_dev) if (rc) { pr_err(GHES_PFX "Failed to map GSI to IRQ for generic hardware error source: %d\n", generic->header.source_id); - goto err_edac_unreg; + goto err; } rc = request_irq(ghes->irq, ghes_irq_func, 0, "GHES IRQ", ghes); if (rc) { pr_err(GHES_PFX "Failed to register IRQ for generic hardware error source: %d\n", generic->header.source_id); - goto err_edac_unreg; + goto err; } break; @@ -1190,8 +1201,7 @@ static int ghes_probe(struct platform_device *ghes_dev) ghes_proc(ghes); return 0; -err_edac_unreg: - ghes_edac_unregister(ghes); + err: if (ghes) { ghes_fini(ghes); @@ -1241,8 +1251,6 @@ static int ghes_remove(struct platform_device *ghes_dev) ghes_fini(ghes); - ghes_edac_unregister(ghes); - kfree(ghes); platform_set_drvdata(ghes_dev, NULL); diff --git a/drivers/edac/Kconfig b/drivers/edac/Kconfig index 96afb2aeed18..fdd8278ca89a 100644 --- a/drivers/edac/Kconfig +++ b/drivers/edac/Kconfig @@ -53,8 +53,8 @@ config EDAC_DECODE_MCE has been initialized. config EDAC_GHES - bool "Output ACPI APEI/GHES BIOS detected errors via EDAC" - depends on ACPI_APEI_GHES && (EDAC=y) + tristate "Output ACPI APEI/GHES BIOS detected errors via EDAC" + depends on ACPI_APEI_GHES help Not all machines support hardware-driven error report. Some of those provide a BIOS-driven error report mechanism via ACPI, using the diff --git a/drivers/edac/edac_mc.h b/drivers/edac/edac_mc.h index 5357800e418d..6d46f30dc657 100644 --- a/drivers/edac/edac_mc.h +++ b/drivers/edac/edac_mc.h @@ -60,6 +60,9 @@ #define edac_pci_printk(ctl, level, fmt, arg...) \ printk(level "EDAC PCI%d: " fmt, ctl->pci_idx, ##arg) +#define edac_pr_err(fmt, arg...) edac_printk(KERN_ERR, "", fmt, ##arg) +#define edac_pr_info(fmt, arg...) edac_printk(KERN_INFO, "", fmt, ##arg) + /* prefixes for edac_printk() and edac_mc_printk() */ #define EDAC_MC "MC" #define EDAC_PCI "PCI" diff --git a/drivers/edac/ghes_edac.c b/drivers/edac/ghes_edac.c index 4e61a6229dd2..20fafc55eb2d 100644 --- a/drivers/edac/ghes_edac.c +++ b/drivers/edac/ghes_edac.c @@ -5,6 +5,9 @@ * License version 2. * * Copyright (c) 2013 by Mauro Carvalho Chehab + * (c) 2017 Borislav Petkov + * + * Borislav Petkov: turn it into a proper module. * * Red Hat Inc. http://www.redhat.com */ @@ -17,7 +20,14 @@ #include "edac_module.h" #include <ras/ras_event.h> -#define GHES_EDAC_REVISION " Ver: 1.0.0" +#define GHES_EDAC_REVISION " Ver: 2.0.0" + +/* + * Hand it into EDAC's core so that we have a device to operate on. + */ +static struct device dummy_dev; + +struct ghes_edac_pvt *ghes_pvt; struct ghes_edac_pvt { struct list_head list; @@ -30,11 +40,6 @@ struct ghes_edac_pvt { char msg[80]; }; -static LIST_HEAD(ghes_reglist); -static DEFINE_MUTEX(ghes_edac_lock); -static int ghes_edac_mc_num; - - /* Memory Device - Type 17 of SMBIOS spec */ struct memdev_dmi_entry { u8 type; @@ -165,24 +170,21 @@ static void ghes_edac_dmidecode(const struct dmi_header *dh, void *arg) } } -void ghes_edac_report_mem_error(struct ghes *ghes, int sev, - struct cper_sec_mem_err *mem_err) +static int report_mem_error(struct notifier_block *nb, unsigned long sev, void *data) { + struct cper_sec_mem_err *mem_err = data; enum hw_event_mc_err_type type; struct edac_raw_error_desc *e; struct mem_ctl_info *mci; - struct ghes_edac_pvt *pvt = NULL; - char *p; + struct ghes_edac_pvt *pvt = ghes_pvt; u8 grain_bits; + char *p; - list_for_each_entry(pvt, &ghes_reglist, list) { - if (ghes == pvt->ghes) - break; - } if (!pvt) { - pr_err("Internal error: Can't find EDAC structure\n"); - return; + edac_pr_err("Internal error: Can't find EDAC structure\n"); + return NOTIFY_DONE; } + mci = pvt->mci; e = &mci->error_desc; @@ -402,23 +404,40 @@ void ghes_edac_report_mem_error(struct ghes *ghes, int sev, /* Report the error via EDAC API */ edac_raw_mc_handle_error(type, mci, e); + + return NOTIFY_DONE; } -EXPORT_SYMBOL_GPL(ghes_edac_report_mem_error); -int ghes_edac_register(struct ghes *ghes, struct device *dev) +static struct notifier_block ghes_nb = { + .notifier_call = report_mem_error, +}; + +static const char * const fake_msg = +"This EDAC driver relies on BIOS to enumerate memory and get error reports.\n" +"Unfortunately, not all BIOSes reflect the memory layout correctly.\n" +"So, the end result of using this driver varies from vendor to vendor.\n" +"If you find incorrect reports, please contact your hardware vendor\n" +"to correct its BIOS."; + +static const char * const super_crap_msg = +"This system has a very crappy BIOS: It doesn't even list the DIMMS.\n" +"Its SMBIOS info is wrong. It is doubtful that the error report would\n" +"work on such system. Use this driver with caution."; + +static int __init ghes_edac_register(void) { + struct ghes_edac_pvt *pvt = ghes_pvt; bool fake = false; int rc, num_dimm = 0; struct mem_ctl_info *mci; struct edac_mc_layer layers[1]; - struct ghes_edac_pvt *pvt; struct ghes_edac_dimm_fill dimm_fill; /* Get the number of DIMMs */ dmi_walk(ghes_edac_count_dimms, &num_dimm); /* Check if we've got a bogus BIOS */ - if (num_dimm == 0) { + if (!num_dimm) { fake = true; num_dimm = 1; } @@ -431,21 +450,17 @@ int ghes_edac_register(struct ghes *ghes, struct device *dev) * We need to serialize edac_mc_alloc() and edac_mc_add_mc(), * to avoid duplicated memory controller numbers */ - mutex_lock(&ghes_edac_lock); - mci = edac_mc_alloc(ghes_edac_mc_num, ARRAY_SIZE(layers), layers, - sizeof(*pvt)); + mci = edac_mc_alloc(1, ARRAY_SIZE(layers), layers, sizeof(*pvt)); if (!mci) { - pr_info("Can't allocate memory for EDAC data\n"); - mutex_unlock(&ghes_edac_lock); + edac_pr_err("Can't allocate memory for EDAC data\n"); return -ENOMEM; } pvt = mci->pvt_info; memset(pvt, 0, sizeof(*pvt)); - list_add_tail(&pvt->list, &ghes_reglist); - pvt->ghes = ghes; pvt->mci = mci; - mci->pdev = dev; + + mci->pdev = &dummy_dev; mci->mtype_cap = MEM_FLAG_EMPTY; mci->edac_ctl_cap = EDAC_FLAG_NONE; @@ -455,21 +470,12 @@ int ghes_edac_register(struct ghes *ghes, struct device *dev) mci->ctl_name = "ghes_edac"; mci->dev_name = "ghes"; - if (!ghes_edac_mc_num) { - if (!fake) { - pr_info("This EDAC driver relies on BIOS to enumerate memory and get error reports.\n"); - pr_info("Unfortunately, not all BIOSes reflect the memory layout correctly.\n"); - pr_info("So, the end result of using this driver varies from vendor to vendor.\n"); - pr_info("If you find incorrect reports, please contact your hardware vendor\n"); - pr_info("to correct its BIOS.\n"); - pr_info("This system has %d DIMM sockets.\n", - num_dimm); - } else { - pr_info("This system has a very crappy BIOS: It doesn't even list the DIMMS.\n"); - pr_info("Its SMBIOS info is wrong. It is doubtful that the error report would\n"); - pr_info("work on such system. Use this driver with caution\n"); - } - } + if (!fake) + edac_pr_info("%s\n", fake_msg); + else + edac_pr_info("%s\n", super_crap_msg); + + edac_pr_info("This system has %d DIMM sockets.\n", num_dimm); if (!fake) { /* @@ -478,13 +484,11 @@ int ghes_edac_register(struct ghes *ghes, struct device *dev) * Keep it in blank for the other memory controllers, as * there's no reliable way to properly credit each DIMM to * the memory controller, as different BIOSes fill the - * DMI bank location fields on different ways + * DMI bank location fields in different ways. */ - if (!ghes_edac_mc_num) { - dimm_fill.count = 0; - dimm_fill.mci = mci; - dmi_walk(ghes_edac_dmidecode, &dimm_fill); - } + dimm_fill.count = 0; + dimm_fill.mci = mci; + dmi_walk(ghes_edac_dmidecode, &dimm_fill); } else { struct dimm_info *dimm = EDAC_DIMM_PTR(mci->layers, mci->dimms, mci->n_layers, 0, 0, 0); @@ -498,30 +502,31 @@ int ghes_edac_register(struct ghes *ghes, struct device *dev) rc = edac_mc_add_mc(mci); if (rc < 0) { - pr_info("Can't register at EDAC core\n"); + edac_pr_err("Can't register with EDAC core\n"); edac_mc_free(mci); - mutex_unlock(&ghes_edac_lock); return -ENODEV; } - ghes_edac_mc_num++; - mutex_unlock(&ghes_edac_lock); + ghes_register_edac_chain(&ghes_nb); + return 0; } -EXPORT_SYMBOL_GPL(ghes_edac_register); +module_init(ghes_edac_register); -void ghes_edac_unregister(struct ghes *ghes) +static void __exit ghes_edac_unregister(void) { struct mem_ctl_info *mci; - struct ghes_edac_pvt *pvt, *tmp; - - list_for_each_entry_safe(pvt, tmp, &ghes_reglist, list) { - if (ghes == pvt->ghes) { - mci = pvt->mci; - edac_mc_del_mc(mci->pdev); - edac_mc_free(mci); - list_del(&pvt->list); - } - } + + ghes_unregister_edac_chain(&ghes_nb); + + mci = find_mci_by_dev(&dummy_dev); + WARN_ON(!mci); + + edac_mc_del_mc(mci->pdev); + edac_mc_free(mci); + } -EXPORT_SYMBOL_GPL(ghes_edac_unregister); +module_exit(ghes_edac_unregister); + +MODULE_LICENSE("GPL"); +MODULE_DESCRIPTION("GHES error decoding module - " GHES_EDAC_REVISION); diff --git a/include/acpi/ghes.h b/include/acpi/ghes.h index 9f26e01186ae..c02b8eb91bd6 100644 --- a/include/acpi/ghes.h +++ b/include/acpi/ghes.h @@ -51,31 +51,8 @@ enum { GHES_SEV_PANIC = 0x3, }; -/* From drivers/edac/ghes_edac.c */ - -#ifdef CONFIG_EDAC_GHES -void ghes_edac_report_mem_error(struct ghes *ghes, int sev, - struct cper_sec_mem_err *mem_err); - -int ghes_edac_register(struct ghes *ghes, struct device *dev); - -void ghes_edac_unregister(struct ghes *ghes); - -#else -static inline void ghes_edac_report_mem_error(struct ghes *ghes, int sev, - struct cper_sec_mem_err *mem_err) -{ -} - -static inline int ghes_edac_register(struct ghes *ghes, struct device *dev) -{ - return 0; -} - -static inline void ghes_edac_unregister(struct ghes *ghes) -{ -} -#endif +void ghes_register_edac_chain(struct notifier_block *nb); +void ghes_unregister_edac_chain(struct notifier_block *nb); static inline int acpi_hest_get_version(struct acpi_hest_generic_data *gdata) {