| Message ID | 51d3d738-cdf5-2992-bba5-c3e1f34096c2@infradead.org (mailing list archive) |
|---|---|
| State | New, archived |
| Headers | show |
On Wed, 13 Jun 2018 at 21:22, Geoff Levand <geoff@infradead.org> wrote: > > Adds a new ACPI init routine acpi_fixup_m400_quirks that adds > a work-around for HPE ProLiant m400 APEI firmware problems. > > The work-around disables APEI when CONFIG_ACPI_APEI is set and > m400 firmware is detected. Without this fixup m400 systems > experience errors like these on startup: > > [Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 2 > [Hardware Error]: event severity: fatal > [Hardware Error]: Error 0, type: fatal > [Hardware Error]: section_type: memory error > [Hardware Error]: error_status: 0x0000000000001300 > [Hardware Error]: error_type: 10, invalid address > Kernel panic - not syncing: Fatal hardware error! > > Signed-off-by: Geoff Levand <geoff@infradead.org> > --- > Hi, > > It seems unlikely there will be any m400 firmware updates to fix > this problem. APEI support is desired for new ARM64 servers coming > to market. Distros are now forced to have their own fixes, not > enable APEI, or let m400 users fend for themselves. Thanks for working on this. This patch works well on m400 with APEI enabled. Tested-by: Riku Voipio <riku.voipio@linaro.org> > -Geoff > > arch/arm64/kernel/acpi.c | 40 ++++++++++++++++++++++++++++++++++++---- > 1 file changed, 36 insertions(+), 4 deletions(-) > > diff --git a/arch/arm64/kernel/acpi.c b/arch/arm64/kernel/acpi.c > index 7b09487ff8fb..3c315c2c7476 100644 > --- a/arch/arm64/kernel/acpi.c > +++ b/arch/arm64/kernel/acpi.c > @@ -31,6 +31,8 @@ > #include <asm/cpu_ops.h> > #include <asm/smp_plat.h> > > +#include <acpi/apei.h> > + > #ifdef CONFIG_ACPI_APEI > # include <linux/efi.h> > # include <asm/pgtable.h> > @@ -177,6 +179,33 @@ static int __init acpi_fadt_sanity_check(void) > return ret; > } > > +/* > + * acpi_fixup_m400_quirks - Work-around for HPE ProLiant m400 APEI firmware > + * problems. > + */ > +static void __init acpi_fixup_m400_quirks(void) > +{ > + acpi_status status; > + struct acpi_table_header *header; > +#if !defined(CONFIG_ACPI_APEI) > + int hest_disable = HEST_DISABLED; > +#endif > + > + if (!IS_ENABLED(CONFIG_ACPI_APEI) || hest_disable != HEST_ENABLED) > + return; > + > + status = acpi_get_table(ACPI_SIG_HEST, 0, &header); > + > + if (ACPI_SUCCESS(status) && !strncmp(header->oem_id, "HPE ", 6) && > + !strncmp(header->oem_table_id, "ProLiant", 8) && > + MIDR_IMPLEMENTOR(read_cpuid_id()) == ARM_CPU_IMP_APM) { > + hest_disable = HEST_DISABLED; > + pr_info("Disabled APEI for m400.\n"); > + } > + > + acpi_put_table(header); > +} > + > /* > * acpi_boot_table_init() called from setup_arch(), always. > * 1. find RSDP and get its address, and then find XSDT > @@ -232,11 +261,14 @@ void __init acpi_boot_table_init(void) > if (acpi_disabled) { > if (earlycon_acpi_spcr_enable) > early_init_dt_scan_chosen_stdout(); > - } else { > - acpi_parse_spcr(earlycon_acpi_spcr_enable, true); > - if (IS_ENABLED(CONFIG_ACPI_BGRT)) > - acpi_table_parse(ACPI_SIG_BGRT, acpi_parse_bgrt); > + return; > } > + > + acpi_parse_spcr(earlycon_acpi_spcr_enable, true); > + if (IS_ENABLED(CONFIG_ACPI_BGRT)) > + acpi_table_parse(ACPI_SIG_BGRT, acpi_parse_bgrt); > + > + acpi_fixup_m400_quirks(); > } > > #ifdef CONFIG_ACPI_APEI > -- > 2.14.1 >
On Wed, Jun 13, 2018 at 11:22:30AM -0700, Geoff Levand wrote: > Adds a new ACPI init routine acpi_fixup_m400_quirks that adds > a work-around for HPE ProLiant m400 APEI firmware problems. > > The work-around disables APEI when CONFIG_ACPI_APEI is set and > m400 firmware is detected. Without this fixup m400 systems > experience errors like these on startup: > > [Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 2 > [Hardware Error]: event severity: fatal > [Hardware Error]: Error 0, type: fatal > [Hardware Error]: section_type: memory error > [Hardware Error]: error_status: 0x0000000000001300 > [Hardware Error]: error_type: 10, invalid address > Kernel panic - not syncing: Fatal hardware error! > > Signed-off-by: Geoff Levand <geoff@infradead.org> > --- > Hi, > > It seems unlikely there will be any m400 firmware updates to fix > this problem. APEI support is desired for new ARM64 servers coming > to market. Distros are now forced to have their own fixes, not > enable APEI, or let m400 users fend for themselves. > > Please consider. > > -Geoff > > arch/arm64/kernel/acpi.c | 40 ++++++++++++++++++++++++++++++++++++---- > 1 file changed, 36 insertions(+), 4 deletions(-) > > diff --git a/arch/arm64/kernel/acpi.c b/arch/arm64/kernel/acpi.c > index 7b09487ff8fb..3c315c2c7476 100644 > --- a/arch/arm64/kernel/acpi.c > +++ b/arch/arm64/kernel/acpi.c > @@ -31,6 +31,8 @@ > #include <asm/cpu_ops.h> > #include <asm/smp_plat.h> > > +#include <acpi/apei.h> > + > #ifdef CONFIG_ACPI_APEI > # include <linux/efi.h> > # include <asm/pgtable.h> > @@ -177,6 +179,33 @@ static int __init acpi_fadt_sanity_check(void) > return ret; > } > > +/* > + * acpi_fixup_m400_quirks - Work-around for HPE ProLiant m400 APEI firmware > + * problems. > + */ > +static void __init acpi_fixup_m400_quirks(void) > +{ > + acpi_status status; > + struct acpi_table_header *header; > +#if !defined(CONFIG_ACPI_APEI) > + int hest_disable = HEST_DISABLED; > +#endif > + > + if (!IS_ENABLED(CONFIG_ACPI_APEI) || hest_disable != HEST_ENABLED) > + return; > + > + status = acpi_get_table(ACPI_SIG_HEST, 0, &header); > + > + if (ACPI_SUCCESS(status) && !strncmp(header->oem_id, "HPE ", 6) && > + !strncmp(header->oem_table_id, "ProLiant", 8) && > + MIDR_IMPLEMENTOR(read_cpuid_id()) == ARM_CPU_IMP_APM) { > + hest_disable = HEST_DISABLED; > + pr_info("Disabled APEI for m400.\n"); > + } > + > + acpi_put_table(header); > +} > + > /* > * acpi_boot_table_init() called from setup_arch(), always. > * 1. find RSDP and get its address, and then find XSDT > @@ -232,11 +261,14 @@ void __init acpi_boot_table_init(void) > if (acpi_disabled) { > if (earlycon_acpi_spcr_enable) > early_init_dt_scan_chosen_stdout(); > - } else { > - acpi_parse_spcr(earlycon_acpi_spcr_enable, true); > - if (IS_ENABLED(CONFIG_ACPI_BGRT)) > - acpi_table_parse(ACPI_SIG_BGRT, acpi_parse_bgrt); > + return; > } > + > + acpi_parse_spcr(earlycon_acpi_spcr_enable, true); > + if (IS_ENABLED(CONFIG_ACPI_BGRT)) > + acpi_table_parse(ACPI_SIG_BGRT, acpi_parse_bgrt); > + > + acpi_fixup_m400_quirks(); > } > > #ifdef CONFIG_ACPI_APEI Quirk actually make code nicer to read. Reviewed-by: Graeme Gregory <graeme.gregory@linaro.org> Thanks we needed this having 90+ m400s Graeme
Hi Geoff, On 13/06/18 19:22, Geoff Levand wrote: > Adds a new ACPI init routine acpi_fixup_m400_quirks that adds > a work-around for HPE ProLiant m400 APEI firmware problems. > > The work-around disables APEI when CONFIG_ACPI_APEI is set and > m400 firmware is detected. Without this fixup m400 systems > experience errors like these on startup: > > [Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 2 > [Hardware Error]: event severity: fatal > [Hardware Error]: Error 0, type: fatal > [Hardware Error]: section_type: memory error > [Hardware Error]: error_status: 0x0000000000001300 "Access to a memory address which is not mapped to any component" > [Hardware Error]: error_type: 10, invalid address > Kernel panic - not syncing: Fatal hardware error! Why is this a problem? Surely this is a valid description of an error. (okay its not particularly useful without the physical address, but the address is optional in that structure) When does this happen during boot? This looks like a driver mapping some non-existent physical address space to see if its device is present... unsurprisingly this doesn't go well. (might also be a typo in the DSDT) Can't we pin down the driver that does this and fix it. Its either wrong for everyone, or still broken after you disable APEI. > It seems unlikely there will be any m400 firmware updates to fix > this problem. What is the problem? This patch looks like it shoots the messenger for bringing bad news. > diff --git a/arch/arm64/kernel/acpi.c b/arch/arm64/kernel/acpi.c > index 7b09487ff8fb..3c315c2c7476 100644 > --- a/arch/arm64/kernel/acpi.c > +++ b/arch/arm64/kernel/acpi.c > @@ -31,6 +31,8 @@ > #include <asm/cpu_ops.h> > #include <asm/smp_plat.h> > > +#include <acpi/apei.h> > + > #ifdef CONFIG_ACPI_APEI > # include <linux/efi.h> > # include <asm/pgtable.h> > @@ -177,6 +179,33 @@ static int __init acpi_fadt_sanity_check(void) > return ret; > } > > +/* > + * acpi_fixup_m400_quirks - Work-around for HPE ProLiant m400 APEI firmware > + * problems. > + */ > +static void __init acpi_fixup_m400_quirks(void) > +{ > + acpi_status status; > + struct acpi_table_header *header; > +#if !defined(CONFIG_ACPI_APEI) > + int hest_disable = HEST_DISABLED; > +#endif Yuck. > + > + if (!IS_ENABLED(CONFIG_ACPI_APEI) || hest_disable != HEST_ENABLED) > + return; > + > + status = acpi_get_table(ACPI_SIG_HEST, 0, &header); > + > + if (ACPI_SUCCESS(status) && !strncmp(header->oem_id, "HPE ", 6) && > + !strncmp(header->oem_table_id, "ProLiant", 8) && You should match the affected range of OEM table revisions too, that way a firmware upgrade should start working, instead of being permanently disabled because we think its unlikely. > + MIDR_IMPLEMENTOR(read_cpuid_id()) == ARM_CPU_IMP_APM) { How is the CPU implementer relevant? You suggest a firmware-update would make this issue go away... > + hest_disable = HEST_DISABLED; > + pr_info("Disabled APEI for m400.\n"); > + } > + > + acpi_put_table(header); > +} > + > /* > * acpi_boot_table_init() called from setup_arch(), always. > * 1. find RSDP and get its address, and then find XSDT Nothing arch-specific here. You're adding this to arch/arm64 because drivers/acpi/apei doesn't have an existing quirks table? Thanks, James
Hi James, Just for background, this is a well known bug in the m400's AEPI/HEST firmware. There are a number of fixes out there the different distros have. I just put together this patch to unify things and have a common 'upstream' fix. On 06/15/2018 04:14 AM, James Morse wrote: > On 13/06/18 19:22, Geoff Levand wrote: >> Adds a new ACPI init routine acpi_fixup_m400_quirks that adds >> a work-around for HPE ProLiant m400 APEI firmware problems. >> >> The work-around disables APEI when CONFIG_ACPI_APEI is set and >> m400 firmware is detected. Without this fixup m400 systems >> experience errors like these on startup: >> >> [Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 2 >> [Hardware Error]: event severity: fatal >> [Hardware Error]: Error 0, type: fatal >> [Hardware Error]: section_type: memory error >> [Hardware Error]: error_status: 0x0000000000001300 > > "Access to a memory address which is not mapped to any component" > > >> [Hardware Error]: error_type: 10, invalid address >> Kernel panic - not syncing: Fatal hardware error! > > Why is this a problem? > > Surely this is a valid description of an error. The firmware bug causes this failure, not bad hardware. > (okay its not particularly useful without the physical address, but the address > is optional in that structure) > > When does this happen during boot? This looks like a driver mapping some > non-existent physical address space to see if its device is present... > unsurprisingly this doesn't go well. > (might also be a typo in the DSDT) > > Can't we pin down the driver that does this and fix it. Its either wrong for > everyone, or still broken after you disable APEI. > > >> It seems unlikely there will be any m400 firmware updates to fix >> this problem. > > What is the problem? This patch looks like it shoots the messenger for bringing > bad news. The news is incorrect, so this patch disables the source (APEI code). >> diff --git a/arch/arm64/kernel/acpi.c b/arch/arm64/kernel/acpi.c >> index 7b09487ff8fb..3c315c2c7476 100644 >> --- a/arch/arm64/kernel/acpi.c >> +++ b/arch/arm64/kernel/acpi.c >> @@ -31,6 +31,8 @@ >> #include <asm/cpu_ops.h> >> #include <asm/smp_plat.h> >> >> +#include <acpi/apei.h> >> + >> #ifdef CONFIG_ACPI_APEI >> # include <linux/efi.h> >> # include <asm/pgtable.h> >> @@ -177,6 +179,33 @@ static int __init acpi_fadt_sanity_check(void) >> return ret; >> } >> >> +/* >> + * acpi_fixup_m400_quirks - Work-around for HPE ProLiant m400 APEI firmware >> + * problems. >> + */ >> +static void __init acpi_fixup_m400_quirks(void) >> +{ >> + acpi_status status; >> + struct acpi_table_header *header; >> +#if !defined(CONFIG_ACPI_APEI) >> + int hest_disable = HEST_DISABLED; >> +#endif > > Yuck. Yes, unfortunately, the hest code conditionally defines hest_disable. >> + >> + if (!IS_ENABLED(CONFIG_ACPI_APEI) || hest_disable != HEST_ENABLED) >> + return; >> + >> + status = acpi_get_table(ACPI_SIG_HEST, 0, &header); >> + >> + if (ACPI_SUCCESS(status) && !strncmp(header->oem_id, "HPE ", 6) && >> + !strncmp(header->oem_table_id, "ProLiant", 8) && > > You should match the affected range of OEM table revisions too, that way a > firmware upgrade should start working, instead of being permanently disabled > because we think its unlikely. The m400 has reached end of life. No one really expects to see any firmware update. I don't know what the effected OEM table revisions are, and I don't think there is an active platform maintainer who could give that info either. If someone can provide the info. I'll update the fix. >> + MIDR_IMPLEMENTOR(read_cpuid_id()) == ARM_CPU_IMP_APM) { > > How is the CPU implementer relevant? That was just a copy of what other fixes had. Should I remove it? > You suggest a firmware-update would make this issue go away... > > >> + hest_disable = HEST_DISABLED; >> + pr_info("Disabled APEI for m400.\n"); >> + } >> + >> + acpi_put_table(header); >> +} >> + >> /* >> * acpi_boot_table_init() called from setup_arch(), always. >> * 1. find RSDP and get its address, and then find XSDT > > Nothing arch-specific here. You're adding this to arch/arm64 because > drivers/acpi/apei doesn't have an existing quirks table? There was a fix submitted that had it in drivers/acpi/scan.c, but the ACPI maintainer said he didn't want the fix in the main ACPI code. See: https://lkml.org/lkml/2018/4/19/1020 (ACPI / scan: Fix regression related to X-Gene UARTs) The m400 is an arm64 platform, so it seems most appropriate to have it in arch/arm64/kernel/acpi.c. I followed what was done for x86 quirks (into arch/x86/kernel/acpi/boot.c), and what was suggested here: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=900581 (linux: Enable Buster kernel features for newer ARM64 servers) Thanks for the review. -Geoff
On Fri, 2018-06-15 at 10:17 -0700, Geoff Levand wrote: > > > + if (ACPI_SUCCESS(status) && !strncmp(header->oem_id, "HPE ", 6) && > > > + !strncmp(header->oem_table_id, "ProLiant", 8) && > > > > You should match the affected range of OEM table revisions too, that way a > > firmware upgrade should start working, instead of being permanently disabled > > because we think its unlikely. > > The m400 has reached end of life. No one really expects to see any firmware > update. I don't know what the effected OEM table revisions are, and I don't > think there is an active platform maintainer who could give that info either. > > If someone can provide the info. I'll update the fix. > > > > + MIDR_IMPLEMENTOR(read_cpuid_id()) == ARM_CPU_IMP_APM) { > > > > How is the CPU implementer relevant? > > That was just a copy of what other fixes had. Should I remove it? It was there because HPE ProLiant strings are generic and you may end up disabling platforms which would otherwise work. It is the ProLiant system based on the APM chipset which is the problem. Thus the check for cpu implementor.
Hi Mark, On 06/15/2018 10:33 AM, Mark Salter wrote: > On Fri, 2018-06-15 at 10:17 -0700, Geoff Levand wrote: >>>> + MIDR_IMPLEMENTOR(read_cpuid_id()) == ARM_CPU_IMP_APM) { >>> >>> How is the CPU implementer relevant? >> >> That was just a copy of what other fixes had. Should I remove it? > > It was there because HPE ProLiant strings are generic and you may end up > disabling platforms which would otherwise work. It is the ProLiant system > based on the APM chipset which is the problem. Thus the check for cpu > implementor. Your original fix that had this cpu implementor check was in the main ACPI code, so would be built for other arches. This is now in arch/arm64/kernel/acpi.c, which will only be built for arm64. Is that enough to limit it, or do we still need the check? -Geoff
On Fri, 2018-06-15 at 11:15 -0700, Geoff Levand wrote: > Hi Mark, > > On 06/15/2018 10:33 AM, Mark Salter wrote: > > On Fri, 2018-06-15 at 10:17 -0700, Geoff Levand wrote: > > > > > + MIDR_IMPLEMENTOR(read_cpuid_id()) == ARM_CPU_IMP_APM) { > > > > > > > > How is the CPU implementer relevant? > > > > > > That was just a copy of what other fixes had. Should I remove it? > > > > It was there because HPE ProLiant strings are generic and you may end up > > disabling platforms which would otherwise work. It is the ProLiant system > > based on the APM chipset which is the problem. Thus the check for cpu > > implementor. > > Your original fix that had this cpu implementor check was in the main > ACPI code, so would be built for other arches. This is now in > arch/arm64/kernel/acpi.c, which will only be built for arm64. Is that > enough to limit it, or do we still need the check? The original code was protected by #ifdef ARM64. But yes, HPE has announced another aarch64 ProLiant system based on Cavium ThunderX2. So we need to allow everything but the APM XGene based ProLiant products.
Hi Geoff, On 15/06/18 18:17, Geoff Levand wrote: > Just for background, this is a well known bug in the m400's AEPI/HEST > firmware. There are a number of fixes out there the different distros > have. Thanks, From: https://bugzilla.redhat.com/show_bug.cgi?id=1574718 This is tripped by the ghes_probe() call, it finds an error has already occurred before ghes code is initialized. Does this still happen if you compile without CONFIG_PCI? (that is easily half the dmesg that has happened before this point). Disabling CONFIG_EFIVAR_FS would be interesting too, as that is firmware-code we can't fix bugs in. Your argument here is the that the firmware-vendor only build-tested their code, and never noticed it notifies a fatal exception on startup. I find this hard to believe, especially as these systems don't have EL3. It's much more likely a driver is causing this, possibly because of bad data in the firmware tables. I'd like to quirk the driver, so we can fix the next error like this, as opposed to blindly continuing. If it really is firmware, what is it doing that causes this error, and where does it run? Disabling APEI is just reacting after the error has occurred, when we could prevent it happening. I can't reproduce this on a Mustang. I assume its the different ACPI tables, not the kernel-config. > I just put together this patch to unify things and have a > common 'upstream' fix. Wouldn't passing 'hest_disable' on the cmdline do the same for all kernel versions? > On 06/15/2018 04:14 AM, James Morse wrote: >> On 13/06/18 19:22, Geoff Levand wrote: >>> Adds a new ACPI init routine acpi_fixup_m400_quirks that adds >>> a work-around for HPE ProLiant m400 APEI firmware problems. >>> >>> The work-around disables APEI when CONFIG_ACPI_APEI is set and >>> m400 firmware is detected. Without this fixup m400 systems >>> experience errors like these on startup: >>> >>> [Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 2 >>> [Hardware Error]: event severity: fatal >>> [Hardware Error]: Error 0, type: fatal >>> [Hardware Error]: section_type: memory error >>> [Hardware Error]: error_status: 0x0000000000001300 >> >> "Access to a memory address which is not mapped to any component" >> >> >>> [Hardware Error]: error_type: 10, invalid address >>> Kernel panic - not syncing: Fatal hardware error! >> >> Why is this a problem? >> >> Surely this is a valid description of an error. > > The firmware bug causes this failure, not bad hardware. I'm not talking about bad hardware here. What I think is happening is a software bug is causing the CPU to make a bad access, which the RAS mechanism is reporting like this because software would never be stupid enough to access an address which is not mapped to any component! The RAS stuff believes this must be address corruption. I think this is a linux-driver bug, or a typo in the firmware tables, that cause $non_existent_address to be mapped and probed. >>> diff --git a/arch/arm64/kernel/acpi.c b/arch/arm64/kernel/acpi.c >>> index 7b09487ff8fb..3c315c2c7476 100644 >>> --- a/arch/arm64/kernel/acpi.c >>> +++ b/arch/arm64/kernel/acpi.c >>> + >>> + if (!IS_ENABLED(CONFIG_ACPI_APEI) || hest_disable != HEST_ENABLED) >>> + return; >>> + >>> + status = acpi_get_table(ACPI_SIG_HEST, 0, &header); >>> + >>> + if (ACPI_SUCCESS(status) && !strncmp(header->oem_id, "HPE ", 6) && >>> + !strncmp(header->oem_table_id, "ProLiant", 8) && >> You should match the affected range of OEM table revisions too, that way a >> firmware upgrade should start working, instead of being permanently disabled >> because we think its unlikely. > > The m400 has reached end of life. No one really expects to see any firmware > update. I don't know what the effected OEM table revisions are, and I don't > think there is an active platform maintainer who could give that info either. > > If someone can provide the info. I'll update the fix. We can start with the version you have. You mention distro's have their own fixes, hopefully they can supply the missing range of values. >>> + MIDR_IMPLEMENTOR(read_cpuid_id()) == ARM_CPU_IMP_APM) { >> >> How is the CPU implementer relevant? > > That was just a copy of what other fixes had. Should I remove it? The conclusion in the rest of this thread was HPE also produces ProLiant boxes with a (presumably) identical HEST, but a totally different architecture. Matching the DMI platform name as well would work round this. (or somewhere only arm64 builds, with an appropriate comment in case we move it). >> Nothing arch-specific here. You're adding this to arch/arm64 because >> drivers/acpi/apei doesn't have an existing quirks table? > > There was a fix submitted that had it in drivers/acpi/scan.c, but the > ACPI maintainer said he didn't want the fix in the main ACPI code. Specifically about a HID-hack in the core device enumeration code, as opposed to in the driver that claims it. > See: > > https://lkml.org/lkml/2018/4/19/1020 (ACPI / scan: Fix regression related to X-Gene UARTs) | 'some X-Gene based platforms (Mustang and M400) with invalid DSDT.' ... sounds familiar ... And: https://bugzilla.redhat.com/attachment.cgi?id=1144903&action=diff Fixes a typo in firmware description of the GIC. Why do we think this is a new kind of firmware bug, and not a repeat of the last two? The only difference is this is being caught by the RAS code. > The m400 is an arm64 platform, so it seems most appropriate to > have it in arch/arm64/kernel/acpi.c. We don't keep all drivers under arch/arm64, I'd really like to find the driver that is causing this, and quirk it there. If we have to bolt the hest-stable-door, drivers/acpi/apei/hest.c looks better, it at least doesn't have to bodge around hest_disabled not being exposed in a compatible way. If the maintainer objects, (as x86 hasn't had to do this yet), then we can try drivers/acpi/arm64/hest_quirks.c. When only once architecture had to quirk stuff based on the ACPI tables, the arch-code was a suitable dumping ground. Now there is more than one, we should do things in a way they are useful to both architectures. Thanks, James
Hi James, Thanks for all the comments, but my lack of access to an m400 platform, and my lack of knowledge about the m400 limits what I can comment on and what I can do. My motivation for submitting this fix was to enable CONFIG_ACPI_APEI in the Debian 10 kernel (for the new Cavium ThunderX2 based systems). The Debian maintainers wanted to either have an upstream kernel fix for this m400 APEI problem or to disable CONFIG_ACPI_APEI. The later would be unfortunate. It means users would need to have custom kernels to get APEI support or go without. Comments follow. On 06/18/2018 09:18 AM, James Morse wrote: > On 15/06/18 18:17, Geoff Levand wrote: > > From: > https://bugzilla.redhat.com/show_bug.cgi?id=1574718 > > This is tripped by the ghes_probe() call, it finds an error has already occurred > before ghes code is initialized. > > Does this still happen if you compile without CONFIG_PCI? (that is easily half > the dmesg that has happened before this point). > Disabling CONFIG_EFIVAR_FS would be interesting too, as that is firmware-code we > can't fix bugs in. Sorry, no m400 to test these on, but looking at the code, I would say it doesn't occur if CONFIG_PCI=n. > Your argument here is the that the firmware-vendor only build-tested their code, > and never noticed it notifies a fatal exception on startup. I find this hard to > believe, especially as these systems don't have EL3. > > It's much more likely a driver is causing this, possibly because of bad data in > the firmware tables. I'd like to quirk the driver, so we can fix the next error > like this, as opposed to blindly continuing. That sounds OK, but which driver? > If it really is firmware, what is it doing that causes this error, and where > does it run? Disabling APEI is just reacting after the error has occurred, when > we could prevent it happening. > > I can't reproduce this on a Mustang. I assume its the different ACPI tables, not > the kernel-config. From what I understand this is only on the m400, not the Mustang. >> I just put together this patch to unify things and have a >> common 'upstream' fix. > > Wouldn't passing 'hest_disable' on the cmdline do the same for all kernel versions? Yes, the current patch essentially just sets hest_disable when an m400 is detected. The cmdline work-around is what some have been using, but is not an acceptable solution for the Debian maintainers. See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=900581 >> On 06/15/2018 04:14 AM, James Morse wrote: >>> On 13/06/18 19:22, Geoff Levand wrote: >>>> Adds a new ACPI init routine acpi_fixup_m400_quirks that adds >>>> a work-around for HPE ProLiant m400 APEI firmware problems. >>>> >>>> The work-around disables APEI when CONFIG_ACPI_APEI is set and >>>> m400 firmware is detected. Without this fixup m400 systems >>>> experience errors like these on startup: >>>> >>>> [Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 2 >>>> [Hardware Error]: event severity: fatal >>>> [Hardware Error]: Error 0, type: fatal >>>> [Hardware Error]: section_type: memory error >>>> [Hardware Error]: error_status: 0x0000000000001300 >>> >>> "Access to a memory address which is not mapped to any component" >>> >>> >>>> [Hardware Error]: error_type: 10, invalid address >>>> Kernel panic - not syncing: Fatal hardware error! >>> >>> Why is this a problem? >>> >>> Surely this is a valid description of an error. >> >> The firmware bug causes this failure, not bad hardware. > > I'm not talking about bad hardware here. What I think is happening is a software > bug is causing the CPU to make a bad access, which the RAS mechanism is > reporting like this because software would never be stupid enough to access an > address which is not mapped to any component! The RAS stuff believes this must > be address corruption. > > I think this is a linux-driver bug, or a typo in the firmware tables, that cause > $non_existent_address to be mapped and probed. From what I know about it the problem is in the access of the firmware tables, and that corrupted data is retrieved. But I am not sure, as there are so many reported m400 firmware problems it is hard to tell what exactly is what. >>>> diff --git a/arch/arm64/kernel/acpi.c b/arch/arm64/kernel/acpi.c >>>> index 7b09487ff8fb..3c315c2c7476 100644 >>>> --- a/arch/arm64/kernel/acpi.c >>>> +++ b/arch/arm64/kernel/acpi.c > >>>> + >>>> + if (!IS_ENABLED(CONFIG_ACPI_APEI) || hest_disable != HEST_ENABLED) >>>> + return; >>>> + >>>> + status = acpi_get_table(ACPI_SIG_HEST, 0, &header); >>>> + >>>> + if (ACPI_SUCCESS(status) && !strncmp(header->oem_id, "HPE ", 6) && >>>> + !strncmp(header->oem_table_id, "ProLiant", 8) && > >>> You should match the affected range of OEM table revisions too, that way a >>> firmware upgrade should start working, instead of being permanently disabled >>> because we think its unlikely. >> >> The m400 has reached end of life. No one really expects to see any firmware >> update. I don't know what the effected OEM table revisions are, and I don't >> think there is an active platform maintainer who could give that info either. >> >> If someone can provide the info. I'll update the fix. > > We can start with the version you have. You mention distro's have their own > fixes, hopefully they can supply the missing range of values. It seems you are living in a dream world... >>>> + MIDR_IMPLEMENTOR(read_cpuid_id()) == ARM_CPU_IMP_APM) { >>> >>> How is the CPU implementer relevant? >> >> That was just a copy of what other fixes had. Should I remove it? > > The conclusion in the rest of this thread was HPE also produces ProLiant boxes > with a (presumably) identical HEST, but a totally different architecture. > > Matching the DMI platform name as well would work round this. (or somewhere only > arm64 builds, with an appropriate comment in case we move it). > > >>> Nothing arch-specific here. You're adding this to arch/arm64 because >>> drivers/acpi/apei doesn't have an existing quirks table? >> >> There was a fix submitted that had it in drivers/acpi/scan.c, but the >> ACPI maintainer said he didn't want the fix in the main ACPI code. > > Specifically about a HID-hack in the core device enumeration code, as opposed to > in the driver that claims it. > > >> See: >> >> https://lkml.org/lkml/2018/4/19/1020 (ACPI / scan: Fix regression related to X-Gene UARTs) > > | 'some X-Gene based platforms (Mustang and M400) with invalid DSDT.' > > ... sounds familiar ... > > And: > https://bugzilla.redhat.com/attachment.cgi?id=1144903&action=diff > > Fixes a typo in firmware description of the GIC. > > Why do we think this is a new kind of firmware bug, and not a repeat of the last > two? The only difference is this is being caught by the RAS code. It could very well be. >> The m400 is an arm64 platform, so it seems most appropriate to >> have it in arch/arm64/kernel/acpi.c. > > We don't keep all drivers under arch/arm64, I'd really like to find the driver > that is causing this, and quirk it there. > > If we have to bolt the hest-stable-door, drivers/acpi/apei/hest.c looks better, > it at least doesn't have to bodge around hest_disabled not being exposed in a > compatible way. If the maintainer objects, (as x86 hasn't had to do this yet), > then we can try drivers/acpi/arm64/hest_quirks.c. We need to call acpi_fixup_m400_quirks early enough to get hest_disable set before it is used. To do that from drivers/acpi/ code we would need have it as arch_initcall, or still call it from acpi_boot_table_init. I think having it as a static routine and the call in arch/arm64/kernel/acpi.c is cleaner. > When only once architecture had to quirk stuff based on the ACPI tables, the > arch-code was a suitable dumping ground. Now there is more than one, we should > do things in a way they are useful to both architectures. As I mentioned in the opening, I don't have access to m400 equipment. There is little I can do. I could move acpi_fixup_m400_quirks into drivers/acpi/apei/hest.c and submit that to the ACPI maintainer, but based on https://lkml.org/lkml/2018/4/19/1020, I don't think it would be accepted. -Geoff
On Mon, 2018-06-18 at 11:04 -0700, Geoff Levand wrote: > Hi James, > > Thanks for all the comments, but my lack of access to an m400 platform, and > my lack of knowledge about the m400 limits what I can comment on and what I > can do. I can take another look at this on an m400 here. I don't believe it is a memory access to physical space with nothing attached to it. I seem to recall an errata with xgene-1 where such accesses cause the cpu to halt. But I could be misremembering that. I have no trouble believing the firmware ras code was untested. It is probably some boilerplate code built in before ras was supported in kernel. But the problem occurs early enough in boot where there can't be that many things that would cause a problem on m400 and not mustang so I'll look again. > > My motivation for submitting this fix was to enable CONFIG_ACPI_APEI in the > Debian 10 kernel (for the new Cavium ThunderX2 based systems). The Debian > maintainers wanted to either have an upstream kernel fix for this m400 APEI > problem or to disable CONFIG_ACPI_APEI. The later would be unfortunate. It > means users would need to have custom kernels to get APEI support or go > without. > > Comments follow. > > On 06/18/2018 09:18 AM, James Morse wrote: > > On 15/06/18 18:17, Geoff Levand wrote: > > > > From: > > https://bugzilla.redhat.com/show_bug.cgi?id=1574718 > > > > This is tripped by the ghes_probe() call, it finds an error has already occurred > > before ghes code is initialized. > > > > Does this still happen if you compile without CONFIG_PCI? (that is easily half > > the dmesg that has happened before this point). > > Disabling CONFIG_EFIVAR_FS would be interesting too, as that is firmware-code we > > can't fix bugs in. > > Sorry, no m400 to test these on, but looking at the code, I would say it doesn't > occur if CONFIG_PCI=n. > > > Your argument here is the that the firmware-vendor only build-tested their code, > > and never noticed it notifies a fatal exception on startup. I find this hard to > > believe, especially as these systems don't have EL3. > > > > It's much more likely a driver is causing this, possibly because of bad data in > > the firmware tables. I'd like to quirk the driver, so we can fix the next error > > like this, as opposed to blindly continuing. > > That sounds OK, but which driver? > > > If it really is firmware, what is it doing that causes this error, and where > > does it run? Disabling APEI is just reacting after the error has occurred, when > > we could prevent it happening. > > > > I can't reproduce this on a Mustang. I assume its the different ACPI tables, not > > the kernel-config. > > From what I understand this is only on the m400, not the Mustang. > > > > I just put together this patch to unify things and have a > > > common 'upstream' fix. > > > > Wouldn't passing 'hest_disable' on the cmdline do the same for all kernel versions? > > Yes, the current patch essentially just sets hest_disable when an m400 is > detected. The cmdline work-around is what some have been using, but is > not an acceptable solution for the Debian maintainers. See > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=900581 > > > > On 06/15/2018 04:14 AM, James Morse wrote: > > > > On 13/06/18 19:22, Geoff Levand wrote: > > > > > Adds a new ACPI init routine acpi_fixup_m400_quirks that adds > > > > > a work-around for HPE ProLiant m400 APEI firmware problems. > > > > > > > > > > The work-around disables APEI when CONFIG_ACPI_APEI is set and > > > > > m400 firmware is detected. Without this fixup m400 systems > > > > > experience errors like these on startup: > > > > > > > > > > [Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 2 > > > > > [Hardware Error]: event severity: fatal > > > > > [Hardware Error]: Error 0, type: fatal > > > > > [Hardware Error]: section_type: memory error > > > > > [Hardware Error]: error_status: 0x0000000000001300 > > > > > > > > "Access to a memory address which is not mapped to any component" > > > > > > > > > > > > > [Hardware Error]: error_type: 10, invalid address > > > > > Kernel panic - not syncing: Fatal hardware error! > > > > > > > > Why is this a problem? > > > > > > > > Surely this is a valid description of an error. > > > > > > The firmware bug causes this failure, not bad hardware. > > > > I'm not talking about bad hardware here. What I think is happening is a software > > bug is causing the CPU to make a bad access, which the RAS mechanism is > > reporting like this because software would never be stupid enough to access an > > address which is not mapped to any component! The RAS stuff believes this must > > be address corruption. > > > > I think this is a linux-driver bug, or a typo in the firmware tables, that cause > > $non_existent_address to be mapped and probed. > > From what I know about it the problem is in the access of the firmware tables, > and that corrupted data is retrieved. But I am not sure, as there are so many > reported m400 firmware problems it is hard to tell what exactly is what. > > > > > > diff --git a/arch/arm64/kernel/acpi.c b/arch/arm64/kernel/acpi.c > > > > > index 7b09487ff8fb..3c315c2c7476 100644 > > > > > --- a/arch/arm64/kernel/acpi.c > > > > > +++ b/arch/arm64/kernel/acpi.c > > > > > + > > > > > + if (!IS_ENABLED(CONFIG_ACPI_APEI) || hest_disable != HEST_ENABLED) > > > > > + return; > > > > > + > > > > > + status = acpi_get_table(ACPI_SIG_HEST, 0, &header); > > > > > + > > > > > + if (ACPI_SUCCESS(status) && !strncmp(header->oem_id, "HPE ", 6) && > > > > > + !strncmp(header->oem_table_id, "ProLiant", 8) && > > > > You should match the affected range of OEM table revisions too, that way a > > > > firmware upgrade should start working, instead of being permanently disabled > > > > because we think its unlikely. > > > > > > The m400 has reached end of life. No one really expects to see any firmware > > > update. I don't know what the effected OEM table revisions are, and I don't > > > think there is an active platform maintainer who could give that info either. > > > > > > If someone can provide the info. I'll update the fix. > > > > We can start with the version you have. You mention distro's have their own > > fixes, hopefully they can supply the missing range of values. > > It seems you are living in a dream world... > > > > > > + MIDR_IMPLEMENTOR(read_cpuid_id()) == ARM_CPU_IMP_APM) { > > > > > > > > How is the CPU implementer relevant? > > > > > > That was just a copy of what other fixes had. Should I remove it? > > > > The conclusion in the rest of this thread was HPE also produces ProLiant boxes > > with a (presumably) identical HEST, but a totally different architecture. > > > > Matching the DMI platform name as well would work round this. (or somewhere only > > arm64 builds, with an appropriate comment in case we move it). > > > > > > > > Nothing arch-specific here. You're adding this to arch/arm64 because > > > > drivers/acpi/apei doesn't have an existing quirks table? > > > > > > There was a fix submitted that had it in drivers/acpi/scan.c, but the > > > ACPI maintainer said he didn't want the fix in the main ACPI code. > > > > Specifically about a HID-hack in the core device enumeration code, as opposed to > > in the driver that claims it. > > > > > > > See: > > > > > > https://lkml.org/lkml/2018/4/19/1020 (ACPI / scan: Fix regression related to X-Gene UARTs) > > > 'some X-Gene based platforms (Mustang and M400) with invalid DSDT.' > > > > ... sounds familiar ... > > > > And: > > https://bugzilla.redhat.com/attachment.cgi?id=1144903&action=diff > > > > Fixes a typo in firmware description of the GIC. > > > > Why do we think this is a new kind of firmware bug, and not a repeat of the last > > two? The only difference is this is being caught by the RAS code. > > It could very well be. > > > > The m400 is an arm64 platform, so it seems most appropriate to > > > have it in arch/arm64/kernel/acpi.c. > > > > We don't keep all drivers under arch/arm64, I'd really like to find the driver > > that is causing this, and quirk it there. > > > > If we have to bolt the hest-stable-door, drivers/acpi/apei/hest.c looks better, > > it at least doesn't have to bodge around hest_disabled not being exposed in a > > compatible way. If the maintainer objects, (as x86 hasn't had to do this yet), > > then we can try drivers/acpi/arm64/hest_quirks.c. > > We need to call acpi_fixup_m400_quirks early enough to get hest_disable set > before it is used. To do that from drivers/acpi/ code we would need have > it as arch_initcall, or still call it from acpi_boot_table_init. I think > having it as a static routine and the call in arch/arm64/kernel/acpi.c is > cleaner. > > > When only once architecture had to quirk stuff based on the ACPI tables, the > > arch-code was a suitable dumping ground. Now there is more than one, we should > > do things in a way they are useful to both architectures. > > As I mentioned in the opening, I don't have access to m400 equipment. There is > little I can do. I could move acpi_fixup_m400_quirks into > drivers/acpi/apei/hest.c and submit that to the ACPI maintainer, but based on > https://lkml.org/lkml/2018/4/19/1020, I don't think it would be accepted. > > -Geoff >
Hi Mark, On 18/06/18 23:18, Mark Salter wrote: > On Mon, 2018-06-18 at 11:04 -0700, Geoff Levand wrote: >> Thanks for all the comments, but my lack of access to an m400 platform, and >> my lack of knowledge about the m400 limits what I can comment on and what I >> can do. > > I can take another look at this on an m400 here. Thanks! > I don't believe it is a > memory access to physical space with nothing attached to it. That is what the CPER records are describing though. > I seem to recall > an errata with xgene-1 where such accesses cause the cpu to halt. But I could > be misremembering that. I have no trouble believing the firmware ras code was > untested. It is probably some boilerplate code built in before ras was supported > in kernel. It would be interesting to know which GHES this error is being found in, and whether the Error Status Block points anywhere (or at an empty block) when Linux is started from UEFI. If there is something in the Error Status Block out of UEFI, then this must be something triggered by UEFI, or a bug that can be fixed by UEFI clearing out the CPER records. https://bugzilla.redhat.com/show_bug.cgi?id=1285107 suggests redhat can rebuild the UEFI firmware for this box. If there is nothing in the Error Status Block when Linux is started, surely Linux is doing something to cause this to happen. I'd like to find out what, as its probably a software bug. (The case where disabling HEST would be the right thing to do is if there is a bogus GHES->GAS entry in GHES.0, the access to which causes GHES.1 to be populated with 'Access to an address not mapped to any component', which we find next. If this is the case it would be better to check GHES entries against the UEFI memory map to check this is memory, and it was reserved.) > But the problem occurs early enough in boot where there can't be > that many things that would cause a problem on m400 and not mustang so I'll > look again. Playing spot the difference in the dmesg, I'd check for smoke coming out of: | acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 | xgene-gpio APMC0D14:00: X-Gene GPIO driver registered. | pcie_pme: probe of 0000:00:00.0:pcie001 failed with error -22 If the firmware description of the GIC is wrong in someway, disabling KVM may be worth testing too. Thanks, James
On Tue, 2018-06-19 at 11:21 +0100, James Morse wrote: > Hi Mark, > > On 18/06/18 23:18, Mark Salter wrote: > > On Mon, 2018-06-18 at 11:04 -0700, Geoff Levand wrote: > > > Thanks for all the comments, but my lack of access to an m400 platform, and > > > my lack of knowledge about the m400 limits what I can comment on and what I > > > can do. > > > > I can take another look at this on an m400 here. > > Thanks! > > > > I don't believe it is a > > memory access to physical space with nothing attached to it. > > That is what the CPER records are describing though. Yes. > > > > I seem to recall > > an errata with xgene-1 where such accesses cause the cpu to halt. But I could > > be misremembering that. I have no trouble believing the firmware ras code was > > untested. It is probably some boilerplate code built in before ras was supported > > in kernel. > > It would be interesting to know which GHES this error is being found in, and > whether the Error Status Block points anywhere (or at an empty block) when Linux > is started from UEFI. > > If there is something in the Error Status Block out of UEFI, then this must be > something triggered by UEFI, or a bug that can be fixed by UEFI clearing out the > CPER records. > > https://bugzilla.redhat.com/show_bug.cgi?id=1285107 > suggests redhat can rebuild the UEFI firmware for this box. > > > If there is nothing in the Error Status Block when Linux is started, surely > Linux is doing something to cause this to happen. I'd like to find out what, as > its probably a software bug. > > > (The case where disabling HEST would be the right thing to do is if there is a > bogus GHES->GAS entry in GHES.0, the access to which causes GHES.1 to be > populated with 'Access to an address not mapped to any component', which we find > next. If this is the case it would be better to check GHES entries against the > UEFI memory map to check this is memory, and it was reserved.) > > > > But the problem occurs early enough in boot where there can't be > > that many things that would cause a problem on m400 and not mustang so I'll > > look again. > > Playing spot the difference in the dmesg, I'd check for smoke coming out of: > > acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 > > xgene-gpio APMC0D14:00: X-Gene GPIO driver registered. > > pcie_pme: probe of 0000:00:00.0:pcie001 failed with error -22 I've eliminated these by building a kernel with minimalized config and hacks (ACPI requires PCI, so I added code to prevent the root complexe from being probed). I also eliminated all the xgene-specific devices from the config (network, sata, etc). Still hit the ghes panic. I'm going to hack something to get to the ghes info earlier in boot and check the things you mention above wrt Error Status Block and GHES.0. > > If the firmware description of the GIC is wrong in someway, disabling KVM may be > worth testing too. > > > Thanks, > > James
On Fri, 2018-06-22 at 11:19 -0400, Mark Salter wrote: > I'm going to hack something to get to the ghes info earlier in boot and > check the things you mention above wrt Error Status Block and GHES.0. So I had to end up instrumenting the EFI stub to see where the error came from. At the start of the stub, there is no GHES.2 error. The error first shows up after the stub's call to ExitBootServices returns. So it looks like the firmware itself is causing the error. There's still a chance that the stub is doing something wrong with the memory map passed to the firmware, so I'll try to eliminate that as well.
Hi Mark, Thanks for shed-ing some light on what is going on here! On 25/06/18 16:34, Mark Salter wrote: > On Fri, 2018-06-22 at 11:19 -0400, Mark Salter wrote: >> I'm going to hack something to get to the ghes info earlier in boot and >> check the things you mention above wrt Error Status Block and GHES.0. > > So I had to end up instrumenting the EFI stub to see where the error came > from. At the start of the stub, there is no GHES.2 error. The error first > shows up after the stub's call to ExitBootServices returns. What's the notification type of GHES.2? I'm guessing POLLed or some kind of IRQ. These systems don't have EL3, so the CPU must continue running while something external generates the CPER records. The records being visible is the last point the faulty-access could have been made, with the window of time depending on how fast this external-thing receives and processes the error. > So it looks > like the firmware itself is causing the error. There's still a chance that > the stub is doing something wrong with the memory map passed to the > firmware, so I'll try to eliminate that as well. adding delay loops will help prove the EFIStub is innocent. Are there any optional drivers being loaded by UEFI? (can you remove any USB mass storage drives for instance). Are redhat able to rebuild UEFI on these systems? (Can it be fixed?) https://bugzilla.redhat.com/show_bug.cgi?id=1285107 is about the m400 description of the GIC, comments 15 and 16 show a UEFI patch to something other than the upstream platforms tree[0], and new firmware being tested. (although this may be wishful thinking) It looks like quirking this based on the DMI platform name and UEFI version will be what we need. We could discard anything in the error status block areas at ghes_probe() time based on this quirk, but we may have missed other problems during boot, giving a false sense of security. Thanks, James [0] Might be wrong, but this is where I look: https://github.com/tianocore/edk2-platforms.git
On Tue, 2018-06-26 at 15:51 +0100, James Morse wrote: > Hi Mark, > > Thanks for shed-ing some light on what is going on here! > > On 25/06/18 16:34, Mark Salter wrote: > > On Fri, 2018-06-22 at 11:19 -0400, Mark Salter wrote: > > > I'm going to hack something to get to the ghes info earlier in boot and > > > check the things you mention above wrt Error Status Block and GHES.0. > > > > So I had to end up instrumenting the EFI stub to see where the error came > > from. At the start of the stub, there is no GHES.2 error. The error first > > shows up after the stub's call to ExitBootServices returns. > > What's the notification type of GHES.2? I'm guessing POLLed or some kind of IRQ. SCI Here's the HEST entry: [028h 0040 2] Subtable Type : 0009 [Generic Hardware Error Source] [02Ah 0042 2] Source Id : 0002 [02Ch 0044 2] Related Source Id : FFFF [02Eh 0046 1] Reserved : 00 [02Fh 0047 1] Enabled : 01 [030h 0048 4] Records To Preallocate : 00000001 [034h 0052 4] Max Sections Per Record : 00000001 [038h 0056 4] Max Raw Data Length : 00000AEC [03Ch 0060 12] Error Status Address : [Generic Address Structure] [03Ch 0060 1] Space ID : 00 [SystemMemory] [03Dh 0061 1] Bit Width : 40 [03Eh 0062 1] Bit Offset : 00 [03Fh 0063 1] Encoded Access Width : 04 [QWord Access:64] [040h 0064 8] Address : 0000004FF7E9F0E0 There are 9 others all identical except for Source ID and address. > These systems don't have EL3, so the CPU must continue running while something > external generates the CPER records. The records being visible is the last point > the faulty-access could have been made, with the window of time depending on how > fast this external-thing receives and processes the error. There's a System Control Processor (slimpro) on the SoC which can interact with the CPU in various ways and which has access to memory and other hw. > > > > So it looks > > like the firmware itself is causing the error. There's still a chance that > > the stub is doing something wrong with the memory map passed to the > > firmware, so I'll try to eliminate that as well. > > adding delay loops will help prove the EFIStub is innocent. Didn't change anything. > > Are there any optional drivers being loaded by UEFI? (can you remove any USB > mass storage drives for instance). The only storage is pci based. There is a USB port but doesn't look like anything is attached to it. I don't have physical access to it. It is one on many moonshot cartridges in a chassis several hundred miles away. > > Are redhat able to rebuild UEFI on these systems? (Can it be fixed?) No. > > https://bugzilla.redhat.com/show_bug.cgi?id=1285107 is about the m400 > description of the GIC, comments 15 and 16 show a UEFI patch to something other > than the upstream platforms tree[0], and new firmware being tested. > (although this may be wishful thinking) HPe would respond to bug reports until m400 reached EOL. They have been pretty clear that no more firmware updates will be done. > > It looks like quirking this based on the DMI platform name and UEFI version will > be what we need. We could discard anything in the error status block areas at > ghes_probe() time based on this quirk, but we may have missed other problems > during boot, giving a false sense of security. > > > Thanks, > > James > > > [0] Might be wrong, but this is where I look: > https://github.com/tianocore/edk2-platforms.git
On 26 June 2018 at 22:20, Mark Salter <msalter@redhat.com> wrote: > On Tue, 2018-06-26 at 15:51 +0100, James Morse wrote: >> Hi Mark, >> >> Thanks for shed-ing some light on what is going on here! >> >> On 25/06/18 16:34, Mark Salter wrote: >> > On Fri, 2018-06-22 at 11:19 -0400, Mark Salter wrote: >> > > I'm going to hack something to get to the ghes info earlier in boot and >> > > check the things you mention above wrt Error Status Block and GHES.0. >> > >> > So I had to end up instrumenting the EFI stub to see where the error came >> > from. At the start of the stub, there is no GHES.2 error. The error first >> > shows up after the stub's call to ExitBootServices returns. >> >> What's the notification type of GHES.2? I'm guessing POLLed or some kind of IRQ. > > SCI > > Here's the HEST entry: > > [028h 0040 2] Subtable Type : 0009 [Generic Hardware Error Source] > [02Ah 0042 2] Source Id : 0002 > [02Ch 0044 2] Related Source Id : FFFF > [02Eh 0046 1] Reserved : 00 > [02Fh 0047 1] Enabled : 01 > [030h 0048 4] Records To Preallocate : 00000001 > [034h 0052 4] Max Sections Per Record : 00000001 > [038h 0056 4] Max Raw Data Length : 00000AEC > > [03Ch 0060 12] Error Status Address : [Generic Address Structure] > [03Ch 0060 1] Space ID : 00 [SystemMemory] > [03Dh 0061 1] Bit Width : 40 > [03Eh 0062 1] Bit Offset : 00 > [03Fh 0063 1] Encoded Access Width : 04 [QWord Access:64] > [040h 0064 8] Address : 0000004FF7E9F0E0 > This is a reserved region in the memory map. Does that apply to the other occurrences as well? > There are 9 others all identical except for Source ID and address. > >> These systems don't have EL3, so the CPU must continue running while something >> external generates the CPER records. The records being visible is the last point >> the faulty-access could have been made, with the window of time depending on how >> fast this external-thing receives and processes the error. > > There's a System Control Processor (slimpro) on the SoC which can interact with > the CPU in various ways and which has access to memory and other hw. > >> >> >> > So it looks >> > like the firmware itself is causing the error. There's still a chance that >> > the stub is doing something wrong with the memory map passed to the >> > firmware, so I'll try to eliminate that as well. >> >> adding delay loops will help prove the EFIStub is innocent. > > Didn't change anything. > >> >> Are there any optional drivers being loaded by UEFI? (can you remove any USB >> mass storage drives for instance). > > The only storage is pci based. There is a USB port but doesn't look like > anything is attached to it. I don't have physical access to it. It is one on > many moonshot cartridges in a chassis several hundred miles away. > >> >> Are redhat able to rebuild UEFI on these systems? (Can it be fixed?) > > No. > >> >> https://bugzilla.redhat.com/show_bug.cgi?id=1285107 is about the m400 >> description of the GIC, comments 15 and 16 show a UEFI patch to something other >> than the upstream platforms tree[0], and new firmware being tested. >> (although this may be wishful thinking) > > HPe would respond to bug reports until m400 reached EOL. They have been pretty > clear that no more firmware updates will be done. > >> >> It looks like quirking this based on the DMI platform name and UEFI version will >> be what we need. We could discard anything in the error status block areas at >> ghes_probe() time based on this quirk, but we may have missed other problems >> during boot, giving a false sense of security. >> >> >> Thanks, >> >> James >> >> >> [0] Might be wrong, but this is where I look: >> https://github.com/tianocore/edk2-platforms.git > > > _______________________________________________ > linux-arm-kernel mailing list > linux-arm-kernel@lists.infradead.org > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
On Wed, 2018-06-27 at 10:48 +0200, Ard Biesheuvel wrote: > On 26 June 2018 at 22:20, Mark Salter <msalter@redhat.com> wrote: > > On Tue, 2018-06-26 at 15:51 +0100, James Morse wrote: > > > Hi Mark, > > > > > > Thanks for shed-ing some light on what is going on here! > > > > > > On 25/06/18 16:34, Mark Salter wrote: > > > > On Fri, 2018-06-22 at 11:19 -0400, Mark Salter wrote: > > > > > I'm going to hack something to get to the ghes info earlier in boot and > > > > > check the things you mention above wrt Error Status Block and GHES.0. > > > > > > > > So I had to end up instrumenting the EFI stub to see where the error came > > > > from. At the start of the stub, there is no GHES.2 error. The error first > > > > shows up after the stub's call to ExitBootServices returns. > > > > > > What's the notification type of GHES.2? I'm guessing POLLed or some kind of IRQ. > > > > SCI > > > > Here's the HEST entry: > > > > [028h 0040 2] Subtable Type : 0009 [Generic Hardware Error Source] > > [02Ah 0042 2] Source Id : 0002 > > [02Ch 0044 2] Related Source Id : FFFF > > [02Eh 0046 1] Reserved : 00 > > [02Fh 0047 1] Enabled : 01 > > [030h 0048 4] Records To Preallocate : 00000001 > > [034h 0052 4] Max Sections Per Record : 00000001 > > [038h 0056 4] Max Raw Data Length : 00000AEC > > > > [03Ch 0060 12] Error Status Address : [Generic Address Structure] > > [03Ch 0060 1] Space ID : 00 [SystemMemory] > > [03Dh 0061 1] Bit Width : 40 > > [03Eh 0062 1] Bit Offset : 00 > > [03Fh 0063 1] Encoded Access Width : 04 [QWord Access:64] > > [040h 0064 8] Address : 0000004FF7E9F0E0 > > > > This is a reserved region in the memory map. Does that apply to the > other occurrences as well? Yes, they are all in the same reserved region. > > > There are 9 others all identical except for Source ID and address. > > > > > These systems don't have EL3, so the CPU must continue running while something > > > external generates the CPER records. The records being visible is the last point > > > the faulty-access could have been made, with the window of time depending on how > > > fast this external-thing receives and processes the error. > > > > There's a System Control Processor (slimpro) on the SoC which can interact with > > the CPU in various ways and which has access to memory and other hw. > > > > > > > > > > > > So it looks > > > > like the firmware itself is causing the error. There's still a chance that > > > > the stub is doing something wrong with the memory map passed to the > > > > firmware, so I'll try to eliminate that as well. > > > > > > adding delay loops will help prove the EFIStub is innocent. > > > > Didn't change anything. > > > > > > > > Are there any optional drivers being loaded by UEFI? (can you remove any USB > > > mass storage drives for instance). > > > > The only storage is pci based. There is a USB port but doesn't look like > > anything is attached to it. I don't have physical access to it. It is one on > > many moonshot cartridges in a chassis several hundred miles away. > > > > > > > > Are redhat able to rebuild UEFI on these systems? (Can it be fixed?) > > > > No. > > > > > > > > https://bugzilla.redhat.com/show_bug.cgi?id=1285107 is about the m400 > > > description of the GIC, comments 15 and 16 show a UEFI patch to something other > > > than the upstream platforms tree[0], and new firmware being tested. > > > (although this may be wishful thinking) > > > > HPe would respond to bug reports until m400 reached EOL. They have been pretty > > clear that no more firmware updates will be done. > > > > > > > > It looks like quirking this based on the DMI platform name and UEFI version will > > > be what we need. We could discard anything in the error status block areas at > > > ghes_probe() time based on this quirk, but we may have missed other problems > > > during boot, giving a false sense of security. > > > > > > > > > Thanks, > > > > > > James > > > > > > > > > [0] Might be wrong, but this is where I look: > > > https://github.com/tianocore/edk2-platforms.git > > > > > > _______________________________________________ > > linux-arm-kernel mailing list > > linux-arm-kernel@lists.infradead.org > > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
Hi Mark, On 26/06/18 21:20, Mark Salter wrote: > On Tue, 2018-06-26 at 15:51 +0100, James Morse wrote: >> On 25/06/18 16:34, Mark Salter wrote: >>> On Fri, 2018-06-22 at 11:19 -0400, Mark Salter wrote: >>>> I'm going to hack something to get to the ghes info earlier in boot and >>>> check the things you mention above wrt Error Status Block and GHES.0. >>> >>> So I had to end up instrumenting the EFI stub to see where the error came >>> from. At the start of the stub, there is no GHES.2 error. The error first >>> shows up after the stub's call to ExitBootServices returns. >> >> What's the notification type of GHES.2? I'm guessing POLLed or some kind of IRQ. >> These systems don't have EL3, so the CPU must continue running while something >> external generates the CPER records. The records being visible is the last point >> the faulty-access could have been made, with the window of time depending on how >> fast this external-thing receives and processes the error. > > There's a System Control Processor (slimpro) on the SoC which can interact with > the CPU in various ways and which has access to memory and other hw. Thanks, saves me guessing! >>> So it looks >>> like the firmware itself is causing the error. There's still a chance that >>> the stub is doing something wrong with the memory map passed to the >>> firmware, so I'll try to eliminate that as well. >> >> adding delay loops will help prove the EFIStub is innocent. > > Didn't change anything. Okay, so just to clarify, a delay before ExitBootServices doesn't cause the error to show up before ExitBootServices, so the error hasn't occurred prior to this point. And a delay after ExitBootServices allows us to see the error before we exit into head.S. (this rules out a bug in head.S) The delays should be long enough to tell us this slimpro isn't generating the error records N seconds after reset. Given this I agree we should disable_hest based on the DMI platform name and the UEFI version number. (it may be earlier firmware didn't have this bug). I don't have anything to test this on, so I've picked the DMI strings out the demsg output on that bugzilla entry. Any chance you could give it a test? >> Are redhat able to rebuild UEFI on these systems? (Can it be fixed?) >> https://bugzilla.redhat.com/show_bug.cgi?id=1285107 is about the m400 >> description of the GIC, comments 15 and 16 show a UEFI patch to something other >> than the upstream platforms tree[0], and new firmware being tested. >> (although this may be wishful thinking) > > HPe would respond to bug reports until m400 reached EOL. They have been pretty > clear that no more firmware updates will be done. Thanks, it was a bit murky from that ticket... Thanks for doing this! James
On Thu, 2018-06-28 at 11:06 +0100, James Morse wrote: > Hi Mark, > > On 26/06/18 21:20, Mark Salter wrote: > > On Tue, 2018-06-26 at 15:51 +0100, James Morse wrote: > > > On 25/06/18 16:34, Mark Salter wrote: > > > > On Fri, 2018-06-22 at 11:19 -0400, Mark Salter wrote: > > > > > I'm going to hack something to get to the ghes info earlier in boot and > > > > > check the things you mention above wrt Error Status Block and GHES.0. > > > > > > > > So I had to end up instrumenting the EFI stub to see where the error came > > > > from. At the start of the stub, there is no GHES.2 error. The error first > > > > shows up after the stub's call to ExitBootServices returns. > > > > > > What's the notification type of GHES.2? I'm guessing POLLed or some kind of IRQ. > > > These systems don't have EL3, so the CPU must continue running while something > > > external generates the CPER records. The records being visible is the last point > > > the faulty-access could have been made, with the window of time depending on how > > > fast this external-thing receives and processes the error. > > > > There's a System Control Processor (slimpro) on the SoC which can interact with > > the CPU in various ways and which has access to memory and other hw. > > Thanks, saves me guessing! > > > > > > So it looks > > > > like the firmware itself is causing the error. There's still a chance that > > > > the stub is doing something wrong with the memory map passed to the > > > > firmware, so I'll try to eliminate that as well. > > > > > > adding delay loops will help prove the EFIStub is innocent. > > > > Didn't change anything. > Just closing the loop on this... > Okay, so just to clarify, a delay before ExitBootServices doesn't cause the > error to show up before ExitBootServices, so the error hasn't occurred prior to > this point. Correct. I have never seen the error before ExitBootServices. > And a delay after ExitBootServices allows us to see the error before we exit > into head.S. (this rules out a bug in head.S) > The delays should be long enough to tell us this slimpro isn't generating the > error records N seconds after reset. No delay needed after ExitBootServices. The error would be there right after the call returns back to stub. > > Given this I agree we should disable_hest based on the DMI platform name and the > UEFI version number. (it may be earlier firmware didn't have this bug). > > > I don't have anything to test this on, so I've picked the DMI strings out the > demsg output on that bugzilla entry. Any chance you could give it a test? > > > > > Are redhat able to rebuild UEFI on these systems? (Can it be fixed?) > > > https://bugzilla.redhat.com/show_bug.cgi?id=1285107 is about the m400 > > > description of the GIC, comments 15 and 16 show a UEFI patch to something other > > > than the upstream platforms tree[0], and new firmware being tested. > > > (although this may be wishful thinking) > > > > HPe would respond to bug reports until m400 reached EOL. They have been pretty > > clear that no more firmware updates will be done. > > Thanks, it was a bit murky from that ticket... > > > Thanks for doing this! > > James
On Wed, 2018-06-27 at 08:25 -0400, Mark Salter wrote: > On Wed, 2018-06-27 at 10:48 +0200, Ard Biesheuvel wrote: > > On 26 June 2018 at 22:20, Mark Salter <msalter@redhat.com> wrote: > > > [040h 0064 8] Address : 0000004FF7E9F0E0 > > > > > > > This is a reserved region in the memory map. Does that apply to the > > other occurrences as well? > > Yes, they are all in the same reserved region. I wonder if it corresponds to some actual peripheral on a different X- gene based platform (e.g. perhaps a reference design)? Not sure what one would do with that information, just curious. Ian.
On Tue, 2018-07-03 at 10:30 +0100, Ian Campbell wrote: > On Wed, 2018-06-27 at 08:25 -0400, Mark Salter wrote: > > On Wed, 2018-06-27 at 10:48 +0200, Ard Biesheuvel wrote: > > > On 26 June 2018 at 22:20, Mark Salter <msalter@redhat.com> wrote: > > > > [040h 0064 8] Address : 0000004FF7E9F0E0 > > > > > > > > > > This is a reserved region in the memory map. Does that apply to the > > > other occurrences as well? > > > > Yes, they are all in the same reserved region. > > I wonder if it corresponds to some actual peripheral on a different X- > gene based platform (e.g. perhaps a reference design)? > > Not sure what one would do with that information, just curious. > > Ian. It's a reserved region of system RAM.
diff --git a/arch/arm64/kernel/acpi.c b/arch/arm64/kernel/acpi.c index 7b09487ff8fb..3c315c2c7476 100644 --- a/arch/arm64/kernel/acpi.c +++ b/arch/arm64/kernel/acpi.c @@ -31,6 +31,8 @@ #include <asm/cpu_ops.h> #include <asm/smp_plat.h> +#include <acpi/apei.h> + #ifdef CONFIG_ACPI_APEI # include <linux/efi.h> # include <asm/pgtable.h> @@ -177,6 +179,33 @@ static int __init acpi_fadt_sanity_check(void) return ret; } +/* + * acpi_fixup_m400_quirks - Work-around for HPE ProLiant m400 APEI firmware + * problems. + */ +static void __init acpi_fixup_m400_quirks(void) +{ + acpi_status status; + struct acpi_table_header *header; +#if !defined(CONFIG_ACPI_APEI) + int hest_disable = HEST_DISABLED; +#endif + + if (!IS_ENABLED(CONFIG_ACPI_APEI) || hest_disable != HEST_ENABLED) + return; + + status = acpi_get_table(ACPI_SIG_HEST, 0, &header); + + if (ACPI_SUCCESS(status) && !strncmp(header->oem_id, "HPE ", 6) && + !strncmp(header->oem_table_id, "ProLiant", 8) && + MIDR_IMPLEMENTOR(read_cpuid_id()) == ARM_CPU_IMP_APM) { + hest_disable = HEST_DISABLED; + pr_info("Disabled APEI for m400.\n"); + } + + acpi_put_table(header); +} + /* * acpi_boot_table_init() called from setup_arch(), always. * 1. find RSDP and get its address, and then find XSDT @@ -232,11 +261,14 @@ void __init acpi_boot_table_init(void) if (acpi_disabled) { if (earlycon_acpi_spcr_enable) early_init_dt_scan_chosen_stdout(); - } else { - acpi_parse_spcr(earlycon_acpi_spcr_enable, true); - if (IS_ENABLED(CONFIG_ACPI_BGRT)) - acpi_table_parse(ACPI_SIG_BGRT, acpi_parse_bgrt); + return; } + + acpi_parse_spcr(earlycon_acpi_spcr_enable, true); + if (IS_ENABLED(CONFIG_ACPI_BGRT)) + acpi_table_parse(ACPI_SIG_BGRT, acpi_parse_bgrt); + + acpi_fixup_m400_quirks(); } #ifdef CONFIG_ACPI_APEI
Adds a new ACPI init routine acpi_fixup_m400_quirks that adds a work-around for HPE ProLiant m400 APEI firmware problems. The work-around disables APEI when CONFIG_ACPI_APEI is set and m400 firmware is detected. Without this fixup m400 systems experience errors like these on startup: [Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 2 [Hardware Error]: event severity: fatal [Hardware Error]: Error 0, type: fatal [Hardware Error]: section_type: memory error [Hardware Error]: error_status: 0x0000000000001300 [Hardware Error]: error_type: 10, invalid address Kernel panic - not syncing: Fatal hardware error! Signed-off-by: Geoff Levand <geoff@infradead.org> --- Hi, It seems unlikely there will be any m400 firmware updates to fix this problem. APEI support is desired for new ARM64 servers coming to market. Distros are now forced to have their own fixes, not enable APEI, or let m400 users fend for themselves. Please consider. -Geoff arch/arm64/kernel/acpi.c | 40 ++++++++++++++++++++++++++++++++++++---- 1 file changed, 36 insertions(+), 4 deletions(-)