mbox series

[v6,00/10] Add ACPI CPER firmware first error injection on ARM emulation

Message ID cover.1723119423.git.mchehab+huawei@kernel.org (mailing list archive)
Headers show
Series Add ACPI CPER firmware first error injection on ARM emulation | expand

Message

Mauro Carvalho Chehab Aug. 8, 2024, 12:26 p.m. UTC
Testing OS kernel ACPI APEI CPER support is tricky, as one depends on
having hardware with special-purpose BIOS and/or hardware.

With QEMU, it becomes a lot easier, as it can be done via QMP.

This series add support for injecting CPER records on ARM emulation.

The QEMU side changes add a QAPI able to do CPER error injection
on ARM, with a raw data parameter, making it very flexible.

A script is provided at the final patch implementing support for
ARM Processor CPER error injection according with ACPI 6.x and 
UEFI 2.9A/2.10 specs, via QMP.

Injecting such errors can be done using the provided script:

	$ ./scripts/ghes_inject.py arm 
	Error injected.

Produces a simple CPER register, properly handled by the Linux
Kernel:

[  794.983753] {4}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 1
[  794.984150] {4}[Hardware Error]: event severity: recoverable
[  794.984391] {4}[Hardware Error]:  Error 0, type: recoverable
[  794.984652] {4}[Hardware Error]:   section_type: ARM processor error
[  794.984926] {4}[Hardware Error]:   MIDR: 0x0000000000000000
[  794.985184] {4}[Hardware Error]:   running state: 0x0
[  794.985411] {4}[Hardware Error]:   Power State Coordination Interface state: 0
[  794.985720] {4}[Hardware Error]:   Error info structure 0:
[  794.985960] {4}[Hardware Error]:   num errors: 2
[  794.986175] {4}[Hardware Error]:    error_type: 0x02: cache error
[  794.986442] {4}[Hardware Error]:    error_info: 0x000000000091000f
[  794.986755] {4}[Hardware Error]:     transaction type: Data Access
[  794.987027] {4}[Hardware Error]:     cache error, operation type: Data write
[  794.987310] {4}[Hardware Error]:     cache level: 2
[  794.987529] {4}[Hardware Error]:     processor context not corrupted
[  794.987867] [Firmware Warn]: GHES: Unhandled processor error type 0x02: cache error

More complex use cases can be done, like:

	$ ./scripts/ghes_inject.py arm --mpidr 0x444 --running --affinity 1 \
	  --error-info 12345678 --vendor 0x13,123,4,5,1 --ctx-array 0,1,2,3,4,5 \
	  -t cache tlb bus micro-arch tlb,micro-arch
	Error injected.

[  899.181246] {5}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 1
[  899.181769] {5}[Hardware Error]: event severity: recoverable
[  899.182069] {5}[Hardware Error]:  Error 0, type: recoverable
[  899.182370] {5}[Hardware Error]:   section_type: ARM processor error
[  899.182689] {5}[Hardware Error]:   MIDR: 0x0000000000000000
[  899.182980] {5}[Hardware Error]:   Multiprocessor Affinity Register (MPIDR): 0x0000000000000000
[  899.183395] {5}[Hardware Error]:   error affinity level: 0
[  899.183683] {5}[Hardware Error]:   running state: 0x1
[  899.183962] {5}[Hardware Error]:   Power State Coordination Interface state: 0
[  899.184332] {5}[Hardware Error]:   Error info structure 0:
[  899.184610] {5}[Hardware Error]:   num errors: 2
[  899.184864] {5}[Hardware Error]:    error_type: 0x02: cache error
[  899.185181] {5}[Hardware Error]:    error_info: 0x0000000000bc614e
[  899.185504] {5}[Hardware Error]:     cache level: 2
[  899.185771] {5}[Hardware Error]:     processor context not corrupted
[  899.186082] {5}[Hardware Error]:   Error info structure 1:
[  899.186366] {5}[Hardware Error]:   num errors: 2
[  899.186613] {5}[Hardware Error]:    error_type: 0x04: TLB error
[  899.186929] {5}[Hardware Error]:    error_info: 0x000000000054007f
[  899.187236] {5}[Hardware Error]:     transaction type: Instruction
[  899.187588] {5}[Hardware Error]:     TLB error, operation type: Instruction fetch
[  899.187962] {5}[Hardware Error]:     TLB level: 1
[  899.188209] {5}[Hardware Error]:     processor context not corrupted
[  899.188535] {5}[Hardware Error]:     the error has not been corrected
[  899.188853] {5}[Hardware Error]:     PC is imprecise
[  899.189114] {5}[Hardware Error]:   Error info structure 2:
[  899.189404] {5}[Hardware Error]:   num errors: 2
[  899.189653] {5}[Hardware Error]:    error_type: 0x08: bus error
[  899.189967] {5}[Hardware Error]:    error_info: 0x00000080d6460fff
[  899.190293] {5}[Hardware Error]:     transaction type: Generic
[  899.190611] {5}[Hardware Error]:     bus error, operation type: Generic read (type of instruction or data request cannot be determined)
[  899.191174] {5}[Hardware Error]:     affinity level at which the bus error occurred: 1
[  899.191563] {5}[Hardware Error]:     processor context corrupted
[  899.191872] {5}[Hardware Error]:     the error has been corrected
[  899.192185] {5}[Hardware Error]:     PC is imprecise
[  899.192445] {5}[Hardware Error]:     Program execution can be restarted reliably at the PC associated with the error.
[  899.192939] {5}[Hardware Error]:     participation type: Local processor observed
[  899.193324] {5}[Hardware Error]:     request timed out
[  899.193596] {5}[Hardware Error]:     address space: External Memory Access
[  899.193945] {5}[Hardware Error]:     memory access attributes:0x20
[  899.194273] {5}[Hardware Error]:     access mode: secure
[  899.194544] {5}[Hardware Error]:   Error info structure 3:
[  899.194838] {5}[Hardware Error]:   num errors: 2
[  899.195088] {5}[Hardware Error]:    error_type: 0x10: micro-architectural error
[  899.195456] {5}[Hardware Error]:    error_info: 0x0000000078da03ff
[  899.195782] {5}[Hardware Error]:   Error info structure 4:
[  899.196070] {5}[Hardware Error]:   num errors: 2
[  899.196331] {5}[Hardware Error]:    error_type: 0x14: TLB error|micro-architectural error
[  899.196733] {5}[Hardware Error]:   Context info structure 0:
[  899.197024] {5}[Hardware Error]:    register context type: AArch64 EL1 context registers
[  899.197427] {5}[Hardware Error]:    00000000: 00000000 00000000
[  899.197741] {5}[Hardware Error]:   Vendor specific error info has 5 bytes:
[  899.198096] {5}[Hardware Error]:    00000000: 13 7b 04 05 01                                   .{...
[  899.198610] [Firmware Warn]: GHES: Unhandled processor error type 0x02: cache error
[  899.199000] [Firmware Warn]: GHES: Unhandled processor error type 0x04: TLB error
[  899.199388] [Firmware Warn]: GHES: Unhandled processor error type 0x08: bus error
[  899.199767] [Firmware Warn]: GHES: Unhandled processor error type 0x10: micro-architectural error
[  899.200194] [Firmware Warn]: GHES: Unhandled processor error type 0x14: TLB error|micro-architectural error

---

v6:
- PNP0C33 device creation moved to aml-build.c;
- acpi_ghes record functions now use ACPI notify parameter,
  instead of source ID;
- the number of source IDs is now automatically calculated;
- some code cleanups and function/var renames;
- some fixes and cleanups at the error injection script;
- ghes cper stub now produces an error if cper JSON is not compiled;
- Offset calculation logic for GHES was refactored;
- Updated documentation to reflect the GHES allocated size;
- Added a x-mpidr object for QOM usage;
- Added a patch making usage of x-mpidr field at ARM injection
  script;

v5:
- CPER guid is now passing as string;
- raw-data is now passed with base64 encode;
- Removed several GPIO left-overs from arm/virt.c changes;
- Lots of cleanups and improvements at the error injection script.
  It now better handles QMP dialog and doesn't print debug messages.
  Also, code was split on two modules, to make easier to add more
  error injection commands.

v4:
- CPER generation moved to happen outside QEMU;
- One patch adding support for mpidr query was removed.

v3:
- patch 1 cleanups with some comment changes and adding another place where
  the poweroff GPIO define should be used. No changes on other patches (except
  due to conflict resolution).

v2:
- added a new patch using a define for GPIO power pin;
- patch 2 changed to also use a define for generic error GPIO pin;
- a couple cleanups at patch 2 removing uneeded else clauses.



Jonathan Cameron (1):
  acpi/ghes: Add support for GED error device

Mauro Carvalho Chehab (9):
  acpi/generic_event_device: add an APEI error device
  arm/virt: Wire up a GED error device for ACPI / GHES
  qapi/ghes-cper: add an interface to do generic CPER error injection
  acpi/ghes: rework the logic to handle HEST source ID
  acpi/ghes: add support for generic error injection via QAPI
  docs: acpi_hest_ghes: fix documentation for CPER size
  scripts/ghes_inject: add a script to generate GHES error inject
  target/arm: add an experimental mpidr arm cpu property object
  scripts/arm_processor_error.py: retrieve mpidr if not filled

 MAINTAINERS                            |  10 +
 docs/specs/acpi_hest_ghes.rst          |   6 +-
 hw/acpi/Kconfig                        |   5 +
 hw/acpi/aml-build.c                    |  10 +
 hw/acpi/generic_event_device.c         |   8 +
 hw/acpi/ghes-stub.c                    |   3 +-
 hw/acpi/ghes.c                         | 308 ++++++++++++++----
 hw/acpi/ghes_cper.c                    |  45 +++
 hw/acpi/ghes_cper_stub.c               |  19 ++
 hw/acpi/meson.build                    |   2 +
 hw/arm/Kconfig                         |   5 +
 hw/arm/virt-acpi-build.c               |   1 +
 hw/arm/virt.c                          |  12 +-
 include/hw/acpi/acpi_dev_interface.h   |   1 +
 include/hw/acpi/aml-build.h            |   2 +
 include/hw/acpi/generic_event_device.h |   1 +
 include/hw/acpi/ghes.h                 |  24 +-
 include/hw/arm/virt.h                  |   1 +
 qapi/ghes-cper.json                    |  55 ++++
 qapi/meson.build                       |   1 +
 qapi/qapi-schema.json                  |   1 +
 scripts/arm_processor_error.py         | 389 ++++++++++++++++++++++
 scripts/ghes_inject.py                 |  48 +++
 scripts/qmp_helper.py                  | 431 +++++++++++++++++++++++++
 target/arm/cpu.c                       |   1 +
 target/arm/cpu.h                       |   1 +
 target/arm/helper.c                    |  10 +-
 27 files changed, 1316 insertions(+), 84 deletions(-)
 create mode 100644 hw/acpi/ghes_cper.c
 create mode 100644 hw/acpi/ghes_cper_stub.c
 create mode 100644 qapi/ghes-cper.json
 create mode 100644 scripts/arm_processor_error.py
 create mode 100755 scripts/ghes_inject.py
 create mode 100644 scripts/qmp_helper.py

Comments

Mauro Carvalho Chehab Aug. 8, 2024, 12:57 p.m. UTC | #1
Em Thu,  8 Aug 2024 14:26:26 +0200
Mauro Carvalho Chehab <mchehab+huawei@kernel.org> escreveu:

> v6:
> - PNP0C33 device creation moved to aml-build.c;
> - acpi_ghes record functions now use ACPI notify parameter,
>   instead of source ID;
> - the number of source IDs is now automatically calculated;
> - some code cleanups and function/var renames;
> - some fixes and cleanups at the error injection script;
> - ghes cper stub now produces an error if cper JSON is not compiled;
> - Offset calculation logic for GHES was refactored;
> - Updated documentation to reflect the GHES allocated size;
> - Added a x-mpidr object for QOM usage;
> - Added a patch making usage of x-mpidr field at ARM injection
>   script;

Forgot to mention: I dropped the PIN cleanup from this series, submitting 
it in separate - and it is not related anymore with this changeset:

https://lore.kernel.org/qemu-devel/ef0e7f5fca6cd94eda415ecee670c3028c671b74.1723121692.git.mchehab+huawei@kernel.org/T/#u

Thanks,
Mauro
Igor Mammedov Aug. 12, 2024, 12:18 p.m. UTC | #2
On Thu, 8 Aug 2024 14:57:35 +0200
Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:

> Em Thu,  8 Aug 2024 14:26:26 +0200
> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> escreveu:
> 
> > v6:
> > - PNP0C33 device creation moved to aml-build.c;
> > - acpi_ghes record functions now use ACPI notify parameter,
> >   instead of source ID;
> > - the number of source IDs is now automatically calculated;
> > - some code cleanups and function/var renames;
> > - some fixes and cleanups at the error injection script;
> > - ghes cper stub now produces an error if cper JSON is not compiled;
> > - Offset calculation logic for GHES was refactored;
> > - Updated documentation to reflect the GHES allocated size;
> > - Added a x-mpidr object for QOM usage;
> > - Added a patch making usage of x-mpidr field at ARM injection
> >   script;

stopping review at 5/10 and expecting a version with
GHES source to error status block mapping fetched from
HEST in guest RAM, instead of pre-calculated offsets
in source code  (as in this series) to avoid migration
issues and keeping compat plumbing manageable down the road.

> Forgot to mention: I dropped the PIN cleanup from this series, submitting 
> it in separate - and it is not related anymore with this changeset:
> 
> https://lore.kernel.org/qemu-devel/ef0e7f5fca6cd94eda415ecee670c3028c671b74.1723121692.git.mchehab+huawei@kernel.org/T/#u
> 
> Thanks,
> Mauro
>
Mauro Carvalho Chehab Aug. 13, 2024, 11:29 p.m. UTC | #3
Em Mon, 12 Aug 2024 14:18:35 +0200
Igor Mammedov <imammedo@redhat.com> escreveu:

> On Thu, 8 Aug 2024 14:57:35 +0200
> Mauro Carvalho Chehab <mchehab+huawei@kernel.org> wrote:
> 
> > Em Thu,  8 Aug 2024 14:26:26 +0200
> > Mauro Carvalho Chehab <mchehab+huawei@kernel.org> escreveu:
> >   
> > > v6:
> > > - PNP0C33 device creation moved to aml-build.c;
> > > - acpi_ghes record functions now use ACPI notify parameter,
> > >   instead of source ID;
> > > - the number of source IDs is now automatically calculated;
> > > - some code cleanups and function/var renames;
> > > - some fixes and cleanups at the error injection script;
> > > - ghes cper stub now produces an error if cper JSON is not compiled;
> > > - Offset calculation logic for GHES was refactored;
> > > - Updated documentation to reflect the GHES allocated size;
> > > - Added a x-mpidr object for QOM usage;
> > > - Added a patch making usage of x-mpidr field at ARM injection
> > >   script;  
> 
> stopping review at 5/10 and expecting a version with
> GHES source to error status block mapping fetched from
> HEST in guest RAM, instead of pre-calculated offsets
> in source code  (as in this series) to avoid migration
> issues and keeping compat plumbing manageable down the road.

Done. Sent a version 7 addressing it, and the other received
feedbacks. On this version, there are just two offsets used
during error injection:

1) the ack offset: represented relative to HEST table;
2) the CPER data offset: relative to /etc/hardware_errors table.

Thanks,
Mauro