mbox series

[v18,0/6] Add ARMv8 RAS virtualization support in QEMU

Message ID 20190906083152.25716-1-zhengxiang9@huawei.com (mailing list archive)
Headers show
Series Add ARMv8 RAS virtualization support in QEMU | expand

Message

Xiang Zheng Sept. 6, 2019, 8:31 a.m. UTC 
In the ARMv8 platform, the CPU error types are synchronous external abort(SEA)
and SError Interrupt (SEI). If exception happens in guest, sometimes it's better
for guest to perform the recovery, because host does not know the detailed
information of guest. For example, if an exception happens in a user-space
application within guest, host does not know which application encounters
errors.

For the ARMv8 SEA/SEI, KVM or host kernel delivers SIGBUS to notify userspace.
After user space gets the notification, it will record the CPER into guest GHES
buffer and inject an exception or IRQ into guest.

In the current implementation, if the type of SIGBUS is BUS_MCEERR_AR, we will
treat it as a synchronous exception, and notify guest with ARMv8 SEA
notification type after recording CPER into guest.

This series of patches are based on Qemu 4.1, which include two parts:
1. Generate APEI/GHES table.
2. Handle the SIGBUS signal, record the CPER in runtime and fill it into guest
   memory, then notify guest according to the type of SIGBUS.

The whole solution was suggested by James(james.morse@arm.com); The solution of
APEI section was suggested by Laszlo(lersek@redhat.com).
Show some discussions in [1].

This series of patches have already been tested on ARM64 platform with RAS
feature enabled:
Show the APEI part verification result in [2].
Show the BUS_MCEERR_AR SIGBUS handling verification result in [3].

---

Since Dongjiu is too busy to do this work, I will finish the rest work on behalf
of him.

---
Change since v17:
1. Improve some commit messages and comments.
2. Fix some code-style problems.
3. Add a *ras* machine option.
4. Move HEST/GHES related structures and macros into "hw/acpi/acpi_ghes.*".
5. Move HWPoison page functions into "include/sysemu/kvm_int.h".
6. Fix some bugs.
7. Improve the design document.

Change since v16:
1. check whether ACPI table is enabled when handling the memory error in the SIGBUS handler.

Change since v15:
1. Add a doc-comment in the proper format for 'include/exec/ram_addr.h'
2. Remove write_part_cpustate_to_list() because there is another bug fix patch
   has been merged "arm: Allow system registers for KVM guests to be changed by QEMU code"
3. Add some comments for kvm_inject_arm_sea() in 'target/arm/kvm64.c'
4. Compare the arm_current_el() return value to 0,1,2,3, not to PSTATE_MODE_* constants.
5. Change the RAS support wasn't introduced before 4.1 QEMU version.
6. Move the no_ras flag  patch to begin in this series

Change since v14:
1. Remove the BUS_MCEERR_AO handling logic because this asynchronous signal was masked by main thread
2. Address some Igor Mammedov's comments(ACPI part)
   1) change the comments for the enum AcpiHestNotifyType definition and remove ditto in patch 1
   2) change some patch commit messages and separate "APEI GHES table generation" patch to more patches.
3. Address some peter's comments(arm64 Synchronous External Abort injection)
   1) change some code notes
   2) using arm_current_el() for current EL
   2) use the helper functions for those (syn_data_abort_*).

Change since v13:
1. Move the patches that set guest ESR and inject virtual SError out of this series
2. Clean and optimize the APEI part patches
3. Update the commit messages and add some comments for the code

Change since v12:
1. Address Paolo's comments to move HWPoisonPage definition to accel/kvm/kvm-all.c
2. Only call kvm_cpu_synchronize_state() when get the BUS_MCEERR_AR signal
3. Only add and enable GPIO-Signal and ARMv8 SEA two hardware error sources
4. Address Michael's comments to not sync SPDX from Linux kernel header file

Change since v11:
Address James's comments(james.morse@arm.com)
1. Check whether KVM has the capability to to set ESR instead of detecting host CPU RAS capability
2. For SIGBUS_MCEERR_AR SIGBUS, use Synchronous-External-Abort(SEA) notification type
   for SIGBUS_MCEERR_AO SIGBUS, use GPIO-Signal notification


Address Shannon's comments(for ACPI part):
1. Unify hest_ghes.c and hest_ghes.h license declaration
2. Remove unnecessary including "qmp-commands.h" in hest_ghes.c
3. Unconditionally add guest APEI table based on James's comments(james.morse@arm.com)
4. Add a option to virt machine for migration compatibility. On new virt machine it's on
   by default while off for old ones, we enabled it since 2.12
5. Refer to the ACPI spec version which introduces Hardware Error Notification first time
6. Add ACPI_HEST_NOTIFY_RESERVED notification type

Address Igor's comments(for ACPI part):
1. Add doc patch first which will describe how it's supposed to work between QEMU/firmware/guest
   OS with expected flows.
2. Move APEI diagrams into doc/spec patch
3. Remove redundant g_malloc in ghes_record_cper()
4. Use build_append_int_noprefix() API to compose whole error status block and whole APEI table,
   and try to get rid of most structures in patch 1, as they will be left unused after that
5. Reuse something like https://github.com/imammedo/qemu/commit/3d2fd6d13a3ea298d2ee814835495ce6241d085c
   to build GAS
6. Remove much offsetof() in the function
7. Build independent tables first and only then build dependent tables passing to it pointers
   to previously build table if necessary.
8. Redefine macro GHES_ACPI_HEST_NOTIFY_RESERVED to ACPI_HEST_ERROR_SOURCE_COUNT to avoid confusion


Address Peter Maydell's comments
1. linux-headers is done as a patch of their own created using scripts/update-linux-headers.sh run against a
   mainline kernel tree
2. Tested whether this patchset builds OK on aarch32
3. Abstract Hwpoison page adding code  out properly into a cpu-independent source file from target/i386/kvm.c,
   such as kvm-all.c
4. Add doc-comment formatted documentation comment for new globally-visible function prototype in a header

---
[1]:
https://lkml.org/lkml/2017/2/27/246
https://patchwork.kernel.org/patch/9633105/
https://patchwork.kernel.org/patch/9925227/

[2]:
Note: the UEFI(QEMU_EFI.fd) is needed if guest want to use ACPI table.

After guest boot up, dump the APEI table, then can see the initialized table
(1) # iasl -p ./HEST -d /sys/firmware/acpi/tables/HEST
(2) # cat HEST.dsl
    /*
     * Intel ACPI Component Architecture
     * AML/ASL+ Disassembler version 20170728 (64-bit version)
     * Copyright (c) 2000 - 2017 Intel Corporation
     *
     * Disassembly of /sys/firmware/acpi/tables/HEST, Mon Sep  5 07:59:17 2016
     *
     * ACPI Data Table [HEST]
     *
     * Format: [HexOffset DecimalOffset ByteLength]  FieldName : FieldValue
     */

    ..................................................................................
    [308h 0776   2]                Subtable Type : 000A [Generic Hardware Error Source V2]
    [30Ah 0778   2]                    Source Id : 0001
    [30Ch 0780   2]            Related Source Id : FFFF
    [30Eh 0782   1]                     Reserved : 00
    [30Fh 0783   1]                      Enabled : 01
    [310h 0784   4]       Records To Preallocate : 00000001
    [314h 0788   4]      Max Sections Per Record : 00000001
    [318h 0792   4]          Max Raw Data Length : 00001000

    [31Ch 0796  12]         Error Status Address : [Generic Address Structure]
    [31Ch 0796   1]                     Space ID : 00 [SystemMemory]
    [31Dh 0797   1]                    Bit Width : 40
    [31Eh 0798   1]                   Bit Offset : 00
    [31Fh 0799   1]         Encoded Access Width : 04 [QWord Access:64]
    [320h 0800   8]                      Address : 00000000785D0040

    [328h 0808  28]                       Notify : [Hardware Error Notification Structure]
    [328h 0808   1]                  Notify Type : 08 [SEA]
    [329h 0809   1]                Notify Length : 1C
    [32Ah 0810   2]   Configuration Write Enable : 0000
    [32Ch 0812   4]                 PollInterval : 00000000
    [330h 0816   4]                       Vector : 00000000
    [334h 0820   4]      Polling Threshold Value : 00000000
    [338h 0824   4]     Polling Threshold Window : 00000000
    [33Ch 0828   4]        Error Threshold Value : 00000000
    [340h 0832   4]       Error Threshold Window : 00000000

    [344h 0836   4]    Error Status Block Length : 00001000
    [348h 0840  12]            Read Ack Register : [Generic Address Structure]
    [348h 0840   1]                     Space ID : 00 [SystemMemory]
    [349h 0841   1]                    Bit Width : 40
    [34Ah 0842   1]                   Bit Offset : 00
    [34Bh 0843   1]         Encoded Access Width : 04 [QWord Access:64]
    [34Ch 0844   8]                      Address : 00000000785D0098

    [354h 0852   8]            Read Ack Preserve : 00000000FFFFFFFE
    [35Ch 0860   8]               Read Ack Write : 0000000000000001

    .....................................................................................

(3) After a synchronous external abort(SEA) happen, Qemu receive a SIGBUS and 
    filled the CPER into guest GHES memory.  For example, according to above table,
    the address that contains the physical address of a block of memory that holds
    the error status data for this abort is 0x00000000785D0040
(4) the address for SEA notification error source is 0x785d80b0
    (qemu) xp /1 0x00000000785D0040
    00000000785d0040: 0x785d80b0

(5) check the content of generic error status block and generic error data entry
    (qemu) xp /100x 0x785d80b0
    00000000785d80b0: 0x00000001 0x00000000 0x00000000 0x00000098
    00000000785d80c0: 0x00000000 0xa5bc1114 0x4ede6f64 0x833e63b8
    00000000785d80d0: 0xb1837ced 0x00000000 0x00000300 0x00000050
    00000000785d80e0: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d80f0: 0x00000000 0x00000000 0x00000000 0x00000000
    00000000785d8100: 0x00000000 0x00000000 0x00000000 0x00004002
(6) check the OSPM's ACK value(for example SEA)
    /* Before OSPM acknowledges the error, check the ACK value */
    (qemu) xp /1 0x00000000785D0098
    00000000785d00f0: 0x00000000

    /* After OSPM acknowledges the error, check the ACK value, it change to 1 from 0 */
    (qemu) xp /1 0x00000000785D0098
    00000000785d00f0: 0x00000001

[3]: KVM deliver "BUS_MCEERR_AR" to Qemu, Qemu record the guest CPER and inject
    synchronous external abort to notify guest, then guest do the recovery.

[ 1552.516170] Synchronous External Abort: synchronous external abort (0x92000410) at 0x000000003751c6b4
[ 1553.074073] {1}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 8
[ 1553.081654] {1}[Hardware Error]: event severity: recoverable
[ 1554.034191] {1}[Hardware Error]:  Error 0, type: recoverable
[ 1554.037934] {1}[Hardware Error]:   section_type: memory error
[ 1554.513261] {1}[Hardware Error]:   physical_address: 0x0000000040fa6000
[ 1554.513944] {1}[Hardware Error]:   error_type: 0, unknown
[ 1555.041451] Memory failure: 0x40fa6: Killing mca-recover:1296 due to hardware memory corruption
[ 1555.373116] Memory failure: 0x40fa6: recovery action for dirty LRU page: Recovered

Dongjiu Geng (6):
  hw/arm/virt: Introduce RAS platform version and RAS machine option
  docs: APEI GHES generation and CPER record description
  ACPI: Add APEI GHES table generation support
  KVM: Move hwpoison page related functions into
    include/sysemu/kvm_int.h
  target-arm: kvm64: inject synchronous External Abort
  target-arm: kvm64: handle SIGBUS signal from kernel or KVM

 accel/kvm/kvm-all.c             |  33 +++
 default-configs/arm-softmmu.mak |   1 +
 docs/specs/acpi_hest_ghes.txt   |  88 ++++++
 hw/acpi/Kconfig                 |   4 +
 hw/acpi/Makefile.objs           |   1 +
 hw/acpi/acpi_ghes.c             | 462 ++++++++++++++++++++++++++++++++
 hw/acpi/aml-build.c             |   2 +
 hw/arm/virt-acpi-build.c        |  12 +
 hw/arm/virt.c                   |  33 +++
 include/hw/acpi/acpi_ghes.h     | 143 ++++++++++
 include/hw/acpi/aml-build.h     |   1 +
 include/hw/arm/virt.h           |   2 +
 include/sysemu/kvm.h            |   2 +-
 include/sysemu/kvm_int.h        |  23 ++
 target/arm/helper.c             |   2 +-
 target/arm/internals.h          |   5 +-
 target/arm/kvm.c                |   3 +
 target/arm/kvm64.c              |  73 +++++
 target/arm/tlb_helper.c         |   2 +-
 target/i386/kvm.c               |  34 ---
 20 files changed, 887 insertions(+), 39 deletions(-)
 create mode 100644 docs/specs/acpi_hest_ghes.txt
 create mode 100644 hw/acpi/acpi_ghes.c
 create mode 100644 include/hw/acpi/acpi_ghes.h

Comments

Xiang Zheng Sept. 17, 2019, 12:39 p.m. UTC | #1 
Hi all,

This patch series has been tested for both TCG and KVM scenes.

1) Test for TCG:
   - Re-compile qemu after applying the patch refered to https://patchwork.kernel.org/cover/10942757/#22640271).
   - Use command line shown below to start qemu:
        ./qemu-system-aarch64 \
                -name guest=ras \
                -machine virt,gic-version=3,ras=on \
                -cpu cortex-a57 \
                -bios /usr/share/edk2/aarch64/QEMU_EFI.fd \
                -nodefaults \
                -kernel ${GUEST_KERNEL} \
                -initrd ${GUEST_FS} \
                -append "rdinit=init console=ttyAMA0 earlycon=pl011,0x9000000" \
                -m 8192 \
                -smp 4 \
                -serial stdio \

   - Send a signal to one of the VCPU threads:
        kill -s SIGBUS 71571

   - The result of test is shown below:

    [   41.194753] {1}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 0
    [   41.197329] {1}[Hardware Error]: event severity: recoverable
    [   41.199078] {1}[Hardware Error]:  Error 0, type: recoverable
    [   41.200829] {1}[Hardware Error]:   section_type: memory error
    [   41.202603] {1}[Hardware Error]:   physical_address: 0x00000000400a1000
    [   41.204649] {1}[Hardware Error]:   error_type: 0, unknown
    [   41.206328] EDAC MC0: 1 UE Unknown on unknown label ( page:0x400a1 offset:0x0 grain:0)
    [   41.208788] Internal error: synchronous external abort: 96000410 [#1] SMP
    [   41.210879] Modules linked in:
    [   41.211823] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 4.19.0+ #8
    [   41.213698] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
    [   41.215812] pstate: 60c00085 (nZCv daIf +PAN +UAO)
    [   41.217296] pc : cpu_do_idle+0x8/0xc
    [   41.218400] lr : arch_cpu_idle+0x2c/0x1b8
    [   41.219629] sp : ffff000009f9bf00
    [   41.220649] x29: ffff000009f9bf00 x28: 0000000000000000
    [   41.222310] x27: 0000000000000000 x26: ffff8001fe471d80
    [   41.223945] x25: 0000000000000000 x24: ffff00000937ba38
    [   41.225581] x23: ffff0000090b3338 x22: ffff000009379000
    [   41.227220] x21: ffff00000937b000 x20: 0000000000000004
    [   41.228871] x19: ffff0000090a6000 x18: 0000000000000000
    [   41.230517] x17: 0000000000000000 x16: 0000000000000000
    [   41.232165] x15: 0000000000000000 x14: 0000000000000000
    [   41.233810] x13: ffff0000089f4da8 x12: 000000000000000e
    [   41.235448] x11: ffff0000089f4d80 x10: 0000000000000af0
    [   41.237101] x9 : ffff000009f9be80 x8 : ffff8001fe4728d0
    [   41.238738] x7 : 0000000000000004 x6 : ffff8001fffbaf30
    [   41.240380] x5 : ffff00000c43b940 x4 : 00008001f6f0c000
    [   41.242030] x3 : 0000000000000001 x2 : ffff000009f9bf00
    [   41.243666] x1 : ffff8001fffb82c8 x0 : ffff0000090a6018
    [   41.245306] Process swapper/2 (pid: 0, stack limit = 0x(____ptrval____))
    [   41.247378] Call trace:
    [   41.248117]  cpu_do_idle+0x8/0xc
    [   41.249111]  do_idle+0x1dc/0x2a8
    [   41.250111]  cpu_startup_entry+0x28/0x30
    [   41.251319]  secondary_start_kernel+0x180/0x1c8
    [   41.252725] Code: a8c17bfd d65f03c0 d5033f9f d503207f (d65f03c0)
    [   41.254606] ---[ end trace 221bc8a614fb5a1d ]---
    [   41.256030] Kernel panic - not syncing: Fatal exception
    [   41.257644] SMP: stopping secondary CPUs
    [   41.258912] Kernel Offset: disabled
    [   41.260011] CPU features: 0x0,22a00238
    [   41.261178] Memory Limit: none
    [   41.262122] ---[ end Kernel panic - not syncing: Fatal exception ]---

2) Test for KVM:
   - Use command line shown below to start qemu:
        ./qemu-system-aarch64 \
            -name guest=ras \
            -machine virt,accel=kvm,gic-version=3,ras=on \
            -cpu host \
            -bios /usr/share/edk2/aarch64/QEMU_EFI.fd \
            -nodefaults \
            -kernel ${GUEST_KERNEL} \
            -initrd ${GUEST_FS} \
            -append "rdinit=init console=ttyAMA0 earlycon=pl011,0x9000000" \
            -m 8192 \
            -smp 4 \
            -serial stdio \

   - Run mca-recover and get the GPA(IPA) of allocated page which would be corrupted on the later.
   - Convert the GPA to HPA and corrupt this HPA via APEI/EINJ.
   - Go back to guest and continue to read this page.

   - The result of test is shown below:

    root@genericarmv8:~/tools# ./mca-recover
    pagesize: 0x1000
    before clear cache
    flags for page 0x2317b2: uptodate active mmap anon swapbacked
    vtop(0xffff9c9e8000) = 0x2317b2000
    Hit any key to access: before read

    after read
    Access at Tue Sep 17 01:41:14 2019

    flags for page 0x2317b2: uptodate active mmap anon swapbacked
    [  403.298539] {1}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 0
    [  403.301421] {1}[Hardware Error]: event severity: recoverable
    [  403.303217] {1}[Hardware Error]:  Error 0, type: recoverable
    [  403.304920] {1}[Hardware Error]:   section_type: memory error
    [  403.306645] {1}[Hardware Error]:   physical_address: 0x00000002317b2000
    [  403.308947] {1}[Hardware Error]:   error_type: 0, unknown
    [  403.310630] WARNING: CPU: 0 PID: 510 at drivers/edac/ghes_edac.c:202 ghes_edac_report_mem_error+0x648/0xb20
    [  403.310630] Modules linked in:
    [  403.310631] CPU: 0 PID: 510 Comm: mca-recover Not tainted 4.19.0+ #8
    [  403.310632] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
    [  403.310632] pstate: 60000005 (nZCv daif -PAN -UAO)
    [  403.310632] pc : ghes_edac_report_mem_error+0x648/0xb20
    [  403.310633] lr : ghes_proc+0x3d8/0x950
    [  403.310633] sp : ffff00000c543b20
    [  403.310633] x29: ffff00000c543b50 x28: ffff8001f5918014
    [  403.310634] x27: 0000000000000000 x26: b1837ced833e63b8
    [  403.310635] x25: 430fbbc1d995e954 x24: 0000000000000002
    [  403.310636] x23: 0000000000000002 x22: ffff0000096ec000
    [  403.310637] x21: ffff000009379000 x20: ffff8001f591805c
    [  403.310638] x19: ffff8001f591e71c x18: ffffffffffffffff
    [  403.310638] x17: 0000000000000000 x16: 0000000000000000
    [  403.310639] x15: ffff000009379708 x14: 0000000000000000
    [  403.310640] x13: 0000000000000002 x12: 317b200000000000
    [  403.310641] x11: 0000000000000000 x10: 0000400200000000
    [  403.310642] x9 : 0000000000000000 x8 : 00000002540be3ff
    [  403.310642] x7 : 0000000000000000 x6 : ffff0000096dce30
    [  403.310643] x5 : 4ede6f64a5bc1114 x4 : 0000000000000000
    [  403.310644] x3 : ffff0000096ec4f0 x2 : ffff8001f591805c
    [  403.310645] x1 : 0000000000000000 x0 : 0000000000110000
    [  403.310646] Call trace:
    [  403.310646]  ghes_edac_report_mem_error+0x648/0xb20
    [  403.310646]  ghes_proc+0x3d8/0x950
    [  403.310647]  ghes_notify_sea+0x3c/0x68
    [  403.310647]  do_sea+0x9c/0x188
    [  403.310647]  do_mem_abort+0x74/0x140
    [  403.310648]  el0_da+0x24/0x28
    [  403.310648] ---[ end trace 651f1abaa6b1de2d ]---
    Recover: sig=7 si=0xffffc9bc5640 v=0xffffc9bc56c0[  403.364295] Memory failure: 0x2317b2: recovery action for dirty LRU page: Recovered
    [  403.364295] Memory failure: 0x2317b2: recovery action for dirty LRU page: Recovered

    Platform memory error at 0x(nil)
    Addr = (nil) lsb=0
    Recovery allocated new page at physical 0x232563000
    Got 2a2a2a2a
Dongjiu Geng Sept. 20, 2019, 2:07 a.m. UTC | #2 
Thanks xiang's continue upstream and test.
Hope maintainer can review it.


On 2019/9/17 20:39, Xiang Zheng wrote:
> Hi all,
> 
> This patch series has been tested for both TCG and KVM scenes.
> 
> 1) Test for TCG:
>    - Re-compile qemu after applying the patch refered to https://patchwork.kernel.org/cover/10942757/#22640271).
>    - Use command line shown below to start qemu:
>         ./qemu-system-aarch64 \
>                 -name guest=ras \
>                 -machine virt,gic-version=3,ras=on \
>                 -cpu cortex-a57 \
>                 -bios /usr/share/edk2/aarch64/QEMU_EFI.fd \
>                 -nodefaults \
>                 -kernel ${GUEST_KERNEL} \
>                 -initrd ${GUEST_FS} \
>                 -append "rdinit=init console=ttyAMA0 earlycon=pl011,0x9000000" \
>                 -m 8192 \
>                 -smp 4 \
>                 -serial stdio \
> 
>    - Send a signal to one of the VCPU threads:
>         kill -s SIGBUS 71571
> 
>    - The result of test is shown below:
> 
>     [   41.194753] {1}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 0
>     [   41.197329] {1}[Hardware Error]: event severity: recoverable
>     [   41.199078] {1}[Hardware Error]:  Error 0, type: recoverable
>     [   41.200829] {1}[Hardware Error]:   section_type: memory error
>     [   41.202603] {1}[Hardware Error]:   physical_address: 0x00000000400a1000
>     [   41.204649] {1}[Hardware Error]:   error_type: 0, unknown
>     [   41.206328] EDAC MC0: 1 UE Unknown on unknown label ( page:0x400a1 offset:0x0 grain:0)
>     [   41.208788] Internal error: synchronous external abort: 96000410 [#1] SMP
>     [   41.210879] Modules linked in:
>     [   41.211823] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 4.19.0+ #8
>     [   41.213698] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
>     [   41.215812] pstate: 60c00085 (nZCv daIf +PAN +UAO)
>     [   41.217296] pc : cpu_do_idle+0x8/0xc
>     [   41.218400] lr : arch_cpu_idle+0x2c/0x1b8
>     [   41.219629] sp : ffff000009f9bf00
>     [   41.220649] x29: ffff000009f9bf00 x28: 0000000000000000
>     [   41.222310] x27: 0000000000000000 x26: ffff8001fe471d80
>     [   41.223945] x25: 0000000000000000 x24: ffff00000937ba38
>     [   41.225581] x23: ffff0000090b3338 x22: ffff000009379000
>     [   41.227220] x21: ffff00000937b000 x20: 0000000000000004
>     [   41.228871] x19: ffff0000090a6000 x18: 0000000000000000
>     [   41.230517] x17: 0000000000000000 x16: 0000000000000000
>     [   41.232165] x15: 0000000000000000 x14: 0000000000000000
>     [   41.233810] x13: ffff0000089f4da8 x12: 000000000000000e
>     [   41.235448] x11: ffff0000089f4d80 x10: 0000000000000af0
>     [   41.237101] x9 : ffff000009f9be80 x8 : ffff8001fe4728d0
>     [   41.238738] x7 : 0000000000000004 x6 : ffff8001fffbaf30
>     [   41.240380] x5 : ffff00000c43b940 x4 : 00008001f6f0c000
>     [   41.242030] x3 : 0000000000000001 x2 : ffff000009f9bf00
>     [   41.243666] x1 : ffff8001fffb82c8 x0 : ffff0000090a6018
>     [   41.245306] Process swapper/2 (pid: 0, stack limit = 0x(____ptrval____))
>     [   41.247378] Call trace:
>     [   41.248117]  cpu_do_idle+0x8/0xc
>     [   41.249111]  do_idle+0x1dc/0x2a8
>     [   41.250111]  cpu_startup_entry+0x28/0x30
>     [   41.251319]  secondary_start_kernel+0x180/0x1c8
>     [   41.252725] Code: a8c17bfd d65f03c0 d5033f9f d503207f (d65f03c0)
>     [   41.254606] ---[ end trace 221bc8a614fb5a1d ]---
>     [   41.256030] Kernel panic - not syncing: Fatal exception
>     [   41.257644] SMP: stopping secondary CPUs
>     [   41.258912] Kernel Offset: disabled
>     [   41.260011] CPU features: 0x0,22a00238
>     [   41.261178] Memory Limit: none
>     [   41.262122] ---[ end Kernel panic - not syncing: Fatal exception ]---
> 
> 2) Test for KVM:
>    - Use command line shown below to start qemu:
>         ./qemu-system-aarch64 \
>             -name guest=ras \
>             -machine virt,accel=kvm,gic-version=3,ras=on \
>             -cpu host \
>             -bios /usr/share/edk2/aarch64/QEMU_EFI.fd \
>             -nodefaults \
>             -kernel ${GUEST_KERNEL} \
>             -initrd ${GUEST_FS} \
>             -append "rdinit=init console=ttyAMA0 earlycon=pl011,0x9000000" \
>             -m 8192 \
>             -smp 4 \
>             -serial stdio \
> 
>    - Run mca-recover and get the GPA(IPA) of allocated page which would be corrupted on the later.
>    - Convert the GPA to HPA and corrupt this HPA via APEI/EINJ.
>    - Go back to guest and continue to read this page.
> 
>    - The result of test is shown below:
> 
>     root@genericarmv8:~/tools# ./mca-recover
>     pagesize: 0x1000
>     before clear cache
>     flags for page 0x2317b2: uptodate active mmap anon swapbacked
>     vtop(0xffff9c9e8000) = 0x2317b2000
>     Hit any key to access: before read
> 
>     after read
>     Access at Tue Sep 17 01:41:14 2019
> 
>     flags for page 0x2317b2: uptodate active mmap anon swapbacked
>     [  403.298539] {1}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 0
>     [  403.301421] {1}[Hardware Error]: event severity: recoverable
>     [  403.303217] {1}[Hardware Error]:  Error 0, type: recoverable
>     [  403.304920] {1}[Hardware Error]:   section_type: memory error
>     [  403.306645] {1}[Hardware Error]:   physical_address: 0x00000002317b2000
>     [  403.308947] {1}[Hardware Error]:   error_type: 0, unknown
>     [  403.310630] WARNING: CPU: 0 PID: 510 at drivers/edac/ghes_edac.c:202 ghes_edac_report_mem_error+0x648/0xb20
>     [  403.310630] Modules linked in:
>     [  403.310631] CPU: 0 PID: 510 Comm: mca-recover Not tainted 4.19.0+ #8
>     [  403.310632] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
>     [  403.310632] pstate: 60000005 (nZCv daif -PAN -UAO)
>     [  403.310632] pc : ghes_edac_report_mem_error+0x648/0xb20
>     [  403.310633] lr : ghes_proc+0x3d8/0x950
>     [  403.310633] sp : ffff00000c543b20
>     [  403.310633] x29: ffff00000c543b50 x28: ffff8001f5918014
>     [  403.310634] x27: 0000000000000000 x26: b1837ced833e63b8
>     [  403.310635] x25: 430fbbc1d995e954 x24: 0000000000000002
>     [  403.310636] x23: 0000000000000002 x22: ffff0000096ec000
>     [  403.310637] x21: ffff000009379000 x20: ffff8001f591805c
>     [  403.310638] x19: ffff8001f591e71c x18: ffffffffffffffff
>     [  403.310638] x17: 0000000000000000 x16: 0000000000000000
>     [  403.310639] x15: ffff000009379708 x14: 0000000000000000
>     [  403.310640] x13: 0000000000000002 x12: 317b200000000000
>     [  403.310641] x11: 0000000000000000 x10: 0000400200000000
>     [  403.310642] x9 : 0000000000000000 x8 : 00000002540be3ff
>     [  403.310642] x7 : 0000000000000000 x6 : ffff0000096dce30
>     [  403.310643] x5 : 4ede6f64a5bc1114 x4 : 0000000000000000
>     [  403.310644] x3 : ffff0000096ec4f0 x2 : ffff8001f591805c
>     [  403.310645] x1 : 0000000000000000 x0 : 0000000000110000
>     [  403.310646] Call trace:
>     [  403.310646]  ghes_edac_report_mem_error+0x648/0xb20
>     [  403.310646]  ghes_proc+0x3d8/0x950
>     [  403.310647]  ghes_notify_sea+0x3c/0x68
>     [  403.310647]  do_sea+0x9c/0x188
>     [  403.310647]  do_mem_abort+0x74/0x140
>     [  403.310648]  el0_da+0x24/0x28
>     [  403.310648] ---[ end trace 651f1abaa6b1de2d ]---
>     Recover: sig=7 si=0xffffc9bc5640 v=0xffffc9bc56c0[  403.364295] Memory failure: 0x2317b2: recovery action for dirty LRU page: Recovered
>     [  403.364295] Memory failure: 0x2317b2: recovery action for dirty LRU page: Recovered
> 
>     Platform memory error at 0x(nil)
>     Addr = (nil) lsb=0
>     Recovery allocated new page at physical 0x232563000
>     Got 2a2a2a2a
> 
>
Dongjiu Geng Sept. 26, 2019, 12:57 p.m. UTC | #3 
ping.....

Hi peter/Igor/all，
  can you review these patches，thanks a lot.




--------------------------------------------------
耿东久 Geng Dongjiu
Mobile: +86-18221809728<tel:+86-18221809728>
Email: gengdongjiu@huawei.com<mailto:gengdongjiu@huawei.com>
发件人：zhengxiang (A) <zhengxiang9@huawei.com>
收件人：pbonzini <pbonzini@redhat.com>;mst <mst@redhat.com>;imammedo <imammedo@redhat.com>;shannon.zhaosl <shannon.zhaosl@gmail.com>;peter.maydell <peter.maydell@linaro.org>;lersek <lersek@redhat.com>;james.morse <james.morse@arm.com>;gengdongjiu <gengdongjiu@huawei.com>;mtosatti <mtosatti@redhat.com>;rth <rth@twiddle.net>;ehabkost <ehabkost@redhat.com>;Jonathan Cameron <jonathan.cameron@huawei.com>;xuwei (O) <xuwei5@huawei.com>;kvm <kvm@vger.kernel.org>;qemu-devel <qemu-devel@nongnu.org>;qemu-arm <qemu-arm@nongnu.org>;Linuxarm <linuxarm@huawei.com>
抄 送：Wanghaibin (D) <wanghaibin.wang@huawei.com>
时 间：2019-09-17 20:40:21
主题Re: [PATCH v18 0/6] Add ARMv8 RAS virtualization support in QEMU

Hi all,

This patch series has been tested for both TCG and KVM scenes.

1) Test for TCG:
   - Re-compile qemu after applying the patch refered to https://patchwork.kernel.org/cover/10942757/#22640271).
   - Use command line shown below to start qemu:
        ./qemu-system-aarch64 \
                -name guest=ras \
                -machine virt,gic-version=3,ras=on \
                -cpu cortex-a57 \
                -bios /usr/share/edk2/aarch64/QEMU_EFI.fd \
                -nodefaults \
                -kernel ${GUEST_KERNEL} \
                -initrd ${GUEST_FS} \
                -append "rdinit=init console=ttyAMA0 earlycon=pl011,0x9000000" \
                -m 8192 \
                -smp 4 \
                -serial stdio \

   - Send a signal to one of the VCPU threads:
        kill -s SIGBUS 71571

   - The result of test is shown below:

    [   41.194753] {1}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 0
    [   41.197329] {1}[Hardware Error]: event severity: recoverable
    [   41.199078] {1}[Hardware Error]:  Error 0, type: recoverable
    [   41.200829] {1}[Hardware Error]:   section_type: memory error
    [   41.202603] {1}[Hardware Error]:   physical_address: 0x00000000400a1000
    [   41.204649] {1}[Hardware Error]:   error_type: 0, unknown
    [   41.206328] EDAC MC0: 1 UE Unknown on unknown label ( page:0x400a1 offset:0x0 grain:0)
    [   41.208788] Internal error: synchronous external abort: 96000410 [#1] SMP
    [   41.210879] Modules linked in:
    [   41.211823] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 4.19.0+ #8
    [   41.213698] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
    [   41.215812] pstate: 60c00085 (nZCv daIf +PAN +UAO)
    [   41.217296] pc : cpu_do_idle+0x8/0xc
    [   41.218400] lr : arch_cpu_idle+0x2c/0x1b8
    [   41.219629] sp : ffff000009f9bf00
    [   41.220649] x29: ffff000009f9bf00 x28: 0000000000000000
    [   41.222310] x27: 0000000000000000 x26: ffff8001fe471d80
    [   41.223945] x25: 0000000000000000 x24: ffff00000937ba38
    [   41.225581] x23: ffff0000090b3338 x22: ffff000009379000
    [   41.227220] x21: ffff00000937b000 x20: 0000000000000004
    [   41.228871] x19: ffff0000090a6000 x18: 0000000000000000
    [   41.230517] x17: 0000000000000000 x16: 0000000000000000
    [   41.232165] x15: 0000000000000000 x14: 0000000000000000
    [   41.233810] x13: ffff0000089f4da8 x12: 000000000000000e
    [   41.235448] x11: ffff0000089f4d80 x10: 0000000000000af0
    [   41.237101] x9 : ffff000009f9be80 x8 : ffff8001fe4728d0
    [   41.238738] x7 : 0000000000000004 x6 : ffff8001fffbaf30
    [   41.240380] x5 : ffff00000c43b940 x4 : 00008001f6f0c000
    [   41.242030] x3 : 0000000000000001 x2 : ffff000009f9bf00
    [   41.243666] x1 : ffff8001fffb82c8 x0 : ffff0000090a6018
    [   41.245306] Process swapper/2 (pid: 0, stack limit = 0x(____ptrval____))
    [   41.247378] Call trace:
    [   41.248117]  cpu_do_idle+0x8/0xc
    [   41.249111]  do_idle+0x1dc/0x2a8
    [   41.250111]  cpu_startup_entry+0x28/0x30
    [   41.251319]  secondary_start_kernel+0x180/0x1c8
    [   41.252725] Code: a8c17bfd d65f03c0 d5033f9f d503207f (d65f03c0)
    [   41.254606] ---[ end trace 221bc8a614fb5a1d ]---
    [   41.256030] Kernel panic - not syncing: Fatal exception
    [   41.257644] SMP: stopping secondary CPUs
    [   41.258912] Kernel Offset: disabled
    [   41.260011] CPU features: 0x0,22a00238
    [   41.261178] Memory Limit: none
    [   41.262122] ---[ end Kernel panic - not syncing: Fatal exception ]---

2) Test for KVM:
   - Use command line shown below to start qemu:
        ./qemu-system-aarch64 \
            -name guest=ras \
            -machine virt,accel=kvm,gic-version=3,ras=on \
            -cpu host \
            -bios /usr/share/edk2/aarch64/QEMU_EFI.fd \
            -nodefaults \
            -kernel ${GUEST_KERNEL} \
            -initrd ${GUEST_FS} \
            -append "rdinit=init console=ttyAMA0 earlycon=pl011,0x9000000" \
            -m 8192 \
            -smp 4 \
            -serial stdio \

   - Run mca-recover and get the GPA(IPA) of allocated page which would be corrupted on the later.
   - Convert the GPA to HPA and corrupt this HPA via APEI/EINJ.
   - Go back to guest and continue to read this page.

   - The result of test is shown below:

    root@genericarmv8:~/tools# ./mca-recover
    pagesize: 0x1000
    before clear cache
    flags for page 0x2317b2: uptodate active mmap anon swapbacked
    vtop(0xffff9c9e8000) = 0x2317b2000
    Hit any key to access: before read

    after read
    Access at Tue Sep 17 01:41:14 2019

    flags for page 0x2317b2: uptodate active mmap anon swapbacked
    [  403.298539] {1}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 0
    [  403.301421] {1}[Hardware Error]: event severity: recoverable
    [  403.303217] {1}[Hardware Error]:  Error 0, type: recoverable
    [  403.304920] {1}[Hardware Error]:   section_type: memory error
    [  403.306645] {1}[Hardware Error]:   physical_address: 0x00000002317b2000
    [  403.308947] {1}[Hardware Error]:   error_type: 0, unknown
    [  403.310630] WARNING: CPU: 0 PID: 510 at drivers/edac/ghes_edac.c:202 ghes_edac_report_mem_error+0x648/0xb20
    [  403.310630] Modules linked in:
    [  403.310631] CPU: 0 PID: 510 Comm: mca-recover Not tainted 4.19.0+ #8
    [  403.310632] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
    [  403.310632] pstate: 60000005 (nZCv daif -PAN -UAO)
    [  403.310632] pc : ghes_edac_report_mem_error+0x648/0xb20
    [  403.310633] lr : ghes_proc+0x3d8/0x950
    [  403.310633] sp : ffff00000c543b20
    [  403.310633] x29: ffff00000c543b50 x28: ffff8001f5918014
    [  403.310634] x27: 0000000000000000 x26: b1837ced833e63b8
    [  403.310635] x25: 430fbbc1d995e954 x24: 0000000000000002
    [  403.310636] x23: 0000000000000002 x22: ffff0000096ec000
    [  403.310637] x21: ffff000009379000 x20: ffff8001f591805c
    [  403.310638] x19: ffff8001f591e71c x18: ffffffffffffffff
    [  403.310638] x17: 0000000000000000 x16: 0000000000000000
    [  403.310639] x15: ffff000009379708 x14: 0000000000000000
    [  403.310640] x13: 0000000000000002 x12: 317b200000000000
    [  403.310641] x11: 0000000000000000 x10: 0000400200000000
    [  403.310642] x9 : 0000000000000000 x8 : 00000002540be3ff
    [  403.310642] x7 : 0000000000000000 x6 : ffff0000096dce30
    [  403.310643] x5 : 4ede6f64a5bc1114 x4 : 0000000000000000
    [  403.310644] x3 : ffff0000096ec4f0 x2 : ffff8001f591805c
    [  403.310645] x1 : 0000000000000000 x0 : 0000000000110000
    [  403.310646] Call trace:
    [  403.310646]  ghes_edac_report_mem_error+0x648/0xb20
    [  403.310646]  ghes_proc+0x3d8/0x950
    [  403.310647]  ghes_notify_sea+0x3c/0x68
    [  403.310647]  do_sea+0x9c/0x188
    [  403.310647]  do_mem_abort+0x74/0x140
    [  403.310648]  el0_da+0x24/0x28
    [  403.310648] ---[ end trace 651f1abaa6b1de2d ]---
    Recover: sig=7 si=0xffffc9bc5640 v=0xffffc9bc56c0[  403.364295] Memory failure: 0x2317b2: recovery action for dirty LRU page: Recovered
    [  403.364295] Memory failure: 0x2317b2: recovery action for dirty LRU page: Recovered

    Platform memory error at 0x(nil)
    Addr = (nil) lsb=0
    Recovery allocated new page at physical 0x232563000
    Got 2a2a2a2a


--

Thanks,
Xiang
Peter Maydell Sept. 27, 2019, 2:03 p.m. UTC | #4 
On Fri, 6 Sep 2019 at 09:33, Xiang Zheng <zhengxiang9@huawei.com> wrote:
>
> In the ARMv8 platform, the CPU error types are synchronous external abort(SEA)
> and SError Interrupt (SEI). If exception happens in guest, sometimes it's better
> for guest to perform the recovery, because host does not know the detailed
> information of guest. For example, if an exception happens in a user-space
> application within guest, host does not know which application encounters
> errors.
>
> For the ARMv8 SEA/SEI, KVM or host kernel delivers SIGBUS to notify userspace.
> After user space gets the notification, it will record the CPER into guest GHES
> buffer and inject an exception or IRQ into guest.
>
> In the current implementation, if the type of SIGBUS is BUS_MCEERR_AR, we will
> treat it as a synchronous exception, and notify guest with ARMv8 SEA
> notification type after recording CPER into guest.
>
> This series of patches are based on Qemu 4.1, which include two parts:
> 1. Generate APEI/GHES table.
> 2. Handle the SIGBUS signal, record the CPER in runtime and fill it into guest
>    memory, then notify guest according to the type of SIGBUS.
>
> The whole solution was suggested by James(james.morse@arm.com); The solution of
> APEI section was suggested by Laszlo(lersek@redhat.com).
> Show some discussions in [1].
>
> This series of patches have already been tested on ARM64 platform with RAS
> feature enabled:
> Show the APEI part verification result in [2].
> Show the BUS_MCEERR_AR SIGBUS handling verification result in [3].
>
> ---
>
> Since Dongjiu is too busy to do this work, I will finish the rest work on behalf
> of him.


Thanks for picking up the work on this patchset, and sorry it's taken me
a while to get to reviewing it. I've now given review comments on the
arm parts of this, which are looking in generally good shape (my comments
are all pretty minor stuff I think). I'll have to leave the ACPI parts
to somebody else to review as that is definitely not my speciality.

thanks
-- PMM