mbox series

[v2,00/41] xen/arm: Add Armv8-R64 MPU support to Xen - Part#1

Message ID 20230113052914.3845596-1-Penny.Zheng@arm.com (mailing list archive)
Headers show
Series xen/arm: Add Armv8-R64 MPU support to Xen - Part#1 | expand

Message

Penny Zheng Jan. 13, 2023, 5:28 a.m. UTC
The Armv8-R architecture profile was designed to support use cases
that have a high sensitivity to deterministic execution. (e.g.
Fuel Injection, Brake control, Drive trains, Motor control etc)

Arm announced Armv8-R in 2013, it is the latest generation Arm
architecture targeted at the Real-time profile. It introduces
virtualization at the highest security level while retaining the
Protected Memory System Architecture (PMSA) based on a Memory
Protection Unit (MPU). In 2020, Arm announced Cortex-R82,
which is the first Arm 64-bit Cortex-R processor based on Armv8-R64.
The latest Armv8-R64 document can be found [1]. And the features of
Armv8-R64 architecture:
  - An exception model that is compatible with the Armv8-A model
  - Virtualization with support for guest operating systems
  - PMSA virtualization using MPUs In EL2.
  - Adds support for the 64-bit A64 instruction set.
  - Supports up to 48-bit physical addressing.
  - Supports three Exception Levels (ELs)
        - Secure EL2 - The Highest Privilege
        - Secure EL1 - RichOS (MMU) or RTOS (MPU)
        - Secure EL0 - Application Workloads
 - Supports only a single Security state - Secure.
 - MPU in EL1 & EL2 is configurable, MMU in EL1 is configurable.

These patch series are implementing the Armv8-R64 MPU support
for Xen, which are based on the discussion of
"Proposal for Porting Xen to Armv8-R64 - DraftC" [2].

We will implement the Armv8-R64 and MPU support in three stages:
1. Boot Xen itself to idle thread, do not create any guests on it.
2. Support to boot MPU and MMU domains on Armv8-R64 Xen.
3. SMP and other advanced features of Xen support on Armv8-R64.

As we have not implemented guest support in part#1 series of MPU
support, Xen can not create any guest in boot time. So in this
patch serie, we provide an extra DNM-commit in the last for users
to test Xen boot to idle on MPU system.

We will split these patches to several parts, this series is the
part#1, v1 is in [3], the full PoC can be found in [4]. More software for
Armv8-R64 can be found in [5];

[1] https://developer.arm.com/documentation/ddi0600/latest
[2] https://lists.xenproject.org/archives/html/xen-devel/2022-05/msg00643.html
[3] https://lists.xenproject.org/archives/html/xen-devel/2022-11/msg00289.html
[4] https://gitlab.com/xen-project/people/weic/xen/-/tree/integration/mpu_v2
[5] https://armv8r64-refstack.docs.arm.com/en/v5.0/

Penny Zheng (28):
  xen/mpu: build up start-of-day Xen MPU memory region map
  xen/mpu: introduce helpers for MPU enablement
  xen/mpu: introduce unified function setup_early_uart to map early UART
  xen/arm64: head: Jump to the runtime mapping in enable_mm()
  xen/arm: introduce setup_mm_mappings
  xen/mpu: plump virt/maddr/mfn convertion in MPU system
  xen/mpu: introduce helper access_protection_region
  xen/mpu: populate a new region in Xen MPU mapping table
  xen/mpu: plump early_fdt_map in MPU systems
  xen/arm: move MMU-specific setup_mm to setup_mmu.c
  xen/mpu: implement MPU version of setup_mm in setup_mpu.c
  xen/mpu: initialize frametable in MPU system
  xen/mpu: introduce "mpu,xxx-memory-section"
  xen/mpu: map MPU guest memory section before static memory
    initialization
  xen/mpu: destroy an existing entry in Xen MPU memory mapping table
  xen/mpu: map device memory resource in MPU system
  xen/mpu: map boot module section in MPU system
  xen/mpu: introduce mpu_memory_section_contains for address range check
  xen/mpu: disable VMAP sub-system for MPU systems
  xen/mpu: disable FIXMAP in MPU system
  xen/mpu: implement MPU version of ioremap_xxx
  xen/mpu: free init memory in MPU system
  xen/mpu: destroy boot modules and early FDT mapping in MPU system
  xen/mpu: Use secure hypervisor timer for AArch64v8R
  xen/mpu: move MMU specific P2M code to p2m_mmu.c
  xen/mpu: implement setup_virt_paging for MPU system
  xen/mpu: re-order xen_mpumap in arch_init_finialize
  xen/mpu: add Kconfig option to enable Armv8-R AArch64 support

Wei Chen (13):
  xen/arm: remove xen_phys_start and xenheap_phys_end from config.h
  xen/arm: make ARM_EFI selectable for Arm64
  xen/arm: adjust Xen TLB helpers for Armv8-R64 PMSA
  xen/arm: add an option to define Xen start address for Armv8-R
  xen/arm64: prepare for moving MMU related code from head.S
  xen/arm64: move MMU related code from head.S to head_mmu.S
  xen/arm64: add .text.idmap for Xen identity map sections
  xen/arm: use PA == VA for EARLY_UART_VIRTUAL_ADDRESS on Armv-8R
  xen/arm: decouple copy_from_paddr with FIXMAP
  xen/arm: split MMU and MPU config files from config.h
  xen/arm: move MMU-specific memory management code to mm_mmu.c/mm_mmu.h
  xen/arm: check mapping status and attributes for MPU copy_from_paddr
  xen/mpu: make Xen boot to idle on MPU systems(DNM)

 xen/arch/arm/Kconfig                      |   44 +-
 xen/arch/arm/Makefile                     |   17 +-
 xen/arch/arm/arm64/Makefile               |    5 +
 xen/arch/arm/arm64/head.S                 |  466 +----
 xen/arch/arm/arm64/head_mmu.S             |  399 ++++
 xen/arch/arm/arm64/head_mpu.S             |  394 ++++
 xen/arch/arm/bootfdt.c                    |   13 +-
 xen/arch/arm/domain_build.c               |    4 +
 xen/arch/arm/include/asm/alternative.h    |   15 +
 xen/arch/arm/include/asm/arm64/flushtlb.h |   25 +
 xen/arch/arm/include/asm/arm64/macros.h   |   51 +
 xen/arch/arm/include/asm/arm64/mpu.h      |  174 ++
 xen/arch/arm/include/asm/arm64/sysregs.h  |   77 +
 xen/arch/arm/include/asm/config.h         |  105 +-
 xen/arch/arm/include/asm/config_mmu.h     |  112 +
 xen/arch/arm/include/asm/config_mpu.h     |   25 +
 xen/arch/arm/include/asm/cpregs.h         |    4 +-
 xen/arch/arm/include/asm/cpuerrata.h      |   12 +
 xen/arch/arm/include/asm/cpufeature.h     |    7 +
 xen/arch/arm/include/asm/early_printk.h   |   13 +
 xen/arch/arm/include/asm/fixmap.h         |   28 +-
 xen/arch/arm/include/asm/flushtlb.h       |   22 +
 xen/arch/arm/include/asm/mm.h             |   78 +-
 xen/arch/arm/include/asm/mm_mmu.h         |   77 +
 xen/arch/arm/include/asm/mm_mpu.h         |   54 +
 xen/arch/arm/include/asm/p2m.h            |   27 +-
 xen/arch/arm/include/asm/p2m_mmu.h        |   28 +
 xen/arch/arm/include/asm/processor.h      |   13 +
 xen/arch/arm/include/asm/setup.h          |   39 +
 xen/arch/arm/kernel.c                     |   31 +-
 xen/arch/arm/mm.c                         | 1340 +-----------
 xen/arch/arm/mm_mmu.c                     | 1376 +++++++++++++
 xen/arch/arm/mm_mpu.c                     | 1056 ++++++++++
 xen/arch/arm/p2m.c                        | 2282 +--------------------
 xen/arch/arm/p2m_mmu.c                    | 2257 ++++++++++++++++++++
 xen/arch/arm/p2m_mpu.c                    |  274 +++
 xen/arch/arm/platforms/Kconfig            |   16 +-
 xen/arch/arm/setup.c                      |  394 +---
 xen/arch/arm/setup_mmu.c                  |  391 ++++
 xen/arch/arm/setup_mpu.c                  |  208 ++
 xen/arch/arm/time.c                       |   14 +-
 xen/arch/arm/traps.c                      |    2 +
 xen/arch/arm/xen.lds.S                    |   10 +-
 xen/arch/x86/Kconfig                      |    1 +
 xen/common/Kconfig                        |    6 +
 xen/common/Makefile                       |    2 +-
 xen/include/xen/vmap.h                    |   93 +-
 47 files changed, 7500 insertions(+), 4581 deletions(-)
 create mode 100644 xen/arch/arm/arm64/head_mmu.S
 create mode 100644 xen/arch/arm/arm64/head_mpu.S
 create mode 100644 xen/arch/arm/include/asm/arm64/mpu.h
 create mode 100644 xen/arch/arm/include/asm/config_mmu.h
 create mode 100644 xen/arch/arm/include/asm/config_mpu.h
 create mode 100644 xen/arch/arm/include/asm/mm_mmu.h
 create mode 100644 xen/arch/arm/include/asm/mm_mpu.h
 create mode 100644 xen/arch/arm/include/asm/p2m_mmu.h
 create mode 100644 xen/arch/arm/mm_mmu.c
 create mode 100644 xen/arch/arm/mm_mpu.c
 create mode 100644 xen/arch/arm/p2m_mmu.c
 create mode 100644 xen/arch/arm/p2m_mpu.c
 create mode 100644 xen/arch/arm/setup_mmu.c
 create mode 100644 xen/arch/arm/setup_mpu.c

Comments

Jan Beulich Jan. 13, 2023, 8:54 a.m. UTC | #1
On 13.01.2023 06:28, Penny Zheng wrote:
>  xen/arch/x86/Kconfig                      |    1 +
>  xen/common/Kconfig                        |    6 +
>  xen/common/Makefile                       |    2 +-
>  xen/include/xen/vmap.h                    |   93 +-

I would like to take a look at these non-Arm changes, but I view it as not
very reasonable to wade through 40 patches just to find those changes. The
titles don't look to help in that either. For such pretty large series,
could you please help non-Arm folks by pointing out in some way where the
non-Arm changes actually are?

Thanks, Jan
Julien Grall Jan. 13, 2023, 9:16 a.m. UTC | #2
Hi,

On 13/01/2023 08:54, Jan Beulich wrote:
> On 13.01.2023 06:28, Penny Zheng wrote:
>>   xen/arch/x86/Kconfig                      |    1 +
>>   xen/common/Kconfig                        |    6 +
>>   xen/common/Makefile                       |    2 +-
>>   xen/include/xen/vmap.h                    |   93 +-
> 
> I would like to take a look at these non-Arm changes, but I view it as not
> very reasonable to wade through 40 patches just to find those changes.

Right, but that's the purpose of the different CC list on each patch. 
AFAICT, Penny respected that and you should have been CC to the three 
patches (#30, #31, #32) touching common/x86 code.

> The
> titles don't look to help in that either. For such pretty large series,
> could you please help non-Arm folks by pointing out in some way where the
> non-Arm changes actually are?

See above. I am not entirely sure what else you are requested here. Do 
you want Penny to be explicit and list the patch modified in the cover 
letter?

Cheers,
Jan Beulich Jan. 13, 2023, 9:28 a.m. UTC | #3
On 13.01.2023 10:16, Julien Grall wrote:
> On 13/01/2023 08:54, Jan Beulich wrote:
>> On 13.01.2023 06:28, Penny Zheng wrote:
>>>   xen/arch/x86/Kconfig                      |    1 +
>>>   xen/common/Kconfig                        |    6 +
>>>   xen/common/Makefile                       |    2 +-
>>>   xen/include/xen/vmap.h                    |   93 +-
>>
>> I would like to take a look at these non-Arm changes, but I view it as not
>> very reasonable to wade through 40 patches just to find those changes.
> 
> Right, but that's the purpose of the different CC list on each patch. 
> AFAICT, Penny respected that and you should have been CC to the three 
> patches (#30, #31, #32) touching common/x86 code.

Right, but I have no way to immediately see which patches I have been
Cc-ed on. Unlike you (iiuc) I'm subscribed to the list, and hence mails
all look the same whether or not I'm CC-ed. Then again I only now
realize that there are ways to filter what I've got - I'm sorry for
not having thought of this earlier.

>> The
>> titles don't look to help in that either. For such pretty large series,
>> could you please help non-Arm folks by pointing out in some way where the
>> non-Arm changes actually are?
> 
> See above. I am not entirely sure what else you are requested here. Do 
> you want Penny to be explicit and list the patch modified in the cover 
> letter?

For a large series mostly touching Arm code, calling out the
"outliers" (when patch titles don't make this clear) could certainly
help. It's not like I'm asking to do such everywhere.

Jan
Ayan Kumar Halder Jan. 24, 2023, 7:31 p.m. UTC | #4
Hi Penny,

On 13/01/2023 05:28, Penny Zheng wrote:
> CAUTION: This message has originated from an External Source. Please use proper judgment and caution when opening attachments, clicking links, or responding to this email.
>
>
> The Armv8-R architecture profile was designed to support use cases
> that have a high sensitivity to deterministic execution. (e.g.
> Fuel Injection, Brake control, Drive trains, Motor control etc)
>
> Arm announced Armv8-R in 2013, it is the latest generation Arm
> architecture targeted at the Real-time profile. It introduces
> virtualization at the highest security level while retaining the
> Protected Memory System Architecture (PMSA) based on a Memory
> Protection Unit (MPU). In 2020, Arm announced Cortex-R82,
> which is the first Arm 64-bit Cortex-R processor based on Armv8-R64.
> The latest Armv8-R64 document can be found [1]. And the features of
> Armv8-R64 architecture:
>    - An exception model that is compatible with the Armv8-A model
>    - Virtualization with support for guest operating systems
>    - PMSA virtualization using MPUs In EL2.
>    - Adds support for the 64-bit A64 instruction set.
>    - Supports up to 48-bit physical addressing.
>    - Supports three Exception Levels (ELs)
>          - Secure EL2 - The Highest Privilege
>          - Secure EL1 - RichOS (MMU) or RTOS (MPU)
>          - Secure EL0 - Application Workloads
>   - Supports only a single Security state - Secure.
>   - MPU in EL1 & EL2 is configurable, MMU in EL1 is configurable.
>
> These patch series are implementing the Armv8-R64 MPU support
> for Xen, which are based on the discussion of
> "Proposal for Porting Xen to Armv8-R64 - DraftC" [2].
>
> We will implement the Armv8-R64 and MPU support in three stages:
> 1. Boot Xen itself to idle thread, do not create any guests on it.
> 2. Support to boot MPU and MMU domains on Armv8-R64 Xen.
> 3. SMP and other advanced features of Xen support on Armv8-R64.
>
> As we have not implemented guest support in part#1 series of MPU
> support, Xen can not create any guest in boot time. So in this
> patch serie, we provide an extra DNM-commit in the last for users
> to test Xen boot to idle on MPU system.
>
> We will split these patches to several parts, this series is the
> part#1, v1 is in [3], the full PoC can be found in [4]. More software for
> Armv8-R64 can be found in [5];
>
> [1] https://developer.arm.com/documentation/ddi0600/latest
> [2] https://lists.xenproject.org/archives/html/xen-devel/2022-05/msg00643.html
> [3] https://lists.xenproject.org/archives/html/xen-devel/2022-11/msg00289.html
> [4] https://gitlab.com/xen-project/people/weic/xen/-/tree/integration/mpu_v2
> [5] https://armv8r64-refstack.docs.arm.com/en/v5.0/
>
> Penny Zheng (28):
>    xen/mpu: build up start-of-day Xen MPU memory region map
>    xen/mpu: introduce helpers for MPU enablement
>    xen/mpu: introduce unified function setup_early_uart to map early UART
>    xen/arm64: head: Jump to the runtime mapping in enable_mm()
>    xen/arm: introduce setup_mm_mappings
>    xen/mpu: plump virt/maddr/mfn convertion in MPU system
>    xen/mpu: introduce helper access_protection_region
>    xen/mpu: populate a new region in Xen MPU mapping table
>    xen/mpu: plump early_fdt_map in MPU systems
>    xen/arm: move MMU-specific setup_mm to setup_mmu.c
>    xen/mpu: implement MPU version of setup_mm in setup_mpu.c
>    xen/mpu: initialize frametable in MPU system
>    xen/mpu: introduce "mpu,xxx-memory-section"
>    xen/mpu: map MPU guest memory section before static memory
>      initialization
>    xen/mpu: destroy an existing entry in Xen MPU memory mapping table
>    xen/mpu: map device memory resource in MPU system
>    xen/mpu: map boot module section in MPU system
>    xen/mpu: introduce mpu_memory_section_contains for address range check
>    xen/mpu: disable VMAP sub-system for MPU systems
>    xen/mpu: disable FIXMAP in MPU system
>    xen/mpu: implement MPU version of ioremap_xxx
>    xen/mpu: free init memory in MPU system
>    xen/mpu: destroy boot modules and early FDT mapping in MPU system
>    xen/mpu: Use secure hypervisor timer for AArch64v8R
>    xen/mpu: move MMU specific P2M code to p2m_mmu.c
>    xen/mpu: implement setup_virt_paging for MPU system
>    xen/mpu: re-order xen_mpumap in arch_init_finialize
>    xen/mpu: add Kconfig option to enable Armv8-R AArch64 support
>
> Wei Chen (13):
>    xen/arm: remove xen_phys_start and xenheap_phys_end from config.h
>    xen/arm: make ARM_EFI selectable for Arm64
>    xen/arm: adjust Xen TLB helpers for Armv8-R64 PMSA
>    xen/arm: add an option to define Xen start address for Armv8-R
>    xen/arm64: prepare for moving MMU related code from head.S
>    xen/arm64: move MMU related code from head.S to head_mmu.S
>    xen/arm64: add .text.idmap for Xen identity map sections
>    xen/arm: use PA == VA for EARLY_UART_VIRTUAL_ADDRESS on Armv-8R
>    xen/arm: decouple copy_from_paddr with FIXMAP
>    xen/arm: split MMU and MPU config files from config.h
>    xen/arm: move MMU-specific memory management code to mm_mmu.c/mm_mmu.h
>    xen/arm: check mapping status and attributes for MPU copy_from_paddr
>    xen/mpu: make Xen boot to idle on MPU systems(DNM)
>
>   xen/arch/arm/Kconfig                      |   44 +-
>   xen/arch/arm/Makefile                     |   17 +-
>   xen/arch/arm/arm64/Makefile               |    5 +
>   xen/arch/arm/arm64/head.S                 |  466 +----
>   xen/arch/arm/arm64/head_mmu.S             |  399 ++++
>   xen/arch/arm/arm64/head_mpu.S             |  394 ++++
>   xen/arch/arm/bootfdt.c                    |   13 +-
>   xen/arch/arm/domain_build.c               |    4 +
>   xen/arch/arm/include/asm/alternative.h    |   15 +
>   xen/arch/arm/include/asm/arm64/flushtlb.h |   25 +
>   xen/arch/arm/include/asm/arm64/macros.h   |   51 +
>   xen/arch/arm/include/asm/arm64/mpu.h      |  174 ++
>   xen/arch/arm/include/asm/arm64/sysregs.h  |   77 +
>   xen/arch/arm/include/asm/config.h         |  105 +-
>   xen/arch/arm/include/asm/config_mmu.h     |  112 +
>   xen/arch/arm/include/asm/config_mpu.h     |   25 +
>   xen/arch/arm/include/asm/cpregs.h         |    4 +-
>   xen/arch/arm/include/asm/cpuerrata.h      |   12 +
>   xen/arch/arm/include/asm/cpufeature.h     |    7 +
>   xen/arch/arm/include/asm/early_printk.h   |   13 +
>   xen/arch/arm/include/asm/fixmap.h         |   28 +-
>   xen/arch/arm/include/asm/flushtlb.h       |   22 +
>   xen/arch/arm/include/asm/mm.h             |   78 +-
>   xen/arch/arm/include/asm/mm_mmu.h         |   77 +
>   xen/arch/arm/include/asm/mm_mpu.h         |   54 +
>   xen/arch/arm/include/asm/p2m.h            |   27 +-
>   xen/arch/arm/include/asm/p2m_mmu.h        |   28 +
>   xen/arch/arm/include/asm/processor.h      |   13 +
>   xen/arch/arm/include/asm/setup.h          |   39 +
>   xen/arch/arm/kernel.c                     |   31 +-
>   xen/arch/arm/mm.c                         | 1340 +-----------
>   xen/arch/arm/mm_mmu.c                     | 1376 +++++++++++++
>   xen/arch/arm/mm_mpu.c                     | 1056 ++++++++++
>   xen/arch/arm/p2m.c                        | 2282 +--------------------
>   xen/arch/arm/p2m_mmu.c                    | 2257 ++++++++++++++++++++
>   xen/arch/arm/p2m_mpu.c                    |  274 +++
>   xen/arch/arm/platforms/Kconfig            |   16 +-
>   xen/arch/arm/setup.c                      |  394 +---
>   xen/arch/arm/setup_mmu.c                  |  391 ++++
>   xen/arch/arm/setup_mpu.c                  |  208 ++
>   xen/arch/arm/time.c                       |   14 +-
>   xen/arch/arm/traps.c                      |    2 +
>   xen/arch/arm/xen.lds.S                    |   10 +-
>   xen/arch/x86/Kconfig                      |    1 +
>   xen/common/Kconfig                        |    6 +
>   xen/common/Makefile                       |    2 +-
>   xen/include/xen/vmap.h                    |   93 +-
>   47 files changed, 7500 insertions(+), 4581 deletions(-)
>   create mode 100644 xen/arch/arm/arm64/head_mmu.S
>   create mode 100644 xen/arch/arm/arm64/head_mpu.S
>   create mode 100644 xen/arch/arm/include/asm/arm64/mpu.h
>   create mode 100644 xen/arch/arm/include/asm/config_mmu.h
>   create mode 100644 xen/arch/arm/include/asm/config_mpu.h
>   create mode 100644 xen/arch/arm/include/asm/mm_mmu.h
>   create mode 100644 xen/arch/arm/include/asm/mm_mpu.h
>   create mode 100644 xen/arch/arm/include/asm/p2m_mmu.h
>   create mode 100644 xen/arch/arm/mm_mmu.c
>   create mode 100644 xen/arch/arm/mm_mpu.c
>   create mode 100644 xen/arch/arm/p2m_mmu.c
>   create mode 100644 xen/arch/arm/p2m_mpu.c
>   create mode 100644 xen/arch/arm/setup_mmu.c
>   create mode 100644 xen/arch/arm/setup_mpu.c
>
> --
> 2.25.1
>
>
I applied this serie and there were some compilation issues :-

1. drivers/passthrough/arm/smmu.c:1240:29: error: ‘P2M_ROOT_LEVEL’ 
undeclared (first use in this function)
  1240 |                 reg |= (2 - P2M_ROOT_LEVEL) << TTBCR_SL0_SHIFT;

2. drivers/passthrough/arm/smmu-v3.c:1211:24: error: ‘P2M_ROOT_LEVEL’ 
undeclared (first use in this function)
  1211 |         vtcr->sl = 2 - P2M_ROOT_LEVEL;

For the above two issues, I have disabled SMMU.

3. /scratch/ayankuma/xen_v8r_64/xen/arch/arm/arm64/head.S:470: undefined 
reference to `init_ttbr'
You might need to wrap with some #ifdef .

Can you provide me the dts and the config file with which you have tested ?

I see that the console got stuck at this line.

"(XEN) Command line: console=dtuart dtuart=serial0"

Looking into setup_static_mappings(),

     for ( uint8_t i = MSINFO_GUEST; i < MSINFO_MAX; i++ )
     {
#ifdef CONFIG_EARLY_PRINTK
         if ( i == MSINFO_DEVICE )
             /*
              * Destroy early UART mapping before mapping device memory 
section.
              * WARNING:console will be inaccessible temporarily.
              */
             destroy_xen_mappings(CONFIG_EARLY_UART_BASE_ADDRESS,
                                  CONFIG_EARLY_UART_BASE_ADDRESS + 
EARLY_UART_SIZE);
#endif
         map_mpu_memory_section_on_boot(i, mpu_section_mattr[i]); 
<<<<----- Is this expected to map "mpu,device-memory-section" ?
     }

- Ayan