[v3,0/5] AWS Nitro Enclave emulation support

Message

Dorjoy Chowdhury Aug. 10, 2024, 4:44 p.m. UTC

This is v3 submission for AWS Nitro Enclave emulation in QEMU. From the QEMU side
the implementation for nitro enclaves is complete. A lot of changes from v2 so all
the patches need to be reviewed. v2 is at:
https://mail.gnu.org/archive/html/qemu-devel/2024-06/msg00012.html

Changes in v3:
    - Support for virtio-nsm device
    - The EIF related logic has been removed from microvm.c i.e., the logic is
contained in enclave related code
    - For vsock emulation in nitro-enclave, now vhost-user-vsock is being used
instead of vhost-vsock (more details in the cover-letter below)
    - updated documentation accordingly

Changes in v2:
    - moved eif.c and eif.h files from hw/i386 to hw/core

Hi,

Hope everyone is doing well. This is a patch series adding AWS Nitro Enclave[1]
emulation support in QEMU. Alexander Graf is mentoring me on this work. I have
a gitlab branch where you can view the patches in the gitlab web UI for each commit:
https://gitlab.com/dorjoy03/qemu/-/tree/nitro-enclave-emulation

AWS nitro enclaves is an Amazon EC2[2] feature that allows creating isolated
execution environments, called enclaves, from Amazon EC2 instances, which are
used for processing highly sensitive data. Enclaves have no persistent storage
and no external networking. The enclave VMs are based on Firecracker microvm
and have a vhost-vsock device for communication with the parent EC2 instance
that spawned it and a Nitro Secure Module (NSM) device for cryptographic
attestation. The parent instance VM always has CID 3 while the enclave VM gets
a dynamic CID. The enclave VMs can communicate with the parent instance over
various ports to CID 3, for example, the init process inside an enclave sends a
heartbeat to port 9000 upon boot, expecting a heartbeat reply, letting the
parent instance know that the enclave VM has successfully booted.

From inside an EC2 instance, nitro-cli[3] is used to spawn an enclave VM using
an EIF (Enclave Image Format)[4] file. EIF files can be built using nitro-cli
as well. The EIF specification can be found in the README of the github
aws-nitro-enclaves-image-format repository[4]. An EIF file contains the kernel,
cmdline and ramdisk(s) in different sections which are used to boot the enclave
VM.

Adding nitro enclave emulation support in QEMU will make the life of AWS Nitro
Enclave users easier as they will be able to test their EIF images locally
without having to run real nitro enclaves which can be difficult for debugging
due to its roots in security. This will also make quick prototyping easier.

In QEMU, the new nitro-enclave machine type is implemented based on the microvm
machine type similar to how AWS Nitro Enclaves are based on Firecracker microvm.

The vsock emulation support is added using vhost-user-vsock device. This is
needed as nitro VMs always talk to parent VM (CID 3) but there is no support for
sibling VM communication in vhost-vsock. So to run nitro-enclave, a process that
does vsock emulation in user-space like vhost-device-vsock[5] from rust-vmm must
be run. I am working on adding proxying using vsock (right now it uses unix
domain socket) to the host machine in vhost-device-vsock which I will be posting
a PR to the rust-vmm repo. This will allow users to run the necessary parent VM
applications in the host machine instead of a separate VM with CID 3. Once that
work is done, I will update the documentation in docs/system/i386/nitro-enclave.rst
accordingly.

A new device virtio-nsm support has been added to QEMU. This device is built-into
the nitro-enclave VM. The virtio-nsm spec can be found here[6].

For local testing you need to generate a hello.eif image by first building
nitro-cli locally[7]. Then you can use nitro-cli to build a hello.eif image[8].
More details about testing can be found in the docs/system/i386/nitro-enclave.rst
file.

Thanks.

Regards,
Dorjoy

[1] https://docs.aws.amazon.com/enclaves/latest/user/nitro-enclave.html
[2] https://aws.amazon.com/ec2/
[3] https://docs.aws.amazon.com/enclaves/latest/user/getting-started.html
[4] https://github.com/aws/aws-nitro-enclaves-image-format?tab=readme-ov-file#enclave-image-file-eif-specification
[5] https://github.com/rust-vmm/vhost-device/tree/main/vhost-device-vsock
[6] https://lists.oasis-open.org/archives/virtio-comment/202310/msg00387.html
[7] https://github.com/aws/aws-nitro-enclaves-cli/blob/main/docs/ubuntu_20.04_how_to_install_nitro_cli_from_github_sources.md
[8] https://github.com/aws/aws-nitro-enclaves-cli/blob/main/examples/x86_64/hello/README.md

Dorjoy Chowdhury (5):
  machine/nitro-enclave: New machine type for AWS Nitro Enclaves
  machine/nitro-enclave: Add vhost-user-vsock device
  device/virtio-nsm: Support for Nitro Secure Module device
  machine/nitro-enclave: Add built-in Nitro Secure Module device
  docs/nitro-enclave: Documentation for nitro-enclave machine type

 MAINTAINERS                              |   17 +
 backends/hostmem-memfd.c                 |    2 -
 configs/devices/i386-softmmu/default.mak |    1 +
 docs/system/i386/nitro-enclave.rst       |   80 +
 hw/core/eif.c                            |  707 ++++++++
 hw/core/eif.h                            |   22 +
 hw/core/machine.c                        |   71 +-
 hw/core/meson.build                      |    3 +
 hw/i386/Kconfig                          |    6 +
 hw/i386/meson.build                      |    1 +
 hw/i386/microvm.c                        |    6 +-
 hw/i386/nitro_enclave.c                  |  299 ++++
 hw/virtio/Kconfig                        |    5 +
 hw/virtio/meson.build                    |    4 +
 hw/virtio/virtio-nsm-pci.c               |   73 +
 hw/virtio/virtio-nsm.c                   | 1929 ++++++++++++++++++++++
 include/hw/boards.h                      |    2 +
 include/hw/i386/microvm.h                |    2 +
 include/hw/i386/nitro_enclave.h          |   56 +
 include/hw/virtio/virtio-nsm.h           |   59 +
 include/sysemu/hostmem.h                 |    2 +
 21 files changed, 3311 insertions(+), 36 deletions(-)
 create mode 100644 docs/system/i386/nitro-enclave.rst
 create mode 100644 hw/core/eif.c
 create mode 100644 hw/core/eif.h
 create mode 100644 hw/i386/nitro_enclave.c
 create mode 100644 hw/virtio/virtio-nsm-pci.c
 create mode 100644 hw/virtio/virtio-nsm.c
 create mode 100644 include/hw/i386/nitro_enclave.h
 create mode 100644 include/hw/virtio/virtio-nsm.h

Comments

Dorjoy Chowdhury Aug. 18, 2024, 11:51 a.m. UTC | #1

Thanks for the reviews. I have now posted a v4 at:
https://mail.gnu.org/archive/html/qemu-devel/2024-08/msg02675.html

Regards,
Dorjoy