diff mbox series

[v4,01/18] nitro_enclaves: Add ioctl interface definition

Message ID 20200622200329.52996-2-andraprs@amazon.com (mailing list archive)
State New, archived
Headers show
Series Add support for Nitro Enclaves | expand

Commit Message

Paraschiv, Andra-Irina June 22, 2020, 8:03 p.m. UTC 
The Nitro Enclaves driver handles the enclave lifetime management. This
includes enclave creation, termination and setting up its resources such
as memory and CPU.

An enclave runs alongside the VM that spawned it. It is abstracted as a
process running in the VM that launched it. The process interacts with
the NE driver, that exposes an ioctl interface for creating an enclave
and setting up its resources.

Signed-off-by: Alexandru Vasile <lexnv@amazon.com>
Signed-off-by: Andra Paraschiv <andraprs@amazon.com>
---
Changelog

v3 -> v4

* Decouple NE ioctl interface from KVM API.
* Add NE API version and the corresponding ioctl call.
* Add enclave / image load flags options.

v2 -> v3

* Remove the GPL additional wording as SPDX-License-Identifier is
  already in place.

v1 -> v2

* Add ioctl for getting enclave image load metadata.
* Update NE_ENCLAVE_START ioctl name to NE_START_ENCLAVE.
* Add entry in Documentation/userspace-api/ioctl/ioctl-number.rst for NE
  ioctls.
* Update NE ioctls definition based on the updated ioctl range for major
  and minor.
---
 .../userspace-api/ioctl/ioctl-number.rst      |   5 +-
 include/linux/nitro_enclaves.h                |  11 ++
 include/uapi/linux/nitro_enclaves.h           | 137 ++++++++++++++++++
 3 files changed, 152 insertions(+), 1 deletion(-)
 create mode 100644 include/linux/nitro_enclaves.h
 create mode 100644 include/uapi/linux/nitro_enclaves.h

Comments

Stefan Hajnoczi June 23, 2020, 8:56 a.m. UTC | #1 
On Mon, Jun 22, 2020 at 11:03:12PM +0300, Andra Paraschiv wrote:
> diff --git a/include/uapi/linux/nitro_enclaves.h b/include/uapi/linux/nitro_enclaves.h
> new file mode 100644
> index 000000000000..3270eb939a97
> --- /dev/null
> +++ b/include/uapi/linux/nitro_enclaves.h
> @@ -0,0 +1,137 @@
> +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
> +/*
> + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.
> + */
> +
> +#ifndef _UAPI_LINUX_NITRO_ENCLAVES_H_
> +#define _UAPI_LINUX_NITRO_ENCLAVES_H_
> +
> +#include <linux/types.h>
> +
> +/* Nitro Enclaves (NE) Kernel Driver Interface */
> +
> +#define NE_API_VERSION (1)
> +
> +/**
> + * The command is used to get the version of the NE API. This way the user space
> + * processes can be aware of the feature sets provided by the NE kernel driver.
> + *
> + * The NE API version is returned as result of this ioctl call.
> + */
> +#define NE_GET_API_VERSION _IO(0xAE, 0x20)
> +
> +/**
> + * The command is used to create a slot that is associated with an enclave VM.
> + *
> + * The generated unique slot id is a read parameter of this command. An enclave
> + * file descriptor is returned as result of this ioctl call. The enclave fd can
> + * be further used with ioctl calls to set vCPUs and memory regions, then start
> + * the enclave.
> + */
> +#define NE_CREATE_VM _IOR(0xAE, 0x21, __u64)

Information that would be useful for the ioctls:

1. Which fd the ioctl must be invoked on (/dev/nitro-enclaves, enclave fd, vCPU fd)

2. Errnos and their meanings

3. Which state(s) the ioctls may be invoked in (e.g. enclave created/started/etc)

> +/* User memory region flags */
> +
> +/* Memory region for enclave general usage. */
> +#define NE_DEFAULT_MEMORY_REGION (0x00)
> +
> +/* Memory region to be set for an enclave (write). */
> +struct ne_user_memory_region {
> +	/**
> +	 * Flags to determine the usage for the memory region (write).
> +	 */
> +	__u64 flags;

Where is the write flag defined?

I guess it's supposed to be:

  #define NE_USER_MEMORY_REGION_FLAG_WRITE (0x01)
Paraschiv, Andra-Irina June 24, 2020, 2:02 p.m. UTC | #2 
On 23/06/2020 11:56, Stefan Hajnoczi wrote:
> On Mon, Jun 22, 2020 at 11:03:12PM +0300, Andra Paraschiv wrote:
>> diff --git a/include/uapi/linux/nitro_enclaves.h b/include/uapi/linux/nitro_enclaves.h
>> new file mode 100644
>> index 000000000000..3270eb939a97
>> --- /dev/null
>> +++ b/include/uapi/linux/nitro_enclaves.h
>> @@ -0,0 +1,137 @@
>> +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
>> +/*
>> + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.
>> + */
>> +
>> +#ifndef _UAPI_LINUX_NITRO_ENCLAVES_H_
>> +#define _UAPI_LINUX_NITRO_ENCLAVES_H_
>> +
>> +#include <linux/types.h>
>> +
>> +/* Nitro Enclaves (NE) Kernel Driver Interface */
>> +
>> +#define NE_API_VERSION (1)
>> +
>> +/**
>> + * The command is used to get the version of the NE API. This way the user space
>> + * processes can be aware of the feature sets provided by the NE kernel driver.
>> + *
>> + * The NE API version is returned as result of this ioctl call.
>> + */
>> +#define NE_GET_API_VERSION _IO(0xAE, 0x20)
>> +
>> +/**
>> + * The command is used to create a slot that is associated with an enclave VM.
>> + *
>> + * The generated unique slot id is a read parameter of this command. An enclave
>> + * file descriptor is returned as result of this ioctl call. The enclave fd can
>> + * be further used with ioctl calls to set vCPUs and memory regions, then start
>> + * the enclave.
>> + */
>> +#define NE_CREATE_VM _IOR(0xAE, 0x21, __u64)
> Information that would be useful for the ioctls:
>
> 1. Which fd the ioctl must be invoked on (/dev/nitro-enclaves, enclave fd, vCPU fd)
>
> 2. Errnos and their meanings
>
> 3. Which state(s) the ioctls may be invoked in (e.g. enclave created/started/etc)

I'll include this info in v5. Indeed, that's useful for the user space 
tooling that interacts with the kernel driver, in addition to the code 
review itself and future refs, to understand how it works.

>
>> +/* User memory region flags */
>> +
>> +/* Memory region for enclave general usage. */
>> +#define NE_DEFAULT_MEMORY_REGION (0x00)
>> +
>> +/* Memory region to be set for an enclave (write). */
>> +struct ne_user_memory_region {
>> +	/**
>> +	 * Flags to determine the usage for the memory region (write).
>> +	 */
>> +	__u64 flags;
> Where is the write flag defined?
>
> I guess it's supposed to be:
>
>    #define NE_USER_MEMORY_REGION_FLAG_WRITE (0x01)

For now, the flags field is included in the NE ioctl interface for 
extensions, it is not part of the NE PCI device interface yet.

The enclave image is copied into enclave memory before the enclave 
memory is carved out of the primary / parent VM. After carving it out 
(when the command request to add memory is sent to the PCI device and it 
is successfully completed), there will be faults if the enclave memory 
is written from the primary / parent VM.

Ah, and just as a note, that "read" / "write" in parentheses means that 
a certain data structure / field is read / written by user space. I 
updated to use "in" / "out" instead of "read" / "write" in v5.

Thank you.

Andra




Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.
Stefan Hajnoczi June 25, 2020, 1:29 p.m. UTC | #3 
On Wed, Jun 24, 2020 at 05:02:54PM +0300, Paraschiv, Andra-Irina wrote:
> On 23/06/2020 11:56, Stefan Hajnoczi wrote:
> > On Mon, Jun 22, 2020 at 11:03:12PM +0300, Andra Paraschiv wrote:
> > > +/* User memory region flags */
> > > +
> > > +/* Memory region for enclave general usage. */
> > > +#define NE_DEFAULT_MEMORY_REGION (0x00)
> > > +
> > > +/* Memory region to be set for an enclave (write). */
> > > +struct ne_user_memory_region {
> > > +	/**
> > > +	 * Flags to determine the usage for the memory region (write).
> > > +	 */
> > > +	__u64 flags;
> > Where is the write flag defined?
> > 
> > I guess it's supposed to be:
> > 
> >    #define NE_USER_MEMORY_REGION_FLAG_WRITE (0x01)
> 
> For now, the flags field is included in the NE ioctl interface for
> extensions, it is not part of the NE PCI device interface yet.
...
> Ah, and just as a note, that "read" / "write" in parentheses means that a
> certain data structure / field is read / written by user space. I updated to
> use "in" / "out" instead of "read" / "write" in v5.

Oops, I got confused. I thought "(write)" was an example of a flag that
can be set on the memory region. Now I realize "write" means this field
is an input to the ioctl. :)

Thanks for updating the docs.

Stefan
Paraschiv, Andra-Irina June 25, 2020, 5:42 p.m. UTC | #4 
On 25/06/2020 16:29, Stefan Hajnoczi wrote:
> On Wed, Jun 24, 2020 at 05:02:54PM +0300, Paraschiv, Andra-Irina wrote:
>> On 23/06/2020 11:56, Stefan Hajnoczi wrote:
>>> On Mon, Jun 22, 2020 at 11:03:12PM +0300, Andra Paraschiv wrote:
>>>> +/* User memory region flags */
>>>> +
>>>> +/* Memory region for enclave general usage. */
>>>> +#define NE_DEFAULT_MEMORY_REGION (0x00)
>>>> +
>>>> +/* Memory region to be set for an enclave (write). */
>>>> +struct ne_user_memory_region {
>>>> +	/**
>>>> +	 * Flags to determine the usage for the memory region (write).
>>>> +	 */
>>>> +	__u64 flags;
>>> Where is the write flag defined?
>>>
>>> I guess it's supposed to be:
>>>
>>>     #define NE_USER_MEMORY_REGION_FLAG_WRITE (0x01)
>> For now, the flags field is included in the NE ioctl interface for
>> extensions, it is not part of the NE PCI device interface yet.
> ...
>> Ah, and just as a note, that "read" / "write" in parentheses means that a
>> certain data structure / field is read / written by user space. I updated to
>> use "in" / "out" instead of "read" / "write" in v5.
> Oops, I got confused. I thought "(write)" was an example of a flag that
> can be set on the memory region. Now I realize "write" means this field
> is an input to the ioctl. :)
>
> Thanks for updating the docs.

I was thinking this may be the case. :) Should be less confusing now, 
with the "in / out" updates.

Thanks also for feedback.

Andra



Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.
Alexander Graf July 2, 2020, 3:24 p.m. UTC | #5 
On 22.06.20 22:03, Andra Paraschiv wrote:
> The Nitro Enclaves driver handles the enclave lifetime management. This
> includes enclave creation, termination and setting up its resources such
> as memory and CPU.
> 
> An enclave runs alongside the VM that spawned it. It is abstracted as a
> process running in the VM that launched it. The process interacts with
> the NE driver, that exposes an ioctl interface for creating an enclave
> and setting up its resources.
> 
> Signed-off-by: Alexandru Vasile <lexnv@amazon.com>
> Signed-off-by: Andra Paraschiv <andraprs@amazon.com>

Reviewed-by: Alexander Graf <graf@amazon.com>


Alex



Amazon Development Center Germany GmbH
Krausenstr. 38
10117 Berlin
Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss
Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B
Sitz: Berlin
Ust-ID: DE 289 237 879
Paraschiv, Andra-Irina July 4, 2020, 8:09 a.m. UTC | #6 
On 02/07/2020 18:24, Alexander Graf wrote:
>
>
> On 22.06.20 22:03, Andra Paraschiv wrote:
>> The Nitro Enclaves driver handles the enclave lifetime management. This
>> includes enclave creation, termination and setting up its resources such
>> as memory and CPU.
>>
>> An enclave runs alongside the VM that spawned it. It is abstracted as a
>> process running in the VM that launched it. The process interacts with
>> the NE driver, that exposes an ioctl interface for creating an enclave
>> and setting up its resources.
>>
>> Signed-off-by: Alexandru Vasile <lexnv@amazon.com>
>> Signed-off-by: Andra Paraschiv <andraprs@amazon.com>
>
> Reviewed-by: Alexander Graf <graf@amazon.com>

Added. Thanks for reviewing the group of patches so far.

Andra



Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.
diff mbox series

Patch

diff --git a/Documentation/userspace-api/ioctl/ioctl-number.rst b/Documentation/userspace-api/ioctl/ioctl-number.rst
index 59472cd6a11d..783440c6719b 100644
--- a/Documentation/userspace-api/ioctl/ioctl-number.rst
+++ b/Documentation/userspace-api/ioctl/ioctl-number.rst
@@ -328,8 +328,11 @@  Code  Seq#    Include File                                           Comments
 0xAC  00-1F  linux/raw.h
 0xAD  00                                                             Netfilter device in development:
                                                                      <mailto:rusty@rustcorp.com.au>
-0xAE  all    linux/kvm.h                                             Kernel-based Virtual Machine
+0xAE  00-1F  linux/kvm.h                                             Kernel-based Virtual Machine
                                                                      <mailto:kvm@vger.kernel.org>
+0xAE  40-FF  linux/kvm.h                                             Kernel-based Virtual Machine
+                                                                     <mailto:kvm@vger.kernel.org>
+0xAE  20-3F  linux/nitro_enclaves.h                                  Nitro Enclaves
 0xAF  00-1F  linux/fsl_hypervisor.h                                  Freescale hypervisor
 0xB0  all                                                            RATIO devices in development:
                                                                      <mailto:vgo@ratio.de>
diff --git a/include/linux/nitro_enclaves.h b/include/linux/nitro_enclaves.h
new file mode 100644
index 000000000000..d91ef2bfdf47
--- /dev/null
+++ b/include/linux/nitro_enclaves.h
@@ -0,0 +1,11 @@ 
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ */
+
+#ifndef _LINUX_NITRO_ENCLAVES_H_
+#define _LINUX_NITRO_ENCLAVES_H_
+
+#include <uapi/linux/nitro_enclaves.h>
+
+#endif /* _LINUX_NITRO_ENCLAVES_H_ */
diff --git a/include/uapi/linux/nitro_enclaves.h b/include/uapi/linux/nitro_enclaves.h
new file mode 100644
index 000000000000..3270eb939a97
--- /dev/null
+++ b/include/uapi/linux/nitro_enclaves.h
@@ -0,0 +1,137 @@ 
+/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
+/*
+ * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ */
+
+#ifndef _UAPI_LINUX_NITRO_ENCLAVES_H_
+#define _UAPI_LINUX_NITRO_ENCLAVES_H_
+
+#include <linux/types.h>
+
+/* Nitro Enclaves (NE) Kernel Driver Interface */
+
+#define NE_API_VERSION (1)
+
+/**
+ * The command is used to get the version of the NE API. This way the user space
+ * processes can be aware of the feature sets provided by the NE kernel driver.
+ *
+ * The NE API version is returned as result of this ioctl call.
+ */
+#define NE_GET_API_VERSION _IO(0xAE, 0x20)
+
+/**
+ * The command is used to create a slot that is associated with an enclave VM.
+ *
+ * The generated unique slot id is a read parameter of this command. An enclave
+ * file descriptor is returned as result of this ioctl call. The enclave fd can
+ * be further used with ioctl calls to set vCPUs and memory regions, then start
+ * the enclave.
+ */
+#define NE_CREATE_VM _IOR(0xAE, 0x21, __u64)
+
+/**
+ * The command is used to set a vCPU for an enclave. A CPU pool needs to be set
+ * for enclave usage, before calling this function. CPU 0 and its siblings need
+ * to remain available for the primary / parent VM, so they cannot be set for
+ * an enclave.
+ *
+ * The vCPU id is a write / read parameter. If its value is 0, then a CPU is
+ * chosen from the enclave CPU pool and returned via this parameter. A vCPU file
+ * descriptor is returned as result of this ioctl call.
+ */
+#define NE_CREATE_VCPU _IOWR(0xAE, 0x22, __u32)
+
+/**
+ * The command is used to get information needed for in-memory enclave image
+ * loading e.g. offset in enclave memory to start placing the enclave image.
+ *
+ * The image load info is a write / read parameter. It includes info provided
+ * by the caller - flags - and returns the offset in enclave memory where to
+ * start placing the enclave image.
+ */
+#define NE_GET_IMAGE_LOAD_INFO _IOWR(0xAE, 0x23, struct ne_image_load_info)
+
+/**
+ * The command is used to set a memory region for an enclave, given the
+ * allocated memory from the userspace.
+ *
+ * The user memory region is a write parameter. It includes info provided
+ * by the caller - flags, memory size and userspace address.
+ */
+#define NE_SET_USER_MEMORY_REGION _IOW(0xAE, 0x24, struct ne_user_memory_region)
+
+/**
+ * The command is used to trigger enclave start after the enclave resources,
+ * such as memory and CPU, have been set.
+ *
+ * The enclave start info is a write / read parameter. It includes info provided
+ * by the caller - enclave cid and flags - and returns the cid (if input cid is
+ * 0).
+ */
+#define NE_START_ENCLAVE _IOWR(0xAE, 0x25, struct ne_enclave_start_info)
+
+/* Image load info flags */
+
+/* Enclave Image Format (EIF) */
+#define NE_EIF_IMAGE (0x01)
+
+/* Info necessary for in-memory enclave image loading (write / read). */
+struct ne_image_load_info {
+	/**
+	 * Flags to determine the enclave image type (e.g. Enclave Image Format
+	 * - EIF) (write).
+	 */
+	__u64 flags;
+
+	/**
+	 * Offset in enclave memory where to start placing the enclave image
+	 * (read).
+	 */
+	__u64 memory_offset;
+};
+
+/* User memory region flags */
+
+/* Memory region for enclave general usage. */
+#define NE_DEFAULT_MEMORY_REGION (0x00)
+
+/* Memory region to be set for an enclave (write). */
+struct ne_user_memory_region {
+	/**
+	 * Flags to determine the usage for the memory region (write).
+	 */
+	__u64 flags;
+
+	/**
+	 * The size, in bytes, of the memory region to be set for an enclave
+	 * (write).
+	 */
+	__u64 memory_size;
+
+	/**
+	 * The start of the userspace allocated memory of the memory region to
+	 * set for an enclave (write).
+	 */
+	__u64 userspace_addr;
+};
+
+/* Enclave start info flags */
+
+/* Start enclave in debug mode. */
+#define NE_ENCLAVE_DEBUG_MODE (0x01)
+
+/* Setup info necessary for enclave start (write / read). */
+struct ne_enclave_start_info {
+	/* Flags for the enclave to start with (e.g. debug mode) (write). */
+	__u64 flags;
+
+	/**
+	 * Context ID (CID) for the enclave vsock device. If 0 as input, the
+	 * CID is autogenerated by the hypervisor and returned back as output
+	 * by the driver (write / read).
+	 */
+	__u64 enclave_cid;
+};
+
+#endif /* _UAPI_LINUX_NITRO_ENCLAVES_H_ */