From patchwork Fri May 22 06:29:29 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paraschiv, Andra-Irina" X-Patchwork-Id: 11564729 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8C7EA14C0 for ; Fri, 22 May 2020 06:30:47 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6DD3B20814 for ; Fri, 22 May 2020 06:30:47 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="QqEMPO5Q" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728253AbgEVGao (ORCPT ); Fri, 22 May 2020 02:30:44 -0400 Received: from smtp-fw-9101.amazon.com ([207.171.184.25]:33642 "EHLO smtp-fw-9101.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728243AbgEVGan (ORCPT ); Fri, 22 May 2020 02:30:43 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1590129043; x=1621665043; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=xNKNBXliT9IaYCIFmyg2+Tw4otdxx+qtzW56Ci2KiNA=; b=QqEMPO5QrelyGJKoA0RzZ3KGfBjUWJolRymj4PhhsBkKj0cR8R4xUusH MkJdM+aX6dAmNFXvxP8a+6/4PJ184uUlL1aMIOY/O8vxZ4pmPfeWmROfS zEHT3H/ggkJ76y+zsWFf/eL40a4PGveyPW/g7L47qq8m396zmrljzJ+qJ c=; IronPort-SDR: yoBQ2n0N4EqS8ZKXRmuKdxaFcyeC45AH0GZ0/E6edd8z2HWMkZEQSv/uqfRArPdwBioEEHPZ07 TjNn1KWUITRg== X-IronPort-AV: E=Sophos;i="5.73,420,1583193600"; d="scan'208";a="36944277" Received: from sea32-co-svc-lb4-vlan3.sea.corp.amazon.com (HELO email-inbound-relay-1d-f273de60.us-east-1.amazon.com) ([10.47.23.38]) by smtp-border-fw-out-9101.sea19.amazon.com with ESMTP; 22 May 2020 06:30:42 +0000 Received: from EX13MTAUEA002.ant.amazon.com (iad55-ws-svc-p15-lb9-vlan2.iad.amazon.com [10.40.159.162]) by email-inbound-relay-1d-f273de60.us-east-1.amazon.com (Postfix) with ESMTPS id D7CC7A2504; Fri, 22 May 2020 06:30:39 +0000 (UTC) Received: from EX13D16EUB003.ant.amazon.com (10.43.166.99) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Fri, 22 May 2020 06:30:10 +0000 Received: from 38f9d34ed3b1.ant.amazon.com (10.43.161.175) by EX13D16EUB003.ant.amazon.com (10.43.166.99) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Fri, 22 May 2020 06:30:01 +0000 From: Andra Paraschiv To: CC: Anthony Liguori , Benjamin Herrenschmidt , Colm MacCarthaigh , "Bjoern Doebel" , David Woodhouse , "Frank van der Linden" , Alexander Graf , "Martin Pohlack" , Matt Wilson , Paolo Bonzini , Balbir Singh , Stefano Garzarella , Stefan Hajnoczi , Stewart Smith , Uwe Dannowski , , , Andra Paraschiv Subject: [PATCH v2 01/18] nitro_enclaves: Add ioctl interface definition Date: Fri, 22 May 2020 09:29:29 +0300 Message-ID: <20200522062946.28973-2-andraprs@amazon.com> X-Mailer: git-send-email 2.20.1 (Apple Git-117) In-Reply-To: <20200522062946.28973-1-andraprs@amazon.com> References: <20200522062946.28973-1-andraprs@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.161.175] X-ClientProxiedBy: EX13D20UWC004.ant.amazon.com (10.43.162.41) To EX13D16EUB003.ant.amazon.com (10.43.166.99) Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The Nitro Enclaves driver handles the enclave lifetime management. This includes enclave creation, termination and setting up its resources such as memory and CPU. An enclave runs alongside the VM that spawned it. It is abstracted as a process running in the VM that launched it. The process interacts with the NE driver, that exposes an ioctl interface for creating an enclave and setting up its resources. Include part of the KVM ioctls in the provided ioctl interface, with additional NE ioctl commands that e.g. triggers the enclave run. Signed-off-by: Alexandru Vasile Signed-off-by: Andra Paraschiv --- .../userspace-api/ioctl/ioctl-number.rst | 5 +- include/linux/nitro_enclaves.h | 23 ++++++ include/uapi/linux/nitro_enclaves.h | 77 +++++++++++++++++++ 3 files changed, 104 insertions(+), 1 deletion(-) create mode 100644 include/linux/nitro_enclaves.h create mode 100644 include/uapi/linux/nitro_enclaves.h diff --git a/Documentation/userspace-api/ioctl/ioctl-number.rst b/Documentation/userspace-api/ioctl/ioctl-number.rst index f759edafd938..8a19b5e871d3 100644 --- a/Documentation/userspace-api/ioctl/ioctl-number.rst +++ b/Documentation/userspace-api/ioctl/ioctl-number.rst @@ -325,8 +325,11 @@ Code Seq# Include File Comments 0xAC 00-1F linux/raw.h 0xAD 00 Netfilter device in development: -0xAE all linux/kvm.h Kernel-based Virtual Machine +0xAE 00-1F linux/kvm.h Kernel-based Virtual Machine +0xAE 40-FF linux/kvm.h Kernel-based Virtual Machine + +0xAE 20-3F linux/nitro_enclaves.h Nitro Enclaves 0xAF 00-1F linux/fsl_hypervisor.h Freescale hypervisor 0xB0 all RATIO devices in development: diff --git a/include/linux/nitro_enclaves.h b/include/linux/nitro_enclaves.h new file mode 100644 index 000000000000..7e593a9fbf8c --- /dev/null +++ b/include/linux/nitro_enclaves.h @@ -0,0 +1,23 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms and conditions of the GNU General Public License, + * version 2, as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, see . + */ + +#ifndef _LINUX_NITRO_ENCLAVES_H_ +#define _LINUX_NITRO_ENCLAVES_H_ + +#include + +#endif /* _LINUX_NITRO_ENCLAVES_H_ */ diff --git a/include/uapi/linux/nitro_enclaves.h b/include/uapi/linux/nitro_enclaves.h new file mode 100644 index 000000000000..98ba59f15b52 --- /dev/null +++ b/include/uapi/linux/nitro_enclaves.h @@ -0,0 +1,77 @@ +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ +/* + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms and conditions of the GNU General Public License, + * version 2, as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, see . + */ + +#ifndef _UAPI_LINUX_NITRO_ENCLAVES_H_ +#define _UAPI_LINUX_NITRO_ENCLAVES_H_ + +#include +#include + +/* Nitro Enclaves (NE) Kernel Driver Interface */ + +/** + * The command is used to get information needed for in-memory enclave image + * loading e.g. offset in enclave memory to start placing the enclave image. + * + * The image load metadata is an in / out data structure. It includes info + * provided by the caller - flags - and returns the offset in enclave memory + * where to start placing the enclave image. + */ +#define NE_GET_IMAGE_LOAD_METADATA _IOWR(0xAE, 0x20, struct image_load_metadata) + +/** + * The command is used to trigger enclave start after the enclave resources, + * such as memory and CPU, have been set. + * + * The enclave start metadata is an in / out data structure. It includes info + * provided by the caller - enclave cid and flags - and returns the slot uid + * and the cid (if input cid is 0). + */ +#define NE_START_ENCLAVE _IOWR(0xAE, 0x21, struct enclave_start_metadata) + +/* Metadata necessary for in-memory enclave image loading. */ +struct image_load_metadata { + /** + * Flags to determine the enclave image type e.g. Enclave Image Format + * (EIF) (in). + */ + __u64 flags; + + /** + * Offset in enclave memory where to start placing the enclave image + * (out). + */ + __u64 memory_offset; +}; + +/* Setup metadata necessary for enclave start. */ +struct enclave_start_metadata { + /* Flags for the enclave to start with (e.g. debug mode) (in). */ + __u64 flags; + + /** + * Context ID (CID) for the enclave vsock device. If 0 as input, the + * CID is autogenerated by the hypervisor and returned back as output + * by the driver (in/out). + */ + __u64 enclave_cid; + + /* Slot unique id mapped to the enclave to start (out). */ + __u64 slot_uid; +}; + +#endif /* _UAPI_LINUX_NITRO_ENCLAVES_H_ */ From patchwork Fri May 22 06:29:30 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paraschiv, Andra-Irina" X-Patchwork-Id: 11564727 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E812114B7 for ; Fri, 22 May 2020 06:30:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id CB0E420814 for ; Fri, 22 May 2020 06:30:42 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="nNzK8xHC" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728210AbgEVGaf (ORCPT ); Fri, 22 May 2020 02:30:35 -0400 Received: from smtp-fw-6002.amazon.com ([52.95.49.90]:45947 "EHLO smtp-fw-6002.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727942AbgEVGaf (ORCPT ); Fri, 22 May 2020 02:30:35 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1590129032; x=1621665032; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Zigc/8odGd+W8vSHjYtkbRyAUXKiGMivG8/OH2O/D0o=; b=nNzK8xHCJXpYOdoIW1X3A3QWpeTZ/aeO9QmJjVencecnlvZIIjbmwT/4 esV1rOVQeGbCo9MWz1OoxZD6z1elgTTYnxJ5Z+jMWwXjTysbYytFYHdaQ AascJQh1EI3RbPHgnqrY6rtLEXCeC5ZMAdB+QKAzubdTXqjIOE7lOXAto Y=; IronPort-SDR: pnH41RZKsigY8ktS6v51kOxebfYVEEXauCuFJ/kndUea/HEjgGUylZRhRQPJmbtsZSWQiXCnr6 uaTHwzHSdGKw== X-IronPort-AV: E=Sophos;i="5.73,420,1583193600"; d="scan'208";a="31607886" Received: from iad12-co-svc-p1-lb1-vlan3.amazon.com (HELO email-inbound-relay-1a-715bee71.us-east-1.amazon.com) ([10.43.8.6]) by smtp-border-fw-out-6002.iad6.amazon.com with ESMTP; 22 May 2020 06:30:21 +0000 Received: from EX13MTAUEA002.ant.amazon.com (iad55-ws-svc-p15-lb9-vlan2.iad.amazon.com [10.40.159.162]) by email-inbound-relay-1a-715bee71.us-east-1.amazon.com (Postfix) with ESMTPS id 569E7A25F7; Fri, 22 May 2020 06:30:19 +0000 (UTC) Received: from EX13D16EUB003.ant.amazon.com (10.43.166.99) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Fri, 22 May 2020 06:30:17 +0000 Received: from 38f9d34ed3b1.ant.amazon.com (10.43.161.175) by EX13D16EUB003.ant.amazon.com (10.43.166.99) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Fri, 22 May 2020 06:30:09 +0000 From: Andra Paraschiv To: CC: Anthony Liguori , Benjamin Herrenschmidt , Colm MacCarthaigh , "Bjoern Doebel" , David Woodhouse , "Frank van der Linden" , Alexander Graf , "Martin Pohlack" , Matt Wilson , Paolo Bonzini , Balbir Singh , Stefano Garzarella , Stefan Hajnoczi , Stewart Smith , Uwe Dannowski , , , Andra Paraschiv Subject: [PATCH v2 02/18] nitro_enclaves: Define the PCI device interface Date: Fri, 22 May 2020 09:29:30 +0300 Message-ID: <20200522062946.28973-3-andraprs@amazon.com> X-Mailer: git-send-email 2.20.1 (Apple Git-117) In-Reply-To: <20200522062946.28973-1-andraprs@amazon.com> References: <20200522062946.28973-1-andraprs@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.161.175] X-ClientProxiedBy: EX13D20UWC004.ant.amazon.com (10.43.162.41) To EX13D16EUB003.ant.amazon.com (10.43.166.99) Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The Nitro Enclaves (NE) driver communicates with a new PCI device, that is exposed to a virtual machine (VM) and handles commands meant for handling enclaves lifetime e.g. creation, termination, setting memory regions. The communication with the PCI device is handled using a MMIO space and MSI-X interrupts. This device communicates with the hypervisor on the host, where the VM that spawned the enclave itself run, e.g. to launch a VM that is used for the enclave. Define the MMIO space of the PCI device, the commands that are provided by this device. Add an internal data structure used as private data for the PCI device driver and the functions for the PCI device init / uninit and command requests handling. Signed-off-by: Alexandru-Catalin Vasile Signed-off-by: Alexandru Ciobotaru Signed-off-by: Andra Paraschiv --- drivers/virt/nitro_enclaves/ne_pci_dev.h | 266 +++++++++++++++++++++++ 1 file changed, 266 insertions(+) create mode 100644 drivers/virt/nitro_enclaves/ne_pci_dev.h diff --git a/drivers/virt/nitro_enclaves/ne_pci_dev.h b/drivers/virt/nitro_enclaves/ne_pci_dev.h new file mode 100644 index 000000000000..1d5d5f4872d6 --- /dev/null +++ b/drivers/virt/nitro_enclaves/ne_pci_dev.h @@ -0,0 +1,266 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms and conditions of the GNU General Public License, + * version 2, as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, see . + */ + +#ifndef _NE_PCI_DEV_H_ +#define _NE_PCI_DEV_H_ + +#include +#include +#include +#include +#include +#include + +/* Nitro Enclaves (NE) PCI device identifier */ + +#define PCI_DEVICE_ID_NE (0xe4c1) +#define PCI_BAR_NE (0x03) + +/* Device registers */ + +/** + * (1 byte) Register to notify the device that the driver is using it + * (Read/Write). + */ +#define NE_ENABLE (0x0000) +#define NE_ENABLE_OFF (0x00) +#define NE_ENABLE_ON (0x01) + +/* (2 bytes) Register to select the device run-time version (Read/Write). */ +#define NE_VERSION (0x0002) +#define NE_VERSION_MAX (0x0001) + +/** + * (4 bytes) Register to notify the device what command was requested + * (Write-Only). + */ +#define NE_COMMAND (0x0004) + +/** + * (4 bytes) Register to notify the driver that a reply or a device event + * is available (Read-Only): + * - Lower half - command reply counter + * - Higher half - out-of-band device event counter + */ +#define NE_EVTCNT (0x000c) +#define NE_EVTCNT_REPLY_SHIFT (0) +#define NE_EVTCNT_REPLY_MASK (0x0000ffff) +#define NE_EVTCNT_REPLY(cnt) (((cnt) & NE_EVTCNT_REPLY_MASK) >> \ + NE_EVTCNT_REPLY_SHIFT) +#define NE_EVTCNT_EVENT_SHIFT (16) +#define NE_EVTCNT_EVENT_MASK (0xffff0000) +#define NE_EVTCNT_EVENT(cnt) (((cnt) & NE_EVTCNT_EVENT_MASK) >> \ + NE_EVTCNT_EVENT_SHIFT) + +/* (240 bytes) Buffer for sending the command request payload (Read/Write). */ +#define NE_SEND_DATA (0x0010) + +/* (240 bytes) Buffer for receiving the command reply payload (Read-Only). */ +#define NE_RECV_DATA (0x0100) + +/* Device MMIO buffer sizes */ + +/* 240 bytes for send / recv buffer. */ +#define NE_SEND_DATA_SIZE (240) +#define NE_RECV_DATA_SIZE (240) + +/* MSI-X interrupt vectors */ + +/* MSI-X vector used for command reply notification. */ +#define NE_VEC_REPLY (0) + +/* MSI-X vector used for out-of-band events e.g. enclave crash. */ +#define NE_VEC_EVENT (1) + +/* Device command types. */ +enum ne_pci_dev_cmd_type { + INVALID_CMD = 0, + ENCLAVE_START = 1, + ENCLAVE_GET_SLOT = 2, + ENCLAVE_STOP = 3, + SLOT_ALLOC = 4, + SLOT_FREE = 5, + SLOT_ADD_MEM = 6, + SLOT_ADD_VCPU = 7, + SLOT_COUNT = 8, + NEXT_SLOT = 9, + SLOT_INFO = 10, + SLOT_ADD_BULK_VCPUS = 11, + MAX_CMD, +}; + +/* Device commands - payload structure for requests and replies. */ + +struct enclave_start_req { + /* Slot unique id mapped to the enclave to start. */ + u64 slot_uid; + + /** + * Context ID (CID) for the enclave vsock device. + * If 0, CID is autogenerated. + */ + u64 enclave_cid; + + /* Flags for the enclave to start with (e.g. debug mode). */ + u64 flags; +} __attribute__ ((__packed__)); + +struct enclave_get_slot_req { + /* Context ID (CID) for the enclave vsock device. */ + u64 enclave_cid; +} __attribute__ ((__packed__)); + +struct enclave_stop_req { + /* Slot unique id mapped to the enclave to stop. */ + u64 slot_uid; +} __attribute__ ((__packed__)); + +struct slot_alloc_req { + /* In order to avoid weird sizeof edge cases. */ + u8 unused; +} __attribute__ ((__packed__)); + +struct slot_free_req { + /* Slot unique id mapped to the slot to free. */ + u64 slot_uid; +} __attribute__ ((__packed__)); + +struct slot_add_mem_req { + /* Slot unique id mapped to the slot to add the memory region to. */ + u64 slot_uid; + + /* Physical address of the memory region to add to the slot. */ + u64 paddr; + + /* Memory size, in bytes, of the memory region to add to the slot. */ + u64 size; +} __attribute__ ((__packed__)); + +struct slot_add_vcpu_req { + /* Slot unique id mapped to the slot to add the vCPU to. */ + u64 slot_uid; + + /* vCPU ID of the CPU to add to the enclave. */ + u32 vcpu_id; +} __attribute__ ((__packed__)); + +struct slot_count_req { + /* In order to avoid weird sizeof edge cases. */ + u8 unused; +} __attribute__ ((__packed__)); + +struct next_slot_req { + /* Slot unique id of the next slot in the iteration. */ + u64 slot_uid; +} __attribute__ ((__packed__)); + +struct slot_info_req { + /* Slot unique id mapped to the slot to get information about. */ + u64 slot_uid; +} __attribute__ ((__packed__)); + +struct slot_add_bulk_vcpus_req { + /* Slot unique id mapped to the slot to add vCPUs to. */ + u64 slot_uid; + + /* Number of vCPUs to add to the slot. */ + u64 nr_vcpus; +} __attribute__ ((__packed__)); + +struct ne_pci_dev_cmd_reply { + s32 rc; + + /* Valid for all commands except SLOT_COUNT. */ + u64 slot_uid; + + /* Valid for ENCLAVE_START command. */ + u64 enclave_cid; + + /* Valid for SLOT_COUNT command. */ + u64 slot_count; + + /* Valid for SLOT_ALLOC and SLOT_INFO commands. */ + u64 mem_regions; + + /* Valid for SLOT_INFO command. */ + u64 mem_size; + + /* Valid for SLOT_INFO command. */ + u64 nr_vcpus; + + /* Valid for SLOT_INFO command. */ + u64 flags; + + /* Valid for SLOT_INFO command. */ + u16 state; +} __attribute__ ((__packed__)); + +/* Nitro Enclaves (NE) PCI device. */ +struct ne_pci_dev { + /* Variable set if a reply has been sent by the PCI device. */ + atomic_t cmd_reply_avail; + + /* Wait queue for handling command reply from the PCI device. */ + wait_queue_head_t cmd_reply_wait_q; + + /* List of the enclaves managed by the PCI device. */ + struct list_head enclaves_list; + + /* Mutex for accessing the list of enclaves. */ + struct mutex enclaves_list_mutex; + + /** + * Work queue for handling out-of-band events triggered by the Nitro + * Hypervisor which require enclave state scanning and propagation to + * the enclave process. + */ + struct workqueue_struct *event_wq; + + /* MMIO region of the PCI device. */ + void __iomem *iomem_base; + + /* Work item for every received out-of-band event. */ + struct work_struct notify_work; + + /* Mutex for accessing the PCI dev MMIO space. */ + struct mutex pci_dev_mutex; +}; + +/** + * ne_do_request - Submit command request to the PCI device based on the command + * type and retrieve the associated reply. + * + * This function uses the ne_pci_dev mutex to handle one command at a time. + * + * @pdev: PCI device to send the command to and receive the reply from. + * @cmd_type: command type of the request sent to the PCI device. + * @cmd_request: command request payload. + * @cmd_request_size: size of the command request payload. + * @cmd_reply: command reply payload. + * @cmd_reply_size: size of the command reply payload. + * + * @returns: 0 on success, negative return value on failure. + */ +int ne_do_request(struct pci_dev *pdev, enum ne_pci_dev_cmd_type cmd_type, + void *cmd_request, size_t cmd_request_size, + struct ne_pci_dev_cmd_reply *cmd_reply, + size_t cmd_reply_size); + +/* Nitro Enclaves (NE) PCI device driver */ +extern struct pci_driver ne_pci_driver; + +#endif /* _NE_PCI_DEV_H_ */ From patchwork Fri May 22 06:29:31 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paraschiv, Andra-Irina" X-Patchwork-Id: 11564733 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CB06314B7 for ; Fri, 22 May 2020 06:31:00 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B4596207FB for ; Fri, 22 May 2020 06:31:00 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="BMtl+fW3" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728322AbgEVGa5 (ORCPT ); Fri, 22 May 2020 02:30:57 -0400 Received: from smtp-fw-2101.amazon.com ([72.21.196.25]:20638 "EHLO smtp-fw-2101.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728200AbgEVGa5 (ORCPT ); Fri, 22 May 2020 02:30:57 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1590129057; x=1621665057; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=+IEnUDauVK6Kxis8XNouhki7DlHBWdmNgib7opoIkYo=; b=BMtl+fW3ebTxb1ziSoJCOgjsJoaweb+zvSvWJeKu1BF6SvoNZIIfmcob ck418BtCmyG/x+sjX0tdeQ+ULy3GLx/Yq0qJB8WAgDxIxFpQqrjkNfX/d RaCQWxK1P1Elvg2hiiS8XcQ66FMZdUHQndrGFER2u1Xa5ekkh5AdUSOEx o=; IronPort-SDR: J/nNhtzoSGxgMBoLAiy5oQWC7pMfbpckNL0V81d/xhBAURoPBmKEaAD0R118eiHhh8qQOB17KU 2GSwKoNZgGJw== X-IronPort-AV: E=Sophos;i="5.73,420,1583193600"; d="scan'208";a="31775372" Received: from iad12-co-svc-p1-lb1-vlan2.amazon.com (HELO email-inbound-relay-1d-74cf8b49.us-east-1.amazon.com) ([10.43.8.2]) by smtp-border-fw-out-2101.iad2.amazon.com with ESMTP; 22 May 2020 06:30:45 +0000 Received: from EX13MTAUEA002.ant.amazon.com (iad55-ws-svc-p15-lb9-vlan2.iad.amazon.com [10.40.159.162]) by email-inbound-relay-1d-74cf8b49.us-east-1.amazon.com (Postfix) with ESMTPS id C7AC4C0836; Fri, 22 May 2020 06:30:41 +0000 (UTC) Received: from EX13D16EUB003.ant.amazon.com (10.43.166.99) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Fri, 22 May 2020 06:30:36 +0000 Received: from 38f9d34ed3b1.ant.amazon.com (10.43.161.175) by EX13D16EUB003.ant.amazon.com (10.43.166.99) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Fri, 22 May 2020 06:30:17 +0000 From: Andra Paraschiv To: CC: Anthony Liguori , Benjamin Herrenschmidt , Colm MacCarthaigh , "Bjoern Doebel" , David Woodhouse , "Frank van der Linden" , Alexander Graf , "Martin Pohlack" , Matt Wilson , Paolo Bonzini , Balbir Singh , Stefano Garzarella , Stefan Hajnoczi , Stewart Smith , Uwe Dannowski , , , Andra Paraschiv Subject: [PATCH v2 03/18] nitro_enclaves: Define enclave info for internal bookkeeping Date: Fri, 22 May 2020 09:29:31 +0300 Message-ID: <20200522062946.28973-4-andraprs@amazon.com> X-Mailer: git-send-email 2.20.1 (Apple Git-117) In-Reply-To: <20200522062946.28973-1-andraprs@amazon.com> References: <20200522062946.28973-1-andraprs@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.161.175] X-ClientProxiedBy: EX13D20UWC004.ant.amazon.com (10.43.162.41) To EX13D16EUB003.ant.amazon.com (10.43.166.99) Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The Nitro Enclaves driver keeps an internal info per each enclave. This is needed to be able to manage enclave resources state, enclave notifications and have a reference of the PCI device that handles command requests for enclave lifetime management. Signed-off-by: Alexandru-Catalin Vasile Signed-off-by: Andra Paraschiv --- drivers/virt/nitro_enclaves/ne_misc_dev.h | 121 ++++++++++++++++++++++ 1 file changed, 121 insertions(+) create mode 100644 drivers/virt/nitro_enclaves/ne_misc_dev.h diff --git a/drivers/virt/nitro_enclaves/ne_misc_dev.h b/drivers/virt/nitro_enclaves/ne_misc_dev.h new file mode 100644 index 000000000000..9d683607502f --- /dev/null +++ b/drivers/virt/nitro_enclaves/ne_misc_dev.h @@ -0,0 +1,121 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms and conditions of the GNU General Public License, + * version 2, as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, see . + */ + +#ifndef _NE_MISC_DEV_H_ +#define _NE_MISC_DEV_H_ + +#include +#include +#include +#include +#include +#include +#include + +/* Entry in vCPU IDs list. */ +struct ne_vcpu_id { + /* CPU id associated with a given slot, apic id on x86. */ + u32 vcpu_id; + + struct list_head vcpu_id_list_entry; +}; + +/* Entry in memory regions list. */ +struct ne_mem_region { + struct list_head mem_region_list_entry; + + /* Number of pages that make up the memory region. */ + unsigned long nr_pages; + + /* Pages that make up the user space memory region. */ + struct page **pages; +}; + +/* Per-enclave data used for enclave lifetime management. */ +struct ne_enclave { + /** + * CPU pool with siblings of already allocated CPUs to an enclave. + * This is used when a CPU pool is set, to be able to know the CPU + * siblings for the hyperthreading (HT) setup. + */ + cpumask_var_t cpu_siblings; + + struct list_head enclave_list_entry; + + /* Mutex for accessing this internal state. */ + struct mutex enclave_info_mutex; + + /** + * Wait queue used for out-of-band event notifications + * triggered from the PCI device event handler to the enclave + * process via the poll function. + */ + wait_queue_head_t eventq; + + /* Variable used to determine if the out-of-band event was triggered. */ + bool has_event; + + /** + * The maximum number of memory regions that can be handled by the + * lower levels. + */ + u64 max_mem_regions; + + /* Enclave memory regions list. */ + struct list_head mem_regions_list; + + /* Enclave process abstraction mm data struct. */ + struct mm_struct *mm; + + /* Number of memory regions associated with the enclave. */ + u64 nr_mem_regions; + + /* Number of vcpus associated with the enclave. */ + u64 nr_vcpus; + + /* PCI device used for enclave lifetime management. */ + struct pci_dev *pdev; + + /* Slot unique id mapped to the enclave. */ + u64 slot_uid; + + /* Enclave state, updated during enclave lifetime. */ + u16 state; + + /* Enclave vCPUs list. */ + struct list_head vcpu_ids_list; +}; + +/* States available for an enclave. */ +enum ne_state { + /* NE_START_ENCLAVE ioctl was never issued for the enclave. */ + NE_STATE_INIT = 0, + + /** + * NE_START_ENCLAVE ioctl was issued and the enclave is running + * as expected. + */ + NE_STATE_RUNNING = 2, + + /* Enclave exited without userspace interaction. */ + NE_STATE_STOPPED = U16_MAX, +}; + +/* Nitro Enclaves (NE) misc device */ +extern struct miscdevice ne_miscdevice; + +#endif /* _NE_MISC_DEV_H_ */ From patchwork Fri May 22 06:29:32 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paraschiv, Andra-Irina" X-Patchwork-Id: 11564731 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B2537138A for ; Fri, 22 May 2020 06:30:51 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 95E1120776 for ; Fri, 22 May 2020 06:30:51 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="WkfGRHr5" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728288AbgEVGas (ORCPT ); Fri, 22 May 2020 02:30:48 -0400 Received: from smtp-fw-9102.amazon.com ([207.171.184.29]:41297 "EHLO smtp-fw-9102.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728243AbgEVGaq (ORCPT ); Fri, 22 May 2020 02:30:46 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1590129046; x=1621665046; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=39me77MSOHxwXqDrOp0ImYHZUNXF9EMGemsZ9hzrmME=; b=WkfGRHr5j8lMQ3FmVaeF/cjdbXB1mSXcg4E1okkofzz0aisafnc6ucRp di195R7JjtPB6zup139y1qUAbwuCL1PsdJIaf7WGBTShSqK0Umhmf5rTf 58hrSlnpGmEjUdIMEp706OrLQjhJpU+ldEetljxc7WN2v2ORdGb8v0VhH g=; IronPort-SDR: QRsjsSaA6999NG6AZNcPK6ysbfZPIrKFf5dfRdQ+8HBiSSBWVOHQnKfx30qqxVgQwWquNli0po UohXakuxJtig== X-IronPort-AV: E=Sophos;i="5.73,420,1583193600"; d="scan'208";a="45244095" Received: from sea32-co-svc-lb4-vlan3.sea.corp.amazon.com (HELO email-inbound-relay-1e-303d0b0e.us-east-1.amazon.com) ([10.47.23.38]) by smtp-border-fw-out-9102.sea19.amazon.com with ESMTP; 22 May 2020 06:30:43 +0000 Received: from EX13MTAUEA002.ant.amazon.com (iad55-ws-svc-p15-lb9-vlan3.iad.amazon.com [10.40.159.166]) by email-inbound-relay-1e-303d0b0e.us-east-1.amazon.com (Postfix) with ESMTPS id 7595EA1FE1; Fri, 22 May 2020 06:30:41 +0000 (UTC) Received: from EX13D16EUB003.ant.amazon.com (10.43.166.99) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Fri, 22 May 2020 06:30:36 +0000 Received: from 38f9d34ed3b1.ant.amazon.com (10.43.161.175) by EX13D16EUB003.ant.amazon.com (10.43.166.99) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Fri, 22 May 2020 06:30:26 +0000 From: Andra Paraschiv To: CC: Anthony Liguori , Benjamin Herrenschmidt , Colm MacCarthaigh , "Bjoern Doebel" , David Woodhouse , "Frank van der Linden" , Alexander Graf , "Martin Pohlack" , Matt Wilson , Paolo Bonzini , Balbir Singh , Stefano Garzarella , Stefan Hajnoczi , Stewart Smith , Uwe Dannowski , , , Andra Paraschiv Subject: [PATCH v2 04/18] nitro_enclaves: Init PCI device driver Date: Fri, 22 May 2020 09:29:32 +0300 Message-ID: <20200522062946.28973-5-andraprs@amazon.com> X-Mailer: git-send-email 2.20.1 (Apple Git-117) In-Reply-To: <20200522062946.28973-1-andraprs@amazon.com> References: <20200522062946.28973-1-andraprs@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.161.175] X-ClientProxiedBy: EX13D20UWC004.ant.amazon.com (10.43.162.41) To EX13D16EUB003.ant.amazon.com (10.43.166.99) Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The Nitro Enclaves PCI device is used by the kernel driver as a means of communication with the hypervisor on the host where the primary VM and the enclaves run. It handles requests with regard to enclave lifetime. Setup the PCI device driver and add support for MSI-X interrupts. Signed-off-by: Alexandru-Catalin Vasile Signed-off-by: Alexandru Ciobotaru Signed-off-by: Andra Paraschiv --- drivers/virt/nitro_enclaves/ne_pci_dev.c | 303 +++++++++++++++++++++++ 1 file changed, 303 insertions(+) create mode 100644 drivers/virt/nitro_enclaves/ne_pci_dev.c diff --git a/drivers/virt/nitro_enclaves/ne_pci_dev.c b/drivers/virt/nitro_enclaves/ne_pci_dev.c new file mode 100644 index 000000000000..8e39e30c882f --- /dev/null +++ b/drivers/virt/nitro_enclaves/ne_pci_dev.c @@ -0,0 +1,303 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms and conditions of the GNU General Public License, + * version 2, as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, see . + */ + +/* Nitro Enclaves (NE) PCI device driver. */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "ne_misc_dev.h" +#include "ne_pci_dev.h" + +#define DEFAULT_TIMEOUT_MSECS (120000) // 120 sec + +#define NE "nitro_enclaves: " + +static const struct pci_device_id ne_pci_ids[] = { + { PCI_DEVICE(PCI_VENDOR_ID_AMAZON, PCI_DEVICE_ID_NE) }, + { 0, } +}; + +MODULE_DEVICE_TABLE(pci, ne_pci_ids); + +/** + * ne_setup_msix - Setup MSI-X vectors for the PCI device. + * + * @pdev: PCI device to setup the MSI-X for. + * + * @returns: 0 on success, negative return value on failure. + */ +static int ne_setup_msix(struct pci_dev *pdev) +{ + struct ne_pci_dev *ne_pci_dev = NULL; + int nr_vecs = 0; + int rc = -EINVAL; + + if (WARN_ON(!pdev)) + return -EINVAL; + + ne_pci_dev = pci_get_drvdata(pdev); + if (WARN_ON(!ne_pci_dev)) + return -EINVAL; + + nr_vecs = pci_msix_vec_count(pdev); + if (nr_vecs < 0) { + rc = nr_vecs; + + dev_err_ratelimited(&pdev->dev, + NE "Error in getting vec count [rc=%d]\n", + rc); + + return rc; + } + + rc = pci_alloc_irq_vectors(pdev, nr_vecs, nr_vecs, PCI_IRQ_MSIX); + if (rc < 0) { + dev_err_ratelimited(&pdev->dev, + NE "Error in alloc MSI-X vecs [rc=%d]\n", + rc); + + return rc; + } + + return 0; +} + +/** + * ne_teardown_msix - Teardown MSI-X vectors for the PCI device. + * + * @pdev: PCI device to teardown the MSI-X for. + */ +static void ne_teardown_msix(struct pci_dev *pdev) +{ + struct ne_pci_dev *ne_pci_dev = NULL; + + if (WARN_ON(!pdev)) + return; + + ne_pci_dev = pci_get_drvdata(pdev); + if (WARN_ON(!ne_pci_dev)) + return; + + pci_free_irq_vectors(pdev); +} + +/** + * ne_pci_dev_enable - Select PCI device version and enable it. + * + * @pdev: PCI device to select version for and then enable. + * + * @returns: 0 on success, negative return value on failure. + */ +static int ne_pci_dev_enable(struct pci_dev *pdev) +{ + u8 dev_enable_reply = 0; + u16 dev_version_reply = 0; + struct ne_pci_dev *ne_pci_dev = NULL; + + if (WARN_ON(!pdev)) + return -EINVAL; + + ne_pci_dev = pci_get_drvdata(pdev); + if (WARN_ON(!ne_pci_dev) || WARN_ON(!ne_pci_dev->iomem_base)) + return -EINVAL; + + iowrite16(NE_VERSION_MAX, ne_pci_dev->iomem_base + NE_VERSION); + + dev_version_reply = ioread16(ne_pci_dev->iomem_base + NE_VERSION); + if (dev_version_reply != NE_VERSION_MAX) { + dev_err_ratelimited(&pdev->dev, + NE "Error in pci dev version cmd\n"); + + return -EIO; + } + + iowrite8(NE_ENABLE_ON, ne_pci_dev->iomem_base + NE_ENABLE); + + dev_enable_reply = ioread8(ne_pci_dev->iomem_base + NE_ENABLE); + if (dev_enable_reply != NE_ENABLE_ON) { + dev_err_ratelimited(&pdev->dev, + NE "Error in pci dev enable cmd\n"); + + return -EIO; + } + + return 0; +} + +/** + * ne_pci_dev_disable - Disable PCI device. + * + * @pdev: PCI device to disable. + */ +static void ne_pci_dev_disable(struct pci_dev *pdev) +{ + u8 dev_disable_reply = 0; + struct ne_pci_dev *ne_pci_dev = NULL; + const unsigned int sleep_time = 10; // 10 ms + unsigned int sleep_time_count = 0; + + if (WARN_ON(!pdev)) + return; + + ne_pci_dev = pci_get_drvdata(pdev); + if (WARN_ON(!ne_pci_dev) || WARN_ON(!ne_pci_dev->iomem_base)) + return; + + iowrite8(NE_ENABLE_OFF, ne_pci_dev->iomem_base + NE_ENABLE); + + /* + * Check for NE_ENABLE_OFF in a loop, to handle cases when the device + * state is not immediately set to disabled and going through a + * transitory state of disabling. + */ + while (sleep_time_count < DEFAULT_TIMEOUT_MSECS) { + dev_disable_reply = ioread8(ne_pci_dev->iomem_base + NE_ENABLE); + if (dev_disable_reply == NE_ENABLE_OFF) + return; + + msleep_interruptible(sleep_time); + sleep_time_count += sleep_time; + } + + dev_disable_reply = ioread8(ne_pci_dev->iomem_base + NE_ENABLE); + if (dev_disable_reply != NE_ENABLE_OFF) + dev_err_ratelimited(&pdev->dev, + NE "Error in pci dev disable cmd\n"); +} + +static int ne_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) +{ + struct ne_pci_dev *ne_pci_dev = NULL; + int rc = -EINVAL; + + ne_pci_dev = kzalloc(sizeof(*ne_pci_dev), GFP_KERNEL); + if (!ne_pci_dev) + return -ENOMEM; + + rc = pci_enable_device(pdev); + if (rc < 0) { + dev_err_ratelimited(&pdev->dev, + NE "Error in pci dev enable [rc=%d]\n", rc); + + goto free_ne_pci_dev; + } + + rc = pci_request_regions_exclusive(pdev, "ne_pci_dev"); + if (rc < 0) { + dev_err_ratelimited(&pdev->dev, + NE "Error in pci request regions [rc=%d]\n", + rc); + + goto disable_pci_dev; + } + + ne_pci_dev->iomem_base = pci_iomap(pdev, PCI_BAR_NE, 0); + if (!ne_pci_dev->iomem_base) { + rc = -ENOMEM; + + dev_err_ratelimited(&pdev->dev, + NE "Error in pci iomap [rc=%d]\n", rc); + + goto release_pci_regions; + } + + pci_set_drvdata(pdev, ne_pci_dev); + + rc = ne_setup_msix(pdev); + if (rc < 0) { + dev_err_ratelimited(&pdev->dev, + NE "Error in pci dev msix setup [rc=%d]\n", + rc); + + goto iounmap_pci_bar; + } + + ne_pci_dev_disable(pdev); + + rc = ne_pci_dev_enable(pdev); + if (rc < 0) { + dev_err_ratelimited(&pdev->dev, + NE "Error in ne_pci_dev enable [rc=%d]\n", + rc); + + goto teardown_msix; + } + + atomic_set(&ne_pci_dev->cmd_reply_avail, 0); + init_waitqueue_head(&ne_pci_dev->cmd_reply_wait_q); + INIT_LIST_HEAD(&ne_pci_dev->enclaves_list); + mutex_init(&ne_pci_dev->enclaves_list_mutex); + mutex_init(&ne_pci_dev->pci_dev_mutex); + + return 0; + +teardown_msix: + ne_teardown_msix(pdev); +iounmap_pci_bar: + pci_set_drvdata(pdev, NULL); + pci_iounmap(pdev, ne_pci_dev->iomem_base); +release_pci_regions: + pci_release_regions(pdev); +disable_pci_dev: + pci_disable_device(pdev); +free_ne_pci_dev: + kzfree(ne_pci_dev); + + return rc; +} + +static void ne_pci_remove(struct pci_dev *pdev) +{ + struct ne_pci_dev *ne_pci_dev = pci_get_drvdata(pdev); + + if (!ne_pci_dev || !ne_pci_dev->iomem_base) + return; + + ne_pci_dev_disable(pdev); + + ne_teardown_msix(pdev); + + pci_set_drvdata(pdev, NULL); + + pci_iounmap(pdev, ne_pci_dev->iomem_base); + + pci_release_regions(pdev); + + pci_disable_device(pdev); + + kzfree(ne_pci_dev); +} + +/* + * TODO: Add suspend / resume functions for power management w/ CONFIG_PM, if + * needed. + */ +struct pci_driver ne_pci_driver = { + .name = "ne_pci_dev", + .id_table = ne_pci_ids, + .probe = ne_pci_probe, + .remove = ne_pci_remove, +}; From patchwork Fri May 22 06:29:33 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paraschiv, Andra-Irina" X-Patchwork-Id: 11564735 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9134C138A for ; Fri, 22 May 2020 06:31:04 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 715B8207FB for ; Fri, 22 May 2020 06:31:04 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="BaNu6Kl0" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728315AbgEVGbB (ORCPT ); Fri, 22 May 2020 02:31:01 -0400 Received: from smtp-fw-33001.amazon.com ([207.171.190.10]:8199 "EHLO smtp-fw-33001.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728267AbgEVGa6 (ORCPT ); Fri, 22 May 2020 02:30:58 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1590129056; x=1621665056; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Oyw7kch753GXE+C8RzyOWUNgZGNNk4NQ14vTj0zqGXs=; b=BaNu6Kl03CaDe3MCGMEUPsWgn9OLj6pyQlenRcGAsg7AuO3E3DZTZwNW ptzb9M16L7m/mEGMCEj4HOH8lyfgS4KylHb5sjxOd/EvQUbwsnFsu2Jvd 5bqLxnLfimdDKWY1Je9/tUFQXO4Yb3WdHBHDSCTwySe8m3I+gAOACkyrf U=; IronPort-SDR: gGxSPSc/gSTq3iFvf7oJoC6msT+BrUDA8nMkqOmWTnv9gAR2sW6HVGsYFaz5Bus/7mF9x8SJln vFQv6WxQIcdw== X-IronPort-AV: E=Sophos;i="5.73,420,1583193600"; d="scan'208";a="46674930" Received: from sea32-co-svc-lb4-vlan2.sea.corp.amazon.com (HELO email-inbound-relay-1d-5dd976cd.us-east-1.amazon.com) ([10.47.23.34]) by smtp-border-fw-out-33001.sea14.amazon.com with ESMTP; 22 May 2020 06:30:53 +0000 Received: from EX13MTAUEA002.ant.amazon.com (iad55-ws-svc-p15-lb9-vlan3.iad.amazon.com [10.40.159.166]) by email-inbound-relay-1d-5dd976cd.us-east-1.amazon.com (Postfix) with ESMTPS id B7D31A22D3; Fri, 22 May 2020 06:30:51 +0000 (UTC) Received: from EX13D16EUB003.ant.amazon.com (10.43.166.99) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Fri, 22 May 2020 06:30:50 +0000 Received: from 38f9d34ed3b1.ant.amazon.com (10.43.162.50) by EX13D16EUB003.ant.amazon.com (10.43.166.99) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Fri, 22 May 2020 06:30:40 +0000 From: Andra Paraschiv To: CC: Anthony Liguori , Benjamin Herrenschmidt , Colm MacCarthaigh , "Bjoern Doebel" , David Woodhouse , "Frank van der Linden" , Alexander Graf , "Martin Pohlack" , Matt Wilson , Paolo Bonzini , Balbir Singh , Stefano Garzarella , Stefan Hajnoczi , Stewart Smith , Uwe Dannowski , , , Andra Paraschiv , kbuild test robot Subject: [PATCH v2 05/18] nitro_enclaves: Handle PCI device command requests Date: Fri, 22 May 2020 09:29:33 +0300 Message-ID: <20200522062946.28973-6-andraprs@amazon.com> X-Mailer: git-send-email 2.20.1 (Apple Git-117) In-Reply-To: <20200522062946.28973-1-andraprs@amazon.com> References: <20200522062946.28973-1-andraprs@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.162.50] X-ClientProxiedBy: EX13D06UWC004.ant.amazon.com (10.43.162.97) To EX13D16EUB003.ant.amazon.com (10.43.166.99) Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The Nitro Enclaves PCI device exposes a MMIO space that this driver uses to submit command requests and to receive command replies e.g. for enclave creation / termination or setting enclave resources. Add logic for handling PCI device command requests based on the given command type. Register an MSI-X interrupt vector for command reply notifications to handle this type of communication events. Signed-off-by: Alexandru-Catalin Vasile Signed-off-by: Andra Paraschiv Fix issue reported in: https://lore.kernel.org/lkml/202004231644.xTmN4Z1z%25lkp@intel.com/ Reported-by: kbuild test robot Signed-off-by: Andra Paraschiv --- drivers/virt/nitro_enclaves/ne_pci_dev.c | 284 +++++++++++++++++++++++ 1 file changed, 284 insertions(+) diff --git a/drivers/virt/nitro_enclaves/ne_pci_dev.c b/drivers/virt/nitro_enclaves/ne_pci_dev.c index 8e39e30c882f..7b2d8e1b49a5 100644 --- a/drivers/virt/nitro_enclaves/ne_pci_dev.c +++ b/drivers/virt/nitro_enclaves/ne_pci_dev.c @@ -42,6 +42,268 @@ static const struct pci_device_id ne_pci_ids[] = { MODULE_DEVICE_TABLE(pci, ne_pci_ids); +/** + * ne_submit_request - Submit command request to the PCI device based on the + * command type. + * + * This function gets called with the ne_pci_dev mutex held. + * + * @pdev: PCI device to send the command to. + * @cmd_type: command type of the request sent to the PCI device. + * @cmd_request: command request payload. + * @cmd_request_size: size of the command request payload. + * + * @returns: 0 on success, negative return value on failure. + */ +static int ne_submit_request(struct pci_dev *pdev, + enum ne_pci_dev_cmd_type cmd_type, + void *cmd_request, size_t cmd_request_size) +{ + struct ne_pci_dev *ne_pci_dev = NULL; + + if (WARN_ON(!pdev)) + return -EINVAL; + + ne_pci_dev = pci_get_drvdata(pdev); + if (WARN_ON(!ne_pci_dev) || WARN_ON(!ne_pci_dev->iomem_base)) + return -EINVAL; + + if (cmd_type <= INVALID_CMD || cmd_type >= MAX_CMD) { + dev_err_ratelimited(&pdev->dev, NE "Invalid cmd type=%u\n", + cmd_type); + + return -EINVAL; + } + + if (!cmd_request) { + dev_err_ratelimited(&pdev->dev, NE "Null cmd request\n"); + + return -EINVAL; + } + + if (cmd_request_size > NE_SEND_DATA_SIZE) { + dev_err_ratelimited(&pdev->dev, + NE "Invalid req size=%zu for cmd type=%u\n", + cmd_request_size, cmd_type); + + return -EINVAL; + } + + memcpy_toio(ne_pci_dev->iomem_base + NE_SEND_DATA, cmd_request, + cmd_request_size); + + iowrite32(cmd_type, ne_pci_dev->iomem_base + NE_COMMAND); + + return 0; +} + +/** + * ne_retrieve_reply - Retrieve reply from the PCI device. + * + * This function gets called with the ne_pci_dev mutex held. + * + * @pdev: PCI device to receive the reply from. + * @cmd_reply: command reply payload. + * @cmd_reply_size: size of the command reply payload. + * + * @returns: 0 on success, negative return value on failure. + */ +static int ne_retrieve_reply(struct pci_dev *pdev, + struct ne_pci_dev_cmd_reply *cmd_reply, + size_t cmd_reply_size) +{ + struct ne_pci_dev *ne_pci_dev = NULL; + + if (WARN_ON(!pdev)) + return -EINVAL; + + ne_pci_dev = pci_get_drvdata(pdev); + if (WARN_ON(!ne_pci_dev) || WARN_ON(!ne_pci_dev->iomem_base)) + return -EINVAL; + + if (!cmd_reply) { + dev_err_ratelimited(&pdev->dev, NE "Null cmd reply\n"); + + return -EINVAL; + } + + if (cmd_reply_size > NE_RECV_DATA_SIZE) { + dev_err_ratelimited(&pdev->dev, NE "Invalid reply size=%zu\n", + cmd_reply_size); + + return -EINVAL; + } + + memcpy_fromio(cmd_reply, ne_pci_dev->iomem_base + NE_RECV_DATA, + cmd_reply_size); + + return 0; +} + +/** + * ne_wait_for_reply - Wait for a reply of a PCI command. + * + * This function gets called with the ne_pci_dev mutex held. + * + * @pdev: PCI device for which a reply is waited. + * + * @returns: 0 on success, negative return value on failure. + */ +static int ne_wait_for_reply(struct pci_dev *pdev) +{ + struct ne_pci_dev *ne_pci_dev = NULL; + int rc = -EINVAL; + + if (WARN_ON(!pdev)) + return -EINVAL; + + ne_pci_dev = pci_get_drvdata(pdev); + if (WARN_ON(!ne_pci_dev)) + return -EINVAL; + + /* + * TODO: Update to _interruptible and handle interrupted wait event + * e.g. -ERESTARTSYS, incoming signals + add / update timeout. + */ + rc = wait_event_timeout(ne_pci_dev->cmd_reply_wait_q, + atomic_read(&ne_pci_dev->cmd_reply_avail) != 0, + msecs_to_jiffies(DEFAULT_TIMEOUT_MSECS)); + if (!rc) { + dev_err_ratelimited(&pdev->dev, + NE "Timeout in wait for reply\n"); + + return -ETIMEDOUT; + } + + return 0; +} + +int ne_do_request(struct pci_dev *pdev, enum ne_pci_dev_cmd_type cmd_type, + void *cmd_request, size_t cmd_request_size, + struct ne_pci_dev_cmd_reply *cmd_reply, size_t cmd_reply_size) +{ + struct ne_pci_dev *ne_pci_dev = NULL; + int rc = -EINVAL; + + if (WARN_ON(!pdev)) + return -EINVAL; + + ne_pci_dev = pci_get_drvdata(pdev); + if (WARN_ON(!ne_pci_dev) || WARN_ON(!ne_pci_dev->iomem_base)) + return -EINVAL; + + if (cmd_type <= INVALID_CMD || cmd_type >= MAX_CMD) { + dev_err_ratelimited(&pdev->dev, NE "Invalid cmd type=%u\n", + cmd_type); + + return -EINVAL; + } + + if (!cmd_request) { + dev_err_ratelimited(&pdev->dev, NE "Null cmd request\n"); + + return -EINVAL; + } + + if (cmd_request_size > NE_SEND_DATA_SIZE) { + dev_err_ratelimited(&pdev->dev, + NE "Invalid req size=%zu for cmd type=%u\n", + cmd_request_size, cmd_type); + + return -EINVAL; + } + + if (!cmd_reply) { + dev_err_ratelimited(&pdev->dev, NE "Null cmd reply\n"); + + return -EINVAL; + } + + if (cmd_reply_size > NE_RECV_DATA_SIZE) { + dev_err_ratelimited(&pdev->dev, NE "Invalid reply size=%zu\n", + cmd_reply_size); + + return -EINVAL; + } + + /* + * Use this mutex so that the PCI device handles one command request at + * a time. + */ + mutex_lock(&ne_pci_dev->pci_dev_mutex); + + atomic_set(&ne_pci_dev->cmd_reply_avail, 0); + + rc = ne_submit_request(pdev, cmd_type, cmd_request, cmd_request_size); + if (rc < 0) { + dev_err_ratelimited(&pdev->dev, + NE "Error in submit request [rc=%d]\n", + rc); + + goto unlock_mutex; + } + + rc = ne_wait_for_reply(pdev); + if (rc < 0) { + dev_err_ratelimited(&pdev->dev, + NE "Error in wait for reply [rc=%d]\n", + rc); + + goto unlock_mutex; + } + + rc = ne_retrieve_reply(pdev, cmd_reply, cmd_reply_size); + if (rc < 0) { + dev_err_ratelimited(&pdev->dev, + NE "Error in retrieve reply [rc=%d]\n", + rc); + + goto unlock_mutex; + } + + atomic_set(&ne_pci_dev->cmd_reply_avail, 0); + + if (cmd_reply->rc < 0) { + dev_err_ratelimited(&pdev->dev, + NE "Error in cmd process logic [rc=%d]\n", + cmd_reply->rc); + + rc = cmd_reply->rc; + + goto unlock_mutex; + } + + mutex_unlock(&ne_pci_dev->pci_dev_mutex); + + return 0; + +unlock_mutex: + mutex_unlock(&ne_pci_dev->pci_dev_mutex); + + return rc; +} + +/** + * ne_reply_handler - Interrupt handler for retrieving a reply matching + * a request sent to the PCI device for enclave lifetime management. + * + * @irq: received interrupt for a reply sent by the PCI device. + * @args: PCI device private data structure. + * + * @returns: IRQ_HANDLED on handled interrupt, IRQ_NONE otherwise. + */ +static irqreturn_t ne_reply_handler(int irq, void *args) +{ + struct ne_pci_dev *ne_pci_dev = (struct ne_pci_dev *)args; + + atomic_set(&ne_pci_dev->cmd_reply_avail, 1); + + /* TODO: Update to _interruptible. */ + wake_up(&ne_pci_dev->cmd_reply_wait_q); + + return IRQ_HANDLED; +} + /** * ne_setup_msix - Setup MSI-X vectors for the PCI device. * @@ -82,7 +344,27 @@ static int ne_setup_msix(struct pci_dev *pdev) return rc; } + /* + * This IRQ gets triggered every time the PCI device responds to a + * command request. The reply is then retrieved, reading from the MMIO + * space of the PCI device. + */ + rc = request_irq(pci_irq_vector(pdev, NE_VEC_REPLY), + ne_reply_handler, 0, "enclave_cmd", ne_pci_dev); + if (rc < 0) { + dev_err_ratelimited(&pdev->dev, + NE "Error in request irq reply [rc=%d]\n", + rc); + + goto free_irq_vectors; + } + return 0; + +free_irq_vectors: + pci_free_irq_vectors(pdev); + + return rc; } /** @@ -101,6 +383,8 @@ static void ne_teardown_msix(struct pci_dev *pdev) if (WARN_ON(!ne_pci_dev)) return; + free_irq(pci_irq_vector(pdev, NE_VEC_REPLY), ne_pci_dev); + pci_free_irq_vectors(pdev); } From patchwork Fri May 22 06:29:34 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paraschiv, Andra-Irina" X-Patchwork-Id: 11564737 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5840E138A for ; Fri, 22 May 2020 06:31:07 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3C35520825 for ; Fri, 22 May 2020 06:31:07 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="qdDor9Aw" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728402AbgEVGbG (ORCPT ); Fri, 22 May 2020 02:31:06 -0400 Received: from smtp-fw-9102.amazon.com ([207.171.184.29]:41329 "EHLO smtp-fw-9102.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728374AbgEVGbC (ORCPT ); Fri, 22 May 2020 02:31:02 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1590129062; x=1621665062; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=BIAcz8LzddijuDlc7ilC9ptMeaoq12NgueDWwEgd1S4=; b=qdDor9AwSzUIZ7LnclKUotpA2BCztxPWNRwIi1Qe9epXPC5a7hiU0jt+ Md4owUxPdeEO0ZyHRVipKUPxMVpDOZBIWn4qWHlEHAiJmE2UnJ6ViHwvL rKQT52i8aYs1LH8kiCCjjdw5gjmdQb56crRZNLyH++IDY2euFr1iPgd49 A=; IronPort-SDR: uIy4+63jZL1BTjOk48MWEL0v8rA7rCN1wJRm3McH0lk/HeuTNfhc1ngjrHtkIHjRpftpvj3l4k x+A/6DYk724A== X-IronPort-AV: E=Sophos;i="5.73,420,1583193600"; d="scan'208";a="45244137" Received: from sea32-co-svc-lb4-vlan3.sea.corp.amazon.com (HELO email-inbound-relay-1a-af6a10df.us-east-1.amazon.com) ([10.47.23.38]) by smtp-border-fw-out-9102.sea19.amazon.com with ESMTP; 22 May 2020 06:31:01 +0000 Received: from EX13MTAUEA002.ant.amazon.com (iad55-ws-svc-p15-lb9-vlan3.iad.amazon.com [10.40.159.166]) by email-inbound-relay-1a-af6a10df.us-east-1.amazon.com (Postfix) with ESMTPS id A1B42A0727; Fri, 22 May 2020 06:30:59 +0000 (UTC) Received: from EX13D16EUB003.ant.amazon.com (10.43.166.99) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Fri, 22 May 2020 06:30:59 +0000 Received: from 38f9d34ed3b1.ant.amazon.com (10.43.162.50) by EX13D16EUB003.ant.amazon.com (10.43.166.99) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Fri, 22 May 2020 06:30:50 +0000 From: Andra Paraschiv To: CC: Anthony Liguori , Benjamin Herrenschmidt , Colm MacCarthaigh , "Bjoern Doebel" , David Woodhouse , "Frank van der Linden" , Alexander Graf , "Martin Pohlack" , Matt Wilson , Paolo Bonzini , Balbir Singh , Stefano Garzarella , Stefan Hajnoczi , Stewart Smith , Uwe Dannowski , , , Andra Paraschiv Subject: [PATCH v2 06/18] nitro_enclaves: Handle out-of-band PCI device events Date: Fri, 22 May 2020 09:29:34 +0300 Message-ID: <20200522062946.28973-7-andraprs@amazon.com> X-Mailer: git-send-email 2.20.1 (Apple Git-117) In-Reply-To: <20200522062946.28973-1-andraprs@amazon.com> References: <20200522062946.28973-1-andraprs@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.162.50] X-ClientProxiedBy: EX13D06UWC004.ant.amazon.com (10.43.162.97) To EX13D16EUB003.ant.amazon.com (10.43.166.99) Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org In addition to the replies sent by the Nitro Enclaves PCI device in response to command requests, out-of-band enclave events can happen e.g. an enclave crashes. In this case, the Nitro Enclaves driver needs to be aware of the event and notify the corresponding user space process that abstracts the enclave. Register an MSI-X interrupt vector to be used for this kind of out-of-band events. The interrupt notifies that the state of an enclave changed and the driver logic scans the state of each running enclave to identify for which this notification is intended. Create an workqueue to handle the out-of-band events. Notify user space enclave process that is using a polling mechanism on the enclave fd. The enclave fd is returned as a result of KVM_CREATE_VM ioctl call. Signed-off-by: Alexandru-Catalin Vasile Signed-off-by: Andra Paraschiv --- drivers/virt/nitro_enclaves/ne_pci_dev.c | 117 +++++++++++++++++++++++ 1 file changed, 117 insertions(+) diff --git a/drivers/virt/nitro_enclaves/ne_pci_dev.c b/drivers/virt/nitro_enclaves/ne_pci_dev.c index 7b2d8e1b49a5..cd2012d9b365 100644 --- a/drivers/virt/nitro_enclaves/ne_pci_dev.c +++ b/drivers/virt/nitro_enclaves/ne_pci_dev.c @@ -304,6 +304,85 @@ static irqreturn_t ne_reply_handler(int irq, void *args) return IRQ_HANDLED; } +/** + * ne_event_work_handler - Work queue handler for notifying enclaves on + * a state change received by the event interrupt handler. + * + * An out-of-band event is being issued by the Nitro Hypervisor when at least + * one enclave is changing state without client interaction. + * + * @work: item containing the Nitro Enclaves PCI device for which a + * out-of-band event was issued. + */ +static void ne_event_work_handler(struct work_struct *work) +{ + struct ne_pci_dev_cmd_reply cmd_reply = {}; + struct ne_enclave *ne_enclave = NULL; + struct ne_pci_dev *ne_pci_dev = + container_of(work, struct ne_pci_dev, notify_work); + int rc = -EINVAL; + struct slot_info_req slot_info_req = {}; + + mutex_lock(&ne_pci_dev->enclaves_list_mutex); + + /* + * Iterate over all enclaves registered for the Nitro Enclaves + * PCI device and determine for which enclave(s) the out-of-band event + * is corresponding to. + */ + list_for_each_entry(ne_enclave, &ne_pci_dev->enclaves_list, + enclave_list_entry) { + mutex_lock(&ne_enclave->enclave_info_mutex); + + /* + * Enclaves that were never started cannot receive out-of-band + * events. + */ + if (ne_enclave->state != NE_STATE_RUNNING) + goto unlock; + + slot_info_req.slot_uid = ne_enclave->slot_uid; + + rc = ne_do_request(ne_enclave->pdev, SLOT_INFO, &slot_info_req, + sizeof(slot_info_req), &cmd_reply, + sizeof(cmd_reply)); + WARN_ON(rc < 0); + + /* Notify enclave process that the enclave state changed. */ + if (ne_enclave->state != cmd_reply.state) { + ne_enclave->state = cmd_reply.state; + + ne_enclave->has_event = true; + + wake_up_interruptible(&ne_enclave->eventq); + } + +unlock: + mutex_unlock(&ne_enclave->enclave_info_mutex); + } + + mutex_unlock(&ne_pci_dev->enclaves_list_mutex); +} + +/** + * ne_event_handler - Interrupt handler for PCI device out-of-band + * events. This interrupt does not supply any data in the MMIO region. + * It notifies a change in the state of any of the launched enclaves. + * + * @irq: received interrupt for an out-of-band event. + * @args: PCI device private data structure. + * + * @returns: IRQ_HANDLED on handled interrupt, IRQ_NONE otherwise. + */ +static irqreturn_t ne_event_handler(int irq, void *args) +{ + struct ne_pci_dev *ne_pci_dev = (struct ne_pci_dev *)args; + + queue_work(ne_pci_dev->event_wq, &ne_pci_dev->notify_work); + + return IRQ_HANDLED; +} + /** * ne_setup_msix - Setup MSI-X vectors for the PCI device. * @@ -359,8 +438,40 @@ static int ne_setup_msix(struct pci_dev *pdev) goto free_irq_vectors; } + ne_pci_dev->event_wq = create_singlethread_workqueue("ne_pci_dev_wq"); + if (!ne_pci_dev->event_wq) { + rc = -ENOMEM; + + dev_err_ratelimited(&pdev->dev, + NE "Cannot get wq for dev events [rc=%d]\n", + rc); + + goto free_reply_irq_vec; + } + + INIT_WORK(&ne_pci_dev->notify_work, ne_event_work_handler); + + /* + * This IRQ gets triggered every time any enclave's state changes. Its + * handler then scans for the changes and propagates them to the user + * space. + */ + rc = request_irq(pci_irq_vector(pdev, NE_VEC_EVENT), + ne_event_handler, 0, "enclave_evt", ne_pci_dev); + if (rc < 0) { + dev_err_ratelimited(&pdev->dev, + NE "Error in request irq event [rc=%d]\n", + rc); + + goto destroy_wq; + } + return 0; +destroy_wq: + destroy_workqueue(ne_pci_dev->event_wq); +free_reply_irq_vec: + free_irq(pci_irq_vector(pdev, NE_VEC_REPLY), ne_pci_dev); free_irq_vectors: pci_free_irq_vectors(pdev); @@ -383,6 +494,12 @@ static void ne_teardown_msix(struct pci_dev *pdev) if (WARN_ON(!ne_pci_dev)) return; + free_irq(pci_irq_vector(pdev, NE_VEC_EVENT), ne_pci_dev); + + flush_work(&ne_pci_dev->notify_work); + flush_workqueue(ne_pci_dev->event_wq); + destroy_workqueue(ne_pci_dev->event_wq); + free_irq(pci_irq_vector(pdev, NE_VEC_REPLY), ne_pci_dev); pci_free_irq_vectors(pdev); From patchwork Fri May 22 06:29:35 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paraschiv, Andra-Irina" X-Patchwork-Id: 11564739 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9C1CC14B7 for ; Fri, 22 May 2020 06:31:17 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 80FE3207FB for ; Fri, 22 May 2020 06:31:17 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="IGhRcXvc" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728455AbgEVGbO (ORCPT ); Fri, 22 May 2020 02:31:14 -0400 Received: from smtp-fw-2101.amazon.com ([72.21.196.25]:20657 "EHLO smtp-fw-2101.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728439AbgEVGbL (ORCPT ); Fri, 22 May 2020 02:31:11 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1590129070; x=1621665070; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=kUqB74ao4cKMBkg75oPcYgcRcd1MtSxGoDTgZYPITEc=; b=IGhRcXvcG6mGftJAezLfuSm4ruiNTvvH4HGsYWmQ1Weo0ErwDBYQZ8oG FYGWfWcp0Oj8Qa4mDoI0DX7vI8IMKtxCY8HPsVsU078ga7TZtekfngSwP MO8cSSffth3490yybtohMoaSpNFS0Zdw3W5JVg1ceBLOsN9B+XXbKmSfR A=; IronPort-SDR: B2vOXkIoQT6uiv4Iv7pzn8SU2LmPombwHceIx/pk4IJrtgBCl/ntiML5tTKNWYf8l71QwbuWVh 37Fl7/3P5fIw== X-IronPort-AV: E=Sophos;i="5.73,420,1583193600"; d="scan'208";a="31775418" Received: from iad12-co-svc-p1-lb1-vlan2.amazon.com (HELO email-inbound-relay-1d-f273de60.us-east-1.amazon.com) ([10.43.8.2]) by smtp-border-fw-out-2101.iad2.amazon.com with ESMTP; 22 May 2020 06:31:10 +0000 Received: from EX13MTAUEA002.ant.amazon.com (iad55-ws-svc-p15-lb9-vlan2.iad.amazon.com [10.40.159.162]) by email-inbound-relay-1d-f273de60.us-east-1.amazon.com (Postfix) with ESMTPS id EC775A26AC; Fri, 22 May 2020 06:31:07 +0000 (UTC) Received: from EX13D16EUB003.ant.amazon.com (10.43.166.99) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Fri, 22 May 2020 06:31:07 +0000 Received: from 38f9d34ed3b1.ant.amazon.com (10.43.162.50) by EX13D16EUB003.ant.amazon.com (10.43.166.99) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Fri, 22 May 2020 06:30:58 +0000 From: Andra Paraschiv To: CC: Anthony Liguori , Benjamin Herrenschmidt , Colm MacCarthaigh , "Bjoern Doebel" , David Woodhouse , "Frank van der Linden" , Alexander Graf , "Martin Pohlack" , Matt Wilson , Paolo Bonzini , Balbir Singh , Stefano Garzarella , Stefan Hajnoczi , Stewart Smith , Uwe Dannowski , , , Andra Paraschiv Subject: [PATCH v2 07/18] nitro_enclaves: Init misc device providing the ioctl interface Date: Fri, 22 May 2020 09:29:35 +0300 Message-ID: <20200522062946.28973-8-andraprs@amazon.com> X-Mailer: git-send-email 2.20.1 (Apple Git-117) In-Reply-To: <20200522062946.28973-1-andraprs@amazon.com> References: <20200522062946.28973-1-andraprs@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.162.50] X-ClientProxiedBy: EX13D06UWC004.ant.amazon.com (10.43.162.97) To EX13D16EUB003.ant.amazon.com (10.43.166.99) Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The Nitro Enclaves driver provides an ioctl interface to the user space for enclave lifetime management e.g. enclave creation / termination and setting enclave resources such as memory and CPU. This ioctl interface is mapped to a Nitro Enclaves misc device. Signed-off-by: Andra Paraschiv --- drivers/virt/nitro_enclaves/ne_misc_dev.c | 196 ++++++++++++++++++++++ drivers/virt/nitro_enclaves/ne_pci_dev.c | 13 ++ 2 files changed, 209 insertions(+) create mode 100644 drivers/virt/nitro_enclaves/ne_misc_dev.c diff --git a/drivers/virt/nitro_enclaves/ne_misc_dev.c b/drivers/virt/nitro_enclaves/ne_misc_dev.c new file mode 100644 index 000000000000..e1866fac8220 --- /dev/null +++ b/drivers/virt/nitro_enclaves/ne_misc_dev.c @@ -0,0 +1,196 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms and conditions of the GNU General Public License, + * version 2, as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, see . + */ + +/** + * Enclave lifetime management driver for Nitro Enclaves (NE). + * Nitro is a hypervisor that has been developed by Amazon. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "ne_misc_dev.h" +#include "ne_pci_dev.h" + +#define MIN_MEM_REGION_SIZE (2 * 1024UL * 1024UL) + +#define NE "nitro_enclaves: " + +#define NE_DEV_NAME "nitro_enclaves" + +#define NE_IMAGE_LOAD_OFFSET (8 * 1024UL * 1024UL) + +static char *ne_cpus; +module_param(ne_cpus, charp, 0644); +MODULE_PARM_DESC(ne_cpus, " - CPU pool used for Nitro Enclaves"); + +/* CPU pool used for Nitro Enclaves. */ +struct ne_cpu_pool { + /* Available CPUs in the pool. */ + cpumask_var_t avail; + struct mutex mutex; +}; + +static struct ne_cpu_pool ne_cpu_pool; + +static int ne_open(struct inode *node, struct file *file) +{ + return 0; +} + +static long ne_ioctl(struct file *file, unsigned int cmd, unsigned long arg) +{ + switch (cmd) { + + default: + return -ENOTTY; + } + + return 0; +} + +static int ne_release(struct inode *inode, struct file *file) +{ + return 0; +} + +static const struct file_operations ne_fops = { + .owner = THIS_MODULE, + .llseek = noop_llseek, + .unlocked_ioctl = ne_ioctl, + .open = ne_open, + .release = ne_release, +}; + +struct miscdevice ne_miscdevice = { + .minor = MISC_DYNAMIC_MINOR, + .name = NE_DEV_NAME, + .fops = &ne_fops, + .mode = 0660, +}; + +static int __init ne_init(void) +{ + unsigned int cpu = 0; + unsigned int cpu_sibling = 0; + int rc = -EINVAL; + + memset(&ne_cpu_pool, 0, sizeof(ne_cpu_pool)); + + if (!zalloc_cpumask_var(&ne_cpu_pool.avail, GFP_KERNEL)) + return -ENOMEM; + + mutex_init(&ne_cpu_pool.mutex); + + rc = cpulist_parse(ne_cpus, ne_cpu_pool.avail); + if (rc < 0) { + pr_err_ratelimited(NE "Error in cpulist parse [rc=%d]\n", rc); + + goto free_cpumask; + } + + /* + * Check if CPU siblings are included in the provided CPU pool. The + * expectation is that CPU cores are made available in the CPU pool for + * enclaves. + */ + for_each_cpu(cpu, ne_cpu_pool.avail) { + for_each_cpu(cpu_sibling, topology_sibling_cpumask(cpu)) { + if (!cpumask_test_cpu(cpu_sibling, ne_cpu_pool.avail)) { + pr_err_ratelimited(NE "CPU %d is not in pool\n", + cpu_sibling); + + rc = -EINVAL; + + goto free_cpumask; + } + } + } + + for_each_cpu(cpu, ne_cpu_pool.avail) { + rc = remove_cpu(cpu); + if (rc != 0) { + pr_err_ratelimited(NE "CPU %d not offlined [rc=%d]\n", + cpu, rc); + + goto online_cpus; + } + } + + rc = pci_register_driver(&ne_pci_driver); + if (rc < 0) { + pr_err_ratelimited(NE "Error in pci register driver [rc=%d]\n", + rc); + + goto online_cpus; + } + + return 0; + +online_cpus: + for_each_cpu(cpu, ne_cpu_pool.avail) + add_cpu(cpu); +free_cpumask: + free_cpumask_var(ne_cpu_pool.avail); + + return rc; +} + +static void __exit ne_exit(void) +{ + unsigned int cpu = 0; + int rc = -EINVAL; + + pci_unregister_driver(&ne_pci_driver); + + if (!ne_cpu_pool.avail) + return; + + for_each_cpu(cpu, ne_cpu_pool.avail) { + rc = add_cpu(cpu); + if (rc != 0) + pr_err_ratelimited(NE "CPU %d not onlined [rc=%d]\n", + cpu, rc); + } + + free_cpumask_var(ne_cpu_pool.avail); +} + +/* TODO: Handle actions such as reboot, kexec. */ + +module_init(ne_init); +module_exit(ne_exit); + +MODULE_AUTHOR("Amazon.com, Inc. or its affiliates"); +MODULE_DESCRIPTION("Nitro Enclaves Driver"); +MODULE_LICENSE("GPL v2"); diff --git a/drivers/virt/nitro_enclaves/ne_pci_dev.c b/drivers/virt/nitro_enclaves/ne_pci_dev.c index cd2012d9b365..7b23262f8c35 100644 --- a/drivers/virt/nitro_enclaves/ne_pci_dev.c +++ b/drivers/virt/nitro_enclaves/ne_pci_dev.c @@ -647,6 +647,15 @@ static int ne_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) goto teardown_msix; } + rc = misc_register(&ne_miscdevice); + if (rc < 0) { + dev_err_ratelimited(&pdev->dev, + NE "Error in misc dev register [rc=%d]\n", + rc); + + goto disable_ne_pci_dev; + } + atomic_set(&ne_pci_dev->cmd_reply_avail, 0); init_waitqueue_head(&ne_pci_dev->cmd_reply_wait_q); INIT_LIST_HEAD(&ne_pci_dev->enclaves_list); @@ -655,6 +664,8 @@ static int ne_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id) return 0; +disable_ne_pci_dev: + ne_pci_dev_disable(pdev); teardown_msix: ne_teardown_msix(pdev); iounmap_pci_bar: @@ -677,6 +688,8 @@ static void ne_pci_remove(struct pci_dev *pdev) if (!ne_pci_dev || !ne_pci_dev->iomem_base) return; + misc_deregister(&ne_miscdevice); + ne_pci_dev_disable(pdev); ne_teardown_msix(pdev); From patchwork Fri May 22 06:29:36 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paraschiv, Andra-Irina" X-Patchwork-Id: 11564741 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 055A4138A for ; Fri, 22 May 2020 06:31:27 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DB505207BC for ; Fri, 22 May 2020 06:31:26 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="e753uDyU" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728243AbgEVGbV (ORCPT ); Fri, 22 May 2020 02:31:21 -0400 Received: from smtp-fw-2101.amazon.com ([72.21.196.25]:20657 "EHLO smtp-fw-2101.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728475AbgEVGbT (ORCPT ); Fri, 22 May 2020 02:31:19 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1590129078; x=1621665078; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=KLTkN11PYw1BrFhTsfFoVnUQqECKaZPdZ/tYIEVKWpQ=; b=e753uDyUYt6T2gxojllc2OHv15XjkG4koY8VGLp3i2EsmPJD/IhinhdF QMA8v4lS0fKuKKH+NU6EXS3g8gxCjFrwIEeBWNtxWLvNDPlsx/tkIu66A DEYRlyxUbYXsPEV62jmIihi39AQTVVQmuffBUZGq6znkaaPUe8804h3R5 g=; IronPort-SDR: CyiohJ0v2eKp/mVeAczusX6eMtdHFllTZr4mZItOIhG9QQRdgNM0mfzA7k7C25Hl5vLLL4teXQ wNd4Cxate2vg== X-IronPort-AV: E=Sophos;i="5.73,420,1583193600"; d="scan'208";a="31775434" Received: from iad12-co-svc-p1-lb1-vlan2.amazon.com (HELO email-inbound-relay-1e-303d0b0e.us-east-1.amazon.com) ([10.43.8.2]) by smtp-border-fw-out-2101.iad2.amazon.com with ESMTP; 22 May 2020 06:31:18 +0000 Received: from EX13MTAUEA002.ant.amazon.com (iad55-ws-svc-p15-lb9-vlan3.iad.amazon.com [10.40.159.166]) by email-inbound-relay-1e-303d0b0e.us-east-1.amazon.com (Postfix) with ESMTPS id 2C5B6A2075; Fri, 22 May 2020 06:31:16 +0000 (UTC) Received: from EX13D16EUB003.ant.amazon.com (10.43.166.99) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Fri, 22 May 2020 06:31:15 +0000 Received: from 38f9d34ed3b1.ant.amazon.com (10.43.162.50) by EX13D16EUB003.ant.amazon.com (10.43.166.99) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Fri, 22 May 2020 06:31:07 +0000 From: Andra Paraschiv To: CC: Anthony Liguori , Benjamin Herrenschmidt , Colm MacCarthaigh , "Bjoern Doebel" , David Woodhouse , "Frank van der Linden" , Alexander Graf , "Martin Pohlack" , Matt Wilson , Paolo Bonzini , Balbir Singh , Stefano Garzarella , Stefan Hajnoczi , Stewart Smith , Uwe Dannowski , , , Andra Paraschiv Subject: [PATCH v2 08/18] nitro_enclaves: Add logic for enclave vm creation Date: Fri, 22 May 2020 09:29:36 +0300 Message-ID: <20200522062946.28973-9-andraprs@amazon.com> X-Mailer: git-send-email 2.20.1 (Apple Git-117) In-Reply-To: <20200522062946.28973-1-andraprs@amazon.com> References: <20200522062946.28973-1-andraprs@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.162.50] X-ClientProxiedBy: EX13D06UWC004.ant.amazon.com (10.43.162.97) To EX13D16EUB003.ant.amazon.com (10.43.166.99) Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Add ioctl command logic for enclave VM creation. It triggers a slot allocation. The enclave resources will be associated with this slot and it will be used as an identifier for triggering enclave run. Return a file descriptor, namely enclave fd. This is further used by the associated user space enclave process to set enclave resources and trigger enclave termination. The poll function is implemented in order to notify the enclave process when an enclave exits without a specific enclave termination command trigger e.g. when an enclave crashes. Signed-off-by: Alexandru Vasile Signed-off-by: Andra Paraschiv --- drivers/virt/nitro_enclaves/ne_misc_dev.c | 169 ++++++++++++++++++++++ 1 file changed, 169 insertions(+) diff --git a/drivers/virt/nitro_enclaves/ne_misc_dev.c b/drivers/virt/nitro_enclaves/ne_misc_dev.c index e1866fac8220..1036221238f4 100644 --- a/drivers/virt/nitro_enclaves/ne_misc_dev.c +++ b/drivers/virt/nitro_enclaves/ne_misc_dev.c @@ -63,6 +63,146 @@ struct ne_cpu_pool { static struct ne_cpu_pool ne_cpu_pool; +static int ne_enclave_open(struct inode *node, struct file *file) +{ + return 0; +} + +static long ne_enclave_ioctl(struct file *file, unsigned int cmd, + unsigned long arg) +{ + switch (cmd) { + default: + return -ENOTTY; + } + + return 0; +} + +static int ne_enclave_release(struct inode *inode, struct file *file) +{ + return 0; +} + +static __poll_t ne_enclave_poll(struct file *file, poll_table *wait) +{ + __poll_t mask = 0; + struct ne_enclave *ne_enclave = file->private_data; + + poll_wait(file, &ne_enclave->eventq, wait); + + if (!ne_enclave->has_event) + return mask; + + mask = POLLHUP; + + return mask; +} + +static const struct file_operations ne_enclave_fops = { + .owner = THIS_MODULE, + .llseek = noop_llseek, + .poll = ne_enclave_poll, + .unlocked_ioctl = ne_enclave_ioctl, + .open = ne_enclave_open, + .release = ne_enclave_release, +}; + +/** + * ne_create_vm_ioctl - Alloc slot to be associated with an enclave. Create + * enclave file descriptor to be further used for enclave resources handling + * e.g. memory regions and CPUs. + * + * This function gets called with the ne_pci_dev enclave mutex held. + * + * @pdev: PCI device used for enclave lifetime management. + * @ne_pci_dev: private data associated with the PCI device. + * @type: type of the virtual machine to be created. + * + * @returns: enclave fd on success, negative return value on failure. + */ +static int ne_create_vm_ioctl(struct pci_dev *pdev, + struct ne_pci_dev *ne_pci_dev, unsigned long type) +{ + struct ne_pci_dev_cmd_reply cmd_reply = {}; + int fd = 0; + struct file *file = NULL; + struct ne_enclave *ne_enclave = NULL; + int rc = -EINVAL; + struct slot_alloc_req slot_alloc_req = {}; + + if (WARN_ON(!pdev) || WARN_ON(!ne_pci_dev)) + return -EINVAL; + + ne_enclave = kzalloc(sizeof(*ne_enclave), GFP_KERNEL); + if (!ne_enclave) + return -ENOMEM; + + if (!zalloc_cpumask_var(&ne_enclave->cpu_siblings, GFP_KERNEL)) { + kzfree(ne_enclave); + + return -ENOMEM; + } + + fd = get_unused_fd_flags(O_CLOEXEC); + if (fd < 0) { + rc = fd; + + pr_err_ratelimited(NE "Error in getting unused fd [rc=%d]\n", + rc); + + goto free_cpumask; + } + + file = anon_inode_getfile("ne-vm", &ne_enclave_fops, ne_enclave, + O_RDWR); + if (IS_ERR(file)) { + rc = PTR_ERR(file); + + pr_err_ratelimited(NE "Error in anon inode get file [rc=%d]\n", + rc); + + goto put_fd; + } + + ne_enclave->pdev = pdev; + + rc = ne_do_request(ne_enclave->pdev, SLOT_ALLOC, &slot_alloc_req, + sizeof(slot_alloc_req), &cmd_reply, + sizeof(cmd_reply)); + if (rc < 0) { + pr_err_ratelimited(NE "Error in slot alloc [rc=%d]\n", rc); + + goto put_file; + } + + init_waitqueue_head(&ne_enclave->eventq); + ne_enclave->has_event = false; + mutex_init(&ne_enclave->enclave_info_mutex); + ne_enclave->max_mem_regions = cmd_reply.mem_regions; + INIT_LIST_HEAD(&ne_enclave->mem_regions_list); + ne_enclave->mm = current->mm; + ne_enclave->slot_uid = cmd_reply.slot_uid; + ne_enclave->state = NE_STATE_INIT; + INIT_LIST_HEAD(&ne_enclave->vcpu_ids_list); + + list_add(&ne_enclave->enclave_list_entry, &ne_pci_dev->enclaves_list); + + fd_install(fd, file); + + return fd; + +put_file: + fput(file); +put_fd: + put_unused_fd(fd); +free_cpumask: + free_cpumask_var(ne_enclave->cpu_siblings); + kzfree(ne_enclave); + + return rc; +} + static int ne_open(struct inode *node, struct file *file) { return 0; @@ -70,7 +210,36 @@ static int ne_open(struct inode *node, struct file *file) static long ne_ioctl(struct file *file, unsigned int cmd, unsigned long arg) { + struct ne_pci_dev *ne_pci_dev = NULL; + struct pci_dev *pdev = pci_get_device(PCI_VENDOR_ID_AMAZON, + PCI_DEVICE_ID_NE, NULL); + + if (WARN_ON(!pdev)) + return -EINVAL; + + ne_pci_dev = pci_get_drvdata(pdev); + if (WARN_ON(!ne_pci_dev)) + return -EINVAL; + switch (cmd) { + case KVM_CREATE_VM: { + int rc = -EINVAL; + unsigned long type = 0; + + if (copy_from_user(&type, (void *)arg, sizeof(type))) { + pr_err_ratelimited(NE "Error in copy from user\n"); + + return -EFAULT; + } + + mutex_lock(&ne_pci_dev->enclaves_list_mutex); + + rc = ne_create_vm_ioctl(pdev, ne_pci_dev, type); + + mutex_unlock(&ne_pci_dev->enclaves_list_mutex); + + return rc; + } default: return -ENOTTY; From patchwork Fri May 22 06:29:37 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paraschiv, Andra-Irina" X-Patchwork-Id: 11564743 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 75CF414B7 for ; Fri, 22 May 2020 06:31:31 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 597B22072C for ; Fri, 22 May 2020 06:31:31 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="IqMJn2O9" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728590AbgEVGba (ORCPT ); Fri, 22 May 2020 02:31:30 -0400 Received: from smtp-fw-33001.amazon.com ([207.171.190.10]:8296 "EHLO smtp-fw-33001.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728481AbgEVGb2 (ORCPT ); Fri, 22 May 2020 02:31:28 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1590129088; x=1621665088; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=1DHL97KDrr4JedjALnLaMkYmTHR9Yl9qB1A7Hs79j1Y=; b=IqMJn2O9AC9sps5L3a4dbfvxIwiXdnYxggiZLZQcjrXpnDiQaD2lAm1n h1BWxTm0aMeSqsf51LuaxTV6r6skerIiVBJbg757WaFlbq9kaRgw/m1VZ nswS4VKJZ55bQyQXlwRDmCLCBoSeNKGsDxHsGI2xEKfNFCnGKG/3Itnw+ Q=; IronPort-SDR: 2EF+aUzhiiGEW/6TzHcybXWTbTQ59RmANHWpsZL4ZK9OpdPTP3zVpSIQiuOzcFsY4NaoG4lyN4 81hW7UQipCYA== X-IronPort-AV: E=Sophos;i="5.73,420,1583193600"; d="scan'208";a="46675022" Received: from sea32-co-svc-lb4-vlan2.sea.corp.amazon.com (HELO email-inbound-relay-1a-16acd5e0.us-east-1.amazon.com) ([10.47.23.34]) by smtp-border-fw-out-33001.sea14.amazon.com with ESMTP; 22 May 2020 06:31:28 +0000 Received: from EX13MTAUEA002.ant.amazon.com (iad55-ws-svc-p15-lb9-vlan2.iad.amazon.com [10.40.159.162]) by email-inbound-relay-1a-16acd5e0.us-east-1.amazon.com (Postfix) with ESMTPS id 17BC4A2587; Fri, 22 May 2020 06:31:26 +0000 (UTC) Received: from EX13D16EUB003.ant.amazon.com (10.43.166.99) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Fri, 22 May 2020 06:31:25 +0000 Received: from 38f9d34ed3b1.ant.amazon.com (10.43.162.50) by EX13D16EUB003.ant.amazon.com (10.43.166.99) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Fri, 22 May 2020 06:31:15 +0000 From: Andra Paraschiv To: CC: Anthony Liguori , Benjamin Herrenschmidt , Colm MacCarthaigh , "Bjoern Doebel" , David Woodhouse , "Frank van der Linden" , Alexander Graf , "Martin Pohlack" , Matt Wilson , Paolo Bonzini , Balbir Singh , Stefano Garzarella , Stefan Hajnoczi , Stewart Smith , Uwe Dannowski , , , Andra Paraschiv Subject: [PATCH v2 09/18] nitro_enclaves: Add logic for enclave vcpu creation Date: Fri, 22 May 2020 09:29:37 +0300 Message-ID: <20200522062946.28973-10-andraprs@amazon.com> X-Mailer: git-send-email 2.20.1 (Apple Git-117) In-Reply-To: <20200522062946.28973-1-andraprs@amazon.com> References: <20200522062946.28973-1-andraprs@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.162.50] X-ClientProxiedBy: EX13D06UWC004.ant.amazon.com (10.43.162.97) To EX13D16EUB003.ant.amazon.com (10.43.166.99) Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org An enclave, before being started, has its resources set. One of its resources is CPU. Add ioctl command logic for enclave vCPU creation. Return as result a file descriptor that is associated with the enclave vCPU. Signed-off-by: Alexandru Vasile Signed-off-by: Andra Paraschiv --- drivers/virt/nitro_enclaves/ne_misc_dev.c | 223 ++++++++++++++++++++++ 1 file changed, 223 insertions(+) diff --git a/drivers/virt/nitro_enclaves/ne_misc_dev.c b/drivers/virt/nitro_enclaves/ne_misc_dev.c index 1036221238f4..8cf262ac1bbc 100644 --- a/drivers/virt/nitro_enclaves/ne_misc_dev.c +++ b/drivers/virt/nitro_enclaves/ne_misc_dev.c @@ -63,6 +63,183 @@ struct ne_cpu_pool { static struct ne_cpu_pool ne_cpu_pool; +static int ne_enclave_vcpu_open(struct inode *node, struct file *file) +{ + return 0; +} + +static long ne_enclave_vcpu_ioctl(struct file *file, unsigned int cmd, + unsigned long arg) +{ + switch (cmd) { + default: + return -ENOTTY; + } + + return 0; +} + +static int ne_enclave_vcpu_release(struct inode *inode, struct file *file) +{ + return 0; +} + +static const struct file_operations ne_enclave_vcpu_fops = { + .owner = THIS_MODULE, + .llseek = noop_llseek, + .unlocked_ioctl = ne_enclave_vcpu_ioctl, + .open = ne_enclave_vcpu_open, + .release = ne_enclave_vcpu_release, +}; + +/** + * ne_get_cpu_from_cpu_pool - Get a CPU from the CPU pool, if it is set. + * + * This function gets called with the ne_enclave mutex held. + * + * @ne_enclave: private data associated with the current enclave. + * @vcpu_id: id of the CPU to be associated with the given slot, apic id on x86. + * + * @returns: 0 on success, negative return value on failure. + */ +static int ne_get_cpu_from_cpu_pool(struct ne_enclave *ne_enclave, u32 *vcpu_id) +{ + unsigned int cpu = 0; + unsigned int cpu_sibling = 0; + + if (WARN_ON(!ne_enclave)) + return -EINVAL; + + if (!vcpu_id) + return -EINVAL; + + /* There are CPU siblings available to choose from. */ + cpu = cpumask_any(ne_enclave->cpu_siblings); + if (cpu < nr_cpu_ids) { + cpumask_clear_cpu(cpu, ne_enclave->cpu_siblings); + + *vcpu_id = cpu; + + return 0; + } + + mutex_lock(&ne_cpu_pool.mutex); + + /* Choose any CPU from the available CPU pool. */ + cpu = cpumask_any(ne_cpu_pool.avail); + if (cpu >= nr_cpu_ids) { + pr_err_ratelimited(NE "No CPUs available in CPU pool\n"); + + mutex_unlock(&ne_cpu_pool.mutex); + + return -EINVAL; + } + + cpumask_clear_cpu(cpu, ne_cpu_pool.avail); + + /* + * Make sure the CPU siblings are not marked as + * available anymore. + */ + for_each_cpu(cpu_sibling, topology_sibling_cpumask(cpu)) { + if (cpu_sibling != cpu) { + cpumask_clear_cpu(cpu_sibling, ne_cpu_pool.avail); + + cpumask_set_cpu(cpu_sibling, ne_enclave->cpu_siblings); + } + } + + mutex_unlock(&ne_cpu_pool.mutex); + + *vcpu_id = cpu; + + return 0; +} + +/** + * ne_create_vcpu_ioctl - Add vCPU to the slot associated with the current + * enclave. Create vCPU file descriptor to be further used for CPU handling. + * + * This function gets called with the ne_enclave mutex held. + * + * @ne_enclave: private data associated with the current enclave. + * @vcpu_id: id of the CPU to be associated with the given slot, apic id on x86. + * + * @returns: vCPU fd on success, negative return value on failure. + */ +static int ne_create_vcpu_ioctl(struct ne_enclave *ne_enclave, u32 vcpu_id) +{ + struct ne_pci_dev_cmd_reply cmd_reply = {}; + int fd = 0; + struct file *file = NULL; + struct ne_vcpu_id *ne_vcpu_id = NULL; + int rc = -EINVAL; + struct slot_add_vcpu_req slot_add_vcpu_req = {}; + + if (WARN_ON(!ne_enclave) || WARN_ON(!ne_enclave->pdev)) + return -EINVAL; + + if (ne_enclave->mm != current->mm) + return -EIO; + + ne_vcpu_id = kzalloc(sizeof(*ne_vcpu_id), GFP_KERNEL); + if (!ne_vcpu_id) + return -ENOMEM; + + fd = get_unused_fd_flags(O_CLOEXEC); + if (fd < 0) { + rc = fd; + + pr_err_ratelimited(NE "Error in getting unused fd [rc=%d]\n", + rc); + + goto free_ne_vcpu_id; + } + + /* TODO: Include (vcpu) id in the ne-vm-vcpu naming. */ + file = anon_inode_getfile("ne-vm-vcpu", &ne_enclave_vcpu_fops, + ne_enclave, O_RDWR); + if (IS_ERR(file)) { + rc = PTR_ERR(file); + + pr_err_ratelimited(NE "Error in anon inode get file [rc=%d]\n", + rc); + + goto put_fd; + } + + slot_add_vcpu_req.slot_uid = ne_enclave->slot_uid; + slot_add_vcpu_req.vcpu_id = vcpu_id; + + rc = ne_do_request(ne_enclave->pdev, SLOT_ADD_VCPU, &slot_add_vcpu_req, + sizeof(slot_add_vcpu_req), &cmd_reply, + sizeof(cmd_reply)); + if (rc < 0) { + pr_err_ratelimited(NE "Error in slot add vcpu [rc=%d]\n", rc); + + goto put_file; + } + + ne_vcpu_id->vcpu_id = vcpu_id; + + list_add(&ne_vcpu_id->vcpu_id_list_entry, &ne_enclave->vcpu_ids_list); + + ne_enclave->nr_vcpus++; + + fd_install(fd, file); + + return fd; + +put_file: + fput(file); +put_fd: + put_unused_fd(fd); +free_ne_vcpu_id: + kzfree(ne_vcpu_id); + + return rc; +} + static int ne_enclave_open(struct inode *node, struct file *file) { return 0; @@ -71,7 +248,53 @@ static int ne_enclave_open(struct inode *node, struct file *file) static long ne_enclave_ioctl(struct file *file, unsigned int cmd, unsigned long arg) { + struct ne_enclave *ne_enclave = file->private_data; + + if (WARN_ON(!ne_enclave)) + return -EINVAL; + switch (cmd) { + case KVM_CREATE_VCPU: { + int rc = -EINVAL; + u32 vcpu_id = 0; + + if (copy_from_user(&vcpu_id, (void *)arg, sizeof(vcpu_id))) { + pr_err_ratelimited(NE "Error in copy from user\n"); + + return -EFAULT; + } + + mutex_lock(&ne_enclave->enclave_info_mutex); + + if (ne_enclave->state != NE_STATE_INIT) { + pr_err_ratelimited(NE "Enclave isn't in init state\n"); + + mutex_unlock(&ne_enclave->enclave_info_mutex); + + return -EINVAL; + } + + /* Use the CPU pool for choosing a CPU for the enclave. */ + rc = ne_get_cpu_from_cpu_pool(ne_enclave, &vcpu_id); + if (rc < 0) { + pr_err_ratelimited(NE "Error in get CPU from pool\n"); + + mutex_unlock(&ne_enclave->enclave_info_mutex); + + return -EINVAL; + } + + rc = ne_create_vcpu_ioctl(ne_enclave, vcpu_id); + + /* Put back the CPU in enclave cpu pool, if add vcpu error. */ + if (rc < 0) + cpumask_set_cpu(vcpu_id, ne_enclave->cpu_siblings); + + mutex_unlock(&ne_enclave->enclave_info_mutex); + + return rc; + } + default: return -ENOTTY; } From patchwork Fri May 22 06:29:38 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paraschiv, Andra-Irina" X-Patchwork-Id: 11564745 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 116C8138A for ; Fri, 22 May 2020 06:31:48 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id EECFC2072C for ; Fri, 22 May 2020 06:31:47 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="c6EYPwLw" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728636AbgEVGbo (ORCPT ); Fri, 22 May 2020 02:31:44 -0400 Received: from smtp-fw-9101.amazon.com ([207.171.184.25]:33810 "EHLO smtp-fw-9101.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728371AbgEVGbo (ORCPT ); Fri, 22 May 2020 02:31:44 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1590129104; x=1621665104; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=rJzeHtcicxOU0HUoPr5xYwGnt2a0NWxVxt6g39XUl0w=; b=c6EYPwLwQqZveFYxqgD+l0SmKcDZM8xMq50Yh2CB3huuZRQjGiejfCIj r34UJo9TFdAJw+MW33RW7Lqyf+nZPWfmhW/Y2PtdyAPmGdBPDgo1kHEEL ykFfCqz4HWvkt+m0mLFk+2p5USD47AXJhO1lM9Z5qAsj0sSOSwLkA5UJX M=; IronPort-SDR: 6QSeRXkxjIg1mY5B2lTZcHSl4+0sPQlPsUYV0coQvY2i9k1b9auDyXyo9tzxXtchqk4lgQuZWr Rwq83ZnYOzEw== X-IronPort-AV: E=Sophos;i="5.73,420,1583193600"; d="scan'208";a="36944429" Received: from sea32-co-svc-lb4-vlan3.sea.corp.amazon.com (HELO email-inbound-relay-1e-17c49630.us-east-1.amazon.com) ([10.47.23.38]) by smtp-border-fw-out-9101.sea19.amazon.com with ESMTP; 22 May 2020 06:31:43 +0000 Received: from EX13MTAUEA002.ant.amazon.com (iad55-ws-svc-p15-lb9-vlan3.iad.amazon.com [10.40.159.166]) by email-inbound-relay-1e-17c49630.us-east-1.amazon.com (Postfix) with ESMTPS id 2E52EA067A; Fri, 22 May 2020 06:31:41 +0000 (UTC) Received: from EX13D16EUB003.ant.amazon.com (10.43.166.99) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Fri, 22 May 2020 06:31:40 +0000 Received: from 38f9d34ed3b1.ant.amazon.com (10.43.161.175) by EX13D16EUB003.ant.amazon.com (10.43.166.99) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Fri, 22 May 2020 06:31:31 +0000 From: Andra Paraschiv To: CC: Anthony Liguori , Benjamin Herrenschmidt , Colm MacCarthaigh , "Bjoern Doebel" , David Woodhouse , "Frank van der Linden" , Alexander Graf , "Martin Pohlack" , Matt Wilson , Paolo Bonzini , Balbir Singh , Stefano Garzarella , Stefan Hajnoczi , Stewart Smith , Uwe Dannowski , , , Andra Paraschiv Subject: [PATCH v2 10/18] nitro_enclaves: Add logic for enclave image load metadata Date: Fri, 22 May 2020 09:29:38 +0300 Message-ID: <20200522062946.28973-11-andraprs@amazon.com> X-Mailer: git-send-email 2.20.1 (Apple Git-117) In-Reply-To: <20200522062946.28973-1-andraprs@amazon.com> References: <20200522062946.28973-1-andraprs@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.161.175] X-ClientProxiedBy: EX13D27UWA001.ant.amazon.com (10.43.160.19) To EX13D16EUB003.ant.amazon.com (10.43.166.99) Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Before setting the memory regions for the enclave, the enclave image needs to be placed in memory. After the memory regions are set, this memory cannot be used anymore by the VM, being carved out. Add ioctl command logic to get the offset in enclave memory where to place the enclave image. Then the user space tooling copies the enclave image in the memory using the giveni memory offset. Signed-off-by: Andra Paraschiv --- drivers/virt/nitro_enclaves/ne_misc_dev.c | 24 +++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/drivers/virt/nitro_enclaves/ne_misc_dev.c b/drivers/virt/nitro_enclaves/ne_misc_dev.c index 8cf262ac1bbc..936c9ce19213 100644 --- a/drivers/virt/nitro_enclaves/ne_misc_dev.c +++ b/drivers/virt/nitro_enclaves/ne_misc_dev.c @@ -295,6 +295,30 @@ static long ne_enclave_ioctl(struct file *file, unsigned int cmd, return rc; } + case NE_GET_IMAGE_LOAD_METADATA: { + struct image_load_metadata image_load_metadata = {}; + + if (copy_from_user(&image_load_metadata, (void *)arg, + sizeof(image_load_metadata))) { + pr_err_ratelimited(NE "Error in copy from user\n"); + + return -EFAULT; + } + + /* TODO: Check flags before setting the memory offset. */ + + image_load_metadata.memory_offset = NE_IMAGE_LOAD_OFFSET; + + if (copy_to_user((void *)arg, &image_load_metadata, + sizeof(image_load_metadata))) { + pr_err_ratelimited(NE "Error in copy to user\n"); + + return -EFAULT; + } + + return 0; + } + default: return -ENOTTY; } From patchwork Fri May 22 06:29:39 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paraschiv, Andra-Irina" X-Patchwork-Id: 11564747 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8C1A7138A for ; Fri, 22 May 2020 06:31:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6B2752072C for ; Fri, 22 May 2020 06:31:57 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="shbGxIFe" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728667AbgEVGby (ORCPT ); Fri, 22 May 2020 02:31:54 -0400 Received: from smtp-fw-2101.amazon.com ([72.21.196.25]:20749 "EHLO smtp-fw-2101.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728593AbgEVGby (ORCPT ); Fri, 22 May 2020 02:31:54 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1590129112; x=1621665112; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Eh/w7fDQ/t+3heysDB45hgJsudaZbKHb1B4FFyT3Zhg=; b=shbGxIFeEEdvFzbC2n/0HyzNsqdfzc1a2DGOrTy0tS2Qca4RYh7HV5+8 Df5555fppNjrGYVODsVxnDMuStnVJ71dQZhumBK5mma9ZC3mhzsgni0Gj DK6kkWNJfPuzer9tcpNcsMCC1Fn7tQOxRlF7YSO0irF2s5hqw7+M1+XmC A=; IronPort-SDR: 8aunz0jmc81HQlFOfh6ASnGvzm/UCtnhSZ+HPYIBkRCPrXfwF4ENwawF0TtCILGUfBCI8RJcqL atSYFoKHsypA== X-IronPort-AV: E=Sophos;i="5.73,420,1583193600"; d="scan'208";a="31775489" Received: from iad12-co-svc-p1-lb1-vlan2.amazon.com (HELO email-inbound-relay-1d-37fd6b3d.us-east-1.amazon.com) ([10.43.8.2]) by smtp-border-fw-out-2101.iad2.amazon.com with ESMTP; 22 May 2020 06:31:51 +0000 Received: from EX13MTAUEA002.ant.amazon.com (iad55-ws-svc-p15-lb9-vlan2.iad.amazon.com [10.40.159.162]) by email-inbound-relay-1d-37fd6b3d.us-east-1.amazon.com (Postfix) with ESMTPS id 8AABA284788; Fri, 22 May 2020 06:31:49 +0000 (UTC) Received: from EX13D16EUB003.ant.amazon.com (10.43.166.99) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Fri, 22 May 2020 06:31:49 +0000 Received: from 38f9d34ed3b1.ant.amazon.com (10.43.161.175) by EX13D16EUB003.ant.amazon.com (10.43.166.99) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Fri, 22 May 2020 06:31:40 +0000 From: Andra Paraschiv To: CC: Anthony Liguori , Benjamin Herrenschmidt , Colm MacCarthaigh , "Bjoern Doebel" , David Woodhouse , "Frank van der Linden" , Alexander Graf , "Martin Pohlack" , Matt Wilson , Paolo Bonzini , Balbir Singh , Stefano Garzarella , Stefan Hajnoczi , Stewart Smith , Uwe Dannowski , , , Andra Paraschiv Subject: [PATCH v2 11/18] nitro_enclaves: Add logic for enclave memory region set Date: Fri, 22 May 2020 09:29:39 +0300 Message-ID: <20200522062946.28973-12-andraprs@amazon.com> X-Mailer: git-send-email 2.20.1 (Apple Git-117) In-Reply-To: <20200522062946.28973-1-andraprs@amazon.com> References: <20200522062946.28973-1-andraprs@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.161.175] X-ClientProxiedBy: EX13D27UWA001.ant.amazon.com (10.43.160.19) To EX13D16EUB003.ant.amazon.com (10.43.166.99) Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Another resource that is being set for an enclave is memory. User space memory regions, that need to be backed by contiguous memory regions, are associated with the enclave. One solution for allocating / reserving contiguous memory regions, that is used for integration, is hugetlbfs. The user space process that is associated with the enclave passes to the driver these memory regions. Add ioctl command logic for setting user space memory region for an enclave. Signed-off-by: Alexandru Vasile Signed-off-by: Andra Paraschiv --- drivers/virt/nitro_enclaves/ne_misc_dev.c | 256 ++++++++++++++++++++++ 1 file changed, 256 insertions(+) diff --git a/drivers/virt/nitro_enclaves/ne_misc_dev.c b/drivers/virt/nitro_enclaves/ne_misc_dev.c index 936c9ce19213..a5c6185613b9 100644 --- a/drivers/virt/nitro_enclaves/ne_misc_dev.c +++ b/drivers/virt/nitro_enclaves/ne_misc_dev.c @@ -240,6 +240,234 @@ static int ne_create_vcpu_ioctl(struct ne_enclave *ne_enclave, u32 vcpu_id) return rc; } +/** + * ne_sanity_check_user_mem_region - Sanity check the userspace memory + * region received during the set user memory region ioctl call. + * + * This function gets called with the ne_enclave mutex held. + * + * @ne_enclave: private data associated with the current enclave. + * @mem_region: user space memory region to be sanity checked. + * + * @returns: 0 on success, negative return value on failure. + */ +static int ne_sanity_check_user_mem_region(struct ne_enclave *ne_enclave, + struct kvm_userspace_memory_region *mem_region) +{ + if (WARN_ON(!ne_enclave)) + return -EINVAL; + + if (!mem_region) + return -EINVAL; + + if ((mem_region->memory_size % MIN_MEM_REGION_SIZE) != 0) { + pr_err_ratelimited(NE "Mem size not multiple of 2 MiB\n"); + + return -EINVAL; + } + + if ((mem_region->userspace_addr & (MIN_MEM_REGION_SIZE - 1)) || + !access_ok((void __user *)(unsigned long)mem_region->userspace_addr, + mem_region->memory_size)) { + pr_err_ratelimited(NE "Invalid user space addr range\n"); + + return -EINVAL; + } + + if ((mem_region->guest_phys_addr + mem_region->memory_size) < + mem_region->guest_phys_addr) { + pr_err_ratelimited(NE "Invalid guest phys addr range\n"); + + return -EINVAL; + } + + return 0; +} + +/** + * ne_set_user_memory_region_ioctl - Add user space memory region to the slot + * associated with the current enclave. + * + * This function gets called with the ne_enclave mutex held. + * + * @ne_enclave: private data associated with the current enclave. + * @mem_region: user space memory region to be associated with the given slot. + * + * @returns: 0 on success, negative return value on failure. + */ +static int ne_set_user_memory_region_ioctl(struct ne_enclave *ne_enclave, + struct kvm_userspace_memory_region *mem_region) +{ + struct ne_pci_dev_cmd_reply cmd_reply = {}; + long gup_rc = 0; + unsigned long i = 0; + struct ne_mem_region *ne_mem_region = NULL; + unsigned long nr_phys_contig_mem_regions = 0; + unsigned long nr_pinned_pages = 0; + struct page **phys_contig_mem_regions = NULL; + int rc = -EINVAL; + struct slot_add_mem_req slot_add_mem_req = {}; + + if (WARN_ON(!ne_enclave) || WARN_ON(!ne_enclave->pdev)) + return -EINVAL; + + if (!mem_region) + return -EINVAL; + + if (ne_enclave->mm != current->mm) + return -EIO; + + rc = ne_sanity_check_user_mem_region(ne_enclave, mem_region); + if (rc < 0) + return rc; + + ne_mem_region = kzalloc(sizeof(*ne_mem_region), GFP_KERNEL); + if (!ne_mem_region) + return -ENOMEM; + + /* + * TODO: Update nr_pages value to handle contiguous virtual address + * ranges mapped to non-contiguous physical regions. Hugetlbfs can give + * 2 MiB / 1 GiB contiguous physical regions. + */ + ne_mem_region->nr_pages = mem_region->memory_size / MIN_MEM_REGION_SIZE; + + ne_mem_region->pages = kcalloc(ne_mem_region->nr_pages, + sizeof(*ne_mem_region->pages), + GFP_KERNEL); + if (!ne_mem_region->pages) { + kzfree(ne_mem_region); + + return -ENOMEM; + } + + phys_contig_mem_regions = kcalloc(ne_mem_region->nr_pages, + sizeof(*phys_contig_mem_regions), + GFP_KERNEL); + if (!phys_contig_mem_regions) { + kzfree(ne_mem_region->pages); + kzfree(ne_mem_region); + + return -ENOMEM; + } + + /* + * TODO: Handle non-contiguous memory regions received from user space. + * Hugetlbfs can give 2 MiB / 1 GiB contiguous physical regions. The + * virtual address space can be seen as contiguous, although it is + * mapped underneath to 2 MiB / 1 GiB physical regions e.g. 8 MiB + * virtual address space mapped to 4 physically contiguous regions of 2 + * MiB. + */ + do { + unsigned long tmp_nr_pages = ne_mem_region->nr_pages - + nr_pinned_pages; + struct page **tmp_pages = ne_mem_region->pages + + nr_pinned_pages; + u64 tmp_userspace_addr = mem_region->userspace_addr + + nr_pinned_pages * MIN_MEM_REGION_SIZE; + + gup_rc = get_user_pages(tmp_userspace_addr, tmp_nr_pages, + FOLL_GET, tmp_pages, NULL); + if (gup_rc < 0) { + rc = gup_rc; + + pr_err_ratelimited(NE "Error in gup [rc=%d]\n", rc); + + unpin_user_pages(ne_mem_region->pages, nr_pinned_pages); + + goto free_mem_region; + } + + nr_pinned_pages += gup_rc; + + } while (nr_pinned_pages < ne_mem_region->nr_pages); + + /* + * TODO: Update checks once physically contiguous regions are collected + * based on the user space address and get_user_pages() results. + */ + for (i = 0; i < ne_mem_region->nr_pages; i++) { + if (!PageHuge(ne_mem_region->pages[i])) { + pr_err_ratelimited(NE "Not a hugetlbfs page\n"); + + goto unpin_pages; + } + + if (huge_page_size(page_hstate(ne_mem_region->pages[i])) != + MIN_MEM_REGION_SIZE) { + pr_err_ratelimited(NE "The page size isn't 2 MiB\n"); + + goto unpin_pages; + } + + /* + * TODO: Update once handled non-contiguous memory regions + * received from user space. + */ + phys_contig_mem_regions[i] = ne_mem_region->pages[i]; + } + + /* + * TODO: Update once handled non-contiguous memory regions received + * from user space. + */ + nr_phys_contig_mem_regions = ne_mem_region->nr_pages; + + if ((ne_enclave->nr_mem_regions + nr_phys_contig_mem_regions) > + ne_enclave->max_mem_regions) { + pr_err_ratelimited(NE "Reached max memory regions %lld\n", + ne_enclave->max_mem_regions); + + goto unpin_pages; + } + + for (i = 0; i < nr_phys_contig_mem_regions; i++) { + u64 phys_addr = page_to_phys(phys_contig_mem_regions[i]); + + slot_add_mem_req.slot_uid = ne_enclave->slot_uid; + slot_add_mem_req.paddr = phys_addr; + /* + * TODO: Update memory size of physical contiguous memory + * region, in case of non-contiguous memory regions received + * from user space. + */ + slot_add_mem_req.size = MIN_MEM_REGION_SIZE; + + rc = ne_do_request(ne_enclave->pdev, SLOT_ADD_MEM, + &slot_add_mem_req, sizeof(slot_add_mem_req), + &cmd_reply, sizeof(cmd_reply)); + if (rc < 0) { + pr_err_ratelimited(NE "Error in slot add mem [rc=%d]\n", + rc); + + /* TODO: Only unpin memory regions not added. */ + goto unpin_pages; + } + + ne_enclave->nr_mem_regions++; + + memset(&slot_add_mem_req, 0, sizeof(slot_add_mem_req)); + memset(&cmd_reply, 0, sizeof(cmd_reply)); + } + + list_add(&ne_mem_region->mem_region_list_entry, + &ne_enclave->mem_regions_list); + + kzfree(phys_contig_mem_regions); + + return 0; + +unpin_pages: + unpin_user_pages(ne_mem_region->pages, ne_mem_region->nr_pages); +free_mem_region: + kzfree(phys_contig_mem_regions); + kzfree(ne_mem_region->pages); + kzfree(ne_mem_region); + + return rc; +} + static int ne_enclave_open(struct inode *node, struct file *file) { return 0; @@ -319,6 +547,34 @@ static long ne_enclave_ioctl(struct file *file, unsigned int cmd, return 0; } + case KVM_SET_USER_MEMORY_REGION: { + struct kvm_userspace_memory_region mem_region = {}; + int rc = -EINVAL; + + if (copy_from_user(&mem_region, (void *)arg, + sizeof(mem_region))) { + pr_err_ratelimited(NE "Error in copy from user\n"); + + return -EFAULT; + } + + mutex_lock(&ne_enclave->enclave_info_mutex); + + if (ne_enclave->state != NE_STATE_INIT) { + pr_err_ratelimited(NE "Enclave isn't in init state\n"); + + mutex_unlock(&ne_enclave->enclave_info_mutex); + + return -EINVAL; + } + + rc = ne_set_user_memory_region_ioctl(ne_enclave, &mem_region); + + mutex_unlock(&ne_enclave->enclave_info_mutex); + + return rc; + } + default: return -ENOTTY; } From patchwork Fri May 22 06:29:40 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paraschiv, Andra-Irina" X-Patchwork-Id: 11564753 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B8A1E138A for ; Fri, 22 May 2020 06:32:26 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A208F207BC for ; Fri, 22 May 2020 06:32:26 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="juikhiwj" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728690AbgEVGcN (ORCPT ); Fri, 22 May 2020 02:32:13 -0400 Received: from smtp-fw-4101.amazon.com ([72.21.198.25]:42883 "EHLO smtp-fw-4101.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728161AbgEVGcN (ORCPT ); Fri, 22 May 2020 02:32:13 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1590129132; x=1621665132; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=vnbxYk56jIHoNZd6W/+t/lpoean4SoKA2FhzfkWsOJ4=; b=juikhiwjHb/WRpcKyg/Kw6i9Kr7yGIRCzzBDzCxzamriErgRwx6Dr/o/ tp+bImjG0V2VvPBrlwobtsaKfG15pJPsd7TbmVgsCoMSsF0qyoLAkLicP dcgqNn6fvN9hBxGLv36jqYb9PAGBx92V3KDjeHA3JNM4h/FwIuBM6PGyS 8=; IronPort-SDR: tptkNhe8SI5h/Yj9NOoF3NtaZ40x4RrtNY2hj3MTU01pgYzt8UZQETKRf5zLO9yt6LMe98zHwq DBp/Ndt5CuwQ== X-IronPort-AV: E=Sophos;i="5.73,420,1583193600"; d="scan'208";a="31715549" Received: from iad12-co-svc-p1-lb1-vlan3.amazon.com (HELO email-inbound-relay-1e-c7c08562.us-east-1.amazon.com) ([10.43.8.6]) by smtp-border-fw-out-4101.iad4.amazon.com with ESMTP; 22 May 2020 06:32:00 +0000 Received: from EX13MTAUEA002.ant.amazon.com (iad55-ws-svc-p15-lb9-vlan2.iad.amazon.com [10.40.159.162]) by email-inbound-relay-1e-c7c08562.us-east-1.amazon.com (Postfix) with ESMTPS id 05F9B24103D; Fri, 22 May 2020 06:31:58 +0000 (UTC) Received: from EX13D16EUB003.ant.amazon.com (10.43.166.99) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Fri, 22 May 2020 06:31:58 +0000 Received: from 38f9d34ed3b1.ant.amazon.com (10.43.161.175) by EX13D16EUB003.ant.amazon.com (10.43.166.99) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Fri, 22 May 2020 06:31:48 +0000 From: Andra Paraschiv To: CC: Anthony Liguori , Benjamin Herrenschmidt , Colm MacCarthaigh , "Bjoern Doebel" , David Woodhouse , "Frank van der Linden" , Alexander Graf , "Martin Pohlack" , Matt Wilson , Paolo Bonzini , Balbir Singh , Stefano Garzarella , Stefan Hajnoczi , Stewart Smith , Uwe Dannowski , , , Andra Paraschiv Subject: [PATCH v2 12/18] nitro_enclaves: Add logic for enclave start Date: Fri, 22 May 2020 09:29:40 +0300 Message-ID: <20200522062946.28973-13-andraprs@amazon.com> X-Mailer: git-send-email 2.20.1 (Apple Git-117) In-Reply-To: <20200522062946.28973-1-andraprs@amazon.com> References: <20200522062946.28973-1-andraprs@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.161.175] X-ClientProxiedBy: EX13D27UWA001.ant.amazon.com (10.43.160.19) To EX13D16EUB003.ant.amazon.com (10.43.166.99) Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org After all the enclave resources are set, the enclave is ready for beginning to run. Add ioctl command logic for starting an enclave after all its resources, memory regions and CPUs, have been set. The enclave start information includes the local channel addressing - vsock CID - and the flags associated with the enclave. Signed-off-by: Alexandru Vasile Signed-off-by: Andra Paraschiv --- drivers/virt/nitro_enclaves/ne_misc_dev.c | 107 ++++++++++++++++++++++ 1 file changed, 107 insertions(+) diff --git a/drivers/virt/nitro_enclaves/ne_misc_dev.c b/drivers/virt/nitro_enclaves/ne_misc_dev.c index a5c6185613b9..2109ed3f2f53 100644 --- a/drivers/virt/nitro_enclaves/ne_misc_dev.c +++ b/drivers/virt/nitro_enclaves/ne_misc_dev.c @@ -468,6 +468,53 @@ static int ne_set_user_memory_region_ioctl(struct ne_enclave *ne_enclave, return rc; } +/** + * ne_enclave_start_ioctl - Trigger enclave start after the enclave resources, + * such as memory and CPU, have been set. + * + * This function gets called with the ne_enclave mutex held. + * + * @ne_enclave: private data associated with the current enclave. + * @enclave_start_metadata: enclave metadata that includes enclave cid and + * flags and the slot uid. + * + * @returns: 0 on success, negative return value on failure. + */ +static int ne_enclave_start_ioctl(struct ne_enclave *ne_enclave, + struct enclave_start_metadata *enclave_start_metadata) +{ + struct ne_pci_dev_cmd_reply cmd_reply = {}; + struct enclave_start_req enclave_start_req = {}; + int rc = -EINVAL; + + if (WARN_ON(!ne_enclave) || WARN_ON(!ne_enclave->pdev)) + return -EINVAL; + + if (!enclave_start_metadata) + return -EINVAL; + + enclave_start_metadata->slot_uid = ne_enclave->slot_uid; + + enclave_start_req.enclave_cid = enclave_start_metadata->enclave_cid; + enclave_start_req.flags = enclave_start_metadata->flags; + enclave_start_req.slot_uid = enclave_start_metadata->slot_uid; + + rc = ne_do_request(ne_enclave->pdev, ENCLAVE_START, &enclave_start_req, + sizeof(enclave_start_req), &cmd_reply, + sizeof(cmd_reply)); + if (rc < 0) { + pr_err_ratelimited(NE "Error in enclave start [rc=%d]\n", rc); + + return rc; + } + + ne_enclave->state = NE_STATE_RUNNING; + + enclave_start_metadata->enclave_cid = cmd_reply.enclave_cid; + + return 0; +} + static int ne_enclave_open(struct inode *node, struct file *file) { return 0; @@ -575,6 +622,66 @@ static long ne_enclave_ioctl(struct file *file, unsigned int cmd, return rc; } + case NE_START_ENCLAVE: { + struct enclave_start_metadata enclave_start_metadata = {}; + int rc = -EINVAL; + + if (copy_from_user(&enclave_start_metadata, (void *)arg, + sizeof(enclave_start_metadata))) { + pr_err_ratelimited(NE "Error in copy from user\n"); + + return -EFAULT; + } + + mutex_lock(&ne_enclave->enclave_info_mutex); + + if (ne_enclave->state != NE_STATE_INIT) { + pr_err_ratelimited(NE "Enclave isn't in init state\n"); + + mutex_unlock(&ne_enclave->enclave_info_mutex); + + return -EINVAL; + } + + if (!ne_enclave->nr_mem_regions) { + pr_err_ratelimited(NE "Enclave has no mem regions\n"); + + mutex_unlock(&ne_enclave->enclave_info_mutex); + + return -EINVAL; + } + + if (!ne_enclave->nr_vcpus) { + pr_err_ratelimited(NE "Enclave has no vcpus\n"); + + mutex_unlock(&ne_enclave->enclave_info_mutex); + + return -EINVAL; + } + + if (!cpumask_empty(ne_enclave->cpu_siblings)) { + pr_err_ratelimited(NE "CPU siblings not used\n"); + + mutex_unlock(&ne_enclave->enclave_info_mutex); + + return -EINVAL; + } + + rc = ne_enclave_start_ioctl(ne_enclave, + &enclave_start_metadata); + + mutex_unlock(&ne_enclave->enclave_info_mutex); + + if (copy_to_user((void *)arg, &enclave_start_metadata, + sizeof(enclave_start_metadata))) { + pr_err_ratelimited(NE "Error in copy to user\n"); + + return -EFAULT; + } + + return rc; + } + default: return -ENOTTY; } From patchwork Fri May 22 06:29:41 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paraschiv, Andra-Irina" X-Patchwork-Id: 11564749 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 761A7138A for ; Fri, 22 May 2020 06:32:17 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 5ABA8207BC for ; Fri, 22 May 2020 06:32:17 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="igUgT41j" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728734AbgEVGcQ (ORCPT ); Fri, 22 May 2020 02:32:16 -0400 Received: from smtp-fw-4101.amazon.com ([72.21.198.25]:42883 "EHLO smtp-fw-4101.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728706AbgEVGcO (ORCPT ); Fri, 22 May 2020 02:32:14 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1590129134; x=1621665134; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=MA1K62X9Pr2El5WkX3JI3Rb+WSIH+i6M6DjD1fPg7MU=; b=igUgT41jDZD4h0T5pF3BRUitQ7/2YALYOH9QQlvmliDzRSYf7FcnCACf jOI4scsz7tpqEBEDsn3ZuzjYMpiVxfMPrpAyL/3TePBczkWHI3aALO5w7 nhItAObg8bTsEHuxxvKDZvuyh1m1l8+Cwg0ujy76LGQbYyOC0jyhXiZLV g=; IronPort-SDR: ThmIP1yxEWKyHoy3CfNR718JsLLd/OruSSwzJgQB4vlgdGoognB56iyc7G/pFCrOOHP4WZ0ott HNSCcQ7gh5eA== X-IronPort-AV: E=Sophos;i="5.73,420,1583193600"; d="scan'208";a="31715560" Received: from iad12-co-svc-p1-lb1-vlan3.amazon.com (HELO email-inbound-relay-1a-715bee71.us-east-1.amazon.com) ([10.43.8.6]) by smtp-border-fw-out-4101.iad4.amazon.com with ESMTP; 22 May 2020 06:32:08 +0000 Received: from EX13MTAUEA002.ant.amazon.com (iad55-ws-svc-p15-lb9-vlan2.iad.amazon.com [10.40.159.162]) by email-inbound-relay-1a-715bee71.us-east-1.amazon.com (Postfix) with ESMTPS id 17244A25F7; Fri, 22 May 2020 06:32:07 +0000 (UTC) Received: from EX13D16EUB003.ant.amazon.com (10.43.166.99) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Fri, 22 May 2020 06:32:06 +0000 Received: from 38f9d34ed3b1.ant.amazon.com (10.43.161.175) by EX13D16EUB003.ant.amazon.com (10.43.166.99) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Fri, 22 May 2020 06:31:57 +0000 From: Andra Paraschiv To: CC: Anthony Liguori , Benjamin Herrenschmidt , Colm MacCarthaigh , "Bjoern Doebel" , David Woodhouse , "Frank van der Linden" , Alexander Graf , "Martin Pohlack" , Matt Wilson , Paolo Bonzini , Balbir Singh , Stefano Garzarella , Stefan Hajnoczi , Stewart Smith , Uwe Dannowski , , , Andra Paraschiv Subject: [PATCH v2 13/18] nitro_enclaves: Add logic for enclave termination Date: Fri, 22 May 2020 09:29:41 +0300 Message-ID: <20200522062946.28973-14-andraprs@amazon.com> X-Mailer: git-send-email 2.20.1 (Apple Git-117) In-Reply-To: <20200522062946.28973-1-andraprs@amazon.com> References: <20200522062946.28973-1-andraprs@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.161.175] X-ClientProxiedBy: EX13D27UWA001.ant.amazon.com (10.43.160.19) To EX13D16EUB003.ant.amazon.com (10.43.166.99) Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org An enclave is associated with an fd that is returned after the enclave creation logic is completed. This enclave fd is further used to setup enclave resources. Once the enclave needs to be terminated, the enclave fd is closed. Add logic for enclave termination, that is mapped to the enclave fd release callback. Free the internal enclave info used for bookkeeping. Signed-off-by: Alexandru Vasile Signed-off-by: Andra Paraschiv --- drivers/virt/nitro_enclaves/ne_misc_dev.c | 177 ++++++++++++++++++++++ 1 file changed, 177 insertions(+) diff --git a/drivers/virt/nitro_enclaves/ne_misc_dev.c b/drivers/virt/nitro_enclaves/ne_misc_dev.c index 2109ed3f2f53..a6f8d2e98e67 100644 --- a/drivers/virt/nitro_enclaves/ne_misc_dev.c +++ b/drivers/virt/nitro_enclaves/ne_misc_dev.c @@ -689,9 +689,186 @@ static long ne_enclave_ioctl(struct file *file, unsigned int cmd, return 0; } +/** + * ne_enclave_remove_all_mem_region_entries - Remove all memory region + * entries from the enclave data structure. + * + * This function gets called with the ne_enclave mutex held. + * + * @ne_enclave: private data associated with the current enclave. + */ +static void ne_enclave_remove_all_mem_region_entries( + struct ne_enclave *ne_enclave) +{ + struct ne_mem_region *ne_mem_region = NULL; + struct ne_mem_region *ne_mem_region_tmp = NULL; + + if (WARN_ON(!ne_enclave)) + return; + + list_for_each_entry_safe(ne_mem_region, ne_mem_region_tmp, + &ne_enclave->mem_regions_list, + mem_region_list_entry) { + list_del(&ne_mem_region->mem_region_list_entry); + + unpin_user_pages(ne_mem_region->pages, + ne_mem_region->nr_pages); + + kzfree(ne_mem_region->pages); + + kzfree(ne_mem_region); + } +} + +/** + * ne_enclave_remove_all_vcpu_id_entries - Remove all vCPU id entries + * from the enclave data structure. + * + * This function gets called with the ne_enclave mutex held. + * + * @ne_enclave: private data associated with the current enclave. + */ +static void ne_enclave_remove_all_vcpu_id_entries(struct ne_enclave *ne_enclave) +{ + unsigned int cpu = 0; + struct ne_vcpu_id *ne_vcpu_id = NULL; + struct ne_vcpu_id *ne_vcpu_id_tmp = NULL; + + if (WARN_ON(!ne_enclave)) + return; + + mutex_lock(&ne_cpu_pool.mutex); + + list_for_each_entry_safe(ne_vcpu_id, ne_vcpu_id_tmp, + &ne_enclave->vcpu_ids_list, + vcpu_id_list_entry) { + list_del(&ne_vcpu_id->vcpu_id_list_entry); + + /* Update the available CPU pool. */ + cpumask_set_cpu(ne_vcpu_id->vcpu_id, ne_cpu_pool.avail); + + kzfree(ne_vcpu_id); + } + + /* If any siblings left in the enclave CPU pool, move to available. */ + for_each_cpu(cpu, ne_enclave->cpu_siblings) { + cpumask_clear_cpu(cpu, ne_enclave->cpu_siblings); + + cpumask_set_cpu(cpu, ne_cpu_pool.avail); + } + + free_cpumask_var(ne_enclave->cpu_siblings); + + mutex_unlock(&ne_cpu_pool.mutex); +} + +/** + * ne_pci_dev_remove_enclave_entry - Remove enclave entry from the data + * structure that is part of the PCI device private data. + * + * This function gets called with the ne_pci_dev enclave mutex held. + * + * @ne_enclave: private data associated with the current enclave. + * @ne_pci_dev: private data associated with the PCI device. + */ +static void ne_pci_dev_remove_enclave_entry(struct ne_enclave *ne_enclave, + struct ne_pci_dev *ne_pci_dev) +{ + struct ne_enclave *ne_enclave_entry = NULL; + struct ne_enclave *ne_enclave_entry_tmp = NULL; + + if (WARN_ON(!ne_enclave) || WARN_ON(!ne_pci_dev)) + return; + + list_for_each_entry_safe(ne_enclave_entry, ne_enclave_entry_tmp, + &ne_pci_dev->enclaves_list, + enclave_list_entry) { + if (ne_enclave_entry->slot_uid == ne_enclave->slot_uid) { + list_del(&ne_enclave_entry->enclave_list_entry); + + break; + } + } +} + static int ne_enclave_release(struct inode *inode, struct file *file) { + struct ne_pci_dev_cmd_reply cmd_reply = {}; + struct enclave_stop_req enclave_stop_request = {}; + struct ne_enclave *ne_enclave = file->private_data; + struct ne_pci_dev *ne_pci_dev = NULL; + int rc = -EINVAL; + struct slot_free_req slot_free_req = {}; + + if (WARN_ON(!ne_enclave)) + return 0; + + /* + * Early exit in case there is an error in the enclave creation logic + * and fput() is called on the cleanup path. + */ + if (!ne_enclave->slot_uid) + return 0; + + if (WARN_ON(!ne_enclave->pdev)) + return -EINVAL; + + ne_pci_dev = pci_get_drvdata(ne_enclave->pdev); + if (WARN_ON(!ne_pci_dev)) + return -EINVAL; + + /* + * Acquire the enclave list mutex before the enclave mutex + * in order to avoid deadlocks with @ref ne_event_work_handler. + */ + mutex_lock(&ne_pci_dev->enclaves_list_mutex); + mutex_lock(&ne_enclave->enclave_info_mutex); + + if (ne_enclave->state != NE_STATE_INIT && + ne_enclave->state != NE_STATE_STOPPED) { + enclave_stop_request.slot_uid = ne_enclave->slot_uid; + + rc = ne_do_request(ne_enclave->pdev, ENCLAVE_STOP, + &enclave_stop_request, + sizeof(enclave_stop_request), &cmd_reply, + sizeof(cmd_reply)); + if (rc < 0) { + pr_err_ratelimited(NE "Error in enclave stop [rc=%d]\n", + rc); + + goto unlock_mutex; + } + + memset(&cmd_reply, 0, sizeof(cmd_reply)); + } + + slot_free_req.slot_uid = ne_enclave->slot_uid; + + rc = ne_do_request(ne_enclave->pdev, SLOT_FREE, &slot_free_req, + sizeof(slot_free_req), &cmd_reply, + sizeof(cmd_reply)); + if (rc < 0) { + pr_err_ratelimited(NE "Error in slot free [rc=%d]\n", rc); + + goto unlock_mutex; + } + + ne_pci_dev_remove_enclave_entry(ne_enclave, ne_pci_dev); + ne_enclave_remove_all_mem_region_entries(ne_enclave); + ne_enclave_remove_all_vcpu_id_entries(ne_enclave); + + mutex_unlock(&ne_enclave->enclave_info_mutex); + mutex_unlock(&ne_pci_dev->enclaves_list_mutex); + + kzfree(ne_enclave); + return 0; + +unlock_mutex: + mutex_unlock(&ne_enclave->enclave_info_mutex); + mutex_unlock(&ne_pci_dev->enclaves_list_mutex); + + return rc; } static __poll_t ne_enclave_poll(struct file *file, poll_table *wait) From patchwork Fri May 22 06:29:42 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paraschiv, Andra-Irina" X-Patchwork-Id: 11564751 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9B2B014B7 for ; Fri, 22 May 2020 06:32:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 84B57207FB for ; Fri, 22 May 2020 06:32:24 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="l2Hm749a" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728760AbgEVGcV (ORCPT ); Fri, 22 May 2020 02:32:21 -0400 Received: from smtp-fw-9101.amazon.com ([207.171.184.25]:33895 "EHLO smtp-fw-9101.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728031AbgEVGcT (ORCPT ); Fri, 22 May 2020 02:32:19 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1590129139; x=1621665139; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=pVzeZuh/oMq5OlkcQrxw5Piwtvs/xx1oUDu7oEGLlNQ=; b=l2Hm749aKvwifT7t9qnJTyzjvOSgHaZMHiWV5KEXmHtkNFkYvdvdN3OZ 24miw58/6wTBWgpz0pkFrz7ohamJJa4X/ybsjAYUokepEpOBMVJqiXDFM fCbp4rSwApguU7jS3poXcr2j1qa17w2W501UQj6uyldoBU1EvZcF1HX9m c=; IronPort-SDR: n9jLwxlKdot8AlUwWNVhJ6ZDbD+2QOQLLiP+FFq4nveiOrQ7egDRuioIZuYcmsR6F6vsreTBQk Vv7p/QlB2IQw== X-IronPort-AV: E=Sophos;i="5.73,420,1583193600"; d="scan'208";a="36944494" Received: from sea32-co-svc-lb4-vlan3.sea.corp.amazon.com (HELO email-inbound-relay-1a-af6a10df.us-east-1.amazon.com) ([10.47.23.38]) by smtp-border-fw-out-9101.sea19.amazon.com with ESMTP; 22 May 2020 06:32:18 +0000 Received: from EX13MTAUEA002.ant.amazon.com (iad55-ws-svc-p15-lb9-vlan3.iad.amazon.com [10.40.159.166]) by email-inbound-relay-1a-af6a10df.us-east-1.amazon.com (Postfix) with ESMTPS id 38AC0A11D4; Fri, 22 May 2020 06:32:16 +0000 (UTC) Received: from EX13D16EUB003.ant.amazon.com (10.43.166.99) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Fri, 22 May 2020 06:32:15 +0000 Received: from 38f9d34ed3b1.ant.amazon.com (10.43.161.175) by EX13D16EUB003.ant.amazon.com (10.43.166.99) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Fri, 22 May 2020 06:32:06 +0000 From: Andra Paraschiv To: CC: Anthony Liguori , Benjamin Herrenschmidt , Colm MacCarthaigh , "Bjoern Doebel" , David Woodhouse , "Frank van der Linden" , Alexander Graf , "Martin Pohlack" , Matt Wilson , Paolo Bonzini , Balbir Singh , Stefano Garzarella , Stefan Hajnoczi , Stewart Smith , Uwe Dannowski , , , Andra Paraschiv Subject: [PATCH v2 14/18] nitro_enclaves: Add Kconfig for the Nitro Enclaves driver Date: Fri, 22 May 2020 09:29:42 +0300 Message-ID: <20200522062946.28973-15-andraprs@amazon.com> X-Mailer: git-send-email 2.20.1 (Apple Git-117) In-Reply-To: <20200522062946.28973-1-andraprs@amazon.com> References: <20200522062946.28973-1-andraprs@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.161.175] X-ClientProxiedBy: EX13D27UWA001.ant.amazon.com (10.43.160.19) To EX13D16EUB003.ant.amazon.com (10.43.166.99) Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Signed-off-by: Andra Paraschiv --- drivers/virt/Kconfig | 2 ++ drivers/virt/nitro_enclaves/Kconfig | 28 ++++++++++++++++++++++++++++ 2 files changed, 30 insertions(+) create mode 100644 drivers/virt/nitro_enclaves/Kconfig diff --git a/drivers/virt/Kconfig b/drivers/virt/Kconfig index 363af2eaf2ba..ae82460cdec2 100644 --- a/drivers/virt/Kconfig +++ b/drivers/virt/Kconfig @@ -32,4 +32,6 @@ config FSL_HV_MANAGER partition shuts down. source "drivers/virt/vboxguest/Kconfig" + +source "drivers/virt/nitro_enclaves/Kconfig" endif diff --git a/drivers/virt/nitro_enclaves/Kconfig b/drivers/virt/nitro_enclaves/Kconfig new file mode 100644 index 000000000000..2298a4bf609b --- /dev/null +++ b/drivers/virt/nitro_enclaves/Kconfig @@ -0,0 +1,28 @@ +# SPDX-License-Identifier: GPL-2.0 +# +# Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. +# +# This program is free software; you can redistribute it and/or modify it +# under the terms and conditions of the GNU General Public License, +# version 2, as published by the Free Software Foundation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, see . + +# Amazon Nitro Enclaves (NE) support. +# Nitro is a hypervisor that has been developed by Amazon. + +config NITRO_ENCLAVES + tristate "Nitro Enclaves Support" + depends on HOTPLUG_CPU + help + This driver consists of support for enclave lifetime management + for Nitro Enclaves (NE). + + To compile this driver as a module, choose M here. + The module will be called nitro_enclaves. From patchwork Fri May 22 06:29:43 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paraschiv, Andra-Irina" X-Patchwork-Id: 11564755 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A2970138A for ; Fri, 22 May 2020 06:32:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 8B893207BC for ; Fri, 22 May 2020 06:32:37 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="kHR9osjL" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728788AbgEVGce (ORCPT ); Fri, 22 May 2020 02:32:34 -0400 Received: from smtp-fw-9102.amazon.com ([207.171.184.29]:41560 "EHLO smtp-fw-9102.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728267AbgEVGce (ORCPT ); Fri, 22 May 2020 02:32:34 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1590129153; x=1621665153; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=IKu5qeN5RGz8z3cFUrYa21vyJqeYbxPu8OnXuStK4g0=; b=kHR9osjLIQ1i/DpIiFY1216xEXO+vEdZrbq3gr5KZWvkMJ+sb2ORssBQ yOLvarCtTWd/r1jz2pNsMYJm20Tzasmb9T9kjB90bWFKKV+lARI6XVGio GjDOSvrDg85F+zM56L/yLsaPeY6G8Uw+PAeHbaRBH8bs2eAMIrYJzBFKu s=; IronPort-SDR: rjGjTri4E9k3hRP5BYLONsLft6Tuhed50qE20F/QMKe4gsenNRCA4jucWZ9f0ol1xCdxSANJ2h N33a2r7Vht9g== X-IronPort-AV: E=Sophos;i="5.73,420,1583193600"; d="scan'208";a="45244360" Received: from sea32-co-svc-lb4-vlan3.sea.corp.amazon.com (HELO email-inbound-relay-1e-a70de69e.us-east-1.amazon.com) ([10.47.23.38]) by smtp-border-fw-out-9102.sea19.amazon.com with ESMTP; 22 May 2020 06:32:33 +0000 Received: from EX13MTAUEA002.ant.amazon.com (iad55-ws-svc-p15-lb9-vlan2.iad.amazon.com [10.40.159.162]) by email-inbound-relay-1e-a70de69e.us-east-1.amazon.com (Postfix) with ESMTPS id B7285A1876; Fri, 22 May 2020 06:32:30 +0000 (UTC) Received: from EX13D16EUB003.ant.amazon.com (10.43.166.99) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Fri, 22 May 2020 06:32:30 +0000 Received: from 38f9d34ed3b1.ant.amazon.com (10.43.161.175) by EX13D16EUB003.ant.amazon.com (10.43.166.99) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Fri, 22 May 2020 06:32:21 +0000 From: Andra Paraschiv To: CC: Anthony Liguori , Benjamin Herrenschmidt , Colm MacCarthaigh , "Bjoern Doebel" , David Woodhouse , "Frank van der Linden" , Alexander Graf , "Martin Pohlack" , Matt Wilson , Paolo Bonzini , Balbir Singh , Stefano Garzarella , Stefan Hajnoczi , Stewart Smith , Uwe Dannowski , , , Andra Paraschiv Subject: [PATCH v2 15/18] nitro_enclaves: Add Makefile for the Nitro Enclaves driver Date: Fri, 22 May 2020 09:29:43 +0300 Message-ID: <20200522062946.28973-16-andraprs@amazon.com> X-Mailer: git-send-email 2.20.1 (Apple Git-117) In-Reply-To: <20200522062946.28973-1-andraprs@amazon.com> References: <20200522062946.28973-1-andraprs@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.161.175] X-ClientProxiedBy: EX13D24UWA003.ant.amazon.com (10.43.160.195) To EX13D16EUB003.ant.amazon.com (10.43.166.99) Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Signed-off-by: Andra Paraschiv --- drivers/virt/Makefile | 2 ++ drivers/virt/nitro_enclaves/Makefile | 23 +++++++++++++++++++++++ 2 files changed, 25 insertions(+) create mode 100644 drivers/virt/nitro_enclaves/Makefile diff --git a/drivers/virt/Makefile b/drivers/virt/Makefile index fd331247c27a..f28425ce4b39 100644 --- a/drivers/virt/Makefile +++ b/drivers/virt/Makefile @@ -5,3 +5,5 @@ obj-$(CONFIG_FSL_HV_MANAGER) += fsl_hypervisor.o obj-y += vboxguest/ + +obj-$(CONFIG_NITRO_ENCLAVES) += nitro_enclaves/ diff --git a/drivers/virt/nitro_enclaves/Makefile b/drivers/virt/nitro_enclaves/Makefile new file mode 100644 index 000000000000..9109aed41070 --- /dev/null +++ b/drivers/virt/nitro_enclaves/Makefile @@ -0,0 +1,23 @@ +# SPDX-License-Identifier: GPL-2.0 +# +# Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. +# +# This program is free software; you can redistribute it and/or modify it +# under the terms and conditions of the GNU General Public License, +# version 2, as published by the Free Software Foundation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, see . + +# Enclave lifetime management support for Nitro Enclaves (NE). + +obj-$(CONFIG_NITRO_ENCLAVES) += nitro_enclaves.o + +nitro_enclaves-y := ne_pci_dev.o ne_misc_dev.o + +ccflags-y += -Wall From patchwork Fri May 22 06:29:44 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paraschiv, Andra-Irina" X-Patchwork-Id: 11564757 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4AB21138A for ; Fri, 22 May 2020 06:32:50 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 24A5F207BC for ; Fri, 22 May 2020 06:32:50 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="j5FLe8FP" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728816AbgEVGcq (ORCPT ); Fri, 22 May 2020 02:32:46 -0400 Received: from smtp-fw-33001.amazon.com ([207.171.190.10]:8476 "EHLO smtp-fw-33001.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728183AbgEVGcq (ORCPT ); Fri, 22 May 2020 02:32:46 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1590129162; x=1621665162; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=W2HeL6GA33boG647C+EUOE88cN/TGT0BZmVgBkDhBVQ=; b=j5FLe8FPjWOQbuqXV13e2BIwwIylnQIMclI0P2mqkAAT4CyTJO/Xmalb zBnygtamtwh94qPFnABmM6WYtLdfkVDh5TO2FT/nh/kdCHEVk/Vb7pnAs YHdnYQjIgkHkuNlUIky0T090lUS7v52smnk71Q4Ldf06f4c2iVmii0VrR U=; IronPort-SDR: G9bQ4eJc3uLuc/03isqzi4eu6fgjVewIBp+aNGXwk6tqv9hr50vw5acgyMSS7hmBVRNj7A9aDu swTl9b7dN0pw== X-IronPort-AV: E=Sophos;i="5.73,420,1583193600"; d="scan'208";a="46675166" Received: from sea32-co-svc-lb4-vlan2.sea.corp.amazon.com (HELO email-inbound-relay-1e-57e1d233.us-east-1.amazon.com) ([10.47.23.34]) by smtp-border-fw-out-33001.sea14.amazon.com with ESMTP; 22 May 2020 06:32:41 +0000 Received: from EX13MTAUEA002.ant.amazon.com (iad55-ws-svc-p15-lb9-vlan3.iad.amazon.com [10.40.159.166]) by email-inbound-relay-1e-57e1d233.us-east-1.amazon.com (Postfix) with ESMTPS id 8A873141809; Fri, 22 May 2020 06:32:39 +0000 (UTC) Received: from EX13D16EUB003.ant.amazon.com (10.43.166.99) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Fri, 22 May 2020 06:32:39 +0000 Received: from 38f9d34ed3b1.ant.amazon.com (10.43.161.175) by EX13D16EUB003.ant.amazon.com (10.43.166.99) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Fri, 22 May 2020 06:32:29 +0000 From: Andra Paraschiv To: CC: Anthony Liguori , Benjamin Herrenschmidt , Colm MacCarthaigh , "Bjoern Doebel" , David Woodhouse , "Frank van der Linden" , Alexander Graf , "Martin Pohlack" , Matt Wilson , Paolo Bonzini , Balbir Singh , Stefano Garzarella , Stefan Hajnoczi , Stewart Smith , Uwe Dannowski , , , Andra Paraschiv Subject: [PATCH v2 16/18] nitro_enclaves: Add sample for ioctl interface usage Date: Fri, 22 May 2020 09:29:44 +0300 Message-ID: <20200522062946.28973-17-andraprs@amazon.com> X-Mailer: git-send-email 2.20.1 (Apple Git-117) In-Reply-To: <20200522062946.28973-1-andraprs@amazon.com> References: <20200522062946.28973-1-andraprs@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.161.175] X-ClientProxiedBy: EX13D24UWA003.ant.amazon.com (10.43.160.195) To EX13D16EUB003.ant.amazon.com (10.43.166.99) Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Signed-off-by: Alexandru Vasile Signed-off-by: Andra Paraschiv --- samples/nitro_enclaves/.gitignore | 2 + samples/nitro_enclaves/Makefile | 28 + .../include/linux/nitro_enclaves.h | 23 + .../include/uapi/linux/nitro_enclaves.h | 77 +++ samples/nitro_enclaves/ne_ioctl_sample.c | 502 ++++++++++++++++++ 5 files changed, 632 insertions(+) create mode 100644 samples/nitro_enclaves/.gitignore create mode 100644 samples/nitro_enclaves/Makefile create mode 100644 samples/nitro_enclaves/include/linux/nitro_enclaves.h create mode 100644 samples/nitro_enclaves/include/uapi/linux/nitro_enclaves.h create mode 100644 samples/nitro_enclaves/ne_ioctl_sample.c diff --git a/samples/nitro_enclaves/.gitignore b/samples/nitro_enclaves/.gitignore new file mode 100644 index 000000000000..827934129c90 --- /dev/null +++ b/samples/nitro_enclaves/.gitignore @@ -0,0 +1,2 @@ +# SPDX-License-Identifier: GPL-2.0 +ne_ioctl_sample diff --git a/samples/nitro_enclaves/Makefile b/samples/nitro_enclaves/Makefile new file mode 100644 index 000000000000..2acdc6e0c492 --- /dev/null +++ b/samples/nitro_enclaves/Makefile @@ -0,0 +1,28 @@ +# SPDX-License-Identifier: GPL-2.0 +# +# Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. +# +# This program is free software; you can redistribute it and/or modify it +# under the terms and conditions of the GNU General Public License, +# version 2, as published by the Free Software Foundation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, see . + +# Enclave lifetime management support for Nitro Enclaves (NE) - ioctl sample +# usage. + +.PHONY: all clean + +CFLAGS += -Wall -Iinclude/ + +all: + $(CC) $(CFLAGS) -o ne_ioctl_sample ne_ioctl_sample.c -lpthread + +clean: + rm -f ne_ioctl_sample diff --git a/samples/nitro_enclaves/include/linux/nitro_enclaves.h b/samples/nitro_enclaves/include/linux/nitro_enclaves.h new file mode 100644 index 000000000000..7e593a9fbf8c --- /dev/null +++ b/samples/nitro_enclaves/include/linux/nitro_enclaves.h @@ -0,0 +1,23 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms and conditions of the GNU General Public License, + * version 2, as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, see . + */ + +#ifndef _LINUX_NITRO_ENCLAVES_H_ +#define _LINUX_NITRO_ENCLAVES_H_ + +#include + +#endif /* _LINUX_NITRO_ENCLAVES_H_ */ diff --git a/samples/nitro_enclaves/include/uapi/linux/nitro_enclaves.h b/samples/nitro_enclaves/include/uapi/linux/nitro_enclaves.h new file mode 100644 index 000000000000..98ba59f15b52 --- /dev/null +++ b/samples/nitro_enclaves/include/uapi/linux/nitro_enclaves.h @@ -0,0 +1,77 @@ +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ +/* + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms and conditions of the GNU General Public License, + * version 2, as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, see . + */ + +#ifndef _UAPI_LINUX_NITRO_ENCLAVES_H_ +#define _UAPI_LINUX_NITRO_ENCLAVES_H_ + +#include +#include + +/* Nitro Enclaves (NE) Kernel Driver Interface */ + +/** + * The command is used to get information needed for in-memory enclave image + * loading e.g. offset in enclave memory to start placing the enclave image. + * + * The image load metadata is an in / out data structure. It includes info + * provided by the caller - flags - and returns the offset in enclave memory + * where to start placing the enclave image. + */ +#define NE_GET_IMAGE_LOAD_METADATA _IOWR(0xAE, 0x20, struct image_load_metadata) + +/** + * The command is used to trigger enclave start after the enclave resources, + * such as memory and CPU, have been set. + * + * The enclave start metadata is an in / out data structure. It includes info + * provided by the caller - enclave cid and flags - and returns the slot uid + * and the cid (if input cid is 0). + */ +#define NE_START_ENCLAVE _IOWR(0xAE, 0x21, struct enclave_start_metadata) + +/* Metadata necessary for in-memory enclave image loading. */ +struct image_load_metadata { + /** + * Flags to determine the enclave image type e.g. Enclave Image Format + * (EIF) (in). + */ + __u64 flags; + + /** + * Offset in enclave memory where to start placing the enclave image + * (out). + */ + __u64 memory_offset; +}; + +/* Setup metadata necessary for enclave start. */ +struct enclave_start_metadata { + /* Flags for the enclave to start with (e.g. debug mode) (in). */ + __u64 flags; + + /** + * Context ID (CID) for the enclave vsock device. If 0 as input, the + * CID is autogenerated by the hypervisor and returned back as output + * by the driver (in/out). + */ + __u64 enclave_cid; + + /* Slot unique id mapped to the enclave to start (out). */ + __u64 slot_uid; +}; + +#endif /* _UAPI_LINUX_NITRO_ENCLAVES_H_ */ diff --git a/samples/nitro_enclaves/ne_ioctl_sample.c b/samples/nitro_enclaves/ne_ioctl_sample.c new file mode 100644 index 000000000000..c9a164b7193e --- /dev/null +++ b/samples/nitro_enclaves/ne_ioctl_sample.c @@ -0,0 +1,502 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms and conditions of the GNU General Public License, + * version 2, as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, see . + */ + +/** + * Sample flow of using the ioctl interface provided by the Nitro Enclaves (NE) + * kernel driver. + * + * Usage + * ----- + * + * Load the nitro_enclaves module, setting also the enclave CPU pool. + * + * See the cpu list section from the kernel documentation. + * https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html + * + * insmod drivers/virt/nitro_enclaves/nitro_enclaves.ko ne_cpus= + * lsmod + * + * Check dmesg for any warnings / errors through the NE driver lifetime / usage. + * The NE logs contain the "nitro_enclaves" pattern. + * + * dmesg + * + * Check the online / offline CPU list. The CPUs from the pool should be + * offlined. + * + * lscpu + * + * Setup hugetlbfs huge pages. + * + * echo > /proc/sys/vm/nr_hugepages + * + * In this example 256 hugepages of 2 MiB are used. + * + * Build and run the NE sample. + * + * make -C samples/nitro_enclaves clean + * make -C samples/nitro_enclaves + * ./samples/nitro_enclaves/ne_ioctl_sample + * + * Unload the nitro_enclaves module. + * + * rmmod nitro_enclaves + * lsmod + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include + +/* Nitro Enclaves (NE) misc device that provides the ioctl interface. */ +#define NE_DEV_NAME "/dev/nitro_enclaves" + +/* Timeout in seconds / milliseconds for each poll event. */ +#define POLL_WAIT_TIME (60) +#define POLL_WAIT_TIME_MS (POLL_WAIT_TIME * 1000) + +/* Amount of time in seconds for the process to keep the enclave alive. */ +#define SLEEP_TIME (300) + +/* Enclave vCPUs metadata. */ +#define DEFAULT_NR_VCPUS (2) + +/* Enclave memory metadata */ +/* Min memory size - 2 MiB */ +#define MIN_MEM_REGION_SIZE (2 * 1024 * 1024) +/* 256 memory regions of 2 MiB */ +#define DEFAULT_NR_MEM_REGIONS (256) + +/* Vsock addressing for enclave image loading heartbeat. */ +#define VSOCK_CID (3) +#define VSOCK_PORT (9000) +#define HEARTBEAT_VALUE (0xb7) + +struct ne_mem_region { + void *mem_addr; + size_t mem_size; +}; + +struct ne_vcpu { + int vcpu_fd; + unsigned int vcpu_id; +}; + +/* Thread function for polling the enclave fd. */ +void *ne_poll_enclave_fd(void *data) +{ + int enclave_fd = *(int *)data; + struct pollfd fds[1] = {}; + int i = 0; + int rc = 0; + + printf("Running from poll thread, enclave fd %d\n", enclave_fd); + + fds[0].fd = enclave_fd; + fds[0].events = POLLIN | POLLERR | POLLHUP; + + /* Keep on polling until the current process is terminated. */ + while (1) { + printf("[iter %d] Polling ...\n", i); + + rc = poll(fds, 1, POLL_WAIT_TIME_MS); + if (rc < 0) { + printf("Error in poll [%m]\n"); + + return NULL; + } + + i++; + + if (!rc) { + printf("Poll: %d seconds elapsed\n", + i * POLL_WAIT_TIME); + + continue; + } + + printf("Poll received value %d\n", fds[0].revents); + } + + return NULL; +} + +/* Allocate memory region that will be used for the enclave. */ +int ne_alloc_mem_region(struct ne_mem_region *ne_mem_region) +{ + if (!ne_mem_region) + return -EINVAL; + + if (!ne_mem_region->mem_size) + return -EINVAL; + + ne_mem_region->mem_addr = mmap(NULL, ne_mem_region->mem_size, + PROT_READ | PROT_WRITE, + MAP_PRIVATE | MAP_ANONYMOUS | + MAP_HUGETLB, -1, 0); + if (ne_mem_region->mem_addr == MAP_FAILED) { + printf("Error in mmap memory [%m]\n"); + + return -1; + } + + return 0; +} + +/* Place enclave image in enclave memory. */ +int ne_load_enclave_image(int enclave_fd, struct ne_mem_region ne_mem_regions[], + char enclave_image_path[]) +{ + struct image_load_metadata image_load_metadata = {}; + int rc = 0; + + if (enclave_fd < 0) + return -EINVAL; + + /* TODO: Set flags based on enclave image type. */ + image_load_metadata.flags = 0; + + rc = ioctl(enclave_fd, NE_GET_IMAGE_LOAD_METADATA, + &image_load_metadata); + if (rc < 0) { + printf("Error in get image load metadata [rc=%d]\n", rc); + + return rc; + } + + printf("Enclave image offset in enclave memory is %lld\n", + image_load_metadata.memory_offset); + + /* + * TODO: Copy enclave image in enclave memory starting from the given + * offset. + */ + + return 0; +} + +/* Wait for a hearbeat from the enclave to check it has booted. */ +int ne_check_enclave_booted(void) +{ + struct sockaddr_vm client_vsock_addr = {}; + socklen_t client_vsock_len = sizeof(client_vsock_addr); + struct pollfd fds[1] = {}; + int rc = 0; + unsigned char recv_buf = 0; + struct sockaddr_vm server_vsock_addr = { + .svm_family = AF_VSOCK, + .svm_cid = VSOCK_CID, + .svm_port = VSOCK_PORT, + }; + int server_vsock_fd = 0; + + server_vsock_fd = socket(AF_VSOCK, SOCK_STREAM, 0); + if (server_vsock_fd < 0) { + rc = server_vsock_fd; + + printf("Error in socket [rc=%d]\n", rc); + + return rc; + } + + rc = bind(server_vsock_fd, (struct sockaddr *)&server_vsock_addr, + sizeof(server_vsock_addr)); + if (rc < 0) { + printf("Error in bind [rc=%d]\n", rc); + + goto out; + } + + rc = listen(server_vsock_fd, 1); + if (rc < 0) { + printf("Error in listen [rc=%d]\n", rc); + + goto out; + } + + fds[0].fd = server_vsock_fd; + fds[0].events = POLLIN; + + rc = poll(fds, 1, POLL_WAIT_TIME_MS); + if (rc < 0) { + printf("Error in poll [%m]\n"); + + goto out; + } + + if (!rc) { + printf("Poll timeout, %d seconds elapsed\n", POLL_WAIT_TIME); + + rc = -ETIMEDOUT; + + goto out; + } + + if ((fds[0].revents & POLLIN) == 0) { + printf("Poll received value %d\n", fds[0].revents); + + rc = -EINVAL; + + goto out; + } + + rc = accept(server_vsock_fd, (struct sockaddr *)&client_vsock_addr, + &client_vsock_len); + if (rc < 0) { + printf("Error in accept [rc=%d]\n", rc); + + goto out; + } + + /* + * Read the heartbeat value that the init process in the enclave sends + * after vsock connect. + */ + rc = read(server_vsock_fd, &recv_buf, sizeof(recv_buf)); + if (rc < 0) { + printf("Error in read [rc=%d]\n", rc); + + goto out; + } + + if (rc != sizeof(recv_buf) || recv_buf != HEARTBEAT_VALUE) { + printf("Read %d instead of %d\n", recv_buf, HEARTBEAT_VALUE); + + goto out; + } + + close(server_vsock_fd); + + return 0; + +out: + close(server_vsock_fd); + + return rc; +} + +/* Set memory region for the given enclave. */ +int ne_set_mem_region(int enclave_fd, struct ne_mem_region ne_mem_region) +{ + struct kvm_userspace_memory_region mem_region = {}; + int rc = 0; + + if (enclave_fd < 0) + return -EINVAL; + + mem_region.slot = 0; + mem_region.memory_size = ne_mem_region.mem_size; + mem_region.userspace_addr = (__u64)ne_mem_region.mem_addr; + mem_region.guest_phys_addr = 0; + + rc = ioctl(enclave_fd, KVM_SET_USER_MEMORY_REGION, &mem_region); + if (rc < 0) { + printf("Error in set user memory region [rc=%d]\n", rc); + + return rc; + } + + return 0; +} + +/* Unmap all the memory regions that were set aside for the enclave. */ +void ne_free_mem_regions(struct ne_mem_region ne_mem_regions[]) +{ + unsigned int i = 0; + + for (i = 0; i < DEFAULT_NR_MEM_REGIONS; i++) + munmap(ne_mem_regions[i].mem_addr, ne_mem_regions[i].mem_size); +} + +/* Create enclave vCPU. */ +int ne_create_vcpu(int enclave_fd, struct ne_vcpu *ne_vcpu) +{ + if (enclave_fd < 0) + return -EINVAL; + + if (!ne_vcpu) + return -EINVAL; + + ne_vcpu->vcpu_fd = ioctl(enclave_fd, KVM_CREATE_VCPU, + &ne_vcpu->vcpu_id); + if (ne_vcpu->vcpu_fd < 0) { + printf("Error in create vcpu [rc=%d]\n", ne_vcpu->vcpu_fd); + + return ne_vcpu->vcpu_fd; + } + + return 0; +} + +/* Release enclave vCPU fd(s). */ +void ne_release_vcpus(struct ne_vcpu ne_vcpus[]) +{ + unsigned int i = 0; + + for (i = 0; i < DEFAULT_NR_VCPUS; i++) + if (ne_vcpus[i].vcpu_fd > 0) + close(ne_vcpus[i].vcpu_fd); +} + +int main(int argc, char *argv[]) +{ + int enclave_fd = 0; + char enclave_image_path[PATH_MAX] = {}; + unsigned int i = 0; + int ne_dev_fd = 0; + struct ne_mem_region ne_mem_regions[DEFAULT_NR_MEM_REGIONS] = {}; + struct enclave_start_metadata ne_start_metadata = {}; + struct ne_vcpu ne_vcpus[DEFAULT_NR_VCPUS] = {}; + int rc = 0; + pthread_t thread_id = 0; + unsigned long type = 0; + + if (argc != 2) { + printf("Usage: %s \n", argv[0]); + + exit(EXIT_FAILURE); + } + + strncpy(enclave_image_path, argv[1], sizeof(enclave_image_path) - 1); + + ne_dev_fd = open(NE_DEV_NAME, O_RDWR | O_CLOEXEC); + if (ne_dev_fd < 0) { + printf("Error in open NE device [rc=%d]\n", ne_dev_fd); + + exit(EXIT_FAILURE); + } + + printf("Creating enclave slot ...\n"); + + enclave_fd = ioctl(ne_dev_fd, KVM_CREATE_VM, &type); + + close(ne_dev_fd); + + if (enclave_fd < 0) { + printf("Error in create enclave slot [rc=%d]\n", enclave_fd); + + exit(EXIT_FAILURE); + } + + printf("Enclave fd %d\n", enclave_fd); + + rc = pthread_create(&thread_id, NULL, ne_poll_enclave_fd, + (void *)&enclave_fd); + if (rc < 0) { + printf("Error in thread create [rc=%d]\n", rc); + + close(enclave_fd); + + exit(EXIT_FAILURE); + } + + for (i = 0; i < DEFAULT_NR_MEM_REGIONS; i++) { + ne_mem_regions[i].mem_size = MIN_MEM_REGION_SIZE; + rc = ne_alloc_mem_region(&ne_mem_regions[i]); + if (rc < 0) { + printf("Error in alloc mem region, iter %d [rc=%d]\n", + i, rc); + + goto release_enclave_fd; + } + } + + rc = ne_load_enclave_image(enclave_fd, ne_mem_regions, + enclave_image_path); + if (rc < 0) { + printf("Error in load enclave image [rc=%d]\n", rc); + + goto release_enclave_fd; + } + + for (i = 0; i < DEFAULT_NR_MEM_REGIONS; i++) { + rc = ne_set_mem_region(enclave_fd, ne_mem_regions[i]); + if (rc < 0) { + printf("Error in set mem region, iter %d [rc=%d]\n", + i, rc); + + goto release_enclave_fd; + } + } + + printf("Enclave memory regions were added\n"); + + for (i = 0; i < DEFAULT_NR_VCPUS; i++) { + /* + * The vCPU is chosen from the enclave vCPU pool, this value is + * not used for now. + */ + ne_vcpus[i].vcpu_id = i; + rc = ne_create_vcpu(enclave_fd, &ne_vcpus[i]); + if (rc < 0) { + printf("Error in create vcpu, iter %d [rc=%d]\n", + i, rc); + + goto release_enclave_vcpu_fds; + } + } + + printf("Enclave vCPUs were created\n"); + + rc = ioctl(enclave_fd, NE_START_ENCLAVE, &ne_start_metadata); + if (rc < 0) { + printf("Error in start enclave [rc=%d]\n", rc); + + goto release_enclave_vcpu_fds; + } + + printf("Enclave started, CID %llu\n", ne_start_metadata.enclave_cid); + + /* + * TODO: Check for enclave hearbeat after it has started to see if it + * has booted. + */ + + printf("Entering sleep for %d seconds ...\n", SLEEP_TIME); + + sleep(SLEEP_TIME); + + ne_release_vcpus(ne_vcpus); + + close(enclave_fd); + + ne_free_mem_regions(ne_mem_regions); + + exit(EXIT_SUCCESS); + +release_enclave_vcpu_fds: + ne_release_vcpus(ne_vcpus); +release_enclave_fd: + close(enclave_fd); + ne_free_mem_regions(ne_mem_regions); + + exit(EXIT_FAILURE); +} From patchwork Fri May 22 06:29:45 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paraschiv, Andra-Irina" X-Patchwork-Id: 11564759 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C1E95138A for ; Fri, 22 May 2020 06:32:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A5ED1207BC for ; Fri, 22 May 2020 06:32:56 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="LSsIGhBx" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728811AbgEVGcw (ORCPT ); Fri, 22 May 2020 02:32:52 -0400 Received: from smtp-fw-9102.amazon.com ([207.171.184.29]:41619 "EHLO smtp-fw-9102.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728232AbgEVGcw (ORCPT ); Fri, 22 May 2020 02:32:52 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1590129171; x=1621665171; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Q2uQ4KONMuRjyhlvAaG5QSJTMRYNPu4HnVm7hH7LvkA=; b=LSsIGhBx76m7eNX0r5r9l04KJi20M6+g5Lfocz+qjd7mIiSpVvBcQ5Q2 GnxSOoSNEq6m8x2scTUVWfOWLa/51eEdSURq2gxf2B+cRAbfcjQIVVUg5 N5jMe844B34y3aUfaLniHF7mwVvxRBWke9h83oDh0hPoLOylPShOx24F0 I=; IronPort-SDR: wUxGTDuEzdIPxsYGPggLv9Uk322dLEDlIZN9vSgVlYZEQkrwXfMZuMIINoaTv5W0jJHvD2r8s7 c+w9+5pfhApw== X-IronPort-AV: E=Sophos;i="5.73,420,1583193600"; d="scan'208";a="45244417" Received: from sea32-co-svc-lb4-vlan3.sea.corp.amazon.com (HELO email-inbound-relay-1d-98acfc19.us-east-1.amazon.com) ([10.47.23.38]) by smtp-border-fw-out-9102.sea19.amazon.com with ESMTP; 22 May 2020 06:32:50 +0000 Received: from EX13MTAUEA002.ant.amazon.com (iad55-ws-svc-p15-lb9-vlan3.iad.amazon.com [10.40.159.166]) by email-inbound-relay-1d-98acfc19.us-east-1.amazon.com (Postfix) with ESMTPS id 10EB2A236E; Fri, 22 May 2020 06:32:48 +0000 (UTC) Received: from EX13D16EUB003.ant.amazon.com (10.43.166.99) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Fri, 22 May 2020 06:32:47 +0000 Received: from 38f9d34ed3b1.ant.amazon.com (10.43.161.175) by EX13D16EUB003.ant.amazon.com (10.43.166.99) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Fri, 22 May 2020 06:32:38 +0000 From: Andra Paraschiv To: CC: Anthony Liguori , Benjamin Herrenschmidt , Colm MacCarthaigh , "Bjoern Doebel" , David Woodhouse , "Frank van der Linden" , Alexander Graf , "Martin Pohlack" , Matt Wilson , Paolo Bonzini , Balbir Singh , Stefano Garzarella , Stefan Hajnoczi , Stewart Smith , Uwe Dannowski , , , Andra Paraschiv Subject: [PATCH v2 17/18] nitro_enclaves: Add overview documentation Date: Fri, 22 May 2020 09:29:45 +0300 Message-ID: <20200522062946.28973-18-andraprs@amazon.com> X-Mailer: git-send-email 2.20.1 (Apple Git-117) In-Reply-To: <20200522062946.28973-1-andraprs@amazon.com> References: <20200522062946.28973-1-andraprs@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.161.175] X-ClientProxiedBy: EX13D24UWA003.ant.amazon.com (10.43.160.195) To EX13D16EUB003.ant.amazon.com (10.43.166.99) Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Signed-off-by: Andra Paraschiv --- Documentation/nitro_enclaves/ne_overview.txt | 86 ++++++++++++++++++++ 1 file changed, 86 insertions(+) create mode 100644 Documentation/nitro_enclaves/ne_overview.txt diff --git a/Documentation/nitro_enclaves/ne_overview.txt b/Documentation/nitro_enclaves/ne_overview.txt new file mode 100644 index 000000000000..be8bb3d84132 --- /dev/null +++ b/Documentation/nitro_enclaves/ne_overview.txt @@ -0,0 +1,86 @@ +Nitro Enclaves +============== + +Nitro Enclaves (NE) is a new Amazon Elastic Compute Cloud (EC2) capability +that allows customers to carve out isolated compute environments within EC2 +instances [1]. + +For example, an application that processes sensitive data and runs in a VM, +can be separated from other applications running in the same VM. This +application then runs in a separate VM than the primary VM, namely an enclave. + +An enclave runs alongside the VM that spawned it. This setup matches low latency +applications needs. The resources that are allocated for the enclave, such as +memory and CPU, are carved out of the primary VM. Each enclave is mapped to a +process running in the primary VM, that communicates with the NE driver via an +ioctl interface. + +In this sense, there are two components: + +1. An enclave abstraction process - a user space process running in the primary +VM guest that uses the provided ioctl interface of the NE driver to spawn an +enclave VM (that's 2 below). + +How does all gets to an enclave VM running on the host? + +There is a NE emulated PCI device exposed to the primary VM. The driver for this +new PCI device is included in the NE driver. + +The ioctl logic is mapped to PCI device commands e.g. the NE_START_ENCLAVE ioctl +maps to an enclave start PCI command or the KVM_SET_USER_MEMORY_REGION maps to +an add memory PCI command. The PCI device commands are then translated into +actions taken on the hypervisor side; that's the Nitro hypervisor running on the +host where the primary VM is running. The Nitro hypervisor is based on core KVM +technology. + +2. The enclave itself - a VM running on the same host as the primary VM that +spawned it. Memory and CPUs are carved out of the primary VM and are dedicated +for the enclave VM. An enclave does not have persistent storage attached. + +An enclave communicates with the primary VM via a local communication channel, +using virtio-vsock [2]. The primary VM has virtio-pci vsock emulated device, +while the enclave VM has a virtio-mmio vsock emulated device. The vsock device +uses eventfd for signaling. The enclave VM sees the usual interfaces - local +APIC and IOAPIC - to get interrupts from virtio-vsock device. The virtio-mmio +device is placed in memory below the typical 4 GiB. + +The application that runs in the enclave needs to be packaged in an enclave +image together with the OS ( e.g. kernel, ramdisk, init ) that will run in the +enclave VM. The enclave VM has its own kernel and follows the standard Linux +boot protocol. + +The kernel bzImage, the kernel command line, the ramdisk(s) are part of the +Enclave Image Format (EIF); plus an EIF header including metadata such as magic +number, eif version, image size and CRC. + +Hash values are computed for the entire enclave image (EIF), the kernel and +ramdisk(s). That's used, for example, to check that the enclave image that is +loaded in the enclave VM is the one that was intended to be run. + +These crypto measurements are included in a signed attestation document +generated by the Nitro Hypervisor and further used to prove the identity of the +enclave; KMS is an example of service that NE is integrated with and that checks +the attestation doc. + +The enclave image (EIF) is loaded in the enclave memory at offset 8 MiB. The +init process in the enclave connects to the vsock CID of the primary VM and a +predefined port - 9000 - to send a heartbeat value - 0xb7. This mechanism is +used to check in the primary VM that the enclave has booted. + +If the enclave VM crashes or gracefully exits, an interrupt event is received by +the NE driver. This event is sent further to the user space enclave process +running in the primary VM via a poll notification mechanism. Then the user space +enclave process can exit. + +The NE driver for enclave lifetime management provides an ioctl interface to the +user space. It includes the NE PCI device driver that is the means of +communication with the hypervisor running on the host where the primary VM and +the enclave are launched. + +The proposed solution is following the KVM model and uses KVM ioctls to be able +to create and set resources for enclaves. Additional NE ioctl commands, besides +the ones provided by KVM, are used to start an enclave and get memory offset for +in-memory enclave image loading. + +[1] https://aws.amazon.com/ec2/nitro/nitro-enclaves/ +[2] http://man7.org/linux/man-pages/man7/vsock.7.html From patchwork Fri May 22 06:29:46 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paraschiv, Andra-Irina" X-Patchwork-Id: 11564761 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D9BBD14B7 for ; Fri, 22 May 2020 06:33:03 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C34D220825 for ; Fri, 22 May 2020 06:33:03 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="h3CAYiHH" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728864AbgEVGc7 (ORCPT ); Fri, 22 May 2020 02:32:59 -0400 Received: from smtp-fw-2101.amazon.com ([72.21.196.25]:20918 "EHLO smtp-fw-2101.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728209AbgEVGc7 (ORCPT ); Fri, 22 May 2020 02:32:59 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1590129179; x=1621665179; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Sf33lwe+INgQJ3nfbsCYbHw2Wj4q8ImFKELp3Gjh7hY=; b=h3CAYiHHmaNscsm1GWjtMtqtk11Rn6g3l8xqNsR6IaEv8HTeo/lDkpOH KLDH7Z3QpB9sdPR9m7TtoMUz3X0FPoWHNnHdcLPt4jmEXiNNn+o6/47ux CnzfD7Z+xudGzgif+V9IvYslsDe+t2H0tsq5fflQ5YCoJ/vOjKBX4NnB3 4=; IronPort-SDR: jIgftmOx8xzurRLAyb1BYUHcvImfS/rto7jUmpbYgEsG9uCTRY5bWzE6bnoysFpHSp3mHouHvh ShEx/EZJJNwQ== X-IronPort-AV: E=Sophos;i="5.73,420,1583193600"; d="scan'208";a="31775604" Received: from iad12-co-svc-p1-lb1-vlan2.amazon.com (HELO email-inbound-relay-1d-37fd6b3d.us-east-1.amazon.com) ([10.43.8.2]) by smtp-border-fw-out-2101.iad2.amazon.com with ESMTP; 22 May 2020 06:32:58 +0000 Received: from EX13MTAUEA002.ant.amazon.com (iad55-ws-svc-p15-lb9-vlan2.iad.amazon.com [10.40.159.162]) by email-inbound-relay-1d-37fd6b3d.us-east-1.amazon.com (Postfix) with ESMTPS id 80E942831A3; Fri, 22 May 2020 06:32:56 +0000 (UTC) Received: from EX13D16EUB003.ant.amazon.com (10.43.166.99) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Fri, 22 May 2020 06:32:56 +0000 Received: from 38f9d34ed3b1.ant.amazon.com (10.43.161.175) by EX13D16EUB003.ant.amazon.com (10.43.166.99) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Fri, 22 May 2020 06:32:47 +0000 From: Andra Paraschiv To: CC: Anthony Liguori , Benjamin Herrenschmidt , Colm MacCarthaigh , "Bjoern Doebel" , David Woodhouse , "Frank van der Linden" , Alexander Graf , "Martin Pohlack" , Matt Wilson , Paolo Bonzini , Balbir Singh , Stefano Garzarella , Stefan Hajnoczi , Stewart Smith , Uwe Dannowski , , , Andra Paraschiv Subject: [PATCH v2 18/18] MAINTAINERS: Add entry for the Nitro Enclaves driver Date: Fri, 22 May 2020 09:29:46 +0300 Message-ID: <20200522062946.28973-19-andraprs@amazon.com> X-Mailer: git-send-email 2.20.1 (Apple Git-117) In-Reply-To: <20200522062946.28973-1-andraprs@amazon.com> References: <20200522062946.28973-1-andraprs@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.161.175] X-ClientProxiedBy: EX13D24UWA003.ant.amazon.com (10.43.160.195) To EX13D16EUB003.ant.amazon.com (10.43.166.99) Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Signed-off-by: Andra Paraschiv --- MAINTAINERS | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/MAINTAINERS b/MAINTAINERS index ecc0749810b0..69fe37999a9e 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -11956,6 +11956,19 @@ S: Maintained T: git git://git.kernel.org/pub/scm/linux/kernel/git/lftan/nios2.git F: arch/nios2/ +NITRO ENCLAVES (NE) +M: Andra Paraschiv +M: Alexandru Vasile +M: Alexandru Ciobotaru +L: linux-kernel@vger.kernel.org +S: Supported +W: https://aws.amazon.com/ec2/nitro/nitro-enclaves/ +F: include/linux/nitro_enclaves.h +F: include/uapi/linux/nitro_enclaves.h +F: drivers/virt/nitro_enclaves/ +F: samples/nitro_enclaves/ +F: Documentation/nitro_enclaves/ + NOHZ, DYNTICKS SUPPORT M: Frederic Weisbecker M: Thomas Gleixner