From patchwork Tue Apr 21 18:41:36 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paraschiv, Andra-Irina" X-Patchwork-Id: 11502067 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AA8B181 for ; Tue, 21 Apr 2020 18:42:19 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 92F3F20738 for ; Tue, 21 Apr 2020 18:42:19 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="dcJohmdr" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729360AbgDUSmQ (ORCPT ); Tue, 21 Apr 2020 14:42:16 -0400 Received: from smtp-fw-9102.amazon.com ([207.171.184.29]:53386 "EHLO smtp-fw-9102.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725870AbgDUSmQ (ORCPT ); Tue, 21 Apr 2020 14:42:16 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1587494536; x=1619030536; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=p3KGxzu0ADrayANgam9z2ixDAMnSo4ulJX3kwnOktHY=; b=dcJohmdrbRTD833Md9aPsOCI9Cn3fSv0r1rELPiU1Wa1mMsBllLTYi8O I4jsmkAkZJLGYSRD56ArD376BSriALd8PKHLgjTm7OOOvOsOuOZrYn4ro VdZcTM7ocQwfmBxXdF/8OCnx/jPd8ySfMi8gxUMXTkvGfnQCvk06rhLC9 Q=; IronPort-SDR: U2pF+64vnqDEzdek/BksJqqn8twIcJln9O+n26hSW7xkaukg9egpCd3UUBG9xeiMWoziJumiLL +FIvD9qptL+g== X-IronPort-AV: E=Sophos;i="5.72,411,1580774400"; d="scan'208";a="38565615" Received: from sea32-co-svc-lb4-vlan3.sea.corp.amazon.com (HELO email-inbound-relay-2a-538b0bfb.us-west-2.amazon.com) ([10.47.23.38]) by smtp-border-fw-out-9102.sea19.amazon.com with ESMTP; 21 Apr 2020 18:42:14 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan3.pdx.amazon.com [10.170.41.166]) by email-inbound-relay-2a-538b0bfb.us-west-2.amazon.com (Postfix) with ESMTPS id 83459A1DBC; Tue, 21 Apr 2020 18:42:13 +0000 (UTC) Received: from EX13D16EUB003.ant.amazon.com (10.43.166.99) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 21 Apr 2020 18:42:12 +0000 Received: from 38f9d34ed3b1.ant.amazon.com (10.43.160.90) by EX13D16EUB003.ant.amazon.com (10.43.166.99) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 21 Apr 2020 18:42:04 +0000 From: Andra Paraschiv To: CC: Anthony Liguori , Benjamin Herrenschmidt , Colm MacCarthaigh , Bjoern Doebel , David Woodhouse , Frank van der Linden , Alexander Graf , Martin Pohlack , Matt Wilson , Paolo Bonzini , Balbir Singh , Stewart Smith , Uwe Dannowski , , , Andra Paraschiv Subject: [PATCH v1 01/15] nitro_enclaves: Add ioctl interface definition Date: Tue, 21 Apr 2020 21:41:36 +0300 Message-ID: <20200421184150.68011-2-andraprs@amazon.com> X-Mailer: git-send-email 2.20.1 (Apple Git-117) In-Reply-To: <20200421184150.68011-1-andraprs@amazon.com> References: <20200421184150.68011-1-andraprs@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.160.90] X-ClientProxiedBy: EX13D25UWC004.ant.amazon.com (10.43.162.201) To EX13D16EUB003.ant.amazon.com (10.43.166.99) Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The Nitro Enclaves driver handles the enclave lifetime management. This includes enclave creation, termination and setting up its resources such as memory and CPU. An enclave runs alongside the VM that spawned it. It is abstracted as a process running in the VM that launched it. The process interacts with the NE driver, that exposes an ioctl interface for creating an enclave and setting up its resources. Include the KVM API as part of the provided ioctl interface, with an additional ENCLAVE_START ioctl command that triggers the enclave run. Signed-off-by: Alexandru Vasile Signed-off-by: Andra Paraschiv --- include/linux/nitro_enclaves.h | 23 +++++++++++++ include/uapi/linux/nitro_enclaves.h | 52 +++++++++++++++++++++++++++++ 2 files changed, 75 insertions(+) create mode 100644 include/linux/nitro_enclaves.h create mode 100644 include/uapi/linux/nitro_enclaves.h diff --git a/include/linux/nitro_enclaves.h b/include/linux/nitro_enclaves.h new file mode 100644 index 000000000000..7e593a9fbf8c --- /dev/null +++ b/include/linux/nitro_enclaves.h @@ -0,0 +1,23 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms and conditions of the GNU General Public License, + * version 2, as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, see . + */ + +#ifndef _LINUX_NITRO_ENCLAVES_H_ +#define _LINUX_NITRO_ENCLAVES_H_ + +#include + +#endif /* _LINUX_NITRO_ENCLAVES_H_ */ diff --git a/include/uapi/linux/nitro_enclaves.h b/include/uapi/linux/nitro_enclaves.h new file mode 100644 index 000000000000..b90dfcf6253a --- /dev/null +++ b/include/uapi/linux/nitro_enclaves.h @@ -0,0 +1,52 @@ +/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */ +/* + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms and conditions of the GNU General Public License, + * version 2, as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, see . + */ + +#ifndef _UAPI_LINUX_NITRO_ENCLAVES_H_ +#define _UAPI_LINUX_NITRO_ENCLAVES_H_ + +#include +#include + +/* Nitro Enclaves (NE) Kernel Driver Interface */ + +/** + * The command is used to trigger enclave start after the enclave resources, + * such as memory and CPU, have been set. + * + * The enclave start metadata is an in / out data structure. It includes + * provided info by the caller - enclave cid and flags - and returns the + * slot uid and the cid (if input cid is 0). + */ +#define NE_ENCLAVE_START _IOWR('B', 0x1, struct enclave_start_metadata) + +/* Setup metadata necessary for enclave start. */ +struct enclave_start_metadata { + /* Flags for the enclave to start with (e.g. debug mode) (in). */ + __u64 flags; + + /** + * Context ID (CID) for the enclave vsock device. If 0 as input, the + * CID is autogenerated by the hypervisor and returned back as output + * by the driver (in/out). + */ + __u64 enclave_cid; + + /* Slot unique id mapped to the enclave to start (out). */ + __u64 slot_uid; +}; + +#endif /* _UAPI_LINUX_NITRO_ENCLAVES_H_ */ From patchwork Tue Apr 21 18:41:37 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paraschiv, Andra-Irina" X-Patchwork-Id: 11502071 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B6A4581 for ; Tue, 21 Apr 2020 18:42:29 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 91C25206D9 for ; Tue, 21 Apr 2020 18:42:29 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="Y303xp9J" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729462AbgDUSm0 (ORCPT ); Tue, 21 Apr 2020 14:42:26 -0400 Received: from smtp-fw-33001.amazon.com ([207.171.190.10]:62215 "EHLO smtp-fw-33001.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729380AbgDUSmY (ORCPT ); Tue, 21 Apr 2020 14:42:24 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1587494543; x=1619030543; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Hcr8IWCBd9vxlMeEDzpaZbC8VQKGocQ+6X2HB/FejJk=; b=Y303xp9Jjbq4gXs0eJFlEwp7olLJ2Z9ny36/Nu8P0AWORfHjQsyC13fz Dw3/nlJmAOBYJ/RG+jNBGo2RqJg/E5xBbTiK/qsLi9Gzp5D2bFUuxSCXC oqdUbYe4S7I1AZYaE4UhCXbD+BupkyIy4sVqubZa4dj2ZogwCyGqbfm0o 8=; IronPort-SDR: Z8UgWe5qohhMnkdOd8WQ/IQCIT9yo2+w8oOvkssHSCxGa79il6QccproX3tanx6htZ/TDJUbin GJHqpp01NmVQ== X-IronPort-AV: E=Sophos;i="5.72,411,1580774400"; d="scan'208";a="39978365" Received: from sea32-co-svc-lb4-vlan3.sea.corp.amazon.com (HELO email-inbound-relay-2a-538b0bfb.us-west-2.amazon.com) ([10.47.23.38]) by smtp-border-fw-out-33001.sea14.amazon.com with ESMTP; 21 Apr 2020 18:42:22 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan3.pdx.amazon.com [10.170.41.166]) by email-inbound-relay-2a-538b0bfb.us-west-2.amazon.com (Postfix) with ESMTPS id 3044FA1DE0; Tue, 21 Apr 2020 18:42:21 +0000 (UTC) Received: from EX13D16EUB003.ant.amazon.com (10.43.166.99) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 21 Apr 2020 18:42:20 +0000 Received: from 38f9d34ed3b1.ant.amazon.com (10.43.160.90) by EX13D16EUB003.ant.amazon.com (10.43.166.99) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 21 Apr 2020 18:42:12 +0000 From: Andra Paraschiv To: CC: Anthony Liguori , Benjamin Herrenschmidt , Colm MacCarthaigh , Bjoern Doebel , David Woodhouse , Frank van der Linden , Alexander Graf , Martin Pohlack , Matt Wilson , Paolo Bonzini , Balbir Singh , Stewart Smith , Uwe Dannowski , , , Andra Paraschiv Subject: [PATCH v1 02/15] nitro_enclaves: Define the PCI device interface Date: Tue, 21 Apr 2020 21:41:37 +0300 Message-ID: <20200421184150.68011-3-andraprs@amazon.com> X-Mailer: git-send-email 2.20.1 (Apple Git-117) In-Reply-To: <20200421184150.68011-1-andraprs@amazon.com> References: <20200421184150.68011-1-andraprs@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.160.90] X-ClientProxiedBy: EX13D25UWC004.ant.amazon.com (10.43.162.201) To EX13D16EUB003.ant.amazon.com (10.43.166.99) Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The Nitro Enclaves (NE) driver communicates with a new PCI device, that is exposed to a virtual machine (VM) and handles commands meant for handling enclaves lifetime e.g. creation, termination, setting memory regions. The communication with the PCI device is handled using a MMIO space and MSI-X interrupts. This device communicates with the hypervisor on the host, where the VM that spawned the enclave itself run, e.g. to launch a VM that is used for the enclave. Define the MMIO space of the PCI device, the commands that are provided by this device. Add an internal data structure used as private data for the PCI device driver and the functions for the PCI device init / uninit and command requests handling. Signed-off-by: Alexandru-Catalin Vasile Signed-off-by: Alexandru Ciobotaru Signed-off-by: Andra Paraschiv --- .../virt/amazon/nitro_enclaves/ne_pci_dev.h | 266 ++++++++++++++++++ 1 file changed, 266 insertions(+) create mode 100644 drivers/virt/amazon/nitro_enclaves/ne_pci_dev.h diff --git a/drivers/virt/amazon/nitro_enclaves/ne_pci_dev.h b/drivers/virt/amazon/nitro_enclaves/ne_pci_dev.h new file mode 100644 index 000000000000..e703419ed29d --- /dev/null +++ b/drivers/virt/amazon/nitro_enclaves/ne_pci_dev.h @@ -0,0 +1,266 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms and conditions of the GNU General Public License, + * version 2, as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, see . + */ + +#ifndef _NE_PCI_DEV_H_ +#define _NE_PCI_DEV_H_ + +#include +#include +#include +#include +#include +#include + +/* Nitro Enclaves (NE) PCI device identifier */ + +#define PCI_DEVICE_ID_NE (0xe4c1) +#define PCI_BAR_NE (0x03) + +/* Device registers */ + +/** + * (1 byte) Register to notify the device that the driver is using it + * (Read/Write). + */ +#define NE_ENABLE (0x0000) +#define NE_ENABLE_ON (0x00) +#define NE_ENABLE_OFF (0x01) + +/* (2 bytes) Register to select the device run-time version (Read/Write). */ +#define NE_VERSION (0x0002) +#define NE_VERSION_MAX (0x0001) + +/** + * (4 bytes) Register to notify the device what command was requested + * (Write-Only). + */ +#define NE_COMMAND (0x0004) + +/** + * (4 bytes) Register to notify the driver that a reply or a device event + * is available (Read-Only): + * - Lower half - command reply counter + * - Higher half - out-of-band device event counter + */ +#define NE_EVTCNT (0x000c) +#define NE_EVTCNT_REPLY_SHIFT (0) +#define NE_EVTCNT_REPLY_MASK (0x0000ffff) +#define NE_EVTCNT_REPLY(cnt) (((cnt) & NE_EVTCNT_REPLY_MASK) >> \ + NE_EVTCNT_REPLY_SHIFT) +#define NE_EVTCNT_EVENT_SHIFT (16) +#define NE_EVTCNT_EVENT_MASK (0xffff0000) +#define NE_EVTCNT_EVENT(cnt) (((cnt) & NE_EVTCNT_EVENT_MASK) >> \ + NE_EVTCNT_EVENT_SHIFT) + +/* (240 bytes) Buffer for sending the command request payload (Read/Write). */ +#define NE_SEND_DATA (0x0010) + +/* (240 bytes) Buffer for receiving the command reply payload (Read-Only). */ +#define NE_RECV_DATA (0x0100) + +/* Device MMIO buffer sizes */ + +/* 240 bytes for send / recv buffer. */ +#define NE_SEND_DATA_SIZE (240) +#define NE_RECV_DATA_SIZE (240) + +/* MSI-X interrupt vectors */ + +/* MSI-X vector used for command reply notification. */ +#define NE_VEC_REPLY (0) + +/* MSI-X vector used for out-of-band events e.g. enclave crash. */ +#define NE_VEC_EVENT (1) + +/* Device command types. */ +enum ne_pci_dev_cmd_type { + INVALID_CMD = 0, + ENCLAVE_START = 1, + ENCLAVE_GET_SLOT = 2, + ENCLAVE_STOP = 3, + SLOT_ALLOC = 4, + SLOT_FREE = 5, + SLOT_ADD_MEM = 6, + SLOT_ADD_VCPU = 7, + SLOT_COUNT = 8, + NEXT_SLOT = 9, + SLOT_INFO = 10, + SLOT_ADD_BULK_VCPUS = 11, + MAX_CMD, +}; + +/* Device commands - payload structure for requests and replies. */ + +struct enclave_start_req { + /* Slot unique id mapped to the enclave to start. */ + u64 slot_uid; + + /** + * Context ID (CID) for the enclave vsock device. + * If 0, CID is autogenerated. + */ + u64 enclave_cid; + + /* Flags for the enclave to start with (e.g. debug mode). */ + u64 flags; +} __attribute__ ((__packed__)); + +struct enclave_get_slot_req { + /* Context ID (CID) for the enclave vsock device. */ + u64 enclave_cid; +} __attribute__ ((__packed__)); + +struct enclave_stop_req { + /* Slot unique id mapped to the enclave to stop. */ + u64 slot_uid; +} __attribute__ ((__packed__)); + +struct slot_alloc_req { + /* In order to avoid weird sizeof edge cases. */ + u8 unused; +} __attribute__ ((__packed__)); + +struct slot_free_req { + /* Slot unique id mapped to the slot to free. */ + u64 slot_uid; +} __attribute__ ((__packed__)); + +struct slot_add_mem_req { + /* Slot unique id mapped to the slot to add the memory region to. */ + u64 slot_uid; + + /* Physical address of the memory region to add to the slot. */ + u64 paddr; + + /* Memory size, in bytes, of the memory region to add to the slot. */ + u64 size; +} __attribute__ ((__packed__)); + +struct slot_add_vcpu_req { + /* Slot unique id mapped to the slot to add the vCPU to. */ + u64 slot_uid; + + /* vCPU ID of the CPU to add to the enclave. */ + u32 vcpu_id; +} __attribute__ ((__packed__)); + +struct slot_count_req { + /* In order to avoid weird sizeof edge cases. */ + u8 unused; +} __attribute__ ((__packed__)); + +struct next_slot_req { + /* Slot unique id of the next slot in the iteration. */ + u64 slot_uid; +} __attribute__ ((__packed__)); + +struct slot_info_req { + /* Slot unique id mapped to the slot to get information about. */ + u64 slot_uid; +} __attribute__ ((__packed__)); + +struct slot_add_bulk_vcpus_req { + /* Slot unique id mapped to the slot to add vCPUs to. */ + u64 slot_uid; + + /* Number of vCPUs to add to the slot. */ + u64 nr_vcpus; +} __attribute__ ((__packed__)); + +struct ne_pci_dev_cmd_reply { + s32 rc; + + /* Valid for all commands except SLOT_COUNT. */ + u64 slot_uid; + + /* Valid for ENCLAVE_START command. */ + u64 enclave_cid; + + /* Valid for SLOT_COUNT command. */ + u64 slot_count; + + /* Valid for SLOT_ALLOC and SLOT_INFO commands. */ + u64 mem_regions; + + /* Valid for SLOT_INFO command. */ + u64 mem_size; + + /* Valid for SLOT_INFO command. */ + u64 nr_vcpus; + + /* Valid for SLOT_INFO command. */ + u64 flags; + + /* Valid for SLOT_INFO command. */ + u16 state; +} __attribute__ ((__packed__)); + +/* Nitro Enclaves (NE) PCI device. */ +struct ne_pci_dev { + /* Variable set if a reply has been sent by the PCI device. */ + atomic_t cmd_reply_avail; + + /* Wait queue for handling command reply from the PCI device. */ + wait_queue_head_t cmd_reply_wait_q; + + /* List of the enclaves managed by the PCI device. */ + struct list_head enclaves_list; + + /* Mutex for accessing the list of enclaves. */ + struct mutex enclaves_list_mutex; + + /** + * Work queue for handling out-of-band events triggered by the Nitro + * Hypervisor which require enclave state scanning and propagation to + * the enclave process. + */ + struct workqueue_struct *event_wq; + + /* MMIO region of the PCI device. */ + void __iomem *iomem_base; + + /* Work item for every received out-of-band event. */ + struct work_struct notify_work; + + /* Mutex for accessing the PCI dev MMIO space. */ + struct mutex pci_dev_mutex; +}; + +/** + * ne_do_request - Submit command request to the PCI device based on the command + * type and retrieve the associated reply. + * + * This function uses the ne_pci_dev mutex to handle one command at a time. + * + * @pdev: PCI device to send the command to and receive the reply from. + * @cmd_type: command type of the request sent to the PCI device. + * @cmd_request: command request payload. + * @cmd_request_size: size of the command request payload. + * @cmd_reply: command reply payload. + * @cmd_reply_size: size of the command reply payload. + * + * @returns: 0 on success, negative return value on failure. + */ +int ne_do_request(struct pci_dev *pdev, enum ne_pci_dev_cmd_type cmd_type, + void *cmd_request, size_t cmd_request_size, + struct ne_pci_dev_cmd_reply *cmd_reply, + size_t cmd_reply_size); + +/* Nitro Enclaves (NE) PCI device driver */ +extern struct pci_driver ne_pci_driver; + +#endif /* _NE_PCI_DEV_H_ */ From patchwork Tue Apr 21 18:41:38 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paraschiv, Andra-Irina" X-Patchwork-Id: 11502073 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 88F5281 for ; Tue, 21 Apr 2020 18:42:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 6B50C2072D for ; Tue, 21 Apr 2020 18:42:37 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="FQ1xIRVP" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729480AbgDUSme (ORCPT ); Tue, 21 Apr 2020 14:42:34 -0400 Received: from smtp-fw-9101.amazon.com ([207.171.184.25]:64501 "EHLO smtp-fw-9101.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729457AbgDUSmd (ORCPT ); Tue, 21 Apr 2020 14:42:33 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1587494552; x=1619030552; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=JvAkbGXE1TkqLW/VIeJ1UyQVKOVRurbP5B+dKLPBzQM=; b=FQ1xIRVPrambeWWZ8V9Px9igxHKBcFcQJxpnIjzDYYy1BG2G9XZGmYD5 DOPsUDCAjLocLYrzhwdVWk1gz/8ttFOdZkp+d7mgov3id03hszbl9TJ4E grSZGe6QCQZAZRncJchBahvjx6Qis5UDPhGqgd6UW+NxjaygfxL5w14sc o=; IronPort-SDR: iTDTXD0VqbwQEv8BZHVf78cDcxHnyu4DA5JEEZvZhTMvucS95gF47zQxlGu8JiElk0mequShib N40L7bUh51kQ== X-IronPort-AV: E=Sophos;i="5.72,411,1580774400"; d="scan'208";a="30232759" Received: from sea32-co-svc-lb4-vlan3.sea.corp.amazon.com (HELO email-inbound-relay-2a-e7be2041.us-west-2.amazon.com) ([10.47.23.38]) by smtp-border-fw-out-9101.sea19.amazon.com with ESMTP; 21 Apr 2020 18:42:31 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan2.pdx.amazon.com [10.170.41.162]) by email-inbound-relay-2a-e7be2041.us-west-2.amazon.com (Postfix) with ESMTPS id DC015A22F5; Tue, 21 Apr 2020 18:42:29 +0000 (UTC) Received: from EX13D16EUB003.ant.amazon.com (10.43.166.99) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 21 Apr 2020 18:42:29 +0000 Received: from 38f9d34ed3b1.ant.amazon.com (10.43.160.90) by EX13D16EUB003.ant.amazon.com (10.43.166.99) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 21 Apr 2020 18:42:20 +0000 From: Andra Paraschiv To: CC: Anthony Liguori , Benjamin Herrenschmidt , Colm MacCarthaigh , Bjoern Doebel , David Woodhouse , Frank van der Linden , Alexander Graf , Martin Pohlack , Matt Wilson , Paolo Bonzini , Balbir Singh , Stewart Smith , Uwe Dannowski , , , Andra Paraschiv Subject: [PATCH v1 03/15] nitro_enclaves: Define enclave info for internal bookkeeping Date: Tue, 21 Apr 2020 21:41:38 +0300 Message-ID: <20200421184150.68011-4-andraprs@amazon.com> X-Mailer: git-send-email 2.20.1 (Apple Git-117) In-Reply-To: <20200421184150.68011-1-andraprs@amazon.com> References: <20200421184150.68011-1-andraprs@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.160.90] X-ClientProxiedBy: EX13D25UWC004.ant.amazon.com (10.43.162.201) To EX13D16EUB003.ant.amazon.com (10.43.166.99) Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The Nitro Enclaves driver keeps an internal info per each enclave. This is needed to be able to manage enclave resources state, enclave notifications and have a reference of the PCI device that handles command requests for enclave lifetime management. Signed-off-by: Alexandru-Catalin Vasile Signed-off-by: Andra Paraschiv --- .../virt/amazon/nitro_enclaves/ne_misc_dev.h | 120 ++++++++++++++++++ 1 file changed, 120 insertions(+) create mode 100644 drivers/virt/amazon/nitro_enclaves/ne_misc_dev.h diff --git a/drivers/virt/amazon/nitro_enclaves/ne_misc_dev.h b/drivers/virt/amazon/nitro_enclaves/ne_misc_dev.h new file mode 100644 index 000000000000..dece3ead86b9 --- /dev/null +++ b/drivers/virt/amazon/nitro_enclaves/ne_misc_dev.h @@ -0,0 +1,120 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms and conditions of the GNU General Public License, + * version 2, as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, see . + */ + +#ifndef _NE_MISC_DEV_H_ +#define _NE_MISC_DEV_H_ + +#include +#include +#include +#include +#include +#include +#include + +/* Entry in vCPU IDs list. */ +struct ne_vcpu_id { + /* CPU id associated with a given slot, apic id on x86. */ + u32 vcpu_id; + + struct list_head vcpu_id_list_entry; +}; + +/* Entry in memory regions list. */ +struct ne_mem_region { + struct list_head mem_region_list_entry; + + /* Number of pages that make up the memory region. */ + unsigned long nr_pages; + + /* Pages that make up the user space memory region. */ + struct page **pages; +}; + +/* Per-enclave data used for enclave lifetime management. */ +struct ne_enclave { + /** + * CPU pool with siblings of already allocated CPUs to an enclave. + * This is used when a CPU pool is set, to be able to know the CPU + * siblings for the hyperthreading (HT) setup. + */ + cpumask_var_t cpu_siblings; + + struct list_head enclave_list_entry; + + /* Mutex for accessing this internal state. */ + struct mutex enclave_info_mutex; + + /** + * Wait queue used for out-of-band event notifications + * triggered from the PCI device event handler to the enclave + * process via the poll function. + */ + wait_queue_head_t eventq; + + /* Variable used to determine if the out-of-band event was triggered. */ + bool has_event; + + /** + * The maximum number of memory regions that can be handled by the + * lower levels. + */ + u64 max_mem_regions; + + /* Enclave memory regions list. */ + struct list_head mem_regions_list; + + /* Enclave process abstraction mm data struct. */ + struct mm_struct *mm; + + /* PCI device used for enclave lifetime management. */ + struct pci_dev *pdev; + + /* Slot unique id mapped to the enclave. */ + u64 slot_uid; + + /* Enclave state, updated during enclave lifetime. */ + u16 state; + + /* Enclave vCPUs list. */ + struct list_head vcpu_ids_list; +}; + +/** + * States available for an enclave. + * + * TODO: Determine if the following states are exposing enough information + * to the kernel driver. + */ +enum ne_state { + /* NE_ENCLAVE_START ioctl was never issued for the enclave. */ + NE_STATE_INIT = 0, + + /** + * NE_ENCLAVE_START ioctl was issued and the enclave is running + * as expected. + */ + NE_STATE_RUNNING = 2, + + /* Enclave exited without userspace interaction. */ + NE_STATE_STOPPED = U16_MAX, +}; + +/* Nitro Enclaves (NE) misc device */ +extern struct miscdevice ne_miscdevice; + +#endif /* _NE_MISC_DEV_H_ */ From patchwork Tue Apr 21 18:41:39 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paraschiv, Andra-Irina" X-Patchwork-Id: 11502075 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8C35E92C for ; Tue, 21 Apr 2020 18:42:45 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 680BE20724 for ; Tue, 21 Apr 2020 18:42:45 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="kLd2F00O" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729497AbgDUSmm (ORCPT ); Tue, 21 Apr 2020 14:42:42 -0400 Received: from smtp-fw-9102.amazon.com ([207.171.184.29]:53454 "EHLO smtp-fw-9102.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729452AbgDUSmk (ORCPT ); Tue, 21 Apr 2020 14:42:40 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1587494559; x=1619030559; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=aMA7GuRQk9xRyNRDUSC1CF1PbdsE/6li9LlY6g7pMyo=; b=kLd2F00OkmG3kfJyj6m1+3dYrRJ1i7dbhver5u3+1hlPrBajKHq0+kfG pfyX4wN+yUdk1zksPU0+9c0/PioB8lKuetMwxVv4bMc3zirPEz3ufBrEl uveJuFYLTQ1vCe2xCMTvuCPqLk5X1m7kmLRbW62IOpbvE2qU1yoQOTAI1 Q=; IronPort-SDR: QqqpYh2L/trLOag45i9lRLGDT6tMUArdG6WL8Lsrs/LDr5xJ0RWar4I6doUWfh3xjW+GiSwVd2 phx0d9bE3bHw== X-IronPort-AV: E=Sophos;i="5.72,411,1580774400"; d="scan'208";a="38565694" Received: from sea32-co-svc-lb4-vlan3.sea.corp.amazon.com (HELO email-inbound-relay-2c-1968f9fa.us-west-2.amazon.com) ([10.47.23.38]) by smtp-border-fw-out-9102.sea19.amazon.com with ESMTP; 21 Apr 2020 18:42:39 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan3.pdx.amazon.com [10.170.41.166]) by email-inbound-relay-2c-1968f9fa.us-west-2.amazon.com (Postfix) with ESMTPS id 0E67CA070F; Tue, 21 Apr 2020 18:42:37 +0000 (UTC) Received: from EX13D16EUB003.ant.amazon.com (10.43.166.99) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 21 Apr 2020 18:42:37 +0000 Received: from 38f9d34ed3b1.ant.amazon.com (10.43.160.90) by EX13D16EUB003.ant.amazon.com (10.43.166.99) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 21 Apr 2020 18:42:28 +0000 From: Andra Paraschiv To: CC: Anthony Liguori , Benjamin Herrenschmidt , Colm MacCarthaigh , Bjoern Doebel , David Woodhouse , Frank van der Linden , Alexander Graf , Martin Pohlack , Matt Wilson , Paolo Bonzini , Balbir Singh , Stewart Smith , Uwe Dannowski , , , Andra Paraschiv Subject: [PATCH v1 04/15] nitro_enclaves: Init PCI device driver Date: Tue, 21 Apr 2020 21:41:39 +0300 Message-ID: <20200421184150.68011-5-andraprs@amazon.com> X-Mailer: git-send-email 2.20.1 (Apple Git-117) In-Reply-To: <20200421184150.68011-1-andraprs@amazon.com> References: <20200421184150.68011-1-andraprs@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.160.90] X-ClientProxiedBy: EX13D25UWC004.ant.amazon.com (10.43.162.201) To EX13D16EUB003.ant.amazon.com (10.43.166.99) Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The Nitro Enclaves PCI device is used by the kernel driver as a means of communication with the hypervisor on the host where the primary VM and the enclaves run. It handles requests with regard to enclave lifetime. Setup the PCI device driver and add support for MSI-X interrupts. Signed-off-by: Alexandru-Catalin Vasile Signed-off-by: Alexandru Ciobotaru Signed-off-by: Andra Paraschiv --- .../virt/amazon/nitro_enclaves/ne_pci_dev.c | 278 ++++++++++++++++++ 1 file changed, 278 insertions(+) create mode 100644 drivers/virt/amazon/nitro_enclaves/ne_pci_dev.c diff --git a/drivers/virt/amazon/nitro_enclaves/ne_pci_dev.c b/drivers/virt/amazon/nitro_enclaves/ne_pci_dev.c new file mode 100644 index 000000000000..8fbee95ea291 --- /dev/null +++ b/drivers/virt/amazon/nitro_enclaves/ne_pci_dev.c @@ -0,0 +1,278 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms and conditions of the GNU General Public License, + * version 2, as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, see . + */ + +/* Nitro Enclaves (NE) PCI device driver. */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "ne_misc_dev.h" +#include "ne_pci_dev.h" + +#define DEFAULT_TIMEOUT_MSECS (120000) // 120 sec + +static const struct pci_device_id ne_pci_ids[] = { + { PCI_DEVICE(PCI_VENDOR_ID_AMAZON, PCI_DEVICE_ID_NE) }, + { 0, } +}; + +MODULE_DEVICE_TABLE(pci, ne_pci_ids); + +/** + * ne_setup_msix - Setup MSI-X vectors for the PCI device. + * + * @pdev: PCI device to setup the MSI-X for. + * @ne_pci_dev: PCI device private data structure. + * + * @returns: 0 on success, negative return value on failure. + */ +static int ne_setup_msix(struct pci_dev *pdev, struct ne_pci_dev *ne_pci_dev) +{ + int nr_vecs = 0; + int rc = -EINVAL; + + BUG_ON(!ne_pci_dev); + + nr_vecs = pci_msix_vec_count(pdev); + if (nr_vecs < 0) { + rc = nr_vecs; + + dev_err_ratelimited(&pdev->dev, + "Failure in getting vec count [rc=%d]\n", + rc); + + return rc; + } + + rc = pci_alloc_irq_vectors(pdev, nr_vecs, nr_vecs, PCI_IRQ_MSIX); + if (rc < 0) { + dev_err_ratelimited(&pdev->dev, + "Failure in alloc MSI-X vecs [rc=%d]\n", + rc); + + goto err_alloc_irq_vecs; + } + + return 0; + +err_alloc_irq_vecs: + return rc; +} + +/** + * ne_pci_dev_enable - Select PCI device version and enable it. + * + * @pdev: PCI device to select version for and then enable. + * @ne_pci_dev: PCI device private data structure. + * + * @returns: 0 on success, negative return value on failure. + */ +static int ne_pci_dev_enable(struct pci_dev *pdev, + struct ne_pci_dev *ne_pci_dev) +{ + u8 dev_enable_reply = 0; + u16 dev_version_reply = 0; + + BUG_ON(!pdev); + BUG_ON(!ne_pci_dev); + BUG_ON(!ne_pci_dev->iomem_base); + + iowrite16(NE_VERSION_MAX, ne_pci_dev->iomem_base + NE_VERSION); + + dev_version_reply = ioread16(ne_pci_dev->iomem_base + NE_VERSION); + if (dev_version_reply != NE_VERSION_MAX) { + dev_err_ratelimited(&pdev->dev, + "Failure in pci dev version cmd\n"); + + return -EIO; + } + + iowrite8(NE_ENABLE_ON, ne_pci_dev->iomem_base + NE_ENABLE); + + dev_enable_reply = ioread8(ne_pci_dev->iomem_base + NE_ENABLE); + if (dev_enable_reply != NE_ENABLE_ON) { + dev_err_ratelimited(&pdev->dev, + "Failure in pci dev enable cmd\n"); + + return -EIO; + } + + return 0; +} + +/** + * ne_pci_dev_disable - Disable PCI device. + * + * @pdev: PCI device to disable. + * @ne_pci_dev: PCI device private data structure. + * + * @returns: 0 on success, negative return value on failure. + */ +static int ne_pci_dev_disable(struct pci_dev *pdev, + struct ne_pci_dev *ne_pci_dev) +{ + u8 dev_disable_reply = 0; + + BUG_ON(!pdev); + BUG_ON(!ne_pci_dev); + BUG_ON(!ne_pci_dev->iomem_base); + + iowrite8(NE_ENABLE_OFF, ne_pci_dev->iomem_base + NE_ENABLE); + + /* + * TODO: Check for NE_ENABLE_OFF in a loop, to handle cases when the + * device state is not immediately set to disabled and going through a + * transitory state of disabling. + */ + dev_disable_reply = ioread8(ne_pci_dev->iomem_base + NE_ENABLE); + if (dev_disable_reply != NE_ENABLE_OFF) { + dev_err_ratelimited(&pdev->dev, + "Failure in pci dev disable cmd\n"); + + return -EIO; + } + + return 0; +} + +static int ne_probe(struct pci_dev *pdev, const struct pci_device_id *id) +{ + struct ne_pci_dev *ne_pci_dev = NULL; + int rc = -EINVAL; + + ne_pci_dev = kzalloc(sizeof(*ne_pci_dev), GFP_KERNEL); + if (!ne_pci_dev) + return -ENOMEM; + + rc = pci_enable_device(pdev); + if (rc < 0) { + dev_err_ratelimited(&pdev->dev, + "Failure in pci dev enable [rc=%d]\n", rc); + + goto err_pci_enable_dev; + } + + rc = pci_request_regions_exclusive(pdev, "ne_pci_dev"); + if (rc < 0) { + dev_err_ratelimited(&pdev->dev, + "Failure in pci request regions [rc=%d]\n", + rc); + + goto err_req_regions; + } + + ne_pci_dev->iomem_base = pci_iomap(pdev, PCI_BAR_NE, 0); + if (!ne_pci_dev->iomem_base) { + rc = -ENOMEM; + + dev_err_ratelimited(&pdev->dev, + "Failure in pci bar mapping [rc=%d]\n", rc); + + goto err_iomap; + } + + rc = ne_setup_msix(pdev, ne_pci_dev); + if (rc < 0) { + dev_err_ratelimited(&pdev->dev, + "Failure in pci dev msix setup [rc=%d]\n", + rc); + + goto err_setup_msix; + } + + rc = ne_pci_dev_disable(pdev, ne_pci_dev); + if (rc < 0) { + dev_err_ratelimited(&pdev->dev, + "Failure in ne_pci_dev disable [rc=%d]\n", + rc); + + goto err_ne_pci_dev_disable; + } + + rc = ne_pci_dev_enable(pdev, ne_pci_dev); + if (rc < 0) { + dev_err_ratelimited(&pdev->dev, + "Failure in ne_pci_dev enable [rc=%d]\n", + rc); + + goto err_ne_pci_dev_enable; + } + + atomic_set(&ne_pci_dev->cmd_reply_avail, 0); + init_waitqueue_head(&ne_pci_dev->cmd_reply_wait_q); + INIT_LIST_HEAD(&ne_pci_dev->enclaves_list); + mutex_init(&ne_pci_dev->enclaves_list_mutex); + mutex_init(&ne_pci_dev->pci_dev_mutex); + + pci_set_drvdata(pdev, ne_pci_dev); + + return 0; + +err_ne_pci_dev_enable: +err_ne_pci_dev_disable: + pci_free_irq_vectors(pdev); +err_setup_msix: + pci_iounmap(pdev, ne_pci_dev->iomem_base); +err_iomap: + pci_release_regions(pdev); +err_req_regions: + pci_disable_device(pdev); +err_pci_enable_dev: + kzfree(ne_pci_dev); + return rc; +} + +static void ne_remove(struct pci_dev *pdev) +{ + struct ne_pci_dev *ne_pci_dev = pci_get_drvdata(pdev); + + if (!ne_pci_dev || !ne_pci_dev->iomem_base) + return; + + ne_pci_dev_disable(pdev, ne_pci_dev); + + pci_set_drvdata(pdev, NULL); + + pci_free_irq_vectors(pdev); + + pci_iounmap(pdev, ne_pci_dev->iomem_base); + + kzfree(ne_pci_dev); + + pci_release_regions(pdev); + + pci_disable_device(pdev); +} + +/* + * TODO: Add suspend / resume functions for power management w/ CONFIG_PM, if + * needed. + */ +struct pci_driver ne_pci_driver = { + .name = "ne_pci_dev", + .id_table = ne_pci_ids, + .probe = ne_probe, + .remove = ne_remove, +}; From patchwork Tue Apr 21 18:41:40 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paraschiv, Andra-Irina" X-Patchwork-Id: 11502077 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 95D0081 for ; Tue, 21 Apr 2020 18:42:58 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 794C9206D9 for ; Tue, 21 Apr 2020 18:42:58 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="LFal6lDD" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729527AbgDUSmz (ORCPT ); Tue, 21 Apr 2020 14:42:55 -0400 Received: from smtp-fw-33001.amazon.com ([207.171.190.10]:62315 "EHLO smtp-fw-33001.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729501AbgDUSmy (ORCPT ); Tue, 21 Apr 2020 14:42:54 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1587494573; x=1619030573; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=hCVuFtt9M+P1AunrCOwfanKEEX8ct24ptrnRC6w3jCI=; b=LFal6lDDDpQAQe4W5FE/CYdts/ePXeYeQIFCQRrqEyP9npZjdAV/Eqhn RNkq4Y83Hv2FLjoqPJB6kZVobmkXBz5U9Kg8+E+rPdWmf9TBHdXbfRwQB mt5EIsF+RNHkYQZVyY9C4bvb09D5mmvQmiKDiXM3QmSAm+Suc3K9jcLp9 k=; IronPort-SDR: 1eC9AHuypUgqQi5opmT0fPhKBsynrA/xsnzQcjJ3chmR2ilYNEKVDxbhhhg6KHfRej3AuCpfui 3PJktKAyrvAw== X-IronPort-AV: E=Sophos;i="5.72,411,1580774400"; d="scan'208";a="39978453" Received: from sea32-co-svc-lb4-vlan3.sea.corp.amazon.com (HELO email-inbound-relay-2b-c7131dcf.us-west-2.amazon.com) ([10.47.23.38]) by smtp-border-fw-out-33001.sea14.amazon.com with ESMTP; 21 Apr 2020 18:42:53 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan2.pdx.amazon.com [10.170.41.162]) by email-inbound-relay-2b-c7131dcf.us-west-2.amazon.com (Postfix) with ESMTPS id D58C6A1DC5; Tue, 21 Apr 2020 18:42:51 +0000 (UTC) Received: from EX13D16EUB003.ant.amazon.com (10.43.166.99) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 21 Apr 2020 18:42:51 +0000 Received: from 38f9d34ed3b1.ant.amazon.com (10.43.160.90) by EX13D16EUB003.ant.amazon.com (10.43.166.99) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 21 Apr 2020 18:42:42 +0000 From: Andra Paraschiv To: CC: Anthony Liguori , Benjamin Herrenschmidt , Colm MacCarthaigh , Bjoern Doebel , David Woodhouse , Frank van der Linden , Alexander Graf , Martin Pohlack , Matt Wilson , Paolo Bonzini , Balbir Singh , Stewart Smith , Uwe Dannowski , , , Andra Paraschiv Subject: [PATCH v1 05/15] nitro_enclaves: Handle PCI device command requests Date: Tue, 21 Apr 2020 21:41:40 +0300 Message-ID: <20200421184150.68011-6-andraprs@amazon.com> X-Mailer: git-send-email 2.20.1 (Apple Git-117) In-Reply-To: <20200421184150.68011-1-andraprs@amazon.com> References: <20200421184150.68011-1-andraprs@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.160.90] X-ClientProxiedBy: EX13D37UWC004.ant.amazon.com (10.43.162.212) To EX13D16EUB003.ant.amazon.com (10.43.166.99) Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The Nitro Enclaves PCI device exposes a MMIO space that this driver uses to submit command requests and to receive command replies e.g. for enclave creation / termination or setting enclave resources. Add logic for handling PCI device command requests based on the given command type. Register an MSI-X interrupt vector for command reply notifications to handle this type of communication events. Signed-off-by: Alexandru-Catalin Vasile Signed-off-by: Andra Paraschiv --- .../virt/amazon/nitro_enclaves/ne_pci_dev.c | 264 ++++++++++++++++++ 1 file changed, 264 insertions(+) diff --git a/drivers/virt/amazon/nitro_enclaves/ne_pci_dev.c b/drivers/virt/amazon/nitro_enclaves/ne_pci_dev.c index 8fbee95ea291..7453d129689a 100644 --- a/drivers/virt/amazon/nitro_enclaves/ne_pci_dev.c +++ b/drivers/virt/amazon/nitro_enclaves/ne_pci_dev.c @@ -40,6 +40,251 @@ static const struct pci_device_id ne_pci_ids[] = { MODULE_DEVICE_TABLE(pci, ne_pci_ids); +/** + * ne_submit_request - Submit command request to the PCI device based on the + * command type. + * + * This function gets called with the ne_pci_dev mutex held. + * + * @pdev: PCI device to send the command to. + * @cmd_type: command type of the request sent to the PCI device. + * @cmd_request: command request payload. + * @cmd_request_size: size of the command request payload. + * + * @returns: 0 on success, negative return value on failure. + */ +static int ne_submit_request(struct pci_dev *pdev, + enum ne_pci_dev_cmd_type cmd_type, + void *cmd_request, size_t cmd_request_size) +{ + struct ne_pci_dev *ne_pci_dev = NULL; + + BUG_ON(!pdev); + + ne_pci_dev = pci_get_drvdata(pdev); + BUG_ON(!ne_pci_dev); + BUG_ON(!ne_pci_dev->iomem_base); + + if (WARN_ON(cmd_type <= INVALID_CMD || cmd_type >= MAX_CMD)) { + dev_err_ratelimited(&pdev->dev, "Invalid cmd type=%d\n", + cmd_type); + + return -EINVAL; + } + + if (WARN_ON(!cmd_request)) + return -EINVAL; + + if (WARN_ON(cmd_request_size > NE_SEND_DATA_SIZE)) { + dev_err_ratelimited(&pdev->dev, + "Invalid req size=%ld for cmd type=%d\n", + cmd_request_size, cmd_type); + + return -EINVAL; + } + + memcpy_toio(ne_pci_dev->iomem_base + NE_SEND_DATA, cmd_request, + cmd_request_size); + + iowrite32(cmd_type, ne_pci_dev->iomem_base + NE_COMMAND); + + return 0; +} + +/** + * ne_retrieve_reply - Retrieve reply from the PCI device. + * + * This function gets called with the ne_pci_dev mutex held. + * + * @pdev: PCI device to receive the reply from. + * @cmd_reply: command reply payload. + * @cmd_reply_size: size of the command reply payload. + * + * @returns: 0 on success, negative return value on failure. + */ +static int ne_retrieve_reply(struct pci_dev *pdev, + struct ne_pci_dev_cmd_reply *cmd_reply, + size_t cmd_reply_size) +{ + struct ne_pci_dev *ne_pci_dev = NULL; + + BUG_ON(!pdev); + + ne_pci_dev = pci_get_drvdata(pdev); + BUG_ON(!ne_pci_dev); + BUG_ON(!ne_pci_dev->iomem_base); + + if (WARN_ON(!cmd_reply)) + return -EINVAL; + + if (WARN_ON(cmd_reply_size > NE_RECV_DATA_SIZE)) { + dev_err_ratelimited(&pdev->dev, "Invalid reply size=%ld\n", + cmd_reply_size); + + return -EINVAL; + } + + memcpy_fromio(cmd_reply, ne_pci_dev->iomem_base + NE_RECV_DATA, + cmd_reply_size); + + return 0; +} + +/** + * ne_wait_for_reply - Wait for a reply of a PCI command. + * + * This function gets called with the ne_pci_dev mutex held. + * + * @pdev: PCI device for which a reply is waited. + * + * @returns: 0 on success, negative return value on failure. + */ +static int ne_wait_for_reply(struct pci_dev *pdev) +{ + struct ne_pci_dev *ne_pci_dev = NULL; + int rc = -EINVAL; + + BUG_ON(!pdev); + + ne_pci_dev = pci_get_drvdata(pdev); + BUG_ON(!ne_pci_dev); + + /* + * TODO: Update to _interruptible and handle interrupted wait event + * e.g. -ERESTARTSYS, incoming signals + add / update timeout. + */ + rc = wait_event_timeout(ne_pci_dev->cmd_reply_wait_q, + atomic_read(&ne_pci_dev->cmd_reply_avail) != 0, + msecs_to_jiffies(DEFAULT_TIMEOUT_MSECS)); + if (!rc) { + pr_err("Wait event timed out when waiting for PCI cmd reply\n"); + + return -ETIMEDOUT; + } + + return 0; +} + +int ne_do_request(struct pci_dev *pdev, enum ne_pci_dev_cmd_type cmd_type, + void *cmd_request, size_t cmd_request_size, + struct ne_pci_dev_cmd_reply *cmd_reply, size_t cmd_reply_size) +{ + struct ne_pci_dev *ne_pci_dev = NULL; + int rc = -EINVAL; + + BUG_ON(!pdev); + + ne_pci_dev = pci_get_drvdata(pdev); + BUG_ON(!ne_pci_dev); + BUG_ON(!ne_pci_dev->iomem_base); + + if (WARN_ON(cmd_type <= INVALID_CMD || cmd_type >= MAX_CMD)) { + dev_err_ratelimited(&pdev->dev, "Invalid cmd type=%d\n", + cmd_type); + + return -EINVAL; + } + + if (WARN_ON(!cmd_request)) + return -EINVAL; + + if (WARN_ON(cmd_request_size > NE_SEND_DATA_SIZE)) { + dev_err_ratelimited(&pdev->dev, + "Invalid req size=%ld for cmd type=%d\n", + cmd_request_size, cmd_type); + + return -EINVAL; + } + + if (WARN_ON(!cmd_reply)) + return -EINVAL; + + if (WARN_ON(cmd_reply_size > NE_RECV_DATA_SIZE)) { + dev_err_ratelimited(&pdev->dev, "Invalid reply size=%ld\n", + cmd_reply_size); + + return -EINVAL; + } + + /* + * Use this mutex so that the PCI device handles one command request at + * a time. + */ + mutex_lock(&ne_pci_dev->pci_dev_mutex); + + atomic_set(&ne_pci_dev->cmd_reply_avail, 0); + + rc = ne_submit_request(pdev, cmd_type, cmd_request, cmd_request_size); + if (rc < 0) { + dev_err_ratelimited(&pdev->dev, + "Failure in submit cmd request [rc=%d]\n", + rc); + + mutex_unlock(&ne_pci_dev->pci_dev_mutex); + + return rc; + } + + rc = ne_wait_for_reply(pdev); + if (rc < 0) { + dev_err_ratelimited(&pdev->dev, + "Failure in wait cmd reply [rc=%d]\n", + rc); + + mutex_unlock(&ne_pci_dev->pci_dev_mutex); + + return rc; + } + + rc = ne_retrieve_reply(pdev, cmd_reply, cmd_reply_size); + if (rc < 0) { + dev_err_ratelimited(&pdev->dev, + "Failure in retrieve cmd reply [rc=%d]\n", + rc); + + mutex_unlock(&ne_pci_dev->pci_dev_mutex); + + return rc; + } + + atomic_set(&ne_pci_dev->cmd_reply_avail, 0); + + if (cmd_reply->rc < 0) { + dev_err_ratelimited(&pdev->dev, + "Failure in cmd process logic [rc=%d]\n", + cmd_reply->rc); + + mutex_unlock(&ne_pci_dev->pci_dev_mutex); + + return cmd_reply->rc; + } + + mutex_unlock(&ne_pci_dev->pci_dev_mutex); + + return 0; +} + +/** + * ne_reply_handler - Interrupt handler for retrieving a reply matching + * a request sent to the PCI device for enclave lifetime management. + * + * @irq: received interrupt for a reply sent by the PCI device. + * @args: PCI device private data structure. + * + * @returns: IRQ_HANDLED on handled interrupt, IRQ_NONE otherwise. + */ +static irqreturn_t ne_reply_handler(int irq, void *args) +{ + struct ne_pci_dev *ne_pci_dev = (struct ne_pci_dev *)args; + + atomic_set(&ne_pci_dev->cmd_reply_avail, 1); + + /* TODO: Update to _interruptible. */ + wake_up(&ne_pci_dev->cmd_reply_wait_q); + + return IRQ_HANDLED; +} + /** * ne_setup_msix - Setup MSI-X vectors for the PCI device. * @@ -75,8 +320,25 @@ static int ne_setup_msix(struct pci_dev *pdev, struct ne_pci_dev *ne_pci_dev) goto err_alloc_irq_vecs; } + /* + * This IRQ gets triggered every time the PCI device responds to a + * command request. The reply is then retrieved, reading from the MMIO + * space of the PCI device. + */ + rc = request_irq(pci_irq_vector(pdev, NE_VEC_REPLY), + ne_reply_handler, 0, "enclave_cmd", ne_pci_dev); + if (rc < 0) { + dev_err_ratelimited(&pdev->dev, + "Failure in allocating irq reply [rc=%d]\n", + rc); + + goto err_req_irq_reply; + } + return 0; +err_req_irq_reply: + pci_free_irq_vectors(pdev); err_alloc_irq_vecs: return rc; } @@ -232,6 +494,7 @@ static int ne_probe(struct pci_dev *pdev, const struct pci_device_id *id) err_ne_pci_dev_enable: err_ne_pci_dev_disable: + free_irq(pci_irq_vector(pdev, NE_VEC_REPLY), ne_pci_dev); pci_free_irq_vectors(pdev); err_setup_msix: pci_iounmap(pdev, ne_pci_dev->iomem_base); @@ -255,6 +518,7 @@ static void ne_remove(struct pci_dev *pdev) pci_set_drvdata(pdev, NULL); + free_irq(pci_irq_vector(pdev, NE_VEC_REPLY), ne_pci_dev); pci_free_irq_vectors(pdev); pci_iounmap(pdev, ne_pci_dev->iomem_base); From patchwork Tue Apr 21 18:41:41 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paraschiv, Andra-Irina" X-Patchwork-Id: 11502081 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5188B81 for ; Tue, 21 Apr 2020 18:43:25 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3102D206F4 for ; Tue, 21 Apr 2020 18:43:25 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="UM2/YvuK" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729574AbgDUSnR (ORCPT ); Tue, 21 Apr 2020 14:43:17 -0400 Received: from smtp-fw-4101.amazon.com ([72.21.198.25]:40570 "EHLO smtp-fw-4101.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729457AbgDUSnQ (ORCPT ); Tue, 21 Apr 2020 14:43:16 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1587494595; x=1619030595; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=L+YfGbNVRac+QAclE7qZ7rnk6HjJJ2K8eilo3W1BJmo=; b=UM2/YvuKb3OqXuLG0O86jLegO1vr9VdYFm8Mk+IZzeePocZKUfPEYdKd YST4nBURMsETQw82mLAARU1rJv5pk2GrjqKsBd7eqcXNVGWJJXnEnmYN7 TBI6Yor7lx22FAQwQ5sFyJ+S35x8nBXMCPsYoNv/dq8iy91xQx2amEgCn o=; IronPort-SDR: e775FMH867kBhx9iO8hRDujbCwmXcID2cz+VYNDBr47aF9Og6gt3gpuLGCtHaSBLTyrdzd+PTt wwZRwjzvBSsQ== X-IronPort-AV: E=Sophos;i="5.72,411,1580774400"; d="scan'208";a="26702238" Received: from iad12-co-svc-p1-lb1-vlan3.amazon.com (HELO email-inbound-relay-2b-8cc5d68b.us-west-2.amazon.com) ([10.43.8.6]) by smtp-border-fw-out-4101.iad4.amazon.com with ESMTP; 21 Apr 2020 18:43:01 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan3.pdx.amazon.com [10.170.41.166]) by email-inbound-relay-2b-8cc5d68b.us-west-2.amazon.com (Postfix) with ESMTPS id DEB49A1A13; Tue, 21 Apr 2020 18:42:59 +0000 (UTC) Received: from EX13D16EUB003.ant.amazon.com (10.43.166.99) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 21 Apr 2020 18:42:59 +0000 Received: from 38f9d34ed3b1.ant.amazon.com (10.43.160.90) by EX13D16EUB003.ant.amazon.com (10.43.166.99) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 21 Apr 2020 18:42:50 +0000 From: Andra Paraschiv To: CC: Anthony Liguori , Benjamin Herrenschmidt , Colm MacCarthaigh , Bjoern Doebel , David Woodhouse , Frank van der Linden , Alexander Graf , Martin Pohlack , Matt Wilson , Paolo Bonzini , Balbir Singh , Stewart Smith , Uwe Dannowski , , , Andra Paraschiv Subject: [PATCH v1 06/15] nitro_enclaves: Handle out-of-band PCI device events Date: Tue, 21 Apr 2020 21:41:41 +0300 Message-ID: <20200421184150.68011-7-andraprs@amazon.com> X-Mailer: git-send-email 2.20.1 (Apple Git-117) In-Reply-To: <20200421184150.68011-1-andraprs@amazon.com> References: <20200421184150.68011-1-andraprs@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.160.90] X-ClientProxiedBy: EX13D37UWC004.ant.amazon.com (10.43.162.212) To EX13D16EUB003.ant.amazon.com (10.43.166.99) Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org In addition to the replies sent by the Nitro Enclaves PCI device in response to command requests, out-of-band enclave events can happen e.g. an enclave crashes. In this case, the Nitro Enclaves driver needs to be aware of the event and notify the corresponding user space process that abstracts the enclave. Register an MSI-X interrupt vector to be used for this kind of out-of-band events. The interrupt notifies that the state of an enclave changed and the driver logic scans the state of each running enclave to identify for which this notification is intended. Create an workqueue to handle the out-of-band events. Notify user space enclave process that is using a polling mechanism on the enclave fd. The enclave fd is returned as a result of KVM_CREATE_VM ioctl call. Signed-off-by: Alexandru-Catalin Vasile Signed-off-by: Andra Paraschiv --- .../virt/amazon/nitro_enclaves/ne_pci_dev.c | 120 ++++++++++++++++++ 1 file changed, 120 insertions(+) diff --git a/drivers/virt/amazon/nitro_enclaves/ne_pci_dev.c b/drivers/virt/amazon/nitro_enclaves/ne_pci_dev.c index 7453d129689a..884acbb92305 100644 --- a/drivers/virt/amazon/nitro_enclaves/ne_pci_dev.c +++ b/drivers/virt/amazon/nitro_enclaves/ne_pci_dev.c @@ -285,6 +285,85 @@ static irqreturn_t ne_reply_handler(int irq, void *args) return IRQ_HANDLED; } +/** + * ne_event_work_handler - Work queue handler for notifying enclaves on + * a state change received by the event interrupt handler. + * + * An out-of-band event is being issued by the Nitro Hypervisor when at least + * one enclave is changing state without client interaction. + * + * @work: item containing the Nitro Enclaves PCI device for which a + * out-of-band event was issued. + */ +static void ne_event_work_handler(struct work_struct *work) +{ + struct ne_pci_dev_cmd_reply cmd_reply = {}; + struct ne_enclave *ne_enclave = NULL; + struct ne_pci_dev *ne_pci_dev = + container_of(work, struct ne_pci_dev, notify_work); + int rc = -EINVAL; + struct slot_info_req slot_info_req = {}; + + mutex_lock(&ne_pci_dev->enclaves_list_mutex); + + /* + * Iterate over all enclaves registered for the Nitro Enclaves + * PCI device and determine for which enclave(s) the out-of-band event + * is corresponding to. + */ + list_for_each_entry(ne_enclave, &ne_pci_dev->enclaves_list, + enclave_list_entry) { + mutex_lock(&ne_enclave->enclave_info_mutex); + + /* + * Enclaves that were never started cannot receive out-of-band + * events. + */ + if (ne_enclave->state != NE_STATE_RUNNING) + goto unlock; + + slot_info_req.slot_uid = ne_enclave->slot_uid; + + rc = ne_do_request(ne_enclave->pdev, SLOT_INFO, &slot_info_req, + sizeof(slot_info_req), &cmd_reply, + sizeof(cmd_reply)); + WARN_ON(rc < 0); + + /* Notify enclave process that the enclave state changed. */ + if (ne_enclave->state != cmd_reply.state) { + ne_enclave->state = cmd_reply.state; + + ne_enclave->has_event = true; + + wake_up_interruptible(&ne_enclave->eventq); + } + +unlock: + mutex_unlock(&ne_enclave->enclave_info_mutex); + } + + mutex_unlock(&ne_pci_dev->enclaves_list_mutex); +} + +/** + * ne_event_handler - Interrupt handler for PCI device out-of-band + * events. This interrupt does not supply any data in the MMIO region. + * It notifies a change in the state of any of the launched enclaves. + * + * @irq: received interrupt for an out-of-band event. + * @args: PCI device private data structure. + * + * @returns: IRQ_HANDLED on handled interrupt, IRQ_NONE otherwise. + */ +static irqreturn_t ne_event_handler(int irq, void *args) +{ + struct ne_pci_dev *ne_pci_dev = (struct ne_pci_dev *)args; + + queue_work(ne_pci_dev->event_wq, &ne_pci_dev->notify_work); + + return IRQ_HANDLED; +} + /** * ne_setup_msix - Setup MSI-X vectors for the PCI device. * @@ -311,6 +390,19 @@ static int ne_setup_msix(struct pci_dev *pdev, struct ne_pci_dev *ne_pci_dev) return rc; } + ne_pci_dev->event_wq = create_singlethread_workqueue("ne_pci_dev_wq"); + if (!ne_pci_dev->event_wq) { + rc = -ENOMEM; + + dev_err_ratelimited(&pdev->dev, + "Cannot get wq for device events [rc=%d]\n", + rc); + + goto err_create_wq; + } + + INIT_WORK(&ne_pci_dev->notify_work, ne_event_work_handler); + rc = pci_alloc_irq_vectors(pdev, nr_vecs, nr_vecs, PCI_IRQ_MSIX); if (rc < 0) { dev_err_ratelimited(&pdev->dev, @@ -335,11 +427,30 @@ static int ne_setup_msix(struct pci_dev *pdev, struct ne_pci_dev *ne_pci_dev) goto err_req_irq_reply; } + /* + * This IRQ gets triggered every time any enclave's state changes. Its + * handler then scans for the changes and propagates them to the user + * space. + */ + rc = request_irq(pci_irq_vector(pdev, NE_VEC_EVENT), + ne_event_handler, 0, "enclave_evt", ne_pci_dev); + if (rc < 0) { + dev_err_ratelimited(&pdev->dev, + "Failure in allocating irq event [rc=%d]\n", + rc); + + goto err_req_irq_event; + } + return 0; +err_req_irq_event: + free_irq(pci_irq_vector(pdev, NE_VEC_REPLY), ne_pci_dev); err_req_irq_reply: pci_free_irq_vectors(pdev); err_alloc_irq_vecs: + destroy_workqueue(ne_pci_dev->event_wq); +err_create_wq: return rc; } @@ -494,8 +605,10 @@ static int ne_probe(struct pci_dev *pdev, const struct pci_device_id *id) err_ne_pci_dev_enable: err_ne_pci_dev_disable: + free_irq(pci_irq_vector(pdev, NE_VEC_EVENT), ne_pci_dev); free_irq(pci_irq_vector(pdev, NE_VEC_REPLY), ne_pci_dev); pci_free_irq_vectors(pdev); + destroy_workqueue(ne_pci_dev->event_wq); err_setup_msix: pci_iounmap(pdev, ne_pci_dev->iomem_base); err_iomap: @@ -518,9 +631,16 @@ static void ne_remove(struct pci_dev *pdev) pci_set_drvdata(pdev, NULL); + free_irq(pci_irq_vector(pdev, NE_VEC_EVENT), ne_pci_dev); free_irq(pci_irq_vector(pdev, NE_VEC_REPLY), ne_pci_dev); pci_free_irq_vectors(pdev); + if (ne_pci_dev->event_wq) { + flush_work(&ne_pci_dev->notify_work); + flush_workqueue(ne_pci_dev->event_wq); + destroy_workqueue(ne_pci_dev->event_wq); + } + pci_iounmap(pdev, ne_pci_dev->iomem_base); kzfree(ne_pci_dev); From patchwork Tue Apr 21 18:41:42 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paraschiv, Andra-Irina" X-Patchwork-Id: 11502079 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3012481 for ; Tue, 21 Apr 2020 18:43:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1236020738 for ; Tue, 21 Apr 2020 18:43:13 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="g7G4nI2l" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729552AbgDUSnK (ORCPT ); Tue, 21 Apr 2020 14:43:10 -0400 Received: from smtp-fw-33001.amazon.com ([207.171.190.10]:62367 "EHLO smtp-fw-33001.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729542AbgDUSnJ (ORCPT ); Tue, 21 Apr 2020 14:43:09 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1587494588; x=1619030588; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=dnaTm1aF+1cLi/7DGn/IqxJTYcFCqrkiSr63xiWSCiw=; b=g7G4nI2lIREJhnV7emZuBPoEw/KX/UDwbE9Uoqx2O7DLChbRKQO8X1F9 lgVDqE5/O6H/Osd3YX4WGWhgmBw1Occhq/+9P96ckM7r7BNhSiu+B6toB /7NnBr/79CmMi97TogLZGqBDyUB91fu76K2BRyUFteUNKswsmV5lkRYwM Y=; IronPort-SDR: QdqEgfSxVuT0T2BMhPVvqLkO1BJBGznv7qxXLC50KC3eSOnm1hiYsqM0GSlwiHQT7QRSEE5hqn wjZ9YR5e2J7Q== X-IronPort-AV: E=Sophos;i="5.72,411,1580774400"; d="scan'208";a="39978497" Received: from sea32-co-svc-lb4-vlan3.sea.corp.amazon.com (HELO email-inbound-relay-2a-1c1b5cdd.us-west-2.amazon.com) ([10.47.23.38]) by smtp-border-fw-out-33001.sea14.amazon.com with ESMTP; 21 Apr 2020 18:43:08 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan2.pdx.amazon.com [10.170.41.162]) by email-inbound-relay-2a-1c1b5cdd.us-west-2.amazon.com (Postfix) with ESMTPS id A704DA06EE; Tue, 21 Apr 2020 18:43:07 +0000 (UTC) Received: from EX13D16EUB003.ant.amazon.com (10.43.166.99) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 21 Apr 2020 18:43:07 +0000 Received: from 38f9d34ed3b1.ant.amazon.com (10.43.160.90) by EX13D16EUB003.ant.amazon.com (10.43.166.99) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 21 Apr 2020 18:42:58 +0000 From: Andra Paraschiv To: CC: Anthony Liguori , Benjamin Herrenschmidt , Colm MacCarthaigh , Bjoern Doebel , David Woodhouse , Frank van der Linden , Alexander Graf , Martin Pohlack , Matt Wilson , Paolo Bonzini , Balbir Singh , Stewart Smith , Uwe Dannowski , , , Andra Paraschiv Subject: [PATCH v1 07/15] nitro_enclaves: Init misc device providing the ioctl interface Date: Tue, 21 Apr 2020 21:41:42 +0300 Message-ID: <20200421184150.68011-8-andraprs@amazon.com> X-Mailer: git-send-email 2.20.1 (Apple Git-117) In-Reply-To: <20200421184150.68011-1-andraprs@amazon.com> References: <20200421184150.68011-1-andraprs@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.160.90] X-ClientProxiedBy: EX13D37UWC004.ant.amazon.com (10.43.162.212) To EX13D16EUB003.ant.amazon.com (10.43.166.99) Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The Nitro Enclaves driver provides an ioctl interface to the user space for enclave lifetime management e.g. enclave creation / termination and setting enclave resources such as memory and CPU. This ioctl interface is mapped to a Nitro Enclaves misc device. Signed-off-by: Andra Paraschiv --- .../virt/amazon/nitro_enclaves/ne_misc_dev.c | 174 ++++++++++++++++++ .../virt/amazon/nitro_enclaves/ne_pci_dev.c | 13 ++ 2 files changed, 187 insertions(+) create mode 100644 drivers/virt/amazon/nitro_enclaves/ne_misc_dev.c diff --git a/drivers/virt/amazon/nitro_enclaves/ne_misc_dev.c b/drivers/virt/amazon/nitro_enclaves/ne_misc_dev.c new file mode 100644 index 000000000000..d22a76ed07e5 --- /dev/null +++ b/drivers/virt/amazon/nitro_enclaves/ne_misc_dev.c @@ -0,0 +1,174 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. + * + * This program is free software; you can redistribute it and/or modify it + * under the terms and conditions of the GNU General Public License, + * version 2, as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + * GNU General Public License for more details. + * + * You should have received a copy of the GNU General Public License + * along with this program; if not, see . + */ + +/** + * Enclave lifetime management driver for Nitro Enclaves (NE). + * Nitro is a hypervisor that has been developed by Amazon. + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "ne_misc_dev.h" +#include "ne_pci_dev.h" + +#define NE_DEV_NAME "nitro_enclaves" + +#define MIN_MEM_REGION_SIZE (2 * 1024UL * 1024UL) + +static char *ne_cpus; +module_param(ne_cpus, charp, 0644); +MODULE_PARM_DESC(ne_cpus, " - CPU pool used for Nitro Enclaves"); + +/* CPU pool used for Nitro Enclaves. */ +struct ne_cpu_pool { + /* Available CPUs in the pool. */ + cpumask_var_t avail; +}; + +static struct ne_cpu_pool ne_cpu_pool; + +static struct mutex ne_cpu_pool_mutex; + +static int ne_open(struct inode *node, struct file *file) +{ + return 0; +} + +static long ne_ioctl(struct file *file, unsigned int cmd, unsigned long arg) +{ + switch (cmd) { + + default: + return -ENOTTY; + } + + return 0; +} + +static int ne_release(struct inode *inode, struct file *file) +{ + return 0; +} + +static const struct file_operations ne_fops = { + .owner = THIS_MODULE, + .llseek = noop_llseek, + .unlocked_ioctl = ne_ioctl, + .open = ne_open, + .release = ne_release, +}; + +struct miscdevice ne_miscdevice = { + .minor = MISC_DYNAMIC_MINOR, + .name = NE_DEV_NAME, + .fops = &ne_fops, + .mode = 0664, +}; + +static int __init ne_init(void) +{ + unsigned int cpu = 0; + int rc = -EINVAL; + + memset(&ne_cpu_pool, 0, sizeof(ne_cpu_pool)); + + if (!zalloc_cpumask_var(&ne_cpu_pool.avail, GFP_KERNEL)) + return -ENOMEM; + + mutex_init(&ne_cpu_pool_mutex); + + rc = cpulist_parse(ne_cpus, ne_cpu_pool.avail); + if (rc < 0) { + pr_err_ratelimited("Failure in cpulist parse [rc=%d]\n", rc); + + goto err_cpulist_parse; + } + + for_each_cpu(cpu, ne_cpu_pool.avail) { + rc = remove_cpu(cpu); + if (rc != 0) { + pr_err_ratelimited("Failure in cpu=%d remove [rc=%d]\n", + cpu, rc); + + goto err_remove_cpu; + } + } + + rc = pci_register_driver(&ne_pci_driver); + if (rc < 0) { + pr_err_ratelimited("Failure in pci register driver [rc=%d]\n", + rc); + + goto err_pci_register_driver; + } + + return 0; + +err_pci_register_driver: +err_remove_cpu: + for_each_cpu(cpu, ne_cpu_pool.avail) + add_cpu(cpu); +err_cpulist_parse: + free_cpumask_var(ne_cpu_pool.avail); + return rc; +} + +static void __exit ne_exit(void) +{ + unsigned int cpu = 0; + int rc = -EINVAL; + + pci_unregister_driver(&ne_pci_driver); + + if (!ne_cpu_pool.avail) + return; + + for_each_cpu(cpu, ne_cpu_pool.avail) { + rc = add_cpu(cpu); + if (WARN_ON(rc != 0)) + pr_err_ratelimited("Failure in cpu=%d add [rc=%d]\n", + cpu, rc); + } + + free_cpumask_var(ne_cpu_pool.avail); +} + +/* TODO: Handle actions such as reboot, kexec. */ + +module_init(ne_init); +module_exit(ne_exit); + +MODULE_AUTHOR("Amazon.com, Inc. or its affiliates"); +MODULE_DESCRIPTION("Nitro Enclaves Driver"); +MODULE_LICENSE("GPL v2"); diff --git a/drivers/virt/amazon/nitro_enclaves/ne_pci_dev.c b/drivers/virt/amazon/nitro_enclaves/ne_pci_dev.c index 884acbb92305..19b3836b08f4 100644 --- a/drivers/virt/amazon/nitro_enclaves/ne_pci_dev.c +++ b/drivers/virt/amazon/nitro_enclaves/ne_pci_dev.c @@ -593,6 +593,15 @@ static int ne_probe(struct pci_dev *pdev, const struct pci_device_id *id) goto err_ne_pci_dev_enable; } + rc = misc_register(&ne_miscdevice); + if (rc < 0) { + dev_err_ratelimited(&pdev->dev, + "Failure in misc dev register [rc=%d]\n", + rc); + + goto err_misc_register; + } + atomic_set(&ne_pci_dev->cmd_reply_avail, 0); init_waitqueue_head(&ne_pci_dev->cmd_reply_wait_q); INIT_LIST_HEAD(&ne_pci_dev->enclaves_list); @@ -603,6 +612,8 @@ static int ne_probe(struct pci_dev *pdev, const struct pci_device_id *id) return 0; +err_misc_register: + ne_pci_dev_disable(pdev, ne_pci_dev); err_ne_pci_dev_enable: err_ne_pci_dev_disable: free_irq(pci_irq_vector(pdev, NE_VEC_EVENT), ne_pci_dev); @@ -627,6 +638,8 @@ static void ne_remove(struct pci_dev *pdev) if (!ne_pci_dev || !ne_pci_dev->iomem_base) return; + misc_deregister(&ne_miscdevice); + ne_pci_dev_disable(pdev, ne_pci_dev); pci_set_drvdata(pdev, NULL); From patchwork Tue Apr 21 18:41:43 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paraschiv, Andra-Irina" X-Patchwork-Id: 11502085 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B888392C for ; Tue, 21 Apr 2020 18:43:38 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9C3012068F for ; Tue, 21 Apr 2020 18:43:38 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="Wv3a2NFU" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729606AbgDUSnZ (ORCPT ); Tue, 21 Apr 2020 14:43:25 -0400 Received: from smtp-fw-33001.amazon.com ([207.171.190.10]:62396 "EHLO smtp-fw-33001.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729540AbgDUSnQ (ORCPT ); Tue, 21 Apr 2020 14:43:16 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1587494597; x=1619030597; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=vsbWuzgjSRqPVgjOvuPHmqAcGQpcFjP8Gy0kRb8w6Sg=; b=Wv3a2NFUSwqgumKKuvgYjN+Q4+hPCE7twqQv1ZHn6M+KyX+Qh4jibtah ehg9eUBrJe36Nr9/UoccXkU9Rbtr15UWma/DzYnQsqXT8oLWlVTo6W/P9 T6IyIM80h0/ZrOdiEx1LxZsyRsf66DVOUCgMTHDudHYIbxKBA0VC1x9zc I=; IronPort-SDR: PltIrDIistU/XYafnYcpmG7MXZt7LkYZ3U8tHqeItECEmLtdhAfmS0mSZfK+ROttI+pgmP4Hg0 n2vfmYCGWDRw== X-IronPort-AV: E=Sophos;i="5.72,411,1580774400"; d="scan'208";a="39978516" Received: from sea32-co-svc-lb4-vlan3.sea.corp.amazon.com (HELO email-inbound-relay-2a-e7be2041.us-west-2.amazon.com) ([10.47.23.38]) by smtp-border-fw-out-33001.sea14.amazon.com with ESMTP; 21 Apr 2020 18:43:16 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan2.pdx.amazon.com [10.170.41.162]) by email-inbound-relay-2a-e7be2041.us-west-2.amazon.com (Postfix) with ESMTPS id B7613A22FD; Tue, 21 Apr 2020 18:43:15 +0000 (UTC) Received: from EX13D16EUB003.ant.amazon.com (10.43.166.99) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 21 Apr 2020 18:43:15 +0000 Received: from 38f9d34ed3b1.ant.amazon.com (10.43.160.90) by EX13D16EUB003.ant.amazon.com (10.43.166.99) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 21 Apr 2020 18:43:06 +0000 From: Andra Paraschiv To: CC: Anthony Liguori , Benjamin Herrenschmidt , Colm MacCarthaigh , Bjoern Doebel , David Woodhouse , Frank van der Linden , Alexander Graf , Martin Pohlack , Matt Wilson , Paolo Bonzini , Balbir Singh , Stewart Smith , Uwe Dannowski , , , Andra Paraschiv Subject: [PATCH v1 08/15] nitro_enclaves: Add logic for enclave vm creation Date: Tue, 21 Apr 2020 21:41:43 +0300 Message-ID: <20200421184150.68011-9-andraprs@amazon.com> X-Mailer: git-send-email 2.20.1 (Apple Git-117) In-Reply-To: <20200421184150.68011-1-andraprs@amazon.com> References: <20200421184150.68011-1-andraprs@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.160.90] X-ClientProxiedBy: EX13D37UWC004.ant.amazon.com (10.43.162.212) To EX13D16EUB003.ant.amazon.com (10.43.166.99) Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Add ioctl command logic for enclave VM creation. It triggers a slot allocation. The enclave resources will be associated with this slot and it will be used as an identifier for triggering enclave run. Return a file descriptor, namely enclave fd. This is further used by the associated user space enclave process to set enclave resources and trigger enclave termination. The poll function is implemented in order to notify the enclave process when an enclave exits without a specific enclave termination command trigger e.g. when an enclave crashes. Signed-off-by: Alexandru Vasile Signed-off-by: Andra Paraschiv --- .../virt/amazon/nitro_enclaves/ne_misc_dev.c | 166 ++++++++++++++++++ 1 file changed, 166 insertions(+) diff --git a/drivers/virt/amazon/nitro_enclaves/ne_misc_dev.c b/drivers/virt/amazon/nitro_enclaves/ne_misc_dev.c index d22a76ed07e5..abbebc7718c2 100644 --- a/drivers/virt/amazon/nitro_enclaves/ne_misc_dev.c +++ b/drivers/virt/amazon/nitro_enclaves/ne_misc_dev.c @@ -60,6 +60,145 @@ static struct ne_cpu_pool ne_cpu_pool; static struct mutex ne_cpu_pool_mutex; +static int ne_enclave_open(struct inode *node, struct file *file) +{ + return 0; +} + +static long ne_enclave_ioctl(struct file *file, unsigned int cmd, + unsigned long arg) +{ + switch (cmd) { + default: + return -ENOTTY; + } + + return 0; +} + +static int ne_enclave_release(struct inode *inode, struct file *file) +{ + return 0; +} + +static __poll_t ne_enclave_poll(struct file *file, poll_table *wait) +{ + __poll_t mask = 0; + struct ne_enclave *ne_enclave = file->private_data; + + poll_wait(file, &ne_enclave->eventq, wait); + + if (!ne_enclave->has_event) + return mask; + + mask = POLLHUP; + + return mask; +} + +static const struct file_operations ne_enclave_fops = { + .owner = THIS_MODULE, + .llseek = noop_llseek, + .poll = ne_enclave_poll, + .unlocked_ioctl = ne_enclave_ioctl, + .open = ne_enclave_open, + .release = ne_enclave_release, +}; + +/** + * ne_create_vm_ioctl - Alloc slot to be associated with an enclave. Create + * enclave file descriptor to be further used for enclave resources handling + * e.g. memory regions and CPUs. + * + * This function gets called with the ne_pci_dev enclave mutex held. + * + * @pdev: PCI device used for enclave lifetime management. + * @ne_pci_dev: private data associated with the PCI device. + * @type: type of the virtual machine to be created. + * + * @returns: enclave fd on success, negative return value on failure. + */ +static int ne_create_vm_ioctl(struct pci_dev *pdev, + struct ne_pci_dev *ne_pci_dev, unsigned long type) +{ + struct ne_pci_dev_cmd_reply cmd_reply = {}; + int fd = 0; + struct file *file = NULL; + struct ne_enclave *ne_enclave = NULL; + int rc = -EINVAL; + struct slot_alloc_req slot_alloc_req = {}; + + BUG_ON(!pdev); + BUG_ON(!ne_pci_dev); + + ne_enclave = kzalloc(sizeof(*ne_enclave), GFP_KERNEL); + if (!ne_enclave) + return -ENOMEM; + + if (!zalloc_cpumask_var(&ne_enclave->cpu_siblings, GFP_KERNEL)) { + kzfree(ne_enclave); + + return -ENOMEM; + } + + fd = get_unused_fd_flags(O_CLOEXEC); + if (fd < 0) { + rc = fd; + + pr_err_ratelimited("Failure in getting unused fd [rc=%d]\n", + rc); + + goto err_get_unused_fd; + } + + file = anon_inode_getfile("ne-vm", &ne_enclave_fops, ne_enclave, + O_RDWR); + if (IS_ERR(file)) { + rc = PTR_ERR(file); + + pr_err_ratelimited("Failure in anon inode get file [rc=%d]\n", + rc); + + goto err_anon_inode_getfile; + } + + ne_enclave->pdev = pdev; + + rc = ne_do_request(ne_enclave->pdev, SLOT_ALLOC, &slot_alloc_req, + sizeof(slot_alloc_req), &cmd_reply, + sizeof(cmd_reply)); + if (rc < 0) { + pr_err_ratelimited("Failure in slot alloc [rc=%d]\n", rc); + + goto err_slot_alloc; + } + + init_waitqueue_head(&ne_enclave->eventq); + ne_enclave->has_event = false; + mutex_init(&ne_enclave->enclave_info_mutex); + ne_enclave->max_mem_regions = cmd_reply.mem_regions; + INIT_LIST_HEAD(&ne_enclave->mem_regions_list); + ne_enclave->mm = current->mm; + ne_enclave->slot_uid = cmd_reply.slot_uid; + ne_enclave->state = NE_STATE_INIT; + INIT_LIST_HEAD(&ne_enclave->vcpu_ids_list); + + list_add(&ne_enclave->enclave_list_entry, &ne_pci_dev->enclaves_list); + + fd_install(fd, file); + + return fd; + +err_slot_alloc: + fput(file); +err_anon_inode_getfile: + put_unused_fd(fd); +err_get_unused_fd: + free_cpumask_var(ne_enclave->cpu_siblings); + kzfree(ne_enclave); + return rc; +} + static int ne_open(struct inode *node, struct file *file) { return 0; @@ -67,7 +206,34 @@ static int ne_open(struct inode *node, struct file *file) static long ne_ioctl(struct file *file, unsigned int cmd, unsigned long arg) { + struct ne_pci_dev *ne_pci_dev = NULL; + struct pci_dev *pdev = pci_get_device(PCI_VENDOR_ID_AMAZON, + PCI_DEVICE_ID_NE, NULL); + + BUG_ON(!pdev); + + ne_pci_dev = pci_get_drvdata(pdev); + BUG_ON(!ne_pci_dev); + switch (cmd) { + case KVM_CREATE_VM: { + int rc = -EINVAL; + unsigned long type = 0; + + if (copy_from_user(&type, (void *)arg, sizeof(type))) { + pr_err_ratelimited("Failure in copy from user\n"); + + return -EFAULT; + } + + mutex_lock(&ne_pci_dev->enclaves_list_mutex); + + rc = ne_create_vm_ioctl(pdev, ne_pci_dev, type); + + mutex_unlock(&ne_pci_dev->enclaves_list_mutex); + + return rc; + } default: return -ENOTTY; From patchwork Tue Apr 21 18:41:44 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paraschiv, Andra-Irina" X-Patchwork-Id: 11502083 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4A7B392C for ; Tue, 21 Apr 2020 18:43:29 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2B40B20736 for ; Tue, 21 Apr 2020 18:43:29 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="BPFAjdJe" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729624AbgDUSn2 (ORCPT ); Tue, 21 Apr 2020 14:43:28 -0400 Received: from smtp-fw-33001.amazon.com ([207.171.190.10]:62436 "EHLO smtp-fw-33001.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729562AbgDUSn1 (ORCPT ); Tue, 21 Apr 2020 14:43:27 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1587494606; x=1619030606; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=3HnYBZCPpA+K5LQ93akasa4zhaILbgx153AkIWoMXFg=; b=BPFAjdJeOPRnOqNlbWcsUgVveavAS4RPjYU6Uzz/zeRSRVty712Bkwuh ElBWGQ7Lq6EUdJFyrUapL9lsX4ZKK8t7rIJ1KeurrCgP+RBJCP2DjYkMU 2ARCDaUx27wBWQQtGIeJA8vpwu1/ktOD1kbTNdvlxVBeSbVqcgtqRzBd1 A=; IronPort-SDR: KQLCPE4wIHrA2RmT9SQMuj1c8weUr9lG/uYzm3aJclaM5Vg4m0Q6ykEj/gn8Q8e/RDkfC3g3dI DwY07vFhlmJw== X-IronPort-AV: E=Sophos;i="5.72,411,1580774400"; d="scan'208";a="39978538" Received: from sea32-co-svc-lb4-vlan3.sea.corp.amazon.com (HELO email-inbound-relay-2b-859fe132.us-west-2.amazon.com) ([10.47.23.38]) by smtp-border-fw-out-33001.sea14.amazon.com with ESMTP; 21 Apr 2020 18:43:25 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan2.pdx.amazon.com [10.170.41.162]) by email-inbound-relay-2b-859fe132.us-west-2.amazon.com (Postfix) with ESMTPS id 8FBAD226806; Tue, 21 Apr 2020 18:43:24 +0000 (UTC) Received: from EX13D16EUB003.ant.amazon.com (10.43.166.99) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 21 Apr 2020 18:43:23 +0000 Received: from 38f9d34ed3b1.ant.amazon.com (10.43.160.90) by EX13D16EUB003.ant.amazon.com (10.43.166.99) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 21 Apr 2020 18:43:14 +0000 From: Andra Paraschiv To: CC: Anthony Liguori , Benjamin Herrenschmidt , Colm MacCarthaigh , Bjoern Doebel , David Woodhouse , Frank van der Linden , Alexander Graf , Martin Pohlack , Matt Wilson , Paolo Bonzini , Balbir Singh , Stewart Smith , Uwe Dannowski , , , Andra Paraschiv Subject: [PATCH v1 09/15] nitro_enclaves: Add logic for enclave vcpu creation Date: Tue, 21 Apr 2020 21:41:44 +0300 Message-ID: <20200421184150.68011-10-andraprs@amazon.com> X-Mailer: git-send-email 2.20.1 (Apple Git-117) In-Reply-To: <20200421184150.68011-1-andraprs@amazon.com> References: <20200421184150.68011-1-andraprs@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.160.90] X-ClientProxiedBy: EX13D37UWC004.ant.amazon.com (10.43.162.212) To EX13D16EUB003.ant.amazon.com (10.43.166.99) Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org An enclave, before being started, has its resources set. One of its resources is CPU. Add ioctl command logic for enclave vCPU creation. Return as result a file descriptor that is associated with the enclave vCPU. Signed-off-by: Alexandru Vasile Signed-off-by: Andra Paraschiv --- .../virt/amazon/nitro_enclaves/ne_misc_dev.c | 210 ++++++++++++++++++ 1 file changed, 210 insertions(+) diff --git a/drivers/virt/amazon/nitro_enclaves/ne_misc_dev.c b/drivers/virt/amazon/nitro_enclaves/ne_misc_dev.c index abbebc7718c2..c9acdfd63daf 100644 --- a/drivers/virt/amazon/nitro_enclaves/ne_misc_dev.c +++ b/drivers/virt/amazon/nitro_enclaves/ne_misc_dev.c @@ -60,6 +60,179 @@ static struct ne_cpu_pool ne_cpu_pool; static struct mutex ne_cpu_pool_mutex; +static int ne_enclave_vcpu_open(struct inode *node, struct file *file) +{ + return 0; +} + +static long ne_enclave_vcpu_ioctl(struct file *file, unsigned int cmd, + unsigned long arg) +{ + switch (cmd) { + default: + return -ENOTTY; + } + + return 0; +} + +static int ne_enclave_vcpu_release(struct inode *inode, struct file *file) +{ + return 0; +} + +static const struct file_operations ne_enclave_vcpu_fops = { + .owner = THIS_MODULE, + .llseek = noop_llseek, + .unlocked_ioctl = ne_enclave_vcpu_ioctl, + .open = ne_enclave_vcpu_open, + .release = ne_enclave_vcpu_release, +}; + +/** + * ne_get_cpu_from_cpu_pool - Get a CPU from the CPU pool, if it is set. + * + * This function gets called with the ne_enclave mutex held. + * + * @ne_enclave: private data associated with the current enclave. + * @vcpu_id: id of the CPU to be associated with the given slot, apic id on x86. + * + * @returns: 0 on success, negative return value on failure. + */ +static int ne_get_cpu_from_cpu_pool(struct ne_enclave *ne_enclave, u32 *vcpu_id) +{ + unsigned int cpu = 0; + unsigned int cpu_sibling = 0; + + BUG_ON(!ne_enclave); + + if (WARN_ON(!vcpu_id)) + return -EINVAL; + + /* There are CPU siblings available to choose from. */ + cpu = cpumask_any(ne_enclave->cpu_siblings); + if (cpu < nr_cpu_ids) { + cpumask_clear_cpu(cpu, ne_enclave->cpu_siblings); + + *vcpu_id = cpu; + + return 0; + } + + mutex_lock(&ne_cpu_pool_mutex); + + /* Choose any CPU from the available CPU pool. */ + cpu = cpumask_any(ne_cpu_pool.avail); + if (cpu >= nr_cpu_ids) { + pr_err_ratelimited("No CPUs available in CPU pool\n"); + + mutex_unlock(&ne_cpu_pool_mutex); + + return -EINVAL; + } + + cpumask_clear_cpu(cpu, ne_cpu_pool.avail); + + /* + * Make sure the CPU siblings are not marked as + * available anymore. + */ + for_each_cpu(cpu_sibling, topology_sibling_cpumask(cpu)) { + if (cpu_sibling != cpu) { + cpumask_clear_cpu(cpu_sibling, ne_cpu_pool.avail); + + cpumask_set_cpu(cpu_sibling, ne_enclave->cpu_siblings); + } + } + + mutex_unlock(&ne_cpu_pool_mutex); + + *vcpu_id = cpu; + + return 0; +} + +/** + * ne_create_vcpu_ioctl - Add vCPU to the slot associated with the current + * enclave. Create vCPU file descriptor to be further used for CPU handling. + * + * This function gets called with the ne_enclave mutex held. + * + * @ne_enclave: private data associated with the current enclave. + * @vcpu_id: id of the CPU to be associated with the given slot, apic id on x86. + * + * @returns: vCPU fd on success, negative return value on failure. + */ +static int ne_create_vcpu_ioctl(struct ne_enclave *ne_enclave, u32 vcpu_id) +{ + struct ne_pci_dev_cmd_reply cmd_reply = {}; + int fd = 0; + struct file *file = NULL; + struct ne_vcpu_id *ne_vcpu_id = NULL; + int rc = -EINVAL; + struct slot_add_vcpu_req slot_add_vcpu_req = {}; + + BUG_ON(!ne_enclave); + BUG_ON(!ne_enclave->pdev); + + if (ne_enclave->mm != current->mm) + return -EIO; + + ne_vcpu_id = kzalloc(sizeof(*ne_vcpu_id), GFP_KERNEL); + if (!ne_vcpu_id) + return -ENOMEM; + + fd = get_unused_fd_flags(O_CLOEXEC); + if (fd < 0) { + rc = fd; + + pr_err_ratelimited("Failure in getting unused fd [rc=%d]\n", + rc); + + goto err_get_unused_fd; + } + + /* TODO: Include (vcpu) id in the ne-vm-vcpu naming. */ + file = anon_inode_getfile("ne-vm-vcpu", &ne_enclave_vcpu_fops, + ne_enclave, O_RDWR); + if (IS_ERR(file)) { + rc = PTR_ERR(file); + + pr_err_ratelimited("Failure in anon inode get file [rc=%d]\n", + rc); + + goto err_anon_inode_getfile; + } + + slot_add_vcpu_req.slot_uid = ne_enclave->slot_uid; + slot_add_vcpu_req.vcpu_id = vcpu_id; + + rc = ne_do_request(ne_enclave->pdev, SLOT_ADD_VCPU, &slot_add_vcpu_req, + sizeof(slot_add_vcpu_req), &cmd_reply, + sizeof(cmd_reply)); + if (rc < 0) { + pr_err_ratelimited("Failure in slot add vcpu [rc=%d]\n", rc); + + goto err_slot_add_vcpu; + } + + ne_vcpu_id->vcpu_id = vcpu_id; + + list_add(&ne_vcpu_id->vcpu_id_list_entry, &ne_enclave->vcpu_ids_list); + + fd_install(fd, file); + + return fd; + +err_slot_add_vcpu: + fput(file); +err_anon_inode_getfile: + put_unused_fd(fd); +err_get_unused_fd: + kzfree(ne_vcpu_id); + return rc; +} + static int ne_enclave_open(struct inode *node, struct file *file) { return 0; @@ -68,7 +241,44 @@ static int ne_enclave_open(struct inode *node, struct file *file) static long ne_enclave_ioctl(struct file *file, unsigned int cmd, unsigned long arg) { + struct ne_enclave *ne_enclave = file->private_data; + + BUG_ON(!ne_enclave); + switch (cmd) { + case KVM_CREATE_VCPU: { + int rc = -EINVAL; + u32 vcpu_id = 0; + + if (copy_from_user(&vcpu_id, (void *)arg, sizeof(vcpu_id))) { + pr_err_ratelimited("Failure in copy from user\n"); + + return -EFAULT; + } + + mutex_lock(&ne_enclave->enclave_info_mutex); + + /* Use the CPU pool for choosing a CPU for the enclave. */ + rc = ne_get_cpu_from_cpu_pool(ne_enclave, &vcpu_id); + if (rc < 0) { + pr_err_ratelimited("Failure in get CPU from pool\n"); + + mutex_unlock(&ne_enclave->enclave_info_mutex); + + return -EINVAL; + } + + rc = ne_create_vcpu_ioctl(ne_enclave, vcpu_id); + + /* Put back the CPU in enclave cpu pool, if add vcpu error. */ + if (rc < 0) + cpumask_set_cpu(vcpu_id, ne_enclave->cpu_siblings); + + mutex_unlock(&ne_enclave->enclave_info_mutex); + + return rc; + } + default: return -ENOTTY; } From patchwork Tue Apr 21 18:41:45 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paraschiv, Andra-Irina" X-Patchwork-Id: 11502087 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 245C181 for ; Tue, 21 Apr 2020 18:43:43 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 061DF206F4 for ; Tue, 21 Apr 2020 18:43:43 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="rnaNfQAc" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729649AbgDUSnm (ORCPT ); Tue, 21 Apr 2020 14:43:42 -0400 Received: from smtp-fw-4101.amazon.com ([72.21.198.25]:40638 "EHLO smtp-fw-4101.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729542AbgDUSnl (ORCPT ); Tue, 21 Apr 2020 14:43:41 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1587494620; x=1619030620; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=iD6hK6n6rAHjJ0UFqEld6zc7icpNZCFSfqUcIfc6BKo=; b=rnaNfQAcXNYBAJ5dfSJVVVntDsLokBEEy1Q8NEBPDqhhCxli4d1kjBzD gHkCiPR8jd38E9nITd3c8jpbYE9T+WhfKos+wFEu7nGSp6y1j8OteQHKk qrf6jaNVG+QSmfGBNjc/TTg+bhvacilCLA+5LEJHWZe0ktIyARLyGLYVH g=; IronPort-SDR: AU4ETh7LHYmR+zEG/Q4H1XhmDw2ir1b3ozvvWnRHSZtg7sB37NkyzlzfvpE1H/LOjR13+g3FzC +cvjKAbAppKA== X-IronPort-AV: E=Sophos;i="5.72,411,1580774400"; d="scan'208";a="26702307" Received: from iad12-co-svc-p1-lb1-vlan3.amazon.com (HELO email-inbound-relay-2c-2225282c.us-west-2.amazon.com) ([10.43.8.6]) by smtp-border-fw-out-4101.iad4.amazon.com with ESMTP; 21 Apr 2020 18:43:39 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan3.pdx.amazon.com [10.170.41.166]) by email-inbound-relay-2c-2225282c.us-west-2.amazon.com (Postfix) with ESMTPS id EEF19A2392; Tue, 21 Apr 2020 18:43:37 +0000 (UTC) Received: from EX13D16EUB003.ant.amazon.com (10.43.166.99) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 21 Apr 2020 18:43:37 +0000 Received: from 38f9d34ed3b1.ant.amazon.com (10.43.161.217) by EX13D16EUB003.ant.amazon.com (10.43.166.99) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 21 Apr 2020 18:43:28 +0000 From: Andra Paraschiv To: CC: Anthony Liguori , Benjamin Herrenschmidt , Colm MacCarthaigh , Bjoern Doebel , David Woodhouse , Frank van der Linden , Alexander Graf , Martin Pohlack , Matt Wilson , Paolo Bonzini , Balbir Singh , Stewart Smith , Uwe Dannowski , , , Andra Paraschiv Subject: [PATCH v1 10/15] nitro_enclaves: Add logic for enclave memory region set Date: Tue, 21 Apr 2020 21:41:45 +0300 Message-ID: <20200421184150.68011-11-andraprs@amazon.com> X-Mailer: git-send-email 2.20.1 (Apple Git-117) In-Reply-To: <20200421184150.68011-1-andraprs@amazon.com> References: <20200421184150.68011-1-andraprs@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.161.217] X-ClientProxiedBy: EX13D17UWC002.ant.amazon.com (10.43.162.61) To EX13D16EUB003.ant.amazon.com (10.43.166.99) Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Another resource that is being set for an enclave is memory. User space memory regions, that needs to be backed by contiguous memory regions, are associated with the enclave. One solution for allocating / reserving contiguous memory regions, that is used for integration, is hugetlbfs. The user space process that is associated with the enclave passes to the driver these memory regions. Add ioctl command logic for setting user space memory region for an enclave. Signed-off-by: Alexandru Vasile Signed-off-by: Andra Paraschiv --- .../virt/amazon/nitro_enclaves/ne_misc_dev.c | 242 ++++++++++++++++++ 1 file changed, 242 insertions(+) diff --git a/drivers/virt/amazon/nitro_enclaves/ne_misc_dev.c b/drivers/virt/amazon/nitro_enclaves/ne_misc_dev.c index c9acdfd63daf..0bd283f73a87 100644 --- a/drivers/virt/amazon/nitro_enclaves/ne_misc_dev.c +++ b/drivers/virt/amazon/nitro_enclaves/ne_misc_dev.c @@ -233,6 +233,228 @@ static int ne_create_vcpu_ioctl(struct ne_enclave *ne_enclave, u32 vcpu_id) return rc; } +/** + * ne_sanity_check_user_mem_region - Sanity check the userspace memory + * region received during the set user memory region ioctl call. + * + * This function gets called with the ne_enclave mutex held. + * + * @ne_enclave: private data associated with the current enclave. + * @mem_region: user space memory region to be sanity checked. + * + * @returns: 0 on success, negative return value on failure. + */ +static int ne_sanity_check_user_mem_region(struct ne_enclave *ne_enclave, + struct kvm_userspace_memory_region *mem_region) +{ + BUG_ON(!ne_enclave); + + if (WARN_ON(!mem_region)) + return -EINVAL; + + if (mem_region->slot > ne_enclave->max_mem_regions) { + pr_err_ratelimited("Mem slot higher than max mem regions\n"); + + return -EINVAL; + } + + if ((mem_region->memory_size % MIN_MEM_REGION_SIZE) != 0) { + pr_err_ratelimited("Mem region size not multiple of 2 MiB\n"); + + return -EINVAL; + } + + if ((mem_region->userspace_addr & (MIN_MEM_REGION_SIZE - 1)) || + !access_ok((void __user *)(unsigned long)mem_region->userspace_addr, + mem_region->memory_size)) { + pr_err_ratelimited("Invalid user space addr range\n"); + + return -EINVAL; + } + + if ((mem_region->guest_phys_addr + mem_region->memory_size) < + mem_region->guest_phys_addr) { + pr_err_ratelimited("Invalid guest phys addr range\n"); + + return -EINVAL; + } + + return 0; +} + +/** + * ne_set_user_memory_region_ioctl - Add user space memory region to the slot + * associated with the current enclave. + * + * This function gets called with the ne_enclave mutex held. + * + * @ne_enclave: private data associated with the current enclave. + * @mem_region: user space memory region to be associated with the given slot. + * + * @returns: 0 on success, negative return value on failure. + */ +static int ne_set_user_memory_region_ioctl(struct ne_enclave *ne_enclave, + struct kvm_userspace_memory_region *mem_region) +{ + struct ne_pci_dev_cmd_reply cmd_reply = {}; + long gup_rc = 0; + unsigned long i = 0; + struct ne_mem_region *ne_mem_region = NULL; + unsigned long nr_phys_contig_mem_regions = 0; + unsigned long nr_pinned_pages = 0; + struct page **phys_contig_mem_regions = NULL; + int rc = -EINVAL; + struct slot_add_mem_req slot_add_mem_req = {}; + + BUG_ON(!ne_enclave); + BUG_ON(!ne_enclave->pdev); + + if (WARN_ON(!mem_region)) + return -EINVAL; + + if (ne_enclave->mm != current->mm) + return -EIO; + + rc = ne_sanity_check_user_mem_region(ne_enclave, mem_region); + if (rc < 0) + return rc; + + ne_mem_region = kzalloc(sizeof(*ne_mem_region), GFP_KERNEL); + if (!ne_mem_region) + return -ENOMEM; + + /* + * TODO: Update nr_pages value to handle contiguous virtual address + * ranges mapped to non-contiguous physical regions. Hugetlbfs can give + * 2 MiB / 1 GiB contiguous physical regions. + */ + ne_mem_region->nr_pages = mem_region->memory_size / MIN_MEM_REGION_SIZE; + + ne_mem_region->pages = kcalloc(ne_mem_region->nr_pages, + sizeof(*ne_mem_region->pages), + GFP_KERNEL); + if (!ne_mem_region->pages) { + kzfree(ne_mem_region); + + return -ENOMEM; + } + + phys_contig_mem_regions = kcalloc(ne_mem_region->nr_pages, + sizeof(*phys_contig_mem_regions), + GFP_KERNEL); + if (!phys_contig_mem_regions) { + kzfree(ne_mem_region->pages); + kzfree(ne_mem_region); + + return -ENOMEM; + } + + /* + * TODO: Handle non-contiguous memory regions received from user space. + * Hugetlbfs can give 2 MiB / 1 GiB contiguous physical regions. The + * virtual address space can be seen as contiguous, although it is + * mapped underneath to 2 MiB / 1 GiB physical regions e.g. 8 MiB + * virtual address space mapped to 4 physically contiguous regions of 2 + * MiB. + */ + do { + unsigned long tmp_nr_pages = ne_mem_region->nr_pages - + nr_pinned_pages; + struct page **tmp_pages = ne_mem_region->pages + + nr_pinned_pages; + u64 tmp_userspace_addr = mem_region->userspace_addr + + nr_pinned_pages * MIN_MEM_REGION_SIZE; + + gup_rc = get_user_pages(tmp_userspace_addr, tmp_nr_pages, + FOLL_GET, tmp_pages, NULL); + if (gup_rc < 0) { + rc = gup_rc; + + pr_err_ratelimited("Failure in gup [rc=%d]\n", rc); + + unpin_user_pages(ne_mem_region->pages, nr_pinned_pages); + + goto err_get_user_pages; + } + + nr_pinned_pages += gup_rc; + + } while (nr_pinned_pages < ne_mem_region->nr_pages); + + /* + * TODO: Update checks once physically contiguous regions are collected + * based on the user space address and get_user_pages() results. + */ + for (i = 0; i < ne_mem_region->nr_pages; i++) { + if (!PageHuge(ne_mem_region->pages[i])) { + pr_err_ratelimited("The page isn't a hugetlbfs page\n"); + + goto err_phys_pages_check; + } + + if (huge_page_size(page_hstate(ne_mem_region->pages[i])) != + MIN_MEM_REGION_SIZE) { + pr_err_ratelimited("The page size isn't 2 MiB\n"); + + goto err_phys_pages_check; + } + + /* + * TODO: Update once handled non-contiguous memory regions + * received from user space. + */ + phys_contig_mem_regions[i] = ne_mem_region->pages[i]; + } + + /* + * TODO: Update once handled non-contiguous memory regions received + * from user space. + */ + nr_phys_contig_mem_regions = ne_mem_region->nr_pages; + + for (i = 0; i < nr_phys_contig_mem_regions; i++) { + u64 phys_addr = page_to_phys(phys_contig_mem_regions[i]); + + slot_add_mem_req.slot_uid = ne_enclave->slot_uid; + slot_add_mem_req.paddr = phys_addr; + /* + * TODO: Update memory size of physical contiguous memory + * region, in case of non-contiguous memory regions received + * from user space. + */ + slot_add_mem_req.size = MIN_MEM_REGION_SIZE; + + rc = ne_do_request(ne_enclave->pdev, SLOT_ADD_MEM, + &slot_add_mem_req, sizeof(slot_add_mem_req), + &cmd_reply, sizeof(cmd_reply)); + if (rc < 0) { + pr_err_ratelimited("Failure in slot add mem [rc=%d]\n", + rc); + + goto err_slot_add_mem; + } + + memset(&slot_add_mem_req, 0, sizeof(slot_add_mem_req)); + memset(&cmd_reply, 0, sizeof(cmd_reply)); + } + + list_add(&ne_mem_region->mem_region_list_entry, + &ne_enclave->mem_regions_list); + + kzfree(phys_contig_mem_regions); + + return 0; + +err_slot_add_mem: +err_phys_pages_check: + unpin_user_pages(ne_mem_region->pages, ne_mem_region->nr_pages); +err_get_user_pages: + kzfree(phys_contig_mem_regions); + kzfree(ne_mem_region->pages); + kzfree(ne_mem_region); + return rc; +} + static int ne_enclave_open(struct inode *node, struct file *file) { return 0; @@ -279,6 +501,26 @@ static long ne_enclave_ioctl(struct file *file, unsigned int cmd, return rc; } + case KVM_SET_USER_MEMORY_REGION: { + struct kvm_userspace_memory_region mem_region = {}; + int rc = -EINVAL; + + if (copy_from_user(&mem_region, (void *)arg, + sizeof(mem_region))) { + pr_err_ratelimited("Failure in copy from user\n"); + + return -EFAULT; + } + + mutex_lock(&ne_enclave->enclave_info_mutex); + + rc = ne_set_user_memory_region_ioctl(ne_enclave, &mem_region); + + mutex_unlock(&ne_enclave->enclave_info_mutex); + + return rc; + } + default: return -ENOTTY; } From patchwork Tue Apr 21 18:41:46 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paraschiv, Andra-Irina" X-Patchwork-Id: 11502091 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4736192C for ; Tue, 21 Apr 2020 18:44:05 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 2B397206D9 for ; Tue, 21 Apr 2020 18:44:05 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="dEHSY03i" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729682AbgDUSoB (ORCPT ); Tue, 21 Apr 2020 14:44:01 -0400 Received: from smtp-fw-6001.amazon.com ([52.95.48.154]:44135 "EHLO smtp-fw-6001.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729679AbgDUSoB (ORCPT ); Tue, 21 Apr 2020 14:44:01 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1587494640; x=1619030640; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=tTIsnBjqE+u3ZnWQbEklQODug9U78pYyqe4mU4uBc4s=; b=dEHSY03iRIklzqdLG50OhP3VrruByRRy7Wjeq1OA+o/p8QSJknN5cWzJ Es/aXy6EjJGq+NJpmvO8PXvffO23F7gqDHLElc6B23D1Z3eM4cfExrwvn f9ANVvcQxujTTSoBQv/uWrk0h21VjiBkBRMmyLkaZF88pjEJjBOmg9lMJ Y=; IronPort-SDR: 1bfE/NL9nQ9wrCFDt9DoWxRX8LrzLfnEPlo1zt6gTs3ktQWjKCYzSxPuO4sTQK/UMHDp8d6JIH RnHSDQO0y6wQ== X-IronPort-AV: E=Sophos;i="5.72,411,1580774400"; d="scan'208";a="28051328" Received: from iad12-co-svc-p1-lb1-vlan3.amazon.com (HELO email-inbound-relay-2b-a7fdc47a.us-west-2.amazon.com) ([10.43.8.6]) by smtp-border-fw-out-6001.iad6.amazon.com with ESMTP; 21 Apr 2020 18:43:47 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan2.pdx.amazon.com [10.170.41.162]) by email-inbound-relay-2b-a7fdc47a.us-west-2.amazon.com (Postfix) with ESMTPS id 68F07C5B80; Tue, 21 Apr 2020 18:43:46 +0000 (UTC) Received: from EX13D16EUB003.ant.amazon.com (10.43.166.99) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 21 Apr 2020 18:43:45 +0000 Received: from 38f9d34ed3b1.ant.amazon.com (10.43.161.217) by EX13D16EUB003.ant.amazon.com (10.43.166.99) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 21 Apr 2020 18:43:37 +0000 From: Andra Paraschiv To: CC: Anthony Liguori , Benjamin Herrenschmidt , Colm MacCarthaigh , Bjoern Doebel , David Woodhouse , Frank van der Linden , Alexander Graf , Martin Pohlack , Matt Wilson , Paolo Bonzini , Balbir Singh , Stewart Smith , Uwe Dannowski , , , Andra Paraschiv Subject: [PATCH v1 11/15] nitro_enclaves: Add logic for enclave start Date: Tue, 21 Apr 2020 21:41:46 +0300 Message-ID: <20200421184150.68011-12-andraprs@amazon.com> X-Mailer: git-send-email 2.20.1 (Apple Git-117) In-Reply-To: <20200421184150.68011-1-andraprs@amazon.com> References: <20200421184150.68011-1-andraprs@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.161.217] X-ClientProxiedBy: EX13D17UWC002.ant.amazon.com (10.43.162.61) To EX13D16EUB003.ant.amazon.com (10.43.166.99) Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org After all the enclave resources are set, the enclave is ready for beginning to run. Add ioctl command logic for starting an enclave after all its resources, memory regions and CPUs, have been set. The enclave start information includes the local channel addressing - vsock CID - and the flags associated with the enclave. Signed-off-by: Alexandru Vasile Signed-off-by: Andra Paraschiv --- .../virt/amazon/nitro_enclaves/ne_misc_dev.c | 83 +++++++++++++++++++ 1 file changed, 83 insertions(+) diff --git a/drivers/virt/amazon/nitro_enclaves/ne_misc_dev.c b/drivers/virt/amazon/nitro_enclaves/ne_misc_dev.c index 0bd283f73a87..f07eb46f7995 100644 --- a/drivers/virt/amazon/nitro_enclaves/ne_misc_dev.c +++ b/drivers/virt/amazon/nitro_enclaves/ne_misc_dev.c @@ -455,6 +455,53 @@ static int ne_set_user_memory_region_ioctl(struct ne_enclave *ne_enclave, return rc; } +/** + * ne_enclave_start_ioctl - Trigger enclave start after the enclave resources, + * such as memory and CPU, have been set. + * + * This function gets called with the ne_enclave mutex held. + * + * @ne_enclave: private data associated with the current enclave. + * @enclave_start_metadata: enclave metadata that includes enclave cid and + * flags and the slot uid. + * + * @returns: 0 on success, negative return value on failure. + */ +static int ne_enclave_start_ioctl(struct ne_enclave *ne_enclave, + struct enclave_start_metadata *enclave_start_metadata) +{ + struct ne_pci_dev_cmd_reply cmd_reply = {}; + struct enclave_start_req enclave_start_req = {}; + int rc = -EINVAL; + + BUG_ON(!ne_enclave); + BUG_ON(!ne_enclave->pdev); + + if (WARN_ON(!enclave_start_metadata)) + return -EINVAL; + + enclave_start_metadata->slot_uid = ne_enclave->slot_uid; + + enclave_start_req.enclave_cid = enclave_start_metadata->enclave_cid; + enclave_start_req.flags = enclave_start_metadata->flags; + enclave_start_req.slot_uid = enclave_start_metadata->slot_uid; + + rc = ne_do_request(ne_enclave->pdev, ENCLAVE_START, &enclave_start_req, + sizeof(enclave_start_req), &cmd_reply, + sizeof(cmd_reply)); + if (rc < 0) { + pr_err_ratelimited("Failure in enclave start [rc=%d]\n", rc); + + return rc; + } + + ne_enclave->state = NE_STATE_RUNNING; + + enclave_start_metadata->enclave_cid = cmd_reply.enclave_cid; + + return 0; +} + static int ne_enclave_open(struct inode *node, struct file *file) { return 0; @@ -521,6 +568,42 @@ static long ne_enclave_ioctl(struct file *file, unsigned int cmd, return rc; } + case NE_ENCLAVE_START: { + struct enclave_start_metadata enclave_start_metadata = {}; + int rc = -EINVAL; + + if (copy_from_user(&enclave_start_metadata, (void *)arg, + sizeof(enclave_start_metadata))) { + pr_err_ratelimited("Failure in copy from user\n"); + + return -EFAULT; + } + + mutex_lock(&ne_enclave->enclave_info_mutex); + + if (!cpumask_empty(ne_enclave->cpu_siblings)) { + pr_err_ratelimited("Enclave has CPU siblings avail\n"); + + mutex_unlock(&ne_enclave->enclave_info_mutex); + + return -EINVAL; + } + + rc = ne_enclave_start_ioctl(ne_enclave, + &enclave_start_metadata); + + mutex_unlock(&ne_enclave->enclave_info_mutex); + + if (copy_to_user((void *)arg, &enclave_start_metadata, + sizeof(enclave_start_metadata))) { + pr_err_ratelimited("Failure in copy to user\n"); + + return -EFAULT; + } + + return rc; + } + default: return -ENOTTY; } From patchwork Tue Apr 21 18:41:47 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paraschiv, Andra-Irina" X-Patchwork-Id: 11502089 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B64EB81 for ; Tue, 21 Apr 2020 18:43:59 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 977602071C for ; Tue, 21 Apr 2020 18:43:59 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="IkTkgwEA" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729666AbgDUSn4 (ORCPT ); Tue, 21 Apr 2020 14:43:56 -0400 Received: from smtp-fw-33001.amazon.com ([207.171.190.10]:62511 "EHLO smtp-fw-33001.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725990AbgDUSn4 (ORCPT ); Tue, 21 Apr 2020 14:43:56 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1587494635; x=1619030635; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=+uXZSd+rA8gs8cQcgD50Ns4MwYbbNyukGlXYfO8A2+0=; b=IkTkgwEAlB2H1j8Oal6l8cNfW3i1YlzCnuOAyClEHnztsQw8KMYC4yMm 1NQryUZRftbKfY6vuPUB4g6cnMxoYKwwhQoOOVgNAvXRYZjqW2a6nxSpT 2F7yLxbLkGTOuILx1rY8PIblycos35SoegPtPAvU4I4BoYV5h/AeYuFuF I=; IronPort-SDR: wSYXh4zgWBWEcWcU/2R6X/NlO+JoCVVnmS/45Lh7Gn8gH4vmX348CEiRh0ka1WQvyLVuw3EMUZ Knn9mzpdl7BA== X-IronPort-AV: E=Sophos;i="5.72,411,1580774400"; d="scan'208";a="39978613" Received: from sea32-co-svc-lb4-vlan3.sea.corp.amazon.com (HELO email-inbound-relay-2c-87a10be6.us-west-2.amazon.com) ([10.47.23.38]) by smtp-border-fw-out-33001.sea14.amazon.com with ESMTP; 21 Apr 2020 18:43:54 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan2.pdx.amazon.com [10.170.41.162]) by email-inbound-relay-2c-87a10be6.us-west-2.amazon.com (Postfix) with ESMTPS id 05C61A17AD; Tue, 21 Apr 2020 18:43:53 +0000 (UTC) Received: from EX13D16EUB003.ant.amazon.com (10.43.166.99) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 21 Apr 2020 18:43:53 +0000 Received: from 38f9d34ed3b1.ant.amazon.com (10.43.161.217) by EX13D16EUB003.ant.amazon.com (10.43.166.99) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 21 Apr 2020 18:43:45 +0000 From: Andra Paraschiv To: CC: Anthony Liguori , Benjamin Herrenschmidt , Colm MacCarthaigh , Bjoern Doebel , David Woodhouse , Frank van der Linden , Alexander Graf , Martin Pohlack , Matt Wilson , Paolo Bonzini , Balbir Singh , Stewart Smith , Uwe Dannowski , , , Andra Paraschiv Subject: [PATCH v1 12/15] nitro_enclaves: Add logic for enclave termination Date: Tue, 21 Apr 2020 21:41:47 +0300 Message-ID: <20200421184150.68011-13-andraprs@amazon.com> X-Mailer: git-send-email 2.20.1 (Apple Git-117) In-Reply-To: <20200421184150.68011-1-andraprs@amazon.com> References: <20200421184150.68011-1-andraprs@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.161.217] X-ClientProxiedBy: EX13D17UWC002.ant.amazon.com (10.43.162.61) To EX13D16EUB003.ant.amazon.com (10.43.166.99) Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org An enclave is associated with an fd that is returned after the enclave creation logic is completed. This enclave fd is further used to setup enclave resources. Once the enclave needs to be terminated, the enclave fd is closed. Add logic for enclave termination, that is mapped to the enclave fd release callback. Free the internal enclave info used for bookkeeping. Signed-off-by: Alexandru Vasile Signed-off-by: Andra Paraschiv --- .../virt/amazon/nitro_enclaves/ne_misc_dev.c | 164 ++++++++++++++++++ 1 file changed, 164 insertions(+) diff --git a/drivers/virt/amazon/nitro_enclaves/ne_misc_dev.c b/drivers/virt/amazon/nitro_enclaves/ne_misc_dev.c index f07eb46f7995..08ba8295d524 100644 --- a/drivers/virt/amazon/nitro_enclaves/ne_misc_dev.c +++ b/drivers/virt/amazon/nitro_enclaves/ne_misc_dev.c @@ -611,8 +611,172 @@ static long ne_enclave_ioctl(struct file *file, unsigned int cmd, return 0; } +/** + * ne_enclave_remove_all_mem_region_entries - Remove all memory region + * entries from the enclave data structure. + * + * This function gets called with the ne_enclave mutex held. + * + * @ne_enclave: private data associated with the current enclave. + */ +static void ne_enclave_remove_all_mem_region_entries( + struct ne_enclave *ne_enclave) +{ + struct ne_mem_region *ne_mem_region = NULL; + struct ne_mem_region *ne_mem_region_tmp = NULL; + + BUG_ON(!ne_enclave); + + list_for_each_entry_safe(ne_mem_region, ne_mem_region_tmp, + &ne_enclave->mem_regions_list, + mem_region_list_entry) { + list_del(&ne_mem_region->mem_region_list_entry); + + unpin_user_pages(ne_mem_region->pages, + ne_mem_region->nr_pages); + + kzfree(ne_mem_region->pages); + + kzfree(ne_mem_region); + } +} + +/** + * ne_enclave_remove_all_vcpu_id_entries - Remove all vCPU id entries + * from the enclave data structure. + * + * This function gets called with the ne_enclave mutex held. + * + * @ne_enclave: private data associated with the current enclave. + */ +static void ne_enclave_remove_all_vcpu_id_entries(struct ne_enclave *ne_enclave) +{ + unsigned int cpu = 0; + struct ne_vcpu_id *ne_vcpu_id = NULL; + struct ne_vcpu_id *ne_vcpu_id_tmp = NULL; + + BUG_ON(!ne_enclave); + + mutex_lock(&ne_cpu_pool_mutex); + + list_for_each_entry_safe(ne_vcpu_id, ne_vcpu_id_tmp, + &ne_enclave->vcpu_ids_list, + vcpu_id_list_entry) { + list_del(&ne_vcpu_id->vcpu_id_list_entry); + + /* Update the available CPU pool. */ + cpumask_set_cpu(ne_vcpu_id->vcpu_id, ne_cpu_pool.avail); + + kzfree(ne_vcpu_id); + } + + /* If any siblings left in the enclave CPU pool, move to available. */ + for_each_cpu(cpu, ne_enclave->cpu_siblings) { + cpumask_clear_cpu(cpu, ne_enclave->cpu_siblings); + + cpumask_set_cpu(cpu, ne_cpu_pool.avail); + } + + free_cpumask_var(ne_enclave->cpu_siblings); + + mutex_unlock(&ne_cpu_pool_mutex); +} + +/** + * ne_pci_dev_remove_enclave_entry - Remove enclave entry from the data + * structure that is part of the PCI device private data. + * + * This function gets called with the ne_pci_dev enclave mutex held. + * + * @ne_enclave: private data associated with the current enclave. + * @ne_pci_dev: private data associated with the PCI device. + */ +static void ne_pci_dev_remove_enclave_entry(struct ne_enclave *ne_enclave, + struct ne_pci_dev *ne_pci_dev) +{ + struct ne_enclave *ne_enclave_entry = NULL; + struct ne_enclave *ne_enclave_entry_tmp = NULL; + + BUG_ON(!ne_enclave); + BUG_ON(!ne_pci_dev); + + list_for_each_entry_safe(ne_enclave_entry, ne_enclave_entry_tmp, + &ne_pci_dev->enclaves_list, + enclave_list_entry) { + if (ne_enclave_entry->slot_uid == ne_enclave->slot_uid) { + list_del(&ne_enclave_entry->enclave_list_entry); + + break; + } + } +} + static int ne_enclave_release(struct inode *inode, struct file *file) { + struct ne_pci_dev_cmd_reply cmd_reply = {}; + struct enclave_stop_req enclave_stop_request = {}; + struct ne_enclave *ne_enclave = file->private_data; + struct ne_pci_dev *ne_pci_dev = NULL; + int rc = -EINVAL; + struct slot_free_req slot_free_req = {}; + + BUG_ON(!ne_enclave); + BUG_ON(!ne_enclave->pdev); + + ne_pci_dev = pci_get_drvdata(ne_enclave->pdev); + BUG_ON(!ne_pci_dev); + + /* + * Acquire the enclave list mutex before the enclave mutex + * in order to avoid deadlocks with @ref ne_event_work_handler. + */ + mutex_lock(&ne_pci_dev->enclaves_list_mutex); + mutex_lock(&ne_enclave->enclave_info_mutex); + + if (ne_enclave->state != NE_STATE_INIT && + ne_enclave->state != NE_STATE_STOPPED) { + enclave_stop_request.slot_uid = ne_enclave->slot_uid; + + rc = ne_do_request(ne_enclave->pdev, ENCLAVE_STOP, + &enclave_stop_request, + sizeof(enclave_stop_request), &cmd_reply, + sizeof(cmd_reply)); + if (WARN_ON(rc < 0)) { + pr_err_ratelimited("Failure in enclave stop [rc=%d]\n", + rc); + + mutex_unlock(&ne_enclave->enclave_info_mutex); + mutex_unlock(&ne_pci_dev->enclaves_list_mutex); + + return rc; + } + + memset(&cmd_reply, 0, sizeof(cmd_reply)); + } + + slot_free_req.slot_uid = ne_enclave->slot_uid; + + rc = ne_do_request(ne_enclave->pdev, SLOT_FREE, &slot_free_req, + sizeof(slot_free_req), &cmd_reply, + sizeof(cmd_reply)); + if (WARN_ON(rc < 0)) { + pr_err_ratelimited("Failure in slot free [rc=%d]\n", rc); + + mutex_unlock(&ne_enclave->enclave_info_mutex); + mutex_unlock(&ne_pci_dev->enclaves_list_mutex); + + return rc; + } + + ne_pci_dev_remove_enclave_entry(ne_enclave, ne_pci_dev); + ne_enclave_remove_all_mem_region_entries(ne_enclave); + ne_enclave_remove_all_vcpu_id_entries(ne_enclave); + + mutex_unlock(&ne_enclave->enclave_info_mutex); + mutex_unlock(&ne_pci_dev->enclaves_list_mutex); + + kzfree(ne_enclave); + return 0; } From patchwork Tue Apr 21 18:41:48 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paraschiv, Andra-Irina" X-Patchwork-Id: 11502093 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 95B5281 for ; Tue, 21 Apr 2020 18:44:08 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 7EA07206D9 for ; Tue, 21 Apr 2020 18:44:08 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="iA7hzwtu" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729700AbgDUSoG (ORCPT ); Tue, 21 Apr 2020 14:44:06 -0400 Received: from smtp-fw-6002.amazon.com ([52.95.49.90]:58103 "EHLO smtp-fw-6002.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729679AbgDUSoE (ORCPT ); Tue, 21 Apr 2020 14:44:04 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1587494644; x=1619030644; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=D9HcJgxc4qqZhJGXAx5AE73mSnX8REqGb4FqR6zgOqg=; b=iA7hzwtuhPX+wtdKgKX2stjaZlpH74+uKnHzk+LVkEoUh7pvNmf1338+ 5Slvw5qoCVSdbspyN3aEjipGvYklSNZdtiGQZjmMdQQDNxlYX0vVBJS8r xT3wK0fY1+c9CJahdW1fUupz5pABLk0RJ4BuHCx0Ci17zamYFsMU8G5tD Y=; IronPort-SDR: E1OEtWh/TXxgF7Tm+5R6eNZav9pOKyWH2vFSqo7w69B4aYS+xleB6sm+8TgyeMqx5xKTYIUPES 43PM+H/wt78A== X-IronPort-AV: E=Sophos;i="5.72,411,1580774400"; d="scan'208";a="26614658" Received: from iad12-co-svc-p1-lb1-vlan3.amazon.com (HELO email-inbound-relay-2b-5bdc5131.us-west-2.amazon.com) ([10.43.8.6]) by smtp-border-fw-out-6002.iad6.amazon.com with ESMTP; 21 Apr 2020 18:44:03 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan3.pdx.amazon.com [10.170.41.166]) by email-inbound-relay-2b-5bdc5131.us-west-2.amazon.com (Postfix) with ESMTPS id BDDE7A1D33; Tue, 21 Apr 2020 18:44:01 +0000 (UTC) Received: from EX13D16EUB003.ant.amazon.com (10.43.166.99) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 21 Apr 2020 18:44:01 +0000 Received: from 38f9d34ed3b1.ant.amazon.com (10.43.161.217) by EX13D16EUB003.ant.amazon.com (10.43.166.99) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 21 Apr 2020 18:43:52 +0000 From: Andra Paraschiv To: CC: Anthony Liguori , Benjamin Herrenschmidt , Colm MacCarthaigh , Bjoern Doebel , David Woodhouse , Frank van der Linden , Alexander Graf , Martin Pohlack , Matt Wilson , Paolo Bonzini , Balbir Singh , Stewart Smith , Uwe Dannowski , , , Andra Paraschiv Subject: [PATCH v1 13/15] nitro_enclaves: Add Kconfig for the Nitro Enclaves driver Date: Tue, 21 Apr 2020 21:41:48 +0300 Message-ID: <20200421184150.68011-14-andraprs@amazon.com> X-Mailer: git-send-email 2.20.1 (Apple Git-117) In-Reply-To: <20200421184150.68011-1-andraprs@amazon.com> References: <20200421184150.68011-1-andraprs@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.161.217] X-ClientProxiedBy: EX13D17UWC002.ant.amazon.com (10.43.162.61) To EX13D16EUB003.ant.amazon.com (10.43.166.99) Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Signed-off-by: Andra Paraschiv --- drivers/virt/Kconfig | 2 ++ drivers/virt/amazon/Kconfig | 28 ++++++++++++++++++++++++++++ 2 files changed, 30 insertions(+) create mode 100644 drivers/virt/amazon/Kconfig diff --git a/drivers/virt/Kconfig b/drivers/virt/Kconfig index 363af2eaf2ba..06bb5cfa191d 100644 --- a/drivers/virt/Kconfig +++ b/drivers/virt/Kconfig @@ -32,4 +32,6 @@ config FSL_HV_MANAGER partition shuts down. source "drivers/virt/vboxguest/Kconfig" + +source "drivers/virt/amazon/Kconfig" endif diff --git a/drivers/virt/amazon/Kconfig b/drivers/virt/amazon/Kconfig new file mode 100644 index 000000000000..57fd0aa58803 --- /dev/null +++ b/drivers/virt/amazon/Kconfig @@ -0,0 +1,28 @@ +# SPDX-License-Identifier: GPL-2.0 +# +# Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. +# +# This program is free software; you can redistribute it and/or modify it +# under the terms and conditions of the GNU General Public License, +# version 2, as published by the Free Software Foundation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, see . + +# Amazon Nitro Enclaves (NE) support. +# Nitro is a hypervisor that has been developed by Amazon. + +config NITRO_ENCLAVES + tristate "Nitro Enclaves Support" + depends on HOTPLUG_CPU + ---help--- + This driver consists of support for enclave lifetime management + for Nitro Enclaves (NE). + + To compile this driver as a module, choose M here. + The module will be called nitro_enclaves. From patchwork Tue Apr 21 18:41:49 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paraschiv, Andra-Irina" X-Patchwork-Id: 11502095 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 24D7192C for ; Tue, 21 Apr 2020 18:44:17 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 0B24F2072D for ; Tue, 21 Apr 2020 18:44:17 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="jT63UHNW" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729699AbgDUSoN (ORCPT ); Tue, 21 Apr 2020 14:44:13 -0400 Received: from smtp-fw-6002.amazon.com ([52.95.49.90]:58124 "EHLO smtp-fw-6002.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729722AbgDUSoM (ORCPT ); Tue, 21 Apr 2020 14:44:12 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1587494651; x=1619030651; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=vLDZfBQ8IqS0q77AJUKPbfKbjqr2sjpRWf9gkII3JQU=; b=jT63UHNWSJ1BXd037dJTypzNDff68uYiHQfzr0VI/waWWvo4METynafQ eQWELpTeglry0AW/tK+2IyW1cRpnPixXFvdEkm1VLJlMZDmmVxbadJm9X h/nv1oJFT2uOlv71XQgP0XM2ePR/jLNPvu+KDuURdgvA4TBpvgsubjEXf Y=; IronPort-SDR: IwQSjZRIsD99Wa8PA+Gd1gho9PjRrq4JftpvLu82XFNO3SaHs14Qo8Ng/RWUjzgLOSGU72LQdL 7IennduioqVg== X-IronPort-AV: E=Sophos;i="5.72,411,1580774400"; d="scan'208";a="26614674" Received: from iad12-co-svc-p1-lb1-vlan3.amazon.com (HELO email-inbound-relay-2a-90c42d1d.us-west-2.amazon.com) ([10.43.8.6]) by smtp-border-fw-out-6002.iad6.amazon.com with ESMTP; 21 Apr 2020 18:44:10 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan3.pdx.amazon.com [10.170.41.166]) by email-inbound-relay-2a-90c42d1d.us-west-2.amazon.com (Postfix) with ESMTPS id 71BE6A1E6D; Tue, 21 Apr 2020 18:44:09 +0000 (UTC) Received: from EX13D16EUB003.ant.amazon.com (10.43.166.99) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 21 Apr 2020 18:44:08 +0000 Received: from 38f9d34ed3b1.ant.amazon.com (10.43.161.217) by EX13D16EUB003.ant.amazon.com (10.43.166.99) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 21 Apr 2020 18:44:00 +0000 From: Andra Paraschiv To: CC: Anthony Liguori , Benjamin Herrenschmidt , Colm MacCarthaigh , Bjoern Doebel , David Woodhouse , Frank van der Linden , Alexander Graf , Martin Pohlack , Matt Wilson , Paolo Bonzini , Balbir Singh , Stewart Smith , Uwe Dannowski , , , Andra Paraschiv Subject: [PATCH v1 14/15] nitro_enclaves: Add Makefile for the Nitro Enclaves driver Date: Tue, 21 Apr 2020 21:41:49 +0300 Message-ID: <20200421184150.68011-15-andraprs@amazon.com> X-Mailer: git-send-email 2.20.1 (Apple Git-117) In-Reply-To: <20200421184150.68011-1-andraprs@amazon.com> References: <20200421184150.68011-1-andraprs@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.161.217] X-ClientProxiedBy: EX13D17UWC002.ant.amazon.com (10.43.162.61) To EX13D16EUB003.ant.amazon.com (10.43.166.99) Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Signed-off-by: Andra Paraschiv --- drivers/virt/Makefile | 2 ++ drivers/virt/amazon/Makefile | 19 +++++++++++++++++ drivers/virt/amazon/nitro_enclaves/Makefile | 23 +++++++++++++++++++++ 3 files changed, 44 insertions(+) create mode 100644 drivers/virt/amazon/Makefile create mode 100644 drivers/virt/amazon/nitro_enclaves/Makefile diff --git a/drivers/virt/Makefile b/drivers/virt/Makefile index fd331247c27a..50a7d792e1c4 100644 --- a/drivers/virt/Makefile +++ b/drivers/virt/Makefile @@ -5,3 +5,5 @@ obj-$(CONFIG_FSL_HV_MANAGER) += fsl_hypervisor.o obj-y += vboxguest/ + +obj-$(CONFIG_NITRO_ENCLAVES) += amazon/ diff --git a/drivers/virt/amazon/Makefile b/drivers/virt/amazon/Makefile new file mode 100644 index 000000000000..9d77bbfc748a --- /dev/null +++ b/drivers/virt/amazon/Makefile @@ -0,0 +1,19 @@ +# SPDX-License-Identifier: GPL-2.0 +# +# Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. +# +# This program is free software; you can redistribute it and/or modify it +# under the terms and conditions of the GNU General Public License, +# version 2, as published by the Free Software Foundation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, see . + +# Enclave lifetime management support for Nitro Enclaves (NE). + +obj-$(CONFIG_NITRO_ENCLAVES) += nitro_enclaves/ diff --git a/drivers/virt/amazon/nitro_enclaves/Makefile b/drivers/virt/amazon/nitro_enclaves/Makefile new file mode 100644 index 000000000000..9109aed41070 --- /dev/null +++ b/drivers/virt/amazon/nitro_enclaves/Makefile @@ -0,0 +1,23 @@ +# SPDX-License-Identifier: GPL-2.0 +# +# Copyright 2020 Amazon.com, Inc. or its affiliates. All Rights Reserved. +# +# This program is free software; you can redistribute it and/or modify it +# under the terms and conditions of the GNU General Public License, +# version 2, as published by the Free Software Foundation. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, see . + +# Enclave lifetime management support for Nitro Enclaves (NE). + +obj-$(CONFIG_NITRO_ENCLAVES) += nitro_enclaves.o + +nitro_enclaves-y := ne_pci_dev.o ne_misc_dev.o + +ccflags-y += -Wall From patchwork Tue Apr 21 18:41:50 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Paraschiv, Andra-Irina" X-Patchwork-Id: 11502097 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1939992C for ; Tue, 21 Apr 2020 18:44:28 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 00CFF2068F for ; Tue, 21 Apr 2020 18:44:28 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="vn/TA35T" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729730AbgDUSoY (ORCPT ); Tue, 21 Apr 2020 14:44:24 -0400 Received: from smtp-fw-33001.amazon.com ([207.171.190.10]:62594 "EHLO smtp-fw-33001.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1729704AbgDUSoX (ORCPT ); Tue, 21 Apr 2020 14:44:23 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1587494663; x=1619030663; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=x7e23kFiXkvUBqIe8e1beT+ixqa5FhXJH3ogwFK7OSU=; b=vn/TA35TOdoOp+o/iwsowk/an8xoEc6JG1no/UuRS3QPLdcbHO63sW6t sECpUucUJTm2x8rfcrz03MHfgjEYc++7mZa4kRkizUHQFWn4Z2Tt9Y8J9 Ww7LK0W14kzwrOhUZ/UfpoQYmFdPH+My0Fbxb9eJ+FNuadkPpYA+zls6/ Y=; IronPort-SDR: 6GB3xe8ItpdTWMUTW+vMRC6TZZZLRue50FgLVl+Eux87Zm7497NnDu+ILeFy356WlBd/pnb4dv tSyjFZ/rAolw== X-IronPort-AV: E=Sophos;i="5.72,411,1580774400"; d="scan'208";a="39978688" Received: from sea32-co-svc-lb4-vlan3.sea.corp.amazon.com (HELO email-inbound-relay-2b-4ff6265a.us-west-2.amazon.com) ([10.47.23.38]) by smtp-border-fw-out-33001.sea14.amazon.com with ESMTP; 21 Apr 2020 18:44:23 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan3.pdx.amazon.com [10.170.41.166]) by email-inbound-relay-2b-4ff6265a.us-west-2.amazon.com (Postfix) with ESMTPS id 456D0A277D; Tue, 21 Apr 2020 18:44:22 +0000 (UTC) Received: from EX13D16EUB003.ant.amazon.com (10.43.166.99) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 21 Apr 2020 18:44:21 +0000 Received: from 38f9d34ed3b1.ant.amazon.com (10.43.162.148) by EX13D16EUB003.ant.amazon.com (10.43.166.99) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 21 Apr 2020 18:44:13 +0000 From: Andra Paraschiv To: CC: Anthony Liguori , Benjamin Herrenschmidt , Colm MacCarthaigh , Bjoern Doebel , David Woodhouse , Frank van der Linden , Alexander Graf , Martin Pohlack , Matt Wilson , Paolo Bonzini , Balbir Singh , Stewart Smith , Uwe Dannowski , , , Andra Paraschiv Subject: [PATCH v1 15/15] MAINTAINERS: Add entry for the Nitro Enclaves driver Date: Tue, 21 Apr 2020 21:41:50 +0300 Message-ID: <20200421184150.68011-16-andraprs@amazon.com> X-Mailer: git-send-email 2.20.1 (Apple Git-117) In-Reply-To: <20200421184150.68011-1-andraprs@amazon.com> References: <20200421184150.68011-1-andraprs@amazon.com> MIME-Version: 1.0 X-Originating-IP: [10.43.162.148] X-ClientProxiedBy: EX13D08UWB002.ant.amazon.com (10.43.161.168) To EX13D16EUB003.ant.amazon.com (10.43.166.99) Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Signed-off-by: Andra Paraschiv --- MAINTAINERS | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/MAINTAINERS b/MAINTAINERS index b816a453b10e..9625fadbd400 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -11956,6 +11956,17 @@ S: Maintained T: git git://git.kernel.org/pub/scm/linux/kernel/git/lftan/nios2.git F: arch/nios2/ +NITRO ENCLAVES (NE) +M: Andra Paraschiv +M: Alexandru Vasile +M: Alexandru Ciobotaru +L: linux-kernel@vger.kernel.org +S: Supported +W: https://aws.amazon.com/ec2/nitro/nitro-enclaves/ +F: include/linux/nitro_enclaves.h +F: include/uapi/linux/nitro_enclaves.h +F: drivers/virt/amazon/nitro_enclaves/ + NOHZ, DYNTICKS SUPPORT M: Frederic Weisbecker M: Thomas Gleixner