From patchwork Mon Oct 11 11:15:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Zhong X-Patchwork-Id: 12549721 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 89F83C433F5 for ; Mon, 11 Oct 2021 11:24:25 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id F2A0660EB1 for ; Mon, 11 Oct 2021 11:24:24 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org F2A0660EB1 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=intel.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:48480 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mZtPn-0004AH-7V for qemu-devel@archiver.kernel.org; Mon, 11 Oct 2021 07:24:20 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:56526) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mZtNJ-0001Ty-QJ for qemu-devel@nongnu.org; Mon, 11 Oct 2021 07:21:45 -0400 Received: from mga04.intel.com ([192.55.52.120]:28238) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mZtNB-00080y-1i for qemu-devel@nongnu.org; Mon, 11 Oct 2021 07:21:45 -0400 X-IronPort-AV: E=McAfee;i="6200,9189,10133"; a="225624843" X-IronPort-AV: E=Sophos;i="5.85,364,1624345200"; d="scan'208";a="225624843" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Oct 2021 04:21:30 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.85,364,1624345200"; d="scan'208";a="490432119" Received: from icx-2s.bj.intel.com ([10.240.193.41]) by orsmga008.jf.intel.com with ESMTP; 11 Oct 2021 04:21:29 -0700 From: Yang Zhong To: qemu-devel@nongnu.org Subject: [PATCH 1/6] numa: Enable numa for SGX EPC sections Date: Mon, 11 Oct 2021 19:15:49 +0800 Message-Id: <20211011111554.12403-2-yang.zhong@intel.com> X-Mailer: git-send-email 2.29.2.334.gfaefdd61ec In-Reply-To: <20211011111554.12403-1-yang.zhong@intel.com> References: <20211011111554.12403-1-yang.zhong@intel.com> MIME-Version: 1.0 Received-SPF: pass client-ip=192.55.52.120; envelope-from=yang.zhong@intel.com; helo=mga04.intel.com X-Spam_score_int: -22 X-Spam_score: -2.3 X-Spam_bar: -- X-Spam_report: (-2.3 / 5.0 requ) RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yang.zhong@intel.com, pbonzini@redhat.com, eblake@redhat.com, philmd@redhat.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" The basic SGX did not enable numa for SGX EPC sections, which result in all EPC sections located in numa node 0. This patch enable SGX numa function in the guest and the EPC section can work with RAM as one numa node. The Guest kernel related log: [ 0.009981] ACPI: SRAT: Node 0 PXM 0 [mem 0x180000000-0x183ffffff] [ 0.009982] ACPI: SRAT: Node 1 PXM 1 [mem 0x184000000-0x185bfffff] The SRAT table can normally show SGX EPC sections menory info in different numa nodes. The SGX EPC numa related command: ...... -m 4G,maxmem=20G \ -smp sockets=2,cores=2 \ -cpu host,+sgx-provisionkey \ -object memory-backend-ram,size=2G,host-nodes=0,policy=bind,id=node0 \ -object memory-backend-epc,id=mem0,size=64M,prealloc=on,host-nodes=0,policy=bind \ -numa node,nodeid=0,cpus=0-1,memdev=node0 \ -object memory-backend-ram,size=2G,host-nodes=1,policy=bind,id=node1 \ -object memory-backend-epc,id=mem1,size=28M,prealloc=on,host-nodes=1,policy=bind \ -numa node,nodeid=1,cpus=2-3,memdev=node1 \ -M sgx-epc.0.memdev=mem0,sgx-epc.0.node=0,sgx-epc.1.memdev=mem1,sgx-epc.1.node=1 \ ...... Signed-off-by: Yang Zhong --- qapi/machine.json | 6 +++++- include/hw/i386/sgx-epc.h | 3 +++ hw/i386/acpi-build.c | 4 ++++ hw/i386/sgx-epc.c | 3 +++ hw/i386/sgx.c | 44 +++++++++++++++++++++++++++++++++++++++ monitor/hmp-cmds.c | 1 + qemu-options.hx | 4 ++-- 7 files changed, 62 insertions(+), 3 deletions(-) diff --git a/qapi/machine.json b/qapi/machine.json index 5db54df298..09b6188e6f 100644 --- a/qapi/machine.json +++ b/qapi/machine.json @@ -1213,6 +1213,7 @@ 'data': { '*id': 'str', 'memaddr': 'size', 'size': 'size', + 'node': 'int', 'memdev': 'str' } } @@ -1288,7 +1289,10 @@ # Since: 6.2 ## { 'struct': 'SgxEPC', - 'data': { 'memdev': 'str' } } + 'data': { 'memdev': 'str', + 'node': 'int' + } +} ## # @SgxEPCProperties: diff --git a/include/hw/i386/sgx-epc.h b/include/hw/i386/sgx-epc.h index 65a68ca753..7a61c52869 100644 --- a/include/hw/i386/sgx-epc.h +++ b/include/hw/i386/sgx-epc.h @@ -25,6 +25,7 @@ #define SGX_EPC_ADDR_PROP "addr" #define SGX_EPC_SIZE_PROP "size" #define SGX_EPC_MEMDEV_PROP "memdev" +#define SGX_EPC_NUMA_NODE_PROP "node" /** * SGXEPCDevice: @@ -38,6 +39,7 @@ typedef struct SGXEPCDevice { /* public */ uint64_t addr; + uint32_t node; HostMemoryBackendEpc *hostmem; } SGXEPCDevice; @@ -56,6 +58,7 @@ typedef struct SGXEPCState { } SGXEPCState; int sgx_epc_get_section(int section_nr, uint64_t *addr, uint64_t *size); +void sgx_epc_build_srat(GArray *table_data); static inline uint64_t sgx_epc_above_4g_end(SGXEPCState *sgx_epc) { diff --git a/hw/i386/acpi-build.c b/hw/i386/acpi-build.c index 81418b7911..563a38992f 100644 --- a/hw/i386/acpi-build.c +++ b/hw/i386/acpi-build.c @@ -2062,6 +2062,10 @@ build_srat(GArray *table_data, BIOSLinker *linker, MachineState *machine) nvdimm_build_srat(table_data); } + if (pcms->sgx_epc.size != 0) { + sgx_epc_build_srat(table_data); + } + /* * TODO: this part is not in ACPI spec and current linux kernel boots fine * without these entries. But I recall there were issues the last time I diff --git a/hw/i386/sgx-epc.c b/hw/i386/sgx-epc.c index 55e2217eae..e5cd2789be 100644 --- a/hw/i386/sgx-epc.c +++ b/hw/i386/sgx-epc.c @@ -21,6 +21,7 @@ static Property sgx_epc_properties[] = { DEFINE_PROP_UINT64(SGX_EPC_ADDR_PROP, SGXEPCDevice, addr, 0), + DEFINE_PROP_UINT32(SGX_EPC_NUMA_NODE_PROP, SGXEPCDevice, node, 0), DEFINE_PROP_LINK(SGX_EPC_MEMDEV_PROP, SGXEPCDevice, hostmem, TYPE_MEMORY_BACKEND_EPC, HostMemoryBackendEpc *), DEFINE_PROP_END_OF_LIST(), @@ -139,6 +140,8 @@ static void sgx_epc_md_fill_device_info(const MemoryDeviceState *md, se->memaddr = epc->addr; se->size = object_property_get_uint(OBJECT(epc), SGX_EPC_SIZE_PROP, NULL); + se->node = object_property_get_uint(OBJECT(epc), SGX_EPC_NUMA_NODE_PROP, + NULL); se->memdev = object_get_canonical_path(OBJECT(epc->hostmem)); info->u.sgx_epc.data = se; diff --git a/hw/i386/sgx.c b/hw/i386/sgx.c index e481e9358f..906facb645 100644 --- a/hw/i386/sgx.c +++ b/hw/i386/sgx.c @@ -19,6 +19,7 @@ #include "exec/address-spaces.h" #include "hw/i386/sgx.h" #include "sysemu/hw_accel.h" +#include "hw/acpi/aml-build.h" #define SGX_MAX_EPC_SECTIONS 8 #define SGX_CPUID_EPC_INVALID 0x0 @@ -27,6 +28,46 @@ #define SGX_CPUID_EPC_SECTION 0x1 #define SGX_CPUID_EPC_MASK 0xF +static int sgx_epc_device_list(Object *obj, void *opaque) +{ + GSList **list = opaque; + + if (object_dynamic_cast(obj, TYPE_SGX_EPC)) { + *list = g_slist_append(*list, DEVICE(obj)); + } + + object_child_foreach(obj, sgx_epc_device_list, opaque); + return 0; +} + +static GSList *sgx_epc_get_device_list(void) +{ + GSList *list = NULL; + + object_child_foreach(qdev_get_machine(), sgx_epc_device_list, &list); + return list; +} + +void sgx_epc_build_srat(GArray *table_data) +{ + GSList *device_list = sgx_epc_get_device_list(); + + for (; device_list; device_list = device_list->next) { + DeviceState *dev = device_list->data; + Object *obj = OBJECT(dev); + uint64_t addr, size; + int node; + + node = object_property_get_uint(obj, SGX_EPC_NUMA_NODE_PROP, + &error_abort); + addr = object_property_get_uint(obj, SGX_EPC_ADDR_PROP, &error_abort); + size = object_property_get_uint(obj, SGX_EPC_SIZE_PROP, &error_abort); + + build_srat_memory(table_data, addr, size, node, MEM_AFFINITY_ENABLED); + } + g_slist_free(device_list); +} + static uint64_t sgx_calc_section_metric(uint64_t low, uint64_t high) { return (low & MAKE_64BIT_MASK(12, 20)) + @@ -156,6 +197,9 @@ void pc_machine_init_sgx_epc(PCMachineState *pcms) /* set the memdev link with memory backend */ object_property_parse(obj, SGX_EPC_MEMDEV_PROP, list->value->memdev, &error_fatal); + /* set the numa node property for sgx epc object */ + object_property_set_uint(obj, SGX_EPC_NUMA_NODE_PROP, list->value->node, + &error_fatal); object_property_set_bool(obj, "realized", true, &error_fatal); object_unref(obj); } diff --git a/monitor/hmp-cmds.c b/monitor/hmp-cmds.c index bcaa41350e..8af26e3e20 100644 --- a/monitor/hmp-cmds.c +++ b/monitor/hmp-cmds.c @@ -1878,6 +1878,7 @@ void hmp_info_memory_devices(Monitor *mon, const QDict *qdict) se->id ? se->id : ""); monitor_printf(mon, " memaddr: 0x%" PRIx64 "\n", se->memaddr); monitor_printf(mon, " size: %" PRIu64 "\n", se->size); + monitor_printf(mon, " node: %" PRId64 "\n", se->node); monitor_printf(mon, " memdev: %s\n", se->memdev); break; default: diff --git a/qemu-options.hx b/qemu-options.hx index 5f375bbfa6..aaa5a1926d 100644 --- a/qemu-options.hx +++ b/qemu-options.hx @@ -127,11 +127,11 @@ SRST ERST DEF("M", HAS_ARG, QEMU_OPTION_M, - " sgx-epc.0.memdev=memid\n", + " sgx-epc.0.memdev=memid,sgx-epc.0.node=numaid\n", QEMU_ARCH_ALL) SRST -``sgx-epc.0.memdev=@var{memid}`` +``sgx-epc.0.memdev=@var{memid},sgx-epc.0.node=@var{numaid}`` Define an SGX EPC section. ERST From patchwork Mon Oct 11 11:15:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Zhong X-Patchwork-Id: 12549719 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 35D63C433EF for ; Mon, 11 Oct 2021 11:24:19 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id EC95360C49 for ; Mon, 11 Oct 2021 11:24:18 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org EC95360C49 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=intel.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:48442 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mZtPl-000497-C7 for qemu-devel@archiver.kernel.org; Mon, 11 Oct 2021 07:24:17 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:56508) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mZtNE-0001TH-Ln for qemu-devel@nongnu.org; Mon, 11 Oct 2021 07:21:40 -0400 Received: from mga04.intel.com ([192.55.52.120]:28240) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mZtNB-0008Kp-1g for qemu-devel@nongnu.org; Mon, 11 Oct 2021 07:21:39 -0400 X-IronPort-AV: E=McAfee;i="6200,9189,10133"; a="225624851" X-IronPort-AV: E=Sophos;i="5.85,364,1624345200"; d="scan'208";a="225624851" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Oct 2021 04:21:32 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.85,364,1624345200"; d="scan'208";a="490432152" Received: from icx-2s.bj.intel.com ([10.240.193.41]) by orsmga008.jf.intel.com with ESMTP; 11 Oct 2021 04:21:30 -0700 From: Yang Zhong To: qemu-devel@nongnu.org Subject: [PATCH 2/6] monitor: Support 'info numa' command Date: Mon, 11 Oct 2021 19:15:50 +0800 Message-Id: <20211011111554.12403-3-yang.zhong@intel.com> X-Mailer: git-send-email 2.29.2.334.gfaefdd61ec In-Reply-To: <20211011111554.12403-1-yang.zhong@intel.com> References: <20211011111554.12403-1-yang.zhong@intel.com> MIME-Version: 1.0 Received-SPF: pass client-ip=192.55.52.120; envelope-from=yang.zhong@intel.com; helo=mga04.intel.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yang.zhong@intel.com, pbonzini@redhat.com, eblake@redhat.com, philmd@redhat.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Add the MEMORY_DEVICE_INFO_KIND_SGX_EPC case for SGX numa info with 'info numa' command in the monitor. Signed-off-by: Yang Zhong --- hw/core/numa.c | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/hw/core/numa.c b/hw/core/numa.c index 510d096a88..1aa05dcf42 100644 --- a/hw/core/numa.c +++ b/hw/core/numa.c @@ -756,6 +756,7 @@ static void numa_stat_memory_devices(NumaNodeMem node_mem[]) PCDIMMDeviceInfo *pcdimm_info; VirtioPMEMDeviceInfo *vpi; VirtioMEMDeviceInfo *vmi; + SgxEPCDeviceInfo *se; for (info = info_list; info; info = info->next) { MemoryDeviceInfo *value = info->value; @@ -781,6 +782,11 @@ static void numa_stat_memory_devices(NumaNodeMem node_mem[]) node_mem[vmi->node].node_mem += vmi->size; node_mem[vmi->node].node_plugged_mem += vmi->size; break; + case MEMORY_DEVICE_INFO_KIND_SGX_EPC: + se = value->u.sgx_epc.data; + node_mem[se->node].node_mem += se->size; + node_mem[se->node].node_plugged_mem = 0; + break; default: g_assert_not_reached(); } From patchwork Mon Oct 11 11:15:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Zhong X-Patchwork-Id: 12549727 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BE9C7C433F5 for ; Mon, 11 Oct 2021 11:26:58 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7D28560EB1 for ; Mon, 11 Oct 2021 11:26:58 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 7D28560EB1 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=intel.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:55168 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mZtSL-0000Hj-FL for qemu-devel@archiver.kernel.org; Mon, 11 Oct 2021 07:26:57 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:56524) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mZtNJ-0001Tw-PN for qemu-devel@nongnu.org; Mon, 11 Oct 2021 07:21:45 -0400 Received: from mga04.intel.com ([192.55.52.120]:28240) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mZtNE-0008Kp-Ij for qemu-devel@nongnu.org; Mon, 11 Oct 2021 07:21:41 -0400 X-IronPort-AV: E=McAfee;i="6200,9189,10133"; a="225624856" X-IronPort-AV: E=Sophos;i="5.85,364,1624345200"; d="scan'208";a="225624856" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Oct 2021 04:21:34 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.85,364,1624345200"; d="scan'208";a="490432173" Received: from icx-2s.bj.intel.com ([10.240.193.41]) by orsmga008.jf.intel.com with ESMTP; 11 Oct 2021 04:21:32 -0700 From: Yang Zhong To: qemu-devel@nongnu.org Subject: [PATCH 3/6] numa: Add SGXEPCSection list for multiple sections Date: Mon, 11 Oct 2021 19:15:51 +0800 Message-Id: <20211011111554.12403-4-yang.zhong@intel.com> X-Mailer: git-send-email 2.29.2.334.gfaefdd61ec In-Reply-To: <20211011111554.12403-1-yang.zhong@intel.com> References: <20211011111554.12403-1-yang.zhong@intel.com> MIME-Version: 1.0 Received-SPF: pass client-ip=192.55.52.120; envelope-from=yang.zhong@intel.com; helo=mga04.intel.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yang.zhong@intel.com, pbonzini@redhat.com, eblake@redhat.com, philmd@redhat.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" The SGXEPCSection list added into SGXInfo to show the multiple SGX EPC sections detailed info, not the total size like before. Signed-off-by: Yang Zhong --- qapi/misc-target.json | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/qapi/misc-target.json b/qapi/misc-target.json index 594fbd1577..89a5a4250a 100644 --- a/qapi/misc-target.json +++ b/qapi/misc-target.json @@ -334,6 +334,21 @@ 'returns': 'SevAttestationReport', 'if': 'TARGET_I386' } +## +# @SGXEPCSection: +# +# Information about intel SGX EPC section info +# +# @index: the SGX epc section index +# +# @size: the size of epc section +# +# Since: 6.2 +## +{ 'struct': 'SGXEPCSection', + 'data': { 'index': 'uint64', + 'size': 'uint64'}} + ## # @SGXInfo: # @@ -347,7 +362,7 @@ # # @flc: true if FLC is supported # -# @section-size: The EPC section size for guest +# @sections: The EPC sections info for guest # # Since: 6.2 ## @@ -356,7 +371,7 @@ 'sgx1': 'bool', 'sgx2': 'bool', 'flc': 'bool', - 'section-size': 'uint64'}, + 'sections': ['SGXEPCSection']}, 'if': 'TARGET_I386' } ## From patchwork Mon Oct 11 11:15:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Zhong X-Patchwork-Id: 12549729 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D227EC433EF for ; Mon, 11 Oct 2021 11:27:00 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 9968960EB1 for ; Mon, 11 Oct 2021 11:27:00 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 9968960EB1 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=intel.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:55152 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mZtSL-0000H6-GN for qemu-devel@archiver.kernel.org; Mon, 11 Oct 2021 07:26:57 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:56564) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mZtNL-0001Wh-DD for qemu-devel@nongnu.org; Mon, 11 Oct 2021 07:21:47 -0400 Received: from mga04.intel.com ([192.55.52.120]:28236) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mZtNJ-0007q1-JO for qemu-devel@nongnu.org; Mon, 11 Oct 2021 07:21:47 -0400 X-IronPort-AV: E=McAfee;i="6200,9189,10133"; a="225624865" X-IronPort-AV: E=Sophos;i="5.85,364,1624345200"; d="scan'208";a="225624865" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Oct 2021 04:21:35 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.85,364,1624345200"; d="scan'208";a="490432195" Received: from icx-2s.bj.intel.com ([10.240.193.41]) by orsmga008.jf.intel.com with ESMTP; 11 Oct 2021 04:21:34 -0700 From: Yang Zhong To: qemu-devel@nongnu.org Subject: [PATCH 4/6] monitor: numa support for 'info sgx' command Date: Mon, 11 Oct 2021 19:15:52 +0800 Message-Id: <20211011111554.12403-5-yang.zhong@intel.com> X-Mailer: git-send-email 2.29.2.334.gfaefdd61ec In-Reply-To: <20211011111554.12403-1-yang.zhong@intel.com> References: <20211011111554.12403-1-yang.zhong@intel.com> MIME-Version: 1.0 Received-SPF: pass client-ip=192.55.52.120; envelope-from=yang.zhong@intel.com; helo=mga04.intel.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yang.zhong@intel.com, pbonzini@redhat.com, eblake@redhat.com, philmd@redhat.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" This patch can enable numa support for 'info sgx' command in the monitor, which can show detailed SGX EPC sections info. (qemu) info sgx SGX support: enabled SGX1 support: enabled SGX2 support: enabled FLC support: enabled SECTION #0: size=67108864 SECTION #1: size=29360128 The QMP interface show: (QEMU) query-sgx {"return": {"sgx": true, "sgx2": true, "sgx1": true, "sections": \ [{"index": 0, "size": 67108864}, {"index": 1, "size": 29360128}], "flc": true}} Signed-off-by: Yang Zhong --- hw/i386/sgx.c | 25 +++++++++++++++++++++++-- target/i386/monitor.c | 11 +++++++++-- 2 files changed, 32 insertions(+), 4 deletions(-) diff --git a/hw/i386/sgx.c b/hw/i386/sgx.c index 906facb645..8af45925c6 100644 --- a/hw/i386/sgx.c +++ b/hw/i386/sgx.c @@ -126,6 +126,28 @@ SGXInfo *sgx_get_capabilities(Error **errp) return info; } +static SGXEPCSectionList *sgx_get_epc_sections_list(void) +{ + GSList *device_list = sgx_epc_get_device_list(); + SGXEPCSectionList *head = NULL, **tail = &head; + SGXEPCSection *section; + uint64_t i = 0; + + for (; device_list; device_list = device_list->next) { + DeviceState *dev = device_list->data; + Object *obj = OBJECT(dev); + + section = g_new0(SGXEPCSection, 1); + section->index = i++; + section->size = object_property_get_uint(obj, SGX_EPC_SIZE_PROP, + &error_abort); + QAPI_LIST_APPEND(tail, section); + } + g_slist_free(device_list); + + return head; +} + SGXInfo *sgx_get_info(Error **errp) { SGXInfo *info = NULL; @@ -144,14 +166,13 @@ SGXInfo *sgx_get_info(Error **errp) return NULL; } - SGXEPCState *sgx_epc = &pcms->sgx_epc; info = g_new0(SGXInfo, 1); info->sgx = true; info->sgx1 = true; info->sgx2 = true; info->flc = true; - info->section_size = sgx_epc->size; + info->sections = sgx_get_epc_sections_list(); return info; } diff --git a/target/i386/monitor.c b/target/i386/monitor.c index 196c1c9e77..08e7d4a425 100644 --- a/target/i386/monitor.c +++ b/target/i386/monitor.c @@ -773,6 +773,7 @@ SGXInfo *qmp_query_sgx(Error **errp) void hmp_info_sgx(Monitor *mon, const QDict *qdict) { Error *err = NULL; + SGXEPCSectionList *section_list, *section; g_autoptr(SGXInfo) info = qmp_query_sgx(&err); if (err) { @@ -787,8 +788,14 @@ void hmp_info_sgx(Monitor *mon, const QDict *qdict) info->sgx2 ? "enabled" : "disabled"); monitor_printf(mon, "FLC support: %s\n", info->flc ? "enabled" : "disabled"); - monitor_printf(mon, "size: %" PRIu64 "\n", - info->section_size); + + section_list = info->sections; + for (section = section_list; section; section = section->next) { + monitor_printf(mon, "SECTION #%" PRId64 ": ", + section->value->index); + monitor_printf(mon, "size=%" PRIu64 "\n", + section->value->size); + } } SGXInfo *qmp_query_sgx_capabilities(Error **errp) From patchwork Mon Oct 11 11:15:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Zhong X-Patchwork-Id: 12549725 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B8C7FC433EF for ; Mon, 11 Oct 2021 11:24:37 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 6F10B60C49 for ; Mon, 11 Oct 2021 11:24:37 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 6F10B60C49 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=intel.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:48912 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mZtQ4-0004SI-CM for qemu-devel@archiver.kernel.org; Mon, 11 Oct 2021 07:24:36 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:56584) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mZtNM-0001bU-Mc for qemu-devel@nongnu.org; Mon, 11 Oct 2021 07:21:48 -0400 Received: from mga04.intel.com ([192.55.52.120]:28240) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mZtNK-0008Kp-54 for qemu-devel@nongnu.org; Mon, 11 Oct 2021 07:21:48 -0400 X-IronPort-AV: E=McAfee;i="6200,9189,10133"; a="225624872" X-IronPort-AV: E=Sophos;i="5.85,364,1624345200"; d="scan'208";a="225624872" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Oct 2021 04:21:37 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.85,364,1624345200"; d="scan'208";a="490432226" Received: from icx-2s.bj.intel.com ([10.240.193.41]) by orsmga008.jf.intel.com with ESMTP; 11 Oct 2021 04:21:36 -0700 From: Yang Zhong To: qemu-devel@nongnu.org Subject: [PATCH 5/6] numa: Enable numa for libvirt interface Date: Mon, 11 Oct 2021 19:15:53 +0800 Message-Id: <20211011111554.12403-6-yang.zhong@intel.com> X-Mailer: git-send-email 2.29.2.334.gfaefdd61ec In-Reply-To: <20211011111554.12403-1-yang.zhong@intel.com> References: <20211011111554.12403-1-yang.zhong@intel.com> MIME-Version: 1.0 Received-SPF: pass client-ip=192.55.52.120; envelope-from=yang.zhong@intel.com; helo=mga04.intel.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yang.zhong@intel.com, pbonzini@redhat.com, eblake@redhat.com, philmd@redhat.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Libvirt need get the detailed host SGX EPC capabilities to support numa function. Libvirt can decide how to allocate host EPC sections to guest numa from host numa info. (QEMU) query-sgx-capabilities {"return": {"sgx": true, "sgx2": true, "sgx1": true, "sections": \ [{"index": 0, "size": 17070817280}, {"index": 1, "size": 17079205888}], "flc": true}} Signed-off-by: Yang Zhong --- hw/i386/sgx.c | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/hw/i386/sgx.c b/hw/i386/sgx.c index 8af45925c6..fe3034060d 100644 --- a/hw/i386/sgx.c +++ b/hw/i386/sgx.c @@ -74,11 +74,13 @@ static uint64_t sgx_calc_section_metric(uint64_t low, uint64_t high) ((high & MAKE_64BIT_MASK(0, 20)) << 32); } -static uint64_t sgx_calc_host_epc_section_size(void) +static SGXEPCSectionList *sgx_calc_host_epc_sections(void) { + SGXEPCSectionList *head = NULL, **tail = &head; + SGXEPCSection *section; uint32_t i, type; uint32_t eax, ebx, ecx, edx; - uint64_t size = 0; + uint32_t j = 0; for (i = 0; i < SGX_MAX_EPC_SECTIONS; i++) { host_cpuid(0x12, i + 2, &eax, &ebx, &ecx, &edx); @@ -92,10 +94,13 @@ static uint64_t sgx_calc_host_epc_section_size(void) break; } - size += sgx_calc_section_metric(ecx, edx); + section = g_new0(SGXEPCSection, 1); + section->index = j++; + section->size = sgx_calc_section_metric(ecx, edx); + QAPI_LIST_APPEND(tail, section); } - return size; + return head; } SGXInfo *sgx_get_capabilities(Error **errp) @@ -119,7 +124,7 @@ SGXInfo *sgx_get_capabilities(Error **errp) info->sgx1 = eax & (1U << 0) ? true : false; info->sgx2 = eax & (1U << 1) ? true : false; - info->section_size = sgx_calc_host_epc_section_size(); + info->sections = sgx_calc_host_epc_sections(); close(fd); From patchwork Mon Oct 11 11:15:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Zhong X-Patchwork-Id: 12549761 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EBEDCC433EF for ; Mon, 11 Oct 2021 11:29:00 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7D74560EB1 for ; Mon, 11 Oct 2021 11:29:00 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 7D74560EB1 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=intel.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:60404 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mZtUH-0003nh-PP for qemu-devel@archiver.kernel.org; Mon, 11 Oct 2021 07:28:59 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:56572) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mZtNL-0001Ya-Sy for qemu-devel@nongnu.org; Mon, 11 Oct 2021 07:21:47 -0400 Received: from mga04.intel.com ([192.55.52.120]:28238) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mZtNK-00080y-56 for qemu-devel@nongnu.org; Mon, 11 Oct 2021 07:21:47 -0400 X-IronPort-AV: E=McAfee;i="6200,9189,10133"; a="225624876" X-IronPort-AV: E=Sophos;i="5.85,364,1624345200"; d="scan'208";a="225624876" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 11 Oct 2021 04:21:39 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.85,364,1624345200"; d="scan'208";a="490432244" Received: from icx-2s.bj.intel.com ([10.240.193.41]) by orsmga008.jf.intel.com with ESMTP; 11 Oct 2021 04:21:37 -0700 From: Yang Zhong To: qemu-devel@nongnu.org Subject: [PATCH 6/6] doc: Add the SGX numa description Date: Mon, 11 Oct 2021 19:15:54 +0800 Message-Id: <20211011111554.12403-7-yang.zhong@intel.com> X-Mailer: git-send-email 2.29.2.334.gfaefdd61ec In-Reply-To: <20211011111554.12403-1-yang.zhong@intel.com> References: <20211011111554.12403-1-yang.zhong@intel.com> MIME-Version: 1.0 Received-SPF: pass client-ip=192.55.52.120; envelope-from=yang.zhong@intel.com; helo=mga04.intel.com X-Spam_score_int: -41 X-Spam_score: -4.2 X-Spam_bar: ---- X-Spam_report: (-4.2 / 5.0 requ) BAYES_00=-1.9, RCVD_IN_DNSWL_MED=-2.3, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: yang.zhong@intel.com, pbonzini@redhat.com, eblake@redhat.com, philmd@redhat.com Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" Add the SGX numa reference command and how to check if SGX numa is support or not with multiple EPC sections. Signed-off-by: Yang Zhong --- docs/system/i386/sgx.rst | 31 +++++++++++++++++++++++++++---- 1 file changed, 27 insertions(+), 4 deletions(-) diff --git a/docs/system/i386/sgx.rst b/docs/system/i386/sgx.rst index f103ae2a2f..9e4ada761f 100644 --- a/docs/system/i386/sgx.rst +++ b/docs/system/i386/sgx.rst @@ -141,8 +141,7 @@ To launch a SGX guest: |qemu_system_x86| \\ -cpu host,+sgx-provisionkey \\ -object memory-backend-epc,id=mem1,size=64M,prealloc=on \\ - -object memory-backend-epc,id=mem2,size=28M \\ - -M sgx-epc.0.memdev=mem1,sgx-epc.1.memdev=mem2 + -M sgx-epc.0.memdev=mem1,sgx-epc.0.node=0 Utilizing SGX in the guest requires a kernel/OS with SGX support. The support can be determined in guest by:: @@ -152,8 +151,32 @@ The support can be determined in guest by:: and SGX epc info by:: $ dmesg | grep sgx - [ 1.242142] sgx: EPC section 0x180000000-0x181bfffff - [ 1.242319] sgx: EPC section 0x181c00000-0x1837fffff + [ 0.182807] sgx: EPC section 0x140000000-0x143ffffff + [ 0.183695] sgx: [Firmware Bug]: Unable to map EPC section to online node. Fallback to the NUMA node 0. + +To launch a SGX numa guest: + +.. parsed-literal:: + + |qemu_system_x86| \\ + -cpu host,+sgx-provisionkey \\ + -object memory-backend-ram,size=2G,host-nodes=0,policy=bind,id=node0 \\ + -object memory-backend-epc,id=mem0,size=64M,prealloc=on,host-nodes=0,policy=bind \\ + -numa node,nodeid=0,cpus=0-1,memdev=node0 \\ + -object memory-backend-ram,size=2G,host-nodes=1,policy=bind,id=node1 \\ + -object memory-backend-epc,id=mem1,size=28M,prealloc=on,host-nodes=1,policy=bind \\ + -numa node,nodeid=1,cpus=2-3,memdev=node1 \\ + -M sgx-epc.0.memdev=mem0,sgx-epc.0.node=0,sgx-epc.1.memdev=mem1,sgx-epc.1.node=1 + +and SGX epc numa info by:: + + $ dmesg | grep sgx + [ 0.369937] sgx: EPC section 0x180000000-0x183ffffff + [ 0.370259] sgx: EPC section 0x184000000-0x185bfffff + + $ dmesg | grep SRAT + [ 0.009981] ACPI: SRAT: Node 0 PXM 0 [mem 0x180000000-0x183ffffff] + [ 0.009982] ACPI: SRAT: Node 1 PXM 1 [mem 0x184000000-0x185bfffff] References ----------