From patchwork Wed Sep 29 22:52:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Tyshchenko X-Patchwork-Id: 12526905 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DDC58C433FE for ; Wed, 29 Sep 2021 22:53:03 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 83C17613A0 for ; Wed, 29 Sep 2021 22:53:03 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 83C17613A0 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.199328.353319 (Exim 4.92) (envelope-from ) id 1mViRO-0005up-7P; Wed, 29 Sep 2021 22:52:42 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 199328.353319; Wed, 29 Sep 2021 22:52:42 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mViRO-0005ug-4F; Wed, 29 Sep 2021 22:52:42 +0000 Received: by outflank-mailman (input) for mailman id 199328; Wed, 29 Sep 2021 22:52:41 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mViRN-0005eO-8A for xen-devel@lists.xenproject.org; Wed, 29 Sep 2021 22:52:41 +0000 Received: from mail-lf1-x12b.google.com (unknown [2a00:1450:4864:20::12b]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id 39071c9b-55bf-45a7-acc5-da34a53ef60c; Wed, 29 Sep 2021 22:52:35 +0000 (UTC) Received: by mail-lf1-x12b.google.com with SMTP id m3so16941328lfu.2 for ; Wed, 29 Sep 2021 15:52:35 -0700 (PDT) Received: from otyshchenko.router ([212.22.223.21]) by smtp.gmail.com with ESMTPSA id g26sm133178lja.16.2021.09.29.15.52.33 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 29 Sep 2021 15:52:34 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 39071c9b-55bf-45a7-acc5-da34a53ef60c DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=6iJhS8sphgwZjkcusuNiu4A9g9hD+Ik9xi5wqFSEXCA=; b=XfpD5CaoOHUa3T1eWlpV4iVp0qd4qArA6lMrXYi/2x+QtbZVzo2qdWz231Bd5KEOz2 1FzHBouCfYWsTOTsVDRiTM0tPP+P0BW22fiNi14MLZKDDyMcAo6P8jykx8KW9/Uaillw gZKewKZ55Tom9y153hfyfvuqu89h85tHAEsg7LDkfYd6QXlj8R5j/u6Y/pPKHROtKlqP HU7RM+N0wFsB3xYN2Zc1ML4/1P5BGsEVatqCPUNvg285Jnz/kxN2pq9WWQG2hS+WTGxn l5H9CxpvbPWIjpIxDnVcZJkKJSw+1aClDjEIimKKQoQX5LsLg+JLmSNgyBfb7vPfySO8 zmrA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=6iJhS8sphgwZjkcusuNiu4A9g9hD+Ik9xi5wqFSEXCA=; b=3QWJVBFBC2EIIMCb1cl4mwgO9pZPCyH9OxYogXT2jFj8yuuvk5uzfaA5EgqAOF+mIM 9BejCxCGJsvB4NvkjjdZ4VomtZRlVfBvLypAzitOmMeBjduvLjbvR/Z/PWHdROej4x5C 4eN/hdLZN9V8u4p7xdFWt0A/0tkuhAtajFTo5BzpvQJh7cIdgzXEP8KnHV9XEV5gm227 ua/2v/kqjfwyxjJEUX07YyrTR8rHkuG47UE47e+FAj9OMAG8WFlFccmiBmzTBqsdmInd ew/R2Lp2m7oIkTE38Xc+I/0E9ttVN4jMddTGML9mkKDXH/jeA2gHxDU8BeV4UXW4pa9/ 2iow== X-Gm-Message-State: AOAM533jybLkzBvycPpSPrgaShL8CI6BwxKd75JPvRZtvAumXJkb1kWX zm2kRC87bgo0UMrxIAUF4D8z5UKBcIhXqw== X-Google-Smtp-Source: ABdhPJxHS/aqZSklH95g0ibf9cFZWdz2JQxvIIqc4Id9zhoH+ECAkzA68y4SqkB2/FDTSIw8AOuHKw== X-Received: by 2002:a05:651c:12c6:: with SMTP id 6mr2570306lje.194.1632955954429; Wed, 29 Sep 2021 15:52:34 -0700 (PDT) From: Oleksandr Tyshchenko To: xen-devel@lists.xenproject.org Cc: Oleksandr Tyshchenko , Ian Jackson , Wei Liu , Anthony PERARD , Andrew Cooper , George Dunlap , Jan Beulich , Julien Grall , Stefano Stabellini , Juergen Gross , Volodymyr Babchuk , =?utf-8?q?Roger_Pau_Monn?= =?utf-8?q?=C3=A9?= Subject: [PATCH V4 1/3] xen: Introduce "gpaddr_bits" field to XEN_SYSCTL_physinfo Date: Thu, 30 Sep 2021 01:52:05 +0300 Message-Id: <1632955927-27911-2-git-send-email-olekstysh@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1632955927-27911-1-git-send-email-olekstysh@gmail.com> References: <1632955927-27911-1-git-send-email-olekstysh@gmail.com> From: Oleksandr Tyshchenko We need to pass info about maximum supported guest address space size to the toolstack on Arm in order to properly calculate the base and size of the extended region (safe range) for the guest. The extended region is unused address space which could be safely used by domain for foreign/grant mappings on Arm. The extended region itself will be handled by the subsequents patch. Use p2m_ipa_bits variable on Arm, the x86 equivalent is hap_paddr_bits. As we change the size of structure bump the interface version. Suggested-by: Julien Grall Signed-off-by: Oleksandr Tyshchenko Reviewed-by: Michal Orzel Reviewed-by: Stefano Stabellini --- Please note, that review comments for the RFC version [1] haven't been addressed yet. It is not forgotten, some clarification is needed. It will be addressed for the next version. [1] https://lore.kernel.org/xen-devel/973f5344-aa10-3ad6-ff02-ad5f358ad279@citrix.com/ Changes RFC -> V2: - update patch subject/description - replace arch-specific sub-struct with common gpaddr_bits field and update code to reflect that Changes V2 -> V3: - make the field uint8_t and add uint8_t pad[7] after - remove leading blanks in libxl.h Changes V3 -> V4: - also print gpaddr_bits from output_physinfo() - add Michal's R-b --- tools/include/libxl.h | 7 +++++++ tools/libs/light/libxl.c | 2 ++ tools/libs/light/libxl_types.idl | 2 ++ tools/xl/xl_info.c | 2 ++ xen/arch/arm/sysctl.c | 2 ++ xen/arch/x86/sysctl.c | 2 ++ xen/include/public/sysctl.h | 4 +++- 7 files changed, 20 insertions(+), 1 deletion(-) diff --git a/tools/include/libxl.h b/tools/include/libxl.h index b9ba16d..63f9550 100644 --- a/tools/include/libxl.h +++ b/tools/include/libxl.h @@ -856,6 +856,13 @@ typedef struct libxl__ctx libxl_ctx; #define LIBXL_HAVE_PHYSINFO_MAX_POSSIBLE_MFN 1 /* + * LIBXL_HAVE_PHYSINFO_GPADDR_BITS + * + * If this is defined, libxl_physinfo has a "gpaddr_bits" field. + */ +#define LIBXL_HAVE_PHYSINFO_GPADDR_BITS 1 + +/* * LIBXL_HAVE_DOMINFO_OUTSTANDING_MEMKB 1 * * If this is defined, libxl_dominfo will contain a MemKB type field called diff --git a/tools/libs/light/libxl.c b/tools/libs/light/libxl.c index 204eb0b..c86624f 100644 --- a/tools/libs/light/libxl.c +++ b/tools/libs/light/libxl.c @@ -405,6 +405,8 @@ int libxl_get_physinfo(libxl_ctx *ctx, libxl_physinfo *physinfo) physinfo->cap_vmtrace = !!(xcphysinfo.capabilities & XEN_SYSCTL_PHYSCAP_vmtrace); + physinfo->gpaddr_bits = xcphysinfo.gpaddr_bits; + GC_FREE; return 0; } diff --git a/tools/libs/light/libxl_types.idl b/tools/libs/light/libxl_types.idl index 3f9fff6..bf27fe6 100644 --- a/tools/libs/light/libxl_types.idl +++ b/tools/libs/light/libxl_types.idl @@ -1061,6 +1061,8 @@ libxl_physinfo = Struct("physinfo", [ ("cap_shadow", bool), ("cap_iommu_hap_pt_share", bool), ("cap_vmtrace", bool), + + ("gpaddr_bits", uint8), ], dir=DIR_OUT) libxl_connectorinfo = Struct("connectorinfo", [ diff --git a/tools/xl/xl_info.c b/tools/xl/xl_info.c index 8383e4a..dfbbeaa 100644 --- a/tools/xl/xl_info.c +++ b/tools/xl/xl_info.c @@ -221,6 +221,8 @@ static void output_physinfo(void) info.cap_vmtrace ? " vmtrace" : "" ); + maybe_printf("gpaddr_bits : %d\n", info.gpaddr_bits); + vinfo = libxl_get_version_info(ctx); if (vinfo) { i = (1 << 20) / vinfo->pagesize; diff --git a/xen/arch/arm/sysctl.c b/xen/arch/arm/sysctl.c index f87944e..91dca4f 100644 --- a/xen/arch/arm/sysctl.c +++ b/xen/arch/arm/sysctl.c @@ -15,6 +15,8 @@ void arch_do_physinfo(struct xen_sysctl_physinfo *pi) { pi->capabilities |= XEN_SYSCTL_PHYSCAP_hvm | XEN_SYSCTL_PHYSCAP_hap; + + pi->gpaddr_bits = p2m_ipa_bits; } long arch_do_sysctl(struct xen_sysctl *sysctl, diff --git a/xen/arch/x86/sysctl.c b/xen/arch/x86/sysctl.c index aff52a1..7b14865 100644 --- a/xen/arch/x86/sysctl.c +++ b/xen/arch/x86/sysctl.c @@ -135,6 +135,8 @@ void arch_do_physinfo(struct xen_sysctl_physinfo *pi) pi->capabilities |= XEN_SYSCTL_PHYSCAP_hap; if ( IS_ENABLED(CONFIG_SHADOW_PAGING) ) pi->capabilities |= XEN_SYSCTL_PHYSCAP_shadow; + + pi->gpaddr_bits = hap_paddr_bits; } long arch_do_sysctl( diff --git a/xen/include/public/sysctl.h b/xen/include/public/sysctl.h index 039ccf8..0450a78 100644 --- a/xen/include/public/sysctl.h +++ b/xen/include/public/sysctl.h @@ -35,7 +35,7 @@ #include "domctl.h" #include "physdev.h" -#define XEN_SYSCTL_INTERFACE_VERSION 0x00000013 +#define XEN_SYSCTL_INTERFACE_VERSION 0x00000014 /* * Read console content from Xen buffer ring. @@ -120,6 +120,8 @@ struct xen_sysctl_physinfo { uint64_aligned_t outstanding_pages; uint64_aligned_t max_mfn; /* Largest possible MFN on this host */ uint32_t hw_cap[8]; + uint8_t gpaddr_bits; + uint8_t pad[7]; }; /* From patchwork Wed Sep 29 22:52:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Tyshchenko X-Patchwork-Id: 12526901 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9DFF3C433EF for ; Wed, 29 Sep 2021 22:53:03 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 34B0F6142A for ; Wed, 29 Sep 2021 22:53:03 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 34B0F6142A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.199329.353330 (Exim 4.92) (envelope-from ) id 1mViRT-0006E6-Fd; Wed, 29 Sep 2021 22:52:47 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 199329.353330; Wed, 29 Sep 2021 22:52:47 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mViRT-0006Du-CZ; Wed, 29 Sep 2021 22:52:47 +0000 Received: by outflank-mailman (input) for mailman id 199329; Wed, 29 Sep 2021 22:52:46 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mViRS-0005eO-8V for xen-devel@lists.xenproject.org; Wed, 29 Sep 2021 22:52:46 +0000 Received: from mail-lf1-x12e.google.com (unknown [2a00:1450:4864:20::12e]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id ae92a60d-2dcd-47cc-a077-514b6a7ec018; Wed, 29 Sep 2021 22:52:36 +0000 (UTC) Received: by mail-lf1-x12e.google.com with SMTP id e15so17151185lfr.10 for ; Wed, 29 Sep 2021 15:52:36 -0700 (PDT) Received: from otyshchenko.router ([212.22.223.21]) by smtp.gmail.com with ESMTPSA id g26sm133178lja.16.2021.09.29.15.52.34 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 29 Sep 2021 15:52:35 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: ae92a60d-2dcd-47cc-a077-514b6a7ec018 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=2up+jpzvkiUGC0ZnAjQlIu5gOHMwSu/FSUWJVQDo7sg=; b=NaYnQ2utvVo47PVBmbrhDtMAasyXppuMpH4+xXi2Uc4sNoJCxJn9ztYmX1D6edSptW VS35YnR0lcDzHKU3dPHumY3TKszcEh1pwkzhiVnDhVQexejVStqGaoD7+50vrI+s/D1l Pw5rHt4eHduwVJNctlxjxpC8Zjw/wPAnI6ogkC4XgWg2pZPajeihCgizDOdYBwRN/Jsb dylbnmpSjXm8cASEKscw5snndzKjfDX1YjzhmPSqbStFoMIBLoibUfi5/XXYhzu7XXh1 BkE4VZqaa9XnqlUHkeEghO/7DYzUxRgSdLZb5nGb9HdCMRDcG8d6SZ0BICiIjsQGypBJ Tg3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=2up+jpzvkiUGC0ZnAjQlIu5gOHMwSu/FSUWJVQDo7sg=; b=4FevMsXYcCBQXQiN5JyIhl1xwhvMGPAO5QEOY7Ux9uDdN2yKeD8yaJSJYN/r3av4OW wbesFCwbrsyVs0EqlYszMLdVGmTuLRnmAITLvOsGcbqptvHkvZlpBS+B6Cyurn4KgFW4 xgBKYr5c/rGYoML9f0+K2QuBwbUVeFYpBU1CH131rEYbh0/wBjRoSpHOTd7NOt9znHnE IeU+HlDX6SfH1LKBYO7vhpM2wVXyzst0RXU2IAy0zCiWOLj3/wR6zofCj9SzgHS2JvTu uzSlHVKYyyvktbD+xQsL+NNeqJnj6c0fs56II7suA16OcdGUiZlYYQlHfekRVbFf/Ryk zaSg== X-Gm-Message-State: AOAM533p+Vy9JfyHARiTqX1X5HdJA1VElbSx0WoixJXfdv930GPcYDY2 YGbD57yKUOIoYXcfc3xlzWRn308aYDeo+Q== X-Google-Smtp-Source: ABdhPJxZf5d54evx4goBRAO/op87y8XxoFBaz3jFucB1o1inkxXir4YOVSyaT/CgGrCzpXo3gaQLkA== X-Received: by 2002:a05:6512:3341:: with SMTP id y1mr2511831lfd.424.1632955955505; Wed, 29 Sep 2021 15:52:35 -0700 (PDT) From: Oleksandr Tyshchenko To: xen-devel@lists.xenproject.org Cc: Oleksandr Tyshchenko , Andrew Cooper , George Dunlap , Ian Jackson , Jan Beulich , Julien Grall , Stefano Stabellini , Wei Liu , Volodymyr Babchuk Subject: [PATCH V4 2/3] xen/arm: Add handling of extended regions for Dom0 Date: Thu, 30 Sep 2021 01:52:06 +0300 Message-Id: <1632955927-27911-3-git-send-email-olekstysh@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1632955927-27911-1-git-send-email-olekstysh@gmail.com> References: <1632955927-27911-1-git-send-email-olekstysh@gmail.com> From: Oleksandr Tyshchenko The extended region (safe range) is a region of guest physical address space which is unused and could be safely used to create grant/foreign mappings instead of wasting real RAM pages from the domain memory for establishing these mappings. The extended regions are chosen at the domain creation time and advertised to it via "reg" property under hypervisor node in the guest device-tree. As region 0 is reserved for grant table space (always present), the indexes for extended regions are 1...N. If extended regions could not be allocated for some reason, Xen doesn't fail and behaves as usual, so only inserts region 0. Please note the following limitations: - The extended region feature is only supported for 64-bit domain currently. - The ACPI case is not covered. *** As Dom0 is direct mapped domain on Arm (e.g. MFN == GFN) the algorithm to choose extended regions for it is different in comparison with the algorithm for non-direct mapped DomU. What is more, that extended regions should be chosen differently whether IOMMU is enabled or not. Provide RAM not assigned to Dom0 if IOMMU is disabled or memory holes found in host device-tree if otherwise. Make sure that extended regions are 2MB-aligned and located within maximum possible addressable physical memory range. The minimum size of extended region is 64MB. The maximum number of extended regions is 128, which is an artificial limit to minimize code changes (we reuse struct meminfo to describe extended regions, so there are an array field for 128 elements). It worth mentioning that unallocated memory solution (when the IOMMU is disabled) will work safely until Dom0 is able to allocate memory outside of the original range. Also introduce command line option to be able to globally enable or disable support for extended regions for Dom0 (enabled by default). Suggested-by: Julien Grall Signed-off-by: Oleksandr Tyshchenko Reviewed-by: Luca Fancellu Reviewed-by: Stefano Stabellini --- Please note, we need to decide which approach to use in find_unallocated_memory(), you can find details at: https://lore.kernel.org/xen-devel/28503e09-44c3-f623-bb8d-8778bb94225f@gmail.com/ Changes RFC -> V2: - update patch description - drop uneeded "extended-region" DT property Changes V2 -> V3: - update patch description - add comment for "size" calculation in add_ext_regions() - clarify "end" calculation in find_unallocated_memory() and find_memory_holes() - only pick up regions with size >= 64MB - allocate reg dynamically instead of keeping on the stack in make_hypervisor_node() - do not show warning for 32-bit domain - drop Linux specific limits EXT_REGION_* - also cover "ranges" property in find_memory_holes() - add command line arg to enable/disable extended region support Changes V3 -> V4: - update opt_ext_regions purpose and comment in code - reorganize make_hypervisor_node() to move allocations after initial checks, allocate only required amount of elements instead of maximum possible --- docs/misc/xen-command-line.pandoc | 11 ++ xen/arch/arm/domain_build.c | 286 +++++++++++++++++++++++++++++++++++++- 2 files changed, 294 insertions(+), 3 deletions(-) diff --git a/docs/misc/xen-command-line.pandoc b/docs/misc/xen-command-line.pandoc index 177e656..5cae4ad 100644 --- a/docs/misc/xen-command-line.pandoc +++ b/docs/misc/xen-command-line.pandoc @@ -1081,6 +1081,17 @@ hardware domain is architecture dependent. Note that specifying zero as domU value means zero, while for dom0 it means to use the default. +### ext_regions (Arm) +> `= ` + +> Default : `true` + +Flag to enable or disable support for extended regions for Dom0. + +Extended regions are ranges of unused address space exposed to Dom0 as +"safe to use" for special memory mappings. Disable if your board device +tree is incomplete. + ### flask > `= permissive | enforcing | late | disabled` diff --git a/xen/arch/arm/domain_build.c b/xen/arch/arm/domain_build.c index d233d63..c5afbe2 100644 --- a/xen/arch/arm/domain_build.c +++ b/xen/arch/arm/domain_build.c @@ -34,6 +34,10 @@ static unsigned int __initdata opt_dom0_max_vcpus; integer_param("dom0_max_vcpus", opt_dom0_max_vcpus); +/* If true, the extended regions support is enabled for dom0 */ +static bool __initdata opt_ext_regions = true; +boolean_param("ext_regions", opt_ext_regions); + static u64 __initdata dom0_mem; static bool __initdata dom0_mem_set; @@ -886,6 +890,232 @@ static int __init make_memory_node(const struct domain *d, return res; } +static int __init add_ext_regions(unsigned long s, unsigned long e, void *data) +{ + struct meminfo *ext_regions = data; + paddr_t start, size; + + if ( ext_regions->nr_banks >= ARRAY_SIZE(ext_regions->bank) ) + return 0; + + /* Both start and size of the extended region should be 2MB aligned */ + start = (s + SZ_2M - 1) & ~(SZ_2M - 1); + if ( start > e ) + return 0; + + /* + * e is actually "end-1" because it is called by rangeset functions + * which are inclusive of the last address. + */ + e += 1; + size = (e - start) & ~(SZ_2M - 1); + if ( size < MB(64) ) + return 0; + + ext_regions->bank[ext_regions->nr_banks].start = start; + ext_regions->bank[ext_regions->nr_banks].size = size; + ext_regions->nr_banks++; + + return 0; +} + +static int __init find_unallocated_memory(const struct kernel_info *kinfo, + struct meminfo *ext_regions) +{ + const struct meminfo *assign_mem = &kinfo->mem; + struct rangeset *unalloc_mem; + paddr_t start, end; + unsigned int i; + int res; + + dt_dprintk("Find unallocated memory for extended regions\n"); + + unalloc_mem = rangeset_new(NULL, NULL, 0); + if ( !unalloc_mem ) + return -ENOMEM; + + /* Start with all available RAM */ + for ( i = 0; i < bootinfo.mem.nr_banks; i++ ) + { + start = bootinfo.mem.bank[i].start; + end = bootinfo.mem.bank[i].start + bootinfo.mem.bank[i].size; + res = rangeset_add_range(unalloc_mem, start, end - 1); + if ( res ) + { + printk(XENLOG_ERR "Failed to add: %#"PRIx64"->%#"PRIx64"\n", + start, end); + goto out; + } + } + + /* Remove RAM assigned to Dom0 */ + for ( i = 0; i < assign_mem->nr_banks; i++ ) + { + start = assign_mem->bank[i].start; + end = assign_mem->bank[i].start + assign_mem->bank[i].size; + res = rangeset_remove_range(unalloc_mem, start, end - 1); + if ( res ) + { + printk(XENLOG_ERR "Failed to remove: %#"PRIx64"->%#"PRIx64"\n", + start, end); + goto out; + } + } + + /* Remove reserved-memory regions */ + for ( i = 0; i < bootinfo.reserved_mem.nr_banks; i++ ) + { + start = bootinfo.reserved_mem.bank[i].start; + end = bootinfo.reserved_mem.bank[i].start + + bootinfo.reserved_mem.bank[i].size; + res = rangeset_remove_range(unalloc_mem, start, end - 1); + if ( res ) + { + printk(XENLOG_ERR "Failed to remove: %#"PRIx64"->%#"PRIx64"\n", + start, end); + goto out; + } + } + + /* Remove grant table region */ + start = kinfo->gnttab_start; + end = kinfo->gnttab_start + kinfo->gnttab_size; + res = rangeset_remove_range(unalloc_mem, start, end - 1); + if ( res ) + { + printk(XENLOG_ERR "Failed to remove: %#"PRIx64"->%#"PRIx64"\n", + start, end); + goto out; + } + + start = 0; + end = (1ULL << p2m_ipa_bits) - 1; + res = rangeset_report_ranges(unalloc_mem, start, end, + add_ext_regions, ext_regions); + if ( res ) + ext_regions->nr_banks = 0; + else if ( !ext_regions->nr_banks ) + res = -ENOENT; + +out: + rangeset_destroy(unalloc_mem); + + return res; +} + +static int __init find_memory_holes(const struct kernel_info *kinfo, + struct meminfo *ext_regions) +{ + struct dt_device_node *np; + struct rangeset *mem_holes; + paddr_t start, end; + unsigned int i; + int res; + + dt_dprintk("Find memory holes for extended regions\n"); + + mem_holes = rangeset_new(NULL, NULL, 0); + if ( !mem_holes ) + return -ENOMEM; + + /* Start with maximum possible addressable physical memory range */ + start = 0; + end = (1ULL << p2m_ipa_bits) - 1; + res = rangeset_add_range(mem_holes, start, end); + if ( res ) + { + printk(XENLOG_ERR "Failed to add: %#"PRIx64"->%#"PRIx64"\n", + start, end); + goto out; + } + + /* + * Remove regions described by "reg" and "ranges" properties where + * the memory is addressable (MMIO, RAM, PCI BAR, etc). + */ + dt_for_each_device_node( dt_host, np ) + { + unsigned int naddr; + u64 addr, size; + + naddr = dt_number_of_address(np); + + for ( i = 0; i < naddr; i++ ) + { + res = dt_device_get_address(np, i, &addr, &size); + if ( res ) + { + printk(XENLOG_ERR "Unable to retrieve address %u for %s\n", + i, dt_node_full_name(np)); + goto out; + } + + start = addr & PAGE_MASK; + end = PAGE_ALIGN(addr + size); + res = rangeset_remove_range(mem_holes, start, end - 1); + if ( res ) + { + printk(XENLOG_ERR "Failed to remove: %#"PRIx64"->%#"PRIx64"\n", + start, end); + goto out; + } + } + + if ( dt_device_type_is_equal(np, "pci" ) ) + { + unsigned int range_size, nr_ranges; + int na, ns, pna; + const __be32 *ranges; + u32 len; + + /* + * Looking for non-empty ranges property which in this context + * describes the PCI host bridge aperture. + */ + ranges = dt_get_property(np, "ranges", &len); + if ( !ranges || !len ) + continue; + + pna = dt_n_addr_cells(np); + na = dt_child_n_addr_cells(np); + ns = dt_child_n_size_cells(np); + range_size = pna + na + ns; + nr_ranges = len / sizeof(__be32) / range_size; + + for ( i = 0; i < nr_ranges; i++, ranges += range_size ) + { + /* Skip the child address and get the parent (CPU) address */ + addr = dt_read_number(ranges + na, pna); + size = dt_read_number(ranges + na + pna, ns); + + start = addr & PAGE_MASK; + end = PAGE_ALIGN(addr + size); + res = rangeset_remove_range(mem_holes, start, end - 1); + if ( res ) + { + printk(XENLOG_ERR "Failed to remove: %#"PRIx64"->%#"PRIx64"\n", + start, end); + goto out; + } + } + } + } + + start = 0; + end = (1ULL << p2m_ipa_bits) - 1; + res = rangeset_report_ranges(mem_holes, start, end, + add_ext_regions, ext_regions); + if ( res ) + ext_regions->nr_banks = 0; + else if ( !ext_regions->nr_banks ) + res = -ENOENT; + +out: + rangeset_destroy(mem_holes); + + return res; +} + static int __init make_hypervisor_node(struct domain *d, const struct kernel_info *kinfo, int addrcells, int sizecells) @@ -893,11 +1123,12 @@ static int __init make_hypervisor_node(struct domain *d, const char compat[] = "xen,xen-"__stringify(XEN_VERSION)"."__stringify(XEN_SUBVERSION)"\0" "xen,xen"; - __be32 reg[4]; + __be32 *reg, *cells; gic_interrupt_t intr; - __be32 *cells; int res; void *fdt = kinfo->fdt; + struct meminfo *ext_regions = NULL; + unsigned int i, nr_ext_regions; dt_dprintk("Create hypervisor node\n"); @@ -919,12 +1150,61 @@ static int __init make_hypervisor_node(struct domain *d, if ( res ) return res; + if ( !opt_ext_regions ) + { + printk(XENLOG_DEBUG "The extended regions support is disabled\n"); + nr_ext_regions = 0; + } + else if ( is_32bit_domain(d) ) + { + printk(XENLOG_DEBUG "The extended regions are only supported for 64-bit guest currently\n"); + nr_ext_regions = 0; + } + else + { + ext_regions = xzalloc(struct meminfo); + if ( !ext_regions ) + return -ENOMEM; + + if ( !is_iommu_enabled(d) ) + res = find_unallocated_memory(kinfo, ext_regions); + else + res = find_memory_holes(kinfo, ext_regions); + + if ( res ) + printk(XENLOG_WARNING "Failed to allocate extended regions\n"); + nr_ext_regions = ext_regions->nr_banks; + } + + reg = xzalloc_array(__be32, (nr_ext_regions + 1) * (addrcells + sizecells)); + if ( !reg ) + { + xfree(ext_regions); + return -ENOMEM; + } + /* reg 0 is grant table space */ cells = ®[0]; dt_child_set_range(&cells, addrcells, sizecells, kinfo->gnttab_start, kinfo->gnttab_size); + /* reg 1...N are extended regions */ + for ( i = 0; i < nr_ext_regions; i++ ) + { + u64 start = ext_regions->bank[i].start; + u64 size = ext_regions->bank[i].size; + + dt_dprintk("Extended region %d: %#"PRIx64"->%#"PRIx64"\n", + i, start, start + size); + + dt_child_set_range(&cells, addrcells, sizecells, start, size); + } + res = fdt_property(fdt, "reg", reg, - dt_cells_to_size(addrcells + sizecells)); + dt_cells_to_size(addrcells + sizecells) * + (nr_ext_regions + 1)); + xfree(ext_regions); + xfree(reg); + if ( res ) return res; From patchwork Wed Sep 29 22:52:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oleksandr Tyshchenko X-Patchwork-Id: 12526899 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 19B36C433F5 for ; Wed, 29 Sep 2021 22:53:03 +0000 (UTC) Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A2D646142A for ; Wed, 29 Sep 2021 22:53:02 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org A2D646142A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=lists.xenproject.org Received: from list by lists.xenproject.org with outflank-mailman.199330.353341 (Exim 4.92) (envelope-from ) id 1mViRY-0006a6-SC; Wed, 29 Sep 2021 22:52:52 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 199330.353341; Wed, 29 Sep 2021 22:52:52 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mViRY-0006Zx-OQ; Wed, 29 Sep 2021 22:52:52 +0000 Received: by outflank-mailman (input) for mailman id 199330; Wed, 29 Sep 2021 22:52:51 +0000 Received: from us1-rack-iad1.inumbo.com ([172.99.69.81]) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1mViRX-0005eO-8S for xen-devel@lists.xenproject.org; Wed, 29 Sep 2021 22:52:51 +0000 Received: from mail-lf1-x132.google.com (unknown [2a00:1450:4864:20::132]) by us1-rack-iad1.inumbo.com (Halon) with ESMTPS id c000e0a0-3405-4bf9-9bbb-3804c7cd0959; Wed, 29 Sep 2021 22:52:37 +0000 (UTC) Received: by mail-lf1-x132.google.com with SMTP id m3so16941616lfu.2 for ; Wed, 29 Sep 2021 15:52:37 -0700 (PDT) Received: from otyshchenko.router ([212.22.223.21]) by smtp.gmail.com with ESMTPSA id g26sm133178lja.16.2021.09.29.15.52.35 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Wed, 29 Sep 2021 15:52:36 -0700 (PDT) X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: c000e0a0-3405-4bf9-9bbb-3804c7cd0959 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=/1JvRrp5W5uJkWjfb1ZqIFuL/StY5VJHuwwhfYh02cI=; b=fiDgjzxCvzMpU+cge8HG1YXhBeDd1ALRtgkbGFFTLjsGfFwVqKESbq9IB7CTe7YvWg c0wrNKIsfAJ1vfz/VOVEPBJNlj81Eado2i3ktjoCI4Cd9dzf+SpEIw+y9c7GPH0uAIzT J2XNG3IDV0nIazOUO+3TW9DFRglJXcUWJpfWpk4dIZVv7dsNRNXo1GpR7leJ5zLIySQB 8ejfP3VByyW1G4srgR4Ao57FET1fXN3hnzwJBrCOEJlr5JoDWo5yQU09QngNYV1GyeYX TbyQ6ZOzvgCLj5wMsFJP2xLK9FS4Tr+CC0uHQxNntYQs2xDkIM/62qJPELr2URNSiihS clPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=/1JvRrp5W5uJkWjfb1ZqIFuL/StY5VJHuwwhfYh02cI=; b=xJh/npI185PD0n3myoJ55OXveMLPNCFNm+I7kvUn//LQvxIQni2NVHqtCGFUK38XgP Rk5hIWh34nM8dBtMvOknrfl9pPjFNp0QS8Af0exhbYJl2HFl4kPRRXdOB2SyM5fMHx9u afAXXwCWvSoIRSgyuVigSdWbDRdTYvIPcewC4iqTrQamrmKNxyKeGASDOxtiMfb2y3xu JygE7f9axvgBvQSH3JatRKZJgYlBz2WvGlBsNH3nr4iAVmkDlwUfezdrtyMynbrX5Zjh nX+06o8qiAXUMzCCNeFHG7LXMRMvnym/WDxYm0Ld1FVe/NN8iB5yrshrC6Z4jq/KLiWt Lgig== X-Gm-Message-State: AOAM530bzviR1kF+gtEuoFZcMUfaub+cYAiUwfQXhgWuiPDzlrC+Q9EG NJkok/M4Hvmm5RBG3UZ9pqOVFOg5TGnpLQ== X-Google-Smtp-Source: ABdhPJzl6QAZiwF0YkTVqnJbdYmYGW4g0VGERnOjOyaoDOewIYtd11OLyFArNNk+PgEOqO6j7hBWKw== X-Received: by 2002:a2e:a446:: with SMTP id v6mr2419279ljn.143.1632955956436; Wed, 29 Sep 2021 15:52:36 -0700 (PDT) From: Oleksandr Tyshchenko To: xen-devel@lists.xenproject.org Cc: Oleksandr Tyshchenko , Ian Jackson , Wei Liu , Anthony PERARD , Juergen Gross , Stefano Stabellini , Julien Grall , Volodymyr Babchuk Subject: [PATCH V4 3/3] libxl/arm: Add handling of extended regions for DomU Date: Thu, 30 Sep 2021 01:52:07 +0300 Message-Id: <1632955927-27911-4-git-send-email-olekstysh@gmail.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1632955927-27911-1-git-send-email-olekstysh@gmail.com> References: <1632955927-27911-1-git-send-email-olekstysh@gmail.com> From: Oleksandr Tyshchenko The extended region (safe range) is a region of guest physical address space which is unused and could be safely used to create grant/foreign mappings instead of wasting real RAM pages from the domain memory for establishing these mappings. The extended regions are chosen at the domain creation time and advertised to it via "reg" property under hypervisor node in the guest device-tree. As region 0 is reserved for grant table space (always present), the indexes for extended regions are 1...N. If extended regions could not be allocated for some reason, Xen doesn't fail and behaves as usual, so only inserts region 0. Please note the following limitations: - The extended region feature is only supported for 64-bit domain currently. - The ACPI case is not covered. *** The algorithm to choose extended regions for non-direct mapped DomU is simpler in comparison with the algorithm for direct mapped Dom0. As we have a lot of unused space above 4GB, provide single 2MB-aligned region from the second RAM bank taking into the account the maximum supported guest address space size and the amount of memory assigned to the guest. The maximum size of the region is 128GB. The minimum size is 64MB. Suggested-by: Julien Grall Signed-off-by: Oleksandr Tyshchenko Reviewed-by: Stefano Stabellini Acked-by: Ian Jackson Reviewed-by: Michal Orzel Tested-by: Michal Orzel --- Changes RFC -> V2: - update patch description - drop uneeded "extended-region" DT property - clear reg array in finalise_ext_region() and add a TODO Changes V2 -> V3: - update patch description, comments in code - only pick up regions with size >= 64MB - move the region calculation to make_hypervisor_node() and drop finalise_ext_region() - extend the list of arguments for make_hypervisor_node() - do not show warning for 32-bit domain - change the region alignment from 1GB to 2MB - move EXT_REGION_SIZE to public/arch-arm.h Changes V3 -> V4: - add R-b, A-b and T-b --- tools/libs/light/libxl_arm.c | 70 +++++++++++++++++++++++++++++++++++++++---- xen/include/public/arch-arm.h | 3 ++ 2 files changed, 68 insertions(+), 5 deletions(-) diff --git a/tools/libs/light/libxl_arm.c b/tools/libs/light/libxl_arm.c index e3140a6..a67b68e 100644 --- a/tools/libs/light/libxl_arm.c +++ b/tools/libs/light/libxl_arm.c @@ -598,9 +598,17 @@ static int make_timer_node(libxl__gc *gc, void *fdt, return 0; } +#define ALIGN_UP_TO_2MB(x) (((x) + MB(2) - 1) & (~(MB(2) - 1))) + static int make_hypervisor_node(libxl__gc *gc, void *fdt, - const libxl_version_info *vers) + const libxl_version_info *vers, + const libxl_domain_build_info *b_info, + const struct xc_dom_image *dom) { + uint64_t region_size = 0, region_base, ramsize, bank1size, + bank1end_align, bank1end_max; + uint8_t gpaddr_bits; + libxl_physinfo physinfo; int res; gic_interrupt intr; @@ -615,9 +623,61 @@ static int make_hypervisor_node(libxl__gc *gc, void *fdt, "xen,xen"); if (res) return res; - /* reg 0 is grant table space */ - res = fdt_property_regs(gc, fdt, GUEST_ROOT_ADDRESS_CELLS, GUEST_ROOT_SIZE_CELLS, - 1,GUEST_GNTTAB_BASE, GUEST_GNTTAB_SIZE); + if (strcmp(dom->guest_type, "xen-3.0-aarch64")) { + LOG(DEBUG, "The extended regions are only supported for 64-bit guest currently"); + goto out; + } + + res = libxl_get_physinfo(CTX, &physinfo); + assert(!res); + + gpaddr_bits = physinfo.gpaddr_bits; + assert(gpaddr_bits >= 32 && gpaddr_bits <= 48); + + /* + * Try to allocate single 2MB-aligned extended region from the second RAM + * bank (above 4GB) taking into the account the maximum supported guest + * address space size and the amount of memory assigned to the guest. + * As the guest memory layout is not populated yet we cannot rely on + * dom->rambank_size[1], so calculate the actual size of the second bank + * using "max_memkb" value. + */ + bank1end_max = min(1ULL << gpaddr_bits, GUEST_RAM1_BASE + GUEST_RAM1_SIZE); + ramsize = b_info->max_memkb * 1024; + if (ramsize <= GUEST_RAM0_SIZE) + bank1size = 0; + else + bank1size = ramsize - GUEST_RAM0_SIZE; + bank1end_align = GUEST_RAM1_BASE + ALIGN_UP_TO_2MB(bank1size); + + if (bank1end_max <= bank1end_align) { + LOG(WARN, "The extended region cannot be allocated, not enough space"); + goto out; + } + + if (bank1end_max - bank1end_align > GUEST_EXT_REGION_MAX_SIZE) { + region_base = bank1end_max - GUEST_EXT_REGION_MAX_SIZE; + region_size = GUEST_EXT_REGION_MAX_SIZE; + } else { + region_base = bank1end_align; + region_size = bank1end_max - bank1end_align; + } + +out: + /* + * The region 0 for grant table space must be always present. If we managed + * to allocate the extended region then insert it as region 1. + */ + if (region_size >= GUEST_EXT_REGION_MIN_SIZE) { + LOG(DEBUG, "Extended region: %#"PRIx64"->%#"PRIx64"\n", + region_base, region_base + region_size); + + res = fdt_property_regs(gc, fdt, GUEST_ROOT_ADDRESS_CELLS, GUEST_ROOT_SIZE_CELLS, + 2, GUEST_GNTTAB_BASE, GUEST_GNTTAB_SIZE, + region_base, region_size); + } else + res = fdt_property_regs(gc, fdt, GUEST_ROOT_ADDRESS_CELLS, GUEST_ROOT_SIZE_CELLS, + 1, GUEST_GNTTAB_BASE, GUEST_GNTTAB_SIZE); if (res) return res; /* @@ -963,7 +1023,7 @@ next_resize: } FDT( make_timer_node(gc, fdt, ainfo, state->clock_frequency) ); - FDT( make_hypervisor_node(gc, fdt, vers) ); + FDT( make_hypervisor_node(gc, fdt, vers, info, dom) ); if (info->arch_arm.vuart == LIBXL_VUART_TYPE_SBSA_UART) FDT( make_vpl011_uart_node(gc, fdt, ainfo, dom) ); diff --git a/xen/include/public/arch-arm.h b/xen/include/public/arch-arm.h index 6b5a5f8..df59933 100644 --- a/xen/include/public/arch-arm.h +++ b/xen/include/public/arch-arm.h @@ -449,6 +449,9 @@ typedef uint64_t xen_callback_t; #define GUEST_RAM_BANK_BASES { GUEST_RAM0_BASE, GUEST_RAM1_BASE } #define GUEST_RAM_BANK_SIZES { GUEST_RAM0_SIZE, GUEST_RAM1_SIZE } +#define GUEST_EXT_REGION_MAX_SIZE xen_mk_ullong(0x2000000000) /* 128GB */ +#define GUEST_EXT_REGION_MIN_SIZE xen_mk_ullong(0x0004000000) /* 64MB */ + /* Current supported guest VCPUs */ #define GUEST_MAX_VCPUS 128