From patchwork Fri Oct 7 10:32:50 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mykyta Poturai X-Patchwork-Id: 13000971 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.xenproject.org (lists.xenproject.org [192.237.175.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2087BC433F5 for ; Fri, 7 Oct 2022 10:33:21 +0000 (UTC) Received: from list by lists.xenproject.org with outflank-mailman.417688.662459 (Exim 4.92) (envelope-from ) id 1ogkfE-0006jA-US; Fri, 07 Oct 2022 10:33:08 +0000 X-Outflank-Mailman: Message body and most headers restored to incoming version Received: by outflank-mailman (output) from mailman id 417688.662459; Fri, 07 Oct 2022 10:33:08 +0000 Received: from localhost ([127.0.0.1] helo=lists.xenproject.org) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1ogkfE-0006cb-D0; Fri, 07 Oct 2022 10:33:08 +0000 Received: by outflank-mailman (input) for mailman id 417688; Fri, 07 Oct 2022 10:33:03 +0000 Received: from se1-gles-sth1-in.inumbo.com ([159.253.27.254] helo=se1-gles-sth1.inumbo.com) by lists.xenproject.org with esmtp (Exim 4.92) (envelope-from ) id 1ogkf8-0004P4-HE for xen-devel@lists.xenproject.org; Fri, 07 Oct 2022 10:33:02 +0000 Received: from mx0b-0039f301.pphosted.com (mx0b-0039f301.pphosted.com [148.163.137.242]) by se1-gles-sth1.inumbo.com (Halon) with ESMTPS id 6a2595cc-462b-11ed-964a-05401a9f4f97; Fri, 07 Oct 2022 12:33:00 +0200 (CEST) Received: from pps.filterd (m0174680.ppops.net [127.0.0.1]) by mx0b-0039f301.pphosted.com (8.17.1.5/8.17.1.5) with ESMTP id 2977q6Qa027397; Fri, 7 Oct 2022 10:32:57 GMT Received: from eur01-db5-obe.outbound.protection.outlook.com (mail-db5eur01lp2055.outbound.protection.outlook.com [104.47.2.55]) by mx0b-0039f301.pphosted.com (PPS) with ESMTPS id 3k208bupyb-6 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 07 Oct 2022 10:32:57 +0000 Received: from VI1PR03MB3758.eurprd03.prod.outlook.com (2603:10a6:803:33::29) by DBBPR03MB6716.eurprd03.prod.outlook.com (2603:10a6:10:207::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5676.38; Fri, 7 Oct 2022 10:32:52 +0000 Received: from VI1PR03MB3758.eurprd03.prod.outlook.com ([fe80::77e8:5cf6:210:7273]) by VI1PR03MB3758.eurprd03.prod.outlook.com ([fe80::77e8:5cf6:210:7273%5]) with mapi id 15.20.5676.038; Fri, 7 Oct 2022 10:32:52 +0000 X-BeenThere: xen-devel@lists.xenproject.org List-Id: Xen developer discussion List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xenproject.org Precedence: list Sender: "Xen-devel" X-Inumbo-ID: 6a2595cc-462b-11ed-964a-05401a9f4f97 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=dR0/TwJm+ndEpUka78218ts99GPs90BfwY3cSG1mIwqscDCXnhjqCvJDJ13aJJt2zn82vLRlQ+A7LKtsChlgAvrjIbprgRFUu/wG5zRzqi/K9H5cRcpwOczVTbHUlYyeRLW9+aZX5VP7JusYCB4KyG9yw77p5ZTytDMeU7OBpDVyD8fFfbuU9SNmh/YWPu84JFBSghsSiRR4RTqiE6uF/nfWJaaYARoyZ+1W50T+fFotHE3Ljj321d3ApeSJu746WktCi1K275R5H+FtKaU6/5Z67y+zlcSs8dyU+wNnU//EV4F7ePPNxcilA5fxN9lMynRwFn95Pj/6ZrS3a6F+vA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=iUPyVMiYcwnXeZDY3vZWTB0lut8ZUfKfufVeIsYI6/w=; b=dlibaf9YhA7eE1ZB9oSwTdZPxhytviaMFRPhQp7W0o2jHultqOnbiMi0T7ZeIiBSJ5zSJS+pr1RZJ3pQJ9bex95/+5rt4qaEt4OgS92kvg07a40i19sb4cIhEya7+Kis+7owS5T0ZMNdUfGMS9vVjgKw9sBuHlopB6ttakYIKFbrMCe83PCit5P4uxLi0AR0AO1illfJR3UlqeVQYSHHfF0GpDRTHOOggsmxfHyb/jZ8Td9WHDVpiXn/7umlF4seie/KpDGhuYfJcULX+WNEd2YqJCLOuJTlNBov+EiIzkl1W9hb4aSEUi83+g/+1GuuWehKZfF3W7Z7jd9QdFPZbg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=epam.com; dmarc=pass action=none header.from=epam.com; dkim=pass header.d=epam.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=epam.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=iUPyVMiYcwnXeZDY3vZWTB0lut8ZUfKfufVeIsYI6/w=; b=Fn0iDdX2TikXB3AyAi+pOR0+uLFccRM/CeJ9RnPNdxwHwICXrWMCE7pRhrM8+3bggmpQwHtkwRRDm3pGYh5Po0+vNBZe/61biYonN6Oqp/4JMQNwnIBYeoAWWbLaL//IHK6u8/HSOIS4ktJuZU6d/i2LsYCUkkocjgGv51Yrhv+hortWzI7DhQiHB7R3Ad+FKAr4Rf9KDGGOKgly33CRiex/L9g0ryW9J8u3XBgAMSvL41Md50B6ipus8hJbr2nj/Mp6Z+hAAlDu3yfm6urx7aiREWgfP+q1ctz3MPawpyrkP145TprDU+EDCS5UUzwLKLRROJha0eVdOs/PKi6XkA== From: Mykyta Poturai To: "xen-devel@lists.xenproject.org" CC: Mirela Simonovic , Stefano Stabellini , Julien Grall , Volodymyr Babchuk , Bertrand Marquis , Saeed Nowshadi , Mykyta Poturai Subject: [PATCH 13/19] xen/arm: Resume memory management on Xen resume Thread-Topic: [PATCH 13/19] xen/arm: Resume memory management on Xen resume Thread-Index: AQHY2jgm3Yyy/xebRkCIsLQOXCUKiw== Date: Fri, 7 Oct 2022 10:32:50 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-traffictypediagnostic: VI1PR03MB3758:EE_|DBBPR03MB6716:EE_ x-ms-office365-filtering-correlation-id: 122b4de9-69b9-4d93-3cf9-08daa84f4a41 x-ld-processed: b41b72d0-4e9f-4c26-8a69-f949f367c91d,ExtAddr x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: WzColpCRZPKvEoVnX4wlJHpV8gVO/0wzHsWCJkXRkeW2I93xyhNNpB4pof3E737pFY51B3aX7h4T2WSQ7xVMqMCB9pv65AmhYt1OgwYjpXymwi030z0pE7k/XMm0blZGO4e4x921vE2wrsWDusf/K3I+L+XYIwnERJZqKNIrEZ/N7adubURg1JESZQPPXUXz01XlMRvFzowkv8Q1uy8p3FNQ3KUBKa/04/B2BX6f+SLYZ8ZkFWElNDg6mNQoThmFd1p90vg9wBfLQn6KanB4JPLKxPN+lckk2mz9sWSA3bpJUuRgFpAYRSmnqzaVSmrsikrhlQU+Ak6Knsewo4LFNHiVUZRISyiV1W4dYhiF/4hAfJWJtUwUapfT3G2HZcUUm38zCC7M97/6xQWSyC/5s3N5N/7GCZgR6QKQHaRA+EozIqI44vGnNBSMK1ZjjU1QAWZstVk0aASptnqRJ/SQVPuEp10eFOhV0fNyPsgxzEpj1kFq8kKBSS8HLlAZyebwghgn5DEvXwuj+Vlew/hxOfyL+Fijxtg6Sbp3njuH7wL/6uJ0y+kAUAzzgcpZWj4NLVtqceK/O/90tQAoSMAJtC3hb240UHyxJ6SYu8pCNnjuCNvqFp/mtkH8LRmWm080U6e1SOpJ0PI8V96p3Fza6wnzJEKV8lHbPztJ3M4NxphfqKniDy0980J87/Km/IZuDmOhufnlhcsP6YvaNwZPd0nK2FQ70G3n0SSGxaIIKUq8i8JCrDPecSjR8/k2npSnLzdwKitHJITkD9h2inkcjA== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:VI1PR03MB3758.eurprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230022)(4636009)(39860400002)(136003)(396003)(366004)(376002)(346002)(451199015)(2616005)(186003)(83380400001)(107886003)(38100700002)(38070700005)(122000001)(41300700001)(2906002)(5660300002)(8936002)(55236004)(6486002)(26005)(71200400001)(6506007)(316002)(478600001)(8676002)(54906003)(76116006)(91956017)(4326008)(64756008)(66446008)(66476007)(66946007)(66556008)(6916009)(6512007)(36756003)(86362001);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?q?Zjm/umPBq/YGOwxrp/GQ++m?= =?iso-8859-1?q?eLKXym0CwDbwjsmcHeLLQizDXkC7pMTD6/X0fPWwMgyW3ynRqnwP/ElHry3W?= =?iso-8859-1?q?v2zqSomxFRJIhbsCqcdeX9j2d/n9U+WNDLbOuOX4UiBMy03Us1rZup8KnNhP?= =?iso-8859-1?q?adCU8JCcGKtnYT1GCi/PB/1bWGJKOg/YFl+UPDgqpwckeNpTPy7Zd6lfWzen?= =?iso-8859-1?q?DY+NvcXrRP94HVLndLqJneR+ZeESCbd4DUjCGhfWgBxi6MgxJh7gq73QZts/?= =?iso-8859-1?q?H8NUY6KAQ137XHhtu9ESFdBhUHr8U5Fviq5m+wCVuy5nm3DOk9rqaF9mqnn2?= =?iso-8859-1?q?JnefMxMc8JmzpXc91j7vd6K+lLhbIb+8vItbb3ZvVnb0EoHZq4rfOQ/uy3D3?= =?iso-8859-1?q?jVGqihxQRpL4xcFP+YwhUxRTrFg19alzPKvmrbznMBRTgKY6RuiuOgmA/5q4?= =?iso-8859-1?q?OAoGH5IO9tS5B6EZh1kZ/Extv/0XQDs9nA0Zd/dQet3mD+fmZjQXQoLQxbj0?= =?iso-8859-1?q?Dk9mbJbPe8buqDa8NFaaoTc+chtLUdJyV+EsMnan5KwrN6zhOsHGAli+OgSu?= =?iso-8859-1?q?LrPB9uLj2gFv0M9v6stdHvbrrVVk5AU3hFZ47GSw4VYLAOECWJxccc5+hBpj?= =?iso-8859-1?q?FGbC67rfnqqw582biZOI9/zwj5hCCqXxCWvtyeTqAccDei8GjilwPm+5GygZ?= =?iso-8859-1?q?OoxHy7dQeWtEvioJNoBiV9cwyquQbnf7FKpprd0KabNSkITKbug40+Gt3dPl?= =?iso-8859-1?q?tANLHxxXfV159LWqOgkgNBDZA/mFphhz/ZUM5RL6uTCQaEQdw+/HXig5yPtO?= =?iso-8859-1?q?/wO1fMS/A6gAHbDQPu94f0/EXHnT3gOUw55zHeKWyWkTarczp7lPA+AaadzV?= =?iso-8859-1?q?qbaIwV88D44BV1Y8InDYQfoDbrXMh40jCWOCbkmt7iwX9TEdx//xqiamkshY?= =?iso-8859-1?q?isF8ivpKeMMw7uAAg5Ugs9baR2c2BmDYpDcj1YjUfWVUHdEXzPI0hvqoMNDa?= =?iso-8859-1?q?6llUwx+WCfxRffCKlrUPLSVCyBYMekJDA066XGBlwDK1ckPsEUd81qZstYCZ?= =?iso-8859-1?q?ssD0RZ0vwI2f1KRHMI8Z6/rbfEIZk1w+ErjgXsxF9qzJ6cHH8EVjepyG/JK0?= =?iso-8859-1?q?QjO3tgSIAr1wQd/yySZxhFCjBHe9PaWHJmM/leHSTiBKR9jnTPqvkGRPrJqO?= =?iso-8859-1?q?KYoAmd2upkBtjl5drxiALu9rBtOt8eI5b7sBts8qpwdYN4zv2WdjLCVfZFZF?= =?iso-8859-1?q?NM7R52uYPbQrRIqc2Md1WTPycn6QJBy/W6dMXdWdInt3HJoOXDGcZzYHNO74?= =?iso-8859-1?q?TblmPHnDn8KMWK15O6VekQS18JG3omkXC9NrkZzIEbvGb5MxCa615WPxM0W7?= =?iso-8859-1?q?sehpiAaJ19NogruGQKUG/1BmETGrBFTrWg58GgNXpGwkK4O2y3uXSV73npl6?= =?iso-8859-1?q?L7XFAKinH5ZJIlBthg6TYVFE1lSJoOO8vJK6BgSBIs36hO4DWm6LEGn3YWNO?= =?iso-8859-1?q?hZUfWKYptIVLXyQ6iOLXXMtAsJDZ6ZXF96p/AP6TSwZpgObQ9z+DA+kHBP52?= =?iso-8859-1?q?LcRrTp5G9Tv7kaG57dLkdBCw/rjxj9A89aataWSSX6gwJp8tHay2MmOfhl8J?= =?iso-8859-1?q?7N2CNQUqUQya/K2JVzwrrQo1I4nvhxctp7j8IYw=3D=3D?= MIME-Version: 1.0 X-OriginatorOrg: epam.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: VI1PR03MB3758.eurprd03.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 122b4de9-69b9-4d93-3cf9-08daa84f4a41 X-MS-Exchange-CrossTenant-originalarrivaltime: 07 Oct 2022 10:32:50.0677 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: b41b72d0-4e9f-4c26-8a69-f949f367c91d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: xYET/DSI/I6FVFtIM4AMvDy5Cd8hfmgqwNOJT8mCKQ/2M4x/8coZWwlX4bY0tODD7hYqGlbqVg7UASd2JwaJrQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DBBPR03MB6716 X-Proofpoint-GUID: 5tysNXNLOO9dVVVqJ4irDYcKjt4IIYyE X-Proofpoint-ORIG-GUID: 5tysNXNLOO9dVVVqJ4irDYcKjt4IIYyE X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.205,Aquarius:18.0.895,Hydra:6.0.528,FMLib:17.11.122.1 definitions=2022-10-06_05,2022-10-07_01,2022-06-22_01 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxscore=0 impostorscore=0 lowpriorityscore=0 phishscore=0 spamscore=0 malwarescore=0 priorityscore=1501 mlxlogscore=418 adultscore=0 bulkscore=0 suspectscore=0 clxscore=1015 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2209130000 definitions=main-2210070064 From: Mirela Simonovic The MMU needs to be enabled in the resume flow before the context can be restored (we need to be able to access the context data by virtual address in order to restore it). The configuration of system registers prior to branching to the routine that sets up the page tables is copied from xen/arch/arm/arm64/head.S. After the MMU is enabled, the content of TTBR0_EL2 is changed to point to init_ttbr (page tables used at runtime). At boot the init_ttbr variable is updated when a secondary CPU is hotplugged. In the scenario where there is only one physical CPU in the system, the init_ttbr would not be initialized for the use in resume flow. To get the variable initialized in all scenarios in this patch we add that the boot CPU updates init_ttbr after it sets the page tables for runtime. After the memory management is resumed, the SCTLR_WXN in SCTLR_EL2 has to be set in order to configure that a mapping cannot be both writable and executable (this was configured prior to suspend). This is done using an existing function (mmu_init_secondary_cpu). Update: moved hyp_resume to head.S to place it near the rest of the start code Signed-off-by: Mirela Simonovic Signed-off-by: Saeed Nowshadi Signed-off-by: Mykyta Poturai --- xen/arch/arm/arm64/entry.S | 2 ++ xen/arch/arm/arm64/head.S | 30 ++++++++++++++++++++++++++++++ xen/arch/arm/mm.c | 1 + xen/arch/arm/suspend.c | 6 ++++++ xen/include/asm-arm/processor.h | 22 ++++++++++++++++++++++ 5 files changed, 61 insertions(+) diff --git a/xen/arch/arm/arm64/entry.S b/xen/arch/arm/arm64/entry.S index fc3811ad0a..f49f1daf46 100644 --- a/xen/arch/arm/arm64/entry.S +++ b/xen/arch/arm/arm64/entry.S @@ -1,4 +1,6 @@ #include +#include +#include #include #include #include diff --git a/xen/arch/arm/arm64/head.S b/xen/arch/arm/arm64/head.S index 8857955699..82d83214dc 100644 --- a/xen/arch/arm/arm64/head.S +++ b/xen/arch/arm/arm64/head.S @@ -958,6 +958,36 @@ ENTRY(efi_xen_start) ENDPROC(efi_xen_start) ENTRY(hyp_resume) + msr DAIFSet, 0xf /* Disable all interrupts */ + + tlbi alle2 + dsb sy /* Ensure completion of TLB flush */ + isb + + ldr x0, =start + adr x19, start /* x19 := paddr (start) */ + sub x20, x19, x0 /* x20 := phys-offset */ + + mov x22, #0 /* x22 := is_secondary_cpu */ + + bl check_cpu_mode + bl cpu_init + bl create_page_tables + bl enable_mmu + + ldr x0, =mmu_resumed /* Explicit vaddr, not RIP-relative */ + br x0 /* Get a proper vaddr into PC */ + +mmu_resumed: + ldr x4, =init_ttbr /* VA of TTBR0_EL2 stashed by CPU 0 */ + ldr x4, [x4] /* Actual value */ + dsb sy + msr TTBR0_EL2, x4 + dsb sy + isb + tlbi alle2 + dsb sy /* Ensure completion of TLB flush */ + isb b . /* diff --git a/xen/arch/arm/mm.c b/xen/arch/arm/mm.c index eea926d823..29cdaff3bf 100644 --- a/xen/arch/arm/mm.c +++ b/xen/arch/arm/mm.c @@ -708,6 +708,7 @@ void __init setup_pagetables(unsigned long boot_phys_offset) switch_ttbr(ttbr); xen_pt_enforce_wnx(); + init_secondary_pagetables(0); #ifdef CONFIG_ARM_32 per_cpu(xen_pgtable, 0) = cpu0_pgtable; diff --git a/xen/arch/arm/suspend.c b/xen/arch/arm/suspend.c index a0258befc9..aa5ee4714b 100644 --- a/xen/arch/arm/suspend.c +++ b/xen/arch/arm/suspend.c @@ -167,6 +167,12 @@ static long system_suspend(void *data) system_state = SYS_STATE_resume; + /* + * SCTLR_WXN needs to be set to configure that a mapping cannot be both + * writable and executable. This is done by mmu_init_secondary_cpu. + */ + mmu_init_secondary_cpu(); + gic_resume(); resume_irqs: diff --git a/xen/include/asm-arm/processor.h b/xen/include/asm-arm/processor.h index 8ab2940f68..ecf97f1ab4 100644 --- a/xen/include/asm-arm/processor.h +++ b/xen/include/asm-arm/processor.h @@ -133,6 +133,28 @@ #define TTBCR_PD1 (_AC(1,U)<<5) /* SCTLR System Control Register. */ +/* HSCTLR is a subset of this. */ +#define SCTLR_TE (_AC(1,U)<<30) +#define SCTLR_AFE (_AC(1,U)<<29) +#define SCTLR_TRE (_AC(1,U)<<28) +#define SCTLR_NMFI (_AC(1,U)<<27) +#define SCTLR_EE (_AC(1,U)<<25) +#define SCTLR_VE (_AC(1,U)<<24) +#define SCTLR_U (_AC(1,U)<<22) +#define SCTLR_FI (_AC(1,U)<<21) +#define SCTLR_WXN (_AC(1,U)<<19) +#define SCTLR_HA (_AC(1,U)<<17) +#define SCTLR_RR (_AC(1,U)<<14) +#define SCTLR_V (_AC(1,U)<<13) +#define SCTLR_I (_AC(1,U)<<12) +#define SCTLR_Z (_AC(1,U)<<11) +#define SCTLR_SW (_AC(1,U)<<10) +#define SCTLR_B (_AC(1,U)<<7) +#define SCTLR_C (_AC(1,U)<<2) +#define SCTLR_A (_AC(1,U)<<1) +#define SCTLR_M (_AC(1,U)<<0) + +#define HSCTLR_BASE _AC(0x30c51878,U) /* Bits specific to SCTLR_EL1 for Arm32 */