From patchwork Wed Jan 27 17:27:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12050683 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CEF73C433DB for ; Wed, 27 Jan 2021 17:35:40 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1856664DA0 for ; Wed, 27 Jan 2021 17:35:40 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1856664DA0 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:List-Subscribe:List-Help:List-Post:List-Archive:List-Unsubscribe :List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:To:From: Reply-To:Cc:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=a6ywbOxFuOV9Zz0N8FvTx/aqE46ipubT08+o0vaw9IE=; b=W3+6f4EXY+SfacBXV/+cImL8cf Za/Sa9loRltA9BbIMZICBxonHuzZxTHiEztHTvER500ae3sQrOQEq2rlCMvWSM+oqgQXC5G8p5X4C 7RzfyN7DgdpLXsFRmYh5RwCk52aodw04P7D8EBOFuzQVEidraW32jevcs2g7IR2j+KpuvU1IAk1qu IGh88p/xXwMeHT4IQ2OTclT7h3pmpI/S5tiwxfUX5ehY0v14oca/Z4EXdRrOKarU2bLmSR5OhFrL+ myTX3GJBx8Z4gEwSuX9gwLK/d0X7Dwz8ZTyqVbVC/Dt4L8+IjmgbG/vSedLyysONc8pKl7PdHwjod O9IkYOgg==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l4ohF-0000MQ-Ku; Wed, 27 Jan 2021 17:33:37 +0000 Received: from mail-qv1-xf29.google.com ([2607:f8b0:4864:20::f29]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1l4ob4-0005or-0y for linux-arm-kernel@lists.infradead.org; Wed, 27 Jan 2021 17:27:24 +0000 Received: by mail-qv1-xf29.google.com with SMTP id h21so1420398qvb.8 for ; Wed, 27 Jan 2021 09:27:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=ASygAqr57jUZ78nCjocRBoBMKp4WVYWpgcDuzWEf5/U=; b=lTW8AsSeGiTqhrvoe5+DBlxNbXH/Jl+2xyBWNDwB76Bg8MNCuvd6VNwkmJ4+h63cZX cuq1Hb5sgFA9btCgFJ1qlG7gFOO9yjiPdnvbbADDuICXgW4ePbFURxqoTvJbgj35gXLl cYTYx1IjAV587qwyGmaRdylhQw3z11+nFnsBEcq0EOLmQsJWymmY++T5KvljbsHwL5TZ JKP3aTXz5mdnMHXMT1rL/M2j51IVvCKZBnD/YmNSWiVDpKFPuhgDLUuH9M8INScH3Ywl j8bNzL4bTdELqLfTU0vHpDFzwa+6thecSerPkkUNYqHGh3aUQs9T6I/ptCm+mYIUblHA Hlfg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ASygAqr57jUZ78nCjocRBoBMKp4WVYWpgcDuzWEf5/U=; b=Aao5P//4IlS3GMenxDS+yuoWcOXJGRxa85d0mknK7LXGBan0HoYQzMCrwrhdKBiKII HQcfHOCxOXun7P2VzHjhaXZSmksIpcD7CIaEWr7v7+XtI7ZKRMwS6OcE8AD8FkONI55e LYzqT6sNS2HaTZgmrqoqgFc+D2W/Vgmxn+137MDj/gKE9LzfCMtsM4whepWedLu7HmT3 trT6ipdEJJsuGaBmAOp84j4MqH/Zs5yUIFya9OZkH2Cm0u44Ue89UYNUGATv5hlHk1Ys Mu+Dj+G7MMb6BGsMcSbfcWJgernQCGpWHkpwfI96W3p5AyAdzFY9q0jOHu9qZnGSNto4 UTXQ== X-Gm-Message-State: AOAM530zamVDwpNq4L1kKVRplSVxyWezTCr27AO2w4MfmRIsE+2cYvty OwbilWIAyEQ5C40e7DQYLubU+g== X-Google-Smtp-Source: ABdhPJxIcrcK4iDLzYXxJWKQD6eG4blKjf34bscTuC6KLr/1IGH2MCDgmQpbj2CQsy/4e2VNcDPU/g== X-Received: by 2002:a0c:a8e7:: with SMTP id h39mr11424788qvc.45.1611768430096; Wed, 27 Jan 2021 09:27:10 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id z2sm1728019qtz.71.2021.01.27.09.27.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Jan 2021 09:27:09 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, jmorris@namei.org, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, maz@kernel.org, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, linux-mm@kvack.org, mark.rutland@arm.com, steve.capper@arm.com, rfontana@redhat.com, tglx@linutronix.de, selindag@gmail.com, tyhicks@linux.microsoft.com Subject: [PATCH v11 1/6] arm64: kexec: add expandable argument to relocation function Date: Wed, 27 Jan 2021 12:27:01 -0500 Message-Id: <20210127172706.617195-2-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210127172706.617195-1-pasha.tatashin@soleen.com> References: <20210127172706.617195-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210127_122714_499309_5DFDAFE2 X-CRM114-Status: GOOD ( 26.60 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Currently, kexec relocation function (arm64_relocate_new_kernel) accepts the following arguments: head: start of array that contains relocation information. entry: entry point for new kernel or purgatory. dtb_mem: first and only argument to entry. The number of arguments cannot be easily expended, because this function is also called from HVC_SOFT_RESTART, which preserves only three arguments (hypervisor abi). And, also arm64_relocate_new_kernel is written in assembly but called without stack, thus no place to move extra arguments to free registers. Soon, we will need to pass more arguments: once we enable MMU we will need to pass information about page tables. Add a new struct: kern_reloc_arg, and place it in kexec safe page (i.e memory that is not overwritten during relocation). Thus, make arm64_relocate_new_kernel to only take one argument, that contains all the needed information. Note: Another benefit of allowing this function to accept more arguments, is that kernel can actually accept up to 4 arguments (x0-x3), however currently only one is used, but if in the future we will need for more (for example, pass information about when previous kernel exited to have a precise measurement in time spent in purgatory), we won't be easilty do that if arm64_relocate_new_kernel can't accept more arguments. Signed-off-by: Pavel Tatashin --- arch/arm64/include/asm/kexec.h | 18 ++++++++++++++++++ arch/arm64/kernel/asm-offsets.c | 9 +++++++++ arch/arm64/kernel/cpu-reset.S | 11 +++-------- arch/arm64/kernel/cpu-reset.h | 8 +++----- arch/arm64/kernel/machine_kexec.c | 27 +++++++++++++++++++++++++-- arch/arm64/kernel/relocate_kernel.S | 21 ++++++++------------- 6 files changed, 66 insertions(+), 28 deletions(-) diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h index 9befcd87e9a8..990185744148 100644 --- a/arch/arm64/include/asm/kexec.h +++ b/arch/arm64/include/asm/kexec.h @@ -90,12 +90,30 @@ static inline void crash_prepare_suspend(void) {} static inline void crash_post_resume(void) {} #endif +/* + * kern_reloc_arg is passed to kernel relocation function as an argument. + * head kimage->head, allows to traverse through relocation segments. + * entry_addr kimage->start, where to jump from relocation function (new + * kernel, or purgatory entry address). + * kern_arg0 first argument to kernel is its dtb address. The other + * arguments are currently unused, and must be set to 0 + */ +struct kern_reloc_arg { + phys_addr_t head; + phys_addr_t entry_addr; + phys_addr_t kern_arg0; + phys_addr_t kern_arg1; + phys_addr_t kern_arg2; + phys_addr_t kern_arg3; +}; + #define ARCH_HAS_KIMAGE_ARCH struct kimage_arch { void *dtb; phys_addr_t dtb_mem; phys_addr_t kern_reloc; + phys_addr_t kern_reloc_arg; /* Core ELF header buffer */ void *elf_headers; unsigned long elf_headers_mem; diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index 301784463587..6067a288f568 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -23,6 +23,7 @@ #include #include #include +#include int main(void) { @@ -150,6 +151,14 @@ int main(void) DEFINE(PTRAUTH_USER_KEY_APGA, offsetof(struct ptrauth_keys_user, apga)); DEFINE(PTRAUTH_KERNEL_KEY_APIA, offsetof(struct ptrauth_keys_kernel, apia)); BLANK(); +#endif +#ifdef CONFIG_KEXEC_CORE + DEFINE(KEXEC_KRELOC_HEAD, offsetof(struct kern_reloc_arg, head)); + DEFINE(KEXEC_KRELOC_ENTRY_ADDR, offsetof(struct kern_reloc_arg, entry_addr)); + DEFINE(KEXEC_KRELOC_KERN_ARG0, offsetof(struct kern_reloc_arg, kern_arg0)); + DEFINE(KEXEC_KRELOC_KERN_ARG1, offsetof(struct kern_reloc_arg, kern_arg1)); + DEFINE(KEXEC_KRELOC_KERN_ARG2, offsetof(struct kern_reloc_arg, kern_arg2)); + DEFINE(KEXEC_KRELOC_KERN_ARG3, offsetof(struct kern_reloc_arg, kern_arg3)); #endif return 0; } diff --git a/arch/arm64/kernel/cpu-reset.S b/arch/arm64/kernel/cpu-reset.S index 37721eb6f9a1..bbf70db43744 100644 --- a/arch/arm64/kernel/cpu-reset.S +++ b/arch/arm64/kernel/cpu-reset.S @@ -16,14 +16,11 @@ .pushsection .idmap.text, "awx" /* - * __cpu_soft_restart(el2_switch, entry, arg0, arg1, arg2) - Helper for - * cpu_soft_restart. + * __cpu_soft_restart(el2_switch, entry, arg) - Helper for cpu_soft_restart. * * @el2_switch: Flag to indicate a switch to EL2 is needed. * @entry: Location to jump to for soft reset. - * arg0: First argument passed to @entry. (relocation list) - * arg1: Second argument passed to @entry.(physical kernel entry) - * arg2: Third argument passed to @entry. (physical dtb address) + * arg: Entry argument * * Put the CPU into the same state as it would be if it had been reset, and * branch to what would be the reset vector. It must be executed with the @@ -47,9 +44,7 @@ SYM_CODE_START(__cpu_soft_restart) hvc #0 // no return 1: mov x8, x1 // entry - mov x0, x2 // arg0 - mov x1, x3 // arg1 - mov x2, x4 // arg2 + mov x0, x2 // arg br x8 SYM_CODE_END(__cpu_soft_restart) diff --git a/arch/arm64/kernel/cpu-reset.h b/arch/arm64/kernel/cpu-reset.h index ed50e9587ad8..7a8720ff186f 100644 --- a/arch/arm64/kernel/cpu-reset.h +++ b/arch/arm64/kernel/cpu-reset.h @@ -11,12 +11,10 @@ #include void __cpu_soft_restart(unsigned long el2_switch, unsigned long entry, - unsigned long arg0, unsigned long arg1, unsigned long arg2); + unsigned long arg); static inline void __noreturn cpu_soft_restart(unsigned long entry, - unsigned long arg0, - unsigned long arg1, - unsigned long arg2) + unsigned long arg) { typeof(__cpu_soft_restart) *restart; @@ -25,7 +23,7 @@ static inline void __noreturn cpu_soft_restart(unsigned long entry, restart = (void *)__pa_symbol(__cpu_soft_restart); cpu_install_idmap(); - restart(el2_switch, entry, arg0, arg1, arg2); + restart(el2_switch, entry, arg); unreachable(); } diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c index 90a335c74442..679db3f1e0c5 100644 --- a/arch/arm64/kernel/machine_kexec.c +++ b/arch/arm64/kernel/machine_kexec.c @@ -43,6 +43,7 @@ static void _kexec_image_info(const char *func, int line, pr_debug(" head: %lx\n", kimage->head); pr_debug(" nr_segments: %lu\n", kimage->nr_segments); pr_debug(" kern_reloc: %pa\n", &kimage->arch.kern_reloc); + pr_debug(" kern_reloc_arg: %pa\n", &kimage->arch.kern_reloc_arg); for (i = 0; i < kimage->nr_segments; i++) { pr_debug(" segment[%lu]: %016lx - %016lx, 0x%lx bytes, %lu pages\n", @@ -59,19 +60,42 @@ void machine_kexec_cleanup(struct kimage *kimage) /* Empty routine needed to avoid build errors. */ } +/* Allocates pages for kexec page table */ +static void *kexec_page_alloc(void *arg) +{ + struct kimage *kimage = (struct kimage *)arg; + struct page *page = kimage_alloc_control_pages(kimage, 0); + + if (!page) + return NULL; + + memset(page_address(page), 0, PAGE_SIZE); + + return page_address(page); +} + int machine_kexec_post_load(struct kimage *kimage) { void *reloc_code = page_to_virt(kimage->control_code_page); + struct kern_reloc_arg *kern_reloc_arg = kexec_page_alloc(kimage); + + if (!kern_reloc_arg) + return -ENOMEM; memcpy(reloc_code, arm64_relocate_new_kernel, arm64_relocate_new_kernel_size); kimage->arch.kern_reloc = __pa(reloc_code); + kimage->arch.kern_reloc_arg = __pa(kern_reloc_arg); + kern_reloc_arg->head = kimage->head; + kern_reloc_arg->entry_addr = kimage->start; + kern_reloc_arg->kern_arg0 = kimage->arch.dtb_mem; kexec_image_info(kimage); /* Flush the reloc_code in preparation for its execution. */ __flush_dcache_area(reloc_code, arm64_relocate_new_kernel_size); flush_icache_range((uintptr_t)reloc_code, (uintptr_t)reloc_code + arm64_relocate_new_kernel_size); + __flush_dcache_area(kern_reloc_arg, sizeof(struct kern_reloc_arg)); return 0; } @@ -192,8 +216,7 @@ void machine_kexec(struct kimage *kimage) * userspace (kexec-tools). * In kexec_file case, the kernel starts directly without purgatory. */ - cpu_soft_restart(kimage->arch.kern_reloc, kimage->head, kimage->start, - kimage->arch.dtb_mem); + cpu_soft_restart(kimage->arch.kern_reloc, kimage->arch.kern_reloc_arg); BUG(); /* Should never get here. */ } diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S index b78ea5de97a4..c92228aeddca 100644 --- a/arch/arm64/kernel/relocate_kernel.S +++ b/arch/arm64/kernel/relocate_kernel.S @@ -8,7 +8,7 @@ #include #include - +#include #include #include #include @@ -26,13 +26,8 @@ * safe memory that has been set up to be preserved during the copy operation. */ SYM_CODE_START(arm64_relocate_new_kernel) - /* Setup the list loop variables. */ - mov x18, x2 /* x18 = dtb address */ - mov x17, x1 /* x17 = kimage_start */ - mov x16, x0 /* x16 = kimage_head */ - mov x14, xzr /* x14 = entry ptr */ - mov x13, xzr /* x13 = copy dest */ /* Check if the new image needs relocation. */ + ldr x16, [x0, #KEXEC_KRELOC_HEAD] /* x16 = kimage_head */ tbnz x16, IND_DONE_BIT, .Ldone raw_dcache_line_size x15, x1 /* x15 = dcache line size */ .Lloop: @@ -73,12 +68,12 @@ SYM_CODE_START(arm64_relocate_new_kernel) isb /* Start new image. */ - mov x0, x18 - mov x1, xzr - mov x2, xzr - mov x3, xzr - br x17 - + ldr x4, [x0, #KEXEC_KRELOC_ENTRY_ADDR] /* x4 = kimage_start */ + ldr x3, [x0, #KEXEC_KRELOC_KERN_ARG3] + ldr x2, [x0, #KEXEC_KRELOC_KERN_ARG2] + ldr x1, [x0, #KEXEC_KRELOC_KERN_ARG1] + ldr x0, [x0, #KEXEC_KRELOC_KERN_ARG0] /* x0 = dtb address */ + br x4 SYM_CODE_END(arm64_relocate_new_kernel) .align 3 /* To keep the 64-bit values below naturally aligned. */ From patchwork Wed Jan 27 17:27:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12050681 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 48D8AC433DB for ; Wed, 27 Jan 2021 17:35:17 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id BBC1160187 for ; Wed, 27 Jan 2021 17:35:16 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BBC1160187 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:List-Subscribe:List-Help:List-Post:List-Archive:List-Unsubscribe :List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:To:From: Reply-To:Cc:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=5+oI0Cyas0ypQoYbjUnCAnh58tcyHC22teXAVnJeNpw=; b=Cep2mcULxtq6/20v3lB0g74Moo Q/t8My9JCXYO3jRqhP2R+fQDgDo/WBN35RLcQt5OHgTTgrjVrdsX6Fz1cqNTKy792Cls7+lDMA7jD ddFFiGNx4fps3PlkHFAzYnZCpe/6uAg+u+buWidJp9r7ZPXlIsQlbWwy38hyRCraOC5VnXBuW+6AY 5JnpK5jnMoINAisPySZD4JR8XVFpK7Hq27kSUuXSQ9fvT0aHkvgHY/nBHu6UknT/Zk5TbNjIEG8vv 7rH7efCUsmeX/tHq0F5kAEloKzd5jwSv/0dHpw5w7FjZT9sxBJv9dSFZAYC/TKA+E/YAdxRWyQMK6 tEgqrv9Q==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l4ogn-0008Pv-SF; Wed, 27 Jan 2021 17:33:10 +0000 Received: from mail-qt1-x832.google.com ([2607:f8b0:4864:20::832]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1l4ob5-0005pl-HV for linux-arm-kernel@lists.infradead.org; Wed, 27 Jan 2021 17:27:21 +0000 Received: by mail-qt1-x832.google.com with SMTP id v3so1973219qtw.4 for ; Wed, 27 Jan 2021 09:27:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=iM7vyMjvp8HFtXYgFN4DAarGfV3AZjiwX6w+vbwvdyg=; b=QMOtRowsZT/Ru/xiPTLv/7JFh1hwOeF1FlFu0MyMoJe50YZ0ukuVerCY5joYBmYk/j dENZG3TuOn7AB27MiwjTj0TrcFWdGNygOEnZ4oGtC8PD8Du3kTYT6MOH+q217OZij3VA xMIee8OoRMSkHIpRftlWwViPysU80FoFq3KYkOC9aLDwgio7a8z5Ksf6AILaSVU0Gxha mSMwr5P5KU4POmhpF7+VQycvc+a+IS3S+icaRsghxQ5TaGvRsRibJJlSlzRdHCgkriq2 mJAX/hxhmlpNg6OOMCgLkRiiCUisawlzo9BxNh/nMUteSZE2Uaa4Vf0zsuQq+CBuEALt cl1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=iM7vyMjvp8HFtXYgFN4DAarGfV3AZjiwX6w+vbwvdyg=; b=CQSOr6nhZXNA1Minl4VXQC9C8hSITnMkKuOj4O/490Ip5ult2+VsZUq7u8fFzjlvDG ZGqRsuQvmuHcEJ+vm2OrxMbxjBRODL/z8FCqEc6P0roJ9YUW7Rh/JGu50SoPSvWuyiez gsnKhpHxod27gGKsiPi6oJYOTQnYH6VaqnPmlOHjxntoARPzFpNnh8pylVRvzFQ8bwpm +Lcj/ENHmKyqz5XkKgCfY8Qw9NSmlWpWj6tS5V9JTEhQv3DKOratRQz2eNc2Oi/M4qno rKk7i2RQQ3xJMGdYOLjybVqFxmMFF+s4y/oUQGfs/+LI46YsjJ+4wJzGDZr3rfKWpRLS f6Dw== X-Gm-Message-State: AOAM533bITmMAAtFkF2GTVIkhKz3B9AEPBko1oCGgbJr7u/itXUqVZjM 4wZz9RI/iofcTsd5D/xdNSt4ug== X-Google-Smtp-Source: ABdhPJz2pmeuW3zsZ09pL8xkdh0pAeu9616r0G+I0/+leO2zQYI1rzh9PJ65p7HB/lI/13nJfQJHcA== X-Received: by 2002:ac8:118c:: with SMTP id d12mr10787471qtj.262.1611768431629; Wed, 27 Jan 2021 09:27:11 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id z2sm1728019qtz.71.2021.01.27.09.27.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Jan 2021 09:27:11 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, jmorris@namei.org, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, maz@kernel.org, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, linux-mm@kvack.org, mark.rutland@arm.com, steve.capper@arm.com, rfontana@redhat.com, tglx@linutronix.de, selindag@gmail.com, tyhicks@linux.microsoft.com Subject: [PATCH v11 2/6] arm64: kexec: use ld script for relocation function Date: Wed, 27 Jan 2021 12:27:02 -0500 Message-Id: <20210127172706.617195-3-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210127172706.617195-1-pasha.tatashin@soleen.com> References: <20210127172706.617195-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210127_122715_832938_84B39176 X-CRM114-Status: GOOD ( 18.08 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Currently, relocation code declares start and end variables which are used to compute it size. The better way to do this is to use ld script incited, and put relocation function in its own section. Soon, relocation function will share the same page with EL2 vectors. So, proper marking is needed. Signed-off-by: Pavel Tatashin --- arch/arm64/include/asm/kexec.h | 4 ++++ arch/arm64/include/asm/sections.h | 1 + arch/arm64/kernel/machine_kexec.c | 17 ++++++++--------- arch/arm64/kernel/relocate_kernel.S | 15 ++------------- arch/arm64/kernel/vmlinux.lds.S | 19 +++++++++++++++++++ 5 files changed, 34 insertions(+), 22 deletions(-) diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h index 990185744148..7f4f9abdf049 100644 --- a/arch/arm64/include/asm/kexec.h +++ b/arch/arm64/include/asm/kexec.h @@ -90,6 +90,10 @@ static inline void crash_prepare_suspend(void) {} static inline void crash_post_resume(void) {} #endif +#if defined(CONFIG_KEXEC_CORE) +extern const char arm64_relocate_new_kernel[]; +#endif + /* * kern_reloc_arg is passed to kernel relocation function as an argument. * head kimage->head, allows to traverse through relocation segments. diff --git a/arch/arm64/include/asm/sections.h b/arch/arm64/include/asm/sections.h index 8ff579361731..ae873eb22205 100644 --- a/arch/arm64/include/asm/sections.h +++ b/arch/arm64/include/asm/sections.h @@ -19,5 +19,6 @@ extern char __exittext_begin[], __exittext_end[]; extern char __irqentry_text_start[], __irqentry_text_end[]; extern char __mmuoff_data_start[], __mmuoff_data_end[]; extern char __entry_tramp_text_start[], __entry_tramp_text_end[]; +extern char __relocate_new_kernel_start[], __relocate_new_kernel_end[]; #endif /* __ASM_SECTIONS_H */ diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c index 679db3f1e0c5..361a4d082093 100644 --- a/arch/arm64/kernel/machine_kexec.c +++ b/arch/arm64/kernel/machine_kexec.c @@ -20,13 +20,10 @@ #include #include #include +#include #include "cpu-reset.h" -/* Global variables for the arm64_relocate_new_kernel routine. */ -extern const unsigned char arm64_relocate_new_kernel[]; -extern const unsigned long arm64_relocate_new_kernel_size; - /** * kexec_image_info - For debugging output. */ @@ -78,13 +75,15 @@ int machine_kexec_post_load(struct kimage *kimage) { void *reloc_code = page_to_virt(kimage->control_code_page); struct kern_reloc_arg *kern_reloc_arg = kexec_page_alloc(kimage); + long func_offset, reloc_size; if (!kern_reloc_arg) return -ENOMEM; - memcpy(reloc_code, arm64_relocate_new_kernel, - arm64_relocate_new_kernel_size); - kimage->arch.kern_reloc = __pa(reloc_code); + func_offset = arm64_relocate_new_kernel - __relocate_new_kernel_start; + reloc_size = __relocate_new_kernel_end - __relocate_new_kernel_start; + memcpy(reloc_code, __relocate_new_kernel_start, reloc_size); + kimage->arch.kern_reloc = __pa(reloc_code) + func_offset; kimage->arch.kern_reloc_arg = __pa(kern_reloc_arg); kern_reloc_arg->head = kimage->head; kern_reloc_arg->entry_addr = kimage->start; @@ -92,9 +91,9 @@ int machine_kexec_post_load(struct kimage *kimage) kexec_image_info(kimage); /* Flush the reloc_code in preparation for its execution. */ - __flush_dcache_area(reloc_code, arm64_relocate_new_kernel_size); + __flush_dcache_area(reloc_code, reloc_size); flush_icache_range((uintptr_t)reloc_code, (uintptr_t)reloc_code + - arm64_relocate_new_kernel_size); + reloc_size); __flush_dcache_area(kern_reloc_arg, sizeof(struct kern_reloc_arg)); return 0; diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S index c92228aeddca..d2a4a0b0d76b 100644 --- a/arch/arm64/kernel/relocate_kernel.S +++ b/arch/arm64/kernel/relocate_kernel.S @@ -14,6 +14,7 @@ #include #include +.pushsection ".kexec_relocate.text", "ax" /* * arm64_relocate_new_kernel - Put a 2nd stage image in place and boot it. * @@ -75,16 +76,4 @@ SYM_CODE_START(arm64_relocate_new_kernel) ldr x0, [x0, #KEXEC_KRELOC_KERN_ARG0] /* x0 = dtb address */ br x4 SYM_CODE_END(arm64_relocate_new_kernel) - -.align 3 /* To keep the 64-bit values below naturally aligned. */ - -.Lcopy_end: -.org KEXEC_CONTROL_PAGE_SIZE - -/* - * arm64_relocate_new_kernel_size - Number of bytes to copy to the - * control_code_page. - */ -.globl arm64_relocate_new_kernel_size -arm64_relocate_new_kernel_size: - .quad .Lcopy_end - arm64_relocate_new_kernel +.popsection diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S index 4c0b0c89ad59..33b0d3c9fd3b 100644 --- a/arch/arm64/kernel/vmlinux.lds.S +++ b/arch/arm64/kernel/vmlinux.lds.S @@ -12,6 +12,7 @@ #include #include #include +#include #include #include @@ -82,6 +83,16 @@ jiffies = jiffies_64; #define HIBERNATE_TEXT #endif +#ifdef CONFIG_KEXEC_CORE +#define KEXEC_TEXT \ + . = ALIGN(SZ_4K); \ + __relocate_new_kernel_start = .; \ + *(.kexec_relocate.text) \ + __relocate_new_kernel_end = .; +#else +#define KEXEC_TEXT +#endif + #ifdef CONFIG_UNMAP_KERNEL_AT_EL0 #define TRAMP_TEXT \ . = ALIGN(PAGE_SIZE); \ @@ -142,6 +153,7 @@ SECTIONS HYPERVISOR_TEXT IDMAP_TEXT HIBERNATE_TEXT + KEXEC_TEXT TRAMP_TEXT *(.fixup) *(.gnu.warning) @@ -316,3 +328,10 @@ ASSERT((__entry_tramp_text_end - __entry_tramp_text_start) == PAGE_SIZE, * If padding is applied before .head.text, virt<->phys conversions will fail. */ ASSERT(_text == KIMAGE_VADDR, "HEAD is misaligned") + +#ifdef CONFIG_KEXEC_CORE +/* kexec relocation code should fit into one KEXEC_CONTROL_PAGE_SIZE */ +ASSERT(__relocate_new_kernel_end - (__relocate_new_kernel_start & ~(SZ_4K - 1)) + <= SZ_4K, "kexec relocation code is too big or misaligned") +ASSERT(KEXEC_CONTROL_PAGE_SIZE >= SZ_4K, "KEXEC_CONTROL_PAGE_SIZE is brokern") +#endif From patchwork Wed Jan 27 17:27:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12050685 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 03A68C433E6 for ; Wed, 27 Jan 2021 17:36:17 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4D77E64DA6 for ; Wed, 27 Jan 2021 17:36:16 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4D77E64DA6 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:List-Subscribe:List-Help:List-Post:List-Archive:List-Unsubscribe :List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:To:From: Reply-To:Cc:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=r1t657QmlXGq4RI+1wOInYwbqlb1bfJNsUr39mz3UfA=; b=eX44RLJvzJ33DSkzJJczs18PsW 3ARJlHgJr4s92HrDpi5O0K1OC5SIy73+AyOQAuRpW4VYTksZ2WDsK3rG3cFRc3XxsFeFRgP6DD5hK GdxOHGBhNTpI4qI/O//nr3P2Jc3GFgzh00xf3IBrDN3LyMO2065swd/RWI9zv5NXGEeLe6SNHIaLf yERMVTb/EL1+dfjOgHPPczhj4MTI9NjWMRNwL2ZSbNhB0Rz2Y+uBCc8fSdFgGKsiKZhf9K7c8Y+0R 10hQMUQmZ/kBkacOg3y6JhqzPccCcM0DTQwInkfjrko5IoaJys8d0XpO1kLGqlQiCb3Itvb0AB7en za4M+O9A==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l4ohe-0000ZM-S7; Wed, 27 Jan 2021 17:34:04 +0000 Received: from mail-qt1-x834.google.com ([2607:f8b0:4864:20::834]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1l4ob7-0005q6-5h for linux-arm-kernel@lists.infradead.org; Wed, 27 Jan 2021 17:27:28 +0000 Received: by mail-qt1-x834.google.com with SMTP id c1so1983733qtc.1 for ; Wed, 27 Jan 2021 09:27:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=lbvVstnavugORmz+7RoYwB1ceirdtrsJhtHKQ5onWhE=; b=QV+R8Re8BFcvEa3KJeWhAPQnSVBkbq5dabdVCP2CjZZ8ZSwcwRvXqu83WHuCg68EzE pP/iQOyEwDyg1Iqy+YnkFzs5hZjA8YahurgylMOElXDiWUjz08sJoZL7cLIybd2FokX4 /AbrCaUi6+ckqv6Ai7V2lFdBNm0v1Ee6D30znoHnux3cOcDF8kaLEoB3XNte8NUC1/V9 mz4tMZapzAEf3Tz97RChN/qhHCaBbbUGMJcCsJihf073iEYNlIIChdftQDBryKBA5mDd ewU32BTqwSlRsSmXhTQGNNPz/Jn6wVmbysMfXLLufw9WsmDRthUkfBaXWHLYiETpzofh EMAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=lbvVstnavugORmz+7RoYwB1ceirdtrsJhtHKQ5onWhE=; b=CsyB8eNFN9wXzwPfw+xORKzxLl+gxocrtZTpsTMiclQTU1MfJ0dv2MMGFOqi3dECAU 5kRE/LjiT49RegNUYAH5pk14qaf5h5ovDRtgOsNm3JQsSOnPzEU6geYphgb30nvYRhWS +CCEOJCcWEo3aRzU0mkl5x6oTBQiVEop1RvHnYCorEbkRudXO39S6cO2ra9LNUhBTLmp UDbVyX7DShIZawTHftqeM3vu5NmjQf4fU/pMWlIEcfgsdQgbwoyw1mH6ZoyIEkVybHJB VN3eRsq64E7mU7D5QcbBNnyyO+fH9HZXJcBHBmh2jDSFGRVOhgdd1k/T2q/SMyHc3bD6 MX4A== X-Gm-Message-State: AOAM532AiK8KrU/J7LnW/GGV7LKkc91SBAXWADVINvJ1lWGMJcRZnODN uARKeGWmaIVfyIvxE3hVhkWAKUE0Py/T7P0T X-Google-Smtp-Source: ABdhPJz6VAe7Hg9z00hIz2UDa6TgtFcqklDUbjVYu90Qteg2VkXL4vXeuEwM1lTUmzQASkqOin5HYw== X-Received: by 2002:a05:622a:201:: with SMTP id b1mr10304370qtx.237.1611768433197; Wed, 27 Jan 2021 09:27:13 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id z2sm1728019qtz.71.2021.01.27.09.27.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Jan 2021 09:27:12 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, jmorris@namei.org, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, maz@kernel.org, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, linux-mm@kvack.org, mark.rutland@arm.com, steve.capper@arm.com, rfontana@redhat.com, tglx@linutronix.de, selindag@gmail.com, tyhicks@linux.microsoft.com Subject: [PATCH v11 3/6] arm64: kexec: kexec may require EL2 vectors Date: Wed, 27 Jan 2021 12:27:03 -0500 Message-Id: <20210127172706.617195-4-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210127172706.617195-1-pasha.tatashin@soleen.com> References: <20210127172706.617195-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210127_122717_533064_ED5AA99A X-CRM114-Status: GOOD ( 18.20 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org If we have a EL2 mode without VHE, the EL2 vectors are needed in order to switch to EL2 and jump to new world with hypervisor privileges. Signed-off-by: Pavel Tatashin --- arch/arm64/include/asm/kexec.h | 5 +++++ arch/arm64/kernel/asm-offsets.c | 1 + arch/arm64/kernel/machine_kexec.c | 9 +++++++- arch/arm64/kernel/relocate_kernel.S | 35 +++++++++++++++++++++++++++++ 4 files changed, 49 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h index 7f4f9abdf049..b96d8a6aac80 100644 --- a/arch/arm64/include/asm/kexec.h +++ b/arch/arm64/include/asm/kexec.h @@ -92,6 +92,7 @@ static inline void crash_post_resume(void) {} #if defined(CONFIG_KEXEC_CORE) extern const char arm64_relocate_new_kernel[]; +extern const char arm64_kexec_el2_vectors[]; #endif /* @@ -101,6 +102,9 @@ extern const char arm64_relocate_new_kernel[]; * kernel, or purgatory entry address). * kern_arg0 first argument to kernel is its dtb address. The other * arguments are currently unused, and must be set to 0 + * el2_vector If present means that relocation routine will go to EL1 + * from EL2 to do the copy, and then back to EL2 to do the jump + * to new world. */ struct kern_reloc_arg { phys_addr_t head; @@ -109,6 +113,7 @@ struct kern_reloc_arg { phys_addr_t kern_arg1; phys_addr_t kern_arg2; phys_addr_t kern_arg3; + phys_addr_t el2_vector; }; #define ARCH_HAS_KIMAGE_ARCH diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index 6067a288f568..8a9475be1b62 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -159,6 +159,7 @@ int main(void) DEFINE(KEXEC_KRELOC_KERN_ARG1, offsetof(struct kern_reloc_arg, kern_arg1)); DEFINE(KEXEC_KRELOC_KERN_ARG2, offsetof(struct kern_reloc_arg, kern_arg2)); DEFINE(KEXEC_KRELOC_KERN_ARG3, offsetof(struct kern_reloc_arg, kern_arg3)); + DEFINE(KEXEC_KRELOC_EL2_VECTOR, offsetof(struct kern_reloc_arg, el2_vector)); #endif return 0; } diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c index 361a4d082093..41d1e3ca13f8 100644 --- a/arch/arm64/kernel/machine_kexec.c +++ b/arch/arm64/kernel/machine_kexec.c @@ -75,19 +75,26 @@ int machine_kexec_post_load(struct kimage *kimage) { void *reloc_code = page_to_virt(kimage->control_code_page); struct kern_reloc_arg *kern_reloc_arg = kexec_page_alloc(kimage); - long func_offset, reloc_size; + long func_offset, vector_offset, reloc_size; if (!kern_reloc_arg) return -ENOMEM; func_offset = arm64_relocate_new_kernel - __relocate_new_kernel_start; reloc_size = __relocate_new_kernel_end - __relocate_new_kernel_start; + vector_offset = arm64_kexec_el2_vectors - __relocate_new_kernel_start; + memcpy(reloc_code, __relocate_new_kernel_start, reloc_size); kimage->arch.kern_reloc = __pa(reloc_code) + func_offset; kimage->arch.kern_reloc_arg = __pa(kern_reloc_arg); kern_reloc_arg->head = kimage->head; kern_reloc_arg->entry_addr = kimage->start; kern_reloc_arg->kern_arg0 = kimage->arch.dtb_mem; + + /* Setup vector table only when EL2 is available, but no VHE */ + if (is_hyp_mode_available() && !is_kernel_in_hyp_mode()) + kern_reloc_arg->el2_vector = __pa(reloc_code) + vector_offset; + kexec_image_info(kimage); /* Flush the reloc_code in preparation for its execution. */ diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S index d2a4a0b0d76b..c6178b1a4e60 100644 --- a/arch/arm64/kernel/relocate_kernel.S +++ b/arch/arm64/kernel/relocate_kernel.S @@ -14,6 +14,17 @@ #include #include +.macro el1_sync_64 + .align 7 + br x4 /* Jump to new world from el2 */ +.endm + +.macro invalid_vector label +\label: + .align 7 + b \label +.endm + .pushsection ".kexec_relocate.text", "ax" /* * arm64_relocate_new_kernel - Put a 2nd stage image in place and boot it. @@ -76,4 +87,28 @@ SYM_CODE_START(arm64_relocate_new_kernel) ldr x0, [x0, #KEXEC_KRELOC_KERN_ARG0] /* x0 = dtb address */ br x4 SYM_CODE_END(arm64_relocate_new_kernel) + +/* el2 vectors - switch el2 here while we restore the memory image. */ + .align 11 +SYM_CODE_START(arm64_kexec_el2_vectors) + invalid_vector el2_sync_invalid_sp0 /* Synchronous EL2t */ + invalid_vector el2_irq_invalid_sp0 /* IRQ EL2t */ + invalid_vector el2_fiq_invalid_sp0 /* FIQ EL2t */ + invalid_vector el2_error_invalid_sp0 /* Error EL2t */ + + invalid_vector el2_sync_invalid_spx /* Synchronous EL2h */ + invalid_vector el2_irq_invalid_spx /* IRQ EL2h */ + invalid_vector el2_fiq_invalid_spx /* FIQ EL2h */ + invalid_vector el2_error_invalid_spx /* Error EL2h */ + + el1_sync_64 /* Synchronous 64-bit EL1 */ + invalid_vector el1_irq_invalid_64 /* IRQ 64-bit EL1 */ + invalid_vector el1_fiq_invalid_64 /* FIQ 64-bit EL1 */ + invalid_vector el1_error_invalid_64 /* Error 64-bit EL1 */ + + invalid_vector el1_sync_invalid_32 /* Synchronous 32-bit EL1 */ + invalid_vector el1_irq_invalid_32 /* IRQ 32-bit EL1 */ + invalid_vector el1_fiq_invalid_32 /* FIQ 32-bit EL1 */ + invalid_vector el1_error_invalid_32 /* Error 32-bit EL1 */ +SYM_CODE_END(arm64_kexec_el2_vectors) .popsection From patchwork Wed Jan 27 17:27:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12050699 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 61D7CC433DB for ; Wed, 27 Jan 2021 17:36:59 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E45AA64DA3 for ; Wed, 27 Jan 2021 17:36:58 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E45AA64DA3 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:List-Subscribe:List-Help:List-Post:List-Archive:List-Unsubscribe :List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:To:From: Reply-To:Cc:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=0y47dBQQ7KcFDxJLqckX4ECHWCoaV0Hs17DCxXHTsE0=; b=Wpv5dL1HFs04J1wXoy4Logxf1g xJZzhbSa49QWn43QGlTD3h/hbXcFiDocwOubLYIP1ncwOE1b4oKJ1njWdgOTxl8gDXpUTqvvqJLbG Z7rkqO6Fp3sCMWAlByMSNgMusQSxaf5GBAab5XNcw3b0Qk+YcZDTok46go96nVzAnza0lGyXfDktP tjmcW9ia5EDUK7TVerheuD+OtFs74t/DnBW0GjyLmvd00tuIcmRax1r4ZpttXZLkSMbZ7svLY0A7C bSlqrzTg9JabAk0XevxZByQaQzeJvGM10N5TG+C24FdoT3WGvlb0CQsBs7iXW7s3Z4MJImKSLAZ66 E2RataEQ==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l4oiW-0000zC-8q; Wed, 27 Jan 2021 17:34:56 +0000 Received: from mail-qt1-x830.google.com ([2607:f8b0:4864:20::830]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1l4ob8-0005rW-S4 for linux-arm-kernel@lists.infradead.org; Wed, 27 Jan 2021 17:27:33 +0000 Received: by mail-qt1-x830.google.com with SMTP id z22so1970405qto.7 for ; Wed, 27 Jan 2021 09:27:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=SdcRL3eX4lOwXH8jydL/qdL0IVgZG3cGF34GsbvsmD8=; b=K3B0SdmdyKapkNJZMDzEgXozpmx8/0k4lvny4yseNM4bQmkE44PvsM9aK1V/2oYXRH L12qNPPnei6EWRQu+on8qdESTEknOP+uHmmDxPCWxucBOWENNtaR7pi86cpaTOj/N3DG DVivHIxQDW5asZvrdaCEGm/dr5D4LQBhigXNZcC1CVlGx9C60VVT+R1H1eSTzxA4DFnk SL/Uq8Xmx68QpPaQN+9N79M1ux+9p6ACjmro9sZwM+mEP3Htd8tjDb9N58tdwoTDJA4T hk0L2UYTRayDnWTuYIzdJPHhljf00W6pbGLcQHZ1rlHDfOQ3aJzz7sSpgBJJtfB2uTaj iVew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=SdcRL3eX4lOwXH8jydL/qdL0IVgZG3cGF34GsbvsmD8=; b=P3E5kC9kRzyb+1r9uxvwo2C+h+fo4MST8/Y+sbt3nB5HIBZK/qUZpCglSjK6HNMf3t 5V9J8EfGwrYFhqwQ1bl/zizf5akUNODl6i5JDJxUL0m6dpyKd6AmGoeEdzy2UY8PSyKa Kr1IZUMG0BHD9eUAAXlmkuJqffp9MxsjfVbLNdLnVzi0pXB4k1nz2jZxos7ATajzjtP6 OsQ1taQ8LXmPCrREn/m/k0lJOXtgioCn8Idet4d5vW7LP7aHv00NJTEKmLHf30q+99M5 KPbd50/GIDKL42uyYLBNI25MpSYhuT8NXY2Ta+zBcWN//ekiJkDRmy6hWkf/R9ZvmF9B w5Qw== X-Gm-Message-State: AOAM530j0mxijMeqOzSVI1g1MDMUFE9Y215Z8ks+k+f5XhqEcT8H+vWN +gD0xUX8VUAvCivQdPktq59uMw== X-Google-Smtp-Source: ABdhPJxlFV1txevvNgU5xnq79+y1SsCRubMlvuyhI8QVtE3ik/wKyoAO5LODelM9gvODmj3UENO8cQ== X-Received: by 2002:aed:2123:: with SMTP id 32mr10909736qtc.325.1611768434815; Wed, 27 Jan 2021 09:27:14 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id z2sm1728019qtz.71.2021.01.27.09.27.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Jan 2021 09:27:14 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, jmorris@namei.org, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, maz@kernel.org, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, linux-mm@kvack.org, mark.rutland@arm.com, steve.capper@arm.com, rfontana@redhat.com, tglx@linutronix.de, selindag@gmail.com, tyhicks@linux.microsoft.com Subject: [PATCH v11 4/6] arm64: kexec: configure trans_pgd page table for kexec Date: Wed, 27 Jan 2021 12:27:04 -0500 Message-Id: <20210127172706.617195-5-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210127172706.617195-1-pasha.tatashin@soleen.com> References: <20210127172706.617195-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210127_122719_255356_0408D1E2 X-CRM114-Status: GOOD ( 22.91 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Configure a page table located in kexec-safe memory that has the following mappings: 1. identity mapping for text of relocation function with executable permission. 2. va mappings for all source ranges 3. va mappings for all destination ranges. Signed-off-by: Pavel Tatashin --- arch/arm64/Kconfig | 2 +- arch/arm64/include/asm/kexec.h | 12 +++++ arch/arm64/kernel/asm-offsets.c | 6 +++ arch/arm64/kernel/machine_kexec.c | 89 ++++++++++++++++++++++++++++++- 4 files changed, 107 insertions(+), 2 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index fc0ed9d6e011..440abd0c0ee1 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -1134,7 +1134,7 @@ config CRASH_DUMP config TRANS_TABLE def_bool y - depends on HIBERNATION + depends on HIBERNATION || KEXEC_CORE config XEN_DOM0 def_bool y diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h index b96d8a6aac80..049cde429b1b 100644 --- a/arch/arm64/include/asm/kexec.h +++ b/arch/arm64/include/asm/kexec.h @@ -105,6 +105,12 @@ extern const char arm64_kexec_el2_vectors[]; * el2_vector If present means that relocation routine will go to EL1 * from EL2 to do the copy, and then back to EL2 to do the jump * to new world. + * trans_ttbr0 idmap for relocation function and its argument + * trans_ttbr1 map for source/destination addresses. + * trans_t0sz t0sz for idmap page in trans_ttbr0 + * src_addr start address for source pages. + * dst_addr start address for destination pages. + * copy_len Number of bytes that need to be copied */ struct kern_reloc_arg { phys_addr_t head; @@ -114,6 +120,12 @@ struct kern_reloc_arg { phys_addr_t kern_arg2; phys_addr_t kern_arg3; phys_addr_t el2_vector; + phys_addr_t trans_ttbr0; + phys_addr_t trans_ttbr1; + unsigned long trans_t0sz; + unsigned long src_addr; + unsigned long dst_addr; + unsigned long copy_len; }; #define ARCH_HAS_KIMAGE_ARCH diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index 8a9475be1b62..06278611451d 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -160,6 +160,12 @@ int main(void) DEFINE(KEXEC_KRELOC_KERN_ARG2, offsetof(struct kern_reloc_arg, kern_arg2)); DEFINE(KEXEC_KRELOC_KERN_ARG3, offsetof(struct kern_reloc_arg, kern_arg3)); DEFINE(KEXEC_KRELOC_EL2_VECTOR, offsetof(struct kern_reloc_arg, el2_vector)); + DEFINE(KEXEC_KRELOC_TRANS_TTBR0, offsetof(struct kern_reloc_arg, trans_ttbr0)); + DEFINE(KEXEC_KRELOC_TRANS_TTBR1, offsetof(struct kern_reloc_arg, trans_ttbr1)); + DEFINE(KEXEC_KRELOC_TRANS_T0SZ, offsetof(struct kern_reloc_arg, trans_t0sz)); + DEFINE(KEXEC_KRELOC_SRC_ADDR, offsetof(struct kern_reloc_arg, src_addr)); + DEFINE(KEXEC_KRELOC_DST_ADDR, offsetof(struct kern_reloc_arg, dst_addr)); + DEFINE(KEXEC_KRELOC_COPY_LEN, offsetof(struct kern_reloc_arg, copy_len)); #endif return 0; } diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c index 41d1e3ca13f8..9588c91f67c6 100644 --- a/arch/arm64/kernel/machine_kexec.c +++ b/arch/arm64/kernel/machine_kexec.c @@ -21,6 +21,7 @@ #include #include #include +#include #include "cpu-reset.h" @@ -71,11 +72,89 @@ static void *kexec_page_alloc(void *arg) return page_address(page); } +/* + * Map source segments starting from src_va, and map destination + * segments starting from dst_va, and return size of copy in + * *copy_len argument. + * Relocation function essentially needs to do: + * memcpy(dst_va, src_va, copy_len); + */ +static int map_segments(struct kimage *kimage, pgd_t *pgdp, + struct trans_pgd_info *info, + unsigned long src_va, + unsigned long dst_va, + unsigned long *copy_len) +{ + unsigned long *ptr = 0; + unsigned long dest = 0; + unsigned long len = 0; + unsigned long entry, addr; + int rc; + + for (entry = kimage->head; !(entry & IND_DONE); entry = *ptr++) { + addr = entry & PAGE_MASK; + + switch (entry & IND_FLAGS) { + case IND_DESTINATION: + dest = addr; + break; + case IND_INDIRECTION: + ptr = __va(addr); + break; + case IND_SOURCE: + rc = trans_pgd_map_page(info, pgdp, __va(addr), + src_va, PAGE_KERNEL); + if (rc) + return rc; + rc = trans_pgd_map_page(info, pgdp, __va(dest), + dst_va, PAGE_KERNEL); + if (rc) + return rc; + dest += PAGE_SIZE; + src_va += PAGE_SIZE; + dst_va += PAGE_SIZE; + len += PAGE_SIZE; + } + } + *copy_len = len; + + return 0; +} + +static int mmu_relocate_setup(struct kimage *kimage, void *reloc_code, + struct kern_reloc_arg *kern_reloc_arg) +{ + struct trans_pgd_info info = { + .trans_alloc_page = kexec_page_alloc, + .trans_alloc_arg = kimage, + }; + pgd_t *trans_pgd = kexec_page_alloc(kimage); + int rc; + + if (!trans_pgd) + return -ENOMEM; + + /* idmap relocation function */ + rc = trans_pgd_idmap_page(&info, &kern_reloc_arg->trans_ttbr0, + &kern_reloc_arg->trans_t0sz, reloc_code); + if (rc) + return rc; + + kern_reloc_arg->src_addr = _PAGE_OFFSET(VA_BITS_MIN); + kern_reloc_arg->dst_addr = _PAGE_OFFSET(VA_BITS_MIN - 1); + kern_reloc_arg->trans_ttbr1 = phys_to_ttbr(__pa(trans_pgd)); + + rc = map_segments(kimage, trans_pgd, &info, kern_reloc_arg->src_addr, + kern_reloc_arg->dst_addr, &kern_reloc_arg->copy_len); + return rc; +} + int machine_kexec_post_load(struct kimage *kimage) { void *reloc_code = page_to_virt(kimage->control_code_page); struct kern_reloc_arg *kern_reloc_arg = kexec_page_alloc(kimage); long func_offset, vector_offset, reloc_size; + int rc = 0; if (!kern_reloc_arg) return -ENOMEM; @@ -95,6 +174,14 @@ int machine_kexec_post_load(struct kimage *kimage) if (is_hyp_mode_available() && !is_kernel_in_hyp_mode()) kern_reloc_arg->el2_vector = __pa(reloc_code) + vector_offset; + /* + * If relocation is not needed, we do not need to enable MMU in + * relocation routine, therefore do not create page tables for + * scenarios such as crash kernel + */ + if (!(kimage->head & IND_DONE)) + rc = mmu_relocate_setup(kimage, reloc_code, kern_reloc_arg); + kexec_image_info(kimage); /* Flush the reloc_code in preparation for its execution. */ @@ -103,7 +190,7 @@ int machine_kexec_post_load(struct kimage *kimage) reloc_size); __flush_dcache_area(kern_reloc_arg, sizeof(struct kern_reloc_arg)); - return 0; + return rc; } /** From patchwork Wed Jan 27 17:27:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12050991 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6BC18C433E0 for ; Wed, 27 Jan 2021 20:25:08 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id EBA3561488 for ; Wed, 27 Jan 2021 20:25:07 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EBA3561488 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:List-Subscribe:List-Help:List-Post:List-Archive:List-Unsubscribe :List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:To:From: Reply-To:Cc:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=rw5feXPvBaXz1B0APzX6mO65AaHsZT3yWiRJZ/a8HSE=; b=Xod4ehZZc0J5FalgJeUzJNlOB0 pd2cHrsvrlxt3rpnPvpRc+jKnJXPP4vao1s4LbnUvL7FI6Evgn5PoKp8BozKcUL3QjWxTPBu4pAnm ZjGK/bO8u7rly7vCTZUktecYcaM4jY0sWQ7qdLzrT58Ebr17qX192peLJB3URzhh0stDj2JHIRvix 55tfIQOWO030OUJlnV6pYAl9pG8Zrq0VPyZ3Lba84iwLtrE0PNsN0IdyFewAlKx1OYMNc8Vxc4AFt bt05/vGvKDj09p16FPIkxiw5y9XtaebAWV/tPGL2wewSdnBz8t9HiC55FvtZrskr2DLb+pKn4Be9l SMcIofxg==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l4ojb-0001Uh-78; Wed, 27 Jan 2021 17:36:03 +0000 Received: from mail-qk1-x72f.google.com ([2607:f8b0:4864:20::72f]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1l4obA-0005sU-Bt for linux-arm-kernel@lists.infradead.org; Wed, 27 Jan 2021 17:27:34 +0000 Received: by mail-qk1-x72f.google.com with SMTP id k193so2475319qke.6 for ; Wed, 27 Jan 2021 09:27:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=RDSr+b90kmy9FHMSi/QfI0SbQbMsMCPJRwK16BQ54DQ=; b=D0vmR3k2J3V59Tf9s9cd/7e81xmYwG30fQDjq0TRTMFYsuLttnJjFOhyv3py9kd7e8 tR5GuoAbb317sw2avBKR0hwLqYH9ztq8Nfx4j3Puh61MMGT3Mk5McC8tw6KOcxr0zlRw mV/fer9zN7LgR0lkpI88m/eb4RjED+TibTTVUsaJyqa2x2EyVDjCGgnfgdho0CaA7dVk l/m3T/XDxDWPoe5jkoOsIpUFsHV4f98IJvnq3Na4WzdaI4x424oe0lbOkQ/mLZyhS6Tp fMkcLsHgwzYABYXvHW/EoysJnRU6e+lZCq0osuDKHNa5I53k0XRd8dxF/APoFx4Jbv0U wDAg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=RDSr+b90kmy9FHMSi/QfI0SbQbMsMCPJRwK16BQ54DQ=; b=DG6Vh611vRUDSC5EfDyKp3sehjnDVNIxAgmRaXmSU7F3Uy3lPNbeZ9IrFJG05R+trR DhK07MhPcV3DK9jGQFN6lub7WCyKlSYHcuTbYyNTE8M1Gy6EBoiNAqyEGDyYJJ2VOiWT RxHb+v3DUhjdBX+iWVwsehh5VBKXutfLYq2GIn1BbxtJdwQnb11BmY0XyIxg0aycUOjg SkD5AYu5yu6fwrzjzP1gjRJv9NTrHYE3BRE9DOvmFVOpo9x1EwlNQLxIrDFgZgHV7YLM OwZJEa4BG7scZweVBrRZ/KkR06opmC2wAgaVsaOW2RepCvlQp7pKem0sKJL0fkl9UEkR KS2g== X-Gm-Message-State: AOAM5326EgsLnzG0Dh+PScU5t8/gT4oxMZ8DI2GcAdWJvY3oVpmUVHI+ XZITBhKAWwsJGncIgpiXilOgsw== X-Google-Smtp-Source: ABdhPJy1F3OJMWVpbhDsfEHVOUOuSIdueuuHWt8/BvaPPfMVcsC5saRFBDHsiaILyCDjZvE/5KZc7w== X-Received: by 2002:a37:2741:: with SMTP id n62mr11561676qkn.43.1611768436393; Wed, 27 Jan 2021 09:27:16 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id z2sm1728019qtz.71.2021.01.27.09.27.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Jan 2021 09:27:15 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, jmorris@namei.org, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, maz@kernel.org, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, linux-mm@kvack.org, mark.rutland@arm.com, steve.capper@arm.com, rfontana@redhat.com, tglx@linutronix.de, selindag@gmail.com, tyhicks@linux.microsoft.com Subject: [PATCH v11 5/6] arm64: kexec: enable MMU during kexec relocation Date: Wed, 27 Jan 2021 12:27:05 -0500 Message-Id: <20210127172706.617195-6-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210127172706.617195-1-pasha.tatashin@soleen.com> References: <20210127172706.617195-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210127_122720_661383_6E820700 X-CRM114-Status: GOOD ( 18.08 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Now, that we have transitional page tables configured, temporarily enable MMU to allow faster relocation of segments to final destination. The performance data: for a moderate size kernel + initramfs: 25M the relocation was taking 0.382s, with enabled MMU it now takes 0.019s only or x20 improvement. The time is proportional to the size of relocation, therefore if initramfs is larger, 100M it could take over a second. Signed-off-by: Pavel Tatashin --- arch/arm64/kernel/relocate_kernel.S | 131 ++++++++++++++++++---------- 1 file changed, 87 insertions(+), 44 deletions(-) diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S index c6178b1a4e60..9c60981a6911 100644 --- a/arch/arm64/kernel/relocate_kernel.S +++ b/arch/arm64/kernel/relocate_kernel.S @@ -4,6 +4,8 @@ * * Copyright (C) Linaro. * Copyright (C) Huawei Futurewei Technologies. + * Copyright (C) 2020, Microsoft Corporation. + * Pavel Tatashin */ #include @@ -14,6 +16,54 @@ #include #include +.macro tlb_invalidate + dsb sy + dsb ish + tlbi vmalle1 + dsb ish + isb +.endm + +.macro turn_off_mmu tmp1, tmp2 + mrs \tmp1, sctlr_el1 + mov_q \tmp2, SCTLR_ELx_FLAGS + bic \tmp1, \tmp1, \tmp2 + pre_disable_mmu_workaround + msr sctlr_el1, \tmp1 + isb +.endm + +.macro turn_on_mmu tmp1, tmp2 + mrs \tmp1, sctlr_el1 + mov_q \tmp2, SCTLR_ELx_FLAGS + orr \tmp1, \tmp1, \tmp2 + msr sctlr_el1, \tmp1 + ic iallu + dsb nsh + isb +.endm + +/* + * Set ttbr0 and ttbr1, called while MMU is disabled, so no need to temporarily + * set zero_page table. Invalidate TLB after new tables are set. + */ +.macro set_ttbr arg, tmp1, tmp2 + ldr \tmp1, [\arg, #KEXEC_KRELOC_TRANS_TTBR0] + msr ttbr0_el1, \tmp1 + ldr \tmp1, [\arg, #KEXEC_KRELOC_TRANS_TTBR1] + offset_ttbr1 \tmp1, \tmp2 + msr ttbr1_el1, \tmp1 + isb +.endm + +/* Set T0SZ to match the requirements of idmap page */ +.macro set_tcr_t0sz arg, tmp1, tmp2 + ldr \tmp2, [\arg, #KEXEC_KRELOC_TRANS_T0SZ] + mrs \tmp1, tcr_el1 + bfi \tmp1, \tmp2, TCR_T0SZ_OFFSET, TCR_TxSZ_WIDTH + msr tcr_el1, \tmp1 +.endm + .macro el1_sync_64 .align 7 br x4 /* Jump to new world from el2 */ @@ -36,56 +86,49 @@ * symbols arm64_relocate_new_kernel and arm64_relocate_new_kernel_end. The * machine_kexec() routine will copy arm64_relocate_new_kernel to the kexec * safe memory that has been set up to be preserved during the copy operation. + * + * This function temporarily enables MMU if kernel relocation is needed. + * Also, if we enter this function at EL2 on non-VHE kernel, we temporarily go + * to EL1 to enable MMU, and escalate back to EL2 at the end to do the jump to + * the new kernel. This is determined by presence of el2_vector. */ SYM_CODE_START(arm64_relocate_new_kernel) - /* Check if the new image needs relocation. */ - ldr x16, [x0, #KEXEC_KRELOC_HEAD] /* x16 = kimage_head */ - tbnz x16, IND_DONE_BIT, .Ldone - raw_dcache_line_size x15, x1 /* x15 = dcache line size */ -.Lloop: - and x12, x16, PAGE_MASK /* x12 = addr */ - - /* Test the entry flags. */ -.Ltest_source: - tbz x16, IND_SOURCE_BIT, .Ltest_indirection - - /* Invalidate dest page to PoC. */ - mov x2, x13 - add x20, x2, #PAGE_SIZE - sub x1, x15, #1 - bic x2, x2, x1 -2: dc ivac, x2 - add x2, x2, x15 - cmp x2, x20 - b.lo 2b - dsb sy - - copy_page x13, x12, x1, x2, x3, x4, x5, x6, x7, x8 - b .Lnext -.Ltest_indirection: - tbz x16, IND_INDIRECTION_BIT, .Ltest_destination - mov x14, x12 /* ptr = addr */ - b .Lnext -.Ltest_destination: - tbz x16, IND_DESTINATION_BIT, .Lnext - mov x13, x12 /* dest = addr */ -.Lnext: - ldr x16, [x14], #8 /* entry = *ptr++ */ - tbz x16, IND_DONE_BIT, .Lloop /* while (!(entry & DONE)) */ -.Ldone: - /* wait for writes from copy_page to finish */ - dsb nsh - ic iallu - dsb nsh - isb - - /* Start new image. */ - ldr x4, [x0, #KEXEC_KRELOC_ENTRY_ADDR] /* x4 = kimage_start */ + mov x20, xzr /* x20 will hold vector value */ + ldr x11, [x0, #KEXEC_KRELOC_COPY_LEN] + cbz x11, 5f /* Check if need to relocate */ + ldr x20, [x0, #KEXEC_KRELOC_EL2_VECTOR] + cbz x20, 2f /* need to reduce to EL1? */ + msr vbar_el2, x20 /* el2_vector present, means */ + adr x1, 2f /* we will do copy in el1 but */ + msr elr_el2, x1 /* do final jump from el2 */ + eret /* Reduce to EL1 */ +2: set_tcr_t0sz x0, x1, x2 /* Set t0sz for idmaped page */ + set_ttbr x0, x1, x2 /* Set our page tables */ + tlb_invalidate + ldr x1, [x0, #KEXEC_KRELOC_DST_ADDR]; /* arg is not idmapped so */ + ldr x2, [x0, #KEXEC_KRELOC_SRC_ADDR]; /* read before MMU is on */ + turn_on_mmu x3, x4 /* Turn MMU back on */ + mov x12, x1 /* x12 dst backup */ +3: copy_page x1, x2, x3, x4, x5, x6, x7, x8, x9, x10 + sub x11, x11, #PAGE_SIZE + cbnz x11, 3b /* page copy loop */ + raw_dcache_line_size x2, x3 /* x2 = dcache line size */ + sub x3, x2, #1 /* x3 = dcache_size - 1 */ + bic x12, x12, x3 +4: dc cvau, x12 /* Flush D-cache */ + add x12, x12, x2 + cmp x12, x1 /* Compare to dst + len */ + b.ne 4b /* D-cache flush loop */ + turn_off_mmu x1, x2 /* Turn off MMU */ + tlb_invalidate /* Invalidate TLB */ +5: ldr x4, [x0, #KEXEC_KRELOC_ENTRY_ADDR] /* x4 = kimage_start */ ldr x3, [x0, #KEXEC_KRELOC_KERN_ARG3] ldr x2, [x0, #KEXEC_KRELOC_KERN_ARG2] ldr x1, [x0, #KEXEC_KRELOC_KERN_ARG1] ldr x0, [x0, #KEXEC_KRELOC_KERN_ARG0] /* x0 = dtb address */ - br x4 + cbnz x20, 6f /* need to escalate to el2? */ + br x4 /* Jump to new world */ +6: hvc #0 /* enters kexec_el1_sync */ SYM_CODE_END(arm64_relocate_new_kernel) /* el2 vectors - switch el2 here while we restore the memory image. */ From patchwork Wed Jan 27 17:27:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 12050787 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 38E7FC433DB for ; Wed, 27 Jan 2021 18:39:58 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D5133601FB for ; Wed, 27 Jan 2021 18:39:57 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D5133601FB Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=soleen.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:List-Subscribe:List-Help:List-Post:List-Archive:List-Unsubscribe :List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:To:From: Reply-To:Cc:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=cLrrceUuNqtfVoGcFpe62vYyPMICXScLVPVm0AzDPfE=; b=Htb+L9mZ3N/WfgO+u7cYkuudM2 6ZU7RdHM1gxO+8g1U8oGpyjFR243fSrF+mqOhZPFJlgTiDNPQDTqn28WGKhjkBsYbEPx5AYOdSdaq NmzRPM8wuPTXtBfCHudsXxgFoN7f+B/5wt71fUglASOUE1jzGWaE6FZTQX8WZi51Z1eRAJAhEHYET N23Y64JHHcirKNcX19KbcjJpIXgGl6kA6Q5ojvkt0Hq4DasCS+roDfTJywTZuRjI5wHKgc93IutbB gzdSvfHHq5aORV7zMSznaPl2HQonOtoJBrzU/AE2r7I/YSo4SIUxr4ns3R1e7is1I5Bj6DqnIOIL2 9uP/ywmw==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1l4pig-0003ES-Vp; Wed, 27 Jan 2021 18:39:11 +0000 Received: from mail-qv1-xf2a.google.com ([2607:f8b0:4864:20::f2a]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1l4obB-0005tV-MY for linux-arm-kernel@lists.infradead.org; Wed, 27 Jan 2021 17:27:36 +0000 Received: by mail-qv1-xf2a.google.com with SMTP id j13so1413569qvu.10 for ; Wed, 27 Jan 2021 09:27:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=cfSnNSnfVduFxCPOvWfs9p+QSvXlJihsz/H5U3Y4H1U=; b=JSF/HlHCJzShZn7ALunS0BkmdIvbsb31kLfU+RnIPOJjHHHbnPC+hKp5GzFf5aNfG0 PCnd0hUx8KnQTQJAs7hc3j8UjBo0nhsavr2YgBKj48WT2dlprkP86SX5YogstiQFgEW1 fQXmcUFNK/whrTvXsAd2ttbgakW8eugEXjKNmuKRJBUbncvkUaYlDCqNXZhKNp4QGgZi s9l3aBEqnDiNAea7t1MlSQvIt3uOdGGZZwDbfriUGm56jfCILeRPYfkcC4aBIVfBChfU myBdivbvXZCSHraxlvN/kcYnz6cOQzbdEGPqEKU9R4Tk+vISJcmeJUo2Io2EH78AfPDL BLQA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=cfSnNSnfVduFxCPOvWfs9p+QSvXlJihsz/H5U3Y4H1U=; b=PdIW7Op9uJzaamIV+qc3aHnduAk7kviTHaFOfzohhlrz3qpTUAhDDf5hJMtWselCTd ayDzWTasrDGRFRB2cAjB6MbP0E8Tp+oHEHB80Sp4roWEmKaJC75vEPaC7C1hHs3g8sey aV/JS7mBVuKAd6QythGP4LZwmUP2Pxn+tZ3XutKBzoyJ6tAnrnl/PKX8PwXRRbtHRJ+/ f4B9BeNqBlD0yy198B6/dW0Jr0XF/9/mET0RtCSd0/YXwSi9r2PFWqGFn6qKfAjDZrZk HDwTX7pOwI9oeGMYvTXbhtdqn0O0mCrFuyCUJ/m99VDGPROnSl11LO3bWib39G/SYbFr SXsA== X-Gm-Message-State: AOAM532Kv06quz80O3uhwbRSTNWuepRtDiPWdRp785QKnFR1YsVE/edX miF7Q3lYrHJvM1pJLd+xPL2hOA== X-Google-Smtp-Source: ABdhPJwHsa1PlzOcEErpv+fEXO2+H7UWl48S+Yfvc0TUYpC69d2qgqgALhqP+OIOSgJyhuzsv33wBA== X-Received: by 2002:a0c:99c4:: with SMTP id y4mr11377443qve.9.1611768437920; Wed, 27 Jan 2021 09:27:17 -0800 (PST) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id z2sm1728019qtz.71.2021.01.27.09.27.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 27 Jan 2021 09:27:17 -0800 (PST) From: Pavel Tatashin To: pasha.tatashin@soleen.com, jmorris@namei.org, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-arm-kernel@lists.infradead.org, maz@kernel.org, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, linux-mm@kvack.org, mark.rutland@arm.com, steve.capper@arm.com, rfontana@redhat.com, tglx@linutronix.de, selindag@gmail.com, tyhicks@linux.microsoft.com Subject: [PATCH v11 6/6] arm64: kexec: remove head from relocation argument Date: Wed, 27 Jan 2021 12:27:06 -0500 Message-Id: <20210127172706.617195-7-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210127172706.617195-1-pasha.tatashin@soleen.com> References: <20210127172706.617195-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210127_122723_726780_8B7B4D9A X-CRM114-Status: GOOD ( 11.91 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Now, that relocation is done using virtual addresses, reloc_arg->head is not needed anymore. Signed-off-by: Pavel Tatashin --- arch/arm64/include/asm/kexec.h | 2 -- arch/arm64/kernel/asm-offsets.c | 1 - arch/arm64/kernel/machine_kexec.c | 1 - 3 files changed, 4 deletions(-) diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h index 049cde429b1b..2fa4109bd582 100644 --- a/arch/arm64/include/asm/kexec.h +++ b/arch/arm64/include/asm/kexec.h @@ -97,7 +97,6 @@ extern const char arm64_kexec_el2_vectors[]; /* * kern_reloc_arg is passed to kernel relocation function as an argument. - * head kimage->head, allows to traverse through relocation segments. * entry_addr kimage->start, where to jump from relocation function (new * kernel, or purgatory entry address). * kern_arg0 first argument to kernel is its dtb address. The other @@ -113,7 +112,6 @@ extern const char arm64_kexec_el2_vectors[]; * copy_len Number of bytes that need to be copied */ struct kern_reloc_arg { - phys_addr_t head; phys_addr_t entry_addr; phys_addr_t kern_arg0; phys_addr_t kern_arg1; diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index 06278611451d..94f050ad6471 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -153,7 +153,6 @@ int main(void) BLANK(); #endif #ifdef CONFIG_KEXEC_CORE - DEFINE(KEXEC_KRELOC_HEAD, offsetof(struct kern_reloc_arg, head)); DEFINE(KEXEC_KRELOC_ENTRY_ADDR, offsetof(struct kern_reloc_arg, entry_addr)); DEFINE(KEXEC_KRELOC_KERN_ARG0, offsetof(struct kern_reloc_arg, kern_arg0)); DEFINE(KEXEC_KRELOC_KERN_ARG1, offsetof(struct kern_reloc_arg, kern_arg1)); diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c index 9588c91f67c6..07da8d623d8e 100644 --- a/arch/arm64/kernel/machine_kexec.c +++ b/arch/arm64/kernel/machine_kexec.c @@ -166,7 +166,6 @@ int machine_kexec_post_load(struct kimage *kimage) memcpy(reloc_code, __relocate_new_kernel_start, reloc_size); kimage->arch.kern_reloc = __pa(reloc_code) + func_offset; kimage->arch.kern_reloc_arg = __pa(kern_reloc_arg); - kern_reloc_arg->head = kimage->head; kern_reloc_arg->entry_addr = kimage->start; kern_reloc_arg->kern_arg0 = kimage->arch.dtb_mem;