From patchwork Wed Jul 31 15:38:50 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 11068409 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 01009112C for ; Wed, 31 Jul 2019 15:39:22 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DF3101FF28 for ; Wed, 31 Jul 2019 15:39:21 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D1E95204FB; Wed, 31 Jul 2019 15:39:21 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 6972F1FF28 for ; Wed, 31 Jul 2019 15:39:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Cc:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=f8LxXPMQ1wSJznE/4k58emuiSdHaruNV/0M1hKxpD1M=; b=U0KX4V4mr6cjWU gsX9JasBoFiPGNxBvMXDYkZ4AiziP08e7GF4ezNfhtGt+cjELuRkB0S3v+CgEusBOaH8bNVHUmFOy A/9AvSQ4yIj/bxkq+PwIPwyWzuae5p6RK24rZ69EqSRZZdBiraYqVbT8qpydePfHBX0TSHXKrLwWz r740kude47NlGfX3CvFXsXjJGH3rr3sxIX6Hc71sob5K2SrpwICgQh70eOVZeJNEYdthtZrE+rZdh fTjQ6sjMQlN8mJUgWRoConMIz2y9EFv5huOx/GUoz18LGshzhHklif3TOPQDSoOoSh8MaIroJ0KgP hbNHf0SK6Vxi9e5wBkUg==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92 #3 (Red Hat Linux)) id 1hsqhE-00082l-Le; Wed, 31 Jul 2019 15:39:20 +0000 Received: from mail-qt1-x844.google.com ([2607:f8b0:4864:20::844]) by bombadil.infradead.org with esmtps (Exim 4.92 #3 (Red Hat Linux)) id 1hsqgw-0007iL-NM for linux-arm-kernel@lists.infradead.org; Wed, 31 Jul 2019 15:39:04 +0000 Received: by mail-qt1-x844.google.com with SMTP id n11so67001780qtl.5 for ; Wed, 31 Jul 2019 08:39:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=O8sp7Ivz1alrz3xXygPalicyjdu3OPu+GkykS7h2DZg=; b=f/mj0qEdjPvowxzxCkz1WBHsoz6X08MjhaCvOP9JKChUzdvXGCDjmIksTr75NmrETe GR0NJA4z1vIEH0dzoq8OvrHbM6EdjK+AohOXseaucGCyJn1Xewil/BIr2BORcK8Y8r+W B1JzbU1hdtMsW+Qq72doDnCa9Cyf/4J1EsfdmNeeVCidWQbLHt+DkzFZUp2EXOkbnkdG EvW/gLPfttAw50rqtj3FU3JdgGfBzBnstUC/k66NtOwuC0Nqlg1fr2V9VIaZji8gnYBA Oa1jc7STRN3fH47wz3Yk1rreSvslGvBiO2R7vofvw5mmc48VryeJE4V7i1rRlbMbWKxH l5nQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=O8sp7Ivz1alrz3xXygPalicyjdu3OPu+GkykS7h2DZg=; b=n48Tf65t6t8qgIL1AnlQnAJ7FYGz631po3OmBNzFsbuIdItSh+98imb7X+ysexCUwv +PNTnm2LCrY3pdiSR0X1338BXeZ+4Uc6d/NN2HDnVSJcbpwyIw43EyQNHC+8gqCfIpCM qPn8Kgx6kvfTCE4nh55oh0tuC9PRmlGLSs2nTM9jjVAaRpy2izlKO9QNryeRNL+dA68u Wj9f7VXowFGjETN8ybdPhZJRosRHTccrfftRXLuRJTgQS9RJWfnM6fVaRYBbi4EakMdt HM+d72EOUrq8S8Dc/4XrlfiUr7U475xPdR/0/VXx1NgjAcFHzsZzvO3Svenkf1iRYgwt phWg== X-Gm-Message-State: APjAAAVOaC2h1+0ist3z1FSeyBFz50ivKpNiLnemjAGBNkG0kY4o7cLm q57LuIJsKNpgDuzMtUUChbE= X-Google-Smtp-Source: APXvYqxvBo52tjJ6/0c0xwHoZtK9MewLNPfKHRcGHVWsI64uRmnRR/4lWkfMBAzaWutbGo9baKuA9g== X-Received: by 2002:a0c:afd5:: with SMTP id t21mr88223958qvc.105.1564587540796; Wed, 31 Jul 2019 08:39:00 -0700 (PDT) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id f25sm35116803qta.81.2019.07.31.08.38.59 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 31 Jul 2019 08:39:00 -0700 (PDT) From: Pavel Tatashin To: pasha.tatashin@soleen.com, jmorris@namei.org, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-doc@vger.kernel.org, linux-arm-kernel@lists.infradead.org, marc.zyngier@arm.com, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, bhsharma@redhat.com Subject: [RFC v2 1/8] kexec: quiet down kexec reboot Date: Wed, 31 Jul 2019 11:38:50 -0400 Message-Id: <20190731153857.4045-2-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190731153857.4045-1-pasha.tatashin@soleen.com> References: <20190731153857.4045-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190731_083902_754778_D2253887 X-CRM114-Status: GOOD ( 10.72 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Here is a regular kexec command sequence and output: ===== $ kexec --reuse-cmdline -i --load Image $ kexec -e [ 161.342002] kexec_core: Starting new kernel Welcome to Buildroot buildroot login: ===== Even when "quiet" kernel parameter is specified, "kexec_core: Starting new kernel" is printed. This message has KERN_EMERG level, but there is no emergency, it is a normal kexec operation, so quiet it down to appropriate KERN_NOTICE. Machines that have slow console baud rate benefit from less output. Signed-off-by: Pavel Tatashin Reviewed-by: Simon Horman --- kernel/kexec_core.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/kernel/kexec_core.c b/kernel/kexec_core.c index d5870723b8ad..2c5b72863b7b 100644 --- a/kernel/kexec_core.c +++ b/kernel/kexec_core.c @@ -1169,7 +1169,7 @@ int kernel_kexec(void) * CPU hotplug again; so re-enable it here. */ cpu_hotplug_enable(); - pr_emerg("Starting new kernel\n"); + pr_notice("Starting new kernel\n"); machine_shutdown(); } From patchwork Wed Jul 31 15:38:51 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 11068411 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 48F0B1399 for ; Wed, 31 Jul 2019 15:39:39 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2D71D1FF28 for ; Wed, 31 Jul 2019 15:39:39 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1F59E204FB; Wed, 31 Jul 2019 15:39:39 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 4736F1FF28 for ; Wed, 31 Jul 2019 15:39:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Cc:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=SDuJN93KZIQN+QRly6lT60GibSD1X5Z4zfXXaEuvEsY=; b=gnGOjRylx0WycL ompGY1QNWQ1Sl93x0tR5IuW5uH7alNmmJMN7rNdWFFqCKiHZaWIFzgHKUW3n3dzil/wC8QDpeYxda 6G0uUsP2HqIScrX0EZuz7ti3qqafUuxKcce4quR7ppNsBiR9UwttN1qdGr147SL92N4+HTpmitNz+ t/PnuEfCoSPllaAAUogQP7h7KSQ3MU6eeqpVsYbZTgGAN88zg8rRwvLC75ossbHBD6VSuZ4lgprE8 7b2lWxMf6c+ErtsKHyyS+zcNjnmLVJcdWCsDLh2lQzK7Ifd/rDObHPkKxMxe5azcgcRTps3SSvGZY HhSsRfyZgccjmfRwhmzQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92 #3 (Red Hat Linux)) id 1hsqhV-0008Mt-Jc; Wed, 31 Jul 2019 15:39:37 +0000 Received: from mail-qt1-x842.google.com ([2607:f8b0:4864:20::842]) by bombadil.infradead.org with esmtps (Exim 4.92 #3 (Red Hat Linux)) id 1hsqgx-0007iN-6L for linux-arm-kernel@lists.infradead.org; Wed, 31 Jul 2019 15:39:05 +0000 Received: by mail-qt1-x842.google.com with SMTP id h21so66968235qtn.13 for ; Wed, 31 Jul 2019 08:39:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=mSPaXkOF4y4c7qEpVZR0NQfcUuKxNUUmh4BAqy8Ck5c=; b=D2Tu5yPz6ThRiLKACYJE0UxapK38kpyJlO332z7cxCc1YZn2NoRKoz21V31F6uDgZ/ SWZxsCEX0j5++knXwsUdKPVXCG5Ne5ONR+GSZxIWsOV2jUozTSWOAwyvMRlunWgbv3fR +P4okaxNEsy+aIW4IGGne9xBVBFiP1Th/SylTO2SLnKASLc7Er+n5/g1hEw0mZaGTQtH HgsCCXq2vgm4k2bgNxO76FnHs3GFnR7uQLbQXylkg9YM4rlIR50pJf12hVDIjkW/aXGj ru9eSaEGXoqKnW9Xcn3D1hx2TggdwnP124NT8IKtKyzC1qTJx9qFkpCzZP1TNmVwLUec baHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=mSPaXkOF4y4c7qEpVZR0NQfcUuKxNUUmh4BAqy8Ck5c=; b=lKswkGcaSIg8/6HNlnSn3+zBhavxGJ+vwAtavRfBYkNfXd3mvhKGIWxKtKStEBxu6f 7pYx34/Qndar5w3eg6rWrZlkgqrvaRA9AhpyX8K9TQBHtpPFmBZ8T+J+pLlzzkfkBdo5 0DoIsV+ZsISWhI3YjpmwLnelMo9fM4CUQvFkpAX8nBUVnr6lmYyF6tUl2kLrv61prnq1 74lpep16qt8nK88aoGVHl/k/s7ZstWH7v3UrZ6HTeexYoAV8KhOU2WxrrwqrOuJeFmkw kugWtJ1D9TjTQGFfxG89V6cC9QFL+uPX23VyKJHi3SI6NvzqnizK9tkOcAL3hqB1WWTs bT4Q== X-Gm-Message-State: APjAAAU7pb2IUFHx/ajZRM1tpkO/+vXKzeGsDVo3cdZgkzKbyNQ3wlkV lLzdydlz/01yXA3sBigW1f0= X-Google-Smtp-Source: APXvYqwnV5hzOxvHii3LH6ElNCKpudgjmJbmwaAb14VIS52ac266RM0J/Dc7QhnTq28gMec0v5rn6w== X-Received: by 2002:a0c:d7cc:: with SMTP id g12mr54325194qvj.220.1564587542099; Wed, 31 Jul 2019 08:39:02 -0700 (PDT) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id f25sm35116803qta.81.2019.07.31.08.39.00 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 31 Jul 2019 08:39:01 -0700 (PDT) From: Pavel Tatashin To: pasha.tatashin@soleen.com, jmorris@namei.org, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-doc@vger.kernel.org, linux-arm-kernel@lists.infradead.org, marc.zyngier@arm.com, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, bhsharma@redhat.com Subject: [RFC v2 2/8] arm64, mm: transitional tables Date: Wed, 31 Jul 2019 11:38:51 -0400 Message-Id: <20190731153857.4045-3-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190731153857.4045-1-pasha.tatashin@soleen.com> References: <20190731153857.4045-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190731_083903_258165_6F9B1173 X-CRM114-Status: GOOD ( 15.48 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP There are cases where normal kernel pages tables, i.e. idmap_pg_dir and swapper_pg_dir are not sufficient because they may be overwritten. This happens when we transition from one world to another: for example during kexec kernel relocation transition, and also during hibernate kernel restore transition. In these cases, if MMU is needed, the page table memory must be allocated from a safe place. Transitional tables is intended to allow just that. Signed-off-by: Pavel Tatashin --- arch/arm64/Kconfig | 4 + arch/arm64/include/asm/pgtable-hwdef.h | 1 + arch/arm64/include/asm/trans_table.h | 66 ++++++ arch/arm64/mm/Makefile | 1 + arch/arm64/mm/trans_table.c | 272 +++++++++++++++++++++++++ 5 files changed, 344 insertions(+) create mode 100644 arch/arm64/include/asm/trans_table.h create mode 100644 arch/arm64/mm/trans_table.c diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 3adcec05b1f6..91a7416ffe4e 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -999,6 +999,10 @@ config CRASH_DUMP For more details see Documentation/admin-guide/kdump/kdump.rst +config TRANS_TABLE + def_bool y + depends on HIBERNATION || KEXEC_CORE + config XEN_DOM0 def_bool y depends on XEN diff --git a/arch/arm64/include/asm/pgtable-hwdef.h b/arch/arm64/include/asm/pgtable-hwdef.h index db92950bb1a0..dcb4f13c7888 100644 --- a/arch/arm64/include/asm/pgtable-hwdef.h +++ b/arch/arm64/include/asm/pgtable-hwdef.h @@ -110,6 +110,7 @@ #define PUD_TABLE_BIT (_AT(pudval_t, 1) << 1) #define PUD_TYPE_MASK (_AT(pudval_t, 3) << 0) #define PUD_TYPE_SECT (_AT(pudval_t, 1) << 0) +#define PUD_SECT_RDONLY (_AT(pudval_t, 1) << 7) /* AP[2] */ /* * Level 2 descriptor (PMD). diff --git a/arch/arm64/include/asm/trans_table.h b/arch/arm64/include/asm/trans_table.h new file mode 100644 index 000000000000..4d7bd0bf36c0 --- /dev/null +++ b/arch/arm64/include/asm/trans_table.h @@ -0,0 +1,66 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (c) 2019, Microsoft Corporation. + * Pavel Tatashin + */ + +#ifndef _ASM_TRANS_TABLE_H +#define _ASM_TRANS_TABLE_H + +#include + +/* + * trans_alloc_page + * - Allocator that should return exactly one uninitilaized page, if this + * allocator fails, trans_table returns -ENOMEM error. + * + * trans_alloc_arg + * - Passed to trans_alloc_page as an argument + * + * trans_flags + * - bitmap with flags that control how page table is filled. + * TRANS_MKWRITE: during page table copy make PTE, PME, and PUD page + * writeable by removing RDONLY flag from PTE. + * TRANS_MKVALID: during page table copy, if PTE present, but not valid, + * make it valid. + * TRANS_CHECKPFN: During page table copy, for every PTE entry check that + * PFN that this PTE points to is valid. Otherwise return + * -ENXIO + * TRANS_FORCEMAP: During page map, if translation exists, force + * overwrite it. Otherwise -ENXIO may be returned by + * trans_table_map_* functions if conflict is detected. + */ + +#define TRANS_MKWRITE (1 << 0) +#define TRANS_MKVALID (1 << 1) +#define TRANS_CHECKPFN (1 << 2) +#define TRANS_FORCEMAP (1 << 3) + +struct trans_table_info { + void * (*trans_alloc_page)(void *); + void *trans_alloc_arg; + unsigned long trans_flags; +}; + +/* Create and empty trans table. */ +int trans_table_create_empty(struct trans_table_info *info, + pgd_t **trans_table); + +/* + * Create trans table and copy entries from from_table to trans_table in range + * [start, end) + */ +int trans_table_create_copy(struct trans_table_info *info, pgd_t **trans_table, + pgd_t *from_table, unsigned long start, + unsigned long end); + +/* + * Add map entry to trans_table for a base-size page at PTE level. + * page: page to be mapped. + * dst_addr: new VA address for the pages + * pgprot: protection for the page. + */ +int trans_table_map_page(struct trans_table_info *info, pgd_t *trans_table, + void *page, unsigned long dst_addr, pgprot_t pgprot); + +#endif /* _ASM_TRANS_TABLE_H */ diff --git a/arch/arm64/mm/Makefile b/arch/arm64/mm/Makefile index 849c1df3d214..3794fff18659 100644 --- a/arch/arm64/mm/Makefile +++ b/arch/arm64/mm/Makefile @@ -6,6 +6,7 @@ obj-y := dma-mapping.o extable.o fault.o init.o \ obj-$(CONFIG_HUGETLB_PAGE) += hugetlbpage.o obj-$(CONFIG_ARM64_PTDUMP_CORE) += dump.o obj-$(CONFIG_ARM64_PTDUMP_DEBUGFS) += ptdump_debugfs.o +obj-$(CONFIG_TRANS_TABLE) += trans_table.o obj-$(CONFIG_NUMA) += numa.o obj-$(CONFIG_DEBUG_VIRTUAL) += physaddr.o KASAN_SANITIZE_physaddr.o += n diff --git a/arch/arm64/mm/trans_table.c b/arch/arm64/mm/trans_table.c new file mode 100644 index 000000000000..d5729eb318b7 --- /dev/null +++ b/arch/arm64/mm/trans_table.c @@ -0,0 +1,272 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +/* + * Copyright (c) 2019, Microsoft Corporation. + * Pavel Tatashin + */ + +/* + * Transitional tables are used during system transferring from one world to + * another: such as during hibernate restore, and kexec reboots. During these + * phases one cannot rely on page table not being overwritten. + * + */ + +#include +#include +#include + +static void *trans_alloc(struct trans_table_info *info) +{ + void *page = info->trans_alloc_page(info->trans_alloc_arg); + + if (page) + clear_page(page); + + return page; +} + +static int trans_table_copy_pte(struct trans_table_info *info, pte_t *dst_ptep, + pte_t *src_ptep, unsigned long start, + unsigned long end) +{ + unsigned long addr = start; + int i = pgd_index(addr); + + do { + pte_t src_pte = READ_ONCE(src_ptep[i]); + + if (pte_none(src_pte)) + continue; + if (info->trans_flags & TRANS_MKWRITE) + src_pte = pte_mkwrite(src_pte); + if (info->trans_flags & TRANS_MKVALID) + src_pte = pte_mkpresent(src_pte); + if (info->trans_flags & TRANS_CHECKPFN) { + if (!pfn_valid(pte_pfn(src_pte))) + return -ENXIO; + } + set_pte(&dst_ptep[i], src_pte); + } while (addr += PAGE_SIZE, i++, addr != end && i < PTRS_PER_PTE); + + return 0; +} + +static int trans_table_copy_pmd(struct trans_table_info *info, pmd_t *dst_pmdp, + pmd_t *src_pmdp, unsigned long start, + unsigned long end) +{ + unsigned long next; + unsigned long addr = start; + int i = pgd_index(addr); + int rc; + + do { + pmd_t src_pmd = READ_ONCE(src_pmdp[i]); + pmd_t dst_pmd = READ_ONCE(dst_pmdp[i]); + pte_t *dst_ptep, *src_ptep; + + next = pmd_addr_end(addr, end); + if (pmd_none(src_pmd)) + continue; + + if (!pmd_table(src_pmd)) { + if (info->trans_flags & TRANS_MKWRITE) + pmd_val(src_pmd) &= ~PMD_SECT_RDONLY; + set_pmd(&dst_pmdp[i], src_pmd); + continue; + } + + if (pmd_none(dst_pmd)) { + pte_t *t = trans_alloc(info); + + if (!t) + return -ENOMEM; + + __pmd_populate(&dst_pmdp[i], __pa(t), PTE_TYPE_PAGE); + dst_pmd = READ_ONCE(dst_pmdp[i]); + } + + src_ptep = __va(pmd_page_paddr(src_pmd)); + dst_ptep = __va(pmd_page_paddr(dst_pmd)); + + rc = trans_table_copy_pte(info, dst_ptep, src_ptep, addr, next); + if (rc) + return rc; + } while (addr = next, i++, addr != end && i < PTRS_PER_PMD); + + return 0; +} + +static int trans_table_copy_pud(struct trans_table_info *info, pud_t *dst_pudp, + pud_t *src_pudp, unsigned long start, + unsigned long end) +{ + unsigned long next; + unsigned long addr = start; + int i = pgd_index(addr); + int rc; + + do { + pud_t src_pud = READ_ONCE(src_pudp[i]); + pud_t dst_pud = READ_ONCE(dst_pudp[i]); + pmd_t *dst_pmdp, *src_pmdp; + + next = pud_addr_end(addr, end); + if (pud_none(src_pud)) + continue; + + if (!pud_table(src_pud)) { + if (info->trans_flags & TRANS_MKWRITE) + pud_val(src_pud) &= ~PUD_SECT_RDONLY; + set_pud(&dst_pudp[i], src_pud); + continue; + } + + if (pud_none(dst_pud)) { + pmd_t *t = trans_alloc(info); + + if (!t) + return -ENOMEM; + + __pud_populate(&dst_pudp[i], __pa(t), PMD_TYPE_TABLE); + dst_pud = READ_ONCE(dst_pudp[i]); + } + + src_pmdp = __va(pud_page_paddr(src_pud)); + dst_pmdp = __va(pud_page_paddr(dst_pud)); + + rc = trans_table_copy_pmd(info, dst_pmdp, src_pmdp, addr, next); + if (rc) + return rc; + } while (addr = next, i++, addr != end && i < PTRS_PER_PUD); + + return 0; +} + +static int trans_table_copy_pgd(struct trans_table_info *info, pgd_t *dst_pgdp, + pgd_t *src_pgdp, unsigned long start, + unsigned long end) +{ + unsigned long next; + unsigned long addr = start; + int i = pgd_index(addr); + int rc; + + do { + pgd_t src_pgd; + pgd_t dst_pgd; + pud_t *dst_pudp, *src_pudp; + + src_pgd = READ_ONCE(src_pgdp[i]); + dst_pgd = READ_ONCE(dst_pgdp[i]); + next = pgd_addr_end(addr, end); + if (pgd_none(src_pgd)) + continue; + + if (pgd_none(dst_pgd)) { + pud_t *t = trans_alloc(info); + + if (!t) + return -ENOMEM; + + __pgd_populate(&dst_pgdp[i], __pa(t), PUD_TYPE_TABLE); + dst_pgd = READ_ONCE(dst_pgdp[i]); + } + + src_pudp = __va(pgd_page_paddr(src_pgd)); + dst_pudp = __va(pgd_page_paddr(dst_pgd)); + + rc = trans_table_copy_pud(info, dst_pudp, src_pudp, addr, next); + if (rc) + return rc; + } while (addr = next, i++, addr != end && i < PTRS_PER_PGD); + + return 0; +} + +int trans_table_create_empty(struct trans_table_info *info, pgd_t **trans_table) +{ + pgd_t *dst_pgdp = trans_alloc(info); + + if (!dst_pgdp) + return -ENOMEM; + + *trans_table = dst_pgdp; + + return 0; +} + +int trans_table_create_copy(struct trans_table_info *info, pgd_t **trans_table, + pgd_t *from_table, unsigned long start, + unsigned long end) +{ + int rc; + + rc = trans_table_create_empty(info, trans_table); + if (rc) + return rc; + + return trans_table_copy_pgd(info, *trans_table, from_table, start, end); +} + +int trans_table_map_page(struct trans_table_info *info, pgd_t *trans_table, + void *page, unsigned long dst_addr, pgprot_t pgprot) +{ + int pgd_idx = pgd_index(dst_addr); + int pud_idx = pud_index(dst_addr); + int pmd_idx = pmd_index(dst_addr); + int pte_idx = pte_index(dst_addr); + pgd_t *pgdp = trans_table; + pgd_t pgd = READ_ONCE(pgdp[pgd_idx]); + pud_t *pudp, pud; + pmd_t *pmdp, pmd; + pte_t *ptep, pte; + + if (pgd_none(pgd)) { + pud_t *t = trans_alloc(info); + + if (!t) + return -ENOMEM; + + __pgd_populate(&pgdp[pgd_idx], __pa(t), PUD_TYPE_TABLE); + pgd = READ_ONCE(pgdp[pgd_idx]); + } + + pudp = __va(pgd_page_paddr(pgd)); + pud = READ_ONCE(pudp[pud_idx]); + if (pud_sect(pud) && !(info->trans_flags & TRANS_FORCEMAP)) { + return -ENXIO; + } else if (pud_none(pud) || pud_sect(pud)) { + pmd_t *t = trans_alloc(info); + + if (!t) + return -ENOMEM; + + __pud_populate(&pudp[pud_idx], __pa(t), PMD_TYPE_TABLE); + pud = READ_ONCE(pudp[pud_idx]); + } + + pmdp = __va(pud_page_paddr(pud)); + pmd = READ_ONCE(pmdp[pmd_idx]); + if (pmd_sect(pmd) && !(info->trans_flags & TRANS_FORCEMAP)) { + return -ENXIO; + } else if (pmd_none(pmd) || pmd_sect(pmd)) { + pte_t *t = trans_alloc(info); + + if (!t) + return -ENOMEM; + + __pmd_populate(&pmdp[pmd_idx], __pa(t), PTE_TYPE_PAGE); + pmd = READ_ONCE(pmdp[pmd_idx]); + } + + ptep = __va(pmd_page_paddr(pmd)); + pte = READ_ONCE(ptep[pte_idx]); + + if (!pte_none(pte) && !(info->trans_flags & TRANS_FORCEMAP)) + return -ENXIO; + + set_pte(&ptep[pte_idx], pfn_pte(virt_to_pfn(page), pgprot)); + + return 0; +} From patchwork Wed Jul 31 15:38:52 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 11068417 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E8E2E1399 for ; Wed, 31 Jul 2019 15:40:24 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D2E362228E for ; Wed, 31 Jul 2019 15:40:24 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C6F7D22701; Wed, 31 Jul 2019 15:40:24 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 0743D2228E for ; Wed, 31 Jul 2019 15:40:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Cc:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=VCy4Qe3ic5wrZFnYFd/ChvstHVWFJCqp0DPITgU8rOY=; b=cYozm6HlPvZV6l lDsUBB0hoEgtviBn0Rl+aBorn/TWci9wqTmWFDIt8sB6nknQCGJytffV9Iy97YBmT0soFBK9XLO+x +O8wY1l+eKpZTbac25ZiSy6gH/3LfK82kiJGXErcbazrBuam9Dxvp2ajXMF7VUMFN15UwITwYlkBS WSaAtKTeJGdA0TLE5XMsN9WGu5VO5mxow2g7Ch+MZg/AQr0579Y1FASzhofQPoNmDdJlygsR2Xvtw 7fGBWwFG4Z4LVXLTPdDB+FUhR1nEv5ln1u4Zshuk+AUcaZp109WRkRxeaLA+h4jFpO4lXXCjbqMkQ MNwcGSXwjvNcAnp60tCA==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92 #3 (Red Hat Linux)) id 1hsqiF-0001rt-CG; Wed, 31 Jul 2019 15:40:23 +0000 Received: from mail-qt1-x843.google.com ([2607:f8b0:4864:20::843]) by bombadil.infradead.org with esmtps (Exim 4.92 #3 (Red Hat Linux)) id 1hsqgy-0007jn-L3 for linux-arm-kernel@lists.infradead.org; Wed, 31 Jul 2019 15:39:07 +0000 Received: by mail-qt1-x843.google.com with SMTP id h21so66968340qtn.13 for ; Wed, 31 Jul 2019 08:39:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=HM9ffM3Ct5ZgrBhxNWuGZauxxnBjbr17mWM+iliA67E=; b=jtW03mPTNpMTc670V7njVtxY3k9/5re7CPdcDVcjT8bs93LHBR/EwAoPKSM5CO66+g hTBBAMk+lFoO1JPuUr6B+jUdyWPMVaL0zSsMzPkWRADDsQOYmuH4TCC+wxAXpjLFKD4w KGJrc5XS1hZQ3yzE5VgSldS4frMGJhch9ftzdFTDTY//0Z0prz68KOF1kf17GnObnsBu 4rAXeRV/85tm0sMjdFmdQL956iLpCT7InnhzG94VJg1cFFlHbBUDU9I6CKN6tqiLxiYO emrZCUupb67hUf2bQ9Va80hRqLAoxfkruZCoyBWZaUtifur9fVAe5BR4zf3V2oTEM6WR Bw7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=HM9ffM3Ct5ZgrBhxNWuGZauxxnBjbr17mWM+iliA67E=; b=sCrAKnddtPNcIGIXiVl/DofYt2ffbfWaiYttVoLdmmbQAvgBxZiqWfvHR5oAEFHCJM viIJbfXmyGPi2OJH5tCKTHYJQhDD4FGbFz8p7R/qR5+1/K9M2KwzlfyjUyuDoso40mB7 EMYo5ubDHSdyWPtLcFdc5ala3FAFp+fzW2kMjakwanXYcV9UQUUW4nfd2rgKGIIjP7i6 XWgXDmLVvr3exIp8txWp2B/HNL0262gNnMsJYkKTgYrPBtbNydLwAlkmbgFiNHxHl3md 8njWy3nM0MV6dd5f2NGkQBghCAQNhLPgBxl2LcRFaYkP62hnOCEi6sd4xsmtLH6VvDiu jQiA== X-Gm-Message-State: APjAAAUIpUhdUVITbHBEOE53Lcqf5XTDFhozCGlZN35V8i9/jWvn5C3K MmW+zpfarKZTERmLe0jOJXE= X-Google-Smtp-Source: APXvYqyfqk/oPFVHamoZx5msXhvQjnnSt2fmBBceeU/q9amOUBgwudcPBaoce1lVzuJHnUhWj0XvRA== X-Received: by 2002:a0c:a8d2:: with SMTP id h18mr88972679qvc.16.1564587543515; Wed, 31 Jul 2019 08:39:03 -0700 (PDT) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id f25sm35116803qta.81.2019.07.31.08.39.02 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 31 Jul 2019 08:39:02 -0700 (PDT) From: Pavel Tatashin To: pasha.tatashin@soleen.com, jmorris@namei.org, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-doc@vger.kernel.org, linux-arm-kernel@lists.infradead.org, marc.zyngier@arm.com, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, bhsharma@redhat.com Subject: [RFC v2 3/8] arm64: hibernate: switch to transtional page tables. Date: Wed, 31 Jul 2019 11:38:52 -0400 Message-Id: <20190731153857.4045-4-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190731153857.4045-1-pasha.tatashin@soleen.com> References: <20190731153857.4045-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190731_083904_711910_2EDDD469 X-CRM114-Status: GOOD ( 18.32 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Transitional page tables provide the needed functionality to setup temporary page tables needed for hibernate resume. Signed-off-by: Pavel Tatashin --- arch/arm64/kernel/hibernate.c | 261 ++++++++-------------------------- 1 file changed, 60 insertions(+), 201 deletions(-) diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c index 9341fcc6e809..4120b03a02fd 100644 --- a/arch/arm64/kernel/hibernate.c +++ b/arch/arm64/kernel/hibernate.c @@ -16,7 +16,6 @@ #define pr_fmt(x) "hibernate: " x #include #include -#include #include #include #include @@ -31,14 +30,12 @@ #include #include #include -#include -#include -#include #include #include #include #include #include +#include #include /* @@ -182,6 +179,12 @@ int arch_hibernation_header_restore(void *addr) } EXPORT_SYMBOL(arch_hibernation_header_restore); +static void * +hibernate_page_alloc(void *arg) +{ + return (void *)get_safe_page((gfp_t)(unsigned long)arg); +} + /* * Copies length bytes, starting at src_start into an new page, * perform cache maintentance, then maps it at the specified address low @@ -196,57 +199,31 @@ EXPORT_SYMBOL(arch_hibernation_header_restore); */ static int create_safe_exec_page(void *src_start, size_t length, unsigned long dst_addr, - phys_addr_t *phys_dst_addr, - void *(*allocator)(gfp_t mask), - gfp_t mask) + phys_addr_t *phys_dst_addr) { - int rc = 0; - pgd_t *pgdp; - pud_t *pudp; - pmd_t *pmdp; - pte_t *ptep; - unsigned long dst = (unsigned long)allocator(mask); - - if (!dst) { - rc = -ENOMEM; - goto out; - } - - memcpy((void *)dst, src_start, length); - __flush_icache_range(dst, dst + length); + struct trans_table_info trans_info = { + .trans_alloc_page = hibernate_page_alloc, + .trans_alloc_arg = (void *)GFP_ATOMIC, + .trans_flags = 0, + }; + void *page = (void *)get_safe_page(GFP_ATOMIC); + pgd_t *trans_table; + int rc; + + if (!page) + return -ENOMEM; - pgdp = pgd_offset_raw(allocator(mask), dst_addr); - if (pgd_none(READ_ONCE(*pgdp))) { - pudp = allocator(mask); - if (!pudp) { - rc = -ENOMEM; - goto out; - } - pgd_populate(&init_mm, pgdp, pudp); - } + memcpy(page, src_start, length); + __flush_icache_range((unsigned long)page, (unsigned long)page + length); - pudp = pud_offset(pgdp, dst_addr); - if (pud_none(READ_ONCE(*pudp))) { - pmdp = allocator(mask); - if (!pmdp) { - rc = -ENOMEM; - goto out; - } - pud_populate(&init_mm, pudp, pmdp); - } - - pmdp = pmd_offset(pudp, dst_addr); - if (pmd_none(READ_ONCE(*pmdp))) { - ptep = allocator(mask); - if (!ptep) { - rc = -ENOMEM; - goto out; - } - pmd_populate_kernel(&init_mm, pmdp, ptep); - } + rc = trans_table_create_empty(&trans_info, &trans_table); + if (rc) + return rc; - ptep = pte_offset_kernel(pmdp, dst_addr); - set_pte(ptep, pfn_pte(virt_to_pfn(dst), PAGE_KERNEL_EXEC)); + rc = trans_table_map_page(&trans_info, trans_table, page, dst_addr, + PAGE_KERNEL_EXEC); + if (rc) + return rc; /* * Load our new page tables. A strict BBM approach requires that we @@ -262,13 +239,12 @@ static int create_safe_exec_page(void *src_start, size_t length, */ cpu_set_reserved_ttbr0(); local_flush_tlb_all(); - write_sysreg(phys_to_ttbr(virt_to_phys(pgdp)), ttbr0_el1); + write_sysreg(phys_to_ttbr(virt_to_phys(trans_table)), ttbr0_el1); isb(); - *phys_dst_addr = virt_to_phys((void *)dst); + *phys_dst_addr = virt_to_phys(page); -out: - return rc; + return 0; } #define dcache_clean_range(start, end) __flush_dcache_area(start, (end - start)) @@ -332,143 +308,6 @@ int swsusp_arch_suspend(void) return ret; } -static void _copy_pte(pte_t *dst_ptep, pte_t *src_ptep, unsigned long addr) -{ - pte_t pte = READ_ONCE(*src_ptep); - - if (pte_valid(pte)) { - /* - * Resume will overwrite areas that may be marked - * read only (code, rodata). Clear the RDONLY bit from - * the temporary mappings we use during restore. - */ - set_pte(dst_ptep, pte_mkwrite(pte)); - } else if (debug_pagealloc_enabled() && !pte_none(pte)) { - /* - * debug_pagealloc will removed the PTE_VALID bit if - * the page isn't in use by the resume kernel. It may have - * been in use by the original kernel, in which case we need - * to put it back in our copy to do the restore. - * - * Before marking this entry valid, check the pfn should - * be mapped. - */ - BUG_ON(!pfn_valid(pte_pfn(pte))); - - set_pte(dst_ptep, pte_mkpresent(pte_mkwrite(pte))); - } -} - -static int copy_pte(pmd_t *dst_pmdp, pmd_t *src_pmdp, unsigned long start, - unsigned long end) -{ - pte_t *src_ptep; - pte_t *dst_ptep; - unsigned long addr = start; - - dst_ptep = (pte_t *)get_safe_page(GFP_ATOMIC); - if (!dst_ptep) - return -ENOMEM; - pmd_populate_kernel(&init_mm, dst_pmdp, dst_ptep); - dst_ptep = pte_offset_kernel(dst_pmdp, start); - - src_ptep = pte_offset_kernel(src_pmdp, start); - do { - _copy_pte(dst_ptep, src_ptep, addr); - } while (dst_ptep++, src_ptep++, addr += PAGE_SIZE, addr != end); - - return 0; -} - -static int copy_pmd(pud_t *dst_pudp, pud_t *src_pudp, unsigned long start, - unsigned long end) -{ - pmd_t *src_pmdp; - pmd_t *dst_pmdp; - unsigned long next; - unsigned long addr = start; - - if (pud_none(READ_ONCE(*dst_pudp))) { - dst_pmdp = (pmd_t *)get_safe_page(GFP_ATOMIC); - if (!dst_pmdp) - return -ENOMEM; - pud_populate(&init_mm, dst_pudp, dst_pmdp); - } - dst_pmdp = pmd_offset(dst_pudp, start); - - src_pmdp = pmd_offset(src_pudp, start); - do { - pmd_t pmd = READ_ONCE(*src_pmdp); - - next = pmd_addr_end(addr, end); - if (pmd_none(pmd)) - continue; - if (pmd_table(pmd)) { - if (copy_pte(dst_pmdp, src_pmdp, addr, next)) - return -ENOMEM; - } else { - set_pmd(dst_pmdp, - __pmd(pmd_val(pmd) & ~PMD_SECT_RDONLY)); - } - } while (dst_pmdp++, src_pmdp++, addr = next, addr != end); - - return 0; -} - -static int copy_pud(pgd_t *dst_pgdp, pgd_t *src_pgdp, unsigned long start, - unsigned long end) -{ - pud_t *dst_pudp; - pud_t *src_pudp; - unsigned long next; - unsigned long addr = start; - - if (pgd_none(READ_ONCE(*dst_pgdp))) { - dst_pudp = (pud_t *)get_safe_page(GFP_ATOMIC); - if (!dst_pudp) - return -ENOMEM; - pgd_populate(&init_mm, dst_pgdp, dst_pudp); - } - dst_pudp = pud_offset(dst_pgdp, start); - - src_pudp = pud_offset(src_pgdp, start); - do { - pud_t pud = READ_ONCE(*src_pudp); - - next = pud_addr_end(addr, end); - if (pud_none(pud)) - continue; - if (pud_table(pud)) { - if (copy_pmd(dst_pudp, src_pudp, addr, next)) - return -ENOMEM; - } else { - set_pud(dst_pudp, - __pud(pud_val(pud) & ~PMD_SECT_RDONLY)); - } - } while (dst_pudp++, src_pudp++, addr = next, addr != end); - - return 0; -} - -static int copy_page_tables(pgd_t *dst_pgdp, unsigned long start, - unsigned long end) -{ - unsigned long next; - unsigned long addr = start; - pgd_t *src_pgdp = pgd_offset_k(start); - - dst_pgdp = pgd_offset_raw(dst_pgdp, start); - do { - next = pgd_addr_end(addr, end); - if (pgd_none(READ_ONCE(*src_pgdp))) - continue; - if (copy_pud(dst_pgdp, src_pgdp, addr, next)) - return -ENOMEM; - } while (dst_pgdp++, src_pgdp++, addr = next, addr != end); - - return 0; -} - /* * Setup then Resume from the hibernate image using swsusp_arch_suspend_exit(). * @@ -484,21 +323,42 @@ int swsusp_arch_resume(void) phys_addr_t phys_hibernate_exit; void __noreturn (*hibernate_exit)(phys_addr_t, phys_addr_t, void *, void *, phys_addr_t, phys_addr_t); + struct trans_table_info trans_info = { + .trans_alloc_page = hibernate_page_alloc, + .trans_alloc_arg = (void *)GFP_ATOMIC, + /* + * Resume will overwrite areas that may be marked read only + * (code, rodata). Clear the RDONLY bit from the temporary + * mappings we use during restore. + */ + .trans_flags = TRANS_MKWRITE, + }; + + /* + * debug_pagealloc will removed the PTE_VALID bit if the page isn't in + * use by the resume kernel. It may have been in use by the original + * kernel, in which case we need to put it back in our copy to do the + * restore. + * + * Before marking this entry valid, check the pfn should be mapped. + */ + if (debug_pagealloc_enabled()) + trans_info.trans_flags |= (TRANS_MKVALID | TRANS_CHECKPFN); /* * Restoring the memory image will overwrite the ttbr1 page tables. * Create a second copy of just the linear map, and use this when * restoring. */ - tmp_pg_dir = (pgd_t *)get_safe_page(GFP_ATOMIC); - if (!tmp_pg_dir) { - pr_err("Failed to allocate memory for temporary page tables.\n"); - rc = -ENOMEM; + rc = trans_table_create_copy(&trans_info, &tmp_pg_dir, + pgd_offset_k(PAGE_OFFSET), PAGE_OFFSET, 0); + if (rc) { + if (rc == -ENOMEM) + pr_err("Failed to allocate memory for temporary page tables.\n"); + else if (rc == -ENXIO) + pr_err("Tried to set PTE for PFN that does not exist\n"); goto out; } - rc = copy_page_tables(tmp_pg_dir, PAGE_OFFSET, 0); - if (rc) - goto out; /* * We need a zero page that is zero before & after resume in order to @@ -523,8 +383,7 @@ int swsusp_arch_resume(void) */ rc = create_safe_exec_page(__hibernate_exit_text_start, exit_size, (unsigned long)hibernate_exit, - &phys_hibernate_exit, - (void *)get_safe_page, GFP_ATOMIC); + &phys_hibernate_exit); if (rc) { pr_err("Failed to create safe executable page for hibernate_exit code.\n"); goto out; From patchwork Wed Jul 31 15:38:53 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 11068413 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 080CA112C for ; Wed, 31 Jul 2019 15:40:04 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E3F1B205E9 for ; Wed, 31 Jul 2019 15:40:03 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D6851209CD; Wed, 31 Jul 2019 15:40:03 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 2464F1FF28 for ; Wed, 31 Jul 2019 15:40:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Cc:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=KmjLIGzxVNfj4j8K3trzB/5LOpbjvNu+aopoafw8Evc=; b=qiGrziGMzH7Dwj cSeq9L9/IEmL07wpoYrQpBkjCWncmieG2JiI0Q+kwqm/Qd5Sm8uZ+cVbCnSbtdvVdUDpI5+D162od FPef+r+PZi9nX+DgpxcHSlaMp9YOoKNxMKb/aCXkYF3KUOBvO/sn4rXDbqgaLUN+8san4L9qNTo89 IC686s4kjFS+iFFB6ZpfdFWwTVEHPJTZKF2NTODUNQxsxPD9/5kX8hVHaftY1UGENaCjzv2JKqoBw T86kFGSwwVxC1bqh2ieBpBVxtyt8QnHU3Jz2aILxkXUxsU7iNFxldjawu+aSJtyzkdo4hmiew+Jif zG2dqfKoSTybgfUJR6eg==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92 #3 (Red Hat Linux)) id 1hsqhl-0000CQ-Oo; Wed, 31 Jul 2019 15:39:53 +0000 Received: from mail-qt1-x844.google.com ([2607:f8b0:4864:20::844]) by bombadil.infradead.org with esmtps (Exim 4.92 #3 (Red Hat Linux)) id 1hsqgz-0007lR-HC for linux-arm-kernel@lists.infradead.org; Wed, 31 Jul 2019 15:39:07 +0000 Received: by mail-qt1-x844.google.com with SMTP id n11so67002003qtl.5 for ; Wed, 31 Jul 2019 08:39:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=a8h6/KaC5Piq1VkFuiQLhvl2ZdeRxBmHZozqoP8J2iw=; b=kozB+ytYyHw9RhluUzblYdsFfY3Dv43ogPWJXWl7P1oNu+oyHMha8I7lwP+NjNLmZt UOmHrGMgf5sYEl0a2/cKbMoV2kq9Z/eQbDzblyi13+Xrm/Tnp3ITIQqK1UfJd/qxpXkS 0aGI974bEakxPT/bxoEOaNl18hQ3yeYLEut1AtR1KIw+epSAEQ54Dfi5T0SdU66hCyjE QihiV01sf1ndWDDZ0NB2//SZoDf37BmsurqPijIqDVf/LkvSW63siZlcfNnAjGWzoWSn 1L7bkl1Cd8jJxCulHiPGHEUuMn2KzotmzW4qJ4xVa+yuSrh8dbaqho+xrP5uejKhShlF +hrg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=a8h6/KaC5Piq1VkFuiQLhvl2ZdeRxBmHZozqoP8J2iw=; b=BXWkPrpyY4yTSX1wn12H3mdPlegWE7z+1UYvZ6/HG1Enc+TzLUB4MtoKhOF2+3Glgd LIFS1Cjrj/ovS4BGOoEEiKBJrgjgSk23WOuYkT/+4gGLOb9RpetxuR+rfHBZlog4FoZ3 qOCJhIvtYyarLLrukaLF8VXVZBUMxqF9LTyUtveyqPjo9ov5HeCE8yJ45MJ1g9sgnICM 8btU9pNgNpmMKLKZCDKfH0dgkfPMHJT2yZsME+DLGc6Xwn1yJ6iqOKSSCh9pz4S1NUV5 MMHYfb+GCTgiS14q/4WLUPbsXmGBDawICq47VRZGaxaMx77R0xxSGNYnNOBd8sCmX8jf Ptyw== X-Gm-Message-State: APjAAAUomemQtTTOMHcxftjcvGUCQCZE0Xs9PMBrVlMmdoe0r+JTe/Un 8Jz26fxqqq1a1wCvxsaf3GM= X-Google-Smtp-Source: APXvYqwuACOTaf8gffAijH2Zzx0ksDXKqJiEVvsEccygqEisgOhp5Vs/4mMgCK39Hh+y7F4PhUqq4w== X-Received: by 2002:a0c:d003:: with SMTP id u3mr90210411qvg.112.1564587544806; Wed, 31 Jul 2019 08:39:04 -0700 (PDT) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id f25sm35116803qta.81.2019.07.31.08.39.03 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 31 Jul 2019 08:39:04 -0700 (PDT) From: Pavel Tatashin To: pasha.tatashin@soleen.com, jmorris@namei.org, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-doc@vger.kernel.org, linux-arm-kernel@lists.infradead.org, marc.zyngier@arm.com, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, bhsharma@redhat.com Subject: [RFC v2 4/8] kexec: add machine_kexec_post_load() Date: Wed, 31 Jul 2019 11:38:53 -0400 Message-Id: <20190731153857.4045-5-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190731153857.4045-1-pasha.tatashin@soleen.com> References: <20190731153857.4045-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190731_083905_603786_2CF18D77 X-CRM114-Status: GOOD ( 11.96 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP It is the same as machine_kexec_prepare(), but is called after segments are loaded. This way, can do processing work with already loaded relocation segments. One such example is arm64: it has to have segments loaded in order to create a page table, but it cannot do it during kexec time, because at that time allocations won't be possible anymore. Signed-off-by: Pavel Tatashin --- kernel/kexec.c | 4 ++++ kernel/kexec_core.c | 6 ++++++ kernel/kexec_file.c | 4 ++++ kernel/kexec_internal.h | 2 ++ 4 files changed, 16 insertions(+) diff --git a/kernel/kexec.c b/kernel/kexec.c index 1b018f1a6e0d..27b71dc7b35a 100644 --- a/kernel/kexec.c +++ b/kernel/kexec.c @@ -159,6 +159,10 @@ static int do_kexec_load(unsigned long entry, unsigned long nr_segments, kimage_terminate(image); + ret = machine_kexec_post_load(image); + if (ret) + goto out; + /* Install the new kernel and uninstall the old */ image = xchg(dest_image, image); diff --git a/kernel/kexec_core.c b/kernel/kexec_core.c index 2c5b72863b7b..8360645d1bbe 100644 --- a/kernel/kexec_core.c +++ b/kernel/kexec_core.c @@ -587,6 +587,12 @@ static void kimage_free_extra_pages(struct kimage *image) kimage_free_page_list(&image->unusable_pages); } + +int __weak machine_kexec_post_load(struct kimage *image) +{ + return 0; +} + void kimage_terminate(struct kimage *image) { if (*image->entry != 0) diff --git a/kernel/kexec_file.c b/kernel/kexec_file.c index b8cc032d5620..cb531d768114 100644 --- a/kernel/kexec_file.c +++ b/kernel/kexec_file.c @@ -391,6 +391,10 @@ SYSCALL_DEFINE5(kexec_file_load, int, kernel_fd, int, initrd_fd, kimage_terminate(image); + ret = machine_kexec_post_load(image); + if (ret) + goto out; + /* * Free up any temporary buffers allocated which are not needed * after image has been loaded diff --git a/kernel/kexec_internal.h b/kernel/kexec_internal.h index 48aaf2ac0d0d..39d30ccf8d87 100644 --- a/kernel/kexec_internal.h +++ b/kernel/kexec_internal.h @@ -13,6 +13,8 @@ void kimage_terminate(struct kimage *image); int kimage_is_destination_range(struct kimage *image, unsigned long start, unsigned long end); +int machine_kexec_post_load(struct kimage *image); + extern struct mutex kexec_mutex; #ifdef CONFIG_KEXEC_FILE From patchwork Wed Jul 31 15:38:54 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 11068415 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 96847112C for ; Wed, 31 Jul 2019 15:40:16 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8065E22701 for ; Wed, 31 Jul 2019 15:40:16 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 7381C22B39; Wed, 31 Jul 2019 15:40:16 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id E317421327 for ; Wed, 31 Jul 2019 15:40:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Cc:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=AD9W61YnBHxmo1K9x/sZLM/+ZaEffZ0hZK8IN0FZhf4=; b=BPviX704fng/v+ 2Rtztjtptka1vyZfxnuY4bDfT2kStBG55auYic3VTWivdq4ovqowzKgLpMMMguvLO/c/1pmH5cs3M YVbii5/y4Jxcc+vV8este1c4rNaGMahCkFkWnPvSmX+3umAUGtKmrkuDax/dR868ooGrYvMwrRCNx UqUeDWhd3JGznko5jNX7vLoXtJqi8O/j6BNTnjn9gFqRVtBBQOrM6RKUIN79F60H7fvnwfcr+mSGz 9l5tWa3oe5B+a8Fijc5Ce/3pNQdD6inQJT8nT0IKbD+ik3C+zz443Ldka1CcNglSAoNSRCCxoze/K CZi1zzZboli689knr5mg==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92 #3 (Red Hat Linux)) id 1hsqi7-0001f3-Gt; Wed, 31 Jul 2019 15:40:15 +0000 Received: from mail-qt1-x842.google.com ([2607:f8b0:4864:20::842]) by bombadil.infradead.org with esmtps (Exim 4.92 #3 (Red Hat Linux)) id 1hsqh0-0007mv-Qd for linux-arm-kernel@lists.infradead.org; Wed, 31 Jul 2019 15:39:09 +0000 Received: by mail-qt1-x842.google.com with SMTP id k10so67098942qtq.1 for ; Wed, 31 Jul 2019 08:39:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=vwP4vIc0noA3GXpIp2IQdFlP9YBsdVIQ86rMMFQHlJM=; b=bmnJDnBXrFzsPdPJlhIiPTmaGeesbqD7rZSzxNAb8BxmIp23jrZdPfMLRFi14l2ssi +MldCdNoTReJdzXbI87YQgXkSZsOY/uTDTfGENd9Z0oAFGj4d08FwR5y5nA3Pyswlt8P 5OHOTrLRllqWgDkSfrheQcXD3/mUhh7DqVH741r+Pg9+Q7blN+6TT2PrWXHOpftkcrcr kBYok4X3V7QSAgSrZWqymy7LSkr3BMjCJqxxhfXbebhSHhsoqSmE1JkNEwPqUHYNS5H1 ROWgpLyL2FgY3FFvQqmz+ZPEu9uztX7qVxnifJW4I+AahWwcJKKJ2s7dFzQUYdF/h2XQ V4vQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=vwP4vIc0noA3GXpIp2IQdFlP9YBsdVIQ86rMMFQHlJM=; b=WzdR45S0bkHy/MP+J2G5ZMwCBgYeTL9Bk+XPtf/LEenXXt2J62Cy3C91pbmkCCDdwU X/nvo3TEgkCe9uFh/3nCGVjtwF4A5T0eid4IhOyyiNuLjMPIS2pXXhof0A7P3hQQS4wW e24aK2c44rDNZYhD3ZWfia55MMeWOfSw5bOidimJmLZfIUyHqzpoYjLJ0i8df85ZI/Ns NsW2IWm/QueLisVSZ1pQuo0B7JVF4rKcQOqH4VBp10YehVBqfGFMcbj0FnOVXzo9d4mI drC/743fKIGtUe4U+hAvyEaCn0ra6Q6PfLVXKHPge3D3zT8tc3UlyILme9I0GgcSA5kp Hslg== X-Gm-Message-State: APjAAAXE4PkFxeJIJyiBM+qOc8fvetMwy4r7IMfW3WOzOi16mgK055JZ fgYQ3d6qvXCbLdiIB7KBeyw= X-Google-Smtp-Source: APXvYqz1tq+CtgHliRp2L5RpkoQXD3KfaK/eiVw7PrmTvCdr9o0f1bBNHtnLrfzZMnLMdMnsyVi8nQ== X-Received: by 2002:a0c:98e9:: with SMTP id g38mr86986095qvd.187.1564587546083; Wed, 31 Jul 2019 08:39:06 -0700 (PDT) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id f25sm35116803qta.81.2019.07.31.08.39.04 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 31 Jul 2019 08:39:05 -0700 (PDT) From: Pavel Tatashin To: pasha.tatashin@soleen.com, jmorris@namei.org, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-doc@vger.kernel.org, linux-arm-kernel@lists.infradead.org, marc.zyngier@arm.com, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, bhsharma@redhat.com Subject: [RFC v2 5/8] arm64, kexec: move relocation function setup and clean up Date: Wed, 31 Jul 2019 11:38:54 -0400 Message-Id: <20190731153857.4045-6-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190731153857.4045-1-pasha.tatashin@soleen.com> References: <20190731153857.4045-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190731_083906_901199_CE835114 X-CRM114-Status: GOOD ( 13.65 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Currently, kernel relocation function is configured in machine_kexec() at the time of kexec reboot by using control_code_page. This operation, however, is more logical to be done during kexec_load, and thus remove from reboot time. Move, setup of this function to newly added machine_kexec_post_load(). In addition, do some cleanup: add infor about reloction function to kexec_image_info(), and remove extra messages from machine_kexec(). Make dtb_mem, always available, if CONFIG_KEXEC_FILE is not configured dtb_mem is set to zero anyway. Signed-off-by: Pavel Tatashin --- arch/arm64/include/asm/kexec.h | 3 +- arch/arm64/kernel/machine_kexec.c | 47 +++++++++++-------------------- 2 files changed, 18 insertions(+), 32 deletions(-) diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h index 12a561a54128..d15ca1ca1e83 100644 --- a/arch/arm64/include/asm/kexec.h +++ b/arch/arm64/include/asm/kexec.h @@ -90,14 +90,15 @@ static inline void crash_prepare_suspend(void) {} static inline void crash_post_resume(void) {} #endif -#ifdef CONFIG_KEXEC_FILE #define ARCH_HAS_KIMAGE_ARCH struct kimage_arch { void *dtb; unsigned long dtb_mem; + unsigned long kern_reloc; }; +#ifdef CONFIG_KEXEC_FILE extern const struct kexec_file_ops kexec_image_ops; struct kimage; diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c index 0df8493624e0..596c9b9657be 100644 --- a/arch/arm64/kernel/machine_kexec.c +++ b/arch/arm64/kernel/machine_kexec.c @@ -42,6 +42,7 @@ static void _kexec_image_info(const char *func, int line, pr_debug(" start: %lx\n", kimage->start); pr_debug(" head: %lx\n", kimage->head); pr_debug(" nr_segments: %lu\n", kimage->nr_segments); + pr_debug(" kern_reloc: %pa\n", &kimage->arch.kern_reloc); for (i = 0; i < kimage->nr_segments; i++) { pr_debug(" segment[%lu]: %016lx - %016lx, 0x%lx bytes, %lu pages\n", @@ -58,6 +59,19 @@ void machine_kexec_cleanup(struct kimage *kimage) /* Empty routine needed to avoid build errors. */ } +int machine_kexec_post_load(struct kimage *kimage) +{ + unsigned long kern_reloc; + + kern_reloc = page_to_phys(kimage->control_code_page); + memcpy(__va(kern_reloc), arm64_relocate_new_kernel, + arm64_relocate_new_kernel_size); + kimage->arch.kern_reloc = kern_reloc; + + kexec_image_info(kimage); + return 0; +} + /** * machine_kexec_prepare - Prepare for a kexec reboot. * @@ -67,8 +81,6 @@ void machine_kexec_cleanup(struct kimage *kimage) */ int machine_kexec_prepare(struct kimage *kimage) { - kexec_image_info(kimage); - if (kimage->type != KEXEC_TYPE_CRASH && cpus_are_stuck_in_kernel()) { pr_err("Can't kexec: CPUs are stuck in the kernel.\n"); return -EBUSY; @@ -143,8 +155,7 @@ static void kexec_segment_flush(const struct kimage *kimage) */ void machine_kexec(struct kimage *kimage) { - phys_addr_t reboot_code_buffer_phys; - void *reboot_code_buffer; + void *reboot_code_buffer = phys_to_virt(kimage->arch.kern_reloc); bool in_kexec_crash = (kimage == kexec_crash_image); bool stuck_cpus = cpus_are_stuck_in_kernel(); @@ -155,30 +166,8 @@ void machine_kexec(struct kimage *kimage) WARN(in_kexec_crash && (stuck_cpus || smp_crash_stop_failed()), "Some CPUs may be stale, kdump will be unreliable.\n"); - reboot_code_buffer_phys = page_to_phys(kimage->control_code_page); - reboot_code_buffer = phys_to_virt(reboot_code_buffer_phys); - kexec_image_info(kimage); - pr_debug("%s:%d: control_code_page: %p\n", __func__, __LINE__, - kimage->control_code_page); - pr_debug("%s:%d: reboot_code_buffer_phys: %pa\n", __func__, __LINE__, - &reboot_code_buffer_phys); - pr_debug("%s:%d: reboot_code_buffer: %p\n", __func__, __LINE__, - reboot_code_buffer); - pr_debug("%s:%d: relocate_new_kernel: %p\n", __func__, __LINE__, - arm64_relocate_new_kernel); - pr_debug("%s:%d: relocate_new_kernel_size: 0x%lx(%lu) bytes\n", - __func__, __LINE__, arm64_relocate_new_kernel_size, - arm64_relocate_new_kernel_size); - - /* - * Copy arm64_relocate_new_kernel to the reboot_code_buffer for use - * after the kernel is shut down. - */ - memcpy(reboot_code_buffer, arm64_relocate_new_kernel, - arm64_relocate_new_kernel_size); - /* Flush the reboot_code_buffer in preparation for its execution. */ __flush_dcache_area(reboot_code_buffer, arm64_relocate_new_kernel_size); @@ -214,12 +203,8 @@ void machine_kexec(struct kimage *kimage) * userspace (kexec-tools). * In kexec_file case, the kernel starts directly without purgatory. */ - cpu_soft_restart(reboot_code_buffer_phys, kimage->head, kimage->start, -#ifdef CONFIG_KEXEC_FILE + cpu_soft_restart(kimage->arch.kern_reloc, kimage->head, kimage->start, kimage->arch.dtb_mem); -#else - 0); -#endif BUG(); /* Should never get here. */ } From patchwork Wed Jul 31 15:38:55 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 11068419 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C907E1399 for ; Wed, 31 Jul 2019 15:40:42 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B12DF205FC for ; Wed, 31 Jul 2019 15:40:42 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9EC00223C6; Wed, 31 Jul 2019 15:40:42 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 98A40212E8 for ; Wed, 31 Jul 2019 15:40:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Cc:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Q5XetAIBES9gSzrSxpU6U9wlZziq70rrub53vHoHwlI=; b=ZtFVl1W4rD6AiF SULyfdxuB49+WQoe1iT6hCJ4yKBFLWddcekKF7afrpr53qU2/VCwVTmgPteq03aaIlARfGow3iTld VxnrS4ak1sAaf2qM3ERpxouvSSqGSC48VG1cU+QlOKLVwQIMll83mrpB7Bs9mwxsKK6bxqCu82y5u +EHZMneQ1XDKfkbzCKk9fnFaUXI9XxWloHdkvQsbAoKEuGhREPMZQMMF5xz9MGOOu8+/ExR5lfX5v DA3Uao12xSyw1yUGvlwQta8LZGLGTHznUrDM1k01QBSG/QDENoK5QBbVniK1xpPe25RqStXhk4C09 aMtD5pi+GswkdWqTidig==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92 #3 (Red Hat Linux)) id 1hsqiX-0002E1-1r; Wed, 31 Jul 2019 15:40:41 +0000 Received: from mail-qt1-x841.google.com ([2607:f8b0:4864:20::841]) by bombadil.infradead.org with esmtps (Exim 4.92 #3 (Red Hat Linux)) id 1hsqh2-0007or-C0 for linux-arm-kernel@lists.infradead.org; Wed, 31 Jul 2019 15:39:11 +0000 Received: by mail-qt1-x841.google.com with SMTP id x22so62087073qtp.12 for ; Wed, 31 Jul 2019 08:39:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=HfSXPiGWGYi/ikK0oDDpXoRIKr8iFDNDIcJZS6CIIOc=; b=j4j3YSNY3h8tHzD3ckH8/z10UuMz2lFPFVEBTi/7pcsCKaM74gvRswce15eYMbz3+G dUWEA9MFRF16sF8VixrP9kw2QiQ0Js1I5NPZjx/SAH9aPOVUMEHMQv8nKCjtdGqbyvyn TMCPgKL8hC1IEhoOzhV4iDSlPfai7FRx7wWlJXMUeUq/9MWC+Yi0sGp4Dk6DU99/56+w JG0nzftoscsOIkME1QBmgagKqVu2VBnqOb02ehAqJvjRVPnj9eexRL1WFCAf2Z6TsHES yyO9k17TGynfiG5ub91YDjn5ZWiU9eeTzAidZ+dFYbICLjus1l0vDwb14Z2022Z6IpiJ xzeQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=HfSXPiGWGYi/ikK0oDDpXoRIKr8iFDNDIcJZS6CIIOc=; b=CtdoSK89MIrXi+jbmgRMIOs/tFAshjaGe0uyiqN23Kol6wQiskJsJzkPer23JCYcbB lv2FUOn5CmrEORiSKY5Ia0504ykUQ27XyESrF+CA76iF7jqoWSeuu3V6mGPFoMzFGudP uMRaN+Yt7Iud8jw8TLJhdI3nJXq0u59dNfJDLTlfX0/1g5d6tnWT/1Sq/gl7uUUXZaag TyBnubvLqFHIM+kwnGjZbFwwh6fXaua/BXjMRswoMM7kluoS+fjXjos+VdWTLngYhrkJ nybomkemXsNddwB8t1kp37p+xWwSkmqN7ndI3gjeptzcHjgHX5ZSlzxk5ipjJjE8kzye Rc2g== X-Gm-Message-State: APjAAAXxVcvhfF+ET/RWNynxjx61jGF5g21Dzx/fM1BrneBsI6uArAVa OdLAwFuDBB7F4bQ3QxaYOB0= X-Google-Smtp-Source: APXvYqy3ereCRLmwOXo+40M/C0he4SX0Oik2C6bodGJN0qN7iK9y+qtO26TmEDu+oYdfzBb8OLjDbA== X-Received: by 2002:a0c:8dc7:: with SMTP id u7mr87970914qvb.41.1564587547424; Wed, 31 Jul 2019 08:39:07 -0700 (PDT) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id f25sm35116803qta.81.2019.07.31.08.39.06 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 31 Jul 2019 08:39:06 -0700 (PDT) From: Pavel Tatashin To: pasha.tatashin@soleen.com, jmorris@namei.org, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-doc@vger.kernel.org, linux-arm-kernel@lists.infradead.org, marc.zyngier@arm.com, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, bhsharma@redhat.com Subject: [RFC v2 6/8] arm64, kexec: add expandable argument to relocation function Date: Wed, 31 Jul 2019 11:38:55 -0400 Message-Id: <20190731153857.4045-7-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190731153857.4045-1-pasha.tatashin@soleen.com> References: <20190731153857.4045-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190731_083908_494611_BF89C52F X-CRM114-Status: GOOD ( 20.36 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Currently, kexec relocation function (arm64_relocate_new_kernel) accepts the following arguments: head: start of array that contains relocation information. entry: entry point for new kernel or purgatory. dtb_mem: first and only argument to entry. The number of arguments cannot be easily expended, because this function is also called from HVC_SOFT_RESTART, which preserves only three arguments. And, also arm64_relocate_new_kernel is written in assembly but called without stack, thus no place to move extra arguments to free registers. Soon, we will need to pass more arguments: once we enable MMU we will need to pass information about page tables. Another benefit of allowing this function to accept more arguments, is that kernel can actually accept up to 4 arguments (x0-x3), however currently only one is used, but if in the future we will need for more (for example, pass information about when previous kernel exited to have a precise measurement in time spent in purgatory), we won't be easilty do that if arm64_relocate_new_kernel can't accept more arguments. So, add a new struct: kern_reloc_arg, and place it in kexec safe page (i.e memory that is not overwritten during relocation). Thus, make arm64_relocate_new_kernel to only take one argument, that contains all the needed information. Signed-off-by: Pavel Tatashin --- arch/arm64/include/asm/kexec.h | 18 ++++++ arch/arm64/kernel/asm-offsets.c | 9 +++ arch/arm64/kernel/cpu-reset.S | 4 +- arch/arm64/kernel/cpu-reset.h | 8 +-- arch/arm64/kernel/machine_kexec.c | 29 +++++++++- arch/arm64/kernel/relocate_kernel.S | 88 ++++++++++------------------- 6 files changed, 87 insertions(+), 69 deletions(-) diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h index d15ca1ca1e83..d5b79d4c7fae 100644 --- a/arch/arm64/include/asm/kexec.h +++ b/arch/arm64/include/asm/kexec.h @@ -90,12 +90,30 @@ static inline void crash_prepare_suspend(void) {} static inline void crash_post_resume(void) {} #endif +/* + * kern_reloc_arg is passed to kernel relocation function as an argument. + * head kimage->head, allows to traverse through relocation segments. + * entry_addr kimage->start, where to jump from relocation function (new + * kernel, or purgatory entry address). + * kern_arg0 first argument to kernel is its dtb address. The other + * arguments are currently unused, and must be set to 0 + */ +struct kern_reloc_arg { + unsigned long head; + unsigned long entry_addr; + unsigned long kern_arg0; + unsigned long kern_arg1; + unsigned long kern_arg2; + unsigned long kern_arg3; +}; + #define ARCH_HAS_KIMAGE_ARCH struct kimage_arch { void *dtb; unsigned long dtb_mem; unsigned long kern_reloc; + unsigned long kern_reloc_arg; }; #ifdef CONFIG_KEXEC_FILE diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index 214685760e1c..900394907fd8 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -23,6 +23,7 @@ #include #include #include +#include int main(void) { @@ -126,6 +127,14 @@ int main(void) #ifdef CONFIG_ARM_SDE_INTERFACE DEFINE(SDEI_EVENT_INTREGS, offsetof(struct sdei_registered_event, interrupted_regs)); DEFINE(SDEI_EVENT_PRIORITY, offsetof(struct sdei_registered_event, priority)); +#endif +#ifdef CONFIG_KEXEC_CORE + DEFINE(KRELOC_HEAD, offsetof(struct kern_reloc_arg, head)); + DEFINE(KRELOC_ENTRY_ADDR, offsetof(struct kern_reloc_arg, entry_addr)); + DEFINE(KRELOC_KERN_ARG0, offsetof(struct kern_reloc_arg, kern_arg0)); + DEFINE(KRELOC_KERN_ARG1, offsetof(struct kern_reloc_arg, kern_arg1)); + DEFINE(KRELOC_KERN_ARG2, offsetof(struct kern_reloc_arg, kern_arg2)); + DEFINE(KRELOC_KERN_ARG3, offsetof(struct kern_reloc_arg, kern_arg3)); #endif return 0; } diff --git a/arch/arm64/kernel/cpu-reset.S b/arch/arm64/kernel/cpu-reset.S index 6ea337d464c4..64c78a42919f 100644 --- a/arch/arm64/kernel/cpu-reset.S +++ b/arch/arm64/kernel/cpu-reset.S @@ -43,9 +43,7 @@ ENTRY(__cpu_soft_restart) hvc #0 // no return 1: mov x18, x1 // entry - mov x0, x2 // arg0 - mov x1, x3 // arg1 - mov x2, x4 // arg2 + mov x0, x2 // arg br x18 ENDPROC(__cpu_soft_restart) diff --git a/arch/arm64/kernel/cpu-reset.h b/arch/arm64/kernel/cpu-reset.h index ed50e9587ad8..7a8720ff186f 100644 --- a/arch/arm64/kernel/cpu-reset.h +++ b/arch/arm64/kernel/cpu-reset.h @@ -11,12 +11,10 @@ #include void __cpu_soft_restart(unsigned long el2_switch, unsigned long entry, - unsigned long arg0, unsigned long arg1, unsigned long arg2); + unsigned long arg); static inline void __noreturn cpu_soft_restart(unsigned long entry, - unsigned long arg0, - unsigned long arg1, - unsigned long arg2) + unsigned long arg) { typeof(__cpu_soft_restart) *restart; @@ -25,7 +23,7 @@ static inline void __noreturn cpu_soft_restart(unsigned long entry, restart = (void *)__pa_symbol(__cpu_soft_restart); cpu_install_idmap(); - restart(el2_switch, entry, arg0, arg1, arg2); + restart(el2_switch, entry, arg); unreachable(); } diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c index 596c9b9657be..d7291a663379 100644 --- a/arch/arm64/kernel/machine_kexec.c +++ b/arch/arm64/kernel/machine_kexec.c @@ -43,6 +43,7 @@ static void _kexec_image_info(const char *func, int line, pr_debug(" head: %lx\n", kimage->head); pr_debug(" nr_segments: %lu\n", kimage->nr_segments); pr_debug(" kern_reloc: %pa\n", &kimage->arch.kern_reloc); + pr_debug(" kern_reloc_arg: %pa\n", &kimage->arch.kern_reloc_arg); for (i = 0; i < kimage->nr_segments; i++) { pr_debug(" segment[%lu]: %016lx - %016lx, 0x%lx bytes, %lu pages\n", @@ -59,14 +60,39 @@ void machine_kexec_cleanup(struct kimage *kimage) /* Empty routine needed to avoid build errors. */ } +/* Allocates pages for kexec page table */ +static void *kexec_page_alloc(void *arg) +{ + struct kimage *kimage = (struct kimage *)arg; + struct page *page = kimage_alloc_control_pages(kimage, 0); + + if (!page) + return NULL; + + return page_address(page); +} + + int machine_kexec_post_load(struct kimage *kimage) { unsigned long kern_reloc; + struct kern_reloc_arg *kern_reloc_arg; kern_reloc = page_to_phys(kimage->control_code_page); memcpy(__va(kern_reloc), arm64_relocate_new_kernel, arm64_relocate_new_kernel_size); + + kern_reloc_arg = kexec_page_alloc(kimage); + if (!kern_reloc_arg) + return -ENOMEM; + memset(kern_reloc_arg, 0, sizeof (struct kern_reloc_arg)); + kimage->arch.kern_reloc = kern_reloc; + kimage->arch.kern_reloc_arg = __pa(kern_reloc_arg); + + kern_reloc_arg->head = kimage->head; + kern_reloc_arg->entry_addr = kimage->start; + kern_reloc_arg->kern_arg0 = kimage->arch.dtb_mem; kexec_image_info(kimage); return 0; @@ -203,8 +229,7 @@ void machine_kexec(struct kimage *kimage) * userspace (kexec-tools). * In kexec_file case, the kernel starts directly without purgatory. */ - cpu_soft_restart(kimage->arch.kern_reloc, kimage->head, kimage->start, - kimage->arch.dtb_mem); + cpu_soft_restart(kimage->arch.kern_reloc, kimage->arch.kern_reloc_arg); BUG(); /* Should never get here. */ } diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S index c1d7db71a726..d352faf7cbe6 100644 --- a/arch/arm64/kernel/relocate_kernel.S +++ b/arch/arm64/kernel/relocate_kernel.S @@ -8,7 +8,7 @@ #include #include - +#include #include #include #include @@ -17,86 +17,58 @@ /* * arm64_relocate_new_kernel - Put a 2nd stage image in place and boot it. * - * The memory that the old kernel occupies may be overwritten when coping the + * The memory that the old kernel occupies may be overwritten when copying the * new image to its final location. To assure that the * arm64_relocate_new_kernel routine which does that copy is not overwritten, * all code and data needed by arm64_relocate_new_kernel must be between the * symbols arm64_relocate_new_kernel and arm64_relocate_new_kernel_end. The * machine_kexec() routine will copy arm64_relocate_new_kernel to the kexec - * control_code_page, a special page which has been set up to be preserved - * during the copy operation. + * safe memory that has been set up to be preserved during the copy operation. */ ENTRY(arm64_relocate_new_kernel) - - /* Setup the list loop variables. */ - mov x18, x2 /* x18 = dtb address */ - mov x17, x1 /* x17 = kimage_start */ - mov x16, x0 /* x16 = kimage_head */ - raw_dcache_line_size x15, x0 /* x15 = dcache line size */ - mov x14, xzr /* x14 = entry ptr */ - mov x13, xzr /* x13 = copy dest */ - /* Clear the sctlr_el2 flags. */ - mrs x0, CurrentEL - cmp x0, #CurrentEL_EL2 + mrs x2, CurrentEL + cmp x2, #CurrentEL_EL2 b.ne 1f - mrs x0, sctlr_el2 + mrs x2, sctlr_el2 ldr x1, =SCTLR_ELx_FLAGS - bic x0, x0, x1 + bic x2, x2, x1 pre_disable_mmu_workaround - msr sctlr_el2, x0 + msr sctlr_el2, x2 isb -1: - - /* Check if the new image needs relocation. */ +1: /* Check if the new image needs relocation. */ + ldr x16, [x0, #KRELOC_HEAD] /* x16 = kimage_head */ tbnz x16, IND_DONE_BIT, .Ldone - + raw_dcache_line_size x15, x1 /* x15 = dcache line size */ .Lloop: and x12, x16, PAGE_MASK /* x12 = addr */ - /* Test the entry flags. */ .Ltest_source: tbz x16, IND_SOURCE_BIT, .Ltest_indirection /* Invalidate dest page to PoC. */ - mov x0, x13 - add x20, x0, #PAGE_SIZE + mov x2, x13 + add x20, x2, #PAGE_SIZE sub x1, x15, #1 - bic x0, x0, x1 -2: dc ivac, x0 - add x0, x0, x15 - cmp x0, x20 + bic x2, x2, x1 +2: dc ivac, x2 + add x2, x2, x15 + cmp x2, x20 b.lo 2b dsb sy - mov x20, x13 - mov x21, x12 - copy_page x20, x21, x0, x1, x2, x3, x4, x5, x6, x7 - - /* dest += PAGE_SIZE */ - add x13, x13, PAGE_SIZE + copy_page x13, x12, x1, x2, x3, x4, x5, x6, x7, x8 b .Lnext - .Ltest_indirection: tbz x16, IND_INDIRECTION_BIT, .Ltest_destination - - /* ptr = addr */ - mov x14, x12 + mov x14, x12 /* ptr = addr */ b .Lnext - .Ltest_destination: tbz x16, IND_DESTINATION_BIT, .Lnext - - /* dest = addr */ - mov x13, x12 - + mov x13, x12 /* dest = addr */ .Lnext: - /* entry = *ptr++ */ - ldr x16, [x14], #8 - - /* while (!(entry & DONE)) */ - tbz x16, IND_DONE_BIT, .Lloop - + ldr x16, [x14], #8 /* entry = *ptr++ */ + tbz x16, IND_DONE_BIT, .Lloop /* while (!(entry & DONE)) */ .Ldone: /* wait for writes from copy_page to finish */ dsb nsh @@ -105,18 +77,16 @@ ENTRY(arm64_relocate_new_kernel) isb /* Start new image. */ - mov x0, x18 - mov x1, xzr - mov x2, xzr - mov x3, xzr - br x17 - -ENDPROC(arm64_relocate_new_kernel) + ldr x4, [x0, #KRELOC_ENTRY_ADDR] /* x4 = kimage_start */ + ldr x3, [x0, #KRELOC_KERN_ARG3] + ldr x2, [x0, #KRELOC_KERN_ARG2] + ldr x1, [x0, #KRELOC_KERN_ARG1] + ldr x0, [x0, #KRELOC_KERN_ARG0] /* x0 = dtb address */ + br x4 +END(arm64_relocate_new_kernel) .ltorg - .align 3 /* To keep the 64-bit values below naturally aligned. */ - .Lcopy_end: .org KEXEC_CONTROL_PAGE_SIZE From patchwork Wed Jul 31 15:38:56 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 11068421 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 06CC0112C for ; Wed, 31 Jul 2019 15:40:58 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E38BA212BE for ; Wed, 31 Jul 2019 15:40:57 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D77752237D; Wed, 31 Jul 2019 15:40:57 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 28D29212E8 for ; Wed, 31 Jul 2019 15:40:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Cc:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=X37nT5LzpRJrCEJC23BvNcQwFGHMmDVm6sRKL1YAbrY=; b=dlgvOXJ56fwk0U jutJYZib5xtJ2MLhcqM+4TGh+LWa9mrX3nQ64+MpzNSnDlxgzy4PQ6Akszf4tpP8+9X3jrmU0caoV E9pZ22Y8K43972+XmLrhC/PHPhxei31Cs8DPpMA9HfyadMMgkH8hOC3c2cn1yHwwO0uNdPZxoOOg6 acY8KxrrjJJ8qdFP/fGu7FUhzI4L1bC/GJs9Wvl0liBu2PJCAAhMh8H1L9WqEl8Y/0bgl5YnwWVWr Fc2VolbAJORpWv2Zen3NE0VCXvZkjvw5U2exiKAEVv7LJSBW3Ddjf6wL/enSIVt/d9JachLxB67jg ZxN37v/kkLzmrPJ5v4WA==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92 #3 (Red Hat Linux)) id 1hsqig-0002R3-PV; Wed, 31 Jul 2019 15:40:50 +0000 Received: from mail-qt1-x841.google.com ([2607:f8b0:4864:20::841]) by bombadil.infradead.org with esmtps (Exim 4.92 #3 (Red Hat Linux)) id 1hsqh3-0007qK-JD for linux-arm-kernel@lists.infradead.org; Wed, 31 Jul 2019 15:39:12 +0000 Received: by mail-qt1-x841.google.com with SMTP id l9so66969265qtu.6 for ; Wed, 31 Jul 2019 08:39:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=2TaYhOiI7KpUUvoumCBN2Lbnal9E4X64rBpNT0SqQZE=; b=i1Elz6C2MClCEZoSwzlpK3Xd+SoeWvaa3h/7LPEK598s+cCkT+uqd6SJfeSDDiZyx8 af7CegDqeGjvoifmnrZrU6ndAa/plTQppagyRJjHdCnYiBUz7SPWoH8PUP5PPK8Cn6r1 bqEx/2VRpQ582wxfdn6ell47OeVd0NWB6mG9rr5x4+9Ksf7BsBRoTuTX0MeapQuLsn1Q RG4VWOLe9L7yEkyRB/kcaDXCDD5H3PAvG+Zxecbt7v2/9zbrB0oa8DKZLX8IQWgojtP1 qiOUhu2zlEBGJoXCPVULBot5rVGAGIOAJmyLvbpyFREH5C2KeMKkZhdIFV8ylzK3tFCD B6Tg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=2TaYhOiI7KpUUvoumCBN2Lbnal9E4X64rBpNT0SqQZE=; b=iuBw+im4sxahfYnxXogX5uqVhnbfD55t9zCIi+GHTUWL3WJ20reMvW+dysJffhXUH6 VVBP/UaJirsOmlpDAMczvRSOKDKTK8heECqZsWch2JUqvEaxwR7HBWv+99hWq5onpYaV Zjg0NHBhzAgZqA6mZQnXtNVI25HkyEY8I2KMxNvZlwX3WNVgHjLWGzwxm6edmTW4e0+0 MkEEvlQGpZ2ks2JW+vXebE4J7dD34mqX1AXuN4yfA69JUdETVtKDDrgQ8M+7zMzW4E4j 1Q7lrCXrU5YlMOPOytCo6qfRybiKjkcDEGjsuan8ALa+zI6OpmUFyyFEbqoKDKUvsdNk HBPQ== X-Gm-Message-State: APjAAAX1gQIuPYaOzz/TpUbRElj7jpqqVPg7nv1qhfW1C4CXYxz0dv40 b2zPAoRLJT5Wf+k913j3nqX/QSrJ X-Google-Smtp-Source: APXvYqyxMmvwl0RXSQBVJwR1Npj7vUzF0+xnqd3FDu11Bt5mJAI7HT5mhKuJmjwDNUagfZCK7qrX4w== X-Received: by 2002:aed:3325:: with SMTP id u34mr84683890qtd.324.1564587548713; Wed, 31 Jul 2019 08:39:08 -0700 (PDT) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id f25sm35116803qta.81.2019.07.31.08.39.07 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 31 Jul 2019 08:39:08 -0700 (PDT) From: Pavel Tatashin To: pasha.tatashin@soleen.com, jmorris@namei.org, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-doc@vger.kernel.org, linux-arm-kernel@lists.infradead.org, marc.zyngier@arm.com, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, bhsharma@redhat.com Subject: [RFC v2 7/8] arm64, kexec: configure transitional page table for kexec Date: Wed, 31 Jul 2019 11:38:56 -0400 Message-Id: <20190731153857.4045-8-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190731153857.4045-1-pasha.tatashin@soleen.com> References: <20190731153857.4045-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190731_083909_770263_8AAB6A4E X-CRM114-Status: GOOD ( 17.65 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Configure a page table located in kexec-safe memory that has the following mappings: 1. mapping for text of relocation function with executable permission. 2. mapping for argument for relocation function. 3. mappings for all source ranges 4. mappings for all destination ranges. 5. mappings for array that contains information about source/destinations. We could make this page table to contain liner addresses, but instead do identity maps (va == pa) for every mapping. This is because the relocation code can be executed at EV2, where ttbr1 might not be available. There is no way to execute relocation code at EV1, because the old world is overwritten and thus no place to trap to to escalator to EV2. Signed-off-by: Pavel Tatashin --- arch/arm64/include/asm/kexec.h | 3 + arch/arm64/kernel/asm-offsets.c | 1 + arch/arm64/kernel/machine_kexec.c | 96 ++++++++++++++++++++++++++++++- 3 files changed, 99 insertions(+), 1 deletion(-) diff --git a/arch/arm64/include/asm/kexec.h b/arch/arm64/include/asm/kexec.h index d5b79d4c7fae..1f226cc76e24 100644 --- a/arch/arm64/include/asm/kexec.h +++ b/arch/arm64/include/asm/kexec.h @@ -97,6 +97,8 @@ static inline void crash_post_resume(void) {} * kernel, or purgatory entry address). * kern_arg0 first argument to kernel is its dtb address. The other * arguments are currently unused, and must be set to 0 + * trans_table: idmap for source and destination pages, as well as for + * relocation text. */ struct kern_reloc_arg { unsigned long head; @@ -105,6 +107,7 @@ struct kern_reloc_arg { unsigned long kern_arg1; unsigned long kern_arg2; unsigned long kern_arg3; + unsigned long trans_table; }; #define ARCH_HAS_KIMAGE_ARCH diff --git a/arch/arm64/kernel/asm-offsets.c b/arch/arm64/kernel/asm-offsets.c index 900394907fd8..002db58b28f3 100644 --- a/arch/arm64/kernel/asm-offsets.c +++ b/arch/arm64/kernel/asm-offsets.c @@ -135,6 +135,7 @@ int main(void) DEFINE(KRELOC_KERN_ARG1, offsetof(struct kern_reloc_arg, kern_arg1)); DEFINE(KRELOC_KERN_ARG2, offsetof(struct kern_reloc_arg, kern_arg2)); DEFINE(KRELOC_KERN_ARG3, offsetof(struct kern_reloc_arg, kern_arg3)); + DEFINE(KRELOC_TRANS_TABLE, offsetof(struct kern_reloc_arg, trans_table)); #endif return 0; } diff --git a/arch/arm64/kernel/machine_kexec.c b/arch/arm64/kernel/machine_kexec.c index d7291a663379..402c8fb48f7e 100644 --- a/arch/arm64/kernel/machine_kexec.c +++ b/arch/arm64/kernel/machine_kexec.c @@ -20,6 +20,7 @@ #include #include #include +#include #include "cpu-reset.h" @@ -72,11 +73,96 @@ static void *kexec_page_alloc(void *arg) return page_address(page); } +/* + * idmap every segment that needs to be relocated. We map pages for + * destination, source, and also array that holds source, and destination + * addresses. + * Ideally, we could linearly map src and dst addresses, so, in relocation + * routine we would need to only do memcpy(dst, src, len), but this is not + * possible, because on armv8.0 EL2 does not have ttbr1, and thus we might + * not have enough linear VA range. So, simply idmap it here, that works + * for both EL1, and EL2. Note: we cannot really do relocation in EL1 and + * later upgrade to EL2 because old world is erased, so there is no where + * to trap. + */ +static int map_segments(struct kimage *kimage, pgd_t *pgdp, + struct trans_table_info *info) +{ + unsigned long *ptr = 0; + unsigned long dest = 0; + unsigned long entry, addr; + int rc; + + for (entry = kimage->head; !(entry & IND_DONE); entry = *ptr++) { + addr = entry & PAGE_MASK; + + switch (entry & IND_FLAGS) { + case IND_DESTINATION: + dest = addr; + break; + case IND_INDIRECTION: + ptr = __va(addr); + rc = trans_table_map_page(info, pgdp, ptr, + addr, PAGE_KERNEL); + if (rc) + return rc; + break; + case IND_SOURCE: + rc = trans_table_map_page(info, pgdp, __va(addr), + addr, PAGE_KERNEL); + if (rc) + return rc; + rc = trans_table_map_page(info, pgdp, __va(dest), + dest, PAGE_KERNEL); + if (rc) + return rc; + dest += PAGE_SIZE; + } + } + return 0; +} + +static int mmu_relocate_setup(struct kimage *kimage, unsigned long kern_reloc, + struct kern_reloc_arg *kern_reloc_arg) +{ + struct trans_table_info info = { + .trans_alloc_page = kexec_page_alloc, + .trans_alloc_arg = kimage, + .trans_flags = 0, + }; + pgd_t *trans_table; + int rc; + + rc = trans_table_create_empty(&info, &trans_table); + if (rc) + return rc; + + rc = map_segments(kimage, trans_table, &info); + if (rc) + return rc; + + /* Map relocation function va == pa */ + rc = trans_table_map_page(&info, trans_table, __va(kern_reloc), + kern_reloc, PAGE_KERNEL_EXEC); + if (rc) + return rc; + + /* Map relocation function argument va == pa */ + rc = trans_table_map_page(&info, trans_table, kern_reloc_arg, + __pa(kern_reloc_arg), PAGE_KERNEL); + if (rc) + return rc; + + kern_reloc_arg->trans_table = __pa(trans_table); + + return 0; +} int machine_kexec_post_load(struct kimage *kimage) { unsigned long kern_reloc; struct kern_reloc_arg *kern_reloc_arg; + int rc = 0; kern_reloc = page_to_phys(kimage->control_code_page); memcpy(__va(kern_reloc), arm64_relocate_new_kernel, @@ -94,8 +180,16 @@ int machine_kexec_post_load(struct kimage *kimage) kern_reloc_arg->entry_addr = kimage->start; kern_reloc_arg->kern_arg0 = kimage->arch.dtb_mem; + /* + * If relocation is not needed, we do not need to enable MMU in + * relocation routine, therefore do not create page tables for + * scenarios such as crash kernel + */ + if (!(kimage->head & IND_DONE)) + rc = mmu_relocate_setup(kimage, kern_reloc, kern_reloc_arg); + kexec_image_info(kimage); - return 0; + return rc; } /** From patchwork Wed Jul 31 15:38:57 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 11068423 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5353D112C for ; Wed, 31 Jul 2019 15:41:08 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3CD92205FC for ; Wed, 31 Jul 2019 15:41:08 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 30E7622B39; Wed, 31 Jul 2019 15:41:08 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 80CDE205FC for ; Wed, 31 Jul 2019 15:41:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Cc:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=5qpRc4nlmMCkl7zU+six0JEm+/520Y+jaFK9UfTMOrg=; b=uZEeusR4XvMt08 KoQm9arueJ1MU2WQ6bwkXa7/lcILPgX2/omVNyhuiD+z5F/V35ecgfv5DZrBflb9xKvcnNOdveJi8 usCmelXBfaIpEw9gVc6tXCU1br/lB0LY7q1gDyr99MlF5fNjNtFSHgcx0Ljpzt7VvNxwcGhGWolgS 168PeGqq8UU/OMygMPX9AbvHzFpgq30+y14ijKBi2f7mTkWLXp27UXwsa8LPVpjAUVmbbOeMWwqAU mBocdr/Xf7bjKkvZb0dpjxglX2ArEIue3t6It4rOWwEE+8p8ggZh9XHdZpwVHTY6991QwRmRCBLfN T3MMrdVlgj19JtjX8Paw==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92 #3 (Red Hat Linux)) id 1hsqiw-0002k8-R4; Wed, 31 Jul 2019 15:41:06 +0000 Received: from mail-qk1-x742.google.com ([2607:f8b0:4864:20::742]) by bombadil.infradead.org with esmtps (Exim 4.92 #3 (Red Hat Linux)) id 1hsqh5-0007s0-Ac for linux-arm-kernel@lists.infradead.org; Wed, 31 Jul 2019 15:39:14 +0000 Received: by mail-qk1-x742.google.com with SMTP id m14so23880862qka.10 for ; Wed, 31 Jul 2019 08:39:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; h=from:to:subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=XYehKYYCRESHDl9g5m8hoNiorXJBXUwWGYZzf/xkBeE=; b=iBD+3oi2nGxdZcxkvKa50mV2lsj6z6x2B6Hz5KjNL2N9LvtXwsP75YXDODriDCfRGa Zav5gYJ8SRdJ2N/7hFq/MVQnkAwOwsE1OgIXhjXNOc8X7icudW4owNOg3v2i44o58U42 8hYwd+m39OKSz88CxJ//EqNRR1+avPGlMP1maR8dr+y/IXFIGPNJpn3MOTIAUEhRzBfK 9hJ8A630GhQ0UQLsndGaACUuca+uAGD0Bml8mH5vWBdb7u+hu0Zlgho+0+i6qa6k0fAi nNiRQ7fcADFmkpBUCqhwHi3wcVxaDvXuNjYxDWa9tjdr7Z1HIVJVjZ4hUkXZwro+Aooa 5qZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=XYehKYYCRESHDl9g5m8hoNiorXJBXUwWGYZzf/xkBeE=; b=YSKyJCRk/sn692yKDV7jc1gn9h3LnqEDIKThO78VkZy5V07Jxiu3MrWF3exEZAAo/F g9UQY8UR0tcX4rrxUc3zhaTGB2GKxLo2LicItbYZg4Q+lapUbYqIquv6YEisssZOjFDE PeRTkwr/jJfm4RAGo2zqUUiDTG1p2uwTXDfQT8WfxOz0TfYjiuU3+UXf91930IQgb3CE R5xXr+MGe0kyh2OJ5eD/gqxqeaZaO3OV2zlojsCxW2P5aaWH3sXEDlsWUxEO7xN9kccs s+qk0L1h8Cukn8HjcI+Pbg+/aUEPfV7Z0okfsyvkEMm1DfwR20ZvGCD6zjRHUQM3tY6s w2hQ== X-Gm-Message-State: APjAAAVyXoy0MAR50OiJwOEvs05qxjkLZSEjwhHXSY1Jx3yKkImH/UIk LZjvuRipxtMklOb80TV3k48= X-Google-Smtp-Source: APXvYqw42qjGXDZoiPLqtufyjBZBkOjr1xdQbWaU20wZzak05OCxDaGez2/WsRBm3vcjtAjblizKLA== X-Received: by 2002:a37:8ac3:: with SMTP id m186mr62365455qkd.476.1564587550167; Wed, 31 Jul 2019 08:39:10 -0700 (PDT) Received: from localhost.localdomain (c-73-69-118-222.hsd1.nh.comcast.net. [73.69.118.222]) by smtp.gmail.com with ESMTPSA id f25sm35116803qta.81.2019.07.31.08.39.08 (version=TLS1_3 cipher=AEAD-AES256-GCM-SHA384 bits=256/256); Wed, 31 Jul 2019 08:39:09 -0700 (PDT) From: Pavel Tatashin To: pasha.tatashin@soleen.com, jmorris@namei.org, sashal@kernel.org, ebiederm@xmission.com, kexec@lists.infradead.org, linux-kernel@vger.kernel.org, corbet@lwn.net, catalin.marinas@arm.com, will@kernel.org, linux-doc@vger.kernel.org, linux-arm-kernel@lists.infradead.org, marc.zyngier@arm.com, james.morse@arm.com, vladimir.murzin@arm.com, matthias.bgg@gmail.com, bhsharma@redhat.com Subject: [RFC v2 8/8] arm64, kexec: enable MMU during kexec relocation Date: Wed, 31 Jul 2019 11:38:57 -0400 Message-Id: <20190731153857.4045-9-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.22.0 In-Reply-To: <20190731153857.4045-1-pasha.tatashin@soleen.com> References: <20190731153857.4045-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190731_083911_454849_E4534082 X-CRM114-Status: GOOD ( 14.66 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Now, that we have transitional page tables configured, temporarily enable MMU to allow faster relocation of segments to final destination. The performance data: for a moderate size kernel + initramfs: 25M the relocation was taking 0.382s, with enabled MMU it now takes 0.019s only or x20 improvement. The time is proportional to the size of relocation, therefore if initramfs is larger, 100M it could take over a second. Signed-off-by: Pavel Tatashin --- arch/arm64/kernel/relocate_kernel.S | 192 ++++++++++++++++++++++------ 1 file changed, 154 insertions(+), 38 deletions(-) diff --git a/arch/arm64/kernel/relocate_kernel.S b/arch/arm64/kernel/relocate_kernel.S index d352faf7cbe6..88fc69adb90d 100644 --- a/arch/arm64/kernel/relocate_kernel.S +++ b/arch/arm64/kernel/relocate_kernel.S @@ -4,6 +4,8 @@ * * Copyright (C) Linaro. * Copyright (C) Huawei Futurewei Technologies. + * Copyright (c) 2019, Microsoft Corporation. + * Pavel Tatashin */ #include @@ -13,6 +15,130 @@ #include #include #include +#include + +/* + * The following code is adoped from "Bare-metal Boot Code for ARMv8-A + * Processors Version 1.0, 5.3.1 Cleaning and invalidating the caches". + * http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.dai0527a + */ +.macro dcache_invalidate tmp0, tmp1, tmp2, tmp3, tmp4, tmp5, tmp6, tmp7, tmp8 + mov \tmp0, #0x0 /* tmp0 = Cache level */ + msr CSSELR_EL1, \tmp0 /* 0x0 for L1, 0x2 for L2 */ + mrs \tmp4, CCSIDR_EL1 /* Read Cache Size ID */ + and \tmp1, \tmp4, #0x7 + add \tmp1, \tmp1, #0x4 /* tmp1 Cache Line Size */ + ldr \tmp3, =0x7fff + and \tmp2, \tmp3, \tmp4, lsr #13 /* tmp2 Cache Set num - 1 */ + ldr \tmp3, =0x3ff + and \tmp3, \tmp3, \tmp4, lsr #3 /* tmp3 Cache Assoc. num - 1 */ + clz \tmp4, \tmp3 /* tmp4 way pos. in the CISW */ + mov \tmp5, #0 /* tmp5 way counter way_loop */ +1: /* way_loop */ + mov \tmp6, #0 /* tmp6 set counter set_loop */ +2: /* set_loop */ + lsl \tmp7, \tmp5, \tmp4 + orr \tmp7, \tmp0, \tmp7 /* Set way */ + lsl \tmp8, \tmp6, \tmp1 + orr \tmp7, \tmp7, \tmp8 /* Set set */ + dc cisw, \tmp7 /* Clean & Inval. cache line */ + add \tmp6, \tmp6, #1 /* Increment set counter */ + cmp \tmp6, \tmp2 /* Last set reached yet? */ + ble 2b /* If not, iterate set_loop, */ + add \tmp5, \tmp5, #1 /* else, next way. */ + cmp \tmp5, \tmp3 /* Last way reached yet? */ + ble 1b /* If not, iterate way_loop. */ +.endm + +/* + * Invalidae all TLB: if we are running at EL2, invalidate all TLB at EL1 & EL2, + * if we are running at EL1 invalidate all current VMID TLB at EL1. + */ +.macro tlb_invalidate tmp + mrs \tmp, CurrentEL + cmp \tmp, #CurrentEL_EL2 + isb + b.ne 1f + dsb sy + tlbi alle2 + tlbi alle1 + dsb ish + isb + b 2f +1: + dsb sy + tlbi vmalle1 + dsb ish + isb +2: +.endm + +.macro turn_off_mmu_el sctlr, tmp1, tmp2 + mrs \tmp1, \sctlr + ldr \tmp2, =SCTLR_ELx_FLAGS + bic \tmp1, \tmp1, \tmp2 + pre_disable_mmu_workaround + msr \sctlr, \tmp1 + isb +.endm + +.macro turn_off_mmu tmp1, tmp2 + turn_off_mmu_el sctlr_el1, \tmp1, \tmp2 /* Turn off MMU at EL1 */ + mrs \tmp1, CurrentEL + cmp \tmp1, #CurrentEL_EL2 + b.ne 1f + turn_off_mmu_el sctlr_el2, \tmp1, \tmp2 /* Turn off MMU at EL2 */ +1: +.endm + +/* Configure TCR_EL2 and MAIR_EL2 */ +.macro tcr_mair_mmu_el2 tmp1, tmp2, tmp3 + mrs \tmp1, tcr_el1 + ldr \tmp2, =TCR_EL2_MASK + and \tmp1, \tmp1, \tmp2 + mov \tmp2, #TCR_EL2_RES1 + orr \tmp1, \tmp1, \tmp2 + ldr \tmp2, =TCR_T0SZ(VA_BITS) + orr \tmp1, \tmp1, \tmp2 + tcr_compute_pa_size \tmp1, #TCR_EL2_PS_SHIFT, \tmp2, \tmp3 + msr tcr_el2, \tmp1 + mrs \tmp1, mair_el1 + msr mair_el2, \tmp1 +.endm + +.macro turn_on_mmu tmp1, tmp2, tmp3 + mrs \tmp1, CurrentEL + cmp \tmp1, #CurrentEL_EL2 + b.ne 1f + tcr_mair_mmu_el2 \tmp1, \tmp2, \tmp3 + ldr \tmp1, =(SCTLR_EL2_RES1 | SCTLR_ELx_FLAGS | ENDIAN_SET_EL2) + msr sctlr_el2, \tmp1 + b 2f +1: mrs \tmp1, sctlr_el1 + ldr \tmp2, =SCTLR_ELx_FLAGS + orr \tmp1, \tmp1, \tmp2 + msr sctlr_el1, \tmp1 +2: ic iallu + dsb nsh + isb +.endm + +.macro set_ttbr_el ttbr_reg, trans_table + phys_to_ttbr \trans_table, \trans_table + msr \ttbr_reg, \trans_table + isb +.endm + +.macro set_ttbr trans_table, tmp + mrs \tmp, CurrentEL + cmp \tmp, #CurrentEL_EL2 + b.ne 1f + set_ttbr_el ttbr0_el2 \trans_table + b 2f +1: + set_ttbr_el ttbr0_el1 \trans_table +2: +.endm /* * arm64_relocate_new_kernel - Put a 2nd stage image in place and boot it. @@ -24,59 +150,49 @@ * symbols arm64_relocate_new_kernel and arm64_relocate_new_kernel_end. The * machine_kexec() routine will copy arm64_relocate_new_kernel to the kexec * safe memory that has been set up to be preserved during the copy operation. + * + * This function temporarily enables MMU if kernel relocation is needed. This is + * done for performance reasons: with MMU-enabled arm64 is much quicker at + * copying pages due to also having enabled caching. */ ENTRY(arm64_relocate_new_kernel) - /* Clear the sctlr_el2 flags. */ - mrs x2, CurrentEL - cmp x2, #CurrentEL_EL2 - b.ne 1f - mrs x2, sctlr_el2 - ldr x1, =SCTLR_ELx_FLAGS - bic x2, x2, x1 - pre_disable_mmu_workaround - msr sctlr_el2, x2 - isb -1: /* Check if the new image needs relocation. */ - ldr x16, [x0, #KRELOC_HEAD] /* x16 = kimage_head */ - tbnz x16, IND_DONE_BIT, .Ldone - raw_dcache_line_size x15, x1 /* x15 = dcache line size */ + /* MMU on EL2 might still be on, turn it off for now */ + turn_off_mmu x1, x2 + dcache_invalidate x1, x2, x3, x4, x5, x6, x7, x8, x9 + tlb_invalidate x1 + + /* Check if the new image needs relocation. */ + ldr x12, [x0, #KRELOC_HEAD] /* x12 = kimage_head */ + tbnz x12, IND_DONE_BIT, .Ldone + ldr x1, [x0, #KRELOC_TRANS_TABLE] + set_ttbr x1, x2 + turn_on_mmu x1, x2, x3 .Lloop: - and x12, x16, PAGE_MASK /* x12 = addr */ + and x2, x12, PAGE_MASK /* x2 = addr */ /* Test the entry flags. */ .Ltest_source: - tbz x16, IND_SOURCE_BIT, .Ltest_indirection - - /* Invalidate dest page to PoC. */ - mov x2, x13 - add x20, x2, #PAGE_SIZE - sub x1, x15, #1 - bic x2, x2, x1 -2: dc ivac, x2 - add x2, x2, x15 - cmp x2, x20 - b.lo 2b - dsb sy - - copy_page x13, x12, x1, x2, x3, x4, x5, x6, x7, x8 + tbz x12, IND_SOURCE_BIT, .Ltest_indirection + copy_page x1, x2, x3, x4, x5, x6, x7, x8, x9, x10 b .Lnext .Ltest_indirection: - tbz x16, IND_INDIRECTION_BIT, .Ltest_destination - mov x14, x12 /* ptr = addr */ + tbz x12, IND_INDIRECTION_BIT, .Ltest_destination + mov x11, x2 /* x11 = ptr */ b .Lnext .Ltest_destination: - tbz x16, IND_DESTINATION_BIT, .Lnext - mov x13, x12 /* dest = addr */ + tbz x12, IND_DESTINATION_BIT, .Lnext + mov x1, x2 /* x1 = dest */ .Lnext: - ldr x16, [x14], #8 /* entry = *ptr++ */ - tbz x16, IND_DONE_BIT, .Lloop /* while (!(entry & DONE)) */ -.Ldone: + ldr x12, [x11], #8 /* x12 = entry = *ptr++ */ + tbz x12, IND_DONE_BIT, .Lloop /* while (!(entry & DONE)) */ /* wait for writes from copy_page to finish */ dsb nsh ic iallu dsb nsh isb - - /* Start new image. */ + turn_off_mmu x1, x2 + dcache_invalidate x1, x2, x3, x4, x5, x6, x7, x8, x9 + tlb_invalidate x1 +.Ldone: /* Start new image. */ ldr x4, [x0, #KRELOC_ENTRY_ADDR] /* x4 = kimage_start */ ldr x3, [x0, #KRELOC_KERN_ARG3] ldr x2, [x0, #KRELOC_KERN_ARG2]