From patchwork Wed Apr 22 14:25:27 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Price X-Patchwork-Id: 11503987 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1A1A814DD for ; Wed, 22 Apr 2020 14:26:02 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E149620787 for ; Wed, 22 Apr 2020 14:26:01 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="g5YTwQKw" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E149620787 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Q1UWg13wZ9e6KCU/XXFDcms3cYde/0lIX+2TbbsKToQ=; b=g5YTwQKwob8Vo4 wQ0poRTdNDgMekbGR4F9jzgdrIsQrgKQob1eE8MC5TuDH/MuwYaRclCrr80OY7jWiA0VnWanu04IS gFZVsUiYbT2XWD+tVTgqs9T+V2rLg6q1AB29DTkXmQTnHS5qacidCWQkX49L81Ad+37hKXqRWt4yx noIbjBOx7y3cUyaki/2Phd5AoiJmZCDFTfMWsWLrhO08qD7f3vW1xFA9HSOLQg7n65lhY2POmor2l YGUXVOx0xpw2xPJq5etTncfSuhAZ7MgM3xQuYQDyOVc9NiHxtX6wew2jk/byaMaJV2287dwHSO7yR x7hjfmjsUbEMDf7x2TYA==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jRGK4-0000ni-2y; Wed, 22 Apr 2020 14:25:56 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jRGJr-0000c6-Ia for linux-arm-kernel@lists.infradead.org; Wed, 22 Apr 2020 14:25:45 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 013421042; Wed, 22 Apr 2020 07:25:43 -0700 (PDT) Received: from e112269-lin.arm.com (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 8755C3F68F; Wed, 22 Apr 2020 07:25:41 -0700 (PDT) From: Steven Price To: linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org Subject: [PATCH 1/4] mm: Add PG_ARCH_2 page flag Date: Wed, 22 Apr 2020 15:25:27 +0100 Message-Id: <20200422142530.32619-2-steven.price@arm.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200422142530.32619-1-steven.price@arm.com> References: <20200422142530.32619-1-steven.price@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200422_072543_696731_6A95A772 X-CRM114-Status: GOOD ( 12.27 ) X-Spam-Score: -2.3 (--) X-Spam-Report: SpamAssassin version 3.4.4 on bombadil.infradead.org summary: Content analysis details: (-2.3 points) pts rule name description ---- ---------------------- -------------------------------------------------- -2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at https://www.dnswl.org/, medium trust [217.140.110.172 listed in list.dnswl.org] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.0 SPF_PASS SPF: sender matches SPF record X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch@vger.kernel.org, Arnd Bergmann , Catalin Marinas , Hugh Dickins , Steven Price , Andrew Morton , Vincenzo Frascino , Will Deacon Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org For arm64 MTE support it is necessary to be able to mark pages that contain user space visible tags that will need to be saved/restored e.g. when swapped out. To support this add a new arch specific flag (PG_ARCH_2) that arch code can opt into using ARCH_USES_PG_ARCH_2. Signed-off-by: Steven Price --- fs/proc/page.c | 3 +++ include/linux/kernel-page-flags.h | 1 + include/linux/page-flags.h | 3 +++ include/trace/events/mmflags.h | 9 ++++++++- mm/Kconfig | 3 +++ tools/vm/page-types.c | 2 ++ 6 files changed, 20 insertions(+), 1 deletion(-) diff --git a/fs/proc/page.c b/fs/proc/page.c index f909243d4a66..1b6cbe0849a8 100644 --- a/fs/proc/page.c +++ b/fs/proc/page.c @@ -217,6 +217,9 @@ u64 stable_page_flags(struct page *page) u |= kpf_copy_bit(k, KPF_PRIVATE_2, PG_private_2); u |= kpf_copy_bit(k, KPF_OWNER_PRIVATE, PG_owner_priv_1); u |= kpf_copy_bit(k, KPF_ARCH, PG_arch_1); +#ifdef CONFIG_ARCH_USES_PG_ARCH_2 + u |= kpf_copy_bit(k, KPF_ARCH_2, PG_arch_2); +#endif return u; }; diff --git a/include/linux/kernel-page-flags.h b/include/linux/kernel-page-flags.h index abd20ef93c98..eee1877a354e 100644 --- a/include/linux/kernel-page-flags.h +++ b/include/linux/kernel-page-flags.h @@ -17,5 +17,6 @@ #define KPF_ARCH 38 #define KPF_UNCACHED 39 #define KPF_SOFTDIRTY 40 +#define KPF_ARCH_2 41 #endif /* LINUX_KERNEL_PAGE_FLAGS_H */ diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index 222f6f7b2bb3..1d4971fe4fee 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -135,6 +135,9 @@ enum pageflags { #if defined(CONFIG_IDLE_PAGE_TRACKING) && defined(CONFIG_64BIT) PG_young, PG_idle, +#endif +#ifdef CONFIG_ARCH_USES_PG_ARCH_2 + PG_arch_2, #endif __NR_PAGEFLAGS, diff --git a/include/trace/events/mmflags.h b/include/trace/events/mmflags.h index 5fb752034386..5d098029a2d8 100644 --- a/include/trace/events/mmflags.h +++ b/include/trace/events/mmflags.h @@ -79,6 +79,12 @@ #define IF_HAVE_PG_IDLE(flag,string) #endif +#ifdef CONFIG_ARCH_USES_PG_ARCH_2 +#define IF_HAVE_PG_ARCH_2(flag,string) ,{1UL << flag, string} +#else +#define IF_HAVE_PG_ARCH_2(flag,string) +#endif + #define __def_pageflag_names \ {1UL << PG_locked, "locked" }, \ {1UL << PG_waiters, "waiters" }, \ @@ -105,7 +111,8 @@ IF_HAVE_PG_MLOCK(PG_mlocked, "mlocked" ) \ IF_HAVE_PG_UNCACHED(PG_uncached, "uncached" ) \ IF_HAVE_PG_HWPOISON(PG_hwpoison, "hwpoison" ) \ IF_HAVE_PG_IDLE(PG_young, "young" ) \ -IF_HAVE_PG_IDLE(PG_idle, "idle" ) +IF_HAVE_PG_IDLE(PG_idle, "idle" ) \ +IF_HAVE_PG_ARCH_2(PG_arch_2, "arch_2" ) #define show_page_flags(flags) \ (flags) ? __print_flags(flags, "|", \ diff --git a/mm/Kconfig b/mm/Kconfig index c1acc34c1c35..60427ccc3cb8 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -867,4 +867,7 @@ config ARCH_HAS_HUGEPD config MAPPING_DIRTY_HELPERS bool +config ARCH_USES_PG_ARCH_2 + bool + endmenu diff --git a/tools/vm/page-types.c b/tools/vm/page-types.c index 58c0eab71bca..0517c744b04e 100644 --- a/tools/vm/page-types.c +++ b/tools/vm/page-types.c @@ -78,6 +78,7 @@ #define KPF_ARCH 38 #define KPF_UNCACHED 39 #define KPF_SOFTDIRTY 40 +#define KPF_ARCH_2 41 /* [48-] take some arbitrary free slots for expanding overloaded flags * not part of kernel API @@ -135,6 +136,7 @@ static const char * const page_flag_names[] = { [KPF_ARCH] = "h:arch", [KPF_UNCACHED] = "c:uncached", [KPF_SOFTDIRTY] = "f:softdirty", + [KPF_ARCH_2] = "H:arch_2", [KPF_READAHEAD] = "I:readahead", [KPF_SLOB_FREE] = "P:slob_free", From patchwork Wed Apr 22 14:25:28 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Price X-Patchwork-Id: 11503991 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 58EF715AB for ; Wed, 22 Apr 2020 14:26:20 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3730B20787 for ; Wed, 22 Apr 2020 14:26:20 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="O2L4qVQv" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3730B20787 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=XgtcwvVSzawul2t7glDVPuoz/OEXzBrPSIue7fpkfXE=; b=O2L4qVQvgEJOF8 LFOPVprUUJAiz97Ys97117eAQNTYOLnOD3tFldMP1sxTqmJZDjrYTCubBj/jSeda/RT3eP/VvyAYt zZnsuadZxDk8Q550NNLfQSq0gyqEb+Soii9Xn7HbhCyDBaL29PBlioTWM3tvwveF5vcRd4M6PO1WX i6MJknjgsVjDkoBwD3f1mFUOMVQc/UbFW9oeT7oW6ZkeEZ5DO5cj18S2OXLBZc3szaFnFl8yk2Tlo BTYif8qKpDaL0BmYz5YCfKetLZOvwVDbPh0oxJ5yfpRYgv4MU3rQFyHv0CbvCruvXOS7ROw7KIP3V P5tDaYYHRspl+jLjsQwA==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jRGKN-00016C-Np; Wed, 22 Apr 2020 14:26:15 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jRGJt-0000cg-4P for linux-arm-kernel@lists.infradead.org; Wed, 22 Apr 2020 14:25:46 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id A482F1045; Wed, 22 Apr 2020 07:25:44 -0700 (PDT) Received: from e112269-lin.arm.com (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 3944B3F68F; Wed, 22 Apr 2020 07:25:43 -0700 (PDT) From: Steven Price To: linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org Subject: [PATCH 2/4] mm: Add arch hooks for saving/restoring tags Date: Wed, 22 Apr 2020 15:25:28 +0100 Message-Id: <20200422142530.32619-3-steven.price@arm.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200422142530.32619-1-steven.price@arm.com> References: <20200422142530.32619-1-steven.price@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200422_072545_261442_C2F23916 X-CRM114-Status: GOOD ( 13.71 ) X-Spam-Score: -2.3 (--) X-Spam-Report: SpamAssassin version 3.4.4 on bombadil.infradead.org summary: Content analysis details: (-2.3 points) pts rule name description ---- ---------------------- -------------------------------------------------- -2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at https://www.dnswl.org/, medium trust [217.140.110.172 listed in list.dnswl.org] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.0 SPF_PASS SPF: sender matches SPF record X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch@vger.kernel.org, Arnd Bergmann , Catalin Marinas , Hugh Dickins , Steven Price , Andrew Morton , Vincenzo Frascino , Will Deacon Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Arm's Memory Tagging Extension (MTE) adds some metadata (taga) to every physical page, when swapping pages out to disk it is necessary to save these tags, and later restore them when reading the pages back. Add some hooks along with dummy implementations to enable the arch code to handle this. Three new hooks are added to the swap code: * arch_prepare_to_swap() and * arch_swap_invalidate_page() / arch_swap_invalidate_area(). One new hook is added to shmem: * arch_swap_restore_tags() Signed-off-by: Steven Price --- include/asm-generic/pgtable.h | 23 +++++++++++++++++++++++ mm/page_io.c | 6 ++++++ mm/shmem.c | 6 ++++++ mm/swapfile.c | 2 ++ 4 files changed, 37 insertions(+) diff --git a/include/asm-generic/pgtable.h b/include/asm-generic/pgtable.h index 329b8c8ca703..306cee75b9ec 100644 --- a/include/asm-generic/pgtable.h +++ b/include/asm-generic/pgtable.h @@ -475,6 +475,29 @@ static inline int arch_unmap_one(struct mm_struct *mm, } #endif +#ifndef __HAVE_ARCH_PREPARE_TO_SWAP +static inline int arch_prepare_to_swap(struct page *page) +{ + return 0; +} +#endif + +#ifndef __HAVE_ARCH_SWAP_INVALIDATE +static inline void arch_swap_invalidate_page(int type, pgoff_t offset) +{ +} + +static inline void arch_swap_invalidate_area(int type) +{ +} +#endif + +#ifndef __HAVE_ARCH_SWAP_RESTORE_TAGS +static inline void arch_swap_restore_tags(swp_entry_t entry, struct page *page) +{ +} +#endif + #ifndef __HAVE_ARCH_PGD_OFFSET_GATE #define pgd_offset_gate(mm, addr) pgd_offset(mm, addr) #endif diff --git a/mm/page_io.c b/mm/page_io.c index 76965be1d40e..7baee316ac99 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -253,6 +253,12 @@ int swap_writepage(struct page *page, struct writeback_control *wbc) unlock_page(page); goto out; } + /* Arch code may have to preserve more data + * than just the page contents, e.g. memory tags + */ + ret = arch_prepare_to_swap(page); + if (ret) + goto out; if (frontswap_store(page) == 0) { set_page_writeback(page); unlock_page(page); diff --git a/mm/shmem.c b/mm/shmem.c index 73754ed7af69..1010b91f267e 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1658,6 +1658,12 @@ static int shmem_swapin_page(struct inode *inode, pgoff_t index, } wait_on_page_writeback(page); + /* + * Some architectures may have to restore extra metadata to the + * physical page after reading from swap + */ + arch_swap_restore_tags(swap, page); + if (shmem_should_replace_page(page, gfp)) { error = shmem_replace_page(&page, gfp, info, index); if (error) diff --git a/mm/swapfile.c b/mm/swapfile.c index 5871a2aa86a5..b39c6520b0cf 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -722,6 +722,7 @@ static void swap_range_free(struct swap_info_struct *si, unsigned long offset, else swap_slot_free_notify = NULL; while (offset <= end) { + arch_swap_invalidate_page(si->type, offset); frontswap_invalidate_page(si->type, offset); if (swap_slot_free_notify) swap_slot_free_notify(si->bdev, offset); @@ -2645,6 +2646,7 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile) frontswap_map = frontswap_map_get(p); spin_unlock(&p->lock); spin_unlock(&swap_lock); + arch_swap_invalidate_area(p->type); frontswap_invalidate_area(p->type); frontswap_map_set(p, NULL); mutex_unlock(&swapon_mutex); From patchwork Wed Apr 22 14:25:29 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Price X-Patchwork-Id: 11504003 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8B6A514DD for ; Wed, 22 Apr 2020 14:26:40 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 57CC12084D for ; Wed, 22 Apr 2020 14:26:40 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="t2jSOi8Z" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 57CC12084D Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=xXXCicR6KAhB4aU+/3Px4BwXGpbuhp9bSZS4rZ9pndE=; b=t2jSOi8ZRbvPIA 9vQ04Dn8/yYvd2br9BRwPOppFwll2lU7aA839iyGPFFoZVCEKkCHOHar1Sfs5dRVFqRg3CvwVruYT aIIAxczJfhUWDRXbLGbRS3ts/Ep6LIFqahd2q+A41THhAPtM3FKlpOmWfIHOZZgNg/NA/aF5ac0t+ 66Jcq1qFm8fHqvvu8AQ9OEyRJVUEogPxyl4RFI4O6sUJGpgoeKLV2HB3hxsn2eoOB0u5pWdL8Lli2 u/YY8Ejl+LGcD0n24/EMIwTJycZg+OHLKspHTcPlfDlTJmFDI3AieQRjn9hp+joBftwuSFDTPrQxU vb4a8PlBkRRo1S30HImQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jRGKh-0001Mo-0S; Wed, 22 Apr 2020 14:26:35 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jRGJv-0000cg-3r for linux-arm-kernel@lists.infradead.org; Wed, 22 Apr 2020 14:25:49 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 559B430E; Wed, 22 Apr 2020 07:25:46 -0700 (PDT) Received: from e112269-lin.arm.com (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id DC69C3F68F; Wed, 22 Apr 2020 07:25:44 -0700 (PDT) From: Steven Price To: linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org Subject: [PATCH 3/4] arm64: mte: Enable swap of tagged pages Date: Wed, 22 Apr 2020 15:25:29 +0100 Message-Id: <20200422142530.32619-4-steven.price@arm.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200422142530.32619-1-steven.price@arm.com> References: <20200422142530.32619-1-steven.price@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200422_072547_268912_4DF43BDC X-CRM114-Status: GOOD ( 18.58 ) X-Spam-Score: -2.3 (--) X-Spam-Report: SpamAssassin version 3.4.4 on bombadil.infradead.org summary: Content analysis details: (-2.3 points) pts rule name description ---- ---------------------- -------------------------------------------------- -2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at https://www.dnswl.org/, medium trust [217.140.110.172 listed in list.dnswl.org] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.0 SPF_PASS SPF: sender matches SPF record X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch@vger.kernel.org, Arnd Bergmann , Catalin Marinas , Hugh Dickins , Steven Price , Andrew Morton , Vincenzo Frascino , Will Deacon Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org When swapping pages out to disk it is necessary to save any tags that have been set, and restore when swapping back in. To do this pages that mapped so user space can access tags are marked with a new page flag (PG_ARCH_2, locally named PG_mte_tagged). When swapping out these pages the tags are stored in memory and later restored when the pages are brought back in. Because shmem can swap pages back in without restoring the userspace PTE it is also necessary to add a hook for shmem. Signed-off-by: Steven Price --- arch/arm64/Kconfig | 2 +- arch/arm64/include/asm/mte.h | 6 ++ arch/arm64/include/asm/pgtable.h | 44 ++++++++++++++ arch/arm64/lib/mte.S | 50 ++++++++++++++++ arch/arm64/mm/Makefile | 2 +- arch/arm64/mm/mteswap.c | 98 ++++++++++++++++++++++++++++++++ 6 files changed, 200 insertions(+), 2 deletions(-) create mode 100644 arch/arm64/mm/mteswap.c diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index af2e6e5dae1b..697d5c6b1d53 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -1615,7 +1615,7 @@ config ARM64_MTE bool "Memory Tagging Extension support" depends on ARM64_AS_HAS_MTE && ARM64_TAGGED_ADDR_ABI select ARCH_USES_HIGH_VMA_FLAGS - select ARCH_NO_SWAP + select ARCH_USES_PG_ARCH_2 help Memory Tagging (part of the ARMv8.5 Extensions) provides architectural support for run-time, always-on detection of diff --git a/arch/arm64/include/asm/mte.h b/arch/arm64/include/asm/mte.h index 0ca2aaff07a1..28bb32b270ee 100644 --- a/arch/arm64/include/asm/mte.h +++ b/arch/arm64/include/asm/mte.h @@ -17,6 +17,10 @@ unsigned long mte_copy_tags_from_user(void *to, const void __user *from, unsigned long n); unsigned long mte_copy_tags_to_user(void __user *to, void *from, unsigned long n); +unsigned long mte_save_page_tags(const void *page_addr, void *tag_storage); +void mte_restore_page_tags(void *page_addr, const void *tag_storage); + +#define PG_mte_tagged PG_arch_2 #ifdef CONFIG_ARM64_MTE void flush_mte_state(void); @@ -26,6 +30,8 @@ long set_mte_ctrl(unsigned long arg); long get_mte_ctrl(void); int mte_ptrace_copy_tags(struct task_struct *child, long request, unsigned long addr, unsigned long data); +void *mte_allocate_tag_storage(void); +void mte_free_tag_storage(char *storage); #else static inline void flush_mte_state(void) { diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 39a372bf8afc..a4ad1b75a1a7 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -80,6 +80,8 @@ extern unsigned long empty_zero_page[PAGE_SIZE / sizeof(unsigned long)]; #define pte_user_exec(pte) (!(pte_val(pte) & PTE_UXN)) #define pte_cont(pte) (!!(pte_val(pte) & PTE_CONT)) #define pte_devmap(pte) (!!(pte_val(pte) & PTE_DEVMAP)) +#define pte_tagged(pte) (!!((pte_val(pte) & PTE_ATTRINDX_MASK) == \ + PTE_ATTRINDX(MT_NORMAL_TAGGED))) #define pte_cont_addr_end(addr, end) \ ({ unsigned long __boundary = ((addr) + CONT_PTE_SIZE) & CONT_PTE_MASK; \ @@ -268,12 +270,17 @@ static inline void __check_racy_pte_update(struct mm_struct *mm, pte_t *ptep, __func__, pte_val(old_pte), pte_val(pte)); } +void mte_sync_tags(pte_t *ptep, pte_t pte); + static inline void set_pte_at(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte) { if (pte_present(pte) && pte_user_exec(pte) && !pte_special(pte)) __sync_icache_dcache(pte); + if (system_supports_mte() && pte_tagged(pte)) + mte_sync_tags(ptep, pte); + __check_racy_pte_update(mm, ptep, pte); set_pte(ptep, pte); @@ -845,6 +852,43 @@ static inline pmd_t pmdp_establish(struct vm_area_struct *vma, extern int kern_addr_valid(unsigned long addr); +#ifdef CONFIG_ARM64_MTE + +#define __HAVE_ARCH_PREPARE_TO_SWAP +int mte_save_tags(struct page *page); +static inline int arch_prepare_to_swap(struct page *page) +{ + if (system_supports_mte()) + return mte_save_tags(page); + return 0; +} + +#define __HAVE_ARCH_SWAP_INVALIDATE +void mte_invalidate_tags(int type, pgoff_t offset); +void mte_invalidate_tags_area(int type); + +static inline void arch_swap_invalidate_page(int type, pgoff_t offset) +{ + if (system_supports_mte()) + mte_invalidate_tags(type, offset); +} + +static inline void arch_swap_invalidate_area(int type) +{ + if (system_supports_mte()) + mte_invalidate_tags_area(type); +} + +#define __HAVE_ARCH_SWAP_RESTORE_TAGS +void mte_restore_tags(swp_entry_t entry, struct page *page); +static inline void arch_swap_restore_tags(swp_entry_t entry, struct page *page) +{ + if (system_supports_mte()) + mte_restore_tags(entry, page); +} + +#endif /* CONFIG_ARM64_MTE */ + #include /* diff --git a/arch/arm64/lib/mte.S b/arch/arm64/lib/mte.S index 45be04a8c73c..df8800dfe891 100644 --- a/arch/arm64/lib/mte.S +++ b/arch/arm64/lib/mte.S @@ -94,3 +94,53 @@ USER(2f, sttrb w4, [x0]) 2: sub x0, x0, x3 // update the number of tags copied ret SYM_FUNC_END(mte_copy_tags_from_user) + +/* + * Save the tags in a page + * x0 - page address + * x1 - tag storage + * + * Returns 0 if all tags are 0, otherwise non-zero + */ +SYM_FUNC_START(mte_save_page_tags) + multitag_transfer_size x7, x5 + mov x3, #0 +1: + mov x2, #0 +2: + ldgm x5, [x0] + orr x2, x2, x5 + add x0, x0, x7 + tst x0, #0xFF // 16 tag values fit in a register, + b.ne 2b // which is 16*4=256 bytes + + str x2, [x1], #8 + + orr x3, x3, x2 // OR together all the tag values + tst x0, #(PAGE_SIZE - 1) + b.ne 1b + + mov x0, x3 + ret +SYM_FUNC_END(mte_save_page_tags) + +/* + * Restore the tags in a page + * x0 - page address + * x1 - tag storage + */ +SYM_FUNC_START(mte_restore_page_tags) + multitag_transfer_size x7, x5 +1: + ldr x2, [x1], #8 +2: + stgm x2, [x0] + add x0, x0, x7 + tst x0, #0xFF + b.ne 2b + + tst x0, #(PAGE_SIZE - 1) + b.ne 1b + + ret +SYM_FUNC_END(mte_restore_page_tags) diff --git a/arch/arm64/mm/Makefile b/arch/arm64/mm/Makefile index e93d696295d0..cd7cb19fc224 100644 --- a/arch/arm64/mm/Makefile +++ b/arch/arm64/mm/Makefile @@ -8,7 +8,7 @@ obj-$(CONFIG_PTDUMP_CORE) += dump.o obj-$(CONFIG_PTDUMP_DEBUGFS) += ptdump_debugfs.o obj-$(CONFIG_NUMA) += numa.o obj-$(CONFIG_DEBUG_VIRTUAL) += physaddr.o -obj-$(CONFIG_ARM64_MTE) += cmppages.o +obj-$(CONFIG_ARM64_MTE) += cmppages.o mteswap.o KASAN_SANITIZE_physaddr.o += n obj-$(CONFIG_KASAN) += kasan_init.o diff --git a/arch/arm64/mm/mteswap.c b/arch/arm64/mm/mteswap.c new file mode 100644 index 000000000000..3f8ab5a6d33b --- /dev/null +++ b/arch/arm64/mm/mteswap.c @@ -0,0 +1,98 @@ +// SPDX-License-Identifier: GPL-2.0-only + +#include +#include +#include +#include +#include + +static DEFINE_XARRAY(mte_pages); + +void *mte_allocate_tag_storage(void) +{ + /* tags granule is 16 bytes, 2 tags stored per byte */ + return kmalloc(PAGE_SIZE / 16 / 2, GFP_KERNEL); +} + +void mte_free_tag_storage(char *storage) +{ + kfree(storage); +} + +int mte_save_tags(struct page *page) +{ + void *tag_storage, *ret; + + if (!test_bit(PG_mte_tagged, &page->flags)) + return 0; + + tag_storage = mte_allocate_tag_storage(); + if (!tag_storage) + return -ENOMEM; + + mte_save_page_tags(page_address(page), tag_storage); + + ret = xa_store(&mte_pages, page_private(page), tag_storage, GFP_KERNEL); + if (WARN(xa_is_err(ret), "Failed to store MTE tags")) { + mte_free_tag_storage(tag_storage); + return xa_err(ret); + } else if (ret) { + mte_free_tag_storage(ret); + } + + return 0; +} + +void mte_restore_tags(swp_entry_t entry, struct page *page) +{ + void *tags = xa_load(&mte_pages, entry.val); + + if (!tags) + return; + + mte_restore_page_tags(page_address(page), tags); + + set_bit(PG_mte_tagged, &page->flags); +} + +void mte_invalidate_tags(int type, pgoff_t offset) +{ + swp_entry_t entry = swp_entry(type, offset); + void *tags = xa_erase(&mte_pages, entry.val); + + mte_free_tag_storage(tags); +} + +void mte_invalidate_tags_area(int type) +{ + swp_entry_t entry = swp_entry(type, 0); + swp_entry_t last_entry = swp_entry(type + 1, 0); + void *tags; + + XA_STATE(xa_state, &mte_pages, entry.val); + + xa_lock(&mte_pages); + xas_for_each(&xa_state, tags, last_entry.val - 1) { + __xa_erase(&mte_pages, xa_state.xa_index); + mte_free_tag_storage(tags); + } + xa_unlock(&mte_pages); +} + +void mte_sync_tags(pte_t *ptep, pte_t pte) +{ + struct page *page = pte_page(pte); + pte_t old_pte = READ_ONCE(*ptep); + swp_entry_t entry; + + set_bit(PG_mte_tagged, &page->flags); + + if (!is_swap_pte(old_pte)) + return; + + entry = pte_to_swp_entry(old_pte); + if (non_swap_entry(entry)) + return; + + mte_restore_tags(entry, page); +} From patchwork Wed Apr 22 14:25:30 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Steven Price X-Patchwork-Id: 11504007 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 78B5515AB for ; Wed, 22 Apr 2020 14:26:53 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 56A7D2082E for ; Wed, 22 Apr 2020 14:26:53 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="hENhUXrt" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 56A7D2082E Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=LvxLdN8if83wpr6nn9TntW32HItFWw/OLmljc4r7bCY=; b=hENhUXrtDpXonv U8fVLX+RLhZbO6Sqv+VS2QsuNoZXb+n+imm+T5QfA7sbPUmVIELy3QsmgMp577594CFRUtSmdBSW7 vXe7tOO6P6tgWPSm7oLcYR+mgn7JjYFdl6i2LJTknamlb7Dqo0Cz/Y8IWF9h1WwvG3Jjv0s4e4HCP hBIVWMKugcPRs/LshNHsVhdjNEyfXx5AC06wrOTCQe8CNr72D5pEoU/movGYublHnraGqNaMBinkp ZzzOfkLYMQoqsoLInAYvxWFLbpDWkXXH3KWXapYGr6X01c5mHbmkFlBF/AJ5MId2txag6R18DnM3m Mq7oQuIy/2Sqr+0+QhPQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jRGKr-0001aU-Ro; Wed, 22 Apr 2020 14:26:45 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jRGJw-0000fy-Cm for linux-arm-kernel@lists.infradead.org; Wed, 22 Apr 2020 14:25:50 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 036B91042; Wed, 22 Apr 2020 07:25:48 -0700 (PDT) Received: from e112269-lin.arm.com (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 8D26A3F68F; Wed, 22 Apr 2020 07:25:46 -0700 (PDT) From: Steven Price To: linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org Subject: [PATCH 4/4] arm64: mte: Save tags when hibernating Date: Wed, 22 Apr 2020 15:25:30 +0100 Message-Id: <20200422142530.32619-5-steven.price@arm.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200422142530.32619-1-steven.price@arm.com> References: <20200422142530.32619-1-steven.price@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200422_072548_562541_C1ED4E1E X-CRM114-Status: GOOD ( 14.60 ) X-Spam-Score: -2.3 (--) X-Spam-Report: SpamAssassin version 3.4.4 on bombadil.infradead.org summary: Content analysis details: (-2.3 points) pts rule name description ---- ---------------------- -------------------------------------------------- -2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at https://www.dnswl.org/, medium trust [217.140.110.172 listed in list.dnswl.org] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.0 SPF_PASS SPF: sender matches SPF record X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch@vger.kernel.org, Arnd Bergmann , Catalin Marinas , Hugh Dickins , Steven Price , Andrew Morton , Vincenzo Frascino , Will Deacon Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org When hibernating the contents of all pages in the system are written to disk, however the MTE tags are not visible to the generic hibernation code. So just before the hibernation image is created copy the tags out of the physical tag storage into standard memory so they will be included in the hibernation image. After hibernation apply the tags back into the physical tag storage. Signed-off-by: Steven Price --- arch/arm64/kernel/hibernate.c | 116 ++++++++++++++++++++++++++++++++++ 1 file changed, 116 insertions(+) diff --git a/arch/arm64/kernel/hibernate.c b/arch/arm64/kernel/hibernate.c index 5b73e92c99e3..b89429778b5d 100644 --- a/arch/arm64/kernel/hibernate.c +++ b/arch/arm64/kernel/hibernate.c @@ -31,6 +31,7 @@ #include #include #include +#include #include #include #include @@ -277,6 +278,115 @@ static int create_safe_exec_page(void *src_start, size_t length, #define dcache_clean_range(start, end) __flush_dcache_area(start, (end - start)) +#ifdef CONFIG_ARM64_MTE + +static DEFINE_XARRAY(mte_pages); + +static int save_tags(struct page *page, unsigned long pfn) +{ + void *tag_storage, *ret; + + tag_storage = mte_allocate_tag_storage(); + if (!tag_storage) + return -ENOMEM; + + mte_save_page_tags(page_address(page), tag_storage); + + ret = xa_store(&mte_pages, pfn, tag_storage, GFP_KERNEL); + if (WARN(xa_is_err(ret), "Failed to store MTE tags")) { + mte_free_tag_storage(tag_storage); + return xa_err(ret); + } else if (WARN(ret, "swsusp: %s: Duplicate entry", __func__)) { + mte_free_tag_storage(ret); + } + + return 0; +} + +static void swsusp_mte_free_storage(void) +{ + XA_STATE(xa_state, &mte_pages, 0); + void *tags; + + xa_lock(&mte_pages); + xas_for_each(&xa_state, tags, ULONG_MAX) { + mte_free_tag_storage(tags); + } + xa_unlock(&mte_pages); + + xa_destroy(&mte_pages); +} + +static int swsusp_mte_save_tags(void) +{ + struct zone *zone; + unsigned long pfn, max_zone_pfn; + int ret = 0; + int n = 0; + + if (!system_supports_mte()) + return 0; + + for_each_populated_zone(zone) { + max_zone_pfn = zone_end_pfn(zone); + for (pfn = zone->zone_start_pfn; pfn < max_zone_pfn; pfn++) { + struct page *page = pfn_to_online_page(pfn); + + if (!page) + continue; + + if (!test_bit(PG_mte_tagged, &page->flags)) + continue; + + ret = save_tags(page, pfn); + n++; + + if (ret) { + swsusp_mte_free_storage(); + goto out; + } + } + } + + pr_info("Saved %d MTE pages\n", n); + +out: + return ret; +} + +static void swsusp_mte_restore_tags(void) +{ + XA_STATE(xa_state, &mte_pages, 0); + int n = 0; + void *tags; + + xa_lock(&mte_pages); + xas_for_each(&xa_state, tags, ULONG_MAX) { + unsigned long pfn = xa_state.xa_index; + struct page *page = pfn_to_online_page(pfn); + + mte_restore_page_tags(page_address(page), tags); + + mte_free_tag_storage(tags); + n++; + } + xa_unlock(&mte_pages); + + pr_info("Restored %d MTE pages\n", n); + + xa_destroy(&mte_pages); +} +#else +static int swsusp_mte_save_tags(void) +{ + return 0; +} + +static void swsusp_mte_restore_tags(void) +{ +} +#endif + int swsusp_arch_suspend(void) { int ret = 0; @@ -294,6 +404,10 @@ int swsusp_arch_suspend(void) /* make the crash dump kernel image visible/saveable */ crash_prepare_suspend(); + ret = swsusp_mte_save_tags(); + if (ret) + return ret; + sleep_cpu = smp_processor_id(); ret = swsusp_save(); } else { @@ -307,6 +421,8 @@ int swsusp_arch_suspend(void) dcache_clean_range(__hyp_text_start, __hyp_text_end); } + swsusp_mte_restore_tags(); + /* make the crash dump kernel image protected again */ crash_post_resume();