From patchwork Tue Feb 8 08:18:51 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 12738393 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5D4C9C433F5 for ; Tue, 8 Feb 2022 09:31:29 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:References: Mime-Version:Message-Id:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=F71Pn9wk/+9ZpdesvHezGS0izBcdNeDb9skWo+oL8/0=; b=jTzPxUtPByiGeqaybzWfjmZnZM WZa6fTJG1sOzH8Yez7k1RDMlsNvgqz6MhUQloD6nbXtMVu3Tflehg0uVQDZBdUTV3I1iTbvXbAFbL Dh6Fh1kdHEQeQn4MyTLA+5NyxttNxz1RRYFldWe7wG0Q7TFvvo6RE8/X7WvWvM2KEkLdqbl83xB22 4Xwb9EI6IbHh4HAOYdOZkV/GnMdkauZhTCh/kx2g5HXBCteKYXRUaBAqDmW5lvArDtOua2EBw+v8S RGE4Dn+n1QtlVBVKXQ1gmnTsCkf4MJULdhBeedzCMH1RvAullo3JkE5Pi9jGl+624epigXnyzgPBX M5USS6sg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nHMpK-00DEgC-Rn; Tue, 08 Feb 2022 09:30:23 +0000 Received: from mail-il1-x149.google.com ([2607:f8b0:4864:20::149]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nHLio-00CtrJ-0L for linux-arm-kernel@lists.infradead.org; Tue, 08 Feb 2022 08:19:35 +0000 Received: by mail-il1-x149.google.com with SMTP id y3-20020a920903000000b002be462612d7so2566749ilg.10 for ; Tue, 08 Feb 2022 00:19:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc:content-transfer-encoding; bh=7qxSpd3xPX+C4yLR1dX/IFNkPZKcmGSNIBx8cClYDYU=; b=XdHFeaYJazRuhnhm8ZTE0z6kV3Vg2D7yOo9RvPCbURUwLrCWauzEIpSc8/2g25BIrC evOKvaKC3ItglVgQLRRj+71Onpkq4YqoQPQlaYkJFEGSNMwESMNIsmqvMcuvONtG+6Qb VVVwB4nH31EcjmbpRGyN1vVaXRF+kDjinCTdzPY9xcQ9AunmrG/UViWo0h4s1mLGpuuw EAaiO3hVdv6BSKZU1oXVYCcz7sw9nFo7DNiU/sQlsuWjLwQt0m4CB8wgyQn6tpwRDHlx IcKDt9ipu8tzeBQl9QbCO7108dJWGw0dQfqHSLo/+el2iwJNyIZtX1gUACJz7GMkp2yw jLTg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc:content-transfer-encoding; bh=7qxSpd3xPX+C4yLR1dX/IFNkPZKcmGSNIBx8cClYDYU=; b=UyUgt3BOK2oLMlz7u5+Swb5+lpz1ML8N2LVUfrVykPFkwTwoNiShgUtRHhXRvgkQTm qHmiItIA4VaRoObh8Cbxennf7OPVyutubnlyOPZgDAmmENXYRM7gpTMHk/IqFewN0aB3 hYWd9NWQuTVjFXuR7pSRivWBIg9oTbt+F5UJJzXMji1LASoc4PMtxYZW9/WK7FNYEx/9 25lKudrnLMjS534KGh8Zn3fpIKLAGDovD6TC2WMKbVTcs8drRTcwxg6DGKtuyx8Ov1BA Au8ABWGnjb7z/Y/YQ16O8yQjsDVhSV831iJxl8YOsOYXSSsAA11AuEHub9Ma4tNowzPV vkpQ== X-Gm-Message-State: AOAM5325q8hQUnfxs8EoAbs5B5rYn95Snx2IXP6+DFQLbzTT05YEYaew nD4WRyFfmrQvxcNdaLn+cThqbsO+DFA= X-Google-Smtp-Source: ABdhPJzbQWEGlbwheroT5ZESJTa7PbZbsGYE8OYoBn//TRQbpRJUpKvYqj4cX5AWQ5q+o4KtvtatSbH1LBY= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:5f31:19c3:21f5:7300]) (user=yuzhao job=sendgmr) by 2002:a92:1e01:: with SMTP id e1mr394250ile.291.1644308368038; Tue, 08 Feb 2022 00:19:28 -0800 (PST) Date: Tue, 8 Feb 2022 01:18:51 -0700 In-Reply-To: <20220208081902.3550911-1-yuzhao@google.com> Message-Id: <20220208081902.3550911-2-yuzhao@google.com> Mime-Version: 1.0 References: <20220208081902.3550911-1-yuzhao@google.com> X-Mailer: git-send-email 2.35.0.263.gb82422642f-goog Subject: [PATCH v7 01/12] mm: x86, arm64: add arch_has_hw_pte_young() From: Yu Zhao To: Andrew Morton , Johannes Weiner , Mel Gorman , Michal Hocko Cc: Andi Kleen , Aneesh Kumar , Barry Song <21cnbao@gmail.com>, Catalin Marinas , Dave Hansen , Hillf Danton , Jens Axboe , Jesse Barnes , Jonathan Corbet , Linus Torvalds , Matthew Wilcox , Michael Larabel , Mike Rapoport , Rik van Riel , Vlastimil Babka , Will Deacon , Ying Huang , linux-arm-kernel@lists.infradead.org, linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, page-reclaim@google.com, x86@kernel.org, Yu Zhao , Brian Geffon , Jan Alexander Steffens , Oleksandr Natalenko , Steven Barrett , Suleiman Souhlal , Daniel Byrne , Donald Carr , " =?utf-8?q?Holger_Hoffst=C3=A4tte?= " , Konstantin Kharlamov , Shuang Zhai , Sofia Trinh X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220208_001934_073994_9737AB1C X-CRM114-Status: GOOD ( 18.06 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Some architectures automatically set the accessed bit in PTEs, e.g., x86 and arm64 v8.2. On architectures that don't have this capability, clearing the accessed bit in a PTE usually triggers a page fault following the TLB miss of this PTE (to emulate the accessed bit). Being aware of this capability can help make better decisions, e.g., whether to spread the work out over a period of time to reduce bursty page faults when trying to clear the accessed bit in many PTEs. Note that theoretically this capability can be unreliable, e.g., hotplugged CPUs might be different from builtin ones. Therefore it shouldn't be used in architecture-independent code that involves correctness, e.g., to determine whether TLB flushes are required (in combination with the accessed bit). Signed-off-by: Yu Zhao Acked-by: Brian Geffon Acked-by: Jan Alexander Steffens (heftig) Acked-by: Oleksandr Natalenko Acked-by: Steven Barrett Acked-by: Suleiman Souhlal Tested-by: Daniel Byrne Tested-by: Donald Carr Tested-by: Holger Hoffstätte Tested-by: Konstantin Kharlamov Tested-by: Shuang Zhai Tested-by: Sofia Trinh Acked-by: Will Deacon --- arch/arm64/include/asm/pgtable.h | 14 ++------------ arch/x86/include/asm/pgtable.h | 6 +++--- include/linux/pgtable.h | 13 +++++++++++++ mm/memory.c | 14 +------------- 4 files changed, 19 insertions(+), 28 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index c4ba047a82d2..990358eca359 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -999,23 +999,13 @@ static inline void update_mmu_cache(struct vm_area_struct *vma, * page after fork() + CoW for pfn mappings. We don't always have a * hardware-managed access flag on arm64. */ -static inline bool arch_faults_on_old_pte(void) -{ - WARN_ON(preemptible()); - - return !cpu_has_hw_af(); -} -#define arch_faults_on_old_pte arch_faults_on_old_pte +#define arch_has_hw_pte_young cpu_has_hw_af /* * Experimentally, it's cheap to set the access flag in hardware and we * benefit from prefaulting mappings as 'old' to start with. */ -static inline bool arch_wants_old_prefaulted_pte(void) -{ - return !arch_faults_on_old_pte(); -} -#define arch_wants_old_prefaulted_pte arch_wants_old_prefaulted_pte +#define arch_wants_old_prefaulted_pte cpu_has_hw_af static inline pgprot_t arch_filter_pgprot(pgprot_t prot) { diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 8a9432fb3802..60b6ce45c2e3 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -1423,10 +1423,10 @@ static inline bool arch_has_pfn_modify_check(void) return boot_cpu_has_bug(X86_BUG_L1TF); } -#define arch_faults_on_old_pte arch_faults_on_old_pte -static inline bool arch_faults_on_old_pte(void) +#define arch_has_hw_pte_young arch_has_hw_pte_young +static inline bool arch_has_hw_pte_young(void) { - return false; + return true; } #endif /* __ASSEMBLY__ */ diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index f4f4077b97aa..c799635f4d79 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -259,6 +259,19 @@ static inline int pmdp_clear_flush_young(struct vm_area_struct *vma, #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ #endif +#ifndef arch_has_hw_pte_young +/* + * Return whether the accessed bit is supported by the local CPU. + * + * This stub assumes accessing through an old PTE triggers a page fault. + * Architectures that automatically set the access bit should overwrite it. + */ +static inline bool arch_has_hw_pte_young(void) +{ + return false; +} +#endif + #ifndef __HAVE_ARCH_PTEP_CLEAR static inline void ptep_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep) diff --git a/mm/memory.c b/mm/memory.c index c125c4969913..a7379196a47e 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -122,18 +122,6 @@ int randomize_va_space __read_mostly = 2; #endif -#ifndef arch_faults_on_old_pte -static inline bool arch_faults_on_old_pte(void) -{ - /* - * Those arches which don't have hw access flag feature need to - * implement their own helper. By default, "true" means pagefault - * will be hit on old pte. - */ - return true; -} -#endif - #ifndef arch_wants_old_prefaulted_pte static inline bool arch_wants_old_prefaulted_pte(void) { @@ -2778,7 +2766,7 @@ static inline bool cow_user_page(struct page *dst, struct page *src, * On architectures with software "accessed" bits, we would * take a double page fault, so mark it accessed here. */ - if (arch_faults_on_old_pte() && !pte_young(vmf->orig_pte)) { + if (!arch_has_hw_pte_young() && !pte_young(vmf->orig_pte)) { pte_t entry; vmf->pte = pte_offset_map_lock(mm, vmf->pmd, addr, &vmf->ptl);