From patchwork Wed Aug 18 06:30:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 12443379 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C5C62C432BE for ; Wed, 18 Aug 2021 06:31:14 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6EFEF60720 for ; Wed, 18 Aug 2021 06:31:14 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 6EFEF60720 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 0AD846B0074; Wed, 18 Aug 2021 02:31:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E3F6F6B0075; Wed, 18 Aug 2021 02:31:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C6A8E6B0078; Wed, 18 Aug 2021 02:31:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0107.hostedemail.com [216.40.44.107]) by kanga.kvack.org (Postfix) with ESMTP id AAB506B0074 for ; Wed, 18 Aug 2021 02:31:13 -0400 (EDT) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 5FE438249980 for ; Wed, 18 Aug 2021 06:31:13 +0000 (UTC) X-FDA: 78487229226.28.BBB4431 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf18.hostedemail.com (Postfix) with ESMTP id 1839D4001EBB for ; Wed, 18 Aug 2021 06:31:12 +0000 (UTC) Received: by mail-yb1-f202.google.com with SMTP id p71-20020a25424a0000b029056092741626so1764846yba.19 for ; Tue, 17 Aug 2021 23:31:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=axDz8FmbUkmwoP5Po1n0WrbURWOp6eK+Sy9KgBfPnX8=; b=tMJdGo0VdmwhtsmfaZ+8rkd4E9lkxdQ44NTW3d4/TNpCinw/fAFDzywEKib15cClPi W29Wg6lpdN+bZgNq9Vd3h4q/1uzr+5hqEfenQ6HyCLa6QbL33fZSCMLzbTULgjUGwpI+ BXmZg8SRzvZbXVtkQBlaLzEKm8WV77UfxsQ8EG6tQhxw6/Eh23ojg4fHfZC7QgWEo1ji DDtk9ZcuKvr1BNn+esg0iIEsM4XejbvsJ+161wy6H40w1DQU0zhTeR6Qc0qXfc3jhHfB OSV1hDsLPUfb0gbAc/64uz9OCgvP0IMwvygx6lIZRwZoCsDlkU53OLv5tpYI+ITQhfO/ zGEA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=axDz8FmbUkmwoP5Po1n0WrbURWOp6eK+Sy9KgBfPnX8=; b=HEyHpRA7YkfffNvslIvW6HuN/6lnGvyQzgi2vghhPPVeVN1+ZUdzybOhUmp8V1aIsd R6fuYvHIpvNaRU142y3ulk0Hu9MLPLhdMWDHgdWfAbIZbPHtXgIshRoGehpK84rKwtDo 5tSnPqV8YuFY1LsWdBgNyDHlrWBWFvpwRw77Zp/UM4h4INljVWXA58iykQGq6VBngklo rsRicm6hIe9fEMon2Pnmx2zr8mAqhb8NSrlKRey0Bl7gmRdM8+cbn9EYrklokoXvCtKk lW64Y3rnhPEoczcQygRAiBGYyJ05hDiyz3qZ18SGS7sYMABsjNr0g83Bu9xDoPa91RVC MSkw== X-Gm-Message-State: AOAM533RJ4Wj2uAqtXyGyNOhUUTEKfFVtgLHkrIUi6WCw6BsLiZ0OUjk DRJPW3BODZlhqj+fBg4YowtzkJpwM9h0gx2aA6A/hNtOB2FHNaK+tsi74IrNsA92Ev/+6euQD6R MDW0nY//owiss6a0Zn+KAhFxRVqnkJyheQOyWmPpErefzhtBP/L5D1Mmw X-Google-Smtp-Source: ABdhPJz9k/2W2LmgADoTsZbfpB5d6btee0nxlX6DF/x7Ki85TCn69a1MVMx/WICufFas9KT4dBOAj2dLDVU= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:41f0:f89:87cd:8bd0]) (user=yuzhao job=sendgmr) by 2002:a25:cf8a:: with SMTP id f132mr9779192ybg.387.1629268272384; Tue, 17 Aug 2021 23:31:12 -0700 (PDT) Date: Wed, 18 Aug 2021 00:30:57 -0600 In-Reply-To: <20210818063107.2696454-1-yuzhao@google.com> Message-Id: <20210818063107.2696454-2-yuzhao@google.com> Mime-Version: 1.0 References: <20210818063107.2696454-1-yuzhao@google.com> X-Mailer: git-send-email 2.33.0.rc1.237.g0d66db33f3-goog Subject: [PATCH v4 01/11] mm: x86, arm64: add arch_has_hw_pte_young() From: Yu Zhao To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Hillf Danton , page-reclaim@google.com, Yu Zhao Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=google.com header.s=20161025 header.b=tMJdGo0V; spf=pass (imf18.hostedemail.com: domain of 3MKkcYQYKCAU3z4mftlttlqj.htrqnsz2-rrp0fhp.twl@flex--yuzhao.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3MKkcYQYKCAU3z4mftlttlqj.htrqnsz2-rrp0fhp.twl@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 1839D4001EBB X-Stat-Signature: ierg7zqi1iutxm8fjk9smhyogpegi6rn X-HE-Tag: 1629268272-765849 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Some architectures set the accessed bit in PTEs automatically, e.g., x86, and arm64 v8.2 and later. On architectures that do not have this capability, clearing the accessed bit in a PTE triggers a page fault following the TLB miss. Being aware of this capability can help make better decisions, i.e., whether to limit the size of each batch of PTEs and the burst of batches when clearing the accessed bit. Signed-off-by: Yu Zhao --- arch/arm64/include/asm/cpufeature.h | 19 ++++++------------- arch/arm64/include/asm/pgtable.h | 10 ++++------ arch/arm64/kernel/cpufeature.c | 19 +++++++++++++++++++ arch/arm64/mm/proc.S | 12 ------------ arch/arm64/tools/cpucaps | 1 + arch/x86/include/asm/pgtable.h | 6 +++--- include/linux/pgtable.h | 12 ++++++++++++ mm/memory.c | 14 +------------- 8 files changed, 46 insertions(+), 47 deletions(-) diff --git a/arch/arm64/include/asm/cpufeature.h b/arch/arm64/include/asm/cpufeature.h index 9bb9d11750d7..2020b9e818c8 100644 --- a/arch/arm64/include/asm/cpufeature.h +++ b/arch/arm64/include/asm/cpufeature.h @@ -776,6 +776,12 @@ static inline bool system_supports_tlb_range(void) cpus_have_const_cap(ARM64_HAS_TLB_RANGE); } +/* Check whether hardware update of the Access flag is supported. */ +static inline bool system_has_hw_af(void) +{ + return IS_ENABLED(CONFIG_ARM64_HW_AFDBM) && cpus_have_const_cap(ARM64_HW_AF); +} + extern int do_emulate_mrs(struct pt_regs *regs, u32 sys_reg, u32 rt); static inline u32 id_aa64mmfr0_parange_to_phys_shift(int parange) @@ -799,19 +805,6 @@ static inline u32 id_aa64mmfr0_parange_to_phys_shift(int parange) } } -/* Check whether hardware update of the Access flag is supported */ -static inline bool cpu_has_hw_af(void) -{ - u64 mmfr1; - - if (!IS_ENABLED(CONFIG_ARM64_HW_AFDBM)) - return false; - - mmfr1 = read_cpuid(ID_AA64MMFR1_EL1); - return cpuid_feature_extract_unsigned_field(mmfr1, - ID_AA64MMFR1_HADBS_SHIFT); -} - static inline bool cpu_has_pan(void) { u64 mmfr1 = read_cpuid(ID_AA64MMFR1_EL1); diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index f09bf5c02891..b63a6a7b62ee 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -993,13 +993,11 @@ static inline void update_mmu_cache(struct vm_area_struct *vma, * page after fork() + CoW for pfn mappings. We don't always have a * hardware-managed access flag on arm64. */ -static inline bool arch_faults_on_old_pte(void) +static inline bool arch_has_hw_pte_young(void) { - WARN_ON(preemptible()); - - return !cpu_has_hw_af(); + return system_has_hw_af(); } -#define arch_faults_on_old_pte arch_faults_on_old_pte +#define arch_has_hw_pte_young arch_has_hw_pte_young /* * Experimentally, it's cheap to set the access flag in hardware and we @@ -1007,7 +1005,7 @@ static inline bool arch_faults_on_old_pte(void) */ static inline bool arch_wants_old_prefaulted_pte(void) { - return !arch_faults_on_old_pte(); + return arch_has_hw_pte_young(); } #define arch_wants_old_prefaulted_pte arch_wants_old_prefaulted_pte diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 0ead8bfedf20..d05de77626f5 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -1650,6 +1650,14 @@ static bool has_hw_dbm(const struct arm64_cpu_capabilities *cap, return true; } +static void cpu_enable_hw_af(struct arm64_cpu_capabilities const *cap) +{ + u64 val = read_sysreg(tcr_el1); + + write_sysreg(val | TCR_HA, tcr_el1); + isb(); + local_flush_tlb_all(); +} #endif #ifdef CONFIG_ARM64_AMU_EXTN @@ -2126,6 +2134,17 @@ static const struct arm64_cpu_capabilities arm64_features[] = { .matches = has_hw_dbm, .cpu_enable = cpu_enable_hw_dbm, }, + { + .desc = "Hardware update of the Access flag", + .type = ARM64_CPUCAP_SYSTEM_FEATURE, + .capability = ARM64_HW_AF, + .sys_reg = SYS_ID_AA64MMFR1_EL1, + .sign = FTR_UNSIGNED, + .field_pos = ID_AA64MMFR1_HADBS_SHIFT, + .min_field_value = 1, + .matches = has_cpuid_feature, + .cpu_enable = cpu_enable_hw_af, + }, #endif { .desc = "CRC32 instructions", diff --git a/arch/arm64/mm/proc.S b/arch/arm64/mm/proc.S index 35936c5ae1ce..b066d5712e3d 100644 --- a/arch/arm64/mm/proc.S +++ b/arch/arm64/mm/proc.S @@ -478,18 +478,6 @@ SYM_FUNC_START(__cpu_setup) * Set the IPS bits in TCR_EL1. */ tcr_compute_pa_size tcr, #TCR_IPS_SHIFT, x5, x6 -#ifdef CONFIG_ARM64_HW_AFDBM - /* - * Enable hardware update of the Access Flags bit. - * Hardware dirty bit management is enabled later, - * via capabilities. - */ - mrs x9, ID_AA64MMFR1_EL1 - and x9, x9, #0xf - cbz x9, 1f - orr tcr, tcr, #TCR_HA // hardware Access flag update -1: -#endif /* CONFIG_ARM64_HW_AFDBM */ msr mair_el1, mair msr tcr_el1, tcr /* diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps index 49305c2e6dfd..d52f50671e60 100644 --- a/arch/arm64/tools/cpucaps +++ b/arch/arm64/tools/cpucaps @@ -35,6 +35,7 @@ HAS_STAGE2_FWB HAS_SYSREG_GIC_CPUIF HAS_TLB_RANGE HAS_VIRT_HOST_EXTN +HW_AF HW_DBM KVM_PROTECTED_MODE MISMATCHED_CACHE_TYPE diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 448cd01eb3ec..3908780fc408 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -1397,10 +1397,10 @@ static inline bool arch_has_pfn_modify_check(void) return boot_cpu_has_bug(X86_BUG_L1TF); } -#define arch_faults_on_old_pte arch_faults_on_old_pte -static inline bool arch_faults_on_old_pte(void) +#define arch_has_hw_pte_young arch_has_hw_pte_young +static inline bool arch_has_hw_pte_young(void) { - return false; + return true; } #endif /* __ASSEMBLY__ */ diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index e24d2c992b11..3a8221fa2c76 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -258,6 +258,18 @@ static inline int pmdp_clear_flush_young(struct vm_area_struct *vma, #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ #endif +#ifndef arch_has_hw_pte_young +static inline bool arch_has_hw_pte_young(void) +{ + /* + * Those arches which have hw access flag feature need to implement + * their own helper. By default, "false" means pagefault will be hit + * on old pte. + */ + return false; +} +#endif + #ifndef __HAVE_ARCH_PTEP_GET_AND_CLEAR static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long address, diff --git a/mm/memory.c b/mm/memory.c index 25fc46e87214..2f96179db219 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -121,18 +121,6 @@ int randomize_va_space __read_mostly = 2; #endif -#ifndef arch_faults_on_old_pte -static inline bool arch_faults_on_old_pte(void) -{ - /* - * Those arches which don't have hw access flag feature need to - * implement their own helper. By default, "true" means pagefault - * will be hit on old pte. - */ - return true; -} -#endif - #ifndef arch_wants_old_prefaulted_pte static inline bool arch_wants_old_prefaulted_pte(void) { @@ -2769,7 +2757,7 @@ static inline bool cow_user_page(struct page *dst, struct page *src, * On architectures with software "accessed" bits, we would * take a double page fault, so mark it accessed here. */ - if (arch_faults_on_old_pte() && !pte_young(vmf->orig_pte)) { + if (!arch_has_hw_pte_young() && !pte_young(vmf->orig_pte)) { pte_t entry; vmf->pte = pte_offset_map_lock(mm, vmf->pmd, addr, &vmf->ptl);