From patchwork Thu Nov 11 04:15:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 12692249 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 605D9C433EF for ; Thu, 11 Nov 2021 04:21:16 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 241B66134F for ; Thu, 11 Nov 2021 04:21:16 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 241B66134F Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:From:Subject:References: Mime-Version:Message-Id:In-Reply-To:Date:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=Xf7EY5ydM+k1Y5/f7O/+I2ny2eoEsKuULAD6E3VZ9oI=; b=xKUMpCoMQkpXYw1UL5svTqDGN/ khk4AuZGYZjCqZJKlDG7kVQfVSiR4FzfukzqgINYQgYZ8GjA7X5bv2JqjnPHdRDP1ga2KcnUFE3pW /EpNEoNpfCCeYLquTlEnnF0WjEfrxsdxvgt6gGfGfeQ4nw6xkpJ0xrU4MxG2csPp9iPCxozEe+wZE 9cfuVW1NTGD1NA6FjG+casBybvIbdBQFQRtqRby6wIlQz3itcPBeFLALgmFDsEZI7G2D3KFE+jiyq mkD1g0hqf/A6idVw5XHJHTfBb6Dqr4JEVIh2UUuaXo4JFxuLGD/O2K+URj7MPpydiQg2VK8qlumAV 7CdeFhQg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1ml1Ys-0072KH-92; Thu, 11 Nov 2021 04:19:43 +0000 Received: from mail-yb1-xb49.google.com ([2607:f8b0:4864:20::b49]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1ml1Ue-006zrv-Q1 for linux-arm-kernel@lists.infradead.org; Thu, 11 Nov 2021 04:15:28 +0000 Received: by mail-yb1-xb49.google.com with SMTP id d8-20020a253608000000b005c202405f52so7526489yba.7 for ; Wed, 10 Nov 2021 20:15:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=UqQ1CAHDKzNIHLLK4QSZngjkwRJTSigVp7v0+NSGo7s=; b=iCTrg5hHtKFPWug2bcoGi1y0JSa/3ibgMhmXVbdRD8JMGzVbhCPL8gwxwh6Hd6I0m4 cxsZfNdVwtSrDVKdEp3d9cqbPMulbzip+86pdgV4njqKhDOi3BIQnPGMz2aT0ZKPV96n xK+tc14myylCnDee3S0ioGi+PpuCZd9TMCv5IHtHpbPhSglKW+PxgU9vaBXFQI4HSxux t/DwjX34g7G80VP9EDeyeShh1C+RWvkNa6UjlGUFqMsDUdu/+mJdn5Mu1DSLhFDu8QPN mMRo4OdwQjqUK31qpe28XbPEKhUEnh0Yicb9/nDJgtLVccFoGmVcmDKfZD7e9ulZQDEX 4kCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=UqQ1CAHDKzNIHLLK4QSZngjkwRJTSigVp7v0+NSGo7s=; b=DDfxB66e4kFnPG+IrsPYOJP8shQbzCDfu2v77YydHRIsvVyVP5+gZAUuSaIBjK5UtF 6PkS2g5ni+9mMUkxbOtPN6iYsZPBx0MaJDrcTWNzONvxjnXj7YfPDuoR7KZUdPB+uibB KyKaVA/MJcE7I9NWw42/rotUBU4K6+Pc9xBRy/ovl2hoS+I49wZnO5P39zUPonrtlfga UiZ0PUC4gwuFdxkgdC3i/rFF76NiCRFfp7osWHApIO9dG0R2NYujlcX+juXQS8amRDqD qXQ3+Wp/FujPL86MijStN8mJL+xu1D9dH6AaJX0dgp1q0pzivfHSY9W8BmY1GLBlAeze KCbw== X-Gm-Message-State: AOAM530qM4U1R+uPyOp0XlwKypS+J+MtfS/+KOrvHAi4nl38t0+yuwbV 6biQfqSf0xi1aufprssBk70LAb08P6U= X-Google-Smtp-Source: ABdhPJwuPl9PJhP6Upxe5eqWLpjhaqFw0snAdbggMzVCEnomOiwgAM1BKhvsCadbArbOq/ns603ARh86ExE= X-Received: from yuzhao.bld.corp.google.com ([2620:15c:183:200:346b:bb72:659e:f91c]) (user=yuzhao job=sendgmr) by 2002:a25:c70f:: with SMTP id w15mr5226488ybe.167.1636604118332; Wed, 10 Nov 2021 20:15:18 -0800 (PST) Date: Wed, 10 Nov 2021 21:15:02 -0700 In-Reply-To: <20211111041510.402534-1-yuzhao@google.com> Message-Id: <20211111041510.402534-3-yuzhao@google.com> Mime-Version: 1.0 References: <20211111041510.402534-1-yuzhao@google.com> X-Mailer: git-send-email 2.34.0.rc0.344.g81b53c2807-goog Subject: [PATCH v5 02/10] mm: x86: add CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG From: Yu Zhao To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, x86@kernel.org, page-reclaim@google.com, holger@applied-asynchrony.com, iam@valdikss.org.ru, Yu Zhao , Konstantin Kharlamov X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20211110_201520_929662_7EC2C7D0 X-CRM114-Status: GOOD ( 17.64 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Some architectures support the accessed bit on non-leaf PMD entries, e.g., x86_64 sets the accessed bit on a non-leaf PMD entry when using it as part of linear address translation [1]. As an optimization, page table walkers who are interested in the accessed bit can skip the PTEs under a non-leaf PMD entry if the accessed bit is cleared on this PMD entry. Although an inline function may be preferable, this capability is added as a configuration option to look consistent when used with the existing macros. [1]: Intel 64 and IA-32 Architectures Software Developer's Manual Volume 3 (June 2021), section 4.8 Signed-off-by: Yu Zhao Tested-by: Konstantin Kharlamov --- arch/Kconfig | 9 +++++++++ arch/x86/Kconfig | 1 + arch/x86/include/asm/pgtable.h | 3 ++- arch/x86/mm/pgtable.c | 5 ++++- include/linux/pgtable.h | 4 ++-- 5 files changed, 18 insertions(+), 4 deletions(-) diff --git a/arch/Kconfig b/arch/Kconfig index 8df1c7102643..7392fcc88777 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -1288,6 +1288,15 @@ config ARCH_HAS_ELFCORE_COMPAT config ARCH_HAS_PARANOID_L1D_FLUSH bool +config ARCH_HAS_NONLEAF_PMD_YOUNG + bool + depends on PGTABLE_LEVELS > 2 + help + Architectures that select this are able to set the accessed bit on + non-leaf PMD entries in addition to leaf PTE entries where pages are + mapped. For them, page table walkers that clear the accessed bit may + stop at non-leaf PMD entries if they do not see the accessed bit. + source "kernel/gcov/Kconfig" source "scripts/gcc-plugins/Kconfig" diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig index d9830e7e1060..245742b79be9 100644 --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -84,6 +84,7 @@ config X86 select ARCH_HAS_PMEM_API if X86_64 select ARCH_HAS_PTE_DEVMAP if X86_64 select ARCH_HAS_PTE_SPECIAL + select ARCH_HAS_NONLEAF_PMD_YOUNG if X86_64 select ARCH_HAS_UACCESS_FLUSHCACHE if X86_64 select ARCH_HAS_COPY_MC if X86_64 select ARCH_HAS_SET_MEMORY diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index c60b16f8b741..36205ec0acac 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -817,7 +817,8 @@ static inline unsigned long pmd_page_vaddr(pmd_t pmd) static inline int pmd_bad(pmd_t pmd) { - return (pmd_flags(pmd) & ~_PAGE_USER) != _KERNPG_TABLE; + return (pmd_flags(pmd) & ~(_PAGE_USER | _PAGE_ACCESSED)) != + (_KERNPG_TABLE & ~_PAGE_ACCESSED); } static inline unsigned long pages_to_mb(unsigned long npg) diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 3481b35cb4ec..a224193d84bf 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -550,7 +550,7 @@ int ptep_test_and_clear_young(struct vm_area_struct *vma, return ret; } -#ifdef CONFIG_TRANSPARENT_HUGEPAGE +#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG) int pmdp_test_and_clear_young(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmdp) { @@ -562,6 +562,9 @@ int pmdp_test_and_clear_young(struct vm_area_struct *vma, return ret; } +#endif + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE int pudp_test_and_clear_young(struct vm_area_struct *vma, unsigned long addr, pud_t *pudp) { diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index a64c6124c137..afb9004b778a 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -211,7 +211,7 @@ static inline int ptep_test_and_clear_young(struct vm_area_struct *vma, #endif #ifndef __HAVE_ARCH_PMDP_TEST_AND_CLEAR_YOUNG -#ifdef CONFIG_TRANSPARENT_HUGEPAGE +#if defined(CONFIG_TRANSPARENT_HUGEPAGE) || defined(CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG) static inline int pmdp_test_and_clear_young(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp) @@ -232,7 +232,7 @@ static inline int pmdp_test_and_clear_young(struct vm_area_struct *vma, BUILD_BUG(); return 0; } -#endif /* CONFIG_TRANSPARENT_HUGEPAGE */ +#endif /* CONFIG_TRANSPARENT_HUGEPAGE || CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG */ #endif #ifndef __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH