From patchwork Wed May 8 19:19:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13659109 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A3993C04FFE for ; Wed, 8 May 2024 19:28:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3F8C86B0096; Wed, 8 May 2024 15:28:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3816C6B00A3; Wed, 8 May 2024 15:28:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1FBA76B00A6; Wed, 8 May 2024 15:28:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 019B46B0096 for ; Wed, 8 May 2024 15:28:49 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id A8010A11B9 for ; Wed, 8 May 2024 19:28:49 +0000 (UTC) X-FDA: 82096215978.05.4BDFA95 Received: from mail-wm1-f54.google.com (mail-wm1-f54.google.com [209.85.128.54]) by imf15.hostedemail.com (Postfix) with ESMTP id B191CA0008 for ; Wed, 8 May 2024 19:28:47 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=DcwZ4N+D; spf=pass (imf15.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.54 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715196527; a=rsa-sha256; cv=none; b=Wx82Ep5rmFizZAs6kLGbJbB0u9zwF5ht4jBWYzg6ADw0tVIRtlNHAm2kc9FZHJZbsX67BI ZRBWO0tSoL02rSl0Ipiivo4027pRe5iEiUv561ZmEj9B4uYOW9rdhfY0XL5mFfauwSjwnD ywbwVMuZQwZCwabR4GjMHTFaoErQXSM= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=DcwZ4N+D; spf=pass (imf15.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.54 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715196527; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SrlnOIvJhP/2Gfg4T9gM6ETbApI/dif7rW/ZR+x6Q9w=; b=kNQMdOrqjnX1U7CNuAEsgjMUnFOfIYGFTGlQRPZ73Z9le3xd5QR2JzWOVeAduIZPjE9Rvw 2nN1bkpW8ZGEimuANBBtAvdTlboG/3Be7TAWmne1vOCZBb9cHwEVk6k9u+vfOckYnp4xfu X60jUZaNY0OFERbYwqUnXvoWG9s8mVU= Received: by mail-wm1-f54.google.com with SMTP id 5b1f17b1804b1-41ecffed96cso651965e9.1 for ; Wed, 08 May 2024 12:28:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1715196526; x=1715801326; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=SrlnOIvJhP/2Gfg4T9gM6ETbApI/dif7rW/ZR+x6Q9w=; b=DcwZ4N+Dcx7K0NT4ADocGnsK01grPBCMFsvFQzftONarcTngv6P6bc3v/UYFkGzOE4 kedZatmRfNLAYGANL6pzTO4b2TWYuGIL4Moy5B/QX5r0lFNt8Myppn6uO7cnI+o7glr0 D17lMV/I4jNuMK9flckaYZJz72r0saLY37qBzRDLr0WbWydF/QGXf4AGIR8uSiZ4SF3h /2EXyS4n/bmFN4PSuEwNoGFOeYtZ/V0VloZxhjVtt/2wLiEOe4NSrfQxTlUoLvJiNEOH a0E71P5zqE3YVLEzLBN/yrJJ/yaTvOayaDgmlFuKLKmbNBKhTVtQIh07HnFPVM9XDu7b EkfA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715196526; x=1715801326; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SrlnOIvJhP/2Gfg4T9gM6ETbApI/dif7rW/ZR+x6Q9w=; b=wN+dwa6H7aVIOJ5DiMoQB//tGeUZCgHwpa3ZQpUamvI+1CGYQ/XZ1BizwOXr+ztqFg vZzjX/9MVT2gdPp67qWvGUMWsC+cnkbIN1eE8hOZneF4xpmreS8cJ7LuYsf2mXdN2wP9 8PCq2iAeJqzCQOhdq/YBiHC2DiKn9Mb3IdBwfObstS9PIbtbo90cUrDX5KJf9IWxihNa d8u7JXCObUCrR7wVOId1QNQfPIQG6zfAoL5KiwUAcml0E5Fs2FZrzP+wGMvjzv6TfnEK ERBdN8KLfDYXbYj4ATBVQeZlIilrIyk3li/HbcWEzdz5YRi01+69iRUOl82xV7ClNy3h JHeQ== X-Forwarded-Encrypted: i=1; AJvYcCXzCYrauhb4LEMUnYV1DlB0w2gGMM7/ENvx9DU+8xqmRZND0ihlZJjssQUWO41JvRc0VZB/L51MmUwDYYbFyifB7I8= X-Gm-Message-State: AOJu0YxBE+SJBlgMCB/NddCJsRt28p4sTf/sTwFYcNR/naKpCYP4vMV9 RpJRzo1UYzF6PXEfDWwOG7ggRK5GPxuTDuApUc/LAqUD3c5y41jmkt06yGCrxuE= X-Google-Smtp-Source: AGHT+IEyIjL1IsEMqXGLkJGiy2tkR0X5tVHC2TKSU1+3Y1r/Xzm4QA5qwQ487USNifryKvOLnO9vbQ== X-Received: by 2002:a5d:6350:0:b0:34c:65ba:5d43 with SMTP id ffacd0b85a97d-34fca621699mr2523239f8f.46.1715196526270; Wed, 08 May 2024 12:28:46 -0700 (PDT) Received: from alex-rivos.ba.rivosinc.com (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id cx8-20020a056000092800b0034e01a80176sm16002694wrb.114.2024.05.08.12.28.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 May 2024 12:28:45 -0700 (PDT) From: Alexandre Ghiti To: Ryan Roberts , Catalin Marinas , Will Deacon , Alexander Potapenko , Marco Elver , Dmitry Vyukov , Paul Walmsley , Palmer Dabbelt , Albert Ou , Ard Biesheuvel , Anup Patel , Atish Patra , Andrey Ryabinin , Andrey Konovalov , Vincenzo Frascino , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-riscv@lists.infradead.org, linux-efi@vger.kernel.org, kvm@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH 09/12] mm, riscv, arm64: Use common ptep_clear_flush_young() function Date: Wed, 8 May 2024 21:19:28 +0200 Message-Id: <20240508191931.46060-10-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240508191931.46060-1-alexghiti@rivosinc.com> References: <20240508191931.46060-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-Stat-Signature: xfio9pwuc9s14ebnth84y6uiinffgs7s X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: B191CA0008 X-HE-Tag: 1715196527-315464 X-HE-Meta: U2FsdGVkX18Q16S+s/bzQNEeWBeiVD4c70mD9ZBz4sTgy6swWN4b+0/osnD6uzT22DeEsUlfw6ZuR5KrcuA4ScH49NLx/J6x+aITMINTB+Qd6lbHySkLeCg3Pru72UgpuD09y2Hd3+AJNQ0JJJMhzCf/vjKt19wu8u1hP5vCAuA86uXNjscY7yGD4m2kX/GDlZiAZvX3MJBGh8AGaUzcmjhu6bsm0DMDx9WSlYtCOTZThgANH31QdAOuFPn2vQ4b6dzuPRfLyMexiOqg+vaq65aFsmT7lKlMZ0pjUy74rBzcScwMpS58a6f+3wZlPriIQI1/WxmswazmPsI2qA8bTfGPZD416N1ICQsqvdC5ni+PQfU1Imaz3yZz+Z0u7NafewAslbPNzNAXbQDpQfM9lY/V4AOgNGwQCL9kwJUuWXV0a05OLs7P5Fkrf08DUxb/mvnR2CJxZdqVD4/KFqTWq3Hr5TUGvM1QCkspd0/jaBax31MANllqiYxXJOpXdJRkGmzYhBZFpaqFAOAef5sX/671LgxYhsD6rjO/G3Lbz6yG2vIvWlF6RH7eJZgHrTXD0diiyVdwfR9c7w8XZALNow5gZf5gpyIMZzV3MJUjl4vWz7GV6x9FmomeLzGt9/wu4qH3vFo8Ffzl3IG70QDuEKXeSKjR/Fk7gzMLiT0kAlUssbj1Wecpjakn8BXXJtLKhY0cIwwxrsamNJ0zhKdU8J+jrF28/NExamNImbMUyjkz+JVP9hi6WFi/yUdXrAz+MAaiJFVJF0KyGl1v6gmnNEY15SLs6uvfHxLIMKQpzZxLCHwquRFHFw5nGEwnsHqXQ9M5GLQ0LtVWmBM4YOgPDsqyk0QoFQXgfCpYjNJnR/RbdCVk9H6IVrl90gegbDOJktKy5yFqIQUi5+ZCgq4cPXLwMTQ5Dk+UHj0a8QQONEOSZKzlxKOOZfO19a4vRq5EfnkEGvWbrc57Vhis9/1 KEKNKFf9 fXJ9qkbHMSScASPjSXiPAp7OXM1F4mwslYdeQkiEsEnfua5yfAET2dXECnJR3CG3YGQ9wdwqJeCfeHBgjyk4lI/n1pS5CBCHTQLK6k6UOwcpunPO7BolKPKr4w/A/C+wbnvllDueobg65CkYhybbTc35H3dv1w3UiTPmTNok7WXXvZFoWxsM6i+wV+RqLbJw7nQNDhaq4Emm5XY5Xphj3pg7yKGTtbxyHOICC6W4/hET9GPwnEXFPihejjaW/6fA1tGFPQMimfkuwlZbK2o5s9AuAnPK2qLaZymsQJMZ092nkADZb4luD6gqC1SbSelpv3eu0HrB8EHTDYC9dEvhTLwGROWwDxVcah8B8NuLmWpeizY2h5uxBzuuZLtb2VH8pyRiFIWQZARbmv0Iit98a5LUCE9mf62a7VUWk4aWp/cOusUDH9vBKKkSmJEcRLtvHoAm/6GNUFrh22pcnwSpRf9uWiEWhRPPHATh195Hz7KMGCuyhLslu4Z+R6UwPxPv/qrZSgDWwLP2T3MzMfLFOE/AJLdWP0PjhOr8UNI3jIyAr60Ao4tnkWh72og== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Make riscv use the contpte aware ptep_clear_flush_young() function from arm64. Note that riscv used to not flush the tlb after clearing the accessed bit, which it does now: this will be improved when we implement svinval support. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/pgtable.h | 22 ++++++++---------- arch/arm64/mm/contpte.c | 21 ----------------- arch/riscv/include/asm/pgtable.h | 12 +++++++--- include/linux/contpte.h | 2 ++ mm/contpte.c | 40 ++++++++++++++++++++++++++++++++ 5 files changed, 61 insertions(+), 36 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 9a8702d1ad00..92c12fb85cb4 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1389,8 +1389,6 @@ extern void contpte_clear_full_ptes(struct mm_struct *mm, unsigned long addr, extern pte_t contpte_get_and_clear_full_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned int nr, int full); -extern int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep); extern void contpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned int nr); extern int contpte_ptep_set_access_flags(struct vm_area_struct *vma, @@ -1479,16 +1477,8 @@ extern int ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); #define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH -static inline int ptep_clear_flush_young(struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep) -{ - pte_t orig_pte = __ptep_get(ptep); - - if (likely(!pte_valid_cont(orig_pte))) - return __ptep_clear_flush_young(vma, addr, ptep); - - return contpte_ptep_clear_flush_young(vma, addr, ptep); -} +extern int ptep_clear_flush_young(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep); #define wrprotect_ptes wrprotect_ptes static __always_inline void wrprotect_ptes(struct mm_struct *mm, @@ -1616,6 +1606,14 @@ static inline void arch_contpte_flush_tlb_range(struct vm_area_struct *vma, __flush_tlb_range(vma, start, end, stride, true, 3); } +static inline void arch_contpte_flush_tlb_range_nosync(struct vm_area_struct *vma, + unsigned long start, + unsigned long end, + unsigned long stride) +{ + __flush_tlb_range_nosync(vma, start, end, stride, true, 3); +} + static inline int arch_contpte_get_first_ncontig(size_t *pgsize) { if (pgsize) diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c index 9bf471633ca4..16940511943c 100644 --- a/arch/arm64/mm/contpte.c +++ b/arch/arm64/mm/contpte.c @@ -45,27 +45,6 @@ pte_t contpte_get_and_clear_full_ptes(struct mm_struct *mm, } EXPORT_SYMBOL_GPL(contpte_get_and_clear_full_ptes); -int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep) -{ - int young; - - young = contpte_ptep_test_and_clear_young(vma, addr, ptep); - - if (young) { - /* - * See comment in __ptep_clear_flush_young(); same rationale for - * eliding the trailing DSB applies here. - */ - addr = ALIGN_DOWN(addr, CONT_PTE_SIZE); - __flush_tlb_range_nosync(vma, addr, addr + CONT_PTE_SIZE, - PAGE_SIZE, true, 3); - } - - return young; -} -EXPORT_SYMBOL_GPL(contpte_ptep_clear_flush_young); - void contpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned int nr) { diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index d39cb24c6c4a..42c7884b8d2e 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -615,6 +615,8 @@ static inline void arch_contpte_flush_tlb_range(struct vm_area_struct *vma, flush_tlb_mm_range(vma->vm_mm, start, end, stride); } +#define arch_contpte_flush_tlb_range_nosync arch_contpte_flush_tlb_range + static inline int arch_contpte_get_first_ncontig(size_t *pgsize) { if (pgsize) @@ -758,9 +760,8 @@ static inline void __ptep_set_wrprotect(struct mm_struct *mm, atomic_long_and(~(unsigned long)_PAGE_WRITE, (atomic_long_t *)ptep); } -#define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH -static inline int ptep_clear_flush_young(struct vm_area_struct *vma, - unsigned long address, pte_t *ptep) +static inline int __ptep_clear_flush_young(struct vm_area_struct *vma, + unsigned long address, pte_t *ptep) { /* * This comment is borrowed from x86, but applies equally to RISC-V: @@ -799,6 +800,9 @@ extern pte_t ptep_get_and_clear(struct mm_struct *mm, #define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG extern int ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); +#define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH +extern int ptep_clear_flush_young(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep); #else /* CONFIG_THP_CONTPTE */ @@ -810,6 +814,8 @@ extern int ptep_test_and_clear_young(struct vm_area_struct *vma, #define ptep_get_and_clear __ptep_get_and_clear #define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG #define ptep_test_and_clear_young __ptep_test_and_clear_young +#define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH +#define ptep_clear_flush_young __ptep_clear_flush_young #endif /* CONFIG_THP_CONTPTE */ diff --git a/include/linux/contpte.h b/include/linux/contpte.h index 38092adbe0d4..76a49ac8b6f5 100644 --- a/include/linux/contpte.h +++ b/include/linux/contpte.h @@ -21,5 +21,7 @@ void contpte_set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte, unsigned int nr); int contpte_ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); +int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep); #endif /* _LINUX_CONTPTE_H */ diff --git a/mm/contpte.c b/mm/contpte.c index 220e9d81f401..600277b1196c 100644 --- a/mm/contpte.c +++ b/mm/contpte.c @@ -48,6 +48,7 @@ * - pte_clear() * - ptep_get_and_clear() * - ptep_test_and_clear_young() + * - ptep_clear_flush_young() */ pte_t huge_ptep_get(pte_t *ptep) @@ -729,4 +730,43 @@ __always_inline int ptep_test_and_clear_young(struct vm_area_struct *vma, return contpte_ptep_test_and_clear_young(vma, addr, ptep); } + +int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep) +{ + int young; + + young = contpte_ptep_test_and_clear_young(vma, addr, ptep); + + if (young) { + /* + * See comment in __ptep_clear_flush_young(); same rationale for + * eliding the trailing DSB applies here. + */ + size_t pgsize; + int ncontig; + + ncontig = arch_contpte_get_num_contig(vma->vm_mm, addr, ptep, + 0, &pgsize); + + addr = ALIGN_DOWN(addr, ncontig * pgsize); + arch_contpte_flush_tlb_range_nosync(vma, addr, + addr + ncontig * pgsize, + pgsize); + } + + return young; +} +EXPORT_SYMBOL_GPL(contpte_ptep_clear_flush_young); + +__always_inline int ptep_clear_flush_young(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep) +{ + pte_t orig_pte = __ptep_get(ptep); + + if (likely(!pte_valid_cont(orig_pte))) + return __ptep_clear_flush_young(vma, addr, ptep); + + return contpte_ptep_clear_flush_young(vma, addr, ptep); +} #endif /* CONFIG_THP_CONTPTE */