From patchwork Wed May 8 19:19:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13659110 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 090E6C04FFE for ; Wed, 8 May 2024 19:29:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 952F86B00A6; Wed, 8 May 2024 15:29:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8DA156B00A7; Wed, 8 May 2024 15:29:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 753816B00A8; Wed, 8 May 2024 15:29:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 51BA46B00A6 for ; Wed, 8 May 2024 15:29:51 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id CA93216079D for ; Wed, 8 May 2024 19:29:50 +0000 (UTC) X-FDA: 82096218540.09.F6B870B Received: from mail-wm1-f43.google.com (mail-wm1-f43.google.com [209.85.128.43]) by imf03.hostedemail.com (Postfix) with ESMTP id E3DE62000C for ; Wed, 8 May 2024 19:29:48 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=One38r9T; dmarc=none; spf=pass (imf03.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.43 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715196589; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=uckq1a6OF7H/9ROfgD12E/DRrLJPxYkFYbl5+hSyETI=; b=WF0jKPTkPY2gBN1mtEPQqXlfhfWLmVXpp5LvHmA+rO71vBlUShVH+cgCqT0G/K2RqW5dqb LUVOGUnoZc9JNGZukgicoTuNYNyFxi2LfZHG2bDjnpdCo0uvgRd/96tg9phQFZZKfPWh9x XcGEEqe5L5ilWjXhsOFr3zt5oVP0HRk= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715196589; a=rsa-sha256; cv=none; b=7SVL3xU3nMGsA35M8aaSScpBuaeHlMnBG3T/4mOexgEq++OUeQ9FlfXT0UURcJn+629Ao/ dbdAHaitJJUkHoDQuoPobCejuW8zTa/QZpB6+23X5J7RdaI5WPiPLrNnCRsYNd2QURiqQU Lqp9rwQpgHWnFxu27cA09vo/7+3L/v0= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=One38r9T; dmarc=none; spf=pass (imf03.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.43 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com Received: by mail-wm1-f43.google.com with SMTP id 5b1f17b1804b1-41ecffed96cso662195e9.1 for ; Wed, 08 May 2024 12:29:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1715196587; x=1715801387; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=uckq1a6OF7H/9ROfgD12E/DRrLJPxYkFYbl5+hSyETI=; b=One38r9TvfvDu8fsKgGJ+fqe7BSTLbYnO5Ng2o7I5Ptu7ayLxmbL8Fm53QVbvQTBOa GzdXEU+3RgYvtAbVORX0jVYQqxD1dme9qYgRzoGLIave2ZvF/mhNSZb8ypXQ0GkF0wqL B8LdTlB8MtFpHy/IT53KAJZKjtBjTLw4pvej4aaoRG9D+0jbf0kK5Cu+ZIRvTXmbC+Gn LuNtHGfJhXVAveObxNshDpo3OCHL7r0QSciFsFr/hLO5zoMQzG+mXEuHaEppBngpDljF BFHsZZ+F7zp/FRdq9wxD+D4kIvjlfqV0L8jQNET0KGIvP6A089VYSPjmsb1zHHVY2Ztq 5Yfg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715196587; x=1715801387; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=uckq1a6OF7H/9ROfgD12E/DRrLJPxYkFYbl5+hSyETI=; b=Mev99o9mHSe3U2RzM/zZ2qQxt1QTK+9JS7kYuSHX+CFLXPtPCeG424c63bXRGHBt+t AsEBvFb+VWtiWlT4/cNlPfAC1NPXgBXaHku40gKRRPY9FOhsylCRaCSGWHDYaGjFVuf+ gptMbNPiV9MtE/buPxlC3ddooeNZtcc73oxFdHpVWJP2Y5m90SxayIkXjHXsOBRKqhJd PYY3bLM63HoQxePhrRvVUvlVtCu8VgctrkIq1u4g/S36FwuKcMhSEjo9Q5ZAgS9FZGcf FbNc5VIA1TWvEAJBB1yvdypdWQmr0kZXkaI7w2TRnxDzAg+hoMe3jaP5q2O14dtShwTv nflw== X-Forwarded-Encrypted: i=1; AJvYcCUGchM6e81n9UsWDGES2UP66wi2u87shzaA88hjjwwKsNyPQmAZ5BKBqMd+bweoCT7qZGfXlIJY7dvdRhmBqbSfVuo= X-Gm-Message-State: AOJu0Yz5zbS+22SKvvpk0CxxFyNMqkq9k/zXCeIu2Yhb+y+Ll0po0Lyx MGaVon39ShP7bMlwiSp8DUfaBzvxhPTxrX9ptOQrWBHZGtK/ZEU70J0cPNp9ExA= X-Google-Smtp-Source: AGHT+IGQ61Vb4NuwkawA/0/DWqDCfOJMo7+D5f/tM30tLGXtIG6xkHE311QLb6FnYOpUeATe3M4jpQ== X-Received: by 2002:a05:600c:4e93:b0:41b:fc3a:f1ef with SMTP id 5b1f17b1804b1-41f71acca18mr25217385e9.33.1715196587426; Wed, 08 May 2024 12:29:47 -0700 (PDT) Received: from alex-rivos.ba.rivosinc.com (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id d16-20020a05600c34d000b00419f572671dsm3314921wmq.20.2024.05.08.12.29.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 May 2024 12:29:47 -0700 (PDT) From: Alexandre Ghiti To: Ryan Roberts , Catalin Marinas , Will Deacon , Alexander Potapenko , Marco Elver , Dmitry Vyukov , Paul Walmsley , Palmer Dabbelt , Albert Ou , Ard Biesheuvel , Anup Patel , Atish Patra , Andrey Ryabinin , Andrey Konovalov , Vincenzo Frascino , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-riscv@lists.infradead.org, linux-efi@vger.kernel.org, kvm@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH 10/12] mm, riscv, arm64: Use common ptep_set_access_flags() function Date: Wed, 8 May 2024 21:19:29 +0200 Message-Id: <20240508191931.46060-11-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240508191931.46060-1-alexghiti@rivosinc.com> References: <20240508191931.46060-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: E3DE62000C X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: a4zqw1gijorn48ftrsq7j5b1u5whrryq X-HE-Tag: 1715196588-15193 X-HE-Meta: U2FsdGVkX18ypckEpArtPu7uREYyh2iditpl9ZDUbC4+K/9nzVpEZNX9OzP2daGFNRMG+95Nfq/CZf0EFg+NzrIWyn/MCIcenf4JPOZ7qcs6HNEcRyR9v0im44ccRoqSiDBTI7oSl5ZOYOiP42Awi7FIN5l/yex8U9FqEGfEPHbZrrUxpgxwG5168GIhKg54VqBWq9YcpvObzyY9ALkodxv8Vfr8bDRgCuAi83k/BwpSJGs7HvpxYpsVcjo0PPEf5x9eSnny+xTwqsCQi/h79k+WKfS5DlHQJiE4tBunkmSa60T+5MxXkzSUb5G6x/jJ04dbi0ig3umrde6LFVuwQaFW6EU3OuXOAN0FBaC8OJkKMfWADTfJCexE6eiSjHUIt4UiFmQMaVhoUPoOuqbbpLU8y7SMT7wv+mMTH9s6X/ELnLJS0JsgNLjBoB58DLZpAz9QXej427IGWq4zB6dg4hC02IrTVxMnREKRJjQQvGYwkA3GMn9LicWAAShBbNd2PwlS8foRe384LnnLoo698Hm/Y3oU5PJvSRopHuL/+J3Rv8bzsZjXj/vMpX/ykDJC3+yNfOlG57C0++IUZ9L1+R+Cv2Z4yC3Un3Ut8QQGrLffKzTKigW5MfwXfklP/0yGY75OghsXYl+rGSPe2Ee+t94C2OS3oYLtJ2rZ07CsDR1tUlqzbhTkCsQJJgdyPO/moVQsP/JG7QJ+YhIOtYzyQdApRQGNe42sN8DFJfiIiNJr4xpV1Zb50Cw2FAnCOVkPG83DO8ll1np4m+W1peTRXR6wkjeiZO3Kwg+Ob0QxYjqe1k53Xrm1hYn6dEhzP3VyryPWeriAEFTc1qIY5cNZwb16XodGv0xIBtWcyMe6tXQb95j0+iQwbKDoVoIt+iL1eZteQq/OdCxCcCr67xnTNs0dumal7md/xblSXgsF2yeEUjhR4yDR+luJnfCVzPahPMzcDbENNtqAfJ7RDcN ILb2QRGL BnSt7Om+mdOuC0A7DOqbO7aXS+9X3u5LzhwzxaxnAo1ZaEC6cK9pNkRShwOUTNtoWUi1RrPhtZcZ/UyJY38XN2g0+PhDHSgFvTUJcKgjvQmldMBRZUEqqBMEzJwAaqvzeCrLRGMGtTfF9uZUV1PYGGKdoC0sJl+rk1eXk1bnVE3RR2/+bwkqAN1y9HlJulUnhwB2fdMhHbpk+ElOEWf63n3W5DXehHiGOKq/c49ATxFwErmjyKvquX0sby1SHSWn19mqThbk2AHtrGVME+9THDL7jrOsaalsTcyrfdnxaeIeFBtON+nLSVU0fx5DlbavK2Y4GrWFLpT0jMmYvMZHsVkY645NDp+6aoELBVvNNGyDzOdmFev32bfTHXZBx8Awqt/qC0IgQPYUgfMfAmfjnM1bX+tCrb9Mb+t3OR3pvc3qx16Ju9e7/17BJ48mLpXQ8LB9GI0hMYasuLW6mcNK9VCsbLtnlWwdZJnedz5ttej0UpJHcr3zB120tC5xT36lcCeUiShlHqb+m0zDo22blI7VilO7yGGZxb3a5mjT9QAO5ekc1K8pcDos1Iw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Make riscv use the contpte aware ptep_set_access_flags() function from arm64. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/pgtable.h | 19 ++-------- arch/arm64/mm/contpte.c | 46 ----------------------- arch/riscv/include/asm/pgtable.h | 10 +++-- include/linux/contpte.h | 3 ++ mm/contpte.c | 63 ++++++++++++++++++++++++++++++++ 5 files changed, 76 insertions(+), 65 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 92c12fb85cb4..6591aab11c67 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1391,9 +1391,6 @@ extern pte_t contpte_get_and_clear_full_ptes(struct mm_struct *mm, unsigned int nr, int full); extern void contpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned int nr); -extern int contpte_ptep_set_access_flags(struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep, - pte_t entry, int dirty); #define pte_batch_hint pte_batch_hint static inline unsigned int pte_batch_hint(pte_t *ptep, pte_t pte) @@ -1512,19 +1509,9 @@ static inline void ptep_set_wrprotect(struct mm_struct *mm, } #define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS -static inline int ptep_set_access_flags(struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep, - pte_t entry, int dirty) -{ - pte_t orig_pte = __ptep_get(ptep); - - entry = pte_mknoncont(entry); - - if (likely(!pte_valid_cont(orig_pte))) - return __ptep_set_access_flags(vma, addr, ptep, entry, dirty); - - return contpte_ptep_set_access_flags(vma, addr, ptep, entry, dirty); -} +extern int ptep_set_access_flags(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + pte_t entry, int dirty); #else /* CONFIG_THP_CONTPTE */ diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c index 16940511943c..5675a61452ac 100644 --- a/arch/arm64/mm/contpte.c +++ b/arch/arm64/mm/contpte.c @@ -62,49 +62,3 @@ void contpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, __wrprotect_ptes(mm, addr, ptep, nr); } EXPORT_SYMBOL_GPL(contpte_wrprotect_ptes); - -int contpte_ptep_set_access_flags(struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep, - pte_t entry, int dirty) -{ - unsigned long start_addr; - pte_t orig_pte; - int i; - - /* - * Gather the access/dirty bits for the contiguous range. If nothing has - * changed, its a noop. - */ - orig_pte = pte_mknoncont(ptep_get(ptep)); - if (pte_val(orig_pte) == pte_val(entry)) - return 0; - - /* - * We can fix up access/dirty bits without having to unfold the contig - * range. But if the write bit is changing, we must unfold. - */ - if (pte_write(orig_pte) == pte_write(entry)) { - /* - * For HW access management, we technically only need to update - * the flag on a single pte in the range. But for SW access - * management, we need to update all the ptes to prevent extra - * faults. Avoid per-page tlb flush in __ptep_set_access_flags() - * and instead flush the whole range at the end. - */ - ptep = arch_contpte_align_down(ptep); - start_addr = addr = ALIGN_DOWN(addr, CONT_PTE_SIZE); - - for (i = 0; i < CONT_PTES; i++, ptep++, addr += PAGE_SIZE) - __ptep_set_access_flags(vma, addr, ptep, entry, 0); - - if (dirty) - __flush_tlb_range(vma, start_addr, addr, - PAGE_SIZE, true, 3); - } else { - __contpte_try_unfold(vma->vm_mm, addr, ptep, orig_pte); - __ptep_set_access_flags(vma, addr, ptep, entry, dirty); - } - - return 1; -} -EXPORT_SYMBOL_GPL(contpte_ptep_set_access_flags); diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 42c7884b8d2e..b151a5aa4de8 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -803,6 +803,10 @@ extern int ptep_test_and_clear_young(struct vm_area_struct *vma, #define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH extern int ptep_clear_flush_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); +#define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS +extern int ptep_set_access_flags(struct vm_area_struct *vma, + unsigned long address, pte_t *ptep, + pte_t entry, int dirty); #else /* CONFIG_THP_CONTPTE */ @@ -816,11 +820,11 @@ extern int ptep_clear_flush_young(struct vm_area_struct *vma, #define ptep_test_and_clear_young __ptep_test_and_clear_young #define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH #define ptep_clear_flush_young __ptep_clear_flush_young +#define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS +#define ptep_set_access_flags __ptep_set_access_flags #endif /* CONFIG_THP_CONTPTE */ -#define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS -#define ptep_set_access_flags __ptep_set_access_flags #define __HAVE_ARCH_PTEP_SET_WRPROTECT #define ptep_set_wrprotect __ptep_set_wrprotect @@ -990,7 +994,7 @@ static inline int pmdp_set_access_flags(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp, pmd_t entry, int dirty) { - return ptep_set_access_flags(vma, address, (pte_t *)pmdp, pmd_pte(entry), dirty); + return __ptep_set_access_flags(vma, address, (pte_t *)pmdp, pmd_pte(entry), dirty); } #define __HAVE_ARCH_PMDP_TEST_AND_CLEAR_YOUNG diff --git a/include/linux/contpte.h b/include/linux/contpte.h index 76a49ac8b6f5..76244b0c678a 100644 --- a/include/linux/contpte.h +++ b/include/linux/contpte.h @@ -23,5 +23,8 @@ int contpte_ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); +int contpte_ptep_set_access_flags(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + pte_t entry, int dirty); #endif /* _LINUX_CONTPTE_H */ diff --git a/mm/contpte.c b/mm/contpte.c index 600277b1196c..9cbbff1f67ad 100644 --- a/mm/contpte.c +++ b/mm/contpte.c @@ -769,4 +769,67 @@ __always_inline int ptep_clear_flush_young(struct vm_area_struct *vma, return contpte_ptep_clear_flush_young(vma, addr, ptep); } + +int contpte_ptep_set_access_flags(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + pte_t entry, int dirty) +{ + unsigned long start_addr; + pte_t orig_pte; + int i; + + /* + * Gather the access/dirty bits for the contiguous range. If nothing has + * changed, its a noop. + */ + orig_pte = pte_mknoncont(ptep_get(ptep)); + if (pte_val(orig_pte) == pte_val(entry)) + return 0; + + /* + * We can fix up access/dirty bits without having to unfold the contig + * range. But if the write bit is changing, we must unfold. + */ + if (pte_write(orig_pte) == pte_write(entry)) { + /* + * For HW access management, we technically only need to update + * the flag on a single pte in the range. But for SW access + * management, we need to update all the ptes to prevent extra + * faults. Avoid per-page tlb flush in __ptep_set_access_flags() + * and instead flush the whole range at the end. + */ + size_t pgsize; + int ncontig; + + ptep = arch_contpte_align_down(ptep); + ncontig = arch_contpte_get_num_contig(vma->vm_mm, addr, ptep, 0, &pgsize); + start_addr = addr = ALIGN_DOWN(addr, ncontig * pgsize); + + for (i = 0; i < ncontig; i++, ptep++, addr += pgsize) + __ptep_set_access_flags(vma, addr, ptep, entry, 0); + + if (dirty) + arch_contpte_flush_tlb_range(vma, start_addr, addr, pgsize); + } else { + __contpte_try_unfold(vma->vm_mm, addr, ptep, orig_pte); + __ptep_set_access_flags(vma, addr, ptep, entry, dirty); + } + + return 1; +} +EXPORT_SYMBOL_GPL(contpte_ptep_set_access_flags); + +__always_inline int ptep_set_access_flags(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + pte_t entry, int dirty) +{ + pte_t orig_pte = __ptep_get(ptep); + + entry = pte_mknoncont(entry); + + if (likely(!pte_valid_cont(orig_pte))) + return __ptep_set_access_flags(vma, addr, ptep, entry, dirty); + + return contpte_ptep_set_access_flags(vma, addr, ptep, entry, dirty); +} #endif /* CONFIG_THP_CONTPTE */