From patchwork Wed May 8 11:18:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13658581 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BC68FC04FFE for ; Wed, 8 May 2024 11:24:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 483826B009B; Wed, 8 May 2024 07:24:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 40C886B0141; Wed, 8 May 2024 07:24:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 289896B0143; Wed, 8 May 2024 07:24:48 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 08C896B009B for ; Wed, 8 May 2024 07:24:48 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 89D80120AAB for ; Wed, 8 May 2024 11:24:47 +0000 (UTC) X-FDA: 82094996214.05.5ABB080 Received: from mail-wm1-f42.google.com (mail-wm1-f42.google.com [209.85.128.42]) by imf06.hostedemail.com (Postfix) with ESMTP id 91285180008 for ; Wed, 8 May 2024 11:24:45 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=gfaCVvmC; spf=pass (imf06.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.42 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1715167485; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=k7TbOelFCBPjD//aDEZ907vn9YCYdQxapVfhTuFqwGo=; b=JIf1ZPwLfUKBlJoXF5mVnchatmdv1FH15B/pK6WxQHh4+ea3DgWuuRuhG1ZIXi+WzuYZu+ MKO5op0Clvs/u4bOvKhO5FQueGSAGWcPcta2Zi0pQorgtvUR1jLmfA1GSMz6a2F+kkqSy8 nHyXtshOv3Y6BMzwRdPvjhKR99RAJYs= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=gfaCVvmC; spf=pass (imf06.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.128.42 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1715167485; a=rsa-sha256; cv=none; b=eDYbeTQiEHNVfTA4hKlPvVylavJoSfbEn9Be4hgvJPvn8GQ4LeNy+tg8e+7RAyN17t2E+c QKGu7qpF7j1L6JVkGeqdg69/KjRulOhfTqveT8OQyiaSErHTcHSj6yMHPifLQ/uu9uRIys UHUjV7yMAgFzzycc1cf/KLB1ihMVI1U= Received: by mail-wm1-f42.google.com with SMTP id 5b1f17b1804b1-41dc9f98e8dso3941755e9.1 for ; Wed, 08 May 2024 04:24:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1715167484; x=1715772284; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=k7TbOelFCBPjD//aDEZ907vn9YCYdQxapVfhTuFqwGo=; b=gfaCVvmCeI+Hxo1OnMsnknIgwvBY/AwEguKZHoeYYrIUiX0QnxbknLFSctJEtJkJ7b f7jMc9m+ORhZpmK8o3hY89HbqmNOotfGKLpcSkzATjf0Qtc0zh5XJg5FyCBl8Nn8p5zC Ptd6mj8UpRA6PmLHZ3aobfjS9XwZbb1k+8bXeGARiPcHlyezmYSyfYPOPPuOuwQ7PqKt xjyZ+X0SXriinZuFV2BghMIDt9BiceZRGqliDTmHPLn2D2SmwfVse4ch+hgm3Evo29E+ SJqEu0LI0vP2TbmlyZUoa/wtjYhPYTqINnr+OaxM8yLccv1NOcjo6Oe1GGQQ51H6nn2q mtbQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715167484; x=1715772284; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=k7TbOelFCBPjD//aDEZ907vn9YCYdQxapVfhTuFqwGo=; b=gb09FbPgKBTvaSYQt7Lso5TFZyqXhNd/REjv6Kj6vd+aY6s9LeZLwTmHBHvwuMwFUf UaoEyFiFYGKMPRZRL8iJ+zxgJKpqDJxBWt5/6R1AVOlpre9LN8IQ0zRHZ/RlPPoW8b3x zyto0cXaFNFd+z4zPg4RM+SJCfvhaRQsHOUEp0KVfVTJG7OUssqPEXP8soRG1EhC9+vn pRgHGTNi7gJOpQ4Q/JcQB+FKC9Mnk/A7HFFJ+1oTUquYCljek7lXWvHg0sMpam/L42DM M10tbfsWqD4WRrDlDLcB+LUTuYNCvW9PG9jCXHsWcVc419eFojOmQ5g7mAy0++PR01Ep hIiw== X-Forwarded-Encrypted: i=1; AJvYcCVcnZ5IHLdW1rYAw6UHtf5OcaFpBSI8I4v9GBBrnqR1vIjTlRqpWnMcemPMF9p9w/zeFhWq8cisn9VCniaixq3Blao= X-Gm-Message-State: AOJu0YyZIaC5srGI02HR/3DlzpXLRtitOWt/CJ+fv//ykKeYROIsL6Dq JBW/yy/YhEYYcpVY7PqcXb4I6SXYBmtEh5vaBX+a9v2f4eb8FQJJwfMypYgUnqI= X-Google-Smtp-Source: AGHT+IGIT5fRmSU5mTHAtfUSvBO8bSYE3S8hCevxI+ZKnPLL6EPNP04SFEUKamUPrL/Drlv8BwdKKg== X-Received: by 2002:a5d:6d52:0:b0:34d:b3a0:5614 with SMTP id ffacd0b85a97d-34fca4326b6mr2190198f8f.8.1715167483962; Wed, 08 May 2024 04:24:43 -0700 (PDT) Received: from localhost.localdomain (amontpellier-656-1-456-62.w92-145.abo.wanadoo.fr. [92.145.124.62]) by smtp.gmail.com with ESMTPSA id dn15-20020a0560000c0f00b0034d9012e74bsm15257565wrb.72.2024.05.08.04.24.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 08 May 2024 04:24:43 -0700 (PDT) From: Alexandre Ghiti To: Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH v2 6/9] mm: Use common huge_ptep_get_and_clear() function for riscv/arm64 Date: Wed, 8 May 2024 13:18:26 +0200 Message-Id: <20240508111829.16891-7-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20240508111829.16891-1-alexghiti@rivosinc.com> References: <20240508111829.16891-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 91285180008 X-Rspamd-Server: rspam06 X-Stat-Signature: bqspgwhf5ae5xndrdsc8e4k3siqz3uss X-HE-Tag: 1715167485-767694 X-HE-Meta: U2FsdGVkX19SuxEHRLmpopSJOmDuxVA7QFJvnXbAo4eD9oS2cjClFc64ojXFQ5s9018MqAoVKxBxRjTWPeJGWJaLDpYoj7UEMi+C0p/rsNV/35h+2Q3lm2Nw3noYX25QxOf8qvZ1/YEtyOepD97mf5edg6zsylhQGTBQdFxoaALanxv2WzczoOCiUNzQTi4SA9461o2TIEBKAOCUuVClRdU2KI3XPbqLDIGk9GEoWr0rGDe+lnEpep/36HLUKE+VwcXQ2H01MEvSELKCUKcY+zFJ1cf9Pcg2mk5ZtP9JsCTTYYohGDP0ztcm+Qxdd0z8hK3eUf2cC9f5WWFZ3/Co0FBqYQtieQxyBcYbvXnc3YdFopfPw0BF85QV0qeaQqQzmWtCW6n0bHUNdQ7e8eIfJWL2DXu/X1d4vN4OSrg7WkfCO7YWNVEO3IPhL1aM14g6WxfxDMTgnm32mQB2tp0xoJ6gI6YVlWr9yT+DJUhurXaGDSAroOVbBzdpNKy1HcR2LD+/OYGaenLZc/79Kcc1NjNMlw7uadXhGVJH1zIFPc3ALDBF+8rK5VJBEF6akhX3tfprERLV6pEWCT1I7Lisn4z4v4sK9FcRNDg0CvTpeHBSlC+P3eAWd+kUWLr0odeZmn54rCcuRfdrQPhynpS9jMjAJxGXWq9cCO4LeSiuvnrOFn+wtwBqpWVpjQ6w008lUNZalkGx1INE1RQ/wxU/qSDJMoXWgmc1SUK5KNcLxuueaZ03xEHWqM8uq5xuPS6+QuCQKblCnejoqIjiS7Pb1MFs+GGwPHAVada/LtzNVMuzaeZSyqtUT27l2IshX6W/BSXBrJckKJm45cHms4XGl1EkI+wDqK0FIwm+WY0a5UFNxLC93ILGmGpqhDhcSkzKR6sdgVZpjHDjXr0DtivQGErkopypz0ktP9CCo/WyTAC9M9s8yyq4n9gvmBzHdo7DbOT7ArSGlaIOg3RcGjU tZPxyQ8f l3jzg4yxj1atAPC6JKfgehysqMIMcZYrhj0iRirjpEABDGLEpgdHVb9IDsxEfQxqDftAviCf7DEAL8vEAAURjzILIQlxcKHrPbV67Y8qP0jAUKiumgwjhugSH+4HEBmAvXgEL+pD8wfZr5UHX5D4iPw024sITOH+IqOCMeURDRtecT+jsT6xQ6E3drToeZ2/2G416xnDQDH3X2DX7dXzox4EFZ2loUvCQSsKFimE19y3TX8LHl6dRG4eEchu4Ax/4B6YEzE8Q59HAwlaw1O8FTMedjJS8f0CUl4Pki4oApfV49fHZ4inB1j1uCii95Hpcs8E0HdoQeKdCGBXZn/OQJy3nqzFyjQICXEVP3MqDxgFMiPBxUbtyUhKGZU1YOGQlD+2nNBWK2EyUTDDje4gaxPhpqNuDR17SeDDep4OV++xit+7RC9JvR7sP+vQmrf+zJsOnBj+LjbcNNVDx2hybahANlOBl7lb17NrFQupReC0tXiwAW8RMidHPFQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: After some adjustments, both architectures have the same implementation so move it to the generic code. Note that get_clear_contig() function is duplicated in the generic and the arm64 code because it is still used by some arm64 functions that will, in the next commits, be moved to the generic code. Once all have been moved, the arm64 version will be removed. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/pgtable.h | 14 +++++++++- arch/arm64/mm/hugetlbpage.c | 19 ++----------- arch/riscv/include/asm/pgtable.h | 4 ++- arch/riscv/mm/hugetlbpage.c | 21 ++------------ mm/contpte.c | 48 ++++++++++++++++++++++++++++++-- 5 files changed, 66 insertions(+), 40 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 2e0415fd5083..7c2938cb70b9 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1644,11 +1644,23 @@ static inline int ptep_set_access_flags(struct vm_area_struct *vma, #endif /* CONFIG_ARM64_CONTPTE */ -static inline int arch_contpte_get_num_contig(pte_t *ptep, unsigned long size, +int find_num_contig(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, size_t *pgsize); + +static inline int arch_contpte_get_num_contig(struct mm_struct *mm, + unsigned long addr, + pte_t *ptep, unsigned long size, size_t *pgsize) { int contig_ptes = 0; + /* + * If the size is not passed, we need to go through the page table to + * find out the number of contiguous ptes. + */ + if (size == 0) + return find_num_contig(mm, addr, ptep, pgsize); + *pgsize = size; switch (size) { diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index cf44837369be..5ace4bf7ce35 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -93,8 +93,8 @@ int pud_huge(pud_t pud) #endif } -static int find_num_contig(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, size_t *pgsize) +int find_num_contig(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, size_t *pgsize) { pgd_t *pgdp = pgd_offset(mm, addr); p4d_t *p4dp; @@ -277,21 +277,6 @@ pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags) return entry; } -pte_t huge_ptep_get_and_clear(struct mm_struct *mm, - unsigned long addr, pte_t *ptep) -{ - int ncontig; - size_t pgsize; - pte_t orig_pte = __ptep_get(ptep); - - if (!pte_cont(orig_pte)) - return __ptep_get_and_clear(mm, addr, ptep); - - ncontig = find_num_contig(mm, addr, ptep, &pgsize); - - return get_clear_contig(mm, addr, ptep, pgsize, ncontig); -} - /* * huge_ptep_set_access_flags will update access flags (dirty, accesssed) * and write permission. diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 0847a7fb8661..d976113a370d 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -532,7 +532,9 @@ static inline void __set_pte_at(struct mm_struct *mm, pte_t *ptep, pte_t pteval) #define PFN_PTE_SHIFT _PAGE_PFN_SHIFT #ifdef CONFIG_RISCV_ISA_SVNAPOT -static inline int arch_contpte_get_num_contig(pte_t *ptep, unsigned long size, +static inline int arch_contpte_get_num_contig(struct mm_struct *mm, + unsigned long addr, + pte_t *ptep, unsigned long size, size_t *pgsize) { unsigned long hugepage_shift; diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c index 437b1df059eb..a757e0b2f090 100644 --- a/arch/riscv/mm/hugetlbpage.c +++ b/arch/riscv/mm/hugetlbpage.c @@ -187,7 +187,7 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, if (!pte_napot(pte)) return ptep_set_access_flags(vma, addr, ptep, pte, dirty); - pte_num = arch_contpte_get_num_contig(ptep, 0, &pgsize); + pte_num = arch_contpte_get_num_contig(vma->vm_mm, addr, ptep, 0, &pgsize); orig_pte = get_clear_contig_flush(mm, addr, ptep, pte_num); @@ -202,21 +202,6 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, return true; } -pte_t huge_ptep_get_and_clear(struct mm_struct *mm, - unsigned long addr, - pte_t *ptep) -{ - pte_t orig_pte = ptep_get(ptep); - int pte_num; - - if (!pte_napot(orig_pte)) - return ptep_get_and_clear(mm, addr, ptep); - - pte_num = arch_contpte_get_num_contig(ptep, 0, NULL); - - return get_clear_contig(mm, addr, ptep, pte_num); -} - void huge_ptep_set_wrprotect(struct mm_struct *mm, unsigned long addr, pte_t *ptep) @@ -231,7 +216,7 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm, return; } - pte_num = arch_contpte_get_num_contig(ptep, 0, &pgsize); + pte_num = arch_contpte_get_num_contig(mm, addr, ptep, 0, &pgsize); orig_pte = get_clear_contig_flush(mm, addr, ptep, pte_num); orig_pte = pte_wrprotect(orig_pte); @@ -249,7 +234,7 @@ pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, if (!pte_napot(pte)) return ptep_clear_flush(vma, addr, ptep); - pte_num = arch_contpte_get_num_contig(ptep, 0, NULL); + pte_num = arch_contpte_get_num_contig(vma->vm_mm, addr, ptep, 0, NULL); return get_clear_contig_flush(vma->vm_mm, addr, ptep, pte_num); } diff --git a/mm/contpte.c b/mm/contpte.c index 22e0de197bd3..68eb1634b922 100644 --- a/mm/contpte.c +++ b/mm/contpte.c @@ -22,6 +22,7 @@ * - huge_ptep_get() * - set_huge_pte_at() * - huge_pte_clear() + * - huge_ptep_get_and_clear() */ pte_t huge_ptep_get(pte_t *ptep) @@ -33,7 +34,7 @@ pte_t huge_ptep_get(pte_t *ptep) if (!pte_present(orig_pte) || !pte_cont(orig_pte)) return orig_pte; - ncontig = arch_contpte_get_num_contig(ptep, + ncontig = arch_contpte_get_num_contig(NULL, 0, ptep, page_size(pte_page(orig_pte)), &pgsize); @@ -87,7 +88,7 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, int i; int ncontig; - ncontig = arch_contpte_get_num_contig(ptep, sz, &pgsize); + ncontig = arch_contpte_get_num_contig(mm, addr, ptep, sz, &pgsize); if (!pte_present(pte)) { for (i = 0; i < ncontig; i++, ptep++, addr += pgsize) @@ -111,8 +112,49 @@ void huge_pte_clear(struct mm_struct *mm, unsigned long addr, int i, ncontig; size_t pgsize; - ncontig = arch_contpte_get_num_contig(ptep, sz, &pgsize); + ncontig = arch_contpte_get_num_contig(mm, addr, ptep, sz, &pgsize); for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) __pte_clear(mm, addr, ptep); } + +static pte_t get_clear_contig(struct mm_struct *mm, + unsigned long addr, + pte_t *ptep, + unsigned long pgsize, + unsigned long ncontig) +{ + pte_t orig_pte = __ptep_get(ptep); + unsigned long i; + + for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) { + pte_t pte = __ptep_get_and_clear(mm, addr, ptep); + + /* + * If HW_AFDBM (arm64) or svadu (riscv) is enabled, then the HW + * could turn on the dirty or accessed bit for any page in the + * set, so check them all. + */ + if (pte_dirty(pte)) + orig_pte = pte_mkdirty(orig_pte); + + if (pte_young(pte)) + orig_pte = pte_mkyoung(orig_pte); + } + return orig_pte; +} + +pte_t huge_ptep_get_and_clear(struct mm_struct *mm, + unsigned long addr, pte_t *ptep) +{ + int ncontig; + size_t pgsize; + pte_t orig_pte = __ptep_get(ptep); + + if (!pte_cont(orig_pte)) + return __ptep_get_and_clear(mm, addr, ptep); + + ncontig = arch_contpte_get_num_contig(mm, addr, ptep, 0, &pgsize); + + return get_clear_contig(mm, addr, ptep, pgsize, ncontig); +}