From patchwork Wed Feb 5 15:09:44 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13961282 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 47569C02192 for ; Wed, 5 Feb 2025 15:10:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CCD696B007B; Wed, 5 Feb 2025 10:10:41 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C7CC16B0089; Wed, 5 Feb 2025 10:10:41 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AF4F36B008A; Wed, 5 Feb 2025 10:10:41 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 918C56B007B for ; Wed, 5 Feb 2025 10:10:41 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 3B582A01D0 for ; Wed, 5 Feb 2025 15:10:41 +0000 (UTC) X-FDA: 83086227882.01.201C121 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf30.hostedemail.com (Postfix) with ESMTP id 7AF6E80012 for ; Wed, 5 Feb 2025 15:10:39 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=none; spf=pass (imf30.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738768239; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zo0qgLaUDpVDO+Osvp+LrXOW2iS80qEjSY2xlQTPbVI=; b=UdoukYe+u3XrJ+v8LDdCFc7IKBxPSBxaOkSJAVQ+3TxD9skjOyiJjijrK0HTzxZ6AFC9wi Q23fzTTm6EwDnoTBZdqH2XKLYBg+uofHpezftM8/JH2omvT73n0h1Py2cHdSycXaeCthgV /glbGFK0KyJt6b058kHLc+TFxmKCpGI= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738768239; a=rsa-sha256; cv=none; b=O/gK2lbpncPfrsMbBBRjSH1+hkYxcGRaw9szRxH/Y4pmzaQICEMkoket9KfxfNEdqr0GcY WR7zvorbQla01i9zFpRFj8Jkxq1tJYVkDwmiKxShjzhL/CtywsGAxShq0O4yqkI2KdRr8o svbpynWeCXMXPH1SK8tCXhfK6Ln9thk= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=none; spf=pass (imf30.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 2DD511515; Wed, 5 Feb 2025 07:11:02 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 11E403F5A1; Wed, 5 Feb 2025 07:10:35 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Muchun Song , Pasha Tatashin , Andrew Morton , Uladzislau Rezki , Christoph Hellwig , Mark Rutland , Ard Biesheuvel , Anshuman Khandual , Dev Jain , Alexandre Ghiti , Steve Capper , Kevin Brodsky Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v1 04/16] arm64: hugetlb: Refine tlb maintenance scope Date: Wed, 5 Feb 2025 15:09:44 +0000 Message-ID: <20250205151003.88959-5-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20250205151003.88959-1-ryan.roberts@arm.com> References: <20250205151003.88959-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Stat-Signature: am7zdzfdf34msrsjfg5aifwtmsjiry4h X-Rspamd-Queue-Id: 7AF6E80012 X-Rspam-User: X-Rspamd-Server: rspam06 X-HE-Tag: 1738768239-741215 X-HE-Meta: U2FsdGVkX1+54yWk52F+71MFVn9NliiM8Iyd0e8idTvc0SLgrOu22wfaQ/+OekWCok/fpbGrOs8pB5oN+mYSKKagVz50z3kgNvaYrtopL4nMQgV50DBZjqZUZtEsoZbuRZotJoO+Q33sWpTMSGMCEW2H2JvgSfGybr3/SLJTgy0oiXAEyM4wucou3xf5JNaXl7MVXKpvvUp0Tw9Jw/7HIzqvnLUs9Cbux24xvfPS2gueJe+I4VBLI6Dl4MTF8f4Z4S174Y61tgOMUmgkpdMVYhVkc+AF6yHyoUqK516mYcWheFij0CJjJiS5qwb+wY5fqCFLapfZqqzlVr4kKBkx+nSPPC2Y3TDoEPy1Mm3tNPKsJmMIiZO9NtLQUq2r6/qNHrl3ragsrwHwjMGBA9XFPL4bhH7LnXAPPdGLWQoHE7CVsDia6LsbkX/7IRVYd26nPc8Cq/i4NkZxXIC9S3cmdT6GMUz6QHzk2hsxZkBMNvwvd/eJdt6A2ylJiDOx8kzbazq5x6pYWRm5iHsnXftgWHb2MI9xDtm0IaZMU3DR/xkD8N8c1AdyqtrSnWb7mtHhJ4ejjhvtwJwC3lbPtErZKyMPs/ERwZ5iCgJpGel0UnA2A2bhHXUOySJAsMZIwy5VPxAcs31WfANMa/NVDCVOjvroUnyakoJuYfDZeVJUBX60sZMm853wOREHh1tJFrVEk+mnmn5LbthNcasSLo+nmrhyvgj6HuzwaRl66j/5P7T7ak8y4X8CEu1lIaEvb6HAmVXlhlgzGV30o5RNiiZAq7L7gyqCgW+dFP4stJ1nVySjUgB2EXugHEUW1HU9RpdcT7Ey/CC8qVchaTbflV6ejDA5UnuWWt2RoLvWDadkdQMc2GFJONXo1yQFfPfshhEc6b3oLbJpa03YdPw5DUkYhY9kejQbGSeUFuSmHlkFp7+u8OAWiBazQ14ZcWvhwD/77mAdUUQgDLQtImr4fRu GgvXyoCj csQUlaphuNO+IIttJ81PTWrxmqOcaV9F39NvyZ3SgkGLVyhkDOyiiPBbQPRmnBqwCNGa/ZEK9hbVCgnc3vgIeGelHPFv1ZYWuDikkCed7NfHFgWojEWdtLYoXsT3si3nC50tPC74GwTov+t2qynjGuyRsvQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When operating on contiguous blocks of ptes (or pmds) for some hugetlb sizes, we must honour break-before-make requirements and clear down the block to invalid state in the pgtable then invalidate the relevant tlb entries before making the pgtable entries valid again. However, the tlb maintenance is currently always done assuming the worst case stride (PAGE_SIZE), last_level (false) and tlb_level (TLBI_TTL_UNKNOWN). We can do much better with the hinting; In reality, we know the stride from the huge_pte pgsize, we are always operating only on the last level, and we always know the tlb_level, again based on pgsize. So let's start providing these hints. Additionally, avoid tlb maintenace in set_huge_pte_at(). Break-before-make is only required if we are transitioning the contiguous pte block from valid -> valid. So let's elide the clear-and-flush ("break") if the pte range was previously invalid. Signed-off-by: Ryan Roberts --- arch/arm64/include/asm/hugetlb.h | 29 +++++++++++++++++++---------- arch/arm64/mm/hugetlbpage.c | 9 ++++++--- 2 files changed, 25 insertions(+), 13 deletions(-) diff --git a/arch/arm64/include/asm/hugetlb.h b/arch/arm64/include/asm/hugetlb.h index 8ab9542d2d22..c38f2944c20d 100644 --- a/arch/arm64/include/asm/hugetlb.h +++ b/arch/arm64/include/asm/hugetlb.h @@ -69,27 +69,36 @@ extern void huge_ptep_modify_prot_commit(struct vm_area_struct *vma, #include -#define __HAVE_ARCH_FLUSH_HUGETLB_TLB_RANGE -static inline void flush_hugetlb_tlb_range(struct vm_area_struct *vma, - unsigned long start, - unsigned long end) +static inline void __flush_hugetlb_tlb_range(struct vm_area_struct *vma, + unsigned long start, + unsigned long end, + unsigned long stride, + bool last_level) { - unsigned long stride = huge_page_size(hstate_vma(vma)); - switch (stride) { case PUD_SIZE: - __flush_tlb_range(vma, start, end, PUD_SIZE, false, 1); + __flush_tlb_range(vma, start, end, PUD_SIZE, last_level, 1); break; case CONT_PMD_SIZE: case PMD_SIZE: - __flush_tlb_range(vma, start, end, PMD_SIZE, false, 2); + __flush_tlb_range(vma, start, end, PMD_SIZE, last_level, 2); break; case CONT_PTE_SIZE: - __flush_tlb_range(vma, start, end, PAGE_SIZE, false, 3); + __flush_tlb_range(vma, start, end, PAGE_SIZE, last_level, 3); break; default: - __flush_tlb_range(vma, start, end, PAGE_SIZE, false, TLBI_TTL_UNKNOWN); + __flush_tlb_range(vma, start, end, PAGE_SIZE, last_level, TLBI_TTL_UNKNOWN); } } +#define __HAVE_ARCH_FLUSH_HUGETLB_TLB_RANGE +static inline void flush_hugetlb_tlb_range(struct vm_area_struct *vma, + unsigned long start, + unsigned long end) +{ + unsigned long stride = huge_page_size(hstate_vma(vma)); + + __flush_hugetlb_tlb_range(vma, start, end, stride, false); +} + #endif /* __ASM_HUGETLB_H */ diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 328eec4bfe55..e870d01d12ea 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -190,8 +190,9 @@ static pte_t get_clear_contig_flush(struct mm_struct *mm, { pte_t orig_pte = get_clear_contig(mm, addr, ptep, pgsize, ncontig); struct vm_area_struct vma = TLB_FLUSH_VMA(mm, 0); + unsigned long end = addr + (pgsize * ncontig); - flush_tlb_range(&vma, addr, addr + (pgsize * ncontig)); + __flush_hugetlb_tlb_range(&vma, addr, end, pgsize, true); return orig_pte; } @@ -216,7 +217,7 @@ static void clear_flush(struct mm_struct *mm, for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) __ptep_get_and_clear(mm, addr, ptep); - flush_tlb_range(&vma, saddr, addr); + __flush_hugetlb_tlb_range(&vma, saddr, addr, pgsize, true); } void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, @@ -245,7 +246,9 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, dpfn = pgsize >> PAGE_SHIFT; hugeprot = pte_pgprot(pte); - clear_flush(mm, addr, ptep, pgsize, ncontig); + /* Only need to "break" if transitioning valid -> valid. */ + if (pte_valid(__ptep_get(ptep))) + clear_flush(mm, addr, ptep, pgsize, ncontig); for (i = 0; i < ncontig; i++, ptep++, addr += pgsize, pfn += dpfn) __set_ptes(mm, addr, ptep, pfn_pte(pfn, hugeprot), 1);