From patchwork Fri Feb 2 08:07:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13542336 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D71E4C47258 for ; Fri, 2 Feb 2024 08:08:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6379F6B0071; Fri, 2 Feb 2024 03:08:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5E8936B0072; Fri, 2 Feb 2024 03:08:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4AF9D6B0074; Fri, 2 Feb 2024 03:08:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 3A7136B0071 for ; Fri, 2 Feb 2024 03:08:23 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id D20FEC0EA0 for ; Fri, 2 Feb 2024 08:08:22 +0000 (UTC) X-FDA: 81746136444.15.FEEA911 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf02.hostedemail.com (Postfix) with ESMTP id 2AB3F80023 for ; Fri, 2 Feb 2024 08:08:20 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf02.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706861301; a=rsa-sha256; cv=none; b=QWLyw+2M5R3pHTugch1jQFkde2V9Jx2awFYMQ8CP+evYbwpXsgPm9aGS1bRxjyhHT+zf3G aDTwOB14RWbn3DfUmNyUQM+GMPSyu/E8y91H+KIz04rwFThZFJAX/Kc+ty9W9C9JJNKkxS zqmTBMTt0JZ/kVczYfhfXGVOrrCwBa0= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf02.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706861301; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dr+JmlJli2R+rlUtzp9TTxxnomA8MeE7u4eqm10nX+k=; b=8jb8/G6sPmP98CJji0+aFA4Rw8Q0SfBTZ9xRSFwgX8qeZWqz/0TIHF2gQeZ19TwB0CpK27 PaXX8Ve1CkjzbH7w/Kip2KJS+fBG1CNGxu9psAU9fdmvbTZhthf1vGLM2IdOLG3GgWxgJY BmnI39DPWnHtV5vNINZP2JIMhwDkPWY= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 85221FEC; Fri, 2 Feb 2024 00:09:02 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 99B143F5A1; Fri, 2 Feb 2024 00:08:16 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v5 01/25] mm: Clarify the spec for set_ptes() Date: Fri, 2 Feb 2024 08:07:32 +0000 Message-Id: <20240202080756.1453939-2-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240202080756.1453939-1-ryan.roberts@arm.com> References: <20240202080756.1453939-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 2AB3F80023 X-Stat-Signature: p135zfub1wsyrhupzffaby7efxn1wgfa X-Rspam-User: X-HE-Tag: 1706861300-537968 X-HE-Meta: U2FsdGVkX19z1mKVJ8FIZuUeFTge1DAusbfoH5bW1NbdHody0bPeqE3NB1UDDNs+QLGWgEtUV2LVi9ASzaXaBJ6JjWxWfMUlaZ9YvgZ5DA3E2WgdEN/c56CwJLdL5IL3QEWEVdmgsclA+tTAgWCIXereayXFgWktCNYaOlFCPEq3zlSpq2YvgVrmofnXqIGedAdA+oIdRaDhKXEKRjrjYCFJTY+sPY6KWfOKDWHQvW9qKkDNc4RXh4oLQK4IiB+g/EteVXgBVgD1iLNUI1LOu6ft4jcJ6H5SxilQWnZdFFTh3I3EGXVjDUyHb2zAHvymiZ6GyPlZGuKHs8FUM5CatIvSqV5JhHpUyP/7tmQvl1mXAIt3Mrwnlam+P/j5NxhBmD3iqmXyAmh+3oheCPTZFhpkvecdeQDlSTDGvFOvxkMbC+nK2Z/NV6eqRQxHyIQwxU4Q7z0HMfkBkbvN/ri3SnCraRRhQP5+wHu9H5s4A38oXKmZ77zVC9Ji5Mg/HcM7gnzYMsnL0zhAT0gWZM13qv9bwdQCbzyqts0G4Tibiwuqs6junmpCzP1m8q1RbqxfLHXdD/SfLc2wupQ5TM7FeR7MTBYk4BfVyiHU25yWlAS7tzq+K/c3eeZtnIWEVLJyyQE9Q4v/iYyzL9fQSz1bZR6PBGwqyDzmDtpvKaU+G/R8sVNzMPqxv2Noyuj1gReBX4+Mr06Jx5cRcq91gqXLCYKDZTmC2Yse4ZYkQlgCDZz2H1pNHMnjOp501HrgWnZPWwKcu9BjJpNNL61WW/sDhsnR9Co6jczG7pNcrdmAoVCNnyd2q5BmR/f8AnExDp6bF2nuTcPUOKhqnh54S1ayBNx+ylRrGT6XbkSmuf2xU9l4YTfhoT6OTr7vejjri+oRhW1L9ibCfsWen6yroEGLjM6qoV0T6Ktf+tvslWEtwQznDCCQO2FykLe3/5kyl72sKH3Cj1wkQ7o9sCYSMIw 5UnL7Uw3 yNPaMzOJn4R9aqryqOxe8/d5jueqcI0VPRLJJBqwTW48Ass+uvoawbLePLqCi4xvWQkUpNrZ53VOuZskhnSWjt5WAbBEd8b+TboeOBVMm9YvfnzsOgY2UQiYbVDGRcH6G6tPE0UWqE1gaBlrvRmlr9f5WqK16raLug+NHdZdoO4jCEvqVPYfkrTY9FbOYawtJOJTeo90fDuQb7W62wY+YKi7zs+MkCRXcCMEjZXOY78iONXiAMgyYLrKuQNiJIQRW3kUvSinefuZKag0vA5hyK2jACZPKfaElHE5CAlGgNYa6L0rT/L2Tfhs/8FA2DcjxHzES X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: set_ptes() spec implies that it can only be used to set a present pte because it interprets the PFN field to increment it. However, set_pte_at() has been implemented on top of set_ptes() since set_ptes() was introduced, and set_pte_at() allows setting a pte to a not-present state. So clarify the spec to state that when nr==1, new state of pte may be present or not present. When nr>1, new state of all ptes must be present. While we are at it, tighten the spec to set requirements around the initial state of ptes; when nr==1 it may be either present or not-present. But when nr>1 all ptes must initially be not-present. All set_ptes() callsites already conform to this requirement. Stating it explicitly is useful because it allows for a simplification to the upcoming arm64 contpte implementation. Signed-off-by: Ryan Roberts Acked-by: David Hildenbrand --- include/linux/pgtable.h | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index f0feae7f89fb..5e7eaf8f2b97 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -229,6 +229,10 @@ static inline pte_t pte_next_pfn(pte_t pte) * @pte: Page table entry for the first page. * @nr: Number of pages to map. * + * When nr==1, initial state of pte may be present or not present, and new state + * may be present or not present. When nr>1, initial state of all ptes must be + * not present, and new state must be present. + * * May be overridden by the architecture, or the architecture can define * set_pte() and PFN_PTE_SHIFT. * From patchwork Fri Feb 2 08:07:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13542337 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 75CC3C47258 for ; Fri, 2 Feb 2024 08:08:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0920D6B0075; Fri, 2 Feb 2024 03:08:27 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0421C6B0078; Fri, 2 Feb 2024 03:08:26 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E737E6B007B; Fri, 2 Feb 2024 03:08:26 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id D8A9D6B0075 for ; Fri, 2 Feb 2024 03:08:26 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 7BFD3A0EFF for ; Fri, 2 Feb 2024 08:08:26 +0000 (UTC) X-FDA: 81746136612.05.635B3D3 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf28.hostedemail.com (Postfix) with ESMTP id DB7C4C0006 for ; Fri, 2 Feb 2024 08:08:24 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf28.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706861305; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=iYGxY8gKE5MXdRT9yHHw6Vo1X8bbi/EsI1w4KtEcUbM=; b=mffPx/uNIaY+vgXLuWYPd2S27vmaaRefivauvbfDSOQVud1j6vYWg3iyg1/PcWo2B+YimL k7/smf1j5CDMCYtbwZitAQTo+bRHNu5kToi9qc1GMD0oTbezO/8EDqu2J3o4TK2EdciPwB GgXcRdlUzU+gVZf7mDy8invDr8ebKKQ= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf28.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706861305; a=rsa-sha256; cv=none; b=4LZBUvuaZyDM6ELIsVcYJZXCDFPr9qyvqSDVcHtjNPGxAZhKMGGFiwu34fPx75GROYQQZ9 8+DlSWucUckR4OqxfL8wsOhnytsyUjxEMTJ+YqvzGsD+Hp+m0h6vVX6ZpA6+/IrNgukYXc Qmpvw+DcyZv/2/pUCc9qWfVi8dGUqnY= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 568E21042; Fri, 2 Feb 2024 00:09:06 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 6AED23F5A1; Fri, 2 Feb 2024 00:08:20 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v5 02/25] mm: thp: Batch-collapse PMD with set_ptes() Date: Fri, 2 Feb 2024 08:07:33 +0000 Message-Id: <20240202080756.1453939-3-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240202080756.1453939-1-ryan.roberts@arm.com> References: <20240202080756.1453939-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: gsoykgewipwyp39sdbidumjp1saek6w4 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: DB7C4C0006 X-HE-Tag: 1706861304-788647 X-HE-Meta: U2FsdGVkX19v1w0UWncnnaq2BMLXXW4NQhrm+Eel2nPhyZNbwxqqeoMsr6zZ9yg4TdjdcjEY0LmZN2QpwXH04F1keRQm1FqhOGLfT2SA/c9nPN5AS3Nvsp+mI2uA1psRsYAwY/EXJ6ol8AIz66wtQxQ8spVNFZCtbLyyer7lRQWo2/SQMwOBlzRpc8Em0VUz5enHMnikgft06qEUiXpCa6kBlJxxjB8jK//nT7PrvxkCMrTlfyNEy5eRisaNWpvcjHcpJE6aUtCf5GuJSbXBlI35gWP/K5Jl5M24OA9NBfI+v14pKXEUNHK02hSIbr+zmZaNsRMSvW/mdlu7k1+1oeQ+q70cy3xleGdn1zJUf/OBrox9C2ZEzbeZs1yl4irzaJsWyIRGMNz5AGnwJwa14cttvEw8QcyIKeuNDk9WV76JpLWJdinxDDgsUfy6jX7dw/wTWuDDRyXyQDUscJKCqH+alFiJl5CMOt0OdrRelGn78qSl/sLA+A+UABkyI7cCWJGjSYH/6SRx9oYa+ny6novI25iLw89lg9lZaPFLBoW6YmdTImEM72F1a3qW0xbaI0BYGIyJu2cuiO/S/2v7DmGTrnoueILh1nvPyXIpKN0o28Prv0qzbrfLvXr6J/hn4kRSEnMZqza9OFyTRBU6ftjIMeiUiww2ZpxnuiwIIckONSKu0ld/1jct5dbAKX08hx6MuJH0tXb/M/+fUSVjQQ4VfFZgbZoGCPVXAx+2jtI2RXYGRudcCQDafK02YlmWNTOkMzGHf8i7pkacNtWtt8FpeiHwn8ku4ketxMEXiw2McW5IRVSDbPzzyj4aUu65cPimDr1A2nr2w+05nF/if+tqEAJh68oAqPofjimf7VnhJ3E4LAC4TXaXbdNJrhaMh5A3N8/nYE9Gz74Hi/X95hqGMZJAZpO1F9tUYU27xioboCrPLXpu/WvitK3YasExOUiVjRpIPXKKdNtMUs8 CrA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Refactor __split_huge_pmd_locked() so that a present PMD can be collapsed to PTEs in a single batch using set_ptes(). This should improve performance a little bit, but the real motivation is to remove the need for the arm64 backend to have to fold the contpte entries. Instead, since the ptes are set as a batch, the contpte blocks can be initially set up pre-folded (once the arm64 contpte support is added in the next few patches). This leads to noticeable performance improvement during split. Acked-by: David Hildenbrand Signed-off-by: Ryan Roberts --- mm/huge_memory.c | 58 +++++++++++++++++++++++++++--------------------- 1 file changed, 33 insertions(+), 25 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 016e20bd813e..14888b15121e 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2579,15 +2579,16 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, pte = pte_offset_map(&_pmd, haddr); VM_BUG_ON(!pte); - for (i = 0, addr = haddr; i < HPAGE_PMD_NR; i++, addr += PAGE_SIZE) { - pte_t entry; - /* - * Note that NUMA hinting access restrictions are not - * transferred to avoid any possibility of altering - * permissions across VMAs. - */ - if (freeze || pmd_migration) { + + /* + * Note that NUMA hinting access restrictions are not transferred to + * avoid any possibility of altering permissions across VMAs. + */ + if (freeze || pmd_migration) { + for (i = 0, addr = haddr; i < HPAGE_PMD_NR; i++, addr += PAGE_SIZE) { + pte_t entry; swp_entry_t swp_entry; + if (write) swp_entry = make_writable_migration_entry( page_to_pfn(page + i)); @@ -2606,25 +2607,32 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, entry = pte_swp_mksoft_dirty(entry); if (uffd_wp) entry = pte_swp_mkuffd_wp(entry); - } else { - entry = mk_pte(page + i, READ_ONCE(vma->vm_page_prot)); - if (write) - entry = pte_mkwrite(entry, vma); - if (!young) - entry = pte_mkold(entry); - /* NOTE: this may set soft-dirty too on some archs */ - if (dirty) - entry = pte_mkdirty(entry); - if (soft_dirty) - entry = pte_mksoft_dirty(entry); - if (uffd_wp) - entry = pte_mkuffd_wp(entry); + + VM_WARN_ON(!pte_none(ptep_get(pte + i))); + set_pte_at(mm, addr, pte + i, entry); } - VM_BUG_ON(!pte_none(ptep_get(pte))); - set_pte_at(mm, addr, pte, entry); - pte++; + } else { + pte_t entry; + + entry = mk_pte(page, READ_ONCE(vma->vm_page_prot)); + if (write) + entry = pte_mkwrite(entry, vma); + if (!young) + entry = pte_mkold(entry); + /* NOTE: this may set soft-dirty too on some archs */ + if (dirty) + entry = pte_mkdirty(entry); + if (soft_dirty) + entry = pte_mksoft_dirty(entry); + if (uffd_wp) + entry = pte_mkuffd_wp(entry); + + for (i = 0; i < HPAGE_PMD_NR; i++) + VM_WARN_ON(!pte_none(ptep_get(pte + i))); + + set_ptes(mm, haddr, pte, entry, HPAGE_PMD_NR); } - pte_unmap(pte - 1); + pte_unmap(pte); if (!pmd_migration) folio_remove_rmap_pmd(folio, page, vma); From patchwork Fri Feb 2 08:07:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13542338 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D9362C47258 for ; Fri, 2 Feb 2024 08:08:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6857E6B007B; Fri, 2 Feb 2024 03:08:32 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6360E6B007D; Fri, 2 Feb 2024 03:08:32 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4FD946B007E; Fri, 2 Feb 2024 03:08:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 3FBF06B007B for ; Fri, 2 Feb 2024 03:08:32 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 0D37A140F2E for ; Fri, 2 Feb 2024 08:08:32 +0000 (UTC) X-FDA: 81746136864.23.F77F58F Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf27.hostedemail.com (Postfix) with ESMTP id 6B28440018 for ; Fri, 2 Feb 2024 08:08:30 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf27.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706861310; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MiB2yK22g/POYHUygPxWvSb+30lGFDNWNkV8pOKxEt0=; b=oZFT4ry1EIw5HziiYHpzbJyKPcu6VapJvtNoTbmNpQjYF1teFwGMMr68x669/9Yiya7osl a78uAeYyn4qoydZSiXFf8BQFoEEGpKIcV1xy2j0LVuGhDUNjvjlK/mnG/DXMALT2flbBgt FarEs3w7ShLbTkhf5K7ucb78OcGuc7w= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf27.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706861310; a=rsa-sha256; cv=none; b=GfmrQISOGyRbanvgx6oiBmYhDzCF2KAMdcrmQnbDOg70gfMPr6GentB+KVNayQO3hIbj4B 5MX9jEgoh37Vyje0yB0k0Q5j3jRn5OjVJTGLUz3/R/Lctg71j4KMFxiNFeh6K/rbKz/ldQ ZLRxByW80FNalUUpnjQX9/v2XCKI4XE= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 2865B169E; Fri, 2 Feb 2024 00:09:10 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 3C7A53F5A1; Fri, 2 Feb 2024 00:08:24 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v5 03/25] mm: Make pte_next_pfn() a wrapper around pte_advance_pfn() Date: Fri, 2 Feb 2024 08:07:34 +0000 Message-Id: <20240202080756.1453939-4-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240202080756.1453939-1-ryan.roberts@arm.com> References: <20240202080756.1453939-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 6B28440018 X-Stat-Signature: fyyi5xds3s4s7icq4exr6jc4cym9c4mm X-Rspam-User: X-HE-Tag: 1706861310-751723 X-HE-Meta: U2FsdGVkX1+M21m1eh+lyBLrLhywq7V8deJQBebESPGh8pNEB8f4+q4Qs1liOIdMRyx3NMF3SNtq3dYIyNfttqgLQ1lAWTo33Xejb8hQGrYkAXf7eGRKeJKTtc0M7JkfoHPMnCI28TOVuoexvgqZowIhdvBKeDc+Yk1u7l6T83OQZUxCROSwK65Ovtr7VyrQ7JlM3hohBzRdgHhtPkzedyzZNSQ+nyav43F57AZvvVIVa2aJLm3jaFUrMuK+b7SCiQ5hbAELu+I0u7NgeIy9sqSut+7Qi8fbOMxYbwhxAAJNhKw8vfTZ8lcJx/dFzVaQpxSRHKUKqKVbEsgiauLki9axI9TxgzhjhtyHdy0QTdElQT0+yeSBfVrFavHcyK7oe/gX6/YWDpejvQ3a55dTsuitzpmNxDVd0KYTop9d0cKCwarhk8AwvkPUFU22whu1e7GKJefCTuUcRT2N8kuDaOf5JKUUAIf0BsnrIiT+/zMJYaRKZkTrxNxsN7ROSi4F0sYtiMzPICIAIviyVBONZyhGBQluwYi6rb/hr1StEagGODSK3nmWArpxGRq+Qg3clV4HEmM+t28fkZ1nJs/KBNOe8yqbEjoIrkqRtYJwc2PngWwV4IzvVMdZFDUhsq1OJCGSinvcw5L6eK5igY1VxIpuTmP9HRjkVw5KQn5Qrgb73Gh8o0YqTDeJ2qw7T56OJrFJIzwGj8BYK/1W9Hd7HvK+QOf8KtOE1Nc8Cm9er3hPxDmCyrO16i+cOoI8dDnSl8Uvq7yOtfTuIfbtdLAHu38qPZalr4b/iiO4ToEINiV1lLe5vrB1Uu896MrBZEQ+SO4ZrKjxZsmc7fz+X/Fyy2JnFuKpBlijzBgf0jTf+sqnHgqgNXewsYOfaaIsj17F1GhUkCldixtIZXVP6BYL6c90czprtFQh+s64r3k/dMs/7i2WyydI4Vv4u95gobh4oD+ffe5kw13kLDJa1PG wf6IyQtg zuVv9+4QBL9D01cuhuvJpHNbXzQm6b/4Xj2nQNkdxfYzEvI2qw4L/qCvEFLHrDFSeCSPLnA/6sm2Jboa/ZY5fCMCFVxry/PcaGFLi6iPzg6bISIyJ+DbgRYxgyIg+NIVp1twu++ckyFxRKE3zQOdvqzFa1+U4xOwgF7OQnhuliY+McrkQGW9msMwPvhKpW52RoV3uW0Nd6pmhvRMn2Gvsyt2nPgSG5I7egmFW77HLIqryQ0FXJWBkaOMw/OX1FnWydKrU/8TOM50aryDn0eyrDoiBXQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The goal is to be able to advance a PTE by an arbitrary number of PFNs. So introduce a new API that takes a nr param. We are going to remove pte_next_pfn() and replace it with pte_advance_pfn(). As a first step, implement pte_next_pfn() as a wrapper around pte_advance_pfn() so that we can incrementally switch the architectures over. Once all arches are moved over, we will change all the core-mm callers to call pte_advance_pfn() directly and remove the wrapper. Signed-off-by: Ryan Roberts --- include/linux/pgtable.h | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 5e7eaf8f2b97..815d92dcb96b 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -214,9 +214,15 @@ static inline int pmd_dirty(pmd_t pmd) #ifndef pte_next_pfn +#ifndef pte_advance_pfn +static inline pte_t pte_advance_pfn(pte_t pte, unsigned long nr) +{ + return __pte(pte_val(pte) + (nr << PFN_PTE_SHIFT)); +} +#endif static inline pte_t pte_next_pfn(pte_t pte) { - return __pte(pte_val(pte) + (1UL << PFN_PTE_SHIFT)); + return pte_advance_pfn(pte, 1); } #endif From patchwork Fri Feb 2 08:07:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13542339 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 00937C47258 for ; Fri, 2 Feb 2024 08:08:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 833A06B007D; Fri, 2 Feb 2024 03:08:34 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7E43E6B007E; Fri, 2 Feb 2024 03:08:34 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 684906B0080; Fri, 2 Feb 2024 03:08:34 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 5A93C6B007D for ; Fri, 2 Feb 2024 03:08:34 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 29F7E1C18BC for ; Fri, 2 Feb 2024 08:08:34 +0000 (UTC) X-FDA: 81746136948.07.C47B2C5 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf22.hostedemail.com (Postfix) with ESMTP id 84F59C0014 for ; Fri, 2 Feb 2024 08:08:32 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=none; spf=pass (imf22.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706861312; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4B3QQ0hn6W/OOMuAWwXMBnWBKPLPbfhbIhE+WOA0G6Q=; b=KrXjflW4tQyxlNVkoXhylmLitdsQQ5X11V54K1ym5vf7Eql6l78CNgg4BvdEPgMgtPudG9 p+MTdYDcE7rPMCzrzikkaBgTOI2Qhp3n4qkEYtmeuFG1CkrYTMFP0wUoDLXP++2S7+iYAN O9Plg8Yp3CGGx7C+0LBAUkU2xQLdTgQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706861312; a=rsa-sha256; cv=none; b=MMPW4RmOHFdaRIhVc9DkzY7Hx9rKWomhq5ILifCPt8xF2UjTmOlYV8eMJdO8xK01A1yTTb eIo64hFuSnuE9/a6ClvRq8TkK1dvNeckV9a4cWJIOFhDfcB4SIaq3ZpUGYruiXyICYNWgM jqlDfkkbrKKOVroSq2PupL+eCoT+vNI= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=none; spf=pass (imf22.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id EDFB71763; Fri, 2 Feb 2024 00:09:13 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 0E4093F5A1; Fri, 2 Feb 2024 00:08:27 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v5 04/25] arm/mm: Convert pte_next_pfn() to pte_advance_pfn() Date: Fri, 2 Feb 2024 08:07:35 +0000 Message-Id: <20240202080756.1453939-5-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240202080756.1453939-1-ryan.roberts@arm.com> References: <20240202080756.1453939-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 84F59C0014 X-Rspam-User: X-Stat-Signature: uqaxkowe8tenudbdhu5ga7apxfc3fa4x X-Rspamd-Server: rspam03 X-HE-Tag: 1706861312-968625 X-HE-Meta: U2FsdGVkX1/0QonBjW9TO2SV3DpwVSCSaNz6+IskCPYleCRZPMX0iqk7M8Ax3QeRq1ey5CE1jloiRRHRzZTrcXy9/VSvwGz+zrf3p4QF4oF8kujx921xjfoN5ExvNyFJHTX+cSrDTo+1yZGB8ecINpwD9pFk8apUe6dsLwYoaPm544cxX5kkFvyLlx4iGnfOlCtmiycPxdnE4+lC31DApmtdPzd2haNVblwFTrL3yApue5lOCN/2VM5b1oKnUyRj4IxRHszLmZ0JPevgo9oGisO0j9rASATJYFndsSlJlDosvLo5erYEzpADn5w62T5RDW8IpXPwIg7UIHEnxYL/invNb9zGJVCeo215ZLDoSPg/WISqoSLoDztWihbSGTd9njhn5+vT3XnaI2OyUR4rP+JgHz4e/Lwk0ENYUBrvvO4cpjMcaj00hciJVGgpKLP3cLHzsk+4iAboaBqiAMITiXBTfjOA0e862b/znuKGABQQaIhDfxJ5TY3eRpdPzJPbjg7XSzFtcHCf4PT8j5G0u2CuDcIFKqKZlPgXhG0z1wo1fqyIP/jaamxSpr4e4XSfqA+lVTBgN2NJonB8uRDA/4O5FC+Pi8qSq+I4BVENbAhZkINVecAk/CbHfAOU8BKm0QY+lkR/3zRaHQPfbPZOCQLsqVI6BUhSiOl25ZvtkK80BTWodUUn14DXBPFEGmSx+ht+CdNH84rIJ6y738PQuQaZIa8Q2cgmmPgx84Avtv9XKPMIPTbi4pQDgF+1wTNC/uN1i9f7yxsPpLSl8aQE1mVg2783DPnR59fimdJPBcYy2vdoxKc6V9EOsfimE7LCLnXlFMlSOzaFRIs+rFLqk3LBwcMdlcIloepiqto/IAUDhO7z9utiKLm6hvYGD2VCsx9z8JhIT4pKuYOe/oc4FW9IjmUCstxutYSGAejzfocq4flssjo2KupDTPSHakYkCa21Qa58/w/nwKdyi8K jt408lc+ E65TZCUHEmKtVg1l+LS5w0n4J42fz0zJjLDaWzDR6LWHqEC0oiS3E/y9nT15Jt08DKc67lyE+mal6XEhAn2dfoXsscp4LGG/E+eOhjmXuEUOpjIHwzszG2PUT2FPdin9Cc3+F0qHnrQhBxbaO3CE66dB2aPM5iDQT9kGzUSB+28WL/v+aB27ITglbkvla1HpvUT+wOrVUiyH11qtN62knwj1DUIAuNi+Yf8Fb8ml3EKTjYUpqYWJuoMQJO/V1W1WNlKAY8oCVPWpwzH703Xl+7oYMtQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Core-mm needs to be able to advance the pfn by an arbitrary amount, so improve the API to do so and change the name. Signed-off-by: Ryan Roberts --- arch/arm/mm/mmu.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/arm/mm/mmu.c b/arch/arm/mm/mmu.c index c24e29c0b9a4..137711c68f2f 100644 --- a/arch/arm/mm/mmu.c +++ b/arch/arm/mm/mmu.c @@ -1814,6 +1814,6 @@ void set_ptes(struct mm_struct *mm, unsigned long addr, if (--nr == 0) break; ptep++; - pteval = pte_next_pfn(pteval); + pteval = pte_advance_pfn(pteval, 1); } } From patchwork Fri Feb 2 08:07:36 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13542340 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 81F14C4828E for ; Fri, 2 Feb 2024 08:08:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1F1D06B0082; Fri, 2 Feb 2024 03:08:38 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1A1D06B0083; Fri, 2 Feb 2024 03:08:38 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 090BD6B0085; Fri, 2 Feb 2024 03:08:38 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id ED10C6B0082 for ; Fri, 2 Feb 2024 03:08:37 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id C6D40140F2E for ; Fri, 2 Feb 2024 08:08:37 +0000 (UTC) X-FDA: 81746137074.11.ED11A0D Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf14.hostedemail.com (Postfix) with ESMTP id 35A06100006 for ; Fri, 2 Feb 2024 08:08:36 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=none; spf=pass (imf14.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706861316; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=AwvDF9Q3mfZgddP7u26CnNByjWvMvI5KLwLV5SoUP/M=; b=M3mqg9cYHpctq++ko7Tvc7Q1FMOxxiUUZBPObCkVquuXkeyHHkS/4c7OPw/JLOwqAF+OKA rdiSTc0hcFwil1jFdhHGTF9IRLr8u6I/HLvOPo2h0iIsvJTucnuyhQMd0T39Dkb45d8xov FaVUGAwB0GmsnvbWSg7hwuTlVeRxI/s= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706861316; a=rsa-sha256; cv=none; b=tKZa4kG6VV7WYMegPFaoj3wkO1trEsrPuGscZ9LxGTIq/3L+g7xIv48We7i58IzMPdTcvV uuxTxhNWN/d5OXW3edSpArN/CfDVdPg7dPQ+S6MKFMuhG7PYvECxLOzmNZBl/I9SwiCWbt ejjiy69Es9AvnyxOtjEYdaRs/DOrHHQ= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=none; spf=pass (imf14.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id BF9921764; Fri, 2 Feb 2024 00:09:17 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id D3F633F5A1; Fri, 2 Feb 2024 00:08:31 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v5 05/25] arm64/mm: Convert pte_next_pfn() to pte_advance_pfn() Date: Fri, 2 Feb 2024 08:07:36 +0000 Message-Id: <20240202080756.1453939-6-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240202080756.1453939-1-ryan.roberts@arm.com> References: <20240202080756.1453939-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 35A06100006 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: tswsfm19nd7sny9poamzryo13rhmgodx X-HE-Tag: 1706861316-330789 X-HE-Meta: U2FsdGVkX1+eYVTNN4P02roWnsnMd/8iFRGdbQGaJSsxBHG34R/4QzjgO73fHZjAM/dAinEcwuWSEVy9e2FVadiXUKXDjdguEFyB2d6VlD5ZaO2ddvxLvWQ8Vv0IZWjp86DHuXr68uDjWIpbSv+/0b79VIu0TNT0X4obwI/I1Z6sMgEF6HsxUMrwtYRFrPcBE6zXoxuwrEHrbDVpFw79TOAwEIaXHw5CNTGT/iazPM0p39XvCK/BBRJwrnyuS3hJBBuAEgDb5kk/x1Eg6IaSU9+xf82FFvAqMzThz9LFLfTd0/27+wjuFubOky+ZYv0lRtavyvPNbQ/Zm9kPbIGo4KtV2TuYMubbkDQZTntK4uR/GUr3FB+YOZxug5Cj74ia55RMIQcsNeYFHDfPaXbR5kUL4pAnJgLQ2qshBXBj+V6Il98E9w4NPwmKSRb4K66uSDx3wQBUVhmGob4qw086dZYrOmDcurKVgeO7h3XjLwQ+YrmGRBZfrBfV1nejMu/pxCzCiYxe772sXV0SFiJk39lblKbnjckxgo70cFF/23qGWn88HahIadWnoeXXTqTDIwkhSyVjvXuaSjKoG4zhCUJtTWumxC7miX7OHVTezGMZBrIxUTTB4p63er1ZMBqabLAW0u/6p/uA8WcXJvbNP+WADX5sxRt4/bDN/VVOGD8SWYGZd9aqN+VY58y0fMYA99nKG5TwlEs6ONh3K4G3jFQUxNqdxEtiXmGt8RrEeOTnWNz6Kckg+yjPsDCGqh0hQBwj3bVq7iAIwIQk6GucqIY9rsGyMcpmfjXA863/TZQyiKVK73QuwACrVDrV8YrAHOppc84l7sWAEdO6amkDo/863bNJu/gGIyIt/NoiRBZ9Q/wTvm701UWUtogBnQVgqh212TMqt5sBgRNll/BUjGRDk+VosPYC5Agn+nKlf7ctiVN+5S8bgQDe9131EFwG4E00MrmIP8paDXLMtKv +r/ExSQ9 dxoRrMrtUCpIqLFkXykFMPm4wkESeZpH2jiOleLumxGvX5k3SXjjwGPqyvLGid6vynF6R3jzrKiwWOeDtuwgObdBg36oChK6ntGOrNMPXxgT/9YvNpW670N4votC3HbD6BYHN46P2y46dKENp/hohmWU6EnoQsdaxWT5E6yuo06tNm0Ir9+oEEcRviGDYzOgqQQcv6w2fPP4Wjb/7/Mb6q8iKpiBRmNezqfsM01MWnF++nc1yuL/0iwwIRBlCieKmmKIyDE2KsAUOR872OdMd+eGmlQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Core-mm needs to be able to advance the pfn by an arbitrary amount, so improve the API to do so and change the name. Signed-off-by: Ryan Roberts --- arch/arm64/include/asm/pgtable.h | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 9428801c1040..6a6cc78cf879 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -351,10 +351,10 @@ static inline pgprot_t pte_pgprot(pte_t pte) return __pgprot(pte_val(pfn_pte(pfn, __pgprot(0))) ^ pte_val(pte)); } -#define pte_next_pfn pte_next_pfn -static inline pte_t pte_next_pfn(pte_t pte) +#define pte_advance_pfn pte_advance_pfn +static inline pte_t pte_advance_pfn(pte_t pte, unsigned long nr) { - return pfn_pte(pte_pfn(pte) + 1, pte_pgprot(pte)); + return pfn_pte(pte_pfn(pte) + nr, pte_pgprot(pte)); } static inline void set_ptes(struct mm_struct *mm, @@ -370,7 +370,7 @@ static inline void set_ptes(struct mm_struct *mm, if (--nr == 0) break; ptep++; - pte = pte_next_pfn(pte); + pte = pte_advance_pfn(pte, 1); } } #define set_ptes set_ptes From patchwork Fri Feb 2 08:07:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13542341 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 65539C4828E for ; Fri, 2 Feb 2024 08:08:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EC3FF6B0085; Fri, 2 Feb 2024 03:08:41 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E72D96B0087; Fri, 2 Feb 2024 03:08:41 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D3B4C6B0088; Fri, 2 Feb 2024 03:08:41 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id C43C96B0085 for ; Fri, 2 Feb 2024 03:08:41 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 834C4A25D7 for ; Fri, 2 Feb 2024 08:08:41 +0000 (UTC) X-FDA: 81746137242.03.AB58172 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf29.hostedemail.com (Postfix) with ESMTP id E6846120016 for ; Fri, 2 Feb 2024 08:08:39 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf29.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706861320; a=rsa-sha256; cv=none; b=WuKOASSbvfuqy5ZjoGdh4+ghWw9eOKf8Kj6MHnRhNSP8qqAry48C4APSZs4NnGA1NzhUgn Pe7udH8pGKe7O/8FUDXNLjI4oiI1j7+/ysSeq7r685jdlH1gpfVDRD3bH3vW6VC8EfbiyB Ios8j4zjovtshV9JVKfdjX4rdyzWYpo= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf29.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706861320; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=52iYv8dj+hH79XZPEMODfoNkUJF+rthPXe736QRckVE=; b=UzYUzwncbxyIMTddjWhDYFrcw2sZWrcsdVvUvVXzvUBFPrg4xKnFBsIakBwmDbmD6lAO8e kZjtAE1dddag1hxaWIw1LctgCAhNkEYlurCrVD86pQdNKaJF5CYXlG7Jw+8dIDtkhHmIW4 ySrL1yN5WG/hBPxHkRyUgo3fsEp+WKQ= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 911E2176A; Fri, 2 Feb 2024 00:09:21 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id A536F3F5A1; Fri, 2 Feb 2024 00:08:35 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v5 06/25] powerpc/mm: Convert pte_next_pfn() to pte_advance_pfn() Date: Fri, 2 Feb 2024 08:07:37 +0000 Message-Id: <20240202080756.1453939-7-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240202080756.1453939-1-ryan.roberts@arm.com> References: <20240202080756.1453939-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: E6846120016 X-Stat-Signature: weqwh4cdjjo9ugs94aw1tukzj7mxgyk9 X-Rspam-User: X-HE-Tag: 1706861319-165724 X-HE-Meta: U2FsdGVkX19ihK8sOK23wW8FEwQg/0CSpJvraIKiPciE6ld9HdgD7JWrUnHpDBdZi4ZqLDva9Nhm3c9erMoeRds12OZkYfvlAIaPZgxnl0E+dYwJbyPvqRrT4OVqYj2+17EtktJCYcC/dLAAJDr45Pzhtx8soY+BjLQQFbQSWg7iUu3xpGg9wWnHU85h4FM7fBX1baUdG47pQdvJ8tUY3vbY+TmTsjWAzUkvK6yoXSCtBNsATuD0R15n8Nsu7ttjz7MBtOV6qh8yL2ct15L6Q2S0p00o2Nfh2jTaTy4voPgChebq0Dho5eP/Bz6Fwz3a/OrgpPIfjHKy+6rcqJF8e7Fc7CxRq8NH4jZmKhKJv9o0PwyO0btClxe1XTUv0yEUwu3Afb2uP9tVtNVkkHRv9CX9/hms7eCo+2a83BSG7uEeVtCehBA9CU8jLPXnTuoCVMMu4dvVcnSgka/XBW/yPVkOsH6au4CCmq3EQjrZYldJC4qvCTHGePHxHK17hhbZJ+LDimU4jawWq5irGJ790PXvtBWRrDfI6mNg6c8SsIPEZg8J1bY5By1KNaxYgNojyaoY8h5Yy+7RLf6yEiuVkGImu4KS9IRyJS92ciXIBs0Vsrr9mnQlpLSVGu/k2Fg5nUBVS+LbUsEKLv05W59HkhBEkcJQySa6st+uSHN0JCXjDJwApPjtdZZGRC1SoqnEyUwgXzCkdZG/o75QEjBg0OSzHnbmNHscPK8nMjRThzIEqVZEJkDrpBMr4802ag/n96TI8+qI9cQeClwHkbSBneIxG+nFbiF03B215E9PNNDkp0qCTHtqJc5L0ifEeHg7Khblu5RWXhiRXwNGB0Gm12fsly7QPubXUY08zQ3jb/Tme+yo9ZWjDP8tarjpTlNRjUdm1YX8pfZh2XkOPkGBKiEu+U0joRzcJ4j9/Fpz1V68frxg5tyPz13UKC+QMzdnmZeXVEvc1XM/gcknd8U oabTIqfK 6E1wGcjlAxnLV3I3S2IvmZ3rqS4hLHYXLT5CBd3cNGR4b1Gx/pRmYhpdQprOTE1dDmHTBHP5ix+KOmSdPMQvqwavtcUC4tenRgdoFcC7iDRc5rpsWPldPynROpAAzlj8rznnz4UAnkvM0jSvWy94vb2K4T3oKP/6ItOvj7uqFWjJVnR7f6pckOaBZuSVGk1QKS3iLSDc13+VV+VdoLrZPFnRYi6fKEdB9DXcg0ARtmoyAgq4dCTO3pLTHcXxuYKethg+Dq5jLPHX3KHogNXyQi3RX9w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Core-mm needs to be able to advance the pfn by an arbitrary amount, so improve the API to do so and change the name. Signed-off-by: Ryan Roberts --- arch/powerpc/mm/pgtable.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/arch/powerpc/mm/pgtable.c b/arch/powerpc/mm/pgtable.c index 549a440ed7f6..6853cdb1290d 100644 --- a/arch/powerpc/mm/pgtable.c +++ b/arch/powerpc/mm/pgtable.c @@ -220,7 +220,7 @@ void set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, break; ptep++; addr += PAGE_SIZE; - pte = pte_next_pfn(pte); + pte = pte_advance_pfn(pte, 1); } } From patchwork Fri Feb 2 08:07:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13542342 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 43B41C4828E for ; Fri, 2 Feb 2024 08:08:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CF7246B0088; Fri, 2 Feb 2024 03:08:45 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C9E406B0089; Fri, 2 Feb 2024 03:08:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B66596B008A; Fri, 2 Feb 2024 03:08:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id A792B6B0088 for ; Fri, 2 Feb 2024 03:08:45 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 7A27A1A0124 for ; Fri, 2 Feb 2024 08:08:45 +0000 (UTC) X-FDA: 81746137410.27.3BC8170 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf13.hostedemail.com (Postfix) with ESMTP id DD06520003 for ; Fri, 2 Feb 2024 08:08:43 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=none; spf=pass (imf13.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706861324; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YPFZqAqLzx2QQ/vmuvlz3bcsLX4NUf8WSSH2bWwChpY=; b=yc557A4QWD2iliU4gu24p/QAJkL1J/FvpsxqmR12CIX5qtrIsRrHwGnZ5lNsgUovHFzVXO u6WGtELaHIgmJWBqT7WdGaJvi/9Q2pi1zaJR5d/wc8iGQ+oE2FN315kxvfRYrAZoIyoGDM 1CbWE3334Cn+7F79OudwTbZYYt+vogE= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706861324; a=rsa-sha256; cv=none; b=bfCrQEnWZ2qxcESg2IsfhBuYUy88cmKG2w07FW9ZOOfQwkcW1QEWbW399ll9mmusqQkFJP eZWSlisbZwcTPGoVGXncREZ2FxcpxVjleYUM08z885njSWDDJtR4Zv1N8RpiZhlV9YxhDy Wx96iw+JM3oY2lqGRY2dlZKZgp+VxG8= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=none; spf=pass (imf13.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6299E176B; Fri, 2 Feb 2024 00:09:25 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 76D9A3F5A1; Fri, 2 Feb 2024 00:08:39 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v5 07/25] x86/mm: Convert pte_next_pfn() to pte_advance_pfn() Date: Fri, 2 Feb 2024 08:07:38 +0000 Message-Id: <20240202080756.1453939-8-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240202080756.1453939-1-ryan.roberts@arm.com> References: <20240202080756.1453939-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: DD06520003 X-Rspam-User: X-Stat-Signature: 1ia4afaotwu95frdjz1msej6c4hyrknp X-Rspamd-Server: rspam03 X-HE-Tag: 1706861323-201774 X-HE-Meta: U2FsdGVkX18s+N8RbTr9zcJ3ilhxJaAHQDApdr09qGYMywvm5toYx3Y2sQA0I9glla+w3ENS2c0caYzbOZvK9i1rSn6KEQ7lVE/9JHui/0DYhWqOoZwpzoqE+sbegO+si2/wZVQx00iD9EAtRLCtMh+TjTGRSf3Mwt6j/0cSEs6ZN/YR4Ke34jOKlNJgOvHP2Y3+ougJ/9aQP+cHWlJfhthewro1t7xwr5J9z9NmvqrDA3VewnHhDkZvu8g0r2WxrkwTjuNZqG+3dQcjbhoG3UPB1lQk89ix4BekU60Kk1eREeCsUp0uAL/kYb/bInhZcI1tgvvzmaha5W7FVmWTTkNQPjxJgXdTdy+7G4eySUrsm0Su0TUQP21wrXkJmghQOtPb4t4w+ta6zZZCexkYm6BRZgfWXDdCQk/2um+cSE+4P9mORIpud8rLyeD5Pue7hc1Gh8BmwcJPb1Z0HYw4WHpLRoIAtAKbJkApWpC9DhMxbQgoGcIKQKncwIjVjeMHPytSYcRHPHfw7nf3EeB3hipuGrlVERu3Sz6spxqzLZHaOuQYPWv/P5Ov2udikS9wQKLAy3btmaatblcqlfE+ea1o0FLTyrM6XdcZs3hYlNSMWbJqHv28EB5qR8/5GMvXxYxergn2QcZ/NCRVrnwlSnVHCROsEuR0sU6X03P2AdcJ+VGoq4FUTXJOIxd8SN0BDGl5PkS4G8+9UqZ7/zG+X0FxgDfLj2P3OBRDu4HGTYc+B2ZjoAd7/UNKGHijMe8pM6eDqAR8+gUyMixtBfJXCygcPyf6TwBtNtssolfvbKQirZuCaCIVJVI8YooGpZuV18KFARwtWMpJiB9ObsXsOOeiJF6p2sYqdlSzhtW3Cj4PhPe0ixhr6kqFWOva4jkMHN8uvKYHJLNM3FR2Rl8eWny0TrKw5hTJrkihgFcjVpsF+MZMrimSpQroLmjPJ+osumhL8ypNG5r6VTQ2eoI ypmCMuwv RI+gjcHzOYDV2t7wrw2B4vopU0NfdvQRzxr9iiE15UW7vyZfHNE275a4f/kAAbTzg4jdRjmZ1srfi8vbCnc00AlSbE/TUTgc1MJrgDoHFWc6JbhBizcxzkqOBMRoMYO1aIudqBcFeaGQW+thuxLKNlZo9+nZ8wVQdSiCWdZ3+4bs7EOZLIJtfmtRSWaIU9ChrKDCEV7WhsvwuCOlRW6KF2quegswfyBj/XhCiHRqSll1ZhjKU77NYzXW6Obux3FslvVmxe5nsk6psL8ifHehSVQvukA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Core-mm needs to be able to advance the pfn by an arbitrary amount, so improve the API to do so and change the name. Signed-off-by: Ryan Roberts --- arch/x86/include/asm/pgtable.h | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/arch/x86/include/asm/pgtable.h b/arch/x86/include/asm/pgtable.h index 9d077bca6a10..b60b0c897b4c 100644 --- a/arch/x86/include/asm/pgtable.h +++ b/arch/x86/include/asm/pgtable.h @@ -956,13 +956,13 @@ static inline int pte_same(pte_t a, pte_t b) return a.pte == b.pte; } -static inline pte_t pte_next_pfn(pte_t pte) +static inline pte_t pte_advance_pfn(pte_t pte, unsigned long nr) { if (__pte_needs_invert(pte_val(pte))) - return __pte(pte_val(pte) - (1UL << PFN_PTE_SHIFT)); - return __pte(pte_val(pte) + (1UL << PFN_PTE_SHIFT)); + return __pte(pte_val(pte) - (nr << PFN_PTE_SHIFT)); + return __pte(pte_val(pte) + (nr << PFN_PTE_SHIFT)); } -#define pte_next_pfn pte_next_pfn +#define pte_advance_pfn pte_advance_pfn static inline int pte_present(pte_t a) { From patchwork Fri Feb 2 08:07:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13542343 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3AA6FC47258 for ; Fri, 2 Feb 2024 08:08:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C522E6B008A; Fri, 2 Feb 2024 03:08:49 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BDB236B008C; Fri, 2 Feb 2024 03:08:49 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AAEA26B0092; Fri, 2 Feb 2024 03:08:49 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 9AE236B008A for ; Fri, 2 Feb 2024 03:08:49 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 3D750405C9 for ; Fri, 2 Feb 2024 08:08:49 +0000 (UTC) X-FDA: 81746137578.25.B06858E Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf17.hostedemail.com (Postfix) with ESMTP id 9327E40016 for ; Fri, 2 Feb 2024 08:08:47 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=none; spf=pass (imf17.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706861327; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=m802SiXxPoDIA4ME1C8BlYO8FuTHIGUVigVkR29YgiU=; b=zhUKWuEx1LB3sLySVQWKuaHOQc/PUqc5tvScUg8jaL86SPvUgpoMG8xCh4K0S3WiWhNazg HxRsEhRyHdmFPBiwUcJz9ytGAAcztXaEYDjzNbleHGsaIE9waEVqOY9Aog3UJ4xdLu95qf nKnp1x05DM16L0N3kLIC4IZSUN1WDso= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706861327; a=rsa-sha256; cv=none; b=6HGLVZ0Yk0Pj81NMQOoXomzywqD1fIYOD5E19uK9gQYK5G+/PAxCdorBoSdhC2TJ8nQPOD knYBAN94EYnguzQxJSqH+MQcWKV84qILK9VcahgvV0uIk1LNqlaG5/Oq95BHTejAniJxtO N1QBkM66P0R2Evw2xULSuVH9pJrHAIU= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=none; spf=pass (imf17.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 34710176C; Fri, 2 Feb 2024 00:09:29 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 489F23F5A1; Fri, 2 Feb 2024 00:08:43 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v5 08/25] mm: Remove pte_next_pfn() and replace with pte_advance_pfn() Date: Fri, 2 Feb 2024 08:07:39 +0000 Message-Id: <20240202080756.1453939-9-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240202080756.1453939-1-ryan.roberts@arm.com> References: <20240202080756.1453939-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 9327E40016 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: q6f8g98163dkxcyi8xmjp37hb86iet4x X-HE-Tag: 1706861327-248857 X-HE-Meta: U2FsdGVkX19vzrQE407omiWkBYSUxzrXOdRqaxnzG9PpgbZJEiis4/BSVMxelugoxW6Hk/IELoNYYEIQdNbit4I/swSoivMMxFwZszkc7kjulewkuQtmPVJ9vyPeQ6d5DBYB2aV2uFfg5LvrTGUC6yoDpcuLa7USuQODvJePelCiwy/mjohc06/G33QuITbJp9o5vLNdWSh+Vs5QZcZiYpGFhqx8rcMczCXqwQ6gnlGL5JHryuF7FXpQz2IyRM8ABoLJ/li0BityQL0QCm5afW09l+8Bq/Xgl6R2L0waEcD2rT20ATBwvJf8mTbvtaNwD3ynPc/bEeKO4Lbjk8d1GIAFcDRpfgQPDf8BZJB0/Q3il8pZP6NoDXSoiBzLXchF0d3AwbnebPq5k513/rhRAozmd8u/JmhvY39L+t87QvddBgLmzLP000yMNnqGxtD2wZeQn3JopLCCf6aC2Lw1KBXtPtket9IbSQ3bFGFFQ5Cq5o/GVBqDSvFuaqO9XCTyq8vbi5+BsFQsTdTZab73yZU6ymAIXZ3kgQpVIrkWRPbH1QckLa/JveREDi7F2H0CPS81TSbv90MLMrq7ebEaZgBl/T77H3JB8dX2+YSwh1Cjf6iUmUAksQvpyD3LO1Co6E1dN2T5eGYXCwmW1mam61J3mrgBOy6btOjJnaNx/j9oK1bysJpNO5jH4zFMz9xzJs+hnE7EBuw+/UEB9zGC97q0waTiakVQF6mrxcnqBColuMMT06O6eNtSmyctMR8dFFcjn9xHH+UovPQcU0XtqbnLFuGbr7wtSyREQsGosLS+mNGeABHkpSMefV8rSdUPz0f3Jf0lJ4CxIoJx+c5gBqoN6uE9UblDU+3JKRvHltNriZ8tanq04q7GRcbJpk4vMScqFHuJhOyC4Mcr4W08PG8HtUv35PLHhm0TDr+PtLl0aRmhObkGxUTXyuORcjpSHck0PA9n/aAMlVhpSZi oD21To/A DdhJRx8nObDjdQee5/NA/cTuFV5ohYHKYfYuQVlqGt3tRxe3oDyf1cRy6i+acGQPy4Vs46ehJC73ytkslqBJynf1m8wumXUFhg8El6hikYFcVv4dJplNmTRJ9aDlm2Xxz1yi2xyFVJpQqbGa6PX+hJTTzbPsynqoS06w8zwOL0U3giWThNPh6DbP1iX9xVhnyONX3npFEIkHtOHR6ue9HBLr+MKg5aSIoZXfTV9lyqQYFmVAiQ6AzSJMyc0pmn2I/eW3UrUvgTxLABAzR2rWCW0BXtA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Now that the architectures are converted over to pte_advance_pfn(), we can remove the pte_next_pfn() wrapper and convert the callers to call pte_advance_pfn(). Signed-off-by: Ryan Roberts --- include/linux/pgtable.h | 9 +-------- mm/memory.c | 4 ++-- 2 files changed, 3 insertions(+), 10 deletions(-) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 815d92dcb96b..50f32cccbd92 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -212,19 +212,12 @@ static inline int pmd_dirty(pmd_t pmd) #define arch_flush_lazy_mmu_mode() do {} while (0) #endif - -#ifndef pte_next_pfn #ifndef pte_advance_pfn static inline pte_t pte_advance_pfn(pte_t pte, unsigned long nr) { return __pte(pte_val(pte) + (nr << PFN_PTE_SHIFT)); } #endif -static inline pte_t pte_next_pfn(pte_t pte) -{ - return pte_advance_pfn(pte, 1); -} -#endif #ifndef set_ptes /** @@ -256,7 +249,7 @@ static inline void set_ptes(struct mm_struct *mm, unsigned long addr, if (--nr == 0) break; ptep++; - pte = pte_next_pfn(pte); + pte = pte_advance_pfn(pte, 1); } arch_leave_lazy_mmu_mode(); } diff --git a/mm/memory.c b/mm/memory.c index 38a010c4d04d..65fbe4f886c1 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -988,7 +988,7 @@ static inline int folio_pte_batch(struct folio *folio, unsigned long addr, { unsigned long folio_end_pfn = folio_pfn(folio) + folio_nr_pages(folio); const pte_t *end_ptep = start_ptep + max_nr; - pte_t expected_pte = __pte_batch_clear_ignored(pte_next_pfn(pte), flags); + pte_t expected_pte = __pte_batch_clear_ignored(pte_advance_pfn(pte, 1), flags); pte_t *ptep = start_ptep + 1; bool writable; @@ -1017,7 +1017,7 @@ static inline int folio_pte_batch(struct folio *folio, unsigned long addr, if (any_writable) *any_writable |= writable; - expected_pte = pte_next_pfn(expected_pte); + expected_pte = pte_advance_pfn(expected_pte, 1); ptep++; } From patchwork Fri Feb 2 08:07:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13542344 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E6527C4828E for ; Fri, 2 Feb 2024 08:08:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 833746B0092; Fri, 2 Feb 2024 03:08:53 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7E3BA6B0093; Fri, 2 Feb 2024 03:08:53 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 636206B0095; Fri, 2 Feb 2024 03:08:53 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 53B9D6B0092 for ; Fri, 2 Feb 2024 03:08:53 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 3026040E94 for ; Fri, 2 Feb 2024 08:08:53 +0000 (UTC) X-FDA: 81746137746.24.B4A5295 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf09.hostedemail.com (Postfix) with ESMTP id 7948F140015 for ; Fri, 2 Feb 2024 08:08:51 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf09.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706861331; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=i0bW7S2cW1fodez/hBNFPu4+tmL6rFgklQoEc6srDy8=; b=bMnXNkML8/8Jf5s4TTZWDkPDcvmCOY2mu4ppHUDLWJopRK3iCGESESYvTj3TXcudfAjSCQ ICDmcu3tZXr+Sz69FyTJEVn2MVFoUA1QeK6Oq0II3BuOIKtZkMe77jbK2t9FizFcRzbm6G jTvFRxEEuovl5DsFkngOMI6a6icLnlY= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf09.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706861331; a=rsa-sha256; cv=none; b=yopEbBiWRIrKD84p8aOXN68fP8E/KSxWzT0WvPPeIPX7AFLoJz2kqRN2OPB97Fj/MpK1RM MwkUMQphjsbE+YrjC8XhN80I+G/I/puE1ikBbxd8OS2K0SXW2HH2jcl8Od9XdUaBqXJ1t3 fpXum/KmmdyfRmniM66WQQTBhV/Em+w= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 0665D19F0; Fri, 2 Feb 2024 00:09:33 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 1AC463F5A1; Fri, 2 Feb 2024 00:08:47 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v5 09/25] arm64/mm: set_pte(): New layer to manage contig bit Date: Fri, 2 Feb 2024 08:07:40 +0000 Message-Id: <20240202080756.1453939-10-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240202080756.1453939-1-ryan.roberts@arm.com> References: <20240202080756.1453939-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 7948F140015 X-Stat-Signature: 7q8b1qihyuwsoc1g9mjo7t6dyhj38iyr X-Rspam-User: X-HE-Tag: 1706861331-508859 X-HE-Meta: U2FsdGVkX19tgiW/WK5TxZ3WZvRtD+9yd2OmY3qWizwxBo+q693oo5ICNSDwrASgCOWexRnxPfOE8FB1GL9qYgN1Izmb7dCpureMhMAJe4M0ZaDZhs9sjwfIh2ioVeZuNSpj0QL64AxA3Eby08sibDSwJsho/rIkBHAFWNeaoKi3EgXgJSHtJeBxZbCFba91p+AK/KH+YvKco8TTSCsT/EHRgUAr1wspaxSv61pP49U2+m5w4XHlkKOT397RIo1FeBf50JqBL7Y2OiQBv/vQWQdzkCVVQf31h/lqI+qr8DIXGUyzaTyL/W/t3z06dnzkUsFmRpjVTgL2yM9AIq5UtD7Gt8SuV2zsBflH76aFUJLX4DXJXdWSH84e6JiDMYd1DZU63XdSaovRN51KGbERORwJRr/qfAMA35WFz/KOjY/zt3Hd0p6NCl0sJFvNtbbnG7OoAfk8WOV3IxaHAznCBjdufISaxOtwB+QkbTETmeEmy6aP92deAe82zUoJJo7JoycZtNkPYEzYKXGuwvWKJJ6D9htCdd0LHgzatA66wBWLHE7b+5nJXIyFKcQf50SVCQCI5SujBekeBrkwUTX8AKoFcSk4zzxoYyhf/OwS5XMIabEAE79sdctAXaMt9nhneSh24eYNErlp8Mc6rOgqu7uce7KJR2BzBSus8AfEa+gNMn0anVUmzDgXUniuPYqQHsqLg1moGFdrYiz11G1GTy5xB+SPvu+Y1HvHFNA1YMFE6ULVKtcYP6muHQ31GDitE7Qva2pshyA0gKde/2xHUUN/VJnUNZ1WftHCvdq+a0AeIcfdSTVlnwiOJXyvKtJH5BjWKusWzyJiOwbV2vq3U+Hf9HivondjNIZhgBl82IUV2J7C36Afp+3RmymLdDA1SeOMz0qLlwYq/jQS5CSGQcp8G1A+/k3EwBZ8DBQjZj9O4lFHEsxLQ18AcMWq+YudhF9yA+r/OdXJVGxQVLz SzDb0eyz suoePzTMd+xiHLW8x5nNCjUmqnBi065xdSXCbIde5axedrKaCWt9jZJYbXDtwJiaO84Log/C8dprKhq6DSXZ48F+EDJu4tUgDKVvHlqIiYWfBYgQnE+GFNus3rPa/blU59Ot8Sw5i+mOsCagJ7eFAycGq/hLhmsOmQMhZyLJNKmrQlT8hbeFbkLRkKqlLPsD9j2MyBJdM6D84GGbBM4hJqXLvX3Mf5Km3x1bwBEjOKwI6YrpWzGnb9xoK8zg896a/RZgsXhKcSfLB60o11ulPNsXpEzPzEKh2wLZGhwYKvjgX70I= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Create a new layer for the in-table PTE manipulation APIs. For now, The existing API is prefixed with double underscore to become the arch-private API and the public API is just a simple wrapper that calls the private API. The public API implementation will subsequently be used to transparently manipulate the contiguous bit where appropriate. But since there are already some contig-aware users (e.g. hugetlb, kernel mapper), we must first ensure those users use the private API directly so that the future contig-bit manipulations in the public API do not interfere with those existing uses. Tested-by: John Hubbard Signed-off-by: Ryan Roberts --- arch/arm64/include/asm/pgtable.h | 11 +++++++---- arch/arm64/kernel/efi.c | 2 +- arch/arm64/mm/fixmap.c | 2 +- arch/arm64/mm/kasan_init.c | 4 ++-- arch/arm64/mm/mmu.c | 2 +- arch/arm64/mm/pageattr.c | 2 +- arch/arm64/mm/trans_pgd.c | 4 ++-- 7 files changed, 15 insertions(+), 12 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 6a6cc78cf879..3cb45e8dbb52 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -93,7 +93,8 @@ static inline pteval_t __phys_to_pte_val(phys_addr_t phys) __pte(__phys_to_pte_val((phys_addr_t)(pfn) << PAGE_SHIFT) | pgprot_val(prot)) #define pte_none(pte) (!pte_val(pte)) -#define pte_clear(mm,addr,ptep) set_pte(ptep, __pte(0)) +#define pte_clear(mm, addr, ptep) \ + __set_pte(ptep, __pte(0)) #define pte_page(pte) (pfn_to_page(pte_pfn(pte))) /* @@ -261,7 +262,7 @@ static inline pte_t pte_mkdevmap(pte_t pte) return set_pte_bit(pte, __pgprot(PTE_DEVMAP | PTE_SPECIAL)); } -static inline void set_pte(pte_t *ptep, pte_t pte) +static inline void __set_pte(pte_t *ptep, pte_t pte) { WRITE_ONCE(*ptep, pte); @@ -366,7 +367,7 @@ static inline void set_ptes(struct mm_struct *mm, for (;;) { __check_safe_pte_update(mm, ptep, pte); - set_pte(ptep, pte); + __set_pte(ptep, pte); if (--nr == 0) break; ptep++; @@ -540,7 +541,7 @@ static inline void __set_pte_at(struct mm_struct *mm, { __sync_cache_and_tags(pte, nr); __check_safe_pte_update(mm, ptep, pte); - set_pte(ptep, pte); + __set_pte(ptep, pte); } static inline void set_pmd_at(struct mm_struct *mm, unsigned long addr, @@ -1138,6 +1139,8 @@ void vmemmap_update_pte(unsigned long addr, pte_t *ptep, pte_t pte); #define vmemmap_update_pte vmemmap_update_pte #endif +#define set_pte __set_pte + #endif /* !__ASSEMBLY__ */ #endif /* __ASM_PGTABLE_H */ diff --git a/arch/arm64/kernel/efi.c b/arch/arm64/kernel/efi.c index 0228001347be..44288a12fc6c 100644 --- a/arch/arm64/kernel/efi.c +++ b/arch/arm64/kernel/efi.c @@ -111,7 +111,7 @@ static int __init set_permissions(pte_t *ptep, unsigned long addr, void *data) pte = set_pte_bit(pte, __pgprot(PTE_PXN)); else if (system_supports_bti_kernel() && spd->has_bti) pte = set_pte_bit(pte, __pgprot(PTE_GP)); - set_pte(ptep, pte); + __set_pte(ptep, pte); return 0; } diff --git a/arch/arm64/mm/fixmap.c b/arch/arm64/mm/fixmap.c index c0a3301203bd..51cd4501816d 100644 --- a/arch/arm64/mm/fixmap.c +++ b/arch/arm64/mm/fixmap.c @@ -121,7 +121,7 @@ void __set_fixmap(enum fixed_addresses idx, ptep = fixmap_pte(addr); if (pgprot_val(flags)) { - set_pte(ptep, pfn_pte(phys >> PAGE_SHIFT, flags)); + __set_pte(ptep, pfn_pte(phys >> PAGE_SHIFT, flags)); } else { pte_clear(&init_mm, addr, ptep); flush_tlb_kernel_range(addr, addr+PAGE_SIZE); diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c index 4c7ad574b946..f659bd98c63f 100644 --- a/arch/arm64/mm/kasan_init.c +++ b/arch/arm64/mm/kasan_init.c @@ -112,7 +112,7 @@ static void __init kasan_pte_populate(pmd_t *pmdp, unsigned long addr, if (!early) memset(__va(page_phys), KASAN_SHADOW_INIT, PAGE_SIZE); next = addr + PAGE_SIZE; - set_pte(ptep, pfn_pte(__phys_to_pfn(page_phys), PAGE_KERNEL)); + __set_pte(ptep, pfn_pte(__phys_to_pfn(page_phys), PAGE_KERNEL)); } while (ptep++, addr = next, addr != end && pte_none(READ_ONCE(*ptep))); } @@ -271,7 +271,7 @@ static void __init kasan_init_shadow(void) * so we should make sure that it maps the zero page read-only. */ for (i = 0; i < PTRS_PER_PTE; i++) - set_pte(&kasan_early_shadow_pte[i], + __set_pte(&kasan_early_shadow_pte[i], pfn_pte(sym_to_pfn(kasan_early_shadow_page), PAGE_KERNEL_RO)); diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index d794b2f4b5a3..7cc1930f0e10 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -175,7 +175,7 @@ static void init_pte(pmd_t *pmdp, unsigned long addr, unsigned long end, do { pte_t old_pte = READ_ONCE(*ptep); - set_pte(ptep, pfn_pte(__phys_to_pfn(phys), prot)); + __set_pte(ptep, pfn_pte(__phys_to_pfn(phys), prot)); /* * After the PTE entry has been populated once, we diff --git a/arch/arm64/mm/pageattr.c b/arch/arm64/mm/pageattr.c index 924843f1f661..a7996d8edf0a 100644 --- a/arch/arm64/mm/pageattr.c +++ b/arch/arm64/mm/pageattr.c @@ -41,7 +41,7 @@ static int change_page_range(pte_t *ptep, unsigned long addr, void *data) pte = clear_pte_bit(pte, cdata->clear_mask); pte = set_pte_bit(pte, cdata->set_mask); - set_pte(ptep, pte); + __set_pte(ptep, pte); return 0; } diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c index 7b14df3c6477..230b607cf881 100644 --- a/arch/arm64/mm/trans_pgd.c +++ b/arch/arm64/mm/trans_pgd.c @@ -41,7 +41,7 @@ static void _copy_pte(pte_t *dst_ptep, pte_t *src_ptep, unsigned long addr) * read only (code, rodata). Clear the RDONLY bit from * the temporary mappings we use during restore. */ - set_pte(dst_ptep, pte_mkwrite_novma(pte)); + __set_pte(dst_ptep, pte_mkwrite_novma(pte)); } else if ((debug_pagealloc_enabled() || is_kfence_address((void *)addr)) && !pte_none(pte)) { /* @@ -55,7 +55,7 @@ static void _copy_pte(pte_t *dst_ptep, pte_t *src_ptep, unsigned long addr) */ BUG_ON(!pfn_valid(pte_pfn(pte))); - set_pte(dst_ptep, pte_mkpresent(pte_mkwrite_novma(pte))); + __set_pte(dst_ptep, pte_mkpresent(pte_mkwrite_novma(pte))); } } From patchwork Fri Feb 2 08:07:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13542345 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CD6FCC47258 for ; Fri, 2 Feb 2024 08:08:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4B0BD6B0095; Fri, 2 Feb 2024 03:08:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4612A6B0096; Fri, 2 Feb 2024 03:08:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 329D86B0098; Fri, 2 Feb 2024 03:08:57 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 22EB56B0095 for ; Fri, 2 Feb 2024 03:08:57 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id EC661120E67 for ; Fri, 2 Feb 2024 08:08:56 +0000 (UTC) X-FDA: 81746137872.25.56C120F Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf28.hostedemail.com (Postfix) with ESMTP id 456DDC0009 for ; Fri, 2 Feb 2024 08:08:55 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf28.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706861335; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5nP/62q17o0sXhInGhbvTK+r1jxCZnzghRQj+6N9kKY=; b=CbRNRNLJBgfTMF4PtkD6JU5+aQeONOy2Wzfu+R2CC89MVc8V5fjbsFx5rt9jrRMuwaPTqE fy4sf//EdH6uDwL00LGbU782VYBwPoj+HHNiJ8mJ9TjeMq10zbJePjVvkPeWPbtDbUTINK XZbb/o0aLhrCB/7WSVu+sNMvJBvLZ+M= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf28.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706861335; a=rsa-sha256; cv=none; b=vXmhDd4XbUVsQWaE4PeQya5ijohaOJpc7AudrVh9lS2a0W4yBLwT4jpCDtkeQHYBiFCf1I 8R5djQsIrygtM0tustHx8LV7aKi+G/hCPTR3ft/IcmGVHa9jwEbUYZxmTRMgA0GzM/LCf5 JNztq6l4ZBoVsyOopQF8Nrnq0DO+oSU= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id CCB8F1A25; Fri, 2 Feb 2024 00:09:36 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id E0BD93F5A1; Fri, 2 Feb 2024 00:08:50 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v5 10/25] arm64/mm: set_ptes()/set_pte_at(): New layer to manage contig bit Date: Fri, 2 Feb 2024 08:07:41 +0000 Message-Id: <20240202080756.1453939-11-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240202080756.1453939-1-ryan.roberts@arm.com> References: <20240202080756.1453939-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: szok1foet9tbogqjkafcb3neymnso38o X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 456DDC0009 X-HE-Tag: 1706861335-644043 X-HE-Meta: U2FsdGVkX1+1LOz1RJy1VMSdE5JmHUSVK2ypBGSGDPfd/KcdlX4XvjnOSN8as5RM3npp1d6saK52B6RJWxLCU941rjyciG0cfjESSjuOIPYtKQtWB8bu9wxm3y7spQPXeusizWBQLxbVBkbrwsfU+oFYqu9C4DEcAo2S3A3ZpkEdyVVqKku4gConcVX3m0IIQXn16q4pwWY4ugPP0uWBNyNJNnmeKKCD9E3iJXk6cZHdgVqoGErHLs693sbiG2Gi79rvf+1/UiKdDFOOjpCKNrP+g5lHBxkNhn7/GUj76VU1uGrmAy9p45O7/bU9YZOhABFJiWLvuTLC9ACmVCLtlEL9LGnBIA2SKD/TREkc68jZvAxksdkNx/+HR+I8Ar4oyEEoow2xnTg/Kq2Hg5NsBa/+42ERKvms5bkyQWVQP8E4dbN7buQ58s17G23L7BL5Je3MGjAvwDNc3rpfrffEhxRARj+KeHp4muzhnzz9cq9q9aUaDssF5J1kj0vOQpLUjzlFoRxwMP9tHt4daAdVHScHpiOYN8OSZWViIrUXTQIQzYeFUOZptrAUfknsFi5sjBxhYzEGFemnZfmU/emrJyQh0JV0kZin6YK+WX1kVWRLsVJ3N6ejS7cuUGdr0gUXy0tf4+PihD/NQNB7S01CVEi23t6/VA3FuxIsuFdt4Mi0bjxVnDhFBKImXvUR8Veypj+0mYudHKHbi+xcaVNYB3iJNZ/iTBDtEYyd8DhQHbb20NtpS1hyFhphYVHd8Py5B/3K0Q4+EE4Ec2E2/m+9KFuiyW4hn4P2L6h/YRdkoV/VbWi2YVU2un/6Mt8xooIhB9nZ16eF6IMVDOKqzjLjAGZqdiJdZ3P+5vyoI03XcIw3S24Z2v2oJO7ITZpCKWFIpWPxXUv+wJqFAfiHFEjmwqJnpl8DqYnxWAEpCR0Eca/jJjb/0IUS/xlvklDu3+IYjgnkFiYm64k3d0BErvL vzmYmCNs bLxAfTjN94coyMwd+tTmeYBpHliLVJWnywj6/DmF1QCRH/u7y1hkB1YTW89QoMtvWTMk9sTYj5prZb4iYNbm0eX5wQK7lXjOhe3i7PF9zqkICLSnbsg1KGGD18QdcHr6Wsa3aZtXoREyNsmub6J/atI0+lLR1hW4wGDhOpAph+bHVa61g4X3Mmr7hIrXbTq0or90zn3S0Qxy01KiqKCiXYdogMCWBW2aAQqaAOY2fgWXajieXUa5kgThibaYBtVgQusiweqWbL1SHFE6f5T/TOp3aLtbSd59W/D8OKHFnHF9rWCA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Create a new layer for the in-table PTE manipulation APIs. For now, The existing API is prefixed with double underscore to become the arch-private API and the public API is just a simple wrapper that calls the private API. The public API implementation will subsequently be used to transparently manipulate the contiguous bit where appropriate. But since there are already some contig-aware users (e.g. hugetlb, kernel mapper), we must first ensure those users use the private API directly so that the future contig-bit manipulations in the public API do not interfere with those existing uses. set_pte_at() is a core macro that forwards to set_ptes() (with nr=1). Instead of creating a __set_pte_at() internal macro, convert all arch users to use set_ptes()/__set_ptes() directly, as appropriate. Callers in hugetlb may benefit from calling __set_ptes() once for their whole range rather than managing their own loop. This is left for future improvement. Tested-by: John Hubbard Signed-off-by: Ryan Roberts --- arch/arm64/include/asm/pgtable.h | 10 +++++----- arch/arm64/kernel/mte.c | 2 +- arch/arm64/kvm/guest.c | 2 +- arch/arm64/mm/fault.c | 2 +- arch/arm64/mm/hugetlbpage.c | 10 +++++----- 5 files changed, 13 insertions(+), 13 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 3cb45e8dbb52..f1fd6c5e3eca 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -358,9 +358,9 @@ static inline pte_t pte_advance_pfn(pte_t pte, unsigned long nr) return pfn_pte(pte_pfn(pte) + nr, pte_pgprot(pte)); } -static inline void set_ptes(struct mm_struct *mm, - unsigned long __always_unused addr, - pte_t *ptep, pte_t pte, unsigned int nr) +static inline void __set_ptes(struct mm_struct *mm, + unsigned long __always_unused addr, + pte_t *ptep, pte_t pte, unsigned int nr) { page_table_check_ptes_set(mm, ptep, pte, nr); __sync_cache_and_tags(pte, nr); @@ -374,7 +374,6 @@ static inline void set_ptes(struct mm_struct *mm, pte = pte_advance_pfn(pte, 1); } } -#define set_ptes set_ptes /* * Huge pte definitions. @@ -1079,7 +1078,7 @@ static inline void arch_swap_restore(swp_entry_t entry, struct folio *folio) #endif /* CONFIG_ARM64_MTE */ /* - * On AArch64, the cache coherency is handled via the set_pte_at() function. + * On AArch64, the cache coherency is handled via the __set_ptes() function. */ static inline void update_mmu_cache_range(struct vm_fault *vmf, struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, @@ -1140,6 +1139,7 @@ void vmemmap_update_pte(unsigned long addr, pte_t *ptep, pte_t pte); #endif #define set_pte __set_pte +#define set_ptes __set_ptes #endif /* !__ASSEMBLY__ */ diff --git a/arch/arm64/kernel/mte.c b/arch/arm64/kernel/mte.c index a41ef3213e1e..dcdcccd40891 100644 --- a/arch/arm64/kernel/mte.c +++ b/arch/arm64/kernel/mte.c @@ -67,7 +67,7 @@ int memcmp_pages(struct page *page1, struct page *page2) /* * If the page content is identical but at least one of the pages is * tagged, return non-zero to avoid KSM merging. If only one of the - * pages is tagged, set_pte_at() may zero or change the tags of the + * pages is tagged, __set_ptes() may zero or change the tags of the * other page via mte_sync_tags(). */ if (page_mte_tagged(page1) || page_mte_tagged(page2)) diff --git a/arch/arm64/kvm/guest.c b/arch/arm64/kvm/guest.c index aaf1d4939739..629145fd3161 100644 --- a/arch/arm64/kvm/guest.c +++ b/arch/arm64/kvm/guest.c @@ -1072,7 +1072,7 @@ int kvm_vm_ioctl_mte_copy_tags(struct kvm *kvm, } else { /* * Only locking to serialise with a concurrent - * set_pte_at() in the VMM but still overriding the + * __set_ptes() in the VMM but still overriding the * tags, hence ignoring the return value. */ try_page_mte_tagging(page); diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 13189322a38f..23d0dfc16686 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -205,7 +205,7 @@ static void show_pte(unsigned long addr) * * It needs to cope with hardware update of the accessed/dirty state by other * agents in the system and can safely skip the __sync_icache_dcache() call as, - * like set_pte_at(), the PTE is never changed from no-exec to exec here. + * like __set_ptes(), the PTE is never changed from no-exec to exec here. * * Returns whether or not the PTE actually changed. */ diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 8116ac599f80..9d7e7315eaa3 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -254,12 +254,12 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, if (!pte_present(pte)) { for (i = 0; i < ncontig; i++, ptep++, addr += pgsize) - set_pte_at(mm, addr, ptep, pte); + __set_ptes(mm, addr, ptep, pte, 1); return; } if (!pte_cont(pte)) { - set_pte_at(mm, addr, ptep, pte); + __set_ptes(mm, addr, ptep, pte, 1); return; } @@ -270,7 +270,7 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, clear_flush(mm, addr, ptep, pgsize, ncontig); for (i = 0; i < ncontig; i++, ptep++, addr += pgsize, pfn += dpfn) - set_pte_at(mm, addr, ptep, pfn_pte(pfn, hugeprot)); + __set_ptes(mm, addr, ptep, pfn_pte(pfn, hugeprot), 1); } pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma, @@ -478,7 +478,7 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, hugeprot = pte_pgprot(pte); for (i = 0; i < ncontig; i++, ptep++, addr += pgsize, pfn += dpfn) - set_pte_at(mm, addr, ptep, pfn_pte(pfn, hugeprot)); + __set_ptes(mm, addr, ptep, pfn_pte(pfn, hugeprot), 1); return 1; } @@ -507,7 +507,7 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm, pfn = pte_pfn(pte); for (i = 0; i < ncontig; i++, ptep++, addr += pgsize, pfn += dpfn) - set_pte_at(mm, addr, ptep, pfn_pte(pfn, hugeprot)); + __set_ptes(mm, addr, ptep, pfn_pte(pfn, hugeprot), 1); } pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, From patchwork Fri Feb 2 08:07:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13542346 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6F8E7C4828E for ; Fri, 2 Feb 2024 08:09:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 06B046B0098; Fri, 2 Feb 2024 03:09:01 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 01B136B0099; Fri, 2 Feb 2024 03:09:00 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DD7616B009A; Fri, 2 Feb 2024 03:09:00 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id CD6E46B0098 for ; Fri, 2 Feb 2024 03:09:00 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id A610F40E96 for ; Fri, 2 Feb 2024 08:09:00 +0000 (UTC) X-FDA: 81746138040.30.7142872 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf07.hostedemail.com (Postfix) with ESMTP id 21B3A40006 for ; Fri, 2 Feb 2024 08:08:58 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf07.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706861339; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=it+CriqSISuFYBxAlX4OlkX2JArsjio+LXEUKYR9urU=; b=vkNX56opphBfGTq/B9QQnRGCsWI58DqkMP4WhW2kokZqdMiSZ9MzNoeKjdSD7Y2fSGv6vf cnmjXY/UUe5kL295ookDjpgfdOofh85Ha+qLv2Wir3Kkvpkwc/Igmv8srYYPX3MrHYJwDm i2WTcAwqznUXFiq7VTCra+i2c2heLBg= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf07.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706861339; a=rsa-sha256; cv=none; b=o6chUhvMu4UO34oe5+ieeItv8QgHg1KWsXh/b1gHmlUlWlyi8BUy89pT89q0Gb861AD9aU +m3AjxHBPkWochsjPfsCvG8xSDJmIjjCxT5duLqmxsaPRmk2gfmEhpaVFH0XTHCkaNniIn lhp9ZYh3lDl8CmFE3arGDkiLPwJOVF0= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 9ED401A2D; Fri, 2 Feb 2024 00:09:40 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id B2CAC3F5A1; Fri, 2 Feb 2024 00:08:54 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v5 11/25] arm64/mm: pte_clear(): New layer to manage contig bit Date: Fri, 2 Feb 2024 08:07:42 +0000 Message-Id: <20240202080756.1453939-12-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240202080756.1453939-1-ryan.roberts@arm.com> References: <20240202080756.1453939-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: xzid7ojh98xrsy4934x6npxfoue9esrh X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 21B3A40006 X-HE-Tag: 1706861338-669181 X-HE-Meta: U2FsdGVkX1+cN1MfBEJuf2CTZMungjqpPADrI2SWx4G0o+3EhTJN9MGRbMoiyOiIJkBwzvw86o9s9jwn2a+4APfM1mLuT2bjiXhfWzTWrJPGEVpi3gH848ucKJQmmF0IbI1wJFAIKt5CQAtuzOrS1H35GcVlzQNzLTECqUp5OlyzCsmJQ1O00KtFh+1WXJL3ahNQ7PUgDnjKPiBRH/SiZqujqT9bm/nmTeIs9EA6hOMOjctCMffpL7zmjHueb/x6RUhLWtS8uV2qfJgXQAqn29KaQa5cOISq6oDUf8DBxfaK/WfbidarXwCzq4maKfTDCcsWlSJo7lD+sox+nqlE4Bes+7m3PK1ePFHPxu3mnp9vRc8ZkE8qBGGSdv5zyAqxZoWCg6s9cKmdtC/wIKWOrmUG32eaHbXvSNWQWm1ERI7ByyeOjgTJDIwCawpdDA1IQ4BjKGRPBtnIsNIPa5mc5NjV2pk5huwbrsFpyKWJkaN2oJDgiy816Wj2leFVs7up/CcP85uBGR4Q4ZkPInhot197PoR4oDETAKIQD7L7jiJipWN2A3huMBIvPfU4tcETe38b8kYAYKsKdyJhysssEpu6vehMLUYU0I2ZuiuFnhVY2E1IIYZXJN05BQNv4Lrh/+MMsZQkqZx9XJcYfLI0bC+iqCEj/tmuTL5QmNPbJD/hKMcKHMKY8C6dyvUdT4J9FNndLGzHp4Fi1XfxoXyQPCjNePAM/xn/UcHLL6v9a0R+6mb+H4onoe4vGDY32BwWAuLzyQI0Khf3B6qohvgFGBEBWNKgUndWxyPaJvurOmrfCtzZ8Cz0AOUOEP8cRD3+eTW87i36O/orfzKpgx2Rot+ncj+0WpG5GkTXWRoCr5GZFEALxTviq31+vNmtnN9gg49DA7ChdW3QhjX01yUA2FbV5nDahwV2GobOw/QlGaQ1zdx7EMy4Bdaut7rMq30tukmkd/YPI3sKmBL6aUe A3g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Create a new layer for the in-table PTE manipulation APIs. For now, The existing API is prefixed with double underscore to become the arch-private API and the public API is just a simple wrapper that calls the private API. The public API implementation will subsequently be used to transparently manipulate the contiguous bit where appropriate. But since there are already some contig-aware users (e.g. hugetlb, kernel mapper), we must first ensure those users use the private API directly so that the future contig-bit manipulations in the public API do not interfere with those existing uses. Tested-by: John Hubbard Signed-off-by: Ryan Roberts --- arch/arm64/include/asm/pgtable.h | 3 ++- arch/arm64/mm/fixmap.c | 2 +- arch/arm64/mm/hugetlbpage.c | 2 +- arch/arm64/mm/mmu.c | 2 +- 4 files changed, 5 insertions(+), 4 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index f1fd6c5e3eca..3b0ff58109c5 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -93,7 +93,7 @@ static inline pteval_t __phys_to_pte_val(phys_addr_t phys) __pte(__phys_to_pte_val((phys_addr_t)(pfn) << PAGE_SHIFT) | pgprot_val(prot)) #define pte_none(pte) (!pte_val(pte)) -#define pte_clear(mm, addr, ptep) \ +#define __pte_clear(mm, addr, ptep) \ __set_pte(ptep, __pte(0)) #define pte_page(pte) (pfn_to_page(pte_pfn(pte))) @@ -1140,6 +1140,7 @@ void vmemmap_update_pte(unsigned long addr, pte_t *ptep, pte_t pte); #define set_pte __set_pte #define set_ptes __set_ptes +#define pte_clear __pte_clear #endif /* !__ASSEMBLY__ */ diff --git a/arch/arm64/mm/fixmap.c b/arch/arm64/mm/fixmap.c index 51cd4501816d..bfc02568805a 100644 --- a/arch/arm64/mm/fixmap.c +++ b/arch/arm64/mm/fixmap.c @@ -123,7 +123,7 @@ void __set_fixmap(enum fixed_addresses idx, if (pgprot_val(flags)) { __set_pte(ptep, pfn_pte(phys >> PAGE_SHIFT, flags)); } else { - pte_clear(&init_mm, addr, ptep); + __pte_clear(&init_mm, addr, ptep); flush_tlb_kernel_range(addr, addr+PAGE_SIZE); } } diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 9d7e7315eaa3..3d73b83cf97f 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -400,7 +400,7 @@ void huge_pte_clear(struct mm_struct *mm, unsigned long addr, ncontig = num_contig_ptes(sz, &pgsize); for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) - pte_clear(mm, addr, ptep); + __pte_clear(mm, addr, ptep); } pte_t huge_ptep_get_and_clear(struct mm_struct *mm, diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 7cc1930f0e10..bcaa5a5d86f8 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -859,7 +859,7 @@ static void unmap_hotplug_pte_range(pmd_t *pmdp, unsigned long addr, continue; WARN_ON(!pte_present(pte)); - pte_clear(&init_mm, addr, ptep); + __pte_clear(&init_mm, addr, ptep); flush_tlb_kernel_range(addr, addr + PAGE_SIZE); if (free_mapped) free_hotplug_page_range(pte_page(pte), From patchwork Fri Feb 2 08:07:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13542347 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 63A8FC4828E for ; Fri, 2 Feb 2024 08:09:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E9C476B009A; Fri, 2 Feb 2024 03:09:04 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E4C3B6B009B; Fri, 2 Feb 2024 03:09:04 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D13ED6B009C; Fri, 2 Feb 2024 03:09:04 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id BF04F6B009A for ; Fri, 2 Feb 2024 03:09:04 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 918C7120E67 for ; Fri, 2 Feb 2024 08:09:04 +0000 (UTC) X-FDA: 81746138208.26.F05990B Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf14.hostedemail.com (Postfix) with ESMTP id CD4A1100009 for ; Fri, 2 Feb 2024 08:09:02 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=none; spf=pass (imf14.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706861343; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ugmxcY8A+ihqkEHAUXwkD3Iz3A7BIhWHSuJUsSWcNFg=; b=C32oAZ5OGaR5D+fTxGECbJ8RPzh7tNcuT3QMJwu1MfFx2wH6jUYhTv0ZfWfuLAsMiiBuNd FVVAIPPRBi9Xr0flpzNnwepbjfEe3LeW4wJBD0+fUh6Gu3QFzY+RJ4Uw69ayokB2Q2w39C KQu6bkqwH+Kv/2nT95CeaG8WzoomqsE= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=none; spf=pass (imf14.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706861343; a=rsa-sha256; cv=none; b=prt2PlMc3oO9HPBIWk+Z2rj7ofO2FvKuvJPXdyrLt8xM4E78cIX4mRrWhsuu5hLkylpa1U 4UgMDdtA2QyR/yqzBslOJdvR+vS8CwfOY9s1hW2IE7efYgbGHfIF+/Dwmq1sCh+I3bblyF P06o2TzY7L9ROSjBth+WP1rBLeXTdOw= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 70FC91A9A; Fri, 2 Feb 2024 00:09:44 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 852FB3F5A1; Fri, 2 Feb 2024 00:08:58 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v5 12/25] arm64/mm: ptep_get_and_clear(): New layer to manage contig bit Date: Fri, 2 Feb 2024 08:07:43 +0000 Message-Id: <20240202080756.1453939-13-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240202080756.1453939-1-ryan.roberts@arm.com> References: <20240202080756.1453939-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: CD4A1100009 X-Rspam-User: X-Stat-Signature: 8z8pyri976iksgzeoxo5k7tmykysmn7s X-Rspamd-Server: rspam01 X-HE-Tag: 1706861342-242968 X-HE-Meta: U2FsdGVkX1/cYfnpvm4eXeeqYnV9oK6LZEfDtxLdPcHR/WMEWYBjnfm2ZHda1dyLmsvVh0UlVf/kV3My/GpgGFz893146kTM93tigDxI/gFqwNs3nguLZHgw5PYqaBhI+HVeIhvQXGxmL3DpxCK/Xjx8jH0WG0LZ7+aWlC0RZbY9j8kcRbSQviWZ+G2bxo0tvpS/msCgptZ5Zx4lLPAqiTa2wYQd65XOfzTU4bD9Tn6d8sTbIb5EoKOaA/6IV3xzIjzb0nRMeuZcsfYU5vZ7chnBVOQSctCt6NWMoevyn/pp/aK3eLXpUqGSKYMzsP+1dyGD78GyhM+eS7DgBKynfbSga3DMz3MXOtcTHVLBl4wNOBy7t2xWHuKnZFk2GUu0X1vnwBGlOizxc0Szpa/gwRrBR7bi/HnWmBadludP4LczwRklnGCMe2MEq+WpbQlOwL0tYJ145u9BHfOk3N3tNGXmeNFJ3K9mlLj5/swpIwcVFQf8aVUj+2za385407ATZpn5dCjcP31f6+SqigzRcLK7547E6rsCNUZs881zgxHxCWNLU3urKtWbVjlD0+DXLOJwz96qlADa+QGjK5bVEBt+PX0FS2nPRcsvhYWU4FNTmgBpbzdLg3Mn5pFkqBQ2R1csdjBDyYGAca/aUC9mMKFBkcd607OOQb3jaKzoQqDfpsr7yxBdhNByi3W3qlxOQ4Gg0l/dxPn2NbxDtRgDhq7/Suh1j84Xi+F0zOT0EaVMMQZQm9YR93NfUqfYLF6uLF/TFrtX2KK61Fe3PxPEc/VQNnzkDfM+8fwZBjtcHwGZm0eUxBOJN0ifL7R6gdGlCCiySEOY/tiU77bAaleBEmZYEuEV0oAnn5ZLyUkxxh1bQYeVQjLwQPP15iks2q/91CwUh/NcDosnRYWapLLPJgO/QhJq8nxB4iRCZbV+zCNx2iNQuSoWn5P/Lbnm7zGMFJ1PB7N7EI7XDLXlEUo QCqTEpA6 3+Yb89zIWiHCI6Wkq8Lnu5qrXbXIGaIlGGu9slJr4ate+Vbg2mUG5c96VxGBfKTFO4X1oNpyYd9bttWquDt/oxAkgK/W68erqetV+Qr2RjgjVLdzCAU8BMf+Ghav5CoRTWYTAXUxaQvZN3hPuZnsyVuR1RINBsUJvzeblIHtpoJjJHr2YY1xy1fEFL/GzVrtPkZqUyDdfyHgixdXb6yKb6/uRwUahdLl0P5QVwyKnwkGAfichBQAQBKCe591By4P8MxoBZ5OvQQY2QEryVCzY/eLZNVnQnoolbMr3P+G3+sytyis= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Create a new layer for the in-table PTE manipulation APIs. For now, The existing API is prefixed with double underscore to become the arch-private API and the public API is just a simple wrapper that calls the private API. The public API implementation will subsequently be used to transparently manipulate the contiguous bit where appropriate. But since there are already some contig-aware users (e.g. hugetlb, kernel mapper), we must first ensure those users use the private API directly so that the future contig-bit manipulations in the public API do not interfere with those existing uses. Tested-by: John Hubbard Signed-off-by: Ryan Roberts --- arch/arm64/include/asm/pgtable.h | 5 +++-- arch/arm64/mm/hugetlbpage.c | 6 +++--- 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 3b0ff58109c5..5f560326116e 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -953,8 +953,7 @@ static inline int pmdp_test_and_clear_young(struct vm_area_struct *vma, } #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ -#define __HAVE_ARCH_PTEP_GET_AND_CLEAR -static inline pte_t ptep_get_and_clear(struct mm_struct *mm, +static inline pte_t __ptep_get_and_clear(struct mm_struct *mm, unsigned long address, pte_t *ptep) { pte_t pte = __pte(xchg_relaxed(&pte_val(*ptep), 0)); @@ -1141,6 +1140,8 @@ void vmemmap_update_pte(unsigned long addr, pte_t *ptep, pte_t pte); #define set_pte __set_pte #define set_ptes __set_ptes #define pte_clear __pte_clear +#define __HAVE_ARCH_PTEP_GET_AND_CLEAR +#define ptep_get_and_clear __ptep_get_and_clear #endif /* !__ASSEMBLY__ */ diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 3d73b83cf97f..7e74e7b67107 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -188,7 +188,7 @@ static pte_t get_clear_contig(struct mm_struct *mm, unsigned long i; for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) { - pte_t pte = ptep_get_and_clear(mm, addr, ptep); + pte_t pte = __ptep_get_and_clear(mm, addr, ptep); /* * If HW_AFDBM is enabled, then the HW could turn on @@ -236,7 +236,7 @@ static void clear_flush(struct mm_struct *mm, unsigned long i, saddr = addr; for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) - ptep_clear(mm, addr, ptep); + __ptep_get_and_clear(mm, addr, ptep); flush_tlb_range(&vma, saddr, addr); } @@ -411,7 +411,7 @@ pte_t huge_ptep_get_and_clear(struct mm_struct *mm, pte_t orig_pte = ptep_get(ptep); if (!pte_cont(orig_pte)) - return ptep_get_and_clear(mm, addr, ptep); + return __ptep_get_and_clear(mm, addr, ptep); ncontig = find_num_contig(mm, addr, ptep, &pgsize); From patchwork Fri Feb 2 08:07:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13542348 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 47DCAC4828E for ; Fri, 2 Feb 2024 08:09:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CC7AE6B009C; Fri, 2 Feb 2024 03:09:08 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C77116B009D; Fri, 2 Feb 2024 03:09:08 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B18BC6B009E; Fri, 2 Feb 2024 03:09:08 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id A05CA6B009C for ; Fri, 2 Feb 2024 03:09:08 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 4A3F11C18E7 for ; Fri, 2 Feb 2024 08:09:08 +0000 (UTC) X-FDA: 81746138376.03.B3F0702 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf06.hostedemail.com (Postfix) with ESMTP id 95320180011 for ; Fri, 2 Feb 2024 08:09:06 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf06.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706861346; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=0Nx3YcaR5GnneSzYgrn5cd9bIJsd379LdaU+eavGkDo=; b=werrBFqIzq56B5ns68cNDIjLUC6VDPL9gLjOdjjTSCrJHz6ao/zeSD/aVdOwQGyUhQXlh/ PE2nt/udQ48afinNlDBWgcd1SCbaDVdqFWFoeB0MO6EUmfRv2ko+QGHHdA2NmEmuGvr9NU TJmuj+Z5jEnlR8cp+LnWSoZ6uLSurog= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf06.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706861346; a=rsa-sha256; cv=none; b=i29NyGvYjEGKrIfwTJZidmvnLxwmfXcnBk1mZoYvXOIZkzy6bYWlfxPujCF68TdXankFgf sotoiAH2uXpKHpe22xKs6qHZ5MBR1Ib1ACDU5tD1sE+Y5GHxcmmNVqiI2Cd6XsVzRFqPVU L/VZ24VAHm4m1607WD1Au5EnQeaJD+g= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 426DA1AC1; Fri, 2 Feb 2024 00:09:48 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 570433F5A1; Fri, 2 Feb 2024 00:09:02 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v5 13/25] arm64/mm: ptep_test_and_clear_young(): New layer to manage contig bit Date: Fri, 2 Feb 2024 08:07:44 +0000 Message-Id: <20240202080756.1453939-14-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240202080756.1453939-1-ryan.roberts@arm.com> References: <20240202080756.1453939-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: 1iqpd9qdm1xde3gjrjkdxyn7fdod3cdg X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 95320180011 X-HE-Tag: 1706861346-64204 X-HE-Meta: U2FsdGVkX1/tfHjSCvkpFXEcTOpf2vhU35SOzo8FV/EpGL/j1ADSXyLJt2cwhk/QYOy0NRfWuWH/1m87lqVlXus7EaVOJRFUIBuAw8C4ggrDv6LZjv/kqD+oT93XOnXoY2Bi3vFNheYbuGmwGSy4PuvQHMwTadD0xn8z2b3hocACayFG4wcNSNDfQCJZ47mGxRc3iPqKdrTo0kbkq2fBO6yOrj60sn1VPg3FbxoiZqH5RUjFaCKJ+bNXbotx5cXaWdfQX5e1pCE28DOoOS7L3omou4b6kF5npFftzJ7eqIjOqteNx/blgEFIDz9M4aRhmoPrgAkMlRXNE3FUK/U+uTHBBtFdRoTGJleLuppAm93w27fFPnOP3Tm+r121Ak5EWPzpVaOX7993xzqptHdnx5CdgVqspc57h5CgRYDxt63crmlM1Q99RRreo2+XI5NKwmPLFy9bMTFYE7osp+ePeerjbqF9049+7wGHm88vwGQ33B6RbgVp+CipJm2geTioJGpUX4hrPLJYs3W/aNbXpHBx+Abc6Zh53kAAoPP4cwuq/hMLDxlFqlzU1SAOPhUEXeyEXWYEZl3HNvmfR/qU16TjxY1zmLgTiGhBUjnp1WgwedeG1KNHdW1fkcQGMgdJU46vgjJfby1yr8Le3cRQ27aCoYUuCtfF20pRsnFtFqS7cIfGOssYr//SnrwbhtrsGtrGxAr+rIfq6jH5Llg9xp+aMOqFl8+CaJfRgNeqvU98uhjf40TpruT7lTBgW/v3KMUXV4qpr1BBN7e9d54+LXll6TQbaI0iwE0EnWjypjQVGS9bSGC0XQyt1qiW8syhUT0JLDgVT18MwQc99Ql0Q8gQzqYWvQtxhE8Vaw153vSfxRMxEJCIo7LQJxTd4FCbe3R4Kkrqtg7XUxrEO/tqF07Tm7FfgzQ7tHOj99Hz6TqOBqDdZIkX4IKUeG6IGci9HJhs1dTa1dy3IuczSfl T0XF8KOR HmKiXr3QjLkMEhnq5yzMsUfZau5uNYEFXA/odaUyca/+iOnoAojOY2khoptDOCTCM/9+vg2IdxLBnKMaMAdqW2mAGfmJFsoKFehnW0xkTqxzkkorPD38QuiG3YcTzHwXFqFHG0GjplF+i2P/phGHcKnfq6lyyKJa794AX+18OFuBq1KeBwg7lgMY5qJ23gNk/M/aD3WRO2Qz/hyh9TQMlPSL10SoIzOkvczH5wXSKuZe6XbsdQW8rc9WlJDbG6xHAhOrj23dlWLWYuPiC3wkf5JB8XVP8sFV7u5a1ktBJcoBrW7s= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Create a new layer for the in-table PTE manipulation APIs. For now, The existing API is prefixed with double underscore to become the arch-private API and the public API is just a simple wrapper that calls the private API. The public API implementation will subsequently be used to transparently manipulate the contiguous bit where appropriate. But since there are already some contig-aware users (e.g. hugetlb, kernel mapper), we must first ensure those users use the private API directly so that the future contig-bit manipulations in the public API do not interfere with those existing uses. Tested-by: John Hubbard Signed-off-by: Ryan Roberts --- arch/arm64/include/asm/pgtable.h | 18 +++++++----------- 1 file changed, 7 insertions(+), 11 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 5f560326116e..77a8b100e1cd 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -899,8 +899,9 @@ static inline bool pud_user_accessible_page(pud_t pud) /* * Atomic pte/pmd modifications. */ -#define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG -static inline int __ptep_test_and_clear_young(pte_t *ptep) +static inline int __ptep_test_and_clear_young(struct vm_area_struct *vma, + unsigned long address, + pte_t *ptep) { pte_t old_pte, pte; @@ -915,18 +916,11 @@ static inline int __ptep_test_and_clear_young(pte_t *ptep) return pte_young(pte); } -static inline int ptep_test_and_clear_young(struct vm_area_struct *vma, - unsigned long address, - pte_t *ptep) -{ - return __ptep_test_and_clear_young(ptep); -} - #define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH static inline int ptep_clear_flush_young(struct vm_area_struct *vma, unsigned long address, pte_t *ptep) { - int young = ptep_test_and_clear_young(vma, address, ptep); + int young = __ptep_test_and_clear_young(vma, address, ptep); if (young) { /* @@ -949,7 +943,7 @@ static inline int pmdp_test_and_clear_young(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp) { - return ptep_test_and_clear_young(vma, address, (pte_t *)pmdp); + return __ptep_test_and_clear_young(vma, address, (pte_t *)pmdp); } #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ @@ -1142,6 +1136,8 @@ void vmemmap_update_pte(unsigned long addr, pte_t *ptep, pte_t pte); #define pte_clear __pte_clear #define __HAVE_ARCH_PTEP_GET_AND_CLEAR #define ptep_get_and_clear __ptep_get_and_clear +#define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG +#define ptep_test_and_clear_young __ptep_test_and_clear_young #endif /* !__ASSEMBLY__ */ From patchwork Fri Feb 2 08:07:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13542349 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CA04DC47258 for ; Fri, 2 Feb 2024 08:09:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5EAC26B009E; Fri, 2 Feb 2024 03:09:12 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 599F06B009F; Fri, 2 Feb 2024 03:09:12 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 414C16B00A0; Fri, 2 Feb 2024 03:09:12 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 311626B009E for ; Fri, 2 Feb 2024 03:09:12 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 0D6EC160F30 for ; Fri, 2 Feb 2024 08:09:12 +0000 (UTC) X-FDA: 81746138544.29.8BAF414 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf11.hostedemail.com (Postfix) with ESMTP id 656B74000C for ; Fri, 2 Feb 2024 08:09:10 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=none; spf=pass (imf11.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706861350; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/BCYDUOM6grDx0BQxG+5TvCOyfppgVqDK+cfK1dp4FA=; b=f5FhTV6PvcezCvo5GD6MndQ8rjrQbNZMlVHOYNOk0Wow52ZcjNBLl8WZLVI0nRLki/wN9U gqtMKsVEuU9uwtiQXUdTwdbdAZWUB/K7o9+qWMCtSxQHkDlPMxeNwqeoOUc7DCNagORC03 xllieERNVTSZeAM+oCLx21LTK5DyBk4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706861350; a=rsa-sha256; cv=none; b=hwa+ODNY9nqyx6DihKxr+TePudFnLWU65BdDNQ5OCDqIqqBsf9BChucVj5do6X5g3V82UU nav3aIxxJgTnykbto64bZM39ba8HgIzZgeW9mIl5vDZKnxhW2f6ajlOGuCwR/px346Bf6S CflDnDV6/gVMKV8QVdo6SdvJ2yqiuuA= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=none; spf=pass (imf11.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 159451A00; Fri, 2 Feb 2024 00:09:52 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 29E883F5A1; Fri, 2 Feb 2024 00:09:06 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v5 14/25] arm64/mm: ptep_clear_flush_young(): New layer to manage contig bit Date: Fri, 2 Feb 2024 08:07:45 +0000 Message-Id: <20240202080756.1453939-15-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240202080756.1453939-1-ryan.roberts@arm.com> References: <20240202080756.1453939-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 656B74000C X-Rspam-User: X-Stat-Signature: bjx63zgj5qghtcgnsuqr4wwjc47zexxb X-Rspamd-Server: rspam03 X-HE-Tag: 1706861350-200017 X-HE-Meta: U2FsdGVkX19ok4KAtyJPdELKawzUuWefpsMcmVOuaP8o4ntVpjbLLuSS5hAZo9TIXC8g+LgPdsGjmNJNDHI92AnkgIosZqUbo9bqqDI6QtAB7mcDjONVHw9YxY1vsiCD4d+NfmjkkGjpO/ojJiN47Z9Lfht0falBlQnizYvk1YYqbfPQKyTtjDU+3910BsBwNcwaH6SCt5Z08zyC3vHir6Zc44QQuiADrHeZzhj41n88c35N9UB0FPcuEI2Im2YewTno65vzf17Hv33iT1gJDEOoXN2V/Pl+VBZExH1Ur0ge+CXUJKO8fyLbVIbWSHRAc5z87ZlO/ezWc/LDxAmalmXmh5B7jhPQtspNJ3tPe66lLpJtbbvkjferldHU8UcFvvHZE/4Kw+tYNi6Z4YX4/7xxsq8E3pUkD4Z1tIYFaZB8GuKmFN601n/HTcXMLUas/BMM45KGEiqYNwU+Ngz4nyAdLlVFrLM41zRhYK8w3cZ9x3zKihTy7gglL1QBtA9oB8XEJ+kBXxQ0QAkA4QhDcaT7OVB1w2Lr3tDY+Q4dmzL+/Vj2Sao/+QupLgMzBP4Ex/IOyGbcBoTOlShXqoMrtOlqSSUcOOFm6d+GMCZr6uYw1U6M0snmi1W0yqQloSFTsD0rhJrdkYGqDiU3okqkzAdbu1v2//ovQN3jXnPhD4CXz+dgEPF0+uhEgl/pq2qVVuWnpdZPdi3l08wTgSVCBxSjAqW34BVCp0b9gukEk8ZjaZ+rEWOhJUaJ87kr6p3rfipjspomRhOsN5vTEIADxbNSUJeTy5fobQpdwShBkCr5GDiDI0ivqzdirbZZmUSAtq/XFjtCzvV/el4ROniaI53UvW9HCy0gF/twobphGw6WcxkP31WMaZqqhzO90SeMsqWbEiX3YUoT2ZzS0RfHezzgU7kdqgPCNIfVE/9ZxKf4KVjgpSkZMfbcL4XZTK34INUMD1eIyLMpXK1u3c6 elw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Create a new layer for the in-table PTE manipulation APIs. For now, The existing API is prefixed with double underscore to become the arch-private API and the public API is just a simple wrapper that calls the private API. The public API implementation will subsequently be used to transparently manipulate the contiguous bit where appropriate. But since there are already some contig-aware users (e.g. hugetlb, kernel mapper), we must first ensure those users use the private API directly so that the future contig-bit manipulations in the public API do not interfere with those existing uses. Tested-by: John Hubbard Signed-off-by: Ryan Roberts --- arch/arm64/include/asm/pgtable.h | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 77a8b100e1cd..2870bc12f288 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -138,7 +138,7 @@ static inline pteval_t __phys_to_pte_val(phys_addr_t phys) * so that we don't erroneously return false for pages that have been * remapped as PROT_NONE but are yet to be flushed from the TLB. * Note that we can't make any assumptions based on the state of the access - * flag, since ptep_clear_flush_young() elides a DSB when invalidating the + * flag, since __ptep_clear_flush_young() elides a DSB when invalidating the * TLB. */ #define pte_accessible(mm, pte) \ @@ -916,8 +916,7 @@ static inline int __ptep_test_and_clear_young(struct vm_area_struct *vma, return pte_young(pte); } -#define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH -static inline int ptep_clear_flush_young(struct vm_area_struct *vma, +static inline int __ptep_clear_flush_young(struct vm_area_struct *vma, unsigned long address, pte_t *ptep) { int young = __ptep_test_and_clear_young(vma, address, ptep); @@ -1138,6 +1137,8 @@ void vmemmap_update_pte(unsigned long addr, pte_t *ptep, pte_t pte); #define ptep_get_and_clear __ptep_get_and_clear #define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG #define ptep_test_and_clear_young __ptep_test_and_clear_young +#define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH +#define ptep_clear_flush_young __ptep_clear_flush_young #endif /* !__ASSEMBLY__ */ From patchwork Fri Feb 2 08:07:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13542350 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AAD33C4828E for ; Fri, 2 Feb 2024 08:09:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3EDE46B00A0; Fri, 2 Feb 2024 03:09:16 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3C4B66B00A1; Fri, 2 Feb 2024 03:09:16 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 28D736B00A2; Fri, 2 Feb 2024 03:09:16 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 16DD46B00A0 for ; Fri, 2 Feb 2024 03:09:16 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id E848CA0B89 for ; Fri, 2 Feb 2024 08:09:15 +0000 (UTC) X-FDA: 81746138670.04.D0F827F Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf24.hostedemail.com (Postfix) with ESMTP id 404EB18001B for ; Fri, 2 Feb 2024 08:09:14 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=none; spf=pass (imf24.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706861354; a=rsa-sha256; cv=none; b=JT0GEKmGTSpJgkWxQQClr98f9sYi1iFUSQ93cPF1Txcml2j9G4ThkpP0tPzEQsJvmbpvtM NKENz4pYd1mMlPbLrNMXm2+tk/fvvD+KnT+RqwDy8fcfo+3svLPwUof4wusrybCuGkfFHK ilZ6V6h4jOPsMggR5+oeS63u4zJa+AU= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=none; spf=pass (imf24.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706861354; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=n5ldPMFAPklGXLjzR2u+4xIeNMfWz8XyKASMmqWuzUc=; b=buG6etnvXAa++GpjKH6Xg6biyR4U76MmpPmNO5fhDe1w06Dm9rMIg+jcUFwSpnwRmx94Xi yRJGD+3bQtWFAf1fLQXgBrUJQrD0Tp/FShFzXRNeQZGFvXw5h6FnrOqcIFknhFDcknwsWz MNpx3EsPe1epOruI7c7lQWmyBkgWLKU= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id DB9861AED; Fri, 2 Feb 2024 00:09:55 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id EFF383F5A1; Fri, 2 Feb 2024 00:09:09 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v5 15/25] arm64/mm: ptep_set_wrprotect(): New layer to manage contig bit Date: Fri, 2 Feb 2024 08:07:46 +0000 Message-Id: <20240202080756.1453939-16-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240202080756.1453939-1-ryan.roberts@arm.com> References: <20240202080756.1453939-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 404EB18001B X-Stat-Signature: g858udrazrbfu4a43ijmtbm17wf18h88 X-HE-Tag: 1706861354-445143 X-HE-Meta: U2FsdGVkX1+fhEG8POSpm2qLJADV5cJIXWhOhf5slnWZm5gWkIqN0vyr1vWEgzPZTLiKSofOkHR7H802d6WiBq4uBmv21IyMiDiiEw9vdhSjCg9gQQm/pMUPQ+BmEb5KYK3k6E0NVIVTwzCZ3q0p8UdzwJh+JCn0SAHCuJOaTKN+mOcMcbQVFStMkvbPXNV9CE6RSNVnEcf3AmzcNLC0qY5XUWvbBGPtw43ddO4BSsWWTXe3etgnQP1cqRfOJ5TABD1bM4hhASMheHn5jf2bJBGImvrM1KNBZMf676VIntoS2djGO2ZxrTcVTrlr1ZqtkgD1buyNM14apc7DbaOHGLJO1+TWPd8KLrXBqQdKl5+9Xacqtjn9++rVaVqwZkavHLdw4TbUECssDq4RutqrSbmJI4UvKC0VP1oYj/tm6QgLPlCisMKMdIrJDGZvZZtXKNCAq5UO+IMzur5iJRWJrTn6+MDfA5vdngYf++bbOObmk+dbMFexR0nb32c/qNCtPnmGcNgvEtEtXXV3nA0JgfXBpMQpWtXHVaGPhgXaa+D69+yAmWHKyvApFtD+qntZF32J1zvfP1XZnDYHDCPQ7b3I+IDny2ISaa4PcZz5txargdJpC57AtZFnNDVQ9OXWaIjBO+4AccjMPB45s8EV4VDgp5GuVokZ3oYpX+bYoOlAeO89gUJUw3C85yw0xko3Vy67nEyvwbeGsTapSKKCeLjJEv2Suodxr/wQcyiwtZ67srW5I3OWAPT+ht9a9pMo+oyu3cRF005xbc/gzF45yvMTPTvKg3n1T9d6Ur89maPS9snztcObzOAaqaISu0lUfTZepBDDNUvdruj6aBlN0f6ejPDdoKVVaNa5bOJLq2H5uT7M9Pxkke5GFBAQM67whdOOKOyRlNM+W4dnBWl9OWsgxmFt1PvsIDw54tYSyOrO6mjFIX/RbaqSmHvSDf3YCRAjbsEafLdUSQe0qtC 3g3dxe4X nQcfDtm/uzlFiGrjh4eTPdCyWXnTytmivuSwlbcHXtw5+NXhNW7CoycsywJnTcRsRxKqmcA9+1VOPL1XgEjXG4dItAbi5nufunABXmm0jOwMOxi2thhvMcTX+xNGsUG0hqjhV55Wy1VL8/rojHao3OYiB//8Hhs8xTgMkOIFkqnab09TsIBSFcR6m2y6RkJd9ftcjc0SfJFsHOF5hVdBFwV6WDVOTq/HzdzIpBeTHoWLN296AlQBe6K9tpUcP/PTkz8QOhoKVaJlgH7x4tWjWbkS6fupiOKK49dNjC2N88iAShdc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Create a new layer for the in-table PTE manipulation APIs. For now, The existing API is prefixed with double underscore to become the arch-private API and the public API is just a simple wrapper that calls the private API. The public API implementation will subsequently be used to transparently manipulate the contiguous bit where appropriate. But since there are already some contig-aware users (e.g. hugetlb, kernel mapper), we must first ensure those users use the private API directly so that the future contig-bit manipulations in the public API do not interfere with those existing uses. Tested-by: John Hubbard Signed-off-by: Ryan Roberts --- arch/arm64/include/asm/pgtable.h | 10 ++++++---- arch/arm64/mm/hugetlbpage.c | 2 +- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 2870bc12f288..4c2d6c483390 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -970,11 +970,11 @@ static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ /* - * ptep_set_wrprotect - mark read-only while trasferring potential hardware + * __ptep_set_wrprotect - mark read-only while trasferring potential hardware * dirty status (PTE_DBM && !PTE_RDONLY) to the software PTE_DIRTY bit. */ -#define __HAVE_ARCH_PTEP_SET_WRPROTECT -static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long address, pte_t *ptep) +static inline void __ptep_set_wrprotect(struct mm_struct *mm, + unsigned long address, pte_t *ptep) { pte_t old_pte, pte; @@ -992,7 +992,7 @@ static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long addres static inline void pmdp_set_wrprotect(struct mm_struct *mm, unsigned long address, pmd_t *pmdp) { - ptep_set_wrprotect(mm, address, (pte_t *)pmdp); + __ptep_set_wrprotect(mm, address, (pte_t *)pmdp); } #define pmdp_establish pmdp_establish @@ -1139,6 +1139,8 @@ void vmemmap_update_pte(unsigned long addr, pte_t *ptep, pte_t pte); #define ptep_test_and_clear_young __ptep_test_and_clear_young #define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH #define ptep_clear_flush_young __ptep_clear_flush_young +#define __HAVE_ARCH_PTEP_SET_WRPROTECT +#define ptep_set_wrprotect __ptep_set_wrprotect #endif /* !__ASSEMBLY__ */ diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 7e74e7b67107..f6612f3e1c07 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -493,7 +493,7 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm, pte_t pte; if (!pte_cont(READ_ONCE(*ptep))) { - ptep_set_wrprotect(mm, addr, ptep); + __ptep_set_wrprotect(mm, addr, ptep); return; } From patchwork Fri Feb 2 08:07:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13542351 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8853FC47258 for ; Fri, 2 Feb 2024 08:09:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 12E956B00A2; Fri, 2 Feb 2024 03:09:20 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0DF396B00A3; Fri, 2 Feb 2024 03:09:20 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EE9146B00A4; Fri, 2 Feb 2024 03:09:19 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id DF83F6B00A2 for ; Fri, 2 Feb 2024 03:09:19 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id BBDFF40E9B for ; Fri, 2 Feb 2024 08:09:19 +0000 (UTC) X-FDA: 81746138838.03.A9FB70E Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf17.hostedemail.com (Postfix) with ESMTP id 1E39140011 for ; Fri, 2 Feb 2024 08:09:17 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=none; spf=pass (imf17.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706861358; a=rsa-sha256; cv=none; b=yTDG+UsEoWJ8MRaIG4to02va+2bInmq3nGsnYYLDZ8hrKrvMnIMEvs1spnReHfz0kHIaU7 O/iFEvjyyNcpDV6O82nOo8suAhgTq7KElLf51vljtgpjrudI1ptUMg397Rh09eFqHB2tfO PsxOXV0NWsQFtCMspk0OiBAe7DZW5oY= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=none; spf=pass (imf17.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706861358; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=WEDowJeJyddXsBri0afqLyUh9g9Stb132m/wGVNOl8Y=; b=yB94f23enf6ei9uztv8nVbOWA6kV0YN+TMX/FCPMzs+o2HnpYvETOz7wmr3pGGqGNB/vFr DWGtI3WtTyxz9zcWHrapFRfWarsBLbfI8DJ/7zlAJgk5418d39aRgrPwDzTdeccPD0IFHF Jef7v4V76e+VAtwl64ZnJy1GQXEhmbU= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id AD2351BA8; Fri, 2 Feb 2024 00:09:59 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id C16503F5A1; Fri, 2 Feb 2024 00:09:13 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v5 16/25] arm64/mm: ptep_set_access_flags(): New layer to manage contig bit Date: Fri, 2 Feb 2024 08:07:47 +0000 Message-Id: <20240202080756.1453939-17-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240202080756.1453939-1-ryan.roberts@arm.com> References: <20240202080756.1453939-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 1E39140011 X-Stat-Signature: dikppewtk6ouptjyjoeomeepauyuyq5m X-HE-Tag: 1706861357-364672 X-HE-Meta: U2FsdGVkX19M6hQJf1b8Tol5UV5XUp/VdJgZiEEsK6lcluGbCfesQj58ToC8ApfLVVIP/6VQP/u9Op4Oxp2K1SlAaqMlUDYNvSBrYKccH6qIMg+HEvzTy41a59RvHV5gBnaV2SH3npGj+MpxTAFMUMoT0haaUBbPUMYJlsEgqvR3Rd2oNINVLVpvrl/dD7AaWoC6tY2UU0Y5a3JLSbklrtrX4bA3zKx0VncJOqY315XmTw+DIQd4u6qq9+iM16KpHVsJAdstZ+0qgwJ/nIeg7UY9/VlGyadQy98eaeP6rRzZZ6SU2ZW6COO9pt8MWa8YGNbsWRFL3c+vzFxvwEelL1rqL9GDhdouRUZpcCU5oqmRvKEVKxA+Bogv3cL07s79O+Y8kVU7nHcuzbSjKYaju9cWbaVAaF31halT9H0yKUP4YMjQzyEVKQKisnFRNRdB3Q+yqbBtiwRLTCNX9jUQJuMmlQGO5Y90f1ny3N1EsBfPDnrmw7TnlxVcqNiF7tKMZ3LGu6TFLIJsJlz43FpqXlBJvk1z6MC5pKEZX3wpyT8tEnQSKVf3nCcsrGrTohPEPS5Wtsbyq1LHpw37x2oFn09fDJN6Hf/VU0qhkablHqdFv6X7C3Bgohw7FF7PXlL9+rbpz3FRDkaLvDQBOYJN05uO/koHiQlaHprFuElhtFU5+WIn2cEHqut36aRUbAJytMsJhuhN/vOZWL+ERmJ0X9ZHtcsI1SYIhTluGpclYJM7BiQflm6LQaZ4o3WOeodfv7Y4h/Uhn1MyMJe2Vo87O1jWKDnm7Ngd3fO3mbUjLfCsAR154EEtv8X2G4eRba0B41X+IeqGgMVl6yRgn7o0flgTYhLLXtLt4EoLhC3cIdDMePqh8B4Mf6KqycyyulTwtnicQF+tii2CveIwiXSCDdhNjzyErd8y2yr+aOc3VLfXazMJLKcD9nMAZXGVZQsxqA1dZzMkmnMpNtZdqfs DG3iYOwf CnP3qWcd6FjiWdKeHZzVE8iNwZq7KZnMoQmjunQKX3UcDSrzXPMVI4FXSi2SjzhsL1Ll/EU74kQ2rgBwswm97Rt5KK3a4Ah61xCojr9rZeboNRLYV0vunk+o2L+d0YB+6EFNI5vHno0SIG3l69QXJxMh0keqUKwSwgo8n+D6QnroBLAER0BRasFnHBIvJE/6PWIkqsh0d0JUiWuwPdmdZlQX+zmuFfWsGPueZ322P29O5TMGcieNITUDZO4WgkKhQBW2xWx6sZI7upm471FrRrAAZ4iNhcfKbwSenPOyp3ViAVGs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Create a new layer for the in-table PTE manipulation APIs. For now, The existing API is prefixed with double underscore to become the arch-private API and the public API is just a simple wrapper that calls the private API. The public API implementation will subsequently be used to transparently manipulate the contiguous bit where appropriate. But since there are already some contig-aware users (e.g. hugetlb, kernel mapper), we must first ensure those users use the private API directly so that the future contig-bit manipulations in the public API do not interfere with those existing uses. Tested-by: John Hubbard Signed-off-by: Ryan Roberts --- arch/arm64/include/asm/pgtable.h | 10 ++++++---- arch/arm64/mm/fault.c | 6 +++--- arch/arm64/mm/hugetlbpage.c | 2 +- 3 files changed, 10 insertions(+), 8 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 4c2d6c483390..fe27a3175618 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -312,7 +312,7 @@ static inline void __check_safe_pte_update(struct mm_struct *mm, pte_t *ptep, /* * Check for potential race with hardware updates of the pte - * (ptep_set_access_flags safely changes valid ptes without going + * (__ptep_set_access_flags safely changes valid ptes without going * through an invalid entry). */ VM_WARN_ONCE(!pte_young(pte), @@ -854,8 +854,7 @@ static inline pmd_t pmd_modify(pmd_t pmd, pgprot_t newprot) return pte_pmd(pte_modify(pmd_pte(pmd), newprot)); } -#define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS -extern int ptep_set_access_flags(struct vm_area_struct *vma, +extern int __ptep_set_access_flags(struct vm_area_struct *vma, unsigned long address, pte_t *ptep, pte_t entry, int dirty); @@ -865,7 +864,8 @@ static inline int pmdp_set_access_flags(struct vm_area_struct *vma, unsigned long address, pmd_t *pmdp, pmd_t entry, int dirty) { - return ptep_set_access_flags(vma, address, (pte_t *)pmdp, pmd_pte(entry), dirty); + return __ptep_set_access_flags(vma, address, (pte_t *)pmdp, + pmd_pte(entry), dirty); } static inline int pud_devmap(pud_t pud) @@ -1141,6 +1141,8 @@ void vmemmap_update_pte(unsigned long addr, pte_t *ptep, pte_t pte); #define ptep_clear_flush_young __ptep_clear_flush_young #define __HAVE_ARCH_PTEP_SET_WRPROTECT #define ptep_set_wrprotect __ptep_set_wrprotect +#define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS +#define ptep_set_access_flags __ptep_set_access_flags #endif /* !__ASSEMBLY__ */ diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index 23d0dfc16686..dbbc06cfb848 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -209,9 +209,9 @@ static void show_pte(unsigned long addr) * * Returns whether or not the PTE actually changed. */ -int ptep_set_access_flags(struct vm_area_struct *vma, - unsigned long address, pte_t *ptep, - pte_t entry, int dirty) +int __ptep_set_access_flags(struct vm_area_struct *vma, + unsigned long address, pte_t *ptep, + pte_t entry, int dirty) { pteval_t old_pteval, pteval; pte_t pte = READ_ONCE(*ptep); diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index f6612f3e1c07..9949b80baac8 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -459,7 +459,7 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, pte_t orig_pte; if (!pte_cont(pte)) - return ptep_set_access_flags(vma, addr, ptep, pte, dirty); + return __ptep_set_access_flags(vma, addr, ptep, pte, dirty); ncontig = find_num_contig(mm, addr, ptep, &pgsize); dpfn = pgsize >> PAGE_SHIFT; From patchwork Fri Feb 2 08:07:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13542352 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4958CC4828E for ; Fri, 2 Feb 2024 08:09:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D15EF6B00A4; Fri, 2 Feb 2024 03:09:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CC6766B00A5; Fri, 2 Feb 2024 03:09:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B66C56B00A6; Fri, 2 Feb 2024 03:09:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id A584A6B00A4 for ; Fri, 2 Feb 2024 03:09:23 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 6C3911A02F6 for ; Fri, 2 Feb 2024 08:09:23 +0000 (UTC) X-FDA: 81746139006.04.9E4A76D Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf05.hostedemail.com (Postfix) with ESMTP id DB83710000E for ; Fri, 2 Feb 2024 08:09:21 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=none; spf=pass (imf05.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706861362; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=EKn7BmrjSmFG23Bgg+izfHM8ytbGtfqkkpIYAkKZT10=; b=igieXL1CT1j+Akic+O/W1QAOsi3cGlixtTCP5SAGXQgxcuVPVbcrZUzycokaldK9vsmv2h sJvkvZzm+tFj3Yynkk9nM08BqTP4Ecs/gZgavI3SC2uTpiG9xpx2zly2JxFXcy4vjYhCbG qzmYoIkb4bT7fbXqsgyTduXk9zLzU8g= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=none; spf=pass (imf05.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706861362; a=rsa-sha256; cv=none; b=eWsjwpYgxS/ZiwxhEZMm8MS/Xm2Q9UQ1mgfmwI0314EkQKgs5TEQbnAbtS0Jb7B2iaOFdZ Oye6bUNnVjaiylMorVuN5kvifnwqifzSaoJF2zOIFIBAnbtQyFC/BdHxw3lD6NZrt/JhMo +wYJ3IQtyKJ36QD7nbR8IvEkKoYljCo= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 9B9861BB0; Fri, 2 Feb 2024 00:10:03 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 936AA3F5A1; Fri, 2 Feb 2024 00:09:17 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v5 17/25] arm64/mm: ptep_get(): New layer to manage contig bit Date: Fri, 2 Feb 2024 08:07:48 +0000 Message-Id: <20240202080756.1453939-18-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240202080756.1453939-1-ryan.roberts@arm.com> References: <20240202080756.1453939-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: DB83710000E X-Rspam-User: X-Stat-Signature: 3s88331ikrt4teyqeci3g3q96zoe9bj3 X-Rspamd-Server: rspam01 X-HE-Tag: 1706861361-190688 X-HE-Meta: U2FsdGVkX188nbHF6yxNYb/3wJ9rEiUWTNV8ENlvu9bMMyCVet0xQgXrIUvZSAXEtynGUR+G6EjgUpT3ojUHIO4XnxAxs5cqnTlcWJ7DnCuO1g26sJAwRSjbW2RiE6NjcXKBpU3CCQF10VIzRIrZ68VvxbXuX82o0ATrIfp7bg3oAw5OEr+gQm0DLrnYcamBA+L4cNSY5fuI/b0q6AxYVO7YopFRgvzgv4YfwKdu40X8Unpy26dZr/2RV+kzT7k/9qAzcT9GlskoJ/2G0VgNyQrJi2KY8nEgsOiqWexQk7akXyCIbRbg9NuUl7fhERtUNrQfheoocyXflz9/Kbqyn7jQLsfpjMMiOae7IBqlYICyKTbR3dGLwYs3ZgyxXrzc2hTImkLT3IhvdnooGag+6DevECn74aPiW87BYo+j9+mh7QwJh40iV6yZNJEEbJgCQwEkWLOpM1izspt8jwaSRXHqAR+jB3OMPnngcLjvUa06nKESxYt62p4JIfxeDw8WyHRnG2q4gFo4SKHkJ/2vwDU2KhTMMY7nnBZq5Xg61VRm2cg23y+reFGN4JOw684+72Iotk/2aFSzjmHeGe6AdPDV4NUWjgJrKBH2/kGAhI4EM0veSm2xQdHfks0LY7ZVs75xUPLjnxhxkFwHBg7OZIWOOJG8Obno/BTrgyTMD/M23IMZETn/VNaBd9/8RZuUKOmyrGibQj73TeL6uKG24s6qvKXDXIfr16tZgmDUEcxA8mZc6E11G7QTwLApTP7UcpQHAT1hG5bYSXwdsYyd7a7T+5ngRQmbUzUR7odItGDEZgXfEaJNTG5H+yiez/9bEXgrtrxiT9FHYZUYF1uTVh0tccgECOxCPIVd+i3mIe5LGvz4sS2laCUpMSmK6fQZakSlBJNiI7HGcqzDKk1hd9DvzAlWegGzKfcIxYTWfIOLlFQpHObvON37lS563nvbXigyQKt1jpt7sqbaXgE OjA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Create a new layer for the in-table PTE manipulation APIs. For now, The existing API is prefixed with double underscore to become the arch-private API and the public API is just a simple wrapper that calls the private API. The public API implementation will subsequently be used to transparently manipulate the contiguous bit where appropriate. But since there are already some contig-aware users (e.g. hugetlb, kernel mapper), we must first ensure those users use the private API directly so that the future contig-bit manipulations in the public API do not interfere with those existing uses. arm64 did not previously define an arch-specific ptep_get(), so override the default version in the arch code, and also define the private __ptep_get() version. Currently they both do the same thing that the default version does (READ_ONCE()). Some arch users (hugetlb) were already using ptep_get() so convert those to the private API. While other callsites were doing direct READ_ONCE(), so convert those to use the appropriate (public/private) API too. Tested-by: John Hubbard Signed-off-by: Ryan Roberts --- arch/arm64/include/asm/pgtable.h | 12 +++++++++--- arch/arm64/kernel/efi.c | 2 +- arch/arm64/mm/fault.c | 4 ++-- arch/arm64/mm/hugetlbpage.c | 18 +++++++++--------- arch/arm64/mm/kasan_init.c | 2 +- arch/arm64/mm/mmu.c | 12 ++++++------ arch/arm64/mm/pageattr.c | 4 ++-- arch/arm64/mm/trans_pgd.c | 2 +- 8 files changed, 31 insertions(+), 25 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index fe27a3175618..7dc6b68ee516 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -276,6 +276,11 @@ static inline void __set_pte(pte_t *ptep, pte_t pte) } } +static inline pte_t __ptep_get(pte_t *ptep) +{ + return READ_ONCE(*ptep); +} + extern void __sync_icache_dcache(pte_t pteval); bool pgattr_change_is_safe(u64 old, u64 new); @@ -303,7 +308,7 @@ static inline void __check_safe_pte_update(struct mm_struct *mm, pte_t *ptep, if (!IS_ENABLED(CONFIG_DEBUG_VM)) return; - old_pte = READ_ONCE(*ptep); + old_pte = __ptep_get(ptep); if (!pte_valid(old_pte) || !pte_valid(pte)) return; @@ -905,7 +910,7 @@ static inline int __ptep_test_and_clear_young(struct vm_area_struct *vma, { pte_t old_pte, pte; - pte = READ_ONCE(*ptep); + pte = __ptep_get(ptep); do { old_pte = pte; pte = pte_mkold(pte); @@ -978,7 +983,7 @@ static inline void __ptep_set_wrprotect(struct mm_struct *mm, { pte_t old_pte, pte; - pte = READ_ONCE(*ptep); + pte = __ptep_get(ptep); do { old_pte = pte; pte = pte_wrprotect(pte); @@ -1130,6 +1135,7 @@ void vmemmap_update_pte(unsigned long addr, pte_t *ptep, pte_t pte); #define vmemmap_update_pte vmemmap_update_pte #endif +#define ptep_get __ptep_get #define set_pte __set_pte #define set_ptes __set_ptes #define pte_clear __pte_clear diff --git a/arch/arm64/kernel/efi.c b/arch/arm64/kernel/efi.c index 44288a12fc6c..9afcc690fe73 100644 --- a/arch/arm64/kernel/efi.c +++ b/arch/arm64/kernel/efi.c @@ -103,7 +103,7 @@ static int __init set_permissions(pte_t *ptep, unsigned long addr, void *data) { struct set_perm_data *spd = data; const efi_memory_desc_t *md = spd->md; - pte_t pte = READ_ONCE(*ptep); + pte_t pte = __ptep_get(ptep); if (md->attribute & EFI_MEMORY_RO) pte = set_pte_bit(pte, __pgprot(PTE_RDONLY)); diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c index dbbc06cfb848..892e8cc8983f 100644 --- a/arch/arm64/mm/fault.c +++ b/arch/arm64/mm/fault.c @@ -191,7 +191,7 @@ static void show_pte(unsigned long addr) if (!ptep) break; - pte = READ_ONCE(*ptep); + pte = __ptep_get(ptep); pr_cont(", pte=%016llx", pte_val(pte)); pte_unmap(ptep); } while(0); @@ -214,7 +214,7 @@ int __ptep_set_access_flags(struct vm_area_struct *vma, pte_t entry, int dirty) { pteval_t old_pteval, pteval; - pte_t pte = READ_ONCE(*ptep); + pte_t pte = __ptep_get(ptep); if (pte_same(pte, entry)) return 0; diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 9949b80baac8..c3db949560f9 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -152,14 +152,14 @@ pte_t huge_ptep_get(pte_t *ptep) { int ncontig, i; size_t pgsize; - pte_t orig_pte = ptep_get(ptep); + pte_t orig_pte = __ptep_get(ptep); if (!pte_present(orig_pte) || !pte_cont(orig_pte)) return orig_pte; ncontig = num_contig_ptes(page_size(pte_page(orig_pte)), &pgsize); for (i = 0; i < ncontig; i++, ptep++) { - pte_t pte = ptep_get(ptep); + pte_t pte = __ptep_get(ptep); if (pte_dirty(pte)) orig_pte = pte_mkdirty(orig_pte); @@ -184,7 +184,7 @@ static pte_t get_clear_contig(struct mm_struct *mm, unsigned long pgsize, unsigned long ncontig) { - pte_t orig_pte = ptep_get(ptep); + pte_t orig_pte = __ptep_get(ptep); unsigned long i; for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) { @@ -408,7 +408,7 @@ pte_t huge_ptep_get_and_clear(struct mm_struct *mm, { int ncontig; size_t pgsize; - pte_t orig_pte = ptep_get(ptep); + pte_t orig_pte = __ptep_get(ptep); if (!pte_cont(orig_pte)) return __ptep_get_and_clear(mm, addr, ptep); @@ -431,11 +431,11 @@ static int __cont_access_flags_changed(pte_t *ptep, pte_t pte, int ncontig) { int i; - if (pte_write(pte) != pte_write(ptep_get(ptep))) + if (pte_write(pte) != pte_write(__ptep_get(ptep))) return 1; for (i = 0; i < ncontig; i++) { - pte_t orig_pte = ptep_get(ptep + i); + pte_t orig_pte = __ptep_get(ptep + i); if (pte_dirty(pte) != pte_dirty(orig_pte)) return 1; @@ -492,7 +492,7 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm, size_t pgsize; pte_t pte; - if (!pte_cont(READ_ONCE(*ptep))) { + if (!pte_cont(__ptep_get(ptep))) { __ptep_set_wrprotect(mm, addr, ptep); return; } @@ -517,7 +517,7 @@ pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, size_t pgsize; int ncontig; - if (!pte_cont(READ_ONCE(*ptep))) + if (!pte_cont(__ptep_get(ptep))) return ptep_clear_flush(vma, addr, ptep); ncontig = find_num_contig(mm, addr, ptep, &pgsize); @@ -550,7 +550,7 @@ pte_t huge_ptep_modify_prot_start(struct vm_area_struct *vma, unsigned long addr * when the permission changes from executable to non-executable * in cases where cpu is affected with errata #2645198. */ - if (pte_user_exec(READ_ONCE(*ptep))) + if (pte_user_exec(__ptep_get(ptep))) return huge_ptep_clear_flush(vma, addr, ptep); } return huge_ptep_get_and_clear(vma->vm_mm, addr, ptep); diff --git a/arch/arm64/mm/kasan_init.c b/arch/arm64/mm/kasan_init.c index f659bd98c63f..9ee16cfce587 100644 --- a/arch/arm64/mm/kasan_init.c +++ b/arch/arm64/mm/kasan_init.c @@ -113,7 +113,7 @@ static void __init kasan_pte_populate(pmd_t *pmdp, unsigned long addr, memset(__va(page_phys), KASAN_SHADOW_INIT, PAGE_SIZE); next = addr + PAGE_SIZE; __set_pte(ptep, pfn_pte(__phys_to_pfn(page_phys), PAGE_KERNEL)); - } while (ptep++, addr = next, addr != end && pte_none(READ_ONCE(*ptep))); + } while (ptep++, addr = next, addr != end && pte_none(__ptep_get(ptep))); } static void __init kasan_pmd_populate(pud_t *pudp, unsigned long addr, diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index bcaa5a5d86f8..8c1ab90bb1e5 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -173,7 +173,7 @@ static void init_pte(pmd_t *pmdp, unsigned long addr, unsigned long end, ptep = pte_set_fixmap_offset(pmdp, addr); do { - pte_t old_pte = READ_ONCE(*ptep); + pte_t old_pte = __ptep_get(ptep); __set_pte(ptep, pfn_pte(__phys_to_pfn(phys), prot)); @@ -182,7 +182,7 @@ static void init_pte(pmd_t *pmdp, unsigned long addr, unsigned long end, * only allow updates to the permission attributes. */ BUG_ON(!pgattr_change_is_safe(pte_val(old_pte), - READ_ONCE(pte_val(*ptep)))); + pte_val(__ptep_get(ptep)))); phys += PAGE_SIZE; } while (ptep++, addr += PAGE_SIZE, addr != end); @@ -854,7 +854,7 @@ static void unmap_hotplug_pte_range(pmd_t *pmdp, unsigned long addr, do { ptep = pte_offset_kernel(pmdp, addr); - pte = READ_ONCE(*ptep); + pte = __ptep_get(ptep); if (pte_none(pte)) continue; @@ -987,7 +987,7 @@ static void free_empty_pte_table(pmd_t *pmdp, unsigned long addr, do { ptep = pte_offset_kernel(pmdp, addr); - pte = READ_ONCE(*ptep); + pte = __ptep_get(ptep); /* * This is just a sanity check here which verifies that @@ -1006,7 +1006,7 @@ static void free_empty_pte_table(pmd_t *pmdp, unsigned long addr, */ ptep = pte_offset_kernel(pmdp, 0UL); for (i = 0; i < PTRS_PER_PTE; i++) { - if (!pte_none(READ_ONCE(ptep[i]))) + if (!pte_none(__ptep_get(&ptep[i]))) return; } @@ -1503,7 +1503,7 @@ pte_t ptep_modify_prot_start(struct vm_area_struct *vma, unsigned long addr, pte * when the permission changes from executable to non-executable * in cases where cpu is affected with errata #2645198. */ - if (pte_user_exec(READ_ONCE(*ptep))) + if (pte_user_exec(ptep_get(ptep))) return ptep_clear_flush(vma, addr, ptep); } return ptep_get_and_clear(vma->vm_mm, addr, ptep); diff --git a/arch/arm64/mm/pageattr.c b/arch/arm64/mm/pageattr.c index a7996d8edf0a..0c4e3ecf989d 100644 --- a/arch/arm64/mm/pageattr.c +++ b/arch/arm64/mm/pageattr.c @@ -36,7 +36,7 @@ bool can_set_direct_map(void) static int change_page_range(pte_t *ptep, unsigned long addr, void *data) { struct page_change_data *cdata = data; - pte_t pte = READ_ONCE(*ptep); + pte_t pte = __ptep_get(ptep); pte = clear_pte_bit(pte, cdata->clear_mask); pte = set_pte_bit(pte, cdata->set_mask); @@ -245,5 +245,5 @@ bool kernel_page_present(struct page *page) return true; ptep = pte_offset_kernel(pmdp, addr); - return pte_valid(READ_ONCE(*ptep)); + return pte_valid(__ptep_get(ptep)); } diff --git a/arch/arm64/mm/trans_pgd.c b/arch/arm64/mm/trans_pgd.c index 230b607cf881..5139a28130c0 100644 --- a/arch/arm64/mm/trans_pgd.c +++ b/arch/arm64/mm/trans_pgd.c @@ -33,7 +33,7 @@ static void *trans_alloc(struct trans_pgd_info *info) static void _copy_pte(pte_t *dst_ptep, pte_t *src_ptep, unsigned long addr) { - pte_t pte = READ_ONCE(*src_ptep); + pte_t pte = __ptep_get(src_ptep); if (pte_valid(pte)) { /* From patchwork Fri Feb 2 08:07:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13542354 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 64665C47258 for ; Fri, 2 Feb 2024 08:09:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E4F646B00A7; Fri, 2 Feb 2024 03:09:32 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DB0196B00A9; Fri, 2 Feb 2024 03:09:32 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C9F336B00AA; Fri, 2 Feb 2024 03:09:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id BB6E46B00A7 for ; Fri, 2 Feb 2024 03:09:32 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 9D8F81A05B2 for ; Fri, 2 Feb 2024 08:09:32 +0000 (UTC) X-FDA: 81746139384.21.0D9EDAB Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf22.hostedemail.com (Postfix) with ESMTP id 09075C0014 for ; Fri, 2 Feb 2024 08:09:30 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=none; spf=pass (imf22.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706861371; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nA5bUrRt6blUgSGh1rucsmmbrNp8un7UcFTvjqqTNjA=; b=AvK2u3cGOWtLLQIKu2hJ+BBwlMNd6bQ1QzTMoxUjGnjDEGlqPRpQM/l015bNxsHEwLXQV3 hPX3RHLq08e0bIjVtbRM8YrKoFom9xJ1+sMJ6/Is7X0BzuIEj/IVX7DT++aCg+5VRiLJe5 u5xvOv2t7PfCpkphVeSCBb4DbF7l4i4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706861371; a=rsa-sha256; cv=none; b=lsn1QUvyk38wiP1Lg6kitc9K6eIdMql9R1bQcYyPSBGP9lxyUeJM9NscPKqeAgoLV+p42J nZp/7+XklZZlzna41HNEgGpGB6mAR7ZPXoADXDh12s39XJTiQabousPu/kjwhKdfCIkhSk zYjTWibA23EcqDRLFjzd3PUAwLVH3mk= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=none; spf=pass (imf22.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 704231BC0; Fri, 2 Feb 2024 00:10:12 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 82DE43F5A1; Fri, 2 Feb 2024 00:09:21 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v5 18/25] arm64/mm: Split __flush_tlb_range() to elide trailing DSB Date: Fri, 2 Feb 2024 08:07:49 +0000 Message-Id: <20240202080756.1453939-19-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240202080756.1453939-1-ryan.roberts@arm.com> References: <20240202080756.1453939-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 09075C0014 X-Rspam-User: X-Stat-Signature: nagr7gqnxge1dxafeqqepcnty6hq3mhf X-Rspamd-Server: rspam03 X-HE-Tag: 1706861370-488643 X-HE-Meta: U2FsdGVkX18ybsdwhkpHHmqLN7fMhiHDHbwUFb5tGkaQoXVHHHOKI6qokU5BAVxNGa9jSg4ER6LUcWFIdaQ7hPwUWVknYct5bd94jO9rvWRxycIFWx3/JGZAm4WZrL4m/CLMy1fW53Fz/4y1IeynEmZwfP/Pymvg/kZs2jneXqeU3u5giJLpQILC4hx8wTsVQD9lAgl0t1uKVIezpEo5i4ywR5kHtXF2i3+abVwlMgZOWxV9GEgLyKMmx3a7NzaOKERst5HAe9W7CUs9AUjz2559sXYloMLmCp8zBZLZHdn5YXO6R5rj1NbdG7yekqdvW/8xGZxIFwlK6q74x+uGaHDUDFiBfsk0C3DxBOlKS40/1G8KTk7Db9mCG6NYwZiE4a8PzkblobHj0ex8nRfnHbFXA9VcbNACETxIxKBlOYkgrz70/ynI7iduN6MlrOFTpdWwbPYM0JsD/bNiewJe67VHdKstQDzoG29+BAu1b+HqfOGad7BIGzJu4E7UWgxmyH7PnmYYGTR3R24FeNwcUi7ro5o/8W9v1fgqOpKmiC/3+9+U0y+Sf9F1ORWrHlLMVgDKxZLjBPNn3CuOxy93BmjcBRH6ojfJJdNpNR/N+TwkhfdaDuqUzK1ZWLSqj1MY71kI21fIs0Us1T3jo+vz3M+78UvmDFv4pCcMKddURDsO7SE9wh+zsvdr915JfurXmWIToA0okdvrzC4rvNcTP/qOEyH6MIa9eHLegA+ovJkrX2E9lrJ8Pl7h9DFJwTBVyHRBCHxQ35ak4J2LgYW12pZa7jSgez53Ktpkc9hMPuYwkx558WHNnJAkM4DpMT9mCIXu7xqFQLvtVSWOU7fiD6EmaVuGLBD2vvwLtC6yY7gLndptlUQYuaU418Ih5mQI3IcnpsVK50F6o/9TbORPYVw5POsZpEkr8//TMkW3DntTBd+Td0ot4mLs0y9VH5q0tThu9B1jKPboEjgWGxX TYr/3YRV bN/dgjWNlYn686ZyFfs5vBbHrjuJf2D7cikUG3rYfmaWh3Nu1xTwx1sDweOJEHs5snsxFyCA3OBGzCRqxD2PubXSfPItIdIsmBAeK5qgQRwextMm71dxEOzxeKIMONyH53Lhv4MBI+v2H7tlHw6Kcf2SY3JR+ltpN/jhXpUVAlm9JPlo08GxA74EpEsLqIRh7m+TbrIGf4X8a/kNJnkaiCsnTqchKBbku9Ld8ZiO81cm8XDwCTux3RSYPp7imvD5L0D+mPSMJvEJOf1Gwl4bzvvFesdmcUonK1sILr69sWMtbRIo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Split __flush_tlb_range() into __flush_tlb_range_nosync() + __flush_tlb_range(), in the same way as the existing flush_tlb_page() arrangement. This allows calling __flush_tlb_range_nosync() to elide the trailing DSB. Forthcoming "contpte" code will take advantage of this when clearing the young bit from a contiguous range of ptes. Tested-by: John Hubbard Signed-off-by: Ryan Roberts Reviewed-by: David Hildenbrand --- arch/arm64/include/asm/tlbflush.h | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h index 79e932a1bdf8..50a765917327 100644 --- a/arch/arm64/include/asm/tlbflush.h +++ b/arch/arm64/include/asm/tlbflush.h @@ -422,7 +422,7 @@ do { \ #define __flush_s2_tlb_range_op(op, start, pages, stride, tlb_level) \ __flush_tlb_range_op(op, start, pages, stride, 0, tlb_level, false, kvm_lpa2_is_enabled()); -static inline void __flush_tlb_range(struct vm_area_struct *vma, +static inline void __flush_tlb_range_nosync(struct vm_area_struct *vma, unsigned long start, unsigned long end, unsigned long stride, bool last_level, int tlb_level) @@ -456,10 +456,19 @@ static inline void __flush_tlb_range(struct vm_area_struct *vma, __flush_tlb_range_op(vae1is, start, pages, stride, asid, tlb_level, true, lpa2_is_enabled()); - dsb(ish); mmu_notifier_arch_invalidate_secondary_tlbs(vma->vm_mm, start, end); } +static inline void __flush_tlb_range(struct vm_area_struct *vma, + unsigned long start, unsigned long end, + unsigned long stride, bool last_level, + int tlb_level) +{ + __flush_tlb_range_nosync(vma, start, end, stride, + last_level, tlb_level); + dsb(ish); +} + static inline void flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end) { From patchwork Fri Feb 2 08:07:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13542353 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3FC15C4828E for ; Fri, 2 Feb 2024 08:09:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C2AC76B00A6; Fri, 2 Feb 2024 03:09:31 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BDAAF6B00A7; Fri, 2 Feb 2024 03:09:31 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A55A06B00A8; Fri, 2 Feb 2024 03:09:31 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 948236B00A6 for ; Fri, 2 Feb 2024 03:09:31 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 672F91607E0 for ; Fri, 2 Feb 2024 08:09:31 +0000 (UTC) X-FDA: 81746139342.03.D84145D Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf01.hostedemail.com (Postfix) with ESMTP id 9DBBB40009 for ; Fri, 2 Feb 2024 08:09:29 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=none; spf=pass (imf01.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706861369; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=P1xu3BLf2o1r+yd8h0LJeygbJrOGinDpUNJMX+u+Vlc=; b=CgELnirJZKm3nNaao4PIeBIBKWACSwwXdh+GxHLlvciwvJ2mZSrh+E/TtjurBnmdUwBAP9 qE2hB6LYUNco0fD8f+KfFIav3/JRilon/nnkOyuxiPLGyz5bVuGOQUiAyoPugrNefJnoG3 QccmYG+Ha6jfBGuVI7sXOF2s4uPSUfY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706861369; a=rsa-sha256; cv=none; b=Z11eN0qOXlqlHf8zwlrz98HK0mtnj6Oh3peIJzLnW6atE4Ra3cPBPpUFI0Lhqq+giGOsLo ehFT1FZ1TfcEq6AdvUWHWLQHmr3urZNbNEz30JKQtY59dALpK4vll1Zsom7IPjgtPUq0HI ekVbquQIYUdauNIADDevOwHvsOkpmFQ= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=none; spf=pass (imf01.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 44A331BB2; Fri, 2 Feb 2024 00:10:11 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 55C663F8A4; Fri, 2 Feb 2024 00:09:25 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v5 19/25] arm64/mm: Wire up PTE_CONT for user mappings Date: Fri, 2 Feb 2024 08:07:50 +0000 Message-Id: <20240202080756.1453939-20-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240202080756.1453939-1-ryan.roberts@arm.com> References: <20240202080756.1453939-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 9DBBB40009 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: qsw9ihjbytxwqe1nwkjktdu1jdpirsb6 X-HE-Tag: 1706861369-343488 X-HE-Meta: U2FsdGVkX1+ozqua0FFtDfwK+DsHh3dYq0yD6KC+xy4BoBcoIp0nJh6ULYwUebN2kD9oR5t61+7Dlc2vz/vXtHQ89gL0J6sWDUJ/1nnCUiTQvdL9KKDtKj3GAHKON69FMXTUGcl2uM4mpKi9kRV/EVvXhAmjhAmpNkMJi1WKC7c5adbDm4Kb5sY08wxRk+rXR6yhBU2lcCvADZVP1GmHVBQr4alkxXg9mZKpRuJvKMiRoOrZ30XiHxZL8swa4kuel8Teq/US8ZZ+GSEuHqdkTNLhJi5a5hCeVqv7XjBdel8n3TaeMhXZn43Q+CCL5s6bK/XUNgvFZiZJLDVQKmiQ0aVH5k38mMgNLiHWl6EhlaKwonlkSs1SeQiWSKfYjqpDPDnE9LEQ96g4gprxD1XiqrE5NyqPVgTeRYHKVyVTBh/U4dQ1HgNUZFylJX9D4XYVxFW+J0NWY7h0h6remA3Ib/5FsjsnnHaKBXv8hot5LjGUfbJICarqsjvoGLCGQAM6lDAlub4+stEgX/lPzGlKxlmF71ft8Zu0PgNNw1blXy4K1GYFsJy/TkZ5SGi4k0kwLC/O2c818kEolR53yZj5OY65lCBJ2leOFItBbEo0Fd90nFt3XgRP8cwN92ANe5EQ/0u+50s4WPLGluf701XECvD4axJkhcQsmqXlWeRoiohPFnU3PjYG3RWL5RKueF/fLEZy+cYhtoeSHgBskodJBdFE0RvVYCJ5q5gcuk5QGeldJgwjGTHzsNiP+Y+TbZsIKh8W+PyTfUfJPFUOLmPRuDVdQngbhkURyXmNI1cWacIl2VydqSx5NQmfhtx55tUdPtGjle0gjqIGWe0Dj2xyTJ+6cLpnWaVBSxhyqLsiwEMR5djeIY12ilfEjNWOC4sKw2o9gZ+r7uq5Sdh+81BW+XIPJwdxJ2M2HGJ49iTNBoNKrehrkdn9kdsuKZSWEU1CZttG7PI35r9puBifZI+ y4FuLShv EAPUrskj7yGsnTMlTNUwoVU/FF1nBIWuGeWDmwxCRNkDF0hNaG1zW4U5S2kWExFjzNfRULEFahMyKPDy5YyDQOg8l3VEuZpvZQHAbzO1OJ22HKQugcVlpd25IbCPEjd9vBAKe+zY0LGUOZT1fPpomfoPBcCl30RNw4kK/qrgW2dw8eJsGiW2eSHG0WtPeGQ/zN4q3eVsANtOGGyro0cvqwEeRheRzu7qeGlRas9njBadpw3ZGjBy2BB1OsXEM/wtXCr1ofixQnJZl1nvmQapS144NHYMlfIUUEDReKe5T+YJro24= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: With the ptep API sufficiently refactored, we can now introduce a new "contpte" API layer, which transparently manages the PTE_CONT bit for user mappings. In this initial implementation, only suitable batches of PTEs, set via set_ptes(), are mapped with the PTE_CONT bit. Any subsequent modification of individual PTEs will cause an "unfold" operation to repaint the contpte block as individual PTEs before performing the requested operation. While, a modification of a single PTE could cause the block of PTEs to which it belongs to become eligible for "folding" into a contpte entry, "folding" is not performed in this initial implementation due to the costs of checking the requirements are met. Due to this, contpte mappings will degrade back to normal pte mappings over time if/when protections are changed. This will be solved in a future patch. Since a contpte block only has a single access and dirty bit, the semantic here changes slightly; when getting a pte (e.g. ptep_get()) that is part of a contpte mapping, the access and dirty information are pulled from the block (so all ptes in the block return the same access/dirty info). When changing the access/dirty info on a pte (e.g. ptep_set_access_flags()) that is part of a contpte mapping, this change will affect the whole contpte block. This is works fine in practice since we guarantee that only a single folio is mapped by a contpte block, and the core-mm tracks access/dirty information per folio. In order for the public functions, which used to be pure inline, to continue to be callable by modules, export all the contpte_* symbols that are now called by those public inline functions. The feature is enabled/disabled with the ARM64_CONTPTE Kconfig parameter at build time. It defaults to enabled as long as its dependency, TRANSPARENT_HUGEPAGE is also enabled. The core-mm depends upon TRANSPARENT_HUGEPAGE to be able to allocate large folios, so if its not enabled, then there is no chance of meeting the physical contiguity requirement for contpte mappings. Tested-by: John Hubbard Signed-off-by: Ryan Roberts Acked-by: Ard Biesheuvel --- arch/arm64/Kconfig | 9 + arch/arm64/include/asm/pgtable.h | 161 ++++++++++++++++++ arch/arm64/mm/Makefile | 1 + arch/arm64/mm/contpte.c | 283 +++++++++++++++++++++++++++++++ 4 files changed, 454 insertions(+) create mode 100644 arch/arm64/mm/contpte.c diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index d86d7f4758b5..1442e8ed95b6 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -2230,6 +2230,15 @@ config UNWIND_PATCH_PAC_INTO_SCS select UNWIND_TABLES select DYNAMIC_SCS +config ARM64_CONTPTE + bool "Contiguous PTE mappings for user memory" if EXPERT + depends on TRANSPARENT_HUGEPAGE + default y + help + When enabled, user mappings are configured using the PTE contiguous + bit, for any mappings that meet the size and alignment requirements. + This reduces TLB pressure and improves performance. + endmenu # "Kernel Features" menu "Boot options" diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 7dc6b68ee516..34892a95403d 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -133,6 +133,10 @@ static inline pteval_t __phys_to_pte_val(phys_addr_t phys) */ #define pte_valid_not_user(pte) \ ((pte_val(pte) & (PTE_VALID | PTE_USER | PTE_UXN)) == (PTE_VALID | PTE_UXN)) +/* + * Returns true if the pte is valid and has the contiguous bit set. + */ +#define pte_valid_cont(pte) (pte_valid(pte) && pte_cont(pte)) /* * Could the pte be present in the TLB? We must check mm_tlb_flush_pending * so that we don't erroneously return false for pages that have been @@ -1135,6 +1139,161 @@ void vmemmap_update_pte(unsigned long addr, pte_t *ptep, pte_t pte); #define vmemmap_update_pte vmemmap_update_pte #endif +#ifdef CONFIG_ARM64_CONTPTE + +/* + * The contpte APIs are used to transparently manage the contiguous bit in ptes + * where it is possible and makes sense to do so. The PTE_CONT bit is considered + * a private implementation detail of the public ptep API (see below). + */ +extern void __contpte_try_unfold(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte); +extern pte_t contpte_ptep_get(pte_t *ptep, pte_t orig_pte); +extern pte_t contpte_ptep_get_lockless(pte_t *orig_ptep); +extern void contpte_set_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte, unsigned int nr); +extern int contpte_ptep_test_and_clear_young(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep); +extern int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep); +extern int contpte_ptep_set_access_flags(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + pte_t entry, int dirty); + +static inline void contpte_try_unfold(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte) +{ + if (unlikely(pte_valid_cont(pte))) + __contpte_try_unfold(mm, addr, ptep, pte); +} + +/* + * The below functions constitute the public API that arm64 presents to the + * core-mm to manipulate PTE entries within their page tables (or at least this + * is the subset of the API that arm64 needs to implement). These public + * versions will automatically and transparently apply the contiguous bit where + * it makes sense to do so. Therefore any users that are contig-aware (e.g. + * hugetlb, kernel mapper) should NOT use these APIs, but instead use the + * private versions, which are prefixed with double underscore. All of these + * APIs except for ptep_get_lockless() are expected to be called with the PTL + * held. + */ + +#define ptep_get ptep_get +static inline pte_t ptep_get(pte_t *ptep) +{ + pte_t pte = __ptep_get(ptep); + + if (likely(!pte_valid_cont(pte))) + return pte; + + return contpte_ptep_get(ptep, pte); +} + +#define ptep_get_lockless ptep_get_lockless +static inline pte_t ptep_get_lockless(pte_t *ptep) +{ + pte_t pte = __ptep_get(ptep); + + if (likely(!pte_valid_cont(pte))) + return pte; + + return contpte_ptep_get_lockless(ptep); +} + +static inline void set_pte(pte_t *ptep, pte_t pte) +{ + /* + * We don't have the mm or vaddr so cannot unfold contig entries (since + * it requires tlb maintenance). set_pte() is not used in core code, so + * this should never even be called. Regardless do our best to service + * any call and emit a warning if there is any attempt to set a pte on + * top of an existing contig range. + */ + pte_t orig_pte = __ptep_get(ptep); + + WARN_ON_ONCE(pte_valid_cont(orig_pte)); + __set_pte(ptep, pte_mknoncont(pte)); +} + +#define set_ptes set_ptes +static inline void set_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte, unsigned int nr) +{ + pte = pte_mknoncont(pte); + + if (likely(nr == 1)) { + contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); + __set_ptes(mm, addr, ptep, pte, 1); + } else { + contpte_set_ptes(mm, addr, ptep, pte, nr); + } +} + +static inline void pte_clear(struct mm_struct *mm, + unsigned long addr, pte_t *ptep) +{ + contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); + __pte_clear(mm, addr, ptep); +} + +#define __HAVE_ARCH_PTEP_GET_AND_CLEAR +static inline pte_t ptep_get_and_clear(struct mm_struct *mm, + unsigned long addr, pte_t *ptep) +{ + contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); + return __ptep_get_and_clear(mm, addr, ptep); +} + +#define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG +static inline int ptep_test_and_clear_young(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep) +{ + pte_t orig_pte = __ptep_get(ptep); + + if (likely(!pte_valid_cont(orig_pte))) + return __ptep_test_and_clear_young(vma, addr, ptep); + + return contpte_ptep_test_and_clear_young(vma, addr, ptep); +} + +#define __HAVE_ARCH_PTEP_CLEAR_YOUNG_FLUSH +static inline int ptep_clear_flush_young(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep) +{ + pte_t orig_pte = __ptep_get(ptep); + + if (likely(!pte_valid_cont(orig_pte))) + return __ptep_clear_flush_young(vma, addr, ptep); + + return contpte_ptep_clear_flush_young(vma, addr, ptep); +} + +#define __HAVE_ARCH_PTEP_SET_WRPROTECT +static inline void ptep_set_wrprotect(struct mm_struct *mm, + unsigned long addr, pte_t *ptep) +{ + contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); + __ptep_set_wrprotect(mm, addr, ptep); +} + +#define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS +static inline int ptep_set_access_flags(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + pte_t entry, int dirty) +{ + pte_t orig_pte = __ptep_get(ptep); + + entry = pte_mknoncont(entry); + + if (likely(!pte_valid_cont(orig_pte))) + return __ptep_set_access_flags(vma, addr, ptep, entry, dirty); + + return contpte_ptep_set_access_flags(vma, addr, ptep, entry, dirty); +} + +#else /* CONFIG_ARM64_CONTPTE */ + #define ptep_get __ptep_get #define set_pte __set_pte #define set_ptes __set_ptes @@ -1150,6 +1309,8 @@ void vmemmap_update_pte(unsigned long addr, pte_t *ptep, pte_t pte); #define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS #define ptep_set_access_flags __ptep_set_access_flags +#endif /* CONFIG_ARM64_CONTPTE */ + #endif /* !__ASSEMBLY__ */ #endif /* __ASM_PGTABLE_H */ diff --git a/arch/arm64/mm/Makefile b/arch/arm64/mm/Makefile index dbd1bc95967d..60454256945b 100644 --- a/arch/arm64/mm/Makefile +++ b/arch/arm64/mm/Makefile @@ -3,6 +3,7 @@ obj-y := dma-mapping.o extable.o fault.o init.o \ cache.o copypage.o flush.o \ ioremap.o mmap.o pgd.o mmu.o \ context.o proc.o pageattr.o fixmap.o +obj-$(CONFIG_ARM64_CONTPTE) += contpte.o obj-$(CONFIG_HUGETLB_PAGE) += hugetlbpage.o obj-$(CONFIG_PTDUMP_CORE) += ptdump.o obj-$(CONFIG_PTDUMP_DEBUGFS) += ptdump_debugfs.o diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c new file mode 100644 index 000000000000..bfb50e6b44c7 --- /dev/null +++ b/arch/arm64/mm/contpte.c @@ -0,0 +1,283 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (C) 2023 ARM Ltd. + */ + +#include +#include +#include + +static inline bool mm_is_user(struct mm_struct *mm) +{ + /* + * Don't attempt to apply the contig bit to kernel mappings, because + * dynamically adding/removing the contig bit can cause page faults. + * These racing faults are ok for user space, since they get serialized + * on the PTL. But kernel mappings can't tolerate faults. + */ + return mm != &init_mm; +} + +static inline pte_t *contpte_align_down(pte_t *ptep) +{ + return (pte_t *)(ALIGN_DOWN((unsigned long)ptep >> 3, CONT_PTES) << 3); +} + +static void contpte_convert(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte) +{ + struct vm_area_struct vma = TLB_FLUSH_VMA(mm, 0); + unsigned long start_addr; + pte_t *start_ptep; + int i; + + start_ptep = ptep = contpte_align_down(ptep); + start_addr = addr = ALIGN_DOWN(addr, CONT_PTE_SIZE); + pte = pfn_pte(ALIGN_DOWN(pte_pfn(pte), CONT_PTES), pte_pgprot(pte)); + + for (i = 0; i < CONT_PTES; i++, ptep++, addr += PAGE_SIZE) { + pte_t ptent = __ptep_get_and_clear(mm, addr, ptep); + + if (pte_dirty(ptent)) + pte = pte_mkdirty(pte); + + if (pte_young(ptent)) + pte = pte_mkyoung(pte); + } + + __flush_tlb_range(&vma, start_addr, addr, PAGE_SIZE, true, 3); + + __set_ptes(mm, start_addr, start_ptep, pte, CONT_PTES); +} + +void __contpte_try_unfold(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte) +{ + /* + * We have already checked that the ptes are contiguous in + * contpte_try_unfold(), so just check that the mm is user space. + */ + + if (!mm_is_user(mm)) + return; + + pte = pte_mknoncont(pte); + contpte_convert(mm, addr, ptep, pte); +} +EXPORT_SYMBOL(__contpte_try_unfold); + +pte_t contpte_ptep_get(pte_t *ptep, pte_t orig_pte) +{ + /* + * Gather access/dirty bits, which may be populated in any of the ptes + * of the contig range. We are guarranteed to be holding the PTL, so any + * contiguous range cannot be unfolded or otherwise modified under our + * feet. + */ + + pte_t pte; + int i; + + ptep = contpte_align_down(ptep); + + for (i = 0; i < CONT_PTES; i++, ptep++) { + pte = __ptep_get(ptep); + + if (pte_dirty(pte)) + orig_pte = pte_mkdirty(orig_pte); + + if (pte_young(pte)) + orig_pte = pte_mkyoung(orig_pte); + } + + return orig_pte; +} +EXPORT_SYMBOL(contpte_ptep_get); + +pte_t contpte_ptep_get_lockless(pte_t *orig_ptep) +{ + /* + * Gather access/dirty bits, which may be populated in any of the ptes + * of the contig range. We may not be holding the PTL, so any contiguous + * range may be unfolded/modified/refolded under our feet. Therefore we + * ensure we read a _consistent_ contpte range by checking that all ptes + * in the range are valid and have CONT_PTE set, that all pfns are + * contiguous and that all pgprots are the same (ignoring access/dirty). + * If we find a pte that is not consistent, then we must be racing with + * an update so start again. If the target pte does not have CONT_PTE + * set then that is considered consistent on its own because it is not + * part of a contpte range. + */ + + pgprot_t orig_prot; + unsigned long pfn; + pte_t orig_pte; + pgprot_t prot; + pte_t *ptep; + pte_t pte; + int i; + +retry: + orig_pte = __ptep_get(orig_ptep); + + if (!pte_valid_cont(orig_pte)) + return orig_pte; + + orig_prot = pte_pgprot(pte_mkold(pte_mkclean(orig_pte))); + ptep = contpte_align_down(orig_ptep); + pfn = pte_pfn(orig_pte) - (orig_ptep - ptep); + + for (i = 0; i < CONT_PTES; i++, ptep++, pfn++) { + pte = __ptep_get(ptep); + prot = pte_pgprot(pte_mkold(pte_mkclean(pte))); + + if (!pte_valid_cont(pte) || + pte_pfn(pte) != pfn || + pgprot_val(prot) != pgprot_val(orig_prot)) + goto retry; + + if (pte_dirty(pte)) + orig_pte = pte_mkdirty(orig_pte); + + if (pte_young(pte)) + orig_pte = pte_mkyoung(orig_pte); + } + + return orig_pte; +} +EXPORT_SYMBOL(contpte_ptep_get_lockless); + +void contpte_set_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte, unsigned int nr) +{ + unsigned long next; + unsigned long end; + unsigned long pfn; + pgprot_t prot; + + /* + * The set_ptes() spec guarantees that when nr > 1, the initial state of + * all ptes is not-present. Therefore we never need to unfold or + * otherwise invalidate a range before we set the new ptes. + * contpte_set_ptes() should never be called for nr < 2. + */ + VM_WARN_ON(nr == 1); + + if (!mm_is_user(mm)) + return __set_ptes(mm, addr, ptep, pte, nr); + + end = addr + (nr << PAGE_SHIFT); + pfn = pte_pfn(pte); + prot = pte_pgprot(pte); + + do { + next = pte_cont_addr_end(addr, end); + nr = (next - addr) >> PAGE_SHIFT; + pte = pfn_pte(pfn, prot); + + if (((addr | next | (pfn << PAGE_SHIFT)) & ~CONT_PTE_MASK) == 0) + pte = pte_mkcont(pte); + else + pte = pte_mknoncont(pte); + + __set_ptes(mm, addr, ptep, pte, nr); + + addr = next; + ptep += nr; + pfn += nr; + + } while (addr != end); +} +EXPORT_SYMBOL(contpte_set_ptes); + +int contpte_ptep_test_and_clear_young(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep) +{ + /* + * ptep_clear_flush_young() technically requires us to clear the access + * flag for a _single_ pte. However, the core-mm code actually tracks + * access/dirty per folio, not per page. And since we only create a + * contig range when the range is covered by a single folio, we can get + * away with clearing young for the whole contig range here, so we avoid + * having to unfold. + */ + + int young = 0; + int i; + + ptep = contpte_align_down(ptep); + addr = ALIGN_DOWN(addr, CONT_PTE_SIZE); + + for (i = 0; i < CONT_PTES; i++, ptep++, addr += PAGE_SIZE) + young |= __ptep_test_and_clear_young(vma, addr, ptep); + + return young; +} +EXPORT_SYMBOL(contpte_ptep_test_and_clear_young); + +int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep) +{ + int young; + + young = contpte_ptep_test_and_clear_young(vma, addr, ptep); + + if (young) { + /* + * See comment in __ptep_clear_flush_young(); same rationale for + * eliding the trailing DSB applies here. + */ + addr = ALIGN_DOWN(addr, CONT_PTE_SIZE); + __flush_tlb_range_nosync(vma, addr, addr + CONT_PTE_SIZE, + PAGE_SIZE, true, 3); + } + + return young; +} +EXPORT_SYMBOL(contpte_ptep_clear_flush_young); + +int contpte_ptep_set_access_flags(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + pte_t entry, int dirty) +{ + unsigned long start_addr; + pte_t orig_pte; + int i; + + /* + * Gather the access/dirty bits for the contiguous range. If nothing has + * changed, its a noop. + */ + orig_pte = pte_mknoncont(ptep_get(ptep)); + if (pte_val(orig_pte) == pte_val(entry)) + return 0; + + /* + * We can fix up access/dirty bits without having to unfold the contig + * range. But if the write bit is changing, we must unfold. + */ + if (pte_write(orig_pte) == pte_write(entry)) { + /* + * For HW access management, we technically only need to update + * the flag on a single pte in the range. But for SW access + * management, we need to update all the ptes to prevent extra + * faults. Avoid per-page tlb flush in __ptep_set_access_flags() + * and instead flush the whole range at the end. + */ + ptep = contpte_align_down(ptep); + start_addr = addr = ALIGN_DOWN(addr, CONT_PTE_SIZE); + + for (i = 0; i < CONT_PTES; i++, ptep++, addr += PAGE_SIZE) + __ptep_set_access_flags(vma, addr, ptep, entry, 0); + + if (dirty) + __flush_tlb_range(vma, start_addr, addr, + PAGE_SIZE, true, 3); + } else { + __contpte_try_unfold(vma->vm_mm, addr, ptep, orig_pte); + __ptep_set_access_flags(vma, addr, ptep, entry, dirty); + } + + return 1; +} +EXPORT_SYMBOL(contpte_ptep_set_access_flags); From patchwork Fri Feb 2 08:07:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13542355 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3E6BDC47258 for ; Fri, 2 Feb 2024 08:09:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B1CB66B00A9; Fri, 2 Feb 2024 03:09:35 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A592A6B00AB; Fri, 2 Feb 2024 03:09:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 685366B00AC; Fri, 2 Feb 2024 03:09:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 55B836B00A9 for ; Fri, 2 Feb 2024 03:09:35 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 3258E801D1 for ; Fri, 2 Feb 2024 08:09:35 +0000 (UTC) X-FDA: 81746139510.01.664E4CC Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf12.hostedemail.com (Postfix) with ESMTP id 7FBF040010 for ; Fri, 2 Feb 2024 08:09:33 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf12.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706861373; a=rsa-sha256; cv=none; b=FYCyWCU1WsfbJd4gdRefcwJSRAg3n4luJoeGtvBzgELmaAJjrwOlz2QyBxp+c9C2ktmvOA y4nxUpSWu65rdQz1ILLN7nlkrbFDBWdUlS0Uujvo0MMIX6lq9HO9f/9pPBcKFEtcjQaRT6 d/TyuPpBTY+80f42EQgWedqF6ewH0y8= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf12.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706861373; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=c/4EEJu2gQomaE2sg7y+ahoqZodMpQ+lo82yJLdFja4=; b=gn9tifKuQ7GF3vYSbAP69CMBCUCxO9eiX9HYAJAFSyqejARHgMJ+sd5k8+GeQgZJKZvJbB oV38PYxzVINcbrzq8xc/9PW/msnLl/lHt15ODx9wGd1W0S2ALepPhAFJ4bB7DrPYOQOVpr RGRkmTik7dsiIxn0rK0eK0+ubX7UhxE= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 1EEE91BCB; Fri, 2 Feb 2024 00:10:15 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 29FC73F8A4; Fri, 2 Feb 2024 00:09:29 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v5 20/25] arm64/mm: Implement new wrprotect_ptes() batch API Date: Fri, 2 Feb 2024 08:07:51 +0000 Message-Id: <20240202080756.1453939-21-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240202080756.1453939-1-ryan.roberts@arm.com> References: <20240202080756.1453939-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 7FBF040010 X-Stat-Signature: 5c1wasi9jk4bob378tbokn6g5xtdbyph X-Rspam-User: X-HE-Tag: 1706861373-616869 X-HE-Meta: U2FsdGVkX19EfbY3UU1Zrh22rvgVfG1QpqH0vl+gTiYp3tyeyJhAE10XHw1LT3OH9aqDbul5Gs+HzEQ4SErqvmfHpKJOuHCRVbTVPPvw5CqPK/AUNUePv4KTZG8PP+dC+MvlEOnQpFrGdADyiATszI5vq6NqVxWYbwkzen1X1vUeiMr6Z54mdLZ60rUmCHlAwedr4KZgMbba0G4fNJFN24WhJTwtxFCQl3itLE9bt3NfgUc3pVfQcwTPJ9I/p5F5AZDCXDDoqav/yw9TG6NLw5MLcdQ9gysmBMRlesf88XNPgghO2IuKe7G3EQxffwiBWMET1ton81ZY+tx8BQinJMWI5JXfBN4rHNyInoOrVC+9Gmt8HuLMaTZomATnBpmacjBJYEQc/pzPgoi/xeC+qWRsrUugIqVQnkfn7+VJnU9hA4YHT8acdeP3wcWTvg/eRhA/UhFKU1JSUSzr0KLiQ5g0NWuDjf4qI9/8/R9U9X5esS/cNzM8IPneT815LjAGJcUQj0RPKMxbAFqicq853g0fGffxnm8dd+dDtlFTG5a3kvuGBJiDR0cChCqJRnrf9yNiQsnTF09rXj4TPQd8Ngt0QbFOYekyDcKksy8Kw0zebmoQ/Wm1AH7WrZrhZj6tswRiFAyzQy+6JiDXHQhBGtf2BfFTZy7dq9XqFC8LRAna8Xxe0vmQshCUIcNxW1F20JpN4wvVYUU/gIIoyyzwyAWtuVsnwKoPz26uZenISCSSHiPUYodRkzh37UY7Pa8cjfVmcL1MHh/x+Sn6WCeV1fXIfaGH4yIGTVtwdo6LSh+7jzLOz1h4307lTmeIiKE5Updiy//2QOvebO8OD0NukLoH+XXBfAySPcaG4iAP15/VvsRzdQTU8mzZ+P2qIuOoAcl0cGgHtLQjTj8B0lPVv15fdg9pwx1UAxuKgwLMSGlZKvDh/tpXJYdfW5FJFd8n1uGSzDmZQnJAOJMHBQZ hdFaf8bJ tjk1VVtB8vOiYOtJnRtF1qfFxXGdHc7zQ9UBIsdHHxGUeBR0AD6IlAd26e0QhNNUHiskT7rB++JpNSxkmo4xq0aeYJJKGbXqc22X6d/J3MdclzhAc9IBjICMN5fgmTDu/crIHvZ6y6tlS5tjLcUGhkITs2ACWdjfzuJcZjn6Jpd/tk+omx42zdK0y8p92IltvBRCTtiUZ/fZxiTuI9iePyUGRpI+87duhmCS5vDQbyFGNLlvup/RrjfEeEh3VYAkxsO5qMFCOWR2NX63eKZ5iWEqylfpZmea1adiLq8ysZMOYY/M= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Optimize the contpte implementation to fix some of the fork performance regression introduced by the initial contpte commit. Subsequent patches will solve it entirely. During fork(), any private memory in the parent must be write-protected. Previously this was done 1 PTE at a time. But the core-mm supports batched wrprotect via the new wrprotect_ptes() API. So let's implement that API and for fully covered contpte mappings, we no longer need to unfold the contpte. This has 2 benefits: - reduced unfolding, reduces the number of tlbis that must be issued. - The memory remains contpte-mapped ("folded") in the parent, so it continues to benefit from the more efficient use of the TLB after the fork. The optimization to wrprotect a whole contpte block without unfolding is possible thanks to the tightening of the Arm ARM in respect to the definition and behaviour when 'Misprogramming the Contiguous bit'. See section D21194 at https://developer.arm.com/documentation/102105/latest/ Tested-by: John Hubbard Signed-off-by: Ryan Roberts --- arch/arm64/include/asm/pgtable.h | 61 ++++++++++++++++++++++++++------ arch/arm64/mm/contpte.c | 35 ++++++++++++++++++ 2 files changed, 86 insertions(+), 10 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 34892a95403d..c07f0d563733 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -978,16 +978,12 @@ static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, } #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ -/* - * __ptep_set_wrprotect - mark read-only while trasferring potential hardware - * dirty status (PTE_DBM && !PTE_RDONLY) to the software PTE_DIRTY bit. - */ -static inline void __ptep_set_wrprotect(struct mm_struct *mm, - unsigned long address, pte_t *ptep) +static inline void ___ptep_set_wrprotect(struct mm_struct *mm, + unsigned long address, pte_t *ptep, + pte_t pte) { - pte_t old_pte, pte; + pte_t old_pte; - pte = __ptep_get(ptep); do { old_pte = pte; pte = pte_wrprotect(pte); @@ -996,6 +992,25 @@ static inline void __ptep_set_wrprotect(struct mm_struct *mm, } while (pte_val(pte) != pte_val(old_pte)); } +/* + * __ptep_set_wrprotect - mark read-only while trasferring potential hardware + * dirty status (PTE_DBM && !PTE_RDONLY) to the software PTE_DIRTY bit. + */ +static inline void __ptep_set_wrprotect(struct mm_struct *mm, + unsigned long address, pte_t *ptep) +{ + ___ptep_set_wrprotect(mm, address, ptep, __ptep_get(ptep)); +} + +static inline void __wrprotect_ptes(struct mm_struct *mm, unsigned long address, + pte_t *ptep, unsigned int nr) +{ + unsigned int i; + + for (i = 0; i < nr; i++, address += PAGE_SIZE, ptep++) + __ptep_set_wrprotect(mm, address, ptep); +} + #ifdef CONFIG_TRANSPARENT_HUGEPAGE #define __HAVE_ARCH_PMDP_SET_WRPROTECT static inline void pmdp_set_wrprotect(struct mm_struct *mm, @@ -1156,6 +1171,8 @@ extern int contpte_ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); extern int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); +extern void contpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr); extern int contpte_ptep_set_access_flags(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t entry, int dirty); @@ -1269,12 +1286,35 @@ static inline int ptep_clear_flush_young(struct vm_area_struct *vma, return contpte_ptep_clear_flush_young(vma, addr, ptep); } +#define wrprotect_ptes wrprotect_ptes +static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr) +{ + if (likely(nr == 1)) { + /* + * Optimization: wrprotect_ptes() can only be called for present + * ptes so we only need to check contig bit as condition for + * unfold, and we can remove the contig bit from the pte we read + * to avoid re-reading. This speeds up fork() which is sensitive + * for order-0 folios. Equivalent to contpte_try_unfold(). + */ + pte_t orig_pte = __ptep_get(ptep); + + if (unlikely(pte_cont(orig_pte))) { + __contpte_try_unfold(mm, addr, ptep, orig_pte); + orig_pte = pte_mknoncont(orig_pte); + } + ___ptep_set_wrprotect(mm, addr, ptep, orig_pte); + } else { + contpte_wrprotect_ptes(mm, addr, ptep, nr); + } +} + #define __HAVE_ARCH_PTEP_SET_WRPROTECT static inline void ptep_set_wrprotect(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { - contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); - __ptep_set_wrprotect(mm, addr, ptep); + wrprotect_ptes(mm, addr, ptep, 1); } #define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS @@ -1306,6 +1346,7 @@ static inline int ptep_set_access_flags(struct vm_area_struct *vma, #define ptep_clear_flush_young __ptep_clear_flush_young #define __HAVE_ARCH_PTEP_SET_WRPROTECT #define ptep_set_wrprotect __ptep_set_wrprotect +#define wrprotect_ptes __wrprotect_ptes #define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS #define ptep_set_access_flags __ptep_set_access_flags diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c index bfb50e6b44c7..c85e64baf03b 100644 --- a/arch/arm64/mm/contpte.c +++ b/arch/arm64/mm/contpte.c @@ -23,6 +23,23 @@ static inline pte_t *contpte_align_down(pte_t *ptep) return (pte_t *)(ALIGN_DOWN((unsigned long)ptep >> 3, CONT_PTES) << 3); } +static void contpte_try_unfold_partial(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr) +{ + /* + * Unfold any partially covered contpte block at the beginning and end + * of the range. + */ + + if (ptep != contpte_align_down(ptep) || nr < CONT_PTES) + contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); + + if (ptep + nr != contpte_align_down(ptep + nr)) + contpte_try_unfold(mm, addr + PAGE_SIZE * (nr - 1), + ptep + nr - 1, + __ptep_get(ptep + nr - 1)); +} + static void contpte_convert(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte) { @@ -236,6 +253,24 @@ int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, } EXPORT_SYMBOL(contpte_ptep_clear_flush_young); +void contpte_wrprotect_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr) +{ + /* + * If wrprotecting an entire contig range, we can avoid unfolding. Just + * set wrprotect and wait for the later mmu_gather flush to invalidate + * the tlb. Until the flush, the page may or may not be wrprotected. + * After the flush, it is guarranteed wrprotected. If its a partial + * range though, we must unfold, because we can't have a case where + * CONT_PTE is set but wrprotect applies to a subset of the PTEs; this + * would cause it to continue to be unpredictable after the flush. + */ + + contpte_try_unfold_partial(mm, addr, ptep, nr); + __wrprotect_ptes(mm, addr, ptep, nr); +} +EXPORT_SYMBOL(contpte_wrprotect_ptes); + int contpte_ptep_set_access_flags(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t entry, int dirty) From patchwork Fri Feb 2 08:07:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13542356 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 900A1C4828E for ; Fri, 2 Feb 2024 08:09:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1CAC96B00AB; Fri, 2 Feb 2024 03:09:39 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 17C966B00AD; Fri, 2 Feb 2024 03:09:39 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 043766B00AE; Fri, 2 Feb 2024 03:09:39 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id EA4666B00AB for ; Fri, 2 Feb 2024 03:09:38 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id CCE23A25D1 for ; Fri, 2 Feb 2024 08:09:38 +0000 (UTC) X-FDA: 81746139636.04.272EDFE Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf09.hostedemail.com (Postfix) with ESMTP id 43002140004 for ; Fri, 2 Feb 2024 08:09:37 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf09.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706861377; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=exgJk3Iib5rVvkUc8EQu+oEfH2dafVkQZy8I6MW4orI=; b=KPrPljD28ZTgz36Q2iOigx8mW+GohL1fUoRRRQ93Xdr6Alr3/DafASo7iY77KHw0+FZ4qP jv9OoQM9k4FljJAAe/gsmcxlh6l5UnyzNUBTcvFH1mWhlvcntvnTyKVYi0jenW184QgjaK 6yDIoVQJtA5KwVxrrDsEe4DyEgIfaQE= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf09.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706861377; a=rsa-sha256; cv=none; b=XSZLSW1JS20jWxE3DVhOdd1fAKBfRoCHrYS+TU5ht0ZAr9EzXrOcxSjoy2aqtRDUX8vrAo GxeL/mx4WxxVBazbIMDU6lCcHXO/9ZBosaNZk4PBqhKmB56U575zPA/XUtJEfmeprYaSYI ex3LzAFRKNZoIIOrJRXxQuVKp68pQ4I= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id E4B631BCA; Fri, 2 Feb 2024 00:10:18 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 047993F5A1; Fri, 2 Feb 2024 00:09:32 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v5 21/25] arm64/mm: Implement new [get_and_]clear_full_ptes() batch APIs Date: Fri, 2 Feb 2024 08:07:52 +0000 Message-Id: <20240202080756.1453939-22-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240202080756.1453939-1-ryan.roberts@arm.com> References: <20240202080756.1453939-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 43002140004 X-Stat-Signature: 4eyc11bf7jpk4ke1hcuejpxn3fgoi95i X-HE-Tag: 1706861377-157740 X-HE-Meta: U2FsdGVkX18T5f3juSUCeFC+aNOp27xoGfTGaumRYhwkIjzxTin2BIvwZEBbhWhqWo/OLL0HtBdF8sHVCgCBq0rvmAqhtYrDB68GHAYoOjj7jrxvdPQHUdToT9AkmSNeWjcBm3nG+ulbH+2ih8dP7gEBPyRIijRWMrxVFkAP2hE1azaV/qXwdMJlE8A2UfvEu9oa6mGe9dC1NQNpidBJhu9kyDJT/UNvXMrCKX9lEkkyiVIKv9bPNzSLl6aIVHTy7bJi1Mzog07x8SAy5O2hqlD6WNs53OqsOhUN+ahLrAOxZS621xxzBEDwEQXtnxKZMw/ULRcKIOmKPAyWwMLmADPKXvOCP/tAoNnPnjUV4GZsrDwIlOKdjzSP8P4DiDIomwZmkG+/NayfAd5jLnUkVd8szfXVnAaaVFi8toLtPycHJ7vX9tZJ5u8UjkgVcWKT+C9krEKpFFVGVW3vr47YSULI7sdINXKfq17HaFlRdfRuizerPrQsot6i6G5SdiVWuhsKRT09Dn4IPmQcetbqzgL9S8M0Qi1tAcVW0vjxHddZhpeI+d87z03TwC9JbZgGNvJya/3MdvxS8wojcIZOhxe6aQR1qkyNvEc+VIgFJiBYF35FoMTYaqv7ZWNrVe49rReygu4DkrZQZ49qLEgrmuPCIQFQPgn+y0ADq9O/t4d9G6/mS9HNTIVOXgqs+L0rdaWRRQXhWW4D2HrtO/rmP+wjJg8fKaRsbuMklU+veaXH5H4w/8u+xQN1XamiHnKbuFSrD97j/uPvP4nRVU32vCDPulYmSfaHsN7WTEhku0hfVm0Om7H+wy2+/a3M9+Sd3B5vzfax2QrS70TSlgmiJWdcHwIolacSrJmzodTANeRoIQITbWMzTA0d8Y3LxcHZmth1PsPf82ts9gTqJvNccfKwWu+srcRWeki5VxbgGZcbx6gk5W/wRggEtWbRaGsnAN9rvWjIEmfQbinCrpi 9qg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Optimize the contpte implementation to fix some of the exit/munmap/dontneed performance regression introduced by the initial contpte commit. Subsequent patches will solve it entirely. During exit(), munmap() or madvise(MADV_DONTNEED), mappings must be cleared. Previously this was done 1 PTE at a time. But the core-mm supports batched clear via the new [get_and_]clear_full_ptes() APIs. So let's implement those APIs and for fully covered contpte mappings, we no longer need to unfold the contpte. This significantly reduces unfolding operations, reducing the number of tlbis that must be issued. Tested-by: John Hubbard Signed-off-by: Ryan Roberts --- arch/arm64/include/asm/pgtable.h | 67 ++++++++++++++++++++++++++++++++ arch/arm64/mm/contpte.c | 17 ++++++++ 2 files changed, 84 insertions(+) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index c07f0d563733..ad04adb7b87f 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -965,6 +965,37 @@ static inline pte_t __ptep_get_and_clear(struct mm_struct *mm, return pte; } +static inline void __clear_full_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr, int full) +{ + for (;;) { + __ptep_get_and_clear(mm, addr, ptep); + if (--nr == 0) + break; + ptep++; + addr += PAGE_SIZE; + } +} + +static inline pte_t __get_and_clear_full_ptes(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, + unsigned int nr, int full) +{ + pte_t pte, tmp_pte; + + pte = __ptep_get_and_clear(mm, addr, ptep); + while (--nr) { + ptep++; + addr += PAGE_SIZE; + tmp_pte = __ptep_get_and_clear(mm, addr, ptep); + if (pte_dirty(tmp_pte)) + pte = pte_mkdirty(pte); + if (pte_young(tmp_pte)) + pte = pte_mkyoung(pte); + } + return pte; +} + #ifdef CONFIG_TRANSPARENT_HUGEPAGE #define __HAVE_ARCH_PMDP_HUGE_GET_AND_CLEAR static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, @@ -1167,6 +1198,11 @@ extern pte_t contpte_ptep_get(pte_t *ptep, pte_t orig_pte); extern pte_t contpte_ptep_get_lockless(pte_t *orig_ptep); extern void contpte_set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte, unsigned int nr); +extern void contpte_clear_full_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr, int full); +extern pte_t contpte_get_and_clear_full_ptes(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, + unsigned int nr, int full); extern int contpte_ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); extern int contpte_ptep_clear_flush_young(struct vm_area_struct *vma, @@ -1254,6 +1290,35 @@ static inline void pte_clear(struct mm_struct *mm, __pte_clear(mm, addr, ptep); } +#define clear_full_ptes clear_full_ptes +static inline void clear_full_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr, int full) +{ + if (likely(nr == 1)) { + contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); + __clear_full_ptes(mm, addr, ptep, nr, full); + } else { + contpte_clear_full_ptes(mm, addr, ptep, nr, full); + } +} + +#define get_and_clear_full_ptes get_and_clear_full_ptes +static inline pte_t get_and_clear_full_ptes(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, + unsigned int nr, int full) +{ + pte_t pte; + + if (likely(nr == 1)) { + contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); + pte = __get_and_clear_full_ptes(mm, addr, ptep, nr, full); + } else { + pte = contpte_get_and_clear_full_ptes(mm, addr, ptep, nr, full); + } + + return pte; +} + #define __HAVE_ARCH_PTEP_GET_AND_CLEAR static inline pte_t ptep_get_and_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep) @@ -1338,6 +1403,8 @@ static inline int ptep_set_access_flags(struct vm_area_struct *vma, #define set_pte __set_pte #define set_ptes __set_ptes #define pte_clear __pte_clear +#define clear_full_ptes __clear_full_ptes +#define get_and_clear_full_ptes __get_and_clear_full_ptes #define __HAVE_ARCH_PTEP_GET_AND_CLEAR #define ptep_get_and_clear __ptep_get_and_clear #define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c index c85e64baf03b..80346108450b 100644 --- a/arch/arm64/mm/contpte.c +++ b/arch/arm64/mm/contpte.c @@ -207,6 +207,23 @@ void contpte_set_ptes(struct mm_struct *mm, unsigned long addr, } EXPORT_SYMBOL(contpte_set_ptes); +void contpte_clear_full_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned int nr, int full) +{ + contpte_try_unfold_partial(mm, addr, ptep, nr); + __clear_full_ptes(mm, addr, ptep, nr, full); +} +EXPORT_SYMBOL(contpte_clear_full_ptes); + +pte_t contpte_get_and_clear_full_ptes(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, + unsigned int nr, int full) +{ + contpte_try_unfold_partial(mm, addr, ptep, nr); + return __get_and_clear_full_ptes(mm, addr, ptep, nr, full); +} +EXPORT_SYMBOL(contpte_get_and_clear_full_ptes); + int contpte_ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep) { From patchwork Fri Feb 2 08:07:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13542357 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B448BC48292 for ; Fri, 2 Feb 2024 08:09:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 43C386B00AD; Fri, 2 Feb 2024 03:09:43 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3EC3C6B00AF; Fri, 2 Feb 2024 03:09:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2B43C6B00B0; Fri, 2 Feb 2024 03:09:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 1CF8E6B00AD for ; Fri, 2 Feb 2024 03:09:43 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id E2706120659 for ; Fri, 2 Feb 2024 08:09:42 +0000 (UTC) X-FDA: 81746139804.05.C48E06D Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf20.hostedemail.com (Postfix) with ESMTP id 152F71C0021 for ; Fri, 2 Feb 2024 08:09:40 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=none; spf=pass (imf20.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706861381; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RG2xpbHo+BZWR1EUw1E88djCi8W1k8Ens4WKUhSzkA8=; b=E0nOH/4eS29Rjng9ljhsG8BweM1PpjXLfxW0DDS10Jgou6bhlqkhuNFpEra2DrbePfkMQp bxXqQFNgdqWT4z/dLN6mnlzv8JP86jWKWs2ywmSrtLAUZFLNx/MazQmQ25TrLu7hTTS3Z4 Z2C9GiaWhcRFTZdAqs+m52v20CriDYQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706861381; a=rsa-sha256; cv=none; b=mgV7ApM6O2pIFPSEPxwKFtqWx0iyeYe6ucIHKkf5dYohdEF0nO39TyfmdPM6+GhrhP+iW4 1R/jq2cwg/Y71GeCEYzR6AK77LQQtqPl7JMuMYUA9vDOft5UfEaWY8lJGgYba1+fBOFCMz tZT+7iqPNmNRJcF8nNYUirLD2VZCAbQ= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=none; spf=pass (imf20.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id B640A1BD0; Fri, 2 Feb 2024 00:10:22 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id CA6BF3F5A1; Fri, 2 Feb 2024 00:09:36 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v5 22/25] mm: Add pte_batch_hint() to reduce scanning in folio_pte_batch() Date: Fri, 2 Feb 2024 08:07:53 +0000 Message-Id: <20240202080756.1453939-23-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240202080756.1453939-1-ryan.roberts@arm.com> References: <20240202080756.1453939-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Stat-Signature: 6d1cw7bh1m5dgeqohqrfkzyzf7wh9wr8 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 152F71C0021 X-Rspam-User: X-HE-Tag: 1706861380-112572 X-HE-Meta: U2FsdGVkX1+CcJT+VKsxVPwJY+CvL5Pc9v5OsUA2SXD44BITqvqE2cmb4AYHIxZLiyhoe6P0HC5xaXNL+Kv8NSE1n3TwcdPIlvnd0lpCtpdEKSgXBW2ywoeIRffeYiUcm0pmoosvNsy07DHfeal38HVt7hkmqvMdCgQMQqVk2vixGIFgPGRQ+YULUvDZ3bfTDhosnXrbaxPgYOHSrLq1Y4MHRKildxOaNaJu258zZcTI2/LpwL2PqT8D76YUwdHhCPFMsiTAaItDiM5LXqyag41ocWyZmdI9zwmPXVPGaCdZlqHFBOuwct9gixEaxsu3ZJhQPj7EONxEjpBSMZDsTyhkbV4rOmeBSBeKUih3zKUmQDs3lfkrHtZ7E1rumOmE+Ml4v9CN7EzaB+1hHAC6CMPsw6nm+qKR/vkWy1Hj3fQFTsrcLlOgK9VA7YjAcoJQ1vUH3bFOuaPMYPoTlt2prHiXKTH//ZScrErHVyRN7o5YlCpllvWShGJvM8hsetEkyZUnpIFiVmc8GNEmmj0vboqbr/D6wrpzbTd3oX6peaKaaZ5Asy9mo6OAH4Sfae0TnQKbAsBM+2A42qQHvUEPyLwDqrvsexUQVBrmuotY3feFL7otYmj1ZvqyoJdVVafStH2vyKgAxwGrQA7mokkULNKkXHpWWzOX0KqH0mJv5seVmrBGz5/L7k3AydlVUrl+d5o1p7z5hpCz+eDtau7S3IMqLyZFsaEOp6+EEOWnA+2NKO0weyKielFs9VgeXOh8nXJc9o2/q+abacMwH+0X7sOzA080zJBYoCpeNUHDIQ8R9GCQDtOBlrfMNjORsZPX9CWxPwq5RfvYY5EQ0SUAgRcHsuiTxf9m+Eu+LG8BAiH9Mwn1z7SLevLQI+RLR8+sXoyThYHaTMQndLgdZGWMFeohiD/Qsl4ekSKGQEG3dk9oOnXZ4XFp+36zwxGFJk+8WeMYF/8oKAeEV+QXsQo iffg4PGO BsLs9FcxxSzpclF8O65D20g0fYXYWuT9E7eiymcQ1KgT+iLh7dT6YmKxJugYw/Q3+XOB6U0/pYoxNB7Ney1/IhqlNves5HBww4SAzAwC7jn96u5OFBw0lPVVq5VJD7Q3aqR9OKm20a03LPoij5juekOBIyMrwXYUdVhobf94yRkjvwG8fZt7ou5bUa/suWpaa5AYf8uHlH+aSN9HdWRiFl+teq0gN2ThPn1iVngyVTMV+4sm6SfyvTcnd+gb/KyBq9JjU4u0/0BCOxcJCZfI2sSq66IbKEwdL8KrQAoGaAwR9WqU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Some architectures (e.g. arm64) can tell from looking at a pte, if some follow-on ptes also map contiguous physical memory with the same pgprot. (for arm64, these are contpte mappings). Take advantage of this knowledge to optimize folio_pte_batch() so that it can skip these ptes when scanning to create a batch. By default, if an arch does not opt-in, folio_pte_batch() returns a compile-time 1, so the changes are optimized out and the behaviour is as before. arm64 will opt-in to providing this hint in the next patch, which will greatly reduce the cost of ptep_get() when scanning a range of contptes. Tested-by: John Hubbard Signed-off-by: Ryan Roberts Acked-by: David Hildenbrand --- include/linux/pgtable.h | 18 ++++++++++++++++++ mm/memory.c | 20 +++++++++++++------- 2 files changed, 31 insertions(+), 7 deletions(-) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 50f32cccbd92..cba31f177d27 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -212,6 +212,24 @@ static inline int pmd_dirty(pmd_t pmd) #define arch_flush_lazy_mmu_mode() do {} while (0) #endif +#ifndef pte_batch_hint +/** + * pte_batch_hint - Number of pages that can be added to batch without scanning. + * @ptep: Page table pointer for the entry. + * @pte: Page table entry. + * + * Some architectures know that a set of contiguous ptes all map the same + * contiguous memory with the same permissions. In this case, it can provide a + * hint to aid pte batching without the core code needing to scan every pte. + * + * May be overridden by the architecture, else pte_batch_hint is always 1. + */ +static inline unsigned int pte_batch_hint(pte_t *ptep, pte_t pte) +{ + return 1; +} +#endif + #ifndef pte_advance_pfn static inline pte_t pte_advance_pfn(pte_t pte, unsigned long nr) { diff --git a/mm/memory.c b/mm/memory.c index 65fbe4f886c1..902665b27702 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -988,16 +988,21 @@ static inline int folio_pte_batch(struct folio *folio, unsigned long addr, { unsigned long folio_end_pfn = folio_pfn(folio) + folio_nr_pages(folio); const pte_t *end_ptep = start_ptep + max_nr; - pte_t expected_pte = __pte_batch_clear_ignored(pte_advance_pfn(pte, 1), flags); - pte_t *ptep = start_ptep + 1; + pte_t expected_pte = __pte_batch_clear_ignored(pte, flags); + pte_t *ptep = start_ptep; bool writable; + int nr; if (any_writable) *any_writable = false; VM_WARN_ON_FOLIO(!pte_present(pte), folio); - while (ptep != end_ptep) { + nr = pte_batch_hint(ptep, pte); + expected_pte = pte_advance_pfn(expected_pte, nr); + ptep += nr; + + while (ptep < end_ptep) { pte = ptep_get(ptep); if (any_writable) writable = !!pte_write(pte); @@ -1011,17 +1016,18 @@ static inline int folio_pte_batch(struct folio *folio, unsigned long addr, * corner cases the next PFN might fall into a different * folio. */ - if (pte_pfn(pte) == folio_end_pfn) + if (pte_pfn(pte) >= folio_end_pfn) break; if (any_writable) *any_writable |= writable; - expected_pte = pte_advance_pfn(expected_pte, 1); - ptep++; + nr = pte_batch_hint(ptep, pte); + expected_pte = pte_advance_pfn(expected_pte, nr); + ptep += nr; } - return ptep - start_ptep; + return min(ptep - start_ptep, max_nr); } /* From patchwork Fri Feb 2 08:07:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13542358 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3A8EDC47258 for ; Fri, 2 Feb 2024 08:09:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C03A06B00AF; Fri, 2 Feb 2024 03:09:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BB4436B00B1; Fri, 2 Feb 2024 03:09:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A7C516B00B2; Fri, 2 Feb 2024 03:09:46 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 984F26B00AF for ; Fri, 2 Feb 2024 03:09:46 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 747E2140F38 for ; Fri, 2 Feb 2024 08:09:46 +0000 (UTC) X-FDA: 81746139972.08.CE75675 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf14.hostedemail.com (Postfix) with ESMTP id CB765100010 for ; Fri, 2 Feb 2024 08:09:44 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=none; spf=pass (imf14.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706861384; a=rsa-sha256; cv=none; b=ta2OrRgj8hmBn3g+tmxu5rXFV5ACbjxmBHvBuXTGbu2Z42GvgdtVVlPw1RBNy0nqoT0CxY 88J0vlDluV0fqrjz1zfgCt1S2XD/oFUIPsHl9mgNt/j0zskTV7KyE225aZbdxURuiBWhTF VfBbYghiZzDhWg25EQlaBZbsctLSqcU= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=none; spf=pass (imf14.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706861384; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=j94fI+UszkXDyyB5sUGdHbBs/R63iGXwtMWb4oxj3U4=; b=3hqAkLpj6CTjL5tCxCQafiupzCyL4xaQCcllZnbW/NKbSteFcYLoY+Msb/GX/YbB4W6TrC vnlZgGYJKNLhGzUNtBbtpgO2EMP9pHG2xIwEJoMtWOnRSQWpHZmY2YRiK4uMaHSzYeyEO6 zftFgkeni8tGXvVA1m4EQ9m9oPfURVo= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 87EF01BF3; Fri, 2 Feb 2024 00:10:26 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 9C0C43F5A1; Fri, 2 Feb 2024 00:09:40 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v5 23/25] arm64/mm: Implement pte_batch_hint() Date: Fri, 2 Feb 2024 08:07:54 +0000 Message-Id: <20240202080756.1453939-24-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240202080756.1453939-1-ryan.roberts@arm.com> References: <20240202080756.1453939-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: CB765100010 X-Stat-Signature: pq3zyq35pcexw9b6i6rnoqfcy3gki5sj X-HE-Tag: 1706861384-878312 X-HE-Meta: U2FsdGVkX18SobMpthYfuXKtlEQQZrxIbH8jsGf06Nlgyj7wF9VpTa2nxkQblj2ZXNpTU9m9tOnfzWyjAubyn69iXPX2bkalIaLyr02rSfLUKDcUSRuNdlPBoyuPkjIB3NTNIQPIF1uVe/WFlDZiJh0VV+rehrv2yMZAdcvo3LclM6L4+oCZBoLnNQPKLlofbBNrZ6YG2kWvo0QPWixwTDd/cvYlZyWwficK4uQ3uZ9EPy5IfpM0DNLeYzAd3WsAag9NFxGx27+ZCk46YxljwnAdAIbmo2/ALyzR+I5zbW2ju+OQQ7rw+XMPdeCR0k7VzDOifW5n/aAlf7mLodKVoAtd7wT2jBmvAfwouwh15uzUPT9XZil2l186ebTft49FIrmZMeIyQNJQ+NHyEVl0qoWgS0NlNgqPGEa/ypI26kmoLWl+mKpSMJF99zaEWVtkDFAXZQe6KuGvUai/bg3of3YYMsXrLOIYvjgsj/c1uEjYAzGi8HThz/NA7CeeUL13umJ4FA+MTg3bDhWKpFm5zl9ZG+H6anGxZ5Dq05zgsQZCDomGd2Nd4V41x3RVJzdCYFDn4bMwF7uWbTpBr5PSlY0hsSm+rz4xOZQGPgrL/bi/L2yOXtSzLZvFcniqgA4r8eL9uSa/sN7FsSzqB8jG4kJGeNnuNEHEZEajxCd3uVttBcQHW5rC4L2gcQZ4vUXe4jt9vhHiHZjFci6N927kRbViAuYK2GqH0dxk3n8QIkh0Sk+63stmX2fBa1ji5bNzpPZRzcZgkqarBovu9oPUeOYBqKkqgcGhP9B/07fYS6W7yQgifqFbyvJVTAkrxgOo4oOLvVHn8ZnlP2ED/ifYb7fbxiUWtJ7E9AY3MMh7y2Z1DphOhJJUnyKwkN+ur5on3e0jF6IhvzAiMtUot4KdmKcPH/BDVsqr1WDsL+75MF7dHT/aCBtIdjiRLR6zuakjcVa2tovTyguYjX+l2/S 7NTZpI/M /inbC94SZdipWixmP5w11yYGBlIwib80Ay0LxIj6htj0Nb0hBJZPXmwr0Ir0LUhXJyJulhuZd3pOzu5Yj2bOGmVSHNOZvapSZiu1qrgRAh0YnKe7r+EEilu/4s//RRrpHv2GjZaoEb9AOKOL2vEU1Jld9bkExnqf1/nS2BRuC/TA/4T8D21OSNeq1J4SwJtcJDL0rzt+ocDvk4ngwi5HvVjcZp9zwYA7i3X62i3GWqq6uYQ5+AsQMKMym2mhU3Lc4pldixjp4n8Odg/ntpITZDgNpeE8mkHe9N2sIscDAZ7C+1Ec= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When core code iterates over a range of ptes and calls ptep_get() for each of them, if the range happens to cover contpte mappings, the number of pte reads becomes amplified by a factor of the number of PTEs in a contpte block. This is because for each call to ptep_get(), the implementation must read all of the ptes in the contpte block to which it belongs to gather the access and dirty bits. This causes a hotspot for fork(), as well as operations that unmap memory such as munmap(), exit and madvise(MADV_DONTNEED). Fortunately we can fix this by implementing pte_batch_hint() which allows their iterators to skip getting the contpte tail ptes when gathering the batch of ptes to operate on. This results in the number of PTE reads returning to 1 per pte. Tested-by: John Hubbard Signed-off-by: Ryan Roberts Reviewed-by: David Hildenbrand Acked-by: Mark Rutland --- arch/arm64/include/asm/pgtable.h | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index ad04adb7b87f..353ea67b5d75 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1220,6 +1220,15 @@ static inline void contpte_try_unfold(struct mm_struct *mm, unsigned long addr, __contpte_try_unfold(mm, addr, ptep, pte); } +#define pte_batch_hint pte_batch_hint +static inline unsigned int pte_batch_hint(pte_t *ptep, pte_t pte) +{ + if (!pte_valid_cont(pte)) + return 1; + + return CONT_PTES - (((unsigned long)ptep >> 3) & (CONT_PTES - 1)); +} + /* * The below functions constitute the public API that arm64 presents to the * core-mm to manipulate PTE entries within their page tables (or at least this From patchwork Fri Feb 2 08:07:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13542359 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 425A7C47258 for ; Fri, 2 Feb 2024 08:09:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C79B76B0072; Fri, 2 Feb 2024 03:09:50 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C2A436B00B3; Fri, 2 Feb 2024 03:09:50 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B19566B00B4; Fri, 2 Feb 2024 03:09:50 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id A2B636B0072 for ; Fri, 2 Feb 2024 03:09:50 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 761CC1C18BC for ; Fri, 2 Feb 2024 08:09:50 +0000 (UTC) X-FDA: 81746140140.14.D5778AA Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf19.hostedemail.com (Postfix) with ESMTP id BE6B61A0016 for ; Fri, 2 Feb 2024 08:09:48 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf19.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706861388; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=yy+Swr2xbZSqDvn3ENPl3Nu9FCFS8f4PKsB1GCbCKzA=; b=NVDZrv/C0tmgSW9+tvJT3vofpVUAWlKJdIAm/E9/2M7bzkvm66bNbBsivtsyNG095B7mb8 4MB9jgYorC+DghuXAwlm4C1xAIZCRBIdwER2B7FU0DFpGqflbCK+je7lOB9JjC0kHNJ9gO f1S6jd4O31eMwm9zBOEBAnsw9ZATA2k= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf19.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706861388; a=rsa-sha256; cv=none; b=ZUtEhIojHNIaArtwNsoVKid2/OAW9j3hDRfUys+ck585R6rFN3U3HhSi3gBwRzyyR5cdcq 5VJlC9dpDnmKf9puacb8sfO9P0E2okinJkIXXZ3M1+YcBmMmatFJO0yw/28gAY+ueRGcea borW37f7VjVH78kp4pFroJibUrq+K/M= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 599431BF7; Fri, 2 Feb 2024 00:10:30 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 6D8E53F5A1; Fri, 2 Feb 2024 00:09:44 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v5 24/25] arm64/mm: __always_inline to improve fork() perf Date: Fri, 2 Feb 2024 08:07:55 +0000 Message-Id: <20240202080756.1453939-25-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240202080756.1453939-1-ryan.roberts@arm.com> References: <20240202080756.1453939-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: BE6B61A0016 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: oupp9qo3swiomde9gyihgoo3jzeoozit X-HE-Tag: 1706861388-894572 X-HE-Meta: U2FsdGVkX19OE4Qr+XZNvzR3/bnCbvlLUjQHpj/uKaJebrpCvO4TNuzm/qvODphJLplO+mlpcosLgUVKEeTzUMnZzI9WEyfTBkVIzwHnw03lPzpSymI8c5Y1aTZP1FrRI4qgb/spffdHZijTi1Isq/bDqTbVXiadPgoKIAP2A+tTAarfsiMrcbUI2DYYPGITR+NnHpe9tyGx+TSS50erKi1s0qe/tE4EDJnT4OqC7NTq69dM/BomfhvzlBEQFQnj56oPPTYQk3Pzv2yKQLK6hdZjtVKSv7tb+7WSDqUAVVptfmVixQj9t/7FTTLN2mTdyQj0Fz+uQapUSWSGzoqXaF3ZMzLFvEQj8BnLzJiq/W6zVel4kgs/+GJByjmh6Fz8iIbD4+aMBYJHcDtFdPNc21PO/Umwq5vBB/Qc1MNL6T62ceUH2+jCO6jyFTiYAVtyqikj05talJehCb5WlhPI8ZZVL6K4oETDE05T6aJRgJISOwLs2jT8n0kshkJ5K7U8WfzYqDbpge1ZhsUbA9TK09vEbP+ZJTGp0AkzZclNaOnpgHtlVImFYODiooWHmRw7uyl05IYFPf/2rlaNAUeHI0u3YRdbznJ6GX4+LneALMwr1loxhei1RamW/AMO/yB7w9GEDbl6xVwY6wm6QO1fcXzWvDlm6W933fbGMqvO5C9Lw7oLYKp2Xo0W9tnZRBDbiwpzjkMFodEi50Q4fnqySY7Ha0zDiS4Ic98Cp6VN03C7EME/Nj70mlu8iV2GlA+sQaAr+9s8ogIpUd/wWeLLZXz5iw4H7CsaLHuQvvlRRWMeTydg1qeNsFM68nXewptapdachnI1ELGmMPJwFHfug/Fwaqx3ipGDYFJ3xTTP+Ky38ryrC/H9xSUXnTIA/xaU8G5I/egnrHdNWKRZCl5S+7T5l4i08HYSLpNUyjmjaEl3oiHRRcLllXcre9yb3HmjfCMP0mimo/kuY0QCk71 Jno3luZO dWlOjW7pVE+RaoBYLvRWbuESC1Sv8XzuoOg+T/qlCPktS0wV41pVELI4xdRt09QAcWeAArSdZys4sbi8++dCy20KAusiDgLyPjg3IdSVJahqN9JIZ8U4xne5qLPhATjuYzkAvFZRsbM5FeYvRTOzFLjBIvhf1WNP2L10A4vOFY8Lfq1zkMbbIgkvfNzjICgGgD+NGzgTixfybn0SoGdCQKs1a0aWkc79Mel+ubxHZhKSc4m5xG6mXz0gyMZv6sP+4EsBHIWYP9Wr2iSJZ8COqmbEZAw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: As set_ptes() and wrprotect_ptes() become a bit more complex, the compiler may choose not to inline them. But this is critical for fork() performance. So mark the functions, along with contpte_try_unfold() which is called by them, as __always_inline. This is worth ~1% on the fork() microbenchmark with order-0 folios (the common case). Signed-off-by: Ryan Roberts Acked-by: Mark Rutland --- arch/arm64/include/asm/pgtable.h | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 353ea67b5d75..cdc310880a3b 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1213,8 +1213,8 @@ extern int contpte_ptep_set_access_flags(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t entry, int dirty); -static inline void contpte_try_unfold(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, pte_t pte) +static __always_inline void contpte_try_unfold(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, pte_t pte) { if (unlikely(pte_valid_cont(pte))) __contpte_try_unfold(mm, addr, ptep, pte); @@ -1279,7 +1279,7 @@ static inline void set_pte(pte_t *ptep, pte_t pte) } #define set_ptes set_ptes -static inline void set_ptes(struct mm_struct *mm, unsigned long addr, +static __always_inline void set_ptes(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte, unsigned int nr) { pte = pte_mknoncont(pte); @@ -1361,8 +1361,8 @@ static inline int ptep_clear_flush_young(struct vm_area_struct *vma, } #define wrprotect_ptes wrprotect_ptes -static inline void wrprotect_ptes(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, unsigned int nr) +static __always_inline void wrprotect_ptes(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, unsigned int nr) { if (likely(nr == 1)) { /* From patchwork Fri Feb 2 08:07:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13542360 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F1DBDC4828E for ; Fri, 2 Feb 2024 08:09:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7E6266B00B3; Fri, 2 Feb 2024 03:09:54 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 796096B00B5; Fri, 2 Feb 2024 03:09:54 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 65D966B00B6; Fri, 2 Feb 2024 03:09:54 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 563316B00B3 for ; Fri, 2 Feb 2024 03:09:54 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 308C5C070B for ; Fri, 2 Feb 2024 08:09:54 +0000 (UTC) X-FDA: 81746140308.15.6321DF1 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf12.hostedemail.com (Postfix) with ESMTP id 777C54000C for ; Fri, 2 Feb 2024 08:09:52 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf12.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706861392; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qS1uNmviXYMG7rzOytsrcMROwkSHH753rn9wtVwEfUs=; b=QJ82Pm3DDeffphI9hO0i/OKrvmGti3js2wV03NXcG4/Cjj1F5FO5OtuhLUpJRFBxW2rZWd zdbZVfV/ek0kpPJElVWhHiq8nmlTUbH/lCkmaEeYEWS7DgbojHKAtEPn1LBpsfpeWpjcyr 4HtkDgxcAu03TMOi+4g2FKCZVtIyQ0I= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf12.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706861392; a=rsa-sha256; cv=none; b=hHQ0MGbQmts038n9wcno9aOVxzp3w6IMwaJQSKaW9oi0RxAVSxuM8hStO5xARM2CFtlcpV VpCL4v3OC9yfwcEHMyEvI/bb48cksYteL4LTKjWlbxMnCe465QYwQIeptc6qyhlN8M00Kn RwjIhR9aaD+O8SOIj7CHpC2qnNJX/ns= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 2B01B1C00; Fri, 2 Feb 2024 00:10:34 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 3F6B93F5A1; Fri, 2 Feb 2024 00:09:48 -0800 (PST) From: Ryan Roberts To: Catalin Marinas , Will Deacon , Ard Biesheuvel , Marc Zyngier , James Morse , Andrey Ryabinin , Andrew Morton , Matthew Wilcox , Mark Rutland , David Hildenbrand , Kefeng Wang , John Hubbard , Zi Yan , Barry Song <21cnbao@gmail.com>, Alistair Popple , Yang Shi , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, x86@kernel.org, linuxppc-dev@lists.ozlabs.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v5 25/25] arm64/mm: Automatically fold contpte mappings Date: Fri, 2 Feb 2024 08:07:56 +0000 Message-Id: <20240202080756.1453939-26-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240202080756.1453939-1-ryan.roberts@arm.com> References: <20240202080756.1453939-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 777C54000C X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: s6y6zafckt6h1mwiz6oeiifzhnwdmgci X-HE-Tag: 1706861392-384508 X-HE-Meta: U2FsdGVkX19BXV519l6co9/7vVM0VlVKG2LJ6/U3yKGaWm1y2RQIz2PuDfGSt2JHX/y6321eDHhDaiIIq71rNMBTyypg60wqP5T0SKDqNEsmXjVRbwKFOfqMY56Du14vSkJ2J6wTxJPBjN6hfO2LQG6K9G38y3sWe6Jgc71OKb0L7wa8ioQGQML57W1E3o/DVSBnO4NprB1f4JIgHXeImTo0+wT/cRCem22WMU2A+VTiBFY2DTm95WSihtgLe5GYcZEmcoqP4T0NcsDCmwSCb3o+gbM+fOnP40V/tXDwp/Y3EES+el6oUWRxGZ/Eo7O2fN52KCewo7XiTq2F+lIOJeEsrQ2KCHNN1JmYAej6KJ+Rh1+M5qEko1NHmP4+uEqvsv/80KMfEJVM5iX3mrhSWWgI/nqXnLdKL8H8d9+jHzXWdq2xVPbIAbOfJPpigcNXmtGrPclHLE3wAWXc8yuzGKiUkKQgwAI0TfBtop6COFmnBWGOOyr/IB1+KEewmGBRHkPEHIk8oOKOCsEqaUMZV2Oe3oTUMdvQ/AYM3G97amf/kiYShiWzI8B142vcoCxSi8CY+IOewtceY3r9e0uIJvk3l8Fb9Fv2GVQQJGD1r8xJA0Qm+yqW2+fB01Lg3dODt0tKhZgfqZ3YIRFuH96CTht+HUCvddMBRNqMJO+l4FI8bgUYjcLnZBizmLAMYjZst3pgfIbOgdRij0z8T20STMTCnRkFFr+449+8K8EIJsZXYTEP02E8z0tL6uWE2f/+FMSrtZS9Rav1SqvEi5S2Hnvhu0s7v7yVwrd6Wxs+mKnCtuq5+vjKwu3m53u9XhJrLHtaz15QEH7MusMMyd7NV56GXUJ0DrZl4tiN5qGzQvRWqMtWCz6BhKcA7WLtgilDIHx8q0kiYqmzb9VSae2nermhTFowh9gEBbjn6fHJmxkvTZdzFNjUSTsD3UUulrvnXw6D+QGHWY8hD8QGum5 k24leFYi kqlFOsaVET3gjPvRmBZo2n6JbXNksvYQrVnLKZQGWoCNzGNamdXzJah8RS0k9OUojAG2j8xhsjI0jG6Bp87YG+6j8vdjd/pk38Bbw3zfqNsMAF3vCWY+yC7iZdfIjbapAez655shZHE6o7++3vtz/ZmCUYUHQyu+A+CSiY2rzFxFxpVdm30FFaUP8jIopKYkhA+01hHGoRi4UJ5MdnuYScKqtQw5alWdT+T3hbG9100dgNl8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: There are situations where a change to a single PTE could cause the contpte block in which it resides to become foldable (i.e. could be repainted with the contiguous bit). Such situations arise, for example, when user space temporarily changes protections, via mprotect, for individual pages, such can be the case for certain garbage collectors. We would like to detect when such a PTE change occurs. However this can be expensive due to the amount of checking required. Therefore only perform the checks when an indiviual PTE is modified via mprotect (ptep_modify_prot_commit() -> set_pte_at() -> set_ptes(nr=1)) and only when we are setting the final PTE in a contpte-aligned block. Signed-off-by: Ryan Roberts --- arch/arm64/include/asm/pgtable.h | 26 +++++++++++++ arch/arm64/mm/contpte.c | 64 ++++++++++++++++++++++++++++++++ 2 files changed, 90 insertions(+) diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index cdc310880a3b..d3357fe4eb89 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1192,6 +1192,8 @@ void vmemmap_update_pte(unsigned long addr, pte_t *ptep, pte_t pte); * where it is possible and makes sense to do so. The PTE_CONT bit is considered * a private implementation detail of the public ptep API (see below). */ +extern void __contpte_try_fold(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte); extern void __contpte_try_unfold(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte); extern pte_t contpte_ptep_get(pte_t *ptep, pte_t orig_pte); @@ -1213,6 +1215,29 @@ extern int contpte_ptep_set_access_flags(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t entry, int dirty); +static __always_inline void contpte_try_fold(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, pte_t pte) +{ + /* + * Only bother trying if both the virtual and physical addresses are + * aligned and correspond to the last entry in a contig range. The core + * code mostly modifies ranges from low to high, so this is the likely + * the last modification in the contig range, so a good time to fold. + * We can't fold special mappings, because there is no associated folio. + */ + + const unsigned long contmask = CONT_PTES - 1; + bool valign = ((addr >> PAGE_SHIFT) & contmask) == contmask; + + if (unlikely(valign)) { + bool palign = (pte_pfn(pte) & contmask) == contmask; + + if (unlikely(palign && + pte_valid(pte) && !pte_cont(pte) && !pte_special(pte))) + __contpte_try_fold(mm, addr, ptep, pte); + } +} + static __always_inline void contpte_try_unfold(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte) { @@ -1287,6 +1312,7 @@ static __always_inline void set_ptes(struct mm_struct *mm, unsigned long addr, if (likely(nr == 1)) { contpte_try_unfold(mm, addr, ptep, __ptep_get(ptep)); __set_ptes(mm, addr, ptep, pte, 1); + contpte_try_fold(mm, addr, ptep, pte); } else { contpte_set_ptes(mm, addr, ptep, pte, nr); } diff --git a/arch/arm64/mm/contpte.c b/arch/arm64/mm/contpte.c index 80346108450b..2c7dafd0552a 100644 --- a/arch/arm64/mm/contpte.c +++ b/arch/arm64/mm/contpte.c @@ -67,6 +67,70 @@ static void contpte_convert(struct mm_struct *mm, unsigned long addr, __set_ptes(mm, start_addr, start_ptep, pte, CONT_PTES); } +void __contpte_try_fold(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte) +{ + /* + * We have already checked that the virtual and pysical addresses are + * correctly aligned for a contpte mapping in contpte_try_fold() so the + * remaining checks are to ensure that the contpte range is fully + * covered by a single folio, and ensure that all the ptes are valid + * with contiguous PFNs and matching prots. We ignore the state of the + * access and dirty bits for the purpose of deciding if its a contiguous + * range; the folding process will generate a single contpte entry which + * has a single access and dirty bit. Those 2 bits are the logical OR of + * their respective bits in the constituent pte entries. In order to + * ensure the contpte range is covered by a single folio, we must + * recover the folio from the pfn, but special mappings don't have a + * folio backing them. Fortunately contpte_try_fold() already checked + * that the pte is not special - we never try to fold special mappings. + * Note we can't use vm_normal_page() for this since we don't have the + * vma. + */ + + unsigned long folio_saddr, folio_eaddr; + unsigned long cont_saddr, cont_eaddr; + pte_t expected_pte, subpte; + struct folio *folio; + struct page *page; + unsigned long pfn; + pte_t *orig_ptep; + pgprot_t prot; + + int i; + + if (!mm_is_user(mm)) + return; + + page = pte_page(pte); + folio = page_folio(page); + folio_saddr = addr - (page - &folio->page) * PAGE_SIZE; + folio_eaddr = folio_saddr + folio_nr_pages(folio) * PAGE_SIZE; + cont_saddr = ALIGN_DOWN(addr, CONT_PTE_SIZE); + cont_eaddr = cont_saddr + CONT_PTE_SIZE; + + if (folio_saddr > cont_saddr || folio_eaddr < cont_eaddr) + return; + + pfn = pte_pfn(pte) - ((addr - cont_saddr) >> PAGE_SHIFT); + prot = pte_pgprot(pte_mkold(pte_mkclean(pte))); + expected_pte = pfn_pte(pfn, prot); + orig_ptep = ptep; + ptep = contpte_align_down(ptep); + + for (i = 0; i < CONT_PTES; i++) { + subpte = pte_mkold(pte_mkclean(__ptep_get(ptep))); + if (!pte_same(subpte, expected_pte)) + return; + expected_pte = pte_advance_pfn(expected_pte, 1); + ptep++; + } + + pte = pte_mkcont(pte); + contpte_convert(mm, addr, orig_ptep, pte); +} +EXPORT_SYMBOL(__contpte_try_fold); + void __contpte_try_unfold(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte) {