From patchwork Thu Feb 15 12:17:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13558301 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6E47EC48BEB for ; Thu, 15 Feb 2024 12:18:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D3AE58D000E; Thu, 15 Feb 2024 07:18:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CC3288D0001; Thu, 15 Feb 2024 07:18:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AEF728D000E; Thu, 15 Feb 2024 07:18:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 9A11D8D0001 for ; Thu, 15 Feb 2024 07:18:11 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 68686160A3F for ; Thu, 15 Feb 2024 12:18:11 +0000 (UTC) X-FDA: 81793940382.07.9AB7ECA Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf20.hostedemail.com (Postfix) with ESMTP id D44971C000D for ; Thu, 15 Feb 2024 12:18:08 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=none; spf=pass (imf20.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707999489; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5v058h4Aw+ECiRT25vTaV+XZu2TbsaL/aTwq/D7+cl0=; b=WT1UUzW/fy1/S3VDdy6HDeYx5Xy5QrxTsgnOyMtP5MiAyJGyBBLYUNkWnUFpe0wml8dppb naS7Ype3k/eLmuUXQubeAWRltKq3hi0n7T7qk9yYDqNm7SLVEqVvByixm9P6G62R4NNV++ yB0KLSI5cQXJiqip9E4apMJEu/SRBB8= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707999489; a=rsa-sha256; cv=none; b=tyvha4VVVhJcexCd/4cN2am4CRW/3Zmv8/0LdZ/QVdi/qkJzc6RxlQu9pNIZs2dHMVd5fN uny6H+f7wHsFs0GRdxRAz3sfmIociaro1h+ZzPOe6uLRj4rqxLbsvDUD2zNBaRSRYwCVHW f0Kx4SzFrYNPQin7VH7jkwE+3JK39k8= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=none; spf=pass (imf20.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id BA08EDA7; Thu, 15 Feb 2024 04:18:48 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 1E9A03F766; Thu, 15 Feb 2024 04:18:06 -0800 (PST) From: Ryan Roberts To: David Hildenbrand , Mark Rutland , Catalin Marinas , Will Deacon , Alexander Shishkin , Jiri Olsa , Ian Rogers , Adrian Hunter , Andrew Morton , Muchun Song Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v1 1/4] mm: Introduce ptep_get_lockless_norecency() Date: Thu, 15 Feb 2024 12:17:53 +0000 Message-Id: <20240215121756.2734131-2-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240215121756.2734131-1-ryan.roberts@arm.com> References: <20240215121756.2734131-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: D44971C000D X-Rspam-User: X-Stat-Signature: arhtc1a5wjqmrpdmggwm9wzso4q31gha X-Rspamd-Server: rspam03 X-HE-Tag: 1707999488-942640 X-HE-Meta: U2FsdGVkX18a4iX+cx5SQKHTohU+3ohkvGE3Mb6nVBZa72ACwkDArmTmiRJGNuh4Zq3xpltAJL913DNg1ApFirSO7pXA2ggvEEauN/CJf+qf2eGysnrq99XJVE15AjvMWWpE488RwHT47zisNzsFX/TMZm+Q3lHyP/GSrBXgGJnXKEOtX63F5rTZgxHiRJoByjqgHqwqIePr7Patdh5dqBlRhueEKPrV0RznW2vX0fQUbzw4sN2so0aysioe5W2/4RHS9F1ZDC/WN8vLEfj9rj23BlFTYvIImBwJQ5onsGSfh/NaxIZg8CBBRsGi16mFBdhWK6EFElIxD7U0samRrFTdcB6P6vGduG+bZwVtVi3cGxVzaA5G4gGpIIIrswvVBkWiZ5rSK08whhRByDhszhvqkP1Qx5zDyt+IMSvKB6RLZCdRuKEA9D3iV+/J5vhKK3QNpw6wK/mRYXhrAPYOaDgFzEpN7cp2CsRV6I8oPvK2D8FK7Ise1ooPCZlTYWaG4T4ykop79zomCl3UlDSaWtcrQCASdGhr1p0JUR3Td0AWKc05A0MsfSUctTr3M6DrCZZRO1zlvkzvb8AUKZwkKnAfqD0Ij1PxqDcOCVc+kgd38k1qcb9u7DXRlDQMPPnYGO2JyikrtyDXhQ3X/KMbzHbxiW6mjmsBbkZZFv3rFNKSg+DsFiYLTq0dsBT9547z/GYYyQy6FoYIKaCG1gecWKC3VfWKwCDpB0E/vGctumRNSGfWtMtwy3Y7NJnHo8R0567C/Mv80vE4e+cOAVuK488OKfwIOV36KUkqxPBaMN9+/3mTiHUbDN7AtYYzygfOeDcvOv4qZPxzdMSdOZUHbVK8YoVipRSUa4gSPE8DfMqZQTqBVnCzPcGCitylsM+jyhSYE4s8gWg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: With the introduction of contpte mapping support for arm64, that architecture's implementation of ptep_get_lockless() has become very complex due to the need to gather access and dirty bits from across all of the ptes in the contpte block. This requires careful implementation to ensure the returned value is consistent (because its not possible to read all ptes atomically), but even in the common case when there is no racing modification, we have to read all ptes, which gives an ~O(n^2) cost if the core-mm is iterating over a range, and performing a ptep_get_lockless() on each pte. Solve this by introducing ptep_get_lockless_norecency(), which does not make any guarantees about access and dirty bits. Therefore it can simply read the single target pte. At the same time, convert all call sites that previously used ptep_get_lockless() but don't care about access and dirty state. We may want to do something similar for ptep_get() (i.e. ptep_get_norecency()) in future; it doesn't suffer from the consistency problem because the PTL serializes it with any modifications, but does suffer the same O(n^2) cost. Signed-off-by: Ryan Roberts --- include/linux/pgtable.h | 37 ++++++++++++++++++++++++++++++++++--- kernel/events/core.c | 2 +- mm/hugetlb.c | 2 +- mm/khugepaged.c | 2 +- mm/memory.c | 2 +- mm/swap_state.c | 2 +- mm/swapfile.c | 2 +- 7 files changed, 40 insertions(+), 9 deletions(-) -- 2.25.1 diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index a36cf4e124b0..9dd40fdbd825 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -528,16 +528,47 @@ static inline pmd_t pmdp_get_lockless(pmd_t *pmdp) #endif /* CONFIG_PGTABLE_LEVELS > 2 */ #endif /* CONFIG_GUP_GET_PXX_LOW_HIGH */ -/* - * We require that the PTE can be read atomically. - */ #ifndef ptep_get_lockless +/** + * ptep_get_lockless - Get a pte without holding the page table lock. Young and + * dirty bits are guaranteed to accurately reflect the state + * of the pte at the time of the call. + * @ptep: Page table pointer for pte to get. + * + * If young and dirty information is not required, use + * ptep_get_lockless_norecency() which can be faster on some architectures. + * + * May be overridden by the architecture; otherwise, implemented using + * ptep_get(), on the assumption that it is atomic. + * + * Context: Any. + */ static inline pte_t ptep_get_lockless(pte_t *ptep) { return ptep_get(ptep); } #endif +#ifndef ptep_get_lockless_norecency +/** + * ptep_get_lockless_norecency - Get a pte without holding the page table lock. + * Young and dirty bits may not be accurate. + * @ptep: Page table pointer for pte to get. + * + * Prefer this over ptep_get_lockless() when young and dirty information is not + * required since it can be faster on some architectures. + * + * May be overridden by the architecture; otherwise, implemented using the more + * precise ptep_get_lockless(). + * + * Context: Any. + */ +static inline pte_t ptep_get_lockless_norecency(pte_t *ptep) +{ + return ptep_get_lockless(ptep); +} +#endif + #ifndef pmdp_get_lockless static inline pmd_t pmdp_get_lockless(pmd_t *pmdp) { diff --git a/kernel/events/core.c b/kernel/events/core.c index f0f0f71213a1..27991312d635 100644 --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -7583,7 +7583,7 @@ static u64 perf_get_pgtable_size(struct mm_struct *mm, unsigned long addr) if (!ptep) goto again; - pte = ptep_get_lockless(ptep); + pte = ptep_get_lockless_norecency(ptep); if (pte_present(pte)) size = pte_leaf_size(pte); pte_unmap(ptep); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 68283e54c899..41dc44eb8454 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -7517,7 +7517,7 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma, } if (pte) { - pte_t pteval = ptep_get_lockless(pte); + pte_t pteval = ptep_get_lockless_norecency(pte); BUG_ON(pte_present(pteval) && !pte_huge(pteval)); } diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 2771fc043b3b..1a6c9ed8237a 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1019,7 +1019,7 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm, } } - vmf.orig_pte = ptep_get_lockless(pte); + vmf.orig_pte = ptep_get_lockless_norecency(pte); if (!is_swap_pte(vmf.orig_pte)) continue; diff --git a/mm/memory.c b/mm/memory.c index 4dd8e35b593a..8e65fa1884f1 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4353,7 +4353,7 @@ static bool pte_range_none(pte_t *pte, int nr_pages) int i; for (i = 0; i < nr_pages; i++) { - if (!pte_none(ptep_get_lockless(pte + i))) + if (!pte_none(ptep_get_lockless_norecency(pte + i))) return false; } diff --git a/mm/swap_state.c b/mm/swap_state.c index 2f540748f7c0..061c6c16c7ff 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -837,7 +837,7 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask, if (!pte) break; } - pentry = ptep_get_lockless(pte); + pentry = ptep_get_lockless_norecency(pte); if (!is_swap_pte(pentry)) continue; entry = pte_to_swp_entry(pentry); diff --git a/mm/swapfile.c b/mm/swapfile.c index d1bd8d1e17bd..c414dd238814 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1857,7 +1857,7 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd, break; } - ptent = ptep_get_lockless(pte); + ptent = ptep_get_lockless_norecency(pte); if (!is_swap_pte(ptent)) continue; From patchwork Thu Feb 15 12:17:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13558302 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6A931C48BF0 for ; Thu, 15 Feb 2024 12:18:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2906B8D0013; Thu, 15 Feb 2024 07:18:13 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1F03D8D0001; Thu, 15 Feb 2024 07:18:13 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0435B8D0013; Thu, 15 Feb 2024 07:18:12 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id DC4D88D0001 for ; Thu, 15 Feb 2024 07:18:12 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 78803A0FC9 for ; Thu, 15 Feb 2024 12:18:12 +0000 (UTC) X-FDA: 81793940424.02.26BC93B Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf13.hostedemail.com (Postfix) with ESMTP id BA04520010 for ; Thu, 15 Feb 2024 12:18:10 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf13.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707999490; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lbpD6M3yiauchTnc5b8hwmaJxEY6qpI+eNSGlv7SoLM=; b=UWKmB8eKAz5df8eJOO96QghArtFLKkXxIzxLClf6EYqMWthlV6bqtNkH7xaSYt1/hTCrcV e/UtC0yasFqc5VwEg9k0JyppFRWzDrym6eiWiXKdbZF3QUf2YQm/qzZ6SUEsXuTcfEU+YB y/l+kuL36qLSouNeh6V0l1wpd+3DwwQ= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf13.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707999490; a=rsa-sha256; cv=none; b=mcrEAlsgxjdzeQbNFkHIUMwyc2JnDDp1FtG9l2H3+rrGOR4b+0hEQ8IJl66usM/ubBGIUK 3is9hrf6JKm9kZAZS4y/e312cHA4tH3K2yD7XpsusCCZHHYFq1wQP30VXf6+uCCsBRCqlY 1l/lwjx2sGMqTT+IXI2fePEeJ9mdWuY= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id B279B1480; Thu, 15 Feb 2024 04:18:50 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 16F853F766; Thu, 15 Feb 2024 04:18:07 -0800 (PST) From: Ryan Roberts To: David Hildenbrand , Mark Rutland , Catalin Marinas , Will Deacon , Alexander Shishkin , Jiri Olsa , Ian Rogers , Adrian Hunter , Andrew Morton , Muchun Song Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v1 2/4] mm/gup: Use ptep_get_lockless_norecency() Date: Thu, 15 Feb 2024 12:17:54 +0000 Message-Id: <20240215121756.2734131-3-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240215121756.2734131-1-ryan.roberts@arm.com> References: <20240215121756.2734131-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: BA04520010 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: jyfxn1rwjtezwn7c4dm48mma51wo6zpt X-HE-Tag: 1707999490-39397 X-HE-Meta: U2FsdGVkX19EzX/XdVAlpE9KBBGukW7vjZ1Ei6wMHdVO0vpnlMMJjAndDeCzbJ/RCMRKmN4q6PNGTNkShyFDGBsKeap0hS8jCdE26AGYOoJ2OTivMTrvvGvo9lQ1X2aSmtqz7fTxZp2dWS+gLbr0MMYkMg9HTnWealPnLXC2PVNYkAUKJKw0NCW+aD8NNsPzRlnSWYV2ep/XvZ85v2cUQDymkzYr0sMzIctKKaXWIUlspKB9q1Ge38z+IPnrSsniJZ+OEtbpVvWt78GTQmIafcYSABpBWmi23aWQMU8xVJNUVdJTGBL/e8jYhg6VXDiIiWSFEoXCOtXbHdbEikmNNgbPKgWM6+qj1by3ddml4uQ9cqWnOg0BcaFEYuzOItdK1aS+UhmNs3Op3jE+hOqQStN8AL90iYoE0WB1djq7hx3GJTmKhxboCCv3NEfnCCI6ZNamWSKhhsyU1tEC0+A9dMsNh/Lb27xwXU1+xRATPIqMvqmkS6Qp2ykWgQwch9CurbEDiYrrYVmnv4k1mqqSlC2TFt0FtrwJyLWbEoRukxtJ9idBYJOvDi/ilefsCHsJ/9eyXxT7kMQmZatjaOkFtxqL8y7FTrVxDhyyln6d3yieNV5Tm3x4/ky8d+SfUB0rQ1I9LdusU/kHypgIv05QtMm87JmXZEhhnLuUgIkjSffiE7vV7V18va+SGW2nTvAyoaMCRZh6/DgsvVZLlzTOV+N+Q1Oz0cfPG/dsqewHQF709MilfO+cAYVF8E4H+jVi6cmAjJQ6cQKVaIYTS40XhmB9QCu1lo1hz4PvRyQvZU2ffaUGpZAp2nPT0v+Pwlo8kfzJNirpQwQc7KM28aIr+b3pXLK+9u2z4E6zMpGui+8aeo7OCcPxNH9lso79oH8Aix0pvXC9jeW5pkrcb14t/pYbLxSKiu3Zq/8Pf3pkJMUDkCDqCmAPZZPe1roHv4t9+9WS3FNHOlFykV+iaYN Ks501R1P HGDRJlAKbtHF1A2lyMreI6aMxYKL1Ew2kjxuUNu/vCbjmQzhwm7KbFxXl8xWsi9JgkeBb9PKKhNaw6/vUekfMG1WLT04vq22KXJIDD5Bd5eeaZDxuUoK9Key9pjxKaXGrgMwinQNxRHO1Vm8USrKx/BncuELSn4gPKhEYaZ+o0WFZOo7qDuvJmy2BYpGCkqVQ/qy8fpaKSL0ckWA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Gup needs to read ptes locklessly, so it uses ptep_get_lockless(). However, the returned access and dirty bits are unimportant so let's switch over to ptep_get_lockless_norecency(). The wrinkle is that gup needs to check that the pte hasn't changed once it has pinned the folio following this model: pte = ptep_get_lockless_norecency(ptep) ... if (!pte_same(pte, ptep_get_lockless(ptep))) // RACE! ... And now that pte may not contain correct access and dirty information, the pte_same() comparison could spuriously fail. So let's introduce a new pte_same_norecency() helper which will ignore the access and dirty bits when doing the comparison. Note that previously, ptep_get() was being used for the comparison; this is technically incorrect because the PTL is not held. I've also converted the comparison to use the preferred pmd_same() helper instead of doing a raw value comparison. As a side-effect, this new approach removes the possibility of concurrent read/write to the page causing a spurious fast gup failure, because the access and dirty bits are no longer used in the comparison. Signed-off-by: Ryan Roberts Acked-by: David Hildenbrand --- include/linux/pgtable.h | 18 ++++++++++++++++++ mm/gup.c | 7 ++++--- 2 files changed, 22 insertions(+), 3 deletions(-) -- 2.25.1 diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 9dd40fdbd825..8123affa8baf 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -936,6 +936,24 @@ static inline int pte_same(pte_t pte_a, pte_t pte_b) } #endif +/** + * pte_same_norecency - Compare pte_a and pte_b, ignoring young and dirty bits, + * if the ptes are present. + * + * @pte_a: First pte to compare. + * @pte_b: Second pte to compare. + * + * Returns 1 if the ptes match, else 0. + */ +static inline int pte_same_norecency(pte_t pte_a, pte_t pte_b) +{ + if (pte_present(pte_a)) + pte_a = pte_mkold(pte_mkclean(pte_a)); + if (pte_present(pte_b)) + pte_b = pte_mkold(pte_mkclean(pte_b)); + return pte_same(pte_a, pte_b); +} + #ifndef __HAVE_ARCH_PTE_UNUSED /* * Some architectures provide facilities to virtualization guests diff --git a/mm/gup.c b/mm/gup.c index df83182ec72d..0f96d0a5ec09 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -2576,7 +2576,7 @@ static int gup_pte_range(pmd_t pmd, pmd_t *pmdp, unsigned long addr, if (!ptep) return 0; do { - pte_t pte = ptep_get_lockless(ptep); + pte_t pte = ptep_get_lockless_norecency(ptep); struct page *page; struct folio *folio; @@ -2617,8 +2617,9 @@ static int gup_pte_range(pmd_t pmd, pmd_t *pmdp, unsigned long addr, goto pte_unmap; } - if (unlikely(pmd_val(pmd) != pmd_val(*pmdp)) || - unlikely(pte_val(pte) != pte_val(ptep_get(ptep)))) { + if (unlikely(!pmd_same(pmd, *pmdp)) || + unlikely(!pte_same_norecency(pte, + ptep_get_lockless_norecency(ptep)))) { gup_put_folio(folio, 1, flags); goto pte_unmap; } From patchwork Thu Feb 15 12:17:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13558303 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 345E5C48BEB for ; Thu, 15 Feb 2024 12:18:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A56498D0014; Thu, 15 Feb 2024 07:18:14 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 98EB28D0001; Thu, 15 Feb 2024 07:18:14 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6FA7F8D0014; Thu, 15 Feb 2024 07:18:14 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 534CC8D0001 for ; Thu, 15 Feb 2024 07:18:14 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 2F851C1041 for ; Thu, 15 Feb 2024 12:18:14 +0000 (UTC) X-FDA: 81793940508.15.22A6FFD Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf08.hostedemail.com (Postfix) with ESMTP id 87C3616000C for ; Thu, 15 Feb 2024 12:18:12 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf08.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707999492; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=uUGDUFHa6ouIpGmhULS0O/3eXPjqJ4rcpqKk6CYa4SM=; b=Aa+cl9ABM+wxu+TuRHDZr7QeKGBJjrDdbuXoSKPGqxCbRDvMkkpV9TbBdnpQ0SDLgyOdgy AyMp97UbWQk39vgepbY15kLEiHa3FcvxDd15Sme35x/HhhFaocHmOsUiApjpE1ecLrdO4+ X+IRsdCCVb8oLjlOtLQMqDtNclfdefA= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf08.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707999492; a=rsa-sha256; cv=none; b=bzROzrbsgwH9l0OO0peJhlsXYNwyrBxI7Vos86lAyAPuHkxW1wpf+B8w5Lfn3foA3nhfIP pl1291JmTh6CpYBjRiJMDRaq0ubF6KeO1nGvj0ReKuCIdwsOvtZ8uRwyBcNqwZ6ZIpEqqR PC8KdorBUl/KMYRFnKNi4rY9lC6J9b0= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id ACE0F14BF; Thu, 15 Feb 2024 04:18:52 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 0F9763F766; Thu, 15 Feb 2024 04:18:09 -0800 (PST) From: Ryan Roberts To: David Hildenbrand , Mark Rutland , Catalin Marinas , Will Deacon , Alexander Shishkin , Jiri Olsa , Ian Rogers , Adrian Hunter , Andrew Morton , Muchun Song Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v1 3/4] mm/memory: Use ptep_get_lockless_norecency() for orig_pte Date: Thu, 15 Feb 2024 12:17:55 +0000 Message-Id: <20240215121756.2734131-4-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240215121756.2734131-1-ryan.roberts@arm.com> References: <20240215121756.2734131-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: 7ah447es8aifumsqqor84eunqpwaj6fx X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 87C3616000C X-HE-Tag: 1707999492-74639 X-HE-Meta: U2FsdGVkX18m39fss1xRn0LarDv5I4ya4PAB4WqUD5e7Td0AygcP/Dki3nMr9a6cSbvg5F4VDbf0R/A1ODwML5QN8ebBEuEbx0l6+TItC4AdeEdgQefLNXjia19IiF3EL4LZnxN+fUCfxrDzYe/lMvgbB7e2PBXz5WGeukAZU0oLqwe5iLUvHMU0Yu9bfwOxXdvyWaqeIpaCjlpbb/V8c6XOijhvjgo08rwZjo/eK4SJgSlkgEklHvVOY1nTlU7yhc8lhsA3vPq5zPxuYQg8HS8qIPvL+m3ylSBqbPJcHnqMm+mmqmjd2+rYiBVSNSlyUJjJhyRaPrMuVo4gyGXUHMD0kxKcK2NmSsBi/yPDHpwPEOmqqb7XlABu7HrGpwSR8L8on0HEfIyxH2mzbLPY4vbki4joI/lSy0qj6FG3YObgoHsJNpU4fsadS8dPxkV/hVFpU/M3qDcG7Nzcv2+FPVc7Z2OZ+vTmZhpxjbTV/Ov8iylxRdz3hYlXl+nfrk8S3E3WEr6Hjxb42b+SwaZclsYXcGULxHwYpnAXE/GESd/wWlJoUs82oxz25RworNa9OBYP32UGIUEi84z4oboSnMTFfLAdLATtmA9WQqokqkAd52BM0vtLJ1nUWbEIw7VwqJizKk92cfxw7wEv1LrJKb3g3UXWtxVrnS3fWyDuNUh7s7NgK9AdEeLj4y5JPkyu9sUkyI8I+DwBA+rND67N+DoePnCecYwYs1r+nWv2qq0VMVWpdUl90lO8SauNmzhMOT8omubRZYhNQxKcyl3gLzqtgOKqyLunSzOHSiW9ZhLrVzul89sAI12rkNxc9gjK8QhoBrZ2ksuqVmV59tLQUX/3wQAmPYTWPk2v7duyAeUDh+1QMr5s4uKT10f8dZnHcZyOeQ9+J0SclQ459qSfMpuqgUBM4dWDxTz1jLxcKT4tY3pSASnMyok6lvzp4NjaRjksqNzCUIp+1FH2Quk CmbkzaOP YxSCGue56Y55AKZ5xRCJeurXLZXw8wybiTgL37wn4UywROdUEVzNARwo+Xk6fTnK9h6e1Qwtctsrsjs96oVqQd/7nT3Vv3/ERPYlnMBIAmaj+B1Guf78AGzAQD6r+1OcKJHQ4t7+zpKrH5jTvx4NTfz2ZOuYU5CwbK9yHysarEhZKHl4YqvwnMw5HCY44prlpzkClXjsemgDJLpo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's convert handle_pte_fault()'s use of ptep_get_lockless() to ptep_get_lockless_norecency() to save orig_pte. There are a number of places that follow this model: orig_pte = ptep_get_lockless(ptep) ... if (!pte_same(orig_pte, ptep_get(ptep))) // RACE! ... So we need to be careful to convert all of those to use pte_same_norecency() so that the access and dirty bits are excluded from the comparison. Additionally there are a couple of places that genuinely rely on the access and dirty bits of orig_pte, but with some careful refactoring, we can use ptep_get() once we are holding the lock to achieve equivalent logic. Signed-off-by: Ryan Roberts --- mm/memory.c | 55 +++++++++++++++++++++++++++++++++-------------------- 1 file changed, 34 insertions(+), 21 deletions(-) -- 2.25.1 diff --git a/mm/memory.c b/mm/memory.c index 8e65fa1884f1..075245314ec3 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3014,7 +3014,7 @@ static inline int pte_unmap_same(struct vm_fault *vmf) #if defined(CONFIG_SMP) || defined(CONFIG_PREEMPTION) if (sizeof(pte_t) > sizeof(unsigned long)) { spin_lock(vmf->ptl); - same = pte_same(ptep_get(vmf->pte), vmf->orig_pte); + same = pte_same_norecency(ptep_get(vmf->pte), vmf->orig_pte); spin_unlock(vmf->ptl); } #endif @@ -3062,11 +3062,14 @@ static inline int __wp_page_copy_user(struct page *dst, struct page *src, * take a double page fault, so mark it accessed here. */ vmf->pte = NULL; - if (!arch_has_hw_pte_young() && !pte_young(vmf->orig_pte)) { + if (!arch_has_hw_pte_young()) { pte_t entry; vmf->pte = pte_offset_map_lock(mm, vmf->pmd, addr, &vmf->ptl); - if (unlikely(!vmf->pte || !pte_same(ptep_get(vmf->pte), vmf->orig_pte))) { + if (likely(vmf->pte)) + entry = ptep_get(vmf->pte); + if (unlikely(!vmf->pte || + !pte_same_norecency(entry, vmf->orig_pte))) { /* * Other thread has already handled the fault * and update local tlb only @@ -3077,9 +3080,11 @@ static inline int __wp_page_copy_user(struct page *dst, struct page *src, goto pte_unlock; } - entry = pte_mkyoung(vmf->orig_pte); - if (ptep_set_access_flags(vma, addr, vmf->pte, entry, 0)) - update_mmu_cache_range(vmf, vma, addr, vmf->pte, 1); + if (!pte_young(entry)) { + entry = pte_mkyoung(entry); + if (ptep_set_access_flags(vma, addr, vmf->pte, entry, 0)) + update_mmu_cache_range(vmf, vma, addr, vmf->pte, 1); + } } /* @@ -3094,7 +3099,8 @@ static inline int __wp_page_copy_user(struct page *dst, struct page *src, /* Re-validate under PTL if the page is still mapped */ vmf->pte = pte_offset_map_lock(mm, vmf->pmd, addr, &vmf->ptl); - if (unlikely(!vmf->pte || !pte_same(ptep_get(vmf->pte), vmf->orig_pte))) { + if (unlikely(!vmf->pte || + !pte_same_norecency(ptep_get(vmf->pte), vmf->orig_pte))) { /* The PTE changed under us, update local tlb */ if (vmf->pte) update_mmu_tlb(vma, addr, vmf->pte); @@ -3369,14 +3375,17 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) * Re-check the pte - we dropped the lock */ vmf->pte = pte_offset_map_lock(mm, vmf->pmd, vmf->address, &vmf->ptl); - if (likely(vmf->pte && pte_same(ptep_get(vmf->pte), vmf->orig_pte))) { + if (likely(vmf->pte)) + entry = ptep_get(vmf->pte); + if (likely(vmf->pte && pte_same_norecency(entry, vmf->orig_pte))) { if (old_folio) { if (!folio_test_anon(old_folio)) { dec_mm_counter(mm, mm_counter_file(old_folio)); inc_mm_counter(mm, MM_ANONPAGES); } } else { - ksm_might_unmap_zero_page(mm, vmf->orig_pte); + /* Needs dirty bit so can't use vmf->orig_pte. */ + ksm_might_unmap_zero_page(mm, entry); inc_mm_counter(mm, MM_ANONPAGES); } flush_cache_page(vma, vmf->address, pte_pfn(vmf->orig_pte)); @@ -3494,7 +3503,7 @@ static vm_fault_t finish_mkwrite_fault(struct vm_fault *vmf, struct folio *folio * We might have raced with another page fault while we released the * pte_offset_map_lock. */ - if (!pte_same(ptep_get(vmf->pte), vmf->orig_pte)) { + if (!pte_same_norecency(ptep_get(vmf->pte), vmf->orig_pte)) { update_mmu_tlb(vmf->vma, vmf->address, vmf->pte); pte_unmap_unlock(vmf->pte, vmf->ptl); return VM_FAULT_NOPAGE; @@ -3883,7 +3892,8 @@ static vm_fault_t remove_device_exclusive_entry(struct vm_fault *vmf) vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); - if (likely(vmf->pte && pte_same(ptep_get(vmf->pte), vmf->orig_pte))) + if (likely(vmf->pte && + pte_same_norecency(ptep_get(vmf->pte), vmf->orig_pte))) restore_exclusive_pte(vma, vmf->page, vmf->address, vmf->pte); if (vmf->pte) @@ -3928,7 +3938,7 @@ static vm_fault_t pte_marker_clear(struct vm_fault *vmf) * quickly from a PTE_MARKER_UFFD_WP into PTE_MARKER_POISONED. * So is_pte_marker() check is not enough to safely drop the pte. */ - if (pte_same(vmf->orig_pte, ptep_get(vmf->pte))) + if (pte_same_norecency(vmf->orig_pte, ptep_get(vmf->pte))) pte_clear(vmf->vma->vm_mm, vmf->address, vmf->pte); pte_unmap_unlock(vmf->pte, vmf->ptl); return 0; @@ -4028,8 +4038,8 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); if (unlikely(!vmf->pte || - !pte_same(ptep_get(vmf->pte), - vmf->orig_pte))) + !pte_same_norecency(ptep_get(vmf->pte), + vmf->orig_pte))) goto unlock; /* @@ -4117,7 +4127,8 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); if (likely(vmf->pte && - pte_same(ptep_get(vmf->pte), vmf->orig_pte))) + pte_same_norecency(ptep_get(vmf->pte), + vmf->orig_pte))) ret = VM_FAULT_OOM; goto unlock; } @@ -4187,7 +4198,8 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) */ vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); - if (unlikely(!vmf->pte || !pte_same(ptep_get(vmf->pte), vmf->orig_pte))) + if (unlikely(!vmf->pte || + !pte_same_norecency(ptep_get(vmf->pte), vmf->orig_pte))) goto out_nomap; if (unlikely(!folio_test_uptodate(folio))) { @@ -4747,7 +4759,7 @@ void set_pte_range(struct vm_fault *vmf, struct folio *folio, static bool vmf_pte_changed(struct vm_fault *vmf) { if (vmf->flags & FAULT_FLAG_ORIG_PTE_VALID) - return !pte_same(ptep_get(vmf->pte), vmf->orig_pte); + return !pte_same_norecency(ptep_get(vmf->pte), vmf->orig_pte); return !pte_none(ptep_get(vmf->pte)); } @@ -5125,7 +5137,7 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) * the pfn may be screwed if the read is non atomic. */ spin_lock(vmf->ptl); - if (unlikely(!pte_same(ptep_get(vmf->pte), vmf->orig_pte))) { + if (unlikely(!pte_same_norecency(ptep_get(vmf->pte), vmf->orig_pte))) { pte_unmap_unlock(vmf->pte, vmf->ptl); goto out; } @@ -5197,7 +5209,8 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) vmf->address, &vmf->ptl); if (unlikely(!vmf->pte)) goto out; - if (unlikely(!pte_same(ptep_get(vmf->pte), vmf->orig_pte))) { + if (unlikely(!pte_same_norecency(ptep_get(vmf->pte), + vmf->orig_pte))) { pte_unmap_unlock(vmf->pte, vmf->ptl); goto out; } @@ -5343,7 +5356,7 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) vmf->address, &vmf->ptl); if (unlikely(!vmf->pte)) return 0; - vmf->orig_pte = ptep_get_lockless(vmf->pte); + vmf->orig_pte = ptep_get_lockless_norecency(vmf->pte); vmf->flags |= FAULT_FLAG_ORIG_PTE_VALID; if (pte_none(vmf->orig_pte)) { @@ -5363,7 +5376,7 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) spin_lock(vmf->ptl); entry = vmf->orig_pte; - if (unlikely(!pte_same(ptep_get(vmf->pte), entry))) { + if (unlikely(!pte_same_norecency(ptep_get(vmf->pte), entry))) { update_mmu_tlb(vmf->vma, vmf->address, vmf->pte); goto unlock; } From patchwork Thu Feb 15 12:17:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13558304 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2D3FFC48BEB for ; Thu, 15 Feb 2024 12:18:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3090E8D0017; Thu, 15 Feb 2024 07:18:16 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 28E888D0001; Thu, 15 Feb 2024 07:18:16 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 105188D0017; Thu, 15 Feb 2024 07:18:16 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id E96E38D0001 for ; Thu, 15 Feb 2024 07:18:15 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id BBBB616108D for ; Thu, 15 Feb 2024 12:18:15 +0000 (UTC) X-FDA: 81793940550.29.89DC049 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf13.hostedemail.com (Postfix) with ESMTP id 3103220005 for ; Thu, 15 Feb 2024 12:18:14 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf13.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1707999494; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=djYJwShBq6wbk5Xppr25LzC5laLJijpWrzDCEzEspN8=; b=t0iRsjoVG3PqbzslPM72JZZmPjlKiDRYAZGfQxQhh1aJRV3u5BodMUP6bCBmwuid0t+hqJ F5HYpS2svATHZJxEfTFNUiWkCffywx4ig26ti0jD28vvTv42xDQeKV+o5Ei2AtCOwHJpoS 7tMXDScvJbeBtjZ8MgTcZhUeb2zkcvM= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf13.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1707999494; a=rsa-sha256; cv=none; b=bMPA+qOaINOWNaWzRvy78F8tacLCVuzEcpq3+5sq9t2/hiykRsic4UKvohjgkBfyP+llBU FfFKZu5NaMAPaA5g3NtFaltA6esqcJuj6rPC+nwXsWuANm49przhD+8hDZ2t8VmVLbyIT0 q9PC4Osyw0Myw6VYWDP8rvfFH1bnYAY= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id A35301FB; Thu, 15 Feb 2024 04:18:54 -0800 (PST) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.26]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 07C203F766; Thu, 15 Feb 2024 04:18:11 -0800 (PST) From: Ryan Roberts To: David Hildenbrand , Mark Rutland , Catalin Marinas , Will Deacon , Alexander Shishkin , Jiri Olsa , Ian Rogers , Adrian Hunter , Andrew Morton , Muchun Song Cc: Ryan Roberts , linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [RFC PATCH v1 4/4] arm64/mm: Override ptep_get_lockless_norecency() Date: Thu, 15 Feb 2024 12:17:56 +0000 Message-Id: <20240215121756.2734131-5-ryan.roberts@arm.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20240215121756.2734131-1-ryan.roberts@arm.com> References: <20240215121756.2734131-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 3103220005 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: whue67oicgghcz7eryqynxnp47cmnwwp X-HE-Tag: 1707999494-15701 X-HE-Meta: U2FsdGVkX19haHf5C8HOd1R91wEYJ0C8WZt8CAoEWRKhU+ywqQvuiB9tiStI3nhyw7zXFp2muo6P+KJKVIZWJx+LSEYQEq2UpELaaSN3xbW3dhe9uIQ3bwYoucs/F2xS+OTcHmrxjGEGRHolkAKS9v+u/EqA85EulLUpwID7Y+yo1SovusLR27PifvCzjrO497E5H+Az1EZTkapxFKl4T2OFPsoh3J5U+XZo+Wqr0Sl8wBz89zJsgarbjhP3KUzoQiu9qZ/LglsNhDdzHd9UC55DLR0PMQ/wcILU+2LRmxz5PhcY8NMpHbU1hMKC+PABINP7QuZhvqpJD2qkGHICidMYiPvD61CK1oihq0rEFbQ8Rw8eHpLDT2DzsejKZD73lD0utD0bzfMrqD4HBgi91RJLpkraVYl/mIhtQ7NAAmALenpB/62sHFBbICrFa6Wz7PgjAKswwLp85+O8HpOWBe8ta8JoVxfHNiCfI1AU3yjtx5H/lVwxfvkTbKHBjYcirYhFXwAB0IGKBK71p5HWQYQpfjnZrGFELXik79AXq5HBxrpNhtFrsXwbz5OWPHOq2IV2RfCi9G0Qciog27kPA7pHzbi4APdSzeuiaf+2NHs5gXXxW2+c3SRHeoy5XhnRghfvxMQHYK2DG/CG3ybG3adfzsOCZEB9AM9q+8/59tWgb9nBETvpCDAc7+rU3MWkRwX01D57U5vSmTJS71xD0gK3PxIbe36/VvrhLJf3kYp4n6gRFy2wU/RBH2FjKmN3+AYciKiSKO/GEETxfhF3T05amPI8yrE1sdPeE++4tAQTta0pUAnSlg5CZOL+aR5CW0k8lQNt3OpZ6CzmwM4zyl0t6kY2aDh/p5Eekrdrt9dXZSmmHTPBe2soZu91fRu/GatHCzF9nLyC2A34A+dUVc65InY2PiBwYp94W0GH+FwtVppM1xbXUN1aba/nI+uZ2AT1S6QUyBerCsQnF8p jXLMlKTh h6EghcMe51MmwgYuJpBWrIvZrLOvDAe8VVJWXvFjBBdEpCsKh6E8wT8xAjvzfDtWshMN0hpD7mJbMVqtCkECmS+XHXJlnjXSdWPuuBBTOdlKg7Aa5H435omyEP9/EsNhs/DxKW0FFzRQl6wVdodnNuDILvDZe/unjI1bW+B+sTcRxp4DPwkRv324wMSh1/uA+HKJCPy7o1dHs9pRfIAWD69escPe7bVHP67OH X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Override ptep_get_lockless_norecency() when CONFIG_ARM64_CONTPTE is enabled. Because this API doesn't require the access and dirty bits to be accurate, for the contpte case, we can avoid reading all ptes in the contpte block to collect those bits, in contrast to ptep_get_lockless(). Signed-off-by: Ryan Roberts Reviewed-by: David Hildenbrand --- arch/arm64/include/asm/pgtable.h | 6 ++++++ 1 file changed, 6 insertions(+) -- 2.25.1 diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 401087e8a43d..c0e4ccf74714 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1287,6 +1287,12 @@ static inline pte_t ptep_get_lockless(pte_t *ptep) return contpte_ptep_get_lockless(ptep); } +#define ptep_get_lockless_norecency ptep_get_lockless_norecency +static inline pte_t ptep_get_lockless_norecency(pte_t *ptep) +{ + return __ptep_get(ptep); +} + static inline void set_pte(pte_t *ptep, pte_t pte) { /*