From patchwork Mon Sep 16 11:07:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dev Jain X-Patchwork-Id: 13805278 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4812CC3ABB2 for ; Mon, 16 Sep 2024 11:08:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C11E16B0082; Mon, 16 Sep 2024 07:08:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BDEF26B0092; Mon, 16 Sep 2024 07:08:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A63806B0093; Mon, 16 Sep 2024 07:08:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 878626B0082 for ; Mon, 16 Sep 2024 07:08:12 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 4BC6F140341 for ; Mon, 16 Sep 2024 11:08:12 +0000 (UTC) X-FDA: 82570327224.20.45FE79A Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf17.hostedemail.com (Postfix) with ESMTP id B92F94000E for ; Mon, 16 Sep 2024 11:08:10 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=none; spf=pass (imf17.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1726484835; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=im3FXfedJUXneslzBR2T8Ij2ESATVXDs3oyBPAS+3Jg=; b=OZARborWyMuyCQ7qB5kmVLsbRN489AjIpHHBKL2JHEuGfXY7TNLqknxMRZDazsqc79siSQ nRlWYcH8qWqApxKTU00B0aqTNscaGc2FTx7e3jUBScbGd5gtL4O0CLW9V7MBvQbnoTZD6O 89Hf2wBEXMKSP1hGHUBLuRTywpyM9Ts= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=none; spf=pass (imf17.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1726484835; a=rsa-sha256; cv=none; b=ke9Y8VeN/I63KhWdjryAts/DUWz7EkFUZqCa2Ntc7z/CRUKFXwzMu2rq/nh3U8jg43kgy+ NWMOwgBydTySdC/y0HmftH10CVTVBunSF22Oi+LpNADGPJH5BfPT76ScuC+DfyNHfhXfwv e04vabWuRGvtfixBXJqJ8YzZlSNy/+g= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 6AAE812FC; Mon, 16 Sep 2024 04:08:39 -0700 (PDT) Received: from e116581.blr.arm.com (e116581.arm.com [10.162.42.11]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id DF6683F64C; Mon, 16 Sep 2024 04:08:04 -0700 (PDT) From: Dev Jain To: akpm@linux-foundation.org, david@redhat.com, willy@infradead.org Cc: ryan.roberts@arm.com, anshuman.khandual@arm.com, baohua@kernel.org, hughd@google.com, ioworker0@gmail.com, wangkefeng.wang@huawei.com, baolin.wang@linux.alibaba.com, gshan@redhat.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Dev Jain Subject: [PATCH v2 1/2] mm: Make pte_range_none() return number of empty PTEs Date: Mon, 16 Sep 2024 16:37:53 +0530 Message-Id: <20240916110754.1236200-2-dev.jain@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240916110754.1236200-1-dev.jain@arm.com> References: <20240916110754.1236200-1-dev.jain@arm.com> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: B92F94000E X-Stat-Signature: j5ts4819is5ibhgk9gyh1oiuonw5afs9 X-HE-Tag: 1726484890-860980 X-HE-Meta: U2FsdGVkX19yRSw06dX/Nve3IWbXAPv5p0ndnhiMzQKP9UjOqGLuIYLS7SLIytIpM7gMDB4nWaZZpclq5Jh740zHQHU0AgjZhlnCE4YkTfgr2Ot4q9WQ8uh4qanQMH9RVlA4kAvH5aZJPZoMc30DxuCNKRYAa7OIGfvmx8dFFf6xKakM8VLt3hP3rzE9eR+ccARoNZ25Z6D3a5DoDF1eEMF3ce3CiZ5BycvpftzXsOt90GW0R01SWbtwQ+IsUy6T+V0pS+4n+iE6rf/YJTyiPkdLEK5VfRbz8qLKetXGSAwOg1CRvKN07hNBDuEQoNgFds7X5l0XmnGAEmrVvKqpEf5kOxZ2bdQD/vROijBke6WNZwavofPKLh+TAO71tBDZMZcyJmtmCimgRC5g7lvMOY7iuiyXJvu8B88mqibfht0EEien0K28OAKTIW+vhta9zjUdr2Z1yW7kV9pNCUZC8DT6ULlA0hiK1oOoEqEqi8nIR39iEcntg5IkIPUvy+JADy3TuSJHjcyq3NCf3mLswoohGnLX+dnCJN1jLcDCuq8GQYx9OSa7Xm1Qiq0tQgMh2MNcFwuxze8fxygnkoh0XcKFbhMx9yryO1TDuPUWpO0+xHBEK1H+wT5YTH4w62GMaoPQ4Tyt2lsiep1JG9XLqsDfS6VrvaaRb4qI+CRDveSmsFYs9zrMsDPTC0nMU52zgLi/ExHXbUo0NpW01RhRiI9bK/Ur2h9wFKNp6Jk+kXzzbaiQmwrw15OXqFTUaRWh5XwzsbCPZ/8Nac4Q8C0L0kO2mcbBchswBdDoy2ciOr67kTyjfIwCMb4Zvm62M85I+EJDail7PjIBcVSTgb8OHc1zbT3q3x+h1v3uQLPTSqRTLp+pfc+Rz7rzk23Ttjxojlw4dK2nZa++jD7nHN2yAie63lwMxfMmmWYopG7H34psnxcCu3wS4Qs62gIiDDPLhY1vxHO+wiCPvgqYvmw Wl7UqHZn cSee1mxze47bLuQhTBoIjTgbqzKYNTRHmTF+a7Pf8DJlI/oc46yYRmlD9e6+k2dMZE4Zh9LklTTajhqtOGQx+Ec/JruHSfK2ucQHVnlaHJ9GZnj1iiBqTVbgG2R10EEcjv/djn+NA1sWRebw6DvtO2SRkDTlhOF0jBu56C69coYfVqm3WYRh1KTe7iZWTxiBXt3m2ow2BJpCuLmw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In preparation for the second patch, make pte_range_none() return the number of contiguous empty PTEs. Signed-off-by: Dev Jain Reviewed-by: Baolin Wang --- mm/memory.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 6469ac99f2f7..8bb1236de93c 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4617,16 +4617,16 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) return ret; } -static bool pte_range_none(pte_t *pte, int nr_pages) +static int pte_range_none(pte_t *pte, int nr_pages) { int i; for (i = 0; i < nr_pages; i++) { if (!pte_none(ptep_get_lockless(pte + i))) - return false; + return i; } - return true; + return nr_pages; } static struct folio *alloc_anon_folio(struct vm_fault *vmf) @@ -4671,7 +4671,7 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf) order = highest_order(orders); while (orders) { addr = ALIGN_DOWN(vmf->address, PAGE_SIZE << order); - if (pte_range_none(pte + pte_index(addr), 1 << order)) + if (pte_range_none(pte + pte_index(addr), 1 << order) == 1 << order) break; order = next_order(&orders, order); } @@ -4787,7 +4787,7 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) if (nr_pages == 1 && vmf_pte_changed(vmf)) { update_mmu_tlb(vma, addr, vmf->pte); goto release; - } else if (nr_pages > 1 && !pte_range_none(vmf->pte, nr_pages)) { + } else if (nr_pages > 1 && pte_range_none(vmf->pte, nr_pages) != nr_pages) { update_mmu_tlb_range(vma, addr, vmf->pte, nr_pages); goto release; } @@ -5121,7 +5121,7 @@ vm_fault_t finish_fault(struct vm_fault *vmf) update_mmu_tlb(vma, addr, vmf->pte); ret = VM_FAULT_NOPAGE; goto unlock; - } else if (nr_pages > 1 && !pte_range_none(vmf->pte, nr_pages)) { + } else if (nr_pages > 1 && pte_range_none(vmf->pte, nr_pages) != nr_pages) { update_mmu_tlb_range(vma, addr, vmf->pte, nr_pages); ret = VM_FAULT_NOPAGE; goto unlock; From patchwork Mon Sep 16 11:07:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Dev Jain X-Patchwork-Id: 13805279 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EBD71C3ABB2 for ; Mon, 16 Sep 2024 11:08:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4732A6B0093; Mon, 16 Sep 2024 07:08:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3F9226B0095; Mon, 16 Sep 2024 07:08:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 24C306B0096; Mon, 16 Sep 2024 07:08:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 03C686B0093 for ; Mon, 16 Sep 2024 07:08:17 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 8644C14033B for ; Mon, 16 Sep 2024 11:08:17 +0000 (UTC) X-FDA: 82570327434.20.463606E Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf05.hostedemail.com (Postfix) with ESMTP id 03E0A10000B for ; Mon, 16 Sep 2024 11:08:15 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=none; spf=pass (imf05.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1726484786; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DKmFxZ7weftyY68zIWFg14sI4661+BB6jesUl0LX+DE=; b=cQhpE1+BFrPbGvJGg+sRWuHVHcrN9odnAopOqbvNQXCFvVLyur5jAREcOR+rMCTp35P+NW 1dYHcS7GyUcFIYU4EIo/XYtsIBXo6vA+/yB6ZVEHTmYyZYDnUpUuyYsHElOvXc/ioV7xcN eSiSLrySbSO3FJnnBPeObE6vGTC9jUo= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1726484786; a=rsa-sha256; cv=none; b=BCaxkyknxmc7SjTnvkCWNAcqRnuQIsOmFiFjU7z6zG4UYsXaW924XxbFeG258MzuxjUVmM rTQqSA1MJmpDWKYmQbMkvUNnIGBtWE23795xvfjA/ra3tjxa4xPoJBGGf/VYIgZtLc7asN XpwBK46PSi4Kwtf1va7+3LmeV7UjXCE= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=none; spf=pass (imf05.hostedemail.com: domain of dev.jain@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=dev.jain@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 7F8571516; Mon, 16 Sep 2024 04:08:44 -0700 (PDT) Received: from e116581.blr.arm.com (e116581.arm.com [10.162.42.11]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 8E6013F64C; Mon, 16 Sep 2024 04:08:10 -0700 (PDT) From: Dev Jain To: akpm@linux-foundation.org, david@redhat.com, willy@infradead.org Cc: ryan.roberts@arm.com, anshuman.khandual@arm.com, baohua@kernel.org, hughd@google.com, ioworker0@gmail.com, wangkefeng.wang@huawei.com, baolin.wang@linux.alibaba.com, gshan@redhat.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Dev Jain Subject: [PATCH v2 2/2] mm: Compute first_set_pte to eliminate evaluating redundant ranges Date: Mon, 16 Sep 2024 16:37:54 +0530 Message-Id: <20240916110754.1236200-3-dev.jain@arm.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20240916110754.1236200-1-dev.jain@arm.com> References: <20240916110754.1236200-1-dev.jain@arm.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 03E0A10000B X-Stat-Signature: bgmihqrpzixqqogbdddibx81y4h7n65s X-HE-Tag: 1726484895-42278 X-HE-Meta: U2FsdGVkX1+63KugMnhKtSiTcw/FUrKnZWDuYX/ZX5yaqEWXkJn9OY7feEgnFzHECt6YzGu283EmEM+gIA4r96AlMWM2lWnF3zHCQuNf2Re1cJAL35fRJ7azShF1oygKfUPLTugDwCx9lul3T8nRCsy7rZbIeQjyEuR+lQHVn3wWZu8xgxD+3PcOLvgnDN9RBY/ncOCkS7E2jGBr54VomaANcJ7rLQkbi9l8v7FP0K7vqjdYod4Zbl3OrI2lgw3orC/U/qDM9pDVE3oaOg0JCQ6pD2hYJQJ3lfGOVfSZI9eJuK4pPVPtj0d2BISLoMHUBc81eGTswNvZ0cSU2aaSZaVBmHzD1yHdiM7XeNUnFMa/TiZ0ElHzJRLnBz6dFWKZhwk9JmRKaFKiEvkdP27c7DFyDOg197jdrzjciq19bCZr+PVjJh7tDknKhdc8rFnN1ZxP3NAVKTcayfH7RfjCOqu1pY7Kvv/UcaGpkFVCHBo7bJAbDz+q32vb0g98g9LlbEZgh5wyILM/7gmskU9qWvlTFkJvGRGlynBUfIL+Y0ACYY/U8gF5Qd68wRzn8GeMZ1A0mlxLdPclfUxrtWMW/Gf6lnLwCFDv7dE7HJDWcJj6dsrj+2Sj31xIcwTTfisoYdDTlZUGTyqLuGjFIOydHdJ+yq+kLHZv2r5t6bUCHAYwFpKw17oVdBVRJqkYTZAPQT8ANmN2UDxK+D/pigrXXYP/nofZW+vIL1IbFNXUUdNWsmQqnsvs+nNo7O1Cj1k+vpFIzk5Dzob1iAxX2R0/ztFIAle+QBk6F98GOvDnqkizxrhUXLEv+ACoIUPiNurjqudl32TAESD54VzCkIIT08BClEHhEElL1Z7iCvhcArk1onK0NbxhpEMrzOxzv2LHgOW9rmBAAZlg8RiHDb/kJHIoobHbjlNtItDkNxdgKmg6W7no6eU79hAt3lSdWf2WvuQvgFZiE74/JrlHQaF tc6nsJ+9 HpH3G/KV5FvJeXST/WaQyAMiKoxWHvBlQAgfIeXOdJ+3fNOLRHFMc9t0I8oUAMnLHZhiuV9+BJPRGZsaruUXEPtbcbgBTW9r0bo5qEUKKF8QIjAjnnrryHylKZtWjge4E4YTFRg6J1asi7P70Ed6ImOmSfGAy/1lmgTT0DwVXwpYcE5TWH4BtpSUS9QUXG/ZoMGIB1yxWyMkHhz8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: For an mTHP allocation, we need to check, for every order, whether for that order, we have enough number of contiguous PTEs empty. Instead of iterating the while loop for every order, use some information, which is the first set PTE found, from the previous iteration to eliminate some cases. The key to understanding the correctness of the patch is that the ranges we want to examine form a strictly decreasing sequence of nested intervals. Suggested-by: Ryan Roberts Signed-off-by: Dev Jain --- mm/memory.c | 20 ++++++++++++++++++-- 1 file changed, 18 insertions(+), 2 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 8bb1236de93c..e81c6abe09ce 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4633,10 +4633,11 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf) { struct vm_area_struct *vma = vmf->vma; #ifdef CONFIG_TRANSPARENT_HUGEPAGE + pte_t *first_set_pte = NULL, *align_pte, *pte; unsigned long orders; struct folio *folio; unsigned long addr; - pte_t *pte; + int max_empty; gfp_t gfp; int order; @@ -4671,8 +4672,23 @@ static struct folio *alloc_anon_folio(struct vm_fault *vmf) order = highest_order(orders); while (orders) { addr = ALIGN_DOWN(vmf->address, PAGE_SIZE << order); - if (pte_range_none(pte + pte_index(addr), 1 << order) == 1 << order) + align_pte = pte + pte_index(addr); + + /* Range to be scanned known to be empty */ + if (align_pte + (1 << order) <= first_set_pte) + break; + + /* Range to be scanned contains first_set_pte */ + if (align_pte <= first_set_pte) + goto repeat; + + /* align_pte > first_set_pte, so need to check properly */ + max_empty = pte_range_none(align_pte, 1 << order); + if (max_empty == 1 << order) break; + + first_set_pte = align_pte + max_empty; +repeat: order = next_order(&orders, order); }