From patchwork Fri Apr 4 15:59:56 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Woodhouse X-Patchwork-Id: 14038660 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E9AC3C36010 for ; Fri, 4 Apr 2025 16:01:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 68EE76B0007; Fri, 4 Apr 2025 12:01:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6669C6B0008; Fri, 4 Apr 2025 12:01:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 476C26B000C; Fri, 4 Apr 2025 12:01:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 22F846B0007 for ; Fri, 4 Apr 2025 12:01:55 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id E5B0DAADBF for ; Fri, 4 Apr 2025 16:01:55 +0000 (UTC) X-FDA: 83296827390.14.BC33759 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf19.hostedemail.com (Postfix) with ESMTP id 68DB51A000B for ; Fri, 4 Apr 2025 16:01:53 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=DXHzSB3s; dmarc=none; spf=none (imf19.hostedemail.com: domain of BATV+be2fdb05a07bd611d225+7894+infradead.org+dwmw2@casper.srs.infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=BATV+be2fdb05a07bd611d225+7894+infradead.org+dwmw2@casper.srs.infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1743782513; a=rsa-sha256; cv=none; b=te1Z9lifbZJb0S4alXxMjcm0m5EtgO0LP7rjOKmnvtAcb37yQc+ZeI6K60eUgixPr5mFjZ V4vDA4Oz2kvcRVYvq/1mWOtSI8enzoLtLgjyi3pIPmjXr7MseEniHTjgDcwwIfoFmsLa3Z HDDfCGIpONnkLupGxOmDqmb2KoU1N5k= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=DXHzSB3s; dmarc=none; spf=none (imf19.hostedemail.com: domain of BATV+be2fdb05a07bd611d225+7894+infradead.org+dwmw2@casper.srs.infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=BATV+be2fdb05a07bd611d225+7894+infradead.org+dwmw2@casper.srs.infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1743782513; h=from:from:sender:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=O+396y0rDBKXn/tGzn/xwAn8Ku375WZ2ZAT9PS+Ub5w=; b=1TPkLyAOwNJur1tuIKDfJCDxRfwTeJYViUgGqIiDCtqQ9ytuguHdoUzwY5W6VL/ecPc73X f7Iyd1n6JvBdUWFY4cRudiCf92f8LZvG5sPxnuoLaq41jyrAJ6qAp910WYLNy/op01Fonw VK7twacm956jrn2qZXNIeUSZxQh4ATI= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Sender:Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From: Reply-To:Content-Type:Content-ID:Content-Description; bh=O+396y0rDBKXn/tGzn/xwAn8Ku375WZ2ZAT9PS+Ub5w=; b=DXHzSB3si2aN61sd7dOw7Dkduk TfChbYRNdb3ACgbes0kuBN7Ps4Sy5/UipeFG8/Xla7pFYuFAZjAv8ghlbw60pID0KBmbGy7KhdoZX xFeg4W6Khe91Ifz1tB9jPFsxfe3bqKisa6rPuuYszWXM+JzP0nj3JG8rUww67uY/Mv9WHKvRsSeQX Ok99/gqCaltwnaMSZvz4drLWaLh2mGbx5jBswPyNZtO7LBeN9BcJh4IWoGNNd4OiLAnX2VY16JZlL QP6zoE/Go4gQZXoAjZGtme0XAK6mXGvw/o5DoYFynT49usw3UBp1GA3ggRsbCxmI+nr/eYZ4vp6BK LUawAvyw==; Received: from [2001:8b0:10b:1::ebe] (helo=i7.infradead.org) by casper.infradead.org with esmtpsa (Exim 4.98.1 #2 (Red Hat Linux)) id 1u0jSa-0000000GFHy-3ANo; Fri, 04 Apr 2025 16:01:01 +0000 Received: from dwoodhou by i7.infradead.org with local (Exim 4.98.1 #2 (Red Hat Linux)) id 1u0jSZ-0000000ERSK-3V46; Fri, 04 Apr 2025 16:59:59 +0100 From: David Woodhouse To: Mike Rapoport Cc: Andrew Morton , "Sauerwein, David" , Anshuman Khandual , Ard Biesheuvel , Catalin Marinas , David Hildenbrand , Marc Zyngier , Mark Rutland , Mike Rapoport , Will Deacon , kvmarm@lists.cs.columbia.edu, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v2 4/7] mm: Optimise SPARSEMEM implementation of for_each_valid_pfn() Date: Fri, 4 Apr 2025 16:59:56 +0100 Message-ID: <20250404155959.3442111-4-dwmw2@infradead.org> X-Mailer: git-send-email 2.49.0 In-Reply-To: <20250404155959.3442111-1-dwmw2@infradead.org> References: <20250404155959.3442111-1-dwmw2@infradead.org> MIME-Version: 1.0 X-SRS-Rewrite: SMTP reverse-path rewritten from by casper.infradead.org. See http://www.infradead.org/rpr.html X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 68DB51A000B X-Stat-Signature: pd4hqymxhpakqnqre9tdwxxpxydojx3h X-HE-Tag: 1743782513-524709 X-HE-Meta: U2FsdGVkX19aMbCisULPSRYnbL4Y+99VT4wO0V+lLyXx+bgNqJZJxf9+RKbhKe3e96vIRtAQpoZw/uDkjJt73y1ic0527QCaevKIFGmiOv+BHGX29SWX6d5Pi8kXhq4S+VrJUI6IbmbLAeoMAEAEv6NFMeHYHK5UamSKAr/gURJ7SbKQTf3UgP4gbSAqp/Gi8xw3/c9yPSenQrEeluqPLc0gBNgPWpEPJ6oFCF/AiLdhH2P8kzpm+2kUdmUDw/zWe0daYmNW4ztNiQMRBDTZpbIBHkAtRfb3LulhRSpzOPCdF39IOjdxdAuc7JNjqSVCcSgjIVlD1qYyrYuZzDRBW9/ypRdhJd2O2X+gScFkhuMbF8s8AH4s80B74w4J1sAjvJUT+RFQZrVcIz0IW6f/kunsR8iQ0GkhtGYAXafpU8Hhw/nagoRJhAM4mDhqswXWSnGEGTj2TdEsRgocljjLQ8cY/PIeni3grBafOKcj1biA/fO4QBJSsvxka4YXLrErnUnro2lxem5MuGbDB0GzSGkr2ZhQr34w37lIR7Jnjqlru7c22YoO9IRUeZPDBWLSrJPV88qLA80XfYcLKM3CUDE41VVwTU9VeqyhCkp1KJgdV2LL5nWnhmYKQK3R5dLMYnMkTzPuZ8zjRM56cRBaxW5YkKH13z6iAEX/Ebb1IUXlOW5gqBjPIQFynXA0twcvIsXGwhrmVJmd1ZqlxiLg/EMXp0ZQty1YxxAKmniZrqhHxP+Pkv3s764D1VvNO86FFzvSf0ARsIDEDIGmqYOhRBCpiB9MVywsxyhls9p8y4Hob7eh3WRFyYAPjYqMgrLuXdDHgZP4DdAfJ0vm0OgqObr5dXnRbXBCmDIGxEauMnRkx/xPrSvYQ7PvTzaiHi7sZunIP7+7jGXgFbelOsEHjKPjCvY8zJ4gFG6iLpkONxFlsnpjbgTdavS9cb6aJRNPQwpn+mIbad7sBRtZvsr 0SWNfW+y 3MqyywD58KfarvuGsYaT92O8kdOVFXEzWzVeAUBqoZEN/hT3hvV197uggNCJ3nOZ4Asfx7ixvC1K8sOEJG1GYHdnBqFQqjjvAcInjlmqmotb/MWCrdtIhQPXK80M6HjA03oBvYeVHnhnMz+GA6Pie6GfhzoZV1MsI9yCgODKUAkbm2cf9yagA64Rk+ky+d7fxtQOTOiprgG78i0/J+3m9Cxld6N4Yfa+HMAIKfCqQgEWgQ/Kqf2cFX5WvvZon1G6M24q0CCR6yGa+NRzCjfz2aNr5TA0J6xH7RZ5TFmR8JlX8VA8oklVDcBA3vnfGbRglIK8W5K1KVySLttEv11r2NJw4yg4OfvXVorpPq37NJdPT/i6oxwWc9dDzuXUExKIg1gFBTtSvNcOoZLeLp6TX6fSmvMSxcvQwDF2popAsaRlnAXFhyYI8ai3NrGVN9+/MUljdKBeopxsLGZNTvGW0nBjdruxlUOv2R3oqLIpiV7xI2vKa7dmuxb3kWvrYcelh/SqAJDx7TAVe51maoOyXpbdmFPkfSw5pgPsg X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: David Woodhouse There's no point in checking the section and subsection bitmap for *every* PFN in the same section; they're either all valid or they aren't. Signed-off-by: David Woodhouse --- include/linux/mmzone.h | 39 +++++++++++++++++++++++++++++---------- 1 file changed, 29 insertions(+), 10 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 67cdf675a4b9..0da1b0ba5d9f 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -2154,21 +2154,20 @@ static inline int pfn_valid(unsigned long pfn) return ret; } -static inline bool first_valid_pfn(unsigned long *p_pfn) +/* Returns -1 (an invalid PFN) if no valid PFN remaining */ +static inline unsigned long first_valid_pfn(unsigned long pfn, unsigned long end_pfn) { - unsigned long pfn = *p_pfn; unsigned long nr = pfn_to_section_nr(pfn); rcu_read_lock_sched(); - while (nr <= __highest_present_section_nr) { + while (nr <= __highest_present_section_nr && pfn < end_pfn) { struct mem_section *ms = __pfn_to_section(pfn); if (valid_section(ms) && (early_section(ms) || pfn_section_first_valid(ms, &pfn))) { - *p_pfn = pfn; rcu_read_unlock_sched(); - return true; + return pfn; } /* Nothing left in this section? Skip to next section */ @@ -2177,14 +2176,34 @@ static inline bool first_valid_pfn(unsigned long *p_pfn) } rcu_read_unlock_sched(); + return (unsigned long)-1; +} - return false; +static inline unsigned long next_valid_pfn(unsigned long pfn, unsigned long end_pfn) +{ + pfn++; + + if (pfn >= end_pfn) + return (unsigned long)-1; + + /* + * Either every PFN within the section (or subsection for VMEMMAP) is + * valid, or none of them are. So there's no point repeating the check + * for every PFN; only call first_valid_pfn() the first time, and when + * crossing a (sub)section boundary (i.e. !(pfn & ~PFN_VALID_MASK)). + */ + if (pfn & (IS_ENABLED(CONFIG_SPARSEMEM_VMEMMAP) ? + PAGE_SUBSECTION_MASK : PAGE_SECTION_MASK)) + return pfn; + + return first_valid_pfn(pfn, end_pfn); } -#define for_each_valid_pfn(_pfn, _start_pfn, _end_pfn) \ - for ((_pfn) = (_start_pfn); \ - first_valid_pfn(&(_pfn)) && (_pfn) < (_end_pfn); \ - (_pfn)++) + +#define for_each_valid_pfn(_pfn, _start_pfn, _end_pfn) \ + for ((_pfn) = first_valid_pfn((_start_pfn), (_end_pfn)); \ + (_pfn) != (unsigned long)-1; \ + (_pfn) = next_valid_pfn((_pfn), (_end_pfn))) #endif