From patchwork Sat Feb 17 02:25:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 13561212 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 78530C48BEF for ; Sat, 17 Feb 2024 02:26:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 009426B00A2; Fri, 16 Feb 2024 21:25:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E82BE6B00AA; Fri, 16 Feb 2024 21:25:55 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C0BA26B00A9; Fri, 16 Feb 2024 21:25:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 990AB6B00A0 for ; Fri, 16 Feb 2024 21:25:55 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 63810120145 for ; Sat, 17 Feb 2024 02:25:55 +0000 (UTC) X-FDA: 81799705470.04.4B4BFF4 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf13.hostedemail.com (Postfix) with ESMTP id C047120005 for ; Sat, 17 Feb 2024 02:25:53 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=FejDrfEI; spf=none (imf13.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1708136753; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+aNucfAnCh67XPHQO85RwlKe2XqWfNSUlxuOSTQj5iw=; b=5RmDgcOO+iNGevog94NVYp9hNBWBJC0rmOoaZZFW4x10oQt7zCsmDQez2Zw202Kjbr/6R2 koBIWFYYCgbgf3KPO5NF3J1KmZeQS4vvJyUHw8SeI5UXYRVELZ4Wanwu1aiT27txeupYYG J9WA30G7zqtT9/ZOs+kv1PwPF3tZ01A= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=FejDrfEI; spf=none (imf13.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1708136753; a=rsa-sha256; cv=none; b=lybm11dMFhqOT/KiciIn6VtIGDJpYkXl/aGf6aRIAW2V6BlKP/yB6vJrSGB2xeh2PApd0B LAfa7AdRgcHhS5WD5NT8lCm7nY9knFSdRzIclmRzgseEQx4aTuodmKyDv5dl3IV38teVsZ 8rqiVZ1p6AULYs74UOWjfLYfIByuUOc= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=+aNucfAnCh67XPHQO85RwlKe2XqWfNSUlxuOSTQj5iw=; b=FejDrfEIOWy7DYFg01udT8aGcx x24y4hdVgL8Kzm+LkOo+9ym9bOqm7jpDL9S1ptDMCFQ5Z78IlkodphVk3N6SI+MyoJjRDM9aglnMl m2AmpRiTdSRqEXYfsqWuIpGE1B62lNJSYtApcNalyA8YGwGIHsNfPUUafAuCYj7PnLthQ5C25IBF0 vSMbasW8KnxGYsyF6WzciNgoDzBoYxx/je0yTG0zUXAKWYUS1EUpi2YFc/Dd82BrlB1jBtf74yhqP slVNA3tWsOAB+CJs0/drv0VfBefj5sJTYQXgCbij810OKd++D9pi4eNiX0C386xAhBIOLMBSiX+tu csFh6tsw==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rbAOi-00000006HDN-46fA; Sat, 17 Feb 2024 02:25:49 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v2 09/18] mm: Handle large folios in free_unref_folios() Date: Sat, 17 Feb 2024 02:25:35 +0000 Message-ID: <20240217022546.1496101-10-willy@infradead.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240217022546.1496101-1-willy@infradead.org> References: <20240217022546.1496101-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: C047120005 X-Rspam-User: X-Stat-Signature: bhg4acxsj9jbff337ihsncs54s9az1xo X-Rspamd-Server: rspam01 X-HE-Tag: 1708136753-828762 X-HE-Meta: U2FsdGVkX19fjbuFw0M90iAQqvesI8wTg4GCbIpipUWjUYok8qKVtVEswJsqB+eWXOoVbaubXp55J6ZsUJHmS7YywS5kbWNEnIjAcfCol2/6bqMeAYKq+++Ca8ISTiBZE7+8NnM1VERgrhgUBoT1ovQVcGoR2mmYqo2eewStjc+BGQ1ZyJLQvOmhylkMK481jb4yN8Xx+amioUA8+/CS1VMF3f6AlpkBNCwaRa5t6nnLdEI61GDBihBjA/a2/jBRjeNaeba4o2HHzB9w/ooxcGvFvowNdY4DOifwf1Pu0jn7z2kDN5LOOgBuS43/bd5gEUwEfEdknCHkzRg1XbvmModHq4NZnB4JEUIxp82QwWq9+U3dHVfOwRtbZlOgDgE5MmFf9OlJkZyYuK48ADXl56Dz8xeXSSYYSi3ezf9cLj1+7Zh1zEIXrP5lUJO+h3hmN1Pt+Cuu8Hqr/VjnBEppIftSmKbPC+vpcSqJKSBgazIuiU49Y2L1UKTBN/LNAs9xUG/Oq0jlgzORVpl2jlBu2Qiv5ZR8zvqG+megRkef7+uxY+nTTdWq+pHynvjRhBWISHh9S7vFjctjKLkCm0Aqqeo1fRhQZ+y0YnUKokIXRy/44Uby5vt0TCc3kTM003QLYCUNykXVs6ocTx0cGhCIAsbhjgi/lM9VfpUEoxJD9ZlcnFIOr1puKy4a+Rd2VhbMJoyKb///VIbk6KO/c4QtL2nMrpny/kM64yijY3R78aNKOy6ThIOioawsDof4rLeDfKGbxzXPfyOjnPAgyxEL7RK9qWee93wHmJhvfrI4tAO344aLoLsxn5WcVrwh5adOf86pHNtcSqxdIMFDrnLTMVM/vNIaKUIvPQPsS8bdkAXWSyb+eh+Y9Y9MHSXs3WDFqI2xFCWGG+52YOPur7Vba8VsgB5XKIXF5AWX9LNElKvt/qLJ7b6OiXLopSabXV21W8MbKIx2fy2865U06ry 97j+qM+X IAZ5PFnAfSKTPy00wCb8CM8vhqn4CgIjXnG3tRmrhPqXwPpAnLd4R8NtOnCkZCcvOv6r/gIkwzqWbTwz4ExNOKYkiDorTAuh1t+1NkMchV4lslywdcHazUhx7JCVD38I3Xya2Yw34wB7orMc3gHj8f2rL0np87XUm5keIp8LKZKRLfd14QaMDuvc3bz+Ap43ESh8Yu+0BkPL/zDsqVMc/1OyRvx7FY0O89kGw X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Call folio_undo_large_rmappable() if needed. free_unref_page_prepare() destroys the ability to call folio_order(), so stash the order in folio->private for the benefit of the second loop. Signed-off-by: Matthew Wilcox (Oracle) --- mm/page_alloc.c | 25 +++++++++++++++++-------- 1 file changed, 17 insertions(+), 8 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 8ef1c5c86472..eca5b153f732 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -2552,7 +2552,7 @@ void free_unref_page(struct page *page, unsigned int order) } /* - * Free a batch of 0-order pages + * Free a batch of folios */ void free_unref_folios(struct folio_batch *folios) { @@ -2565,19 +2565,25 @@ void free_unref_folios(struct folio_batch *folios) for (i = 0, j = 0; i < folios->nr; i++) { struct folio *folio = folios->folios[i]; unsigned long pfn = folio_pfn(folio); - if (!free_unref_page_prepare(&folio->page, pfn, 0)) + unsigned int order = folio_order(folio); + + if (order > 0 && folio_test_large_rmappable(folio)) + folio_undo_large_rmappable(folio); + if (!free_unref_page_prepare(&folio->page, pfn, order)) continue; /* - * Free isolated folios directly to the allocator, see - * comment in free_unref_page. + * Free isolated folios and orders not handled on the PCP + * directly to the allocator, see comment in free_unref_page. */ migratetype = get_pcppage_migratetype(&folio->page); - if (unlikely(is_migrate_isolate(migratetype))) { + if (!pcp_allowed_order(order) || + is_migrate_isolate(migratetype)) { free_one_page(folio_zone(folio), &folio->page, pfn, - 0, migratetype, FPI_NONE); + order, migratetype, FPI_NONE); continue; } + folio->private = (void *)(unsigned long)order; if (j != i) folios->folios[j] = folio; j++; @@ -2587,7 +2593,9 @@ void free_unref_folios(struct folio_batch *folios) for (i = 0; i < folios->nr; i++) { struct folio *folio = folios->folios[i]; struct zone *zone = folio_zone(folio); + unsigned int order = (unsigned long)folio->private; + folio->private = NULL; migratetype = get_pcppage_migratetype(&folio->page); /* Different zone requires a different pcp lock */ @@ -2606,7 +2614,7 @@ void free_unref_folios(struct folio_batch *folios) if (unlikely(!pcp)) { pcp_trylock_finish(UP_flags); free_one_page(zone, &folio->page, - folio_pfn(folio), 0, + folio_pfn(folio), order, migratetype, FPI_NONE); locked_zone = NULL; continue; @@ -2622,7 +2630,8 @@ void free_unref_folios(struct folio_batch *folios) migratetype = MIGRATE_MOVABLE; trace_mm_page_free_batched(&folio->page); - free_unref_page_commit(zone, pcp, &folio->page, migratetype, 0); + free_unref_page_commit(zone, pcp, &folio->page, migratetype, + order); } if (pcp) {