From patchwork Mon Jan 13 17:57:29 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13937876 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0682EC02184 for ; Mon, 13 Jan 2025 18:00:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 89FDD6B00A2; Mon, 13 Jan 2025 13:00:34 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 850826B00A3; Mon, 13 Jan 2025 13:00:34 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6F17E6B00A4; Mon, 13 Jan 2025 13:00:34 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 46C526B00A2 for ; Mon, 13 Jan 2025 13:00:34 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id E8DFB160222 for ; Mon, 13 Jan 2025 18:00:33 +0000 (UTC) X-FDA: 83003193546.21.63E2BE5 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) by imf29.hostedemail.com (Postfix) with ESMTP id 5EB7512000E for ; Mon, 13 Jan 2025 18:00:31 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=M3Birquf; spf=pass (imf29.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736791231; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=w3/A4K7Bt53NzbRkeVq3PRJRV4Gi+Nq+TM9VfYB4Gnc=; b=h9nZTj3SMr9ADtz88b0MJpGBkElwOetsNB9lMEIFf8HGGZpMDpv2ZRVTvkD9mJ2BmJ32Tr alcIvDf5GbSCQeIFJ3hC67SpB+/EwUJt5QBoKw3pwjXwUdKKNnZuQKcgsXQ+cGkGDC5tuv O8HlRitWAt8YZs633xSc2dlZ0njYjeA= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=M3Birquf; spf=pass (imf29.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736791231; a=rsa-sha256; cv=none; b=XjgMOVVvSJHBZkeKT4/x26XD990jmB0Cpe7MY0N7f2YoHcZxfJVYmMa+a3vfZOGiwR4P9t r2KJDoLAMqIj8xdrvgY/dXizXgDUMHTjKrWlOvmyhHRB+VXO5KNMgA8Tc/mKzrFiyp9NUf pdkuYX87Hp4MgN1tFihzn6PmgYw/rgA= Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-21619108a6bso79469725ad.3 for ; Mon, 13 Jan 2025 10:00:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1736791230; x=1737396030; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=w3/A4K7Bt53NzbRkeVq3PRJRV4Gi+Nq+TM9VfYB4Gnc=; b=M3Birquf+PXe6u2xiez7B3HCh6K3G1BFEkeTLllZ9oMXxD3ElYJr/vyrPjE2fo/yqg hES3aa2FaLw6L/uepv1Ri2EtUwVzwwcRV8GYslbFaQLhWC/d3qniwaKk+kHCPtE7SVYP yuvM+6As9gCM3D4wy8muBPwAc2zfsiJF9XWkhHnGfus1MxDLM9zKR7ZkmJuLrTwUUIVv Q9/igm3+ViL+mHepJNIJIFOylP9/Br/mdT7cSzhB7CvuWyu28DQK6jbO0t5OqEtFTi4c Oty92FTR1ahZ/UYJ0VTgVx7Kf9KtTSFCQNBFU7tevA4n3S96K+UjA0vDLGD3PDL0XkHu doyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736791230; x=1737396030; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=w3/A4K7Bt53NzbRkeVq3PRJRV4Gi+Nq+TM9VfYB4Gnc=; b=OBbB+pt8DaiL6d3brIZvabyXe6wGmYPhrwRHr+ZhBmpz0wlYuKOHDagz6ujGrH4Rhg q9c2N+wawqb+n5fbQTN8D2McIZqOsAmTbWgWhNAmAWIcYiod5K0PSY7aBw/eAQx/UCoo Q59ZctGWJAU3U2kMEP0M78j4he96IpP4zfAa2B8+u2Fxc+DyHhn8xI8xr1O5w2O6jVWF OwT83baY+YoOzKiKsX+54B/RNFffQjewQ0HzUXORaXPeM2dM8MSWVi0zDyC1kwluJrSO wjStwPNWwRau62XndJzwHHoTosRJ7R0EecF6Wf8ZE0vXNf79dlDbTaFdsFFuQ4rdCddj ABkw== X-Gm-Message-State: AOJu0YyfC7pFm54uaTiwkwV7hgaAqfA1CYFEwZzi5FLnA/iEWkpeaiH4 Czeq7QXKRkR6MP1tNrzOCdL800Z2kjepVDeyxHJkgsHbdA+AmDjWzWbwKL1FyZc= X-Gm-Gg: ASbGncs9JyDnzZA8koMT9B9+1VfBI9/uGp0lP81P6LhH3A5vPl6VIw9gYbcZ9ca3AXS a8Je9OxsYtRYK6M2ENwCaF+QqkuFPbOuj/rxvxt1nlfjVW2Ae5igI+EXc4K9lee8kYMLMi3nW24 RktxHySOmrqukhgHrF+owDy0a9KRasE+99gkuiCgdFh47UCCfG3dyeav+OLJXC4qVNIibC7/M65 NumXCwUy72fRsKc92TWwVcfp+EGuDT3tsPryAbJoDAvZwrhUeFOgf1p6r2L8dj6l2hT9wp9LvMx 4A== X-Google-Smtp-Source: AGHT+IEb+lacgc6xktF1aaBt7hTsj1nPaNZqgDFnny/A4HmF9V2aouf4o4ZmssdJsfpBSua/NdU+2Q== X-Received: by 2002:a17:902:ea09:b0:216:2a36:5b2e with SMTP id d9443c01a7336-21a83f76879mr320846885ad.32.1736791229397; Mon, 13 Jan 2025 10:00:29 -0800 (PST) Received: from KASONG-MC4.tencent.com ([115.171.41.132]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-21a9f21aba7sm57023635ad.113.2025.01.13.10.00.26 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 13 Jan 2025 10:00:28 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Chris Li , Barry Song , Ryan Roberts , Hugh Dickins , Yosry Ahmed , "Huang, Ying" , Baoquan He , Nhat Pham , Johannes Weiner , Kalesh Singh , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v4 10/13] mm, swap: simplify percpu cluster updating Date: Tue, 14 Jan 2025 01:57:29 +0800 Message-ID: <20250113175732.48099-11-ryncsn@gmail.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250113175732.48099-1-ryncsn@gmail.com> References: <20250113175732.48099-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Stat-Signature: 7cgszzbq8dgrc7bpgoua94s5mnog6xkt X-Rspamd-Queue-Id: 5EB7512000E X-Rspam-User: X-HE-Tag: 1736791231-962225 X-HE-Meta: U2FsdGVkX1/mwJNV9jganSF+/NTs31Q3NLL4MWElofMpUZ+D4lStouTz3u7osYo3Rr6oBl+MlaXELy57Z3NgXKSTdldem9TlIoaslcFdQkUpDohpZ49Zz65VJorW9KP0M3hk4r/DPg/7DuNnH8eadRn1BUxh8Z1vcUq6QjMQ4NfqlEzVWMOleBBIEuhgi2EMRTT31JqrR4m0MWR5UY2iBBnCZ8rCCc/G/+qkht13zel1bM35ZF85i4hXkLJy0O21K1QhG9nk3v1YbChhXC8Lu5Q5WQzrPRfJyTemLWPOo5vbuGPF7+vXo4WVNAAj9TKqhUc96aabRRn8w6iB2Jafew3WTa82oBGEcfC+zIQSnEmCpuKPvwIXYJnltL7aC+KVpcyteNYu0VrBphf9LMpKC8R9gEWiFJ7BAZ1nmQcDhUBlJo0q0T7dIhQpswcdfNXKtkUHsz/6H2S1xM46e+W1YypSSmniIday67vtQvThYta1H+TOLBVVbF9bQ/34nR7ouwk1J/FbmKqi3Fk4errfQFUOD9jPOhKCzGbXslM4RqOGiN4zQ/Gft4cLRWwaNnmVaVc6M5PsMYcy2pSXdZrk3f0shgjsy1O51jYtpmVlezmFp84QQx7CV+LbaVXDh46GRn5ReTjEf8raw5PbtafXNTTp4EpdYjK088mHc9DDJUkQ42Wg7eUg5zaHXFqm4+gtWgflXRcvOWGcvlhpSJsEHPqhvHYs9ikEI2qoiWjBSpVDndShwR7rcuTRx8/MvL6UQWiIExTCoz6qZ0/OP7GB7DKj5H3UjXvSIE6xf1yvAwwaJmyzfXW6ZMQr8DVsTZ7F3R765TwxwjFwxCUifbyO4fh9lPkIQDjObT1y6jPmVJtJK1rRqeb8stSID3ebYXgm8ifUeKipqEAzYYmQNO+/g+jEzG7+jWg5vRghucvK0GkKcQVsFcCoGlPV1X0Z0X59EFwtl6C0If9YKyuj67i JvBx87+d zcvAwTHcsXHYUGGJZNpUnGWQ7M72X3f2lllOHdqLEkhtXSxqZ0rGQOapj7VDOsuwgFVq370s4WsT22M2x+KdOXseuGTKaTNf9yEDSy55Grj9x9tDLjySHJ0q+8SNA1IJzKqkLMQUPkrL0c8VeEW3SJSuJWAIRkZBpmv2EGUwzAvU5PoVtmE2zASwfzV/nU5VS4KzC/Wk7+W8glRJIAG3RtB9qalIZQvqBUFJ9sSLbZ0MRTSFAjTUqdeRHwBwy6t4f0k9HkzOyLn+ZTbtJ3bi5Slyx30khrGUyHnlcxgsH/IXJtTha+Z/QuXdH0EwYMONO8LJWWYXkrzXOB/uiWs+vw+66mlIF/ZobVJV9n9/agaPUEjay10fPd7rPzB+tSWSp/C/4cXLcb2uMCE1aPKsRTfnlIv+uNlU7vfYmBmarI8j3nib/QvmvmoEpyjwF9/GSiZwn/qs3sYzhSt7YQXzbiYBUrherYXIB8hPgqFjwrLstzUM3wWzikJJdOPdgxYsOWSbTH8m+YMiNMpEI1OUALEGAbQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song Instead of using a returning argument, we can simply store the next cluster offset to the fixed percpu location, which reduce the stack usage and simplify the function: Object size: ./scripts/bloat-o-meter mm/swapfile.o mm/swapfile.o.new add/remove: 0/0 grow/shrink: 0/2 up/down: 0/-271 (-271) Function old new delta get_swap_pages 2847 2733 -114 alloc_swap_scan_cluster 894 737 -157 Total: Before=30833, After=30562, chg -0.88% Stack usage: Before: swapfile.c:1190:5:get_swap_pages 240 static After: swapfile.c:1185:5:get_swap_pages 216 static Signed-off-by: Kairui Song --- include/linux/swap.h | 4 +-- mm/swapfile.c | 66 +++++++++++++++++++------------------------- 2 files changed, 31 insertions(+), 39 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index c4ff31cb6bde..4c1d2e69689f 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -275,9 +275,9 @@ enum swap_cluster_flags { * The first page in the swap file is the swap header, which is always marked * bad to prevent it from being allocated as an entry. This also prevents the * cluster to which it belongs being marked free. Therefore 0 is safe to use as - * a sentinel to indicate next is not valid in percpu_cluster. + * a sentinel to indicate an entry is not valid. */ -#define SWAP_NEXT_INVALID 0 +#define SWAP_ENTRY_INVALID 0 #ifdef CONFIG_THP_SWAP #define SWAP_NR_ORDERS (PMD_ORDER + 1) diff --git a/mm/swapfile.c b/mm/swapfile.c index 489ac6997a0c..6da2f3aa55fb 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -765,23 +765,23 @@ static bool cluster_alloc_range(struct swap_info_struct *si, struct swap_cluster return true; } -static unsigned int alloc_swap_scan_cluster(struct swap_info_struct *si, unsigned long offset, - unsigned int *foundp, unsigned int order, +/* Try use a new cluster for current CPU and allocate from it. */ +static unsigned int alloc_swap_scan_cluster(struct swap_info_struct *si, + struct swap_cluster_info *ci, + unsigned long offset, + unsigned int order, unsigned char usage) { - unsigned long start = offset & ~(SWAPFILE_CLUSTER - 1); + unsigned int next = SWAP_ENTRY_INVALID, found = SWAP_ENTRY_INVALID; + unsigned long start = ALIGN_DOWN(offset, SWAPFILE_CLUSTER); unsigned long end = min(start + SWAPFILE_CLUSTER, si->max); unsigned int nr_pages = 1 << order; bool need_reclaim, ret; - struct swap_cluster_info *ci; - ci = &si->cluster_info[offset / SWAPFILE_CLUSTER]; lockdep_assert_held(&ci->lock); - if (end < nr_pages || ci->count + nr_pages > SWAPFILE_CLUSTER) { - offset = SWAP_NEXT_INVALID; + if (end < nr_pages || ci->count + nr_pages > SWAPFILE_CLUSTER) goto out; - } for (end -= nr_pages; offset <= end; offset += nr_pages) { need_reclaim = false; @@ -795,34 +795,27 @@ static unsigned int alloc_swap_scan_cluster(struct swap_info_struct *si, unsigne * cluster has no flag set, and change of list * won't cause fragmentation. */ - if (!cluster_is_usable(ci, order)) { - offset = SWAP_NEXT_INVALID; + if (!cluster_is_usable(ci, order)) goto out; - } if (cluster_is_empty(ci)) offset = start; /* Reclaim failed but cluster is usable, try next */ if (!ret) continue; } - if (!cluster_alloc_range(si, ci, offset, usage, order)) { - offset = SWAP_NEXT_INVALID; - goto out; - } - *foundp = offset; - if (ci->count == SWAPFILE_CLUSTER) { - offset = SWAP_NEXT_INVALID; - goto out; - } + if (!cluster_alloc_range(si, ci, offset, usage, order)) + break; + found = offset; offset += nr_pages; + if (ci->count < SWAPFILE_CLUSTER && offset <= end) + next = offset; break; } - if (offset > end) - offset = SWAP_NEXT_INVALID; out: relocate_cluster(si, ci); unlock_cluster(ci); - return offset; + __this_cpu_write(si->percpu_cluster->next[order], next); + return found; } /* Return true if reclaimed a whole cluster */ @@ -891,8 +884,8 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si, int o if (cluster_is_usable(ci, order)) { if (cluster_is_empty(ci)) offset = cluster_offset(si, ci); - offset = alloc_swap_scan_cluster(si, offset, &found, - order, usage); + found = alloc_swap_scan_cluster(si, ci, offset, + order, usage); } else { unlock_cluster(ci); } @@ -903,8 +896,8 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si, int o new_cluster: ci = isolate_lock_cluster(si, &si->free_clusters); if (ci) { - offset = alloc_swap_scan_cluster(si, cluster_offset(si, ci), - &found, order, usage); + found = alloc_swap_scan_cluster(si, ci, cluster_offset(si, ci), + order, usage); if (found) goto done; } @@ -917,8 +910,8 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si, int o unsigned int frags = 0, frags_existing; while ((ci = isolate_lock_cluster(si, &si->nonfull_clusters[order]))) { - offset = alloc_swap_scan_cluster(si, cluster_offset(si, ci), - &found, order, usage); + found = alloc_swap_scan_cluster(si, ci, cluster_offset(si, ci), + order, usage); if (found) goto done; /* Clusters failed to allocate are moved to frag_clusters */ @@ -935,8 +928,8 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si, int o * per-CPU usage, but they could contain newly released * reclaimable (eg. lazy-freed swap cache) slots. */ - offset = alloc_swap_scan_cluster(si, cluster_offset(si, ci), - &found, order, usage); + found = alloc_swap_scan_cluster(si, ci, cluster_offset(si, ci), + order, usage); if (found) goto done; frags++; @@ -962,21 +955,20 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si, int o */ while ((ci = isolate_lock_cluster(si, &si->frag_clusters[o]))) { atomic_long_dec(&si->frag_cluster_nr[o]); - offset = alloc_swap_scan_cluster(si, cluster_offset(si, ci), - &found, order, usage); + found = alloc_swap_scan_cluster(si, ci, cluster_offset(si, ci), + 0, usage); if (found) goto done; } while ((ci = isolate_lock_cluster(si, &si->nonfull_clusters[o]))) { - offset = alloc_swap_scan_cluster(si, cluster_offset(si, ci), - &found, order, usage); + found = alloc_swap_scan_cluster(si, ci, cluster_offset(si, ci), + 0, usage); if (found) goto done; } } done: - __this_cpu_write(si->percpu_cluster->next[order], offset); local_unlock(&si->percpu_cluster->lock); return found; @@ -3200,7 +3192,7 @@ static struct swap_cluster_info *setup_clusters(struct swap_info_struct *si, cluster = per_cpu_ptr(si->percpu_cluster, cpu); for (i = 0; i < SWAP_NR_ORDERS; i++) - cluster->next[i] = SWAP_NEXT_INVALID; + cluster->next[i] = SWAP_ENTRY_INVALID; local_lock_init(&cluster->lock); }