From patchwork Tue Dec 24 14:38:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13920198 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 55474E7718E for ; Tue, 24 Dec 2024 14:40:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DA9D86B00A1; Tue, 24 Dec 2024 09:40:15 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D342C6B00A2; Tue, 24 Dec 2024 09:40:15 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B85246B00A3; Tue, 24 Dec 2024 09:40:15 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 968FA6B00A1 for ; Tue, 24 Dec 2024 09:40:15 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 3F11D42721 for ; Tue, 24 Dec 2024 14:40:15 +0000 (UTC) X-FDA: 82930111488.21.BFFBEBB Received: from mail-pj1-f46.google.com (mail-pj1-f46.google.com [209.85.216.46]) by imf03.hostedemail.com (Postfix) with ESMTP id 749A120015 for ; Tue, 24 Dec 2024 14:39:55 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=IRvSFmIk; spf=pass (imf03.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.216.46 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1735051194; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OeMbPM60Q8v3Qfaz9R6/9c2/Ea7K6oyXleY7O4H/Tp8=; b=7YXj8JtZUXAUkqOy4NVcrxkO6G/NbrbvX6tqKuzalqy1kCXsvftlS+AzVeg3m/GhGZ6mJg 6FV4aMbbHEAUiNe8FzFcvGL3tQskYBhxri7Z3rHRoESqXNvfkNJ3Xhcj/6zVeJS6ckkXPq Aoz6u1HnvaxAkw/+kgw0FnnPRVbL00I= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1735051194; a=rsa-sha256; cv=none; b=DVxcp2ElOE6sWPYAq1Jy/Tc1cyJm9agRIgwCZenOGhf9ILQhfBZJ1UbDwu1upRntvMvsSz 2sm8kaI2aNjOt07jULpee+7a4Sw60OjGWDUTPApizTIiXyW+TuJ/zX4MFlA+djlsov951H bBP9TBofaiSX4Qw5PLXjziMrXzOaHRA= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=IRvSFmIk; spf=pass (imf03.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.216.46 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-pj1-f46.google.com with SMTP id 98e67ed59e1d1-2f13acbe29bso5404989a91.1 for ; Tue, 24 Dec 2024 06:40:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1735051211; x=1735656011; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=OeMbPM60Q8v3Qfaz9R6/9c2/Ea7K6oyXleY7O4H/Tp8=; b=IRvSFmIkgMGqvmrUPbkchfre5QebmdSl10tAb+mXQAEv+BSwNAiLZJ7Sebz/1+nPp8 ctQPkuFSjo1gZiX+KC1+RmNtktRPElTXTQrt8LZFAN/RBqMSJL2bp81ICgf6r8Sj2/S0 ZS8Ct9vKHaKHTjFaksEYwWdMv7DCJBKcUfehp4lSKb2rRn6EmeVAIAyJHH76t9X2zK5C v61pSJ/Trix66pAzoxsC3w+1fOpOZbWz1EoTy+h0xW5xKNaQYMOL01x89kt0Dizlksqt 7pubSRqHGCYXBZzUOSLE/WkoSnoWZktJlznrEb5GTQeVp25fEsRcWsubpJ+lo30ABeVc geng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735051211; x=1735656011; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=OeMbPM60Q8v3Qfaz9R6/9c2/Ea7K6oyXleY7O4H/Tp8=; b=frYOQdk3efptKVpFnBaNL/BTOdc/6Yb9mOhIPIK8y/x8cqUtJ+I8+ATiVO2VIdOw4S Kx2i8HT3qyYn1vyFTq59QK08Xb3rX11pwrzDyioeoPLnPAodqxrGIYKklBB7CAgV2SUD EmVz4AkkyY9IprOb2GqPiqLuMwgZ3OQhB5P/M9ttZLhEITDwO0vPPxjq0ZFClYA0LLmO LV9uPMs0lgHYSfGYKLKoJX2t1GZHljUIHcYM2bEtVpX9GsYNfqRAxJTKJplTjMjIL6d5 HM5knOANn7K7ytBbi1i9LflQ1vZUTSXhj0r+SF2uR3guLLD1XW4SLn9DvIKFpRdJhV4b bqMg== X-Gm-Message-State: AOJu0YxHz77YHDw1CwOYvCrCPWTOBkmSNQAxIBZym1XUhwn1uC8piMIK cac/Azo44BmZy2UZDWsyV3kdO67eA0BShRhsFsAEjtgklGu4YAlcLkn6BKCvy0s= X-Gm-Gg: ASbGncueSgyFybpvVI/ulEbWcluCddexgRvlV4FTEpLCkCcug16GzmkX7/ZSP9VcLfU TAMMAOYFeJhZR6Zxl+nAaaj63/tWXXb5Qlx4MfOGkZfMIIhKLEz7HNH3rOZqGGZeQwpEeL72OQa LXHt5wTE1MDQUzYvlfnguIJIAJWn2a6r7egQ3wCfwOraV7QppH7BFjYd6xv7jHwB0WKewH9ZL7b O7DGrdwVLfEFPWGpZrPmFsKD3b5Yv6FyxDlGEA1hLWvq1PXg8SWqs2PE/MR7V249wN893q6haaL Zw== X-Google-Smtp-Source: AGHT+IGbB/29Y+1ZgUGrQ+JERD995978jUTdlYdu6QWEKgLiCQEmWerJkgBsrHNrNTQtYolF6Xq0WQ== X-Received: by 2002:a17:90a:d64f:b0:2ee:9661:eafb with SMTP id 98e67ed59e1d1-2f4536d18a7mr24570769a91.12.1735051211431; Tue, 24 Dec 2024 06:40:11 -0800 (PST) Received: from KASONG-MC4.tencent.com ([115.171.41.189]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-219dc9cdec1sm90598735ad.136.2024.12.24.06.40.07 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 24 Dec 2024 06:40:11 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Chris Li , Barry Song , Ryan Roberts , Hugh Dickins , Yosry Ahmed , "Huang, Ying" , Nhat Pham , Johannes Weiner , Kalesh Singh , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v2 10/13] mm, swap: simplify percpu cluster updating Date: Tue, 24 Dec 2024 22:38:08 +0800 Message-ID: <20241224143811.33462-11-ryncsn@gmail.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20241224143811.33462-1-ryncsn@gmail.com> References: <20241224143811.33462-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 749A120015 X-Stat-Signature: 4ojh1d83zjoeqne11f86gy6ogt5ehz3u X-Rspam-User: X-HE-Tag: 1735051195-351472 X-HE-Meta: U2FsdGVkX1/KV70RlT/eaCBFTRRyNAIVHujp9dMe1Y1WIJtVpCvbeQjbZ585r//YrGtODs5KDT3uTiOXV6lvHUh6oI7I4SW14uUGROKdpmUwWtYMi+WoY2fosy/fgMHKruT0Mb/C3glx2624EEubOX1VVYj7LEI3xJ4TbD6VibPMe7YSWBqKXRrZYrb8d6k64In1/zRdMvu/u1cM0PIuyfJXYeJMGH5RoHGtDxgxLf961dOVrjPpfkD3Zc0reJ8yzkN3dvL22K/69LtCimvdxaEC5hEim3ANROiOBm1dr6mItxIkC0OlLFclMiyUPi6P+lBO9R4P+kcWcLV2HrKEejx//nt7O4mFztEtzcHIz+WYDRj6hOCxNjlYJ+l0UqalaNNz0SlhJ53pXyZCwOzde/4N0dHK+LEUTTHaINyLCeWiAk570T2193vV9I3PqwYipl/j3HnHek6vdQJDwhA+Bk7DxNeIkftaoybRaFSK1UdHJxUKWgAKWl+IpdooYUWVWr4GKADKexOZMYpGbrLLhHF10I1a7kgR5LqzCrkKveSeOqizJaEVUtG/zz+AL/2Dis4+LiPHCYH1cmIFg6k3UlBK3PlOG658osCIGcD6egzSZ+unvZr50O4zsyDoF3dgaA8XAcjNEuVAvVBeWJhFuGBJ+gsPo2MhVizgje5R6Gi8VbFSNusqC1w0GvnZ9mIVLA7SG8kN77TyVIPX2J7rMpF+pRk4SSzNY251Sz4uJr7lcyfaWpWdoTgpW8vJ59aLnFZWbYhwUmaVjrL1lrwSo7unCjEtvyAeXzTgvTXxEoDU50mHaSgrXiYqd/odxBIejqucLatzfVDTuKcantMRAYj1lOzj7qdtNx6rWZ25BVtHGj2FFdjUCUxGcucjrgQUHsIlijj/YZbGyBv1dGk8cpI5IrPHsQsR3KoD89rVknovx0Qc+AyZWXdPCEsORH3s/6SeJyhBZEW20xD3AGA ucXbUMj2 kzNL1/ZGuYp6TCi2tQCzQYY/EvISygTs5JAqxqKEVkn7Da4EkHhcx5A3AqUEN3x8ZTkybNDiensGBl8/8BdLv4HNtG+7bvTZauyxg7gOOblqHqR5bZKF+s0uRAJddi6utZL7AQK5LUSb+dYHTjvyO05lqyuVxdskRQkDSdHwNxJRA80j3YVOT35lFtSFCTs0zUzfmLnJH1NrvZMqk7dwpUuL7Y3uPYHuLJRYgKDIf5xXwC+nfj9SiiIWI/tFTrSoZ6fcF/igOaeAYz9Pgxbwc9P7mpqeHZIqca6ZAQ+0+bj2VdPhTQnME6iNdHDA2hGKPw1/FNxKvZAwLmu4tw6a2o5pMz6kdifYyqplzjscO2FiDAZsYxRtw4L9ZY3236qybnloC5yhXGvzG3FZyq+mKdzTQ3WnvtsimtriS7LXcyt4rnL9zwrFNdUuYTMI4aRd5EDnv+KOJrO487aGHBALqZAL+XYrfZ7ziEnOKm7l60010Em9PA15UcLoJBzbZH4sDqPLQRo6JcL5o1xl64gp/O16uMA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.001635, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song Instead of using a returning argument, we can simply store the next cluster offset to the fixed percpu location, which reduce the stack usage and simplify the function: Object size: ./scripts/bloat-o-meter mm/swapfile.o mm/swapfile.o.new add/remove: 0/0 grow/shrink: 0/2 up/down: 0/-271 (-271) Function old new delta get_swap_pages 2847 2733 -114 alloc_swap_scan_cluster 894 737 -157 Total: Before=30833, After=30562, chg -0.88% Stack usage: Before: swapfile.c:1190:5:get_swap_pages 240 static After: swapfile.c:1185:5:get_swap_pages 216 static Signed-off-by: Kairui Song --- include/linux/swap.h | 4 +-- mm/swapfile.c | 66 +++++++++++++++++++------------------------- 2 files changed, 31 insertions(+), 39 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index c4ff31cb6bde..4c1d2e69689f 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -275,9 +275,9 @@ enum swap_cluster_flags { * The first page in the swap file is the swap header, which is always marked * bad to prevent it from being allocated as an entry. This also prevents the * cluster to which it belongs being marked free. Therefore 0 is safe to use as - * a sentinel to indicate next is not valid in percpu_cluster. + * a sentinel to indicate an entry is not valid. */ -#define SWAP_NEXT_INVALID 0 +#define SWAP_ENTRY_INVALID 0 #ifdef CONFIG_THP_SWAP #define SWAP_NR_ORDERS (PMD_ORDER + 1) diff --git a/mm/swapfile.c b/mm/swapfile.c index bb1ef9192d99..ac170acf55a7 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -759,23 +759,23 @@ static bool cluster_alloc_range(struct swap_info_struct *si, struct swap_cluster return true; } -static unsigned int alloc_swap_scan_cluster(struct swap_info_struct *si, unsigned long offset, - unsigned int *foundp, unsigned int order, +/* Try use a new cluster for current CPU and allocate from it. */ +static unsigned int alloc_swap_scan_cluster(struct swap_info_struct *si, + struct swap_cluster_info *ci, + unsigned long offset, + unsigned int order, unsigned char usage) { - unsigned long start = offset & ~(SWAPFILE_CLUSTER - 1); + unsigned int next = SWAP_ENTRY_INVALID, found = SWAP_ENTRY_INVALID; + unsigned long start = ALIGN_DOWN(offset, SWAPFILE_CLUSTER); unsigned long end = min(start + SWAPFILE_CLUSTER, si->max); unsigned int nr_pages = 1 << order; bool need_reclaim, ret; - struct swap_cluster_info *ci; - ci = &si->cluster_info[offset / SWAPFILE_CLUSTER]; lockdep_assert_held(&ci->lock); - if (end < nr_pages || ci->count + nr_pages > SWAPFILE_CLUSTER) { - offset = SWAP_NEXT_INVALID; + if (end < nr_pages || ci->count + nr_pages > SWAPFILE_CLUSTER) goto out; - } for (end -= nr_pages; offset <= end; offset += nr_pages) { need_reclaim = false; @@ -789,34 +789,27 @@ static unsigned int alloc_swap_scan_cluster(struct swap_info_struct *si, unsigne * cluster has no flag set, and change of list * won't cause fragmentation. */ - if (!cluster_is_usable(ci, order)) { - offset = SWAP_NEXT_INVALID; + if (!cluster_is_usable(ci, order)) goto out; - } if (cluster_is_free(ci)) offset = start; /* Reclaim failed but cluster is usable, try next */ if (!ret) continue; } - if (!cluster_alloc_range(si, ci, offset, usage, order)) { - offset = SWAP_NEXT_INVALID; - goto out; - } - *foundp = offset; - if (ci->count == SWAPFILE_CLUSTER) { - offset = SWAP_NEXT_INVALID; - goto out; - } + if (!cluster_alloc_range(si, ci, offset, usage, order)) + break; + found = offset; offset += nr_pages; + if (ci->count < SWAPFILE_CLUSTER && offset <= end) + next = offset; break; } - if (offset > end) - offset = SWAP_NEXT_INVALID; out: relocate_cluster(si, ci); unlock_cluster(ci); - return offset; + __this_cpu_write(si->percpu_cluster->next[order], next); + return found; } /* Return true if reclaimed a whole cluster */ @@ -885,8 +878,8 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si, int o if (cluster_is_usable(ci, order)) { if (cluster_is_free(ci)) offset = cluster_offset(si, ci); - offset = alloc_swap_scan_cluster(si, offset, &found, - order, usage); + found = alloc_swap_scan_cluster(si, ci, offset, + order, usage); } else { unlock_cluster(ci); } @@ -897,8 +890,8 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si, int o new_cluster: ci = cluster_isolate_lock(si, &si->free_clusters); if (ci) { - offset = alloc_swap_scan_cluster(si, cluster_offset(si, ci), - &found, order, usage); + found = alloc_swap_scan_cluster(si, ci, cluster_offset(si, ci), + order, usage); if (found) goto done; } @@ -911,8 +904,8 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si, int o unsigned int frags = 0, frags_existing; while ((ci = cluster_isolate_lock(si, &si->nonfull_clusters[order]))) { - offset = alloc_swap_scan_cluster(si, cluster_offset(si, ci), - &found, order, usage); + found = alloc_swap_scan_cluster(si, ci, cluster_offset(si, ci), + order, usage); /* * With `fragmenting` set to true, it will surely take * the cluster off nonfull list @@ -932,8 +925,8 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si, int o * per-CPU usage, but they could contain newly released * reclaimable (eg. lazy-freed swap cache) slots. */ - offset = alloc_swap_scan_cluster(si, cluster_offset(si, ci), - &found, order, usage); + found = alloc_swap_scan_cluster(si, ci, cluster_offset(si, ci), + order, usage); if (found) goto done; frags++; @@ -959,21 +952,20 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si, int o */ while ((ci = cluster_isolate_lock(si, &si->frag_clusters[o]))) { atomic_long_dec(&si->frag_cluster_nr[o]); - offset = alloc_swap_scan_cluster(si, cluster_offset(si, ci), - &found, order, usage); + found = alloc_swap_scan_cluster(si, ci, cluster_offset(si, ci), + 0, usage); if (found) goto done; } while ((ci = cluster_isolate_lock(si, &si->nonfull_clusters[o]))) { - offset = alloc_swap_scan_cluster(si, cluster_offset(si, ci), - &found, order, usage); + found = alloc_swap_scan_cluster(si, ci, cluster_offset(si, ci), + 0, usage); if (found) goto done; } } done: - __this_cpu_write(si->percpu_cluster->next[order], offset); local_unlock(&si->percpu_cluster->lock); return found; @@ -3190,7 +3182,7 @@ static struct swap_cluster_info *setup_clusters(struct swap_info_struct *si, cluster = per_cpu_ptr(si->percpu_cluster, cpu); for (i = 0; i < SWAP_NR_ORDERS; i++) - cluster->next[i] = SWAP_NEXT_INVALID; + cluster->next[i] = SWAP_ENTRY_INVALID; local_lock_init(&cluster->lock); }