From patchwork Tue Jun 18 23:26:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ryan Roberts X-Patchwork-Id: 13703159 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E7965C27C4F for ; Tue, 18 Jun 2024 23:27:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 029B76B03D8; Tue, 18 Jun 2024 19:27:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EF7DF6B03DA; Tue, 18 Jun 2024 19:27:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D49506B03DB; Tue, 18 Jun 2024 19:27:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id B87656B03D8 for ; Tue, 18 Jun 2024 19:27:06 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 32E2E1C1AD8 for ; Tue, 18 Jun 2024 23:27:06 +0000 (UTC) X-FDA: 82245597252.19.8F607FA Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf30.hostedemail.com (Postfix) with ESMTP id 8346580002 for ; Tue, 18 Jun 2024 23:27:04 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=none; spf=pass (imf30.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1718753220; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=V/EvMUGX8lBU3QjX45BQlJi6IpIDFkRP1Em73pvzTkE=; b=gETIt1TJjHJ/bfdWiRxJ6hij18DNLmpm+UfycJeFhOfGuOPtItBpKIwqvHM6eszfplZOt+ qF61EIIlo7IdXc8AZqUAWtw0TUw+WDFUX/wfckmyympylf8W1BpSFJk9YKrPuMe4QJF3qO bkFZ/Y+DwXWtCKGXV+qYOAVV+p1aGDU= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=none; spf=pass (imf30.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1718753220; a=rsa-sha256; cv=none; b=FSyzBvVlUN10ZkNhQ2CUDeBZ2OByCyDoKvKHM6RR2WISH6QjLwPh4QnDZzDyFVpzK3RRqH OOs+ZUOVC1vFoFrO3A3Csn2J2W1It498pr6rRqoBgKeUIJ1yb/g37rZctDMzZ37FtwfkO1 0/Wd3R4pd7hALMHhOaf0C3IH+TRgcjk= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 5D21A1477; Tue, 18 Jun 2024 16:27:28 -0700 (PDT) Received: from e125769.cambridge.arm.com (e125769.cambridge.arm.com [10.1.196.27]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id ED9C93F64C; Tue, 18 Jun 2024 16:27:01 -0700 (PDT) From: Ryan Roberts To: Andrew Morton , Chris Li , Kairui Song , "Huang, Ying" , Kalesh Singh , Barry Song , Hugh Dickins , David Hildenbrand Cc: Ryan Roberts , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 3/5] mm: swap: Track allocation order for clusters Date: Wed, 19 Jun 2024 00:26:43 +0100 Message-ID: <20240618232648.4090299-4-ryan.roberts@arm.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240618232648.4090299-1-ryan.roberts@arm.com> References: <20240618232648.4090299-1-ryan.roberts@arm.com> MIME-Version: 1.0 X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 8346580002 X-Stat-Signature: 7rstpoyjzhswrfrqsuns96e59j978g1d X-Rspam-User: X-HE-Tag: 1718753224-624551 X-HE-Meta: U2FsdGVkX19j8VciIBrQmwXXKC3WWH6sLQhmRbnhnO2rw0Lo3PVkMbuL9mYnGe6kDS2CaWUoclF+NTc9pz4pD7PvtSn8VL5GnARr3EeLXB9cFOhXHpwyaROG8b+MdRa53yr501L8b1mhYfd+mSmR53ESKKcDnSEh3iwzBLguqK3lZPe++HLrqaYlxIY/SIZesc9eYfXjEyksv2bhghqKEbDeR3JEJaHyDTqqrYjrU3rdHYiKzDKef3IpiEUgMgQZXNSQcLSE6MY7izQj10WviYr7rwBq1bfnktUrLNi/VNbGhK2/worq/6NT5PyZn2/rU9t4M3hR8HgoDRvm5vHTV+TLV1M7/qzkvZP3R9nPBsCaPk+Qf7MHCjN6vi4TYl1tQDwmJI/t5B9K39GA8lKCQzty0eRDZ0JEs2BMBDAtT06ihi+fNS6tTwFVqkQVkSh2D7mohgxzWLinfNWOcdmd4pTMiEnK2Ho14xCH4oqhEaMxzzIixXTz7EgMfebfO/9whjZe2zCY6Iqz3LWbuOs842dfk7epcmSiIV0Wl0XqjpjByfll1AzD9UsXaz/d+wajmoUbG9EHWfXt0bBKjzS1uj0EBEoj8bV1+paTXIFfa96YZ/JXADzTy36xfnfvpEj3gFBjnQ7oW/XvYUK4Xmf3a/FqXASI5NHOPFatVX159C5q+7NI8gByYHSQNK4RkMK9H6x8Yrpf8QDWXWOenFIcm/ujt9bDXWpFIra2E8ZNXdYfDm60IHAgSIBKvia+pH6CHdXfBb+f0i5RZN3O1boc7jIKUMrv+jixVy14AOqOicIrPlz9OmYpM7qIXhMJ0NMC7i8f+ptm3/snzU7pU+Y2MPIhBoST+elWtjNQmLrFeaPl/JMF1sK+zfaK3ebnS9UrQk1bZvXkZvhyArp95q8r1Y8rAWsqv4wrJohS7SwNJxS8j8saF0UuQ6xCq6y9yHSD0/75tC+np/Ai+zQp0Pr bxNamC0Y aTnQ7rc26TvoeWSMfQE5EGE/2+L6PWMuA9Kx4T6AsdSLkwXWtqeWmVqtZIwlt5OKOndYo7ANg/8TAv4wpVLntOBwscx2wqdAjsioXyiVgOoM0sDuTXSUuaQf8eG3mv+X8qgpBrM4GqZwJFAkqZfDPO34XXw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add an `order` field to `struct swap_cluster_info`, which applies to allocated clusters (i.e. those not on the free list) and tracks the swap entry order that the cluster should be used to allocate. A future commit will use this information to scan partially filled clusters to find appropriate free swap entries for allocation. Note that it is still possible that order-0 swap entries will be allocated in clusters that indicate a higher order due to the order-0 scanning mechanism. The maximum order we ever expect to see is 13 - PMD-size on arm64 with 64K base pages. 13 fits into 4 bits, so let's steal 4 unused flags bits for this purpose to avoid making `struct swap_cluster_info` any bigger. Signed-off-by: Ryan Roberts --- include/linux/swap.h | 3 ++- mm/swapfile.c | 24 +++++++++++++++--------- 2 files changed, 17 insertions(+), 10 deletions(-) -- 2.43.0 diff --git a/include/linux/swap.h b/include/linux/swap.h index 66566251ba31..2a40fe02d281 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -255,7 +255,8 @@ struct swap_cluster_info { * cluster */ unsigned int data:24; - unsigned int flags:8; + unsigned int flags:4; + unsigned int order:4; }; #define CLUSTER_FLAG_FREE 1 /* This cluster is free */ #define CLUSTER_FLAG_NEXT_NULL 2 /* This cluster has no next cluster */ diff --git a/mm/swapfile.c b/mm/swapfile.c index 30e79739dfdc..7b13f02a7ac2 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -307,11 +307,13 @@ static inline void cluster_set_count(struct swap_cluster_info *info, info->data = c; } -static inline void cluster_set_count_flag(struct swap_cluster_info *info, - unsigned int c, unsigned int f) +static inline void cluster_set_count_flag_order(struct swap_cluster_info *info, + unsigned int c, unsigned int f, + unsigned int o) { info->flags = f; info->data = c; + info->order = o; } static inline unsigned int cluster_next(struct swap_cluster_info *info) @@ -330,6 +332,7 @@ static inline void cluster_set_next_flag(struct swap_cluster_info *info, { info->flags = f; info->data = n; + info->order = 0; } static inline bool cluster_is_free(struct swap_cluster_info *info) @@ -346,6 +349,7 @@ static inline void cluster_set_null(struct swap_cluster_info *info) { info->flags = CLUSTER_FLAG_NEXT_NULL; info->data = 0; + info->order = 0; } static inline struct swap_cluster_info *lock_cluster(struct swap_info_struct *si, @@ -521,13 +525,14 @@ static void swap_users_ref_free(struct percpu_ref *ref) complete(&si->comp); } -static void alloc_cluster(struct swap_info_struct *si, unsigned long idx) +static void alloc_cluster(struct swap_info_struct *si, unsigned long idx, + int order) { struct swap_cluster_info *ci = si->cluster_info; VM_BUG_ON(cluster_list_first(&si->free_clusters) != idx); cluster_list_del_first(&si->free_clusters, ci); - cluster_set_count_flag(ci + idx, 0, 0); + cluster_set_count_flag_order(ci + idx, 0, 0, order); } static void free_cluster(struct swap_info_struct *si, unsigned long idx) @@ -556,14 +561,15 @@ static void free_cluster(struct swap_info_struct *si, unsigned long idx) */ static void add_cluster_info_page(struct swap_info_struct *p, struct swap_cluster_info *cluster_info, unsigned long page_nr, - unsigned long count) + int order) { unsigned long idx = page_nr / SWAPFILE_CLUSTER; + unsigned long count = 1 << order; if (!cluster_info) return; if (cluster_is_free(&cluster_info[idx])) - alloc_cluster(p, idx); + alloc_cluster(p, idx, order); VM_BUG_ON(cluster_count(&cluster_info[idx]) + count > SWAPFILE_CLUSTER); cluster_set_count(&cluster_info[idx], @@ -577,7 +583,7 @@ static void add_cluster_info_page(struct swap_info_struct *p, static void inc_cluster_info_page(struct swap_info_struct *p, struct swap_cluster_info *cluster_info, unsigned long page_nr) { - add_cluster_info_page(p, cluster_info, page_nr, 1); + add_cluster_info_page(p, cluster_info, page_nr, 0); } /* @@ -964,7 +970,7 @@ static int scan_swap_map_slots(struct swap_info_struct *si, goto done; } memset(si->swap_map + offset, usage, nr_pages); - add_cluster_info_page(si, si->cluster_info, offset, nr_pages); + add_cluster_info_page(si, si->cluster_info, offset, order); unlock_cluster(ci); swap_range_alloc(si, offset, nr_pages); @@ -1060,7 +1066,7 @@ static void swap_free_cluster(struct swap_info_struct *si, unsigned long idx) ci = lock_cluster(si, offset); memset(si->swap_map + offset, 0, SWAPFILE_CLUSTER); - cluster_set_count_flag(ci, 0, 0); + cluster_set_count_flag_order(ci, 0, 0, 0); free_cluster(si, idx); unlock_cluster(ci); swap_range_free(si, offset, SWAPFILE_CLUSTER);