From patchwork Tue Oct 22 19:24:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13846078 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0259FCDD0D7 for ; Tue, 22 Oct 2024 19:30:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 87B196B008C; Tue, 22 Oct 2024 15:30:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8291D6B0092; Tue, 22 Oct 2024 15:30:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6A2A76B0095; Tue, 22 Oct 2024 15:30:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 490BC6B008C for ; Tue, 22 Oct 2024 15:30:02 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 74818140464 for ; Tue, 22 Oct 2024 19:29:44 +0000 (UTC) X-FDA: 82702228098.21.B413511 Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) by imf07.hostedemail.com (Postfix) with ESMTP id 5BD3F40012 for ; Tue, 22 Oct 2024 19:29:37 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="c/RpbDCf"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf07.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1729625288; a=rsa-sha256; cv=none; b=CvQWkco5XXEA014ylqOaULP4Cfz01flRx7dDAgbhRK5eqUFWSXB4DfS4E86hIlMit9BofB jj/NrU0yg1g5cgSWD3LoO+Q1ag4Po+quCtgTgGsJAh6aFOhre6q35/HMfcMAituexqScM1 E3hjYo0hh6nivxKhK0duD351kpNHNLE= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="c/RpbDCf"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf07.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=ryncsn@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729625288; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=De6FNmrDX2CsofmiG4+hb8nOrMhL9lIlNsptB1OQpyI=; b=nyaP6Fx3lqZSyPxANgHsjaS3HssmYqnvh3kwRdqHfBchpFvFSlSGNmY3urH+i3ZOGqWoVK WaazyslGhd2pMhaVL69WlfQ5iVaH45wQkIlAR6iqxt2NOzj0Wck9AI6+HbjKS+VauK8fwH Uh86FBe/HX9gfYTuEbDPBez2os2rKgs= Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-20c6f492d2dso69674395ad.0 for ; Tue, 22 Oct 2024 12:29:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1729625398; x=1730230198; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=De6FNmrDX2CsofmiG4+hb8nOrMhL9lIlNsptB1OQpyI=; b=c/RpbDCf3vNwtuy1dfO5m4aswzitcQDmojS+uJnOaCHQT4yq44cJXN3gpr3TmQiX20 RJ+i6mxXpQDAIm+jS0hFGzVprDFvXg3CUQNvy0J+YuG31DMySHe/95Epg2SG7ZKiLuv0 EsjGM50rugT6aGib1OPpBVoNlEOOz7UOAEmRg0utC3882DD1d8R7mjx1WV72pdvjwye5 jyUwasKyqxKq2IgdfoRlmVI6tsm5kutsikCTueJvp64SYN8fB89F8SLxF+GleadvV8rp 5UVszudHgqV6h5+VKhHX7CD4GcnHxkqAXsnyA6srnNVgCqUxFadW4sQH7X/+bxs6F6JG Xehg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729625398; x=1730230198; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=De6FNmrDX2CsofmiG4+hb8nOrMhL9lIlNsptB1OQpyI=; b=KVi2QLaqbPf+pecNzQlHLzXjG2JzR3d52n1P81IsMPMdLFTnUiFxOFuMJk7krO6jLx nFVsRcvl0jQzz2kax2TdumVn2h9y+xjruyEk/yQ0YCao80uXAVyIIn9QKwD752ek9Srp EcIccifEZpYi28ffls/IeONeoTZfO2l0ZO807jIYg3aFtRcJIFwJoE8KxeZEZBp83Aw6 6Zur4d/ikhIaTDohpxfM+0b6+MCcVsWw8o54SkH5wgQFDsy+00Rmu5MU/mYIod0iAuqr 6LRX8j+EpASNxlUy4ldg7vo+ifWGYP6LrozXlR71Jm+XZUtDiE7Su170qFajX0ks9jZG x+vQ== X-Gm-Message-State: AOJu0YywszfwDcZW0H/WPobp8lIuqRI7kf4+tTQOBdUo8N2QNJkavtmE g/3Zy5bGbUmT6rnvIrAFJCRZMFzi5jTgBT2qF37g2Yo/7P7r1Xw4mH2ilkqJ7FQ= X-Google-Smtp-Source: AGHT+IG648J4NAU1H2vV7qvnmugjAT+R3eqfpZK2sgM9OZGK+vlJTzeCTjXDpzcoLaWx1v4aNcZg4Q== X-Received: by 2002:a17:903:18e:b0:20b:6918:30b5 with SMTP id d9443c01a7336-20fa9e9f77emr3186585ad.41.1729625398260; Tue, 22 Oct 2024 12:29:58 -0700 (PDT) Received: from KASONG-MC4.tencent.com ([106.37.123.36]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-20e7f0d9f05sm45895305ad.186.2024.10.22.12.29.54 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 22 Oct 2024 12:29:57 -0700 (PDT) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Chris Li , Barry Song , Ryan Roberts , Hugh Dickins , Yosry Ahmed , "Huang, Ying" , Tim Chen , Nhat Pham , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH 04/13] mm, swap: use cluster lock for HDD Date: Wed, 23 Oct 2024 03:24:42 +0800 Message-ID: <20241022192451.38138-5-ryncsn@gmail.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241022192451.38138-1-ryncsn@gmail.com> References: <20241022192451.38138-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspamd-Queue-Id: 5BD3F40012 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: mab5oo5p9cz39nmif4ccrntja6cxjo3t X-HE-Tag: 1729625377-655070 X-HE-Meta: U2FsdGVkX19pk5IpK0qj+Zd7nRX62IrPTdW8NhRhnBesPGrgI2pVEqNgCEjW8rHJJWgSJwcMr+uQmQR9IoawkuUjQqGCe7NdoT4oZYEmHOLWVcQIlTUImhLbgwpHjcBhUfkVwiD0g4GRwWibxU4cYizsb1bFJJ6AGfdA/Zd3Z2HV7Yw460Yp7K4sFZhvcM1deVV1/lJcLS4GDlA06Ay78/sTQ+Nx1AHCpREPprEvPSRtmCXL1uJNg7vw/HZdC4dG3hkOwiCVJ4DseirBBdUA8a/n7WI8CWkJN63p9kMAIk2xBWJfSVRdaFYwKVxPDmZyWlkjEKn40tNjXkKV1v3i3XFuqQLfmrB9uHT0TqYoPc/D7LPJLqOAJelDdhtIhrPXQHfDfgOU6X1dvfuRLbcFvW59a0k6Dt1Bea46V2Lol6mwRZ/DzzRZG08E26GkyOO9xtqCbaTSj/JN15W4vwkrhoeIFrJ1V+SjjBNYruS5eXtcFJvkYVw9lztOL4yDBimQMq9MAizdVH+bzkC7jCDvTwHZ7wZauqmKab3/xR1Z1TEBktEa4G5jNyrPyQnLipk30HR4Oi8bJVjxi1fLccObjqbuIdA7dBkV4iho8niLZSjTIjY/X5+HU/+rKgupvOUJudELmwBB5807sn9t0ZUsPSxoR9NEJIg3YTYBRG6N3bHXzqSsRT7moaoKep2lQXFyxK8pIToaHHdmZpKIndaQj3m6XGIrAl6ocdp+tHarr1m5Z1iGWwp+0hz233gqeqRGVj8VRfyy54MKZrPQ3/QcAr/aZ1yTdNpRSmk49PggHA2LiOPfA74cDXgocfP96YqRiF2k+PTtelof27Q8NDbDSlTkepn3rN5Le9ozCozAIf1dWBZAEJXd/wFSc25/bDjawFLITP61gGUxz7SlLLwA3zn+0DwC1XRvMuE7kEUebEkAOzmuzBHkwT8KgvvyPPHkbfqmZDRK5gJwxAlraTJ W7pQHfYD pTLUs7gPK07NDMRIsDpphszEZ8YjV9YFVWlCnKQWx1zSDnCBqnCqVxY5BmL0pSnxduxZlDASQ5eow479v3y3c1t0t6Ef7rGZCyAue/6hyIfFH9IBC8UH4BcuPOcODR7L2G7+QGpmSS/YxhNOeZ05qqQbQ6nSOt/vr6aIix0SJc9B4+bBVS7j8Prgh9s6Mc8U2TwC+CNK1LCkOvQE7Kr4YDqjTlsw5d9Py0jKxFgMVDx4N7FuVZEPXMZEItI75dmOX5Q8LpV4TIBz1lA7Xh3N0ZT/JCeWWeHBGGvJ7fMIAxNOH2St4Q0Ur9ZhUoOJt1XGszLaJoziQuqcIrgahfkO9sTsbWOC0mZc7GOAotr2r+Iv3B2zxzqASkhnFuV/WY9CX+IIZOyjj2ukhz29PucY03CoSnU2Nj1kEdr9bswYYfOQpsSODFuMo2T671sbPvJooSWJXhZHnZ0OzPxRdkQnhmNtYeKDWQCIuCVGjFE32pmE+9PkFelAGQOXUZBRSUVyJ2vQfKFN8Gqub3rdZnthATZhuJaCdXBsqfpUm7Fi7z4mNUEY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song Cluster lock (ci->lock) was introduce to reduce contention for certain operations. Using cluster lock for HDD is not helpful as HDD have a poor performance, so locking isn't the bottleneck. But having different set of locks for HDD / non-HDD prevents further rework of device lock (si->lock). This commit just changed all lock_cluster_or_swap_info to lock_cluster, which is a safe and straight conversion since cluster info is always allocated now, also removed all cluster_info related checks. Suggested-by: Chris Li Signed-off-by: Kairui Song --- mm/swapfile.c | 107 ++++++++++++++++---------------------------------- 1 file changed, 34 insertions(+), 73 deletions(-) diff --git a/mm/swapfile.c b/mm/swapfile.c index ffdf7eedecb5..f8e70bb5f1d7 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -58,10 +58,9 @@ static void swap_entry_range_free(struct swap_info_struct *si, swp_entry_t entry static void swap_range_alloc(struct swap_info_struct *si, unsigned long offset, unsigned int nr_entries); static bool folio_swapcache_freeable(struct folio *folio); -static struct swap_cluster_info *lock_cluster_or_swap_info( - struct swap_info_struct *si, unsigned long offset); -static void unlock_cluster_or_swap_info(struct swap_info_struct *si, - struct swap_cluster_info *ci); +static struct swap_cluster_info *lock_cluster(struct swap_info_struct *si, + unsigned long offset); +static void unlock_cluster(struct swap_cluster_info *ci); static DEFINE_SPINLOCK(swap_lock); static unsigned int nr_swapfiles; @@ -222,9 +221,9 @@ static int __try_to_reclaim_swap(struct swap_info_struct *si, * swap_map is HAS_CACHE only, which means the slots have no page table * reference or pending writeback, and can't be allocated to others. */ - ci = lock_cluster_or_swap_info(si, offset); + ci = lock_cluster(si, offset); need_reclaim = swap_is_has_cache(si, offset, nr_pages); - unlock_cluster_or_swap_info(si, ci); + unlock_cluster(ci); if (!need_reclaim) goto out_unlock; @@ -404,45 +403,15 @@ static inline struct swap_cluster_info *lock_cluster(struct swap_info_struct *si { struct swap_cluster_info *ci; - ci = si->cluster_info; - if (ci) { - ci += offset / SWAPFILE_CLUSTER; - spin_lock(&ci->lock); - } - return ci; -} - -static inline void unlock_cluster(struct swap_cluster_info *ci) -{ - if (ci) - spin_unlock(&ci->lock); -} - -/* - * Determine the locking method in use for this device. Return - * swap_cluster_info if SSD-style cluster-based locking is in place. - */ -static inline struct swap_cluster_info *lock_cluster_or_swap_info( - struct swap_info_struct *si, unsigned long offset) -{ - struct swap_cluster_info *ci; - - /* Try to use fine-grained SSD-style locking if available: */ - ci = lock_cluster(si, offset); - /* Otherwise, fall back to traditional, coarse locking: */ - if (!ci) - spin_lock(&si->lock); + ci = &si->cluster_info[offset / SWAPFILE_CLUSTER]; + spin_lock(&ci->lock); return ci; } -static inline void unlock_cluster_or_swap_info(struct swap_info_struct *si, - struct swap_cluster_info *ci) +static inline void unlock_cluster(struct swap_cluster_info *ci) { - if (ci) - unlock_cluster(ci); - else - spin_unlock(&si->lock); + spin_unlock(&ci->lock); } /* Add a cluster to discard list and schedule it to do discard */ @@ -558,9 +527,6 @@ static void inc_cluster_info_page(struct swap_info_struct *si, unsigned long idx = page_nr / SWAPFILE_CLUSTER; struct swap_cluster_info *ci; - if (!cluster_info) - return; - ci = cluster_info + idx; ci->count++; @@ -576,9 +542,6 @@ static void inc_cluster_info_page(struct swap_info_struct *si, static void dec_cluster_info_page(struct swap_info_struct *si, struct swap_cluster_info *ci, int nr_pages) { - if (!si->cluster_info) - return; - VM_BUG_ON(ci->count < nr_pages); VM_BUG_ON(cluster_is_free(ci)); lockdep_assert_held(&si->lock); @@ -995,8 +958,6 @@ static int cluster_alloc_swap(struct swap_info_struct *si, { int n_ret = 0; - VM_BUG_ON(!si->cluster_info); - while (n_ret < nr) { unsigned long offset = cluster_alloc_swap_entry(si, order, usage); @@ -1036,10 +997,10 @@ static int scan_swap_map_slots(struct swap_info_struct *si, } /* - * Swapfile is not block device or not using clusters so unable + * Swapfile is not block device so unable * to allocate large entries. */ - if (!(si->flags & SWP_BLKDEV) || !si->cluster_info) + if (!(si->flags & SWP_BLKDEV)) return 0; } @@ -1279,9 +1240,9 @@ static unsigned char __swap_entry_free(struct swap_info_struct *si, unsigned long offset = swp_offset(entry); unsigned char usage; - ci = lock_cluster_or_swap_info(si, offset); + ci = lock_cluster(si, offset); usage = __swap_entry_free_locked(si, offset, 1); - unlock_cluster_or_swap_info(si, ci); + unlock_cluster(ci); if (!usage) free_swap_slot(entry); @@ -1304,14 +1265,14 @@ static bool __swap_entries_free(struct swap_info_struct *si, if (nr > SWAPFILE_CLUSTER - offset % SWAPFILE_CLUSTER) goto fallback; - ci = lock_cluster_or_swap_info(si, offset); + ci = lock_cluster(si, offset); if (!swap_is_last_map(si, offset, nr, &has_cache)) { - unlock_cluster_or_swap_info(si, ci); + unlock_cluster(ci); goto fallback; } for (i = 0; i < nr; i++) WRITE_ONCE(si->swap_map[offset + i], SWAP_HAS_CACHE); - unlock_cluster_or_swap_info(si, ci); + unlock_cluster(ci); if (!has_cache) { for (i = 0; i < nr; i++) @@ -1367,7 +1328,7 @@ static void cluster_swap_free_nr(struct swap_info_struct *si, DECLARE_BITMAP(to_free, BITS_PER_LONG) = { 0 }; int i, nr; - ci = lock_cluster_or_swap_info(si, offset); + ci = lock_cluster(si, offset); while (nr_pages) { nr = min(BITS_PER_LONG, nr_pages); for (i = 0; i < nr; i++) { @@ -1375,18 +1336,18 @@ static void cluster_swap_free_nr(struct swap_info_struct *si, bitmap_set(to_free, i, 1); } if (!bitmap_empty(to_free, BITS_PER_LONG)) { - unlock_cluster_or_swap_info(si, ci); + unlock_cluster(ci); for_each_set_bit(i, to_free, BITS_PER_LONG) free_swap_slot(swp_entry(si->type, offset + i)); if (nr == nr_pages) return; bitmap_clear(to_free, 0, BITS_PER_LONG); - ci = lock_cluster_or_swap_info(si, offset); + ci = lock_cluster(si, offset); } offset += nr; nr_pages -= nr; } - unlock_cluster_or_swap_info(si, ci); + unlock_cluster(ci); } /* @@ -1425,9 +1386,9 @@ void put_swap_folio(struct folio *folio, swp_entry_t entry) if (!si) return; - ci = lock_cluster_or_swap_info(si, offset); + ci = lock_cluster(si, offset); if (size > 1 && swap_is_has_cache(si, offset, size)) { - unlock_cluster_or_swap_info(si, ci); + unlock_cluster(ci); spin_lock(&si->lock); swap_entry_range_free(si, entry, size); spin_unlock(&si->lock); @@ -1435,14 +1396,14 @@ void put_swap_folio(struct folio *folio, swp_entry_t entry) } for (int i = 0; i < size; i++, entry.val++) { if (!__swap_entry_free_locked(si, offset + i, SWAP_HAS_CACHE)) { - unlock_cluster_or_swap_info(si, ci); + unlock_cluster(ci); free_swap_slot(entry); if (i == size - 1) return; - lock_cluster_or_swap_info(si, offset); + lock_cluster(si, offset); } } - unlock_cluster_or_swap_info(si, ci); + unlock_cluster(ci); } static int swp_entry_cmp(const void *ent1, const void *ent2) @@ -1506,9 +1467,9 @@ int swap_swapcount(struct swap_info_struct *si, swp_entry_t entry) struct swap_cluster_info *ci; int count; - ci = lock_cluster_or_swap_info(si, offset); + ci = lock_cluster(si, offset); count = swap_count(si->swap_map[offset]); - unlock_cluster_or_swap_info(si, ci); + unlock_cluster(ci); return count; } @@ -1531,7 +1492,7 @@ int swp_swapcount(swp_entry_t entry) offset = swp_offset(entry); - ci = lock_cluster_or_swap_info(si, offset); + ci = lock_cluster(si, offset); count = swap_count(si->swap_map[offset]); if (!(count & COUNT_CONTINUED)) @@ -1554,7 +1515,7 @@ int swp_swapcount(swp_entry_t entry) n *= (SWAP_CONT_MAX + 1); } while (tmp_count & COUNT_CONTINUED); out: - unlock_cluster_or_swap_info(si, ci); + unlock_cluster(ci); return count; } @@ -1569,8 +1530,8 @@ static bool swap_page_trans_huge_swapped(struct swap_info_struct *si, int i; bool ret = false; - ci = lock_cluster_or_swap_info(si, offset); - if (!ci || nr_pages == 1) { + ci = lock_cluster(si, offset); + if (nr_pages == 1) { if (swap_count(map[roffset])) ret = true; goto unlock_out; @@ -1582,7 +1543,7 @@ static bool swap_page_trans_huge_swapped(struct swap_info_struct *si, } } unlock_out: - unlock_cluster_or_swap_info(si, ci); + unlock_cluster(ci); return ret; } @@ -3412,7 +3373,7 @@ static int __swap_duplicate(swp_entry_t entry, unsigned char usage, int nr) offset = swp_offset(entry); VM_WARN_ON(nr > SWAPFILE_CLUSTER - offset % SWAPFILE_CLUSTER); VM_WARN_ON(usage == 1 && nr > 1); - ci = lock_cluster_or_swap_info(si, offset); + ci = lock_cluster(si, offset); err = 0; for (i = 0; i < nr; i++) { @@ -3467,7 +3428,7 @@ static int __swap_duplicate(swp_entry_t entry, unsigned char usage, int nr) } unlock_out: - unlock_cluster_or_swap_info(si, ci); + unlock_cluster(ci); return err; }