From patchwork Mon Jan 13 17:57:25 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Kairui Song X-Patchwork-Id: 13937872 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B2C8AC02180 for ; Mon, 13 Jan 2025 18:00:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 459A96B0085; Mon, 13 Jan 2025 13:00:20 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 408F56B009E; Mon, 13 Jan 2025 13:00:20 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 283BB6B009F; Mon, 13 Jan 2025 13:00:20 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 028C16B0085 for ; Mon, 13 Jan 2025 13:00:19 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id B6EB712024D for ; Mon, 13 Jan 2025 18:00:19 +0000 (UTC) X-FDA: 83003192958.21.98DFD71 Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) by imf02.hostedemail.com (Postfix) with ESMTP id B576480023 for ; Mon, 13 Jan 2025 18:00:17 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=YBW3pmIx; spf=pass (imf02.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.179 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736791217; h=from:from:sender:reply-to:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=qtETt4hJiQh7efUCU6F69OscBxra6w0MDgbP2RHhprU=; b=qL08Wd2lDCEM8QdTRJ7Eo6q+34WWkk2eZpQeMar4ha91/dV9k50S8vETcmO4CdlOi0H0au f061nr4v9JY3houoqlFXxEbk+azJMbAep53Li8r6aEBPwlb50J8FwzRh04QnKhVjhZY2di uQMW1g1UhAA66xVrVuu6tteaqSt78Fg= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=YBW3pmIx; spf=pass (imf02.hostedemail.com: domain of ryncsn@gmail.com designates 209.85.214.179 as permitted sender) smtp.mailfrom=ryncsn@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736791217; a=rsa-sha256; cv=none; b=ayekbS/GIa63Bhw6cXf9jTOogHEUobTJnzt8Q8NfSEc2fVPXDYRMSEfl0o7BtSF/NWyM+R NvyJIO2vGN2p/sTc4hBya1MPGnMNduFVlJUr2OTIybd+BTaH2xkrPqyH7KwTVQ4MQpSJbo DgcRGk/E2KRcLiSf1SydJoSTYAi8G18= Received: by mail-pl1-f179.google.com with SMTP id d9443c01a7336-21644aca3a0so101078745ad.3 for ; Mon, 13 Jan 2025 10:00:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1736791216; x=1737396016; darn=kvack.org; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:from:to:cc:subject :date:message-id:reply-to; bh=qtETt4hJiQh7efUCU6F69OscBxra6w0MDgbP2RHhprU=; b=YBW3pmIxDt9XhbiuT7sJoj+h6bmz1KGOTGU4CH7E03Y0JzVghxPgJV071BpW3yNf5u LMnIH2NvgBwMBme0Hb68hDtlKllZ8a+1IguDiMoM2bwulUZ1cjC0H5IFBWSekCnolf1X tSRpsmpECE01M3klVok3H6gqCgTkNwhWgesZ9l4De8J1pa365Hap8gOemx8b9zr6tmyh B1JQXlxwlkUUoFAZIB009oVCwRPsyWm0nYtbr/xdr39Ef1hrMeuZ5GIw/4zYeRQIgJ7l 8PBl7WzqldxuGACQoE2yDppvQcKI8rsswiZW4icNIm28BY6UeD5x/6ouORg4VI+UgLMS XR4g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736791216; x=1737396016; h=content-transfer-encoding:mime-version:reply-to:references :in-reply-to:message-id:date:subject:cc:to:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=qtETt4hJiQh7efUCU6F69OscBxra6w0MDgbP2RHhprU=; b=sT28cxTH0R6rjVEU/HwYYY4ENOPZhivXyXxbYyqrH1XDThcD1jzrmoAKrlfcWYvm32 wMnA9WoXuYC6BGRTn3H6rkg9KBkzwx1gV5hOXMGcBT7fUrZAQXfipD/x0IUBJjy4SobF VSezVZtSYag8s0zWQKf5puu4XtF/ip0byZUWPj7Gx+Iy/FvulqpnzJN0DhXYYv6ZvGJx LekHPYJ7+OIpApx5DK9l+OPMwlhAaZtDdHNFHLAxH7ty5hGaZUfj5Qnjes+9MiEOv5Uz LgZYBUmxHUCpaE5Ah4SxLb2+ujnffF4HT9iW4sKf1S8aF0XzZJpvFfBZhrHt+NZdPTtg mPnw== X-Gm-Message-State: AOJu0YxY4s+FX5sO379KgakYH3TrUW4pCeMX8SsNo9jay+tvAaDNT0K/ zyTIIFjoe0IwXjKOY4qVpJhv4oxose/efVAC18EX4TiMbyXHSFwsDUtHwHFuzIY= X-Gm-Gg: ASbGncsyiPW1QdMZzzVW0dd68IlKP/VzAc+KsJZ6TClmXVO+VN8+8HMbCm63RibJI91 ijY7fSIDtMifLM3HlGaR84KjUhUedKhy/up9ktaEItmF8Ny3LImh32XilkORSpkUMrcsuUVEdqS 61jsNUiMNZBurXvmX5TT80/c1pDDKEy5eYcO8nG7IhoYEKcd3N4wSE3tLNaYI+q/vIXm20cSarO KiP6PIIVu2E7lx1Mq0LJa2c42yKOvJMoTQaNAunr2cXfXyMPFPtBpH6smvJwD/+oOZJdc3W/yzm bw== X-Google-Smtp-Source: AGHT+IHgJeS+srOGqdry/3K8xXT02L3VLVOokYD22RUqbvTi1GzphHH6M9xyOJ1unQcXSjNg3M09tw== X-Received: by 2002:a17:902:db0e:b0:215:a57e:88e7 with SMTP id d9443c01a7336-21a83f48cd6mr314107555ad.3.1736791213848; Mon, 13 Jan 2025 10:00:13 -0800 (PST) Received: from KASONG-MC4.tencent.com ([115.171.41.132]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-21a9f21aba7sm57023635ad.113.2025.01.13.10.00.10 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 13 Jan 2025 10:00:13 -0800 (PST) From: Kairui Song To: linux-mm@kvack.org Cc: Andrew Morton , Chris Li , Barry Song , Ryan Roberts , Hugh Dickins , Yosry Ahmed , "Huang, Ying" , Baoquan He , Nhat Pham , Johannes Weiner , Kalesh Singh , linux-kernel@vger.kernel.org, Kairui Song Subject: [PATCH v4 06/13] mm, swap: clean up plist removal and adding Date: Tue, 14 Jan 2025 01:57:25 +0800 Message-ID: <20250113175732.48099-7-ryncsn@gmail.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250113175732.48099-1-ryncsn@gmail.com> References: <20250113175732.48099-1-ryncsn@gmail.com> Reply-To: Kairui Song MIME-Version: 1.0 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: B576480023 X-Rspam-User: X-Stat-Signature: wadizi6s7jtwgt8mxtftzsgiwi7pj7y3 X-HE-Tag: 1736791217-66013 X-HE-Meta: U2FsdGVkX1+G8qielALEX2Ubm3xtnglths03/u67kBUjt58IpeiWhG6TAaEyDMpqwilNBKlZCyjgCuAfU4RxYa+IqDxa0GrCfD3C/nu5QMjL/vwcDo51GcbfqQhPfsyFowoxGETll70Gwn5Q6x2ICqWFoBwfZFAWiHv+GcJYfhKh0pk89wNLPWEog8dUE54831NEba5r1WKkmnSPmGyX6GWYAHwK8q9p0wIDHg2OHqFzw+9n+4zKGoxNxzxjCqh4h1YZFAOa+z17a36Chv0jfB3GXR0723cT5FflA4bQ07zKFDeBVEAwajMmEdY+7pe9sZ652PxuNXgzBN9t9GYvlY7TV/SMbAZRurAKSUTEVlsq6dZ96GTab9iUgwwX+lDGqaj3oHmg02BHwKwWKJnbK/EnKxoS/KGKfI3a8U7oKEqWD4clRkg6CZ9fw3pZRkquyCDIcGe1CUfp1lLGfIFLeDjDhXpNxeEeYD7+DMb1dO9Rn4SjOLr9u0soNnowbcC26zAn5Go3JH9RkPxadzxaQQ7Pbl1dXbKhosz35mTG5VAj/FKX6EySlA2BPr5qCxR+EpS2Kk5DjJnyx/NAhqvdbdwC6kCS3JV7JIctQ0N1rRmqtnMBowxgFLXWujh0ET+wF+/lN+6X1iR7TNqDZKaYJmpDGbLKqkLftlVjp0gCKVkRFjlIr/f41XPkckb48j7YRok9CWt4WsC3NHTM0fEiThC6I3c96yG/xVj+Po+3wvSFWmpnQ5eBVr+PYcvSdkfc5ahxD4YTgVw3e8V3tmThG9sqUCQq7dIHUk7BPtrtbYs/gXYYTQZaF2MIc6UlcWlvqt1G9sl+TeBBM80qlLzKdnyl0S1x/2x2v3jaS08mad1AGVFojeSRkc/5cBysp8FisZQdRui1tq//GJBKnhhKJjoZzfrAqMVHYSwypWhrq/Dt96qEONuOIef7yXYRjzn8skHGbJKy9SGUoWRfpcq svtQ8A6m RD2HlY9I2gVbhjsZyN4mtFfVkDTRQVkJlld5UEGdM0yMZAlmX3uIeyncAqxKp8IsVcA0+4DDA79zMLX5dmT4+4KhVvE1M7DOwIjLsm8XbFE3qUmeEhkE1BQZkLg/damrfhSrfBNVizQPFpt6LwGLjb0wEqEV8+NCRbKcl6YHTaYyR9VIlZoGdyNveqPyA8AFUpbA5V3jc36/oNcD3cMMup/0BlLznf5gG0kgrKQ5+cLGCpdeNDCmysly9uEGV8NjetqtQC4UvODStTC9RCQwHs72vwmzhwQ9KXQLygz88i7mHL/PtEKtI2nA2KRWmfzYgV10tcdwgiwK7NIctBHQKEjXU3WlMgZD5WqCPMeDyWPa+yR8j+0hodi79o6SnVj5kCJEDBjLcdwdXzR+Gf9wfF7/6W+oe5aARprBHQnmw4feQBNTEfS4lzNhagDXPdkpYBnX8cg35UOQH++LDA9gFG+qYb29bjKS7xW36xD9VG5kMJu6RBOLN8ydWPhB9ad/ZmIrP5b8JFLvZPKDGEZ401WERkA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Kairui Song When the swap device is full (inuse_pages == pages), it should be removed from the allocation available plist. If any slot is freed, the swap device should be added back to the plist. Additionally, during swapon or swapoff, the swap device is forcefully added or removed. Currently, the condition (inuse_pages == pages) is checked after every counter update, then remove or add the device accordingly. This is serialized by si->lock. This commit decouples it from the protection of si->lock and reworked plist removal and adding, making it possible to get rid of the hard dependency on si->lock in allocation path in later commits. To achieve this, simply using another lock is not an optimal approach, as the overhead is observable for a hot counter, and may cause complex locking issues. Thus, this commit manages to make it a lock-free atomic operation, by embedding the plist state into the second highest bit of the atomic counter. Simply making the counter an atomic will not work, if the update and plist status check are not performed atomically, we may miss an addition or removal. With the embedded info we can update the counter and check the plist status with single atomic operations, and avoid any extra overheads: If the counter is full (inuse_pages == pages) and the off-list bit is unset, we attempt to remove it from the plist. If the counter is not full (inuse_pages != pages) and the off-list bit is set, we attempt to add it to the plist. Removing, adding and bit update is serialized with a lock, which is a cold path. Ordinary counter updates will be lock-free. Signed-off-by: Kairui Song --- include/linux/swap.h | 2 +- mm/swapfile.c | 186 +++++++++++++++++++++++++++++++------------ 2 files changed, 138 insertions(+), 50 deletions(-) diff --git a/include/linux/swap.h b/include/linux/swap.h index 0c222017b5c6..e1eeea6307cd 100644 --- a/include/linux/swap.h +++ b/include/linux/swap.h @@ -307,7 +307,7 @@ struct swap_info_struct { /* list of cluster that are fragmented or contented */ unsigned int frag_cluster_nr[SWAP_NR_ORDERS]; unsigned int pages; /* total of usable pages of swap */ - unsigned int inuse_pages; /* number of those currently in use */ + atomic_long_t inuse_pages; /* number of those currently in use */ struct percpu_cluster __percpu *percpu_cluster; /* per cpu's swap location */ struct rb_root swap_extent_root;/* root of the swap extent rbtree */ struct block_device *bdev; /* swap device or bdev of swap file */ diff --git a/mm/swapfile.c b/mm/swapfile.c index 2686032d3510..91faf2073006 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -128,6 +128,26 @@ static inline unsigned char swap_count(unsigned char ent) return ent & ~SWAP_HAS_CACHE; /* may include COUNT_CONTINUED flag */ } +/* + * Use the second highest bit of inuse_pages counter as the indicator + * if one swap device is on the available plist, so the atomic can + * still be updated arithmetically while having special data embedded. + * + * inuse_pages counter is the only thing indicating if a device should + * be on avail_lists or not (except swapon / swapoff). By embedding the + * off-list bit in the atomic counter, updates no longer need any lock + * to check the list status. + * + * This bit will be set if the device is not on the plist and not + * usable, will be cleared if the device is on the plist. + */ +#define SWAP_USAGE_OFFLIST_BIT (1UL << (BITS_PER_TYPE(atomic_t) - 2)) +#define SWAP_USAGE_COUNTER_MASK (~SWAP_USAGE_OFFLIST_BIT) +static long swap_usage_in_pages(struct swap_info_struct *si) +{ + return atomic_long_read(&si->inuse_pages) & SWAP_USAGE_COUNTER_MASK; +} + /* Reclaim the swap entry anyway if possible */ #define TTRS_ANYWAY 0x1 /* @@ -717,7 +737,7 @@ static void swap_reclaim_full_clusters(struct swap_info_struct *si, bool force) int nr_reclaim; if (force) - to_scan = si->inuse_pages / SWAPFILE_CLUSTER; + to_scan = swap_usage_in_pages(si) / SWAPFILE_CLUSTER; while (!list_empty(&si->full_clusters)) { ci = list_first_entry(&si->full_clusters, struct swap_cluster_info, list); @@ -872,42 +892,128 @@ static unsigned long cluster_alloc_swap_entry(struct swap_info_struct *si, int o return found; } -static void __del_from_avail_list(struct swap_info_struct *si) +/* SWAP_USAGE_OFFLIST_BIT can only be set by this helper. */ +static void del_from_avail_list(struct swap_info_struct *si, bool swapoff) { int nid; + unsigned long pages; + + spin_lock(&swap_avail_lock); + + if (swapoff) { + /* + * Forcefully remove it. Clear the SWP_WRITEOK flags for + * swapoff here so it's synchronized by both si->lock and + * swap_avail_lock, to ensure the result can be seen by + * add_to_avail_list. + */ + lockdep_assert_held(&si->lock); + si->flags &= ~SWP_WRITEOK; + atomic_long_or(SWAP_USAGE_OFFLIST_BIT, &si->inuse_pages); + } else { + /* + * If not called by swapoff, take it off-list only if it's + * full and SWAP_USAGE_OFFLIST_BIT is not set (strictly + * si->inuse_pages == pages), any concurrent slot freeing, + * or device already removed from plist by someone else + * will make this return false. + */ + pages = si->pages; + if (!atomic_long_try_cmpxchg(&si->inuse_pages, &pages, + pages | SWAP_USAGE_OFFLIST_BIT)) + goto skip; + } - assert_spin_locked(&si->lock); for_each_node(nid) plist_del(&si->avail_lists[nid], &swap_avail_heads[nid]); + +skip: + spin_unlock(&swap_avail_lock); } -static void del_from_avail_list(struct swap_info_struct *si) +/* SWAP_USAGE_OFFLIST_BIT can only be cleared by this helper. */ +static void add_to_avail_list(struct swap_info_struct *si, bool swapon) { + int nid; + long val; + unsigned long pages; + spin_lock(&swap_avail_lock); - __del_from_avail_list(si); + + /* Corresponding to SWP_WRITEOK clearing in del_from_avail_list */ + if (swapon) { + lockdep_assert_held(&si->lock); + si->flags |= SWP_WRITEOK; + } else { + if (!(READ_ONCE(si->flags) & SWP_WRITEOK)) + goto skip; + } + + if (!(atomic_long_read(&si->inuse_pages) & SWAP_USAGE_OFFLIST_BIT)) + goto skip; + + val = atomic_long_fetch_and_relaxed(~SWAP_USAGE_OFFLIST_BIT, &si->inuse_pages); + + /* + * When device is full and device is on the plist, only one updater will + * see (inuse_pages == si->pages) and will call del_from_avail_list. If + * that updater happen to be here, just skip adding. + */ + pages = si->pages; + if (val == pages) { + /* Just like the cmpxchg in del_from_avail_list */ + if (atomic_long_try_cmpxchg(&si->inuse_pages, &pages, + pages | SWAP_USAGE_OFFLIST_BIT)) + goto skip; + } + + for_each_node(nid) + plist_add(&si->avail_lists[nid], &swap_avail_heads[nid]); + +skip: spin_unlock(&swap_avail_lock); } -static void swap_range_alloc(struct swap_info_struct *si, - unsigned int nr_entries) +/* + * swap_usage_add / swap_usage_sub of each slot are serialized by ci->lock + * within each cluster, so the total contribution to the global counter should + * always be positive and cannot exceed the total number of usable slots. + */ +static bool swap_usage_add(struct swap_info_struct *si, unsigned int nr_entries) { - WRITE_ONCE(si->inuse_pages, si->inuse_pages + nr_entries); - if (si->inuse_pages == si->pages) { - del_from_avail_list(si); + long val = atomic_long_add_return_relaxed(nr_entries, &si->inuse_pages); - if (vm_swap_full()) - schedule_work(&si->reclaim_work); + /* + * If device is full, and SWAP_USAGE_OFFLIST_BIT is not set, + * remove it from the plist. + */ + if (unlikely(val == si->pages)) { + del_from_avail_list(si, false); + return true; } + + return false; } -static void add_to_avail_list(struct swap_info_struct *si) +static void swap_usage_sub(struct swap_info_struct *si, unsigned int nr_entries) { - int nid; + long val = atomic_long_sub_return_relaxed(nr_entries, &si->inuse_pages); - spin_lock(&swap_avail_lock); - for_each_node(nid) - plist_add(&si->avail_lists[nid], &swap_avail_heads[nid]); - spin_unlock(&swap_avail_lock); + /* + * If device is not full, and SWAP_USAGE_OFFLIST_BIT is set, + * remove it from the plist. + */ + if (unlikely(val & SWAP_USAGE_OFFLIST_BIT)) + add_to_avail_list(si, false); +} + +static void swap_range_alloc(struct swap_info_struct *si, + unsigned int nr_entries) +{ + if (swap_usage_add(si, nr_entries)) { + if (vm_swap_full()) + schedule_work(&si->reclaim_work); + } } static void swap_range_free(struct swap_info_struct *si, unsigned long offset, @@ -925,8 +1031,6 @@ static void swap_range_free(struct swap_info_struct *si, unsigned long offset, for (i = 0; i < nr_entries; i++) clear_bit(offset + i, si->zeromap); - if (si->inuse_pages == si->pages) - add_to_avail_list(si); if (si->flags & SWP_BLKDEV) swap_slot_free_notify = si->bdev->bd_disk->fops->swap_slot_free_notify; @@ -946,7 +1050,7 @@ static void swap_range_free(struct swap_info_struct *si, unsigned long offset, */ smp_wmb(); atomic_long_add(nr_entries, &nr_swap_pages); - WRITE_ONCE(si->inuse_pages, si->inuse_pages - nr_entries); + swap_usage_sub(si, nr_entries); } static int cluster_alloc_swap(struct swap_info_struct *si, @@ -1036,19 +1140,6 @@ int get_swap_pages(int n_goal, swp_entry_t swp_entries[], int entry_order) plist_requeue(&si->avail_lists[node], &swap_avail_heads[node]); spin_unlock(&swap_avail_lock); spin_lock(&si->lock); - if ((si->inuse_pages == si->pages) || !(si->flags & SWP_WRITEOK)) { - spin_lock(&swap_avail_lock); - if (plist_node_empty(&si->avail_lists[node])) { - spin_unlock(&si->lock); - goto nextsi; - } - WARN(!(si->flags & SWP_WRITEOK), - "swap_info %d in list but !SWP_WRITEOK\n", - si->type); - __del_from_avail_list(si); - spin_unlock(&si->lock); - goto nextsi; - } n_ret = scan_swap_map_slots(si, SWAP_HAS_CACHE, n_goal, swp_entries, order); spin_unlock(&si->lock); @@ -1057,7 +1148,6 @@ int get_swap_pages(int n_goal, swp_entry_t swp_entries[], int entry_order) cond_resched(); spin_lock(&swap_avail_lock); -nextsi: /* * if we got here, it's likely that si was almost full before, * and since scan_swap_map_slots() can drop the si->lock, @@ -1789,7 +1879,7 @@ unsigned int count_swap_pages(int type, int free) if (sis->flags & SWP_WRITEOK) { n = sis->pages; if (free) - n -= sis->inuse_pages; + n -= swap_usage_in_pages(sis); } spin_unlock(&sis->lock); } @@ -2124,7 +2214,7 @@ static int try_to_unuse(unsigned int type) swp_entry_t entry; unsigned int i; - if (!READ_ONCE(si->inuse_pages)) + if (!swap_usage_in_pages(si)) goto success; retry: @@ -2137,7 +2227,7 @@ static int try_to_unuse(unsigned int type) spin_lock(&mmlist_lock); p = &init_mm.mmlist; - while (READ_ONCE(si->inuse_pages) && + while (swap_usage_in_pages(si) && !signal_pending(current) && (p = p->next) != &init_mm.mmlist) { @@ -2165,7 +2255,7 @@ static int try_to_unuse(unsigned int type) mmput(prev_mm); i = 0; - while (READ_ONCE(si->inuse_pages) && + while (swap_usage_in_pages(si) && !signal_pending(current) && (i = find_next_to_unuse(si, i)) != 0) { @@ -2200,7 +2290,7 @@ static int try_to_unuse(unsigned int type) * folio_alloc_swap(), temporarily hiding that swap. It's easy * and robust (though cpu-intensive) just to keep retrying. */ - if (READ_ONCE(si->inuse_pages)) { + if (swap_usage_in_pages(si)) { if (!signal_pending(current)) goto retry; return -EINTR; @@ -2227,7 +2317,7 @@ static void drain_mmlist(void) unsigned int type; for (type = 0; type < nr_swapfiles; type++) - if (swap_info[type]->inuse_pages) + if (swap_usage_in_pages(swap_info[type])) return; spin_lock(&mmlist_lock); list_for_each_safe(p, next, &init_mm.mmlist) @@ -2406,7 +2496,6 @@ static void setup_swap_info(struct swap_info_struct *si, int prio, static void _enable_swap_info(struct swap_info_struct *si) { - si->flags |= SWP_WRITEOK; atomic_long_add(si->pages, &nr_swap_pages); total_swap_pages += si->pages; @@ -2423,9 +2512,8 @@ static void _enable_swap_info(struct swap_info_struct *si) */ plist_add(&si->list, &swap_active_head); - /* add to available list if swap device is not full */ - if (si->inuse_pages < si->pages) - add_to_avail_list(si); + /* Add back to available list */ + add_to_avail_list(si, true); } static void enable_swap_info(struct swap_info_struct *si, int prio, @@ -2523,7 +2611,7 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile) goto out_dput; } spin_lock(&p->lock); - del_from_avail_list(p); + del_from_avail_list(p, true); if (p->prio < 0) { struct swap_info_struct *si = p; int nid; @@ -2541,7 +2629,6 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile) plist_del(&p->list, &swap_active_head); atomic_long_sub(p->pages, &nr_swap_pages); total_swap_pages -= p->pages; - p->flags &= ~SWP_WRITEOK; spin_unlock(&p->lock); spin_unlock(&swap_lock); @@ -2721,7 +2808,7 @@ static int swap_show(struct seq_file *swap, void *v) } bytes = K(si->pages); - inuse = K(READ_ONCE(si->inuse_pages)); + inuse = K(swap_usage_in_pages(si)); file = si->swap_file; len = seq_file_path(swap, file, " \t\n\\"); @@ -2838,6 +2925,7 @@ static struct swap_info_struct *alloc_swap_info(void) } spin_lock_init(&p->lock); spin_lock_init(&p->cont_lock); + atomic_long_set(&p->inuse_pages, SWAP_USAGE_OFFLIST_BIT); init_completion(&p->comp); return p; @@ -3335,7 +3423,7 @@ void si_swapinfo(struct sysinfo *val) struct swap_info_struct *si = swap_info[type]; if ((si->flags & SWP_USED) && !(si->flags & SWP_WRITEOK)) - nr_to_be_unused += READ_ONCE(si->inuse_pages); + nr_to_be_unused += swap_usage_in_pages(si); } val->freeswap = atomic_long_read(&nr_swap_pages) + nr_to_be_unused; val->totalswap = total_swap_pages + nr_to_be_unused;