From patchwork Tue Jan 30 01:36:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Johannes Weiner X-Patchwork-Id: 13536685 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B79C3C47DA9 for ; Tue, 30 Jan 2024 01:43:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F004E6B00BE; Mon, 29 Jan 2024 20:42:39 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E894E6B00BF; Mon, 29 Jan 2024 20:42:39 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C8E786B00C0; Mon, 29 Jan 2024 20:42:39 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id B3F5E6B00BE for ; Mon, 29 Jan 2024 20:42:39 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 8A99BA1D03 for ; Tue, 30 Jan 2024 01:42:39 +0000 (UTC) X-FDA: 81734278038.08.FA9B1EA Received: from mail-qv1-f53.google.com (mail-qv1-f53.google.com [209.85.219.53]) by imf28.hostedemail.com (Postfix) with ESMTP id C6CCEC000B for ; Tue, 30 Jan 2024 01:42:37 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=3HavyYuJ; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf28.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.53 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706578957; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nthkOb34nC29m9Z7rdKEYXbNdpH1xuatn2MKs2M9M7s=; b=FOhj2RVOfJuGIclEYYTbOcf7pxeFDjsJ5RaZ/Ow0XbjBq/YtIU74u5WMeA6ali4Glpq2ey 41+3oWLwcTRLVioID+9WJiUidadUx+S/Rtp+i89MoiXHh0TDt1mluVGE3IlC/0179sYKEC AnR9DmwdJQVCY+LbbcjmYcHP/hM7AjU= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=cmpxchg-org.20230601.gappssmtp.com header.s=20230601 header.b=3HavyYuJ; dmarc=pass (policy=none) header.from=cmpxchg.org; spf=pass (imf28.hostedemail.com: domain of hannes@cmpxchg.org designates 209.85.219.53 as permitted sender) smtp.mailfrom=hannes@cmpxchg.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706578957; a=rsa-sha256; cv=none; b=D5nIsh5+dp226sSkAh6pVs9IG4fJ0K2CPOeWMZ9nn5xML3aw4lJSZ5VuHg2DIPvbNmNUIM v6rrSlbRZLSq/2Z4jW7orawq6zQ3nglZueFFMxKMaCHETmqh7SLBLP3xgSrmzj6XzhAb3V mc7tShKwjnyFNrgxN/tWEbSV+JGkxmk= Received: by mail-qv1-f53.google.com with SMTP id 6a1803df08f44-68c4fb9e7e0so11623896d6.2 for ; Mon, 29 Jan 2024 17:42:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20230601.gappssmtp.com; s=20230601; t=1706578957; x=1707183757; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=nthkOb34nC29m9Z7rdKEYXbNdpH1xuatn2MKs2M9M7s=; b=3HavyYuJvTUY8zmJzN73Wjnc9MhGh+bCEu2BsnckYot5oq5uCYlFU8haPfzSy1N50n lFQRqbCuk1VG5xPZ2sq2zA4kGXex6yfxrlWD95S3PvJmCCFnXZTmuG3KArqoefC56bIA B/V1RcjQFZC1YmrgI619TRhUgWp7W/17RqW7XtOpQVWtQH4L5/gTijnkZp26hSVvBf3f WZMIJfAgBfiI7yKoWFlkZ52BiXagqumEpWB+I82PtM3Tp5Ka89AJIs+7YVB9hEu0ivyK cK+Ekx3vBwaIDAc30/KahvV1g3EslRC0omqLx2QgVR+/P8W9FmEC/ZzLytuXYS0By6v7 ZHsQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706578957; x=1707183757; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=nthkOb34nC29m9Z7rdKEYXbNdpH1xuatn2MKs2M9M7s=; b=wSXQ0tGT3t7WCwpMZEvx5AQ8faSSWkcPz9nKs8KWnHgG+lARVNUmDerBchRsJ4MJM6 +7ZUq1DFSRbgCvK25Ob2Fv8Jw2+FFevGmSE9SR25sgPaoutkVcCDz8x5oaWmbP7KIjFx gy/7eifgVMEMIF6BjsRvrs8XsXrivQQnYLDVRerwXMIYoNzH8B4AW9lFeCjgork3YQTH eA1RkRTiPQS3iRFV9eMyTo+0Lgujz2SuDPnT5atjPjv0nJ2XCP0SsGLgI/3ga1c22TPj 1LPkhVmxK+lEITmcYk0d/eHsheWUzZBVndm3GBtwapqymC4VBYJcat9+FeyiBKrmdpWB SVEg== X-Gm-Message-State: AOJu0YxpqxQ5l/IDixeEEWanbYZLZSdhSRFHkcAg8dClMtCbrVj7Sj8U ic95HyNb0/L46j01PZ4vjmiVMUBvop8vQD/E1y6lTwGV6tYPUtnmmb62o1zvs8Q= X-Google-Smtp-Source: AGHT+IGTKeUUuo7KoC1RLq1MuNgQsEnaX6DaZ/35944su1CBVjphvILdr8AWGuYvZXd46VFaBb9AjQ== X-Received: by 2002:a05:6214:c68:b0:68c:498e:d09f with SMTP id t8-20020a0562140c6800b0068c498ed09fmr4765609qvj.54.1706578956903; Mon, 29 Jan 2024 17:42:36 -0800 (PST) Received: from localhost (2603-7000-0c01-2716-da5e-d3ff-fee7-26e7.res6.spectrum.com. [2603:7000:c01:2716:da5e:d3ff:fee7:26e7]) by smtp.gmail.com with ESMTPSA id p16-20020a0cc3d0000000b00686ac3226ccsm3935915qvi.114.2024.01.29.17.42.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 29 Jan 2024 17:42:36 -0800 (PST) From: Johannes Weiner To: Andrew Morton Cc: Nhat Pham , Yosry Ahmed , Chengming Zhou , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH 19/20] mm: zswap: function ordering: writeback Date: Mon, 29 Jan 2024 20:36:55 -0500 Message-ID: <20240130014208.565554-20-hannes@cmpxchg.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240130014208.565554-1-hannes@cmpxchg.org> References: <20240130014208.565554-1-hannes@cmpxchg.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: C6CCEC000B X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: wd4qnji194j5ujqgoz5bkxniz3chxgis X-HE-Tag: 1706578957-905289 X-HE-Meta: U2FsdGVkX18iOQtoFHTQEd2xEfjWdVqBbC73lQ5+BxlHsZ2rOHGt7STB5CubN5bkcthWmVuQNe+denwA8FDHhe+BmYdo+KFjVXC0Xojcb6N0s9omQzz29U6fHjM/nT665UY5w9B6gtmZ5tYMO89wAeTRBsi1VwtsWeecgBbnzJH3rgbac8N9YInwnjVWE0nZn0I4GOsa812M0KGxxTdA8FgWj8E1dKhZlYSrnZC5++lFxLRpiuQ+oSZ7zgbO8ltedk7FX8yP6OvyjkSho8NyE9iLtrTuEz72C1x/P6B5e9NxyTEOCGiEDd0YeyeGK9R/mybVG33MMjMmxPRgMwm6buEAoa4iZp696O31LQ6KFeVj+ZUsWHORjQuZ+iynWXUPtST9Z3m19RGvdzxr910dyoyvca6fhJqiSKvwWBvJLuv+0hT966Ti3JerbinIF7AWUsaTylVefK4vBAzFgAF1i21/cJDdAIuJELYlu+9Cj2oPHY/cyOkdLlSSF26Ht8VYULrMK/yE+r6T6ucQ7cRAzRDXKt25CQcQ2sM8d/+rR2aVcYbgi+exgerna0+yCJW4exkf7vVC+UET9ycGxJ5kH4/ikJotUrTPArYDZi8BfSxQuC1UI2nGD80uw6dZNilGXJ4qNGiKNLvIenlPH8GeK6ABrHhsem7IQ6jyWY9V+jXR6csYUEbnNvxa/8Y+Yz7J56sE6QjbyVOX4Bs9iW96eoHbpHv+IbsZwUlgloTdO52xdBzRtGSECDX5D9oP0bi180kF0WLaygcuN86txwKDCrps2uXtuYOPX+BWKtYr1zmGMkPRagi6hGyMQhfZVkkjwYi9p6PueUMHHTP5+m4UtwBrcuqjYK5TxFlmFHzkdyfcqqjT9cj8ZNfkU/znddBq4jTcsudiqvehjTsaH0LoeUyqMaxpId9NNv4Orj/7+uxN/GZX5AM1uOklUTEvDQeoNzWHjxWePhpGsZH2BHH nVpKSSg1 MnF6PeT87Oln7GelZFsVUXSdbmlOt+4yn1VKFF0LsQTddPMJgrgfcNCRmvc2albw7ySCcu+trJkxjlHl9jHpq5vUPF3Y8b9L2knFyiWg5+YHWgqQP0FdG4hM8TTBZeUGuHF4MWiojR5kE1jZRYBowgGP/coDmgD4C+EFG40AeldT6Wv68jRHkAw1C5UB1HCxNk9uTvnui54hyuSoavB3MwaYM5JHu1s9mS2j1RPlsAIUh5y1OYqif4Bz4obOIU5poRdf9BEn4XFKrqtrA+mmNfyciw3ZIQjmqIkYka7jWHAZEt2NKrFuJloxsBnVW8LUj+kp7uTE117Q+5iXROjqYrFBrQOwuc8CcJ1yclsnGcNypwvtIrWdiaFs1VR9e7R8OkJTYlV0wUlqLsZWzg0bn94WFBX6nfhU8lGK2Ies++Pza5a2S+X3pmbCEMv+COrO1yACT X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Shrinking needs writeback. Naturally, move the writeback code above the shrinking code. Delete the forward decl. Signed-off-by: Johannes Weiner Reviewed-by: Nhat Pham --- mm/zswap.c | 183 ++++++++++++++++++++++++++--------------------------- 1 file changed, 90 insertions(+), 93 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index acd7dcd1e0f2..0cb3437d47eb 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -276,9 +276,6 @@ static inline struct zswap_tree *swap_zswap_tree(swp_entry_t swp) pr_debug("%s pool %s/%s\n", msg, (p)->tfm_name, \ zpool_get_type((p)->zpools[0])) -static int zswap_writeback_entry(struct zswap_entry *entry, - swp_entry_t swpentry); - static bool zswap_is_full(void) { return totalram_pages() * zswap_max_pool_percent / 100 < @@ -1163,6 +1160,96 @@ static void zswap_decompress(struct zswap_entry *entry, struct page *page) zpool_unmap_handle(zpool, entry->handle); } +/********************************* +* writeback code +**********************************/ +/* + * Attempts to free an entry by adding a folio to the swap cache, + * decompressing the entry data into the folio, and issuing a + * bio write to write the folio back to the swap device. + * + * This can be thought of as a "resumed writeback" of the folio + * to the swap device. We are basically resuming the same swap + * writeback path that was intercepted with the zswap_store() + * in the first place. After the folio has been decompressed into + * the swap cache, the compressed version stored by zswap can be + * freed. + */ +static int zswap_writeback_entry(struct zswap_entry *entry, + swp_entry_t swpentry) +{ + struct zswap_tree *tree; + struct folio *folio; + struct mempolicy *mpol; + bool folio_was_allocated; + struct writeback_control wbc = { + .sync_mode = WB_SYNC_NONE, + }; + + /* try to allocate swap cache folio */ + mpol = get_task_policy(current); + folio = __read_swap_cache_async(swpentry, GFP_KERNEL, mpol, + NO_INTERLEAVE_INDEX, &folio_was_allocated, true); + if (!folio) + return -ENOMEM; + + /* + * Found an existing folio, we raced with swapin or concurrent + * shrinker. We generally writeback cold folios from zswap, and + * swapin means the folio just became hot, so skip this folio. + * For unlikely concurrent shrinker case, it will be unlinked + * and freed when invalidated by the concurrent shrinker anyway. + */ + if (!folio_was_allocated) { + folio_put(folio); + return -EEXIST; + } + + /* + * folio is locked, and the swapcache is now secured against + * concurrent swapping to and from the slot. Verify that the + * swap entry hasn't been invalidated and recycled behind our + * backs (our zswap_entry reference doesn't prevent that), to + * avoid overwriting a new swap folio with old compressed data. + */ + tree = swap_zswap_tree(swpentry); + spin_lock(&tree->lock); + if (zswap_rb_search(&tree->rbroot, swp_offset(swpentry)) != entry) { + spin_unlock(&tree->lock); + delete_from_swap_cache(folio); + folio_unlock(folio); + folio_put(folio); + return -ENOMEM; + } + + /* Safe to deref entry after the entry is verified above. */ + zswap_entry_get(entry); + spin_unlock(&tree->lock); + + zswap_decompress(entry, &folio->page); + + count_vm_event(ZSWPWB); + if (entry->objcg) + count_objcg_event(entry->objcg, ZSWPWB); + + spin_lock(&tree->lock); + zswap_invalidate_entry(tree, entry); + zswap_entry_put(entry); + spin_unlock(&tree->lock); + + /* folio is up to date */ + folio_mark_uptodate(folio); + + /* move it to the tail of the inactive list after end_writeback */ + folio_set_reclaim(folio); + + /* start writeback */ + __swap_writepage(folio, &wbc); + folio_put(folio); + + return 0; +} + /********************************* * shrinker functions **********************************/ @@ -1419,96 +1506,6 @@ static void shrink_worker(struct work_struct *w) zswap_pool_put(pool); } -/********************************* -* writeback code -**********************************/ -/* - * Attempts to free an entry by adding a folio to the swap cache, - * decompressing the entry data into the folio, and issuing a - * bio write to write the folio back to the swap device. - * - * This can be thought of as a "resumed writeback" of the folio - * to the swap device. We are basically resuming the same swap - * writeback path that was intercepted with the zswap_store() - * in the first place. After the folio has been decompressed into - * the swap cache, the compressed version stored by zswap can be - * freed. - */ -static int zswap_writeback_entry(struct zswap_entry *entry, - swp_entry_t swpentry) -{ - struct zswap_tree *tree; - struct folio *folio; - struct mempolicy *mpol; - bool folio_was_allocated; - struct writeback_control wbc = { - .sync_mode = WB_SYNC_NONE, - }; - - /* try to allocate swap cache folio */ - mpol = get_task_policy(current); - folio = __read_swap_cache_async(swpentry, GFP_KERNEL, mpol, - NO_INTERLEAVE_INDEX, &folio_was_allocated, true); - if (!folio) - return -ENOMEM; - - /* - * Found an existing folio, we raced with swapin or concurrent - * shrinker. We generally writeback cold folios from zswap, and - * swapin means the folio just became hot, so skip this folio. - * For unlikely concurrent shrinker case, it will be unlinked - * and freed when invalidated by the concurrent shrinker anyway. - */ - if (!folio_was_allocated) { - folio_put(folio); - return -EEXIST; - } - - /* - * folio is locked, and the swapcache is now secured against - * concurrent swapping to and from the slot. Verify that the - * swap entry hasn't been invalidated and recycled behind our - * backs (our zswap_entry reference doesn't prevent that), to - * avoid overwriting a new swap folio with old compressed data. - */ - tree = swap_zswap_tree(swpentry); - spin_lock(&tree->lock); - if (zswap_rb_search(&tree->rbroot, swp_offset(swpentry)) != entry) { - spin_unlock(&tree->lock); - delete_from_swap_cache(folio); - folio_unlock(folio); - folio_put(folio); - return -ENOMEM; - } - - /* Safe to deref entry after the entry is verified above. */ - zswap_entry_get(entry); - spin_unlock(&tree->lock); - - zswap_decompress(entry, &folio->page); - - count_vm_event(ZSWPWB); - if (entry->objcg) - count_objcg_event(entry->objcg, ZSWPWB); - - spin_lock(&tree->lock); - zswap_invalidate_entry(tree, entry); - zswap_entry_put(entry); - spin_unlock(&tree->lock); - - /* folio is up to date */ - folio_mark_uptodate(folio); - - /* move it to the tail of the inactive list after end_writeback */ - folio_set_reclaim(folio); - - /* start writeback */ - __swap_writepage(folio, &wbc); - folio_put(folio); - - return 0; -} - static int zswap_is_page_same_filled(void *ptr, unsigned long *value) { unsigned long *page;