From patchwork Mon Mar 25 23:50:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13603190 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1507BC54E64 for ; Mon, 25 Mar 2024 23:50:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 398826B0099; Mon, 25 Mar 2024 19:50:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 322536B009A; Mon, 25 Mar 2024 19:50:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0FCC16B009B; Mon, 25 Mar 2024 19:50:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id DECCE6B0099 for ; Mon, 25 Mar 2024 19:50:37 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id B0026A03C9 for ; Mon, 25 Mar 2024 23:50:37 +0000 (UTC) X-FDA: 81937208514.19.FCFF2E6 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf22.hostedemail.com (Postfix) with ESMTP id EF5F1C0007 for ; Mon, 25 Mar 2024 23:50:35 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=JBMZOzew; spf=pass (imf22.hostedemail.com: domain of 3yw0CZgoKCCYaQUTaCJOGFIQQING.EQONKPWZ-OOMXCEM.QTI@flex--yosryahmed.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3yw0CZgoKCCYaQUTaCJOGFIQQING.EQONKPWZ-OOMXCEM.QTI@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711410636; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=z6nu3U2cdQmEtB2lnDGjl91RMjz6aut26G90+PFcykM=; b=6Y9v8bMfmC0AuCAVcX0eVhLu5EFq57N2dYKyNIH7W0WTiQdpeQYjb/nDobUXbe1gpYn3D/ K9UQyr/51p9QuzepX8kduXi9x0g/ULt9Uvec8gjdYLmijJXqSynDCZKRPCn35334zjFaMS 6Vg4c0Yf2wPntNYXsoCTRGu+QHKTpQ0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711410636; a=rsa-sha256; cv=none; b=XbDJb8IWttUVXnAv/yVEo1c2s+YngQ+8eN5czcugfnS4P16yPShwEcGVjnRE2/ZcddiLgu dcgRZkd6sEULL+cFHahkfWVZXkH5xNbxjRQYspvLYRGbZ15SKzZZMPmCjSVOTxhnhQaOvW pqMBZr20T2VcAxkFXE2GQqrgud3zB8o= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=JBMZOzew; spf=pass (imf22.hostedemail.com: domain of 3yw0CZgoKCCYaQUTaCJOGFIQQING.EQONKPWZ-OOMXCEM.QTI@flex--yosryahmed.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3yw0CZgoKCCYaQUTaCJOGFIQQING.EQONKPWZ-OOMXCEM.QTI@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dbf618042daso7713071276.0 for ; Mon, 25 Mar 2024 16:50:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1711410635; x=1712015435; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=z6nu3U2cdQmEtB2lnDGjl91RMjz6aut26G90+PFcykM=; b=JBMZOzewLcGqH4BfisGsBnLO4DpJ4UWl4/75QE1luth2Tn0uL0V/7ijkVYHfQar+lb +rir/3cGD3AO32BpIU47pgW7kC4PqnpGqy9YwvDsGYEWaLsugQHIBHDB5R0fiKGyHtwO vHkjHMlEh4lYALkobMnvMHUxOIhI4l5NPD9aqn8u4JKv9NcoysPHqT2DDGxTjx4Drj4F vUvq1evGm/SrQXXGMJyk0nQ/h3ryV3+27yA8ZQ5e6oQdYN4VCecuBSoTdM7ZIHf/xFEA qHC8Nzv4PcvwNIJlVV2mQ+hsak90rJomiPG6c55okZXLxGc9t3C0ujiim5R1ZVVVgXMD GMtQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711410635; x=1712015435; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=z6nu3U2cdQmEtB2lnDGjl91RMjz6aut26G90+PFcykM=; b=CX/qE3aGI1rGIfjmn71w48EL6HmquaTSn1V0Z8N8HZv4pabFZlQLIefqj8Ah7WHJBb eHTOX8TxVjbvidp+x+Op3v/O7cnft3S9laEBID+Tv9lw75AwAqPysrvs52UQEmjpGyVO EWBKkrO4YN3lM+6Lmor+qIz0jORTAN23ILpXdVZbFznyhuhDoK+MwK12XQUwMgvls/aT 29H5JHPdhAsZh28LtZADVctcYn1DURHjIp7ts+vbz+zwALGW+IMBTfsbLbapoBtzgFB3 MoPSvv/uShhYgGvuto+gq+XzSbScnKu/bs5PTTEF9OsjhREnRDgB/7mnHm9YJ5uzMqbh wFnA== X-Forwarded-Encrypted: i=1; AJvYcCVXDbU6XgNB/2VoeSJ6MsjLsS7dbbwnV6l3UCnp9oKp+g8p/GVtVYPMslAfRsSIi5kL/8S7cKUzYWQ8m88wXSk6g3M= X-Gm-Message-State: AOJu0Yy7Ns0JBl++vF80MGeb0+xmqEtoG+ejouy69u2UG3jGBMpMfyiH QZx1UX7FvfhummfqJ5O1JLHzdUK3MnbUlPq+Ah01F82QdopCJBRRw09O0rDE+cD3gEE7TtN6rfy Os2PpdLP9tmbiyPIGRw== X-Google-Smtp-Source: AGHT+IHhwuSMeZ8KbkpdVbgoG6k+kjBoalv5THVzyluQgGq+kifTjQF/ACTCwAHFlLZp6s7t/niB6q7ZIp/Hk5oK X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:29b4]) (user=yosryahmed job=sendgmr) by 2002:a05:6902:218d:b0:dda:c565:1025 with SMTP id dl13-20020a056902218d00b00ddac5651025mr317614ybb.2.1711410635103; Mon, 25 Mar 2024 16:50:35 -0700 (PDT) Date: Mon, 25 Mar 2024 23:50:17 +0000 In-Reply-To: <20240325235018.2028408-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20240325235018.2028408-1-yosryahmed@google.com> X-Mailer: git-send-email 2.44.0.396.g6e790dbe36-goog Message-ID: <20240325235018.2028408-10-yosryahmed@google.com> Subject: [RFC PATCH 9/9] mm: zswap: use zswap_entry_free() for partially initialized entries From: Yosry Ahmed To: Andrew Morton Cc: Johannes Weiner , Nhat Pham , Chengming Zhou , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yosry Ahmed X-Rspamd-Queue-Id: EF5F1C0007 X-Rspam-User: X-Stat-Signature: dwb798bi6bbd19nbjxxh7oy7c78wr3jc X-Rspamd-Server: rspam03 X-HE-Tag: 1711410635-884521 X-HE-Meta: U2FsdGVkX1+3W64ABLJwacUhx0FHWXBDZqi0lz7JV6QUbM7S+hDRZ+cTq+f4XF+x2lgcVEIXHAF/UmRPL8qFqU+2KG80/wLtOFot9q0Gy8DiMT4nsVl23rHgaAPevyAKdHolecc3Db08owZ7VW6wMtNx/Iyur+G4kpx5VH9VmHs4bdpGAH4bKrpEJUtsntiU3WXaeWgEsWz6w0gndBhUM7QeLBYINcIJAIjybJ1a/nAdKNyvuX0FnZ72pR0fKppMoMvvuWUHoMBGy2lgbQGAbCcpUZCqEtSmWqHcH+qk5rK+rVtajcoebo/rkZJgzU6LIgKxUmz+q33Zf9snkQptMT5p6XnvOItJINkMZaZMVUh5nSA1vHqE+N+FQNMcXFsUm9AY0TPrXtDa1ihYPLT0cv/I6kEbyAFpI03+pQxBD9UfBunDgtFpAe7XfnOA7aXqixxrnk6dzCLubYjQC3avvmfMTLiN/Uf3mpu8qT33yJv3qBKAGej1M8MtqpGYyqlT2KcB6Oo4vI+puVznzEWNnwp1fSZ1cLRR37GVV13mIfuQPg1dVJbFLoHUNtpkKfEmCiwVrQf7A6neM3C84SVL6OlkPPVrOslEweULZsWI+xyOZL9AFVQsuTYw/UNkS53hOh7/ulTmjNPVEr0fqITvMsN0/fdCC3bgQfa/zeCek5Z1SBLu7R6dtgQIEyKPX26PP3eNz0T35yCyyBvyUmBB4ueNv2Z9Rl1WaWUEiioX2mejatRm36a+1odpY6I+unmGaC2DVUybWDpJRy7bEVxMClqGOuSFzcmfkZJQp+rt+1t7FpPiK87GKsfO7pIPCYIhJmdrneHaC+Wbx7JOKZUJdCmaT5HHc5GHIR+BAaGu/A/Z8r8dcNnz77VkLCRvfaswCaptnRouqUOgFKszawnAZbPGhkb7fxWAcpRVlaxA2ZgQHKZPuL7pnqKwyMkABWF7FCesovOEBqJqKdYEVjN yP7WP783 /Ck4pw68BwTm/oDH7gjw/ypjUSD+ZxleC/3ldhwWxUs2iXBH/fyHTiH4jRkjoL+V9TzXfgFvqzlU1zOh1mpw44ufufLQfIJ0py0LXtqNZFPm4nMgwEb/YgQICNU5S7ChD7GMcOkqemeGQXlc8HAIZ5B7oIE9gXVPqdWbKQmjRp+0w28MdnKd8yoKieavk3ivZb5DhO9ESXZ611slOPOCUi4zEQcHU5SGOwBnJ/4zkDTB/zHwQaS2yRf5BwoawnVac4w7vGSLEvFEsYazvnzjdYMATB5tmnhWKPumxo9SFF2TsWd8/diEPlA2yfHSI81PX9i5Nl0hFp7e3GNVeGPzTmEYPGjTGOa5ku6Egx6MI5rr4xEiSEZan1fFSQGUjXGn5nXAFCt3/V77urd6hdmFnGfSD+mjYM5vgEyeCX1bj+UVZfhUqOfTUfSgvKSHretlVFwenRaNNvKLs+MMSBX1yX24DUVKs041Y6mmS0FGFJ5DGtMTzeFEgsck71bqILxDfpRfFPwYtqziXTkX7UjRGWqRXCrq1YkDbJHfYYh/swONFvx7aEr3tR0+D7K4Yc/nXhAo1EimBj93T2g9Lc+ouvjAE9qlyNWOe73uh14AODsAucYE6EeMWUcBGJDbb5SgopGMgkMfloWrKfvCnZWu0bUWYF0K7iC1WiQ6ZkVrMuaLyu0A= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: zswap_entry_free() performs four types of cleanups before freeing a zswap_entry: - Deletes the entry from the LRU. - Frees compressed memory. - Puts the pool reference. - Uncharges the compressed memory and puts the objcg. zswap_entry_free() always expects a fully initialized entry. Allow zswap_entry_free() to handle partially initialized entries by making it possible to identify what cleanups are needed as follows: - Allocate entries with __GFP_ZERO and initialize zswap_entry.lru when the entry is allocated. Points are NULL and length is zero upon initialization. - Use zswap_entry.length to identify if there is compressed memory to free. This is possible now that zero-filled pages are handled separately, so a length of zero means we did not successfully compress the page. - Only initialize entry->objcg after the memory is charged in zswap_store(). With this in place, use zswap_entry_free() in the failure path of zswap_store() to cleanup partially initialized entries. This simplifies the cleanup code in zswap_store(). While we are at it, rename the remaining cleanup labels to more meaningful names. Signed-off-by: Yosry Ahmed --- mm/zswap.c | 62 ++++++++++++++++++++++++++---------------------------- 1 file changed, 30 insertions(+), 32 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 9357328d940af..c50f9df230ca3 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -774,12 +774,13 @@ void zswap_memcg_offline_cleanup(struct mem_cgroup *memcg) **********************************/ static struct kmem_cache *zswap_entry_cache; -static struct zswap_entry *zswap_entry_cache_alloc(gfp_t gfp, int nid) +static struct zswap_entry *zswap_entry_cache_alloc(int nid) { struct zswap_entry *entry; - entry = kmem_cache_alloc_node(zswap_entry_cache, gfp, nid); - if (!entry) - return NULL; + entry = kmem_cache_alloc_node(zswap_entry_cache, + GFP_KERNEL | __GFP_ZERO, nid); + if (entry) + INIT_LIST_HEAD(&entry->lru); return entry; } @@ -795,9 +796,12 @@ static struct zpool *zswap_find_zpool(struct zswap_entry *entry) static void zswap_entry_free(struct zswap_entry *entry) { - zswap_lru_del(&zswap_list_lru, entry); - zpool_free(zswap_find_zpool(entry), entry->handle); - zswap_pool_put(entry->pool); + if (!list_empty(&entry->lru)) + zswap_lru_del(&zswap_list_lru, entry); + if (entry->length) + zpool_free(zswap_find_zpool(entry), entry->handle); + if (entry->pool) + zswap_pool_put(entry->pool); if (entry->objcg) { obj_cgroup_uncharge_zswap(entry->objcg, entry->length); obj_cgroup_put(entry->objcg); @@ -1447,7 +1451,7 @@ bool zswap_store(struct folio *folio) return false; if (!zswap_enabled) - goto check_old; + goto erase_old; /* Check cgroup limits */ objcg = get_obj_cgroup_from_folio(folio); @@ -1455,54 +1459,52 @@ bool zswap_store(struct folio *folio) memcg = get_mem_cgroup_from_objcg(objcg); if (shrink_memcg(memcg)) { mem_cgroup_put(memcg); - goto reject; + goto put_objcg; } mem_cgroup_put(memcg); } if (zswap_is_folio_zero_filled(folio)) { if (zswap_store_zero_filled(tree, offset, objcg)) - goto reject; + goto put_objcg; goto stored; } if (!zswap_non_zero_filled_pages_enabled) - goto reject; + goto put_objcg; if (!zswap_check_limit()) - goto reject; + goto put_objcg; - entry = zswap_entry_cache_alloc(GFP_KERNEL, folio_nid(folio)); + entry = zswap_entry_cache_alloc(folio_nid(folio)); if (!entry) { zswap_reject_kmemcache_fail++; - goto reject; + goto put_objcg; } - /* if entry is successfully added, it keeps the reference */ entry->pool = zswap_pool_current_get(); if (!entry->pool) - goto freepage; + goto free_entry; if (objcg) { memcg = get_mem_cgroup_from_objcg(objcg); if (memcg_list_lru_alloc(memcg, &zswap_list_lru, GFP_KERNEL)) { mem_cgroup_put(memcg); - goto put_pool; + goto free_entry; } mem_cgroup_put(memcg); } if (!zswap_compress(folio, entry)) - goto put_pool; - - entry->swpentry = swp; - entry->objcg = objcg; + goto free_entry; if (zswap_tree_store(tree, offset, entry)) - goto store_failed; + goto free_entry; - if (objcg) + if (objcg) { obj_cgroup_charge_zswap(objcg, entry->length); + entry->objcg = objcg; + } /* * We finish initializing the entry while it's already in xarray. @@ -1514,7 +1516,7 @@ bool zswap_store(struct folio *folio) * The publishing order matters to prevent writeback from seeing * an incoherent entry. */ - INIT_LIST_HEAD(&entry->lru); + entry->swpentry = swp; zswap_lru_add(&zswap_list_lru, entry); stored: @@ -1525,17 +1527,13 @@ bool zswap_store(struct folio *folio) return true; -store_failed: - zpool_free(zswap_find_zpool(entry), entry->handle); -put_pool: - zswap_pool_put(entry->pool); -freepage: - zswap_entry_cache_free(entry); -reject: +free_entry: + zswap_entry_free(entry); +put_objcg: obj_cgroup_put(objcg); if (zswap_pool_reached_full) queue_work(shrink_wq, &zswap_shrink_work); -check_old: +erase_old: /* * If the zswap store fails or zswap is disabled, we must invalidate the * possibly stale entry which was previously stored at this offset.