From patchwork Wed Jun 21 09:30:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13286960 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E005EEB64D7 for ; Wed, 21 Jun 2023 09:30:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 589048D0002; Wed, 21 Jun 2023 05:30:16 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 50FF18D0001; Wed, 21 Jun 2023 05:30:16 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 389A88D0002; Wed, 21 Jun 2023 05:30:16 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 251998D0001 for ; Wed, 21 Jun 2023 05:30:16 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id D2FB1A08BE for ; Wed, 21 Jun 2023 09:30:15 +0000 (UTC) X-FDA: 80926233990.23.A9BB579 Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) by imf26.hostedemail.com (Postfix) with ESMTP id C5B5114001C for ; Wed, 21 Jun 2023 09:30:13 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=U6f0qHda; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf26.hostedemail.com: domain of 3JMOSZAoKCCEVLPOV7EJBADLLDIB.9LJIFKRU-JJHS79H.LOD@flex--yosryahmed.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3JMOSZAoKCCEVLPOV7EJBADLLDIB.9LJIFKRU-JJHS79H.LOD@flex--yosryahmed.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687339813; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=ZkMJYW9AaEkYEFdiIfyRJ8y7x5cNBiWOkfQ1Jk9vOOI=; b=0GrGaDO4FIKpGaSlrKmGN+ETjN0ZyXQlrQsRPuycdGX0RDtsjWOBRZwNRnF6JJrqz5B+FO qrYSHWkvT8UjUo12Li1CySyvKgCHcYW8TBSsifl2zHYhu56qoWBjsWb5M5OvFdzm3y0cyi LlzkPxcobaCIh5hCCR5kisLcnP7jCCI= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=U6f0qHda; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf26.hostedemail.com: domain of 3JMOSZAoKCCEVLPOV7EJBADLLDIB.9LJIFKRU-JJHS79H.LOD@flex--yosryahmed.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3JMOSZAoKCCEVLPOV7EJBADLLDIB.9LJIFKRU-JJHS79H.LOD@flex--yosryahmed.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687339813; a=rsa-sha256; cv=none; b=7ctVYtatszXFruflBf1tRTw59Cwh9kvwcky/F2igaYj8crOBvvFGROp8JuVp7TMOmrgKDS jh06XwcwthoEpIVHeEGOKr6iun+ztAE/sruND/dmC4REj8SCdLQ3gyKCESRXHhdhI4vnsU XkbS+OT0f9tJh0z8jzhrMtudiT7/9nE= Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-25b79a5cf1aso4094328a91.1 for ; Wed, 21 Jun 2023 02:30:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1687339812; x=1689931812; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=ZkMJYW9AaEkYEFdiIfyRJ8y7x5cNBiWOkfQ1Jk9vOOI=; b=U6f0qHdaGld6GTFVyXxfloqd25G2EbbnWjUmTySziT7GSoTmtZppD3J4/pbhq/vTr1 FFrgcAlhqTCcqPc2fBGDrjxKgSAihBui7o41fev/08bRLTT4nyGNLaIHeGTUSzNvnAAO DN/9CTNj+eA4WdTANg86emSsQY4nvBEGRRhB/ZAJJuI3AxPKCnmFTdhWqcoSem0mF/pq JjHJy7Ca+IFbWiqi0l8H/LHQLNn5f6UIuqc8lDTvNY+QGwaGWGGdJaRy+kKc52aSFnt5 U9RMgouucsiKJp4d+Z9Uh5kG6aeiyZGJ4RJjONYN8P1iFu+BzlK3XuBwRqVb5knD0dzE vjVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687339812; x=1689931812; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=ZkMJYW9AaEkYEFdiIfyRJ8y7x5cNBiWOkfQ1Jk9vOOI=; b=emsEu1nCA0+n8fAnfIgpgbvCUuVXne3ucuH2PuwNZXtAjWfwLvicByG5fnA7XQvsrC 7E+ucrUB94ANL6fbXSqp6NWDxLaBJbaNIxYyuOL2AA/maMoDHYNo+7w1afMFnWnXhR/U oskRkASP6Xuzw55W2ravXmbnBV3/X7zzYDoL5cS2c70McgDZlvSAEPN9NkQK/wSLZv1W CqwzfwX8NY+rh2mtwad6mEGG7KCqUKVHL9VPjFggrWfHELupAC5hdYlpM71Mbf8QShrh LLGlAmVA/H0gUqpUfL5/+TxxuFJgICKXZjLbmok2d/TwhsMHoSH8RmuLIC4bU811zbph HC1g== X-Gm-Message-State: AC+VfDyaAAEuPl1U+EYpt5z6KNR/+6tFMKgnm0/ZFQTuKNYOqL4h7cvW Yo2wxYG2dbJOG2I/UrfYUQQVamkpzrgxoMbl X-Google-Smtp-Source: ACHHUZ7l9rcK0816aFWbVC6lPdhWa3cUm/9YUOdHHYlROSnCwgf2ZSIN+kchQK1Tab23ig2CAx/x3hozCaVcEVHE X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:7f:e700:c0a8:2327]) (user=yosryahmed job=sendgmr) by 2002:a17:90a:404c:b0:25e:c3a3:ccc7 with SMTP id k12-20020a17090a404c00b0025ec3a3ccc7mr1891887pjg.3.1687339812393; Wed, 21 Jun 2023 02:30:12 -0700 (PDT) Date: Wed, 21 Jun 2023 09:30:09 +0000 Mime-Version: 1.0 X-Mailer: git-send-email 2.41.0.162.gfafddb0af9-goog Message-ID: <20230621093009.637544-1-yosryahmed@google.com> Subject: [PATCH] mm: zswap: fix double invalidate with exclusive loads From: Yosry Ahmed To: Andrew Morton Cc: Domenico Cerasuolo , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Konrad Rzeszutek Wilk , Seth Jennings , Dan Streetman , Vitaly Wool , Johannes Weiner , Nhat Pham , Yu Zhao , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yosry Ahmed X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: C5B5114001C X-Stat-Signature: oyuus31n9za4c1f9iz3phk7q748ygeog X-Rspam-User: X-HE-Tag: 1687339813-255249 X-HE-Meta: U2FsdGVkX19oRGpyMnM9YRbsVWi47+/sh71Za1NPdi+2xJh2mV9bUOmTbG9fiH718CCd1aeHK2Tf8CRHi+HFavnjEn4RtRdF6BhDMGRTFwlECq6PL1kRE8ddxMugf59G6vZdLO2ilCJK0AjM1V+i2PzmPH3l51yS/MbKiTA6rwRxHM/vqWZ0cLZbJXWIEIb4VUY/7TJlcxNuo/jsDd2Gm8Q6mn4VsWbWXO0qOAYVRnC4398IUfuX3bz0c2w3NiV7gVFORKOjWCSPwuhgOFCOm9ydBIiFD5+jstR+PpVBuKgBDBQp/5Ju6gCOH8GV1h/oe4iBWidbKicspgU7GuJS+T5teW38DmAqZVr1dOHC/dJ7n7tH4X643VYQZVAR9W8e7lscDjWAKNeU2JABPbFA+0pgxAOckDbg4ttj0Sl36JjaqPfwnsE28JuEiPzZHvN7jcvHXbiQ+21XnQRWqdgcOjBwJV6mnp6BcSgFYeENZoEHs4HUi2JeCLJcTeVOdNJwLvagmaWltxBr7dowqkNE0I2G3zmmMdVfI+yR1/yMGToXcGDRMNb7VcGe4VqvxxZ+XXGGEIpI4Z4eGwLL90pV0lFFudDsKtIXCy9pKMSwVu68vscYTQke51x+eDI+kpLjc2dHRHAhsL6FFElCO5zvZjSWQ0uVmdLL2PoCZxHZ8NajasUsbpcUKQLuEDaF/1WDi0vLpVO2Sk853KME1Bc9l1SFkLl0adSoC5mcyDT4+AKxMz6lJc1IPeiDSqQ6Z2L5bRP1ru3R5DH69SphVbvX3HWwDRlat8CdkKQQyEaeJ0rI+Xmd5myRSv5+iUPO3wUaC9KymEHzkobEZdA0CjF+iF6IegJK4QwFKxF5Z29wZXL9ni0HL+0ZrofxHS8WV0PoLQ9ohY1l0VZx1U1jSKB4R8ylWq9bSCJIWrIEMxpdbSyET359/lj3C8XvFsd+n+hXbr77trZv3RIYBcTL+hC zs/9MAff ve7Y2caylU+Ls1efrtzWXK0isOB4PcwzO/XgUpbueBNIvADcJIbSG5Smy0OwiwMvx/HLqD3moqBjAJYVGp7viDpUVhUfvw9vQ/j4S8zVYVHijQtjsF8tZceiwEemKliSRq/g6DVZHAq4lz95ec4Ag8V9PLm7EBgWczs0fECX0JvMBMQxFg8w9KvvqAoR5RBvUmSRhyXg3DG8+H8m/Ypy0g7oTqNt+tszP2D9MA8MR7nzv4sM1o8+OX7OjE6bfsfUctelb9togPkyKXC0uRA550iDTP7sUK2OsjAWJzz8tY/oe1fipj4xLpctSByObewoTWsk+n+UYm0m00nH6Bu5OPECHPfU7eso7lVRqyITRZLHPlpq6HH76EXErYTnsrQqI+7wotaPTEmTiyK+4h+WKGzbZ5rPiuZxFL4VMswfGw0M/61lOxD7l2GXPmVoSjEZ0838v507hOPUwv6upo/k8sNvtkWAM/Ymrndaob0AkRH3LIcloQNudEzJGAaJ1ENHu0UN8G9is3XI/3qLlkcizJBT23oHJDtuswnSfT8a3yFroqQb5l0NCMjzH11Z/6uAQUJymmVYzUeLBc6rRcyAVeUQdB8/O6enxOaxH0vBnQ1u7HAzujWyzQWwm4wjad7cLGOMb0J/78HjdTaYZXab8ANbbCKxRDHNuoSxCG7Hu+U3ml8iNth5B1+nFmQQMRVIbxPa9fvHAll5KLmTxHWwMpZqK7PdxxWBMsSNKV5uuF6D1og4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If exclusive loads are enabled for zswap, we invalidate the entry before returning from zswap_frontswap_load(), after dropping the local reference. However, the tree lock is dropped during decompression after the local reference is acquired, so the entry could be invalidated before we drop the local ref. If this happens, the entry is freed once we drop the local ref, and zswap_invalidate_entry() tries to invalidate an already freed entry. Fix this by: (a) Making sure zswap_invalidate_entry() is always called with a local ref held, to avoid being called on a freed entry. (b) Making sure zswap_invalidate_entry() only drops the ref if the entry was actually on the rbtree. Otherwise, another invalidation could have already happened, and the initial ref is already dropped. With these changes, there is no need to check that there is no need to make sure the entry still exists in the tree in zswap_reclaim_entry() before invalidating it, as zswap_reclaim_entry() will make this check internally. Fixes: b9c91c43412f ("mm: zswap: support exclusive loads") Reported-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> Signed-off-by: Yosry Ahmed Reviewed-by: Domenico Cerasuolo --- mm/zswap.c | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 87b204233115..62195f72bf56 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -355,12 +355,14 @@ static int zswap_rb_insert(struct rb_root *root, struct zswap_entry *entry, return 0; } -static void zswap_rb_erase(struct rb_root *root, struct zswap_entry *entry) +static bool zswap_rb_erase(struct rb_root *root, struct zswap_entry *entry) { if (!RB_EMPTY_NODE(&entry->rbnode)) { rb_erase(&entry->rbnode, root); RB_CLEAR_NODE(&entry->rbnode); + return true; } + return false; } /* @@ -599,14 +601,16 @@ static struct zswap_pool *zswap_pool_find_get(char *type, char *compressor) return NULL; } +/* + * If the entry is still valid in the tree, drop the initial ref and remove it + * from the tree. This function must be called with an additional ref held, + * otherwise it may race with another invalidation freeing the entry. + */ static void zswap_invalidate_entry(struct zswap_tree *tree, struct zswap_entry *entry) { - /* remove from rbtree */ - zswap_rb_erase(&tree->rbroot, entry); - - /* drop the initial reference from entry creation */ - zswap_entry_put(tree, entry); + if (zswap_rb_erase(&tree->rbroot, entry)) + zswap_entry_put(tree, entry); } static int zswap_reclaim_entry(struct zswap_pool *pool) @@ -659,8 +663,7 @@ static int zswap_reclaim_entry(struct zswap_pool *pool) * swapcache. Drop the entry from zswap - unless invalidate already * took it out while we had the tree->lock released for IO. */ - if (entry == zswap_rb_search(&tree->rbroot, swpoffset)) - zswap_invalidate_entry(tree, entry); + zswap_invalidate_entry(tree, entry); put_unlock: /* Drop local reference */ @@ -1466,7 +1469,6 @@ static int zswap_frontswap_load(unsigned type, pgoff_t offset, count_objcg_event(entry->objcg, ZSWPIN); freeentry: spin_lock(&tree->lock); - zswap_entry_put(tree, entry); if (!ret && zswap_exclusive_loads_enabled) { zswap_invalidate_entry(tree, entry); *exclusive = true; @@ -1475,6 +1477,7 @@ static int zswap_frontswap_load(unsigned type, pgoff_t offset, list_move(&entry->lru, &entry->pool->lru); spin_unlock(&entry->pool->lru_lock); } + zswap_entry_put(tree, entry); spin_unlock(&tree->lock); return ret;