From patchwork Tue Jan 7 07:47:24 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13928316 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 60B48E77199 for ; Tue, 7 Jan 2025 07:47:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E9C996B00BF; Tue, 7 Jan 2025 02:47:45 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E4BC46B00C0; Tue, 7 Jan 2025 02:47:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D3A776B00C1; Tue, 7 Jan 2025 02:47:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id AFFFE6B00BF for ; Tue, 7 Jan 2025 02:47:45 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 70FEF1A078C for ; Tue, 7 Jan 2025 07:47:45 +0000 (UTC) X-FDA: 82979876490.10.C5E554D Received: from mail-pj1-f74.google.com (mail-pj1-f74.google.com [209.85.216.74]) by imf04.hostedemail.com (Postfix) with ESMTP id 9D69F40009 for ; Tue, 7 Jan 2025 07:47:43 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ru0gSVzc; spf=pass (imf04.hostedemail.com: domain of 3Htx8ZwoKCA0B154Bnuzrqt11tyr.p1zyv07A-zzx8npx.14t@flex--yosryahmed.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3Htx8ZwoKCA0B154Bnuzrqt11tyr.p1zyv07A-zzx8npx.14t@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736236063; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TgkmCWVczOKXI/I991UVFS0Ip8/4HV0rkcV91lhH2tc=; b=gdAlikm64M/KWppaI777xCFttIIPbpzXRA7UickroZyPu5+nqiDalV4/xm8qkcQqe0idii KOv79OtEqZMIGQn+PvEzGK47JIAUAVL+vNKzLUZ0krwenl38/AXTg/iCy/pvBAMFMEVbQ1 fgyM6TdRVDwELNmDlWfkCdr1ma/xOIY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736236063; a=rsa-sha256; cv=none; b=qC591GfUj6WNUlYvhphekB2kjJ0XbI6VxllEiH4N0f+kbuolcIFy3ZEokM5cEKz5ywSXgO iXXvRwNn+jDPgEy6Mx47x/rrduatzJNwJowh47bTjJD+QJq6KrLTwBUyK0IEhWVEcbNQ7P Kvh/+xGnlC6wuedni92yU8X92kHCrzk= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ru0gSVzc; spf=pass (imf04.hostedemail.com: domain of 3Htx8ZwoKCA0B154Bnuzrqt11tyr.p1zyv07A-zzx8npx.14t@flex--yosryahmed.bounces.google.com designates 209.85.216.74 as permitted sender) smtp.mailfrom=3Htx8ZwoKCA0B154Bnuzrqt11tyr.p1zyv07A-zzx8npx.14t@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-pj1-f74.google.com with SMTP id 98e67ed59e1d1-2ef79403c5eso33290190a91.0 for ; Mon, 06 Jan 2025 23:47:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1736236062; x=1736840862; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=TgkmCWVczOKXI/I991UVFS0Ip8/4HV0rkcV91lhH2tc=; b=ru0gSVzcS9H2VpVtaoowwZsiNrJ6+Y1xbat/COHsmAlEHZjm+VU+Fb0n9rTdoNHDG5 +0rWEdSSIye3b6qya4w4m36C6/26mC9goOAyMShhqD2+plNVvmBYVWpgBhhgh9vVh4Jr K7pSWK83gGPLQv1KoqJc37BhXIw0wicu4IkXqT8yzCGiiUCEDc8wa1OT7sJu+ztRbNNe lSqtB5gxEG1CbmTcjoV9vb0BNTBP0TZMnUnyF7tNmQjb79mOVyWfv1GZRZXBnVMC5+9m hiiJ/UmM4eJRe4bKOoJLeq53wbB8bzX+Lb1G7txng62qWzszuUFNx7xc15M6v03Fingu y8sg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736236062; x=1736840862; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=TgkmCWVczOKXI/I991UVFS0Ip8/4HV0rkcV91lhH2tc=; b=k11Y12crv9M3mEhGQzyev46pH4sHqbzNf3k+PWsUA5hYp+3a8pILhsjbhkiZyoh9VN Nf2NqmU8ThDcxiQ5qKEYuQXZxSR0IQYRnzIVTiJIP5OWVCW2IDvZNf2wcAyjs9e8/viF FRxv4I5nEQVHV/rGnM8zL8QeoX3WO34tsk4GLB5wwadaqRl+LdwyNUyE9Nc4f2VVsu0T fNDwsWuuLthZe/KM4LnByP876WIH+NsW2NMGdSJxRElImrWJy/P4uxS2xe/UmKve5D3J NmRjVkjznrUZbNjF+Opr8cSYvsUb4bPyDBVIV4xBrEHC9SjtSB5yllD9ZEkqxjW52Zpa M0fQ== X-Forwarded-Encrypted: i=1; AJvYcCXDm5jRfWarEACZsjH1gEt1adeI968M5eyRA9lj0qpxXH7bRNRg8QL5LvQ4QP3X7qXGA4fRkUu7Eg==@kvack.org X-Gm-Message-State: AOJu0YwM7UUxLGUeIdYFztULzRGaujvWReDpCRdr5P7G5s8B1CCuu3LB JzpRiY4SOUxozOxGxoCkbHArKPs+i9CId8hfJeCeNIPqccGap87r7lbRb0QSjkT8eSNhkF/sZA5 FLFSdR6OytyNZ51bLbg== X-Google-Smtp-Source: AGHT+IHQ7AZqxFj9pAZrCsvqiHbUVAdUbqvqSZXExiCCJxdWdkFbZ6wGHRFwryywnMMfUT9qvbfOq0nZYKQovlUN X-Received: from pjbee7.prod.google.com ([2002:a17:90a:fc47:b0:2ef:8d43:14d8]) (user=yosryahmed job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:6cb:b0:2ee:863e:9fff with SMTP id 98e67ed59e1d1-2f452e22560mr94099372a91.10.1736236062518; Mon, 06 Jan 2025 23:47:42 -0800 (PST) Date: Tue, 7 Jan 2025 07:47:24 +0000 In-Reply-To: <20250107074724.1756696-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20250107074724.1756696-1-yosryahmed@google.com> X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog Message-ID: <20250107074724.1756696-2-yosryahmed@google.com> Subject: [PATCH RESEND 2/2] mm: zswap: use SRCU to synchronize with CPU hotunplug From: Yosry Ahmed To: Andrew Morton Cc: Johannes Weiner , Nhat Pham , Chengming Zhou , Vitaly Wool , Barry Song , Sam Sun , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yosry Ahmed , stable@vger.kernel.org X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 9D69F40009 X-Stat-Signature: accrsd7fb8qz64g9ro3owrg7w3r4t3ax X-Rspam-User: X-HE-Tag: 1736236063-92215 X-HE-Meta: U2FsdGVkX18jM1QiGRCNZCuYy2YSiBf9UFaO4HLB37wWyToXsNsqC6QUAuqkiukrlov6zZKiC8A3StdQAZSu3fobDeuwQlwFXovviKtrE0cF7KZc38ZePSdcxhqDCdCk5Y2ZzlNvYhhSuP27yWuoRUQP74oMhc/6RbejkPclVGvB8mFOIBnEmJ2/S27PN+Rlsj7J6RAqZw5GXFCO42QbTJxWJajbxfk+7I6fHQ6onqYT7TWY4uZEXa4r0uMgt+irk9P7G00If2SeVd33vb5Ybj+4TShsLjTazvZBtQFtgfcdRE04UcyWvBVjuNlQ1BFGgwj0/UjbSQ0XqoG+G1m0e8KOVuwYZej8NxYJ00rvP1NGfuOT48q+uIO+ciiRR+rgF4WfQLZx7oE6SP3a9fuajd4q3ER1vicTdzqdO+Pa0a/erJK4w45K5GxcZYOJaBNy/W+JRZ4jAGYaGNCEfZnKTUFalbC9Tdevhf5pII4+SwGPJSGxyK4S8DjRi0rQ7BiKR7rqbi4C43D51NKldQ2DJeQFCEyT2z5Cji1LksXreOOXhRU1PC8q2nE8pd2edSSzdCPChdnwhPk8/OxnPNOa9Sf5kUp3gaXPkvT8ETPQlh/0DUovRveaiay+Vu90gUD5JbhOx3vQOcyCf2io0tZVdsKwL00O9OP5S6e1FPz9tWZcluhgehF9eJpKJvLns+BQBsUuJJZy2OYWKx6h6rv++xJjXv4ItWXg6ei57qmQdgB7XwfS/RtN9gZKggsdtTQ1AhfT5+fkP6zor3rH+2Ugh9kcwijUqBxY0ACSBOsgjoWcR0qtWpmTaMJjbe4Ob5Va0L3IYQowPobNUgjKs2ohvXvL63ur8Q1EFOBlUHyk16BpEhLWdRMTbXvPbuJEVRVId3oIvLwkPE6PjyXZDS00IzdFDV6DwWaANMAPPtG9MnEyntchn6/U6bYMOj1/sNtS/8F5iPye7XWNtDx1tsP +4jDA+hU ci9bx7NFgJKDAgNVq7KJ6PjXmrmNp2LNjI8EyBLCf8+Qf6O7bIIdyzgZv3pUYFSX7EEb8lodBCnfKvmXCLdiJ8mGFx+4hJn27J2Wfgneuff82dlDP0wy6jwwTJFe0ZxOYRezyU21mOmcXNPCoLBNKgA04nb0D6fp0RVRpd3FNAflbXk+yl+XUGXh1tjgMX5YOlxHV5Ym5pp8TpSx5IEprF/KPBhCL3Zb4SnpNChzNhC1qtFftGm0aAgjZWR2+1xKfV0s83R3CJkdpQTBgP/R31B9fbMnPW37zVbXufAc3RetHjTAHN97QVtPxLeWER4RbUood5lJBOH4JbVhgZ5N9Un/+vSqjgrNiG2US6UYn9qCboStxqYT68zVcrkQyGS5bmquBtgbv9FY5D1QohyELtDpwPdzeTGqhZC8HybqDnHLfLvFASdXNw+ubeuC1E8IJADG43KLbEK9MEqB0gweWMXyddzfcIrR6UvPiKALDOlVWvrA24I1mJZdaxXOnsAqDSf2KYmNSBAMRyCCbuC1iDwbiG6gAnPf8oqwDNv+gTgtrhle1xC77tbA/9f2yjYyXvf/QxoEtPFuijtLNJtdXDqLGfK/q5ehTMvD8+jcGal12KSdPK6a5drv8JL29yf0tyoX5+krKx4MEBlOwGq5ng2v2Oscr/RFn/AtmKsXFW5it/rfGB2MYhuVoXA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In zswap_compress() and zswap_decompress(), the per-CPU acomp_ctx of the current CPU at the beginning of the operation is retrieved and used throughout. However, since neither preemption nor migration are disabled, it is possible that the operation continues on a different CPU. If the original CPU is hotunplugged while the acomp_ctx is still in use, we run into a UAF bug as the resources attached to the acomp_ctx are freed during hotunplug in zswap_cpu_comp_dead(). The problem was introduced in commit 1ec3b5fe6eec ("mm/zswap: move to use crypto_acomp API for hardware acceleration") when the switch to the crypto_acomp API was made. Prior to that, the per-CPU crypto_comp was retrieved using get_cpu_ptr() which disables preemption and makes sure the CPU cannot go away from under us. Preemption cannot be disabled with the crypto_acomp API as a sleepable context is needed. Commit 8ba2f844f050 ("mm/zswap: change per-cpu mutex and buffer to per-acomp_ctx") increased the UAF surface area by making the per-CPU buffers dynamic, adding yet another resource that can be freed from under zswap compression/decompression by CPU hotunplug. There are a few ways to fix this: (a) Add a refcount for acomp_ctx. (b) Disable migration while using the per-CPU acomp_ctx. (c) Use SRCU to wait for other CPUs using the acomp_ctx of the CPU being hotunplugged. Normal RCU cannot be used as a sleepable context is required. Implement (c) since it's simpler than (a), and (b) involves using migrate_disable() which is apparently undesired (see huge comment in include/linux/preempt.h). Fixes: 1ec3b5fe6eec ("mm/zswap: move to use crypto_acomp API for hardware acceleration") Cc: Signed-off-by: Yosry Ahmed Reported-by: Johannes Weiner Closes: https://lore.kernel.org/lkml/20241113213007.GB1564047@cmpxchg.org/ Reported-by: Sam Sun Closes: https://lore.kernel.org/lkml/CAEkJfYMtSdM5HceNsXUDf5haghD5+o2e7Qv4OcuruL4tPg6OaQ@mail.gmail.com/ Signed-off-by: Yosry Ahmed Reported-by: syzbot Signed-off-by: Andrew Morton --- mm/zswap.c | 31 ++++++++++++++++++++++++++++--- 1 file changed, 28 insertions(+), 3 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index f6316b66fb236..add1406d693b8 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -864,12 +864,22 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node) return ret; } +DEFINE_STATIC_SRCU(acomp_srcu); + static int zswap_cpu_comp_dead(unsigned int cpu, struct hlist_node *node) { struct zswap_pool *pool = hlist_entry(node, struct zswap_pool, node); struct crypto_acomp_ctx *acomp_ctx = per_cpu_ptr(pool->acomp_ctx, cpu); if (!IS_ERR_OR_NULL(acomp_ctx)) { + /* + * Even though the acomp_ctx should not be currently in use on + * @cpu, it may still be used by compress/decompress operations + * that started on @cpu and migrated to a different CPU. Wait + * for such usages to complete, any news usages would be a bug. + */ + synchronize_srcu(&acomp_srcu); + if (!IS_ERR_OR_NULL(acomp_ctx->req)) acomp_request_free(acomp_ctx->req); if (!IS_ERR_OR_NULL(acomp_ctx->acomp)) @@ -880,6 +890,18 @@ static int zswap_cpu_comp_dead(unsigned int cpu, struct hlist_node *node) return 0; } +static struct crypto_acomp_ctx *acomp_ctx_get_cpu(struct crypto_acomp_ctx __percpu *acomp_ctx, + int *srcu_idx) +{ + *srcu_idx = srcu_read_lock(&acomp_srcu); + return raw_cpu_ptr(acomp_ctx); +} + +static void acomp_ctx_put_cpu(int srcu_idx) +{ + srcu_read_unlock(&acomp_srcu, srcu_idx); +} + static bool zswap_compress(struct page *page, struct zswap_entry *entry, struct zswap_pool *pool) { @@ -889,12 +911,12 @@ static bool zswap_compress(struct page *page, struct zswap_entry *entry, unsigned int dlen = PAGE_SIZE; unsigned long handle; struct zpool *zpool; + int srcu_idx; char *buf; gfp_t gfp; u8 *dst; - acomp_ctx = raw_cpu_ptr(pool->acomp_ctx); - + acomp_ctx = acomp_ctx_get_cpu(pool->acomp_ctx, &srcu_idx); mutex_lock(&acomp_ctx->mutex); dst = acomp_ctx->buffer; @@ -950,6 +972,7 @@ static bool zswap_compress(struct page *page, struct zswap_entry *entry, zswap_reject_alloc_fail++; mutex_unlock(&acomp_ctx->mutex); + acomp_ctx_put_cpu(srcu_idx); return comp_ret == 0 && alloc_ret == 0; } @@ -958,9 +981,10 @@ static void zswap_decompress(struct zswap_entry *entry, struct folio *folio) struct zpool *zpool = entry->pool->zpool; struct scatterlist input, output; struct crypto_acomp_ctx *acomp_ctx; + int srcu_idx; u8 *src; - acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); + acomp_ctx = acomp_ctx_get_cpu(entry->pool->acomp_ctx, &srcu_idx); mutex_lock(&acomp_ctx->mutex); src = zpool_map_handle(zpool, entry->handle, ZPOOL_MM_RO); @@ -990,6 +1014,7 @@ static void zswap_decompress(struct zswap_entry *entry, struct folio *folio) if (src != acomp_ctx->buffer) zpool_unmap_handle(zpool, entry->handle); + acomp_ctx_put_cpu(srcu_idx); } /*********************************