From patchwork Mon Jan 6 23:52:46 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13927901 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2B561E77188 for ; Mon, 6 Jan 2025 23:52:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 993DF6B00BC; Mon, 6 Jan 2025 18:52:55 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 91A3C6B00BD; Mon, 6 Jan 2025 18:52:55 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7BB376B00BE; Mon, 6 Jan 2025 18:52:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 5BC556B00BC for ; Mon, 6 Jan 2025 18:52:55 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id DDEF41C6C52 for ; Mon, 6 Jan 2025 23:52:54 +0000 (UTC) X-FDA: 82978679868.09.EC9E4B9 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) by imf06.hostedemail.com (Postfix) with ESMTP id 2757518000E for ; Mon, 6 Jan 2025 23:52:52 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=gOG9EjR9; spf=pass (imf06.hostedemail.com: domain of 302x8ZwoKCOAaQUTaCJOGFIQQING.EQONKPWZ-OOMXCEM.QTI@flex--yosryahmed.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=302x8ZwoKCOAaQUTaCJOGFIQQING.EQONKPWZ-OOMXCEM.QTI@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736207573; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=u7O8GUIoD1UZG/rG8KBOAiEwEk9YL8KfYsJ7/2RZcq0=; b=QmiH8CbLUMlagTK5acA/Vc/ZOfG2GNPkrLTyS1az/CayG1/gASpIP9WGvGjW627y2S3N+/ Yy1Uuj+ZC8echpviYX+++uZ0pqHjT9OY8kRV23AXA4YkDPbODiWPVaDzkY7bhv4Ps1XjEB ISTSHmmPV9vozLI1A+ewzs4RVSFFElY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736207573; a=rsa-sha256; cv=none; b=WQp5Hj3K9N7GKAiJ0bxV4MiCWFd6KfQWRj1VlIMsqSHfBz95EgrMQOKujw5C61U6QwwrTn 1YKc6T15jg7Eg9r8klXjhJgKXDqYoR+sQrGeONmXAOtYV8mruhkgtcZ+NoDKH2TJS6qtb/ Vinla7Y5hPys1+EKvnAWupBkTJHckYY= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=gOG9EjR9; spf=pass (imf06.hostedemail.com: domain of 302x8ZwoKCOAaQUTaCJOGFIQQING.EQONKPWZ-OOMXCEM.QTI@flex--yosryahmed.bounces.google.com designates 209.85.216.73 as permitted sender) smtp.mailfrom=302x8ZwoKCOAaQUTaCJOGFIQQING.EQONKPWZ-OOMXCEM.QTI@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-pj1-f73.google.com with SMTP id 98e67ed59e1d1-2ef9204f898so20994532a91.2 for ; Mon, 06 Jan 2025 15:52:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1736207572; x=1736812372; darn=kvack.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=u7O8GUIoD1UZG/rG8KBOAiEwEk9YL8KfYsJ7/2RZcq0=; b=gOG9EjR9Q9Web/Nk2isd3jPi174cPEMHDC0C1Jn9dsa0wOtGhbs2E//cJ2p2mEefMn rd7YqV29PEI2yoAJQxMN5B54ia/kI8hT1YDTW7s35XuHi4jGTAR4/9FC0gM9p6levrPx amtVLCEQVmQ6IFayZoxp3oI2F+dlTFh6qKBfvBnySZn74JGgBURUib+0nDNJcwpPQFt5 oK4emFa5kvKIv8rPB53dxrFD4F0opd6Je7F3R+5qaglEs1FwVVgKHRGymqjAQ0c23Ygk X425OQWIns9pZviCZ16f9SLfIvH8A27qv+V8Z7wGTt5TcKzakRv+QsrHaNB+URVOsrEr Wrhw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736207572; x=1736812372; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=u7O8GUIoD1UZG/rG8KBOAiEwEk9YL8KfYsJ7/2RZcq0=; b=eWfWCZRWHOesO4ta5Xtp06QfCCPa1YfJbqWQ9M+j8hQltW647Z+IlIfTsqv+vlnw2P 6GlfMf3qSrR4BR1dmkcXVWNiWr6F4HLICmTmbqwsMNRnJiAnkNpS4zObupP/gJR/wInM CV+uVxt/cl9InvDIcVEGseAAZE9MLtILTv0JYJwwgZj2/yrgQUDPUg3lFafKgtKXAZV8 EthkioxUAHg5Oo1iC3GNYpbKXhaLpkVaUYY357tTCyyKjY/bNdmygidXKHClV3RSSMus MwbPwLw5mC99cjzM7S4/3cYwn3zDYLU0O9eK8zMEX5lAhi1XjnnzMtb1UrQBZlA7zL6k oxLw== X-Forwarded-Encrypted: i=1; AJvYcCUTNAXED/kZHMo7dE6hjn9Qrby7+noZ4sE/tuaalCCYKO1gN10H7rrwueSjoixvLgj2j5VQQaEGxw==@kvack.org X-Gm-Message-State: AOJu0YxnnWAd5rdeYMo8UFPckZ+aFnyBwGZJToCwLbMSPaePDhQa2vKj jJvgbxB+NFknDaiIk6+xSruPhEpdmEUgWlwY1F04zm0UQzdfG3ROpmtqC197seWa2RpupJpso6Y LcyvgwJK/gwIH4x6byg== X-Google-Smtp-Source: AGHT+IEuLrc6Y2QxDWJppEIUEse6ofoa305x2MBQtNxB8/VQXUzNW5wB0Z45mFMFJNRnuQnoIvYGpDktiR5T4yjm X-Received: from pjh16.prod.google.com ([2002:a17:90b:3f90:b0:2ea:5c73:542c]) (user=yosryahmed job=prod-delivery.src-stubby-dispatcher) by 2002:a17:90b:5347:b0:2ee:3fa7:ef4d with SMTP id 98e67ed59e1d1-2f452ec37bamr89128041a91.24.1736207571666; Mon, 06 Jan 2025 15:52:51 -0800 (PST) Date: Mon, 6 Jan 2025 23:52:46 +0000 Mime-Version: 1.0 X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog Message-ID: <20250106235248.1501064-1-yosryahmed@google.com> Subject: [PATCH] mm: zswap: fix race between [de]compression and CPU hotunplug From: Yosry Ahmed To: Andrew Morton Cc: Johannes Weiner , Nhat Pham , Chengming Zhou , Vitaly Wool , Barry Song , Sam Sun , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yosry Ahmed , stable@vger.kernel.org X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 2757518000E X-Stat-Signature: 7qrs4xjj33mcgdaiedge31sanhp8p64z X-Rspam-User: X-HE-Tag: 1736207572-904422 X-HE-Meta: U2FsdGVkX18YE3qKAaOPpTbYixlvaWcCOa6T5jqlkQEVEQUonkDSZqGuM9S0+x9tzecO+AjDbOzDnQ0nJTOcQsxWj6Kci/Vwie8M1/fGCbTjNiwGYgKRQahgPH/AIfg/kglC9H9jL3xGgRsFED2jChpOS5S+ljxZ+/lECzJMlO/8IWf5vVoFuETb/oVIh0hwMnJKy5oVK73s+yXkq66g2lmFWmXHLddI2bJ2NSeePFdDtK8+MuVM9Vm4+0UBe3SI9O5Jq4dZPZUN6gNivYcPb5I1OALXCASfLDlh1bOhOqMh6lScVSc6sHWtE5Ao/mO2Wt99OpgQxSugi+MCzO5mXBxe+PbhjYBAXHDc1PtdoqmlJu3SrTF6Dtpqd/6i6Qwmxvv/PCQwl7LNT48wQ8GMFl3YCgRCDLsrQUKpwlIQFUH1Hd40mm6EAQkRoCyU0RkDMNWHn7/pulE6O2OTilmVPwaQXlgbSIbm5AoLUOxMkH+g3UNJsbwT0BM0FDo9acsNYycyrlqhf+HFoIQb5QmVYbTUlIDYm/h1MpKoVsHemuzpNlbCC090k3JkX62oVtERKrR064RMNCc1om2M2vkzW3Ei9/FcMUYTFniuq3EXxNjHufh9YV7npepzYeW8rEwbYERd8ukbw/EZZN8RQDfa/XowtLygZxgHmXndAmzMzcgAMAnPMJPEN308nLkrIqgrEds53OujtdcdO9whTo08JshCULSG/7uqvh0MNH7SHixsgQqjupG/y3w3zYu1tGXTZYnn3sc5mhLimxatT1QgocblXWnnXMHULCQjQUv7KXpV3uJG2gbtSZuTv6t03Oq3BfEjfu3NKExQ1+WC+I79AYHuVeHoF8kuUz3QzU1NPCGGITj0x2Q4KuQJImSm6xGcQcgS4SKWAp9uPxP/r8Qo3G1idQzSGxquyKm0zIgkpMh81nJViFPEzNljcfX0wPsP4fTlBwaftqh1pZU1plw 6KfH8OKX EOl4H6xX6YBlwYdsdn9Ws5Br7dg0DK2r09YCzPWyFz4QFuPOj7bAalhda0bcBgrNlbi5N04GCF9PFya3ex2enPGiweHo1dEFGys+oZkXfYE61s/pcuFN0rtnvIxfVbThGfIiofa5EV0DS/YAIn9YL3nmGUhhCGE2iCCDOquWZyizfvQK43t/iVw6etUIXwG+8B7g72bS4xEVt+boBaGQtA6FYinzUEh6ZXD1OobpB0O+rGtN3Uh1WrrWmofffE3J3SVY1xLvJY445AC4YTiBW6FDGTi8p4ciVhHIJCXdBhN5MWiIzf/baboMVTtL8TtJ6GJjnV63OH1WbZNbubhnfeB8IaI9Siy4ycbt4v0ik5lDun/CbZdpHJPzG0HLVKR5vkdPe6+T1pnnbvo3W4vmBV3azDv62siNhvqULxoj4Ofra5JL+cpHmNVTbrOZR++YrUZSiEcjQmDgcq3EdEw3DNiunf6MgN33yqn4xQwi6OXfbUEkcAH1le7YpZUHZ0ek8ZUTUSZYxFRltpnjyUdZhawf5x+NQ5WUQVcZI5huRgEh/cvQIT/HBXFhRLB4qi64Tl9jx5rfOHqMZJA2R6qCjYXXBeAmzSYw4zi/gtkJczPmCpcNmRE+vX1Y8+YmYkYvy3+47bzJWtlAD9cPbDPzodTc8NMfK3DISQ/Ao41nOGK1VJxNLnMowdZoliw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In zswap_compress() and zswap_decompress(), the per-CPU acomp_ctx of the current CPU at the beginning of the operation is retrieved and used throughout. However, since neither preemption nor migration are disabled, it is possible that the operation continues on a different CPU. If the original CPU is hotunplugged while the acomp_ctx is still in use, we run into a UAF bug as the resources attached to the acomp_ctx are freed during hotunplug in zswap_cpu_comp_dead(). The problem was introduced in commit 1ec3b5fe6eec ("mm/zswap: move to use crypto_acomp API for hardware acceleration") when the switch to the crypto_acomp API was made. Prior to that, the per-CPU crypto_comp was retrieved using get_cpu_ptr() which disables preemption and makes sure the CPU cannot go away from under us. Preemption cannot be disabled with the crypto_acomp API as a sleepable context is needed. Commit 8ba2f844f050 ("mm/zswap: change per-cpu mutex and buffer to per-acomp_ctx") increased the UAF surface area by making the per-CPU buffers dynamic, adding yet another resource that can be freed from under zswap compression/decompression by CPU hotunplug. There are a few ways to fix this: (a) Add a refcount for acomp_ctx. (b) Disable migration while using the per-CPU acomp_ctx. (c) Disable CPU hotunplug while using the per-CPU acomp_ctx by holding the CPUs read lock. Implement (c) since it's simpler than (a), and (b) involves using migrate_disable() which is apparently undesired (see huge comment in include/linux/preempt.h). Fixes: 1ec3b5fe6eec ("mm/zswap: move to use crypto_acomp API for hardware acceleration") Reported-by: Johannes Weiner Closes: https://lore.kernel.org/lkml/20241113213007.GB1564047@cmpxchg.org/ Reported-by: Sam Sun Closes: https://lore.kernel.org/lkml/CAEkJfYMtSdM5HceNsXUDf5haghD5+o2e7Qv4OcuruL4tPg6OaQ@mail.gmail.com/ Cc: Signed-off-by: Yosry Ahmed --- mm/zswap.c | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index f6316b66fb236..5a27af8d86ea9 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -880,6 +880,18 @@ static int zswap_cpu_comp_dead(unsigned int cpu, struct hlist_node *node) return 0; } +/* Prevent CPU hotplug from freeing up the per-CPU acomp_ctx resources */ +static struct crypto_acomp_ctx *acomp_ctx_get_cpu(struct crypto_acomp_ctx __percpu *acomp_ctx) +{ + cpus_read_lock(); + return raw_cpu_ptr(acomp_ctx); +} + +static void acomp_ctx_put_cpu(void) +{ + cpus_read_unlock(); +} + static bool zswap_compress(struct page *page, struct zswap_entry *entry, struct zswap_pool *pool) { @@ -893,8 +905,7 @@ static bool zswap_compress(struct page *page, struct zswap_entry *entry, gfp_t gfp; u8 *dst; - acomp_ctx = raw_cpu_ptr(pool->acomp_ctx); - + acomp_ctx = acomp_ctx_get_cpu(pool->acomp_ctx); mutex_lock(&acomp_ctx->mutex); dst = acomp_ctx->buffer; @@ -950,6 +961,7 @@ static bool zswap_compress(struct page *page, struct zswap_entry *entry, zswap_reject_alloc_fail++; mutex_unlock(&acomp_ctx->mutex); + acomp_ctx_put_cpu(); return comp_ret == 0 && alloc_ret == 0; } @@ -960,7 +972,7 @@ static void zswap_decompress(struct zswap_entry *entry, struct folio *folio) struct crypto_acomp_ctx *acomp_ctx; u8 *src; - acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); + acomp_ctx = acomp_ctx_get_cpu(entry->pool->acomp_ctx); mutex_lock(&acomp_ctx->mutex); src = zpool_map_handle(zpool, entry->handle, ZPOOL_MM_RO); @@ -990,6 +1002,7 @@ static void zswap_decompress(struct zswap_entry *entry, struct folio *folio) if (src != acomp_ctx->buffer) zpool_unmap_handle(zpool, entry->handle); + acomp_ctx_put_cpu(); } /*********************************