From patchwork Thu Dec 19 21:24:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13915834 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7ACCEE77184 for ; Thu, 19 Dec 2024 21:24:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ED2946B007B; Thu, 19 Dec 2024 16:24:43 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E82636B0082; Thu, 19 Dec 2024 16:24:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D49EE6B0083; Thu, 19 Dec 2024 16:24:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id B49D46B007B for ; Thu, 19 Dec 2024 16:24:43 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 5FFD842C25 for ; Thu, 19 Dec 2024 21:24:43 +0000 (UTC) X-FDA: 82912986072.09.3937B05 Received: from mail-pg1-f201.google.com (mail-pg1-f201.google.com [209.85.215.201]) by imf05.hostedemail.com (Postfix) with ESMTP id 9EC6E100011 for ; Thu, 19 Dec 2024 21:23:37 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=G0BMeq3z; spf=pass (imf05.hostedemail.com: domain of 3GI9kZwoKCAs9z329lsxporzzrwp.nzxwty58-xxv6lnv.z2r@flex--yosryahmed.bounces.google.com designates 209.85.215.201 as permitted sender) smtp.mailfrom=3GI9kZwoKCAs9z329lsxporzzrwp.nzxwty58-xxv6lnv.z2r@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1734643456; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=u7O8GUIoD1UZG/rG8KBOAiEwEk9YL8KfYsJ7/2RZcq0=; b=XGidznFAYgMz3M1d09Aqher7hGOVgHJgmifVj8Ra9j0dEUXxgRkDnJK0hvL8Vf1ZZvZ4VM foCUab6cmRilo6lgmRKF7hvMSk6uboYrbqZQfwm1pL53GW9MwtogMqYV2GQPnwQB+ujxQ5 PWWbbsuBarYd8ZskCHUtRrrTGYlGav8= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=G0BMeq3z; spf=pass (imf05.hostedemail.com: domain of 3GI9kZwoKCAs9z329lsxporzzrwp.nzxwty58-xxv6lnv.z2r@flex--yosryahmed.bounces.google.com designates 209.85.215.201 as permitted sender) smtp.mailfrom=3GI9kZwoKCAs9z329lsxporzzrwp.nzxwty58-xxv6lnv.z2r@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1734643456; a=rsa-sha256; cv=none; b=ClcFPtgudBLotrqPSuOXIWItmGZg2EsuOBM5NOzgcZzBCdtHDyLitQseMkuEErWd9vX8GI PydMs6cTLVRBgywV0CGsTSw5y/UCecdVofsvtSH0FmfKEVRfkQBVyEHiTQOUl/FSE/DnLh 96o1PypP5aevZivTLSJM+o3XAJ9SQdE= Received: by mail-pg1-f201.google.com with SMTP id 41be03b00d2f7-7eaac1e95ffso776980a12.2 for ; Thu, 19 Dec 2024 13:24:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1734643480; x=1735248280; darn=kvack.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=u7O8GUIoD1UZG/rG8KBOAiEwEk9YL8KfYsJ7/2RZcq0=; b=G0BMeq3zoyHz1xeOKQgI05h0vwop/nkMHqXZjgOVd8yCqKhxvgOK0Ncn9IMRutyI27 wr3cNP9ra9AEvAa5bNnkBvjFdMD4MJQk4cchpAsurLkUq4a6Xbj30xjCxfo8hh8VLVWE oUguTi6frSeVrrdBGo+QeuYWCdIvX3TpEp5ayRvs36ZScyL2A0VNH+iFpeeW1AF69OGU T7OxSNFA0YNugRiY0rPFanK5Czk/pMcXfvcBxnhntchOY6xbKWlvuC7Wng+4gbUa/6oS Kuj2ul0i+8r3GVhkFGh4r7yWu4+lvzb6akUnQ/R7Nn0HxB+EwGdi3B2kD30pAwgUdKoM +MJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734643480; x=1735248280; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=u7O8GUIoD1UZG/rG8KBOAiEwEk9YL8KfYsJ7/2RZcq0=; b=cRdQSWxLzCdNz7FRIQa51VyDZkyjDn4F4mAnmKBystcXIW2uREEwvax3FCerVihmgo 3cZkF2Qkt9M+Jl3U35gBmXlqGNh1FGhXfzRHt7Qp4V439YU6L+FdET0Myhn+MDq8rMg8 +IRkUieyzvPxGtigBOzgi8uHS5i9EhMXpy3/xLH7vqRWo7KBa1YNDUVnp48toes9FXFy dbKCOKBQLn8JW07ZXimGmABTDIpPNH6yHtvUTfaNxzz3fe7+m4vi6IvA3pKIhJCYl0OL P1/0e4JVP4brRa5q79au6lYZHBnOydxebr8FEd9822uyQsO+QV6mOtGZILcWLamJg0wm QMEQ== X-Forwarded-Encrypted: i=1; AJvYcCXzvBbZn8Beu7HrqO6U5c7GjxkgT63rF8MenJ6tPYyQL2bY5YHTZNddCuzFXj7pNmjXU6+Tapl8UA==@kvack.org X-Gm-Message-State: AOJu0Ywqj0bXxfybL1QozuzfaNt5GrZ5D5HaCnYRxsgfC7S4divmKZhW /fgdLAPND/DUowMr/KVHyietDz/u0xTFtN5UIXN17aONdedOzSp8bsAdjZ8BAsJFBcY4mJIYH8c ZmqwBvVHHsL6Z/YWEAw== X-Google-Smtp-Source: AGHT+IFnH6UQrf3C3P3rhQlihyUOrX9sGtuPFMiRoQuLliycrZpnMFxS2E+N7rAXPa1MSWZDj7o1ChPfPBNiASrl X-Received: from pgbfu11.prod.google.com ([2002:a05:6a02:4a8b:b0:7fd:5569:7b79]) (user=yosryahmed job=prod-delivery.src-stubby-dispatcher) by 2002:a05:6a20:7487:b0:1e1:ab63:c5ed with SMTP id adf61e73a8af0-1e5e048b22emr887197637.23.1734643480234; Thu, 19 Dec 2024 13:24:40 -0800 (PST) Date: Thu, 19 Dec 2024 21:24:37 +0000 Mime-Version: 1.0 X-Mailer: git-send-email 2.47.1.613.gc27f4b7a9f-goog Message-ID: <20241219212437.2714151-1-yosryahmed@google.com> Subject: [PATCH] mm: zswap: fix race between [de]compression and CPU hotunplug From: Yosry Ahmed To: Andrew Morton Cc: Johannes Weiner , Nhat Pham , Chengming Zhou , Vitaly Wool , Barry Song , Sam Sun , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yosry Ahmed , stable@vger.kernel.org X-Rspamd-Queue-Id: 9EC6E100011 X-Rspamd-Server: rspam12 X-Stat-Signature: cf31srm7gth1ipceyfopmo6aptw9k13d X-Rspam-User: X-HE-Tag: 1734643417-237470 X-HE-Meta: U2FsdGVkX1+D1xc0Tv1AYkXo43YYZCgmn5kYf0OO1ahf0TAVgcp5aVxPWpq2mPSddYo5FII3MKHsDigKJk4pYwNm+EVMlDDU8BnDrJEHccO8+aGNt3p/zPzASmgstsZO1s1d4LOgQ1GvVXQWzvDUBX7eWxwTA7KoYbgFHodkB+EVXIDdMl7OYvgANBiYn2p2BxfqJAia3l7KLhCdoZAScs188VHhVh8sfW3J+0sdPpDWGuWWmym6cm89DLswhg4mUFE8LihBDjoH1X2IEQhfzEq1xhan4t6KV29VU2+U0C44HvuinMUvZATogPSF1P+WaO/dbM/Mw9ZvRaHiAODEkB626ROO4pyA6lboIAnImmZyZU7xB/t9PBETkPONeCHDJ/wHf/wxC2z81RV0YORiVCiD8VapCpuJmjKA8wsH+UTuyiiL5CLcNY/emmswQ/ztK9Ro9sACi16iF1le2mk4wqHkNuHQqiOXCgN1mEcoWOfJHOLOSeGkA87hsoaMzpI0wxrxNoA38dNALRwS6GdAlUrfCw7t3/5fPGUdgVVDR/kgCuEBulO4YIQnkR0VgyvbbWM16X+C3I7tOT66XSwZycJgsJE2hinifJL34lsj5ZvrH/9r0+J+qEvMNDLzeHfeCSCRdD6G1b61l6qgvMKYPJcI9IfSqlWbWvWgLOUtRkwj9IH0XeFX7/uTK2TPvzEf+ycGLkn8vUGWhw0ydn5GdLQktJFe112tmRiWZniYMhch0Mz8MQTiIYzi+j5VRERt7rYbhDAacMBn3TmQbSIqDygP5mfRppk/4pZ2ntBy3Ju8v2K9ddto9FrCsxBCNlJ93kWa8XhpAVrTGSi9F0SafsW/yASuKqf2CZteYppRZOtKw1U/jnxuhlYqKr68g+ISslp92ZX1l7LAxNEenFShShlsha2r9tjSvHFdxRxSMuJI69T8zv94ktUzCH+jeCYoK7kDduF57YtNFEPq0qc XA7XkQRW SAcFaXdHJpmvdF/6eOfuLl9MLuviJYdwfnGRLnNukIRGvTapbm5yy10ufBSoqsp1X5Ca5GDrcM+Jupq/GBeXviVyMiiygBvRqh81wNxAuaAVCtu4CYD3mUVlRHgy/CXu/mUw3p3RrZOB8YrOHvDc4I1rVi27UYk1nCOYfthJHPspQyVsZUowvA9Qc+HcjY8ONYLJG7AXX0vl+2OlYjo7Cn0VgJFt8RIwBYw6BnC3ZXuu0RZr9Gi4TbEOogNLCKy/JnHz7F9yTVkfu67tGqN2/wx0UJHSSOFQGeQgMP/D+LL5UfLSFJoDYVtve+PJb3+DVkYpTOl6OLoHUF5RsPKPWq00UFK82fYvCnEaLg0pN6aHmQM8KA7EGnxl5IsiHA8eWy9mS//WFX9h1drgi+aQZW8x9T5jSCYL9Xw/DBmQGLAfZjCEdaBKsSLCZ8JQ2y4+0FDNYsn2K0c+7Thanksqs+h/oA2lO4QDM2OZJPtcIg/R+ul2NBtdCzZn6Jsn7Rn0RIjXXOwwH6bjMChXUQ2DDS6jmuy+zVCKCCIVJTRwHQ1ahj/lBhqjR0RZEdL0wi7Qrdc3KEN/83aftwXRL7SuDxN+116aGp8tGXDY04ftP93dHkq39CeSuXxnDdrduGk1lhquem/JUAEgnUzBCyn92soQwEFX+KdjA3bE7W7fPMinvTqgkJVM+LAEt/g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In zswap_compress() and zswap_decompress(), the per-CPU acomp_ctx of the current CPU at the beginning of the operation is retrieved and used throughout. However, since neither preemption nor migration are disabled, it is possible that the operation continues on a different CPU. If the original CPU is hotunplugged while the acomp_ctx is still in use, we run into a UAF bug as the resources attached to the acomp_ctx are freed during hotunplug in zswap_cpu_comp_dead(). The problem was introduced in commit 1ec3b5fe6eec ("mm/zswap: move to use crypto_acomp API for hardware acceleration") when the switch to the crypto_acomp API was made. Prior to that, the per-CPU crypto_comp was retrieved using get_cpu_ptr() which disables preemption and makes sure the CPU cannot go away from under us. Preemption cannot be disabled with the crypto_acomp API as a sleepable context is needed. Commit 8ba2f844f050 ("mm/zswap: change per-cpu mutex and buffer to per-acomp_ctx") increased the UAF surface area by making the per-CPU buffers dynamic, adding yet another resource that can be freed from under zswap compression/decompression by CPU hotunplug. There are a few ways to fix this: (a) Add a refcount for acomp_ctx. (b) Disable migration while using the per-CPU acomp_ctx. (c) Disable CPU hotunplug while using the per-CPU acomp_ctx by holding the CPUs read lock. Implement (c) since it's simpler than (a), and (b) involves using migrate_disable() which is apparently undesired (see huge comment in include/linux/preempt.h). Fixes: 1ec3b5fe6eec ("mm/zswap: move to use crypto_acomp API for hardware acceleration") Reported-by: Johannes Weiner Closes: https://lore.kernel.org/lkml/20241113213007.GB1564047@cmpxchg.org/ Reported-by: Sam Sun Closes: https://lore.kernel.org/lkml/CAEkJfYMtSdM5HceNsXUDf5haghD5+o2e7Qv4OcuruL4tPg6OaQ@mail.gmail.com/ Cc: Signed-off-by: Yosry Ahmed Acked-by: Barry Song Reviewed-by: Chengming Zhou --- mm/zswap.c | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index f6316b66fb236..5a27af8d86ea9 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -880,6 +880,18 @@ static int zswap_cpu_comp_dead(unsigned int cpu, struct hlist_node *node) return 0; } +/* Prevent CPU hotplug from freeing up the per-CPU acomp_ctx resources */ +static struct crypto_acomp_ctx *acomp_ctx_get_cpu(struct crypto_acomp_ctx __percpu *acomp_ctx) +{ + cpus_read_lock(); + return raw_cpu_ptr(acomp_ctx); +} + +static void acomp_ctx_put_cpu(void) +{ + cpus_read_unlock(); +} + static bool zswap_compress(struct page *page, struct zswap_entry *entry, struct zswap_pool *pool) { @@ -893,8 +905,7 @@ static bool zswap_compress(struct page *page, struct zswap_entry *entry, gfp_t gfp; u8 *dst; - acomp_ctx = raw_cpu_ptr(pool->acomp_ctx); - + acomp_ctx = acomp_ctx_get_cpu(pool->acomp_ctx); mutex_lock(&acomp_ctx->mutex); dst = acomp_ctx->buffer; @@ -950,6 +961,7 @@ static bool zswap_compress(struct page *page, struct zswap_entry *entry, zswap_reject_alloc_fail++; mutex_unlock(&acomp_ctx->mutex); + acomp_ctx_put_cpu(); return comp_ret == 0 && alloc_ret == 0; } @@ -960,7 +972,7 @@ static void zswap_decompress(struct zswap_entry *entry, struct folio *folio) struct crypto_acomp_ctx *acomp_ctx; u8 *src; - acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); + acomp_ctx = acomp_ctx_get_cpu(entry->pool->acomp_ctx); mutex_lock(&acomp_ctx->mutex); src = zpool_map_handle(zpool, entry->handle, ZPOOL_MM_RO); @@ -990,6 +1002,7 @@ static void zswap_decompress(struct zswap_entry *entry, struct folio *folio) if (src != acomp_ctx->buffer) zpool_unmap_handle(zpool, entry->handle); + acomp_ctx_put_cpu(); } /*********************************