From patchwork Wed Feb 12 06:26:59 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13971029 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2E0DEC02198 for ; Wed, 12 Feb 2025 06:32:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6A5756B0085; Wed, 12 Feb 2025 01:32:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 62FE46B0088; Wed, 12 Feb 2025 01:32:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4B401280001; Wed, 12 Feb 2025 01:32:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 2AB996B0085 for ; Wed, 12 Feb 2025 01:32:11 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 927381A0E8B for ; Wed, 12 Feb 2025 06:32:10 +0000 (UTC) X-FDA: 83110322820.16.1D24992 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) by imf25.hostedemail.com (Postfix) with ESMTP id A9BB7A0008 for ; Wed, 12 Feb 2025 06:32:08 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=JUYJl8F1; spf=pass (imf25.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.169 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739341928; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Nbviwh7jX+fyZgx+5zWo4yndOTBdlLf1Q5tRKthE6k8=; b=ltxbY3lgZmXPHFCCm+sS+EIuLjDQpws62bIXejzMUJArrslkUcg/mJMhdyOVFDkOVK2gn8 N4k4T4ldWF9TS6+QGfIkIbGiRn2UgXliG+z8fyCWvioKw6MML6XQ4UdlILxJxSgKI0kvLy BmDeOA1Xq+DbvgacZRuk9JKFfKm9x3I= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739341928; a=rsa-sha256; cv=none; b=jWmMl9oXulVVtqnHE51h1NVtIoVjs7RXH0HbiAQjXQ6XEINKBPU9iwn0L0Rnc0MW9r7Hoh DWXK+2qmCQoN4fxgB8R3jqwISItIb40tvZcSrUXBRI1nS5DmUi2xk0ix3XBfIthYuvzw9J jTipttWVxn/CYSXdsefn3NzTyIqmYDk= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=JUYJl8F1; spf=pass (imf25.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.169 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-21f92258aa6so81130125ad.3 for ; Tue, 11 Feb 2025 22:32:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739341927; x=1739946727; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Nbviwh7jX+fyZgx+5zWo4yndOTBdlLf1Q5tRKthE6k8=; b=JUYJl8F1TGBHIzcetpxsg0Qg1tOWY7H+e6oQnR+U1AMz6c3QKXDiPBdmZJXRq5XsdI 9NuAmXHMl6s4TS7N55jBEk8dv2rV0ehHO0EtvS9gSjCBSB+0BrXWGQ8hWomsTteg4jgh 3r7mx2iighfat5TDoCVeXAXH2bea0qNAdEr8w= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739341927; x=1739946727; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Nbviwh7jX+fyZgx+5zWo4yndOTBdlLf1Q5tRKthE6k8=; b=gWjRRFdjSc5X/1FuhTSUzYWYOqlg0tbipVJgURb0frF3vzAHfm79xiPE0DTc8WlGDa EmwTiqK6+k4Hrjnc/7p9lEHaBfXxQF4L/DEVyIBdt8Bv5E6hCenf6IFr+RYzWt+QAosm UPI9qil8Cj57j/kWTmh+jIDRQeWh+DVBxPDvEcUoHkMcxClixxK6y+fknu9uxWIH6jlu DDgZwyGBx7A6KhaKEtAGNCwtgZ3kgOZcHweKyOkdkA3zQdEHShpZeETw1WqpKzkvBXjf NLtYzdWmHaNSs0eJwbubLd7ACmAZyf9TwDABoBZDCdzhriSdftzgJ+QKsVWIAOqcMkwo H0xA== X-Forwarded-Encrypted: i=1; AJvYcCU0JMG14efDEGFX5CAzcl9dMKJInnERttLe9YhPriv8XiDe7rk010UZ1na4Shn2NnvJMjpI027/tg==@kvack.org X-Gm-Message-State: AOJu0YzFwoqeHMYWk2XDAoYni0XfkLJ13o+pf4FnZ5qmxaE9/kxP41Lb yUXCYlMqynmnz9cGMY3IrSgVUz9lFyBWjH7QOsWgQxKwgpvhSzgnzZ6zvnTQwQ== X-Gm-Gg: ASbGncuJnJyOKXuWoHXV7pwDlLgITsefutvCBdMXo3AWOl+01tedCj15WsiwicieVxn GtkdyfJ6qWe5QB+lscMCPcjz0OJtHJctFBQWkGLy0GwKZVrUQIj3E6GBLTcfBNMcstviv0tsuhS pdffnfifS3HYGfjV2c4GLiWOr0nVWH7Y4oScek5AqRmzxLHS2Zp8v+CLcxRtGcoyqYnwSkbbVaJ DIUVccqN/zlDitu1QaDx7o0XV5amMgOwpohA7R2SDt/6vQzdVD/9u8BKwFfJWZRqGv80ooDY+og 1hFgKuxv2QN5KrhNNg== X-Google-Smtp-Source: AGHT+IFSvyC57xadlYDsJ2pjEf3GwKRx0RFUmqWoHBxBXjtPHZ1A+qKCAZJUZUiYg24snMECPbVRZA== X-Received: by 2002:a17:902:ea11:b0:216:46f4:7e3d with SMTP id d9443c01a7336-220bbad6f94mr31042255ad.15.1739341925761; Tue, 11 Feb 2025 22:32:05 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:69f5:6852:451e:8142]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-220c34f443bsm2839685ad.5.2025.02.11.22.32.03 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 11 Feb 2025 22:32:05 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v5 01/18] zram: sleepable entry locking Date: Wed, 12 Feb 2025 15:26:59 +0900 Message-ID: <20250212063153.179231-2-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog In-Reply-To: <20250212063153.179231-1-senozhatsky@chromium.org> References: <20250212063153.179231-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: A9BB7A0008 X-Rspamd-Server: rspam07 X-Stat-Signature: mcjqxg1xagyh5ks5ngu6ixk4x36oyr98 X-HE-Tag: 1739341928-182063 X-HE-Meta: U2FsdGVkX19V9DTktW5YU0aGVECqZsmZqu8c0hcBIFhV2jDOvEk4xUa7+LxpDFCuLteupP7BSgVdQzenvvybYFjwtzPWPs/q0PQeLMODF1gz2WLKYGbDNr44EQJy/ED0XYvUS6Q8lMrIPDaTM6DFcwGOaq9Y+gcjG5M/K+xbKvIaFlIunooFjRa4LHsXXMM55ONFv4V1syVJ/LMk3cOFiPi5q6Z3xXLR2xXL+/5YiZ5V28SyDMG16zXplN1ByVF69GhtOEo4Cx153GVHAwU25enU97QMiIBqky1C2/Mh4oE2RhqpGCwqt3hX9Ojl+X/Va4Arn+TMSE2QxYUSIHLXSyv4hp1bI+N04DVB73PlH2S0spxEhtpENs5W2v+6ZglYq7sbBTEPfGWfhiEieIgMlrdOaIn/6ILFNt4PUjVwXyz2ehSJe9krMXig9kKPZ4MMf2D8xML222cgZ95onuM/wlW0lPJMPftm/obnzXBU1II1M+7LzRZ4oO73pffIZDkjtv88cTWJC8FZRAEBbDJiGL8z/lVZptt8Aol05zhECkM/HTpbXvgxsPFoyd7v9D3oYxGOkO/fxIpUu6OnjT07VT6HbAq0DihnIiyqQMUJnF1E2Sj+lB7EMTg4x8Cvs+t6IReW8K6MetWhrSzuZD7fR7bNrBuGVbeIYcmL21UytJ4M1AtaMrciTlvPdFuAm3YCZ+IW5+MN7JMtDwL1ers/8EFF70x2shoqe8G5YTEej2OwcOEBkvljPFmawhj/2qcfq59DzIZYoGkxiyYwOZy3zTvZtVD3RYLYRYCkr3p2+ed4M4rnWfBc1Btk58I/7+BX7qWhBHxBeEs/T3Xdvx//9My32QW4QNaLf9QXpzpCqJFTlHjznMefJJUBhtfLX6MVBG4JJye5AIKdD06Yd3HpZuUseEkwp1SJtK2OB1mCxvmYrc4NbZ+eGM6KOuWLv/YrEcnX0LdBs2dlrKFY2VP +HgoNam4 1Qlrt71LiNTLNALaCLnHxIxlLD+zq29aYgJXcmWnIJbiqB5crs98iAyMmK5++PaRFX3VeKuTJa74G9JJzVQQDV9pqP0E0OWSDVcBOoVAl8YTIjdAOFCyOyGxfaNt95LXntShD6nemYSuNnsn4T1BEfzqAOV3HWxRSZ0J+D3+ETq7t4ygkkiXvd5b8tp2w0Fisd8SUu9D2LeiZkSLVgAdiwKiHw1ukRIAVhZnOroneGYftpdT+xI6bcSHiKPGJMozU2F4eSUX0weqvpEM1kQJptg/rLfuItcgir464ys/JW66RenWhfPRQYj6U0A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Concurrent modifications of meta table entries is now handled by per-entry spin-lock. This has a number of shortcomings. First, this imposes atomic requirements on compression backends. zram can call both zcomp_compress() and zcomp_decompress() under entry spin-lock, which implies that we can use only compression algorithms that don't schedule/sleep/wait during compression and decompression. This, for instance, makes it impossible to use some of the ASYNC compression algorithms (H/W compression, etc.) implementations. Second, this can potentially trigger watchdogs. For example, entry re-compression with secondary algorithms is performed under entry spin-lock. Given that we chain secondary compression algorithms and that some of them can be configured for best compression ratio (and worst compression speed) zram can stay under spin-lock for quite some time. Having a per-entry mutex (or, for instance, a rw-semaphore) significantly increases sizeof() of each entry and hence the meta table. Therefore entry locking returns back to bit locking, as before, however, this time also preempt-rt friendly, because if waits-on-bit instead of spinning-on-bit. Lock owners are also now permitted to schedule, which is a first step on the path of making zram non-atomic. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 65 ++++++++++++++++++++++++++++------- drivers/block/zram/zram_drv.h | 20 +++++++---- 2 files changed, 67 insertions(+), 18 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 9f5020b077c5..3708436f1d1f 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -58,19 +58,57 @@ static void zram_free_page(struct zram *zram, size_t index); static int zram_read_from_zspool(struct zram *zram, struct page *page, u32 index); -static int zram_slot_trylock(struct zram *zram, u32 index) +static void zram_slot_lock_init(struct zram *zram, u32 index) { - return spin_trylock(&zram->table[index].lock); +#ifdef CONFIG_DEBUG_LOCK_ALLOC + lockdep_init_map(&zram->table[index].lockdep_map, "zram-entry->lock", + &zram->table_lockdep_key, 0); +#endif +} + +/* + * entry locking rules: + * + * 1) Lock is exclusive + * + * 2) lock() function can sleep waiting for the lock + * + * 3) Lock owner can sleep + * + * 4) Use TRY lock variant when in atomic context + * - must check return value and handle locking failers + */ +static __must_check bool zram_slot_try_lock(struct zram *zram, u32 index) +{ + unsigned long *lock = &zram->table[index].flags; + + if (!test_and_set_bit_lock(ZRAM_ENTRY_LOCK, lock)) { +#ifdef CONFIG_DEBUG_LOCK_ALLOC + mutex_acquire(&zram->table[index].lockdep_map, 0, 1, _RET_IP_); +#endif + return true; + } + return false; } static void zram_slot_lock(struct zram *zram, u32 index) { - spin_lock(&zram->table[index].lock); + unsigned long *lock = &zram->table[index].flags; + +#ifdef CONFIG_DEBUG_LOCK_ALLOC + mutex_acquire(&zram->table[index].lockdep_map, 0, 0, _RET_IP_); +#endif + wait_on_bit_lock(lock, ZRAM_ENTRY_LOCK, TASK_UNINTERRUPTIBLE); } static void zram_slot_unlock(struct zram *zram, u32 index) { - spin_unlock(&zram->table[index].lock); + unsigned long *lock = &zram->table[index].flags; + +#ifdef CONFIG_DEBUG_LOCK_ALLOC + mutex_release(&zram->table[index].lockdep_map, _RET_IP_); +#endif + clear_and_wake_up_bit(ZRAM_ENTRY_LOCK, lock); } static inline bool init_done(struct zram *zram) @@ -93,7 +131,6 @@ static void zram_set_handle(struct zram *zram, u32 index, unsigned long handle) zram->table[index].handle = handle; } -/* flag operations require table entry bit_spin_lock() being held */ static bool zram_test_flag(struct zram *zram, u32 index, enum zram_pageflags flag) { @@ -1473,15 +1510,11 @@ static bool zram_meta_alloc(struct zram *zram, u64 disksize) huge_class_size = zs_huge_class_size(zram->mem_pool); for (index = 0; index < num_pages; index++) - spin_lock_init(&zram->table[index].lock); + zram_slot_lock_init(zram, index); + return true; } -/* - * To protect concurrent access to the same index entry, - * caller should hold this table index entry's bit_spinlock to - * indicate this index entry is accessing. - */ static void zram_free_page(struct zram *zram, size_t index) { unsigned long handle; @@ -2321,7 +2354,7 @@ static void zram_slot_free_notify(struct block_device *bdev, zram = bdev->bd_disk->private_data; atomic64_inc(&zram->stats.notify_free); - if (!zram_slot_trylock(zram, index)) { + if (!zram_slot_try_lock(zram, index)) { atomic64_inc(&zram->stats.miss_free); return; } @@ -2625,6 +2658,10 @@ static int zram_add(void) if (ret) goto out_cleanup_disk; +#ifdef CONFIG_DEBUG_LOCK_ALLOC + lockdep_register_key(&zram->table_lockdep_key); +#endif + zram_debugfs_register(zram); pr_info("Added device: %s\n", zram->disk->disk_name); return device_id; @@ -2681,6 +2718,10 @@ static int zram_remove(struct zram *zram) */ zram_reset_device(zram); +#ifdef CONFIG_DEBUG_LOCK_ALLOC + lockdep_unregister_key(&zram->table_lockdep_key); +#endif + put_disk(zram->disk); kfree(zram); return 0; diff --git a/drivers/block/zram/zram_drv.h b/drivers/block/zram/zram_drv.h index db78d7c01b9a..63b933059cb6 100644 --- a/drivers/block/zram/zram_drv.h +++ b/drivers/block/zram/zram_drv.h @@ -28,7 +28,6 @@ #define ZRAM_SECTOR_PER_LOGICAL_BLOCK \ (1 << (ZRAM_LOGICAL_BLOCK_SHIFT - SECTOR_SHIFT)) - /* * ZRAM is mainly used for memory efficiency so we want to keep memory * footprint small and thus squeeze size and zram pageflags into a flags @@ -46,6 +45,7 @@ /* Flags for zram pages (table[page_no].flags) */ enum zram_pageflags { ZRAM_SAME = ZRAM_FLAG_SHIFT, /* Page consists the same element */ + ZRAM_ENTRY_LOCK, /* entry access lock bit */ ZRAM_WB, /* page is stored on backing_device */ ZRAM_PP_SLOT, /* Selected for post-processing */ ZRAM_HUGE, /* Incompressible page */ @@ -58,13 +58,18 @@ enum zram_pageflags { __NR_ZRAM_PAGEFLAGS, }; -/*-- Data structures */ - -/* Allocated for each disk page */ +/* + * Allocated for each disk page. We use bit-lock (ZRAM_ENTRY_LOCK bit + * of flags) to save memory. There can be plenty of entries and standard + * locking primitives (e.g. mutex) will significantly increase sizeof() + * of each entry and hence of the meta table. + */ struct zram_table_entry { unsigned long handle; - unsigned int flags; - spinlock_t lock; + unsigned long flags; +#ifdef CONFIG_DEBUG_LOCK_ALLOC + struct lockdep_map lockdep_map; +#endif #ifdef CONFIG_ZRAM_TRACK_ENTRY_ACTIME ktime_t ac_time; #endif @@ -137,5 +142,8 @@ struct zram { struct dentry *debugfs_dir; #endif atomic_t pp_in_progress; +#ifdef CONFIG_DEBUG_LOCK_ALLOC + struct lock_class_key table_lockdep_key; +#endif }; #endif From patchwork Wed Feb 12 06:27:00 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13971030 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 93574C0219E for ; Wed, 12 Feb 2025 06:32:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 23A026B0089; Wed, 12 Feb 2025 01:32:15 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1C387280001; Wed, 12 Feb 2025 01:32:15 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 014876B008C; Wed, 12 Feb 2025 01:32:14 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id D44176B0089 for ; Wed, 12 Feb 2025 01:32:14 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 8F2944B577 for ; Wed, 12 Feb 2025 06:32:14 +0000 (UTC) X-FDA: 83110322988.03.465DD4E Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) by imf01.hostedemail.com (Postfix) with ESMTP id 9FFFB4000F for ; Wed, 12 Feb 2025 06:32:12 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=K9F0UrpF; spf=pass (imf01.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.176 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739341932; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=j7H+x+6xpsdKzVL/D/QPxhemiaH/onPBYnSp5xsYcj0=; b=S9scDRjs9xQpSgAdmC31WJQSwrdJqGWJJ1R1aQimiwwDetdqfmnQAzeOzU+dJ61jC8CwYD BlSF/ZB9pv3x0nYm+VVX3W0lel5NsMftR0EL25EpetVMxTlq4LSTyVmyGHix42s9IiTBdn mvna86AumhZzOWGI7gljpfbBYom3vWA= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=K9F0UrpF; spf=pass (imf01.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.176 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739341932; a=rsa-sha256; cv=none; b=ZPZHh6tqsebSxK0J2iTIFirqxjVfWDBXQCIBNqv96AJqDSNgzNin50sMMjV9+gD0M1r7Pa pH7pFoxy5cOVZ1XBsVVjU1XKQyh9NsWtePCA++iBcn4Ed/LikMapdMWzFJm61+aG1SMozi qVoX7lbd1XUdUr1af0T8gXBNZaT66o8= Received: by mail-pl1-f176.google.com with SMTP id d9443c01a7336-21fa56e1583so45799575ad.3 for ; Tue, 11 Feb 2025 22:32:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739341931; x=1739946731; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=j7H+x+6xpsdKzVL/D/QPxhemiaH/onPBYnSp5xsYcj0=; b=K9F0UrpFHbLhfbG1PbviE5G1uB2u2E/+go0x9B/J0KNGg35u2ibTz6/hYec31wM5JV w4ssBuu9QGw5nk9AZyGa4rQRzhsoybf0SY6iZPf13G2Ff64axxU4u7cLO6HwULNtwt/+ 8vTVOQLwJGsBFMz7Wx0BxyM3J9KVkQ7tNAmLY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739341931; x=1739946731; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=j7H+x+6xpsdKzVL/D/QPxhemiaH/onPBYnSp5xsYcj0=; b=bFCVytqoC7GZM0XoRoCsnYX2dNSZmUR6RVybD+AJ+sQfgs6J3nPO6uAQ2gaf/qpFJ/ aoyliyhlDbUJQ/GFk46aoUUL0toDffcC84KPZAiabK02ttWIEUnyn/R4mwrD9xiMml5F v5UNMZouKafWT1Y26pGKsPWMiOjDrm+yC+Frqj6B0dcvluqLT7I/h500Iq/lNFrVIVG7 6tSX3LeZT9v9YvSQ2ao+1BnRDCXj1eWzYbtF3DkFo9VKwkEqpIzMOTYEA0z2Pi2jdVHS 8+Og1QBCYeQSuV3DIO3KlpyaNoZ9MMu/QAvePdolWRnn+0itNZdKGybxiuu/vc0KgQUV o9Kw== X-Forwarded-Encrypted: i=1; AJvYcCVg2WoUAAAk7rCQCMKZuRF5OIkSIb5hVPARH9Ag6uW2pEIFOHsUqTwHmUVAtfNk83rpjVrWNT0ukg==@kvack.org X-Gm-Message-State: AOJu0YxLxEUp0TAtUAZPykX/7dXGcvZRsp5FP8pJf+vdESXeqjmchCXU 1Y1sehxOo03nGvg2GxVJrEM8EmvmIfw+NN7fhlV6dlF+FX4iuwNUChesSlKXJA== X-Gm-Gg: ASbGncuzLnreb/Kg87AJsQGA2VHayCMy1Fqk/76fAWBeE7n1bahMVrQBcN11T4jNFyk KWpTVck4luV1tRqefNe4hB3k8a7uvH51yuBVTFq0b7byJQ+Fg66Fhn8DG4PjNdPKijailCEswez 08x0bVc3GdZHTrD4v7/8Ihj2TC9+BkghxTYIdaYNl9+YyF7j4LyudYk7Cv6kcDdYw3iBkmh0BEQ TzYOjx+t0Zvle3aDalUYkkesP2WV4uEepQNFBfKrqRlmjTo+8h5esnkszZrwqUi5pO66kdpa6GJ ZR5hQNvgAniLWEe87w== X-Google-Smtp-Source: AGHT+IHq2P1ROpgHywWa5M8Xpzdi4NwWEFRypribFsvZL+hUr50EoXZA3FxWT+dt8VM20Q3kk3TokQ== X-Received: by 2002:a17:903:2f86:b0:216:2259:a4bd with SMTP id d9443c01a7336-220bbca3a98mr26603225ad.52.1739341931537; Tue, 11 Feb 2025 22:32:11 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:69f5:6852:451e:8142]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-21f93c7e363sm48246805ad.94.2025.02.11.22.32.09 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 11 Feb 2025 22:32:11 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v5 02/18] zram: permit preemption with active compression stream Date: Wed, 12 Feb 2025 15:27:00 +0900 Message-ID: <20250212063153.179231-3-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog In-Reply-To: <20250212063153.179231-1-senozhatsky@chromium.org> References: <20250212063153.179231-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 9FFFB4000F X-Stat-Signature: pb4upwe4rxtutepn11qpmd93jkk1po5a X-Rspamd-Server: rspam03 X-HE-Tag: 1739341932-986618 X-HE-Meta: U2FsdGVkX18UEWZ4AsSRA1L1nJuF0ZKivlUGv2T0aqfMbGdV6BnlYDeyO3MHRiG8kIdVL+7u8I8XYbKhqeC+rKT1TiZBlBNo/m8vXLa93X4EQC59yA04wG7nnHMGtcQb1gXBtDFES6ePUlxQkIB+3nyE1E2OmgoL6HbGm2TSfu+tWaU52XlhGrCvx0ticJN7lFifUbqYQbnZIEUtW0e+xEtytWSeksar/TtECdrudMz6EFweCD3YDjnspJSXrj4lwBmw2amYoWkmSvwz3W2+Bm5giain6t+eRRmDQkKTqQ3swVNa4u22x/cbu/vNV+ek2n5/WTha89PcIDpBlZNpqI8EDSqm17r1rY0ka5IXVla67S+KVY/PQwDUXQs0S1K8sfVPfYPuoycZvYtxKyV4HbRY3pnhGBRvVApuid8xAb1wyfvW2CSSfQgEGsUOTsaRacvN6JKsk7s7NqF2HtuaX3uNeDfbuXnKI6TdXNCMENAeS680dNLcIdNM9UEyGgZCF+V31A1hcTjsxQq+gY2Vr7PHCoOmIqPw4VOomtbAY1loUbB777g3GoKY5NVOB7DW1ySNnZFP41ty66u5eBcBob+ySQ01ap1Z+8tB4XZ145wZHc9CuNZpWrVHPM7vbmfjRzrIBx52I8jD+woSmo9V0AFW7VL+nej5gsesZxQAsXAtwEfgbFNa1LxFgsa8yiKLTIc35Pd51qUxXlBM4PA8x4QhnoBs/ltSOHw0N4/9ecea9IWynk0MorcXYC7NV8ZlYK0LCtgvtAX2nYYQYcjUlcAIVxRHJukHpG+lvfmWCFRLZtRMyOKvtMhxl790wrEdDOrXz1uRmRjmKlxLdGqrAzXY5d/1V12BGggfEDkulN95gvGz6fkcRcGYV85r1EDwr6Sgqmvb4vQmXi1Sn7cdmvD/UvFzIjIg+kuJnqf1HPKz2T7DHgNFfEfpp8PA7C09x+VBJPHRKxlbsgiistf v6MoNFtI 6W9andPlAksg+mDVQ/Uv30wg9iCqRAEzMri28nKAIXCgzDsdZDuDCQWzO1aBaja7d+/x+7AbLIoKlGo99GX7/ckfAGj36s5/D5WzlLCFGjwhuRgNqM7VkZuLpw+pAJHVcGwwbAaWOsqMIMCzYHCwm3ux3oFfVC1d1hdyAYPXqAas4eph9xUyTGMvShHzITVAJiDGzTrW8iWikXKBuUAqAo6ctkVg1Cy3XsUQOQ/aQKb3VQxLnKbQdnOq/hvm1QNzNJGaqAvMVAdpQVLeb6u82MmdwPr8yk8yeGZJHqW7j8HN6Jtws42yTmAvQ4g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Currently, per-CPU stream access is done from a non-preemptible (atomic) section, which imposes the same atomicity requirements on compression backends as entry spin-lock, and makes it impossible to use algorithms that can schedule/wait/sleep during compression and decompression. Switch to preemptible per-CPU model, similar to the one used in zswap. Instead of a per-CPU local lock, each stream carries a mutex which is locked throughout entire time zram uses it for compression or decompression, so that cpu-dead event waits for zram to stop using a particular per-CPU stream and release it. Suggested-by: Yosry Ahmed Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zcomp.c | 36 +++++++++++++++++++++++++---------- drivers/block/zram/zcomp.h | 6 +++--- drivers/block/zram/zram_drv.c | 20 +++++++++---------- 3 files changed, 39 insertions(+), 23 deletions(-) diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c index bb514403e305..e83dd9a80a81 100644 --- a/drivers/block/zram/zcomp.c +++ b/drivers/block/zram/zcomp.c @@ -7,6 +7,7 @@ #include #include #include +#include #include #include @@ -54,6 +55,7 @@ static int zcomp_strm_init(struct zcomp *comp, struct zcomp_strm *zstrm) { int ret; + mutex_init(&zstrm->lock); ret = comp->ops->create_ctx(comp->params, &zstrm->ctx); if (ret) return ret; @@ -109,13 +111,29 @@ ssize_t zcomp_available_show(const char *comp, char *buf) struct zcomp_strm *zcomp_stream_get(struct zcomp *comp) { - local_lock(&comp->stream->lock); - return this_cpu_ptr(comp->stream); + for (;;) { + struct zcomp_strm *zstrm = raw_cpu_ptr(comp->stream); + + /* + * Inspired by zswap + * + * stream is returned with ->mutex locked which prevents + * cpu_dead() from releasing this stream under us, however + * there is still a race window between raw_cpu_ptr() and + * mutex_lock(), during which we could have been migrated + * to a CPU that has already destroyed its stream. If so + * then unlock and re-try on the current CPU. + */ + mutex_lock(&zstrm->lock); + if (likely(zstrm->buffer)) + return zstrm; + mutex_unlock(&zstrm->lock); + } } -void zcomp_stream_put(struct zcomp *comp) +void zcomp_stream_put(struct zcomp_strm *zstrm) { - local_unlock(&comp->stream->lock); + mutex_unlock(&zstrm->lock); } int zcomp_compress(struct zcomp *comp, struct zcomp_strm *zstrm, @@ -151,12 +169,9 @@ int zcomp_decompress(struct zcomp *comp, struct zcomp_strm *zstrm, int zcomp_cpu_up_prepare(unsigned int cpu, struct hlist_node *node) { struct zcomp *comp = hlist_entry(node, struct zcomp, node); - struct zcomp_strm *zstrm; + struct zcomp_strm *zstrm = per_cpu_ptr(comp->stream, cpu); int ret; - zstrm = per_cpu_ptr(comp->stream, cpu); - local_lock_init(&zstrm->lock); - ret = zcomp_strm_init(comp, zstrm); if (ret) pr_err("Can't allocate a compression stream\n"); @@ -166,10 +181,11 @@ int zcomp_cpu_up_prepare(unsigned int cpu, struct hlist_node *node) int zcomp_cpu_dead(unsigned int cpu, struct hlist_node *node) { struct zcomp *comp = hlist_entry(node, struct zcomp, node); - struct zcomp_strm *zstrm; + struct zcomp_strm *zstrm = per_cpu_ptr(comp->stream, cpu); - zstrm = per_cpu_ptr(comp->stream, cpu); + mutex_lock(&zstrm->lock); zcomp_strm_free(comp, zstrm); + mutex_unlock(&zstrm->lock); return 0; } diff --git a/drivers/block/zram/zcomp.h b/drivers/block/zram/zcomp.h index ad5762813842..23b8236b9090 100644 --- a/drivers/block/zram/zcomp.h +++ b/drivers/block/zram/zcomp.h @@ -3,7 +3,7 @@ #ifndef _ZCOMP_H_ #define _ZCOMP_H_ -#include +#include #define ZCOMP_PARAM_NO_LEVEL INT_MIN @@ -31,7 +31,7 @@ struct zcomp_ctx { }; struct zcomp_strm { - local_lock_t lock; + struct mutex lock; /* compression buffer */ void *buffer; struct zcomp_ctx ctx; @@ -77,7 +77,7 @@ struct zcomp *zcomp_create(const char *alg, struct zcomp_params *params); void zcomp_destroy(struct zcomp *comp); struct zcomp_strm *zcomp_stream_get(struct zcomp *comp); -void zcomp_stream_put(struct zcomp *comp); +void zcomp_stream_put(struct zcomp_strm *zstrm); int zcomp_compress(struct zcomp *comp, struct zcomp_strm *zstrm, const void *src, unsigned int *dst_len); diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 3708436f1d1f..43f460a45e3e 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1608,7 +1608,7 @@ static int read_compressed_page(struct zram *zram, struct page *page, u32 index) ret = zcomp_decompress(zram->comps[prio], zstrm, src, size, dst); kunmap_local(dst); zs_unmap_object(zram->mem_pool, handle); - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zstrm); return ret; } @@ -1769,14 +1769,14 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) kunmap_local(mem); if (unlikely(ret)) { - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zstrm); pr_err("Compression failed! err=%d\n", ret); zs_free(zram->mem_pool, handle); return ret; } if (comp_len >= huge_class_size) { - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zstrm); return write_incompressible_page(zram, page, index); } @@ -1800,7 +1800,7 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) __GFP_HIGHMEM | __GFP_MOVABLE); if (IS_ERR_VALUE(handle)) { - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zstrm); atomic64_inc(&zram->stats.writestall); handle = zs_malloc(zram->mem_pool, comp_len, GFP_NOIO | __GFP_HIGHMEM | @@ -1812,7 +1812,7 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) } if (!zram_can_store_page(zram)) { - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zstrm); zs_free(zram->mem_pool, handle); return -ENOMEM; } @@ -1820,7 +1820,7 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) dst = zs_map_object(zram->mem_pool, handle, ZS_MM_WO); memcpy(dst, zstrm->buffer, comp_len); - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zstrm); zs_unmap_object(zram->mem_pool, handle); zram_slot_lock(zram, index); @@ -1979,7 +1979,7 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, kunmap_local(src); if (ret) { - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zstrm); return ret; } @@ -1989,7 +1989,7 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, /* Continue until we make progress */ if (class_index_new >= class_index_old || (threshold && comp_len_new >= threshold)) { - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zstrm); continue; } @@ -2047,13 +2047,13 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, __GFP_HIGHMEM | __GFP_MOVABLE); if (IS_ERR_VALUE(handle_new)) { - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zstrm); return PTR_ERR((void *)handle_new); } dst = zs_map_object(zram->mem_pool, handle_new, ZS_MM_WO); memcpy(dst, zstrm->buffer, comp_len_new); - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zstrm); zs_unmap_object(zram->mem_pool, handle_new); From patchwork Wed Feb 12 06:27:01 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13971031 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 65BF1C02198 for ; Wed, 12 Feb 2025 06:32:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C72346B008C; Wed, 12 Feb 2025 01:32:20 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BFB446B0092; Wed, 12 Feb 2025 01:32:20 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A762D280001; Wed, 12 Feb 2025 01:32:20 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 868466B008C for ; Wed, 12 Feb 2025 01:32:20 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 40283A0E8D for ; Wed, 12 Feb 2025 06:32:20 +0000 (UTC) X-FDA: 83110323240.17.FE70F9D Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) by imf19.hostedemail.com (Postfix) with ESMTP id 5E9121A0006 for ; Wed, 12 Feb 2025 06:32:18 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=RD3kvKCW; spf=pass (imf19.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.177 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739341938; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=48U7icBiaj5vHELEBDYpmRQUsi6H+QG8AACYxOChctw=; b=a/x1uUZT8ix6dV2eUTibqS2bPCQ0EHcetY6s83RwmIRAqpHaPNKP9dmcRE+UFom6mbiqaw v0pdapA8lIGcqSkTf118RZLizkv8MTyCvPRScodNKnCF3HFrPUj4339nHV9E8q6LB88g1i mebYAuCc/471wUMPsa69SXGyAfK1yLY= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=RD3kvKCW; spf=pass (imf19.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.177 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739341938; a=rsa-sha256; cv=none; b=IyQLOf8Vs6STA7yG6ZJFa138lcUso9Ko/c1tpxQDAQQWH+yrhZ1FgihmJY/CfqvdDAY9zs ihOliO2+Hr+eShAAi3rZZPywEhevQR2t3WJFAlUdpbbunk+TaZ4tMRIRSGHeTNB6A3eaQD vz5jR+ccu2UKiBLYsVj730WQXtMYqzY= Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-21f5268cf50so70832935ad.1 for ; Tue, 11 Feb 2025 22:32:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739341937; x=1739946737; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=48U7icBiaj5vHELEBDYpmRQUsi6H+QG8AACYxOChctw=; b=RD3kvKCWk2oUXFA6JeFkiV5W57twjMmyg4LY9JLtPtXT7qh+roDnoOjfcLyn1fThNf 0hbNkB7wW9sVHXkgawS0fWMkcsFbvJanrgSVWqBnP1QjJji7IYMRJ+mRYYjfh0ca5DsX 6ODrY+1qtnqFeh+Lyz+nK1Oy/iID4CNW6evHs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739341937; x=1739946737; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=48U7icBiaj5vHELEBDYpmRQUsi6H+QG8AACYxOChctw=; b=oqlmjq3tu64Z6Ll+YEIE+o37Z3CazDapbTFO1Hd9IBftdkbrfEbPTyfny840jULtQ+ ff8uwphAaUBs1VXKGLZ3RCdjZpWmmyqbWYVXJiIC9AFmwscsXMgIGxT1oCMuMKQaRXrP 0/fS0MPI5wYUza2t6BTorzo4fhHx/BHvjgIerVOTEJ0mHj5DGtZEU5uAQ0aYb0dON/dF A5kwpsv69XFu2X1R0nYNUvB2zgggTc9XXJuqOmIR0KkWrNigTRX3CSgNlMY0HHuvtGvn z81r6fav+JvjKMsOD6V2C98PMJZxp/armw1FYHH9hmfLhnoUcpi2raFMgga0QfIenFt2 /7PA== X-Forwarded-Encrypted: i=1; AJvYcCWxphgjltbLHY7KaiQRtw1t71loFziu5VQC5bAxmRfG0Y1AXcTxQU4vBmfQmLKZjV6cb3FgMxyZcw==@kvack.org X-Gm-Message-State: AOJu0Ywe+xEQeqwun7BDNcpf7wcclzCVwm8SC8XaXYZ48anopxRph8Ij d7DY+T2TDEmLYq+Hg2RPWxY20PD2quGwAqNSthlw+uHYfOORoiZE6U2CVundTW/wFOJWgEIm6M4 = X-Gm-Gg: ASbGncu+LrVNQrqWHwzntiVNeFv7KnXcTgG8cIk2xYD3tMRlVAHIcD4hzVI+zvopesV vUB3lQSD95IX0ubI5i+fpVpCjsm33SITG4plgLb+tSTzXhx8er2mEukaznq4JT4yXEnTGaLcRkd TYoZ6E+e414M7CtWlNVAwKsZqG9AsybYRO78bYCRYN0phrQDTNiuhaBSoFgQlc/BbxKfiE5xJY1 glyT6jhNLV05OpJykNRZrMKOAfQ7hvHLXti8sk84OI/sn+g7rWboAlOhIBt7H9nLu3mIoFIfeAD FDtZu1IrWKSAt0xPsQ== X-Google-Smtp-Source: AGHT+IGlZ7sjxw8CKEDe37B21DbU3g+Tt4XgUZOISnGOFp7x2JKzUfH/1KMDV9dAFdU8C0jsA6InIg== X-Received: by 2002:a05:6a00:1c8e:b0:730:9659:ff4b with SMTP id d2e1a72fcca58-7322c5febedmr3482199b3a.19.1739341937213; Tue, 11 Feb 2025 22:32:17 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:69f5:6852:451e:8142]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-73048ae7c24sm10384533b3a.78.2025.02.11.22.32.15 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 11 Feb 2025 22:32:16 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v5 03/18] zram: remove crypto include Date: Wed, 12 Feb 2025 15:27:01 +0900 Message-ID: <20250212063153.179231-4-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog In-Reply-To: <20250212063153.179231-1-senozhatsky@chromium.org> References: <20250212063153.179231-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 5E9121A0006 X-Stat-Signature: np3ygm8w3wbdzyx849jg1c8z8br6y6so X-Rspam-User: X-HE-Tag: 1739341938-984768 X-HE-Meta: U2FsdGVkX19CtdAviiuUVlrkGODJ4Bhq1hB08fFyQBDqDcPAwPNKAT0iZEm8Qd4peTaDxIIPUyLaEUYa5M1T/EaviWRf72rfowMuf/Ti/MKtj0AIxFVn+ry2CebOlxOn7gzpsoQ0k5vJ8Vh076Hi7Js8t1uI6h3oeGNzCPVsODXJ4WvHs5yVxZm5GcOJAPM6dfm009XDzUFo/ddYPze9usroiz2paSBQKrQOK0069OywGonePylbJw9qDdqwnp7OuPB33U0nidW/B2VIWtkGXxZ6PjHQlBDC1KmE9AtnBTArolZyADeM7BKskCAa1mwKyNY4z9vr0RWLKsZa6GUM7bhvTlpubt/dw6sDsD8oh2hTnNhqTWzLADWxpx0imkjlfZEwKdkJ80AP14njq0yIaduTSOrXtpmPMHBblyjdcsbXZgPcv6jw1lZUala1hNjTkvCFPhcti4aveXWUWiEDRYdlLXcki5PQMC8i8IL4+msGVUms1YbPYfTYn+vu/HhlGxQJa6TEMj5R2P7OZVxZBBDWGfFjYsOZUgbZn7wjHlXOB/3PpI/RMaEb0URIquafYcdwfPzFrSQv57XF530I5Ho6bsqOswxk8fjtx2QBoE3hjSM/xA5T7OP0L32/BGbArQaamXgHz3jxYe7saZeg6u2UhV7u1/k839iFKbP7esxuJJINlpdfI3QTpKGO5P6hK/UzP5EpLNR/RkH+FN2PL9j50woReOmWz+3etzs19kh0rLbJPOE1EGqg7sKO+8VrsQvFtcqsLDw54eaVzQ5MpgeZEN0CP+d1onbfmbFj29NPnSTMDveUXh372ioiYRWWzW8pG4IfodN6ZRRxrgJOuf7jdTgremFWtjjQa9Ddk+lG594PDlFfShOW3hZYnABGRC3skZ2z8nfdVBQQhQ9ZVbcjp4BSx4yauEcQ0CnCmDvpdeZkgG4AdScRpTfmcmct/9sV/2/JqVaPLfkOwjB fxgixRZ5 dCtJmxEekvuiOgDTgP96xY01BLRY30/radhegfexjF8a7Qa4ML9TNcJqLiIotAuPEekM+mhNFL0IhMNcTGJssAwlJBKwt/k88ydLuY7RCPShpab7g9dbCj5innKnGA4YK1sipEyirxRP0fbxkRFg4OfNRIvS67l45V05zlsd6dt7iPMF8RYMBibC7gI/vdZ6KOohRh0nSS4CgO7ZlILohoxS0FQWKqbysAgbLMDuq29b+mm8tG4gzaoRAYzHApNtUVNHhDGh95PpsqX+BIuCDWGrroX4r7FGrYjBNX4uSWBOe+OOslIsd63CBMIwFcf3wLiRR8zArYhZHXz5v4Ji0V1SdzdD1fXbJ5GHn X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Remove a leftover crypto header include. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zcomp.c | 1 - drivers/block/zram/zram_drv.c | 4 +++- drivers/block/zram/zram_drv.h | 1 - 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c index e83dd9a80a81..c393243eeb5c 100644 --- a/drivers/block/zram/zcomp.c +++ b/drivers/block/zram/zcomp.c @@ -8,7 +8,6 @@ #include #include #include -#include #include #include "zcomp.h" diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 43f460a45e3e..12fb260e3355 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -44,6 +44,8 @@ static DEFINE_MUTEX(zram_index_mutex); static int zram_major; static const char *default_compressor = CONFIG_ZRAM_DEF_COMP; +#define ZRAM_MAX_ALGO_NAME_SZ 64 + /* Module params (documentation at end) */ static unsigned int num_devices = 1; /* @@ -1149,7 +1151,7 @@ static int __comp_algorithm_store(struct zram *zram, u32 prio, const char *buf) size_t sz; sz = strlen(buf); - if (sz >= CRYPTO_MAX_ALG_NAME) + if (sz >= ZRAM_MAX_ALGO_NAME_SZ) return -E2BIG; compressor = kstrdup(buf, GFP_KERNEL); diff --git a/drivers/block/zram/zram_drv.h b/drivers/block/zram/zram_drv.h index 63b933059cb6..97c98fa07954 100644 --- a/drivers/block/zram/zram_drv.h +++ b/drivers/block/zram/zram_drv.h @@ -17,7 +17,6 @@ #include #include -#include #include "zcomp.h" From patchwork Wed Feb 12 06:27:02 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13971032 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DE341C02198 for ; Wed, 12 Feb 2025 06:32:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 71B0F280002; Wed, 12 Feb 2025 01:32:26 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6A45C280001; Wed, 12 Feb 2025 01:32:26 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 51DF0280002; Wed, 12 Feb 2025 01:32:26 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 30C30280001 for ; Wed, 12 Feb 2025 01:32:26 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id B5AA4C0E5A for ; Wed, 12 Feb 2025 06:32:25 +0000 (UTC) X-FDA: 83110323450.30.ED4F967 Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) by imf23.hostedemail.com (Postfix) with ESMTP id ADCB3140004 for ; Wed, 12 Feb 2025 06:32:23 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=oc16v3M2; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf23.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.177 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739341943; a=rsa-sha256; cv=none; b=D1gUaR2K5Jo1qm0eIJPd4gy/x3Kh7N0Ky3OxC+a4XK4lyFqXrdwPHSUi7VqUj1f3GXF3Hm FaN6q1KhAIt9olSV2eodk5tUMlP5tDn45Q7Qx4veHVLojxvfm1nDKbA6NpIIamZbwypcG8 OOTXjEBXG38q4gMZQ43LNlY1PIEFPKE= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=oc16v3M2; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf23.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.177 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739341943; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=h2AQA5cUglzG6yhVm3aKj+975OMRekO7mg4bKTRqdCs=; b=WJnpanFb70S6LcovkEM3qgO4IV8Jy71z6MLZOIjS+F5fewXpgEBxW28YVB+J+w+nmKyK/B anHiQTF6KB17foCIK5QLiKC6waeNfJ7P2Gqg4RcF2+HUdSH/foXKsWPGlpAzKuEweVCr+u DeQSQdLR5F/jLfUavPAyrcDfhPj+89M= Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-21f6022c2c3so8169405ad.0 for ; Tue, 11 Feb 2025 22:32:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739341942; x=1739946742; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=h2AQA5cUglzG6yhVm3aKj+975OMRekO7mg4bKTRqdCs=; b=oc16v3M2vQFMlRypHc0eDEuezAgkQ03MMrwIarcLE+XVThK3BRSWTn6Bd777gp6T0P rtCRvuO1TcVeSdzkRN83mLbQZL9tQ1cJIXM2LhkUba0flN7kE68VEczJnt4QuefLJl/W vq1DwbyB4rnF2lQhQZcUpzot+7eDG5Rus3ifY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739341942; x=1739946742; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=h2AQA5cUglzG6yhVm3aKj+975OMRekO7mg4bKTRqdCs=; b=ZoeSlBGt9Jy1JqAin4xu/WB6/9bV2jtDvhiz4qLS9lw72zVnCribcE1toCuvteZWDp EMIsJjeFavpPGvaaOTvX5Y704BkeNbBEqnV4a9MKzKA/vX90Qh2zw4LaGrcqpWx0oHaP Ap/cdppHyx1nq9CloNNW5z+CfailTSsSzY6YCURbkq7x6MbxgsaU8D8UmreKCfr2xTvO gX6bhUIfarjyOdMzhBZN3Ru4RG1HDxTdrXY91+esT69/eI6qOvhUQtDWqy8e5Br1fbpz L96b7ErKrEA+vei8QZb8GiFbigl3a3JYLq+JiZfw40idp2g+4Z1bPXcivHtdqct6RC+a g0OQ== X-Forwarded-Encrypted: i=1; AJvYcCVfYbGKLgP0La1sCaRffC6LfnY2KNJMmsC7jZmN5ZlxdMizoB+J743jBNq1+OF5sf0OD2+Sug3A/A==@kvack.org X-Gm-Message-State: AOJu0Yw/BgEKUH68ZKNBCg6yaWEC0o08x8Xk++IyVxUi2LMkblzPJ/C5 6zZtrlWWfM2PcYLJuJPmqbPRkugqqRiNwxivLnDuB8oc+rfQ9fBdM43GbEMiXw== X-Gm-Gg: ASbGnct9Xl82ZCk7av1QWsyDb0hlaj7aE2/J7M1/kfkM0ohGL0KxLzEuLukmaE3DAGd QVdzcolqqSoWsh9GVB6p4oKwloSO8igzmZ0xpy+8CljMzhssPAzCg2GHtsEVSGO+TFF+ZkcHFNe 4mj8+KdF/jNmTsfoARsRoOc4q17q2gpCTNhUUciE/CX5Mc3vndUsTlVp/3hTFaZxKr0BUXuiBah XQKuWc6q6l37Vl9845YqU0nIxjLmuYtaRGLoInOe86KzPqqUY+c2gxanGIGnxjpRuRWGEmSJbK7 hdcow9MshNO42TTvWQ== X-Google-Smtp-Source: AGHT+IFvCaT4j3C930RCMRawLzcITwesvr3FnYejPnLHFg8E648OUmGxFSWdbEBwV5fB9AnLznY3OA== X-Received: by 2002:a05:6a21:398f:b0:1d9:a94:feec with SMTP id adf61e73a8af0-1ee5cea8113mr3339152637.2.1739341942625; Tue, 11 Feb 2025 22:32:22 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:69f5:6852:451e:8142]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-730918f6ec7sm4475974b3a.37.2025.02.11.22.32.20 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 11 Feb 2025 22:32:22 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v5 04/18] zram: remove max_comp_streams device attr Date: Wed, 12 Feb 2025 15:27:02 +0900 Message-ID: <20250212063153.179231-5-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog In-Reply-To: <20250212063153.179231-1-senozhatsky@chromium.org> References: <20250212063153.179231-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: ADCB3140004 X-Rspamd-Server: rspam12 X-Stat-Signature: cfpjci4kzjrxsmo756npzmwprzihh3bk X-HE-Tag: 1739341943-720229 X-HE-Meta: U2FsdGVkX1/3OYcCYRHomsyJiX/SE/YuISGN5cVu5BbnDvk1blBbNSYUFil3DzrGcAopwkUDPHx79hbje36HeRcjC7Bw8OoypcaToehnaGgVUSnGBnbJN7LhMj2GcrS2iYUu1qWbVCN5e57ZvVvRK6uGjFd5QzOMNrAm6Wgssgg+iSeWDzhH1dL7zGe34iXSMGJHLgfU0QMv1SwWZ/RFPF0hUfdpO8TJpP3CXEoaxrnuMO9DV4z4H0h5lN5hBQbqQwuaqvtlvh/Y+8fsat+G5mjQ1gKu19vOEBZNl5cedk5y1oKd0PZoOqhuI+Ga+esmrOaxrGNhawYjf70mXmqt0np/Bb+0LYEzm3tLGLIQPyNclhPD/oEiCchBVlwnzjnbk5WxBPijxZxa9PXJ5zD5LwXL83D9fJiRy47mHPgZ8T5HjoOA+0AdZIW/j06SsnYUp9KKS9b7LRnPmUuFyzPJ2TE75MOYmrPHevfk1L+4tCK3HgCY7lJzQFXBESPSd2u44eEoYp58OSRA8o80f4IVQlK5LIQBK/vr5gdk64ckW6dJ7smiyaz+CTb1y4lEc+TOFmkS0tMEPLZMYH2++eGuoWjKYyZwZIEHFA9qAWcFzDJrBmHErM7Dx6hXR5cbrdQPJiX/EPjhNstTe4t5WfGJx+p9+A03dYgyOI/7ONSfbt+MRJ4UdYkkshsnXIrtjl4hijHbS3ePOsdS9r6xcKE0phrpNT63y7rH7opsb84LfvuCXbMmVqQ7ygL+u/rXqpvpnLuF676NmhibECK5khKWVdnwo5HzMuc4UdNQJfESSzntJlppkCNQwGNUjoa0TaW/dcoayzzKbPjnbmWPqdKPjanIIBdBIrDRQt07h9SAq93oAXGrnQBOdiOKGYMvPtM/GUo6KoOQlpdgx2gyXGOCSVSCnsuZxZvMkFD6T/SNxOHOdgdsta5d9VpyrgdMLO7xN3x6AS1J28Ffh+0SgOl ZAZeBeLE mxfmM3aVkCRrUCwHnpvDbTK0c71bTqbcpYxpyrr7AhKjOnv8bXRpoawwMpkjNG0T4lbdIoN7/jGwqnqbwNTvdtERKTQmr6Dw9mLWICbaSb/1xKGV6gAiSyR3IbkVeSfVFRXGRyXEheBRskFXxlmP0S3Cra/+NSPTa+MVk7DgZKAWQaTfUElfyDmImcLNgkpNfDN1BdwWLr/WH4P1r+WN5N0TrmWU69vGo60GqVvHGj33iCqzU5d0hy9CjG+BEL/FTyGL7q/lKdiPKlYUH7uYL9vImbrpa2ZmcCDMS3NvBFJYluZSlrPWwJ+x2OI8RvIU8mxOJTh/o9u53Bwp2a3aqF0BDC/gHDtVFGsKZVI68wkGiSwCrEa6vNG0Lr4GkYHSMLmO+ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: max_comp_streams device attribute has been defunct since May 2016 when zram switched to per-CPU compression streams, remove it. Signed-off-by: Sergey Senozhatsky --- Documentation/ABI/testing/sysfs-block-zram | 8 ----- Documentation/admin-guide/blockdev/zram.rst | 36 ++++++--------------- drivers/block/zram/zram_drv.c | 23 ------------- 3 files changed, 10 insertions(+), 57 deletions(-) diff --git a/Documentation/ABI/testing/sysfs-block-zram b/Documentation/ABI/testing/sysfs-block-zram index 1ef69e0271f9..36c57de0a10a 100644 --- a/Documentation/ABI/testing/sysfs-block-zram +++ b/Documentation/ABI/testing/sysfs-block-zram @@ -22,14 +22,6 @@ Description: device. The reset operation frees all the memory associated with this device. -What: /sys/block/zram/max_comp_streams -Date: February 2014 -Contact: Sergey Senozhatsky -Description: - The max_comp_streams file is read-write and specifies the - number of backend's zcomp_strm compression streams (number of - concurrent compress operations). - What: /sys/block/zram/comp_algorithm Date: February 2014 Contact: Sergey Senozhatsky diff --git a/Documentation/admin-guide/blockdev/zram.rst b/Documentation/admin-guide/blockdev/zram.rst index 1576fb93f06c..9bdb30901a93 100644 --- a/Documentation/admin-guide/blockdev/zram.rst +++ b/Documentation/admin-guide/blockdev/zram.rst @@ -54,7 +54,7 @@ The list of possible return codes: If you use 'echo', the returned value is set by the 'echo' utility, and, in general case, something like:: - echo 3 > /sys/block/zram0/max_comp_streams + echo foo > /sys/block/zram0/comp_algorithm if [ $? -ne 0 ]; then handle_error fi @@ -73,21 +73,7 @@ This creates 4 devices: /dev/zram{0,1,2,3} num_devices parameter is optional and tells zram how many devices should be pre-created. Default: 1. -2) Set max number of compression streams -======================================== - -Regardless of the value passed to this attribute, ZRAM will always -allocate multiple compression streams - one per online CPU - thus -allowing several concurrent compression operations. The number of -allocated compression streams goes down when some of the CPUs -become offline. There is no single-compression-stream mode anymore, -unless you are running a UP system or have only 1 CPU online. - -To find out how many streams are currently available:: - - cat /sys/block/zram0/max_comp_streams - -3) Select compression algorithm +2) Select compression algorithm =============================== Using comp_algorithm device attribute one can see available and @@ -107,7 +93,7 @@ Examples:: For the time being, the `comp_algorithm` content shows only compression algorithms that are supported by zram. -4) Set compression algorithm parameters: Optional +3) Set compression algorithm parameters: Optional ================================================= Compression algorithms may support specific parameters which can be @@ -138,7 +124,7 @@ better the compression ratio, it even can take negatives values for some algorithms), for other algorithms `level` is acceleration level (the higher the value the lower the compression ratio). -5) Set Disksize +4) Set Disksize =============== Set disk size by writing the value to sysfs node 'disksize'. @@ -158,7 +144,7 @@ There is little point creating a zram of greater than twice the size of memory since we expect a 2:1 compression ratio. Note that zram uses about 0.1% of the size of the disk when not in use so a huge zram is wasteful. -6) Set memory limit: Optional +5) Set memory limit: Optional ============================= Set memory limit by writing the value to sysfs node 'mem_limit'. @@ -177,7 +163,7 @@ Examples:: # To disable memory limit echo 0 > /sys/block/zram0/mem_limit -7) Activate +6) Activate =========== :: @@ -188,7 +174,7 @@ Examples:: mkfs.ext4 /dev/zram1 mount /dev/zram1 /tmp -8) Add/remove zram devices +7) Add/remove zram devices ========================== zram provides a control interface, which enables dynamic (on-demand) device @@ -208,7 +194,7 @@ execute:: echo X > /sys/class/zram-control/hot_remove -9) Stats +8) Stats ======== Per-device statistics are exported as various nodes under /sys/block/zram/ @@ -228,8 +214,6 @@ mem_limit WO specifies the maximum amount of memory ZRAM can writeback_limit WO specifies the maximum amount of write IO zram can write out to backing device as 4KB unit writeback_limit_enable RW show and set writeback_limit feature -max_comp_streams RW the number of possible concurrent compress - operations comp_algorithm RW show and change the compression algorithm algorithm_params WO setup compression algorithm parameters compact WO trigger memory compaction @@ -310,7 +294,7 @@ a single line of text and contains the following stats separated by whitespace: Unit: 4K bytes ============== ============================================================= -10) Deactivate +9) Deactivate ============== :: @@ -318,7 +302,7 @@ a single line of text and contains the following stats separated by whitespace: swapoff /dev/zram0 umount /dev/zram1 -11) Reset +10) Reset ========= Write any positive value to 'reset' sysfs node:: diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 12fb260e3355..e0e64b2610d6 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1104,27 +1104,6 @@ static void zram_debugfs_register(struct zram *zram) {}; static void zram_debugfs_unregister(struct zram *zram) {}; #endif -/* - * We switched to per-cpu streams and this attr is not needed anymore. - * However, we will keep it around for some time, because: - * a) we may revert per-cpu streams in the future - * b) it's visible to user space and we need to follow our 2 years - * retirement rule; but we already have a number of 'soon to be - * altered' attrs, so max_comp_streams need to wait for the next - * layoff cycle. - */ -static ssize_t max_comp_streams_show(struct device *dev, - struct device_attribute *attr, char *buf) -{ - return scnprintf(buf, PAGE_SIZE, "%d\n", num_online_cpus()); -} - -static ssize_t max_comp_streams_store(struct device *dev, - struct device_attribute *attr, const char *buf, size_t len) -{ - return len; -} - static void comp_algorithm_set(struct zram *zram, u32 prio, const char *alg) { /* Do not free statically defined compression algorithms */ @@ -2541,7 +2520,6 @@ static DEVICE_ATTR_WO(reset); static DEVICE_ATTR_WO(mem_limit); static DEVICE_ATTR_WO(mem_used_max); static DEVICE_ATTR_WO(idle); -static DEVICE_ATTR_RW(max_comp_streams); static DEVICE_ATTR_RW(comp_algorithm); #ifdef CONFIG_ZRAM_WRITEBACK static DEVICE_ATTR_RW(backing_dev); @@ -2563,7 +2541,6 @@ static struct attribute *zram_disk_attrs[] = { &dev_attr_mem_limit.attr, &dev_attr_mem_used_max.attr, &dev_attr_idle.attr, - &dev_attr_max_comp_streams.attr, &dev_attr_comp_algorithm.attr, #ifdef CONFIG_ZRAM_WRITEBACK &dev_attr_backing_dev.attr, From patchwork Wed Feb 12 06:27:03 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13971033 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 91639C021A5 for ; Wed, 12 Feb 2025 06:32:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0DB1D280003; Wed, 12 Feb 2025 01:32:32 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 06262280001; Wed, 12 Feb 2025 01:32:31 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E1E9D280003; Wed, 12 Feb 2025 01:32:31 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id BBA62280001 for ; Wed, 12 Feb 2025 01:32:31 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 43941B1327 for ; Wed, 12 Feb 2025 06:32:31 +0000 (UTC) X-FDA: 83110323702.29.4E6F412 Received: from mail-pj1-f52.google.com (mail-pj1-f52.google.com [209.85.216.52]) by imf22.hostedemail.com (Postfix) with ESMTP id 63B4FC000A for ; Wed, 12 Feb 2025 06:32:29 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=EJtEQ0sH; spf=pass (imf22.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.216.52 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739341949; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=QEMWhAClB1d5xqcLAEaUS1zNcFLgRU1myRFnr10zi+A=; b=CEaB991rzGfQyhhjIPV09WYHYN+v9WmUI4vM1VYCSq6sqqZqvRag0qxtzUb+wuePvZMd8H e7RMRZQVK2BT8whABhvxrvGFy80Frm7VNg1Rk41ZrxHqIqXOfWpvK98Jy8zqLc28IYFe6G l+d+RzMOeoUwR6gbvTM2kJbTJ0GOImI= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739341949; a=rsa-sha256; cv=none; b=y1csRUn4nSrGCzOuTotP5r3ren2e/j8Fg7af+r/IcOMlF0G7hhQVHcgy2JYuuiNOSPLp9v tJQ1vISpTyGqGAwTcOf27i9wmwmCjny0u2TcmdzGNPGX6YMksci5eW+1Kz+RAxHUIOsvaZ U6igQQEi8qw3DPlcSJetwnYhOOmcpr8= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=EJtEQ0sH; spf=pass (imf22.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.216.52 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org Received: by mail-pj1-f52.google.com with SMTP id 98e67ed59e1d1-2fa3fe04dd2so6249462a91.0 for ; Tue, 11 Feb 2025 22:32:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739341948; x=1739946748; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=QEMWhAClB1d5xqcLAEaUS1zNcFLgRU1myRFnr10zi+A=; b=EJtEQ0sHkoJlClzWBW8roHSxA8Wqogw+jqPOTAGCbEB/p04ZXGqfbkxWTAuE3q5eqW 3kp4h34OK6JQjpTlWOa10x6UqtZE7TgqFqrt/h5KapveLUuC7Ai2f3H1rt/3rUqpG3aI 8oB0WiQr+pkaXKvEtCcUZlqTNx2YzwFq35x/o= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739341948; x=1739946748; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QEMWhAClB1d5xqcLAEaUS1zNcFLgRU1myRFnr10zi+A=; b=ilaV0BY2DhBxcUGRlD6uVIJnPCKyOmW0qmUUTRVaPCvyqZy9WHCWE/LWneLC5hgOvc HITLUdk3HCg0D62Dst+jlIO1ApHo72Lhm+kbRbZ5ODKrGRLKuUMUutWSoHXFjSy/tSL5 zb+iR+0sSKinn9ji9vHkl0AxaMVLWm2xiEAGFQj0oQNamnzaz9+56v+xWCLQ/5lOXRAP fcvbX3L7Y9FXeCf8aA8ceyOXUQ+nxxKn3PktCk81F1tLGJ8MQ0Mfi2QwIX/jix1zRFXE Hx/lFjWh8SHjx8gQKDZjP+T+cmRCfM0CFlcjdDRVRJo0ny1WfPopYbgVAxSHAPW+mjXT W4Wg== X-Forwarded-Encrypted: i=1; AJvYcCVSKZ49+6Bfg+cVGZekFghbPmeYX8Nl9C6lmLsZMC2ZuE+uH5RZE1PR02938VpAE0koBZ2d5qNY9A==@kvack.org X-Gm-Message-State: AOJu0Yy6HA0SIEKGOOCNAqAFuPshkmJSnTK8GaJouMgPmGXclHCuVNDj 9hlMix1Nn9c9mLvrK16azOMKnMRP+RbN27WlRltjvn2PfrF+8w4Mf8tFdDjqCQ== X-Gm-Gg: ASbGnctPDoeR8lNVJWbKkuBGWi77Qpobi8+UYzhz3NHmjmxrz6JFysU8np3bLMiTGd8 fOz+Xo1KYxtvHOIZDvGYxkdTCUhet3/cbbCSCyxVlS+V+HSI8nFNFicC7tAmlhu75Je7ZIFFLNn 3Xuy+L/Vh7id9T5ZfVLPOe5inXERySDSIuZt9VRsUaru6mb5rs1PDGRIvkk49YnppO+Bs9ccfWz AnjJLHb7Oeu8BP0/9bPPi5yTAUh7+Nd7ZIyQQLamT7px4SVKyZWzSRV+xDls6BiUdgFFJHZq39+ wHvRX9UMh7X8ehwLfg== X-Google-Smtp-Source: AGHT+IFA+EnqkupBVK3/MEPCQX2ZjyTTOUvx22Z0kt6I03ZDDZCylIMnWUqaHODE62jxmfspsUlwkQ== X-Received: by 2002:a05:6a00:804:b0:728:e2cc:bfd6 with SMTP id d2e1a72fcca58-7322c43107bmr3332267b3a.18.1739341948039; Tue, 11 Feb 2025 22:32:28 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:69f5:6852:451e:8142]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-730862e72a4sm5749487b3a.103.2025.02.11.22.32.25 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 11 Feb 2025 22:32:27 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v5 05/18] zram: remove two-staged handle allocation Date: Wed, 12 Feb 2025 15:27:03 +0900 Message-ID: <20250212063153.179231-6-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog In-Reply-To: <20250212063153.179231-1-senozhatsky@chromium.org> References: <20250212063153.179231-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 63B4FC000A X-Rspamd-Server: rspam07 X-Stat-Signature: enb4nb83jcesr5g8rm99r3a33qrgpphp X-HE-Tag: 1739341949-764000 X-HE-Meta: U2FsdGVkX18Hxl132uFL5nBQ2+l+3tkSUurxsL/WtIzT8qVrnrbWJ9Lp1sKowOxYxCQx/srs10E3SG5uMKzXh0BVf6xIS3FbPG7tU5rZJ1+whOixTrjG9wUXQVJHmwJBkCWmvNtFKp6XyYrbazKXnZgqUIds6llKpKYaVUs1rFpyKG41znEtA9LchNO2O2He8SEpwyD/bJDOBnjqshWFbgkcrEA3MoRKSWmnGgJm0HPOKNS0AMC7uhL6uDaJ3SeY0//sFj5/4Y9AOzJjbFFQZQc0k2aCuJXwjF9rtxS5BDcffqbB0xXh8lg1jlb6iWmXmfIwHkyZNfwg8NtHBJrbSUrDYETu8pojp+ZU/8Zy6EgRlVLaf8MPYj5RWWpCC3Kdg4Pxss56kuz5MfQRSJNv9H3xQFlrvTQgOkp9qvvcKRrv/gM2I77la2cvyHRNWYKG/3bELlBWRnZZwTWt/85HOc/eZwOkcJsks/halnCpp6cDB5Hc25UHDMUrK/98GI8BBUWcNtRcvMEsiqHh7kzo8dzs/A0fxjZLRW4/lefWZ64+Re7EGbpKhVCyrHfXmZ1hyZUI1ayXTnpG+Tcglj9uzB0nj75ukn00T3wSkmErLjWDkOrYJQzPJc1T2Nm1ys5MlgeiF7fq3MTskxE9P+RHD0KnFQhV0ocnte74JcJO2MeUiEoq0W3JB6yFnvLcerQ8X61idD79fyb4lCHuKM2Wep6Ui5zb994BCN3jD6ZKhBIzp8e5IJgiH73dVUbvwklnP6cfm7sgsBNleQWCpkv4ETkzrWcsIR+3Lsz7xGjtSJ6O0CfLrd0sZQPmPYC9U3O7ZY6AL1S2GE5g6+ythPDrHDNOatKgIgoWLOZJGVgy0KrImEp8My/9NcjZO3TOnbmVUTUEWPYvAlqgdY8AQ2s+oyjS7r6vZCPVbAQ0H0Il8UhiZmhXseNynsdVB7/JwW4vJ3sUUJJkSttaBuM98Fg JtpPFHhl epcEByrHpan8rGmYWCkbOm9rCCbbm2zKQMUGpWpuuGLulQun9h5LlnfRSJ/vb4+SU8sxUzRM0pM9EqbCs/aZ+uZaS5aKjPYGDc+smvwgAOrWfazCj4pvl9CV+MXf/Wa2eDQgEI3DJCyugrNiJqU1mLvq2o3uIK2XTlYVIi2X5Qgo0vqrgZmyG6dGpr0LiAK/sdwG6q2sQwn2cHF/RZ3IRRroHh9l0rUfE7RhHKbm1C75oGHrRrJlbaMXR9vMb+xOoYeu7vDykY0ZsaSjAgUP5mlSGAOGlJrmekO0XqHB28Lme34MebOEhPuZOgg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Previously zram write() was atomic which required us to pass __GFP_KSWAPD_RECLAIM to zsmalloc handle allocation on a fast path and attempt a slow path allocation (with recompression) when the fast path failed. Since it's not atomic anymore we can permit direct reclaim during allocation, and remove fast allocation path and, also, drop the recompression path (which should reduce CPU/battery usage). Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 38 ++++++----------------------------- 1 file changed, 6 insertions(+), 32 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index e0e64b2610d6..6384c61c03bf 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1724,11 +1724,11 @@ static int write_incompressible_page(struct zram *zram, struct page *page, static int zram_write_page(struct zram *zram, struct page *page, u32 index) { int ret = 0; - unsigned long handle = -ENOMEM; - unsigned int comp_len = 0; + unsigned long handle; + unsigned int comp_len; void *dst, *mem; struct zcomp_strm *zstrm; - unsigned long element = 0; + unsigned long element; bool same_filled; /* First, free memory allocated to this slot (if any) */ @@ -1742,7 +1742,6 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) if (same_filled) return write_same_filled_page(zram, element, index); -compress_again: zstrm = zcomp_stream_get(zram->comps[ZRAM_PRIMARY_COMP]); mem = kmap_local_page(page); ret = zcomp_compress(zram->comps[ZRAM_PRIMARY_COMP], zstrm, @@ -1752,7 +1751,6 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) if (unlikely(ret)) { zcomp_stream_put(zstrm); pr_err("Compression failed! err=%d\n", ret); - zs_free(zram->mem_pool, handle); return ret; } @@ -1761,35 +1759,11 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) return write_incompressible_page(zram, page, index); } - /* - * handle allocation has 2 paths: - * a) fast path is executed with preemption disabled (for - * per-cpu streams) and has __GFP_DIRECT_RECLAIM bit clear, - * since we can't sleep; - * b) slow path enables preemption and attempts to allocate - * the page with __GFP_DIRECT_RECLAIM bit set. we have to - * put per-cpu compression stream and, thus, to re-do - * the compression once handle is allocated. - * - * if we have a 'non-null' handle here then we are coming - * from the slow path and handle has already been allocated. - */ - if (IS_ERR_VALUE(handle)) - handle = zs_malloc(zram->mem_pool, comp_len, - __GFP_KSWAPD_RECLAIM | - __GFP_NOWARN | - __GFP_HIGHMEM | - __GFP_MOVABLE); + handle = zs_malloc(zram->mem_pool, comp_len, + GFP_NOIO | __GFP_HIGHMEM | __GFP_MOVABLE); if (IS_ERR_VALUE(handle)) { zcomp_stream_put(zstrm); - atomic64_inc(&zram->stats.writestall); - handle = zs_malloc(zram->mem_pool, comp_len, - GFP_NOIO | __GFP_HIGHMEM | - __GFP_MOVABLE); - if (IS_ERR_VALUE(handle)) - return PTR_ERR((void *)handle); - - goto compress_again; + return PTR_ERR((void *)handle); } if (!zram_can_store_page(zram)) { From patchwork Wed Feb 12 06:27:04 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13971034 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0F8A5C02198 for ; Wed, 12 Feb 2025 06:32:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8A3D7280004; Wed, 12 Feb 2025 01:32:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 82C22280001; Wed, 12 Feb 2025 01:32:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6CC47280004; Wed, 12 Feb 2025 01:32:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 4896D280001 for ; Wed, 12 Feb 2025 01:32:36 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id C45DD80C8D for ; Wed, 12 Feb 2025 06:32:35 +0000 (UTC) X-FDA: 83110323870.22.98EC05B Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) by imf18.hostedemail.com (Postfix) with ESMTP id E211D1C000B for ; Wed, 12 Feb 2025 06:32:33 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=SGmR7Cst; spf=pass (imf18.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.181 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739341953; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MP/iUhuHVLA6wFxJBrfyM09o41KGkhO8ufnVXdUQ+KM=; b=2BJb+sTV/lH4sJgO9/PmUG5fHlVpmTKZBjIcltcUt8UTTw3nbmpWb3KpHtudBb7Pzn9/kL G2H3PdjpXmVnSY58zaoF+3ge0QK0B/ZQuSAJbVnmOUJNvbrM6iBWEtk12s8WXCn8l6NMc9 jsLhFCJbPW2Ktn1e/79lMfPC6VxJg58= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=SGmR7Cst; spf=pass (imf18.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.181 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739341953; a=rsa-sha256; cv=none; b=SF7DuodHiHXv/x9esJIjBrvUhWN+fIxUToAL28aOG3Z/7V+NMvexMfFSnNfpCxVa65IQCj wh4P7598CyPK2ZqexUEMLF5n4SoEx2rccdu1e6kGDwc6nceHP4Xsa8PTjQ3xEkvCIc4fhE Y8scxD+G3lXRYT1gJn1a8xiIAC/e9AE= Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-21f818a980cso57281705ad.3 for ; Tue, 11 Feb 2025 22:32:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739341953; x=1739946753; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MP/iUhuHVLA6wFxJBrfyM09o41KGkhO8ufnVXdUQ+KM=; b=SGmR7Cst+/WCOBs//xeN9gBBd2A2DEdpu3uzPAFsW9xNRtWvlBGP3flT7Ar4IILtwn tK2nS7h2dUj76BqlJltB1tiy6IGz6GPkO9O2/sDe6wdIbQ/to0IbXm2jqv0uceYnIM3G tJCeU5t0AXHRhQL7Dl9sl2SBlUD1GAAqZcjAc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739341953; x=1739946753; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MP/iUhuHVLA6wFxJBrfyM09o41KGkhO8ufnVXdUQ+KM=; b=CHX4Z/0kHmydvI42bCYnBHpwT2L4eIQ3c4sZeXrFHGiEsCp5loIZVn4VOqJspg5UgF 1rvELBR4cna3SDbsKcemTgic06mlbMX6rMr8x4IRHcQmb+zaUl9ikPYCnvkS03e4h0Nd IagJ4hfmHFc8o6hjiSIXITFJvSVjcWjwgj1fFnrwzJHPwRgV7T/izPG5Gz92/ZmpCQtu kE3H2J9HK5yJiCKa+dnViM6bWSCWKfNBa/vdTPZ7AcatKShvyRfnkmhUl2tUzZvB51k1 w+St3E1RgObSY0xfXY5WL2K3b3NHbMk9yI90hVG9IdtAQPSaqHCOeOSvkmEOMYF7IsN/ ckoQ== X-Forwarded-Encrypted: i=1; AJvYcCXWgoukVFDv1jSWGaBAFwO4qrspBEb00a5Ko75b8me8tfblVlaoDwA9IeG8ZcEqkKJyjO5wtP3fYw==@kvack.org X-Gm-Message-State: AOJu0Yxug4QdND/RfT7smRB+PTYdBlCEwsm5sEspZQBhGSfPsHie3FAJ kGB2t0LVLJRQKPuKGL6g+oIaOzt2VBjXBWLbpDdqiWCYDUELNE6O/0nmPrM4TQ== X-Gm-Gg: ASbGncuuYH7nFpmRZdkjwCqGLTgkSDhYIxFfbgj4yUw6rcKDhD9lL2sZH1+9c+/V7E4 bi3GPKiYvDZX5sypoV19YvgO4908p2lmABAZLO0yTlzfSVgBfhzi01e3jH6V9PU0E3EZ+dZl7Wh 1WKR7Bpok9fKwtzDxcIG1QPivEduAP6y0aZ0NWV65P4Dn099ZjyFjWBZxrZI0GpsXi3ARMGpRlx ed8fQQBo3DnLWtNI8A/RmETFVkh2KAznILBCp520xaQiymmrBHulFyxdedrqCNH4lg6q1F4HmHI idv7RRJEWTsHKtqtGA== X-Google-Smtp-Source: AGHT+IE7SyVtE81IRMZ7QuZhJlVL0CH3JofgG8lKolY4V/dbLjW0GVG8rwvz0sFKRCNuv1wj7rDucg== X-Received: by 2002:a05:6a21:6b17:b0:1cf:27bf:8e03 with SMTP id adf61e73a8af0-1ee5c840a69mr4005891637.26.1739341952903; Tue, 11 Feb 2025 22:32:32 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:69f5:6852:451e:8142]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-73048ae7f6esm10423017b3a.74.2025.02.11.22.32.30 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 11 Feb 2025 22:32:32 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v5 06/18] zram: remove writestall zram_stats member Date: Wed, 12 Feb 2025 15:27:04 +0900 Message-ID: <20250212063153.179231-7-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog In-Reply-To: <20250212063153.179231-1-senozhatsky@chromium.org> References: <20250212063153.179231-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: E211D1C000B X-Stat-Signature: as3djy5b5syhbqpxaeudfgoi9wn64kqq X-Rspamd-Server: rspam03 X-HE-Tag: 1739341953-504418 X-HE-Meta: U2FsdGVkX19TiagQ3Yec5sBPUVWVis4BXzN7mXOe2M8t1hpMIXuoe3ObJetEFGu7efxkcfPPjMaSeJ11Dpqwz309RNJ099n5jpJYV5EG8fZZZkkq4SkHwlFNIGCHy1cWTKpGpBZIFQTEUjmRG4XLrcsEXRgOhgVtKYp2mKgEkWe8QMF5536Uc5vjuwkfQ8peFrrMGTreE3IFamWLtJydBYTi1dbow9xBloWKIjubZzShfhVCmA6V63sB1/cgSiUgy0jy3Zz2lr3DkGbZ82P0/khg9RXoERENsnoKp8/539/1b4bRpwt4hviG81ovikcsuPQ8FZHWR6tum2VaN54T7iHChzOYfDqru0sYhTk5WU9MUTm2SYOwNJ2j3phri/Dlf/dNpwMKr3qJkd54v76eQr03Yxzg3HDV0hLIcgXx/AwTEar7Kv5J7fjwmpaBqDnNGaqBqbHEFsTu2XATa3zHUswwg7ffFyurtBb+gm4A9wBYguimvaG9vGAAX95R//MTQPleXVM8AiYtCo6/PHElCXkOv/GBewp0PfmZKPqQzKpLnNjZZkJAUK9qwUS0eQp58kCdT/q3bqAVQQA86FLJE7mo0yot7jp9jS/CeQH4vCCbxRfVR9Zx8R49T2KjtiK+hwoDIqnKcFsnFOoiX8vOpwvxcCAcuFA04zhlQXTl63wQZVPR2rHURNaX5g0tbWdHrBBfAkAnh4+IIP+7xDkAlzfUAl25CrqtTw8jGPx8rNj7HiVWJ1W9ZeTpRFLK2iVSN9mXtkzzzCayuiIdOFFPyF9uFJJR/tKlVb0Cfkbf2k08DsEAciaGNr83amEx/Sr8lAiYXU6QuWQAX65E4ovD0AuNC8656C9UhbwwSkZjL8mDYbQ7hkxY+0GNuCMAyoFfBRdKyOqdewsKamuXK7XuLZKOF8m2MmaArZNg3tpMy3KnkR8tgOVm9wc2An3IoqX6p6iURtgA4K6bMWzeL1q uCzl5+R7 8jp6KQzta4M1D8OsQ3uPuJDQ+eP4k+wCARF3kQrIx3Q+RMi7yTY1+U55lv8ik32z5H97xTuMnmYG+qsMK08WnOTVgLfTkZpIxnJeFJL761zCpnDTLNZeW5j04fxkzyKiXB7H/2LFImQu6kQBlrN83ZBk6NlDurbBSh8vKo+h90p25GpX/JHUN8DGZpjzKKUcSY7puO+HMCU4fDdtPGs6I0e9t5Tg+ISNabh/X17ErTprbCQEGKv6XGi2OZsZvZp3SZb6Sshqfz8MHhqOiERJ0dE3IyZEWS/Nz/3jkL+R/B1tH9EKhYmRP+RrUz29icAYCFuABmSkaqiZH8kTfwi5i1ZNea+TLmAPqm2hxe0LygVu+a0A= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000019, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: There is no zsmalloc handle allocation slow path now and writestall is not possible any longer. Remove it from zram_stats. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 3 +-- drivers/block/zram/zram_drv.h | 1 - 2 files changed, 1 insertion(+), 3 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 6384c61c03bf..7e2694079760 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1438,9 +1438,8 @@ static ssize_t debug_stat_show(struct device *dev, down_read(&zram->init_lock); ret = scnprintf(buf, PAGE_SIZE, - "version: %d\n%8llu %8llu\n", + "version: %d\n0 %8llu\n", version, - (u64)atomic64_read(&zram->stats.writestall), (u64)atomic64_read(&zram->stats.miss_free)); up_read(&zram->init_lock); diff --git a/drivers/block/zram/zram_drv.h b/drivers/block/zram/zram_drv.h index 97c98fa07954..b9528a62521e 100644 --- a/drivers/block/zram/zram_drv.h +++ b/drivers/block/zram/zram_drv.h @@ -84,7 +84,6 @@ struct zram_stats { atomic64_t huge_pages_since; /* no. of huge pages since zram set up */ atomic64_t pages_stored; /* no. of pages currently stored */ atomic_long_t max_used_pages; /* no. of maximum pages stored */ - atomic64_t writestall; /* no. of write slow paths */ atomic64_t miss_free; /* no. of missed free */ #ifdef CONFIG_ZRAM_WRITEBACK atomic64_t bd_count; /* no. of pages in backing device */ From patchwork Wed Feb 12 06:27:05 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13971035 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 80002C0219E for ; Wed, 12 Feb 2025 06:32:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 08B94280005; Wed, 12 Feb 2025 01:32:41 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 03BB2280001; Wed, 12 Feb 2025 01:32:40 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DF7F6280005; Wed, 12 Feb 2025 01:32:40 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id BCCF4280001 for ; Wed, 12 Feb 2025 01:32:40 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 7AEE0B0E4B for ; Wed, 12 Feb 2025 06:32:40 +0000 (UTC) X-FDA: 83110324080.16.7EA00AF Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by imf06.hostedemail.com (Postfix) with ESMTP id 9EA45180002 for ; Wed, 12 Feb 2025 06:32:38 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=aVik6pD2; spf=pass (imf06.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.180 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739341958; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=cyCqO8rpD053xgod9EGtthWwAT9KzKKgkQ9kmE/51BA=; b=VmGmTJ5/zGdxxHHbKwXRuj+BfoYio12P3GFYuHUAyxZocxCOeb1HqXJgfhjaD/eRnnhON1 UfsOVYf7//wxqoQdhcf3F0PEDyca0Yijd01lRogjdlU1JlGoRiKIMicBHrCQrHhNjs42LN QAVy/aDYKMZFE0RE+4Vjf0Xa48sJemk= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=aVik6pD2; spf=pass (imf06.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.180 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739341958; a=rsa-sha256; cv=none; b=faDFMNk1zXkgxWOz7n6zJ/KxVS/AleiyMwdv30Be+qNHwKsW53W54WQILAVwXGG76fF7pz gTpRFxrnWbo0f5KZl74W9YgnQpzGeJLevqqjNWLiWR8x9hs7v53mrsTjHQS/mBMnHNmC9x MOEsF3WQyOZK9LdSpX9273AAqxMHnQU= Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-21f61b01630so7534965ad.1 for ; Tue, 11 Feb 2025 22:32:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739341957; x=1739946757; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=cyCqO8rpD053xgod9EGtthWwAT9KzKKgkQ9kmE/51BA=; b=aVik6pD2GFQvZqAkZdLxCrmX3teTH1kF9+GE+xEY/J/Qvqb0FJ6buXYzbHYDrh3m2R f6Dzd8tiihp/ivkmOVqp9em9qEWoUFRc3zcfnv2yU2Y1Gyxp6e7P3Z8UCDa8eIU46v3j 9Q5YW5Et4bD9/LyF2Xb8ZEEkaVud4Hx3Cd5dg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739341957; x=1739946757; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cyCqO8rpD053xgod9EGtthWwAT9KzKKgkQ9kmE/51BA=; b=MFmQuOp6tLGGoLhO2OvnirWcPCHiNSvovZurhfDh2dpW/pihkOOH1is/mC2rMfHE6X N1Azr0JAtpIpY8GhhCUfglQ/kI9/F6/jcYYqyMI1kD4erQDI6L2OGxffIbdIV7BYZwLx c7PsDbKy0rGZ/YA8A4pLkJPRxLgx9JihTGLSuLV69O0z+piml00MrrH6v92fxtVdwRWd 87VLsivrO9pUD9cy63uKxDmtEWhUrRHbB8h+MQsidhoKfc/EgkUirdo1dIPxCeaqd0n6 d0e3FzjunAgtxX1cBH80HJ9fT0IZmixbXuzXw6JDUWch+ZnOQa0LRko5/gK7W8sCKsYA lavA== X-Forwarded-Encrypted: i=1; AJvYcCU0DcNVXYQmaike910AzB1K8x38EJl6bJPjZCeA75Bgq9OdFS5XW/Y3RDAThwmbZqkUSbI76+wXlw==@kvack.org X-Gm-Message-State: AOJu0YxW+kg7PlIDqpnW15oHDQTQnAScHaHvh7yXn5AP4eNmCKqwl3UP 4WNVRCTzeW9oJ/qmuDIzGzi3NsgjCvz0PINFgOtAOHw8pqtqS2FWSJgxvPiKQg== X-Gm-Gg: ASbGncsJcPTaUmYXkN9yG13Emqm3CEjSQoofBpwETOmtkqiwxudhJpZKqtm5Vm//0gY 6s6KWuvJ1BRbdhu8xHDYOY5qF059c/E7qktQzMbDXJJttx7JFSlMSK4V0LWP6Isn6f2mqInpRlZ mkf3ANGTsM/Cq8zvyyqgC6kGovyaCTLd2DoR9xEryriZz4lQ/mFL0n8hVOMPtm0i+kEqCi18sRk FoEAScXt7q+/EmzyEBE/GPx9tjoJacHHnjLiVBs8XDxZALQfcE6ZYAV13nDagW5aaFY9AszIac1 k7rW1ylXeSaI83FKPg== X-Google-Smtp-Source: AGHT+IEC8Cc8dAP8Ht+pVcdn6EfMAD6X2400PsIqNRPLot1jyPj+LyGyqkVdRWRCN4K6E/Zgz3CsHA== X-Received: by 2002:a05:6a21:150d:b0:1e0:d5be:bf75 with SMTP id adf61e73a8af0-1ee5cf4b707mr3005496637.17.1739341957604; Tue, 11 Feb 2025 22:32:37 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:69f5:6852:451e:8142]) by smtp.gmail.com with UTF8SMTPSA id 41be03b00d2f7-ad5489a2ddesm5668053a12.53.2025.02.11.22.32.35 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 11 Feb 2025 22:32:37 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v5 07/18] zram: limit max recompress prio to num_active_comps Date: Wed, 12 Feb 2025 15:27:05 +0900 Message-ID: <20250212063153.179231-8-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog In-Reply-To: <20250212063153.179231-1-senozhatsky@chromium.org> References: <20250212063153.179231-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 9EA45180002 X-Stat-Signature: 61y4msceof3uir3kj9cp3koxubay4mex X-HE-Tag: 1739341958-371476 X-HE-Meta: U2FsdGVkX18yXTNW17zln8NgC5JND1tDIsk9SoTrwONUaB167BxCtgHSepqBg713QgdVCbvETS+sT5g1veAM4nR+3nXRe2Man7SRPRq0l3b0WwOP8InwQ3K+M3nx6sicFdovhIaiMKBQ8TYKARTp9S5iqpH8CNZCnb0yP/DXmKPGODn2kW8N4TujYEn9hbr2WoJ5/xKEWrfsO6nKaArv1CGwdF7/RwQAJv+tqUSQu1/jizZIBjjNa2Z8d4+/YOzd90bBst0YdbxTohjpM92g9cuTWv4Xh/W6FBo5sR8GP2uzjYqo217r4EprfyPGygCNKmd/0ZP90J2EXUbkJfuuRTk2stabreT3Fu2HN3hqRZXQuX7TUbIkoytRAb3oANYLMCX+/g+TKFVJoameDUQBEK3c5T80S4HyU8vF3JF3J3Q6MC6dgVl5FQ6F6SqCIGXuNYa1cruBuL07FsKMEV9l75VsirjPUB365WmgBsY/3CV3JOnu8V8LLYRxZ/zI/KqzbmXTxl8nuS6wxj6679b1yBY1Weejh/k0Hn6yBSUR0MKpRnne9UwEbX8UGsJXtSFk6C53JQLFrwQIgxA4vUI0ATYCAKAX5Onv1nVbcPHvYEHoSKF0quG6mIfk+IfVSrdVDQR4KtDCaUJWqpmjiGIjeNL2HKsFpANXBR/eCnmQ9d26fBH7vqwQDcVT5ME5rPAlMNGSb76j1lNlMbnBNcjRF8Dl+5tYI+aXHf26Zl7EezdnLJ70Q+BCAG68EugHAv4oLBG+ksmwUMvInnMlaZJTZdXcZwbRiDI+jpowTGr5TfjRXtp26Dzavz8AM0J6ViGw80LushXTAPfj5LTZE7dH0ZxrJIblWqnps3l+15bGzZYnonyx0PvAMf2UMlDfVK43Lp24NdjNy9TZ9qBX4cIn2W92tebxIqAVDSLe1KBCe8am8/8ckp9PXkAjtPFaWXo2WabV6Ktnunog3tBe7rU 0EO6mh5Q pdwy9Li7LAz6CfuZh/qqzFzoTh/I5baTjgqK5fAbDvvH/iiMdWNNOTzFd91rWK/OwbLRW/cVuq9hW/yg5T8E4ZJrqzeKaBk+CKWeXwEDdS4AG9Jkouir4L2mA6FvxEBNrYIiwix0wT7igIR3sNmXi7MpxH2rPhL913MSzzK8azrM5CLKCGHVs7s/t3U+ZyqwT/2y7V47lQRTTGJM2Us8CaK3C7An3rPeoL1xjKPjsn+SJvYkx0j+73TrhfEDBQh+bkSZwZLGQN+umeCntg23JqEpUKecjdlEh+7qG51t7KrYYZ5gFKxERGk5qIQ6wd8jzgEpxcZ6CbKtCm8OUcdrh0Izycd3yEHgSZ0ZNVQNR3XL1RtU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Use the actual number of algorithms zram was configure with instead of theoretical limit of ZRAM_MAX_COMPS. Also make sure that min prio is not above max prio. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 7e2694079760..27148f3e5ae9 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -2026,16 +2026,19 @@ static ssize_t recompress_store(struct device *dev, struct device_attribute *attr, const char *buf, size_t len) { - u32 prio = ZRAM_SECONDARY_COMP, prio_max = ZRAM_MAX_COMPS; struct zram *zram = dev_to_zram(dev); char *args, *param, *val, *algo = NULL; u64 num_recomp_pages = ULLONG_MAX; struct zram_pp_ctl *ctl = NULL; struct zram_pp_slot *pps; u32 mode = 0, threshold = 0; + u32 prio, prio_max; struct page *page; ssize_t ret; + prio = ZRAM_SECONDARY_COMP; + prio_max = zram->num_active_comps; + args = skip_spaces(buf); while (*args) { args = next_arg(args, ¶m, &val); @@ -2088,7 +2091,7 @@ static ssize_t recompress_store(struct device *dev, if (prio == ZRAM_PRIMARY_COMP) prio = ZRAM_SECONDARY_COMP; - prio_max = min(prio + 1, ZRAM_MAX_COMPS); + prio_max = prio + 1; continue; } } @@ -2116,7 +2119,7 @@ static ssize_t recompress_store(struct device *dev, continue; if (!strcmp(zram->comp_algs[prio], algo)) { - prio_max = min(prio + 1, ZRAM_MAX_COMPS); + prio_max = prio + 1; found = true; break; } @@ -2128,6 +2131,12 @@ static ssize_t recompress_store(struct device *dev, } } + prio_max = min(prio_max, (u32)zram->num_active_comps); + if (prio >= prio_max) { + ret = -EINVAL; + goto release_init_lock; + } + page = alloc_page(GFP_KERNEL); if (!page) { ret = -ENOMEM; From patchwork Wed Feb 12 06:27:06 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13971036 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66766C02198 for ; Wed, 12 Feb 2025 06:32:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E1FCF280006; Wed, 12 Feb 2025 01:32:45 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DD097280001; Wed, 12 Feb 2025 01:32:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C2427280006; Wed, 12 Feb 2025 01:32:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id A47F7280001 for ; Wed, 12 Feb 2025 01:32:45 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 3B328A0EA2 for ; Wed, 12 Feb 2025 06:32:45 +0000 (UTC) X-FDA: 83110324290.05.161E9AD Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) by imf21.hostedemail.com (Postfix) with ESMTP id 5E20F1C000C for ; Wed, 12 Feb 2025 06:32:43 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=BBRMUlRy; spf=pass (imf21.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.171 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739341963; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GyNDz1JqW9fZ5TiB72j3x2sTVp+ebBvZsmBm0h5y3g0=; b=q7ITXhagKPMqLfk3huk04aN1vnREy5s7Hd8wrR5bCtue0aMRcVW3XL1lU9CYKyzON6qlbB yNOAyg4U+ljPqBE6U3phE2B+5rXRsFM5kLlQZ7dg0lFpb/7gnXP4H098Rs7QOLLfpIKURc 6Kbbxm+KEaAePrgpOrxgiE0t/+D9EKI= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=BBRMUlRy; spf=pass (imf21.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.171 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739341963; a=rsa-sha256; cv=none; b=7IwDbB1TLrQ2MdX7/VD/0gOyipttDaCL3aVv83GUAae1KzoYc0qRUudaIFKbN7UXM2M7GZ /qJOGuMyGIleTPYl+/ifHgj2qPFpJ619fG7nkpLuMprE05QJ5UirXzyN6OS5to3IpopY3X /6IB+ogohdXA9u2ZJLrGPk3/wBFF7oA= Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-21f49bd087cso87524105ad.0 for ; Tue, 11 Feb 2025 22:32:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739341962; x=1739946762; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=GyNDz1JqW9fZ5TiB72j3x2sTVp+ebBvZsmBm0h5y3g0=; b=BBRMUlRyuBKA/YOaubuueu+DehlSSe8ZQVESxmJj4f1FykvGcFvGpKohbNqMQ4brNJ I6vMdTm6YqkpD1MkY22Yv6gXcU10mqXikMueqgQ3TOuyQUwhQCAbt91vDaPbsXmV6/fL z1kjggE53+VJ9ZhGK3kWe+uEau+bsd431Qh+k= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739341962; x=1739946762; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GyNDz1JqW9fZ5TiB72j3x2sTVp+ebBvZsmBm0h5y3g0=; b=VLSYjEnCmD3sB8LR3DyZDENgbZw+1X32yXcom0vGsI5RRsV2qYn7St+ICp4UL8wt3K itT0IhSR0Whs/9zdA0fT/CWPP3L7BtNfZVtqEsxA9DTSxw7xPFb/WXLaHtRUjzxlplPD XFBNpOTMfUl+zs4HUMqMvmSEP5K8S3nNFSrkCmIJY6yPM7IOmk0p2CjeYYLwaXMqnrec VV8KuJzu1QKYKodeeelIrvxo8dx2TkxkUDDTn6fNP18rMzT9tMx/LTa+S6VxYoJNNjud syxSe/heliUpoSitQiKARteIQWNQByrLgvTlqLc71tdQGMJku1HkaHlWTwGU7bahvc5P 15Jw== X-Forwarded-Encrypted: i=1; AJvYcCUZa6UKJb+0FZLo0erl2cBvSCNrr+YgP71ZH0H+F3S9QBgRXr31BPBBtNYOno58VoOLiwiJCKKBNA==@kvack.org X-Gm-Message-State: AOJu0YxbyQ3J/jy8bpOmFw/q71vw8KlR0NZhvsXL+lX0N0aX8fCO3vQ7 C6C18D2ScHLJouCUhle1sl9ABnmfHejWz0jeKO0PVn74z0YcCNtV4B2jNIKwyS9f43JCufoCvak = X-Gm-Gg: ASbGncuqkikluMCHo3VK0010/gErxWbX+2fcg4V4T9wdyv6xjYjpIWfcOk7BFtK2WQO ZZnP/pm+bkPii9dj4DuDqj/BdB/Z0nvihb7gZ34oLxSyMsmtxJ5UyBWu5MElSTAqf+JF3j8U+NR AXMpi8J67u+oHPg/7kTThse5w1vVJDyy6tTi4Rb4YxyNmeBYENVdq8/UP/jz0GB7f0Dne5cQEES VrJomF2HpM21/Xw5XHgkKNyuCq4FG9bAPFYLt6bSGs5eFG+t2WTq/GubHyw+/RHgeLXr7p/5sMc NpLEUKvbhYHX6ZEnMw== X-Google-Smtp-Source: AGHT+IHuvm8HCsSAsDPLpNBHM4fkgErNxht0F4ETwuQK+JzFh06XIBG0WxJwETNpSs2/VqRQClDZHQ== X-Received: by 2002:a05:6a21:600b:b0:1e1:af74:a235 with SMTP id adf61e73a8af0-1ee5c790de5mr4045324637.24.1739341962338; Tue, 11 Feb 2025 22:32:42 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:69f5:6852:451e:8142]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-7308e4881fesm4730281b3a.46.2025.02.11.22.32.40 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 11 Feb 2025 22:32:42 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v5 08/18] zram: filter out recomp targets based on priority Date: Wed, 12 Feb 2025 15:27:06 +0900 Message-ID: <20250212063153.179231-9-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog In-Reply-To: <20250212063153.179231-1-senozhatsky@chromium.org> References: <20250212063153.179231-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 5E20F1C000C X-Stat-Signature: qrde3apaduf719ybcu67ekk8gs7id547 X-Rspam-User: X-HE-Tag: 1739341963-266637 X-HE-Meta: U2FsdGVkX1/Mxo7dd8NWuyffX1BtZsOyd6xZ+9k1Cu48rIgYDYKzbO18Jn0/HeB8EaX7HkS05pBpS89XUi1Y4K1tZdcCOjpZG66tT77xrOKhksJluv7GQMpsCz2V3i2+Z8UF9OInX8n3uEp70hnmIm/sxU/tJB8bZJwxiyIrGiKLhzdkhm22rMi26qiOlSKnZ09PGk4oMdn9hDfWMN+lpiviwoHePbBCeNepKcq+eSoRbakSzY+2f5bYOethBA1nsP4BM9HU5oKwGy1kFZ3leJpzKjJ616fYeR9FG6LIu0O3eDrQqEjFJiLWQIYcTiFRTCeTS8arltmVp524Wcwt4W2jwTFFohOKtkRjsJ8rRamZ+gWv+lcClKys2dio5t3bEMOilsGKfRtqeVrvWLXXRkqjNIIsJZK08HdIzRrnSY4jGJ2sb+A3HIPaedWbNJw6np8W3jMl83bWkWimc44yB4KpbT/tu8SwOmHsFLGAAdh1PshHR2cNtGYWjicIiDMSZYBSlq14xmf/komW4nV0Q+Eh9iifNATgqKcs+Es5jbrpH7YQkAeAu9/sDZ0blTQk6rlNDSTNpAub3ZjqVy3JoDCD028lhIQlx70xrpN9ygS88WoLgpd2kOprcNCU8Retqa7Y441gjqe8HdYzcEDxOB7kd+3+Q5Si0vS53FOpnEBf1nOPL1ghr0vXDk5B9pZaNJUk603vcp3XSZpHX8gU/oLQdvQnnRhK5fmOhyaShIcP5XN0L5pMdAkmUD5SudvaE++bC6n0vgvKrR4KTY1rYnTKTp8EG6tABw/9VDz7dlTYPzB3hoX+30CWPOXn6t5ryYbEEk9HRhx9CPSruVErUJ0j+JGWKszzpZ1C/74MFcayt8TXcp5KboIFi3ZLhJdQAKLG3WvOyxlEjIfZdFEuAW9rXmCHTRa3mUUOGqJ3KXMMcvaA6+KEG93KLuIwVsj1Q2MMpufNmk9l5lCGcBW C0L07Hx6 0jWLLlLKr2mdey+MVOmUDLRPFH0m5m8hWYkT8PD6D9EftUrNn9n6zx6gx/PNQwRoPqa/lF3vX5CzTUVRID6PtxyFwwnXGY4NaJfasfO1IvAhstjCVV1jjVR5LFJ2bjL8vKEFSsRWvJsN0N/6Vi8H3Y5MhCJytPQY66BGk5J/q2P6kP5yvY068HZqJPXxquXdKFom74ik5+GVSVSxb8pRbldbppyFVuNHcRiY8dlzIm7LdQ9/zcbNyno58f+X+zsqF4+mGQKWQYerhqx+tIb2Kfri8uR9c6C+GtAdc4xOnnTXhG1DWwRpjPnH0Tw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Do no select for post processing slots that are already compressed with same or higher priority compression algorithm. This should save some memory, as previously we would still put those entries into corresponding post-processing buckets and filter them out later in recompress_slot(). Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 27148f3e5ae9..31bdf5e0ff74 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1822,7 +1822,7 @@ static int zram_bvec_write(struct zram *zram, struct bio_vec *bvec, #define RECOMPRESS_IDLE (1 << 0) #define RECOMPRESS_HUGE (1 << 1) -static int scan_slots_for_recompress(struct zram *zram, u32 mode, +static int scan_slots_for_recompress(struct zram *zram, u32 mode, u32 prio_max, struct zram_pp_ctl *ctl) { unsigned long nr_pages = zram->disksize >> PAGE_SHIFT; @@ -1854,6 +1854,10 @@ static int scan_slots_for_recompress(struct zram *zram, u32 mode, zram_test_flag(zram, index, ZRAM_INCOMPRESSIBLE)) goto next; + /* Already compressed with same of higher priority */ + if (zram_get_priority(zram, index) + 1 >= prio_max) + goto next; + pps->index = index; place_pp_slot(zram, ctl, pps); pps = NULL; @@ -1910,6 +1914,16 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, zram_clear_flag(zram, index, ZRAM_IDLE); class_index_old = zs_lookup_class_index(zram->mem_pool, comp_len_old); + + prio = max(prio, zram_get_priority(zram, index) + 1); + /* + * Recompression slots scan should not select slots that are + * already compressed with a higher priority algorithm, but + * just in case + */ + if (prio >= prio_max) + return 0; + /* * Iterate the secondary comp algorithms list (in order of priority) * and try to recompress the page. @@ -1918,13 +1932,6 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, if (!zram->comps[prio]) continue; - /* - * Skip if the object is already re-compressed with a higher - * priority algorithm (or same algorithm). - */ - if (prio <= zram_get_priority(zram, index)) - continue; - num_recomps++; zstrm = zcomp_stream_get(zram->comps[prio]); src = kmap_local_page(page); @@ -2149,7 +2156,7 @@ static ssize_t recompress_store(struct device *dev, goto release_init_lock; } - scan_slots_for_recompress(zram, mode, ctl); + scan_slots_for_recompress(zram, mode, prio_max, ctl); ret = len; while ((pps = select_pp_slot(ctl))) { From patchwork Wed Feb 12 06:27:07 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13971037 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EDB72C0219E for ; Wed, 12 Feb 2025 06:32:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 80657280007; Wed, 12 Feb 2025 01:32:51 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 78F5E280001; Wed, 12 Feb 2025 01:32:51 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5E159280007; Wed, 12 Feb 2025 01:32:51 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 3F141280001 for ; Wed, 12 Feb 2025 01:32:51 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 0319C140E11 for ; Wed, 12 Feb 2025 06:32:50 +0000 (UTC) X-FDA: 83110324542.15.A595CDA Received: from mail-pj1-f47.google.com (mail-pj1-f47.google.com [209.85.216.47]) by imf20.hostedemail.com (Postfix) with ESMTP id 207C11C000C for ; Wed, 12 Feb 2025 06:32:48 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=RqPfZycC; spf=pass (imf20.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.216.47 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739341969; a=rsa-sha256; cv=none; b=JoeLkiF0mW2KmFBTWuUwqptY2V+KW7X/KcT773+pD6QdUH0GwSDURKD89MJ22S5wBEdSQf pV5SqpnRrmJ1SHA2nyKfFI19mF878r2ix2yI7icrlqReDgenoMuJnkn/qi9gOkGiJmNDQk AQ47dsj3J88nfBggXkn8eFA72+ZwZ7o= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=RqPfZycC; spf=pass (imf20.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.216.47 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739341969; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=cLnpAJqzl8agqjBt4hqeQuPEXwmT47X6Rb4B201CoRE=; b=3FnB86zPPS6uw5Q32wWAekIkDUcIC3dkj2XUNjCBO8lizjTIXbsWkdrczFAvp4/pKeZUGA 8u3RZ1QbQYu2d+IvRV2mdxXnr8OAT58VBvU8emEfrSNlKOB35nIosivoWB6N/BZkC+iFNN fmBFQU5M/ckwcWMwnZ0CSJrPS45RaJM= Received: by mail-pj1-f47.google.com with SMTP id 98e67ed59e1d1-2fa19e1d027so8560809a91.0 for ; Tue, 11 Feb 2025 22:32:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739341968; x=1739946768; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=cLnpAJqzl8agqjBt4hqeQuPEXwmT47X6Rb4B201CoRE=; b=RqPfZycCoaPc2oGwF6AHV3f1A7D/7ZI1QL3q0n1L4pSjoEJnA4A3kTRJ5rx6IxqO3H CDd2Z+lwxeP1CuD3locCzb0VpygyHmd9f8ScEjbcr9Ja4r8AGue+t/sh9mE9uQqSSLZ/ 8a/euQMWYr96GcULAQg9OnycWsXczHl9YBpBg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739341968; x=1739946768; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cLnpAJqzl8agqjBt4hqeQuPEXwmT47X6Rb4B201CoRE=; b=vXW9DbvlUZl0RjLp5aABqg7MttsBdjl0brS71igkTuKehFC8dRRgebbE5UapE7jvtD mNN6VAP60JL2YfajC+VQMfgDv4xElsKbWR401jxGBjiNrP4CksnXKy+tLEq2EB8i2wr1 EKnVg4faz1pH86lmedbFkxffNxLX7yRDRpjCVS9oOeu4lWEC8S0aMS4sLOvmT3xKn8f2 CDGB9jxEumIizdXVDrMwfcsDtzbhgx1kGsg0tGtsKbaE0swppfVcyyhZ8qv5s7udTcYh pcWWhVSg6B0FsVXlrL8VQRTzGHuw1Jl8SzQKgJMGjQ3UBZQ7muQWvpMbff4TZA1S+GTd 2oXg== X-Forwarded-Encrypted: i=1; AJvYcCX5keCpCslOGcFnAPEAkTR4npdA2gdPG2BaENsAN2PcRqKKWl20vivo9tKrDDK2DiVDX/tSpSYr/w==@kvack.org X-Gm-Message-State: AOJu0YwXGnV2ZujrByuhza9u9wtSCWxLivljshDyl7UXi9dKX5jLE0Is +iPqUFr4Lb0Sm9/RhnBEGqF8Z8C9xq6G3v0fbeVZgsro0XqTx6EF5R/IcvRb7A== X-Gm-Gg: ASbGncts6/qHP1vj52obAL0hBzP7KQfTu55oJHn7gEAPcBTWg3SfBV0v2bdcVUdLR9F UFyIABUQzk2E4DR5BOyp1sdoxbbRJe3PlwdJ60U0Hb0lsFQNeKDXfGgSPnMv9OKWq4KdWK2vCax ujaekgbUPOqsTrMvUorylaRzVKMjqfeFl0hHaqGF444nd4O2ciTjUfbeI9/JgDOuEIw166+bquB Zl2+d4+FygMbxzpCbEBcqbwIeh9jFPVXzGdpFo1a7kgRbm4h0dxgK3K/jyIy4Ihj3tXo74RciQX kgUspl1PGdPlRyHnQw== X-Google-Smtp-Source: AGHT+IHiT+N5GVxrJj2d7+UgbwxnPgo/4IOHt2XM/rWhD+WNLZnaiOqWIvo76vE7u+pn65YNGazKSQ== X-Received: by 2002:a05:6a00:1798:b0:732:24ad:8e08 with SMTP id d2e1a72fcca58-7322c376a72mr3048146b3a.1.1739341967995; Tue, 11 Feb 2025 22:32:47 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:69f5:6852:451e:8142]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-73084e182cdsm5759136b3a.99.2025.02.11.22.32.45 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 11 Feb 2025 22:32:47 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v5 09/18] zram: rework recompression loop Date: Wed, 12 Feb 2025 15:27:07 +0900 Message-ID: <20250212063153.179231-10-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog In-Reply-To: <20250212063153.179231-1-senozhatsky@chromium.org> References: <20250212063153.179231-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 207C11C000C X-Stat-Signature: 1p7sj3ijs3bwmtby53jthrt9h44yynij X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1739341968-842124 X-HE-Meta: U2FsdGVkX183JSpD4qXqqmeNeqwt0Bxotsuaa5Wsrq+PA5viRgQ8notssLkCovt2ztloKsJvSpnwtM+VojeAtgxsQa+J0TXHHZrKR4idQpd7qk/I3w9ObZ6uOE0TmsOSatwxoWJutvQpKdgH8ZrQLYV8GGFNwlANgDrpYcMpofn5M3Fr1Uo/TBDpVXWqH92QN/6DY2xeKMpymUYIBMHcvXcZGXcK6UjEusnrbF9p1Qgxa+XDBF8nBS/rzBJp0P/6wsoNniPiTdmV8Lk83lJzmsBNdPF8UhEm5ak+ksDyJUPwQOM42f42pqBgTrtv699rJ2HPVDamUnIJ28f6bqbaBrTtoLOruQVOZk0tR5s05aRYgOSyA1eovAoNpMq7jC8mXAh0uc1q+vtWUbBad2PTgKG7ETmzaLf80l5Tj/PnXUO3yNlfhYBde+Z6R3jJz9aKmZbsAWH7DFK8RUPmIM7ge+l6DauBC1bh8/acDZwBeQHI9AFtcCy2dKgTZO7mevTDkWlw0EjuyoviKV2M+N9qGGj9QDtuk2RAIEeuaZ+75K4xumdLnI8bhuEpjHHuKyFdSB0Bmua/50QSEKYNQXFIkEd9Cld4PPjYLF8pfmhyV5gq3YLq8Jop9r6NpdfPjh0GWw4GyGqvUUYweSsLAVezCOuK0knEfpawiYChZcmpz5QUK9ajeXemHMAfWI2TGt/Tc74QdzSAQBxY5MaDiHxnDxlG/xeVyrwaSwUaqgjcqW+Mw7G7hiASx+fAxNaHQ9JTRnMPJ+O7OFzA6cmQm/dqXHuhReLABP3Pe4rojXDSiErq/RvjgH8741CSwlhHrz6jSGS71Y53vzDldLI/RmbNBPCm1EH1GFGo6xC3ytIz204cY1bMpNuStA+Ghe3kGcyhw2FKSgX5iZNfLMHpGSwSndD62UXp9FWRh+DRAa8YNddbh2uJytdbBoyKDFeCsuBq9cCafZuhjehBVnfch7w V8RjiRqv YlgCE24x8MiAO0M6P10MlIcJEYnXz/Iyv3bLaMkAWShPEGZ6LYCf0OoPb+QFEYTuS34hiCLfSEyGajvj7NLEN02PPU02NvDMrcCGUa6tkYRyZ4lANgA23Y6zPlqNgYD7vwzwMcqFImwBAHstrYj2gP9LoAsvJnXRLgG2Zt2FvCsnb5wDf5hJLG9qyJP+nXuO0HYq5GbfpGK38jSY6ifHWHOGb7lQ8Vn8GzY2xdBEkY+w+yb11BCYCWrJUMmad3+fRl3EIKVeU39033447LwP1S8/WgXBL7Et9JdgiLYbSPHT7PVY2vWCRQ6dRMQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This reworks recompression loop handling: - set a rule that stream-put NULLs the stream pointer If the loop returns with a non-NULL stream then it's a successfull recompression, otherwise the stream should always be NULL. - do not count the number of recompressions Mark object as incompressible as soon as the algorithm with the highest priority failed to compress that object. - count compression errors as resource usage Even if compression has failed, we still need to bump num_recomp_pages counter. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 53 +++++++++++++---------------------- 1 file changed, 19 insertions(+), 34 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 31bdf5e0ff74..7c4c296181a8 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1887,9 +1887,8 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, unsigned int comp_len_new; unsigned int class_index_old; unsigned int class_index_new; - u32 num_recomps = 0; void *src, *dst; - int ret; + int ret = 0; handle_old = zram_get_handle(zram, index); if (!handle_old) @@ -1932,7 +1931,6 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, if (!zram->comps[prio]) continue; - num_recomps++; zstrm = zcomp_stream_get(zram->comps[prio]); src = kmap_local_page(page); ret = zcomp_compress(zram->comps[prio], zstrm, @@ -1941,7 +1939,8 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, if (ret) { zcomp_stream_put(zstrm); - return ret; + zstrm = NULL; + break; } class_index_new = zs_lookup_class_index(zram->mem_pool, @@ -1951,6 +1950,7 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, if (class_index_new >= class_index_old || (threshold && comp_len_new >= threshold)) { zcomp_stream_put(zstrm); + zstrm = NULL; continue; } @@ -1958,14 +1958,6 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, break; } - /* - * We did not try to recompress, e.g. when we have only one - * secondary algorithm and the page is already recompressed - * using that algorithm - */ - if (!zstrm) - return 0; - /* * Decrement the limit (if set) on pages we can recompress, even * when current recompression was unsuccessful or did not compress @@ -1975,38 +1967,31 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, if (*num_recomp_pages) *num_recomp_pages -= 1; - if (class_index_new >= class_index_old) { + /* Compression error */ + if (ret) + return ret; + + if (!zstrm) { /* * Secondary algorithms failed to re-compress the page - * in a way that would save memory, mark the object as - * incompressible so that we will not try to compress - * it again. + * in a way that would save memory. * - * We need to make sure that all secondary algorithms have - * failed, so we test if the number of recompressions matches - * the number of active secondary algorithms. + * Mark the object incompressible if the max-priority + * algorithm couldn't re-compress it. */ - if (num_recomps == zram->num_active_comps - 1) - zram_set_flag(zram, index, ZRAM_INCOMPRESSIBLE); + if (prio < zram->num_active_comps) + return 0; + zram_set_flag(zram, index, ZRAM_INCOMPRESSIBLE); return 0; } - /* Successful recompression but above threshold */ - if (threshold && comp_len_new >= threshold) - return 0; - /* - * No direct reclaim (slow path) for handle allocation and no - * re-compression attempt (unlike in zram_write_bvec()) since - * we already have stored that object in zsmalloc. If we cannot - * alloc memory for recompressed object then we bail out and - * simply keep the old (existing) object in zsmalloc. + * We are holding per-CPU stream mutex and entry lock so better + * avoid direct reclaim. Allocation error is not fatal since + * we still have the old object in the mem_pool. */ handle_new = zs_malloc(zram->mem_pool, comp_len_new, - __GFP_KSWAPD_RECLAIM | - __GFP_NOWARN | - __GFP_HIGHMEM | - __GFP_MOVABLE); + GFP_NOWAIT | __GFP_HIGHMEM | __GFP_MOVABLE); if (IS_ERR_VALUE(handle_new)) { zcomp_stream_put(zstrm); return PTR_ERR((void *)handle_new); From patchwork Wed Feb 12 06:27:08 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13971038 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B477DC0219E for ; Wed, 12 Feb 2025 06:32:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 43684280008; Wed, 12 Feb 2025 01:32:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3BE7C280001; Wed, 12 Feb 2025 01:32:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 238AF280008; Wed, 12 Feb 2025 01:32:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id F2E56280001 for ; Wed, 12 Feb 2025 01:32:55 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id B6D7FA0E96 for ; Wed, 12 Feb 2025 06:32:55 +0000 (UTC) X-FDA: 83110324710.07.F0C6B09 Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) by imf30.hostedemail.com (Postfix) with ESMTP id E3E8F8000B for ; Wed, 12 Feb 2025 06:32:53 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=DCnyq0mn; spf=pass (imf30.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.177 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739341974; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=J+hKSGhtq9SEMO5RmXUCmDriDLy9gufp9/zvmYkmXcc=; b=QRyV206neetQVXHDOA4/qfbWBQ+yMP+pW9OFTopfJqY6x04r+sayv/CYbx++0YskGA57Rs GPljejNQvsmX4luQOnAgZTVNIzAwOWdmb+KgHzV3/dvkDNYxFYh0dT6qk3i8z122f8LUBp yKIlz3ITuH29rSKvKMAcsE3ElhzIrGU= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=DCnyq0mn; spf=pass (imf30.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.177 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739341974; a=rsa-sha256; cv=none; b=nqXTWlwRGiv1gPzXQ80sobZeDQd0J6eF4wzRf8YGqLaT/OyLg7kv1Kik6Wc2qH5W7FEA77 e9iSVOJtvLbqO1f4JP5Toaj76TpGyRPE6cCgXEWTkEbXJWzBCMVp4opbVw1LCfSfTnVRSZ Obx7pDmwlOPyu0jiAFQbdmmIIFih9io= Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-21f6a47d617so68242805ad.2 for ; Tue, 11 Feb 2025 22:32:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739341973; x=1739946773; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=J+hKSGhtq9SEMO5RmXUCmDriDLy9gufp9/zvmYkmXcc=; b=DCnyq0mnAjijhrB8+8tpdOf1lq39A/V15sZ2xu1QxsqTdn9o77Q25lVorJKmWhLdSh oiMHrPG58zOynQC6qCf/d6MtcQgOid154R6X0tMgkJzOOUp38JbhlxkF56XuksUYo7Ne BeuWQ/JpWOCftIfhDmj091QM6KHooOGmYbATA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739341973; x=1739946773; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=J+hKSGhtq9SEMO5RmXUCmDriDLy9gufp9/zvmYkmXcc=; b=epKkKhMwspPP6EeFhEhdSR5qZbzNscsaJ/K9ccxAc3gncltowJIdOcWTfY0KLtUADJ 3dSXZHFpQSLRycIZmsGXW6lwDRSZfN/yB6x6SHGTOXI5Q+v+4S9tEpxVwlV1h64+X60X HRAu3HFIM11kfcHUJkJpnAHukH9fXXVAyFNnmYNqQsLCPO1MsJjLIb6ldxp+2xI7F+6W IXOvq94o+zIv6if9dEXejz6yhTF+VhKWsV77lLmipLU0a/ibkyPjNLDsKri2qj0wcRFs 4DSw+iwb0XA/edYKEEUgduK2fqCrWpixsUaW+GGX/S0K+2ARW9XqocsiiI8mygPcxXXH jyMw== X-Forwarded-Encrypted: i=1; AJvYcCV7KmjabVuxdfJsIgMp/XnBTIb8DPZ3uvXG4rlwvi3xKVJZmcz0sE7m1LzYLJ7oyyyb8zKCZSeo0g==@kvack.org X-Gm-Message-State: AOJu0Yx/CFie1OPpWpunfjjZoMSsZuqVbkrkqMBp67fPNEe5Y2DT/bts viRktihqOqSzP7TUxZRISVTHJb5eH/hUjFB37+jndhhm6CUdsVjZxLJn207F+g== X-Gm-Gg: ASbGnctFbpsUwvS6boqXnUtGVWmEHY6BKLcChtUMBQM82PNki197OZNGqLWnrcATtIr 4iqcLuL6b/RRHLIbtxcZeF67ToAHv51lkTdLRfdvHCkbtncJrdmioQTGU++DqtX+OmkEFjU1Rwl 5UU74t+KM7HOdDfR0KCrtopjoT5qli/ex7+GE7+66Ij2+oyMP9tBmi+ozeBWcoxQhM+ekx7cjl+ Eqi3wUouFL+S8eg/7wVmmXPfVOP354olLztSn6VpoiucWlu+dZmL0ggazCLWHalX/cXVInjvAJ+ 1R0nmfiF46U/woSTww== X-Google-Smtp-Source: AGHT+IGDl9mV+HtGUyn4myzfZBpkMQ7To3I2eCgVPnnWhAaFxnQ0KIUaHGWBSJNUrky9tjmUsZBSrQ== X-Received: by 2002:a05:6a20:3943:b0:1ee:650b:ac22 with SMTP id adf61e73a8af0-1ee650bb853mr710537637.40.1739341972819; Tue, 11 Feb 2025 22:32:52 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:69f5:6852:451e:8142]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-73077975f05sm7037787b3a.14.2025.02.11.22.32.50 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 11 Feb 2025 22:32:52 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v5 10/18] zsmalloc: factor out pool locking helpers Date: Wed, 12 Feb 2025 15:27:08 +0900 Message-ID: <20250212063153.179231-11-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog In-Reply-To: <20250212063153.179231-1-senozhatsky@chromium.org> References: <20250212063153.179231-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: E3E8F8000B X-Stat-Signature: 3f5m6j4fz1dswmudt8wxdbgu7he7efyy X-HE-Tag: 1739341973-544835 X-HE-Meta: U2FsdGVkX19jCl9K4HJGtKXyfILlIfptqJp/PCeNHj76KguM3IUX/mtfaL2CTT8LoCpTLcKcP8LkzM272aZD0fQkGL+Z8anICVTcdZoeK8jGdccLx257fmtLOVwjgv2gUCDuxh3B4InF+XfRWG9CiaLDOPlYYm0nUjhcAsgoVvwQreqzVoNbxL3a3dqmNTcsKosCcykZnVVcpKe2R9ViA0xQ2Zs8FLecz5oW7VBxYA8ig5yQ+7pSL92ifl4XncooEqRb3FWqMY1zXDHcqVfVzKoLRvnA+R7c4MKVhsWk/TEwGwcwj24+5LzkTt3Nj4E+TITpI5GXjCXG1C2rO6WMyd2j3/IDLxnkgfxa8erVqYr3BwNrvqv1dL5De/7w1Gz7x0G6aDJPaFc6DHsTMxF/OOsS+nds9PU9u+Ig6zEbBFUGstVnRlcPte6rnfgU3hEbj3Xuf0UvliNobHdlSg7RoNjuhYTXD89YEOY2eGN0DwZ8eSEBzmGh7kQaygMhJsK5zhhSLBYkzRXRMORucnPUWKuwcFlAyVx1KP9iAHlOamhJXfqUlwqBTjp/m63e6qzwais/paVasGfKd2W6n3OSISLxUpOwyt6V2ChileYS1rz5fbAHpfX6QCbAEVXgurNOV6oakKQX0+BR+IHVPmrciF50o9QEKsZkBTWbFI9vqRkmHMXEC+k6JPZi+60+w64PgcKsvqXySiIsJt5f2oyg5nMy9IaCdgRrvla8jOjIrpCmPExKCaNSISV2wvwij+V9+YsG26ewjtnx9pBgAHeyoCGr6/KDw6hd3dbzo+7OY9LHtpSH3k8y0tDFKOZEjW2+KJpB9tImyWuMigqz7vanH/sCqfxf/n3JN+k81+WBO/ZCJJ9hxR+xnSeKLDMIIKhyYbuxz7sYL1qCrY6QSTHJDG1LJqAUwB4V6Tskmc5DBYkRl4SCgN+WARbDSXmE4UfihVU299rKLhHuyAEaIzM 0/3kXvLX du7dh1ddvIi3ahcd54juchT6j83uvQykpBla/oB5+Vk9DYZOi4qUaquuuVYkVCUfuI8TLekgPgq2+ylJ3fl5ObxXWC5o1z6DWnCQ6Qo8xuO9bq59Qvr2AzI5h8d5kgM9FunK0H37PXvAO7mCAh2KQg9JuWTwaOP4BTU8wWBscrW4v+D63LqP0S3aj0Ff5KyFEgQQeQQIgcBRPRInyWycDByCODnA+zYP2D1jgccTzE7AqJdpHO4JnF5VGh3XR4GFV95JasLHtRayOp7firtckhP5pPPJ2GhHRk7kZnkcUuHdWi4cyMLchh/tTWw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: We currently have a mix of migrate_{read,write}_lock() helpers that lock zspages, but it's zs_pool that actually has a ->migrate_lock access to which is opene-coded. Factor out pool migrate locking into helpers, zspage migration locking API will be renamed to reduce confusion. It's worth mentioning that zsmalloc locks sync not only migration, but also compaction. Signed-off-by: Sergey Senozhatsky --- mm/zsmalloc.c | 63 +++++++++++++++++++++++++++++++++++---------------- 1 file changed, 44 insertions(+), 19 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 6d0e47f7ae33..47c638df47c5 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -18,7 +18,7 @@ /* * lock ordering: * page_lock - * pool->migrate_lock + * pool->lock * class->lock * zspage->lock */ @@ -224,10 +224,35 @@ struct zs_pool { struct work_struct free_work; #endif /* protect page/zspage migration */ - rwlock_t migrate_lock; + rwlock_t lock; atomic_t compaction_in_progress; }; +static void pool_write_unlock(struct zs_pool *pool) +{ + write_unlock(&pool->lock); +} + +static void pool_write_lock(struct zs_pool *pool) +{ + write_lock(&pool->lock); +} + +static void pool_read_unlock(struct zs_pool *pool) +{ + read_unlock(&pool->lock); +} + +static void pool_read_lock(struct zs_pool *pool) +{ + read_lock(&pool->lock); +} + +static bool pool_lock_is_contended(struct zs_pool *pool) +{ + return rwlock_is_contended(&pool->lock); +} + static inline void zpdesc_set_first(struct zpdesc *zpdesc) { SetPagePrivate(zpdesc_page(zpdesc)); @@ -1206,7 +1231,7 @@ void *zs_map_object(struct zs_pool *pool, unsigned long handle, BUG_ON(in_interrupt()); /* It guarantees it can get zspage from handle safely */ - read_lock(&pool->migrate_lock); + pool_read_lock(pool); obj = handle_to_obj(handle); obj_to_location(obj, &zpdesc, &obj_idx); zspage = get_zspage(zpdesc); @@ -1218,7 +1243,7 @@ void *zs_map_object(struct zs_pool *pool, unsigned long handle, * which is smaller granularity. */ migrate_read_lock(zspage); - read_unlock(&pool->migrate_lock); + pool_read_unlock(pool); class = zspage_class(pool, zspage); off = offset_in_page(class->size * obj_idx); @@ -1450,16 +1475,16 @@ void zs_free(struct zs_pool *pool, unsigned long handle) return; /* - * The pool->migrate_lock protects the race with zpage's migration + * The pool->lock protects the race with zpage's migration * so it's safe to get the page from handle. */ - read_lock(&pool->migrate_lock); + pool_read_lock(pool); obj = handle_to_obj(handle); obj_to_zpdesc(obj, &f_zpdesc); zspage = get_zspage(f_zpdesc); class = zspage_class(pool, zspage); spin_lock(&class->lock); - read_unlock(&pool->migrate_lock); + pool_read_unlock(pool); class_stat_sub(class, ZS_OBJS_INUSE, 1); obj_free(class->size, obj); @@ -1793,10 +1818,10 @@ static int zs_page_migrate(struct page *newpage, struct page *page, pool = zspage->pool; /* - * The pool migrate_lock protects the race between zpage migration + * The pool lock protects the race between zpage migration * and zs_free. */ - write_lock(&pool->migrate_lock); + pool_write_lock(pool); class = zspage_class(pool, zspage); /* @@ -1833,7 +1858,7 @@ static int zs_page_migrate(struct page *newpage, struct page *page, * Since we complete the data copy and set up new zspage structure, * it's okay to release migration_lock. */ - write_unlock(&pool->migrate_lock); + pool_write_unlock(pool); spin_unlock(&class->lock); migrate_write_unlock(zspage); @@ -1956,7 +1981,7 @@ static unsigned long __zs_compact(struct zs_pool *pool, * protect the race between zpage migration and zs_free * as well as zpage allocation/free */ - write_lock(&pool->migrate_lock); + pool_write_lock(pool); spin_lock(&class->lock); while (zs_can_compact(class)) { int fg; @@ -1983,14 +2008,14 @@ static unsigned long __zs_compact(struct zs_pool *pool, src_zspage = NULL; if (get_fullness_group(class, dst_zspage) == ZS_INUSE_RATIO_100 - || rwlock_is_contended(&pool->migrate_lock)) { + || pool_lock_is_contended(pool)) { putback_zspage(class, dst_zspage); dst_zspage = NULL; spin_unlock(&class->lock); - write_unlock(&pool->migrate_lock); + pool_write_unlock(pool); cond_resched(); - write_lock(&pool->migrate_lock); + pool_write_lock(pool); spin_lock(&class->lock); } } @@ -2002,7 +2027,7 @@ static unsigned long __zs_compact(struct zs_pool *pool, putback_zspage(class, dst_zspage); spin_unlock(&class->lock); - write_unlock(&pool->migrate_lock); + pool_write_unlock(pool); return pages_freed; } @@ -2014,10 +2039,10 @@ unsigned long zs_compact(struct zs_pool *pool) unsigned long pages_freed = 0; /* - * Pool compaction is performed under pool->migrate_lock so it is basically + * Pool compaction is performed under pool->lock so it is basically * single-threaded. Having more than one thread in __zs_compact() - * will increase pool->migrate_lock contention, which will impact other - * zsmalloc operations that need pool->migrate_lock. + * will increase pool->lock contention, which will impact other + * zsmalloc operations that need pool->lock. */ if (atomic_xchg(&pool->compaction_in_progress, 1)) return 0; @@ -2139,7 +2164,7 @@ struct zs_pool *zs_create_pool(const char *name) return NULL; init_deferred_free(pool); - rwlock_init(&pool->migrate_lock); + rwlock_init(&pool->lock); atomic_set(&pool->compaction_in_progress, 0); pool->name = kstrdup(name, GFP_KERNEL); From patchwork Wed Feb 12 06:27:09 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13971039 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 91E48C0219E for ; Wed, 12 Feb 2025 06:33:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1782328000A; Wed, 12 Feb 2025 01:33:01 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1274D280009; Wed, 12 Feb 2025 01:33:01 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EE2C028000A; Wed, 12 Feb 2025 01:33:00 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id C9FDF280009 for ; Wed, 12 Feb 2025 01:33:00 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 89886B14B7 for ; Wed, 12 Feb 2025 06:33:00 +0000 (UTC) X-FDA: 83110324920.03.6F05DDC Received: from mail-pj1-f50.google.com (mail-pj1-f50.google.com [209.85.216.50]) by imf29.hostedemail.com (Postfix) with ESMTP id B0AB9120002 for ; Wed, 12 Feb 2025 06:32:58 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=jK1qRgHX; spf=pass (imf29.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.216.50 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739341978; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=1Z728glTbJRUijcb8TCkc0Ox5AYTm//47KQTAd0HAqQ=; b=icrLtmzu/aeIpX93BXjQOW1a7S8CNx3CQ5jcnIARFMNehzdk6vH3nPVwiF8pBnAUyKEutI YT5iP+Ko2nP4E/jsJkceH9F4RFI5CVN8kE0CvP47H/THjyEiT002OpKEyB/5vFMlz2VDme G722zgQE1VksOa9XFWGpTZz332sKLug= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=jK1qRgHX; spf=pass (imf29.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.216.50 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739341978; a=rsa-sha256; cv=none; b=vqAFRkpI1SjnBG1Io6rIIoPSuKbd8oV91xrVCmuWv1v8dCqhQXKgsgU+5PfFHW/Y71pTdZ 0rwwp7ahDGvghUlW31QcBz/V0y1KkvQCGBJWohx6OgPfuiRrRTicFp6yaOGevmwuSEwF4Q i3d9E8K1jON8+fQGY886O7FvvpazTYw= Received: by mail-pj1-f50.google.com with SMTP id 98e67ed59e1d1-2fa2eb7eb45so779063a91.1 for ; Tue, 11 Feb 2025 22:32:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739341978; x=1739946778; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=1Z728glTbJRUijcb8TCkc0Ox5AYTm//47KQTAd0HAqQ=; b=jK1qRgHXDSkdsnMP+NxFmESyAntb3bZvpbcDzqM5xYe40Tzkh8tFPtjpasj8AvvMJh GPtiQ9lWKjhpPp1Fj1795XpnHaSgJcilUWOIS/Z0eEQEM4z3BuR0j+EVJtNtVPY8friq 5o5LUJVI0wVE87l4IiA+W2+uw0AfPqOCQOW9I= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739341978; x=1739946778; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1Z728glTbJRUijcb8TCkc0Ox5AYTm//47KQTAd0HAqQ=; b=IdrthtHg8Eob+bCdXbNjxHxSF4pu8PgifQWdccxxnk9seFrhEReqIurml2JdFFroNn fNhF/fEy9Ffhpucrt0UD2zOOqCYHnKj119oHoJQumjYJAvssBReBV9W6l7yhk5mM5OQW gQSSKiEHES5cODJ7hjFb3DjL9PjigPDZfBo3Qof3tCodIUlWs9Vqq/fpad8zOl81ZK2f DlbV2s9sFwdmOT3Zh6M2KFMcf1j4gUXEGTXC9L2mjE87lUiCunhQ5UCjdJ8cqCUVI1ox GzJLOLRYybxf5oGiAoMeKAphP2beCPES1trOwdjGcaLisW2WzIUme/LpEEvpuaLLpWIo SPjw== X-Forwarded-Encrypted: i=1; AJvYcCW9yyx8VpYENOQKxfNkeMeVKFo9hptzSosa4rG0T03ELgXddlO/o5pZr4ajBqBeHI8OtiEZYq4H+Q==@kvack.org X-Gm-Message-State: AOJu0Yx83CjstUrPHxnLkpS53rwc/l0988jyeWXwBE6R7u8YYbsDjts5 tFj0Zlq6fb8bOWgWQ3RyCiXxgMM92joJ6JGTjVY23YNrMVKMPh14xe128kK23g== X-Gm-Gg: ASbGncur+NzikPO2bNcZJGmntUDT7dDqlXKT8VAEXM9GCiVAJtO88lOHtXTSZl5rsVO kPX8Z1oLbNg2wVeoPOwC9oPyYWlESugqVC2bcS7/hkapXUPHv0X02wgr2VCRozqMi8NstC6nVCA F0urS3hTthxnRYlHZs085AQwPdUaXNERf6GqLLQxcHUwL2simt5LYLThZeAGIPNbflzgb+B3NM9 1Mo4yd1zmgRTYKi+yG2HJkDlDvNaKf64ICJaDUD3KdzlqPDWb5J9bHHD6FkrApqNwb5IPSvCGTS lvJsTkwh6+fOiP82Lw== X-Google-Smtp-Source: AGHT+IH9novtJAP2nahMJtf23e6d5DXnQj2BtTSpA3K6387kmaUUk7tqOB+00czVkoYIJ9iJRKWKIQ== X-Received: by 2002:a17:90b:2b45:b0:2fa:1c09:3cee with SMTP id 98e67ed59e1d1-2faa0913acemr9497572a91.9.1739341977607; Tue, 11 Feb 2025 22:32:57 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:69f5:6852:451e:8142]) by smtp.gmail.com with UTF8SMTPSA id 98e67ed59e1d1-2fbf98cffddsm658335a91.21.2025.02.11.22.32.55 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 11 Feb 2025 22:32:57 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v5 11/18] zsmalloc: factor out size-class locking helpers Date: Wed, 12 Feb 2025 15:27:09 +0900 Message-ID: <20250212063153.179231-12-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog In-Reply-To: <20250212063153.179231-1-senozhatsky@chromium.org> References: <20250212063153.179231-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: B0AB9120002 X-Stat-Signature: 31ydqnfpy1moo373srzrgcwt5nhy8peh X-Rspam-User: X-HE-Tag: 1739341978-988025 X-HE-Meta: U2FsdGVkX18ggElPVKeLecKoRxP4kAyJrcyDe90awrdznR9crdTQd6dYRjRl+IuC3TnKX9P0qIpmZ84tZsByxGWk8MzwpVMK1h97JZ/Q/JfkeMUT2QFy2Iad//0HITy9Vgudynd84RaxwgS1Kc3jwCbIAsCW++w6SBEzXXGuoOB7RlY8ZBi7gPNpwjTRIZfzGE+LqcYcgO+OIlc8r3T5xwYXpDiKLT5L1ARx7EPGTI3FwwSbHV9p2iPR1FoBwqswGrEz++R+XHqXKtpL3RW0cy6YEmQPCuz7zvMcp/tEZNHTVpsTUdrnSmM810piKRCX3ey02i1CS0YsXnQx7v+C3Cu9RdRlEz9ztDAbHCDvENMAYGyqvnAGOHXi8cQl3TzdFacZBPxgk14rXB1cRFPPL2zWi8vLCMpV2hyO7Ejnrl9YNRCfPclpSGksNqgrRJmUsxVE+cO7wQB1F+0ecuGoZCFPJYg8D3GcuNR5cZlATSSNaCDQyGswyyENsqUwqDB5PpYqJtifnVIWeENpjNppc/7GE0FsNFD8OaCbtnB9z3zZ97Yzl6JzvICFsxpiy6dVAllJ4Rl4Zb25kwIUgRt0j11rjeOSebNV4knm9qeO6B21S5JdJCf8Suy85Obf02Z/Q56K4059KT50atCwxSCHj6aA+I349s9FD096fqy/IX1nb88VD8tc3CPAou5NZw6tSbS6cYnuFtdOqGoeIu0yIMGnF5n8nyx0j/BUCbyRe566ywWTJdRX2sQYFqP9WGB0ZWF9s+S2kXfbivDCVOZWCxP/sT3F+QdTujSsuLLNa8UtqUiT5XvIEFHneLPkP/B5V/f0mnd56QoDAq5jQe1xFyLNR1j0ZJd4vZt3VWA3EAmaNR1aBQNYMhG01oklyyBDyCFqEFxH3UoUQnrFg9LfAKG6F3YFAquFejlU1XwLioq0XI3QersbtG7X2i/vWQnccBrgKC1RDZ924cDGIzv QnPEHr7s 9GKxWB/P1GceCOJTwjJUhmwEHkkwyYFJVQO84BGTPTU/oFrv5u2OzgQAcUgVPMnt87fN9WzB2SzLD1wYUZpNTLIMaXVknm0XQ4eXiQpeBSlB5KSFYXsX4Wx/Gwy32CCuG5yl6MebuNLi7m/KmGE5ZhNbnjG6W7goY+xcJ8tDhyqGN7nffagy6Z4zRNtr2OXKyXKwC7fGI8ymunf9inDR6Qw6XsqbyZq5a2ZN+Rx889ImbGQ3PLNIhPjOTUnbMFSLuBU04aEw2LQiMIp2yDiGQspv4xrskgAdobRADODP92c1sH5MoxLqUNOUbIw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Move open-coded size-class locking to dedicated helpers. Signed-off-by: Sergey Senozhatsky Reviewed-by: Yosry Ahmed --- mm/zsmalloc.c | 47 ++++++++++++++++++++++++++++------------------- 1 file changed, 28 insertions(+), 19 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 47c638df47c5..c82c24b8e6a4 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -253,6 +253,16 @@ static bool pool_lock_is_contended(struct zs_pool *pool) return rwlock_is_contended(&pool->lock); } +static void size_class_lock(struct size_class *class) +{ + spin_lock(&class->lock); +} + +static void size_class_unlock(struct size_class *class) +{ + spin_unlock(&class->lock); +} + static inline void zpdesc_set_first(struct zpdesc *zpdesc) { SetPagePrivate(zpdesc_page(zpdesc)); @@ -613,8 +623,7 @@ static int zs_stats_size_show(struct seq_file *s, void *v) if (class->index != i) continue; - spin_lock(&class->lock); - + size_class_lock(class); seq_printf(s, " %5u %5u ", i, class->size); for (fg = ZS_INUSE_RATIO_10; fg < NR_FULLNESS_GROUPS; fg++) { inuse_totals[fg] += class_stat_read(class, fg); @@ -624,7 +633,7 @@ static int zs_stats_size_show(struct seq_file *s, void *v) obj_allocated = class_stat_read(class, ZS_OBJS_ALLOCATED); obj_used = class_stat_read(class, ZS_OBJS_INUSE); freeable = zs_can_compact(class); - spin_unlock(&class->lock); + size_class_unlock(class); objs_per_zspage = class->objs_per_zspage; pages_used = obj_allocated / objs_per_zspage * @@ -1399,7 +1408,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) class = pool->size_class[get_size_class_index(size)]; /* class->lock effectively protects the zpage migration */ - spin_lock(&class->lock); + size_class_lock(class); zspage = find_get_zspage(class); if (likely(zspage)) { obj_malloc(pool, zspage, handle); @@ -1410,7 +1419,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) goto out; } - spin_unlock(&class->lock); + size_class_unlock(class); zspage = alloc_zspage(pool, class, gfp); if (!zspage) { @@ -1418,7 +1427,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) return (unsigned long)ERR_PTR(-ENOMEM); } - spin_lock(&class->lock); + size_class_lock(class); obj_malloc(pool, zspage, handle); newfg = get_fullness_group(class, zspage); insert_zspage(class, zspage, newfg); @@ -1429,7 +1438,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) /* We completely set up zspage so mark them as movable */ SetZsPageMovable(pool, zspage); out: - spin_unlock(&class->lock); + size_class_unlock(class); return handle; } @@ -1483,7 +1492,7 @@ void zs_free(struct zs_pool *pool, unsigned long handle) obj_to_zpdesc(obj, &f_zpdesc); zspage = get_zspage(f_zpdesc); class = zspage_class(pool, zspage); - spin_lock(&class->lock); + size_class_lock(class); pool_read_unlock(pool); class_stat_sub(class, ZS_OBJS_INUSE, 1); @@ -1493,7 +1502,7 @@ void zs_free(struct zs_pool *pool, unsigned long handle) if (fullness == ZS_INUSE_RATIO_0) free_zspage(pool, class, zspage); - spin_unlock(&class->lock); + size_class_unlock(class); cache_free_handle(pool, handle); } EXPORT_SYMBOL_GPL(zs_free); @@ -1827,7 +1836,7 @@ static int zs_page_migrate(struct page *newpage, struct page *page, /* * the class lock protects zpage alloc/free in the zspage. */ - spin_lock(&class->lock); + size_class_lock(class); /* the migrate_write_lock protects zpage access via zs_map_object */ migrate_write_lock(zspage); @@ -1859,7 +1868,7 @@ static int zs_page_migrate(struct page *newpage, struct page *page, * it's okay to release migration_lock. */ pool_write_unlock(pool); - spin_unlock(&class->lock); + size_class_unlock(class); migrate_write_unlock(zspage); zpdesc_get(newzpdesc); @@ -1903,10 +1912,10 @@ static void async_free_zspage(struct work_struct *work) if (class->index != i) continue; - spin_lock(&class->lock); + size_class_lock(class); list_splice_init(&class->fullness_list[ZS_INUSE_RATIO_0], &free_pages); - spin_unlock(&class->lock); + size_class_unlock(class); } list_for_each_entry_safe(zspage, tmp, &free_pages, list) { @@ -1914,10 +1923,10 @@ static void async_free_zspage(struct work_struct *work) lock_zspage(zspage); class = zspage_class(pool, zspage); - spin_lock(&class->lock); + size_class_lock(class); class_stat_sub(class, ZS_INUSE_RATIO_0, 1); __free_zspage(pool, class, zspage); - spin_unlock(&class->lock); + size_class_unlock(class); } }; @@ -1982,7 +1991,7 @@ static unsigned long __zs_compact(struct zs_pool *pool, * as well as zpage allocation/free */ pool_write_lock(pool); - spin_lock(&class->lock); + size_class_lock(class); while (zs_can_compact(class)) { int fg; @@ -2012,11 +2021,11 @@ static unsigned long __zs_compact(struct zs_pool *pool, putback_zspage(class, dst_zspage); dst_zspage = NULL; - spin_unlock(&class->lock); + size_class_unlock(class); pool_write_unlock(pool); cond_resched(); pool_write_lock(pool); - spin_lock(&class->lock); + size_class_lock(class); } } @@ -2026,7 +2035,7 @@ static unsigned long __zs_compact(struct zs_pool *pool, if (dst_zspage) putback_zspage(class, dst_zspage); - spin_unlock(&class->lock); + size_class_unlock(class); pool_write_unlock(pool); return pages_freed; From patchwork Wed Feb 12 06:27:10 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13971040 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9CB54C0219E for ; Wed, 12 Feb 2025 06:33:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 28B9628000B; Wed, 12 Feb 2025 01:33:06 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 237DB280009; Wed, 12 Feb 2025 01:33:06 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 08ABE28000B; Wed, 12 Feb 2025 01:33:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id DA640280009 for ; Wed, 12 Feb 2025 01:33:05 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 8FB77A0E96 for ; Wed, 12 Feb 2025 06:33:05 +0000 (UTC) X-FDA: 83110325130.22.0C702D1 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) by imf30.hostedemail.com (Postfix) with ESMTP id A66468000C for ; Wed, 12 Feb 2025 06:33:03 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=Op+QrTOD; spf=pass (imf30.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.171 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739341983; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BwlPMmfdw4MbZw7eUNqN2h6GCS8i0cwiXU5uZLysFGk=; b=XKaqHG6fjDKERdL6Me8aaApZ8KqZnFI0AlrVr6ncYc2nndDtWqklgXVJ0bNbVCbBU7R9zm SfitFp6Krk09m1AaOu6KjZ0SotlcDMTquf2LCtjoIK5kqLBXB69L+WA0H3f/Rm4piXReQ+ W5Oe5F+v6RTNZcVGIX+DWAXpEF98vYk= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=Op+QrTOD; spf=pass (imf30.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.171 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739341983; a=rsa-sha256; cv=none; b=VhWddzPl2K8bFEyAUMQSvw8SaL+gRxiBtFCsBvu7LzWfRjmJmkHK5f5K8jhgUt4Gj2Zdw1 NffxHizB1nn0R9llZW0dj6jY9YI0j82ORp0M3sahkKJMeYforbLRvqjK+MWkAKYls6yqhz E2OQ/4CbM6oxai2WhMCKExzKXqc5uQk= Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-21f6d2642faso99516415ad.1 for ; Tue, 11 Feb 2025 22:33:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739341982; x=1739946782; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BwlPMmfdw4MbZw7eUNqN2h6GCS8i0cwiXU5uZLysFGk=; b=Op+QrTODvwkUECr1xwJftm4g8kYMHtbCwapeWDEzhukJPTaQM6dH9wWeEIRhvuUGpS 6tLDWB2MNA9PjX5wvyl10BY269x81lJfa6tPKN9BC6CI6V/C764H+gxxfZczw8s1VFbP 1TpXz3J/m1/OqAmjzZa7AjJyVPsiV1urboK5k= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739341982; x=1739946782; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BwlPMmfdw4MbZw7eUNqN2h6GCS8i0cwiXU5uZLysFGk=; b=IVaVbGF8reUwm/ab428ZhXCUezhpgnWb1EQNoxJAAwbQnV+JeVWIACcSeM/+FoW/GH RC2ATQcMaXFS742fbKYuJYqb2SJww8ansBqTcd1dU/F0HyeDk8UGigV8ahVikt1lIt1x xctghteOzJRFmvZkePZtOQiP6Q2xGFhTLBfxyrVSvv98kvRLfFKjc2O1zFkICOLcApzp oXzx5ckfv/g5YeyiGLFTj3N54uzGA0TCeqDWH+TQtMFa2C8mgm7PtbCQnRS+fna5a/qK BqB0avSvj82AyefwuiEi0eomWl0xeX3bcFezTsSObf8+10v/Jpl0K/q6ZbuEu7i8ZZX7 j7cg== X-Forwarded-Encrypted: i=1; AJvYcCW4v14OepzLRuYpzpC8tV2qa5mFnM4yx3tOTnXfZH4AW6qzf8J6zHeBsJLmNt1Vy42r0VgxU6D/9w==@kvack.org X-Gm-Message-State: AOJu0YzbGyOnUrafnkVSLE0JvJM8V819Cg32+7vMog+qQq0YkQ5uHOrm GuPFSZ1HUpH3v7Pt1o6tFa1sOTTrYxO514r1xHp7fya1uE38meeTyxBV8LAFIvJzmNs9gCbbd1k = X-Gm-Gg: ASbGnctgn/R6oY7+zP9+UOEIlY4okx3eZr54V/lTA684qoWTvMadRdryNNvawpRijWx gGFJkblC2hsyocP1LY+DoKvkfLefpNYiPbbCaJXYVGb9IPsX9XkROMErGI44YW+wMB8MpKwhQwD y62rH86qT0Mzr/FeNj00IpXVROdcCVaDYX0WKZzCK7FIn0uTxxz/Ai98UeVU+GAvFVjMzue1iOb 0AkRjeV5RGXlzPGLoXeYoLIno2NLLfpjYLuYzxXchv7owU+slv7oV/o76CW7HkIwv+yAxY1Rcaf qsEb0vb82SqWUSp19Q== X-Google-Smtp-Source: AGHT+IGBqbR5rsHnduepMOHV/sDsvhxru/3OZYfP4l3JaDq83WR0FHW+hFPpJFYdg17e8CIob6LH9g== X-Received: by 2002:a05:6a00:1804:b0:732:24ad:8e15 with SMTP id d2e1a72fcca58-7322c39ac1amr3180602b3a.11.1739341982467; Tue, 11 Feb 2025 22:33:02 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:69f5:6852:451e:8142]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-7308e22e7e3sm4826244b3a.6.2025.02.11.22.33.00 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 11 Feb 2025 22:33:02 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v5 12/18] zsmalloc: make zspage lock preemptible Date: Wed, 12 Feb 2025 15:27:10 +0900 Message-ID: <20250212063153.179231-13-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog In-Reply-To: <20250212063153.179231-1-senozhatsky@chromium.org> References: <20250212063153.179231-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: A66468000C X-Stat-Signature: t4j6oj7cidndgiqtkj31jf1yy3ms46z4 X-HE-Tag: 1739341983-578893 X-HE-Meta: U2FsdGVkX1/9Luv3sPv57uXk89mIlgl/I0Ck0/M+hb5pH/lI6YOgsTVXbz4W3cNxDNHdWlIjjGstx/rxc61DCPkbB2ynG5Nq8M+AVm7A5/+JrfPhhcPfdl+r/H0Q4gWZ2fcXkp8YbKSB9JQmAQUEeGSIMYHI1umRsZlzQ+VESQZaxSqPCktssYMep+CF9V4QEmFnO1lzrgbOWkCrEpp2LXRJp2tF+A+B/t/igN//HFJDwBjoqzI6uuhDyLs1ZRrRzEtJx1F+yPFJ0zHlGqEPuwg8Jy6kxCzgHLa1NC6CsYuetxrNmpR7y2jnBavjUDHsfY4NPtdk1rpMqTRxFaCNImlNA+rOjdKNT70SvBua9fbG1W26Qp61rFAPsHEtESJu9L24Hz6OSggXJ66J80+BKLWhq85HQPwz+RBpm5JKNFs+XNNCKsIUbQ0ukPQ4snBZCvPde8Xm/R90dOrlGOOo6Eu+IZ8UCyxW2MV8OYybuR8a96SyGqIvEtmdpfYc9mwHbfDo2oW/YvFhQA6vtg7NUgvMm7YIf3G3+QFG05shbzhc2Sc6cqE0zUJAgh8AK4Z/jiKCajx2TALP4Q311UHIaQdPM0+eu+cGBePBLcs0olWHd0hVmSN7kByrq+4ZmJpnOU6/EjlCnjTzclxSNknjiBBv9+ALbriQgYVf/dQNtDoddZEw2nMUTDzl7SeBrU6Uu6Ov/Fs4IAgdt/aBJ3EchCC8WYDE+kSlGqLG9ck7+F/7DQ1jvw8KDwMuLa4ymwA8NzyBho2wQ48sV1I/GmLWLGtDHuPK6FdKHf/zxkk1BcJfA6I3Q4o/Kf9wO0ckfGQyWuHXHGvQWrTFPVDE38rRZjVQAMjhrnhFORA5bXCKQho2NRDyaOf53iL16lTlfbqOYD+1YwoSEKAukGnhYgr4rX67NyTwD3mYLdD83ETHVR0+2N0CTEmxsbZ2H+WgCjLSkTopii5E7tHBRrru1Jo xIZzlwx8 G/aCrC+Tk5u4MKd9Az293fABDfvEotF7GIpI4vyOAqjIYCWcTXf1LkxX0HFu8H0WgfAvBxDAgIr7+nMvEoflZ+QfMZtdKYl/piXrj14ya+yY9XqLOwRVhOuzENOkbZjo/rXi0vIFwBF21J8IZjBRf2K2GK8QaSG1icO+a2q2EO+Xz3dHd/6XbyKedH++A4V9gxJTjDQiyzMVNo2rQIWepLbfrzQKODMNl+sRxXOXQjq0OxOWHCMafMMZCz6df2tkDt99pJEvJI6gmqfffwfyjqp/amUYtw3rE+75B6AFXmheMvRJuY27NkphQpw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Switch over from rwlock_t to a atomic_t variable that takes negative value when the page is under migration, or positive values when the page is used by zsmalloc users (object map, etc.) Using a rwsem per-zspage is a little too memory heavy, a simple atomic_t should suffice. zspage lock is a leaf lock for zs_map_object(), where it's read-acquired. Since this lock now permits preemption extra care needs to be taken when it is write-acquired - all writers grab it in atomic context, so they cannot spin and wait for (potentially preempted) reader to unlock zspage. There are only two writers at this moment - migration and compaction. In both cases we use write-try-lock and bail out if zspage is read locked. Writers, on the other hand, never get preempted, so readers can spin waiting for the writer to unlock zspage. With this we can implement a preemptible object mapping. Signed-off-by: Sergey Senozhatsky Cc: Yosry Ahmed --- mm/zsmalloc.c | 183 +++++++++++++++++++++++++++++++++++--------------- 1 file changed, 128 insertions(+), 55 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index c82c24b8e6a4..80261bb78cf8 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -226,6 +226,9 @@ struct zs_pool { /* protect page/zspage migration */ rwlock_t lock; atomic_t compaction_in_progress; +#ifdef CONFIG_DEBUG_LOCK_ALLOC + struct lock_class_key lockdep_key; +#endif }; static void pool_write_unlock(struct zs_pool *pool) @@ -292,6 +295,9 @@ static inline void free_zpdesc(struct zpdesc *zpdesc) __free_page(page); } +#define ZS_PAGE_UNLOCKED 0 +#define ZS_PAGE_WRLOCKED -1 + struct zspage { struct { unsigned int huge:HUGE_BITS; @@ -304,7 +310,11 @@ struct zspage { struct zpdesc *first_zpdesc; struct list_head list; /* fullness list */ struct zs_pool *pool; - rwlock_t lock; + atomic_t lock; + +#ifdef CONFIG_DEBUG_LOCK_ALLOC + struct lockdep_map lockdep_map; +#endif }; struct mapping_area { @@ -314,6 +324,88 @@ struct mapping_area { enum zs_mapmode vm_mm; /* mapping mode */ }; +static void zspage_lock_init(struct zspage *zspage) +{ +#ifdef CONFIG_DEBUG_LOCK_ALLOC + lockdep_init_map(&zspage->lockdep_map, "zsmalloc-page", + &zspage->pool->lockdep_key, 0); +#endif + + atomic_set(&zspage->lock, ZS_PAGE_UNLOCKED); +} + +/* + * zspage locking rules: + * + * 1) writer-lock is exclusive + * + * 2) writer-lock owner cannot sleep + * + * 3) writer-lock owner cannot spin waiting for the lock + * - caller (e.g. compaction and migration) must check return value and + * handle locking failures + * - there is only TRY variant of writer-lock function + * + * 4) reader-lock owners (multiple) can sleep + * + * 5) reader-lock owners can spin waiting for the lock, in any context + * - existing readers (even preempted ones) don't block new readers + * - writer-lock owners never sleep, always unlock at some point + */ +static void zspage_read_lock(struct zspage *zspage) +{ + atomic_t *lock = &zspage->lock; + int old = atomic_read_acquire(lock); + +#ifdef CONFIG_DEBUG_LOCK_ALLOC + rwsem_acquire_read(&zspage->lockdep_map, 0, 0, _RET_IP_); +#endif + + do { + if (old == ZS_PAGE_WRLOCKED) { + cpu_relax(); + old = atomic_read_acquire(lock); + continue; + } + } while (!atomic_try_cmpxchg_acquire(lock, &old, old + 1)); +} + +static void zspage_read_unlock(struct zspage *zspage) +{ +#ifdef CONFIG_DEBUG_LOCK_ALLOC + rwsem_release(&zspage->lockdep_map, _RET_IP_); +#endif + atomic_dec_return_release(&zspage->lock); +} + +static __must_check bool zspage_try_write_lock(struct zspage *zspage) +{ + atomic_t *lock = &zspage->lock; + int old = ZS_PAGE_UNLOCKED; + + WARN_ON_ONCE(preemptible()); + + preempt_disable(); + if (atomic_try_cmpxchg_acquire(lock, &old, ZS_PAGE_WRLOCKED)) { +#ifdef CONFIG_DEBUG_LOCK_ALLOC + rwsem_acquire(&zspage->lockdep_map, 0, 1, _RET_IP_); +#endif + return true; + } + + preempt_enable(); + return false; +} + +static void zspage_write_unlock(struct zspage *zspage) +{ +#ifdef CONFIG_DEBUG_LOCK_ALLOC + rwsem_release(&zspage->lockdep_map, _RET_IP_); +#endif + atomic_set_release(&zspage->lock, ZS_PAGE_UNLOCKED); + preempt_enable(); +} + /* huge object: pages_per_zspage == 1 && maxobj_per_zspage == 1 */ static void SetZsHugePage(struct zspage *zspage) { @@ -325,12 +417,6 @@ static bool ZsHugePage(struct zspage *zspage) return zspage->huge; } -static void migrate_lock_init(struct zspage *zspage); -static void migrate_read_lock(struct zspage *zspage); -static void migrate_read_unlock(struct zspage *zspage); -static void migrate_write_lock(struct zspage *zspage); -static void migrate_write_unlock(struct zspage *zspage); - #ifdef CONFIG_COMPACTION static void kick_deferred_free(struct zs_pool *pool); static void init_deferred_free(struct zs_pool *pool); @@ -1026,7 +1112,9 @@ static struct zspage *alloc_zspage(struct zs_pool *pool, return NULL; zspage->magic = ZSPAGE_MAGIC; - migrate_lock_init(zspage); + zspage->pool = pool; + zspage->class = class->index; + zspage_lock_init(zspage); for (i = 0; i < class->pages_per_zspage; i++) { struct zpdesc *zpdesc; @@ -1049,8 +1137,6 @@ static struct zspage *alloc_zspage(struct zs_pool *pool, create_page_chain(class, zspage, zpdescs); init_zspage(class, zspage); - zspage->pool = pool; - zspage->class = class->index; return zspage; } @@ -1251,7 +1337,7 @@ void *zs_map_object(struct zs_pool *pool, unsigned long handle, * zs_unmap_object API so delegate the locking from class to zspage * which is smaller granularity. */ - migrate_read_lock(zspage); + zspage_read_lock(zspage); pool_read_unlock(pool); class = zspage_class(pool, zspage); @@ -1311,7 +1397,7 @@ void zs_unmap_object(struct zs_pool *pool, unsigned long handle) } local_unlock(&zs_map_area.lock); - migrate_read_unlock(zspage); + zspage_read_unlock(zspage); } EXPORT_SYMBOL_GPL(zs_unmap_object); @@ -1705,18 +1791,18 @@ static void lock_zspage(struct zspage *zspage) /* * Pages we haven't locked yet can be migrated off the list while we're * trying to lock them, so we need to be careful and only attempt to - * lock each page under migrate_read_lock(). Otherwise, the page we lock + * lock each page under zspage_read_lock(). Otherwise, the page we lock * may no longer belong to the zspage. This means that we may wait for * the wrong page to unlock, so we must take a reference to the page - * prior to waiting for it to unlock outside migrate_read_lock(). + * prior to waiting for it to unlock outside zspage_read_lock(). */ while (1) { - migrate_read_lock(zspage); + zspage_read_lock(zspage); zpdesc = get_first_zpdesc(zspage); if (zpdesc_trylock(zpdesc)) break; zpdesc_get(zpdesc); - migrate_read_unlock(zspage); + zspage_read_unlock(zspage); zpdesc_wait_locked(zpdesc); zpdesc_put(zpdesc); } @@ -1727,41 +1813,16 @@ static void lock_zspage(struct zspage *zspage) curr_zpdesc = zpdesc; } else { zpdesc_get(zpdesc); - migrate_read_unlock(zspage); + zspage_read_unlock(zspage); zpdesc_wait_locked(zpdesc); zpdesc_put(zpdesc); - migrate_read_lock(zspage); + zspage_read_lock(zspage); } } - migrate_read_unlock(zspage); + zspage_read_unlock(zspage); } #endif /* CONFIG_COMPACTION */ -static void migrate_lock_init(struct zspage *zspage) -{ - rwlock_init(&zspage->lock); -} - -static void migrate_read_lock(struct zspage *zspage) __acquires(&zspage->lock) -{ - read_lock(&zspage->lock); -} - -static void migrate_read_unlock(struct zspage *zspage) __releases(&zspage->lock) -{ - read_unlock(&zspage->lock); -} - -static void migrate_write_lock(struct zspage *zspage) -{ - write_lock(&zspage->lock); -} - -static void migrate_write_unlock(struct zspage *zspage) -{ - write_unlock(&zspage->lock); -} - #ifdef CONFIG_COMPACTION static const struct movable_operations zsmalloc_mops; @@ -1803,7 +1864,7 @@ static bool zs_page_isolate(struct page *page, isolate_mode_t mode) } static int zs_page_migrate(struct page *newpage, struct page *page, - enum migrate_mode mode) + enum migrate_mode mode) { struct zs_pool *pool; struct size_class *class; @@ -1819,15 +1880,12 @@ static int zs_page_migrate(struct page *newpage, struct page *page, VM_BUG_ON_PAGE(!zpdesc_is_isolated(zpdesc), zpdesc_page(zpdesc)); - /* We're committed, tell the world that this is a Zsmalloc page. */ - __zpdesc_set_zsmalloc(newzpdesc); - /* The page is locked, so this pointer must remain valid */ zspage = get_zspage(zpdesc); pool = zspage->pool; /* - * The pool lock protects the race between zpage migration + * The pool->lock protects the race between zpage migration * and zs_free. */ pool_write_lock(pool); @@ -1837,8 +1895,15 @@ static int zs_page_migrate(struct page *newpage, struct page *page, * the class lock protects zpage alloc/free in the zspage. */ size_class_lock(class); - /* the migrate_write_lock protects zpage access via zs_map_object */ - migrate_write_lock(zspage); + /* the zspage write_lock protects zpage access via zs_map_object */ + if (!zspage_try_write_lock(zspage)) { + size_class_unlock(class); + pool_write_unlock(pool); + return -EINVAL; + } + + /* We're committed, tell the world that this is a Zsmalloc page. */ + __zpdesc_set_zsmalloc(newzpdesc); offset = get_first_obj_offset(zpdesc); s_addr = kmap_local_zpdesc(zpdesc); @@ -1869,7 +1934,7 @@ static int zs_page_migrate(struct page *newpage, struct page *page, */ pool_write_unlock(pool); size_class_unlock(class); - migrate_write_unlock(zspage); + zspage_write_unlock(zspage); zpdesc_get(newzpdesc); if (zpdesc_zone(newzpdesc) != zpdesc_zone(zpdesc)) { @@ -2005,9 +2070,11 @@ static unsigned long __zs_compact(struct zs_pool *pool, if (!src_zspage) break; - migrate_write_lock(src_zspage); + if (!zspage_try_write_lock(src_zspage)) + break; + migrate_zspage(pool, src_zspage, dst_zspage); - migrate_write_unlock(src_zspage); + zspage_write_unlock(src_zspage); fg = putback_zspage(class, src_zspage); if (fg == ZS_INUSE_RATIO_0) { @@ -2267,7 +2334,9 @@ struct zs_pool *zs_create_pool(const char *name) * trigger compaction manually. Thus, ignore return code. */ zs_register_shrinker(pool); - +#ifdef CONFIG_DEBUG_LOCK_ALLOC + lockdep_register_key(&pool->lockdep_key); +#endif return pool; err: @@ -2304,6 +2373,10 @@ void zs_destroy_pool(struct zs_pool *pool) kfree(class); } +#ifdef CONFIG_DEBUG_LOCK_ALLOC + lockdep_unregister_key(&pool->lockdep_key); +#endif + destroy_cache(pool); kfree(pool->name); kfree(pool); From patchwork Wed Feb 12 06:27:11 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13971041 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2F44EC021A4 for ; Wed, 12 Feb 2025 06:33:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B3A9F28000C; Wed, 12 Feb 2025 01:33:10 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AE9B8280009; Wed, 12 Feb 2025 01:33:10 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9657928000C; Wed, 12 Feb 2025 01:33:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 717EC280009 for ; Wed, 12 Feb 2025 01:33:10 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 2714C80DC1 for ; Wed, 12 Feb 2025 06:33:10 +0000 (UTC) X-FDA: 83110325340.25.4141AB8 Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) by imf28.hostedemail.com (Postfix) with ESMTP id 449E5C0003 for ; Wed, 12 Feb 2025 06:33:08 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=RAmeb8f4; spf=pass (imf28.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.216.43 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739341988; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=dCEIYmpOSL5kMgSv8CIDbNXJeiLt2Oo14JvQ8JkkWQI=; b=ML0f9mEq/qaMM1Jdpqe429/25pcODXygHGWwBuIDTOG9hrlLBSg9HLNLfOozW1nrXlfJjq qn4vDSXi/mfG1rby6wMiSViHUGDmnIHL67H1DjTTndgNmxMslt9jZvtzI4qNBztvnFyoQH AcTtqUHVhOI4lKt1/lULDDo9RAOUUbw= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=RAmeb8f4; spf=pass (imf28.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.216.43 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739341988; a=rsa-sha256; cv=none; b=tCPzxjUFPUjXiDiVLEgDkL4D8/XlniNS27N2UAmc6JesSN/UUVnu0xpY3kJEWgjQlXNzCG 3eM+27wQE+dv+3Y8AIarih8ZnL/k1jCkeEp2O/t60Tqb6KEJzeaQ+DXPBSxc0MP/zJQJbq ynP7xRMByz9Si0vxhPYFhiNts0Asu78= Received: by mail-pj1-f43.google.com with SMTP id 98e67ed59e1d1-2ee9a780de4so9786069a91.3 for ; Tue, 11 Feb 2025 22:33:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739341987; x=1739946787; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=dCEIYmpOSL5kMgSv8CIDbNXJeiLt2Oo14JvQ8JkkWQI=; b=RAmeb8f45JprPDL7JMC0CgN35Me0c7nNBeg4dDlXZcsXQUNstdD94lnr1lo1Lgqurv NsB2Flb/p/vuT995Fdt1UpSPSCsaENRJU7BBfu2Pv1m2D5tVb5OX6CV/A8j3dd5OKaZ3 rMWm6jLal1s+16JU83+ZXR6xlk2ADdTAzM7NA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739341987; x=1739946787; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dCEIYmpOSL5kMgSv8CIDbNXJeiLt2Oo14JvQ8JkkWQI=; b=jvHBaNhm5+rhvgoJzo9LbII0ySGsKOU24ivjjZVoSaaqCMp3n+kEnIAtQ12knEJc8+ WC8eUR0UOGxs+MrMEirImQ07F9DSS15YnqBxPF0SlboXSMRxzWTpwNS1gw9IApviSCaJ +iPoD3L+UiKfIJM0uBu0Ti/2p3Twv86ZWXNe9kNFgmPgtvQoWltECiQgGN/EB06oUqkD aB08p72EqpA80TLWCbyPOl0NzeZEPA1rEfGeC1EnLqqG7lH9c31oQkyKNHUgm48cFsie WkdCMV6WUqo+X+U+1dmc1RarcJ2r6Cji0BL6XkaMGDjnbd1dsoxlko1PgD9dGDwJaTk4 96hA== X-Forwarded-Encrypted: i=1; AJvYcCWCmJvEZzD275Bf88QI/fNjkmJGe4mXy+8ecKP1thA44ELEiURY6efXfn+wnPSXLQC95LSY1pFuQg==@kvack.org X-Gm-Message-State: AOJu0YzUIDCeua2tMXiyL6xsJpMI/nmZoliWBxg8VZif8CrdPQrtsqJJ QTyM1zlJLshKKTpmvbIAPKos6U5oFKNUXEyxZx73xQbFCil9Tmlqdhcpif6/Ed9xAk2VqLopNw8 = X-Gm-Gg: ASbGnctwnQdtn+ZFUjDWFVD7TcLg/mddf6IMRL4zSULpTgsIcLTXL2PONKRfEXY5fOy dFDf497cqt84QNVaxHeGZOSIxa/7WZXuGh+bIlb2DMorb5d+Jj3ZHrjo7az9pLij3IUJnc7ttuo Osa4MIydKCIRsNAj56+jHnJX0L+ntTK/HmCN2oeb3CZLdeZjpK2sKJljFS5jk9gXW0eEkTFILc2 awB+hiffR8gex7eigvVjkbXmGDYzTCgdoufcLv48l+5pKusJBDfOwZoz0qfzNDX511jBYzPMKgC NJ55AAffmoX2HRKX2w== X-Google-Smtp-Source: AGHT+IEcBz98pD8dO8lfqL3ofXePIyCx5p8Uf93CWTvB4grPFDM00/5pX1hnLpil/qbdFRArW8IpbA== X-Received: by 2002:a17:90b:568d:b0:2ea:3f34:f18f with SMTP id 98e67ed59e1d1-2fbf5c0f4b5mr2776207a91.19.1739341987103; Tue, 11 Feb 2025 22:33:07 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:69f5:6852:451e:8142]) by smtp.gmail.com with UTF8SMTPSA id 98e67ed59e1d1-2fbf973c42esm672421a91.0.2025.02.11.22.33.04 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 11 Feb 2025 22:33:06 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v5 13/18] zsmalloc: introduce new object mapping API Date: Wed, 12 Feb 2025 15:27:11 +0900 Message-ID: <20250212063153.179231-14-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog In-Reply-To: <20250212063153.179231-1-senozhatsky@chromium.org> References: <20250212063153.179231-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 449E5C0003 X-Stat-Signature: kn14ij1mxa7f7ndqts6mef6mso9e3kga X-HE-Tag: 1739341988-290931 X-HE-Meta: U2FsdGVkX18mk5zlwSDuwKpJQ6W/fxydQHQE/jLXms91azatsw4z6jktBa7RyZnbzRYVcyWf0qvIo4SdZlgjlEMQPbkTXNeU0OYdm/zc7f6XCiaXDQlZ7CvJj62y+qvhBUlMVWL9Oh/gEkIzOiaRHC2mxRERZErkZYhCSUQjopPVj3+XZbDdFugZz34uAeEmFwhEB6tdCOviPOgjInBJi3WkPpFIHgLqmEWP4i+IkWSzC+tl8PBGiUnL11T4NuXsdv99QxaHoNp9rFDHeTSULSzMkR+ULRc0I9lLkId6A7K61+s0UpweOiY2eQ2mHfzd0IDWZaMYNEI+LJLi5u9A88SeerzyB9lS8qpjfLpfjqP1iIhMdFGo6doO1QS20albLIRzNNswpeNZGHjOzIPMOha4teQtLRQtigfPQC03VZ0nM8Wy5l+0aqr7GV6udZqH4OEp8/MiJfMXiflv23h/dwJQ2P4p4ASd8/y/0tB0fZ2kQXsyLo+57Vbuxu1JTqnWDkFTT+Ra+lis5ziz9gPNu88pBe+iwRpRs7snnkXD6FbTRBsJzi2T1cT49dWL5Z62bN6SZihhR4tQViuwDG1JD9Ewb8UpsNTCIunVAyBaDJec14Q1sHrQ02VNUnxeP7EofMossb39A1LXEXXtdYuTLZS6CjL1OCEe2pBL9+gqYdSHYOtLs8fqY/rWYTfB9TMp5ew+JgiR/KKyUKECiSDW/JL69mdRlciNdmy2Zx/i7cQ4BmtEW9jhAozhfmbX5LC6zyGz5Yc16PaHpUkcQqundx0IKhKa+pi4hvyGoiK1PuItJT30BEUEXhz2mkUcRUcNR/RPNcVZttEKioi/av2hcEmLvAgTLbgsNKeshAAgYV0GBng0axlYEDPEVvhxOVTP4++RPnKNxRHbqyYuofjNQLFdeUhS8rl4NCZuKuzzfdNgUS2CYk1Yi93cJrNcIxe0wC2qEdKaMTnsnEJIKG+ MH46W5Ia ZkvsjdQvpAm/oUIQ6+bsTfsQ5Vju/1bayXrsc4MRyr4oh4cpG/22i6AONqMs6TQK5bt5hf3cD5td4+fYky6QVPKnLEoiE1k6U71p7DHt6rpoirZyXN1IEirZ4Kmt5ANZiB53cZkufIyzXyGfeuZN0jgytfEeSAAzPKy6Q2wjynDQUAzk7cEzvS+9OBlyWQ+MD4npQiQHisjLJ0jnAtzzJgXbOyMf73LCuL/pvAv8boUAbCWqZhoVgMtYjXwh/x6yBAhoEpaRriaqhgLfMYYhjJdd0VXcUxn/w+dZLSO2qduZ5/3/zblOABezyvw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Current object mapping API is a little cumbersome. First, it's inconsistent, sometimes it returns with page-faults disabled and sometimes with page-faults enabled. Second, and most importantly, it enforces atomicity restrictions on its users. zs_map_object() has to return a liner object address which is not always possible because some objects span multiple physical (non-contiguous) pages. For such objects zsmalloc uses a per-CPU buffer to which object's data is copied before a pointer to that per-CPU buffer is returned back to the caller. This leads to another, final, issue - extra memcpy(). Since the caller gets a pointer to per-CPU buffer it can memcpy() data only to that buffer, and during zs_unmap_object() zsmalloc will memcpy() from that per-CPU buffer to physical pages that object in question spans across. New API splits functions by access mode: - zs_obj_read_begin(handle, local_copy) Returns a pointer to handle memory. For objects that span two physical pages a local_copy buffer is used to store object's data before the address is returned to the caller. Otherwise the object's page is kmap_local mapped directly. - zs_obj_read_end(handle, buf) Unmaps the page if it was kmap_local mapped by zs_obj_read_begin(). - zs_obj_write(handle, buf, len) Copies len-bytes from compression buffer to handle memory (takes care of objects that span two pages). This does not need any additional (e.g. per-CPU) buffers and writes the data directly to zsmalloc pool pages. In terms of performance, on a synthetic and completely reproducible test that allocates fixed number of objects of fixed sizes and iterates over those objects, first mapping in RO then in RW mode: OLD API ======= 10 runs 369,205,778 instructions # 0.80 insn per cycle 40,467,926 branches # 113.732 M/sec 369,002,122 instructions # 0.62 insn per cycle 40,426,145 branches # 189.361 M/sec 369,036,706 instructions # 0.63 insn per cycle 40,430,860 branches # 204.105 M/sec [..] NEW API ======= 10 runs 265,799,293 instructions # 0.51 insn per cycle 29,834,567 branches # 170.281 M/sec 265,765,970 instructions # 0.55 insn per cycle 29,829,019 branches # 161.602 M/sec 265,764,702 instructions # 0.51 insn per cycle 29,828,015 branches # 189.677 M/sec [..] Difference at 95.0% confidence -1.03219e+08 +/- 55308.7 -27.9705% +/- 0.0149878% (Student's t, pooled s = 58864.4) The old API will stay around until the remaining users switch to the new one. After that we'll also remove zsmalloc per-CPU buffer and CPU hotplug handling. Signed-off-by: Sergey Senozhatsky Reviewed-by: Yosry Ahmed --- include/linux/zsmalloc.h | 8 +++ mm/zsmalloc.c | 129 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 137 insertions(+) diff --git a/include/linux/zsmalloc.h b/include/linux/zsmalloc.h index a48cd0ffe57d..7d70983cf398 100644 --- a/include/linux/zsmalloc.h +++ b/include/linux/zsmalloc.h @@ -58,4 +58,12 @@ unsigned long zs_compact(struct zs_pool *pool); unsigned int zs_lookup_class_index(struct zs_pool *pool, unsigned int size); void zs_pool_stats(struct zs_pool *pool, struct zs_pool_stats *stats); + +void *zs_obj_read_begin(struct zs_pool *pool, unsigned long handle, + void *local_copy); +void zs_obj_read_end(struct zs_pool *pool, unsigned long handle, + void *handle_mem); +void zs_obj_write(struct zs_pool *pool, unsigned long handle, + void *handle_mem, size_t mem_len); + #endif diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 80261bb78cf8..e40268f3b655 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -1401,6 +1401,135 @@ void zs_unmap_object(struct zs_pool *pool, unsigned long handle) } EXPORT_SYMBOL_GPL(zs_unmap_object); +void *zs_obj_read_begin(struct zs_pool *pool, unsigned long handle, + void *local_copy) +{ + struct zspage *zspage; + struct zpdesc *zpdesc; + unsigned long obj, off; + unsigned int obj_idx; + struct size_class *class; + void *addr; + + WARN_ON(in_interrupt()); + + /* Guarantee we can get zspage from handle safely */ + pool_read_lock(pool); + obj = handle_to_obj(handle); + obj_to_location(obj, &zpdesc, &obj_idx); + zspage = get_zspage(zpdesc); + + /* Make sure migration doesn't move any pages in this zspage */ + zspage_read_lock(zspage); + pool_read_unlock(pool); + + class = zspage_class(pool, zspage); + off = offset_in_page(class->size * obj_idx); + + if (off + class->size <= PAGE_SIZE) { + /* this object is contained entirely within a page */ + addr = kmap_local_zpdesc(zpdesc); + addr += off; + } else { + size_t sizes[2]; + + /* this object spans two pages */ + sizes[0] = PAGE_SIZE - off; + sizes[1] = class->size - sizes[0]; + addr = local_copy; + + memcpy_from_page(addr, zpdesc_page(zpdesc), + off, sizes[0]); + zpdesc = get_next_zpdesc(zpdesc); + memcpy_from_page(addr + sizes[0], + zpdesc_page(zpdesc), + 0, sizes[1]); + } + + if (!ZsHugePage(zspage)) + addr += ZS_HANDLE_SIZE; + + return addr; +} +EXPORT_SYMBOL_GPL(zs_obj_read_begin); + +void zs_obj_read_end(struct zs_pool *pool, unsigned long handle, + void *handle_mem) +{ + struct zspage *zspage; + struct zpdesc *zpdesc; + unsigned long obj, off; + unsigned int obj_idx; + struct size_class *class; + + obj = handle_to_obj(handle); + obj_to_location(obj, &zpdesc, &obj_idx); + zspage = get_zspage(zpdesc); + class = zspage_class(pool, zspage); + off = offset_in_page(class->size * obj_idx); + + if (off + class->size <= PAGE_SIZE) { + if (!ZsHugePage(zspage)) + off += ZS_HANDLE_SIZE; + handle_mem -= off; + kunmap_local(handle_mem); + } + + zspage_read_unlock(zspage); +} +EXPORT_SYMBOL_GPL(zs_obj_read_end); + +void zs_obj_write(struct zs_pool *pool, unsigned long handle, + void *handle_mem, size_t mem_len) +{ + struct zspage *zspage; + struct zpdesc *zpdesc; + unsigned long obj, off; + unsigned int obj_idx; + struct size_class *class; + + WARN_ON(in_interrupt()); + + /* Guarantee we can get zspage from handle safely */ + pool_read_lock(pool); + obj = handle_to_obj(handle); + obj_to_location(obj, &zpdesc, &obj_idx); + zspage = get_zspage(zpdesc); + + /* Make sure migration doesn't move any pages in this zspage */ + zspage_read_lock(zspage); + pool_read_unlock(pool); + + class = zspage_class(pool, zspage); + off = offset_in_page(class->size * obj_idx); + + if (off + class->size <= PAGE_SIZE) { + /* this object is contained entirely within a page */ + void *dst = kmap_local_zpdesc(zpdesc); + + if (!ZsHugePage(zspage)) + off += ZS_HANDLE_SIZE; + memcpy(dst + off, handle_mem, mem_len); + kunmap_local(dst); + } else { + /* this object spans two pages */ + size_t sizes[2]; + + off += ZS_HANDLE_SIZE; + sizes[0] = PAGE_SIZE - off; + sizes[1] = mem_len - sizes[0]; + + memcpy_to_page(zpdesc_page(zpdesc), off, + handle_mem, sizes[0]); + zpdesc = get_next_zpdesc(zpdesc); + memcpy_to_page(zpdesc_page(zpdesc), 0, + handle_mem + sizes[0], sizes[1]); + } + + zspage_read_unlock(zspage); +} +EXPORT_SYMBOL_GPL(zs_obj_write); + /** * zs_huge_class_size() - Returns the size (in bytes) of the first huge * zsmalloc &size_class. From patchwork Wed Feb 12 06:27:12 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13971042 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BC1A9C02198 for ; Wed, 12 Feb 2025 06:33:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4F97A28000D; Wed, 12 Feb 2025 01:33:16 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 481F5280009; Wed, 12 Feb 2025 01:33:16 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2AF4428000D; Wed, 12 Feb 2025 01:33:16 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 0614A280009 for ; Wed, 12 Feb 2025 01:33:15 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 85D02C0E60 for ; Wed, 12 Feb 2025 06:33:15 +0000 (UTC) X-FDA: 83110325550.25.F7F8C99 Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) by imf05.hostedemail.com (Postfix) with ESMTP id AE25D100006 for ; Wed, 12 Feb 2025 06:33:13 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=BQ9pWKXf; spf=pass (imf05.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.174 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739341993; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=G+0Z+4xYmRhftKTrRiF3jNVLaBZmv0cRLATopAZYYvQ=; b=IT8erpC02LBBYbao4tPEcBxlCgvTOCdvDlnNYId+CIAs3OkpYWLR5almR6bzWry53VCT9r XVcbh2nEQFnZBOvQbMnbMK3xvCTv5guRNvqnW+mXw0JgcfBQg6otfSjCILFCNrpGp5F7Sd aKs8tY1LZoBtYE5zNupSQqsbmCeOA20= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=BQ9pWKXf; spf=pass (imf05.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.174 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739341993; a=rsa-sha256; cv=none; b=6+OiWq4b/c3K+OmW4ztRXLpWDO+gn8aWzPESMl/7MZHV2iz8LntIHJZ/8Sml9RZozzO/89 QljQjUqIC8NZXoSwLfsLXntycPlPap0cy7pSVCaXruS4uuhI4Fe7XroMRr4nFtmk7yD9A1 YBhZ/SEyX38/P9pYRf477D8qbIS1nfE= Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-21f6022c2c3so8183695ad.0 for ; Tue, 11 Feb 2025 22:33:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739341992; x=1739946792; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=G+0Z+4xYmRhftKTrRiF3jNVLaBZmv0cRLATopAZYYvQ=; b=BQ9pWKXfqh79c06BQPRyRZWi0TXVHxypXjOFqfXMUaW+HApPTtNl9hPC6Iud1tO91O 7V36A3u7C4igt3lW8WoCyLSkKDna2a3PGTiFEq0ZRv1lQ7+iORXzup+3dyJsWF4pgaz5 YrjzTlU6Nn1wC+IbV8Q7EjZZrqPD9cXdQvS0w= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739341992; x=1739946792; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=G+0Z+4xYmRhftKTrRiF3jNVLaBZmv0cRLATopAZYYvQ=; b=ahpm6yUuQyLDAU2YcFG0HIU8zehpdCcyOEzVSW1XTX4Sbsd96nowQ4ipEB6mgdg4TY 0gU92gj17ABKNrtgdc7+Ea9Y+pl7E7z3pxJbcT+Hjk9qyKqImqGjFZagfOjULVIh1bSJ cuhzJLhvLKMK0wiq2E3OUXuqe6MjjfJ5s+2KadKW6BIlbUmGGq9pueN5ujqEujCRW2yF H0s9bMb8WEqJWLL6ecFlEhkweRbWFf8A4WSxObjcHXNpveKcffT1ST7PO86o+y/rYwgy aozxdrHrx/oNnxgSaQ7netMTtL9ByVEvCDrV/zBsiVHSraVWaQT6PgC2FHtii02lXdjb 338g== X-Forwarded-Encrypted: i=1; AJvYcCVUGXT87IPRVFsooKNt6YiYvfZMl1at66JC3gQFGRow58YG5SwohvCARTAcXgIR2ZOkdTiLWTZiBQ==@kvack.org X-Gm-Message-State: AOJu0YxXdIqUwtHzpv6rqbKfzAc4cG7prL+/La1MpGv9AfXfypcTYHUR /WEWWC9PVq1BCLQx2ov765rcES604zNpO3S72tmcK6c4Qa963Gbdme5cXA81tqS/CshGaQW+Wng = X-Gm-Gg: ASbGnct6ihisPYPAm/0ssvADaVX8tMEjACnoJj+3h1bToVrGdhvVGir9cb6ACCrt1G/ 00/eChsWkh9zjnV/rekuTT5/1Oe2JFek8tAsJrEHB2J14DdQEa/DuKpnKlSh9WBEdQ2KQ+JDE6K uGsx0i9fhh14Ly+iL/Miht0vKOjGJkK0y5pl2+r+KaA6cbQ5bKBfzLJh05PoTpsuVsxv+RHBGzF OCTZi47mQ5heHN2vTCVBA+wfN+oqa9pHsDygbJbB/xduQXL4UbgEdn5jhvU0vw9NnzmPHBjIo7n mQMH9H4EaeNxXMEkZQ== X-Google-Smtp-Source: AGHT+IEhnNCxLoE2CNoFgq8/HUot/UmlliKB3Yg2LwQwc2F/d9hoWw6hVmqWymhRbVtPXN7CHslIlw== X-Received: by 2002:a05:6a21:9101:b0:1e0:d3e9:1f8 with SMTP id adf61e73a8af0-1ee5cedffe2mr2971435637.10.1739341992642; Tue, 11 Feb 2025 22:33:12 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:69f5:6852:451e:8142]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-73093ab315csm4424643b3a.143.2025.02.11.22.33.10 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 11 Feb 2025 22:33:12 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v5 14/18] zram: switch to new zsmalloc object mapping API Date: Wed, 12 Feb 2025 15:27:12 +0900 Message-ID: <20250212063153.179231-15-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog In-Reply-To: <20250212063153.179231-1-senozhatsky@chromium.org> References: <20250212063153.179231-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: AE25D100006 X-Stat-Signature: ibe61qeoew8xe8ttucdnpq96z9ktm8om X-HE-Tag: 1739341993-741337 X-HE-Meta: U2FsdGVkX1+yk4Lge7XGJ5iC2qGo8x+AoJ1EMNOfjX4hQ0ChVsIyYZpoudFKEi7hKitcN6PYUrSYpCGEUeCUyD0PhjpqZXUxFSvTPH0IqKUR319ehL4CgJWERrrcKiv1Vu1LadbkCPBswaKOx1sodLQemUqCB6VsguMrrrYUDazH6rxYcnfskM61XHus4vYihPqJU4/2KMQBZwCDEijF9FrzNBA57s/linXAVwseKA7eJjQGT2mRyav03LMH5fcNTuE7J7AdszqRApFKkjOYPyk4NHig8g/Zp6avThs/OMweTD1q9LWeeG6Y566cYgSLDRcPH+xcezEdF+k8/APAX1P3SdTv+0B5vpbrPtuavNk/Rx2aMNbBfzl8+yS0AbmUgc7gkMc+tkLY8yTYBf+WJf6TvrsICLPraHVrsDERLSF9ZmZp3BES84pWsGrX4BHVdZaTIbCij4SpodPOVBXVP9JdSClF3J8xSNQzG4MUc+b3fw8UcpPu4bEVg40hFtZOccEhybe8/cN+SXdjUhXNgmy7UZXRpQbyLkYNotyRmbygugyT4OASdY9T5QLrR8MhNQd2+d+WvOOUwvgN2EK6ol3bi1cCs3La5TutYZkt+vtXHrFN6CFqPdDyYORrvrUZkK5ym2dx1wh8T5mpGhOOgEqXmjixi8WvQHMM2WifY4YLbJTYggc6fN3c4xOja6WT0bV/v8Gwwkiy1kHIYXcPHZuL41fhjGKgujRnimXuuIK/Un+HXPZGlklYAXPNuY+2HcGYQtFaHe/JUys7e6SblV0y0uARy4uZfMtvQ07B8X/R1fmgNUjZi3hqx6sVWvhmvxVXs8d14wj0FEjNN/kyHhoRqQaP4hxZh3VRwbZaDELxtorQ5PScu3wWMjcr2d7JLwmcPf64EYZC/KBJsfSx2Tdtc7ZWkEHgcHydaJJPCMehy7Z6ZNCjWbZXQonHUa3m5Vfzt4pN2hyAF0kgOoY kORcJn5M ESGXCT49PLL3mC4u+dlbOQyi4jATCgday+oWSrmoGJWGZB+FkqQ4UqCH68o35FwP8S0khCH4fkDLdWIfkhKp7kNr7f+HuVPQSIY8FlYJ6V+tDdoIx56IcuDEmLZTH/XZ934ZADxsPFQO9jc0uvhwyv/vsWVR0GPpbcEseIyzmwD7obT5JPZP5uHqJsgDVqL4NDxcmY8vPk6g5714N+EGewFIqkJix2phs2XfN3HwbVhmCqEOL3JV0RbjbZYeW7e4sdi5fnfOVWC8g7TG+d5aSk/1CpWteyvAn9qcgiGzg598uQgwCwgFJ54k/TQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Use new read/write zsmalloc object API. For cases when RO mapped object spans two physical pages (requires temp buffer) compression streams now carry around one extra physical page. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zcomp.c | 4 +++- drivers/block/zram/zcomp.h | 2 ++ drivers/block/zram/zram_drv.c | 28 ++++++++++------------------ 3 files changed, 15 insertions(+), 19 deletions(-) diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c index c393243eeb5c..61a9c3ed6f7a 100644 --- a/drivers/block/zram/zcomp.c +++ b/drivers/block/zram/zcomp.c @@ -46,6 +46,7 @@ static const struct zcomp_ops *backends[] = { static void zcomp_strm_free(struct zcomp *comp, struct zcomp_strm *zstrm) { comp->ops->destroy_ctx(&zstrm->ctx); + vfree(zstrm->local_copy); vfree(zstrm->buffer); zstrm->buffer = NULL; } @@ -59,12 +60,13 @@ static int zcomp_strm_init(struct zcomp *comp, struct zcomp_strm *zstrm) if (ret) return ret; + zstrm->local_copy = vzalloc(PAGE_SIZE); /* * allocate 2 pages. 1 for compressed data, plus 1 extra for the * case when compressed size is larger than the original one */ zstrm->buffer = vzalloc(2 * PAGE_SIZE); - if (!zstrm->buffer) { + if (!zstrm->buffer || !zstrm->local_copy) { zcomp_strm_free(comp, zstrm); return -ENOMEM; } diff --git a/drivers/block/zram/zcomp.h b/drivers/block/zram/zcomp.h index 23b8236b9090..25339ed1e07e 100644 --- a/drivers/block/zram/zcomp.h +++ b/drivers/block/zram/zcomp.h @@ -34,6 +34,8 @@ struct zcomp_strm { struct mutex lock; /* compression buffer */ void *buffer; + /* local copy of handle memory */ + void *local_copy; struct zcomp_ctx ctx; }; diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 7c4c296181a8..c6310077c221 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1561,11 +1561,11 @@ static int read_incompressible_page(struct zram *zram, struct page *page, void *src, *dst; handle = zram_get_handle(zram, index); - src = zs_map_object(zram->mem_pool, handle, ZS_MM_RO); + src = zs_obj_read_begin(zram->mem_pool, handle, NULL); dst = kmap_local_page(page); copy_page(dst, src); kunmap_local(dst); - zs_unmap_object(zram->mem_pool, handle); + zs_obj_read_end(zram->mem_pool, handle, src); return 0; } @@ -1583,11 +1583,11 @@ static int read_compressed_page(struct zram *zram, struct page *page, u32 index) prio = zram_get_priority(zram, index); zstrm = zcomp_stream_get(zram->comps[prio]); - src = zs_map_object(zram->mem_pool, handle, ZS_MM_RO); + src = zs_obj_read_begin(zram->mem_pool, handle, zstrm->local_copy); dst = kmap_local_page(page); ret = zcomp_decompress(zram->comps[prio], zstrm, src, size, dst); kunmap_local(dst); - zs_unmap_object(zram->mem_pool, handle); + zs_obj_read_end(zram->mem_pool, handle, src); zcomp_stream_put(zstrm); return ret; @@ -1683,7 +1683,7 @@ static int write_incompressible_page(struct zram *zram, struct page *page, u32 index) { unsigned long handle; - void *src, *dst; + void *src; /* * This function is called from preemptible context so we don't need @@ -1700,11 +1700,9 @@ static int write_incompressible_page(struct zram *zram, struct page *page, return -ENOMEM; } - dst = zs_map_object(zram->mem_pool, handle, ZS_MM_WO); src = kmap_local_page(page); - memcpy(dst, src, PAGE_SIZE); + zs_obj_write(zram->mem_pool, handle, src, PAGE_SIZE); kunmap_local(src); - zs_unmap_object(zram->mem_pool, handle); zram_slot_lock(zram, index); zram_set_flag(zram, index, ZRAM_HUGE); @@ -1725,7 +1723,7 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) int ret = 0; unsigned long handle; unsigned int comp_len; - void *dst, *mem; + void *mem; struct zcomp_strm *zstrm; unsigned long element; bool same_filled; @@ -1771,11 +1769,8 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) return -ENOMEM; } - dst = zs_map_object(zram->mem_pool, handle, ZS_MM_WO); - - memcpy(dst, zstrm->buffer, comp_len); + zs_obj_write(zram->mem_pool, handle, zstrm->buffer, comp_len); zcomp_stream_put(zstrm); - zs_unmap_object(zram->mem_pool, handle); zram_slot_lock(zram, index); zram_set_handle(zram, index, handle); @@ -1887,7 +1882,7 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, unsigned int comp_len_new; unsigned int class_index_old; unsigned int class_index_new; - void *src, *dst; + void *src; int ret = 0; handle_old = zram_get_handle(zram, index); @@ -1997,12 +1992,9 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, return PTR_ERR((void *)handle_new); } - dst = zs_map_object(zram->mem_pool, handle_new, ZS_MM_WO); - memcpy(dst, zstrm->buffer, comp_len_new); + zs_obj_write(zram->mem_pool, handle_new, zstrm->buffer, comp_len_new); zcomp_stream_put(zstrm); - zs_unmap_object(zram->mem_pool, handle_new); - zram_free_page(zram, index); zram_set_handle(zram, index, handle_new); zram_set_obj_size(zram, index, comp_len_new); From patchwork Wed Feb 12 06:27:13 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13971043 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 80DF1C02198 for ; Wed, 12 Feb 2025 06:33:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0DC1D28000F; Wed, 12 Feb 2025 01:33:21 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 08A7F28000E; Wed, 12 Feb 2025 01:33:20 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E47FF28000F; Wed, 12 Feb 2025 01:33:20 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id C495528000E for ; Wed, 12 Feb 2025 01:33:20 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 83D034B4F9 for ; Wed, 12 Feb 2025 06:33:20 +0000 (UTC) X-FDA: 83110325760.25.2BF3EF2 Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) by imf09.hostedemail.com (Postfix) with ESMTP id A5A19140009 for ; Wed, 12 Feb 2025 06:33:18 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=OBJ4s0xD; spf=pass (imf09.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.170 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739341998; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zGWR8uwxFV58sLam3cHzynn2zEt595XLKCTA6OHBb4A=; b=lpYl1iCVEOS1Q8jx+D0RYE7SN9hKWrCMHkTRI4WSeOQwaPrtSi0C/OGawNHlR1FfGntgcy enjcCoY1OMsdQhFfhq+Dc5VqKgQpSk3F7K2xE6uuq9KIQ7/OA1xuiluLldI8qaYYQe4zAf ZsdFUkYR9esxJv/TZh480SxTTjSucBo= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=OBJ4s0xD; spf=pass (imf09.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.170 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739341998; a=rsa-sha256; cv=none; b=K+blsKhrVyKM292nkoqJJ5hqB8XQgMkhCg7vD/I/i8h2IMWEQIoM6GgskUAAuKsEzQn2sW 3/VSZB6LnoTGSVw9uhQvx+dKH7WfCCXLzDtmXbHa6d0jpxAwtcqQegas7OdlDFHs1FFFHb B0l29XSfeWMBxuxBQIwrtm4hucfqHCM= Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-21f6f18b474so55257925ad.1 for ; Tue, 11 Feb 2025 22:33:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739341997; x=1739946797; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=zGWR8uwxFV58sLam3cHzynn2zEt595XLKCTA6OHBb4A=; b=OBJ4s0xDlIIKcIprw27SjJ4DPmoMNFoDMP1NnhCdzDhaE3q50xtPSzbDaFMmFz2Yt7 bK3HkZy2GbrD5xTnutFYmh0sq5qcGx5hP/JABEeKZOD3DhWobE+u3TYfL/7hx/B0gWGC mS5QYwU3ZYOw5m/tnI1MS6I0h9cRDlWNxR3+4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739341997; x=1739946797; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=zGWR8uwxFV58sLam3cHzynn2zEt595XLKCTA6OHBb4A=; b=jphkSyXPlegwP++xNP/vwd16rNEOxzmeseBqzki6xg2L/KMKH1mUCSVvjzRkLbI1Rs Hy/OEJADqJ5NErcRZ23/LuNbQOxqUOLaxyBIHeby94gYAun/acwxFCiL6COhi07dfG3k NYZpmehZ0q9o4qBqN4i8aP9GCmYvzGhMNVG7qYpyFMxnVBWGKZhxzy+5iFciBS+4YU1M axzRlx7U9kvqW0V52mMQXK2vgbQoOx5NCJv/x6Rsw+lAAJJo7SS2UF7Sfjx0qjFNzlQw 1KDN6bd0jDyYmiuRMPEcGL6P50Nby7KAMQWnxq0ZdhG9jcDsgUxfC4G6ezgBh52iEaLj 2aQA== X-Forwarded-Encrypted: i=1; AJvYcCWCTAFn6zIkp6jhrmNgqSHppguavgY4X2TvUmOYgUxBolMUFoWCe4wbJQ0JkcPkkPyehcEgTRJx+A==@kvack.org X-Gm-Message-State: AOJu0YwsRjxnHWufC6Y19DugBixrW31uEn737i7ZHeXHowpe/qmscsN5 6DstGqdeQbi1/QMlVJRN/Ni/KmjKBn8feMlc/Dhop2KIrO3NAW3m5KU28sABu2Dyfmyrt+gsXK4 = X-Gm-Gg: ASbGnctmPITGKhPazcebwYBCpoW9dNjB3VFpm60kgESBEVrB1y0jRBd+J8HqHesuOqI C8QcLYwuTobSKEbEwZG5QE6m4V9ZiLjpw+ddB1HJ5YTnfL9+KbKi8OUPBiNwtv8JxqjYJeYb2Np Q78pXUlNqx839nVISZpAwhscbX4xi2dobl3R2ujMYSTL0LLBbFZkCC1l4tGC/cu70PDswPuRO92 sCA7ZTkh59rX1vsCAecgkJKas26nNavNFQ6utwbaMQCJ60Twur7PRFqB4od1Ybz2k1jbye99voj SaljgF1Ccro/aPp02w== X-Google-Smtp-Source: AGHT+IFzbH9qNa8XB8LsSRHPJwmV7Dgu2lkCSlxYEmJLzkmFDmA/w+WWTWaak6t0Xswa1OMVGZCLYg== X-Received: by 2002:a05:6a20:728d:b0:1ee:6187:7fb3 with SMTP id adf61e73a8af0-1ee61878104mr1837250637.15.1739341997477; Tue, 11 Feb 2025 22:33:17 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:69f5:6852:451e:8142]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-73218c206a6sm2581185b3a.49.2025.02.11.22.33.15 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 11 Feb 2025 22:33:17 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v5 15/18] zram: permit reclaim in zstd custom allocator Date: Wed, 12 Feb 2025 15:27:13 +0900 Message-ID: <20250212063153.179231-16-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog In-Reply-To: <20250212063153.179231-1-senozhatsky@chromium.org> References: <20250212063153.179231-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: A5A19140009 X-Stat-Signature: 65rco4bos1fkic8sigi6d47hesp9ktzo X-HE-Tag: 1739341998-479584 X-HE-Meta: U2FsdGVkX1/NtRW+fGVU+LkIf2ZhkinoF/YGfdLaIqP5Fqno32JjWIAETLH7UKATf6s4HP25Sacz7NWw8dcnP1pHNKgY+yntqXAd0vliS0W8zul4/GOsBhNM+g1OifTBVjosPwgk+MSq55C1C7a7S+x9u+uxj0+bPaeMuNJDUPTjdwVLKH9xKP6OgrKMJW/NlXUW7RI5wkD7u/pD/EJw0dmiKtetO2CYFaH0RH9uG3yjzFkWp0f3gCLj1ZXhTlwN4I/LcGD0MClDckLNe1toQzHogQniYKvPfKN8/uiWA5gaLu1jWBK5ciRPmTNkhLw1DvFS+bqc6nd69T1kWZ6ETbUb07qcsG3J/vGd4OAtulGo78vEBnv7o9xp85R2OzteHgNKTiKuT2imG9PtTD7AEkfrE25gzHR5FqQsXhuS29y+ZI767tP3/iaOyIQN/EHXUS5cOlIo/yfr5yUim828eJGmDVzCURG/+L7MOokt8ld4NEYQsMtevR4WYIMV9yIzwjWSO8iiz5aEFQwKeLsqKFjNW5LMyBVTO2/x8okMa3HK5Zjwdx2HC04pSs1ur9Ye5dv563JFYxmiJL7aMwjsFoCbXCAkMNwdNjfqki09BNT403lKZaU1iMtqkYQsxo+AUmG/v5mNsO8jUAPC7vUrdTf9MvFrv/dLoFRHt/dtc8kzuq3PWyhjXw90z2vMDwmUsNvcycmQD7KQUxWEXJIcyNVBNtuwHA3hcSAH6WAC4xXzwzYSxik0aH6fh8mNxVKgb6ST8z9KRL0OI4fINI7MreVyX0WVWmHQ5o23Z1KUmFNUza9H2LgFo/6smX1i+h0OxmKY/OeiRRBQTlxyIheHmZyjBqwqO1+y7sMJ0EwWk7ssTJoV4OWbnbDX1cbvTL1XWj1pt8vJVAmvR71G7AMKDtTxr3tt6OaT9MZ+o7wXv8NYJGbKANkDKw01rQx1cTA/Twwe5LBMHkq5/DgBz8g ber+UUw4 Dg9Tew75YD53uLxzGQ+H/yUSpT56+K4H6fMlrynslf4V+Hl8c6Autdx70uVXxZ6C3dg1IVWHY9Y+fJURS8rehWUz8InMIkEbWSZUSPiBf0lWHN6GultmRUJDqpSpj/fkcNTel/jd7hKgVctXOapN+gOh0RISc7ei15JB7RNxzxlMTBCx088qnF4M4JPLpabDUuadK5XiLqttDCDu9bGzyQDJ+o4V7Oe6exbSLEOGp8+yi8HLP9PqQ90olhhOHjGfPi8bg6YDWtDInpzfo55Xm8V9nGpN+rjMrFdkoUu7ClbMSe67yLq80CrOJ0A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When configured with pre-trained compression/decompression dictionary support, zstd requires custom memory allocator, which it calls internally from compression()/decompression() routines. That means allocation from atomic context (either under entry spin-lock, or per-CPU local-lock or both). Now, with non-atomic zram read()/write(), those limitations are relaxed and we can allow direct and indirect reclaim. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/backend_zstd.c | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-) diff --git a/drivers/block/zram/backend_zstd.c b/drivers/block/zram/backend_zstd.c index 1184c0036f44..53431251ea62 100644 --- a/drivers/block/zram/backend_zstd.c +++ b/drivers/block/zram/backend_zstd.c @@ -24,19 +24,14 @@ struct zstd_params { /* * For C/D dictionaries we need to provide zstd with zstd_custom_mem, * which zstd uses internally to allocate/free memory when needed. - * - * This means that allocator.customAlloc() can be called from zcomp_compress() - * under local-lock (per-CPU compression stream), in which case we must use - * GFP_ATOMIC. - * - * Another complication here is that we can be configured as a swap device. */ static void *zstd_custom_alloc(void *opaque, size_t size) { - if (!preemptible()) + /* Technically this should not happen */ + if (WARN_ON_ONCE(!preemptible())) return kvzalloc(size, GFP_ATOMIC); - return kvzalloc(size, __GFP_KSWAPD_RECLAIM | __GFP_NOWARN); + return kvzalloc(size, GFP_NOIO | __GFP_NOWARN); } static void zstd_custom_free(void *opaque, void *address) From patchwork Wed Feb 12 06:27:14 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13971044 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EEEF2C0219E for ; Wed, 12 Feb 2025 06:33:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7E02F280010; Wed, 12 Feb 2025 01:33:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7901F28000E; Wed, 12 Feb 2025 01:33:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 62FEE280010; Wed, 12 Feb 2025 01:33:25 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 3EADD28000E for ; Wed, 12 Feb 2025 01:33:25 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id E75EDC0E56 for ; Wed, 12 Feb 2025 06:33:24 +0000 (UTC) X-FDA: 83110325928.21.C81E6A6 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) by imf15.hostedemail.com (Postfix) with ESMTP id 1D7D3A0004 for ; Wed, 12 Feb 2025 06:33:22 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=CbaBgCeH; spf=pass (imf15.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.169 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739342003; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=fDJRc+CzjHHBPozwYIKOoKTE7RtcGO1bK7SyyTKpM/I=; b=FjlkYQWgASybuY27Osrrt0MWufFDCh81WWFpajVsu/MSnvfVSn9ORX6Y2s+q7oubAsT5nC TBeH8vhM9ZPU4tgqKrOz/hA+Zl0+WwfvfmAr+CD1Jv3IzHd+gQC9QjaLKpYSXod9sj6gQq FQoXuv+X6kjratQWY0e0n7whcTDwAZw= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=CbaBgCeH; spf=pass (imf15.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.169 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739342003; a=rsa-sha256; cv=none; b=o3ZWkS87FSIRZdQSb9M2t+aeXNPTozBWjThw5fnpVc5EPilc9PokQsTlzLKUU7AMyd7DCN qCpBZzYmCqfTJct4pv1yZJ3/2ojAn2hX+kPwFvTdlj6xGjKQ+NBXca5YBWUCRfe3mJx52a MOjJVg+yFIP3tUPTZP7CCCqcVHEryuc= Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-220bfdfb3f4so6552615ad.2 for ; Tue, 11 Feb 2025 22:33:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739342002; x=1739946802; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=fDJRc+CzjHHBPozwYIKOoKTE7RtcGO1bK7SyyTKpM/I=; b=CbaBgCeHBxm61SjhoPCnz7v54FxQcu2lkhdjA2nB79kb6kmnXey6kwNkSQeFxI0m4L hrPxiSodgB9OH23Sad/jFKT7iXJB2efpIGwY8sdG0CyHN67Ute74LPL80mnwgbkoHwuz Rcuies9hoAv81L/w6SNuSxn6nEoKfKJGeMKmw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739342002; x=1739946802; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=fDJRc+CzjHHBPozwYIKOoKTE7RtcGO1bK7SyyTKpM/I=; b=jCnmKCGqNcio4eVsGOJCdEQEYr6PYI0k3fgfTXhziCgsOPjGp44F9d1a4RofmnAI47 fOaV1YXL5dxNGkfmr4JJCeOFOnNEHuBFyr/Xdjd8RBYYKWLsNajqaFBNzs68S10/Wl5s GoGNTLI/UcSFv+VVwyHpB3Z0k51/KVM6Js/bmXQXD/RDu6U6a7pZp271X5xaWt+OPUub N+xdTtFaF2OjzhtUbHF6wjG8XDlaozkluuDRZ3IFb45y+mGLmm06FkdzjsHWDegMDq2D c0O1r3oaH9E4ET5ccGgGt/Qc2nj2u51cYSXeqdl8lyPuF+SKuOD/Sz23jiqULj+GTv82 LVtA== X-Forwarded-Encrypted: i=1; AJvYcCU6Bzmag23vVRKwnGZ5lxIJHq6FNhydKEzSScKp7WHpqkiAuGCPjZjeXqc+UQtVgsRoQEpUw6QbOg==@kvack.org X-Gm-Message-State: AOJu0YzOCaMUCb9qeVN+QiV+5WT2ZHfpa2g56HQhPi4JRuvnMUeCiBdQ 0dzP9TQKb8OJxQ1bqttQaytAa+yD0gqZoppQY9MQ0verIBys3m7C+fKYX/TfQQ== X-Gm-Gg: ASbGnctqf1q2I9zawU5gWHD7SuqSrG2xdXsAL3sQgSHuvT6C/FnCP+p1cYjN0Dfq7iM l+RPNj/K0gJz8c8IzbAtLiSb7AzcL3pFgFkTaQJMwDZVpo3SgkeZkkbIuEmgO1tdLBwFKHekqEs wbBfHLNMtS36xNhDm7+4NzyOvr8YDdlzYo0nxdYUUQRJKtZBHCcXh3nzY/2WOM41JeeXGvPD28p 7NRyh7K0Rw9aFe81eCSuuRWIQU/W9MbsdWjbyiN6mgvej3hIPBLIN/Jzydw3+aG17J+VpcHJ2ge zIGxq20KICP5riaJ7g== X-Google-Smtp-Source: AGHT+IGOuYujF/oHMf514Cu6FZkIg94rGPYgxxAA4Ed/XcTGYHy4Z7A3N+oerldYL8V8e8DNCN1OUw== X-Received: by 2002:a17:902:da87:b0:216:3297:92a4 with SMTP id d9443c01a7336-220bbc73ab5mr34950945ad.46.1739342002127; Tue, 11 Feb 2025 22:33:22 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:69f5:6852:451e:8142]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-21f3687bde1sm106393605ad.195.2025.02.11.22.33.20 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 11 Feb 2025 22:33:21 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v5 16/18] zram: do not leak page on recompress_store error path Date: Wed, 12 Feb 2025 15:27:14 +0900 Message-ID: <20250212063153.179231-17-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog In-Reply-To: <20250212063153.179231-1-senozhatsky@chromium.org> References: <20250212063153.179231-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 1D7D3A0004 X-Stat-Signature: p7zzfmr8hrhaf8jxrz3f6f8pejkok8kt X-HE-Tag: 1739342002-179775 X-HE-Meta: U2FsdGVkX1+c3iRa6zZ0oz5TZ/BWbYVKCkLB8PX4UgSXp7esTjHEm1fmClfhEempPUpmkhhU35H8iPHHhHc4KYEzgVzkyGVsKzZY2OSP1b74szh59FOicdrh2JrMCgKbhJfCT+KM8TO6NwkC8m3M9KB7NwWr5AyBCfhdWuWusFxskx3EK9nUBDmkrp3jBjUF/ZJLCDmhm12KupwqnnS+KoeoU9MYPXk6eOUZOssnhE+ebSHL7UVUaXxEV6IKZzNAqaj0XoxDm6zye51SJNJvW9YR5hEJGa48NGEIIw0ss/OzIbz2z0/SXzU1cLSseS6MSDX9FSbygKXxhPWrVIr3rQ8E4NLihUDMQz/2trt+8l/Zg3rpRkGLcLmPYFXwlFGF82AZ7/Mr0e2aBhyJVCSZZum9C0iLWEH3rqFXD3J+ro5h9pt1FRjmDfd2v7p/wOyhfZFJNIi1ZSAUeV1ACga4of9Rym9u5+wQjvnkLq9bglml6C1KmO4bPn0j6LUjViA/kHN+UtrMM5jFEpTXeqzevqiEkwGiqE91XVHVTBrup2zPPB/Lk8/vt5H4dAVLatriqnnwtiGEeUHOSVB6WgnHuHzoVoXtmICOTUDc7gPbvvm1RmCyCUi0EiXhXc0HupF+y0ScW8WnR819034HbVRHQzwEBQd8i6rWOYLJPdgGpx8+MpAJ9P86GdPOsVj1i6VUIyJmhreeeL5CxeeUUXPeOQleGuIWdytoJ5AyXC9BF+SqOUfs9LY5hVMulSAV5zZS1bM4vO2ileTY513UKJcfs0YbSoNHs6bRL0QbVm1yetffHndlMziV7FdBG7+nH3yCsASK8X7yNiHqorP7sRVe9nzSdm5aDP3lw54oCoSeGO/883SFsfRt0uMxSTN+aMsmw1ylRgMvS9a2pAnplOPy/mK1QkuTudQUqtBeB/8N6UucX61QWlWqPznlnMJN/Qp5lF5sGH5ZubsJl5TSs2o T2VMR7Wn TTov2PITV8VPVQ4TtpIRbgodhdoh+OCN62Jz67x00m+BiIuV2TedkPkt2397m1sRiXAZ17IqOp1jGg6FcD1da5GHmicdsptPHJ+7lhbW8xyTjtIMyEEgFjRJ9X48KP6+p6EsPvTQifEzrX0EgZrk2Y151Oiw+b67XPtS4/0SteCJtz4nqIGp8BBZ8yXKn4vN/l48EOKXO3IY20SgZRsfjftUbD1sUt1glSemNdXwBT8VN+7AzsCBLSn9w9Rg9SL9+nq7tdtvYcWqDfO+XWosuPnkISNByFHzKYPLUW4Ac5FBrc0Gb9N9gQrfKSbkp/qBrq7Uxx6roW+AgviyLjJ6sYGmSVX8br7O5OTwIJX28j7behnA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Ensure the page used for local object data is freed on error out path. Fixes: 3f909a60cec1 ("zram: rework recompress target selection strategy") Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index c6310077c221..f4644c29f74e 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -2017,7 +2017,7 @@ static ssize_t recompress_store(struct device *dev, struct zram_pp_slot *pps; u32 mode = 0, threshold = 0; u32 prio, prio_max; - struct page *page; + struct page *page = NULL; ssize_t ret; prio = ZRAM_SECONDARY_COMP; @@ -2161,9 +2161,9 @@ static ssize_t recompress_store(struct device *dev, cond_resched(); } - __free_page(page); - release_init_lock: + if (page) + __free_page(page); release_pp_ctl(zram, ctl); atomic_set(&zram->pp_in_progress, 0); up_read(&zram->init_lock); From patchwork Wed Feb 12 06:27:15 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13971045 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 17434C0219E for ; Wed, 12 Feb 2025 06:33:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A2A54280011; Wed, 12 Feb 2025 01:33:30 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9D9E128000E; Wed, 12 Feb 2025 01:33:30 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 85289280011; Wed, 12 Feb 2025 01:33:30 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 62F7628000E for ; Wed, 12 Feb 2025 01:33:30 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 16C6380D6C for ; Wed, 12 Feb 2025 06:33:30 +0000 (UTC) X-FDA: 83110326180.06.6AAC917 Received: from mail-pj1-f54.google.com (mail-pj1-f54.google.com [209.85.216.54]) by imf04.hostedemail.com (Postfix) with ESMTP id 411F640006 for ; Wed, 12 Feb 2025 06:33:28 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=ZNDHTwgq; spf=pass (imf04.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.216.54 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739342008; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=a8HQ6KKZTY10gG1FVyTeRkLW7p1KkTSFkDVXwYemgoI=; b=4wkoakqXzB/CVlx+leLRTt+YsBv3wL+pbN7oQR52J18y77SOLsYTsJGkB6hU4SpnXR30nm EhHsZEZKz4GaGeVYeNOKlA+JzBBQwVh707V2ESO2WkqaxNI6BTaqthyr8BY9XjydXLzH0I SBisu1kc9JgyhRSqWqfj1RVGkDAFjb0= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=ZNDHTwgq; spf=pass (imf04.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.216.54 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739342008; a=rsa-sha256; cv=none; b=8o/nFlmd++MbjeXxQ4u5lwJj92IqcluGZ+nhsyIPlkD/0mG/yX4vhGHiMC+IuAt7Vy2Cu8 jLBvQZuqScR11ibFgbA5uzgfK3LycwGc5Dw3pQXacO8/aQSTUJMnK6AgtXJyCj0qSjTgS5 soSJbsgsWo1sFytBhN7hCKGV6a8M45w= Received: by mail-pj1-f54.google.com with SMTP id 98e67ed59e1d1-2fa1a3c88c5so8505086a91.3 for ; Tue, 11 Feb 2025 22:33:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739342007; x=1739946807; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=a8HQ6KKZTY10gG1FVyTeRkLW7p1KkTSFkDVXwYemgoI=; b=ZNDHTwgqHMmhVZQ8NvTvHdmPY4a1h3Uls670cQNgZmIDijdD10Cz5RffXn3frg+yhn c6sa+oaxYPT1VAl2GaXDiHt7gFRnKw0R5vOkBHMGHGuFr0X/LBy0yIwPcLKcpoCSqA4x 9rpSFdpuUkdPpkcaQNIl/S5DBDMHpFUq7KIMk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739342007; x=1739946807; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=a8HQ6KKZTY10gG1FVyTeRkLW7p1KkTSFkDVXwYemgoI=; b=AFD1MquwdpP1qFB59h6ZlykRsYDtPG80EqyPhYkPgz40BwxPJFwnOZbCEkLF2oFjl7 /jy6hQV8d8XoNq1eamZ7Yei2jgSl3YUxGsjSKQ7DbUOvqzsMr5GbaguZrikohecGlYVh hc6sHJID10dmdyZIdY50xrQQ1AbhQlSGDom2p2KmSlagGTBAivk2IW/2eYIdiaZ6akNM nmwbOv0MY73AA9KHIXUSIfFSTQvhqmYi9hk9K0aXoOqw4y7DQ2ct5jCdt/NjkrZjfya1 LPLAa7jwHrFgtLZu/zgWXgHoWUKxWraNRHSYwm9S5PcW1w68AxGdXdpkvi4cSjHQ7zXi 8Fgw== X-Forwarded-Encrypted: i=1; AJvYcCXrvmKH6Z4LTwj/vnbjMi4P9+2ogyGVO8lPq3y3kL/3MMl3h77iskHzaYTnUbTLkjnsDLp+4j6kwQ==@kvack.org X-Gm-Message-State: AOJu0YzfogStkWgyz1zhIhZjnqmf8WaQ7fshtdMnHIDbOUm4f2IYl2GX oLOnZn65m+UQZDcto069RFQSblo44mhQtkNOUwLAXWNvBPlTezLCpVhUVxBNfA== X-Gm-Gg: ASbGncuzekOzN2XjcaG/f4cSu/If7YNIHqoK6We7Hwg3h52HN45uceLl3wtoSFg3w0a ItDduEsGyT2Zg2a2Ro01eDuJksSPLYZfUthSvNnDy66/J0w3a5z2TvucDS8JyCRM1T5MQKRL54N hZC9QHwui/4HT0XahDU+OSBCm7mIgoisVjvWssTPvN8KXj/x2UvBvcO48ZdsbH4u7kG+p9mgabs k47ze/oRFNce8sQStcL2cWH/CbefpUNVP7tqPfF4KxzNBfy4tHzboJ0r9VoDhD8A+Gx70JLI8Bn 2FmOPBW7CdiHYgLI3Q== X-Google-Smtp-Source: AGHT+IEQRw5PJmq7wTBgXFBNQOsAEHuqWdKMzAKbQj8udw0upyNAtfwplObq30KxfZI8V98xY/jwVA== X-Received: by 2002:a05:6a00:22d3:b0:730:8e97:bd76 with SMTP id d2e1a72fcca58-7322c591b87mr2706438b3a.9.1739342007145; Tue, 11 Feb 2025 22:33:27 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:69f5:6852:451e:8142]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-730851fa25bsm5556237b3a.180.2025.02.11.22.33.25 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 11 Feb 2025 22:33:26 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v5 17/18] zram: do not leak page on writeback_store error path Date: Wed, 12 Feb 2025 15:27:15 +0900 Message-ID: <20250212063153.179231-18-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog In-Reply-To: <20250212063153.179231-1-senozhatsky@chromium.org> References: <20250212063153.179231-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 411F640006 X-Stat-Signature: 7fs5nkqtceojzsaez1biagmn3bcwtah5 X-HE-Tag: 1739342008-176834 X-HE-Meta: U2FsdGVkX1+NZjt5O4tGuOzwAPxNB+6KrYpjOTyQZuNh0bDd3IkAeUnw7o8tUhR8Uuw28p68+MttFRI8OvckijKmIUkS1SheL5deQV9EPNxpNgx/sJXtAarlXxrXhvxxIRuOBFSqwLHCke/oEiFXLIqdIfwLwTTKhlSb5MZiCmpHQW66CEuXowKOlDc2lbEg2AENtTNypyzU6mCU2777IPHWJkVp3eLPpTzq/2lKGLRtBA482PdlhMi++Gu2E6Gw8WdVgiv0sKJZKdUUsISPOFQefyLv7ubhyRvcXi1pVvM0fckU/R8Kt+nkbDxX04X9903sYQ/HKTkTyNDLCnFHH9+fKD5DKrWX7BorJRKCjr3GKTSj8Cag61S5qfuiXG2BJPLgYLESmvNsUpnX22ox28sDK+6POD4Ro7ixM6wd/X10Zmpf8Jm2GiE4OPzbNLAaS+TcT6TdvER7pMZoQdUs8aksaj+f1vpwNhcOgjgRYxEUP3ewZY2TjkFTTVTFNVFX9tba6t2h6iYEviJa2pyOdZ86Y6KC2EHjjnc8LLSi4sSQ5YnqezJIkWW4VBYtApDntGsHXsp+Oxb8m4tUaat9lhZKs2gfza/ofE3U5EkEmq74A8kjUyQGphzmCgH8BS9JgZwh8JoRU8e9hDAPAz/JnYD4gWkoNu1Kup5BkmykcAXl0BcIMV15DBbnEs3ypPFGIvwaxpKeQdXuPQkQOEjDuwVnQZ0ys+NRGGxkn6JqAouqpuENTD5GGCCXAFJ5kIqIrxWZVRpNIMLlc9lKSi2zpDd+93JUdutGPOgNPm9ruPXcgMAjyY3oaaQtmHlvqXBWTdylv0/Qbdki26SSYqhRv3gULRbPAuVlSvfUasCuRJrkFV6gufSyjBzre0/NlYLAt6xpY172wyw8FVMa39b/7KHCQQNzcquBRsPXPVZDGRgLfgNAy+80e0yYUFnaqmBSeUzwi/TtMmcOUVplwNg FelgR+1P w1Mr1H2rLYU9wcm3PGdSXh5pQ73M8t9rn+9jIT4UuLPWca+ST04Wnz1yyi0LPjshLB+43pkoteT75ySROlIaVhLe7/IT9PwsWBnaDhqIEv17KvfTyoxjZwwNhECVQWo57nTIFCLVLVEskE0N7gWCAgaQn6xd6pJXXtA0efjsPNbB6S1o4fsg+7YrcQ5IoP2rLLZw1Badx1q0QMcGszfHwE4vUBbipTWLvDjBPTUa90hVGY/weO3/T65nIH4UpUi42BjucabEeW7KvUK90BatnTYTAWC0NE5Vcsw0DHrE4sYU1+eOgt5mr9KKTYflILcjBBIPk1+nzA3EJsX5SCuvwNp/MDMLXBb/KO5sCG+3w9VpI1+w= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Ensure the page used for local object data is freed on error out path. Fixes: 330edc2bc059 (zram: rework writeback target selection strategy) Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index f4644c29f74e..10239aea5ce0 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -787,7 +787,7 @@ static ssize_t writeback_store(struct device *dev, unsigned long index = 0; struct bio bio; struct bio_vec bio_vec; - struct page *page; + struct page *page = NULL; ssize_t ret = len; int mode, err; unsigned long blk_idx = 0; @@ -929,8 +929,10 @@ static ssize_t writeback_store(struct device *dev, if (blk_idx) free_block_bdev(zram, blk_idx); - __free_page(page); + release_init_lock: + if (page) + __free_page(page); release_pp_ctl(zram, ctl); atomic_set(&zram->pp_in_progress, 0); up_read(&zram->init_lock); From patchwork Wed Feb 12 06:27:16 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13971046 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 161A2C0219E for ; Wed, 12 Feb 2025 06:33:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8CC1B280013; Wed, 12 Feb 2025 01:33:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 878B3280012; Wed, 12 Feb 2025 01:33:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6F3E7280013; Wed, 12 Feb 2025 01:33:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 4F631280012 for ; Wed, 12 Feb 2025 01:33:36 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 0D1FCB1750 for ; Wed, 12 Feb 2025 06:33:36 +0000 (UTC) X-FDA: 83110326432.01.D03FA56 Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) by imf12.hostedemail.com (Postfix) with ESMTP id E84B540003 for ; Wed, 12 Feb 2025 06:33:33 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=ImWPS5Jf; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf12.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.170 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1739342014; a=rsa-sha256; cv=none; b=t8eEwFwjJf+MT7Yti68FO+EM3Znh7Qsyf4QX0vvP0GNtD7oY1YiTluxb07GZuCKnnNKJUY hllhzvxlHLkEA+pzewe4DwULZ3aPtKrucNccHySnUUlSaiv6rb7n7ClvKJ4K70M0+UhqMh GJnrEQ12vqUgrPVWBQRuWcVJjXnhBNo= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=ImWPS5Jf; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf12.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.170 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1739342014; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=tBUgKAKzM6MxWLflFY2zWKudIzejP2AyvCz3ga5cMyM=; b=UFFyDpIQlDu7a+craJgowRqa7pqrZNJdubAWWmsVTEIUdjkFMpjaz0v+Eq3eyN15Qtniw6 Qvk5J2ZwsDQFn1s0JEMNWZ9bwiJ4sgj0A7kuRhu0Jz10yY6yWO9BXp6rfeRZ7fN8rQBbym 0dA44JkWsopJgZpJstep0+hZ50X2+QY= Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-21f53ad05a0so82905655ad.3 for ; Tue, 11 Feb 2025 22:33:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1739342013; x=1739946813; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=tBUgKAKzM6MxWLflFY2zWKudIzejP2AyvCz3ga5cMyM=; b=ImWPS5Jfq6yavDn0/X/a9d9uA4rMb9ygxMHueqK+xmwOsWa52831CihSztLc0BE2fm zO/psVA6xDxE7hbPXv2o/qjPKmaIvbTa5k8j3oOYCBbOIN2Gm28TbLnfU+NdDyLlByyp cE+foKazxJ/CYehjb3v8NK5EDobC6eNl3P1tg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1739342013; x=1739946813; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tBUgKAKzM6MxWLflFY2zWKudIzejP2AyvCz3ga5cMyM=; b=T1SqvO99G465yIeGCiWTqORZRniGeuBPiStBd60GZ+xYg1NaKxN2y4dFmkcj3UqyoJ WkOOg6bxJFVOVeYv3+HVP8Aam6wW9xCBamQXD1/NdNB9mVvQ3x73H0yPM54uQnZ4+fUx uXVTiYmMVNEMJeQ75GcMnGIu8eRaXAbxcLr/iCd3V/4mvPAbqkVzICojW5JF8e4GicEh DUkwNjdAx+pm8rtyudx+hfCFnC56gWFArsIfYXzd/JN9wV5OKANLkj+VUwdxfzdcjaNg 695xVE+/k1QZemyO90UWFjYdft/0Gpu+Y5OQCmdxXIH4XSYTIsEeMZq2WWkiGVSO97OE UUSQ== X-Forwarded-Encrypted: i=1; AJvYcCVCC0a8OJ/NtMyOHI5YLzkOgIyMIZ3a3m5DoAp0BkgeHoyzwWtnIK5BXehYIekdfBqZOxh/Eu+LjA==@kvack.org X-Gm-Message-State: AOJu0YxWQ0qDLbM4LuiBE636MmIu9zCKCt8ocxsgBRkpe+6iLFLs7U4S g717/pLFIx4Qq9ixGbMyt098iqgrRlurR4w3kNFG3f6N/3Minpu3V/H6B/oV3A== X-Gm-Gg: ASbGncs3mzpMO9GLImICqHiiDc30QNk8sItBspOorG6nsC6oGqx7eYgYi2+qa2u0cIa 1uHmTNJoc2nZRh2KRpwTqvt7mH2nTJzEgontecTZrIPLaXloL4bFo0egy3PoPJH/rl5FMIcKs/J /r6lqo3BG13/IIpVyXFITZUWWrHPbsyUGyi2Lp7iyl1foBIWR66oo8YQnNRXBLqFIr9nOQ1RR7d UtCatmvSkss/rACLSvVFI7HSp02/CVFpr5Cq8jzQ0Znk4FC8UkRgK/scpulsGtseOBfqSNFPzzL zy+qJFTrkdQRiIHo3g== X-Google-Smtp-Source: AGHT+IFlFIHAdO/k5TBM9JhT/XTZlmlIvZZXfAUmXRCed2AuW1r8OudMsiWeuA3jTbNKnlIXBGxP7A== X-Received: by 2002:a05:6a21:3996:b0:1e1:afa9:d39b with SMTP id adf61e73a8af0-1ee5e530b58mr3670243637.7.1739342012925; Tue, 11 Feb 2025 22:33:32 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:69f5:6852:451e:8142]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-73083999abasm5932358b3a.10.2025.02.11.22.33.30 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 11 Feb 2025 22:33:32 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Kairui Song , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v5 18/18] zram: add might_sleep to zcomp API Date: Wed, 12 Feb 2025 15:27:16 +0900 Message-ID: <20250212063153.179231-19-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.502.g6dc24dfdaf-goog In-Reply-To: <20250212063153.179231-1-senozhatsky@chromium.org> References: <20250212063153.179231-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: E84B540003 X-Rspamd-Server: rspam12 X-Stat-Signature: jw586cz1ts7kxiqqrjycthkwkgyfu19q X-HE-Tag: 1739342013-650907 X-HE-Meta: U2FsdGVkX1/WVSI56yrF4ju4DnO6r3qgSOXts14JyC/VMVIMtQvJOVPePPt421Nyhvc6UEgwWK+bbqspG+kcFSGCoLAIUp6BiTD0uvv7I+kWlJljZFp8rRGngWrd5L+rcVgtppMa/S7nOzTZ3W/z8r2QIgaR5uVRIifPgoJqLjiDzG2mwYhViGL91uwW6GFXyUjlhulVl3oa7vA8dYSHHrFENooLw3YfjmgsyvEvdRBJMQdhQWuBPsl13YksAZB3Qa7LSnToYMG2VxSFrRqdeoKOc4K1Qmzhv2G6Dt8sJ0pV0bTLZmu443+9HejT7oYaqqk7WQEIsEBkOseo68ujE8M1cvnIj8ft50pvpuV8Qpn7A21Hm6Df6Q5kzu1HgRxt9kNtO4DnVSbs6PKWT9kjrjuTXCFETbh9LohIKMmGCej3xmX/owAdsYKOL0YakmEqNOwncZ0TS+V+jbLRyi4ThoKz7hx5b0lO7ZPyLdPQQkz2ryex72dzXtV9zRFHB429u1khE/Spl9M2OaJ5lyAMAyml+1NKvv7Lr1gSMx9n60aWCCLdUUBnJvBPMQCZ1laaFicdYGVZ3niGomJmCNM13hwj+FeP9ur0jo60oGsJSDxaKH9IlXwWP4TrDlRXP00x+NtOCX/h0bb5zN6+vfq9VM0Ee0w/9Fr9irxOKWEfUTasVMaVvDukH5nOP7Qo2NGzGCJcUlIsFHq7dOdVxYeN+v8Ih3sMpT2QiwvF5JQgo1U6MaPzMEt2ZGqpYEGGDFKddajg/lYljlPeqcGG1HCv/0yJM7Kk/RlfLE1VbF0ISCl3PKM9hI6t4DCWd7zaul399ZRvT/x/EP+CTptwZOqo+F+gz6jxBeVxlFHBCRcYqj8AbOxyGmjCsySWzaO4qzjexjl2dHg5JqZZrZBa/AfLDkv9Fm80PzPkIGebUtXyo1JIhAWI0Hx9LfLE81sGx0RD0uF2gsEl9GLJ92jwpcq 9yYsSGIp Vvt/HxO/QhM/tjt9YoJoNibzQEo7TaOkHnnEp4FzcFxvcJ/SOr75NvgYzeH0KU1RYIUh5YZlin7Hw9GQ7cKxruETdmXYOy8OZkosJ4JvlK/8sgQusUcU2VvTmDdqe7pxL4ino4H9JcH4EPvrPScpSi7THLbi3wjETNYIu1u/rvzJ++SWW2p/Gw03YmSrfKCirRXKGwgjKdgYHilj9NUSkKNp6lq9EVkG+0Cwh2JfCc732L37pSVsVGp/+H2wc3WidPT0nuCKPGGfZY5HLNq8J9xkoRlO5Z+6grToOlMDrUgqS8LomQgWlBx9nWwLeJ5dTb35/JM3ASEvp56aTXSSAgnUKLr4ayW+VGEMr2ias5xcYGK4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000026, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Explicitly state that zcomp compress/decompress must be called from non-atomic context. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zcomp.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c index 61a9c3ed6f7a..217a77e09dc7 100644 --- a/drivers/block/zram/zcomp.c +++ b/drivers/block/zram/zcomp.c @@ -148,6 +148,7 @@ int zcomp_compress(struct zcomp *comp, struct zcomp_strm *zstrm, }; int ret; + might_sleep(); ret = comp->ops->compress(comp->params, &zstrm->ctx, &req); if (!ret) *dst_len = req.dst_len; @@ -164,6 +165,7 @@ int zcomp_decompress(struct zcomp *comp, struct zcomp_strm *zstrm, .dst_len = PAGE_SIZE, }; + might_sleep(); return comp->ops->decompress(comp->params, &zstrm->ctx, &req); }