From patchwork Wed Jan 22 05:57:39 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13946858 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1C53BC02182 for ; Wed, 22 Jan 2025 05:59:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9F0916B0085; Wed, 22 Jan 2025 00:59:14 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 99F196B0088; Wed, 22 Jan 2025 00:59:14 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 819A16B0089; Wed, 22 Jan 2025 00:59:14 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 5C75F6B0085 for ; Wed, 22 Jan 2025 00:59:14 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 16C75458C9 for ; Wed, 22 Jan 2025 05:59:14 +0000 (UTC) X-FDA: 83034035028.04.3D8ADBA Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) by imf18.hostedemail.com (Postfix) with ESMTP id 0B3471C000E for ; Wed, 22 Jan 2025 05:59:11 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=ADFoH4To; spf=pass (imf18.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.169 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1737525552; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=86ToOuUyh/4wwxhxClpdCo8Ac9RuY+5nejrk9oqhwi8=; b=AqF40xR8cd0v3Dnkcxly/hJGi6xmViMIgoJpeITsa0YsG2EYhua6KLYSBVhqpeF3sczYv5 t0UdvM5B1fYIRDrJa2Omo3voTn06v7RjAAtGMoFqmeCXKDRuK6AAGhvqSBEcWHigTS62pM SZDk+8bu9W5N2CHksk6SQASSFybUAYw= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=ADFoH4To; spf=pass (imf18.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.169 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1737525552; a=rsa-sha256; cv=none; b=1xYUFT/IqLuAJia4KHPipi6SaKE0QX2fCg+wGU46rlb3JLeKFtnIIgnA9hbvFVYF4PLhZB UbG4iJJ4/masRb8PfU2AIRe3EDDcO/4wRB3JdVuvFAoLzbrifjbzroagBYIo5e+S9/SXAA XUPEIfTlAmA1vk/zv+ay23DXyxVNKNo= Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-216395e151bso6880115ad.0 for ; Tue, 21 Jan 2025 21:59:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1737525551; x=1738130351; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=86ToOuUyh/4wwxhxClpdCo8Ac9RuY+5nejrk9oqhwi8=; b=ADFoH4ToaC4wKV6FaMTLgj/o+H6Bqurxssye8Eau+nACDTEjrRKnPtBWOq8K8S4zqK MEl40sdnBxikeIDy1vNbhZ3qm+lRPNVRfIoapaVrYvVtXTE1jBNehQTtovfFHVrXqX9e TS47TNH1ELIHouUL6jjte39NmcNmzfmo/84J0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737525551; x=1738130351; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=86ToOuUyh/4wwxhxClpdCo8Ac9RuY+5nejrk9oqhwi8=; b=Ld0k7zaxgpFeCE47dNqm3wfGFrzCXMu/P8ZcUk5yfsCNzA7xVoztO8/F9uzv6EJ2mQ 3e4CR4vtNFuIPM+I0TjxusFTCi7JpK0tlSgl2JR3G3yB0pncf//IkGRwQAFDqqaNTZMY eofDdl5LtTRbcH2YDCdPCkc9YO78k4mCjvWfLWsT0oT+D7UWwj2GktGMvD0JEuOw8TP6 +WOZIiZZggyp4Z2BCtbd8ye7GRqVOmzJ0zS0B7GPKPNnMhjuZdxYfxqGq1E5P7mr07q/ nup5NBH7eqlfr37bcpPCs5l9uI0r9wfvsK5SXJjjqEFNQAraafi+TBiFgmxZtfNEREIu Nvnw== X-Gm-Message-State: AOJu0YwHqaMhFvjNKYLVadK8rchI0hDwQHLTMxDpU30DPBSts243B9oT wyEv7fWawcLOhm13F04PXLTpo964Mto7aRg2PuTXcOgPK0G4cLZpj0Tx61vDiQ== X-Gm-Gg: ASbGncsrgHCZvRw8+Ce23cEgaLwWoc77Y4ixO8LylFeEy8ntIlkXTHafPSu9k9qSso1 YlsYXxrwmnRAjOCzMIfD6mi0BzepfLyW+BSnY+23treDhJc17TKRGEEZwunRrmRrXHVJy42jATo +6LXaayAlr8OyjShTV1DrWPvyabqcDuDEVjFqMoAmWVmOApt46QISjTLVBpwE9Zhx2MnbtKsw8u CMCSjE4w6P8oIx+TvHsK3/MkztOMOxnkC/tSm3eGVuXHFdGWs+mjxSpIEx7RzHfkF59rK4x X-Google-Smtp-Source: AGHT+IFI3izmvrB4GENXvTqLLu8ZEkzBN0mPpSFCervBudtCE84arWbb2mVIRxYrPs+7DZVxq2jA1A== X-Received: by 2002:a17:902:cec5:b0:215:a56f:1e50 with SMTP id d9443c01a7336-21c35c8b62fmr273018845ad.8.1737525550752; Tue, 21 Jan 2025 21:59:10 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:2902:8f0f:12b3:c251]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-21c2d4042desm87675045ad.242.2025.01.21.21.59.08 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 21 Jan 2025 21:59:10 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH 1/7] zram: switch to non-atomic entry locking Date: Wed, 22 Jan 2025 14:57:39 +0900 Message-ID: <20250122055831.3341175-2-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.0.rc2.279.g1de40edade-goog In-Reply-To: <20250122055831.3341175-1-senozhatsky@chromium.org> References: <20250122055831.3341175-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 0B3471C000E X-Stat-Signature: 1cnduotf4gbbjzbj9uxcghsktjd4ow4i X-Rspam-User: X-HE-Tag: 1737525551-850062 X-HE-Meta: U2FsdGVkX1/7CnskfS/Qe/p/RBG/r3aDQCBCHqAY/llUijo38B8zfxQDusm9gOlDFATayO7zGghurFq1n19uzRdt0JfLj1aXJrKJaIUJg+Bs7DroiyVluqtf85acEPfNRvnCO5eX8Jzdor2M8z879abSzuIOb0qNr9nUN83zvUyF8rY6zer4K+WfK1ioQEvjKoIHrhrxX7CNTWcmX8uFB3Rh1qL8BFNR5hYGrtBrhod960T7Q5AJjb9+MEWTVUSIIgwI3JHw/G3eYPC55ae+yg5sMxOfN6F9toF/FI2CFLo1S1zP9JTQflpZIa4ZZrBBmxreuw/Q/4ftbK/nR3PvW1wQnTmdvVhZtXOIf6qbca9/eZRlRIHCb7oUI0Vsm0PUCMzqPjfw+DvPYdDYgTkRSkRPKMH0n8kvu7dSMvi8wZ3bGEht5sjSAlBgBH1P0jNjeUGHvlGPd6mZChDfYgfiQGHivORsY5u9lZIKcI2YVnRbGzlf586YPqz9O3ii4lrDnybLRSRwKOx8DrDnl4UNGQRkvp/77FWGRUNAOE9IPQr7whQpJaJQ/SSVwMhDRmBmA3HMTkykRalrWcfflPrHbW3hyeWZhIUREDdwdYQQzo1sK2hv5j9FU7Xy6MOABiZxMY439g31Qv/I4kiRgfR4W04IBMZ7HojI1/CzbmSrNnd/BxUVvqbWOJmC7euqUHQA3DzOeornHpOasL7DZYizCDwBfRRl4mpjYj47V6puGp1Qr08jwpGefWMg/Br4tX1WGu+ZpzNoU2tOA7PvHoYFS+7ffD71A3fRDyIQa7dVb2tZ6ZxI6z1J9TEfxcZ+hq1TLC/T8Jd+nz7DY1nNNboN7tNuv0wqKJ/av9uvtK0I32UA3APuh+DRKOBXn7L9Cns1HiXmf8VvGV+dZ6ngwnN98EhvYvIOh7x506piBPdsklo5kZsfFfCltaweXwNUkbfQf5/meo0k7/wCsTAqtbL 2uwBwBJF pAdZHYAalDQ3LN5C5TU3QHpSrpNyp/8JCOo2ZZDYOddBuCA+NKP5abgUiqbMdMR49C+r4cQanIa7lgaKgy82OkL1QKBEqo0Vx/iOgQElyPdy8pOvnc8Wr05yaZy5QBZe2e+lU2UyFQB4X0oZZg43BNbkoloMU4EnOCOOxSNMlCIfG0cT56L1EUbbGF9WZBnFmtnr9MfXx/jeolUcES8XiyT3MPjp7TUHHvFC1fVS6TEnn3jmauI9s7RgLgkgQUUz++P6ZLwUzV7f9eKiBZYCZTY/oCjzErArua5SjS9w+Hs2pwx5E0bbTJnib+g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Concurrent modifications of meta table entries is now handled by per-entry spin-lock. This has a number of shortcomings. First, this imposes atomic requirements on compression backends. zram can call both zcomp_compress() and zcomp_decompress() under entry spin-lock, which implies that we can use only compression algorithms that don't schedule/sleep/wait during compression and decompression. This, for instance, makes it impossible to use some of the ASYNC compression algorithms (H/W compression, etc.) implementations. Second, this can potentially trigger watchdogs. For example, entry re-compression with secondary algorithms is performed under entry spin-lock. Given that we chain secondary compression algorithms and that some of them can be configured for best compression ratio (and worst compression speed) zram can stay under spin-lock for quite some time. Do not use per-entry spin-locks and instead switch to a bucket RW-sem locking scheme. Each rw-lock controls access to a number of zram entries (a bucket). Each bucket is configured to protect 12 (for the time being) pages of zram disk, in order to minimize memory footprint of bucket locks - we cannot afford a mutex of rwsem per-entry, that can use hundreds of megabytes on a relatively common setup, yet we still want some degree of parallelism wrt entries access. Bucket lock is taken in write-mode only to modify zram entry, compression and zsmalloc handle allocation performed outside of bucket lock scopr; decompression is performed under bucket read-lock. At a glance there doesn't seem to be any immediate performance difference: time make -j$(nproc) ARCH=x86 all modules # defconfig, 24-vCPU VM BEFORE ------ Kernel: arch/x86/boot/bzImage is ready (#1) 1371.43user 158.84system 1:30.70elapsed 1687%CPU (0avgtext+0avgdata 825620maxresident)k 32504inputs+1768352outputs (259major+51536895minor)pagefaults 0swaps AFTER ----- Kernel: arch/x86/boot/bzImage is ready (#1) 1366.79user 158.82system 1:31.17elapsed 1673%CPU (0avgtext+0avgdata 825676maxresident)k 32504inputs+1768352outputs (263major+51538123minor)pagefaults 0swaps Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 151 ++++++++++++++++++++-------------- drivers/block/zram/zram_drv.h | 8 +- 2 files changed, 98 insertions(+), 61 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 9f5020b077c5..7eb7feba3cac 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -58,19 +58,44 @@ static void zram_free_page(struct zram *zram, size_t index); static int zram_read_from_zspool(struct zram *zram, struct page *page, u32 index); -static int zram_slot_trylock(struct zram *zram, u32 index) +static u32 zram_bucket_idx(u32 index) { - return spin_trylock(&zram->table[index].lock); + return index / ZRAM_PAGES_PER_BUCKET_LOCK; } -static void zram_slot_lock(struct zram *zram, u32 index) +static int zram_slot_write_trylock(struct zram *zram, u32 index) { - spin_lock(&zram->table[index].lock); + u32 idx = zram_bucket_idx(index); + + return down_write_trylock(&zram->locks[idx].lock); +} + +static void zram_slot_write_lock(struct zram *zram, u32 index) +{ + u32 idx = zram_bucket_idx(index); + + down_write(&zram->locks[idx].lock); +} + +static void zram_slot_write_unlock(struct zram *zram, u32 index) +{ + u32 idx = zram_bucket_idx(index); + + up_write(&zram->locks[idx].lock); +} + +static void zram_slot_read_lock(struct zram *zram, u32 index) +{ + u32 idx = zram_bucket_idx(index); + + down_read(&zram->locks[idx].lock); } -static void zram_slot_unlock(struct zram *zram, u32 index) +static void zram_slot_read_unlock(struct zram *zram, u32 index) { - spin_unlock(&zram->table[index].lock); + u32 idx = zram_bucket_idx(index); + + up_read(&zram->locks[idx].lock); } static inline bool init_done(struct zram *zram) @@ -93,7 +118,6 @@ static void zram_set_handle(struct zram *zram, u32 index, unsigned long handle) zram->table[index].handle = handle; } -/* flag operations require table entry bit_spin_lock() being held */ static bool zram_test_flag(struct zram *zram, u32 index, enum zram_pageflags flag) { @@ -229,9 +253,9 @@ static void release_pp_slot(struct zram *zram, struct zram_pp_slot *pps) { list_del_init(&pps->entry); - zram_slot_lock(zram, pps->index); + zram_slot_write_lock(zram, pps->index); zram_clear_flag(zram, pps->index, ZRAM_PP_SLOT); - zram_slot_unlock(zram, pps->index); + zram_slot_write_unlock(zram, pps->index); kfree(pps); } @@ -394,11 +418,11 @@ static void mark_idle(struct zram *zram, ktime_t cutoff) * * And ZRAM_WB slots simply cannot be ZRAM_IDLE. */ - zram_slot_lock(zram, index); + zram_slot_write_lock(zram, index); if (!zram_allocated(zram, index) || zram_test_flag(zram, index, ZRAM_WB) || zram_test_flag(zram, index, ZRAM_SAME)) { - zram_slot_unlock(zram, index); + zram_slot_write_unlock(zram, index); continue; } @@ -410,7 +434,7 @@ static void mark_idle(struct zram *zram, ktime_t cutoff) zram_set_flag(zram, index, ZRAM_IDLE); else zram_clear_flag(zram, index, ZRAM_IDLE); - zram_slot_unlock(zram, index); + zram_slot_write_unlock(zram, index); } } @@ -709,7 +733,7 @@ static int scan_slots_for_writeback(struct zram *zram, u32 mode, INIT_LIST_HEAD(&pps->entry); - zram_slot_lock(zram, index); + zram_slot_write_lock(zram, index); if (!zram_allocated(zram, index)) goto next; @@ -731,7 +755,7 @@ static int scan_slots_for_writeback(struct zram *zram, u32 mode, place_pp_slot(zram, ctl, pps); pps = NULL; next: - zram_slot_unlock(zram, index); + zram_slot_write_unlock(zram, index); } kfree(pps); @@ -822,7 +846,7 @@ static ssize_t writeback_store(struct device *dev, } index = pps->index; - zram_slot_lock(zram, index); + zram_slot_read_lock(zram, index); /* * scan_slots() sets ZRAM_PP_SLOT and relases slot lock, so * slots can change in the meantime. If slots are accessed or @@ -833,7 +857,7 @@ static ssize_t writeback_store(struct device *dev, goto next; if (zram_read_from_zspool(zram, page, index)) goto next; - zram_slot_unlock(zram, index); + zram_slot_read_unlock(zram, index); bio_init(&bio, zram->bdev, &bio_vec, 1, REQ_OP_WRITE | REQ_SYNC); @@ -860,7 +884,7 @@ static ssize_t writeback_store(struct device *dev, } atomic64_inc(&zram->stats.bd_writes); - zram_slot_lock(zram, index); + zram_slot_write_lock(zram, index); /* * Same as above, we release slot lock during writeback so * slot can change under us: slot_free() or slot_free() and @@ -882,7 +906,7 @@ static ssize_t writeback_store(struct device *dev, zram->bd_wb_limit -= 1UL << (PAGE_SHIFT - 12); spin_unlock(&zram->wb_limit_lock); next: - zram_slot_unlock(zram, index); + zram_slot_write_unlock(zram, index); release_pp_slot(zram, pps); cond_resched(); @@ -1001,7 +1025,7 @@ static ssize_t read_block_state(struct file *file, char __user *buf, for (index = *ppos; index < nr_pages; index++) { int copied; - zram_slot_lock(zram, index); + zram_slot_read_lock(zram, index); if (!zram_allocated(zram, index)) goto next; @@ -1019,13 +1043,13 @@ static ssize_t read_block_state(struct file *file, char __user *buf, ZRAM_INCOMPRESSIBLE) ? 'n' : '.'); if (count <= copied) { - zram_slot_unlock(zram, index); + zram_slot_read_unlock(zram, index); break; } written += copied; count -= copied; next: - zram_slot_unlock(zram, index); + zram_slot_read_unlock(zram, index); *ppos += 1; } @@ -1451,37 +1475,44 @@ static void zram_meta_free(struct zram *zram, u64 disksize) zs_destroy_pool(zram->mem_pool); vfree(zram->table); zram->table = NULL; + vfree(zram->locks); + zram->locks = NULL; } static bool zram_meta_alloc(struct zram *zram, u64 disksize) { - size_t num_pages, index; + size_t num_ents, index; - num_pages = disksize >> PAGE_SHIFT; - zram->table = vzalloc(array_size(num_pages, sizeof(*zram->table))); + num_ents = disksize >> PAGE_SHIFT; + zram->table = vzalloc(array_size(num_ents, sizeof(*zram->table))); if (!zram->table) - return false; + goto error; + + num_ents /= ZRAM_PAGES_PER_BUCKET_LOCK; + zram->locks = vzalloc(array_size(num_ents, sizeof(*zram->locks))); + if (!zram->locks) + goto error; zram->mem_pool = zs_create_pool(zram->disk->disk_name); - if (!zram->mem_pool) { - vfree(zram->table); - zram->table = NULL; - return false; - } + if (!zram->mem_pool) + goto error; if (!huge_class_size) huge_class_size = zs_huge_class_size(zram->mem_pool); - for (index = 0; index < num_pages; index++) - spin_lock_init(&zram->table[index].lock); + for (index = 0; index < num_ents; index++) + init_rwsem(&zram->locks[index].lock); + return true; + +error: + vfree(zram->table); + zram->table = NULL; + vfree(zram->locks); + zram->locks = NULL; + return false; } -/* - * To protect concurrent access to the same index entry, - * caller should hold this table index entry's bit_spinlock to - * indicate this index entry is accessing. - */ static void zram_free_page(struct zram *zram, size_t index) { unsigned long handle; @@ -1602,17 +1633,17 @@ static int zram_read_page(struct zram *zram, struct page *page, u32 index, { int ret; - zram_slot_lock(zram, index); + zram_slot_read_lock(zram, index); if (!zram_test_flag(zram, index, ZRAM_WB)) { /* Slot should be locked through out the function call */ ret = zram_read_from_zspool(zram, page, index); - zram_slot_unlock(zram, index); + zram_slot_read_unlock(zram, index); } else { /* * The slot should be unlocked before reading from the backing * device. */ - zram_slot_unlock(zram, index); + zram_slot_read_unlock(zram, index); ret = read_from_bdev(zram, page, zram_get_handle(zram, index), parent); @@ -1655,10 +1686,10 @@ static int zram_bvec_read(struct zram *zram, struct bio_vec *bvec, static int write_same_filled_page(struct zram *zram, unsigned long fill, u32 index) { - zram_slot_lock(zram, index); + zram_slot_write_lock(zram, index); zram_set_flag(zram, index, ZRAM_SAME); zram_set_handle(zram, index, fill); - zram_slot_unlock(zram, index); + zram_slot_write_unlock(zram, index); atomic64_inc(&zram->stats.same_pages); atomic64_inc(&zram->stats.pages_stored); @@ -1693,11 +1724,11 @@ static int write_incompressible_page(struct zram *zram, struct page *page, kunmap_local(src); zs_unmap_object(zram->mem_pool, handle); - zram_slot_lock(zram, index); + zram_slot_write_lock(zram, index); zram_set_flag(zram, index, ZRAM_HUGE); zram_set_handle(zram, index, handle); zram_set_obj_size(zram, index, PAGE_SIZE); - zram_slot_unlock(zram, index); + zram_slot_write_unlock(zram, index); atomic64_add(PAGE_SIZE, &zram->stats.compr_data_size); atomic64_inc(&zram->stats.huge_pages); @@ -1718,9 +1749,9 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) bool same_filled; /* First, free memory allocated to this slot (if any) */ - zram_slot_lock(zram, index); + zram_slot_write_lock(zram, index); zram_free_page(zram, index); - zram_slot_unlock(zram, index); + zram_slot_write_unlock(zram, index); mem = kmap_local_page(page); same_filled = page_same_filled(mem, &element); @@ -1790,10 +1821,10 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); zs_unmap_object(zram->mem_pool, handle); - zram_slot_lock(zram, index); + zram_slot_write_lock(zram, index); zram_set_handle(zram, index, handle); zram_set_obj_size(zram, index, comp_len); - zram_slot_unlock(zram, index); + zram_slot_write_unlock(zram, index); /* Update stats */ atomic64_inc(&zram->stats.pages_stored); @@ -1850,7 +1881,7 @@ static int scan_slots_for_recompress(struct zram *zram, u32 mode, INIT_LIST_HEAD(&pps->entry); - zram_slot_lock(zram, index); + zram_slot_write_lock(zram, index); if (!zram_allocated(zram, index)) goto next; @@ -1871,7 +1902,7 @@ static int scan_slots_for_recompress(struct zram *zram, u32 mode, place_pp_slot(zram, ctl, pps); pps = NULL; next: - zram_slot_unlock(zram, index); + zram_slot_write_unlock(zram, index); } kfree(pps); @@ -2162,7 +2193,7 @@ static ssize_t recompress_store(struct device *dev, if (!num_recomp_pages) break; - zram_slot_lock(zram, pps->index); + zram_slot_write_lock(zram, pps->index); if (!zram_test_flag(zram, pps->index, ZRAM_PP_SLOT)) goto next; @@ -2170,7 +2201,7 @@ static ssize_t recompress_store(struct device *dev, &num_recomp_pages, threshold, prio, prio_max); next: - zram_slot_unlock(zram, pps->index); + zram_slot_write_unlock(zram, pps->index); release_pp_slot(zram, pps); if (err) { @@ -2217,9 +2248,9 @@ static void zram_bio_discard(struct zram *zram, struct bio *bio) } while (n >= PAGE_SIZE) { - zram_slot_lock(zram, index); + zram_slot_write_lock(zram, index); zram_free_page(zram, index); - zram_slot_unlock(zram, index); + zram_slot_write_unlock(zram, index); atomic64_inc(&zram->stats.notify_free); index++; n -= PAGE_SIZE; @@ -2248,9 +2279,9 @@ static void zram_bio_read(struct zram *zram, struct bio *bio) } flush_dcache_page(bv.bv_page); - zram_slot_lock(zram, index); + zram_slot_write_lock(zram, index); zram_accessed(zram, index); - zram_slot_unlock(zram, index); + zram_slot_write_unlock(zram, index); bio_advance_iter_single(bio, &iter, bv.bv_len); } while (iter.bi_size); @@ -2278,9 +2309,9 @@ static void zram_bio_write(struct zram *zram, struct bio *bio) break; } - zram_slot_lock(zram, index); + zram_slot_write_lock(zram, index); zram_accessed(zram, index); - zram_slot_unlock(zram, index); + zram_slot_write_unlock(zram, index); bio_advance_iter_single(bio, &iter, bv.bv_len); } while (iter.bi_size); @@ -2321,13 +2352,13 @@ static void zram_slot_free_notify(struct block_device *bdev, zram = bdev->bd_disk->private_data; atomic64_inc(&zram->stats.notify_free); - if (!zram_slot_trylock(zram, index)) { + if (!zram_slot_write_trylock(zram, index)) { atomic64_inc(&zram->stats.miss_free); return; } zram_free_page(zram, index); - zram_slot_unlock(zram, index); + zram_slot_write_unlock(zram, index); } static void zram_comp_params_reset(struct zram *zram) diff --git a/drivers/block/zram/zram_drv.h b/drivers/block/zram/zram_drv.h index db78d7c01b9a..b272ede404b0 100644 --- a/drivers/block/zram/zram_drv.h +++ b/drivers/block/zram/zram_drv.h @@ -28,6 +28,7 @@ #define ZRAM_SECTOR_PER_LOGICAL_BLOCK \ (1 << (ZRAM_LOGICAL_BLOCK_SHIFT - SECTOR_SHIFT)) +#define ZRAM_PAGES_PER_BUCKET_LOCK 12 /* * ZRAM is mainly used for memory efficiency so we want to keep memory @@ -63,13 +64,17 @@ enum zram_pageflags { /* Allocated for each disk page */ struct zram_table_entry { unsigned long handle; + /* u32 suffice for flags + u32 padding */ unsigned int flags; - spinlock_t lock; #ifdef CONFIG_ZRAM_TRACK_ENTRY_ACTIME ktime_t ac_time; #endif }; +struct zram_bucket_lock { + struct rw_semaphore lock; +}; + struct zram_stats { atomic64_t compr_data_size; /* compressed size of pages stored */ atomic64_t failed_reads; /* can happen when memory is too low */ @@ -101,6 +106,7 @@ struct zram_stats { struct zram { struct zram_table_entry *table; + struct zram_bucket_lock *locks; struct zs_pool *mem_pool; struct zcomp *comps[ZRAM_MAX_COMPS]; struct zcomp_params params[ZRAM_MAX_COMPS]; From patchwork Wed Jan 22 05:57:40 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13946859 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C306AC0218C for ; Wed, 22 Jan 2025 05:59:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 339E56B0089; Wed, 22 Jan 2025 00:59:19 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2E8676B008A; Wed, 22 Jan 2025 00:59:19 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 163186B008C; Wed, 22 Jan 2025 00:59:19 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id E35C76B0089 for ; Wed, 22 Jan 2025 00:59:18 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 7124B45C3E for ; Wed, 22 Jan 2025 05:59:18 +0000 (UTC) X-FDA: 83034035196.27.33F3782 Received: from mail-pj1-f48.google.com (mail-pj1-f48.google.com [209.85.216.48]) by imf21.hostedemail.com (Postfix) with ESMTP id 92F4B1C0003 for ; Wed, 22 Jan 2025 05:59:16 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b="OrGY25/D"; spf=pass (imf21.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.216.48 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1737525556; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nNJBFEkJ6QI9GsDPc+DtKnpVKRZ1iP1KV3SxEj7y/i4=; b=0c2YD0dm7UR3HUgvW/WXxd3PBzoANab1GHn02OyDryLfo1x8h4842KgAsq7WLkW7fU+Z9q c64oVEZJSp8umgi99oY1qy4ittvGfER/Gkbb8gW5KEPStus9PaD9Gqza/ms0Uxmc2ILWLs I1poD1IIlWMC87z821JopACIwX06Q4Q= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b="OrGY25/D"; spf=pass (imf21.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.216.48 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1737525556; a=rsa-sha256; cv=none; b=GAjtmNRqVM08D/WvwM8DzDhLuNAST3GhRlEnPkgY3/i05bXTerPBYBOp+jvFi8ZwXOHgjE NzXJN8UnhCIh5zCA9aUjPZnzJTJ6fC2y4dIgRRvADbe3uizci2SF2PYM9832xcLhpZksoa u5iXhraGbRNzl+Qcn899dYPVaA3uQ5Q= Received: by mail-pj1-f48.google.com with SMTP id 98e67ed59e1d1-2ef72924e53so11147059a91.3 for ; Tue, 21 Jan 2025 21:59:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1737525555; x=1738130355; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=nNJBFEkJ6QI9GsDPc+DtKnpVKRZ1iP1KV3SxEj7y/i4=; b=OrGY25/DBfdgF/Ph1KZxxzRFRVK5acNxcuiBfRhRDmnyO4hvsQo8AkqdABlG/l/Av4 CFL2HCwkKooIXbi2z7IwiYTgVbX7mwtUCNfAZyW86SrYG1QTqa5W49HBIG9qCyg6NNrf wfLst3dGnN4/Vc5061RiSvFfHxt+2EH7Wa+jE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737525555; x=1738130355; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=nNJBFEkJ6QI9GsDPc+DtKnpVKRZ1iP1KV3SxEj7y/i4=; b=n0CO1fxM3wzQ5F/iEiMQTyB3CJbkBY5EFFRt5ZydJ0OgRYsnsenfIHDiWOuT1JnQKi KNZjLGh4hRGmo8VbX95IhGh6WnZts1klJdoxEF15iRS9S1EdLlSyvwRiSaqbDNymYd1N 1zuViAN9XVKpYiZQ+53DgPTUy6z1tpCI0ripCPXSHrGPlkiXBPcCghop+mMPb2VGNaeU dJIA8Mf7wdrdeYyxM2nh0jp7qaJSfdR0hQh37tYNx/ksglB5FilFsTjfQps6rcZsYbKd 3ITKlg5Vxw2Ig18KfaLHyg91tfohe4akgiVb3PwTbRgY1yiy41S898nFtQDseBaGZKrO Wp5A== X-Gm-Message-State: AOJu0YzjGtHQkblWHX2dYbtFoLC4ANQVPOojfsZNQuG/7fYudhpipdP4 J9Hx8JTFUWaYI7iqU4keVDqrMzyZHzNR/LgfozMcSat9CdZ0xlcXBFpnXqD5/Q== X-Gm-Gg: ASbGnct99BTG+YCnVFRj6OaH0wnf/Pbsv/K9ZDnOIgIcuLLd/PNPWZphNa2nL9A34/2 QOHbH/D5Yw6sf5YJcIbzatgUI7ExpEvfvBe3IgB+KxydvVjLrkd7j0VBmPegz3iknOC0qHuHgcH UNNEfjDMwLWLo0Xw8l9PfU8zrxiRPsI7BLr1sDK2+YRDHrOx5Hd2j5RvQppVljmzI5nDEP/0/FR KQt5blmccVHnqVOTC8E+MEH2VAaJRzWonVgyud/IzVI13WBGF3qWbLlRVnVVl1xp0plJNie X-Google-Smtp-Source: AGHT+IFkiDY8ubThsiSVPvY+v+ZW4p7rcixmuj2cYd+76LLUa3zPITbS5RuEp5qCovKlriMjDRnvpw== X-Received: by 2002:a17:90b:540b:b0:2f2:8bdd:cd8b with SMTP id 98e67ed59e1d1-2f782d972a2mr28408515a91.29.1737525555228; Tue, 21 Jan 2025 21:59:15 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:2902:8f0f:12b3:c251]) by smtp.gmail.com with UTF8SMTPSA id 98e67ed59e1d1-2f7e6a78ba1sm618280a91.14.2025.01.21.21.59.13 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 21 Jan 2025 21:59:14 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH 2/7] zram: do not use per-CPU compression streams Date: Wed, 22 Jan 2025 14:57:40 +0900 Message-ID: <20250122055831.3341175-3-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.0.rc2.279.g1de40edade-goog In-Reply-To: <20250122055831.3341175-1-senozhatsky@chromium.org> References: <20250122055831.3341175-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 92F4B1C0003 X-Stat-Signature: f6nfkyhyjeabuuo48djnmo7n1ddbfoch X-Rspam-User: X-HE-Tag: 1737525556-260575 X-HE-Meta: U2FsdGVkX1/kxdBgb7Z4n6iMvc+R/DpNarVImZjjkbu+rYpDF7TCXigPavukztqRdsFaA23566jXjVi5PJpD1P6uB6DlzJM5oIoDcFGXU2b84srOQEPj+uQ7mhcDezml8q1HIjJ+r+9yirfbTGD/KqoMSzyMT2mRuvpsjSipbRrW90ql7lsu8QtuIr0Ur1SRpzv9RWFz5eVwooVb6VWovEG+vruqqzzCyTl4c+5HnuHmutahQd3FH+H/bQMU4VVoIrATng8ob2weBN1xD7CGX031yH3GCdD3i88u6zZenypuFBfshwAXR1gPTGVjb8oBhUU3k1BveD5ltRqvX7bbfqmlT+Ffxog6zMBbdsoGRp9Qfsf8khB+hG66EdOXVOPuecCytcFE/vCNK3QMmv3AuV/rWD/okBMgfWuctsX2DcWvg0yrcKFNNqmL1CsmSGnoHWMRQNF4P0yEyfBABogzkzQSAYRqQByUNxuLuaqlMzVsMYqEKz5KpLdIZdiieIgsm+KWMOXPG7Y9UA/gtV/jO4Aw150BEhTjTtjP5KXDwhLe+3CO5wWUrVXAgEe7pEOxzzl7nlSitN+XdrelopndLltUkdBGiMGzZK3FeETnd+MuxOj6z+qgelSkMwS9JybQmfqaIRKabvhJ6xohcs+9h2W4hgSa2eApaKT86ITe5jUHDy8JFTH64XsXaQZhdPHK7u3xgrj4ynq6VJdBYGg0/CfwqGhCVW0lTsql/SJzp7A5nx4Rdas8H282kPQqjaEkOaqXTl2uen3bPyucIXN1Cfk+pBMiChoa+jdolWaAejhvZ4FPfKQBkutusTkhgUCuJ+CMZWmD8Crc+FPHBEXqk3Bu4WkREhIuaEHKwsxEeNg89xT9/iTtI2ZewA4Wg5pZzUUcItZSkEf8qepBb22v7Vkh1i3f0DzkvGOZn+HN/hVtSJ+vCTMefq3EQdn18Ix/T1fV82+UjSxAw/vxsy3 XdqV6DXD cOrVcXNL1A+cMFE1miYvYBi9ItbKzlQ7+Cuq+stP++t0yG5cdRX0cFqvdwf3yFaFfHKRKMHJfiRXGGWCff/rLFWkUaGzpYRJ6p4nE18/qN/5bCN5xDhP5ujo3FHuYYoNcRZvQJCyWypBc0CPo9LfYLIW2McCT3MvtnWsItJNlfHqTIfjQb9PMWDky+wJHK2jBo9DMyEdQwbtWnUW1/3EgIkf/hZBSuzobIBdCNzA92OGxf2tnZwkwCW6fIhLTDzDqc+GKepGTbX7G7TxsaaEprs3rZBMss8Ssr31Mw/yy348IDsnuqmU/A2Pj0g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Similarly to per-entry spin-lock per-CPU compression streams also have a number of shortcoming. First, per-CPU stream access has to be done from a non-preemptible (atomic) section, which imposes the same atomicity requirements on compression backends as entry spin-lock do and makes it impossible to use algorithms that can schedule/wait/sleep during compression and decompression. Second, per-CPU streams noticeably increase memory usage (actually more like wastage) of secondary compression streams. The problem is that secondary compression streams are allocated per-CPU, just like the primary streams are. Yet we never use more that one secondary stream at a time, because recompression is a single threaded action. Which means that remaining num_online_cpu() - 1 streams are allocated for nothing, and this is per-priority list (we can have several secondary compression algorithms). Depending on the algorithm this may lead to a significant memory wastage, in addition each stream also carries a workmem buffer (2 physical pages). Instead of per-CPU streams, maintain a list of idle compression streams and allocate new streams on-demand (something that we used to do many years ago). So that zram read() and write() become non-atomic and ease requirements on the compression algorithm implementation. This also means that we now should have only one secondary stream per-priority list. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zcomp.c | 162 +++++++++++++++++++--------------- drivers/block/zram/zcomp.h | 17 ++-- drivers/block/zram/zram_drv.c | 29 +++--- include/linux/cpuhotplug.h | 1 - 4 files changed, 108 insertions(+), 101 deletions(-) diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c index bb514403e305..5d8298fc2616 100644 --- a/drivers/block/zram/zcomp.c +++ b/drivers/block/zram/zcomp.c @@ -43,31 +43,40 @@ static const struct zcomp_ops *backends[] = { NULL }; -static void zcomp_strm_free(struct zcomp *comp, struct zcomp_strm *zstrm) +static void zcomp_strm_free(struct zcomp *comp, struct zcomp_strm *strm) { - comp->ops->destroy_ctx(&zstrm->ctx); - vfree(zstrm->buffer); - zstrm->buffer = NULL; + comp->ops->destroy_ctx(&strm->ctx); + vfree(strm->buffer); + kfree(strm); } -static int zcomp_strm_init(struct zcomp *comp, struct zcomp_strm *zstrm) +static struct zcomp_strm *zcomp_strm_alloc(struct zcomp *comp) { + struct zcomp_strm *strm; int ret; - ret = comp->ops->create_ctx(comp->params, &zstrm->ctx); - if (ret) - return ret; + strm = kzalloc(sizeof(*strm), GFP_KERNEL); + if (!strm) + return NULL; + + INIT_LIST_HEAD(&strm->entry); + + ret = comp->ops->create_ctx(comp->params, &strm->ctx); + if (ret) { + kfree(strm); + return NULL; + } /* - * allocate 2 pages. 1 for compressed data, plus 1 extra for the - * case when compressed size is larger than the original one + * allocate 2 pages. 1 for compressed data, plus 1 extra in case if + * compressed data is larger than the original one. */ - zstrm->buffer = vzalloc(2 * PAGE_SIZE); - if (!zstrm->buffer) { - zcomp_strm_free(comp, zstrm); - return -ENOMEM; + strm->buffer = vzalloc(2 * PAGE_SIZE); + if (!strm->buffer) { + zcomp_strm_free(comp, strm); + return NULL; } - return 0; + return strm; } static const struct zcomp_ops *lookup_backend_ops(const char *comp) @@ -109,13 +118,59 @@ ssize_t zcomp_available_show(const char *comp, char *buf) struct zcomp_strm *zcomp_stream_get(struct zcomp *comp) { - local_lock(&comp->stream->lock); - return this_cpu_ptr(comp->stream); + struct zcomp_strm *strm; + + might_sleep(); + + while (1) { + spin_lock(&comp->strm_lock); + if (!list_empty(&comp->idle_strm)) { + strm = list_first_entry(&comp->idle_strm, + struct zcomp_strm, + entry); + list_del(&strm->entry); + spin_unlock(&comp->strm_lock); + return strm; + } + + /* cannot allocate new stream, wait for an idle one */ + if (comp->avail_strm >= num_online_cpus()) { + spin_unlock(&comp->strm_lock); + wait_event(comp->strm_wait, + !list_empty(&comp->idle_strm)); + continue; + } + + /* allocate new stream */ + comp->avail_strm++; + spin_unlock(&comp->strm_lock); + + strm = zcomp_strm_alloc(comp); + if (strm) + break; + + spin_lock(&comp->strm_lock); + comp->avail_strm--; + spin_unlock(&comp->strm_lock); + wait_event(comp->strm_wait, !list_empty(&comp->idle_strm)); + } + + return strm; } -void zcomp_stream_put(struct zcomp *comp) +void zcomp_stream_put(struct zcomp *comp, struct zcomp_strm *strm) { - local_unlock(&comp->stream->lock); + spin_lock(&comp->strm_lock); + if (comp->avail_strm <= num_online_cpus()) { + list_add(&strm->entry, &comp->idle_strm); + spin_unlock(&comp->strm_lock); + wake_up(&comp->strm_wait); + return; + } + + comp->avail_strm--; + spin_unlock(&comp->strm_lock); + zcomp_strm_free(comp, strm); } int zcomp_compress(struct zcomp *comp, struct zcomp_strm *zstrm, @@ -148,61 +203,19 @@ int zcomp_decompress(struct zcomp *comp, struct zcomp_strm *zstrm, return comp->ops->decompress(comp->params, &zstrm->ctx, &req); } -int zcomp_cpu_up_prepare(unsigned int cpu, struct hlist_node *node) -{ - struct zcomp *comp = hlist_entry(node, struct zcomp, node); - struct zcomp_strm *zstrm; - int ret; - - zstrm = per_cpu_ptr(comp->stream, cpu); - local_lock_init(&zstrm->lock); - - ret = zcomp_strm_init(comp, zstrm); - if (ret) - pr_err("Can't allocate a compression stream\n"); - return ret; -} - -int zcomp_cpu_dead(unsigned int cpu, struct hlist_node *node) -{ - struct zcomp *comp = hlist_entry(node, struct zcomp, node); - struct zcomp_strm *zstrm; - - zstrm = per_cpu_ptr(comp->stream, cpu); - zcomp_strm_free(comp, zstrm); - return 0; -} - -static int zcomp_init(struct zcomp *comp, struct zcomp_params *params) -{ - int ret; - - comp->stream = alloc_percpu(struct zcomp_strm); - if (!comp->stream) - return -ENOMEM; - - comp->params = params; - ret = comp->ops->setup_params(comp->params); - if (ret) - goto cleanup; - - ret = cpuhp_state_add_instance(CPUHP_ZCOMP_PREPARE, &comp->node); - if (ret < 0) - goto cleanup; - - return 0; - -cleanup: - comp->ops->release_params(comp->params); - free_percpu(comp->stream); - return ret; -} - void zcomp_destroy(struct zcomp *comp) { - cpuhp_state_remove_instance(CPUHP_ZCOMP_PREPARE, &comp->node); + struct zcomp_strm *strm; + + while (!list_empty(&comp->idle_strm)) { + strm = list_first_entry(&comp->idle_strm, + struct zcomp_strm, + entry); + list_del(&strm->entry); + zcomp_strm_free(comp, strm); + } + comp->ops->release_params(comp->params); - free_percpu(comp->stream); kfree(comp); } @@ -229,7 +242,12 @@ struct zcomp *zcomp_create(const char *alg, struct zcomp_params *params) return ERR_PTR(-EINVAL); } - error = zcomp_init(comp, params); + INIT_LIST_HEAD(&comp->idle_strm); + init_waitqueue_head(&comp->strm_wait); + spin_lock_init(&comp->strm_lock); + + comp->params = params; + error = comp->ops->setup_params(comp->params); if (error) { kfree(comp); return ERR_PTR(error); diff --git a/drivers/block/zram/zcomp.h b/drivers/block/zram/zcomp.h index ad5762813842..62330829db3f 100644 --- a/drivers/block/zram/zcomp.h +++ b/drivers/block/zram/zcomp.h @@ -3,10 +3,10 @@ #ifndef _ZCOMP_H_ #define _ZCOMP_H_ -#include - #define ZCOMP_PARAM_NO_LEVEL INT_MIN +#include + /* * Immutable driver (backend) parameters. The driver may attach private * data to it (e.g. driver representation of the dictionary, etc.). @@ -31,7 +31,7 @@ struct zcomp_ctx { }; struct zcomp_strm { - local_lock_t lock; + struct list_head entry; /* compression buffer */ void *buffer; struct zcomp_ctx ctx; @@ -60,16 +60,15 @@ struct zcomp_ops { const char *name; }; -/* dynamic per-device compression frontend */ struct zcomp { - struct zcomp_strm __percpu *stream; + struct list_head idle_strm; + spinlock_t strm_lock; + u32 avail_strm; + wait_queue_head_t strm_wait; const struct zcomp_ops *ops; struct zcomp_params *params; - struct hlist_node node; }; -int zcomp_cpu_up_prepare(unsigned int cpu, struct hlist_node *node); -int zcomp_cpu_dead(unsigned int cpu, struct hlist_node *node); ssize_t zcomp_available_show(const char *comp, char *buf); bool zcomp_available_algorithm(const char *comp); @@ -77,7 +76,7 @@ struct zcomp *zcomp_create(const char *alg, struct zcomp_params *params); void zcomp_destroy(struct zcomp *comp); struct zcomp_strm *zcomp_stream_get(struct zcomp *comp); -void zcomp_stream_put(struct zcomp *comp); +void zcomp_stream_put(struct zcomp *comp, struct zcomp_strm *strm); int zcomp_compress(struct zcomp *comp, struct zcomp_strm *zstrm, const void *src, unsigned int *dst_len); diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 7eb7feba3cac..b217c29448ce 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -31,7 +31,6 @@ #include #include #include -#include #include #include @@ -1606,7 +1605,7 @@ static int read_compressed_page(struct zram *zram, struct page *page, u32 index) ret = zcomp_decompress(zram->comps[prio], zstrm, src, size, dst); kunmap_local(dst); zs_unmap_object(zram->mem_pool, handle); - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zram->comps[prio], zstrm); return ret; } @@ -1767,14 +1766,14 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) kunmap_local(mem); if (unlikely(ret)) { - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP], zstrm); pr_err("Compression failed! err=%d\n", ret); zs_free(zram->mem_pool, handle); return ret; } if (comp_len >= huge_class_size) { - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP], zstrm); return write_incompressible_page(zram, page, index); } @@ -1798,7 +1797,7 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) __GFP_HIGHMEM | __GFP_MOVABLE); if (IS_ERR_VALUE(handle)) { - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP], zstrm); atomic64_inc(&zram->stats.writestall); handle = zs_malloc(zram->mem_pool, comp_len, GFP_NOIO | __GFP_HIGHMEM | @@ -1810,7 +1809,7 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) } if (!zram_can_store_page(zram)) { - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP], zstrm); zs_free(zram->mem_pool, handle); return -ENOMEM; } @@ -1818,7 +1817,7 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) dst = zs_map_object(zram->mem_pool, handle, ZS_MM_WO); memcpy(dst, zstrm->buffer, comp_len); - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP], zstrm); zs_unmap_object(zram->mem_pool, handle); zram_slot_write_lock(zram, index); @@ -1977,7 +1976,7 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, kunmap_local(src); if (ret) { - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zram->comps[prio], zstrm); return ret; } @@ -1987,7 +1986,7 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, /* Continue until we make progress */ if (class_index_new >= class_index_old || (threshold && comp_len_new >= threshold)) { - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zram->comps[prio], zstrm); continue; } @@ -2045,13 +2044,13 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, __GFP_HIGHMEM | __GFP_MOVABLE); if (IS_ERR_VALUE(handle_new)) { - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zram->comps[prio], zstrm); return PTR_ERR((void *)handle_new); } dst = zs_map_object(zram->mem_pool, handle_new, ZS_MM_WO); memcpy(dst, zstrm->buffer, comp_len_new); - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zram->comps[prio], zstrm); zs_unmap_object(zram->mem_pool, handle_new); @@ -2799,7 +2798,6 @@ static void destroy_devices(void) zram_debugfs_destroy(); idr_destroy(&zram_index_idr); unregister_blkdev(zram_major, "zram"); - cpuhp_remove_multi_state(CPUHP_ZCOMP_PREPARE); } static int __init zram_init(void) @@ -2809,15 +2807,9 @@ static int __init zram_init(void) BUILD_BUG_ON(__NR_ZRAM_PAGEFLAGS > sizeof(zram_te.flags) * 8); - ret = cpuhp_setup_state_multi(CPUHP_ZCOMP_PREPARE, "block/zram:prepare", - zcomp_cpu_up_prepare, zcomp_cpu_dead); - if (ret < 0) - return ret; - ret = class_register(&zram_control_class); if (ret) { pr_err("Unable to register zram-control class\n"); - cpuhp_remove_multi_state(CPUHP_ZCOMP_PREPARE); return ret; } @@ -2826,7 +2818,6 @@ static int __init zram_init(void) if (zram_major <= 0) { pr_err("Unable to get major number\n"); class_unregister(&zram_control_class); - cpuhp_remove_multi_state(CPUHP_ZCOMP_PREPARE); return -EBUSY; } diff --git a/include/linux/cpuhotplug.h b/include/linux/cpuhotplug.h index 6cc5e484547c..092ace7db8ee 100644 --- a/include/linux/cpuhotplug.h +++ b/include/linux/cpuhotplug.h @@ -119,7 +119,6 @@ enum cpuhp_state { CPUHP_MM_ZS_PREPARE, CPUHP_MM_ZSWP_POOL_PREPARE, CPUHP_KVM_PPC_BOOK3S_PREPARE, - CPUHP_ZCOMP_PREPARE, CPUHP_TIMERS_PREPARE, CPUHP_TMIGR_PREPARE, CPUHP_MIPS_SOC_PREPARE, From patchwork Wed Jan 22 05:57:41 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13946860 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 34FE0C02181 for ; Wed, 22 Jan 2025 05:59:24 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BC5306B008C; Wed, 22 Jan 2025 00:59:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B73E56B0092; Wed, 22 Jan 2025 00:59:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A3D376B0093; Wed, 22 Jan 2025 00:59:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 82E1F6B008C for ; Wed, 22 Jan 2025 00:59:23 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 2D5851C8696 for ; Wed, 22 Jan 2025 05:59:23 +0000 (UTC) X-FDA: 83034035406.27.CFDF3CD Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) by imf24.hostedemail.com (Postfix) with ESMTP id 6223C180005 for ; Wed, 22 Jan 2025 05:59:21 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b="R/PWTpVc"; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf24.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.177 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1737525561; a=rsa-sha256; cv=none; b=7mUVo4H694+HHOxraRpxp/dRlN6ab4VC0F6Z94beBDlPgjJexQ0xh6AiQ9zqv2W1c7bSFA B5Hv1N1FUHIDPFerTtwbKMWhCfyKv346kxgCgK4vUDVxN58vV5XBhpivzlNE8pdMmYgV2l csuVueJYFmmLugpvqp5yjn9rDWx2AkU= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b="R/PWTpVc"; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf24.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.177 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1737525561; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SJ+ga7y6+gKF81TWcu7LgFAJ2gVF/zL45vImj5BpZyE=; b=PyFzudcQQdziKr6Z1EhCcRRRwVOKlUNjDTe2v4RtUDhDbKLSV8ObQrXZCUl3yfwKizqgSU DZBwYZLNoWZj7jTRhsCmR303rWL4f9LafOFAElR5aRXsnegZ9kTHpopoN6wddXwKAIFLfs 2uemzmlwVbM2VM55BsBOp17Xy649CkU= Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-2166360285dso124805185ad.1 for ; Tue, 21 Jan 2025 21:59:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1737525560; x=1738130360; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=SJ+ga7y6+gKF81TWcu7LgFAJ2gVF/zL45vImj5BpZyE=; b=R/PWTpVc1dHOgs2i8++xc8+10Ufma15+j38nKJoJWd8jxmWvYprz/CcnNXBA3V6iyh y54R9VqZYO2Vg5xrqDVAg0PFFmeoQuXM3BSqoGJQJ2d/zorj1LRqfwOB2yiyLD3fVuZ8 GbQEu4b3u8bmOUlzn30aO3fpWvqC5hol15vbE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737525560; x=1738130360; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SJ+ga7y6+gKF81TWcu7LgFAJ2gVF/zL45vImj5BpZyE=; b=t2+dxj5Yrx8oZs/mpmwhRjtiZDCC9YNiH1zdoAbZGkntiIHJHJCm2PgvngewHQZ2fo gddFHvu7mIH2HG4fGH353Bi5x4gbmasBQzd/CVt3AR8T289ceFcCVUgEislsxaqe0cpg mqORj0s1ljByvtD4G+oSCcmlFMwxkkjKSka1lU/Bj2xgH/0NI5k49Ijsu0xxQt/8AZ5a TFm80xMZaqqOuZi9MNNZBQYE0bz2XRW/80jhWXk/m4fbW35AZ8fi/dQXAMzoTSTSe6j9 6mTN8+cLDvC2JOLMsQ9gDXiwEFICEjuswpuZbHDmZIikeKzXpkxTAoDhd5mcamw7FukU jyug== X-Gm-Message-State: AOJu0YzUIChCxyYTaFSSiZ9ygry8zLIHdNjLhG7sZaFqtMyqCGSr3Aye pRR+kHBNNF7HcadQIMuFhaoJr/67/G9cAIV+m2HKA9kte7wy9gHsZ1TO+Bqi/A== X-Gm-Gg: ASbGnctngpJhcN13bawUP5g8zGnYaCSQ5/oMoFru5QRWO8UGl88ImAxnFzPWcRhuGjK iII75Vwt0SDeKr2/lj2iVhQuZ4AF4er+R+85cV7jLS6KUs9kUUiVSUK9teCzqMbdQF3E2X7a+w6 0yxoME7qvsekxppGkn9nYzO3CnxtprifYjSLa8pTIGo3DSzlfjj7cDWYKrozKR+akOw5P893w3D k2b/p8oDkYQWtoRCAofMqzK1hr72yxBomV1pzU3V3cJUi6WcUl9RoqQtWpv6mFLXlYbve4QDqWX 1tXWRWE= X-Google-Smtp-Source: AGHT+IFBMlFbJctVy/ypJUX6lRvd/RQFlHEGcsLvYlfwp9P4XLmEWnmR34KjHtXYACtXltbyrx4LEQ== X-Received: by 2002:a05:6a20:2455:b0:1d9:3957:8a14 with SMTP id adf61e73a8af0-1eb2146512cmr29760238637.1.1737525560105; Tue, 21 Jan 2025 21:59:20 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:2902:8f0f:12b3:c251]) by smtp.gmail.com with UTF8SMTPSA id 41be03b00d2f7-abe1a03aa6csm302287a12.2.2025.01.21.21.59.18 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 21 Jan 2025 21:59:19 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH 3/7] zram: remove two-staged handle allocation Date: Wed, 22 Jan 2025 14:57:41 +0900 Message-ID: <20250122055831.3341175-4-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.0.rc2.279.g1de40edade-goog In-Reply-To: <20250122055831.3341175-1-senozhatsky@chromium.org> References: <20250122055831.3341175-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 6223C180005 X-Rspamd-Server: rspam10 X-Stat-Signature: k5em9s54kiu7eraj14jjdgfeqwsprucy X-HE-Tag: 1737525561-78125 X-HE-Meta: U2FsdGVkX19v5mFQzjtC3Ayz4Zz2SimpIU+l2fs8dRfKYT41x0nzg+R2yiNoHxXnCEMpSIlq0ZWiqBpd2nEsm1Ss/AC4c8F3ITdWymv3wdZ7XU7ljxTj/Mjf0bstJdpuArk+b1EYqXmzz1AokFh8HIqYPEqeMSzVbmcr2FpBLukN6AGGhqlZkxkqf3U1uwp9UjTf1IzvpyPoN+c8WsREugNitCP4aqlRI6V1fAGcKr1qhsHJiW9cPtGSYcYPwRM7LzjxSKMwfzXqRo3C6SJtLA7cyYN878UJQG2jmH1X9wfQH5hP9rXmWFNX2hqT2n4lxkNH6ooQ5/7q5U450jvzeFaWRyh7AGZtPezlMugXZNypwdx34iqXCbNS7QYceUVsqG8gJHani2eOFFAMjxsMWO7lSr65TKSPOD9zk9F02LzCNJGynxYLpPd+YltzMGRpOqHhCo3ebqXh04o7io1buBCfzT8Cz6pHk7cxGDJ9Y87P6B18wdLVvEDltlYHHRcUwR6VPMYZgYcfv0OfLg0tnhCCVhl37oZPk5I12JvBsal4U6aZuOM05bOLFJ4aoRw6c9mq2MNFR8nAaNJdOczXTVjC8lf1H49LPGAB+O00LPYo8ZBZd/u/moPeYukscMcNa2JwSYQ5+A2DalYUbglYAQJAIZ8zXeX7W53x6UgMdJYn5bKipw49LjlgTCyPB9IdeVQ1N/4ZOEMrMOiZyETD6Q0kiPnU+uKbZ5NnML+4E12ldY8VKYstZtUGlPnMoqvtOEHjLpXWEe3dc7c5EDfMeGk7ns2/mtxIBtAU7GAWw59o25j43D+P7gYsoNSd7RyiNvZ7UzhWd43MFSV0wOL7VSRBxNl4RkxCuHxuad5Y6SX4BfA8nSJ4lusHlQQGLbNgBRF0zf8y9qP4tESNuqHbQLp2apd+jT9O4RwYOU54NOUknPtoe086qVZmNfNrW5qzkInmuDJo9XjpAblft2n zZ353ntP HKy9DcS1oKmbncn9a7m++2QzHexXkSnEtC6ffuje92zdLEeWmVwh0iwNQvRtQ2HRJxu+94CYENYnLFwkW2uOvRCr6lB50VXDhXip7zWA4C5CMjroiWDqO5xhSMbnPU2ICcgBJmhETh2oFYpy8I9Iu2n+ONf47rt6MGuMC5qUweUc/vm27+KtM2+S8/gxJLP+DKmYL8Q2540QAb3A1JDtJ+7W433BqBYwPMEYYjwv0AsIbKaegcttJ2RZPLR9NFAfKApLKV7PLo+yZ07eZy84CfcU1ylqaMgI+LrNQvp18+kWT2HNIHjhxFVE9fQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Previously zram write() was atomic which required us to pass __GFP_KSWAPD_RECLAIM to zsmalloc handle allocation on a fast path and attempt a slow path allocation (with recompression) when the fast path failed. Since it's not atomic anymore we can permit direct reclaim during allocation, and remove fast allocation path and, also, drop the recompression path (which should reduce CPU/battery usage). Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 39 ++++++----------------------------- 1 file changed, 6 insertions(+), 33 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index b217c29448ce..8029e0fe864a 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1740,11 +1740,11 @@ static int write_incompressible_page(struct zram *zram, struct page *page, static int zram_write_page(struct zram *zram, struct page *page, u32 index) { int ret = 0; - unsigned long handle = -ENOMEM; - unsigned int comp_len = 0; + unsigned long handle; + unsigned int comp_len; void *dst, *mem; struct zcomp_strm *zstrm; - unsigned long element = 0; + unsigned long element; bool same_filled; /* First, free memory allocated to this slot (if any) */ @@ -1758,7 +1758,6 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) if (same_filled) return write_same_filled_page(zram, element, index); -compress_again: zstrm = zcomp_stream_get(zram->comps[ZRAM_PRIMARY_COMP]); mem = kmap_local_page(page); ret = zcomp_compress(zram->comps[ZRAM_PRIMARY_COMP], zstrm, @@ -1777,36 +1776,10 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) return write_incompressible_page(zram, page, index); } - /* - * handle allocation has 2 paths: - * a) fast path is executed with preemption disabled (for - * per-cpu streams) and has __GFP_DIRECT_RECLAIM bit clear, - * since we can't sleep; - * b) slow path enables preemption and attempts to allocate - * the page with __GFP_DIRECT_RECLAIM bit set. we have to - * put per-cpu compression stream and, thus, to re-do - * the compression once handle is allocated. - * - * if we have a 'non-null' handle here then we are coming - * from the slow path and handle has already been allocated. - */ + handle = zs_malloc(zram->mem_pool, comp_len, + GFP_NOIO | __GFP_HIGHMEM | __GFP_MOVABLE); if (IS_ERR_VALUE(handle)) - handle = zs_malloc(zram->mem_pool, comp_len, - __GFP_KSWAPD_RECLAIM | - __GFP_NOWARN | - __GFP_HIGHMEM | - __GFP_MOVABLE); - if (IS_ERR_VALUE(handle)) { - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP], zstrm); - atomic64_inc(&zram->stats.writestall); - handle = zs_malloc(zram->mem_pool, comp_len, - GFP_NOIO | __GFP_HIGHMEM | - __GFP_MOVABLE); - if (IS_ERR_VALUE(handle)) - return PTR_ERR((void *)handle); - - goto compress_again; - } + return PTR_ERR((void *)handle); if (!zram_can_store_page(zram)) { zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP], zstrm); From patchwork Wed Jan 22 05:57:42 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13946861 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C9A51C02182 for ; Wed, 22 Jan 2025 05:59:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 583246B0093; Wed, 22 Jan 2025 00:59:28 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 531F36B0095; Wed, 22 Jan 2025 00:59:28 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3D3A56B0096; Wed, 22 Jan 2025 00:59:28 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 1F8796B0093 for ; Wed, 22 Jan 2025 00:59:28 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id CC8E7C12FA for ; Wed, 22 Jan 2025 05:59:27 +0000 (UTC) X-FDA: 83034035574.30.156ACF4 Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) by imf24.hostedemail.com (Postfix) with ESMTP id EEDC2180008 for ; Wed, 22 Jan 2025 05:59:25 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=Y1UBDom+; spf=pass (imf24.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.181 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1737525566; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MU55qMB62NlP1Hfg8Lf4TsS0i/+gjPtdvpdv0f7R4WY=; b=DQ9V7ygXJOVMwdZ0DOcPM76jISbFL3+PUQX2VpW+ti+PO4rdqK9fbUg29sIjtSCs5wzoSs /ik8wZcosmqW768vyhuE2aZbwzqbKm9wshX3BSRjYiZLXBjJLGvmn7DS/Irgw2+pUZ/vbN tsMl+3/Q8WDiS9xCIKVWF2gsXO8uXCI= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1737525566; a=rsa-sha256; cv=none; b=tBU8NAtj99XTzSITs6Ai+sl3ePgIMgqnckBsCvazr9kZX9FjM0gzvHL1H01RjWXM9FLIpn MRScyRdeyG4WOWLBiXTzMmqd/8/cT5wseklUgG6QCzScWiI5SpX5dmxBxB1Z6IrEy9VYIY YcSi0P4XKaldb+1XCqyapZGHfUDqnm4= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=Y1UBDom+; spf=pass (imf24.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.181 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-2163dc5155fso123848275ad.0 for ; Tue, 21 Jan 2025 21:59:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1737525565; x=1738130365; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MU55qMB62NlP1Hfg8Lf4TsS0i/+gjPtdvpdv0f7R4WY=; b=Y1UBDom+3SLSD7o0UVkGmdyn2X+wRsfRkk0pLbI9XzeC0NNPQPJFkYXNaLZiOMKf4t 2/nrNpXN9u1yEmK20srU2kKHZbO5S88WqMXtVZIi+zlbPzGoD9gLrshHBh0HW3Q5oPxl of0DS5Cz5wxq9gRr+pSNf353L0Klhn7nZ0MR4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737525565; x=1738130365; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MU55qMB62NlP1Hfg8Lf4TsS0i/+gjPtdvpdv0f7R4WY=; b=jaIcdU5GkusN8Ut1lKieEdMqmgD0lNP8MeNtvk3M4BTsQqbyZ0nx9LKyhGS87s0h0M A3jNiwiZ4Tkh2aYQbcsi8EJSOOK9L3w3XyFubSBOdKXyJOnQ3RGbpcMNs9TvuwfAeONr 4R1l7qjwFqaYF7zpZ+rq3TCLEJBmTzDipzsyzW6LV5eCcwGO7GUnzxlmSw4/d0jPH9L5 iIb9UAT3EtstfDf58feDanHVAxG9y5g1jIvFM4D+vETZbRls1oR/Y4vENWyPI5a9Puvi VG6UUD9hXYh4wnEtiVTwJY58bjaCW9gyxypJxYbkilo/R+NHq5vm9HJZ1tBwNJuUm7nA qvEA== X-Gm-Message-State: AOJu0YyQAVnlesA70UPhq8XjZSL2XiHKZdndx0sHRZbLcXvE5PBN7hB7 6pskU4bCwozDZ4NEBOHonetRrp9wjKSoK5sffotAuLe8evfQ14dn1Omi8T6MNQ== X-Gm-Gg: ASbGncts5uk82ymxlrYIMtXGdEHcDbJ2RvpTsW505mwupMcUbfPnk0h0ZQS+Cne1WC9 Hb7l6V00/LqtgummuJtUz/mHOHYClXK1uvdX/5flh2iHZbLc67dSLOpDFLzoy2krkTBLGiNN2eR C9MiXNjQfsAlqhAWqrwf+2FZk/PWQfRwBwDuMiKcxbgNo9RI2REZykztmsz/qcRO5OOsMdglleK 9Ej/iXqQm2MAUCIFYFh6+C9bXrFQSk3T3RP1VxTCGoWbZ+V6Luq1rN3h7nJURQuxjH/pUBF X-Google-Smtp-Source: AGHT+IHVvjN7ukzTgGkrLyv03/2Ys7IGx6CgvGp7QzNoO8SNF+YE8BGXBhQxnVlgYQWffGVAiCf6Ng== X-Received: by 2002:a05:6a20:3d8d:b0:1e0:f495:1bd9 with SMTP id adf61e73a8af0-1eb2144d37bmr33294873637.8.1737525564931; Tue, 21 Jan 2025 21:59:24 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:2902:8f0f:12b3:c251]) by smtp.gmail.com with UTF8SMTPSA id 41be03b00d2f7-a9bdf0b7579sm9763985a12.73.2025.01.21.21.59.23 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 21 Jan 2025 21:59:24 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH 4/7] zram: permit reclaim in zstd custom allocator Date: Wed, 22 Jan 2025 14:57:42 +0900 Message-ID: <20250122055831.3341175-5-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.0.rc2.279.g1de40edade-goog In-Reply-To: <20250122055831.3341175-1-senozhatsky@chromium.org> References: <20250122055831.3341175-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: EEDC2180008 X-Stat-Signature: 93inn13aow7y6kp8nyizohwsb7jr1rgk X-Rspam-User: X-Rspamd-Server: rspam12 X-HE-Tag: 1737525565-449558 X-HE-Meta: U2FsdGVkX1/V4Ib7urQikLaSqc5xRH5b9F414LooSRG7ed8uycvZkHLjc1rHVf2yZkh48rZjUSIjfUhMDYA0ZiTmzwge99+mHGwWmEcWmDmhnVUfdjM7Fi8zXPD8Ydc6dBot0QjigeG390FUNZv/Xw0vmKwcVWSQGZ7uS0u7bS9HkdeKDZ7R/FUE4yySc1kKnRAEomFNG+BLzbie2Ajz4M/Un2FKbPTZMnz+8a83qigOOucf7jyUaAABOLRWWGW2fCps/rBJ1jSH/Tl3M6WaPTECEIXW/3Y997Wh4+GatjlS42DHTi6DYLQDWEV2etQkyos8ZyTnJG5Ox3H6dYqkFqLImaE/zCSKquHJq8LEwMBr8TbFt8gZnz4sWxqeHAIgJBMkDhphIt0/SSrMvAnhigzfTr3aiwH0pw9Gx6Gvq7n+ilwPQz5IoYmMTmgNGT+e5Nw2XpUxl/2NEh3w6M8HJIWWCjZiS2fBo0wVDLCHB0VMwRqQuyNB+Df7k71C9nmMTAGIFvjL41WXLeV4WBtlcNmY+bbMLBCJK+eL/N4pnbapKQ9G/3xqcua0UDmjXHWl8Sdxfo/jtUNUhJzZ89Tkdlx5WxJGbkJgGyxFZhgXL3XfyXutXeCDKABfjeNz/JcxU3q1x7CDe4s+SvlNudN6O48Fp6EKs1yGKuZDcfS5NSOOPiNZsq+x/zaUvXzQU8/N+pIqyaTQ8DA0U2YiAqIRCiXu9ay2k1/zu/S815/veOdWwIQ4Q7ggqbPBu6ntgSLxIkzUJby8//kWaTsfXXS7j9VTECyD2B3eZaDJgXZT5R+gZ0ap22d3B6+LmICQTJiDouVII5A/CoBytj1xYWFy3k0mgV3kUOFDipeO7OjChRInnPQ9TgD3wY6ZNuPIq6WMVwDgLkuDRClz/k3R4IqsGRcOk2OQoA7puAqGixctFWwal2oo+fKmVVmbakhpt+eI7cSDtKO/w93g94lq0dy GxgLOSMl yi4czqz2lCnDRqzVegcoz8pDlwcmMTYF6lWCQuoFIRNAIXSJluRGViX3yl679jxGF+7Scra3T1nUyyDzvEKdy4ZMh4KaTRQdJsI709rpEOPAiXjFVfs+F+Zzbmf8mGnL3BSkBrSBGR7NzJQwZXwTVQWeYSeFMZbKqmcqFLQ6NtNyJnv0FsQK0lX/WHvzmQjUHP2C6IP5DcaU8X4nO/3Vikf1ptWD+LaNiQPMkQz6FMw+vHaGcjVr7kq7EsXFUPKbNYk8YVvcQH2DXUbHMZcNGJ8XlQ3t0SEb8/2oxm89wwC+U38Y0onIjt0YrZw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000093, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When configured with pre-trained compression/decompression dictionary support, zstd requires custom memory allocator, which it calls internally from compression()/decompression() routines. This was a tad problematic, because that would mean allocation from atomic context (either under entry spin-lock, or per-CPU local-lock or both). Now, with non-atomic zram write(), those limitations are relaxed and we can allow direct and indirect reclaim during allocations. The tricky part is zram read() path, which is still atomic in one particular case (read_compressed_page()), due to zsmalloc handling of object mapping. However, in zram in order to read() something one has to write() it first, and write() is when zstd allocates required internal state memory, and write() path is non-atomic. Because of this write() allocation, in theory, zstd should not call its allocator from the atomic read() path. Keep the non-preemptible branch, just in case if zstd allocates memory from read(), but WARN_ON_ONCE() if it happens. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/backend_zstd.c | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-) diff --git a/drivers/block/zram/backend_zstd.c b/drivers/block/zram/backend_zstd.c index 1184c0036f44..53431251ea62 100644 --- a/drivers/block/zram/backend_zstd.c +++ b/drivers/block/zram/backend_zstd.c @@ -24,19 +24,14 @@ struct zstd_params { /* * For C/D dictionaries we need to provide zstd with zstd_custom_mem, * which zstd uses internally to allocate/free memory when needed. - * - * This means that allocator.customAlloc() can be called from zcomp_compress() - * under local-lock (per-CPU compression stream), in which case we must use - * GFP_ATOMIC. - * - * Another complication here is that we can be configured as a swap device. */ static void *zstd_custom_alloc(void *opaque, size_t size) { - if (!preemptible()) + /* Technically this should not happen */ + if (WARN_ON_ONCE(!preemptible())) return kvzalloc(size, GFP_ATOMIC); - return kvzalloc(size, __GFP_KSWAPD_RECLAIM | __GFP_NOWARN); + return kvzalloc(size, GFP_NOIO | __GFP_NOWARN); } static void zstd_custom_free(void *opaque, void *address) From patchwork Wed Jan 22 05:57:43 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13946862 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C70CDC02182 for ; Wed, 22 Jan 2025 05:59:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5D0D8280002; Wed, 22 Jan 2025 00:59:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 580BD280001; Wed, 22 Jan 2025 00:59:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 421A0280002; Wed, 22 Jan 2025 00:59:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 23706280001 for ; Wed, 22 Jan 2025 00:59:33 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id D357FA134E for ; Wed, 22 Jan 2025 05:59:32 +0000 (UTC) X-FDA: 83034035784.22.C5ACFB2 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) by imf08.hostedemail.com (Postfix) with ESMTP id 002D816000E for ; Wed, 22 Jan 2025 05:59:30 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b="dQp/lPW8"; spf=pass (imf08.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.171 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1737525571; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=RlO1kRVQLX8+VpLXbuNkwAyTGyYAX3nxybEwBuf8gq8=; b=wBt278VhCSlrgG72hsfinxVaHqOnqluaFpTvpe35pGJvGOlA/ffPPPpyuFeEUN7nAXc0Ye GeAqCJ1eDL7OZ24DLxlQTIjsWvXSrO5YAFk+sNBVOUMxW02KWltJQtfWo+G9spPuDgmBYn MXrL4byyJxVGY4lJuMqhESIBWyBKyZs= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b="dQp/lPW8"; spf=pass (imf08.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.171 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1737525571; a=rsa-sha256; cv=none; b=nwl7W9eYurphaYzhUoPa0gdnGdW6WvNkHfoD2G4huWjfoPrmBQEBINZJKtMpEJ1CyY3UrW wRjOh0cKBsgOYx6WHMAQmwnIx5K8FxG7K/OPfO0q+r3G8t8H10SHBSQFZ6SFaooCikPsFD 2I6k/19Bz/l40Z8mEzYxlNJ0rIZmIo0= Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-21680814d42so106110145ad.2 for ; Tue, 21 Jan 2025 21:59:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1737525570; x=1738130370; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=RlO1kRVQLX8+VpLXbuNkwAyTGyYAX3nxybEwBuf8gq8=; b=dQp/lPW8RpMDKYkS97INi7y1SBMM0M4NVCgzguw/0S2gh9KuJtGaX9lMphLMShxmK1 Qerjw0qS30ST5Df8nRpG0HYYHZcMjmVlmxy0XE7ZMMjO9Cj+zak2U/pLd+PMMNOyWmYe pc7vZvyCaIlM9duLStATezKHTJRwmedQGBHAM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737525570; x=1738130370; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RlO1kRVQLX8+VpLXbuNkwAyTGyYAX3nxybEwBuf8gq8=; b=mdLoAYFs43dpVGTnUBZ+ig/GQ5B35YfDckb6BGN72/tJ9iKjSLhvoYg8Il4XwCvdIO FoFwgkOJ/OflKDaIEKnB0YAjNYcpPfkjIY0smIJYwpsrWTemXmLpqrUGHSyusMlX5unm Mp3yUDM4jUTNbmckBc7AddZOdi3jjRpbL1G4TEkAde4ivmgPGdIcXvEh8eruTDLkLBgt NxSDWJxCiN10ZLqfF3+kPO9Azl1JXpDlQM/3uIqMdelYRtH9yZsVOJNK5JydxzuFP9RC 8MPL7gz0Fa8feK7kmaB0XFIbPoLcdCMibqUPG5ghGa7QWhKflSPOQ2T3beZN38tWq4CV sY+Q== X-Gm-Message-State: AOJu0Yz6SGsJEU9Gokrzhu0bV3k7i1ut0pr6M07NlPFyJjIQp5P0ENYN zti/KcOFBwCkYXpgLAplo+0mT0F7z/wXDT5GpZwrJsxjVRByEl+w4ywu4xNnOQ== X-Gm-Gg: ASbGncu/nzum8M4pygLk7kWwFoSLWfXkWOOxqpMoFLKCuMZFJ6x0TqY2beVRRk5Gf9F RH80os+FXWAVzpFsVYXNG9bZ8TACoQVevjzMe6QLSOA5kJSmDz/wcD+OSqbjbdE6ftz5ayr+pxN d2TJidgWJCyfwPOqzE2bQNOflnKJt9s4bFR2X0mK3UR4Bup5wy4R9Y8r8aWxdRtWnYSeGeva+ft oRDVd181sWWZHM2urtMqYCFwxv5tHWjR3G7osSF3zgyP+hIvwSW+IeGrj5g4DufDAIJdKIP X-Google-Smtp-Source: AGHT+IFDdKEw/ZLrD/5ATxrQr6EltWEOuPwzfWGcOrPfuy+nWB9ln8JMINNEvYbVGHeZ24N0e5vrbA== X-Received: by 2002:a17:902:f644:b0:211:8404:a957 with SMTP id d9443c01a7336-21c355f6aa9mr356031145ad.41.1737525569847; Tue, 21 Jan 2025 21:59:29 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:2902:8f0f:12b3:c251]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-21d6217cd6fsm20187155ad.189.2025.01.21.21.59.28 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 21 Jan 2025 21:59:29 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH 5/7] zram: permit reclaim in recompression handle allocation Date: Wed, 22 Jan 2025 14:57:43 +0900 Message-ID: <20250122055831.3341175-6-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.0.rc2.279.g1de40edade-goog In-Reply-To: <20250122055831.3341175-1-senozhatsky@chromium.org> References: <20250122055831.3341175-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 002D816000E X-Stat-Signature: w6mbodzwfda795y41eqwh3u91jqpkz73 X-HE-Tag: 1737525570-713062 X-HE-Meta: U2FsdGVkX19U6wyjIEtpVrpmPWHKVvDpcnHoPrToHqbT8aD+Mqcd2OMZb3dCR96Ii/E9vvl/Go9E7QhQYkrWut/z6rzNfzgcWxGu2cqgFqyaWLw3UIdWwwX5EFY5EgoFC8lqOti9kbcYZg8UkEAqo26LuPcz+vLnfhHs1WRAQWwy6ZgZO4kfRSrcP6LgUedZNsxIdWi1+f7o65d5A04ufyKg1c8TjtPkouNZRDmdMxEKh0NDJgQA8flA49vWK363RVO7cMqp9fSFS9yzoA05r2ze7B+JP4GFAUZTr4KqGTG4Ybh7cQ++iZiJfdwQiLhKfTQa/zq6QwW0t2v7BGsGm29fR5vY0Oa3vSzSAThZ7FsByMkDYfmhTQ4B903scMEDB9sYoBhFDoUOcGdWcDb3vwHDp5Jqk3ExpAgzddvbf8g/ayguvZVoXkCbrXJKzU2OgnDdlC1s7S5cG7AO8KuU1RMgjhs+wyrSyL0Yn32hQwglKDJCuJSIhQO3jPG4i9DUQIpD6Y/684ctVJkO5BoZZw6tQkwoMF/e6LMnGDot+7u7a8NRTUq9AelgFU3GrOZuVOxcrK06tlQx2fBB4udbqZ0djOmViKv0y8DgMHQDto8HEzuFvFFStB3BOmuMXSz04Adsx1rHNktGe5A+bRDcVDoeRT8ate0fXwWVcDRtAVkoIJsAgb1nj+Ltj4mB4RxHLozgwMapxgrgewoeakjaZgW1uV3C/GLy3Wr+MKj3CMZqeNOckpu+ayGxJEeOXs+OSJjcrAWt2S95N290GVdXcr/+UckZFvskNVwtcnG9MShE5tJ7dl7hDbTX07n8fJMAmcSOUtqvs18cgIFOPKq8k3RUtR7JQu2MbRI0hjfVtcUJn2SO7jQgj+Q04v+LZd+OP/4ycFW+gt9z06AvcxvBbEvnmw3g3Hhry6Qcoap7M0ORIaKvkszTRJFqqMBb51G63OHb8OuNvAbM2kaTKEC fT6toITw yuBvvkhuo/zG099XvppKheKJ2LdcSO5hXOQBa8KI0a7sEggyCDLdlHTuy6yNAGSVAnXz7CtOfcj5mom+NxarQIY3hq5AkKR5Vac0AjFzDzbWgIw5RV03xnf0RM8/qCg0eWUYlysfPKvQfGIluxoKGymvL4ho9NkehMhc68nuA+wBLzTcNdP2VQAqlNjTInz5A8R9dTmvZoIMNlTXKPuangl8B3OLK5d7N3/N32p+YurGz0qSeg6sJWjsetA9hoVGqP4FHwugL11ut2WJLMZj5rOycuiKeAkLifsCwAa7dqTs+h2R8SU7BChJiGw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000056, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Recompression path can now permit direct reclaim during new zs_handle allocation, because it's not atomic anymore. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 12 +++--------- 1 file changed, 3 insertions(+), 9 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 8029e0fe864a..faccf9923391 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -2005,17 +2005,11 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, return 0; /* - * No direct reclaim (slow path) for handle allocation and no - * re-compression attempt (unlike in zram_write_bvec()) since - * we already have stored that object in zsmalloc. If we cannot - * alloc memory for recompressed object then we bail out and - * simply keep the old (existing) object in zsmalloc. + * If we cannot alloc memory for recompressed object then we bail out + * and simply keep the old (existing) object in zsmalloc. */ handle_new = zs_malloc(zram->mem_pool, comp_len_new, - __GFP_KSWAPD_RECLAIM | - __GFP_NOWARN | - __GFP_HIGHMEM | - __GFP_MOVABLE); + GFP_NOIO | __GFP_HIGHMEM | __GFP_MOVABLE); if (IS_ERR_VALUE(handle_new)) { zcomp_stream_put(zram->comps[prio], zstrm); return PTR_ERR((void *)handle_new); From patchwork Wed Jan 22 05:57:44 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13946863 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 16060C02181 for ; Wed, 22 Jan 2025 05:59:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A1638280003; Wed, 22 Jan 2025 00:59:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9C6B3280001; Wed, 22 Jan 2025 00:59:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8684C280003; Wed, 22 Jan 2025 00:59:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 67715280001 for ; Wed, 22 Jan 2025 00:59:37 -0500 (EST) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 3342B473D2 for ; Wed, 22 Jan 2025 05:59:37 +0000 (UTC) X-FDA: 83034035994.01.5C153CA Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) by imf13.hostedemail.com (Postfix) with ESMTP id 5553220009 for ; Wed, 22 Jan 2025 05:59:35 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=liE1IAIj; spf=pass (imf13.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.171 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1737525575; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BvDi8LZx+/IWsVw6XTpR4rzD/goHjjTNwP1dgLi65OA=; b=AryIHSDnYHQxDLCri8Bp/PzkzfqiMC+ZQZuHzzNqyFqQkAL4E8oHyW9qgWpDu1IxGGcQhF 54Aad3h+ALDY5kpMT7PJ/Ex5QTjaE/rZ+McwxDwKZoxm0qksotRmj3qPE4PM+dPMz9DP1X rKJphHWLfi8QiSGtilvv7C9uTxWvQTA= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=liE1IAIj; spf=pass (imf13.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.171 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1737525575; a=rsa-sha256; cv=none; b=VCwj4Rqh8VzmSHTpUxnqYdF4CDfm8DmT1/tiXzXnaXGxmHQMWn1PJ08ucR4JYXli1ySDzQ WFPl0MNNMjlM1lG7pRtnxQHHZmX1vwVKXGOtHclJihDPw5Upe1driVp3B2jTqCW9nUqKBQ 8LuSKIYg667nE145Xd5JW1YrUobJimk= Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-21bc1512a63so118799135ad.1 for ; Tue, 21 Jan 2025 21:59:35 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1737525574; x=1738130374; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BvDi8LZx+/IWsVw6XTpR4rzD/goHjjTNwP1dgLi65OA=; b=liE1IAIje78Xs3+dj4axrRTMNC8eG56c6mZLT7mhPduhPLDErvb7HwjmfIl9UyM5T4 9OuXX5Efq4o3I4AKeXl5tyK8m3pwjgcUfgZwzL/NGYm1Pn+1tK5G8aRsjPuRkigprx8j JoH8cm9T1iXSQ7Tqqslga+2Og3ml5uiR2L8sA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737525574; x=1738130374; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BvDi8LZx+/IWsVw6XTpR4rzD/goHjjTNwP1dgLi65OA=; b=Mjg9GhfuMFkDqGdd9h9VgKQnq6K1SdWOqLPtFN9WW1EncOLIpI1Pcvma8lSRSoYknU ebLED4NTGnwxf6dR28XzRu28m9saADbXZ++4V/ieEFaxEEolC6O3AQtdsh0MQoC62d8N zzuNFywlNoRMS4T4lcaSPeotvQSAWzlDBxpDt3cEqIYIBnphsyJIQ6LxiK22dQZ5Aewo YaUG+/SKnVTa/YXu9+YZYr0xK9BV2J2SsmmINy413q4mz4ZvbQSJOkcI4lYon/gw4ovD 2ZNKeWW6YK1GCumnlvizfCHwoYULxC68kJsfyGSop6J4NUeP7vUkjVOjQwbGa8oYkJQm bqmQ== X-Gm-Message-State: AOJu0Yyr7/k6wFPO0O0BUm6AffLvfbm4KwilpRRrLPP/PCfYz9epTz1f eyHDkIq+VjJaQQECJhin0s/NBrrbd+ow7YxfHWbZP9BXiYri5J/ZetDE/FboUp/XJzjgaDvlW7w = X-Gm-Gg: ASbGncsMDoyd8hZWK5bF/Nd6L4fGbsJRll8LnamFABnttND9et+ZLi7+9s9v40TUKa7 MYDZu9E7RT72Jpsay5zWBu9p8YD8nmU5U29Zq1dqyuSSaaKyCv2Jxp5i0o6irEWbUHt62nPLscd ULZqQVgEpEgJsijLnQwQcHqgI/Q/6lhg4x5LnHoT0Ed6twJOlbBgoJbVdSm+1FyV4qCzkR3FYhk iDEzQuFXOqxkemKPC8O9EUXtK8Dn758vNPG8Alp/pmdq/F+FQE/EQIkivoPtucMnpAyGTdQ X-Google-Smtp-Source: AGHT+IF5DI4LMRdnglCGj0EuJR+qLfX7sRa9uFUVFiVN11N888cUyl0GHxvq1Dybvd7Sp+FVeBy9+g== X-Received: by 2002:a17:902:ea05:b0:216:3d72:1712 with SMTP id d9443c01a7336-21c3579359cmr340704225ad.48.1737525574148; Tue, 21 Jan 2025 21:59:34 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:2902:8f0f:12b3:c251]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-21c2d4027fcsm87386355ad.214.2025.01.21.21.59.32 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 21 Jan 2025 21:59:33 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH 6/7] zram: remove writestall zram_stats member Date: Wed, 22 Jan 2025 14:57:44 +0900 Message-ID: <20250122055831.3341175-7-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.0.rc2.279.g1de40edade-goog In-Reply-To: <20250122055831.3341175-1-senozhatsky@chromium.org> References: <20250122055831.3341175-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 5553220009 X-Stat-Signature: he54gd7gw5pfoort4rd5gadyopwzqrss X-Rspam-User: X-HE-Tag: 1737525575-230365 X-HE-Meta: U2FsdGVkX1/DihzhVbRT4bfjkagPV9FULp12mtIqVDKwR9nLta4eEtBpcp2NxNWL5AIcC8SLQRKiwyHEtKvKKTlkj+oJuz6INIFH0lM1t37d+RIq7zXhT2Tc1Ju1iQ/d8pLsVOu5SdOM64lXyyqML+GKmajDWLAFGub/4UW6pcfpU9B02soNa58Vhij5707Wv4S2ckzFKiqe8Q0sfDluFlFU2cXwtI5qUtTeDZ/e6C5VV3mox4LwlYkuEYpSRVERmzX7+aw+Vgi4WkjNlXdzjyltzpGJ4fb5POrgMzfW7sc0JHXTKo7Gd4qXNqpu1lZbPXJ36ZWRnkT1h02L7hBM3R07TRmYyejzrfW3cSH0v+/Mhbo/jLClo/3nS288yxd516zFRO6qxTXnOTMfKNNbrMw5ie5QOOtXPeYZc/CGKjOP2hqYMYHCPkiloVhx78eIuxAVylg+uYtyiTg+2vB6TIZ/SirDLxAeH+skn4RQIy3f7Ls/4sg9JgeVfJnNDgoP0hkY6lc/mP5u8Z58yR1RsUmH4w+sIoJOLPOqx3jpQW4eceX/iR6KG4qJvZYSnq6U4IB4s3Rb0f8Sif4Kx/WaZ2BmymkGECwkOtqzHMXCLjbEuMi2dBuC5Jm11tZcZiEJdFf1VyzUHNuiSyb+g1X9E8DTLxVdqb4atm+02svebYKoNaXMvOCW+PuKqrVR0rcFEjEPde0Tuv4gr8Mqn0vO4mkoqmbsv4rjfPGF7hMKV6nm8D6HZLGqQlPaTU5I2UPZI9EErT1Kqx3l5mCeEa5FvAM2M1ip8ZdCagflrt7KInkCJpir6UAQ514cdkqvCxaYdkKR9ovT52jg/gM0C4QD+WdliHaAekOJ5M3JXQrHsRz8CLST4K6nLfKbvMz7SizdNYiaGpuinUbZRftTVcP3S4akdp9IXESVGvfUyQuWfwBqmS95w8YJL7Y8K8SSGr9b+HHafEpflPE8r5/Jiy7 C+2IBYQp s1Ll2yfo/9/Bss/nVHPCTEsxxUWXLcSDRaNzV6w4vNwkSOCjLRQGTFOYvWmXrYAmnPf5QDgFmYoKq66GQQBKHoCAdS/8DNdEA+1wFgt3zxSVZ+oEoKggqBLegwHZbkXuU1cESgaib3bPSh5ZX8mK+kDKmmEXpyq+qdgBXaNAwPjWDHuOwZKi284sce4xVJjrARa65iS1cuSBrn/RP8b5oUU9pbV5ZonGyBsK/jfiKuYQZ3+ePNf3CY6PlVxZAIpuvgUKyKHvfhuD3x1EnWuL+2l/WXy+hPk8c1omUmOMIEBNeMOjH1tH+f3/nAG8CnuVwQDeghn2SzO740pkt9ETuCgHsldXpNzMcaARZ1Lt/4sla6IU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000001, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: There is no zsmalloc handle allocation slow path now and writestall is not possible any longer. Remove it from zram_stats. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 3 +-- drivers/block/zram/zram_drv.h | 1 - 2 files changed, 1 insertion(+), 3 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index faccf9923391..d516f968321e 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1443,9 +1443,8 @@ static ssize_t debug_stat_show(struct device *dev, down_read(&zram->init_lock); ret = scnprintf(buf, PAGE_SIZE, - "version: %d\n%8llu %8llu\n", + "version: %d\n0 %8llu\n", version, - (u64)atomic64_read(&zram->stats.writestall), (u64)atomic64_read(&zram->stats.miss_free)); up_read(&zram->init_lock); diff --git a/drivers/block/zram/zram_drv.h b/drivers/block/zram/zram_drv.h index b272ede404b0..4f707dabed12 100644 --- a/drivers/block/zram/zram_drv.h +++ b/drivers/block/zram/zram_drv.h @@ -85,7 +85,6 @@ struct zram_stats { atomic64_t huge_pages_since; /* no. of huge pages since zram set up */ atomic64_t pages_stored; /* no. of pages currently stored */ atomic_long_t max_used_pages; /* no. of maximum pages stored */ - atomic64_t writestall; /* no. of write slow paths */ atomic64_t miss_free; /* no. of missed free */ #ifdef CONFIG_ZRAM_WRITEBACK atomic64_t bd_count; /* no. of pages in backing device */ From patchwork Wed Jan 22 05:57:45 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13946864 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 80CCFC02182 for ; Wed, 22 Jan 2025 05:59:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 13D6B280004; Wed, 22 Jan 2025 00:59:42 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0C5D4280001; Wed, 22 Jan 2025 00:59:42 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EA9C0280004; Wed, 22 Jan 2025 00:59:41 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id CB535280001 for ; Wed, 22 Jan 2025 00:59:41 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 753A314138E for ; Wed, 22 Jan 2025 05:59:41 +0000 (UTC) X-FDA: 83034036162.30.8AC5C35 Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) by imf07.hostedemail.com (Postfix) with ESMTP id 9DFCA40007 for ; Wed, 22 Jan 2025 05:59:39 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=SziaP01p; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf07.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.177 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1737525579; a=rsa-sha256; cv=none; b=MpMisBKbbHzXmawlWqtqaJMlmhFlnYodAFJ5Pdp2bqIseIUgZyPUwM7FiIGGArYWOPyTKH erZciicOc31CDPBpIQSMpUXeaWebApvY5aFjvGHVqfrx6Cneq0YFgBa1ebVkWH9007j53y WPXdLdZkFZ4MA8a+xDkXEkj6Koz0Zhw= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=SziaP01p; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf07.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.177 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1737525579; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=URQGVN2RLRBAm/FHMFqf5kTEz8apzib8vA7ECF9oUaE=; b=l77rvw3N1Mf/qF9Iu0LerdG3ycBNB9v5Q/p7XyM6l9UEcai15SGHKpUL/OFr6tCNcP2nXL rAUw8cLfjBxc+tfLhDHWLOlpG2pOmMaBJrLSTfmow61FFaAw2ImkQlQPC2k5wBOaRyodQu t/y8p5JOWUnpYDl8KXCpxFC4OPUDIDU= Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-2166022c5caso102999515ad.2 for ; Tue, 21 Jan 2025 21:59:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1737525578; x=1738130378; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=URQGVN2RLRBAm/FHMFqf5kTEz8apzib8vA7ECF9oUaE=; b=SziaP01pLwV1RKpB+Nw6e+8ZxpuBUeyFaDXwk3TKIta5lRG4119v77wiewapiFfgf3 ru7jOz+Mc+nSgdquqOyoTNWDyg5CVboY8MhKqXgGwxYyWx1KPf170+/Y8PcCGhJz769q 5GalG2DcLxBZgjP/WAA6G5GkzgLzDGIic+5pE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737525578; x=1738130378; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=URQGVN2RLRBAm/FHMFqf5kTEz8apzib8vA7ECF9oUaE=; b=vybKPEdhHG4QuZYzLLsbdh3MurisoE0dpx/2DeLfDqPWPNl2wjMExqiE7o20KgvFvb QxhA7tG7pe9zTl/ZSqo8NVlVkYrD3BiGvvN+G2sk941zfYVMUrqHO6hZEI3g5FLUQAod J1Zzv9FAyhn5amHOjP1MeUYnvIB+wUjrk20Zrodmmf5ZetB7VcFLJAEGs2ic1ecoAe8I bHq2scHzqJbRDjUskbaIvkqyo3vJxmKvjMQmDs4N11zFzwoTZ0v39+LRVmCETjzZTWNu MIYC3uABXi8V8n0AqzL3fh2J+pIjQpN82z0fA/9m05Rw+e652WRqQXMZjd1dAJbUoYez vIQg== X-Gm-Message-State: AOJu0YxvvQuDsqSQc+bj1ACq+n4ibYzqg49lF+AbFjv8zE+tg7BJ2Spg OvnoGsU6EM4NU5Dj+qHHjHYq7YkDVZDv8ofQwuvf7f3WCRUQwfo2FO6JOCxvVdoquqthD4WrmRc = X-Gm-Gg: ASbGncvSRtBqwhqEjyNg6/fkQ9/8bCrb1cNxvLzWYA2TMbgyJ9DoM0KwMlmXdzeYgDJ /marALRfi/KdChAN1uVlwdLIvNxu7ZqmGDOR3i5NvZGHbbQPIQ1hsE5Hdw5s8gH1apq8HVzCffg y6FZnzM7mWmzi4gLNjxsjN9zRT53O2NMp7lZMeD+TFq8PxCmxE4UNXW384JhiKOp8gUX9PxeRIT C0tNjAK6WRoZxJvVyDnen3DdeeY8GaDsbsVIS2Dq85CqFFfglWcXUjPp+9QZ//IgAaGKyCP X-Google-Smtp-Source: AGHT+IHfDORXl8892ZPFv7Yg9s3aa+pqcjz8wYjoVNgZaPfPzukdMK7LQbRYaeoP/wmNeG24p8MYWA== X-Received: by 2002:a17:902:d2c5:b0:216:779a:d5f3 with SMTP id d9443c01a7336-21c353edbd7mr334668355ad.14.1737525578444; Tue, 21 Jan 2025 21:59:38 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:2902:8f0f:12b3:c251]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-21c2d3ac3e2sm86558565ad.139.2025.01.21.21.59.36 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 21 Jan 2025 21:59:38 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH 7/7] zram: unlock slot bucket during recompression Date: Wed, 22 Jan 2025 14:57:45 +0900 Message-ID: <20250122055831.3341175-8-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.0.rc2.279.g1de40edade-goog In-Reply-To: <20250122055831.3341175-1-senozhatsky@chromium.org> References: <20250122055831.3341175-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 9DFCA40007 X-Rspamd-Server: rspam10 X-Stat-Signature: wmx6td1qc4qa9uabe7sdirmxwcjimud8 X-HE-Tag: 1737525579-953084 X-HE-Meta: U2FsdGVkX19jewCQND0fjKZDBTclRnVjxKrHj1OwG6A6J7GZ11ohdw10MLsUpk3dUfCqCP790jwn+iidWQp9C8DYMur1u1hMZFrqWnifao693zJ+YWoyuVOK0WCwzuzSZUML35jdCwxmhQ1mDtZCOUt1ljYNGlC4EMg30ald1mdVWyNtR/A9c9x3awjnoJl3Ss1YPwc99n1IgB8r/DL9GbYWtWFluhCmC4DHKawtUrgzGQA42GGpj+dRqH3LzzpypBX1qRDMNSuQLSL37I+PudVhRKQCsSXiT+/PjGs8mAJhgHsj31p5wSLkiijOuF6SIO4idrfTsf9bj8E3dcrmRGtlzuQcp7PKqjPzLdvAZ7aKvHdBdu8s1A2Tuk0AghhmtCTQejeiOADN1W1uoRTX0Zrl+8iYOxZUaikWmKxqvV/q3QXFg3yALxygsfG7qtVvTUbfNiYhiCyI7HaGEFH/cVIA8uqZIEhvsWpccPt93ZJryrGljLvYdfSKRJ02V4Y0SXOfeNwbL5PSz6xp0YOmMM1LyooE1rNlgiY+0BnXpkA07doyfr9c5HwmylHWrPxDPRBvUdqBldDlcdz8/XLgKTdonGMmirr8yTCMivuzJYugbbRN+SZjcgT9bbcFHkug82nFvCCbmYHKmRRYyXldwOqhX5n5EbR+6sf9LN9TKkwte61yppC/sWNhgx1/AWfxsAhbSQiZCmCpmqhEoojd3EGobKrarkYFpw/V2+78n6Q4GVMCa8Z45nMxcI4EidrLbkOBJWcsdH1cVsXjbYz9mkx4bHydS8c2MaLPL3jzLK/sJjHNR1hOBzr5PyKUJvBpYhsSEQ/z8+16IaDkRawuWfJP3N5QvLQQYuRi+S9kYprJ9CceicDcgU2f3/BdKdZQTUCYOiVu9b6Tba5F5SuCuyNkLIVea5di51P6oa8kIMcZ1MgWZnW5RmHpOHrnx3jn5sslTMDiiV9fk3qLbzF 8ZMmofJg 3T62NyeMzTu9PMloAdr2EaXc2uIqqcsTTv5mY62uLZ82/zVyHgqFJfUeMS/7vINlFEl7NntjiBagjRd3npISJ6u1Coye8opBJtvMFv6mEM4AHIkK0BKWCjacjxi+yTP0kOGjoF82V9gzPsDwN3til3tck9O09JSyg7RIUC4EY/bZRbcUWRyssYR1T0WB62KDRR63lnMc3PFMXRqkCQ2M/MSc19ynXReGiPBMV32QDPOSMXgrtNcd4mof6y6eSVZMrOmIeZ9vJ4Jv98VFcUZg8TndFQEtyZ5loaO3oCZx4NTrlo/QrafXeB1PApXAKMUHJeigE1uvHNUK/SM0Il0Ae07glmQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: As of now recompress_slot() is called under slot bucket write-lock, which is suboptimal as it blocks access to a huge number of entries. The good news is that recompression, like writeback, makes a local copy of slot data (we need to decompress it anyway) before post-processing so we can unlock slot bucket once we have that local copy. Unlock the bucket write-lock before recompression loop (secondary algorithms can be tried out one by one, in order of priority) and re-acquire it right after the loop. There is one more potentially costly operation recompress_slot() does - new zs_handle allocation, which can schedule(). Release the bucket write-lock before zsmalloc allocation and grab it again after the allocation. In both cases, once the bucket lock is re-acquired we examine slot's ZRAM_PP_SLOT flag to make sure that the slot has not been modified by a concurrent operation. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 53 +++++++++++++++++++++++++---------- 1 file changed, 38 insertions(+), 15 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index d516f968321e..0413438e4500 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1925,6 +1925,14 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, zram_clear_flag(zram, index, ZRAM_IDLE); class_index_old = zs_lookup_class_index(zram->mem_pool, comp_len_old); + + /* + * Set prio to one past current slot's compression prio, so that + * we automatically skip lower priority algorithms. + */ + prio = zram_get_priority(zram, index) + 1; + /* Slot data copied out - unlock its bucket */ + zram_slot_write_unlock(zram, index); /* * Iterate the secondary comp algorithms list (in order of priority) * and try to recompress the page. @@ -1933,13 +1941,6 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, if (!zram->comps[prio]) continue; - /* - * Skip if the object is already re-compressed with a higher - * priority algorithm (or same algorithm). - */ - if (prio <= zram_get_priority(zram, index)) - continue; - num_recomps++; zstrm = zcomp_stream_get(zram->comps[prio]); src = kmap_local_page(page); @@ -1947,10 +1948,8 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, src, &comp_len_new); kunmap_local(src); - if (ret) { - zcomp_stream_put(zram->comps[prio], zstrm); - return ret; - } + if (ret) + break; class_index_new = zs_lookup_class_index(zram->mem_pool, comp_len_new); @@ -1966,6 +1965,19 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, break; } + zram_slot_write_lock(zram, index); + /* Compression error */ + if (ret) { + zcomp_stream_put(zram->comps[prio], zstrm); + return ret; + } + + /* Slot has been modified concurrently */ + if (!zram_test_flag(zram, index, ZRAM_PP_SLOT)) { + zcomp_stream_put(zram->comps[prio], zstrm); + return 0; + } + /* * We did not try to recompress, e.g. when we have only one * secondary algorithm and the page is already recompressed @@ -2003,17 +2015,28 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, if (threshold && comp_len_new >= threshold) return 0; - /* - * If we cannot alloc memory for recompressed object then we bail out - * and simply keep the old (existing) object in zsmalloc. - */ + /* zsmalloc handle allocation can schedule, unlock slot's bucket */ + zram_slot_write_unlock(zram, index); handle_new = zs_malloc(zram->mem_pool, comp_len_new, GFP_NOIO | __GFP_HIGHMEM | __GFP_MOVABLE); + zram_slot_write_lock(zram, index); + + /* + * If we couldn't allocate memory for recompressed object then bail + * out and simply keep the old (existing) object in mempool. + */ if (IS_ERR_VALUE(handle_new)) { zcomp_stream_put(zram->comps[prio], zstrm); return PTR_ERR((void *)handle_new); } + /* Slot has been modified concurrently */ + if (!zram_test_flag(zram, index, ZRAM_PP_SLOT)) { + zcomp_stream_put(zram->comps[prio], zstrm); + zs_free(zram->mem_pool, handle_new); + return 0; + } + dst = zs_map_object(zram->mem_pool, handle_new, ZS_MM_WO); memcpy(dst, zstrm->buffer, comp_len_new); zcomp_stream_put(zram->comps[prio], zstrm);