From patchwork Fri Jan 31 09:06:00 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13955127 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 384CDC0218F for ; Fri, 31 Jan 2025 09:07:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BC1422800E4; Fri, 31 Jan 2025 04:07:12 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B6EFB2800E2; Fri, 31 Jan 2025 04:07:12 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9C2302800E4; Fri, 31 Jan 2025 04:07:12 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 73B2F2800E2 for ; Fri, 31 Jan 2025 04:07:12 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 05AC616092A for ; Fri, 31 Jan 2025 09:07:12 +0000 (UTC) X-FDA: 83067167904.29.687D8B8 Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) by imf17.hostedemail.com (Postfix) with ESMTP id 06BBB40006 for ; Fri, 31 Jan 2025 09:07:09 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=C+G2IsoQ; spf=pass (imf17.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.216.43 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738314430; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=qaFN5XJ8A1km1G4xPnULAZdvrmEAmo6pRkSTLZcPqY0=; b=Ry4O9VHbuinVoyIxZH4nMzZPQZsM4+2LZwo3pX6CZ8MgvO9xweBEHVulMiNhxbAsoQhgcC W7UL+NODJYoPodWYSMxThfBiwoTeoirw0Mg88KZ0Tum2dkR7aYSQEccJVv+Q5nSKu9bxl3 HJZ0pitIFaNTdcQbmVJLF6mcJjx8Mlk= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=C+G2IsoQ; spf=pass (imf17.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.216.43 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738314430; a=rsa-sha256; cv=none; b=Sfs1H2o5waHIwgxRbboSpfg4nM5nlWLi/BqGbAhXpTsnSdTpTsr9tXGgkzbeJ0yMKG6x8T l2sz37MLO8uhOypdnVtY4gVI8Gi2lCLpoDFIp3QAhdclZTQFIC+2HdJO7mJ4bx5izkh6QF tc880sji3rEz9whGtk1SgScRm3qwULU= Received: by mail-pj1-f43.google.com with SMTP id 98e67ed59e1d1-2f78a4ca5deso2326325a91.0 for ; Fri, 31 Jan 2025 01:07:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738314429; x=1738919229; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=qaFN5XJ8A1km1G4xPnULAZdvrmEAmo6pRkSTLZcPqY0=; b=C+G2IsoQViax19lbPEnZN9GmYheSxOh0wsrMQPHS1pgUB5dwESSu8i0EGXu9bL7Qn0 Qp80G4S9G3Rakt9yFL83zYazbW8dV3Gvz4FqIrNy8AZCqiMZ3fMlRzcw/UQuqD9rLX6U 6FCV0b54Va/gPMW/HTCYJYLPY7sFCUxCqwUqw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738314429; x=1738919229; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qaFN5XJ8A1km1G4xPnULAZdvrmEAmo6pRkSTLZcPqY0=; b=XXIQTYNwcjGiGQgHIDcJlkNNBX9YB+51B729SjcNckSFbEbWH+VJrp3xG9pjca/4Aw 7zPbjaSt5c6XjjLmlhJc1hRDzAPjz29/Pw9LQWK68hdCPFjooQkOQe9eHO+QRgsfgtXC v9BrWEwu4fJZ2uGrKZzcWMlwM/UjxjxTME1lxDPZ2Mrw4b96o4BoIPwEpHxvBzIoTyax kvZe9h38BzvGJcM+xJmdInQ64ULA+o3WoVHOBsxO2tz3Y4ibpxfFPxQkTv8FLjy1wNr7 g4238WCsLXiqxvCJ5Fv6RO/V2V00uHTHBzlIWzV1xX2Xf25/gwbNCI9rvwH12GpSPt8m c1xg== X-Forwarded-Encrypted: i=1; AJvYcCXtmJvDUiV+2CHJtRuyU19ErAgM0RSdAuSZOpKP0GbqKiopuPpr1tDsNaurIWwx7Nb2Q7MPjnD/gA==@kvack.org X-Gm-Message-State: AOJu0YyCfkqvsaATuabWT5VlF7iCraVWEGf48dM2nQZmPAR2oUQXo6gx qGx9n8MOI67ks3Behf8Z+V8G7+rbrJ+Z2AInvG55mOCYaas0gKUs/Lv2VFhyfg== X-Gm-Gg: ASbGnctzjPZvgfr+nUQGTxAn0Cuoz4MiwhqspO4G5lQHj9wCZrmqLm8qCtBJ9ueLjvr Q0cyLWRwDCaOODqNoFIhQWM+63mGlaL2yB19CTEiZB6tTg8ThAto2z/C9VKCVyR5yFk60t8xAKh cSAbpxBXPm0mjM4XGBankPmWTo/EtRT83O6j6/wgsuL+1iWXDjMYnO7W+fFMxK+ImOLgQ+sUmBI 2fW4MDdDbP0VuyHWhxkyzmRnHQ/Oy6jWjcOj9UnHhwNcFwD6ds9pVKHlDxWTyMZmLSNoHelmydC yJY+CN/EN+5nETBZkw== X-Google-Smtp-Source: AGHT+IGIXN+n6f5l+y68Non+b6GFZDbdRRf5AaVlS7GSpPKO4QWiW4YKeYlyUraz6wfdYS57hfAKuA== X-Received: by 2002:a17:90a:d890:b0:2ee:e317:69ab with SMTP id 98e67ed59e1d1-2f83aa65ed2mr19463617a91.0.1738314428603; Fri, 31 Jan 2025 01:07:08 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:c752:be9d:3368:16fa]) by smtp.gmail.com with UTF8SMTPSA id 98e67ed59e1d1-2f83bf93e97sm5869177a91.31.2025.01.31.01.07.06 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 31 Jan 2025 01:07:08 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv4 01/17] zram: switch to non-atomic entry locking Date: Fri, 31 Jan 2025 18:06:00 +0900 Message-ID: <20250131090658.3386285-2-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.362.g079036d154-goog In-Reply-To: <20250131090658.3386285-1-senozhatsky@chromium.org> References: <20250131090658.3386285-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 06BBB40006 X-Stat-Signature: bzwiebscmzs56oac7h4g51ueeqe5cuig X-Rspam-User: X-HE-Tag: 1738314429-767751 X-HE-Meta: U2FsdGVkX1+nCgIANk7JtWQ52RBEWWxZz8OCDtpzbVpds5ymp6aljzApB1CUjVVkGPYCpCsSQbq34+C5+qoaLX6DBsTw3XVHsJw/AX3hDTkn/P50WT1hr+mADu8RxoW+JJ7GBMqJSI6FMp+coOSEYr0Cifnj+sIX1diXPIPcZaLGXFH4eaJsJpMhE7/azIoXauEkP4KuOFTHyJ+IcXw0QjL60GNlm5hD9WEDyfseprLzYBE2zegrAwr9YN0OFB8+B78cx2tuGtEVowymzS8hmO0tRXvu0ZLA2pF/h3GMX0wxjeRxzDTE6N7oTzKTjW/qRvKXGjINcXzRD7W4A6+IqaFUzGhtVFZImS/O3fQMQr36bdwZUo/b8+aacY+4/O6VuOZ62ncoNqIlrfktononu9tbEpegWKk86ZHo/ACec2sWVVABtC3hK+BBBotlAA78kXjQDDA9/Fbxjq3y7LhHZLg6kLU64u3tyIRqtOt1CYJJcJwY6FtxQvzg8vuy/SktYf9hdsMWbxd4TDFxwI9CeUduX8n58uE4xIGPlWMQBR0uzbN6B6tdhWCpx1/wNIogqywtSBJTIFRmlxMbak3Rz/pVqXW2rA7ndrVB+FuslC/bhQe+N7QmJcjBzMJbWsJZiUVKQKSxwMhIJJ02I+T4fq4q6E48peFirLbwaZgtwd9L9rsodD+c5xbU4a91RFH+bWJ8Ivl21nO/4Qvs+7ObIwbvETueqp5NQHr9JOcb7BdjKcI6GFwBPDTzSsS/kdwNThca3podX+t9E8700uO9rXyD/hKkztJg0w6pmRnGtJthCPs7qrpeRFt8Z0eiEjDhMsexJOcp/9H1421KM660Xak9f8T6ehhxM4rl+5/sDYYW9ckHrS+UifL/fdZsL71fdBjOYrnYByPCijafjnt6sJ+NBSEgBWfDFwW9iWSHMz1iGB2887TYDlgmZRViOY4oAcZl/SzQ6BNVm+RV5nM atxdEQHx F/5jNo9bGW4g1W+dzsop2IYMrnJVFcD3iXJ0+Vai5pskdcTCjm7IO4Kzp7oaZZRt+suPHfauyL8kqUQXQOFrLeJne33XiiicpWNoXKHUQI5mDxVza6Smx/PUAoC0vL8xiLwVRGQWPxE8yKdXof70up+3oZUBCf4xgIPwVfzOObriAfGC+BlaTyVj1m/HuVTi/S/5HlJ3exlNJo1a48dNG4unWvzRDvWsHPq2VyxIyp9mLlHvuxtRW/2GnTdNreipcmnrGYsuuC+r04P5CYafzxugBAj3ZerbXvmIYdBkhgZ+zs8reAemNi0Zo3w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Concurrent modifications of meta table entries is now handled by per-entry spin-lock. This has a number of shortcomings. First, this imposes atomic requirements on compression backends. zram can call both zcomp_compress() and zcomp_decompress() under entry spin-lock, which implies that we can use only compression algorithms that don't schedule/sleep/wait during compression and decompression. This, for instance, makes it impossible to use some of the ASYNC compression algorithms (H/W compression, etc.) implementations. Second, this can potentially trigger watchdogs. For example, entry re-compression with secondary algorithms is performed under entry spin-lock. Given that we chain secondary compression algorithms and that some of them can be configured for best compression ratio (and worst compression speed) zram can stay under spin-lock for quite some time. Do not use per-entry spin-locks and instead convert it to an atomic_t variable which open codes reader-writer type of lock. This permits preemption from slot_lock section, also reduces the sizeof() zram entry when lockdep is enabled. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 126 ++++++++++++++++++++-------------- drivers/block/zram/zram_drv.h | 6 +- 2 files changed, 79 insertions(+), 53 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 9f5020b077c5..1c2df2341704 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -58,19 +58,50 @@ static void zram_free_page(struct zram *zram, size_t index); static int zram_read_from_zspool(struct zram *zram, struct page *page, u32 index); -static int zram_slot_trylock(struct zram *zram, u32 index) +static bool zram_slot_try_write_lock(struct zram *zram, u32 index) { - return spin_trylock(&zram->table[index].lock); + atomic_t *lock = &zram->table[index].lock; + int old = ZRAM_ENTRY_UNLOCKED; + + return atomic_try_cmpxchg(lock, &old, ZRAM_ENTRY_WRLOCKED); +} + +static void zram_slot_write_lock(struct zram *zram, u32 index) +{ + atomic_t *lock = &zram->table[index].lock; + int old = atomic_read(lock); + + do { + if (old != ZRAM_ENTRY_UNLOCKED) { + cond_resched(); + old = atomic_read(lock); + continue; + } + } while (!atomic_try_cmpxchg(lock, &old, ZRAM_ENTRY_WRLOCKED)); +} + +static void zram_slot_write_unlock(struct zram *zram, u32 index) +{ + atomic_set(&zram->table[index].lock, ZRAM_ENTRY_UNLOCKED); } -static void zram_slot_lock(struct zram *zram, u32 index) +static void zram_slot_read_lock(struct zram *zram, u32 index) { - spin_lock(&zram->table[index].lock); + atomic_t *lock = &zram->table[index].lock; + int old = atomic_read(lock); + + do { + if (old == ZRAM_ENTRY_WRLOCKED) { + cond_resched(); + old = atomic_read(lock); + continue; + } + } while (!atomic_try_cmpxchg(lock, &old, old + 1)); } -static void zram_slot_unlock(struct zram *zram, u32 index) +static void zram_slot_read_unlock(struct zram *zram, u32 index) { - spin_unlock(&zram->table[index].lock); + atomic_dec(&zram->table[index].lock); } static inline bool init_done(struct zram *zram) @@ -93,7 +124,6 @@ static void zram_set_handle(struct zram *zram, u32 index, unsigned long handle) zram->table[index].handle = handle; } -/* flag operations require table entry bit_spin_lock() being held */ static bool zram_test_flag(struct zram *zram, u32 index, enum zram_pageflags flag) { @@ -229,9 +259,9 @@ static void release_pp_slot(struct zram *zram, struct zram_pp_slot *pps) { list_del_init(&pps->entry); - zram_slot_lock(zram, pps->index); + zram_slot_write_lock(zram, pps->index); zram_clear_flag(zram, pps->index, ZRAM_PP_SLOT); - zram_slot_unlock(zram, pps->index); + zram_slot_write_unlock(zram, pps->index); kfree(pps); } @@ -394,11 +424,11 @@ static void mark_idle(struct zram *zram, ktime_t cutoff) * * And ZRAM_WB slots simply cannot be ZRAM_IDLE. */ - zram_slot_lock(zram, index); + zram_slot_write_lock(zram, index); if (!zram_allocated(zram, index) || zram_test_flag(zram, index, ZRAM_WB) || zram_test_flag(zram, index, ZRAM_SAME)) { - zram_slot_unlock(zram, index); + zram_slot_write_unlock(zram, index); continue; } @@ -410,7 +440,7 @@ static void mark_idle(struct zram *zram, ktime_t cutoff) zram_set_flag(zram, index, ZRAM_IDLE); else zram_clear_flag(zram, index, ZRAM_IDLE); - zram_slot_unlock(zram, index); + zram_slot_write_unlock(zram, index); } } @@ -709,7 +739,7 @@ static int scan_slots_for_writeback(struct zram *zram, u32 mode, INIT_LIST_HEAD(&pps->entry); - zram_slot_lock(zram, index); + zram_slot_write_lock(zram, index); if (!zram_allocated(zram, index)) goto next; @@ -731,7 +761,7 @@ static int scan_slots_for_writeback(struct zram *zram, u32 mode, place_pp_slot(zram, ctl, pps); pps = NULL; next: - zram_slot_unlock(zram, index); + zram_slot_write_unlock(zram, index); } kfree(pps); @@ -822,7 +852,7 @@ static ssize_t writeback_store(struct device *dev, } index = pps->index; - zram_slot_lock(zram, index); + zram_slot_read_lock(zram, index); /* * scan_slots() sets ZRAM_PP_SLOT and relases slot lock, so * slots can change in the meantime. If slots are accessed or @@ -833,7 +863,7 @@ static ssize_t writeback_store(struct device *dev, goto next; if (zram_read_from_zspool(zram, page, index)) goto next; - zram_slot_unlock(zram, index); + zram_slot_read_unlock(zram, index); bio_init(&bio, zram->bdev, &bio_vec, 1, REQ_OP_WRITE | REQ_SYNC); @@ -860,7 +890,7 @@ static ssize_t writeback_store(struct device *dev, } atomic64_inc(&zram->stats.bd_writes); - zram_slot_lock(zram, index); + zram_slot_write_lock(zram, index); /* * Same as above, we release slot lock during writeback so * slot can change under us: slot_free() or slot_free() and @@ -882,7 +912,7 @@ static ssize_t writeback_store(struct device *dev, zram->bd_wb_limit -= 1UL << (PAGE_SHIFT - 12); spin_unlock(&zram->wb_limit_lock); next: - zram_slot_unlock(zram, index); + zram_slot_write_unlock(zram, index); release_pp_slot(zram, pps); cond_resched(); @@ -1001,7 +1031,7 @@ static ssize_t read_block_state(struct file *file, char __user *buf, for (index = *ppos; index < nr_pages; index++) { int copied; - zram_slot_lock(zram, index); + zram_slot_read_lock(zram, index); if (!zram_allocated(zram, index)) goto next; @@ -1019,13 +1049,13 @@ static ssize_t read_block_state(struct file *file, char __user *buf, ZRAM_INCOMPRESSIBLE) ? 'n' : '.'); if (count <= copied) { - zram_slot_unlock(zram, index); + zram_slot_read_unlock(zram, index); break; } written += copied; count -= copied; next: - zram_slot_unlock(zram, index); + zram_slot_read_unlock(zram, index); *ppos += 1; } @@ -1473,15 +1503,11 @@ static bool zram_meta_alloc(struct zram *zram, u64 disksize) huge_class_size = zs_huge_class_size(zram->mem_pool); for (index = 0; index < num_pages; index++) - spin_lock_init(&zram->table[index].lock); + atomic_set(&zram->table[index].lock, ZRAM_ENTRY_UNLOCKED); + return true; } -/* - * To protect concurrent access to the same index entry, - * caller should hold this table index entry's bit_spinlock to - * indicate this index entry is accessing. - */ static void zram_free_page(struct zram *zram, size_t index) { unsigned long handle; @@ -1602,17 +1628,17 @@ static int zram_read_page(struct zram *zram, struct page *page, u32 index, { int ret; - zram_slot_lock(zram, index); + zram_slot_read_lock(zram, index); if (!zram_test_flag(zram, index, ZRAM_WB)) { /* Slot should be locked through out the function call */ ret = zram_read_from_zspool(zram, page, index); - zram_slot_unlock(zram, index); + zram_slot_read_unlock(zram, index); } else { /* * The slot should be unlocked before reading from the backing * device. */ - zram_slot_unlock(zram, index); + zram_slot_read_unlock(zram, index); ret = read_from_bdev(zram, page, zram_get_handle(zram, index), parent); @@ -1655,10 +1681,10 @@ static int zram_bvec_read(struct zram *zram, struct bio_vec *bvec, static int write_same_filled_page(struct zram *zram, unsigned long fill, u32 index) { - zram_slot_lock(zram, index); + zram_slot_write_lock(zram, index); zram_set_flag(zram, index, ZRAM_SAME); zram_set_handle(zram, index, fill); - zram_slot_unlock(zram, index); + zram_slot_write_unlock(zram, index); atomic64_inc(&zram->stats.same_pages); atomic64_inc(&zram->stats.pages_stored); @@ -1693,11 +1719,11 @@ static int write_incompressible_page(struct zram *zram, struct page *page, kunmap_local(src); zs_unmap_object(zram->mem_pool, handle); - zram_slot_lock(zram, index); + zram_slot_write_lock(zram, index); zram_set_flag(zram, index, ZRAM_HUGE); zram_set_handle(zram, index, handle); zram_set_obj_size(zram, index, PAGE_SIZE); - zram_slot_unlock(zram, index); + zram_slot_write_unlock(zram, index); atomic64_add(PAGE_SIZE, &zram->stats.compr_data_size); atomic64_inc(&zram->stats.huge_pages); @@ -1718,9 +1744,9 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) bool same_filled; /* First, free memory allocated to this slot (if any) */ - zram_slot_lock(zram, index); + zram_slot_write_lock(zram, index); zram_free_page(zram, index); - zram_slot_unlock(zram, index); + zram_slot_write_unlock(zram, index); mem = kmap_local_page(page); same_filled = page_same_filled(mem, &element); @@ -1790,10 +1816,10 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); zs_unmap_object(zram->mem_pool, handle); - zram_slot_lock(zram, index); + zram_slot_write_lock(zram, index); zram_set_handle(zram, index, handle); zram_set_obj_size(zram, index, comp_len); - zram_slot_unlock(zram, index); + zram_slot_write_unlock(zram, index); /* Update stats */ atomic64_inc(&zram->stats.pages_stored); @@ -1850,7 +1876,7 @@ static int scan_slots_for_recompress(struct zram *zram, u32 mode, INIT_LIST_HEAD(&pps->entry); - zram_slot_lock(zram, index); + zram_slot_write_lock(zram, index); if (!zram_allocated(zram, index)) goto next; @@ -1871,7 +1897,7 @@ static int scan_slots_for_recompress(struct zram *zram, u32 mode, place_pp_slot(zram, ctl, pps); pps = NULL; next: - zram_slot_unlock(zram, index); + zram_slot_write_unlock(zram, index); } kfree(pps); @@ -2162,7 +2188,7 @@ static ssize_t recompress_store(struct device *dev, if (!num_recomp_pages) break; - zram_slot_lock(zram, pps->index); + zram_slot_write_lock(zram, pps->index); if (!zram_test_flag(zram, pps->index, ZRAM_PP_SLOT)) goto next; @@ -2170,7 +2196,7 @@ static ssize_t recompress_store(struct device *dev, &num_recomp_pages, threshold, prio, prio_max); next: - zram_slot_unlock(zram, pps->index); + zram_slot_write_unlock(zram, pps->index); release_pp_slot(zram, pps); if (err) { @@ -2217,9 +2243,9 @@ static void zram_bio_discard(struct zram *zram, struct bio *bio) } while (n >= PAGE_SIZE) { - zram_slot_lock(zram, index); + zram_slot_write_lock(zram, index); zram_free_page(zram, index); - zram_slot_unlock(zram, index); + zram_slot_write_unlock(zram, index); atomic64_inc(&zram->stats.notify_free); index++; n -= PAGE_SIZE; @@ -2248,9 +2274,9 @@ static void zram_bio_read(struct zram *zram, struct bio *bio) } flush_dcache_page(bv.bv_page); - zram_slot_lock(zram, index); + zram_slot_write_lock(zram, index); zram_accessed(zram, index); - zram_slot_unlock(zram, index); + zram_slot_write_unlock(zram, index); bio_advance_iter_single(bio, &iter, bv.bv_len); } while (iter.bi_size); @@ -2278,9 +2304,9 @@ static void zram_bio_write(struct zram *zram, struct bio *bio) break; } - zram_slot_lock(zram, index); + zram_slot_write_lock(zram, index); zram_accessed(zram, index); - zram_slot_unlock(zram, index); + zram_slot_write_unlock(zram, index); bio_advance_iter_single(bio, &iter, bv.bv_len); } while (iter.bi_size); @@ -2321,13 +2347,13 @@ static void zram_slot_free_notify(struct block_device *bdev, zram = bdev->bd_disk->private_data; atomic64_inc(&zram->stats.notify_free); - if (!zram_slot_trylock(zram, index)) { + if (!zram_slot_try_write_lock(zram, index)) { atomic64_inc(&zram->stats.miss_free); return; } zram_free_page(zram, index); - zram_slot_unlock(zram, index); + zram_slot_write_unlock(zram, index); } static void zram_comp_params_reset(struct zram *zram) diff --git a/drivers/block/zram/zram_drv.h b/drivers/block/zram/zram_drv.h index db78d7c01b9a..e20538cdf565 100644 --- a/drivers/block/zram/zram_drv.h +++ b/drivers/block/zram/zram_drv.h @@ -28,7 +28,6 @@ #define ZRAM_SECTOR_PER_LOGICAL_BLOCK \ (1 << (ZRAM_LOGICAL_BLOCK_SHIFT - SECTOR_SHIFT)) - /* * ZRAM is mainly used for memory efficiency so we want to keep memory * footprint small and thus squeeze size and zram pageflags into a flags @@ -58,13 +57,14 @@ enum zram_pageflags { __NR_ZRAM_PAGEFLAGS, }; -/*-- Data structures */ +#define ZRAM_ENTRY_UNLOCKED 0 +#define ZRAM_ENTRY_WRLOCKED (-1) /* Allocated for each disk page */ struct zram_table_entry { unsigned long handle; unsigned int flags; - spinlock_t lock; + atomic_t lock; #ifdef CONFIG_ZRAM_TRACK_ENTRY_ACTIME ktime_t ac_time; #endif From patchwork Fri Jan 31 09:06:01 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13955128 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B9214C0218D for ; Fri, 31 Jan 2025 09:07:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 495BB2800E9; Fri, 31 Jan 2025 04:07:16 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 41D972800E2; Fri, 31 Jan 2025 04:07:16 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 248AB2800E9; Fri, 31 Jan 2025 04:07:16 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id F0A182800E2 for ; Fri, 31 Jan 2025 04:07:15 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id AC11E8098C for ; Fri, 31 Jan 2025 09:07:15 +0000 (UTC) X-FDA: 83067168030.18.EBE0401 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) by imf12.hostedemail.com (Postfix) with ESMTP id C24B040003 for ; Fri, 31 Jan 2025 09:07:13 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=QnLHBJPp; spf=pass (imf12.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.169 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738314433; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ZM/BjG4bEVLcBszqat2kHDfQpVGFwc0ijbl9RU545So=; b=cAkPdSPYV9/GNkha/vCd3P4wq3cPjT2DtM7cpgFz91XHC4U5U79PwpxXhzk03/pAgk+2yv ybsFK19+BHz0nPcnMBVkmkTldCdnfkUWHOVRfBW4XJryJqEeOp+YCLQ/V2F+kWTKGOoVqW 2b1ROfcsL6TxziUAow3SSQokHy4IXUE= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=QnLHBJPp; spf=pass (imf12.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.169 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738314433; a=rsa-sha256; cv=none; b=R7xRk2gr/QFm+R5YBUmK/Y1vqz3brnjt2XeawlWqCgOQ9iAyXj77979AYJ0wvLOHWrpM2i lMR5ez78fesBRMXppr8j0XfGzr60+gcX+NiStnA3OyoX7YSVzYxDiXPfkP4njMN433csTy rm/rUdV3YujFYwMKCQoN+7tcEmi4qDU= Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-21670dce0a7so36068235ad.1 for ; Fri, 31 Jan 2025 01:07:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738314432; x=1738919232; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ZM/BjG4bEVLcBszqat2kHDfQpVGFwc0ijbl9RU545So=; b=QnLHBJPpmjZPUJD5pAnoSnABbLQyT37cwiJt3ypZ20d60u3+sbESjbfK4Un2rqand2 pZ561008bdmo5jaO6x1NMBows1yQbjmAognZYYN1rltTW/eUSSVjCW9ShtM2EAwH2OKV 9uYbcYZeQp2ZWfEg73uushqjnfh1nxebjVrZk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738314432; x=1738919232; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ZM/BjG4bEVLcBszqat2kHDfQpVGFwc0ijbl9RU545So=; b=Ljg2u8eZO/xm22MPAhvnkSz1ve++8oZyqiCf9mCUF4lDtJU8sMwRO7isEHYCDtRV4x 0smxxx7PsQkNZvZV/+Zdgq/MoBGQgV2GpSYX8Gq6x6pUSKkOFYwoWTkLX4nUqV72Kd0e g1v38hpn7Ql+uSDR83Qb+6ZXaqY1c5WNQvXohHW8mVzXatOcn5hXTHNptrc/85bgDNxz 1h8qp1yDoWZQonKjxmYCP1p+pBEzKFJAbqey4f3VVuVOcFxu6hD9iRuFvIRiX6qM/dnE FgOksoXVW5IyiVDIt6giX6DR5+VTcJkoJzgJ3JEhe+zzDoQR3y7Aqk/odiqQEmQyPvcg 1hag== X-Forwarded-Encrypted: i=1; AJvYcCWkCCcBST2OveVPn/dJkPg5jFXy+3cn3VyYyRcDFme4wOTTj/7aE5S2HVPI8KT2sCcqMmnJa3BKWA==@kvack.org X-Gm-Message-State: AOJu0YwvVBpfPH6Sf6vxqmaU6nsCH5kGoXH3Rvs3eTwu9leBnvDukISJ bb+aIsiTAMOp/EMHO0Yuk1rVGwPSU8tyClu/kMYzj4Bc1h15fT+ezrdiwYqACg== X-Gm-Gg: ASbGncsr20b5jAbnzSKg9SWarj7KeICUfCR1jjlDfdWwp2Tv6IToj9e59LPFGC3xKpi e9YzxYczAGS/RiLPoDLRFmd21gm5vF4Mu4m9lnjPif8ObBepvsrUq7VsZiGN/9zpFDCs546xqO4 rbwNMZmBAnrAlxUkQl3L0bcJbmRIpS4UEAHpyMrPz3JKx84pGiHw3ZYFyD+KcosuZKRYTesQPkk Jsu/AZxavttkmOKzwhaQp9EKM0lDXZiPAyo5gY+4QRyslpnFpvNOcZBQUIVuSi8OLuvW/1V7W7t kswBksBbkTM+bQjz+w== X-Google-Smtp-Source: AGHT+IHUYlW/s/BROjpWDLIYuBNea13AbLtp5vL/iIvdkHnTpWdyOu8kZuE13OYSyQwa0btUHKfWmA== X-Received: by 2002:a05:6a21:9102:b0:1e1:faa:d8cf with SMTP id adf61e73a8af0-1ed7a6e1efcmr19676241637.40.1738314432483; Fri, 31 Jan 2025 01:07:12 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:c752:be9d:3368:16fa]) by smtp.gmail.com with UTF8SMTPSA id 41be03b00d2f7-acebddbb0besm2609414a12.16.2025.01.31.01.07.10 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 31 Jan 2025 01:07:12 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv4 02/17] zram: do not use per-CPU compression streams Date: Fri, 31 Jan 2025 18:06:01 +0900 Message-ID: <20250131090658.3386285-3-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.362.g079036d154-goog In-Reply-To: <20250131090658.3386285-1-senozhatsky@chromium.org> References: <20250131090658.3386285-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: C24B040003 X-Stat-Signature: xz9a4mfkxynmqk7nno3onbuqce16x1m6 X-Rspam-User: X-HE-Tag: 1738314433-280625 X-HE-Meta: U2FsdGVkX18/MBHDYp4d9L+2q0D3F4glBCEUlwixG9Xtqsg+3oU/wNjVDBJ23xAYot6mNvXYWKzT9XrfvIMsZSM/Bzm1g6VEEh4l3gpPYVegdaR1x08rR+wB+hM7GWM57wIpyfGaj0CAD3gV7WX0h4SVn6OMUSQQefFrmpZNmchnZB0lR6HCLRSUXIgOKnG8geM1O1j9yG8BnNuOhaQa4DjW/GNM165qOupp0/D8CDoduqOYhtm/AXh5AeAuoYl3cUUFHvlP4MJRk/VcFJK3MjOa+h5XVjCn3FcAQL9Xi1vNqjHf5WpiGpzZ/DoZPKpN7KsIz0RZnddRxZ0RXD74OLgq/AhiZK/d6uHHPBigUxJwStcyCFW0BdH8qy1kUJAPOrr8LTGaqpBW1I4zN0rWhc5lOp4WxPVQaMLEliFwsXnkWQg/9RdxA9YPWQ+BYYXjW7HtNdFaxxLIpMIc1IAf6/TZGk5UAcZYlf15uPsGZsCsMY7P1oUO5RPioKMWRCQ17SVJjLeESZ+wtXQRR3+4hqJBxfl3Q1pg6IRKNrOqRzQVHtsAilWZq1GkliGKabeUm+hNzqkjVs+7VBmCI1CH6VPpZqd0Xrk2Nwm/g/Npygr1fKPW6zChkSjGLjt2dv5ZXv0yC+cn6K/uB2c6oMsnN01bE1jT4rvX6rtFPxJBuzX0mr/CPBmJfhMI95kJOWRHot9ehZVe4Yrd70kpC1Y+OpQ577iqlZsEYHTt3td7dcNDW5CiVXJOjdk0Ootc6fRLlWsjsF3TBqaBBEOlJ4Wcsh6IqbRk85Rk0i6mhv0sZC3tDIdzmoryK06ksKnVXA/b1HV6RAeoMNzYre+ARjulibUCpMEX+LbD4TwFjJLFmx4O88mMt9zOUkIExsMOK6bX+TxoLm457GgjEkmCikeAaqo+DKdDS9mO3YPAyRLq9VfLO8Iv/ueMS1O4iLaOimrYJtE64Z37J89WfPj5OoL /V9iw4Fa RkgbCsWKwsEzCmYF+H394uSh95cIv9/iICEfu5cl0NYhHibby6e5ZP++YRijkBdmUWTEfqBhS0D4JnAcn+lwtGE3AArnHpwFQl+iRuju2/fFU4Oi1rIV1B9iDne9nRwq6lESnDzHTTkWhCriAU2DW3l9D75vkRZaL4r8Wl7o6GoJSOimHPNLiQD1i6Gou95VJ6ilK4JM6rpUhqhBQU+njJsWszrZPSoUntyGqq+/LAYPabXyERL6Y8XehN6Ms9tw5evA5JvcidYSAfkeSRc7eKryZ1pp+oqHyoLbT0t6axd4vpZtZ+FvJ5wm9Xg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Similarly to per-entry spin-lock per-CPU compression streams also have a number of shortcoming. First, per-CPU stream access has to be done from a non-preemptible (atomic) section, which imposes the same atomicity requirements on compression backends as entry spin-lock do and makes it impossible to use algorithms that can schedule/wait/sleep during compression and decompression. Second, per-CPU streams noticeably increase memory usage (actually more like wastage) of secondary compression streams. The problem is that secondary compression streams are allocated per-CPU, just like the primary streams are. Yet we never use more that one secondary stream at a time, because recompression is a single threaded action. Which means that remaining num_online_cpu() - 1 streams are allocated for nothing, and this is per-priority list (we can have several secondary compression algorithms). Depending on the algorithm this may lead to a significant memory wastage, in addition each stream also carries a workmem buffer (2 physical pages). Instead of per-CPU streams, maintain a list of idle compression streams and allocate new streams on-demand (something that we used to do many years ago). So that zram read() and write() become non-atomic and ease requirements on the compression algorithm implementation. This also means that we now should have only one secondary stream per-priority list. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zcomp.c | 164 +++++++++++++++++++--------------- drivers/block/zram/zcomp.h | 17 ++-- drivers/block/zram/zram_drv.c | 29 +++--- include/linux/cpuhotplug.h | 1 - 4 files changed, 109 insertions(+), 102 deletions(-) diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c index bb514403e305..982c769d5831 100644 --- a/drivers/block/zram/zcomp.c +++ b/drivers/block/zram/zcomp.c @@ -6,7 +6,7 @@ #include #include #include -#include +#include #include #include @@ -43,31 +43,40 @@ static const struct zcomp_ops *backends[] = { NULL }; -static void zcomp_strm_free(struct zcomp *comp, struct zcomp_strm *zstrm) +static void zcomp_strm_free(struct zcomp *comp, struct zcomp_strm *strm) { - comp->ops->destroy_ctx(&zstrm->ctx); - vfree(zstrm->buffer); - zstrm->buffer = NULL; + comp->ops->destroy_ctx(&strm->ctx); + vfree(strm->buffer); + kfree(strm); } -static int zcomp_strm_init(struct zcomp *comp, struct zcomp_strm *zstrm) +static struct zcomp_strm *zcomp_strm_alloc(struct zcomp *comp) { + struct zcomp_strm *strm; int ret; - ret = comp->ops->create_ctx(comp->params, &zstrm->ctx); - if (ret) - return ret; + strm = kzalloc(sizeof(*strm), GFP_KERNEL); + if (!strm) + return NULL; + + INIT_LIST_HEAD(&strm->entry); + + ret = comp->ops->create_ctx(comp->params, &strm->ctx); + if (ret) { + kfree(strm); + return NULL; + } /* - * allocate 2 pages. 1 for compressed data, plus 1 extra for the - * case when compressed size is larger than the original one + * allocate 2 pages. 1 for compressed data, plus 1 extra in case if + * compressed data is larger than the original one. */ - zstrm->buffer = vzalloc(2 * PAGE_SIZE); - if (!zstrm->buffer) { - zcomp_strm_free(comp, zstrm); - return -ENOMEM; + strm->buffer = vzalloc(2 * PAGE_SIZE); + if (!strm->buffer) { + zcomp_strm_free(comp, strm); + return NULL; } - return 0; + return strm; } static const struct zcomp_ops *lookup_backend_ops(const char *comp) @@ -109,13 +118,59 @@ ssize_t zcomp_available_show(const char *comp, char *buf) struct zcomp_strm *zcomp_stream_get(struct zcomp *comp) { - local_lock(&comp->stream->lock); - return this_cpu_ptr(comp->stream); + struct zcomp_strm *strm; + + might_sleep(); + + while (1) { + spin_lock(&comp->strm_lock); + if (!list_empty(&comp->idle_strm)) { + strm = list_first_entry(&comp->idle_strm, + struct zcomp_strm, + entry); + list_del(&strm->entry); + spin_unlock(&comp->strm_lock); + return strm; + } + + /* cannot allocate new stream, wait for an idle one */ + if (comp->avail_strm >= num_online_cpus()) { + spin_unlock(&comp->strm_lock); + wait_event(comp->strm_wait, + !list_empty(&comp->idle_strm)); + continue; + } + + /* allocate new stream */ + comp->avail_strm++; + spin_unlock(&comp->strm_lock); + + strm = zcomp_strm_alloc(comp); + if (strm) + break; + + spin_lock(&comp->strm_lock); + comp->avail_strm--; + spin_unlock(&comp->strm_lock); + wait_event(comp->strm_wait, !list_empty(&comp->idle_strm)); + } + + return strm; } -void zcomp_stream_put(struct zcomp *comp) +void zcomp_stream_put(struct zcomp *comp, struct zcomp_strm *strm) { - local_unlock(&comp->stream->lock); + spin_lock(&comp->strm_lock); + if (comp->avail_strm <= num_online_cpus()) { + list_add(&strm->entry, &comp->idle_strm); + spin_unlock(&comp->strm_lock); + wake_up(&comp->strm_wait); + return; + } + + comp->avail_strm--; + spin_unlock(&comp->strm_lock); + zcomp_strm_free(comp, strm); } int zcomp_compress(struct zcomp *comp, struct zcomp_strm *zstrm, @@ -148,61 +203,19 @@ int zcomp_decompress(struct zcomp *comp, struct zcomp_strm *zstrm, return comp->ops->decompress(comp->params, &zstrm->ctx, &req); } -int zcomp_cpu_up_prepare(unsigned int cpu, struct hlist_node *node) -{ - struct zcomp *comp = hlist_entry(node, struct zcomp, node); - struct zcomp_strm *zstrm; - int ret; - - zstrm = per_cpu_ptr(comp->stream, cpu); - local_lock_init(&zstrm->lock); - - ret = zcomp_strm_init(comp, zstrm); - if (ret) - pr_err("Can't allocate a compression stream\n"); - return ret; -} - -int zcomp_cpu_dead(unsigned int cpu, struct hlist_node *node) -{ - struct zcomp *comp = hlist_entry(node, struct zcomp, node); - struct zcomp_strm *zstrm; - - zstrm = per_cpu_ptr(comp->stream, cpu); - zcomp_strm_free(comp, zstrm); - return 0; -} - -static int zcomp_init(struct zcomp *comp, struct zcomp_params *params) -{ - int ret; - - comp->stream = alloc_percpu(struct zcomp_strm); - if (!comp->stream) - return -ENOMEM; - - comp->params = params; - ret = comp->ops->setup_params(comp->params); - if (ret) - goto cleanup; - - ret = cpuhp_state_add_instance(CPUHP_ZCOMP_PREPARE, &comp->node); - if (ret < 0) - goto cleanup; - - return 0; - -cleanup: - comp->ops->release_params(comp->params); - free_percpu(comp->stream); - return ret; -} - void zcomp_destroy(struct zcomp *comp) { - cpuhp_state_remove_instance(CPUHP_ZCOMP_PREPARE, &comp->node); + struct zcomp_strm *strm; + + while (!list_empty(&comp->idle_strm)) { + strm = list_first_entry(&comp->idle_strm, + struct zcomp_strm, + entry); + list_del(&strm->entry); + zcomp_strm_free(comp, strm); + } + comp->ops->release_params(comp->params); - free_percpu(comp->stream); kfree(comp); } @@ -229,7 +242,12 @@ struct zcomp *zcomp_create(const char *alg, struct zcomp_params *params) return ERR_PTR(-EINVAL); } - error = zcomp_init(comp, params); + INIT_LIST_HEAD(&comp->idle_strm); + init_waitqueue_head(&comp->strm_wait); + spin_lock_init(&comp->strm_lock); + + comp->params = params; + error = comp->ops->setup_params(comp->params); if (error) { kfree(comp); return ERR_PTR(error); diff --git a/drivers/block/zram/zcomp.h b/drivers/block/zram/zcomp.h index ad5762813842..62330829db3f 100644 --- a/drivers/block/zram/zcomp.h +++ b/drivers/block/zram/zcomp.h @@ -3,10 +3,10 @@ #ifndef _ZCOMP_H_ #define _ZCOMP_H_ -#include - #define ZCOMP_PARAM_NO_LEVEL INT_MIN +#include + /* * Immutable driver (backend) parameters. The driver may attach private * data to it (e.g. driver representation of the dictionary, etc.). @@ -31,7 +31,7 @@ struct zcomp_ctx { }; struct zcomp_strm { - local_lock_t lock; + struct list_head entry; /* compression buffer */ void *buffer; struct zcomp_ctx ctx; @@ -60,16 +60,15 @@ struct zcomp_ops { const char *name; }; -/* dynamic per-device compression frontend */ struct zcomp { - struct zcomp_strm __percpu *stream; + struct list_head idle_strm; + spinlock_t strm_lock; + u32 avail_strm; + wait_queue_head_t strm_wait; const struct zcomp_ops *ops; struct zcomp_params *params; - struct hlist_node node; }; -int zcomp_cpu_up_prepare(unsigned int cpu, struct hlist_node *node); -int zcomp_cpu_dead(unsigned int cpu, struct hlist_node *node); ssize_t zcomp_available_show(const char *comp, char *buf); bool zcomp_available_algorithm(const char *comp); @@ -77,7 +76,7 @@ struct zcomp *zcomp_create(const char *alg, struct zcomp_params *params); void zcomp_destroy(struct zcomp *comp); struct zcomp_strm *zcomp_stream_get(struct zcomp *comp); -void zcomp_stream_put(struct zcomp *comp); +void zcomp_stream_put(struct zcomp *comp, struct zcomp_strm *strm); int zcomp_compress(struct zcomp *comp, struct zcomp_strm *zstrm, const void *src, unsigned int *dst_len); diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 1c2df2341704..8d5974ea8ff8 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -31,7 +31,6 @@ #include #include #include -#include #include #include @@ -1601,7 +1600,7 @@ static int read_compressed_page(struct zram *zram, struct page *page, u32 index) ret = zcomp_decompress(zram->comps[prio], zstrm, src, size, dst); kunmap_local(dst); zs_unmap_object(zram->mem_pool, handle); - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zram->comps[prio], zstrm); return ret; } @@ -1762,14 +1761,14 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) kunmap_local(mem); if (unlikely(ret)) { - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP], zstrm); pr_err("Compression failed! err=%d\n", ret); zs_free(zram->mem_pool, handle); return ret; } if (comp_len >= huge_class_size) { - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP], zstrm); return write_incompressible_page(zram, page, index); } @@ -1793,7 +1792,7 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) __GFP_HIGHMEM | __GFP_MOVABLE); if (IS_ERR_VALUE(handle)) { - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP], zstrm); atomic64_inc(&zram->stats.writestall); handle = zs_malloc(zram->mem_pool, comp_len, GFP_NOIO | __GFP_HIGHMEM | @@ -1805,7 +1804,7 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) } if (!zram_can_store_page(zram)) { - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP], zstrm); zs_free(zram->mem_pool, handle); return -ENOMEM; } @@ -1813,7 +1812,7 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) dst = zs_map_object(zram->mem_pool, handle, ZS_MM_WO); memcpy(dst, zstrm->buffer, comp_len); - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP], zstrm); zs_unmap_object(zram->mem_pool, handle); zram_slot_write_lock(zram, index); @@ -1972,7 +1971,7 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, kunmap_local(src); if (ret) { - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zram->comps[prio], zstrm); return ret; } @@ -1982,7 +1981,7 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, /* Continue until we make progress */ if (class_index_new >= class_index_old || (threshold && comp_len_new >= threshold)) { - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zram->comps[prio], zstrm); continue; } @@ -2040,13 +2039,13 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, __GFP_HIGHMEM | __GFP_MOVABLE); if (IS_ERR_VALUE(handle_new)) { - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zram->comps[prio], zstrm); return PTR_ERR((void *)handle_new); } dst = zs_map_object(zram->mem_pool, handle_new, ZS_MM_WO); memcpy(dst, zstrm->buffer, comp_len_new); - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zram->comps[prio], zstrm); zs_unmap_object(zram->mem_pool, handle_new); @@ -2794,7 +2793,6 @@ static void destroy_devices(void) zram_debugfs_destroy(); idr_destroy(&zram_index_idr); unregister_blkdev(zram_major, "zram"); - cpuhp_remove_multi_state(CPUHP_ZCOMP_PREPARE); } static int __init zram_init(void) @@ -2804,15 +2802,9 @@ static int __init zram_init(void) BUILD_BUG_ON(__NR_ZRAM_PAGEFLAGS > sizeof(zram_te.flags) * 8); - ret = cpuhp_setup_state_multi(CPUHP_ZCOMP_PREPARE, "block/zram:prepare", - zcomp_cpu_up_prepare, zcomp_cpu_dead); - if (ret < 0) - return ret; - ret = class_register(&zram_control_class); if (ret) { pr_err("Unable to register zram-control class\n"); - cpuhp_remove_multi_state(CPUHP_ZCOMP_PREPARE); return ret; } @@ -2821,7 +2813,6 @@ static int __init zram_init(void) if (zram_major <= 0) { pr_err("Unable to get major number\n"); class_unregister(&zram_control_class); - cpuhp_remove_multi_state(CPUHP_ZCOMP_PREPARE); return -EBUSY; } diff --git a/include/linux/cpuhotplug.h b/include/linux/cpuhotplug.h index 6cc5e484547c..092ace7db8ee 100644 --- a/include/linux/cpuhotplug.h +++ b/include/linux/cpuhotplug.h @@ -119,7 +119,6 @@ enum cpuhp_state { CPUHP_MM_ZS_PREPARE, CPUHP_MM_ZSWP_POOL_PREPARE, CPUHP_KVM_PPC_BOOK3S_PREPARE, - CPUHP_ZCOMP_PREPARE, CPUHP_TIMERS_PREPARE, CPUHP_TMIGR_PREPARE, CPUHP_MIPS_SOC_PREPARE, From patchwork Fri Jan 31 09:06:02 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13955129 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 99A2FC0218D for ; Fri, 31 Jan 2025 09:07:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2BEAF2800EB; Fri, 31 Jan 2025 04:07:20 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 26F022800EA; Fri, 31 Jan 2025 04:07:20 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0E9DF2800EB; Fri, 31 Jan 2025 04:07:20 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id DE11F2800EA for ; Fri, 31 Jan 2025 04:07:19 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 659FF16095B for ; Fri, 31 Jan 2025 09:07:19 +0000 (UTC) X-FDA: 83067168198.17.BEB9A0E Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) by imf23.hostedemail.com (Postfix) with ESMTP id 7E7C6140002 for ; Fri, 31 Jan 2025 09:07:17 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=dHL3vqd2; spf=pass (imf23.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.178 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738314437; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=n/jN2+B1aQ/0Z82UnSahO/vWpa5e/0HHbY39bdZtI8w=; b=5w+IpUeA3qfVMtlxfm475QHwasxYsGbT09qD+G7C79YbKTtLot7LOy4XFHCkGoV7jF1UPI GO3gcfLU/SuGtlRMtbdiRZNBrXMNabA82kFlsPPGju66B/QZ615A31DkVqYSn9YTqAFnxD vRBlnhXRUO9aSFdnAhQ2q5pyLPLHEpA= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738314437; a=rsa-sha256; cv=none; b=qmaCplpgJq3GLF3xPB/3qJoNY5b0XWQXux29Lnuyx6p7aFq9b1imCadNo89aav8Tbqotzl K5GrtiJk8tVTbc+rBUTxyoTBttviKdMETVen6A12g6Yykh14eVIu4US47tdBmPJlWR+IUQ 3ZwcVjhWs/i5M1kytmUAD3tnYHvrmrc= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=dHL3vqd2; spf=pass (imf23.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.178 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-21619108a6bso29591365ad.3 for ; Fri, 31 Jan 2025 01:07:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738314436; x=1738919236; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=n/jN2+B1aQ/0Z82UnSahO/vWpa5e/0HHbY39bdZtI8w=; b=dHL3vqd2hX0A0VDnTNRGX1a4ccqcPCuxhzsenDiOSIw4gzHr9X64KCnk7hflxgTs/e uaGFEHeDesoQtvmnWaHYS9NsVsnuJTziX04cHzOv7O4VlYWQKUBZEVdAXXuvbbcj6jrO idfDKSh8Olc+WLsCLvSlygsDgYQEK179sNDVE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738314436; x=1738919236; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=n/jN2+B1aQ/0Z82UnSahO/vWpa5e/0HHbY39bdZtI8w=; b=vDuRXRHDebmwOjxAbxLKy8qxKz/KLYRrWHJCV2D+aNEhlAIU/PWse6EyT1uZznMvq2 MM1v1/kDKB7ho4yO0nnMl9TQ+GpV/tVTZM9yJRTtIH6E88qUhg4aiYwJwOUuPt6g3z+1 l+wDbsRueHnCKQNy5v6f6QqCLip7sDZF1fIdMJ/K6MqMsqZkKn8kdPPYYf23lWCsyP75 g74uXSyVKZh2+mqtGmbp2JpAE8EdEitxUAmHez2i7lFiXMr5r2Hj+kZnWm5USwLPGEsC zS++6w/AH2+OC2C54py5eLhSnuneDAwltOMaj/NhJWvxlsjImMkjLxx1qEJG9AO8Rob5 sIKA== X-Forwarded-Encrypted: i=1; AJvYcCVFVb8ixREbGR/vKmyWss8/p7Ag1Okuukne2fNyiglcyHVleTqm7cAHPKV7Ex0BuaFc/lSk7q7E4g==@kvack.org X-Gm-Message-State: AOJu0YxakwLYIydNomg4DjERFw0zIPEoUQjNeDT/DTHuar/xaCwbuWOo YYl6TC0xM7upI4CrOuCr6IuicdbNtB8iROVmi4zDk4ZByG8T4XGxms5nNJ0t3Q== X-Gm-Gg: ASbGncsV9XQSLz+PoxEXB0n/H7j1KLsFNzyHey+PJwrzWps8UtceX1nTaO01+6Sel7O pKBEley4DtKsK4ajzqdPuve6wZmPWlJSGaO8BXdcoF83GeWoRZyTnmQwGaE66nKx67BWuv7OItU dXMPwhy6KI/Y4gvwixFD+50ncvcsNC4utxcoEn+KTbhOUXyS+Wp/0kiIOVx0cMpcL2KYzukBs5M NsI7FQ9QoSiO160rWYCMuxUa56HCfOgvtcAIzfoeCQj3fexNuzpQLUSbXi5j0vYjrq5WuztJq63 0qPkrACELkYgnzM3rw== X-Google-Smtp-Source: AGHT+IFuc7qdcZdQN6B06WAX8++wNkf8GmtQ583C/YsjPsWR7di3HOqQJ5UOroWbF4X70ISjCdoQzA== X-Received: by 2002:a05:6a00:3e13:b0:72a:a9d9:5a82 with SMTP id d2e1a72fcca58-72fd096a1bemr14710114b3a.0.1738314436186; Fri, 31 Jan 2025 01:07:16 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:c752:be9d:3368:16fa]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-72ff8ae7c9dsm575712b3a.71.2025.01.31.01.07.14 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 31 Jan 2025 01:07:15 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv4 03/17] zram: remove crypto include Date: Fri, 31 Jan 2025 18:06:02 +0900 Message-ID: <20250131090658.3386285-4-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.362.g079036d154-goog In-Reply-To: <20250131090658.3386285-1-senozhatsky@chromium.org> References: <20250131090658.3386285-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 7E7C6140002 X-Stat-Signature: 5howc8xky9qdijom8mecm5sq1dsi8hqr X-Rspam-User: X-Rspamd-Server: rspam12 X-HE-Tag: 1738314437-187438 X-HE-Meta: U2FsdGVkX1+nyI4zsggVfuQsq0/wAUGnD0NcVkurE3HBMFNu/UcFWj9p2c9QBDbfby0pVce0EbLCOH6YzQWAsWKuIokIEtjLogpu1tiaiIHPjOKzreA80lPE8DN3IdEIKOLlVSZQLUTP3molu7EkaW4ymwb3LdNjTP6zMK0nZRuqKpnMI35BEPpP1mPCpRE5t0tq4ItmeTpZJoz2bx2NQosngow3BVNu+pWfdfVvBiLrlYaHoxBR2VNNWpDbc/W32s/+wl62ZLVZgAlFUsZCa5WsBGdypDUQbz7rWIKUB+2DB7xZNem0V+fUbPveyX+865vAeApeDxFpKpTVn1CYoHEcrKxjarJBCDyEQPn4ZcvGfjt8IZvsZRgEp4S8QPHY+9Cn7tOJxb49LOg0WpOY3CMEluje8k8yIK6MqlawWHqjkpq2MvgWzcvUYQatd+coKmct0jVDUMFgpx633ASKbczl5x6Ateo4+bsGlje4CJemL7qxal0djsinIIBmkn+TLZPuE5IiRXy2g8cU5aBaOPJeAW6KZbMoLulXu2jiTVRsn94CjPSCQKgpfIGl62AkVyvRzxq1zxuVaNS/D+kJD1vxNRM2NJmSF7MKoOn/mh4vzn2yC+drBpOdEYpFIu5vQDdG96jiLi1HK5ZmeZOdzILWmPnhBaP2zt+Eo9UrnwFfCg1uxEPoqJ4eWxNcK1qmouiAIMzNozCb3NwbTy+EezdL7MesfTB+i46HKjaYe4QbFEKq2xiOlrgfkFrUjFk4zDECIhEb+PnKLRtGHD/mlzCyr5ohPrNr+db5V8krQw5XQ2xH/z7PqJQQoHWfiK5C94xbMqHTXtCNg3zqQub0yudITWNPDYenZ/FE7C7wnpDsNRWT8HEXWwSkcFwL6PALO5v2yVUnI17nhwByidUcn7kE0Y3v+5Rx1aR3wZ8SGnZMjPpX9MjqS7lAr67iTdE2fdc4dk0XjlQgNY19sQK q3PYWsE6 MmDw0FBwwGNcK9g2U7D6rRukL2hpACBLuZUWueUpy+tkwTksQa12VqNkBR2JlKqlJxYhOU8dZK47nKXVMbjYnm39j9dhaqOwkWdi9L7n/kxi3m8VhCSMui1RAQaQ/q+a/V7IPLioqAKviRD1lH8R03pXjjok+uXNzUUnUk24mhv34yvD8A85NsvWnM4V8YJjsFBsFkb+li/OV/PC5MZkTW6RBx7cdC/kMlgRMTGJolgfxyh4Du5EIqjI2LY/FUv7lJIRL1jleC6L00CoCNcRat3MdROOVGAcCe0F/fOxAt+VwmvZHaZvn3U5q0gy9Hm8BXyZcRy6Qbs0Pkv0T1ZVH/agInA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Remove a leftover crypto header include. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zcomp.c | 1 - drivers/block/zram/zram_drv.c | 4 +++- drivers/block/zram/zram_drv.h | 1 - 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c index 982c769d5831..efd5919808d9 100644 --- a/drivers/block/zram/zcomp.c +++ b/drivers/block/zram/zcomp.c @@ -7,7 +7,6 @@ #include #include #include -#include #include #include "zcomp.h" diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 8d5974ea8ff8..6239fcc340b6 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -43,6 +43,8 @@ static DEFINE_MUTEX(zram_index_mutex); static int zram_major; static const char *default_compressor = CONFIG_ZRAM_DEF_COMP; +#define ZRAM_MAX_ALGO_NAME_SZ 64 + /* Module params (documentation at end) */ static unsigned int num_devices = 1; /* @@ -1141,7 +1143,7 @@ static int __comp_algorithm_store(struct zram *zram, u32 prio, const char *buf) size_t sz; sz = strlen(buf); - if (sz >= CRYPTO_MAX_ALG_NAME) + if (sz >= ZRAM_MAX_ALGO_NAME_SZ) return -E2BIG; compressor = kstrdup(buf, GFP_KERNEL); diff --git a/drivers/block/zram/zram_drv.h b/drivers/block/zram/zram_drv.h index e20538cdf565..3ae2988090b3 100644 --- a/drivers/block/zram/zram_drv.h +++ b/drivers/block/zram/zram_drv.h @@ -17,7 +17,6 @@ #include #include -#include #include "zcomp.h" From patchwork Fri Jan 31 09:06:03 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13955130 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6A62DC0218D for ; Fri, 31 Jan 2025 09:07:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 026D62800F0; Fri, 31 Jan 2025 04:07:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id F19F52800EF; Fri, 31 Jan 2025 04:07:24 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DBB0B2800F0; Fri, 31 Jan 2025 04:07:24 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id BA70C2800EF for ; Fri, 31 Jan 2025 04:07:24 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 7B319A089B for ; Fri, 31 Jan 2025 09:07:24 +0000 (UTC) X-FDA: 83067168408.25.E50CDF4 Received: from mail-pj1-f50.google.com (mail-pj1-f50.google.com [209.85.216.50]) by imf19.hostedemail.com (Postfix) with ESMTP id 8A9451A0004 for ; Fri, 31 Jan 2025 09:07:22 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=dkRDQ2O9; spf=pass (imf19.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.216.50 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738314442; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TXbFDLYuuOhKufVYkqbgI//dvCPqRmfyGlKUIOvD+TU=; b=6UNCiz+bzvxWXhetPsPHoiKStK062+TqqpGNRJLqc/Id3OdqYAgNZFUl+XJvuWyNQqiBFP YB28a2J7qnxYy2+jXmOqmpTm2DA+5j9N/0WIKvQJp6u2PcuUHEdhJ2HyFzv7rao+2SKbLd Ea4dPtX5/zQpQru8DhHMNcz1DY/WGYA= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=dkRDQ2O9; spf=pass (imf19.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.216.50 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738314442; a=rsa-sha256; cv=none; b=exuiWwjnC7DGR80md1zEzJ3/VjxRhssgHJJF5i0H01iUbtcPL6Isvi0d7xgp5cD4WX9q1v UufkgyRtRYtmRtwxRdF2788sQeYxYKuTXT9SSvuCnfkJyB2SkNufm+yKYAsglr8A9ABSJA an8RzyG74q5l5brNGrDidFNR++X6EXs= Received: by mail-pj1-f50.google.com with SMTP id 98e67ed59e1d1-2f13acbe29bso4435079a91.1 for ; Fri, 31 Jan 2025 01:07:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738314441; x=1738919241; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=TXbFDLYuuOhKufVYkqbgI//dvCPqRmfyGlKUIOvD+TU=; b=dkRDQ2O9xLDu8ISvnIPhgNapABFztkyD/TOR7O71Yx0yYixH05fPdrtWv5xBxl9UQ3 nMgDwQh3maugSyxfh2DEgrSTRB1TKmrDvED1p5h1s/8l13Q36I2Q5SvzRuhKpLP/hFNz tAoVFOEfSBk85e9hzDHOS2o3eNx4TgvIid4YE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738314441; x=1738919241; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TXbFDLYuuOhKufVYkqbgI//dvCPqRmfyGlKUIOvD+TU=; b=femiRqp6wDR08tWf5tbf5lptMHnpRFD+7byAuiF0+oCl/ENaEGmpJOjvBlXwdZ29RM eYu5ihha6HMLfhUewNiIxlB4m82gwZlGu6c6Gd/t+qjP8ZIjLc3o3yI88e53C92bWPgl YyJtuVjc9SUVy8ZR07/DGWzev/RmPsBpg1iiBObh36FHx+FqAUB+V21LrwGga5Yf2gKG Zz1MjqZ0Ayk8gCYcahxRX2fm1D8ETygXFWMkNg2Nm9R+ZB+NpQBvXF/tcrfGJcOdlfxi IxlFU1PzZ3bT3wRX0bDwUNFvIgBsRI/dgs+4K8gcxqCLcNKR7NZXgDsLu9qjkE+fEqiB eaOw== X-Forwarded-Encrypted: i=1; AJvYcCW2Vg0a5El1q3A3k1PzosJF67Nl4I7Dix/ImC4likFujZ0BeLCqPVnflI/pPNymzvEi017K4x25Eg==@kvack.org X-Gm-Message-State: AOJu0Yx9UYTo+f3CM38gXCR1MkVBy3dUF66mlod+u488ZOKMVTw++ER+ 6v3v8BwZiAfuWzstQAFv5FFWCuiAlU7LdGrxWulrG9JUctyCZacpc9GjWKpetoBSOJ3PCyGcbfQ = X-Gm-Gg: ASbGncsxU1YLwOdvFjx6xIHGuv+n0xUdC3LEWHfU6L8JHbkmHPLLT/8wHaM1xdNyU05 RwQl95dhIfqALCR0J1vwVhDV8iL9dtPS939HMbrpoqFCWERavR2IQ84b2HY7sn7XuNkoqn5w95w vYstznFJFJwv4Y3WwhjF8lI751KVp8rdKq/gl9jQjXc3w4z8RxM18dLMi2PiP6zYljuCZMpgCb9 Okzr28ENnARc/W/weiPNpwWr97ZDg+dbZYLB2AWYBucPrqkytDSvuYOpXU2Ytu1F6bdB2WZg+NZ n8vVN9ibsndtLaRiKQ== X-Google-Smtp-Source: AGHT+IEKH7u2FXrx6jWcyNcJb/PaxSPMsTyCbU1VktjzxFUwBwz+4J7kQF6itN5PPT+0HWOIKATLLw== X-Received: by 2002:a17:90b:2dc7:b0:2ee:9229:e4bd with SMTP id 98e67ed59e1d1-2f84633eaeemr10228618a91.2.1738314440902; Fri, 31 Jan 2025 01:07:20 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:c752:be9d:3368:16fa]) by smtp.gmail.com with UTF8SMTPSA id 98e67ed59e1d1-2f83bccd590sm5442929a91.12.2025.01.31.01.07.18 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 31 Jan 2025 01:07:20 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv4 04/17] zram: remove max_comp_streams device attr Date: Fri, 31 Jan 2025 18:06:03 +0900 Message-ID: <20250131090658.3386285-5-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.362.g079036d154-goog In-Reply-To: <20250131090658.3386285-1-senozhatsky@chromium.org> References: <20250131090658.3386285-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 8A9451A0004 X-Stat-Signature: 43uramues5pmsr5cbo3npu8w9gm11ds6 X-Rspam-User: X-HE-Tag: 1738314442-771775 X-HE-Meta: U2FsdGVkX18laiEaYiSehOUhvuV6IHgkngjYT6vEn81BgVUQlg5lcGlTAj9Pc6P9G/RWSeh3Uo8ggEeFe0zbpPcZz4mtoqz0NEkB7c9b1JTasJSr+3deNazV/RTa7TzhZN5hscyrykHRR3FyCs5rxrfVZq290NDKZqCeikO1shJv5tYSrcUnN1zgkmGA/pOQyDGQiyI3nF4Wl/4oC7NTDRkakfOXseeBjxkDDezDt/yK/DOGY3cQHpTleHSeGJhySietHiAZeWbe+ma8tcXiGOukv/IFKA1/8xw+PyITm4P/9EDsV1rkiz7mz/MnhTuSxtkTHi4SyCRdsHx6NX8EAwqLg63wu/pz5/3bmyWKml12j+6rtBTik857VQaYnuC3R8mUp1ABWbhS7P3tFs7i5nxkD6QLlvQBKOtZ5XoFEaE2po3MGbkVLI5rSNs/biUQTm++5YEvn4hyR9fCFCltaKziRgsGmG6yuS/cUqrqlbnb+rkyen1HTqkkgYWzoTWVTk+V4IafLr58xY1uoJooZ+dnYPiSdZ7vkr/2eeBbRilkYXAXF/BBssyIX+02uHF/YMrbdhQEFDXiXMvv/txa0O+KT1GJZC91L+ZQhiXse/TL7OdrncJB8Ef7Vh0h5u9R36tqkdiNrVqVDv399okjdbbaxo786wSMcGkw+mouiKlPnYV2AKMA/UyeDI2MdC7Jaisnmq/TXVL61CyjvXLH0WUntdUy9U0PjFyPeti34t1Gwf/DdnIz6OzpKuT/C09IzVDa6taWLmFwRRZeXOv4BlIFFcA3lgw1xaFRbK+8oVHg2il/KxHk6M7F0qCN2l+Df7S7KKh5pJNXeeSWj6+fFFGj18v0I2dU9EyeRIrKTvRN3I8OGOIjUzVrjUbwP7IroI6gwPNZg1oefKJhqeqHCrAxxhUvrW37UVIu5DugRNLcJ6hED/9S1oul7nfmHG6+nAkmNXBn6R9rYIzVUs+ EwhxbXLl 1HyiKneZMPwfA97R3ZnHlTV9yj/H6z5YmutoLDMSzvfIRfLBJyzE+4OvNrpjKBnj9jpBY7/L/JtUjSeYBkD8qfbNooFVoBRM8r78Yskr9rckPy9IB05tV+AO9aq59nmcvUKmUT0FN3yfhm3Gm/fdvIV6VPeEB4ZJ1tmKKRwYzEBDbt9HSClbcZiIhyZPySaZsuMuKajCD/WTVfAhbAyTPuM/7zw/+X5f59gSeT1wnJdLDfGwsK4bmqfq2norixWe80M6fWBQwBx+rb8iBB8ZzNWu6RnXkfMIjffFgi2DzRT2QexPAAUEaeV4Tfr1mt8e9G5McNZiS+dIV1M43ObPnmR0qWB5R0IRAynqo3rTRHduVMwKp+fA7kYr9WDyRhJsWlXRu X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: max_comp_streams device attribute has been defunct since May 2016 when zram switched to per-CPU compression streams, remove it. Signed-off-by: Sergey Senozhatsky --- Documentation/ABI/testing/sysfs-block-zram | 8 ----- Documentation/admin-guide/blockdev/zram.rst | 36 ++++++--------------- drivers/block/zram/zram_drv.c | 23 ------------- 3 files changed, 10 insertions(+), 57 deletions(-) diff --git a/Documentation/ABI/testing/sysfs-block-zram b/Documentation/ABI/testing/sysfs-block-zram index 1ef69e0271f9..36c57de0a10a 100644 --- a/Documentation/ABI/testing/sysfs-block-zram +++ b/Documentation/ABI/testing/sysfs-block-zram @@ -22,14 +22,6 @@ Description: device. The reset operation frees all the memory associated with this device. -What: /sys/block/zram/max_comp_streams -Date: February 2014 -Contact: Sergey Senozhatsky -Description: - The max_comp_streams file is read-write and specifies the - number of backend's zcomp_strm compression streams (number of - concurrent compress operations). - What: /sys/block/zram/comp_algorithm Date: February 2014 Contact: Sergey Senozhatsky diff --git a/Documentation/admin-guide/blockdev/zram.rst b/Documentation/admin-guide/blockdev/zram.rst index 1576fb93f06c..9bdb30901a93 100644 --- a/Documentation/admin-guide/blockdev/zram.rst +++ b/Documentation/admin-guide/blockdev/zram.rst @@ -54,7 +54,7 @@ The list of possible return codes: If you use 'echo', the returned value is set by the 'echo' utility, and, in general case, something like:: - echo 3 > /sys/block/zram0/max_comp_streams + echo foo > /sys/block/zram0/comp_algorithm if [ $? -ne 0 ]; then handle_error fi @@ -73,21 +73,7 @@ This creates 4 devices: /dev/zram{0,1,2,3} num_devices parameter is optional and tells zram how many devices should be pre-created. Default: 1. -2) Set max number of compression streams -======================================== - -Regardless of the value passed to this attribute, ZRAM will always -allocate multiple compression streams - one per online CPU - thus -allowing several concurrent compression operations. The number of -allocated compression streams goes down when some of the CPUs -become offline. There is no single-compression-stream mode anymore, -unless you are running a UP system or have only 1 CPU online. - -To find out how many streams are currently available:: - - cat /sys/block/zram0/max_comp_streams - -3) Select compression algorithm +2) Select compression algorithm =============================== Using comp_algorithm device attribute one can see available and @@ -107,7 +93,7 @@ Examples:: For the time being, the `comp_algorithm` content shows only compression algorithms that are supported by zram. -4) Set compression algorithm parameters: Optional +3) Set compression algorithm parameters: Optional ================================================= Compression algorithms may support specific parameters which can be @@ -138,7 +124,7 @@ better the compression ratio, it even can take negatives values for some algorithms), for other algorithms `level` is acceleration level (the higher the value the lower the compression ratio). -5) Set Disksize +4) Set Disksize =============== Set disk size by writing the value to sysfs node 'disksize'. @@ -158,7 +144,7 @@ There is little point creating a zram of greater than twice the size of memory since we expect a 2:1 compression ratio. Note that zram uses about 0.1% of the size of the disk when not in use so a huge zram is wasteful. -6) Set memory limit: Optional +5) Set memory limit: Optional ============================= Set memory limit by writing the value to sysfs node 'mem_limit'. @@ -177,7 +163,7 @@ Examples:: # To disable memory limit echo 0 > /sys/block/zram0/mem_limit -7) Activate +6) Activate =========== :: @@ -188,7 +174,7 @@ Examples:: mkfs.ext4 /dev/zram1 mount /dev/zram1 /tmp -8) Add/remove zram devices +7) Add/remove zram devices ========================== zram provides a control interface, which enables dynamic (on-demand) device @@ -208,7 +194,7 @@ execute:: echo X > /sys/class/zram-control/hot_remove -9) Stats +8) Stats ======== Per-device statistics are exported as various nodes under /sys/block/zram/ @@ -228,8 +214,6 @@ mem_limit WO specifies the maximum amount of memory ZRAM can writeback_limit WO specifies the maximum amount of write IO zram can write out to backing device as 4KB unit writeback_limit_enable RW show and set writeback_limit feature -max_comp_streams RW the number of possible concurrent compress - operations comp_algorithm RW show and change the compression algorithm algorithm_params WO setup compression algorithm parameters compact WO trigger memory compaction @@ -310,7 +294,7 @@ a single line of text and contains the following stats separated by whitespace: Unit: 4K bytes ============== ============================================================= -10) Deactivate +9) Deactivate ============== :: @@ -318,7 +302,7 @@ a single line of text and contains the following stats separated by whitespace: swapoff /dev/zram0 umount /dev/zram1 -11) Reset +10) Reset ========= Write any positive value to 'reset' sysfs node:: diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 6239fcc340b6..dd987e3942c7 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1096,27 +1096,6 @@ static void zram_debugfs_register(struct zram *zram) {}; static void zram_debugfs_unregister(struct zram *zram) {}; #endif -/* - * We switched to per-cpu streams and this attr is not needed anymore. - * However, we will keep it around for some time, because: - * a) we may revert per-cpu streams in the future - * b) it's visible to user space and we need to follow our 2 years - * retirement rule; but we already have a number of 'soon to be - * altered' attrs, so max_comp_streams need to wait for the next - * layoff cycle. - */ -static ssize_t max_comp_streams_show(struct device *dev, - struct device_attribute *attr, char *buf) -{ - return scnprintf(buf, PAGE_SIZE, "%d\n", num_online_cpus()); -} - -static ssize_t max_comp_streams_store(struct device *dev, - struct device_attribute *attr, const char *buf, size_t len) -{ - return len; -} - static void comp_algorithm_set(struct zram *zram, u32 prio, const char *alg) { /* Do not free statically defined compression algorithms */ @@ -2533,7 +2512,6 @@ static DEVICE_ATTR_WO(reset); static DEVICE_ATTR_WO(mem_limit); static DEVICE_ATTR_WO(mem_used_max); static DEVICE_ATTR_WO(idle); -static DEVICE_ATTR_RW(max_comp_streams); static DEVICE_ATTR_RW(comp_algorithm); #ifdef CONFIG_ZRAM_WRITEBACK static DEVICE_ATTR_RW(backing_dev); @@ -2555,7 +2533,6 @@ static struct attribute *zram_disk_attrs[] = { &dev_attr_mem_limit.attr, &dev_attr_mem_used_max.attr, &dev_attr_idle.attr, - &dev_attr_max_comp_streams.attr, &dev_attr_comp_algorithm.attr, #ifdef CONFIG_ZRAM_WRITEBACK &dev_attr_backing_dev.attr, From patchwork Fri Jan 31 09:06:04 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13955131 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66D92C0218D for ; Fri, 31 Jan 2025 09:07:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E5A9D2800F1; Fri, 31 Jan 2025 04:07:28 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E13FB2800EF; Fri, 31 Jan 2025 04:07:28 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CA97A2800F1; Fri, 31 Jan 2025 04:07:28 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id A8D4F2800EF for ; Fri, 31 Jan 2025 04:07:28 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 1A8B712097C for ; Fri, 31 Jan 2025 09:07:28 +0000 (UTC) X-FDA: 83067168576.02.E507DE1 Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by imf26.hostedemail.com (Postfix) with ESMTP id 296E0140004 for ; Fri, 31 Jan 2025 09:07:25 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=aRvxUvGl; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf26.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.180 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738314446; a=rsa-sha256; cv=none; b=k6m+pVC7ar8d8HoUFiU/pH9mmFZQHba6iAQxXE7RrFxi4H3MjdHe+0VRL+150d0+8TCS5G B7AgqiSxcll3TZ47WDRS7UJtqJjYM/lhNfUUlDHcwmnbN6CwacJffTtnrlRTRzpag931QN ZJREkI4to2UUYp1vyR0QnUn20Fn/C4s= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=aRvxUvGl; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf26.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.180 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738314446; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MtsIOf/TBIswacsBKgt0lP8Ca9NRj+2uNLPxrfHTdS0=; b=Bd3VgJKB19YNJWmEuU5kO7yPezWMZ1eroqQa0TzQjn4dH01oD7ZWZWPXEovV00uXhgN8AV 1ffzqcAkPBhClDwu80T3e90qVJceMvo5EqQ+ZDKKJQk/EpptI8sMnSy3RbuARZcw4boNPC XIf95yp1Naau0qVq8hqLMEChukCyEbo= Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-2163dc5155fso30730735ad.0 for ; Fri, 31 Jan 2025 01:07:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738314445; x=1738919245; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MtsIOf/TBIswacsBKgt0lP8Ca9NRj+2uNLPxrfHTdS0=; b=aRvxUvGlPYbVEJ2toYDMmZ6hIZrX7cu5yBd2XetOHfyV/cOasyPWbrPCrP9S872GbM sl5NwCZMPPgjWJqqcQwYlwQffgh+0UH1GLTteLNSbl5BS0HmbLqYHkz5NIN5gmoUX5zI IytcYew1t2d6VZ32/jy7aViko+/kYwg+7p1So= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738314445; x=1738919245; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MtsIOf/TBIswacsBKgt0lP8Ca9NRj+2uNLPxrfHTdS0=; b=Gp7yYth9QsVaCos7p3wa0NPBR8XX5bk4OTnUHYSwZopbhaJyWvYc+m0saw8xauZ3+j bzn1/cm1OvVpuQi4ptQfR3pxTZyylVSQEYHtVTwmdhYkhJaFg1GKmpEKmvvRIH19jFAY WAVn+NbUmhLK9LNW/RI9vpjR21jFKMvP4Sitlr9fXGlB6azahAwbmxZuRvJMTCm8Z1k6 3kaIyxE2JBLSWXXhNSF6iVpuSPNqNwjBo/TwY7l2z/tDJ+mrUN9nYzNlFR48i8m5OT5n bWwH+0VsNCQ20nwoX/GZrEIQvytwAtsokuWEtSASrvU3jwyaY73juNiVtjIdPfIXTbz9 kYMA== X-Forwarded-Encrypted: i=1; AJvYcCVP1dkT//mB7+oYGWvPXCg3/9Q9fwnFCRtbwD7EnuDYY0bb4oyvNxeIyQWqai8ozP76beTjnh601g==@kvack.org X-Gm-Message-State: AOJu0YxtDK4fqICrvGGnAL7cDaxyobjoxh05JaP1rtCjbTINIgVyoC1X a9G64/Qfv1waZvl2GOMMo5NQ5HtHiP0iBp8YeVcFZihSE8Hwd+m+Ls1m/tNJgg== X-Gm-Gg: ASbGncuUmcb2KK7zb6PUs9kTxtT3MRzm2ESz/gBBFETHUTCkNOkoMU+8A9W8vbDkfHT kEk7KzyM0G6xJnCLllLaY+jI3FMxdj9W8tNhtVTvZ3+nMwYFerVEEQ/JzGS7MvlMO8Z0JkqUVfS P5R9jlI6+vfuDuhmkuMxTsV6k/NyhxZsjwJ3alaFUXidi597tDNruMfwTftRX3i+T5ZBchPlr1j gxB4ECCWaLuICR718Gmf2wCCOcazlGhWKpBw6XdOQD/Sbn3ICkN/vjQfbHSodZcjb9fqD+Xub3w e7ZzUWC79P82Oamjqg== X-Google-Smtp-Source: AGHT+IG1fS5h4JbjO4Mc37VY88UPXs9TdEujSTlHdi2szESpBE3nfr8tk5rye/+mNhmuHHPPmpGn9w== X-Received: by 2002:a17:902:c40a:b0:212:63c0:d9e7 with SMTP id d9443c01a7336-21dd7b61a82mr166145155ad.0.1738314444996; Fri, 31 Jan 2025 01:07:24 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:c752:be9d:3368:16fa]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-21de32eb9f1sm25752285ad.115.2025.01.31.01.07.23 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 31 Jan 2025 01:07:24 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv4 05/17] zram: remove two-staged handle allocation Date: Fri, 31 Jan 2025 18:06:04 +0900 Message-ID: <20250131090658.3386285-6-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.362.g079036d154-goog In-Reply-To: <20250131090658.3386285-1-senozhatsky@chromium.org> References: <20250131090658.3386285-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 296E0140004 X-Rspamd-Server: rspam10 X-Stat-Signature: bws9y5fehz4swfkmazc7jdadqno7hjcd X-HE-Tag: 1738314445-576200 X-HE-Meta: U2FsdGVkX183f6hysPOSoYmbL/udtjFuWYobJsnezcmWwNekkyWsx9Lwe2QGNBXhVShiJTf/ydutylivxbKhqXJUcN4Xt6k8rVCMt13li943uwplzmDc2xW/qohVtmnJJ869rotcH7hcDIBKFYJwIzhJsSd0krooNtZSlzldmzZmIhGHS0tIsDvfHc9box1YqZsls2LPDiwStorqqi/Rta3fa2uwA4mRteuVWQqzgTQcRjI7UulEUapQTfEh3GFqJpBZKDGhNsXH3EeCTxRnev87jeAlCmucVO6PomDtQpQ7qgYv8xS2/+8DLMBgKlJxy2VUDpcXt3v+qpIGCEKmNVNJNScmBtWJ8BLchzSO+vV4GT97RH4xYT7rLrcRiQRFj7+UQzwOBTjNIfEtCG8XlFJ7jtnRx6SZBG4rXZ1F9F+QO0++i7IhHA5RNnVdbwa4fyEUCcHCQfNMfCgSLroOTUmaQ8dCtRmgZbL4lIoiJosUoq6p9gFNZ9oBaB7gscFSn6qcRW8tz0NAk32K7b1IQEUyxkZzZa2l6EoSfA8EB0FpkD9uRxFFwhh6j2I3WkOHg3y0zOduJ6M9doLQJNnmaAZY+SJ2owQUNjK9rkNyQAAA4LuCIuQsNV9igUzR2zY7DsdtvScyDQ/yMNxvEDlnl4zYIpSuBhBZhQn/WrEr+XfkyCq5LY6k6D1c4TY2LcrhGhaifPd8RBieGvsj5zJJ3FDKuREqkOzyYZB09SbNx1o8tsVwfOuydmx8UZDVMK0PhabeOvQu1llPMIBN8Y0IpbE2LSy11xw2eJS6kZAiIUtbkpRepT7D241u28MnoP5pUyOQFLge23j6Vh4B6Y0kE2lMnh0b0qtJWuLAW7McuOqnXJRoPsyhWftUu4JBY9G8JPnLL2wb0qzKQ9yWlTcqv6lfJDW6Q66mhFVJKGsmkzeivYRUWE1QN+5F1W1FE3Q+AykBR4tVlYgpL2TwQXk zcaym3qw cK7AoiL2Jsde3dp4ASvsMCMRuK54YwmAdvTUjkrz53/DxpejIWVppUlfPlMnW9m7jne4IaYvGzjN+dkAPOIehp6LLSIhxI3hqC0Jadi6eqAjDPpfXoXlG1U2ykxm2Bevcpj1cCi7UAyq8WuWEeHHExBdF7dKlptLI6A6IAoY350sNQtmPGvCTM0/px9NE9pMxUzu6J25i1xPagEXY/r0wHulJvI9HYfnkmtsVCTI6Q2sKGl5W9QLEXug6lX8qTCy8sYSSyEu7TZcxWl09O18iS7YYKh3rUqnbBy2dILmiKqgy9jJsV1l4KgibOw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Previously zram write() was atomic which required us to pass __GFP_KSWAPD_RECLAIM to zsmalloc handle allocation on a fast path and attempt a slow path allocation (with recompression) when the fast path failed. Since it's not atomic anymore we can permit direct reclaim during allocation, and remove fast allocation path and, also, drop the recompression path (which should reduce CPU/battery usage). Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 41 +++++------------------------------ 1 file changed, 6 insertions(+), 35 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index dd987e3942c7..0404f5e35cb4 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1716,11 +1716,11 @@ static int write_incompressible_page(struct zram *zram, struct page *page, static int zram_write_page(struct zram *zram, struct page *page, u32 index) { int ret = 0; - unsigned long handle = -ENOMEM; - unsigned int comp_len = 0; + unsigned long handle; + unsigned int comp_len; void *dst, *mem; struct zcomp_strm *zstrm; - unsigned long element = 0; + unsigned long element; bool same_filled; /* First, free memory allocated to this slot (if any) */ @@ -1734,7 +1734,6 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) if (same_filled) return write_same_filled_page(zram, element, index); -compress_again: zstrm = zcomp_stream_get(zram->comps[ZRAM_PRIMARY_COMP]); mem = kmap_local_page(page); ret = zcomp_compress(zram->comps[ZRAM_PRIMARY_COMP], zstrm, @@ -1743,8 +1742,6 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) if (unlikely(ret)) { zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP], zstrm); - pr_err("Compression failed! err=%d\n", ret); - zs_free(zram->mem_pool, handle); return ret; } @@ -1753,36 +1750,10 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) return write_incompressible_page(zram, page, index); } - /* - * handle allocation has 2 paths: - * a) fast path is executed with preemption disabled (for - * per-cpu streams) and has __GFP_DIRECT_RECLAIM bit clear, - * since we can't sleep; - * b) slow path enables preemption and attempts to allocate - * the page with __GFP_DIRECT_RECLAIM bit set. we have to - * put per-cpu compression stream and, thus, to re-do - * the compression once handle is allocated. - * - * if we have a 'non-null' handle here then we are coming - * from the slow path and handle has already been allocated. - */ + handle = zs_malloc(zram->mem_pool, comp_len, + GFP_NOIO | __GFP_HIGHMEM | __GFP_MOVABLE); if (IS_ERR_VALUE(handle)) - handle = zs_malloc(zram->mem_pool, comp_len, - __GFP_KSWAPD_RECLAIM | - __GFP_NOWARN | - __GFP_HIGHMEM | - __GFP_MOVABLE); - if (IS_ERR_VALUE(handle)) { - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP], zstrm); - atomic64_inc(&zram->stats.writestall); - handle = zs_malloc(zram->mem_pool, comp_len, - GFP_NOIO | __GFP_HIGHMEM | - __GFP_MOVABLE); - if (IS_ERR_VALUE(handle)) - return PTR_ERR((void *)handle); - - goto compress_again; - } + return PTR_ERR((void *)handle); if (!zram_can_store_page(zram)) { zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP], zstrm); From patchwork Fri Jan 31 09:06:05 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13955132 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2F998C0218D for ; Fri, 31 Jan 2025 09:07:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B249D2800F2; Fri, 31 Jan 2025 04:07:32 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AD3832800EF; Fri, 31 Jan 2025 04:07:32 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 974FC2800F2; Fri, 31 Jan 2025 04:07:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 79EB22800EF for ; Fri, 31 Jan 2025 04:07:32 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 367361C7B27 for ; Fri, 31 Jan 2025 09:07:32 +0000 (UTC) X-FDA: 83067168744.15.AEDD86E Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) by imf07.hostedemail.com (Postfix) with ESMTP id 4BE904000E for ; Fri, 31 Jan 2025 09:07:30 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=nz3zx3mD; spf=pass (imf07.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.171 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738314450; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=loywN5MAU0/S7v3h/0kjXpkRnG0c8JJBzlP3UvQhSOM=; b=2Re9D3Hh4ESCYn9L3zOahfwSMPwoQ9+3uLu47EQ8ykjIWR+6qrmet+jGoOqrx0HsVU4KVj MCrCEhin54Ib5LLOT0lFnetatYh2fWWz+lPCTFaliPpqXCD8U5mSx2f5ho1gLReUdr//xA IfwrePjNWt1JkhvwckD8V2FIv0L9QQc= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=nz3zx3mD; spf=pass (imf07.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.171 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738314450; a=rsa-sha256; cv=none; b=xYkd3BefBTdM2L1inxRSNxSFzFhPo/5/upXmynnUcG5RxVLhRvfjz/BWSdWLvBRMMMIO+0 q9CNOtYcJSaFzpoEoxjIBiFhW9sEZOfmehvjfPXvlXT5ZC17BSqIipYE1BdtzJ/Tee/qU3 1UMdz3B85gcxQDbAOJ2ou1boZlFvDX0= Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-21670dce0a7so36071665ad.1 for ; Fri, 31 Jan 2025 01:07:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738314449; x=1738919249; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=loywN5MAU0/S7v3h/0kjXpkRnG0c8JJBzlP3UvQhSOM=; b=nz3zx3mDF9XOI1mwl2n1M2PvxeRe695lZamVPuHlno1I0rq2fAidoGbD032NYnM7kA 297yF5q/OgxXjVTnn0d+9+MgR0MCUxC3/ebPnLpW8pYro/FC3q1CcB/1Rzw/hbD1gyLo 2GF3DfI/RfUJVjl+9NWDS2HyE8NV9sIk3dcqY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738314449; x=1738919249; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=loywN5MAU0/S7v3h/0kjXpkRnG0c8JJBzlP3UvQhSOM=; b=OsSEEFDNDCC3NktCld4bVKX/pY4k4mmygNtKzt0O+zMnK9mwv+HZJ9Za5OFQh93XTj zh06KGASfn98uSXaS/NxrmcjdxauVXY6LXBQZg/thzvnUjCPIxDZ3mqOCW7pkVaEDtrs CzexfaDXcvgI/qCLWY+WPFTTPbeLJIrRkdS9q2SXZ9dHabPJyZPLt6cGK+PjDbqCprQF hwqA0GTFxZPEQRUoOApzLRM7WDmAWVTyWRGWCc95x8bPPEOm/AGRvvdCcAUORhGmmxvD 5j2JEse8LiFqnHRWuYPNU19o09c7LoIDFcps3djGHO+cINJWp1pJUCmHmmin2IqQeSQR 2zYg== X-Forwarded-Encrypted: i=1; AJvYcCXYa0oqDnQjZ13N4vFdAAn2ouvqoRXlm0XBEFPETcRBaVKsKlaHTAslBi49YFTW5G0PjPeFByDKbA==@kvack.org X-Gm-Message-State: AOJu0YxEpKQPJqgFii/zOIj10RXgVEbyl0ko2d9ezZkc9sDy/saGN7Xt 7D/JPFbiY80birM3ICq2k0Ua6/QYxkm3YdrSuyuih34OQI81b1DOPWjYw8m4lw== X-Gm-Gg: ASbGnctkRLxKGbYBAn8v47/Kdq2EhKwyRk9YK3zaFfLtRVXNWljjloXLIG2F75rH7Xr jULGoIjZpT+0xkkZ9kA6L8ARlucS1pz5BEZu7msiTqXty3LiktnfrB0f6GdQ3lznWIJUHEZDCMw HB2nKR4fwRpfVQL6bdesXn+0wsIHwmtmEXv2gPYv81L9cHB+q/QIMNFerlgMxmZKJC34hgtyXNN wGSe/F0AJOyrG9jClp/pgSGlxRhVCJtkwY6XMpgH1NdoQG4oQO4B3NhrmJL+K9Tj1ngXHKmqQPo jJJlVio5YvKcCsZ8qg== X-Google-Smtp-Source: AGHT+IF+0FKD5xX6XHXYkP+petrD53hSmc7lcGie6WBqsrwN7A64ICQ6XETnp7Q7BlhjDUtZHY0kTg== X-Received: by 2002:a17:902:cf0a:b0:215:a80b:f6f9 with SMTP id d9443c01a7336-21dd7c449ffmr131180925ad.8.1738314449146; Fri, 31 Jan 2025 01:07:29 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:c752:be9d:3368:16fa]) by smtp.gmail.com with UTF8SMTPSA id 98e67ed59e1d1-2f83bcd0d9csm5896824a91.14.2025.01.31.01.07.27 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 31 Jan 2025 01:07:28 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv4 06/17] zram: permit reclaim in zstd custom allocator Date: Fri, 31 Jan 2025 18:06:05 +0900 Message-ID: <20250131090658.3386285-7-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.362.g079036d154-goog In-Reply-To: <20250131090658.3386285-1-senozhatsky@chromium.org> References: <20250131090658.3386285-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 4BE904000E X-Stat-Signature: paegjiy11rhisdkxkhc9wxm6741p64ws X-Rspam-User: X-HE-Tag: 1738314450-793991 X-HE-Meta: U2FsdGVkX19/ua9Qeq9GJj9ZqC1L7CEAKCrZyljrAX4Ts83LPZMQCdbwLtNo990rtcu1ktYBYfdxjNmVGZWStwMDKrxMsa/LMBnGBoIC32ruZ23ouEofNZlf3tlvQVIAuSxGxeyOAV6LPg0TMkzkpBAAq83M9lq94RSpzkxE4WhZy7qazBBETy8bsqlzV0aTn6Ei9LVLPlS3gZ5e+UE+IWbcbZnyFU4/QVlAgLyI7BY7Vss8TqXa+hHGtKe6yvwQ6NkNf/sCCzo5KJJcU5q160FpXozbV5v1zsODPDO02StarC3DE9b1kYT7IZVH00NYARJOJ16qYzgDZoBSCWe+9EwA5cXp1b4km+/bVp/cO1bp8AR2pA6a50NjHCxDgK2u08BLKNRbMcpc05L7msdQz0L0rW4sjQR9I95ifqjdQBbhCH4puC08eJ29i/mHnLFz855qMOO7g6R7fjIz3VzuoCcUVY9n67nxNvFuCocQaGrWzP/Z+mhrv8f8MN0IJCWYyveZNBGZW16oQcEaHoQcDUqlzj3XB5bJRK1ve6T2WL9mBPslU2sXw863HLgvsDoOtkBDZkmBYqa1SgsOqvxeBNlKUFDrZGUaoUyDPRtshFFu25mHoY/+rHl/9UH/H+yhu3rtgxO/lWf+MjiKghLYwXxSuoZQqi8h46HBTK6vkd6KOYygLZ+9FjCNbY2tEArKbN7illYse6cv/5ct/YvhDV9nSP7yPzQ+AvSWoxy9B79jMH0AM6kGrNmRPC8mf++LtM5iJhJeGbsK7FrhB79sYVHzA2o20W8gyVYoNjzNlJN45ug2koA+exFy0wU0AxwAAZWlUUjQ4yf6f9zMUQ7n0iUkds51etl/BDiH0KT+K4u7jMWP2hHjFFU/W3bKULKYnNGPscSYoSmWbkxK/UO1/uwF6Cayjap5ARm0wPWkqZu+994i4ra3Q2m11/5bomNrDdWm1oS2rXQ1IqYw5H7 eyrRB+Cc wf6C83ziNdlS7dEoJMlGu+Cl8tWQkKfaVe5GD/jQqnvJDiLD+bvCkSFM5SX7ezbgFQQV7210vP12BdCHawTgvvMg1aiMZwqmi377a9qGT/1lhalD/wHE8Z0wTAQNm067raQUEBTk1mKAVx7sjkXImyuI5oVzmx3KN/Ya4iVLty4+khmCbIfC8/1ZQWrnq+UqtRc1qQaeOElUnlxEoLqocaOlR2jSAaJK9ZKxry8cwPiqqCZzHVo6D0wK6QBcGcAFuaDc+YQCKh6NL9ZAJmbnsA9QBb4bzbz9KO/Hl+HGK3tEhmtXn65lBpp89Hg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When configured with pre-trained compression/decompression dictionary support, zstd requires custom memory allocator, which it calls internally from compression()/decompression() routines. This was a tad problematic, because that would mean allocation from atomic context (either under entry spin-lock, or per-CPU local-lock or both). Now, with non-atomic zram write(), those limitations are relaxed and we can allow direct and indirect reclaim during allocations. The tricky part is zram read() path, which is still atomic in one particular case (read_compressed_page()), due to zsmalloc handling of object mapping. However, in zram in order to read() something one has to write() it first, and write() is when zstd allocates required internal state memory, and write() path is non-atomic. Because of this write() allocation, in theory, zstd should not call its allocator from the atomic read() path. Keep the non-preemptible branch, just in case if zstd allocates memory from read(), but WARN_ON_ONCE() if it happens. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/backend_zstd.c | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-) diff --git a/drivers/block/zram/backend_zstd.c b/drivers/block/zram/backend_zstd.c index 1184c0036f44..53431251ea62 100644 --- a/drivers/block/zram/backend_zstd.c +++ b/drivers/block/zram/backend_zstd.c @@ -24,19 +24,14 @@ struct zstd_params { /* * For C/D dictionaries we need to provide zstd with zstd_custom_mem, * which zstd uses internally to allocate/free memory when needed. - * - * This means that allocator.customAlloc() can be called from zcomp_compress() - * under local-lock (per-CPU compression stream), in which case we must use - * GFP_ATOMIC. - * - * Another complication here is that we can be configured as a swap device. */ static void *zstd_custom_alloc(void *opaque, size_t size) { - if (!preemptible()) + /* Technically this should not happen */ + if (WARN_ON_ONCE(!preemptible())) return kvzalloc(size, GFP_ATOMIC); - return kvzalloc(size, __GFP_KSWAPD_RECLAIM | __GFP_NOWARN); + return kvzalloc(size, GFP_NOIO | __GFP_NOWARN); } static void zstd_custom_free(void *opaque, void *address) From patchwork Fri Jan 31 09:06:06 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13955133 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D777DC0218D for ; Fri, 31 Jan 2025 09:07:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 68FCC2800F3; Fri, 31 Jan 2025 04:07:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 641B12800EF; Fri, 31 Jan 2025 04:07:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4E1822800F3; Fri, 31 Jan 2025 04:07:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 2E2082800EF for ; Fri, 31 Jan 2025 04:07:36 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id EBC0E140918 for ; Fri, 31 Jan 2025 09:07:35 +0000 (UTC) X-FDA: 83067168870.24.92C2421 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) by imf13.hostedemail.com (Postfix) with ESMTP id 0AEA220009 for ; Fri, 31 Jan 2025 09:07:33 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=JMJEXjwr; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf13.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.169 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738314454; a=rsa-sha256; cv=none; b=zNbcvEzpzKxDaAL30Ae1PcQtBxrp34V9VKruU4kWJs1r5SGTgrIOgMAn6IEy5+cms2xmTj 2EHXlI5G9bH7GBgmlDkLjwJduFOON7KdfhA2uq1b7+PbhWPZ97Z/zdzwk96tXtstTTqRsg aZ8tvo023ZDT7jNSB9dqO8Qwu3fnBwM= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=JMJEXjwr; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf13.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.169 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738314454; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=yIS0Akli6K3ICDNolLXeCGhuZ0MDBrb+zo33V4R+vJw=; b=cGCLWslzyYcl9BFpDQibeTCfr+eMmW2M7wR80hp0y0AhUwA1K6Ho7E51Npfl/EGznz1f8D GWg1T1LZSJpucCM1ca53lLLTHBcgxyPqFwqQSDjkOKKhH+qRw4VfpbXb9qfobZMIp8e4FH /QJR+bxlIiAOGE7tUsEGr9semdYHmaE= Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-21c2f1b610dso38757695ad.0 for ; Fri, 31 Jan 2025 01:07:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738314453; x=1738919253; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=yIS0Akli6K3ICDNolLXeCGhuZ0MDBrb+zo33V4R+vJw=; b=JMJEXjwrCg3hl4pAhdAqjhkoCURiVEESU80+F+e4LMh4fWvGdHptHnChNkVmmDBxpH iClytS0lF/loFYTWAUvflReNjGaqtvDavBx0yTaknxWql7asV7Ys+x4SkPATEUl0SQpj ieP/Hl7VDH1eNM2rklLDGZH+VvNQWoqmhR1NE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738314453; x=1738919253; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=yIS0Akli6K3ICDNolLXeCGhuZ0MDBrb+zo33V4R+vJw=; b=HaBVl2I6EsM/x4ej36EJfOVENF54r3YyOXIrM+/nMj24igL4OO99Osvh+Y6jv7RUFa SH7I9AHFvS2UnArYJSTSb0yK6Z1wJECBiYjKb+bUbDFIxioQH2FwheH+/O9pDIrWjxn+ HPOdeabmIq1rcawH2VZRRuhqf6Po+9PEh2Unf8SRDdcy/MgUQWhzcVSoQi72RlmZGmNE iiqkR/YHJVVcRfERgAUFmK0uXzt9VCQcvVxyfiZzNaaBIjHh0X5tSQNDCtHE9s9Jpzhf zYhcGv2UwEbMiL5jpgFHAJy1sl3umWyRfOJ/Rch9RT6Rh9sy4wHTEak1LHqJIMrozjMy qLtw== X-Forwarded-Encrypted: i=1; AJvYcCW/DHHToW4iNPzaJLlVNSQP1F9WAXv980uMskAO5oGnIxYRFIAkzn1Mp7o8slRJw6lzWH9eOMDvcA==@kvack.org X-Gm-Message-State: AOJu0YyZoZY5m/01v0+soa56JXuXQuXLRqMEQVxo0R7w1kSjvDYbz2uW zIQGgl0uLgppvhyV8owVRrpIdrVLBZ5+3khySvvPxKk6kYmtvSDaDu8cSfwWS0Lsx6ZG5PkEVf4 = X-Gm-Gg: ASbGncv9UMxND9KAW1a+lFT8JQ/0t4USnv1yC1V5XSDuZzi5rNI1byYN+viG5MO89En h2ws3j6WNRtsgqMawLzWLhgv44OCksV4JfdSZIBkZZA2xFJIYW5axc3RhMnqGGLL9AXbE5SjwWE /qzj0VP/d+MYNazOxzHMyrZQHR/kj9weuPPigpY3WCjZIBv+ICWZG1Jx3zuK5efLrXo5PvUkcHF Slpxgw+Jlku7+EJbuQ7kok3+a9n+BuIlgVdwuQKyRKL+2qOkZlZQqCaEgVdLPMHhZuk9rZCrmDw lBFMqtQU7jj3eR9pwg== X-Google-Smtp-Source: AGHT+IGDstI0UvROyDnrpX/fcD3zobh8n0WWgBv7PtyE0EXq8xnCiPT08ZRIDCaJlns/HLlSY9XhVw== X-Received: by 2002:a05:6a20:3d86:b0:1e1:bf32:7d3a with SMTP id adf61e73a8af0-1ed7a6c8c72mr16361840637.26.1738314452868; Fri, 31 Jan 2025 01:07:32 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:c752:be9d:3368:16fa]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-72fe6a1a75bsm2781771b3a.167.2025.01.31.01.07.31 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 31 Jan 2025 01:07:32 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv4 07/17] zram: permit reclaim in recompression handle allocation Date: Fri, 31 Jan 2025 18:06:06 +0900 Message-ID: <20250131090658.3386285-8-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.362.g079036d154-goog In-Reply-To: <20250131090658.3386285-1-senozhatsky@chromium.org> References: <20250131090658.3386285-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 0AEA220009 X-Stat-Signature: qoi4kctf54tsjog91e7ouzfohhc14ys6 X-HE-Tag: 1738314453-718255 X-HE-Meta: U2FsdGVkX19X1KcaF0NFxOV9PGCntBXzCivESbU7WqMZ1K6FV+5zfvYGc1f2RCd0mfwkG3SCcrPkzHWmRIGryyYBfaaPeA8rQddFB0sJkBW+92FcjZyLuzWX45b/updNmBSGTAMoiMUqMWKbIM45Blyu2EhcxqwDfbN9lnxvjGFXhRXnTWuELwuVKBCjuabqkd8rwAMb908CRXIrW1Mc52CstP9HrRli2Mhqx9gRx3yqqDSyLbS/9FRBTXeKdkTUeZwLad4gSqIpBahcgaYGWZ1HinBLptZHuOzrdZzo08DowvFe3G2/6aneXtFDQnwXT/GZrYc1TwKUG6xmyqqihD+Z+5TEQXqzmIWNs1t7k1EiSncOAClzdTYnXKW1bGw6sXZUf/Lx2JreDsOVZKhoMUmV9ss/8bHSyGfFvSCg1bf8jErSG430IWkpba4cWqY0X7x6aaUP1iQadakjlG7XEmRX3e+DAEf7+OgbvrNDDsiF6oCKem6IPSFpoJNIPwrKv4YUrIYgPSKi0DkBN/grYXs16OJTxfyBtJkGuHzBjG8RjQW7gn+Lv41YWfK2PNBu227ur5YWgJZyFQT2LGdEUMPvkwz8EBhNOy6cUEd6hZ6rBlmdK1xSHQ58ucbj7brnImkFlpa5aU+dyiKrVVA3rn9sA0uSItZu5djTUGnYKz/bAVEhUtlB2SWSwsTn5WhQ7y7PpjKAyBiq6d+OnZE+ifGpp8HVaCWbLx1qAK1448Wl7lmtZIrU/PKkbDjdJDHMd24OaMCX5VmtSRRu8lw96qoIeK30H6MOU4ukDxW3NrGd56szoBy4RJDRWv13VMjN2QPhky6lA67GVdTk+KQFc1nGUPAFGUmOSr1hDF94XtUPZYJ5+VHM6cjP0jYzPle1Wt1DTICtJWCLCOl56A4KsCiu8HPOY/rxUV3u8ukReoY8GNstC/fU1Uf6MXMh9YBaMiqiUn3vzZ2th/HkxeP +LZNpOrW WqffY7kHguooXyABjhGafezX/qYHifLcDAuMsh+vGJkpnwKvSxiIMsTr+vYI2DoSXp94Ra8f4hCtYzToA+PuEghD2OgzimI/N17YQKGvg4bShWEVdhwWh0VC0buIJVvJFzmcdiYe14UuaY3JBgTnei6r7Q6Q3slJjUn0um866okR8GbZPcIEmnYIUAoZnnHUriGHdMV6VCX1dPehOGu9MebTwWb0QBrsIR3QSLAEEsVZiwjX0fIGWZR4ralQF0GE4uwemUbNzokOk5tcmQ0CXOpMsD8oJ3b6SZD5cooxMZdlp3p5L7VgYDaz2+A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Recompression path can now permit direct reclaim during new zs_handle allocation, because it's not atomic anymore. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 12 +++--------- 1 file changed, 3 insertions(+), 9 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 0404f5e35cb4..33a7bfa53861 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1979,17 +1979,11 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, return 0; /* - * No direct reclaim (slow path) for handle allocation and no - * re-compression attempt (unlike in zram_write_bvec()) since - * we already have stored that object in zsmalloc. If we cannot - * alloc memory for recompressed object then we bail out and - * simply keep the old (existing) object in zsmalloc. + * If we cannot alloc memory for recompressed object then we bail out + * and simply keep the old (existing) object in zsmalloc. */ handle_new = zs_malloc(zram->mem_pool, comp_len_new, - __GFP_KSWAPD_RECLAIM | - __GFP_NOWARN | - __GFP_HIGHMEM | - __GFP_MOVABLE); + GFP_NOIO | __GFP_HIGHMEM | __GFP_MOVABLE); if (IS_ERR_VALUE(handle_new)) { zcomp_stream_put(zram->comps[prio], zstrm); return PTR_ERR((void *)handle_new); From patchwork Fri Jan 31 09:06:07 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13955134 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AAE6BC0218D for ; Fri, 31 Jan 2025 09:07:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 385722800F4; Fri, 31 Jan 2025 04:07:40 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2BD8C2800EF; Fri, 31 Jan 2025 04:07:40 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 15EA02800F4; Fri, 31 Jan 2025 04:07:40 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id EBDB62800EF for ; Fri, 31 Jan 2025 04:07:39 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id A0EC645C69 for ; Fri, 31 Jan 2025 09:07:39 +0000 (UTC) X-FDA: 83067169038.17.F5E63BA Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) by imf28.hostedemail.com (Postfix) with ESMTP id AE8D3C0006 for ; Fri, 31 Jan 2025 09:07:37 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=g3NVrxGv; spf=pass (imf28.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.170 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738314457; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=oU99tlqEx3VP4Fr6vsoC6pVJUm4rK42GNNjPkPWz9Uo=; b=S1TQo7T1qZBPvUvMvGFh6mYB6G3u7D8n4HydOuA0jCTb3WdLCgt/ZJuUr+aftVcg0ha/lR 8A+Gl8nKVNJKfeAKXTScrYD6ZyVan/sgYoSfIIc+RVjItRvvxB3x46FfCBa7F0QspXGvw1 2N+aR+xxDWQqoZfwan/ZD5/biM0atyc= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=g3NVrxGv; spf=pass (imf28.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.170 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738314457; a=rsa-sha256; cv=none; b=ZOAaJEzkvINi5fjIT49RUnD84TFL9WLTw3YQ3HGMcYWW7/JMGRt2Fu5npBlr2y7c0bvz6Q dz6g4oBPCJekfvu2XM7ywcaVaLIp2qn10mg4qFw23mJ3FC5FkHyIalTwKRNqQa8fVe9KPm O5lrFvUwRk6I8Hs6doJ5SKK8YoD8MMU= Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-21669fd5c7cso29285185ad.3 for ; Fri, 31 Jan 2025 01:07:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738314456; x=1738919256; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=oU99tlqEx3VP4Fr6vsoC6pVJUm4rK42GNNjPkPWz9Uo=; b=g3NVrxGvHVvwzwS6W1wtCd8DeHyJCO4tXM0rdMpKpoU9E0KegE8Cr3nhmXYw4wkcYu BfAtvWoYL1ZFvNQbUPmR8fU8YaslcaVXuGnJ3cG8VxnIjBM7SziXx3piSU+jAqQS8rRl deiB2tjqg1/dRXyIGO2O0YpX7N/mA/XYbap3I= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738314456; x=1738919256; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=oU99tlqEx3VP4Fr6vsoC6pVJUm4rK42GNNjPkPWz9Uo=; b=vtB1ZVaT2rUz1H1J0J6YQhgFS31xlnennwN38leqMWCDrky25tmw9dTls8hvbX2FSX onRFr94FCGm7cAsOhA/VnEbKKINfYDIpKpkjLHoB5dDTTyURV2zshdWXySEasmfffRwR Ly5H3EWuDK+w4TP41cVHtLTtWFAIaOMvBgx2UZAv2A70bgbhpM6c4vD7awOL6Yu/perg VE96BdrZDaM1fuH0D1w7t35To5I8prX73Z8974VYxeTH795eBPFGk/taopax4kx3wncp 6KWhevP9QZHMzg13znvWMHdYKmL58HQYpoWMazHbh68lokmCDFR7ct9ZLtoHIGKenEOF 7a6A== X-Forwarded-Encrypted: i=1; AJvYcCUDPgZmpqDPaLzoGhGpDij3U/v+3F+u6w2SyRx8de9a00MfRWVw+9AZTGs+/iFnJOjQYRY1lnikQA==@kvack.org X-Gm-Message-State: AOJu0Yx5bMuUfZ/rJbTBmY4mUHfmRk7wR4WVFEMawU/B+CH6FadWlN10 04/x2+JTZEakc6mynBLxM9y3plSJ7TL19KzZHNRjlq7xm+sUwiHZAx0R3YhkCg== X-Gm-Gg: ASbGncuaON6yZXCafm4Q+veBeugFSIRHMm+8DpJWAUsWA59M0tnnh0GWgYKFGZcCw6z 7GKQSJPQbilm3iKmDQI2qWLl27Qm5ryNL+IhltwK/jjRuX2Oaiu9DSVbd/g6ttUUS95zn/abuLj EmPxMZcepgmy25hPxVEYW80yCkH544/464onYQwOJOlLoeDMwIWctuV0DubSCTbT+REFWA3NayH ZOECBmb03wyOnNQNxhFtV3lwWn8MSFXvfEAMw56xlREulcyDTsOWVa81/Y0YowJEdwnO3TzP+7e lIhQTrt++xYjACPAng== X-Google-Smtp-Source: AGHT+IGQyeOJINoq+3olFAJs8BwSF3tpMesiAuDGnBGSVFEb9/qgBFf2sNzJkKESZdssbJAVfH6Lfg== X-Received: by 2002:a05:6a21:33a8:b0:1db:915b:ab11 with SMTP id adf61e73a8af0-1ed7a640d42mr17388523637.24.1738314456497; Fri, 31 Jan 2025 01:07:36 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:c752:be9d:3368:16fa]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-72fe6a1a74esm2745081b3a.159.2025.01.31.01.07.34 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 31 Jan 2025 01:07:36 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv4 08/17] zram: remove writestall zram_stats member Date: Fri, 31 Jan 2025 18:06:07 +0900 Message-ID: <20250131090658.3386285-9-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.362.g079036d154-goog In-Reply-To: <20250131090658.3386285-1-senozhatsky@chromium.org> References: <20250131090658.3386285-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: AE8D3C0006 X-Stat-Signature: bj8mje5encdqfabr6a3yymhfdodqms6x X-Rspamd-Server: rspam08 X-Rspam-User: X-HE-Tag: 1738314457-781057 X-HE-Meta: U2FsdGVkX19b9OEo7fjJH2rDi47WJhfnl7LmVE1YgmsdvEKyKoV5bs7h0Vzb6NKrl78pZKZ8ttSP7nnPUBhuFTr3Do0CpYDEexmzd7UHOsYw4H1xgJy+pvrPsRVYStScs+1e4e2P8v0tIYZQcJH8dgxOp0uaxIsU9EsCDLIjRNqp86jxW03gmS/VQ+Ci4D/taVaiiLes/464t/OXkMrBfE+nYsgUdnOzVIXI/YYdH21g67dSR+DXVC3CNMpSPW4q3HwzkGyuLi1pA7+cko59E1qCD7AHfUImfK95OQwAqQuCj/szTweYbSwJBSStROknnJbV3b5Vo/X0ed9nF/90FKRYVLbX9yhq1JppwgEGPpo2Q+iPcb+eohMa0tZjtPEkK7SCniU0ZmxcTN9fRb6WnowjbRUTw/z1AhmohCmAAzUy/O8l/dfqCyi/HsRcwfTWElNtBvZAhPvZ2cnGjHRsR40a1o96IGFa+Z3EjlDRbzGbnKNc40CG2nZ6SoU0riwMfJSq7c3ZVo9/Puo6LlcoOwmZk52L829Cbma1WaYjcB35qwRpiu1LUDx3VwamEjaerzIr41lXwahodmH17cmJdJbyuK4njHaHE8qWzwy9Ky8fIuPJC1zTsPqt8xO765KiLlT1I6eJgJV9eIAVQrZLO6Kghq6JRWwn9rfxA+lQjnvyeipJgKECORnPT15pMWTY833iQT1ZhGmPp95ukp8eWL5xAWr5ZCShn39z1k8s8XtAMKzD9Dsnx5pyo7hipVh70NLpmUCu7Eyuzk1MoknmHEFMhRP+lLoqSZ4QlAtr/zsG0C3z9HhUyQSEJP/UI1qHJ23iGs+7dlbJ7KqOv+/lmNu6BDGWPX1GFcZALz0PiErhlSDSshe7b5H12OJkO+CLQoE/BBJUdCw1AN3J6O8PeFphgXP4c697ZyyBwcqKSLuFNjmdSzwpI0R4wQkQ9Szvg6GWhJ6iUwOjxGChyHZ edCy/DNK /XNLaaOoBm5SA9gTdFL3wVeU7fRRhgvjZu40Sxfm7p5dMMeh4dyl6K2MHwmxHIsVE80aMAtqF2Cj9zDx1eWYj/nChNkPAWhOjb2O8mNz3rt2H9gN90XbgGNuZEG1OEmabGTbeglTstIyZKM2/y0a5pbQ0UdbSEbEAy3cIonjqozKehXXkWxgP0on6BbN9k73iBFS5IYKWpnNP6cOyhZ7Kx1wHRRRyGgWiRa2+1qdLsPo3MyQyjKhuglGhEdZZVOnzK721up+Qj1wqLH8+WTb+Hf+C/Js4TZAFvOAO5ZvDqfzXQ/aeiyf3IdYW3TcpX8ThoMAeOzVbB1aKtLHTWoGe98EV5vCEWgrWLlqidUaGdb2IsF8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000001, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: There is no zsmalloc handle allocation slow path now and writestall is not possible any longer. Remove it from zram_stats. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 3 +-- drivers/block/zram/zram_drv.h | 1 - 2 files changed, 1 insertion(+), 3 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 33a7bfa53861..35fca4c468a7 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1430,9 +1430,8 @@ static ssize_t debug_stat_show(struct device *dev, down_read(&zram->init_lock); ret = scnprintf(buf, PAGE_SIZE, - "version: %d\n%8llu %8llu\n", + "version: %d\n0 %8llu\n", version, - (u64)atomic64_read(&zram->stats.writestall), (u64)atomic64_read(&zram->stats.miss_free)); up_read(&zram->init_lock); diff --git a/drivers/block/zram/zram_drv.h b/drivers/block/zram/zram_drv.h index 3ae2988090b3..219d405fc26e 100644 --- a/drivers/block/zram/zram_drv.h +++ b/drivers/block/zram/zram_drv.h @@ -79,7 +79,6 @@ struct zram_stats { atomic64_t huge_pages_since; /* no. of huge pages since zram set up */ atomic64_t pages_stored; /* no. of pages currently stored */ atomic_long_t max_used_pages; /* no. of maximum pages stored */ - atomic64_t writestall; /* no. of write slow paths */ atomic64_t miss_free; /* no. of missed free */ #ifdef CONFIG_ZRAM_WRITEBACK atomic64_t bd_count; /* no. of pages in backing device */ From patchwork Fri Jan 31 09:06:08 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13955135 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 60DD7C0218D for ; Fri, 31 Jan 2025 09:07:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E596E28028B; Fri, 31 Jan 2025 04:07:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E08552800EF; Fri, 31 Jan 2025 04:07:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C823328028B; Fri, 31 Jan 2025 04:07:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id A3B302800EF for ; Fri, 31 Jan 2025 04:07:44 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 50AD0C0956 for ; Fri, 31 Jan 2025 09:07:44 +0000 (UTC) X-FDA: 83067169248.25.6FA05B6 Received: from mail-pj1-f50.google.com (mail-pj1-f50.google.com [209.85.216.50]) by imf08.hostedemail.com (Postfix) with ESMTP id 6BE6F160007 for ; Fri, 31 Jan 2025 09:07:42 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b="aN/z1WIL"; spf=pass (imf08.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.216.50 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738314462; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=i68glYQZ+E+noE0wy+Y9J0JFMp7tHf3uQmMwsu8iKww=; b=NXpG5KB6kpiPZAH1XcpM1kMrX4Ykr3vEMVg3EQht7xNSs2nc/owFdWCMGOlg6fTSw96QaT e7PWgtR/++ycZTcLbsyLpQxxjY0y9+T0FejaPqp/V/BiaxqFZIZuN0GAyuqg4ab+lv3VWQ 4CbXv9SgsVQ+AyyZWB2dGEOZ85gpx0s= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b="aN/z1WIL"; spf=pass (imf08.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.216.50 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738314462; a=rsa-sha256; cv=none; b=aBKuOK1tWSAxNdzSarA00l8HroJii+n+KE8XQYld4A2Hb+R3juqMuvLJNO3OPEJu5eul5x Xu9R2C7TSGYAMbpFe+PzzC18Gv5v1fQc7YHq7NSNq4Uj/nOD0jicD5EVmpawyWwkKXAsOa GVmn5Qgr91nxLtf36lVdz5y/zfhW79c= Received: by mail-pj1-f50.google.com with SMTP id 98e67ed59e1d1-2f43d17b0e3so3018201a91.0 for ; Fri, 31 Jan 2025 01:07:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738314461; x=1738919261; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=i68glYQZ+E+noE0wy+Y9J0JFMp7tHf3uQmMwsu8iKww=; b=aN/z1WILdx3o3fSl4kEbrNuH/96wih5jnTX2vecyDR/Yb86RsOC64nCc5VmDE/ncTY S7cMuffvr4FH0kxA7quHc4Bn5liKiDDh6HltqQ+jtMmY1C15xjz49h3MEyWC5OmSNCek j8/Xf7V90ccdTpWbJBefbKhZgt4XHfoPrcRnQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738314461; x=1738919261; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=i68glYQZ+E+noE0wy+Y9J0JFMp7tHf3uQmMwsu8iKww=; b=fTwwRwfrs0dQcJ5L2XHXRPj07vJWicemOe71vsnEz+XX87ycFTU4INBekTP+a22sAy RVq40zFOOW5X0O3T/iXv0CRg2UDwx4NxHw0GpqtCZwFo/875KYYjPDi/+bjNMhSPccQ+ jBefmlwkUZL5lKUV8PDmwRGYw3OEF42qMX98G5fH+O4+iulSpbpbzyJhEmuoM20Fj3nR tgVAKgNzLkoteyQ7DJTY9aUFv2mHQvqs2Hbn0SFyrnIuJ/6lYCSD6IRTPAjr2xPWsBSD abDudsWPpOy8sm+aFCHIWnEnD1Z3VDOdHrhkb8gJxlIa5swDsiZGiR0VF3jV9hH/vuyl 6pmw== X-Forwarded-Encrypted: i=1; AJvYcCU/WsHF3DWsjk49ujNHRutI+K5d5KJr5VqXvInE+imEMJ2eCh3XXpzccWzUxDBF869O5BflqvGWWw==@kvack.org X-Gm-Message-State: AOJu0Yzek658eDYvlTrdsitXlKoSSzwp9SxrY9tpzQomSnWI5TZl4rAp d+TGhM98rRhQNXtFo8MAlpjN8A1rlQBhkA53rlrRF7ObJ6hBMdXqqabzogk3hg== X-Gm-Gg: ASbGnctrSNDJ/M1ahwQocQ6CNOeO5q+6nAVq/54PNOKXAeGI+Z8wj12JOujob0Vvvi6 4sx3Wl7dAyfIHaWgFWfOv0CH4PnuoEtoLTcprk7Xd4xrSwIXa98l6UWwIUx4quDA7uJ/1athZbG P3gdN0vxwSEBGQwitUzuOK6VLwxanHNKBqLe0dzFrK/VZWm2xa3WvJNbJ9eDHgu36ItPNqeElNa RTqOfYDa7402ZNnzS2SCKYJWEBHNwTB5hRmkX9N7suzEnUBEoqfOWHkIgpIHTlTOBqV5JAPBZCC Tnh2xXan+ACvj8ckzg== X-Google-Smtp-Source: AGHT+IEm5A+dR0l39EGWgA22kpqEPsUgTOFh4GGf32kwJ92J5tCR45G0PnUGYdzf2axpCJd9jEM3rw== X-Received: by 2002:a17:90b:2702:b0:2f4:4003:f3ea with SMTP id 98e67ed59e1d1-2f83ac84706mr17607671a91.33.1738314461307; Fri, 31 Jan 2025 01:07:41 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:c752:be9d:3368:16fa]) by smtp.gmail.com with UTF8SMTPSA id 98e67ed59e1d1-2f84897a504sm3109092a91.7.2025.01.31.01.07.39 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 31 Jan 2025 01:07:40 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv4 09/17] zram: limit max recompress prio to num_active_comps Date: Fri, 31 Jan 2025 18:06:08 +0900 Message-ID: <20250131090658.3386285-10-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.362.g079036d154-goog In-Reply-To: <20250131090658.3386285-1-senozhatsky@chromium.org> References: <20250131090658.3386285-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 6BE6F160007 X-Stat-Signature: f7fmp9oj1p1nw1nso7t1wgq4f3h36eeq X-Rspamd-Server: rspam08 X-Rspam-User: X-HE-Tag: 1738314462-816082 X-HE-Meta: U2FsdGVkX18xf0ioc9YDHKQPiWoXbBVhCnEy880MFAAsBv6dHRGZvhurjzjd1aBFUHMaAR9Pb2afS5ehJfXlIfsciA78aMeXXe8kV5cjUQVbAX8mqFgYq/Af/T00iuTGjXzHiMCnxwk9HNmIUQInOJ6Or+95yCpWPka3r48JnY9OavHJ7xAJbjj3H6OU7iEHvumpmuhA1LslSRMjy7hOAw1L3FecVhjXO+RDn82ncpEB4SXIu1XqhsnErTcabP2omLOo0sMfpsvxZRfEXtytDXwzHuXMLYDXhDbQ/0/hTggvWPeJdLnGk7OBGBvEq1GVQqYxHbuiUvo6lzjrUpKy3Gc0UNmGRUrtYCmWuXAgLzYh4ELBE+Fy5MzcHFyc50emOMnOkPzG+YgzeXT7YerJnUaCWYi4JbSwr0iQFkU9E0RLyjrkFyNPWyWLP1W+DfBAbNgjq+BL8TJmGzn+CIo60PsJLGqcGNFKadyhTiA69kU9doDFLgNatrw0qUYPDOT1hElTm1lfHiVuzcDUcSgZ6Q8XDxE49bW+1/WhXnNO2i5nQNeAIzu4wqbJ92vS15EtwEYnGRmmh26QYchlM8FHTsDl2uiXbIlUO4iN8r+/EB8AwWHUo3Sb1euJ+LuRBD1iOBZ0XqL+nJf3pFsDDfxf06Ys8KS8EdiK7CUptetI264sLDq4JfW1v1mp8Y314hErAd1OF5e6jA4Wqb0Z7/JeJJUkqvF6nNEd5Ln+DwhzERyW9QzbnyCePwHRftmkvuIZShrVAZ8mZeQdHy4Aa6FHyDtgwM+cgxzizCBEY8is6kIawbBQ62jtccfeFMU2Y3mcEUnTcGd0GjQ2xZord18kFqd+Bpmot2ev81RZSrRN7PI2d9+mdmL3a0jEcqDcHQsbACFhP536jqQR/LKwcIRcSXa46YdeBosgXTsix3Y7GS5mY8jO8wgcsg218Ix4mldXTnKrpXFywMAYn06rY08 KT/G9/HP uiVfKJBGPhgNMQruP4lFWwumrStPEJABs433Lw2RVN5FSuEPP6vPYIdr82cjNXScfnCKarrtdO1epC4xO2h4KmdNP3jgAMgppGktXBCHAl/00wVOU67HYHX5wTRoZVEFCx0PpU9s/JzORdMsdiIPrMPtS6mylkMbPSDZm9GF+ydS/ivER3fwr0E4F2c9bcLr1ET+55ViHlk7QXvsIAVH34rRtVmGTVmsz2ngOFE1cqVmyXLxculPUTUOH4CflhSCoiQPzUmJRG0iHbCR/V/oBMFtK1ayIIFHIQ7qGVd3oCXA1vWHcj+3+PCBC63f+x/JIYwIXO86zAykp+PjXrWdE4bVCxxvaZZXo0Vqp0NAuo7oHEKA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Use the actual number of algorithms zram was configure with instead of theoretical limit of ZRAM_MAX_COMPS. Also make sure that min prio is not above max prio. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 35fca4c468a7..c500ace0d02f 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -2009,16 +2009,19 @@ static ssize_t recompress_store(struct device *dev, struct device_attribute *attr, const char *buf, size_t len) { - u32 prio = ZRAM_SECONDARY_COMP, prio_max = ZRAM_MAX_COMPS; struct zram *zram = dev_to_zram(dev); char *args, *param, *val, *algo = NULL; u64 num_recomp_pages = ULLONG_MAX; struct zram_pp_ctl *ctl = NULL; struct zram_pp_slot *pps; u32 mode = 0, threshold = 0; + u32 prio, prio_max; struct page *page; ssize_t ret; + prio = ZRAM_SECONDARY_COMP; + prio_max = zram->num_active_comps; + args = skip_spaces(buf); while (*args) { args = next_arg(args, ¶m, &val); @@ -2071,7 +2074,7 @@ static ssize_t recompress_store(struct device *dev, if (prio == ZRAM_PRIMARY_COMP) prio = ZRAM_SECONDARY_COMP; - prio_max = min(prio + 1, ZRAM_MAX_COMPS); + prio_max = prio + 1; continue; } } @@ -2099,7 +2102,7 @@ static ssize_t recompress_store(struct device *dev, continue; if (!strcmp(zram->comp_algs[prio], algo)) { - prio_max = min(prio + 1, ZRAM_MAX_COMPS); + prio_max = prio + 1; found = true; break; } @@ -2111,6 +2114,12 @@ static ssize_t recompress_store(struct device *dev, } } + prio_max = min(prio_max, (u32)zram->num_active_comps); + if (prio >= prio_max) { + ret = -EINVAL; + goto release_init_lock; + } + page = alloc_page(GFP_KERNEL); if (!page) { ret = -ENOMEM; From patchwork Fri Jan 31 09:06:09 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13955136 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9D7B4C0218D for ; Fri, 31 Jan 2025 09:07:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 30F2128028D; Fri, 31 Jan 2025 04:07:49 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 248F22800EF; Fri, 31 Jan 2025 04:07:49 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0C27C28028D; Fri, 31 Jan 2025 04:07:49 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id DECBF2800EF for ; Fri, 31 Jan 2025 04:07:48 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id A22E2C09CE for ; Fri, 31 Jan 2025 09:07:48 +0000 (UTC) X-FDA: 83067169416.09.FF31F45 Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) by imf28.hostedemail.com (Postfix) with ESMTP id C7032C0004 for ; Fri, 31 Jan 2025 09:07:46 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=Sq6riHCX; spf=pass (imf28.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.173 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738314466; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4r0pcOBNOcRsnBO2Q2AoDymvU1L0RHp2karwxkPTEA0=; b=3fNsvo6Z00dNGQkusXypDQ4LQSH2BOQ4AnjxBrezP/Z8GWZUHsGwme/3FXYoya+7j81aX7 3Bu/nRjzjX2SxTxLATT6TfRMf+bZ4y9mT9veCwJ4EESekxZCT0+AsGzad+TsACqnwv8/6j jSb+eyV7lUxiYHMxjfeAa7EqX3j+Ins= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=Sq6riHCX; spf=pass (imf28.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.173 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738314466; a=rsa-sha256; cv=none; b=yEilVzNMva10ghZNBksMEdmlEjvhO5eDRWbXHCveYye1zMhEWkYEEC5VhcWwF6lpIQeZth of8au0qNfhqzMVYgQIHMUHd7DM3XZp3I8iWNu7TC1Q6gMtuCyGS8nnQSSrXrZXaQOmaxcS MmirL/Wux9e11jKqUfMWRQGepLiquS0= Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-21619108a6bso29598705ad.3 for ; Fri, 31 Jan 2025 01:07:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738314465; x=1738919265; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=4r0pcOBNOcRsnBO2Q2AoDymvU1L0RHp2karwxkPTEA0=; b=Sq6riHCXnj3egfX0LJwUixgA44i4q4uh3mZsNkeyN7ymd5GKw/fpDoPnpW2RQOWPvl okAjb8zBXDTFUreSCsa5D013fT4jZYkNnYhDd162sMclRZH4/jVF5eWo+94m21IfDJWc qaSqXANTMgRRL2524c6paNwqoBRneUrWGmxXk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738314465; x=1738919265; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4r0pcOBNOcRsnBO2Q2AoDymvU1L0RHp2karwxkPTEA0=; b=FEfLTzKBPqBgn0+njGGmzU1r6mIkvcEZEDGd/34cM6iPUACAd86Ajw5ZFFSmi/ci5i DHt7m9dIO67bMIujA1gDYxr1YFAy4ZGuy3dztCr9rjslywT9lksK78RkzIkw1u1w0xxT F3/jqNPEcXxJPGijWGVnA+PuTBaDvEENbejUxvPQgt83XeBBS0hbKLwiUsgqIMuXuOKo 7tyQENoU/oVCgW2Jxt3qrBgj/UuKyUqnAzVApUq7q8KJoAXrln+xn9+/JE7JZB9OVLgI uVokv5w5ZrIhmgirb/4RhS2vVCeFknAI/+T4g+kmqb1Cpd3eixQLwHt7hBOWpTZgrvbf abmQ== X-Forwarded-Encrypted: i=1; AJvYcCV02U5Gm0DXSlnISogcxefki+tzWRpdLiwiKZDCCboG4OwPjLiv9kmpwawXcl7IdJwaX8GWUeo7Pg==@kvack.org X-Gm-Message-State: AOJu0YyQh9ae93OHPogInk5MrAuf6cUp8r6aQ8brQF8EJTU0F96l4+NP vCUHvAHndxsVWzT8fFn47EkYlKUAjGmrKFsOF+ehh+hx9cDJJOT9ATbhkADGzxh2vWFeR9PVfgc = X-Gm-Gg: ASbGncvxjXrYvymFzXbzKGyI5rWxqX0zXDM44wZIhY4pir+IVps58EKKHHXWKzcAb+1 y8yDNEuuW/xE6CI7V2cIgQlrMT6ju9unG2j61pFB6iz7V/wcEF3UXb2EwlRVA7+pb4zNH9PAqKU GUQkt2NCAEutK/5wvPTT7ZQrOz2sXNnvhWBit03erwN5yEGbEkLWUWbVoZFjAQ+rLSxfhq+fDVo U3GpU16TdD+1phl671hnWhqjnddiqudKAtdtKYuhPUMfWkSQAdnpdf17oxxm4Jce4EkK0QytdFn QTluvxyFWNxRQ1HE4A== X-Google-Smtp-Source: AGHT+IFVF6qQXP7j68CfVjgHMYTUsYqMzGDzrbpNfx9vlhjUIvaCbiwn0ENvclvQGAhdyEYDWrJB0Q== X-Received: by 2002:a05:6a20:2d06:b0:1e6:8f30:df53 with SMTP id adf61e73a8af0-1ed7a4dcd34mr15160965637.16.1738314465661; Fri, 31 Jan 2025 01:07:45 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:c752:be9d:3368:16fa]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-72fe69cd801sm2784910b3a.123.2025.01.31.01.07.43 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 31 Jan 2025 01:07:45 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv4 10/17] zram: filter out recomp targets based on priority Date: Fri, 31 Jan 2025 18:06:09 +0900 Message-ID: <20250131090658.3386285-11-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.362.g079036d154-goog In-Reply-To: <20250131090658.3386285-1-senozhatsky@chromium.org> References: <20250131090658.3386285-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: C7032C0004 X-Stat-Signature: p1c81ax4jzx1b9tzgeccuqqfmrfniayj X-Rspam-User: X-HE-Tag: 1738314466-18093 X-HE-Meta: U2FsdGVkX19tw23ZF9UlCKoUW618NvPuue4UaSZU7MRTxqN1Yi3hGsjANMwF2FFEj2hGSwyjiA5hIUsoEGstDu49WMqNytrRrt+JcBxhBN5CCaE+ORoBaEJYbYdjf857cC/C1v8vzOTVxEGZwIwZ1ikjYWfT4DabInD3bvHSiYFQL1M+AI0eDygOkc/Sc2JuKEpjFEQlZx328kyyxlyIiRJehSiEA8t3gzYBVUK+/N2pOg/6NKHz8cD8rQCPbV9T1DJr8S6k/t9hE+DKyJrPJsJaPoDmkx9IcJRZl1NSLnGudM8GwD6hLSkb6bUuCpqWhDOaet2AJKwS2SWqmTOu5wWkron6XZTyU9x5kmkhlRszuRhCi6bpVFeTtXa3ndNYSCw/+UdpLPe2pNXmug5MOvFZS0zEWJPau3y+75GH8DwFJue9mrZc4y+jwd7e/r4axa6rhLB54TJE4u0XOFgjZ8Hsp9uNbJtxb74dnek+RYJdfKH+XxON5sZt/WH+NP9JGkSSeyQG2kRpHNtyRfgr7TgmBQOqjlJmb36EwulxYxpt0XoYf7PWgplrTs7pvZDNnT/HkNjdy739VdmAtXgO2753C+DsF9GuYfTTjZki8UA6tK0yzuc+EuUT1TGFbgbfRoFOZHwTq77B7mV9snCuF8JRFQ31hcwa6CCxZshhRvayZPvJt6pVhke3o2PqbVTxZhJD8KPfVdw3KXfDsh8rMA7KwemkrzhCrfIDb+MNLTCmMoqMs04yPPrlOPCFZOAP+CS+Xzij+XnsXyw6XwaWYx0OIpT2adP1sYdr/35RQSYhDT4Dvg2m59bJ7LW7BFFgOzLu+0CMLgspcMPMvjuiki1ODlSs0HgLtxmXyDZQ+D2mHzRAxMYFJySupxwTJwLFi5s1F9artgazwk+I058BKkwGYcLoxQquyHxWznvVoGYkYtVmtUsZuTixikZrAypC40MkLppCYY2doa2dTPt N7D4s1r9 Iyq7qbAWrwEkpFhgJfxqE8tYUz58Ym71bjIALToA2n5UpN2uhml679UYRic2Xo+lunR3TOyVp5w0SEQzbC+Ry7SyJnbxMWGwNRyvIMxm7jlliKZk2AVooRvJeTh0SacJ5p9P0y5V/uyaJ/KjWW4MeyPTuG9mGg37Su1FkjMqcMwqHfQhwt9qlJ+7nw9Zp2QCPG2F7RMs/84Om4YEaUNEiYfFQ27gSDIasJ4LlHVM74ZFQcRWeLsGUq1SPSJLVHecHzvfXw0wjf6OVLJluKXQ86p2EntClnYsbA/0lxZArUpLY8riXtNupJmRKag== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Do no select for post processing slots that are already compressed with same or higher priority compression algorithm. This should save some memory, as previously we would still put those entries into corresponding post-processing buckets and filter them out later in recompress_slot(). Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index c500ace0d02f..256439361367 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1811,7 +1811,7 @@ static int zram_bvec_write(struct zram *zram, struct bio_vec *bvec, #define RECOMPRESS_IDLE (1 << 0) #define RECOMPRESS_HUGE (1 << 1) -static int scan_slots_for_recompress(struct zram *zram, u32 mode, +static int scan_slots_for_recompress(struct zram *zram, u32 mode, u32 prio_max, struct zram_pp_ctl *ctl) { unsigned long nr_pages = zram->disksize >> PAGE_SHIFT; @@ -1843,6 +1843,10 @@ static int scan_slots_for_recompress(struct zram *zram, u32 mode, zram_test_flag(zram, index, ZRAM_INCOMPRESSIBLE)) goto next; + /* Already compressed with same of higher priority */ + if (zram_get_priority(zram, index) + 1 >= prio_max) + goto next; + pps->index = index; place_pp_slot(zram, ctl, pps); pps = NULL; @@ -2132,7 +2136,7 @@ static ssize_t recompress_store(struct device *dev, goto release_init_lock; } - scan_slots_for_recompress(zram, mode, ctl); + scan_slots_for_recompress(zram, mode, prio_max, ctl); ret = len; while ((pps = select_pp_slot(ctl))) { From patchwork Fri Jan 31 09:06:10 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13955137 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6EC5EC0218D for ; Fri, 31 Jan 2025 09:07:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ECEE228028E; Fri, 31 Jan 2025 04:07:53 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E7D782800EF; Fri, 31 Jan 2025 04:07:53 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CF6A928028E; Fri, 31 Jan 2025 04:07:53 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id ACB9C2800EF for ; Fri, 31 Jan 2025 04:07:53 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 3073180945 for ; Fri, 31 Jan 2025 09:07:53 +0000 (UTC) X-FDA: 83067169626.29.BDDE29F Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) by imf22.hostedemail.com (Postfix) with ESMTP id 4A97CC0018 for ; Fri, 31 Jan 2025 09:07:51 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=RT9OFZl+; spf=pass (imf22.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.172 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738314471; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=w3R6ZrvyVvJXmo0SkC31+WiXIWMORonA8dJRkgz1Hic=; b=xxa28zbnCz3liMjEr4xbLeMTNu5xSXXXQiH2bUn9sCR9YconnICo2hiSlbswYW7WrXubZr KveJsdxoJlEjLYKB/eKnN1o9Jj+1A1gOnXJPCMRBw23U87af+gC9z1bZ4K7crZIEYp6nzu v8HqzgtzxD5lvn9d5HZYgwkcdiGuHGM= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=RT9OFZl+; spf=pass (imf22.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.172 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738314471; a=rsa-sha256; cv=none; b=DC0E95B7/VqYyuRbjTPR8AWE1V5esnESRBuASjBW9HO9wAfVpLIgPXMClgfpTaRYhSo+oj izC7qCexjsJrgzrf8EEe1h1ivc8Outh6jFKtqQKJJIbpHegoDcrZN6qDcD3yVZRAChE5b5 uGoCBv1Mr0Qf0/cP/+4nkreoZvfkJOw= Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-2165cb60719so32206905ad.0 for ; Fri, 31 Jan 2025 01:07:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738314470; x=1738919270; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=w3R6ZrvyVvJXmo0SkC31+WiXIWMORonA8dJRkgz1Hic=; b=RT9OFZl+ZaCu76KTs3+8Ns4vfCusBN6POghvD/PaSVeat1NuHUb3skPvaSDOGxVmyM qgBMt9KTMPncRvRBJ+J0BcyMLW55RXummkUAIysXepU6qd558kcx8oGgJv6eyE4ke33V V0yIDKqAIrprkI26kSOoZWhh6kX7N9sWoKvdY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738314470; x=1738919270; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=w3R6ZrvyVvJXmo0SkC31+WiXIWMORonA8dJRkgz1Hic=; b=HR3uhLsNpYqBTV20QSmHuCiAiZobENp77FLoUyAH/kd/HT3Jhkgrsy5wqc4Rehl4Ge qjoEcwDJN5FpA6bj+RXXC5GODFbew7BulxPc6Wo+MIA56VKwFePq0jEQz9sfjMCTCWA/ SwEcTHhX7bLaGJnGXFCoDGf9e+gJ0Zhi/IuaddoX/jdmNpPCkI2d7RAykLn+sKGje20F p6uwU6ogvunmqV90WsvEbYSyNXKWbQJ5aJ4KLiDyv7wdCk1wC401vCkn6i8Pc2e3LhLY W2tbmNOFfosp++tNUagGtyv7zpZU1n8bamBocZr/EckHKVO7xJC4mFaSc1kZDHTzly53 mw0Q== X-Forwarded-Encrypted: i=1; AJvYcCU7+2UCgOdV7MKOa1UPKqq0fDfrl2sRQf1akce0/cKvbZS05dGs/iIATtktUPC7tfM8J36Dh6PtYQ==@kvack.org X-Gm-Message-State: AOJu0Yz7cbM5j/7gkLAaErD8e/7mFyrfexDg2BY4oWbtr/1OOhzyjSzX xWd3Y2YqOPu2QLL9q5i6lLGndFj0/A55NhVXyVTIy2ersG9CmZCTHspvZ8Tn7KqxmP0CyeG0EmY = X-Gm-Gg: ASbGncupwYpYJD1KpW+U0Fv/BhoOD732gtUb+ikMo4X87xpGRUiduE2E4wGiymkccbh 52vNyrJAvhm3z2snbsCEYlKF53kqC789QIIlQ1qnDC494Aq630BhVgr01re5Y5UvyxzrF4q3RDn 0Jl44qF2CTIOzqY4XBU5FMcICSlBVvncgbAf0K069fMlej6Vj5EnL1FqMhXEFVqBoNC2CPMy0Is V6ShVdMttaXnvj/kNKJFH6lHThnS/U0lr3MktOs9a2AjI+wqTDIVPVlWuseX87pR7R08MYIUDe/ 2uJGjAxELXQzd5SfPg== X-Google-Smtp-Source: AGHT+IHLXvsFhFTC2/CpnCNKtJm8HspbzNrFJ2stc8XPxlxgUVecxgwWhc2fHSMoQmdPv+pgQ7aEHA== X-Received: by 2002:a17:903:18c:b0:215:522d:72d6 with SMTP id d9443c01a7336-21dd7dcd025mr173931335ad.38.1738314470104; Fri, 31 Jan 2025 01:07:50 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:c752:be9d:3368:16fa]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-21de3302a68sm26120825ad.180.2025.01.31.01.07.48 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 31 Jan 2025 01:07:49 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv4 11/17] zram: unlock slot during recompression Date: Fri, 31 Jan 2025 18:06:10 +0900 Message-ID: <20250131090658.3386285-12-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.362.g079036d154-goog In-Reply-To: <20250131090658.3386285-1-senozhatsky@chromium.org> References: <20250131090658.3386285-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 4A97CC0018 X-Stat-Signature: 6yxonrhhnzr8rz3c4tm35e5chrc6xkpb X-Rspam-User: X-HE-Tag: 1738314471-921255 X-HE-Meta: U2FsdGVkX18nx9IADe5zScGa4TpHcOTPpwc1AFjroqeeWcxI38r0JikN91TVUOjGGnS2LdohZUTuzEsVcfZUjjcQ93pQgRzESNRyh9R/Z/J3zH9i8lMq24xjACvA9RxVAbc91EzUnkegfTcRjL7Js0VxpQQ44KqypkwbGfA9HaknYcCadULLM3cMr36T8o41iQdgs1Py3Hf6gCYyLpggitpTS8NJNC2wkB2EaqwjG8CqcDPnlEmlUgLH37K082cDwWiyoKmAv2lGS/xrqIBah3wJ7jDEMG+jFGKWb55w9O6208cy5D2iIPFHXwiFydgUgvdFC7dzAEE8tlBJLGh9XwRGLtRvewbmc05v7iA/T9xRUGn6iGDIbR5kuZRcpXnfDJ3zx+KYFWBh8QH1836DDoaAnm92rmPZS2pRjTlSl/BQgwc/bhh7QNpiyZrCDWC0OlJHKPyfTe+Hjde9wXD2TtEewrNP2naqPsBZUDHxzavVd3KPvdzAJGhpkvhha/DBQyemwqEQPsWwnYsZDOoiPVjGuoQXpAyWvaQsbV25HwTuNeWdCOh0pBfb9r9/ku956vKiFm86qdyTxbod+PlyV1Vx3o9INNxJKOvn8KcqYUKIOVEol8CO10OdHlFAlMAicRYtbfMOeDM3Z13msgpzo/9bxOyxWdVvXqboPlSkgWadFsmZJmrGnUQ4FiPhI7nhEaaWFlqq7BcKcbLdFBP2tY1L+EnIRRAlRagAWipxjlvGXzQ+k1EmSVuToukrr0jr+CCvh8wEpqrLaEnC9rN3a4ppAO494ghiOZVK06UBc6nG47eXjvAvpUj/IT5mH7q0Fo+h3ZvwZMZpISZeJVu3+IALSvQEIrNsnTAsokYbhSXVZtMTN4mVnCAWz0UFMOPDgPcGQSoThehJukalmzpSlxRn0aE0jSZ4EPEQQL0eLZeKJdsvS0dXC6af8UUfD6FMoosuVXV/9xbEDhMAKdV oh4mhN98 Qshg5qB3nJ9ImHbgsMNk9TgoWcTYA5RgRO/PVddwPKqqOvGj4lMq291MeMgnoKt59GU3AlD+JEPnCTpNN4jl6fp/Zd1eqHvRgevqY0w/pOSTDDugnMdG57wRHRH5wprIBRiuN7b4R7r7U5djOtVCJwEEVjCsWiJJwbbj9z3AQXvrnzqcixv6HpstKucftacBp1h8A2fTGGE/IA+yHtM//5NnU939nhFVJRDbF6Y4MnPQ8sfIH2igxECn35vyv9etO4Z08mnT8Vgx5nmjnrptJuREutM0zTiX56i1GoLhDn2N0LK4Q6J8S7LYTwkLVrQD3Vc2tUHb20rZOPyCIUrRMUJwehL3JwWuYvo9G X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Recompression, like writeback, makes a local copy of slot data (we need to decompress it anyway) before post-processing so we can unlock slot-entry once we have that local copy. Unlock the entry write-lock before recompression loop (secondary algorithms can be tried out one by one, in order of priority) and re-acquire it right after the loop. There is one more potentially costly operation recompress_slot() does - new zs_handle allocation, which can schedule(). Release the slot-entry write-lock before zsmalloc allocation and grab it again after the allocation. In both cases, once the slot-lock is re-acquired we examine slot's ZRAM_PP_SLOT flag to make sure that the slot has not been modified by a concurrent operation. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 80 +++++++++++++++++++---------------- 1 file changed, 44 insertions(+), 36 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 256439361367..cfbb3072ee9e 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1869,14 +1869,13 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, u64 *num_recomp_pages, u32 threshold, u32 prio, u32 prio_max) { - struct zcomp_strm *zstrm = NULL; + struct zcomp_strm *zstrm; unsigned long handle_old; unsigned long handle_new; unsigned int comp_len_old; unsigned int comp_len_new; unsigned int class_index_old; unsigned int class_index_new; - u32 num_recomps = 0; void *src, *dst; int ret; @@ -1903,6 +1902,13 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, zram_clear_flag(zram, index, ZRAM_IDLE); class_index_old = zs_lookup_class_index(zram->mem_pool, comp_len_old); + prio = max(prio, zram_get_priority(zram, index) + 1); + /* Slot data copied out - unlock its bucket */ + zram_slot_write_unlock(zram, index); + /* Recompression slots scan takes care of this, but just in case */ + if (prio >= prio_max) + return 0; + /* * Iterate the secondary comp algorithms list (in order of priority) * and try to recompress the page. @@ -1911,24 +1917,14 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, if (!zram->comps[prio]) continue; - /* - * Skip if the object is already re-compressed with a higher - * priority algorithm (or same algorithm). - */ - if (prio <= zram_get_priority(zram, index)) - continue; - - num_recomps++; zstrm = zcomp_stream_get(zram->comps[prio]); src = kmap_local_page(page); ret = zcomp_compress(zram->comps[prio], zstrm, src, &comp_len_new); kunmap_local(src); - if (ret) { - zcomp_stream_put(zram->comps[prio], zstrm); - return ret; - } + if (ret) + break; class_index_new = zs_lookup_class_index(zram->mem_pool, comp_len_new); @@ -1937,6 +1933,7 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, if (class_index_new >= class_index_old || (threshold && comp_len_new >= threshold)) { zcomp_stream_put(zram->comps[prio], zstrm); + zstrm = NULL; continue; } @@ -1944,14 +1941,7 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, break; } - /* - * We did not try to recompress, e.g. when we have only one - * secondary algorithm and the page is already recompressed - * using that algorithm - */ - if (!zstrm) - return 0; - + zram_slot_write_lock(zram, index); /* * Decrement the limit (if set) on pages we can recompress, even * when current recompression was unsuccessful or did not compress @@ -1961,37 +1951,55 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, if (*num_recomp_pages) *num_recomp_pages -= 1; - if (class_index_new >= class_index_old) { + /* Compression error */ + if (ret) { + zcomp_stream_put(zram->comps[prio], zstrm); + return ret; + } + + if (!zstrm) { /* * Secondary algorithms failed to re-compress the page - * in a way that would save memory, mark the object as - * incompressible so that we will not try to compress - * it again. + * in a way that would save memory. * - * We need to make sure that all secondary algorithms have - * failed, so we test if the number of recompressions matches - * the number of active secondary algorithms. + * Mark the object incompressible if the max-priority + * algorithm couldn't re-compress it. */ - if (num_recomps == zram->num_active_comps - 1) + if (prio < zram->num_active_comps) + return 0; + if (zram_test_flag(zram, index, ZRAM_PP_SLOT)) zram_set_flag(zram, index, ZRAM_INCOMPRESSIBLE); return 0; } - /* Successful recompression but above threshold */ - if (threshold && comp_len_new >= threshold) + /* Slot has been modified concurrently */ + if (!zram_test_flag(zram, index, ZRAM_PP_SLOT)) { + zcomp_stream_put(zram->comps[prio], zstrm); return 0; + } - /* - * If we cannot alloc memory for recompressed object then we bail out - * and simply keep the old (existing) object in zsmalloc. - */ + /* zsmalloc handle allocation can schedule, unlock slot's bucket */ + zram_slot_write_unlock(zram, index); handle_new = zs_malloc(zram->mem_pool, comp_len_new, GFP_NOIO | __GFP_HIGHMEM | __GFP_MOVABLE); + zram_slot_write_lock(zram, index); + + /* + * If we couldn't allocate memory for recompressed object then bail + * out and simply keep the old (existing) object in mempool. + */ if (IS_ERR_VALUE(handle_new)) { zcomp_stream_put(zram->comps[prio], zstrm); return PTR_ERR((void *)handle_new); } + /* Slot has been modified concurrently */ + if (!zram_test_flag(zram, index, ZRAM_PP_SLOT)) { + zcomp_stream_put(zram->comps[prio], zstrm); + zs_free(zram->mem_pool, handle_new); + return 0; + } + dst = zs_map_object(zram->mem_pool, handle_new, ZS_MM_WO); memcpy(dst, zstrm->buffer, comp_len_new); zcomp_stream_put(zram->comps[prio], zstrm); From patchwork Fri Jan 31 09:06:11 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13955138 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 43B05C0218F for ; Fri, 31 Jan 2025 09:07:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C106828028F; Fri, 31 Jan 2025 04:07:58 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BBEF92800EF; Fri, 31 Jan 2025 04:07:58 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A379128028F; Fri, 31 Jan 2025 04:07:58 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 864DA2800EF for ; Fri, 31 Jan 2025 04:07:58 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 410C9160968 for ; Fri, 31 Jan 2025 09:07:58 +0000 (UTC) X-FDA: 83067169836.04.BA1EDB4 Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) by imf20.hostedemail.com (Postfix) with ESMTP id 501261C0016 for ; Fri, 31 Jan 2025 09:07:56 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=irRfTbg8; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf20.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.177 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738314476; a=rsa-sha256; cv=none; b=yTLzZR8/0ldPwGmXd+AbBVUvcaYFpF76wflHmGwUvjRUKOytJjVXzF2SYSGuJwiE59lauI Ql9YFpC5b0LZadsVaLNfsr8AOKbgATEaKUiUjmvdUYPgFaVZx+X6QUIzJ6SGjmVA3DOCBN g/mrZZXEdgSOaM0JWKtrfXcOw5sF9yU= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=irRfTbg8; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf20.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.177 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738314476; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=aqDkbsViMBfIzGLM8hcQE+HJuYFQMLk+aSWGFS+XSh0=; b=neerG/vjr78g8LRiMarvC1j1sxe/Nr7u3JbCkXjacouXSPiVHMk33Cr67YObvApQKJCy1R 1Xy956NqkbLF44B0N8tuoV6NlcVOISTAHaQrqA1GLyed97AgR7jetikrtbZFGMjpGfBUYX iIkE/DW6a/eA2sX3L02/jdjsmiF0Rfg= Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-21634338cfdso41848585ad.2 for ; Fri, 31 Jan 2025 01:07:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738314475; x=1738919275; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=aqDkbsViMBfIzGLM8hcQE+HJuYFQMLk+aSWGFS+XSh0=; b=irRfTbg8UsQo/NlRGgz/WoClroBQr9cDaMjnno5lhq7K7QkuheEZe1srh7kbBRnM25 m1v6bU4VGtuG8YbtlVbkEd9zQHnqz9yPz/PpJsGidBOxA5njVwn8aQFIfEeXDr3BYG0G gEf/cWMoBBRQLEAniRK/4GKacyX5IXip8ilug= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738314475; x=1738919275; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=aqDkbsViMBfIzGLM8hcQE+HJuYFQMLk+aSWGFS+XSh0=; b=Bnf2ci6OeiegtqQz4eoeTeaZRAkooT1E0N/bOUNr5i5m2+nhpn+XiBWUFxH4BIXnwk gpqjefZAOS1n+rBdRgOS8RU2P0pwCxe0Uk/ni1V/NCoWQp3EEfqqJBlP+0enIsFRuo4z eUr0oEApH9TWaSuPwi/3siOQ79rtpFKaDw437Q6kc4L8XO9IUAg28DBJWyhcBQR+IqSG OpvFSMMQH8ymH4XzU11NX3CCAGSzpdK2i+7W/sY1yD2mbmBVrcumAKDdt8ZhzkY9Hv0B 8rBymt6dgRTBjW9OlJ4X3z97ZvxeBVSDdYmk29Pjp2IQERXW8KQoYVYlkDzXuR+1DrP+ Gg9Q== X-Forwarded-Encrypted: i=1; AJvYcCVtY+EDUJxbtRTyAbSxXQjiNO963PMIJs+fNir3eKdl8vDqCpgarTZUeijhpCa47y76KPi3mRqpNA==@kvack.org X-Gm-Message-State: AOJu0Yz7DUZHIAZuhmvSEDZDEOSDtaMIv/8aBL0O/RE9V/f87ZThf1UJ ISeO5mhzG6inJQb5qodtnv/yL4i8JdT+S7A756Qp8CO84ehv6J4pfnQ8304XDw== X-Gm-Gg: ASbGncsH4aTkb0FrR+ONRa6hfPlZKBRSMOO4+Pt7CN/015AlrBbx4hvYpQ21Ecg1qxi Po9JclqWV4K85t7xmM/VOxZkSc/IbLYuEnyh6fYQfM2kKTwbzXTPr74oDd7502l93zoQIQUBfEN 7MvuznBMy8fe45/8YpHblnH2ZuUV8aqKIofSeF6YdFC04nNPUBgvIQzlnqMcatvF7sTFs7qjzIr 8R2KqdxWcV+7cwUznwXF0xPkeXThGNUOaf/7i4vHBd6m96xZ5r3wLC8y2qxkZS6Aq4kDhBQgmIA 4jFgPfQ8kNAfrqxcbg== X-Google-Smtp-Source: AGHT+IGX15C7/nD8ive9k+yk6hUyhUaEscDdlRfPOcsZ6APSPwhqAsXH+BxwUfuwKmae5iWyxWBwog== X-Received: by 2002:a17:902:ebc2:b0:21b:d105:26b8 with SMTP id d9443c01a7336-21eddbf3366mr33372725ad.7.1738314475027; Fri, 31 Jan 2025 01:07:55 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:c752:be9d:3368:16fa]) by smtp.gmail.com with UTF8SMTPSA id 98e67ed59e1d1-2f83bfc0ddbsm5391902a91.45.2025.01.31.01.07.52 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 31 Jan 2025 01:07:54 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky , Yosry Ahmed Subject: [PATCHv4 12/17] zsmalloc: factor out pool locking helpers Date: Fri, 31 Jan 2025 18:06:11 +0900 Message-ID: <20250131090658.3386285-13-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.362.g079036d154-goog In-Reply-To: <20250131090658.3386285-1-senozhatsky@chromium.org> References: <20250131090658.3386285-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 501261C0016 X-Stat-Signature: 5485y6qu6tt5bogyxxtj69h5dqxfetq1 X-HE-Tag: 1738314476-706979 X-HE-Meta: U2FsdGVkX19CNiAGrgu6Dn58t46u23BOLQjziyTtdQ8x7593+If5IrEvTMsh73aUFo4Nn9pl06LI2eAHEmkG1f9i8HJsR4vZ6ZbRQt4241Wt7HNUaQektpEUbhCaU6z1ycbAxR8xEgBf92AQe0xAPL5+KBo6CgjrJtVOTOtQ1DMoD5ajN8dgW5E25F3K8/3fmiDbup7sF3LH/n7mk8+dsgKqs+CkXso8r2LW9OVEYwfrQWz0wu4AOUcoClOik37R09uy7i7Cz9bLlUSzH2a05lkahzIHOKb7rExWy5Q5EE4b/YIDj/VX5W48kAFq2DTFNYtW0CQcTtopJ/3ktU7Ti9jPa3VhQX+WvlACXp1fgD7QcFxNhCHEtBctUljyeWxUrA+58QJjc0TQecR1YzrTXwQEvvIZwV9+SzPGdmia31+9Hbah9SFXXB2yh3Cs5pWWQcw3iKTWejYWoo3JWVo1Zgx7WwOCwgL5b2FczUj6FYF0d4ydYeXrm+aQDqOVqYI7vrxTZXZWInlbJUfYyNXFxmCTUOIbXyLPk20GTYfV66lZjgpV+iwLyNS81VW9QBJF5QRvyyYYwAiVm2FUeN/cJ/RSi+dt8OeNxeq46WzwzXPkW1lQtvZTdf9xD2YWs80HBj4KV8DlT46ESSZcEekbiolWILXK+NM/T3BFHYU9/lXC7IeT88CQonalyE1zz3DD9KVaXVcFwfklJ4PgvFrpivKC6agmOs9iNlZqm9LxeqlaOwLjVg0xuBdbJ+lgG2QkjqP9/05WXix7H763m3esQm3oZnCyFN4IAX9rOH1Dv2h+nO4I8JPpYSIq1vCDADXCL8BRMgxVLaNo3gdj8hN+rsDFfPHfNJqkGpRiEo34R8Oaz93b+8bM7MC4dPhWhfu6bewzMBO395HflNCNmKepOerJf0vPcX7RKrzN5JuAvWxi+msqozSx/Qi/PXQG+oHRig79WjsQcCmFe06gcBF lWxkVWt7 +N2AxUxFLxx+YPIjNdxHtbDs5mz/a2j5pj+ev2CvsiM5+CL2Yf7L5+tLvnzexj86yz3Xf3S5jfScYkymphAf6EYHnc8KLCT/itn1rqd8+TSBe8jhhVc30xsvpGK9ehwfLTLhRy3Bhe6MlyYEAExohoS1OVPDepkZamIeT2akjxcPj70RtCtKY3+HOUYV1f2J6+J94Mfq0vG/BCrasRAlJuezyCT1hvLPPhmltd9ltevOVVqf28CaOsVMSKWNdxC2sBc6ovawT13sz77qW96ag/52lV2fhTQkpI7oPyc1DUQHs9CSEWQHmJrownw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: We currently have a mix of migrate_{read,write}_lock() helpers that lock zspages, but it's zs_pool that actually has a ->migrate_lock access to which is opene-coded. Factor out pool migrate locking into helpers, zspage migration locking API will be renamed to reduce confusion. It's worth mentioning that zsmalloc locks sync not only migration, but also compaction. Signed-off-by: Sergey Senozhatsky Cc: Yosry Ahmed --- mm/zsmalloc.c | 69 +++++++++++++++++++++++++++++++++++---------------- 1 file changed, 47 insertions(+), 22 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 817626a351f8..c129596ab960 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -18,7 +18,7 @@ /* * lock ordering: * page_lock - * pool->migrate_lock + * pool->lock * class->lock * zspage->lock */ @@ -224,10 +224,35 @@ struct zs_pool { struct work_struct free_work; #endif /* protect page/zspage migration */ - rwlock_t migrate_lock; + rwlock_t lock; atomic_t compaction_in_progress; }; +static void pool_write_unlock(struct zs_pool *pool) +{ + write_unlock(&pool->lock); +} + +static void pool_write_lock(struct zs_pool *pool) +{ + write_lock(&pool->lock); +} + +static void pool_read_unlock(struct zs_pool *pool) +{ + read_unlock(&pool->lock); +} + +static void pool_read_lock(struct zs_pool *pool) +{ + read_lock(&pool->lock); +} + +static bool pool_lock_is_contended(struct zs_pool *pool) +{ + return rwlock_is_contended(&pool->lock); +} + static inline void zpdesc_set_first(struct zpdesc *zpdesc) { SetPagePrivate(zpdesc_page(zpdesc)); @@ -290,7 +315,7 @@ static bool ZsHugePage(struct zspage *zspage) return zspage->huge; } -static void migrate_lock_init(struct zspage *zspage); +static void lock_init(struct zspage *zspage); static void migrate_read_lock(struct zspage *zspage); static void migrate_read_unlock(struct zspage *zspage); static void migrate_write_lock(struct zspage *zspage); @@ -992,7 +1017,7 @@ static struct zspage *alloc_zspage(struct zs_pool *pool, return NULL; zspage->magic = ZSPAGE_MAGIC; - migrate_lock_init(zspage); + lock_init(zspage); for (i = 0; i < class->pages_per_zspage; i++) { struct zpdesc *zpdesc; @@ -1206,7 +1231,7 @@ void *zs_map_object(struct zs_pool *pool, unsigned long handle, BUG_ON(in_interrupt()); /* It guarantees it can get zspage from handle safely */ - read_lock(&pool->migrate_lock); + pool_read_lock(pool); obj = handle_to_obj(handle); obj_to_location(obj, &zpdesc, &obj_idx); zspage = get_zspage(zpdesc); @@ -1218,7 +1243,7 @@ void *zs_map_object(struct zs_pool *pool, unsigned long handle, * which is smaller granularity. */ migrate_read_lock(zspage); - read_unlock(&pool->migrate_lock); + pool_read_unlock(pool); class = zspage_class(pool, zspage); off = offset_in_page(class->size * obj_idx); @@ -1450,16 +1475,16 @@ void zs_free(struct zs_pool *pool, unsigned long handle) return; /* - * The pool->migrate_lock protects the race with zpage's migration + * The pool->lock protects the race with zpage's migration * so it's safe to get the page from handle. */ - read_lock(&pool->migrate_lock); + pool_read_lock(pool); obj = handle_to_obj(handle); obj_to_zpdesc(obj, &f_zpdesc); zspage = get_zspage(f_zpdesc); class = zspage_class(pool, zspage); spin_lock(&class->lock); - read_unlock(&pool->migrate_lock); + pool_read_unlock(pool); class_stat_sub(class, ZS_OBJS_INUSE, 1); obj_free(class->size, obj); @@ -1703,7 +1728,7 @@ static void lock_zspage(struct zspage *zspage) } #endif /* CONFIG_COMPACTION */ -static void migrate_lock_init(struct zspage *zspage) +static void lock_init(struct zspage *zspage) { rwlock_init(&zspage->lock); } @@ -1793,10 +1818,10 @@ static int zs_page_migrate(struct page *newpage, struct page *page, pool = zspage->pool; /* - * The pool migrate_lock protects the race between zpage migration + * The pool lock protects the race between zpage migration * and zs_free. */ - write_lock(&pool->migrate_lock); + pool_write_lock(pool); class = zspage_class(pool, zspage); /* @@ -1833,7 +1858,7 @@ static int zs_page_migrate(struct page *newpage, struct page *page, * Since we complete the data copy and set up new zspage structure, * it's okay to release migration_lock. */ - write_unlock(&pool->migrate_lock); + pool_write_unlock(pool); spin_unlock(&class->lock); migrate_write_unlock(zspage); @@ -1956,7 +1981,7 @@ static unsigned long __zs_compact(struct zs_pool *pool, * protect the race between zpage migration and zs_free * as well as zpage allocation/free */ - write_lock(&pool->migrate_lock); + pool_write_lock(pool); spin_lock(&class->lock); while (zs_can_compact(class)) { int fg; @@ -1983,14 +2008,14 @@ static unsigned long __zs_compact(struct zs_pool *pool, src_zspage = NULL; if (get_fullness_group(class, dst_zspage) == ZS_INUSE_RATIO_100 - || rwlock_is_contended(&pool->migrate_lock)) { + || pool_lock_is_contended(pool)) { putback_zspage(class, dst_zspage); dst_zspage = NULL; spin_unlock(&class->lock); - write_unlock(&pool->migrate_lock); + pool_write_unlock(pool); cond_resched(); - write_lock(&pool->migrate_lock); + pool_write_lock(pool); spin_lock(&class->lock); } } @@ -2002,7 +2027,7 @@ static unsigned long __zs_compact(struct zs_pool *pool, putback_zspage(class, dst_zspage); spin_unlock(&class->lock); - write_unlock(&pool->migrate_lock); + pool_write_unlock(pool); return pages_freed; } @@ -2014,10 +2039,10 @@ unsigned long zs_compact(struct zs_pool *pool) unsigned long pages_freed = 0; /* - * Pool compaction is performed under pool->migrate_lock so it is basically + * Pool compaction is performed under pool->lock so it is basically * single-threaded. Having more than one thread in __zs_compact() - * will increase pool->migrate_lock contention, which will impact other - * zsmalloc operations that need pool->migrate_lock. + * will increase pool->lock contention, which will impact other + * zsmalloc operations that need pool->lock. */ if (atomic_xchg(&pool->compaction_in_progress, 1)) return 0; @@ -2139,7 +2164,7 @@ struct zs_pool *zs_create_pool(const char *name) return NULL; init_deferred_free(pool); - rwlock_init(&pool->migrate_lock); + rwlock_init(&pool->lock); atomic_set(&pool->compaction_in_progress, 0); pool->name = kstrdup(name, GFP_KERNEL); From patchwork Fri Jan 31 09:06:12 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13955139 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 356FDC0218D for ; Fri, 31 Jan 2025 09:08:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B991A280290; Fri, 31 Jan 2025 04:08:02 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B486F2800EF; Fri, 31 Jan 2025 04:08:02 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 99FA7280290; Fri, 31 Jan 2025 04:08:02 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 762CD2800EF for ; Fri, 31 Jan 2025 04:08:02 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 2975E80945 for ; Fri, 31 Jan 2025 09:08:02 +0000 (UTC) X-FDA: 83067170004.08.CAFB8AD Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) by imf26.hostedemail.com (Postfix) with ESMTP id 3927514000D for ; Fri, 31 Jan 2025 09:08:00 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=hkpZUgGo; spf=pass (imf26.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.179 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738314480; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IMJPEUQI7RZrpU8Uw6E+fMbBL3rBKdqOy7rMXGa3DVQ=; b=7ttbmv0/AL1oeQ/dCly+ig/dJImNfSb/EA9dijsaptrQkc9AYKcEdELYzbVDm6bZmEUngx gVqFC9uuuI/Ubk1prquMWfFRGo+Wduypau7BKicWzSG8jcEWoNFUmwdO3E11PKDKccqmtK Qv1JrBsOlM4kpE/xvoagOUN80qhw+XY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738314480; a=rsa-sha256; cv=none; b=BnWCgTgs95nwBwK2CAzhQ59M6OHvuoirBS2dbjJfdzfEKDqM7i4xKJGOlMrBbi3lOqeWjk khMqPOFf1CdZUbz6huBzUBInmafUVSaBxH6DYHJX1kZTjUBkr2tf9TtdsX3GI8GuVrCP/O oO8s1ngUQj2SEPE5oHzAxyHYCOFy+/o= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=hkpZUgGo; spf=pass (imf26.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.179 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org Received: by mail-pl1-f179.google.com with SMTP id d9443c01a7336-2163dc5155fso30739575ad.0 for ; Fri, 31 Jan 2025 01:07:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738314479; x=1738919279; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=IMJPEUQI7RZrpU8Uw6E+fMbBL3rBKdqOy7rMXGa3DVQ=; b=hkpZUgGogK16hR4UUA7ncqGEIw1+BAl5BxtGDJDzSqRJvLV0IH2xj6LAl4UXBFqjSC gZCt5WrrqlorGkY8e3reeBojdBlqI+KG8LRJ6L5jOYodv6oPxUm66NGdu0oVlcpNGbR4 2DyzGw/rxdPadaaSiq37kTXu5kLOdtw5kMjrg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738314479; x=1738919279; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=IMJPEUQI7RZrpU8Uw6E+fMbBL3rBKdqOy7rMXGa3DVQ=; b=shZzHw4ardfq/JHKTVCioKDkzuJOn9HUEIfy4+ToWX5DaZWNfnZ8RFgUbKrwHo0KBo 2yGVD5hx0CH9lO/xv8IHyZqE60wbDY4zW/+/QVzbtQwlSEF+X2g0bmfZu3QCwR4GwUfu BmJOxRh2iFayyK6lm3mjAI2fNDZmx7s/6cn4C754xn3Nrr5OyV1/6OxVpw8J34ip8/ec 0GemAjuBq8S7qlPHMswSGt3ADNzn01+sOqvac3O6dyqZuz13D/tT+hrwH1VSWLUpBr9y FQf7/yoVpfFybk8nf5IDeptZGBKcEhCyddRJOpzVANhHkVQDtWt6G8F47wyonk60tXhD i1jA== X-Forwarded-Encrypted: i=1; AJvYcCX78l4jYt2yDZ3V2ZQ2lvyCaCAni8D0d3PIFvLpxO8pYp5h0KDsrNd2J8nqScre0MIkkuoXGx2PHA==@kvack.org X-Gm-Message-State: AOJu0Yw5Sel3biOeKx8rY6tdNUJRDUBax9EolR7PkY7U1W2LqcWZV5Pm yFYq428WEsCrKs8eBy8daDkRCuH5Dah7P2lSOpwQL8w9l26PU71xxn369D1ABA== X-Gm-Gg: ASbGncvdxuJrzy/lJpSBSqQADcEI+XmZC5hdrht6VPW/DZsG7dYhbSktXR6LkZg2dHx sIooIY5rmNoq5uC0oRcGlsQWizhmW3Eq69Ei53euLnfg8qDG5jHMdAA7zuDdHecIWDWi+e0oNzC rO/GX3tF5dksMECb8upKipTZ2g6285/4DTVtMhcERe6Ie7YBH5df7GK+8D0mDcjZQv4tejuQhWv RrFV0r/zyoICCGoZxarmRH7i6rPqo5cP9TAqOF/JW8MRETG2TgpRd3eUmzoBp8nLo6x+6UlzFPl 9QgGgrUN3HD82XQNZw== X-Google-Smtp-Source: AGHT+IHhqjztMQY9nyoxNY57ZCZdxtWzFwxbZ4LRhwZuQGquqZr7M2SqvkrDWTSh2oSFv8q0NO6MeA== X-Received: by 2002:a17:902:ea06:b0:215:352c:af73 with SMTP id d9443c01a7336-21dd7c57eb0mr137738715ad.18.1738314478999; Fri, 31 Jan 2025 01:07:58 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:c752:be9d:3368:16fa]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-21de31f5a60sm25837715ad.61.2025.01.31.01.07.57 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 31 Jan 2025 01:07:58 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky , Yosry Ahmed Subject: [PATCHv4 13/17] zsmalloc: factor out size-class locking helpers Date: Fri, 31 Jan 2025 18:06:12 +0900 Message-ID: <20250131090658.3386285-14-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.362.g079036d154-goog In-Reply-To: <20250131090658.3386285-1-senozhatsky@chromium.org> References: <20250131090658.3386285-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Stat-Signature: ycbta3n6pa4k7i4594zze7h9r1g4huna X-Rspam-User: X-Rspamd-Queue-Id: 3927514000D X-Rspamd-Server: rspam03 X-HE-Tag: 1738314480-758529 X-HE-Meta: U2FsdGVkX1+ftu26S1pP2/ogM0FKner4poYvGqlUIgrg02ne1UM0ZT2DAIuBys67E0/VC1JcNuoOFb5mMLJkgpv3H/o9DiObawLMvJBcIZNQ8vPZm0P988xAfCSUc5FOjfH0ySmpyjl3DvWyE3MUwzJcPJ2nT0z3BnBEqguu1NERW4Vrujta3psTHhcZZ45f8IWl70c9IfKYay4g3xHr0l4VbqsrDlwYwmoDwpqe1fpBiA2D6sGuebGRkitcs5tHq1Tu5MXqt760g1uJBpINlwzXozEVzDtQ9AQf7ZZQ2ewoMJMWkKrWcxtrtoloJAi41KZJVMCnEiVQc/tfjhAIb81Ogh7tlfAytB6vPNKI2YsVozJRIzCkiwfxk039vR6BzpBOy8Xt2vpyRb3TPyovqVj4yPKpbmf5rd6h2r7e6WArsj5YrYYCXaCjQXxy9GRBOKFjDQ1Hk+mqK4vuSD9OyLpf9s4Dqf6C1j3BtJu2yLfe/V1O8tWp1+H4pYl9z2vUTbjNRtDeBUnLIGLOQflsUavyaGe3SDm8itH5Di3wjVbUnbJgL6tR4oiWR50LWwR/PCydyBLN5t8DLR0gfE5OIycN23H8jiM+n/ZLf8+6/RVs2/H6FAWy0L5NEXUtpIQL1o0yukpLmA4j9lFfgSn110fi0sJxL2Q+Tot1J4jenfwf2P7HSjESGYdA2OUYDc3xyARCMT2PhxUc/cCciihLq3IUn2hOH7XaZqcS7a3V8e9KoMZ2ueZDwjSWJCQblmIPdXnq/q5N2nEDuoQyj4b2nQued467xIhq1ZH1TmOEtBwzpDl/vwB77GtXHFV2k0Q0aXer0Xdax9LC326v4NWXQa6dvUEYUja8uyPai2JCBZ60VwmJpMI2XDRDO3IOqRdhyP7KYPdCa3pR0gd6t0j8ghvICP7PW5ezQfw8ajh2WeXTza0C5YCqcQD+Y0f7yjKDKtYeq/fIc87h5Kvg35x INlILLhE ybv4Zh4KsjMLS4MofJT3Ut+O4oRNQ5IkgBDNeqyAwBk6DwhtLxZmeGouh+xAZOVC/cDc43VpwuDEf8/kglSTjOXk/N/nVEhoBCrVNGddiLInxmEAe9cOVX3uMSnmPw8PMt8+vg3uDSTfZa1PAGXqgOqWvO0jP3F/rW5NOCcy4dalxCJMbVuuveGoEaIdZNGADQ/4GKQZycOBv66WmwLNR7ahFttQRL0jeixzuw7CvGWVf91B+I2mCH/u/1K9TDPKTr5NEEkMCjTV1AErrXHDt0WH48P1DpQb05bd44g6uwvCh9mWoSr77gJZINQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Move open-coded size-class locking to dedicated helpers. Signed-off-by: Sergey Senozhatsky Reviewed-by: Yosry Ahmed --- mm/zsmalloc.c | 47 ++++++++++++++++++++++++++++------------------- 1 file changed, 28 insertions(+), 19 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index c129596ab960..4b4c77bc08f9 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -253,6 +253,16 @@ static bool pool_lock_is_contended(struct zs_pool *pool) return rwlock_is_contended(&pool->lock); } +static void size_class_lock(struct size_class *class) +{ + spin_lock(&class->lock); +} + +static void size_class_unlock(struct size_class *class) +{ + spin_unlock(&class->lock); +} + static inline void zpdesc_set_first(struct zpdesc *zpdesc) { SetPagePrivate(zpdesc_page(zpdesc)); @@ -613,8 +623,7 @@ static int zs_stats_size_show(struct seq_file *s, void *v) if (class->index != i) continue; - spin_lock(&class->lock); - + size_class_lock(class); seq_printf(s, " %5u %5u ", i, class->size); for (fg = ZS_INUSE_RATIO_10; fg < NR_FULLNESS_GROUPS; fg++) { inuse_totals[fg] += class_stat_read(class, fg); @@ -624,7 +633,7 @@ static int zs_stats_size_show(struct seq_file *s, void *v) obj_allocated = class_stat_read(class, ZS_OBJS_ALLOCATED); obj_used = class_stat_read(class, ZS_OBJS_INUSE); freeable = zs_can_compact(class); - spin_unlock(&class->lock); + size_class_unlock(class); objs_per_zspage = class->objs_per_zspage; pages_used = obj_allocated / objs_per_zspage * @@ -1399,7 +1408,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) class = pool->size_class[get_size_class_index(size)]; /* class->lock effectively protects the zpage migration */ - spin_lock(&class->lock); + size_class_lock(class); zspage = find_get_zspage(class); if (likely(zspage)) { obj_malloc(pool, zspage, handle); @@ -1410,7 +1419,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) goto out; } - spin_unlock(&class->lock); + size_class_unlock(class); zspage = alloc_zspage(pool, class, gfp); if (!zspage) { @@ -1418,7 +1427,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) return (unsigned long)ERR_PTR(-ENOMEM); } - spin_lock(&class->lock); + size_class_lock(class); obj_malloc(pool, zspage, handle); newfg = get_fullness_group(class, zspage); insert_zspage(class, zspage, newfg); @@ -1429,7 +1438,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) /* We completely set up zspage so mark them as movable */ SetZsPageMovable(pool, zspage); out: - spin_unlock(&class->lock); + size_class_unlock(class); return handle; } @@ -1483,7 +1492,7 @@ void zs_free(struct zs_pool *pool, unsigned long handle) obj_to_zpdesc(obj, &f_zpdesc); zspage = get_zspage(f_zpdesc); class = zspage_class(pool, zspage); - spin_lock(&class->lock); + size_class_lock(class); pool_read_unlock(pool); class_stat_sub(class, ZS_OBJS_INUSE, 1); @@ -1493,7 +1502,7 @@ void zs_free(struct zs_pool *pool, unsigned long handle) if (fullness == ZS_INUSE_RATIO_0) free_zspage(pool, class, zspage); - spin_unlock(&class->lock); + size_class_unlock(class); cache_free_handle(pool, handle); } EXPORT_SYMBOL_GPL(zs_free); @@ -1827,7 +1836,7 @@ static int zs_page_migrate(struct page *newpage, struct page *page, /* * the class lock protects zpage alloc/free in the zspage. */ - spin_lock(&class->lock); + size_class_lock(class); /* the migrate_write_lock protects zpage access via zs_map_object */ migrate_write_lock(zspage); @@ -1859,7 +1868,7 @@ static int zs_page_migrate(struct page *newpage, struct page *page, * it's okay to release migration_lock. */ pool_write_unlock(pool); - spin_unlock(&class->lock); + size_class_unlock(class); migrate_write_unlock(zspage); zpdesc_get(newzpdesc); @@ -1903,10 +1912,10 @@ static void async_free_zspage(struct work_struct *work) if (class->index != i) continue; - spin_lock(&class->lock); + size_class_lock(class); list_splice_init(&class->fullness_list[ZS_INUSE_RATIO_0], &free_pages); - spin_unlock(&class->lock); + size_class_unlock(class); } list_for_each_entry_safe(zspage, tmp, &free_pages, list) { @@ -1914,10 +1923,10 @@ static void async_free_zspage(struct work_struct *work) lock_zspage(zspage); class = zspage_class(pool, zspage); - spin_lock(&class->lock); + size_class_lock(class); class_stat_sub(class, ZS_INUSE_RATIO_0, 1); __free_zspage(pool, class, zspage); - spin_unlock(&class->lock); + size_class_unlock(class); } }; @@ -1982,7 +1991,7 @@ static unsigned long __zs_compact(struct zs_pool *pool, * as well as zpage allocation/free */ pool_write_lock(pool); - spin_lock(&class->lock); + size_class_lock(class); while (zs_can_compact(class)) { int fg; @@ -2012,11 +2021,11 @@ static unsigned long __zs_compact(struct zs_pool *pool, putback_zspage(class, dst_zspage); dst_zspage = NULL; - spin_unlock(&class->lock); + size_class_unlock(class); pool_write_unlock(pool); cond_resched(); pool_write_lock(pool); - spin_lock(&class->lock); + size_class_lock(class); } } @@ -2026,7 +2035,7 @@ static unsigned long __zs_compact(struct zs_pool *pool, if (dst_zspage) putback_zspage(class, dst_zspage); - spin_unlock(&class->lock); + size_class_unlock(class); pool_write_unlock(pool); return pages_freed; From patchwork Fri Jan 31 09:06:13 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13955140 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B1516C0218D for ; Fri, 31 Jan 2025 09:08:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 447B2280293; Fri, 31 Jan 2025 04:08:07 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3F59D2800EF; Fri, 31 Jan 2025 04:08:07 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 26EF6280293; Fri, 31 Jan 2025 04:08:07 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 023EF2800EF for ; Fri, 31 Jan 2025 04:08:06 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id AB37C1C6FF8 for ; Fri, 31 Jan 2025 09:08:06 +0000 (UTC) X-FDA: 83067170172.25.B25F260 Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) by imf02.hostedemail.com (Postfix) with ESMTP id BE22580014 for ; Fri, 31 Jan 2025 09:08:04 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=Buqq49Sx; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf02.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.170 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738314484; a=rsa-sha256; cv=none; b=kSsy55psNa7cOT3tC0Gib2cUnbEIv4HZtmGQuFTxxoStCmu6DlFZ5iIzy5auHNOlJwNv1Z CC1S0TdMVEkgWyVTxp4DuqBlLObEwvpQDpDyLdvriu3LAkM8fK5ZTCvFU6g3di4t35liWx 7Z816HJbsZs4SVcFgqInaiT9urgL6io= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=Buqq49Sx; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf02.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.170 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738314484; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=K14zF8RGObITiXxNWEiw4I5DkmsIF3aHWMHaZssq3hI=; b=YQ74A7S6CjUAPwXsdpOuc5oEdl/aDr9BFif9AmCKBWsMRd5ifnIyUmxIalB8k/FXoYFDTu 4PIG9yqv79BFgBVzn7ybTBdx8IWYCp4O/X5W1VZssn6yOIATl5tk+rQg9Lm/TKxl8ogn1I 7j9JIOpjShHPvgZCU7SlwbVEogkXc08= Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-21bc1512a63so32839735ad.1 for ; Fri, 31 Jan 2025 01:08:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738314483; x=1738919283; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=K14zF8RGObITiXxNWEiw4I5DkmsIF3aHWMHaZssq3hI=; b=Buqq49Sx/QE6hjy6Spduc8z55ux6O1LBxhVGd7cDTiZeT8Um3XR9F2PxgJe7McyBn8 QcHZCiLUDgFeCMXtQVpUQy3pufSXduLAAZOgIWsvuZh2FaCIZb+BgZNTouPF8jf22aib F1XSf8bRYZoETuhfm1qP/TQU/che5lkVkCOIU= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738314483; x=1738919283; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=K14zF8RGObITiXxNWEiw4I5DkmsIF3aHWMHaZssq3hI=; b=hfHiRiWN+vI6U3G/Xm+hSIFSXHLnNWUrdId0aPIBp9kcNYdip/bQRAuuMTJsIHFUbp c7e/E6AMTVvokg+HQTlg9xNjSjda4Wftw+f102bmTWzAcT6OKLzQ9YJp9Iv+osPOw8ms cx7lU1UR5zmxb7n8HcQsGRK/UzZWCu36K4S+tO11zHJCZpn3E53sKr3539ANrfV16Rgz 4H3J9NH4J5ruVZlJlcU8Wj6JERtjyft/kDJroMNpQF70yaHh38KAR2jirx4eqoXyUezp tw3Xw8lYXWU2+51hAi0rdiAKXL1rmE5Zl2dZPN/E/IEnNmOTw3ojCXqFX/v7uIEy+t0q 1nbw== X-Forwarded-Encrypted: i=1; AJvYcCUO/n0Mn90rST7qmBidl8V4v40+yC1odhm2owS7SQ5o9bUCzBDlSWV9YdcIHHJqBD5JNwyB5AxdGw==@kvack.org X-Gm-Message-State: AOJu0Yyq4rBWUiA1uoadB2Ga1jbmjWOA6p+6F860j1WAcJu0XEHLXjRL z4/rXGy7yWy6PgIZyC4HjNvnftzbMQ7LxifiUqI2hNEL2+BwUykhMj1LGHpHmQ== X-Gm-Gg: ASbGnctejeEhkdle9gBJ5XZqFsKoxFzciZ0mftNQs9RdYqXqQBRVPNvER6KtzARGmwf 9UBr8dbleTQ0NNGJviGJWbWcYM+j4+iUMBVE35FhyUYLV9/apr7yus339QRu9cPyDAhBO6lCJiy bqp6tYOREO0jvBmKOLJYYnxL7GHXmereIhuZ/mSjf8CK1qhIXNVFp7/PVAfbX6rY6KG0HulM9mW E17Y08bJkPkV/n9/n02koxSNw72UYpVm2CQzIwjn2+5xRp+f3hB1faCz07sZfUDBG5aNjpzXkh6 tqNIws8Ce2ZmGDzS5Q== X-Google-Smtp-Source: AGHT+IFoGKrZNGGlmGxZx2UZTouqjuhCYFoIKMcMR38ulc4zYXbCr002OD1N6TYgHMj1w9Keyos0HA== X-Received: by 2002:a17:902:ecce:b0:211:3275:3fe with SMTP id d9443c01a7336-21dd7c65555mr158196745ad.17.1738314483547; Fri, 31 Jan 2025 01:08:03 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:c752:be9d:3368:16fa]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-21de31f80c2sm26197065ad.72.2025.01.31.01.08.01 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 31 Jan 2025 01:08:03 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky , Yosry Ahmed Subject: [PATCHv4 14/17] zsmalloc: make zspage lock preemptible Date: Fri, 31 Jan 2025 18:06:13 +0900 Message-ID: <20250131090658.3386285-15-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.362.g079036d154-goog In-Reply-To: <20250131090658.3386285-1-senozhatsky@chromium.org> References: <20250131090658.3386285-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: BE22580014 X-Stat-Signature: i891nm9kztbxxnpzob5k8kc7k5efzpcy X-HE-Tag: 1738314484-595646 X-HE-Meta: U2FsdGVkX19Cdhp/hufyU7kmSu3VYJdU0tML7sjn35AhBgXou2WLJakNAExAP81zZT0mFphcq8clwADDLZaG4T3IMHDnrxxkXxEf+bu3LXrfj3zoLEsIz501WBjhuVHGhZE3IAgkgP20/ANkSN02Facjuig6eDVDdnAbVonuN0wRxGEE7V2Sw5fW6RrQEO3w+tikeFapCVPL/mfGbOJkxVyU70k9IgJ6u7kOedV9MYOL5nePfFSFX2pQddG6R+od1SFERNJBeb1E0pnqwkXwR5J56rjQN8wjeGLqh3XSvw5wj4L08cmRvuVglElLZPzSn96LJsUBZtMFHWtXI8H3ZXWICoAK/ODZLQEnH5awzc3cdYtN6JEKA8Jgz1D4OOeiytd2FcWxlbc6dwW+0dHtGYPw1kLvoBhjEeHVXHylJSKMLLAsU2WDXurQ+wLSll++yaRDmgsO1iUjAGcGLET2rPG8Wvps2xxeo2SVAnwwh0F3OG8uVPkQsOV3giu2mV6y49D8IWaDHkAWhaBREQwHJpngZnNPba9WQ1cRHvXtRmeX1dt3PS4JCeBooaPu6i6RtqqBvFOjx5LM2L5Gl22UjO6dcdTVPeg2MFA7HGXJP33y10oeNrt2kA2RwBYGScq1FHEMkLcgRNjzPfZJHYWYefb3XUWS6YE8DEEf69gp0MlkQjAUMJztcSGmEG38ssprYwwBVMS5GjuLczqOcLtT6KALLyLW0tEWBEjIc17v86GyLhre5MElpcKlpsDjLz4c58Ml46PK4lhS6ChzQagV5sCEAy8WWPZvEEakGZZzBneD6RZKVGcVjOh1yu/yOm5pc1ochF5iw1MItd+WHmrb22iUfC0QKs8a3Uleoasrzr+Lq9IGbiRA9LnNWqE2n4HTMgK9DpA/1ehGxAboktqkD4c86G8G8mYmyvuxhmSVYAyORGCqtcBJu5SZ5zKLSmtYFpmZfEY/+nN3iFPlsPi XWiMrTRa CO9zg72wLOt6Eh66JCdF/0ShkVQ1nm+VA2h89lCeCgsa5F81WjSiJPZB3IivK3qgpP1CgbbsQ6xfdXStgMQH6/P+/APD+v02SRHYCxRenms3BUtiRoL08M2bf6hex81+w8MWPDbyU/zt4RxQDy1x5Fyul4v2h9+ZaGnwceqCfMK1Y2PIQQY1CSbD60vHRV6Hk/JLyn5xwjoOlSF/8eSPcAevCjwP7IGAMuLO2T8t29/U8ZOMSw8pzGaJ1IcdXmQCEMKw5hGCu7eEMezRIwaIrmBqCpmlpRXshO0+Gjy/XW+B1yx4+n3qM+2M4qw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Switch over from rwlock_t to a atomic_t variable that takes negative value when the page is under migration, or positive values when the page is used by zsmalloc users (object map, etc.) Using a rwsem per-zspage is a little too memory heavy, a simple atomic_t should suffice. zspage lock is a leaf lock for zs_map_object(), where it's read-acquired. Since this lock now permits preemption extra care needs to be taken when it is write-acquired - all writers grab it in atomic context, so they cannot spin and wait for (potentially preempted) reader to unlock zspage. There are only two writers at this moment - migration and compaction. In both cases we use write-try-lock and bail out if zspage is read locked. Writers, on the other hand, never get preempted, so readers can spin waiting for the writer to unlock zspage. With this we can implement a preemptible object mapping. Signed-off-by: Sergey Senozhatsky Cc: Yosry Ahmed --- mm/zsmalloc.c | 135 +++++++++++++++++++++++++++++++------------------- 1 file changed, 83 insertions(+), 52 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 4b4c77bc08f9..f5b5fe732e50 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -292,6 +292,9 @@ static inline void free_zpdesc(struct zpdesc *zpdesc) __free_page(page); } +#define ZS_PAGE_UNLOCKED 0 +#define ZS_PAGE_WRLOCKED -1 + struct zspage { struct { unsigned int huge:HUGE_BITS; @@ -304,7 +307,7 @@ struct zspage { struct zpdesc *first_zpdesc; struct list_head list; /* fullness list */ struct zs_pool *pool; - rwlock_t lock; + atomic_t lock; }; struct mapping_area { @@ -314,6 +317,59 @@ struct mapping_area { enum zs_mapmode vm_mm; /* mapping mode */ }; +static void zspage_lock_init(struct zspage *zspage) +{ + atomic_set(&zspage->lock, ZS_PAGE_UNLOCKED); +} + +/* + * zspage lock permits preemption on the reader-side (there can be multiple + * readers). Writers (exclusive zspage ownership), on the other hand, are + * always run in atomic context and cannot spin waiting for a (potentially + * preempted) reader to unlock zspage. This, basically, means that writers + * can only call write-try-lock and must bail out if it didn't succeed. + * + * At the same time, writers cannot reschedule under zspage write-lock, + * so readers can spin waiting for the writer to unlock zspage. + */ +static void zspage_read_lock(struct zspage *zspage) +{ + atomic_t *lock = &zspage->lock; + int old = atomic_read(lock); + + do { + if (old == ZS_PAGE_WRLOCKED) { + cpu_relax(); + old = atomic_read(lock); + continue; + } + } while (!atomic_try_cmpxchg(lock, &old, old + 1)); +} + +static void zspage_read_unlock(struct zspage *zspage) +{ + atomic_dec(&zspage->lock); +} + +static bool zspage_try_write_lock(struct zspage *zspage) +{ + atomic_t *lock = &zspage->lock; + int old = ZS_PAGE_UNLOCKED; + + preempt_disable(); + if (atomic_try_cmpxchg(lock, &old, ZS_PAGE_WRLOCKED)) + return true; + + preempt_enable(); + return false; +} + +static void zspage_write_unlock(struct zspage *zspage) +{ + atomic_set(&zspage->lock, ZS_PAGE_UNLOCKED); + preempt_enable(); +} + /* huge object: pages_per_zspage == 1 && maxobj_per_zspage == 1 */ static void SetZsHugePage(struct zspage *zspage) { @@ -325,12 +381,6 @@ static bool ZsHugePage(struct zspage *zspage) return zspage->huge; } -static void lock_init(struct zspage *zspage); -static void migrate_read_lock(struct zspage *zspage); -static void migrate_read_unlock(struct zspage *zspage); -static void migrate_write_lock(struct zspage *zspage); -static void migrate_write_unlock(struct zspage *zspage); - #ifdef CONFIG_COMPACTION static void kick_deferred_free(struct zs_pool *pool); static void init_deferred_free(struct zs_pool *pool); @@ -1026,7 +1076,7 @@ static struct zspage *alloc_zspage(struct zs_pool *pool, return NULL; zspage->magic = ZSPAGE_MAGIC; - lock_init(zspage); + zspage_lock_init(zspage); for (i = 0; i < class->pages_per_zspage; i++) { struct zpdesc *zpdesc; @@ -1251,7 +1301,7 @@ void *zs_map_object(struct zs_pool *pool, unsigned long handle, * zs_unmap_object API so delegate the locking from class to zspage * which is smaller granularity. */ - migrate_read_lock(zspage); + zspage_read_lock(zspage); pool_read_unlock(pool); class = zspage_class(pool, zspage); @@ -1311,7 +1361,7 @@ void zs_unmap_object(struct zs_pool *pool, unsigned long handle) } local_unlock(&zs_map_area.lock); - migrate_read_unlock(zspage); + zspage_read_unlock(zspage); } EXPORT_SYMBOL_GPL(zs_unmap_object); @@ -1705,18 +1755,18 @@ static void lock_zspage(struct zspage *zspage) /* * Pages we haven't locked yet can be migrated off the list while we're * trying to lock them, so we need to be careful and only attempt to - * lock each page under migrate_read_lock(). Otherwise, the page we lock + * lock each page under zspage_read_lock(). Otherwise, the page we lock * may no longer belong to the zspage. This means that we may wait for * the wrong page to unlock, so we must take a reference to the page - * prior to waiting for it to unlock outside migrate_read_lock(). + * prior to waiting for it to unlock outside zspage_read_lock(). */ while (1) { - migrate_read_lock(zspage); + zspage_read_lock(zspage); zpdesc = get_first_zpdesc(zspage); if (zpdesc_trylock(zpdesc)) break; zpdesc_get(zpdesc); - migrate_read_unlock(zspage); + zspage_read_unlock(zspage); zpdesc_wait_locked(zpdesc); zpdesc_put(zpdesc); } @@ -1727,41 +1777,16 @@ static void lock_zspage(struct zspage *zspage) curr_zpdesc = zpdesc; } else { zpdesc_get(zpdesc); - migrate_read_unlock(zspage); + zspage_read_unlock(zspage); zpdesc_wait_locked(zpdesc); zpdesc_put(zpdesc); - migrate_read_lock(zspage); + zspage_read_lock(zspage); } } - migrate_read_unlock(zspage); + zspage_read_unlock(zspage); } #endif /* CONFIG_COMPACTION */ -static void lock_init(struct zspage *zspage) -{ - rwlock_init(&zspage->lock); -} - -static void migrate_read_lock(struct zspage *zspage) __acquires(&zspage->lock) -{ - read_lock(&zspage->lock); -} - -static void migrate_read_unlock(struct zspage *zspage) __releases(&zspage->lock) -{ - read_unlock(&zspage->lock); -} - -static void migrate_write_lock(struct zspage *zspage) -{ - write_lock(&zspage->lock); -} - -static void migrate_write_unlock(struct zspage *zspage) -{ - write_unlock(&zspage->lock); -} - #ifdef CONFIG_COMPACTION static const struct movable_operations zsmalloc_mops; @@ -1803,7 +1828,7 @@ static bool zs_page_isolate(struct page *page, isolate_mode_t mode) } static int zs_page_migrate(struct page *newpage, struct page *page, - enum migrate_mode mode) + enum migrate_mode mode) { struct zs_pool *pool; struct size_class *class; @@ -1819,15 +1844,12 @@ static int zs_page_migrate(struct page *newpage, struct page *page, VM_BUG_ON_PAGE(!zpdesc_is_isolated(zpdesc), zpdesc_page(zpdesc)); - /* We're committed, tell the world that this is a Zsmalloc page. */ - __zpdesc_set_zsmalloc(newzpdesc); - /* The page is locked, so this pointer must remain valid */ zspage = get_zspage(zpdesc); pool = zspage->pool; /* - * The pool lock protects the race between zpage migration + * The pool->lock protects the race between zpage migration * and zs_free. */ pool_write_lock(pool); @@ -1837,8 +1859,15 @@ static int zs_page_migrate(struct page *newpage, struct page *page, * the class lock protects zpage alloc/free in the zspage. */ size_class_lock(class); - /* the migrate_write_lock protects zpage access via zs_map_object */ - migrate_write_lock(zspage); + /* the zspage write_lock protects zpage access via zs_map_object */ + if (!zspage_try_write_lock(zspage)) { + size_class_unlock(class); + pool_write_unlock(pool); + return -EINVAL; + } + + /* We're committed, tell the world that this is a Zsmalloc page. */ + __zpdesc_set_zsmalloc(newzpdesc); offset = get_first_obj_offset(zpdesc); s_addr = kmap_local_zpdesc(zpdesc); @@ -1869,7 +1898,7 @@ static int zs_page_migrate(struct page *newpage, struct page *page, */ pool_write_unlock(pool); size_class_unlock(class); - migrate_write_unlock(zspage); + zspage_write_unlock(zspage); zpdesc_get(newzpdesc); if (zpdesc_zone(newzpdesc) != zpdesc_zone(zpdesc)) { @@ -2005,9 +2034,11 @@ static unsigned long __zs_compact(struct zs_pool *pool, if (!src_zspage) break; - migrate_write_lock(src_zspage); + if (!zspage_try_write_lock(src_zspage)) + break; + migrate_zspage(pool, src_zspage, dst_zspage); - migrate_write_unlock(src_zspage); + zspage_write_unlock(src_zspage); fg = putback_zspage(class, src_zspage); if (fg == ZS_INUSE_RATIO_0) { From patchwork Fri Jan 31 09:06:14 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13955141 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DFDD9C0218D for ; Fri, 31 Jan 2025 09:08:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4BCE928029C; Fri, 31 Jan 2025 04:08:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 444CD280299; Fri, 31 Jan 2025 04:08:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2BFAC28029C; Fri, 31 Jan 2025 04:08:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 0C084280299 for ; Fri, 31 Jan 2025 04:08:11 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id B7E15A097A for ; Fri, 31 Jan 2025 09:08:10 +0000 (UTC) X-FDA: 83067170340.15.1217D4E Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) by imf05.hostedemail.com (Postfix) with ESMTP id C9364100005 for ; Fri, 31 Jan 2025 09:08:08 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=LT7dUjPK; spf=pass (imf05.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.172 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738314488; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rSDpkbxqSvfVMXf6MtRLSP9PYlXCwlCf1VxOLTEJdKQ=; b=1WyaafqtY6IPQpKJrCkWSOPtoUqT7ONRNmGQi7iw6sqkQHNkPPp+8yZbhkqD8KumOQa6Zr A7FeoYA8p/YSnG07utQlCHL5MY7CpmMuuu99d2T5OqXTqYf/IQCcdKsO1joXTwdiP9YFUn Gn1mzqccPjZEZSsHOEr5HWNQnyi4wXE= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=LT7dUjPK; spf=pass (imf05.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.172 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738314488; a=rsa-sha256; cv=none; b=mN+120rYOvEc70jjoDj+wuTVNhyRUic8yYHwZaPgcwjveREiHSrgWr/DQCCc4pKCH/NeXK 7bG30DgNt7d8nn/RgArDYA2jmK4UOtxt3t/K5f8lODSWp4z55UOG1FiZbCjIr7KxDrMncq 3WP2VRW+gB9FYLOHcLBHLI0tOkS5tKU= Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-21669fd5c7cso29292415ad.3 for ; Fri, 31 Jan 2025 01:08:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738314488; x=1738919288; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=rSDpkbxqSvfVMXf6MtRLSP9PYlXCwlCf1VxOLTEJdKQ=; b=LT7dUjPKAs0hnAAmelBIRxm5tGdYaVoQjHE4XAoXZDKqwsmw9WA8Y8ixhL6W945tC1 Ci/jPI78quTXvnkF1m5fWU9vbAp1aJW7vOd7TNbn9QIeUGpX+EGbJ6FxpqgHjhML4TPo PiM1D9qkoJWMlsx5ZIdqzlF3mlktnwZJDIFPA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738314488; x=1738919288; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rSDpkbxqSvfVMXf6MtRLSP9PYlXCwlCf1VxOLTEJdKQ=; b=cfCz0E5zcQ7MB+8MiE48TOfITz4ShoVYYMwmEipLJHdnTkloBLKXNB0SC9hEh7fsV9 wmsXgB4SqFUweNIV8ap9IEN8tlVGduLrzJZOIQgevX3Duh7hxEFVUocgmKw8UPs74kB9 4krbrZ0F3eQgx0ou0igxirlRApMDRnZeLVZwDN4aTvPcwWJ8b5boHyi4jzXlO0/5cmOm sO3NYN4RVpUDhgAJ9a0B5iPtYfncEUlkZU/XL463daMn20HmllQ91cI3E3SI+Ys+bffA fYKyA2Jp3TscaKjFvDi/cq2gRi5zjVHUdtJRwU19Rc5aOY1HA4znb8o86txXyUKc2h/v WY1A== X-Forwarded-Encrypted: i=1; AJvYcCWEX8MzsxcxLXOt+cmuNo75FQN+N4GJVOOcVD7R19jkCFji/OF0HLbOaaF9FUGkWUp40wO9GY/WWA==@kvack.org X-Gm-Message-State: AOJu0YwrMF1V/wZSEVXhy+3I6iCsNqEdBF5v48ldSNkbNzuhtIdZnn1L XPvxQwaefhwYyhVbWETEAwOsAWIc2VUqxoRUp6SrMnwZjUpq15L8y/fhGAkJdQ== X-Gm-Gg: ASbGncvOGzHtjmafKnlGdXCUyWj6iEoFaxSOIGBmXQ5lu0qGerkgp1eiyaR/o9YrbhC RoyDQQdFXfFJKHmEARvxOE5s0UmQaWtCu/ayX6vqJUQWxLdCzcsovBfotc/bS1+9InhNFrJFOav casAVgIhqJD461a/+sXAuosW7DT05ObcM1UuuS/UvAelXOi5/3m5BPeKnkF8JNEyM1NZaBK2Zv5 8e9qGIWZwMRbkNPzB33QvXkkr++eKQxJFAZou1CyqarK0AisQr3moN8rbyHlo6P23IaDDCJfyce 6NHglX680P2IyE6X1A== X-Google-Smtp-Source: AGHT+IFbB/ylqlS/xqv7OLfNnPKv/KBkJRu1pTXCHuKZ+Ljs5D2OLYSsCuLflKkc6dvQm0HS88PAuQ== X-Received: by 2002:a17:903:2f8e:b0:215:b75f:a1cb with SMTP id d9443c01a7336-21dd9fa74bcmr137340835ad.9.1738314487509; Fri, 31 Jan 2025 01:08:07 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:c752:be9d:3368:16fa]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-21de32f44a9sm26237855ad.143.2025.01.31.01.08.05 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 31 Jan 2025 01:08:07 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky , Yosry Ahmed Subject: [PATCHv4 15/17] zsmalloc: introduce new object mapping API Date: Fri, 31 Jan 2025 18:06:14 +0900 Message-ID: <20250131090658.3386285-16-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.362.g079036d154-goog In-Reply-To: <20250131090658.3386285-1-senozhatsky@chromium.org> References: <20250131090658.3386285-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: C9364100005 X-Stat-Signature: 8tthfwuj9oqipczf9wxus585ma7h9qmi X-Rspam-User: X-HE-Tag: 1738314488-424332 X-HE-Meta: U2FsdGVkX1/Sj8lcBNPCigApoRRpyBYXo6vzGFIKzPFDDct2qXo7aovNV6fsMvbs6oATWWBXK3HrOSCaJCLYEhltBs+PG4eUBhSvZ636ps/RecXammYw22EYWPjQe+Oi3SfpSeOCv5YfMWNsYwZnuHK0lsb/cHa/64YxZZjt58mL0IcWN13oReg38a2Mgft89K6QBJlABS9HvwZOzy9TBzw1MlDpPxCyhUccjemy7DHf9hI4JQLVtxxEWvQ3G/MFhKf04gnaXAHlAjnou/V0HnSnQ71nGbd/CYg81mG4TsQDB018UTucf9NNrqbb1Z1P8JL5ISYgni2MF8AFmhCxJjl/td1IP2y23btHDZTB+k/sNLO0mrUL8t7tjdstoi0KLHU6IJqDa0OnJ635XDIQ1YkGWtWKvshHH4cgMl0BO6Q1VKr7i9ccCzL82bkAEcoxVcdSXf6AvWV4fy/R+j+Vs6/4K0SjQnMMRKiq+tLf9N8fIxjjU1UTY7mq91qc4CXPWLW1M07ZbobCNlx+v/kgx97zlRLcHKD66LBjJxGVn2NSTikLETZLLd90hTMLPfMj1OPTHrXYYTRbhkqlB2UfsR0t/zDWpDEHBRYp4SZpusbpl3Vzj33gcfnudjxpDb5SfzG0N+ZgXK4gaOsDixW8Sk1I31vFY7BmhtyX2Z5jqQYSmsIp6CVqz9DzaHfiOWbF1VGVq420/Roxn2I5xNLVHI1wvsMFo9UjrUPpL2jnaAQBM377bCtktl+1th9kZxhYkAqRYzT0L1UrqC/zevk3Uh8XOJsZxXcBpWYWw++1wXeHIZ5Gp02NFEtwZ/Swiy+NJ76yO+jIZqS03VYzEUhjSeEI9BvjsIRTjMVs5HDO8cc3PM1Z1J2QYb/fknEA2tc1RsDvIOS166hzc/Y/elF9rpoSqrxpa6lrDnq8aWsHCsoNwqv4ivGBNpln7nS0dMBw/IOSR+Bhn2KEuX3XZm5 9GyzYJ7m 5hx7cruAaV8og0GAo0VWai2wj3ubsFHlv1ikJUVRZoMWegvUndEQqYEFc5BxnQTsPWXh50TgtrECFlQcG/dB1MePrERp1Y/iw9JlHBUCILNzilABBoquQOsLOlgA8GVgtvtbrj8CnWs8AiuyY41uDU0I+L4FHpwUo5TInxe6+XEzF1HCP2jYlrTk0UfTk9YXrUtpEikrPFDLGHUbg0G/yR+rEO58rvvNMRvSPVqf36TVbPE99+GUgSbFBwNIW7J42/928NbebzDpPKkOreUfoSoDdG2s4gg8+ZN6xppD2VptOGFFRPZSzmco1vg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Current object mapping API is a little cumbersome. First, it's inconsistent, sometimes it returns with page-faults disabled and sometimes with page-faults enabled. Second, and most importantly, it enforces atomicity restrictions on its users. zs_map_object() has to return a liner object address which is not always possible because some objects span multiple physical (non-contiguous) pages. For such objects zsmalloc uses a per-CPU buffer to which object's data is copied before a pointer to that per-CPU buffer is returned back to the caller. This leads to another, final, issue - extra memcpy(). Since the caller gets a pointer to per-CPU buffer it can memcpy() data only to that buffer, and during zs_unmap_object() zsmalloc will memcpy() from that per-CPU buffer to physical pages that object in question spans across. New API splits functions by access mode: - zs_obj_read_begin(handle, local_copy) Returns a pointer to handle memory. For objects that span two physical pages a local_copy buffer is used to store object's data before the address is returned to the caller. Otherwise the object's page is kmap_local mapped directly. - zs_obj_read_end(handle, buf) Unmaps the page if it was kmap_local mapped by zs_obj_read_begin(). - zs_obj_write(handle, buf, len) Copies len-bytes from compression buffer to handle memory (takes care of objects that span two pages). This does not need any additional (e.g. per-CPU) buffers and writes the data directly to zsmalloc pool pages. The old API will stay around until the remaining users switch to the new one. After that we'll also remove zsmalloc per-CPU buffer and CPU hotplug handling. Signed-off-by: Sergey Senozhatsky Reviewed-by: Yosry Ahmed --- include/linux/zsmalloc.h | 8 +++ mm/zsmalloc.c | 129 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 137 insertions(+) diff --git a/include/linux/zsmalloc.h b/include/linux/zsmalloc.h index a48cd0ffe57d..7d70983cf398 100644 --- a/include/linux/zsmalloc.h +++ b/include/linux/zsmalloc.h @@ -58,4 +58,12 @@ unsigned long zs_compact(struct zs_pool *pool); unsigned int zs_lookup_class_index(struct zs_pool *pool, unsigned int size); void zs_pool_stats(struct zs_pool *pool, struct zs_pool_stats *stats); + +void *zs_obj_read_begin(struct zs_pool *pool, unsigned long handle, + void *local_copy); +void zs_obj_read_end(struct zs_pool *pool, unsigned long handle, + void *handle_mem); +void zs_obj_write(struct zs_pool *pool, unsigned long handle, + void *handle_mem, size_t mem_len); + #endif diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index f5b5fe732e50..f9d840f77b18 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -1365,6 +1365,135 @@ void zs_unmap_object(struct zs_pool *pool, unsigned long handle) } EXPORT_SYMBOL_GPL(zs_unmap_object); +void *zs_obj_read_begin(struct zs_pool *pool, unsigned long handle, + void *local_copy) +{ + struct zspage *zspage; + struct zpdesc *zpdesc; + unsigned long obj, off; + unsigned int obj_idx; + struct size_class *class; + void *addr; + + WARN_ON(in_interrupt()); + + /* Guarantee we can get zspage from handle safely */ + pool_read_lock(pool); + obj = handle_to_obj(handle); + obj_to_location(obj, &zpdesc, &obj_idx); + zspage = get_zspage(zpdesc); + + /* Make sure migration doesn't move any pages in this zspage */ + zspage_read_lock(zspage); + pool_read_unlock(pool); + + class = zspage_class(pool, zspage); + off = offset_in_page(class->size * obj_idx); + + if (off + class->size <= PAGE_SIZE) { + /* this object is contained entirely within a page */ + addr = kmap_local_zpdesc(zpdesc); + addr += off; + } else { + size_t sizes[2]; + + /* this object spans two pages */ + sizes[0] = PAGE_SIZE - off; + sizes[1] = class->size - sizes[0]; + addr = local_copy; + + memcpy_from_page(addr, zpdesc_page(zpdesc), + off, sizes[0]); + zpdesc = get_next_zpdesc(zpdesc); + memcpy_from_page(addr + sizes[0], + zpdesc_page(zpdesc), + 0, sizes[1]); + } + + if (!ZsHugePage(zspage)) + addr += ZS_HANDLE_SIZE; + + return addr; +} +EXPORT_SYMBOL_GPL(zs_obj_read_begin); + +void zs_obj_read_end(struct zs_pool *pool, unsigned long handle, + void *handle_mem) +{ + struct zspage *zspage; + struct zpdesc *zpdesc; + unsigned long obj, off; + unsigned int obj_idx; + struct size_class *class; + + obj = handle_to_obj(handle); + obj_to_location(obj, &zpdesc, &obj_idx); + zspage = get_zspage(zpdesc); + class = zspage_class(pool, zspage); + off = offset_in_page(class->size * obj_idx); + + if (off + class->size <= PAGE_SIZE) { + if (!ZsHugePage(zspage)) + off += ZS_HANDLE_SIZE; + handle_mem -= off; + kunmap_local(handle_mem); + } + + zspage_read_unlock(zspage); +} +EXPORT_SYMBOL_GPL(zs_obj_read_end); + +void zs_obj_write(struct zs_pool *pool, unsigned long handle, + void *handle_mem, size_t mem_len) +{ + struct zspage *zspage; + struct zpdesc *zpdesc; + unsigned long obj, off; + unsigned int obj_idx; + struct size_class *class; + + WARN_ON(in_interrupt()); + + /* Guarantee we can get zspage from handle safely */ + pool_read_lock(pool); + obj = handle_to_obj(handle); + obj_to_location(obj, &zpdesc, &obj_idx); + zspage = get_zspage(zpdesc); + + /* Make sure migration doesn't move any pages in this zspage */ + zspage_read_lock(zspage); + pool_read_unlock(pool); + + class = zspage_class(pool, zspage); + off = offset_in_page(class->size * obj_idx); + + if (off + class->size <= PAGE_SIZE) { + /* this object is contained entirely within a page */ + void *dst = kmap_local_zpdesc(zpdesc); + + if (!ZsHugePage(zspage)) + off += ZS_HANDLE_SIZE; + memcpy(dst + off, handle_mem, mem_len); + kunmap_local(dst); + } else { + /* this object spans two pages */ + size_t sizes[2]; + + off += ZS_HANDLE_SIZE; + sizes[0] = PAGE_SIZE - off; + sizes[1] = mem_len - sizes[0]; + + memcpy_to_page(zpdesc_page(zpdesc), off, + handle_mem, sizes[0]); + zpdesc = get_next_zpdesc(zpdesc); + memcpy_to_page(zpdesc_page(zpdesc), 0, + handle_mem + sizes[0], sizes[1]); + } + + zspage_read_unlock(zspage); +} +EXPORT_SYMBOL_GPL(zs_obj_write); + /** * zs_huge_class_size() - Returns the size (in bytes) of the first huge * zsmalloc &size_class. From patchwork Fri Jan 31 09:06:15 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13955142 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 74242C02193 for ; Fri, 31 Jan 2025 09:08:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0C40B2802A2; Fri, 31 Jan 2025 04:08:15 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 074CC280299; Fri, 31 Jan 2025 04:08:15 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E57942802A2; Fri, 31 Jan 2025 04:08:14 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id C35A4280299 for ; Fri, 31 Jan 2025 04:08:14 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 6FE388098C for ; Fri, 31 Jan 2025 09:08:14 +0000 (UTC) X-FDA: 83067170508.21.BB321D8 Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) by imf18.hostedemail.com (Postfix) with ESMTP id 8BBE61C0014 for ; Fri, 31 Jan 2025 09:08:12 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=AvOVwnVZ; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf18.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.173 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738314492; a=rsa-sha256; cv=none; b=UEt3lU77mkCwp4S7j+Y1k9EkeOEbppBFegY5LQXIYGm1pqeuVaGTwM/MwYpegtS8+HduyT Siel8Eh6PKDAk4PwJnFZ1NZ9eKOGy9KFwix0YQZGwrw3EDK4ksIiRHy3PC2lcSHzFyIF0Y k7IYMIu/2IbIsoM36mxduqAXh9xwGlM= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=AvOVwnVZ; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf18.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.173 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738314492; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YsBQiXFOiDXEVopox/DL4fBI7MjKngkv74+OC5sfCK8=; b=AZWJxnm/p7e6loIkA5JQkomNgn2/DlzCzEqX6KqVKwXmKCs71Hq3ucXJ5RWV1WOwAiKuuS +6UD1dzekMPu/gkOAS+fT3fdvh3c6Gls4Kcpv5Fx2IsTwSSz+VMVIeLtBqX2hTS2lS0OY0 CnMyuyvnK3KIPkUeLwiDfrZnkwTIGCk= Received: by mail-pl1-f173.google.com with SMTP id d9443c01a7336-2164b1f05caso30026115ad.3 for ; Fri, 31 Jan 2025 01:08:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738314491; x=1738919291; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=YsBQiXFOiDXEVopox/DL4fBI7MjKngkv74+OC5sfCK8=; b=AvOVwnVZsiCbKZPp4vnAXQs3Aq1hvlVla79J0Mc6csu8nZX4WO5zcEBZUE30g0rEeP jULULugHlgMHmPtIUvGvXe+Y4yLKulN/5y2PFqE1U2OcBBn5hA+BN5ybbpnzskNat8nn 8osbSkHmZq5aWL5MppL6JrkNo7S9BeGpTLQnk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738314491; x=1738919291; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YsBQiXFOiDXEVopox/DL4fBI7MjKngkv74+OC5sfCK8=; b=vb2iUSnSnc5iPFSEeRYL1KoPYxa3gFwQnVzR3/SwPRWuVYY8x9pvN85Rvkp8NSL3T1 67dmzAfUPnQMISXUpkedjAg3jfis28zTMYwkxDYvR/YARv+wzSNThl5O01TRpye8x4Gp Kuweh/sU9XmIs47dlVfwXizMdudNSI8sqynrwAQhLTbWYXNQj1bBAsfzMlpW42L+Mv6E bjxlosr63brkBvLan+PaJvVM1gviMPqJqBvidm2CoDtHMtqoerMPHVsOGh2aT/rtbO+K IB5bgg/ida0Ktv9tmHbAM7BPXhU+DLpDDb8w7f4hxiDag5owc7oya3lqGUBdiNgapI1w qZLQ== X-Forwarded-Encrypted: i=1; AJvYcCWvlfB45RK05zxZYKk4cR+5B7FL6bmCrp6XvOlDfFFseJcxPKPvcgURG7lygF5txxtG+sJVI7Irvw==@kvack.org X-Gm-Message-State: AOJu0YyqhuJ7v9ErN6RmO+CPMfbNqdYdm9We6LmLiUldn9eVS2oUgzLv o94D27/9r93pxdjSpU/feNmZ/tyxPF6tNjqADdVVVa+A/rIqMqc+h8tNVhjuNGFF028/BlXbYBo = X-Gm-Gg: ASbGncup/DcVnA4nYwExEb03ctTYFFRtwaOMgvaQsZYB+CQTOxZYSUs7JdjtzDnZxlg TimLzlVqSDCY/WpOpKpeFSKo1okF7TNCqvc808DARqF2XyjycAYBPRVj5FInamisH3irN4hfbJ/ eyqM3+K0n3Bo2apMAVGJs93+vnxL4azu6uS759iProDfwkNxAuBiKR2WADCSZ5dXEh2vZVWxsHs Q722ECaOUm4vNIRk6gLnZ/Qn7ulTkANtc6cahj+eVTnGAgYyMgsTDzYYXlhFYV4eW0SWSIuBMSY UGZwZJs0k53hKTOvUw== X-Google-Smtp-Source: AGHT+IENlfyVLhUqijJuU7j/zhaZph0w7819UX7xL0oXUri1pMuVzz1XrUZpnmESTer0aRPTpzNpUQ== X-Received: by 2002:a17:902:dacd:b0:216:50fb:5dfc with SMTP id d9443c01a7336-21dd7c3cddemr135410395ad.9.1738314491281; Fri, 31 Jan 2025 01:08:11 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:c752:be9d:3368:16fa]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-21de33032dbsm25983505ad.191.2025.01.31.01.08.09 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 31 Jan 2025 01:08:10 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv4 16/17] zram: switch to new zsmalloc object mapping API Date: Fri, 31 Jan 2025 18:06:15 +0900 Message-ID: <20250131090658.3386285-17-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.362.g079036d154-goog In-Reply-To: <20250131090658.3386285-1-senozhatsky@chromium.org> References: <20250131090658.3386285-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 8BBE61C0014 X-Stat-Signature: 9nu94f3q6kei1pa3adq9r5co5jxjibg3 X-HE-Tag: 1738314492-237157 X-HE-Meta: U2FsdGVkX188qZPVN8ALxP8l5p52Yrs2sOffies52I+oDs4XKiTVLcD4bwwi9BRqj1Y14C2IweT1BzSYil5aqdR7wo3Cp/5Fo9Vm9hdBRsFX7Hx7ZwuUZjfgbESdxf/ZO2ovIpOMm3WYP5ez9nzBup8wFZiqArJ1ZqpdveUrvzEy5zzjIZOEVRnoF0cDZ6gRUSgtoYrbTCPgnBtYi5/SLCSSpCiQFhRrZgNAtlBDsUhEgz6gmaRtiRyrp7LgEdgqHEVZ/44KME/35a13wA8IGdle+yYAStey6G6247DyLrKmPWvi0zSe9bdxQHimoZD1mCiK0O1lL7yQHgP2/jGDBrOWc7rJUuknAbv50jmDwHDyNWX6cxSCp3aOItGiCop1mNlxvovID/Fapb4os+gep3UHe1fxoGG+SVIvrA7n0FEijiXSQ+UYkKzXw8FlraJepmoX9jDeD5wiRd+7sm03aKl/M6ZXw6Y6X7gyuYxjn8vNlVN0b2Pxe7A3bt07n9I/qOmR7QAih8550WbiGEgpR9mQNZMOfc+v4h0V/zCMl2OI5eteOiHzhDeP6WxbwMTVwPNrsdMGAaXk1rV8m9hwUcMiCnr2SUZ2nZn9OC5D3IcM6GsDV6LjocpA41/OHbCbVSZlXNeDUFfmuChai2JO9qPHv2gp017XxYTzlISyWrCdBpcwoZ99Kk5DWKr10jIQ83fq9Z9pl1hJMLclOurw9WCgAmnhZuz1Br7L95c0F5Sz9EsKGsqggm2ztTmd1Sxjnp8tSaZ2feSmtWj89ixeAgHpS1nsb9KbnrTKFZUnQZVGIfPU1UZbBDJlcuJLoHoScJ1W4CFve7Rs6vHwToy4t1eW8HL7BOvGKqXFehtxvt8y1bqt7LBrqHER5/M7aBVvbGrn6LOk/r2/6FTK6/YiDHWoYAVPcKeslXtQhhs9KOCj1OsqrBYKi9DCLIXTFk8lHH6LC27/73C7+FVt7ke wB5bwA2M UFNk/LMM8c9W9BK9Hedn/xaccJY0Ex7HhccQGVmmJLBiTyXq8MqMfN2L34vepgzpLkrJ1sT9UD5LWzSC9KZp+NyNwxD0UO99bq5AHwej+0EGd+Rku/yh6frCF6rxAgRu1NP+EKbpQoUvlZqnvG8u+4L2xH4QZ7nJ0WmUacCn9ObeVKYEjGeeDVMCydFnkjFxnSmwS9whz9GcHtb98unLdcSEi+zgdE5jxTojLtOwxjAL6V1KkOK30Djj9QtDYLI0kFXT5UihK3uPwm04chcAPQfEwkyhorQz5tgfmMtJXgZEizqPBGLTcIawkCQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Use new read/write zsmalloc object API. For cases when RO mapped object spans two physical pages (requires temp buffer) compression streams now carry around one extra physical page. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zcomp.c | 4 +++- drivers/block/zram/zcomp.h | 2 ++ drivers/block/zram/zram_drv.c | 28 ++++++++++------------------ 3 files changed, 15 insertions(+), 19 deletions(-) diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c index efd5919808d9..675f2a51ad5f 100644 --- a/drivers/block/zram/zcomp.c +++ b/drivers/block/zram/zcomp.c @@ -45,6 +45,7 @@ static const struct zcomp_ops *backends[] = { static void zcomp_strm_free(struct zcomp *comp, struct zcomp_strm *strm) { comp->ops->destroy_ctx(&strm->ctx); + vfree(strm->local_copy); vfree(strm->buffer); kfree(strm); } @@ -66,12 +67,13 @@ static struct zcomp_strm *zcomp_strm_alloc(struct zcomp *comp) return NULL; } + strm->local_copy = vzalloc(PAGE_SIZE); /* * allocate 2 pages. 1 for compressed data, plus 1 extra in case if * compressed data is larger than the original one. */ strm->buffer = vzalloc(2 * PAGE_SIZE); - if (!strm->buffer) { + if (!strm->buffer || !strm->local_copy) { zcomp_strm_free(comp, strm); return NULL; } diff --git a/drivers/block/zram/zcomp.h b/drivers/block/zram/zcomp.h index 62330829db3f..9683d4aa822d 100644 --- a/drivers/block/zram/zcomp.h +++ b/drivers/block/zram/zcomp.h @@ -34,6 +34,8 @@ struct zcomp_strm { struct list_head entry; /* compression buffer */ void *buffer; + /* local copy of handle memory */ + void *local_copy; struct zcomp_ctx ctx; }; diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index cfbb3072ee9e..f85502ae7dce 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1553,11 +1553,11 @@ static int read_incompressible_page(struct zram *zram, struct page *page, void *src, *dst; handle = zram_get_handle(zram, index); - src = zs_map_object(zram->mem_pool, handle, ZS_MM_RO); + src = zs_obj_read_begin(zram->mem_pool, handle, NULL); dst = kmap_local_page(page); copy_page(dst, src); kunmap_local(dst); - zs_unmap_object(zram->mem_pool, handle); + zs_obj_read_end(zram->mem_pool, handle, src); return 0; } @@ -1575,11 +1575,11 @@ static int read_compressed_page(struct zram *zram, struct page *page, u32 index) prio = zram_get_priority(zram, index); zstrm = zcomp_stream_get(zram->comps[prio]); - src = zs_map_object(zram->mem_pool, handle, ZS_MM_RO); + src = zs_obj_read_begin(zram->mem_pool, handle, zstrm->local_copy); dst = kmap_local_page(page); ret = zcomp_decompress(zram->comps[prio], zstrm, src, size, dst); kunmap_local(dst); - zs_unmap_object(zram->mem_pool, handle); + zs_obj_read_end(zram->mem_pool, handle, src); zcomp_stream_put(zram->comps[prio], zstrm); return ret; @@ -1675,7 +1675,7 @@ static int write_incompressible_page(struct zram *zram, struct page *page, u32 index) { unsigned long handle; - void *src, *dst; + void *src; /* * This function is called from preemptible context so we don't need @@ -1692,11 +1692,9 @@ static int write_incompressible_page(struct zram *zram, struct page *page, return -ENOMEM; } - dst = zs_map_object(zram->mem_pool, handle, ZS_MM_WO); src = kmap_local_page(page); - memcpy(dst, src, PAGE_SIZE); + zs_obj_write(zram->mem_pool, handle, src, PAGE_SIZE); kunmap_local(src); - zs_unmap_object(zram->mem_pool, handle); zram_slot_write_lock(zram, index); zram_set_flag(zram, index, ZRAM_HUGE); @@ -1717,7 +1715,7 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) int ret = 0; unsigned long handle; unsigned int comp_len; - void *dst, *mem; + void *mem; struct zcomp_strm *zstrm; unsigned long element; bool same_filled; @@ -1760,11 +1758,8 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) return -ENOMEM; } - dst = zs_map_object(zram->mem_pool, handle, ZS_MM_WO); - - memcpy(dst, zstrm->buffer, comp_len); + zs_obj_write(zram->mem_pool, handle, zstrm->buffer, comp_len); zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP], zstrm); - zs_unmap_object(zram->mem_pool, handle); zram_slot_write_lock(zram, index); zram_set_handle(zram, index, handle); @@ -1876,7 +1871,7 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, unsigned int comp_len_new; unsigned int class_index_old; unsigned int class_index_new; - void *src, *dst; + void *src; int ret; handle_old = zram_get_handle(zram, index); @@ -2000,12 +1995,9 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, return 0; } - dst = zs_map_object(zram->mem_pool, handle_new, ZS_MM_WO); - memcpy(dst, zstrm->buffer, comp_len_new); + zs_obj_write(zram->mem_pool, handle_new, zstrm->buffer, comp_len_new); zcomp_stream_put(zram->comps[prio], zstrm); - zs_unmap_object(zram->mem_pool, handle_new); - zram_free_page(zram, index); zram_set_handle(zram, index, handle_new); zram_set_obj_size(zram, index, comp_len_new); From patchwork Fri Jan 31 09:06:16 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13955143 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 89347C0218D for ; Fri, 31 Jan 2025 09:08:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 188EE2802A4; Fri, 31 Jan 2025 04:08:19 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1395E2802A3; Fri, 31 Jan 2025 04:08:19 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ECB582802A4; Fri, 31 Jan 2025 04:08:18 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id CA62A2802A3 for ; Fri, 31 Jan 2025 04:08:18 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 8DD64808C9 for ; Fri, 31 Jan 2025 09:08:18 +0000 (UTC) X-FDA: 83067170676.21.DFBB4B5 Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) by imf12.hostedemail.com (Postfix) with ESMTP id A8A4C40003 for ; Fri, 31 Jan 2025 09:08:16 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=eetupifH; spf=pass (imf12.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.170 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738314496; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=DQcqfpSqnss0R3EkpuRQBovJzgNoPvoL6ThFstuCphQ=; b=2MDVJcHwZNXkmunreqUSQJgrd2Ut+YXVFFuZO31mOs2x+TdQWDTDtr6sbYvTQJWv0+2SD/ jQ/205c1e770tRfpdNua4qshy3smTyswYmSKSrb0XFO0HuJ/q9tKxmBFVEf4mYmEKYS40q ThCSTCNAYaZxWDa1jtSpaW9jkEG1WCU= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=eetupifH; spf=pass (imf12.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.170 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738314496; a=rsa-sha256; cv=none; b=4YsJgFEsMRYBidlEARr6TjFYeoIm6UMmyWQflgrHns1qgGuX3SK14WEhc10UPgInv8H7cx pIugNuRV+mTix3Dfo+L9/HxiK+4KmDmY2P5fbJxVE7SOA2UpAq4yWoEbdchXbJXK74kZq1 kS5MPOgF3k050pVCucHek4LtN37k8/s= Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-216728b1836so30043955ad.0 for ; Fri, 31 Jan 2025 01:08:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1738314495; x=1738919295; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=DQcqfpSqnss0R3EkpuRQBovJzgNoPvoL6ThFstuCphQ=; b=eetupifHp6gtWyZKrI4d4WzwxtBWdDj3Lilfu4SG5AQxY5jLsiQKFToioDOHpyJOWt 0Q11xbgvDuc070893MkUcJRYae/NMRyYI5wftQUYpNbdxwBedRuXQjAokpFNo8bY6/NX dMNuIvEm09g02B4DNK2MgViqC6waDUTSEu7iM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738314495; x=1738919295; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DQcqfpSqnss0R3EkpuRQBovJzgNoPvoL6ThFstuCphQ=; b=wauW31yaIZSmsFNghPHpadPQFTCqBToOds2/V05XtPi6TB8dj7ylprfyG7vJ6ZSCNI lOxQMTrinYO3APnQU7tLE9QcE5YCIl5+Q7KU1fXHU2XVskvhI7GhsHksRatCnNtxLKo0 QZK5HSOVi8LrkxOieA6VVcaLo5aXB49zVgYPbfgztc3eYn6zukd0uwxFa680VLJPpI61 ZDM83phO+B6D+Zhtfra8sthzOhO/pSNBCXGRr2rMBVJ0DpUR3Wjq8uJUvpc5OCcb3TzI f0uF5Oj4gggYj4fmRbswN4bsvIILYr1xahdxIugDHa0HQKXO1+pUNCSGJaihx1J1muhs hPLQ== X-Forwarded-Encrypted: i=1; AJvYcCU5I5Cho4mMqLpvY6UNThlPwivUsHg6EyGQHupxQW4xFth5jqIb4H9B3bQfLzWoIpl2vVvX7oJI8w==@kvack.org X-Gm-Message-State: AOJu0YwCKRakbmBREQ6HZlRuA0CE7hM309J+6udvWdsKdukkEbA7L982 072VyLTH3xA6Y1vqMl9jyCZggtA0cHbALxajehSVj23VXpRUYY1YVqlr0xxtPg== X-Gm-Gg: ASbGncvXtsP1lm1lfVcwzqtZEV0wENWIhrRtFV/Owr10jb1qK79qViI9eImsTtMxoXC wQGoCuTKy/SkUrlGwJHIU3FSUjcTyT+93Ff/oXgH+CJvBddnjoylhB4URtZ9my1POnguTOHvsqW OGo/W6ynm94Z/4ka3rXOOE8bBKPEy4LJLkukf89rt+zg+xlzSZ5v3+445qrj1oFKk6qz3e0giWB 5w00t7evsc4oOP3CSA1nGM9V74hIRgWa/DUWtWggcJvNLieCzfxvsj/6Bd0HS6X7CKAt5SSRH77 BhFc+DLGXIH1sRwXlg== X-Google-Smtp-Source: AGHT+IEmt7Kd9WrX0z/MrdYcFEjWtu/6bbm1WyY2+91BuQrMC+eYVRLixUf3IzSlvbxl7TXEySCQzg== X-Received: by 2002:a17:903:2f8c:b0:215:b9a6:5cb9 with SMTP id d9443c01a7336-21dd7c3555bmr174992725ad.5.1738314495518; Fri, 31 Jan 2025 01:08:15 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:c752:be9d:3368:16fa]) by smtp.gmail.com with UTF8SMTPSA id 98e67ed59e1d1-2f8489d3982sm3396499a91.24.2025.01.31.01.08.13 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 31 Jan 2025 01:08:15 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCHv4 17/17] zram: add might_sleep to zcomp API Date: Fri, 31 Jan 2025 18:06:16 +0900 Message-ID: <20250131090658.3386285-18-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.362.g079036d154-goog In-Reply-To: <20250131090658.3386285-1-senozhatsky@chromium.org> References: <20250131090658.3386285-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: A8A4C40003 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: c7dkr81b5o1m5rz6qbjh8gq9mxk645ow X-HE-Tag: 1738314496-396547 X-HE-Meta: U2FsdGVkX19MIn/wbi9o8l+eqPr3hXV7Fh0vKNukKtF/jXJXXCOyhxOcQOmQulbm5RypyNECr20Im6BcqZRBypotrtervj0+2f9Cc7PikmZtIiaShcOtPNPx7F5dsnBqBXR9Jtgn7zviDRiELJSFFPtqiTeIB0Yck2OazraA3t4lFmypZXaYCU1cONMWrPoaQq2k27xG3uaPwrFo6+YddCJlAnE+mj/v6Wl9IYwG1w4s2PJ739shEkCcsKZgABsUcWTbbWLkNDdzHWEW6PyBpKyXhPSoOaqDuLuYFJX4HojNUDHpYzxdpwSR0eQSwZDHMoCSFMVmCIuxNOh9VNEumjp8Ofvf+fbUV8oYNP4MQuVv2HMVvavlQ6ArTQAJvOz5NUNGa6ueziq9WYGGeOnkq6qRFhWqZ1R8cMjDw2YPmTX6SPuOxSBxDk6rMUf0Jwwle4I8G9gIAgfTOZr1TRHD8kgJWjhU30lS4IHaf4ehyQmpQrO+jH00TxpEwd9+eYTXR213syXV4pcSe9fZK1I8JgBqxkPBYbfU7AMwi2zukeRX5Jesm58VoARj3dh62oVGVaHWl9BNa89d0ImPUvVR3yb1cbAMSL/hVfI9tlvgAVFKCZHCmF3zcyE5CfD+rXKhL59xlWfKBwfmonmp0ndz/mUdzLWTlkdHdINrD/xr6Qs5TMMSEFTqB4pGMZYX1NOIoF7xDeDwhN0MsZkqmYTsSvHvdXnPQpzzPaSQ0BEM2ernOhV+spXd/lkAeEi9l1ZTfPB6A2Esc3Tlo7KQs60iZv/kHLFIg8m4w+Q7XDSx0HjZJa+wM56lJjmt6cAMaGM4bXPr2uiV0Phn7iKBpQ4eZQOUc+ejxYLq8Ga/1VggoujIzEAY4TeonuaCUV0QsoLo6QNcmWjLG8sgTgT+jV7gsyia5WrUNz8+ihgGfRnf3zfhIKOu3juMufbFe9Jh2oS5i1ivCiNmTWN/Bkr7FlR VkqEtyqD kgwSAlXGDjRMWGMACa3CW65ofw9wuj6lCYDxABQKk1Nc1DBrySP9XMeipnDnzimy0nM1ftHnRnxxbLtKNTM0z88ULoEq1MQ6NXoF/uavsJuiMy/K5Hqa6YjZTZQaMJ/L0fPfOTqp9Tn4xFGvX7QjeRdlV7UYiPTDrwJ8fol/HIor/+JKRx7GUPw2AlSsjQuhwGZfHznVKch0JWyvORhKk5Zg2Y/G58UIf6A9qTnAJK7FFJOhplFoLfq1ZK/h8A3q/So1svGThr/Q9IJCT8kxmTRAS6VqHzI91SleHyWEaHTrYxlWQWyFBnKTTxP823tILwL1aMjatePYvEo6htSr4GHWJKxY4NXqQOjUpprQQ1b+kKBY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Explicitly state that zcomp compress/decompress must be called from non-atomic context. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zcomp.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c index 675f2a51ad5f..f4235735787b 100644 --- a/drivers/block/zram/zcomp.c +++ b/drivers/block/zram/zcomp.c @@ -185,6 +185,7 @@ int zcomp_compress(struct zcomp *comp, struct zcomp_strm *zstrm, }; int ret; + might_sleep(); ret = comp->ops->compress(comp->params, &zstrm->ctx, &req); if (!ret) *dst_len = req.dst_len; @@ -201,6 +202,7 @@ int zcomp_decompress(struct zcomp *comp, struct zcomp_strm *zstrm, .dst_len = PAGE_SIZE, }; + might_sleep(); return comp->ops->decompress(comp->params, &zstrm->ctx, &req); }