From patchwork Fri Feb 21 09:37:54 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13985077 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5678FC021AA for ; Fri, 21 Feb 2025 09:38:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CC7406B009D; Fri, 21 Feb 2025 04:38:51 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C75FD280003; Fri, 21 Feb 2025 04:38:51 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AEFD4280001; Fri, 21 Feb 2025 04:38:51 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 917326B009D for ; Fri, 21 Feb 2025 04:38:51 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 43314140B7D for ; Fri, 21 Feb 2025 09:38:51 +0000 (UTC) X-FDA: 83143452462.20.5A5761B Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) by imf22.hostedemail.com (Postfix) with ESMTP id 5AE1BC0004 for ; Fri, 21 Feb 2025 09:38:49 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=Zeq7v2RB; spf=pass (imf22.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.176 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740130729; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=tEamf86sHM0Ptusd1ivjV+FzA+4m9NR9VbFHQMTSZrc=; b=VycV2PsUU8yS0LYVdae3z6bUAOYIkVmx1MLFCWAf2qo+/K6oQ65Y3CzslwcRNOu+I3CdR6 OibJgeSC9cnai0KaZ3qciUlYmC74D5XtH+SNOYF2rTBw6WiPBa1Yh1Cgml/q5lDBa9d9Sf Hq7OmiUrd8Z2uH6Gu2iXJZ+Et11bkD8= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=Zeq7v2RB; spf=pass (imf22.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.176 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740130729; a=rsa-sha256; cv=none; b=aqVCkz5gmNRsYdKDPVzIXbyC2ozinwgQ28lrKrIsTBqWFxu5bu4sGFPRSt4e5vGCQm+oaJ iviyzhzPdtW2qIsiJu2UrUDeXOzJ99VQA6myCY/6nHizW2tfjlNPyB3CPcInDKMlAnoNUs u4JJUohXq/60zJc4YCNFwZyLwOvlnmY= Received: by mail-pl1-f176.google.com with SMTP id d9443c01a7336-221050f3f00so39929765ad.2 for ; Fri, 21 Feb 2025 01:38:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740130728; x=1740735528; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=tEamf86sHM0Ptusd1ivjV+FzA+4m9NR9VbFHQMTSZrc=; b=Zeq7v2RBZBEGpr1NUItDPPFKLMvEOcD681LdzuASqA6746EOeSWENEc2hWy1/91nY4 N9aea0Ni92tgl4o57RQXO6ENMlLqIu3kYLFKdZoU+wv7ygB0mpGbU9tUDEWsgEu5SS2o YELMcPrt87U3pVIZn/Q97XIwF4OMdNyQEP8e4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740130728; x=1740735528; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tEamf86sHM0Ptusd1ivjV+FzA+4m9NR9VbFHQMTSZrc=; b=ucsNjLnjRdPR8o8ebrWjYJoUs/Yn3j4PCwpP3Ak/8yMJR89TmyJ5Ez6pZ2Rnjtw1Zk ssdqV0ZqxiL5CsnFUepYA8nwlsM8Hi5ROfrkx4zco6j7kKxXIVFeJN3tOURHqS9Trzvk IdigEQQrrlIZo0p/BQv/5ygVbom+0bro0T7O5husQZVzm+4J0sPcojAkCJI4OzVUspIf n3OYniRtpNDGPVncarlvJqU+xZtYmzf+An25I5fqQp2XTw5CoI0RIcWT002PmpzinLH4 HMrQ+2QJkA4tXihkKD2FJBNzHNMsmNysbaZFsDJypE3i/sOyJJKPqCSbtJWkS9FKUAVP 1DGg== X-Forwarded-Encrypted: i=1; AJvYcCXUjVGm58L+AvPBGrwWbi7ZwaPMobRr2fR4W5PsupgFd2RJsXk8z0FqSDeJxSL2LhoC3y/SlYmE/w==@kvack.org X-Gm-Message-State: AOJu0YwP1fRKenL3zYF+3KfmYBKG/B95+htS1cUhMB4TV15SmO8BOSvx Tdk0JGshBS/tg4WM9i8Jf7kwxHgDejC3jIAr8Rsh/MXfBBHVUYDgnGhbPHTUEQ== X-Gm-Gg: ASbGncvyQcQmP4gnlxkeAbzALNQ8b3tnoZZzGx7crHsWxn5Ji4XftIMKgTlbcgh9hGn oqtwUPDAy37CVnvG83IpTFrOu1MxfHMZ79g1kVC9WPbesqUzN8OcD88Wdui7Sg76YBm42wcB3Sl iB6mn4QnzvW+3ZAJKLqZzaS3Rj2ePnB2SCj7jKtIRH5xaOBDL6cWeecihP3ulA8U4PSz2vpL//p ZxWjxCd5K44WpwAq6Cj4MrDyWDCbGAp7ORYzNoOjq6pbvIrYCUhjdyIy5veHOk40+6kHdz7wmG+ tsGWbrgxzp75hVtAu6Df32AHziM= X-Google-Smtp-Source: AGHT+IF6XPk6PTdI7TGgSavL9+4vaZz+bvaqDiLmdjUKVz/91xc480zPlBnpKvLh5Q/90+8Bn/sTSA== X-Received: by 2002:a17:903:2350:b0:21f:515:d61 with SMTP id d9443c01a7336-2219ff4e78emr42043375ad.21.1740130728316; Fri, 21 Feb 2025 01:38:48 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:f987:e1e:3dbb:2191]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-220d556fb40sm132682925ad.193.2025.02.21.01.38.45 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 21 Feb 2025 01:38:47 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v7 01/17] zram: sleepable entry locking Date: Fri, 21 Feb 2025 18:37:54 +0900 Message-ID: <20250221093832.1949691-2-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <20250221093832.1949691-1-senozhatsky@chromium.org> References: <20250221093832.1949691-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 5AE1BC0004 X-Stat-Signature: 5ot45gfjn77p6xn4jazmcxwj8js1qtd7 X-HE-Tag: 1740130729-565502 X-HE-Meta: U2FsdGVkX1/hdGMy3P41sq04GmsVMTFGzsfAc+nVrV1g+9c4JmNRGQn1JnAqHvGm0ka+0OKaHFrm6xLDn3ZSRCha0Z92tvljj42duIRxjNyN8K1Na36dJkrD/nTTUW29g1VnZnUzUqSvbUxw11BciTsNmt1USe6potzSEtfLq57LfYDPpfMKeoiNX3LpLWVAWUGCvHLvb3d0FDgvVFm3G6SbdstW5sKd1vo1HrQFxvsrnTcHmrjV5WkggD2GUtVz+yB8wG/I7uDqtcpFdI5ZVqOzGl3PGCByJhV4Bt8lAwD564kY2VDWdEu9orrWl/jYYpHYw6VUUUh5YiwHMKsHlvbGj2lmsfLv/3tj4SLH69nMdXDRb4hEPkWG4lXQrZG24flyGxReHR7dhhcaPWPlzU5PrSmoJsPKWGQPAx35Jay8OIKlM020BikqtallZgaoBH+q8NDB/Xmu+X25mvZIjJUh23Nzj4QvE7zikLHcFlcCDjKnbhUyaigx4bCcGYy3HT2N3gWXGPjR/ZUODSiu6LPX6NJ1FnpihgK5zroPYnSOoXshbULKnSd/rO8azU+v/MX/QunQGYUK0k4iF72G/iDCBBGm/ogQSIHvcgCWVkgXwYcxKnAAWQiep7mmdam7ykt2xOs8CIlVBCSs6YWXF5EsJgGO8di4r+8HO4aMjTcAgzmF6xBqSY9ph+eC0gm4dlzRfEt+yDZP1vp4yntxgAaFaXLPxaAQ/XmZvq7VNDH/sisPzIrlFk0hyxsEUg577dZPUInGdt1O3tBGS+VwBJx7vH+rWeUhQyDE6vfdmgz5Qf4+CUKehfKbPEr6eNDJYPokE2aaQ58002YhPtTQhNwSnV4NNd9QwDo2J3diD09cbegvSQwiUkcPI8T0sbDlUeaitOCnTq93VSQw2j11rXyH6Myn/6M1UBzCepcYQ1BID78Cclj33OwdwJCV8G0ikn9t1TG1VP/r/n929NP cpTx5hJV ALf5/cvadO7iYrL6q2KHAs6MchJBlJmAV+MFT/OvjPv1CRpSQlmrHjJXc2Bg3FZqACcZFRRFH3OzL1CofYNsb8JDDmtcsFgOBhpUsxAL7G0AJG+FDJPKSrYL7sPHf4ikq9vkplBUWMs6Ka+jTUfQEYQI+tAV+8AE/4+Q0IYUEpQctl5aJvVMOCI309kIa5owXhVdRY9rtSj9hKazUkbvljB6NxdGHJhzuEd8XZb+HxfoVw/NwHuza4aaanOIv7JmHXiQFHElMJMOdOyLa7zweHfYnvyg5oWkHo6lMzRJAQayoTZyRgDt1uuN8SA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Concurrent modifications of meta table entries is now handled by per-entry spin-lock. This has a number of shortcomings. First, this imposes atomic requirements on compression backends. zram can call both zcomp_compress() and zcomp_decompress() under entry spin-lock, which implies that we can use only compression algorithms that don't schedule/sleep/wait during compression and decompression. This, for instance, makes it impossible to use some of the ASYNC compression algorithms (H/W compression, etc.) implementations. Second, this can potentially trigger watchdogs. For example, entry re-compression with secondary algorithms is performed under entry spin-lock. Given that we chain secondary compression algorithms and that some of them can be configured for best compression ratio (and worst compression speed) zram can stay under spin-lock for quite some time. Having a per-entry mutex (or, for instance, a rw-semaphore) significantly increases sizeof() of each entry and hence the meta table. Therefore entry locking returns back to bit locking, as before, however, this time also preempt-rt friendly, because if waits-on-bit instead of spinning-on-bit. Lock owners are also now permitted to schedule, which is a first step on the path of making zram non-atomic. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 68 +++++++++++++++++++++++++++++------ drivers/block/zram/zram_drv.h | 20 +++++++---- 2 files changed, 71 insertions(+), 17 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 9f5020b077c5..31bce7c3ce8b 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -58,19 +58,62 @@ static void zram_free_page(struct zram *zram, size_t index); static int zram_read_from_zspool(struct zram *zram, struct page *page, u32 index); -static int zram_slot_trylock(struct zram *zram, u32 index) +static void zram_slot_lock_init(struct zram *zram, u32 index) { - return spin_trylock(&zram->table[index].lock); +#ifdef CONFIG_DEBUG_LOCK_ALLOC + lockdep_init_map(&zram->table[index].dep_map, + "zram->table[index].lock", + &zram->lock_class, 0); +#endif +} + +#ifdef CONFIG_DEBUG_LOCK_ALLOC +#define slot_dep_map(zram, index) (&(zram)->table[(index)].dep_map) +#else +#define slot_dep_map(zram, index) NULL +#endif + +/* + * entry locking rules: + * + * 1) Lock is exclusive + * + * 2) lock() function can sleep waiting for the lock + * + * 3) Lock owner can sleep + * + * 4) Use TRY lock variant when in atomic context + * - must check return value and handle locking failers + */ +static __must_check bool zram_slot_trylock(struct zram *zram, u32 index) +{ + unsigned long *lock = &zram->table[index].flags; + + if (!test_and_set_bit_lock(ZRAM_ENTRY_LOCK, lock)) { + mutex_acquire(slot_dep_map(zram, index), 0, 1, _RET_IP_); + lock_acquired(slot_dep_map(zram, index), _RET_IP_); + return true; + } + + lock_contended(slot_dep_map(zram, index), _RET_IP_); + return false; } static void zram_slot_lock(struct zram *zram, u32 index) { - spin_lock(&zram->table[index].lock); + unsigned long *lock = &zram->table[index].flags; + + mutex_acquire(slot_dep_map(zram, index), 0, 0, _RET_IP_); + wait_on_bit_lock(lock, ZRAM_ENTRY_LOCK, TASK_UNINTERRUPTIBLE); + lock_acquired(slot_dep_map(zram, index), _RET_IP_); } static void zram_slot_unlock(struct zram *zram, u32 index) { - spin_unlock(&zram->table[index].lock); + unsigned long *lock = &zram->table[index].flags; + + mutex_release(slot_dep_map(zram, index), _RET_IP_); + clear_and_wake_up_bit(ZRAM_ENTRY_LOCK, lock); } static inline bool init_done(struct zram *zram) @@ -93,7 +136,6 @@ static void zram_set_handle(struct zram *zram, u32 index, unsigned long handle) zram->table[index].handle = handle; } -/* flag operations require table entry bit_spin_lock() being held */ static bool zram_test_flag(struct zram *zram, u32 index, enum zram_pageflags flag) { @@ -1473,15 +1515,11 @@ static bool zram_meta_alloc(struct zram *zram, u64 disksize) huge_class_size = zs_huge_class_size(zram->mem_pool); for (index = 0; index < num_pages; index++) - spin_lock_init(&zram->table[index].lock); + zram_slot_lock_init(zram, index); + return true; } -/* - * To protect concurrent access to the same index entry, - * caller should hold this table index entry's bit_spinlock to - * indicate this index entry is accessing. - */ static void zram_free_page(struct zram *zram, size_t index) { unsigned long handle; @@ -2625,6 +2663,10 @@ static int zram_add(void) if (ret) goto out_cleanup_disk; +#ifdef CONFIG_DEBUG_LOCK_ALLOC + lockdep_register_key(&zram->lock_class); +#endif + zram_debugfs_register(zram); pr_info("Added device: %s\n", zram->disk->disk_name); return device_id; @@ -2681,6 +2723,10 @@ static int zram_remove(struct zram *zram) */ zram_reset_device(zram); +#ifdef CONFIG_DEBUG_LOCK_ALLOC + lockdep_unregister_key(&zram->lock_class); +#endif + put_disk(zram->disk); kfree(zram); return 0; diff --git a/drivers/block/zram/zram_drv.h b/drivers/block/zram/zram_drv.h index db78d7c01b9a..794c9234e627 100644 --- a/drivers/block/zram/zram_drv.h +++ b/drivers/block/zram/zram_drv.h @@ -28,7 +28,6 @@ #define ZRAM_SECTOR_PER_LOGICAL_BLOCK \ (1 << (ZRAM_LOGICAL_BLOCK_SHIFT - SECTOR_SHIFT)) - /* * ZRAM is mainly used for memory efficiency so we want to keep memory * footprint small and thus squeeze size and zram pageflags into a flags @@ -46,6 +45,7 @@ /* Flags for zram pages (table[page_no].flags) */ enum zram_pageflags { ZRAM_SAME = ZRAM_FLAG_SHIFT, /* Page consists the same element */ + ZRAM_ENTRY_LOCK, /* entry access lock bit */ ZRAM_WB, /* page is stored on backing_device */ ZRAM_PP_SLOT, /* Selected for post-processing */ ZRAM_HUGE, /* Incompressible page */ @@ -58,13 +58,18 @@ enum zram_pageflags { __NR_ZRAM_PAGEFLAGS, }; -/*-- Data structures */ - -/* Allocated for each disk page */ +/* + * Allocated for each disk page. We use bit-lock (ZRAM_ENTRY_LOCK bit + * of flags) to save memory. There can be plenty of entries and standard + * locking primitives (e.g. mutex) will significantly increase sizeof() + * of each entry and hence of the meta table. + */ struct zram_table_entry { unsigned long handle; - unsigned int flags; - spinlock_t lock; + unsigned long flags; +#ifdef CONFIG_DEBUG_LOCK_ALLOC + struct lockdep_map dep_map; +#endif #ifdef CONFIG_ZRAM_TRACK_ENTRY_ACTIME ktime_t ac_time; #endif @@ -137,5 +142,8 @@ struct zram { struct dentry *debugfs_dir; #endif atomic_t pp_in_progress; +#ifdef CONFIG_DEBUG_LOCK_ALLOC + struct lock_class_key lock_class; +#endif }; #endif From patchwork Fri Feb 21 09:37:55 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13985078 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 81D82C021B3 for ; Fri, 21 Feb 2025 09:38:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 17B306B00A0; Fri, 21 Feb 2025 04:38:57 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 12C04280003; Fri, 21 Feb 2025 04:38:57 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F0F32280001; Fri, 21 Feb 2025 04:38:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id CF7386B00A0 for ; Fri, 21 Feb 2025 04:38:56 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 8EF651C840E for ; Fri, 21 Feb 2025 09:38:56 +0000 (UTC) X-FDA: 83143452672.24.DF1ECFD Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by imf08.hostedemail.com (Postfix) with ESMTP id 92E16160005 for ; Fri, 21 Feb 2025 09:38:54 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=eTHZutQD; spf=pass (imf08.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.180 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740130734; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3J2F96AB7zgvGtDzJY4571zbkjr+U9L0eHeRaKN2Txw=; b=5Sc3MP9Ql76LWZgomsdi29Ry/YJPPskBQR0KoSvlgOIHHgO5wiMQTZbOJfRulh2Sxp6XK8 o8JyTm+1qGO4zm3iCOHoL2gOS5AhEQNrz6X+gURGjEWOdPY3AJqLNzomI5jUb7XwCDC+V9 NSxINA+tU5skTFqDb414g5nNihrWQD8= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=eTHZutQD; spf=pass (imf08.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.180 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740130734; a=rsa-sha256; cv=none; b=0K0qa8zWADrmmaAu7wN6y9/Fe4MWs/BGeY22g0K5qbCBhXSKzZgbEjqW609tQz+4juIZmo nsFPEv9zrSY6BxPixnKJO1KQWFTHXXZnp2YJD/7bW2SpIfSDmfQwxbQLbbWX5b39mbsVfD 62erRYXt2qERvF3Oe0MM2WFR5sbW/CU= Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-219f8263ae0so39317275ad.0 for ; Fri, 21 Feb 2025 01:38:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740130733; x=1740735533; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=3J2F96AB7zgvGtDzJY4571zbkjr+U9L0eHeRaKN2Txw=; b=eTHZutQDk59mQJI3CuDMga0W0IzX9iRrd4K1Z97ENkqJVKB+kx1JRTc8eGKmChha4N DRvyHM8Cm9kqZAMUCtd+a4a2QhQI7BpfhRjzFVLn6a2JS+bMH90yLay29CLrZJQ1h81q NGxHqCT6c72eztTchiLYu7C3YwLoQpCcmo1Wc= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740130733; x=1740735533; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3J2F96AB7zgvGtDzJY4571zbkjr+U9L0eHeRaKN2Txw=; b=YuLkJEXehApNsVvbWQDSTxVHWKJH/NUWIhadWxh/Pvd+kedhuigrvs/TQGiLW9pYRQ 9k61Ru3pp+3tAjX5FoTB/lOB8YB7XV89fqtCQ2bL/Hset1Wwp3ztpAlFswAcf4X8sM5y 3QKzhUvWKPHR5ZAlWWeTUI+tLyUCyBcbFnNlKj3qZc1XZUjg5RcNTepC0r+eq1TgbV/3 z32rS+ZQ2e/6XaKkJ1KFVnyXyN7yX4Flc3vMNLcRJKb8uUqVj83C29wU3vw1fVuDmVDr oNIZ9Qd4u1h9RiM8SQ09jQSpkcGdBjm0TQ3FaHMgw9MSXDKdUt7F2byofFG8jp94GOOF Cl7Q== X-Forwarded-Encrypted: i=1; AJvYcCWcVP+95gExCnoi4NvoNytkw6OMP7QA5M6nRyaQRGWmxAankigUXbZrUQdk8AIlV/dF9LQXxIp36Q==@kvack.org X-Gm-Message-State: AOJu0YytjJMxqz9T2cw144PkoNdsjgJesx0No9PoJyXG8vSeKOj+xX8w XIbLHD2qPX43PwXEtgGCSmz/C00nZwELozlSGVMOzzIKaLocBrTbtrISqlTn0Q== X-Gm-Gg: ASbGnctB/LL3yBvbRTM1jidb3+tP6/adqt6rUrEUG/HGBVtxB/9V3Upu2rQJ8hLlkCv d1/8HbXhQb/ctZYU7RVlQSrThbDS5/BaVmwnMuGVv+3y+wEQB5hhhVpTMPVKnqhUqQhF1vu3MVL C13kQp9MX00mt9flTt4nUST3Sl21p4eHXDZAHBv3auJPlsBTEMAVNy6KpJTM8uaCzxvN6nZwtJY 0TjpvaKV8I0boYC5GC3ofUIQ1qeh6l7dqpgCXYOtk1aDAD6eGjSc83D+Nvoz0lDADGPZrsBvo8A cN4e3KzKbUwXITTOrpgMiJ2mr8o= X-Google-Smtp-Source: AGHT+IGOi6SPciorjNACJD3sIjQHlvMRrWmIILFhO0P0XVNPDa2JYcssns8HG8BXkS0LFd21ZLAyMA== X-Received: by 2002:a05:6a21:3288:b0:1ee:e655:97ea with SMTP id adf61e73a8af0-1eef55b505cmr3612717637.41.1740130733499; Fri, 21 Feb 2025 01:38:53 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:f987:e1e:3dbb:2191]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-73242546867sm15269312b3a.24.2025.02.21.01.38.51 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 21 Feb 2025 01:38:53 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v7 02/17] zram: permit preemption with active compression stream Date: Fri, 21 Feb 2025 18:37:55 +0900 Message-ID: <20250221093832.1949691-3-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <20250221093832.1949691-1-senozhatsky@chromium.org> References: <20250221093832.1949691-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 92E16160005 X-Stat-Signature: bci9eq5491xzo4er1itdk356bwatd6k8 X-HE-Tag: 1740130734-384930 X-HE-Meta: U2FsdGVkX19L9a1dEbzVs6jnRSZvEAehdJEnOUre5+06keDrT2QM9Knvso9Hvmh0X4VdEw1BqnqZkNMdvb6vGVF5sDRDNbZibqec8Wn/fazIhc6Ie1Ayuh2vP+aXDvNV7LAqlBuVwfQ9mHssFeconR0qcspePczSdUTYMtcb/XKi9MLHd0lXg3eF2Qa4U1FVVa/FXsj6qH7FcSlRH344YVJhe/KmRLzj68zPKdswI2TDHJ85hQIL3kN8u0oF7et/ypaPbj1TA2L1KBlTnKJD/2xLNgWWSXh5jllTbd62vf8xONIl5Ug2lV5R7QwtUwpSz4nYlaJ/QA2QnhqBuO0QTmnvShbtruvSbXAuxYkhWsITYp7d2Pue4/bDysZqy6uZln6IgDsTe+NHoAAMJ1J0GEtqHkOc7HMvXE98cpgANmb40B9HekTL05VC9HGvp1kFYx8N9+le1TsRzOoQw4HuOVci9aPhqJectLK6MDhQcT8l45DspQypjp0VyXRJNm3MEsPn4YTtu+w03rjpcppSpXBFhw8NuPEjbL7b1h0Dq0S7nRvHsODDiZYhnOpdHWOAoV5ykPy5EV4fJ1+M5xno+FBH7t8lZXRxXk8r6hHBYXi/ToTGbeA4b2YkIKaPFyBXBhB5jJvbMjJkULinnkLC1DKYIZ6L7QfEXMoX2DgDOtwJx9xVTWZsb+x/jm4NBwyOzauq5krg2VwLpJZlM/hKKOWhJJnaXy2Lj8s+PbOq19/IzbJmdzkSuankZUjeeh/LIxAMQyn88KZQ97Y7tWYNUdsTbutU/F9ayLgO5OJ9PDtSMvlk65MO/W4M/RApZfr1gASDSVyYHsMSKzBv20dTlYpkuLIvH5S9eT4/14j5NvRwEs/xu5yw4BE1y/27TwhlwgklTvIzToVjBvrvw0FRCSKs6oNq3/jMZfnEeJKEC+Xk25m5CSXzXLkp27JF1H6PFQAs+2zgYY/j/iCn3ne WGXv5WyL obbN/eRm9dKyu48/JuBULGnOTW7HPPN+eAFMbERduUyC5thmiyV46BSYjBN794aOTgj4Ll0qh+PO1GKqcN7dlCNZReFNu/CMtPuMqYPt/lptsqPPUdWV8INxTWj92OjrRZ++kNN/FB5HeKfNHdEnFqqzSKDSsuKUNB/nVwiQlOY2kbA8fqHiWuuouSAROW7ETnZ5uBiPrQR44PkjEBuK6yoajvZhBEuT8C/rF3GRnbZ4cqwjq4e7QI2FSPDt+AQ1Jp/v//6/3k0imOiSxadnHji82Lhn0jPQss9jidB6OJ7smZve5ERSI0dHzjQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Currently, per-CPU stream access is done from a non-preemptible (atomic) section, which imposes the same atomicity requirements on compression backends as entry spin-lock, and makes it impossible to use algorithms that can schedule/wait/sleep during compression and decompression. Switch to preemptible per-CPU model, similar to the one used in zswap. Instead of a per-CPU local lock, each stream carries a mutex which is locked throughout entire time zram uses it for compression or decompression, so that cpu-dead event waits for zram to stop using a particular per-CPU stream and release it. Suggested-by: Yosry Ahmed Reviewed-by: Yosry Ahmed Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zcomp.c | 41 +++++++++++++++++++++++++---------- drivers/block/zram/zcomp.h | 6 ++--- drivers/block/zram/zram_drv.c | 20 ++++++++--------- 3 files changed, 42 insertions(+), 25 deletions(-) diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c index bb514403e305..53e4c37441be 100644 --- a/drivers/block/zram/zcomp.c +++ b/drivers/block/zram/zcomp.c @@ -6,7 +6,7 @@ #include #include #include -#include +#include #include #include @@ -109,13 +109,29 @@ ssize_t zcomp_available_show(const char *comp, char *buf) struct zcomp_strm *zcomp_stream_get(struct zcomp *comp) { - local_lock(&comp->stream->lock); - return this_cpu_ptr(comp->stream); + for (;;) { + struct zcomp_strm *zstrm = raw_cpu_ptr(comp->stream); + + /* + * Inspired by zswap + * + * stream is returned with ->mutex locked which prevents + * cpu_dead() from releasing this stream under us, however + * there is still a race window between raw_cpu_ptr() and + * mutex_lock(), during which we could have been migrated + * from a CPU that has already destroyed its stream. If + * so then unlock and re-try on the current CPU. + */ + mutex_lock(&zstrm->lock); + if (likely(zstrm->buffer)) + return zstrm; + mutex_unlock(&zstrm->lock); + } } -void zcomp_stream_put(struct zcomp *comp) +void zcomp_stream_put(struct zcomp_strm *zstrm) { - local_unlock(&comp->stream->lock); + mutex_unlock(&zstrm->lock); } int zcomp_compress(struct zcomp *comp, struct zcomp_strm *zstrm, @@ -151,12 +167,9 @@ int zcomp_decompress(struct zcomp *comp, struct zcomp_strm *zstrm, int zcomp_cpu_up_prepare(unsigned int cpu, struct hlist_node *node) { struct zcomp *comp = hlist_entry(node, struct zcomp, node); - struct zcomp_strm *zstrm; + struct zcomp_strm *zstrm = per_cpu_ptr(comp->stream, cpu); int ret; - zstrm = per_cpu_ptr(comp->stream, cpu); - local_lock_init(&zstrm->lock); - ret = zcomp_strm_init(comp, zstrm); if (ret) pr_err("Can't allocate a compression stream\n"); @@ -166,16 +179,17 @@ int zcomp_cpu_up_prepare(unsigned int cpu, struct hlist_node *node) int zcomp_cpu_dead(unsigned int cpu, struct hlist_node *node) { struct zcomp *comp = hlist_entry(node, struct zcomp, node); - struct zcomp_strm *zstrm; + struct zcomp_strm *zstrm = per_cpu_ptr(comp->stream, cpu); - zstrm = per_cpu_ptr(comp->stream, cpu); + mutex_lock(&zstrm->lock); zcomp_strm_free(comp, zstrm); + mutex_unlock(&zstrm->lock); return 0; } static int zcomp_init(struct zcomp *comp, struct zcomp_params *params) { - int ret; + int ret, cpu; comp->stream = alloc_percpu(struct zcomp_strm); if (!comp->stream) @@ -186,6 +200,9 @@ static int zcomp_init(struct zcomp *comp, struct zcomp_params *params) if (ret) goto cleanup; + for_each_possible_cpu(cpu) + mutex_init(&per_cpu_ptr(comp->stream, cpu)->lock); + ret = cpuhp_state_add_instance(CPUHP_ZCOMP_PREPARE, &comp->node); if (ret < 0) goto cleanup; diff --git a/drivers/block/zram/zcomp.h b/drivers/block/zram/zcomp.h index ad5762813842..23b8236b9090 100644 --- a/drivers/block/zram/zcomp.h +++ b/drivers/block/zram/zcomp.h @@ -3,7 +3,7 @@ #ifndef _ZCOMP_H_ #define _ZCOMP_H_ -#include +#include #define ZCOMP_PARAM_NO_LEVEL INT_MIN @@ -31,7 +31,7 @@ struct zcomp_ctx { }; struct zcomp_strm { - local_lock_t lock; + struct mutex lock; /* compression buffer */ void *buffer; struct zcomp_ctx ctx; @@ -77,7 +77,7 @@ struct zcomp *zcomp_create(const char *alg, struct zcomp_params *params); void zcomp_destroy(struct zcomp *comp); struct zcomp_strm *zcomp_stream_get(struct zcomp *comp); -void zcomp_stream_put(struct zcomp *comp); +void zcomp_stream_put(struct zcomp_strm *zstrm); int zcomp_compress(struct zcomp *comp, struct zcomp_strm *zstrm, const void *src, unsigned int *dst_len); diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 31bce7c3ce8b..3fd1608bbe9e 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1613,7 +1613,7 @@ static int read_compressed_page(struct zram *zram, struct page *page, u32 index) ret = zcomp_decompress(zram->comps[prio], zstrm, src, size, dst); kunmap_local(dst); zs_unmap_object(zram->mem_pool, handle); - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zstrm); return ret; } @@ -1774,14 +1774,14 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) kunmap_local(mem); if (unlikely(ret)) { - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zstrm); pr_err("Compression failed! err=%d\n", ret); zs_free(zram->mem_pool, handle); return ret; } if (comp_len >= huge_class_size) { - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zstrm); return write_incompressible_page(zram, page, index); } @@ -1805,7 +1805,7 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) __GFP_HIGHMEM | __GFP_MOVABLE); if (IS_ERR_VALUE(handle)) { - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zstrm); atomic64_inc(&zram->stats.writestall); handle = zs_malloc(zram->mem_pool, comp_len, GFP_NOIO | __GFP_HIGHMEM | @@ -1817,7 +1817,7 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) } if (!zram_can_store_page(zram)) { - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zstrm); zs_free(zram->mem_pool, handle); return -ENOMEM; } @@ -1825,7 +1825,7 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) dst = zs_map_object(zram->mem_pool, handle, ZS_MM_WO); memcpy(dst, zstrm->buffer, comp_len); - zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP]); + zcomp_stream_put(zstrm); zs_unmap_object(zram->mem_pool, handle); zram_slot_lock(zram, index); @@ -1984,7 +1984,7 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, kunmap_local(src); if (ret) { - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zstrm); return ret; } @@ -1994,7 +1994,7 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, /* Continue until we make progress */ if (class_index_new >= class_index_old || (threshold && comp_len_new >= threshold)) { - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zstrm); continue; } @@ -2052,13 +2052,13 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, __GFP_HIGHMEM | __GFP_MOVABLE); if (IS_ERR_VALUE(handle_new)) { - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zstrm); return PTR_ERR((void *)handle_new); } dst = zs_map_object(zram->mem_pool, handle_new, ZS_MM_WO); memcpy(dst, zstrm->buffer, comp_len_new); - zcomp_stream_put(zram->comps[prio]); + zcomp_stream_put(zstrm); zs_unmap_object(zram->mem_pool, handle_new); From patchwork Fri Feb 21 09:37:56 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13985079 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1FD02C021B3 for ; Fri, 21 Feb 2025 09:39:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B3F416B00A3; Fri, 21 Feb 2025 04:39:04 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AEFA06B00A4; Fri, 21 Feb 2025 04:39:04 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9B7616B00AA; Fri, 21 Feb 2025 04:39:04 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 7B3606B00A3 for ; Fri, 21 Feb 2025 04:39:04 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 35784140C0A for ; Fri, 21 Feb 2025 09:39:04 +0000 (UTC) X-FDA: 83143453008.25.4D5B896 Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) by imf26.hostedemail.com (Postfix) with ESMTP id 44A34140004 for ; Fri, 21 Feb 2025 09:39:02 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=gyiR1K95; spf=pass (imf26.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.169 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740130742; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=F1Fr47oZvnwPkNkfXAk51rgR9W9ztXQBGqxFE9BRxTc=; b=MsmKUDdd854DNjeKO0RCJ7i7Q5d6z6Qh4mptiulhAFTjOhhgrc0UO7ps9+catXuHRfNXbY VmxLUJlS4l6NVdNRK1By8xx/xnMfdHnDT0UXTcM8j8dqgbSFxKl/Z8+EPA5DDL8BvMplNe g4klMPuWFvwGwosOHhqiUKnwpkZZ8CA= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740130742; a=rsa-sha256; cv=none; b=Z3QJiDcmuQHobF8iKNHM9lJKuYMIggO1wGsCK/V/60lAvTOD0HM/UrrKwkjsqU44zcAuAs 6TNgAflQgxqPPl6RMqECq7LTqMFqWP1eICroo3SXujHEIlP8lf59B5f3M+tVJlGHkwE1nG i9CENL4jsdqvPmbDbQLy3yOGanXFNUM= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=gyiR1K95; spf=pass (imf26.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.169 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-2211cd4463cso37181965ad.2 for ; Fri, 21 Feb 2025 01:39:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740130741; x=1740735541; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=F1Fr47oZvnwPkNkfXAk51rgR9W9ztXQBGqxFE9BRxTc=; b=gyiR1K95vcNo+KrW2+kTbObwlQrnAzaO3d5RJCo/nOWIU2FUO4Hj185kBH7FJAM+DZ pX1lzqqEgSUPwuEehOh3yt3ReUED3CXFHVMSplwMIdaEukuxE/8wtfUERvR08s5NoPVX YTM9Ga8fryZztOZRjVhOSL//C/UvjHPvOm9Fg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740130741; x=1740735541; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=F1Fr47oZvnwPkNkfXAk51rgR9W9ztXQBGqxFE9BRxTc=; b=uohoNLvih1hcTJjhEmsYZOubn/mk5FIYmVFPA92iWpYk95JDohlcoa43jrbG2C7SbR BcW+K052Ph/PX86VcG/I8bqNF6iPxsOdtAXonbfDmstVYcyx5bBf/XCJ7NCbGJdqeh8P v+yYgT499j9z9W8VAFh6lsCaEuzN9uvKd/GIhYaDseVgm9q7hBwj520NzaJNhhLdY028 XBIQeq6KOl+85YFVJnfzhvv+gskW6XJ3vlAlk3OLtSYGr98gtxSqC4y1GKsv0O+Z5ocR PrNFO+bRdJF8wxLCpbmdWRFDUkVMofK8k2oIYcddqTSq/eoHY2ALXd30P7Squ4LhxFJV 0vBA== X-Forwarded-Encrypted: i=1; AJvYcCXyqoyj5LtdY8aMfIxnpIDuBlg3GjgPeJtL+M39kELGfpcYT1gpRJP5gA7PnnJ8zHhB6lOVrRZ1MQ==@kvack.org X-Gm-Message-State: AOJu0Yziq+/ITkWtI7tMZq1EYzy+ZNBBHFV9MXsyBva76Zc/aXQJ71iI bWK40mlTA8mdx/Wg22bVFv2so2dTgE6e800/gUQa5Fxs9jUSsLehy7oWxrnyDg== X-Gm-Gg: ASbGncsrMRiOaqvuDrLIlGTJqPy3CpY6wc9wEe8lZcbfaAj1Ep3s/UScFzwgZVKkM65 VRzz5UUy0PNbLXuvbeVlq+oidyAJSpsN95RYB6dx4/Tz68LwaDgU8B2eaGbBuQ1fyaIk/P2gu+a GZmI+NyjhSDE4MRU794afRsyda/IWkinXcBZaVQxpYuy0YyIYG3IJMFA3TM8yuadyBzs9aSfPzg kNbFlBl+ApgpTt61Nyc6K+VYd24lgXtGv22k2SN68kkuXEO9jSvSlo7fP+hOkc8UG4/OqzSW/3S PiFDGViVQwT7vpz38I5PgxB5HiQ= X-Google-Smtp-Source: AGHT+IFRUnCZlCQ55FhwarcrVaKRWTrZj4RkPpnwVEGcDwDMY39AagQltM3YaVXCI6t8ZJ/r6pw83w== X-Received: by 2002:a05:6a00:4fd3:b0:732:5b10:572b with SMTP id d2e1a72fcca58-73426cb2b0bmr3161816b3a.10.1740130741168; Fri, 21 Feb 2025 01:39:01 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:f987:e1e:3dbb:2191]) by smtp.gmail.com with UTF8SMTPSA id 41be03b00d2f7-adb5813a687sm11978723a12.20.2025.02.21.01.38.58 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 21 Feb 2025 01:39:00 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v7 03/17] zram: remove unused crypto include Date: Fri, 21 Feb 2025 18:37:56 +0900 Message-ID: <20250221093832.1949691-4-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <20250221093832.1949691-1-senozhatsky@chromium.org> References: <20250221093832.1949691-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 44A34140004 X-Rspamd-Server: rspam07 X-Stat-Signature: ny9juxuawezzjksmhszfoh74686bmssd X-HE-Tag: 1740130742-2368 X-HE-Meta: U2FsdGVkX1+VqQgqJ3haEMeHMF/6+ZNfr2BZHSyvb8OLd78MO0flfWtQHXkmlJ39e+qc7VfTzaL+gMhDm7U03zz5TShZ7bbN8+UB8qg89kBL0xX1ZpLQxi3VRarOaRhpWLJT5PAD3MkRl059Wvp/Y7QCRcVRiHdqOnGdDUAtiirVIHtfmGu85l1rWQMwwdU4eb/GKs/C2Pbfj7O8Ms8M4ZsRjJk7h4tQfMIeass2NJXkcE9VcwK26cEzpCfJIK/giZ67MQrBT2WqTYzjs/PhcVsjx71F5InabQGiUWB9gjaWY+pIUufSvu3jCpXGRM7uTGheiBIaVli6dScj+F6FO/fkHZ9P4dIQAgdwNNtYCNPvEs4OulOx3wz0iblddr6ti4zTV2YLy2nXoCHgpCKsI88bnSa9u7tgxqSjNih/LShwYWyv0mpStfdzS92AQGu0xVXYn9fMwZMRgK0UXPjmPWBKwAw32s+zb76io9CsfJmJtXMYC174dBbfnNZbHRxCJxihgO3vLM+6dy9n3EGDOXnvA6cWMb0O/Qgz5OxpGNJ96vuOQxzOCU5QLqlqhwz27+u/q7waPWaHFLBxyl/xALkTHv/zC8sig1+/ayzOt/xMm9QWTR19T/MCfj/KXKALR2ltmxSryXHjwm5vpIQkMFBaXJNZMytotddNuMFUDW3AT0FcimAu/F2+Y3V1vj8Ei/sCC0YXK6NCTnwHlV3OYCBQJ8eYXKRgRsZi3xe/FvVd6BxBd8C4OORpnwgOIL2TtAaZdVbkNbkf5Wr29tfEJP+rzwSX1B+bEgcPPdi4zCj1CwCDx3uNbDAqeq3ZYHM9/v3JAn9HfM5ggehYH1F3SoaDYorMaG7zRZ+gkRdgkepvJw0BXVmSk9wCnrtDkONlHSi1wKcC5EuJQivTorCAmM155cFOWWSUXt+utfuCx70rsxoAo8Nz1em15Es7qQnO4ZP/+Hzr+MLOV2G6XOy lDpXrFf4 80Fx7Wnnui75zgzSMCPkxMqCoQ5D4sMH+7j3Z55PaTKgTPcqYP20sV1BOpcNeEabynnlRJc2He+x9LPE2PnY7VXgHZockI+RLqWb6znReYSqeEmpUiFamZarxkvE1Gxb/ZHz08E8WAdnkuoPPzfxXzWjZa5NJnDMovgsKJEAdYYaw62BPzaBT6wwXCj6JvFBNyLxujBRv/k+0bOXkKHa/EtntFyFWU5vQrQXdRT2XEfAUiloeorJ1/NhXZyumJsafozIm9jF4con32Ma9WiwtdaMWB4SYrxenM001ZMAZkaqLiCEyR3BtUiMv7Xx0VauxkALwgpPXe87s4t+WCIUu4UZU1eV55TkLS40c X-Bogosity: Ham, tests=bogofilter, spamicity=0.000037, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: We stopped using crypto API (for the time being), so remove its include and replace CRYPTO_MAX_ALG_NAME with a local define. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zcomp.c | 1 - drivers/block/zram/zram_drv.c | 4 +++- drivers/block/zram/zram_drv.h | 1 - 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c index 53e4c37441be..cfdde2e0748a 100644 --- a/drivers/block/zram/zcomp.c +++ b/drivers/block/zram/zcomp.c @@ -7,7 +7,6 @@ #include #include #include -#include #include #include "zcomp.h" diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 3fd1608bbe9e..424ed1128fb3 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -44,6 +44,8 @@ static DEFINE_MUTEX(zram_index_mutex); static int zram_major; static const char *default_compressor = CONFIG_ZRAM_DEF_COMP; +#define ZRAM_MAX_ALGO_NAME_SZ 128 + /* Module params (documentation at end) */ static unsigned int num_devices = 1; /* @@ -1154,7 +1156,7 @@ static int __comp_algorithm_store(struct zram *zram, u32 prio, const char *buf) size_t sz; sz = strlen(buf); - if (sz >= CRYPTO_MAX_ALG_NAME) + if (sz >= ZRAM_MAX_ALGO_NAME_SZ) return -E2BIG; compressor = kstrdup(buf, GFP_KERNEL); diff --git a/drivers/block/zram/zram_drv.h b/drivers/block/zram/zram_drv.h index 794c9234e627..2c380ea9a816 100644 --- a/drivers/block/zram/zram_drv.h +++ b/drivers/block/zram/zram_drv.h @@ -17,7 +17,6 @@ #include #include -#include #include "zcomp.h" From patchwork Fri Feb 21 09:37:57 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13985080 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 94B93C021AA for ; Fri, 21 Feb 2025 09:39:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 30365280001; Fri, 21 Feb 2025 04:39:10 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2DA676B00AB; Fri, 21 Feb 2025 04:39:10 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1A27E280001; Fri, 21 Feb 2025 04:39:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id F1D5C6B00AA for ; Fri, 21 Feb 2025 04:39:09 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 7911E1C85DE for ; Fri, 21 Feb 2025 09:39:09 +0000 (UTC) X-FDA: 83143453218.10.17116BE Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) by imf18.hostedemail.com (Postfix) with ESMTP id 79ED71C0004 for ; Fri, 21 Feb 2025 09:39:07 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=hgIufcmq; spf=pass (imf18.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.177 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740130747; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=gNAdjSZPdBAH3LrqV88ZZwXLFy97UrlICUdu5mXaygY=; b=HKaqRoSzaHHaij2k1Jbq5QGFdMycH4t2aFmdfGcAwY34L8jAx54UUoSXVhnLhFcR567JrD t+xT561OG9m6xBNihVki/0PvhUG0ix7aBrWcUvBXHb5S/ndDjyPE0Y/J6XXBuA6wRzqUwh iBpn+/eOfwlNBJbIN4arAyukE+Hjo6E= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=hgIufcmq; spf=pass (imf18.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.177 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740130747; a=rsa-sha256; cv=none; b=JwKc07/meiN/cehESGZH0q9pkoiSLLEEINguPeIQYcAPvUf9h4k4KMmfAPRq6t5CB0M+o6 7O6n56lbU9CguhWZiZ5mtMjZvUGn/x1aGgg3eh7x64kHzW/rBE61ZsEpCVQaXYQdV2nbin WOKjICCz/72ZNkHuwnUVVN7kwn04EJ4= Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-22185cddbffso45276755ad.1 for ; Fri, 21 Feb 2025 01:39:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740130746; x=1740735546; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=gNAdjSZPdBAH3LrqV88ZZwXLFy97UrlICUdu5mXaygY=; b=hgIufcmqyd0nevtnnouEG41wy+2I7GiPmbpWKoANc0rGmhf2K3OHgDn6aDWQh4RSqT 9AJJ8ipRxMfk8oRHQTUt/q6VrTGfWSmmy9LbJBAUuViUOL61UKiDwvWkA+UH0UnyoOUx otJSwaAIymzAoBER8wO9S/KxdeTBm5CFKrurY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740130746; x=1740735546; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gNAdjSZPdBAH3LrqV88ZZwXLFy97UrlICUdu5mXaygY=; b=RSq/huXF0mU36e61zhBEPlPF02I6yFsLZz/gb2yHLxFf/ptFPNeEMddpgTK70u44xa jiWiHq3aINupyER+qhd4zzvXcS/obG/m1EASNLiiFK35QtZlE3Y9VjESp2PZCjj2eWV9 DcuBZuaYps0+byTxMTEP/tB8wH8rkwG3MICEp24YcU59lK4BWunHDvsXoD3shsUlAKqN Y3ZJSiioNS8Rc2MM13QmLinr2PebwxthKVvl/WBKvUFnAc8kWXmSizese4W2Jgaw6yFn GpBqXiRREYZOyY0zI+cwclAiG+I8VkP8Av1/TGnKHCxGFbSN74zady7o/ExHfOrUxOe4 3kwA== X-Forwarded-Encrypted: i=1; AJvYcCUH7Ipv+X9Fl0JLUtEqWfEHItlv9Xd4zMJxk0SXYsQMPCkwfi7OFBzFlR5+Cc+3zzNQimQJoJSHNA==@kvack.org X-Gm-Message-State: AOJu0Yyqs31Xa7A+oGO9h/LaAbDm3AS0GK9/irgpVShjdi+vtTCb4Nfx t/egurLZ5wmGdMMC/uevgw/jJD05W15cteDVDzx7Lr1reCeQ9TNH5e5ro7Jryg== X-Gm-Gg: ASbGncv5AtCSpPokJxz4RZFYMDdXZyOyoaALqYdaTG4uwISvNJf4AbKTS62g+wWpdxg W75NuH71/IpOrhjBX8C3a4ee/HiZOwYaiilS3QpeIAEX7thzBOvNbWEOKl/YX00vdBzsDe9n+O0 PasRuuha0CfRK4Fm7K3kAsNAqq25PSTluuwBqaJmnMduizoKkvO6j97GCm/DzDw4hcS+wHkKkgA R2pLktU3HnZy3mJ7Rlgi1Ia9XVJLHldIeT8aanvvoS8vSN7N7u22YrWSBTDSPF/FVwtegUtw59V oEBtkkRR9iMYyEDN751KQrUugo4= X-Google-Smtp-Source: AGHT+IFV3Kv4elyYdLzHIwLUKC2Z7+Or+dH9Vd4/YSg3c0Jz7xi+JB7UxI+JSjhGhix3fONvjyLTTw== X-Received: by 2002:a05:6a00:2e14:b0:730:9637:b2ff with SMTP id d2e1a72fcca58-73425ca201fmr3949487b3a.7.1740130746241; Fri, 21 Feb 2025 01:39:06 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:f987:e1e:3dbb:2191]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-7326b5ef448sm11220174b3a.173.2025.02.21.01.39.03 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 21 Feb 2025 01:39:05 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v7 04/17] zram: remove max_comp_streams device attr Date: Fri, 21 Feb 2025 18:37:57 +0900 Message-ID: <20250221093832.1949691-5-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <20250221093832.1949691-1-senozhatsky@chromium.org> References: <20250221093832.1949691-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 79ED71C0004 X-Stat-Signature: uhoubkmf9h3pma7wxdbmnoh5sf1x86fp X-HE-Tag: 1740130747-177243 X-HE-Meta: U2FsdGVkX1+wK95HkHdolcMMm6HdNhpX9yI0KGJMxMHP4FepmbG8YbRSuLRAvFDKHdop4xiaAltdKXgNJDg3qn+JPbwHkk5WGU++gpURhAseFisP6kiXe+7NxdtsZzL1RDheB4CgcQDrlyT5kx2wJFgS7eBkrVpmJCu13GZWAh0Zg5VupIqvDqwfqAWbk7pVcFecAOGaWt3Yfy3AZFvXt+hY3G/ehwaNa70DjMMWiL9lvpsTXzEIwPCMOhL7sU4omZJ8qjiLEdXpFaoQi4XTwp8OvJ1A5u0cJUMPu0sf9MY+pUZwYZiYll5u5QrghcF4KZ3B5JbBafECoccIzP3UzbSez6N51/MTlg5gycrOArA3sOtPRoieZVIABErciVpLHK7oS82z8x1yN3nCvuYtL4puE9xyBwKYj19/uZTJLYhVkDhWVpIc3T9T9yf0lIhIpouRGClsJ+CK7iN8pSynqMEEAEKYQdVhQLLcfHvqkkLbvlQ7sbiuj5XFYBB/AyXvXJbotOCOhECsUEh/Telc47H9Pj32IIRdWxHCZyIlIrHC7rgYAs1U69Zm5fMtp2QQjCXAyG+sm5XRP/7X8OK/BucJYkOe2Vf5HHkl+Oa4D6eVP10uTNsBCmUZrPuJ5HdMePsZ1uQsdl1D84jfUsT4fvMvftyAHgLfOgXbSLdayRz8Xwb2q4+k8Ugi9Qpiq89sedX1Nrf2prESg4/8SayrDWj9o/d2oju41UrL4Zx7HJJ/1QSBbJsx3rhcU8I0GJoPRU4OAGS7Ev1DXdylTl4g0yjySn9FKIfiG77rxHqtAUcNgNtVwCaRAYs3wFHfOubIvXDLg2M/QBoPXqWAhip9Ued1ngSjOXYM5acxHpmlxNMTZpHPuJESJpCCUirI/tWcqxde3GJQFKyORghC+WQKshpuJP46+cyXzJ8ae5GvAo//vGE7we715/lHXKUrXfQZ4BTrtIrheH+qdyT088/ WGz4jsM6 seqCp3twUvWRyjivaiEL6ZAqNI+qjuLiynv3xdyAcAW4UXCtHV7M8xr9IyTUdPUogSkCioNTm+oaZB3adzyU3knxMf9bZjevBJBKscSprHRUn3AC/UO7/T7BGp82uagEapDbbS4wKFFdMx5M6WQtA+Gu3iUP5SGKKMmQauYyr9GZuAhauajHxgJbTT0uaUxzICm1wF4Anls5IhkQUObKGxnuF0owEC7RzQilg6tR5gG6Awog4OnDNY0zo/tHEqWh6oP+2BR9rUJ9xZ++dLjt9FGNco2cpQWku3lThSVduVVJkC40P1Sc4gVQp2NSYaKn0Ps6ndfT41mAsGtFcc9hOkC4Y/BCuqSzPb5lIKWLGqwuSbCTkAaSpNMsM6INtQ3hzu7FY X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: max_comp_streams device attribute has been defunct since May 2016 when zram switched to per-CPU compression streams, remove it. Signed-off-by: Sergey Senozhatsky --- Documentation/ABI/testing/sysfs-block-zram | 8 ----- Documentation/admin-guide/blockdev/zram.rst | 36 ++++++--------------- drivers/block/zram/zram_drv.c | 23 ------------- 3 files changed, 10 insertions(+), 57 deletions(-) diff --git a/Documentation/ABI/testing/sysfs-block-zram b/Documentation/ABI/testing/sysfs-block-zram index 1ef69e0271f9..36c57de0a10a 100644 --- a/Documentation/ABI/testing/sysfs-block-zram +++ b/Documentation/ABI/testing/sysfs-block-zram @@ -22,14 +22,6 @@ Description: device. The reset operation frees all the memory associated with this device. -What: /sys/block/zram/max_comp_streams -Date: February 2014 -Contact: Sergey Senozhatsky -Description: - The max_comp_streams file is read-write and specifies the - number of backend's zcomp_strm compression streams (number of - concurrent compress operations). - What: /sys/block/zram/comp_algorithm Date: February 2014 Contact: Sergey Senozhatsky diff --git a/Documentation/admin-guide/blockdev/zram.rst b/Documentation/admin-guide/blockdev/zram.rst index 714a5171bfc0..7ad4c86f8258 100644 --- a/Documentation/admin-guide/blockdev/zram.rst +++ b/Documentation/admin-guide/blockdev/zram.rst @@ -54,7 +54,7 @@ The list of possible return codes: If you use 'echo', the returned value is set by the 'echo' utility, and, in general case, something like:: - echo 3 > /sys/block/zram0/max_comp_streams + echo foo > /sys/block/zram0/comp_algorithm if [ $? -ne 0 ]; then handle_error fi @@ -73,21 +73,7 @@ This creates 4 devices: /dev/zram{0,1,2,3} num_devices parameter is optional and tells zram how many devices should be pre-created. Default: 1. -2) Set max number of compression streams -======================================== - -Regardless of the value passed to this attribute, ZRAM will always -allocate multiple compression streams - one per online CPU - thus -allowing several concurrent compression operations. The number of -allocated compression streams goes down when some of the CPUs -become offline. There is no single-compression-stream mode anymore, -unless you are running a UP system or have only 1 CPU online. - -To find out how many streams are currently available:: - - cat /sys/block/zram0/max_comp_streams - -3) Select compression algorithm +2) Select compression algorithm =============================== Using comp_algorithm device attribute one can see available and @@ -107,7 +93,7 @@ Examples:: For the time being, the `comp_algorithm` content shows only compression algorithms that are supported by zram. -4) Set compression algorithm parameters: Optional +3) Set compression algorithm parameters: Optional ================================================= Compression algorithms may support specific parameters which can be @@ -138,7 +124,7 @@ better the compression ratio, it even can take negatives values for some algorithms), for other algorithms `level` is acceleration level (the higher the value the lower the compression ratio). -5) Set Disksize +4) Set Disksize =============== Set disk size by writing the value to sysfs node 'disksize'. @@ -158,7 +144,7 @@ There is little point creating a zram of greater than twice the size of memory since we expect a 2:1 compression ratio. Note that zram uses about 0.1% of the size of the disk when not in use so a huge zram is wasteful. -6) Set memory limit: Optional +5) Set memory limit: Optional ============================= Set memory limit by writing the value to sysfs node 'mem_limit'. @@ -177,7 +163,7 @@ Examples:: # To disable memory limit echo 0 > /sys/block/zram0/mem_limit -7) Activate +6) Activate =========== :: @@ -188,7 +174,7 @@ Examples:: mkfs.ext4 /dev/zram1 mount /dev/zram1 /tmp -8) Add/remove zram devices +7) Add/remove zram devices ========================== zram provides a control interface, which enables dynamic (on-demand) device @@ -208,7 +194,7 @@ execute:: echo X > /sys/class/zram-control/hot_remove -9) Stats +8) Stats ======== Per-device statistics are exported as various nodes under /sys/block/zram/ @@ -228,8 +214,6 @@ mem_limit WO specifies the maximum amount of memory ZRAM can writeback_limit WO specifies the maximum amount of write IO zram can write out to backing device as 4KB unit writeback_limit_enable RW show and set writeback_limit feature -max_comp_streams RW the number of possible concurrent compress - operations comp_algorithm RW show and change the compression algorithm algorithm_params WO setup compression algorithm parameters compact WO trigger memory compaction @@ -310,7 +294,7 @@ a single line of text and contains the following stats separated by whitespace: Unit: 4K bytes ============== ============================================================= -10) Deactivate +9) Deactivate ============== :: @@ -318,7 +302,7 @@ a single line of text and contains the following stats separated by whitespace: swapoff /dev/zram0 umount /dev/zram1 -11) Reset +10) Reset ========= Write any positive value to 'reset' sysfs node:: diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 424ed1128fb3..d3b70aae2660 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1109,27 +1109,6 @@ static void zram_debugfs_register(struct zram *zram) {}; static void zram_debugfs_unregister(struct zram *zram) {}; #endif -/* - * We switched to per-cpu streams and this attr is not needed anymore. - * However, we will keep it around for some time, because: - * a) we may revert per-cpu streams in the future - * b) it's visible to user space and we need to follow our 2 years - * retirement rule; but we already have a number of 'soon to be - * altered' attrs, so max_comp_streams need to wait for the next - * layoff cycle. - */ -static ssize_t max_comp_streams_show(struct device *dev, - struct device_attribute *attr, char *buf) -{ - return scnprintf(buf, PAGE_SIZE, "%d\n", num_online_cpus()); -} - -static ssize_t max_comp_streams_store(struct device *dev, - struct device_attribute *attr, const char *buf, size_t len) -{ - return len; -} - static void comp_algorithm_set(struct zram *zram, u32 prio, const char *alg) { /* Do not free statically defined compression algorithms */ @@ -2546,7 +2525,6 @@ static DEVICE_ATTR_WO(reset); static DEVICE_ATTR_WO(mem_limit); static DEVICE_ATTR_WO(mem_used_max); static DEVICE_ATTR_WO(idle); -static DEVICE_ATTR_RW(max_comp_streams); static DEVICE_ATTR_RW(comp_algorithm); #ifdef CONFIG_ZRAM_WRITEBACK static DEVICE_ATTR_RW(backing_dev); @@ -2568,7 +2546,6 @@ static struct attribute *zram_disk_attrs[] = { &dev_attr_mem_limit.attr, &dev_attr_mem_used_max.attr, &dev_attr_idle.attr, - &dev_attr_max_comp_streams.attr, &dev_attr_comp_algorithm.attr, #ifdef CONFIG_ZRAM_WRITEBACK &dev_attr_backing_dev.attr, From patchwork Fri Feb 21 09:37:58 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13985081 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7E9F6C021AA for ; Fri, 21 Feb 2025 09:39:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0B8BB280003; Fri, 21 Feb 2025 04:39:15 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0681A6B00AC; Fri, 21 Feb 2025 04:39:14 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E4BEE280003; Fri, 21 Feb 2025 04:39:14 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id C89136B00AB for ; Fri, 21 Feb 2025 04:39:14 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 714B5140C1C for ; Fri, 21 Feb 2025 09:39:14 +0000 (UTC) X-FDA: 83143453428.22.CAA60A8 Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) by imf16.hostedemail.com (Postfix) with ESMTP id 8CEA5180002 for ; Fri, 21 Feb 2025 09:39:12 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=RlbcXKgk; spf=pass (imf16.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.172 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740130752; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=owpeQWEBy2AJNnuQaGP0HmEMVbn5ZGw0EqTWg3icB+A=; b=RfmJf6R3ZSHhvnGVewPqIjrm3KLgf8PsfwmntWnuUV8hjx+XOAhoaZjWrYI4elTlNy5OqZ WxTzTy94r2vc1y+kRgC4StJbBMmQXqzlfNPIlZUxLMejth9NLRRo1mL6ZBEhQTlnmVKs+w bDZhq9GVsB34KAU34sEr2ygvWGSgEzw= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=RlbcXKgk; spf=pass (imf16.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.172 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740130752; a=rsa-sha256; cv=none; b=4mKRbHP+MqWuwUbrs/+ObwZLWwjj+fdLghxkbp5fAZCI7iYSHtAY8uH4jVbX0f1N7N+2aC sz3P3u7imZ2EVVeJXKtumCA7zFGJIkqyivs/sDhF775g4e1qUGEwBewzwerEa4DU/ac68F DtXtkf6Zx8B3NOFObvmc3zQ0mzzsbxs= Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-221057b6ac4so34809875ad.2 for ; Fri, 21 Feb 2025 01:39:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740130751; x=1740735551; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=owpeQWEBy2AJNnuQaGP0HmEMVbn5ZGw0EqTWg3icB+A=; b=RlbcXKgkmaQanrwnWfxhLKuRB3zMVSU8TdlflS51mGiCYqJdPfN0WmWQhQtcnzfEKz 6gcs5tb4ELMS9InVNNwEtE/7oIx4c0GELb8dVDDaWOHT8bdidBow05gKL9gG9UQhU9R3 SJUjvgoAaOuLrLhU8PUXzKmQ+40oIS5fyNnCw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740130751; x=1740735551; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=owpeQWEBy2AJNnuQaGP0HmEMVbn5ZGw0EqTWg3icB+A=; b=tdQEesXj4VkdNn+GbeJxtAv1tbOfVXYc7BrV//wcqoHzy3NtnANDOTT4DUiyJ/ah3b E17yTN56Q5/xc6n7Wwj/i4Dc4mHQYurZ20FE5HrzFV4YWwUlJn8bH0Y3TCE7LgAjrF7R HCQLf+XBucgrQWaav5gXhN04viu1G/xqGsb1UVsqs1oJTYhKuYHJjBPPQF236nlcvsfR W0URzkHKIzpRIAQmA2BF3KwNBaYKcFWEQ75zgxdB6VDnsrZOJ/9AbIcSQPiKOl5eGSQQ LoXNlBClZizaflrQMiGOpmTGG27gX21khbbFkAvkbuRUXTJJqfYMAVerwZUXxL6h+nUn NxVA== X-Forwarded-Encrypted: i=1; AJvYcCXkUke76FHd5sQFSYPXVxzQxFoD95B5OY23J3UlfJzKG73M9xNubQifT2pLYKsUO01pJNhnAUE6Rg==@kvack.org X-Gm-Message-State: AOJu0YzdGUlvh/Kt33qA315h1qpdZU/LCgoa3/vOWEGC1J40nCHyFnH9 w2Jy5MNxeAqVHZcnHu/5zTtPcKXCggi0Kxzo5KlXRINZtSoyjvUckU42+b1SUw== X-Gm-Gg: ASbGnctdEkp1uIiqSBtgSZKsvduJNnibuW6FZ3feoD/phZnUZyF5+c6TORi58y9B928 NGOOF5JWwaTPHEND/WcWvVkL03jRl1hdQriTRJMiXj/ncgxD7FkJLR63S9YXefRfDEbT3w1+1Eo V26QyxIWiYMgkPdhQtZ8Cq+QF5B9p2kZzawYUPNhGgn34hUE0nrsyv8x364LSyISaWFlx8zxCdo hUQ4J0z7KXQ4/ra0W7FG86M4zs+iHXC12wi4wmYnpxSMbe7US65jXf1JtNvFB4fzZbgn4zxVaeN 8flf3U4gngmqaVXzJedLiGKS0VA= X-Google-Smtp-Source: AGHT+IEr9op50ypLXHrxjr4hZ7IvQcDqM446lcCXIuVBN0RImmgqBI3rJCeE1hqij4hU017n9SwtSg== X-Received: by 2002:a17:903:2309:b0:220:eaaf:f6ec with SMTP id d9443c01a7336-221a0ec9501mr37135415ad.5.1740130751439; Fri, 21 Feb 2025 01:39:11 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:f987:e1e:3dbb:2191]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-220d5367e97sm132666595ad.61.2025.02.21.01.39.08 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 21 Feb 2025 01:39:11 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v7 05/17] zram: remove two-staged handle allocation Date: Fri, 21 Feb 2025 18:37:58 +0900 Message-ID: <20250221093832.1949691-6-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <20250221093832.1949691-1-senozhatsky@chromium.org> References: <20250221093832.1949691-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 8CEA5180002 X-Stat-Signature: d1g83e7ucnqq9awwce9h943txd55n31h X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1740130752-523252 X-HE-Meta: U2FsdGVkX19Oqzy/Xby1olNktYNU3OSCeP1aTDBh57ilbcm1LcrwYE4ChhFqgEW/O0XgbVfIR2o67Bd3tFa6dfy1Gb2rAsuyQqxZJWAuHCf0Mr9hBQ784MezRsmDTGG+Omq4fkb21510pne5hQVqkwOClzfLjMiQ4FefYB+YKHKnzVHCwVzSnJT0HJnbxAfqJXSfeUssHr+1DH1PITk34SnzleC9o7tosZMTJvgmxbrX2dqkHRgOWNF65HsUjdovthTPGRH7QYHgVXWNsxeCNv3gssXDnJi4qC1UDglPN1TMuFYweEUXywpUgZOGgznzkstaCHLnZrMaaY5/xWbDAt/hgOGyfuDxpyEjamrQ9nsR16+3+IZYJojielKKP1HOzRqOuD8imz6a3bfQqSQQz1JFaVIhQ0I8XdEtSMGHXSAJq88os8AEPXZo/j3K9PRvvDJGo9PAJS4RgoPSdDqGIKYUNO3BXg3EkO+rZypVQgWFXJFOezFTkTlx2w2UersP5vdVf4rqifAqsTQHSBKmkNMLLjK+MchPuwVNybsdjnedLNqWGgqijc8T9sXEAoYcNo/K7NGqYdr/XZAwSvmEAD9WJPdpXtbv1vND1Xf2GlWPRsq64qgQd4ehe+Rgv5aw6XyrMCi+dKlVKvRmvXsmG28vh8fpHW+NTI64mRFLjFrKST6e4dFjCOY7I3Ye97k9Qw8CAiFOALFTNfRxk/BK6Z0gr3u2AUYWrDK3HpLpeARC+MvrnF5YrgwNRTl/C0SmZVetqLdDNQwrhhIdxckWTK/Q9xwFjaNdHouWCUwWkk1wLYef1i0uSrESg+TRdYQiH/fo1wlOsrXSu3FApQta5tgc0W4qClEyt2PZyTUmlXWZwrWmvzx9GsML8enjmVZvUE9BnIDcshxPQNwhtMCAScdBVRt5Bv/O5CaNVukcwxJvEfmDvDZATQmW9utiP/F2tjNUz8pGgQ5CFL+Wf5Q kebMqlmq hNtrT7YGZiqYoL1T3fUhswMjJIUf53kra/w+qfmXoJHgooS1q5iYYyKjGZWxR+OBCXJFt9oVQpk+MpiVU19rz3XNGSIgU761nV3rygiLXz2kj4tyDVnEHvgRJ0RggoFajoBtobHgG1WEjQcc//rKL2tkIj6+1PQ7rJC/IydPQNepoC7h5UYGtMrjLpNBcjWTWfYsZqZO/3bMD4gvNwAgAQfjZwqDvD4PFj/BluN88TFCWT7dNxuwlushe6dblRiW6RqZ4SoSnYUZv2k6ShLyYgxK6JbxDZ3TC44cehC5UUD5yY6ReDyLMj54N2A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Previously zram write() was atomic which required us to pass __GFP_KSWAPD_RECLAIM to zsmalloc handle allocation on a fast path and attempt a slow path allocation (with recompression) when the fast path failed. Since it's not atomic anymore we can permit direct reclaim during allocation, and remove fast allocation path and, also, drop the recompression path (which should reduce CPU/battery usage). Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 38 ++++++----------------------------- 1 file changed, 6 insertions(+), 32 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index d3b70aae2660..b805a3661101 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1729,11 +1729,11 @@ static int write_incompressible_page(struct zram *zram, struct page *page, static int zram_write_page(struct zram *zram, struct page *page, u32 index) { int ret = 0; - unsigned long handle = -ENOMEM; - unsigned int comp_len = 0; + unsigned long handle; + unsigned int comp_len; void *dst, *mem; struct zcomp_strm *zstrm; - unsigned long element = 0; + unsigned long element; bool same_filled; /* First, free memory allocated to this slot (if any) */ @@ -1747,7 +1747,6 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) if (same_filled) return write_same_filled_page(zram, element, index); -compress_again: zstrm = zcomp_stream_get(zram->comps[ZRAM_PRIMARY_COMP]); mem = kmap_local_page(page); ret = zcomp_compress(zram->comps[ZRAM_PRIMARY_COMP], zstrm, @@ -1757,7 +1756,6 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) if (unlikely(ret)) { zcomp_stream_put(zstrm); pr_err("Compression failed! err=%d\n", ret); - zs_free(zram->mem_pool, handle); return ret; } @@ -1766,35 +1764,11 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) return write_incompressible_page(zram, page, index); } - /* - * handle allocation has 2 paths: - * a) fast path is executed with preemption disabled (for - * per-cpu streams) and has __GFP_DIRECT_RECLAIM bit clear, - * since we can't sleep; - * b) slow path enables preemption and attempts to allocate - * the page with __GFP_DIRECT_RECLAIM bit set. we have to - * put per-cpu compression stream and, thus, to re-do - * the compression once handle is allocated. - * - * if we have a 'non-null' handle here then we are coming - * from the slow path and handle has already been allocated. - */ - if (IS_ERR_VALUE(handle)) - handle = zs_malloc(zram->mem_pool, comp_len, - __GFP_KSWAPD_RECLAIM | - __GFP_NOWARN | - __GFP_HIGHMEM | - __GFP_MOVABLE); + handle = zs_malloc(zram->mem_pool, comp_len, + GFP_NOIO | __GFP_HIGHMEM | __GFP_MOVABLE); if (IS_ERR_VALUE(handle)) { zcomp_stream_put(zstrm); - atomic64_inc(&zram->stats.writestall); - handle = zs_malloc(zram->mem_pool, comp_len, - GFP_NOIO | __GFP_HIGHMEM | - __GFP_MOVABLE); - if (IS_ERR_VALUE(handle)) - return PTR_ERR((void *)handle); - - goto compress_again; + return PTR_ERR((void *)handle); } if (!zram_can_store_page(zram)) { From patchwork Fri Feb 21 09:37:59 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13985082 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8DAFEC021AA for ; Fri, 21 Feb 2025 09:39:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 23F34280005; Fri, 21 Feb 2025 04:39:20 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1EED5280004; Fri, 21 Feb 2025 04:39:20 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0B701280005; Fri, 21 Feb 2025 04:39:20 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id E0FC2280004 for ; Fri, 21 Feb 2025 04:39:19 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 9603E80BF2 for ; Fri, 21 Feb 2025 09:39:19 +0000 (UTC) X-FDA: 83143453638.19.DA5D738 Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) by imf08.hostedemail.com (Postfix) with ESMTP id A514D16000E for ; Fri, 21 Feb 2025 09:39:17 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=T+RFtPRl; spf=pass (imf08.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.172 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740130757; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=dWUXn5zzM3sEuYVHsX5O+HQ9Km/ronw7RO7r8BNsmmQ=; b=dROefSOY8g0OCJLmRs25dtixr7Jm87X14Sv7hCt/4A4torLq0iFEAVvjc7S+3drzyIWlQ5 bhnpQVr8ccrOcH48Bg1wZbHB2TslUy1qFk8cNb6GSn8+Kz4VLzvjQAl7W/hxxRgt/fzx85 3GcK5TJLQa5shEYmwhPUq17//hR86D0= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=T+RFtPRl; spf=pass (imf08.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.172 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740130757; a=rsa-sha256; cv=none; b=JMIcTjVXOs0haZgxTED42VvUW9hp3yxAgw0Vce4hfBJwCa3J/dtCcx9y0Ea2cMN635sr9n JyqqB+ac9Ag0wlyOPR3iiiiDMBYaVkd5HnwofDnyvDXh6fAIyF+tjXH9402EQ7CRZmO0Kh x+K4thvC7ViPPUlfMsvOFooNxKtN0/k= Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-22100006bc8so32895325ad.0 for ; Fri, 21 Feb 2025 01:39:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740130756; x=1740735556; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=dWUXn5zzM3sEuYVHsX5O+HQ9Km/ronw7RO7r8BNsmmQ=; b=T+RFtPRlZotGOSO/+hkmfnH+G+VwG7u/6t5TEvfI3BSHDxR3NyjV9vAFys/mJHDVF+ THnyQoE4PTpuQdXkPBtcxSbpzPlYUOQ7sMRsJrZB14eNGv3+wmd5kPy8GbTmzE53FmXO sMul9bXDw5kVWGGDhwNAKMAC4C1JPCHExurMw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740130756; x=1740735556; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dWUXn5zzM3sEuYVHsX5O+HQ9Km/ronw7RO7r8BNsmmQ=; b=o757/HrCwLSU1M2MhZT2b4nec5aoiNzKk0yAPEt23YR/DR9E46E9d61DsQiPwvXUs0 U58uKdu91Apqihlj8MITfO30cJeynYqNGDY3b4WB++HntyismOLLR59hp5FSGKRtj41W gUrB3GVfzRVHRLoj49C6v+5tPM6VJLMM1eiwreZb7grOLmsWmQjVPS8UAa7qA6VHlFfK xv+6kueAPhuXEFVm2cKoVsWEWLZ1763eYJOXTNPiPYcAQJ9sUX6bAHcy5+5hpEKluT/M slnliL97KmCvZ6RQ0A12Iqt+nSg4opi2r9f+mYQkWY5dtWC0ztS0r/U8bOKzpreZByRS k7LQ== X-Forwarded-Encrypted: i=1; AJvYcCUkXNuS2mjKK6+ReGV9ebAR0qOI5mxyp7PziIots/cPIK8oiKRW6eTklxzPPu2MMuScuMYDPZ1OyA==@kvack.org X-Gm-Message-State: AOJu0YzL51ecuebT0ouEBi8nbScYP50L4LBlAUeJdKkH95UmakkrNzIo eSRmmP4bGJbo/5zlpzDHVbn9/pviV2q649vi6pnffvPyTj7FVHjEjCJYEhuQdSr/CvlGKepEQSI = X-Gm-Gg: ASbGncucko2nH0AbV8OP8tKr9iMcRRIFPTUnXyLD3sLJTY6paCPU2wucs8XE/fa/wQ8 iHvVz+RcIh/aZjExyCKTITrIfukDpR8DZmxsK44ZmnSoYkwxYMkoqibSph9HX1UV4xiLMsp5zrC saBSNsxkXqsoDNA7xqbEa4SId6dd0v5/eYOT00vAjikDs2n7wyud9qwmUzjZUVY4OXwir1p8b26 8Ajtn5RGCKEi5tYwY/iJPz9GQfOiTj4myKpqOaBkM4B8As3P/IMltVePcJaHE/yR2XWen508NSq 8j4C2P7GRqxb54BjvJdsp99WCN4= X-Google-Smtp-Source: AGHT+IHVbAaNiCZXUdV+FymTI/dwqaElFp7dSQG1wMJdlZh6amw8nQFYdiY2vLz0zNb+VFzK/PfYdg== X-Received: by 2002:a05:6a20:9144:b0:1ee:db10:a4cd with SMTP id adf61e73a8af0-1eef3c889b0mr4622607637.13.1740130756632; Fri, 21 Feb 2025 01:39:16 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:f987:e1e:3dbb:2191]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-73287ac0ea9sm7967823b3a.152.2025.02.21.01.39.14 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 21 Feb 2025 01:39:16 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v7 06/17] zram: remove writestall zram_stats member Date: Fri, 21 Feb 2025 18:37:59 +0900 Message-ID: <20250221093832.1949691-7-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <20250221093832.1949691-1-senozhatsky@chromium.org> References: <20250221093832.1949691-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: hudb36xechq8ysaykenergrzjgkkbe98 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: A514D16000E X-HE-Tag: 1740130757-223513 X-HE-Meta: U2FsdGVkX18o23cyw8fV6QjfVO3DLkuSr9aOm0U+x4x9fMQmekB+BgEpuFlA/5/KscK3C619HGwB5GAZHxYDPohyFLDN1isTr+BnppOTN12Rkg6kBPIM10T1X3ww7FRrfNBE2CMojPTqilGGblaAY/GrCZSZYJMLFFsUr2q/vDIugTDNYxTBLa6TbrBnfK4LgRiF6dZmgaiBL6vyoCC8pdVPEB3DsY5jldAJ26ijKLBaxEEk8qLFKdbgWtS2Jrge9vcmA+MCg0rHqWtrA0zPKA8CgL6C3aOawA3k4xjN//Zbd5uRqx+3MhKLgBrkcpxXrgqRAks2mhiZ16xhIHCg2Vn11xXsdLc9J6Br+iPt5UVv3++insptfS0dU2j5NxmVCHeXAFUDeBmZdLliOUO9ul0BgTEbfMEB+U1fVEj+EVFIpmCNb/hIA2VwODnoE3KoMK7a9ugeLKSkFeNFXxX0+v/2+3XVO9lyRqarW+WPeHG9LR4l6U0NTgkOnZCA/T7lcjzlOIUmUy47vq0lYupGqcM+LRkRs5Utm7iFIXTLBiB2aHzSV6i9254zq7aRgoNHBhI1oEa7S4LGaUV9Yjr9MZKs7TUR1RoikiVf1XHvLbj1beS8zSaq9tLFfF+DEXGvcmMt3OHxLKmHqhEgKZkTHHCgssoXyAPq6FMUlIc5SGRisiZSut7t0x+3HGHDJ0JRqNC/W7wuLGxm2dHHL0jbbM2NXTrfeGCMXZtNxLjOlkKMSnSVd5rVVp4bENn47BALzzx3fpBJwVuYWBnoKsKdDfIV3SIgmvBoLrlPTxfzHb7gzpYRVMu1y+ehVZldf76rE7wtRWNHPjCB0Qr7DFOGRVOfdMFLI4Im0IAQYLmDGGNRv43zgQiwG3oPTjuDtI+nUb8plKr4u8SKIU/+br/e8/af9DJKFBQKHwS+V3pgpdT6pmZBsC2gDD/c9IkfDnDO8Dlib4+Tg1zUuD9JdY/ 3ELcq8Nf ayMm1N7m8d17dfj5xuN8ZEqSoWnl7rMDV5RJKY6lAOEim2O5ZNYZF7ob6S4lE+dbz3EQ7vhz718HIASEHlnn6amUmglPLssomUhbPAM8l4mpXq2V3bM1C/3zojd9E+vaVHyoKDsXJwkVgLO9XKRTqgO4hascGzr7EPjgR825XpXPQU69IDs1Hu3blyHAuNqmYI1jiApX9Sm17H4sBfr4DiWFx6LxgR6ShQDhfxY4LzkUNXrM7xouYUJVAn6LYgWhiLLzCMX7k5ASpogmoSuhI13Y2ppVgbWsZW7aQm5bY+hswd5RJi1BwLeuwS7sRe07Y/+7ekIc7pA8hzQf+PAMqWG4pa6XoDJuNUZb5GjHsnbVVDK0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000001, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: There is no zsmalloc handle allocation slow path now and writestall is not possible any longer. Remove it from zram_stats. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 3 +-- drivers/block/zram/zram_drv.h | 1 - 2 files changed, 1 insertion(+), 3 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index b805a3661101..f59fea0f04ec 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1443,9 +1443,8 @@ static ssize_t debug_stat_show(struct device *dev, down_read(&zram->init_lock); ret = scnprintf(buf, PAGE_SIZE, - "version: %d\n%8llu %8llu\n", + "version: %d\n0 %8llu\n", version, - (u64)atomic64_read(&zram->stats.writestall), (u64)atomic64_read(&zram->stats.miss_free)); up_read(&zram->init_lock); diff --git a/drivers/block/zram/zram_drv.h b/drivers/block/zram/zram_drv.h index 2c380ea9a816..59c75154524f 100644 --- a/drivers/block/zram/zram_drv.h +++ b/drivers/block/zram/zram_drv.h @@ -84,7 +84,6 @@ struct zram_stats { atomic64_t huge_pages_since; /* no. of huge pages since zram set up */ atomic64_t pages_stored; /* no. of pages currently stored */ atomic_long_t max_used_pages; /* no. of maximum pages stored */ - atomic64_t writestall; /* no. of write slow paths */ atomic64_t miss_free; /* no. of missed free */ #ifdef CONFIG_ZRAM_WRITEBACK atomic64_t bd_count; /* no. of pages in backing device */ From patchwork Fri Feb 21 09:38:00 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13985083 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A54EFC021AA for ; Fri, 21 Feb 2025 09:39:25 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 39EBB280006; Fri, 21 Feb 2025 04:39:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 34F08280004; Fri, 21 Feb 2025 04:39:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2168E280006; Fri, 21 Feb 2025 04:39:25 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 02B4D280004 for ; Fri, 21 Feb 2025 04:39:24 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id AEFFBB20B7 for ; Fri, 21 Feb 2025 09:39:24 +0000 (UTC) X-FDA: 83143453848.11.E97FB3A Received: from mail-pj1-f52.google.com (mail-pj1-f52.google.com [209.85.216.52]) by imf21.hostedemail.com (Postfix) with ESMTP id D11071C0008 for ; Fri, 21 Feb 2025 09:39:22 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=RzymTe55; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf21.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.216.52 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740130762; a=rsa-sha256; cv=none; b=qt6VxR2q5rGD1ZQ6WuU3j/BP2RPh5R9E7yiwlnyRgmbHnBGlb9ANxYf8vSILHvcUrNwIjS 4P8ZCRlnVhkd6i1es9YrWsV5asLvF3n/2ENj5RZu2O8/YcKo//xawYAoeKkVEr88R8f+nl nr1IMRisMre+n4KUyS1q5wIIIwwdlIM= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=RzymTe55; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf21.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.216.52 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740130762; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YiO1FosDR2lrWYSluoJ0c1MfeSV2ozE15nkpwKYw8aM=; b=dNUSe3ySOCnb1eHDlp36y3mfSNxNssaAHiszFxroQgxhyx0egGdwAaYTiSTChi2WZVH9fH 8S0WH9hM5IdGcULx5BW85CUSYJDlq3PVGco5bKs4ha6fC1L3anFzGdHZNC6eA6eF6fKOh1 8JpKXVAQHsPsnnSSWf6sk0DgqSXSzDM= Received: by mail-pj1-f52.google.com with SMTP id 98e67ed59e1d1-2fa5af6d743so3036803a91.3 for ; Fri, 21 Feb 2025 01:39:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740130762; x=1740735562; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=YiO1FosDR2lrWYSluoJ0c1MfeSV2ozE15nkpwKYw8aM=; b=RzymTe55e0f4TU34TqtLnBbJn1l3YN7IORsQEZ5KDISBZEtAWN+JZngXS3xFZs/vmx P3gBElQfzTYdBZDzdEL9FNqtRpJ5awMuEROFeIctFjTKNvmnZ9CRVrPjXVzqAKL6c5tm VEA/jWvNBYKgWnxbbeFSeSMDlJfzDItDxz4ys= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740130762; x=1740735562; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YiO1FosDR2lrWYSluoJ0c1MfeSV2ozE15nkpwKYw8aM=; b=Lsu5d46/A2QOMB7DujRoUdqFFYZLQkU0Gr4CXwqj37Fhk0I+W9euMxsK+JNoyQSvH/ ZKkZygKklw1i7BRUqqUpBJ9dJmvrExss4J8fTY/GgVVZ5MXMqkOoR/mm7K6UENcleAW8 Jj9uWGw9Is+6ZH6mZe8ghWvGm5V0Z9sN2dKPzMRE4YqDZoAM+0G32D+81wLduSrwyVK/ GWZhEwa4TUG7r/gjrWNhKDz720nkFkQVtp7G+1V5OxSXb1JvHEMHcDXz64UW5ndGrE5E Hee8mRYkXKzzQEe4OqQF3ruEABGcSMM4VQRwdfia1snb6E6Spp3/ghYNYpub/1dPkxxj Abtg== X-Forwarded-Encrypted: i=1; AJvYcCXLJHTjZJ4+ma6iArhOZanmWRyx9qWqCMVSp/HrycYwMOZs8AtwRmChsZhGWT4OU6GjAn53Ekub3w==@kvack.org X-Gm-Message-State: AOJu0YzthnByMDlkZhJQDUkwoCmN16W1e6O1+12R0GEpOJUNAgoUokfO t2P7ETez4PjyFGmaX6IM1r0PSCBHDiifO6b0sO6wpv2TtVmRu2nmigHe74vurQ== X-Gm-Gg: ASbGncvl/6EMmxI/iqnDbVBgN0W2gczpv3+Uy8GOzt4caiqx53phS6wONcrxhPPlYK/ 6sVntAlcTiT4P9ZDWZc0fZMTneekFIT2HnJVwf6cQ2P1phaMgiB4+Yh97DNjWgQ+V6BWHvQcf78 YQFvtBOKAN+ER7C60Fx/2WXjY6eo2jjeIyUyq4Fx3HC1Z46c7H1R+UQEE721IFX5pM4xL74Ly6k 8e+w0BcyT2FPSql70bIB6VXLnWLMa69MQBox2JnvZ9OGX0D/mhwbSbxXjOVmwRLGny2BtCODwNa tmrQdX27uySvCYhKiNTQgoJtoPA= X-Google-Smtp-Source: AGHT+IFFEglapRmDrpRV77iwnKxQ9APz+Jgfl7/ScCUUMY+lOfpVb4pqyKKGcqHkCqH1z6i+q1OeRw== X-Received: by 2002:a17:90b:3d0e:b0:2ea:37b4:5373 with SMTP id 98e67ed59e1d1-2fce78aec58mr4681268a91.10.1740130761841; Fri, 21 Feb 2025 01:39:21 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:f987:e1e:3dbb:2191]) by smtp.gmail.com with UTF8SMTPSA id 98e67ed59e1d1-2fceb065299sm981337a91.29.2025.02.21.01.39.19 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 21 Feb 2025 01:39:21 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v7 07/17] zram: limit max recompress prio to num_active_comps Date: Fri, 21 Feb 2025 18:38:00 +0900 Message-ID: <20250221093832.1949691-8-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <20250221093832.1949691-1-senozhatsky@chromium.org> References: <20250221093832.1949691-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: D11071C0008 X-Stat-Signature: qwcteffig49ab7ygjwa34rau43opxzw3 X-Rspam-User: X-HE-Tag: 1740130762-874313 X-HE-Meta: U2FsdGVkX18vD9ttRXRmnDQmq1kNU5hcf9f3eineeVNuFunIX1tW83j2kBpjpxl58ekVMfOZJHdfOirLgcLlJ1HWENSXTWmQR+c9Ww9bqWKGhSVBdPYvBN4CdrgKe9kR5t+aSKkTEsCoirOtBp77DzoSe70alRGLWq86n28H37uu9KFXqtDJgxigDBj7UORS4JinYBbNX2v1ZxVFh+2svGRbOtlrQTk+8WE7lmhXYuE7+EQPRjvjWMiKEcKE9li6LUD9+QuozsZVybCLcTpLutrNw2BSzATyVyYZyw8Ao4G4d4paYrWapZFAr9UHurLTjxIkd2ze7YtYc2x+K7R7mffvWgmTtih4UtDfYP+yFhS65jfNm7wurjoPyM3N3xtO5qP5BXRetnZ+B5Ok/GdPhAfFgxA2EVM0QetxaC5PjBUUi/D3RbLi6edhave5JRX1FxNLzUuFgo/s00/aXGkPpbJknQQiT9bPhUjod9q+dfq0vOUEyZgsjzNtt4fd8k98HybKeJcuyLkrfoTum7gQokxTAf8BSA5BwR4j7gr/jUGwrn8W3WSLHTM3y+QXvCLYdj8dfThV9kvbN8BY37IgkXJMuLQAa22Q3dn876O6gpPae6n4M7rvua+xfJUxTMo8/SSIPbcq0ehvBFesemzc1UDOxVU/ttVTaYITa1eYSBxjxdJ2hPdPIj+ABFF9aj5KOZIPebV4MKIZ6vrfIoZIJ6AQZTvW1/2FjVHiGgyuoMb4HKyGgy2lalIKo8hT2ghUetEagAJyFSu48CucLJ2/NJiahW140K9uc5PuBL1ENrmac4BeRsSxGu+mjoquiipyuIo1x9Sf95KEBDxejW9G0lYLwM7WoGoKpzwK0dg2u9uQzpnb03dlYvcjgNb11i2Cii6Qkc4++dTGzKgtmGJqqtBDNjahYvBKcYL9OVZUQ2/YgqX9vYOY8oJ4jgni61r7Q3tLM68qxqjsQkD3z8/ fbQKYXbS bCccgsEK9mYoFCmoKAO7urcFuQg4GBYbGo4+tXpnvpfw0QK3n8XB7+JR76AppEX5BIoCBOHL5xiIUoJh/TYw/ltgbPA2hhEbGVKgjaRbujuA8Pnl6UrKyLtWT7SdAUEZeoxn8jlvh7s8xQobhsl3GOmiPhxmn2ApTz0AWARmBTnK/OB7sPVm7sQCxXFj4HFUCru2y6tyC8vwq+ZDJcsXYQqon5yW3lcuzwQw8EZ/8TjxCUS1QVDwAYT0U6CIEByEP7thjPOb3NQ8eckIGpQ/E7Cg362f5ySY3EwInfcyre2HhSB34ok+upPx/E/gd+XW4DXHdpuas4DyAKYCJfxW4qzU70kitvIHVv5KsPG38Ukz/yUA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000025, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Use the actual number of algorithms zram was configure with instead of theoretical limit of ZRAM_MAX_COMPS. Also make sure that min prio is not above max prio. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index f59fea0f04ec..a886827f0b3c 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -2031,16 +2031,19 @@ static ssize_t recompress_store(struct device *dev, struct device_attribute *attr, const char *buf, size_t len) { - u32 prio = ZRAM_SECONDARY_COMP, prio_max = ZRAM_MAX_COMPS; struct zram *zram = dev_to_zram(dev); char *args, *param, *val, *algo = NULL; u64 num_recomp_pages = ULLONG_MAX; struct zram_pp_ctl *ctl = NULL; struct zram_pp_slot *pps; u32 mode = 0, threshold = 0; + u32 prio, prio_max; struct page *page; ssize_t ret; + prio = ZRAM_SECONDARY_COMP; + prio_max = zram->num_active_comps; + args = skip_spaces(buf); while (*args) { args = next_arg(args, ¶m, &val); @@ -2093,7 +2096,7 @@ static ssize_t recompress_store(struct device *dev, if (prio == ZRAM_PRIMARY_COMP) prio = ZRAM_SECONDARY_COMP; - prio_max = min(prio + 1, ZRAM_MAX_COMPS); + prio_max = prio + 1; continue; } } @@ -2121,7 +2124,7 @@ static ssize_t recompress_store(struct device *dev, continue; if (!strcmp(zram->comp_algs[prio], algo)) { - prio_max = min(prio + 1, ZRAM_MAX_COMPS); + prio_max = prio + 1; found = true; break; } @@ -2133,6 +2136,12 @@ static ssize_t recompress_store(struct device *dev, } } + prio_max = min(prio_max, (u32)zram->num_active_comps); + if (prio >= prio_max) { + ret = -EINVAL; + goto release_init_lock; + } + page = alloc_page(GFP_KERNEL); if (!page) { ret = -ENOMEM; From patchwork Fri Feb 21 09:38:01 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13985084 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 85EB5C021AA for ; Fri, 21 Feb 2025 09:39:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1A3B7280007; Fri, 21 Feb 2025 04:39:31 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 15403280004; Fri, 21 Feb 2025 04:39:31 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F360E280007; Fri, 21 Feb 2025 04:39:30 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id D43CB280004 for ; Fri, 21 Feb 2025 04:39:30 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 8A600140308 for ; Fri, 21 Feb 2025 09:39:30 +0000 (UTC) X-FDA: 83143454100.16.23AAAD1 Received: from mail-pj1-f46.google.com (mail-pj1-f46.google.com [209.85.216.46]) by imf07.hostedemail.com (Postfix) with ESMTP id A4F1A40007 for ; Fri, 21 Feb 2025 09:39:28 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=GdZNhQel; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf07.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.216.46 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740130768; a=rsa-sha256; cv=none; b=d6B+SD2Pc4YZ0UgPBS2QNMvFuHLUSPC7DdCJSyjTjoWiggqB1DMnL849cCThDmu22mUW0A A+kj/yYk8MY1P17rG0P6p6oMgDcvKx+CwhbWzn9gaLXFYx1Yxtr04vhdOrAqWeUwasdSLr K6Tt7UKUnf5cGPM9elOZXsLh93gvI/Y= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=GdZNhQel; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf07.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.216.46 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740130768; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6Ti7CmEaorHMBg5DWvmUEjK7cf0LUhuuwHacW6S7Q+w=; b=RRi1iUL6kuAVAplvz27ieYAve8TdIYAxXL9senhUc8otpRS8QqhkX+APt5deb3tjgasi8Z o8nsU/EaxogVtI3c10AzfpKruW26SeDUlLeo6CTalTcJ2efsynk8ZoXqGgqIIuBN17gBUO KeCI43ZimMXWmWHShNp11sP5d2tKQH4= Received: by mail-pj1-f46.google.com with SMTP id 98e67ed59e1d1-2fa5af6d743so3036905a91.3 for ; Fri, 21 Feb 2025 01:39:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740130767; x=1740735567; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=6Ti7CmEaorHMBg5DWvmUEjK7cf0LUhuuwHacW6S7Q+w=; b=GdZNhQelpN3eCP2ICA/TGSTk70j0+p8BOH/OFi5OX8/5Nj7AXMAal6qbzAkxebo+g4 nsJtlP0CPbwk4qACgO9ncp0H5vzlbEY/a4VqE/QUApRSP9oBJEbIF3yktzlSEiyEKHaM Te0uwO2Gi3BtNj0fPWmmpUrX9R40+XVZLTqkI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740130767; x=1740735567; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6Ti7CmEaorHMBg5DWvmUEjK7cf0LUhuuwHacW6S7Q+w=; b=cwwcHT0u1l4kaKULy8nEJmPj5ll6LLECOf9Tfn6uPzmeo8A0s1tcn8RKU+SwE6qPzN yJUGSYxZgkLHbi7Ar8DWU3wPDHyxZQDnSm2a5pb3Zffzn2UJEaEQC1bs0DjQWquDvs2G RBHnuWuREFNwe6ajn8cCRHpc8PS7MiQHGwJtZ06OdSZeYeVNdV9Nfoy0Z9wOvFeVo58j 6FTbQ8i6aNbC9BmZ+Qm+AJ3u6Y6LAhxMvnSohK6Oppn0fhvOI2RKqaIXVLtkoojHoTa3 Nv+xVip/tsNlLH5g3raQMXS3FpnqXDNVk8VXCvolj7xOH6KH6trd5x7JrjDjf/pqB84T kQaw== X-Forwarded-Encrypted: i=1; AJvYcCX1+bwYIxsg5YVjF1spt0o8PLg5VxJfeVgt3hXwjk3Aw2gTdGLCyRkq8QOw2uiZ6Xts68qPm8hUKg==@kvack.org X-Gm-Message-State: AOJu0Yxm8NvaJdVByF9GQQ4eumQk4hxpPMPtx6439HjEeI3MFop0szLz S3Efufjv1eCQ3ST2u2yAWzR/b6O9MWINOXetPulHaW1yjCp/gg8AhmhM6dQpCTPBfIS5gqfT7a4 = X-Gm-Gg: ASbGncuoje54n5TuXw0IOSr6pKe42bzLFGajAUOHcWYou6mHQ+3txyJ0wn0NS1kz+n/ xGObJdIGKaRd+bX4S/vKVM1uRzScEXE7vWct7Dzi2+1JZ537IrI0UTqFcXCfkiRNZaHdA5embK8 UsG7FrdQSSwX38zHgXs7Gk7PotXIWdHkMTCqq24Rn+6lkkadLXEIZxn8yl0Akj1/TpmajNfDJjN ZRe7huS47Lwy3NeZZOE1nOUKfzscOIdEdl4jRiB/4SPv3tgy45PsU4SbCUN7cXLJLPFddRRknY8 V5gO7QzigNNS9T3twbeYc6V8gnU= X-Google-Smtp-Source: AGHT+IFkEw5rhjwVGmuiSJO9ZqK4LoqieklxjbVlzdvGmp36x+X05fC+JwKkd7RRpGwQHuo2CguWWw== X-Received: by 2002:a17:90b:5303:b0:2fa:1a8a:cfe8 with SMTP id 98e67ed59e1d1-2fce7b3ad90mr4368050a91.29.1740130767623; Fri, 21 Feb 2025 01:39:27 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:f987:e1e:3dbb:2191]) by smtp.gmail.com with UTF8SMTPSA id 98e67ed59e1d1-2fceb05f715sm900785a91.24.2025.02.21.01.39.25 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 21 Feb 2025 01:39:27 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v7 08/17] zram: filter out recomp targets based on priority Date: Fri, 21 Feb 2025 18:38:01 +0900 Message-ID: <20250221093832.1949691-9-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <20250221093832.1949691-1-senozhatsky@chromium.org> References: <20250221093832.1949691-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: A4F1A40007 X-Rspamd-Server: rspam12 X-Stat-Signature: i4fxwtkizjixhkuspjg4biy5ds6d3brx X-HE-Tag: 1740130768-296195 X-HE-Meta: U2FsdGVkX1+/L1Fh8gJTgcqYUYYP26gup9s2mt74vUeumiatCAaUcdqJf/v9OJKvXh4NICGFcypfnSytBOdR8X5NV8QXh+/hLdj7eC0gaO1EiMuE4uHMiAt9XtAKQHVVEi6Z9r2P0sY92qRdVGfv/fj0rXj7+KeJfptshVjGNDZfgnF1t48By9DtvYXGOMJ4MFynyGQXsbBxuW93sm876IfGChcVbiPnbadGDIrh0CqZGXiIHx2JAHESTWBsNARrqWdIqDCesiWHG9/fmPjzWvFW6+Z+Fy4bAB78NzGnXpxke6RsisAamXe6dArWc8xwbNMXehcinm8uBIexi1xELbsl6jrodlnSOIU5LfqasmhsF9RGx4f57QzL3f7ztbc0h8tsZifZi3a4TS7+AXHbC4AZf0VsgwnFWiypDs2cEV0/jSK+tB51YAJ6HN2qPv4ECxZR5G8+ahpyZ4oPgNoZ2DVviIVNmZ+G4JseJRpeMxmUfj4LJxVg6bbWXa3s2bFSO+dpDjK+4sHZfxSFwypNYGpqQ4Gi8WguQGCxFySEEk0zMYUExdhnktNvPNvpQ7giNL7F1pVzeZ3r+FjgH76ZIKnNS15qPORWZFRRhNOGQyJCl2L9Sy3nUKo/i+6wg+PcLLOMY3YLGZb+phHvA86SW4GRsvX6wgu7kxuLoYBDN0Yt7wK0KioSILnPEtojAnv73GrkF2IeOJAL06+MldcE09mvh9Md3DpAWnCY3eegWezi4fxPt89zkd/YJxmLhlAmaIyFYi2vUIgH1BoCkK7GOudjHYrJtky2229zWbsN6l13n+Ce4bBVQ33YpLnySFVTCNPdsUe1aiG0o6IQQVsC8MFhplnGysXfXBfp+pTgIpflC+9zEJqnlYP4WXuKSf2E13vQHlTG7mL+jxRawODe7P3QHZjv+Ki3jxZnW+UEg6mBKC7Rd5b9FocJWpLoJUqZgx4zlkTZ77Dmw8FBIjF EW68B1OD J6kK6Vy3bsjAU0yI30nh+Pkw6/TuWXZNJDyIv+pl6fzZOTl4KTaB/4AY8nZ3ibsSXQXLk/+PIJrzh5hwjf1EsccpydXWQrrj6p+TUjPhD5Y84QLSFYv3coqhZgDfbEvXwDRI+AZzBBIlZmEJUjpzV7BQCcVkm1BaFUMKyus+Y8KyIViFeIw375uM7sdw0yu0fBX6v5Egogsqia0+EL8D3H9TeNJPv8WGKbPM/VIjE8yfB0n91zGSHMt2Y9BkatFy5fIRPMXoxvalCEMPFUsdgNt7ndm8ve5mTZ2XSQM+hjbN3s6I9BsaT8SuCnA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Do no select for post processing slots that are already compressed with same or higher priority compression algorithm. This should save some memory, as previously we would still put those entries into corresponding post-processing buckets and filter them out later in recompress_slot(). Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 25 ++++++++++++++++--------- 1 file changed, 16 insertions(+), 9 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index a886827f0b3c..a1880e621895 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1827,7 +1827,7 @@ static int zram_bvec_write(struct zram *zram, struct bio_vec *bvec, #define RECOMPRESS_IDLE (1 << 0) #define RECOMPRESS_HUGE (1 << 1) -static int scan_slots_for_recompress(struct zram *zram, u32 mode, +static int scan_slots_for_recompress(struct zram *zram, u32 mode, u32 prio_max, struct zram_pp_ctl *ctl) { unsigned long nr_pages = zram->disksize >> PAGE_SHIFT; @@ -1859,6 +1859,10 @@ static int scan_slots_for_recompress(struct zram *zram, u32 mode, zram_test_flag(zram, index, ZRAM_INCOMPRESSIBLE)) goto next; + /* Already compressed with same of higher priority */ + if (zram_get_priority(zram, index) + 1 >= prio_max) + goto next; + pps->index = index; place_pp_slot(zram, ctl, pps); pps = NULL; @@ -1915,6 +1919,16 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, zram_clear_flag(zram, index, ZRAM_IDLE); class_index_old = zs_lookup_class_index(zram->mem_pool, comp_len_old); + + prio = max(prio, zram_get_priority(zram, index) + 1); + /* + * Recompression slots scan should not select slots that are + * already compressed with a higher priority algorithm, but + * just in case + */ + if (prio >= prio_max) + return 0; + /* * Iterate the secondary comp algorithms list (in order of priority) * and try to recompress the page. @@ -1923,13 +1937,6 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, if (!zram->comps[prio]) continue; - /* - * Skip if the object is already re-compressed with a higher - * priority algorithm (or same algorithm). - */ - if (prio <= zram_get_priority(zram, index)) - continue; - num_recomps++; zstrm = zcomp_stream_get(zram->comps[prio]); src = kmap_local_page(page); @@ -2154,7 +2161,7 @@ static ssize_t recompress_store(struct device *dev, goto release_init_lock; } - scan_slots_for_recompress(zram, mode, ctl); + scan_slots_for_recompress(zram, mode, prio_max, ctl); ret = len; while ((pps = select_pp_slot(ctl))) { From patchwork Fri Feb 21 09:38:02 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13985085 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4DB62C021B3 for ; Fri, 21 Feb 2025 09:39:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DB1116B00AC; Fri, 21 Feb 2025 04:39:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D610C280008; Fri, 21 Feb 2025 04:39:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C01D1280004; Fri, 21 Feb 2025 04:39:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id A0B076B00AC for ; Fri, 21 Feb 2025 04:39:36 -0500 (EST) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 63FFD80BB6 for ; Fri, 21 Feb 2025 09:39:36 +0000 (UTC) X-FDA: 83143454352.21.7063D2C Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) by imf03.hostedemail.com (Postfix) with ESMTP id 788712000A for ; Fri, 21 Feb 2025 09:39:34 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=OQwAwACR; spf=pass (imf03.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.172 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740130774; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ooRAT30N+1IineDWVp9J6MTWf65J1t58JBJO7W1ETBg=; b=eQLz/rJ+poIURn8QW78eiXBJqt9tw8zpqktarF4RQEKPkj0M9xMYG+BRwW6ZX3W4rN29FT TvG9X/azw+jYXMpU4K49pLqk3TGfW0FXx3LjDjfsO4/EvGUJJQ6jYt+uvWrMw17Rd/0F07 v2AM+vw4OuIpacmdDy416K7CE8FxL2c= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=OQwAwACR; spf=pass (imf03.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.172 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740130774; a=rsa-sha256; cv=none; b=1Vxht/hZ752nFUN1ucNC0MMBIhbpvFfJWNxhMRij6sVjFwI9nxI8w/AwS7UVnhZ3iKy2SN 9PpT7M4JmrX7vuTWNwX1SJ28D98EW9E8Bkw7g2hZdBkbfoQrSCFhX9DWGYBNsGfaBS5YQU /NQlSkTg4PkzRTFzHEh4HJ3uvEJn1cQ= Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-21f2339dcfdso32869645ad.1 for ; Fri, 21 Feb 2025 01:39:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740130773; x=1740735573; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ooRAT30N+1IineDWVp9J6MTWf65J1t58JBJO7W1ETBg=; b=OQwAwACRw1jZHrE8CZ0M+HrRaxdKNipgWzCacgtbNuHMjR/Azk4srd8KhftoKtUyrh IX1TxSj63m6c2f8xJdgwuZth3I+x1GsjUSDh1+okzawcmpMhD9Ek1erRjBktgC5GLObS TKizPhihHtTCuGYTRHQdefHw/7ZKxba69jNK4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740130773; x=1740735573; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ooRAT30N+1IineDWVp9J6MTWf65J1t58JBJO7W1ETBg=; b=V839gEw/g0aSw17qSpsOMAVmctS/U6sZEXVq22lhPSUolHS5XiLBw0TrGtJxtDapdu DdnE8Ojw0/3wctrmgO0dn/FKkx17JEiQT94+X0mrtJd7XTD4fLblmxWZtHtw+RMgdfZp hhbEW6Zj8mp2CEXbJrQF6/a/j+lzSBN/jYScpr75Qai1Npwa6X+i3v2KLQfWJ34iE7HV Wb0vl4/eKExRDV5R08imnk3QvO4TZojetaI4wuge7ORbvZhgq0+/gLW8PvSifKWe72Ld Y6RfnUuPD1LVPGaz7hY2vHZoLn7ce+fkDDHZoVEd8d0x2aDx/6bWxc0avvDvLImh87nk d1ew== X-Forwarded-Encrypted: i=1; AJvYcCWkgOvEkH7FepsOUC1AZhvGPWLv61RYQSUulha01V8HJakPm56JE8CEyyXiwY3ziXkyl+UZdCNaPQ==@kvack.org X-Gm-Message-State: AOJu0YzzLQiqyyngaG7ZO8hJEZE225rulJJkqCuUPXOaE4C+JZKGAst1 TouvHz0WRwYYcxw4AaW9RELKwcv65anID6lcc+sBciikUQfI/DJmq4I0jwnx0g== X-Gm-Gg: ASbGncvpq40u8rOl8YL1gcO55hcxxYZJtBk2FVCnQfRNoUhZI/O31NhhMsXc9FQIu3m tH2qPi2V0GThuTlg24MAmHK6FMuEI7gAPb235xTS/zwptBoQuKR7F2//6zhXS8BOoNfbB6DqCDt zpViGnHpysdw+dq/D4vCROWhQ2AwydU6JGe58ayL9kBWKQMiMD7rS2HNkqUYcIjqQ8puLDgqUiE F75E0ldhLBVPFeStI1wO+TclPpwtRNvzjrI/394Vc1av9gageiCCDWh/irxGtMb6vQbS3vMRgmE F6SKa+fbG0NV9B5wLKSoV+YMFOo= X-Google-Smtp-Source: AGHT+IGRF3A51Ju1CJymIu4QEKwvzsIx5m8/kDSMhapHrOeLOdsD4MZEno2i5dRKAI8Yq2xbHCHWYA== X-Received: by 2002:a17:903:2311:b0:21f:5063:d3ca with SMTP id d9443c01a7336-2218c5575dbmr109218605ad.16.1740130773425; Fri, 21 Feb 2025 01:39:33 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:f987:e1e:3dbb:2191]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-73242761538sm15686426b3a.139.2025.02.21.01.39.30 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 21 Feb 2025 01:39:33 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v7 09/17] zram: rework recompression loop Date: Fri, 21 Feb 2025 18:38:02 +0900 Message-ID: <20250221093832.1949691-10-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <20250221093832.1949691-1-senozhatsky@chromium.org> References: <20250221093832.1949691-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Queue-Id: 788712000A X-Stat-Signature: ub347eg9i5fufiu8yw8aaknoji7rkrcn X-Rspamd-Server: rspam03 X-HE-Tag: 1740130774-397378 X-HE-Meta: U2FsdGVkX1+irMgpy0LpmtClfV8sPLQGf/jfFPmnrCj8M5//aBiavsDBXhvDETr0iDgj67cKTyYYPfYLITw/LEO7YCAKSLuZuTDsqhBM4YqJ7/qlWPjVjFIJLn2z+TYpVlcH/InKE+MLOXqth78ZyZjluJ+rLKiZdl7Mq8QJE1UZtc2tAymBiyAT+Hn/7P4/cSS+J5t2pRai+M7Fnu61IUA5yTWDmM837fUiM5nuDJJwn9bsgW1hKQ+QpEjYSGLqaV9mP7wUAt+EUMq39dlxSoAhatFuYnd9xnqRCxDDdYrB9yGG11VIj5J67dv2HeKl/JFCjsRrmUz1J8nkNRRfOHy+gQHgkWeuxuPLFo6Tj5/y4NivPNLxEI535VEJ/obn08Myx625yQCnrcuVroRKaVBLY59QrZv/OJi8dlgi0Mrb+bhkoo1yeAXvLxA1MtsH+l22yiQigMNjQV6REBwXLa3JnUoQsch5z2FVqqVKlD78xOShirKuD9yEche8Ng5pWTgoBta8VDjpxW7zbkZ9WxriP0lmY6C/7PEJMy5f33LeTKbo0sAgdkVogSgy8p4gZSebu5HRtuqrJ9VsLlW8hqkyGjyS3m0StYU7CX1lIQo9tDJ9rgKYS0/JMQhTexJmyx5DQ4SjdhMsqO+Ij8IZpq/mq0Lf25p336tKXsDtttLlfgWXnaWWh6Lm1b786gYDOuTYFWOg+m4AT4/PftW4qs4O+c3I2/kxvN4Pe9/G1FnEYRq0AhWA72XL++Mx48tvnmSY6PbdyfBi/XT5wBh2ObcsvqWP+ao/eaMqXBueuwLJe1xp93FOV0slyXG5V6cjbwJ32+R/lqVUEtyCWSgKjxpFRYLEvvqvIuJ4x3P26Fo0erseUiU2DtzOwNHiVo7lFUAiK3E+LMz6TaYzhNrGvMrmizGGGLtmCstBHnTnm8j/DJKa6jXP2c2NtOGckfS8Ith0hojokUxbhGI6tCU 7VhXf6fF DOO9BsDzSMtHtqf4VEZIFS2mZFHhAEYvJA3NHRzfra4oAvZBFRh2O4YmKLuxs6Qb6pZBwjD1vTUDDeLO9OvZS6ledcGi9tp8U9tJJ2woA6NaDh1ilAtcI8z9kzGXolgilnDl786dEXHZRDlb+Im3cbAJzDrlADrP8fWE9jJFNSoBZSyZ6hHX+BMMk9+sL9VUIUs2IrfJtONamh50l4uZnaw9M+NQqjMkW1+APunRF8oT0lsY+Kz8zmRni2kjuNAcVJgF4nK4GTgIKpKNU1EuU9T0RXU+wHQx4u1d2sEDno6kV51jD4l6rpTaWDw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This reworks recompression loop handling: - set a rule that stream-put NULLs the stream pointer If the loop returns with a non-NULL stream then it's a successfull recompression, otherwise the stream should always be NULL. - do not count the number of recompressions Mark object as incompressible as soon as the algorithm with the highest priority failed to compress that object. - count compression errors as resource usage Even if compression has failed, we still need to bump num_recomp_pages counter. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 53 +++++++++++++---------------------- 1 file changed, 19 insertions(+), 34 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index a1880e621895..814e64f4ee38 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1892,9 +1892,8 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, unsigned int comp_len_new; unsigned int class_index_old; unsigned int class_index_new; - u32 num_recomps = 0; void *src, *dst; - int ret; + int ret = 0; handle_old = zram_get_handle(zram, index); if (!handle_old) @@ -1937,7 +1936,6 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, if (!zram->comps[prio]) continue; - num_recomps++; zstrm = zcomp_stream_get(zram->comps[prio]); src = kmap_local_page(page); ret = zcomp_compress(zram->comps[prio], zstrm, @@ -1946,7 +1944,8 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, if (ret) { zcomp_stream_put(zstrm); - return ret; + zstrm = NULL; + break; } class_index_new = zs_lookup_class_index(zram->mem_pool, @@ -1956,6 +1955,7 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, if (class_index_new >= class_index_old || (threshold && comp_len_new >= threshold)) { zcomp_stream_put(zstrm); + zstrm = NULL; continue; } @@ -1963,14 +1963,6 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, break; } - /* - * We did not try to recompress, e.g. when we have only one - * secondary algorithm and the page is already recompressed - * using that algorithm - */ - if (!zstrm) - return 0; - /* * Decrement the limit (if set) on pages we can recompress, even * when current recompression was unsuccessful or did not compress @@ -1980,38 +1972,31 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, if (*num_recomp_pages) *num_recomp_pages -= 1; - if (class_index_new >= class_index_old) { + /* Compression error */ + if (ret) + return ret; + + if (!zstrm) { /* * Secondary algorithms failed to re-compress the page - * in a way that would save memory, mark the object as - * incompressible so that we will not try to compress - * it again. + * in a way that would save memory. * - * We need to make sure that all secondary algorithms have - * failed, so we test if the number of recompressions matches - * the number of active secondary algorithms. + * Mark the object incompressible if the max-priority + * algorithm couldn't re-compress it. */ - if (num_recomps == zram->num_active_comps - 1) - zram_set_flag(zram, index, ZRAM_INCOMPRESSIBLE); + if (prio < zram->num_active_comps) + return 0; + zram_set_flag(zram, index, ZRAM_INCOMPRESSIBLE); return 0; } - /* Successful recompression but above threshold */ - if (threshold && comp_len_new >= threshold) - return 0; - /* - * No direct reclaim (slow path) for handle allocation and no - * re-compression attempt (unlike in zram_write_bvec()) since - * we already have stored that object in zsmalloc. If we cannot - * alloc memory for recompressed object then we bail out and - * simply keep the old (existing) object in zsmalloc. + * We are holding per-CPU stream mutex and entry lock so better + * avoid direct reclaim. Allocation error is not fatal since + * we still have the old object in the mem_pool. */ handle_new = zs_malloc(zram->mem_pool, comp_len_new, - __GFP_KSWAPD_RECLAIM | - __GFP_NOWARN | - __GFP_HIGHMEM | - __GFP_MOVABLE); + GFP_NOWAIT | __GFP_HIGHMEM | __GFP_MOVABLE); if (IS_ERR_VALUE(handle_new)) { zcomp_stream_put(zstrm); return PTR_ERR((void *)handle_new); From patchwork Fri Feb 21 09:38:03 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13985086 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C9BA0C021B3 for ; Fri, 21 Feb 2025 09:39:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5C25F6B00AE; Fri, 21 Feb 2025 04:39:42 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5736F280008; Fri, 21 Feb 2025 04:39:42 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3EC2A280004; Fri, 21 Feb 2025 04:39:42 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 205BF6B00AE for ; Fri, 21 Feb 2025 04:39:42 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id A2AFF1A0BA6 for ; Fri, 21 Feb 2025 09:39:41 +0000 (UTC) X-FDA: 83143454562.04.4002AB1 Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) by imf08.hostedemail.com (Postfix) with ESMTP id B8CF416000A for ; Fri, 21 Feb 2025 09:39:39 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=gJ+U52h1; spf=pass (imf08.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.178 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740130779; a=rsa-sha256; cv=none; b=phQot4rgOqobbNCPLTkrs2uioANN5Fmcas8PUQd/G+OfUYcTmjxFELFQysK7vTxcDzrMMt 8asBBCVvbyJKATBO6ASGqxBUsCdr5khJV2zLxRlHxTKyrdlGN8L6NGNT0S258Gd/oZcYxj KTjYm75f/SuQ5xFnAGpAA2BO8aX0sb4= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=gJ+U52h1; spf=pass (imf08.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.178 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740130779; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5HvQEcjJeXsHC85jXzgvN7MoOE9VPDedZeSd2dLCsl4=; b=GXv+/ysnpWhGCjHCfYNcyvUjP7ajCktC9DuW161tgaZFhbBESBLrU9MumpyLpXdfjtWVVI epi8W3LGzuSfNWYuJmPj5i+wOZbsQEI21roQGjy5GrjibJHZmZ8qH4xqeMUrRkJe7gUqO4 NkrqPLqkj/Q9KxGKYJrlK4z1bDIspFQ= Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-220f048c038so34294475ad.2 for ; Fri, 21 Feb 2025 01:39:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740130778; x=1740735578; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5HvQEcjJeXsHC85jXzgvN7MoOE9VPDedZeSd2dLCsl4=; b=gJ+U52h11Uka+VemE+FkjTa+/WX3B6B2c8FeN2xHFpAuhQLVHwkZWjrWUkso+2rBQH XLSI5yBEHYbIoTMQxwUuBgFR2Nc8jZF0BhbJkVjOSpfZZ0PqkDBnR0aGH5K9iOFg1hwg 1QBDBeh1/hFuwaP6qI9odh+v2Ojcr6D1Yzzh0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740130778; x=1740735578; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5HvQEcjJeXsHC85jXzgvN7MoOE9VPDedZeSd2dLCsl4=; b=qW49lsOXLrlJk915SqVKKl6XkLuqzVmNnNeDGORHCDyy6b7lVU/C+VNotNodUz5+vV VU2/5of8sDX59AGHYEUVuBH1bSSFX2rqNoFigfaesfIT1xHVHvcgsaGKZ7H2tvHpn/+E RlLPCfdafthBKuWHiY5NPunnpfiqVFu+mJiWdgCMKZilzNqQxppI2JfoHRy5TyxQmTXO +baBt2KfWXbEjEi94Fvk/K8ikuGSRU5fnn4IoIxTcBtMXCA46sjUz2Y18ED1R5IxOVhr GeQo5J3+8X6V7hh5rWn8DOGQTPL0JquHnPskpWBBd1WSAJsessRqQE/E7ZB5uzexiq+X rRxw== X-Forwarded-Encrypted: i=1; AJvYcCXv4g9L2vT6Vgd9hpiyNUQqEFZEFMXIOl/hwIw73P1I+1mwJHRxhl5AVmg9h93s8aT2sINVW03ufg==@kvack.org X-Gm-Message-State: AOJu0Yxbs9K74BqlB38rDf1uVct9//V0Tb0yIsY8K3yjUegfxdz1+cxN Yt+2kpQgsZFG+TWfKkZEaUJb5WwH3ZTSX/hBchTChJSDkH5wjbMStbmWq8RVwQ== X-Gm-Gg: ASbGncvjAjPzYKHGbPvj9XW49Awlr7x3Pc+XPf9d6GrtxAyPiw3vwq7aGT3hN+wtRpH RAWNNA+vRuhovkoKp4fFeaKq4wCC+1E6lHOEewp86SLYw642vzJ91+TNlrpOMjRO1zPtTujczMs tlRrgAqqFEiSUm6Hik4D5o7E48Jv9xDk5sIAoFLgyvtyPefC+eBNJhStuvBRFBY3jrDnScgvlpV ZZ9tpANt6OT47XGia+pgy2Uk8EHxOyuVIcNaSsQyTjfwmqZic4iygVOHC32+36NoFIx81VFQQX/ +ZJXU0L8hJu6HXdYEvEPgUtFNuI= X-Google-Smtp-Source: AGHT+IFJSxFh0kIH2ZWbOryRmu8nKtsDecZCxDfSknljeAwZwRajmYNb6qKYJjX34w+hU0iVAuRclw== X-Received: by 2002:a05:6a21:6004:b0:1ee:c6bf:7c59 with SMTP id adf61e73a8af0-1eef3cdcb24mr4527195637.23.1740130778665; Fri, 21 Feb 2025 01:39:38 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:f987:e1e:3dbb:2191]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-73274bdf0aasm9692864b3a.125.2025.02.21.01.39.36 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 21 Feb 2025 01:39:38 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v7 10/17] zsmalloc: rename pool lock Date: Fri, 21 Feb 2025 18:38:03 +0900 Message-ID: <20250221093832.1949691-11-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <20250221093832.1949691-1-senozhatsky@chromium.org> References: <20250221093832.1949691-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: B8CF416000A X-Stat-Signature: cpxmwiqpjqdwe87ii3ja9nez6ehg6s1i X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1740130779-637034 X-HE-Meta: U2FsdGVkX1+60oHkN4KdbB98IgIVR5dp5rg4gdmQ6JXv1QdEEkIp0rODFDczQBZhVmlThsRA1VuB28Goesieb8UWMt7L3RDDgPFHFUYmJF6xCSnMpdSQdymwMA7NBJVdc7Bq3BFEb1q4ahuv2JZXHQ99C44guXv0+e8XU5AqeGxMFRrbTEyHKfftY8fHJkXVUM7rF/IPEHAqG2n9g5ex5KSN+bselFNFvyV/RRLz3EBp89He00dkeiwRQnMVITpjmOW/MxXvhBd4p6GtXqp6oSpks2354pdZ7soWCsTCdk42jgknqpu0JTrRBaOTU5qnBQ0TkBaUuvOyTsXI1ElujZvePrdq991NsntrH0gprHcbx2g3pSwuq4/jAb8doH3fDHUZfX++KvJTmcfkB2/lrwu2pZpQt4OPA/UO5Vey6ezaV/8Leym7RPmdUaUQkcBAxx7DznfbzaUWb40uAF288iSXfW5SdU7CnusR+m1OIkRiWkfMc+oIaNTxvyGxfWewiJii/UK7afssB/y/JNxJ9s90cjTgFpi81Y8IQz5y/xSbWk5rc9l4wIiiyRAqvidjFNzm38+TPzG7N5vOqbSUe8ZJbB/Q8jiqsiVUhpVfnNYrOjz5YRxR0nTU0keGeCAUxSIejUK+Pn4bdeIagb4TOBo0XH3f1jM6W0wQ+LwFylTZ+GxkoRii0Iit9I5/OSD2TofzCY6bdQ6FuqnsPfUT5vSeFaPL0iCy2mvXuh9zPfw2JW6xaO7kYIUJ2TSocakYv1+wTw+vY6Xv/ygv74rfBqHmLzQzD0vROHr6ozFVvuEpKlgEXqk119b1WxqLsQzz3f2Z3dz7OLu+LDIoVVBaw9H8ta6J0FSm4y9p2AsfmIv+opmQAfxjzIcI9/T0/YkO/I73lp8fHq/tSfrGd2Vrwi88oGfQ+slIF0E2YNITaKZuolXbw2tPKp7GY/FAizwSXq9qsItccJqRVFpZ/b5 rH7TxO3S GQQNAr1CLUWq7922Wa0rok3RyO6PSKHeT9BxmYlg/bs6JrzQ354gZC9Q8ymzFTzdhRqzkNVPO2x/XkoxJfpm7vLhw/XiKSOdnJOQucZ1uEY2NcHXE6/jzVKxRvjLcTz6RCPXu/ido5AwPHATA/+ja73h60LdcJcK7Lj+zOLANFHFKbW/RhIt5BxJKu7WMRbx56xs1ZVSgih39qAejp75Dco8IgfH16yFOAL58Db57/P9cDFw1ZhLxU+gWRkzC8le2M9BXqoZrLFadaWqqnOmciPPkMVh8zKqw0W6qtWsSdvfUgnk3/hGsplrJig== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The old name comes from the times when the pool did not have compaction (defragmentation). Rename it to ->lock because these days it synchronizes not only migration. Reviewed-by: Yosry Ahmed Signed-off-by: Sergey Senozhatsky --- mm/zsmalloc.c | 38 +++++++++++++++++++------------------- 1 file changed, 19 insertions(+), 19 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 817626a351f8..1424ee73cbb5 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -18,7 +18,7 @@ /* * lock ordering: * page_lock - * pool->migrate_lock + * pool->lock * class->lock * zspage->lock */ @@ -223,8 +223,8 @@ struct zs_pool { #ifdef CONFIG_COMPACTION struct work_struct free_work; #endif - /* protect page/zspage migration */ - rwlock_t migrate_lock; + /* protect zspage migration/compaction */ + rwlock_t lock; atomic_t compaction_in_progress; }; @@ -1206,7 +1206,7 @@ void *zs_map_object(struct zs_pool *pool, unsigned long handle, BUG_ON(in_interrupt()); /* It guarantees it can get zspage from handle safely */ - read_lock(&pool->migrate_lock); + read_lock(&pool->lock); obj = handle_to_obj(handle); obj_to_location(obj, &zpdesc, &obj_idx); zspage = get_zspage(zpdesc); @@ -1218,7 +1218,7 @@ void *zs_map_object(struct zs_pool *pool, unsigned long handle, * which is smaller granularity. */ migrate_read_lock(zspage); - read_unlock(&pool->migrate_lock); + read_unlock(&pool->lock); class = zspage_class(pool, zspage); off = offset_in_page(class->size * obj_idx); @@ -1450,16 +1450,16 @@ void zs_free(struct zs_pool *pool, unsigned long handle) return; /* - * The pool->migrate_lock protects the race with zpage's migration + * The pool->lock protects the race with zpage's migration * so it's safe to get the page from handle. */ - read_lock(&pool->migrate_lock); + read_lock(&pool->lock); obj = handle_to_obj(handle); obj_to_zpdesc(obj, &f_zpdesc); zspage = get_zspage(f_zpdesc); class = zspage_class(pool, zspage); spin_lock(&class->lock); - read_unlock(&pool->migrate_lock); + read_unlock(&pool->lock); class_stat_sub(class, ZS_OBJS_INUSE, 1); obj_free(class->size, obj); @@ -1796,7 +1796,7 @@ static int zs_page_migrate(struct page *newpage, struct page *page, * The pool migrate_lock protects the race between zpage migration * and zs_free. */ - write_lock(&pool->migrate_lock); + write_lock(&pool->lock); class = zspage_class(pool, zspage); /* @@ -1833,7 +1833,7 @@ static int zs_page_migrate(struct page *newpage, struct page *page, * Since we complete the data copy and set up new zspage structure, * it's okay to release migration_lock. */ - write_unlock(&pool->migrate_lock); + write_unlock(&pool->lock); spin_unlock(&class->lock); migrate_write_unlock(zspage); @@ -1956,7 +1956,7 @@ static unsigned long __zs_compact(struct zs_pool *pool, * protect the race between zpage migration and zs_free * as well as zpage allocation/free */ - write_lock(&pool->migrate_lock); + write_lock(&pool->lock); spin_lock(&class->lock); while (zs_can_compact(class)) { int fg; @@ -1983,14 +1983,14 @@ static unsigned long __zs_compact(struct zs_pool *pool, src_zspage = NULL; if (get_fullness_group(class, dst_zspage) == ZS_INUSE_RATIO_100 - || rwlock_is_contended(&pool->migrate_lock)) { + || rwlock_is_contended(&pool->lock)) { putback_zspage(class, dst_zspage); dst_zspage = NULL; spin_unlock(&class->lock); - write_unlock(&pool->migrate_lock); + write_unlock(&pool->lock); cond_resched(); - write_lock(&pool->migrate_lock); + write_lock(&pool->lock); spin_lock(&class->lock); } } @@ -2002,7 +2002,7 @@ static unsigned long __zs_compact(struct zs_pool *pool, putback_zspage(class, dst_zspage); spin_unlock(&class->lock); - write_unlock(&pool->migrate_lock); + write_unlock(&pool->lock); return pages_freed; } @@ -2014,10 +2014,10 @@ unsigned long zs_compact(struct zs_pool *pool) unsigned long pages_freed = 0; /* - * Pool compaction is performed under pool->migrate_lock so it is basically + * Pool compaction is performed under pool->lock so it is basically * single-threaded. Having more than one thread in __zs_compact() - * will increase pool->migrate_lock contention, which will impact other - * zsmalloc operations that need pool->migrate_lock. + * will increase pool->lock contention, which will impact other + * zsmalloc operations that need pool->lock. */ if (atomic_xchg(&pool->compaction_in_progress, 1)) return 0; @@ -2139,7 +2139,7 @@ struct zs_pool *zs_create_pool(const char *name) return NULL; init_deferred_free(pool); - rwlock_init(&pool->migrate_lock); + rwlock_init(&pool->lock); atomic_set(&pool->compaction_in_progress, 0); pool->name = kstrdup(name, GFP_KERNEL); From patchwork Fri Feb 21 09:38:04 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13985087 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ECB6FC021B3 for ; Fri, 21 Feb 2025 09:39:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 85E7A6B00B0; Fri, 21 Feb 2025 04:39:47 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 80A2C280008; Fri, 21 Feb 2025 04:39:47 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 686EC280004; Fri, 21 Feb 2025 04:39:47 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 48EE46B00B0 for ; Fri, 21 Feb 2025 04:39:47 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id F10E1C0BC6 for ; Fri, 21 Feb 2025 09:39:46 +0000 (UTC) X-FDA: 83143454772.04.A3C54D2 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) by imf28.hostedemail.com (Postfix) with ESMTP id 0CAF7C0002 for ; Fri, 21 Feb 2025 09:39:44 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=jbV24U1I; spf=pass (imf28.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.182 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740130785; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=c43sUKc+c1340lwZZrmUjsY+PbdZGHYK9fwL3MZ7pbk=; b=13X4v9jV3LXjYCb+DuC2AHAk6XEJlYBht923dhYtjbrPbAZ1juXPjInbi1GHFLRnIuvqZN CHWsj8RBnW1TRvZCCzyriNSVNHThMznpSv5re7AaNLiFYWMApANn2GuKxRf8Q/QESSEHUs SOqi7ty1yhr1IrVLmLBhqfzB9rFZGPc= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=jbV24U1I; spf=pass (imf28.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.182 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740130785; a=rsa-sha256; cv=none; b=b8F++FfxQ2V1Yr4Zgqej9GOZgxDQ5deAIICuq4NeXBUx3pbHD6yIRn4qBNPLoI71hs1lkg t++o73h5T7j/xQCYhawX/NIjv44NgePM58G0H9NqOdR7TETyJRmcTN1i/9XFn0tn8CsrkA m7zZqub+kkc+7PlZWkqtPA4H+3V9cVI= Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-220c665ef4cso30668425ad.3 for ; Fri, 21 Feb 2025 01:39:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740130784; x=1740735584; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=c43sUKc+c1340lwZZrmUjsY+PbdZGHYK9fwL3MZ7pbk=; b=jbV24U1I/+QTU0v0GvihGFu8ekIFwefAXOlEu7bWXL1jTiQh8hQrq6vSlwU4cPCzPc 2rj3P30KAhVDKd5LbmhIOkh14ypqmuaEbBx5hieVblVK7nz1+4jgrBuzYZy/O8kCBdvq p4zEySu9aROv8GE29ZIpfRZjULlx6/G6ZBJ6c= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740130784; x=1740735584; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=c43sUKc+c1340lwZZrmUjsY+PbdZGHYK9fwL3MZ7pbk=; b=cYe4EcgxLUOYUfRE3aiHcVucD/axamEUB+2r0RQaY+TPwSgzOQjEzySubGLpvRfcvi A7oGl3UeRYLKbz8T6Of8IhNjRDCQAE6Jo8zq6PFjyXRO+udAUIs98Q0Ywa+Qr4syszrz 6Dm6sHuSVUaQPY2TuSMQtg7RP0SvNuyy8QebPVXz8U6ciNLqr5p+NJOaAR7LQRAVxRs3 rGI9zi/dcky6G4164svSdqxIck1R0bSiNCmILiesz/rRsNiIWa3Y0igqqq+er16I2bxP kgHEwpyTqSKLfTRv3DPpOj3Xl5WsJb5963lj0R2btDuufwckFs+x6F68FpW4Nuvjsj3k /+Jw== X-Forwarded-Encrypted: i=1; AJvYcCXz8Bn02BTN0uo4fXmWljPjcHTuiCG6O/IxkDeZP83f2dBWEIh3Kn/VizmiOTdTxMPKN21gfbypeQ==@kvack.org X-Gm-Message-State: AOJu0YxKpmIjZRcY2S4pY0BJLDL5DdBdagOYysErf+RvClWNtId7wX2e VtSdBE27VyCGWMctRvfX5USj1W2QDnztzOR8LCtomo49CLvDi1rZa1lGyH2VIA== X-Gm-Gg: ASbGncupGWAJw0g9xwiv9F6jSvXKeiztzrDt4kzT5Od+Wa0ZWigtPYktZa4Dls0rjwm mDn2aqqsRg0pICsc3HTKKq2hDe2hRWzkk1rK8gseBtjLP2zBXGcAlu+oMjdaht1c05Z5xjJzg4j MZ/OialGG/X2G510hNng4wuJUsLTJPf1n/uEHXCpIr6eLbiWnW2PIJhlfaNGEUSngqIYTAfQqK9 ADVKc5rwkA2XA0XNS9rJlONo2XBDBrwiF4gj5uTc3MyzH1FHb9rPGlZ7kHpwY7BqAbaf86Y1XrB jvkiyKr40tWQhZcW/8i1jp6Ui08= X-Google-Smtp-Source: AGHT+IGQLO+1U1vS6Lct91u4p8i4NavSONuQ6g0x/XphxsfnVjWvF85vBjeEIzZsrJ0hPSWktsRKtw== X-Received: by 2002:a17:902:e783:b0:219:e4b0:4286 with SMTP id d9443c01a7336-2219ffc00d7mr41532145ad.29.1740130783941; Fri, 21 Feb 2025 01:39:43 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:f987:e1e:3dbb:2191]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-220d545d4bcsm132919005ad.144.2025.02.21.01.39.41 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 21 Feb 2025 01:39:43 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v7 11/17] zsmalloc: make zspage lock preemptible Date: Fri, 21 Feb 2025 18:38:04 +0900 Message-ID: <20250221093832.1949691-12-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <20250221093832.1949691-1-senozhatsky@chromium.org> References: <20250221093832.1949691-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 0CAF7C0002 X-Stat-Signature: nbrcxt8zaynf34ghkpfno3a8ni7jrwur X-HE-Tag: 1740130784-729288 X-HE-Meta: U2FsdGVkX1/pJcemnVWSMUaPkqt06MeUULAJqQeMkcHU5u+jX9Siffh1jr2Pm6G2jQ/RqqHdnVsgWKnjue7sye606ankVZjO6U3zHNHNOPDgcAvuSyEvMFF3uHxRyYK+0afr6iWlJYiKUflV/f7Vm/fE8GjKv53QDJSy+mrEmUgsQH+6P0GmqQ+bXXQoKYe3Fn0wzeE/Z11tnlH+Fk/zkBXmP5XM1DZNwCxCQsS0Dbc8Qfjy4Cs5YeVFcUsfl0kdWTjfI0AkkIjUowsvhi7ow/D70lANyJsDl5VmVnDnrb54B5AZAV+5Jhk/cDz64nTrHjFEJo3vlROKtmXZL56JHtsNq5k52gRtjEkJ/Bn2OYlyL7IgB6ObNuQvIDnK7CJlCLUKRge48Sa6A/IuR/PCjNAgN0CJY52CTOXY6Mw1XbgaYo6pKP/z1fWNYu4EWd5bDr6IQ5BlKyhqo1Scq6pwdHfuJC8ofG4OnBAwVxA8cCpd9QW7m6NMU9l8VYvrMfga3PxBE2x6ZtHL3ua9gJb4cZDPNAHVptwGoEPoa/ymVVezgG/4kM/kMX3oTPWsgIKiWgzbpJ/f26udW6vOk8t9QNN1DebKTS/b5+hezxBD/NuLh2lLrtFFdZxPUoXMw5NkOCz6yn3xdvC7xbf+yIpxAqHK6vtojIDOQYAOhIZsimB8pzk830H1GVdF7d7fGIPAxWi4gdoZ0sQHRRywiXfxzOMfqL8LpaujsSYE7fojz61LRkHFxqbXDlSnn7WbQZSkodQL4gaUNooBAFcDZulDL/aXj6ATU1ZafejTRJJEGg9RZBHn6yr4ZzyB343/NOmLpnv78o8xX8gqR0jhGNCnlOn0Z0BHeeH/AbHsaTiVBZ7TCwvzEWOQ7heQD5ruBMmM/1Qjcrn6FZ1y3pzR0E6lHDWFWH313tBq/SWtEJFsO7Pyxe1O9os3tY2/GA93y/Dyb1KTZ1jrqQrGiJpyQRq hcLxaK/y 8z661OXAf5gWsuTc2In4M1jHLhsSlBH1zPYBOSEl8BTg1yAorhhfk4CeUJHhgqW9sfQVXF5bOGtW5jQyrFOeTAmGpvLuyDY8SHfvVEY0BvGANPGO1eKnxmI2nVtyNBz/dyEMj95v09k2u9w2TfgOmUu2ofi8OJV4cju15NQT51pmR3/m7cXUgH3QMY/iYN+QC7R5DkgcMSUm+3T8U1ILL6NKLck0OCkEm7hkQsPgQ8t3gqzzx670lXJ0qk9Ks95HFyTfGYVf+OzklAmfy7ibOWBoHEi7Azq3jVzP3jrrUgFh9zASpP4XZIFbVrFdfNArV3SPRtvtK12FrjqM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In order to implement preemptible object mapping we need a zspage lock that satisfies several preconditions: - it should be reader-write type of a lock - it should be possible to hold it from any context, but also being preemptible if the context allows it - we never sleep while acquiring but can sleep while holding in read mode An rwsemaphore doesn't suffice, due to atomicity requirements, rwlock doesn't satisfy due to reader-preemptability requirement. It's also worth to mention, that per-zspage rwsem is a little too memory heavy (we can easily have double digits megabytes used only on rwsemaphores). Switch over from rwlock_t to a atomic_t-based implementation of a reader-writer semaphore that satisfies all of the preconditions. The spin-lock based zspage_lock is suggested by Hillf Danton. Suggested-by: Hillf Danton Signed-off-by: Sergey Senozhatsky --- mm/zsmalloc.c | 189 +++++++++++++++++++++++++++++++++++--------------- 1 file changed, 135 insertions(+), 54 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 1424ee73cbb5..250f1fddaf34 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -226,6 +226,9 @@ struct zs_pool { /* protect zspage migration/compaction */ rwlock_t lock; atomic_t compaction_in_progress; +#ifdef CONFIG_DEBUG_LOCK_ALLOC + struct lock_class_key lock_class; +#endif }; static inline void zpdesc_set_first(struct zpdesc *zpdesc) @@ -257,6 +260,18 @@ static inline void free_zpdesc(struct zpdesc *zpdesc) __free_page(page); } +#define ZS_PAGE_UNLOCKED 0 +#define ZS_PAGE_WRLOCKED -1 + +struct zspage_lock { + spinlock_t lock; + int cnt; + +#ifdef CONFIG_DEBUG_LOCK_ALLOC + struct lockdep_map dep_map; +#endif +}; + struct zspage { struct { unsigned int huge:HUGE_BITS; @@ -269,7 +284,7 @@ struct zspage { struct zpdesc *first_zpdesc; struct list_head list; /* fullness list */ struct zs_pool *pool; - rwlock_t lock; + struct zspage_lock zsl; }; struct mapping_area { @@ -279,6 +294,91 @@ struct mapping_area { enum zs_mapmode vm_mm; /* mapping mode */ }; +#ifdef CONFIG_DEBUG_LOCK_ALLOC +#define zsl_dep_map(zsl) (&(zsl)->dep_map) +#else +#define zsl_dep_map(zsl) NULL +#endif + +static void zspage_lock_init(struct zspage *zspage) +{ +#ifdef CONFIG_DEBUG_LOCK_ALLOC + lockdep_init_map(&zspage->zsl.dep_map, "zspage->lock", + &zspage->pool->lock_class, 0); +#endif + spin_lock_init(&zspage->zsl.lock); + zspage->zsl.cnt = ZS_PAGE_UNLOCKED; +} + +/* + * The zspage lock can be held from atomic contexts, but it needs to remain + * preemptible when held for reading because it remains held outside of those + * atomic contexts, otherwise we unnecessarily lose preemptibility. + * + * To achieve this, the following rules are enforced on readers and writers: + * + * - Writers are blocked by both writers and readers, while readers are only + * blocked by writers (i.e. normal rwlock semantics). + * + * - Writers are always atomic (to allow readers to spin waiting for them). + * + * - Writers always use trylock (as the lock may be held be sleeping readers). + * + * - Readers may spin on the lock (as they can only wait for atomic writers). + * + * - Readers may sleep while holding the lock (as writes only use trylock). + */ +static void zspage_read_lock(struct zspage *zspage) +{ + struct zspage_lock *zsl = &zspage->zsl; + + rwsem_acquire_read(zsl_dep_map(zsl), 0, 0, _RET_IP_); + + spin_lock(&zsl->lock); + zsl->cnt++; + spin_unlock(&zsl->lock); + + lock_acquired(zsl_dep_map(zsl), _RET_IP_); +} + +static void zspage_read_unlock(struct zspage *zspage) +{ + struct zspage_lock *zsl = &zspage->zsl; + + rwsem_release(zsl_dep_map(zsl), _RET_IP_); + + spin_lock(&zsl->lock); + zsl->cnt--; + spin_unlock(&zsl->lock); +} + +static __must_check bool zspage_write_trylock(struct zspage *zspage) +{ + struct zspage_lock *zsl = &zspage->zsl; + + spin_lock(&zsl->lock); + if (zsl->cnt == ZS_PAGE_UNLOCKED) { + zsl->cnt = ZS_PAGE_WRLOCKED; + rwsem_acquire(zsl_dep_map(zsl), 0, 1, _RET_IP_); + lock_acquired(zsl_dep_map(zsl), _RET_IP_); + return true; + } + + lock_contended(zsl_dep_map(zsl), _RET_IP_); + spin_unlock(&zsl->lock); + return false; +} + +static void zspage_write_unlock(struct zspage *zspage) +{ + struct zspage_lock *zsl = &zspage->zsl; + + rwsem_release(zsl_dep_map(zsl), _RET_IP_); + + zsl->cnt = ZS_PAGE_UNLOCKED; + spin_unlock(&zsl->lock); +} + /* huge object: pages_per_zspage == 1 && maxobj_per_zspage == 1 */ static void SetZsHugePage(struct zspage *zspage) { @@ -290,12 +390,6 @@ static bool ZsHugePage(struct zspage *zspage) return zspage->huge; } -static void migrate_lock_init(struct zspage *zspage); -static void migrate_read_lock(struct zspage *zspage); -static void migrate_read_unlock(struct zspage *zspage); -static void migrate_write_lock(struct zspage *zspage); -static void migrate_write_unlock(struct zspage *zspage); - #ifdef CONFIG_COMPACTION static void kick_deferred_free(struct zs_pool *pool); static void init_deferred_free(struct zs_pool *pool); @@ -992,7 +1086,9 @@ static struct zspage *alloc_zspage(struct zs_pool *pool, return NULL; zspage->magic = ZSPAGE_MAGIC; - migrate_lock_init(zspage); + zspage->pool = pool; + zspage->class = class->index; + zspage_lock_init(zspage); for (i = 0; i < class->pages_per_zspage; i++) { struct zpdesc *zpdesc; @@ -1015,8 +1111,6 @@ static struct zspage *alloc_zspage(struct zs_pool *pool, create_page_chain(class, zspage, zpdescs); init_zspage(class, zspage); - zspage->pool = pool; - zspage->class = class->index; return zspage; } @@ -1217,7 +1311,7 @@ void *zs_map_object(struct zs_pool *pool, unsigned long handle, * zs_unmap_object API so delegate the locking from class to zspage * which is smaller granularity. */ - migrate_read_lock(zspage); + zspage_read_lock(zspage); read_unlock(&pool->lock); class = zspage_class(pool, zspage); @@ -1277,7 +1371,7 @@ void zs_unmap_object(struct zs_pool *pool, unsigned long handle) } local_unlock(&zs_map_area.lock); - migrate_read_unlock(zspage); + zspage_read_unlock(zspage); } EXPORT_SYMBOL_GPL(zs_unmap_object); @@ -1671,18 +1765,18 @@ static void lock_zspage(struct zspage *zspage) /* * Pages we haven't locked yet can be migrated off the list while we're * trying to lock them, so we need to be careful and only attempt to - * lock each page under migrate_read_lock(). Otherwise, the page we lock + * lock each page under zspage_read_lock(). Otherwise, the page we lock * may no longer belong to the zspage. This means that we may wait for * the wrong page to unlock, so we must take a reference to the page - * prior to waiting for it to unlock outside migrate_read_lock(). + * prior to waiting for it to unlock outside zspage_read_lock(). */ while (1) { - migrate_read_lock(zspage); + zspage_read_lock(zspage); zpdesc = get_first_zpdesc(zspage); if (zpdesc_trylock(zpdesc)) break; zpdesc_get(zpdesc); - migrate_read_unlock(zspage); + zspage_read_unlock(zspage); zpdesc_wait_locked(zpdesc); zpdesc_put(zpdesc); } @@ -1693,41 +1787,16 @@ static void lock_zspage(struct zspage *zspage) curr_zpdesc = zpdesc; } else { zpdesc_get(zpdesc); - migrate_read_unlock(zspage); + zspage_read_unlock(zspage); zpdesc_wait_locked(zpdesc); zpdesc_put(zpdesc); - migrate_read_lock(zspage); + zspage_read_lock(zspage); } } - migrate_read_unlock(zspage); + zspage_read_unlock(zspage); } #endif /* CONFIG_COMPACTION */ -static void migrate_lock_init(struct zspage *zspage) -{ - rwlock_init(&zspage->lock); -} - -static void migrate_read_lock(struct zspage *zspage) __acquires(&zspage->lock) -{ - read_lock(&zspage->lock); -} - -static void migrate_read_unlock(struct zspage *zspage) __releases(&zspage->lock) -{ - read_unlock(&zspage->lock); -} - -static void migrate_write_lock(struct zspage *zspage) -{ - write_lock(&zspage->lock); -} - -static void migrate_write_unlock(struct zspage *zspage) -{ - write_unlock(&zspage->lock); -} - #ifdef CONFIG_COMPACTION static const struct movable_operations zsmalloc_mops; @@ -1769,7 +1838,7 @@ static bool zs_page_isolate(struct page *page, isolate_mode_t mode) } static int zs_page_migrate(struct page *newpage, struct page *page, - enum migrate_mode mode) + enum migrate_mode mode) { struct zs_pool *pool; struct size_class *class; @@ -1785,9 +1854,6 @@ static int zs_page_migrate(struct page *newpage, struct page *page, VM_BUG_ON_PAGE(!zpdesc_is_isolated(zpdesc), zpdesc_page(zpdesc)); - /* We're committed, tell the world that this is a Zsmalloc page. */ - __zpdesc_set_zsmalloc(newzpdesc); - /* The page is locked, so this pointer must remain valid */ zspage = get_zspage(zpdesc); pool = zspage->pool; @@ -1803,8 +1869,15 @@ static int zs_page_migrate(struct page *newpage, struct page *page, * the class lock protects zpage alloc/free in the zspage. */ spin_lock(&class->lock); - /* the migrate_write_lock protects zpage access via zs_map_object */ - migrate_write_lock(zspage); + /* the zspage write_lock protects zpage access via zs_map_object */ + if (!zspage_write_trylock(zspage)) { + spin_unlock(&class->lock); + write_unlock(&pool->lock); + return -EINVAL; + } + + /* We're committed, tell the world that this is a Zsmalloc page. */ + __zpdesc_set_zsmalloc(newzpdesc); offset = get_first_obj_offset(zpdesc); s_addr = kmap_local_zpdesc(zpdesc); @@ -1835,7 +1908,7 @@ static int zs_page_migrate(struct page *newpage, struct page *page, */ write_unlock(&pool->lock); spin_unlock(&class->lock); - migrate_write_unlock(zspage); + zspage_write_unlock(zspage); zpdesc_get(newzpdesc); if (zpdesc_zone(newzpdesc) != zpdesc_zone(zpdesc)) { @@ -1971,9 +2044,11 @@ static unsigned long __zs_compact(struct zs_pool *pool, if (!src_zspage) break; - migrate_write_lock(src_zspage); + if (!zspage_write_trylock(src_zspage)) + break; + migrate_zspage(pool, src_zspage, dst_zspage); - migrate_write_unlock(src_zspage); + zspage_write_unlock(src_zspage); fg = putback_zspage(class, src_zspage); if (fg == ZS_INUSE_RATIO_0) { @@ -2233,7 +2308,9 @@ struct zs_pool *zs_create_pool(const char *name) * trigger compaction manually. Thus, ignore return code. */ zs_register_shrinker(pool); - +#ifdef CONFIG_DEBUG_LOCK_ALLOC + lockdep_register_key(&pool->lock_class); +#endif return pool; err: @@ -2270,6 +2347,10 @@ void zs_destroy_pool(struct zs_pool *pool) kfree(class); } +#ifdef CONFIG_DEBUG_LOCK_ALLOC + lockdep_unregister_key(&pool->lock_class); +#endif + destroy_cache(pool); kfree(pool->name); kfree(pool); From patchwork Fri Feb 21 09:38:05 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13985088 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7AFAEC021B3 for ; Fri, 21 Feb 2025 09:39:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0B8DD280004; Fri, 21 Feb 2025 04:39:53 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 069D96B00B3; Fri, 21 Feb 2025 04:39:52 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E269F280004; Fri, 21 Feb 2025 04:39:52 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id C29016B00B2 for ; Fri, 21 Feb 2025 04:39:52 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 7E1BAA1BFB for ; Fri, 21 Feb 2025 09:39:52 +0000 (UTC) X-FDA: 83143455024.28.DB7F60C Received: from mail-pl1-f169.google.com (mail-pl1-f169.google.com [209.85.214.169]) by imf22.hostedemail.com (Postfix) with ESMTP id 8EA7AC0002 for ; Fri, 21 Feb 2025 09:39:50 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=H+mAIsMP; spf=pass (imf22.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.169 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740130790; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=r+V102a8CvzSvRQ+sxK0ffvIfOAyV8bBQDBTXVAo23s=; b=pXqSqGDu3Xb7GTXrS5qcFjG+/GXSSjkIDpKclPgrSy3c8kwRrxIdEC3ld/nX3STI7B02nc YgvcEugoNMB0B1mHp5n2OoM9/2XPF8Z3NmKtxU+1WVaNCNEmURUy+8+g5jrm5mCc+wLXu8 YXv/RxNSUoMrBTYCSU6Ll3+gfHa6DIE= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=H+mAIsMP; spf=pass (imf22.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.169 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740130790; a=rsa-sha256; cv=none; b=ULC7GBJA+xwLoXEXE5wcQcxtlqYlRIWef+Fq1NCAvZJ0/NMiU1sXMimKbfEEzeOI6ION1X /mRtFyG/pSqqUwNIMdvfhFzGUoPoQZ585X4RfO8y0l5wbpvNejHH3XWL+WCsjn1kpLL037 ZBAg3iKWsilirU7cW2/F+4yMml+KhRw= Received: by mail-pl1-f169.google.com with SMTP id d9443c01a7336-22114b800f7so35341855ad.2 for ; Fri, 21 Feb 2025 01:39:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740130789; x=1740735589; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=r+V102a8CvzSvRQ+sxK0ffvIfOAyV8bBQDBTXVAo23s=; b=H+mAIsMPjDCemiwb7PwLbLxQm0cXRCr8CixkFk736H703JSHTE1jZd/a0Iz+uQd625 RKvd+ynYgJiRm+rC/wYWWqAD7A2HM2vHDRrJ/YAzrseDwARQvFSt2GUVbgA0Ls8CGHNu dHNN/4BIA1qs4zwOMzVWBP6T7xvcRiUWMumzw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740130789; x=1740735589; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=r+V102a8CvzSvRQ+sxK0ffvIfOAyV8bBQDBTXVAo23s=; b=npftPjS1qvtYoGj8DwZmLrEAT+xngBZSt5eWLRKDEm70FiU4nA1+k1827s5D09Jw58 42eXnUv3Hjvra+REDuu124AcVkWE1gfceXX7yhzK4ChoNWa/01kLAEDOAnIE72lmLKr9 t7HStLdxNcypcHokr/JNDjYgKlOr6o+60BB1/P56Xud8vvuZuq07Lj87prT7sjcVGIEo Vr+S0vjWKcKGLmdR8wDKDWHQoSNGXWlWulp0F3o0IcaVeKpKFbU/ASiax2ujzDGnBMY8 z6pTpLMiukEr0WU3yoUWUjVEuGPCgyv7nfaKIYvKjCzk9YTjDznGby/tjD5g6pMluZJl mEeQ== X-Forwarded-Encrypted: i=1; AJvYcCXoDFEZZGdCLTmNNoaKD5mCdz3x8yfIRzIBmtUYe/qc0XX9LN2GFjFAilO64vW8jcjswZavp6F8DQ==@kvack.org X-Gm-Message-State: AOJu0Yzh5/UTCVF4r6+iv24eeeqymuTed7PpSp07vVz1z45o4C1nSOcl aGEzQbVYHt9e7IsvspEjRtKg6+NgqgLTkUV70QOeS89GM5cPhaODwDlMlkR+IQ== X-Gm-Gg: ASbGnctWUb3UsU1Xvmqxw5BuZywsP4H4yf92DzivbGCUS4cycU8WyAelo0PnOHLmn5j G784lTvfnMCZi9GAkfn5vYxc53UZisf8a+IoZCKabZPHi10KmmlIVAtK1jMkI4vV74mYvTk9Iem 8GEfhzrpeNOM1BN5uWDnUxUjRneFET8XkgW3y1yEcd4bdY6cUgqIXacX15F1b2CziCvv+YD4nx3 a2/mnwcinaRdCs4Gp6x2o+YYKhsUEIBKIDVEePq20FPwTV+PaQZOPyTTm2ARG5h3T/UlzDRNCd/ s/IW9ztbP1ZAtngp4QN7qKxkvzE= X-Google-Smtp-Source: AGHT+IFJYSryWZFsvikewIDDOjJ5biVq7e7y/3wvbBGO0tIKWvvm6+aR0zFU9xY7HWK8MXDKvHfkow== X-Received: by 2002:a17:902:f712:b0:221:7e36:b13e with SMTP id d9443c01a7336-2219ff50d62mr52630715ad.12.1740130789480; Fri, 21 Feb 2025 01:39:49 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:f987:e1e:3dbb:2191]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-220d55866ecsm133407275ad.212.2025.02.21.01.39.46 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 21 Feb 2025 01:39:49 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v7 12/17] zsmalloc: introduce new object mapping API Date: Fri, 21 Feb 2025 18:38:05 +0900 Message-ID: <20250221093832.1949691-13-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <20250221093832.1949691-1-senozhatsky@chromium.org> References: <20250221093832.1949691-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 8EA7AC0002 X-Stat-Signature: 4rrwj1tafkqephks9diisy1oreepyofr X-HE-Tag: 1740130790-647884 X-HE-Meta: U2FsdGVkX19X0S2O/tLnr4+utQQ7Lh9+p2kkUtiY1EbX/DPPojqhT7T5kSeHiAJz0Derb78kJ29oCljFc5Ag43FBdovBlwIqW7gA5pkO7k7NU85C1FBGE85tYzJ0mQwABthUnCFUhYQ/BD4Wz/sU1/HMURyrMQN2K2g4TzGMC64UV95h+lYJ/OBNTfsrBCX7YYg3F5Y4W8tiCqmyV2+WQ/hyySgJRS+bbp5edUFY76VzkY4HW5TdunOVRv8Evu090XQLWPRdXyLJS7RhdFijH6p5t5xdDy0/qVo0ol7gP4FPJHcSXBWkfZ4vdH2VxW1fvT7jFib2W9rTgQJDANpw7Q9t7cuf8hBZ3jJ8GOqmQ3U+2KWEI0J+GWBQ7mDGYrIlGe6FC9rpSkGK86fN2LxpuLX9uz5Szlw+JuiojK/AWzMOeuMB1nOYpq83IfcXPJIxRhd6ts3eL9l59USfuVz8MiUC2Frh99IO++iwo6lkcZEfiuN2VToPtgQP1sn7UR+0IsXg3JmSI6QsiyVe09+aAqjYoI9YL7xvWETEOCTptq7iD9erTek+GmKTvQIzyOfhNnwkHwufkXs6pZEt7r9pCMjWh6L1c1Vto+j/qHr8G2410Mpy2o7GcS9KPFA2ubIlp7xCuOXNikbaj+eRAmxZul29hN7sxzesU20rTCunkvAy1l5WTeVhaRE3++vrlzdr1lS7UuGcWe3VUg5k6QL90clbZEUJt21QQfduREAH66t9Ny78+ca7UF6kVje0W+eZOZ7yBpX0tR8up/NVGfeGYtkDleQG7jV6LocS0Jc0rvqu9sEMaRdDA3DCaVbhFo8mmJbMwZCZ/fxaGlnEuJKYLzeSlETChMEYZoDtijwqXv3baCmPYhcdiF0MKZ4ls91XzIJ6FL+iwDxIbS1815Xsl0MrUtZ5/4AryJ65nJG0AQNupJvk8xkj/FNAliFu0qQTyh+xZOlJdD09bHy51PJ Q0wMTh1b HPqg7jR3sl2VVrhYQaBK6UoP5lFq2qGUfovp7jhFF8F8ugfxvnRj46dAYtWvZcrxZlEp788W5O0LpJgoAzLu6jaTT1AVqTAOudeFLEF3LywLnsiykamhcfjQK2S9oK+OPQUk9Bf0zqQn3kcEKiPYk4dgq0HgnlDLTxVT0DlBWV8h2T1l748PDuT/hcxqq88WvzruOxKJ4GTm9PUalJntexjMMViFTbpFbtWxZ7/BRJhooWvX+csnskzq0PHmu/nMRyZ3+IeSQi9omgIcjVxeDHJmFFmnfOJV67tDAbB9mB2IX/Q1D8eF4GUeslw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Current object mapping API is a little cumbersome. First, it's inconsistent, sometimes it returns with page-faults disabled and sometimes with page-faults enabled. Second, and most importantly, it enforces atomicity restrictions on its users. zs_map_object() has to return a liner object address which is not always possible because some objects span multiple physical (non-contiguous) pages. For such objects zsmalloc uses a per-CPU buffer to which object's data is copied before a pointer to that per-CPU buffer is returned back to the caller. This leads to another, final, issue - extra memcpy(). Since the caller gets a pointer to per-CPU buffer it can memcpy() data only to that buffer, and during zs_unmap_object() zsmalloc will memcpy() from that per-CPU buffer to physical pages that object in question spans across. New API splits functions by access mode: - zs_obj_read_begin(handle, local_copy) Returns a pointer to handle memory. For objects that span two physical pages a local_copy buffer is used to store object's data before the address is returned to the caller. Otherwise the object's page is kmap_local mapped directly. - zs_obj_read_end(handle, buf) Unmaps the page if it was kmap_local mapped by zs_obj_read_begin(). - zs_obj_write(handle, buf, len) Copies len-bytes from compression buffer to handle memory (takes care of objects that span two pages). This does not need any additional (e.g. per-CPU) buffers and writes the data directly to zsmalloc pool pages. In terms of performance, on a synthetic and completely reproducible test that allocates fixed number of objects of fixed sizes and iterates over those objects, first mapping in RO then in RW mode: OLD API ======= 3 first results out of 10 369,205,778 instructions # 0.80 insn per cycle 40,467,926 branches # 113.732 M/sec 369,002,122 instructions # 0.62 insn per cycle 40,426,145 branches # 189.361 M/sec 369,036,706 instructions # 0.63 insn per cycle 40,430,860 branches # 204.105 M/sec [..] NEW API ======= 3 first results out of 10 265,799,293 instructions # 0.51 insn per cycle 29,834,567 branches # 170.281 M/sec 265,765,970 instructions # 0.55 insn per cycle 29,829,019 branches # 161.602 M/sec 265,764,702 instructions # 0.51 insn per cycle 29,828,015 branches # 189.677 M/sec [..] T-test on all 10 runs ===================== Difference at 95.0% confidence -1.03219e+08 +/- 55308.7 -27.9705% +/- 0.0149878% (Student's t, pooled s = 58864.4) The old API will stay around until the remaining users switch to the new one. After that we'll also remove zsmalloc per-CPU buffer and CPU hotplug handling. The split of map(RO) and map(WO) into read_{begin/end}/write is suggested by Yosry Ahmed. Suggested-by: Yosry Ahmed Signed-off-by: Sergey Senozhatsky --- include/linux/zsmalloc.h | 8 +++ mm/zsmalloc.c | 129 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 137 insertions(+) diff --git a/include/linux/zsmalloc.h b/include/linux/zsmalloc.h index a48cd0ffe57d..7d70983cf398 100644 --- a/include/linux/zsmalloc.h +++ b/include/linux/zsmalloc.h @@ -58,4 +58,12 @@ unsigned long zs_compact(struct zs_pool *pool); unsigned int zs_lookup_class_index(struct zs_pool *pool, unsigned int size); void zs_pool_stats(struct zs_pool *pool, struct zs_pool_stats *stats); + +void *zs_obj_read_begin(struct zs_pool *pool, unsigned long handle, + void *local_copy); +void zs_obj_read_end(struct zs_pool *pool, unsigned long handle, + void *handle_mem); +void zs_obj_write(struct zs_pool *pool, unsigned long handle, + void *handle_mem, size_t mem_len); + #endif diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 250f1fddaf34..71d030500d2b 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -1375,6 +1375,135 @@ void zs_unmap_object(struct zs_pool *pool, unsigned long handle) } EXPORT_SYMBOL_GPL(zs_unmap_object); +void *zs_obj_read_begin(struct zs_pool *pool, unsigned long handle, + void *local_copy) +{ + struct zspage *zspage; + struct zpdesc *zpdesc; + unsigned long obj, off; + unsigned int obj_idx; + struct size_class *class; + void *addr; + + WARN_ON(in_interrupt()); + + /* Guarantee we can get zspage from handle safely */ + read_lock(&pool->lock); + obj = handle_to_obj(handle); + obj_to_location(obj, &zpdesc, &obj_idx); + zspage = get_zspage(zpdesc); + + /* Make sure migration doesn't move any pages in this zspage */ + zspage_read_lock(zspage); + read_unlock(&pool->lock); + + class = zspage_class(pool, zspage); + off = offset_in_page(class->size * obj_idx); + + if (off + class->size <= PAGE_SIZE) { + /* this object is contained entirely within a page */ + addr = kmap_local_zpdesc(zpdesc); + addr += off; + } else { + size_t sizes[2]; + + /* this object spans two pages */ + sizes[0] = PAGE_SIZE - off; + sizes[1] = class->size - sizes[0]; + addr = local_copy; + + memcpy_from_page(addr, zpdesc_page(zpdesc), + off, sizes[0]); + zpdesc = get_next_zpdesc(zpdesc); + memcpy_from_page(addr + sizes[0], + zpdesc_page(zpdesc), + 0, sizes[1]); + } + + if (!ZsHugePage(zspage)) + addr += ZS_HANDLE_SIZE; + + return addr; +} +EXPORT_SYMBOL_GPL(zs_obj_read_begin); + +void zs_obj_read_end(struct zs_pool *pool, unsigned long handle, + void *handle_mem) +{ + struct zspage *zspage; + struct zpdesc *zpdesc; + unsigned long obj, off; + unsigned int obj_idx; + struct size_class *class; + + obj = handle_to_obj(handle); + obj_to_location(obj, &zpdesc, &obj_idx); + zspage = get_zspage(zpdesc); + class = zspage_class(pool, zspage); + off = offset_in_page(class->size * obj_idx); + + if (off + class->size <= PAGE_SIZE) { + if (!ZsHugePage(zspage)) + off += ZS_HANDLE_SIZE; + handle_mem -= off; + kunmap_local(handle_mem); + } + + zspage_read_unlock(zspage); +} +EXPORT_SYMBOL_GPL(zs_obj_read_end); + +void zs_obj_write(struct zs_pool *pool, unsigned long handle, + void *handle_mem, size_t mem_len) +{ + struct zspage *zspage; + struct zpdesc *zpdesc; + unsigned long obj, off; + unsigned int obj_idx; + struct size_class *class; + + WARN_ON(in_interrupt()); + + /* Guarantee we can get zspage from handle safely */ + read_lock(&pool->lock); + obj = handle_to_obj(handle); + obj_to_location(obj, &zpdesc, &obj_idx); + zspage = get_zspage(zpdesc); + + /* Make sure migration doesn't move any pages in this zspage */ + zspage_read_lock(zspage); + read_unlock(&pool->lock); + + class = zspage_class(pool, zspage); + off = offset_in_page(class->size * obj_idx); + + if (off + class->size <= PAGE_SIZE) { + /* this object is contained entirely within a page */ + void *dst = kmap_local_zpdesc(zpdesc); + + if (!ZsHugePage(zspage)) + off += ZS_HANDLE_SIZE; + memcpy(dst + off, handle_mem, mem_len); + kunmap_local(dst); + } else { + /* this object spans two pages */ + size_t sizes[2]; + + off += ZS_HANDLE_SIZE; + sizes[0] = PAGE_SIZE - off; + sizes[1] = mem_len - sizes[0]; + + memcpy_to_page(zpdesc_page(zpdesc), off, + handle_mem, sizes[0]); + zpdesc = get_next_zpdesc(zpdesc); + memcpy_to_page(zpdesc_page(zpdesc), 0, + handle_mem + sizes[0], sizes[1]); + } + + zspage_read_unlock(zspage); +} +EXPORT_SYMBOL_GPL(zs_obj_write); + /** * zs_huge_class_size() - Returns the size (in bytes) of the first huge * zsmalloc &size_class. From patchwork Fri Feb 21 09:38:06 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13985089 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B86DFC021AA for ; Fri, 21 Feb 2025 09:39:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4899328000A; Fri, 21 Feb 2025 04:39:58 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 43A04280008; Fri, 21 Feb 2025 04:39:58 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2B33628000A; Fri, 21 Feb 2025 04:39:58 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 0B7EE280008 for ; Fri, 21 Feb 2025 04:39:58 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id C1352160CC0 for ; Fri, 21 Feb 2025 09:39:57 +0000 (UTC) X-FDA: 83143455234.15.FE16228 Received: from mail-pl1-f181.google.com (mail-pl1-f181.google.com [209.85.214.181]) by imf15.hostedemail.com (Postfix) with ESMTP id CB5A5A0009 for ; Fri, 21 Feb 2025 09:39:55 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=juPO7tpq; spf=pass (imf15.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.181 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740130795; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+7nX2BQHP+jBaMIBfki0Sibf1caPlu6CBS1i9yoAT5c=; b=MX5VirKIuSnI1YOLksDyHeO7a/d5GV/63htc15qAbp6GI4Re/1e8mCBUMZwNSOwo5qbQw8 67GL9RM2laqtYchhLiX90eo8e93Yn5ABWmymYbrjeYL9l0IRvlN6kZNrx/a2wC/Qi45I/Y etIAQFb6IU/lJv4El/f8wRFmujJymYw= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=juPO7tpq; spf=pass (imf15.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.181 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740130795; a=rsa-sha256; cv=none; b=7F85nZrVgYj889S5loEj/SsTOx0AE5nMsXuR433KJQNOFT8kNF0qg9lKrUsZ8yhWBs5UWQ N3XsyvOS8mUNjA2ZPxSvgqORsCiRoz+FEYMn+DK0FTqdQVykwtSqCgcegzy2208x8l5sCy BomWReFiz7ifxV1dRz59ypsfyCfnTbw= Received: by mail-pl1-f181.google.com with SMTP id d9443c01a7336-220f048c038so34297995ad.2 for ; Fri, 21 Feb 2025 01:39:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740130795; x=1740735595; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+7nX2BQHP+jBaMIBfki0Sibf1caPlu6CBS1i9yoAT5c=; b=juPO7tpqxhHJ24gpgMkcH4t77rVhiK1/mC3EE1Zq6MN8QLxgDg7KvtNv33iNh66xJ4 t6wmzac7yRFqFS6J0JARs1vpkA3h0Vm/RO0syz2gg39/kPAwZAhp/rdH6Jmo7q51G6x9 SbuB/XGpcaW3CxijQZe18GjazNIdI2wmE8fRY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740130795; x=1740735595; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+7nX2BQHP+jBaMIBfki0Sibf1caPlu6CBS1i9yoAT5c=; b=LyjImZnY1Q+xwTFgHOTISh/9IDuGOyc7nRGziVEuLoDM3yvQlt42hcBQWOS8yek6/1 UkSSJ0vvMAaYT+SQVtrSUhbyLKq2fZNKaDhfnIhuiQu3q0EdxyWbnZMzgFjIwCju4Jxr 8mIp8Mib/2zp7cLiDXVpjO9ljlYdzuCbtJ7x5N+8qa26iQ9Ct4lqjUjxGI530+JAgjxr Ai+wmsLAxU7+QViHoXVif1bouHqdNrlzANG08/cS1RKLlJcSnRd3ebb+VCk4vdh/8KOv SPFju7j+c3YONm4wXLrPlri+rdlO7zt+gbn2IM/kD8NTpq0cn0MVGlZNbZM95g2z9VB6 fgaw== X-Forwarded-Encrypted: i=1; AJvYcCUljAqMflPilTgcRMdJwB5coH2wHAwwEGuRpnE8d9f4LqyJXKr3qw+nGgT749Ie4/NabH8W28TCBQ==@kvack.org X-Gm-Message-State: AOJu0YyMle1h1yjBhgDqIp2B4cDjqB+gZSVAGJQwzdM1/muh+dIFIn6P YBASFnqOf8aKaXAl2eedqqcM9F5+37G/epk/ihAxvReDtmNtkOIl5E2Vcj0Iow== X-Gm-Gg: ASbGncs5qqXUq/cNXXEgkM/Uaah4vZXcb/+bf5ALDHbuQHcAp5D9RIWb02UtQvx05R+ JPIX1tYwWyarS0PXiwSzMRY4qQV1RXEqV69Gb8Y3oUBzodQYSpbSfGH+BzSHuiplptbBOF6d+S4 iYDQ+cE5pvTomF1OEpvUhXH133Y/CgwSgpcXXiUeyw4g6Fei01VY2Q7jcmHNBZfn8Q8cpNQp9gm mGxGCe9BGIOVNNjmcPIa16ztaEjZURA5GZ0qMR2G/FsvZiyUAQ8yDi+EcjjNt5CjjoZhXCqBgcG UU76tSrpNzFrVsRSN8G7KBiiPkQ= X-Google-Smtp-Source: AGHT+IHnX9TtZ5+pJhaRuJdP+fitjuJuk97DulMxTJbwpxnruxJEAlVkN6hwAldQgSdlVmFyU1l8KA== X-Received: by 2002:a05:6a20:9184:b0:1ee:c093:e24c with SMTP id adf61e73a8af0-1eef3dc4dfemr5404148637.30.1740130794756; Fri, 21 Feb 2025 01:39:54 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:f987:e1e:3dbb:2191]) by smtp.gmail.com with UTF8SMTPSA id 41be03b00d2f7-adbf21517eesm12895632a12.13.2025.02.21.01.39.52 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 21 Feb 2025 01:39:54 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v7 13/17] zram: switch to new zsmalloc object mapping API Date: Fri, 21 Feb 2025 18:38:06 +0900 Message-ID: <20250221093832.1949691-14-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <20250221093832.1949691-1-senozhatsky@chromium.org> References: <20250221093832.1949691-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: w17ehzr6o15skc8am4iy1o84tn6sdnbq X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: CB5A5A0009 X-HE-Tag: 1740130795-559775 X-HE-Meta: U2FsdGVkX1+xCVb5gmzEwRhWeu7OrkQttGDOOa4GdwQFnijxWhThdfPWVhMQL7Z6SMA/MNqMRiBbjSsoxXHGOaALIjpd2uoT7O7cFoq2vcxMJ8+P0P8/1PrZZ8ZWBgfIdELLsnTE5q2adwMBRUG4vlh5roVp4ARq3CXVSg6Ph+bDUk7WmnJcrd+EA7mVQjRoj5CrMUYZTtBAm5jYa8UiwTF/cWmy6shalk1I4CQiEOOuDiHbJuujKH38avZNuzVH4OEXpeHfw80iZOBaSkc3VJzLZDNJhMVPAcSF57RiaLVqmKmWmC6gVyJGPJqdeitT2HV6zTbrN2Nbv/+fUhX5XN+5SIT4nMLIT7g7mTFAEJxGbnxqDZOeQXlPS3HhcTLNMbNQ32uSD2GpOPVlFdS24K6cvDGjNBTOrG+BYcg9GNEFddwcp5cMSl3n9lRX/2LlDK0v28yOH3PaNoTSDMci1iBRahdGRhKNjsJYoKEtd7WOCjSaQ554ptiWnGnCs8GBEpHYc3zNMibuKf45hAS0L7NSrKRjvWxVNKifFgDf+FbWmP9rA3n90gBYuQ9fonO0ZKRG3aTXLuwkZQveYABqUoZr7ru5JHAtsSkxwzIhLHYNV2nq+oXjeP1uXQxTzGROztRhFPSLHemEDCfJP3ME6TbeWil274YD/4HXyxZT8coJXTQyadN4JMWBWbc8dkvOneRxm/2QGeB3YCvJV3bdOd9wN9/SPPZcqKexHYnGYOPA436Clr6HReCb9K0Pk0SRPOsFiKP5LFeYNyBfsLkbIDeQGPQhofMBzbzoydYcs3cH10EqXzAuZHWUdJbGBWJVGph8W+/jZr6W5hj85HjX3sP7zmPeZnL16oOgIb5ZfaFcAK6bIHOkrrLDy3kYFca0GZB/NyaFBZAMExUy9L+0eMqdMliX5Hm8BwY6OjmUb1cUetm0c7VOByW6Co3CiTqIggXFhe1qMmzsdYEn9lo aJ+97ykt G7ilgrSjwYqdN0aZ6tQAHFYz5P/CpTynUE67JHDdFVdaGZjRYjU7jn/KG6eLDYq04hyP+Df+dfRcjj0x0eU4OrpAe9VQGnGTZ8nYw6t5ryjJFJFNHOZlo9BIaTfANw9CGIodbgAVgqhyRaYkKhpPIuhbPwP5p8T9AxVW2Z8pP6/4Xlah8AGTJKITlr6mlTFxfeGy5cUZpWVcKUM3WTEU2igFESPPQUsMgopPUv069Q2Qr0G7IqMof45eQzCjVuHCFxjk952EGgobPHs8XMKmcfw1eqNQYCcpmhdrgZ+f2d7BDv4TRwhUsuZnkQg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Use new read/write zsmalloc object API. For cases when RO mapped object spans two physical pages (requires temp buffer) compression streams now carry around one extra physical page. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zcomp.c | 4 +++- drivers/block/zram/zcomp.h | 2 ++ drivers/block/zram/zram_drv.c | 28 ++++++++++------------------ 3 files changed, 15 insertions(+), 19 deletions(-) diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c index cfdde2e0748a..a1d627054bb1 100644 --- a/drivers/block/zram/zcomp.c +++ b/drivers/block/zram/zcomp.c @@ -45,6 +45,7 @@ static const struct zcomp_ops *backends[] = { static void zcomp_strm_free(struct zcomp *comp, struct zcomp_strm *zstrm) { comp->ops->destroy_ctx(&zstrm->ctx); + vfree(zstrm->local_copy); vfree(zstrm->buffer); zstrm->buffer = NULL; } @@ -57,12 +58,13 @@ static int zcomp_strm_init(struct zcomp *comp, struct zcomp_strm *zstrm) if (ret) return ret; + zstrm->local_copy = vzalloc(PAGE_SIZE); /* * allocate 2 pages. 1 for compressed data, plus 1 extra for the * case when compressed size is larger than the original one */ zstrm->buffer = vzalloc(2 * PAGE_SIZE); - if (!zstrm->buffer) { + if (!zstrm->buffer || !zstrm->local_copy) { zcomp_strm_free(comp, zstrm); return -ENOMEM; } diff --git a/drivers/block/zram/zcomp.h b/drivers/block/zram/zcomp.h index 23b8236b9090..25339ed1e07e 100644 --- a/drivers/block/zram/zcomp.h +++ b/drivers/block/zram/zcomp.h @@ -34,6 +34,8 @@ struct zcomp_strm { struct mutex lock; /* compression buffer */ void *buffer; + /* local copy of handle memory */ + void *local_copy; struct zcomp_ctx ctx; }; diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 814e64f4ee38..47ccda207bf3 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1566,11 +1566,11 @@ static int read_incompressible_page(struct zram *zram, struct page *page, void *src, *dst; handle = zram_get_handle(zram, index); - src = zs_map_object(zram->mem_pool, handle, ZS_MM_RO); + src = zs_obj_read_begin(zram->mem_pool, handle, NULL); dst = kmap_local_page(page); copy_page(dst, src); kunmap_local(dst); - zs_unmap_object(zram->mem_pool, handle); + zs_obj_read_end(zram->mem_pool, handle, src); return 0; } @@ -1588,11 +1588,11 @@ static int read_compressed_page(struct zram *zram, struct page *page, u32 index) prio = zram_get_priority(zram, index); zstrm = zcomp_stream_get(zram->comps[prio]); - src = zs_map_object(zram->mem_pool, handle, ZS_MM_RO); + src = zs_obj_read_begin(zram->mem_pool, handle, zstrm->local_copy); dst = kmap_local_page(page); ret = zcomp_decompress(zram->comps[prio], zstrm, src, size, dst); kunmap_local(dst); - zs_unmap_object(zram->mem_pool, handle); + zs_obj_read_end(zram->mem_pool, handle, src); zcomp_stream_put(zstrm); return ret; @@ -1688,7 +1688,7 @@ static int write_incompressible_page(struct zram *zram, struct page *page, u32 index) { unsigned long handle; - void *src, *dst; + void *src; /* * This function is called from preemptible context so we don't need @@ -1705,11 +1705,9 @@ static int write_incompressible_page(struct zram *zram, struct page *page, return -ENOMEM; } - dst = zs_map_object(zram->mem_pool, handle, ZS_MM_WO); src = kmap_local_page(page); - memcpy(dst, src, PAGE_SIZE); + zs_obj_write(zram->mem_pool, handle, src, PAGE_SIZE); kunmap_local(src); - zs_unmap_object(zram->mem_pool, handle); zram_slot_lock(zram, index); zram_set_flag(zram, index, ZRAM_HUGE); @@ -1730,7 +1728,7 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) int ret = 0; unsigned long handle; unsigned int comp_len; - void *dst, *mem; + void *mem; struct zcomp_strm *zstrm; unsigned long element; bool same_filled; @@ -1776,11 +1774,8 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) return -ENOMEM; } - dst = zs_map_object(zram->mem_pool, handle, ZS_MM_WO); - - memcpy(dst, zstrm->buffer, comp_len); + zs_obj_write(zram->mem_pool, handle, zstrm->buffer, comp_len); zcomp_stream_put(zstrm); - zs_unmap_object(zram->mem_pool, handle); zram_slot_lock(zram, index); zram_set_handle(zram, index, handle); @@ -1892,7 +1887,7 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, unsigned int comp_len_new; unsigned int class_index_old; unsigned int class_index_new; - void *src, *dst; + void *src; int ret = 0; handle_old = zram_get_handle(zram, index); @@ -2002,12 +1997,9 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, return PTR_ERR((void *)handle_new); } - dst = zs_map_object(zram->mem_pool, handle_new, ZS_MM_WO); - memcpy(dst, zstrm->buffer, comp_len_new); + zs_obj_write(zram->mem_pool, handle_new, zstrm->buffer, comp_len_new); zcomp_stream_put(zstrm); - zs_unmap_object(zram->mem_pool, handle_new); - zram_free_page(zram, index); zram_set_handle(zram, index, handle_new); zram_set_obj_size(zram, index, comp_len_new); From patchwork Fri Feb 21 09:38:07 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13985090 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 85AD9C021AA for ; Fri, 21 Feb 2025 09:40:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 17DA328000D; Fri, 21 Feb 2025 04:40:04 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 12DDB28000B; Fri, 21 Feb 2025 04:40:04 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EE8B228000D; Fri, 21 Feb 2025 04:40:03 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id D0B1828000B for ; Fri, 21 Feb 2025 04:40:03 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 89676B09B0 for ; Fri, 21 Feb 2025 09:40:03 +0000 (UTC) X-FDA: 83143455486.14.EB3FF21 Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) by imf03.hostedemail.com (Postfix) with ESMTP id 982FC20003 for ; Fri, 21 Feb 2025 09:40:01 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=nEX1ahKf; spf=pass (imf03.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.216.43 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740130801; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=cvgHEnC+lRfOrRLv2RIVOYlOBk4EIx2tIzOwTFt2Qag=; b=8IokFDVkZ8zDU7aPtE102vDE4Y9nMlg04t+LXFnviNMQFrZaswHQoDyyyRBP+0CAj/QyIF binPgr3pDM/6DEuIhUwlNm9/ax/nSDgG/EHF16W050msQViHQDi5tcZr5aqt83e1YQAc/x +8YAWKIhTcl5Igowm6SI6RVVVkhrMNs= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=nEX1ahKf; spf=pass (imf03.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.216.43 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740130801; a=rsa-sha256; cv=none; b=nI8GuwTiOUwB7wK5ln9ZHmWODU4H2Ek9BBeqt4FuzEYyboO+kjKmx1/uoycc00HKVJcGF2 j2SjK4sXKWuTfEJ8silqw4D71Fdrum7soAK59QF8gY1tsNXtYTTM1gZEJg8vQ8NWQWmXn1 sScbhxNOAD26EPJ5FT9Um+amGvuZ0uk= Received: by mail-pj1-f43.google.com with SMTP id 98e67ed59e1d1-2fc32756139so3016140a91.1 for ; Fri, 21 Feb 2025 01:40:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740130800; x=1740735600; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=cvgHEnC+lRfOrRLv2RIVOYlOBk4EIx2tIzOwTFt2Qag=; b=nEX1ahKfKSvOcpBtwO3DzKJ9qf2BNMuKFkoCD5kPF02Qs7NpJfvfBJ5aUpcZJVDVCF uYz6h6Zkn89qcW2bE3e74ffqsGygzoVwVIADQ/QcY5Hjhmpjsk57yUaHwDcYm6n/LtN2 OS7WL4QhbKKJN2BGCMJme/Ibka8vXBQowOMb0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740130800; x=1740735600; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cvgHEnC+lRfOrRLv2RIVOYlOBk4EIx2tIzOwTFt2Qag=; b=raupolVshID4YifJLM7ZvM6MEjRPK0pC9WmM7wwPjevvzTWpTHWT5CF7B9Vh6WfdmH //vR5meoHGSlKkEHmvHP0bjwCXI5JCnnYevu0pF+Xc5NuOodY5xhpYXTVlnLF6rMwaW8 LTYldVLzdxGaqvBOBDJsinQ7F5MyJGDGBw9ZDJZYjQPuUyDVzZdwymXj5wK5IkR26xPN gKWoyl4GfEKRFloxhlIFPS7tKJN4pta3PGOzTg/NDkyON9xQtFR45ypLEFO2+h982amg sJ3rTWMUT42RODMsVzxwosBdhW8jyUYypd+/RxApWZvWIl2V1RbhZZhVxhTf+iOTBoX4 gCEA== X-Forwarded-Encrypted: i=1; AJvYcCWIj9peIsMNDM32s03zb4g4D9AE+qRmd9I8ltFtJngWcM64asAxgPUSMRhPawEMxcXh+cr/93fOqw==@kvack.org X-Gm-Message-State: AOJu0YzoIBFq3f69Eh8SkvHNUNAdUFt5Us0ag+qxgtk2hdHog4bHWaTk dXYWI9at5Xj2QGF3ucAuSn95bP46uCPFv4Bu9AI2tBMpA59Pfc/69kUWd4BT4Q== X-Gm-Gg: ASbGncvJ6rcKdIr3BYom8PvNzSF82R9uYAcA7/2tmqf02H3FSksSj+W5qsPMaw5WJNA j69dH+1LJu7OymaiS2Mw5fcC9gA8B3jv49cUX6dGwEE9rxf8zXra+ucfFGg01UhUQXG8KI69/cL s8GvXZ4rTjxImzvT66ER+lUOfukuLEDVtgSlXfrse0RWi/GsTAH29pvE636Iy1QYzf6mdLqsbx/ QuWg8MKvzPL/cuOgqL649wJ1JoGLmHsKljCFm3SqUmcI0CNzmepPbWA1R2PhXzVrh79SS3+zRet jx4xs/vXnDQ3Ey5H/ugkBrG0PYg= X-Google-Smtp-Source: AGHT+IEeaNCIbiqEVAo1RmizyW1uxrb4z6DMOQbzYKHkv/Y97ooxMJdlMx5xw1SmW1HKoPKFG7FKUw== X-Received: by 2002:a17:90b:1811:b0:2ee:aed2:c15c with SMTP id 98e67ed59e1d1-2fce873aeffmr3562067a91.28.1740130800521; Fri, 21 Feb 2025 01:40:00 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:f987:e1e:3dbb:2191]) by smtp.gmail.com with UTF8SMTPSA id 98e67ed59e1d1-2fceb12dc27sm898120a91.45.2025.02.21.01.39.57 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 21 Feb 2025 01:40:00 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v7 14/17] zram: permit reclaim in zstd custom allocator Date: Fri, 21 Feb 2025 18:38:07 +0900 Message-ID: <20250221093832.1949691-15-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <20250221093832.1949691-1-senozhatsky@chromium.org> References: <20250221093832.1949691-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: 982FC20003 X-Stat-Signature: 1fn78aqs3sby4cutxpsdy5xm76iewx5p X-HE-Tag: 1740130801-43979 X-HE-Meta: U2FsdGVkX1+fO5DLkmFEP0YBclYfIQ2YcK5iO2FAKyYJ1FAjjdDSUavjrls+IWekKH3BIMTUnPLCfPWrWoc/WrUE6gg4lYrfAcyRYVno6mgmhdSMyoDX/4asJr2odK61MzmicCTKYkw30Tq5j2YFlyiZTKo0W/2XAUwfhWzwLysWiTI9i0JFbNmSZkKIjJS8VDJn+D9PMwjJNQI0onYIBIqQ0PjFWVuNp1dRJkpNniNo8zYGA7Bo/j/yirZMIanqV4i4IfDOQcKBxShpv38pl8AJUO0KFnndc/x/QWWO+RRZtsims/2CxuznnBAZQfxdVqwZDeHvsdMVXo0WIkihLUIXzoA8y8P4KJ6dqzgMFFLN+9VEI7uH650iVWlxAmxfMzAldthLDSx0yiRzmvwN5ECG/TW8KFiof29ccVHKnw2mBFJG7vPd7V9IYGHp2V8Y/GAbVFih48YD2DzMzoZeJR2NTQ8qSLaI8uTUCuHvNZ40AmXAiYWiujh+ZnyIDpXWqVoOyqyREJ0yVlOoFeAIrUZ0MexiPUr9W/rIB4GPLa4IfQnwwIRhih1Vl7VYNn1X2Tg0ZnLUunTDMqNYlb/toXx3wCxSol2NIG+9g13G5f/S4jSdfkfgu6ZC11DsAcxDNJh22wRwyF+VLDf4EOcwPIsaWG8+pVKmJQZgHP1qHX79r8RKmg48N0b7Q/yCO99+Gcyvs6j3OeYGkgjDA/8yPjED34oOJ7dQei/JTp9H4myBHPW+gwUFpELZOmAggsY76I4XtCN2acBt3WXyHcShFlLUsW4a/p5cKgOxwqs0wFDiMUy46t+IBlAOEm2LY7dIsbVIakqyOZMs6uARVYiNyVhcBTCyMh4O+2Nu7B1EMEclHzJefQDsy+vowRHC4Nwts0r5kp1xjgCcCnn40ZX6eT5b0u2th0QMxQdIG8Kfe2DyxJf5nmJaSi3r9bj785UOfiwvl2CSdLLbwdDILXQ oZvjbmmp yaYnkHTAn+UYbhOsexkJbJQT3xtQrrrbyWAULQ2/f/P84Ps4Yfju6JBkCb/H3nUnRlhpPVKgF15+Apscun9ayf2EMWN0Yw17m2P2MDoFsG67gnGIcHwoJ1ZABOTEEt46qCb4WIVHdRpaEbFXZ210qYdNmj6B/M+QWgqH+j3AGHdMIs4rjMXp6NKOj9liCIzC86Cc5YPVPouGZhxHIXC/SHlIalVSTbnfUBkvUEg2tan+McIBo3OKtsH82cf4r+biaUahlXDsu1iO02jCitEvJNnXm0aU2DLD0tNCLrRK20SBJ7VwC1f7hyYD3YiNvSOMIVDFo X-Bogosity: Ham, tests=bogofilter, spamicity=0.000003, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When configured with pre-trained compression/decompression dictionary support, zstd requires custom memory allocator, which it calls internally from compression()/decompression() routines. That means allocation from atomic context (either under entry spin-lock, or per-CPU local-lock or both). Now, with non-atomic zram read()/write(), those limitations are relaxed and we can allow direct and indirect reclaim. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/backend_zstd.c | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-) diff --git a/drivers/block/zram/backend_zstd.c b/drivers/block/zram/backend_zstd.c index 1184c0036f44..53431251ea62 100644 --- a/drivers/block/zram/backend_zstd.c +++ b/drivers/block/zram/backend_zstd.c @@ -24,19 +24,14 @@ struct zstd_params { /* * For C/D dictionaries we need to provide zstd with zstd_custom_mem, * which zstd uses internally to allocate/free memory when needed. - * - * This means that allocator.customAlloc() can be called from zcomp_compress() - * under local-lock (per-CPU compression stream), in which case we must use - * GFP_ATOMIC. - * - * Another complication here is that we can be configured as a swap device. */ static void *zstd_custom_alloc(void *opaque, size_t size) { - if (!preemptible()) + /* Technically this should not happen */ + if (WARN_ON_ONCE(!preemptible())) return kvzalloc(size, GFP_ATOMIC); - return kvzalloc(size, __GFP_KSWAPD_RECLAIM | __GFP_NOWARN); + return kvzalloc(size, GFP_NOIO | __GFP_NOWARN); } static void zstd_custom_free(void *opaque, void *address) From patchwork Fri Feb 21 09:38:08 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13985091 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7B743C021AA for ; Fri, 21 Feb 2025 09:40:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0E0F828000F; Fri, 21 Feb 2025 04:40:10 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 043F028000B; Fri, 21 Feb 2025 04:40:09 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DD70D28000F; Fri, 21 Feb 2025 04:40:09 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id B233328000B for ; Fri, 21 Feb 2025 04:40:09 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 6C8FF120BB5 for ; Fri, 21 Feb 2025 09:40:09 +0000 (UTC) X-FDA: 83143455738.12.F3C850E Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) by imf11.hostedemail.com (Postfix) with ESMTP id 8763D4000C for ; Fri, 21 Feb 2025 09:40:07 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=fzqUw2M0; spf=pass (imf11.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.170 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740130807; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=X/8bmw52gxKP9HlvW7ud00ppVmPtFMCISSlcsi1PIXo=; b=XLxIpD57YutxMZ/0FGVp5w0BwF65Bv8iptLa4LdL2pBZF9IyeHy7Mpo6xdtOvSjT5096JC jQFz8UpcYLsmqoKAWqfuO7+RMG1rnxqaX/azoCzTDAfLUleks9jn2q0z3H4nvHMxpj41wX d7mkkiQ8rSq6so4e2CS9WAoUTw9hWo4= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=fzqUw2M0; spf=pass (imf11.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.170 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740130807; a=rsa-sha256; cv=none; b=QyumnQc7hGYISGEalMq2F1LWzOpYH53sX3d3B8j+QVnP/L1rxKmBleATBOFwYDqz0SW1ZR vjxVx8Eb+ohMp+ceWdl4xhbrQwpHJ1yHx9g+VWztpEXz3i0LsRXXQU9J0X13xZrip56h8t k4s29tkbwHoaqjB5fx8Brur2IBR/nS8= Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-220f048c038so34300935ad.2 for ; Fri, 21 Feb 2025 01:40:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740130806; x=1740735606; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=X/8bmw52gxKP9HlvW7ud00ppVmPtFMCISSlcsi1PIXo=; b=fzqUw2M0ifr5O9nUAm9bYjlwxgqf2+0BYv88F7DUs6AQwRDG+Yq9uj7CxXPTC4PY29 AHB2BhhwaebdNo4iuvTWwNr57p1g9acVQH/HH5ps+VN6wjdmUutDMv9n7xDr1CEbuXgr 6dsD1pa4nnm2CRXV7r7VYZ4vMnEoVr2EfviQM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740130806; x=1740735606; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=X/8bmw52gxKP9HlvW7ud00ppVmPtFMCISSlcsi1PIXo=; b=ll8QDzu1NPUYNb91xW0iBvk9DqSRTTfcpakgmbDAKFuMwfIwdtY0SFlv669Fmjcnzc dur4FgZaA7eaIald+QakoeHjNqUMKXvamFCh4LHrlcfowZHBFGJr2UTY/zjAJT+8JIi1 /VtX61+E2wBVZaowsrsNKJLM5GX1U9JQjbBY9bFhrpzwPggkNyMO/zUK/D03eY+cX4A5 7utfHOT3toBc3Wo25N8+zz+cFMw2QcvuK9DD/p/UEABakpIiEYVKVyFS3XvCyFwgAMVy Q+EdvYjs1cWQv+M3c4BnBwLod7FcybnNW3BXh/7L5QBlXoM3uhUwO9XcJHROMH4tOMrq I3HQ== X-Forwarded-Encrypted: i=1; AJvYcCUKI2cQl30jzABW9ONHWZhIb5jsSLHcA5cuwkVwbqa3v/Ue4/6p4yekIvd9b6uAdu3y4zXTz3ID6g==@kvack.org X-Gm-Message-State: AOJu0Yw6ZOmNdeZj+cJgxJAEG0AQvV0YvDfp4BNn2jbhkDMRTMHh5fEy zzZRTNNE0+YUAGiQRiCL0siPezMMDRMRZJwAG8v0pzV7MsH34V5UnfbVOwaMbA== X-Gm-Gg: ASbGnct1wOPqKZFhzHcbZ/c21Ubl3tf4NoOOGDYk/kmob+pgY+bUaRreJbci4HI2Xoa Dx3hQcraR9FJPipaYmtSO8tIXoLY5IeTqhPKWl360ZUHxpjSirhlhwWnWtc5y5iI1hr5Z1a695M iyDR26HBMjS0x4edveYkyLFtcUIHhPkkErB25heaIXFW0bXderEX9Q5E9oNTnMT+DwwdXNP2xQo ONmE01dOohkHKMcnKcS7Fp55gieRprAT0dnDJam6Q//AZ2RKra/nNuRl0lL8FlPlXU/n/n5wrFs fb6lxAoA+jlaDu6ceJ/FJLHRL1o= X-Google-Smtp-Source: AGHT+IFiDsUOFrogVCXhwG7jE5UhzUG2HXzmhm8iuLxdnK1gwrZ2PnYz7hKVVRLeqXubsVEuTmgEmg== X-Received: by 2002:a05:6a00:c89:b0:730:91fc:f9c9 with SMTP id d2e1a72fcca58-73426d6b971mr4177006b3a.16.1740130806309; Fri, 21 Feb 2025 01:40:06 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:f987:e1e:3dbb:2191]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-73242568aaasm15111670b3a.41.2025.02.21.01.40.03 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 21 Feb 2025 01:40:06 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v7 15/17] zram: do not leak page on recompress_store error path Date: Fri, 21 Feb 2025 18:38:08 +0900 Message-ID: <20250221093832.1949691-16-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <20250221093832.1949691-1-senozhatsky@chromium.org> References: <20250221093832.1949691-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 8763D4000C X-Stat-Signature: czxaqmxdj9q7s7tgy3udi4ursrgzwm9a X-HE-Tag: 1740130807-839103 X-HE-Meta: U2FsdGVkX19LJgAw8NrLfQ7jd+RyWIRCnAujWM+BvF44JPgogPSz089I7487qsTiYEUzbF38WGBLDEZv31U1e7c6D0zTO3+NlHiF1L6lEwnSzCr2cE3agm/Xs8LiWiKeJHfSEUNAxY6inD0Ljy0xRGgAFCBAqYw1Vx8pjnrGjkoBdcABcZtXtjhQ2FtGDtCA3ZlzrZYS8SRpLh/qLo8f+TtFezBsoeh8Yd12Se4MVArZBDb52FMVcPl0oazQp5FaJLJlExGThUWmqGJad4GPAaKE+j4KGKLXfijpQ8Uok7kzDwY2wMv0qs50mw97tZ5o6JPtXAVobmeMvbmoX6n8ZgBEcF7zGjb+Hqdd8hfh0CylPQ8ObK9/8srFkVJA04zvDzAX7m/DZpJkDlEPg3BpP8ac6lxzzSX9SseCid/lxjHNAn/FMGzb4g+y3+Uj2BuuFZWfHiFyBRC0KniWmzvsil1f355VrIxfhWgc6xXWGpMGM8pOy55jcFyUp/ZZlw+jklOhY2HPDQhCYiy7QK+cKFNARvdbvNhHERs3p8g0Fre8wpNoDiW0FGqd88guMAVdJJj1PWCXxYZJnfYP9TzZLYvO2u2b0IIU9g3ATV+ERzcHiv+3E1Ryw/RHzttxXu2oWdKVBstE+diVJDrwBdPzJDgEXEpaDMPvtE/GgW1b576NHeoxYzBTP1Q3wbGW6G2sU6Ha5he5N9jNaOSE1mgISojPS1L2GA1E1j0DI2mxv1wpyxg1ty12gbSeXcuqvJdIaqhaYYXxH+jq6PiEtEvgX1ylONSs+HNSfOcSej5j/C13rK1LkSIh7gEYlDnNiBFOdn7Qs8DFa4sgEX5dtAAVFHb2OrG63Axh58i/VFMcofilqNeTGFDCyZI+vM20OWSxE88dFbaWaWm55TD33TwrsQ3QRhAZDxJj3NacQRE21fSY/JwPAvkREOrTc4VAgCtTmy4cn0tw1beRX2XA39o X1dFWpPf FPPmBkqqbl4ZlfcpHJkBtv7OmnUQrFux+McAuocReCNhu1FVfTb95kbmzJJUWhP5xUJ6kVcQE1oGlWaVCGFNovjNjXH0de3EXMA5Z4Cw/+logFEFqH8G/qwjSsB0I1oQZ0zUTxGnxOUgyLPj5XE9XxQ2CGQK5YJE6GTIE+9smgWa98lE251jsODzJMI5Ltsg0mLdK5OtyNLIe44294x8QLlmSv8S69f07bFODOk002wdbigrR9dTxleYVp5vBG0I2zyQQc0s2KEtFepLk7Yh2pZ1BM/jJx5jVWl/038iDHGXkiNpebThOb8GxS+Fvh1TGyizG1AUKtSTIIFnxvtMNfO42V8cv5C6/bqJ1ibGh+L1Z7vY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000005, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Ensure the page used for local object data is freed on error out path. Fixes: 3f909a60cec1 ("zram: rework recompress target selection strategy") Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 47ccda207bf3..48639a48ddb8 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -2022,7 +2022,7 @@ static ssize_t recompress_store(struct device *dev, struct zram_pp_slot *pps; u32 mode = 0, threshold = 0; u32 prio, prio_max; - struct page *page; + struct page *page = NULL; ssize_t ret; prio = ZRAM_SECONDARY_COMP; @@ -2166,9 +2166,9 @@ static ssize_t recompress_store(struct device *dev, cond_resched(); } - __free_page(page); - release_init_lock: + if (page) + __free_page(page); release_pp_ctl(zram, ctl); atomic_set(&zram->pp_in_progress, 0); up_read(&zram->init_lock); From patchwork Fri Feb 21 09:38:09 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13985092 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B8209C021AA for ; Fri, 21 Feb 2025 09:40:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 48CFD280010; Fri, 21 Feb 2025 04:40:17 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 416ED28000B; Fri, 21 Feb 2025 04:40:17 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 28CB8280010; Fri, 21 Feb 2025 04:40:17 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 06A8E28000B for ; Fri, 21 Feb 2025 04:40:17 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id B04AA120B83 for ; Fri, 21 Feb 2025 09:40:16 +0000 (UTC) X-FDA: 83143456032.05.BC61825 Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) by imf13.hostedemail.com (Postfix) with ESMTP id CD75520002 for ; Fri, 21 Feb 2025 09:40:14 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=H3sJZf1r; spf=pass (imf13.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.174 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740130814; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0hGRfBbi2fKBhcLL3toY8LXz3XIHS2H2SZvxy9aDrUM=; b=lJHVfH1vha5lcGh9utbVj+1Mia/mQkrFsYB2EfqSm8CrxXuXJKlueC5PToKBRgDhKc/39z yt1A5EfniGMG88x8oueuVWcO/Bw2WA877ATOkgq0UZ4lq6dRLjpksOzvbKkDjTF4gUi59o kriIgz5okhMRfwLk+sUTBpCG4nK3kAg= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=H3sJZf1r; spf=pass (imf13.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.174 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740130814; a=rsa-sha256; cv=none; b=oDxmw/lKF4Dt9fR84+4BmWd8t6DF5+2U85hkapDt1jAM1s4NSo1tzmuxHH9BaixjeDzGTV 1bpycXCTIDj1QHjq1d9b/0WwE25JTUa0Kz5Jq915rsLewY5LTPyLQDjIQfCCVXy0BHlLSM QcvkJkmcXVVhE8eQYZFsE+BoS6ySQPk= Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-220c8eb195aso40122135ad.0 for ; Fri, 21 Feb 2025 01:40:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740130814; x=1740735614; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0hGRfBbi2fKBhcLL3toY8LXz3XIHS2H2SZvxy9aDrUM=; b=H3sJZf1r6Q1P3T6pevjCtHrl7LcdMfjgYkg9x56OLGm7SRN1JJS4QTyL7sXH+nEzfl Nwkz1nOxLSLJ7t3Yfp3t/k0VvJd5XBDuoDbque+s6Ubd1RAUqnbHPG6jliqmfwAKMaEZ WOApHVSYYFbkCQXKHjU95L7dqzqvVMr4aRHLs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740130814; x=1740735614; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0hGRfBbi2fKBhcLL3toY8LXz3XIHS2H2SZvxy9aDrUM=; b=i9K3PVvEiR0mznbEGAcFvP9zvcARLY0p+dvNzyduqD8to1ORxA3SgHQ2d32I1DwFgt WDmT4lIx4YHyCRBBeW2OhsrdcJqQFtKevYoYEX2B2hxcbiFn9P6hTgEZNQ6GWAzhv/5X 4UBpCRyf4cZYh0JTYS2AiMrzOePn4IqMdIjhnYlucqnxNUVhtoLXtBPrXNd1F88oWsMQ tiOVH/ZxeQ2KIhbh41pT5axF2mLxV2CEd3tSCve/tfARwJtfN1XhefbEsMsM07QPr0fZ 4yyVGiIRYsPwx1cIMbRsDwneSjDKjlKrIywT+OHByP9BQdqF+7HZ8xo3Eh0KDA40glQr H+rQ== X-Forwarded-Encrypted: i=1; AJvYcCW5Ihq3itZX+d83w957yeU3h1DSfh21n0rHs50h4iyRins6I8mH+XarmNO53FTWUgrWvlYquy2JJw==@kvack.org X-Gm-Message-State: AOJu0YwBkR9ZdN+PHmMZP26ZAA4qhiFARhTLXpU9h0qgZB5kNzkwSGUs lI8mY3IH5jg1v6qATrQmzY5rKWgcQCWSuxGGfXTOc8yDoFySgGaoaWmGWzvWbBtK/PEmyT6DvtM = X-Gm-Gg: ASbGncs8gSeJU9+HIHbj+LJrHDyzd+FZDaA0TB97ORuGoIK/NSxRnEQH9w7aevi3dMA QAnO0ek37y42oPMNpuW1rOzQzTHOBxg0vrQbWftBHfBj5NPXmlCcAWsInrrfn19jwuxpLuDuysk zrscm9ZnvCFBcRCuDS2ZLEGJdD87amraaIL+zP4ZYEueyCVN76B+dKNngpDcbQDber9f10r+0Ai CtQrD36zS2F4RANDJZZ/h4TI9pkDvIihfwUgrk5P0vLdZGKMAA3egiLgpHF2bo7NsJYqnOffUWq L4Unef6SsmYvG8LpDRQFVhV57/w= X-Google-Smtp-Source: AGHT+IHnD4WYtByveH7o9SWK4ZPshLs+PqzDTaY7xc5G/UiEwzsRQaEFLwJdetxPgoe922xVEvOq1g== X-Received: by 2002:a17:902:e892:b0:220:fae9:8a6 with SMTP id d9443c01a7336-2219ff3127emr51606815ad.4.1740130813846; Fri, 21 Feb 2025 01:40:13 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:f987:e1e:3dbb:2191]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-220d545d051sm131952525ad.108.2025.02.21.01.40.11 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 21 Feb 2025 01:40:13 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v7 16/17] zram: do not leak page on writeback_store error path Date: Fri, 21 Feb 2025 18:38:09 +0900 Message-ID: <20250221093832.1949691-17-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <20250221093832.1949691-1-senozhatsky@chromium.org> References: <20250221093832.1949691-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: CD75520002 X-Stat-Signature: 7bf8k7ag6q16xj3oxcamy3s36x6q4uai X-Rspam-User: X-HE-Tag: 1740130814-714112 X-HE-Meta: U2FsdGVkX19jLXqkquNk+dIJdkQfQRmnR0foJUoNyAqWXUq+ME6ERtq8OWuFY3/VVkOIK1WvtuTMlIpgdd5iR21RUiCRweLFT91Wz8UZJpYhJnn1bxFbJu65qtKnMCr4Yy3ojfdWFPytCN7gancvh8BP9+6H6oiXjSuJEvH3ho+mWWvC9gT4CLRgJ1Fij/RHk0FkfL2EPNReyMGGY3KA9X3QKNtggfnIk2wF3arGK11Tew+gFhHeQQDC+aOh/+ZeN6HPi/7vQygOf69Ki/TTDdZqJPP+UUeJOtWu/ck+4COeuFm/Gk53Wyn6+lY24tVJgLvt0dXP5tFBPuBnAydBWA+LqBZOr6J31PtLOSyLXG4V2Iy7IgjELdGOkG1JQ7W5dtuINKOOev9KXZuhoXoNExGyHd30FkPHrAjRLqAanW6u8DYrlN3LG8r4LkwywwzO+q2N+8BcogN+nGLnf9UinUHDKnAD/plH1PFbSYeD0OaYB0Wgr/mM/7S1qvAFAx8ZURGbZOw4cnxlnNoLyzzGFfTX+XiQ8kgGt/jzE8ZkHnHbyGxmLlICtkUvoYwMlGruNuHSsB/5L9rBHOjsU5RwnyLFyDqanJ5OISf/obyNvfYIWHkESUlokwVPWAMwkH+ZqXAB6j4J4MGBAiOlpeenI2DZkoA1+GhMbXh1FY+5OEYiQRAiS/RLN9SMSsdzUDY3BqdIeZAd5r8WKC7uRvDMQv3EOOHZYiULk9bG1tuKeUjvLii4/QsaMIsCBQa180rKMnux+wM2HtCV/Nuv4+kkl0ROjzTWIgMlDpjwOBGLAi6A81Tu2aAHyaw/+G7z9gJ6MFZ91YZ6uhGkgQLlMrml418udgczxNfSZSBbhCv3PA3w8Qr88LJnllECu6EGbD553bJ/KfyVFE6BbByJZuv+bHtIkbMHCUqOKT9rKMSPFalvAmb7bjcViE6K5LkeoWryDylQJ3/bN3vgpDM/B5L n9beXCB7 yLipakpboQvnpw7gbXcBwrJjlM9GCb8LCfW3mAMU0SSgCnbjkkjkQ+vH9sYKkZU7OnwdLCpK+vbr4HmzWFDZ568lE10/0SpVYBlhccNBhemiVEB954XI2OnBxk9sgpG3W/wxd65lxGqhrNNFfjaocmDv2RqSEiXlMjnBQgZqL3f4FQa48vG0L6ftbOtSPhr1s9DyHx8lifpxv2LyEXLQIwc18WYxw5ISlKWThrRgT+wD7WJvw1iQ/xD3QVrOiQU0+pK2TO3WUJnqksokMQi2gbCv4/ZbZ6TXmvN6z7tXlS6Tz8ibiWhUvErroY0qUXfNLrZ/7gZ3HH9+DBD0gyquIRSCqOwQgaXjWf5gjKD/sSMayB3g= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Ensure the page used for local object data is freed on error out path. Fixes: 330edc2bc059 (zram: rework writeback target selection strategy) Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 48639a48ddb8..7a5c5b861fe5 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -792,7 +792,7 @@ static ssize_t writeback_store(struct device *dev, unsigned long index = 0; struct bio bio; struct bio_vec bio_vec; - struct page *page; + struct page *page = NULL; ssize_t ret = len; int mode, err; unsigned long blk_idx = 0; @@ -934,8 +934,10 @@ static ssize_t writeback_store(struct device *dev, if (blk_idx) free_block_bdev(zram, blk_idx); - __free_page(page); + release_init_lock: + if (page) + __free_page(page); release_pp_ctl(zram, ctl); atomic_set(&zram->pp_in_progress, 0); up_read(&zram->init_lock); From patchwork Fri Feb 21 09:38:10 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13985093 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3B8B9C021B3 for ; Fri, 21 Feb 2025 09:40:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BDF2D280011; Fri, 21 Feb 2025 04:40:22 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B66376B008A; Fri, 21 Feb 2025 04:40:22 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 991C1280011; Fri, 21 Feb 2025 04:40:22 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 74A116B0088 for ; Fri, 21 Feb 2025 04:40:22 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 247AC80BE9 for ; Fri, 21 Feb 2025 09:40:22 +0000 (UTC) X-FDA: 83143456284.28.C550FB5 Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by imf07.hostedemail.com (Postfix) with ESMTP id 4B66E40011 for ; Fri, 21 Feb 2025 09:40:20 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=BpC9uQx5; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf07.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.180 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1740130820; a=rsa-sha256; cv=none; b=OZBemTWLtD+gZk5P3uFXrJdwlkQ9rnJDc7IaCkjezC0vyISXK3Zf+lVg7YmmiVhasn8lc8 crEkPSsy5upNodOvRI+oY0QJPAaM+xdLstg9bQcRcisxLjlGSJsyfpb5RETuqH2tLde2yI Os9mztcx8qZxWatiPCUkiN0RecPvYQs= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=BpC9uQx5; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf07.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.180 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1740130820; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=sFE2Vkh4W4kK9IrN6erQ+4Og/P/7Lb06X2GD4vSsp54=; b=7mgYLVcpoS/pELg0k/zxvhPpgfYrW9l53iyCfdPKv/8pw5/LGTykXGEt0CL/ElkRHbOg0R wonGhzY2FFpn9j0AFAibyfO1R0bsM2z8OJmApVS4rGc0UUBNBhkwQ4Tfx5U3ZZPwrjcN6x FwFVBD4DcPx5SyUbMxLCYCjL11G6glI= Received: by mail-pl1-f180.google.com with SMTP id d9443c01a7336-22185cddbffso45291445ad.1 for ; Fri, 21 Feb 2025 01:40:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1740130819; x=1740735619; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=sFE2Vkh4W4kK9IrN6erQ+4Og/P/7Lb06X2GD4vSsp54=; b=BpC9uQx5PuNepHr8OxkIOWNttNB1kZDUT5Sy36dEI1hBY9HmW4LUIAzK0aeYIJAtRS dBZlryuNgUfYxCVaTIiISTYI7VWL6AqMzvFQs/GlAj6bx+gABLMwefoxHeYvcmJywm4F ZTw3Ah2USbJS7qd1P3PAoTUHXDm72UxpOtitY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740130819; x=1740735619; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=sFE2Vkh4W4kK9IrN6erQ+4Og/P/7Lb06X2GD4vSsp54=; b=Hn2ML7x354JCC+NrYp32Fbp7CKzS8cXFuX0krkSNCklhmuP6YDfQzW1hvk6hX7YD5g lYsl6RppxHnB967C6jlvWxmjRhoHEJslaV/4aGbHUMw9dooXtHUHQ9mrtUvIENKnMfAQ SHr8OBMd57v/g+fH+0voO+LLXt/ED2RpxAO6hfUbaARGJlsbHXUt4hVHP2e3Gody5bVc MP98gAIEa3OJBJ1+sPUmiklwxxVzGaG3VVe9v003DmCNnN7+3duX35qCSMa6UAzQzBoy 0XZoT2BK1XbUqW5LTxds0IMLWsLQ27Al1pc0t1BQaOMpjG3dclI6T/raLYR+UgwznfZs 2xNQ== X-Forwarded-Encrypted: i=1; AJvYcCVUXSt9GDGtIDDNhiIHDTJM4Ul3yLR1c1654cLagXE59DMpAGIXToZ/Plf4jZgZmXF5Ws7uLAat0w==@kvack.org X-Gm-Message-State: AOJu0YznEDg36FQPbYlFoJFq2VfZugUAii+LTv18ipiQSpRH9hKlUHdd hJA23KrFXNzLkvXe81x7ctz78kT8QysBTfEsm3YRHyhmilyXvnjlKfpmSws76g== X-Gm-Gg: ASbGncvyX8R6eDLklxg3dH3N31NlxyM0TM/Kjf9KBKqNHhrOqZXakDndt9vItiMQTcR zgrPkOVW+Y1xu1XNKnUkvnFBTVjDL+rU+B8lgySQJdcytMFuRGQnytIIbfxKHPYp1XmVSpLQEnj K/mLgJJyMkSBRUMPkPO/JHAnliTYGB7MCd+DuxJCl+arC8pTDQZ01trgJswQIfwbdpr4lp/2/cx lSNkONrTkl3BCWwfQ7uD/EpDQxODquMZ8DNhpe4ZMivffGG+Nm+Ibht4n99YpGR1lkJNz3LAikp 1rtdtrx8SE3/EvobsfG5rSTiPzM= X-Google-Smtp-Source: AGHT+IHaMrwpRWXBtbqLfO7E9WJyzm0Z3N+2WmGGSFh+bCgjQ+uCawv48dxk9E1SSvQ/nXFDOCUNQA== X-Received: by 2002:a05:6a00:3a1e:b0:732:5875:eb95 with SMTP id d2e1a72fcca58-734140898f3mr13714952b3a.4.1740130819336; Fri, 21 Feb 2025 01:40:19 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:f987:e1e:3dbb:2191]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-732425463ebsm15677215b3a.19.2025.02.21.01.40.16 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 21 Feb 2025 01:40:19 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton Cc: Yosry Ahmed , Hillf Danton , Kairui Song , Sebastian Andrzej Siewior , Minchan Kim , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [PATCH v7 17/17] zram: add might_sleep to zcomp API Date: Fri, 21 Feb 2025 18:38:10 +0900 Message-ID: <20250221093832.1949691-18-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.601.g30ceb7b040-goog In-Reply-To: <20250221093832.1949691-1-senozhatsky@chromium.org> References: <20250221093832.1949691-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 4B66E40011 X-Stat-Signature: 48yd4fpanq1x6is4eq8nts5idkqpi41k X-Rspam-User: X-HE-Tag: 1740130820-698288 X-HE-Meta: U2FsdGVkX1/YBoGWuEasaIjMUoTTHvzFbjKtPK3YT2chSSh7Nixj1TbRykyagrcMd6HPDBWK1J8tYqh6b08v10NYYss0bcq7xqRjiB2+MLehJWvzIxlpJKOBui3Y8i+EKaxLbsbBSZuwBjib9ncuBG8LYaEcJtFw4RaGE6NOOl9TM4JNygq/92MR7i9VPjf30NyxEB20yxSTNmn7Vo332y7CUgSJjv9Ltrkux+wNJtmRttn3Fj4qDKvtsp/vxAwjUsC9/8o7dHVXUMt0wyuhaukCXNqFA7n4eLxxbca0B4mrmPg5SCxsixqJHGoApqOY191xDwGa6bot0zhzKp3W+rwbsgPpdaRNt/9ABXDljDskzXk0JYI0V405OLjzK1f6LWIWNgJtFblE4Zn6HI6M3bIpG5iynXvqTeb7/NPIJBK5LKa29yjZZ0WJdvEbqga7C0FOeKJLvju3Qk+BQNa3uMXVptmfgNLh7gfP/FEmbMHaOgV0j/Kxa+vDReFIx0gpz31bgJttKvR3ANv6NGeumNKCP8GnzrGOEbWHnsn/KvZ6uc4haqN5kJ3EOgA3+o4ea/PHxQC8lFutSUIgNJlrzzQ8ZuVn6VlXnzjDId3fp4FoQTLTetBLQd8POvTQl1QivlVau3B+VUUOVgEZThcP1X79Tx6Q+D2Dsi2KGuIrbIf1v1dZe6Ena7flcQDDUH9z1aTsLtVtKpdiVW3t+Ok2jHE71SxekgkMryYFMMdOkNpMNwwkM4Bs9ujseet4kGBzlh3Sb9pKBXHaU33q5yPCDfDANOLoi1F4OrKCEtFJWcQrbYaHpAHwpaA28Grn49bVkcjPoI8CVFj1IfwWIzI+i4MuvnQHZgkxt34bxNFL+DOQhvYKyF8ST46ixDfnAMa3p4eVvF1aMx5kYLxy1/8wBAp/lyVQQuzk/H5NLHifiajGDTNhaMILsHk7GvI/4wNoPWRYvUMqvLnCHnBgb1e ubQJzkN5 jHyN56XGMS0Z1ZhWk4uDR2V5O/xveY6Z9Yi6/XsINmwk8Of3ScKfpZ7HpXDPu2BhIHeh59p59fByAk/a1EyyRzq9E9yKssD0e85sN+4l7ghmdozmv0sWnEb3lB95JG4LdAriY8p92KkK7iK3wQkET4Ji1UQDMJht35i5/BAMltWsx+BNMu3ZINlhjn/6XwJGGYZHouwp/70yMyzN18WvOhRQ/X2eWOL3Em9WRsFF0nS/kjj07KLL6A0qcw8PG0rju+xHz+yGOsHYp3tJ8Anhd+P4w2JA4eiBH9kCFeCxVzDezUVc4FRIY5AWpu2tgxXU2z8jxWLvEO7cOTtKTSm5dlJZqpuJUZdEtn5Wh9nQJsvhR238= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000004, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Explicitly state that zcomp compress/decompress must be called from non-atomic context. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zcomp.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c index a1d627054bb1..d26a58c67e95 100644 --- a/drivers/block/zram/zcomp.c +++ b/drivers/block/zram/zcomp.c @@ -146,6 +146,7 @@ int zcomp_compress(struct zcomp *comp, struct zcomp_strm *zstrm, }; int ret; + might_sleep(); ret = comp->ops->compress(comp->params, &zstrm->ctx, &req); if (!ret) *dst_len = req.dst_len; @@ -162,6 +163,7 @@ int zcomp_decompress(struct zcomp *comp, struct zcomp_strm *zstrm, .dst_len = PAGE_SIZE, }; + might_sleep(); return comp->ops->decompress(comp->params, &zstrm->ctx, &req); }