From patchwork Mon Jan 27 07:59:26 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13951053 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E3F58C0218F for ; Mon, 27 Jan 2025 08:03:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6DEA5280132; Mon, 27 Jan 2025 03:03:14 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 68E1C28012F; Mon, 27 Jan 2025 03:03:14 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 52E91280132; Mon, 27 Jan 2025 03:03:14 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 33DD028012F for ; Mon, 27 Jan 2025 03:03:14 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id DE434122070 for ; Mon, 27 Jan 2025 08:03:13 +0000 (UTC) X-FDA: 83052491466.09.D3786A9 Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) by imf01.hostedemail.com (Postfix) with ESMTP id 135BB40012 for ; Mon, 27 Jan 2025 08:03:11 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b="oY/PpW8Z"; spf=pass (imf01.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.177 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1737964992; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9BKLEcG6n3DfmSf8Cjanopzri1qniom6oLk870hy4qs=; b=LBr4/ZtYe6Ul+7aURtI1rDhgl2T1hLOhLRa433sO7Zh4Pyscdki3H0aHXpuRSWnHsyv7xZ UMmphlwRGzx3ymyyXryyDeS4NVSxejfzBqf7OBPHQY1fcJDdCDu0jO7v3CNpyygY0IdfM5 zqKfdu9WeaRc1F47CtXx26WQXVxG8Ic= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b="oY/PpW8Z"; spf=pass (imf01.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.177 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1737964992; a=rsa-sha256; cv=none; b=0xR8P5BaiLzSvR0QDqcIEfR1qSgdL396yq9wMvQd/uP0hO2VQJ12keWpmqbxlnLoy52jGC P4QEix6ui/nOaoP1bex8Fo0L41rqDOhSXmX1INsSq03BY00mmTS93eLWFC17AsmUOvDOEy rbQOe0H8E56oAiR52LjOC7BuAt+tg6Y= Received: by mail-pl1-f177.google.com with SMTP id d9443c01a7336-21649a7bcdcso67392005ad.1 for ; Mon, 27 Jan 2025 00:03:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1737964991; x=1738569791; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9BKLEcG6n3DfmSf8Cjanopzri1qniom6oLk870hy4qs=; b=oY/PpW8Z+TNWX+VzsHwt+xem2kxGTDGKKR9vrzGzV5kVEVJraTwSZHoksi44+azxLO FALwGGqONOWJ1o85Aeb8L8LbLQbQ8d4vOMQ9Frabj6LdKBqB5150NC/lsVEB4VBF5BST ikCXL64Ye7R8mPmASGepwjN7BqWVAytWdbIZ8= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737964991; x=1738569791; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9BKLEcG6n3DfmSf8Cjanopzri1qniom6oLk870hy4qs=; b=Cl5JmCWgi7N9zXYJhXUREIrWjgQKxaeDIJmI04hPi55MKQqfOXJLKxSPiv6P5QhyxR ykMUU1vumS0StPXtQX7iwlhRjYahh5DHCOA6+4WcrJj0tq7DvNdfhFm1rTlu6G3m375v wia01eSJ6RGTqmHaqke86uLSoyc788+YXZYuI4zDP9ECdD4kcu15OC5fxXxYwVas4Ss3 /6AvrFmRjVMa8WWaZzNln5Rt8JAr5vt8HCzf66rwPQvPhfkzSwe+dgT0oycUjNfXaDTu 6INf1SM4phWhqZaZUGZE6FangTC66uLFddbZ0uKFpJV1FW0Oqts10KgH65ReQvR7/EJv KMRQ== X-Gm-Message-State: AOJu0YynsAAw0Fl8YjrPrURz3v8+7XUWyx20BTOsKlLEUMhi49y5Zl2b XVMrBS6VZDy21EqT6fzlXYw4fL07lqgdFTIln7RV8tjTJuuC7UNJh2MNE1e6+g== X-Gm-Gg: ASbGncsgg9RH3uaxR3MQnTJysqnZTpaqr6FIDgXFfpOqUg8NEanEiPqxAFf/PJShh3o pg5i+BOoCVRkHEVms5qowqpN3wVD5xQMg7wogjSQQTkozD/1ZKf/DGzdfwF8gI/zdeO/hNDs0Aa M+ltbeVwP03fZ8DZVI34dZyLm4Tha/dGaU34X8l8iV/QKZms/S51N3rcpXKDmZ3PtteuuAp8iJ7 dmsuFP+h1NRWgGBhaUjY8oDyvAMB9kQR3DZr96iGX69Lqx9RP4G8tRwRqpuaBmyWwNnozLnx8IW gEPrCUA= X-Google-Smtp-Source: AGHT+IGkzFE/72UtLpMeB1N62Pwf7Zxax/xK4f/D9pilE4rZeetABBV2aUoAWwiZJw+ncza8QLiWeg== X-Received: by 2002:a17:903:124d:b0:21a:8dec:e57a with SMTP id d9443c01a7336-21c35619607mr540586485ad.48.1737964990824; Mon, 27 Jan 2025 00:03:10 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:566d:6152:c049:8d3a]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-21da3d9c9f2sm57195075ad.42.2025.01.27.00.03.08 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 27 Jan 2025 00:03:10 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim , Johannes Weiner , Yosry Ahmed , Nhat Pham Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [RFC PATCH 1/6] zram: deffer slot free notification Date: Mon, 27 Jan 2025 16:59:26 +0900 Message-ID: <20250127080254.1302026-2-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog In-Reply-To: <20250127080254.1302026-1-senozhatsky@chromium.org> References: <20250127080254.1302026-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 135BB40012 X-Stat-Signature: rt49j6twqpi1zya8x1utdd3o74c7fchw X-Rspamd-Server: rspam08 X-Rspam-User: X-HE-Tag: 1737964991-289767 X-HE-Meta: U2FsdGVkX1/wRRErf2cDb8M1ZnAv/jvtSVOpuYXM3d0TbEx27TcDQWWBJejcaLOaLxVtMwSI5UvvLoby17j9KE8BjCuEAqRT6OMtA3NVjA+ypGSpmFo9W/otisnv6VehWUrMVs2b1A7M+JkRc4nfofL/NKUnyelXORG+Mv0yKHDtXEeWn/HlKIK0WMGCv0d5R4Jd+td4dxcsfMKDzRsscby/EGI0bRgS6xQ0MHlCP9AnSfyGrcDKcXxdqPcx3sGzG7+7+sSQUrOQVrvQgDfh5cQKBpw9Zn4nEr8hM1RCh1/mYbqNKBpC3X9mGntSM22VyFqG/+MmQ6yAMehRsKnxMDqLeB91E6MNny5wCRN4Ap4NhmVh+aeF7OrmKioqoxwmuZ897OkzOu5ERNLEtj2+PEdRKNv98Sc7BG1tE4Abh38AptqMkJHRs6nm8anbjhwSAgkAuoQsmACdgDftQ84425TrT6hcSHKbwqsveCceoUlXD0AEzuCxdFJhOm3Y4djQQ/Vzt4gu9dfAoyDfEo9fXf8NKhwrT+Ikg9LPHb/5KVuKEb5QojFl4tO39d7B/PvU4he4a1ZNfmhWshR3pFPBuW0OBb7Z/KlrUDADqXebXdrZNyQ7z/6o9+JGNS3FbzsRtkdV4/BF6u8valoO+awBQsPzRmnUTzLdv4A05PRNpJrdCi/rkJNAiFdcWrHOaeLhYPYUMr18C78iMzhyH5h74jJDPiplK0glJ5g2lEMYfoVvDvwpgpitCvsEDf8mq27RYR2RrDUeHSPcmxFrIIpnHlbp5xQ8BLxfC6iI5fIJD0XLvr8r8osuYmgKXewjPMe3s/4Mu5BWfRaCRmUEWFg0/Gou2TzJMqKrLL1u9nULcVXJlUpIY+/Rykm0yfN/V0+pLN1tfemDiE4eB04K+RTivSbbnEmFFisOdZMzWgipuylyfiWubT+UXRdcu5QYYnXWobxTY8x3sHgqWbGBHwt G1M78oPR aDO6YAl+FLV8BJ5UJoU+tnrylmfpUSiKeZH48Olmy2ggJxCRtdSzJMC53z+RHwE+Fe6mlAeBH6TcZfgNuxQcD8rDL9U7F8uCeUzVNulIPWi+B1u7hokBQD3zT/Bmy36mmchcWj07NlFhCHZsYs3iSHEs3FokPscsWbje7ACFDckpC1USR5OrxNUIWFoPX5YGmNkIR6XsuDkYHC3apIafoijcPRJbIcRC1hVzLIb6DqUyu8y4Z3Jlf4m8JUHvufvmoAkeUVg1+N9tFKMSwS5M+g/hc+pbKrhJrdItC0BzkH9fxoZLcZjigoJwtew== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: As of this moment ->swap_slot_free_notify is called from atomic section (under spin-lock) which makes it impossible to make zsmalloc fully preemptible. Deffer slot-free to a non-atomic context. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zram_drv.c | 66 +++++++++++++++++++++++++++++++++-- drivers/block/zram/zram_drv.h | 4 +++ 2 files changed, 68 insertions(+), 2 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index ad3e8885b0d2..9c72beb86ab0 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -2315,9 +2315,52 @@ static void zram_submit_bio(struct bio *bio) } } +static void async_slot_free(struct work_struct *work) +{ + struct zram *zram = container_of(work, struct zram, slot_free_work); + + spin_lock(&zram->slot_free_lock); + while (!list_empty(&zram->slot_free_list)) { + struct zram_pp_slot *pps; + + pps = list_first_entry(&zram->slot_free_list, + struct zram_pp_slot, + entry); + list_del_init(&pps->entry); + spin_unlock(&zram->slot_free_lock); + + zram_slot_write_lock(zram, pps->index); + if (zram_test_flag(zram, pps->index, ZRAM_PP_SLOT)) + zram_free_page(zram, pps->index); + zram_slot_write_unlock(zram, pps->index); + + kfree(pps); + spin_lock(&zram->slot_free_lock); + } + spin_unlock(&zram->slot_free_lock); +}; + +static void zram_kick_slot_free(struct zram *zram) +{ + schedule_work(&zram->slot_free_work); +} + +static void zram_flush_slot_free(struct zram *zram) +{ + flush_work(&zram->slot_free_work); +} + +static void zram_init_slot_free(struct zram *zram) +{ + spin_lock_init(&zram->slot_free_lock); + INIT_LIST_HEAD(&zram->slot_free_list); + INIT_WORK(&zram->slot_free_work, async_slot_free); +} + static void zram_slot_free_notify(struct block_device *bdev, - unsigned long index) + unsigned long index) { + struct zram_pp_slot *pps; struct zram *zram; zram = bdev->bd_disk->private_data; @@ -2328,7 +2371,24 @@ static void zram_slot_free_notify(struct block_device *bdev, return; } - zram_free_page(zram, index); + if (zram_test_flag(zram, index, ZRAM_PP_SLOT)) + goto out; + + pps = kzalloc(sizeof(*pps), GFP_ATOMIC); + if (!pps) { + atomic64_inc(&zram->stats.miss_free); + goto out; + } + + INIT_LIST_HEAD(&pps->entry); + pps->index = index; + zram_set_flag(zram, index, ZRAM_PP_SLOT); + spin_lock(&zram->slot_free_lock); + list_add(&pps->entry, &zram->slot_free_list); + spin_unlock(&zram->slot_free_lock); + + zram_kick_slot_free(zram); +out: zram_slot_write_unlock(zram, index); } @@ -2473,6 +2533,7 @@ static ssize_t reset_store(struct device *dev, /* Make sure all the pending I/O are finished */ sync_blockdev(disk->part0); + zram_flush_slot_free(zram); zram_reset_device(zram); mutex_lock(&disk->open_mutex); @@ -2618,6 +2679,7 @@ static int zram_add(void) atomic_set(&zram->pp_in_progress, 0); zram_comp_params_reset(zram); comp_algorithm_set(zram, ZRAM_PRIMARY_COMP, default_compressor); + zram_init_slot_free(zram); /* Actual capacity set using sysfs (/sys/block/zram/disksize */ set_capacity(zram->disk, 0); diff --git a/drivers/block/zram/zram_drv.h b/drivers/block/zram/zram_drv.h index b7e250d6fa02..27ca269f4a4e 100644 --- a/drivers/block/zram/zram_drv.h +++ b/drivers/block/zram/zram_drv.h @@ -134,5 +134,9 @@ struct zram { struct dentry *debugfs_dir; #endif atomic_t pp_in_progress; + + spinlock_t slot_free_lock; + struct list_head slot_free_list; + struct work_struct slot_free_work; }; #endif From patchwork Mon Jan 27 07:59:27 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13951054 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A6B5CC0218F for ; Mon, 27 Jan 2025 08:03:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 382BE280133; Mon, 27 Jan 2025 03:03:20 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3349E28012F; Mon, 27 Jan 2025 03:03:20 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1D2F2280133; Mon, 27 Jan 2025 03:03:20 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id F263228012F for ; Mon, 27 Jan 2025 03:03:19 -0500 (EST) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id A3CB41A20C8 for ; Mon, 27 Jan 2025 08:03:19 +0000 (UTC) X-FDA: 83052491718.23.A48D733 Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) by imf26.hostedemail.com (Postfix) with ESMTP id AA04114001C for ; Mon, 27 Jan 2025 08:03:17 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=MfbDQpOf; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf26.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.172 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1737964997; a=rsa-sha256; cv=none; b=jj8TiDjkHmS9WccNUALmS6DZqgpSErCGFVw4RumDtIW5W9hooLJtORxuqaoNQPB5TcATNG M4jzdmhdNex06lTxpRwLJcL5W3K1tWxnRRPeWcn8Gl43/hva2/d9Qfu3f3W8TLWSnVhUpn 0hYYTbA4X4nPhYP0YmWEN6LZ89W/8ck= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=MfbDQpOf; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf26.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.172 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1737964997; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=sJOiK/AZ7SDP8ZEH670moHkVzGkxeiUJkhPpwvQZYoo=; b=cO8V19Fm+AqJ2VjsxgtuWKkJjINkhgzuvyb3zW5XSYsCP6OEs3kZt31PrqVWdlM/Lbrqmt d8Nw68uezE6Dzmbl7psyZW24J48c2AoHrZWiXtHUJZ7Uqyp3GTo7PiNQ3E6CZooP0qkSqy 7SvyxDjpiTugmN1qP2NTBeluyfeoLUE= Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-21649a7bcdcso67393805ad.1 for ; Mon, 27 Jan 2025 00:03:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1737964996; x=1738569796; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=sJOiK/AZ7SDP8ZEH670moHkVzGkxeiUJkhPpwvQZYoo=; b=MfbDQpOfHl990ESNbRf8GnITkxztjBX2akTsaxJB7kSvQl2pt2v7kWJnE88/GSCFfr 2exbl8wjZJUTQcei0NY4aWtqEoPt4GAcuuucyAsj2IYgodRRcvTFGPO4xPzIYhof3JJg U1ugyKkqs6RBA4jcj1iLgDwUwCptK2Io48ZNw= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737964996; x=1738569796; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=sJOiK/AZ7SDP8ZEH670moHkVzGkxeiUJkhPpwvQZYoo=; b=FrxfSuUljsrRxrgtEfz0+qL9xLTZAFKo0IU1YR+E9K6aZ/YSgYBJzyM8tPjgAinfEP GtWPdK+phsSVNUc6CPUzKLBXxVobaRmy/TaqA0w+K/I/S12xCUwc9BjeW3NdUPiidiEA WhB/xal6nuUcfegpix9S42B0spL7D8owKYN1rZz8nrZ1cZehoRn4bOSmvEubeverzy4S dRJyB/bHyx4waN2zXzLp8w+4oZE4Ce6oxZaTZpzVWGTdI+ZYwUm0UfO4LKV741phgMgU 2Z/UvjIb0TFy2VDCeqsPj6fwZL9NOvI+ABQCpofY/ftLIaZ2NfHTp6QofHR1VYSFbKMu I9Iw== X-Gm-Message-State: AOJu0Yz9VcqfGjE4h+S3wH+5nVIxRd4mu1QRP9pa6SZSBC0DeMs4Db2I TFYTTjMYfHFyr/5PWD4SEtcQhVIULYHPnZ9fZOT+de3z0qf+3quOBrmZ6R4TAA== X-Gm-Gg: ASbGncvW131ITnbJ5h9YYaEga/mhZsOk8P9AbHn51GDdmihXjUyD4DsvfugBtBiXlOy 95fX5IRIoP2xWjkZ/BslWUeatdnwp5AaBFGBeOya4gO4QrAvqUjE61rMWObegpdVSPqUycjPvA9 AdLkLzwLrrLkKG/6f833NH0APRL/pOj3xUL6NYaKMBnZlJ95+PQqNeYYAFxdFDCTe2hBzsteL12 ZYy9cfq98pA3lyCFlVNMxLEHaYUzgWPUb9+F9K5r52uMDnk28QGk2wiPmBkAKyzAHmsVJnYHSlf YTEq2SU= X-Google-Smtp-Source: AGHT+IHumB/yavt3u5eoYPAQOgJWZc93KAkbBNAvMwvcgxL7JyG5xBcpCGjcvKI+xBBcHvQIx6IveQ== X-Received: by 2002:a17:902:d2c8:b0:216:5e6e:68ae with SMTP id d9443c01a7336-21c3558c76bmr601303625ad.31.1737964996493; Mon, 27 Jan 2025 00:03:16 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:566d:6152:c049:8d3a]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-21da3d9dd80sm57265565ad.33.2025.01.27.00.03.14 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 27 Jan 2025 00:03:16 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim , Johannes Weiner , Yosry Ahmed , Nhat Pham Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [RFC PATCH 2/6] zsmalloc: make zspage lock preemptible Date: Mon, 27 Jan 2025 16:59:27 +0900 Message-ID: <20250127080254.1302026-3-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog In-Reply-To: <20250127080254.1302026-1-senozhatsky@chromium.org> References: <20250127080254.1302026-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: AA04114001C X-Stat-Signature: bbxnxdex7po866e1g6w4rxjmdpdx1pgo X-Rspam-User: X-HE-Tag: 1737964997-91772 X-HE-Meta: U2FsdGVkX1/iqVdD3Ekhkxb1RWPir4xaf6tutRLKEV11EIHLQv5MLscTbQyI126LypjD/mptVZpocBuZyFHZ+U7RWxkhe/Y6M0KNQNqjzOOMSES5+/sGV8m4culKr5R6+xHmnN6/MF5IAgQcSrC0weTDkzlMoZB55zvQXFJgTjl1BUfLPmbTvq/JMQqyU4HtA5WBySr10V34zY+lssyCunJ3mi/0yZ+Q5ZW/y2tUu55iQgx5N5OZRZ2WgdlPNqkmyONK2G9bxeyOO/ki8lRTizK4DcnkRLJUKAsAPvD4yPQ4tfXTa7WEgnGJUQMfo8/kK1fM+5D5PIrQYlttMXBhw9pL8dm048FOLRl9089fvJ5dysxo3NBVbf2b+cBbTj+gTWrslAJayOv76nNIMRNRU7V46hx+NvnSQiOhLMWXWT3UmiuAH9Tk3P3SlsE5lO8HVI9SjWF3r8cnjszU/8I0k59zgfL3bp/xm9W10EArtED2fhokr4Kv7ggFD/JU0W2Igr5igXPqYJskoI2AJEJEJYSlG7js0eL3MYDunxZ+pWWuciWGwrUZUvosm34F9YqW9SGG+llH/X0vhGZYK5alrdzPK/F5bu0TBHVuxAVPZppid5zrba5LgJ44tiC4TOCeEhggh/+lpn00xK2wQ+0402fZvR1uqJFTfY4zR+gt1ccygTu9S/mxMsgAGs46NayNV/CmOyHS1mJFxC43jnmpHCfdaq9vGsD7GXDTorhmiOjh7dlXkHCIm842clnDnueu3hozZgyNswJHTADAO40ywLC6y/CczIPaATYSuh0DfhPNKzHFGRMQG/nE1PvjXD73KdfIIteUh8LvzuG25/Bv+Hj2WlyLP6gg+CKZKP4oUk8w3pRL5ffPMIwvdc1TpmyRv6F8WwQxf5zY1kUOYd8vpJbRRaeY4QJp+pVSe3ntcqrCW/PtC/FXoVG/sViR6AM0wxQO+w5QExqzoMCGrS2 vjIHsBec oviLX5UgGioRTzedZWxLc/DnleH3AxtgyeAEe+98iLeQE3XFanCaambgivOuI3nmL8sdSfAKeq1jtuygnubiMLFxsDA+dPrceX6LB8c9NQqekHgXUSCoXJzQwmxb452rxVwSDyFrXc5ijjdUhh0Ff2vI1S0GsgEB1iC59VumRDx3ewjI5dLaZyOb+c5YZayo+b1UmIwjEAOYKPIHMG1Uq6L1Ejqix4jX9aLyN9fQU0bnavnnsSVvW44DwtnFFEs945UAgbe8W+etOr1DWH80Feg4FnV31p95l3rlkf3ILe+9XZUtZpSQ/kIZguw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Switch over from rwlock_t to a atomic_t variable that takes negative value when the page is under migration, or positive values when the page is used by zsmalloc users (object map, etc.) Using a rwsem per-zspage is a little too memory heavy, a simple atomic_t should suffice, after all we only need to mark zspage as either used-for-write or used-for-read. This is needed to make zsmalloc preemtible in the future. Signed-off-by: Sergey Senozhatsky --- mm/zsmalloc.c | 112 +++++++++++++++++++++++++++++--------------------- 1 file changed, 66 insertions(+), 46 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 817626a351f8..28a75bfbeaa6 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -257,6 +257,9 @@ static inline void free_zpdesc(struct zpdesc *zpdesc) __free_page(page); } +#define ZS_PAGE_UNLOCKED 0 +#define ZS_PAGE_WRLOCKED -1 + struct zspage { struct { unsigned int huge:HUGE_BITS; @@ -269,7 +272,7 @@ struct zspage { struct zpdesc *first_zpdesc; struct list_head list; /* fullness list */ struct zs_pool *pool; - rwlock_t lock; + atomic_t lock; }; struct mapping_area { @@ -290,11 +293,53 @@ static bool ZsHugePage(struct zspage *zspage) return zspage->huge; } -static void migrate_lock_init(struct zspage *zspage); -static void migrate_read_lock(struct zspage *zspage); -static void migrate_read_unlock(struct zspage *zspage); -static void migrate_write_lock(struct zspage *zspage); -static void migrate_write_unlock(struct zspage *zspage); +static void zspage_lock_init(struct zspage *zspage) +{ + atomic_set(&zspage->lock, ZS_PAGE_UNLOCKED); +} + +static void zspage_read_lock(struct zspage *zspage) +{ + atomic_t *lock = &zspage->lock; + int old; + + while (1) { + old = atomic_read(lock); + if (old == ZS_PAGE_WRLOCKED) { + cpu_relax(); + continue; + } + + if (atomic_cmpxchg(lock, old, old + 1) == old) + return; + + cpu_relax(); + } +} + +static void zspage_read_unlock(struct zspage *zspage) +{ + atomic_dec(&zspage->lock); +} + +static void zspage_write_lock(struct zspage *zspage) +{ + atomic_t *lock = &zspage->lock; + int old; + + while (1) { + old = atomic_cmpxchg(lock, ZS_PAGE_UNLOCKED, ZS_PAGE_WRLOCKED); + if (old == ZS_PAGE_UNLOCKED) + return; + + cpu_relax(); + } +} + +static void zspage_write_unlock(struct zspage *zspage) +{ + atomic_set(&zspage->lock, ZS_PAGE_UNLOCKED); +} #ifdef CONFIG_COMPACTION static void kick_deferred_free(struct zs_pool *pool); @@ -992,7 +1037,7 @@ static struct zspage *alloc_zspage(struct zs_pool *pool, return NULL; zspage->magic = ZSPAGE_MAGIC; - migrate_lock_init(zspage); + zspage_lock_init(zspage); for (i = 0; i < class->pages_per_zspage; i++) { struct zpdesc *zpdesc; @@ -1217,7 +1262,7 @@ void *zs_map_object(struct zs_pool *pool, unsigned long handle, * zs_unmap_object API so delegate the locking from class to zspage * which is smaller granularity. */ - migrate_read_lock(zspage); + zspage_read_lock(zspage); read_unlock(&pool->migrate_lock); class = zspage_class(pool, zspage); @@ -1277,7 +1322,7 @@ void zs_unmap_object(struct zs_pool *pool, unsigned long handle) } local_unlock(&zs_map_area.lock); - migrate_read_unlock(zspage); + zspage_read_unlock(zspage); } EXPORT_SYMBOL_GPL(zs_unmap_object); @@ -1671,18 +1716,18 @@ static void lock_zspage(struct zspage *zspage) /* * Pages we haven't locked yet can be migrated off the list while we're * trying to lock them, so we need to be careful and only attempt to - * lock each page under migrate_read_lock(). Otherwise, the page we lock + * lock each page under zspage_read_lock(). Otherwise, the page we lock * may no longer belong to the zspage. This means that we may wait for * the wrong page to unlock, so we must take a reference to the page - * prior to waiting for it to unlock outside migrate_read_lock(). + * prior to waiting for it to unlock outside zspage_read_lock(). */ while (1) { - migrate_read_lock(zspage); + zspage_read_lock(zspage); zpdesc = get_first_zpdesc(zspage); if (zpdesc_trylock(zpdesc)) break; zpdesc_get(zpdesc); - migrate_read_unlock(zspage); + zspage_read_unlock(zspage); zpdesc_wait_locked(zpdesc); zpdesc_put(zpdesc); } @@ -1693,41 +1738,16 @@ static void lock_zspage(struct zspage *zspage) curr_zpdesc = zpdesc; } else { zpdesc_get(zpdesc); - migrate_read_unlock(zspage); + zspage_read_unlock(zspage); zpdesc_wait_locked(zpdesc); zpdesc_put(zpdesc); - migrate_read_lock(zspage); + zspage_read_lock(zspage); } } - migrate_read_unlock(zspage); + zspage_read_unlock(zspage); } #endif /* CONFIG_COMPACTION */ -static void migrate_lock_init(struct zspage *zspage) -{ - rwlock_init(&zspage->lock); -} - -static void migrate_read_lock(struct zspage *zspage) __acquires(&zspage->lock) -{ - read_lock(&zspage->lock); -} - -static void migrate_read_unlock(struct zspage *zspage) __releases(&zspage->lock) -{ - read_unlock(&zspage->lock); -} - -static void migrate_write_lock(struct zspage *zspage) -{ - write_lock(&zspage->lock); -} - -static void migrate_write_unlock(struct zspage *zspage) -{ - write_unlock(&zspage->lock); -} - #ifdef CONFIG_COMPACTION static const struct movable_operations zsmalloc_mops; @@ -1803,8 +1823,8 @@ static int zs_page_migrate(struct page *newpage, struct page *page, * the class lock protects zpage alloc/free in the zspage. */ spin_lock(&class->lock); - /* the migrate_write_lock protects zpage access via zs_map_object */ - migrate_write_lock(zspage); + /* the zspage_write_lock protects zpage access via zs_map_object */ + zspage_write_lock(zspage); offset = get_first_obj_offset(zpdesc); s_addr = kmap_local_zpdesc(zpdesc); @@ -1835,7 +1855,7 @@ static int zs_page_migrate(struct page *newpage, struct page *page, */ write_unlock(&pool->migrate_lock); spin_unlock(&class->lock); - migrate_write_unlock(zspage); + zspage_write_unlock(zspage); zpdesc_get(newzpdesc); if (zpdesc_zone(newzpdesc) != zpdesc_zone(zpdesc)) { @@ -1971,9 +1991,9 @@ static unsigned long __zs_compact(struct zs_pool *pool, if (!src_zspage) break; - migrate_write_lock(src_zspage); + zspage_write_lock(src_zspage); migrate_zspage(pool, src_zspage, dst_zspage); - migrate_write_unlock(src_zspage); + zspage_write_unlock(src_zspage); fg = putback_zspage(class, src_zspage); if (fg == ZS_INUSE_RATIO_0) { From patchwork Mon Jan 27 07:59:28 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13951055 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DEDC3C0218C for ; Mon, 27 Jan 2025 08:03:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 70436280116; Mon, 27 Jan 2025 03:03:27 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6B6042800E8; Mon, 27 Jan 2025 03:03:27 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 555C7280116; Mon, 27 Jan 2025 03:03:27 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 3708F2800E8 for ; Mon, 27 Jan 2025 03:03:27 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id CBB78820C3 for ; Mon, 27 Jan 2025 08:03:26 +0000 (UTC) X-FDA: 83052492012.27.C3C1F1B Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) by imf19.hostedemail.com (Postfix) with ESMTP id E74AA1A0006 for ; Mon, 27 Jan 2025 08:03:24 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b="c92d/bH0"; spf=pass (imf19.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.170 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1737965005; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PXEed/HJJxd4qZRhg1cqdVLafAkTWhNb+3EhlC5igm8=; b=5WZu2q/+7lvg/pVixp7YLQaTjCYe9fScXZ8oavsJWhEDo3dLqxFhdHZtfSMwc1mIvfkt8G kIt4tijXYu9vQpyzAVTYAzv/sPmV8DZA8EHcR6/w5wPSF+4B6L/1Ow7JDnzwYt5hXa5VGq Ugb5UeXRPRQTruoTn5lKwGJ5s2cpfBI= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b="c92d/bH0"; spf=pass (imf19.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.170 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1737965005; a=rsa-sha256; cv=none; b=DBVxYQ3Y9dJdsLSiazg6xC9yYaZrBLPuFO/4S0FFxYI2J4e2HfNCGWMVfBzACWmHsWm2hT Qu83/anTuH445Qj9YlmZbT3IosCR9VNd/RqSG7IgKiNZvd8TX+B+uYXaKkYv0M/+cFTcAK WHkQv3UaV0PDPGcJod97ZmEREoaVJS8= Received: by mail-pl1-f170.google.com with SMTP id d9443c01a7336-2164b662090so77919525ad.1 for ; Mon, 27 Jan 2025 00:03:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1737965004; x=1738569804; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=PXEed/HJJxd4qZRhg1cqdVLafAkTWhNb+3EhlC5igm8=; b=c92d/bH02dvUolx+U2AStQtdH5vf5/ymPEp4YLdNaYkDVugn2Y2USXdsWo0pBjiJ0h 5LEbBQHMvvuJybTHZV63Rhf4y1/XBVSpPoNJLQjxHxlNZa8NwSt/K+rRCH9OugTmBKX2 VbS2Yh20mIVL/KZIX8MKNEBVicDX6eKnith+4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737965004; x=1738569804; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=PXEed/HJJxd4qZRhg1cqdVLafAkTWhNb+3EhlC5igm8=; b=iJ4OMHKeitYNNbGJBY4vMudq8JJ08okgEeHPoajjd2ydLyvAVdZmCEvgjK7t1jUY2G j6te5G/n9JV1IiW3Yj24CR+DyJPD9VTsRuUGuAqL411Ub5xSk/Vmcrdpa5Y1YpY1QH7I A/d4+Dd87ORlSlf+ixDtZrG+74T1igDiTbpgenxsP00Ko+3sWEg5Vt/VpCtcics00mGM v4E7cDHzne0v/+UVOUf/ziFSMQc9VsS7MDgm7vkkaF6Yino52s7VGjwLIaTmyufMkI6z GzTBpUPBEMqzmmfARmmPBmL8QGncuJYIXpV9WccT5JNL+LQmbngpRNq81+dYz/L9I5uW Q20g== X-Gm-Message-State: AOJu0YxlrpVHqg49IHJogIAUGTAcjp8zYCCxv/XzVf9qOJmK13o2V6wS GP0ssxS0ZF5ol+1lndtO9qnT8+ElSlhDFVhSwjJCZLZwuVLJ/NTp2uw3h3tIyQ== X-Gm-Gg: ASbGncvC/0QmqJMaSi6Ypk/EBoIEAFTKc510yr+U1QOobG6LK62QqF0GpXT24Iu3FJb 0aPyHFNqcxSywsHf+5V0Q1vEB4cey/Yzwwh7TxCFGBrXNag3rC/NL44mLrhvs5R2X7fgy9XkkZI oh8ilKP2CeoTPSGduTOaPAbriRKDN2L3NdmOLygiQiGmtysUNOh2+ggvlwIpvgSaSbxYKabY5MQ 3Gvlw0GskWd6/eZ3i9D/JX73SCQgTaR5fUm+crsrVY4eqVrodAYFxi05swugg2jwJD+a6f3tiq0 eEJezuI= X-Google-Smtp-Source: AGHT+IEL57Fr+lPzD2FowIHfSuVgiI3V3YUg02jovjSl+Fh1RpWQYSxO6GJ+HfktJR9cPnagmL7JRg== X-Received: by 2002:a05:6a00:2e15:b0:728:e906:e45a with SMTP id d2e1a72fcca58-72dafbfd7e4mr61554805b3a.24.1737965002160; Mon, 27 Jan 2025 00:03:22 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:566d:6152:c049:8d3a]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-72f8a78ee7bsm6686774b3a.170.2025.01.27.00.03.19 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 27 Jan 2025 00:03:21 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim , Johannes Weiner , Yosry Ahmed , Nhat Pham Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [RFC PATCH 3/6] zsmalloc: convert to sleepable pool lock Date: Mon, 27 Jan 2025 16:59:28 +0900 Message-ID: <20250127080254.1302026-4-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog In-Reply-To: <20250127080254.1302026-1-senozhatsky@chromium.org> References: <20250127080254.1302026-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: E74AA1A0006 X-Stat-Signature: eh4j9pp7oj3rycsxs464b1grsh36m7xc X-HE-Tag: 1737965004-973344 X-HE-Meta: U2FsdGVkX1+c/iLbBxv2/f7gXS9BDnQCMFUhh1ccRn61jISeslz12Ts54ZVgtbRppwqok7iqIGv5TKXYMsDXsJwtqJ0/uwQSluFD/gUB9NEYKk8jS/IgSG09q6nx0JOJzvUDSYN5RswnjsZd/QZXumhRt/pkbhpM1Qps9OnanMuK53WqUvqr20ja0KIW8kl632bqx6UqWa7QqjfhXjMzvRXbcGnvrfZlNuuBhrsn96IdYGPLl53A0TqVBrMVCWEwbeoVlaHTcmERBWzRozrxtlIo1VZEcd/Flk5FOYhKkQ0JtifURcf/5m7IKpG8QeUeO0/Ibfnr50DB5bEm9YfVUkeoRm84ytlmptvzB6ekIiEjrK9B5FSnIZgoB80XZRBNZy3BEwnM9BU+pxH5EmEF8tCQKFdcWg6sXvaj4ofb7kMzt0eMxZw6Div4edMxny/iPHRmMr2aV42xzw1NIbR1OqHM3b7KgbESnu/LLYzS2yZS7RwKD8SVgcL3Af+IsP0TRMUfNv4zeqUjr5DguJl5o7EsgWac7knL2XlXXbssNX0t9GsgTPdAXWjVhi1jFczYoPuhS+CRARGma+Oqk6VCmnJqgJaB3C4ocm8kROqLpbAzVzX/nhPbF2AdHejHs4NMhUI/sDX9K/HvLehgXdzHLCvanX5MO1WUjoijnzPYN312y7KtKrTV7Z6W59bmVm18tcJ5QT2g1k8nKPqhZ5+TysyhaXgimgGGpnmI7nFvXd71GP2Xv7/EMX2Md+AGslq/H6/nVUU5UTcQle38TTT9ehfbfdCZKryvlJHwt4RG2F46A21HWQR7AJurIUSEqIHPGn9NvA4Q5b0daUytwRjOphqWOHOkx+cxcIxWIVAGQaJy86tm+SVt1PNx/DPYjwNReyjmSuRde2AdydAsRz+kdqP+Wgb+GstzUnnc6aQcbdjonLjdryhlawUewWPt6dDyOySbwT9xq+j4jHYuoZX F361G92d t54x559wUc+vqCtiqQ4zfL56xcTPkFh3xe1cFeIGIHov1h65wX/0YZ9cFYES+kHHuwSZWfAUY6awAyf+B7Ma9sgEBhl4G5O/+YZvM+zxIesR8oHKY7DIhdkpe7MK9PhuyZRIwMQAgTcae0ajPPDTjlcVMcx9UPjFWbjauybEUJSRZC6WH0koALsaY8+5JKYOIEIANe19VVa9hAuOYmLY9pmOSlq2SCkbOMak6nCZH3CgzQZdtw2Kc6aM9Wuy17M2kgOq0o4l99KyHcMZcFMUHpRd/uEILGGbcWGNPUG5fkt+qpUSwIYTN3HSw7w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Switch over from rwlock_t to rwsemaphore, also introduce simple helpers to lock/unlock the pool. This is needed to make zsmalloc preemptible in the future. Signed-off-by: Sergey Senozhatsky --- mm/zsmalloc.c | 58 ++++++++++++++++++++++++++++++++++++--------------- 1 file changed, 41 insertions(+), 17 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 28a75bfbeaa6..751871ec533f 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -204,8 +204,8 @@ struct link_free { }; struct zs_pool { - const char *name; - + /* protect page/zspage migration */ + struct rw_semaphore migrate_lock; struct size_class *size_class[ZS_SIZE_CLASSES]; struct kmem_cache *handle_cachep; struct kmem_cache *zspage_cachep; @@ -216,6 +216,7 @@ struct zs_pool { /* Compact classes */ struct shrinker *shrinker; + atomic_t compaction_in_progress; #ifdef CONFIG_ZSMALLOC_STAT struct dentry *stat_dentry; @@ -223,11 +224,34 @@ struct zs_pool { #ifdef CONFIG_COMPACTION struct work_struct free_work; #endif - /* protect page/zspage migration */ - rwlock_t migrate_lock; - atomic_t compaction_in_progress; + const char *name; }; +static void pool_write_unlock(struct zs_pool *pool) +{ + up_write(&pool->migrate_lock); +} + +static void pool_write_lock(struct zs_pool *pool) +{ + down_write(&pool->migrate_lock); +} + +static void pool_read_unlock(struct zs_pool *pool) +{ + up_read(&pool->migrate_lock); +} + +static void pool_read_lock(struct zs_pool *pool) +{ + down_read(&pool->migrate_lock); +} + +static bool zspool_lock_is_contended(struct zs_pool *pool) +{ + return rwsem_is_contended(&pool->migrate_lock); +} + static inline void zpdesc_set_first(struct zpdesc *zpdesc) { SetPagePrivate(zpdesc_page(zpdesc)); @@ -1251,7 +1275,7 @@ void *zs_map_object(struct zs_pool *pool, unsigned long handle, BUG_ON(in_interrupt()); /* It guarantees it can get zspage from handle safely */ - read_lock(&pool->migrate_lock); + pool_read_lock(pool); obj = handle_to_obj(handle); obj_to_location(obj, &zpdesc, &obj_idx); zspage = get_zspage(zpdesc); @@ -1263,7 +1287,7 @@ void *zs_map_object(struct zs_pool *pool, unsigned long handle, * which is smaller granularity. */ zspage_read_lock(zspage); - read_unlock(&pool->migrate_lock); + pool_read_unlock(pool); class = zspage_class(pool, zspage); off = offset_in_page(class->size * obj_idx); @@ -1498,13 +1522,13 @@ void zs_free(struct zs_pool *pool, unsigned long handle) * The pool->migrate_lock protects the race with zpage's migration * so it's safe to get the page from handle. */ - read_lock(&pool->migrate_lock); + pool_read_lock(pool); obj = handle_to_obj(handle); obj_to_zpdesc(obj, &f_zpdesc); zspage = get_zspage(f_zpdesc); class = zspage_class(pool, zspage); spin_lock(&class->lock); - read_unlock(&pool->migrate_lock); + pool_read_unlock(pool); class_stat_sub(class, ZS_OBJS_INUSE, 1); obj_free(class->size, obj); @@ -1816,7 +1840,7 @@ static int zs_page_migrate(struct page *newpage, struct page *page, * The pool migrate_lock protects the race between zpage migration * and zs_free. */ - write_lock(&pool->migrate_lock); + pool_write_lock(pool); class = zspage_class(pool, zspage); /* @@ -1853,7 +1877,7 @@ static int zs_page_migrate(struct page *newpage, struct page *page, * Since we complete the data copy and set up new zspage structure, * it's okay to release migration_lock. */ - write_unlock(&pool->migrate_lock); + pool_write_unlock(pool); spin_unlock(&class->lock); zspage_write_unlock(zspage); @@ -1976,7 +2000,7 @@ static unsigned long __zs_compact(struct zs_pool *pool, * protect the race between zpage migration and zs_free * as well as zpage allocation/free */ - write_lock(&pool->migrate_lock); + pool_write_lock(pool); spin_lock(&class->lock); while (zs_can_compact(class)) { int fg; @@ -2003,14 +2027,14 @@ static unsigned long __zs_compact(struct zs_pool *pool, src_zspage = NULL; if (get_fullness_group(class, dst_zspage) == ZS_INUSE_RATIO_100 - || rwlock_is_contended(&pool->migrate_lock)) { + || zspool_lock_is_contended(pool)) { putback_zspage(class, dst_zspage); dst_zspage = NULL; spin_unlock(&class->lock); - write_unlock(&pool->migrate_lock); + pool_write_unlock(pool); cond_resched(); - write_lock(&pool->migrate_lock); + pool_write_lock(pool); spin_lock(&class->lock); } } @@ -2022,7 +2046,7 @@ static unsigned long __zs_compact(struct zs_pool *pool, putback_zspage(class, dst_zspage); spin_unlock(&class->lock); - write_unlock(&pool->migrate_lock); + pool_write_unlock(pool); return pages_freed; } @@ -2159,7 +2183,7 @@ struct zs_pool *zs_create_pool(const char *name) return NULL; init_deferred_free(pool); - rwlock_init(&pool->migrate_lock); + init_rwsem(&pool->migrate_lock); atomic_set(&pool->compaction_in_progress, 0); pool->name = kstrdup(name, GFP_KERNEL); From patchwork Mon Jan 27 07:59:29 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13951056 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 63C45C0218C for ; Mon, 27 Jan 2025 08:03:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EA797280117; Mon, 27 Jan 2025 03:03:30 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E562E2800E8; Mon, 27 Jan 2025 03:03:30 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CCF04280117; Mon, 27 Jan 2025 03:03:30 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id AB7492800E8 for ; Mon, 27 Jan 2025 03:03:30 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 57AF6B480A for ; Mon, 27 Jan 2025 08:03:30 +0000 (UTC) X-FDA: 83052492180.09.F8FE7D0 Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) by imf04.hostedemail.com (Postfix) with ESMTP id 791A64000E for ; Mon, 27 Jan 2025 08:03:28 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=dcOSqiOC; spf=pass (imf04.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.175 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1737965008; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kn1+fRD/5z0gRUNjM1y0pmoc8sr5OaAxZlYjVZdB2hU=; b=nB2k4QW8QA2L8PY4sL2uXMYJpll05skUhoWqzfbK8qbYJhgyiBaVHjZ5/LNvEgONsE+ivT TJFB+k80WmaA+zKtGzNmmfpkh0zf/0RuTiPVP3CkUKONQvM3r36LwhGN7y/ZxsGVzIKFQ0 v1LF159CMC0JotOk987FflU0i9U/8qM= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=dcOSqiOC; spf=pass (imf04.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.175 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1737965008; a=rsa-sha256; cv=none; b=cdpPGjvUQ/TEngm3eIC4f2/p5DbMApmJzCqTMiPvG8cLk3IHQZC71axLmV8/R1xj0YTXuF BzpZO1zv2aB0I3wEgIARN0ZpSQ/onAQWs2/n5L4Tw4taotMxqOQhtv6k9HkbhE4/lGlSnw vhu0s246r2ABz2cFKW0P0x7S9R942kE= Received: by mail-pl1-f175.google.com with SMTP id d9443c01a7336-21634338cfdso96463985ad.2 for ; Mon, 27 Jan 2025 00:03:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1737965007; x=1738569807; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kn1+fRD/5z0gRUNjM1y0pmoc8sr5OaAxZlYjVZdB2hU=; b=dcOSqiOCMvxuel8gYB9kuTz5mbnSRv9uE5SdOclMcbTQg0kbQT9VZFjvwOuu/nHpyy EfzJEdRkqkpYiD731pLPa8ZCIyTr4ppQqHheqTY0YhxuoQiPQzHkr+A9sez9jN/iEwhL Jj2tcsrlPvp1Se+qY8Z7kQxC1PNRubyuhtHZM= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737965007; x=1738569807; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kn1+fRD/5z0gRUNjM1y0pmoc8sr5OaAxZlYjVZdB2hU=; b=peNA5O360rxl2+6dEnjtZtFWo7clFrHwSe9o8EKDUj7uX/k/ARo/X7YGo3erin82fX O8z0Xr1Sr1KEs99cLN3qxc3++gqaTkAMyaC0WLsh1/ptbbeNSHG8DsgoLGEwGPEKaomC HzYoNbeJBd9JOg2vqLYTIp73tBKYh8bT5M5kNN3Y1/0b1JHzOxQnQvxBLLrkoRbg67FL +YQRkH9uk4svYdwzAKwGy4rq+lIWbWsL/bLzO32MmPzDtvAfr8N9WebFPLNgPehyRuHU xIFPKG3zXTZtsuAlzQvOOtHlfgXSaO6Zc8w80c4LDpSarnkaQPp+rTwTcbn4C3Ojsput 34Fw== X-Gm-Message-State: AOJu0Yx4I6+G+3/pOM5VOYN0r+8zRRv9CMcxNpMoQRXG+wm5vZUNRfQb iyNBfwITxjylHBhuM9rFfhD5AMr8txxhcuI0sR2oP2/nfZIrm08hsBRBfxm/Ig== X-Gm-Gg: ASbGncuI4iTBn8m9aPmpu7Qtrt7OTyiRZav8k7RoYJCMUS9Of7SIwHBLD23PhU8n95a feCfOqR0EJW9XAztsajrHEYzVxPrViq91PZzJuqmGPptxUE9IoMAO2HjlwfIhfTyvVfPGLs6MJ1 4rhNcOBnaT+0cqxnL0lTogVJSk67Lag3GuyRstt9RujmOr+FGLyq/CeKoqJVK8VmoFJ5tcn2Nsl ZRdcc9Fj36vLYDbh/n8+aigPSkMYD+g7UmzRuVeW+BQjOnKNhqxke2fvZUZ/MjC/seP+tyKLl73 9RbjK7M= X-Google-Smtp-Source: AGHT+IE0KtyysZw7cacIrE1leqh4Co/IJ/3WMyHKfL3/R554bfYapuUKAhoc+fS50PnMBkNIdu63xg== X-Received: by 2002:a17:902:ecc6:b0:217:803a:e47a with SMTP id d9443c01a7336-21c3553b21cmr580819105ad.4.1737965007128; Mon, 27 Jan 2025 00:03:27 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:566d:6152:c049:8d3a]) by smtp.gmail.com with UTF8SMTPSA id d9443c01a7336-21da3ea3081sm56832255ad.62.2025.01.27.00.03.24 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 27 Jan 2025 00:03:26 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim , Johannes Weiner , Yosry Ahmed , Nhat Pham Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [RFC PATCH 4/6] zsmalloc: make class lock sleepable Date: Mon, 27 Jan 2025 16:59:29 +0900 Message-ID: <20250127080254.1302026-5-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog In-Reply-To: <20250127080254.1302026-1-senozhatsky@chromium.org> References: <20250127080254.1302026-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 791A64000E X-Stat-Signature: xkzfmo9s3pfcigynrwtuobxndfkpg1tw X-Rspamd-Server: rspam08 X-Rspam-User: X-HE-Tag: 1737965008-265976 X-HE-Meta: U2FsdGVkX19q+fM55cO49/fNo4ZfG5siFiVR8d3vCxNSFD5/n5Rw/7xp3hxTFXx46ycAtfVNQ7KmJXqDgeXeH7mvAeNMgySx6wpqrfsaHwXV6s5BBUAMUZeLBpZHapAo2H6ZEI0fzNmj37rUCnFY2/sj/guky+22w6FykIXstr+DKSetCBwTEfWYPkKhGmSOmxaSdpsBzkVqJWarcf6WhKMg9MJ2P/G6MRN3ZrR1Bz0U9nLstSlfiGHcB/lnA+dK4QP4gImrwhTVJ12MuibxMAt5CBLaqrBl6F3wbDqc/aEZwef8pT6H1NbABJfie/RAxkWVfYX/eR9ywBgqrKRBB7pf2T2BgFsV8oRmpePQLWjMu5xwg3FxVyL+kc+UB/MBq3s1XbkFc01z2Bsr96o7FdtMHcbLrTvmwIOmysDz8pJ8ZSz/Yz9sJ2+g0/ScTKyO3nQqg4++jAPbiFTQAwovrzlyPe+auztgWZzyrCjxWvSgpNYnyeP1ZWN6ykr3/SHBJUbzkWXxbroRh9S/PUH9I/5kvmRuNCfro8f54ep7uMG0gapXyAAdGNgsPeNhGkfWC0XdrKkVw6XY1+mG1qVfm2ESKfS4umCpmFoBVfPmZ6uZJ2u4gWagJH9IZTbBtUtMDHrR/uVcHY716vzlWSyOXsionPYxEt+eltoWEEn+SGHhW0voDv3oPMEEXs+kmPnhuXd+aSwPOGGIuAeTKOHB9Q1Ofu1wvwcZikup1JLumvkIxmRHpPugXuNib5Mj/z3MxRzM7EcSYBANU41B50POLf2vJvG/oKXfSAfjBgh9zYYzL6h0wGKqVYuhEziL3eFGiAp5nrbQSJVyO/dW1zRayEts8JU5NLXUT8u34Gm93OeU0PCRsOr2N5p4e9kRWRPniwNzVS+Fw3TrhqsNigVv7sRZ7TNxSvFytwPIEhFAXlBUuZGcDDM7r51i5L5r1zZ8+A4UbdFT/rfg44NVYue wHynr+Vi RWWwaC8F4NG+8fEEbMEFMxjKmcUgdOVIQuCTZv9OJtFr43gHZSoTXcMm1eqyTwoILDba/Jf29QlBYLBIYXyXkuJOdhrtuu8T9/yWP7MYmX4nIAfoyHzuboV6ZVGt25wDfR920/qLO5rfb0tp/6f1ukeTOCq77vBYVDwMG4/CAmCD5fNG5PpPjKucCc2CUwf/o8FY6icVhQxCalMYRRE2PoVMtUuqmKm6ipWE0bH5yVpLwBQzOWCpvUR3khXld9U78SkFslfRHunoMjMPgrsiXk9l3uHC2lCu91KYSwd4W4ee5b4ZnuVj/XY4CtQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Switch over from spin-lock to mutex, also introduce simple helpers to lock/unlock size class. This is needed to make zsmalloc preemptible in the future. Signed-off-by: Sergey Senozhatsky --- mm/zsmalloc.c | 54 ++++++++++++++++++++++++++++----------------------- 1 file changed, 30 insertions(+), 24 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 751871ec533f..a5c1f9852072 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -168,7 +168,7 @@ static struct dentry *zs_stat_root; static size_t huge_class_size; struct size_class { - spinlock_t lock; + struct mutex lock; struct list_head fullness_list[NR_FULLNESS_GROUPS]; /* * Size of objects stored in this class. Must be multiple @@ -252,6 +252,16 @@ static bool zspool_lock_is_contended(struct zs_pool *pool) return rwsem_is_contended(&pool->migrate_lock); } +static void size_class_lock(struct size_class *class) +{ + mutex_lock(&class->lock); +} + +static void size_class_unlock(struct size_class *class) +{ + mutex_unlock(&class->lock); +} + static inline void zpdesc_set_first(struct zpdesc *zpdesc) { SetPagePrivate(zpdesc_page(zpdesc)); @@ -657,8 +667,7 @@ static int zs_stats_size_show(struct seq_file *s, void *v) if (class->index != i) continue; - spin_lock(&class->lock); - + size_class_lock(class); seq_printf(s, " %5u %5u ", i, class->size); for (fg = ZS_INUSE_RATIO_10; fg < NR_FULLNESS_GROUPS; fg++) { inuse_totals[fg] += class_stat_read(class, fg); @@ -668,7 +677,7 @@ static int zs_stats_size_show(struct seq_file *s, void *v) obj_allocated = class_stat_read(class, ZS_OBJS_ALLOCATED); obj_used = class_stat_read(class, ZS_OBJS_INUSE); freeable = zs_can_compact(class); - spin_unlock(&class->lock); + size_class_unlock(class); objs_per_zspage = class->objs_per_zspage; pages_used = obj_allocated / objs_per_zspage * @@ -926,8 +935,6 @@ static void __free_zspage(struct zs_pool *pool, struct size_class *class, { struct zpdesc *zpdesc, *next; - assert_spin_locked(&class->lock); - VM_BUG_ON(get_zspage_inuse(zspage)); VM_BUG_ON(zspage->fullness != ZS_INUSE_RATIO_0); @@ -1443,7 +1450,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) class = pool->size_class[get_size_class_index(size)]; /* class->lock effectively protects the zpage migration */ - spin_lock(&class->lock); + size_class_lock(class); zspage = find_get_zspage(class); if (likely(zspage)) { obj_malloc(pool, zspage, handle); @@ -1453,8 +1460,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) goto out; } - - spin_unlock(&class->lock); + size_class_unlock(class); zspage = alloc_zspage(pool, class, gfp); if (!zspage) { @@ -1462,7 +1468,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) return (unsigned long)ERR_PTR(-ENOMEM); } - spin_lock(&class->lock); + size_class_lock(class); obj_malloc(pool, zspage, handle); newfg = get_fullness_group(class, zspage); insert_zspage(class, zspage, newfg); @@ -1473,7 +1479,7 @@ unsigned long zs_malloc(struct zs_pool *pool, size_t size, gfp_t gfp) /* We completely set up zspage so mark them as movable */ SetZsPageMovable(pool, zspage); out: - spin_unlock(&class->lock); + size_class_unlock(class); return handle; } @@ -1527,7 +1533,7 @@ void zs_free(struct zs_pool *pool, unsigned long handle) obj_to_zpdesc(obj, &f_zpdesc); zspage = get_zspage(f_zpdesc); class = zspage_class(pool, zspage); - spin_lock(&class->lock); + size_class_lock(class); pool_read_unlock(pool); class_stat_sub(class, ZS_OBJS_INUSE, 1); @@ -1537,7 +1543,7 @@ void zs_free(struct zs_pool *pool, unsigned long handle) if (fullness == ZS_INUSE_RATIO_0) free_zspage(pool, class, zspage); - spin_unlock(&class->lock); + size_class_unlock(class); cache_free_handle(pool, handle); } EXPORT_SYMBOL_GPL(zs_free); @@ -1846,7 +1852,7 @@ static int zs_page_migrate(struct page *newpage, struct page *page, /* * the class lock protects zpage alloc/free in the zspage. */ - spin_lock(&class->lock); + size_class_lock(class); /* the zspage_write_lock protects zpage access via zs_map_object */ zspage_write_lock(zspage); @@ -1878,7 +1884,7 @@ static int zs_page_migrate(struct page *newpage, struct page *page, * it's okay to release migration_lock. */ pool_write_unlock(pool); - spin_unlock(&class->lock); + size_class_unlock(class); zspage_write_unlock(zspage); zpdesc_get(newzpdesc); @@ -1922,10 +1928,10 @@ static void async_free_zspage(struct work_struct *work) if (class->index != i) continue; - spin_lock(&class->lock); + size_class_lock(class); list_splice_init(&class->fullness_list[ZS_INUSE_RATIO_0], &free_pages); - spin_unlock(&class->lock); + size_class_unlock(class); } list_for_each_entry_safe(zspage, tmp, &free_pages, list) { @@ -1933,10 +1939,10 @@ static void async_free_zspage(struct work_struct *work) lock_zspage(zspage); class = zspage_class(pool, zspage); - spin_lock(&class->lock); + size_class_lock(class); class_stat_sub(class, ZS_INUSE_RATIO_0, 1); __free_zspage(pool, class, zspage); - spin_unlock(&class->lock); + size_class_unlock(class); } }; @@ -2001,7 +2007,7 @@ static unsigned long __zs_compact(struct zs_pool *pool, * as well as zpage allocation/free */ pool_write_lock(pool); - spin_lock(&class->lock); + size_class_lock(class); while (zs_can_compact(class)) { int fg; @@ -2031,11 +2037,11 @@ static unsigned long __zs_compact(struct zs_pool *pool, putback_zspage(class, dst_zspage); dst_zspage = NULL; - spin_unlock(&class->lock); + size_class_unlock(class); pool_write_unlock(pool); cond_resched(); pool_write_lock(pool); - spin_lock(&class->lock); + size_class_lock(class); } } @@ -2045,7 +2051,7 @@ static unsigned long __zs_compact(struct zs_pool *pool, if (dst_zspage) putback_zspage(class, dst_zspage); - spin_unlock(&class->lock); + size_class_unlock(class); pool_write_unlock(pool); return pages_freed; @@ -2255,7 +2261,7 @@ struct zs_pool *zs_create_pool(const char *name) class->index = i; class->pages_per_zspage = pages_per_zspage; class->objs_per_zspage = objs_per_zspage; - spin_lock_init(&class->lock); + mutex_init(&class->lock); pool->size_class[i] = class; fullness = ZS_INUSE_RATIO_0; From patchwork Mon Jan 27 07:59:30 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13951057 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0E7BEC0218C for ; Mon, 27 Jan 2025 08:03:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 900B2280119; Mon, 27 Jan 2025 03:03:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8B11E2800E8; Mon, 27 Jan 2025 03:03:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 703C5280119; Mon, 27 Jan 2025 03:03:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 5057B2800E8 for ; Mon, 27 Jan 2025 03:03:36 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 01C15B486C for ; Mon, 27 Jan 2025 08:03:35 +0000 (UTC) X-FDA: 83052492432.27.B4966B4 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) by imf03.hostedemail.com (Postfix) with ESMTP id 23A6F20008 for ; Mon, 27 Jan 2025 08:03:33 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=UDdlosUV; spf=pass (imf03.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.171 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1737965014; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=fOh8kdIggNW/MgiQjrbJnv2UXDF8Y7xtTwQgOkh+AAY=; b=n/zVInfZNCQs7PMpAgXKmE46GQ/eoDApwdVxsbt+oS58sg7ZOTe0UG+R/1LHjwRiDYBjtZ VusOvttplxCFhSH0GeOuN3juf7Duv0OcEYKlj+CZ7HTxk+hSH6jU18Y9jyZopqwYqcu74M ka73eXW/n/FlKo/ePF6fHahhoTSwz6w= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=UDdlosUV; spf=pass (imf03.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.171 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1737965014; a=rsa-sha256; cv=none; b=hrawkEJXG5BifMXQ6p3Yp6oVg/+QTuJRQyTR+Td0Hn7+sj6yP8ScbFvq9BaIGsFZt3wwzM XomoLKeFzqu3sF05/dke2WhcgPx7mHmMLckCbjhCtJbHKcokFqQefrDz+ncAYEsMjI6RsR jvYRMXzDTPS69WpxMrhS/2sB9mKhnHw= Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-21631789fcdso67002515ad.1 for ; Mon, 27 Jan 2025 00:03:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1737965013; x=1738569813; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=fOh8kdIggNW/MgiQjrbJnv2UXDF8Y7xtTwQgOkh+AAY=; b=UDdlosUV5qctQ0zyWjRrRtRQ5O+DE9JcGjh/enHYbY8OJiYugBCzVGjDirj7s6kfyL aIsycqxYh2ViUqMoxsHaECjo7CaZ41MLjkWQgv5Fu9b16BVHMRXQMfwQjMJ7daKkKbjl X+xdXnTxHklO6y8OYYKArAzDzPGFRoimS9mKg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737965013; x=1738569813; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=fOh8kdIggNW/MgiQjrbJnv2UXDF8Y7xtTwQgOkh+AAY=; b=sXyo3NO8hT0xjEBA4SUkRcsp/gbFn3eEXG70vFKpw8onF7uXqDJUxcMTfsDnSBK3Uo Dl7nC/wQFNTC7X+/ITfgn4xjufEQWi6msNaLNOYfbW8FCiiIQac6/Hsb1skjYUfhr5sT OqkThVwWZNh1p0rKyiMfFeuZMktWihcAiz52bNKoKAOPi8AfSRGG5woWNkjnBkGceY+u JXryJ27WCu1HIginETTOUixXxKaEfRvw9BQ4nYioafh+oHwioXzcuCIsDAU2PcAw/5Sf eOyMy6tlBTBJk8QpShL2C7cx3lzcu/ONsbRrsZXmVMcMCLJUOfHe0XZvjANL0S5bMIkD nMBQ== X-Gm-Message-State: AOJu0YzHzktWf9xi9288Z1ojkPxFxCSQLJgnwAzruO6VVUH8bYlDVFz/ oQPtdk6R4dL6JjYVVKdfb5rDMh+sFMTTTijeCqt/0AQ/k8gQVmfBMz6P2HfH/A== X-Gm-Gg: ASbGnctcdSrYMR1kiFYDK3kN5K5JMEoSSC8X8nG1Nffkxugj31BUUjXnP3vdPkzBUC3 zNKZeU6Ia+3/vKiW1CcZysqtoU7JxwlGfXYwdZVj3jq336CTu2POzY/EfD8c28Dbbf80+aPptjS l/iS5hgPEkoK8hOvVyWJ8wfRyMydV0jO7NlY3zdZ/ku+bMczevbujCwxiXAemUWldwbu2yAKKBN YpOsy7qR/PxvKglnzwXCO+wBFD4C2Xbzm9QPSFpRCOdgQyctd7RBkAZHXn9cj9dBxm3YNIea5dK 6JBd7WI= X-Google-Smtp-Source: AGHT+IHCMZnq78KzilDYy3SoIuDu6zyyr9bdzoDlo65Vrz4W4RnlHVGP0xG9ma7YfvS5hlyVHvHmww== X-Received: by 2002:a05:6a00:ac4:b0:725:d9b6:3952 with SMTP id d2e1a72fcca58-72f7d1d9573mr28441245b3a.3.1737965012759; Mon, 27 Jan 2025 00:03:32 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:566d:6152:c049:8d3a]) by smtp.gmail.com with UTF8SMTPSA id d2e1a72fcca58-72f8a78dfa2sm6426964b3a.157.2025.01.27.00.03.30 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 27 Jan 2025 00:03:32 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim , Johannes Weiner , Yosry Ahmed , Nhat Pham Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [RFC PATCH 5/6] zsmalloc: introduce handle mapping API Date: Mon, 27 Jan 2025 16:59:30 +0900 Message-ID: <20250127080254.1302026-6-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog In-Reply-To: <20250127080254.1302026-1-senozhatsky@chromium.org> References: <20250127080254.1302026-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 23A6F20008 X-Stat-Signature: w9z5ojndep8wndjd86bqbx47fjt7ryfd X-Rspam-User: X-HE-Tag: 1737965013-952958 X-HE-Meta: U2FsdGVkX19C1lNoiEd8zxt5MJgE4eVPfwZfhdkVhudieaPKgFGbzaWTL7wiAhTnh/w9PPzfMN6S6CoNbvAU0AX7V8EPnOd3Hsi5SI9QOHYPF0TWavpEx2T/JUlSyEo1o5o4eHr9X6X83BRtxNaMHLEg6GqJ2nsUOQ/YvaRr/H2n19kRnUOEOYB55Rm4BZv/vb59XS3vAGl5hf6l2j4ji49rb77MCkBj3Ytk9X9/1j/n4Xgi4V+eSvA7DJHh6At0eDCQmyKR/A+zbEEtHBxfNmSH8TJY+VAYc0a1EBSI8vYWkKPO81vjfkRaKGLqNj/JPu+uWT3MelmsPNpM68LP+Ax0YTC+VpzDcj8F1giS9ox5W1n/blg1HUUGYpwIoKcUoOXHINQTVixossbzNcf8LFHJ0gHcOnYlbwgMQ4ExVlqzWSCv6zHXGBGZ9JBvytFbYrwgU2sYBgCjOLA4yySrLSIcWJyLm5dtajEjZ1/BHsdzOBoC+00ljWRhccSjt3pMQywyskx3NZbYjG1HQOc/BnaXsyHWSzjP/WhfDqIkMXW/1lf3UYl7273BgF4WSgkHdh/i27oczgfvuopz/WNl5DeOHRJ9MIn+3OEx2wbE7R2DdmiO7ngoYzcnObndjYLFza1b+xLL96iLu91sXoiRyZqicLFIs4ypylgPgxKyblXArv/OhXncYs9qZbovZvZa1HXpAuZgSHvxyQ5uzEB4JaQhL1zqM7kuLLBxaT/8I5lsLI3PrXKUOg0FDwzw5/bvD9ssV+vIR5np7Cf9+u0uw+KteakZVq1ma4yZPpkBKy1UXLeTV8agyCESjLLWGghka8SEvh9FCQiNqPaY2qTVbKqP7O4D0F9TjuOTFkf24/kKpP4esHU2Kdxfn/VoaupQy6hWnSth8+vhwhqbpJsXN5rgxEDkh898K6DAcXzh6vIxPg/Wzq0+KT0eihlB055ickHOALYli3iaXsCz3Fm NxaXCZ20 N36abyoB86GVkLpk/ax8EQg/okpuxTCuNk3JQZ1OcRWcThtlAKjVR4iWOMCy5AyE+2EWZ9EESsEvmhqstPSVJjbYH/lVMgDoETY1WHgxny4J0YCh80WacFuXzvT04hB2DvV1rL1jYqw+kPg+IoONn0kt37RWK6WfMe7FdvSuZ29JtZ9Q3lqMYVu0L7Moyn7b6Rj7wvB/dgNd5dxf/YT9wuUFEcHxwemt/MVxzxbBn6dCSpT6yprj/cd0nhKhS3rDE4cRyI9Ge28jXy3nVysUVhnd90PEiGToP1cQjs7SUBmDsMfAfcTMNYCvKlA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Introduce new API to map/unmap zsmalloc handle/object. The key difference is that this API does not impose atomicity restrictions on its users, unlike zs_map_object() which returns with page-faults and preemption disabled - handle mapping API does not need a per-CPU vm-area because the users are required to provide an aux buffer for objects that span several physical pages. Keep zs_map_object/zs_unmap_object for the time being, as there are still users of it, but eventually old API will be removed. Signed-off-by: Sergey Senozhatsky --- include/linux/zsmalloc.h | 29 ++++++++ mm/zsmalloc.c | 148 ++++++++++++++++++++++++++++----------- 2 files changed, 138 insertions(+), 39 deletions(-) diff --git a/include/linux/zsmalloc.h b/include/linux/zsmalloc.h index a48cd0ffe57d..72d84537dd38 100644 --- a/include/linux/zsmalloc.h +++ b/include/linux/zsmalloc.h @@ -58,4 +58,33 @@ unsigned long zs_compact(struct zs_pool *pool); unsigned int zs_lookup_class_index(struct zs_pool *pool, unsigned int size); void zs_pool_stats(struct zs_pool *pool, struct zs_pool_stats *stats); + +struct zs_handle_mapping { + unsigned long handle; + /* Points to start of the object data either within local_copy or + * within local_mapping. This is what callers should use to access + * or modify handle data. + */ + void *handle_mem; + + enum zs_mapmode mode; + union { + /* + * Handle object data copied, because it spans across several + * (non-contiguous) physical pages. This pointer should be + * set by the zs_map_handle() caller beforehand and should + * never be accessed directly. + */ + void *local_copy; + /* + * Handle object mapped directly. Should never be used + * directly. + */ + void *local_mapping; + }; +}; + +int zs_map_handle(struct zs_pool *pool, struct zs_handle_mapping *map); +void zs_unmap_handle(struct zs_pool *pool, struct zs_handle_mapping *map); + #endif diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index a5c1f9852072..281bba4a3277 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -1132,18 +1132,14 @@ static inline void __zs_cpu_down(struct mapping_area *area) area->vm_buf = NULL; } -static void *__zs_map_object(struct mapping_area *area, - struct zpdesc *zpdescs[2], int off, int size) +static void zs_obj_copyin(void *buf, struct zpdesc *zpdesc, int off, int size) { + struct zpdesc *zpdescs[2]; size_t sizes[2]; - char *buf = area->vm_buf; - - /* disable page faults to match kmap_local_page() return conditions */ - pagefault_disable(); - /* no read fastpath */ - if (area->vm_mm == ZS_MM_WO) - goto out; + zpdescs[0] = zpdesc; + zpdescs[1] = get_next_zpdesc(zpdesc); + BUG_ON(!zpdescs[1]); sizes[0] = PAGE_SIZE - off; sizes[1] = size - sizes[0]; @@ -1151,21 +1147,17 @@ static void *__zs_map_object(struct mapping_area *area, /* copy object to per-cpu buffer */ memcpy_from_page(buf, zpdesc_page(zpdescs[0]), off, sizes[0]); memcpy_from_page(buf + sizes[0], zpdesc_page(zpdescs[1]), 0, sizes[1]); -out: - return area->vm_buf; } -static void __zs_unmap_object(struct mapping_area *area, - struct zpdesc *zpdescs[2], int off, int size) +static void zs_obj_copyout(void *buf, struct zpdesc *zpdesc, int off, int size) { + struct zpdesc *zpdescs[2]; size_t sizes[2]; - char *buf; - /* no write fastpath */ - if (area->vm_mm == ZS_MM_RO) - goto out; + zpdescs[0] = zpdesc; + zpdescs[1] = get_next_zpdesc(zpdesc); + BUG_ON(!zpdescs[1]); - buf = area->vm_buf; buf = buf + ZS_HANDLE_SIZE; size -= ZS_HANDLE_SIZE; off += ZS_HANDLE_SIZE; @@ -1176,10 +1168,6 @@ static void __zs_unmap_object(struct mapping_area *area, /* copy per-cpu buffer to object */ memcpy_to_page(zpdesc_page(zpdescs[0]), off, buf, sizes[0]); memcpy_to_page(zpdesc_page(zpdescs[1]), 0, buf + sizes[0], sizes[1]); - -out: - /* enable page faults to match kunmap_local() return conditions */ - pagefault_enable(); } static int zs_cpu_prepare(unsigned int cpu) @@ -1260,6 +1248,8 @@ EXPORT_SYMBOL_GPL(zs_get_total_pages); * against nested mappings. * * This function returns with preemption and page faults disabled. + * + * NOTE: this function is deprecated and will be removed. */ void *zs_map_object(struct zs_pool *pool, unsigned long handle, enum zs_mapmode mm) @@ -1268,10 +1258,8 @@ void *zs_map_object(struct zs_pool *pool, unsigned long handle, struct zpdesc *zpdesc; unsigned long obj, off; unsigned int obj_idx; - struct size_class *class; struct mapping_area *area; - struct zpdesc *zpdescs[2]; void *ret; /* @@ -1309,12 +1297,14 @@ void *zs_map_object(struct zs_pool *pool, unsigned long handle, goto out; } - /* this object spans two pages */ - zpdescs[0] = zpdesc; - zpdescs[1] = get_next_zpdesc(zpdesc); - BUG_ON(!zpdescs[1]); + ret = area->vm_buf; + /* disable page faults to match kmap_local_page() return conditions */ + pagefault_disable(); + if (mm != ZS_MM_WO) { + /* this object spans two pages */ + zs_obj_copyin(area->vm_buf, zpdesc, off, class->size); + } - ret = __zs_map_object(area, zpdescs, off, class->size); out: if (likely(!ZsHugePage(zspage))) ret += ZS_HANDLE_SIZE; @@ -1323,13 +1313,13 @@ void *zs_map_object(struct zs_pool *pool, unsigned long handle, } EXPORT_SYMBOL_GPL(zs_map_object); +/* NOTE: this function is deprecated and will be removed. */ void zs_unmap_object(struct zs_pool *pool, unsigned long handle) { struct zspage *zspage; struct zpdesc *zpdesc; unsigned long obj, off; unsigned int obj_idx; - struct size_class *class; struct mapping_area *area; @@ -1340,23 +1330,103 @@ void zs_unmap_object(struct zs_pool *pool, unsigned long handle) off = offset_in_page(class->size * obj_idx); area = this_cpu_ptr(&zs_map_area); - if (off + class->size <= PAGE_SIZE) + if (off + class->size <= PAGE_SIZE) { kunmap_local(area->vm_addr); - else { - struct zpdesc *zpdescs[2]; + goto out; + } - zpdescs[0] = zpdesc; - zpdescs[1] = get_next_zpdesc(zpdesc); - BUG_ON(!zpdescs[1]); + if (area->vm_mm != ZS_MM_RO) + zs_obj_copyout(area->vm_buf, zpdesc, off, class->size); + /* enable page faults to match kunmap_local() return conditions */ + pagefault_enable(); - __zs_unmap_object(area, zpdescs, off, class->size); - } +out: local_unlock(&zs_map_area.lock); - zspage_read_unlock(zspage); } EXPORT_SYMBOL_GPL(zs_unmap_object); +void zs_unmap_handle(struct zs_pool *pool, struct zs_handle_mapping *map) +{ + struct zspage *zspage; + struct zpdesc *zpdesc; + unsigned long obj, off; + unsigned int obj_idx; + struct size_class *class; + + obj = handle_to_obj(map->handle); + obj_to_location(obj, &zpdesc, &obj_idx); + zspage = get_zspage(zpdesc); + class = zspage_class(pool, zspage); + off = offset_in_page(class->size * obj_idx); + + if (off + class->size <= PAGE_SIZE) { + kunmap_local(map->local_mapping); + goto out; + } + + if (map->mode != ZS_MM_RO) + zs_obj_copyout(map->local_copy, zpdesc, off, class->size); + +out: + zspage_read_unlock(zspage); +} +EXPORT_SYMBOL_GPL(zs_unmap_handle); + +int zs_map_handle(struct zs_pool *pool, struct zs_handle_mapping *map) +{ + struct zspage *zspage; + struct zpdesc *zpdesc; + unsigned long obj, off; + unsigned int obj_idx; + struct size_class *class; + + WARN_ON(in_interrupt()); + + /* It guarantees it can get zspage from handle safely */ + pool_read_lock(pool); + obj = handle_to_obj(map->handle); + obj_to_location(obj, &zpdesc, &obj_idx); + zspage = get_zspage(zpdesc); + + /* + * migration cannot move any zpages in this zspage. Here, class->lock + * is too heavy since callers would take some time until they calls + * zs_unmap_object API so delegate the locking from class to zspage + * which is smaller granularity. + */ + zspage_read_lock(zspage); + pool_read_unlock(pool); + + class = zspage_class(pool, zspage); + off = offset_in_page(class->size * obj_idx); + + if (off + class->size <= PAGE_SIZE) { + /* this object is contained entirely within a page */ + map->local_mapping = kmap_local_zpdesc(zpdesc); + map->handle_mem = map->local_mapping + off; + goto out; + } + + if (WARN_ON_ONCE(!map->local_copy)) { + zspage_read_unlock(zspage); + return -EINVAL; + } + + map->handle_mem = map->local_copy; + if (map->mode != ZS_MM_WO) { + /* this object spans two pages */ + zs_obj_copyin(map->local_copy, zpdesc, off, class->size); + } + +out: + if (likely(!ZsHugePage(zspage))) + map->handle_mem += ZS_HANDLE_SIZE; + + return 0; +} +EXPORT_SYMBOL_GPL(zs_map_handle); + /** * zs_huge_class_size() - Returns the size (in bytes) of the first huge * zsmalloc &size_class. From patchwork Mon Jan 27 07:59:31 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Sergey Senozhatsky X-Patchwork-Id: 13951058 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2799EC0218F for ; Mon, 27 Jan 2025 08:03:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AFC16280137; Mon, 27 Jan 2025 03:03:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AAAFB2800E8; Mon, 27 Jan 2025 03:03:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 90076280137; Mon, 27 Jan 2025 03:03:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id B322E2800E8 for ; Mon, 27 Jan 2025 03:03:41 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 6D022B48E8 for ; Mon, 27 Jan 2025 08:03:41 +0000 (UTC) X-FDA: 83052492642.11.BB36329 Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) by imf23.hostedemail.com (Postfix) with ESMTP id 93049140009 for ; Mon, 27 Jan 2025 08:03:39 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b="mv/B1sg0"; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf23.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.176 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1737965019; a=rsa-sha256; cv=none; b=ViUeoJaVn1LO1ak02Lk4w6Ia2Qz5Aakoc9oXg+AZMOHA/Jl4Z3wNG7qNnWtsjkMCdDELdd 3HGzLtk4fX7Bz3PSssC4aQg9Kv5VvZ4XdKu1yUJpkwNkVY2LyZO8yrlsFMoDn5H0oVhJse 04EINJdVkH7GDcy6vxo25IImRmmUDJE= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b="mv/B1sg0"; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf23.hostedemail.com: domain of senozhatsky@chromium.org designates 209.85.214.176 as permitted sender) smtp.mailfrom=senozhatsky@chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1737965019; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=651sDMCEJHysCKLNgE9iOdg6zJ4Jwg3Nj3A7ATDjnVc=; b=EHA9oHKqbgxRZUs7hF7VI+OVwnYfI0ZK1biKLyGzlVGusF6qLnbZLYpZ3pQVvtuF1sD8cn Md7p9S4e2Po6VZN4INcHQr/68v5S0+d0/Wji9cjJLMTF14TIpsHOq0/mP5QEJcpD8r8XeP BHPJn5FU7/jiKHclaRXOW5ktS1Xc3Vs= Received: by mail-pl1-f176.google.com with SMTP id d9443c01a7336-2163dc5155fso70549235ad.0 for ; Mon, 27 Jan 2025 00:03:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; t=1737965018; x=1738569818; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=651sDMCEJHysCKLNgE9iOdg6zJ4Jwg3Nj3A7ATDjnVc=; b=mv/B1sg018n/Rv28BbU3vVsgJPB0/DpJ/IZC58XoUfUjCgYWgcH2ldXCMnNwy+FK6C 0+oSE4cyBENce2clTFmIEaUIDSfSOjR7lHSfXE/PBYWEEt5akS4E9ZmoJFhJ5qOvVlAY cNonW+Vpdo0wCiiVnMIy4KmBU+lG7MtP4yYOQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737965018; x=1738569818; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=651sDMCEJHysCKLNgE9iOdg6zJ4Jwg3Nj3A7ATDjnVc=; b=GShOD9uqua28WuvWQKEVjrduUiGpPVTGhZ6D6xsEYTvzeygITn73YLXmA5uCUJ0Cd4 vSr+Qqz4SOKWQsgSLogxhB8vBsrllfslsAFO5fn7BG6HhZG7ul5NiDRsbu3axO2/podp QTgvN8zxD74bbCPrixEKdTCy2/YNI00uUSuLyw/S39qtCbBK4HB3M/M/RC06rXMe2y0v AXJBRhwEuOwfaqV1XH2eNoMcAXNtn689r5z6aAQDZkLMjGbxIJmNFjevfTl51yxsMvRH Fl3PGmZtOD2XjqbF2Ly7Vs/d5uuSpdNFYZbXD978By21cqbFKNK7iDm/rca0GjxTYYnH 9QyQ== X-Gm-Message-State: AOJu0YyUwklWMReQtIZHN/AKrtnhEvunFVU/U7HF/DPD13h3/iHS/GD1 U/8u5zm8Gb9O9rGn66elAp6fP3oYMD4ncNXggafXfx/hkwbhOiCH4lmebiJC38ksr7oUWBFqWm8 = X-Gm-Gg: ASbGncv1rWiHSPx/f7UX2dCoWLrv32hEqAwHnZKBrFtsMTYwIZY5v1AIj7dn4FzpedD zdFB/gZfzChdcMiKVI44jHliICbwtDEeh7aSvY2Hq6NNH8xHhsTHh7QKuvAAE9HZd+VCPEhkh1V kk1lLkr70lBY8IVetooDAqZkNnvm511yfzjRGurrauMPDNyNwYm0/Dh9LAeSViqAf9yAZRKnMzk OKcsXqTZmq4IXEkWNdOR7Nl3wZ2x/krGtGlsRgeknkaAQpFx4SQuAMZtmER6JN+h/QaZmpFg7Sy 7SVDiDY= X-Google-Smtp-Source: AGHT+IF4OL0p2X71WwYse+YFBRvC81hL1zRDbOr33Q/MhbIwiQRVy63K3CZ31U6CYc/ujJCK075RbA== X-Received: by 2002:a17:902:db0a:b0:216:7761:cc49 with SMTP id d9443c01a7336-21c355f70c7mr542469735ad.47.1737965018362; Mon, 27 Jan 2025 00:03:38 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:566d:6152:c049:8d3a]) by smtp.gmail.com with UTF8SMTPSA id 41be03b00d2f7-ac496bbce61sm5749314a12.68.2025.01.27.00.03.36 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 27 Jan 2025 00:03:38 -0800 (PST) From: Sergey Senozhatsky To: Andrew Morton , Minchan Kim , Johannes Weiner , Yosry Ahmed , Nhat Pham Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Sergey Senozhatsky Subject: [RFC PATCH 6/6] zram: switch over to zshandle mapping API Date: Mon, 27 Jan 2025 16:59:31 +0900 Message-ID: <20250127080254.1302026-7-senozhatsky@chromium.org> X-Mailer: git-send-email 2.48.1.262.g85cc9f2d1e-goog In-Reply-To: <20250127080254.1302026-1-senozhatsky@chromium.org> References: <20250127080254.1302026-1-senozhatsky@chromium.org> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 93049140009 X-Stat-Signature: rpph1eph759wc7jtpxutxprkg419ezwd X-Rspam-User: X-HE-Tag: 1737965019-886251 X-HE-Meta: U2FsdGVkX19CWlRUP81nu4L5Z/Fn0LqliMgqRYVmUiK9cQdL+d3MoOlb4sDZv+0aV+zqx0kdsE74Xb2BZhyjx1pkF12yohPWORPtKsmUA1taGBxe0i/C8a/TDMvTkEB22fTOX+8ZoxUQcGeNICAibuUTxhFQEgfAlHmdD2BK2WnbLU4z3jvXjiewA8/8BxuVS7rfIXAaIyqHGhTuBAkFz9oww0m2bVrOPqt410WrXK5FIXQgR6vMJoRzXCDo0603NI3OiSQ03K+UM3+afZx5ctE8xLD3qRa2iM3nNuKtWGA8pjtJXzhRynw3QhNDQrU6e4zShmFJQF/kJhIEpiFiMNhXGsp4gSRJFyseQvOR4LweTR774jSPL+/XBHQ16mMHjo29JGM6p1k3MlYt9wlNGc7Y/ekXT+OUxxu+0JxAYx38PNY0sd1KXUdXlm+5Ex0bBzdwYdMuJ3thkd5vAS3nQ7q3cqrVVRFfGWxG7DIWozUfEzRWQbQrxgGfvFqu6B+bXVqy8yzm8YEYXFgU0vpn/WdotOKGBEtjADfLZlNz6HuSiUPdNAznVMREJWBYKZNCmjQt0C3ND556t/jZkZXBwZ3ZjgK+ZtRwsJ3I1rcYirDybcLvJh6V/EE0ve2ozy3GypA39KRDmCuKCVUsUDnHmYefWFqfdrw5OFjnMWZzEzm6MtqcO030zNeHTBL/zz+HHNO3EhQAN4NsOPdEfxedrw1JlJ/lziWe82fve/jWLAqGWxJWTRO7YGbuhEbD2l/DAoqzZRRQJ0FJRvzees9/MyAfq+LVCRuzfXGgnHSES7+R13aCp5QRBUkoAMlzJ0MH1zE837jxwqYCJkj2+9sqUmdtOTMktOzc7Ur9TfyL848do//dsGPWBqUBTRu0bgD+qh98C5OoFYQSJN/3uwsNMELtrktxzG5iG/uld4MWGM1k4bm62wDC66WC2NclFMytAoqrQNQ5gqa/LK4kh08 yI9xkWWZ iKkdZ+JQzYaB6b4Ml7T8tVVUtyTg4qJ3xN4bLzvr85onZzJUErv8/8/bZ3sRsv4UzEWRFvi9gODsd6u6jXQPo9WKQ+vfiBVb080fYC/sC1TFhWXTtRaBD7Hcv6+4kWOUkqBW6NXlY049POY0nGArvHmvTTrs3OT1m0NjTVDWdazJC4CYSEm+xNiCsp8kiHubzZj0pbMyDgL9bKojcaUjxAWthSUxAuD4zCfH7EoH5aC3kbtzm0xuKjlbKWMSmieKFk69huBrR0X7lj1r8bjaG9R1CroKpXRW8WIF8Pw2JBKuy3pH/s4j8mowrSw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Use new zsmalloc handle mapping API so now zram read() becomes preemptible. Signed-off-by: Sergey Senozhatsky --- drivers/block/zram/zcomp.c | 4 +- drivers/block/zram/zcomp.h | 2 + drivers/block/zram/zram_drv.c | 103 ++++++++++++++++++---------------- 3 files changed, 61 insertions(+), 48 deletions(-) diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c index efd5919808d9..9b373ab1ee0b 100644 --- a/drivers/block/zram/zcomp.c +++ b/drivers/block/zram/zcomp.c @@ -45,6 +45,7 @@ static const struct zcomp_ops *backends[] = { static void zcomp_strm_free(struct zcomp *comp, struct zcomp_strm *strm) { comp->ops->destroy_ctx(&strm->ctx); + vfree(strm->handle_mem_copy); vfree(strm->buffer); kfree(strm); } @@ -66,12 +67,13 @@ static struct zcomp_strm *zcomp_strm_alloc(struct zcomp *comp) return NULL; } + strm->handle_mem_copy = vzalloc(PAGE_SIZE); /* * allocate 2 pages. 1 for compressed data, plus 1 extra in case if * compressed data is larger than the original one. */ strm->buffer = vzalloc(2 * PAGE_SIZE); - if (!strm->buffer) { + if (!strm->buffer || !strm->handle_mem_copy) { zcomp_strm_free(comp, strm); return NULL; } diff --git a/drivers/block/zram/zcomp.h b/drivers/block/zram/zcomp.h index 62330829db3f..f003f09820a5 100644 --- a/drivers/block/zram/zcomp.h +++ b/drivers/block/zram/zcomp.h @@ -34,6 +34,8 @@ struct zcomp_strm { struct list_head entry; /* compression buffer */ void *buffer; + /* handle object memory copy */ + void *handle_mem_copy; struct zcomp_ctx ctx; }; diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 9c72beb86ab0..120055b11520 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -1558,37 +1558,43 @@ static int read_same_filled_page(struct zram *zram, struct page *page, static int read_incompressible_page(struct zram *zram, struct page *page, u32 index) { - unsigned long handle; - void *src, *dst; + struct zs_handle_mapping hm; + void *dst; - handle = zram_get_handle(zram, index); - src = zs_map_object(zram->mem_pool, handle, ZS_MM_RO); + hm.handle = zram_get_handle(zram, index); + hm.mode = ZS_MM_RO; + + zs_map_handle(zram->mem_pool, &hm); dst = kmap_local_page(page); - copy_page(dst, src); + copy_page(dst, hm.handle_mem); kunmap_local(dst); - zs_unmap_object(zram->mem_pool, handle); + zs_unmap_handle(zram->mem_pool, &hm); return 0; } static int read_compressed_page(struct zram *zram, struct page *page, u32 index) { + struct zs_handle_mapping hm; struct zcomp_strm *zstrm; - unsigned long handle; unsigned int size; - void *src, *dst; + void *dst; int ret, prio; - handle = zram_get_handle(zram, index); size = zram_get_obj_size(zram, index); prio = zram_get_priority(zram, index); zstrm = zcomp_stream_get(zram->comps[prio]); - src = zs_map_object(zram->mem_pool, handle, ZS_MM_RO); + hm.handle = zram_get_handle(zram, index); + hm.mode = ZS_MM_RO; + hm.local_copy = zstrm->handle_mem_copy; + + zs_map_handle(zram->mem_pool, &hm); dst = kmap_local_page(page); - ret = zcomp_decompress(zram->comps[prio], zstrm, src, size, dst); + ret = zcomp_decompress(zram->comps[prio], zstrm, + hm.handle_mem, size, dst); kunmap_local(dst); - zs_unmap_object(zram->mem_pool, handle); + zs_unmap_handle(zram->mem_pool, &hm); zcomp_stream_put(zram->comps[prio], zstrm); return ret; @@ -1683,33 +1689,34 @@ static int write_same_filled_page(struct zram *zram, unsigned long fill, static int write_incompressible_page(struct zram *zram, struct page *page, u32 index) { - unsigned long handle; - void *src, *dst; + struct zs_handle_mapping hm; + void *src; /* * This function is called from preemptible context so we don't need * to do optimistic and fallback to pessimistic handle allocation, * like we do for compressible pages. */ - handle = zs_malloc(zram->mem_pool, PAGE_SIZE, - GFP_NOIO | __GFP_HIGHMEM | __GFP_MOVABLE); - if (IS_ERR_VALUE(handle)) - return PTR_ERR((void *)handle); + hm.handle = zs_malloc(zram->mem_pool, PAGE_SIZE, + GFP_NOIO | __GFP_HIGHMEM | __GFP_MOVABLE); + if (IS_ERR_VALUE(hm.handle)) + return PTR_ERR((void *)hm.handle); if (!zram_can_store_page(zram)) { - zs_free(zram->mem_pool, handle); + zs_free(zram->mem_pool, hm.handle); return -ENOMEM; } - dst = zs_map_object(zram->mem_pool, handle, ZS_MM_WO); + hm.mode = ZS_MM_WO; + zs_map_handle(zram->mem_pool, &hm); src = kmap_local_page(page); - memcpy(dst, src, PAGE_SIZE); + memcpy(hm.handle_mem, src, PAGE_SIZE); kunmap_local(src); - zs_unmap_object(zram->mem_pool, handle); + zs_unmap_handle(zram->mem_pool, &hm); zram_slot_write_lock(zram, index); zram_set_flag(zram, index, ZRAM_HUGE); - zram_set_handle(zram, index, handle); + zram_set_handle(zram, index, hm.handle); zram_set_obj_size(zram, index, PAGE_SIZE); zram_slot_write_unlock(zram, index); @@ -1724,9 +1731,9 @@ static int write_incompressible_page(struct zram *zram, struct page *page, static int zram_write_page(struct zram *zram, struct page *page, u32 index) { int ret = 0; - unsigned long handle; + struct zs_handle_mapping hm; unsigned int comp_len; - void *dst, *mem; + void *mem; struct zcomp_strm *zstrm; unsigned long element; bool same_filled; @@ -1758,25 +1765,26 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index) return write_incompressible_page(zram, page, index); } - handle = zs_malloc(zram->mem_pool, comp_len, - GFP_NOIO | __GFP_HIGHMEM | __GFP_MOVABLE); - if (IS_ERR_VALUE(handle)) - return PTR_ERR((void *)handle); + hm.handle = zs_malloc(zram->mem_pool, comp_len, + GFP_NOIO | __GFP_HIGHMEM | __GFP_MOVABLE); + if (IS_ERR_VALUE(hm.handle)) + return PTR_ERR((void *)hm.handle); if (!zram_can_store_page(zram)) { zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP], zstrm); - zs_free(zram->mem_pool, handle); + zs_free(zram->mem_pool, hm.handle); return -ENOMEM; } - dst = zs_map_object(zram->mem_pool, handle, ZS_MM_WO); - - memcpy(dst, zstrm->buffer, comp_len); + hm.mode = ZS_MM_WO; + hm.local_copy = zstrm->handle_mem_copy; + zs_map_handle(zram->mem_pool, &hm); + memcpy(hm.handle_mem, zstrm->buffer, comp_len); + zs_unmap_handle(zram->mem_pool, &hm); zcomp_stream_put(zram->comps[ZRAM_PRIMARY_COMP], zstrm); - zs_unmap_object(zram->mem_pool, handle); zram_slot_write_lock(zram, index); - zram_set_handle(zram, index, handle); + zram_set_handle(zram, index, hm.handle); zram_set_obj_size(zram, index, comp_len); zram_slot_write_unlock(zram, index); @@ -1875,14 +1883,14 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, u32 prio_max) { struct zcomp_strm *zstrm = NULL; + struct zs_handle_mapping hm; unsigned long handle_old; - unsigned long handle_new; unsigned int comp_len_old; unsigned int comp_len_new; unsigned int class_index_old; unsigned int class_index_new; u32 num_recomps = 0; - void *src, *dst; + void *src; int ret; handle_old = zram_get_handle(zram, index); @@ -2000,34 +2008,35 @@ static int recompress_slot(struct zram *zram, u32 index, struct page *page, /* zsmalloc handle allocation can schedule, unlock slot's bucket */ zram_slot_write_unlock(zram, index); - handle_new = zs_malloc(zram->mem_pool, comp_len_new, - GFP_NOIO | __GFP_HIGHMEM | __GFP_MOVABLE); + hm.handle = zs_malloc(zram->mem_pool, comp_len_new, + GFP_NOIO | __GFP_HIGHMEM | __GFP_MOVABLE); zram_slot_write_lock(zram, index); /* * If we couldn't allocate memory for recompressed object then bail * out and simply keep the old (existing) object in mempool. */ - if (IS_ERR_VALUE(handle_new)) { + if (IS_ERR_VALUE(hm.handle)) { zcomp_stream_put(zram->comps[prio], zstrm); - return PTR_ERR((void *)handle_new); + return PTR_ERR((void *)hm.handle); } /* Slot has been modified concurrently */ if (!zram_test_flag(zram, index, ZRAM_PP_SLOT)) { zcomp_stream_put(zram->comps[prio], zstrm); - zs_free(zram->mem_pool, handle_new); + zs_free(zram->mem_pool, hm.handle); return 0; } - dst = zs_map_object(zram->mem_pool, handle_new, ZS_MM_WO); - memcpy(dst, zstrm->buffer, comp_len_new); + hm.mode = ZS_MM_WO; + hm.local_copy = zstrm->handle_mem_copy; + zs_map_handle(zram->mem_pool, &hm); + memcpy(hm.handle_mem, zstrm->buffer, comp_len_new); + zs_unmap_handle(zram->mem_pool, &hm); zcomp_stream_put(zram->comps[prio], zstrm); - zs_unmap_object(zram->mem_pool, handle_new); - zram_free_page(zram, index); - zram_set_handle(zram, index, handle_new); + zram_set_handle(zram, index, hm.handle); zram_set_obj_size(zram, index, comp_len_new); zram_set_priority(zram, index, prio);