From patchwork Sat Nov 19 00:15:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nhat Pham X-Patchwork-Id: 13049322 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3E432C4332F for ; Sat, 19 Nov 2022 00:15:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CA1CD6B0072; Fri, 18 Nov 2022 19:15:40 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B8E218E0001; Fri, 18 Nov 2022 19:15:40 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9E28E6B007E; Fri, 18 Nov 2022 19:15:40 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 807706B0072 for ; Fri, 18 Nov 2022 19:15:40 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 59BBA80377 for ; Sat, 19 Nov 2022 00:15:40 +0000 (UTC) X-FDA: 80148273240.15.4EE16B3 Received: from mail-pj1-f42.google.com (mail-pj1-f42.google.com [209.85.216.42]) by imf10.hostedemail.com (Postfix) with ESMTP id 03472C0004 for ; Sat, 19 Nov 2022 00:15:39 +0000 (UTC) Received: by mail-pj1-f42.google.com with SMTP id y14-20020a17090a2b4e00b002189a1b84d4so70916pjc.2 for ; Fri, 18 Nov 2022 16:15:39 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=TdXYXXujE5c6Z8uTdZ5KokARMRQa3FrJU+bkNzs2RLY=; b=NHkIkh8RaR25rj7+Wlw0rbe1vzjooV8Fsu/f07L3zCG5lUFzcdTGGBtND54rMrvFn2 1nztmuMnmKeDkLPZsShRcJCn8KfZDSS2xoMxkJ+8eWbfouubZv8PETaQAHxCEEQyuRcP 7dtpmnLSJFtricXu29dRABoMD0uTm0Xv+9F3R8Z0zOZfzFHEh/jZfd89vph1ur+jnvoi qe4O0YDtzONH4Dze1Yr/u8DHfwdnbozJEOewCLH0RyGcNrdx/Q/rLMIXB5yDuzL+iHp+ RKiSsuMYn6PCIoOSsyr2vmZRFTqbXCde8fgnZhLJQEkH9ydslMP7E4Ww+Oh0mh53K7e2 lK5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=TdXYXXujE5c6Z8uTdZ5KokARMRQa3FrJU+bkNzs2RLY=; b=1ffGgIPgBLNKyjaG8y1Jy/uArP79mZEUBSs5lhG26z7dSKK8RJg85NkT3uKUBUafdC gnlYqxd4oOHiR7WCDs1G8ulZIExb8hVcCj56rOQ1BEPmeQ4XpGApTtxiFkuBxTVtb1Gm rypzYNWvC98NKcQ0iOhOMe+mUlWjWPJlhjryIGX3Hdff4YHURPaIq9Uz9lPxCcSb5DYR lzh9KFw1/GE3RKyLJVPWoW58l7d9tKM9fOI10TzyIoHB+2VheEYeuR1gFc/codfZEHoy U3IQmGlQ3vzsKfyY21OhenKcdJdwzjwCMdEod2vd8LVO7lOi1RcPz6fIBKV8hDs/ebjq PH2w== X-Gm-Message-State: ANoB5pnFhmZcQ+Cw7ZHIo+ht+Uj7r2Lwj7sDH3uvMT71QpL3tJIxeVbP NI1RpB6hD7hdiJkyYF6NMEk= X-Google-Smtp-Source: AA0mqf7QZyc8yIJTE5WAZ0IZUUaf0nqx+pCwp3EewXFYXciM0TOgmftRCHrHwYQYKAVKePw7j1Pq+w== X-Received: by 2002:a17:902:9a03:b0:186:9f20:e7e2 with SMTP id v3-20020a1709029a0300b001869f20e7e2mr1578912plp.174.1668816939044; Fri, 18 Nov 2022 16:15:39 -0800 (PST) Received: from localhost (fwdproxy-prn-010.fbsv.net. [2a03:2880:ff:a::face:b00c]) by smtp.gmail.com with ESMTPSA id x22-20020a170902821600b0017f48a9e2d6sm4202332pln.292.2022.11.18.16.15.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 18 Nov 2022 16:15:38 -0800 (PST) From: Nhat Pham To: akpm@linux-foundation.org Cc: hannes@cmpxchg.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, minchan@kernel.org, ngupta@vflare.org, senozhatsky@chromium.org, sjenning@redhat.com, ddstreet@ieee.org, vitaly.wool@konsulko.com Subject: [PATCH v6 1/6] zswap: fix writeback lock ordering for zsmalloc Date: Fri, 18 Nov 2022 16:15:31 -0800 Message-Id: <20221119001536.2086599-2-nphamcs@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221119001536.2086599-1-nphamcs@gmail.com> References: <20221119001536.2086599-1-nphamcs@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1668816940; a=rsa-sha256; cv=none; b=fJdxSipM/WkDGICA7fjN7fM0HL/BAdn7uvBiomLVKpMuaLZAFx2v+hBUsBR34sJXRnE4dK yayux/6KeSGyrSH0VAnJ1eguf9B6OMl1vfCYQcBpG87TCEcq5TsDLUzrj6aVqG6a8eEBiz KqsJM9Ewc5e0EtCWhqRBPGDWP0e+gNQ= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=NHkIkh8R; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf10.hostedemail.com: domain of nphamcs@gmail.com designates 209.85.216.42 as permitted sender) smtp.mailfrom=nphamcs@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1668816940; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=TdXYXXujE5c6Z8uTdZ5KokARMRQa3FrJU+bkNzs2RLY=; b=gmWBy44+ZZYiWFupBsIhXVA1+ipk5ZeF8iIpJnkbHqF73G2t2eVS0zPmyXlJY1sUznH1+n JVPgWoiaNlbWWbWdd+VtXGl2ioIeQ0JZAur0vhgnitGZWjyXzgcwdc3IapPDRVEKnraiyB jlClrnc8z64lj+4U7Up7+E0JOiiimTo= Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=NHkIkh8R; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf10.hostedemail.com: domain of nphamcs@gmail.com designates 209.85.216.42 as permitted sender) smtp.mailfrom=nphamcs@gmail.com X-Rspam-User: X-Stat-Signature: 5ng7mzrtw1taxuzkzqis9cz8cugg6tey X-Rspamd-Queue-Id: 03472C0004 X-Rspamd-Server: rspam11 X-HE-Tag: 1668816939-62573 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Johannes Weiner zswap's customary lock order is tree->lock before pool->lock, because the tree->lock protects the entries' refcount, and the free callbacks in the backends acquire their respective pool locks to dispatch the backing object. zsmalloc's map callback takes the pool lock, so zswap must not grab the tree->lock while a handle is mapped. This currently only happens during writeback, which isn't implemented for zsmalloc. In preparation for it, move the tree->lock section out of the mapped entry section Signed-off-by: Johannes Weiner Signed-off-by: Nhat Pham --- mm/zswap.c | 37 ++++++++++++++++++++----------------- 1 file changed, 20 insertions(+), 17 deletions(-) -- 2.30.2 diff --git a/mm/zswap.c b/mm/zswap.c index 2d48fd59cc7a..2d69c1d678fe 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -958,7 +958,7 @@ static int zswap_writeback_entry(struct zpool *pool, unsigned long handle) }; if (!zpool_can_sleep_mapped(pool)) { - tmp = kmalloc(PAGE_SIZE, GFP_ATOMIC); + tmp = kmalloc(PAGE_SIZE, GFP_KERNEL); if (!tmp) return -ENOMEM; } @@ -968,6 +968,7 @@ static int zswap_writeback_entry(struct zpool *pool, unsigned long handle) swpentry = zhdr->swpentry; /* here */ tree = zswap_trees[swp_type(swpentry)]; offset = swp_offset(swpentry); + zpool_unmap_handle(pool, handle); /* find and ref zswap entry */ spin_lock(&tree->lock); @@ -975,20 +976,12 @@ static int zswap_writeback_entry(struct zpool *pool, unsigned long handle) if (!entry) { /* entry was invalidated */ spin_unlock(&tree->lock); - zpool_unmap_handle(pool, handle); kfree(tmp); return 0; } spin_unlock(&tree->lock); BUG_ON(offset != entry->offset); - src = (u8 *)zhdr + sizeof(struct zswap_header); - if (!zpool_can_sleep_mapped(pool)) { - memcpy(tmp, src, entry->length); - src = tmp; - zpool_unmap_handle(pool, handle); - } - /* try to allocate swap cache page */ switch (zswap_get_swap_cache_page(swpentry, &page)) { case ZSWAP_SWAPCACHE_FAIL: /* no memory or invalidate happened */ @@ -1006,6 +999,14 @@ static int zswap_writeback_entry(struct zpool *pool, unsigned long handle) acomp_ctx = raw_cpu_ptr(entry->pool->acomp_ctx); dlen = PAGE_SIZE; + zhdr = zpool_map_handle(pool, handle, ZPOOL_MM_RO); + src = (u8 *)zhdr + sizeof(struct zswap_header); + if (!zpool_can_sleep_mapped(pool)) { + memcpy(tmp, src, entry->length); + src = tmp; + zpool_unmap_handle(pool, handle); + } + mutex_lock(acomp_ctx->mutex); sg_init_one(&input, src, entry->length); sg_init_table(&output, 1); @@ -1015,6 +1016,11 @@ static int zswap_writeback_entry(struct zpool *pool, unsigned long handle) dlen = acomp_ctx->req->dlen; mutex_unlock(acomp_ctx->mutex); + if (!zpool_can_sleep_mapped(pool)) + kfree(tmp); + else + zpool_unmap_handle(pool, handle); + BUG_ON(ret); BUG_ON(dlen != PAGE_SIZE); @@ -1045,7 +1051,11 @@ static int zswap_writeback_entry(struct zpool *pool, unsigned long handle) zswap_entry_put(tree, entry); spin_unlock(&tree->lock); - goto end; + return ret; + +fail: + if (!zpool_can_sleep_mapped(pool)) + kfree(tmp); /* * if we get here due to ZSWAP_SWAPCACHE_EXIST @@ -1054,17 +1064,10 @@ static int zswap_writeback_entry(struct zpool *pool, unsigned long handle) * if we free the entry in the following put * it is also okay to return !0 */ -fail: spin_lock(&tree->lock); zswap_entry_put(tree, entry); spin_unlock(&tree->lock); -end: - if (zpool_can_sleep_mapped(pool)) - zpool_unmap_handle(pool, handle); - else - kfree(tmp); - return ret; } From patchwork Sat Nov 19 00:15:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nhat Pham X-Patchwork-Id: 13049323 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B04A3C43219 for ; Sat, 19 Nov 2022 00:15:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4B3B78E0001; Fri, 18 Nov 2022 19:15:42 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 416716B007E; Fri, 18 Nov 2022 19:15:42 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2400F8E0001; Fri, 18 Nov 2022 19:15:42 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 0CCFA6B007D for ; Fri, 18 Nov 2022 19:15:42 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id D609E120538 for ; Sat, 19 Nov 2022 00:15:41 +0000 (UTC) X-FDA: 80148273282.27.2A28159 Received: from mail-pg1-f173.google.com (mail-pg1-f173.google.com [209.85.215.173]) by imf13.hostedemail.com (Postfix) with ESMTP id 7E0102000A for ; Sat, 19 Nov 2022 00:15:41 +0000 (UTC) Received: by mail-pg1-f173.google.com with SMTP id v3so6339416pgh.4 for ; Fri, 18 Nov 2022 16:15:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=HZvzcMHAUb+r8vph2vKfcXCw1G+EouOeN0ss2yr8n7g=; b=hM3jHRqvmUOS23MLnToWW+gukgJ5URhKESdrNtGSY70GCQExpAwFM4v8mLrYILYaSZ o7BlEmhMWruCgk2DNQEhjWp6+CFLP4WnY4FSnnCDBXkEgUoBQP+CRxtiApMY0Aud0C2t EOAfqJTL/pd8k9qLlLGBPfkhUD3vI5oOREPErhNl0UTQw8UvuN6mF3hAo4ZUPUI7hNYm 48WftE7CwCRH0WCskYNEGYJBJPbP7yaZZcErJTzxF4cwXAbqBCC2WNNAA6HTXenZwdbs 9+gdXyCK1H7c40kVRKKR+p3ZqeaaIV+1WGtneXuim+vXKl900/Lh2B7/lEu4RH8ME2um zOgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HZvzcMHAUb+r8vph2vKfcXCw1G+EouOeN0ss2yr8n7g=; b=MPv1LQ71yaShtZehj/sqedTwW0GcjSVkPxZU6flDsScgkvC99S16qrNuFQT0Nel+2k jAzo0tCfYA1lfi3mSyhwGM4pIspeMnprPLb1xeG5O0GTmSf9kspMdQgZ1YOJDGIpiVhn e5/YfkwpQYdMm9Bo34dioBMSKzKIMEFUMlGZfVipcbKL77+4FdXq4dJOrcwgN1hDGtBd YGYhNHQDkjaJcyKEPbj4FQoMOMgD7Q1hAk6iq7UFvB6MJvmENFoJTgZAuZXLRNFAfiyC 0Sut1iwZDZaYx3oko/trhvfweyOrq+I2W3h5kTe1mQnMzP7y6FvShQeaHwX6C7Qb9g6S cq+Q== X-Gm-Message-State: ANoB5pkY/Yz4fEHdDPGzKHKOxHuGtoe8SV9wrC807NWZU/jLvSnTE8mr DMEaneKc2p1+EAMU4lwMN50= X-Google-Smtp-Source: AA0mqf7OmpiqMlm/S892R5kNsfKjFN6KBDSnZ8Q+A9z84JFEF7y6W9KQn0O+bLMfc3v816RDgv+h9A== X-Received: by 2002:a63:4d7:0:b0:46e:d8af:9ff6 with SMTP id 206-20020a6304d7000000b0046ed8af9ff6mr9030196pge.98.1668816940285; Fri, 18 Nov 2022 16:15:40 -0800 (PST) Received: from localhost (fwdproxy-prn-014.fbsv.net. [2a03:2880:ff:e::face:b00c]) by smtp.gmail.com with ESMTPSA id h13-20020a170902680d00b001837b19ebb8sm4259585plk.244.2022.11.18.16.15.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 18 Nov 2022 16:15:39 -0800 (PST) From: Nhat Pham To: akpm@linux-foundation.org Cc: hannes@cmpxchg.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, minchan@kernel.org, ngupta@vflare.org, senozhatsky@chromium.org, sjenning@redhat.com, ddstreet@ieee.org, vitaly.wool@konsulko.com Subject: [PATCH v6 2/6] zpool: clean out dead code Date: Fri, 18 Nov 2022 16:15:32 -0800 Message-Id: <20221119001536.2086599-3-nphamcs@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221119001536.2086599-1-nphamcs@gmail.com> References: <20221119001536.2086599-1-nphamcs@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1668816941; a=rsa-sha256; cv=none; b=gzwQNzpegwC1Lqg6AdZ5emmbQjCBCtWjeDWttsUsoOcdKCfD9SPKEyWh/VlDejacGoZXGt UpqcV4jD2jW/A18D0D7ZZyQEcFFq5QoiUsDo33FgU5I8dVrnUJ/wgLvFC02Rj52yt8hNXN iDdc+cMCCV0YcN9KzYvO08+FVbN8yAU= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=hM3jHRqv; spf=pass (imf13.hostedemail.com: domain of nphamcs@gmail.com designates 209.85.215.173 as permitted sender) smtp.mailfrom=nphamcs@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1668816941; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HZvzcMHAUb+r8vph2vKfcXCw1G+EouOeN0ss2yr8n7g=; b=Ym9zWB/bySKNTwlanP83gEZ9iy/U26BtSb8B8z+tT8monILxpF/apMe4tTpgCIwviAhUqJ 1zRNkZ92uvpfN+OxixIe+xDMxN8LodPF+I8iQC6SjOVxpWolFwxPclMZKN6DOePbfnD5sQ UK/2uTGflG3Fh77OmK5LkGixrS0NCw0= X-Stat-Signature: 5pdidcsaptmdenkufifu87ga4epd8q4m X-Rspamd-Queue-Id: 7E0102000A Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=hM3jHRqv; spf=pass (imf13.hostedemail.com: domain of nphamcs@gmail.com designates 209.85.215.173 as permitted sender) smtp.mailfrom=nphamcs@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1668816941-397797 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Johannes Weiner There is a lot of provision for flexibility that isn't actually needed or used. Zswap (the only zpool user) always passes zpool_ops with an .evict method set. The backends who reclaim only do so for zswap, so they can also directly call zpool_ops without indirection or checks. Finally, there is no need to check the retries parameters and bail with -EINVAL in the reclaim function, when that's called just a few lines below with a hard-coded 8. There is no need to duplicate the evictable and sleep_mapped attrs from the driver in zpool_ops. Signed-off-by: Johannes Weiner Signed-off-by: Nhat Pham Reviewed-by: Sergey Senozhatsky --- mm/z3fold.c | 36 +++++------------------------------- mm/zbud.c | 32 +++++--------------------------- mm/zpool.c | 10 ++-------- 3 files changed, 12 insertions(+), 66 deletions(-) -- 2.30.2 diff --git a/mm/z3fold.c b/mm/z3fold.c index cf71da10d04e..a4de0c317ac7 100644 --- a/mm/z3fold.c +++ b/mm/z3fold.c @@ -68,9 +68,6 @@ * Structures *****************/ struct z3fold_pool; -struct z3fold_ops { - int (*evict)(struct z3fold_pool *pool, unsigned long handle); -}; enum buddy { HEADLESS = 0, @@ -138,8 +135,6 @@ struct z3fold_header { * @stale: list of pages marked for freeing * @pages_nr: number of z3fold pages in the pool. * @c_handle: cache for z3fold_buddy_slots allocation - * @ops: pointer to a structure of user defined operations specified at - * pool creation time. * @zpool: zpool driver * @zpool_ops: zpool operations structure with an evict callback * @compact_wq: workqueue for page layout background optimization @@ -158,7 +153,6 @@ struct z3fold_pool { struct list_head stale; atomic64_t pages_nr; struct kmem_cache *c_handle; - const struct z3fold_ops *ops; struct zpool *zpool; const struct zpool_ops *zpool_ops; struct workqueue_struct *compact_wq; @@ -907,13 +901,11 @@ static inline struct z3fold_header *__z3fold_alloc(struct z3fold_pool *pool, * z3fold_create_pool() - create a new z3fold pool * @name: pool name * @gfp: gfp flags when allocating the z3fold pool structure - * @ops: user-defined operations for the z3fold pool * * Return: pointer to the new z3fold pool or NULL if the metadata allocation * failed. */ -static struct z3fold_pool *z3fold_create_pool(const char *name, gfp_t gfp, - const struct z3fold_ops *ops) +static struct z3fold_pool *z3fold_create_pool(const char *name, gfp_t gfp) { struct z3fold_pool *pool = NULL; int i, cpu; @@ -949,7 +941,6 @@ static struct z3fold_pool *z3fold_create_pool(const char *name, gfp_t gfp, if (!pool->release_wq) goto out_wq; INIT_WORK(&pool->work, free_pages_work); - pool->ops = ops; return pool; out_wq: @@ -1230,10 +1221,6 @@ static int z3fold_reclaim_page(struct z3fold_pool *pool, unsigned int retries) slots.pool = (unsigned long)pool | (1 << HANDLES_NOFREE); spin_lock(&pool->lock); - if (!pool->ops || !pool->ops->evict || retries == 0) { - spin_unlock(&pool->lock); - return -EINVAL; - } for (i = 0; i < retries; i++) { if (list_empty(&pool->lru)) { spin_unlock(&pool->lock); @@ -1319,17 +1306,17 @@ static int z3fold_reclaim_page(struct z3fold_pool *pool, unsigned int retries) } /* Issue the eviction callback(s) */ if (middle_handle) { - ret = pool->ops->evict(pool, middle_handle); + ret = pool->zpool_ops->evict(pool->zpool, middle_handle); if (ret) goto next; } if (first_handle) { - ret = pool->ops->evict(pool, first_handle); + ret = pool->zpool_ops->evict(pool->zpool, first_handle); if (ret) goto next; } if (last_handle) { - ret = pool->ops->evict(pool, last_handle); + ret = pool->zpool_ops->evict(pool->zpool, last_handle); if (ret) goto next; } @@ -1593,26 +1580,13 @@ static const struct movable_operations z3fold_mops = { * zpool ****************/ -static int z3fold_zpool_evict(struct z3fold_pool *pool, unsigned long handle) -{ - if (pool->zpool && pool->zpool_ops && pool->zpool_ops->evict) - return pool->zpool_ops->evict(pool->zpool, handle); - else - return -ENOENT; -} - -static const struct z3fold_ops z3fold_zpool_ops = { - .evict = z3fold_zpool_evict -}; - static void *z3fold_zpool_create(const char *name, gfp_t gfp, const struct zpool_ops *zpool_ops, struct zpool *zpool) { struct z3fold_pool *pool; - pool = z3fold_create_pool(name, gfp, - zpool_ops ? &z3fold_zpool_ops : NULL); + pool = z3fold_create_pool(name, gfp); if (pool) { pool->zpool = zpool; pool->zpool_ops = zpool_ops; diff --git a/mm/zbud.c b/mm/zbud.c index 6348932430b8..3acd26193920 100644 --- a/mm/zbud.c +++ b/mm/zbud.c @@ -74,10 +74,6 @@ struct zbud_pool; -struct zbud_ops { - int (*evict)(struct zbud_pool *pool, unsigned long handle); -}; - /** * struct zbud_pool - stores metadata for each zbud pool * @lock: protects all pool fields and first|last_chunk fields of any @@ -90,8 +86,6 @@ struct zbud_ops { * @lru: list tracking the zbud pages in LRU order by most recently * added buddy. * @pages_nr: number of zbud pages in the pool. - * @ops: pointer to a structure of user defined operations specified at - * pool creation time. * @zpool: zpool driver * @zpool_ops: zpool operations structure with an evict callback * @@ -110,7 +104,6 @@ struct zbud_pool { }; struct list_head lru; u64 pages_nr; - const struct zbud_ops *ops; struct zpool *zpool; const struct zpool_ops *zpool_ops; }; @@ -212,12 +205,11 @@ static int num_free_chunks(struct zbud_header *zhdr) /** * zbud_create_pool() - create a new zbud pool * @gfp: gfp flags when allocating the zbud pool structure - * @ops: user-defined operations for the zbud pool * * Return: pointer to the new zbud pool or NULL if the metadata allocation * failed. */ -static struct zbud_pool *zbud_create_pool(gfp_t gfp, const struct zbud_ops *ops) +static struct zbud_pool *zbud_create_pool(gfp_t gfp) { struct zbud_pool *pool; int i; @@ -231,7 +223,6 @@ static struct zbud_pool *zbud_create_pool(gfp_t gfp, const struct zbud_ops *ops) INIT_LIST_HEAD(&pool->buddied); INIT_LIST_HEAD(&pool->lru); pool->pages_nr = 0; - pool->ops = ops; return pool; } @@ -419,8 +410,7 @@ static int zbud_reclaim_page(struct zbud_pool *pool, unsigned int retries) unsigned long first_handle = 0, last_handle = 0; spin_lock(&pool->lock); - if (!pool->ops || !pool->ops->evict || list_empty(&pool->lru) || - retries == 0) { + if (list_empty(&pool->lru)) { spin_unlock(&pool->lock); return -EINVAL; } @@ -444,12 +434,12 @@ static int zbud_reclaim_page(struct zbud_pool *pool, unsigned int retries) /* Issue the eviction callback(s) */ if (first_handle) { - ret = pool->ops->evict(pool, first_handle); + ret = pool->zpool_ops->evict(pool->zpool, first_handle); if (ret) goto next; } if (last_handle) { - ret = pool->ops->evict(pool, last_handle); + ret = pool->zpool_ops->evict(pool->zpool, last_handle); if (ret) goto next; } @@ -524,25 +514,13 @@ static u64 zbud_get_pool_size(struct zbud_pool *pool) * zpool ****************/ -static int zbud_zpool_evict(struct zbud_pool *pool, unsigned long handle) -{ - if (pool->zpool && pool->zpool_ops && pool->zpool_ops->evict) - return pool->zpool_ops->evict(pool->zpool, handle); - else - return -ENOENT; -} - -static const struct zbud_ops zbud_zpool_ops = { - .evict = zbud_zpool_evict -}; - static void *zbud_zpool_create(const char *name, gfp_t gfp, const struct zpool_ops *zpool_ops, struct zpool *zpool) { struct zbud_pool *pool; - pool = zbud_create_pool(gfp, zpool_ops ? &zbud_zpool_ops : NULL); + pool = zbud_create_pool(gfp); if (pool) { pool->zpool = zpool; pool->zpool_ops = zpool_ops; diff --git a/mm/zpool.c b/mm/zpool.c index 68facc193496..fc3a9893e107 100644 --- a/mm/zpool.c +++ b/mm/zpool.c @@ -21,9 +21,6 @@ struct zpool { struct zpool_driver *driver; void *pool; - const struct zpool_ops *ops; - bool evictable; - bool can_sleep_mapped; }; static LIST_HEAD(drivers_head); @@ -177,9 +174,6 @@ struct zpool *zpool_create_pool(const char *type, const char *name, gfp_t gfp, zpool->driver = driver; zpool->pool = driver->create(name, gfp, ops, zpool); - zpool->ops = ops; - zpool->evictable = driver->shrink && ops && ops->evict; - zpool->can_sleep_mapped = driver->sleep_mapped; if (!zpool->pool) { pr_err("couldn't create %s pool\n", type); @@ -380,7 +374,7 @@ u64 zpool_get_total_size(struct zpool *zpool) */ bool zpool_evictable(struct zpool *zpool) { - return zpool->evictable; + return zpool->driver->shrink; } /** @@ -391,7 +385,7 @@ bool zpool_evictable(struct zpool *zpool) */ bool zpool_can_sleep_mapped(struct zpool *zpool) { - return zpool->can_sleep_mapped; + return zpool->driver->sleep_mapped; } MODULE_LICENSE("GPL"); From patchwork Sat Nov 19 00:15:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nhat Pham X-Patchwork-Id: 13049324 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 22806C4321E for ; Sat, 19 Nov 2022 00:15:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B1A968E0002; Fri, 18 Nov 2022 19:15:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id ACA2E6B007E; Fri, 18 Nov 2022 19:15:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 944E56B0080; Fri, 18 Nov 2022 19:15:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 8358D6B007D for ; Fri, 18 Nov 2022 19:15:44 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 4B0D1409AF for ; Sat, 19 Nov 2022 00:15:44 +0000 (UTC) X-FDA: 80148273408.04.31E7667 Received: from mail-pf1-f177.google.com (mail-pf1-f177.google.com [209.85.210.177]) by imf21.hostedemail.com (Postfix) with ESMTP id E526F1C0009 for ; Sat, 19 Nov 2022 00:15:43 +0000 (UTC) Received: by mail-pf1-f177.google.com with SMTP id d192so6393516pfd.0 for ; Fri, 18 Nov 2022 16:15:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xlNO430pFQOka1esCEqa7IqA23nNPCeXNP57iQ3X88k=; b=dou3yaHyJKMOuyzUjM+AISJf6d71pZrUfWgikbtI9j3Y6hoNgor6yN25JPdwAaQ20T bmEir2kFtNskQqAa6VxLNCDTkWDBSHlNjmNr1BjrGtRvWaoVjBjwCe2CNlGrMB9dPazV zVupTfokhJ0RXHrTqeO8XRh2Sl5a0XajITvW/0w0eEWrXHyjTEthY+XOlM1/P3Qx+FOk PojrXY03qZdpjSSaqO9sYspzBL7r770DMqA1qyL4NgGRnuKeKEWvSnSjInRtGVvk6Vgd LqD/1+XOJo0RiC3b6nmNF2vpTvmAVhB1Dh2dkK+s9F02Uq+tVt1tf0f7PGy4eLuCIXyD 42JQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xlNO430pFQOka1esCEqa7IqA23nNPCeXNP57iQ3X88k=; b=FG0ekwnboYmdpE3s5Fugwv6rjODs1kSdrJyXbUoalrWWnXFO9s5lCoM2gVNZDBVbtg R1z5ZqCGsKevr+dCKvahSzIitcwOc49u/itaOlU/rE+6hRoG2lhjUJWWkG49jfRmOLKP vElOx1UZ4lFsNL/wKajG0BVUsZQ3w8CQvAKntsCkPSR8Y3FtcNQSUm4Of0s251VR8GTr +9MS8Nr9v2wiOLi727F4ehc0ISGZrhKA88HzL0wroWiiKam21hY/bG3Ebf5LcmXceDzO pEtBe8HdbDOWubcFpnKN6BjFDNYSjnyC92e4uf9P3g/zGPBaMCDHJAFZ69Be4TnBQw5O g5sQ== X-Gm-Message-State: ANoB5pla8e/8mh4iQR2YlZAmKrKYnb8enxY5EXzidkIM5nRS5BXEgPV6 kUvmFbUA/JClj42UmDTxw2Q= X-Google-Smtp-Source: AA0mqf6yUmUHE0UvTfSdN0ChRKmCdP7x9U3+tTT0bCMv0jXDGck8BJGrPxpLEzKY2+HNMBmDm17XBg== X-Received: by 2002:a63:d241:0:b0:439:8688:a98d with SMTP id t1-20020a63d241000000b004398688a98dmr8511296pgi.424.1668816943018; Fri, 18 Nov 2022 16:15:43 -0800 (PST) Received: from localhost (fwdproxy-prn-017.fbsv.net. [2a03:2880:ff:11::face:b00c]) by smtp.gmail.com with ESMTPSA id u139-20020a627991000000b00561cf757749sm3751245pfc.183.2022.11.18.16.15.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 18 Nov 2022 16:15:42 -0800 (PST) From: Nhat Pham To: akpm@linux-foundation.org Cc: hannes@cmpxchg.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, minchan@kernel.org, ngupta@vflare.org, senozhatsky@chromium.org, sjenning@redhat.com, ddstreet@ieee.org, vitaly.wool@konsulko.com Subject: [PATCH v6 4/6] zsmalloc: Add a LRU to zs_pool to keep track of zspages in LRU order Date: Fri, 18 Nov 2022 16:15:34 -0800 Message-Id: <20221119001536.2086599-5-nphamcs@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221119001536.2086599-1-nphamcs@gmail.com> References: <20221119001536.2086599-1-nphamcs@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1668816944; a=rsa-sha256; cv=none; b=rSKM0j8DERvAFzIC27Xgf2QCX50QInQggaGOSDygMPH1GTEOb0adLAhv7vNSFp5G/uLrow DjncOchoq6abtDEookKlw8yeL+XxgVD6IQ5AdR/fqt/UWvxlVUp4XuKPxK1tfkgpzesxbj zjKwHIZJZ4Sb+1/NlYTl9+a/ZcvR6rM= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=dou3yaHy; spf=pass (imf21.hostedemail.com: domain of nphamcs@gmail.com designates 209.85.210.177 as permitted sender) smtp.mailfrom=nphamcs@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1668816944; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xlNO430pFQOka1esCEqa7IqA23nNPCeXNP57iQ3X88k=; b=oAy05/Cw+hbkrsrXO/Yuc6h3vsDedi1J0QCYryywygIUNDTwl25GVDbbSya0Hy2L8MJerU PmRQu/qoJIlQfUVL29BzuvOeF+o+SuHV3xseBohJZoyd2quuiXO24mPIacgyg0vh6EuAAy 2ZUifDWcwmDzyhyQQqBpNe6RiDeEpGM= X-Stat-Signature: ipuoih6wscsd9esu17hadz7nwg8pfd7k X-Rspamd-Queue-Id: E526F1C0009 Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=dou3yaHy; spf=pass (imf21.hostedemail.com: domain of nphamcs@gmail.com designates 209.85.210.177 as permitted sender) smtp.mailfrom=nphamcs@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam04 X-Rspam-User: X-HE-Tag: 1668816943-624625 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This helps determines the coldest zspages as candidates for writeback. Signed-off-by: Nhat Pham Acked-by: Johannes Weiner Acked-by: Minchan Kim --- mm/zsmalloc.c | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) -- 2.30.2 diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 326faa751f0a..7dd464b5a6a5 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -239,6 +239,11 @@ struct zs_pool { /* Compact classes */ struct shrinker shrinker; +#ifdef CONFIG_ZPOOL + /* List tracking the zspages in LRU order by most recently added object */ + struct list_head lru; +#endif + #ifdef CONFIG_ZSMALLOC_STAT struct dentry *stat_dentry; #endif @@ -260,6 +265,12 @@ struct zspage { unsigned int freeobj; struct page *first_page; struct list_head list; /* fullness list */ + +#ifdef CONFIG_ZPOOL + /* links the zspage to the lru list in the pool */ + struct list_head lru; +#endif + struct zs_pool *pool; #ifdef CONFIG_COMPACTION rwlock_t lock; @@ -953,6 +964,9 @@ static void free_zspage(struct zs_pool *pool, struct size_class *class, } remove_zspage(class, zspage, ZS_EMPTY); +#ifdef CONFIG_ZPOOL + list_del(&zspage->lru); +#endif __free_zspage(pool, class, zspage); } @@ -998,6 +1012,10 @@ static void init_zspage(struct size_class *class, struct zspage *zspage) off %= PAGE_SIZE; } +#ifdef CONFIG_ZPOOL + INIT_LIST_HEAD(&zspage->lru); +#endif + set_freeobj(zspage, 0); } @@ -1249,6 +1267,15 @@ void *zs_map_object(struct zs_pool *pool, unsigned long handle, obj_to_location(obj, &page, &obj_idx); zspage = get_zspage(page); +#ifdef CONFIG_ZPOOL + /* Move the zspage to front of pool's LRU */ + if (mm == ZS_MM_WO) { + if (!list_empty(&zspage->lru)) + list_del(&zspage->lru); + list_add(&zspage->lru, &pool->lru); + } +#endif + /* * migration cannot move any zpages in this zspage. Here, pool->lock * is too heavy since callers would take some time until they calls @@ -1967,6 +1994,9 @@ static void async_free_zspage(struct work_struct *work) VM_BUG_ON(fullness != ZS_EMPTY); class = pool->size_class[class_idx]; spin_lock(&pool->lock); +#ifdef CONFIG_ZPOOL + list_del(&zspage->lru); +#endif __free_zspage(pool, class, zspage); spin_unlock(&pool->lock); } @@ -2278,6 +2308,10 @@ struct zs_pool *zs_create_pool(const char *name) */ zs_register_shrinker(pool); +#ifdef CONFIG_ZPOOL + INIT_LIST_HEAD(&pool->lru); +#endif + return pool; err: From patchwork Sat Nov 19 00:15:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nhat Pham X-Patchwork-Id: 13049325 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 57DD2C4167B for ; Sat, 19 Nov 2022 00:15:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ECA166B007D; Fri, 18 Nov 2022 19:15:45 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E4E816B007E; Fri, 18 Nov 2022 19:15:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C53996B0080; Fri, 18 Nov 2022 19:15:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id A8F606B007D for ; Fri, 18 Nov 2022 19:15:45 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 81B3E1C6068 for ; Sat, 19 Nov 2022 00:15:45 +0000 (UTC) X-FDA: 80148273450.25.1ED7B53 Received: from mail-pf1-f169.google.com (mail-pf1-f169.google.com [209.85.210.169]) by imf23.hostedemail.com (Postfix) with ESMTP id 28B92140004 for ; Sat, 19 Nov 2022 00:15:44 +0000 (UTC) Received: by mail-pf1-f169.google.com with SMTP id y203so6352109pfb.4 for ; Fri, 18 Nov 2022 16:15:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ZXVKU8ju9i6lgVo4GHLlSQvxR+3nZ3i/iGJQSsIEw+M=; b=jBhZspZhLzlDshrZ/lo4FyZvP0Y26uOgONnXztvGdE4w/RlHjf7VoFHEy/c1QXw9OC 3sh/pvMH4xjovvN5ZMgrFxs7uaMbEz6EfixhJHiZYxopq/6bP14oIYnEtgTOHe1bmjEt 2pH3951rnmqnaeRLCM26DP3goHH8vWmo0GFgnf81JZW/KSDg+nnAJfgZasbKgYIZsw66 ld/4s7hw9iZC3UYE461byM9rk8f9S8QzWQ4Q7tAQW9aA+fMgeZ6rfu/441Wsqap1fvsg TKwiBsOZCHowCBL25wiaNVJupPCK9Vqkzg0FSskz55asJlf6oPn+0Nu52DBiMPBxuC63 O5hA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ZXVKU8ju9i6lgVo4GHLlSQvxR+3nZ3i/iGJQSsIEw+M=; b=moh2jOZuXn2lS/pN8zzgSfXYSRNSqtayWsvnyvML7zmrRuRbe0sVyq0k9T4w8rrY0x Nhc5REcowTonzjjO4vKcquTlqVldOovg86ZMKOmRKllOhChEiS3UF45Yj1KZo+IKiDE9 k3AwGuQhrzFvi0YavcqglHlgbxjKOP98BNvgMfGF9qhac3fCGuL3AhGFkrdTUtZxFNud 4sLrIt2EH1McKwYhBXvsXXWRZU01x6As6t6FvpfNGnNTnLYMkc1sOJxuZbMcfBhjz6fP oE44eysUgTszAWO24oZbpHjikJKSiUYHur9RNvjLCvluMOjzR5+uKiqCl9e4p9F5GR+N Q9vA== X-Gm-Message-State: ANoB5pmSUuC0hr1VgDJtzsCKb0RZ5O7bi5oQRyKNsUS/bOfRjVBdqHkP Am3GfdI0SQJasWhUIgxTRZg= X-Google-Smtp-Source: AA0mqf4o0mZwA9ALBxUb5i0SL4h26XRRE7ZV45iOu6VSl418vkZYGDCrR1JlEljlL4scZsw87eJCWQ== X-Received: by 2002:a63:4701:0:b0:476:d784:1c44 with SMTP id u1-20020a634701000000b00476d7841c44mr8385602pga.431.1668816944233; Fri, 18 Nov 2022 16:15:44 -0800 (PST) Received: from localhost (fwdproxy-prn-120.fbsv.net. [2a03:2880:ff:78::face:b00c]) by smtp.gmail.com with ESMTPSA id x2-20020a626302000000b0056d3b8f530csm3827237pfb.34.2022.11.18.16.15.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 18 Nov 2022 16:15:43 -0800 (PST) From: Nhat Pham To: akpm@linux-foundation.org Cc: hannes@cmpxchg.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, minchan@kernel.org, ngupta@vflare.org, senozhatsky@chromium.org, sjenning@redhat.com, ddstreet@ieee.org, vitaly.wool@konsulko.com Subject: [PATCH v6 5/6] zsmalloc: Add zpool_ops field to zs_pool to store evict handlers Date: Fri, 18 Nov 2022 16:15:35 -0800 Message-Id: <20221119001536.2086599-6-nphamcs@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221119001536.2086599-1-nphamcs@gmail.com> References: <20221119001536.2086599-1-nphamcs@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1668816945; a=rsa-sha256; cv=none; b=NpqjP44SZzXDJTNQjnJ61n/r4aCtgQulxg/DChEQDI/SX+j2PkrhJi0uGL4Ygr6FPyBoRg FDOqggOrYSTE9iEFtWw3jRBT0bcIeLfz6xEQGnhEKtVi1G2NaDZMsDL3K+aYOKZxTJsD1b CFSBoAAfBCSsGMjs9fbVfbI8luUqbOM= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=jBhZspZh; spf=pass (imf23.hostedemail.com: domain of nphamcs@gmail.com designates 209.85.210.169 as permitted sender) smtp.mailfrom=nphamcs@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1668816945; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ZXVKU8ju9i6lgVo4GHLlSQvxR+3nZ3i/iGJQSsIEw+M=; b=iv7iItdPR6h7A1FWtxMicm9Wcj5s6mUrYYarPpHfxXDW/v0JmdncvLEjbf9TksflQNurI1 rs6AeCmH+jRiEx6INKNjNUrdtY7DtdtXVibmkl1VQRN+XkT12Yi53SrGO/qEm8bSJPuKnT EOSPbGUguSHEqmFtEq2ZUCfecH8mups= Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=jBhZspZh; spf=pass (imf23.hostedemail.com: domain of nphamcs@gmail.com designates 209.85.210.169 as permitted sender) smtp.mailfrom=nphamcs@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 28B92140004 X-Rspam-User: X-Stat-Signature: gzuz8k1g7co3w1fk16bpkiayxop36gn7 X-HE-Tag: 1668816944-185640 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This adds a new field to zs_pool to store evict handlers for writeback, analogous to the zbud allocator. Signed-off-by: Nhat Pham Acked-by: Minchan Kim Acked-by: Johannes Weiner --- mm/zsmalloc.c | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) -- 2.30.2 diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 7dd464b5a6a5..9920f3584511 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -242,6 +242,8 @@ struct zs_pool { #ifdef CONFIG_ZPOOL /* List tracking the zspages in LRU order by most recently added object */ struct list_head lru; + struct zpool *zpool; + const struct zpool_ops *zpool_ops; #endif #ifdef CONFIG_ZSMALLOC_STAT @@ -382,7 +384,14 @@ static void *zs_zpool_create(const char *name, gfp_t gfp, * different contexts and its caller must provide a valid * gfp mask. */ - return zs_create_pool(name); + struct zs_pool *pool = zs_create_pool(name); + + if (pool) { + pool->zpool = zpool; + pool->zpool_ops = zpool_ops; + } + + return pool; } static void zs_zpool_destroy(void *pool) From patchwork Sat Nov 19 00:15:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nhat Pham X-Patchwork-Id: 13049326 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AB2DAC4332F for ; Sat, 19 Nov 2022 00:15:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 418616B007E; Fri, 18 Nov 2022 19:15:47 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2B4B76B0080; Fri, 18 Nov 2022 19:15:47 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0E17A8E0003; Fri, 18 Nov 2022 19:15:47 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id EAC666B007E for ; Fri, 18 Nov 2022 19:15:46 -0500 (EST) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id C903DAAEEC for ; Sat, 19 Nov 2022 00:15:46 +0000 (UTC) X-FDA: 80148273492.14.189EA00 Received: from mail-pj1-f42.google.com (mail-pj1-f42.google.com [209.85.216.42]) by imf20.hostedemail.com (Postfix) with ESMTP id 67D4A1C000F for ; Sat, 19 Nov 2022 00:15:46 +0000 (UTC) Received: by mail-pj1-f42.google.com with SMTP id w15-20020a17090a380f00b0021873113cb4so6075902pjb.0 for ; Fri, 18 Nov 2022 16:15:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Co9VEJF4pdxmrIDsS8H+BTo/CtOiZYr3gRWH6+3Vpr4=; b=T0YKjRwwzXJG2yRWFWfCM687bYqCIIoadzwEjQFvRTsy4BBI7Q1v1cPIo3piv1EOu5 ksLLpkjjRbaCjmKa2qR9lbumsFOCXTXPEgF858ZOaWAvmuU21JTnd9/H72E1Y9xmuQmO BBbiGmtGmXmdVqKR6lULR+/UERUdy94CZEgLg5tu12N+NXZB3grZXVL8fYawPBSHrIbT 9grObhJ7iztQZ9tpmLwnTxgrlApibIBEAHtjdNYiAmLEArbEXfa+1m/hsVNTOYQTZM41 LqH8ZFFGKdde707H4Q1MHM/Q3gbdcfZbkFlQ4qTC/+S04DVQKr4lNJBwZlj/bUU9xsFE t8Lg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Co9VEJF4pdxmrIDsS8H+BTo/CtOiZYr3gRWH6+3Vpr4=; b=J7e9EyzAy1PrgwKM7cRgxfeDbYgdTTDQSvsaO7pxONbjnBWLguQ9vfV0QhrWfC6Nea qkekWxdpwuEz0B4eqEBVyb5yLSE+tDxJ5zwJMq3nC1vcvi8G/VOU6v8VDKEB+yag9XLi eygOGycU2YOCP7QbOuVMdNsyQTN/8FC8TGBIoeEnXdeqPJQedHNH/9L/xLu6kNw9YmN4 lg8t9TF+3UX1+PfUMnmWYO7HuzJWuYUC3YQBrOg1FF7QxllW6yrM7hy9Onm61ccJubWi Eeu+fisd8u00KPmsvP8Ufi2fLvPULkMQ7ly6AVSMQVns5fRUrospfitXyaVSFeg8KkeW LrEQ== X-Gm-Message-State: ANoB5pnE28YCI7LpEL6aCKMCr/xiMwiRaEwSGmxxiuU5FFx/BIt5uiGG FSrlNHMDg2i3Uz9VlznJFkw= X-Google-Smtp-Source: AA0mqf6B6K/1xRL8XHotAHoGiBG7qKCqhD0g8IdKbNhXqLYxxJbDoyXr34M0AscDiqFOxLyMlV3RkQ== X-Received: by 2002:a17:903:2055:b0:188:bed6:3fbe with SMTP id q21-20020a170903205500b00188bed63fbemr1812139pla.38.1668816945445; Fri, 18 Nov 2022 16:15:45 -0800 (PST) Received: from localhost (fwdproxy-prn-018.fbsv.net. [2a03:2880:ff:12::face:b00c]) by smtp.gmail.com with ESMTPSA id o14-20020a170902d4ce00b00186acb14c4asm4418285plg.67.2022.11.18.16.15.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 18 Nov 2022 16:15:45 -0800 (PST) From: Nhat Pham To: akpm@linux-foundation.org Cc: hannes@cmpxchg.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, minchan@kernel.org, ngupta@vflare.org, senozhatsky@chromium.org, sjenning@redhat.com, ddstreet@ieee.org, vitaly.wool@konsulko.com Subject: [PATCH v6 6/6] zsmalloc: Implement writeback mechanism for zsmalloc Date: Fri, 18 Nov 2022 16:15:36 -0800 Message-Id: <20221119001536.2086599-7-nphamcs@gmail.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221119001536.2086599-1-nphamcs@gmail.com> References: <20221119001536.2086599-1-nphamcs@gmail.com> MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1668816946; a=rsa-sha256; cv=none; b=mrz6tqYqI3ey/X/d+ybX9XYtQXh9GcmBDnP8QMYVPX9kcUuX4iEfQncdljNMWSIdRPpLCh AWpAedcZoQeo0EtjiuEUa2+Q95drfms/bkytiwP78QOAmH5i0/SmcLDUcsZVhaaUCAN/C3 q5PBBuTAVXNqqKuNGkiVs5cEubrorxE= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=T0YKjRww; spf=pass (imf20.hostedemail.com: domain of nphamcs@gmail.com designates 209.85.216.42 as permitted sender) smtp.mailfrom=nphamcs@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1668816946; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Co9VEJF4pdxmrIDsS8H+BTo/CtOiZYr3gRWH6+3Vpr4=; b=sRQ2iDgIq3t2qiq13zd2+tGkuxYeGQNb3qj60NjbB9mzfcWiXdIl8arNh5wPhY1l99r5ip qAPFe1HwKZY1nNyktR71+WnFHmuZGFoe9fMFuze/I5Z8Fjd90rnWIUj1eK9HXHrnycyMLL Z1Xhq/kmSIuJUT8MsK2smdECdPF3wks= X-Stat-Signature: mkrapaxggqkpg4zatmirwyahyd7wb5eb X-Rspamd-Queue-Id: 67D4A1C000F Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=T0YKjRww; spf=pass (imf20.hostedemail.com: domain of nphamcs@gmail.com designates 209.85.216.42 as permitted sender) smtp.mailfrom=nphamcs@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam04 X-Rspam-User: X-HE-Tag: 1668816946-215022 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This commit adds the writeback mechanism for zsmalloc, analogous to the zbud allocator. Zsmalloc will attempt to determine the coldest zspage (i.e least recently used) in the pool, and attempt to write back all the stored compressed objects via the pool's evict handler. Signed-off-by: Nhat Pham Acked-by: Johannes Weiner Acked-by: Minchan Kim --- mm/zsmalloc.c | 193 +++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 182 insertions(+), 11 deletions(-) -- 2.30.2 diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 9920f3584511..3fba04e10227 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -271,12 +271,13 @@ struct zspage { #ifdef CONFIG_ZPOOL /* links the zspage to the lru list in the pool */ struct list_head lru; + bool under_reclaim; + /* list of unfreed handles whose objects have been reclaimed */ + unsigned long *deferred_handles; #endif struct zs_pool *pool; -#ifdef CONFIG_COMPACTION rwlock_t lock; -#endif }; struct mapping_area { @@ -297,10 +298,11 @@ static bool ZsHugePage(struct zspage *zspage) return zspage->huge; } -#ifdef CONFIG_COMPACTION static void migrate_lock_init(struct zspage *zspage); static void migrate_read_lock(struct zspage *zspage); static void migrate_read_unlock(struct zspage *zspage); + +#ifdef CONFIG_COMPACTION static void migrate_write_lock(struct zspage *zspage); static void migrate_write_lock_nested(struct zspage *zspage); static void migrate_write_unlock(struct zspage *zspage); @@ -308,9 +310,6 @@ static void kick_deferred_free(struct zs_pool *pool); static void init_deferred_free(struct zs_pool *pool); static void SetZsPageMovable(struct zs_pool *pool, struct zspage *zspage); #else -static void migrate_lock_init(struct zspage *zspage) {} -static void migrate_read_lock(struct zspage *zspage) {} -static void migrate_read_unlock(struct zspage *zspage) {} static void migrate_write_lock(struct zspage *zspage) {} static void migrate_write_lock_nested(struct zspage *zspage) {} static void migrate_write_unlock(struct zspage *zspage) {} @@ -413,6 +412,27 @@ static void zs_zpool_free(void *pool, unsigned long handle) zs_free(pool, handle); } +static int zs_reclaim_page(struct zs_pool *pool, unsigned int retries); + +static int zs_zpool_shrink(void *pool, unsigned int pages, + unsigned int *reclaimed) +{ + unsigned int total = 0; + int ret = -EINVAL; + + while (total < pages) { + ret = zs_reclaim_page(pool, 8); + if (ret < 0) + break; + total++; + } + + if (reclaimed) + *reclaimed = total; + + return ret; +} + static void *zs_zpool_map(void *pool, unsigned long handle, enum zpool_mapmode mm) { @@ -451,6 +471,7 @@ static struct zpool_driver zs_zpool_driver = { .malloc_support_movable = true, .malloc = zs_zpool_malloc, .free = zs_zpool_free, + .shrink = zs_zpool_shrink, .map = zs_zpool_map, .unmap = zs_zpool_unmap, .total_size = zs_zpool_total_size, @@ -924,6 +945,25 @@ static int trylock_zspage(struct zspage *zspage) return 0; } +#ifdef CONFIG_ZPOOL +/* + * Free all the deferred handles whose objects are freed in zs_free. + */ +static void free_handles(struct zs_pool *pool, struct zspage *zspage) +{ + unsigned long handle = (unsigned long)zspage->deferred_handles; + + while (handle) { + unsigned long nxt_handle = handle_to_obj(handle); + + cache_free_handle(pool, handle); + handle = nxt_handle; + } +} +#else +static inline void free_handles(struct zs_pool *pool, struct zspage *zspage) {} +#endif + static void __free_zspage(struct zs_pool *pool, struct size_class *class, struct zspage *zspage) { @@ -938,6 +978,9 @@ static void __free_zspage(struct zs_pool *pool, struct size_class *class, VM_BUG_ON(get_zspage_inuse(zspage)); VM_BUG_ON(fg != ZS_EMPTY); + /* Free all deferred handles from zs_free */ + free_handles(pool, zspage); + next = page = get_first_page(zspage); do { VM_BUG_ON_PAGE(!PageLocked(page), page); @@ -1023,6 +1066,8 @@ static void init_zspage(struct size_class *class, struct zspage *zspage) #ifdef CONFIG_ZPOOL INIT_LIST_HEAD(&zspage->lru); + zspage->under_reclaim = false; + zspage->deferred_handles = NULL; #endif set_freeobj(zspage, 0); @@ -1535,12 +1580,26 @@ void zs_free(struct zs_pool *pool, unsigned long handle) obj_free(class->size, obj); class_stat_dec(class, OBJ_USED, 1); + +#ifdef CONFIG_ZPOOL + if (zspage->under_reclaim) { + /* + * Reclaim needs the handles during writeback. It'll free + * them along with the zspage when it's done with them. + * + * Record current deferred handle at the memory location + * whose address is given by handle. + */ + record_obj(handle, (unsigned long)zspage->deferred_handles); + zspage->deferred_handles = (unsigned long *)handle; + spin_unlock(&pool->lock); + return; + } +#endif fullness = fix_fullness_group(class, zspage); - if (fullness != ZS_EMPTY) - goto out; + if (fullness == ZS_EMPTY) + free_zspage(pool, class, zspage); - free_zspage(pool, class, zspage); -out: spin_unlock(&pool->lock); cache_free_handle(pool, handle); } @@ -1740,7 +1799,7 @@ static enum fullness_group putback_zspage(struct size_class *class, return fullness; } -#ifdef CONFIG_COMPACTION +#if defined(CONFIG_ZPOOL) || defined(CONFIG_COMPACTION) /* * To prevent zspage destroy during migration, zspage freeing should * hold locks of all pages in the zspage. @@ -1782,6 +1841,24 @@ static void lock_zspage(struct zspage *zspage) } migrate_read_unlock(zspage); } +#endif /* defined(CONFIG_ZPOOL) || defined(CONFIG_COMPACTION) */ + +#ifdef CONFIG_ZPOOL +/* + * Unlocks all the pages of the zspage. + * + * pool->lock must be held before this function is called + * to prevent the underlying pages from migrating. + */ +static void unlock_zspage(struct zspage *zspage) +{ + struct page *page = get_first_page(zspage); + + do { + unlock_page(page); + } while ((page = get_next_page(page)) != NULL); +} +#endif /* CONFIG_ZPOOL */ static void migrate_lock_init(struct zspage *zspage) { @@ -1798,6 +1875,7 @@ static void migrate_read_unlock(struct zspage *zspage) __releases(&zspage->lock) read_unlock(&zspage->lock); } +#ifdef CONFIG_COMPACTION static void migrate_write_lock(struct zspage *zspage) { write_lock(&zspage->lock); @@ -2362,6 +2440,99 @@ void zs_destroy_pool(struct zs_pool *pool) } EXPORT_SYMBOL_GPL(zs_destroy_pool); +#ifdef CONFIG_ZPOOL +static int zs_reclaim_page(struct zs_pool *pool, unsigned int retries) +{ + int i, obj_idx, ret = 0; + unsigned long handle; + struct zspage *zspage; + struct page *page; + enum fullness_group fullness; + + /* Lock LRU and fullness list */ + spin_lock(&pool->lock); + if (list_empty(&pool->lru)) { + spin_unlock(&pool->lock); + return -EINVAL; + } + + for (i = 0; i < retries; i++) { + struct size_class *class; + + zspage = list_last_entry(&pool->lru, struct zspage, lru); + list_del(&zspage->lru); + + /* zs_free may free objects, but not the zspage and handles */ + zspage->under_reclaim = true; + + class = zspage_class(pool, zspage); + fullness = get_fullness_group(class, zspage); + + /* Lock out object allocations and object compaction */ + remove_zspage(class, zspage, fullness); + + spin_unlock(&pool->lock); + + /* Lock backing pages into place */ + lock_zspage(zspage); + + obj_idx = 0; + page = zspage->first_page; + while (1) { + handle = find_alloced_obj(class, page, &obj_idx); + if (!handle) { + page = get_next_page(page); + if (!page) + break; + obj_idx = 0; + continue; + } + + /* + * This will write the object and call zs_free. + * + * zs_free will free the object, but the + * under_reclaim flag prevents it from freeing + * the zspage altogether. This is necessary so + * that we can continue working with the + * zspage potentially after the last object + * has been freed. + */ + ret = pool->zpool_ops->evict(pool->zpool, handle); + if (ret) + goto next; + + obj_idx++; + } + +next: + /* For freeing the zspage, or putting it back in the pool and LRU list. */ + spin_lock(&pool->lock); + zspage->under_reclaim = false; + + if (!get_zspage_inuse(zspage)) { + /* + * Fullness went stale as zs_free() won't touch it + * while the page is removed from the pool. Fix it + * up for the check in __free_zspage(). + */ + zspage->fullness = ZS_EMPTY; + + __free_zspage(pool, class, zspage); + spin_unlock(&pool->lock); + return 0; + } + + putback_zspage(class, zspage); + list_add(&zspage->lru, &pool->lru); + unlock_zspage(zspage); + } + + spin_unlock(&pool->lock); + return -EAGAIN; +} +#endif /* CONFIG_ZPOOL */ + static int __init zs_init(void) { int ret;