From patchwork Thu Jul 4 11:24:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Baolin Wang X-Patchwork-Id: 13723639 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8D4C8C31D97 for ; Thu, 4 Jul 2024 11:25:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AA6466B00E4; Thu, 4 Jul 2024 07:25:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A2CF56B00E5; Thu, 4 Jul 2024 07:25:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8AB1F6B00E6; Thu, 4 Jul 2024 07:25:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 6674B6B00E4 for ; Thu, 4 Jul 2024 07:25:25 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 250F1160D9D for ; Thu, 4 Jul 2024 11:25:25 +0000 (UTC) X-FDA: 82301839410.22.4AAC7A7 Received: from out30-97.freemail.mail.aliyun.com (out30-97.freemail.mail.aliyun.com [115.124.30.97]) by imf02.hostedemail.com (Postfix) with ESMTP id 0CB3F8001B for ; Thu, 4 Jul 2024 11:25:22 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=linux.alibaba.com header.s=default header.b=xYOmMn7I; spf=pass (imf02.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.97 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=linux.alibaba.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720092298; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=JZrqpGDPBtXPHidCmEmeC3ZJ9JHyMn+COl4BmjCUDX8=; b=l7mRm2G8eM+JvNy9Qmc6HlaugFL2Wf8LREAiUzm7NO/tVFfYbROThLRYIaQiY25wZq6jeR Yi6qk0Wli3ZRpKwugJK2EngaLkY2mdaahY42AYq6pPkydgq8uALsEmvLMICIsfu8RjdY1F H1/RRtX9wcXAVz+QScAIOAwKHqwMX0Q= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720092298; a=rsa-sha256; cv=none; b=4yDzNtjRn27BwPURO3+S+INgWi4IWGT2f+VN245k6wyRmf6Uh7eSuQTKBxT/IV9jFrnOHP NJ3y/Hdb3e35ENKdC92cuOG4mRQfrsusvW2CIs7JHu9uTSsDGHG3ttFy7MlVeOSOTnoqDF 441M9GCA5abgljW0Vjz+Cj6zWV9/a6A= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=linux.alibaba.com header.s=default header.b=xYOmMn7I; spf=pass (imf02.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.97 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=linux.alibaba.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1720092316; h=From:To:Subject:Date:Message-Id:MIME-Version; bh=JZrqpGDPBtXPHidCmEmeC3ZJ9JHyMn+COl4BmjCUDX8=; b=xYOmMn7IqJwvjMHo5oxtjlCcNa7bkIPKQaJrYJYo0D+ZB0r6Y8Fv2JVgI52Om5OHZzWVLUOFLn4d+ch/xJqEo3gZLS0FX8CKQq4krCi+dJsJfbuWinj8GNMg1k9U9wHfeKbUkT/NHhLFyl1EDL0UxSMgqMMiE7O5qbqA5Cb4EHw= X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R331e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=maildocker-contentspam033037067111;MF=baolin.wang@linux.alibaba.com;NM=1;PH=DS;RN=17;SR=0;TI=SMTPD_---0W9qjOyu_1720092314; Received: from localhost(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0W9qjOyu_1720092314) by smtp.aliyun-inc.com; Thu, 04 Jul 2024 19:25:15 +0800 From: Baolin Wang To: akpm@linux-foundation.org, hughd@google.com Cc: willy@infradead.org, david@redhat.com, wangkefeng.wang@huawei.com, chrisl@kernel.org, ying.huang@intel.com, 21cnbao@gmail.com, ryan.roberts@arm.com, shy828301@gmail.com, ziy@nvidia.com, ioworker0@gmail.com, da.gomez@samsung.com, p.raghav@samsung.com, baolin.wang@linux.alibaba.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v3 07/10] mm: shmem: support large folio allocation for shmem_replace_folio() Date: Thu, 4 Jul 2024 19:24:56 +0800 Message-Id: X-Mailer: git-send-email 2.39.3 In-Reply-To: References: MIME-Version: 1.0 X-Stat-Signature: 8pigoymmby7geoixumf994asqo7b18ri X-Rspamd-Queue-Id: 0CB3F8001B X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1720092322-588683 X-HE-Meta: U2FsdGVkX18CvpxURsavzIN/9olhZxZ3EXTMsxr0Qe5SZVU/39ot5sq5bph9FmqSzIOqjrb+v0bzG9i8uT3i2yKOOvUgx4U3H7QXnJJgZ3f2sAiBuSFS3056YizuIUKvZp7cfJWLRCzZkPVs+Q4//SmCoUzDst5h2yILVQqtliGmw4vGPvrz9l+9AefKNw3cJeWWJ/Chk0jx42TDheIIX5qGNmb5lh5xy//Kj1haAi+1PW7EB7+jT0QvrUYe2Y1YaLkXjl+yAk2W5d9B7K3W4/IxQvLfdhW+ESFXsYWxlapXWpIGTVzTsQ2mPv2WryL2OJUF5hbyhx0bMNgpTsqfP8wXdFiKxGmGHXKF2KSkr2sCPR9q8Q1m/1288ZDt/fbqURCN+/URhUVxZJCdo1MH3L7lWfoo27uD2QmcK7row5+qmKjbg6SrT265zGYvRCL7a7Mc+NWeC5OJqlLoJCoBSeOfiRDU7f+YTt6zYHNiP6sPgw8lMNUPhyMLWBlG8jUY8QbhFHNQwkYfNKrzuyPWMdYqax2LmrKvB475wrIZZg4lOgUShZe1oIurAivMOyZpX4Ckh+GVkJN7JyUTH9194WqrFvkPUHB6QXvBHlfnMHuhRC6J4nLpzDgkK0HaVcMn9iAiEv0/3RVv9e+1G8KTcuYlbq8ad5F9pkVK0GVCL5Vy5hr4D0kiz9rpHt3A4APLArsakRUuO7bfNTRZkYFLGIhGZr6hxeBeODzRs9c0QmWgy33dxpWvygnx32oucOCP2SHZbIV8rm+VMabWmVCwimSWhOG+D1F0f/qkhJVpoV2uTyg51Ls17M8MBhoi3iqUhZjoCr9PXOnWTFkEu0AQ5oCXNz7AP9zGGAxf84NeAAOLDnVQ8x1m02N+EAXUVMquF4bXLd1mVIJVL0CVzwxO4w1Yrdw5qvYDIX0l5MP6T/fT/dhbNTNARI/yYtC5X1wTRiGZlNrjW7A8oeGvRn3 hOnMzLCZ N8Os8gkEK23ylKHNobu2E95m9GXpPOeqI9gezeSd8nUfyiNIbSEW7Airq1WB8Jkk0r8kTjKRE12sWZZQKilDXR6KCH0BENC/kqtZMTX8aAxr5wqQ85R/ZXiLHhv3kpWwbMtAw+qPki+3XB06bpgwEu547HQb6D44a/4brSQ2DCNedcKtNXvpFa3sXHd7J8UmOmRsmJ/+4E4QGBp2tJVe9E/YCl28MiM6LFRvRljN2P0SuZnACBVYpkwXhAPTWvopbyT64CSniEGblcY6vwk/FUdJRqIJuJ1qmBuSK X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: To support large folio swapin for shmem in the following patches, add large folio allocation for the new replacement folio in shmem_replace_folio(). Moreover large folios occupy N consecutive entries in the swap cache instead of using multi-index entries like the page cache, therefore we should replace each consecutive entries in the swap cache instead of using the shmem_replace_entry(). As well as updating statistics and folio reference count using the number of pages in the folio. Signed-off-by: Baolin Wang --- mm/shmem.c | 54 +++++++++++++++++++++++++++++++----------------------- 1 file changed, 31 insertions(+), 23 deletions(-) diff --git a/mm/shmem.c b/mm/shmem.c index c1b6cdc61619..1718da7eefab 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -1889,28 +1889,24 @@ static bool shmem_should_replace_folio(struct folio *folio, gfp_t gfp) static int shmem_replace_folio(struct folio **foliop, gfp_t gfp, struct shmem_inode_info *info, pgoff_t index) { - struct folio *old, *new; - struct address_space *swap_mapping; - swp_entry_t entry; - pgoff_t swap_index; - int error; - - old = *foliop; - entry = old->swap; - swap_index = swap_cache_index(entry); - swap_mapping = swap_address_space(entry); + struct folio *new, *old = *foliop; + swp_entry_t entry = old->swap; + struct address_space *swap_mapping = swap_address_space(entry); + pgoff_t swap_index = swap_cache_index(entry); + XA_STATE(xas, &swap_mapping->i_pages, swap_index); + int nr_pages = folio_nr_pages(old); + int error = 0, i; /* * We have arrived here because our zones are constrained, so don't * limit chance of success by further cpuset and node constraints. */ gfp &= ~GFP_CONSTRAINT_MASK; - VM_BUG_ON_FOLIO(folio_test_large(old), old); - new = shmem_alloc_folio(gfp, 0, info, index); + new = shmem_alloc_folio(gfp, folio_order(old), info, index); if (!new) return -ENOMEM; - folio_get(new); + folio_ref_add(new, nr_pages); folio_copy(new, old); flush_dcache_folio(new); @@ -1920,18 +1916,25 @@ static int shmem_replace_folio(struct folio **foliop, gfp_t gfp, new->swap = entry; folio_set_swapcache(new); - /* - * Our caller will very soon move newpage out of swapcache, but it's - * a nice clean interface for us to replace oldpage by newpage there. - */ + /* Swap cache still stores N entries instead of a high-order entry */ xa_lock_irq(&swap_mapping->i_pages); - error = shmem_replace_entry(swap_mapping, swap_index, old, new); + for (i = 0; i < nr_pages; i++) { + void *item = xas_load(&xas); + + if (item != old) { + error = -ENOENT; + break; + } + + xas_store(&xas, new); + xas_next(&xas); + } if (!error) { mem_cgroup_replace_folio(old, new); - __lruvec_stat_mod_folio(new, NR_FILE_PAGES, 1); - __lruvec_stat_mod_folio(new, NR_SHMEM, 1); - __lruvec_stat_mod_folio(old, NR_FILE_PAGES, -1); - __lruvec_stat_mod_folio(old, NR_SHMEM, -1); + __lruvec_stat_mod_folio(new, NR_FILE_PAGES, nr_pages); + __lruvec_stat_mod_folio(new, NR_SHMEM, nr_pages); + __lruvec_stat_mod_folio(old, NR_FILE_PAGES, -nr_pages); + __lruvec_stat_mod_folio(old, NR_SHMEM, -nr_pages); } xa_unlock_irq(&swap_mapping->i_pages); @@ -1951,7 +1954,12 @@ static int shmem_replace_folio(struct folio **foliop, gfp_t gfp, old->private = NULL; folio_unlock(old); - folio_put_refs(old, 2); + /* + * The old folio are removed from swap cache, drop the 'nr_pages' + * reference, as well as one temporary reference getting from swap + * cache. + */ + folio_put_refs(old, nr_pages + 1); return error; }