From patchwork Fri Oct 18 10:48:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 13841573 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AAA44D2FFEF for ; Fri, 18 Oct 2024 10:50:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 411766B008A; Fri, 18 Oct 2024 06:50:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 399CF6B008C; Fri, 18 Oct 2024 06:50:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2155A6B0092; Fri, 18 Oct 2024 06:50:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id EFC0A6B008A for ; Fri, 18 Oct 2024 06:50:42 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id CFA00121A05 for ; Fri, 18 Oct 2024 10:50:31 +0000 (UTC) X-FDA: 82686404220.01.4ABD913 Received: from mail-qk1-f176.google.com (mail-qk1-f176.google.com [209.85.222.176]) by imf02.hostedemail.com (Postfix) with ESMTP id 0198580009 for ; Fri, 18 Oct 2024 10:50:19 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=bIe7KXk3; spf=pass (imf02.hostedemail.com: domain of usamaarif642@gmail.com designates 209.85.222.176 as permitted sender) smtp.mailfrom=usamaarif642@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729248494; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=q2+OQEgh0fCG5oZB9bP27eMnsujYhSchw3YP6zGdLsg=; b=wysqw2Qt2QaDDbSjbp2Cy+nIyOobUrlmh4S5umGxufpxlWWtu+Y6ea0VbtI71dyu79qd3g txmVJp2FLlYx2JeS2WlmFdMAdnsRCAV/fq9kOjCjEMcx+Nn2xiJFTBMaKMdVPQP6ssXANz QPLWSwHfW2zn02CWavTV/zr2z8qUUJk= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1729248494; a=rsa-sha256; cv=none; b=xwc1rcwdNt2tbnR7OdHjldtplQSuup7/AYrS4s4OyZY/Q37If5/TDRC38Uc32AoeZdrhJt CTZMAn/B/WKV3GEP+UEkzahstTMU5ZQEOgwlHo5Gf+yXRqE2iUnSubdjw1V0EfhcgpSohO rjl36kyUlZdWXhh5iCneRG+9c15F0Bo= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=bIe7KXk3; spf=pass (imf02.hostedemail.com: domain of usamaarif642@gmail.com designates 209.85.222.176 as permitted sender) smtp.mailfrom=usamaarif642@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-qk1-f176.google.com with SMTP id af79cd13be357-7b155cca097so75147085a.1 for ; Fri, 18 Oct 2024 03:50:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1729248640; x=1729853440; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=q2+OQEgh0fCG5oZB9bP27eMnsujYhSchw3YP6zGdLsg=; b=bIe7KXk3owCrKMi2HzKtZNmm1DFBR8qy+TKCbRjc3Ve1OXJDmUy7oAQqPmf1RCvWrV 2PcNwLuj8q5ZZQJeUF4/ay9lyP6ewWgGGLo8ZEwpUbHFZlxy6B+oA//wZx3p6yKfBJd6 Bf8VzDFuDCy+FJ4bWuEIm5sI5TPGZpDKCl3/rkr6WxRSmKx2V3i5diu+tEeT+r8MKPSM kbd4p99WcucyDd6omEnMMm3ZxoQCEXTiRKJKoMrDbDHlZcfbw/7U7DZtcH0OI/a6jeFr YoOTaT0e8CyflS+0Ng1Z1yi4EKiMKFJdise3FL5RkkbYIgDm+DJ/jsDHYVxtQrRnnhFl Wfcg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729248640; x=1729853440; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=q2+OQEgh0fCG5oZB9bP27eMnsujYhSchw3YP6zGdLsg=; b=pKlNmPdail4nsERZs0pD28lCf9QHmTtHJ3Zn/XPcTF7FJMCIeJGhqo/C//meXx0jST rQOQIE3QFfhezPen3dPHpTdnc9/7wO5ODwnIavb0o8CPBuH3l8skYt++dcmk/6w+r6c+ M+VfVUepDkXAq2TDjrX9PIKF+PEbIEb+EOkRs/R5ab/70PGKibHFnSPUJWmlfAp/dZTJ dwQUmamDFg+qhQPwgDX9xp7lg9C83ENI0xBZmQLpY8LnLwKhLGTKPxlOGRpY2nWeyn8/ +hqA91sqHKYQ+S+z44k1nPmwHIhoUOVsMhodtVDlz8/PbCL2VV7/ZYGzO0Ttam0Xhf0k ROhA== X-Forwarded-Encrypted: i=1; AJvYcCWhvw4tziYAIIcfhuRY7OsEFkef3fOLsdOhn+HFjVLuRjE5ARntYFLos710QcX2FW+Hm46ON4XW8A==@kvack.org X-Gm-Message-State: AOJu0YwS3lQjSVoTh1CBlvbHKB4JE+KumisjZHHCRvFliOgr4SeIZnnP +kuocug6ThB2QfglvcUxdZpOzQdAseLpj7DFo5laOPaCJ/1ssJdL X-Google-Smtp-Source: AGHT+IExnPa9OX7aimLs3HML3KX6QH8HCRkuDhHElKsmz6WwfmgBG4EaxhA/mKXEbRMU3PnKMeQz8Q== X-Received: by 2002:a05:620a:4708:b0:7b1:3c19:a665 with SMTP id af79cd13be357-7b14d51d341mr914254285a.19.1729248639792; Fri, 18 Oct 2024 03:50:39 -0700 (PDT) Received: from localhost (fwdproxy-ash-017.fbsv.net. [2a03:2880:20ff:11::face:b00c]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7b156fe5b6fsm57894985a.82.2024.10.18.03.50.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 18 Oct 2024 03:50:39 -0700 (PDT) From: Usama Arif To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: hannes@cmpxchg.org, david@redhat.com, willy@infradead.org, kanchana.p.sridhar@intel.com, yosryahmed@google.com, nphamcs@gmail.com, chengming.zhou@linux.dev, ryan.roberts@arm.com, ying.huang@intel.com, 21cnbao@gmail.com, riel@surriel.com, shakeel.butt@linux.dev, kernel-team@meta.com, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, Usama Arif Subject: [RFC 1/4] mm/zswap: skip swapcache for swapping in zswap pages Date: Fri, 18 Oct 2024 11:48:39 +0100 Message-ID: <20241018105026.2521366-2-usamaarif642@gmail.com> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20241018105026.2521366-1-usamaarif642@gmail.com> References: <20241018105026.2521366-1-usamaarif642@gmail.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 0198580009 X-Stat-Signature: hubd9w3oxk8t4g8efcazsausz6qpw9jd X-HE-Tag: 1729248619-154767 X-HE-Meta: U2FsdGVkX1+vVQqnYse7PQ4rEtSkpWmqmpHYHKuU0gRfrQuZ95NwzNj6bmSSTEySsOUIPoB2pnEpvaFk7U0tLHU/FdFSNzAkjXiCy04PnknCFf0NIjPN7/ZC9E7AZm8V5bJK9obPnNJ+IfKuDRXI05NFMyKQGGLgJXmL09FIiB0cJyflHviGOMRfHrplzCK6WpWHP9GEquhZUL8+XaGbr3Dn0CDDMgMIW1s4lN7zjPraY/+ENdUusZ39Cozqo9exRocB0oz/3vrR5SynanVltbi/4XIcZF8CMCYEka4ECmCl6oY1ny8PpKYvCXR1+8iItXf1d+g3xs9PqYDz7HcWoxMGirtU2yakRqjVELxeddXCdwXjDuseD8zr/wGGbvTyRgpnI1eC9ihXSrswpWK/My5dCq/KX65YF+kCDPoeGyQRkyqezwS5XsYzrTJaQ736QIUjGwUD+Gt/49HKEDISdjyikznvkNqxCux8AXr71FGvN0iaY199byoaZMX0eOD0eCjXgOF8cIQZUWwBfCutB6f6wG3DtyNWpYFTB3i50Z1jrIJwtcOYV07+7ww712DH7aNanKcXJG+Li2PUrP2obKuRp+GN9Iq6kVdfzqehTrkBic5G5o7VH2PZ4/mZZO43BD3uCVKGFNS50O0VFjPfmRyp9OoDYd+4Q8BFm9k7EltwhbWVjyA+PYZEMx6eOsl2UC9t2KitcBv/yUHPYULti+Mo4NqxvCYLXfIeTWj7GeoVImG/viGuvHRt6jfnNbALIN17Vk8J55xxEXpRkCOk6i24hoo+FCzjJKNEFk92miNfqSpspRWJi34+FJj47urzZ2ZlR66OjM+AB4YAc1DDfVaiBxOQ1O8s376LEjnWTpWrSTqNop3xTHpsHcqdsu9RnDJcpVDY2j0+8u53ZLCcPObkk4RrJRWPO8jghf6Mj6gAD0eSnkiDigPkmc8Z7uOJfOm9TGsVXLcH4IX0aSF GN+p4VZP tdvwMdu/uEKweRbT3NIifFMCVCJJ+LnpMIdMwqGRtnqYbuo/kUsMOYjIRqitMxrOhhqJ8cXyvl7dXUXVbvqNpks0uLakQCrdbnC3smx5gbthG8enihzAtJKbsH3h+ZTNy0fd6ZpNjjfFSw/5L9FNLuxOhA1nvIuyUMKUKfVfCF4OXUlXFRzxQCXFHJvG9m9g1QNq24BLmPZyIJH27/i9uUHuoIeL1yJ/XEXW3wuc7dj3UU+v6ZnSEnYfmPoUz5sRYd+G1MoMmlDPbIYi4VAKtSDLEikWfmAMX7w01mauFdurLaPT3nNdB/7/GgmSgfDtd70QToUFZ1blMKDxRvtqPe33tLNuYQrbTQmTZMKlfj9Cj0qS8mZHbW5FiuwwQ6iRNCc8loNm2g9cY3OLYwwMw1nSvbvcgqjca44vB X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: As mentioned in [1], there is a significant improvement in no readahead swapin performance for super fast devices when skipping swapcache. With large folio zswapin support added in later patches, this will also mean this path will also act as "readahead" by swapping in multiple pages into large folios. further improving performance. [1] https://lore.kernel.org/all/1505886205-9671-5-git-send-email-minchan@kernel.org/T/#m5a792a04dfea20eb7af4c355d00503efe1c86a93 Signed-off-by: Usama Arif --- include/linux/zswap.h | 6 ++++++ mm/memory.c | 3 ++- mm/page_io.c | 1 - mm/zswap.c | 46 +++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 54 insertions(+), 2 deletions(-) diff --git a/include/linux/zswap.h b/include/linux/zswap.h index d961ead91bf1..e418d75db738 100644 --- a/include/linux/zswap.h +++ b/include/linux/zswap.h @@ -27,6 +27,7 @@ struct zswap_lruvec_state { unsigned long zswap_total_pages(void); bool zswap_store(struct folio *folio); bool zswap_load(struct folio *folio); +bool zswap_present_test(swp_entry_t swp, int nr_pages); void zswap_invalidate(swp_entry_t swp); int zswap_swapon(int type, unsigned long nr_pages); void zswap_swapoff(int type); @@ -49,6 +50,11 @@ static inline bool zswap_load(struct folio *folio) return false; } +static inline bool zswap_present_test(swp_entry_t swp, int nr_pages) +{ + return false; +} + static inline void zswap_invalidate(swp_entry_t swp) {} static inline int zswap_swapon(int type, unsigned long nr_pages) { diff --git a/mm/memory.c b/mm/memory.c index 03e5452dd0c0..49d243131169 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4289,7 +4289,8 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) swapcache = folio; if (!folio) { - if (data_race(si->flags & SWP_SYNCHRONOUS_IO) && + if ((data_race(si->flags & SWP_SYNCHRONOUS_IO) || + zswap_present_test(entry, 1)) && __swap_count(entry) == 1) { /* skip swapcache */ folio = alloc_swap_folio(vmf); diff --git a/mm/page_io.c b/mm/page_io.c index 4aa34862676f..2a15b197968a 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -602,7 +602,6 @@ void swap_read_folio(struct folio *folio, struct swap_iocb **plug) unsigned long pflags; bool in_thrashing; - VM_BUG_ON_FOLIO(!folio_test_swapcache(folio) && !synchronous, folio); VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); VM_BUG_ON_FOLIO(folio_test_uptodate(folio), folio); diff --git a/mm/zswap.c b/mm/zswap.c index 7f00cc918e7c..f4b03071b2fb 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1576,6 +1576,52 @@ bool zswap_store(struct folio *folio) return ret; } +static bool swp_offset_in_zswap(unsigned int type, pgoff_t offset) +{ + return (offset >> SWAP_ADDRESS_SPACE_SHIFT) < nr_zswap_trees[type]; +} + +/* Returns true if the entire folio is in zswap */ +bool zswap_present_test(swp_entry_t swp, int nr_pages) +{ + pgoff_t offset = swp_offset(swp), tree_max_idx; + int max_idx = 0, i = 0, tree_offset = 0; + unsigned int type = swp_type(swp); + struct zswap_entry *entry = NULL; + struct xarray *tree; + + while (i < nr_pages) { + tree_offset = offset + i; + /* Check if the tree exists. */ + if (!swp_offset_in_zswap(type, tree_offset)) + return false; + + tree = swap_zswap_tree(swp_entry(type, tree_offset)); + XA_STATE(xas, tree, tree_offset); + + tree_max_idx = tree_offset % SWAP_ADDRESS_SPACE_PAGES ? + ALIGN(tree_offset, SWAP_ADDRESS_SPACE_PAGES) : + ALIGN(tree_offset + 1, SWAP_ADDRESS_SPACE_PAGES); + max_idx = min(offset + nr_pages, tree_max_idx) - 1; + rcu_read_lock(); + xas_for_each(&xas, entry, max_idx) { + if (xas_retry(&xas, entry)) + continue; + i++; + } + rcu_read_unlock(); + /* + * If xas_for_each exits because entry is NULL and + * the number of entries checked are less then max idx, + * then zswap does not contain the entire folio. + */ + if (!entry && offset + i <= max_idx) + return false; + } + + return true; +} + bool zswap_load(struct folio *folio) { swp_entry_t swp = folio->swap; From patchwork Fri Oct 18 10:48:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 13841574 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0C9AFD2FFEC for ; Fri, 18 Oct 2024 10:50:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 82FDD6B008C; Fri, 18 Oct 2024 06:50:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7E12A6B0092; Fri, 18 Oct 2024 06:50:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6A7416B0093; Fri, 18 Oct 2024 06:50:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 4EDA46B008C for ; Fri, 18 Oct 2024 06:50:44 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 2F8A8A18AF for ; Fri, 18 Oct 2024 10:50:22 +0000 (UTC) X-FDA: 82686404472.10.8074FBF Received: from mail-qk1-f171.google.com (mail-qk1-f171.google.com [209.85.222.171]) by imf30.hostedemail.com (Postfix) with ESMTP id 8FE908000A for ; Fri, 18 Oct 2024 10:50:21 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=b6XNjzLC; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf30.hostedemail.com: domain of usamaarif642@gmail.com designates 209.85.222.171 as permitted sender) smtp.mailfrom=usamaarif642@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1729248534; a=rsa-sha256; cv=none; b=jUsInVmpiVZQhTu3oUtDw1qAGeVZfNM3geN2pQyua4hGCd9Hro3vGE5UfnI8M7P021vVHb U70yt9bV7fGPnnBuXbyX72Y/11I0UKyp18+v/QpJRZGZEADbZ6syuZ5q+vBKVEIwS070HW e+lMT1/4QXk6jgVwGCGRA8XM1DM4CRc= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=b6XNjzLC; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf30.hostedemail.com: domain of usamaarif642@gmail.com designates 209.85.222.171 as permitted sender) smtp.mailfrom=usamaarif642@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729248534; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=I0lh5Dj84nRQ80twxUOjXO55WZ5GvafoDcM5bsLvOt8=; b=Y1/Oa7Ag6biC/Yyj47NJ0Ed9w1PR6jRpynmfhobL+napW/zW9x8khjWfmeC6EIudg5+qk/ HIoKXzk0QV1QsCchCEnlDsfqdjcMbf5Qxv97jD7id/XRShZWHrsSYiWEAVUUqtRBDrETcI rwzCp8EyxTmYaW9ApigST3lgj9FtJz0= Received: by mail-qk1-f171.google.com with SMTP id af79cd13be357-7b14077ec5aso262723885a.1 for ; Fri, 18 Oct 2024 03:50:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1729248641; x=1729853441; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=I0lh5Dj84nRQ80twxUOjXO55WZ5GvafoDcM5bsLvOt8=; b=b6XNjzLC09IHpYZYIIoQ3T8lAVcoMRmor9rzEufITjvlqd45o+axw0TnjxXd/nTzZb o1sfdhE5Cjca4f0HygpjtKS7lfh710mUcHrcLQva4tSV5Q8u3FUxWTM5gsfL0jxRnKsz DnXjuJM+s3T703gR6s6BzuoYwWfaNveCOmpW0Io1M+obVpwBg7n/GQXXr+iFXZrJEGJN qLmqRyVTUqjFCgs43Ycub8fWyNme3t537qs3V7HSBOmL0RMATzI0SjWFAmEmqlXWFrmz 8JDXPcVMfqM1aZFqPbD+becWSXlFfGU85SPen/xZrp79rOMaJZGIMoTDQKVNCyvqs6yv 0omw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729248641; x=1729853441; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=I0lh5Dj84nRQ80twxUOjXO55WZ5GvafoDcM5bsLvOt8=; b=O1pn95y7yNC2XzKHs0ZOOrLeZNouE5uNQL6vNQ5cMGR4hqAd761gXrNK9gifQAuL31 lkRoHcfSLQ2NwfMEeWcn4BnzHKWhyKe8MMj1ljHgSr85a8vu3cTWyKkAobqwicJyT5qh y7x/ciH/a/mFRBH0AWkxzxxJS8CehMteDGhZDuoFqx5feZSl4KO6h1P688NFZSWQZx27 jAB2vVlN9L+WVpJW+I98bnOTgjUrSy6z6HqKrnLsPx3BYOPiO+r1V2dct2mgU9+ispzZ Lw9hP4X7sLEaKNqd++bUQsy/GIbrPgrWsfYeWfPPSt1gze9i+1K/gkPxOZ/QM66Zjhck tn7A== X-Forwarded-Encrypted: i=1; AJvYcCVT7hurGcZVCYNM3zPED2lDAlllEis4vc3fiKGoJD7U6bg1vjilPULlanmzGjYWyhbtQrSXe6VY2A==@kvack.org X-Gm-Message-State: AOJu0Yy1FflbZVRPaCQ2mA3gSwHPlSMNIGm8RNCW3LxST+iQ/r7plSFV ayOL3nJenbptYKx8UIwXpdrbNRCrxsd5Ke/dhybWXYWlrhMTREVF X-Google-Smtp-Source: AGHT+IES+vksoooguMP9yHq2t67dioxSjxiMZLvUVEe1xr8iSRA7X5ppyMkALt+mUyKvRjjB7jAf4A== X-Received: by 2002:ac8:5e0f:0:b0:460:38f5:8753 with SMTP id d75a77b69052e-460af182969mr26970431cf.26.1729248641313; Fri, 18 Oct 2024 03:50:41 -0700 (PDT) Received: from localhost (fwdproxy-ash-018.fbsv.net. [2a03:2880:20ff:12::face:b00c]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-460ae96d3ccsm6189371cf.32.2024.10.18.03.50.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 18 Oct 2024 03:50:40 -0700 (PDT) From: Usama Arif To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: hannes@cmpxchg.org, david@redhat.com, willy@infradead.org, kanchana.p.sridhar@intel.com, yosryahmed@google.com, nphamcs@gmail.com, chengming.zhou@linux.dev, ryan.roberts@arm.com, ying.huang@intel.com, 21cnbao@gmail.com, riel@surriel.com, shakeel.butt@linux.dev, kernel-team@meta.com, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, Usama Arif Subject: [RFC 2/4] mm/zswap: modify zswap_decompress to accept page instead of folio Date: Fri, 18 Oct 2024 11:48:40 +0100 Message-ID: <20241018105026.2521366-3-usamaarif642@gmail.com> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20241018105026.2521366-1-usamaarif642@gmail.com> References: <20241018105026.2521366-1-usamaarif642@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 8FE908000A X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: ybffj51thopsw45rac1r8wnqyghbyyzu X-HE-Tag: 1729248621-52934 X-HE-Meta: U2FsdGVkX1914HaosCU8jvKub7kaHA0AInneW78flKmn4hd2wCwTWv5JHS8s4+L7Cy4EorKfckJXttmryQyBY8cfuGKc0YZBkRQN8qsCV8x4vx22/DIMmmQyTTh7aodFmyOBPP9Xng9hT/WOJQSBv0vY7udxEZ09UJRbxlUdEQbhMq2fHI3Erq1VCFKPHLcemAbbld4rSvTVv9H2oVrmZVSxo1Ygn2SRa/a7kFG7QWp8hAgnwuJgOHCt4y7kSw0uzXSgusFGSH69917P+yxBwnhQ5kOurGbZOMEBW6eOCeUnMJ0LTG01YDi/1fxKpgaOD1M+ewcIVbEeWkH+9GOYMab6PzoGSL8oqOC6QBR/4TEYP5ZchA2K/JbFE0xH6r5KagWFfsdrZeFd9jD0pHaPp8d3RVgjkl2KWIRu4Y5mXyvMhL+WiWvvQAbr19G8fVl1rGGK1ECPg7TDgYQrkCmdd2LWqR710ZE/DsMPNfmagMB6bY1ini7E5THH9exYlRWQDSs5+xvWRcf1oozLm6Kp07olOmK9Cigt2h1p0xlM3ClRfx0POX1x4hdAiZ6X1AKkQhAKmjFpDXjhb4Qa0GnX7sOG1pylf/CvSOxqsW+OYQFMJaJuVBeI+gQR/8d1h3ekcABxDb8OPoHcWggrqjh1PNRpUikAM6Cu1qLC8b0DsdQQPxT0SIFiEhjWGYMml8+jnXyYNHSfrHfGyHcd2Y/5qpp3n0GC7yHe19JGE13308APwfl5ILw3vzmSFrTALD0hbmktf9WljMgdh2gTrZ+SGttCT0uWyw+ppbrtx/j8ETrAZFzgt0jEKM5uDdbENPfdbBUSXaj4HJhuVwigJ8b50C3CJUA4dHDatbmAS4zzPxPuBMzhxHGfZo/ld0v3TamqCP5klEJtRU+j0D1DKhgHN0LQ8RLCEOmXAP6JE7RwElpZmMopsEBdDGL+yoJ12ynadt/5pSjsJR65/q9CYjZ sRpoU+/K 6/IJJz8JZhpvw18wV0CHV02dla75iWCTq0418g+PFHTJgRQp6Xy9ikA3zr5eDI/AomnjrRDJlziPWmsv2Mh1GZUun65SXxvmrGisf77U41X3kgdGb/FGaePYZheIG+leMmyK0T+IENkSaWP/M337sCa5udvww3lnzVJ4DrxHxqkFaxgd4AyM8+lmHD9B5IVqm54v6eCBLE+3yrnb9LfCJWo6eB9/Va0vAI+GtVJCwanFdI5FPTLxe1SGLAcHhiz+igVk2yY2UMiPVjmbLAvOFloR7vn1VWSdqQ7oRVqbEumGvc1Uvt6jzeniWliPb6QvUGR0kPE1JcQTJDdRcM0Gw+1ELRaVQTopmoiA70zucF29DkRj+Yq+OJ5CbZEVpoxHgodixd/8u7SQSaMnrMmHk3DKG6z+tI2zfgae4prOzg95m0CImVtjogl20MQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.003328, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This is a prerequisite for zswap_load to be able to decompress large folios. zswap_load will iterate through each page in a folio and decompress into it. Signed-off-by: Usama Arif --- mm/zswap.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index f4b03071b2fb..9cc91ae31116 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -953,7 +953,7 @@ static bool zswap_compress(struct page *page, struct zswap_entry *entry, return comp_ret == 0 && alloc_ret == 0; } -static void zswap_decompress(struct zswap_entry *entry, struct folio *folio) +static void zswap_decompress(struct zswap_entry *entry, struct page *page) { struct zpool *zpool = entry->pool->zpool; struct scatterlist input, output; @@ -982,7 +982,7 @@ static void zswap_decompress(struct zswap_entry *entry, struct folio *folio) sg_init_one(&input, src, entry->length); sg_init_table(&output, 1); - sg_set_folio(&output, folio, PAGE_SIZE, 0); + sg_set_page(&output, page, PAGE_SIZE, 0); acomp_request_set_params(acomp_ctx->req, &input, &output, entry->length, PAGE_SIZE); BUG_ON(crypto_wait_req(crypto_acomp_decompress(acomp_ctx->req), &acomp_ctx->wait)); BUG_ON(acomp_ctx->req->dlen != PAGE_SIZE); @@ -1055,7 +1055,7 @@ static int zswap_writeback_entry(struct zswap_entry *entry, return -ENOMEM; } - zswap_decompress(entry, folio); + zswap_decompress(entry, &folio->page); count_vm_event(ZSWPWB); if (entry->objcg) @@ -1666,7 +1666,7 @@ bool zswap_load(struct folio *folio) if (!entry) return false; - zswap_decompress(entry, folio); + zswap_decompress(entry, &folio->page); count_vm_event(ZSWPIN); if (entry->objcg) From patchwork Fri Oct 18 10:48:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 13841575 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7D740D2FFEC for ; Fri, 18 Oct 2024 10:50:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 12B706B0096; Fri, 18 Oct 2024 06:50:46 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 067406B0095; Fri, 18 Oct 2024 06:50:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DADB06B0096; Fri, 18 Oct 2024 06:50:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id B5A1F6B0093 for ; Fri, 18 Oct 2024 06:50:45 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 20330806A8 for ; Fri, 18 Oct 2024 10:50:35 +0000 (UTC) X-FDA: 82686404598.05.9BE3157 Received: from mail-yb1-f177.google.com (mail-yb1-f177.google.com [209.85.219.177]) by imf29.hostedemail.com (Postfix) with ESMTP id DFD0212000C for ; Fri, 18 Oct 2024 10:50:28 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=WxT9wu5Z; spf=pass (imf29.hostedemail.com: domain of usamaarif642@gmail.com designates 209.85.219.177 as permitted sender) smtp.mailfrom=usamaarif642@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729248569; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=huWuZ5uHt0Ar/gjZBjsyETlUQ5fMZuwQtvUTxwg+3Fk=; b=TiTdddzUCDALyhL3ZwEkYJJkr0P2/h13BVKQyCWeddfx8hromi1B4fmFWo45rbuWpL2lAz X4guYixlgbk7oScx/oolWDjh2XLDM4Kwa+CvU0jhPtKj4dA59eENs4R281xvr8p/l/HpGJ BOdYzCPjTnaJFaHijuoveZdN/dGoI8w= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=WxT9wu5Z; spf=pass (imf29.hostedemail.com: domain of usamaarif642@gmail.com designates 209.85.219.177 as permitted sender) smtp.mailfrom=usamaarif642@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1729248569; a=rsa-sha256; cv=none; b=uRTNMyF+lAOkWIZmXkBxi9R4m6pcW7OwA6QrLlu+LEdp3c8ryrSFvTUrxgJJBNDbHYGNrL Z/sLQuBH+18g2gahME85bb2lMycfFZ+NdW85R5CgN0VseycQukrfVVMVVqJzKlk2LoM9ro LHvx6acUEamlMgu1o3ZyHvDjtZlx3ew= Received: by mail-yb1-f177.google.com with SMTP id 3f1490d57ef6-e290554afb4so1970278276.0 for ; Fri, 18 Oct 2024 03:50:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1729248642; x=1729853442; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=huWuZ5uHt0Ar/gjZBjsyETlUQ5fMZuwQtvUTxwg+3Fk=; b=WxT9wu5ZrfSS60xuPRVcng7vLuSb9L7kHhiLaEdZvAZCduBXNe10yIPO0ThX/OSAAD Iq0WDHUCDYDwKpWvmnVxVY1DpQfWiyKPZ8OLQjwMubyJuH1A9ebpxgedCJqO3A5nMnPd HvvgpOqOIUD4PhBc+WTijll2avzRrVBKh+ZSXYbrc1NPSw+qctkeLpRzJ7E/KJtEhHh7 z9TFZQWmRwYSSyfyJPNQJZtqmN0Hhgp86WSXJOTvrB+QGBDDwpD3Zh2+QHcdfA3x8W0/ KCEwEYarh4/kvoqV9suSjkX0zNnc0tzmR/HjORFad2wGj7HbAwEdt4UoIAMHJaMW8Jgz EVbQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729248642; x=1729853442; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=huWuZ5uHt0Ar/gjZBjsyETlUQ5fMZuwQtvUTxwg+3Fk=; b=WmG5mS9ra2k3oO53wmKz24Ixe7VQsSqIG2R9hqZHXJlR8Rlw2LR5UjN54O5XYwaUWm ZpBZzanD52cy6OolU9HBgCT5E3bDTIMtkv9tEU7svYeD2o90qYpNp88NgYi6NB9YHEAa /1gLFdzlC6hDPMxF6r5Xq7+sIXrdfbYKWo4v4dgzM7PZpyYI7IM9ledOs8Abo02fDCaS i2x7pLPDvEBuzjWVxOsjWTXF+UYTmgkF7mwarh0EOCzOt/W5+6/kjHE10mnPr4hm64ij g5NlV6TouqLiyIP2NYd/9SaIUZFVbU74cJa+2EAd5SJmdfoekPkdp2gHz8244c52Srgv Am3A== X-Forwarded-Encrypted: i=1; AJvYcCVA+lLwMQUXNj3Cywe6Npy3Qwf7+JyU4QKvVDWxkxrq19Oa3IlEGfmZKvYDx1NHbnIYzBpLog67qQ==@kvack.org X-Gm-Message-State: AOJu0YwceUD3h8guktmO8gLUUYgFub4hk/8r8tMGo8mkZZBxyZq585lW hWpKWK1Nz7x78DEmspRmxdqSuBz9b83x90AnNiWM0LefdgDCP2Rd X-Google-Smtp-Source: AGHT+IH7UKKLWj1Gm56LuvSXibMHdJ9P4Zdin80hAy+JrAjlN+8W7nDtiVAWx8XbC+00QhotrmRWLQ== X-Received: by 2002:a05:6902:2b8b:b0:e28:fbbf:7406 with SMTP id 3f1490d57ef6-e2bb12f6a58mr1693699276.15.1729248642473; Fri, 18 Oct 2024 03:50:42 -0700 (PDT) Received: from localhost (fwdproxy-ash-113.fbsv.net. [2a03:2880:20ff:71::face:b00c]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-460aea455f9sm6044301cf.64.2024.10.18.03.50.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 18 Oct 2024 03:50:42 -0700 (PDT) From: Usama Arif To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: hannes@cmpxchg.org, david@redhat.com, willy@infradead.org, kanchana.p.sridhar@intel.com, yosryahmed@google.com, nphamcs@gmail.com, chengming.zhou@linux.dev, ryan.roberts@arm.com, ying.huang@intel.com, 21cnbao@gmail.com, riel@surriel.com, shakeel.butt@linux.dev, kernel-team@meta.com, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, Usama Arif Subject: [RFC 3/4] mm/zswap: add support for large folio zswapin Date: Fri, 18 Oct 2024 11:48:41 +0100 Message-ID: <20241018105026.2521366-4-usamaarif642@gmail.com> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20241018105026.2521366-1-usamaarif642@gmail.com> References: <20241018105026.2521366-1-usamaarif642@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam03 X-Rspam-User: X-Rspamd-Queue-Id: DFD0212000C X-Stat-Signature: 618undr1uppw5c5jawxdc9tspg5cu56x X-HE-Tag: 1729248628-697046 X-HE-Meta: U2FsdGVkX19yax0s90exD38J4X6QdjOU1+oUeWkbcX2pqj8G0CJLgsQZWo75GAW/7ub1G5hXw7KW6v7MOgUfXAEKuovnj+sCtToRoaAPC4mS2ISrlM0YCNQELcyMXfakeU9TUWsiPGPVXk+XS6nfkdWps8r4nVXCPGUcdYzjVizN1p7rBKRpzumQYTezy4/yCs0O+BPT1WTGFVSoPle+dAuv0/Yo8kacgvdmZ/eMtTiYFlnjo7JeNU8iOj5WmeQVz3HUMXmEWFdFjUeFeQy+quMPB99WqkG9dPOlHhBujrLrafgf5HqIJicNFZ1YswWPud8MyyT9ZhLCUr31IqODQq9KHgBuNH4KDY1lbBAo0OuU8FS8i3SHbDPfP9i9oWpJIA0gkp4odA0zvyyUqYoZIXE77WYIAVoCYYyID7RkhW8YaegZBHCXsxVkBwSIeGT2mJr0sLG7DOQjePjyP/VKnBTcpoG1hhZ0/K2SpASKQKZReCa+LXxYxLLlvPf6/K56ImpdVHNVlxzOghVO2u/fMR9/9fmIb+2Zmq/1uChqse0RRzq8Uzy9235csWfOfnsrk23j5XNnEg73aDR1WmU4J7MLQ9LjJbrMj8xBhXDbD3+f8nz17+Qm1KPFoAg2Ap/Da5m4gJCKCNk7aj85Oj0+eOi5nmfFPuIdtpQ8vjmZcWMR5SXgem63Mwm6HgbTqh2UxH8jehQxe1NWo1nr6RnKybDvdc3ye21I4V7egHu+50qvYoTgKQejnyTLLWKZQrOu96tj1iomzQ5rSDK1OYdj6/qi1WIo0irnTW9BOakh8a1q/W64OTHiy06+3N87SidGSTvBRUbwwwZqnG/ndfkIUMe4m5Pz2V52QHuWUD/Dl2hY77hd9UgMpgJa3dhd9qllH5+p/Njok94G5S8u0JeLZvZgwnA1o3I3yUQg57Elci+DtjTgNP6un7j9pNV1y3Be2oP/L3aAQdVk53XVgib qT+SwQB1 22CEaobfZVPmbh4Uxo7gmt8nWF9Db+tQwqKUyO/vHVbGp9+SQtkPHtGzygveh/xmJuJG3blx4udMCLcChjZGZu1vrfSq3pvpCRcyKqPNl/iFKYdXiT2d/03JDSATGb5pHf/2Zd515wdSPt/nSrIr8bn11SJb+jvLvQLcYaZg4tjHmWn2hZDKZll0aJ8LsSJgUM34bUea4Ur2adUa6OooPKIuIfRdgG2dRTyos1cbTjZ5JB+PkXwVGcakC3eZL5utkzPa295MBSK01njctEbl7JFuTbz1igQpy4DNiZmovJinbUsitxURcMWHtSR9BzF3/QRBc9XEz3Danx0cCkVrZ0StzBdCQIkbVFsvidLMpaXA92kMVZn3OSFdSU0MzJVUaMD94 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: At time of folio allocation, alloc_swap_folio checks if the entire folio is in zswap to determine folio order. During swap_read_folio, zswap_load will check if the entire folio is in zswap, and if it is, it will iterate through the pages in folio and decompress them. This will mean the benefits of large folios (fewer page faults, batched PTE and rmap manipulation, reduced lru list, TLB coalescing (for arm64 and amd) are not lost at swap out when using zswap. This patch does not add support for hybrid backends (i.e. folios partly present swap and zswap). Signed-off-by: Usama Arif --- mm/memory.c | 13 +++------- mm/zswap.c | 68 ++++++++++++++++++++++++----------------------------- 2 files changed, 34 insertions(+), 47 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 49d243131169..75f7b9f5fb32 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4077,13 +4077,14 @@ static bool can_swapin_thp(struct vm_fault *vmf, pte_t *ptep, int nr_pages) /* * swap_read_folio() can't handle the case a large folio is hybridly - * from different backends. And they are likely corner cases. Similar - * things might be added once zswap support large folios. + * from different backends. And they are likely corner cases. */ if (unlikely(swap_zeromap_batch(entry, nr_pages, NULL) != nr_pages)) return false; if (unlikely(non_swapcache_batch(entry, nr_pages) != nr_pages)) return false; + if (unlikely(!zswap_present_test(entry, nr_pages))) + return false; return true; } @@ -4130,14 +4131,6 @@ static struct folio *alloc_swap_folio(struct vm_fault *vmf) if (unlikely(userfaultfd_armed(vma))) goto fallback; - /* - * A large swapped out folio could be partially or fully in zswap. We - * lack handling for such cases, so fallback to swapping in order-0 - * folio. - */ - if (!zswap_never_enabled()) - goto fallback; - entry = pte_to_swp_entry(vmf->orig_pte); /* * Get a list of all the (large) orders below PMD_ORDER that are enabled diff --git a/mm/zswap.c b/mm/zswap.c index 9cc91ae31116..a5aa86c24060 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1624,59 +1624,53 @@ bool zswap_present_test(swp_entry_t swp, int nr_pages) bool zswap_load(struct folio *folio) { + int nr_pages = folio_nr_pages(folio); swp_entry_t swp = folio->swap; + unsigned int type = swp_type(swp); pgoff_t offset = swp_offset(swp); bool swapcache = folio_test_swapcache(folio); - struct xarray *tree = swap_zswap_tree(swp); + struct xarray *tree; struct zswap_entry *entry; + int i; VM_WARN_ON_ONCE(!folio_test_locked(folio)); if (zswap_never_enabled()) return false; - /* - * Large folios should not be swapped in while zswap is being used, as - * they are not properly handled. Zswap does not properly load large - * folios, and a large folio may only be partially in zswap. - * - * Return true without marking the folio uptodate so that an IO error is - * emitted (e.g. do_swap_page() will sigbus). - */ - if (WARN_ON_ONCE(folio_test_large(folio))) - return true; - - /* - * When reading into the swapcache, invalidate our entry. The - * swapcache can be the authoritative owner of the page and - * its mappings, and the pressure that results from having two - * in-memory copies outweighs any benefits of caching the - * compression work. - * - * (Most swapins go through the swapcache. The notable - * exception is the singleton fault on SWP_SYNCHRONOUS_IO - * files, which reads into a private page and may free it if - * the fault fails. We remain the primary owner of the entry.) - */ - if (swapcache) - entry = xa_erase(tree, offset); - else - entry = xa_load(tree, offset); - - if (!entry) + if (!zswap_present_test(folio->swap, nr_pages)) return false; - zswap_decompress(entry, &folio->page); + for (i = 0; i < nr_pages; ++i) { + tree = swap_zswap_tree(swp_entry(type, offset + i)); + /* + * When reading into the swapcache, invalidate our entry. The + * swapcache can be the authoritative owner of the page and + * its mappings, and the pressure that results from having two + * in-memory copies outweighs any benefits of caching the + * compression work. + * + * (Swapins with swap count > 1 go through the swapcache. + * For swap count == 1, the swapcache is skipped and we + * remain the primary owner of the entry.) + */ + if (swapcache) + entry = xa_erase(tree, offset + i); + else + entry = xa_load(tree, offset + i); - count_vm_event(ZSWPIN); - if (entry->objcg) - count_objcg_events(entry->objcg, ZSWPIN, 1); + zswap_decompress(entry, folio_page(folio, i)); - if (swapcache) { - zswap_entry_free(entry); - folio_mark_dirty(folio); + if (entry->objcg) + count_objcg_events(entry->objcg, ZSWPIN, 1); + if (swapcache) + zswap_entry_free(entry); } + count_vm_events(ZSWPIN, nr_pages); + if (swapcache) + folio_mark_dirty(folio); + folio_mark_uptodate(folio); return true; } From patchwork Fri Oct 18 10:48:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Usama Arif X-Patchwork-Id: 13841576 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 36C62D2FFEC for ; Fri, 18 Oct 2024 10:50:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B49616B0095; Fri, 18 Oct 2024 06:50:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A9E746B0098; Fri, 18 Oct 2024 06:50:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8CADB6B0099; Fri, 18 Oct 2024 06:50:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 6864B6B0095 for ; Fri, 18 Oct 2024 06:50:47 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 3F66AA18CD for ; Fri, 18 Oct 2024 10:50:25 +0000 (UTC) X-FDA: 82686404766.02.AD0E3BA Received: from mail-vs1-f52.google.com (mail-vs1-f52.google.com [209.85.217.52]) by imf26.hostedemail.com (Postfix) with ESMTP id 3239614000B for ; Fri, 18 Oct 2024 10:50:37 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="KWS/qCBC"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf26.hostedemail.com: domain of usamaarif642@gmail.com designates 209.85.217.52 as permitted sender) smtp.mailfrom=usamaarif642@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729248484; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BRHby0h+uR8HMvD4At3d64Ma8NriZDDBQPHvfHlsfXk=; b=fk6OGp3+Z1Wup+OLpuZ6uKGy+zHUuMviGsdj+6/iTPR8hIiZTi9wvwVllZi1dT1gZybbEe x9wdRTf9u+rYgqsLIce9glCJX3+AsaX0wUXJH3dJju1c2P6QZJn7gnL5MTwe/Ik+Relm4g gaEZtpdzqMneSgF3AzYX+qJxLVDUJzg= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1729248484; a=rsa-sha256; cv=none; b=zbLn3Pd57Fgn3EOsw/8dkbBg+kDFLohpQZFdkj+dUMDOptavyFvx6FIdZ9ujOiZdymEsRW S9/DyBgnoH6LoX4KkWXf/Ly4MmBYiL5boLBWnFoq+9AkPltJ5JVp3RjTqX8z6FA5oKUce7 gPoKV+uP5BWc4ggmXh3wp8XLZv9pgus= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="KWS/qCBC"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf26.hostedemail.com: domain of usamaarif642@gmail.com designates 209.85.217.52 as permitted sender) smtp.mailfrom=usamaarif642@gmail.com Received: by mail-vs1-f52.google.com with SMTP id ada2fe7eead31-4a5c4d8ee99so651558137.1 for ; Fri, 18 Oct 2024 03:50:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1729248644; x=1729853444; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=BRHby0h+uR8HMvD4At3d64Ma8NriZDDBQPHvfHlsfXk=; b=KWS/qCBCdNclE/xu49dXtOfrQvS0ITjd5ZZq1I58zv9n1v567j3B3OW30MF0vKGZqr yVIznX+ujtvUVxhifedIqoBfOnaeWUT5an/Q+Ae4BAGoDkJfdIfzAuwVCKBWAzMa8uA1 4vAp3/qmyfJt3zRVLbr2cUrYS1okd1T3kTBHZWTx/Vd7DhvTL5OTcEvyCNDltjDLKEdv 5K7LmQpNvAH4ZiA5NfwcPoDeAH0HsI8SYG2zI4CdFR7NRQIuiR9YR1qzKksv+9IH9GZH hyF20M7ktXMgpMz+AIrM3DusQrBJyazeCSFA3bs5f+pe5hD/dMf8hlAaPAjHZWXvxwHq lG8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729248644; x=1729853444; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=BRHby0h+uR8HMvD4At3d64Ma8NriZDDBQPHvfHlsfXk=; b=HMthws11Y/jdbMqtukxUEnPTD/jJKZXtRwEeemtxe0tCIi/7wsCSS8clA6ZUnO/0Oa Y6HX7TCtOz4+dqEI1up1p4CG+AB2ZD2fdwOw/HxYU5losZVvzCYzOj8NjlOD3w938RRc tiub6FHqm6Fyg8yTRo9huarPgZXpu7FmluVf5Nqa/ja/fd+GCbggcJF5RsCmQTyZ7+o1 xoqaqExI0jPrpMSIbXwDcs361PbwcjyG5uH6H5J+SZfVlA3d5po6Ke6bssZkG4tosiMb ihhrXZp1etYfZLWA8NMUe7vxa4PTiRIzO01SOs9yHr9P49ovhr+TaW0P1Oyei7Z41Ko8 cspg== X-Forwarded-Encrypted: i=1; AJvYcCUBgZBHmQP8no/P1CrDk44wTcH6fVBQN57Qe8NUMpAdna+m3mLA2P9s4URA3S3jH6oQhr1BaAX1Bw==@kvack.org X-Gm-Message-State: AOJu0YyniRt+EknGzji4NsizNFSvu0REEv5rg82E7hBYDmNcK1zgQLga kHHL5S0d048r84RUGlyG5QTARsVfESGu0OP3gCysdBQd1eyzVtvm X-Google-Smtp-Source: AGHT+IHykbO3DzKF2g7UHtYVZ/jRuN9Qj89t9+anwaQLDnm38LEAsEEjNjDjKrfF3i0TApP/IeNGHg== X-Received: by 2002:a05:6102:d89:b0:4a4:7928:637c with SMTP id ada2fe7eead31-4a5d6aae56emr1935661137.8.1729248644408; Fri, 18 Oct 2024 03:50:44 -0700 (PDT) Received: from localhost (fwdproxy-ash-013.fbsv.net. [2a03:2880:20ff:d::face:b00c]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7b156fa81b7sm58410885a.68.2024.10.18.03.50.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 18 Oct 2024 03:50:43 -0700 (PDT) From: Usama Arif To: akpm@linux-foundation.org, linux-mm@kvack.org Cc: hannes@cmpxchg.org, david@redhat.com, willy@infradead.org, kanchana.p.sridhar@intel.com, yosryahmed@google.com, nphamcs@gmail.com, chengming.zhou@linux.dev, ryan.roberts@arm.com, ying.huang@intel.com, 21cnbao@gmail.com, riel@surriel.com, shakeel.butt@linux.dev, kernel-team@meta.com, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, Usama Arif Subject: [RFC 4/4] mm/zswap: count successful large folio zswap loads Date: Fri, 18 Oct 2024 11:48:42 +0100 Message-ID: <20241018105026.2521366-5-usamaarif642@gmail.com> X-Mailer: git-send-email 2.43.5 In-Reply-To: <20241018105026.2521366-1-usamaarif642@gmail.com> References: <20241018105026.2521366-1-usamaarif642@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 3239614000B X-Stat-Signature: fwghtmck9gn1ks9dq45e7p5tmibe4qcp X-Rspam-User: X-HE-Tag: 1729248637-436543 X-HE-Meta: U2FsdGVkX1+u3TDUhkngb7L4Yd9McJ/QR9cnk0/eHHWB+7pXdWj+/2C4nBgOte4ehOzXYV4GJM31xO0IQuI1sGPGlRX+8bWkKJnUj9WZYo8aGOngzlFQjtYHtxptaotGgdjy3Tjr06oyWix/9n7eOHVc9uR79hfL+8t52gorMc4cMkcjEMV6xlb5VDwzh2G0JteQe5pVBk8H+lWEQIYB3DvtWCjBVhKmtvuaowK+mlMxQAULO9HfmHRhM3i4op/y53SiHM4wK+hn9o3pqnJDa+WSToeTWD7vGwPTGdTNmq6S9l6FRfwkKPQum8u7EVOxMiugxQCrEFqZg7uyWdinUT+nSzpi/qvUM33j99riaAAQ/Y96VG/VACjxOSSqjDekJOqmRYoZIK5VAmuoTRTk1BtyI3lMCuYsjSV/2w8jNeEthMI2nJpvGyU8FVG+Qwn4l0jKRRCW2wEiZCuXGs/DoOySDjas69BbN5/hI7eNdtnKMKtmFR9brCWRHHih4Fyt6SbCDtdkT7k8svqNOaarDNH2tcU1pwprDaJfkJ2wvHUgRb2+3lqvOhbHbWi/8XDRPqBe58uuVyEaDmTNvbDgvjth6/eLoiq+9uppa1q2bL18VZQj/LIjNCON5TQT6qxEoxUy2S8pS9zYLbJfCo148HHES/x1rLnlt1SzcoxxCuP+qUKunX2x7XEhHhK17S7bfvHyKHgfavawjGw1rnWK56yLKdCxuEONWrCh362KJSJwLSr/coynS1UN+DZ1iyUb7xnW3rXzVT/ZImx1Gp7/iMLL5nIecFRDQ/8464OWlA5h8VXRTsvYOmEQYRwuLEVUTwNZLyPmtfFTQZwwZuWa1x7hO9Rjv57gOhJHNQ5ECJWRvLGrod3PmUr1FfEJpHMkEN19CqpovCCiIS3W1v84+qPc+0mojbk/iBW2NVK/2aarI8pGbSWROFEhWYfpdOZfkgVHrQ3dn6s5BD/ZyhH 0KfL3mKF TdqCrsBhwgv/GvIb6pGlQ8u4uOiboa4rYCCj3GlvaCy5DTMjfbEhbUP+l4MZtphW5BnmV8SVacGIYj2WBNJpQVlQvdN/W9/MpgP5pjp4SXdsTLFFkRnVv0bw/LaZyRZXTIO+PKYYCXiYMs/bqc0mSdmG9FrQpbROTjuDsw5mwA7DX8NYqTV/y9E8hY+L0LK353SZRbiQLdrNEOgII3qMkYIeFvfZVFbBSC4jnw4WPUIed3ajgow2PmyJjcVdnCu2wN7dNyiSP2Vze2Tx+w0l6qprbzRsA9EcO4ZchfSTSG6LHlEzwLxxFZnUrO9cHoPy680fSoiZK7ve2pMJ5h1L+FssY8nfp9xLZ+MBEi07qb+ZjarXubkQMSsu8MDRQj79Uda1I X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Added a new MTHP_STAT_ZSWPIN entry to the sysfs transparent_hugepage stats so that successful large folio zswap stores can be accounted under the per-order sysfs zswpin stats: /sys/kernel/mm/transparent_hugepage/hugepages-*kB/stats/zswpin Signed-off-by: Usama Arif --- Documentation/admin-guide/mm/transhuge.rst | 3 +++ include/linux/huge_mm.h | 1 + mm/huge_memory.c | 3 +++ mm/page_io.c | 1 + 4 files changed, 8 insertions(+) diff --git a/Documentation/admin-guide/mm/transhuge.rst b/Documentation/admin-guide/mm/transhuge.rst index 2a171ed5206e..68a9790908b2 100644 --- a/Documentation/admin-guide/mm/transhuge.rst +++ b/Documentation/admin-guide/mm/transhuge.rst @@ -534,6 +534,9 @@ zswpout is incremented every time a huge page is swapped out to zswap in one piece without splitting. +zswpin + is incremented every time a huge page is swapped in from zswap. + swpout is incremented every time a huge page is swapped out to a non-zswap swap device in one piece without splitting. diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 3eca60f3d512..28a275d3107a 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -120,6 +120,7 @@ enum mthp_stat_item { MTHP_STAT_ANON_FAULT_FALLBACK, MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE, MTHP_STAT_ZSWPOUT, + MTHP_STAT_ZSWPIN, MTHP_STAT_SWPOUT, MTHP_STAT_SWPOUT_FALLBACK, MTHP_STAT_SHMEM_ALLOC, diff --git a/mm/huge_memory.c b/mm/huge_memory.c index a7b05f4c2a5e..587f7dd81500 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -612,6 +612,7 @@ DEFINE_MTHP_STAT_ATTR(anon_fault_alloc, MTHP_STAT_ANON_FAULT_ALLOC); DEFINE_MTHP_STAT_ATTR(anon_fault_fallback, MTHP_STAT_ANON_FAULT_FALLBACK); DEFINE_MTHP_STAT_ATTR(anon_fault_fallback_charge, MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE); DEFINE_MTHP_STAT_ATTR(zswpout, MTHP_STAT_ZSWPOUT); +DEFINE_MTHP_STAT_ATTR(zswpin, MTHP_STAT_ZSWPIN); DEFINE_MTHP_STAT_ATTR(swpout, MTHP_STAT_SWPOUT); DEFINE_MTHP_STAT_ATTR(swpout_fallback, MTHP_STAT_SWPOUT_FALLBACK); #ifdef CONFIG_SHMEM @@ -631,6 +632,7 @@ static struct attribute *anon_stats_attrs[] = { &anon_fault_fallback_charge_attr.attr, #ifndef CONFIG_SHMEM &zswpout_attr.attr, + &zswpin_attr.attr, &swpout_attr.attr, &swpout_fallback_attr.attr, #endif @@ -662,6 +664,7 @@ static struct attribute_group file_stats_attr_grp = { static struct attribute *any_stats_attrs[] = { #ifdef CONFIG_SHMEM &zswpout_attr.attr, + &zswpin_attr.attr, &swpout_attr.attr, &swpout_fallback_attr.attr, #endif diff --git a/mm/page_io.c b/mm/page_io.c index 2a15b197968a..477f9d4fc009 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -620,6 +620,7 @@ void swap_read_folio(struct folio *folio, struct swap_iocb **plug) folio_unlock(folio); goto finish; } else if (zswap_load(folio)) { + count_mthp_stat(folio_order(folio), MTHP_STAT_ZSWPIN); folio_unlock(folio); goto finish; }