From patchwork Sat Aug 17 05:09:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Sridhar, Kanchana P" X-Patchwork-Id: 13766983 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 09D55C531DC for ; Sat, 17 Aug 2024 05:09:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7CB5B8D00C2; Sat, 17 Aug 2024 01:09:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 77AFC8D00B8; Sat, 17 Aug 2024 01:09:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 640FC8D00C2; Sat, 17 Aug 2024 01:09:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 46B998D00B8 for ; Sat, 17 Aug 2024 01:09:31 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id B69DF1602B3 for ; Sat, 17 Aug 2024 05:09:30 +0000 (UTC) X-FDA: 82460559300.03.C944C1D Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.15]) by imf26.hostedemail.com (Postfix) with ESMTP id 22E00140012 for ; Sat, 17 Aug 2024 05:09:27 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=oKrnXtOl; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf26.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 198.175.65.15 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723871308; a=rsa-sha256; cv=none; b=KxpPvpYTNS/QSgA5dgnYX212E6lKK7x+0KZ40EHRi+vRKDFReVQVeGp1mPsSUG+eq/fMZ7 YSaR/1IU0E2evETQHkX9SjJAK6D4mluMJA3GS1My6TPVyDzcTNOh3SoF5LcTm1wl77Tk2R ThJ4X+kriQsK/K1fAbEvn4N6TGJHtsw= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=oKrnXtOl; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf26.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 198.175.65.15 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723871308; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=naruCmolxM1VTIhLrUrisUJHcgVK8pYYLgjjkbmUCc0=; b=iQf4mKIx3Qo1ZjU5qcL5NYOXC06yykaDgAYCsxnwvW3VsUaeMo0fJF7cdYNP64iVilYHwJ NOiW3QF0N7WFkUK86acg9ut3VEL1R+vGhpcTRQHIf2kIO3BxbCGIH6yw8gZNh/ehx92A+e 8LJR3KyPhv7L0Pe9L0xmE9iFDcvdcqI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1723871368; x=1755407368; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=8aJTLnzMPmjHtUmt/CSvhY9AKnZREM8rFjycd3MYym0=; b=oKrnXtOlNgZzN/Z2G65xZA45MFO26Jk0kknqB3x1bXy2PXmaTohPXD4O ydzGLP4tn8gDcYKcJe/bMxqOhhgiKOEJG4xJNl5joysBuF5UT8oKv60h7 O7awbUHihYyEzNvvhIXp0YCewIjQVQgR6cynCXSiEtpN6MtWPPiF7IgL/ kH1rJveQjF7JCm1SMJ2T8bjO6zgWLI0ce9ciOov99c5N4eYpChkz2aWjr PFyz5amvuMAyJIk0IZx5bnigXcZ/WTE/iWlWeq4/61xVGOE9aW0c+2+VW BfptlzSZJmzUgxzIPVCLjfReD9UplZuWZmhR77IWE2lWV4ZvQCHkeZWUa Q==; X-CSE-ConnectionGUID: 0G94JpK7SZarsuQkhtsLVA== X-CSE-MsgGUID: G0KxMJ2iQb6KlmBkEhWe1Q== X-IronPort-AV: E=McAfee;i="6700,10204,11166"; a="25929465" X-IronPort-AV: E=Sophos;i="6.10,154,1719903600"; d="scan'208";a="25929465" Received: from orviesa006.jf.intel.com ([10.64.159.146]) by orvoesa107.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Aug 2024 22:09:21 -0700 X-CSE-ConnectionGUID: SVdXdoIIRwOh7Pc03Kd8zg== X-CSE-MsgGUID: Uu+OYl+RRTiJZwt1cW3OLg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,154,1719903600"; d="scan'208";a="60141492" Received: from jf5300-b11a338t.jf.intel.com ([10.242.51.6]) by orviesa006.jf.intel.com with ESMTP; 16 Aug 2024 22:09:21 -0700 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosryahmed@google.com, nphamcs@gmail.com, ryan.roberts@arm.com, ying.huang@intel.com, 21cnbao@gmail.com, akpm@linux-foundation.org Cc: nanhai.zou@intel.com, wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v3 1/4] mm: zswap: zswap_is_folio_same_filled() takes an index in the folio. Date: Fri, 16 Aug 2024 22:09:18 -0700 Message-Id: <20240817050921.18462-2-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20240817050921.18462-1-kanchana.p.sridhar@intel.com> References: <20240817050921.18462-1-kanchana.p.sridhar@intel.com> MIME-Version: 1.0 X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 22E00140012 X-Stat-Signature: 387pxdjukqueosjeuioazaiy56cgn4su X-Rspam-User: X-HE-Tag: 1723871367-291562 X-HE-Meta: U2FsdGVkX1/hIgSrWXNNjDxkRzhOTzesQ0v6FxRJPZNQTMHIgoPhB+COz8qqCZKit29a196aA8g2nnxk5CehyzinhBRTKzT0ws7wWfmGmlus6isCl7tZJ3xB4EYEh/4Q7HQEQGUDn6rodzTujba0AfhFhLXL+8sDxzo31WUf9wcMYwYvub2j4Oopq4YSEbBoYEjaNn8kdoTwgjmjdYPgaWz/w33CVdZ1rbNTgUjD4Lp2rExMatVDyVeFyBQJ0i7TpUb2rls/pE4NZQ3sNm7+O2YoDox7wUTgoO2MuJHoodStHxvEhkhRJnwkNh3+ep2OsQ0e+z/av/E4+cKs246UcnKXKCBmfVkVXZwP+KGQQ4+AQ5LqdcsJ2FKblCOL47qVg1Mpa4F6BzryZZtuj8PIMp8HGpkNIN8ZVZ0SIu6VNFsk+Kv0lMrn2N/GgYJq9VTJxBJLUxbNUCMgDlNY1l/T2siAZW0gRvpwgu2/pyImayyhcWPvOMVJL5L2UyMMCtAvH4XeJ/+PACGik3DkYR+KpjqLTQTNGKD6FaIlIbHHHiiJA3pyrULPmdroqhoyzCsWVz6u0Y8fkyVT4ckgPRYvp9/iRe29MZD5KeakkEinyqcU7YBrH/xhRAmxBLtyp6LNo2u4GIWNBcBj4eruBI/AJO+luhgLjeW1jeuiTmtX+uua1lbSCu0YUONRC0iRdf4ADbWMtnR2qos5xr9YIYKBGV/uKt3ml9KWBpc9Q0E36vK8C8fZeGBsJaU0gjeJXvrfOwhx0ACPf3jQU08UgDF3q6Ro3REfyRrZQkVKLRPuqJRczRjnMo5zkpOKpu2RJJrvV//7E7eg71EjucS/0oFxbSJ7qjNBftlLbPFxSO22hHN/qTeC1HqxT7oshOZ2StF5uwwgfXSiSDtbdvQuiiIXSCX1oy79dUgAqs1+V5ofvs6y8aDjJaFOLCH3zUw0L/U6JZFsu+MQy8VC/u99QEM c+pLpiQ2 O7/Tnc8CPqwUT8gCKbDyxy0rbaPUCO5VXa/xpAOPwLhpZuRQTD9dkw6PO/sPvATyPyZmcLnSZAkDoM6g8vF4DgqCVAXPBOKfNVfqcF+2hYI22aYRUOn51qa+q8SIsAefyd46ypPfwY+Z4G1DxbyVm7yx7Zf5OSA0qiOsVODwZC/61t8oIXnTr99O+9z7exJ+X8NzbV5G+sjUwCwg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This change is being made so that zswap_store can process mTHP folios. Modified zswap_is_folio_same_filled() to work for any-order folios, by accepting an additional "index" parameter to arrive at the page within the folio to run the same-filled page check. Signed-off-by: Kanchana P Sridhar --- mm/zswap.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index adeaf9c97fde..6c5c656ec282 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -1358,14 +1358,14 @@ static void shrink_worker(struct work_struct *w) /********************************* * same-filled functions **********************************/ -static bool zswap_is_folio_same_filled(struct folio *folio, unsigned long *value) +static bool zswap_is_folio_same_filled(struct folio *folio, long index, unsigned long *value) { unsigned long *data; unsigned long val; unsigned int pos, last_pos = PAGE_SIZE / sizeof(*data) - 1; bool ret = false; - data = kmap_local_folio(folio, 0); + data = kmap_local_folio(folio, index * PAGE_SIZE); val = data[0]; if (val != data[last_pos]) @@ -1435,7 +1435,7 @@ bool zswap_store(struct folio *folio) goto reject; } - if (zswap_is_folio_same_filled(folio, &value)) { + if (zswap_is_folio_same_filled(folio, 0, &value)) { entry->length = 0; entry->value = value; atomic_inc(&zswap_same_filled_pages); From patchwork Sat Aug 17 05:09:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Sridhar, Kanchana P" X-Patchwork-Id: 13766985 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F1EE8C531DE for ; Sat, 17 Aug 2024 05:09:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AD72D8D00C4; Sat, 17 Aug 2024 01:09:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A8C0D8D00B8; Sat, 17 Aug 2024 01:09:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 816E68D00C4; Sat, 17 Aug 2024 01:09:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 674DB8D00B8 for ; Sat, 17 Aug 2024 01:09:34 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id DC85A120492 for ; Sat, 17 Aug 2024 05:09:33 +0000 (UTC) X-FDA: 82460559426.11.1FC2D14 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.15]) by imf24.hostedemail.com (Postfix) with ESMTP id 9B004180012 for ; Sat, 17 Aug 2024 05:09:31 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=J2G2eljl; spf=pass (imf24.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 198.175.65.15 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723871334; a=rsa-sha256; cv=none; b=w1T5aYRD680tCIFS0kpip1n8kXyVCsT+YGOuAZT7V1ASTvXd8bWumu+ejVt3YbHsRm3qDa 6DegDunALpJoobXlL8x5eC8azPAEhc85hOA/Tr3ZT0qUGO9aiHD48sD7DYazO6HTRifLNq 2mYhXQjH/SUwB6xZB6TsL1YhHbk2rmk= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=J2G2eljl; spf=pass (imf24.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 198.175.65.15 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723871334; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+HNbSX4Nfg/DT3bgx+GYIXGCDR7l37iAnJ1JwzYXhAE=; b=mypDgFmlUgUwANxPM4t5tqHwKsBgs12YLJx5y4H86Mj28fVM07C08TnHxnFTXkt2WgtYHG Cm2WZaPSJwhSjs6WTrDY55EP9VZXOGwtPXnLxh9FlkD6Kqy1Vi8Owiztu2Up0bWz0a6Hr3 quemDuiwRUoudfoAlEN7RULd3fGKNe4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1723871371; x=1755407371; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Hs5osIBnEQbfhkvnt0UdOWrL7Zx29/pIwKoBT11shSE=; b=J2G2eljl8XjLQXIonBIP852J2wfAuNlWx1Sgmgk/mB66ajf4Vh4WEd4Q RDlr2aK0xgemBu1IRxV8MBe/RaLppt9HH9bwFsByCEN+odIVpLMaLlx+z DHIcaJF9l1zujajY30a6tGZumjBckk6Ae2/JAlWFVcTwk/c8m70W/hSyE XjN6n5b7nCOz3Gu6CZosCZ7Ff3cYay/38Mhj0GE07/nfcpo0PXeLVUxWS bblDe3NprRmQUoHZLm61Mvu2SMQweKcXje0C2iv7ZAX+FZz1aTBSPNU1c E04JQOMn/dpjcqK8EKMCOEG0P/wKV1e3xNo3V/zRZIBUE+mcbs5bxSBu/ g==; X-CSE-ConnectionGUID: YAhfnObdQUqKojeGiZNNxQ== X-CSE-MsgGUID: 4wMTp97BRHuzqpxns7XWHQ== X-IronPort-AV: E=McAfee;i="6700,10204,11166"; a="25929473" X-IronPort-AV: E=Sophos;i="6.10,154,1719903600"; d="scan'208";a="25929473" Received: from orviesa006.jf.intel.com ([10.64.159.146]) by orvoesa107.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Aug 2024 22:09:21 -0700 X-CSE-ConnectionGUID: 45aK0Pq1RiiWmgfsLRGP+A== X-CSE-MsgGUID: BXaEkahsTVqlUNwVVqM3Ow== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,154,1719903600"; d="scan'208";a="60141495" Received: from jf5300-b11a338t.jf.intel.com ([10.242.51.6]) by orviesa006.jf.intel.com with ESMTP; 16 Aug 2024 22:09:21 -0700 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosryahmed@google.com, nphamcs@gmail.com, ryan.roberts@arm.com, ying.huang@intel.com, 21cnbao@gmail.com, akpm@linux-foundation.org Cc: nanhai.zou@intel.com, wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v3 2/4] mm: zswap: zswap_store() extended to handle mTHP folios. Date: Fri, 16 Aug 2024 22:09:19 -0700 Message-Id: <20240817050921.18462-3-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20240817050921.18462-1-kanchana.p.sridhar@intel.com> References: <20240817050921.18462-1-kanchana.p.sridhar@intel.com> MIME-Version: 1.0 X-Stat-Signature: h9ss9ut67wokcg7mxf8n3d6fkj5c5i9x X-Rspamd-Queue-Id: 9B004180012 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1723871371-907349 X-HE-Meta: U2FsdGVkX193tgW1dkzdy+pGHaXHX+OpRWBCPauZzbm8asjlhf03Uqgm5IT+Oh++o1uraSwgXSDVso6/65iyBRUpYiIPtnsheW/RNdiq/8CsecAO00SJaU3AKMoKU3S8HIYS5Qu+B2CbecB5jVjv3c/YOgY+9WXytgR3777DElDXPmgWJGYDp+ZCaSD87yeHR9s3C1fEyA6/DOUApt3Khpnpq+3kQMqWvci+EksbVOfNF+Ys9sFu7fXUMzm8YURv/xFTZ4SlBn6uj8EpcBWEMTfjNKmiYLxYqWLGy4Blv+J3DM1HuZvDNVJLit+cQP+9BnYf4FsQuf1wFtnMwkqKnU58M7venHCzuY79FrwT9Ml3dpMOu5HzCS98ML5CBqANzU3HnHRtTRjE/LxgD2EV7UsBryJBatJIROaWQfVlPX5nbEQgmQqymO78b8dWtqk69hLNNI6gi5hoSv+jEvJw+Rz0W4BCpWiM+9IH9mvGieJ3OBObUpnPUrIqIv18qggfaePqZG7gPo7/sI/Cqai+vBd3+c+6tOCioj5Tf7InjSEXtIBA3wry+dsEBLxqUlxfyREEoD9I8Rqe4YnVXGNhKJRZJblmYkK51bhyR/vtXSmyYVcEuceKFpZIOqWH2oBYnOMXJyYsV8WUVvOJ1LbBzcUfUB+xKP8eV+K69JTL14mA9lei+5Cgyl2/oDOxK37WNAERRCZBhMsEcrJwJxfzM/zdUZ4EvNfvvFEYoPB+UXmsV16q91uEcn2cyln2cfYo/HGPYaoHHeo81o441rgSze6gg/e8hEr+mWJMaUWB/Yi2mt7DJmhLW98Z+YBNnRYendFQ00jLte1RypClGlDfJhbE19QdUQaYq8PWMTvWymecwczZh8eplby2J+lW7N+YcVgfkPLA9k8/m3p2KhulKIUkDuPjY2rMuxVpuG+xVVYtumSjk8IxS2ywiWuVSEcQdR+rjFnwC28pGdc+ppg KRnTy2Bn 6cAnzSSLn/tz3w/EvhmMgXcGPJ1XH/4OLOC18c4Q83fBA/ydfNX2Q+wbSSEco5m5zpIW94KiYNuuoEVI2nwfshyFl0DBS0dzSHlk+FRXi7MaIJY4BT30eG0/VzF2Yhi93ox6qxDWt94ymvF2ymAoMBOdY+fNzMEiXFUlzeZ7yB85t5o0LAYl+mL3z9SHDpr0qsc0z99OCDUbR3X84zM0kc9vhGFzdThA0oOFz4j78dVHWXDIUloaZZ7HQfmgB+Af3FYy+OHhtBXLHe07NHI+RJyMTm7lRFy3AWEkCv6Sr7SCAaWc/waBJqPPA4lwlS1cG1KCd X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: zswap_store() will now process and store mTHP and PMD-size THP folios. This change reuses and adapts the functionality in Ryan Roberts' RFC patch [1]: "[RFC,v1] mm: zswap: Store large folios without splitting" [1] https://lore.kernel.org/linux-mm/20231019110543.3284654-1-ryan.roberts@arm.com/T/#u This patch provides a sequential implementation of storing an mTHP in zswap_store() by iterating through each page in the folio to compress and store it in the zswap zpool. Towards this goal, zswap_compress() is modified to take a page instead of a folio as input. Each page's swap offset is stored as a separate zswap entry. If an error is encountered during the store of any page in the mTHP, all previous pages/entries stored will be invalidated. Thus, an mTHP is either entirely stored in ZSWAP, or entirely not stored in ZSWAP. This forms the basis for building batching of pages during zswap store of large folios, by compressing batches of up to say, 8 pages in an mTHP in parallel in hardware, with the Intel In-Memory Analytics Accelerator (Intel IAA). Also, addressed some of the RFC comments from the discussion in [1]. Co-developed-by: Ryan Roberts Signed-off-by: Signed-off-by: Kanchana P Sridhar --- mm/zswap.c | 234 +++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 172 insertions(+), 62 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index 6c5c656ec282..7a712be2f3cb 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -884,7 +884,7 @@ static int zswap_cpu_comp_dead(unsigned int cpu, struct hlist_node *node) return 0; } -static bool zswap_compress(struct folio *folio, struct zswap_entry *entry) +static bool zswap_compress(struct page *page, struct zswap_entry *entry) { struct crypto_acomp_ctx *acomp_ctx; struct scatterlist input, output; @@ -902,7 +902,7 @@ static bool zswap_compress(struct folio *folio, struct zswap_entry *entry) dst = acomp_ctx->buffer; sg_init_table(&input, 1); - sg_set_folio(&input, folio, PAGE_SIZE, 0); + sg_set_page(&input, page, PAGE_SIZE, 0); /* * We need PAGE_SIZE * 2 here since there maybe over-compression case, @@ -1394,36 +1394,83 @@ static void zswap_fill_folio(struct folio *folio, unsigned long value) /********************************* * main API **********************************/ -bool zswap_store(struct folio *folio) + +/* + * Returns true if the entry was successfully + * stored in the xarray, and false otherwise. + */ +static bool zswap_store_entry(struct xarray *tree, + struct zswap_entry *entry) { - swp_entry_t swp = folio->swap; - pgoff_t offset = swp_offset(swp); - struct xarray *tree = swap_zswap_tree(swp); - struct zswap_entry *entry, *old; - struct obj_cgroup *objcg = NULL; - struct mem_cgroup *memcg = NULL; - unsigned long value; + struct zswap_entry *old; + pgoff_t offset = swp_offset(entry->swpentry); - VM_WARN_ON_ONCE(!folio_test_locked(folio)); - VM_WARN_ON_ONCE(!folio_test_swapcache(folio)); + old = xa_store(tree, offset, entry, GFP_KERNEL); - /* Large folios aren't supported */ - if (folio_test_large(folio)) + if (xa_is_err(old)) { + int err = xa_err(old); + + WARN_ONCE(err != -ENOMEM, "unexpected xarray error: %d\n", err); + zswap_reject_alloc_fail++; return false; + } - if (!zswap_enabled) - goto check_old; + /* + * We may have had an existing entry that became stale when + * the folio was redirtied and now the new version is being + * swapped out. Get rid of the old. + */ + if (old) + zswap_entry_free(old); - /* Check cgroup limits */ - objcg = get_obj_cgroup_from_folio(folio); - if (objcg && !obj_cgroup_may_zswap(objcg)) { - memcg = get_mem_cgroup_from_objcg(objcg); - if (shrink_memcg(memcg)) { - mem_cgroup_put(memcg); - goto reject; - } - mem_cgroup_put(memcg); + return true; +} + +/* + * If the zswap store fails or zswap is disabled, we must invalidate the + * possibly stale entries which were previously stored at the offsets + * corresponding to each page of the folio. Otherwise, writeback could + * overwrite the new data in the swapfile. + * + * This is called after the store of the i-th offset in a large folio has + * failed. All zswap entries in the folio must be deleted. This helps make + * sure that a swapped-out mTHP is either entirely stored in zswap, or + * entirely not stored in zswap. + * + * This is also called if zswap_store() is invoked, but zswap is not enabled. + * All offsets for the folio are deleted from zswap in this case. + */ +static void zswap_delete_stored_offsets(struct xarray *tree, + pgoff_t offset, + long nr_pages) +{ + struct zswap_entry *entry; + long i; + + for (i = 0; i < nr_pages; ++i) { + entry = xa_erase(tree, offset + i); + if (entry) + zswap_entry_free(entry); } +} + +/* + * Stores the page at specified "index" in a folio. + */ +static bool zswap_store_page(struct folio *folio, long index, + struct obj_cgroup *objcg, + struct zswap_pool *pool) +{ + swp_entry_t swp = folio->swap; + int type = swp_type(swp); + pgoff_t offset = swp_offset(swp) + index; + struct page *page = folio_page(folio, index); + struct xarray *tree = swap_zswap_tree(swp); + struct zswap_entry *entry; + unsigned long value; + + if (objcg) + obj_cgroup_get(objcg); if (zswap_check_limits()) goto reject; @@ -1435,7 +1482,7 @@ bool zswap_store(struct folio *folio) goto reject; } - if (zswap_is_folio_same_filled(folio, 0, &value)) { + if (zswap_is_folio_same_filled(folio, index, &value)) { entry->length = 0; entry->value = value; atomic_inc(&zswap_same_filled_pages); @@ -1443,42 +1490,20 @@ bool zswap_store(struct folio *folio) } /* if entry is successfully added, it keeps the reference */ - entry->pool = zswap_pool_current_get(); - if (!entry->pool) + if (!zswap_pool_get(pool)) goto freepage; - if (objcg) { - memcg = get_mem_cgroup_from_objcg(objcg); - if (memcg_list_lru_alloc(memcg, &zswap_list_lru, GFP_KERNEL)) { - mem_cgroup_put(memcg); - goto put_pool; - } - mem_cgroup_put(memcg); - } + entry->pool = pool; - if (!zswap_compress(folio, entry)) + if (!zswap_compress(page, entry)) goto put_pool; store_entry: - entry->swpentry = swp; + entry->swpentry = swp_entry(type, offset); entry->objcg = objcg; - old = xa_store(tree, offset, entry, GFP_KERNEL); - if (xa_is_err(old)) { - int err = xa_err(old); - - WARN_ONCE(err != -ENOMEM, "unexpected xarray error: %d\n", err); - zswap_reject_alloc_fail++; + if (!zswap_store_entry(tree, entry)) goto store_failed; - } - - /* - * We may have had an existing entry that became stale when - * the folio was redirtied and now the new version is being - * swapped out. Get rid of the old. - */ - if (old) - zswap_entry_free(old); if (objcg) { obj_cgroup_charge_zswap(objcg, entry->length); @@ -1512,7 +1537,7 @@ bool zswap_store(struct folio *folio) else { zpool_free(entry->pool->zpool, entry->handle); put_pool: - zswap_pool_put(entry->pool); + zswap_pool_put(pool); } freepage: zswap_entry_cache_free(entry); @@ -1520,16 +1545,101 @@ bool zswap_store(struct folio *folio) obj_cgroup_put(objcg); if (zswap_pool_reached_full) queue_work(shrink_wq, &zswap_shrink_work); -check_old: + + return false; +} + +/* + * Modified to store mTHP folios. Each page in the mTHP will be compressed + * and stored sequentially. + */ +bool zswap_store(struct folio *folio) +{ + long nr_pages = folio_nr_pages(folio); + swp_entry_t swp = folio->swap; + pgoff_t offset = swp_offset(swp); + struct xarray *tree = swap_zswap_tree(swp); + struct obj_cgroup *objcg = NULL; + struct mem_cgroup *memcg = NULL; + struct zswap_pool *pool; + bool ret = false; + long index; + + VM_WARN_ON_ONCE(!folio_test_locked(folio)); + VM_WARN_ON_ONCE(!folio_test_swapcache(folio)); + + if (!zswap_enabled) + goto reject; + /* - * If the zswap store fails or zswap is disabled, we must invalidate the - * possibly stale entry which was previously stored at this offset. - * Otherwise, writeback could overwrite the new data in the swapfile. + * Check cgroup limits: + * + * The cgroup zswap limit check is done once at the beginning of an + * mTHP store, and not within zswap_store_page() for each page + * in the mTHP. We do however check the zswap pool limits at the + * start of zswap_store_page(). What this means is, the cgroup + * could go over the limits by at most (HPAGE_PMD_NR - 1) pages. + * However, the per-store-page zswap pool limits check should + * hopefully trigger the cgroup aware and zswap LRU aware global + * reclaim implemented in the shrinker. If this assumption holds, + * the cgroup exceeding the zswap limits could potentially be + * resolved before the next zswap_store, and if it is not, the next + * zswap_store would fail the cgroup zswap limit check at the start. */ - entry = xa_erase(tree, offset); - if (entry) - zswap_entry_free(entry); - return false; + objcg = get_obj_cgroup_from_folio(folio); + if (objcg && !obj_cgroup_may_zswap(objcg)) { + memcg = get_mem_cgroup_from_objcg(objcg); + if (shrink_memcg(memcg)) { + mem_cgroup_put(memcg); + goto put_objcg; + } + mem_cgroup_put(memcg); + } + + if (zswap_check_limits()) + goto put_objcg; + + pool = zswap_pool_current_get(); + if (!pool) + goto put_objcg; + + if (objcg) { + memcg = get_mem_cgroup_from_objcg(objcg); + if (memcg_list_lru_alloc(memcg, &zswap_list_lru, GFP_KERNEL)) { + mem_cgroup_put(memcg); + goto put_pool; + } + mem_cgroup_put(memcg); + } + + /* + * Store each page of the folio as a separate entry. If we fail to store + * a page, unwind by removing all the previous pages we stored. + */ + for (index = 0; index < nr_pages; ++index) { + if (!zswap_store_page(folio, index, objcg, pool)) + goto put_pool; + } + + ret = true; + +put_pool: + zswap_pool_put(pool); +put_objcg: + obj_cgroup_put(objcg); + if (zswap_pool_reached_full) + queue_work(shrink_wq, &zswap_shrink_work); +reject: + /* + * If the zswap store fails or zswap is disabled, we must invalidate + * the possibly stale entries which were previously stored at the + * offsets corresponding to each page of the folio. Otherwise, + * writeback could overwrite the new data in the swapfile. + */ + if (!ret) + zswap_delete_stored_offsets(tree, offset, nr_pages); + + return ret; } bool zswap_load(struct folio *folio) From patchwork Sat Aug 17 05:09:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Sridhar, Kanchana P" X-Patchwork-Id: 13766986 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2CBBCC52D7F for ; Sat, 17 Aug 2024 05:09:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 163238D00C5; Sat, 17 Aug 2024 01:09:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0EB528D00B8; Sat, 17 Aug 2024 01:09:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EF46D8D00C5; Sat, 17 Aug 2024 01:09:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id D2B7E8D00B8 for ; Sat, 17 Aug 2024 01:09:35 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 719B9804BA for ; Sat, 17 Aug 2024 05:09:35 +0000 (UTC) X-FDA: 82460559510.02.C719038 Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.15]) by imf26.hostedemail.com (Postfix) with ESMTP id 5C045140003 for ; Sat, 17 Aug 2024 05:09:33 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=e2uUr1So; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf26.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 198.175.65.15 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723871312; a=rsa-sha256; cv=none; b=rbDovDTQyEgPXIIq+KBA3qVRHTzuwuYLAy7klA6hd4oXwo90rXIBsrf9xyUaPlSTCiL5o3 ZC69+sPXoafMzmqtFgLlwEHKSel9JEh7YUSikLnChN2wWIwyd0GDhZq+wdKoIsWhtb/x0O JYAEHqALPRNxeQuAaKZkzAPIJaaFXJs= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=e2uUr1So; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf26.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 198.175.65.15 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723871312; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=lGWbhjtSJVg8G5Ir1pHmR/keo5/N7hZqSr5qDj7PThk=; b=nb8LBdy5iyHswS75htqxm7NeQa4JR1cip+9A6IvDx/6ADbOApDLWJn/41t2vV58GhfIeUi f/EtsOTx7DXgdogbP3ar4R6b5/qB5Wv0Kjc9or5FQtq3qrMInmUNa1NzaWU2acK6bZMslW mqvB0t44Z01+9e8g9JuVmfEaCTCxGaI= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1723871373; x=1755407373; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Rhd+o6cIvQZsonqH5BjtpwXRBNEn4sYl6ZxLr6ObqFU=; b=e2uUr1So7SnNOyJHFr5iFbqgYY4YoT/Lp3SM8jvrS/MjtPjYI3C86Xrt FLXah55MHo6DJjQMO0mrU85PkpHMMjCY+weZ1zN2kMcGEDZotiYfvCb61 TOdmEUW7MLq/855IxfZ/iUTRrVA3gf0HSCjjJp357Y+E+rC5qkyV6N3hz I5Bo/QAYRbrUcfdReAg0HJTeDtM+K5V/1kcs7K9tpvrkvMZj3y6U+pn1d Iv/kVINwiU5He3l5hhDU/KTmADjnlR2AAS2MCT6O48vdyFCb1IwrL5QkK Y5SifEKURc0E+fHgQavcoD4n4iMXEGUwfoypaWmDuzvBmGVGguRRqKnLV w==; X-CSE-ConnectionGUID: bIfizAMbRU2NRhtdWgU4rw== X-CSE-MsgGUID: xrMggtMMSXu19jH5iWxj2A== X-IronPort-AV: E=McAfee;i="6700,10204,11166"; a="25929480" X-IronPort-AV: E=Sophos;i="6.10,154,1719903600"; d="scan'208";a="25929480" Received: from orviesa006.jf.intel.com ([10.64.159.146]) by orvoesa107.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Aug 2024 22:09:21 -0700 X-CSE-ConnectionGUID: ZSoOEOTiSpKi7SFuA8U4wg== X-CSE-MsgGUID: xlveTXNuQU62fSdNy5hU2w== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,154,1719903600"; d="scan'208";a="60141498" Received: from jf5300-b11a338t.jf.intel.com ([10.242.51.6]) by orviesa006.jf.intel.com with ESMTP; 16 Aug 2024 22:09:21 -0700 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosryahmed@google.com, nphamcs@gmail.com, ryan.roberts@arm.com, ying.huang@intel.com, 21cnbao@gmail.com, akpm@linux-foundation.org Cc: nanhai.zou@intel.com, wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v3 3/4] mm: Add MTHP_STAT_ZSWPOUT to sysfs per-order mthp stats. Date: Fri, 16 Aug 2024 22:09:20 -0700 Message-Id: <20240817050921.18462-4-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20240817050921.18462-1-kanchana.p.sridhar@intel.com> References: <20240817050921.18462-1-kanchana.p.sridhar@intel.com> MIME-Version: 1.0 X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 5C045140003 X-Stat-Signature: 75y34h9yiczpkwsrj77kozsw17kr1g7k X-Rspam-User: X-HE-Tag: 1723871373-827633 X-HE-Meta: U2FsdGVkX1+Dh5QHYjy2TxCFyUYDQbsrmuGgMoCu/CRh3TsCCPC2xPu/IrcAiyAgPjoi7rrRPaZJnXdkRK8aMVHEh0hBAYZWzL07bXQUJHieM8e7YCOmzf12nckTktfRsat0u+e5HkK7GUgaEbFtOUROCgNh1AB5pAFqqPI0Mx1Tlng8h8M8N7o5+W9wJ1H9GtH59n5v/ogjBbgUcOovSSfC8yZ0aq7xSYlxYUu2XyCcq8LnhWANuJ3qXLFLqx+UkpPtEzyIovFN+OnAFeENoWnUa5wxN8iO11ExBopneatbRizusjU7rdvNuU2rcXrfA7m6XDGLdPTKNnFzud2u2Bn0LvyOIaD2+sV4pszBbvW09HRG2++1bnJ6DSCYDjNUI3TwL3loVx4YR1V/xwdmT5H5s8ziG8B4F/Z3wSdAt7TN11lxqX5xpKchSl3vumJoKWkrwbEsTcwupXCyHQk8aSp3EEThCASjBqW/bZGar3UkezI+oHbpnLVHZR1OhmcwgBCBJRGN5XMwcH3gghdp53kjQRDNfws32MpSj262Y72tFQJ0Svps/ReEv6Sc62djRimvgddUW1C4g2tsL5ugY/lsEWG4v7oCU7M7QYTAqw5QaLS1NhLK53l4Ns7JAOiQQoRbD2kljQi/TXvy8ahv3XyKKzNXUGFSVtO/YDO/FPGeXWc9W9HJMqlYU6kREgOSPHhEzCcH9JVVcE+T0dGeSw3+OCHzXlMivDiPT/egBeir70R1mZbK4LziVujB01aGqfOr+6n6pZ63p3d9pXSI3SwbPsgGD4mIzIodWGftzy0igHMXmoeMyoINgTCYE8Tgns20AzI13n0xyz1WvOuEB3c7ZdXluTYUWVtAFmXf/nFQKVEBQr8AWAz3QvCosDGqNWDSq8kBqLO8zFXTMoFh+Z9EeeAsaHX0pe/8nv1vFs92jPyGs5gXtl2ZxT73jGQCeyIBmfUp04BPOuIuRbA HzMglzMA rFMZrvJ2ZBYjOlh1YM+9QPJjpf0q0kUeXRTQU3iGVEXVLDAM5mUiZC3Gi6XOCVwBo702j77DsC12H35JzpH3lvx2zGqLiiJZNclz+Bfj5F4/lhCnfrV5d4CpTvelVyH4+jK1ENw1+BDXS59klw1QhF56LZGPkEOewyJeDw5eeYJ5bFqNwGJL688pjFyNJ1Rwpn2TF56dWl+Tc574= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add a new MTHP_STAT_ZSWPOUT entry to the sysfs mTHP stats so that per-order mTHP folio ZSWAP stores can be accounted. Signed-off-by: Kanchana P Sridhar --- include/linux/huge_mm.h | 1 + mm/huge_memory.c | 2 ++ 2 files changed, 3 insertions(+) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index e25d9ebfdf89..44609d84f2dd 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -273,6 +273,7 @@ enum mthp_stat_item { MTHP_STAT_ANON_FAULT_ALLOC, MTHP_STAT_ANON_FAULT_FALLBACK, MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE, + MTHP_STAT_ZSWPOUT, MTHP_STAT_SWPOUT, MTHP_STAT_SWPOUT_FALLBACK, MTHP_STAT_SHMEM_ALLOC, diff --git a/mm/huge_memory.c b/mm/huge_memory.c index f4be468e06a4..7e97b6ed6ff1 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -574,6 +574,7 @@ static struct kobj_attribute _name##_attr = __ATTR_RO(_name) DEFINE_MTHP_STAT_ATTR(anon_fault_alloc, MTHP_STAT_ANON_FAULT_ALLOC); DEFINE_MTHP_STAT_ATTR(anon_fault_fallback, MTHP_STAT_ANON_FAULT_FALLBACK); DEFINE_MTHP_STAT_ATTR(anon_fault_fallback_charge, MTHP_STAT_ANON_FAULT_FALLBACK_CHARGE); +DEFINE_MTHP_STAT_ATTR(zswpout, MTHP_STAT_ZSWPOUT); DEFINE_MTHP_STAT_ATTR(swpout, MTHP_STAT_SWPOUT); DEFINE_MTHP_STAT_ATTR(swpout_fallback, MTHP_STAT_SWPOUT_FALLBACK); DEFINE_MTHP_STAT_ATTR(shmem_alloc, MTHP_STAT_SHMEM_ALLOC); @@ -587,6 +588,7 @@ static struct attribute *stats_attrs[] = { &anon_fault_alloc_attr.attr, &anon_fault_fallback_attr.attr, &anon_fault_fallback_charge_attr.attr, + &zswpout_attr.attr, &swpout_attr.attr, &swpout_fallback_attr.attr, &shmem_alloc_attr.attr, From patchwork Sat Aug 17 05:09:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Sridhar, Kanchana P" X-Patchwork-Id: 13766987 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ADA5CC531DE for ; Sat, 17 Aug 2024 05:09:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 904468D00C6; Sat, 17 Aug 2024 01:09:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 88D9F8D00B8; Sat, 17 Aug 2024 01:09:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7302D8D00C6; Sat, 17 Aug 2024 01:09:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 526DD8D00B8 for ; Sat, 17 Aug 2024 01:09:36 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 07F18C04B7 for ; Sat, 17 Aug 2024 05:09:36 +0000 (UTC) X-FDA: 82460559552.15.A1E3A5A Received: from mgamail.intel.com (mgamail.intel.com [198.175.65.15]) by imf24.hostedemail.com (Postfix) with ESMTP id EDC20180011 for ; Sat, 17 Aug 2024 05:09:33 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=IRMIC7t4; spf=pass (imf24.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 198.175.65.15 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1723871336; a=rsa-sha256; cv=none; b=KJGeKUemR+pFZbSZVxjIDPgaFAR0vXnUJNCVLtMJV9NTsso6JJ8Rju5SrXXVISrYZ5Lw8s ot+e4Yrtf9Qnn5qQvSld+sgEMeEQpHX0QGjyNdY8xm8Miuj6wmPBkY/NTiJt8NuItnZRq9 jY8Hh6FyTpeVutdV8g5wdzm7pZYFW3o= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=IRMIC7t4; spf=pass (imf24.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 198.175.65.15 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1723871336; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Hv0pZRH2O3uWuKJq4mHJ3ytAE+yKARc7SzdSBN4A/oo=; b=QvuJ+l8/8kHKkiPYqgt7vxwg2Gt1Upg6JAtT10bzj1vxlYBoo4ir+YWRKrlb2lNwPFRhJc VWpq9vyCrxNLHwKts6BicCl475Uzpfi6fH8uChmSuL218fo+AE1EmiaLcxj5NWkCJ0a6Ys 5o6/kNYCDIOnQlVykqTFOOYbhDp0NFE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1723871374; x=1755407374; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=pIv8/Aymq35M3mk+Ip2dqi+IlPE8wYobMr4oum9ku1Y=; b=IRMIC7t49MfzNvoQj+3dSQ89Z9c8EjHVrrfFq0rQmKYwJsQ9JPtS5FW4 UL9Eo2cx8WuWG7aAQZSZplozbj42MRHPC13l2SDwH/RGIblUl5bzNM4CW Ahqo7mQztxTfOGEPNYWzq83zZhEQvX59nMWttiSPeF5kpQVPrt/K0sBsz cU+BhQgnDfcRB+reAcI0uQ3ryTghr6WGxBDdy+OOcaaUvDkclOyz49YsR epqOBFbVXOSi7qSgUrIx3tev1IAbdFrroOUU43akSyjGGm2auuOZubzUF nxd+Pv61DrHkVFO9Kcf9eF73mK1bwmRTlWZZtaPg9PrZM/5FqQJKrOvwO Q==; X-CSE-ConnectionGUID: XPnez9x6Sem/k15xRGmz0A== X-CSE-MsgGUID: W6GovgOXS/ujYMrEMdlKJA== X-IronPort-AV: E=McAfee;i="6700,10204,11166"; a="25929487" X-IronPort-AV: E=Sophos;i="6.10,154,1719903600"; d="scan'208";a="25929487" Received: from orviesa006.jf.intel.com ([10.64.159.146]) by orvoesa107.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 16 Aug 2024 22:09:22 -0700 X-CSE-ConnectionGUID: G3ES7ewjRXi5YjsXLL3eQQ== X-CSE-MsgGUID: fZPmCcW6RFSDaGvIag7BFg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.10,154,1719903600"; d="scan'208";a="60141501" Received: from jf5300-b11a338t.jf.intel.com ([10.242.51.6]) by orviesa006.jf.intel.com with ESMTP; 16 Aug 2024 22:09:22 -0700 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosryahmed@google.com, nphamcs@gmail.com, ryan.roberts@arm.com, ying.huang@intel.com, 21cnbao@gmail.com, akpm@linux-foundation.org Cc: nanhai.zou@intel.com, wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v3 4/4] mm: swap: Count successful mTHP ZSWAP stores in sysfs mTHP stats. Date: Fri, 16 Aug 2024 22:09:21 -0700 Message-Id: <20240817050921.18462-5-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20240817050921.18462-1-kanchana.p.sridhar@intel.com> References: <20240817050921.18462-1-kanchana.p.sridhar@intel.com> MIME-Version: 1.0 X-Stat-Signature: cbksdfaejqu4dkado5y9mdbm31ogmkzr X-Rspamd-Queue-Id: EDC20180011 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1723871373-431488 X-HE-Meta: U2FsdGVkX1+ZF+VHXP+nHie+fh3ch0DPYoOvxh4RFw7JdlLVesctOCoOLTR/fi9N6kPMgXYfCYxxAZ6cC9zpWKLa6lyVZxLG4kyP0Y7ERI0dgcHS5gadEysZ6kH8KnF3y10d468H352LqghShjl/EnWPWksQBVJgDRQLW/Wf6alC6/JTPeDkSXMPXlZBnmn28uRT+rnBpjmXJPMVvcq4Rk4QDtE04LXONwmqkqTyLIhlVpN4b39T64SUGZgoGkPcRSLsogWui/hhQq4SV9qXkkMR3aM7aBLA1Q7ErY33qLeOt2QmEu9wMi0G9KoBf2dM+UcFPC8/2uO9U2ujZIOFltwv820Tym7Jv8JaXLi0lnfgbJGS2PPRzUxVxUnkrGJaCMdssssvEwaTOLVX4JNm3ufWkGYQN4LWtKvKvRYFFFoAAxGiB2QhAQ1wFeBv0oZ49C9+5Bilyh2gWvAaI4amP9f9xqg6eaJyevpHailwmRzFwwO7v9x4jaAUnTNMpKviNUzQodJu8IIFJiZyaw3z+wl0qxQZlDPmbBFq2ltRDlbOtYHQxwwCZ6tp55hNHhdkdDvAbW62eilX0tGGdGF22QsdTOUZowHxToz5esb7WxD2gjrewi59UfLwZcpdDolOXrbqXHPMCFnGJSyxE1XdM4kLKSIl6l3/Kyay9Djx5bkfglfv0YJ6FXTtMJeULva+mbGBka+1lr4v7wuAhnp+bsYspnT1vGGQZG/Py7hiavx1LBQwK8VwfQiWGPS+2WWZecF3oBn49l2KtdB7NXk/UUCUaxqE0hDqM7Pf4tft5rmNGwLIRreWr+ufLMMB7hyC+wDY4ynMDJlYQd4FeTCXmXfL8jAiZsvCtG8/JaYlxV32bx7mnoqmEaYJ/+2kpz36spytVPUvPZlFi118BrC+3AZT7xaNbdj31UsHeEV6bGXAOXiIZ1jB5jGyEFjcDC3k9x9j0Nsc6kKUUCfmc3X EEPSTtjF 90KHPYBprPkgMkcoLshcx24x4kwnvHjaw4NmQ/lXCiROkrc90sJxqkW5/U1yaSlnUNs+rPdhmlKeVmKdFg3qqH3aC3GfSvtOt9CkV9dPzjVJaYewCukXCz32oKILSzLQRZL2WYBTjhgyf0zp42vqVAgPAcw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: If zswap_store() successfully stores an mTHP, it will be counted under the per-order sysfs "zswpout" stats: /sys/kernel/mm/transparent_hugepage/hugepages-*kB/stats/zswpout Other block dev/fs mTHP swap-out events will be counted under the existing sysfs "swpout" stats: /sys/kernel/mm/transparent_hugepage/hugepages-*kB/stats/swpout Signed-off-by: Kanchana P Sridhar --- mm/page_io.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/mm/page_io.c b/mm/page_io.c index ff8c99ee3af7..debd04fbdfd0 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -172,6 +172,12 @@ int generic_swapfile_activate(struct swap_info_struct *sis, goto out; } +static inline void count_mthp_zswpout_vm_event(struct folio *folio) +{ + if (IS_ENABLED(CONFIG_THP_SWAP)) + count_mthp_stat(folio_order(folio), MTHP_STAT_ZSWPOUT); +} + /* * We may have stale swap cache pages in memory: notice * them here and get rid of the unnecessary final write. @@ -196,6 +202,7 @@ int swap_writepage(struct page *page, struct writeback_control *wbc) return ret; } if (zswap_store(folio)) { + count_mthp_zswpout_vm_event(folio); folio_unlock(folio); return 0; }