From patchwork Sun Nov 3 03:21:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Sridhar, Kanchana P" X-Patchwork-Id: 13860293 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 48ED7E6780A for ; Sun, 3 Nov 2024 03:21:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2D13F6B0093; Sat, 2 Nov 2024 23:21:24 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 047B56B0095; Sat, 2 Nov 2024 23:21:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C96846B0093; Sat, 2 Nov 2024 23:21:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 8632B6B0095 for ; Sat, 2 Nov 2024 23:21:23 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 47247812EF for ; Sun, 3 Nov 2024 03:21:23 +0000 (UTC) X-FDA: 82743332280.21.61B7B92 Received: from mgamail.intel.com (mgamail.intel.com [192.198.163.14]) by imf12.hostedemail.com (Postfix) with ESMTP id 608CD40005 for ; Sun, 3 Nov 2024 03:21:07 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=VAKRaRP+; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf12.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.14 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1730603960; a=rsa-sha256; cv=none; b=XQW/qkisoxp7ECgPZfBlV/wUIOoESi93WNWjm7kKbf35yqFcWTe55vMqZ2rDCYa/ngsnkM Y4kgJ9PE0dfZ7X0ivszLnonaf6v5fyfrNA9h5d7bZgsAj8Js4x+AM1CxGlqzv9ZYUTjIGB xzy3EEut2kwDotgksKPaVgVnIBaUZgY= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=VAKRaRP+; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf12.hostedemail.com: domain of kanchana.p.sridhar@intel.com designates 192.198.163.14 as permitted sender) smtp.mailfrom=kanchana.p.sridhar@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1730603960; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Qxz8u9Shn/SiPqBxAwwlyMbQX8CsM2ppl302cd3vziA=; b=7WOI7SlJc6i9goQmgUBelSIeUTFnlRSmLh9WYWp1b1Cs1lNT+PPOz5e042BoaodbjemYgK 7Gw6b9xeq3BwAT1N41L0048/90hAjqMMsZdWnMk/qiKJONddiKnpbirEu6HDrp3psaswzK qwbdbOh6iMahZUP3AL73VJ+sP9eJAog= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1730604081; x=1762140081; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=e3w3jsrwSRn0skEK+1OKRkuNEeuFjmM/VVAG1YViexY=; b=VAKRaRP+8Ee/4avyfGGba45eHC14TsWm4gBWRvwiD79KcANRXcr7nNrv NeYJQGmjIPk04+GnXnaIexg2BkItBk6Pb7oLToBRiMMvhlrFbV5Jf7e2L YHADuAiISrlmg5MQohOdjTxyCxIvPOjjq4EzW76OzvzJ7ivHExlUTDlrU COehzVp7C2WKv8sVHKqBIRRAtnulCejsiyHMnzuNAKogNuzryevqmiwwT HUuaLs+8HnrnRgwcyVPjUg7XYt1U3NPT63ibpkHYLfMyL+UPpavrBDdBn qCG5A0xcHu77FvTNjOn3Z3ZC7iFpRcuVfkdMg8j5XV7VipyCrEvn5LfDl g==; X-CSE-ConnectionGUID: LK5ZjHL8Q3qIjgkjy0GvEg== X-CSE-MsgGUID: KZTgbbjCQu+y/g7ltd95eA== X-IronPort-AV: E=McAfee;i="6700,10204,11244"; a="30548298" X-IronPort-AV: E=Sophos;i="6.11,254,1725346800"; d="scan'208";a="30548298" Received: from orviesa006.jf.intel.com ([10.64.159.146]) by fmvoesa108.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 02 Nov 2024 20:21:13 -0700 X-CSE-ConnectionGUID: PVoUksXZR4mAcYTV3a/VVg== X-CSE-MsgGUID: azt2Ht8kSBu5utEhSx2oiA== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="6.11,254,1725346800"; d="scan'208";a="83451891" Received: from unknown (HELO JF5300-B11A338T.jf.intel.com) ([10.242.51.115]) by orviesa006.jf.intel.com with ESMTP; 02 Nov 2024 20:21:14 -0700 From: Kanchana P Sridhar To: linux-kernel@vger.kernel.org, linux-mm@kvack.org, hannes@cmpxchg.org, yosryahmed@google.com, nphamcs@gmail.com, chengming.zhou@linux.dev, usamaarif642@gmail.com, ryan.roberts@arm.com, ying.huang@intel.com, 21cnbao@gmail.com, akpm@linux-foundation.org, linux-crypto@vger.kernel.org, herbert@gondor.apana.org.au, davem@davemloft.net, clabbe@baylibre.com, ardb@kernel.org, ebiggers@google.com, surenb@google.com, kristen.c.accardi@intel.com, zanussi@kernel.org Cc: wajdi.k.feghali@intel.com, vinodh.gopal@intel.com, kanchana.p.sridhar@intel.com Subject: [PATCH v2 08/13] mm: zswap: acomp_ctx mutex lock/unlock optimizations. Date: Sat, 2 Nov 2024 20:21:06 -0700 Message-Id: <20241103032111.333282-9-kanchana.p.sridhar@intel.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20241103032111.333282-1-kanchana.p.sridhar@intel.com> References: <20241103032111.333282-1-kanchana.p.sridhar@intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 608CD40005 X-Stat-Signature: 5do7p5w647r9x8b34zpouipcwf3qpxet X-Rspam-User: X-Rspamd-Server: rspam05 X-HE-Tag: 1730604067-366890 X-HE-Meta: U2FsdGVkX1+lOV9AP8h1au7wIPp/dpGBLJlp8ID4+PFjkhUzxFuG9JwybQ76opLi+OJNmlwosdxe1UKupDcdHfFu+9kLw5L5B0oxZlOfoCHoUzd8hS9S/F8k/qTfQb19Oqni4W2WfDGhviMttIubt6InFjYWd8g9GN63oTxjUxSTazdh8Tlaa6oPD0S4gFFsDs5RBIgyJh0PFC8s00QolfoeFZgTvFcp9CP0HfZUzbfF6D8vuROXl9wdXz7yqcKF5zgMBT9WQfG2VY8/OJ1lpjnDWFO3YozefimNxj8ErykXYgIRWtW1dYptN15h1adF0yTHyOVjUpjP6/WIGIYeOlaggCxDqSXdKFvgl6l+V3BNKmbCKUT+5pvjkLha4k/sM+4TW1ObSUY0Sx5vAs47BTOgRzdPL6qBrYwPq8P5LwmCV9gF+jzJOLXeLFLRdnzG3qYCxq11JVB+WoFe8C/iEkazyuS+6Um00bXTAEgzVFMm7ZB2rkyeRRdNF0EwMi3uMb5mzdf6dbIafXICiBqRcYUEr5a+x+BrwC+ZDPPs1BiOHv7MEJpoHAdr7udBufqvOBaYMmhoNuWMZw0sMSgPUpejSA/cRDTW6hfh/fdJd3dv34vI5f2/eU6dzE/JSIm7y6Ts3dEjdFP4Li1KmF2FJClShODUyRA2uGH9y3i8g7Gt2WDTSPVG6KCyjpzYnvNvWc6wlTEOFAVn7//5lHa2cOGj+QmzYrpJC7sBgSkz/hnCYrW2suMhrmcjMKWoHTDYjbi+E8RnO6HcAKwF+EGstzAYIACsRVvQ6buxx+z7IBENiUebv5nhD97lA7G0GZ4w8N9HMQDsN+J6HOFEADgCTgcqSOKRLwN1oRhsDAYxSyFjcRHzCVw80G9hehR4D3vkt4dKJQrw1YMOr8nJkOH77r2/lbUiu0rO8F+do2d+K4T3RBeoyqwvrQXPz0cQMXEHmkSrfbbG9249xjpRKkM yXcBI3/5 Hikar3s20izycg8kVzazJ99G1sx5ETmrT37ckZyORxWWQW2M1oN6CNsGyLRui6+aEqbij8cQ6P5nMOae8dxDEzQkxtAaSJiN51UVLWPaYiH5ODtXTKdckWT3JLMp5Rq4Ad73e/FDlQB0BO8FvTqBPC/KB8XFeKriYIUCLE85xziJuF5MnvCNJfHAy6S2hXBV0cb2lU0Va63PeGkrGY60s/nTgX1lozyAz6F/h3HALDoaqssdgL/zTJ80+qk5f6d5FabiyOnX/AfMMpG0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This patch implements two changes with respect to the acomp_ctx mutex lock: 1) The mutex lock is not acquired/released in zswap_compress(). Instead, zswap_store() acquires the mutex lock once before compressing each page in a large folio, and releases the lock once all pages in the folio have been compressed. This should reduce some compute cycles in case of large folio stores. 2) In zswap_decompress(), the mutex lock is released after the conditional zpool_unmap_handle() based on "src != acomp_ctx->buffer" rather than before. This ensures that the value of "src" obtained earlier does not change. If the mutex lock is released before the comparison of "src" it is possible that another call to reclaim by the same process could obtain the mutex lock and over-write the value of "src". Signed-off-by: Kanchana P Sridhar --- mm/zswap.c | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-) diff --git a/mm/zswap.c b/mm/zswap.c index f6316b66fb23..3e899fa61445 100644 --- a/mm/zswap.c +++ b/mm/zswap.c @@ -880,6 +880,9 @@ static int zswap_cpu_comp_dead(unsigned int cpu, struct hlist_node *node) return 0; } +/* + * The acomp_ctx->mutex must be locked/unlocked in the calling procedure. + */ static bool zswap_compress(struct page *page, struct zswap_entry *entry, struct zswap_pool *pool) { @@ -895,8 +898,6 @@ static bool zswap_compress(struct page *page, struct zswap_entry *entry, acomp_ctx = raw_cpu_ptr(pool->acomp_ctx); - mutex_lock(&acomp_ctx->mutex); - dst = acomp_ctx->buffer; sg_init_table(&input, 1); sg_set_page(&input, page, PAGE_SIZE, 0); @@ -949,7 +950,6 @@ static bool zswap_compress(struct page *page, struct zswap_entry *entry, else if (alloc_ret) zswap_reject_alloc_fail++; - mutex_unlock(&acomp_ctx->mutex); return comp_ret == 0 && alloc_ret == 0; } @@ -986,10 +986,16 @@ static void zswap_decompress(struct zswap_entry *entry, struct folio *folio) acomp_request_set_params(acomp_ctx->req, &input, &output, entry->length, PAGE_SIZE); BUG_ON(crypto_wait_req(crypto_acomp_decompress(acomp_ctx->req), &acomp_ctx->wait)); BUG_ON(acomp_ctx->req->dlen != PAGE_SIZE); - mutex_unlock(&acomp_ctx->mutex); if (src != acomp_ctx->buffer) zpool_unmap_handle(zpool, entry->handle); + + /* + * It is safer to unlock the mutex after the check for + * "src != acomp_ctx->buffer" so that the value of "src" + * does not change. + */ + mutex_unlock(&acomp_ctx->mutex); } /********************************* @@ -1487,6 +1493,7 @@ bool zswap_store(struct folio *folio) { long nr_pages = folio_nr_pages(folio); swp_entry_t swp = folio->swap; + struct crypto_acomp_ctx *acomp_ctx; struct obj_cgroup *objcg = NULL; struct mem_cgroup *memcg = NULL; struct zswap_pool *pool; @@ -1526,6 +1533,9 @@ bool zswap_store(struct folio *folio) mem_cgroup_put(memcg); } + acomp_ctx = raw_cpu_ptr(pool->acomp_ctx); + mutex_lock(&acomp_ctx->mutex); + for (index = 0; index < nr_pages; ++index) { struct page *page = folio_page(folio, index); ssize_t bytes; @@ -1547,6 +1557,7 @@ bool zswap_store(struct folio *folio) ret = true; put_pool: + mutex_unlock(&acomp_ctx->mutex); zswap_pool_put(pool); put_objcg: obj_cgroup_put(objcg);