From patchwork Sat Apr 29 08:27:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Yin, Fengwei" X-Patchwork-Id: 13226929 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8225EC77B60 for ; Sat, 29 Apr 2023 08:28:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C6DBE900003; Sat, 29 Apr 2023 04:28:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C1E7F6B0078; Sat, 29 Apr 2023 04:28:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AC055900003; Sat, 29 Apr 2023 04:28:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 9D1026B0075 for ; Sat, 29 Apr 2023 04:28:12 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 2FFE6AD2F1 for ; Sat, 29 Apr 2023 08:28:12 +0000 (UTC) X-FDA: 80733751224.21.40A2EB5 Received: from mga02.intel.com (mga02.intel.com [134.134.136.20]) by imf09.hostedemail.com (Postfix) with ESMTP id E2C40140010 for ; Sat, 29 Apr 2023 08:28:08 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=eMaiRqYo; spf=pass (imf09.hostedemail.com: domain of fengwei.yin@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682756889; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=QIb5VC8fScnLrNFoH1L9Vm25I+k0kAFjlpSRyh1Eafs=; b=NgYenSzuoqtULWBmpbg3JhKGuwR8UtP+w37qLBpx7x314m8Fs6v1u1qlvcZoqeUTFUOMRH p7UfcFaF1+xQGHkCZua5JJ8nSx9XfWiz8I2lxQJ1nfp7YO+g6m9CIaBHOJJo2YR/AkWSRV VJtvW7s9EumdcspDy/1pBoX94vcMyJY= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=eMaiRqYo; spf=pass (imf09.hostedemail.com: domain of fengwei.yin@intel.com designates 134.134.136.20 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682756889; a=rsa-sha256; cv=none; b=kWtzx1L4OspZczPyUX7/07ykCAggLePeKQlzPB34ya3k0KRVMSmKCTiwph/RNGxdB6fUgv H3QG0C+vG4DZpLYTi6uoFvDu8yaxqhsctnYDnTxgnNTp3bn1ZAhVFJIvaCDWyeBPLjcHG5 zwtx0jK4rpJU/lQik0/vc8A8VAxicQQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1682756889; x=1714292889; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=uboI4DJOSLiHmn6GUQHT1uZ6ZwStA/UXAc4gTwXHDSo=; b=eMaiRqYowm1NvPly4y2nnKkIUBr8kMj4yspyL/FqDtdXTYvSDrfpNOzO U2gwbo+ioHGZiAtXMLs1fZImw2fNYOFO7ewTFBueMmWo0WSSF325KY/V6 xGzJxVjAeLiUG7qWwBPzFfKU1qJt9tvkCeEyjrhJyJgahX+k0XZBeI0er CsEuh/UfqCrsUh2SWD9sjuvvAWSS6Jj94WCCLBo6408zHFQD8dk9R4rFv CkwcCSjZNmq4vSdZqsM88DnM6BPshYqaW1SJKPxXPmDO/aT83VuucN4Al smfLhalBR5+gIXbgSSUTqnAKmtvCiz73bDGE8wloYMopYgqW0oGiZBRNg A==; X-IronPort-AV: E=McAfee;i="6600,9927,10694"; a="337016609" X-IronPort-AV: E=Sophos;i="5.99,236,1677571200"; d="scan'208";a="337016609" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 29 Apr 2023 01:28:07 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10694"; a="725681201" X-IronPort-AV: E=Sophos;i="5.99,236,1677571200"; d="scan'208";a="725681201" Received: from fyin-dev.sh.intel.com ([10.239.159.32]) by orsmga008.jf.intel.com with ESMTP; 29 Apr 2023 01:28:05 -0700 From: Yin Fengwei To: linux-mm@kvack.org, akpm@linux-foundation.org, willy@infradead.org, kirill@shutemov.name, yuzhao@google.com, ryan.roberts@arm.com, ying.huang@intel.com Cc: fengwei.yin@intel.com Subject: [PATCH v3 1/2] THP: avoid lock when check whether THP is in deferred list Date: Sat, 29 Apr 2023 16:27:58 +0800 Message-Id: <20230429082759.1600796-2-fengwei.yin@intel.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230429082759.1600796-1-fengwei.yin@intel.com> References: <20230429082759.1600796-1-fengwei.yin@intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: E2C40140010 X-Stat-Signature: nkj8zzrq9prkw7u3xn9ynoshp8nznu54 X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1682756888-706250 X-HE-Meta: U2FsdGVkX1+25/7D+TcEjaMVBaviC+HtwaaUSHRrNdaRmm0TgU7OF1yNPT92B94UgXKJEA9wes/REK0gLb5AOVf4K3vn2JKzy2QWGEj2fGURPU5+qDasWpg6lZw//52QekqYR8DHb2SKTX7Z8v44gcIpMsr+wtxLy1v/48cZ/sZT0wR3FIS+J9wx6f4TFhnPc17psrO1W4G6ExkA7d9j24c53X6PD/T8xj2s6oL6poEZt0MK80izWz+80urqdcehdOOdGRL6TcYQ5tdldwzkw61dGsqRJc9NJp1KFIKt4u9MIfePZHD8Aj/2jz94epmbm8uP80HBjphb+iK21AZdSFNvAZduabdnziZEvI2qLrftD2HvHYisyHUdHrYGk2cP8qEP2j8qsiwO2Wi68FTPZVFWnQDrMiARTgC9vlgLG7Y99XnhglQvwVgxR6jfQocKdcE+wnGmiy5omnbutuSSuGx92zcsjO0dj6EKlo1Tc5WEyG/+j9TuP5+U3R5LvXkjXnUnZh5upUF2IrMvRND8fM+/79LujX7YTsOgxVMRJuHZm/uApdNmUsJVqbe5pA4knNJRwLebsMUGOZt7ZTMETo1RWvG1/2dhB2r23rHkoypnXPbCHadGZaQzVsb8tnUBt+4CygD+H/csdIgOyTuCFA0ap1+VBGcN75YYWyKixj3wJWR6/TCWZewGoCssV18DMmnWAJOJjJWW9GoOTAWuJy7Fk916RT0bsrUY1nSUvfswAKW149Qx9Z/AQ4KL5WofO3SthhcaxXl97mrtBi9PBsUzZ93LXSv2sxKeOCYDru1je4/XAn3QOIvEKvFyH6YB2m+bdxFkOJe70JJSojVsAMu22tWF+msEEHBSREBwfolHuLiFEimiPS94O9CDupi/L7waNtyswocP2Rz81JvblJ7Vn/veeEIKx9NrS5Psg4vXQ4kwd3az4m9EONLqRrrfIyvd+gKdbYCiC/n4vtK spxrEEun K7QGY6TcpajOfgthg2ZCMmsgFDLiiFAMTkmlRHgVICZStAil7BjzDTLAx4UDUKKxlhTQ4sSer4t/vTmRjfRSgksYrtgf6m8Szrz0VtBy342JIKmzptj/n0thxIqsP3pDgL/oqSyHE9SEtqmGt3nBNfcKrng== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: free_transhuge_page() acquires split queue lock then check whether the THP was added to deferred list or not. It brings high deferred queue lock contention. It's safe to check whether the THP is in deferred list or not without holding the deferred queue lock in free_transhuge_page() because when code hit free_transhuge_page(), there is no one tries to add the folio to _deferred_list. Running page_fault1 of will-it-scale + order 2 folio for anonymous mapping with 96 processes on an Ice Lake 48C/96T test box, we could see the 61% split_queue_lock contention: - 63.02% 0.01% page_fault1_pro [kernel.kallsyms] [k] free_transhuge_page - 63.01% free_transhuge_page + 62.91% _raw_spin_lock_irqsave With this patch applied, the split_queue_lock contention is less than 1%. Signed-off-by: Yin Fengwei Acked-by: Kirill A. Shutemov Reviewed-by: "Huang, Ying" --- mm/huge_memory.c | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 032fb0ef9cd1..2a1df2c24c8e 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2799,12 +2799,19 @@ void free_transhuge_page(struct page *page) struct deferred_split *ds_queue = get_deferred_split_queue(folio); unsigned long flags; - spin_lock_irqsave(&ds_queue->split_queue_lock, flags); - if (!list_empty(&folio->_deferred_list)) { - ds_queue->split_queue_len--; - list_del(&folio->_deferred_list); + /* + * At this point, there is no one trying to add the folio to + * deferred_list. If folio is not in deferred_list, it's safe + * to check without acquiring the split_queue_lock. + */ + if (data_race(!list_empty(&folio->_deferred_list))) { + spin_lock_irqsave(&ds_queue->split_queue_lock, flags); + if (!list_empty(&folio->_deferred_list)) { + ds_queue->split_queue_len--; + list_del(&folio->_deferred_list); + } + spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); } - spin_unlock_irqrestore(&ds_queue->split_queue_lock, flags); free_compound_page(page); }