From patchwork Wed Jun 28 10:56:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Zhu, Lipeng" X-Patchwork-Id: 13295565 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CDEA4EB64D7 for ; Wed, 28 Jun 2023 10:54:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 56F1B8D0003; Wed, 28 Jun 2023 06:54:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 51EC48D0001; Wed, 28 Jun 2023 06:54:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3E7FE8D0003; Wed, 28 Jun 2023 06:54:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 2CBF38D0001 for ; Wed, 28 Jun 2023 06:54:35 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id F0D7A160D69 for ; Wed, 28 Jun 2023 10:54:34 +0000 (UTC) X-FDA: 80951848068.22.7D423DA Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by imf15.hostedemail.com (Postfix) with ESMTP id 6E1F7A0007 for ; Wed, 28 Jun 2023 10:54:32 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=ZBNfiGx3; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf15.hostedemail.com: domain of lipeng.zhu@intel.com designates 134.134.136.100 as permitted sender) smtp.mailfrom=lipeng.zhu@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687949673; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=nsumsjhgsw+tLflpUjMBGVaApXQRvl3LyMohr9qkN6o=; b=1yBovSQiN8cLpFMNzM1bPhuJrhzxQQdeBEKxv4QsBHbCHt5EMhtWF0vqtdx7v30G3rwG4h kKE/twf+culdYuWknR9KFjoENUqKqmmgSuxwT9L7l3uhsYkGocMzrvfzpxEIl2e97RNbO8 /UJcs1jfPoBPMJ40bxDQcBKI8tzDOPk= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=ZBNfiGx3; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf15.hostedemail.com: domain of lipeng.zhu@intel.com designates 134.134.136.100 as permitted sender) smtp.mailfrom=lipeng.zhu@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687949673; a=rsa-sha256; cv=none; b=SEJYmDQ1RlrPRVNGbmuowhqX6eqhdATWDMvOTxN8VT4k2Czy7ulwJDfAgLdk16q23gNEit r+R/GILA14JLta5CDn+o4CHDtbJeMkWnEYeAOqv6d6/2+3o/tFIoORTkng6aOcPZjalttF zrs+f3IWWUWDbcPN3yFnONyZ2Z8SVjM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1687949672; x=1719485672; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=WgvMUlURPSf3sLuU7HCU3MKlq0fqlttjYS4ABXRfEVQ=; b=ZBNfiGx3JtIOv00I2fGJQT0mkgWuIt1/V/Pf5UntyCP0EnBkOPuSnjWp putoQtgeIjTyAC/fHhAwA4ARNk9hiC5NgcS7Cnar04qmy13LbrOyyeoUf 83f4YNP/E1GmwkM7wvTg9B+YYANIujlASloM6i+gPy6sHaXdDE4Nu1xwk sze1vWkkdzvb5beSaoMdHxH4YpN0IrHGywZPUoSJTGZNv0H3HPwVnfUlz xbiJHTwWcssyhh8URxKZZSZpbu7ivnNDJzmOnl9UeI+hKsRhljvoXNhyq iBBY6tb+USkNIVRiV0aJ80werll17AjrI9v8BPtPJhzLE0hkyr6breFbg g==; X-IronPort-AV: E=McAfee;i="6600,9927,10754"; a="427818564" X-IronPort-AV: E=Sophos;i="6.01,165,1684825200"; d="scan'208";a="427818564" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga105.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 28 Jun 2023 03:54:30 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10754"; a="752196704" X-IronPort-AV: E=Sophos;i="6.01,165,1684825200"; d="scan'208";a="752196704" Received: from linux-pnp-server-30.sh.intel.com ([10.239.146.163]) by orsmga001.jf.intel.com with ESMTP; 28 Jun 2023 03:54:27 -0700 From: "Zhu, Lipeng" To: akpm@linux-foundation.org, viro@zeniv.linux.org.uk, brauner@kernel.org Cc: linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, pan.deng@intel.com, yu.ma@intel.com, tianyou.li@intel.com, tim.c.chen@linux.intel.com, "Zhu, Lipeng" Subject: [PATCH] fs/address_space: add alignment padding for i_map and i_mmap_rwsem to mitigate a false sharing. Date: Wed, 28 Jun 2023 18:56:25 +0800 Message-Id: <20230628105624.150352-1-lipeng.zhu@intel.com> X-Mailer: git-send-email 2.39.1 MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 6E1F7A0007 X-Stat-Signature: izsxknefdaybujqoh5w6w6jnna1c8wtj X-Rspam-User: X-HE-Tag: 1687949672-996908 X-HE-Meta: U2FsdGVkX1+JT4kGhNZEhsCb47Or6YLk5qc9ctmMzJmtkE9RiJI4ateFoHaELvAdR/dC5+O2jgAXKR+5xsBdZPGsm3ngNMX/NPgNZQgBvvMulMp5do9yEWB807gHy4y/9iYZJVzKOsDw3fColwpX4kmPXpYecUmG/a79HKVTznMBMQoYUYtNfu08zbEz0ecc0JSVR2zt63pFdIu8YRFaLAkYMZzISl1Zj6dlWtuvdKJt7aja2eiQjzLuUDtVmSQhMcOmUvDeYNQCSliiFvdQLZ/IyjHMEiOLexWbwy2IEIa9HCqNkLzOPyk5DalHrfpH8DQ+wbD8gzVOO9bslq0PoCnIjmIyGb/lUXgbs0tFxBhI0gy8ACayVrH+fsHsavIPs6ETQ4ZIUw0xoNGVjVoyQIuGUf0/dSFsU7Hy1cDJUsiDGUWfNSbfnKIiPrFBvEbxgIJW2iagTt/ruvu8rx7mtxLJmd3RRSNWLazDWeSFIqK+OGyqN9Nio3ccK+QG3F5dkJD647UaP6CXYPeIAuz4KZQmFHmPnzKTKNYBp4GKyW6YEm34kZrpcd076+sqQLaWDgY7JYQkAFYGZKXASEDxsf7CtoI2HBX9BzRfIC4lpXCJDeelf8vOv4TY+x8+Rw1B2VmxoIHSaHatTl7vPoZ8jFwoeeUbZw/0TRjYQkUYjf1JtyF7LzZjYwNcdeY6bQspxtF+AymB4GkUmrAno1SbLtF04oqAG9esqjzqXh0s35Y8wVUxFngrQ21CkiYxI+e1ad9gcQu9+BEhPCnWQEN5coyjwK1igay3wjtQxqyvWXS+iO+GY80Sk/lZH0Q/S/fGdEj9rBBwb7F22KaM1Z292KuuulLR5UQPqY/+nqGReTBlWZfRBAKiW4rJPD2cbJrOCXHLbj77JdzPIo5I1k4mF1IPkJK2LsCpLRNCac97JjJSJqsAbQGH/U4Hscox/Ie3i7aq3fUp3bGY+ycKQnR jPClbAvH PpV962xzc0yyH7eGj4d6nQk1Xde04GxjcAzUkIsU6Nw5Io6sA+rgy4wBRfJsfQ6RPxElxGSUBsDpOqvx7aFJSGrLF/aHWiAZCT4ao0RbncycGsVaU3nkv6VDRkA5qjmSkJMEQEFOPmK10RoAeDgtniuKuzIp8lk2Hh9GsTDWiIIWGEsFohQ/XypMORooi5OTzov62zhRDGXf8YjAfJPG32EA+AjxAL5+hPfvqtPZ0oZNTwYw97IbDxWeJ1ECl51EsU/MoUAa0Ln+D6nUW7SoGmGUVxg9g3uqT4Y+CfxJnxkxyG/kT1c5ZAVMibTwwXF9dVdkX8gEVVeg8kp5ICdOyThvmTqGlUQwyKaZYQ+apjfrBluJD/KdL0icawYNYAA2qgdqf X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When running UnixBench/Shell Scripts, we observed high false sharing for accessing i_mmap against i_mmap_rwsem. UnixBench/Shell Scripts are typical load/execute command test scenarios, the i_mmap will be accessed frequently to insert/remove vma_interval_tree. Meanwhile, the i_mmap_rwsem is frequently loaded. Unfortunately, they are in the same cacheline. The patch places the i_mmap and i_mmap_rwsem in separate cache lines to avoid this false sharing problem. With this patch, on Intel Sapphire Rapids 2 sockets 112c/224t platform, based on kernel v6.4-rc4, the 224 parallel score is improved ~2.5% for UnixBench/Shell Scripts case. And perf c2c tool shows the false sharing is resolved as expected, the symbol vma_interval_tree_remove disappeared in cache line 0 after this change. Baseline: ================================================= Shared Cache Line Distribution Pareto ================================================= ------------------------------------------------------------- 0 13642 19392 9012 63 0xff1ddd3f0c8a3b00 ------------------------------------------------------------- 9.22% 7.37% 0.00% 0.00% 0x0 0 1 0xffffffffab344052 518 334 354 5490 160 [k] vma_interval_tree_remove [kernel.kallsyms] vma_interval_tree_remove+18 0 1 0.71% 0.73% 0.00% 0.00% 0x8 0 1 0xffffffffabb9a21f 574 338 458 1991 160 [k] rwsem_down_write_slowpath [kernel.kallsyms] rwsem_down_write_slowpath+655 0 1 0.52% 0.71% 5.34% 6.35% 0x8 0 1 0xffffffffabb9a236 1080 597 390 4848 160 [k] rwsem_down_write_slowpath [kernel.kallsyms] rwsem_down_write_slowpath+678 0 1 0.56% 0.47% 26.39% 6.35% 0x8 0 1 0xffffffffabb9a5ec 1327 1037 587 8537 160 [k] down_write [kernel.kallsyms] down_write+28 0 1 0.11% 0.08% 15.72% 1.59% 0x8 0 1 0xffffffffab17082b 1618 1077 735 7303 160 [k] up_write [kernel.kallsyms] up_write+27 0 1 0.01% 0.02% 0.08% 0.00% 0x8 0 1 0xffffffffabb9a27d 1594 593 512 53 43 [k] rwsem_down_write_slowpath [kernel.kallsyms] rwsem_down_write_slowpath+749 0 1 0.00% 0.01% 0.00% 0.00% 0x8 0 1 0xffffffffabb9a0c4 0 323 518 97 74 [k] rwsem_down_write_slowpath [kernel.kallsyms] rwsem_down_write_slowpath+308 0 1 44.74% 49.78% 0.00% 0.00% 0x10 0 1 0xffffffffab170995 609 344 430 26841 160 [k] rwsem_spin_on_owner [kernel.kallsyms] rwsem_spin_on_owner+53 0 1 26.62% 22.39% 0.00% 0.00% 0x10 0 1 0xffffffffab170965 514 347 437 13364 160 [k] rwsem_spin_on_owner [kernel.kallsyms] rwsem_spin_on_owner+5 0 1 With this change: ------------------------------------------------------------- 0 12726 18554 9039 49 0xff157a0f25b90c40 ------------------------------------------------------------- 0.90% 0.72% 0.00% 0.00% 0x0 1 1 0xffffffffa5f9a21f 532 353 461 2200 160 [k] rwsem_down_write_slowpath [kernel.kallsyms] rwsem_down_write_slowpath+655 0 1 0.53% 0.70% 5.16% 6.12% 0x0 1 1 0xffffffffa5f9a236 1196 670 403 4774 160 [k] rwsem_down_write_slowpath [kernel.kallsyms] rwsem_down_write_slowpath+678 0 1 0.68% 0.51% 25.91% 6.12% 0x0 1 1 0xffffffffa5f9a5ec 1049 807 540 8552 160 [k] down_write [kernel.kallsyms] down_write+28 0 1 0.09% 0.06% 16.50% 2.04% 0x0 1 1 0xffffffffa557082b 1693 1351 758 7317 160 [k] up_write [kernel.kallsyms] up_write+27 0 1 0.01% 0.00% 0.00% 0.00% 0x0 1 1 0xffffffffa5f9a0c4 543 0 491 89 68 [k] rwsem_down_write_slowpath [kernel.kallsyms] rwsem_down_write_slowpath+308 0 1 0.00% 0.01% 0.02% 0.00% 0x0 1 1 0xffffffffa5f9a27d 0 597 742 45 40 [k] rwsem_down_write_slowpath [kernel.kallsyms] rwsem_down_write_slowpath+749 0 1 49.29% 53.01% 0.00% 0.00% 0x8 1 1 0xffffffffa5570995 580 310 413 27106 160 [k] rwsem_spin_on_owner [kernel.kallsyms] rwsem_spin_on_owner+53 0 1 28.60% 24.12% 0.00% 0.00% 0x8 1 1 0xffffffffa5570965 490 321 419 13244 160 [k] rwsem_spin_on_owner [kernel.kallsyms] rwsem_spin_on_owner+5 0 1 Reviewed-by: Tim Chen Signed-off-by: Lipeng Zhu --- include/linux/fs.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/linux/fs.h b/include/linux/fs.h index c85916e9f7db..d3dd8dcc9b8b 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -434,7 +434,7 @@ struct address_space { atomic_t nr_thps; #endif struct rb_root_cached i_mmap; - struct rw_semaphore i_mmap_rwsem; + struct rw_semaphore i_mmap_rwsem ____cacheline_aligned_in_smp; unsigned long nrpages; pgoff_t writeback_index; const struct address_space_operations *a_ops;