From patchwork Tue Jan 19 07:20:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Feng Tang X-Patchwork-Id: 12029311 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CCFC0C433DB for ; Tue, 19 Jan 2021 07:20:23 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 515D323133 for ; Tue, 19 Jan 2021 07:20:22 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 515D323133 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=intel.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 59B798D002D; Tue, 19 Jan 2021 02:20:22 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 54BA78D002C; Tue, 19 Jan 2021 02:20:22 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3ECA38D002D; Tue, 19 Jan 2021 02:20:22 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0024.hostedemail.com [216.40.44.24]) by kanga.kvack.org (Postfix) with ESMTP id 2541A8D002C for ; Tue, 19 Jan 2021 02:20:22 -0500 (EST) Received: from smtpin09.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id DE45A1EE6 for ; Tue, 19 Jan 2021 07:20:21 +0000 (UTC) X-FDA: 77721676242.09.tax32_5d1812127550 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin09.hostedemail.com (Postfix) with ESMTP id C5DC9180AD822 for ; Tue, 19 Jan 2021 07:20:21 +0000 (UTC) X-HE-Tag: tax32_5d1812127550 X-Filterd-Recvd-Size: 4864 Received: from mga04.intel.com (mga04.intel.com [192.55.52.120]) by imf22.hostedemail.com (Postfix) with ESMTP for ; Tue, 19 Jan 2021 07:20:20 +0000 (UTC) IronPort-SDR: 23M+0eGGfWz0JHBw5Z/+nR/a1ydBbo5SYL8MMUrGSbqCswPJ2ZF5zG4bsF6twBcDarxAu13zoI LPZmixlvDOIw== X-IronPort-AV: E=McAfee;i="6000,8403,9868"; a="176316951" X-IronPort-AV: E=Sophos;i="5.79,358,1602572400"; d="scan'208";a="176316951" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by fmsmga104.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Jan 2021 23:20:17 -0800 IronPort-SDR: +pddMODqK4D9Q1CXSJYFXNT2tmK+1OptCc/ARAdB9VqrDBNI8bnaK4xoxfT7VSAZNhhO5JEAqw dTC2cZ2phryg== X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.79,358,1602572400"; d="scan'208";a="355465943" Received: from shbuild999.sh.intel.com ([10.239.147.98]) by fmsmga008.fm.intel.com with ESMTP; 18 Jan 2021 23:20:15 -0800 From: Feng Tang To: Andrew Morton , Michal Hocko , Johannes Weiner , Roman Gushchin , Shakeel Butt , linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Feng Tang Subject: [PATCH v2] mm: page_counter: relayout structure to reduce false sharing Date: Tue, 19 Jan 2021 15:20:14 +0800 Message-Id: <1611040814-33449-1-git-send-email-feng.tang@intel.com> X-Mailer: git-send-email 2.7.4 MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When checking a memory cgroup related performance regression [1], from the perf c2c profiling data, we found high false sharing for accessing 'usage' and 'parent'. On 64 bit system, the 'usage' and 'parent' are close to each other, and easy to be in one cacheline (for cacheline size == 64+ B). 'usage' is usally written, while 'parent' is usually read as the cgroup's hierarchical counting nature. So move the 'parent' to the end of the structure to make sure they are in different cache lines. Following are some performance data with the patch, against v5.11-rc1. [ In the data, A means a platform with 2 sockets 48C/96T, B is a platform of 4 sockests 72C/144T, and if a %stddev will be shown bigger than 2%, P100/P50 means number of test tasks equals to 100%/50% of nr_cpu] will-it-scale/malloc1 --------------------- v5.11-rc1 v5.11-rc1+patch A-P100 15782 ± 2% -0.1% 15765 ± 3% will-it-scale.per_process_ops A-P50 21511 +8.9% 23432 will-it-scale.per_process_ops B-P100 9155 +2.2% 9357 will-it-scale.per_process_ops B-P50 10967 +7.1% 11751 ± 2% will-it-scale.per_process_ops will-it-scale/pagefault2 ------------------------ v5.11-rc1 v5.11-rc1+patch A-P100 79028 +3.0% 81411 will-it-scale.per_process_ops A-P50 183960 ± 2% +4.4% 192078 ± 2% will-it-scale.per_process_ops B-P100 85966 +9.9% 94467 ± 3% will-it-scale.per_process_ops B-P50 198195 +9.8% 217526 will-it-scale.per_process_ops fio (4k/1M is block size) ------------------------- v5.11-rc1 v5.11-rc1+patch A-P50-r-4k 16881 ± 2% +1.2% 17081 ± 2% fio.read_bw_MBps A-P50-w-4k 3931 +4.5% 4111 ± 2% fio.write_bw_MBps A-P50-r-1M 15178 -0.2% 15154 fio.read_bw_MBps A-P50-w-1M 3924 +0.1% 3929 fio.write_bw_MBps [1].https://lore.kernel.org/lkml/20201102091543.GM31092@shao2-debian/ Signed-off-by: Feng Tang Reviewed-by: Roman Gushchin Cc: Johannes Weiner Cc: Michal Hocko Reviewed-by: Shakeel Butt Acked-by: Johannes Weiner Acked-by: Michal Hocko --- Changelogs: v2: * Adjust the format of performance data to be more readable, as suggested by Michal Hocko include/linux/page_counter.h | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/include/linux/page_counter.h b/include/linux/page_counter.h index 85bd413..6795913 100644 --- a/include/linux/page_counter.h +++ b/include/linux/page_counter.h @@ -12,7 +12,6 @@ struct page_counter { unsigned long low; unsigned long high; unsigned long max; - struct page_counter *parent; /* effective memory.min and memory.min usage tracking */ unsigned long emin; @@ -27,6 +26,14 @@ struct page_counter { /* legacy */ unsigned long watermark; unsigned long failcnt; + + /* + * 'parent' is placed here to be far from 'usage' to reduce + * cache false sharing, as 'usage' is written mostly while + * parent is frequently read for cgroup's hierarchical + * counting nature. + */ + struct page_counter *parent; }; #if BITS_PER_LONG == 32