From patchwork Mon Mar 29 12:06:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mel Gorman X-Patchwork-Id: 12170001 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 95F2BC433DB for ; Mon, 29 Mar 2021 12:07:03 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 136FA61934 for ; Mon, 29 Mar 2021 12:07:02 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 136FA61934 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=techsingularity.net Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 438BD6B0071; Mon, 29 Mar 2021 08:07:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3E6F06B007D; Mon, 29 Mar 2021 08:07:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2D5C56B007E; Mon, 29 Mar 2021 08:07:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0123.hostedemail.com [216.40.44.123]) by kanga.kvack.org (Postfix) with ESMTP id 0D5256B007D for ; Mon, 29 Mar 2021 08:07:02 -0400 (EDT) Received: from smtpin37.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id A8F9D180373C8 for ; Mon, 29 Mar 2021 12:07:01 +0000 (UTC) X-FDA: 77972785842.37.B69DBE9 Received: from outbound-smtp13.blacknight.com (outbound-smtp13.blacknight.com [46.22.139.230]) by imf19.hostedemail.com (Postfix) with ESMTP id D2E2090009EF for ; Mon, 29 Mar 2021 12:06:53 +0000 (UTC) Received: from mail.blacknight.com (pemlinmail01.blacknight.ie [81.17.254.10]) by outbound-smtp13.blacknight.com (Postfix) with ESMTPS id 3D5731C3019 for ; Mon, 29 Mar 2021 13:06:59 +0100 (IST) Received: (qmail 17289 invoked from network); 29 Mar 2021 12:06:59 -0000 Received: from unknown (HELO stampy.112glenside.lan) (mgorman@techsingularity.net@[84.203.22.4]) by 81.17.254.9 with ESMTPA; 29 Mar 2021 12:06:59 -0000 From: Mel Gorman To: Linux-MM Cc: Linux-RT-Users , LKML , Chuck Lever , Jesper Dangaard Brouer , Matthew Wilcox , Mel Gorman Subject: [RFC PATCH 0/6] Use local_lock for pcp protection and reduce stat overhead Date: Mon, 29 Mar 2021 13:06:42 +0100 Message-Id: <20210329120648.19040-1-mgorman@techsingularity.net> X-Mailer: git-send-email 2.26.2 MIME-Version: 1.0 X-Rspamd-Queue-Id: D2E2090009EF X-Stat-Signature: d53hunkcdkwfdyxksqduo65h4p8ipw68 X-Rspamd-Server: rspam02 Received-SPF: none (techsingularity.net>: No applicable sender policy available) receiver=imf19; identity=mailfrom; envelope-from=""; helo=outbound-smtp13.blacknight.com; client-ip=46.22.139.230 X-HE-DKIM-Result: none/none X-HE-Tag: 1617019613-291024 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This series requires patches in Andrew's tree so the series is also available at git://git.kernel.org/pub/scm/linux/kernel/git/mel/linux.git mm-percpu-local_lock-v1r15 tldr: Jesper and Chuck, it would be nice to verify if this series helps the allocation rate of the bulk page allocator. RT people, this *partially* addresses some problems PREEMPT_RT has with the page allocator but it needs review. The PCP (per-cpu page allocator in page_alloc.c) share locking requirements with vmstat which is inconvenient and causes some issues. Possibly because of that, the PCP list and vmstat share the same per-cpu space meaning that it's possible that vmstat updates dirty cache lines holding per-cpu lists across CPUs unless padding is used. The series splits that structure and separates the locking. Second, PREEMPT_RT considers the following sequence to be unsafe as documented in Documentation/locking/locktypes.rst local_irq_disable(); spin_lock(&lock); The pcp allocator has this sequence for rmqueue_pcplist (local_irq_save) -> __rmqueue_pcplist -> rmqueue_bulk (spin_lock). This series explicitly separates the locking requirements for the PCP list (local_lock) and stat updates (irqs disabled). Once that is done, the length of time IRQs are disabled can be reduced and in some cases, IRQ disabling can be replaced with preempt_disable. After that, it was very obvious that zone_statistics in particular has way too much overhead and leaves IRQs disabled for longer than necessary. It has perfectly accurate counters requiring IRQs be disabled for parallel RMW sequences when inaccurate ones like vm_events would do. The series makes the NUMA statistics (NUMA_HIT and friends) inaccurate counters that only require preempt be disabled. Finally the bulk page allocator can then do all the stat updates in bulk with IRQs enabled which should improve the efficiency of the bulk page allocator. Technically, this could have been done without the local_lock and vmstat conversion work and the order simply reflects the timing of when different series were implemented. No performance data is included because despite the overhead of the stats, it's within the noise for most workloads but Jesper and Chuck may observe a significant different with the same tests used for the bulk page allocator. The series is more likely to be interesting to the RT folk in terms of slowing getting the PREEMPT tree into mainline. drivers/base/node.c | 18 +-- include/linux/mmzone.h | 29 +++-- include/linux/vmstat.h | 65 ++++++----- mm/mempolicy.c | 2 +- mm/page_alloc.c | 173 ++++++++++++++++------------ mm/vmstat.c | 254 +++++++++++++++-------------------------- 6 files changed, 254 insertions(+), 287 deletions(-)