From patchwork Tue Sep 21 16:13:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicolas Saenz Julienne X-Patchwork-Id: 12508153 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id BF1E9C433FE for ; Tue, 21 Sep 2021 16:13:39 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 311D7611BD for ; Tue, 21 Sep 2021 16:13:39 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 311D7611BD Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 7427A6B0075; Tue, 21 Sep 2021 12:13:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6F2886B0078; Tue, 21 Sep 2021 12:13:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 593A4940007; Tue, 21 Sep 2021 12:13:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0006.hostedemail.com [216.40.44.6]) by kanga.kvack.org (Postfix) with ESMTP id 4A27A6B0075 for ; Tue, 21 Sep 2021 12:13:38 -0400 (EDT) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 0662B8249980 for ; Tue, 21 Sep 2021 16:13:38 +0000 (UTC) X-FDA: 78612076116.11.E23F116 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by imf08.hostedemail.com (Postfix) with ESMTP id 9D3BA30000AF for ; Tue, 21 Sep 2021 16:13:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1632240817; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vXWudoCYk+miAPRPv+924oKRNUMxKATIYtsekauJQeA=; b=ZL3W5rI25QFGxvUzsLLO1XlsKrUxalKP7rsfK1ayCn0ayjE4SB4H/OqoZjIXZFMh9tiaOB Wwm3f0l/gO/5hX6SA9r0Y+7MVSe3QkPiDy05LhN7RRR/WUK4PMx9LL4OHd8tlth9zEr+n/ auidJD7BXIoZI3hIifW25C7GKR3wJ+I= Received: from mail-wr1-f71.google.com (mail-wr1-f71.google.com [209.85.221.71]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-263-wns3EzUZPWi6DNy9wCs69Q-1; Tue, 21 Sep 2021 12:13:35 -0400 X-MC-Unique: wns3EzUZPWi6DNy9wCs69Q-1 Received: by mail-wr1-f71.google.com with SMTP id e1-20020adfa741000000b0015e424fdd01so8809631wrd.11 for ; Tue, 21 Sep 2021 09:13:35 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=vXWudoCYk+miAPRPv+924oKRNUMxKATIYtsekauJQeA=; b=ukXLa/abhVoiJK3A1KUzP2Zqs24eN1LCoiKHRoWkh+mfWBe5rnJmg4KEQpeGefPXkr yWjqjXIImloxidBa2FcySDFl5p50dmlmkeLeV7LRudbalMHlZQUPq0i1uuFdgYZ4FvJF pkjUjYR5vaHXHFYQXzBTUbKOYJA1iUdBTHlwLRkLjrIAdLihy9RdmxY/D0NjTdA2hEiM rmQMu66I48tw7negin1mjL9WqyHUlHYpLGfgEJT7s25bnrE9q4eDQAJBza6zYr0c3+C5 8vYKMCzeNgRm2CTOVtRw4ty5pLw97k8cfDt/C0baPCxvoaQ8hEAH9BsqbSuQmMig+Qi8 5eGg== X-Gm-Message-State: AOAM532quOkhO+s+XjpMA2l0m0qAIId6N3EGucLPEHktKFR7Xnnk93nO Ul5Z8oJUgwtFVuSAewll7V9i/sXvbMOG2XkTGXvkFNUaC6dHuJd477ZKqRtvgsrB0MXu1aftvuj HXPbjyJZtEJ0= X-Received: by 2002:a05:600c:ac5:: with SMTP id c5mr5643291wmr.191.1632240814683; Tue, 21 Sep 2021 09:13:34 -0700 (PDT) X-Google-Smtp-Source: ABdhPJy02a6TNjxvk76ZUdxL9bhj9bDessU6ld3p2HrKam7H2usXY6Tt6rrZgJ6GdFwngIshxJ2HPw== X-Received: by 2002:a05:600c:ac5:: with SMTP id c5mr5643253wmr.191.1632240814460; Tue, 21 Sep 2021 09:13:34 -0700 (PDT) Received: from vian.redhat.com ([2a0c:5a80:1d03:b900:c3d1:5974:ce92:3123]) by smtp.gmail.com with ESMTPSA id t1sm19786477wrz.39.2021.09.21.09.13.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Sep 2021 09:13:34 -0700 (PDT) From: Nicolas Saenz Julienne To: akpm@linux-foundation.org, frederic@kernel.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, tglx@linutronix.de, cl@linux.com, peterz@infradead.org, juri.lelli@redhat.com, mingo@redhat.com, mtosatti@redhat.com, nilal@redhat.com, mgorman@suse.de, ppandit@redhat.com, williams@redhat.com, bigeasy@linutronix.de, anna-maria@linutronix.de, linux-rt-users@vger.kernel.org, Nicolas Saenz Julienne Subject: [PATCH 1/6] mm/swap: Introduce lru_cpu_needs_drain() Date: Tue, 21 Sep 2021 18:13:19 +0200 Message-Id: <20210921161323.607817-2-nsaenzju@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210921161323.607817-1-nsaenzju@redhat.com> References: <20210921161323.607817-1-nsaenzju@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ZL3W5rI2; spf=none (imf08.hostedemail.com: domain of nsaenzju@redhat.com has no SPF policy when checking 216.205.24.124) smtp.mailfrom=nsaenzju@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 9D3BA30000AF X-Stat-Signature: phr1hs3k6corkke8t3esbkwf4oh31gwz X-HE-Tag: 1632240817-451366 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000208, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Upcoming patches will need to check whether a CPU needs to drain its LRU pagevecs on multiple locations. So move the check into its own function. Signed-off-by: Nicolas Saenz Julienne --- mm/swap.c | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/mm/swap.c b/mm/swap.c index 04b678342c02..e7f9e4018ccf 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -726,6 +726,17 @@ static void lru_add_drain_per_cpu(struct work_struct *dummy) lru_add_drain(); } +static bool lru_cpu_needs_drain(int cpu) +{ + return pagevec_count(&per_cpu(lru_pvecs.lru_add, cpu)) || + data_race(pagevec_count(&per_cpu(lru_rotate.pvec, cpu))) || + pagevec_count(&per_cpu(lru_pvecs.lru_deactivate_file, cpu)) || + pagevec_count(&per_cpu(lru_pvecs.lru_deactivate, cpu)) || + pagevec_count(&per_cpu(lru_pvecs.lru_lazyfree, cpu)) || + need_activate_page_drain(cpu) || + has_bh_in_lru(cpu, NULL); +} + /* * Doesn't need any cpu hotplug locking because we do rely on per-cpu * kworkers being shut down before our page_alloc_cpu_dead callback is @@ -808,14 +819,7 @@ inline void __lru_add_drain_all(bool force_all_cpus) for_each_online_cpu(cpu) { struct work_struct *work = &per_cpu(lru_add_drain_work, cpu); - if (force_all_cpus || - pagevec_count(&per_cpu(lru_pvecs.lru_add, cpu)) || - data_race(pagevec_count(&per_cpu(lru_rotate.pvec, cpu))) || - pagevec_count(&per_cpu(lru_pvecs.lru_deactivate_file, cpu)) || - pagevec_count(&per_cpu(lru_pvecs.lru_deactivate, cpu)) || - pagevec_count(&per_cpu(lru_pvecs.lru_lazyfree, cpu)) || - need_activate_page_drain(cpu) || - has_bh_in_lru(cpu, NULL)) { + if (force_all_cpus || lru_cpu_needs_drain(cpu)) { INIT_WORK(work, lru_add_drain_per_cpu); queue_work_on(cpu, mm_percpu_wq, work); __cpumask_set_cpu(cpu, &has_work); From patchwork Tue Sep 21 16:13:20 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicolas Saenz Julienne X-Patchwork-Id: 12508155 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AAD1DC433EF for ; Tue, 21 Sep 2021 16:13:42 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 3C1426126A for ; Tue, 21 Sep 2021 16:13:42 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 3C1426126A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id C72D26B0078; Tue, 21 Sep 2021 12:13:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C22796B007B; Tue, 21 Sep 2021 12:13:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A9B4E6B007D; Tue, 21 Sep 2021 12:13:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0067.hostedemail.com [216.40.44.67]) by kanga.kvack.org (Postfix) with ESMTP id 9C1B86B0078 for ; Tue, 21 Sep 2021 12:13:41 -0400 (EDT) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 64A4C2D224 for ; Tue, 21 Sep 2021 16:13:41 +0000 (UTC) X-FDA: 78612076242.17.4EE4376 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by imf12.hostedemail.com (Postfix) with ESMTP id ECBDF10000AB for ; Tue, 21 Sep 2021 16:13:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1632240820; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=6mnRYMlCMcOKDKQfbrjGHyzvFIZ6fLhADshZ7k2vuLI=; b=Mp3icW+NlvR190nCq0xbOoydy6LsdwB7eArVGcabpnADtzSIPmLwSYkVTSoIZIGS3lqTiX yXPUZ3iTSV5xox5kKnukYi3R950BW4bUUQv5TvLAwkuEgrdtt9l2e2TbWhR2TEFQgpLOTc NBhxbtfTOxotL+kCGpDZLwYoJze/GOo= Received: from mail-wr1-f70.google.com (mail-wr1-f70.google.com [209.85.221.70]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-265-mhnW-JTUMLiJsMvtetrQTw-1; Tue, 21 Sep 2021 12:13:37 -0400 X-MC-Unique: mhnW-JTUMLiJsMvtetrQTw-1 Received: by mail-wr1-f70.google.com with SMTP id c15-20020a5d4ccf000000b0015dff622f39so9243751wrt.21 for ; Tue, 21 Sep 2021 09:13:37 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=6mnRYMlCMcOKDKQfbrjGHyzvFIZ6fLhADshZ7k2vuLI=; b=MOOrgx0Xv4ah/41KPnZSDGhft2Zp5LuefOG1jLn3cJew+MFR81SD+TNm6b6FKjOznu JQcRABaZI+wiLCcg+iVOf6EFsFRmIhd3Ba4U2tQ9jq7FHhCsF2BzGOgTDm8WmLJi2cX+ wRvzMYhMtoRZ4/Y5LS5hhchaRCYo+zwCSnbtTdb/YYw9CztrefQS9zl3FVIHaO1qHR9t dGFLuEAwoDdYAFK8fPhbguxlVhNHduF/6vqdQ54Bu5iIt1Nf0iqFxV8ZBXXfkBs/oiue I+8ZAyLdvdotat2P3K0n/ICTRvIic7yg7DzSm9H6EEDvV/JL+dlwrmgonEdE1D77nCBB H0fQ== X-Gm-Message-State: AOAM530IvoqiAFDFJBv/sJbUWYWIng8lRygKF+TxNdiOFX8RvoN0TRRW RgbzIrDB9NanDsGN2ksOgquJqznwldmvq/vO860PjtYduIWQvFIg1idqOtSx4VkA9tEa/4wszfQ rrYFZCiMzYt0= X-Received: by 2002:a1c:c918:: with SMTP id f24mr5518711wmb.61.1632240815855; Tue, 21 Sep 2021 09:13:35 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyVoZ7kYejG472I/F9ywvyNwhKIXJJdZ5S1KHl3KZ2v+nsWpdgVQIW8uMFSa48gONQWppIi6Q== X-Received: by 2002:a1c:c918:: with SMTP id f24mr5518680wmb.61.1632240815623; Tue, 21 Sep 2021 09:13:35 -0700 (PDT) Received: from vian.redhat.com ([2a0c:5a80:1d03:b900:c3d1:5974:ce92:3123]) by smtp.gmail.com with ESMTPSA id t1sm19786477wrz.39.2021.09.21.09.13.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Sep 2021 09:13:35 -0700 (PDT) From: Nicolas Saenz Julienne To: akpm@linux-foundation.org, frederic@kernel.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, tglx@linutronix.de, cl@linux.com, peterz@infradead.org, juri.lelli@redhat.com, mingo@redhat.com, mtosatti@redhat.com, nilal@redhat.com, mgorman@suse.de, ppandit@redhat.com, williams@redhat.com, bigeasy@linutronix.de, anna-maria@linutronix.de, linux-rt-users@vger.kernel.org, Nicolas Saenz Julienne Subject: [PATCH 2/6] mm/swap: Introduce alternative per-cpu LRU cache locking Date: Tue, 21 Sep 2021 18:13:20 +0200 Message-Id: <20210921161323.607817-3-nsaenzju@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210921161323.607817-1-nsaenzju@redhat.com> References: <20210921161323.607817-1-nsaenzju@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Mp3icW+N; spf=none (imf12.hostedemail.com: domain of nsaenzju@redhat.com has no SPF policy when checking 216.205.24.124) smtp.mailfrom=nsaenzju@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Stat-Signature: iuhi3i1zmoeb6gyctwbe9qcr6696uigj X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: ECBDF10000AB X-HE-Tag: 1632240820-382258 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: 'struct lru_pvecs' and 'struct lru_rotate' per-CPU pagevecs are protected using local locks. While performance savvy, this doesn't allow for remote access to these structures. CPUs requiring system-wide LRU cache drains get around this by scheduling drain work on all CPUs. That said, some select setups like systems with NOHZ_FULL CPUs, aren't well suited to this, as they can't handle interruptions of any sort. To mitigate this, introduce an alternative locking scheme using spinlocks that will permit remotely accessing these per-CPU pagevecs. It's disabled by default, with no functional change to regular users, and enabled through the 'remote_pcpu_cache_access' static key. Upcoming patches will make use of this static key. The naming of the static key is left vague on purpose as it is planned to also enable a similar locking setup to access mm/page_alloc.c's per-cpu page lists remotely. This is based on previous work by Thomas Gleixner, Anna-Maria Gleixner, and Sebastian Andrzej Siewior[1]. [1] https://patchwork.kernel.org/project/linux-mm/patch/20190424111208.24459-3-bigeasy@linutronix.de/ Signed-off-by: Nicolas Saenz Julienne --- mm/internal.h | 2 + mm/swap.c | 152 +++++++++++++++++++++++++++++++++++++++++--------- 2 files changed, 128 insertions(+), 26 deletions(-) diff --git a/mm/internal.h b/mm/internal.h index 18256e32a14c..5a2cef7cd394 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -32,6 +32,8 @@ /* Do not use these with a slab allocator */ #define GFP_SLAB_BUG_MASK (__GFP_DMA32|__GFP_HIGHMEM|~__GFP_BITS_MASK) +extern struct static_key_false remote_pcpu_cache_access; + void page_writeback_init(void); static inline void *folio_raw_mapping(struct folio *folio) diff --git a/mm/swap.c b/mm/swap.c index e7f9e4018ccf..bcf73bd563a6 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -46,13 +46,27 @@ /* How many pages do we try to swap or page in/out together? */ int page_cluster; +/* + * On some setups, like with nohz_full, CPUs might be too busy to handle + * per-cpu drain work, leading to unwarranted interruptions and hangs. This + * key, when enabled, allows for remote draining of these per-cpu caches/page + * lists at the cost of more constraining locking. + */ +__ro_after_init DEFINE_STATIC_KEY_FALSE(remote_pcpu_cache_access); + +struct lru_cache_locks { + local_lock_t local; + spinlock_t spin; +}; + /* Protecting only lru_rotate.pvec which requires disabling interrupts */ struct lru_rotate { - local_lock_t lock; + struct lru_cache_locks locks; struct pagevec pvec; }; static DEFINE_PER_CPU(struct lru_rotate, lru_rotate) = { - .lock = INIT_LOCAL_LOCK(lock), + .locks.local = INIT_LOCAL_LOCK(lru_rotate.locks.local), + .locks.spin = __SPIN_LOCK_UNLOCKED(lru_rotate.locks.spin), }; /* @@ -60,7 +74,7 @@ static DEFINE_PER_CPU(struct lru_rotate, lru_rotate) = { * by disabling preemption (and interrupts remain enabled). */ struct lru_pvecs { - local_lock_t lock; + struct lru_cache_locks locks; struct pagevec lru_add; struct pagevec lru_deactivate_file; struct pagevec lru_deactivate; @@ -70,9 +84,94 @@ struct lru_pvecs { #endif }; static DEFINE_PER_CPU(struct lru_pvecs, lru_pvecs) = { - .lock = INIT_LOCAL_LOCK(lock), + .locks.local = INIT_LOCAL_LOCK(lru_pvecs.locks.local), + .locks.spin = __SPIN_LOCK_UNLOCKED(lru_pvecs.locks.spin), }; +static inline void lru_cache_lock(struct lru_cache_locks *locks) +{ + if (static_branch_unlikely(&remote_pcpu_cache_access)) { + /* Avoid migration between this_cpu_ptr() and spin_lock() */ + migrate_disable(); + spin_lock(this_cpu_ptr(&locks->spin)); + } else { + local_lock(&locks->local); + } +} + +static inline void lru_cache_lock_irqsave(struct lru_cache_locks *locks, + unsigned long *flagsp) +{ + if (static_branch_unlikely(&remote_pcpu_cache_access)) { + /* Avoid migration between this_cpu_ptr() and spin_lock_irqsave() */ + migrate_disable(); + spin_lock_irqsave(this_cpu_ptr(&locks->spin), *flagsp); + } else { + local_lock_irqsave(&locks->local, *flagsp); + } +} + +/* + * The lru_cache_lock_cpu()/lru_cache_lock_irqsave_cpu() flavor of functions + * should only be used from remote CPUs when 'remote_pcpu_cache_access' is + * enabled or the target CPU is dead. Otherwise, it can still be called on the + * local CPU with migration disabled. + */ +static inline void lru_cache_lock_cpu(struct lru_cache_locks *locks, int cpu) +{ + if (static_branch_unlikely(&remote_pcpu_cache_access)) + spin_lock(per_cpu_ptr(&locks->spin, cpu)); + else + local_lock(&locks->local); +} + +static inline void lru_cache_lock_irqsave_cpu(struct lru_cache_locks *locks, + unsigned long *flagsp, int cpu) +{ + if (static_branch_unlikely(&remote_pcpu_cache_access)) + spin_lock_irqsave(per_cpu_ptr(&locks->spin, cpu), *flagsp); + else + local_lock_irqsave(&locks->local, *flagsp); +} + +static inline void lru_cache_unlock(struct lru_cache_locks *locks) +{ + if (static_branch_unlikely(&remote_pcpu_cache_access)) { + spin_unlock(this_cpu_ptr(&locks->spin)); + migrate_enable(); + } else { + local_unlock(&locks->local); + } +} + +static inline void lru_cache_unlock_irqrestore(struct lru_cache_locks *locks, + unsigned long flags) +{ + if (static_branch_unlikely(&remote_pcpu_cache_access)) { + spin_unlock_irqrestore(this_cpu_ptr(&locks->spin), flags); + migrate_enable(); + } else { + local_unlock_irqrestore(&locks->local, flags); + } +} + +static inline void lru_cache_unlock_cpu(struct lru_cache_locks *locks, int cpu) +{ + if (static_branch_unlikely(&remote_pcpu_cache_access)) + spin_unlock(per_cpu_ptr(&locks->spin, cpu)); + else + local_unlock(&locks->local); +} + +static inline void lru_cache_unlock_irqrestore_cpu(struct lru_cache_locks *locks, + unsigned long flags, int cpu) +{ + if (static_branch_unlikely(&remote_pcpu_cache_access)) + spin_unlock_irqrestore(per_cpu_ptr(&locks->spin, cpu), flags); + else + local_unlock_irqrestore(&locks->local, flags); +} + /* * This path almost never happens for VM activity - pages are normally * freed via pagevecs. But it gets used by networking. @@ -245,11 +344,11 @@ void folio_rotate_reclaimable(struct folio *folio) unsigned long flags; folio_get(folio); - local_lock_irqsave(&lru_rotate.lock, flags); + lru_cache_lock_irqsave(&lru_rotate.locks, &flags); pvec = this_cpu_ptr(&lru_rotate.pvec); if (pagevec_add_and_need_flush(pvec, &folio->page)) pagevec_lru_move_fn(pvec, pagevec_move_tail_fn); - local_unlock_irqrestore(&lru_rotate.lock, flags); + lru_cache_unlock_irqrestore(&lru_rotate.locks, flags); } } @@ -341,11 +440,11 @@ static void folio_activate(struct folio *folio) struct pagevec *pvec; folio_get(folio); - local_lock(&lru_pvecs.lock); + lru_cache_lock(&lru_pvecs.locks); pvec = this_cpu_ptr(&lru_pvecs.activate_page); if (pagevec_add_and_need_flush(pvec, &folio->page)) pagevec_lru_move_fn(pvec, __activate_page); - local_unlock(&lru_pvecs.lock); + lru_cache_unlock(&lru_pvecs.locks); } } @@ -372,7 +471,7 @@ static void __lru_cache_activate_folio(struct folio *folio) struct pagevec *pvec; int i; - local_lock(&lru_pvecs.lock); + lru_cache_lock(&lru_pvecs.locks); pvec = this_cpu_ptr(&lru_pvecs.lru_add); /* @@ -394,7 +493,7 @@ static void __lru_cache_activate_folio(struct folio *folio) } } - local_unlock(&lru_pvecs.lock); + lru_cache_unlock(&lru_pvecs.locks); } /* @@ -453,11 +552,11 @@ void folio_add_lru(struct folio *folio) VM_BUG_ON_FOLIO(folio_test_lru(folio), folio); folio_get(folio); - local_lock(&lru_pvecs.lock); + lru_cache_lock(&lru_pvecs.locks); pvec = this_cpu_ptr(&lru_pvecs.lru_add); if (pagevec_add_and_need_flush(pvec, &folio->page)) __pagevec_lru_add(pvec); - local_unlock(&lru_pvecs.lock); + lru_cache_unlock(&lru_pvecs.locks); } EXPORT_SYMBOL(folio_add_lru); @@ -592,8 +691,9 @@ static void lru_lazyfree_fn(struct page *page, struct lruvec *lruvec) /* * Drain pages out of the cpu's pagevecs. - * Either "cpu" is the current CPU, and preemption has already been - * disabled; or "cpu" is being hot-unplugged, and is already dead. + * Either "cpu" is the current CPU, and preemption has already been disabled, + * or we're remotely flushing pvecs with the 'remote_pcpu_cache_access' key + * enabled, or "cpu" is being hot-unplugged and is already dead. */ void lru_add_drain_cpu(int cpu) { @@ -608,9 +708,9 @@ void lru_add_drain_cpu(int cpu) unsigned long flags; /* No harm done if a racing interrupt already did this */ - local_lock_irqsave(&lru_rotate.lock, flags); + lru_cache_lock_irqsave_cpu(&lru_rotate.locks, &flags, cpu); pagevec_lru_move_fn(pvec, pagevec_move_tail_fn); - local_unlock_irqrestore(&lru_rotate.lock, flags); + lru_cache_unlock_irqrestore_cpu(&lru_rotate.locks, flags, cpu); } pvec = &per_cpu(lru_pvecs.lru_deactivate_file, cpu); @@ -649,12 +749,12 @@ void deactivate_file_page(struct page *page) if (likely(get_page_unless_zero(page))) { struct pagevec *pvec; - local_lock(&lru_pvecs.lock); + lru_cache_lock(&lru_pvecs.locks); pvec = this_cpu_ptr(&lru_pvecs.lru_deactivate_file); if (pagevec_add_and_need_flush(pvec, page)) pagevec_lru_move_fn(pvec, lru_deactivate_file_fn); - local_unlock(&lru_pvecs.lock); + lru_cache_unlock(&lru_pvecs.locks); } } @@ -671,12 +771,12 @@ void deactivate_page(struct page *page) if (PageLRU(page) && PageActive(page) && !PageUnevictable(page)) { struct pagevec *pvec; - local_lock(&lru_pvecs.lock); + lru_cache_lock(&lru_pvecs.locks); pvec = this_cpu_ptr(&lru_pvecs.lru_deactivate); get_page(page); if (pagevec_add_and_need_flush(pvec, page)) pagevec_lru_move_fn(pvec, lru_deactivate_fn); - local_unlock(&lru_pvecs.lock); + lru_cache_unlock(&lru_pvecs.locks); } } @@ -693,28 +793,28 @@ void mark_page_lazyfree(struct page *page) !PageSwapCache(page) && !PageUnevictable(page)) { struct pagevec *pvec; - local_lock(&lru_pvecs.lock); + lru_cache_lock(&lru_pvecs.locks); pvec = this_cpu_ptr(&lru_pvecs.lru_lazyfree); get_page(page); if (pagevec_add_and_need_flush(pvec, page)) pagevec_lru_move_fn(pvec, lru_lazyfree_fn); - local_unlock(&lru_pvecs.lock); + lru_cache_unlock(&lru_pvecs.locks); } } void lru_add_drain(void) { - local_lock(&lru_pvecs.lock); + lru_cache_lock(&lru_pvecs.locks); lru_add_drain_cpu(smp_processor_id()); - local_unlock(&lru_pvecs.lock); + lru_cache_unlock(&lru_pvecs.locks); } void lru_add_drain_cpu_zone(struct zone *zone) { - local_lock(&lru_pvecs.lock); + lru_cache_lock(&lru_pvecs.locks); lru_add_drain_cpu(smp_processor_id()); drain_local_pages(zone); - local_unlock(&lru_pvecs.lock); + lru_cache_unlock(&lru_pvecs.locks); } #ifdef CONFIG_SMP From patchwork Tue Sep 21 16:13:21 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicolas Saenz Julienne X-Patchwork-Id: 12508157 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0CB20C4332F for ; Tue, 21 Sep 2021 16:13:44 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id ACD5B610A1 for ; Tue, 21 Sep 2021 16:13:43 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org ACD5B610A1 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 4D2236B007B; Tue, 21 Sep 2021 12:13:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 482EE6B007D; Tue, 21 Sep 2021 12:13:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2D4DE940007; Tue, 21 Sep 2021 12:13:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0100.hostedemail.com [216.40.44.100]) by kanga.kvack.org (Postfix) with ESMTP id 1F1C36B007B for ; Tue, 21 Sep 2021 12:13:43 -0400 (EDT) Received: from smtpin27.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id D679B1813CB8D for ; Tue, 21 Sep 2021 16:13:42 +0000 (UTC) X-FDA: 78612076284.27.AAB2376 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf06.hostedemail.com (Postfix) with ESMTP id 7F301801A8A7 for ; Tue, 21 Sep 2021 16:13:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1632240822; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Skz+wSCIMyvocC+cWmr1kqnk80+x9gFo8RgnyvxY9jI=; b=gAMB2NjgJr+vAF4dvhZslkQxSUgOsgAc4E+sxpR0VcJLbOOmkydb+hTLJX4BcO+T+EoMhQ tyvx5/GP/bPTDZwR4rG4eu9w1eiNYY4/FrQqS3BOTvSt/5TGGMYzJAYQ3q6se7tM0vJesM iGP0gPCNB9TkbuMpDVL3sXFKQOCPZrc= Received: from mail-wr1-f71.google.com (mail-wr1-f71.google.com [209.85.221.71]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-420-5x2JoL9ZNQKq_AHU7iDtLw-1; Tue, 21 Sep 2021 12:13:38 -0400 X-MC-Unique: 5x2JoL9ZNQKq_AHU7iDtLw-1 Received: by mail-wr1-f71.google.com with SMTP id c15-20020a5d4ccf000000b0015dff622f39so9243822wrt.21 for ; Tue, 21 Sep 2021 09:13:38 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Skz+wSCIMyvocC+cWmr1kqnk80+x9gFo8RgnyvxY9jI=; b=GbSueWdZC7XgHaRP5rZPCVvupLTomb1ANKifHgcKeJ24mF1ZYbEI39aC4DvKbKN0OW FtklEON/Xi6u0jWMg1JZutAxrtMBgBWlYzf3ZY68d9sj2v0WaSrFWJisFw2IJIrO54w+ uRxX2Rm0QRkrED+dQK6P0zsEtZ99DoBymSVcspedAqTDcYjkRUA1cnMM5B1h6Mc/UOpn ez8FQVNwA+G1vUSXvro95SmMnBqyJAVmf73oMQLtwspJeBd1UlsqYqVSfZfmUqk3kKmg uwtFpgy8u2sFoXmMVM9qqve2uHKfQ4K10lFAFQ7+bSfeCjFmnCd4kmjQRUTuJJzcVKud iXyg== X-Gm-Message-State: AOAM533NNGiPloPWlgQuigmpF6yNsJtDkJm5ZLvDvP3xuTGBBMqQfKPx G/LVr24MDRKZ69IN5aVlbdyNcqbWMooRm6rQds18MO0vdqas7Q8R5peKWaquaVZY+Ltgsl2+cuU +5ozoxvISex0= X-Received: by 2002:a05:6000:c3:: with SMTP id q3mr35713557wrx.361.1632240817400; Tue, 21 Sep 2021 09:13:37 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwc8lfAIXvvAGBd2lf3kHl9jl9Ma+SH7iopQjrQOmNgpGD7Dcwtktf6tIufdAsdVPxyyoBuiQ== X-Received: by 2002:a05:6000:c3:: with SMTP id q3mr35713524wrx.361.1632240817263; Tue, 21 Sep 2021 09:13:37 -0700 (PDT) Received: from vian.redhat.com ([2a0c:5a80:1d03:b900:c3d1:5974:ce92:3123]) by smtp.gmail.com with ESMTPSA id t1sm19786477wrz.39.2021.09.21.09.13.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Sep 2021 09:13:37 -0700 (PDT) From: Nicolas Saenz Julienne To: akpm@linux-foundation.org, frederic@kernel.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, tglx@linutronix.de, cl@linux.com, peterz@infradead.org, juri.lelli@redhat.com, mingo@redhat.com, mtosatti@redhat.com, nilal@redhat.com, mgorman@suse.de, ppandit@redhat.com, williams@redhat.com, bigeasy@linutronix.de, anna-maria@linutronix.de, linux-rt-users@vger.kernel.org, Nicolas Saenz Julienne Subject: [PATCH 3/6] mm/swap: Allow remote LRU cache draining Date: Tue, 21 Sep 2021 18:13:21 +0200 Message-Id: <20210921161323.607817-4-nsaenzju@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210921161323.607817-1-nsaenzju@redhat.com> References: <20210921161323.607817-1-nsaenzju@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=gAMB2Njg; spf=none (imf06.hostedemail.com: domain of nsaenzju@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=nsaenzju@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 7F301801A8A7 X-Stat-Signature: eykpykhf7bbt97rtmewoi5dgdnd45xxq X-HE-Tag: 1632240822-692683 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000005, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Some setups, notably NOHZ_FULL CPUs, are too busy to handle the per-cpu drain work queued by __lru_add_drain_all(). So introduce new a mechanism to remotely drain the per-cpu lists. It is made possible by using a more constraining locking scheme, which is disabled by default and can be enabled through the 'remote_pcpu_cache_access' static key. Regular users shouldn't see any functional or performance change. Upcoming patches will make use of the key. Based on previous work by Thomas Gleixner, Anna-Maria Gleixner, and Sebastian Andrzej Siewior[1]. [1] https://patchwork.kernel.org/project/linux-mm/cover/20190424111208.24459-1-bigeasy@linutronix.de/ Signed-off-by: Nicolas Saenz Julienne --- mm/swap.c | 32 +++++++++++++++++++++----------- 1 file changed, 21 insertions(+), 11 deletions(-) diff --git a/mm/swap.c b/mm/swap.c index bcf73bd563a6..59e96a2520d5 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -915,19 +915,29 @@ inline void __lru_add_drain_all(bool force_all_cpus) WRITE_ONCE(lru_drain_gen, lru_drain_gen + 1); smp_mb(); - cpumask_clear(&has_work); - for_each_online_cpu(cpu) { - struct work_struct *work = &per_cpu(lru_add_drain_work, cpu); - - if (force_all_cpus || lru_cpu_needs_drain(cpu)) { - INIT_WORK(work, lru_add_drain_per_cpu); - queue_work_on(cpu, mm_percpu_wq, work); - __cpumask_set_cpu(cpu, &has_work); + if (static_branch_unlikely(&remote_pcpu_cache_access)) { + for_each_online_cpu(cpu) { + if (force_all_cpus || lru_cpu_needs_drain(cpu)) { + lru_cache_lock_cpu(&lru_pvecs.locks, cpu); + lru_add_drain_cpu(cpu); + lru_cache_unlock_cpu(&lru_pvecs.locks, cpu); + } + } + } else { + cpumask_clear(&has_work); + for_each_online_cpu(cpu) { + struct work_struct *work = &per_cpu(lru_add_drain_work, cpu); + + if (force_all_cpus || lru_cpu_needs_drain(cpu)) { + INIT_WORK(work, lru_add_drain_per_cpu); + queue_work_on(cpu, mm_percpu_wq, work); + __cpumask_set_cpu(cpu, &has_work); + } } - } - for_each_cpu(cpu, &has_work) - flush_work(&per_cpu(lru_add_drain_work, cpu)); + for_each_cpu(cpu, &has_work) + flush_work(&per_cpu(lru_add_drain_work, cpu)); + } done: mutex_unlock(&lock); From patchwork Tue Sep 21 16:13:22 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicolas Saenz Julienne X-Patchwork-Id: 12508159 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 54465C4332F for ; Tue, 21 Sep 2021 16:13:46 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 069F8610A1 for ; Tue, 21 Sep 2021 16:13:45 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 069F8610A1 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id A9E466B007D; Tue, 21 Sep 2021 12:13:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A29FF6B007E; Tue, 21 Sep 2021 12:13:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7DD3E940007; Tue, 21 Sep 2021 12:13:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0177.hostedemail.com [216.40.44.177]) by kanga.kvack.org (Postfix) with ESMTP id 6F32C6B007D for ; Tue, 21 Sep 2021 12:13:43 -0400 (EDT) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 2EB302D4C4 for ; Tue, 21 Sep 2021 16:13:43 +0000 (UTC) X-FDA: 78612076326.12.314A844 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf16.hostedemail.com (Postfix) with ESMTP id DB447F00008E for ; Tue, 21 Sep 2021 16:13:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1632240822; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gPP7p4f7+ZBDSuJ4Fq24/tKP4H9YfAjznqdigXY0rzo=; b=FJzT0BG4v/1Bz0q1QYvMxhxj1X+U3AnTfGCoXc8Zups642a177ja1MvY0BnTgLgv62AAXr ZWJY2phQ9oL+Ssu3QCvkxi3kRAxgCujmOOj+B0qHjH7y1lMwp1yusyfjgId5kkp6kPI/7z OMuRvxBlsBqeNCOVRxp9jlp4NZ5X+Sc= Received: from mail-wr1-f71.google.com (mail-wr1-f71.google.com [209.85.221.71]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-590-lMUKkPZsNjCU3shjzrdyjw-1; Tue, 21 Sep 2021 12:13:40 -0400 X-MC-Unique: lMUKkPZsNjCU3shjzrdyjw-1 Received: by mail-wr1-f71.google.com with SMTP id s14-20020adff80e000000b001601b124f50so2435834wrp.5 for ; Tue, 21 Sep 2021 09:13:40 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=gPP7p4f7+ZBDSuJ4Fq24/tKP4H9YfAjznqdigXY0rzo=; b=iY1jAnfJxy9X4118E4dDoYFxdsv7UJd1DVLc37T3VUY4T7h88XtVSWxnhWxR+EHatd sglzaAu2v2Xi2RWRLrD7pHHOSniXU5jFaxI5YRgwUcwSBn3jpuQHijxk0I7A0sJHZxwN mOPtL2i39P0dkUz7E/9rhaqW0vaBBw768DxkSeUyYA9VQsgIkzNNXLk4iFP3pGllqSkF EGhHiNaujGdrcEinJK1wo3ebu2HXnHYLrHel1HlHk2eIi5fpUjmVQuAZ9LASrHgUH2pk 1mUOXLNB+fZNgxvs3Sg+D87VH1tbp82xl//n47oOFVIyCUr+DuKy1nkoIaxvjX6YuSiD KUCQ== X-Gm-Message-State: AOAM533+l+znp+1YKw00+zkxoN1UFk6X01V93OfX0NSclvIXYCnvQ61O e6VPYU0X0bOY2thBAnqh8Pjb7olUYrXMFa3awTijEnUE1uONhgydngZZyibcrqUodaTNpyC+2Ds i9Sl3b44aTfw= X-Received: by 2002:a05:600c:4105:: with SMTP id j5mr5481448wmi.138.1632240819125; Tue, 21 Sep 2021 09:13:39 -0700 (PDT) X-Google-Smtp-Source: ABdhPJzOTu0J3jQPhjB0rCNk6fDnFpnh+/g5dnPXrEPxVjKH2eCKCxgBKKRAmpV4rD/xGG21/HWfXg== X-Received: by 2002:a05:600c:4105:: with SMTP id j5mr5481424wmi.138.1632240818845; Tue, 21 Sep 2021 09:13:38 -0700 (PDT) Received: from vian.redhat.com ([2a0c:5a80:1d03:b900:c3d1:5974:ce92:3123]) by smtp.gmail.com with ESMTPSA id t1sm19786477wrz.39.2021.09.21.09.13.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Sep 2021 09:13:38 -0700 (PDT) From: Nicolas Saenz Julienne To: akpm@linux-foundation.org, frederic@kernel.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, tglx@linutronix.de, cl@linux.com, peterz@infradead.org, juri.lelli@redhat.com, mingo@redhat.com, mtosatti@redhat.com, nilal@redhat.com, mgorman@suse.de, ppandit@redhat.com, williams@redhat.com, bigeasy@linutronix.de, anna-maria@linutronix.de, linux-rt-users@vger.kernel.org, Nicolas Saenz Julienne Subject: [PATCH 4/6] mm/page_alloc: Introduce alternative per-cpu list locking Date: Tue, 21 Sep 2021 18:13:22 +0200 Message-Id: <20210921161323.607817-5-nsaenzju@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210921161323.607817-1-nsaenzju@redhat.com> References: <20210921161323.607817-1-nsaenzju@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=FJzT0BG4; spf=none (imf16.hostedemail.com: domain of nsaenzju@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=nsaenzju@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: DB447F00008E X-Stat-Signature: hithdk1sfjfe6w9fg4g53nbjieb41e7r X-HE-Tag: 1632240822-200656 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: page_alloc.c's per-cpu page lists are currently protected using local locks. While performance savvy, this doesn't allow for remote access to these structures. CPUs requiring system-wide per-cpu list drains get around this by scheduling drain work on all CPUs. That said, some select setups like systems with NOHZ_FULL CPUs, aren't well suited to this, as they can't handle interruptions of any sort. To mitigate this, introduce an alternative locking scheme using spinlocks that will permit remotely accessing these per-cpu page lists. It's disabled by default, with no functional change to regular users, and enabled through the 'remote_pcpu_cache_access' static key. Upcoming patches will make use of this static key. This is based on previous work by Thomas Gleixner, Anna-Maria Gleixner, and Sebastian Andrzej Siewior[1]. [1] https://patchwork.kernel.org/project/linux-mm/patch/20190424111208.24459-3-bigeasy@linutronix.de/ Signed-off-by: Nicolas Saenz Julienne --- mm/page_alloc.c | 87 ++++++++++++++++++++++++++++++++++++++----------- 1 file changed, 68 insertions(+), 19 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 3b610b05d9b8..3244eb2ab51b 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -123,10 +123,12 @@ static DEFINE_MUTEX(pcp_batch_high_lock); #define MIN_PERCPU_PAGELIST_HIGH_FRACTION (8) struct pagesets { - local_lock_t lock; + local_lock_t local; + spinlock_t spin; }; static DEFINE_PER_CPU(struct pagesets, pagesets) = { - .lock = INIT_LOCAL_LOCK(lock), + .local = INIT_LOCAL_LOCK(pagesets.local), + .spin = __SPIN_LOCK_UNLOCKED(pagesets.spin), }; #ifdef CONFIG_USE_PERCPU_NUMA_NODE_ID @@ -207,6 +209,52 @@ static int __init early_init_on_free(char *buf) } early_param("init_on_free", early_init_on_free); +static inline void pagesets_lock_irqsave(struct pagesets *locks, + unsigned long *flagsp) +{ + if (static_branch_unlikely(&remote_pcpu_cache_access)) { + /* Avoid migration between this_cpu_ptr() and spin_lock_irqsave() */ + migrate_disable(); + spin_lock_irqsave(this_cpu_ptr(&locks->spin), *flagsp); + } else { + local_lock_irqsave(&locks->local, *flagsp); + } +} + +/* + * pagesets_lock_irqsave_cpu() should only be used from remote CPUs when + * 'remote_pcpu_cache_access' is enabled or the target CPU is dead. Otherwise, + * it can still be called on the local CPU with migration disabled. + */ +static inline void pagesets_lock_irqsave_cpu(struct pagesets *locks, + unsigned long *flagsp, int cpu) +{ + if (static_branch_unlikely(&remote_pcpu_cache_access)) + spin_lock_irqsave(per_cpu_ptr(&locks->spin, cpu), *flagsp); + else + local_lock_irqsave(&locks->local, *flagsp); +} + +static inline void pagesets_unlock_irqrestore(struct pagesets *locks, + unsigned long flags) +{ + if (static_branch_unlikely(&remote_pcpu_cache_access)) { + spin_unlock_irqrestore(this_cpu_ptr(&locks->spin), flags); + migrate_enable(); + } else { + local_unlock_irqrestore(&locks->local, flags); + } +} + +static inline void pagesets_unlock_irqrestore_cpu(struct pagesets *locks, + unsigned long flags, int cpu) +{ + if (static_branch_unlikely(&remote_pcpu_cache_access)) + spin_unlock_irqrestore(per_cpu_ptr(&locks->spin, cpu), flags); + else + local_unlock_irqrestore(&locks->local, flags); +} + /* * A cached value of the page's pageblock's migratetype, used when the page is * put on a pcplist. Used to avoid the pageblock migratetype lookup when @@ -3064,12 +3112,12 @@ void drain_zone_pages(struct zone *zone, struct per_cpu_pages *pcp) unsigned long flags; int to_drain, batch; - local_lock_irqsave(&pagesets.lock, flags); + pagesets_lock_irqsave(&pagesets, &flags); batch = READ_ONCE(pcp->batch); to_drain = min(pcp->count, batch); if (to_drain > 0) free_pcppages_bulk(zone, to_drain, pcp); - local_unlock_irqrestore(&pagesets.lock, flags); + pagesets_unlock_irqrestore(&pagesets, flags); } #endif @@ -3077,21 +3125,22 @@ void drain_zone_pages(struct zone *zone, struct per_cpu_pages *pcp) * Drain pcplists of the indicated processor and zone. * * The processor must either be the current processor and the - * thread pinned to the current processor or a processor that - * is not online. + * thread pinned to the current processor, a processor that + * is not online, or a remote processor while 'remote_pcpu_cache_access' is + * enabled. */ static void drain_pages_zone(unsigned int cpu, struct zone *zone) { unsigned long flags; struct per_cpu_pages *pcp; - local_lock_irqsave(&pagesets.lock, flags); + pagesets_lock_irqsave_cpu(&pagesets, &flags, cpu); pcp = per_cpu_ptr(zone->per_cpu_pageset, cpu); if (pcp->count) free_pcppages_bulk(zone, pcp->count, pcp); - local_unlock_irqrestore(&pagesets.lock, flags); + pagesets_unlock_irqrestore_cpu(&pagesets, flags, cpu); } /* @@ -3402,9 +3451,9 @@ void free_unref_page(struct page *page, unsigned int order) migratetype = MIGRATE_MOVABLE; } - local_lock_irqsave(&pagesets.lock, flags); + pagesets_lock_irqsave(&pagesets, &flags); free_unref_page_commit(page, pfn, migratetype, order); - local_unlock_irqrestore(&pagesets.lock, flags); + pagesets_unlock_irqrestore(&pagesets, flags); } /* @@ -3439,7 +3488,7 @@ void free_unref_page_list(struct list_head *list) set_page_private(page, pfn); } - local_lock_irqsave(&pagesets.lock, flags); + pagesets_lock_irqsave(&pagesets, &flags); list_for_each_entry_safe(page, next, list, lru) { pfn = page_private(page); set_page_private(page, 0); @@ -3460,12 +3509,12 @@ void free_unref_page_list(struct list_head *list) * a large list of pages to free. */ if (++batch_count == SWAP_CLUSTER_MAX) { - local_unlock_irqrestore(&pagesets.lock, flags); + pagesets_unlock_irqrestore(&pagesets, flags); batch_count = 0; - local_lock_irqsave(&pagesets.lock, flags); + pagesets_lock_irqsave(&pagesets, &flags); } } - local_unlock_irqrestore(&pagesets.lock, flags); + pagesets_unlock_irqrestore(&pagesets, flags); } /* @@ -3639,7 +3688,7 @@ static struct page *rmqueue_pcplist(struct zone *preferred_zone, struct page *page; unsigned long flags; - local_lock_irqsave(&pagesets.lock, flags); + pagesets_lock_irqsave(&pagesets, &flags); /* * On allocation, reduce the number of pages that are batch freed. @@ -3650,7 +3699,7 @@ static struct page *rmqueue_pcplist(struct zone *preferred_zone, pcp->free_factor >>= 1; list = &pcp->lists[order_to_pindex(migratetype, order)]; page = __rmqueue_pcplist(zone, order, migratetype, alloc_flags, pcp, list); - local_unlock_irqrestore(&pagesets.lock, flags); + pagesets_unlock_irqrestore(&pagesets, flags); if (page) { __count_zid_vm_events(PGALLOC, page_zonenum(page), 1); zone_statistics(preferred_zone, zone, 1); @@ -5270,7 +5319,7 @@ unsigned long __alloc_pages_bulk(gfp_t gfp, int preferred_nid, goto failed; /* Attempt the batch allocation */ - local_lock_irqsave(&pagesets.lock, flags); + pagesets_lock_irqsave(&pagesets, &flags); pcp = this_cpu_ptr(zone->per_cpu_pageset); pcp_list = &pcp->lists[order_to_pindex(ac.migratetype, 0)]; @@ -5300,7 +5349,7 @@ unsigned long __alloc_pages_bulk(gfp_t gfp, int preferred_nid, nr_populated++; } - local_unlock_irqrestore(&pagesets.lock, flags); + pagesets_unlock_irqrestore(&pagesets, flags); __count_zid_vm_events(PGALLOC, zone_idx(zone), nr_account); zone_statistics(ac.preferred_zoneref->zone, zone, nr_account); @@ -5309,7 +5358,7 @@ unsigned long __alloc_pages_bulk(gfp_t gfp, int preferred_nid, return nr_populated; failed_irq: - local_unlock_irqrestore(&pagesets.lock, flags); + pagesets_unlock_irqrestore(&pagesets, flags); failed: page = __alloc_pages(gfp, 0, preferred_nid, nodemask); From patchwork Tue Sep 21 16:13:23 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicolas Saenz Julienne X-Patchwork-Id: 12508161 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 935EBC433EF for ; Tue, 21 Sep 2021 16:13:48 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 49D11610A1 for ; Tue, 21 Sep 2021 16:13:48 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 49D11610A1 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id 8F2616B007E; Tue, 21 Sep 2021 12:13:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8A0AC940007; Tue, 21 Sep 2021 12:13:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 680E16B0081; Tue, 21 Sep 2021 12:13:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0111.hostedemail.com [216.40.44.111]) by kanga.kvack.org (Postfix) with ESMTP id 549E66B007E for ; Tue, 21 Sep 2021 12:13:44 -0400 (EDT) Received: from smtpin23.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 130068249980 for ; Tue, 21 Sep 2021 16:13:44 +0000 (UTC) X-FDA: 78612076368.23.8E02354 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf04.hostedemail.com (Postfix) with ESMTP id 9DB815000303 for ; Tue, 21 Sep 2021 16:13:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1632240822; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kD1gPXuN5hXdBx7xkUXEssekkfNTX/T0+M6z2qHPbrk=; b=FtYLoBdhRPXdwWtwAhmKFgYY3amkPQvmcOcD3NFQfcWnxxA4rg/cAqj7MnqXRji/mjp9R8 z7uj0yCb8qOv5ifBU0+FhhV8nyn+O440c7S3mGsZuInB1+MyCD4dQxkP5A+U07Mi7XXtGO hKd8EsQVLvxce4FkKpb9vrjUqoRwx6M= Received: from mail-wr1-f69.google.com (mail-wr1-f69.google.com [209.85.221.69]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-158-Uwt67j4mNwS8C_kS1-3E-g-1; Tue, 21 Sep 2021 12:13:41 -0400 X-MC-Unique: Uwt67j4mNwS8C_kS1-3E-g-1 Received: by mail-wr1-f69.google.com with SMTP id r9-20020a5d4989000000b0015d0fbb8823so9239938wrq.18 for ; Tue, 21 Sep 2021 09:13:40 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=kD1gPXuN5hXdBx7xkUXEssekkfNTX/T0+M6z2qHPbrk=; b=SZPxc59OHJQlMeYM1XjMHVDybrlfMQCvtEP6eiXiDxWVSoyH5dnlCi3vXfJakrrzNp 4TvVkrnCCAIg1iLcK81mcRnSJm5MItuTjxgVvLMPJ6Ewto0P6+Czv97bsKS60JqL5Om+ tfJ4TPAUPVg+5SBaja0PqViA3gxXDSW6F9pSsYoYR6kJ9HsIRVp+0iiyKRCG6WmwV2lR 0CCty/X4Z4dRdT65/e8ycQxbL9li3evBnyX/6Vhj5sdxwUf+2DKfuoT+MmaCy0XyMO1U QTdCqplbgWYg/oz8P7sW5cIC22O4dxGlNltksOliCkvNgbQylpoLZgpSMWJNpAOqAZQE zKiA== X-Gm-Message-State: AOAM533Un66lH0wBtsiDL5zEwXB+3JmaUsIO5cQu3SkPtSJ2gZF+ewxy uOkIVCfs7txuT2yKGfc7gfPtU9IcPhk3T6gLaAK6RKmE2w/ShkaATn2dEnZ8o9rT8CiVdxP/lJg jGYA4cbVFDbs= X-Received: by 2002:adf:f890:: with SMTP id u16mr35026727wrp.388.1632240819873; Tue, 21 Sep 2021 09:13:39 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwILlB4+iYeakvY1cY1M2iraZrOJrl8W7MLtYotoTLsG/waIBsVBPTg6O9yF1Fb9VfMjynvqg== X-Received: by 2002:adf:f890:: with SMTP id u16mr35026696wrp.388.1632240819716; Tue, 21 Sep 2021 09:13:39 -0700 (PDT) Received: from vian.redhat.com ([2a0c:5a80:1d03:b900:c3d1:5974:ce92:3123]) by smtp.gmail.com with ESMTPSA id t1sm19786477wrz.39.2021.09.21.09.13.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Sep 2021 09:13:39 -0700 (PDT) From: Nicolas Saenz Julienne To: akpm@linux-foundation.org, frederic@kernel.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, tglx@linutronix.de, cl@linux.com, peterz@infradead.org, juri.lelli@redhat.com, mingo@redhat.com, mtosatti@redhat.com, nilal@redhat.com, mgorman@suse.de, ppandit@redhat.com, williams@redhat.com, bigeasy@linutronix.de, anna-maria@linutronix.de, linux-rt-users@vger.kernel.org, Nicolas Saenz Julienne Subject: [PATCH 5/6] mm/page_alloc: Allow remote per-cpu page list draining Date: Tue, 21 Sep 2021 18:13:23 +0200 Message-Id: <20210921161323.607817-6-nsaenzju@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210921161323.607817-1-nsaenzju@redhat.com> References: <20210921161323.607817-1-nsaenzju@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Stat-Signature: ttq56mwf9n87t9yhjzj4hszyw8p6g976 Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=FtYLoBdh; dmarc=pass (policy=none) header.from=redhat.com; spf=none (imf04.hostedemail.com: domain of nsaenzju@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=nsaenzju@redhat.com X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 9DB815000303 X-HE-Tag: 1632240822-907245 X-Bogosity: Ham, tests=bogofilter, spamicity=0.001047, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Some setups, notably NOHZ_FULL CPUs, are too busy to handle the per-cpu drain work queued by __drain_all_pages(). So introduce new a mechanism to remotely drain the per-cpu lists. It is made possible by using a more constraining locking scheme, which is disabled by default and can be enabled through the 'remote_pcpu_cache_access' static key. Regular users shouldn't see any functional or performance change. Upcoming patches will make use of the static key. This is based on previous work by Thomas Gleixner, Anna-Maria Gleixner, and Sebastian Andrzej Siewior[1]. [1] https://patchwork.kernel.org/project/linux-mm/patch/20190424111208.24459-3-bigeasy@linutronix.de/ Signed-off-by: Nicolas Saenz Julienne --- mm/page_alloc.c | 24 +++++++++++++++++------- 1 file changed, 17 insertions(+), 7 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 3244eb2ab51b..717df675ea06 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -3268,15 +3268,25 @@ static void __drain_all_pages(struct zone *zone, bool force_all_cpus) cpumask_clear_cpu(cpu, &cpus_with_pcps); } - for_each_cpu(cpu, &cpus_with_pcps) { - struct pcpu_drain *drain = per_cpu_ptr(&pcpu_drain, cpu); + if (static_branch_unlikely(&remote_pcpu_cache_access)) { + for_each_cpu(cpu, &cpus_with_pcps) { + if (zone) { + drain_pages_zone(cpu, zone); + } else { + drain_pages(cpu); + } + } + } else { + for_each_cpu(cpu, &cpus_with_pcps) { + struct pcpu_drain *drain = per_cpu_ptr(&pcpu_drain, cpu); - drain->zone = zone; - INIT_WORK(&drain->work, drain_local_pages_wq); - queue_work_on(cpu, mm_percpu_wq, &drain->work); + drain->zone = zone; + INIT_WORK(&drain->work, drain_local_pages_wq); + queue_work_on(cpu, mm_percpu_wq, &drain->work); + } + for_each_cpu(cpu, &cpus_with_pcps) + flush_work(&per_cpu_ptr(&pcpu_drain, cpu)->work); } - for_each_cpu(cpu, &cpus_with_pcps) - flush_work(&per_cpu_ptr(&pcpu_drain, cpu)->work); mutex_unlock(&pcpu_drain_mutex); } From patchwork Tue Sep 21 16:13:24 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicolas Saenz Julienne X-Patchwork-Id: 12508165 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7AB45C433F5 for ; Tue, 21 Sep 2021 16:13:53 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 2ADBE61214 for ; Tue, 21 Sep 2021 16:13:53 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 2ADBE61214 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id DD3866B0081; Tue, 21 Sep 2021 12:13:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D0CA56B0082; Tue, 21 Sep 2021 12:13:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B86F86B0083; Tue, 21 Sep 2021 12:13:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0189.hostedemail.com [216.40.44.189]) by kanga.kvack.org (Postfix) with ESMTP id A98326B0081 for ; Tue, 21 Sep 2021 12:13:47 -0400 (EDT) Received: from smtpin26.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 764BB24398 for ; Tue, 21 Sep 2021 16:13:47 +0000 (UTC) X-FDA: 78612076494.26.90A303B Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by imf10.hostedemail.com (Postfix) with ESMTP id 1E814600198F for ; Tue, 21 Sep 2021 16:13:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1632240826; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=zqnAk8CEPT4G20qz1UpPoVf1omLqowS17E+dGhKhwuM=; b=EmohStatTcPutgZHeYBRv5AvqAoSSuZ8BxOGcs8s9I5dqvZ2UknUvRizd29KyDTeWDAlHG eIQymN1cI+J1q43J0LWPQhNIjtxkuK8DsQgbOPeKz1sSogR7aKO1BH7/K12qgF+tNUis+1 uZLPs7QTj1enDheN3zO4fBqsuhiATpQ= Received: from mail-wr1-f69.google.com (mail-wr1-f69.google.com [209.85.221.69]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-589-p3BoXyvhP7Wa1rGKMKqmnA-1; Tue, 21 Sep 2021 12:13:43 -0400 X-MC-Unique: p3BoXyvhP7Wa1rGKMKqmnA-1 Received: by mail-wr1-f69.google.com with SMTP id r5-20020adfb1c5000000b0015cddb7216fso9266901wra.3 for ; Tue, 21 Sep 2021 09:13:42 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=zqnAk8CEPT4G20qz1UpPoVf1omLqowS17E+dGhKhwuM=; b=aYO8x7FVdDMvJl+e629doXi2+smGBchepZ26UXT1/4aebDnI6OOsGrrE7mwAVrCDFs +oe2XAMZVNjxz3NoZDtfQrqHmScuuofIHdMXq1JYCciWoPZPC2LpTRgF79U4335IQPLO L3wdxfvS6qb4KFX1UZzTcY+myhXXDce27+JZmYxrnuXTuypUcE4oJG/CrQVx+BzkRpqL 1Lh/f9itFKtnPVGpodB6sTO0kgVF71T8TCZ9qNWbDxkB4/G5VL6Gh4pe8cnntDcIEtPK eOcnRufN/scnPCPNjBkh9Lm0iwrqBsPtzDNV5MQZMrbs9rIzWX6y7nUCmBgWdPjoMSmz F0kw== X-Gm-Message-State: AOAM530ICuCwS87dgH9IOnLyesZ09L0VOnTiGk11N7kh/LnHippU08Pq 2pZ2VAy77F6O7y9B8bl2tulIrWzTCQVusYQjozCq1Sn3P4lhv9UXYiAJNO8et4cDgafaaSGVpmG 78jlbrQv6nUU= X-Received: by 2002:a5d:510b:: with SMTP id s11mr35121609wrt.79.1632240821967; Tue, 21 Sep 2021 09:13:41 -0700 (PDT) X-Google-Smtp-Source: ABdhPJz2JyQmA+xjBWA1pYPhaUZl3UgvqGOZtEA5D9vdOoOLPYTSVO682BJB72/4LybppWdkc2NU+A== X-Received: by 2002:a5d:510b:: with SMTP id s11mr35121588wrt.79.1632240821777; Tue, 21 Sep 2021 09:13:41 -0700 (PDT) Received: from vian.redhat.com ([2a0c:5a80:1d03:b900:c3d1:5974:ce92:3123]) by smtp.gmail.com with ESMTPSA id t1sm19786477wrz.39.2021.09.21.09.13.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 21 Sep 2021 09:13:41 -0700 (PDT) From: Nicolas Saenz Julienne To: akpm@linux-foundation.org, frederic@kernel.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, tglx@linutronix.de, cl@linux.com, peterz@infradead.org, juri.lelli@redhat.com, mingo@redhat.com, mtosatti@redhat.com, nilal@redhat.com, mgorman@suse.de, ppandit@redhat.com, williams@redhat.com, bigeasy@linutronix.de, anna-maria@linutronix.de, linux-rt-users@vger.kernel.org, Nicolas Saenz Julienne Subject: [PATCH 6/6] sched/isolation: Enable 'remote_pcpu_cache_access' on NOHZ_FULL systems Date: Tue, 21 Sep 2021 18:13:24 +0200 Message-Id: <20210921161323.607817-7-nsaenzju@redhat.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20210921161323.607817-1-nsaenzju@redhat.com> References: <20210921161323.607817-1-nsaenzju@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=EmohStat; spf=none (imf10.hostedemail.com: domain of nsaenzju@redhat.com has no SPF policy when checking 216.205.24.124) smtp.mailfrom=nsaenzju@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 1E814600198F X-Stat-Signature: 6fawmrdqd1wkxfr4gwq7fqn5c7a7jhaf X-HE-Tag: 1632240826-187222 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When enabled, 'remote_pcpu_cache_access' allows for remote draining of mm/swap.c's per-cpu LRU caches and mm/page_alloc.c's per-cpu page lists instead of using per-cpu drain work. This comes at the cost of more constraining locking, but NOHZ_FULL setups need this nonetheless: processes running on isolated CPUs are sensitive to any sort of interruption and preempting them in order to satisfy a housekeeping task is bound to break their functional guarantees (i.e. latency, bandwidth, etc...). So enable 'remote_pcpu_cache_access' after having successfully initialized NOHZ_FULL. This is based on previous work by Thomas Gleixner, Anna-Maria Gleixner, and Sebastian Andrzej Siewior[1]. [1] https://patchwork.kernel.org/project/linux-mm/patch/20190424111208.24459-3-bigeasy@linutronix.de/ Signed-off-by: Nicolas Saenz Julienne --- kernel/sched/isolation.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/kernel/sched/isolation.c b/kernel/sched/isolation.c index 7f06eaf12818..4fc4c76f27ab 100644 --- a/kernel/sched/isolation.c +++ b/kernel/sched/isolation.c @@ -8,6 +8,7 @@ * */ #include "sched.h" +#include "../../mm/internal.h" DEFINE_STATIC_KEY_FALSE(housekeeping_overridden); EXPORT_SYMBOL_GPL(housekeeping_overridden); @@ -137,11 +138,17 @@ static int __init housekeeping_setup(char *str, enum hk_flags flags) static int __init housekeeping_nohz_full_setup(char *str) { unsigned int flags; + int ret; flags = HK_FLAG_TICK | HK_FLAG_WQ | HK_FLAG_TIMER | HK_FLAG_RCU | HK_FLAG_MISC | HK_FLAG_KTHREAD; - return housekeeping_setup(str, flags); + ret = housekeeping_setup(str, flags); + if (ret) + /* Avoid LRU cache and mm/page_alloc.c's pcplists drain work */ + static_branch_enable(&remote_pcpu_cache_access); + + return ret; } __setup("nohz_full=", housekeeping_nohz_full_setup);