From patchwork Wed Nov 2 02:02:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leonardo Bras X-Patchwork-Id: 13027672 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D0734C4332F for ; Wed, 2 Nov 2022 02:03:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6CF4A8E0003; Tue, 1 Nov 2022 22:03:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 67FA78E0002; Tue, 1 Nov 2022 22:03:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 548608E0003; Tue, 1 Nov 2022 22:03:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 417E48E0002 for ; Tue, 1 Nov 2022 22:03:59 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 10DE916039B for ; Wed, 2 Nov 2022 02:03:59 +0000 (UTC) X-FDA: 80086856598.17.9CC8428 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf01.hostedemail.com (Postfix) with ESMTP id B5AF840002 for ; Wed, 2 Nov 2022 02:03:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1667354636; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=i2RYvVpIxnZB4mRhcfM8kpuAulQpsyxXS11J9gDFkxg=; b=KMaVYxxzPFKvnkJvt0um+/4+8V6xJ+rpCSq1eVr0Jhxat67GfqBgSPSB+5lMiLkHC7l+4x NT/2o0+1jafMGD1ZwfTWUZxPn9fz67ml4yYYvHqqdpo4q+ZEDrLrYhMsNmJJLIEsZZWR0r zJHD9jRb8gjxtrv0/jU6mrSkBknqh3w= Received: from mail-ot1-f69.google.com (mail-ot1-f69.google.com [209.85.210.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-76-MuCe6HlrNQ-F7Jd3soiYbw-1; Tue, 01 Nov 2022 22:03:54 -0400 X-MC-Unique: MuCe6HlrNQ-F7Jd3soiYbw-1 Received: by mail-ot1-f69.google.com with SMTP id l31-20020a9d1b22000000b0066c48e9249fso4641262otl.5 for ; Tue, 01 Nov 2022 19:03:54 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=i2RYvVpIxnZB4mRhcfM8kpuAulQpsyxXS11J9gDFkxg=; b=Q9ezC1bDZf2k9mQ9V+XE7QXLeJh/Soe2SLPtsamjc4BoOhX0OUl1aMfS01MRXzai+d DPTcWgYGL9t6wfEh4MpE77lzmkxe7HZbjwWv5DhMymjpoYiFKVgdr7xCNtAYk/lmEZoo NTBa+LkMqht1yvhYMiT9dLctLfF2wAESPNCvJjobhSi1XdyqXJ58wph4lTyjmQ2XUOLO SxB+7fpDVXgQyxKhHjr1ewaeSp0bjHwl2UUsAikrLAnW7poNMsIMfridIeH9Vm/IBEzT OrBeLW51tiuJAM7dA32/jw3/A4KmfXpYb8R2PwXrZd8NvbjfdOokEODFXt3c8N/ROKUc XaIg== X-Gm-Message-State: ACrzQf0gbPT//K39OYw+YxYR3j1L66ZH7M0PWJjd0QyWpXwREBeo2uRb 0kBbKEBmbwjmyW672kn29agkcQqpk3KHj8A8//HV44EimzE9+mnXr/IKwTMknW9pC2s7XvcngjV BoIXTam3FxMQ= X-Received: by 2002:a05:6870:4212:b0:13c:d544:8e2a with SMTP id u18-20020a056870421200b0013cd5448e2amr10233057oac.28.1667354634218; Tue, 01 Nov 2022 19:03:54 -0700 (PDT) X-Google-Smtp-Source: AMsMyM5TucfJr70Pv/6ho03ND1IeMhGN5op58YZ9rbaUJ10xOKWU2YOlELtwuMUjwXZ9y/ol3qVj0Q== X-Received: by 2002:a05:6870:4212:b0:13c:d544:8e2a with SMTP id u18-20020a056870421200b0013cd5448e2amr10233044oac.28.1667354633998; Tue, 01 Nov 2022 19:03:53 -0700 (PDT) Received: from LeoBras.redhat.com ([2804:1b3:a802:1099:7cb2:3a49:6197:5307]) by smtp.gmail.com with ESMTPSA id h15-20020a9d6f8f000000b00665919f7823sm4526624otq.8.2022.11.01.19.03.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 01 Nov 2022 19:03:53 -0700 (PDT) From: Leonardo Bras To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira , Valentin Schneider , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , Frederic Weisbecker , Leonardo Bras , Phil Auld , Marcelo Tosatti Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v1 3/3] mm/memcontrol: Add drain_remote_stock(), avoid drain_stock on isolated cpus Date: Tue, 1 Nov 2022 23:02:43 -0300 Message-Id: <20221102020243.522358-4-leobras@redhat.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20221102020243.522358-1-leobras@redhat.com> References: <20221102020243.522358-1-leobras@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=KMaVYxxz; spf=pass (imf01.hostedemail.com: domain of leobras@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=leobras@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1667354636; a=rsa-sha256; cv=none; b=sisO1cl9Sdx28izlV2Fqi36WKyFWg1sWrl0K8GmVFeDIRJw4wFCJcUiHe4R7YmyFAGcgOz XaVk7r96kym1JZ8VoHNDV1YdrxijltcjljJlSt4ilpvQI+SdqS0EG4Gdn4XFVOd4Vl8Plt HNgbzfJhDtnZIqkkGY1BS3ExPYa9cWE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1667354636; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=i2RYvVpIxnZB4mRhcfM8kpuAulQpsyxXS11J9gDFkxg=; b=DKU2lG8gxZexp4fyH5nrjC2jLoAZDhBnQMdOZT9+yaJCqeVfnw3+Zwy/iOmPuZetSqR6zF LmwgWJF9sovrKlaT2OcTedaeg5taEqjaW+s3/eHscB4RXADVMQLkzag4vD6NGfyJjEpl5W aHJhURb2/D3Y+MnSF2xlbOd+dPl3Sok= X-Stat-Signature: 499ojimk7u5qo5igeykeqs4m5jmstmyc X-Rspamd-Queue-Id: B5AF840002 Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=KMaVYxxz; spf=pass (imf01.hostedemail.com: domain of leobras@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=leobras@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspam-User: X-Rspamd-Server: rspam12 X-HE-Tag: 1667354636-539863 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When drain_all_stock() is called, some CPUs will be required to have their per-CPU caches drained. This currently happens by scheduling a call to drain_local_stock() to run in each affected CPU. This, as a consequence, may end up scheduling work to CPUs that are isolated, and therefore should have as little interruption as possible. In order to avoid this, make drain_all_stock() able to detect isolated CPUs and schedule draining the perCPU stock to happen in another non-isolated CPU. But since the current implementation only allows the drain to happen in local CPU, implement a function to drain stock on a remote CPU: drain_remote_stock(). Given both drain_local_stock() and drain_remote_stock() do almost the same work, implement a inline drain_stock_helper() that is called by both. Also, since drain_stock() will be able to run on a remote CPU, protect memcg_hotplug_cpu_dead() with stock_lock. Signed-off-by: Leonardo Bras --- mm/memcontrol.c | 47 ++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 38 insertions(+), 9 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index add46da2e6df1..7ad6e4f4b79ef 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -30,6 +30,7 @@ #include #include #include +#include #include #include #include @@ -2263,7 +2264,7 @@ static void drain_stock(struct memcg_stock_pcp *stock) stock->cached = NULL; } -static void drain_local_stock(struct work_struct *dummy) +static inline void drain_stock_helper(int cpu) { struct memcg_stock_pcp *stock; struct obj_cgroup *old = NULL; @@ -2271,10 +2272,9 @@ static void drain_local_stock(struct work_struct *dummy) /* * The only protection from cpu hotplug (memcg_hotplug_cpu_dead) vs. - * drain_stock races is that we always operate on local CPU stock - * here with IRQ disabled + * drain_stock races is stock_lock, a percpu spinlock. */ - stock = this_cpu_ptr(&memcg_stock); + stock = per_cpu_ptr(&memcg_stock, cpu); spin_lock_irqsave(&stock->stock_lock, flags); old = drain_obj_stock(stock); @@ -2286,6 +2286,16 @@ static void drain_local_stock(struct work_struct *dummy) obj_cgroup_put(old); } +static void drain_remote_stock(struct work_struct *work) +{ + drain_stock_helper(atomic_long_read(&work->data)); +} + +static void drain_local_stock(struct work_struct *dummy) +{ + drain_stock_helper(smp_processor_id()); +} + /* * Cache charges(val) to local per_cpu area. * This will be consumed by consume_stock() function, later. @@ -2352,10 +2362,16 @@ static void drain_all_stock(struct mem_cgroup *root_memcg) if (flush && !test_and_set_bit(FLUSHING_CACHED_CHARGE, &stock->flags)) { - if (cpu == curcpu) + if (cpu == curcpu) { drain_local_stock(&stock->work); - else + } else if (housekeeping_cpu(cpu, HK_TYPE_WQ)) { schedule_work_on(cpu, &stock->work); + } else { + int hkcpu = housekeeping_any_cpu_from(HK_TYPE_WQ, cpu); + + atomic_long_set(&stock->work.data, cpu); + schedule_work_on(hkcpu, &stock->work); + } } } migrate_enable(); @@ -2367,7 +2383,9 @@ static int memcg_hotplug_cpu_dead(unsigned int cpu) struct memcg_stock_pcp *stock; stock = &per_cpu(memcg_stock, cpu); + spin_lock(&stock->stock_lock); drain_stock(stock); + spin_unlock(&stock->stock_lock); return 0; } @@ -7272,9 +7290,20 @@ static int __init mem_cgroup_init(void) cpuhp_setup_state_nocalls(CPUHP_MM_MEMCQ_DEAD, "mm/memctrl:dead", NULL, memcg_hotplug_cpu_dead); - for_each_possible_cpu(cpu) - INIT_WORK(&per_cpu_ptr(&memcg_stock, cpu)->work, - drain_local_stock); + /* + * CPUs that are isolated should not spend cpu time for stock draining, + * so allow them to export this task to the nearest housekeeping enabled + * cpu available. + */ + for_each_possible_cpu(cpu) { + if (housekeeping_cpu(cpu, HK_TYPE_WQ)) { + INIT_WORK(&per_cpu_ptr(&memcg_stock, cpu)->work, + drain_local_stock); + } else { + INIT_WORK(&per_cpu_ptr(&memcg_stock, cpu)->work, + drain_remote_stock); + } + } for_each_node(node) { struct mem_cgroup_tree_per_node *rtpn;