From patchwork Wed Jan 25 07:35:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leonardo Bras X-Patchwork-Id: 13115048 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ED998C54E94 for ; Wed, 25 Jan 2023 07:36:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 54C7D6B007B; Wed, 25 Jan 2023 02:36:09 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 505696B007D; Wed, 25 Jan 2023 02:36:09 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 39DB96B007E; Wed, 25 Jan 2023 02:36:09 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 2B1176B007B for ; Wed, 25 Jan 2023 02:36:09 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id E01531406B6 for ; Wed, 25 Jan 2023 07:36:08 +0000 (UTC) X-FDA: 80392512816.24.D7303B0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf29.hostedemail.com (Postfix) with ESMTP id CC89B120007 for ; Wed, 25 Jan 2023 07:36:06 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=AJBuLT3m; spf=pass (imf29.hostedemail.com: domain of leobras@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=leobras@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674632166; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=cta4F9u/56j6UuYJx1HDD/ltBRbZgzhdAn9u+4PdIyQ=; b=qO8Hjkws5OeKn/2bb6b6oBbYsgIB9OiNqbCQm0lj06cn3BFMIIjFVnYohRC7/E+VcBcV1T 6kr5FjSV1zyYM/qqG4prqdm5OAkZcoIWdHbrN90NywEER/cJVLxnhWA6Uq+QcMvHmqIRkK 7neo/l0RPgIYTVeuBjk1IWPltcDkcd4= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=AJBuLT3m; spf=pass (imf29.hostedemail.com: domain of leobras@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=leobras@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674632166; a=rsa-sha256; cv=none; b=qZa2RDItWI0OoNEyFVXbIeFmrCqmJKhNCtEhhI9QeoaXjFVY/FdM448KrdLfEzQKN779/P blSZ/r2U/Fn6zSTzbAu/1ZniVFx5P0/K7br8KXBFuaW+wLEjrblhKVWj9dGp7rWMpywmxj 9rJJ5I+6K4jV3cIuvkRngZZ+kpLEhPA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1674632166; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cta4F9u/56j6UuYJx1HDD/ltBRbZgzhdAn9u+4PdIyQ=; b=AJBuLT3mysKio1fQzv/uX1FEm6r+H7xR3VNF0RV24Susp5SfYPFB6ZPxqDGOKpXkqyXb4k sS4FllFziE7W2oo4I0LCO1CKlLfnTQ4CsMIfP5okp66pwdWqYzj7W6dqdBGiOUV4NKZEqn X4dsysDPN5nhZOhYEZVTdxsDVB8D2qc= Received: from mail-ot1-f69.google.com (mail-ot1-f69.google.com [209.85.210.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-66-QiR7kdENNdepimXRSrpb9w-1; Wed, 25 Jan 2023 02:36:05 -0500 X-MC-Unique: QiR7kdENNdepimXRSrpb9w-1 Received: by mail-ot1-f69.google.com with SMTP id by2-20020a056830608200b00683e3e829d5so8870550otb.17 for ; Tue, 24 Jan 2023 23:36:04 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cta4F9u/56j6UuYJx1HDD/ltBRbZgzhdAn9u+4PdIyQ=; b=mL/5gKVejYdJE9HsefEGzr+glz/uBmHUsX9hF60HOtiKw6zsfKP8EnSnvq66m6078G e9PTpW6f0RTXl8krA0CVVLyfHSL2rvV044y9QLgTP6Z8KGNyhzP4WtoRfzFgslrys3VZ duzKnF0SJjzqjb3PKkUYpqJqRlp+qzUJM3ODFpSBYTc+HQouu41L/FQA55DAo823kGVz Fzou0JHK3rYzXBzGxa+IjeYbyGES4n6/5QcjsLplQuR8XnSZcFmd9kERYUzOQCBVT5IV 3sB9ze/oTbRYBR9HVxE0bHpTQJrjB1B7j4ZRYRE+0PlkfycnSi6oOn2/qu2CuytrVkEu yPpA== X-Gm-Message-State: AFqh2kqay3UPogEjnLdtOFZiiCob0FXozznLXhaZ43HzVssmtg/SvRjg 1yfndAM+MQYJ5dHSFy8ud16x1BPlqjHXCCAE5mRttoqHXhCb0usVT1TfxrMykP88OWbxO+QNSeQ /pXIZEBCdyCc= X-Received: by 2002:a4a:d307:0:b0:4f5:2a03:b22a with SMTP id g7-20020a4ad307000000b004f52a03b22amr14131792oos.4.1674632164302; Tue, 24 Jan 2023 23:36:04 -0800 (PST) X-Google-Smtp-Source: AMrXdXsEUpYG3GHiKEkFMX0XaGUeQBQJBtt1RE/vEB6cze/oEgxS7/YGFSUZjjX5dJWUBMfQRU1UMg== X-Received: by 2002:a4a:d307:0:b0:4f5:2a03:b22a with SMTP id g7-20020a4ad307000000b004f52a03b22amr14131784oos.4.1674632164048; Tue, 24 Jan 2023 23:36:04 -0800 (PST) Received: from LeoBras.redhat.com ([2804:1b3:a800:14fa:9361:c141:6c70:c877]) by smtp.gmail.com with ESMTPSA id x189-20020a4a41c6000000b0050dc79bb80esm1538802ooa.27.2023.01.24.23.35.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Jan 2023 23:36:03 -0800 (PST) From: Leonardo Bras To: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , Marcelo Tosatti Cc: Leonardo Bras , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 4/5] mm/memcontrol: Perform all stock drain in current CPU Date: Wed, 25 Jan 2023 04:35:01 -0300 Message-Id: <20230125073502.743446-5-leobras@redhat.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230125073502.743446-1-leobras@redhat.com> References: <20230125073502.743446-1-leobras@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: hm5t3rgr7xcdxq5zw1mscieie4ua3mbm X-Rspamd-Queue-Id: CC89B120007 X-HE-Tag: 1674632166-864681 X-HE-Meta: U2FsdGVkX1+upbQjoR+G/8jNNI84xnc6Yq5gzpgr9qqAu1LTInpMbbH9O4SQJG3NVv5RnzqfHLZx+O1r33f7Y9OAwR9/osghW9IgpxX1TLYs/MaDrEeyP7nZFzcJvZHeni392c02Vtp5BFCUIbsl1cDjLnOufc1p5PmKWeT52HJZI2ogEd2rCBaHBrbtC/nw8ZvFvZB92lpYiIsLM8SbcdwK2tTvzmYEIGHtyUnlh7uqeEbGWD48m9r5bS5qe9QoPoG/Q1pAkWxph7cUAPc3mr15GuDJEXTsPGG8QpuT1JnygYdyqiCGy4Tf2UApBmXnzAfLvtnJDi0o2AxptI00Bcg7lC3eDP+hav8kgnnXoDfi0rh6CYLpHnvZGK95fT4nx27R6VHyfWjNeN+rh7Eqs/mYP5h2Mu4J5+/4z5xkwY7jVAbD57wBp8k5GXGqA9TopeT4Wp3JptdmyLuC6zK2VNye1IIWjSTaAHgJajRDvJ8q1ZFtVbrEGmfe7P40bFvsZcezKIsfa9LURnRbYwxHHJKzchTOgOJduK/K+3p0T80aLhRzKgqKSoKNWo6BkILWlCbFbl2l4j9woTx1uHjiiJsR1v8AyWYVBxKhHsl+MnVkVWLYA59fSJmjuXF4rxULWnxo6YxfNupYI9OIMzfqDiH9AN4iGHdF3z6Qx6h5YsGOT3dptu7YR6A9hd9cm3YRE2qkt09HhOV30jUm4J5B6ZE26kq4S3q8PJzngtbstdXA0e8/QZJzwzXzduWWTHJQ/NnWRpXnG2JZNCmD2SHtWe3BaQqsZO/Q/VoAy2/3Tw5IdRMmwaDBrfJIz9S4JhP8NsKv45gwweKhEkLMOCShepk3UJaunsybmJAb92nWsUndBG4hTDapjike2b8ncGM8zGwL13HDjozwFEVxQrQIu/bpwks93D1ClJTak9rE+ircxzphaA8k/Jnld5ukntLcOZrp5ib1hkcfrYhAUTO xHmSPM26 wcA7DKThGpkMyMvG7C8Po1CKDJzv1r/XujgQlpg6fr4VFBjWTerstC1EDRuCGrZLd/6f5OpUW02kuKoT7oSYP/QtIPn+ppJil2TkANpjKtk4RH6wbn6cYl66Sylo+Py/h7NfzgbIQFD/ynhU+m+gbFNko23rQrs23ZUw0Kd/I1TIMC2dtcbwNB/oaVPqAApo7TScB8jYms0E8H0hjTxv0JktzmsMjjo2OighkTqjd88Myj5k2prz3EBZXZ0Miuy0a6NEQxIZRR5v68CARvWQETvf6BKwbj5dIkA3H9ebyZ+GaTti0XQSbTWkW/MDmGpYyNt/gTXVfIYnPyLIYDYulh8B/fiInz2CYNRa7xR9pd84nerg7bu5qxr3YDFHdBO1WqJK/z7h/OWLKjc0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000012, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When drain_all_stock() is called, some CPUs will be required to have their per-CPU caches drained. This currently happens by scheduling a call to drain_local_stock() to run in each affected CPU. This, as a consequence, may end up scheduling work to CPUs that are isolated, and therefore should have as little interruption as possible. In order to avoid this, run all CPUs stock drain directly from the current CPU. This should be fine as long as drain_all_stock() runs fast enough so it don't often cause contention on consume_stock(), refill_stock(), mod_objcg_state(), consume_obj_stock() or refill_obj_stock(). Also, since drain_all_stock() will be able to run on a remote CPU, protect memcg_hotplug_cpu_dead() with stock_lock. Signed-off-by: Leonardo Bras --- mm/memcontrol.c | 29 +++++++++-------------------- 1 file changed, 9 insertions(+), 20 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 373fa78c4d881..5b7f7c2e0232f 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2184,7 +2184,6 @@ struct memcg_stock_pcp { int nr_slab_unreclaimable_b; #endif - struct work_struct work; unsigned long flags; #define FLUSHING_CACHED_CHARGE 0 }; @@ -2269,18 +2268,15 @@ static void drain_stock(struct memcg_stock_pcp *stock) stock->cached = NULL; } -static void drain_local_stock(struct work_struct *dummy) +static void drain_stock_from(struct memcg_stock_pcp *stock) { - struct memcg_stock_pcp *stock; struct obj_cgroup *old = NULL; unsigned long flags; /* - * The only protection from cpu hotplug (memcg_hotplug_cpu_dead) vs. - * drain_stock races is that we always operate on local CPU stock - * here with IRQ disabled + * The protection from cpu hotplug (memcg_hotplug_cpu_dead) vs. + * drain_stock races is stock_lock, a percpu spinlock. */ - stock = this_cpu_ptr(&memcg_stock); spin_lock_irqsave(&stock->stock_lock, flags); old = drain_obj_stock(stock); @@ -2329,7 +2325,7 @@ static void refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages) */ static void drain_all_stock(struct mem_cgroup *root_memcg) { - int cpu, curcpu; + int cpu; /* If someone's already draining, avoid adding running more workers. */ if (!mutex_trylock(&percpu_charge_mutex)) @@ -2341,7 +2337,6 @@ static void drain_all_stock(struct mem_cgroup *root_memcg) * per-cpu data. CPU up doesn't touch memcg_stock at all. */ migrate_disable(); - curcpu = smp_processor_id(); for_each_online_cpu(cpu) { struct memcg_stock_pcp *stock = &per_cpu(memcg_stock, cpu); struct mem_cgroup *memcg; @@ -2357,12 +2352,8 @@ static void drain_all_stock(struct mem_cgroup *root_memcg) rcu_read_unlock(); if (flush && - !test_and_set_bit(FLUSHING_CACHED_CHARGE, &stock->flags)) { - if (cpu == curcpu) - drain_local_stock(&stock->work); - else - schedule_work_on(cpu, &stock->work); - } + !test_and_set_bit(FLUSHING_CACHED_CHARGE, &stock->flags)) + drain_stock_from(stock); } migrate_enable(); mutex_unlock(&percpu_charge_mutex); @@ -2373,7 +2364,9 @@ static int memcg_hotplug_cpu_dead(unsigned int cpu) struct memcg_stock_pcp *stock; stock = &per_cpu(memcg_stock, cpu); + spin_lock(&stock->stock_lock); drain_stock(stock); + spin_unlock(&stock->stock_lock); return 0; } @@ -7328,7 +7321,7 @@ __setup("cgroup.memory=", cgroup_memory); */ static int __init mem_cgroup_init(void) { - int cpu, node; + int node; /* * Currently s32 type (can refer to struct batched_lruvec_stat) is @@ -7341,10 +7334,6 @@ static int __init mem_cgroup_init(void) cpuhp_setup_state_nocalls(CPUHP_MM_MEMCQ_DEAD, "mm/memctrl:dead", NULL, memcg_hotplug_cpu_dead); - for_each_possible_cpu(cpu) - INIT_WORK(&per_cpu_ptr(&memcg_stock, cpu)->work, - drain_local_stock); - for_each_node(node) { struct mem_cgroup_tree_per_node *rtpn;