From patchwork Fri Apr 9 23:18:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Waiman Long X-Patchwork-Id: 12195213 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2891CC43460 for ; Fri, 9 Apr 2021 23:19:37 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B628D610A4 for ; Fri, 9 Apr 2021 23:19:36 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B628D610A4 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 5394E6B006E; Fri, 9 Apr 2021 19:19:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 49AC66B0070; Fri, 9 Apr 2021 19:19:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 251096B0071; Fri, 9 Apr 2021 19:19:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0076.hostedemail.com [216.40.44.76]) by kanga.kvack.org (Postfix) with ESMTP id F19056B006E for ; Fri, 9 Apr 2021 19:19:35 -0400 (EDT) Received: from smtpin31.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id B6E18181C223A for ; Fri, 9 Apr 2021 23:19:35 +0000 (UTC) X-FDA: 78014397510.31.D48290C Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by imf10.hostedemail.com (Postfix) with ESMTP id 8A46F40002CF for ; Fri, 9 Apr 2021 23:19:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1618010374; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:in-reply-to:in-reply-to:references:references; bh=uZv/d8ZgMLy4t6LltwZ6A+YmlSKW3HmeIredudZwsTQ=; b=Sq5qd5TgS1VIs2diKhF2922629XjH1vCA8CaFlg4P3NZETqwGiZBZeyoYTUStLbU9Y0CQj bqRriu4BSYm4QIaXMorD+55Ku4hoKpqmxE5uFQDH9BtM1wfzpUJvV/2hTTV5bsLG7wr8CF mOLY27zJk8F3u0gwWqlqI7KgvGIYG6A= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-231-x0CQ-5e6NMmY3Sg76oVD1g-1; Fri, 09 Apr 2021 19:19:31 -0400 X-MC-Unique: x0CQ-5e6NMmY3Sg76oVD1g-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 16F898030A0; Fri, 9 Apr 2021 23:19:28 +0000 (UTC) Received: from llong.com (ovpn-113-226.rdu2.redhat.com [10.10.113.226]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9445B6F965; Fri, 9 Apr 2021 23:19:25 +0000 (UTC) From: Waiman Long To: Johannes Weiner , Michal Hocko , Vladimir Davydov , Andrew Morton , Tejun Heo , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Vlastimil Babka , Roman Gushchin Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, Shakeel Butt , Muchun Song , Alex Shi , Chris Down , Yafang Shao , Alexander Duyck , Wei Yang , Masayoshi Mizuma , Waiman Long Subject: [PATCH 1/5] mm/memcg: Pass both memcg and lruvec to mod_memcg_lruvec_state() Date: Fri, 9 Apr 2021 19:18:38 -0400 Message-Id: <20210409231842.8840-2-longman@redhat.com> In-Reply-To: <20210409231842.8840-1-longman@redhat.com> References: <20210409231842.8840-1-longman@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 8A46F40002CF X-Stat-Signature: 9qno34dq9b4ypc7b9fumwgq4wesandrc Received-SPF: none (redhat.com>: No applicable sender policy available) receiver=imf10; identity=mailfrom; envelope-from=""; helo=us-smtp-delivery-124.mimecast.com; client-ip=216.205.24.124 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1618010371-102047 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The caller of mod_memcg_lruvec_state() has both memcg and lruvec readily available. So both of them are now passed to mod_memcg_lruvec_state() and __mod_memcg_lruvec_state(). The __mod_memcg_lruvec_state() is updated to allow either of the two parameters to be set to null. This makes mod_memcg_lruvec_state() equivalent to mod_memcg_state() if lruvec is null. Signed-off-by: Waiman Long Acked-by: Roman Gushchin Reviewed-by: Shakeel Butt --- include/linux/memcontrol.h | 12 +++++++----- mm/memcontrol.c | 19 +++++++++++++------ mm/slab.h | 2 +- 3 files changed, 21 insertions(+), 12 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 0c04d39a7967..95f12996e66c 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -955,8 +955,8 @@ static inline unsigned long lruvec_page_state_local(struct lruvec *lruvec, return x; } -void __mod_memcg_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, - int val); +void __mod_memcg_lruvec_state(struct mem_cgroup *memcg, struct lruvec *lruvec, + enum node_stat_item idx, int val); void __mod_lruvec_kmem_state(void *p, enum node_stat_item idx, int val); static inline void mod_lruvec_kmem_state(void *p, enum node_stat_item idx, @@ -969,13 +969,14 @@ static inline void mod_lruvec_kmem_state(void *p, enum node_stat_item idx, local_irq_restore(flags); } -static inline void mod_memcg_lruvec_state(struct lruvec *lruvec, +static inline void mod_memcg_lruvec_state(struct mem_cgroup *memcg, + struct lruvec *lruvec, enum node_stat_item idx, int val) { unsigned long flags; local_irq_save(flags); - __mod_memcg_lruvec_state(lruvec, idx, val); + __mod_memcg_lruvec_state(memcg, lruvec, idx, val); local_irq_restore(flags); } @@ -1369,7 +1370,8 @@ static inline unsigned long lruvec_page_state_local(struct lruvec *lruvec, return node_page_state(lruvec_pgdat(lruvec), idx); } -static inline void __mod_memcg_lruvec_state(struct lruvec *lruvec, +static inline void __mod_memcg_lruvec_state(struct mem_cgroup *memcg, + struct lruvec *lruvec, enum node_stat_item idx, int val) { } diff --git a/mm/memcontrol.c b/mm/memcontrol.c index e064ac0d850a..d66e1e38f8ac 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -799,20 +799,27 @@ parent_nodeinfo(struct mem_cgroup_per_node *pn, int nid) return mem_cgroup_nodeinfo(parent, nid); } -void __mod_memcg_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, - int val) +/* + * Either one of memcg or lruvec can be NULL, but not both. + */ +void __mod_memcg_lruvec_state(struct mem_cgroup *memcg, struct lruvec *lruvec, + enum node_stat_item idx, int val) { struct mem_cgroup_per_node *pn; - struct mem_cgroup *memcg; long x, threshold = MEMCG_CHARGE_BATCH; + /* Update lruvec */ pn = container_of(lruvec, struct mem_cgroup_per_node, lruvec); - memcg = pn->memcg; + + if (!memcg) + memcg = pn->memcg; /* Update memcg */ __mod_memcg_state(memcg, idx, val); - /* Update lruvec */ + if (!lruvec) + return; + __this_cpu_add(pn->lruvec_stat_local->count[idx], val); if (vmstat_item_in_bytes(idx)) @@ -848,7 +855,7 @@ void __mod_lruvec_state(struct lruvec *lruvec, enum node_stat_item idx, /* Update memcg and lruvec */ if (!mem_cgroup_disabled()) - __mod_memcg_lruvec_state(lruvec, idx, val); + __mod_memcg_lruvec_state(NULL, lruvec, idx, val); } void __mod_lruvec_page_state(struct page *page, enum node_stat_item idx, diff --git a/mm/slab.h b/mm/slab.h index 076582f58f68..bc6c7545e487 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -293,7 +293,7 @@ static inline void mod_objcg_state(struct obj_cgroup *objcg, rcu_read_lock(); memcg = obj_cgroup_memcg(objcg); lruvec = mem_cgroup_lruvec(memcg, pgdat); - mod_memcg_lruvec_state(lruvec, idx, nr); + mod_memcg_lruvec_state(memcg, lruvec, idx, nr); rcu_read_unlock(); } From patchwork Fri Apr 9 23:18:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Waiman Long X-Patchwork-Id: 12195215 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 84E6CC433ED for ; Fri, 9 Apr 2021 23:19:39 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 317F1610A4 for ; Fri, 9 Apr 2021 23:19:39 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 317F1610A4 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C67236B0070; Fri, 9 Apr 2021 19:19:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BC7A16B0071; Fri, 9 Apr 2021 19:19:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A680F6B0072; Fri, 9 Apr 2021 19:19:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0224.hostedemail.com [216.40.44.224]) by kanga.kvack.org (Postfix) with ESMTP id 8705F6B0070 for ; Fri, 9 Apr 2021 19:19:38 -0400 (EDT) Received: from smtpin38.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id CA68CBEFE for ; Fri, 9 Apr 2021 23:19:37 +0000 (UTC) X-FDA: 78014397594.38.0EACC87 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by imf10.hostedemail.com (Postfix) with ESMTP id A070340002D7 for ; Fri, 9 Apr 2021 23:19:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1618010377; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:in-reply-to:in-reply-to:references:references; bh=ptUYwKpXpvlSb7C/Hll6413ogimsLnOMUVrWZt43Ltc=; b=W/HDZWQKhlmH+NpEwU3IXtZabPxWuZh/K8ysbil6L0njazV42eAuXB2WLPzn/c5NBVJcuw YDw5ObAVqUpbVjitGQe9QxGxpxq598UX4M0eg7C3dLEqyRc4e+qtHDS/2s9mWA1LNrc5W9 pxkZCkAPbF5C6yoV1yQ/Lb9BFY75ncI= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-439-rwia5fCfMoCyAeRrf6IlTA-1; Fri, 09 Apr 2021 19:19:33 -0400 X-MC-Unique: rwia5fCfMoCyAeRrf6IlTA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 2B8D81883521; Fri, 9 Apr 2021 23:19:30 +0000 (UTC) Received: from llong.com (ovpn-113-226.rdu2.redhat.com [10.10.113.226]) by smtp.corp.redhat.com (Postfix) with ESMTP id 3726E6EF50; Fri, 9 Apr 2021 23:19:28 +0000 (UTC) From: Waiman Long To: Johannes Weiner , Michal Hocko , Vladimir Davydov , Andrew Morton , Tejun Heo , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Vlastimil Babka , Roman Gushchin Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, Shakeel Butt , Muchun Song , Alex Shi , Chris Down , Yafang Shao , Alexander Duyck , Wei Yang , Masayoshi Mizuma , Waiman Long Subject: [PATCH 2/5] mm/memcg: Introduce obj_cgroup_uncharge_mod_state() Date: Fri, 9 Apr 2021 19:18:39 -0400 Message-Id: <20210409231842.8840-3-longman@redhat.com> In-Reply-To: <20210409231842.8840-1-longman@redhat.com> References: <20210409231842.8840-1-longman@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: A070340002D7 X-Stat-Signature: rfnmp79w4fdsb9bu7934hogpw8yepnw1 Received-SPF: none (redhat.com>: No applicable sender policy available) receiver=imf10; identity=mailfrom; envelope-from=""; helo=us-smtp-delivery-124.mimecast.com; client-ip=216.205.24.124 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1618010373-432454 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In memcg_slab_free_hook()/pcpu_memcg_free_hook(), obj_cgroup_uncharge() is followed by mod_objcg_state()/mod_memcg_state(). Each of these function call goes through a separate irq_save/irq_restore cycle. That is inefficient. Introduce a new function obj_cgroup_uncharge_mod_state() that combines them with a single irq_save/irq_restore cycle. Signed-off-by: Waiman Long Reviewed-by: Shakeel Butt Acked-by: Roman Gushchin --- include/linux/memcontrol.h | 2 ++ mm/memcontrol.c | 31 +++++++++++++++++++++++++++---- mm/percpu.c | 9 ++------- mm/slab.h | 6 +++--- 4 files changed, 34 insertions(+), 14 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 95f12996e66c..6890f999c1a3 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1592,6 +1592,8 @@ struct obj_cgroup *get_obj_cgroup_from_current(void); int obj_cgroup_charge(struct obj_cgroup *objcg, gfp_t gfp, size_t size); void obj_cgroup_uncharge(struct obj_cgroup *objcg, size_t size); +void obj_cgroup_uncharge_mod_state(struct obj_cgroup *objcg, size_t size, + struct pglist_data *pgdat, int idx); extern struct static_key_false memcg_kmem_enabled_key; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index d66e1e38f8ac..b19100c68aa0 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -3225,12 +3225,9 @@ static bool obj_stock_flush_required(struct memcg_stock_pcp *stock, return false; } -static void refill_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes) +static void __refill_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes) { struct memcg_stock_pcp *stock; - unsigned long flags; - - local_irq_save(flags); stock = this_cpu_ptr(&memcg_stock); if (stock->cached_objcg != objcg) { /* reset if necessary */ @@ -3243,7 +3240,14 @@ static void refill_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes) if (stock->nr_bytes > PAGE_SIZE) drain_obj_stock(stock); +} + +static void refill_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes) +{ + unsigned long flags; + local_irq_save(flags); + __refill_obj_stock(objcg, nr_bytes); local_irq_restore(flags); } @@ -3292,6 +3296,25 @@ void obj_cgroup_uncharge(struct obj_cgroup *objcg, size_t size) refill_obj_stock(objcg, size); } +void obj_cgroup_uncharge_mod_state(struct obj_cgroup *objcg, size_t size, + struct pglist_data *pgdat, int idx) +{ + unsigned long flags; + struct mem_cgroup *memcg; + struct lruvec *lruvec = NULL; + + local_irq_save(flags); + __refill_obj_stock(objcg, size); + + rcu_read_lock(); + memcg = obj_cgroup_memcg(objcg); + if (pgdat) + lruvec = mem_cgroup_lruvec(memcg, pgdat); + __mod_memcg_lruvec_state(memcg, lruvec, idx, -(int)size); + rcu_read_unlock(); + local_irq_restore(flags); +} + #endif /* CONFIG_MEMCG_KMEM */ /* diff --git a/mm/percpu.c b/mm/percpu.c index 6596a0a4286e..24ecd51c688c 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -1631,13 +1631,8 @@ static void pcpu_memcg_free_hook(struct pcpu_chunk *chunk, int off, size_t size) objcg = chunk->obj_cgroups[off >> PCPU_MIN_ALLOC_SHIFT]; chunk->obj_cgroups[off >> PCPU_MIN_ALLOC_SHIFT] = NULL; - obj_cgroup_uncharge(objcg, size * num_possible_cpus()); - - rcu_read_lock(); - mod_memcg_state(obj_cgroup_memcg(objcg), MEMCG_PERCPU_B, - -(size * num_possible_cpus())); - rcu_read_unlock(); - + obj_cgroup_uncharge_mod_state(objcg, size * num_possible_cpus(), + NULL, MEMCG_PERCPU_B); obj_cgroup_put(objcg); } diff --git a/mm/slab.h b/mm/slab.h index bc6c7545e487..677cdc52e641 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -366,9 +366,9 @@ static inline void memcg_slab_free_hook(struct kmem_cache *s_orig, continue; objcgs[off] = NULL; - obj_cgroup_uncharge(objcg, obj_full_size(s)); - mod_objcg_state(objcg, page_pgdat(page), cache_vmstat_idx(s), - -obj_full_size(s)); + obj_cgroup_uncharge_mod_state(objcg, obj_full_size(s), + page_pgdat(page), + cache_vmstat_idx(s)); obj_cgroup_put(objcg); } } From patchwork Fri Apr 9 23:18:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Waiman Long X-Patchwork-Id: 12195217 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 31C93C433B4 for ; Fri, 9 Apr 2021 23:19:42 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D6230611F1 for ; Fri, 9 Apr 2021 23:19:41 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D6230611F1 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 6E8776B0071; Fri, 9 Apr 2021 19:19:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 647BD6B0072; Fri, 9 Apr 2021 19:19:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 49B086B0073; Fri, 9 Apr 2021 19:19:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0187.hostedemail.com [216.40.44.187]) by kanga.kvack.org (Postfix) with ESMTP id 2DEE56B0071 for ; Fri, 9 Apr 2021 19:19:41 -0400 (EDT) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 8769E182293E6 for ; Fri, 9 Apr 2021 23:19:40 +0000 (UTC) X-FDA: 78014397720.25.6F181A3 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf25.hostedemail.com (Postfix) with ESMTP id 86AED600010E for ; Fri, 9 Apr 2021 23:19:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1618010378; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:in-reply-to:in-reply-to:references:references; bh=NSPhrn8yNtaplSMa5znJDxoNcNPpKM5qvIFqs/P3czs=; b=KvfkCHsLIyd0csQYta0AdjRic3Egd/dOsp1fgQ3rEr1uhLiSumFbzpwlXJt/MsDr40WTyc 7Am8/ZuAsZLK7yS1ynp3wNOybr/MyxthgpnEJ5icexQ4GRGncy7qXY694HQBdAIwebnrto jd70Vi/VJ05V5/IhZwPzK4gykA1snqk= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-370-vOwjM07sPWCRRfEdoxI2Vw-1; Fri, 09 Apr 2021 19:19:34 -0400 X-MC-Unique: vOwjM07sPWCRRfEdoxI2Vw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 3DB6710054F6; Fri, 9 Apr 2021 23:19:32 +0000 (UTC) Received: from llong.com (ovpn-113-226.rdu2.redhat.com [10.10.113.226]) by smtp.corp.redhat.com (Postfix) with ESMTP id 4D9911B400; Fri, 9 Apr 2021 23:19:30 +0000 (UTC) From: Waiman Long To: Johannes Weiner , Michal Hocko , Vladimir Davydov , Andrew Morton , Tejun Heo , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Vlastimil Babka , Roman Gushchin Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, Shakeel Butt , Muchun Song , Alex Shi , Chris Down , Yafang Shao , Alexander Duyck , Wei Yang , Masayoshi Mizuma , Waiman Long Subject: [PATCH 3/5] mm/memcg: Cache vmstat data in percpu memcg_stock_pcp Date: Fri, 9 Apr 2021 19:18:40 -0400 Message-Id: <20210409231842.8840-4-longman@redhat.com> In-Reply-To: <20210409231842.8840-1-longman@redhat.com> References: <20210409231842.8840-1-longman@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Rspamd-Queue-Id: 86AED600010E X-Stat-Signature: o9j67oc43ic13imxdzzwmtdy3nwuoj7e X-Rspamd-Server: rspam02 Received-SPF: none (redhat.com>: No applicable sender policy available) receiver=imf25; identity=mailfrom; envelope-from=""; helo=us-smtp-delivery-124.mimecast.com; client-ip=170.10.133.124 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1618010377-145072 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Before the new slab memory controller with per object byte charging, charging and vmstat data update happen only when new slab pages are allocated or freed. Now they are done with every kmem_cache_alloc() and kmem_cache_free(). This causes additional overhead for workloads that generate a lot of alloc and free calls. The memcg_stock_pcp is used to cache byte charge for a specific obj_cgroup to reduce that overhead. To further reducing it, this patch makes the vmstat data cached in the memcg_stock_pcp structure as well until it accumulates a page size worth of update or when other cached data change. On a 2-socket Cascade Lake server with instrumentation enabled and this patch applied, it was found that about 17% (946796 out of 5515184) of the time when __mod_obj_stock_state() is called leads to an actual call to mod_objcg_state() after initial boot. When doing parallel kernel build, the figure was about 16% (21894614 out of 139780628). So caching the vmstat data reduces the number of calls to mod_objcg_state() by more than 80%. Signed-off-by: Waiman Long --- mm/memcontrol.c | 78 +++++++++++++++++++++++++++++++++++++++++++------ mm/slab.h | 26 +++++++---------- 2 files changed, 79 insertions(+), 25 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index b19100c68aa0..539c3b632e47 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2220,7 +2220,10 @@ struct memcg_stock_pcp { #ifdef CONFIG_MEMCG_KMEM struct obj_cgroup *cached_objcg; + struct pglist_data *cached_pgdat; unsigned int nr_bytes; + int vmstat_idx; + int vmstat_bytes; #endif struct work_struct work; @@ -3157,6 +3160,21 @@ void __memcg_kmem_uncharge_page(struct page *page, int order) css_put(&memcg->css); } +static inline void mod_objcg_state(struct obj_cgroup *objcg, + struct pglist_data *pgdat, + enum node_stat_item idx, int nr) +{ + struct mem_cgroup *memcg; + struct lruvec *lruvec = NULL; + + rcu_read_lock(); + memcg = obj_cgroup_memcg(objcg); + if (pgdat) + lruvec = mem_cgroup_lruvec(memcg, pgdat); + __mod_memcg_lruvec_state(memcg, lruvec, idx, nr); + rcu_read_unlock(); +} + static bool consume_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes) { struct memcg_stock_pcp *stock; @@ -3207,6 +3225,14 @@ static void drain_obj_stock(struct memcg_stock_pcp *stock) stock->nr_bytes = 0; } + if (stock->vmstat_bytes) { + mod_objcg_state(old, stock->cached_pgdat, stock->vmstat_idx, + stock->vmstat_bytes); + stock->vmstat_bytes = 0; + stock->vmstat_idx = 0; + stock->cached_pgdat = NULL; + } + obj_cgroup_put(old); stock->cached_objcg = NULL; } @@ -3251,6 +3277,48 @@ static void refill_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes) local_irq_restore(flags); } +static void __mod_obj_stock_state(struct obj_cgroup *objcg, + struct pglist_data *pgdat, int idx, int nr) +{ + struct memcg_stock_pcp *stock = this_cpu_ptr(&memcg_stock); + + if (stock->cached_objcg != objcg) { + /* Output the current data as is */ + } else if (!stock->vmstat_bytes) { + /* Save the current data */ + stock->vmstat_bytes = nr; + stock->vmstat_idx = idx; + stock->cached_pgdat = pgdat; + nr = 0; + } else if ((stock->cached_pgdat != pgdat) || + (stock->vmstat_idx != idx)) { + /* Output the cached data & save the current data */ + swap(nr, stock->vmstat_bytes); + swap(idx, stock->vmstat_idx); + swap(pgdat, stock->cached_pgdat); + } else { + stock->vmstat_bytes += nr; + if (abs(nr) > PAGE_SIZE) { + nr = stock->vmstat_bytes; + stock->vmstat_bytes = 0; + } else { + nr = 0; + } + } + if (nr) + mod_objcg_state(objcg, pgdat, idx, nr); +} + +void mod_obj_stock_state(struct obj_cgroup *objcg, struct pglist_data *pgdat, + int idx, int nr) +{ + unsigned long flags; + + local_irq_save(flags); + __mod_obj_stock_state(objcg, pgdat, idx, nr); + local_irq_restore(flags); +} + int obj_cgroup_charge(struct obj_cgroup *objcg, gfp_t gfp, size_t size) { struct mem_cgroup *memcg; @@ -3300,18 +3368,10 @@ void obj_cgroup_uncharge_mod_state(struct obj_cgroup *objcg, size_t size, struct pglist_data *pgdat, int idx) { unsigned long flags; - struct mem_cgroup *memcg; - struct lruvec *lruvec = NULL; local_irq_save(flags); __refill_obj_stock(objcg, size); - - rcu_read_lock(); - memcg = obj_cgroup_memcg(objcg); - if (pgdat) - lruvec = mem_cgroup_lruvec(memcg, pgdat); - __mod_memcg_lruvec_state(memcg, lruvec, idx, -(int)size); - rcu_read_unlock(); + __mod_obj_stock_state(objcg, pgdat, idx, -(int)size); local_irq_restore(flags); } diff --git a/mm/slab.h b/mm/slab.h index 677cdc52e641..ae971975d9fc 100644 --- a/mm/slab.h +++ b/mm/slab.h @@ -239,6 +239,8 @@ static inline bool kmem_cache_debug_flags(struct kmem_cache *s, slab_flags_t fla #ifdef CONFIG_MEMCG_KMEM int memcg_alloc_page_obj_cgroups(struct page *page, struct kmem_cache *s, gfp_t gfp, bool new_page); +void mod_obj_stock_state(struct obj_cgroup *objcg, struct pglist_data *pgdat, + int idx, int nr); static inline void memcg_free_page_obj_cgroups(struct page *page) { @@ -283,20 +285,6 @@ static inline bool memcg_slab_pre_alloc_hook(struct kmem_cache *s, return true; } -static inline void mod_objcg_state(struct obj_cgroup *objcg, - struct pglist_data *pgdat, - enum node_stat_item idx, int nr) -{ - struct mem_cgroup *memcg; - struct lruvec *lruvec; - - rcu_read_lock(); - memcg = obj_cgroup_memcg(objcg); - lruvec = mem_cgroup_lruvec(memcg, pgdat); - mod_memcg_lruvec_state(memcg, lruvec, idx, nr); - rcu_read_unlock(); -} - static inline void memcg_slab_post_alloc_hook(struct kmem_cache *s, struct obj_cgroup *objcg, gfp_t flags, size_t size, @@ -324,8 +312,9 @@ static inline void memcg_slab_post_alloc_hook(struct kmem_cache *s, off = obj_to_index(s, page, p[i]); obj_cgroup_get(objcg); page_objcgs(page)[off] = objcg; - mod_objcg_state(objcg, page_pgdat(page), - cache_vmstat_idx(s), obj_full_size(s)); + mod_obj_stock_state(objcg, page_pgdat(page), + cache_vmstat_idx(s), + obj_full_size(s)); } else { obj_cgroup_uncharge(objcg, obj_full_size(s)); } @@ -408,6 +397,11 @@ static inline void memcg_slab_free_hook(struct kmem_cache *s, void **p, int objects) { } + +static void mod_obj_stock_state(struct obj_cgroup *objcg, + struct pglist_data *pgdat, int idx, int nr) +{ +} #endif /* CONFIG_MEMCG_KMEM */ static inline struct kmem_cache *virt_to_cache(const void *obj) From patchwork Fri Apr 9 23:18:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Waiman Long X-Patchwork-Id: 12195219 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E0FE9C43460 for ; Fri, 9 Apr 2021 23:19:43 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 7B973610C8 for ; Fri, 9 Apr 2021 23:19:43 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7B973610C8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 365036B0072; Fri, 9 Apr 2021 19:19:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3299B6B0074; Fri, 9 Apr 2021 19:19:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0F43E6B0074; Fri, 9 Apr 2021 19:19:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0068.hostedemail.com [216.40.44.68]) by kanga.kvack.org (Postfix) with ESMTP id E548D6B0072 for ; Fri, 9 Apr 2021 19:19:41 -0400 (EDT) Received: from smtpin20.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id A8FEDBF10 for ; Fri, 9 Apr 2021 23:19:41 +0000 (UTC) X-FDA: 78014397762.20.A1147E1 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [63.128.21.124]) by imf14.hostedemail.com (Postfix) with ESMTP id 8551FC0007DD for ; Fri, 9 Apr 2021 23:19:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1618010380; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:in-reply-to:in-reply-to:references:references; bh=oXVjTSR+aFPIAEydZVFSTtWuZOXw9aSIOWMRbXqKvKY=; b=DWl18srn/l1oe92QdXp/yzIdVnmcbmj65NoXX4kqoNfP+843JpT5eXZgcU0FxqW/WeqHwK D8eFvcqsfAZf2IQtfoHcOADnH6SVx55P2ezZ4l7qkdhRSNJQJRBRWHG6vG1guIaGfBd2Vc 1ZN0HuUXTtZd537uPwIi/UiE226RjDo= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-518-lGPiRvXiMlu_e_FpTWZXTQ-1; Fri, 09 Apr 2021 19:19:36 -0400 X-MC-Unique: lGPiRvXiMlu_e_FpTWZXTQ-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 49D94107ACC7; Fri, 9 Apr 2021 23:19:34 +0000 (UTC) Received: from llong.com (ovpn-113-226.rdu2.redhat.com [10.10.113.226]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5DCD46EF50; Fri, 9 Apr 2021 23:19:32 +0000 (UTC) From: Waiman Long To: Johannes Weiner , Michal Hocko , Vladimir Davydov , Andrew Morton , Tejun Heo , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Vlastimil Babka , Roman Gushchin Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, Shakeel Butt , Muchun Song , Alex Shi , Chris Down , Yafang Shao , Alexander Duyck , Wei Yang , Masayoshi Mizuma , Waiman Long Subject: [PATCH 4/5] mm/memcg: Separate out object stock data into its own struct Date: Fri, 9 Apr 2021 19:18:41 -0400 Message-Id: <20210409231842.8840-5-longman@redhat.com> In-Reply-To: <20210409231842.8840-1-longman@redhat.com> References: <20210409231842.8840-1-longman@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 8551FC0007DD X-Stat-Signature: 7ccy9owg9tmdu5kxamfojexhonas5oub Received-SPF: none (redhat.com>: No applicable sender policy available) receiver=imf14; identity=mailfrom; envelope-from=""; helo=us-smtp-delivery-124.mimecast.com; client-ip=63.128.21.124 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1618010375-362645 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The object stock data stored in struct memcg_stock_pcp are independent of the other page based data stored there. Separating them out into their own struct to highlight the independency. Signed-off-by: Waiman Long Acked-by: Roman Gushchin --- mm/memcontrol.c | 41 ++++++++++++++++++++++++++--------------- 1 file changed, 26 insertions(+), 15 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 539c3b632e47..69f728383efe 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2214,17 +2214,22 @@ void unlock_page_memcg(struct page *page) } EXPORT_SYMBOL(unlock_page_memcg); -struct memcg_stock_pcp { - struct mem_cgroup *cached; /* this never be root cgroup */ - unsigned int nr_pages; - +struct obj_stock { #ifdef CONFIG_MEMCG_KMEM struct obj_cgroup *cached_objcg; struct pglist_data *cached_pgdat; unsigned int nr_bytes; int vmstat_idx; int vmstat_bytes; +#else + int dummy[0]; #endif +}; + +struct memcg_stock_pcp { + struct mem_cgroup *cached; /* this never be root cgroup */ + unsigned int nr_pages; + struct obj_stock obj; struct work_struct work; unsigned long flags; @@ -2234,12 +2239,12 @@ static DEFINE_PER_CPU(struct memcg_stock_pcp, memcg_stock); static DEFINE_MUTEX(percpu_charge_mutex); #ifdef CONFIG_MEMCG_KMEM -static void drain_obj_stock(struct memcg_stock_pcp *stock); +static void drain_obj_stock(struct obj_stock *stock); static bool obj_stock_flush_required(struct memcg_stock_pcp *stock, struct mem_cgroup *root_memcg); #else -static inline void drain_obj_stock(struct memcg_stock_pcp *stock) +static inline void drain_obj_stock(struct obj_stock *stock) { } static bool obj_stock_flush_required(struct memcg_stock_pcp *stock, @@ -2249,6 +2254,13 @@ static bool obj_stock_flush_required(struct memcg_stock_pcp *stock, } #endif +static inline struct obj_stock *current_obj_stock(void) +{ + struct memcg_stock_pcp *stock = this_cpu_ptr(&memcg_stock); + + return &stock->obj; +} + /** * consume_stock: Try to consume stocked charge on this cpu. * @memcg: memcg to consume from. @@ -2315,7 +2327,7 @@ static void drain_local_stock(struct work_struct *dummy) local_irq_save(flags); stock = this_cpu_ptr(&memcg_stock); - drain_obj_stock(stock); + drain_obj_stock(&stock->obj); drain_stock(stock); clear_bit(FLUSHING_CACHED_CHARGE, &stock->flags); @@ -3177,13 +3189,13 @@ static inline void mod_objcg_state(struct obj_cgroup *objcg, static bool consume_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes) { - struct memcg_stock_pcp *stock; + struct obj_stock *stock; unsigned long flags; bool ret = false; local_irq_save(flags); - stock = this_cpu_ptr(&memcg_stock); + stock = current_obj_stock(); if (objcg == stock->cached_objcg && stock->nr_bytes >= nr_bytes) { stock->nr_bytes -= nr_bytes; ret = true; @@ -3194,7 +3206,7 @@ static bool consume_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes) return ret; } -static void drain_obj_stock(struct memcg_stock_pcp *stock) +static void drain_obj_stock(struct obj_stock *stock) { struct obj_cgroup *old = stock->cached_objcg; @@ -3242,8 +3254,8 @@ static bool obj_stock_flush_required(struct memcg_stock_pcp *stock, { struct mem_cgroup *memcg; - if (stock->cached_objcg) { - memcg = obj_cgroup_memcg(stock->cached_objcg); + if (stock->obj.cached_objcg) { + memcg = obj_cgroup_memcg(stock->obj.cached_objcg); if (memcg && mem_cgroup_is_descendant(memcg, root_memcg)) return true; } @@ -3253,9 +3265,8 @@ static bool obj_stock_flush_required(struct memcg_stock_pcp *stock, static void __refill_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes) { - struct memcg_stock_pcp *stock; + struct obj_stock *stock = current_obj_stock(); - stock = this_cpu_ptr(&memcg_stock); if (stock->cached_objcg != objcg) { /* reset if necessary */ drain_obj_stock(stock); obj_cgroup_get(objcg); @@ -3280,7 +3291,7 @@ static void refill_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes) static void __mod_obj_stock_state(struct obj_cgroup *objcg, struct pglist_data *pgdat, int idx, int nr) { - struct memcg_stock_pcp *stock = this_cpu_ptr(&memcg_stock); + struct obj_stock *stock = current_obj_stock(); if (stock->cached_objcg != objcg) { /* Output the current data as is */ From patchwork Fri Apr 9 23:18:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Waiman Long X-Patchwork-Id: 12195221 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E9DA0C43462 for ; Fri, 9 Apr 2021 23:19:45 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9C3ED610A6 for ; Fri, 9 Apr 2021 23:19:45 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9C3ED610A6 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B05E26B0073; Fri, 9 Apr 2021 19:19:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AB6F76B0074; Fri, 9 Apr 2021 19:19:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 894AC6B0075; Fri, 9 Apr 2021 19:19:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0094.hostedemail.com [216.40.44.94]) by kanga.kvack.org (Postfix) with ESMTP id 2C66A6B0073 for ; Fri, 9 Apr 2021 19:19:42 -0400 (EDT) Received: from smtpin18.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id E83E4181D75AB for ; Fri, 9 Apr 2021 23:19:41 +0000 (UTC) X-FDA: 78014397762.18.B325D52 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by imf08.hostedemail.com (Postfix) with ESMTP id 296E680192D5 for ; Fri, 9 Apr 2021 23:19:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1618010381; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:in-reply-to:in-reply-to:references:references; bh=VV6+ghNH1/AQ98hZ8A1K9EsRVAqRThkZsOYM3AOhXtQ=; b=U9P6F80ihpeprx0Kyuq97PW0JAH2Baa5O2x8Wtts6I08aOLBslR/N6he2fTE5NsomHCXaC ylckoa2vW9xE9xucQGGsJtLbpSByDOSx4NXofYBUJVrbEx3F/fKo/bABiAh+Or7ks+fDFh EeP4D35j20p0wWbJ1qcAKChrNc0OtsI= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-317-KuKrdhoMNLmBEUiaohrPzA-1; Fri, 09 Apr 2021 19:19:39 -0400 X-MC-Unique: KuKrdhoMNLmBEUiaohrPzA-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 5D8931020C20; Fri, 9 Apr 2021 23:19:36 +0000 (UTC) Received: from llong.com (ovpn-113-226.rdu2.redhat.com [10.10.113.226]) by smtp.corp.redhat.com (Postfix) with ESMTP id 6B4FC1B400; Fri, 9 Apr 2021 23:19:34 +0000 (UTC) From: Waiman Long To: Johannes Weiner , Michal Hocko , Vladimir Davydov , Andrew Morton , Tejun Heo , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Vlastimil Babka , Roman Gushchin Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org, Shakeel Butt , Muchun Song , Alex Shi , Chris Down , Yafang Shao , Alexander Duyck , Wei Yang , Masayoshi Mizuma , Waiman Long Subject: [PATCH 5/5] mm/memcg: Optimize user context object stock access Date: Fri, 9 Apr 2021 19:18:42 -0400 Message-Id: <20210409231842.8840-6-longman@redhat.com> In-Reply-To: <20210409231842.8840-1-longman@redhat.com> References: <20210409231842.8840-1-longman@redhat.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 296E680192D5 X-Stat-Signature: da7w6wjzya9rxnz5ochqu3pjxwyfffuf Received-SPF: none (redhat.com>: No applicable sender policy available) receiver=imf08; identity=mailfrom; envelope-from=""; helo=us-smtp-delivery-124.mimecast.com; client-ip=216.205.24.124 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1618010372-675554 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Most kmem_cache_alloc() calls are from user context. With instrumentation enabled, the measured amount of kmem_cache_alloc() calls from non-task context was about 0.01% of the total. The irq disable/enable sequence used in this case to access content from object stock is slow. To optimize for user context access, there are now two object stocks for task context and interrupt context access respectively. The task context object stock can be accessed after disabling preemption which is cheap in non-preempt kernel. The interrupt context object stock can only be accessed after disabling interrupt. User context code can access interrupt object stock, but not vice versa. The mod_objcg_state() function is also modified to make sure that memcg and lruvec stat updates are done with interrupted disabled. The downside of this change is that there are more data stored in local object stocks and not reflected in the charge counter and the vmstat arrays. However, this is a small price to pay for better performance. Signed-off-by: Waiman Long Reported-by: kernel test robot Acked-by: Roman Gushchin --- mm/memcontrol.c | 71 +++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 57 insertions(+), 14 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 69f728383efe..00c9074e42e5 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2229,7 +2229,8 @@ struct obj_stock { struct memcg_stock_pcp { struct mem_cgroup *cached; /* this never be root cgroup */ unsigned int nr_pages; - struct obj_stock obj; + struct obj_stock task_obj; + struct obj_stock irq_obj; struct work_struct work; unsigned long flags; @@ -2254,11 +2255,46 @@ static bool obj_stock_flush_required(struct memcg_stock_pcp *stock, } #endif +/* + * Most kmem_cache_alloc() calls are from user context. The irq disable/enable + * sequence used in this case to access content from object stock is slow. + * To optimize for user context access, there are now two object stocks for + * task context and interrupt context access respectively. + * + * The task context object stock can be accessed by disabling preemption only + * which is cheap in non-preempt kernel. The interrupt context object stock + * can only be accessed after disabling interrupt. User context code can + * access interrupt object stock, but not vice versa. + */ static inline struct obj_stock *current_obj_stock(void) { struct memcg_stock_pcp *stock = this_cpu_ptr(&memcg_stock); - return &stock->obj; + return in_task() ? &stock->task_obj : &stock->irq_obj; +} + +#define get_obj_stock(flags) \ +({ \ + struct memcg_stock_pcp *stock; \ + struct obj_stock *obj_stock; \ + \ + if (in_task()) { \ + preempt_disable(); \ + (flags) = -1L; \ + obj_stock = &stock->task_obj; \ + } else { \ + local_irq_save(flags); \ + obj_stock = &stock->irq_obj; \ + } \ + obj_stock; \ +}) + +static inline void put_obj_stock(unsigned long flags) +{ + if (flags == -1L) + preempt_enable(); + else + local_irq_restore(flags); } /** @@ -2327,7 +2363,9 @@ static void drain_local_stock(struct work_struct *dummy) local_irq_save(flags); stock = this_cpu_ptr(&memcg_stock); - drain_obj_stock(&stock->obj); + drain_obj_stock(&stock->irq_obj); + if (in_task()) + drain_obj_stock(&stock->task_obj); drain_stock(stock); clear_bit(FLUSHING_CACHED_CHARGE, &stock->flags); @@ -3183,7 +3221,7 @@ static inline void mod_objcg_state(struct obj_cgroup *objcg, memcg = obj_cgroup_memcg(objcg); if (pgdat) lruvec = mem_cgroup_lruvec(memcg, pgdat); - __mod_memcg_lruvec_state(memcg, lruvec, idx, nr); + mod_memcg_lruvec_state(memcg, lruvec, idx, nr); rcu_read_unlock(); } @@ -3193,7 +3231,7 @@ static bool consume_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes) unsigned long flags; bool ret = false; - local_irq_save(flags); + stock = get_obj_stock(flags); stock = current_obj_stock(); if (objcg == stock->cached_objcg && stock->nr_bytes >= nr_bytes) { @@ -3201,7 +3239,7 @@ static bool consume_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes) ret = true; } - local_irq_restore(flags); + put_obj_stock(flags); return ret; } @@ -3254,8 +3292,13 @@ static bool obj_stock_flush_required(struct memcg_stock_pcp *stock, { struct mem_cgroup *memcg; - if (stock->obj.cached_objcg) { - memcg = obj_cgroup_memcg(stock->obj.cached_objcg); + if (in_task() && stock->task_obj.cached_objcg) { + memcg = obj_cgroup_memcg(stock->task_obj.cached_objcg); + if (memcg && mem_cgroup_is_descendant(memcg, root_memcg)) + return true; + } + if (stock->irq_obj.cached_objcg) { + memcg = obj_cgroup_memcg(stock->irq_obj.cached_objcg); if (memcg && mem_cgroup_is_descendant(memcg, root_memcg)) return true; } @@ -3283,9 +3326,9 @@ static void refill_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes) { unsigned long flags; - local_irq_save(flags); + get_obj_stock(flags); __refill_obj_stock(objcg, nr_bytes); - local_irq_restore(flags); + put_obj_stock(flags); } static void __mod_obj_stock_state(struct obj_cgroup *objcg, @@ -3325,9 +3368,9 @@ void mod_obj_stock_state(struct obj_cgroup *objcg, struct pglist_data *pgdat, { unsigned long flags; - local_irq_save(flags); + get_obj_stock(flags); __mod_obj_stock_state(objcg, pgdat, idx, nr); - local_irq_restore(flags); + put_obj_stock(flags); } int obj_cgroup_charge(struct obj_cgroup *objcg, gfp_t gfp, size_t size) @@ -3380,10 +3423,10 @@ void obj_cgroup_uncharge_mod_state(struct obj_cgroup *objcg, size_t size, { unsigned long flags; - local_irq_save(flags); + get_obj_stock(flags); __refill_obj_stock(objcg, size); __mod_obj_stock_state(objcg, pgdat, idx, -(int)size); - local_irq_restore(flags); + put_obj_stock(flags); } #endif /* CONFIG_MEMCG_KMEM */