From patchwork Fri Nov 26 04:06:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 12640085 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BA7D9C433EF for ; Fri, 26 Nov 2021 04:07:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2CD066B0075; Thu, 25 Nov 2021 23:06:50 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 27D246B0078; Thu, 25 Nov 2021 23:06:50 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1452B6B007B; Thu, 25 Nov 2021 23:06:50 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0171.hostedemail.com [216.40.44.171]) by kanga.kvack.org (Postfix) with ESMTP id 0626A6B0075 for ; Thu, 25 Nov 2021 23:06:50 -0500 (EST) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id AFE9818503B21 for ; Fri, 26 Nov 2021 04:06:39 +0000 (UTC) X-FDA: 78849744918.25.42EC419 Received: from mail-pg1-f169.google.com (mail-pg1-f169.google.com [209.85.215.169]) by imf23.hostedemail.com (Postfix) with ESMTP id 176A190000AF for ; Fri, 26 Nov 2021 04:06:32 +0000 (UTC) Received: by mail-pg1-f169.google.com with SMTP id q16so6963439pgq.10 for ; Thu, 25 Nov 2021 20:06:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=dxrYhHc6aKpqxzzlVemaKlJCRKgcCo/zpvcwMEFod1g=; b=3V+niI3eAGEI5joOyhomPDevZI7q++8ohy0CwrDueX/T1rts+xnHL3Zch+LcP2J5B4 Elrcyy2kHzVGkFFJo8IHD2q7z1naXv+/4MtHX4Qbi/XQ7VqlHZFooSUUm6ICLjxTq3Pn jZJFl1HzrkARG8yp1mYHVrlDGl4Uc0p5XPNEU67Ot2dsTvrP5wMOJ3qKF22x4eKrbYs3 8I88SlZPAi8ymtZpCWOWR1Rs+M7zpuKIJeTMAi9WfSXoTZCz2mtEFOL2qCkcuxWxmeXd 03FAWX5Krb1fHSU/phNtzEQPI7/c/wVnzMJmkkekTW9+mYVCHcABNAbwk5AiXdXZSUb5 ukQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=dxrYhHc6aKpqxzzlVemaKlJCRKgcCo/zpvcwMEFod1g=; b=r3iLsXCG/ol7d5SyIxGtDVB1RRMp88Yj9CBJcOI6mS+i4qBN87F9d6KBKmxAEk1pG4 iY0iW3V27OGCiZFB9i8PRcAAjkkD0nTGvPFw9ZK/YebiSTTAQHpue8CiJFvlbzqi2rUB y02N7Q/txtVNqVF94geyXrES8N8ECvRjszwull21nxxYts4vpkCda1j4yg+yZHzItkhk Owi620pIRztBtXLsChO2t2BfDHXIFud4t0Ed7TEJR7cnZZObcKwgGYPtfGchFc7cHANH zSqFheIlruKwKq+puWsLrTA9gYFicVFwSpysxBFS4aaMo7wlvl24j16Mb6j+DHwb6OPw IgDw== X-Gm-Message-State: AOAM532cXZ0zPFYQKzj3egGXz56mvUsAMEp8oLkbquTYIJqZt53fT+Av 31x26cPoUMO9dYcolVdM1R2vHQ== X-Google-Smtp-Source: ABdhPJzi55oolxqTbTy0h1eAiBDK69Il6vfa929IKg6Wco4j8PtxJNIhVNBic+MiFM9sOf8no2sKjg== X-Received: by 2002:a05:6a00:21c4:b0:4a7:ec46:34b7 with SMTP id t4-20020a056a0021c400b004a7ec4634b7mr11126252pfj.15.1637899597210; Thu, 25 Nov 2021 20:06:37 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([139.177.225.226]) by smtp.gmail.com with ESMTPSA id f4sm5406045pfj.61.2021.11.25.20.06.33 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 25 Nov 2021 20:06:36 -0800 (PST) From: Qi Zheng To: akpm@linux-foundation.org, dennis@kernel.org, tj@kernel.org, cl@linux.com Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, songmuchun@bytedance.com, Qi Zheng Subject: [PATCH] mm: memcg/percpu: account extra objcg space to memory cgroups Date: Fri, 26 Nov 2021 12:06:06 +0800 Message-Id: <20211126040606.97836-1-zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) MIME-Version: 1.0 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 176A190000AF X-Stat-Signature: jpaz9ss6rcwoa1aqpjkdzhybz7xqdoyh Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=3V+niI3e; spf=pass (imf23.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.215.169 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com X-HE-Tag: 1637899592-796040 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Similar to slab memory allocator, for each accounted percpu object there is an extra space which is used to store obj_cgroup membership. Charge it too. Signed-off-by: Qi Zheng --- mm/percpu-internal.h | 17 +++++++++++++++++ mm/percpu.c | 10 +++++----- 2 files changed, 22 insertions(+), 5 deletions(-) diff --git a/mm/percpu-internal.h b/mm/percpu-internal.h index 639662c20c82..030a176b5b8e 100644 --- a/mm/percpu-internal.h +++ b/mm/percpu-internal.h @@ -113,6 +113,23 @@ static inline int pcpu_chunk_map_bits(struct pcpu_chunk *chunk) return pcpu_nr_pages_to_map_bits(chunk->nr_pages); } +#ifdef CONFIG_MEMCG_KMEM +/** + * pcpu_obj_full_size - helper to calculate size of each accounted object + * @size: size of area to allocate in bytes + * + * For each accounted object there is an extra space which is used to store + * obj_cgroup membership. Charge it too. + */ +static inline size_t pcpu_obj_full_size(size_t size) +{ + size_t extra_size = + size / PCPU_MIN_ALLOC_SIZE * sizeof(struct obj_cgroup *); + + return size * num_possible_cpus() + extra_size; +} +#endif /* CONFIG_MEMCG_KMEM */ + #ifdef CONFIG_PERCPU_STATS #include diff --git a/mm/percpu.c b/mm/percpu.c index 293009cc03ef..f5c9a3faadee 100644 --- a/mm/percpu.c +++ b/mm/percpu.c @@ -1635,7 +1635,7 @@ static bool pcpu_memcg_pre_alloc_hook(size_t size, gfp_t gfp, if (!objcg) return true; - if (obj_cgroup_charge(objcg, gfp, size * num_possible_cpus())) { + if (obj_cgroup_charge(objcg, gfp, pcpu_obj_full_size(size))) { obj_cgroup_put(objcg); return false; } @@ -1656,10 +1656,10 @@ static void pcpu_memcg_post_alloc_hook(struct obj_cgroup *objcg, rcu_read_lock(); mod_memcg_state(obj_cgroup_memcg(objcg), MEMCG_PERCPU_B, - size * num_possible_cpus()); + pcpu_obj_full_size(size)); rcu_read_unlock(); } else { - obj_cgroup_uncharge(objcg, size * num_possible_cpus()); + obj_cgroup_uncharge(objcg, pcpu_obj_full_size(size)); obj_cgroup_put(objcg); } } @@ -1676,11 +1676,11 @@ static void pcpu_memcg_free_hook(struct pcpu_chunk *chunk, int off, size_t size) return; chunk->obj_cgroups[off >> PCPU_MIN_ALLOC_SHIFT] = NULL; - obj_cgroup_uncharge(objcg, size * num_possible_cpus()); + obj_cgroup_uncharge(objcg, pcpu_obj_full_size(size)); rcu_read_lock(); mod_memcg_state(obj_cgroup_memcg(objcg), MEMCG_PERCPU_B, - -(size * num_possible_cpus())); + -pcpu_obj_full_size(size)); rcu_read_unlock(); obj_cgroup_put(objcg);