From patchwork Tue Jul 30 15:01:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 13747518 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C3910C3DA70 for ; Tue, 30 Jul 2024 15:02:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3C7BF6B009A; Tue, 30 Jul 2024 11:02:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3505C6B00A1; Tue, 30 Jul 2024 11:02:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1CAA06B00A2; Tue, 30 Jul 2024 11:02:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id EC1F86B00A1 for ; Tue, 30 Jul 2024 11:02:37 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 89F84A43F7 for ; Tue, 30 Jul 2024 15:02:37 +0000 (UTC) X-FDA: 82396735554.17.B6EBB0E Received: from mail-qk1-f173.google.com (mail-qk1-f173.google.com [209.85.222.173]) by imf25.hostedemail.com (Postfix) with ESMTP id 5F912A0037 for ; Tue, 30 Jul 2024 15:02:33 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b=xGXw0ucR; spf=pass (imf25.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.222.173 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=pass (policy=none) header.from=soleen.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1722351725; a=rsa-sha256; cv=none; b=V2KTg/6YVMqejpg2XIFeDy7Xgxa2Axlk0pTtlc5UF2i8FTYUK4ibDLXX2bjyQyEDWuxkDw CFbz9ryyNEx+jrKtOynfSojL0eiazeOvKlQtuDI7UGNQGKgEJ+DYTuaxxI7Ji3j7r5bKza WHnz/2pVGguhl5MD/N0GACi+CxEFG+w= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b=xGXw0ucR; spf=pass (imf25.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.222.173 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=pass (policy=none) header.from=soleen.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1722351725; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=qK7WOu+hv7tl+cxtg7CXPtDPtjLG3K7kUlaYAaWdes0=; b=KaL9s6wjaKEgtJ2liIK2XT9XyZhGPsfXN/yfOIMQsZiuqcv8Px4HRxKsXPmbelfamyYjTW f4Tw6GhThDcG/Jn9UQ86O4vgZH3194huemJECJusBRFj7TqswDx4TLPYshA6+jBOtDkaR4 GKn5zNX0vEhss4tVnzOhNKfe3vk6E70= Received: by mail-qk1-f173.google.com with SMTP id af79cd13be357-7a1d0ad7113so319575385a.2 for ; Tue, 30 Jul 2024 08:02:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen-com.20230601.gappssmtp.com; s=20230601; t=1722351752; x=1722956552; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=qK7WOu+hv7tl+cxtg7CXPtDPtjLG3K7kUlaYAaWdes0=; b=xGXw0ucRoSX3KJKbrlrXuxlkXqRPIekaaCR9PFcR5B87lYQeOyTKIUH2ZZQeHOSP9V LO6AjiZ1DTE2Bh1Xz32DE3SHkAyb1qL4qiooYK/h2dnVQYF9YCC1R3ql7X+Y3klnvGyu hUSe2d5DgPo0abtbciV74cVxeBlItdxktdi32Kjxk3OIWp4PuuaGII33IWFCxzhzVeMA RTH1yhl5JIXy/xA0zeeuH6lE3bJeaGiD3qKSvScaDIcQwnAWVOxEjAU6U1uUDhOzOZjI mc/kTm3BW1YQmJR5TTDjIeUZ2KMoXL3jy2ZrELWz8b0zPZh2VH6r9ZxPLaHNwm7nDXa8 cY1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722351752; x=1722956552; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=qK7WOu+hv7tl+cxtg7CXPtDPtjLG3K7kUlaYAaWdes0=; b=wjOf086WSjbicmRAxRVR4/m4hox1IEcy2Gdtzrp7Be7IfQOY24cmycr/mArCbHgTDb 6JkHCcXir/TKh9A4OPA19jBX8BcozQMWNwQCTIzAdJpxAojEiaEmzEdQHdfCUBuIXxKd djcFZ14r846TiBVXjvp3wEzdTXBatKl/tW8zfiZe7/1m4NL3wLY36BHwEEZWJZUlFa7x UYEyO2dg84m2uA16m0RSiP5/hKH2kraeU5GsJDXPRk6QQdowp+Dffh07QXQytFzIzhIs XAvu31UjxtYJ7/Y9WB+FC9UiK8aD4Rc9MF5TvLCNRujlrfEeeWrztUuUeRJBJarYz6Qa 7mBg== X-Forwarded-Encrypted: i=1; AJvYcCVQYulriWIXsKHb/qyXUmyZKrrZYXGMdfbFA6vhIGTHmMjqlnTcm84aHls3jeucJEuP5RWxsq+PFFnN9GJEFDM/kWE= X-Gm-Message-State: AOJu0Yz+tCQsv9lBSkJaXt2dG6GbSYwKY87BdPcH/49enGnY/fmWe83p pnkrq1sNl6QsfgVgjo9BxeC9D09+d6cmPA41UD7pq6kTwvxn4Q6AqQJNxoPo2/o= X-Google-Smtp-Source: AGHT+IGuBFGt6MK7hI5v2XIYYwSP41dPGObuYRlJA5D+J3MWpue7Ez/s4WblQus1adMdjSSsNHR4iQ== X-Received: by 2002:a05:620a:179f:b0:79c:e7d:22b6 with SMTP id af79cd13be357-7a1e525fafbmr1390496785a.39.1722351722274; Tue, 30 Jul 2024 08:02:02 -0700 (PDT) Received: from soleen.c.googlers.com.com (197.5.86.34.bc.googleusercontent.com. [34.86.5.197]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a1d73efffdsm645934285a.69.2024.07.30.08.02.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 30 Jul 2024 08:02:01 -0700 (PDT) From: Pasha Tatashin To: akpm@linux-foundation.org, jpoimboe@kernel.org, pasha.tatashin@soleen.com, kent.overstreet@linux.dev, peterz@infradead.org, nphamcs@gmail.com, cerasuolodomenico@gmail.com, surenb@google.com, lizhijian@fujitsu.com, willy@infradead.org, shakeel.butt@linux.dev, vbabka@suse.cz, ziy@nvidia.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, yosryahmed@google.com Subject: [PATCH v6 1/3] memcg: increase the valid index range for memcg stats Date: Tue, 30 Jul 2024 15:01:56 +0000 Message-ID: <20240730150158.832783-2-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.46.0.rc1.232.g9752f9e123-goog In-Reply-To: <20240730150158.832783-1-pasha.tatashin@soleen.com> References: <20240730150158.832783-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Stat-Signature: 97pomtgkoyxm11h9g8m3gxi5iifxswry X-Rspamd-Queue-Id: 5F912A0037 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1722351753-869746 X-HE-Meta: U2FsdGVkX1+OYp/94hLI3CPHGH7FztXWONee81jNyty0+OZKFM/wzqlAqrS/u0cT+yUVdDmzj3q1ksagBJFfYkLV55HQbVY/hUgmuYRlPcqqgqWNmVmXyMLS0ksieD2tWVraANRW4RvHPWBADbO785Mq5TmQJXZc2Luu11pvYEKT2KXJAxaQD/0HhcBbJ1kptaArrOdRs/20EdvBstlExZV+fMVnDKDg/hlzgFt4sJ3WGq4d97AUQS56rzfaP8AWBjH7rz/jNTyZFRWF4eiRd6Z/gUeBwla6pG0RZZUQ7crbN7ZwJ/lkrLTuZ7GodV6qKDE3X2r9JzAtdIqc5T23BknIQWLuVAhNE1se++6LvPpS8zk4BihrlaQ5R01ocR5QBtIEepM5VvSnheNOc77qWJA3gvMjbJFF6zHdqo3tzotaFpyeP1T7Kj1KGXYuwEXnfhhMg5CQTfKz712cm9KOneV1jgR1PH2lmfWyrblNLyegL7giLYvofaaqO+C9aPhVccfapGEWxHfweGg355IpdFMNT8KiUeQr+y7FaUGkTiVI3zVijIwl+Xs1U+ssxO9qKbKMKcAm5RJoYKDHyLaLosrJdaiCtJxqvUOrqWy1G0YTiGVNX0A/7/WjJTsAU50b+Rnm0DlG53ls2QCa5QyT3LjMt6EUOYNz5MwWdDRlK3tYThG/85xFWMGtCWUbCDykeKdWeaB5GAuOqoD0AZf/z3rggxHOtB3+cR726xRWFhgm0jHW6nH+hxKr1qBFeWFEGU2cvxZPpGcxWPA32EDjHNpNggEAdKiQwczeD0d8IYdTyP88mnd8Ht60Jp3op5Y94ITcv3Zkfv5hA9tirQYi3LFENpZ0nvB0oGZh5234uR/1o13aYUo9w+BQ+j7gLUG83ceIPZqAFN+LylaZ8Kgtx9DsnR061ZYGjNry7VO71hfyn5dwnmgmXNEHmzAGqtowyAIXH5zPL0nFz3Qq6wg vHQu+GQP 8IheaI/8D87CQG2ifzutQDzrdCO+2aDDvOw87dDc9FDbG+IBbXSOHtyztXac7pvXPjMsETb68VW02y6Mr+j2bUKhHLq3sMMDXIdBhno4SrkVLpEgs6MYz7gmF5++K69XwAjZVeHTaa34rRFtKK5VL8AlxCpfSnwpbt2jAzk2pyaTg3KD6y2bZUJI5uwJeb3QDDYa26DxECNCvnmHVvcsbLtOplxZVpok1w1N7p9HWSXh54pYAyxTBGQyFN7flGc0+JZ2cF63FeOd7bgnwdLtFRj5HJXHmJ6twoX2yUSUptYtbJWSn0xFAgl6LObXUvgnE/Z2V3a6XF49HtHLV5dqN26Ujv8lyM5LiH+LhxV4uIkthrto= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000011, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Shakeel Butt At the moment the valid index for the indirection tables for memcg stats and events is < S8_MAX. These indirection tables are used in performance critical codepaths. With the latest addition to the vm_events, the NR_VM_EVENT_ITEMS has gone over S8_MAX. One way to resolve is to increase the entry size of the indirection table from int8_t to int16_t but this will increase the potential number of cachelines needed to access the indirection table. This patch took a different approach and make the valid index < U8_MAX. In this way the size of the indirection tables will remain same and we only need to invalid index check from less than 0 to equal to U8_MAX. In this approach we have also removed a subtraction from the performance critical codepaths. Signed-off-by: Shakeel Butt Co-developed-by: Pasha Tatashin Signed-off-by: Pasha Tatashin Reviewed-by: Yosry Ahmed --- mm/memcontrol.c | 50 +++++++++++++++++++++++++++---------------------- 1 file changed, 28 insertions(+), 22 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 960371788687..84f383952d32 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -320,24 +320,27 @@ static const unsigned int memcg_stat_items[] = { #define NR_MEMCG_NODE_STAT_ITEMS ARRAY_SIZE(memcg_node_stat_items) #define MEMCG_VMSTAT_SIZE (NR_MEMCG_NODE_STAT_ITEMS + \ ARRAY_SIZE(memcg_stat_items)) -static int8_t mem_cgroup_stats_index[MEMCG_NR_STAT] __read_mostly; +#define BAD_STAT_IDX(index) ((u32)(index) >= U8_MAX) +static u8 mem_cgroup_stats_index[MEMCG_NR_STAT] __read_mostly; static void init_memcg_stats(void) { - int8_t i, j = 0; + u8 i, j = 0; - BUILD_BUG_ON(MEMCG_NR_STAT >= S8_MAX); + BUILD_BUG_ON(MEMCG_NR_STAT >= U8_MAX); - for (i = 0; i < NR_MEMCG_NODE_STAT_ITEMS; ++i) - mem_cgroup_stats_index[memcg_node_stat_items[i]] = ++j; + memset(mem_cgroup_stats_index, U8_MAX, sizeof(mem_cgroup_stats_index)); - for (i = 0; i < ARRAY_SIZE(memcg_stat_items); ++i) - mem_cgroup_stats_index[memcg_stat_items[i]] = ++j; + for (i = 0; i < NR_MEMCG_NODE_STAT_ITEMS; ++i, ++j) + mem_cgroup_stats_index[memcg_node_stat_items[i]] = j; + + for (i = 0; i < ARRAY_SIZE(memcg_stat_items); ++i, ++j) + mem_cgroup_stats_index[memcg_stat_items[i]] = j; } static inline int memcg_stats_index(int idx) { - return mem_cgroup_stats_index[idx] - 1; + return mem_cgroup_stats_index[idx]; } struct lruvec_stats_percpu { @@ -369,7 +372,7 @@ unsigned long lruvec_page_state(struct lruvec *lruvec, enum node_stat_item idx) return node_page_state(lruvec_pgdat(lruvec), idx); i = memcg_stats_index(idx); - if (WARN_ONCE(i < 0, "%s: missing stat item %d\n", __func__, idx)) + if (WARN_ONCE(BAD_STAT_IDX(i), "%s: missing stat item %d\n", __func__, idx)) return 0; pn = container_of(lruvec, struct mem_cgroup_per_node, lruvec); @@ -392,7 +395,7 @@ unsigned long lruvec_page_state_local(struct lruvec *lruvec, return node_page_state(lruvec_pgdat(lruvec), idx); i = memcg_stats_index(idx); - if (WARN_ONCE(i < 0, "%s: missing stat item %d\n", __func__, idx)) + if (WARN_ONCE(BAD_STAT_IDX(i), "%s: missing stat item %d\n", __func__, idx)) return 0; pn = container_of(lruvec, struct mem_cgroup_per_node, lruvec); @@ -435,21 +438,24 @@ static const unsigned int memcg_vm_event_stat[] = { }; #define NR_MEMCG_EVENTS ARRAY_SIZE(memcg_vm_event_stat) -static int8_t mem_cgroup_events_index[NR_VM_EVENT_ITEMS] __read_mostly; +static u8 mem_cgroup_events_index[NR_VM_EVENT_ITEMS] __read_mostly; static void init_memcg_events(void) { - int8_t i; + u8 i; + + BUILD_BUG_ON(NR_VM_EVENT_ITEMS >= U8_MAX); - BUILD_BUG_ON(NR_VM_EVENT_ITEMS >= S8_MAX); + memset(mem_cgroup_events_index, U8_MAX, + sizeof(mem_cgroup_events_index)); for (i = 0; i < NR_MEMCG_EVENTS; ++i) - mem_cgroup_events_index[memcg_vm_event_stat[i]] = i + 1; + mem_cgroup_events_index[memcg_vm_event_stat[i]] = i; } static inline int memcg_events_index(enum vm_event_item idx) { - return mem_cgroup_events_index[idx] - 1; + return mem_cgroup_events_index[idx]; } struct memcg_vmstats_percpu { @@ -621,7 +627,7 @@ unsigned long memcg_page_state(struct mem_cgroup *memcg, int idx) long x; int i = memcg_stats_index(idx); - if (WARN_ONCE(i < 0, "%s: missing stat item %d\n", __func__, idx)) + if (WARN_ONCE(BAD_STAT_IDX(i), "%s: missing stat item %d\n", __func__, idx)) return 0; x = READ_ONCE(memcg->vmstats->state[i]); @@ -662,7 +668,7 @@ void __mod_memcg_state(struct mem_cgroup *memcg, enum memcg_stat_item idx, if (mem_cgroup_disabled()) return; - if (WARN_ONCE(i < 0, "%s: missing stat item %d\n", __func__, idx)) + if (WARN_ONCE(BAD_STAT_IDX(i), "%s: missing stat item %d\n", __func__, idx)) return; __this_cpu_add(memcg->vmstats_percpu->state[i], val); @@ -675,7 +681,7 @@ unsigned long memcg_page_state_local(struct mem_cgroup *memcg, int idx) long x; int i = memcg_stats_index(idx); - if (WARN_ONCE(i < 0, "%s: missing stat item %d\n", __func__, idx)) + if (WARN_ONCE(BAD_STAT_IDX(i), "%s: missing stat item %d\n", __func__, idx)) return 0; x = READ_ONCE(memcg->vmstats->state_local[i]); @@ -694,7 +700,7 @@ static void __mod_memcg_lruvec_state(struct lruvec *lruvec, struct mem_cgroup *memcg; int i = memcg_stats_index(idx); - if (WARN_ONCE(i < 0, "%s: missing stat item %d\n", __func__, idx)) + if (WARN_ONCE(BAD_STAT_IDX(i), "%s: missing stat item %d\n", __func__, idx)) return; pn = container_of(lruvec, struct mem_cgroup_per_node, lruvec); @@ -810,7 +816,7 @@ void __count_memcg_events(struct mem_cgroup *memcg, enum vm_event_item idx, if (mem_cgroup_disabled()) return; - if (WARN_ONCE(i < 0, "%s: missing stat item %d\n", __func__, idx)) + if (WARN_ONCE(BAD_STAT_IDX(i), "%s: missing stat item %d\n", __func__, idx)) return; memcg_stats_lock(); @@ -823,7 +829,7 @@ unsigned long memcg_events(struct mem_cgroup *memcg, int event) { int i = memcg_events_index(event); - if (WARN_ONCE(i < 0, "%s: missing stat item %d\n", __func__, event)) + if (WARN_ONCE(BAD_STAT_IDX(i), "%s: missing stat item %d\n", __func__, event)) return 0; return READ_ONCE(memcg->vmstats->events[i]); @@ -833,7 +839,7 @@ unsigned long memcg_events_local(struct mem_cgroup *memcg, int event) { int i = memcg_events_index(event); - if (WARN_ONCE(i < 0, "%s: missing stat item %d\n", __func__, event)) + if (WARN_ONCE(BAD_STAT_IDX(i), "%s: missing stat item %d\n", __func__, event)) return 0; return READ_ONCE(memcg->vmstats->events_local[i]); From patchwork Tue Jul 30 15:01:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 13747519 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3DCB2C3DA70 for ; Tue, 30 Jul 2024 15:02:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C7C356B00A5; Tue, 30 Jul 2024 11:02:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C05376B00A6; Tue, 30 Jul 2024 11:02:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A56886B00A7; Tue, 30 Jul 2024 11:02:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 7F05F6B00A5 for ; Tue, 30 Jul 2024 11:02:47 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 3A81140369 for ; Tue, 30 Jul 2024 15:02:47 +0000 (UTC) X-FDA: 82396735974.04.825D0BA Received: from mail-qk1-f169.google.com (mail-qk1-f169.google.com [209.85.222.169]) by imf02.hostedemail.com (Postfix) with ESMTP id D3E1880036 for ; Tue, 30 Jul 2024 15:02:33 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b=tcneG1Ol; spf=pass (imf02.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.222.169 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=pass (policy=none) header.from=soleen.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1722351700; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=p8PfWoyQ/BZ4vL4BkdBJYX/P6ybvfxcH/D6ib7uXKjM=; b=Uv+GSlr0d/Vj8Rwe/aAXSnivsGX2hlpF3uA1qoEVKmb1IuB7TYPbX3UJKRDuZemOLRcW2T FlfGUqtAsOHKh65XECMm/9id4gzTGfLlHQyDfwIBMiyDRLH6aWgGq0w4TxqLUOIyu7Jh7y enuO8EwDPTppgbVXEj9beh9P82N+vuQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1722351700; a=rsa-sha256; cv=none; b=WcgeECTFHFSX2yLWRDuaLf0DCFQohtDztPxsxRBBcV73GvM9To2vqs2KsOhgdkXEfYWflR vfSYtUGdci96sfxkpkMTUFgZgYmQXUm06EGmTtbtOoqIk97PrI9gt1NnUUMAanFQxWzJ+N Beo0qSvuvKxO3t9xFWhjuwrwkXQSl3Q= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b=tcneG1Ol; spf=pass (imf02.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.222.169 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=pass (policy=none) header.from=soleen.com Received: by mail-qk1-f169.google.com with SMTP id af79cd13be357-7a1df0a93eeso261202285a.1 for ; Tue, 30 Jul 2024 08:02:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen-com.20230601.gappssmtp.com; s=20230601; t=1722351753; x=1722956553; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=p8PfWoyQ/BZ4vL4BkdBJYX/P6ybvfxcH/D6ib7uXKjM=; b=tcneG1OlUTtM+T93k6On/Rmpna0qYE6Qlm5/f2ASEEsw8AgQG+ap8jO6rXQjjzbdpU UZE6kpjV+tiPX+lHySfZ7IFgV1qwFhUUUp4NmlIFPArfY6DdlHTH5gahpnc4pjH2lKxf qNc8+fXRnlNCxhvjdrsJ0/Jk1zuAv+WxPpt0UDugDtF7+gV0CMOREqlEhWa5Kiiv5W6k No5eAWERY0cwOBQGHo3ncSklAixobMn0pPwtl3Pj74SH0hplsMA09ETUuo+FjvKpRV/D WPHkd1qxEUuL107jqF/mGCj5QXwHAbZU2D594YNChE/wRXbhMNohK3FXjqbvAIlMG0rX N3yQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722351753; x=1722956553; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=p8PfWoyQ/BZ4vL4BkdBJYX/P6ybvfxcH/D6ib7uXKjM=; b=v654hD9ojZtALJYp4lBszM0lIDfmAEYwBdGKCW9Ou42xb0VWioCbJ9mU5Qx/MGf49/ MFKnjTB38OqzTOmr6mKvddfdOF/WgmayIydEvZXWg33orRg8sWBz9VtnUEHAijPsl3JC EoF+0J+lg817qEh2zmc8T7g6rIX+F2t+p+XZbXjloA/h/zYP5dbPsgEWjhoaNHdbCcNd 8cjr1vN02a4DJghrggEXPrhJpW18DzD0Hm+zdtiJBs3fPUIR5taspojSbNlR+jzR8YtG hYcKhbF9xRjZKr0Yq2OuWnw3Rr24eSBVgUuCexWiDm0w1cbKjJzZiilwZNPDfZe8etNL Mbjw== X-Forwarded-Encrypted: i=1; AJvYcCXVgRk1z05vIAfpJBDIXSMy2BAxFqwO7HcN+4oeK80uaj0AkPKeMlzLy8UXl2x8O1DrTQEQ5G+CdabnWdpeU8L0+Qc= X-Gm-Message-State: AOJu0YwwbnGn8R1oQjosfTTgO7Yxd48zmfSYhzVXs4he8HjruWnqDapm +Xv4ZZcqmqgMfN1pNyQiSvsKL4WgcslNkCRsjHONc19mIRgXvPKle7GLXgk5UXc= X-Google-Smtp-Source: AGHT+IE62T9UYgoFVIkcGb7YcoXTufdF8VzWoHsKdhWNi0ZH/AY4EA/1XwSY1gXhI3Nz4aPyj1TOGA== X-Received: by 2002:a05:620a:1981:b0:7a1:dbf6:f762 with SMTP id af79cd13be357-7a1e524c6d3mr1039593585a.20.1722351752925; Tue, 30 Jul 2024 08:02:32 -0700 (PDT) Received: from soleen.c.googlers.com.com (197.5.86.34.bc.googleusercontent.com. [34.86.5.197]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a1d73efffdsm645934285a.69.2024.07.30.08.02.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 30 Jul 2024 08:02:32 -0700 (PDT) From: Pasha Tatashin To: akpm@linux-foundation.org, jpoimboe@kernel.org, pasha.tatashin@soleen.com, kent.overstreet@linux.dev, peterz@infradead.org, nphamcs@gmail.com, cerasuolodomenico@gmail.com, surenb@google.com, lizhijian@fujitsu.com, willy@infradead.org, shakeel.butt@linux.dev, vbabka@suse.cz, ziy@nvidia.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, yosryahmed@google.com Subject: [PATCH v6 2/3] vmstat: Kernel stack usage histogram Date: Tue, 30 Jul 2024 15:01:57 +0000 Message-ID: <20240730150158.832783-3-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.46.0.rc1.232.g9752f9e123-goog In-Reply-To: <20240730150158.832783-1-pasha.tatashin@soleen.com> References: <20240730150158.832783-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: D3E1880036 X-Stat-Signature: fg5sopfdbiey1u4eonwwxt3tsys8sgmn X-Rspamd-Server: rspam09 X-Rspam-User: X-HE-Tag: 1722351753-139749 X-HE-Meta: U2FsdGVkX1/0g9oCJSsqjiKXeALlKubYEI0/2jiQbTYFCK9VOTr53AQm8YOpb+iB7MTumkWmX+lYCNw4ceNAc8T1aOcQGLTaI2y7v98+nLR+uGRibHN0JiK/edQq/mL6kZm5fOlnHd7NTYrK7grzBLRGES/wgk7SyA76t57iw/WisOI8HcniuUqtwDkaqAjmXjSWiPZ6pt7YKFC3zUkvUZwpt8zMt4/ks1JnabQ2+E1LUeAUFR65t1+3f9Gq8YHarUvyOY3IeH/v4g5IuhGEgdigihyadOiDGrYd7pScM6uUi8gemXQyUOm2mjXVeJI8KXK5qm6Mqa436ZigWifDPmIlKvRIP58Z3gRE5+vR1wKmKS5ASIzTNa0Nw7cEWDhWGeY6zvIK2zJ1j41EhX4GBTADP1Gc5x/QfXQPSLhQstBTxO5bJsmzsLS60uf8BQtNRcaguq7BeIF8c1SGmYEN4Kl+K1tmMjl6Xh0TPdpVkUQBBDOQTJJ9OXNutUFQvKR8+EuuDf8Go1ae4f8UZhwSoSCbadIWxtSWUDBNlN7TtaEEwGy61A3ANsd1cBn6e0kSNG+lCwzJRviEovhRX227irBPq7Aw9L2MT6angYb+sB5H3wNW/KkRT/Q3mQdg5snUZ4epjbv26lCyUHx1FvLSEdrcSK119i3SYO9qcfUapgps0hrqTqmKR/pE2mpshkK9KW3GGFiNf+1Js34OkfQVbZL+PK6sUaLxOuyFwwLplSrAV1yrjn+ouW1NSXAFXuXz2IG8B1n+XqNMwz45CniIFZFA3A9FIRijrfwW/5F0roUtHPN/sZKO+dUHCWGtd9/eYV1y/pPtyhTEw20hWZkf6Kzc+bv5TExyXUZ/m+yx7PLbxVOzN89fVSX3RJyvLG2eWhpMRDit05f+W9KTujv9Hu1mA3sP2g0Ka9O+sJs/27H+5q+fcT7Wlir29sP4Prs+kido68SHAjjt9iXdc33 Mj/Nbdb0 G1u6YEfD3hthSGMo6ZFe0vtepb3Gt0xbIqRMPmInng88/P3+03NjTQYwjxrPbtYJ3kX8mfAplR6rGmtS5Okygn7BnK2YM01A0QM7a6rkOltVINMofXX6yOQ35YimuzIiqK06WGxcTjsYz4LdsQuRUolm7TZkTgnbm+cwiZNRSbYWqFysOwyxHKzzL/nRqE00Nn52ss6V9WWYgMyfBHyiG5faqiFQ4pXFY4S1RnrfVxXut+0CYfdIPfnWj/SIzSzKK7Zd8uJ9aT7Dec6EqsrmiuHEeYLe+jZtGsq9y6mkZxebh79INhgA5Q5YSK/rGbJPV+9nsPQtMX5F5/vyqligE/ZKOUA5sDFXFPovc00ugmWkpEPqt67WJcEGRz7Rk8bC0uqqZengK1JRKgzubKjC5H2UlBxU4V3/mFxGjBlDoomeGCek= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: As part of the dynamic kernel stack project, we need to know the amount of data that can be saved by reducing the default kernel stack size [1]. Provide a kernel stack usage histogram to aid in optimizing kernel stack sizes and minimizing memory waste in large-scale environments. The histogram divides stack usage into power-of-two buckets and reports the results in /proc/vmstat. This information is especially valuable in environments with millions of machines, where even small optimizations can have a significant impact. The histogram data is presented in /proc/vmstat with entries like "kstack_1k", "kstack_2k", and so on, indicating the number of threads that exited with stack usage falling within each respective bucket. Example outputs: Intel: $ grep kstack /proc/vmstat kstack_1k 3 kstack_2k 188 kstack_4k 11391 kstack_8k 243 kstack_16k 0 ARM with 64K page_size: $ grep kstack /proc/vmstat kstack_1k 1 kstack_2k 340 kstack_4k 25212 kstack_8k 1659 kstack_16k 0 kstack_32k 0 kstack_64k 0 Note: once the dynamic kernel stack is implemented it will depend on the implementation the usability of this feature: On hardware that supports faults on kernel stacks, we will have other metrics that show the total number of pages allocated for stacks. On hardware where faults are not supported, we will most likely have some optimization where only some threads are extended, and for those, these metrics will still be very useful. [1] https://lwn.net/Articles/974367 Signed-off-by: Pasha Tatashin Reviewed-by: Kent Overstreet Acked-by: Shakeel Butt --- include/linux/vm_event_item.h | 24 ++++++++++++++++++++++ kernel/exit.c | 38 +++++++++++++++++++++++++++++++++++ mm/vmstat.c | 24 ++++++++++++++++++++++ 3 files changed, 86 insertions(+) diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h index 747943bc8cc2..37ad1c16367a 100644 --- a/include/linux/vm_event_item.h +++ b/include/linux/vm_event_item.h @@ -154,6 +154,30 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, VMA_LOCK_RETRY, VMA_LOCK_MISS, #endif +#ifdef CONFIG_DEBUG_STACK_USAGE + KSTACK_1K, +#if THREAD_SIZE > 1024 + KSTACK_2K, +#endif +#if THREAD_SIZE > 2048 + KSTACK_4K, +#endif +#if THREAD_SIZE > 4096 + KSTACK_8K, +#endif +#if THREAD_SIZE > 8192 + KSTACK_16K, +#endif +#if THREAD_SIZE > 16384 + KSTACK_32K, +#endif +#if THREAD_SIZE > 32768 + KSTACK_64K, +#endif +#if THREAD_SIZE > 65536 + KSTACK_REST, +#endif +#endif /* CONFIG_DEBUG_STACK_USAGE */ NR_VM_EVENT_ITEMS }; diff --git a/kernel/exit.c b/kernel/exit.c index 7430852a8571..64bfc2bae55b 100644 --- a/kernel/exit.c +++ b/kernel/exit.c @@ -778,6 +778,43 @@ static void exit_notify(struct task_struct *tsk, int group_dead) } #ifdef CONFIG_DEBUG_STACK_USAGE +/* Count the maximum pages reached in kernel stacks */ +static inline void kstack_histogram(unsigned long used_stack) +{ +#ifdef CONFIG_VM_EVENT_COUNTERS + if (used_stack <= 1024) + count_vm_event(KSTACK_1K); +#if THREAD_SIZE > 1024 + else if (used_stack <= 2048) + count_vm_event(KSTACK_2K); +#endif +#if THREAD_SIZE > 2048 + else if (used_stack <= 4096) + count_vm_event(KSTACK_4K); +#endif +#if THREAD_SIZE > 4096 + else if (used_stack <= 8192) + count_vm_event(KSTACK_8K); +#endif +#if THREAD_SIZE > 8192 + else if (used_stack <= 16384) + count_vm_event(KSTACK_16K); +#endif +#if THREAD_SIZE > 16384 + else if (used_stack <= 32768) + count_vm_event(KSTACK_32K); +#endif +#if THREAD_SIZE > 32768 + else if (used_stack <= 65536) + count_vm_event(KSTACK_64K); +#endif +#if THREAD_SIZE > 65536 + else + count_vm_event(KSTACK_REST); +#endif +#endif /* CONFIG_VM_EVENT_COUNTERS */ +} + static void check_stack_usage(void) { static DEFINE_SPINLOCK(low_water_lock); @@ -785,6 +822,7 @@ static void check_stack_usage(void) unsigned long free; free = stack_not_used(current); + kstack_histogram(THREAD_SIZE - free); if (free >= lowest_to_date) return; diff --git a/mm/vmstat.c b/mm/vmstat.c index 04a1cb6cc636..c7d52a9660c3 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1417,6 +1417,30 @@ const char * const vmstat_text[] = { "vma_lock_retry", "vma_lock_miss", #endif +#ifdef CONFIG_DEBUG_STACK_USAGE + "kstack_1k", +#if THREAD_SIZE > 1024 + "kstack_2k", +#endif +#if THREAD_SIZE > 2048 + "kstack_4k", +#endif +#if THREAD_SIZE > 4096 + "kstack_8k", +#endif +#if THREAD_SIZE > 8192 + "kstack_16k", +#endif +#if THREAD_SIZE > 16384 + "kstack_32k", +#endif +#if THREAD_SIZE > 32768 + "kstack_64k", +#endif +#if THREAD_SIZE > 65536 + "kstack_rest", +#endif +#endif #endif /* CONFIG_VM_EVENT_COUNTERS || CONFIG_MEMCG */ }; #endif /* CONFIG_PROC_FS || CONFIG_SYSFS || CONFIG_NUMA || CONFIG_MEMCG */ From patchwork Tue Jul 30 15:01:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 13747517 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D4A88C3DA49 for ; Tue, 30 Jul 2024 15:02:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 070126B00A1; Tue, 30 Jul 2024 11:02:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EE96B6B00A2; Tue, 30 Jul 2024 11:02:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D14CA6B00A3; Tue, 30 Jul 2024 11:02:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id B170B6B00A1 for ; Tue, 30 Jul 2024 11:02:38 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 2063CC0371 for ; Tue, 30 Jul 2024 15:02:38 +0000 (UTC) X-FDA: 82396735596.27.CD6F7C4 Received: from mail-qk1-f172.google.com (mail-qk1-f172.google.com [209.85.222.172]) by imf24.hostedemail.com (Postfix) with ESMTP id 2B31018002E for ; Tue, 30 Jul 2024 15:02:34 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b=jZz+bIr9; dmarc=pass (policy=none) header.from=soleen.com; spf=pass (imf24.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.222.172 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1722351710; a=rsa-sha256; cv=none; b=CMr4b7mmiVVRB7XBz4yiiTG5uKVG/FtSgR0LA3tVdOIkGpq+zf1in3J4RD/CSbqhRXqSWl vwrssfJFis88uhWn7nhxSCxQtQH01QxySD19dIBjJL0p8gvNVIiVb7RWOiZpkvXVbcjr4+ uWQ0JHm8/54qBkhoX5V/EUlmNlh+LWo= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b=jZz+bIr9; dmarc=pass (policy=none) header.from=soleen.com; spf=pass (imf24.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.222.172 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1722351710; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=2aHnBGDy68xwYXXDjrfSbMfL0AOPRRbHUN6FtVmc8lU=; b=8BWFiAC4cTN3JtrQ3EKrdwtmutyVjgG5YvNXw6GpJBHda3zO904iP2IRW48a/AMjn6g0X/ QJoRap5diBZyqYzBpSU/JrrJXYeiCZHD2vmgKnqVGMPmfYp4i2HaWAvRR2k8hUcrIysRUj wp81POVBns732mVQIFqWfxa0VSVfT8Y= Received: by mail-qk1-f172.google.com with SMTP id af79cd13be357-7a1d7a544d0so326074985a.1 for ; Tue, 30 Jul 2024 08:02:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen-com.20230601.gappssmtp.com; s=20230601; t=1722351754; x=1722956554; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=2aHnBGDy68xwYXXDjrfSbMfL0AOPRRbHUN6FtVmc8lU=; b=jZz+bIr9mzq38RSca2gAyvyA+mgwPVp5OlP05PrLaFnVwS/WBkTdBk8IVQLB6BUn0I jtCQnKjebcWxEyhAW0NKoMuSn0fckhkr7agmueY+VCG8mNroIJwJQF+ecPtJsuBk0Jwc ZtKu8rj+dSunfOt2JnYYx8BOLNYkoXWoQLBeoO4bvq8OyUGMDF9gbSLmFf+XMVIm2bcN qMKUcLgkGVDLxY2ofjLhaLEjBwYHQjaweaLbn3yx/tBnDEC/Cd+x+WSfetBG7OgJNTQR qm5fRFcP0lfu5Hrgl9dX/x4BwLZHokTihK/QxQeUlCW+w/meZ7AOtvwKe3Q697XXkRxB 0idQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722351754; x=1722956554; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=2aHnBGDy68xwYXXDjrfSbMfL0AOPRRbHUN6FtVmc8lU=; b=r2YRTk7BEhyrjhtsKtiswWRbMr6TDXP+Ex6iGJxxfE4VlewPVTHtwJlMpzd5ClKEiN CYuyBUYc6LfVHl8GAuF6uCyL+iO/TESTMw9DFbdUS/ug1WVvup38ogUb4idm5Tik3vaZ MtJguL3WqN1O2LBLC9dTyBrSkrq8g/ZJuc/7cCm/0lL+s8B+OVOFYYUW1BzfuQXXRu0k 18T5SseWEbcn7Q42BpMC9xIC7J6Z46R7Yo8gY/7rUPqVPiYvi6/eq++peUhhzBzw37jh hiDy7nnE8TE4UkGV6YhcCs/VsbqC20OlsXJxxvcl7RTFNOnZR8qZfQENutB/hGOYCpCE mK9g== X-Forwarded-Encrypted: i=1; AJvYcCVTqsk4rE8NwlWNGPvvy4MdYBPGHtCtmb5Mk2ARWEQt+OxBUWrLutvpo1IacUgtlI6/Txqf9v9TOU1eqpALJYtbwRQ= X-Gm-Message-State: AOJu0Yzw33CKpaUi+gdm/urbHyU0oyfiGSPcg399tW8h5p/YkcK7EaL+ jtl/w3xggYdHYSYnoaiEUjEI4jft/+ABne2/25VXWrmiQvM7lP6bZVpfWkfZPXU= X-Google-Smtp-Source: AGHT+IFxLC8+/Z/VDyAUb7tf0WUqsksRLJR+UaYeagUb+QXKem8cCoH3WlCYwUUUbTE/PYYdq//rKQ== X-Received: by 2002:a05:620a:269a:b0:7a1:d08d:e2fa with SMTP id af79cd13be357-7a1e5229e64mr1516298285a.1.1722351754104; Tue, 30 Jul 2024 08:02:34 -0700 (PDT) Received: from soleen.c.googlers.com.com (197.5.86.34.bc.googleusercontent.com. [34.86.5.197]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a1d73efffdsm645934285a.69.2024.07.30.08.02.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 30 Jul 2024 08:02:33 -0700 (PDT) From: Pasha Tatashin To: akpm@linux-foundation.org, jpoimboe@kernel.org, pasha.tatashin@soleen.com, kent.overstreet@linux.dev, peterz@infradead.org, nphamcs@gmail.com, cerasuolodomenico@gmail.com, surenb@google.com, lizhijian@fujitsu.com, willy@infradead.org, shakeel.butt@linux.dev, vbabka@suse.cz, ziy@nvidia.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org, yosryahmed@google.com Subject: [PATCH v6 3/3] task_stack: uninline stack_not_used Date: Tue, 30 Jul 2024 15:01:58 +0000 Message-ID: <20240730150158.832783-4-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.46.0.rc1.232.g9752f9e123-goog In-Reply-To: <20240730150158.832783-1-pasha.tatashin@soleen.com> References: <20240730150158.832783-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 2B31018002E X-Stat-Signature: gp4nxdp113bk5q5keh7of6jdp49ce14c X-Rspam-User: X-HE-Tag: 1722351754-336399 X-HE-Meta: U2FsdGVkX18kwg0T/sjTt3KIKMIw/IW0P94ZHe9p4iC1d488VuQiCb/nabMVbS/lpmqXKAjSRvZA/5v1yYnhhsJusno/4MvGxQtbIRMPcOS2hDma0sM9xGWQ7kUtti11ekoKJR6vFGa3fm3OlBFZcRDsL64ID3JSVxUt7sM5qjKM2iX5Xp+kTalNYesjMbX9Zs53XKXMUVkNt/vz8WamUrPIaNpS/JMwo/HyXycqbNtRV5z74wzWWbAsvhiFxTqR5oulpA28kJCW2GwoQ2NisJBm3+qN9V5vAt+uUAWv5IKZWFstQuPgmkqBtan6fzWDFA+B8TWEulcausXKh1ESJFgI7RDqtUE7Y0IOMBk7o/AQxTWgjlgyqKE9wYKG5MAc7wGoyIyjLOic869C6244QhCNACHHP5AL2T5PEYTlHBw4wyRwGyqRvRQRGRnZffCUg3aJkTEoTGIYRQwoSliSZCjFLBCV0vzeOC7DdkNlRlc2u2iUJXwfa/KVVSb7tgK0gt8ziZY45U5r3wEPxasJE7cC6CmQ4F5lyCnxAgkY4q+vWBDOUosdtC67gYNKPm9EXRgsfdEO+7ZhVRQ8tLBOgM2Py/ihAaiFdOlThlzpGrJ3az7pOS/QV+R5YrKbI5qIOs0Ch9jj3cvgxCc5G07c0QPY9KwG968DhHgDH+Kr4b/o3aP66ovDW7Wi0uXVt07CsbDroJorQYUVuR1sRQZw0LzsEKQ8KX15ZAanSvWjCq3qtBlmH+lzljqYsjGAO2DkIc5/gOJaR0vPaLg1D5BExHarj9COtXgENF98DlgX776B9G2sFJTR+Rxw5qhptpyTbvdXbJiehj8ENzaJsa5XbPCQS1zkQiKe7XqIQu+0ftBEJl5sFVBimm8qFSudyZR5gxYoux03bCPaJ0VwL+vWk1f2TI860i2f3hmaApPVTRpOXTgMYTbCSxQwFnM+t9psNHmKxtnk45qJS/nO2S8 6yo2CM7B R5c/hM+WnjhZ3RkBy6yrmU33qSwP6LTk9i0lFdhazaCrRHbEi4Icu1aG36RofP4h8c/WpjTJrHNxaKHTmRvpaZVkIqGg3Iq9RTIMhOhAwV8NpOezz6Wr668A6BpDfY/mxW2oGP11LqHUjNddCwkQCzSKiEoJzm3YsnPxESj+y5rMcj3LDwvIx8Cy39kENfWXXfjYYQ6nSaQMC7ZKbTI2WgnPJZWEy2yHmVl/B3rlF9/DHh1XQpY8TLHhDT/5loC1HOdLnyWyGjp37fABc7H9zALFOXyZud+NaBENoTJYC6tZJMvK0Qh61W0fAtO//XlfCVw3KwDBP1FJDPQKgjvkfLyC4zFhX1vdruIAXXcn6+TegeGQZFc1a9a6+rpOtueU1YoSjhjf3zBzzEJWMoxcE9Fuq1XI16m+i0DgT X-Bogosity: Ham, tests=bogofilter, spamicity=0.000004, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Given that stack_not_used() is not performance critical function uninline it. Signed-off-by: Pasha Tatashin Acked-by: Shakeel Butt --- include/linux/sched/task_stack.h | 18 +++--------------- kernel/exit.c | 19 +++++++++++++++++++ kernel/sched/core.c | 4 +--- 3 files changed, 23 insertions(+), 18 deletions(-) diff --git a/include/linux/sched/task_stack.h b/include/linux/sched/task_stack.h index ccd72b978e1f..bf10bdb487dd 100644 --- a/include/linux/sched/task_stack.h +++ b/include/linux/sched/task_stack.h @@ -95,23 +95,11 @@ static inline int object_is_on_stack(const void *obj) extern void thread_stack_cache_init(void); #ifdef CONFIG_DEBUG_STACK_USAGE +unsigned long stack_not_used(struct task_struct *p); +#else static inline unsigned long stack_not_used(struct task_struct *p) { - unsigned long *n = end_of_stack(p); - - do { /* Skip over canary */ -# ifdef CONFIG_STACK_GROWSUP - n--; -# else - n++; -# endif - } while (!*n); - -# ifdef CONFIG_STACK_GROWSUP - return (unsigned long)end_of_stack(p) - (unsigned long)n; -# else - return (unsigned long)n - (unsigned long)end_of_stack(p); -# endif + return 0; } #endif extern void set_task_stack_end_magic(struct task_struct *tsk); diff --git a/kernel/exit.c b/kernel/exit.c index 64bfc2bae55b..45085a0e7c16 100644 --- a/kernel/exit.c +++ b/kernel/exit.c @@ -778,6 +778,25 @@ static void exit_notify(struct task_struct *tsk, int group_dead) } #ifdef CONFIG_DEBUG_STACK_USAGE +unsigned long stack_not_used(struct task_struct *p) +{ + unsigned long *n = end_of_stack(p); + + do { /* Skip over canary */ +# ifdef CONFIG_STACK_GROWSUP + n--; +# else + n++; +# endif + } while (!*n); + +# ifdef CONFIG_STACK_GROWSUP + return (unsigned long)end_of_stack(p) - (unsigned long)n; +# else + return (unsigned long)n - (unsigned long)end_of_stack(p); +# endif +} + /* Count the maximum pages reached in kernel stacks */ static inline void kstack_histogram(unsigned long used_stack) { diff --git a/kernel/sched/core.c b/kernel/sched/core.c index a9f655025607..b275f4f27e3c 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -7405,7 +7405,7 @@ EXPORT_SYMBOL(io_schedule); void sched_show_task(struct task_struct *p) { - unsigned long free = 0; + unsigned long free; int ppid; if (!try_get_task_stack(p)) @@ -7415,9 +7415,7 @@ void sched_show_task(struct task_struct *p) if (task_is_running(p)) pr_cont(" running task "); -#ifdef CONFIG_DEBUG_STACK_USAGE free = stack_not_used(p); -#endif ppid = 0; rcu_read_lock(); if (pid_alive(p))