From patchwork Wed Jul 24 20:33:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 13741343 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 60CABC3DA61 for ; Wed, 24 Jul 2024 20:33:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F02C86B008A; Wed, 24 Jul 2024 16:33:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EB1076B008C; Wed, 24 Jul 2024 16:33:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D2A696B0092; Wed, 24 Jul 2024 16:33:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id B341E6B008A for ; Wed, 24 Jul 2024 16:33:29 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 29FFC1C2085 for ; Wed, 24 Jul 2024 20:33:29 +0000 (UTC) X-FDA: 82375796538.12.FEFA6C7 Received: from mail-qk1-f175.google.com (mail-qk1-f175.google.com [209.85.222.175]) by imf02.hostedemail.com (Postfix) with ESMTP id 639F780007 for ; Wed, 24 Jul 2024 20:33:27 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b=PAiu9UvF; spf=pass (imf02.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.222.175 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=pass (policy=none) header.from=soleen.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1721853143; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=za/dqz+eg2q+LXUxzWgwz8U+r7avopRJxf8/6QSZyz0=; b=EAHck3QKMZcOZsv8i5N8+GTfKeB9oCYfk0EVCAAXWhk1IY6EupPWeWCYxSvCfodV1xAPq8 jWV8+u3mdd3OIDyrK4g3C/crdtffLMtqpL3uZipslQCjNstew9/3nPUIOFk1YAvUL/K2Nn Bgk9le1pnVhUCymIUNivl3scijrBzF4= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b=PAiu9UvF; spf=pass (imf02.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.222.175 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=pass (policy=none) header.from=soleen.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1721853143; a=rsa-sha256; cv=none; b=XzdTnnrkDZ18nKOZEQSsb2+Taquy/aRIvDJ9hBaa43Cfttdwerxvm+f72J6bnULW0LAt+Q TEdVzI3qLCsw4xecanubVetaD+oYqQakuQ9LGFWYMNtlxJ7bLNuDPyfXZfo6mvdJXXQcGK 1i9yAt63Yl5Un/RKIVgxe35rC74d9TU= Received: by mail-qk1-f175.google.com with SMTP id af79cd13be357-7a1d3959ad5so28366585a.0 for ; Wed, 24 Jul 2024 13:33:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen-com.20230601.gappssmtp.com; s=20230601; t=1721853206; x=1722458006; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=za/dqz+eg2q+LXUxzWgwz8U+r7avopRJxf8/6QSZyz0=; b=PAiu9UvFzO9C1CPBnvsPma2r4FeUcJqLti9f2xre2goTMXc82xnl27RMPygq9QSuag ta9Tjq45WFQmAvXyPdysnyD0cPjydCft1r8s1ds0acWMQmRTiXBKisuCiiyEFng0Guf0 u/+5giWJd6wSxlprBNCtHx/pbSt3PuE/0E5IzF4/yZyv5cM0gvFXOIe6oRu5X0mSG/p5 PFs/NKdiJsU9udK5v5BNUU11a85bCCRcc7TeWALgLU2m+Z57aBTmOgjER5zXZVDYC3pY zRz1inAy0PIgkq2JlZK5URtQ2swtGbIA2MZvS8i/XGJNYOzNA9Sagijuc8+KXIcAFdQW 9uDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721853206; x=1722458006; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=za/dqz+eg2q+LXUxzWgwz8U+r7avopRJxf8/6QSZyz0=; b=ujJzXwXTdR14K3zHdkyw4Mpd5SPGT64asfQPm2xCxxEdxAlORZ0ieicq+mKEh3H37/ cn33ndk186/0bPLzwj8Ak0DAtkPxsAOtur4UAAn7mWV/BkQfk3FKMtP4LDjHzA/H37ja d+0z1CbX87kYPAicQ9fjxI7pc3eFbQXgpeg8xid2omkPhg9FvkrH6N6LJBDdtDcG42KA Hwd7+TNkoYxfkgSi2kq8JsrZrfZ0bkhsc3Q4ZgJoNgsmvkJfLttwLx7vH83GAU6Hcaaj MefvKs6eJpOMsUWtbfqrs1xXa2OWg2+aOIx7tuqTrQP21qXrUcQ/KEJVhmPkXwY4lryM joAw== X-Forwarded-Encrypted: i=1; AJvYcCXwVeqAZ3obLviEYJM4cxbgSxCtha++9n2UmJiMs0ds60iJ5VG9NDVQd1aidE8mIgP6bUkGrZtu+jeu1IEeebZSMc0= X-Gm-Message-State: AOJu0YxCqwkGjz71N61P7JgnSvsoPrgcNVS1f/Zh4YfDK/2v8BAFulOK 9F0GTGI6NOYoUDslIllt9dCBUuueqnrPjRtqz/0l71NhEoXpcldn4RYp3xme47Y= X-Google-Smtp-Source: AGHT+IExTwTAOD/5fd9WfUJdqA5+PovpxcivQTu3nfLVJy2iPs6JHKTbh5T2ftGVvWPvF45SGasYCg== X-Received: by 2002:a05:620a:199b:b0:79f:7c0:ebfd with SMTP id af79cd13be357-7a1ccd334bfmr551254285a.16.1721853206521; Wed, 24 Jul 2024 13:33:26 -0700 (PDT) Received: from soleen.c.googlers.com.com (197.5.86.34.bc.googleusercontent.com. [34.86.5.197]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a1d73b1786sm466485a.33.2024.07.24.13.33.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 24 Jul 2024 13:33:25 -0700 (PDT) From: Pasha Tatashin To: akpm@linux-foundation.org, jpoimboe@kernel.org, pasha.tatashin@soleen.com, kent.overstreet@linux.dev, peterz@infradead.org, nphamcs@gmail.com, cerasuolodomenico@gmail.com, surenb@google.com, lizhijian@fujitsu.com, willy@infradead.org, shakeel.butt@linux.dev, vbabka@suse.cz, ziy@nvidia.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v5 1/3] memcg: increase the valid index range for memcg stats Date: Wed, 24 Jul 2024 20:33:20 +0000 Message-ID: <20240724203322.2765486-2-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.45.2.1089.g2a221341d9-goog In-Reply-To: <20240724203322.2765486-1-pasha.tatashin@soleen.com> References: <20240724203322.2765486-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 639F780007 X-Stat-Signature: qi7qmpnzq5xkuncqpn7n8zg3btejmwa9 X-Rspam-User: X-HE-Tag: 1721853207-522950 X-HE-Meta: U2FsdGVkX1/+8oemfnHqrKHmyiv5c7xP3wLWh5or3iYk5D5RU7ZQCmx+npeNpimdjQYk11p7W6ChagHLORL9oEJYHiNvorPEPaqh6y9XBn5Ys4/YWvp6xgZc+22IlNB/Ljn+G/n+evwTrungPgWYxruG7gLFXA79mODa42RcmdoVNMUYwPVyDGzv8ahuoYBb6r9CN9u4bVbrPyE2LrVrY58WvadCwylVG1BEiHl7pMA5smMwpM/DSF9TECoAuNHKhh0V2bJBfLXiZYIloIp3e+4MOomVMQatTxRMYuTDd/fA46QyChLZXvHM9K8xamzd+SPSgEg4bBBXRFpPBwWmfsAXXcPvyQt9KroZ8+cnTLMjcZdEu5iSxUSNsioBoYvVxqqoYWtx+yfnprva5kuZbUTGeKwTFiQNi+NXk/6nZ/9c2dRFAWe5t94pu6ebKo+sibvrSoLCUYjV1CQRe8qEywX1Ba4rJVhnUGd4E2jgzN2Zhc3fUh9ZrbDUdEi9CAGS/t4Da+qcjdmGwpdyI5Se5YJchrZG7sIIVO3zuHnxWXlV/a2GVc5pEpIhxMXYOMup4eo7MPnDnHa5eoX876CdZZVBThElCxEA0xJKEhWML6SoMEV7ZB04T4VvVHHVoVENCD7PN6Stb+4yEuMEq4rlCdzMHzLf10wV4wobjq25Om47Ur2nFfacDkeh9oNJ4ddAuJrqAARg1RSwZO2rEnnS8G3ab1hE+pjxwFf5HxBeJv1mivECyGrY87x32wrNo4K0FoykKobQE0guNnH86GlI38HredIXj182P4UrFuiEldPZXBkS5JFmrhYmzgxqLBJALRPpyGsKpbMUw5Qp2+ea/U3A6xdaBCDaA1DJTHdS/484ALw7hkrTg/lbANrblQvJqyZwyUxfWQ5G9seieH1YPoThdzUrWRR0MNGCpSa40/GOhYsqPb+1PfjRbXx1DQ79GzZQtMsZ9QRq99FQkTf cgVra3t2 YrcoctpCD41hKANBqTf6Sl0exERCZW2CLN9HNX3fiaxfL5x+wWDCP7+FZQDR0s0U8GDhqo9eSOz0f8T+u079hxQsCMbSc/PwffkGB/cxwZ3Q6tAMrxoHmSEPkmUPm+gELlIXSmIRbTYaA3bJn5mAPNP3+WxKqRdBhi6LEe2bPMyzUdbtwhaur2BJNiLIi2+HKl36P9elFfCY6RxOF5k9KrsFCe67uFYOoLl3OiF1qL5oRjJsve+iN/0ixPqLALFgnawkp1EvZBKNNDSGQ8TAFRDCPrTB5jM2WCQXIMGi11nYDjN9zO2uSbxTVfqgqpJkF0es2Mw10cLD9I4w6GKD90SGJGBj15mkFkl9gW+NH53sC+CI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000002, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Shakeel Butt At the moment the valid index for the indirection tables for memcg stats and events is < S8_MAX. These indirection tables are used in performance critical codepaths. With the latest addition to the vm_events, the NR_VM_EVENT_ITEMS has gone over S8_MAX. One way to resolve is to increase the entry size of the indirection table from int8_t to int16_t but this will increase the potential number of cachelines needed to access the indirection table. This patch took a different approach and make the valid index < U8_MAX. In this way the size of the indirection tables will remain same and we only need to invalid index check from less than 0 to equal to U8_MAX. In this approach we have also removed a subtraction from the performance critical codepaths. Signed-off-by: Shakeel Butt Co-developed-by: Pasha Tatashin Signed-off-by: Pasha Tatashin --- mm/memcontrol.c | 50 +++++++++++++++++++++++++++---------------------- 1 file changed, 28 insertions(+), 22 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 960371788687..2fdeece7f1f8 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -320,24 +320,27 @@ static const unsigned int memcg_stat_items[] = { #define NR_MEMCG_NODE_STAT_ITEMS ARRAY_SIZE(memcg_node_stat_items) #define MEMCG_VMSTAT_SIZE (NR_MEMCG_NODE_STAT_ITEMS + \ ARRAY_SIZE(memcg_stat_items)) -static int8_t mem_cgroup_stats_index[MEMCG_NR_STAT] __read_mostly; +#define IS_INVALID(index) ((index) == U8_MAX) +static u8 mem_cgroup_stats_index[MEMCG_NR_STAT] __read_mostly; static void init_memcg_stats(void) { - int8_t i, j = 0; + u8 i, j = 0; - BUILD_BUG_ON(MEMCG_NR_STAT >= S8_MAX); + BUILD_BUG_ON(MEMCG_NR_STAT >= U8_MAX); - for (i = 0; i < NR_MEMCG_NODE_STAT_ITEMS; ++i) - mem_cgroup_stats_index[memcg_node_stat_items[i]] = ++j; + memset(mem_cgroup_stats_index, U8_MAX, sizeof(mem_cgroup_stats_index)); - for (i = 0; i < ARRAY_SIZE(memcg_stat_items); ++i) - mem_cgroup_stats_index[memcg_stat_items[i]] = ++j; + for (i = 0; i < NR_MEMCG_NODE_STAT_ITEMS; ++i, ++j) + mem_cgroup_stats_index[memcg_node_stat_items[i]] = j; + + for (i = 0; i < ARRAY_SIZE(memcg_stat_items); ++i, ++j) + mem_cgroup_stats_index[memcg_stat_items[i]] = j; } static inline int memcg_stats_index(int idx) { - return mem_cgroup_stats_index[idx] - 1; + return mem_cgroup_stats_index[idx]; } struct lruvec_stats_percpu { @@ -369,7 +372,7 @@ unsigned long lruvec_page_state(struct lruvec *lruvec, enum node_stat_item idx) return node_page_state(lruvec_pgdat(lruvec), idx); i = memcg_stats_index(idx); - if (WARN_ONCE(i < 0, "%s: missing stat item %d\n", __func__, idx)) + if (WARN_ONCE(IS_INVALID(i), "%s: missing stat item %d\n", __func__, idx)) return 0; pn = container_of(lruvec, struct mem_cgroup_per_node, lruvec); @@ -392,7 +395,7 @@ unsigned long lruvec_page_state_local(struct lruvec *lruvec, return node_page_state(lruvec_pgdat(lruvec), idx); i = memcg_stats_index(idx); - if (WARN_ONCE(i < 0, "%s: missing stat item %d\n", __func__, idx)) + if (WARN_ONCE(IS_INVALID(i), "%s: missing stat item %d\n", __func__, idx)) return 0; pn = container_of(lruvec, struct mem_cgroup_per_node, lruvec); @@ -435,21 +438,24 @@ static const unsigned int memcg_vm_event_stat[] = { }; #define NR_MEMCG_EVENTS ARRAY_SIZE(memcg_vm_event_stat) -static int8_t mem_cgroup_events_index[NR_VM_EVENT_ITEMS] __read_mostly; +static u8 mem_cgroup_events_index[NR_VM_EVENT_ITEMS] __read_mostly; static void init_memcg_events(void) { - int8_t i; + u8 i; + + BUILD_BUG_ON(NR_VM_EVENT_ITEMS >= U8_MAX); - BUILD_BUG_ON(NR_VM_EVENT_ITEMS >= S8_MAX); + memset(mem_cgroup_events_index, U8_MAX, + sizeof(mem_cgroup_events_index)); for (i = 0; i < NR_MEMCG_EVENTS; ++i) - mem_cgroup_events_index[memcg_vm_event_stat[i]] = i + 1; + mem_cgroup_events_index[memcg_vm_event_stat[i]] = i; } static inline int memcg_events_index(enum vm_event_item idx) { - return mem_cgroup_events_index[idx] - 1; + return mem_cgroup_events_index[idx]; } struct memcg_vmstats_percpu { @@ -621,7 +627,7 @@ unsigned long memcg_page_state(struct mem_cgroup *memcg, int idx) long x; int i = memcg_stats_index(idx); - if (WARN_ONCE(i < 0, "%s: missing stat item %d\n", __func__, idx)) + if (WARN_ONCE(IS_INVALID(i), "%s: missing stat item %d\n", __func__, idx)) return 0; x = READ_ONCE(memcg->vmstats->state[i]); @@ -662,7 +668,7 @@ void __mod_memcg_state(struct mem_cgroup *memcg, enum memcg_stat_item idx, if (mem_cgroup_disabled()) return; - if (WARN_ONCE(i < 0, "%s: missing stat item %d\n", __func__, idx)) + if (WARN_ONCE(IS_INVALID(i), "%s: missing stat item %d\n", __func__, idx)) return; __this_cpu_add(memcg->vmstats_percpu->state[i], val); @@ -675,7 +681,7 @@ unsigned long memcg_page_state_local(struct mem_cgroup *memcg, int idx) long x; int i = memcg_stats_index(idx); - if (WARN_ONCE(i < 0, "%s: missing stat item %d\n", __func__, idx)) + if (WARN_ONCE(IS_INVALID(i), "%s: missing stat item %d\n", __func__, idx)) return 0; x = READ_ONCE(memcg->vmstats->state_local[i]); @@ -694,7 +700,7 @@ static void __mod_memcg_lruvec_state(struct lruvec *lruvec, struct mem_cgroup *memcg; int i = memcg_stats_index(idx); - if (WARN_ONCE(i < 0, "%s: missing stat item %d\n", __func__, idx)) + if (WARN_ONCE(IS_INVALID(i), "%s: missing stat item %d\n", __func__, idx)) return; pn = container_of(lruvec, struct mem_cgroup_per_node, lruvec); @@ -810,7 +816,7 @@ void __count_memcg_events(struct mem_cgroup *memcg, enum vm_event_item idx, if (mem_cgroup_disabled()) return; - if (WARN_ONCE(i < 0, "%s: missing stat item %d\n", __func__, idx)) + if (WARN_ONCE(IS_INVALID(i), "%s: missing stat item %d\n", __func__, idx)) return; memcg_stats_lock(); @@ -823,7 +829,7 @@ unsigned long memcg_events(struct mem_cgroup *memcg, int event) { int i = memcg_events_index(event); - if (WARN_ONCE(i < 0, "%s: missing stat item %d\n", __func__, event)) + if (WARN_ONCE(IS_INVALID(i), "%s: missing stat item %d\n", __func__, event)) return 0; return READ_ONCE(memcg->vmstats->events[i]); @@ -833,7 +839,7 @@ unsigned long memcg_events_local(struct mem_cgroup *memcg, int event) { int i = memcg_events_index(event); - if (WARN_ONCE(i < 0, "%s: missing stat item %d\n", __func__, event)) + if (WARN_ONCE(IS_INVALID(i), "%s: missing stat item %d\n", __func__, event)) return 0; return READ_ONCE(memcg->vmstats->events_local[i]); From patchwork Wed Jul 24 20:33:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 13741344 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5B460C3DA64 for ; Wed, 24 Jul 2024 20:33:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5B2016B0092; Wed, 24 Jul 2024 16:33:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5658B6B0093; Wed, 24 Jul 2024 16:33:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 403886B0095; Wed, 24 Jul 2024 16:33:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 193776B0092 for ; Wed, 24 Jul 2024 16:33:31 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id B589380BDB for ; Wed, 24 Jul 2024 20:33:30 +0000 (UTC) X-FDA: 82375796580.28.572F997 Received: from mail-oo1-f43.google.com (mail-oo1-f43.google.com [209.85.161.43]) by imf02.hostedemail.com (Postfix) with ESMTP id DCB2580004 for ; Wed, 24 Jul 2024 20:33:28 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b="py/ODG4G"; spf=pass (imf02.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.161.43 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=pass (policy=none) header.from=soleen.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1721853184; a=rsa-sha256; cv=none; b=jQn9Oi6XuwprgtXxM1FrgSCYPnwXEsdXOlNsb82OoQCiyZ9CPX4tp6nrMQdc0HPhEGMRnE QFHTaFhN4vi2fYUxuND1yeTOWKNhNtye8LeexyQ+xpNT7NJFGqGW+XfRhxs6Th+thZfgl5 TUwd/kFZgiOvnFCPkXNDM/e3p32pKDA= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b="py/ODG4G"; spf=pass (imf02.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.161.43 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=pass (policy=none) header.from=soleen.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1721853184; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=cJxm/byA41k6HHb6hL73ZdxkTNW3paktP68/wTRBIYU=; b=B1ObBPij/dFCPzUKLBIOwmsFEFYKraKvGmo8h+7MRGy+CpldkXXh9aBaLIxyjcwsMcovhK 7+xuEljm6zDThtVK2V8ZEE7V2mQVODihcbkoknLo7fXJ8K2ujr7pglm1oLLIHVmegPu0LG ti4KR5ied00uQ5RRn6ctfV+pasKTA7Q= Received: by mail-oo1-f43.google.com with SMTP id 006d021491bc7-5c6661bca43so177068eaf.0 for ; Wed, 24 Jul 2024 13:33:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen-com.20230601.gappssmtp.com; s=20230601; t=1721853208; x=1722458008; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=cJxm/byA41k6HHb6hL73ZdxkTNW3paktP68/wTRBIYU=; b=py/ODG4GzvOlnKykgEPCS88g/uobljJ0XpX3qSgvt3GnuHnZHOOssACuvTnIbdO7Zq jnXwvKvypVzvGLoRbhV4QxCJfX4R7eKq9BYX5QcE0V/JnIG1MNhFIF+Rm/0512D5wYs1 ctQib35zl/4m0J2nAizmgxOXUoDZ5zaBcmZpAblatffhQKpD7TtxoOIVHnp1Wnf8/XGb F8szC+jTWOG12yjzspf9ItZnFzhfNFGLNybY9J6KYFK2x5J3lwDZNgZ5hYWtP/BzklWw R69QyOHvLfgNnz0JQooR1KvCRp7e3cayW6KZV/Xq6cfODPor7l9i5V3n0gR8uiu6GqQg chOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721853208; x=1722458008; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cJxm/byA41k6HHb6hL73ZdxkTNW3paktP68/wTRBIYU=; b=FeY8DJ1CNzhGeQIeqHD4BVLp9/kvvtaoBaodiIzPA3kuLWRVeOcS7DyXfUz/bSZk1i eF3f+Z0jKLIam6mPLJVdhHouZDLkCLxAG3hJRO5l/qcjXdAFADNp2U8ERM7FHhJSlqQQ S45NPkf/obVl/BFdNNfQ7mCWDlL1g5d8SPmn395OVBbdPun9CI8TwwzAog/c7+Ha5ddG CL9lOKAhnwZBlv6PbBKXVHOeaOCJ8Sw78YD4UFeLbibA8Vj1GWFi1Ym8K0uKniuj+FVD mo5+3fx9QmQAnZZa+1IaCGWni9+p+SQfNAw/7U7zCnQEJ3zNyAUGUfRw1TVuNsa4Jtc0 Qu6A== X-Forwarded-Encrypted: i=1; AJvYcCVQO441oXdiJZL04L5oPMImxwQMEza4jjoy/tQ6EjHehTV+oZHFVkFY4AK4JgGxExixFA6jKKACzzqMU6vvT0Jm5Nc= X-Gm-Message-State: AOJu0YyFMW/FTR+MzdDAkeTgtGeH4w6Ag3aTRG4YKAJnymnhQ/K4NOSy 4hSZHNjzakxj/KI1+n7Q0t08ajtHAawqI6cotY+F1g0Gzu88ISYi0q0PvhaCO5Q= X-Google-Smtp-Source: AGHT+IHE52f2bOrGfZsPUD1tpSaklQrRe5ujqaQavwqiUXjGH9LGmhslRRkcpkvz99rconqiOHZ7Tg== X-Received: by 2002:a05:6358:2484:b0:1aa:bde7:5725 with SMTP id e5c5f4694b2df-1acf8ae554fmr128120855d.28.1721853207710; Wed, 24 Jul 2024 13:33:27 -0700 (PDT) Received: from soleen.c.googlers.com.com (197.5.86.34.bc.googleusercontent.com. [34.86.5.197]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a1d73b1786sm466485a.33.2024.07.24.13.33.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 24 Jul 2024 13:33:26 -0700 (PDT) From: Pasha Tatashin To: akpm@linux-foundation.org, jpoimboe@kernel.org, pasha.tatashin@soleen.com, kent.overstreet@linux.dev, peterz@infradead.org, nphamcs@gmail.com, cerasuolodomenico@gmail.com, surenb@google.com, lizhijian@fujitsu.com, willy@infradead.org, shakeel.butt@linux.dev, vbabka@suse.cz, ziy@nvidia.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v5 2/3] vmstat: Kernel stack usage histogram Date: Wed, 24 Jul 2024 20:33:21 +0000 Message-ID: <20240724203322.2765486-3-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.45.2.1089.g2a221341d9-goog In-Reply-To: <20240724203322.2765486-1-pasha.tatashin@soleen.com> References: <20240724203322.2765486-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Stat-Signature: frirrud8foi8r8te8yumpsaa5of9fha1 X-Rspamd-Queue-Id: DCB2580004 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1721853208-408497 X-HE-Meta: U2FsdGVkX1+TQA9pXMWib4NI+VBSrp1QS+iKuyLUHX9P6twrbx7cVeTJiRYWXkt2LXsLi9t2j9l1p3DavhulJ9bZe15oT5uxwNsGT37mvJoLwQF73PZbBWgGViPmCX1X123NGG7ng1J48vIursVoVDGm5XJADGywaPu6SiiPWtCOeerBrbVv4AtJUyQCDUewsIF/xLJ50Bz8AJ5LEewAhkIU4X9rDXymzIoBXhK+jlVUJpwgzFNUSyZ3jpX/nd2LDFYlWp15LmbTR0SUpKgZYPSNv+gVUioFH+n4hyQwIyPIEG9MG9SrlYqkV0ZJZCLkRHe6VF8KTEV5rodKivqq08EKKzsz2tz/CYV3MTlejstoV4ftS1xvt1HAcRiebU7gs8ruWd7YG1vIxXkx1EiN5T+TBpP0ddCgVZyliTntyHjjXkZg15aSl/VTJQAeIQgN8gx5oHvwBSdBlbEtlQzvVYkaFDpLfbT+ScVzbRvLIFjEMZHKIj2s1POqBwHw4K+hIK/sGDIv4Dn8l3EHKfdCOJxeXkmoVY1aqBqQFKINBcLkheLJw7TdQ66KhtsDaVc1Drad34Kcz1sMbTyNjIAmEMCfMmyqLwAalxFzOYSuwMdPnqvl9ZbcyU8jCnHqgqtwPtYQbGi0PGFS/trqDYGkloMEHyvjnJ7RqyHuAMbs2hQadDUokEAJ/rzask1BR1LHFv5ru/H9BfKWRl4KBvEr4XU6CpKNm3NjdNiG1M+hBUJ1Ja/diRGue0PUOwG7iRUgnkrZ2qLa3QmW9tDcgbLxeSypq9iXLCp0yOt6WO6MWLIs/8bEIzRxxHpdtMui+P3ATEyKoi8bLwZzSJojVcHeWJZsqfIurVy0h505ShyoonpgK5SOXoDHGULnTUVoZRL+9EHX9xgDH3b44awDXAGlX/cO6FCmEV304alFmUS0OvJI9h/Uik8cnKH1PTWkvqkVL7waj041Ftyj/fG4YWH UsqLEPEa MEaKzVACNw72orTlG5uk53wspLcXunnuB82cOTbIo/RlDn2ROr3TAKw6wY9zQ49j3088CqyJaZPxoWXf1FFSUgdTkDJq0kIVYPi5HdvtsdhohUcxAfiNxpVZFxoTgHl71cHHpEfTb07BQxi76FL5qomN8mIk4hxUiIZpMnAFM7aciEeKVsWWKnbEnI7+BgOhTWFiKqO050X52kUTakX4leU6W4EayiQ/T+hmzcoNtNW9+HwBlpBmUbA4zcREXrQAMTzGY6M2teyPn33unB85jQU6yJZ+iNATFUzzC7tqR0gf3kAkoWzo48FkcirdB8+eBrgA5hFiUShzpxyom3tKsgHPsFH42YeUbXDSlCRBgKeeQb8uao51EErb/iBGYqUB9QigaGm3Zzp2Vlte4FKuLlfLsZdHL9WhBlcIvel9WBkEuxJM= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: As part of the dynamic kernel stack project, we need to know the amount of data that can be saved by reducing the default kernel stack size [1]. Provide a kernel stack usage histogram to aid in optimizing kernel stack sizes and minimizing memory waste in large-scale environments. The histogram divides stack usage into power-of-two buckets and reports the results in /proc/vmstat. This information is especially valuable in environments with millions of machines, where even small optimizations can have a significant impact. The histogram data is presented in /proc/vmstat with entries like "kstack_1k", "kstack_2k", and so on, indicating the number of threads that exited with stack usage falling within each respective bucket. Example outputs: Intel: $ grep kstack /proc/vmstat kstack_1k 3 kstack_2k 188 kstack_4k 11391 kstack_8k 243 kstack_16k 0 ARM with 64K page_size: $ grep kstack /proc/vmstat kstack_1k 1 kstack_2k 340 kstack_4k 25212 kstack_8k 1659 kstack_16k 0 kstack_32k 0 kstack_64k 0 Note: once the dynamic kernel stack is implemented it will depend on the implementation the usability of this feature: On hardware that supports faults on kernel stacks, we will have other metrics that show the total number of pages allocated for stacks. On hardware where faults are not supported, we will most likely have some optimization where only some threads are extended, and for those, these metrics will still be very useful. [1] https://lwn.net/Articles/974367 Signed-off-by: Pasha Tatashin Reviewed-by: Kent Overstreet Acked-by: Shakeel Butt --- include/linux/vm_event_item.h | 24 ++++++++++++++++++++++ kernel/exit.c | 38 +++++++++++++++++++++++++++++++++++ mm/vmstat.c | 24 ++++++++++++++++++++++ 3 files changed, 86 insertions(+) diff --git a/include/linux/vm_event_item.h b/include/linux/vm_event_item.h index 747943bc8cc2..37ad1c16367a 100644 --- a/include/linux/vm_event_item.h +++ b/include/linux/vm_event_item.h @@ -154,6 +154,30 @@ enum vm_event_item { PGPGIN, PGPGOUT, PSWPIN, PSWPOUT, VMA_LOCK_RETRY, VMA_LOCK_MISS, #endif +#ifdef CONFIG_DEBUG_STACK_USAGE + KSTACK_1K, +#if THREAD_SIZE > 1024 + KSTACK_2K, +#endif +#if THREAD_SIZE > 2048 + KSTACK_4K, +#endif +#if THREAD_SIZE > 4096 + KSTACK_8K, +#endif +#if THREAD_SIZE > 8192 + KSTACK_16K, +#endif +#if THREAD_SIZE > 16384 + KSTACK_32K, +#endif +#if THREAD_SIZE > 32768 + KSTACK_64K, +#endif +#if THREAD_SIZE > 65536 + KSTACK_REST, +#endif +#endif /* CONFIG_DEBUG_STACK_USAGE */ NR_VM_EVENT_ITEMS }; diff --git a/kernel/exit.c b/kernel/exit.c index 7430852a8571..64bfc2bae55b 100644 --- a/kernel/exit.c +++ b/kernel/exit.c @@ -778,6 +778,43 @@ static void exit_notify(struct task_struct *tsk, int group_dead) } #ifdef CONFIG_DEBUG_STACK_USAGE +/* Count the maximum pages reached in kernel stacks */ +static inline void kstack_histogram(unsigned long used_stack) +{ +#ifdef CONFIG_VM_EVENT_COUNTERS + if (used_stack <= 1024) + count_vm_event(KSTACK_1K); +#if THREAD_SIZE > 1024 + else if (used_stack <= 2048) + count_vm_event(KSTACK_2K); +#endif +#if THREAD_SIZE > 2048 + else if (used_stack <= 4096) + count_vm_event(KSTACK_4K); +#endif +#if THREAD_SIZE > 4096 + else if (used_stack <= 8192) + count_vm_event(KSTACK_8K); +#endif +#if THREAD_SIZE > 8192 + else if (used_stack <= 16384) + count_vm_event(KSTACK_16K); +#endif +#if THREAD_SIZE > 16384 + else if (used_stack <= 32768) + count_vm_event(KSTACK_32K); +#endif +#if THREAD_SIZE > 32768 + else if (used_stack <= 65536) + count_vm_event(KSTACK_64K); +#endif +#if THREAD_SIZE > 65536 + else + count_vm_event(KSTACK_REST); +#endif +#endif /* CONFIG_VM_EVENT_COUNTERS */ +} + static void check_stack_usage(void) { static DEFINE_SPINLOCK(low_water_lock); @@ -785,6 +822,7 @@ static void check_stack_usage(void) unsigned long free; free = stack_not_used(current); + kstack_histogram(THREAD_SIZE - free); if (free >= lowest_to_date) return; diff --git a/mm/vmstat.c b/mm/vmstat.c index 73d791d1caad..6e3347789eb2 100644 --- a/mm/vmstat.c +++ b/mm/vmstat.c @@ -1417,6 +1417,30 @@ const char * const vmstat_text[] = { "vma_lock_retry", "vma_lock_miss", #endif +#ifdef CONFIG_DEBUG_STACK_USAGE + "kstack_1k", +#if THREAD_SIZE > 1024 + "kstack_2k", +#endif +#if THREAD_SIZE > 2048 + "kstack_4k", +#endif +#if THREAD_SIZE > 4096 + "kstack_8k", +#endif +#if THREAD_SIZE > 8192 + "kstack_16k", +#endif +#if THREAD_SIZE > 16384 + "kstack_32k", +#endif +#if THREAD_SIZE > 32768 + "kstack_64k", +#endif +#if THREAD_SIZE > 65536 + "kstack_rest", +#endif +#endif #endif /* CONFIG_VM_EVENT_COUNTERS || CONFIG_MEMCG */ }; #endif /* CONFIG_PROC_FS || CONFIG_SYSFS || CONFIG_NUMA || CONFIG_MEMCG */ From patchwork Wed Jul 24 20:33:22 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 13741345 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 17FABC3DA61 for ; Wed, 24 Jul 2024 20:33:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BA4C86B0093; Wed, 24 Jul 2024 16:33:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B31976B0095; Wed, 24 Jul 2024 16:33:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9A6DF6B0096; Wed, 24 Jul 2024 16:33:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 7DA236B0093 for ; Wed, 24 Jul 2024 16:33:31 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 3A204C0C16 for ; Wed, 24 Jul 2024 20:33:31 +0000 (UTC) X-FDA: 82375796622.23.3A77AF8 Received: from mail-qk1-f169.google.com (mail-qk1-f169.google.com [209.85.222.169]) by imf02.hostedemail.com (Postfix) with ESMTP id 671718001D for ; Wed, 24 Jul 2024 20:33:29 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b=zAgAwmWS; dmarc=pass (policy=none) header.from=soleen.com; spf=pass (imf02.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.222.169 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1721853170; a=rsa-sha256; cv=none; b=Pf2DHON8vwARGVMy26qbCjttJ+n3MD9mxYwCyHq4SeK67rH/dDf/ryG3IszeYVSjRsN6bo p16GigTJCyUcOwcAjFaJBeTypWid74OFVv6pQIA/lYxMyf0yrfDGKVdM2KGz7Zggyes3kl YfpNjTjcWPqTGHy26VXZPJlXqtHtrDk= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b=zAgAwmWS; dmarc=pass (policy=none) header.from=soleen.com; spf=pass (imf02.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.222.169 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1721853170; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=EKGazB9ClhJRcAnoyFkGgUjUokS0gyXTYpAGog1A94A=; b=NSF4cVRws3h10SQptRac4JdjVUG2jyDbVblfi2/8waxsJtBM5dAVEWnvZmHB/K8Lk9JOq0 Y3zrteIv6rOPQuX4uJVbn3bQBwu/wYEMZNSQ/NTe2knt8kpuJus1puhojtmg7XaDsYIVAs xjZdH+KgUXCgs5xtPGSzME/kYo0I3Tc= Received: by mail-qk1-f169.google.com with SMTP id af79cd13be357-79f323f084eso9238085a.3 for ; Wed, 24 Jul 2024 13:33:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen-com.20230601.gappssmtp.com; s=20230601; t=1721853208; x=1722458008; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=EKGazB9ClhJRcAnoyFkGgUjUokS0gyXTYpAGog1A94A=; b=zAgAwmWSU1kM7X3hssM0ieJKN1tCK1XYXpevcWGX3tX+jNL9lLrw40vgqC9UW1MuDa a6R+ZA/bIAVavyQATIM1KFz2IPS0G8MiSog1IwwTzAhBwzvvd2c0tZW9f1jStRrwPdjQ v4WP7jljTbdGpDGzronPVxPfkFyG4bCVv2MJXBMXcpL9xyCHFZw5iTDi5nCsUOSl2aRb T8KVwipZkGYWhvEIVfSsg3Y4YODCgenfCNP2IKk+rMp2jj7P4YtAEaZKmGPwI1jcuLOC CHHCeqgqBesIUIEOArC8PTxG0Hd8+oLUvedIkl1aZNx3Li2mCNOX3km8UtdeEoidfVBi gmmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721853208; x=1722458008; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=EKGazB9ClhJRcAnoyFkGgUjUokS0gyXTYpAGog1A94A=; b=sv4E/aDQJLNDYNq/AgIvIeDDaKB7m/P+ZuD8uk4NHzTMaXI6CVhWbCjQSYgobxjywo 9ho/kPqhdcsy0erNJtl90YnYu85CxhB98LiEL/2s46/vuh8pMwXMAe+pPC2j60X+BjMC WBrr4Lk4sNNGYXuv+cbAn8GNbV5AcNiQGtX3zbJat41xTbyP4I3Q1roBuGFnF96rNHYL 77aDpsOscxE4b8WACaEwqplAZvlbi7DxZNMwHuKupua797qGf5GVqcFGGeKDszCvVCJw 3WmMUMFe4/xQZ3UXHuebxLDGRpUaG02bPQmUrIQAP4b+FJ40pxU9JTBHWVnuBw06mzTl RGXw== X-Forwarded-Encrypted: i=1; AJvYcCV6J/QQzSj/T9+ZHRB98t9jmCZTwOwRnI+liJWhq19FYBkZ/DavyzpMgXI8thS7kfAvq6RxPi5ani9VbYi9KavaiK8= X-Gm-Message-State: AOJu0YwZ2UDaX7qVSzUxgnQfAAWqjXtipog1mFyIP9P3+wW7Waaq7NwE +I1hFHYs8UlI4ERNVBCGcYOO2gMKYNjienvKk5HPxTfu4LywIwQCdOKX+bhvV+Y= X-Google-Smtp-Source: AGHT+IGdbA77q0Ryd5vfCl/zff1MduhOE0tjAnp/J35IRonbXQFsuFvYQZxmL8WXYNZns3Wr9gC6Dg== X-Received: by 2002:a05:620a:40d3:b0:79f:17a8:b83 with SMTP id af79cd13be357-7a1d45850b0mr104907085a.33.1721853208537; Wed, 24 Jul 2024 13:33:28 -0700 (PDT) Received: from soleen.c.googlers.com.com (197.5.86.34.bc.googleusercontent.com. [34.86.5.197]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a1d73b1786sm466485a.33.2024.07.24.13.33.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 24 Jul 2024 13:33:28 -0700 (PDT) From: Pasha Tatashin To: akpm@linux-foundation.org, jpoimboe@kernel.org, pasha.tatashin@soleen.com, kent.overstreet@linux.dev, peterz@infradead.org, nphamcs@gmail.com, cerasuolodomenico@gmail.com, surenb@google.com, lizhijian@fujitsu.com, willy@infradead.org, shakeel.butt@linux.dev, vbabka@suse.cz, ziy@nvidia.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v5 3/3] task_stack: uninline stack_not_used Date: Wed, 24 Jul 2024 20:33:22 +0000 Message-ID: <20240724203322.2765486-4-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.45.2.1089.g2a221341d9-goog In-Reply-To: <20240724203322.2765486-1-pasha.tatashin@soleen.com> References: <20240724203322.2765486-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 671718001D X-Stat-Signature: 5q54eeumoqjyii7g9mggiy33wrhzdhpo X-Rspam-User: X-HE-Tag: 1721853209-380999 X-HE-Meta: U2FsdGVkX180fx5lccIb6R5DkEPdivlug/erNvpH5O2my8G2Hp2FndYC+A4g980Ag79tKSAKKBUlq+cpiZQRPHa+dBvrj7Whj0TcIN5xjGb03vaHvODJrd6ZSVpLM0OppCrG0RTLcjLko40cYfFme47LE79m9cJRXD91KvE3mSvRjWQZKRNJaTzUCRECO3+6A9X3KACZ4iVIKth+MDXJTwD3kKoaasN+qOHVE3RHon97m8cW5eB9dsBqUNRC6Y9sBV1yzw3/ZJHPtFGujjO6qL5hKIkPT75ad4s4Pj/AbOA3jYHfIN/rfz+Iim+hSQ4Ik31BNNehRVZVmS7+WV7jl/T4+0bmUw88eJ9qlVmt99o2f5Ivvq7SKtvMPUqpTTtNpRf32pQPwicZcoO0Fx+I0oQgsUFtqKDqNgHkq2chf3BoeGIfeNN9abgzMenIi3D2KQVJ9MQF7NwGg5JWJpNUyUCfkih5dUuqpVewn87ta5xeuiN7tm+EzqBXv6rPvbhq0IaVUFE6vBdc8u/W0TNXpCXXvCGCb7tU+xvOzW3FbSuSenn/ZNtN2Q+VGHue/at3JYn6qFPKzIdN2gAhg8DhmHcwa8lA04jphv0paJ1u9IkscfRYfyFYyYAWtsfST0v6xQYNMbJ8rQqp9qUo5YSnhlCewSyiwuS72uI/AKFCUVjv7fs4gdC4ClovU+eVRERcWpiDiUADBw14RmeoGXKBWXBWFyknAxPybJTcB8lbC3tMzKzHM3ZDCoogw2ex5HXJdNO9AregxUGql7/losjwq8F8AW9Hy7OTRGmIqLWxKW2+GISPx2C5KYF4Ug5O9WZP88TIfy/ui0yx6KOYH1xQAALX0evRWLeuZgncTgrgUZem3sO6WKCQWQPAlRDVYXnrQA51PwgwRGNvINcB3Y4Ju/JP3lrI1en9gkivykFShMlqqeHel1LllQVMM7KTPBeQ45oStAfuoipcpPVzvab B7wXzIHX 6mM0ObvwhXiigbg4NJPep2ZNa2Hxc6VQ//3W3dkCtiUZilDTz+GhT4Aax7t12hgrHSZBKor/4OaeymHnEFTsTQoa5ORfLXkFT8Nx/CQyHtJ5JHzYZilEea0w5WDHYL9lCiho6FQY03/XI1L7HxrUNzTDqcAuNr83Xk+M58051ZcTYZQWS/PKnRktcZ15ELS7RZu0l/I0bYVpnSKYvgNAIblurkpbYxrJXzDSA5uDhkzkOh200D0y621CXPKreYQeE5Ta68sEyrcWLhuM6T831afuOr557l28Fp40jfIoGLmKrfBQwX8Pd5iorfJOm3D2CqrVgPcT3apd7ydKvxQkQobt+65JdWPLKwe02K1ccPLubp6wZq0vwnFxcnKv+E/bZ2RHtSvf/T6L2hb1h1OuylQSVTKnn0bj0s4Ih X-Bogosity: Ham, tests=bogofilter, spamicity=0.000022, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Given that stack_not_used() is not performance critical function uninline it. Signed-off-by: Pasha Tatashin Acked-by: Shakeel Butt --- include/linux/sched/task_stack.h | 18 +++--------------- kernel/exit.c | 19 +++++++++++++++++++ kernel/sched/core.c | 4 +--- 3 files changed, 23 insertions(+), 18 deletions(-) diff --git a/include/linux/sched/task_stack.h b/include/linux/sched/task_stack.h index ccd72b978e1f..bf10bdb487dd 100644 --- a/include/linux/sched/task_stack.h +++ b/include/linux/sched/task_stack.h @@ -95,23 +95,11 @@ static inline int object_is_on_stack(const void *obj) extern void thread_stack_cache_init(void); #ifdef CONFIG_DEBUG_STACK_USAGE +unsigned long stack_not_used(struct task_struct *p); +#else static inline unsigned long stack_not_used(struct task_struct *p) { - unsigned long *n = end_of_stack(p); - - do { /* Skip over canary */ -# ifdef CONFIG_STACK_GROWSUP - n--; -# else - n++; -# endif - } while (!*n); - -# ifdef CONFIG_STACK_GROWSUP - return (unsigned long)end_of_stack(p) - (unsigned long)n; -# else - return (unsigned long)n - (unsigned long)end_of_stack(p); -# endif + return 0; } #endif extern void set_task_stack_end_magic(struct task_struct *tsk); diff --git a/kernel/exit.c b/kernel/exit.c index 64bfc2bae55b..45085a0e7c16 100644 --- a/kernel/exit.c +++ b/kernel/exit.c @@ -778,6 +778,25 @@ static void exit_notify(struct task_struct *tsk, int group_dead) } #ifdef CONFIG_DEBUG_STACK_USAGE +unsigned long stack_not_used(struct task_struct *p) +{ + unsigned long *n = end_of_stack(p); + + do { /* Skip over canary */ +# ifdef CONFIG_STACK_GROWSUP + n--; +# else + n++; +# endif + } while (!*n); + +# ifdef CONFIG_STACK_GROWSUP + return (unsigned long)end_of_stack(p) - (unsigned long)n; +# else + return (unsigned long)n - (unsigned long)end_of_stack(p); +# endif +} + /* Count the maximum pages reached in kernel stacks */ static inline void kstack_histogram(unsigned long used_stack) { diff --git a/kernel/sched/core.c b/kernel/sched/core.c index ae5ef3013a55..f5861f64e960 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -7405,7 +7405,7 @@ EXPORT_SYMBOL(io_schedule); void sched_show_task(struct task_struct *p) { - unsigned long free = 0; + unsigned long free; int ppid; if (!try_get_task_stack(p)) @@ -7415,9 +7415,7 @@ void sched_show_task(struct task_struct *p) if (task_is_running(p)) pr_cont(" running task "); -#ifdef CONFIG_DEBUG_STACK_USAGE free = stack_not_used(p); -#endif ppid = 0; rcu_read_lock(); if (pid_alive(p))