From patchwork Wed Jul 24 20:33:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 13741343 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 60CABC3DA61 for ; Wed, 24 Jul 2024 20:33:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F02C86B008A; Wed, 24 Jul 2024 16:33:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EB1076B008C; Wed, 24 Jul 2024 16:33:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D2A696B0092; Wed, 24 Jul 2024 16:33:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id B341E6B008A for ; Wed, 24 Jul 2024 16:33:29 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 29FFC1C2085 for ; Wed, 24 Jul 2024 20:33:29 +0000 (UTC) X-FDA: 82375796538.12.FEFA6C7 Received: from mail-qk1-f175.google.com (mail-qk1-f175.google.com [209.85.222.175]) by imf02.hostedemail.com (Postfix) with ESMTP id 639F780007 for ; Wed, 24 Jul 2024 20:33:27 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b=PAiu9UvF; spf=pass (imf02.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.222.175 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=pass (policy=none) header.from=soleen.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1721853143; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=za/dqz+eg2q+LXUxzWgwz8U+r7avopRJxf8/6QSZyz0=; b=EAHck3QKMZcOZsv8i5N8+GTfKeB9oCYfk0EVCAAXWhk1IY6EupPWeWCYxSvCfodV1xAPq8 jWV8+u3mdd3OIDyrK4g3C/crdtffLMtqpL3uZipslQCjNstew9/3nPUIOFk1YAvUL/K2Nn Bgk9le1pnVhUCymIUNivl3scijrBzF4= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b=PAiu9UvF; spf=pass (imf02.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.222.175 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=pass (policy=none) header.from=soleen.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1721853143; a=rsa-sha256; cv=none; b=XzdTnnrkDZ18nKOZEQSsb2+Taquy/aRIvDJ9hBaa43Cfttdwerxvm+f72J6bnULW0LAt+Q TEdVzI3qLCsw4xecanubVetaD+oYqQakuQ9LGFWYMNtlxJ7bLNuDPyfXZfo6mvdJXXQcGK 1i9yAt63Yl5Un/RKIVgxe35rC74d9TU= Received: by mail-qk1-f175.google.com with SMTP id af79cd13be357-7a1d3959ad5so28366585a.0 for ; Wed, 24 Jul 2024 13:33:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen-com.20230601.gappssmtp.com; s=20230601; t=1721853206; x=1722458006; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=za/dqz+eg2q+LXUxzWgwz8U+r7avopRJxf8/6QSZyz0=; b=PAiu9UvFzO9C1CPBnvsPma2r4FeUcJqLti9f2xre2goTMXc82xnl27RMPygq9QSuag ta9Tjq45WFQmAvXyPdysnyD0cPjydCft1r8s1ds0acWMQmRTiXBKisuCiiyEFng0Guf0 u/+5giWJd6wSxlprBNCtHx/pbSt3PuE/0E5IzF4/yZyv5cM0gvFXOIe6oRu5X0mSG/p5 PFs/NKdiJsU9udK5v5BNUU11a85bCCRcc7TeWALgLU2m+Z57aBTmOgjER5zXZVDYC3pY zRz1inAy0PIgkq2JlZK5URtQ2swtGbIA2MZvS8i/XGJNYOzNA9Sagijuc8+KXIcAFdQW 9uDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721853206; x=1722458006; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=za/dqz+eg2q+LXUxzWgwz8U+r7avopRJxf8/6QSZyz0=; b=ujJzXwXTdR14K3zHdkyw4Mpd5SPGT64asfQPm2xCxxEdxAlORZ0ieicq+mKEh3H37/ cn33ndk186/0bPLzwj8Ak0DAtkPxsAOtur4UAAn7mWV/BkQfk3FKMtP4LDjHzA/H37ja d+0z1CbX87kYPAicQ9fjxI7pc3eFbQXgpeg8xid2omkPhg9FvkrH6N6LJBDdtDcG42KA Hwd7+TNkoYxfkgSi2kq8JsrZrfZ0bkhsc3Q4ZgJoNgsmvkJfLttwLx7vH83GAU6Hcaaj MefvKs6eJpOMsUWtbfqrs1xXa2OWg2+aOIx7tuqTrQP21qXrUcQ/KEJVhmPkXwY4lryM joAw== X-Forwarded-Encrypted: i=1; AJvYcCXwVeqAZ3obLviEYJM4cxbgSxCtha++9n2UmJiMs0ds60iJ5VG9NDVQd1aidE8mIgP6bUkGrZtu+jeu1IEeebZSMc0= X-Gm-Message-State: AOJu0YxCqwkGjz71N61P7JgnSvsoPrgcNVS1f/Zh4YfDK/2v8BAFulOK 9F0GTGI6NOYoUDslIllt9dCBUuueqnrPjRtqz/0l71NhEoXpcldn4RYp3xme47Y= X-Google-Smtp-Source: AGHT+IExTwTAOD/5fd9WfUJdqA5+PovpxcivQTu3nfLVJy2iPs6JHKTbh5T2ftGVvWPvF45SGasYCg== X-Received: by 2002:a05:620a:199b:b0:79f:7c0:ebfd with SMTP id af79cd13be357-7a1ccd334bfmr551254285a.16.1721853206521; Wed, 24 Jul 2024 13:33:26 -0700 (PDT) Received: from soleen.c.googlers.com.com (197.5.86.34.bc.googleusercontent.com. [34.86.5.197]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a1d73b1786sm466485a.33.2024.07.24.13.33.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 24 Jul 2024 13:33:25 -0700 (PDT) From: Pasha Tatashin To: akpm@linux-foundation.org, jpoimboe@kernel.org, pasha.tatashin@soleen.com, kent.overstreet@linux.dev, peterz@infradead.org, nphamcs@gmail.com, cerasuolodomenico@gmail.com, surenb@google.com, lizhijian@fujitsu.com, willy@infradead.org, shakeel.butt@linux.dev, vbabka@suse.cz, ziy@nvidia.com, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v5 1/3] memcg: increase the valid index range for memcg stats Date: Wed, 24 Jul 2024 20:33:20 +0000 Message-ID: <20240724203322.2765486-2-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.45.2.1089.g2a221341d9-goog In-Reply-To: <20240724203322.2765486-1-pasha.tatashin@soleen.com> References: <20240724203322.2765486-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 639F780007 X-Stat-Signature: qi7qmpnzq5xkuncqpn7n8zg3btejmwa9 X-Rspam-User: X-HE-Tag: 1721853207-522950 X-HE-Meta: U2FsdGVkX1/+8oemfnHqrKHmyiv5c7xP3wLWh5or3iYk5D5RU7ZQCmx+npeNpimdjQYk11p7W6ChagHLORL9oEJYHiNvorPEPaqh6y9XBn5Ys4/YWvp6xgZc+22IlNB/Ljn+G/n+evwTrungPgWYxruG7gLFXA79mODa42RcmdoVNMUYwPVyDGzv8ahuoYBb6r9CN9u4bVbrPyE2LrVrY58WvadCwylVG1BEiHl7pMA5smMwpM/DSF9TECoAuNHKhh0V2bJBfLXiZYIloIp3e+4MOomVMQatTxRMYuTDd/fA46QyChLZXvHM9K8xamzd+SPSgEg4bBBXRFpPBwWmfsAXXcPvyQt9KroZ8+cnTLMjcZdEu5iSxUSNsioBoYvVxqqoYWtx+yfnprva5kuZbUTGeKwTFiQNi+NXk/6nZ/9c2dRFAWe5t94pu6ebKo+sibvrSoLCUYjV1CQRe8qEywX1Ba4rJVhnUGd4E2jgzN2Zhc3fUh9ZrbDUdEi9CAGS/t4Da+qcjdmGwpdyI5Se5YJchrZG7sIIVO3zuHnxWXlV/a2GVc5pEpIhxMXYOMup4eo7MPnDnHa5eoX876CdZZVBThElCxEA0xJKEhWML6SoMEV7ZB04T4VvVHHVoVENCD7PN6Stb+4yEuMEq4rlCdzMHzLf10wV4wobjq25Om47Ur2nFfacDkeh9oNJ4ddAuJrqAARg1RSwZO2rEnnS8G3ab1hE+pjxwFf5HxBeJv1mivECyGrY87x32wrNo4K0FoykKobQE0guNnH86GlI38HredIXj182P4UrFuiEldPZXBkS5JFmrhYmzgxqLBJALRPpyGsKpbMUw5Qp2+ea/U3A6xdaBCDaA1DJTHdS/484ALw7hkrTg/lbANrblQvJqyZwyUxfWQ5G9seieH1YPoThdzUrWRR0MNGCpSa40/GOhYsqPb+1PfjRbXx1DQ79GzZQtMsZ9QRq99FQkTf cgVra3t2 YrcoctpCD41hKANBqTf6Sl0exERCZW2CLN9HNX3fiaxfL5x+wWDCP7+FZQDR0s0U8GDhqo9eSOz0f8T+u079hxQsCMbSc/PwffkGB/cxwZ3Q6tAMrxoHmSEPkmUPm+gELlIXSmIRbTYaA3bJn5mAPNP3+WxKqRdBhi6LEe2bPMyzUdbtwhaur2BJNiLIi2+HKl36P9elFfCY6RxOF5k9KrsFCe67uFYOoLl3OiF1qL5oRjJsve+iN/0ixPqLALFgnawkp1EvZBKNNDSGQ8TAFRDCPrTB5jM2WCQXIMGi11nYDjN9zO2uSbxTVfqgqpJkF0es2Mw10cLD9I4w6GKD90SGJGBj15mkFkl9gW+NH53sC+CI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000002, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Shakeel Butt At the moment the valid index for the indirection tables for memcg stats and events is < S8_MAX. These indirection tables are used in performance critical codepaths. With the latest addition to the vm_events, the NR_VM_EVENT_ITEMS has gone over S8_MAX. One way to resolve is to increase the entry size of the indirection table from int8_t to int16_t but this will increase the potential number of cachelines needed to access the indirection table. This patch took a different approach and make the valid index < U8_MAX. In this way the size of the indirection tables will remain same and we only need to invalid index check from less than 0 to equal to U8_MAX. In this approach we have also removed a subtraction from the performance critical codepaths. Signed-off-by: Shakeel Butt Co-developed-by: Pasha Tatashin Signed-off-by: Pasha Tatashin --- mm/memcontrol.c | 50 +++++++++++++++++++++++++++---------------------- 1 file changed, 28 insertions(+), 22 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 960371788687..2fdeece7f1f8 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -320,24 +320,27 @@ static const unsigned int memcg_stat_items[] = { #define NR_MEMCG_NODE_STAT_ITEMS ARRAY_SIZE(memcg_node_stat_items) #define MEMCG_VMSTAT_SIZE (NR_MEMCG_NODE_STAT_ITEMS + \ ARRAY_SIZE(memcg_stat_items)) -static int8_t mem_cgroup_stats_index[MEMCG_NR_STAT] __read_mostly; +#define IS_INVALID(index) ((index) == U8_MAX) +static u8 mem_cgroup_stats_index[MEMCG_NR_STAT] __read_mostly; static void init_memcg_stats(void) { - int8_t i, j = 0; + u8 i, j = 0; - BUILD_BUG_ON(MEMCG_NR_STAT >= S8_MAX); + BUILD_BUG_ON(MEMCG_NR_STAT >= U8_MAX); - for (i = 0; i < NR_MEMCG_NODE_STAT_ITEMS; ++i) - mem_cgroup_stats_index[memcg_node_stat_items[i]] = ++j; + memset(mem_cgroup_stats_index, U8_MAX, sizeof(mem_cgroup_stats_index)); - for (i = 0; i < ARRAY_SIZE(memcg_stat_items); ++i) - mem_cgroup_stats_index[memcg_stat_items[i]] = ++j; + for (i = 0; i < NR_MEMCG_NODE_STAT_ITEMS; ++i, ++j) + mem_cgroup_stats_index[memcg_node_stat_items[i]] = j; + + for (i = 0; i < ARRAY_SIZE(memcg_stat_items); ++i, ++j) + mem_cgroup_stats_index[memcg_stat_items[i]] = j; } static inline int memcg_stats_index(int idx) { - return mem_cgroup_stats_index[idx] - 1; + return mem_cgroup_stats_index[idx]; } struct lruvec_stats_percpu { @@ -369,7 +372,7 @@ unsigned long lruvec_page_state(struct lruvec *lruvec, enum node_stat_item idx) return node_page_state(lruvec_pgdat(lruvec), idx); i = memcg_stats_index(idx); - if (WARN_ONCE(i < 0, "%s: missing stat item %d\n", __func__, idx)) + if (WARN_ONCE(IS_INVALID(i), "%s: missing stat item %d\n", __func__, idx)) return 0; pn = container_of(lruvec, struct mem_cgroup_per_node, lruvec); @@ -392,7 +395,7 @@ unsigned long lruvec_page_state_local(struct lruvec *lruvec, return node_page_state(lruvec_pgdat(lruvec), idx); i = memcg_stats_index(idx); - if (WARN_ONCE(i < 0, "%s: missing stat item %d\n", __func__, idx)) + if (WARN_ONCE(IS_INVALID(i), "%s: missing stat item %d\n", __func__, idx)) return 0; pn = container_of(lruvec, struct mem_cgroup_per_node, lruvec); @@ -435,21 +438,24 @@ static const unsigned int memcg_vm_event_stat[] = { }; #define NR_MEMCG_EVENTS ARRAY_SIZE(memcg_vm_event_stat) -static int8_t mem_cgroup_events_index[NR_VM_EVENT_ITEMS] __read_mostly; +static u8 mem_cgroup_events_index[NR_VM_EVENT_ITEMS] __read_mostly; static void init_memcg_events(void) { - int8_t i; + u8 i; + + BUILD_BUG_ON(NR_VM_EVENT_ITEMS >= U8_MAX); - BUILD_BUG_ON(NR_VM_EVENT_ITEMS >= S8_MAX); + memset(mem_cgroup_events_index, U8_MAX, + sizeof(mem_cgroup_events_index)); for (i = 0; i < NR_MEMCG_EVENTS; ++i) - mem_cgroup_events_index[memcg_vm_event_stat[i]] = i + 1; + mem_cgroup_events_index[memcg_vm_event_stat[i]] = i; } static inline int memcg_events_index(enum vm_event_item idx) { - return mem_cgroup_events_index[idx] - 1; + return mem_cgroup_events_index[idx]; } struct memcg_vmstats_percpu { @@ -621,7 +627,7 @@ unsigned long memcg_page_state(struct mem_cgroup *memcg, int idx) long x; int i = memcg_stats_index(idx); - if (WARN_ONCE(i < 0, "%s: missing stat item %d\n", __func__, idx)) + if (WARN_ONCE(IS_INVALID(i), "%s: missing stat item %d\n", __func__, idx)) return 0; x = READ_ONCE(memcg->vmstats->state[i]); @@ -662,7 +668,7 @@ void __mod_memcg_state(struct mem_cgroup *memcg, enum memcg_stat_item idx, if (mem_cgroup_disabled()) return; - if (WARN_ONCE(i < 0, "%s: missing stat item %d\n", __func__, idx)) + if (WARN_ONCE(IS_INVALID(i), "%s: missing stat item %d\n", __func__, idx)) return; __this_cpu_add(memcg->vmstats_percpu->state[i], val); @@ -675,7 +681,7 @@ unsigned long memcg_page_state_local(struct mem_cgroup *memcg, int idx) long x; int i = memcg_stats_index(idx); - if (WARN_ONCE(i < 0, "%s: missing stat item %d\n", __func__, idx)) + if (WARN_ONCE(IS_INVALID(i), "%s: missing stat item %d\n", __func__, idx)) return 0; x = READ_ONCE(memcg->vmstats->state_local[i]); @@ -694,7 +700,7 @@ static void __mod_memcg_lruvec_state(struct lruvec *lruvec, struct mem_cgroup *memcg; int i = memcg_stats_index(idx); - if (WARN_ONCE(i < 0, "%s: missing stat item %d\n", __func__, idx)) + if (WARN_ONCE(IS_INVALID(i), "%s: missing stat item %d\n", __func__, idx)) return; pn = container_of(lruvec, struct mem_cgroup_per_node, lruvec); @@ -810,7 +816,7 @@ void __count_memcg_events(struct mem_cgroup *memcg, enum vm_event_item idx, if (mem_cgroup_disabled()) return; - if (WARN_ONCE(i < 0, "%s: missing stat item %d\n", __func__, idx)) + if (WARN_ONCE(IS_INVALID(i), "%s: missing stat item %d\n", __func__, idx)) return; memcg_stats_lock(); @@ -823,7 +829,7 @@ unsigned long memcg_events(struct mem_cgroup *memcg, int event) { int i = memcg_events_index(event); - if (WARN_ONCE(i < 0, "%s: missing stat item %d\n", __func__, event)) + if (WARN_ONCE(IS_INVALID(i), "%s: missing stat item %d\n", __func__, event)) return 0; return READ_ONCE(memcg->vmstats->events[i]); @@ -833,7 +839,7 @@ unsigned long memcg_events_local(struct mem_cgroup *memcg, int event) { int i = memcg_events_index(event); - if (WARN_ONCE(i < 0, "%s: missing stat item %d\n", __func__, event)) + if (WARN_ONCE(IS_INVALID(i), "%s: missing stat item %d\n", __func__, event)) return 0; return READ_ONCE(memcg->vmstats->events_local[i]);