From patchwork Sat Feb 3 04:46:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13543863 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 93034C4828F for ; Sat, 3 Feb 2024 04:46:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E9D1B6B0078; Fri, 2 Feb 2024 23:46:17 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DF3646B007E; Fri, 2 Feb 2024 23:46:17 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CC78E6B0078; Fri, 2 Feb 2024 23:46:17 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id BCF426B0078 for ; Fri, 2 Feb 2024 23:46:17 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 91EFC1A0437 for ; Sat, 3 Feb 2024 04:46:17 +0000 (UTC) X-FDA: 81749255994.02.6CDA76F Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf25.hostedemail.com (Postfix) with ESMTP id 00589A0003 for ; Sat, 3 Feb 2024 04:46:15 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=2EW8u7AV; spf=pass (imf25.hostedemail.com: domain of 3F8W9ZQoKCMwG6A9Gsz4wvy66y3w.u64305CF-442Dsu2.69y@flex--yosryahmed.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3F8W9ZQoKCMwG6A9Gsz4wvy66y3w.u64305CF-442Dsu2.69y@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706935576; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=bIjjFEohc7BizHW5snB3BX6UTrsWYnDTvUNfCUqP8fo=; b=AS6NmAwFOQm985XYQNbU/nrki1wH7jCMB2IgyQImiE7OZHycwuVL1s/ryl9qLLmDwGhPlc zZqNYjswtx/nYrqo8GLKQt+KIPU5BUe5ugUibl1yXgo1ey+mD4xUK9YSaVBDERC8p2nv5y TTCRU2ZnOC03GXrkyUN1h/DTv4VDW5w= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=2EW8u7AV; spf=pass (imf25.hostedemail.com: domain of 3F8W9ZQoKCMwG6A9Gsz4wvy66y3w.u64305CF-442Dsu2.69y@flex--yosryahmed.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3F8W9ZQoKCMwG6A9Gsz4wvy66y3w.u64305CF-442Dsu2.69y@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706935576; a=rsa-sha256; cv=none; b=gBqiBjXEnH13/hoG6zLwSmktetFuCCXWo3JTDCYhDdSrAGZDSiHv0/RxmD+8PJToKWIpX+ s+KPqtULzE8R9Kv17t3Jdpbk2Oj+GpEDIqyFe9h9DsA4JNHI+ulCVxXzbvXdBr2OwWbVh9 IlORD1/fAzOyWwthAunzYBs51rwPl0Y= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dc64b659a9cso4633468276.3 for ; Fri, 02 Feb 2024 20:46:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706935575; x=1707540375; darn=kvack.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=bIjjFEohc7BizHW5snB3BX6UTrsWYnDTvUNfCUqP8fo=; b=2EW8u7AVr9sZpzc+6jDDkR1PJSY0t9zBlZZMCNruHfiIInSkPksArUEcs8r3S5XRnU enz198qjelkVWr8AvYoZRkzwvMWyu6WnSqpS475gfmavuTVLPl9ZaGkN+UBJ8nO2Wn7z jk2CW53Ycmniim3UrBysJbPU9N548zFuhQbyrhN2Qoz5SqovIMrdo1NxoTids+FoPz0T BjLvGB1TVV6UMsp2WMONmD9mpoVLebrXqlF0u1mzX+eRkUxqARK2670YinluSqtMrdF/ qzfLXrEGx+NDngJp/oZqS4md1GPPvx43TJ1vBGGKtzfsFLGmfLoZPeG4L4gM/pahSb7+ c6qg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706935575; x=1707540375; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=bIjjFEohc7BizHW5snB3BX6UTrsWYnDTvUNfCUqP8fo=; b=QJBTiELQU3qa/ZwDU83HHNSX4Qxoa0oplrPvtyck/ugrHdMZ7aCWC99E4Cbw4Scp9v fwDl01L7+tUwm3MeNed2cW24mbHj21ixo73t+nMhE6UMwLekbyXDp+Po/VUysbQ2aOh0 m/O9a4Z0VFvrQUoocBCdZIP6NEshBQbMFlzyvXX0lic8cQ+xGGV9iblssp1/Y6czqNp7 eXQpRkib2xdrPKrRqgbaV56iTAWq46cNazIZiSCyqqWnSynYyYyIgQUbK9zzMrxkOf+w ODst1PTps61L91/d65qYSXyBbqegGE7NmBF2s/QsLTr9C/kzFhqlFOEQx9fDOZDocd8L VKNQ== X-Gm-Message-State: AOJu0Yz9cEpSVm+euCRcKHja4qBo/p4f9HHRTApdIHSfqQGOYdrygy7Y yZ/fBndB+Lvb355pF9f2btVnB9/p/1l8PlgOVnfpRyJzEdj3Z//Ujcli7xsvoTnQmLFPFJ1Tuhr wxsXMCBP1kDRem9ywng== X-Google-Smtp-Source: AGHT+IGAenCSeE/ByPCnMPf0P8sNA6h6ocBEKdDYaJFQ9TcJgAm0rONYTdeZI9tq91q/gs/t0jXDXM0Zkaj7OSD5 X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:29b4]) (user=yosryahmed job=sendgmr) by 2002:a05:6902:1748:b0:dc6:f21f:64ac with SMTP id bz8-20020a056902174800b00dc6f21f64acmr761350ybb.12.1706935575018; Fri, 02 Feb 2024 20:46:15 -0800 (PST) Date: Sat, 3 Feb 2024 04:46:12 +0000 Mime-Version: 1.0 X-Mailer: git-send-email 2.43.0.594.gd9cf4e227d-goog Message-ID: <20240203044612.1234216-1-yosryahmed@google.com> Subject: [PATCH mm-hotfixes-unstable v2] mm: memcg: fix struct memcg_vmstats_percpu size and alignment From: Yosry Ahmed To: Andrew Morton Cc: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yosry Ahmed , Greg Thelen X-Rspamd-Queue-Id: 00589A0003 X-Rspam-User: X-Stat-Signature: yqwwoakfm96swhkc13pacpuyp48m48mu X-Rspamd-Server: rspam01 X-HE-Tag: 1706935575-239755 X-HE-Meta: U2FsdGVkX190gu9MOq2NhDwQ706NWpqEXRUvMScAhWKQLlJn6dMXcqmWkwtSXQeiRknOVUHosUdOViHFsrL7AbnTfP/lWlpggUmP4JbcunYxchUxuImpG4ciYX9rpJFkmFHWbbS0N2hqBW91/r+y2SIyAokF0oSPZDC0uekt00hwyBPmha6wQbxs+k0L3z3G/RITLuvP/+ECgyjnZE+I/L6lrTTzPQ6NiBM6jXGpEZPMoZYmX8cW8WSnwh3HPR2BnGe7NqfV2BV0xhZmJEXhT8HqhDCc+2G6PV0yP3i6UZA6fZZ2Uf352G3+1bXnxd0pTeue63iNdCTJULYeC+zYP2HqqZ4VWT66eaGRRxOa8PINq5QETP8MKbsnC1D2NmCv4gWVrrj41BicdnTwxGJRAxkBNJ6RyGJIPehMf36tWx+RuSXuZQt2amCWD3OGzw/5HoAV3ApHmT01HeY0e9TlnS6w0sFNeE7u/hfnNXDjEtJ2D1sE/O+0QS164RvsMj4knyYGmTNV7R/Y2Zo+q7bbcxiNHvRKijg2gzA9s3g5N2IcR8NsxA1kl5aRTHyOkJ575EYXW8w8mRiD546gWMEPnkx1tKK3cWSBSEcRur2nkrGSxORdoUrO0xqqcGqdAa1YldgvmeLfs0bC+t5pNv2Lxjl+pfbDG18DF6Y09Q3ski6Ax3diiGAaxP1ROXMyfPmDFUzRiBxOJ6uozhDAcWWqpfjX3h5fHCJYqz7Np7IYIKviSgZkluYRIawn7ezu32NpKDzAUTrRClrf4rHiH8LRURVtroWN9gebPnUJaU5mHpCCoTSXVS7AiGZmyQj7f/5lJParfip+VPQ2PkJi9zYUfHz5Fr30QfckByu4SajR4ewUio9iWCDjUvpDjTOGSRlWAOy3zM7iXtd0/+vlDm8jomRjd8Ua4JsUugZZ9ifC3coSqDej1MR74d0qGanS4m9G7xJvRGcBlhmipRIdgrh Q5i6MaIi hmK3x9mIwEV76Wp1GLH8qQ8lNBlJH+PP2ZY2qkSdOlZDeYtr/eblO/gAKZlpMcdpE/DGj3iH5+PKawITnYqlw2j77qIrW68jVsIZzfTrRo6FRRphGutOmGc4Uq6Rjly19qWGUO21G7+EAH3Z5bGBHNCOPvvgXEmeJpew6PrQ+kFp3E5pL5Ark9TCqmRExJyQtP9wDSE/rx3YnF2BC/0pGw6yfmDdELnGERKv0em7tpw40bp9H2RNC2qWpdjaDJXtq5tk5NhlgzHe5dAgswtwtxhxpV7tqwTnf+keL4UbZO+pdTtXJDH9+O7uw09A1V8QCmZURBELcF6zJqoP8aP7y3+675oqLNSL2Gc/xptbIWqhLhO/WLYAKq5ui53/jnbNG64DgDgI3caTRHsZFM7eLQ7+U5YWmookhzhg97raaTVQR2Ub0ktwVetMfOe8UuLlhE0smggBztR+tLCdfcAkNM86GoEyfMFxty1QFAuP+0+9G4G53GIwW0LB3hdIrzmt8wh0n5rS0iarXTrepQSlNbAlclic+qoUZPjP7+uQXiHQn+deO127DdKQDP8WdDsvLeRNUxux3eUFhJDByEC23XCWWu7ag5t2JHnikquDSLV/+vVw2z3KAtA9sRxT1LDSYP6AaogYI8gsytPhRwWRKck3nbfp3OZ3kl9nEVujrXj0Jo1JcxpNYPBisL2eDrFZtp8z+DgxKUftI9Qi6DLl/0mRcSOxHfrPvuUlKNyNu4gMdouvNS7I9xHlOug== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Commit da10d7e140196 ("mm: memcg: optimize parent iteration in memcg_rstat_updated()") added two additional pointers to the end of struct memcg_vmstats_percpu with CACHELINE_PADDING to put them in a separate cacheline. This caused the struct size to increase from 1200 to 1280 on my config (80 extra bytes instead of 16). Upon revisiting, the relevant struct members do not need to be on a separate cacheline, they just need to fit in a single one. This is a percpu struct, so there shouldn't be any contention on that cacheline anyway. Move the members to the beginning of the struct and make sure the struct itself is cacheline aligned. Add a comment about the members that need to fit together in a cacheline. The struct size is now 1216 on my config with this change. Fixes: da10d7e14019 ("mm: memcg: optimize parent iteration in memcg_rstat_updated()") Reported-by: Greg Thelen Signed-off-by: Yosry Ahmed --- v1 -> v2: - Moved ____cacheline_aligned to the end of the struct definition as recommended by Shakeel. v1: https://lore.kernel.org/lkml/20240203003414.1067730-1-yosryahmed@google.com/ --- mm/memcontrol.c | 21 ++++++++++----------- 1 file changed, 10 insertions(+), 11 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index d9ca0fdbe4ab0..1ed40f9d3a277 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -621,6 +621,15 @@ static inline int memcg_events_index(enum vm_event_item idx) } struct memcg_vmstats_percpu { + /* Stats updates since the last flush */ + unsigned int stats_updates; + + /* Cached pointers for fast iteration in memcg_rstat_updated() */ + struct memcg_vmstats_percpu *parent; + struct memcg_vmstats *vmstats; + + /* The above should fit a single cacheline for memcg_rstat_updated() */ + /* Local (CPU and cgroup) page state & events */ long state[MEMCG_NR_STAT]; unsigned long events[NR_MEMCG_EVENTS]; @@ -632,17 +641,7 @@ struct memcg_vmstats_percpu { /* Cgroup1: threshold notifications & softlimit tree updates */ unsigned long nr_page_events; unsigned long targets[MEM_CGROUP_NTARGETS]; - - /* Fit members below in a single cacheline for memcg_rstat_updated() */ - CACHELINE_PADDING(_pad1_); - - /* Stats updates since the last flush */ - unsigned int stats_updates; - - /* Cached pointers for fast iteration in memcg_rstat_updated() */ - struct memcg_vmstats_percpu *parent; - struct memcg_vmstats *vmstats; -}; +} ____cacheline_aligned; struct memcg_vmstats { /* Aggregated (CPU and subtree) page state & events */