From patchwork Wed Jan 25 07:34:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leonardo Bras X-Patchwork-Id: 13115046 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 21DF3C27C76 for ; Wed, 25 Jan 2023 07:35:55 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9B9AC6B0075; Wed, 25 Jan 2023 02:35:54 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9695A6B0078; Wed, 25 Jan 2023 02:35:54 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 80C236B007B; Wed, 25 Jan 2023 02:35:54 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 713FA6B0075 for ; Wed, 25 Jan 2023 02:35:54 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 4DD45C04BD for ; Wed, 25 Jan 2023 07:35:54 +0000 (UTC) X-FDA: 80392512228.12.B733B25 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf25.hostedemail.com (Postfix) with ESMTP id 4F9D9A0004 for ; Wed, 25 Jan 2023 07:35:52 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=iRRbMiVl; spf=pass (imf25.hostedemail.com: domain of leobras@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=leobras@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674632152; a=rsa-sha256; cv=none; b=t0ce6o3wpPwrT+mwz1LHDSDxXuPf6Av3e6x6fzUnKGSUXJyB++WAS46GGOjvB35I02+rg9 JzNY6h2XLmNAjfiaT/LjgIUWdBd0Ziu2vpleXV2LLyuwd9SHKEEy5OxBht2l62bma7xRxN LUFtqAy5rf9KBU2fZbsphFTM0ECzqkc= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=iRRbMiVl; spf=pass (imf25.hostedemail.com: domain of leobras@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=leobras@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674632152; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=KVG76aWPU3FY34u+v28shR/7Qg472WeM1BX+arjDbFU=; b=N+uYeyJX8vKfdzHthYXT2ZLfufQcU5iLXc4+64XL/NcUwZCKEHrCrsjFyO7bhIFeSBz+Tr jvDBW2GevCXx/ememkV2m6KFx5vW/GfliZjoX3ADrJlMj+7/9Ovl9HYWPUJZR3hBUe1o0h j2qcfpZ90XzbDpK1rM318M8MQ6bL8B8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1674632151; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KVG76aWPU3FY34u+v28shR/7Qg472WeM1BX+arjDbFU=; b=iRRbMiVl6RZKBLIn0RBTwIA/asmhzguVrV+5U1CCgd8BBrwPbW6IZyw7vuS4W9zwW/wlPP hqXHhPbNv+kjqgb1Hwf6WK1nYrg5f6yrfkJRv8A3D4H5hjFmOou2Vz0M8FSR0tODs2O3Un /uXvyzFG7hd4/7hk3Ke9NHq1Dt9fdR8= Received: from mail-oi1-f200.google.com (mail-oi1-f200.google.com [209.85.167.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-614-qtfqlQ_FP2ip4pllzwjdIw-1; Wed, 25 Jan 2023 02:35:50 -0500 X-MC-Unique: qtfqlQ_FP2ip4pllzwjdIw-1 Received: by mail-oi1-f200.google.com with SMTP id z8-20020a056808064800b0036aae651f2aso5586512oih.17 for ; Tue, 24 Jan 2023 23:35:50 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KVG76aWPU3FY34u+v28shR/7Qg472WeM1BX+arjDbFU=; b=4saPrwpmbF4J/lk9c+QYSbmd3WIkdE9cgjdK29H39L0uIjolge603J8FtOOoFVezzB Y6WXt5KXgw1aN4Zc7mzSmOSzcCZvu8KzexlPmUFow1Vt08nVQWvKASzvqk9efPuhk1ll 3a6V3OxJ7pwabRyIMHfwKlDjToXRLZ9LlRXvhWhL9gGoSEtU7MjWnp1KnUHfsLOi/fcj 0EYzlSJbXY4BErj996b+mkm9yRg8du/YboNUIuGLvjAiysqyWf00WucpSD3ESPt6G4+U JJTXYVYHucXi7CqAnU45m4jFt/KPCHaHQdjMGTAKWJxb91d1uswsahkaQLMcO27bRBJ7 3lqQ== X-Gm-Message-State: AFqh2koKK14o5t0mB7s9NSnb/cb6DBFS1bZB3/SdAOlkbSzIAXukdieN cN6iUnMMO9kxrRY7PS5fA+SIW3ZNVjQDPQd5WoSZer61sZ/ZksWWnkaKO+qaT6GslvxHeltXZWP 5MwCIKBLc8/4= X-Received: by 2002:a05:6870:8dc5:b0:15f:74b2:9168 with SMTP id lq5-20020a0568708dc500b0015f74b29168mr14415646oab.4.1674632149720; Tue, 24 Jan 2023 23:35:49 -0800 (PST) X-Google-Smtp-Source: AMrXdXva0ztnWMLaJL80saQFKdei152TK3so9LzytflfDAcnAzjhuVwqsk6VSRdFSyzYlYX+q5Awzg== X-Received: by 2002:a05:6870:8dc5:b0:15f:74b2:9168 with SMTP id lq5-20020a0568708dc500b0015f74b29168mr14415637oab.4.1674632149469; Tue, 24 Jan 2023 23:35:49 -0800 (PST) Received: from LeoBras.redhat.com ([2804:1b3:a800:14fa:9361:c141:6c70:c877]) by smtp.gmail.com with ESMTPSA id x189-20020a4a41c6000000b0050dc79bb80esm1538802ooa.27.2023.01.24.23.35.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Jan 2023 23:35:48 -0800 (PST) From: Leonardo Bras To: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , Marcelo Tosatti Cc: Leonardo Bras , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 1/5] mm/memcontrol: Align percpu memcg_stock to cache Date: Wed, 25 Jan 2023 04:34:58 -0300 Message-Id: <20230125073502.743446-2-leobras@redhat.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230125073502.743446-1-leobras@redhat.com> References: <20230125073502.743446-1-leobras@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspam-User: X-Rspamd-Queue-Id: 4F9D9A0004 X-Rspamd-Server: rspam01 X-Stat-Signature: 8mfa9asj8jiitjtxrw6cg3su3gnmmo8p X-HE-Tag: 1674632152-251831 X-HE-Meta: U2FsdGVkX18eF4NnfspS/suUjE8BtvQVikdLW8LeYtlGwQJaMz/QQG+axD70LOCg0iXq5AxCmISSSx1sEQO8x96q1KsaRfousnWrAQTK52CDjF/Ny2d4ymC5w4vVHOLRFPyyWA0MiTl0VExpgyuvBMoNlVRucCUEP2a8lMV9AN0Zr/znn/zg+NbvYIUcf5NfTyfhIdfP/5/yr8S7GaEC9VX7NCa9u8HLy9R9KvtzMD11fXgyfxDibAe3RsKmtxzwIapzBLeP3vHYuv0pte3P9rtpCahVMiv8+zs/cPd7RrtAMU+WoRlOpdl6VP2Zr/gBTCX2QD1iTiLdYlHlWDolfNvBbn3VR3yAmi8LLDxQj6lUvzysfayHAVpqi4jP7jFahVae46zLhCdKMy5bJHUsgu9WTjFsEnPel9PKi4K+anXUArkUirkP/SRwQ4g1P4F+HquOGnCjHq3SFhBt9GzZRNy6JcTpS6b4O0XZHNhGCLAOSQGmu/RthEUR82x3da5kZa9AFQVaxBHNfI+35Lf6H+qAUtXVcfA1jvMFmIadlCVgwX+sk4peYedv1XA/o4+pDXUMLEgf7Mn/1UQOi6bMesvw7Se9rKgaASUSJ+JjoEnXMPkSvmDKxXdAogaRf3sQl/YJfUCgwLXzT8r9P+J3tobSNB99s9slpKsPJlAn8nADUBBB/qrD/UTN/Hec7MEomarqDIIJYgTQWxbsH2TZXF+QyDQuBOUgtDXxh5rw+WHKUn4FOvioS65BfhMP1ITeB4i/2aMRZsnljtDlb8wQNhBgxs3XwXuRdn095voM+BB1VpYc0UO2u6iC9MEgwQR/k2Uroghdskcm9SbNxT/H7z6lsytvXYFXgtbTeCBwcYloGrUFa6TU98ATxS5LVLp8muPY6OO+q9Ml2B+d/TxPjXX7jOyFQdzYIvFTGeCEkWkrfCPZ1aotr2bj4DJvY6I7lI/t++sw6vVsK98dQGe X3K28boM 3FkP6jq18W5uZUEXuydKWtatGn7lGWP0amY3TUHH1k8Lm8KlVRag4EBs30H4LF8zfwbnsNhv4cbmY/eJ027sJAEb6V0ZSFN6+21bhlBWAk0pdn3ngpSTJ4XcmEbGb1CVfFzsnESfrVdvs95KTcNOd7bzKB3Bzh03pQjG3jn/nG8QxG7TQFmUr83GTdbkEfVAOovVRFw13hodvFDAH1ujk5uUSC/qXjYAZEvBzX+haGCutxBsVO9JjgeYIxZ8jGpAw4EZ3vgvK4bD44/4utmNt18RvbVsio/ajn5mN/UpQmhg7BIkMpk5bWnKJ40XPl7VmyIA11jb4XOJOi10wAp9F4Jd8zDdN/qJPRpoPX3nrmS4eSYS3q/6tUkr7SinDSUusujWdkhrrDQU/8SrFRTllVfsNvnXX6WeL6jLhxr5ePHrYE7egSjZJqiFGLw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When a struct smaller than a cacheline has an instance that is not aligned to cache block size, it could happen that data can be spread across two cachelines. For memcg_stock this could mean the need to fetch and get cache-exclusivity in 2 cachelines instead of 1 when we bounce the cacheline between local cpu functions and drain_all_stock(), which does remote read/write. This could also mean some false-sharing costs being paid due to the cacheline being shared between 2 unrelated structures. Avoid this issue by getting memcg_stock cacheline-aligned. Signed-off-by: Leonardo Bras --- mm/memcontrol.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index ab457f0394ab6..f8e86b88b3c7a 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2188,7 +2188,8 @@ struct memcg_stock_pcp { unsigned long flags; #define FLUSHING_CACHED_CHARGE 0 }; -static DEFINE_PER_CPU(struct memcg_stock_pcp, memcg_stock) = { + +static DEFINE_PER_CPU_SHARED_ALIGNED(struct memcg_stock_pcp, memcg_stock) = { .stock_lock = INIT_LOCAL_LOCK(stock_lock), }; static DEFINE_MUTEX(percpu_charge_mutex); From patchwork Wed Jan 25 07:34:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leonardo Bras X-Patchwork-Id: 13115047 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7700CC54E94 for ; Wed, 25 Jan 2023 07:36:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CEEAC6B0078; Wed, 25 Jan 2023 02:35:59 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C9E166B007B; Wed, 25 Jan 2023 02:35:59 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B3E4A6B007D; Wed, 25 Jan 2023 02:35:59 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id A19206B0078 for ; Wed, 25 Jan 2023 02:35:59 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 82B761A0935 for ; Wed, 25 Jan 2023 07:35:59 +0000 (UTC) X-FDA: 80392512438.19.5B38E3D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf13.hostedemail.com (Postfix) with ESMTP id 80AEF20017 for ; Wed, 25 Jan 2023 07:35:57 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=WbnmkwW1; spf=pass (imf13.hostedemail.com: domain of leobras@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=leobras@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674632157; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=1+Ul5sQzoaw3CKL+Rr3WW09PeQYvbUe0nkd8DHiRlc0=; b=jgXiS40bg12XvHYb3PngJoBHF2tSwODpYSvYe2eMjTX1gdo2wANzAXtUdFjSmsh54l42Oq e46R9b0J4Kx52zn1tweBVQKR5fIO79qw7Vrm/cSSpZGgErzegSXTKD7ENp9UTLK9q1/OgG 98bfZjJ8mHAKF/7U5JkHmpi0I+ZuaAY= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=WbnmkwW1; spf=pass (imf13.hostedemail.com: domain of leobras@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=leobras@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674632157; a=rsa-sha256; cv=none; b=CMjCmetwGcgK3S0188PbWIv5SBAtpfUPeUJo01MZXXGN4Ah8fsX69Z8Q33f0PJA/viLR23 1x+A3twl8mdD4BApHEujSM1/NubM3KvA6VWOUCmtVQtrWW8jbl71u0hTmAjdcu4EKRmhs3 LczLzo1nJG7hGs62LEFoBqg1UiSrK1s= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1674632156; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1+Ul5sQzoaw3CKL+Rr3WW09PeQYvbUe0nkd8DHiRlc0=; b=WbnmkwW1rjLl2CrGo0eHYIjtpH9FjTq+my6XNyhCLj+L2YFWrjOmcDYBH17Q3dCmpryowT Kj1X46yQCi92HK09BZjgii9SGjj5axtkzSJrD3N8ZwjAG4wLES4ljYJxB+n8vMyfBBQdLL 8Ly5jijp3M7HE7ryNozOu8OyjzR31GY= Received: from mail-oi1-f199.google.com (mail-oi1-f199.google.com [209.85.167.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-304-_dtnnLXdP1eOd2K7q2zkhQ-1; Wed, 25 Jan 2023 02:35:55 -0500 X-MC-Unique: _dtnnLXdP1eOd2K7q2zkhQ-1 Received: by mail-oi1-f199.google.com with SMTP id u132-20020aca608a000000b00364e4f26dd3so5496981oib.3 for ; Tue, 24 Jan 2023 23:35:54 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1+Ul5sQzoaw3CKL+Rr3WW09PeQYvbUe0nkd8DHiRlc0=; b=btwjcBsgj9MvTKIsORtZOoaQN+J7n0bjiyD7CcVoCT8YAOFgHS/aUeyJQzHa4UDJcE HdV02C3THVK5dyuh34ncdpOkTEJoz6Msq8+AgJbzSmTR1o3eesxrO+8yZQhtCz3aVGoX FjUlbKA6fFm39U2h9YRiQIKAuDWAq08XiHY5h8g4ljg4J1O3o/ki0rE6muIx7YEzxxZ3 TSGKlCWMPxDJ5yzjjJj+Qh0oNRa2w3R5jnvund8DmqqJ7FdpVQhXJzvSSBiPpvbQGuzy JmvvkJngDEAdTR1SM5MeqAw2Q7myzHzPmHB0U1zMMKsiOU08Ymw38R8BdgHN2q3BsjPW m1Gw== X-Gm-Message-State: AFqh2krsz7c+jjZPzpxWD2/cLY2l7uKGfQdauFw1Jciud0CJG3gcLTQv CB0EO1OsAWTyIkYHV8CSord4EqDAA4gxyTE5hBDrFftG3bdOqBWk2wRmuuu0OGpQL2BPCU/eKOu 1216T8SdidXg= X-Received: by 2002:a05:6870:9124:b0:15f:c278:ef49 with SMTP id o36-20020a056870912400b0015fc278ef49mr9943007oae.6.1674632154322; Tue, 24 Jan 2023 23:35:54 -0800 (PST) X-Google-Smtp-Source: AMrXdXsrBVKLDiSLeQadT62Qslz+uBK/ThJ9hDHTsthiDQWfDXU1byL6/MWOVePm8CfYx4MAxndfSg== X-Received: by 2002:a05:6870:9124:b0:15f:c278:ef49 with SMTP id o36-20020a056870912400b0015fc278ef49mr9943000oae.6.1674632154115; Tue, 24 Jan 2023 23:35:54 -0800 (PST) Received: from LeoBras.redhat.com ([2804:1b3:a800:14fa:9361:c141:6c70:c877]) by smtp.gmail.com with ESMTPSA id x189-20020a4a41c6000000b0050dc79bb80esm1538802ooa.27.2023.01.24.23.35.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Jan 2023 23:35:53 -0800 (PST) From: Leonardo Bras To: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , Marcelo Tosatti Cc: Leonardo Bras , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 2/5] mm/memcontrol: Change stock_lock type from local_lock_t to spinlock_t Date: Wed, 25 Jan 2023 04:34:59 -0300 Message-Id: <20230125073502.743446-3-leobras@redhat.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230125073502.743446-1-leobras@redhat.com> References: <20230125073502.743446-1-leobras@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: 5pc58sx65hy66dg5dtgi59ix9hpgsn97 X-Rspamd-Queue-Id: 80AEF20017 X-HE-Tag: 1674632157-145043 X-HE-Meta: U2FsdGVkX1+SIHUbbRWVX8pEmHrUb1CEJRFI6L7SUn40wfagtI4F3bHXpX1iw6aRx1f5WcLzLDoozThmhZg/yPyh5ClbGG87TGRAMm2Qpy3Am2maGFbKkMMpaNOIPxyIwBXy9Piw3qTbuWIC6h1m13kounDNUiBFvGIDyKWYaApDnxxTp4PoMCBhjfTgkcZWxaWXMzy2UUTr168N7w4q0ektVreQgPxbEh7g7oaAeGIgpViRn/0HjQ0MuEJZkXEc80Qqu+jS2lqlrbUi75B5FiE5NzuYT00Y/VzsHopGeQPJ+AvRTe2a+XVK6FPiNoZoPWVy/d2xhrawReULze4OoLDvbXu/UtO9Latb/l02CBjDUWvC2ZLPV3WtkEkOYoqzlbakHr0dbBpHRHFl/VS3/3M947DSqYWmuv1AlKyg6XJ6fNEi5Hrqa6lkFHMjGyOi9B4i2SVYuzPInb0X3EgqRN/PVYuIM3z/g3EW7SRAHQAmWO7G+kJoBSBgHAXaHscflymTeFAIRKPMmwnuuHMNABLkis2yEWNJ6Z/R1mLO6d6RLqbbE2cVGieBqwYguQpPCvX/KAqIpYDRn3nn0M3edyoPvGEHxE52dVwEWvh9Fx4nh8oPIHUs8//tk/Rx1lCrhFMMTVa9pICKW3xpUlg4KwJCljjwRlU8oNBylZUxU8myt4HF9iu5SSa1UxTqEd3q28RWbaZpZzR+SkD+hAs5/QmrGx5T4QIvmKzgwYiCm2At5eKCXL6Kb4c4zNNbWBOPIVpzb5z9k2ChsWpz2ufGsNJV6YVJYP1vtK5oJJC+iRBPVUYu1McZkGCpbmRo6rxTCdnpcnwA5XIsmACiBVrSyaWb2JG/MQOC0OXKab8CV6FZDv1Sx94/qwgSzL9VjYQ/npgyauHe4iw6BB/rCVn3rvdj6112BxWeaBncpRsrU3h567c2MpyTYYk6XbcMgEeWOqrOKpAFWrElPrSWMbT AAMC07GQ s8Xe+N2GKDwZzgBdIkihgDSph5h/Nqvtr0Oc/jMkECc6Y9CNYUIRYOtYg/Wwoe6Y1cNqGCW8KD61UMD8UKP30fila8SdHyf0IlJpc+nCvCvNo1dpm3dXbHdOOzi1F65a8PSzH8rJNE3GfxMDQMKBdtiUepHuc7DtOgTZ2jU6a/ZyAR9x6Gnw8/0qJ/6xeYs6VAY3PjMDaJkLMb00GjOqvamegHn2bLmbHcOxNQ7I0s9eC78Nw3kTG84z3LheFo2TzNsir4DqEsa/EtqMcKXF79R/+sRBmAFNU6Zo/kWTxOyDwbIENPc+0UNkhMWyrS/k/aiSwMLK/5IDf9Ue/7xaVIqjVF5hcw8fBj6tmoxIFjOWLeeG6yvWOdodunxel5iqgfzUdNCpqmGE40e8lCj5tpWWTxdeF+I6MysdeycDAHSi/G2EKYMJSZjg0FQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In this context, since it's using per-cpu variables, changing from local_lock to spinlock should not deal much impact in performance and can allow operations such as stock draining to happen in remote cpus. Why performance would probably not get impacted: 1 - Since the lock is in the same cache line as the information that is written next, there is no much extra memory access cost for using the lock. 2 - Since it's a percpu struct, there should be rare for other cpu to share this cacheline, so there should be rare to need cacheline invalidation, and writing to the lock should be cheap since there is always a write to next struct members. 3 - Even the write in (2) could be pipelined and batched with following writes to the cacheline (such as nr_pages member), further decreasing the impact of this change. Suggested-by: Marcelo Tosatti Signed-off-by: Leonardo Bras --- mm/memcontrol.c | 38 ++++++++++++++++++++------------------ 1 file changed, 20 insertions(+), 18 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index f8e86b88b3c7a..1d5c108413c83 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2172,7 +2172,7 @@ void unlock_page_memcg(struct page *page) } struct memcg_stock_pcp { - local_lock_t stock_lock; + spinlock_t stock_lock; /* Protects the percpu struct */ struct mem_cgroup *cached; /* this never be root cgroup */ unsigned int nr_pages; @@ -2190,7 +2190,7 @@ struct memcg_stock_pcp { }; static DEFINE_PER_CPU_SHARED_ALIGNED(struct memcg_stock_pcp, memcg_stock) = { - .stock_lock = INIT_LOCAL_LOCK(stock_lock), + .stock_lock = __SPIN_LOCK_UNLOCKED(stock_lock), }; static DEFINE_MUTEX(percpu_charge_mutex); @@ -2235,15 +2235,15 @@ static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages) if (nr_pages > MEMCG_CHARGE_BATCH) return ret; - local_lock_irqsave(&memcg_stock.stock_lock, flags); - stock = this_cpu_ptr(&memcg_stock); + spin_lock_irqsave(&stock->stock_lock, flags); + if (memcg == stock->cached && stock->nr_pages >= nr_pages) { stock->nr_pages -= nr_pages; ret = true; } - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + spin_unlock_irqrestore(&stock->stock_lock, flags); return ret; } @@ -2280,14 +2280,14 @@ static void drain_local_stock(struct work_struct *dummy) * drain_stock races is that we always operate on local CPU stock * here with IRQ disabled */ - local_lock_irqsave(&memcg_stock.stock_lock, flags); - stock = this_cpu_ptr(&memcg_stock); + spin_lock_irqsave(&stock->stock_lock, flags); + old = drain_obj_stock(stock); drain_stock(stock); clear_bit(FLUSHING_CACHED_CHARGE, &stock->flags); - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + spin_unlock_irqrestore(&stock->stock_lock, flags); if (old) obj_cgroup_put(old); } @@ -2315,10 +2315,12 @@ static void __refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages) static void refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages) { unsigned long flags; + struct memcg_stock_pcp *stock; - local_lock_irqsave(&memcg_stock.stock_lock, flags); + stock = this_cpu_ptr(&memcg_stock); + spin_lock_irqsave(&stock->stock_lock, flags); __refill_stock(memcg, nr_pages); - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + spin_unlock_irqrestore(&stock->stock_lock, flags); } /* @@ -3165,8 +3167,8 @@ void mod_objcg_state(struct obj_cgroup *objcg, struct pglist_data *pgdat, unsigned long flags; int *bytes; - local_lock_irqsave(&memcg_stock.stock_lock, flags); stock = this_cpu_ptr(&memcg_stock); + spin_lock_irqsave(&stock->stock_lock, flags); /* * Save vmstat data in stock and skip vmstat array update unless @@ -3218,7 +3220,7 @@ void mod_objcg_state(struct obj_cgroup *objcg, struct pglist_data *pgdat, if (nr) mod_objcg_mlstate(objcg, pgdat, idx, nr); - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + spin_unlock_irqrestore(&stock->stock_lock, flags); if (old) obj_cgroup_put(old); } @@ -3229,15 +3231,15 @@ static bool consume_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes) unsigned long flags; bool ret = false; - local_lock_irqsave(&memcg_stock.stock_lock, flags); - stock = this_cpu_ptr(&memcg_stock); + spin_lock_irqsave(&stock->stock_lock, flags); + if (objcg == stock->cached_objcg && stock->nr_bytes >= nr_bytes) { stock->nr_bytes -= nr_bytes; ret = true; } - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + spin_unlock_irqrestore(&stock->stock_lock, flags); return ret; } @@ -3327,9 +3329,9 @@ static void refill_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes, unsigned long flags; unsigned int nr_pages = 0; - local_lock_irqsave(&memcg_stock.stock_lock, flags); - stock = this_cpu_ptr(&memcg_stock); + spin_lock_irqsave(&stock->stock_lock, flags); + if (stock->cached_objcg != objcg) { /* reset if necessary */ old = drain_obj_stock(stock); obj_cgroup_get(objcg); @@ -3345,7 +3347,7 @@ static void refill_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes, stock->nr_bytes &= (PAGE_SIZE - 1); } - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + spin_unlock_irqrestore(&stock->stock_lock, flags); if (old) obj_cgroup_put(old); From patchwork Wed Jan 25 07:35:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leonardo Bras X-Patchwork-Id: 13115050 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 52B7FC54E94 for ; Wed, 25 Jan 2023 07:36:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E27466B007E; Wed, 25 Jan 2023 02:36:19 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D614A6B0080; Wed, 25 Jan 2023 02:36:19 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B17AD6B0081; Wed, 25 Jan 2023 02:36:19 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 9C6C16B007E for ; Wed, 25 Jan 2023 02:36:19 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 1E403A08C7 for ; Wed, 25 Jan 2023 07:36:19 +0000 (UTC) X-FDA: 80392513278.24.BC4D6CD Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf29.hostedemail.com (Postfix) with ESMTP id D237A120008 for ; Wed, 25 Jan 2023 07:36:16 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=fJDjSUtZ; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf29.hostedemail.com: domain of leobras@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=leobras@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674632177; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=i0PZh+yTsPG9Po4mo5Kh71H2anPuGxn31rIFI1zWHXw=; b=E5BB9frte3ujN3TJO5Q/2bCsxlLdfsil8pZt30B0vGwszhLy0KIKQfyGLDr4Ja+ptUE/x4 thGCN3HAN3blr2UwtS6AVyssxHJvrH9h4YIDdTox98H7KGKnInRAnUFDAM4Oom4oPeUrZ0 v7J2x1NP3dckAmJS29mOK/L4qp/QMzU= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=fJDjSUtZ; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf29.hostedemail.com: domain of leobras@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=leobras@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674632177; a=rsa-sha256; cv=none; b=25nAxaObUVFi2YDSTNGsJMB2xBfpNtiGk9riOnvsVXC7S9Mu/UkoLtE2X59npu54qR2tPD P8/h/asoShahRDfOIVXopAQAwKfyLPDLqZeFXm9WteFaus+otvuyv363oZ+r4of9WnDB4V 9skn97/aHZG9J3BgS1Gx/F+J34bO0IU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1674632176; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=i0PZh+yTsPG9Po4mo5Kh71H2anPuGxn31rIFI1zWHXw=; b=fJDjSUtZZw3RUOLTxlL4fZgdCMI7Xfm+Tq+7ISCwQLMZa0d6ABZdmJDrxzBfRPxG2xm7+Q hrIfS+YBHvuGGJYTLiP9tbzmogBVTUSiCVxNQGe8xoB+dw2qGzZI3tQ3HrzN4A2vo1ysnI pCOGX2aabJCzAQ1pnR1Wsu1E0FBFF+w= Received: from mail-ot1-f70.google.com (mail-ot1-f70.google.com [209.85.210.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-649-LqxRd6sDPe20fGmJo8OcmQ-1; Wed, 25 Jan 2023 02:35:59 -0500 X-MC-Unique: LqxRd6sDPe20fGmJo8OcmQ-1 Received: by mail-ot1-f70.google.com with SMTP id e20-20020a9d5614000000b0068401872536so8726855oti.3 for ; Tue, 24 Jan 2023 23:35:59 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=i0PZh+yTsPG9Po4mo5Kh71H2anPuGxn31rIFI1zWHXw=; b=o9JB1dOPTJ1qdmNYvEZFJAvivBxP1uJCJaSeg/78Pl5Uok6BlqoNjpsS4Nm+s+FsZ1 OTiIRzol+50WOCpAu4goNgNUpDILaXWuxAYQA6fZKo5G44kpIYTE91wnf6xkFY/El9jj AVeqC08qoGARRQrkoBlR6gMmXjbP7Ai5dUD0JTIXdTAJ158X04a5phMecbmBqf4Osr6m lVFvmcv79VNg+gd3Wfm5vgwGbhEeTOr8S6egp23gAHB5BSILpQaXzyfY+27yKmIU9G5W vRObybSEfG+Un+XYuE0rtt228cmMJYrwlqj0oJmicKSbX/4vlayrc7WGFHM/BNMMKp0m u3cw== X-Gm-Message-State: AO0yUKVzBYl/okhsIAzEa7pDU4rC9/jqNy3e6vH5SdD7uPkzS5KOvzjt 2bPMHCIPdFbr875787T9jEMNN/q+PRS5lThlNdNBYA7NY3tKSGilF0936RtTeZuEvLCMrWMzG9g AJNhsJfKk+RY= X-Received: by 2002:a05:6870:a788:b0:163:2d2d:beb with SMTP id x8-20020a056870a78800b001632d2d0bebmr1040182oao.26.1674632159219; Tue, 24 Jan 2023 23:35:59 -0800 (PST) X-Google-Smtp-Source: AK7set8+5KjsnMLSgwOG2ncR10Nu23wlcmyAQ1nW0yNjNQg2bEEedKAnOT5WZ8vqhPBMDlxLuA4+1w== X-Received: by 2002:a05:6870:a788:b0:163:2d2d:beb with SMTP id x8-20020a056870a78800b001632d2d0bebmr1040175oao.26.1674632158988; Tue, 24 Jan 2023 23:35:58 -0800 (PST) Received: from LeoBras.redhat.com ([2804:1b3:a800:14fa:9361:c141:6c70:c877]) by smtp.gmail.com with ESMTPSA id x189-20020a4a41c6000000b0050dc79bb80esm1538802ooa.27.2023.01.24.23.35.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Jan 2023 23:35:58 -0800 (PST) From: Leonardo Bras To: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , Marcelo Tosatti Cc: Leonardo Bras , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 3/5] mm/memcontrol: Reorder memcg_stock_pcp members to avoid holes Date: Wed, 25 Jan 2023 04:35:00 -0300 Message-Id: <20230125073502.743446-4-leobras@redhat.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230125073502.743446-1-leobras@redhat.com> References: <20230125073502.743446-1-leobras@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: D237A120008 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: agpbz6bmctkq7t11ywm4nxtordn1tp48 X-HE-Tag: 1674632176-870714 X-HE-Meta: U2FsdGVkX18EVHHy8ezxg/FfrLMoCEXwZN4jmm1AwFab5W/YtNpsSf/ZoGEiwBthBuiR2L3sXvxqv/+vIdHNdMPk3+ZYkpV5nqGtLQrCI0k55uS8a0KEw1atWYIG9jC87EEgwot4O0S6Thor3e22mmI/Zw1zMXeLiOCji3ykCP+Fv052QMk5w96xraZskqS+Yhz3Ksd9Etb1+x+3HbqBMY08+o1JgTcyE1wVoWYfYNap3I7y+a2MrKwK5OdcjOLIarO7aIusA8XjMYsMLvfyRWhU7tNh5otDIdGUsG8c//Gq5dblFNQTWJ1KcqhP5BwA+BKPHQYV3XlTUXKM/KffFjoM2F8f1eH2mdCz1aESxWWA7Efypcz5t++PgRrORjFdGMFdiN8N9jSlsTluB16joqLIFPKmc6ZNOa+lSy7aRzpEL07yxSRlt5IaWTQBy5bvXq1NjGwPfm6gI5c+eHvKaldlzx1U+qz/bnd13V/OqR46mjUIhDxsjdvPH3ugvIRhf1g009zr/kogKZZ6AZdItBu72He9xIxdAQJdlmeI4OV43pi4VWTYGUoL1dOWSPXEdldAs8HAZqDFHvCPWmbEKwUx0U8qdxzbXOLn3gmnhD6tuwPapeEPq/B/WNXDhOvSDp4FU+jPo3T1HVDQKS1VOUPSmNeTjABWKJ0nfRaBOAJeBfHrAtS01lsNsQ3HxF1p6UoFI0SydZNQgze3qim3WUz3M6fNdnDNpTzqIlCddCm15SEnGMi9zT7vBlftbEzEaG0DDk/h8dU6h8VwxmO/7PjOgKOHrtD8w9anzh7gYr9SuSWYWmBKpdPHeFaLlLb8h69hiiXaZqIKQqvYbkGuuBErIv0TSLKgZIVHn0Gh1jjA0k97Bxv08fe2kR19a07n8yGQHILCAFvSlwZwt5VXRn3pwioFWTcNRKN7PxEyirEPQtcc4NSS88mouRKKBPdWw3/8EJdmVNXr4qDE5Ci cetTEjkw SL6anAsmLiPjOUqdCUB28M3QWHqHvK/rVtCE168WZ6QDQXRoaNfGHEGzr5A/LfXqPre0ZRZ2+ReHn/iCmLuE4ecOFv0AFFO3pWRlU1T2oHq+DNI2TMoa4b8/Srz9x3odMU+/Nge9RtPprBUd4JCZSXKBeIGZTcLo7MTrHSo2ujHfUR0FcPSUZTu15gmE/uDzA1H7b65JNdxJWQpkMkbL1XpxiOAeBO94P2FqWh+uJeCnKf8lKNBfM90WDrET9S1TIvA5vkBYSsOma5p/N4PtTq8wdoEPMBkN3H6ISiolTxW160HS3zn065oJj5kt5TDrrtK+IjvBvB20tp52Ek4jqiMppnSxjKIdKCZzMklfl/p5mPaZcg/M2gn7jqmnZyFOi6SSlV+xXqLsH+XqxkGV9NDT96MzoOEdbjqu9XMx8KNNYpPNeoBjkZ7Qkhw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In 64-bit architectures, the current layout of memcg_stock_pcp should look like this: struct memcg_stock_pcp { spinlock_t stock_lock; /* 0 4 */ /* 4 bytes hole */ struct mem_cgroup * cached; /* 8 8 */ unsigned int nr_pages; /* 16 4 */ /* 4 bytes hole */ [...] }; This happens because pointers will have 8 bytes (64-bit) and ints will have 4 bytes. Those 2 holes are avoided if we reorder nr_pages and cached, effectivelly putting nr_pages in the first hole, and saving 8 bytes. Signed-off-by: Leonardo Bras --- mm/memcontrol.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 1d5c108413c83..373fa78c4d881 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2173,8 +2173,8 @@ void unlock_page_memcg(struct page *page) struct memcg_stock_pcp { spinlock_t stock_lock; /* Protects the percpu struct */ - struct mem_cgroup *cached; /* this never be root cgroup */ unsigned int nr_pages; + struct mem_cgroup *cached; /* this never be root cgroup */ #ifdef CONFIG_MEMCG_KMEM struct obj_cgroup *cached_objcg; From patchwork Wed Jan 25 07:35:01 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leonardo Bras X-Patchwork-Id: 13115048 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ED998C54E94 for ; Wed, 25 Jan 2023 07:36:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 54C7D6B007B; Wed, 25 Jan 2023 02:36:09 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 505696B007D; Wed, 25 Jan 2023 02:36:09 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 39DB96B007E; Wed, 25 Jan 2023 02:36:09 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 2B1176B007B for ; Wed, 25 Jan 2023 02:36:09 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id E01531406B6 for ; Wed, 25 Jan 2023 07:36:08 +0000 (UTC) X-FDA: 80392512816.24.D7303B0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf29.hostedemail.com (Postfix) with ESMTP id CC89B120007 for ; Wed, 25 Jan 2023 07:36:06 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=AJBuLT3m; spf=pass (imf29.hostedemail.com: domain of leobras@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=leobras@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674632166; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=cta4F9u/56j6UuYJx1HDD/ltBRbZgzhdAn9u+4PdIyQ=; b=qO8Hjkws5OeKn/2bb6b6oBbYsgIB9OiNqbCQm0lj06cn3BFMIIjFVnYohRC7/E+VcBcV1T 6kr5FjSV1zyYM/qqG4prqdm5OAkZcoIWdHbrN90NywEER/cJVLxnhWA6Uq+QcMvHmqIRkK 7neo/l0RPgIYTVeuBjk1IWPltcDkcd4= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=AJBuLT3m; spf=pass (imf29.hostedemail.com: domain of leobras@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=leobras@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674632166; a=rsa-sha256; cv=none; b=qZa2RDItWI0OoNEyFVXbIeFmrCqmJKhNCtEhhI9QeoaXjFVY/FdM448KrdLfEzQKN779/P blSZ/r2U/Fn6zSTzbAu/1ZniVFx5P0/K7br8KXBFuaW+wLEjrblhKVWj9dGp7rWMpywmxj 9rJJ5I+6K4jV3cIuvkRngZZ+kpLEhPA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1674632166; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=cta4F9u/56j6UuYJx1HDD/ltBRbZgzhdAn9u+4PdIyQ=; b=AJBuLT3mysKio1fQzv/uX1FEm6r+H7xR3VNF0RV24Susp5SfYPFB6ZPxqDGOKpXkqyXb4k sS4FllFziE7W2oo4I0LCO1CKlLfnTQ4CsMIfP5okp66pwdWqYzj7W6dqdBGiOUV4NKZEqn X4dsysDPN5nhZOhYEZVTdxsDVB8D2qc= Received: from mail-ot1-f69.google.com (mail-ot1-f69.google.com [209.85.210.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-66-QiR7kdENNdepimXRSrpb9w-1; Wed, 25 Jan 2023 02:36:05 -0500 X-MC-Unique: QiR7kdENNdepimXRSrpb9w-1 Received: by mail-ot1-f69.google.com with SMTP id by2-20020a056830608200b00683e3e829d5so8870550otb.17 for ; Tue, 24 Jan 2023 23:36:04 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=cta4F9u/56j6UuYJx1HDD/ltBRbZgzhdAn9u+4PdIyQ=; b=mL/5gKVejYdJE9HsefEGzr+glz/uBmHUsX9hF60HOtiKw6zsfKP8EnSnvq66m6078G e9PTpW6f0RTXl8krA0CVVLyfHSL2rvV044y9QLgTP6Z8KGNyhzP4WtoRfzFgslrys3VZ duzKnF0SJjzqjb3PKkUYpqJqRlp+qzUJM3ODFpSBYTc+HQouu41L/FQA55DAo823kGVz Fzou0JHK3rYzXBzGxa+IjeYbyGES4n6/5QcjsLplQuR8XnSZcFmd9kERYUzOQCBVT5IV 3sB9ze/oTbRYBR9HVxE0bHpTQJrjB1B7j4ZRYRE+0PlkfycnSi6oOn2/qu2CuytrVkEu yPpA== X-Gm-Message-State: AFqh2kqay3UPogEjnLdtOFZiiCob0FXozznLXhaZ43HzVssmtg/SvRjg 1yfndAM+MQYJ5dHSFy8ud16x1BPlqjHXCCAE5mRttoqHXhCb0usVT1TfxrMykP88OWbxO+QNSeQ /pXIZEBCdyCc= X-Received: by 2002:a4a:d307:0:b0:4f5:2a03:b22a with SMTP id g7-20020a4ad307000000b004f52a03b22amr14131792oos.4.1674632164302; Tue, 24 Jan 2023 23:36:04 -0800 (PST) X-Google-Smtp-Source: AMrXdXsEUpYG3GHiKEkFMX0XaGUeQBQJBtt1RE/vEB6cze/oEgxS7/YGFSUZjjX5dJWUBMfQRU1UMg== X-Received: by 2002:a4a:d307:0:b0:4f5:2a03:b22a with SMTP id g7-20020a4ad307000000b004f52a03b22amr14131784oos.4.1674632164048; Tue, 24 Jan 2023 23:36:04 -0800 (PST) Received: from LeoBras.redhat.com ([2804:1b3:a800:14fa:9361:c141:6c70:c877]) by smtp.gmail.com with ESMTPSA id x189-20020a4a41c6000000b0050dc79bb80esm1538802ooa.27.2023.01.24.23.35.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Jan 2023 23:36:03 -0800 (PST) From: Leonardo Bras To: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , Marcelo Tosatti Cc: Leonardo Bras , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 4/5] mm/memcontrol: Perform all stock drain in current CPU Date: Wed, 25 Jan 2023 04:35:01 -0300 Message-Id: <20230125073502.743446-5-leobras@redhat.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230125073502.743446-1-leobras@redhat.com> References: <20230125073502.743446-1-leobras@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: hm5t3rgr7xcdxq5zw1mscieie4ua3mbm X-Rspamd-Queue-Id: CC89B120007 X-HE-Tag: 1674632166-864681 X-HE-Meta: U2FsdGVkX1+upbQjoR+G/8jNNI84xnc6Yq5gzpgr9qqAu1LTInpMbbH9O4SQJG3NVv5RnzqfHLZx+O1r33f7Y9OAwR9/osghW9IgpxX1TLYs/MaDrEeyP7nZFzcJvZHeni392c02Vtp5BFCUIbsl1cDjLnOufc1p5PmKWeT52HJZI2ogEd2rCBaHBrbtC/nw8ZvFvZB92lpYiIsLM8SbcdwK2tTvzmYEIGHtyUnlh7uqeEbGWD48m9r5bS5qe9QoPoG/Q1pAkWxph7cUAPc3mr15GuDJEXTsPGG8QpuT1JnygYdyqiCGy4Tf2UApBmXnzAfLvtnJDi0o2AxptI00Bcg7lC3eDP+hav8kgnnXoDfi0rh6CYLpHnvZGK95fT4nx27R6VHyfWjNeN+rh7Eqs/mYP5h2Mu4J5+/4z5xkwY7jVAbD57wBp8k5GXGqA9TopeT4Wp3JptdmyLuC6zK2VNye1IIWjSTaAHgJajRDvJ8q1ZFtVbrEGmfe7P40bFvsZcezKIsfa9LURnRbYwxHHJKzchTOgOJduK/K+3p0T80aLhRzKgqKSoKNWo6BkILWlCbFbl2l4j9woTx1uHjiiJsR1v8AyWYVBxKhHsl+MnVkVWLYA59fSJmjuXF4rxULWnxo6YxfNupYI9OIMzfqDiH9AN4iGHdF3z6Qx6h5YsGOT3dptu7YR6A9hd9cm3YRE2qkt09HhOV30jUm4J5B6ZE26kq4S3q8PJzngtbstdXA0e8/QZJzwzXzduWWTHJQ/NnWRpXnG2JZNCmD2SHtWe3BaQqsZO/Q/VoAy2/3Tw5IdRMmwaDBrfJIz9S4JhP8NsKv45gwweKhEkLMOCShepk3UJaunsybmJAb92nWsUndBG4hTDapjike2b8ncGM8zGwL13HDjozwFEVxQrQIu/bpwks93D1ClJTak9rE+ircxzphaA8k/Jnld5ukntLcOZrp5ib1hkcfrYhAUTO xHmSPM26 wcA7DKThGpkMyMvG7C8Po1CKDJzv1r/XujgQlpg6fr4VFBjWTerstC1EDRuCGrZLd/6f5OpUW02kuKoT7oSYP/QtIPn+ppJil2TkANpjKtk4RH6wbn6cYl66Sylo+Py/h7NfzgbIQFD/ynhU+m+gbFNko23rQrs23ZUw0Kd/I1TIMC2dtcbwNB/oaVPqAApo7TScB8jYms0E8H0hjTxv0JktzmsMjjo2OighkTqjd88Myj5k2prz3EBZXZ0Miuy0a6NEQxIZRR5v68CARvWQETvf6BKwbj5dIkA3H9ebyZ+GaTti0XQSbTWkW/MDmGpYyNt/gTXVfIYnPyLIYDYulh8B/fiInz2CYNRa7xR9pd84nerg7bu5qxr3YDFHdBO1WqJK/z7h/OWLKjc0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000012, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When drain_all_stock() is called, some CPUs will be required to have their per-CPU caches drained. This currently happens by scheduling a call to drain_local_stock() to run in each affected CPU. This, as a consequence, may end up scheduling work to CPUs that are isolated, and therefore should have as little interruption as possible. In order to avoid this, run all CPUs stock drain directly from the current CPU. This should be fine as long as drain_all_stock() runs fast enough so it don't often cause contention on consume_stock(), refill_stock(), mod_objcg_state(), consume_obj_stock() or refill_obj_stock(). Also, since drain_all_stock() will be able to run on a remote CPU, protect memcg_hotplug_cpu_dead() with stock_lock. Signed-off-by: Leonardo Bras --- mm/memcontrol.c | 29 +++++++++-------------------- 1 file changed, 9 insertions(+), 20 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 373fa78c4d881..5b7f7c2e0232f 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2184,7 +2184,6 @@ struct memcg_stock_pcp { int nr_slab_unreclaimable_b; #endif - struct work_struct work; unsigned long flags; #define FLUSHING_CACHED_CHARGE 0 }; @@ -2269,18 +2268,15 @@ static void drain_stock(struct memcg_stock_pcp *stock) stock->cached = NULL; } -static void drain_local_stock(struct work_struct *dummy) +static void drain_stock_from(struct memcg_stock_pcp *stock) { - struct memcg_stock_pcp *stock; struct obj_cgroup *old = NULL; unsigned long flags; /* - * The only protection from cpu hotplug (memcg_hotplug_cpu_dead) vs. - * drain_stock races is that we always operate on local CPU stock - * here with IRQ disabled + * The protection from cpu hotplug (memcg_hotplug_cpu_dead) vs. + * drain_stock races is stock_lock, a percpu spinlock. */ - stock = this_cpu_ptr(&memcg_stock); spin_lock_irqsave(&stock->stock_lock, flags); old = drain_obj_stock(stock); @@ -2329,7 +2325,7 @@ static void refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages) */ static void drain_all_stock(struct mem_cgroup *root_memcg) { - int cpu, curcpu; + int cpu; /* If someone's already draining, avoid adding running more workers. */ if (!mutex_trylock(&percpu_charge_mutex)) @@ -2341,7 +2337,6 @@ static void drain_all_stock(struct mem_cgroup *root_memcg) * per-cpu data. CPU up doesn't touch memcg_stock at all. */ migrate_disable(); - curcpu = smp_processor_id(); for_each_online_cpu(cpu) { struct memcg_stock_pcp *stock = &per_cpu(memcg_stock, cpu); struct mem_cgroup *memcg; @@ -2357,12 +2352,8 @@ static void drain_all_stock(struct mem_cgroup *root_memcg) rcu_read_unlock(); if (flush && - !test_and_set_bit(FLUSHING_CACHED_CHARGE, &stock->flags)) { - if (cpu == curcpu) - drain_local_stock(&stock->work); - else - schedule_work_on(cpu, &stock->work); - } + !test_and_set_bit(FLUSHING_CACHED_CHARGE, &stock->flags)) + drain_stock_from(stock); } migrate_enable(); mutex_unlock(&percpu_charge_mutex); @@ -2373,7 +2364,9 @@ static int memcg_hotplug_cpu_dead(unsigned int cpu) struct memcg_stock_pcp *stock; stock = &per_cpu(memcg_stock, cpu); + spin_lock(&stock->stock_lock); drain_stock(stock); + spin_unlock(&stock->stock_lock); return 0; } @@ -7328,7 +7321,7 @@ __setup("cgroup.memory=", cgroup_memory); */ static int __init mem_cgroup_init(void) { - int cpu, node; + int node; /* * Currently s32 type (can refer to struct batched_lruvec_stat) is @@ -7341,10 +7334,6 @@ static int __init mem_cgroup_init(void) cpuhp_setup_state_nocalls(CPUHP_MM_MEMCQ_DEAD, "mm/memctrl:dead", NULL, memcg_hotplug_cpu_dead); - for_each_possible_cpu(cpu) - INIT_WORK(&per_cpu_ptr(&memcg_stock, cpu)->work, - drain_local_stock); - for_each_node(node) { struct mem_cgroup_tree_per_node *rtpn; From patchwork Wed Jan 25 07:35:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leonardo Bras X-Patchwork-Id: 13115049 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 99F2DC27C76 for ; Wed, 25 Jan 2023 07:36:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 301A06B007D; Wed, 25 Jan 2023 02:36:19 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2B4136B007E; Wed, 25 Jan 2023 02:36:19 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 153956B0080; Wed, 25 Jan 2023 02:36:19 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 049846B007D for ; Wed, 25 Jan 2023 02:36:19 -0500 (EST) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id D0D03A0742 for ; Wed, 25 Jan 2023 07:36:18 +0000 (UTC) X-FDA: 80392513236.29.84CFFFA Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf05.hostedemail.com (Postfix) with ESMTP id C8BCE100009 for ; Wed, 25 Jan 2023 07:36:16 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=T2omN7L1; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf05.hostedemail.com: domain of leobras@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=leobras@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674632176; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=NWp8AY96vdXmiMbopROrFgs2cU/VADiUd7iIJJXBCxY=; b=MeDHeGzk8hELqtW8vYc9TBHGS2J9TYJOORVJdnkvLYb8dccU/iq17qSjYHTvsXFTTvKHTE 2ShApHQsSmMQUvZcINN5LWevEnE84Y99RdUNjog3a5pZyhBFxH30mXs7rWwgrZGonI6Cxn SvPwJuujQ0bodLgfo9vVPKSv9vua2ok= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=T2omN7L1; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf05.hostedemail.com: domain of leobras@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=leobras@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674632176; a=rsa-sha256; cv=none; b=6TFMqFPgntOsMApFGof4wa3+Ezwn+NqFQ2hrvg3jLf2aL/9Q/u2/auqUPkk9gfFT3tumt6 IxH91Unc9kAOWQ63Y1cMxMmhV0g9IMhOrInXkLYp9lbxQ0vQ+OHd5TCygbHGhoGuPNoyXS BKv2w6gf162tQLmsw7bYqE0b9LVLHE0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1674632176; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NWp8AY96vdXmiMbopROrFgs2cU/VADiUd7iIJJXBCxY=; b=T2omN7L1w1tR17xqH0l0c1D8Q4TUgVs1DahEWa2hT13FSHuVJxPAatSHpztQhVvkEjzM0u Z0X7kPTDAW2VacAi/YdlhP+jbaYqfrP+YJxZP3MVFze2KYZ1QOGDcvsE77gxWylCEJ48QV /WbqOOGnGF3v8Lvqn9wY+hdU0IohzC4= Received: from mail-oi1-f198.google.com (mail-oi1-f198.google.com [209.85.167.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-278-Ef7YekK2MpKblw2zrgAbyQ-1; Wed, 25 Jan 2023 02:36:10 -0500 X-MC-Unique: Ef7YekK2MpKblw2zrgAbyQ-1 Received: by mail-oi1-f198.google.com with SMTP id t25-20020a056808159900b0035ecfd3fa78so5506492oiw.4 for ; Tue, 24 Jan 2023 23:36:09 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=NWp8AY96vdXmiMbopROrFgs2cU/VADiUd7iIJJXBCxY=; b=M+HJPkT/G1Dty8jZys9B0mq7FT/uwq35GTW1fvdpyxO68NHB9xHJJhsyw63jaNJdIq IZqZqCwCSjyZqWv232mlzvaZdaEO9gwgQItprMzg5oNVtUj8IhwiVPhu27ShH4u2Suk9 vCifTaJe7chc12KwoqlagU3eDDczZgWUgV8qkHhFYHAShkyjzZy0+BaR049Vsxzi+Ghm h+LwhzeD+1WOGqFdLTvpy3RDnFfLex2tUO/PN/I1PajIAZBsyHpLr/WF+zGh2DXNGti0 cBI4l8CqmxXKV9Angb50qwJuN5OCmLLCb/L5CXrZhdRJwAfhGrrD6avRa1dTUiUoQ6vX vVyg== X-Gm-Message-State: AFqh2kryqL628C+OBvPI2HykEnw4dFquCgZjIr+i33+IafqlIcPbDrbE RPvAAqXImxVm7U43mqlKJJJu7GZMS0IaJX5VZjf48b8Kf9+LfQLlxxUeFQKiCqA34xzQhkrAFjW l0xW0BRilSn0= X-Received: by 2002:a05:6870:e0ce:b0:15f:3bb9:7b3d with SMTP id a14-20020a056870e0ce00b0015f3bb97b3dmr17462985oab.28.1674632169341; Tue, 24 Jan 2023 23:36:09 -0800 (PST) X-Google-Smtp-Source: AMrXdXv6r5ehrqKjuOqxpudgJdHfNcykat5kT9ZAeiiKrUzWSSAJYK3YalJM5sJ2WYW3DmqLUhiPTw== X-Received: by 2002:a05:6870:e0ce:b0:15f:3bb9:7b3d with SMTP id a14-20020a056870e0ce00b0015f3bb97b3dmr17462978oab.28.1674632169128; Tue, 24 Jan 2023 23:36:09 -0800 (PST) Received: from LeoBras.redhat.com ([2804:1b3:a800:14fa:9361:c141:6c70:c877]) by smtp.gmail.com with ESMTPSA id x189-20020a4a41c6000000b0050dc79bb80esm1538802ooa.27.2023.01.24.23.36.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Jan 2023 23:36:08 -0800 (PST) From: Leonardo Bras To: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , Marcelo Tosatti Cc: Leonardo Bras , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 5/5] mm/memcontrol: Remove flags from memcg_stock_pcp Date: Wed, 25 Jan 2023 04:35:02 -0300 Message-Id: <20230125073502.743446-6-leobras@redhat.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230125073502.743446-1-leobras@redhat.com> References: <20230125073502.743446-1-leobras@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: C8BCE100009 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: qwh664b84izz5yjdeibr46b9p1efddtu X-HE-Tag: 1674632176-927905 X-HE-Meta: U2FsdGVkX18cNHkpkBqt+Vk+MKdszRH+HWg/5T3FogdbKbSmzI3eGtZkK6r6DtXSnwGesbcts8YIGHYB1D/ktxMES38GiPwTa3oCe9mgUwmG3T7SuFm8mAfvfSQhKkeTqsBHCT8U4B+T8qv7YuupZzF16nXUsUCv4z89E0vWNixElbrE7d9ukd8q+Z/kqvSK0mP//cteCnNGCCRVWQUrkw+aVJBBDVGiG2+/H1e1bY0emuFfaQ29ZwuP/iwOr4dFouAT7F5L4eNmXzUFOIWZk9Md1JuKTt3vMGRFhniLnUwN38UKUmmW5dy++05JyqQZwD113C7cT9e12JKcQ9tMcZC5R3qOfa9dTHxXGgvG+xUGbiWedjUV/MWqeGHeRBt5peTx9CMh4BGTNuSxChnrZw31WFBgFBFeR8MosIb1u6+SuouNtGQPJyVMipu9LulC9gy+p61MQw1FhqnBXgZaPssPUjYUNGP2ImdL1RqSs7qUOVSr4e4r8Oc//qQzzl3LP8zHNZTkI4mKiGCg80Fsw0ALDD9X/9ImAeBMTT1rLie/f3eW26dCeWpoiYZVPEOvyXKK30CoH1epceoVg6Z1WzG7Hf+0CMht+/8Rt/0/9SzztEYYSvc8xSWp5j5M6vZCQk/7xtiH2qNfDzURNIKe0sO1Nk38bGnbTSt7/uUPYlFVTks+cmwMVIHpq+wVNGjczvA1Kas3RuOHQ9CE0DQSbS5L1EzKuSyLm70qA7+HLPVn/Gpd5vVlzWq2AoEpVF0yJw9kpi615wSHXLviQhIxtptZQv44IyZ8/rgeKC2lg9b8rjtdhruaZc4KdCl2RqWVYhIomScQavpBPcd2zNE1ztAAG4CsSBFZBxCnnhLAtisM77Hlkhy3cUcbdpavdG4Vo/bLb1JMsTSw4xox3ygg3T8smSAJYaSf/QJR66z1SpssqRG0qLS1W5n0pip9LlUc8xhCtLbgBW8caqooDVX znQjm3QW z9P0lLElD/D3r5rQ1d/GWuP9WeBg7JKL7sJq7vxHauPKlH24T7F8b5c7RPm8AahWIKEzz0DBQPFa/XRb4/uch0SxPx+ya78eTwAlc4UqdpWlxXrnmRKZoWsVOI8k4l++xp8dQFEqimHG98wX5jlVvaGv/WzMwhiXXNzSgwXWt1s1bTO4PBABm5vcGs3jznQ52f2Sf3h7z86aljVeFErzsDiNqiowzudBZlYbSB2z0naD+rRmYRVjc3Tq3XlKi5lIUtNX9CFMayWAVDkDwAX2X2yHP/0oLSVm0bDCzkFm96JA8e4EEZP2VnslaZi/Lx3LgVY/I+uUgFuPoMD/zgKcbmfpyYCcqjiYmreZZrSkTnvQ1L7OgGrw6ww2yZsls2GwMCIaCIltdM54iXypj2+BhdkNb7sSuKcFLGriKS1YQE4HASVoqERCQnuybtvX6xgC7ORmsaEqapbNTr1OPTBMigaZ7eVlg8pCkShZG X-Bogosity: Ham, tests=bogofilter, spamicity=0.000006, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The flags member of struct memcg_stock_pcp has only one used bit: FLUSHING_CACHED_CHARGE Both struct member and flag were created to avoid scheduling multiple instances of kworkers running drain_local_stock() for a single cpu. How could this scenario happen before: - drain_all_stock() gets called, get ownership of percpu_charge_mutex, schedules a drain_local_stock() on cpu X, and drops ownership of percpu_charge_mutex. - Another thread calls drain_all_stock(), get ownership of percpu_charge_mutex, schedules a drain_local_stock() on cpu X, ... Since the stock draining is now performed by the thread running drain_all_stock(), and happens before letting go of the percpu_charge_mutex, there is no chance of another drain happening between test_and_set_bit() and clear_bit(), so flags is now useless. Remove the flags member of memcg_stock_pcp, its usages and the FLUSHING_CACHED_CHARGE define. Signed-off-by: Leonardo Bras --- mm/memcontrol.c | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 5b7f7c2e0232f..60712f69595e4 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2183,9 +2183,6 @@ struct memcg_stock_pcp { int nr_slab_reclaimable_b; int nr_slab_unreclaimable_b; #endif - - unsigned long flags; -#define FLUSHING_CACHED_CHARGE 0 }; static DEFINE_PER_CPU_SHARED_ALIGNED(struct memcg_stock_pcp, memcg_stock) = { @@ -2281,7 +2278,6 @@ static void drain_stock_from(struct memcg_stock_pcp *stock) old = drain_obj_stock(stock); drain_stock(stock); - clear_bit(FLUSHING_CACHED_CHARGE, &stock->flags); spin_unlock_irqrestore(&stock->stock_lock, flags); if (old) @@ -2351,8 +2347,7 @@ static void drain_all_stock(struct mem_cgroup *root_memcg) flush = true; rcu_read_unlock(); - if (flush && - !test_and_set_bit(FLUSHING_CACHED_CHARGE, &stock->flags)) + if (flush) drain_stock_from(stock); } migrate_enable();