From patchwork Wed Jan 25 07:34:59 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leonardo Bras X-Patchwork-Id: 13115047 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7700CC54E94 for ; Wed, 25 Jan 2023 07:36:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CEEAC6B0078; Wed, 25 Jan 2023 02:35:59 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C9E166B007B; Wed, 25 Jan 2023 02:35:59 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B3E4A6B007D; Wed, 25 Jan 2023 02:35:59 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id A19206B0078 for ; Wed, 25 Jan 2023 02:35:59 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 82B761A0935 for ; Wed, 25 Jan 2023 07:35:59 +0000 (UTC) X-FDA: 80392512438.19.5B38E3D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf13.hostedemail.com (Postfix) with ESMTP id 80AEF20017 for ; Wed, 25 Jan 2023 07:35:57 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=WbnmkwW1; spf=pass (imf13.hostedemail.com: domain of leobras@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=leobras@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674632157; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=1+Ul5sQzoaw3CKL+Rr3WW09PeQYvbUe0nkd8DHiRlc0=; b=jgXiS40bg12XvHYb3PngJoBHF2tSwODpYSvYe2eMjTX1gdo2wANzAXtUdFjSmsh54l42Oq e46R9b0J4Kx52zn1tweBVQKR5fIO79qw7Vrm/cSSpZGgErzegSXTKD7ENp9UTLK9q1/OgG 98bfZjJ8mHAKF/7U5JkHmpi0I+ZuaAY= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=WbnmkwW1; spf=pass (imf13.hostedemail.com: domain of leobras@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=leobras@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674632157; a=rsa-sha256; cv=none; b=CMjCmetwGcgK3S0188PbWIv5SBAtpfUPeUJo01MZXXGN4Ah8fsX69Z8Q33f0PJA/viLR23 1x+A3twl8mdD4BApHEujSM1/NubM3KvA6VWOUCmtVQtrWW8jbl71u0hTmAjdcu4EKRmhs3 LczLzo1nJG7hGs62LEFoBqg1UiSrK1s= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1674632156; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1+Ul5sQzoaw3CKL+Rr3WW09PeQYvbUe0nkd8DHiRlc0=; b=WbnmkwW1rjLl2CrGo0eHYIjtpH9FjTq+my6XNyhCLj+L2YFWrjOmcDYBH17Q3dCmpryowT Kj1X46yQCi92HK09BZjgii9SGjj5axtkzSJrD3N8ZwjAG4wLES4ljYJxB+n8vMyfBBQdLL 8Ly5jijp3M7HE7ryNozOu8OyjzR31GY= Received: from mail-oi1-f199.google.com (mail-oi1-f199.google.com [209.85.167.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-304-_dtnnLXdP1eOd2K7q2zkhQ-1; Wed, 25 Jan 2023 02:35:55 -0500 X-MC-Unique: _dtnnLXdP1eOd2K7q2zkhQ-1 Received: by mail-oi1-f199.google.com with SMTP id u132-20020aca608a000000b00364e4f26dd3so5496981oib.3 for ; Tue, 24 Jan 2023 23:35:54 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1+Ul5sQzoaw3CKL+Rr3WW09PeQYvbUe0nkd8DHiRlc0=; b=btwjcBsgj9MvTKIsORtZOoaQN+J7n0bjiyD7CcVoCT8YAOFgHS/aUeyJQzHa4UDJcE HdV02C3THVK5dyuh34ncdpOkTEJoz6Msq8+AgJbzSmTR1o3eesxrO+8yZQhtCz3aVGoX FjUlbKA6fFm39U2h9YRiQIKAuDWAq08XiHY5h8g4ljg4J1O3o/ki0rE6muIx7YEzxxZ3 TSGKlCWMPxDJ5yzjjJj+Qh0oNRa2w3R5jnvund8DmqqJ7FdpVQhXJzvSSBiPpvbQGuzy JmvvkJngDEAdTR1SM5MeqAw2Q7myzHzPmHB0U1zMMKsiOU08Ymw38R8BdgHN2q3BsjPW m1Gw== X-Gm-Message-State: AFqh2krsz7c+jjZPzpxWD2/cLY2l7uKGfQdauFw1Jciud0CJG3gcLTQv CB0EO1OsAWTyIkYHV8CSord4EqDAA4gxyTE5hBDrFftG3bdOqBWk2wRmuuu0OGpQL2BPCU/eKOu 1216T8SdidXg= X-Received: by 2002:a05:6870:9124:b0:15f:c278:ef49 with SMTP id o36-20020a056870912400b0015fc278ef49mr9943007oae.6.1674632154322; Tue, 24 Jan 2023 23:35:54 -0800 (PST) X-Google-Smtp-Source: AMrXdXsrBVKLDiSLeQadT62Qslz+uBK/ThJ9hDHTsthiDQWfDXU1byL6/MWOVePm8CfYx4MAxndfSg== X-Received: by 2002:a05:6870:9124:b0:15f:c278:ef49 with SMTP id o36-20020a056870912400b0015fc278ef49mr9943000oae.6.1674632154115; Tue, 24 Jan 2023 23:35:54 -0800 (PST) Received: from LeoBras.redhat.com ([2804:1b3:a800:14fa:9361:c141:6c70:c877]) by smtp.gmail.com with ESMTPSA id x189-20020a4a41c6000000b0050dc79bb80esm1538802ooa.27.2023.01.24.23.35.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Jan 2023 23:35:53 -0800 (PST) From: Leonardo Bras To: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , Marcelo Tosatti Cc: Leonardo Bras , cgroups@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 2/5] mm/memcontrol: Change stock_lock type from local_lock_t to spinlock_t Date: Wed, 25 Jan 2023 04:34:59 -0300 Message-Id: <20230125073502.743446-3-leobras@redhat.com> X-Mailer: git-send-email 2.39.1 In-Reply-To: <20230125073502.743446-1-leobras@redhat.com> References: <20230125073502.743446-1-leobras@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: 5pc58sx65hy66dg5dtgi59ix9hpgsn97 X-Rspamd-Queue-Id: 80AEF20017 X-HE-Tag: 1674632157-145043 X-HE-Meta: U2FsdGVkX1+SIHUbbRWVX8pEmHrUb1CEJRFI6L7SUn40wfagtI4F3bHXpX1iw6aRx1f5WcLzLDoozThmhZg/yPyh5ClbGG87TGRAMm2Qpy3Am2maGFbKkMMpaNOIPxyIwBXy9Piw3qTbuWIC6h1m13kounDNUiBFvGIDyKWYaApDnxxTp4PoMCBhjfTgkcZWxaWXMzy2UUTr168N7w4q0ektVreQgPxbEh7g7oaAeGIgpViRn/0HjQ0MuEJZkXEc80Qqu+jS2lqlrbUi75B5FiE5NzuYT00Y/VzsHopGeQPJ+AvRTe2a+XVK6FPiNoZoPWVy/d2xhrawReULze4OoLDvbXu/UtO9Latb/l02CBjDUWvC2ZLPV3WtkEkOYoqzlbakHr0dbBpHRHFl/VS3/3M947DSqYWmuv1AlKyg6XJ6fNEi5Hrqa6lkFHMjGyOi9B4i2SVYuzPInb0X3EgqRN/PVYuIM3z/g3EW7SRAHQAmWO7G+kJoBSBgHAXaHscflymTeFAIRKPMmwnuuHMNABLkis2yEWNJ6Z/R1mLO6d6RLqbbE2cVGieBqwYguQpPCvX/KAqIpYDRn3nn0M3edyoPvGEHxE52dVwEWvh9Fx4nh8oPIHUs8//tk/Rx1lCrhFMMTVa9pICKW3xpUlg4KwJCljjwRlU8oNBylZUxU8myt4HF9iu5SSa1UxTqEd3q28RWbaZpZzR+SkD+hAs5/QmrGx5T4QIvmKzgwYiCm2At5eKCXL6Kb4c4zNNbWBOPIVpzb5z9k2ChsWpz2ufGsNJV6YVJYP1vtK5oJJC+iRBPVUYu1McZkGCpbmRo6rxTCdnpcnwA5XIsmACiBVrSyaWb2JG/MQOC0OXKab8CV6FZDv1Sx94/qwgSzL9VjYQ/npgyauHe4iw6BB/rCVn3rvdj6112BxWeaBncpRsrU3h567c2MpyTYYk6XbcMgEeWOqrOKpAFWrElPrSWMbT AAMC07GQ s8Xe+N2GKDwZzgBdIkihgDSph5h/Nqvtr0Oc/jMkECc6Y9CNYUIRYOtYg/Wwoe6Y1cNqGCW8KD61UMD8UKP30fila8SdHyf0IlJpc+nCvCvNo1dpm3dXbHdOOzi1F65a8PSzH8rJNE3GfxMDQMKBdtiUepHuc7DtOgTZ2jU6a/ZyAR9x6Gnw8/0qJ/6xeYs6VAY3PjMDaJkLMb00GjOqvamegHn2bLmbHcOxNQ7I0s9eC78Nw3kTG84z3LheFo2TzNsir4DqEsa/EtqMcKXF79R/+sRBmAFNU6Zo/kWTxOyDwbIENPc+0UNkhMWyrS/k/aiSwMLK/5IDf9Ue/7xaVIqjVF5hcw8fBj6tmoxIFjOWLeeG6yvWOdodunxel5iqgfzUdNCpqmGE40e8lCj5tpWWTxdeF+I6MysdeycDAHSi/G2EKYMJSZjg0FQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In this context, since it's using per-cpu variables, changing from local_lock to spinlock should not deal much impact in performance and can allow operations such as stock draining to happen in remote cpus. Why performance would probably not get impacted: 1 - Since the lock is in the same cache line as the information that is written next, there is no much extra memory access cost for using the lock. 2 - Since it's a percpu struct, there should be rare for other cpu to share this cacheline, so there should be rare to need cacheline invalidation, and writing to the lock should be cheap since there is always a write to next struct members. 3 - Even the write in (2) could be pipelined and batched with following writes to the cacheline (such as nr_pages member), further decreasing the impact of this change. Suggested-by: Marcelo Tosatti Signed-off-by: Leonardo Bras --- mm/memcontrol.c | 38 ++++++++++++++++++++------------------ 1 file changed, 20 insertions(+), 18 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index f8e86b88b3c7a..1d5c108413c83 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2172,7 +2172,7 @@ void unlock_page_memcg(struct page *page) } struct memcg_stock_pcp { - local_lock_t stock_lock; + spinlock_t stock_lock; /* Protects the percpu struct */ struct mem_cgroup *cached; /* this never be root cgroup */ unsigned int nr_pages; @@ -2190,7 +2190,7 @@ struct memcg_stock_pcp { }; static DEFINE_PER_CPU_SHARED_ALIGNED(struct memcg_stock_pcp, memcg_stock) = { - .stock_lock = INIT_LOCAL_LOCK(stock_lock), + .stock_lock = __SPIN_LOCK_UNLOCKED(stock_lock), }; static DEFINE_MUTEX(percpu_charge_mutex); @@ -2235,15 +2235,15 @@ static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages) if (nr_pages > MEMCG_CHARGE_BATCH) return ret; - local_lock_irqsave(&memcg_stock.stock_lock, flags); - stock = this_cpu_ptr(&memcg_stock); + spin_lock_irqsave(&stock->stock_lock, flags); + if (memcg == stock->cached && stock->nr_pages >= nr_pages) { stock->nr_pages -= nr_pages; ret = true; } - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + spin_unlock_irqrestore(&stock->stock_lock, flags); return ret; } @@ -2280,14 +2280,14 @@ static void drain_local_stock(struct work_struct *dummy) * drain_stock races is that we always operate on local CPU stock * here with IRQ disabled */ - local_lock_irqsave(&memcg_stock.stock_lock, flags); - stock = this_cpu_ptr(&memcg_stock); + spin_lock_irqsave(&stock->stock_lock, flags); + old = drain_obj_stock(stock); drain_stock(stock); clear_bit(FLUSHING_CACHED_CHARGE, &stock->flags); - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + spin_unlock_irqrestore(&stock->stock_lock, flags); if (old) obj_cgroup_put(old); } @@ -2315,10 +2315,12 @@ static void __refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages) static void refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages) { unsigned long flags; + struct memcg_stock_pcp *stock; - local_lock_irqsave(&memcg_stock.stock_lock, flags); + stock = this_cpu_ptr(&memcg_stock); + spin_lock_irqsave(&stock->stock_lock, flags); __refill_stock(memcg, nr_pages); - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + spin_unlock_irqrestore(&stock->stock_lock, flags); } /* @@ -3165,8 +3167,8 @@ void mod_objcg_state(struct obj_cgroup *objcg, struct pglist_data *pgdat, unsigned long flags; int *bytes; - local_lock_irqsave(&memcg_stock.stock_lock, flags); stock = this_cpu_ptr(&memcg_stock); + spin_lock_irqsave(&stock->stock_lock, flags); /* * Save vmstat data in stock and skip vmstat array update unless @@ -3218,7 +3220,7 @@ void mod_objcg_state(struct obj_cgroup *objcg, struct pglist_data *pgdat, if (nr) mod_objcg_mlstate(objcg, pgdat, idx, nr); - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + spin_unlock_irqrestore(&stock->stock_lock, flags); if (old) obj_cgroup_put(old); } @@ -3229,15 +3231,15 @@ static bool consume_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes) unsigned long flags; bool ret = false; - local_lock_irqsave(&memcg_stock.stock_lock, flags); - stock = this_cpu_ptr(&memcg_stock); + spin_lock_irqsave(&stock->stock_lock, flags); + if (objcg == stock->cached_objcg && stock->nr_bytes >= nr_bytes) { stock->nr_bytes -= nr_bytes; ret = true; } - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + spin_unlock_irqrestore(&stock->stock_lock, flags); return ret; } @@ -3327,9 +3329,9 @@ static void refill_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes, unsigned long flags; unsigned int nr_pages = 0; - local_lock_irqsave(&memcg_stock.stock_lock, flags); - stock = this_cpu_ptr(&memcg_stock); + spin_lock_irqsave(&stock->stock_lock, flags); + if (stock->cached_objcg != objcg) { /* reset if necessary */ old = drain_obj_stock(stock); obj_cgroup_get(objcg); @@ -3345,7 +3347,7 @@ static void refill_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes, stock->nr_bytes &= (PAGE_SIZE - 1); } - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + spin_unlock_irqrestore(&stock->stock_lock, flags); if (old) obj_cgroup_put(old);