From patchwork Wed Nov 2 02:02:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Leonardo Bras X-Patchwork-Id: 13027670 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A38DFC4332F for ; Wed, 2 Nov 2022 02:03:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 263E98E0001; Tue, 1 Nov 2022 22:03:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1EC8B6B0074; Tue, 1 Nov 2022 22:03:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 03F118E0001; Tue, 1 Nov 2022 22:03:46 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id E1E036B0073 for ; Tue, 1 Nov 2022 22:03:46 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id BB5B71C62D0 for ; Wed, 2 Nov 2022 02:03:46 +0000 (UTC) X-FDA: 80086856052.19.60DE6B6 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf08.hostedemail.com (Postfix) with ESMTP id 601B0160002 for ; Wed, 2 Nov 2022 02:03:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1667354625; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CMSAt3GthE2zWUBTJjLiVIVQICwCKfwnNgpg1Ix0oWM=; b=VTYqX+oBfJTKfIr/hUEjVrAwQ5eU1flJmRBcaqpCDAKndBHpTohSyN9M/xdqQPVsW0W1n9 sR6WEXNoTSgFCFsSuzry8kmWI1ObrPUQvEohVE1R1FdBJT3vmbpLjYTNA7xHTolSyUqLP/ rvl5DBdMR/Df5t2j3mPMmdGk5H0+X0U= Received: from mail-ot1-f72.google.com (mail-ot1-f72.google.com [209.85.210.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-453-9JVLCAhkNImbkOgT3e7H9w-1; Tue, 01 Nov 2022 22:03:44 -0400 X-MC-Unique: 9JVLCAhkNImbkOgT3e7H9w-1 Received: by mail-ot1-f72.google.com with SMTP id l31-20020a9d1b22000000b0066c48e9249fso4641092otl.5 for ; Tue, 01 Nov 2022 19:03:44 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=CMSAt3GthE2zWUBTJjLiVIVQICwCKfwnNgpg1Ix0oWM=; b=IH19u4N8MULdQckAz9WKT39XdybSvFRzpa0nO91ydQ4qeZtiCvPv+vESMOqSTdGThL VrOgVCiVo0rVSUZ7Y8++SVTMadZLfdIRl1yvPk+0/ToTTS04xiSMVmIV+lBb2n+TvALL AaLgELsvVB2WhqAXuc+UP7Y0erxVTVI1E4e1AZUQBf8cLieHkcQ3gyCplcJUhutc5N3z mEOxRxetCtFIUCyV+9Q6DOlFzyZs9wowUPja/OhH/VTF7HwLWQH/lSOAB1CMLXsfwsej eASpoCy6dIno3RIc3Jmfzlxjb4btC11sx88wHLNfaLjGZU+bs1J9OXkmoPtEiU1z1ZiR Q+jQ== X-Gm-Message-State: ACrzQf2z8UM89eZD38l130f0UnFrlzGme9/xBUjHzgh0WAtiV3RgYFco OYG8RZhKdc+V0md2jEMEQ97FGaMtK4Idq4QLyqXUA6BCwWE0y+vqz9fbRrmnfnPqvhdhoyhMdbQ QYK/WGd8gw1E= X-Received: by 2002:a05:6870:d250:b0:13b:3100:abcc with SMTP id h16-20020a056870d25000b0013b3100abccmr13089122oac.3.1667354624067; Tue, 01 Nov 2022 19:03:44 -0700 (PDT) X-Google-Smtp-Source: AMsMyM5zv2uj8jD1EUkakgeYhALybaybY7reFpZn0f4powVIbb55KSIq40hrRxQHM4RpX4sgR1KnYw== X-Received: by 2002:a05:6870:d250:b0:13b:3100:abcc with SMTP id h16-20020a056870d25000b0013b3100abccmr13089090oac.3.1667354623748; Tue, 01 Nov 2022 19:03:43 -0700 (PDT) Received: from LeoBras.redhat.com ([2804:1b3:a802:1099:7cb2:3a49:6197:5307]) by smtp.gmail.com with ESMTPSA id h15-20020a9d6f8f000000b00665919f7823sm4526624otq.8.2022.11.01.19.03.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 01 Nov 2022 19:03:43 -0700 (PDT) From: Leonardo Bras To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira , Valentin Schneider , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , Frederic Weisbecker , Leonardo Bras , Phil Auld , Marcelo Tosatti Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v1 1/3] =?utf-8?q?sched/isolation=3A_Add_housekeep=C3=ADng?= =?utf-8?q?=5Fany=5Fcpu=5Ffrom=28=29?= Date: Tue, 1 Nov 2022 23:02:41 -0300 Message-Id: <20221102020243.522358-2-leobras@redhat.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20221102020243.522358-1-leobras@redhat.com> References: <20221102020243.522358-1-leobras@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1667354626; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=CMSAt3GthE2zWUBTJjLiVIVQICwCKfwnNgpg1Ix0oWM=; b=bJnxdkXiPCyXYTgmhIC966MGx1wZWDroy5HqThE3nEx6Ow52XCOL/7sRyz7zTAy5emECny AoO7eBoej6InQGvHfZlor+MvGrOz9j6YfS9At3z+O7Dl72tnFStHUNLQcoKzV+tggiPBHl Ld9EKdtlMITIAPm+FNW5vQ2fLRynuA8= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=VTYqX+oB; spf=pass (imf08.hostedemail.com: domain of leobras@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=leobras@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1667354626; a=rsa-sha256; cv=none; b=njGM81OxSOqWo9JtPNF609VM6XxAmS5ifbgoyR3r3SZa2FcZdwODQmfT2/+3RNG9sE78qw 7z97SqyPJJbn4P7scBJH+HJVlhtlzQks4zCzRcaoKN1XY0sT36j+bBN802YWhCciax4jeM CfVkVCzubAVwHhElAjGuW54+vr/9liY= X-Stat-Signature: hyr7jqqh7xrkmtbt4dob7xyuyi1wbuux X-Rspamd-Queue-Id: 601B0160002 X-Rspamd-Server: rspam06 X-Rspam-User: Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=VTYqX+oB; spf=pass (imf08.hostedemail.com: domain of leobras@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=leobras@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-HE-Tag: 1667354626-229162 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: As of today, there is a function called housekeepíng_any_cpu() that returns a housekeeping cpu near the current one. This function is very useful to help delegate tasks to other cpus when the current one is isolated. It also comes with the benefit of looking for cpus in the same NUMA node as the current cpu, so any memory activity could be faster in NUMA systems. On the other hand, there is no function like that to find housekeeping cpus in the same NUMA node of another CPU. Change housekeepíng_any_cpu() into housekeepíng_any_cpu_from(), so it accepts a cpu_start parameter and can find cpus in the same NUMA node as any given CPU. Also, reimplements housekeepíng_any_cpu() as an inline function that calls housekeepíng_any_cpu_from() with cpu_start = current cpu. Signed-off-by: Leonardo Bras --- include/linux/sched/isolation.h | 11 ++++++++--- kernel/sched/isolation.c | 8 ++++---- 2 files changed, 12 insertions(+), 7 deletions(-) diff --git a/include/linux/sched/isolation.h b/include/linux/sched/isolation.h index 8c15abd67aed9..95b65be44f19f 100644 --- a/include/linux/sched/isolation.h +++ b/include/linux/sched/isolation.h @@ -20,7 +20,7 @@ enum hk_type { #ifdef CONFIG_CPU_ISOLATION DECLARE_STATIC_KEY_FALSE(housekeeping_overridden); -extern int housekeeping_any_cpu(enum hk_type type); +extern int housekeeping_any_cpu_from(enum hk_type type, int cpu_start); extern const struct cpumask *housekeeping_cpumask(enum hk_type type); extern bool housekeeping_enabled(enum hk_type type); extern void housekeeping_affine(struct task_struct *t, enum hk_type type); @@ -29,9 +29,9 @@ extern void __init housekeeping_init(void); #else -static inline int housekeeping_any_cpu(enum hk_type type) +static inline int housekeeping_any_cpu_from(enum hk_type type, int cpu_start) { - return smp_processor_id(); + return cpu_start; } static inline const struct cpumask *housekeeping_cpumask(enum hk_type type) @@ -58,4 +58,9 @@ static inline bool housekeeping_cpu(int cpu, enum hk_type type) return true; } +static inline int housekeeping_any_cpu(enum hk_type type) +{ + return housekeeping_any_cpu_from(type, smp_processor_id()); +} + #endif /* _LINUX_SCHED_ISOLATION_H */ diff --git a/kernel/sched/isolation.c b/kernel/sched/isolation.c index 373d42c707bc5..6ebeac11bb350 100644 --- a/kernel/sched/isolation.c +++ b/kernel/sched/isolation.c @@ -36,22 +36,22 @@ bool housekeeping_enabled(enum hk_type type) } EXPORT_SYMBOL_GPL(housekeeping_enabled); -int housekeeping_any_cpu(enum hk_type type) +int housekeeping_any_cpu_from(enum hk_type type, int cpu_start) { int cpu; if (static_branch_unlikely(&housekeeping_overridden)) { if (housekeeping.flags & BIT(type)) { - cpu = sched_numa_find_closest(housekeeping.cpumasks[type], smp_processor_id()); + cpu = sched_numa_find_closest(housekeeping.cpumasks[type], cpu_start); if (cpu < nr_cpu_ids) return cpu; return cpumask_any_and(housekeeping.cpumasks[type], cpu_online_mask); } } - return smp_processor_id(); + return cpu_start; } -EXPORT_SYMBOL_GPL(housekeeping_any_cpu); +EXPORT_SYMBOL_GPL(housekeeping_any_cpu_from); const struct cpumask *housekeeping_cpumask(enum hk_type type) { From patchwork Wed Nov 2 02:02:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leonardo Bras X-Patchwork-Id: 13027671 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B808CC433FE for ; Wed, 2 Nov 2022 02:03:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 549176B0073; Tue, 1 Nov 2022 22:03:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4F97B8E0002; Tue, 1 Nov 2022 22:03:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 399EA6B0075; Tue, 1 Nov 2022 22:03:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 274516B0073 for ; Tue, 1 Nov 2022 22:03:52 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 042E51A0701 for ; Wed, 2 Nov 2022 02:03:51 +0000 (UTC) X-FDA: 80086856304.08.4C25CF2 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf09.hostedemail.com (Postfix) with ESMTP id 84EED140003 for ; Wed, 2 Nov 2022 02:03:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1667354630; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=9awyJRy6zbIYnxU2Cyp8GDi5vivjY8I9wO3N/Lj5YCI=; b=bc3Ujp9aqaxeqvitBjJN3DjFbvEm2TT5p0CzTxi6SkMDw3arwE3uIJQ5489P8FcKfwXwdC XkbPPJPaIM4wFUzcawyIuYZuT9iehsbGrLUcBpPsQKXz8hJp1BQ01s5tnP4gBBV+7erqmd hQx6EK+54221FafkCD2NQjKzzplVtUY= Received: from mail-oa1-f70.google.com (mail-oa1-f70.google.com [209.85.160.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-629-RCKhKmD_Mg6m5CGU5rKwLw-1; Tue, 01 Nov 2022 22:03:49 -0400 X-MC-Unique: RCKhKmD_Mg6m5CGU5rKwLw-1 Received: by mail-oa1-f70.google.com with SMTP id 586e51a60fabf-13cbfc38be2so4656611fac.0 for ; Tue, 01 Nov 2022 19:03:49 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9awyJRy6zbIYnxU2Cyp8GDi5vivjY8I9wO3N/Lj5YCI=; b=FUCStFveJPlVWbP+IeCuSwOoGr9AiLKd2almpZrK4OfxzRZnujPEWA800CAOK4trVg HNUPopnMCduuEIfbQBLL19VXj4ekbPBgZRMQuAdsVpxGWIQLbuelshg0xK4mXlAdCcU7 7Dz+epOOYLimgWSpvSFjyP0QmpdcHCz8Alk9kZvaPCmqfk8JYWU2x97GzV0RDbGRpm7+ H4tsm/txDB6XDUKRsKN4cIdjqjLAfpRLKfeXN9Tt4lBwEtnv7vv+aJxlWvzXkKdPOIat +OQhOCmKhPvO1OA/117+MkhWI1cKH0PzVHSFB3D6H2kwkkaMzzcEM9EFgqTf0q5vxZKf U4Xw== X-Gm-Message-State: ACrzQf1qS4gMZliTyN4GiZSEKpPL5DYrRqQyps1TRsCq5cp0En2RMisq /Qud097noDtGaH72fBakn29ktglMkAXPa9DpKjsXB3GLc1pnM/VVgy35OkyB0qE1JuLdPgul1m6 wny3p+9o9GGQ= X-Received: by 2002:a05:6871:58b:b0:13c:be46:a02 with SMTP id u11-20020a056871058b00b0013cbe460a02mr11636082oan.8.1667354629093; Tue, 01 Nov 2022 19:03:49 -0700 (PDT) X-Google-Smtp-Source: AMsMyM5ltoJltHI6Z/nEB+eev0WW7uJWB140pg3Bn+RcqlseVCQsPXYU6LZG5IjvE4MMVOkxV19MIA== X-Received: by 2002:a05:6871:58b:b0:13c:be46:a02 with SMTP id u11-20020a056871058b00b0013cbe460a02mr11636072oan.8.1667354628892; Tue, 01 Nov 2022 19:03:48 -0700 (PDT) Received: from LeoBras.redhat.com ([2804:1b3:a802:1099:7cb2:3a49:6197:5307]) by smtp.gmail.com with ESMTPSA id h15-20020a9d6f8f000000b00665919f7823sm4526624otq.8.2022.11.01.19.03.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 01 Nov 2022 19:03:48 -0700 (PDT) From: Leonardo Bras To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira , Valentin Schneider , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , Frederic Weisbecker , Leonardo Bras , Phil Auld , Marcelo Tosatti Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v1 2/3] mm/memcontrol: Change stock_lock type from local_lock_t to spinlock_t Date: Tue, 1 Nov 2022 23:02:42 -0300 Message-Id: <20221102020243.522358-3-leobras@redhat.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20221102020243.522358-1-leobras@redhat.com> References: <20221102020243.522358-1-leobras@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=bc3Ujp9a; spf=pass (imf09.hostedemail.com: domain of leobras@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=leobras@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1667354631; a=rsa-sha256; cv=none; b=aaOyiwFCYqZ2nL1P4d3C3h8U0ZhO1Qp2VYTQYacJqflRd2HNppHX5eFwduRyBKip79fXFY MqO8fNs1zJNyxb0S8mmIaapICHqgfq1ReqVCFA+B6g+zUwdzHb5C9LxuFp1PrBJStt/vaI a/jDTG8z91fcT+ytm35iWUM5iCFE3HI= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1667354631; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9awyJRy6zbIYnxU2Cyp8GDi5vivjY8I9wO3N/Lj5YCI=; b=soDF63IY73kaA7jTKp1P1tchC7Yd/eXK2qr5dhpZ+HwxAw+cXyBipZKkbwe4JAahWbSI1f RvVfiJi9PbnQ8V0PddNSnHyaEzKOY7FJfoUl8t1mDWBq9/FTeL5kxud0i+3MVngI1ONRf3 df/cHy5BwDm5SN5+/TwLsJ9iS53H9MI= X-Stat-Signature: kx6hf1rxs78d6zsncttktxqp1irwdudj X-Rspamd-Queue-Id: 84EED140003 X-Rspam-User: Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=bc3Ujp9a; spf=pass (imf09.hostedemail.com: domain of leobras@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=leobras@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspamd-Server: rspam01 X-HE-Tag: 1667354631-401209 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In this context, since it's using per-cpu variables, changing from local_lock to spinlock should not deal much impact in performance and can allow operations such as stock draining to happen in remote cpus. Why performance would probably not get impacted: 1 - Since the lock is in the same cache line as the information that is used next, there is no extra memory access for caching the lock. 2 - Since it's a percpu struct, there should be no other cpu sharing this cacheline, so there is no need for cacheline invalidation, and writing to the lock should be as fast as the next struct members. 3 - Even the write in (2) could be pipelined and batched with following writes to the cacheline (such as nr_pages member), further decreasing the impact of this change. Suggested-by: Marcelo Tosatti Signed-off-by: Leonardo Bras --- mm/memcontrol.c | 38 ++++++++++++++++++++------------------ 1 file changed, 20 insertions(+), 18 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 2d8549ae1b300..add46da2e6df1 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2167,7 +2167,7 @@ void unlock_page_memcg(struct page *page) } struct memcg_stock_pcp { - local_lock_t stock_lock; + spinlock_t stock_lock; /* Protects the percpu struct */ struct mem_cgroup *cached; /* this never be root cgroup */ unsigned int nr_pages; @@ -2184,7 +2184,7 @@ struct memcg_stock_pcp { #define FLUSHING_CACHED_CHARGE 0 }; static DEFINE_PER_CPU(struct memcg_stock_pcp, memcg_stock) = { - .stock_lock = INIT_LOCAL_LOCK(stock_lock), + .stock_lock = __SPIN_LOCK_UNLOCKED(stock_lock), }; static DEFINE_MUTEX(percpu_charge_mutex); @@ -2229,15 +2229,15 @@ static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages) if (nr_pages > MEMCG_CHARGE_BATCH) return ret; - local_lock_irqsave(&memcg_stock.stock_lock, flags); - stock = this_cpu_ptr(&memcg_stock); + spin_lock_irqsave(&stock->stock_lock, flags); + if (memcg == stock->cached && stock->nr_pages >= nr_pages) { stock->nr_pages -= nr_pages; ret = true; } - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + spin_unlock_irqrestore(&stock->stock_lock, flags); return ret; } @@ -2274,14 +2274,14 @@ static void drain_local_stock(struct work_struct *dummy) * drain_stock races is that we always operate on local CPU stock * here with IRQ disabled */ - local_lock_irqsave(&memcg_stock.stock_lock, flags); - stock = this_cpu_ptr(&memcg_stock); + spin_lock_irqsave(&stock->stock_lock, flags); + old = drain_obj_stock(stock); drain_stock(stock); clear_bit(FLUSHING_CACHED_CHARGE, &stock->flags); - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + spin_unlock_irqrestore(&stock->stock_lock, flags); if (old) obj_cgroup_put(old); } @@ -2309,10 +2309,12 @@ static void __refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages) static void refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages) { unsigned long flags; + struct memcg_stock_pcp *stock; - local_lock_irqsave(&memcg_stock.stock_lock, flags); + stock = this_cpu_ptr(&memcg_stock); + spin_lock_irqsave(&stock->stock_lock, flags); __refill_stock(memcg, nr_pages); - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + spin_unlock_irqrestore(&stock->stock_lock, flags); } /* @@ -3157,8 +3159,8 @@ void mod_objcg_state(struct obj_cgroup *objcg, struct pglist_data *pgdat, unsigned long flags; int *bytes; - local_lock_irqsave(&memcg_stock.stock_lock, flags); stock = this_cpu_ptr(&memcg_stock); + spin_lock_irqsave(&stock->stock_lock, flags); /* * Save vmstat data in stock and skip vmstat array update unless @@ -3210,7 +3212,7 @@ void mod_objcg_state(struct obj_cgroup *objcg, struct pglist_data *pgdat, if (nr) mod_objcg_mlstate(objcg, pgdat, idx, nr); - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + spin_unlock_irqrestore(&stock->stock_lock, flags); if (old) obj_cgroup_put(old); } @@ -3221,15 +3223,15 @@ static bool consume_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes) unsigned long flags; bool ret = false; - local_lock_irqsave(&memcg_stock.stock_lock, flags); - stock = this_cpu_ptr(&memcg_stock); + spin_lock_irqsave(&stock->stock_lock, flags); + if (objcg == stock->cached_objcg && stock->nr_bytes >= nr_bytes) { stock->nr_bytes -= nr_bytes; ret = true; } - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + spin_unlock_irqrestore(&stock->stock_lock, flags); return ret; } @@ -3319,9 +3321,9 @@ static void refill_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes, unsigned long flags; unsigned int nr_pages = 0; - local_lock_irqsave(&memcg_stock.stock_lock, flags); - stock = this_cpu_ptr(&memcg_stock); + spin_lock_irqsave(&stock->stock_lock, flags); + if (stock->cached_objcg != objcg) { /* reset if necessary */ old = drain_obj_stock(stock); obj_cgroup_get(objcg); @@ -3337,7 +3339,7 @@ static void refill_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes, stock->nr_bytes &= (PAGE_SIZE - 1); } - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + spin_unlock_irqrestore(&stock->stock_lock, flags); if (old) obj_cgroup_put(old); From patchwork Wed Nov 2 02:02:43 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leonardo Bras X-Patchwork-Id: 13027672 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D0734C4332F for ; Wed, 2 Nov 2022 02:03:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6CF4A8E0003; Tue, 1 Nov 2022 22:03:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 67FA78E0002; Tue, 1 Nov 2022 22:03:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 548608E0003; Tue, 1 Nov 2022 22:03:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 417E48E0002 for ; Tue, 1 Nov 2022 22:03:59 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 10DE916039B for ; Wed, 2 Nov 2022 02:03:59 +0000 (UTC) X-FDA: 80086856598.17.9CC8428 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf01.hostedemail.com (Postfix) with ESMTP id B5AF840002 for ; Wed, 2 Nov 2022 02:03:56 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1667354636; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=i2RYvVpIxnZB4mRhcfM8kpuAulQpsyxXS11J9gDFkxg=; b=KMaVYxxzPFKvnkJvt0um+/4+8V6xJ+rpCSq1eVr0Jhxat67GfqBgSPSB+5lMiLkHC7l+4x NT/2o0+1jafMGD1ZwfTWUZxPn9fz67ml4yYYvHqqdpo4q+ZEDrLrYhMsNmJJLIEsZZWR0r zJHD9jRb8gjxtrv0/jU6mrSkBknqh3w= Received: from mail-ot1-f69.google.com (mail-ot1-f69.google.com [209.85.210.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-76-MuCe6HlrNQ-F7Jd3soiYbw-1; Tue, 01 Nov 2022 22:03:54 -0400 X-MC-Unique: MuCe6HlrNQ-F7Jd3soiYbw-1 Received: by mail-ot1-f69.google.com with SMTP id l31-20020a9d1b22000000b0066c48e9249fso4641262otl.5 for ; Tue, 01 Nov 2022 19:03:54 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=i2RYvVpIxnZB4mRhcfM8kpuAulQpsyxXS11J9gDFkxg=; b=Q9ezC1bDZf2k9mQ9V+XE7QXLeJh/Soe2SLPtsamjc4BoOhX0OUl1aMfS01MRXzai+d DPTcWgYGL9t6wfEh4MpE77lzmkxe7HZbjwWv5DhMymjpoYiFKVgdr7xCNtAYk/lmEZoo NTBa+LkMqht1yvhYMiT9dLctLfF2wAESPNCvJjobhSi1XdyqXJ58wph4lTyjmQ2XUOLO SxB+7fpDVXgQyxKhHjr1ewaeSp0bjHwl2UUsAikrLAnW7poNMsIMfridIeH9Vm/IBEzT OrBeLW51tiuJAM7dA32/jw3/A4KmfXpYb8R2PwXrZd8NvbjfdOokEODFXt3c8N/ROKUc XaIg== X-Gm-Message-State: ACrzQf0gbPT//K39OYw+YxYR3j1L66ZH7M0PWJjd0QyWpXwREBeo2uRb 0kBbKEBmbwjmyW672kn29agkcQqpk3KHj8A8//HV44EimzE9+mnXr/IKwTMknW9pC2s7XvcngjV BoIXTam3FxMQ= X-Received: by 2002:a05:6870:4212:b0:13c:d544:8e2a with SMTP id u18-20020a056870421200b0013cd5448e2amr10233057oac.28.1667354634218; Tue, 01 Nov 2022 19:03:54 -0700 (PDT) X-Google-Smtp-Source: AMsMyM5TucfJr70Pv/6ho03ND1IeMhGN5op58YZ9rbaUJ10xOKWU2YOlELtwuMUjwXZ9y/ol3qVj0Q== X-Received: by 2002:a05:6870:4212:b0:13c:d544:8e2a with SMTP id u18-20020a056870421200b0013cd5448e2amr10233044oac.28.1667354633998; Tue, 01 Nov 2022 19:03:53 -0700 (PDT) Received: from LeoBras.redhat.com ([2804:1b3:a802:1099:7cb2:3a49:6197:5307]) by smtp.gmail.com with ESMTPSA id h15-20020a9d6f8f000000b00665919f7823sm4526624otq.8.2022.11.01.19.03.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 01 Nov 2022 19:03:53 -0700 (PDT) From: Leonardo Bras To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira , Valentin Schneider , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , Frederic Weisbecker , Leonardo Bras , Phil Auld , Marcelo Tosatti Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v1 3/3] mm/memcontrol: Add drain_remote_stock(), avoid drain_stock on isolated cpus Date: Tue, 1 Nov 2022 23:02:43 -0300 Message-Id: <20221102020243.522358-4-leobras@redhat.com> X-Mailer: git-send-email 2.38.1 In-Reply-To: <20221102020243.522358-1-leobras@redhat.com> References: <20221102020243.522358-1-leobras@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=KMaVYxxz; spf=pass (imf01.hostedemail.com: domain of leobras@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=leobras@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1667354636; a=rsa-sha256; cv=none; b=sisO1cl9Sdx28izlV2Fqi36WKyFWg1sWrl0K8GmVFeDIRJw4wFCJcUiHe4R7YmyFAGcgOz XaVk7r96kym1JZ8VoHNDV1YdrxijltcjljJlSt4ilpvQI+SdqS0EG4Gdn4XFVOd4Vl8Plt HNgbzfJhDtnZIqkkGY1BS3ExPYa9cWE= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1667354636; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=i2RYvVpIxnZB4mRhcfM8kpuAulQpsyxXS11J9gDFkxg=; b=DKU2lG8gxZexp4fyH5nrjC2jLoAZDhBnQMdOZT9+yaJCqeVfnw3+Zwy/iOmPuZetSqR6zF LmwgWJF9sovrKlaT2OcTedaeg5taEqjaW+s3/eHscB4RXADVMQLkzag4vD6NGfyJjEpl5W aHJhURb2/D3Y+MnSF2xlbOd+dPl3Sok= X-Stat-Signature: 499ojimk7u5qo5igeykeqs4m5jmstmyc X-Rspamd-Queue-Id: B5AF840002 Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=KMaVYxxz; spf=pass (imf01.hostedemail.com: domain of leobras@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=leobras@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspam-User: X-Rspamd-Server: rspam12 X-HE-Tag: 1667354636-539863 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When drain_all_stock() is called, some CPUs will be required to have their per-CPU caches drained. This currently happens by scheduling a call to drain_local_stock() to run in each affected CPU. This, as a consequence, may end up scheduling work to CPUs that are isolated, and therefore should have as little interruption as possible. In order to avoid this, make drain_all_stock() able to detect isolated CPUs and schedule draining the perCPU stock to happen in another non-isolated CPU. But since the current implementation only allows the drain to happen in local CPU, implement a function to drain stock on a remote CPU: drain_remote_stock(). Given both drain_local_stock() and drain_remote_stock() do almost the same work, implement a inline drain_stock_helper() that is called by both. Also, since drain_stock() will be able to run on a remote CPU, protect memcg_hotplug_cpu_dead() with stock_lock. Signed-off-by: Leonardo Bras --- mm/memcontrol.c | 47 ++++++++++++++++++++++++++++++++++++++--------- 1 file changed, 38 insertions(+), 9 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index add46da2e6df1..7ad6e4f4b79ef 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -30,6 +30,7 @@ #include #include #include +#include #include #include #include @@ -2263,7 +2264,7 @@ static void drain_stock(struct memcg_stock_pcp *stock) stock->cached = NULL; } -static void drain_local_stock(struct work_struct *dummy) +static inline void drain_stock_helper(int cpu) { struct memcg_stock_pcp *stock; struct obj_cgroup *old = NULL; @@ -2271,10 +2272,9 @@ static void drain_local_stock(struct work_struct *dummy) /* * The only protection from cpu hotplug (memcg_hotplug_cpu_dead) vs. - * drain_stock races is that we always operate on local CPU stock - * here with IRQ disabled + * drain_stock races is stock_lock, a percpu spinlock. */ - stock = this_cpu_ptr(&memcg_stock); + stock = per_cpu_ptr(&memcg_stock, cpu); spin_lock_irqsave(&stock->stock_lock, flags); old = drain_obj_stock(stock); @@ -2286,6 +2286,16 @@ static void drain_local_stock(struct work_struct *dummy) obj_cgroup_put(old); } +static void drain_remote_stock(struct work_struct *work) +{ + drain_stock_helper(atomic_long_read(&work->data)); +} + +static void drain_local_stock(struct work_struct *dummy) +{ + drain_stock_helper(smp_processor_id()); +} + /* * Cache charges(val) to local per_cpu area. * This will be consumed by consume_stock() function, later. @@ -2352,10 +2362,16 @@ static void drain_all_stock(struct mem_cgroup *root_memcg) if (flush && !test_and_set_bit(FLUSHING_CACHED_CHARGE, &stock->flags)) { - if (cpu == curcpu) + if (cpu == curcpu) { drain_local_stock(&stock->work); - else + } else if (housekeeping_cpu(cpu, HK_TYPE_WQ)) { schedule_work_on(cpu, &stock->work); + } else { + int hkcpu = housekeeping_any_cpu_from(HK_TYPE_WQ, cpu); + + atomic_long_set(&stock->work.data, cpu); + schedule_work_on(hkcpu, &stock->work); + } } } migrate_enable(); @@ -2367,7 +2383,9 @@ static int memcg_hotplug_cpu_dead(unsigned int cpu) struct memcg_stock_pcp *stock; stock = &per_cpu(memcg_stock, cpu); + spin_lock(&stock->stock_lock); drain_stock(stock); + spin_unlock(&stock->stock_lock); return 0; } @@ -7272,9 +7290,20 @@ static int __init mem_cgroup_init(void) cpuhp_setup_state_nocalls(CPUHP_MM_MEMCQ_DEAD, "mm/memctrl:dead", NULL, memcg_hotplug_cpu_dead); - for_each_possible_cpu(cpu) - INIT_WORK(&per_cpu_ptr(&memcg_stock, cpu)->work, - drain_local_stock); + /* + * CPUs that are isolated should not spend cpu time for stock draining, + * so allow them to export this task to the nearest housekeeping enabled + * cpu available. + */ + for_each_possible_cpu(cpu) { + if (housekeeping_cpu(cpu, HK_TYPE_WQ)) { + INIT_WORK(&per_cpu_ptr(&memcg_stock, cpu)->work, + drain_local_stock); + } else { + INIT_WORK(&per_cpu_ptr(&memcg_stock, cpu)->work, + drain_remote_stock); + } + } for_each_node(node) { struct mem_cgroup_tree_per_node *rtpn;