From patchwork Wed Nov 2 02:02:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Leonardo Bras X-Patchwork-Id: 13027669 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 68660C433FE for ; Wed, 2 Nov 2022 02:03:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EC0A66B0072; Tue, 1 Nov 2022 22:03:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E6FCF8E0002; Tue, 1 Nov 2022 22:03:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D11998E0001; Tue, 1 Nov 2022 22:03:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id BCB0D6B0072 for ; Tue, 1 Nov 2022 22:03:44 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 866F3141159 for ; Wed, 2 Nov 2022 02:03:44 +0000 (UTC) X-FDA: 80086855968.07.025B2D5 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf10.hostedemail.com (Postfix) with ESMTP id DCB7CC0003 for ; Wed, 2 Nov 2022 02:03:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1667354623; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=gev6P7R24DpRq36R8wXz2kmWLBFI3J+84j/AqTpnyIw=; b=dwXkNRTc+XTUgCAu+HOo40il5cjk/O907nXitbrfgpFNaF4qgYsEooCaGqbewr35HMQTPt G4Dxhj2wbqMrgFLXqNQ6QraOqsCt8IeT4UU+cIk287yLTe+TAyyXjp+/6UgLxZCxZa9AUh 8LHzMGNZ90vDhGDx+vv6wCocZAhh1V4= Received: from mail-oi1-f199.google.com (mail-oi1-f199.google.com [209.85.167.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-581-Eg-apmOSMeChiKj4qJSKZQ-1; Tue, 01 Nov 2022 22:03:39 -0400 X-MC-Unique: Eg-apmOSMeChiKj4qJSKZQ-1 Received: by mail-oi1-f199.google.com with SMTP id o12-20020a056808124c00b00353f308fb4bso7112652oiv.22 for ; Tue, 01 Nov 2022 19:03:39 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=gev6P7R24DpRq36R8wXz2kmWLBFI3J+84j/AqTpnyIw=; b=p4DHHVQS3lvF/8nAdQR/AFsBHBvDwXi7KHwPGu4zuDuocugnlOOj5I1v16dWDIA4dx Df91bGgr+2mBPgdeksoCOOufL7mHjXVjQLkcj6GE0vfYvN9sFoddhFpOkE78nGdOURmm jPT4zoTGNe5TPhkuZc/cmzUTlc19c5Hb5taYRq8td21rS5KeoGQAkCzZRL0jRuvtq+LY Rl3nhqHuDH4SFt8GU91yavrrcH7U6qZrkmdBNDZyjazZNuY05vojw99KTedgWLpQ4n/J GKA5hZKk601AvHV+Dc75l4mngMOxaNZGYFjNs9O9SqHagjGyMYBZQ+Fji99hFJrIeOfE jQNg== X-Gm-Message-State: ACrzQf2kyJKqG0T2i4oausSTyqhZBKffYELEERaxhKwBnkkk0Y7n0YI3 R80D70f8+2q8Gqhl3iHHeTkukY+Gl2HS6VCXuH7/KlmotZ/6aE5CGy4HyNb6bgZM1qW/iL7JwXB 6iDkzPpW8UUE= X-Received: by 2002:a05:6830:3704:b0:660:fe76:3cb7 with SMTP id bl4-20020a056830370400b00660fe763cb7mr10900889otb.21.1667354618882; Tue, 01 Nov 2022 19:03:38 -0700 (PDT) X-Google-Smtp-Source: AMsMyM7DG+li/A7VpDvIKqUuzfPg7gI21EhhlFF8vXWlawemkea0Kk/ubRWrfCId3EkcApts7tUZ1g== X-Received: by 2002:a05:6830:3704:b0:660:fe76:3cb7 with SMTP id bl4-20020a056830370400b00660fe763cb7mr10900864otb.21.1667354618638; Tue, 01 Nov 2022 19:03:38 -0700 (PDT) Received: from LeoBras.redhat.com ([2804:1b3:a802:1099:7cb2:3a49:6197:5307]) by smtp.gmail.com with ESMTPSA id h15-20020a9d6f8f000000b00665919f7823sm4526624otq.8.2022.11.01.19.03.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 01 Nov 2022 19:03:38 -0700 (PDT) From: Leonardo Bras To: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira , Valentin Schneider , Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , Frederic Weisbecker , Leonardo Bras , Phil Auld , Marcelo Tosatti Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v1 0/3] Avoid scheduling cache draining to isolated cpus Date: Tue, 1 Nov 2022 23:02:40 -0300 Message-Id: <20221102020243.522358-1-leobras@redhat.com> X-Mailer: git-send-email 2.38.1 MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1667354624; a=rsa-sha256; cv=none; b=VFVqaACkuZH21BAaldMrYSx6kRJPNopgKJ9VEFXawEdNYJo+Vmp7F7aZPYXnJ5AKfWYnl5 GG//S9WHVHDGjKaH+17jXHxyqNbcuOuDMeMVHhuG0WsYqGN6q91biT0lzZDx8gvid3FTaA bWEnW71W63tDPcNo163mNR65tm6gkaA= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=dwXkNRTc; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf10.hostedemail.com: domain of leobras@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=leobras@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1667354624; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=gev6P7R24DpRq36R8wXz2kmWLBFI3J+84j/AqTpnyIw=; b=mIUiqDsq349TBYqkvhcV5VHRztMV08hcYaj/l+LcEl28RaVmSl758+bC0eshHdMf+vJXRg aFUpCn5L8uMzqXcwdETAP7yN4+pLUM8L/MLipfn5idubJReStLJJdyBBvlpc9vl+225k08 CXwNbdSyWQUfdpkQubeq66LM2iqLUY0= X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: DCB7CC0003 X-Rspam-User: Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=dwXkNRTc; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf10.hostedemail.com: domain of leobras@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=leobras@redhat.com X-Stat-Signature: fsz8dfzb9c9jmc59nea9zgfp19scexdk X-HE-Tag: 1667354623-470755 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Patch #1 expands housekeepíng_any_cpu() so we can find housekeeping cpus closer (NUMA) to any desired CPU, instead of only the current CPU. ### Performance argument that motivated the change: There could be an argument of why would that be needed, since the current CPU is probably acessing the current cacheline, and so having a CPU closer to the current one is always the best choice since the cache invalidation will take less time. OTOH, there could be cases like this which uses perCPU variables, and we can have up to 3 different CPUs touching the cacheline: C1 - Isolated CPU: The perCPU data 'belongs' to this one C2 - Scheduling CPU: Schedule some work to be done elsewhere, current cpu C3 - Housekeeping CPU: This one will do the work Most of the times the cacheline is touched, it should be by C1. Some times a C2 will schedule work to run on C3, since C1 is isolated. If C1 and C2 are in different NUMA nodes, we could have C3 either in C2 NUMA node (housekeeping_any_cpu()) or in C1 NUMA node (housekeeping_any_cpu_from(C1). If C3 is in C2 NUMA node, there will be a faster invalidation when C3 tries to get cacheline exclusivity, and then a slower invalidation when this happens in C1, when it's working in its data. If C3 is in C1 NUMA node, there will be a slower invalidation when C3 tries to get cacheline exclusivity, and then a faster invalidation when this happens in C1. The thing is: it should be better to wait less when doing kernel work on an isolated CPU, even at the cost of some housekeeping CPU waiting a few more cycles. ### Patch #2 changes the locking strategy of memcg_stock_pcp->stock_lock from local_lock to spinlocks, so it can be later used to do remote percpu cache draining on patch #3. Most performance concerns should be pointed in the commit log. Patch #3 implements the remote per-CPU cache drain, making use of both patches #2 and #3. Performance-wise, in non-isolated scenarios, it should introduce an extra function call and a single test to check if the CPU is isolated. On scenarios with isolation enabled on boot, it will also introduce an extra test to check in the cpumask if the CPU is isolated. If it is, there will also be an extra read of the cpumask to look for a housekeeping CPU. Please, provide any feedback on that! Thanks a lot for reading! Leonardo Bras (3): sched/isolation: Add housekeepíng_any_cpu_from() mm/memcontrol: Change stock_lock type from local_lock_t to spinlock_t mm/memcontrol: Add drain_remote_stock(), avoid drain_stock on isolated cpus include/linux/sched/isolation.h | 11 +++-- kernel/sched/isolation.c | 8 ++-- mm/memcontrol.c | 83 ++++++++++++++++++++++----------- 3 files changed, 69 insertions(+), 33 deletions(-)