From patchwork Sat Jun 22 03:58:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leonardo Bras X-Patchwork-Id: 13708196 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BDF28C27C53 for ; Sat, 22 Jun 2024 03:58:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 233458D01B4; Fri, 21 Jun 2024 23:58:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1E40D8D01AF; Fri, 21 Jun 2024 23:58:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 00FD78D01B4; Fri, 21 Jun 2024 23:58:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id D52208D01AF for ; Fri, 21 Jun 2024 23:58:53 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 51BA9C132D for ; Sat, 22 Jun 2024 03:58:53 +0000 (UTC) X-FDA: 82257168546.23.C0A610F Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf10.hostedemail.com (Postfix) with ESMTP id 338ADC0008 for ; Sat, 22 Jun 2024 03:58:51 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=go9e3Ji5; spf=pass (imf10.hostedemail.com: domain of leobras@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=leobras@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1719028721; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=z0Dg1NIU4Aju17NE5wHHed1iDaZiJauMtlBEOZtTK7I=; b=5xcBIGJXNKdC50yRPZqh3IXvhvZqxf9aLZzAF4c9qoeT3NgItkXo36kWHbUatEqBRwMowJ 4LIbi3AEjdRdbQ6p/nt1je7TTVcY48Cy3F3nP/Eg5oQojxnjV0JRgeOLStb8SAr3jFIz7I lcYWrlwE4VLl0CUWnmiapo/2w9ZX2sM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1719028721; a=rsa-sha256; cv=none; b=lqqeowVRZ9SMb/mUtovx2ndeKQSlc0hhSrE+OFGFU/7It1g/eh5HQ5w4v52bdn0wzdKNTJ 2J9fBInAT2QUeRaNPu1Pj1Y98ugHgw7wiYwSfdkuHJCkbiWZkttVk8YDrApRX0wzI47lfX HzxKLOCbuthTpUX12pq6FNhOSObgubg= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=go9e3Ji5; spf=pass (imf10.hostedemail.com: domain of leobras@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=leobras@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1719028730; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=z0Dg1NIU4Aju17NE5wHHed1iDaZiJauMtlBEOZtTK7I=; b=go9e3Ji5vXMQq5MBlZ7Gdr6tsInirddnvOGUQN6wNLflCnTgh4Bfy5MJRssFx3ReA8+0Xf cHQmOQeV15lTx3QtWvnDGBqyfpGxZyYgaBn5D3GciL5sT6bEZHKbYEIitn5RGTX80MGeP1 ElqRk9kJKMDy0Q1EKTNTxOYiYtfCyA4= Received: from mail-pl1-f200.google.com (mail-pl1-f200.google.com [209.85.214.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-473-nJLn2vGNPYOCSXykC0yqaQ-1; Fri, 21 Jun 2024 23:58:46 -0400 X-MC-Unique: nJLn2vGNPYOCSXykC0yqaQ-1 Received: by mail-pl1-f200.google.com with SMTP id d9443c01a7336-1f9a0cb228eso26053475ad.1 for ; Fri, 21 Jun 2024 20:58:46 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719028725; x=1719633525; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=z0Dg1NIU4Aju17NE5wHHed1iDaZiJauMtlBEOZtTK7I=; b=hIQFckLJBHZEoqdwwjj2cslBkPSDixpgldBxobOXpSkdO7WAf6EbQgxvSAeTBGnQ6b 4nKXqYd4rm1dZLdj4Hr/0EIPaNHOV5/SbkSGfmVDnn9KvPLh5l9JI2WS5y4pbz2I6Q5t l0L5NpINRpGu0gTLVgvDFQ7U7R0WYcMcr3TrTsY/SXQ1a9sKygxb9GBcvEsNn692NvzO uJIFtSrHgKLSR8l3piuCsZzSIPYSbgCyth0CwxNwfPw9zN+aCMOJBCAKlaEYEHnGA512 EhsPttyi3yXng4dx5nZJksKkOt2x4vjtjqaNHgIeeVOb+eWTdGhBBchm7yf8s/dpsnl7 ljDw== X-Forwarded-Encrypted: i=1; AJvYcCU3aDPEkDOQ89q2sw/YgPnn7YuzBEfAScVuUtNw3PDr9G9zPI6Zg7Xymgz/ltCvkvVtjmP4kB9TxiC3F/ogbrFQlKs= X-Gm-Message-State: AOJu0YzrLABhC9A1RvJ3hb88cUEfAskJYxBz/l87JbLEs0yb1xiA7S2p vQaS0WQPqItOhU5VCz+MY32IoczBiYvcPPrBTPaaHGl21x2+sZE1icEbvFX/vL0yaRlwA6Hf1Ru CzZP+qLNTvT+yWuEudnZvwbOJvq7+/dEyzO06Ni/stVgglE5/ X-Received: by 2002:a17:903:2444:b0:1f9:d0da:5b42 with SMTP id d9443c01a7336-1f9d0da608amr58943045ad.46.1719028725069; Fri, 21 Jun 2024 20:58:45 -0700 (PDT) X-Google-Smtp-Source: AGHT+IE+Z05JsNeYced1ZM2XKeZjLOv76WXxQ8159J4vFgdL3RzUnlY7H/PhH/+1Ug253Ae9W8bh0A== X-Received: by 2002:a17:903:2444:b0:1f9:d0da:5b42 with SMTP id d9443c01a7336-1f9d0da608amr58942855ad.46.1719028724605; Fri, 21 Jun 2024 20:58:44 -0700 (PDT) Received: from LeoBras.redhat.com ([2804:1b3:a801:c138:e21d:3579:5747:ad1]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-1f9eb32b9edsm21832365ad.118.2024.06.21.20.58.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Jun 2024 20:58:44 -0700 (PDT) From: Leonardo Bras To: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Vlastimil Babka , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Leonardo Bras , Thomas Gleixner , Marcelo Tosatti Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 1/4] Introducing qpw_lock() and per-cpu queue & flush work Date: Sat, 22 Jun 2024 00:58:09 -0300 Message-ID: <20240622035815.569665-2-leobras@redhat.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240622035815.569665-1-leobras@redhat.com> References: <20240622035815.569665-1-leobras@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: 338ADC0008 X-Stat-Signature: c4b7qq4ttk4wb93wi7mojf7pmxnse6wm X-Rspamd-Server: rspam09 X-Rspam-User: X-HE-Tag: 1719028731-833469 X-HE-Meta: U2FsdGVkX19Z3RdznlPyCGZzYrSNaHiSytf4VKIeiU/JTsdJUjWgtoBSzNo4+5EtCWB0G7e+y8LDeYF0WlYlLwOYNzl5Zdf7h4mLLwLZRlsc/HGQGaQ8uUlZjjSk7P8lrCgivp/wxnMA+5KBMVwNnMSUm7VBXbeQPgvn2xFc+4reW8+OIRejeN9X2MdceZ58gCns1AxFXB+h6igoKqlsis/omYEFzOze3dXF14nK44DnTv9KQYbifHIPLneOzK5A/mPH8lb6kWEzYd3dVUuRbeCYnpeJjwV4mtpsy62hK3HRoRIc60xmbhS5zul/3P2bhKpD+Zvorhpxi+fzVPUx8wByAkZvGNP81TyCBkzlFVb2iLyVGkvdKNI56HcpjD4F7xqhqXamuXLVcUdCGaZE24IvpiZYIEBwWRHiYMJuTul/OENXYplY6BeDgZqLp4F2eAMT3D1q/juPWVtFyC4GOo1u1pOxlbIP4hfC4pAgg1WKCfhQr4BPMGA1P3zY/ftjAFlSuAcCAJq3/kmmIdvs3vtwY8pd8J6DwxlQ6lKtEwY9bcp4/SQbcXEXMQ5DzVFWS1WcVBMV+Rr2IssH0AmhmFCEqm5cJn6BEoxnCbT+BIV6at9KvbQuoS2FK6bLCGaQcCJAjqFFrM3MxaNLlwM+8v2QILkzaD2Uz72iXuSH89sWidWsEyhUQN1aALj6cbQVv+5qgKDkIRZB+Xa95J/0TIAtyc+4oHo8GhH5glrNyumV0rh+F2vrYN+kWWkJ/32J2rc6SWIBRMim7LWxPr5/FnlG/2YgM5j35QkNzIb7zba6JGmfQm8YS80RXiwNhtPvOkgu6tzNL8KR83B5XTvcq32yZ08+VLBgmJRuTloOspCB8utdFdIxWRS7FH3u5pXDEcK22v8pNdI+zd7bzpukrpvvV6I10xL9/VRrnHeLJXX5bsChRGAIIkt23k0nmIIBs29w3p00q4JZw2PxcwU FZK/T9pg sknyRKyKAH3g4kDCnpOlG8CGLhR0VkxNJM+Ffv8HlwAP+lokDSd806XxE/MczGYe/HSWA5t/8dyvnnZPKCG8o4zdThGBiG380UTfekBl8Sple/QE4s+f0MK/NaGCCAKLTzt7sC7UlN6QnCqDdpHbOI5upedeY6gFbRfiMT0XnbZeI+OfZppKlYd4E0rJSq4vSu/hRFmQHwY2v5oUrnS9/JvHu/UVRb8g5dWwb0BJ0TuKjNjNNL28UWP5Bl2thr/s2v2jgIRofobUbJZzEmVyhgPIHuXQNWdqOYnfxVBV9JaYqPi/2kO6CI4OIqNSavsyE1U7BJl9dhk9uzspmEHOZ/JKcgmkQ+BI56mTkZbdadB18dB+xdKnubDHWg/SnzDxb7J3Mf6JJmcem8ZKGPy2kKvDIlvwulPcb6J8kShFf+CHUHe/URmPlmGDvuMSNGRzy2u3na5IqrIqVwNjPMOYJn46+lA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Some places in the kernel implement a parallel programming strategy consisting on local_locks() for most of the work, and some rare remote operations are scheduled on target cpu. This keeps cache bouncing low since cacheline tends to be mostly local, and avoids the cost of locks in non-RT kernels, even though the very few remote operations will be expensive due to scheduling overhead. On the other hand, for RT workloads this can represent a problem: getting an important workload scheduled out to deal with some unrelated task is sure to introduce unexpected deadline misses. It's interesting, though, that local_lock()s in RT kernels become spinlock(). We can make use of those to avoid scheduling work on a remote cpu by directly updating another cpu's per_cpu structure, while holding it's spinlock(). In order to do that, it's necessary to introduce a new set of functions to make it possible to get another cpu's per-cpu "local" lock (qpw_{un,}lock*) and also the corresponding queue_percpu_work_on() and flush_percpu_work() helpers to run the remote work. On non-RT kernels, no changes are expected, as every one of the introduced helpers work the exactly same as the current implementation: qpw_{un,}lock*() -> local_{un,}lock*() (ignores cpu parameter) queue_percpu_work_on() -> queue_work_on() flush_percpu_work() -> flush_work() For RT kernels, though, qpw_{un,}lock*() will use the extra cpu parameter to select the correct per-cpu structure to work on, and acquire the spinlock for that cpu. queue_percpu_work_on() will just call the requested function in the current cpu, which will operate in another cpu's per-cpu object. Since the local_locks() become spinlock()s in PREEMPT_RT, we are safe doing that. flush_percpu_work() then becomes a no-op since no work is actually scheduled on a remote cpu. Some minimal code rework is needed in order to make this mechanism work: The calls for local_{un,}lock*() on the functions that are currently scheduled on remote cpus need to be replaced by qpw_{un,}lock_n*(), so in RT kernels they can reference a different cpu. It's also necessary to use a qpw_struct instead of a work_struct, but it just contains a work struct and, in PREEMPT_RT, the target cpu. This should have almost no impact on non-RT kernels: few this_cpu_ptr() will become per_cpu_ptr(,smp_processor_id()). On RT kernels, this should improve performance and reduce latency by removing scheduling noise. Signed-off-by: Leonardo Bras --- include/linux/qpw.h | 88 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 88 insertions(+) create mode 100644 include/linux/qpw.h diff --git a/include/linux/qpw.h b/include/linux/qpw.h new file mode 100644 index 000000000000..ea2686a01e5e --- /dev/null +++ b/include/linux/qpw.h @@ -0,0 +1,88 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_QPW_H +#define _LINUX_QPW_H + +#include "linux/local_lock.h" +#include "linux/workqueue.h" + +#ifndef CONFIG_PREEMPT_RT + +struct qpw_struct { + struct work_struct work; +}; + +#define qpw_lock(lock, cpu) \ + local_lock(lock) + +#define qpw_unlock(lock, cpu) \ + local_unlock(lock) + +#define qpw_lock_irqsave(lock, flags, cpu) \ + local_lock_irqsave(lock, flags) + +#define qpw_unlock_irqrestore(lock, flags, cpu) \ + local_unlock_irqrestore(lock, flags) + +#define queue_percpu_work_on(c, wq, qpw) \ + queue_work_on(c, wq, &(qpw)->work) + +#define flush_percpu_work(qpw) \ + flush_work(&(qpw)->work) + +#define qpw_get_cpu(qpw) \ + smp_processor_id() + +#define INIT_QPW(qpw, func, c) \ + INIT_WORK(&(qpw)->work, (func)) + +#else /* !CONFIG_PREEMPT_RT */ + +struct qpw_struct { + struct work_struct work; + int cpu; +}; + +#define qpw_lock(__lock, cpu) \ + do { \ + migrate_disable(); \ + spin_lock(per_cpu_ptr((__lock), cpu)); \ + } while (0) + +#define qpw_unlock(__lock, cpu) \ + do { \ + spin_unlock(per_cpu_ptr((__lock), cpu)); \ + migrate_enable(); \ + } while (0) + +#define qpw_lock_irqsave(lock, flags, cpu) \ + do { \ + typecheck(unsigned long, flags); \ + flags = 0; \ + qpw_lock(lock, cpu); \ + } while (0) + +#define qpw_unlock_irqrestore(lock, flags, cpu) \ + qpw_unlock(lock, cpu) + +#define queue_percpu_work_on(c, wq, qpw) \ + do { \ + struct qpw_struct *__qpw = (qpw); \ + WARN_ON((c) != __qpw->cpu); \ + __qpw->work.func(&__qpw->work); \ + } while (0) + +#define flush_percpu_work(qpw) \ + do {} while (0) + +#define qpw_get_cpu(w) \ + container_of((w), struct qpw_struct, work)->cpu + +#define INIT_QPW(qpw, func, c) \ + do { \ + struct qpw_struct *__qpw = (qpw); \ + INIT_WORK(&__qpw->work, (func)); \ + __qpw->cpu = (c); \ + } while (0) + +#endif /* CONFIG_PREEMPT_RT */ +#endif /* LINUX_QPW_H */ From patchwork Sat Jun 22 03:58:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leonardo Bras X-Patchwork-Id: 13708197 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 50868C27C53 for ; Sat, 22 Jun 2024 03:58:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D3BD68D01B5; Fri, 21 Jun 2024 23:58:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CE0E48D01AF; Fri, 21 Jun 2024 23:58:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B31B68D01B5; Fri, 21 Jun 2024 23:58:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 92F7E8D01AF for ; Fri, 21 Jun 2024 23:58:56 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 4053E1A13C0 for ; Sat, 22 Jun 2024 03:58:56 +0000 (UTC) X-FDA: 82257168672.17.C9E76D5 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf09.hostedemail.com (Postfix) with ESMTP id 32CF414001A for ; Sat, 22 Jun 2024 03:58:54 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=TrmswcNM; spf=pass (imf09.hostedemail.com: domain of leobras@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=leobras@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1719028728; a=rsa-sha256; cv=none; b=DREaxbeo1KLcM5S4UhbZz8uyqCyg55uRpk4/1dCLPGCf4eOmuRneRLDY2VUo6hd1a+y/fk iKgwaiS2Agbv5H7VwbERFMajGzcLSX90+XaOKZmNLu3Zp1XDDg6zYWu0QjypeEAoc1wrgR AbwlEWmWwSD334HMCV/PlWqUbuQXTc8= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=TrmswcNM; spf=pass (imf09.hostedemail.com: domain of leobras@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=leobras@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1719028728; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YZ33pz4CG9kAvALadE1yZYVBLwtylkbWD53nilFvhZA=; b=nlWYV+gFNJY9FGJAnUQWhITpiiDYV0g1DST6pHSHlqzvdxXaQZCb5YE1X2DebUPyr9ett6 GcGcNLRqo1s5mSqpCe0AO79nGZxFcFCyAN8zR0K1jet7gzPbRpvpV1/6STbq/4xpqeQo4U d2fgb2VOLuDW/OmNLD5UAi/y6Aq71Ko= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1719028733; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YZ33pz4CG9kAvALadE1yZYVBLwtylkbWD53nilFvhZA=; b=TrmswcNMiK457t4jXxGFe966qyRJLFxtsNDjh5mNNXNo2IlLalKUeBoBgT6uFOZaJJQJ/5 mU5NuXYF3CNDyRHA9sjmU8WFCWyQpQrsFFn/cIxyv2FQqDzQnksQ4xxw9rv6KaqE0BFR7N DgZfnfUYjAtucyG5L8xPBnO00DSnywg= Received: from mail-pl1-f199.google.com (mail-pl1-f199.google.com [209.85.214.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-223-BhsJnZJ_N6aqFwdNVHgRZQ-1; Fri, 21 Jun 2024 23:58:51 -0400 X-MC-Unique: BhsJnZJ_N6aqFwdNVHgRZQ-1 Received: by mail-pl1-f199.google.com with SMTP id d9443c01a7336-1f9a6356a02so23069095ad.3 for ; Fri, 21 Jun 2024 20:58:51 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719028731; x=1719633531; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YZ33pz4CG9kAvALadE1yZYVBLwtylkbWD53nilFvhZA=; b=EM8bJdZzUVmpnca5yCojzNFGsjLEp2DOTnuNI2CGQTXyzBBrhURYDwrNarl2PVIph2 qW2tH7/VUFOjtFmzWgWd/VcqIDNG78T5KHIWkhYzMliTEVvaoNn23VSEBDFqJ+0GInL/ nHCKoXC+vz6HYOpb/z271j2LDEW4rBQiVWFYo9mbgl2/+E8yB9tepWi3i2Z2EY0i04on Tj0VVzu8j4hwWG6YTMf+t+4PfbqTgX9tebU/0KabCyhVekMbGUeLIr0yF+dojomav9lJ pQGxDH4rrgFAB1W6Dt1AercTn9o6Y6Y0Ee6qeAYYEenFtxK1ozH6SxG/rKS6Jz3wGgXQ 3ZTw== X-Forwarded-Encrypted: i=1; AJvYcCUMtRMpSOU0qksoMRuerA9w1NjMLsJ5xspt3TGoO4Mu/QZLQxGf72sQXkZRPu8a17Wui3g91TuTkv7T4wQZ9acoKks= X-Gm-Message-State: AOJu0YwxYKb8SXfDATS75Z213DRJgWJf0ocz7O/+0bprTyFLcixGCjGL 7j/jP7rSTYopTcvP99cBHwb0d8SF56XWZRZNr/CPfwz3BI5pCe7SgAnTA3TzSIaN9S5PTullppd IgFzzv40g7CsUJ+bxrn+O+dZJajazraT+YvQyAhqLa3EpSWFz X-Received: by 2002:a17:903:1cf:b0:1f9:9d40:c9bf with SMTP id d9443c01a7336-1f9aa4121cdmr114843275ad.18.1719028730781; Fri, 21 Jun 2024 20:58:50 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHt4UWWF8YWLxNNykxeuK6L5NtoG3cPihSQQtBcVaMFhxxpNx9bPrFdzerXafMwk+8wW98i8A== X-Received: by 2002:a17:903:1cf:b0:1f9:9d40:c9bf with SMTP id d9443c01a7336-1f9aa4121cdmr114843195ad.18.1719028730370; Fri, 21 Jun 2024 20:58:50 -0700 (PDT) Received: from LeoBras.redhat.com ([2804:1b3:a801:c138:e21d:3579:5747:ad1]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-1f9eb32b9edsm21832365ad.118.2024.06.21.20.58.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Jun 2024 20:58:49 -0700 (PDT) From: Leonardo Bras To: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Vlastimil Babka , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Leonardo Bras , Thomas Gleixner , Marcelo Tosatti Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 2/4] swap: apply new queue_percpu_work_on() interface Date: Sat, 22 Jun 2024 00:58:10 -0300 Message-ID: <20240622035815.569665-3-leobras@redhat.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240622035815.569665-1-leobras@redhat.com> References: <20240622035815.569665-1-leobras@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Stat-Signature: dk1y1cphgc3jget7ryzqphqpfh3ku5ug X-Rspamd-Queue-Id: 32CF414001A X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1719028734-194649 X-HE-Meta: U2FsdGVkX18TjkiCUkXABss9R8R4fqU+YgBOO4K1lrZRgNtpMcUMwHLDBxu1GJEY110x8GREVbJTBXrjviGsvlbzNnMRF+pYxPMV3KwV9EyTCotzKVC6JbXZX5DotZavx0r/hMp9+2fGSY2f6kSFiDwG5pXwhEs1J9z1zVDtcbDd/NgDLrfmksRU/Q+bOMMgsRafuRi6ksuHdf15famKpLwqijeULpnKFzcFfrAdCEIri55ZDJzL3cFZcwph+/I98eCDGwFFAX8kP75F77/fz/8nR7oF5tqfFUwsRjJ97gojVHvY42SQKm1iLQ/ewWlWfhS5SLpaI5kWwDq5z8JXBcEgYSroG87a2FL1/BDwNcOLiEXJLZtmBJ0dsEUdUfOHPizLbex3GiceQzE3UGkVmUJacj2YRLHszZvk2woWXNFTQBMKpkbrIkktUBjoco3O8aJF+ArG0ccDOOF75xY7N9VJfJtzo3EDpAKrPS3niGD/MGMIMCV0nOflkcgOIeFNDRzY/VqNEIO0W2yZmOnqaQAk7L6mYop0zWEDvd2In5e0q4ST/zUaJ158Ss1WXyi/l4lUycqsRG82l5H/lzF5d8cpf6Y7ZNEk4GWuwDLGCZAbk9QAVNELadYCGc56rWCqCXkCjcdTl6hdRfsDA11we+qWtbunrJ5ddfV7dsARTmJlgzEyW/rHYNsjxeShFIGpCj7RUn73i9b0Qd2rR9BZBqWVElZfczBdbWYiSyID/yunB9JeqfGZS3JR9O0PWmSfTkTsxUlDXpH/RMfKrXBzz2eca0KSY+7ne/heZrXVijsVRAYNCwXYMmVudU9oaQPFhrBNJzb/4kjBBDLYyJ84d4CmbjNDrHti/gxVgXh7kRMHkRpWwYWT+H8aSVx9RzxHR/GAFZogIhvofbz0GMqLOmWs4swRwDFpQF/GQELpMcEFRsNmA2dNl19h+FkKBjxeyGAd8ggxRbmC99c+TpC nEW1VYbf HNuimg1yesg1/GIKiD7n8QEOIIRKqCTJf7gkyP6s1QKNMkq63jp12+MGTLREBafw+QTYLFMUhFGwDofL0Q3LdUO/fmbLGmBPfyV8Nf/FsT0FHdlPgoDS4mINLDhe3tGbIrvJQ5bCJAaVnGiv9HqQSei3/SH0XdeykhDzMSN3YOI08uARZppCdBP69XFsKso1YYI2223Rj7P4Vb5gC7vEx96/Jqfn6cmHUt3915tsTJfP8S+BmO3qnWsPldlHBeQseYJwk4KhjNtsJAaBDNdYIWvcWz20cRdy3zJDqj4BzbFE132b+pku7iO/cAWdhSywsD9Z/WkM5wiThd8q0dIsMV9vVvgLhbEZ6Tl82K9u5aDcImugr1moj8cfSvOJ4x2/ZarpdinH/dRyqawHs+MLBcL9+13+b9E65/GqpZTqaKrzEElQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Make use of the new qpw_{un,}lock*() and queue_percpu_work_on() interface to improve performance & latency on PREEMTP_RT kernels. For functions that may be scheduled in a different cpu, replace local_{un,}lock*() by qpw_{un,}lock*(), and replace schedule_work_on() by queue_percpu_work_on(). The same happens for flush_work() and flush_percpu_work(). The change requires allocation of qpw_structs instead of a work_structs, and changing parameters of a few functions to include the cpu parameter. This should bring no relevant performance impact on non-RT kernels: For functions that may be scheduled in a different cpu, the local_*lock's this_cpu_ptr() becomes a per_cpu_ptr(smp_processor_id()). Signed-off-by: Leonardo Bras --- mm/swap.c | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/mm/swap.c b/mm/swap.c index 67786cb77130..c1a61b7cd71a 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -28,21 +28,21 @@ #include #include #include #include #include #include #include #include #include #include -#include +#include #include #include "internal.h" #define CREATE_TRACE_POINTS #include /* How many pages do we try to swap or page in/out together? As a power of 2 */ int page_cluster; const int page_cluster_max = 31; @@ -758,45 +758,45 @@ void lru_add_drain(void) local_unlock(&cpu_fbatches.lock); mlock_drain_local(); } /* * It's called from per-cpu workqueue context in SMP case so * lru_add_drain_cpu and invalidate_bh_lrus_cpu should run on * the same cpu. It shouldn't be a problem in !SMP case since * the core is only one and the locks will disable preemption. */ -static void lru_add_and_bh_lrus_drain(void) +static void lru_add_and_bh_lrus_drain(int cpu) { - local_lock(&cpu_fbatches.lock); - lru_add_drain_cpu(smp_processor_id()); - local_unlock(&cpu_fbatches.lock); + qpw_lock(&cpu_fbatches.lock, cpu); + lru_add_drain_cpu(cpu); + qpw_unlock(&cpu_fbatches.lock, cpu); invalidate_bh_lrus_cpu(); mlock_drain_local(); } void lru_add_drain_cpu_zone(struct zone *zone) { local_lock(&cpu_fbatches.lock); lru_add_drain_cpu(smp_processor_id()); drain_local_pages(zone); local_unlock(&cpu_fbatches.lock); mlock_drain_local(); } #ifdef CONFIG_SMP -static DEFINE_PER_CPU(struct work_struct, lru_add_drain_work); +static DEFINE_PER_CPU(struct qpw_struct, lru_add_drain_qpw); -static void lru_add_drain_per_cpu(struct work_struct *dummy) +static void lru_add_drain_per_cpu(struct work_struct *w) { - lru_add_and_bh_lrus_drain(); + lru_add_and_bh_lrus_drain(qpw_get_cpu(w)); } static bool cpu_needs_drain(unsigned int cpu) { struct cpu_fbatches *fbatches = &per_cpu(cpu_fbatches, cpu); /* Check these in order of likelihood that they're not zero */ return folio_batch_count(&fbatches->lru_add) || data_race(folio_batch_count(&per_cpu(lru_rotate.fbatch, cpu))) || folio_batch_count(&fbatches->lru_deactivate_file) || @@ -882,31 +882,31 @@ static inline void __lru_add_drain_all(bool force_all_cpus) * * If the paired barrier is done at any later step, e.g. after the * loop, CPU #x will just exit at (C) and miss flushing out all of its * added pages. */ WRITE_ONCE(lru_drain_gen, lru_drain_gen + 1); smp_mb(); cpumask_clear(&has_work); for_each_online_cpu(cpu) { - struct work_struct *work = &per_cpu(lru_add_drain_work, cpu); + struct qpw_struct *qpw = &per_cpu(lru_add_drain_qpw, cpu); if (cpu_needs_drain(cpu)) { - INIT_WORK(work, lru_add_drain_per_cpu); - queue_work_on(cpu, mm_percpu_wq, work); + INIT_QPW(qpw, lru_add_drain_per_cpu, cpu); + queue_percpu_work_on(cpu, mm_percpu_wq, qpw); __cpumask_set_cpu(cpu, &has_work); } } for_each_cpu(cpu, &has_work) - flush_work(&per_cpu(lru_add_drain_work, cpu)); + flush_percpu_work(&per_cpu(lru_add_drain_qpw, cpu)); done: mutex_unlock(&lock); } void lru_add_drain_all(void) { __lru_add_drain_all(false); } #else @@ -939,21 +939,21 @@ void lru_cache_disable(void) * * Since v5.1 kernel, synchronize_rcu() is guaranteed to wait on * preempt_disable() regions of code. So any CPU which sees * lru_disable_count = 0 will have exited the critical * section when synchronize_rcu() returns. */ synchronize_rcu_expedited(); #ifdef CONFIG_SMP __lru_add_drain_all(true); #else - lru_add_and_bh_lrus_drain(); + lru_add_and_bh_lrus_drain(smp_processor_id()); #endif } /** * folios_put_refs - Reduce the reference count on a batch of folios. * @folios: The folios. * @refs: The number of refs to subtract from each folio. * * Like folio_put(), but for a batch of folios. This is more efficient * than writing the loop yourself as it will optimise the locks which need From patchwork Sat Jun 22 03:58:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leonardo Bras X-Patchwork-Id: 13708198 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AB6B7C27C53 for ; Sat, 22 Jun 2024 03:59:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 234948D01B6; Fri, 21 Jun 2024 23:59:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 195A48D01AF; Fri, 21 Jun 2024 23:59:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F29DE8D01B6; Fri, 21 Jun 2024 23:59:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id D309B8D01AF for ; Fri, 21 Jun 2024 23:59:01 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 953FE8133D for ; Sat, 22 Jun 2024 03:59:01 +0000 (UTC) X-FDA: 82257168882.03.B944AED Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf13.hostedemail.com (Postfix) with ESMTP id 84BB52000E for ; Sat, 22 Jun 2024 03:58:59 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=dNnBhZ3f; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf13.hostedemail.com: domain of leobras@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=leobras@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1719028731; a=rsa-sha256; cv=none; b=ayjWtvv47Uc4kMHlN7vDTraWL7fCNDiytMMimR0oqxRuqkTc6gizHQeUrrh7ViEBIpM/qx IdEFMla1UKiEq7+WXFMILa58SXFYb5xW4Yw5OVb13R33bb+mlVi56pbXIRVMgjcXn3TUE2 Q05vhZrea/Ncwzvv8jCviaFb9aZgvMA= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=dNnBhZ3f; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf13.hostedemail.com: domain of leobras@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=leobras@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1719028731; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kqjqAcyBF16xwLE61W3REKh7p8uOAAMqSPa12WNm3GE=; b=U0SJ5EXzYLICFGq2rZ8ZGxoc70vTtMdErc4P56Ny1GsIKTUWIHMynvyF6Xy57SPJ63moCJ czrG+xlHaqtq8xAUQJJcmew6vwYmanhL6lrAGncwWfVnSuMRui8x+kdk2tKyy3n7TRcaa5 de1NjulPNErF7zPu1vMfL636KZn8X+0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1719028738; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kqjqAcyBF16xwLE61W3REKh7p8uOAAMqSPa12WNm3GE=; b=dNnBhZ3fu37tK5ALUrxBoemtAGQoLee+hGA9NmvhWqTNvA/ENabHvL6Zayqdu4gqVI+S2v OLXhIN/q+42+0QT7JZGCgbVqyrAQvKEboFwZlMnVFVB5Umc7+XTG3CuOp//3aV5KzGKo9f adwW0aRLooNasXc5S+7Kdd4sic8g0dY= Received: from mail-pj1-f72.google.com (mail-pj1-f72.google.com [209.85.216.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-635-98AKigi3Pg2VZYwYFCrJpw-1; Fri, 21 Jun 2024 23:58:57 -0400 X-MC-Unique: 98AKigi3Pg2VZYwYFCrJpw-1 Received: by mail-pj1-f72.google.com with SMTP id 98e67ed59e1d1-2c78c2b7f4bso2698480a91.0 for ; Fri, 21 Jun 2024 20:58:57 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719028736; x=1719633536; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kqjqAcyBF16xwLE61W3REKh7p8uOAAMqSPa12WNm3GE=; b=qEj3rj67PQv9PWfmTyJG9AU9KGxX/A7jcJzQQp2WOaLXKup34tC4uJeZAMwCj2zp2s 42j1JQb28atPc6MoteDCTO5uMA6IGXvlvoqaQjjwLNXiMKh185OA1lIR9pBXshDuQF3T /Y10JvapLROpzSx0XiJc3zR+onfjUqIYU3g11tjroiaw1IJk61P7JtdWA2VyaGbY+45i Mp6MSsIPVkDqtLiFtZ3tXwLVGDnl/fd/aRsgt9uqN+4uOrZZ0+G6vfER1kGOvn9+G+dL GxjkqLKHXvIkfclFaJL5gfHzv/zM82b/j8hcA6USWMhvvydk7pCEm41UTYMJKMpaQsKf Smtw== X-Forwarded-Encrypted: i=1; AJvYcCW0Chde0cV1lhMTfX+c1j3NTxAd7/Q5M2CQ49HQmF9rkICLbVfojB2UETEH0yczZSlSpQy5bEGOyfC+dFUd+JXkqz4= X-Gm-Message-State: AOJu0Yzyu3X3lj9pSEKRAFxdfymq1CV0C9IACizpJHFUX6ix8+GNi3sN uOIUc5TpEvYX64BBA21qq1/W9X4rNtFWe+pbaLEb3+XvILeQP83/K1YTAnIQVAdxZY9d/Q+EN/p O0CrxBN15By/5fe3opaPn0ztUUVxH5kTpLGDIz+Q8cHx/A23L X-Received: by 2002:a17:903:1cf:b0:1f6:92f1:b01c with SMTP id d9443c01a7336-1f9aa481248mr125471875ad.69.1719028736461; Fri, 21 Jun 2024 20:58:56 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGsa/g3e768QiFMvXJv3hzlQlYWrPA/q475LVIhwI75gKqm/BT5XtIgr5pD5KmuF/MDMe+IuQ== X-Received: by 2002:a17:903:1cf:b0:1f6:92f1:b01c with SMTP id d9443c01a7336-1f9aa481248mr125471745ad.69.1719028736115; Fri, 21 Jun 2024 20:58:56 -0700 (PDT) Received: from LeoBras.redhat.com ([2804:1b3:a801:c138:e21d:3579:5747:ad1]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-1f9eb32b9edsm21832365ad.118.2024.06.21.20.58.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Jun 2024 20:58:55 -0700 (PDT) From: Leonardo Bras To: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Vlastimil Babka , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Leonardo Bras , Thomas Gleixner , Marcelo Tosatti Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 3/4] memcontrol: apply new queue_percpu_work_on() interface Date: Sat, 22 Jun 2024 00:58:11 -0300 Message-ID: <20240622035815.569665-4-leobras@redhat.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240622035815.569665-1-leobras@redhat.com> References: <20240622035815.569665-1-leobras@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: 84BB52000E X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 7pazzgy9q4owjq7kkyjd6tmm9czrgu6u X-HE-Tag: 1719028739-499332 X-HE-Meta: U2FsdGVkX1/bS3Awcmezr7H/7mN2gRpzC7awDkF4xN13Pg4XsM1bGoA/SS4lfQW2WDvcET3dcKEOllChEBf4egffEAomxPBOW3eur+jT7hysYg91bulhEztFWSajYYX1E1SWEHzODXylGqwL0cM4jGSOtxNFt8dh0vyt6VQoSRAPLEfzaALRkWXrlZTRGRHtTxrtX1L5pOSjdX2O6clbnS+syra++u9/q7YvSt3k5DaLsFtM2wB+Z6VE3Mbxk/x+DJti/qG2TEiTbEj7vh7yl81ZiT3ltU4CIDLtVX5Bzff7ANyF3lbudDbp4lvpy0v1lIFdbRDqiIqIK1lQTrb//ncPAg7Xy2xZfdm+c/Ew76owj9Lw7vOYY4/iKPRqQduUQ6cHruVCEWrUl+AUR7XGjOeDPQ8sW6PPXu7iAGwSno3oU1FOvriV8hF6ehlyBMX9PpOLVtc0O0FGy0IcbO2WnsDk0ioeOX8CbvhqM0NbrWyfQbIu26MaV9JDg6E6vUeycYYHg4vfmz0c0oeek6yAdN3MPTA/xXuA7sSiQpNehORX9CS8i6oL515zwEdP23z4vm1vWLPYQeTlzuELGXOgYnikYyDLuvhiLGl+J32bhl1uicN/vdY/suglRzj6mIchr05xrobDzN8UjOf8lkIgCXBp/HMbg0RRN5skaSRBKv73wDDI2kl/5/nfW8i5Xl6Aq21+ttEC6keSon4sMCiqklW1kA7lMZz9MDI+DyVxlL/3pJIFzTiRr95eUijqynuNn/DmKtw06D5Ft99d99LSC1rKkT/OWD3nv83m9eE8aRpG77CL1vVj4Bj0GzhJ8yl2qnnabavPeXYWEiw2LXJvwQjOgu4mhwdi/6VH6UPLtiFr4zRDpKblFsZbzi+3W2qqcZkx5sd4QJtBVT3OsCQF/T6ulODB2nIDsQG9MiA7COs4nLJsr/yDFyDwJv5dxxRFKVzv5LvEGT5XqmJ/y58 XlQdkA5F nYACY3vDTbrfAwyYGvArQBlNa+OfhX2vGFOnDNjHNIhCtqyT4SC7ukoB2i1q/tg2YYS6Jiqnksd8nsVt41WEodGYcpDg8I0QNLF9dq4cdc6F5AqOmDeWpZnPPZRyaVrP86bRms2BabXTP32JFBF8Layoj68lAewfmHIwhc2KexPKR5ZpGcAMqZDlsZyF6nifLdsqcGrqTvfWgMqfz9+Vx0/7ITSed82lWXXZwM+6dH89UUoWB9dQuV3M3AyzRYqO1oDg4lKckBUvd8xQ0xFLIJqHRunO/iy/JQJXoKIQRWOGdYWxnIHNWgbqOYuxhukpwW739x+Ie725X5gLnU28ykcH/5zKWkiVofVYenuEb3VZWyu3cMFdtpK54H1D3LRsyPC8ugMSeohyR8h0paFKUaX5II0Jprwok+4USfZpxPx9nXepta4hkwCQ9FcwmDm09HevTrbcu6dtR2aJ1Y8rgbVF17w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Make use of the new qpw_{un,}lock*() and queue_percpu_work_on() interface to improve performance & latency on PREEMTP_RT kernels. For functions that may be scheduled in a different cpu, replace local_{un,}lock*() by qpw_{un,}lock*(), and replace schedule_work_on() by queue_percpu_work_on(). This change requires allocation of qpw_structs instead of a work_structs. This should bring no relevant performance impact on non-RT kernels: For functions that may be scheduled in a different cpu, the local_*lock's this_cpu_ptr() becomes a per_cpu_ptr(smp_processor_id()). Signed-off-by: Leonardo Bras --- mm/memcontrol.c | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 71fe2a95b8bd..18a987f8c998 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -59,20 +59,21 @@ #include #include #include #include #include #include #include #include #include #include +#include #include "internal.h" #include #include #include "slab.h" #include "swap.h" #include #include @@ -2415,21 +2416,21 @@ struct memcg_stock_pcp { unsigned int nr_pages; #ifdef CONFIG_MEMCG_KMEM struct obj_cgroup *cached_objcg; struct pglist_data *cached_pgdat; unsigned int nr_bytes; int nr_slab_reclaimable_b; int nr_slab_unreclaimable_b; #endif - struct work_struct work; + struct qpw_struct qpw; unsigned long flags; #define FLUSHING_CACHED_CHARGE 0 }; static DEFINE_PER_CPU(struct memcg_stock_pcp, memcg_stock) = { .stock_lock = INIT_LOCAL_LOCK(stock_lock), }; static DEFINE_MUTEX(percpu_charge_mutex); #ifdef CONFIG_MEMCG_KMEM static struct obj_cgroup *drain_obj_stock(struct memcg_stock_pcp *stock); @@ -2503,39 +2504,40 @@ static void drain_stock(struct memcg_stock_pcp *stock) if (do_memsw_account()) page_counter_uncharge(&old->memsw, stock_pages); WRITE_ONCE(stock->nr_pages, 0); } css_put(&old->css); WRITE_ONCE(stock->cached, NULL); } -static void drain_local_stock(struct work_struct *dummy) +static void drain_local_stock(struct work_struct *w) { struct memcg_stock_pcp *stock; struct obj_cgroup *old = NULL; unsigned long flags; + int cpu = qpw_get_cpu(w); /* * The only protection from cpu hotplug (memcg_hotplug_cpu_dead) vs. * drain_stock races is that we always operate on local CPU stock * here with IRQ disabled */ - local_lock_irqsave(&memcg_stock.stock_lock, flags); + qpw_lock_irqsave(&memcg_stock.stock_lock, flags, cpu); - stock = this_cpu_ptr(&memcg_stock); + stock = per_cpu_ptr(&memcg_stock, cpu); old = drain_obj_stock(stock); drain_stock(stock); clear_bit(FLUSHING_CACHED_CHARGE, &stock->flags); - local_unlock_irqrestore(&memcg_stock.stock_lock, flags); + qpw_unlock_irqrestore(&memcg_stock.stock_lock, flags, cpu); obj_cgroup_put(old); } /* * Cache charges(val) to local per_cpu area. * This will be consumed by consume_stock() function, later. */ static void __refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages) { struct memcg_stock_pcp *stock; @@ -2592,23 +2594,23 @@ static void drain_all_stock(struct mem_cgroup *root_memcg) if (memcg && READ_ONCE(stock->nr_pages) && mem_cgroup_is_descendant(memcg, root_memcg)) flush = true; else if (obj_stock_flush_required(stock, root_memcg)) flush = true; rcu_read_unlock(); if (flush && !test_and_set_bit(FLUSHING_CACHED_CHARGE, &stock->flags)) { if (cpu == curcpu) - drain_local_stock(&stock->work); + drain_local_stock(&stock->qpw.work); else if (!cpu_is_isolated(cpu)) - schedule_work_on(cpu, &stock->work); + queue_percpu_work_on(cpu, system_wq, &stock->qpw); } } migrate_enable(); mutex_unlock(&percpu_charge_mutex); } static int memcg_hotplug_cpu_dead(unsigned int cpu) { struct memcg_stock_pcp *stock; @@ -7956,22 +7958,22 @@ static int __init mem_cgroup_init(void) * used for per-memcg-per-cpu caching of per-node statistics. In order * to work fine, we should make sure that the overfill threshold can't * exceed S32_MAX / PAGE_SIZE. */ BUILD_BUG_ON(MEMCG_CHARGE_BATCH > S32_MAX / PAGE_SIZE); cpuhp_setup_state_nocalls(CPUHP_MM_MEMCQ_DEAD, "mm/memctrl:dead", NULL, memcg_hotplug_cpu_dead); for_each_possible_cpu(cpu) - INIT_WORK(&per_cpu_ptr(&memcg_stock, cpu)->work, - drain_local_stock); + INIT_QPW(&per_cpu_ptr(&memcg_stock, cpu)->qpw, + drain_local_stock, cpu); for_each_node(node) { struct mem_cgroup_tree_per_node *rtpn; rtpn = kzalloc_node(sizeof(*rtpn), GFP_KERNEL, node); rtpn->rb_root = RB_ROOT; rtpn->rb_rightmost = NULL; spin_lock_init(&rtpn->lock); soft_limit_tree.rb_tree_per_node[node] = rtpn; From patchwork Sat Jun 22 03:58:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leonardo Bras X-Patchwork-Id: 13708199 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CDFCAC27C53 for ; Sat, 22 Jun 2024 03:59:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 603CC8E0001; Fri, 21 Jun 2024 23:59:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5B3248D01AF; Fri, 21 Jun 2024 23:59:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 406348E0001; Fri, 21 Jun 2024 23:59:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 1EC848D01AF for ; Fri, 21 Jun 2024 23:59:08 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id C72A1C132D for ; Sat, 22 Jun 2024 03:59:07 +0000 (UTC) X-FDA: 82257169134.08.AEE2C77 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf23.hostedemail.com (Postfix) with ESMTP id B483114000C for ; Sat, 22 Jun 2024 03:59:05 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ZqB8sRot; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf23.hostedemail.com: domain of leobras@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=leobras@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1719028733; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=gr/4mKEcg1BO/1nqaoL0eifI3AwTZgGcUhIFK99x5ho=; b=ufNxrF33YNneG9wJqM9Wg9hpI6S388bhm9h45WhF0ds2EHOKqCTnsbnE+TOTZOrP/kzj54 BKq2oswNn9qJC70JAUKAEYzIkF/uE2k2qwXU4Kq8umu5TgtypH2qZWpUOGVLcoebBho/Dh gPVxi1PNHe1kw8xm0iS4jlDTxeg3QM4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1719028733; a=rsa-sha256; cv=none; b=ByMQ/PlaMiNVA9qRAggPiQ43AqT1XkOJ8wXtPEEV3paPmCQm39RzDNc5CM/C2d41T4O+Xr dqUYInulOHoNM7X53hrYYGgFpoervVjHQCN8tANCja+T+HVv6uB52a66ziNB3lVHcOcQz4 IRByhnVNPu4COFERASDACfmTakq1CqY= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ZqB8sRot; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf23.hostedemail.com: domain of leobras@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=leobras@redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1719028745; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gr/4mKEcg1BO/1nqaoL0eifI3AwTZgGcUhIFK99x5ho=; b=ZqB8sRotdJfAI1qLOxA+hvDct0foVVmDBnBXsrKbFloTTQ5e4O4Ebg2o1MT5uHtarPU0Tc nxKvvRJvMZmbbYq9aQrIk4tlAAKeWmUhQbpaDkv710cguXpCas/IwOxKVaJKftu/rcWON+ Hi1aRlvnrVIeMGRHXimpiDiCFHeRXGU= Received: from mail-pj1-f69.google.com (mail-pj1-f69.google.com [209.85.216.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-530-aQPd91bAP5WeiR42VgUZuw-1; Fri, 21 Jun 2024 23:59:03 -0400 X-MC-Unique: aQPd91bAP5WeiR42VgUZuw-1 Received: by mail-pj1-f69.google.com with SMTP id 98e67ed59e1d1-2c7a6c639bfso2854904a91.3 for ; Fri, 21 Jun 2024 20:59:03 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719028742; x=1719633542; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=gr/4mKEcg1BO/1nqaoL0eifI3AwTZgGcUhIFK99x5ho=; b=Xbkr6ZR4oSd8CX95uA8Dxc2kCm4hLCtjhZYPoTJN9yX5BCeWsrCsXmkyzytobdUOWa 1yxhoQT1Fwf3AU0g5H08HVRy2KtZYmGe7xWYyz/0WdxYJOVj7NDg40AbVg2FDXtT7GWp GO5UIs7gKnuAlMA60nYTtSPdb94RwssscVDcdj7teUrtxaUhFlpsnsZwzQhwhgsFGRxs uMznJniIwnRzVDuj4kKLROZ20yuaL0felEiq7vKpqJc60npvngbpKU9M4mnw8MsMombF wewgBT/pfZ61wPvoxCUHS9AlDkk9JxBt5Pa4GbaZPl1m8QJdVMjVq6WnjfjbQrf0m67n KNIg== X-Forwarded-Encrypted: i=1; AJvYcCUY8IF/x0m9sBHKhs8nJMQznDDtQfcp3ez1qHMjQllKQTLRm9Ew9K+bijBETpTLaDUeI/XMlNJhBgmdBxjqL5CIZZs= X-Gm-Message-State: AOJu0YzEx3tLj1m+zOBjiB7W2l1nlOP3PJFZmoYGvvGeI8s7xhzvXAJF ReaJFQ8dqx1/3VAerc+7b0fD8utwKgECDU7bcgcUAcHENRQLJdUf3DQMFsf1DCoTdDmwBmGqQJf Cic2I+M82rUj09HjuC7zNhZwPGBD2hx9iHFy1pllStcXimI+S X-Received: by 2002:a17:90a:6006:b0:2c3:cc6:636e with SMTP id 98e67ed59e1d1-2c7b59fa68cmr10387330a91.2.1719028742279; Fri, 21 Jun 2024 20:59:02 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHbqsAmN/2usMDnEbfUaqXha6LBh1OFap3JUlsvia0ZFkQTobQ3J2rvz24LicWySb2pIwxdrA== X-Received: by 2002:a17:90a:6006:b0:2c3:cc6:636e with SMTP id 98e67ed59e1d1-2c7b59fa68cmr10387308a91.2.1719028741834; Fri, 21 Jun 2024 20:59:01 -0700 (PDT) Received: from LeoBras.redhat.com ([2804:1b3:a801:c138:e21d:3579:5747:ad1]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-1f9eb32b9edsm21832365ad.118.2024.06.21.20.58.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 21 Jun 2024 20:59:01 -0700 (PDT) From: Leonardo Bras To: Johannes Weiner , Michal Hocko , Roman Gushchin , Shakeel Butt , Muchun Song , Andrew Morton , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Vlastimil Babka , Hyeonggon Yoo <42.hyeyoo@gmail.com>, Leonardo Bras , Thomas Gleixner , Marcelo Tosatti Cc: linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, linux-mm@kvack.org Subject: [RFC PATCH v1 4/4] slub: apply new queue_percpu_work_on() interface Date: Sat, 22 Jun 2024 00:58:12 -0300 Message-ID: <20240622035815.569665-5-leobras@redhat.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20240622035815.569665-1-leobras@redhat.com> References: <20240622035815.569665-1-leobras@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: B483114000C X-Stat-Signature: 55nzjhkne6fwnaxgttm87ms3anq7an1c X-Rspam-User: X-HE-Tag: 1719028745-45734 X-HE-Meta: U2FsdGVkX18Bb+N0Uq1VpiS+mbywQUNItff3zE2qJZJsNPMtQ6FXaneqi5E6Oe9FsShkHDawSXe0xkfobYek6wSioWMi1AW5FCM3dYcRXv1B4e+rv72SjWXMysY2pQ0LwbAH2aiyALCSXvQtz+/GUVpM6ZbsScGMaRMJUFGMf5Ro2SUmGXwisSDLwVw1eqEoKbkHFPmc+QoRDQtuT3wazaLJz7HVjs8jZGxDi3xH0/oI7RA2Hbk0dT/imD5HRWDzpNuoSljSArZHqxQvR3+wrGE0Rjn5Wp/I0rtcj20L1IoEMJcD1vjjnu5q2n/O5WqAiprQHcLIN4YAl9/h1vphUj8u/Ruu1D3HDNCNZgwtnN8t+gC035dy37bPfT5n420XsbWAFtLfK2bs0kPD0RYUy8UTZHRzRxawo+cFdiSPyXm+Ft94dXFVg/xHHuZ5lOMluXNlXk6uSmAqiEktZhnS417JQJ/vOtEQKSs8vBdu5E2NgJiG6t1ftimfQOQnB/hTnlHTLdJM+7tzWYoLCn5rblUmXRmAsUEpMb/Z6rhHrTV2bPwPXA827pHW+iinrHAEH7WU7gqwN9XP5CpsbOlwMLSOUxh9G/kSg4LrKxQDWfit3v0aiBGfFesNuf5COTtzWCPq41leuu6po1styx8Qs3wOJ2S9w4634i6NV/jYaXDQr3xS3ov51MHC6USYuMrzPivixo5aYvtNvbQ1d3VfIf5JuAB9RUi4exAUQkN6Kl4fGfxIlAKmE03TRfUX+fM3eLD1FfKg6ytw914LLUCXpzjbjyu0irNETFV/uDb45KRBMBxRyZh9RjBxIfWWVY05W4elqReR6oM2bZFwE7p91KbSOgZpjFp94t1Z9aysxCw2X13kkp+0RmYPVuFfvl374D3D75VMKAqlOI9xQ0w7Q10lWZnRPnx2P0aPrZ6gUM+fpkUlUVjnElFU+w7k+SHltAAX+4djj+MpNPf4nyM 5mhslcuw hzSAyKFau6pqImjg79Dr1mNK1kywiqLAgg4b3XcMgQHKZYAkz+ZgEbBH7RD0ziVJnTMZm3qJH/yxD82AMgt4q9VvNDnzMa30k5tdTRckNyTNOyFpEJztV93E18xNji1nbffoSVLcVtx0f7V2UDDaWC8oOGkvktRxUtxo1B9KPToGJXJm5lvNzFYQnQg2N35TsVz7OZekeVX2y//kL8C/6twl9L+3kImQiFMsA5IuwRO9z06RTOiLzSH+Ubf+XHD3Yy1/TE4xi5RVWvIk+Ucb5RckkwP39kXHubkupEOdIg4zlqNWnGZIL5e//uxzKeSB382hXCGj4BfWioa1ZgMcFGdaZ6oBEkN2CIlpKaoUEVyzKixnkJvSDz//EwiWdeLn53aISbheg/FKydJ+nlioPD1j9kJQrrrpQJRoL5UBovcR504Ab9TU32+a2Soxb5KH56Wf7bSjH5jDqiK0PwArgyWg2og== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Make use of the new qpw_{un,}lock*() and queue_percpu_work_on() interface to improve performance & latency on PREEMTP_RT kernels. For functions that may be scheduled in a different cpu, replace local_{un,}lock*() by qpw_{un,}lock*(), and replace schedule_work_on() by queue_percpu_work_on(). The same happens for flush_work() and flush_percpu_work(). This change requires allocation of qpw_structs instead of a work_structs, and changing parameters of a few functions to include the cpu parameter. This should bring no relevant performance impact on non-RT kernels: For functions that may be scheduled in a different cpu, the local_*lock's this_cpu_ptr() becomes a per_cpu_ptr(smp_processor_id()). Signed-off-by: Leonardo Bras --- mm/slub.c | 26 +++++++++++++++----------- 1 file changed, 15 insertions(+), 11 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 1373ac365a46..5cd91541906e 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -35,20 +35,21 @@ #include #include #include #include #include #include #include #include #include #include +#include #include #include #include "internal.h" /* * Lock order: * 1. slab_mutex (Global Mutex) * 2. node->list_lock (Spinlock) @@ -3073,36 +3074,37 @@ static void put_cpu_partial(struct kmem_cache *s, struct slab *slab, int drain) } #else /* CONFIG_SLUB_CPU_PARTIAL */ static inline void put_partials(struct kmem_cache *s) { } static inline void put_partials_cpu(struct kmem_cache *s, struct kmem_cache_cpu *c) { } #endif /* CONFIG_SLUB_CPU_PARTIAL */ -static inline void flush_slab(struct kmem_cache *s, struct kmem_cache_cpu *c) +static inline void flush_slab(struct kmem_cache *s, struct kmem_cache_cpu *c, + int cpu) { unsigned long flags; struct slab *slab; void *freelist; - local_lock_irqsave(&s->cpu_slab->lock, flags); + qpw_lock_irqsave(&s->cpu_slab->lock, flags, cpu); slab = c->slab; freelist = c->freelist; c->slab = NULL; c->freelist = NULL; c->tid = next_tid(c->tid); - local_unlock_irqrestore(&s->cpu_slab->lock, flags); + qpw_unlock_irqrestore(&s->cpu_slab->lock, flags, cpu); if (slab) { deactivate_slab(s, slab, freelist); stat(s, CPUSLAB_FLUSH); } } static inline void __flush_cpu_slab(struct kmem_cache *s, int cpu) { struct kmem_cache_cpu *c = per_cpu_ptr(s->cpu_slab, cpu); @@ -3115,82 +3117,84 @@ static inline void __flush_cpu_slab(struct kmem_cache *s, int cpu) if (slab) { deactivate_slab(s, slab, freelist); stat(s, CPUSLAB_FLUSH); } put_partials_cpu(s, c); } struct slub_flush_work { - struct work_struct work; + struct qpw_struct qpw; struct kmem_cache *s; bool skip; }; +static DEFINE_PER_CPU(struct slub_flush_work, slub_flush); + /* * Flush cpu slab. * * Called from CPU work handler with migration disabled. */ static void flush_cpu_slab(struct work_struct *w) { struct kmem_cache *s; struct kmem_cache_cpu *c; struct slub_flush_work *sfw; + int cpu = qpw_get_cpu(w); - sfw = container_of(w, struct slub_flush_work, work); + sfw = &per_cpu(slub_flush, cpu); s = sfw->s; - c = this_cpu_ptr(s->cpu_slab); + c = per_cpu_ptr(s->cpu_slab, cpu); if (c->slab) - flush_slab(s, c); + flush_slab(s, c, cpu); put_partials(s); } static bool has_cpu_slab(int cpu, struct kmem_cache *s) { struct kmem_cache_cpu *c = per_cpu_ptr(s->cpu_slab, cpu); return c->slab || slub_percpu_partial(c); } static DEFINE_MUTEX(flush_lock); -static DEFINE_PER_CPU(struct slub_flush_work, slub_flush); static void flush_all_cpus_locked(struct kmem_cache *s) { struct slub_flush_work *sfw; unsigned int cpu; lockdep_assert_cpus_held(); mutex_lock(&flush_lock); for_each_online_cpu(cpu) { sfw = &per_cpu(slub_flush, cpu); if (!has_cpu_slab(cpu, s)) { sfw->skip = true; continue; } - INIT_WORK(&sfw->work, flush_cpu_slab); + INIT_QPW(&sfw->qpw, flush_cpu_slab, cpu); sfw->skip = false; sfw->s = s; - queue_work_on(cpu, flushwq, &sfw->work); + queue_percpu_work_on(cpu, flushwq, &sfw->qpw); } for_each_online_cpu(cpu) { sfw = &per_cpu(slub_flush, cpu); if (sfw->skip) continue; - flush_work(&sfw->work); + flush_percpu_work(&sfw->qpw); } mutex_unlock(&flush_lock); } static void flush_all(struct kmem_cache *s) { cpus_read_lock(); flush_all_cpus_locked(s); cpus_read_unlock();