From patchwork Thu Nov 7 20:20:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13867065 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9BBD3D5D689 for ; Thu, 7 Nov 2024 20:20:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AA4E56B00AC; Thu, 7 Nov 2024 15:20:48 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A53856B00AD; Thu, 7 Nov 2024 15:20:48 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8CFC16B00AE; Thu, 7 Nov 2024 15:20:48 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 648E86B00AC for ; Thu, 7 Nov 2024 15:20:48 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 05FDFC0268 for ; Thu, 7 Nov 2024 20:20:48 +0000 (UTC) X-FDA: 82760416494.15.54AEE03 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf24.hostedemail.com (Postfix) with ESMTP id 3AA5718000A for ; Thu, 7 Nov 2024 20:20:41 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=vSvDbcN1; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf24.hostedemail.com: domain of 3HSEtZwYKCFQKGL3wA2AA270.yA8749GJ-886Hwy6.AD2@flex--yuzhao.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3HSEtZwYKCFQKGL3wA2AA270.yA8749GJ-886Hwy6.AD2@flex--yuzhao.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1731010675; a=rsa-sha256; cv=none; b=HFnptqauNCej3zxUDCOerMgnisewSdL4uZ+juYjh+inMBj5uek6Hj/hOSiOHBY0CyKSHwB oBE60S1APJZpNmZXjMLeldSUXz4bpc8XFcRvmgquMvxdEx6Fl32OQsl1YQFx95Zv5i5M8Q 0EdIX2fxrj2U1asl0ZrWmcAxWCdUUro= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=vSvDbcN1; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf24.hostedemail.com: domain of 3HSEtZwYKCFQKGL3wA2AA270.yA8749GJ-886Hwy6.AD2@flex--yuzhao.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3HSEtZwYKCFQKGL3wA2AA270.yA8749GJ-886Hwy6.AD2@flex--yuzhao.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1731010675; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=v35HyYfzjB2bG4AOilBZdKULCbIxbvNUBJLvIql+6B0=; b=kxUXePf5fj6m3Z0xD1RT/2LksXBK4bWap6mkbbM/uhpr384GVzBrML7Vlne69q586HtEfq q4zwa+To0JZwtTVjPedxoH68E4+BUfxVV6OWeYPu5191L4d/GDEwn1EVVo0cSkasqojcJU Mbg5xBEIEvQgzVIigmXg6EWZ77mkVXk= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-e293b3e014aso2109924276.3 for ; Thu, 07 Nov 2024 12:20:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1731010845; x=1731615645; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=v35HyYfzjB2bG4AOilBZdKULCbIxbvNUBJLvIql+6B0=; b=vSvDbcN16Tw8D5YKTR4uXJvFiQjKRApzF1v2qLWpd6dsGnoC6SVHB8wjL6rHHadI90 jZ0tc36+DTwW+OTJW6d/CudSVYKc2uOQiwKRTNwOILpqByXfn5OXDMPP7lzUJYIdkqeS O63jl9dt0CkJiWM2p/r7GNIDQHL903347BYcMkgWbwceTeaEYYVVfcyxff8Fa7HxtXQA NNZWVQTWkm/tNW/iUhsTLfae7g+Nya1Y3d2AM+y0mk3F6BH7NMrJKFjoQQTL2VGlhqqb oqX7PTD4VkA2ILIZ81KR1P6NnU0eJyW0gsxAXaTcgyzotrcO4P95H0gM4nSSLM4hYvK5 YFpw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731010845; x=1731615645; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=v35HyYfzjB2bG4AOilBZdKULCbIxbvNUBJLvIql+6B0=; b=GRkQsoIKnqEq13gN0yN0j0SXdqhLkMWAdy6h48mGMIcQWRv3YpTTkG6AkdPM2iyywm FxbNDBo6rAxRqdARmEy/sfscq/cMARA1RHEwCBcFW0MU/DwCSdwy+Dxjl+LUJFlxpscU 8X0/lpcPRki7S4U/9+D/n51QzyEpQqqtUbYcfpu6kFR+zajZcU4qTNiWLzXhTTcYNy9A iz/Mk/3lExq63fRg6fNfY+R694KzPOAHlMgYuenEcLbZ1tvEhexmbyPc61T9HALlmTzy gZgDzn776lUBcZ/jATmmlnxuK9ZmmcCnn9A1+MIze4qie4sJOrPN3WsqnokTTJucoia6 1rQA== X-Forwarded-Encrypted: i=1; AJvYcCXCHgWD3mkN56OsXAdrMRcn6U5Fq5k0I6+N1+yRw40I2OH41XkOygBdAIVIDgv+OKMGnuNBoe4OsA==@kvack.org X-Gm-Message-State: AOJu0YxjkAzGfnzzARYUdkcxcVVfbhaedvot1Owy62EIYNSGlLN1jkSI 2jzfiAZAc68Q87c6XvDkosi/F3oxsnl5EGMF2gQ9G0Td7WDC7L10MSHBRTXJU2GKdnFOTPNaAfR mEQ== X-Google-Smtp-Source: AGHT+IH5+g5PrNVgp5Pr5OTYOBdUbFjFulnS4uSn6ZxFeBhddvMAobpkpn0NbUnR+mioP6ARIrKUFKFKqUg= X-Received: from yuzhao2.bld.corp.google.com ([2a00:79e0:2e28:6:a4c0:c64f:6cdd:91f8]) (user=yuzhao job=sendgmr) by 2002:a25:a249:0:b0:e28:e510:6ab1 with SMTP id 3f1490d57ef6-e337f8faff5mr210276.8.1731010845128; Thu, 07 Nov 2024 12:20:45 -0800 (PST) Date: Thu, 7 Nov 2024 13:20:30 -0700 In-Reply-To: <20241107202033.2721681-1-yuzhao@google.com> Mime-Version: 1.0 References: <20241107202033.2721681-1-yuzhao@google.com> X-Mailer: git-send-email 2.47.0.277.g8800431eea-goog Message-ID: <20241107202033.2721681-4-yuzhao@google.com> Subject: [PATCH v2 3/6] irqchip/gic-v3: support SGI broadcast From: Yu Zhao To: Andrew Morton , Catalin Marinas , Marc Zyngier , Muchun Song , Thomas Gleixner , Will Deacon Cc: Douglas Anderson , Mark Rutland , Nanyong Sun , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Yu Zhao X-Stat-Signature: b9ukwj7nr1dz5affe8pmwt6pjuhm4j56 X-Rspamd-Queue-Id: 3AA5718000A X-Rspamd-Server: rspam08 X-Rspam-User: X-HE-Tag: 1731010841-957065 X-HE-Meta: U2FsdGVkX1/2s3b8TIRpKtHQvz2GjdQ2O7d33ufps6doVTiD+d2nTNHDX6X8wAAub9JLXRZUyzcr54rEt7WUNUXzAXx/3LYodZ5UN8xv7i4tA6quPwM0yYSyiSJOBGdoueNJHszBA/QmZGOE2KAD8Hs/DJURLFDZ5tV2r7KtIyVey1XGbD/sIwsQV8zv6BVAhtiOdP6kwHzvcT8OXpHUcBZJ3EoEW1qEtSIdAFkliG43XOj9G47ck2bOXeboVg5w5DRSUnWe9EJm86vIBEFXNeXZ6Yyosf4uSvluRCLlcd2z7pBaMgw6CKseBn1l5QitlhBMWj24iKw7EwPK20kKF1Oog/s20gC67Pn5bc7XFwKoSu4m3iUW7ef7VjgOzaJlN0OmgrccAu5CzLx8EPUc7tv8PhmyqsUHCdJGGtzjnTBPn2H7YGl2IzrYJHLGZburPZ+Cc3uzzoBusKaCVXz5UgmddConuvyUL26+9Xu9KB7AtAxP5biujUXnze0CVx5YXbXQqeLi1G2XJY+mvOQVovCXJb4XHlTHaDAX4A1WzdJ9di+Chj5884kEBlGJIr0sv3R0RdfX49YAQiSiwT3mdLdP7c/UHhQ5DLIMJ2N5wEglBAkgReb4KGRRnOjFwX/yn/5kM/R5+obYjgGbQL8U9l30Wx6Fs3sQTF3NRu4sD+mIMG4ulMM32QoxHmDndf6Xkb8CKrsAk7OMnKdaOTNkBnB7suzlrL+n7qB/hViUAQzmDDL/XXtfASwY9BJp/xCOsbIjfR3jdP/2SfC16wOFsDUzeQGIOFQwcJxwiam3wU386oaMoO9Rk5NVYLr9rAOBOeTeDjO0SKhOxmkZ4h19RSlL8F2jkAXTDFWXwUZf3fY1ymohTTXx76rlxCbR5FIUTJKqJSOXGngkCyn4M7McIUXMALhp4rkm6oubuirrCySBJ5/ARGKG99WASOiDi5JWkgT8inmju7iLch35aP8 hRn30Gjk 5W7weBbEKZkgvq14x8ia1wibfnzl1e2Eh2JtZfp+xQfngubfIiWpeaoD/bVsa21IG46kittW1paj5Un9Hl0es5UyQHNSoiWYOStH86On4HzdnmZXTEFcb+hhpRCuqIFtXbFONEGZ/q16eaWaX2skiAZaOtUq9nNLW52RtZgx1ni/Jpp09A5dMlMLQ/kGrgufmuagnRhyoZs/jRMoTpenmtpEfho5DHDbW2RhTndrpKy3TCalhL2u1Y7+6Iab6QxDlbBeYeNVajJklyw4DY5gC28JkHzJ5I6CxAEQweZzjio9STP4s+CJNkRbOoeylprtqq9sm5Y6dvqsbEI2qDPEeCXF+bpnEEvGjX5yv4ppqJeRoYRy9bI+ervJQmSfTWhc/ZTbxmijK15Bn6+I+MAxSi4R1U46McHqexbgBW7ZivBu1IU8DrRGUR/21Y6LabYA/IdCG8JQiMWQPdXw1JKIjbkJOZ8k8pNmzj0dViweD5xBeRqB27MvTALbbcc5orMMFCRru8Y3je/hM1nCQwCGz/1/38cAhstgTRlyqkpuky0VIai8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: GIC v3 and later support SGI broadcast, i.e., the mode that routes interrupts to all PEs in the system excluding the local CPU. Supporting this mode can avoid looping through all the remote CPUs when broadcasting SGIs, especially for systems with 200+ CPUs. The performance improvement can be measured with the rest of this series booted with "hugetlb_free_vmemmap=on irqchip.gicv3_pseudo_nmi=1": cd /sys/kernel/mm/hugepages/ echo 600 >hugepages-1048576kB/nr_hugepages echo 2048kB >hugepages-1048576kB/demote_size perf record -g time echo 600 >hugepages-1048576kB/demote" With 80 CPUs: gic_ipi_send_mask() bash sys time Before: 38.14% 0m10.513s After: 0.20% 0m5.132s Signed-off-by: Yu Zhao --- drivers/irqchip/irq-gic-v3.c | 31 ++++++++++++++++++++++++++++--- 1 file changed, 28 insertions(+), 3 deletions(-) diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c index ce87205e3e82..7ebe870e4608 100644 --- a/drivers/irqchip/irq-gic-v3.c +++ b/drivers/irqchip/irq-gic-v3.c @@ -1322,6 +1322,7 @@ static void gic_cpu_init(void) #define MPIDR_TO_SGI_RS(mpidr) (MPIDR_RS(mpidr) << ICC_SGI1R_RS_SHIFT) #define MPIDR_TO_SGI_CLUSTER_ID(mpidr) ((mpidr) & ~0xFUL) +#define MPIDR_TO_SGI_TARGET_LIST(mpidr) (1 << ((mpidr) & 0xf)) /* * gic_starting_cpu() is called after the last point where cpuhp is allowed @@ -1356,7 +1357,7 @@ static u16 gic_compute_target_list(int *base_cpu, const struct cpumask *mask, mpidr = gic_cpu_to_affinity(cpu); while (cpu < nr_cpu_ids) { - tlist |= 1 << (mpidr & 0xf); + tlist |= MPIDR_TO_SGI_TARGET_LIST(mpidr); next_cpu = cpumask_next(cpu, mask); if (next_cpu >= nr_cpu_ids) @@ -1394,9 +1395,20 @@ static void gic_send_sgi(u64 cluster_id, u16 tlist, unsigned int irq) gic_write_sgi1r(val); } +static void gic_broadcast_sgi(unsigned int irq) +{ + u64 val; + + val = BIT_ULL(ICC_SGI1R_IRQ_ROUTING_MODE_BIT) | (irq << ICC_SGI1R_SGI_ID_SHIFT); + + pr_devel("CPU %d: broadcasting SGI %u\n", smp_processor_id(), irq); + gic_write_sgi1r(val); +} + static void gic_ipi_send_mask(struct irq_data *d, const struct cpumask *mask) { - int cpu; + int cpu = smp_processor_id(); + bool self = cpumask_test_cpu(cpu, mask); if (WARN_ON(d->hwirq >= 16)) return; @@ -1407,6 +1419,19 @@ static void gic_ipi_send_mask(struct irq_data *d, const struct cpumask *mask) */ dsb(ishst); + if (cpumask_weight(mask) + !self == num_online_cpus()) { + /* Broadcast to all but self */ + gic_broadcast_sgi(d->hwirq); + if (self) { + unsigned long mpidr = gic_cpu_to_affinity(cpu); + + /* Send to self */ + gic_send_sgi(MPIDR_TO_SGI_CLUSTER_ID(mpidr), + MPIDR_TO_SGI_TARGET_LIST(mpidr), d->hwirq); + } + goto done; + } + for_each_cpu(cpu, mask) { u64 cluster_id = MPIDR_TO_SGI_CLUSTER_ID(gic_cpu_to_affinity(cpu)); u16 tlist; @@ -1414,7 +1439,7 @@ static void gic_ipi_send_mask(struct irq_data *d, const struct cpumask *mask) tlist = gic_compute_target_list(&cpu, mask, cluster_id); gic_send_sgi(cluster_id, tlist, d->hwirq); } - +done: /* Force the above writes to ICC_SGI1R_EL1 to be executed */ isb(); }