From patchwork Mon Oct 21 04:22:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 13843542 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 593E0D3C933 for ; Mon, 21 Oct 2024 04:22:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CD5126B008A; Mon, 21 Oct 2024 00:22:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C83C46B008C; Mon, 21 Oct 2024 00:22:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AB0BE6B0092; Mon, 21 Oct 2024 00:22:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 746466B008A for ; Mon, 21 Oct 2024 00:22:33 -0400 (EDT) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id C7F2AA145C for ; Mon, 21 Oct 2024 04:22:05 +0000 (UTC) X-FDA: 82696312692.04.A661831 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf01.hostedemail.com (Postfix) with ESMTP id A5D4340011 for ; Mon, 21 Oct 2024 04:22:19 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=LR76z8Fp; spf=pass (imf01.hostedemail.com: domain of 3BtcVZwYKCEsB7Cun1t11tyr.p1zyv07A-zzx8npx.14t@flex--yuzhao.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3BtcVZwYKCEsB7Cun1t11tyr.p1zyv07A-zzx8npx.14t@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729484501; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rF+TQYGEDP6+t2V2Dte+w8ImgbG43FZ1I4yOJyMV9jY=; b=0eHllPlWoiqM5KnpjZrnOUwRuDCAzeKir1jCuK564AJSFA2eSwDa9+OFC7538nn2KQGMGQ pprbCJUiTX9g/Ewf9OOHhYAM5ejmMk+RbhJRz2cGnnIe911gKKO+fXtwFuSfCqfRey94O6 tpb0QE9t+puWJ9AJksKhWDKctzhrH2Y= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=LR76z8Fp; spf=pass (imf01.hostedemail.com: domain of 3BtcVZwYKCEsB7Cun1t11tyr.p1zyv07A-zzx8npx.14t@flex--yuzhao.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3BtcVZwYKCEsB7Cun1t11tyr.p1zyv07A-zzx8npx.14t@flex--yuzhao.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1729484501; a=rsa-sha256; cv=none; b=m9CF9TmDLDqjT2Je7xNBhSLdmGINe8AW74ie5NCIGnvyr0/pyOYSDDP0mnIkh71CTz4yD9 A9EoHrA/BBxn0/VdD3/9gWztr0sjuc48FoXEnVxfXkMonC7EgN1EUuxmTPjFWvAgkYvilG fKjpHZLObZCIPA64zoRSJ08hDQArrlA= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-e2939e25402so6261955276.2 for ; Sun, 20 Oct 2024 21:22:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1729484550; x=1730089350; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=rF+TQYGEDP6+t2V2Dte+w8ImgbG43FZ1I4yOJyMV9jY=; b=LR76z8FpVonfntEgmlk0BnpHEik25crT+toL5oBeoQITMMVe6PNoa+8EQ5hUTcJm4U HyWYzkPpI+RLO5dQaBJRxOnWj7RPhwBGy8nHuC22SEmwtJ6C04TSmuT6RD64w0vlzxuC bg8EIfDzC3dEES4PnT+lZMHUOT+c3+mrA71CiFiY9hR8dazp3f0Zk4SIxyowQ0CyzeRQ 0XbYhmb1aY3PP7uhkPS4xh7Unm29RW07mIRN6NSksS/SL11/Pwsre3eQvk18BXDb59Ii AZwXWuxPc+HGTa3l4snxUH4NpMyxUQzrPdNYMJ0ZQDUgycBw9Jmql7UkmRuYYhsvlt9C Os4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729484550; x=1730089350; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=rF+TQYGEDP6+t2V2Dte+w8ImgbG43FZ1I4yOJyMV9jY=; b=vSNDd+8qZSJ3rYH54FLADXwu1ZpqSm+P4NfTDGQsqm5m1JjitJmm1vK6Qll8s7vnDA dIr/DBT+8sARGIuCkAayXW5wZ0hhDpZHfMFC3wvC2wz8XR3FoTICW9yeaCL8f11VKf3c 5RacCnf3zdb7zf8xvu61MsCL54Du7lwqEyBvB/5C7yUt3BYujgznQhE2qeqS7yR6c4W7 RGaSWQdY2jW/CvrHoj+wkjHkFpkx3gip6ZRLQbwRQMXBnT4DHI5NwtQNMvbesDq1NVGE RU0u1b6/1/PAc106KNZ2EyDc52wpv4D9YMVLrTXk9k5KB4p9bBaJ2nNgYaOda3zg2tVM mSBA== X-Forwarded-Encrypted: i=1; AJvYcCWCkmTKXdN1O1y2CQnotBthS/SMJY9pXoUy+ur0JXtu6sksCUVoM36K5Zq+L202BxEDIfI/wJOoWQ==@kvack.org X-Gm-Message-State: AOJu0YzvDVPJM7Lz9wtGaDjGfvv4v7XwWd7KdrkbjHoNhKNGgpgZUM8c 5+0wPwPZXKblFjP79+VdziuukKTVvdQiHjsN9JEbpNUjPGzBUrqCxRiHoJ2j5iZxYM9QwXwx3Jb SpQ== X-Google-Smtp-Source: AGHT+IEeIIe5Xg98lfzs9VJWYOJ07F6ek87y8El+d0cpjxlVvqWrNUTXfkoh0Qdzc1+S/vFogSJ2meafeF0= X-Received: from yuzhao2.bld.corp.google.com ([2a00:79e0:2e28:6:1569:9ef4:20ab:abf9]) (user=yuzhao job=sendgmr) by 2002:a25:800f:0:b0:e05:6532:166 with SMTP id 3f1490d57ef6-e2bb11cccaemr14439276.1.1729484550141; Sun, 20 Oct 2024 21:22:30 -0700 (PDT) Date: Sun, 20 Oct 2024 22:22:15 -0600 In-Reply-To: <20241021042218.746659-1-yuzhao@google.com> Mime-Version: 1.0 References: <20241021042218.746659-1-yuzhao@google.com> X-Mailer: git-send-email 2.47.0.rc1.288.g06298d1525-goog Message-ID: <20241021042218.746659-4-yuzhao@google.com> Subject: [PATCH v1 3/6] irqchip/gic-v3: support SGI broadcast From: Yu Zhao To: Andrew Morton , Catalin Marinas , Marc Zyngier , Muchun Song , Thomas Gleixner , Will Deacon Cc: Douglas Anderson , Mark Rutland , Nanyong Sun , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Yu Zhao X-Rspam-User: X-Stat-Signature: qfzibruy4thsbc3ecai88u3wr8ab6msj X-Rspamd-Queue-Id: A5D4340011 X-Rspamd-Server: rspam11 X-HE-Tag: 1729484539-845459 X-HE-Meta: U2FsdGVkX1+W9APWGIaXvDLLfTh4IGertmIYpR3IsBs3rAA0F+X3eihuHHmsY3weUNvYP4rlWk5qYVbRWgzxGnLUAuSIYhDOuJV9qrM1WwEsBxIDHXMUNgW4nSnoIVJ4TDm7eSmkYCKMoUm3Y/nMJGW1jwJFUtn0h6Teomyty5L+a+52BERq+OvPU6jQJlvfef37f9ptkQ5rcGHO+uLUGsfaiigok+B10g+0SJnr4Fv9ZD/pMPmPv7i6O/4kDi/VLQB4ar+s9Rdnz3DwjUxzl+FwAeABDrE/WWwmu+vh5XGGIS1Ihf9DBHeKPq96UqdYR2WCLJ0HrBpn+e2g6MRrrzBO9tc2weHysOXb14VgPpoPba+7dhXwpsjZAQiZ0/acprxCiK3Nk5lIXdwPJl1R1NInKTV0FXnZ0RlnAwPg4kax1zMAe0Qfq0KEQdB2BbM+zFL3lUeAAbrovSGKjCRbNfkhuOrwTt67HKqa+HMXTc1wjPQlpgzSYZ+To0DQc/tlAmM21YcB+UJbKAHejwotpiz8QcM5wXlv52z/N15B9Wh3wrqs4vx3UCpOfRG4KaT/Bgb0P2pSeuCFXW+9hCPn6g1YXUmsp6GfBn3Hq+XvaqZPVz0uWz1S7CQQD33zsop6BPXzhWKHrh0By6XnD4UdCyOkKHjAL9oG0AZz9KkOEQOmvsDChlSR1dmxrSOOcn/VgaH0ShD08ygop8V83tybDPMaFAfdAZ4M5jybYMdK9td1gsyCMptMfZalnS/HYyYR2vffo9SkUDOru+C83nz0y1xVccZBj2y+16L+7uqZn9r0+3rHl9g9h/GhjT2+nwOEqW1wANr9tbm+Qqd3v2sFtZiAClOSX5TeRWxjZFHafx91y7SYvVFfTdFPGRPfeNOpTeVY/fKrqvq9+vnCeOm64k8mCdVnIgioFybBiecibWChKn8ggHcPpfG0OSQXgS8QsezCWS/FMRPrBABsyiF NhOk9434 F+W0Q4t/CZQLVTBblB1/3vD2oQmFHVjrBVUpxcolMS72s3YaSScTlD6R1GCcSEzLerVp6GjblAUutOxq4xJSv99/3xvk2AxlNGvsiReMa2O4VHsx+mpqz1zoNKez4FNtJ3+gihdvsc32bqTsOTGUnUKhZj4gAtG4Pvlt+SduuMlz2gYu+Yn3yQh2BDm2alpXdfoJRrEiQFF+NflrM/XkCn3jj8F9I9exAp5zoy3gQBDPUHSOqT7bjbItLPCpQntEtP4m60RTueFP5Ys+kEzCgK44Whm6w3uR8+0WIW5WPczrz8gAlODw4OyKFUZEoJEmqS/NM1cWVbiQET9oJShi8UyLeHytM+AIvuxGR+lq9h4S4JrMimhcWGN/5CKznV4bLNDFgyMhD18zqhIMcxxAsyy7zVS+dLWE6tZjsoEJCVOtb25qeTdpsuTUkPQHQRtVqs0MMlD2dv4ZjbJ2lo9v1fcl4HpCFd9Pe5WMaOrT6kmonFOFYb6bEzcEmx43ipORXPuM5NzXP7pR7nhrxV/mWyIN0F0RGoxeTDvXMygkvS2ok2as= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: GIC v3 and later support SGI broadcast, i.e., the mode that routes interrupts to all PEs in the system excluding the local CPU. Supporting this mode can avoid looping through all the remote CPUs when broadcasting SGIs, especially for systems with 200+ CPUs. The performance improvement can be measured with the rest of this series booted with "hugetlb_free_vmemmap=on irqchip.gicv3_pseudo_nmi=1": cd /sys/kernel/mm/hugepages/ echo 600 >hugepages-1048576kB/nr_hugepages echo 2048kB >hugepages-1048576kB/demote_size perf record -g -- bash -c "echo 600 >hugepages-1048576kB/demote" gic_ipi_send_mask() bash sys time Before: 38.14% 0m10.513s After: 0.20% 0m5.132s Signed-off-by: Yu Zhao --- drivers/irqchip/irq-gic-v3.c | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c index ce87205e3e82..42c39385e1b9 100644 --- a/drivers/irqchip/irq-gic-v3.c +++ b/drivers/irqchip/irq-gic-v3.c @@ -1394,9 +1394,20 @@ static void gic_send_sgi(u64 cluster_id, u16 tlist, unsigned int irq) gic_write_sgi1r(val); } +static void gic_broadcast_sgi(unsigned int irq) +{ + u64 val; + + val = BIT(ICC_SGI1R_IRQ_ROUTING_MODE_BIT) | (irq << ICC_SGI1R_SGI_ID_SHIFT); + + pr_devel("CPU %d: broadcasting SGI %u\n", smp_processor_id(), irq); + gic_write_sgi1r(val); +} + static void gic_ipi_send_mask(struct irq_data *d, const struct cpumask *mask) { int cpu; + cpumask_t broadcast; if (WARN_ON(d->hwirq >= 16)) return; @@ -1407,6 +1418,13 @@ static void gic_ipi_send_mask(struct irq_data *d, const struct cpumask *mask) */ dsb(ishst); + cpumask_copy(&broadcast, cpu_present_mask); + cpumask_clear_cpu(smp_processor_id(), &broadcast); + if (cpumask_equal(&broadcast, mask)) { + gic_broadcast_sgi(d->hwirq); + goto done; + } + for_each_cpu(cpu, mask) { u64 cluster_id = MPIDR_TO_SGI_CLUSTER_ID(gic_cpu_to_affinity(cpu)); u16 tlist; @@ -1414,7 +1432,7 @@ static void gic_ipi_send_mask(struct irq_data *d, const struct cpumask *mask) tlist = gic_compute_target_list(&cpu, mask, cluster_id); gic_send_sgi(cluster_id, tlist, d->hwirq); } - +done: /* Force the above writes to ICC_SGI1R_EL1 to be executed */ isb(); }