From patchwork Tue Jul 7 05:28:40 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pingfan Liu X-Patchwork-Id: 11647639 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8226513B6 for ; Tue, 7 Jul 2020 05:30:46 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5313720758 for ; Tue, 7 Jul 2020 05:30:46 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="sTL/eqOh"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Xg0hEmod" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5313720758 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:MIME-Version:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:In-Reply-To:References:List-Owner; bh=3zA3JoRbYQZO+PJ67JqI/0M8A9POsd6RSkGt6voW+I0=; b=sTL/eqOhfHuZaEheRCM/UKkQQS QzrnXRtO4d5PM6PSYY/a2GOKLdVRNFIrO5RQflB5h1UnanmXXbA47UwVhic0E4x7IQT8YzAsiQ5sT +xJ1DRbH3zRKLeA7vUU2INEOii12qm5xNQZVENcSKsl8WLv10e3UF+x2Ji19+TdcHtg8TpD5pDNbK DOia+1iso84ncNBzqKLIElwTFuXKB1FUB6LThDb3XXNAXzGWKAycXirEOLmpabpETZMdG2izx/qf3 EhysqtzSqhlRvzxrGKdjt6PQ+DF1wjXcduVUO9jgg4X47lc2ud6wAwQo3A6OM455BwptaJNm+jkzW uPH5BjDQ==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jsgAB-0005rM-7I; Tue, 07 Jul 2020 05:29:03 +0000 Received: from mail-pf1-x443.google.com ([2607:f8b0:4864:20::443]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jsgA7-0005r0-ML for linux-arm-kernel@lists.infradead.org; Tue, 07 Jul 2020 05:29:00 +0000 Received: by mail-pf1-x443.google.com with SMTP id 207so17751381pfu.3 for ; Mon, 06 Jul 2020 22:28:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id; bh=uFHn5zsnYTPcM+kwbYbrXR4LwROAWUUrCcvx4zRGDR0=; b=Xg0hEmod21383Q74Qm5F0ZnQIhE9lWkm1uvCF9bJXgOtG5I4oOB6XwquOBa2oaOTgN zbJdEPgdd2vjVLrMWdkWB681BHuowHPleeBGlvJLKvlfcM1z009AFDnPtVrj3Pgh31OF KWk401l2uO9gdMCQxqaz1PGGvjUZuKnI52X0baKtsu/YiFDlGxnGHIUMjsca8TQcSJcg c81mE7G2PcCdCKX7blut13XvRGluKfNGFAV1RRwK/qBGWhh3MiVa74owvh7jRnt/lxjH Y8V9UfQpqCr1vK4SWUXmbA0p8zcdj0dcK/QRQMqXg+xyoHdjkwx7lpjCG+7VJCtzOwKr lwGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=uFHn5zsnYTPcM+kwbYbrXR4LwROAWUUrCcvx4zRGDR0=; b=Dz4m2qVBENQf2CBJsa2qiDdHtJHqHSchvRMjr59A2AFu7GXieg+iqZVklNyLZMSkjf JG0BTRH9q2ff/oU3A9BZDzGGCTF+/caaiA379FD/1Xnvxz5BZq19kzy0AvqJ/eay4rpt G24t5bVK0yq1R/4SJE4yIDYrMy4NppbYnlO5o2fBpx1+U7D+QnSswOJ+x4FJON31m/La qaahFE4Cg+4pI1qQP9Zti0h/VhnzWW1nKVab6hRlpBpSWN5YznUNcaLHIbR+bUojTvvr PtcmctFvtCmOsyBReS2W0SwO9zecmUXM2wMNTcOHY1zYYounlHXL4MtGYuoo4li4OfRC R/yQ== X-Gm-Message-State: AOAM531Xx4O5LCQpVw0DM1BJqNbNQLafM6BEoIKMY7+BnSiQdInGgHLt WaTgXU8UkVsN4+3hXWuNIbmUaNQ= X-Google-Smtp-Source: ABdhPJxWrKTJH81HTgPIWDTNqVGQ1MLJjY+sqWaHd0mE+LIadMZ99h3y2XuFF05Esu3t0s36Uhzz7Q== X-Received: by 2002:a62:cd89:: with SMTP id o131mr14911303pfg.195.1594099737129; Mon, 06 Jul 2020 22:28:57 -0700 (PDT) Received: from mylaptop.redhat.com ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id 204sm10014335pfx.3.2020.07.06.22.28.53 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Mon, 06 Jul 2020 22:28:56 -0700 (PDT) From: Pingfan Liu To: linux-arm-kernel@lists.infradead.org Subject: [PATCHv2] arm64/mm: save memory access in check_and_switch_context() fast switch path Date: Tue, 7 Jul 2020 13:28:40 +0800 Message-Id: <1594099720-17982-1-git-send-email-kernelfans@gmail.com> X-Mailer: git-send-email 2.7.5 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200707_012859_799127_C0A6DC75 X-CRM114-Status: GOOD ( 14.27 ) X-Spam-Score: -0.2 (/) X-Spam-Report: SpamAssassin version 3.4.4 on merlin.infradead.org summary: Content analysis details: (-0.2 points) pts rule name description ---- ---------------------- -------------------------------------------------- 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider [kernelfans[at]gmail.com] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:443 listed in] [list.dnswl.org] -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Jean-Philippe Brucker , Vladimir Murzin , Steve Capper , Catalin Marinas , Pingfan Liu , Will Deacon MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org On arm64, smp_processor_id() reads a per-cpu `cpu_number` variable, using the per-cpu offset stored in the tpidr_el1 system register. In some cases we generate a per-cpu address with a sequence like: cpu_ptr = &per_cpu(ptr, smp_processor_id()); Which potentially incurs a cache miss for both `cpu_number` and the in-memory `__per_cpu_offset` array. This can be written more optimally as: cpu_ptr = this_cpu_ptr(ptr); Which only needs the offset from tpidr_el1, and does not need to load from memory. Signed-off-by: Pingfan Liu Cc: Catalin Marinas Cc: Will Deacon Cc: Steve Capper Cc: Mark Rutland Cc: Vladimir Murzin Cc: Jean-Philippe Brucker To: linux-arm-kernel@lists.infradead.org --- v1 -> v2: improve commit log --- arch/arm64/include/asm/mmu_context.h | 6 ++---- arch/arm64/mm/context.c | 10 ++++++---- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h index ab46187..808c3be 100644 --- a/arch/arm64/include/asm/mmu_context.h +++ b/arch/arm64/include/asm/mmu_context.h @@ -175,7 +175,7 @@ static inline void cpu_replace_ttbr1(pgd_t *pgdp) * take CPU migration into account. */ #define destroy_context(mm) do { } while(0) -void check_and_switch_context(struct mm_struct *mm, unsigned int cpu); +void check_and_switch_context(struct mm_struct *mm); #define init_new_context(tsk,mm) ({ atomic64_set(&(mm)->context.id, 0); 0; }) @@ -214,8 +214,6 @@ enter_lazy_tlb(struct mm_struct *mm, struct task_struct *tsk) static inline void __switch_mm(struct mm_struct *next) { - unsigned int cpu = smp_processor_id(); - /* * init_mm.pgd does not contain any user mappings and it is always * active for kernel addresses in TTBR1. Just set the reserved TTBR0. @@ -225,7 +223,7 @@ static inline void __switch_mm(struct mm_struct *next) return; } - check_and_switch_context(next, cpu); + check_and_switch_context(next); } static inline void diff --git a/arch/arm64/mm/context.c b/arch/arm64/mm/context.c index d702d60..a206655 100644 --- a/arch/arm64/mm/context.c +++ b/arch/arm64/mm/context.c @@ -198,9 +198,10 @@ static u64 new_context(struct mm_struct *mm) return idx2asid(asid) | generation; } -void check_and_switch_context(struct mm_struct *mm, unsigned int cpu) +void check_and_switch_context(struct mm_struct *mm) { unsigned long flags; + unsigned int cpu; u64 asid, old_active_asid; if (system_supports_cnp()) @@ -222,9 +223,9 @@ void check_and_switch_context(struct mm_struct *mm, unsigned int cpu) * relaxed xchg in flush_context will treat us as reserved * because atomic RmWs are totally ordered for a given location. */ - old_active_asid = atomic64_read(&per_cpu(active_asids, cpu)); + old_active_asid = atomic64_read(this_cpu_ptr(&active_asids)); if (old_active_asid && asid_gen_match(asid) && - atomic64_cmpxchg_relaxed(&per_cpu(active_asids, cpu), + atomic64_cmpxchg_relaxed(this_cpu_ptr(&active_asids), old_active_asid, asid)) goto switch_mm_fastpath; @@ -236,10 +237,11 @@ void check_and_switch_context(struct mm_struct *mm, unsigned int cpu) atomic64_set(&mm->context.id, asid); } + cpu = smp_processor_id(); if (cpumask_test_and_clear_cpu(cpu, &tlb_flush_pending)) local_flush_tlb_all(); - atomic64_set(&per_cpu(active_asids, cpu), asid); + atomic64_set(this_cpu_ptr(&active_asids), asid); raw_spin_unlock_irqrestore(&cpu_asid_lock, flags); switch_mm_fastpath: