From patchwork Fri Jul 3 05:44:39 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pingfan Liu X-Patchwork-Id: 11640905 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B013D912 for ; Fri, 3 Jul 2020 05:46:47 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 88C6A20771 for ; Fri, 3 Jul 2020 05:46:47 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="UAjCJlOE"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="EbqFjUfg" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 88C6A20771 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:MIME-Version:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:In-Reply-To:References:List-Owner; bh=dkO64JeWYcczYpn48bkItnWkGfiQ1j415LmVEzN9d3k=; b=UAjCJlOEUMCn9SAJaHOZIZQO7W 5L6pNndfg00LvnbLSjJQhEcZcN11+W4RZLgQMTULelCp8jeDvy9HaNt+mgPMLl4Q6n8V7JNzOFJn1 Jv+8ycGYN9lzWJ/rxAov48AelXbX0eS5qMeekpcM8Odvj0CpV/gRO8vfSIUrQBJfRDmpFN0NRQp6O Vq7fNhWs1c3ZXtvcIB4NgoXmr6Kv8vei/Q3NKL5kKo5Ap6ZSmPPWiWmooDOQHKIBhlRoKYJZ0/scz sU9Qk89Lghq1tEn7RnKMXU4ysrrtH5JcoYVw4Ip6hNrItDQ8do5xq2ldE0z8GyBNo3LDCCRTIbWOc V+tNbe3g==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jrEVO-0002QG-Cq; Fri, 03 Jul 2020 05:44:58 +0000 Received: from mail-pf1-x444.google.com ([2607:f8b0:4864:20::444]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jrEVL-0002MO-He for linux-arm-kernel@lists.infradead.org; Fri, 03 Jul 2020 05:44:56 +0000 Received: by mail-pf1-x444.google.com with SMTP id b16so13509286pfi.13 for ; Thu, 02 Jul 2020 22:44:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id; bh=2s2L/gSOUDYovOjE5PEtlLm9hIsLPplRV9RFD5NgN0g=; b=EbqFjUfgbliw9XykrVlCU+VxIvoCIn/jwynTZoYnM4mjQR+wREJW1loe5lrcO0af3i SokWBVM3ciBXc7pGGc2lXgBwEDbxg2MtUdWpi+mgmmf6HDe5jTcjakIlGoB13jZEGSbf ls45zYH8Ayx1IEx70wQYkOhzVh9paJtePmE3hJvRPdXRC5ShCXY26U3Cc3RQF4GB7Pve +FtmX+rINu1GJuNoA1aXFclpqRf6u/tMXPZX6oSSO2xQD7h6NcpnNVbaO6a+3wILm+QO K8hmREISt8oE0iwatsskAIaYj2b7W9LmTtFec+Xyx0hi8BCh6MpgmdWz6lqp/IReyeqK UswQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=2s2L/gSOUDYovOjE5PEtlLm9hIsLPplRV9RFD5NgN0g=; b=fL9UMLZ8u9MunX3aMU08OhMW8Z3DfDVA548jL6Rir63LJZK+/Lry6YOtZtwRUXQbC/ Uz7Sr+vjjuvSOT0qyVgSjTxekbgPfNjGPtoctoPxsbmbutuPQOkc/aOtFe1QkYUkwuo+ MLlJgQ2BcfO0HWmXWnCEFyeL562ZAwxEGl3d17zTLtWqZR2FiOCmyhVxSgy3uJHKtRdB jWp2kKK5yc2/FQFxLKFISYU4RRWuSuV50Aawi0h/E6Xw/xJrfusrtyObLU0h9aP5C6QU V8hLyk5CwWnLDl5gj5l5Vutxc7n+sPDovfew+P/IXlk+Ip0paD9wwcZW/3ydyaxqGPeW cl9A== X-Gm-Message-State: AOAM531oURKDIETerOZuIUufun6tKnPDUFp1mdeogfkAZYeHuavbaFFY 0XtPaBOJtEQG3K1n8pgd7F6iTxc= X-Google-Smtp-Source: ABdhPJwaNK8dWn/fbvCxELOUiO1WTn/8U3fp4Le3R4cKyDs5gDLqHrnixR/hm7JXmQHEMAY3Kw+scg== X-Received: by 2002:a63:371d:: with SMTP id e29mr27713935pga.153.1593755091866; Thu, 02 Jul 2020 22:44:51 -0700 (PDT) Received: from mylaptop.redhat.com ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id l23sm1473868pjy.45.2020.07.02.22.44.48 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 02 Jul 2020 22:44:51 -0700 (PDT) From: Pingfan Liu To: linux-arm-kernel@lists.infradead.org Subject: [PATCH] arm64/mm: save memory access in check_and_switch_context() fast switch path Date: Fri, 3 Jul 2020 13:44:39 +0800 Message-Id: <1593755079-2160-1-git-send-email-kernelfans@gmail.com> X-Mailer: git-send-email 2.7.5 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200703_014455_594905_218351CF X-CRM114-Status: GOOD ( 13.34 ) X-Spam-Score: -0.2 (/) X-Spam-Report: SpamAssassin version 3.4.4 on merlin.infradead.org summary: Content analysis details: (-0.2 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2607:f8b0:4864:20:0:0:0:444 listed in] [list.dnswl.org] 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider [kernelfans[at]gmail.com] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Mark Rutland , Jean-Philippe Brucker , Vladimir Murzin , Steve Capper , Catalin Marinas , Pingfan Liu , Will Deacon MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org The cpu_number and __per_cpu_offset cost two different cache lines, and may not exist after a heavy user space load. By replacing per_cpu(active_asids, cpu) with this_cpu_ptr(&active_asids) in fast path, register is used and these memory access are avoided. Signed-off-by: Pingfan Liu Cc: Catalin Marinas Cc: Will Deacon Cc: Steve Capper Cc: Mark Rutland Cc: Vladimir Murzin Cc: Jean-Philippe Brucker To: linux-arm-kernel@lists.infradead.org --- arch/arm64/include/asm/mmu_context.h | 6 ++---- arch/arm64/mm/context.c | 10 ++++++---- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h index ab46187..808c3be 100644 --- a/arch/arm64/include/asm/mmu_context.h +++ b/arch/arm64/include/asm/mmu_context.h @@ -175,7 +175,7 @@ static inline void cpu_replace_ttbr1(pgd_t *pgdp) * take CPU migration into account. */ #define destroy_context(mm) do { } while(0) -void check_and_switch_context(struct mm_struct *mm, unsigned int cpu); +void check_and_switch_context(struct mm_struct *mm); #define init_new_context(tsk,mm) ({ atomic64_set(&(mm)->context.id, 0); 0; }) @@ -214,8 +214,6 @@ enter_lazy_tlb(struct mm_struct *mm, struct task_struct *tsk) static inline void __switch_mm(struct mm_struct *next) { - unsigned int cpu = smp_processor_id(); - /* * init_mm.pgd does not contain any user mappings and it is always * active for kernel addresses in TTBR1. Just set the reserved TTBR0. @@ -225,7 +223,7 @@ static inline void __switch_mm(struct mm_struct *next) return; } - check_and_switch_context(next, cpu); + check_and_switch_context(next); } static inline void diff --git a/arch/arm64/mm/context.c b/arch/arm64/mm/context.c index d702d60..a206655 100644 --- a/arch/arm64/mm/context.c +++ b/arch/arm64/mm/context.c @@ -198,9 +198,10 @@ static u64 new_context(struct mm_struct *mm) return idx2asid(asid) | generation; } -void check_and_switch_context(struct mm_struct *mm, unsigned int cpu) +void check_and_switch_context(struct mm_struct *mm) { unsigned long flags; + unsigned int cpu; u64 asid, old_active_asid; if (system_supports_cnp()) @@ -222,9 +223,9 @@ void check_and_switch_context(struct mm_struct *mm, unsigned int cpu) * relaxed xchg in flush_context will treat us as reserved * because atomic RmWs are totally ordered for a given location. */ - old_active_asid = atomic64_read(&per_cpu(active_asids, cpu)); + old_active_asid = atomic64_read(this_cpu_ptr(&active_asids)); if (old_active_asid && asid_gen_match(asid) && - atomic64_cmpxchg_relaxed(&per_cpu(active_asids, cpu), + atomic64_cmpxchg_relaxed(this_cpu_ptr(&active_asids), old_active_asid, asid)) goto switch_mm_fastpath; @@ -236,10 +237,11 @@ void check_and_switch_context(struct mm_struct *mm, unsigned int cpu) atomic64_set(&mm->context.id, asid); } + cpu = smp_processor_id(); if (cpumask_test_and_clear_cpu(cpu, &tlb_flush_pending)) local_flush_tlb_all(); - atomic64_set(&per_cpu(active_asids, cpu), asid); + atomic64_set(this_cpu_ptr(&active_asids), asid); raw_spin_unlock_irqrestore(&cpu_asid_lock, flags); switch_mm_fastpath: