From patchwork Wed Feb 23 05:21:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Junaid Shahid X-Patchwork-Id: 12756365 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C7E4DC433EF for ; Wed, 23 Feb 2022 05:23:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5FF718D0008; Wed, 23 Feb 2022 00:23:58 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 587088D0001; Wed, 23 Feb 2022 00:23:58 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 44E328D0008; Wed, 23 Feb 2022 00:23:58 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0112.hostedemail.com [216.40.44.112]) by kanga.kvack.org (Postfix) with ESMTP id 341F68D0001 for ; Wed, 23 Feb 2022 00:23:58 -0500 (EST) Received: from smtpin30.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id EB6559DAFA for ; Wed, 23 Feb 2022 05:23:57 +0000 (UTC) X-FDA: 79172902914.30.26DA092 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf13.hostedemail.com (Postfix) with ESMTP id 8168F20005 for ; Wed, 23 Feb 2022 05:23:57 +0000 (UTC) Received: by mail-yb1-f202.google.com with SMTP id a12-20020a056902056c00b0061dc0f2a94aso26531129ybt.6 for ; Tue, 22 Feb 2022 21:23:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=LI1Qxw8YovJHKPYFPYoCf4datmDsbayp3z46DL8Kp8Q=; b=U7xDPHt7nJyB2GTAgVc+5mVNa5D9AiEIkeTSJpYyuShkSe5G+Xo37Cmu4z2S0ODbd3 Wm5Eeq8cvy6TNvAv3A7bWuNv++i7NTg2KMLHM/m4eHjfRoQqwVJ2y/Yi5MDF7xrDbsQI j5wRAvkCT4cYk3Bc/Ve5H0pU1FD7r6VuCExamGfvXcE67Mlg9W1l1RFlwTf48DP0g9j8 zrQbvmaWMqoIe6J4oo0xoCcquiVzfC7OAHWIqYk7V7cZwcAdfqwpYM8UaM/ZPk1iYlJM o8gS8/KkwFxTjJeXBM9VLMqxRp+ccKcT56QtTvv+HXupkPEETHBW4Z+e7zRt00MQFKUT ixgA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=LI1Qxw8YovJHKPYFPYoCf4datmDsbayp3z46DL8Kp8Q=; b=lreRIxzYqpcrF5DEakYs/76yaCIBGctQFlJaP7ITCbOCJTgVuIosytNvNyHeSMXNWa 8s0Meo40PunE16bfp8n7PaDiDQ0VDA/k3adHAf73BybMIBMxfZMnAerM0ZeklzF0Dd1J kvrMmNW0En9jhvg+i21Jinapq7vfWEuTW3KenWywm3YfgTStCMUUNHHsSD9xqwmkH04S Mi/9KWthV/0AggLvMEpOKFvkJYOE1Kvj0VDmpohwXVfH4hLhru4BghjTnAsd/R5BzHYl ztEZAwnWZzvrj3QRP0M78MEttmpo0GYxyuNsL0RKXzkowdW2MQg1UfwliCUpMWcOOsUO g49A== X-Gm-Message-State: AOAM530fxlukstIK/OFNEe+/YSeKLBfYGa5xck36gqZciZgsllJAGE/z CbKOmA8Kr/lWRVdHyFpenN4yBYSYlVz/ X-Google-Smtp-Source: ABdhPJxfznelYhWgkiOo62fJh4kZWSbldwDVUL5AZ4s809e7D3FsyXFvg8XSNCQZyLZ5Vxg+VGGrfUoGnpy5 X-Received: from js-desktop.svl.corp.google.com ([2620:15c:2cd:202:ccbe:5d15:e2e6:322]) (user=junaids job=sendgmr) by 2002:a81:5cc3:0:b0:2d0:a2d0:9c0e with SMTP id q186-20020a815cc3000000b002d0a2d09c0emr27666033ywb.270.1645593836836; Tue, 22 Feb 2022 21:23:56 -0800 (PST) Date: Tue, 22 Feb 2022 21:21:41 -0800 In-Reply-To: <20220223052223.1202152-1-junaids@google.com> Message-Id: <20220223052223.1202152-6-junaids@google.com> Mime-Version: 1.0 References: <20220223052223.1202152-1-junaids@google.com> X-Mailer: git-send-email 2.35.1.473.g83b2b277ed-goog Subject: [RFC PATCH 05/47] mm: asi: Make __get_current_cr3_fast() ASI-aware From: Junaid Shahid To: linux-kernel@vger.kernel.org Cc: kvm@vger.kernel.org, pbonzini@redhat.com, jmattson@google.com, pjt@google.com, oweisse@google.com, alexandre.chartre@oracle.com, rppt@linux.ibm.com, dave.hansen@linux.intel.com, peterz@infradead.org, tglx@linutronix.de, luto@kernel.org, linux-mm@kvack.org X-Rspamd-Queue-Id: 8168F20005 X-Stat-Signature: z3xcwf3mwkntoecgo8rn7sp93mnwymo9 X-Rspam-User: Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=U7xDPHt7; spf=pass (imf13.hostedemail.com: domain of 37MQVYgcKCOMOZSFNIXLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--junaids.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=37MQVYgcKCOMOZSFNIXLTTLQJ.HTRQNSZc-RRPaFHP.TWL@flex--junaids.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspamd-Server: rspam05 X-HE-Tag: 1645593837-535245 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When ASI is active, __get_current_cr3_fast() adjusts the returned CR3 value accordingly to reflect the actual ASI CR3. Signed-off-by: Junaid Shahid --- arch/x86/include/asm/asi.h | 7 +++++++ arch/x86/mm/tlb.c | 20 ++++++++++++++++++-- 2 files changed, 25 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/asi.h b/arch/x86/include/asm/asi.h index 7702332c62e8..95557211dabd 100644 --- a/arch/x86/include/asm/asi.h +++ b/arch/x86/include/asm/asi.h @@ -112,6 +112,11 @@ static inline void asi_intr_exit(void) } } +static inline pgd_t *asi_pgd(struct asi *asi) +{ + return asi->pgd; +} + #else /* CONFIG_ADDRESS_SPACE_ISOLATION */ static inline void asi_intr_enter(void) { } @@ -120,6 +125,8 @@ static inline void asi_intr_exit(void) { } static inline void asi_init_thread_state(struct thread_struct *thread) { } +static inline pgd_t *asi_pgd(struct asi *asi) { return NULL; } + #endif /* CONFIG_ADDRESS_SPACE_ISOLATION */ #endif diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 88d9298720dc..25bee959d1d3 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -17,6 +17,7 @@ #include #include #include +#include #include "mm_internal.h" @@ -1073,12 +1074,27 @@ void flush_tlb_kernel_range(unsigned long start, unsigned long end) */ unsigned long __get_current_cr3_fast(void) { - unsigned long cr3 = build_cr3(this_cpu_read(cpu_tlbstate.loaded_mm)->pgd, - this_cpu_read(cpu_tlbstate.loaded_mm_asid)); + unsigned long cr3; + pgd_t *pgd; + u16 asid = this_cpu_read(cpu_tlbstate.loaded_mm_asid); + struct asi *asi = asi_get_current(); + + if (asi) + pgd = asi_pgd(asi); + else + pgd = this_cpu_read(cpu_tlbstate.loaded_mm)->pgd; + + cr3 = build_cr3(pgd, asid); /* For now, be very restrictive about when this can be called. */ VM_WARN_ON(in_nmi() || preemptible()); + /* + * CR3 is unstable if the target ASI is unrestricted + * and a restricted ASI is currently loaded. + */ + VM_WARN_ON_ONCE(asi && asi_is_target_unrestricted()); + VM_BUG_ON(cr3 != __read_cr3()); return cr3; }