From patchwork Fri Jul 12 17:00:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Jackman X-Patchwork-Id: 13731992 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id CE1E0C2BD09 for ; Fri, 12 Jul 2024 17:01:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6A33E6B00A0; Fri, 12 Jul 2024 13:01:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 62C736B00A1; Fri, 12 Jul 2024 13:01:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 42FB56B00A2; Fri, 12 Jul 2024 13:01:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 1487D6B00A0 for ; Fri, 12 Jul 2024 13:01:22 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 7276E140787 for ; Fri, 12 Jul 2024 17:01:21 +0000 (UTC) X-FDA: 82331716362.06.65EE14D Received: from mail-wm1-f74.google.com (mail-wm1-f74.google.com [209.85.128.74]) by imf03.hostedemail.com (Postfix) with ESMTP id ED78920033 for ; Fri, 12 Jul 2024 17:01:18 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=SJvoR4yd; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf03.hostedemail.com: domain of 3XWGRZggKCKAJACKMANBGOOGLE.COMLINUX-MMKVACK.ORG@flex--jackmanb.bounces.google.com designates 209.85.128.74 as permitted sender) smtp.mailfrom=3XWGRZggKCKAJACKMANBGOOGLE.COMLINUX-MMKVACK.ORG@flex--jackmanb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720803653; a=rsa-sha256; cv=none; b=nhGuGgFd3UC9V9Fzyxt2sh+1KEoaQre4jhe0tajg6dfIcWauAA0YdG6EvrqX1VzoS7Hxh8 V9CrgfGc7XPZdaOv8n6F8iCHi9jgShRalheXGFLkjwuDhhqJ53DjPmg7sHHjE3If4VqR+X S3XJOYzZkTQoUlRYV6+gp8Nk2CXMTqM= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=SJvoR4yd; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf03.hostedemail.com: domain of 3XWGRZggKCKAJACKMANBGOOGLE.COMLINUX-MMKVACK.ORG@flex--jackmanb.bounces.google.com designates 209.85.128.74 as permitted sender) smtp.mailfrom=3XWGRZggKCKAJACKMANBGOOGLE.COMLINUX-MMKVACK.ORG@flex--jackmanb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720803653; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=STKCp8fX1eTmkj5Bbc4ElFS0AmmAwYWDzWQm15HV9Io=; b=Kvhhx2Gfzof39zjYt1Wdg0G6hKyI7+LjVHSiZz9K6ytB7WoMf/S90NMi3Orl9lFTb3sfUp ZeBJ3ogRReHyirtkvfGlvnYMwUcJIfHP7SkJIGqMqs6wc9hY5/Y7xXyAuMqmI0X7bXXJh2 CWDXPkNn9Y3cFwHtfX3iGoyArqyWhsc= Received: by mail-wm1-f74.google.com with SMTP id 5b1f17b1804b1-4272718b9b0so14630545e9.1 for ; Fri, 12 Jul 2024 10:01:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1720803677; x=1721408477; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=STKCp8fX1eTmkj5Bbc4ElFS0AmmAwYWDzWQm15HV9Io=; b=SJvoR4ydmpHToPGYiENBBydKQIrUsYHIOHV2AkjK6Kvze9geU/ZU0aJmRwDeZdMqjK iX2RtvzuhWlQlpGmiZ27DHAJwaHE/2q7K3TD4wmDUxMBkEoUiBidiGZNnDtweKWoWLTr TvfTI/8daH9m2Umy2iMjiQQUzZS3nu8N3FxfpPu5+p16qmXk3HR4SzrgxKRjbQpcPJpM ZeGnzJZ5kBdFFWf8n2bexTvV79T8wvEm9wHrQmJQPO42aQGNrzUrXOA0HD9A0R6za+Aj Cp0VBjQEg47deP0IBbW0lPTQqK6cHwcmgVWLGx9hpf3yu+ElMRbiaLbYfYc7WFaLvA2H V6kw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720803677; x=1721408477; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=STKCp8fX1eTmkj5Bbc4ElFS0AmmAwYWDzWQm15HV9Io=; b=WaFX4CFKg74qxSUq66vuuIKqGR8i8Dr6jir40ssKzU2wrwdfBkePrJ8Z2kMSdDVkz4 4f7a2aKtsExn1vdEq5paVexXYjPucKl0QKbxBxfLCeX0VaAy4/LN6OoVgVyMVoTug7vn 4JkRUoxV25/n1ZPAMrzcvQLlNHwBSiblRXZxW1J/YH9t3F0pVnFEzkxW9TfotlEu3tBJ of0I2wiGPJ3XLEGcae3zDtA8r2o7xewf9A1cDIohmUsCUkJD+EOCcecM4jd7OdgbpUpa y1UXzvuu+DrIwi3Oz7aJmdqh5h+Ra8diPCLs9oiDaVPDuCcgssJ0BU7h89ANelJz6TmJ Pkhw== X-Forwarded-Encrypted: i=1; AJvYcCWCCsi/eS+KaqGN6R9H9D3JwsQxCKRY/YJPndbU+wcb9tOprWlKXFr2Tx74tU6xv1UksNuKYx+BxYPCh30yR4hhzSE= X-Gm-Message-State: AOJu0YxPjgHZ5dVFttaHxek8CdiNTqdXloJhJBUOEPHoNFE7iL2nRFu8 v9r9IgL6MOglBuV0lHfs70fMtYbVSf/RdOp1oTgoa3G1RnEcQ7v9fTKzUagvD93ZvirV4LWH3I7 wh90AlxD20g== X-Google-Smtp-Source: AGHT+IFyt7OjPNZMl6PbkdfYNDPh8L+dPRBgusnHtzyUmHnXcEp/uNI1T+enUzqfuMGxALw2f3LiWe2F0njYig== X-Received: from beeg.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:11db]) (user=jackmanb job=sendgmr) by 2002:a05:600c:358a:b0:426:5ff0:1b48 with SMTP id 5b1f17b1804b1-426708ef8b5mr784025e9.4.1720803677551; Fri, 12 Jul 2024 10:01:17 -0700 (PDT) Date: Fri, 12 Jul 2024 17:00:27 +0000 In-Reply-To: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> Mime-Version: 1.0 References: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> X-Mailer: b4 0.14-dev Message-ID: <20240712-asi-rfc-24-v1-9-144b319a40d8@google.com> Subject: [PATCH 09/26] mm: asi: Make __get_current_cr3_fast() ASI-aware From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , Sean Christopherson , Paolo Bonzini , Alexandre Chartre , Liran Alon , Jan Setje-Eilers , Catalin Marinas , Will Deacon , Mark Rutland , Andrew Morton , Mel Gorman , Lorenzo Stoakes , David Hildenbrand , Vlastimil Babka , Michal Hocko , Khalid Aziz , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Valentin Schneider , Paul Turner , Reiji Watanabe , Junaid Shahid , Ofir Weisse , Yosry Ahmed , Patrick Bellasi , KP Singh , Alexandra Sandulescu , Matteo Rizzo , Jann Horn Cc: x86@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, kvm@vger.kernel.org, Brendan Jackman X-Rspamd-Queue-Id: ED78920033 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 1iipqoype7jxigohy85a8k8xa3xfky34 X-HE-Tag: 1720803678-419373 X-HE-Meta: U2FsdGVkX1/zIvsLmupD+OFhN6aySst+ygtF60K4uhw0G0Y0p5azMaiICVlAg3FGIEDlg3JM0Uhg642gsOCQVBPgjzpqj7iNHNY30rpluLRNBn+J/mBWcE8Vy3vXxo7pOll1Vgitlv7gEQDH8PnbE3JARVbDy9PqupXdtIDVeqyVXL1folyjN5gLCkjfmSVIMdsLHy132P2k9IWC8FqpSNmA9tFqspVlJOhEc3XjCJVvqlMseWRlsSkraKEaRnx2sCSlze8MYhUWXh5nuE4gnUA2f6M7ttaQkgg4LFwRB5cqnLw4yCLgMX2cApx9RulefhO5GijvfFA4g9M9jp22idLHSYc9NARiNfh8VPEbY5btkWx0dTdQIVwgaJE1vkBNQTL1g0cD3h6pv+5/KQWR1/6y46XUfk2OU3f07NaE0i1VhSinbXADNPb0aMGd1B9FxQW9TtRxG724f4nlNcIFdZ9iQI76MmKXTfe/JQrVq7GXjKDroGsjyKbp2p6dQncyN9L/VXWyjXphT39d5c+nSMD6vM9NQb6jAJ+Y3hSok7Ps43LCVS8pDITSLVWpScHgeA+y52NjmgTwwokMu3Cc9fmkR9bq8oRCO4oksRTdyzN9IUSNowMfvVcK5RL6r3236gDidkDdPkMcY9NU35Xn8K0ljKoN1AGTdSUT8iuUJhMh1NP6dVzk1wraTpoVLoYVV17lYI3eFkHYzw3bmwEvuD4z4HQnGUmhRmWqqav2DSLzPCXx/VGJpINaGDuSgI2J8dZuhAX6a2BNhFDD4dAtNmhlcyIOFpQ/E0ivveN92wqeK2x53c2xn4BOxAt3vHsuY61KtlSB5Ii0+K3eZMhuC0lpsLmbxUFkNJiJclpX3TlgS/qWyNBqVYcen++FJrTsI+XjsfaTuzUtyAHsB77bmPVaW3SvQJbFNq+4BpJx28PpxP0auKdDQ01t+kVynLZz6z6c94xsHZkWzPXoCcc 7Fdidih/ uplBhTal9+V62KAI23wN1uvPQGdfHNA+tOyojsn9/P+HOpztSUus0gi/eDdKSWgiQmO6HS4Rww3oA8FJTcq6G/VB9bvUtXShI5EuAnz/TUNYNAaqJdi7PLgQBIhEkb4NSRNbbAU8PaZR7Q4SombPA36w37H0vkYeR3Cdc7V73RmXfRTzFSsunWGQPgbHv/7PiPR/sNQYSB7NBeliqcwDXabNQkbpeXG+v4sck1uFg1qEqwPqrSDda/C8OcLGWnIFVAVu9pzrYsxaQeRJVq02VOBtANwscx/CSakV0kQR4CPy6HHj/5WAqGuiY6N9q74IurVX6VVtszHAyl7ek0oO8u4AC/Z/JDQeOBXeWhIqS40neY/n4MHsMr+CfGBNnttUJ1osm2FR9FGVh0RCDeAKdIjrrMIrVSBGlZ03X+VoHr8gRHBJklLv2Xyi2ohuwG29hTDLFiNLLT5zmaG+E/cjr6aUm69m8wlQ6FFEkp1G8UYJ1fPLQt6RUQCwxOvfcIzKsuLMHK/utYUhdr+7CrH/XByXFB4CPBgG4Qz5W9FuCUswHf9yUsdmf1vG818hThMgZ5hoiixmbVos8EbYgMq5az3gRH4EbQF+ihUaPFaWrc7i7Ae3Lj8RWX4lT9A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Junaid Shahid When ASI is active, __get_current_cr3_fast() adjusts the returned CR3 value accordingly to reflect the actual ASI CR3. Signed-off-by: Junaid Shahid Signed-off-by: Brendan Jackman --- arch/x86/mm/tlb.c | 27 +++++++++++++++++++++++---- 1 file changed, 23 insertions(+), 4 deletions(-) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 34d61b56d33f..02f73a71d4ea 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -18,6 +18,7 @@ #include #include #include +#include #include #include "mm_internal.h" @@ -1125,14 +1126,32 @@ void flush_tlb_kernel_range(unsigned long start, unsigned long end) */ inline_or_noinstr unsigned long __get_current_cr3_fast(void) { - unsigned long cr3 = - build_cr3(this_cpu_read(cpu_tlbstate.loaded_mm)->pgd, - this_cpu_read(cpu_tlbstate.loaded_mm_asid), - tlbstate_lam_cr3_mask()); + unsigned long cr3; + pgd_t *pgd; + u16 asid = this_cpu_read(cpu_tlbstate.loaded_mm_asid); + struct asi *asi = asi_get_current(); + u16 pcid; + + if (asi) { + pgd = asi_pgd(asi); + pcid = asi_pcid(asi, asid); + } else { + pgd = this_cpu_read(cpu_tlbstate.loaded_mm)->pgd; + pcid = kern_pcid(asid); + } + + cr3 = build_cr3_pcid(pgd, pcid, tlbstate_lam_cr3_mask(), false); /* For now, be very restrictive about when this can be called. */ VM_WARN_ON(in_nmi() || preemptible()); + /* + * Outside of the ASI critical section, an ASI-restricted CR3 is + * unstable because an interrupt (including an inner interrupt, if we're + * already in one) could cause a persistent asi_exit. + */ + VM_WARN_ON_ONCE(asi && (asi_is_relaxed() || asi_intr_nest_depth())); + VM_BUG_ON(cr3 != __read_cr3()); return cr3; }