From patchwork Fri Jul 12 17:00:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Jackman X-Patchwork-Id: 13732007 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7A3B6C2BD09 for ; Fri, 12 Jul 2024 17:02:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D17D16B00BC; Fri, 12 Jul 2024 13:02:06 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C97E36B00BD; Fri, 12 Jul 2024 13:02:06 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AC0646B00BE; Fri, 12 Jul 2024 13:02:06 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 78C8A6B00BC for ; Fri, 12 Jul 2024 13:02:06 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 1523B80C08 for ; Fri, 12 Jul 2024 17:02:06 +0000 (UTC) X-FDA: 82331718252.07.3FB8064 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf26.hostedemail.com (Postfix) with ESMTP id D3EFE140013 for ; Fri, 12 Jul 2024 17:02:03 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=JAke7Bbm; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf26.hostedemail.com: domain of 3imGRZggKCM02tv35t6uz77z4x.v75416DG-553Etv3.7Az@flex--jackmanb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3imGRZggKCM02tv35t6uz77z4x.v75416DG-553Etv3.7Az@flex--jackmanb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720803698; a=rsa-sha256; cv=none; b=kNHwBzN84GU300TWYW7x6NTbydxkfHHpeM0e2yAlqafjVBXt3Ax/qbkTLkkf8tTCt9Nbqj NI+a8r8BbGe6qtOCgkIvCk8Xm5nMqn2O9Yc9BsD8iwBZa9MP50axBbgZw14LwiVKRp6TuE v7jwu83CF1NK80QLd98U24I9q/wNFXs= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=JAke7Bbm; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf26.hostedemail.com: domain of 3imGRZggKCM02tv35t6uz77z4x.v75416DG-553Etv3.7Az@flex--jackmanb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3imGRZggKCM02tv35t6uz77z4x.v75416DG-553Etv3.7Az@flex--jackmanb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720803698; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=2i/EY2B7r1Db+9uC+ppZJ+s/pV+Uwv0UgdTvi6uATis=; b=bXUBO3WeLhIqMNDXq4yhr6OjjL/9CburV4VgiWn8uVB3kUv94QjFJ/REJdNuECcTTTxvoB Q2uiLTXxS8WiwVWK8uCnv5yPmhL8ctQKh0DhcnPvw0OsN+ojtGRe5Ic1vB9aSO4gkBPJaA Y2hU7NY22Rlyk5/qOCGRV3a330lCCsI= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-65be82a3241so37994247b3.0 for ; Fri, 12 Jul 2024 10:02:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1720803723; x=1721408523; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=2i/EY2B7r1Db+9uC+ppZJ+s/pV+Uwv0UgdTvi6uATis=; b=JAke7BbmaejGHLCyVnduEaXDBK6wtNRVYUWb26hIRrGVErxEvWVzBAr7gwAUTBwkYg CqIEu8lZkOLPbXfzmrEGzebU02L3xhZ8J9lHIQfngwOVEVrOJWU2gkKTlysPP5mypmOb gCS4pfjyB+V8wF7MGR2D8BTKGTTiOnJZ6ApLW7R1N7WN9Mrgyst0rOLY4ac6mmOESWIq S1yh0Fij1ERgQ1cbNUu/jGRaRb0zyisPyqtuOHH6RiTfmwPhRdb4YgEHrxLwe1JBqGR/ L6JETlGS5olq6dbwYIHH86URvVqejlcZ5n7IRnhjZnZguYPOg0b0b5ecAjX/TNRByou4 C4DA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720803723; x=1721408523; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=2i/EY2B7r1Db+9uC+ppZJ+s/pV+Uwv0UgdTvi6uATis=; b=U98sAww18MQu3Y0DdPDw1sSTR89Pk4ri1yPiqpSDy9x4OZE6P5sVUfXH8l1ZjDQ0dc uDIOQ3H2cDqGe/WmrMAmFxiWQKqdneE50CbySadb7FIQ/4WC49F0MaxWVGSI1P2WmT3C fkKnkKuC/9ruUDDcI+Wqv7gSqUtCqzO+mzVFv7ev9iGXTZxoEeAcPwXAoWJZLsk+qfbs MB4oGiLQbVpHV8IKrPBLH3tRMECJ/Bmkx43l9kvJV7RsnkkS9QpLdRkypuUtwLPIWizg F/7sTderZwNh6ecBJ/pjtTr5tljofEJ3jOgfrpxmKqujKciekAEWKS6L2/wyPn4JVPjF FsJQ== X-Forwarded-Encrypted: i=1; AJvYcCVgxf4keoicM8N5ogAvwrP4mpUpZCdMnmNjN8ygfFyaBJO071GYvH/SuxMEnn09Xnq8G3iDy+xZUrXKYGkvkuWvpxI= X-Gm-Message-State: AOJu0YwgVmnQRovpTAmK86l+Uw6DWx2ZDK9aNoPmFNNy5+lxo/IZXZRX FvWgZpqZjqJ1MawuHYWJbceljSh7HxS8ptN7DtUoWrO4dAmkdZfb1XpA8llMMOmwUbTE7l7Wes1 XwvnqF7dfcg== X-Google-Smtp-Source: AGHT+IFNgdBalbrX784MJSkG7g28jv3ta6SaiIVV2y32vALYWfCkCKFBAZMdIemQeL1Zf8yLiAeIuqRY45gfTg== X-Received: from beeg.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:11db]) (user=jackmanb job=sendgmr) by 2002:a05:6902:154d:b0:dfe:fe5e:990a with SMTP id 3f1490d57ef6-e041b1134demr24774276.9.1720803722717; Fri, 12 Jul 2024 10:02:02 -0700 (PDT) Date: Fri, 12 Jul 2024 17:00:42 +0000 In-Reply-To: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> Mime-Version: 1.0 References: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> X-Mailer: b4 0.14-dev Message-ID: <20240712-asi-rfc-24-v1-24-144b319a40d8@google.com> Subject: [PATCH 24/26] mm: asi: Make TLB flushing correct under ASI From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , Sean Christopherson , Paolo Bonzini , Alexandre Chartre , Liran Alon , Jan Setje-Eilers , Catalin Marinas , Will Deacon , Mark Rutland , Andrew Morton , Mel Gorman , Lorenzo Stoakes , David Hildenbrand , Vlastimil Babka , Michal Hocko , Khalid Aziz , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Valentin Schneider , Paul Turner , Reiji Watanabe , Junaid Shahid , Ofir Weisse , Yosry Ahmed , Patrick Bellasi , KP Singh , Alexandra Sandulescu , Matteo Rizzo , Jann Horn Cc: x86@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, kvm@vger.kernel.org, Brendan Jackman X-Rspamd-Queue-Id: D3EFE140013 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: ufubqdkkamsfqn6fwsn57gn1xj9xhygs X-HE-Tag: 1720803723-214610 X-HE-Meta: U2FsdGVkX1/YkLSkBTS5ALNhBFlfhFJjw9Dr/sL9sn0wAHi1goIOno+3EWQI/zVc3KYI8wM51YOXhgUwE0ukcsYk9u0CRjvvo011mUmj9ypzPN5NMeI357q6BPK2dtORlTvJIR+Nif4j/TZCyu0P/Z0R1Gj/Xn4srOhHYjnmi8gKBCp5aeZxJXOGBgTXPloVTK053TERs+Zox/1LSoKdh4iFceEEfietNZqdhEgXTCKNadbBGB/QkrM/dvIwOVmTModUiupOhbGmVvRegnipmznboF9lpLochrjA0ehbUcLz/xiycmA+SkLrGh3/VMKJZ9JcnBS6x0BOXd1ydmQ6GAWLITaBreDUF3y0S36MnBdXBuLRlVk0e6VXmrkWbkbq8Ym++iJG/ZeEFnrOjlHq6/uEBG9wD5hmJzi0TTKVLJmLjZVqrFSztdj/1WwyuGxZGBqVCtPYxibvDfZfJiTx1lc0TLZThjAeph4XAQ3Y2u/PpKwSDvm3TFI/zRoVGNvHEYP4dKb/2Smz5xVuraSZkx22z/5YvqyX/n63ekp/JUZFnHMPreBxER6sJmBRax2L/CR9zo8H+f78/fCpNULNlDhaIVvzhoA+Fdl/aDTTZRJ+5VaQt5WKtb4KOnC9wfVUSEPpZJCVX+OMBGTRkKbO+UMCHmzGpy1940EaF+yajYrTVI5/0PhD6Yglfd+vJLQLNrnjkZ8Y5DrWjQyyjrYA1htqR+OBeUzVBrqgyfNs7bSTem9jFOlz7guPMJjK2zWeyT0mmxwmFItvjf4sn/tvKjmTgtRd9yHw4jZpFql0uvbOqn3nEBhzZiBymRN+zvsVpYK3u0Wz8UCTE8nB7mAUwa9gh4GQ9ogQlDDs3uEAaTb5jcHZ7QAeugeCHiuGhQaPEFMeqclKEd50gzSIIGz7STcD+aK6Q8pFmMv8Qk2AopNV0julUmtMnYZl3sdHw5/OfLLihVzwOdquBzUmsrD aG5Kf4P9 Fyw+DwamyqRS8aoEHtq6G9NIxSNnUnwFZyCeh4j6mj38sLZRMUcm9XbYgV5kvZQ7QYJ9mReOc87cTo6zHTisMNzkOpvK6YP2WGUkhwy/+hLJseLhy9k+I5jjNlWqCmeqD5hkdx8mbof7GcLDDTE0Ooecv9K3+pWc1w/EmtJHDlxuEhOw/pgCPgJWTnQiwKUrQibvrFb0vHLlu8oSSHNPVqilmlxoR2c+6bjTyOHbSEJbhheVYibWwI9+pWHw1bCeVzjUL+cbIKmIrS8Lrr6w5z05LJx1UpvWWyTUdDp8/2ugZs6bEb7gUL0sNAa1Pb3ubz8+tXNiyOt/9hpD0QAZ/0CrszhYyrkGrSZlvyh3aGQO1JbFlK+uCp5FQ/zcjfK+TJ9GnxJGUMpMhZPAdoNm9C6fDc/DzPSfmHb9O0h3sMnYjIOeBo6NVw3FAUvXTbY82PCjky1Lga7Iy08ETeoR/ao3e0a02ZRvz3rIEebowlavnKqcGn/DJk67hQoVb/oPU4H5PYGH48Q55B7dYyltIy9z/GnNqVPKngj4e X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This is the absolute minimum change for TLB flushing to be correct under ASI. There are two arguably orthogonal changes in here but they feel small enough for a single commit. .:: CR3 stabilization As noted in the comment ASI can destabilize CR3, but we can stabilize it again by calling asi_exit, this makes it safe to read CR3 and write it back. This is enough to be correct - we don't have to worry about invalidating the other ASI address space (i.e. we don't need to invalidate the restricted address space if we are currently unrestricted / vice versa) because we currently never set the noflush bit in CR3 for ASI transitions. Even without using CR3's noflush bit there are trivial optimizations still on the table here: on where invpcid_flush_single_context is available (i.e. with the INVPCID_SINGLE feature) we can use that in lieu of the CR3 read/write, and avoid the extremely costly asi_exit. .:: Invalidating kernel mappings Before ASI, with KPTI off we always either disable PCID or use global mappings for kernel memory. However ASI disables global kernel mappings regardless of factors. So we need to invalidate other address spaces to trigger a flush when we switch into them. Note that there is currently a pointless write of cpu_tlbstate.invalidate_other in the case of KPTI and !PCID. We've added another case of that (ASI, !KPTI and !PCID). I think that's preferable to expanding the conditional in flush_tlb_one_kernel. Signed-off-by: Brendan Jackman --- arch/x86/mm/tlb.c | 23 +++++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index a9804274049e..1d9a300fe788 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -219,7 +219,7 @@ static void clear_asid_other(void) * This is only expected to be set if we have disabled * kernel _PAGE_GLOBAL pages. */ - if (!static_cpu_has(X86_FEATURE_PTI)) { + if (!static_cpu_has(X86_FEATURE_PTI) && !static_cpu_has(X86_FEATURE_ASI)) { WARN_ON_ONCE(1); return; } @@ -1178,15 +1178,19 @@ void flush_tlb_one_kernel(unsigned long addr) * use PCID if we also use global PTEs for the kernel mapping, and * INVLPG flushes global translations across all address spaces. * - * If PTI is on, then the kernel is mapped with non-global PTEs, and - * __flush_tlb_one_user() will flush the given address for the current - * kernel address space and for its usermode counterpart, but it does - * not flush it for other address spaces. + * If PTI or ASI is on, then the kernel is mapped with non-global PTEs, + * and __flush_tlb_one_user() will flush the given address for the + * current kernel address space and, if PTI is on, for its usermode + * counterpart, but it does not flush it for other address spaces. */ flush_tlb_one_user(addr); - if (!static_cpu_has(X86_FEATURE_PTI)) + /* Nothing more to do if PTI and ASI are completely off. */ + if (!static_cpu_has(X86_FEATURE_PTI) && !static_cpu_has(X86_FEATURE_ASI)) { + VM_WARN_ON_ONCE(static_cpu_has(X86_FEATURE_PCID) && + !(__default_kernel_pte_mask & _PAGE_GLOBAL)); return; + } /* * See above. We need to propagate the flush to all other address @@ -1275,6 +1279,13 @@ STATIC_NOPV void native_flush_tlb_local(void) invalidate_user_asid(this_cpu_read(cpu_tlbstate.loaded_mm_asid)); + /* + * Restricted ASI CR3 is unstable outside of critical section, so we + * couldn't flush via a CR3 read/write. + */ + if (!asi_in_critical_section()) + asi_exit(); + /* If current->mm == NULL then the read_cr3() "borrows" an mm */ native_write_cr3(__native_read_cr3()); }