From patchwork Fri Jul 12 17:00:26 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Jackman X-Patchwork-Id: 13731991 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DDF0BC3DA45 for ; Fri, 12 Jul 2024 17:01:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9B3CA6B009E; Fri, 12 Jul 2024 13:01:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9104F6B009F; Fri, 12 Jul 2024 13:01:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7176D6B00A0; Fri, 12 Jul 2024 13:01:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 4B51B6B009E for ; Fri, 12 Jul 2024 13:01:18 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id D53EF80BEC for ; Fri, 12 Jul 2024 17:01:17 +0000 (UTC) X-FDA: 82331716194.05.E4BF2A0 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) by imf24.hostedemail.com (Postfix) with ESMTP id BC77D180038 for ; Fri, 12 Jul 2024 17:01:15 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=jRZHhJcD; spf=pass (imf24.hostedemail.com: domain of 3WmGRZggKCJ0G79HJ7K8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--jackmanb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3WmGRZggKCJ0G79HJ7K8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720803649; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IU+0QFZ2g2My69wAuJPkncJIr3fDXfYxT6Ca6fNVeD8=; b=Un1Sq7lrJmHx/nf5vTPdk+2T34HhXTJ6t29Z08NfsovFHg+8FiuXtMw/KxkfuhMewxWyfY EwvwVoK1jbAvHrzK2XwdGvK7Oi43GBJl6yF/xF6xTf5eGntBShPzKj+iV4+HH5JnG87hcd q8l4+w3/yxQoUXUvUdd8G9MeD47Fy9s= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=jRZHhJcD; spf=pass (imf24.hostedemail.com: domain of 3WmGRZggKCJ0G79HJ7K8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--jackmanb.bounces.google.com designates 209.85.128.201 as permitted sender) smtp.mailfrom=3WmGRZggKCJ0G79HJ7K8DLLDIB.9LJIFKRU-JJHS79H.LOD@flex--jackmanb.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720803649; a=rsa-sha256; cv=none; b=IekJ4a9gYUpPDvderVu3DWpOBja3Vv2/0l/pbjlq9ODuKRCMMm/AnZcAPnlkySLK0bByZl 9GgOn6P3LNQ3RM73GQz3lRZQPBLDZjj+p98dqjn5tEuQ9qsxggI1gMEnWP24ut6LBXpYk+ tmSyHssZRweulQXnKvzaOZMY8bRftvc= Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-64b9800b377so41502337b3.0 for ; Fri, 12 Jul 2024 10:01:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1720803675; x=1721408475; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=IU+0QFZ2g2My69wAuJPkncJIr3fDXfYxT6Ca6fNVeD8=; b=jRZHhJcDi4xULJ3u96PgeJdikVCG/kV1q/faDHOg0ydeP2xND2+d+KJxeXnih+ge3+ VVJC+i2nAADbzFLrjcY5DI1+/LXHAtCZQXpGvogQb3PsbKjL40Mu/7rcAbGGrmQId0DJ Hv/Nm4g/cVIYdvdY900Cxw7/dVn2STSaAX7O7VCI2Brqb//JEwZt/OrWOOrRkh5f0C51 88x6QOCQ3B+b/x4N+iGFq4qY0OqKlPwgTzD7obB1XUKOByw9Ekm3mdUhAZBxinwjuHBr HxwHa2cQ/nosdbnM/Jxyxp5GTyeYvbjO0uWcrVgHk7CMla0xywHmhsxkLgU/t2Bhvly9 qJMg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720803675; x=1721408475; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=IU+0QFZ2g2My69wAuJPkncJIr3fDXfYxT6Ca6fNVeD8=; b=aMor4bWQnTLrRIac0iW+Yy/hX1ugt+blg47UzE0+Crx5zYdxj7uveTkBLxBdT60Ewm 2ECuLnizfWI0YGOfftvkY6z5FoZjn8O5dt1X7UuVRGjp7bxOqUwVPKdoqPAs8pPobQVp 3RsuE7fP6hp0KPB4vgQB6ON5UFAW6KK6RRdfbdTUa5NwArNSuo1kuQjWKyKBr1BXCaRj bktIc26Yjvn4Hs48ot0Q4zeeSotRXEjca10Gj3Pc3v2Piv+Pdwe+/7DIUg6xX1O+jNcO 7KiBUsok/3ai9rjZbzSlydblAAssXOSE4C+GEOAzc+ARmQoj/S38G1Xhxy5MF83U6X9W XLpw== X-Forwarded-Encrypted: i=1; AJvYcCWO28c/coZ+i+f4w/FMjN1N/JudnHVzdH9ef0iWH9NSsIKw0J/0hPV1NnJbqdxX7TNtrCvo5H8NtwofczcN4oM/qvo= X-Gm-Message-State: AOJu0YyABITeV0XeEH1yXBpJiP2v8bCS4b8uTsUSbRC3K8vlRqAjmvg/ HInfh/ZGSqgwjPKBz2C+XzAcujKHxK1+91ngbEgjSo2L6u1HEQYMO9kvDf5QJR7mSXiuh4Djeq1 ImwhvhJ8Blw== X-Google-Smtp-Source: AGHT+IH090AhnVnRaBqnnakPZ4/V5j0nTOpLdhxmJg4znTZivgjhg0KK45kar3tFbJqN9PhohJA5PmDXL/BkUA== X-Received: from beeg.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:11db]) (user=jackmanb job=sendgmr) by 2002:a25:945:0:b0:e05:7113:920c with SMTP id 3f1490d57ef6-e058a6cfb8dmr11142276.6.1720803674689; Fri, 12 Jul 2024 10:01:14 -0700 (PDT) Date: Fri, 12 Jul 2024 17:00:26 +0000 In-Reply-To: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> Mime-Version: 1.0 References: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> X-Mailer: b4 0.14-dev Message-ID: <20240712-asi-rfc-24-v1-8-144b319a40d8@google.com> Subject: [PATCH 08/26] mm: asi: Use separate PCIDs for restricted address spaces From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , Sean Christopherson , Paolo Bonzini , Alexandre Chartre , Liran Alon , Jan Setje-Eilers , Catalin Marinas , Will Deacon , Mark Rutland , Andrew Morton , Mel Gorman , Lorenzo Stoakes , David Hildenbrand , Vlastimil Babka , Michal Hocko , Khalid Aziz , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Valentin Schneider , Paul Turner , Reiji Watanabe , Junaid Shahid , Ofir Weisse , Yosry Ahmed , Patrick Bellasi , KP Singh , Alexandra Sandulescu , Matteo Rizzo , Jann Horn Cc: x86@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, kvm@vger.kernel.org, Brendan Jackman X-Stat-Signature: xgien3jca1wj9gf8cbrmncm8f9hdsbg4 X-Rspam-User: X-Rspamd-Queue-Id: BC77D180038 X-Rspamd-Server: rspam02 X-HE-Tag: 1720803675-420100 X-HE-Meta: U2FsdGVkX19jZNuRUpf2vvFNq58nPPTeXuSFtf5BHMyNyEd8zM8FtnO5C91/I+rjFsZVrLeiJNn1rmKJP8F82BE+uh4BeIXzXkwn1DzMOpkoQNA/QYGxlkxMRd7TFnnolgPB5kyFKZsPtLZaAt9R+jdb1QH7sIIe5m7ERcXk1a7DqZ1lBBBegfIpUGdkc27J8ibCmBoWwrj3WjicEOuNf3XrTuBhY3r0bBbHm5GjicjfJ9nEwsVerP27sdv7Rhk9mUV3xhrkm86AvwdsG15OKEfhoNSt5wuEjVXdY6TSw0uLL1MSktq2DmvLz6bNd75SX960T+5F+52aNrdDu2a97hrynJdch/W2sf/NNvcbsO+jD3tEDjj1t492WyS10c/fNT/Qew+DFR5YrMo50rzUm4Uimv/nNSUXKPtOeyMPuXnoLw75sgDw0EcnLH5ItGmJth9vpM45WkqEkG5QoS6iFBDjyc/9/pNuXzk7WjRiKhTQ5+fVLOm0HgLUmTKyM17an1TfXlbXDGhGCsK+QOOiqNB8XLmYwl+zeM0ex6xhVmvYmOT5b+cMYfAbBlMPwdhoVGWEvaqiYPo8H3m7G0JdwJ3uIGHOb0sO45S82p3xGBBo2KebiqcNG6puwP96gO8/tcrSKkXWCl0JD+r9iS0BwgMnAIVQYF4NFhCZRsQNDDK88La0l8Z9ZxQwUWJYu85a4FxX8N1oJw15bUcrT3dL7v2DiD6Mb1+opZgRXWCBwSf1J40ZK7ej6Ub0n06rW8R87EPboGjq/iQ1gvtGNC4ysHrV7vsK2XCk4DOzGO8SYjnf1uOpyGtJN5/r7/zJROQyAEyE79iFB6ob2TnG9yX1dyZ3sxcwK0HztfNI3NZ241KMcxbcS3sJ7shapUjWmpeoCZcNX9ZcFo4WOGrqNm8Tq70nE2fEuw3IQLL3Ela4hywpX3/GJpcr3Y00WaFT30UYsNvJAQWR+Rwy86YBaMr 4+AvqVxl SJNUCzkA33rFgxgb4eQ859reEsDkKuc6nCOyeXeFl207Q9iV5Tbwh0CNqy1nBueTxuQejMs2U2dkzsod0iNES3bTfOsNCSzDgp0nxm/pHxh1o/dlzBVO/e0lPa27YzY7PjQ7m8ZOL/gZv7EAtyqc/YcH4PSaGFI+CvauHiBPgoMp4lDAPA30BC+Oq2fCNk4U3iG8pSGCbkxS0p4j4hXeXqQsnE0gyHw9y55wwRJln4OMDSHPb4OPM5YeOrCWd1xNgHHO7GCdw52E8DFbSc3OR6kbkZMQcL71G/h99XK3tUbWLHAuxk5L46qFrrub4F/C509gUQ4xryPQxJxbwUC3PvjaBKCpo4wybkMUEQ0KQeOKuC3IE4KOFVQEGdNspfseiVEjLma2oEvMNCb+FpCuMPbeFqZGTMZgaVoqSbx9bQxzu30y2/BZS96kDOQHbl99GBo8tYzkrSqeadrvNpfqnFup2pleEVGdpYNhXa1ozrU7AB4oJ+MQG4vMnpV0WIZ0aBh+2ox/2eMqASbVicTLzPFWtgbhOs7DbVRX/ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Junaid Shahid Each restricted address space is assigned a separate PCID. Since currently only one ASI instance per-class exists for a given process, the PCID is just derived from the class index. This commit only sets the appropriate PCID when switching CR3, but does not actually use the NOFLUSH bit. That will be done by later patches. Signed-off-by: Junaid Shahid Signed-off-by: Brendan Jackman --- arch/x86/include/asm/asi.h | 10 +++++++++- arch/x86/include/asm/tlbflush.h | 3 +++ arch/x86/mm/asi.c | 7 ++++--- arch/x86/mm/tlb.c | 44 +++++++++++++++++++++++++++++++++++++---- 4 files changed, 56 insertions(+), 8 deletions(-) diff --git a/arch/x86/include/asm/asi.h b/arch/x86/include/asm/asi.h index df34a8c0560b..1a19a925300c 100644 --- a/arch/x86/include/asm/asi.h +++ b/arch/x86/include/asm/asi.h @@ -69,7 +69,14 @@ #define static_asi_enabled() cpu_feature_enabled(X86_FEATURE_ASI) #define ASI_MAX_NUM_ORDER 2 -#define ASI_MAX_NUM (1 << ASI_MAX_NUM_ORDER) +/* + * We include an ASI identifier in the higher bits of PCID to use + * different PCID for restricted ASIs from non-restricted ASIs (see asi_pcid). + * The ASI identifier we use for this is asi_index + 1, as asi_index + * starts from 0. The -1 below for ASI_MAX_NUM comes from this PCID + * space availability. + */ +#define ASI_MAX_NUM ((1 << ASI_MAX_NUM_ORDER) - 1) struct asi_hooks { /* @@ -101,6 +108,7 @@ struct asi { struct asi_class *class; struct mm_struct *mm; int64_t ref_count; + u16 index; }; DECLARE_PER_CPU_ALIGNED(struct asi *, curr_asi); diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index ed847567b25d..3605f6b99da7 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -392,6 +392,9 @@ static inline bool huge_pmd_needs_flush(pmd_t oldpmd, pmd_t newpmd) #define huge_pmd_needs_flush huge_pmd_needs_flush unsigned long build_cr3(pgd_t *pgd, u16 asid, unsigned long lam); +unsigned long build_cr3_pcid(pgd_t *pgd, u16 pcid, unsigned long lam, bool noflush); + +u16 asi_pcid(struct asi *asi, u16 asid); #ifdef CONFIG_ADDRESS_MASKING static inline u64 tlbstate_lam_cr3_mask(void) diff --git a/arch/x86/mm/asi.c b/arch/x86/mm/asi.c index 2cd8e93a4415..0ba156f879d3 100644 --- a/arch/x86/mm/asi.c +++ b/arch/x86/mm/asi.c @@ -140,6 +140,7 @@ int asi_init(struct mm_struct *mm, int asi_index, struct asi **out_asi) asi->class = &asi_class[asi_index]; asi->mm = mm; + asi->index = asi_index; exit_unlock: if (err) @@ -174,6 +175,7 @@ EXPORT_SYMBOL_GPL(asi_destroy); noinstr void __asi_enter(void) { u64 asi_cr3; + u16 pcid; struct asi *target = asi_get_target(current); /* @@ -200,9 +202,8 @@ noinstr void __asi_enter(void) */ this_cpu_write(curr_asi, target); - asi_cr3 = build_cr3(target->pgd, - this_cpu_read(cpu_tlbstate.loaded_mm_asid), - tlbstate_lam_cr3_mask()); + pcid = asi_pcid(target, this_cpu_read(cpu_tlbstate.loaded_mm_asid)); + asi_cr3 = build_cr3_pcid(target->pgd, pcid, tlbstate_lam_cr3_mask(), false); write_cr3(asi_cr3); if (target->class->ops.post_asi_enter) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 9a5afeac9654..34d61b56d33f 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -98,7 +98,12 @@ # define PTI_CONSUMED_PCID_BITS 0 #endif -#define CR3_AVAIL_PCID_BITS (X86_CR3_PCID_BITS - PTI_CONSUMED_PCID_BITS) +#define ASI_CONSUMED_PCID_BITS ASI_MAX_NUM_ORDER +#define ASI_PCID_BITS_SHIFT CR3_AVAIL_PCID_BITS +#define CR3_AVAIL_PCID_BITS (X86_CR3_PCID_BITS - PTI_CONSUMED_PCID_BITS - \ + ASI_CONSUMED_PCID_BITS) + +static_assert(BIT(CR3_AVAIL_PCID_BITS) > TLB_NR_DYN_ASIDS); /* * ASIDs are zero-based: 0->MAX_AVAIL_ASID are valid. -1 below to account @@ -155,18 +160,23 @@ static inline u16 user_pcid(u16 asid) return ret; } +static inline unsigned long __build_cr3(pgd_t *pgd, u16 pcid, unsigned long lam) +{ + return __sme_pa_nodebug(pgd) | pcid | lam; +} + inline_or_noinstr unsigned long build_cr3(pgd_t *pgd, u16 asid, unsigned long lam) { - unsigned long cr3 = __sme_pa_nodebug(pgd) | lam; + u16 pcid = 0; if (static_cpu_has(X86_FEATURE_PCID)) { VM_WARN_ON_ONCE(asid > MAX_ASID_AVAILABLE); - cr3 |= kern_pcid(asid); + pcid = kern_pcid(asid); } else { VM_WARN_ON_ONCE(asid != 0); } - return cr3; + return __build_cr3(pgd, pcid, lam); } static inline unsigned long build_cr3_noflush(pgd_t *pgd, u16 asid, @@ -181,6 +191,19 @@ static inline unsigned long build_cr3_noflush(pgd_t *pgd, u16 asid, return build_cr3(pgd, asid, lam) | CR3_NOFLUSH; } +inline_or_noinstr unsigned long build_cr3_pcid(pgd_t *pgd, u16 pcid, + unsigned long lam, bool noflush) +{ + u64 noflush_bit = 0; + + if (!static_cpu_has(X86_FEATURE_PCID)) + pcid = 0; + else if (noflush) + noflush_bit = CR3_NOFLUSH; + + return __build_cr3(pgd, pcid, lam) | noflush_bit; +} + /* * We get here when we do something requiring a TLB invalidation * but could not go invalidate all of the contexts. We do the @@ -995,6 +1018,19 @@ static void put_flush_tlb_info(void) #endif } +#ifdef CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION + +inline_or_noinstr u16 asi_pcid(struct asi *asi, u16 asid) +{ + return kern_pcid(asid) | ((asi->index + 1) << ASI_PCID_BITS_SHIFT); +} + +#else /* CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION */ + +u16 asi_pcid(struct asi *asi, u16 asid) { return kern_pcid(asid); } + +#endif /* CONFIG_MITIGATION_ADDRESS_SPACE_ISOLATION */ + void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, unsigned long end, unsigned int stride_shift, bool freed_tables)