From patchwork Fri Jul 12 17:00:24 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Brendan Jackman X-Patchwork-Id: 13731989 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 11DC0C2BD09 for ; Fri, 12 Jul 2024 17:01:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5B88F6B009B; Fri, 12 Jul 2024 13:01:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4CC9C6B009C; Fri, 12 Jul 2024 13:01:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2CFBE6B009D; Fri, 12 Jul 2024 13:01:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 0388A6B009B for ; Fri, 12 Jul 2024 13:01:12 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 7A5A9140BFA for ; Fri, 12 Jul 2024 17:01:12 +0000 (UTC) X-FDA: 82331715984.25.9D30483 Received: from mail-wr1-f73.google.com (mail-wr1-f73.google.com [209.85.221.73]) by imf11.hostedemail.com (Postfix) with ESMTP id 6A63240034 for ; Fri, 12 Jul 2024 17:01:10 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=cBht38FA; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf11.hostedemail.com: domain of 3VGGRZggKCJcA13BD1E27FF7C5.3FDC9ELO-DDBM13B.FI7@flex--jackmanb.bounces.google.com designates 209.85.221.73 as permitted sender) smtp.mailfrom=3VGGRZggKCJcA13BD1E27FF7C5.3FDC9ELO-DDBM13B.FI7@flex--jackmanb.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720803645; a=rsa-sha256; cv=none; b=49J69kB5fjNlfQ2DjB8gvYouqXWhu9hhvnPBqtlTmxY+MuhZagpzEnJSrVGP/eIrApxcEs 4pupx9gGkXgdQ5SZIuBqpCYAwQzI85eoCV2MKzALF6jTDA7D3r7p1x0WOYtAigOWbY2WUq ShPnb1AUaR6xt/hVQbfnhJ7+a4dSYr4= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=cBht38FA; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf11.hostedemail.com: domain of 3VGGRZggKCJcA13BD1E27FF7C5.3FDC9ELO-DDBM13B.FI7@flex--jackmanb.bounces.google.com designates 209.85.221.73 as permitted sender) smtp.mailfrom=3VGGRZggKCJcA13BD1E27FF7C5.3FDC9ELO-DDBM13B.FI7@flex--jackmanb.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720803645; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=qrAVAbVK+nVNUznV+52/ScTYsIGKlGAcY8a4rOPWm6Y=; b=eQOP2xIVg6mdxNoTWRfbf+EicMwKLDHcGziujtp+UQTno0oIl8qaAIZ7Y8BGRSZ3w8ZJx1 TdJRAQ/L6/d5OafO4Wwa5NebP6SgEeUV7GCvl5EXpQOgLiIbzMF/n39uvCfetFkQo5Q5Vx T0v7WhWF1Xs5nkco3zwk/JX7FDz658g= Received: by mail-wr1-f73.google.com with SMTP id ffacd0b85a97d-36785e6c1e6so411038f8f.3 for ; Fri, 12 Jul 2024 10:01:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1720803669; x=1721408469; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=qrAVAbVK+nVNUznV+52/ScTYsIGKlGAcY8a4rOPWm6Y=; b=cBht38FA581BiWP4Ilk3xQudUL3a5rD2+SxJtAfSiPWQQZpuIqb5dJKcExb7+9XIA2 DorgaQD93EIiiHrZ/+rqp2JnqpjPNP6RNTaA6D47rsGALFdcKHJ8ir3m3YnBQ6+cHjz2 wNflVM20nmd7tXlY+u9EPvdF6lnOnu0qjvq8U/EMHugtT8hcY+JgjOt1gR1ImNKldv3i LhpNpkG5SKwsTB0FeC00fs2hkyiKXitcIcoq2oDJ3RMAyrFZiOJjE5ONZt3xIQ+MB4Yo JBfXcvNq7kO6q/5AxT3SUw2jKvNk1A5VfDYXSMSn/hNaRRoDod/xCo22vO955VlQbR3W 2+Ig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1720803669; x=1721408469; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=qrAVAbVK+nVNUznV+52/ScTYsIGKlGAcY8a4rOPWm6Y=; b=PxXJC0GIcTWQQIxduGNUhEBA4Uxi0t7ULoRuUEUYBJm+NHUlMwv3yzc7uUB09Y39Ih gIFPAAHrvhfIzF39hl5adJJ7adA6yqe03jUsPJJGX9VxNz0z3U7zYjr4DJMtqmB1jEF2 fptOo7ufOaSO1yZevNpN05gkGNv1QZv59cVst/7Rop7/j8GUorwmc6maoTmbl6TWtKCO 2khk56EAiMJKscYVLbGbRt30tTcBoSK5anoreHFR2b+OSHZJScWdin06kotCNX8c6BTw TrFesaf/Axt8yy8n2S4wBGZi72+xgaC5ydjirFSXhDvC0CwBn83V5U96fQZsYQdz9zTY q3UQ== X-Forwarded-Encrypted: i=1; AJvYcCX7WchaT1sqyJmSgUGwPrD4p7B2zpa7foYcjNw+iy9NqcKB5ETshVzX8VAvH91EuddT87detrSBG+6184Coy9oqcDo= X-Gm-Message-State: AOJu0YzHnYNYHA4L113ekt5mXIKnur29O0JZLblgiO5Zm2pRzNc3hTOU DjPPQdKZsKmPmHaRLBu/yjn/CLB1FejN6f6QN2Z8a5mbKM6CPlN4v3qj0cQj2yhTnV/U8SG7RGa YjohDLh1HUA== X-Google-Smtp-Source: AGHT+IF3kvDD67OIZ2BMm13dMbd7zSJf6yY9zVib7vZL6oHM8iz/nEJhVOpklDYifHh85TWN8ozTDRdKs7e20g== X-Received: from beeg.c.googlers.com ([fda3:e722:ac3:cc00:28:9cb1:c0a8:11db]) (user=jackmanb job=sendgmr) by 2002:a05:6000:187:b0:367:60dc:9ec4 with SMTP id ffacd0b85a97d-367cea68085mr18401f8f.6.1720803668718; Fri, 12 Jul 2024 10:01:08 -0700 (PDT) Date: Fri, 12 Jul 2024 17:00:24 +0000 In-Reply-To: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> Mime-Version: 1.0 References: <20240712-asi-rfc-24-v1-0-144b319a40d8@google.com> X-Mailer: b4 0.14-dev Message-ID: <20240712-asi-rfc-24-v1-6-144b319a40d8@google.com> Subject: [PATCH 06/26] mm: asi: ASI support in interrupts/exceptions From: Brendan Jackman To: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , Sean Christopherson , Paolo Bonzini , Alexandre Chartre , Liran Alon , Jan Setje-Eilers , Catalin Marinas , Will Deacon , Mark Rutland , Andrew Morton , Mel Gorman , Lorenzo Stoakes , David Hildenbrand , Vlastimil Babka , Michal Hocko , Khalid Aziz , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Valentin Schneider , Paul Turner , Reiji Watanabe , Junaid Shahid , Ofir Weisse , Yosry Ahmed , Patrick Bellasi , KP Singh , Alexandra Sandulescu , Matteo Rizzo , Jann Horn Cc: x86@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, kvm@vger.kernel.org, Brendan Jackman X-Rspamd-Queue-Id: 6A63240034 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: nn45y83a7srx3j4us1t59ftbgdkr8dpj X-HE-Tag: 1720803670-252161 X-HE-Meta: U2FsdGVkX18pGIAYXNNVlIUjRXHnUNaBuWFEdWGaWpViszZYQDI6zESFJ5ioe39vRSE90w/JAeYv6FVQ09Du1BQQXWM8A1qaINSOvHWMZ0nXQyQmhzSV80oWm3xVoWzWbWtYdvugTDOwwPmTUAMS1oEEFoGqPcTl0LjFAetHARbLaxnwDrsdiFPu1ZjtfTrO6aEcllOT4lCn488VwXw7fzk/upwxE3Se9iXdSXzPXWoIR/VkaTOBwEzfowlpJeE1VWyuujGFZbY2kTittorxW0vdlMXpWHK1+WI3+sieG7+dkpmBZhsKigmo1++hDyP7/lUDInc/DEwN/iXYh3HQ0S9VIB48aKo+s3lp3odcKlgC3M5bmYl+/MVAEsNcUtQDLbYDf5qYW/K1/inzvb3hcYlqWUZzcacU75vXrgIBgpt4f7jVY5XntblQkMTrvKnq45Uu6qDmCTHhhEgNqdPNIGmVjF0tcCWVI3UTEA8B1qpTo2w2TzdBcIsQKBvyDo+wJcWOtybB1zZb6FrurGfygGA7pMMDZ4nDC7iTyXxsZ5UqCBDVYPv6CgJRcZ3BiPWE/T6Ii5bpWcvDqpB1aVOPABP+JRZ/fEF5xTyq+FJiNeLEgzupjOXxtO6GmNIMFtFX9IajNIcxLedQ436bNZu/QVrsE6ylb75zUov8WhS1ZQ5I96ub9ljThKJwJJt1KhtfgIWdP6ekwqh2xkStC44ssOUnY92oEA2/CPvn4Mwmk/8g31Ta/evXOVK9Pie1hJL/wI8nxzQ9OMqLpCMpmEkAXfxPCgJdMHSWi8J0KaEQ/6c6FEK0ry0/CX14KlEod0hDCzr7GZk0U09DO3V9ZBhW1tWv0xsm/gPAdJF7z19VwdKT6t5s9Pg2hy4BsQUY76f2fnSprEsF5Q+uFg3eQL3CI6yrO+OZYBcFPa6oGf0O+ypIJdhVIhzHh3pCV42E3N+iR855ROMKJRla6yJcAcY yPWx2NRl vsAJ1BQtRPWqP/RYPQUJgIuf1DDV0Z3SvhmQjilSZyE+w4Q+ySchXJYPpkJjRv+SGXaeBHvju5Pn21c5jbGu3kD/iCZPO38GxI3YWj2sWemw5pVA9mrds3AYbsQM0jzIMheEuFkXBlNBGT2mSzJHYBUzT/+cF/9Ek/HTTNY+xAV5pp/7KWKzJ1O3bbdr9BW/5QkdoXxhh1Qxjgc9ad8EiK+RexoCeJIJLkAixpuJplNh3Ft+Bbg50qswkM0gXsAW73PGun4v1ZPZZl8toahnXUgv+q2CN0QEnesJhk5e7HPzNL+1n5S7Xu4H3Vz6u3VA6mDbC1BN7APzppFbof6AOzJNqG/IcfsrGYyB+Ilz2PzeACvJ38YbEu410OYZ6uNZHxX4Xe77a3ngi2SyrcYxaAE/rW0jGKURKc0HeYIYIsOARJ541A/FQRJ+ZTGFoXfnsEhXbgKe7QjEVAR2kWYBPWRBOAB8184dDgsFCiKM/ooLVVZFqyxXmcj20sMYdi3+azWkm7pSgL1dvEDidExpjbLJe9Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add support for potentially switching address spaces from within interrupts/exceptions/NMIs etc. An interrupt does not automatically switch to the unrestricted address space. It can switch if needed to access some memory not available in the restricted address space, using the normal asi_exit call. On return from the outermost interrupt, if the target address space was the restricted address space (e.g. we were in the critical code path between ASI Enter and VM Enter), the restricted address space will be automatically restored. Otherwise, execution will continue in the unrestricted address space until the next explicit ASI Enter. In order to keep track of when to restore the restricted address space, an interrupt/exception nesting depth counter is maintained per-task. An alternative implementation without needing this counter is also possible, but the counter unlocks an additional nice-to-have benefit by allowing detection of whether or not we are currently executing inside an exception context, which would be useful in a later patch. Note that for KVM on SVM, this is not actually necessary as NMIs are in fact maskable via CLGI. It's not clear to me if VMX has something equivalent but we will need this infrastructure in place for userspace support anyway. Signed-off-by: Junaid Shahid Signed-off-by: Brendan Jackman --- arch/x86/include/asm/asi.h | 68 ++++++++++++++++++++++++++++++++++++++-- arch/x86/include/asm/idtentry.h | 50 ++++++++++++++++++++++++----- arch/x86/include/asm/processor.h | 5 +++ arch/x86/kernel/process.c | 2 ++ arch/x86/kernel/traps.c | 22 +++++++++++++ arch/x86/mm/asi.c | 5 ++- include/asm-generic/asi.h | 10 ++++++ 7 files changed, 151 insertions(+), 11 deletions(-) diff --git a/arch/x86/include/asm/asi.h b/arch/x86/include/asm/asi.h index 04ba2ec7fd28..df34a8c0560b 100644 --- a/arch/x86/include/asm/asi.h +++ b/arch/x86/include/asm/asi.h @@ -127,6 +127,11 @@ void asi_relax(void); /* Immediately exit the restricted address space if in it */ void asi_exit(void); +static inline void asi_init_thread_state(struct thread_struct *thread) +{ + thread->asi_state.intr_nest_depth = 0; +} + /* The target is the domain we'll enter when returning to process context. */ static __always_inline struct asi *asi_get_target(struct task_struct *p) { @@ -167,9 +172,10 @@ static __always_inline bool asi_is_relaxed(void) /* * Is the current task in the critical section? * - * This is just the inverse of !asi_is_relaxed(). We have both functions in order to - * help write intuitive client code. In particular, asi_is_tense returns false - * when ASI is disabled, which is judged to make user code more obvious. + * This is just the inverse of !asi_is_relaxed(). We have both functions in + * order to help write intuitive client code. In particular, asi_is_tense + * returns false when ASI is disabled, which is judged to make user code more + * obvious. */ static __always_inline bool asi_is_tense(void) { @@ -181,6 +187,62 @@ static __always_inline pgd_t *asi_pgd(struct asi *asi) return asi ? asi->pgd : NULL; } +static __always_inline void asi_intr_enter(void) +{ + if (static_asi_enabled() && asi_is_tense()) { + current->thread.asi_state.intr_nest_depth++; + barrier(); + } +} + +void __asi_enter(void); + +static __always_inline void asi_intr_exit(void) +{ + if (static_asi_enabled() && asi_is_tense()) { + /* + * If an access to sensitive memory got reordered after the + * decrement, the #PF handler for that access would see a value + * of 0 for the counter and re-__asi_enter before returning to + * the faulting access, triggering an infinite PF loop. + */ + barrier(); + + if (--current->thread.asi_state.intr_nest_depth == 0) { + /* + * If the decrement got reordered after __asi_enter, an + * interrupt that came between __asi_enter and the + * decrement would always see a nonzero value for the + * counter so it wouldn't call __asi_enter again and we + * would return to process context in the wrong address + * space. + */ + barrier(); + __asi_enter(); + } + } +} + +/* + * Returns the nesting depth of interrupts/exceptions that have interrupted the + * ongoing critical section. If the current task is not in a critical section + * this is 0. + */ +static __always_inline int asi_intr_nest_depth(void) +{ + return current->thread.asi_state.intr_nest_depth; +} + +/* + * Remember that interrupts/exception don't count as the critical section. If + * you want to know if the current task is in the critical section use + * asi_is_tense(). + */ +static __always_inline bool asi_in_critical_section(void) +{ + return asi_is_tense() && !asi_intr_nest_depth(); +} + #define INIT_MM_ASI(init_mm) \ .asi_init_lock = __MUTEX_INITIALIZER(init_mm.asi_init_lock), diff --git a/arch/x86/include/asm/idtentry.h b/arch/x86/include/asm/idtentry.h index 749c7411d2f1..446aed5ebe18 100644 --- a/arch/x86/include/asm/idtentry.h +++ b/arch/x86/include/asm/idtentry.h @@ -12,6 +12,7 @@ #include #include +#include typedef void (*idtentry_t)(struct pt_regs *regs); @@ -55,12 +56,15 @@ static __always_inline void __##func(struct pt_regs *regs); \ \ __visible noinstr void func(struct pt_regs *regs) \ { \ - irqentry_state_t state = irqentry_enter(regs); \ + irqentry_state_t state; \ \ + asi_intr_enter(); \ + state = irqentry_enter(regs); \ instrumentation_begin(); \ __##func (regs); \ instrumentation_end(); \ irqentry_exit(regs, state); \ + asi_intr_exit(); \ } \ \ static __always_inline void __##func(struct pt_regs *regs) @@ -102,12 +106,15 @@ static __always_inline void __##func(struct pt_regs *regs, \ __visible noinstr void func(struct pt_regs *regs, \ unsigned long error_code) \ { \ - irqentry_state_t state = irqentry_enter(regs); \ + irqentry_state_t state; \ \ + asi_intr_enter(); \ + state = irqentry_enter(regs); \ instrumentation_begin(); \ __##func (regs, error_code); \ instrumentation_end(); \ irqentry_exit(regs, state); \ + asi_intr_exit(); \ } \ \ static __always_inline void __##func(struct pt_regs *regs, \ @@ -139,7 +146,16 @@ static __always_inline void __##func(struct pt_regs *regs, \ * is required before the enter/exit() helpers are invoked. */ #define DEFINE_IDTENTRY_RAW(func) \ -__visible noinstr void func(struct pt_regs *regs) +static __always_inline void __##func(struct pt_regs *regs); \ + \ +__visible noinstr void func(struct pt_regs *regs) \ +{ \ + asi_intr_enter(); \ + __##func (regs); \ + asi_intr_exit(); \ +} \ + \ +static __always_inline void __##func(struct pt_regs *regs) /** * DEFINE_FREDENTRY_RAW - Emit code for raw FRED entry points @@ -178,7 +194,18 @@ noinstr void fred_##func(struct pt_regs *regs) * is required before the enter/exit() helpers are invoked. */ #define DEFINE_IDTENTRY_RAW_ERRORCODE(func) \ -__visible noinstr void func(struct pt_regs *regs, unsigned long error_code) +static __always_inline void __##func(struct pt_regs *regs, \ + unsigned long error_code); \ + \ +__visible noinstr void func(struct pt_regs *regs, unsigned long error_code)\ +{ \ + asi_intr_enter(); \ + __##func (regs, error_code); \ + asi_intr_exit(); \ +} \ + \ +static __always_inline void __##func(struct pt_regs *regs, \ + unsigned long error_code) /** * DECLARE_IDTENTRY_IRQ - Declare functions for device interrupt IDT entry @@ -209,14 +236,17 @@ static void __##func(struct pt_regs *regs, u32 vector); \ __visible noinstr void func(struct pt_regs *regs, \ unsigned long error_code) \ { \ - irqentry_state_t state = irqentry_enter(regs); \ + irqentry_state_t state; \ u32 vector = (u32)(u8)error_code; \ \ + asi_intr_enter(); \ + state = irqentry_enter(regs); \ instrumentation_begin(); \ kvm_set_cpu_l1tf_flush_l1d(); \ run_irq_on_irqstack_cond(__##func, regs, vector); \ instrumentation_end(); \ irqentry_exit(regs, state); \ + asi_intr_exit(); \ } \ \ static noinline void __##func(struct pt_regs *regs, u32 vector) @@ -256,12 +286,15 @@ static __always_inline void instr_##func(struct pt_regs *regs) \ \ __visible noinstr void func(struct pt_regs *regs) \ { \ - irqentry_state_t state = irqentry_enter(regs); \ + irqentry_state_t state; \ \ + asi_intr_enter(); \ + state = irqentry_enter(regs); \ instrumentation_begin(); \ instr_##func (regs); \ instrumentation_end(); \ irqentry_exit(regs, state); \ + asi_intr_exit(); \ } \ \ void fred_##func(struct pt_regs *regs) \ @@ -295,12 +328,15 @@ static __always_inline void instr_##func(struct pt_regs *regs) \ \ __visible noinstr void func(struct pt_regs *regs) \ { \ - irqentry_state_t state = irqentry_enter(regs); \ + irqentry_state_t state; \ \ + asi_intr_enter(); \ + state = irqentry_enter(regs); \ instrumentation_begin(); \ instr_##func (regs); \ instrumentation_end(); \ irqentry_exit(regs, state); \ + asi_intr_exit(); \ } \ \ void fred_##func(struct pt_regs *regs) \ diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h index a42f03ff3edc..5b10b3c09b6a 100644 --- a/arch/x86/include/asm/processor.h +++ b/arch/x86/include/asm/processor.h @@ -494,6 +494,11 @@ struct thread_struct { struct { /* Domain to enter when returning to process context. */ struct asi *target; + /* + * The depth of interrupt/exceptions interrupting an ASI + * critical section + */ + int intr_nest_depth; } asi_state; #endif diff --git a/arch/x86/kernel/process.c b/arch/x86/kernel/process.c index b8441147eb5e..ca2391079e59 100644 --- a/arch/x86/kernel/process.c +++ b/arch/x86/kernel/process.c @@ -96,6 +96,8 @@ int arch_dup_task_struct(struct task_struct *dst, struct task_struct *src) #ifdef CONFIG_VM86 dst->thread.vm86 = NULL; #endif + asi_init_thread_state(&dst->thread); + /* Drop the copied pointer to current's fpstate */ dst->thread.fpu.fpstate = NULL; diff --git a/arch/x86/kernel/traps.c b/arch/x86/kernel/traps.c index 4fa0b17e5043..ca0d0b9fe955 100644 --- a/arch/x86/kernel/traps.c +++ b/arch/x86/kernel/traps.c @@ -64,6 +64,7 @@ #include #include #include +#include #include #include #include @@ -414,6 +415,27 @@ DEFINE_IDTENTRY_DF(exc_double_fault) } #endif + /* + * Do an asi_exit() only here because a #DF usually indicates + * the system is in a really bad state, and we don't want to + * cause any additional issue that would prevent us from + * printing a correct stack trace. + * + * The additional issues are not related to a possible triple + * fault, which can only occurs if a fault is encountered while + * invoking this handler, but here we are already executing it. + * Instead, an ASI-induced #PF here could potentially end up + * getting another #DF. For example, if there was some issue in + * invoking the #PF handler. The handler for the second #DF + * could then again cause an ASI-induced #PF leading back to the + * same recursion. + * + * This is not needed in the espfix64 case above, since that + * code is about turning a #DF into a #GP which is okay to + * handle in the restricted domain. That's also why we don't + * asi_exit() in the #GP handler. + */ + asi_exit(); irqentry_nmi_enter(regs); instrumentation_begin(); notify_die(DIE_TRAP, str, regs, error_code, X86_TRAP_DF, SIGSEGV); diff --git a/arch/x86/mm/asi.c b/arch/x86/mm/asi.c index 21207a3e8b17..2cd8e93a4415 100644 --- a/arch/x86/mm/asi.c +++ b/arch/x86/mm/asi.c @@ -171,7 +171,7 @@ void asi_destroy(struct asi *asi) } EXPORT_SYMBOL_GPL(asi_destroy); -static noinstr void __asi_enter(void) +noinstr void __asi_enter(void) { u64 asi_cr3; struct asi *target = asi_get_target(current); @@ -186,6 +186,7 @@ static noinstr void __asi_enter(void) * disabling preemption should be fine. */ VM_BUG_ON(preemptible()); + VM_BUG_ON(current->thread.asi_state.intr_nest_depth != 0); if (!target || target == this_cpu_read(curr_asi)) return; @@ -246,6 +247,8 @@ noinstr void asi_exit(void) asi = this_cpu_read(curr_asi); if (asi) { + WARN_ON_ONCE(asi_in_critical_section()); + if (asi->class->ops.pre_asi_exit) asi->class->ops.pre_asi_exit(); diff --git a/include/asm-generic/asi.h b/include/asm-generic/asi.h index d0a451f9d0b7..fa0bbf899a09 100644 --- a/include/asm-generic/asi.h +++ b/include/asm-generic/asi.h @@ -38,6 +38,8 @@ static inline bool asi_is_relaxed(void) { return true; } static inline bool asi_is_tense(void) { return false; } +static inline bool asi_in_critical_section(void) { return false; } + static inline void asi_exit(void) { } static inline bool asi_is_restricted(void) { return false; } @@ -48,6 +50,14 @@ static inline struct asi *asi_get_target(struct task_struct *p) { return NULL; } static inline pgd_t *asi_pgd(struct asi *asi) { return NULL; } +static inline void asi_init_thread_state(struct thread_struct *thread) { } + +static inline void asi_intr_enter(void) { } + +static inline int asi_intr_nest_depth(void) { return 0; } + +static inline void asi_intr_exit(void) { } + #define static_asi_enabled() false static inline void asi_check_boottime_disable(void) { }