From patchwork Thu Mar 7 13:39:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13585600 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4A8D2C54E49 for ; Thu, 7 Mar 2024 13:39:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C8C046B0181; Thu, 7 Mar 2024 08:39:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C3DB16B0183; Thu, 7 Mar 2024 08:39:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9F3616B0184; Thu, 7 Mar 2024 08:39:25 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 89C4A6B0181 for ; Thu, 7 Mar 2024 08:39:25 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 544BDA1816 for ; Thu, 7 Mar 2024 13:39:25 +0000 (UTC) X-FDA: 81870349890.15.55CE6AA Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf04.hostedemail.com (Postfix) with ESMTP id 4F48A40017 for ; Thu, 7 Mar 2024 13:39:23 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=kXVyWS8H; spf=pass (imf04.hostedemail.com: domain of 3isPpZQoKCO0ndhgnPWbTSVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--yosryahmed.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3isPpZQoKCO0ndhgnPWbTSVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709818763; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MPmHpgEfpH39wv9P3N7+5I+Np9ZrrLnfHGs5cKPvN1A=; b=ALs/ruDTFPlpNMPhSaYCozzasCGdW9xpO+476DC2faxHqwqtXODTpKpyq0dDlGkjeKYQGG Outpk8gYcV9blmqfJnZJzNVoH/4I66/WP80Ivk1L6wFY7BvOPw7EPrGF8YMzoZPoGMtYee 6sRhePNtICfAPlV6b+8djWN5hvObmS8= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709818763; a=rsa-sha256; cv=none; b=RwZ74Sbg4TCYjAHF8OQGX/IUCoKFxHC3rjI4JBrJfLKatYyGV2jhT+je/q6z/9ZlFLhXY9 sap5mDXxNOSLUPTF+j6GmLJplEiuvBB8hpNQIpZ8+jAAVEsjZH+P6QJOgJIq59bOH4Vn2r +HqFafR8/12XWBnrixk30+XyfSenUMs= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=kXVyWS8H; spf=pass (imf04.hostedemail.com: domain of 3isPpZQoKCO0ndhgnPWbTSVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--yosryahmed.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3isPpZQoKCO0ndhgnPWbTSVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dc74ac7d015so1160516276.0 for ; Thu, 07 Mar 2024 05:39:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1709818762; x=1710423562; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=MPmHpgEfpH39wv9P3N7+5I+Np9ZrrLnfHGs5cKPvN1A=; b=kXVyWS8HUj1PF4hL+o0KEuQyn4iyyeL0SpU7Pxjf8Rzj90lIrIksHYHhuZa5pWyVdD cdFevNzJH2/vzpBoLuyjYqStzL3BIi/p+yeaSzl3r0i8Dhy1SNueSiE4IFpd9FKw+Ilg JfHelsINLVyRVAkTieiu2CcPlPTrKUSZFYhwL8DNCRaqpWsESkrft5dDTyPr3qfV4q9l 6Lf/FDOKe9uaP4aCeEwz70uPcbiTqXae+zLqx/sfo3SBog9d+wpwk9Mvy4QYcoSEBEuu A2T5lBDEkCzibWpGAlr8SDtoQX8Zxyi//DvDpTlxGZ004BUHRs9ufkSbn/1E3CblA5wi dBAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1709818762; x=1710423562; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=MPmHpgEfpH39wv9P3N7+5I+Np9ZrrLnfHGs5cKPvN1A=; b=FIgHFm55k0g9bvMGBMXQRTcs6Pzo9A/1PHhv41dVoLSXrPEQfR2NhGA/RKsqldDxwa Fkv4azEa7coTuHM2z2jhgo+l+fSpUA97IU8YNyoonNT5EFYYuNuxm8lQ1/uhbCovlyBO Q2LlKHuefKuEuUW6Kkr0jQeRQBonnRNXt6Hn9vPJXldx3nywvjvuYNVxyc04O9Iq32gG xW7hs4U7pk0a9i4BzZI98xDNrt7zAcUx/n9SphZi4+mQDxaeITlXRGDkFotFXJGoM3ka 8YdU8ursOp+QrLPljcooxvme7Rdn7yh3KKn3+otRV45LgA3UE6Dj+OtXRwv4Cg2jIUeL rm2w== X-Forwarded-Encrypted: i=1; AJvYcCVJ8vEVoJqZGJIrCya3Oxf71GjYOeeVeIwbe6QdH9YFzFEmwWETApG/hXK0/GzdQMo4tgkvKTI40PjWWFYed9bSWyI= X-Gm-Message-State: AOJu0YxWtQvaU5h2lh96+ZfSSDY/v8XHmkj/r/nAcEpCY5UnOKMMUJAP xlqdnxR7r4ulG3dSQlJcxHGkiIXjVMIoilAt3OhKfCZ4ZLxu9oMqWsxj8R27DoatROcZp/lhxcs O0sIy4p9lRW9IqCXoSQ== X-Google-Smtp-Source: AGHT+IERNwqp0BbMH+HdatOkbB3X7SeaUZ0qNtYllODJdaNR0SQYYHmwI9BfJm6ZZYu3OtMlHLptB1kSZCzrhOZk X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:29b4]) (user=yosryahmed job=sendgmr) by 2002:a05:6902:2183:b0:dcb:fb69:eadc with SMTP id dl3-20020a056902218300b00dcbfb69eadcmr781549ybb.6.1709818762290; Thu, 07 Mar 2024 05:39:22 -0800 (PST) Date: Thu, 7 Mar 2024 13:39:15 +0000 In-Reply-To: <20240307133916.3782068-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20240307133916.3782068-1-yosryahmed@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240307133916.3782068-3-yosryahmed@google.com> Subject: [RFC PATCH 2/3] x86/mm: make sure LAM is up-to-date during context switching From: Yosry Ahmed To: Andrew Morton Cc: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , Peter Zijlstra , Andy Lutomirski , "Kirill A. Shutemov" , x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yosry Ahmed X-Stat-Signature: ox7t866pxe9wh6d4rq6hn65c7311m8u4 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 4F48A40017 X-Rspam-User: X-HE-Tag: 1709818763-533278 X-HE-Meta: U2FsdGVkX186QPnL1tbs3PlgFRGswh+sPMZ7cijjbM1bb7IWe+X5DTxAleu7ttLbSv30fIrqhF263isbE+9D7WRjIMha2eaTJK7o2fWuZ3SYOPRtkYLhZwDpehw5lqze5nj6LXi/Flz1vTucw9b0NDAZfJpJWK2DQYO3cvY5LJNDIxZNuTPGhC7Ip6oOTYbFdEuM5nl7GxARyh1uIGRYBnU2qisEw/HGMyXfs0pYnaX+CyA/yhskIegYsQi5LLc8cFMDC1l985KYcfifOl/GD5X+DEqLLc9i9G0yrHI+4FznSh/gWUgvOxcuK1i+uXREoGLwcO/vtZT05MPYJIGS0HIO/c+ASYcsLpifA1KlzT6j4ZHHIMxgiW9YoXfgEBah8fAbN1AZ99znX5QRAE9r2sdQSYk5ivdqQ+oL7VpSj3Q8T/yQgbV7I5CURYyY/Nv15YHmaH0wUMhrsPlWrPZDn64fFIgHGseJy3oLG4+OmyZQi/WcsMk/r2gDFnMtHmtOnmbddW8sFCCgiZH1NGVn/14YVpIu2ugL85ZxYJdj1JM+CqFr/or15vmJqw5TnX1q3QTeJ66MNNpX5wrlr9IHxcZCRoieDLYrcNHv5OXau38he2IYFx1FMxtsi3cd0CG7s3U3ghH05mG5LmcHhL7nPdgzgGx4iEojA5/+uip6PIh00S1b1iqiv8j0Vjzf+VTN9iO4Ue8J5DZlGedqmYGHdfMHFuLN4kHDiv6k1asw+HdWMITQsvL3GALM3Sy6+c01319r6vH19IVijWiwQeCNHvzJXKe66+83P0rgYqPUUZxLmOt9zLy5DAGWT8Qt43E71T0iuHq4YfZyK6AToEqbiN6CMABmLCiSofp2Bk4+mnoo//CxQvKJsUrYSdsj8Hn8snNJCIu3Vui2D31gCPtUn9FqXYSd6rVjsWJa8n7vhkyNzeGNFRm2dy9WVd7Ieuw1dfi9zEkrh4XF1juyBX/ 9pW8CqcW 9IYpxF8ieJIaNJt7odXovO6TN276qj2KrUuDUF9tnm5rhroA2HCpEfowbLZKueYBqsdVCJncXS1S7fFNn4od7KJMTbQZW7sK6TiGxrFDBkH+2q6Yhlwhv41pmYsIdAJZXrWx5+o45fyENIZy/Hofh7GQk2baZOPbOz5x6uCQ7hulnzB6LNYbFSEc5giRA80fkSXsd+EQ3rgwl+Jn29DdMCPtnpBcykWw+j9L7qVkZ8hz9QjFQXXHoTM0VZxBXK2fFXay0N8dsD48sm87vneBmDC8oBaJyVKiiNHFEgA+oIrD6rl8EybGg96Es0zgezC03ehpn1hTvfFswMpJbdE/w7dtKCI25l3Qlaw8k3apa9+RaZ30KmXurV1LXmeH1HCk0mU57Zijkd15CJoxhXcq/t1g8Dwm/W6Rk+vmZ4BdEsnYvJQ1YTbnROIdgMyOMAXfQADH31VX4CANgN7WnjusWhnC+J66EQ/IjsVzN1TlCalxWHK1IGWXg83/PA11fFfjjkapva/ZePqgLFXv5L7tNeSYcdNNcTfkZnFhomvLFLsk6GbVpHZx3+VgbJ4NGRfMkEYeEeQ4XZTvbnz/ud56iAn1QzmoiCVIusbOUXiV1y5OIZgBE1jxaTkCg3WDZVTPFPfEc6LOHEe7/+hk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: During context switching, if we are not switching to new mm and no TLB flush is needed, we do not write CR3. However, it is possible that a user thread enables LAM while a kthread is running on a different CPU with the old LAM CR3 mask. If the kthread context switches into any thread of that user process, it may not write CR3 with the new LAM mask, which would cause the user thread to run with a misconfigured CR3 that disables LAM on the CPU. Fix this by making sure we write a new CR3 if LAM is not up-to-date. No problems were observed in practice, this was found by code inspection. Not that it is possible that mm->context.lam_cr3_mask changes throughout switch_mm_irqs_off(). But since LAM can only be enabled by a single-threaded process on its own behalf, in that case we cannot be switching to a user thread in that same process, we can only be switching to another kthread using the borrowed mm or a different user process, which should be fine. Fixes: 82721d8b25d7 ("x86/mm: Handle LAM on context switch") Signed-off-by: Yosry Ahmed --- arch/x86/mm/tlb.c | 50 ++++++++++++++++++++++++++++------------------- 1 file changed, 30 insertions(+), 20 deletions(-) diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 2975d3f89a5de..3610c23499085 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -503,11 +503,12 @@ void switch_mm_irqs_off(struct mm_struct *unused, struct mm_struct *next, { struct mm_struct *prev = this_cpu_read(cpu_tlbstate.loaded_mm); u16 prev_asid = this_cpu_read(cpu_tlbstate.loaded_mm_asid); + u64 cpu_tlb_gen = this_cpu_read(cpu_tlbstate.ctxs[prev_asid].tlb_gen); bool was_lazy = this_cpu_read(cpu_tlbstate_shared.is_lazy); + bool need_flush = false, need_lam_update = false; unsigned cpu = smp_processor_id(); unsigned long new_lam; u64 next_tlb_gen; - bool need_flush; u16 new_asid; /* We don't want flush_tlb_func() to run concurrently with us. */ @@ -570,32 +571,41 @@ void switch_mm_irqs_off(struct mm_struct *unused, struct mm_struct *next, !cpumask_test_cpu(cpu, mm_cpumask(next)))) cpumask_set_cpu(cpu, mm_cpumask(next)); + /* + * tlbstate_lam_cr3_mask() may be outdated if a different thread + * has enabled LAM while we were borrowing its mm on this CPU. + * Make sure we update CR3 in case we are switching to another + * thread in that process. + */ + if (tlbstate_lam_cr3_mask() != mm_lam_cr3_mask(next)) + need_lam_update = true; + /* * If the CPU is not in lazy TLB mode, we are just switching * from one thread in a process to another thread in the same * process. No TLB flush required. */ - if (!was_lazy) - return; + if (was_lazy) { + /* + * Read the tlb_gen to check whether a flush is needed. + * If the TLB is up to date, just use it. The barrier + * synchronizes with the tlb_gen increment in the TLB + * shootdown code. + */ + smp_mb(); + next_tlb_gen = atomic64_read(&next->context.tlb_gen); + if (cpu_tlb_gen < next_tlb_gen) { + /* + * TLB contents went out of date while we were + * in lazy mode. + */ + new_asid = prev_asid; + need_flush = true; + } + } - /* - * Read the tlb_gen to check whether a flush is needed. - * If the TLB is up to date, just use it. - * The barrier synchronizes with the tlb_gen increment in - * the TLB shootdown code. - */ - smp_mb(); - next_tlb_gen = atomic64_read(&next->context.tlb_gen); - if (this_cpu_read(cpu_tlbstate.ctxs[prev_asid].tlb_gen) == - next_tlb_gen) + if (!need_flush && !need_lam_update) return; - - /* - * TLB contents went out of date while we were in lazy - * mode. Fall through to the TLB switching code below. - */ - new_asid = prev_asid; - need_flush = true; } else { /* * Apply process to process speculation vulnerability