From patchwork Tue Jul 2 13:21:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13719647 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7EEADC3064D for ; Tue, 2 Jul 2024 13:21:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0B4C26B0095; Tue, 2 Jul 2024 09:21:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 03F066B0096; Tue, 2 Jul 2024 09:21:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D36D36B0098; Tue, 2 Jul 2024 09:21:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id AD48E6B0095 for ; Tue, 2 Jul 2024 09:21:47 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 650DD12207F for ; Tue, 2 Jul 2024 13:21:47 +0000 (UTC) X-FDA: 82294875054.20.5FF0255 Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf27.hostedemail.com (Postfix) with ESMTP id A027940008 for ; Tue, 2 Jul 2024 13:21:45 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Y8ZKZE8U; spf=pass (imf27.hostedemail.com: domain of 36P6DZgoKCDElbfelNUZRQTbbTYR.PbZYVahk-ZZXiNPX.beT@flex--yosryahmed.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=36P6DZgoKCDElbfelNUZRQTbbTYR.PbZYVahk-ZZXiNPX.beT@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1719926483; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/1dVnzezxakkrWVDD24n/jxIBS1JtulZK9cr/kIqgsY=; b=F7tUZpyIg5f7Mun//8xKXcEDKpKrTUlODQLd9DUYkXpH5bx2bfaMNCJ44nOGs7XUC8VMNZ 4rtJF3oLm9jpjlnnf5ROE5PELiHWZ7Q/T9es1s05MEl1VfdCsnBiQc1uo5kwNOMGcnkDIN CFKvWcPGuXPgYaVsJ2CLDJkXG9wqjj4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1719926483; a=rsa-sha256; cv=none; b=tR8HRlQAAIBktvwW3q9oRiYNAq5bxHq9YDPH1qBZA063oL5cz4ObBRmGLQb79EfGukV1pn 2XC8LZSJxjtK6sMfRYQeViBFoQvC1A5VtJqc/xr9xnzv06DX7gBd1kB2c4hozQ0UsOk6+3 zJqChcV5M9uTu4/Z0JD7w9Hc9uq0SYo= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=Y8ZKZE8U; spf=pass (imf27.hostedemail.com: domain of 36P6DZgoKCDElbfelNUZRQTbbTYR.PbZYVahk-ZZXiNPX.beT@flex--yosryahmed.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=36P6DZgoKCDElbfelNUZRQTbbTYR.PbZYVahk-ZZXiNPX.beT@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-62f8a1b2969so73931597b3.3 for ; Tue, 02 Jul 2024 06:21:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1719926505; x=1720531305; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=/1dVnzezxakkrWVDD24n/jxIBS1JtulZK9cr/kIqgsY=; b=Y8ZKZE8U+A5pnqu9ulvw3399sOxI+gMtM+jxlnC80VdWLDSUX3llgPO0Th1qvfKM7m xwOBc7ZVV3WgrxpCYlQGnKEt+8cN1P6n3rX8x+9rVFJx6wJWbCALoVpUnOBXiBQzluhx vOq2CVfrgrR0L2vLev94KcqZzsef3KKUP1yCNYsGEV1CKhFtDiL6EW7K+JwsJw3uXrBX NqdPEYJRzI0NPAxqOeRAQntuMaaojScF3am+gapdIp1SXytHmqZfwGG6VdfHu5Dp0H/i aA0BQZ0Chwf5fX4LfiHyDazYDMUSR/QMUGgvQzGfZQtTxgAG//Z/LioMUrV4pVSxgc8s EaCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719926505; x=1720531305; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/1dVnzezxakkrWVDD24n/jxIBS1JtulZK9cr/kIqgsY=; b=NNpMpeqMoCH02UIvC6N/W/bC8MSObecXarC6PioYhtcG6uHULm5c2PSrT6cuI+6h8Q LzsZagQNE4FhXWCd2VQjCXdI/+8rdBlWBH8E+UQpdQzulP3zencA4AOdZjT+Ppyd1uD5 jzCUqR9jtrleVXozZSwb+iyNksV4bAoYxEjB6TqtIfjXmDbJIUmFBskqnSYsERuZ/bxg soJub3GFDv/lAj/bOpMIm0TQmbdnwXyHiUuF0toiw08n+UttXp3ehgQjfZ2NXkPbT/ot DmTvj1wawub2hlWMeuVT3SvPkNR0AH4WypWVskgMXMhOQneu8a7sKjgITaG/WndmVp+t uQsg== X-Forwarded-Encrypted: i=1; AJvYcCUV4bkBvcI9FQH14a+Y7rGX61D/7wcVz7zXqHVD2duYWGsQU5+MBR1R+EIoLfwcy2TvMcmv1c1tPFDev6aoPfzviO4= X-Gm-Message-State: AOJu0YxLYLtZfBw2a/MeCgCqwmq4GJInoqiMtsRrAfLDBwNek9H7NN+Z QEFUI7jLSN2lZzLLiU6mFcj9Pvf80xmW25goF/li8CjzFchto0B3j/VTKR9McHpdmi4Wf1llkMS b10uiM2ou/N5iq2YLQw== X-Google-Smtp-Source: AGHT+IHghuJQ8nulRpb2JfS2RyKKlSW+eKRndDAd54bTQK3f3q6ozzQyqfLRhWOFhsZ+LsF1X/jGe3dnEw1HTZPM X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:29b4]) (user=yosryahmed job=sendgmr) by 2002:a0d:e885:0:b0:61c:89a4:dd5f with SMTP id 00721157ae682-64c72b5434amr740227b3.0.1719926504753; Tue, 02 Jul 2024 06:21:44 -0700 (PDT) Date: Tue, 2 Jul 2024 13:21:37 +0000 In-Reply-To: <20240702132139.3332013-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20240702132139.3332013-1-yosryahmed@google.com> X-Mailer: git-send-email 2.45.2.803.g4e1b14247a-goog Message-ID: <20240702132139.3332013-2-yosryahmed@google.com> Subject: [RESEND PATCH v3 1/3] x86/mm: Use IPIs to synchronize LAM enablement From: Yosry Ahmed To: x86@kernel.org Cc: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , "Kirill A. Shutemov" , Rick Edgecombe , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yosry Ahmed X-Rspamd-Queue-Id: A027940008 X-Stat-Signature: sfuxfbmxhu4ygjqoc49ucf7x5pe9ohur X-Rspamd-Server: rspam09 X-Rspam-User: X-HE-Tag: 1719926505-7548 X-HE-Meta: U2FsdGVkX19vFP6aZ+a16T/i0BhL04m9r0k52tZQUJ4GsnMsfID2yjyN3xc7TxdcUjV/eBRo1RKkLt1Kr9a+nkxAsimArRCbG8KbLaAHfopYOvlc1xmNB7imozAFTQ+KPYly4P2QYkq495GoF4VP5I7aCwk//M9+4ZkVsDM5DsPnokTW95/7NtlW3ZcfjieVBOBqwCNqEvcS9Sle3O7rW4BKQQ11QUQgmrlgcBK/Oerc5LDtwLdtphE3f1Ar5m76W4uBL/pgv+3sb0Wa6vNRu6AHdNEggZRxonEeQ7hmC+MR0LHNljWz5KLdK5xSqB1wuSU5IEwve/7IW/o4GLcJetFTP7ZS0Xx106TARg1gBAr/vMgoTV5oQbPGarCE1JrA1R4PNyxl563cUtoijmpkAbxcPuMygCSv9k4PDYE7lCKEJd3mfe8iAqHsTBpF+kVbdyxwRwO0I2nXfIp2MxaPuTPV6/rE7Z4Wp6qbT6KOBKKYLGtRuhdeA/nVQ52C0Le8ftM4yGS8h9sr7lMKqmWl+tjHQ8ZJMu0A9GOZNbTl2GElJU20JoowyCIBSXLdtoXNJqL+TWIbKhzr85kKU7X2+NwpNoygk4wrP5uY4mk4eC8IggNiHXlaIn2xQcogtjoDEnaHw9Y+DQx3UEvOI9B+YaXm9tONaSVC+LAphCIv8ECEJzX9Qw9Cw8W9s+ERG5eUEsfH3NNTMs98n9gRqhb/TtOa+/mGtXGhefUYjSPTYELf2VTxQIGxxF1oen4SUVbk/08XmyiNtSx9p/wWqN/jjilD+qZDc8/aJ7I0lNZYe9de8HrSeOJXkZmUO360rQdq4Fs8G0Jjlcj75WzOZGAvuyzWHBFj3NvBWY5mNUZhypou5HSHihlvq2Yau8Ku8JuDSxDnp2TnsfYzwR972lmknKE/n2i6nTowONVWCTs7jymNKzVlPh/2K4SFgsmIcmvaG6btwEa1fxzAh8T/3P2 5wJQ4Vef e3B/pccdRt/7DGqZ5wdQrJOOr47xkh0c2arDoTAC6Cn/uCVSn89XEc4duzPSLdz/ZfLbZcE6JMBslS7Q4bDrPSZ7b7mDxyyxSFI2kW78v2l48spLffBmteEczOyPF4UIyzp2WciklBrdlkr0bEfXRyVwxwS0vnztUmlpK+PcNZc6LN0IEtxncc2kEu4/FAfnP9ul/9TW9jCndcqvk30PGb82hqoLog0MO1kjvlMxKmp/1AsmR116zOs5S4EzCzTW0zkhWxDpPJfs3A6H9RkMfMLEaDxW3nRw4R6+5Ne/gHKc5aUFwH/Q/W5RdGQb0ruKei2Dl19ORhlLYxowBD/sjRGOhJjRiZxjgInggz4P2ofG494kvxycPz+5bIiePzuD3p0qT6XHqseuDMGqkJZOB1JWZdD/1fpN8dK2uHGSWoU7Pf+DuKjZw9gF2SOOMRojAZaeaXNP0Lv9SwPKPnjOgRvkrzfX2sdXmlMlxmCetb6aPcDmyQ7CUlH3dyYh57PI6TAr2yhnixqRpdgrHpEW4h/8C4vrzkXeLR1DegK9sT7u5rtB9nu5IXp64hvtycqhK5hhz33ApWbt0sLgMNgGldhEofivQt5v+iyNBovvGGIgz6mSA7oc5LpCnHrN8ofkjzdJS X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: LAM can only be enabled when a process is single-threaded. But _kernel_ threads can temporarily use a single-threaded process's mm. If LAM is enabled by a userspace process while a kthread is using its mm, the kthread will not observe LAM enablement (i.e. LAM will be disabled in CR3). This could be fine for the kthread itself, as LAM only affects userspace addresses. However, if the kthread context switches to a thread in the same userspace process, CR3 may or may not be updated because the mm_struct doesn't change (based on pending TLB flushes). If CR3 is not updated, the userspace thread will run incorrectly with LAM disabled, which may cause page faults when using tagged addresses. Example scenario: CPU 1 CPU 2 /* kthread */ kthread_use_mm() /* user thread */ prctl_enable_tagged_addr() /* LAM enabled on CPU 2 */ /* LAM disabled on CPU 1 */ context_switch() /* to CPU 1 */ /* Switching to user thread */ switch_mm_irqs_off() /* CR3 not updated */ /* LAM is still disabled on CPU 1 */ Synchronize LAM enablement by sending an IPI from prctl_enable_tagged_addr() to all CPUs running with the mm_struct to enable LAM. This makes sure LAM is enabled on CPU 1 in the above scenario before prctl_enable_tagged_addr() returns and userspace starts using tagged addresses, and before it's possible to run the userspace process on CPU 1. In switch_mm_irqs_off(), move reading the LAM mask until after mm_cpumask() is updated. This ensures that if an outdated LAM mask is written to CR3, an IPI is received to update it right after IRQs are re-enabled. Fixes: 82721d8b25d7 ("x86/mm: Handle LAM on context switch") Suggested-by: Andy Lutomirski Reviewed-by: Kirill A. Shutemov Change-Id: I7fd573a9db5fe5284d69bc46f9b3758f1f9fb467 Signed-off-by: Yosry Ahmed --- arch/x86/kernel/process_64.c | 13 +++++++++++-- arch/x86/mm/tlb.c | 7 +++---- 2 files changed, 14 insertions(+), 6 deletions(-) diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index 6d3d20e3e43a9..e1ce0dfd24258 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -798,6 +798,16 @@ static long prctl_map_vdso(const struct vdso_image *image, unsigned long addr) #define LAM_U57_BITS 6 +static void enable_lam_func(void *__mm) +{ + struct mm_struct *mm = __mm; + + if (this_cpu_read(cpu_tlbstate.loaded_mm) == mm) { + write_cr3(__read_cr3() | mm->context.lam_cr3_mask); + set_tlbstate_lam_mode(mm); + } +} + static int prctl_enable_tagged_addr(struct mm_struct *mm, unsigned long nr_bits) { if (!cpu_feature_enabled(X86_FEATURE_LAM)) @@ -830,8 +840,7 @@ static int prctl_enable_tagged_addr(struct mm_struct *mm, unsigned long nr_bits) return -EINVAL; } - write_cr3(__read_cr3() | mm->context.lam_cr3_mask); - set_tlbstate_lam_mode(mm); + on_each_cpu_mask(mm_cpumask(mm), enable_lam_func, mm, true); set_bit(MM_CONTEXT_LOCK_LAM, &mm->context.flags); mmap_write_unlock(mm); diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 44ac64f3a047c..a041d2ecd8380 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -503,9 +503,9 @@ void switch_mm_irqs_off(struct mm_struct *unused, struct mm_struct *next, { struct mm_struct *prev = this_cpu_read(cpu_tlbstate.loaded_mm); u16 prev_asid = this_cpu_read(cpu_tlbstate.loaded_mm_asid); - unsigned long new_lam = mm_lam_cr3_mask(next); bool was_lazy = this_cpu_read(cpu_tlbstate_shared.is_lazy); unsigned cpu = smp_processor_id(); + unsigned long new_lam; u64 next_tlb_gen; bool need_flush; u16 new_asid; @@ -619,9 +619,7 @@ void switch_mm_irqs_off(struct mm_struct *unused, struct mm_struct *next, cpumask_clear_cpu(cpu, mm_cpumask(prev)); } - /* - * Start remote flushes and then read tlb_gen. - */ + /* Start receiving IPIs and then read tlb_gen (and LAM below) */ if (next != &init_mm) cpumask_set_cpu(cpu, mm_cpumask(next)); next_tlb_gen = atomic64_read(&next->context.tlb_gen); @@ -633,6 +631,7 @@ void switch_mm_irqs_off(struct mm_struct *unused, struct mm_struct *next, barrier(); } + new_lam = mm_lam_cr3_mask(next); set_tlbstate_lam_mode(next); if (need_flush) { this_cpu_write(cpu_tlbstate.ctxs[new_asid].ctx_id, next->context.ctx_id); From patchwork Tue Jul 2 13:21:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13719648 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EDB6FC30658 for ; Tue, 2 Jul 2024 13:21:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4641B6B0099; Tue, 2 Jul 2024 09:21:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3EBB86B009A; Tue, 2 Jul 2024 09:21:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1EC0F6B009B; Tue, 2 Jul 2024 09:21:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id E733B6B0099 for ; Tue, 2 Jul 2024 09:21:49 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 5EC981C3753 for ; Tue, 2 Jul 2024 13:21:49 +0000 (UTC) X-FDA: 82294875138.08.0DA2ABA Received: from mail-yw1-f202.google.com (mail-yw1-f202.google.com [209.85.128.202]) by imf25.hostedemail.com (Postfix) with ESMTP id 8870BA001F for ; Tue, 2 Jul 2024 13:21:47 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="KV/Jbu1T"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf25.hostedemail.com: domain of 36v6DZgoKCDMndhgnPWbTSVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--yosryahmed.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=36v6DZgoKCDMndhgnPWbTSVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--yosryahmed.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1719926482; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3p0qtnzDEL/NiIySpMNGGFo+d5LOoZGd/e3zeHTmMTw=; b=I8scqpzLCcML7nzVeytf3UCF09JQbfFx2FtVRY16mH4eYZiRiFYz7lK8Se/BBTXiiMo7iv /HXmGXw25VBbcYTptTq+mK6rU3Ew+Cde4V6HhALppl5LZQ4zL1Md9a4fl6GelJiZ5k+8PJ /tH5fPDNKbxizNzB0GCmAtqW9Qt3KtE= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1719926482; a=rsa-sha256; cv=none; b=YlVGnsFdfiNXNgp81bl46RGc2sw9bs/k9wMSn4iwqLChfU5D9qbi+dC9yeIwlaDoYJv5Y+ h5ahJ1ora/lZ9IDF9qMnNqVi0C18CdSX+H7rAvGs44+v3vSXbENaTIZoRovn7VoHswHp3U dzuGH+z/Vy9m6FhnJxtpTUmI18WC02k= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="KV/Jbu1T"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf25.hostedemail.com: domain of 36v6DZgoKCDMndhgnPWbTSVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--yosryahmed.bounces.google.com designates 209.85.128.202 as permitted sender) smtp.mailfrom=36v6DZgoKCDMndhgnPWbTSVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--yosryahmed.bounces.google.com Received: by mail-yw1-f202.google.com with SMTP id 00721157ae682-65033922659so10130957b3.0 for ; Tue, 02 Jul 2024 06:21:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1719926506; x=1720531306; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=3p0qtnzDEL/NiIySpMNGGFo+d5LOoZGd/e3zeHTmMTw=; b=KV/Jbu1Td+lhQgrUbsq+7nIetkaEs95i4GjjwJtGu6kCZkcxwjtKpNwLbuxoPUNo0h 7G1IKvB/KIUiEWG0dXZ+sB+F1Pw9X5/1MsTkdQsoJsM4G+RhW64wrAL+oXrbLDcsuEqB Xo1Wz5mCsh7t+VxwsxB8D2tHiFSGp2ISsRWrH1RcUrQhfwHlz9M6qAWbNMGupaGmtH33 TVwHDMbpb7QgNG9Pxh41FIYwlLXm1x0+6pf4amIwVrDl6NthaVBUwIXNFQcjoIAuTbqu Jh+Bc3+3dmjg98MQDt3OTcc1NIbPqeZvZfMswTXVvN7C4J3myh5xOWKp1zbORCBYx2JD S4Rw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719926506; x=1720531306; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=3p0qtnzDEL/NiIySpMNGGFo+d5LOoZGd/e3zeHTmMTw=; b=JsCFFM6NEDj86YGWI9+7Z7lnaodsVfVW0rFj5L/zcZ31DE1KoQLGb1z+GaGJvpvaS/ 2RNR2/p4U25XcS0AbRw+OCNe6j6Vk2+8TshKOg6ocoqJm+v0a133Y735FbH5/oKhCqlS N/3y+wTf+xkg1DG48tnteoecNdBbyQ7+6zCdAoIwgNx6EdnlBTTrWOmGgMdcMIck4vLM DaGWRoXIrmejrgSfcyzc0CApSCB8CJTc+8Nsi5BrL/k7DNRanU59+IhSxHd8BgJddOZi KXTrM6YHm2uUcT+jxK9WV/4vuTTlkiIgFTU4MBc+krWj06NKDVwYdMbsUqgum+cH1wtH opnw== X-Forwarded-Encrypted: i=1; AJvYcCV9qQnpbrcIGm9BQLYtveYY8iVSSjHxKPUbJEtplOMZE7iuT4wO4InjwnpN1AOkED6A4lRYY6IOYsFXPllO7pkQuEk= X-Gm-Message-State: AOJu0Ywk9mU3pe68GiH2NfoQ0ytgQ/XgVf6SZ14uH+bEnuiQ87DNTHRP XC0PkWm1SslWpI170l1zX2SSSl7bSKLbuiCt4ty+HlPVD55iRhaHGdkxv6xjtN3ok9xKC7AgJZk UJ1FV6K8GM7Uh/lWtsQ== X-Google-Smtp-Source: AGHT+IE5yOxXHFsN9w/WFqj9GD2Jlu4F+FUycVLVxC5570M2iHpzwIws1dqPsB11Seccelx0/1egw1GKNVlINZNd X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:29b4]) (user=yosryahmed job=sendgmr) by 2002:a0d:e644:0:b0:62c:f623:3a4e with SMTP id 00721157ae682-64c6e5c126dmr289017b3.0.1719926506628; Tue, 02 Jul 2024 06:21:46 -0700 (PDT) Date: Tue, 2 Jul 2024 13:21:38 +0000 In-Reply-To: <20240702132139.3332013-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20240702132139.3332013-1-yosryahmed@google.com> X-Mailer: git-send-email 2.45.2.803.g4e1b14247a-goog Message-ID: <20240702132139.3332013-3-yosryahmed@google.com> Subject: [RESEND PATCH v3 2/3] x86/mm: Fix LAM inconsistency during context switch From: Yosry Ahmed To: x86@kernel.org Cc: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , "Kirill A. Shutemov" , Rick Edgecombe , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yosry Ahmed X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 8870BA001F X-Stat-Signature: gkcswt3bimnae7fao8shzq49w4s6zk4c X-Rspam-User: X-HE-Tag: 1719926507-786884 X-HE-Meta: U2FsdGVkX18maGm5w0Yq2FR+QMbOssQemsoxNIEcRf5t6kYpXJYjqUvXKsaFnc8Bsn/GSTcwAh5TbZSqXse9dcRdBuaACUcjWqSV6ALuwTrVppo1tiI05lmLn74JdY3u0OXn6thdaMSASYumUTdBzy6tzrKNsy88oU24LC3o8Jn4cNYmhSXYy9VTXQ5qxQy1tQixRJGxzInF20PyB7Uqv5/myTF+MGo2zGydDB53ecodyDm/Qm4wJEGIbEL7iTLE3fsK8eN9x2JKdXBYTxQxxi+JyY13AIMrkPGuEA+1PsXPJouH6wvWpBxWNJcHQBS5g+1qxSM2WCWfCRJlu/tHLLlhOyC63LfRrjDwQ33AdGf2KlzYWKlYePgXGNHZgZRxX+14uoMknB9nhyf5KwxeBnqddx6Ttwh2JS2HMdKMqBLtyXfl84+a3ajKloWR+6E6C+YT/j872IkR6m7kFXh8TrTFlYBXCEln79uCmreDetsKPJPj/9/VO0kdmz9L+T4GHdXM1lb1TFWWrdXyOJt/rvMd6hs1P+NHyQl+2eMNnwyNdYcK6dpt0bdaWy6u8pCibEO0pTN27k8pU38mPuIsdjDsG7yrmXmeZyzU4Uvxt/YhDDtMN6Cs79vCpsY3yd/1FOwBjRORjdpbGKLYT9MgOIvdA8M+kuNdn3nUCuhNkqrc7DCHIzk3EzkEASzwj4I0GRcr19Bb5szxQA0z0fbig9TBTFBEUJPQF1drCgEuBIMBB4g8xq2rxuAvVyuCneRcmqR9+Ldcr3zxPMabVt8FAkW4o96EEKCgrkvpUAlaZsb3LfXOz0WEz8j4e32VoqXsWyjsBQv/h7Pnrxqu9uA7uLjenfi0iBapn+ir3CLkN+n4MGH4wokxp0vx7NYhtKkFGAnY9gPBevrgLtGFeV8WDhXtke++XdkkkBFN3KDkAMcQ/t+DBlEWCUXuD199TnV2ncOmFMp92CldO+zEPf9 AbOeYhqa cZ+6nhAzsDFkLj+4mE8V7t+KfQBu3CwjY6AVcayhT83Khj59oiRFZxfx/XVr6LDFCOdYWNeu0415xaC7UWj7GJRnI3Xr15VChe7lkMsqOSrhBMTUHEtY7pZM2gNSi/qQuOx99mHAPRvLXS7KlrPTVvtnxD1EyjtL1OzkD/ksq/FsGIh7qEkrH7T0KHAw2Co6PlEDF/1Px18BNoexfwfGXEc7Qjn/63Tyb+REkI6JHDNUtZ9HZr4swnmsiyIgWMllu//fGbut8r1yE2AF3pqA/+/1PIVlOOkiR6dbJSAHJmn5RMeF0cY7w3xPvqjKwClHBI1giTBaYnvsQ0khsl4fPGqK0+x35fJy9MfNCC1JNgKZMsLcSIiVRaJVvq0ySxIDrjWtOWb6OjolU7hY5tpFpWqxM0jsMweZi5g5RSPMfNEkbCxVred4TdK2HYMB7AlNUO1yMIwKwQXakoQAA8kKLCH+jkxdUfjJ5ZqDzkkcLxi7QbnhYFN900aVXBLOAITLbWA5w93aEGNPWo4S3Rk6xViy0lXXVPrZ8Rnsl8468Vyu/ajZ0Pt8wroZ+NkXuMfNTwy6+ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: LAM can only be enabled when a process is single-threaded. But _kernel_ threads can temporarily use a single-threaded process's mm. That means that a context-switching kernel thread can race and observe the mm's LAM metadata (mm->context.lam_cr3_mask) change. The context switch code does two logical things with that metadata: populate CR3 and populate 'cpu_tlbstate.lam'. If it hits this race, 'cpu_tlbstate.lam' and CR3 can end up out of sync. This de-synchronization is currently harmless. But it is confusing and might lead to warnings or real bugs. Update set_tlbstate_lam_mode() to take in the LAM mask and untag mask instead of an mm_struct pointer, and while we are at it, rename it to cpu_tlbstate_update_lam(). This should also make it clearer that we are updating cpu_tlbstate. In switch_mm_irqs_off(), read the LAM mask once and use it for both the cpu_tlbstate update and the CR3 update. Reviewed-by: Kirill A. Shutemov Change-Id: I8bcf94bbf28ebdbbe75e3939e712246a029f84b6 Signed-off-by: Yosry Ahmed --- arch/x86/include/asm/mmu_context.h | 8 +++++++- arch/x86/include/asm/tlbflush.h | 9 ++++----- arch/x86/kernel/process_64.c | 6 ++++-- arch/x86/mm/tlb.c | 8 +++++--- 4 files changed, 20 insertions(+), 11 deletions(-) diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index 8dac45a2c7fcf..19091ebb86338 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -88,7 +88,13 @@ static inline void switch_ldt(struct mm_struct *prev, struct mm_struct *next) #ifdef CONFIG_ADDRESS_MASKING static inline unsigned long mm_lam_cr3_mask(struct mm_struct *mm) { - return mm->context.lam_cr3_mask; + /* + * When switch_mm_irqs_off() is called for a kthread, it may race with + * LAM enablement. switch_mm_irqs_off() uses the LAM mask to do two + * things: populate CR3 and populate 'cpu_tlbstate.lam'. Make sure it + * reads a single value for both. + */ + return READ_ONCE(mm->context.lam_cr3_mask); } static inline void dup_lam(struct mm_struct *oldmm, struct mm_struct *mm) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 25726893c6f4d..69e79fff41b80 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -399,11 +399,10 @@ static inline u64 tlbstate_lam_cr3_mask(void) return lam << X86_CR3_LAM_U57_BIT; } -static inline void set_tlbstate_lam_mode(struct mm_struct *mm) +static inline void cpu_tlbstate_update_lam(unsigned long lam, u64 untag_mask) { - this_cpu_write(cpu_tlbstate.lam, - mm->context.lam_cr3_mask >> X86_CR3_LAM_U57_BIT); - this_cpu_write(tlbstate_untag_mask, mm->context.untag_mask); + this_cpu_write(cpu_tlbstate.lam, lam >> X86_CR3_LAM_U57_BIT); + this_cpu_write(tlbstate_untag_mask, untag_mask); } #else @@ -413,7 +412,7 @@ static inline u64 tlbstate_lam_cr3_mask(void) return 0; } -static inline void set_tlbstate_lam_mode(struct mm_struct *mm) +static inline void cpu_tlbstate_update_lam(unsigned long lam, u64 untag_mask) { } #endif diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index e1ce0dfd24258..26a853328f2d4 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -801,10 +801,12 @@ static long prctl_map_vdso(const struct vdso_image *image, unsigned long addr) static void enable_lam_func(void *__mm) { struct mm_struct *mm = __mm; + unsigned long lam; if (this_cpu_read(cpu_tlbstate.loaded_mm) == mm) { - write_cr3(__read_cr3() | mm->context.lam_cr3_mask); - set_tlbstate_lam_mode(mm); + lam = mm_lam_cr3_mask(mm); + write_cr3(__read_cr3() | lam); + cpu_tlbstate_update_lam(lam, mm_untag_mask(mm)); } } diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index a041d2ecd8380..1fe9ba33c5805 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -11,6 +11,7 @@ #include #include #include +#include #include #include @@ -632,7 +633,6 @@ void switch_mm_irqs_off(struct mm_struct *unused, struct mm_struct *next, } new_lam = mm_lam_cr3_mask(next); - set_tlbstate_lam_mode(next); if (need_flush) { this_cpu_write(cpu_tlbstate.ctxs[new_asid].ctx_id, next->context.ctx_id); this_cpu_write(cpu_tlbstate.ctxs[new_asid].tlb_gen, next_tlb_gen); @@ -651,6 +651,7 @@ void switch_mm_irqs_off(struct mm_struct *unused, struct mm_struct *next, this_cpu_write(cpu_tlbstate.loaded_mm, next); this_cpu_write(cpu_tlbstate.loaded_mm_asid, new_asid); + cpu_tlbstate_update_lam(new_lam, mm_untag_mask(next)); if (next != prev) { cr4_update_pce_mm(next); @@ -697,6 +698,7 @@ void initialize_tlbstate_and_flush(void) int i; struct mm_struct *mm = this_cpu_read(cpu_tlbstate.loaded_mm); u64 tlb_gen = atomic64_read(&init_mm.context.tlb_gen); + unsigned long lam = mm_lam_cr3_mask(mm); unsigned long cr3 = __read_cr3(); /* Assert that CR3 already references the right mm. */ @@ -704,7 +706,7 @@ void initialize_tlbstate_and_flush(void) /* LAM expected to be disabled */ WARN_ON(cr3 & (X86_CR3_LAM_U48 | X86_CR3_LAM_U57)); - WARN_ON(mm_lam_cr3_mask(mm)); + WARN_ON(lam); /* * Assert that CR4.PCIDE is set if needed. (CR4.PCIDE initialization @@ -723,7 +725,7 @@ void initialize_tlbstate_and_flush(void) this_cpu_write(cpu_tlbstate.next_asid, 1); this_cpu_write(cpu_tlbstate.ctxs[0].ctx_id, mm->context.ctx_id); this_cpu_write(cpu_tlbstate.ctxs[0].tlb_gen, tlb_gen); - set_tlbstate_lam_mode(mm); + cpu_tlbstate_update_lam(lam, mm_untag_mask(mm)); for (i = 1; i < TLB_NR_DYN_ASIDS; i++) this_cpu_write(cpu_tlbstate.ctxs[i].ctx_id, 0); From patchwork Tue Jul 2 13:21:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13719649 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1AE63C30658 for ; Tue, 2 Jul 2024 13:21:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D37F56B009A; Tue, 2 Jul 2024 09:21:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CCCD86B009B; Tue, 2 Jul 2024 09:21:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AEDD46B009C; Tue, 2 Jul 2024 09:21:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 84D906B009A for ; Tue, 2 Jul 2024 09:21:51 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 46986A3EDD for ; Tue, 2 Jul 2024 13:21:51 +0000 (UTC) X-FDA: 82294875222.20.401AD91 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf04.hostedemail.com (Postfix) with ESMTP id 8563940012 for ; Tue, 2 Jul 2024 13:21:49 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ecvwrb8T; spf=pass (imf04.hostedemail.com: domain of 37P6DZgoKCDUpfjipRYdVUXffXcV.TfdcZelo-ddbmRTb.fiX@flex--yosryahmed.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=37P6DZgoKCDUpfjipRYdVUXffXcV.TfdcZelo-ddbmRTb.fiX@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1719926479; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ZE+7BbB3PE5kkYQMGtkK8hHnOxQPaKRTiUn2+3lo1Q4=; b=ChLdMLKtnL7ARIKIVAZ707WPeqqRFSHbVtbyWUy1thxAf405IHkhofJuLwVF09kSf/gqIx 0S56WhtNBoG29zZImWPA/BHg2TGbrfy9CBbY1ZwPrHrXnDPynLZpmfKru+XFEpPseOmeR9 sXZDIzWTOUVncNrTDDT/egvoLrQVJb0= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=ecvwrb8T; spf=pass (imf04.hostedemail.com: domain of 37P6DZgoKCDUpfjipRYdVUXffXcV.TfdcZelo-ddbmRTb.fiX@flex--yosryahmed.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=37P6DZgoKCDUpfjipRYdVUXffXcV.TfdcZelo-ddbmRTb.fiX@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1719926479; a=rsa-sha256; cv=none; b=lItn0xlqjmtCmveQdWHIDIS4nYDMo/cTx5cJN9viBjIUABCxoJlvww9L1r9P3pCmgj4oTC 9om4GoBEbfOHRxP5G2tbhrrII1VI/TXHqGy2CvowWEmCbcFjJ9KtYa3qBsnHuMETldqh3+ +x3ySfnJ2lWu3AUxD+Lx4MVleEmM8/A= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-e03641334e0so5885595276.3 for ; Tue, 02 Jul 2024 06:21:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1719926508; x=1720531308; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=ZE+7BbB3PE5kkYQMGtkK8hHnOxQPaKRTiUn2+3lo1Q4=; b=ecvwrb8TNo1/UZgd0BwVExc1b8Djhehq3jytKBsFs1m1jwAFteNHgqQgw2qkq9UrGj 4YLTNVkj4FAso4xJIYlT1n9+dcmBUQLXb9IKz1HY4drtXefJulKIySOJDHfWJgjISpAe qeDWU1OD+RCkJk/+FslKP9mp02AihUu1/2wWnRldLngln4+Xze6xEHO39HzlwrAiX0Qu 6skY7Vqaxo/75nYkJzEblWJ5qh2hPCB6eEfS+nJumioLSVANCDvAc+EsF62NH774D5/+ LnSeKcODaxcCHyrDesY++ac50qhCEcJUfay10XTuQCWd/EGtjGI1jkfYvzx6qjoyPR1m 6JZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1719926508; x=1720531308; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ZE+7BbB3PE5kkYQMGtkK8hHnOxQPaKRTiUn2+3lo1Q4=; b=uTmVgedAZhwcz7KgKzJnF+cS8Li13RM988tyMoPrA+GrcLckRLXv60RBC4Ol2HnYsW NZL9vb2wPSwabuGxqgP7nSdYzT2AXBqw+2T4CCSPAZ6AepfEZ/x/ZiOSaQjmsQUxm412 bFmKmTr9wHUzEln/L6Pg12jBJmgeiNVmHy/6TaEtpRxf024jju4QDppNrdLICDiJgtaB hqa2x7tpFachugGNF0i6+OB7Fa+2Ctlz9FQXABigUy4l+LFwyyDnOQNayS7UKGOjU8np ljEjxEyzx0i0rCQna4cjc4TUfO5gklrhdU3j7HNE5tsveeTD9Clpibolf/MGN7w84+ob 8E8A== X-Forwarded-Encrypted: i=1; AJvYcCW6k5Ar6bcSJ+qc8RKoHi9/MY8BRhpZHgsL8CqV5Ng9EAjlwydqLmehoLKTNr8eLAUNXtBVlIope0xHxu+X8YOWm2w= X-Gm-Message-State: AOJu0YzxPX3S00AMg3kdS9aLGwdPr6iUwePfuqnw73xETZvSaR6VR1cx Wmw0YUk24gKYbnYfouoBhRzCQNXSyWPX1rQJsTWW9YSEGACb3VP8DdTCpxaRDCv4Vu8rZBk0DoU d66i9Kc+uLbXrOLFW1g== X-Google-Smtp-Source: AGHT+IEqYo+qKpL7nHdD6wl5aRfdtl10keDX5LkNKDvlJA032DzVMnyh1cjIdyzuauJzkL055yfoCx7EJnhJcG3G X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:29b4]) (user=yosryahmed job=sendgmr) by 2002:a05:6902:10c2:b0:e03:a5a5:3204 with SMTP id 3f1490d57ef6-e03a5a534e7mr542276.6.1719926508503; Tue, 02 Jul 2024 06:21:48 -0700 (PDT) Date: Tue, 2 Jul 2024 13:21:39 +0000 In-Reply-To: <20240702132139.3332013-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20240702132139.3332013-1-yosryahmed@google.com> X-Mailer: git-send-email 2.45.2.803.g4e1b14247a-goog Message-ID: <20240702132139.3332013-4-yosryahmed@google.com> Subject: [RESEND PATCH v3 3/3] x86/mm: Cleanup prctl_enable_tagged_addr() nr_bits error checking From: Yosry Ahmed To: x86@kernel.org Cc: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , "Kirill A. Shutemov" , Rick Edgecombe , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yosry Ahmed X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 8563940012 X-Stat-Signature: 5ge3skog7jg6hyxk7trjthy41gc1nmxj X-Rspam-User: X-HE-Tag: 1719926509-114021 X-HE-Meta: U2FsdGVkX1+9yMkpQ5ZIH1EY8acdygCQqX5N1Ht4XhJpeewEev9kQ4VmKj9rtuLO+pEH8M7FqAFci7dRKFsUHZI+H4Gxf76IeMLl4xLcYmksRArs6QGA080TNwvZgAvYeDVZ6hpLWsJx/b8DH76viGSzHwzbntCHb2M+0GsgLt1Nf1mu3W9393M1uYOzQMglrql+N1xzM0YAXAkpWgmc3mA+8f58m8uLpfpr4DPGkR3EqZNGcC5+9QtH5nLxfFTIiVEwkdEPBkZAKvoKyp9XT19YNhOYrDiUNkO0ovUDZaY8q8JqY74f9WSn19kcEmRrPHuWv4jTYO5imvs2eYCikxjmYZN9ttfjzwdhhF2e2PiF+l+2r/BDcwvR1FtAf8NtiN0x5qccEMIk3+7XAiUIcCJSyMlpHWc0TLG/S2wReWFRBKYKmo24LFnLUVfXlMxbf+XfcGw/PMMUIE+D040vQvwsYpkl82Nhpsd7Dy3F6JXrzz+JMFYVSFDox/drq2AAP739owl5GcCQxe+GDKm7OWLRj3QWNOkWjVhZTTuneqYWSrtjy9OEVWbD0aQ4auoqbq7eytLKTxDHh+IYtLncc1zPSMa0AALgwAMufcWCxwG0Mzb0CZdL4q4nq83PfuSxvWtpwcV4f71bZBIAdOzX3m7RwRvkes6KqxIwQEOD0lhr/Z8gbSouZSij88HsXztbTtpCbnHHLoAvwsBcNlH2DBFFz+FymnCqGAkI4vk2gysrFREWQwMJIwcOE3L8bNaiEku82EzKuEmgEpAzb4bsxT0qxV3wqKplcj3e6HJenCdDF5uatvWBPwGtzGiTzdKcFV/dnbfvM3PM6ksBfPykpCItecwBkyQGl1Jbrbj5eAxZMM9unuylfR1U5otgQRmbq31BemDeoNs78mFaKxzNfIU1XCrc3yZ3hQYl8IIsz4UVMxyDkci9JOsICyEl25rR0olUl8x8WIFxHG//vFf tLaqfgyl FvrpPkJaOis1kssi95j1lJJv350DuS8x3AmNrTcgAs4lVFWWDqE19Xr5hgObc9t6rDnNE8ODvgx0RnGDh4LflOe0xyyRvVhics0ldVPBg2bqtenbPb6T8an7iRXKHIPTK4HFd95oewg5daLFKjzaTM+9hvq6gCej9J5P8qfSV26rkb1C4Ed7tHEnTm1Co2QiRmAawlGLCTfstVwYC6DX8As5fxjFXaLy1UENb6ANl4MjXIpfYEmLpv+omwap9jy/ZXulVRI7fZTCvLdygvzFNGfdN9CIhGG7eA6jD6D3fCZzTiz0sM6Rblc2bbLB3aJeG+/p5ZP9QH9WJYdB7FNV/g5tVIDcTmMwvcXMLtXhrROJ9Z5RyhCi0xXBuIGDq5PGbVjGsMAImmHmvmoO4HTgDm9GPn2hWGgoTLjp9nDcUHzPGriIKCPby9JcPbB8Txh39LuDXWIgKBvv9cwTLhQeHOSlmrQg14iSPAWjfOJX2Brz+xhcypkLteoQVbAXKbJPS8GMULwSAyl8UM6+MccwO6HKXm3098u6iDX89QOLW8tcpxL8FjcZOxfQk/sTX6EmCw64POGLvQipl3gfb4TaCdie/sj2J5c/yuZItu860qOsVCcBha/zRVbtpAw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: There are two separate checks in prctl_enable_tagged_addr() that nr_bits is in the correct range. The checks are arranged such the correct case is sandwiched between both error cases, which do exactly the same thing. Simplify the if condition and pull the correct case outside with the rest of the success code path. Reviewed-by: Kirill A. Shutemov Change-Id: Ia4f78de99d02f8ce28787346404d59ef13ca3d92 Signed-off-by: Yosry Ahmed --- arch/x86/kernel/process_64.c | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-) diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index 26a853328f2d4..c62098332f05c 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -831,17 +831,13 @@ static int prctl_enable_tagged_addr(struct mm_struct *mm, unsigned long nr_bits) return -EBUSY; } - if (!nr_bits) { - mmap_write_unlock(mm); - return -EINVAL; - } else if (nr_bits <= LAM_U57_BITS) { - mm->context.lam_cr3_mask = X86_CR3_LAM_U57; - mm->context.untag_mask = ~GENMASK(62, 57); - } else { + if (!nr_bits || nr_bits > LAM_U57_BITS) { mmap_write_unlock(mm); return -EINVAL; } + mm->context.lam_cr3_mask = X86_CR3_LAM_U57; + mm->context.untag_mask = ~GENMASK(62, 57); on_each_cpu_mask(mm_cpumask(mm), enable_lam_func, mm, true); set_bit(MM_CONTEXT_LOCK_LAM, &mm->context.flags);