From patchwork Tue Mar 12 15:56:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13590248 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6606EC54E5D for ; Tue, 12 Mar 2024 15:56:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DE1348D005A; Tue, 12 Mar 2024 11:56:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D91478D0057; Tue, 12 Mar 2024 11:56:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C59318D005A; Tue, 12 Mar 2024 11:56:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id B45618D0057 for ; Tue, 12 Mar 2024 11:56:53 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 8AC6F80720 for ; Tue, 12 Mar 2024 15:56:53 +0000 (UTC) X-FDA: 81888840306.13.BE6074D Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf28.hostedemail.com (Postfix) with ESMTP id ED6FEC0008 for ; Tue, 12 Mar 2024 15:56:51 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=HVbd3oyo; spf=pass (imf28.hostedemail.com: domain of 3Q3vwZQoKCDImcgfmOVaSRUccUZS.QcaZWbil-aaYjOQY.cfU@flex--yosryahmed.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3Q3vwZQoKCDImcgfmOVaSRUccUZS.QcaZWbil-aaYjOQY.cfU@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710259012; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=Utpo4un8WURVOryq3iaezlwySVNXykpwBlPdcQcpqSc=; b=kFsiGX56c21nw4vkuS2sADqECTYQw2bNWyQGI2v+M2joVYC8Ku8DNbKy4yetiWjZs+7oV5 Guxl5RxG4rJ9aDvX+yWDBHlL7gF2xtQvgu0lxqcKfA4ZU7VLBaY9QJPZ8XVwMA3rKjF5KF 3UvQ4HI/Mg5y4j1TevtXCAnTCyMhj7s= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710259012; a=rsa-sha256; cv=none; b=4Bw8OZHZfB7EOMrLG2iijQl/iRhg6sySdhPFKWZQlxwHmbHthaBxgs0SmRxwdquWogJVNZ QEt0lgdRGEG5jnOds0sJ/BhNffEx5GnTrqyCAw3upowRabkMKJcg+gazGbGOObps542b0o QE4MyTnANF/QY3I6p4Cd7APSPjljyAk= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=HVbd3oyo; spf=pass (imf28.hostedemail.com: domain of 3Q3vwZQoKCDImcgfmOVaSRUccUZS.QcaZWbil-aaYjOQY.cfU@flex--yosryahmed.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3Q3vwZQoKCDImcgfmOVaSRUccUZS.QcaZWbil-aaYjOQY.cfU@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dce775fa8adso9791485276.1 for ; Tue, 12 Mar 2024 08:56:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1710259011; x=1710863811; darn=kvack.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=Utpo4un8WURVOryq3iaezlwySVNXykpwBlPdcQcpqSc=; b=HVbd3oyoyiRViQGT9ZUIy1uCPwQqTcrhoOluLKKPZ0AN2BYPM/cbH0mBSF/AoIPnbD ZhhKZDuoeQ22MgQuQ5GUowgcgq20vaTsCAGScgockpsdDI6PrczYYJwlzpExg78zagE6 al8TNWNHZx5JOTHeIFnXfJQ9egVFl1lNsGhdf22idAN6UzKfKOIX7khfhqDWLUIgPiU6 XK5wnj5M4eKlCMl/xA/xNCM3Ts6cb1CDD8DNUdNARzNKzmkUziDIBuwQifzbskWDNjqQ 8DABP6NZmEhb7zDIk8XP6hbMX3l/JBto7V+asAX/42YvwrJA4wF9ercBkLWADOh4pQR7 sXjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710259011; x=1710863811; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=Utpo4un8WURVOryq3iaezlwySVNXykpwBlPdcQcpqSc=; b=CEokwOQkKVpNVgY7j2i9s+c/YSKhm2osHtepuGKe4wO0ueFtWBQxgVTPweVvqjV0Ky vQap2BGGIrUAaGFPfqzJgpH+hYf2ENBFgWjswlIPAbvXjVPUGafcziZOxWFyE15YBA9v 1HvkQoffXqzv9LvFnZJbpHMfkfUAlttRstj9ieiJJT/p/yijgIguQdFI0YLcYLD9gNsE it0H9riiSGalNXU6VgVIYmhcqn/Iy+kU7wsnIQSflnU5zo6l+a573jfk/cP+K4HA1/Nk vtOJDEs6HBjcIAf7K/xO3Ymc1gZAMRpB9oufo+wzB1i2qkxvA8KdwureC0VHpQuXghO5 n0RQ== X-Forwarded-Encrypted: i=1; AJvYcCWmxflNHKwf9WYvcK2AXuQSlxpktJGWFirjf/N5INucZ7vy05/HivcvUk1D9ngVVAR2UJHAZSGcfHjgHITlzHe1Whg= X-Gm-Message-State: AOJu0YyRwhATlZFnVXYQ6KAH2E7N/03peXIFD+7Odzz/bu2cOAPKyb6E IodoWv3oTaxZ7HWd7Lyu4qdr69Og2bX3I0gFgw4btVbGGjQZH7ZDd6EZnlA4+DGVHEGbfN3q4kr oSxOwTZc8X0J3x94+5Q== X-Google-Smtp-Source: AGHT+IFTM3Svt5aG2KDAZZ5idfNa42+hKp13eDZYuCEzCHp68sCT3zJVywoPrbfqfLH7MdR0l2OCOvYs3V662FOa X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:29b4]) (user=yosryahmed job=sendgmr) by 2002:a05:6902:10c2:b0:dc6:e1ed:bd1a with SMTP id w2-20020a05690210c200b00dc6e1edbd1amr2596454ybu.2.1710259011049; Tue, 12 Mar 2024 08:56:51 -0700 (PDT) Date: Tue, 12 Mar 2024 15:56:39 +0000 Mime-Version: 1.0 X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240312155641.4003683-1-yosryahmed@google.com> Subject: [PATCH v2 1/3] x86/mm: Use IPIs to synchronize LAM enablement From: Yosry Ahmed To: x86@kernel.org Cc: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , "Kirill A. Shutemov" , Rick Edgecombe , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yosry Ahmed X-Rspamd-Queue-Id: ED6FEC0008 X-Rspam-User: X-Stat-Signature: 9unoguuchmzxi7ju67sagg1dsbwz6x44 X-Rspamd-Server: rspam03 X-HE-Tag: 1710259011-57069 X-HE-Meta: U2FsdGVkX1+04gh6n34tGF9qsNJrKD7IeNR/K9vKAyoA7fVyxBHomjyZddjRfN/+nO3YKMfCIfUO9a8HQduZ+J0xebOQDSS2WnkPNDyt5vjjvTQdbSAnjKwep+FYTpbab6Vh8PNJ3OZ35JK37YCkt+IGm3B47+ByHpS0ovhwV652bc6ShilYQibFZJG/EYPD1Y5/9QufJ5Nkh+KyL7au5Z797hgPsFUo/AXh2BO1Ar+1Er3EH053Le3h3OcIdL0zvGhoz4P3Q2uctMfWL4ria6WAyOsoYfa4FI/iJsRZaL1/8PmjmUIZe1asaPIZYlb5MAq4JxQfegrmuNlQbGjfQ9Vk2O5lrFfSVUfYX3vSx26T9hOcfUUDjZICUqVbn22sx+BnegdXLVV33U5I52b9YBsF8Sb+f+FgkBXxB5xMJJdpqEpYKj+d2tkxDjtyveKkRbeMwE0zwp66XU70yChQ++SzzI0xRDhV240SSMt8IYbgXp3n7bhhUVT2l3Px0aS2eaRIX93U2kAfVfJIiHnCuxHg3uaGiaDslq7p71lIl7yCguJHm4HF6hMN/mjrvLKCWDAV+KBH0oiyYN87ES/M7UnEKq4/iUEjLSvl6mvtkh30ycdzZTpSfzCCHlSbiL+iIaTYckTVTmnzITuE8ddUAoYGLP5xSLMf4tp53lPjz16HdMfD/9bbUmqlLT1PLC7qZvdO09HXEXVF+FGGv400/CO8gDucy92qC8n7q4xWCIXG6suXoKKjnOfplcNy3qTztrdY8NKEM7q3dEs2oSwWOJtr4ye7xSAp7IfNAnT43A21rFmAFwULQ/QBTt91BDs+WZGI66lp2G2/VBwVdmxiB2wWxuSzEDH0XIGW6uUdPWiiOp3SrLjyLHDNytJs6Bjt3Roh4uDK0Sdk9F4hb+BZ9LtZccqXQjqVR8wQ0Rx3wwp4HZN73fXrScV07WeagEsB4TohXXYVJ07PeEuFbA/ UXx4qwQq 1YnpKw4Ac8xw7qVYZUyTfQgyKWLmtqs5DbHA5dIv5G4wZTClnTesHzKA3XFcsnnTR/H4b5ODBbM9WLODhzAnMC6NyIwsbtouehtwjG8Y2BsiLt+V0gUuF5j4yEOr20QTk8bqNnlaGyB4i1pnM0xon+X9ivTPhWX3WuwQONXC0sf6A9co+5fN7EHJ05fpWdXPWABwM17K/IMkTo52Va5DwYTx7yk0kC703qb7tABm2L0yqutZJdhswWtxZ+1c7a9z17vJGeh2K9Ru2HeDJVNniL9gXBlThMVNz9usSxVb872oG1F5UitTHfPo3gWTtwDezRROS2mmzyr6JFR5DhZiQ+FzzQgorKZZJSvs13aWucDoAmueFk0fPgTHYle0S+5Hczt6n X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: LAM can only be enabled when a process is single-threaded. But _kernel_ threads can temporarily use a single-threaded process's mm. If LAM is enabled by a userspace process while a kthread is using its mm, the kthread will not observe LAM enablement (i.e. LAM will be disabled in CR3). This could be fine for the kthread itself, as LAM only affects userspace addresses. However, if the kthread context switches to a thread in the same userspace process, CR3 may or may not be updated because the mm_struct doesn't change (based on pending TLB flushes). If CR3 is not updated, the userspace thread will run incorrectly with LAM disabled, which may cause page faults when using tagged addresses. Example scenario: CPU 1 CPU 2 /* kthread */ kthread_use_mm() /* user thread */ prctl_enable_tagged_addr() /* LAM enabled on CPU 2 */ /* LAM disabled on CPU 1 */ context_switch() /* to CPU 1 */ /* Switching to user thread */ switch_mm_irqs_off() /* CR3 not updated */ /* LAM is still disabled on CPU 1 */ Synchronize LAM enablement by sending an IPI from prctl_enable_tagged_addr() to all CPUs running with the mm_struct to enable LAM. This makes sure LAM is enabled on CPU 1 in the above scenario before prctl_enable_tagged_addr() returns and userspace starts using tagged addresses, and before it's possible to run the userspace process on CPU 1. In switch_mm_irqs_off(), move reading the LAM mask until after mm_cpumask() is updated. This ensures that if an outdated LAM mask is written to CR3, an IPI is received to update it right after IRQs are re-enabled. Fixes: 82721d8b25d7 ("x86/mm: Handle LAM on context switch") Suggested-by: Andy Lutomirski Signed-off-by: Yosry Ahmed Reviewed-by: Kirill A. Shutemov --- v1 -> v2: - Defer dereferences in enable_lam_func() until after we check cpu_tlbstate.loaded_mm. - Collect Rbs. --- arch/x86/kernel/process_64.c | 13 +++++++++++-- arch/x86/mm/tlb.c | 7 +++---- 2 files changed, 14 insertions(+), 6 deletions(-) diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index 33b268747bb7b..76e91fc68c5f3 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -750,6 +750,16 @@ static long prctl_map_vdso(const struct vdso_image *image, unsigned long addr) #define LAM_U57_BITS 6 +static void enable_lam_func(void *__mm) +{ + struct mm_struct *mm = __mm; + + if (this_cpu_read(cpu_tlbstate.loaded_mm) == mm) { + write_cr3(__read_cr3() | mm->context.lam_cr3_mask); + set_tlbstate_lam_mode(mm); + } +} + static int prctl_enable_tagged_addr(struct mm_struct *mm, unsigned long nr_bits) { if (!cpu_feature_enabled(X86_FEATURE_LAM)) @@ -782,8 +792,7 @@ static int prctl_enable_tagged_addr(struct mm_struct *mm, unsigned long nr_bits) return -EINVAL; } - write_cr3(__read_cr3() | mm->context.lam_cr3_mask); - set_tlbstate_lam_mode(mm); + on_each_cpu_mask(mm_cpumask(mm), enable_lam_func, mm, true); set_bit(MM_CONTEXT_LOCK_LAM, &mm->context.flags); mmap_write_unlock(mm); diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 5768d386efab6..e8feb2e154db2 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -497,9 +497,9 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, { struct mm_struct *real_prev = this_cpu_read(cpu_tlbstate.loaded_mm); u16 prev_asid = this_cpu_read(cpu_tlbstate.loaded_mm_asid); - unsigned long new_lam = mm_lam_cr3_mask(next); bool was_lazy = this_cpu_read(cpu_tlbstate_shared.is_lazy); unsigned cpu = smp_processor_id(); + unsigned long new_lam; u64 next_tlb_gen; bool need_flush; u16 new_asid; @@ -622,9 +622,7 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, cpumask_clear_cpu(cpu, mm_cpumask(real_prev)); } - /* - * Start remote flushes and then read tlb_gen. - */ + /* Start receiving IPIs and then read tlb_gen (and LAM below) */ if (next != &init_mm) cpumask_set_cpu(cpu, mm_cpumask(next)); next_tlb_gen = atomic64_read(&next->context.tlb_gen); @@ -636,6 +634,7 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, barrier(); } + new_lam = mm_lam_cr3_mask(next); set_tlbstate_lam_mode(next); if (need_flush) { this_cpu_write(cpu_tlbstate.ctxs[new_asid].ctx_id, next->context.ctx_id); From patchwork Tue Mar 12 15:56:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13590249 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 292D9C54E58 for ; Tue, 12 Mar 2024 15:56:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B493B8D005B; Tue, 12 Mar 2024 11:56:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AF92C8D0057; Tue, 12 Mar 2024 11:56:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9C1688D005B; Tue, 12 Mar 2024 11:56:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 896868D0057 for ; Tue, 12 Mar 2024 11:56:55 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 340DD1A0551 for ; Tue, 12 Mar 2024 15:56:55 +0000 (UTC) X-FDA: 81888840390.29.0EC23A0 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf20.hostedemail.com (Postfix) with ESMTP id 818491C0011 for ; Tue, 12 Mar 2024 15:56:53 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=4PuZlJ3Q; spf=pass (imf20.hostedemail.com: domain of 3RHvwZQoKCDMndhgnPWbTSVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--yosryahmed.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3RHvwZQoKCDMndhgnPWbTSVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710259013; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=dGVz1SMb2zVErgYS/f8v/LIvaYP5LVM/6U9nkrrlBVE=; b=L1Wo9IN2bWfvs3zp4gpV0P8bkNZ/GjBnJz1fJCqqEsTY3AtP0vaCXnjTvNSIROYgH3aT99 q7llrBy5JCR9G97QMqEoENRlI3XR88oq8UP/lKFiIiU8c5HPuTMTk7Si4QQ/7Fwz3lOJ87 SAoXOfrGG4jfZAa9qs1LauVSEUfOAFQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710259013; a=rsa-sha256; cv=none; b=FYLEzpAwfIGu8hJeqfautkRKMUootjE+2TjfTuNjO2ezypUPYkwdavRxqE5bkllrq6l3gq +4RmXMVTgkH3dW8MO3eWCXarhX488cf+wLY0ss8zZFRF31zPBYFeH7GOtPnvrgRCseNqDX LXfoEl4x9WJYOLTOOIQPosJfj9QoNME= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=4PuZlJ3Q; spf=pass (imf20.hostedemail.com: domain of 3RHvwZQoKCDMndhgnPWbTSVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--yosryahmed.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3RHvwZQoKCDMndhgnPWbTSVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dc693399655so10700320276.1 for ; Tue, 12 Mar 2024 08:56:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1710259012; x=1710863812; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=dGVz1SMb2zVErgYS/f8v/LIvaYP5LVM/6U9nkrrlBVE=; b=4PuZlJ3Qw3DiaH0hHg/n+lRd2Lpx9BFWZsQnlsZz3o3B9etsj1t6pF9bonJ72H2pST 76qcf/KMI6T/63W5qnXHLy2Q6/FtQ3suh5Xn0Pg4+f0n5GTam+AyYyTF6jcvJyg/b4n9 CgyRdhJKmAsFgcn/rkfp0sSXSVqw0eb3bxX1ppOnRErc0lB2at6BajHDbhijypnk3i25 o87Fd5TOaJy36LP8mn/1qx6+zEL6mdn2OYnB8MKkLaV9yccoeqn9Eu348VML9u8ccWns sywCA9R+w5unCh5cRNw9EAEoYqlsrgNLLFVRoQPt3GfJniD0doPJu3E3/bIBaPqrvmlS sC3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710259012; x=1710863812; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=dGVz1SMb2zVErgYS/f8v/LIvaYP5LVM/6U9nkrrlBVE=; b=ZBT0Bi/lnuXZE4hJspOZ1FjV3xhb//rvqSRyCzvvERUuua7doIf+f7T7a0hE18yoIK rGWPMX4LiKI0oN0M7n5xgtUKWN8DIkHKfl+5sGTwnWG5eBMs/aOmC9YNpE8GfK4BgKv9 kg4IDsA2MIwqI9tSvkB2nWwkO3YqwaHSydD96cEcFi9IridhL9kTypfZ65qThk8Sqc/6 5XAb/23qJSqtEJTbepa1IME1o3XrZH46xFvZ1/PZk2+WxRdR3rtwmf1/C9mLBJecPvCj aP2m5c8VjnTZx7Z8ky7hiVR+pOpa0aAEZLKG3Ept55YOoEwBngXBST84+8Q9c8x9hxv3 7cig== X-Forwarded-Encrypted: i=1; AJvYcCUa+wLBTrzUMgG2+INYKgnZ/n22IV4DgLiFVHpLit58xg01nSg0dvF228tOLM8twFotH9ewNIhjL6FRaIMxPRW0n0c= X-Gm-Message-State: AOJu0YzhNXazRPUDTxL19hPT2qeMLLs2spwKBtP7uO0aK6px35VFVSJQ Rj3HA1KV5oU90/EHlv7JWhjXTeFyNfegpWxHYQtY8UeBT34cbwnIQ0+NzCMv9R35BNO+0KWS7Q7 nBiBi503YALxGCe5ZwQ== X-Google-Smtp-Source: AGHT+IEWr4BUvfejWnq0y5OUGkVPSF87K17RjNhILNkA1XXGVsqD9DHHQ9r1YalvT3IW9/ZqycxovNLaT6NPWZZp X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:29b4]) (user=yosryahmed job=sendgmr) by 2002:a05:6902:70a:b0:dc7:68b5:4f3d with SMTP id k10-20020a056902070a00b00dc768b54f3dmr2745152ybt.11.1710259012661; Tue, 12 Mar 2024 08:56:52 -0700 (PDT) Date: Tue, 12 Mar 2024 15:56:40 +0000 In-Reply-To: <20240312155641.4003683-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20240312155641.4003683-1-yosryahmed@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240312155641.4003683-2-yosryahmed@google.com> Subject: [PATCH v2 2/3] x86/mm: Fix LAM inconsistency during context switch From: Yosry Ahmed To: x86@kernel.org Cc: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , "Kirill A. Shutemov" , Rick Edgecombe , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yosry Ahmed X-Rspamd-Queue-Id: 818491C0011 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: sq7jytnw655a4qq7tafxmrpfjdzwybhw X-HE-Tag: 1710259013-675853 X-HE-Meta: U2FsdGVkX1+p/9V55AXHUTcXln+nYOrs6X5OAMAdBQ2I/k0/M/0HyzD9xXZvH/kG2wjHLCD0Df0VFAz/qTBYjrcJ2OqwgHJ46RPSphq3n2fCmlW767oGMskuSahLbf2/mDeKk4aXfzLPpv+TogNRzMtYC937Gq+Mj+4busfiIIRgfiC6O20qqtMUtDj7l1DMsCBam8NXbF3s2QdIDQGJMsPUPsCNRg3lcAQ1f/s9BWgY7gz0rOVDvud7NaKyinz2yp05yfA18r9UTt099hOxw3xwsGSlg5x9s6uMeUau8qegLB/IStW4rJ2tzGdMCD68yFmFwuPi46tnsnxwto7ciKbT0YehYLJbSDQVC5zFhy+hAJZRaTSaK0mUjkCAuRB8lmjjjkykw0aayzWlOvj6sooPnFBo5cCDhNDdhgUOUfY5yttFYfqqq7ZsyUlltkA1qT/ipVOMaVL0/7tNM2TyeFEvHG1ynQS4tCox6ZhoajVXD+sO0r/dxt7LNU5Pzlgt+JQ20L971kzQ0zYRhSX8kzq0PJz6HoPCPzhcNfjxUZRArhfh+c0J11so5sJuhkTcbNeH6xNpXybjrNXZGTBaoD4cfzPHQrSamSCLwcmx+jz5fmTdstMLNTcV7NTnWfbVi095xqMwRHxZD7TT9Ld2TKqiylr/mS7CvEq50TQ7L6/xBF1+vuDUUfnIjVx2dISDNHviVgwA8z52Itzq0j6rPcBU2mqe6HtUV+pLbxbf6xb/hRUEwdQwnHFA/fG6a9sTFf5hyzxIUTZOAhpmAShcnro9UW6L65x+17JWF4eIlhh4HxdAN5tKgOA0A5cWnnidLr6oqwY1PilFCCVebb2UR5majehnVOGI7R/cEy7zKS1QiaKv+yhfzzZBamlMx14zzwAOHWYuReflg4eDsTwyLy5v0fKbHtZVGMpniKw0aUzlNGZGCuJKHfHElPFsMiES94fFAxmvMx362y0qNUW OU8mhHqr qjiC07cfMz1GAoPV+YHJPukXkbQ/j4hD2hJIJdX7R4N3Mh0UXzS3ObS43kHjsyN1NA72EJVy+sh/fo4AMfjYRX4H5lUbPG4fbeXt1VovWBJMPLAqcXTjffCeBt4SACCy5J+7asMn3y3X8nqlfw5hqHaSgmwMlr9XlC7Ta0swXhUtVcPxtmndIdGfO0DX/1AEnKj5wuqQwVOxdRwdNiySNcIeB7NEDPlwef9m9YeowvnmdWou5uydc4G4A5VEgjxKOXSToGxobqAVE1Gtq/keXXGwVSllefidQmu+5wfcX7XAxwbB4U+u4Tl4l6crUmot9T7y6FVPZkrFiz8qC1E3WN+HYwr9NeqlzpKV3XjM7HgaZ/SN95OQTTxBdXfpILVsWSM54 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: LAM can only be enabled when a process is single-threaded. But _kernel_ threads can temporarily use a single-threaded process's mm. That means that a context-switching kernel thread can race and observe the mm's LAM metadata (mm->context.lam_cr3_mask) change. The context switch code does two logical things with that metadata: populate CR3 and populate 'cpu_tlbstate.lam'. If it hits this race, 'cpu_tlbstate.lam' and CR3 can end up out of sync. This de-synchronization is currently harmless. But it is confusing and might lead to warnings or real bugs. Update set_tlbstate_lam_mode() to take in the LAM mask and untag mask instead of an mm_struct pointer, and while we are at it, rename it to cpu_tlbstate_update_lam(). This should also make it clearer that we are updating cpu_tlbstate. In switch_mm_irqs_off(), read the LAM mask once and use it for both the cpu_tlbstate update and the CR3 update. Signed-off-by: Yosry Ahmed Reviewed-by: Kirill A. Shutemov --- arch/x86/include/asm/mmu_context.h | 8 +++++++- arch/x86/include/asm/tlbflush.h | 9 ++++----- arch/x86/kernel/process_64.c | 6 ++++-- arch/x86/mm/tlb.c | 8 +++++--- 4 files changed, 20 insertions(+), 11 deletions(-) diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index 8dac45a2c7fcf..19091ebb86338 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -88,7 +88,13 @@ static inline void switch_ldt(struct mm_struct *prev, struct mm_struct *next) #ifdef CONFIG_ADDRESS_MASKING static inline unsigned long mm_lam_cr3_mask(struct mm_struct *mm) { - return mm->context.lam_cr3_mask; + /* + * When switch_mm_irqs_off() is called for a kthread, it may race with + * LAM enablement. switch_mm_irqs_off() uses the LAM mask to do two + * things: populate CR3 and populate 'cpu_tlbstate.lam'. Make sure it + * reads a single value for both. + */ + return READ_ONCE(mm->context.lam_cr3_mask); } static inline void dup_lam(struct mm_struct *oldmm, struct mm_struct *mm) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 25726893c6f4d..69e79fff41b80 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -399,11 +399,10 @@ static inline u64 tlbstate_lam_cr3_mask(void) return lam << X86_CR3_LAM_U57_BIT; } -static inline void set_tlbstate_lam_mode(struct mm_struct *mm) +static inline void cpu_tlbstate_update_lam(unsigned long lam, u64 untag_mask) { - this_cpu_write(cpu_tlbstate.lam, - mm->context.lam_cr3_mask >> X86_CR3_LAM_U57_BIT); - this_cpu_write(tlbstate_untag_mask, mm->context.untag_mask); + this_cpu_write(cpu_tlbstate.lam, lam >> X86_CR3_LAM_U57_BIT); + this_cpu_write(tlbstate_untag_mask, untag_mask); } #else @@ -413,7 +412,7 @@ static inline u64 tlbstate_lam_cr3_mask(void) return 0; } -static inline void set_tlbstate_lam_mode(struct mm_struct *mm) +static inline void cpu_tlbstate_update_lam(unsigned long lam, u64 untag_mask) { } #endif diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index 76e91fc68c5f3..3dd8c1f51e45c 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -753,10 +753,12 @@ static long prctl_map_vdso(const struct vdso_image *image, unsigned long addr) static void enable_lam_func(void *__mm) { struct mm_struct *mm = __mm; + unsigned long lam; if (this_cpu_read(cpu_tlbstate.loaded_mm) == mm) { - write_cr3(__read_cr3() | mm->context.lam_cr3_mask); - set_tlbstate_lam_mode(mm); + lam = mm_lam_cr3_mask(mm); + write_cr3(__read_cr3() | lam); + cpu_tlbstate_update_lam(lam, mm_untag_mask(mm)); } } diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index e8feb2e154db2..b2f74c451b3d7 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -11,6 +11,7 @@ #include #include #include +#include #include #include @@ -635,7 +636,6 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, } new_lam = mm_lam_cr3_mask(next); - set_tlbstate_lam_mode(next); if (need_flush) { this_cpu_write(cpu_tlbstate.ctxs[new_asid].ctx_id, next->context.ctx_id); this_cpu_write(cpu_tlbstate.ctxs[new_asid].tlb_gen, next_tlb_gen); @@ -654,6 +654,7 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, this_cpu_write(cpu_tlbstate.loaded_mm, next); this_cpu_write(cpu_tlbstate.loaded_mm_asid, new_asid); + cpu_tlbstate_update_lam(new_lam, mm_untag_mask(next)); if (next != real_prev) { cr4_update_pce_mm(next); @@ -700,6 +701,7 @@ void initialize_tlbstate_and_flush(void) int i; struct mm_struct *mm = this_cpu_read(cpu_tlbstate.loaded_mm); u64 tlb_gen = atomic64_read(&init_mm.context.tlb_gen); + unsigned long lam = mm_lam_cr3_mask(mm); unsigned long cr3 = __read_cr3(); /* Assert that CR3 already references the right mm. */ @@ -707,7 +709,7 @@ void initialize_tlbstate_and_flush(void) /* LAM expected to be disabled */ WARN_ON(cr3 & (X86_CR3_LAM_U48 | X86_CR3_LAM_U57)); - WARN_ON(mm_lam_cr3_mask(mm)); + WARN_ON(lam); /* * Assert that CR4.PCIDE is set if needed. (CR4.PCIDE initialization @@ -726,7 +728,7 @@ void initialize_tlbstate_and_flush(void) this_cpu_write(cpu_tlbstate.next_asid, 1); this_cpu_write(cpu_tlbstate.ctxs[0].ctx_id, mm->context.ctx_id); this_cpu_write(cpu_tlbstate.ctxs[0].tlb_gen, tlb_gen); - set_tlbstate_lam_mode(mm); + cpu_tlbstate_update_lam(lam, mm_untag_mask(mm)); for (i = 1; i < TLB_NR_DYN_ASIDS; i++) this_cpu_write(cpu_tlbstate.ctxs[i].ctx_id, 0); From patchwork Tue Mar 12 15:56:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13590250 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 63B78C54E60 for ; Tue, 12 Mar 2024 15:56:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D03778D005C; Tue, 12 Mar 2024 11:56:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C33AC8D0057; Tue, 12 Mar 2024 11:56:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A3A1C8D005C; Tue, 12 Mar 2024 11:56:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 88A788D0057 for ; Tue, 12 Mar 2024 11:56:57 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 54094807B3 for ; Tue, 12 Mar 2024 15:56:57 +0000 (UTC) X-FDA: 81888840474.08.C810AA3 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf20.hostedemail.com (Postfix) with ESMTP id 6F67F1C000D for ; Tue, 12 Mar 2024 15:56:55 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=RkzKdPtb; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf20.hostedemail.com: domain of 3RnvwZQoKCDUpfjipRYdVUXffXcV.TfdcZelo-ddbmRTb.fiX@flex--yosryahmed.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3RnvwZQoKCDUpfjipRYdVUXffXcV.TfdcZelo-ddbmRTb.fiX@flex--yosryahmed.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710259015; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zlk+aNCie6HXTjVy40lS6l4zYMw7smybyoWxAq6T2Xk=; b=4uddBqNcl2b8qzTG0io2R+xLtQezKIYM+liXDZ1pnxWt24IN7XAphG0PWCjQ/WSUO3OMkv 40anaC1P0iNqX5G0EjqNQ4V5ijFFbB7JJYnwwicxOuTIWNUX3ypxg31GfWNT5bT4gG6gwe +LFH0kiiNKZz/HbJ/yl9JN3xg9ShXcw= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=RkzKdPtb; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf20.hostedemail.com: domain of 3RnvwZQoKCDUpfjipRYdVUXffXcV.TfdcZelo-ddbmRTb.fiX@flex--yosryahmed.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3RnvwZQoKCDUpfjipRYdVUXffXcV.TfdcZelo-ddbmRTb.fiX@flex--yosryahmed.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710259015; a=rsa-sha256; cv=none; b=b8cfdU29dD59i4AaKcuv5QQlDA3zjAxMVrLPaO+e03s2HKcH/9vax32uY9rJ+ObHXt1lgO DwL6U5XEEiylMtfgIgOFMq9WhtCkUGhxbKffGU8e+e5owsz03r0VuDz1Apr7Rydpa2BEMb P5tVFfdAzD/C2dv7xs1If4EfVU01m+8= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dc3645a6790so10054947276.0 for ; Tue, 12 Mar 2024 08:56:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1710259014; x=1710863814; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=zlk+aNCie6HXTjVy40lS6l4zYMw7smybyoWxAq6T2Xk=; b=RkzKdPtbyPpKXxoOBm4xbtpC4K7/e5fJV7+zKVgdPVJd+UxZJtJotIjxPBOUR6C16/ WWVc9Lt0e4jHvNz/rJxfPQukIK2ogiHXsCRp8trOxlYo4joBTObcGpz6OGXkF1TU2s6h 9rfhrpjVtM2FQpakBubjrcL0+NlR54nwneQ1u/OugDlJ4ViDdmcekH85OBnOnsPr5OIE 3Bdu3NV7YsMDB7nUbwEfl5Hlb44XdisJCoK6WSHjI2Kf1OGZvCwgkESLQTCw3pTfp0ly PL85rr32nodlnjv/AE2gf/XDtQSSgTK5rxtdiQb/QPoPyU48T9q9pXrQnTxZMus3Ucpc 5p9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710259014; x=1710863814; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=zlk+aNCie6HXTjVy40lS6l4zYMw7smybyoWxAq6T2Xk=; b=qhDG48IeW8pI0zD0uiBuVnsizxhfXvPBKh/tar7k8Zn7qJawYEuZ2novYAed4Lzw21 u0GHz6F5qKYCK5slNijdkXvTeBfwmgYX/wNiA8i0flnnOhnpmITw0uQQVfFCMahhVdiL Z+cDjKy9VnBeP4Vjc0QB91AktI3YHHZ+HldDpIu/Xx9AqeJ4POeeDVhxic0UG530/IBY T6zTBQn8Hji90eTxBKVaMf+xAiRq7NsQBnoZO06sBTLrAaFcp0h21g63zbruQhjwWm/U yiz4vFE+mx+cV60EM6vjfA8NbLJskQWym/GMhUI+8abA8aDxML9iFacBy8y6lJFyqcXi TKNw== X-Forwarded-Encrypted: i=1; AJvYcCUvpbZckHLjwvXjsuSubUfOiCNV19pLeBOdotZoTwKOUkaSh1aXnSf4ZuW02SjQbW4h360m1k2sHTB5vL/oUXP23jE= X-Gm-Message-State: AOJu0YzFo0mZAi6lCnx95xVI5GWYTv14r6RhD2eKVkvYjyNwbMb3buNG HBr8he5hJTRqyX3T9dfKOzRbQj3HR8a5y2R/6X+dlJJmvyDUfV0HtmN7herUNWErfxrLzaVmP78 YZmOE/CWaA7AeURvQGQ== X-Google-Smtp-Source: AGHT+IHyJoqCC0NaihyP5D4D/Ot3zHwDly5rqaMSbNaNRucYa8YyWEm+aw8zRYzIK3OVa86EMuwxIXsQ2czaLmfd X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:29b4]) (user=yosryahmed job=sendgmr) by 2002:a05:6902:110a:b0:dcc:50ca:e153 with SMTP id o10-20020a056902110a00b00dcc50cae153mr2676138ybu.7.1710259014520; Tue, 12 Mar 2024 08:56:54 -0700 (PDT) Date: Tue, 12 Mar 2024 15:56:41 +0000 In-Reply-To: <20240312155641.4003683-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20240312155641.4003683-1-yosryahmed@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240312155641.4003683-3-yosryahmed@google.com> Subject: [PATCH v2 3/3] x86/mm: Cleanup prctl_enable_tagged_addr() nr_bits error checking From: Yosry Ahmed To: x86@kernel.org Cc: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , "Kirill A. Shutemov" , Rick Edgecombe , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yosry Ahmed X-Rspamd-Queue-Id: 6F67F1C000D X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: qk7dm9whj8kmfcw7zbf8zx9psdi9bo6w X-HE-Tag: 1710259015-356599 X-HE-Meta: U2FsdGVkX18StIGXI3PB8HH9q9IUl4caF6GYHR5lfurWn169pFeylGGAfsRV6UfSBlE2epozpreOGp86zi99QGyKkTx/5CmCDEEhvdhKI6RgvSxYRlvQNHjZ7Td2cPiD0bGOMrwci71uxrUTFL8uFZockaFvjkGytMOBFd6+MeBPtZyE4TUPp3DtpHrK/0SX83OKh+StZdJ9uIhxuf6e52itlgPbQ5X3wpxYayy+K0JVTrzw1uU6qaojV4Rp9R71yGdYmJ2jlNifSje30DL40tXLh7OF8C2OF+/uPpgtQeKS3/SCGsmdQESgw4pNMFhwAd3q8Pw1qgVxOeEzT93cU+BsIdhG0uj+nrZeIorPq80T00GjSm15Pz48+oOLtjGpf4Cj3Xl4mtyaXxS4Ys1wI0ImcFdzPJuZVj85NZuAWA6DVynYRVRJTg8t/h+hn1mgtx2qGL839DgArnaIcP9qIiSy4XCS7IzhFyX83xzof2D2S//qV/p6EtZ1KffWCurp4vEj2/a288godMk40hT9sI0MkMv2UDtW/+LhGmegpTY3u9dgfaU1WD2FLTHXWK+T2wL0IcmCTQeHujY7jm864SiXhAqXVAm/hbbaK0IAZK3QkRqI3MHibZTZOJUw7o5LWRqTvFOTyFlaW9odmJJ20zF/ELAxsV0GD3f0NYXNiaMjmX0VumwNrv2EOZJqt4JgDLq5SfTNMU8E4uLmjSBqdpR/H0q1heNYYZWq45vQM0XtacQH3NnKXaB1EqCsnh3Dw6bqT2BqRNkp8rnHEsd9UtRgSJUJgEn/uGQj0991ZKq23GEr0chJqNI4pPsdmpzvhEAZE/rvpUz/oNq2xoqC0/4j6ruvy1Uzv0g761Wk42/oIoETVp3esl8BKi5tP1F2mXYWy6jKiwuVAC6gwNwOSfpct/GNN6PCT3bhb6JQNM4BGqOtVWORDBnysENMVECbAfesYPFSTWA3Dgnamvq 3qSj12PS 8YW1gX4bH/XymZtCGGIS5G64QjL3Vy/J3w2rzvRGFBZ/7zTOrH+QOYU33vq3N6bNhd7dybvnyeAEfxt+NMJEkClXa+0vwgg4TPb2ii9WcD43iGpFvvCwKwKwlZJ1AiCAGh51qtlm8Qu6oiyOwxDLYJDWWYxfifOdUE1alh/8wpWInd6uK7BZG1R8G7siKxKE/gQ5SaWXkX4Fh4uke3BaGmQ3g8DOt8B1MhWFlXQ/tqunnYivyAxnDDbPPwieQKWHARLy6YhopLz9UGwVWvf9s795A240F0IZYr6mnu62qAPsoua2HQKJjOp7RXIsAyPz/SG92wzTfveoFqQp+ftbjTf2n1T5gTumDVhYqpeyYAzg6V3sQ3Uhh9PJY+VNdRcg0asDaQsJNYyPqPYExIRMYKSUXmOetpL+hkfylBLYSSBOkebSr5QE3GNuY7GvOYPUXIdO6SK/jUfq7dwSIaUZOmegrXlPyKKu9j2IhoCwaDIZawITECAGtplBaqJFrKd4J0cOiBc3k5Scuq/UrqXSsr9kikg+xPJqlcTU7y+DENuUpd74zhZiFGJkGvoqe29ZwMI7cagrtqfTWv37IYODZOmQLQvtVYQ4vBl7o7oXhlWMxNiKM2KTSTGXfqmFydzY2Titi9kA9Bzb0ibcXQe3Hi6C2nUJvtHBjvQkb+GMW7WFbWptQTLGS7iMJ5WvcqTBOsZ/bvZFrxeIiZJ3OacgXR1y/qA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: There are two separate checks in prctl_enable_tagged_addr() that nr_bits is in the correct range. The checks are arranged such the correct case is sandwiched between both error cases, which do exactly the same thing. Simplify the if condition and pull the correct case outside with the rest of the success code path. Signed-off-by: Yosry Ahmed Reviewed-by: Kirill A. Shutemov --- arch/x86/kernel/process_64.c | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-) diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index 3dd8c1f51e45c..8349431136530 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -783,17 +783,13 @@ static int prctl_enable_tagged_addr(struct mm_struct *mm, unsigned long nr_bits) return -EBUSY; } - if (!nr_bits) { - mmap_write_unlock(mm); - return -EINVAL; - } else if (nr_bits <= LAM_U57_BITS) { - mm->context.lam_cr3_mask = X86_CR3_LAM_U57; - mm->context.untag_mask = ~GENMASK(62, 57); - } else { + if (!nr_bits || nr_bits > LAM_U57_BITS) { mmap_write_unlock(mm); return -EINVAL; } + mm->context.lam_cr3_mask = X86_CR3_LAM_U57; + mm->context.untag_mask = ~GENMASK(62, 57); on_each_cpu_mask(mm_cpumask(mm), enable_lam_func, mm, true); set_bit(MM_CONTEXT_LOCK_LAM, &mm->context.flags);