From patchwork Tue Mar 12 03:59:49 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13589505 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A9876C54E49 for ; Tue, 12 Mar 2024 03:59:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 229E76B0141; Mon, 11 Mar 2024 23:59:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1D8C16B0145; Mon, 11 Mar 2024 23:59:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0A2486B0143; Mon, 11 Mar 2024 23:59:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id EE6956B017E for ; Mon, 11 Mar 2024 23:59:56 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 9E79580D96 for ; Tue, 12 Mar 2024 03:59:56 +0000 (UTC) X-FDA: 81887033592.21.DEEDBC6 Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf16.hostedemail.com (Postfix) with ESMTP id ED5AD180006 for ; Tue, 12 Mar 2024 03:59:54 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=E0SdI+SX; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf16.hostedemail.com: domain of 3OtPvZQoKCNUPFJIP18D547FF7C5.3FDC9ELO-DDBM13B.FI7@flex--yosryahmed.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3OtPvZQoKCNUPFJIP18D547FF7C5.3FDC9ELO-DDBM13B.FI7@flex--yosryahmed.bounces.google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710215995; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=H0vvm77KC92mPCMDKE7JIHK7Q88ob4yruXgItrPO0iI=; b=2ALLYI/BA9StcsNhO8JxHQFHJ4m+kuC0Al8w9y8GPUc/xOLw25UgKRJO3DQnFGIispD/IY wH5XzZkyowKW9/DhOIVzrCFI6y9MaFZLY0BkMukaBKTG1Y33Hlx9oenkVnOsr/a8f0zxYA ZxUFV7i9ajl8Q7UPv2HgBVZNMp8uj7c= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=E0SdI+SX; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf16.hostedemail.com: domain of 3OtPvZQoKCNUPFJIP18D547FF7C5.3FDC9ELO-DDBM13B.FI7@flex--yosryahmed.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3OtPvZQoKCNUPFJIP18D547FF7C5.3FDC9ELO-DDBM13B.FI7@flex--yosryahmed.bounces.google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710215995; a=rsa-sha256; cv=none; b=Pffl/lN02jF2wih2ERKqb4FZrWigyGGCiiF3Ni1e6sqabxG5Y9dBPJO3YM1EJ8aUhw6et/ B9N4ZaIsz8edeBqbL8dQTo+9EpeMbvdwskVLGozG+zCvToQmDeuruJrs53zL9e3MoA6ufn R+QanDlLcgjy8vFKYsMB0punS/n0mPc= Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dd169dd4183so305602276.3 for ; Mon, 11 Mar 2024 20:59:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1710215994; x=1710820794; darn=kvack.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=H0vvm77KC92mPCMDKE7JIHK7Q88ob4yruXgItrPO0iI=; b=E0SdI+SX+84Wh7sl0TgXUJ/x4XHAyzIvijfqksNzUUhEH+Yvizuq00CTBnbrdYMCQ9 7VAfkjOVKVQLeXjesX2dVs6+9FBi1BhB7weUVcWJpjlPD7BQubR+/qy1OSf0ZxUpvcv4 iS4n4SlpiNLXaZVdO2cwje0QzhJmRXVDg2XOFK/ID7FvZT3hsLs+06H5qR1H2zdOL2uc Q5OxjsXnRbxIQz0W+QOEsUi78EMK0jLdqZVgq+lkXQO34jKRXwn2hh/ash/ys3JhXcML IrhmvyvMWNg9aLJNMgc79UlRov1juuBKeOTZV731EME0OaV2hQ9BFX3U+dblcItJ2F64 Djhg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710215994; x=1710820794; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=H0vvm77KC92mPCMDKE7JIHK7Q88ob4yruXgItrPO0iI=; b=cA1vvc6+W4hxOxMZb6pTh7UrkbEYoPKOiD4GHT73gbjEeIA155iq8AbFiV3IeMaba9 KtxF+Ov809BVRBod7MkwYlKKwpiBtzqrP387QPcGYT4P57XlQ5SSSOgMG2gk82ZS1xBX EzJaKQ/gCSP/aGRyRK6jQDxucp/Q/+NnX8ByDWTJ9o9r7/QIaSRy8Y9XwdvBKMcym7wn CIueOv07nsb1ejaK9L9rfS22IDXBcyjw3piq99pMOJAIhiYugbWRBr7i9HSm1eUxsSWz IkoQaYyK8rilMczao0AHS0+YhhBk29tAzbYqEQDFvPQs186Kbo0nvCvXsY8symd2WjEv wM9A== X-Forwarded-Encrypted: i=1; AJvYcCVcDOSfDpao72Bc7G/l85ou8B/8c1L72LwP5GRe/JYZGY4FkU08JDZzp74S/WDlYs01Siix+kDgd/xrYvm5nxj0Gww= X-Gm-Message-State: AOJu0YyDjnDsH3i+cHtZrlt+ltyzBSzjGYiUjO7Kv+q4feU9BTmONrSt 8jONfZ9APahZC6iiC6KKglWp3sAIvMliKx5qXn4RyGrZKIERYRz5Xdon/1B7Axu8fq883FXGDxj HBsROrsl6SBKPL6iYwg== X-Google-Smtp-Source: AGHT+IE3j2RbmWLGWhXwtCN+tJydKqNDYdq63uVBUe/GpO70ij5hxpAGdtBQ8Cwz7kcE3p6RwDKKNDFJe0VFKiIx X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:29b4]) (user=yosryahmed job=sendgmr) by 2002:a05:6902:f02:b0:dcd:2f2d:7a0f with SMTP id et2-20020a0569020f0200b00dcd2f2d7a0fmr427882ybb.9.1710215994014; Mon, 11 Mar 2024 20:59:54 -0700 (PDT) Date: Tue, 12 Mar 2024 03:59:49 +0000 Mime-Version: 1.0 X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240312035951.3535980-1-yosryahmed@google.com> Subject: [PATCH 1/3] x86/mm: Use IPIs to synchronize LAM enablement From: Yosry Ahmed To: x86@kernel.org Cc: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , "Kirill A. Shutemov" , Rick Edgecombe , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yosry Ahmed X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: ED5AD180006 X-Stat-Signature: pc5otxuywab5teziwxp1nnm1y8e3sf6b X-Rspam-User: X-HE-Tag: 1710215994-607089 X-HE-Meta: U2FsdGVkX19i/Mmac58M9cO6J977hR/X8vhWZQpmO59qWrQTun02je2j2FOXsDgK0fgqyVbQycXAtl5A2WwWGtDE/1gRutZ48jdPXXtFtOIqF9BtTQnzVF0QL1fdx35RX3yuQCyg6+J/6sUxYSwvPZkm+4neZLKfRWdRahS85epLQISBNj1paEA2a2tcSdD49wHcq+FxW4oTTMVzA7yOjatPSHQxCYRFCoQuhp8zDkW2MBixsQBNxiG8tCk/vJzeCrgupWElRd2lvd7/wcO0cMFuP5Xm65gImQN4tYWiO8GtYCLCepaPl/mcLR2+daE172MGbUJdsUzhIwgg5HZjcHAPz/hz5GxOSkJjqukahTMV+H0MD2aYD9IMSnKb7NE6SyLqjxkkhdgnkykceIJpZIqilDu9pABcwWUf68VhXME1jgolv88zdAf5J5iZP0TNtNy50HTqpIEpLbT5Hb98VjPmdNX+wAy27gl1EqXrnzViyVA0cPdAbmHq4V485In0BmO/sARoihbjcLuEtTmaNpiqvA/NdpegvJLAHx0Qw2S/oJHRDXJZ4AryB1zjtjPjBn+6HYEj3SYlyI6yFow5YlYm0kpQRZ9BQBs20yp7x+GnFY0BnusGd573n5tR6TCdmk0zU3qUV4PI4G3aDk59Q4Abuic3F+UE6V2SUNnRKxns5VLj0L9giC8CjGGCymUO7MnGM7nJoBMkj18gvWNgKghXmVJWEqIynonIQ6Bw0Lj9jGhXJ/Ip3jMiICHbNlsXLwJGqq2zecV/Ppzd0NXP38jw+vsIuerlZ9j9hqw1TqZoUAPUCUuNIJEJqVG69RzEow8d4Fac2ehSiWcaIq9cOxwbVdt4OeErWiZRh5ZPdscmXipC6TtafWaHVfa7/NFEjARFVbubQc5pTEJ6l74QO8C31hkny6CnbVUaS7mwN6GOihwpgQtVowwrCG3wTlC/0eNNqseEOQxmPkSMiZ3 2M79jOur cw7XUu4eL00/d1+SJ62pCtQVwz4aU7C1j+ikwYmj6LZJ6lDyBHcTCgL09bgVhWboder8mRvJMdptxC4sa5psYZS42jgUW94HkoimOg6D01mUif1Z9Y1+U5glmb+QXDsJqWN8QIOYib9xR0BeNaKghrGi0ynchzwfEsQVkO8DPrHoHFz99Equa0mn7E5dVBzLS/D52Nk5q4cp6dvpum2NEYYUObMs7/j4Ezs3JQIZnqDVmyhngjj9GqI7GaxT/5iPN8ZLWexDX78Z/iAgd/FOdSGDjVIMzVVmAc3Y+Hrk+lE5lDR4s1Ci1N+Vzr+3Kh63KXcEbrPV0R23xz/JSfuF41SG9/rluqfIrsUA9MjyF1ccG9+3R6nYo4PfRbS4A1gGeldnfql01z2aDFR4R0F6HT4kSkutGlq0OGrO0PrS0nyKZeztF1X/jggoGaNsgQKC/DmJFAM+gsaC0J9qn8qqr7UNoTq4gFfZ9FYsJoB39TQTtOvz1nyFqC2s5ZppQdDmonsT41/z/3bjVvHgD3hTk8O9krrwz1dig7yitRDLlM2ZTLGqxrVGNbMEvdaa4rZrTDESM7EBN8RDjgOYQREqNo1RUPmv9Sm7ZoWxML/VoD/4L8o/IyR8Pd4cxqyebZekEyn08Y04RBS/6n+jOHsJ5zJaVoaQuWxzjvLxVSoGBQv7HzFMGynXB93CUyvmpZZl7zew5r1RUDe+Ouzg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: LAM can only be enabled when a process is single-threaded. But _kernel_ threads can temporarily use a single-threaded process's mm. If LAM is enabled by a userspace process while a kthread is using its mm, the kthread will not observe LAM enablement (i.e. LAM will be disabled in CR3). This could be fine for the kthread itself, as LAM only affects userspace addresses. However, if the kthread context switches to a thread in the same userspace process, CR3 may or may not be updated because the mm_struct doesn't change (based on pending TLB flushes). If CR3 is not updated, the userspace thread will run incorrectly with LAM disabled, which may cause page faults when using tagged addresses. Example scenario: CPU 1 CPU 2 /* kthread */ kthread_use_mm() /* user thread */ prctl_enable_tagged_addr() /* LAM enabled on CPU 2 */ /* LAM disabled on CPU 1 */ context_switch() /* to CPU 1 */ /* Switching to user thread */ switch_mm_irqs_off() /* CR3 not updated */ /* LAM is still disabled on CPU 1 */ Synchronize LAM enablement by sending an IPI from prctl_enable_tagged_addr() to all CPUs running with the mm_struct to enable LAM. This makes sure LAM is enabled on CPU 1 in the above scenario before prctl_enable_tagged_addr() returns and userspace starts using tagged addresses, and before it's possible to run the userspace process on CPU 1. In switch_mm_irqs_off(), move reading the LAM mask until after mm_cpumask() is updated. This ensures that if an outdated LAM mask is written to CR3, an IPI is received to update it right after IRQs are re-enabled. Fixes: 82721d8b25d7 ("x86/mm: Handle LAM on context switch") Suggested-by: Andy Lutomirski Signed-off-by: Yosry Ahmed Reviewed-by: Kirill A. Shutemov --- arch/x86/kernel/process_64.c | 13 +++++++++++-- arch/x86/mm/tlb.c | 7 +++---- 2 files changed, 14 insertions(+), 6 deletions(-) diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index 33b268747bb7b..76e91fc68c5f3 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -750,6 +750,16 @@ static long prctl_map_vdso(const struct vdso_image *image, unsigned long addr) #define LAM_U57_BITS 6 +static void enable_lam_func(void *__mm) +{ + struct mm_struct *mm = __mm; + + if (this_cpu_read(cpu_tlbstate.loaded_mm) == mm) { + write_cr3(__read_cr3() | mm->context.lam_cr3_mask); + set_tlbstate_lam_mode(mm); + } +} + static int prctl_enable_tagged_addr(struct mm_struct *mm, unsigned long nr_bits) { if (!cpu_feature_enabled(X86_FEATURE_LAM)) @@ -782,8 +792,7 @@ static int prctl_enable_tagged_addr(struct mm_struct *mm, unsigned long nr_bits) return -EINVAL; } - write_cr3(__read_cr3() | mm->context.lam_cr3_mask); - set_tlbstate_lam_mode(mm); + on_each_cpu_mask(mm_cpumask(mm), enable_lam_func, mm, true); set_bit(MM_CONTEXT_LOCK_LAM, &mm->context.flags); mmap_write_unlock(mm); diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 5768d386efab6..e8feb2e154db2 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -497,9 +497,9 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, { struct mm_struct *real_prev = this_cpu_read(cpu_tlbstate.loaded_mm); u16 prev_asid = this_cpu_read(cpu_tlbstate.loaded_mm_asid); - unsigned long new_lam = mm_lam_cr3_mask(next); bool was_lazy = this_cpu_read(cpu_tlbstate_shared.is_lazy); unsigned cpu = smp_processor_id(); + unsigned long new_lam; u64 next_tlb_gen; bool need_flush; u16 new_asid; @@ -622,9 +622,7 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, cpumask_clear_cpu(cpu, mm_cpumask(real_prev)); } - /* - * Start remote flushes and then read tlb_gen. - */ + /* Start receiving IPIs and then read tlb_gen (and LAM below) */ if (next != &init_mm) cpumask_set_cpu(cpu, mm_cpumask(next)); next_tlb_gen = atomic64_read(&next->context.tlb_gen); @@ -636,6 +634,7 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, barrier(); } + new_lam = mm_lam_cr3_mask(next); set_tlbstate_lam_mode(next); if (need_flush) { this_cpu_write(cpu_tlbstate.ctxs[new_asid].ctx_id, next->context.ctx_id); From patchwork Tue Mar 12 03:59:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13589506 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 157CAC54E58 for ; Tue, 12 Mar 2024 03:59:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 956DD6B014D; Mon, 11 Mar 2024 23:59:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 92C4C6B0180; Mon, 11 Mar 2024 23:59:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7A5C66B0181; Mon, 11 Mar 2024 23:59:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 691F46B014D for ; Mon, 11 Mar 2024 23:59:58 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 4389C8049C for ; Tue, 12 Mar 2024 03:59:58 +0000 (UTC) X-FDA: 81887033676.11.926D31A Received: from mail-yb1-f202.google.com (mail-yb1-f202.google.com [209.85.219.202]) by imf21.hostedemail.com (Postfix) with ESMTP id 743221C0003 for ; Tue, 12 Mar 2024 03:59:56 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=1yoUKlvR; spf=pass (imf21.hostedemail.com: domain of 3O9PvZQoKCNYQGKJQ29E658GG8D6.4GEDAFMP-EECN24C.GJ8@flex--yosryahmed.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3O9PvZQoKCNYQGKJQ29E658GG8D6.4GEDAFMP-EECN24C.GJ8@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710215996; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GoKed6yKAS+GwnXsUmVoWRiO+MONN8g5vxEH6EyDAwM=; b=qI+EC9JWlpAV35j/mrcLuGCGqqGW/xmOt2jzB6qWsBAbIIMhS+BQwHgDHpLQiAgSWn8pDR hqzcfEVcNCMNC+bGEvaEgZ0AerIKoG47TYCj4eBAFG7kVsJ11u2EbWQ55Knm516l6wyKsI X0UgwFP9gygypRcof7lOoVd4tXaKYDs= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710215996; a=rsa-sha256; cv=none; b=CD3uQuGbp3XnpElr7cnsTlVpI9ULQAh4iPiLazgslQSGY9FTA9Aj47IVAjGnRRmds+oXs6 GoCdUPtcShIIoVoJvDiXForqB/Bq7xISSv+GmtOh3UqxvNP5smW9RS66C+WfK8uBSONgi/ 5PcHn/kxnguQU/qslD3qfQyp6Od9ZQ4= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=1yoUKlvR; spf=pass (imf21.hostedemail.com: domain of 3O9PvZQoKCNYQGKJQ29E658GG8D6.4GEDAFMP-EECN24C.GJ8@flex--yosryahmed.bounces.google.com designates 209.85.219.202 as permitted sender) smtp.mailfrom=3O9PvZQoKCNYQGKJQ29E658GG8D6.4GEDAFMP-EECN24C.GJ8@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f202.google.com with SMTP id 3f1490d57ef6-dc6b2682870so8396024276.0 for ; Mon, 11 Mar 2024 20:59:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1710215995; x=1710820795; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=GoKed6yKAS+GwnXsUmVoWRiO+MONN8g5vxEH6EyDAwM=; b=1yoUKlvRxnGBD7RuvXFLIZhbl9whEfzePMBx6ZPwvjJdjr8yNNmmEpZzXvqzqG3gy8 eaS9+m2iZMNXlzagS04L9EobfIYS3uENvvlTxT/gNYpT3AZZqDUM9sOzcJ7dRSxJIX1Z BLvLkrFO5cyZNF6+XOY/LQ+gk1TadQrBk2/5Lj8ppBgy/jXF/6DGCOEXcE3C8y4kjbZ8 ntY3fYl7crqBQdlMqt1hafs6ITf+TrmZ8O+zMdMo94cHV/2BXQ2TsdPuzZRg/pk1STb9 09G0ekdOtwoGA45J8IMiSxOgAu85IpJwS6E0VL1h6K1rlDKRwybURvmujMW/UTs3dGM5 ZtYA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710215995; x=1710820795; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=GoKed6yKAS+GwnXsUmVoWRiO+MONN8g5vxEH6EyDAwM=; b=BUF9FKKu9klegz0x8IvMXtWC7ffwW36oX16vXap4unOQ8eQuqZy9UXolG5B0DU5Mks x46cOC3QpD08VGIZdViX69fd2q1IuC7T9D7IvcT002ZXqD11Ayo2TG3omKqvdaN3uNPj Omc9cSQM/QAGzNmwmCD7NIk2QhC+WWr4qPCBIDuHF/n5RAQ88xexOvEEPA4HymAwydZr y3djWrYD4gsZXtSlMgAGLQtXTDd/aiNlSlYWUIwjqHUx9nipA4CziptWBfRizi5r24qq sRzhd7Ry7ID1aIE9oUZRqimxceH9Pz4VzAOkwilwHgNGymmGUCUbPssuPHIPizhgfTVE X2tw== X-Forwarded-Encrypted: i=1; AJvYcCW9kBIDAlcRBlYFQlzbufFEfE1kP6/nOG0qa5/mZwHfDuD+acPYOIA9VZffjBP37QNfrV36x6mbk4braArOPQtcerE= X-Gm-Message-State: AOJu0Yy1arGNTbkFBDEGtBbjZpFM34fELlFSuWrQTHC+YJvUT4UeCUB5 hF5yuuoMEMWB9SvySEz0QrZvBbtrhMJvvOWJLSRUhp4deehhb4mQp2RpE5g1nLW6dyXfmPxRJhc kYjb7ZDM0Gf/u87Hs6w== X-Google-Smtp-Source: AGHT+IExFRL/azN3q8HgBq6A2xtMpxQXpFaPTK8KJYQ1jXBub6ECtGohzNaeSuMsvvLKLtLSLkGboXwUHiFLXv1f X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:29b4]) (user=yosryahmed job=sendgmr) by 2002:a05:6902:1081:b0:dc7:8e30:e2e3 with SMTP id v1-20020a056902108100b00dc78e30e2e3mr2416970ybu.2.1710215995630; Mon, 11 Mar 2024 20:59:55 -0700 (PDT) Date: Tue, 12 Mar 2024 03:59:50 +0000 In-Reply-To: <20240312035951.3535980-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20240312035951.3535980-1-yosryahmed@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240312035951.3535980-2-yosryahmed@google.com> Subject: [PATCH 2/3] x86/mm: Fix LAM inconsistency during context switch From: Yosry Ahmed To: x86@kernel.org Cc: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , "Kirill A. Shutemov" , Rick Edgecombe , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yosry Ahmed X-Rspamd-Queue-Id: 743221C0003 X-Rspam-User: X-Stat-Signature: w8fka8xjwd8scbnip9qfjia8akfw1yuc X-Rspamd-Server: rspam03 X-HE-Tag: 1710215996-127249 X-HE-Meta: U2FsdGVkX1+jxzj4DZuCqaj6bC7g/FUSnrjm+1nYYkN06GmdNbjDRymQzoRjfBNk7UDooYjmurCfwxvildX8G2sB9bMaEUYw/0G9xxc0lUgi1jVjc5Zy2Rg1+8GGtbSDNwhHQBoH/K61d8+nhymKmsNcFzFjAoy/NkjXeWF+3Zniy7h+g5k3Nkmf5w8Pf52CRSOQ/5C7TOySc0zekGNt7I7Zcyh2pQ70T0izZ6aUYbkOpiwhvV+uS+z8ur+gaiDAsWlet4QjSo8KYq7HZnU6FbURrd5QDfuB+eqx9+M9uHrjpBQ3kVhuyy48yD25ZubSVpdn31yVqO+Xk8oQzZmurrH2B45fYmjOGnUusSoO59TXeghwzJjTo/jqtjKXS8hu3f9H6OjM66Tak35BNdWvJ6ZElyOzq/8C5p0idp7VYYVfISzI1FqkMVkXy+7fu313WY/DcLGTP0Khw6y3CdWty/wBRliBhFpzc2j3gVA3CGbbt66OpOuQDlX/CQA/P46yf5iI+z2yWatrhQic/pVAh5ZJu52Ja/yPcPAF0w+KMwdkIW2hGY3JO9X65NqoUnMOLLVGN70KFT1WYn9YBXzlb6pS8TpkvoHfpflAWJiQwz1WFkIHZ4bBHeI8FVZrsyyaei3l2zJAcaBxX1B9ypfEHTpTfwHJTCQ32aPleDYN3opXqEnYrvA36RS3qvCACN+n8/RA/zB85KuCrNQAocEordxlboW4lA6E3Fm7JjwghPxtg4zFR2tyOWYJC/FxokJfajChufB5jfYhoYp2SLqArfk9zUoZj8tSuqjZy9XzjiCVrFbGDdg+cki6C8vovjtiuhTFQayJp7e4P0fwLIDDbmVNFFqu3Vf0oTNsX4BGU2LxFuouZ5z9h0JPwRbp0ccuFn68QVsx7ebYg0DD1Hjf/xntwb/dcVB6oUWsLuU0wPu7wGlH6WO0/YLtgbaymg7V0skofpjo8gy69DGyvqo vha5CGWr GgwgUTsm5pWXfYI0Q5NG0eE+MbKh5EAq+qduuJToqW8irtqT2bNLF+TGQ6AEn9EDOHx3ZsLSahDtY6LGJX/krJ4NBOBHU33fSAlSZ9iRMdD6blJZImK7Br6HzKfRcsBPgnwum4CGN5EjVBHBuScaNPZA2tllKXcNI7SwXTOOfsszxqJWsUdVTsZ5l7TsDifZtGPjCjfTL6zDseAxNd4cAhJEQBy5lBE2YlHFTSc8Fv5EQZwMGeDywA1ziy18YVarOVl+fcgVI11V/uDYu+EeMy0AnTkt8RIO0yqWPslI20hSfgKZiauLxLzqVeUkWEWTkpG42T8mN1DfZ5HqcdO4VGvGUJsOTJl429vmWze8E+IhwALHg94bP74Ny8fkuB8riAjJdJfpyX/wJzFP5Agcn2p6E3ptjd87w1JxNdZIiiMDwn1y4onQNJEDK4dBffR/HWZNan8RCzdZNaMNJ5KEUQ8avz4BofzIdZlv7RTaIWuaDeQB9Y19aB5uhe8wjbrLOnuS4PHgOI4DFtzohfm9h+qCI2ue4NkgeoCo4/C7d1cruLG5mtPxbJ6OcXeQoRUS1YA1woikFLzWH8kyG1PthYswV99rhMmRRgHhlfp4mAkqdiuiVIUh8gCRR7uEY+8Y0rXG7w8Nk9az37yo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: LAM can only be enabled when a process is single-threaded. But _kernel_ threads can temporarily use a single-threaded process's mm. That means that a context-switching kernel thread can race and observe the mm's LAM metadata (mm->context.lam_cr3_mask) change. The context switch code does two logical things with that metadata: populate CR3 and populate 'cpu_tlbstate.lam'. If it hits this race, 'cpu_tlbstate.lam' and CR3 can end up out of sync. This de-synchronization is currently harmless. But it is confusing and might lead to warnings or real bugs. Update set_tlbstate_lam_mode() to take in the LAM mask and untag mask instead of an mm_struct pointer, and while we are at it, rename it to cpu_tlbstate_update_lam(). This should also make it clearer that we are updating cpu_tlbstate. In switch_mm_irqs_off(), read the LAM mask once and use it for both the cpu_tlbstate update and the CR3 update. Signed-off-by: Yosry Ahmed --- arch/x86/include/asm/mmu_context.h | 8 +++++++- arch/x86/include/asm/tlbflush.h | 9 ++++----- arch/x86/kernel/process_64.c | 6 ++++-- arch/x86/mm/tlb.c | 8 +++++--- 4 files changed, 20 insertions(+), 11 deletions(-) diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index 8dac45a2c7fcf..19091ebb86338 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -88,7 +88,13 @@ static inline void switch_ldt(struct mm_struct *prev, struct mm_struct *next) #ifdef CONFIG_ADDRESS_MASKING static inline unsigned long mm_lam_cr3_mask(struct mm_struct *mm) { - return mm->context.lam_cr3_mask; + /* + * When switch_mm_irqs_off() is called for a kthread, it may race with + * LAM enablement. switch_mm_irqs_off() uses the LAM mask to do two + * things: populate CR3 and populate 'cpu_tlbstate.lam'. Make sure it + * reads a single value for both. + */ + return READ_ONCE(mm->context.lam_cr3_mask); } static inline void dup_lam(struct mm_struct *oldmm, struct mm_struct *mm) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 25726893c6f4d..69e79fff41b80 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -399,11 +399,10 @@ static inline u64 tlbstate_lam_cr3_mask(void) return lam << X86_CR3_LAM_U57_BIT; } -static inline void set_tlbstate_lam_mode(struct mm_struct *mm) +static inline void cpu_tlbstate_update_lam(unsigned long lam, u64 untag_mask) { - this_cpu_write(cpu_tlbstate.lam, - mm->context.lam_cr3_mask >> X86_CR3_LAM_U57_BIT); - this_cpu_write(tlbstate_untag_mask, mm->context.untag_mask); + this_cpu_write(cpu_tlbstate.lam, lam >> X86_CR3_LAM_U57_BIT); + this_cpu_write(tlbstate_untag_mask, untag_mask); } #else @@ -413,7 +412,7 @@ static inline u64 tlbstate_lam_cr3_mask(void) return 0; } -static inline void set_tlbstate_lam_mode(struct mm_struct *mm) +static inline void cpu_tlbstate_update_lam(unsigned long lam, u64 untag_mask) { } #endif diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index 76e91fc68c5f3..748d2b3bdb985 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -753,10 +753,12 @@ static long prctl_map_vdso(const struct vdso_image *image, unsigned long addr) static void enable_lam_func(void *__mm) { struct mm_struct *mm = __mm; + unsigned long lam = mm_lam_cr3_mask(mm); + u64 untag_mask = mm_untag_mask(mm); if (this_cpu_read(cpu_tlbstate.loaded_mm) == mm) { - write_cr3(__read_cr3() | mm->context.lam_cr3_mask); - set_tlbstate_lam_mode(mm); + write_cr3(__read_cr3() | lam); + cpu_tlbstate_update_lam(lam, untag_mask); } } diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index e8feb2e154db2..b2f74c451b3d7 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -11,6 +11,7 @@ #include #include #include +#include #include #include @@ -635,7 +636,6 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, } new_lam = mm_lam_cr3_mask(next); - set_tlbstate_lam_mode(next); if (need_flush) { this_cpu_write(cpu_tlbstate.ctxs[new_asid].ctx_id, next->context.ctx_id); this_cpu_write(cpu_tlbstate.ctxs[new_asid].tlb_gen, next_tlb_gen); @@ -654,6 +654,7 @@ void switch_mm_irqs_off(struct mm_struct *prev, struct mm_struct *next, this_cpu_write(cpu_tlbstate.loaded_mm, next); this_cpu_write(cpu_tlbstate.loaded_mm_asid, new_asid); + cpu_tlbstate_update_lam(new_lam, mm_untag_mask(next)); if (next != real_prev) { cr4_update_pce_mm(next); @@ -700,6 +701,7 @@ void initialize_tlbstate_and_flush(void) int i; struct mm_struct *mm = this_cpu_read(cpu_tlbstate.loaded_mm); u64 tlb_gen = atomic64_read(&init_mm.context.tlb_gen); + unsigned long lam = mm_lam_cr3_mask(mm); unsigned long cr3 = __read_cr3(); /* Assert that CR3 already references the right mm. */ @@ -707,7 +709,7 @@ void initialize_tlbstate_and_flush(void) /* LAM expected to be disabled */ WARN_ON(cr3 & (X86_CR3_LAM_U48 | X86_CR3_LAM_U57)); - WARN_ON(mm_lam_cr3_mask(mm)); + WARN_ON(lam); /* * Assert that CR4.PCIDE is set if needed. (CR4.PCIDE initialization @@ -726,7 +728,7 @@ void initialize_tlbstate_and_flush(void) this_cpu_write(cpu_tlbstate.next_asid, 1); this_cpu_write(cpu_tlbstate.ctxs[0].ctx_id, mm->context.ctx_id); this_cpu_write(cpu_tlbstate.ctxs[0].tlb_gen, tlb_gen); - set_tlbstate_lam_mode(mm); + cpu_tlbstate_update_lam(lam, mm_untag_mask(mm)); for (i = 1; i < TLB_NR_DYN_ASIDS; i++) this_cpu_write(cpu_tlbstate.ctxs[i].ctx_id, 0); From patchwork Tue Mar 12 03:59:51 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13589507 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 533CCC54E49 for ; Tue, 12 Mar 2024 04:00:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6983E6B0181; Tue, 12 Mar 2024 00:00:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6464F6B0182; Tue, 12 Mar 2024 00:00:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4EE956B0183; Tue, 12 Mar 2024 00:00:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 3BC0E6B0181 for ; Tue, 12 Mar 2024 00:00:00 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id D21FB1C106A for ; Tue, 12 Mar 2024 03:59:59 +0000 (UTC) X-FDA: 81887033718.30.1AA5E07 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf19.hostedemail.com (Postfix) with ESMTP id 2F6DA1A000C for ; Tue, 12 Mar 2024 03:59:58 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=gdtrCXj+; spf=pass (imf19.hostedemail.com: domain of 3PdPvZQoKCNgSIMLS4BG87AIIAF8.6IGFCHOR-GGEP46E.ILA@flex--yosryahmed.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3PdPvZQoKCNgSIMLS4BG87AIIAF8.6IGFCHOR-GGEP46E.ILA@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710215998; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=UAJ+3ue8h2GtGeMmVF7aptze9cHfLRbMGaK/pSc8DQk=; b=cVnqsMVtejCoNic03zYi9undY3Fh2lkEfPxVplhemw/eQldARygK5yTn82hDSAf/JhTI0z 04w1cXXJByIfHmU8mtn9Zeh1mXrhzcM/AMs+1xfwC1zX9v+AaeJGGOkFqgGIxdWMjpdgOL fx2k+00MWX7OTTwryI45car7dkmC0CE= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=gdtrCXj+; spf=pass (imf19.hostedemail.com: domain of 3PdPvZQoKCNgSIMLS4BG87AIIAF8.6IGFCHOR-GGEP46E.ILA@flex--yosryahmed.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3PdPvZQoKCNgSIMLS4BG87AIIAF8.6IGFCHOR-GGEP46E.ILA@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710215998; a=rsa-sha256; cv=none; b=qv44Ke+cQGF+E17XgdLfSkOxELKHtDlULHtmI7EBaSfvMmt0yvACac2GhYjQPHDYagzVyB 3wvKXKjPtotIzc1LFli0wdI/awpvaATMWurNikDnum6uVLoBh95muTWT2h+325adV/ySuB LdT68uq7hLRXg/L/ndaTIAqvY0ZD2bA= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dd0ae66422fso10008189276.0 for ; Mon, 11 Mar 2024 20:59:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1710215997; x=1710820797; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=UAJ+3ue8h2GtGeMmVF7aptze9cHfLRbMGaK/pSc8DQk=; b=gdtrCXj+CKpi8jNlRjTE3mTpschFMKGslZpRqkTT9OK1eRCS7+GyINDLhCa6nE0XoX EogvG/wGcOkJDTgUftQGsXaqQBdk/qaMJy1agu2Dm8+1/NCdHkn1uImQN0iT80Dus4qA SeZMKUGjkyAYjjxuyuGYx29UYB2I+me8YwZCHQOVg46z9YSB/woV6umxsX3fHuHaiCg7 z+SbzJ7csQP6kMIoklULWn0vF9idLy1dt/V/b9TjKTCgeCUwV5fGQiHOCmr0fgwG5+sV qRoVj/HtbUYzzYlLG/p0w5HF1nhW140sy7IJm0M+W8mFYmUqHOxUOmyPnVZT5vXuHXUG +L9g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1710215997; x=1710820797; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=UAJ+3ue8h2GtGeMmVF7aptze9cHfLRbMGaK/pSc8DQk=; b=LJgycmPsSg+l1x0CKcl1I7ZatNwSqZ22JTDeRLoyJ6XZMKnnTUmAwsPLh2qJQCmCNu 5EHfUW57Cjl+NPne5pUo2WEriiAsc5SXX0KKp9OBTjNf7VEX13zpF7vfoy+xhjX7+kDr QwK+5iUWfevk5oO+23n5p7eQkl8nTZCxqNVi3+hNRrLjAtaJg9/gXZA7SSkGs5USK/El yv8aiUgTvr2lolinRLXtS22zymTRlVWmV/MA1e+ba24F/g5iUUw4qtGE7OGZuphog/Yh q/imIlMVlJcSJrtE+xVSt76c4Pr7RUq+YZjPK6sznnHoi/22OHWZtLmvXUgv2sMDyJUh qlDQ== X-Forwarded-Encrypted: i=1; AJvYcCVvlteFcmfs1rR3gqtiL7RtlARCW1wVN5UCu+RcAhw7etZ3EKml5c+nVPv44MzcynZuUn9g9pCH50o849dy3WmXcco= X-Gm-Message-State: AOJu0YyNANzPjPU5ZELQ8eaWzWyRrUAD0zSChBTKL1KDG2wVWnxGLrtk cZe0omhThWAJEroJptpcKGqyixlkj+dDg5FAej9M3rqYdXbG320K2EDceBxC2UKIZHAGIY7Erce rw4O1weoJ79B5ynQq/A== X-Google-Smtp-Source: AGHT+IHHHQjD/GEWT164UF9qNBAO6Zdv0oaS7UwIqMuvXE4Wlv6Tn55WWEmKnE3lYPTXnbsqaib7icEdWoIEAoBS X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:29b4]) (user=yosryahmed job=sendgmr) by 2002:a25:98c5:0:b0:dc6:e884:2342 with SMTP id m5-20020a2598c5000000b00dc6e8842342mr2710694ybo.5.1710215997281; Mon, 11 Mar 2024 20:59:57 -0700 (PDT) Date: Tue, 12 Mar 2024 03:59:51 +0000 In-Reply-To: <20240312035951.3535980-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20240312035951.3535980-1-yosryahmed@google.com> X-Mailer: git-send-email 2.44.0.278.ge034bb2e1d-goog Message-ID: <20240312035951.3535980-3-yosryahmed@google.com> Subject: [PATCH 3/3] x86/mm: Cleanup prctl_enable_tagged_addr() nr_bits error checking From: Yosry Ahmed To: x86@kernel.org Cc: Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , "Kirill A. Shutemov" , Rick Edgecombe , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yosry Ahmed X-Rspamd-Queue-Id: 2F6DA1A000C X-Rspam-User: X-Stat-Signature: tuxway7uauqni68agnu3b989ea5a11kj X-Rspamd-Server: rspam01 X-HE-Tag: 1710215998-262722 X-HE-Meta: U2FsdGVkX18w87EE23m7vn3lIQRy81Loa34KUq1Kkz2nR/JsodBM3JqtG48+C2C5JC0tGzKToDeGH5tTkM6C/9/yiGi/3MnWt31uuedPlAPB0C3jk2QHBKAS7qAEsLb86VmKBq3Tg2GJy7SIfUkQyMVNprJ8bu/man8KPNn/rra2uW4l8Tm87wHs/dddVG2HUXT6hvKwh31Iip1EroD3YybvVIHjOnEhoYbI8KBA1svxo6NlOAHSRM+CKtoPPAUgwVY3+S7KRZ6rF2QabM3H+squBfg3WAq9N7Iz/DGKDZYhx2zyHjPpORfSao2VKVEL99s0V/O3JHxGy+hqTn8log+K1xcid/Hk5V2q9vBp08ncVRoLfXl5o2+tjq7UKOC51PVmLQzu0Mj1l9yu3qwIeS8cRSdaTe0n/9OvH21HrVqrayWXnu8E/goCxOHavlm30wSnAFEMZHigMfpSoLdjBg6FUhqybjM56RX6S8o/V9vc6Pvs5hmCYnqiuOvKWVklDJetlBZ2l3ZgIt5mT2JgHkuIvISvXytUnkuqZXrKX+3ex/KlgWYoxHDSmVyGQ8HhgC8YVqdDvHXZ0XR0kfOl6EG9suPGaDTxNf38bYidrzqh4GqTTNCD/x0HLBuWCYC/cpJOGY5SuuhvG3ytgSP2AtRQrh23nY4yJ+YoLk8BFDtxj8/S+30KoopyAO1W74U7gdCsila9laOPhj/mDXMNbqf4ahBXR9KVuhCNbE8mQMocem3eQ4221d/p6nI/QhIxTPtLe7B3UHdjeujuL4RggxOV5L5d7qvr91Jsh3+P98eIiqE23lwQ+JGBVOFVhGxD4zFmjfQznoaSnLza2tJ5DzBcpE5fzXEruAqCLo69mzTjl76WmI4xErtjjwsF4g5syt+W+Lg9c01/BG3dryrXCgED/wDqhY5ySuYWcRo9cOzH4ayfG9ZMWZZVkiUdHsRt/KrOxWbdp6SQLDXmOwO OhyVMv+i 0s3hDenoCgOUzGEGzpqX8NsEhjGRL71Z6jBiwRk/I0r04uAYl7N+76URXDIZT3kArbQphDlT3diZx+UxVuomYbF3/nQ0XlXfEWzgprpZ8jfd27VZgPif3UvscFIdPYQ/+dmfP4VA2DBNyjj3c0aPsI/YSkycy94otTftPc6AuyrLAbQop8tnlvFhcypLbDKyM7pv5dT+9l3RBjxbMiUfYqZeQWayeZhY5g0vCntY08yeQM2RAK3M7IadtUPjVH4bj5CahG3QIIE/uO0si5XB+jrfFNHtN6aTjlkyA9gP2Lyz3TGGwVD4Afjwmk4XcA+87Q7bjbgEFj0Ec/bEqKX6NRGcROna5k6scxcc+DBGn1eQ9Ut80RZtIa8XYw+eyQoSyt91va495AdGRE79zzMIyYflJv6rIi3THKXV3LbW6FF3dtwQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: There are two separate checks in prctl_enable_tagged_addr() that nr_bits is in the correct range. The checks are arranged such the correct case is sandwiched between both error cases, which do exactly the same thing. Simplify the if condition and pull the correct case outside with the rest of the success code path. Signed-off-by: Yosry Ahmed Reviewed-by: Kirill A. Shutemov --- arch/x86/kernel/process_64.c | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-) diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index 748d2b3bdb985..0608c4df4e95d 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -783,17 +783,13 @@ static int prctl_enable_tagged_addr(struct mm_struct *mm, unsigned long nr_bits) return -EBUSY; } - if (!nr_bits) { - mmap_write_unlock(mm); - return -EINVAL; - } else if (nr_bits <= LAM_U57_BITS) { - mm->context.lam_cr3_mask = X86_CR3_LAM_U57; - mm->context.untag_mask = ~GENMASK(62, 57); - } else { + if (!nr_bits || nr_bits > LAM_U57_BITS) { mmap_write_unlock(mm); return -EINVAL; } + mm->context.lam_cr3_mask = X86_CR3_LAM_U57; + mm->context.untag_mask = ~GENMASK(62, 57); on_each_cpu_mask(mm_cpumask(mm), enable_lam_func, mm, true); set_bit(MM_CONTEXT_LOCK_LAM, &mm->context.flags);