From patchwork Thu Apr 18 01:28:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13634054 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 05EC1C4345F for ; Thu, 18 Apr 2024 01:28:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E0F936B007B; Wed, 17 Apr 2024 21:28:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DBFAB6B0082; Wed, 17 Apr 2024 21:28:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CAE7F6B0083; Wed, 17 Apr 2024 21:28:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id AB4A36B007B for ; Wed, 17 Apr 2024 21:28:41 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 55ABA12104D for ; Thu, 18 Apr 2024 01:28:41 +0000 (UTC) X-FDA: 82020918042.25.5F4210F Received: from mail-pg1-f202.google.com (mail-pg1-f202.google.com [209.85.215.202]) by imf17.hostedemail.com (Postfix) with ESMTP id 9361640005 for ; Thu, 18 Apr 2024 01:28:39 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=tsgOGAXs; spf=pass (imf17.hostedemail.com: domain of 3RncgZgoKCO0ndhgnPWbTSVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--yosryahmed.bounces.google.com designates 209.85.215.202 as permitted sender) smtp.mailfrom=3RncgZgoKCO0ndhgnPWbTSVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1713403719; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=FYDQyLdFdWMdeiUN2dEgWolx8IdIlWszL4UA+NDjWJ0=; b=WdfrdYHYjaxrCp0N/emb+xBPA4gweS6iGBjDXC/g90YHUPapPMbqbaSqmRpvjcH0UzwS0K Onzn2GYVyf2VsTsHPShKmxBNh6mdkczxgjJrhug2rtGqX3ADqES/2tOePp+JgulyHyAzJm zntSXUKNE3qe7hznVrn+LagySx8AyUM= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=tsgOGAXs; spf=pass (imf17.hostedemail.com: domain of 3RncgZgoKCO0ndhgnPWbTSVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--yosryahmed.bounces.google.com designates 209.85.215.202 as permitted sender) smtp.mailfrom=3RncgZgoKCO0ndhgnPWbTSVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1713403719; a=rsa-sha256; cv=none; b=BUxmnSdoN2BDQwOHOZ/4UZHh3cXJc6EvFS99i+u2BYEpgq291Nw3jy9PbqNiH0WA8PooxR Kq78UQqKXc8UxEoMZqfOfDqe7mIPASOHX1i7Z8bLrDUqK5la/ZEm9jpBvNiM0SfxYzw8a2 i3grBk/VQHxQDawDLFr4VUCrmGXWelU= Received: by mail-pg1-f202.google.com with SMTP id 41be03b00d2f7-5d8dd488e09so372847a12.2 for ; Wed, 17 Apr 2024 18:28:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1713403718; x=1714008518; darn=kvack.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=FYDQyLdFdWMdeiUN2dEgWolx8IdIlWszL4UA+NDjWJ0=; b=tsgOGAXsDrin+uawxPmOdtlQCx0WpFbxldDBxLivjtL8A0p6lZ68+n4oEOmChE7V9I bGOYRJa135BwBHFZnlUN0beVoA/qWNvrAUymIqFQHWxID8AP0tlsZX+H2nX60EHWt5Hm UQXlJuHFiNr334Bm+nahwXJaOkuWsY7/UG6qZCN0YhvZJsJpPprwGluOOBbYspWBhsNM PXBRcQM1mv2MqK3KHo+DNS3k3gTjYOgFioPaADvKUQk4AP5oyv9+KvQnOvSexdkaPnl4 IdrRLkFgzkzRmokak9FKjzCyUmE5iTwrW3xKFo4GevVKxplzh/p9LyMk2iWjKf22Orp0 YK4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713403718; x=1714008518; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=FYDQyLdFdWMdeiUN2dEgWolx8IdIlWszL4UA+NDjWJ0=; b=CQd485my55JaAmVE53HzXrIhvKNrRFdzS8B7AkHEloLSyUPBvbsYrmukM83TSsevwJ SrzUtut7//f6+Q85bSHLApHzBYsyHpdFvxXAY6v87SkmiuaeP+Do+vroIy4HZDMCU7Xf VZebeMyvYOSd0X1CBqjScLYN41TGm0bQSELF/bKc5EUhtGq6NGQJH5JG5l1NfVhadTmK OgQMalIJpAk5YVMkK4h6gPZ8I0IhNfaB57o2MXwAuxcnEdgz7yb3QFG8YoQ/7rZFFSIi GPqp06QmSChIZ4zn6BQTAyywlimxjimAnk+ZuZEvGsZxrz3jMuleIsPhvaUPtVgmfQCi zcKg== X-Forwarded-Encrypted: i=1; AJvYcCUoR6sY5uwrkmFGSoVRIZRWVD3Jn47Eng23sEpkv/I6Y9QldncbbwmdoTWK9LkTeLtoI/zEQHMOYecJw0upuQsxMNk= X-Gm-Message-State: AOJu0YxIRxonp/gKUiusp4TgtO+rFOJedzq9A4LfItQL9m6SzgYn6p75 q2V6E7UBMqCI4Jvvxf2ylQHOY5sSeA1//L+QtrBcBXztj7wKk4QyAmylW1UnHy+ADX2/gLyKWMS rLa86LS/ywf4cTM4L4g== X-Google-Smtp-Source: AGHT+IEqdMaTCeR+5N661ePS75QOvFa8jZaaiw9BcHvMUuJwBncrqCmIT7qsnybEQWIN2tFvBp2petyzPSOYJ3zy X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:29b4]) (user=yosryahmed job=sendgmr) by 2002:a63:e57:0:b0:5dc:20e1:c894 with SMTP id 23-20020a630e57000000b005dc20e1c894mr3697pgo.4.1713403718174; Wed, 17 Apr 2024 18:28:38 -0700 (PDT) Date: Thu, 18 Apr 2024 01:28:33 +0000 Mime-Version: 1.0 X-Mailer: git-send-email 2.44.0.683.g7961c838ac-goog Message-ID: <20240418012835.3360429-1-yosryahmed@google.com> Subject: [PATCH tip:x86/mm v3 1/3] x86/mm: Use IPIs to synchronize LAM enablement From: Yosry Ahmed To: x86@kernel.org, Thomas Gleixner , Ingo Molnar Cc: Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , "Kirill A. Shutemov" , Rick Edgecombe , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yosry Ahmed X-Rspamd-Queue-Id: 9361640005 X-Stat-Signature: b1cxc5ournp7ij8uxq4oawn9xduh7cop X-Rspam-User: X-Rspamd-Server: rspam07 X-HE-Tag: 1713403719-48131 X-HE-Meta: U2FsdGVkX19ine58QrxJxqbjAjc0sUYWZD1DuPmO223wfACQsWMoj+LDvYnLIk420CqQP3vGWmDRSOhZj1AyjcrO10e8rtLQ3TMymBXkrkCv0aLtpI3lWm2hOM951oT0Hk8UsKNI837JEOcH4NKM6phlEdfqjFBTh0Je+iJaTDALszrW87fvLE7J2Ld696SWezviYbLbTsEWpN2G6aCn7S/3pF9YtZmbvQWtsfr4+FWBQOJWypwnXxaRI+DL6o5cgu/8UCl6TK7DaRf9s72tZi/3djJQXDnM9cgqmMSxFWoMTLAVyL5jSR4XGsgHlIBKA5ov4zKY5c41a5c2EXuxY4EfGLem9ygwxDalFky9yAepFIeQJcfQtKQVB8yIN7JOXhV2YXdp87FKE8TcVocsgh7QmWYItZH333sHDv7qTCMc+8b2Y0R04xaksj1LqLZVv68ldSI/ycxE26zaFcAxDZo0iEMxJQ6hVo0t6mfXb2d5QkVG1XyQsnvaOCosDNYQTwUnJe6whzQXR7nNg52qTRUR28cDCStOie/4cevUcE2CZ3dA2wgtciXKP2RkrkiPUbtwAhO5nK1h3FyHvwnwSkbtVaXu7M19kzM9je1ici898+tXizMhK+kg9bibOc+McPxl8JgsdLH/0SwD3afg+1bmk2K6AaHhd39Z3dhyh7HG9D+0G9Dgs9AaMzCmYb86f3KqHsgAIEcYjpT+4a7o7pUgjNESr1vWKC0dFZLBxhQR5u5qUmgkhap/Jr8/h60Fc2wTGuGfY/OEs8dtoUP/3nUd9A3TppIqTIhOJbB5/mnL0VI0kGNXhLc5ANN97Qii5HyEIbTTJfrlo+sCRGU8Q/Bbwpvtm9IQJq0S/zDV6CyQKZT+UJLZV7GxsENl+lMf1vl2RywmvL8YPJK1qtBlD0B37+thZ/aiEkc6iq+jxBuAY9OXYMbrrnw7+0PQYlbtzpAc79x/UNu+G2UvmJO 7xekMoGK FuuUmo7bee8zdBKxJRK4j3zIHM9Sl/JlgWpy2pfTv021Rp9wOYTDzNCLp4dyT971GpgDSrgT6C4TwjXRZ5/6xJjjnOmrTk37sSjs/fsMWbSG1Oo93j46O8A6E/+Vd51hnxZUGGGK4fxO9dBeZaKiSK8YK1czNeNU0XbNLtoFEVdxVFOxJ1A35tPveFXKv6zQr2jG2Zg2IMZ/R2wA1W7zdvAwGnYlDj2nZk+tHItrk3wa6yLs9Ae2l4s3SPtdBwcPfoVmEb9QogDE1CEO6gH8NxlJ+nr5w+8czs8I58hsdvLDnb8MqY+TUKmVXArhiVuqFD3Eot5BBTimNakg9V9T0P7lFWALiXFGw3xaQEPOnYyAUlb6Q2Ux4XQj3PVLEgdW7IuYYtdRsMFTvFymssJ0VpehpIurRIVaHylDIf0gJ2/ynTgmkzFGzVjotYq/qFyH4PZxWv9hnyhNZgHyYq4o1Q+ZEoKuonYWUxCxUGeapKI48BWpOedWWMSE3fBmeKCAEmQ3AjyfFkhU9WfL/CScv8/fyRIK2aKCaqnnohdnX55HIQUWW5HQzQIQUzFcv3Nh+XekGAO665wk4qz9eetTG63RCz+l2RKAkoitBucO4CXvNB7TaxubclVwnOBwl4N8nop+axvbhXwc6MutNlCHdhLsGZhTwOLMSrx/nyjfxAs+hqTfX2pyJzBVkWegY+vR4V20nQsN5fT63TMB0PRa22w28Qg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: LAM can only be enabled when a process is single-threaded. But _kernel_ threads can temporarily use a single-threaded process's mm. If LAM is enabled by a userspace process while a kthread is using its mm, the kthread will not observe LAM enablement (i.e. LAM will be disabled in CR3). This could be fine for the kthread itself, as LAM only affects userspace addresses. However, if the kthread context switches to a thread in the same userspace process, CR3 may or may not be updated because the mm_struct doesn't change (based on pending TLB flushes). If CR3 is not updated, the userspace thread will run incorrectly with LAM disabled, which may cause page faults when using tagged addresses. Example scenario: CPU 1 CPU 2 /* kthread */ kthread_use_mm() /* user thread */ prctl_enable_tagged_addr() /* LAM enabled on CPU 2 */ /* LAM disabled on CPU 1 */ context_switch() /* to CPU 1 */ /* Switching to user thread */ switch_mm_irqs_off() /* CR3 not updated */ /* LAM is still disabled on CPU 1 */ Synchronize LAM enablement by sending an IPI from prctl_enable_tagged_addr() to all CPUs running with the mm_struct to enable LAM. This makes sure LAM is enabled on CPU 1 in the above scenario before prctl_enable_tagged_addr() returns and userspace starts using tagged addresses, and before it's possible to run the userspace process on CPU 1. In switch_mm_irqs_off(), move reading the LAM mask until after mm_cpumask() is updated. This ensures that if an outdated LAM mask is written to CR3, an IPI is received to update it right after IRQs are re-enabled. Fixes: 82721d8b25d7 ("x86/mm: Handle LAM on context switch") Suggested-by: Andy Lutomirski Signed-off-by: Yosry Ahmed Reviewed-by: Kirill A. Shutemov --- v2 -> v3: - Rebased on top of the latest tip:x86/mm after v6.9-rc3. - Collected R-b on patch 2 (thanks!). --- arch/x86/kernel/process_64.c | 13 +++++++++++-- arch/x86/mm/tlb.c | 7 +++---- 2 files changed, 14 insertions(+), 6 deletions(-) diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index 7062b84dd467d..c27798f23ef82 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -798,6 +798,16 @@ static long prctl_map_vdso(const struct vdso_image *image, unsigned long addr) #define LAM_U57_BITS 6 +static void enable_lam_func(void *__mm) +{ + struct mm_struct *mm = __mm; + + if (this_cpu_read(cpu_tlbstate.loaded_mm) == mm) { + write_cr3(__read_cr3() | mm->context.lam_cr3_mask); + set_tlbstate_lam_mode(mm); + } +} + static int prctl_enable_tagged_addr(struct mm_struct *mm, unsigned long nr_bits) { if (!cpu_feature_enabled(X86_FEATURE_LAM)) @@ -830,8 +840,7 @@ static int prctl_enable_tagged_addr(struct mm_struct *mm, unsigned long nr_bits) return -EINVAL; } - write_cr3(__read_cr3() | mm->context.lam_cr3_mask); - set_tlbstate_lam_mode(mm); + on_each_cpu_mask(mm_cpumask(mm), enable_lam_func, mm, true); set_bit(MM_CONTEXT_LOCK_LAM, &mm->context.flags); mmap_write_unlock(mm); diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 44ac64f3a047c..a041d2ecd8380 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -503,9 +503,9 @@ void switch_mm_irqs_off(struct mm_struct *unused, struct mm_struct *next, { struct mm_struct *prev = this_cpu_read(cpu_tlbstate.loaded_mm); u16 prev_asid = this_cpu_read(cpu_tlbstate.loaded_mm_asid); - unsigned long new_lam = mm_lam_cr3_mask(next); bool was_lazy = this_cpu_read(cpu_tlbstate_shared.is_lazy); unsigned cpu = smp_processor_id(); + unsigned long new_lam; u64 next_tlb_gen; bool need_flush; u16 new_asid; @@ -619,9 +619,7 @@ void switch_mm_irqs_off(struct mm_struct *unused, struct mm_struct *next, cpumask_clear_cpu(cpu, mm_cpumask(prev)); } - /* - * Start remote flushes and then read tlb_gen. - */ + /* Start receiving IPIs and then read tlb_gen (and LAM below) */ if (next != &init_mm) cpumask_set_cpu(cpu, mm_cpumask(next)); next_tlb_gen = atomic64_read(&next->context.tlb_gen); @@ -633,6 +631,7 @@ void switch_mm_irqs_off(struct mm_struct *unused, struct mm_struct *next, barrier(); } + new_lam = mm_lam_cr3_mask(next); set_tlbstate_lam_mode(next); if (need_flush) { this_cpu_write(cpu_tlbstate.ctxs[new_asid].ctx_id, next->context.ctx_id); From patchwork Thu Apr 18 01:28:34 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13634055 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56076C04FFF for ; Thu, 18 Apr 2024 01:28:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 97DEB6B0082; Wed, 17 Apr 2024 21:28:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 890076B0083; Wed, 17 Apr 2024 21:28:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6E3656B0085; Wed, 17 Apr 2024 21:28:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 51A236B0082 for ; Wed, 17 Apr 2024 21:28:43 -0400 (EDT) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id F0EA7141011 for ; Thu, 18 Apr 2024 01:28:42 +0000 (UTC) X-FDA: 82020918084.10.8CAD1DF Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf11.hostedemail.com (Postfix) with ESMTP id 38F3640007 for ; Thu, 18 Apr 2024 01:28:41 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=xhm8U7x8; spf=pass (imf11.hostedemail.com: domain of 3SHcgZgoKCO8pfjipRYdVUXffXcV.TfdcZelo-ddbmRTb.fiX@flex--yosryahmed.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3SHcgZgoKCO8pfjipRYdVUXffXcV.TfdcZelo-ddbmRTb.fiX@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1713403721; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=z8iDG18ujjZI6cYwd6gK6QfWKPv05SJ9UjlFR07I8FE=; b=fSAAvZ66Mka0QFS8Az8TpCYAcn2qKGuHCu6qh2+8M/qttsrLWcaI0R099Ij+GyKPBbrYQ/ LGv60M4zNShWbKW80rmGp2tjnlkTMsSj+o5WffnOqEXmww77LY+cprG7O5Y8zSMFkgLvHQ uHmmweCHJgcKirX8VfJr/goCChC7Gis= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=xhm8U7x8; spf=pass (imf11.hostedemail.com: domain of 3SHcgZgoKCO8pfjipRYdVUXffXcV.TfdcZelo-ddbmRTb.fiX@flex--yosryahmed.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3SHcgZgoKCO8pfjipRYdVUXffXcV.TfdcZelo-ddbmRTb.fiX@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1713403721; a=rsa-sha256; cv=none; b=hrFNIYsBnVvmZVM+QeF3/rxa0qfTgxJ3xkF33pjhwfL1Ostdk1HMA9NbfXa5hxW1mFT4NV L+smTM+K4pkFiOhd+FXVB3KmXadl3g8C9bVCBOvgqmhnUJnByyxv8XNoKLbJIPzQNV+Z/p xbIPqH7cCjr9bT7nlqcJtYryGHpdBUE= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-de4691a0918so155312276.0 for ; Wed, 17 Apr 2024 18:28:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1713403720; x=1714008520; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=z8iDG18ujjZI6cYwd6gK6QfWKPv05SJ9UjlFR07I8FE=; b=xhm8U7x8MYPsHlrigtXR/Y3GZpO7+ClTKwQHvh/d/bJlyJ5XZgIVOYgmV9qtZi7xRl 7ypXdWPbIuKHF+kzd48k64vsk2FJ3QCc5U7WSJ9wpsOtkOfdExIv9bKT/JANBHQeIoli WSRj8PtQiSwv5j9Xvs13gSYfyQk+aqf0LjZQAH8HIegCpcLIAaFcuQ/3T3TS5kMqo9lB X68bIxXRBFaHtBiYRUyTl4Atd6HZATzSKd3cOlQn4qj+y1IUWJ7pyt1ducEMmQtzPcy9 4yz4aDOhL3WBPBuwq7bpWIjgrOQXyYoYOvATTiKGXJayjUtNUmREBINoNDmvQUOL1sig IoQA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713403720; x=1714008520; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=z8iDG18ujjZI6cYwd6gK6QfWKPv05SJ9UjlFR07I8FE=; b=auelHX7LLzN7fjRdORwirx548Qx9Jrn9PkxmiJfXX1c4TFF3FNYo00Sy0h9r7X/cwt Y5szmP1q3m65wH+p5Qkyfy9ix+Pce+bp4tWQ9ppYiOG+lNFzJz81ekfdPJYPuPg//81U mW7cs6fiqeiBZVFU1Jp3ZL8Yvry0AkKtjXiDlEhx8ggxH5ckC7LEylDHXJw9W4WKr7XS UXgI0+JiYFnT/83tU+uw1cvVY2Qo9YK0P0oIXZwO7mPuG0U+Be7CkS9BR0ZWZ/D7EcsX 6z2Uk+thunzfKp3VNckJcfZ374br6XbalKMrMfhAV3s/3WZKnqUB4jEeKMgka1SAU8ax uQ+A== X-Forwarded-Encrypted: i=1; AJvYcCXqzIBW4TC38mAuy+laopN0DfiPNx35H7ER5LPQgG2kV+ELJTxTwNotldFzzM6pQ/xGmKYmgGzEp+Xh6maP99xuOoo= X-Gm-Message-State: AOJu0YwEsmn1S1WX5FQ5wAUydu3L4BlsMu/RGUIvFhPBsEctSLRzmFxY mpDa8ooS6JklGN1CZ9gVtoeL2pFErBmcP4ethueZlVWxHXIx3LohApm9qE4dRfkLUcUQayCW8mC X9tnOR2HXdOSI2QbJ+g== X-Google-Smtp-Source: AGHT+IHowabqTIUQAhwGR6H3lDw/MbSxyXp0scJMwSRdUOmCkDeIF9VXDDt3wLoCkgZ1GCn8fBoG90voSyasNhI9 X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:29b4]) (user=yosryahmed job=sendgmr) by 2002:a05:6902:2b86:b0:dbe:d0a9:2be3 with SMTP id fj6-20020a0569022b8600b00dbed0a92be3mr255747ybb.3.1713403720188; Wed, 17 Apr 2024 18:28:40 -0700 (PDT) Date: Thu, 18 Apr 2024 01:28:34 +0000 In-Reply-To: <20240418012835.3360429-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20240418012835.3360429-1-yosryahmed@google.com> X-Mailer: git-send-email 2.44.0.683.g7961c838ac-goog Message-ID: <20240418012835.3360429-2-yosryahmed@google.com> Subject: [PATCH tip:x86/mm v3 2/3] x86/mm: Fix LAM inconsistency during context switch From: Yosry Ahmed To: x86@kernel.org, Thomas Gleixner , Ingo Molnar Cc: Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , "Kirill A. Shutemov" , Rick Edgecombe , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yosry Ahmed X-Rspamd-Queue-Id: 38F3640007 X-Stat-Signature: 9m7taig7eo5yz1py4gggn51pxtmb4gkj X-Rspam-User: X-Rspamd-Server: rspam07 X-HE-Tag: 1713403721-647648 X-HE-Meta: U2FsdGVkX18oppGb6DBWzgaYQqBM+IPVjFK3g313Wbm/LoIhhHHSQGvJ/qUZiyNKUVOew0+qtZdvFKxWLIV/LmIcifJ/D0TwiFsUjX93QcyNfVu/ifFEn46UlHD0L4g2Q3GpL2+wHmh0V4/mBg7HFX5NG+zU/UjcsOkl2G5LardfOvUQQEwqOZNu/j9yXykn5XiZF1jWnQrB8tesiKhvE8pA3crb58ONaE3i6WXyKlR3oV9fw15V3fTFc3lFulSDEjiMaL16WLFTRMtp0kGI2IltKRAEusWco2dNx06EaJHnFIWPl6HAa8aChqfbdP3l1ri/V9HJK4qcGXLsJ4te987c3cXMOst3PwW/UMqcb7RvkqenoVJjBv+fTnlPmKthFPPhhgb3y9AA2pzMIadXP8oIFGssr7/izpwHbpyAWBaHWU8u30F7jSs8okN35RKOJ/NOqosB/xI36EQMtOjKsxg0nUsxZCzMz90Z29K/GYjvwFPvc3mMjp6reynK6LdT51SqBQvzl4BP0OrhuVVM4Wm1wo8UgD4VqnQUN3bRP3oO479ImKlDr94pd7AI+mif+vsW2c6PHweXLB91KI9razxQIdyt/EGzAwILFVw95+ICR3uPj6G9dHy3PJhnWPpfRGEIfgAc9gYN8NikRISEDItwLDKbUKnK0byv9MQ7YJtzrm9Dn5nGsfaNYiXrwxOPRo7UCm/HSxbt8jT1Sgfa83jq0SOp0bEdnDLTzhmLOCy1sNl1ysmLopHTbrOP7eRg2S1MCz7JUbJN0ZAo3vGfQogRDJs9ag7KE2tZAUo4qSFE+t9XgyPt7Aqn6VZekq6XX36xC3nyUHLay2dprg1GQq6UPsMRikP5JxoF6G96iL/Z4Xc+9T/nsyRx4lT0j17cmeHa2rK7lVpQhigcfMhQ3cw3d5IV9V8oNaCA85hNwZScSKwZHut33fAqJ6M8Po+Sj+tUYLm5sngj5LstweD BiJ3641f xZJGH5mEB5hZ+ikPrQ+u6NOVKuSFY3FQSqtt6gQnmrbQdyCa3MS7sTdMUL7UlZijC9nJ+4CvuDU9n4bbC5T409zkxQlXWv1+LJVpqIDL80LAjDDJU/7Rx97gtiMK9BiToDLPQ6TA6BwbuoraOUsleujvpQoym0+42hjhvXADCEI/KfFnR+NU3cQDvqkI64P2we2FiVrNXLAEq2vi4saF7DDDX1gFfmjMEYqKT7aNjLIyXcGzmpseYSM4+/UNkLbguZngdq3NX4YaaXJKC+1IcijeBmyyipYPnSC2crmEQJ6SVdekZFudlvTOXi9pnBR26bcXKbbZzBo59tkqXyA/HAUabSO6QB+NrvvkRilKPVsJtSnLgnvXV4YObRRhcsRnkLVG3UDk8qcK5ZQZJVydp242/M5VgltTk5tFCnGBfIPunPNSDhwhjNbr4SVdbdrb5c26HhpkGJU7q5i8Ad3fzv9sJzMlwHkFQs0qeUvjAI+iPdsaCrAadpODmTiuuCQhuqXagL2Ik5pdIl4y4PxHcVAthuDw9GVvKxnFFyI+tbXP7fR+IL48FrJ1OApyeIktIPUzHHk8Oo8sV23VCiYEArBjL7OJKLWhMmnHrsk4i5iw/pFPCJWflCIWy7vIddLnL5Xj8t15QbXkJHsM/SWRJRVaQCLA0lQ8hqPSn6WgNvFjjdjI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: LAM can only be enabled when a process is single-threaded. But _kernel_ threads can temporarily use a single-threaded process's mm. That means that a context-switching kernel thread can race and observe the mm's LAM metadata (mm->context.lam_cr3_mask) change. The context switch code does two logical things with that metadata: populate CR3 and populate 'cpu_tlbstate.lam'. If it hits this race, 'cpu_tlbstate.lam' and CR3 can end up out of sync. This de-synchronization is currently harmless. But it is confusing and might lead to warnings or real bugs. Update set_tlbstate_lam_mode() to take in the LAM mask and untag mask instead of an mm_struct pointer, and while we are at it, rename it to cpu_tlbstate_update_lam(). This should also make it clearer that we are updating cpu_tlbstate. In switch_mm_irqs_off(), read the LAM mask once and use it for both the cpu_tlbstate update and the CR3 update. Signed-off-by: Yosry Ahmed Reviewed-by: Kirill A. Shutemov --- arch/x86/include/asm/mmu_context.h | 8 +++++++- arch/x86/include/asm/tlbflush.h | 9 ++++----- arch/x86/kernel/process_64.c | 6 ++++-- arch/x86/mm/tlb.c | 8 +++++--- 4 files changed, 20 insertions(+), 11 deletions(-) diff --git a/arch/x86/include/asm/mmu_context.h b/arch/x86/include/asm/mmu_context.h index 8dac45a2c7fcf..19091ebb86338 100644 --- a/arch/x86/include/asm/mmu_context.h +++ b/arch/x86/include/asm/mmu_context.h @@ -88,7 +88,13 @@ static inline void switch_ldt(struct mm_struct *prev, struct mm_struct *next) #ifdef CONFIG_ADDRESS_MASKING static inline unsigned long mm_lam_cr3_mask(struct mm_struct *mm) { - return mm->context.lam_cr3_mask; + /* + * When switch_mm_irqs_off() is called for a kthread, it may race with + * LAM enablement. switch_mm_irqs_off() uses the LAM mask to do two + * things: populate CR3 and populate 'cpu_tlbstate.lam'. Make sure it + * reads a single value for both. + */ + return READ_ONCE(mm->context.lam_cr3_mask); } static inline void dup_lam(struct mm_struct *oldmm, struct mm_struct *mm) diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 25726893c6f4d..69e79fff41b80 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -399,11 +399,10 @@ static inline u64 tlbstate_lam_cr3_mask(void) return lam << X86_CR3_LAM_U57_BIT; } -static inline void set_tlbstate_lam_mode(struct mm_struct *mm) +static inline void cpu_tlbstate_update_lam(unsigned long lam, u64 untag_mask) { - this_cpu_write(cpu_tlbstate.lam, - mm->context.lam_cr3_mask >> X86_CR3_LAM_U57_BIT); - this_cpu_write(tlbstate_untag_mask, mm->context.untag_mask); + this_cpu_write(cpu_tlbstate.lam, lam >> X86_CR3_LAM_U57_BIT); + this_cpu_write(tlbstate_untag_mask, untag_mask); } #else @@ -413,7 +412,7 @@ static inline u64 tlbstate_lam_cr3_mask(void) return 0; } -static inline void set_tlbstate_lam_mode(struct mm_struct *mm) +static inline void cpu_tlbstate_update_lam(unsigned long lam, u64 untag_mask) { } #endif diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index c27798f23ef82..cda2918f88f99 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -801,10 +801,12 @@ static long prctl_map_vdso(const struct vdso_image *image, unsigned long addr) static void enable_lam_func(void *__mm) { struct mm_struct *mm = __mm; + unsigned long lam; if (this_cpu_read(cpu_tlbstate.loaded_mm) == mm) { - write_cr3(__read_cr3() | mm->context.lam_cr3_mask); - set_tlbstate_lam_mode(mm); + lam = mm_lam_cr3_mask(mm); + write_cr3(__read_cr3() | lam); + cpu_tlbstate_update_lam(lam, mm_untag_mask(mm)); } } diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index a041d2ecd8380..1fe9ba33c5805 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -11,6 +11,7 @@ #include #include #include +#include #include #include @@ -632,7 +633,6 @@ void switch_mm_irqs_off(struct mm_struct *unused, struct mm_struct *next, } new_lam = mm_lam_cr3_mask(next); - set_tlbstate_lam_mode(next); if (need_flush) { this_cpu_write(cpu_tlbstate.ctxs[new_asid].ctx_id, next->context.ctx_id); this_cpu_write(cpu_tlbstate.ctxs[new_asid].tlb_gen, next_tlb_gen); @@ -651,6 +651,7 @@ void switch_mm_irqs_off(struct mm_struct *unused, struct mm_struct *next, this_cpu_write(cpu_tlbstate.loaded_mm, next); this_cpu_write(cpu_tlbstate.loaded_mm_asid, new_asid); + cpu_tlbstate_update_lam(new_lam, mm_untag_mask(next)); if (next != prev) { cr4_update_pce_mm(next); @@ -697,6 +698,7 @@ void initialize_tlbstate_and_flush(void) int i; struct mm_struct *mm = this_cpu_read(cpu_tlbstate.loaded_mm); u64 tlb_gen = atomic64_read(&init_mm.context.tlb_gen); + unsigned long lam = mm_lam_cr3_mask(mm); unsigned long cr3 = __read_cr3(); /* Assert that CR3 already references the right mm. */ @@ -704,7 +706,7 @@ void initialize_tlbstate_and_flush(void) /* LAM expected to be disabled */ WARN_ON(cr3 & (X86_CR3_LAM_U48 | X86_CR3_LAM_U57)); - WARN_ON(mm_lam_cr3_mask(mm)); + WARN_ON(lam); /* * Assert that CR4.PCIDE is set if needed. (CR4.PCIDE initialization @@ -723,7 +725,7 @@ void initialize_tlbstate_and_flush(void) this_cpu_write(cpu_tlbstate.next_asid, 1); this_cpu_write(cpu_tlbstate.ctxs[0].ctx_id, mm->context.ctx_id); this_cpu_write(cpu_tlbstate.ctxs[0].tlb_gen, tlb_gen); - set_tlbstate_lam_mode(mm); + cpu_tlbstate_update_lam(lam, mm_untag_mask(mm)); for (i = 1; i < TLB_NR_DYN_ASIDS; i++) this_cpu_write(cpu_tlbstate.ctxs[i].ctx_id, 0); From patchwork Thu Apr 18 01:28:35 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13634056 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A4EFFC4345F for ; Thu, 18 Apr 2024 01:28:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2C6AC6B0085; Wed, 17 Apr 2024 21:28:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 253516B0088; Wed, 17 Apr 2024 21:28:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 02BC66B0087; Wed, 17 Apr 2024 21:28:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id D7A306B0083 for ; Wed, 17 Apr 2024 21:28:44 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 86D4D4107A for ; Thu, 18 Apr 2024 01:28:44 +0000 (UTC) X-FDA: 82020918168.29.57AFE56 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf27.hostedemail.com (Postfix) with ESMTP id CE1CC40011 for ; Thu, 18 Apr 2024 01:28:42 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="Zigxbb/W"; spf=pass (imf27.hostedemail.com: domain of 3SXcgZgoKCPAqgkjqSZeWVYggYdW.Ugedafmp-eecnSUc.gjY@flex--yosryahmed.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3SXcgZgoKCPAqgkjqSZeWVYggYdW.Ugedafmp-eecnSUc.gjY@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1713403722; a=rsa-sha256; cv=none; b=sUNrp2N7sLX2N8y04tmbQkTEBHDaHvbIcso1bOt1WcPMw7rLhOSH97lFOmchIz8UVYg2QV r1bSuDAu9UkqCDF2QPtXif0qpsTYINCGxH5U/WKsCzGQ4XOtjFjb8QQhMC0mWTh62l1Lgb OVj53mFYk2i454VEM1nw3eT0ZXfmw4A= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b="Zigxbb/W"; spf=pass (imf27.hostedemail.com: domain of 3SXcgZgoKCPAqgkjqSZeWVYggYdW.Ugedafmp-eecnSUc.gjY@flex--yosryahmed.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3SXcgZgoKCPAqgkjqSZeWVYggYdW.Ugedafmp-eecnSUc.gjY@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1713403722; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=1IocH4dK3cU7wr7fT/1eXbTtilS5jPEYt4JFcTK0tuc=; b=yjOGKXZ2bPZjObAgurRRuy9MJOi30P7yoiSEMqBGX2LzUKn75IUhuvOronzttwI2PxVi8/ ZzbGwt7xANpLlv4jcOuuMcJ4sZ3iB86CAWNzlchaFYi4Mq1fmZMS9dUVA3mY52+qbUoSBk 1gzm4lkaoBTGBtL9vCFl5S+JI6zu7ho= Received: by mail-yb1-f201.google.com with SMTP id 3f1490d57ef6-dcc58cddb50so791784276.0 for ; Wed, 17 Apr 2024 18:28:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1713403722; x=1714008522; darn=kvack.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=1IocH4dK3cU7wr7fT/1eXbTtilS5jPEYt4JFcTK0tuc=; b=Zigxbb/W3398FPOPVWdMa9IBTbb3Zp4PE8yviklaM/CVLNxkxASy1zbtImsq9sBNF2 niWmlR5lfEUTT+XvrNbuJP5IRASdofEsjSEqmcIU5j11DeQfpU7WnurKVF7dQvSSHPqw 73lLhSpOs8LUDVq+NpEwiNwNgnG/wiTUQNsH3wtS2rFyUAio9Ah5P6YMtOZBfo8Gavd/ nXZ1otRjVOTjRHXpNDuBv7iYkXcBUpnRyadq+r0MdtIeZYBato8Z1AiOlfNk04/+1DHp ZL7in4ZrQylF6kONEdLspO4w9P5hJRORgMOSQe1zbj6S9eSoJHo0/FZ7uK8UOevSnsJU zknw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713403722; x=1714008522; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=1IocH4dK3cU7wr7fT/1eXbTtilS5jPEYt4JFcTK0tuc=; b=u2kBat8AgpsVAMlqUKMOjJGYlYg1PyGUO4enOEBz0RXKesuJUHD7psLu1+tbR96C6k CbzqjKqisoE6Z/yAfnCP7UBgxwt1SB49ppqnXhbSbgx7RrOGihhKIRLhKwfF2Ov8nF7J CWV0sC7Twtb9LVlu3RpJ8BlSAJRxLRNOlHn0h5ZtN4aKFy76RBX3yaRATGI4iJ1WtF0V nx7+pudxfzrJxTvgG76QEq0PNkIzYV7x2TbbHUXZ+ua8noQviL22wWcSHxp1u0t/QA4n EErcxZA40eKCgTZBXVuLZ0NG577gtiNZ0f2YEEujzaWgWldDxwTZvcDSfv2chCdjlF5d QXPg== X-Forwarded-Encrypted: i=1; AJvYcCVgYp3Vf55FhltCo3gqMjJ8eEqS3GNcseoDAbegY55aN+enf7oBQnA78QA6Dsjffl6As/i/u75DMpDR+4lIhYwFSkI= X-Gm-Message-State: AOJu0YwlcDX5F27Q3VL5YdyIzZspBqDd9CzL3KSzI1TVU+LQ0d+j6opw vV4L9rOOa03cFIR+aZeUlfJcqNIJXy/Anv1kSOS2OGxWDKhIwysHB9gvGLGM3bkwJ9LvTb0jW37 F0YQ7bKvjqqjQ/DjAkw== X-Google-Smtp-Source: AGHT+IHSfSOAwqPT5+RVg+FkCynH2REwXQqip28l12UA6TCmdrVfJ3C58BqU4WK+J+oNidSiUh15Dp0ZWHsFTsRk X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:29b4]) (user=yosryahmed job=sendgmr) by 2002:a05:6902:704:b0:dc6:e1ed:bd1a with SMTP id k4-20020a056902070400b00dc6e1edbd1amr320570ybt.2.1713403721892; Wed, 17 Apr 2024 18:28:41 -0700 (PDT) Date: Thu, 18 Apr 2024 01:28:35 +0000 In-Reply-To: <20240418012835.3360429-1-yosryahmed@google.com> Mime-Version: 1.0 References: <20240418012835.3360429-1-yosryahmed@google.com> X-Mailer: git-send-email 2.44.0.683.g7961c838ac-goog Message-ID: <20240418012835.3360429-3-yosryahmed@google.com> Subject: [PATCH tip:x86/mm v3 3/3] x86/mm: Cleanup prctl_enable_tagged_addr() nr_bits error checking From: Yosry Ahmed To: x86@kernel.org, Thomas Gleixner , Ingo Molnar Cc: Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , "Kirill A. Shutemov" , Rick Edgecombe , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yosry Ahmed X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: CE1CC40011 X-Stat-Signature: k4zkdqgwwwgghdsgmg55n3ixypuafpba X-Rspam-User: X-HE-Tag: 1713403722-229969 X-HE-Meta: U2FsdGVkX18siPtLmSl37cs5apkbPuNAaD+8Y4vw9mWS8eISI8XNS8ydMhOBxfVhOGdN6NfaAXz5j+YXVYFyJo1G+W0JJBH1xnrG9iKXX9qfGeK7mE7ldqQgdEt5atedGqKMKWLGZtQC0YVvjmMBwBMnJ2HOWU46iqaCI9vHpRPZV9vCa2KP9+y4cqQfvJDG/vrqyWfp7CHBf4Qoj/LVOIAqYUbf9FbStzSPoR1Km9bv50JJzIX+zdW/5LnSn1HQHgimpINWkauMCj0Tc/FvjPD7AB4T2e/00szETGPtXxqe8z6kx7U6KB81koAo/GiIVdG+iR3Mdk9c4dXt05KPAuXFU+vHQgf3ms0FKGm9zxlW6OEVZZYyCuIMrGc5OUX5MgzzkiAMimfQmLsmkdY8eKiTxrD3+1oCPoX8vqfJPVybj1att+GDh3aazlHbISPmutS84L8khr87OsQZoIsqspL64unsRJfZspgAJ59uOL82KvTXQC94ACNvVCtCTI/zF24TWbjdKXEdRKYEL4pUXMt9+uKqu8jxSxfuJBV0Z0HbZlAQcWnjEeef/sCkDuBcnchzJ00pCWCiDuuX9gWDgx2SIVIaFwNqUO8FW3U5a6yGQoUeLMDV1zAw5UHrgvAiCBVaq/fE3qKgE4FtSAE5iJQB/xtuzx4N/mpwVRii/euyNRXUJdgdJYJ/RVdDuRekeUxRLH4Q1MEFDCa1il7r09tkHCMtoXjbz5U1BpVb2t0VqL5OTbefUrcd4KR6m7jWa9gq8sj0uV6J+/nwDNm+EDqB1O6Ys1hdaxZ/GeC6nj+2qsTtsRzgQbHpNP6trz9/JmMrVSiu5gkFrUfxm0IImX243UwzJq6GiTEZOajDSviDIs5yV8rAbG4sHLNT9iU0w3p6/rfzmoGtl9svVOYpW8H7wqnStiVusxrV2smy6OjHiZMEibycwIlmw3q9nTBMnRn0/ketwgbSXcueKGE BTTjbdaE kwRRaSctQvk4rtmI7W4z3b+cYwpW9Ka+0q/YrAd6d4L7e6VUaDbNyuspZ4JX8umqPN5/PjGHjreW+ZweT/M/O2fLK1vtXMrFwxktQwb410wGgF2GvnuJ3OYIA/X5mMHHChywCPVBMJ4ejLAGvwwSaweUQ4obGuxXCoWVcDuoA6O1sONdxFZMUaodgiBigAFAJJpUnN7q40GrxzaDh5EeuSjOt0ApYyTQN6tKzR1gMUjBFIwor+74dLFQDa2b5RVPFjrEaeUAri3kXMF5qBxYLzvQJoVZN5EH4fUidIHFcMNPKp589v/3+lGO34DjZJynO+BeohY295GvixBBPHQTXyZS01XBG8XdissLRLfnY6z6HxC10tn+CVHvOl69wknpM/VZ89AEYtga9lFB5ed9TxsITmQ1u8C74jYs0b62CKVsNsm+y4eMqElE/sCTj8is7qUfAly+g8LkgirQ0odcAh1nuZSUAfLNYLsfySzE1otZdvwC/K8YqQAqstuHmE+hkLNyXbaNk4Zi6BBHVyxNvRM+fnTo40cgfh6uacbF37Kwfahg1rZ5gR+a4dXbB6AZ6IV9s+Tg3teAeXfT+SNpgoOA/rMGeLQAAya2+AkJaASBa864s2wkMLS24jQ1dkZCOjf1o1ajSD1HO4OkE6LMWQgmVysCEw3yA8i08hXN653B0KMSOh8vF52H5gQON58sbNzw6EUJW3h12DNS02Rmj8zTc1w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: There are two separate checks in prctl_enable_tagged_addr() that nr_bits is in the correct range. The checks are arranged such the correct case is sandwiched between both error cases, which do exactly the same thing. Simplify the if condition and pull the correct case outside with the rest of the success code path. Signed-off-by: Yosry Ahmed Reviewed-by: Kirill A. Shutemov --- arch/x86/kernel/process_64.c | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-) diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index cda2918f88f99..baded5b4907c9 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -831,17 +831,13 @@ static int prctl_enable_tagged_addr(struct mm_struct *mm, unsigned long nr_bits) return -EBUSY; } - if (!nr_bits) { - mmap_write_unlock(mm); - return -EINVAL; - } else if (nr_bits <= LAM_U57_BITS) { - mm->context.lam_cr3_mask = X86_CR3_LAM_U57; - mm->context.untag_mask = ~GENMASK(62, 57); - } else { + if (!nr_bits || nr_bits > LAM_U57_BITS) { mmap_write_unlock(mm); return -EINVAL; } + mm->context.lam_cr3_mask = X86_CR3_LAM_U57; + mm->context.untag_mask = ~GENMASK(62, 57); on_each_cpu_mask(mm_cpumask(mm), enable_lam_func, mm, true); set_bit(MM_CONTEXT_LOCK_LAM, &mm->context.flags);