From patchwork Thu Apr 18 01:28:33 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yosry Ahmed X-Patchwork-Id: 13634054 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 05EC1C4345F for ; Thu, 18 Apr 2024 01:28:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E0F936B007B; Wed, 17 Apr 2024 21:28:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DBFAB6B0082; Wed, 17 Apr 2024 21:28:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CAE7F6B0083; Wed, 17 Apr 2024 21:28:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id AB4A36B007B for ; Wed, 17 Apr 2024 21:28:41 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 55ABA12104D for ; Thu, 18 Apr 2024 01:28:41 +0000 (UTC) X-FDA: 82020918042.25.5F4210F Received: from mail-pg1-f202.google.com (mail-pg1-f202.google.com [209.85.215.202]) by imf17.hostedemail.com (Postfix) with ESMTP id 9361640005 for ; Thu, 18 Apr 2024 01:28:39 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=tsgOGAXs; spf=pass (imf17.hostedemail.com: domain of 3RncgZgoKCO0ndhgnPWbTSVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--yosryahmed.bounces.google.com designates 209.85.215.202 as permitted sender) smtp.mailfrom=3RncgZgoKCO0ndhgnPWbTSVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1713403719; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=FYDQyLdFdWMdeiUN2dEgWolx8IdIlWszL4UA+NDjWJ0=; b=WdfrdYHYjaxrCp0N/emb+xBPA4gweS6iGBjDXC/g90YHUPapPMbqbaSqmRpvjcH0UzwS0K Onzn2GYVyf2VsTsHPShKmxBNh6mdkczxgjJrhug2rtGqX3ADqES/2tOePp+JgulyHyAzJm zntSXUKNE3qe7hznVrn+LagySx8AyUM= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=tsgOGAXs; spf=pass (imf17.hostedemail.com: domain of 3RncgZgoKCO0ndhgnPWbTSVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--yosryahmed.bounces.google.com designates 209.85.215.202 as permitted sender) smtp.mailfrom=3RncgZgoKCO0ndhgnPWbTSVddVaT.RdbaXcjm-bbZkPRZ.dgV@flex--yosryahmed.bounces.google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1713403719; a=rsa-sha256; cv=none; b=BUxmnSdoN2BDQwOHOZ/4UZHh3cXJc6EvFS99i+u2BYEpgq291Nw3jy9PbqNiH0WA8PooxR Kq78UQqKXc8UxEoMZqfOfDqe7mIPASOHX1i7Z8bLrDUqK5la/ZEm9jpBvNiM0SfxYzw8a2 i3grBk/VQHxQDawDLFr4VUCrmGXWelU= Received: by mail-pg1-f202.google.com with SMTP id 41be03b00d2f7-5d8dd488e09so372847a12.2 for ; Wed, 17 Apr 2024 18:28:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1713403718; x=1714008518; darn=kvack.org; h=cc:to:from:subject:message-id:mime-version:date:from:to:cc:subject :date:message-id:reply-to; bh=FYDQyLdFdWMdeiUN2dEgWolx8IdIlWszL4UA+NDjWJ0=; b=tsgOGAXsDrin+uawxPmOdtlQCx0WpFbxldDBxLivjtL8A0p6lZ68+n4oEOmChE7V9I bGOYRJa135BwBHFZnlUN0beVoA/qWNvrAUymIqFQHWxID8AP0tlsZX+H2nX60EHWt5Hm UQXlJuHFiNr334Bm+nahwXJaOkuWsY7/UG6qZCN0YhvZJsJpPprwGluOOBbYspWBhsNM PXBRcQM1mv2MqK3KHo+DNS3k3gTjYOgFioPaADvKUQk4AP5oyv9+KvQnOvSexdkaPnl4 IdrRLkFgzkzRmokak9FKjzCyUmE5iTwrW3xKFo4GevVKxplzh/p9LyMk2iWjKf22Orp0 YK4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1713403718; x=1714008518; h=cc:to:from:subject:message-id:mime-version:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=FYDQyLdFdWMdeiUN2dEgWolx8IdIlWszL4UA+NDjWJ0=; b=CQd485my55JaAmVE53HzXrIhvKNrRFdzS8B7AkHEloLSyUPBvbsYrmukM83TSsevwJ SrzUtut7//f6+Q85bSHLApHzBYsyHpdFvxXAY6v87SkmiuaeP+Do+vroIy4HZDMCU7Xf VZebeMyvYOSd0X1CBqjScLYN41TGm0bQSELF/bKc5EUhtGq6NGQJH5JG5l1NfVhadTmK OgQMalIJpAk5YVMkK4h6gPZ8I0IhNfaB57o2MXwAuxcnEdgz7yb3QFG8YoQ/7rZFFSIi GPqp06QmSChIZ4zn6BQTAyywlimxjimAnk+ZuZEvGsZxrz3jMuleIsPhvaUPtVgmfQCi zcKg== X-Forwarded-Encrypted: i=1; AJvYcCUoR6sY5uwrkmFGSoVRIZRWVD3Jn47Eng23sEpkv/I6Y9QldncbbwmdoTWK9LkTeLtoI/zEQHMOYecJw0upuQsxMNk= X-Gm-Message-State: AOJu0YxIRxonp/gKUiusp4TgtO+rFOJedzq9A4LfItQL9m6SzgYn6p75 q2V6E7UBMqCI4Jvvxf2ylQHOY5sSeA1//L+QtrBcBXztj7wKk4QyAmylW1UnHy+ADX2/gLyKWMS rLa86LS/ywf4cTM4L4g== X-Google-Smtp-Source: AGHT+IEqdMaTCeR+5N661ePS75QOvFa8jZaaiw9BcHvMUuJwBncrqCmIT7qsnybEQWIN2tFvBp2petyzPSOYJ3zy X-Received: from yosry.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:29b4]) (user=yosryahmed job=sendgmr) by 2002:a63:e57:0:b0:5dc:20e1:c894 with SMTP id 23-20020a630e57000000b005dc20e1c894mr3697pgo.4.1713403718174; Wed, 17 Apr 2024 18:28:38 -0700 (PDT) Date: Thu, 18 Apr 2024 01:28:33 +0000 Mime-Version: 1.0 X-Mailer: git-send-email 2.44.0.683.g7961c838ac-goog Message-ID: <20240418012835.3360429-1-yosryahmed@google.com> Subject: [PATCH tip:x86/mm v3 1/3] x86/mm: Use IPIs to synchronize LAM enablement From: Yosry Ahmed To: x86@kernel.org, Thomas Gleixner , Ingo Molnar Cc: Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Andy Lutomirski , Peter Zijlstra , "Kirill A. Shutemov" , Rick Edgecombe , Andrew Morton , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Yosry Ahmed X-Rspamd-Queue-Id: 9361640005 X-Stat-Signature: b1cxc5ournp7ij8uxq4oawn9xduh7cop X-Rspam-User: X-Rspamd-Server: rspam07 X-HE-Tag: 1713403719-48131 X-HE-Meta: U2FsdGVkX19ine58QrxJxqbjAjc0sUYWZD1DuPmO223wfACQsWMoj+LDvYnLIk420CqQP3vGWmDRSOhZj1AyjcrO10e8rtLQ3TMymBXkrkCv0aLtpI3lWm2hOM951oT0Hk8UsKNI837JEOcH4NKM6phlEdfqjFBTh0Je+iJaTDALszrW87fvLE7J2Ld696SWezviYbLbTsEWpN2G6aCn7S/3pF9YtZmbvQWtsfr4+FWBQOJWypwnXxaRI+DL6o5cgu/8UCl6TK7DaRf9s72tZi/3djJQXDnM9cgqmMSxFWoMTLAVyL5jSR4XGsgHlIBKA5ov4zKY5c41a5c2EXuxY4EfGLem9ygwxDalFky9yAepFIeQJcfQtKQVB8yIN7JOXhV2YXdp87FKE8TcVocsgh7QmWYItZH333sHDv7qTCMc+8b2Y0R04xaksj1LqLZVv68ldSI/ycxE26zaFcAxDZo0iEMxJQ6hVo0t6mfXb2d5QkVG1XyQsnvaOCosDNYQTwUnJe6whzQXR7nNg52qTRUR28cDCStOie/4cevUcE2CZ3dA2wgtciXKP2RkrkiPUbtwAhO5nK1h3FyHvwnwSkbtVaXu7M19kzM9je1ici898+tXizMhK+kg9bibOc+McPxl8JgsdLH/0SwD3afg+1bmk2K6AaHhd39Z3dhyh7HG9D+0G9Dgs9AaMzCmYb86f3KqHsgAIEcYjpT+4a7o7pUgjNESr1vWKC0dFZLBxhQR5u5qUmgkhap/Jr8/h60Fc2wTGuGfY/OEs8dtoUP/3nUd9A3TppIqTIhOJbB5/mnL0VI0kGNXhLc5ANN97Qii5HyEIbTTJfrlo+sCRGU8Q/Bbwpvtm9IQJq0S/zDV6CyQKZT+UJLZV7GxsENl+lMf1vl2RywmvL8YPJK1qtBlD0B37+thZ/aiEkc6iq+jxBuAY9OXYMbrrnw7+0PQYlbtzpAc79x/UNu+G2UvmJO 7xekMoGK FuuUmo7bee8zdBKxJRK4j3zIHM9Sl/JlgWpy2pfTv021Rp9wOYTDzNCLp4dyT971GpgDSrgT6C4TwjXRZ5/6xJjjnOmrTk37sSjs/fsMWbSG1Oo93j46O8A6E/+Vd51hnxZUGGGK4fxO9dBeZaKiSK8YK1czNeNU0XbNLtoFEVdxVFOxJ1A35tPveFXKv6zQr2jG2Zg2IMZ/R2wA1W7zdvAwGnYlDj2nZk+tHItrk3wa6yLs9Ae2l4s3SPtdBwcPfoVmEb9QogDE1CEO6gH8NxlJ+nr5w+8czs8I58hsdvLDnb8MqY+TUKmVXArhiVuqFD3Eot5BBTimNakg9V9T0P7lFWALiXFGw3xaQEPOnYyAUlb6Q2Ux4XQj3PVLEgdW7IuYYtdRsMFTvFymssJ0VpehpIurRIVaHylDIf0gJ2/ynTgmkzFGzVjotYq/qFyH4PZxWv9hnyhNZgHyYq4o1Q+ZEoKuonYWUxCxUGeapKI48BWpOedWWMSE3fBmeKCAEmQ3AjyfFkhU9WfL/CScv8/fyRIK2aKCaqnnohdnX55HIQUWW5HQzQIQUzFcv3Nh+XekGAO665wk4qz9eetTG63RCz+l2RKAkoitBucO4CXvNB7TaxubclVwnOBwl4N8nop+axvbhXwc6MutNlCHdhLsGZhTwOLMSrx/nyjfxAs+hqTfX2pyJzBVkWegY+vR4V20nQsN5fT63TMB0PRa22w28Qg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: LAM can only be enabled when a process is single-threaded. But _kernel_ threads can temporarily use a single-threaded process's mm. If LAM is enabled by a userspace process while a kthread is using its mm, the kthread will not observe LAM enablement (i.e. LAM will be disabled in CR3). This could be fine for the kthread itself, as LAM only affects userspace addresses. However, if the kthread context switches to a thread in the same userspace process, CR3 may or may not be updated because the mm_struct doesn't change (based on pending TLB flushes). If CR3 is not updated, the userspace thread will run incorrectly with LAM disabled, which may cause page faults when using tagged addresses. Example scenario: CPU 1 CPU 2 /* kthread */ kthread_use_mm() /* user thread */ prctl_enable_tagged_addr() /* LAM enabled on CPU 2 */ /* LAM disabled on CPU 1 */ context_switch() /* to CPU 1 */ /* Switching to user thread */ switch_mm_irqs_off() /* CR3 not updated */ /* LAM is still disabled on CPU 1 */ Synchronize LAM enablement by sending an IPI from prctl_enable_tagged_addr() to all CPUs running with the mm_struct to enable LAM. This makes sure LAM is enabled on CPU 1 in the above scenario before prctl_enable_tagged_addr() returns and userspace starts using tagged addresses, and before it's possible to run the userspace process on CPU 1. In switch_mm_irqs_off(), move reading the LAM mask until after mm_cpumask() is updated. This ensures that if an outdated LAM mask is written to CR3, an IPI is received to update it right after IRQs are re-enabled. Fixes: 82721d8b25d7 ("x86/mm: Handle LAM on context switch") Suggested-by: Andy Lutomirski Signed-off-by: Yosry Ahmed Reviewed-by: Kirill A. Shutemov --- v2 -> v3: - Rebased on top of the latest tip:x86/mm after v6.9-rc3. - Collected R-b on patch 2 (thanks!). --- arch/x86/kernel/process_64.c | 13 +++++++++++-- arch/x86/mm/tlb.c | 7 +++---- 2 files changed, 14 insertions(+), 6 deletions(-) diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index 7062b84dd467d..c27798f23ef82 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -798,6 +798,16 @@ static long prctl_map_vdso(const struct vdso_image *image, unsigned long addr) #define LAM_U57_BITS 6 +static void enable_lam_func(void *__mm) +{ + struct mm_struct *mm = __mm; + + if (this_cpu_read(cpu_tlbstate.loaded_mm) == mm) { + write_cr3(__read_cr3() | mm->context.lam_cr3_mask); + set_tlbstate_lam_mode(mm); + } +} + static int prctl_enable_tagged_addr(struct mm_struct *mm, unsigned long nr_bits) { if (!cpu_feature_enabled(X86_FEATURE_LAM)) @@ -830,8 +840,7 @@ static int prctl_enable_tagged_addr(struct mm_struct *mm, unsigned long nr_bits) return -EINVAL; } - write_cr3(__read_cr3() | mm->context.lam_cr3_mask); - set_tlbstate_lam_mode(mm); + on_each_cpu_mask(mm_cpumask(mm), enable_lam_func, mm, true); set_bit(MM_CONTEXT_LOCK_LAM, &mm->context.flags); mmap_write_unlock(mm); diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index 44ac64f3a047c..a041d2ecd8380 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -503,9 +503,9 @@ void switch_mm_irqs_off(struct mm_struct *unused, struct mm_struct *next, { struct mm_struct *prev = this_cpu_read(cpu_tlbstate.loaded_mm); u16 prev_asid = this_cpu_read(cpu_tlbstate.loaded_mm_asid); - unsigned long new_lam = mm_lam_cr3_mask(next); bool was_lazy = this_cpu_read(cpu_tlbstate_shared.is_lazy); unsigned cpu = smp_processor_id(); + unsigned long new_lam; u64 next_tlb_gen; bool need_flush; u16 new_asid; @@ -619,9 +619,7 @@ void switch_mm_irqs_off(struct mm_struct *unused, struct mm_struct *next, cpumask_clear_cpu(cpu, mm_cpumask(prev)); } - /* - * Start remote flushes and then read tlb_gen. - */ + /* Start receiving IPIs and then read tlb_gen (and LAM below) */ if (next != &init_mm) cpumask_set_cpu(cpu, mm_cpumask(next)); next_tlb_gen = atomic64_read(&next->context.tlb_gen); @@ -633,6 +631,7 @@ void switch_mm_irqs_off(struct mm_struct *unused, struct mm_struct *next, barrier(); } + new_lam = mm_lam_cr3_mask(next); set_tlbstate_lam_mode(next); if (need_flush) { this_cpu_write(cpu_tlbstate.ctxs[new_asid].ctx_id, next->context.ctx_id);