From patchwork Mon Jul 18 12:02:08 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 12921687 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6BE63C43334 for ; Mon, 18 Jul 2022 19:37:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0A3D86B007E; Mon, 18 Jul 2022 15:37:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0545A6B0080; Mon, 18 Jul 2022 15:37:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E36E36B0081; Mon, 18 Jul 2022 15:37:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id CFA196B007E for ; Mon, 18 Jul 2022 15:37:20 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id A42FF809 for ; Mon, 18 Jul 2022 19:37:20 +0000 (UTC) X-FDA: 79701229440.16.82ACFF7 Received: from relay5.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by imf05.hostedemail.com (Postfix) with ESMTP id 0276310006D for ; Mon, 18 Jul 2022 19:37:19 +0000 (UTC) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id A8B0F20B4C for ; Mon, 18 Jul 2022 19:37:19 +0000 (UTC) X-FDA: 79701229398.01.0A437E0 Received: from mail-pj1-f43.google.com (mail-pj1-f43.google.com [209.85.216.43]) by imf02.hostedemail.com (Postfix) with ESMTP id 388BA80043 for ; Mon, 18 Jul 2022 19:37:19 +0000 (UTC) Received: by mail-pj1-f43.google.com with SMTP id n4-20020a17090a73c400b001f1e87432c2so924615pjk.3 for ; Mon, 18 Jul 2022 12:37:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Oo4Pz1r5XHW7nwxpGgJZANPDnH94I0NGToeI6/zKa1I=; b=Yta4aSxqKdJJamMX5Ka9KLD4dubuyf5nvRkNeJsovDB9Ytj1BQ5qWdRgLzIHHHERB6 5sXf3niGo/72wXP0jlBMi/vVkFJX8hLRvbMlrw7tatNFrNNgmNMB9HbzNICkzbNcXs3w 6cu7pwr4AsBJZ3fn8/KYDPZk8nX+6Rg3Ljpt2/TGFqbUZ7cKL97bfzfYsiBq4vhT+Kw9 MVxwsvKLWzOIIRIJm9ZWmpzyPYTX/gBlEmxcuQU3TzAh1tBYia4QkzmOzqCE6j5LO15n QaXqO0JTJLp//5mmCRV7FullleEzXsADDRB6uSgI3/pl3o0omrcymnmvGc3DAicOGAQh YnEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Oo4Pz1r5XHW7nwxpGgJZANPDnH94I0NGToeI6/zKa1I=; b=ju1ztVNcc1qktWOOyZ1zP7vywxNHqaYmb3gdut6bfYr9lzejTETrJRMG5bnQ+5MDHD VxmpES83I7VNvFLy7D1wLGDYnk/tAuEN+lBUxlkZE0oEHP5l6P+7kQH+gDrplU2F+hjk +mav3b6LiIU9epKRzDA0wPJVaLpUv+wsMhBZ/ys4kToPJlmLCySNvl/hiwLdUhhayUY3 bMour5W+fm+o5SCGfltch4vYkjjOK0wgt6+q6ZB14/UdT9/U1RUN+W2Is/WNcMJQDZ7U Fag0o4K5MaqGkYL6FNBVVLiy5dt13949LQEQm5NqDLPDLud8hRr93f9Xst6m7AKzoyFU 1zOg== X-Gm-Message-State: AJIora+zflD28i7dAS51IQwFiQWF3xDNu/PKDH4/FnBX3hSdwNzIgrye cIgpjcpxE36QoYPam2V674GybQ/kQxgFzg== X-Google-Smtp-Source: AGRyM1v14qCZwBb3XJU5MS/ZzWDs+xXqoaSfao3rsct5ND+6jaOGoGucVBBiblDjUX+cd9G34nsdiw== X-Received: by 2002:a17:902:7106:b0:16c:6c95:6153 with SMTP id a6-20020a170902710600b0016c6c956153mr29356100pll.166.1658173037752; Mon, 18 Jul 2022 12:37:17 -0700 (PDT) Received: from sc2-haas01-esx0118.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id q6-20020a170902a3c600b0016bc4a6ce28sm9907887plb.98.2022.07.18.12.37.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Jul 2022 12:37:17 -0700 (PDT) From: Nadav Amit X-Google-Original-From: Nadav Amit To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Andrew Morton , Mike Rapoport , Axel Rasmussen , Nadav Amit , Andrea Arcangeli , Andrew Cooper , Andy Lutomirski , Dave Hansen , David Hildenbrand , Peter Xu , Peter Zijlstra , Thomas Gleixner , Will Deacon , Yu Zhao , Nick Piggin Subject: [RFC PATCH 10/14] x86/mm: introduce relaxed TLB flushes Date: Mon, 18 Jul 2022 05:02:08 -0700 Message-Id: <20220718120212.3180-11-namit@vmware.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220718120212.3180-1-namit@vmware.com> References: <20220718120212.3180-1-namit@vmware.com> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1658173040; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Oo4Pz1r5XHW7nwxpGgJZANPDnH94I0NGToeI6/zKa1I=; b=db3vXujvG+LZIL9BIHCqEP0YevMse7Bfaa/WSI8mi3IfnPX5+5xT8KfKUayvMXgXaVlLQT s0B0oVJBBgjai6FcOx4IT7/+/UDKfH84ga7PG59i0nq4P0xWqMF4RlXR8H11aCeomSV05d ftLvYB8AWyLueY4pvqzt9PJ0LwbYgRM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1658173040; a=rsa-sha256; cv=none; b=Caz15EcojeI9EOzdVj/zxSq8ov5f5rT6nRrQi7/QTP9ZFrPMfMe6gtGZWY9X2laSDPSpFf rG2hnvfIVEJV4a8gMpeUBUwqMMalEX3KnObIbivHvARcHw5T6HyxNFhDCQvmtX1ow4I6hm 7rfn9qCTZI07FpwvtxSHSTO3mVa+rMc= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=Yta4aSxq; dmarc=pass (policy=none) header.from=gmail.com; spf=none (imf05.hostedemail.com: domain of MAILER-DAEMON@hostedemail.com has no SPF policy when checking 216.40.44.16) smtp.mailfrom=MAILER-DAEMON@hostedemail.com X-Stat-Signature: mis3z9t4nuxa5tdxxtrsbgwx5b7bdwj4 X-Rspamd-Queue-Id: 0276310006D X-HE-Tag-Orig: 1658173038-930647 X-Rspam-User: Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=Yta4aSxq; dmarc=pass (policy=none) header.from=gmail.com; spf=none (imf05.hostedemail.com: domain of MAILER-DAEMON@hostedemail.com has no SPF policy when checking 216.40.44.16) smtp.mailfrom=MAILER-DAEMON@hostedemail.com X-Rspamd-Server: rspam06 X-HE-Tag: 1658173039-338064 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Nadav Amit Introduce relaxed TLB flushes in x86. When protection is removed from PTEs (i.e., PTEs become writeable or executable), relaxed TLB flushes would be used. Relaxed TLB flushes do flush the local TLB, but do not flush remote TLBs. If later a spurious page-fault is encountered, and the local TLB generation is found to be out of sync with the mm's TLB generation, a full TLB flush takes place to prevent further spurious page-faults from occurring. Cc: Andrea Arcangeli Cc: Andrew Cooper Cc: Andrew Morton Cc: Andy Lutomirski Cc: Dave Hansen Cc: David Hildenbrand Cc: Peter Xu Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Will Deacon Cc: Yu Zhao Cc: Nick Piggin Signed-off-by: Nadav Amit --- arch/x86/include/asm/tlb.h | 3 ++- arch/x86/include/asm/tlbflush.h | 9 +++++---- arch/x86/kernel/alternative.c | 2 +- arch/x86/kernel/ldt.c | 3 ++- arch/x86/mm/tlb.c | 4 ++-- 5 files changed, 12 insertions(+), 9 deletions(-) diff --git a/arch/x86/include/asm/tlb.h b/arch/x86/include/asm/tlb.h index 1bfe979bb9bc..51c85136f9a8 100644 --- a/arch/x86/include/asm/tlb.h +++ b/arch/x86/include/asm/tlb.h @@ -20,7 +20,8 @@ static inline void tlb_flush(struct mmu_gather *tlb) end = tlb->end; } - flush_tlb_mm_range(tlb->mm, start, end, stride_shift, tlb->freed_tables); + flush_tlb_mm_range(tlb->mm, start, end, stride_shift, tlb->freed_tables, + tlb->strict); } /* diff --git a/arch/x86/include/asm/tlbflush.h b/arch/x86/include/asm/tlbflush.h index 77d4810e5a5d..230cd1d24fe6 100644 --- a/arch/x86/include/asm/tlbflush.h +++ b/arch/x86/include/asm/tlbflush.h @@ -220,23 +220,24 @@ void flush_tlb_multi(const struct cpumask *cpumask, #endif #define flush_tlb_mm(mm) \ - flush_tlb_mm_range(mm, 0UL, TLB_FLUSH_ALL, 0UL, true) + flush_tlb_mm_range(mm, 0UL, TLB_FLUSH_ALL, 0UL, true, true) #define flush_tlb_range(vma, start, end) \ flush_tlb_mm_range((vma)->vm_mm, start, end, \ ((vma)->vm_flags & VM_HUGETLB) \ ? huge_page_shift(hstate_vma(vma)) \ - : PAGE_SHIFT, false) + : PAGE_SHIFT, false, true) extern void flush_tlb_all(void); extern void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, unsigned long end, unsigned int stride_shift, - bool freed_tables); + bool freed_tables, bool strict); extern void flush_tlb_kernel_range(unsigned long start, unsigned long end); static inline void flush_tlb_page(struct vm_area_struct *vma, unsigned long a) { - flush_tlb_mm_range(vma->vm_mm, a, a + PAGE_SIZE, PAGE_SHIFT, false); + flush_tlb_mm_range(vma->vm_mm, a, a + PAGE_SIZE, PAGE_SHIFT, false, + true); } static inline u64 inc_mm_tlb_gen(struct mm_struct *mm) diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c index e257f6c80372..48945a47fd76 100644 --- a/arch/x86/kernel/alternative.c +++ b/arch/x86/kernel/alternative.c @@ -1099,7 +1099,7 @@ static void *__text_poke(text_poke_f func, void *addr, const void *src, size_t l */ flush_tlb_mm_range(poking_mm, poking_addr, poking_addr + (cross_page_boundary ? 2 : 1) * PAGE_SIZE, - PAGE_SHIFT, false); + PAGE_SHIFT, false, true); if (func == text_poke_memcpy) { /* diff --git a/arch/x86/kernel/ldt.c b/arch/x86/kernel/ldt.c index 525876e7b9f4..7c7bc97324bc 100644 --- a/arch/x86/kernel/ldt.c +++ b/arch/x86/kernel/ldt.c @@ -372,7 +372,8 @@ static void unmap_ldt_struct(struct mm_struct *mm, struct ldt_struct *ldt) } va = (unsigned long)ldt_slot_va(ldt->slot); - flush_tlb_mm_range(mm, va, va + nr_pages * PAGE_SIZE, PAGE_SHIFT, false); + flush_tlb_mm_range(mm, va, va + nr_pages * PAGE_SIZE, PAGE_SHIFT, false, + true); } #else /* !CONFIG_PAGE_TABLE_ISOLATION */ diff --git a/arch/x86/mm/tlb.c b/arch/x86/mm/tlb.c index ff3bcc55435e..ec5033d28a97 100644 --- a/arch/x86/mm/tlb.c +++ b/arch/x86/mm/tlb.c @@ -974,7 +974,7 @@ void flush_tlb_fix_spurious_fault(struct vm_area_struct *vma, void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, unsigned long end, unsigned int stride_shift, - bool freed_tables) + bool freed_tables, bool strict) { struct flush_tlb_info *info; u64 new_tlb_gen; @@ -1000,7 +1000,7 @@ void flush_tlb_mm_range(struct mm_struct *mm, unsigned long start, * a local TLB flush is needed. Optimize this use-case by calling * flush_tlb_func_local() directly in this case. */ - if (cpumask_any_but(mm_cpumask(mm), cpu) < nr_cpu_ids) { + if (strict && cpumask_any_but(mm_cpumask(mm), cpu) < nr_cpu_ids) { flush_tlb_multi(mm_cpumask(mm), info); } else if (mm == this_cpu_read(cpu_tlbstate.loaded_mm)) { lockdep_assert_irqs_enabled();