From patchwork Tue Jun 12 07:16:20 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 10459503 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id C2F87603B4 for ; Tue, 12 Jun 2018 07:16:52 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AFA792863D for ; Tue, 12 Jun 2018 07:16:52 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A24F828676; Tue, 12 Jun 2018 07:16:52 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID, DKIM_VALID_AU, FREEMAIL_FROM, MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0C2602863D for ; Tue, 12 Jun 2018 07:16:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D93966B0271; Tue, 12 Jun 2018 03:16:50 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id D1CA96B0272; Tue, 12 Jun 2018 03:16:50 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BBFDC6B0273; Tue, 12 Jun 2018 03:16:50 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl0-f70.google.com (mail-pl0-f70.google.com [209.85.160.70]) by kanga.kvack.org (Postfix) with ESMTP id 739816B0271 for ; Tue, 12 Jun 2018 03:16:50 -0400 (EDT) Received: by mail-pl0-f70.google.com with SMTP id bf1-v6so13566998plb.2 for ; Tue, 12 Jun 2018 00:16:50 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=d4TVD9wsNscDaKLrUOqcUiGr2UY3sYJQXf9huR7qTP8=; b=N43MIUTJ6cf1WrT0tw/3G7TYLPBoFK9q0McgGMcVRx4gJUtQ9kxTWhdNPgZ6MwU0MZ Fc3QTJppAc/d304WIMOXv5Fn0k0RcvpA/fu6y475bhfXBGTF9WGBaT6zPQpUVRBEvmOB DOYyeIJ1xoE9yRlo6NMs0L0rS225yAjIS2CYD6FeFC5+YE2YRn5U0dKJznI/MxdAUhxi 0a01oIXGpXLDB63tymW3BjIfBBnF08X2QayF1sS92y/DZ6TLEglV80sNBD8oZb63Y0gV o8/BvhGUuIWfQT7hUTg3lNGAX3WhUEjkVXttUOuJOQ7icAfjo1xiJbUpkx/+a/ESjr6H QJXw== X-Gm-Message-State: APt69E1HwI3kXPqXZWcsNOHioF0GbkEsZ6NQbghrOIDzmQthck2h5ID3 3hFgsNJu8rf3ybZkAy7XbyJ2/94b4DPWbgNFggoxhjXHmGqum/ncj3a4N/sKodLEIL35xWEz9jF Olf/LGho6Hls5XGtCTgdx3IYzSTdRRPbRZbO6qZUbjnvLKFrxEC9TLojfa4lUhaIWILjzwuTXp5 iprxzYnbtYipocosqDXZJWCFR6qycaGNTgx3sSmwKAOxD3Hk5jz2Al1zJ2zDNVfKW2suZ4H90pp +2tYb2F0aWqdLeBFI3xIhg5LdtmH+F93U2zbNAbu9QP6i6PtsL56jrVeIPVqzpV4hYjKiBuqaM8 nZkH6MgB/DjJIFhPLgzGD7btWyi5pbnD14g2n5rDQIm92/zCyOw0zJ5IorAxNkl603cYMxRAQyj c X-Received: by 2002:a62:a6dd:: with SMTP id r90-v6mr2599244pfl.60.1528787810108; Tue, 12 Jun 2018 00:16:50 -0700 (PDT) X-Received: by 2002:a62:a6dd:: with SMTP id r90-v6mr2599187pfl.60.1528787808960; Tue, 12 Jun 2018 00:16:48 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1528787808; cv=none; d=google.com; s=arc-20160816; b=hdLWa4S/9ghI+b81Zks3MqH18gvl0VxLGMecuFScBptnIKzn4ZYzdNRFRRFeibpuTc 7U2XIcx8tHpcCp4clEBo5ZJ7EUaxbxq1NCD2jogYaYFnKxW/8V2WxQEbKKMYga4clRm2 zsDmsA5BH6qMVvKXKVNRH6E4SZ8RPBDZpZX8RKNLQIZRHF5T6h2CWcXn5NlNvXajJ/3K h/6nco+O7hbtDTe3KoAOTFjLTDdCH5kshHF3jrjtEM53FUXcF/TKzZyXINKdQgWikDp7 7Vlar1hidWMXSipd9Or7ZrqQRbxRP2vyyhipsDB4ZaybG3LRypIER8g3swdCEsxWhyvj 0ouQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:arc-authentication-results; bh=d4TVD9wsNscDaKLrUOqcUiGr2UY3sYJQXf9huR7qTP8=; b=CV+mglAga+0PNZOovl16XY1ikU0xhSpcGnGmA0X01s8GwqTWrZys1hLymoVAdQIVVB RDkiZNtRf5832ixFnZdDFSQ0b+OneFVO1PIJfD1UqLgYVhFkmA55ANrWVZLeDtKL4VLu AJj/bPW1/lA0wGA+S/+5BhO6RUKHl7MeHeKUuLFkiuv2PZ5NBPmIVWHL0k0ZJR4AVg6d X/nxu++THdpPMRiBXC/H2iE2K0V7GdQZmlXMBETKYOz0jtnpjRE89Aio3okt94M3Aks+ lN57joZQPviajz2lqQOjwbR4oMPGwak1QBsLMSqgPDaQVmCTye33DnjpfZv5n8T6eBw1 81RQ== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=LmqOk7vH; spf=pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id s4-v6sor65238pfg.84.2018.06.12.00.16.48 for (Google Transport Security); Tue, 12 Jun 2018 00:16:48 -0700 (PDT) Received-SPF: pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) client-ip=209.85.220.65; Authentication-Results: mx.google.com; dkim=pass header.i=@gmail.com header.s=20161025 header.b=LmqOk7vH; spf=pass (google.com: domain of npiggin@gmail.com designates 209.85.220.65 as permitted sender) smtp.mailfrom=npiggin@gmail.com; dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=d4TVD9wsNscDaKLrUOqcUiGr2UY3sYJQXf9huR7qTP8=; b=LmqOk7vH6iM+T+W6bBZtqJ97J4yKkz8z1HgbAk5D0XjgjCpoZ2HRa3DjqhBTP1p09s CPAsK8cJOAPWy67+H29/RqYv2oZtYAMyDjVYDl08NYSfFGZoOD9mGZfDQcQBK0STOpZp rjaMcVJS3yyXeY3Es3udb+8AZ08BM06K94feTrDGZpwVPuUe6OATptrsu2He3TlyPY0U Hv5whww99Zys2tJrn57hkJe1hWq1Tan/vbSJGhcr5x+jT0tju0dENHLVOGaTzSeOrb4Z cO42XT/2gBae9ljaXALCDskktmzf6aELf/mdWUTaTpe7xbg4aAg+6kKwvmZVhucIOv8r GHdw== X-Google-Smtp-Source: ADUXVKLAg/O2mAd/KWXCK8fOoS30Qd3C96LF+V/bhsjC+YkcVcy2r7+Q6d/BR/nNpBZWcLlPW1a4dw== X-Received: by 2002:a62:e117:: with SMTP id q23-v6mr2580516pfh.75.1528787808472; Tue, 12 Jun 2018 00:16:48 -0700 (PDT) Received: from roar.local0.net (59-102-70-78.tpgi.com.au. [59.102.70.78]) by smtp.gmail.com with ESMTPSA id q24-v6sm649324pfh.26.2018.06.12.00.16.43 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 12 Jun 2018 00:16:47 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org Cc: Nicholas Piggin , linuxppc-dev@lists.ozlabs.org, linux-arch@vger.kernel.org, "Aneesh Kumar K . V" , Minchan Kim , Mel Gorman , Nadav Amit , Andrew Morton , Linus Torvalds Subject: [RFC PATCH 2/3] mm: mmu_gather track of invalidated TLB ranges explicitly for more precise flushing Date: Tue, 12 Jun 2018 17:16:20 +1000 Message-Id: <20180612071621.26775-3-npiggin@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180612071621.26775-1-npiggin@gmail.com> References: <20180612071621.26775-1-npiggin@gmail.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP The mmu_gather APIs keep track of the invalidated address range including the span covered by invalidated page table pages. Page table pages with no ptes (and therefore could not have TLB entries) still need to be involved in the invalidation if the processor caches intermediate levels of the page table. This allows a backwards compatible / legacy implementation to cache page tables without modification, if they invalidate their page table cache using their existing tlb invalidation instructions. However this additional flush range is not necessary if the architecture provides explicit page table cache management, or if it ensures that page table cache entries will never be instantiated if they did not reach a valid pte. This is very noticable on powerpc in the exec path, in shift_arg_pages where the TLB flushing for the page table teardown is a very large range that gets implemented as a full process flush. This patch provides page_start and page_end fields to mmu_gather which architectures can use to optimise their TLB flushing. Signed-off-by: Nicholas Piggin --- include/asm-generic/tlb.h | 27 +++++++++++++++++++++++++-- mm/memory.c | 4 +++- 2 files changed, 28 insertions(+), 3 deletions(-) diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index faddde44de8c..a006f702b4c2 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -96,6 +96,8 @@ struct mmu_gather { #endif unsigned long start; unsigned long end; + unsigned long page_start; + unsigned long page_end; /* we are in the middle of an operation to clear * a full mm and can make some optimizations */ unsigned int fullmm : 1, @@ -128,13 +130,25 @@ static inline void __tlb_adjust_range(struct mmu_gather *tlb, tlb->end = max(tlb->end, address + range_size); } +static inline void __tlb_adjust_page_range(struct mmu_gather *tlb, + unsigned long address, + unsigned int range_size) +{ + tlb->page_start = min(tlb->page_start, address); + tlb->page_end = max(tlb->page_end, address + range_size); +} + + static inline void __tlb_reset_range(struct mmu_gather *tlb) { if (tlb->fullmm) { tlb->start = tlb->end = ~0; + tlb->page_start = tlb->page_end = ~0; } else { tlb->start = TASK_SIZE; tlb->end = 0; + tlb->page_start = TASK_SIZE; + tlb->page_end = 0; } } @@ -210,12 +224,14 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb, #define tlb_remove_tlb_entry(tlb, ptep, address) \ do { \ __tlb_adjust_range(tlb, address, PAGE_SIZE); \ + __tlb_adjust_page_range(tlb, address, PAGE_SIZE); \ __tlb_remove_tlb_entry(tlb, ptep, address); \ } while (0) #define tlb_remove_huge_tlb_entry(h, tlb, ptep, address) \ do { \ __tlb_adjust_range(tlb, address, huge_page_size(h)); \ + __tlb_adjust_page_range(tlb, address, huge_page_size(h)); \ __tlb_remove_tlb_entry(tlb, ptep, address); \ } while (0) @@ -230,6 +246,7 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb, #define tlb_remove_pmd_tlb_entry(tlb, pmdp, address) \ do { \ __tlb_adjust_range(tlb, address, HPAGE_PMD_SIZE); \ + __tlb_adjust_page_range(tlb, address, HPAGE_PMD_SIZE); \ __tlb_remove_pmd_tlb_entry(tlb, pmdp, address); \ } while (0) @@ -244,6 +261,7 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb, #define tlb_remove_pud_tlb_entry(tlb, pudp, address) \ do { \ __tlb_adjust_range(tlb, address, HPAGE_PUD_SIZE); \ + __tlb_adjust_page_range(tlb, address, HPAGE_PUD_SIZE); \ __tlb_remove_pud_tlb_entry(tlb, pudp, address); \ } while (0) @@ -262,6 +280,11 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb, * architecture to do its own odd thing, not cause pain for others * http://lkml.kernel.org/r/CA+55aFzBggoXtNXQeng5d_mRoDnaMBE5Y+URs+PHR67nUpMtaw@mail.gmail.com * + * Powerpc (Book3S 64-bit) with the radix MMU has an architected "page + * walk cache" that is invalidated with a specific instruction. It uses + * need_flush_all to issue this instruction, which is set by its own + * __p??_free_tlb functions. + * * For now w.r.t page table cache, mark the range_size as PAGE_SIZE */ @@ -273,7 +296,7 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb, #define pmd_free_tlb(tlb, pmdp, address) \ do { \ - __tlb_adjust_range(tlb, address, PAGE_SIZE); \ + __tlb_adjust_range(tlb, address, PAGE_SIZE); \ __pmd_free_tlb(tlb, pmdp, address); \ } while (0) @@ -288,7 +311,7 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb, #ifndef __ARCH_HAS_5LEVEL_HACK #define p4d_free_tlb(tlb, pudp, address) \ do { \ - __tlb_adjust_range(tlb, address, PAGE_SIZE); \ + __tlb_adjust_range(tlb, address, PAGE_SIZE); \ __p4d_free_tlb(tlb, pudp, address); \ } while (0) #endif diff --git a/mm/memory.c b/mm/memory.c index 9d472e00fc2d..a46896b85e54 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -277,8 +277,10 @@ void arch_tlb_finish_mmu(struct mmu_gather *tlb, { struct mmu_gather_batch *batch, *next; - if (force) + if (force) { __tlb_adjust_range(tlb, start, end - start); + __tlb_adjust_page_range(tlb, start, end - start); + } tlb_flush_mmu(tlb);