From patchwork Tue Sep 4 11:45:31 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Will Deacon X-Patchwork-Id: 10587207 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AA6B5112B for ; Tue, 4 Sep 2018 11:45:37 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 978A829261 for ; Tue, 4 Sep 2018 11:45:37 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8C3A52926A; Tue, 4 Sep 2018 11:45:37 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DC0C429261 for ; Tue, 4 Sep 2018 11:45:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 370036B6D54; Tue, 4 Sep 2018 07:45:23 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 323AF6B6D55; Tue, 4 Sep 2018 07:45:23 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0FB176B6D56; Tue, 4 Sep 2018 07:45:23 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-oi0-f70.google.com (mail-oi0-f70.google.com [209.85.218.70]) by kanga.kvack.org (Postfix) with ESMTP id D99716B6D54 for ; Tue, 4 Sep 2018 07:45:22 -0400 (EDT) Received: by mail-oi0-f70.google.com with SMTP id w185-v6so3795703oig.19 for ; Tue, 04 Sep 2018 04:45:22 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references; bh=vfGjlN9R0u/blO0JGqukNKHV35fKWjX32T6xo/ATi24=; b=PFlnxQWnfHShscwpqwDSVl+Jy3Ewc2lbgyLziMzQcYsH/g4TijlXIt315oMn9RrX8C t0H20EopdLER2v/b+sc2T6bSXSKvaTXJ3UTJ8pf6YW78Uiho3VhFgT5PJy+iPPXd/AFF uxL81+XKIGiTSM4cSDBKO8PLg79yC4POpyZhFKbmprwYEH/U30RTz9o800RS99T+YUsm eK1vWQLddwo/bRTWPlRTFTMoSQ8PIoOaTyKdFVSPNHweBzoBC9I5FnefUN+9bMcepnQ2 kl/oPHPYaqYsBfPpgPbTX0fe3Z0c2Z6cTrF3Jr5x9gNhPa40BAakb4oLSbNYKhNJ9ky+ HtgQ== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of will.deacon@arm.com designates 217.140.101.70 as permitted sender) smtp.mailfrom=will.deacon@arm.com X-Gm-Message-State: APzg51BPedfEIgr6XQV3vAW4jCasJuvYL+wx1nSX8h6mG3T/yHDXuPbe E0yqunt5X3FzKhpyI8EJ38GAGe0MAuOebJKsOf103XrLJnsnquOidv7IT8KneBaCwviR1d4HOXN dPu1YbzPHHb+8n40AsvxaNlzrSfv83GVPWqy6c0BjPz+OziCE9Yz8aS1033bh/suOmA== X-Received: by 2002:aca:5155:: with SMTP id f82-v6mr22824072oib.272.1536061521344; Tue, 04 Sep 2018 04:45:21 -0700 (PDT) X-Google-Smtp-Source: ANB0VdZXHZMaVUN//HOItEWTZtdmdBFv4Dpdp26fCKYDEyWrLOTbyZmxGvO0KYwuK6pHkjwdePpq X-Received: by 2002:aca:5155:: with SMTP id f82-v6mr22823995oib.272.1536061520413; Tue, 04 Sep 2018 04:45:20 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1536061520; cv=none; d=google.com; s=arc-20160816; b=b84DMfUS0vm5bgJBlt+UQzdRP+/nknS+52hNXaDY6fo0Xugofqamxe4pmu8VV0+cwL 22//v82V2u8zI5CkJPe40Ox2Jh61dLQKo0A+TGciu5ncBxtHF28jH+PVVLhD6JqheITP dkTWVUOkRuXhuzLypebHq1hYZsbo2wSgihUlLknG+reL69KFwLqN1GTGPQ1r0yW4vF3V 78LBRqaZhTYok4PeFN5m1D3W2GntZNN0scvQ4QiP327kW1tPoJxl/wpnUVOLHTVpX3o2 6vI6Uzmqi67LsrfRBYYnn4Zk3GYOFJu1cyr5S7L2uHzyS25oIqm7fHOr/S/9WIpePB0v u9wA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :arc-authentication-results; bh=vfGjlN9R0u/blO0JGqukNKHV35fKWjX32T6xo/ATi24=; b=rfjvhocMVRjvTZqJ+GRuf/BfAAiJYOP1MgmH6JQqLjqSV2M28lJtQRp0FVYYI4kRob xbFkQ+/0SLsfYSX5VH9iTdrnpLW5AhfmR/dowC3cjFBVulNDmGVcL2q1HhrmEEKZplUM 9QXkz7fcfQomEjlhyuqGyBOoYPb4TKd3X+cgJxTvCI5bb4NF3IoKrpPB3Zfev/WCz4oM Zvsj0izhWOMFVZANVbqPLQtjP07tNJhTGeZG8URXmFd5yZ2Lm124ABaAJAJsVvs6FjHm pfd6YnRbrRFh5r2ketdtoUf+PeQq/Siz9c95U2pmjodeiK73dVvrL+yCWMGdk46jiQpn 9HQg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of will.deacon@arm.com designates 217.140.101.70 as permitted sender) smtp.mailfrom=will.deacon@arm.com Received: from foss.arm.com (usa-sjc-mx-foss1.foss.arm.com. [217.140.101.70]) by mx.google.com with ESMTP id e80-v6si4549869oib.276.2018.09.04.04.45.20 for ; Tue, 04 Sep 2018 04:45:20 -0700 (PDT) Received-SPF: pass (google.com: domain of will.deacon@arm.com designates 217.140.101.70 as permitted sender) client-ip=217.140.101.70; Authentication-Results: mx.google.com; spf=pass (google.com: domain of will.deacon@arm.com designates 217.140.101.70 as permitted sender) smtp.mailfrom=will.deacon@arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.72.51.249]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id CF4CD15BE; Tue, 4 Sep 2018 04:45:19 -0700 (PDT) Received: from edgewater-inn.cambridge.arm.com (usa-sjc-imap-foss1.foss.arm.com [10.72.51.249]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id A1B5D3F93D; Tue, 4 Sep 2018 04:45:19 -0700 (PDT) Received: by edgewater-inn.cambridge.arm.com (Postfix, from userid 1000) id A58AD1AE35FF; Tue, 4 Sep 2018 12:45:33 +0100 (BST) From: Will Deacon To: linux-kernel@vger.kernel.org Cc: peterz@infradead.org, npiggin@gmail.com, linux-mm@kvack.org, kirill.shutemov@linux.intel.com, akpm@linux-foundation.org, mhocko@suse.com, aneesh.kumar@linux.vnet.ibm.com Subject: [PATCH v2 3/5] asm-generic/tlb: Track which levels of the page tables have been cleared Date: Tue, 4 Sep 2018 12:45:31 +0100 Message-Id: <1536061533-16188-4-git-send-email-will.deacon@arm.com> X-Mailer: git-send-email 2.1.4 In-Reply-To: <1536061533-16188-1-git-send-email-will.deacon@arm.com> References: <1536061533-16188-1-git-send-email-will.deacon@arm.com> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP It is common for architectures with hugepage support to require only a single TLB invalidation operation per hugepage during unmap(), rather than iterating through the mapping at a PAGE_SIZE increment. Currently, however, the level in the page table where the unmap() operation occurs is not stored in the mmu_gather structure, therefore forcing architectures to issue additional TLB invalidation operations or to give up and over-invalidate by e.g. invalidating the entire TLB. Ideally, we could add an interval rbtree to the mmu_gather structure, which would allow us to associate the correct mapping granule with the various sub-mappings within the range being invalidated. However, this is costly in terms of book-keeping and memory management, so instead we approximate by keeping track of the page table levels that are cleared and provide a means to query the smallest granule required for invalidation. Acked-by: Peter Zijlstra (Intel) Acked-by: Nicholas Piggin Signed-off-by: Will Deacon --- include/asm-generic/tlb.h | 58 ++++++++++++++++++++++++++++++++++++++++------- mm/memory.c | 4 +++- 2 files changed, 53 insertions(+), 9 deletions(-) diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 2b444ad94566..9791e98122a0 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -116,6 +116,14 @@ struct mmu_gather { */ unsigned int freed_tables : 1; + /* + * at which levels have we cleared entries? + */ + unsigned int cleared_ptes : 1; + unsigned int cleared_pmds : 1; + unsigned int cleared_puds : 1; + unsigned int cleared_p4ds : 1; + struct mmu_gather_batch *active; struct mmu_gather_batch local; struct page *__pages[MMU_GATHER_BUNDLE]; @@ -150,6 +158,10 @@ static inline void __tlb_reset_range(struct mmu_gather *tlb) tlb->end = 0; } tlb->freed_tables = 0; + tlb->cleared_ptes = 0; + tlb->cleared_pmds = 0; + tlb->cleared_puds = 0; + tlb->cleared_p4ds = 0; } static inline void tlb_flush_mmu_tlbonly(struct mmu_gather *tlb) @@ -199,6 +211,25 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb, } #endif +static inline unsigned long tlb_get_unmap_shift(struct mmu_gather *tlb) +{ + if (tlb->cleared_ptes) + return PAGE_SHIFT; + if (tlb->cleared_pmds) + return PMD_SHIFT; + if (tlb->cleared_puds) + return PUD_SHIFT; + if (tlb->cleared_p4ds) + return P4D_SHIFT; + + return PAGE_SHIFT; +} + +static inline unsigned long tlb_get_unmap_size(struct mmu_gather *tlb) +{ + return 1UL << tlb_get_unmap_shift(tlb); +} + /* * In the case of tlb vma handling, we can optimise these away in the * case where we're doing a full MM flush. When we're doing a munmap, @@ -232,13 +263,19 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb, #define tlb_remove_tlb_entry(tlb, ptep, address) \ do { \ __tlb_adjust_range(tlb, address, PAGE_SIZE); \ + tlb->cleared_ptes = 1; \ __tlb_remove_tlb_entry(tlb, ptep, address); \ } while (0) -#define tlb_remove_huge_tlb_entry(h, tlb, ptep, address) \ - do { \ - __tlb_adjust_range(tlb, address, huge_page_size(h)); \ - __tlb_remove_tlb_entry(tlb, ptep, address); \ +#define tlb_remove_huge_tlb_entry(h, tlb, ptep, address) \ + do { \ + unsigned long _sz = huge_page_size(h); \ + __tlb_adjust_range(tlb, address, _sz); \ + if (_sz == PMD_SIZE) \ + tlb->cleared_pmds = 1; \ + else if (_sz == PUD_SIZE) \ + tlb->cleared_puds = 1; \ + __tlb_remove_tlb_entry(tlb, ptep, address); \ } while (0) /** @@ -252,6 +289,7 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb, #define tlb_remove_pmd_tlb_entry(tlb, pmdp, address) \ do { \ __tlb_adjust_range(tlb, address, HPAGE_PMD_SIZE); \ + tlb->cleared_pmds = 1; \ __tlb_remove_pmd_tlb_entry(tlb, pmdp, address); \ } while (0) @@ -266,6 +304,7 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb, #define tlb_remove_pud_tlb_entry(tlb, pudp, address) \ do { \ __tlb_adjust_range(tlb, address, HPAGE_PUD_SIZE); \ + tlb->cleared_puds = 1; \ __tlb_remove_pud_tlb_entry(tlb, pudp, address); \ } while (0) @@ -291,7 +330,8 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb, #define pte_free_tlb(tlb, ptep, address) \ do { \ __tlb_adjust_range(tlb, address, PAGE_SIZE); \ - tlb->freed_tables = 1; \ + tlb->freed_tables = 1; \ + tlb->cleared_pmds = 1; \ __pte_free_tlb(tlb, ptep, address); \ } while (0) #endif @@ -300,7 +340,8 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb, #define pmd_free_tlb(tlb, pmdp, address) \ do { \ __tlb_adjust_range(tlb, address, PAGE_SIZE); \ - tlb->freed_tables = 1; \ + tlb->freed_tables = 1; \ + tlb->cleared_puds = 1; \ __pmd_free_tlb(tlb, pmdp, address); \ } while (0) #endif @@ -310,7 +351,8 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb, #define pud_free_tlb(tlb, pudp, address) \ do { \ __tlb_adjust_range(tlb, address, PAGE_SIZE); \ - tlb->freed_tables = 1; \ + tlb->freed_tables = 1; \ + tlb->cleared_p4ds = 1; \ __pud_free_tlb(tlb, pudp, address); \ } while (0) #endif @@ -321,7 +363,7 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb, #define p4d_free_tlb(tlb, pudp, address) \ do { \ __tlb_adjust_range(tlb, address, PAGE_SIZE); \ - tlb->freed_tables = 1; \ + tlb->freed_tables = 1; \ __p4d_free_tlb(tlb, pudp, address); \ } while (0) #endif diff --git a/mm/memory.c b/mm/memory.c index c467102a5cbc..9135f48e8d84 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -267,8 +267,10 @@ void arch_tlb_finish_mmu(struct mmu_gather *tlb, { struct mmu_gather_batch *batch, *next; - if (force) + if (force) { + __tlb_reset_range(tlb); __tlb_adjust_range(tlb, start, end - start); + } tlb_flush_mmu(tlb);