From patchwork Wed Jul 25 14:06:38 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 10544079 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E694517FD for ; Wed, 25 Jul 2018 14:14:07 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D5A842A07E for ; Wed, 25 Jul 2018 14:14:07 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C9BBB2A0B5; Wed, 25 Jul 2018 14:14:07 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.4 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,DKIM_VALID,FREEMAIL_FROM,MAILING_LIST_MULTI,SUSPICIOUS_RECIPS autolearn=no version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id CA41A2A07E for ; Wed, 25 Jul 2018 14:14:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=jgdNioMhPtvYb/4Y5cePYHLS84mObFohQBa5AJD2xSo=; b=Lsxnx5OfIhNEJpis5SdMPR3A/4 R3tk5tlrmMObrfT3SKmwNpkfIDHEgQ0ZxNEmMRuSdqzJ5J2QvVFm7TW6qLsyWM26dnD8Ki9vYfWRr +Ek2YZXFl6hqLNOibcr/VCJZrt1yfyBa3obsGdZG1PHXIxqMVILv8+HfsytW+s0/A/ENUwC81wix6 3Rvrrfa7NBrA8Y1GuHgjvAdWda0bmqtXE2Lwtamwa0kAEm2oLOFdlOCUXN74wVimMw/ocHRN/DDF4 fu6D2hqDYw7xSL5PbzhREHS7ChEqBvj+Lu+XzkklYoZ4/WZaeuNLcdR2Xy9nvwTT0+7JIi4d5EAWQ qFV8G0tA==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1fiKYF-0007OX-BF; Wed, 25 Jul 2018 14:14:03 +0000 Received: from mail-pg1-x544.google.com ([2607:f8b0:4864:20::544]) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1fiKYB-0007MI-IS for linux-arm-kernel@lists.infradead.org; Wed, 25 Jul 2018 14:14:00 +0000 Received: by mail-pg1-x544.google.com with SMTP id f1-v6so5405740pgq.12 for ; Wed, 25 Jul 2018 07:13:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=qIuwOscxl1EW4BeFkvSG6ETyVL4wwncYhTvemKPOb6Q=; b=dzwz6C7ksXXQUMV7dQjFsq+iLH4YhL4uk/iB+Bxk0woJ/mzxIeP+44zCyoQ36g30+P 6ug7/AVQxzn1Qb/Wq9Myr07U8uH4d1n5mGVHBURmbp3DM/dSZkPTXA4KAvirXXDtsrb6 a59RxONojKXjsmW98j8qJOYWW2/S63AL6zWxeZUoOuw8Vv1ITBDIHqqys1KDDg5CulUw U0vWA1AIq9Rhsiecum0dpBKgLF3nXsW/pWWdUGdJ0L/96+xh/Pkd2fJWwFOQXBvYzVnB qluufQQSbzmh+J7kD+Wl2psjbZmnNoxeALjDYjs1gk3Lxba/EBJa4WwVMfGEBa92Vfma a2ig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=qIuwOscxl1EW4BeFkvSG6ETyVL4wwncYhTvemKPOb6Q=; b=FDvamoGYn7g21MAAgTnaUi3BqsVPBAU4SteEEpEcJ3oZ6zxRE6BHurlpOVd5DcjU+b hz4M0du2LDPG0cDh/b0PGWDTEq6UnZXsgxLL6WeME0cOH6ZRLIUpjGugYOoyTzZp3nkg x/iZcckpnePpuu1Zqsm4pOXbz9ZhyHNF2ad9f6XDFoTvBMp0shRR0ly+IwbGMJQpcB90 C6cwLJSG2rWScMcbJMp9PWtB3tcCtuPeewxVEQl4bodVV3PawR0Dy8NmUHbSNShOaB+i xbNEzd3VrrRP9Z4CCfvciO7PTf874m7YLVIyWEPajeeQY9r51Wym1aUMhn6XcZl37st6 aa+g== X-Gm-Message-State: AOUpUlGTJdBqOOsguKr6Ok6d/qh5yi6UDz7R9A2wjoKooOrMSnqi/08Q Dh4u2LqXv7TYE3Eo+ogbexI= X-Google-Smtp-Source: AAOMgpfwf14R92RD5z2T+Q6XLzanWkVmVxsh3MPbqEVttaK81oTV2itdmfykO/Td2y3hCxZfnhMX+Q== X-Received: by 2002:a63:4450:: with SMTP id t16-v6mr20744377pgk.102.1532527615190; Wed, 25 Jul 2018 07:06:55 -0700 (PDT) Received: from roar.au.ibm.com ([61.69.188.107]) by smtp.gmail.com with ESMTPSA id d132-v6sm20923421pga.10.2018.07.25.07.06.52 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 25 Jul 2018 07:06:54 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org Subject: [RFC PATCH 1/4] mm: move tlb_table_flush to tlb_flush_mmu_free Date: Thu, 26 Jul 2018 00:06:38 +1000 Message-Id: <20180725140641.30372-2-npiggin@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180725140641.30372-1-npiggin@gmail.com> References: <20180725140641.30372-1-npiggin@gmail.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20180725_071359_628766_7A5F6590 X-CRM114-Status: GOOD ( 12.66 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-arm-kernel@lists.infradead.org, Nicholas Piggin MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP There is no need to call this from tlb_flush_mmu_tlbonly, it logically belongs with tlb_flush_mmu_free. This allows some code consolidation with a subsequent fix. Signed-off-by: Nicholas Piggin --- mm/memory.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 7206a634270b..bc053d5e9d41 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -245,9 +245,6 @@ static void tlb_flush_mmu_tlbonly(struct mmu_gather *tlb) tlb_flush(tlb); mmu_notifier_invalidate_range(tlb->mm, tlb->start, tlb->end); -#ifdef CONFIG_HAVE_RCU_TABLE_FREE - tlb_table_flush(tlb); -#endif __tlb_reset_range(tlb); } @@ -255,6 +252,9 @@ static void tlb_flush_mmu_free(struct mmu_gather *tlb) { struct mmu_gather_batch *batch; +#ifdef CONFIG_HAVE_RCU_TABLE_FREE + tlb_table_flush(tlb); +#endif for (batch = &tlb->local; batch && batch->nr; batch = batch->next) { free_pages_and_swap_cache(batch->pages, batch->nr); batch->nr = 0; From patchwork Wed Jul 25 14:06:39 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 10544073 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 529A3180E for ; Wed, 25 Jul 2018 14:07:34 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 412B32022C for ; Wed, 25 Jul 2018 14:07:34 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 34EEE206E2; Wed, 25 Jul 2018 14:07:34 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.4 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,DKIM_VALID,FREEMAIL_FROM,MAILING_LIST_MULTI,SUSPICIOUS_RECIPS autolearn=no version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id E88CD2022C for ; Wed, 25 Jul 2018 14:07:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=TzD6lumWC210nINDxXSJzo/ZdqP2GyTka0QdK1LnKsU=; b=dW3bb4CtSNmLDtVz5ofTq6vDxl eBMdcY48Xj+N+lETAn2yacqlZ4PLfw0EhPu0mCAVkIaSfjOpvknIgzsFWPXPp659yhUE1cp2G2ptZ Yjok32fjzt+CPC5tGLrGSYjIf6k5mzgMcZdSZN59+F31jDVd0n30tMt6uBt3pjqaLW/7C097tkrmF lcZi2wi0bfKosyq0lsE2NrlsRpKmA4lMB4+f4CeG4Yq06TnmI6ju5rAbF6lU9ywizKiJoZdAxgZrb q61vKGw77YY0i2yCPyA5i6xmz0KX0rZLJpV4NfVaI+UPCuBGnawopjrScZ5q9X3AdVG/P0I5H4NFM k/iyUMzA==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1fiKRt-0004AH-JC; Wed, 25 Jul 2018 14:07:29 +0000 Received: from mail-pl0-x244.google.com ([2607:f8b0:400e:c01::244]) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1fiKRZ-0003ps-8a for linux-arm-kernel@lists.infradead.org; Wed, 25 Jul 2018 14:07:27 +0000 Received: by mail-pl0-x244.google.com with SMTP id m1-v6so3340862plt.6 for ; Wed, 25 Jul 2018 07:06:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=7UxphlsljR+NlBYsTH8Esis+vxdkqmrjRFhGi68hhJU=; b=XYwUKztUXh78GoFkh5YY9kJxt1SDxMMuG1PsEcnsnNXBRKhePtsbFe3AnAfT9CnG3F 76/aXAEa+jKfRKzD3EGLQyJIfLNGmBxyLEU0g26pqCCN/WXJp6Lujfr1lcShyyN7SPG0 qjVS9IbZDC422uQn8tJgc3XkOhmB2XNE/iJPkLuBI8+/Rt/gEfGPYhVpzTYms+vjMJ9s QYzif4mG6CltSruM3MlDSCP5mE/r+BLYQGDtpyb/sflG1t/JHyat7zeoRQ/rC8gaq3cg onN8GAHAJnDbEckhAAw0JxwVqJKWGpbISgM0TewXOwIdXnNuIJkzCEMnnPSzUa7SNFd0 G6LQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=7UxphlsljR+NlBYsTH8Esis+vxdkqmrjRFhGi68hhJU=; b=i615YfszzZKyUNE9ba4huhKkJaZ+c2riKqahoX8ckdComs43bJko91aN4dAf+uiwzP odoz7YXdN0ZLXl5ZVg+f4yV/9RmWvum6fgaJNBbIPcBHdoub5YwN0/k4rjJIWY/GtObI mNd90se4t/2kh8XLiyNYdpFToT4fSQmDgy2uXz6xoU3Fbd2JF9gDyBTdE0StWYNZH/xY Xyt5gwAMTZ07pkq0dOuGv/XodKdNamHi8JNfYjLIfP3WjeWgzb5BcDjd+GwA8xmVfU9K 4PAH6FSArffh8ZdBV168ZbDxGetgPUwEXWj+wv9tw1a+usGuhf2YWryb29FFZKQIwnce lt/A== X-Gm-Message-State: AOUpUlEi3UpCxizdEKrOfD+KJoKDYaPsmnssmwE6x9gggpz8EKM0diIr V81Ph+ZCX+CZ/UEVRP77m8A= X-Google-Smtp-Source: AAOMgpdCqh7xbrmOwc7Mb7NkJWlmmf0rHmvB7dgvBNPoBO+OeJs5vUqKg07Ng8VHsiopNqqLXn0+RA== X-Received: by 2002:a17:902:2f43:: with SMTP id s61-v6mr20949070plb.274.1532527618840; Wed, 25 Jul 2018 07:06:58 -0700 (PDT) Received: from roar.au.ibm.com ([61.69.188.107]) by smtp.gmail.com with ESMTPSA id d132-v6sm20923421pga.10.2018.07.25.07.06.55 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 25 Jul 2018 07:06:57 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org Subject: [RFC PATCH 2/4] mm: mmu_notifier fix for tlb_end_vma Date: Thu, 26 Jul 2018 00:06:39 +1000 Message-Id: <20180725140641.30372-3-npiggin@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180725140641.30372-1-npiggin@gmail.com> References: <20180725140641.30372-1-npiggin@gmail.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20180725_070709_599196_6B0198A4 X-CRM114-Status: GOOD ( 13.50 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-arm-kernel@lists.infradead.org, Nicholas Piggin MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP The generic tlb_end_vma does not call invalidate_range mmu notifier, and it resets resets the mmu_gather range, which means the notifier won't be called on part of the range in case of an unmap that spans multiple vmas. ARM64 seems to be the only arch I could see that has notifiers and uses the generic tlb_end_vma. I have not actually tested it. --- include/asm-generic/tlb.h | 17 +++++++++++++---- mm/memory.c | 10 ---------- 2 files changed, 13 insertions(+), 14 deletions(-) diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index 3063125197ad..b3353e21f3b3 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -15,6 +15,7 @@ #ifndef _ASM_GENERIC__TLB_H #define _ASM_GENERIC__TLB_H +#include #include #include #include @@ -138,6 +139,16 @@ static inline void __tlb_reset_range(struct mmu_gather *tlb) } } +static inline void tlb_flush_mmu_tlbonly(struct mmu_gather *tlb) +{ + if (!tlb->end) + return; + + tlb_flush(tlb); + mmu_notifier_invalidate_range(tlb->mm, tlb->start, tlb->end); + __tlb_reset_range(tlb); +} + static inline void tlb_remove_page_size(struct mmu_gather *tlb, struct page *page, int page_size) { @@ -186,10 +197,8 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb, #define __tlb_end_vma(tlb, vma) \ do { \ - if (!tlb->fullmm && tlb->end) { \ - tlb_flush(tlb); \ - __tlb_reset_range(tlb); \ - } \ + if (!tlb->fullmm) \ + tlb_flush_mmu_tlbonly(tlb); \ } while (0) #ifndef tlb_end_vma diff --git a/mm/memory.c b/mm/memory.c index bc053d5e9d41..135d18b31e44 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -238,16 +238,6 @@ void arch_tlb_gather_mmu(struct mmu_gather *tlb, struct mm_struct *mm, __tlb_reset_range(tlb); } -static void tlb_flush_mmu_tlbonly(struct mmu_gather *tlb) -{ - if (!tlb->end) - return; - - tlb_flush(tlb); - mmu_notifier_invalidate_range(tlb->mm, tlb->start, tlb->end); - __tlb_reset_range(tlb); -} - static void tlb_flush_mmu_free(struct mmu_gather *tlb) { struct mmu_gather_batch *batch; From patchwork Wed Jul 25 14:06:40 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 10544075 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D00531805 for ; Wed, 25 Jul 2018 14:08:00 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BD8112A0E0 for ; Wed, 25 Jul 2018 14:08:00 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B1E022A0E3; Wed, 25 Jul 2018 14:08:00 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.4 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,DKIM_VALID,FREEMAIL_FROM,MAILING_LIST_MULTI,SUSPICIOUS_RECIPS autolearn=no version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 5CE272A0E0 for ; Wed, 25 Jul 2018 14:08:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=EfMuOistqsAlX1rj21a6hKMTCOmU1rja2ZjSzGUdq2g=; b=sYhmOpyWchlGAKs0SdnpAv9PaX b2OQV+10JX6g6wwNVrQqUJYFn1iJpTk4OtYer37mmeSJawS56S+b2vVWGsC1uPscPLDvYOmusHDac bfiLtO/WchRh8F8nAH3ynoshLKN9sTt4HevGVxFsdWmUvv3hVTYddnqtq7rW4HwUkmEICX93MslCc 6U4SnaIeA6xbX1QnLDlojM/gIhbh/Phxbt4pXheQQkB8YzM3/FbSGEC5zHxQ/m24BckluuXQflcl5 jxUNC5M/11u1KZiN0ohSQ3kNI1yN/kuSWfqCfEM4kw6BOHnS/MJSDrRP00PhOsEPFiff80qyEohnL g1xYmJ+g==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1fiKSH-0004SU-Hr; Wed, 25 Jul 2018 14:07:53 +0000 Received: from mail-pl0-x242.google.com ([2607:f8b0:400e:c01::242]) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1fiKRc-0003r2-Qt for linux-arm-kernel@lists.infradead.org; Wed, 25 Jul 2018 14:07:28 +0000 Received: by mail-pl0-x242.google.com with SMTP id 6-v6so3360095plb.0 for ; Wed, 25 Jul 2018 07:07:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=/jOJ/+W2JSaRGc382cEHsBJ4YMQ1luV4W/5/qP+a/jo=; b=AVI4jmNYjgqDSMI3WJjS/MJXnTw/tSu8EiFLIssrPgdZhTHZLaKvPOCoEpnbETjqS4 WHjd/x8OQXdeOlIJZUbPRVxOpMGMwrcDzsTehdv1ptJRS1/qcrW9aF1kTRIh0XLb0sib XV1qaHllNNEDcAI65jnDZZ7P3MreQAr0+cZP1hvfEyu+KAC9ZUfhPnMW50xPszAp0y3H C9/m8i+9xcJCwcTacvN8SqAkzeONsqUQeD9sYBf05f2IZuZh+RGeze6vtqjx0oYHku1A l/GAg75BAYlWzv78239VYONTyVhDgOtMQjcHuhGA10RhHYWoC4q1YscpehOQ/d52IXnC LYfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=/jOJ/+W2JSaRGc382cEHsBJ4YMQ1luV4W/5/qP+a/jo=; b=sW1H3qme47K5BEH2MiKyxlDWMq1IrjgIimPiyyY3t2mxJfKNcguCOtOvoa2HSED7e1 HwIhuKt+7Qrj/94+sMMSziCvybtEhwELqW978lvmn29gGs97xJ/+SstWaBbkYpi6Z7YH Ty14pzMpTsvu0XEDSmtUiu8tuEAgh17xIRfizzSK0j3lGod20F28Dw1+Hjgfmqcx1i+W olvZ/ww/xKMJwwy758eU4AIuyyu4uX0p5qG+Igm4vlN7lsx58Z+puG3tTdaGw8Atzu/O xnaNW3aqFvjRxFhwMGxZW3c9C55VhBBjYUGKoCWDn6UiH2VMzEWuG/7/7n48AaUBD/fA AwFg== X-Gm-Message-State: AOUpUlEaaILmIxFu7MSfIPuCrxH4b/CV6Kp1awkbOwMfMwnRyxA+KGtW CZsBbuZF0YphJRKnryoyE8g= X-Google-Smtp-Source: AAOMgpdxnMOTaEEmTL31SXnAKEmdYiRjkCxG7VwMPoXvmtNuKHYh+0w2cwP9DLIdwrLmpng7F58Sfw== X-Received: by 2002:a17:902:8697:: with SMTP id g23-v6mr21137522plo.292.1532527622385; Wed, 25 Jul 2018 07:07:02 -0700 (PDT) Received: from roar.au.ibm.com ([61.69.188.107]) by smtp.gmail.com with ESMTPSA id d132-v6sm20923421pga.10.2018.07.25.07.06.59 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 25 Jul 2018 07:07:01 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org Subject: [RFC PATCH 3/4] mm: allow arch to have tlb_flush caled on an empty TLB range Date: Thu, 26 Jul 2018 00:06:40 +1000 Message-Id: <20180725140641.30372-4-npiggin@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180725140641.30372-1-npiggin@gmail.com> References: <20180725140641.30372-1-npiggin@gmail.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20180725_070713_405552_C9159072 X-CRM114-Status: GOOD ( 13.67 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-arm-kernel@lists.infradead.org, Nicholas Piggin MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP powerpc wants to de-couple page table caching structure flushes from TLB flushes, which will make it possible to have mmu_gather with freed page table pages but no TLB range. These must be sent to tlb_flush, so allow the arch to specify when mmu_gather with empty ranges should have tlb_flush called. Signed-off-by: Nicholas Piggin --- include/asm-generic/tlb.h | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index b3353e21f3b3..b320c0cc8996 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -139,14 +139,27 @@ static inline void __tlb_reset_range(struct mmu_gather *tlb) } } +/* + * arch_tlb_mustflush specifies if tlb_flush is to be called even if the + * TLB range is empty (this can be the case for freeing page table pages + * if the arch does not adjust TLB range to cover them). + */ +#ifndef arch_tlb_mustflush +#define arch_tlb_mustflush(tlb) false +#endif + static inline void tlb_flush_mmu_tlbonly(struct mmu_gather *tlb) { - if (!tlb->end) + unsigned long start = tlb->start; + unsigned long end = tlb->end; + + if (!(end || arch_tlb_mustflush(tlb))) return; tlb_flush(tlb); - mmu_notifier_invalidate_range(tlb->mm, tlb->start, tlb->end); __tlb_reset_range(tlb); + if (end) + mmu_notifier_invalidate_range(tlb->mm, start, end); } static inline void tlb_remove_page_size(struct mmu_gather *tlb, From patchwork Wed Jul 25 14:06:41 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 10544077 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 269AB9093 for ; Wed, 25 Jul 2018 14:13:28 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 130CF2A0A9 for ; Wed, 25 Jul 2018 14:13:28 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 069612A0B5; Wed, 25 Jul 2018 14:13:28 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.4 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,DKIM_VALID,FREEMAIL_FROM,MAILING_LIST_MULTI,SUSPICIOUS_RECIPS autolearn=no version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 93C912A0AA for ; Wed, 25 Jul 2018 14:13:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=bSf7QPgWBhNorjlkpK7SvaRTche44kQTXXIR1rFhxlU=; b=VIUJzi50N2CHmf9f9u6Hrme8Em HEJM+icuu16Pmnf24Gno85a7GA1NSvGcZ1zEBeyFI+ZRdpbNF0ddAGjL3syAd61zRmsqTQbyujbql N9B8nJS7LbJGLPAyfboQ96vGaNYBnFXm0tk+WB80FXPhxkBi3B7H5rNc5g5ty1Fm19jUda/9g2qu9 MW1YJeBNQ/xrmFmVvauBCq8e71J6lGp2uL6hi0zvIs6X5z6NVppwRSOdz/BdO+5UZadel4DyZLmRx bRoVa7x4tWczOLSeb3RzM4BdxnlGpmbUU0SF3nEPGq7j0UMkn4ZYTcby1Ly9YHzU1SK0flufY2XQZ mPl0UPvw==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1fiKXa-00071l-49; Wed, 25 Jul 2018 14:13:22 +0000 Received: from merlin.infradead.org ([2001:8b0:10b:1231::1]) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1fiKXY-00071G-32 for linux-arm-kernel@bombadil.infradead.org; Wed, 25 Jul 2018 14:13:20 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=merlin.20170209; h=References:In-Reply-To:Message-Id:Date: Subject:Cc:To:From:Sender:Reply-To:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=bEa2VC1PhTklj7TrxhDQMGrxkMdS1Gq+3HFfcqOkvaE=; b=yzXyb5IbukMpd1bXdec06lZ5B E5xJ9srLxOcfIGs+6ZoFQfRO7uWO08DP2W+WdeyqXaRkzK41RWS0z0kH+s3bnXBNx6Cd/0mekRqOO /CGqg61qPKAG7dHQgfSJmW/xLQI70smaHrQWuSE3WXRNMs2oqAwjzO3Iy1LVBm0KzXI8hSxkFDWn/ LcD8XeaoWFFEl9kvvCqs1C9wCFGdmO0QA1jrtJWy9BgFxnv5KgYKMYmmAlC979aLq1CiVUcIV/wMn fkkxcvjaoWAUD+yQ0t1wb8jTWluKjnutr46ZivCvdq5AvhwxJ0lLQNqqIEagDmroMJuifTuRTQh85 jyDcJBcqw==; Received: from mail-pg1-x542.google.com ([2607:f8b0:4864:20::542]) by merlin.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1fiKXU-0002ZP-Uz for linux-arm-kernel@lists.infradead.org; Wed, 25 Jul 2018 14:13:18 +0000 Received: by mail-pg1-x542.google.com with SMTP id e6-v6so5418998pgv.2 for ; Wed, 25 Jul 2018 07:13:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=bEa2VC1PhTklj7TrxhDQMGrxkMdS1Gq+3HFfcqOkvaE=; b=s/oaPYvUXPdAEv1dgAf1aUK8hLBsA4DKGN5fR0/nvxAdQrOg3/KjjiNtDOA7QaQXWz NEzTiowWzU3LAgvm7OI0XcufZkXEoHCS6LVDtWDQvm18bpOnMwPHS52W4Qso3tHkoyi5 B8iSqrksRtRCUClCWEVLQGu7XgPhGCEREcWB8BIKl0cm4A2dJig9mDzc78FAgmxeSQ5c ic7WTfiRxTcuCnUU9DfhJxF/CiyT/PYxg80ATjMSUXRcnQdbkDn3u+352F83n3lZq342 /IGaDM9kJ27wVD5LcdiBh5wbGEtAdF1CVwRIiD1Hk89zzAJK9ARB+44ObGUaSgoxkWDZ aKgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=bEa2VC1PhTklj7TrxhDQMGrxkMdS1Gq+3HFfcqOkvaE=; b=iHxinQlOqoZBbn5Z3lx/QNtMZUcvRu2wmTrZC+WK1Wzs/2ikSxNSpzLvbIy5bEwe4d Jet6wIFTh5sokm21zVGNM6CUeqi1OxZvE7yHBp/yuE+pOPntzbc9MbxUWpN3ZsWJEzci H4Lh5BcVtbGo8RIXJEuJ2MFu19g/fyA0vk5dbp5gQf50dcNANYM3LLJSPZQjaKShh1Ck Ym51oSo+IjFmdI/5oBYQprGqkPQOBF9O6slcbMpc2/2IVbv3Tm8pg2b04b0l2an3tWZa dl5ScVFAmNv8FpUy/MlX6Z3kFiFbsNiXruqDSLbLPUaa/KP3hZwlERHGwyPM5bujgXO3 0Z1g== X-Gm-Message-State: AOUpUlEiq1YCujf+B7S8d9J9a1bJ7pc24Ru6w9+rgXhUdCCYiE02zUb1 bQlHMevh0H21Bv95eMP8wUk= X-Google-Smtp-Source: AAOMgpcAXooHLfepiFNZvQ1EZXOE1oPa36WafIdHB3mJLvOw0mOJ8azm5i49n1gtoOVCew/36zGZlA== X-Received: by 2002:a65:660a:: with SMTP id w10-v6mr20323698pgv.366.1532527626243; Wed, 25 Jul 2018 07:07:06 -0700 (PDT) Received: from roar.au.ibm.com ([61.69.188.107]) by smtp.gmail.com with ESMTPSA id d132-v6sm20923421pga.10.2018.07.25.07.07.02 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 25 Jul 2018 07:07:05 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org Subject: [RFC PATCH 4/4] powerpc/64s/radix: optimise TLB flush with precise TLB ranges in mmu_gather Date: Thu, 26 Jul 2018 00:06:41 +1000 Message-Id: <20180725140641.30372-5-npiggin@gmail.com> X-Mailer: git-send-email 2.17.0 In-Reply-To: <20180725140641.30372-1-npiggin@gmail.com> References: <20180725140641.30372-1-npiggin@gmail.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20180725_101317_105709_A51AE932 X-CRM114-Status: GOOD ( 20.16 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-arm-kernel@lists.infradead.org, Nicholas Piggin MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP The mmu_gather APIs keep track of the invalidated address range, and the generic page table freeing accessors expand the invalidated range to cover the addresses corresponding to the page tables even if there are no ptes and therefore no TLB entries to invalidate. This is done for architectures that have paging structure caches that are invalidated with their TLB invalidate instructions (e.g., x86). powerpc/64s/radix does have a "page walk cache" (PWC), but it is invalidated with a specific instruction and tracked independently in the mmu_gather (using the need_flush_all flag to indicate PWC must be flushed). Therefore TLB invalidation does not have to be expanded to cover freed page tables. This patch defines p??_free_tlb functions for 64s, which do not expand the TLB flush range over page table pages. This brings the number of tlbiel instructions required by a kernel compile from 33M to 25M, most avoided from exec => shift_arg_pages(). Signed-off-by: Nicholas Piggin --- arch/powerpc/include/asm/tlb.h | 34 ++++++++++++++++++++++++++++++++++ arch/powerpc/mm/tlb-radix.c | 10 ++++++++++ include/asm-generic/tlb.h | 5 +++++ 3 files changed, 49 insertions(+) diff --git a/arch/powerpc/include/asm/tlb.h b/arch/powerpc/include/asm/tlb.h index 9138baccebb0..5d3107f2b014 100644 --- a/arch/powerpc/include/asm/tlb.h +++ b/arch/powerpc/include/asm/tlb.h @@ -30,6 +30,40 @@ #define __tlb_remove_tlb_entry __tlb_remove_tlb_entry #define tlb_remove_check_page_size_change tlb_remove_check_page_size_change +#ifdef CONFIG_PPC_BOOK3S_64 +/* + * powerpc book3s hash does not have page table structure caches, and + * radix requires explicit management with PWC invalidate tlb type, so + * there is no need to expand the mmu_gather range over invalidated page + * table pages like the generic code does. + */ + +#define pte_free_tlb(tlb, ptep, address) \ + do { \ + __pte_free_tlb(tlb, ptep, address); \ + } while (0) + +#define pmd_free_tlb(tlb, pmdp, address) \ + do { \ + __pmd_free_tlb(tlb, pmdp, address); \ + } while (0) + +#define pud_free_tlb(tlb, pudp, address) \ + do { \ + __pud_free_tlb(tlb, pudp, address); \ + } while (0) + +/* + * Radix sets need_flush_all when page table pages have been unmapped + * and the PWC needs flushing. Generic code must call our tlb_flush + * even on empty ranges in this case. + * + * This will always be false for hash. + */ +#define arch_tlb_mustflush(tlb) (tlb->need_flush_all) + +#endif + extern void tlb_flush(struct mmu_gather *tlb); /* Get the generic bits... */ diff --git a/arch/powerpc/mm/tlb-radix.c b/arch/powerpc/mm/tlb-radix.c index 1135b43a597c..238b20a513e7 100644 --- a/arch/powerpc/mm/tlb-radix.c +++ b/arch/powerpc/mm/tlb-radix.c @@ -862,6 +862,16 @@ void radix__tlb_flush(struct mmu_gather *tlb) unsigned long start = tlb->start; unsigned long end = tlb->end; + /* + * This can happen if need_flush_all is set due to a page table + * invalidate, but no ptes under it freed (see arch_tlb_mustflush). + * Set end = start to prevent any TLB flushing here (only PWC). + */ + if (!end) { + WARN_ON_ONCE(!tlb->need_flush_all); + end = start; + } + /* * if page size is not something we understand, do a full mm flush * diff --git a/include/asm-generic/tlb.h b/include/asm-generic/tlb.h index b320c0cc8996..a55ef1425f0d 100644 --- a/include/asm-generic/tlb.h +++ b/include/asm-generic/tlb.h @@ -285,6 +285,11 @@ static inline void tlb_remove_check_page_size_change(struct mmu_gather *tlb, * http://lkml.kernel.org/r/CA+55aFzBggoXtNXQeng5d_mRoDnaMBE5Y+URs+PHR67nUpMtaw@mail.gmail.com * * For now w.r.t page table cache, mark the range_size as PAGE_SIZE + * + * Update: powerpc (Book3S 64-bit, radix MMU) has an architected page table + * cache (called PWC), and invalidates it specifically. It sets the + * need_flush_all flag to indicate the PWC requires flushing, so it defines + * its own p??_free_tlb functions which do not expand the TLB range. */ #ifndef pte_free_tlb