From patchwork Wed Mar 14 08:48:23 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chintan Pandya X-Patchwork-Id: 10281663 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 84CE0602C2 for ; Wed, 14 Mar 2018 08:50:19 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7547328791 for ; Wed, 14 Mar 2018 08:50:19 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6968328794; Wed, 14 Mar 2018 08:50:19 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID autolearn=unavailable version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id D857F28791 for ; Wed, 14 Mar 2018 08:50:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=8AUTuC+gv1403cmOGxH8rJDOdq/avUwgbLZAFFZbeKE=; b=jWajOy6qXsiily/GxfK8V6ifAT 94nr+Fe9Qf62fkLwIMGX8Wftll1Wlajz2UwQBSQQJyHgOkAAWV6RuXDBnQ7VC1jqUncIAreuetf2L tAod7chg7fo6ZIF5ZLKeDkht8+yosokKaR9s0uMr9RoK0UWLREgBv6Gr2Sld+3ds4Fi+BbgyqM6PM HSVkyas/2BekLGRJc0shzNfk8qeXwSIEz6H6LadAC/lhE+jngfPwE1eoWqmFKSiY1OoVlwJn11+Ou PqgrSjFxuLJ9TJ02IyTUT8RE8rjODdSud01xAMGGIznePMU9KSGVS3xzHhd86yuql5RL+L+j5zjme b3EKT2VQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1ew26t-0004vX-D8; Wed, 14 Mar 2018 08:50:11 +0000 Received: from smtp.codeaurora.org ([198.145.29.96]) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1ew25m-0003t8-Gt for linux-arm-kernel@lists.infradead.org; Wed, 14 Mar 2018 08:49:10 +0000 Received: by smtp.codeaurora.org (Postfix, from userid 1000) id 1260B608BF; Wed, 14 Mar 2018 08:48:52 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=codeaurora.org; s=default; t=1521017332; bh=0cbaYAReSctfUKsE9ry3OLwsRwLHi76ohNnH7VdlyXA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=b+288mJzf3AksEepNAiC+59gdA93SWkqUsKkQ9TpPcoqAiIEtyIf0BcziDwB/ovT7 V4PEgmNStcB0mNsGxXfQIM5urFxB2xtAX4lZssvdvK/2VC46Sw7QXaZaFK26m+sxJR PUlcsuy5O8WcXCIffaSuE6ymX0n8al7CR2Ulx1eA= Received: from cpandya-linux.qualcomm.com (blr-c-bdr-fw-01_globalnat_allzones-outside.qualcomm.com [103.229.19.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: cpandya@smtp.codeaurora.org) by smtp.codeaurora.org (Postfix) with ESMTPSA id 877C3608CB; Wed, 14 Mar 2018 08:48:46 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=codeaurora.org; s=default; t=1521017331; bh=0cbaYAReSctfUKsE9ry3OLwsRwLHi76ohNnH7VdlyXA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=YczjPr3lQcLJmtqrTfjqq+nx4hRDOxkl9mhaTITykULOfH4lo14nTyaPFT6XxfyN+ XtN/ZbGRUF/i5OxEn2Ly7qrFJZ0hHtuS2NzN5n97RgZD7ZkXXHAz7pr2QRPWUsWM7n va1MpsLAcdZtlHQTsoGDJkppqf5QHHEXWTj3MSv8= DMARC-Filter: OpenDMARC Filter v1.3.2 smtp.codeaurora.org 877C3608CB Authentication-Results: pdx-caf-mail.web.codeaurora.org; dmarc=none (p=none dis=none) header.from=codeaurora.org Authentication-Results: pdx-caf-mail.web.codeaurora.org; spf=none smtp.mailfrom=cpandya@codeaurora.org From: Chintan Pandya To: catalin.marinas@arm.com, will.deacon@arm.com, arnd@arndb.de Subject: [PATCH v1 2/4] ioremap: Invalidate TLB after huge mappings Date: Wed, 14 Mar 2018 14:18:23 +0530 Message-Id: <1521017305-28518-3-git-send-email-cpandya@codeaurora.org> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1521017305-28518-1-git-send-email-cpandya@codeaurora.org> References: <1521017305-28518-1-git-send-email-cpandya@codeaurora.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20180314_014903_248000_BA778422 X-CRM114-Status: GOOD ( 14.06 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, linux-arch@vger.kernel.org, toshi.kani@hpe.com, ard.biesheuvel@linaro.org, marc.zyngier@arm.com, gregkh@linuxfoundation.org, linux-kernel@vger.kernel.org, kristina.martsenko@arm.com, takahiro.akashi@linaro.org, james.morse@arm.com, Chintan Pandya , tglx@linutronix.de, akpm@linux-foundation.org, linux-arm-kernel@lists.infradead.org MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP If huge mappings are enabled, they can override valid intermediate previous mappings. Some MMU can speculatively pre-fetch these intermediate entries even after unmap. That's because unmap will clear only last level entries in page table keeping intermediate (pud/pmd) entries still valid. This can potentially lead to stale TLB entries which needs invalidation after map. Some more info: https://lkml.org/lkml/2017/12/23/3 There is one noted case for ARM64 where such stale TLB entries causes 3rd level translation fault even after correct (huge) mapping is available. See the case below (reproduced locally with tests), [17505.330123] Unable to handle kernel paging request at virtual address ffffff801ae00000 [17505.338100] pgd = ffffff800a761000 [17505.341566] [ffffff801ae00000] *pgd=000000017e1be003, *pud=000000017e1be003, *pmd=00e8000098000f05 [17505.350704] ------------[ cut here ]------------ [17505.355362] Kernel BUG at ffffff8008238c30 [verbose debug info unavailable] [17505.362375] Internal error: Oops: 96000007 [#1] PREEMPT SMP [17505.367996] Modules linked in: [17505.371114] CPU: 6 PID: 488 Comm: chintan-ioremap Not tainted 4.9.81+ #160 [17505.378039] Hardware name: Qualcomm Technologies, Inc. SDM845 v1 MTP (DT) [17505.384885] task: ffffffc0e3e61180 task.stack: ffffffc0e3e70000 [17505.390868] PC is at io_remap_test+0x2e0/0x444 [17505.395352] LR is at io_remap_test+0x2d0/0x444 [17505.399835] pc : [] lr : [] pstate: 60c00005 [17505.407282] sp : ffffffc0e3e73d70 [17505.410624] x29: ffffffc0e3e73d70 x28: ffffff801ae00008 [17505.416031] x27: ffffff801ae00010 x26: ffffff801ae00018 [17505.421436] x25: ffffff801ae00020 x24: ffffff801adfffe0 [17505.426840] x23: ffffff801adfffe8 x22: ffffff801adffff0 [17505.432244] x21: ffffff801adffff8 x20: ffffff801ae00000 [17505.437648] x19: 0000000000000005 x18: 0000000000000000 [17505.443052] x17: 00000000b3409452 x16: 00000000923da470 [17505.448456] x15: 0000000071c9763c x14: 00000000a15658fa [17505.453860] x13: 000000005cae96bf x12: 00000000e6d5c44a [17505.459264] x11: 0140000000000000 x10: ffffff80099a1000 [17505.464668] x9 : 0000000000000000 x8 : ffffffc0e3e73d68 [17505.470072] x7 : ffffff80099d3220 x6 : 0000000000000015 [17505.475476] x5 : 00000c00004ad32a x4 : 000000000000000a [17505.480880] x3 : 000000000682aaab x2 : 0000001345c2ad2e [17505.486284] x1 : 7d78d61de56639ba x0 : 0000000000000001 Hence, invalidate once we override pmd/pud with huge mappings. Signed-off-by: Chintan Pandya --- lib/ioremap.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/lib/ioremap.c b/lib/ioremap.c index b808a39..c1e1341 100644 --- a/lib/ioremap.c +++ b/lib/ioremap.c @@ -13,6 +13,7 @@ #include #include #include +#include #ifdef CONFIG_HAVE_ARCH_HUGE_VMAP static int __read_mostly ioremap_p4d_capable; @@ -92,8 +93,10 @@ static inline int ioremap_pmd_range(pud_t *pud, unsigned long addr, if (ioremap_pmd_enabled() && ((next - addr) == PMD_SIZE) && IS_ALIGNED(phys_addr + addr, PMD_SIZE)) { - if (pmd_set_huge(pmd, phys_addr + addr, prot)) + if (pmd_set_huge(pmd, phys_addr + addr, prot)) { + flush_tlb_pgtable(&init_mm, addr); continue; + } } if (ioremap_pte_range(pmd, addr, next, phys_addr + addr, prot)) @@ -118,8 +121,10 @@ static inline int ioremap_pud_range(p4d_t *p4d, unsigned long addr, if (ioremap_pud_enabled() && ((next - addr) == PUD_SIZE) && IS_ALIGNED(phys_addr + addr, PUD_SIZE)) { - if (pud_set_huge(pud, phys_addr + addr, prot)) + if (pud_set_huge(pud, phys_addr + addr, prot)) { + flush_tlb_pgtable(&init_mm, addr); continue; + } } if (ioremap_pmd_range(pud, addr, next, phys_addr + addr, prot))