From patchwork Wed Jun 6 07:01:21 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Chintan Pandya X-Patchwork-Id: 10449751 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 75E8C60467 for ; Wed, 6 Jun 2018 07:02:50 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3960D29878 for ; Wed, 6 Jun 2018 07:02:49 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 36E09298EB; Wed, 6 Jun 2018 07:02:49 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,MAILING_LIST_MULTI autolearn=unavailable version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 077E72994A for ; Wed, 6 Jun 2018 07:02:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=UDmdToD3tB2HfQlb03jryV/+6TEHkGIVZD4CzeC9WDw=; b=CJNtC/kiamvZIMmaZFtJrkkdw4 gsaNXjACb7smiaLuqCS4NC3gjJAEQO5B05ywH5EorJ31C4H7KFieGoskyXjNQeZu0kpDlunSMQthC yhzogIQm7eq+JNS1YQvTDf8QXa1kXfsLOqdvPEmrRBUUgCKyljHWu3TcGnEQH3o1KtRwV/CR/fBg0 BTb/z+kWBROKEdmkDmtWKsYA1uxwwugWxaElCEDSCPHTzgG8cgo3O4+SCg5F+Uh2g2/elpVftl7DF 1e5POlpnSOlMmSqJah2GgHN0hm8vF3r13PK2KI6ff3gWQfWTYE02XifVJaIZOUWBF8+w3kfQcdS3Q /ZDaDUDQ==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1fQSSt-00008A-Sp; Wed, 06 Jun 2018 07:02:39 +0000 Received: from smtp.codeaurora.org ([198.145.29.96]) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1fQSSC-00080X-7J for linux-arm-kernel@lists.infradead.org; Wed, 06 Jun 2018 07:02:05 +0000 Received: by smtp.codeaurora.org (Postfix, from userid 1000) id 7B98560860; Wed, 6 Jun 2018 07:01:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=codeaurora.org; s=default; t=1528268508; bh=P2K1eN8Aa4WX+7TfFHFmMjm2xarj/+V2t44oaJEKkVA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=XhJK+bbDsoDoO4Z/MnGvWPgpHDk1FFLWDydW/JGAo4GxuuOP96N8oeu9T2MMhGZYS GboTDRpcW2KYgGW0187ATFmd8Ziki+NBabEp3T8QhfecONX2akvWwBwsUvSx7oQKbJ 1R8z/k/C6dakAtMt/APn9TtPEiSnV4vovkMKP/+I= Received: from cpandya-linux.qualcomm.com (blr-c-bdr-fw-01_globalnat_allzones-outside.qualcomm.com [103.229.19.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: cpandya@smtp.codeaurora.org) by smtp.codeaurora.org (Postfix) with ESMTPSA id 49B21607C7; Wed, 6 Jun 2018 07:01:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=codeaurora.org; s=default; t=1528268507; bh=P2K1eN8Aa4WX+7TfFHFmMjm2xarj/+V2t44oaJEKkVA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=jNbwaeIOQTSs2Rodnb7NUgKv6HK5QXt17i8hil1+/jUzYIDSpYNFjRPxFKiHdzTE/ j2mNv3gPU/VbUsxtvmGKWRZWFlkcNOsqWcnz8urs8l3r/IgvyjAGMA0GjngnyLd6JR x77rRVF8DW8oSWy3jqkb9eLkgTT0NWsW9PYkndKU= DMARC-Filter: OpenDMARC Filter v1.3.2 smtp.codeaurora.org 49B21607C7 Authentication-Results: pdx-caf-mail.web.codeaurora.org; dmarc=none (p=none dis=none) header.from=codeaurora.org Authentication-Results: pdx-caf-mail.web.codeaurora.org; spf=none smtp.mailfrom=cpandya@codeaurora.org From: Chintan Pandya To: will.deacon@arm.com, catalin.marinas@arm.com, mark.rutland@arm.com, akpm@linux-foundation.org Subject: [PATCH v13 3/3] arm64: Implement page table free interfaces Date: Wed, 6 Jun 2018 12:31:21 +0530 Message-Id: <1528268481-19299-4-git-send-email-cpandya@codeaurora.org> X-Mailer: git-send-email 1.9.1 In-Reply-To: <1528268481-19299-1-git-send-email-cpandya@codeaurora.org> References: <1528268481-19299-1-git-send-email-cpandya@codeaurora.org> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20180606_000156_433505_32F4655F X-CRM114-Status: GOOD ( 12.98 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Chintan Pandya , linux-arm-kernel@lists.infradead.org, toshi.kani@hpe.com, linux-kernel@vger.kernel.org MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP arm64 requires break-before-make. Originally, before setting up new pmd/pud entry for huge mapping, in few cases, the modifying pmd/pud entry was still valid and pointing to next level page table as we only clear off leaf PTE in unmap leg. a) This was resulting into stale entry in TLBs (as few TLBs also cache intermediate mapping for performance reasons) b) Also, modifying pmd/pud was the only reference to next level page table and it was getting lost without freeing it. So, page leaks were happening. Implement pud_free_pmd_page() and pmd_free_pte_page() to enforce BBM and also free the leaking page tables. Implementation requires, 1) Clearing off the current pud/pmd entry 2) Invalidation of TLB 3) Freeing of the un-used next level page tables Signed-off-by: Chintan Pandya Reviewed-by: Will Deacon --- arch/arm64/mm/mmu.c | 48 ++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 44 insertions(+), 4 deletions(-) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 8ae5d7a..65f8627 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -45,6 +45,7 @@ #include #include #include +#include #define NO_BLOCK_MAPPINGS BIT(0) #define NO_CONT_MAPPINGS BIT(1) @@ -977,12 +978,51 @@ int pmd_clear_huge(pmd_t *pmdp) return 1; } -int pud_free_pmd_page(pud_t *pud, unsigned long addr) +int pmd_free_pte_page(pmd_t *pmdp, unsigned long addr) { - return pud_none(*pud); + pte_t *table; + pmd_t pmd; + + pmd = READ_ONCE(*pmdp); + + /* No-op for empty entry and WARN_ON for valid entry */ + if (!pmd_present(pmd) || !pmd_table(pmd)) { + VM_WARN_ON(!pmd_table(pmd)); + return 1; + } + + table = pte_offset_kernel(pmdp, addr); + pmd_clear(pmdp); + __flush_tlb_kernel_pgtable(addr); + pte_free_kernel(NULL, table); + return 1; } -int pmd_free_pte_page(pmd_t *pmd, unsigned long addr) +int pud_free_pmd_page(pud_t *pudp, unsigned long addr) { - return pmd_none(*pmd); + pmd_t *table; + pmd_t *pmdp; + pud_t pud; + unsigned long next, end; + + pud = READ_ONCE(*pudp); + + /* No-op for empty entry and WARN_ON for valid entry */ + if (!pud_present(pud) || !pud_table(pud)) { + VM_WARN_ON(!pud_table(pud)); + return 1; + } + + table = pmd_offset(pudp, addr); + pmdp = table; + next = addr; + end = addr + PUD_SIZE; + do { + pmd_free_pte_page(pmdp, next); + } while (pmdp++, next += PMD_SIZE, next != end); + + pud_clear(pudp); + __flush_tlb_kernel_pgtable(addr); + pmd_free(NULL, table); + return 1; }