From patchwork Tue Mar 12 00:57:46 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 10848451 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EA0C76C2 for ; Tue, 12 Mar 2019 00:58:20 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D41D229409 for ; Tue, 12 Mar 2019 00:58:20 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D20DA29415; Tue, 12 Mar 2019 00:58:20 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 5C6FC29409 for ; Tue, 12 Mar 2019 00:58:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version :Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=/XIuhMMDjeaJkH5oNB6S7JH3wjHpHs+JTud/Mc1cpws=; b=R8h6EBYzXiDa6z D4a8e04+ZaW+t1SI+cZHwDTwIcuSlSgpPuLPl88lzoJSm1Y+mYmcaagT0NlxI9DGlikgRbryL1hC6 ADnlRog1XKYQCKwDkxBJBEvq5fofaSDXtdg8ItyXz5WzEZEqQmmDbnyk2A0of/GnlAxWdzBSygnuc pMtKcrA7KHGtdKN5tZpzFsF2LrFd3x2FvlVOygij9VX6FwBTHo/LSahIB7EZyEwFBO9KBS7zpMP4J HO0i9YLsVSM6jC8bw2Q9T/xJOAVs4F6bNatsu5OzBUp9ul1fzxbjMIsCFh4BpDzkC9HwQOvjQZg3y vCXG4NnDTNNE8ByBzG0w==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1h3VkH-0003dv-U9; Tue, 12 Mar 2019 00:58:17 +0000 Received: from mail-yw1-xc4a.google.com ([2607:f8b0:4864:20::c4a]) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1h3Vk8-0003Ui-H1 for linux-arm-kernel@lists.infradead.org; Tue, 12 Mar 2019 00:58:10 +0000 Received: by mail-yw1-xc4a.google.com with SMTP id o66so913483ywc.3 for ; Mon, 11 Mar 2019 17:58:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=0Yb2ivGv5k2eEZFqDaeb2AMWK/OeRIAqkI0UIANBdSk=; b=YtoTCa1FsAa4SxSSy2t1RCEeLGZsCq3F9MmC82/HiNbwzc/7heOZhhMADajPy01n9b agSe/kw18UTudBiU7ksU8/SHwvlZlPyV9LhN6fTTgC8Pm6CxdX5WamzES1EemO0KXy/j decP+RthgSoxH0b7lDx3YgrKtEniFzHFTV9YD3AosSfoHuoGpJSZcij+2TA2ZsJbnnyd To+zhj0MEY+b7krt7yyKqMyeZJG/PoA+A8X8hCbGrS2l76904BOZGQ2g93sMN7LaVVvu yc2xlRSPhv8JbiOW+JO7nL/y8uBBxIr5HeyVp0+8N+nq2xCyLduG0F8nOgQDZH2I33Hs 22eQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=0Yb2ivGv5k2eEZFqDaeb2AMWK/OeRIAqkI0UIANBdSk=; b=dvWKykCHw6I0k9gZ3F/vj5ZWaXxno8pJK06TKLfy3jyD1RVbLGUwxgQOGqE8N+Pndv CTcxZGcFgHsrTfaCkwhI+wWXWIMtrMPtZMDq5cLACLZBN6bWfpJ7mC9hu27XsBK5ECZm HNasij84Rzpc5ZXX+ni9OWyfNKNgb/wlOQzK3LBGClPEAap71A4MB8OZnw6aOfpA3Fx5 3wn3kTfLSgkVvblpIHBsfDF2IIf7wb2vAr+LNkx7qI3VQcZ9amDPgJasPPoaLztxEkCD +2DnkQYiRemH9uEMRt4Xeg5cBPJcpzqwcj0e4WyDnr/EekMvCs8K4qaiUDr91flI7zGI S1QA== X-Gm-Message-State: APjAAAVsUR0Vc4Wkf9QU2x2HHKgXyU29IsUF6H3JuBNk8x5S/7wEhv0U lv42/DDPyMGrHehRN1tokF96roUUG4o= X-Google-Smtp-Source: APXvYqw+ll+HDtKzYX+8qYICIA45HG4qpRTQvEyf0AUjyPzgDXuQrmIOJztXx4aVqQDr6yzeNnT78JHSF9Y= X-Received: by 2002:a25:949:: with SMTP id u9mr15976942ybm.8.1552352284196; Mon, 11 Mar 2019 17:58:04 -0700 (PDT) Date: Mon, 11 Mar 2019 18:57:46 -0600 In-Reply-To: <20190310011906.254635-1-yuzhao@google.com> Message-Id: <20190312005749.30166-1-yuzhao@google.com> Mime-Version: 1.0 References: <20190310011906.254635-1-yuzhao@google.com> X-Mailer: git-send-email 2.21.0.360.g471c308f928-goog Subject: [PATCH v4 1/4] arm64: mm: use appropriate ctors for page tables From: Yu Zhao To: Catalin Marinas , Will Deacon , Mark Rutland X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190311_175808_560334_8798AB85 X-CRM114-Status: GOOD ( 12.64 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch@vger.kernel.org, Yu Zhao , Ard Biesheuvel , Peter Zijlstra , linux-kernel@vger.kernel.org, Nick Piggin , Jun Yao , linux-mm@kvack.org, "Aneesh Kumar K . V" , Chintan Pandya , Joel Fernandes , "Kirill A . Shutemov" , Andrew Morton , Laura Abbott , linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP For pte page, use pgtable_page_ctor(); for pmd page, use pgtable_pmd_page_ctor(); and for the rest (pud, p4d and pgd), don't use any. For now, we don't select ARCH_ENABLE_SPLIT_PMD_PTLOCK and pgtable_pmd_page_ctor() is a nop. When we do in patch 3, we make sure pmd is not folded so we won't mistakenly call pgtable_pmd_page_ctor() on pud or p4d. Acked-by: Mark Rutland Signed-off-by: Yu Zhao --- arch/arm64/mm/mmu.c | 36 ++++++++++++++++++++++++------------ 1 file changed, 24 insertions(+), 12 deletions(-) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index b6f5aa52ac67..f704b291f2c5 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -98,7 +98,7 @@ pgprot_t phys_mem_access_prot(struct file *file, unsigned long pfn, } EXPORT_SYMBOL(phys_mem_access_prot); -static phys_addr_t __init early_pgtable_alloc(void) +static phys_addr_t __init early_pgtable_alloc(int shift) { phys_addr_t phys; void *ptr; @@ -173,7 +173,7 @@ static void init_pte(pmd_t *pmdp, unsigned long addr, unsigned long end, static void alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, unsigned long end, phys_addr_t phys, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(void), + phys_addr_t (*pgtable_alloc)(int), int flags) { unsigned long next; @@ -183,7 +183,7 @@ static void alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, if (pmd_none(pmd)) { phys_addr_t pte_phys; BUG_ON(!pgtable_alloc); - pte_phys = pgtable_alloc(); + pte_phys = pgtable_alloc(PAGE_SHIFT); __pmd_populate(pmdp, pte_phys, PMD_TYPE_TABLE); pmd = READ_ONCE(*pmdp); } @@ -207,7 +207,7 @@ static void alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, static void init_pmd(pud_t *pudp, unsigned long addr, unsigned long end, phys_addr_t phys, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(void), int flags) + phys_addr_t (*pgtable_alloc)(int), int flags) { unsigned long next; pmd_t *pmdp; @@ -245,7 +245,7 @@ static void init_pmd(pud_t *pudp, unsigned long addr, unsigned long end, static void alloc_init_cont_pmd(pud_t *pudp, unsigned long addr, unsigned long end, phys_addr_t phys, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(void), int flags) + phys_addr_t (*pgtable_alloc)(int), int flags) { unsigned long next; pud_t pud = READ_ONCE(*pudp); @@ -257,7 +257,7 @@ static void alloc_init_cont_pmd(pud_t *pudp, unsigned long addr, if (pud_none(pud)) { phys_addr_t pmd_phys; BUG_ON(!pgtable_alloc); - pmd_phys = pgtable_alloc(); + pmd_phys = pgtable_alloc(PMD_SHIFT); __pud_populate(pudp, pmd_phys, PUD_TYPE_TABLE); pud = READ_ONCE(*pudp); } @@ -293,7 +293,7 @@ static inline bool use_1G_block(unsigned long addr, unsigned long next, static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end, phys_addr_t phys, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(void), + phys_addr_t (*pgtable_alloc)(int), int flags) { unsigned long next; @@ -303,7 +303,7 @@ static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end, if (pgd_none(pgd)) { phys_addr_t pud_phys; BUG_ON(!pgtable_alloc); - pud_phys = pgtable_alloc(); + pud_phys = pgtable_alloc(PUD_SHIFT); __pgd_populate(pgdp, pud_phys, PUD_TYPE_TABLE); pgd = READ_ONCE(*pgdp); } @@ -344,7 +344,7 @@ static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end, static void __create_pgd_mapping(pgd_t *pgdir, phys_addr_t phys, unsigned long virt, phys_addr_t size, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(void), + phys_addr_t (*pgtable_alloc)(int), int flags) { unsigned long addr, length, end, next; @@ -370,11 +370,23 @@ static void __create_pgd_mapping(pgd_t *pgdir, phys_addr_t phys, } while (pgdp++, addr = next, addr != end); } -static phys_addr_t pgd_pgtable_alloc(void) +static phys_addr_t pgd_pgtable_alloc(int shift) { void *ptr = (void *)__get_free_page(PGALLOC_GFP); - if (!ptr || !pgtable_page_ctor(virt_to_page(ptr))) - BUG(); + BUG_ON(!ptr); + + /* + * Call proper page table ctor in case later we need to + * call core mm functions like apply_to_page_range() on + * this pre-allocated page table. + * + * We don't select ARCH_ENABLE_SPLIT_PMD_PTLOCK if pmd is + * folded, and if so pgtable_pmd_page_ctor() becomes nop. + */ + if (shift == PAGE_SHIFT) + BUG_ON(!pgtable_page_ctor(virt_to_page(ptr))); + else if (shift == PMD_SHIFT) + BUG_ON(!pgtable_pmd_page_ctor(virt_to_page(ptr))); /* Ensure the zeroed page is visible to the page table walker */ dsb(ishst); From patchwork Tue Mar 12 00:57:47 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 10848447 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1AB6B139A for ; Tue, 12 Mar 2019 00:58:13 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0131D293FD for ; Tue, 12 Mar 2019 00:58:13 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id F349829417; Tue, 12 Mar 2019 00:58:12 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 9E271293FD for ; Tue, 12 Mar 2019 00:58:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version :Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=xODsdmSSV9iGg+bijgTNPYBxr6fW6kPrrAi6Wx+FBFI=; b=DhfU+/lu1lGsgw N+EzHJJDnhHF4+HNQ7KTYAmsGTnxZvf0APqhj0ESkghEemiDhuOXtdyW3AVv1EEnI9URysG/Eempi ZXCC8Q9m1LxZuYzUcl3GtSEqLWCLRYMmwrEceVhlsJL2K4hXhmj3g4NqYmLwCAX+IESfvrjbmkdGC 3ddpFKLzdPn6oU4bSr50sO3/H3O+xwpMs2+SsOdizVYGyu2Em27T6ng9Y3ElvbHibq/efoWouFnl2 MoqYbRLrFRRfb6CUd2nEOytDYB6mPZs4c06Gf5i+ut3Lixnro7Os/MSYiXKJ+acHtqzFANRq2kwiZ p1bFMkegO+TMmDF6zm1A==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1h3VkB-0003WF-1K; Tue, 12 Mar 2019 00:58:11 +0000 Received: from mail-io1-xd49.google.com ([2607:f8b0:4864:20::d49]) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1h3Vk7-0003Ug-S9 for linux-arm-kernel@lists.infradead.org; Tue, 12 Mar 2019 00:58:09 +0000 Received: by mail-io1-xd49.google.com with SMTP id z22so599163iog.5 for ; Mon, 11 Mar 2019 17:58:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=0N1MhcRbdixhLcojGjz1JbD9gYvKsU3sALAMDz5L+zQ=; b=m2SSklts5axBtBFT99k5kWRqt2k7AZ/UFLe5QDWp2m4MrS9nPTzvu+lW6Cw1GTX91x q7m9lX4AdtHIlQvrmJR+9nAGmjI1Sxwo9tCXJq3a4TfLEoryCkDdcW+2PmNyW2gIWRBs wTPKiJ0yJ7eAQJXABh3sNDsdXtcGTVXBZd0z6nTiMfX/ZIfTqQq83M55OBXw3KfljcNy jwcNX81PR159z9bI7I9PlNagdpSG7F8sKSGKprXJJKepemxzBfeWNYASGdTYUFTIqFFc lrAn/W9ub2uGjX9hl5ahbzfE78V49Svncz9HAlXjfo8dyifGlJnK9/F5LsE8kRnM/321 lOaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=0N1MhcRbdixhLcojGjz1JbD9gYvKsU3sALAMDz5L+zQ=; b=e+l5Mv1A1KLykKqqRU9eaH1nDnnBNhHrB/deapxlrOoY8Bt4Tx1BkZaAIJRC/BcWS7 AgrSorUglMvSSzeTwPT/8sPCFUjBI8zBcuQsLGdU/ahdX4MPs57MaKKSlPRe94n4sg73 9fGxs1Qj7FVq3ZjxO+YAVhn/auFUOi0WSkzT5pDJVDayHnm8xwqlaPyQ5EuNgQP3/sor 3OCDId+rgKjgjObe2IwwVT/Aw992lUmtpvyLMpnib5kVJTHpvUWFdFPxeJ8WNOQguDTy OggQRuI71tMggzQvdWMOQ4jvSAfp5RQX9Q1eKPyIhOA1VgN4XTeDNeLUiG4RVLpPDKDJ HoUg== X-Gm-Message-State: APjAAAVtNG++Hvhiz4WSssfpWsCqhDf2elKZE5CZQ/DPJ676AvphWfBK DS/gp11UGE5tCGiNwsHO9K2rRszBd6Y= X-Google-Smtp-Source: APXvYqw5KDZ39HU8Icwu4S6RpSSs7BszMpNyZQ5WJg5vOVgn5k90ODytif1OzlfBC2AtFGb/6BjcJbfFcJE= X-Received: by 2002:a24:5a04:: with SMTP id v4mr546460ita.37.1552352285759; Mon, 11 Mar 2019 17:58:05 -0700 (PDT) Date: Mon, 11 Mar 2019 18:57:47 -0600 In-Reply-To: <20190312005749.30166-1-yuzhao@google.com> Message-Id: <20190312005749.30166-2-yuzhao@google.com> Mime-Version: 1.0 References: <20190310011906.254635-1-yuzhao@google.com> <20190312005749.30166-1-yuzhao@google.com> X-Mailer: git-send-email 2.21.0.360.g471c308f928-goog Subject: [PATCH v4 2/4] arm64: mm: don't call page table ctors for init_mm From: Yu Zhao To: Catalin Marinas , Will Deacon , Mark Rutland X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190311_175807_931406_9FAA051A X-CRM114-Status: GOOD ( 12.51 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch@vger.kernel.org, Yu Zhao , Ard Biesheuvel , Peter Zijlstra , linux-kernel@vger.kernel.org, Nick Piggin , Jun Yao , linux-mm@kvack.org, "Aneesh Kumar K . V" , Chintan Pandya , Joel Fernandes , "Kirill A . Shutemov" , Andrew Morton , Laura Abbott , linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP init_mm doesn't require page table lock to be initialized at any level. Add a separate page table allocator for it, and the new one skips page table ctors. The ctors allocate memory when ALLOC_SPLIT_PTLOCKS is set. Not calling them avoids memory leak in case we call pte_free_kernel() on init_mm. Acked-by: Mark Rutland Signed-off-by: Yu Zhao --- arch/arm64/mm/mmu.c | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index f704b291f2c5..d1dc2a2777aa 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -370,6 +370,16 @@ static void __create_pgd_mapping(pgd_t *pgdir, phys_addr_t phys, } while (pgdp++, addr = next, addr != end); } +static phys_addr_t pgd_kernel_pgtable_alloc(int shift) +{ + void *ptr = (void *)__get_free_page(PGALLOC_GFP); + BUG_ON(!ptr); + + /* Ensure the zeroed page is visible to the page table walker */ + dsb(ishst); + return __pa(ptr); +} + static phys_addr_t pgd_pgtable_alloc(int shift) { void *ptr = (void *)__get_free_page(PGALLOC_GFP); @@ -594,7 +604,7 @@ static int __init map_entry_trampoline(void) /* Map only the text into the trampoline page table */ memset(tramp_pg_dir, 0, PGD_SIZE); __create_pgd_mapping(tramp_pg_dir, pa_start, TRAMP_VALIAS, PAGE_SIZE, - prot, pgd_pgtable_alloc, 0); + prot, pgd_kernel_pgtable_alloc, 0); /* Map both the text and data into the kernel page table */ __set_fixmap(FIX_ENTRY_TRAMP_TEXT, pa_start, prot); @@ -1070,7 +1080,8 @@ int arch_add_memory(int nid, u64 start, u64 size, struct vmem_altmap *altmap, flags = NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS; __create_pgd_mapping(swapper_pg_dir, start, __phys_to_virt(start), - size, PAGE_KERNEL, pgd_pgtable_alloc, flags); + size, PAGE_KERNEL, pgd_kernel_pgtable_alloc, + flags); return __add_pages(nid, start >> PAGE_SHIFT, size >> PAGE_SHIFT, altmap, want_memblock); From patchwork Tue Mar 12 00:57:48 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 10848453 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4346E139A for ; Tue, 12 Mar 2019 00:58:44 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2CC262932B for ; Tue, 12 Mar 2019 00:58:44 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2049C2940F; Tue, 12 Mar 2019 00:58:44 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 6678229411 for ; Tue, 12 Mar 2019 00:58:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version :Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=KcDF8moAxvfRySveAPsyvEpyMDIscNF/Kta5k4zeUQ4=; b=gTWZYKbvjd3kyx pvC0As9aDc9qamW8fFwYhF6SOwi+Z3DLxBwsS+w/Sh9vHfsRlCssTOVvSfpF7h1qoIyvy1MiIXUC8 CuWegim5enN+T+9WWj8sG0Jf8tQhWCND9WKfSheDTTAcl8x4yr11TgUlCLzWNh36USz3yEVMAb1PN /NfS/bGtXE86qMbG4JtJNJH3MdRxM2S0ZMQMEukWmUNJEPfmi4/ckN3nfm2tw2O3ftWk0d1PswuGa MC4wSbMCeu+rjCZzKF69yzpZlspRn2ZUbXpBXtb/CivzaIkblNmlRat3LphdzJxvdJSCBOqCyonFM mWvSX+yOPAUJXYyVKelw==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1h3Vka-00043K-S0; Tue, 12 Mar 2019 00:58:36 +0000 Received: from mail-it1-x149.google.com ([2607:f8b0:4864:20::149]) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1h3Vk9-0003Uj-SO for linux-arm-kernel@lists.infradead.org; Tue, 12 Mar 2019 00:58:11 +0000 Received: by mail-it1-x149.google.com with SMTP id v12so787145itv.9 for ; Mon, 11 Mar 2019 17:58:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=srduQGUMd+1T0rMo1nAAhODDuDFnWSZKCOfw6uJonls=; b=oedD/eggmV+SIgYB11JrwAoZLhFWdb50S/XNtFqL8BtNX+SDe5Glssl7AopEbDqcNA HCKLJ/V3bjISgT2y2KxWp3VykVnCmF5Y6Eu7pAUj8ZigjSaNi2yJ90jUlcOctuxB1Fbm A3C127QaAa54k+3O0L7KF5QaqVgnfjPJr6a85bI04USgXp04seBuyvB1LjH9V5YVSO4n 9poZ+ZPTh65xeVRUyWd030y18tK6PcucnaLZg/I1TUKe2kC6W9D9AdSqG6CQmwyqYYKV 4olxX5RbFrrKkNYKcwWg7HTaRF8nnkjP+yLqbV7GoSTMRn558HpMjY8QWgCmjB5DLMAj nSOA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=srduQGUMd+1T0rMo1nAAhODDuDFnWSZKCOfw6uJonls=; b=DkJnOfY2VOjOh5uoYDDyVuXYqGbwzb68kXDNAsnSAoVgPkGyS4z/agQ9zevGBz4RIo QiYQTrYrjzopy/YgOY6bX8kIJUAO6J2L/wacNggiUI/wnOnKkyaIYmLTBBpitP9KlyMp S51c47E80vFX/EdpO9fS7EUc3qLazuZ/6vBb5F/NerIMO9bIb/VzY2OuDAyZ03pyh+Ku Y8p/N254Mn2Wmoy7a8Z1Jpc8LcT0fJNEHO0Ffh6+hbJlxjH36c4WpxSgJewHuq1SZOCV tTtvxELeGtzp0KZ4O/3GlPrJ9z6Pqfp2DlxdcxDGLB/IioB9mSkGuQuMYjzmzghwJqzb 3MVA== X-Gm-Message-State: APjAAAUZeHgOsP9zY5+3zL/uHs6/tRqcpVJfFgckRzMt5C0OsCYUIU+1 haIACQwvwh5FhjqGDGyLrduSOgLjovA= X-Google-Smtp-Source: APXvYqxqCOwYBxF/7ICjEeopmZPTb2uYLa3V2NjdBJYx9miVxqzRT+/6zNK7zIhloXUYTCZtvdQJhkLhhWk= X-Received: by 2002:a24:e984:: with SMTP id f126mr583102ith.1.1552352287403; Mon, 11 Mar 2019 17:58:07 -0700 (PDT) Date: Mon, 11 Mar 2019 18:57:48 -0600 In-Reply-To: <20190312005749.30166-1-yuzhao@google.com> Message-Id: <20190312005749.30166-3-yuzhao@google.com> Mime-Version: 1.0 References: <20190310011906.254635-1-yuzhao@google.com> <20190312005749.30166-1-yuzhao@google.com> X-Mailer: git-send-email 2.21.0.360.g471c308f928-goog Subject: [PATCH v4 3/4] arm64: mm: call ctor for stage2 pmd page From: Yu Zhao To: Catalin Marinas , Will Deacon , Mark Rutland X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190311_175809_911795_E814CE51 X-CRM114-Status: GOOD ( 12.62 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch@vger.kernel.org, Yu Zhao , Ard Biesheuvel , Peter Zijlstra , linux-kernel@vger.kernel.org, Nick Piggin , Jun Yao , linux-mm@kvack.org, "Aneesh Kumar K . V" , Chintan Pandya , Joel Fernandes , "Kirill A . Shutemov" , Andrew Morton , Laura Abbott , linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Call pgtable_pmd_page_dtor() for pmd page allocated by mmu_memory_cache_alloc() so kernel won't crash when it's freed through stage2_pmd_free()->pmd_free()->pgtable_pmd_page_dtor(). This is needed if we are going to enable split pmd pt lock. Signed-off-by: Yu Zhao --- arch/arm64/include/asm/stage2_pgtable.h | 15 ++++++++++++--- virt/kvm/arm/mmu.c | 13 +++++++++++-- 2 files changed, 23 insertions(+), 5 deletions(-) diff --git a/arch/arm64/include/asm/stage2_pgtable.h b/arch/arm64/include/asm/stage2_pgtable.h index 5412fa40825e..0d9207144257 100644 --- a/arch/arm64/include/asm/stage2_pgtable.h +++ b/arch/arm64/include/asm/stage2_pgtable.h @@ -174,10 +174,19 @@ static inline bool stage2_pud_present(struct kvm *kvm, pud_t pud) return 1; } -static inline void stage2_pud_populate(struct kvm *kvm, pud_t *pud, pmd_t *pmd) +static inline int stage2_pud_populate(struct kvm *kvm, pud_t *pud, pmd_t *pmd) { - if (kvm_stage2_has_pmd(kvm)) - pud_populate(NULL, pud, pmd); + if (!kvm_stage2_has_pmd(kvm)) + return 0; + + /* paired with pgtable_pmd_page_dtor() in pmd_free() below */ + if (!pgtable_pmd_page_ctor(virt_to_page(pmd))) { + free_page((unsigned long)pmd); + return -ENOMEM; + } + + pud_populate(NULL, pud, pmd); + return 0; } static inline pmd_t *stage2_pmd_offset(struct kvm *kvm, diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c index e9d28a7ca673..11922d84be83 100644 --- a/virt/kvm/arm/mmu.c +++ b/virt/kvm/arm/mmu.c @@ -1037,6 +1037,7 @@ static pud_t *stage2_get_pud(struct kvm *kvm, struct kvm_mmu_memory_cache *cache static pmd_t *stage2_get_pmd(struct kvm *kvm, struct kvm_mmu_memory_cache *cache, phys_addr_t addr) { + int ret; pud_t *pud; pmd_t *pmd; @@ -1048,7 +1049,9 @@ static pmd_t *stage2_get_pmd(struct kvm *kvm, struct kvm_mmu_memory_cache *cache if (!cache) return NULL; pmd = mmu_memory_cache_alloc(cache); - stage2_pud_populate(kvm, pud, pmd); + ret = stage2_pud_populate(kvm, pud, pmd); + if (ret) + return ERR_PTR(ret); get_page(virt_to_page(pud)); } @@ -1061,6 +1064,9 @@ static int stage2_set_pmd_huge(struct kvm *kvm, struct kvm_mmu_memory_cache pmd_t *pmd, old_pmd; pmd = stage2_get_pmd(kvm, cache, addr); + if (IS_ERR(pmd)) + return PTR_ERR(pmd); + VM_BUG_ON(!pmd); old_pmd = *pmd; @@ -1198,6 +1204,7 @@ static int stage2_set_pte(struct kvm *kvm, struct kvm_mmu_memory_cache *cache, phys_addr_t addr, const pte_t *new_pte, unsigned long flags) { + int ret; pud_t *pud; pmd_t *pmd; pte_t *pte, old_pte; @@ -1227,7 +1234,9 @@ static int stage2_set_pte(struct kvm *kvm, struct kvm_mmu_memory_cache *cache, if (!cache) return 0; /* ignore calls from kvm_set_spte_hva */ pmd = mmu_memory_cache_alloc(cache); - stage2_pud_populate(kvm, pud, pmd); + ret = stage2_pud_populate(kvm, pud, pmd); + if (ret) + return ret; get_page(virt_to_page(pud)); } From patchwork Tue Mar 12 00:57:49 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 10848455 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BF8B2139A for ; Tue, 12 Mar 2019 00:58:50 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A792A29401 for ; Tue, 12 Mar 2019 00:58:50 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A54C429403; Tue, 12 Mar 2019 00:58:50 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.2 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED,DKIM_VALID,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 4BA5729414 for ; Tue, 12 Mar 2019 00:58:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version :Message-Id:In-Reply-To:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=to3pfuCJn/YhJJf1qh4GDYA9m3I+dhXSUxgz8rwEaNg=; b=Axhvq8Ca0AxPMj tg8lx5iN5ge6S6vCKLgtGNgu6IBKB7qsyyImx3zpvQtehq99p08rV3Gocw5LkPfmRUCaSJC2txXgC KVvolaNFZTOz+Y36EGFcVrOTGxhZqjdubcNOV1vHMtM5Un8Kn9mEmgct/QKN8M6bfXRVU/FVDN1T1 oB7JYjZ0BjFZQDs7nfdieZogOtEzdw7lVHDQqEkBV+A4hV+juyHjkpkdtzfhFTcTAMsvccKGiK/Vp 0dKkAXuvigssexiKDMENnFFfx95JbTC7gz0OomlQtzBYE2wR/BL5aKtiBfma6PLKxrRBR/14cfOnV fwzjv7inmDT7z/oHWq8w==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1h3Vkm-0004ER-1t; Tue, 12 Mar 2019 00:58:48 +0000 Received: from mail-it1-x149.google.com ([2607:f8b0:4864:20::149]) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1h3VkA-0003VE-71 for linux-arm-kernel@lists.infradead.org; Tue, 12 Mar 2019 00:58:11 +0000 Received: by mail-it1-x149.google.com with SMTP id 190so762395itl.7 for ; Mon, 11 Mar 2019 17:58:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=GuInaEquY9cKCo+/R3bLL1KFcpfX5BwU8EHfq1hu08M=; b=HBNdWh+/H/UQD3+xTCC5Gk1vTOFdi8buqG7IOzQCbqWRid+PcTT/T4iKPfX1Muy57H mA9gcEpPcKUv1DaJBopn5SU1wwyHnz+qqvQzVQiKuH4IuAZHw8WmPDHRTekZ4RoJuPoI k8fyIUxtuU7UBUmfFhSlmXIqDkwkNUo2QrkJLaPkIgWzkEwroQQwBNFou6cniBzdTUeU pguYiLIcrrFrKPeCm6lJ2yAvRHkaGHnYB9ACSfOKeP/GH7Q1zppaEI6Nk5aSRMS5Mt3C 9GOsSCQ8e1rnKd9uYUOK1sUbi9ceU8L2Dtpr15+FePDmyFg81CK03OWXDcm0/xCUaK9T fuqw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=GuInaEquY9cKCo+/R3bLL1KFcpfX5BwU8EHfq1hu08M=; b=sMjUBMaHvYRu27Ehf+TBMFLDD60Ua2E4HlzZBZ/u6Qdvej7Y5ikVY3Vt/9PITI/1YO V4Xi7v2xIP2bxmeN9rDFRX9fyu/tm5vVvXm4f2wfIifELGxwwV8uQ/KYLAlJX6JVr1wc RRExw01AiahpVXIm0qrPPKdY6TECifJWCBJbo1Js/oi09Qf1KaEOa/ioBbOB9EXvCxLm xNj9QLyyfMFHox0ZviMez3Q2ZMkmbrK/ibQoXzVJIuOvw+ngZD4+yokLQ5TcjJjGZFiD 1AteAxpMFAD0xtOAOek/mjeo1BrbqMj30vAjyaidLy7nqX6ra1qJNbWH7SRiE6IFiNH+ RZAw== X-Gm-Message-State: APjAAAW8/8HnzrYBGfZ0A2sN/vDO94kwZ1Cv9lpT6i78LecYpD3nF569 Ygb9I3t2a4kPA8NH2aprjrkAaDCd4Lw= X-Google-Smtp-Source: APXvYqxhul1R8mnVTm2ANYck/U0sQICDmCd8jsCIgw/3U8ahE12sg7YNMdshwxSVxfvSc2y++gz3qAvURMY= X-Received: by 2002:a24:6b55:: with SMTP id v82mr545054itc.37.1552352288828; Mon, 11 Mar 2019 17:58:08 -0700 (PDT) Date: Mon, 11 Mar 2019 18:57:49 -0600 In-Reply-To: <20190312005749.30166-1-yuzhao@google.com> Message-Id: <20190312005749.30166-4-yuzhao@google.com> Mime-Version: 1.0 References: <20190310011906.254635-1-yuzhao@google.com> <20190312005749.30166-1-yuzhao@google.com> X-Mailer: git-send-email 2.21.0.360.g471c308f928-goog Subject: [PATCH v4 4/4] arm64: mm: enable per pmd page table lock From: Yu Zhao To: Catalin Marinas , Will Deacon , Mark Rutland X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190311_175810_251381_787E84C5 X-CRM114-Status: GOOD ( 14.60 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-arch@vger.kernel.org, Yu Zhao , Ard Biesheuvel , Peter Zijlstra , linux-kernel@vger.kernel.org, Nick Piggin , Jun Yao , linux-mm@kvack.org, "Aneesh Kumar K . V" , Chintan Pandya , Joel Fernandes , "Kirill A . Shutemov" , Andrew Morton , Laura Abbott , linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Switch from per mm_struct to per pmd page table lock by enabling ARCH_ENABLE_SPLIT_PMD_PTLOCK. This provides better granularity for large system. I'm not sure if there is contention on mm->page_table_lock. Given the option comes at no cost (apart from initializing more spin locks), why not enable it now. We only do so when pmd is not folded, so we don't mistakenly call pgtable_pmd_page_ctor() on pud or p4d in pgd_pgtable_alloc(). Signed-off-by: Yu Zhao --- arch/arm64/Kconfig | 3 +++ arch/arm64/include/asm/pgalloc.h | 12 +++++++++++- arch/arm64/include/asm/tlb.h | 5 ++++- 3 files changed, 18 insertions(+), 2 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index cfbf307d6dc4..a3b1b789f766 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -872,6 +872,9 @@ config ARCH_WANT_HUGE_PMD_SHARE config ARCH_HAS_CACHE_LINE_SIZE def_bool y +config ARCH_ENABLE_SPLIT_PMD_PTLOCK + def_bool y if PGTABLE_LEVELS > 2 + config SECCOMP bool "Enable seccomp to safely compute untrusted bytecode" ---help--- diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index 52fa47c73bf0..dabba4b2c61f 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -33,12 +33,22 @@ static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long addr) { - return (pmd_t *)__get_free_page(PGALLOC_GFP); + struct page *page; + + page = alloc_page(PGALLOC_GFP); + if (!page) + return NULL; + if (!pgtable_pmd_page_ctor(page)) { + __free_page(page); + return NULL; + } + return page_address(page); } static inline void pmd_free(struct mm_struct *mm, pmd_t *pmdp) { BUG_ON((unsigned long)pmdp & (PAGE_SIZE-1)); + pgtable_pmd_page_dtor(virt_to_page(pmdp)); free_page((unsigned long)pmdp); } diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index 106fdc951b6e..4e3becfed387 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -62,7 +62,10 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, unsigned long addr) { - tlb_remove_table(tlb, virt_to_page(pmdp)); + struct page *page = virt_to_page(pmdp); + + pgtable_pmd_page_dtor(page); + tlb_remove_table(tlb, page); } #endif