From patchwork Sun Mar 10 01:19:04 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 10846071 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B770014DE for ; Sun, 10 Mar 2019 01:19:14 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 95923289A1 for ; Sun, 10 Mar 2019 01:19:14 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 8816E289BA; Sun, 10 Mar 2019 01:19:14 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E2150289A1 for ; Sun, 10 Mar 2019 01:19:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C1D598E0003; Sat, 9 Mar 2019 20:19:11 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id BA5DB8E0002; Sat, 9 Mar 2019 20:19:11 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A467F8E0003; Sat, 9 Mar 2019 20:19:11 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-io1-f69.google.com (mail-io1-f69.google.com [209.85.166.69]) by kanga.kvack.org (Postfix) with ESMTP id 786678E0002 for ; Sat, 9 Mar 2019 20:19:11 -0500 (EST) Received: by mail-io1-f69.google.com with SMTP id m15so1281159ioc.16 for ; Sat, 09 Mar 2019 17:19:11 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=0Yb2ivGv5k2eEZFqDaeb2AMWK/OeRIAqkI0UIANBdSk=; b=ojYIcLdmE3uM0Zmj8AF/cbNmPJH8poJWJr8yDpljxLoFBhaPxcKtNB5fZO9nDspOAs 0t/Bv5PDV4q8b9fyKkPJ+aElqh1Gjwk+0YjjCj5ukL9UzVoAYswYfAkBHx+eEBl0U1Sz scH4MBEmpo70VK83yDVkfyP9xWK6AZLPjGLfLbjvbfqFEqaoTVRkxRC+MbOvWWE6tKLZ crNJteKzTORSOJekLS6KpLyDi2bogzuxCHABTdRdBHgbSnWYm/wHTCEIFplwk//d8mQC orf0pFvKhd9sxCjoObstcGpk+tkFV2yZhUVXcjv+Jlrhbe2/MDpc9gfJTKiCI2DpdpvA XpuQ== X-Gm-Message-State: APjAAAXpeBFhQ9BjBPWqHwzk0MzbU5iPOA8r9Q/JanXhPyMdzd88TdAM 3KIZoVw/sthWJd+IuevGwljZ3QUHX1g5XMx+6CnLMD7t9qybEZ9KogKeHY6jayL37cgXUqsYACd BN3TzO0vKBVwi4pbkH+CfeqMgcEkJYnFszks9ObflwafQz0RX74/g66YYOpNq0hX/FaT2U7IhGK L2P9MSS7t2jJfzLUOVVUAAbZoDyhbwQMndEoJg0O/42aw3hMwY8DcDL2ptRCdShnefYQDNtV24D 3/BMRlNOpgmizLBc/QFeRUT3wmH8ELtI0enhVSqXMi3qhha3aOPLk0uq6gcI64ah0DU0mt/Uh7c KzDJeh/cdJe+GQFW6fwVfaiYEGZ2890CHnAqrt0wtaMww3/gjQfwy5pomJiFpsRaNePps/T8VKZ t X-Received: by 2002:a02:8899:: with SMTP id n25mr15126857jaj.7.1552180751189; Sat, 09 Mar 2019 17:19:11 -0800 (PST) X-Received: by 2002:a02:8899:: with SMTP id n25mr15126834jaj.7.1552180750133; Sat, 09 Mar 2019 17:19:10 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1552180750; cv=none; d=google.com; s=arc-20160816; b=TdVR4+jcjqCiOPdThNhPQ/4PLN/aaiZFoX0HEe9sEBc4WK+wFBrYNn4eXBZrF6B+jE wj1yVJTcsEZlQK75AXNkNAgRZ9da4qg35c3uC3qCpFJApjnyzGBjLqsVIoaF2fpIxkvj KAmcdh6NNU1idt9Crrgli/DNgfhBCL/jvUYZ8SlwiNwzVIePI+h6twDIMlmeJ6mNy3O/ W2AG8+PEKQGBKZoYV/BiCFBb17JTKAPQIbkycxKA3ntUudOFamTwoydQg5UdjSHA0Dyl QsGLS+bCQM9xXQEctxnfEjxGLyoDvXS+t2KsEhasQNLty6/gNb15NaaQ+aMX0U3arBv3 7onQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:dkim-signature; bh=0Yb2ivGv5k2eEZFqDaeb2AMWK/OeRIAqkI0UIANBdSk=; b=SR/O2Nvldlo4WUHw+lro0KAuKOqvekvAUV9TJY4RLBnRIjIA/hfvNJ+k1cA6hQU7/K 3fC/kiF/PnISU51IIwYB5COMLTUCLpDjK4cRFsydYJxcOV+RArG8Uvwj1px7ViOgDAjh 4qyaaHY3bcuuy5yf6oiRlJIVaeC9STmVWk4WCu80GR9s6Qim2kvsgHyLRwI7gdLKmet2 81TqUefC/IbdBbJ/ORWt1a6ctHR/GmUMMkYpWyWOjno+6Exe3m8cqAA8dBj1MV+r+uNK tJ/Rmhm60+Xn4+d+r0pnhsdpWkh/3oYFMXw5QbisFpNtoKJj9E9A4TYw63DiLqwqXems im1w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=oPNZFD0u; spf=pass (google.com: domain of 3dwaexaykcnqokp70e6ee6b4.2ecb8dkn-ccal02a.eh6@flex--yuzhao.bounces.google.com designates 209.85.220.73 as permitted sender) smtp.mailfrom=3DWaEXAYKCNQOKP70E6EE6B4.2ECB8DKN-CCAL02A.EH6@flex--yuzhao.bounces.google.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from mail-sor-f73.google.com (mail-sor-f73.google.com. [209.85.220.73]) by mx.google.com with SMTPS id p67sor21828739itp.32.2019.03.09.17.19.10 for (Google Transport Security); Sat, 09 Mar 2019 17:19:10 -0800 (PST) Received-SPF: pass (google.com: domain of 3dwaexaykcnqokp70e6ee6b4.2ecb8dkn-ccal02a.eh6@flex--yuzhao.bounces.google.com designates 209.85.220.73 as permitted sender) client-ip=209.85.220.73; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=oPNZFD0u; spf=pass (google.com: domain of 3dwaexaykcnqokp70e6ee6b4.2ecb8dkn-ccal02a.eh6@flex--yuzhao.bounces.google.com designates 209.85.220.73 as permitted sender) smtp.mailfrom=3DWaEXAYKCNQOKP70E6EE6B4.2ECB8DKN-CCAL02A.EH6@flex--yuzhao.bounces.google.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=0Yb2ivGv5k2eEZFqDaeb2AMWK/OeRIAqkI0UIANBdSk=; b=oPNZFD0u4/uhlNdesjj0nps43el+5vEaWETEUy12QTiHxMV+XS5yFbmZKISRtxFNG5 dKWWoBBhXtJZPfcIT/oFsQor0CVaWiLlLS87tKNCYTv85G2XZg+Hv9r6tfnPFbIGY6Un SNGdRMg05dZskRUyqM2LIB5r6U28IbDBz5PJK/sgVrxMjBJtfSiF4GDger33PFbzQ3/V RdWI9nwiV6I7gLmISzKZ3dWTOEkVTwixb/Nob1jM7uiO4IABEGaAA33lL84qG5hGXKGb XMyAh6HfjdLukWrrG6lkqNzh+60rHUSwrcfxjOOObGKwXrTy2XupT/tlf5HU2vReBnn3 ld9w== X-Google-Smtp-Source: APXvYqzb7py1oPWQlx4fRa9Z+HVEK1bN2fB0qOGJMFr6UeE7WrMjrMYMPcQraL6qYk6xnNQti0o0/6kxf8A= X-Received: by 2002:a05:660c:484:: with SMTP id a4mr16369624itk.15.1552180749805; Sat, 09 Mar 2019 17:19:09 -0800 (PST) Date: Sat, 9 Mar 2019 18:19:04 -0700 In-Reply-To: <20190218231319.178224-1-yuzhao@google.com> Message-Id: <20190310011906.254635-1-yuzhao@google.com> Mime-Version: 1.0 References: <20190218231319.178224-1-yuzhao@google.com> X-Mailer: git-send-email 2.21.0.360.g471c308f928-goog Subject: [PATCH v3 1/3] arm64: mm: use appropriate ctors for page tables From: Yu Zhao To: Catalin Marinas , Will Deacon , Mark Rutland Cc: "Aneesh Kumar K . V" , Andrew Morton , Nick Piggin , Peter Zijlstra , Joel Fernandes , "Kirill A . Shutemov" , Ard Biesheuvel , Chintan Pandya , Jun Yao , Laura Abbott , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, Yu Zhao X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP For pte page, use pgtable_page_ctor(); for pmd page, use pgtable_pmd_page_ctor(); and for the rest (pud, p4d and pgd), don't use any. For now, we don't select ARCH_ENABLE_SPLIT_PMD_PTLOCK and pgtable_pmd_page_ctor() is a nop. When we do in patch 3, we make sure pmd is not folded so we won't mistakenly call pgtable_pmd_page_ctor() on pud or p4d. Acked-by: Mark Rutland Signed-off-by: Yu Zhao --- arch/arm64/mm/mmu.c | 36 ++++++++++++++++++++++++------------ 1 file changed, 24 insertions(+), 12 deletions(-) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index b6f5aa52ac67..f704b291f2c5 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -98,7 +98,7 @@ pgprot_t phys_mem_access_prot(struct file *file, unsigned long pfn, } EXPORT_SYMBOL(phys_mem_access_prot); -static phys_addr_t __init early_pgtable_alloc(void) +static phys_addr_t __init early_pgtable_alloc(int shift) { phys_addr_t phys; void *ptr; @@ -173,7 +173,7 @@ static void init_pte(pmd_t *pmdp, unsigned long addr, unsigned long end, static void alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, unsigned long end, phys_addr_t phys, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(void), + phys_addr_t (*pgtable_alloc)(int), int flags) { unsigned long next; @@ -183,7 +183,7 @@ static void alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, if (pmd_none(pmd)) { phys_addr_t pte_phys; BUG_ON(!pgtable_alloc); - pte_phys = pgtable_alloc(); + pte_phys = pgtable_alloc(PAGE_SHIFT); __pmd_populate(pmdp, pte_phys, PMD_TYPE_TABLE); pmd = READ_ONCE(*pmdp); } @@ -207,7 +207,7 @@ static void alloc_init_cont_pte(pmd_t *pmdp, unsigned long addr, static void init_pmd(pud_t *pudp, unsigned long addr, unsigned long end, phys_addr_t phys, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(void), int flags) + phys_addr_t (*pgtable_alloc)(int), int flags) { unsigned long next; pmd_t *pmdp; @@ -245,7 +245,7 @@ static void init_pmd(pud_t *pudp, unsigned long addr, unsigned long end, static void alloc_init_cont_pmd(pud_t *pudp, unsigned long addr, unsigned long end, phys_addr_t phys, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(void), int flags) + phys_addr_t (*pgtable_alloc)(int), int flags) { unsigned long next; pud_t pud = READ_ONCE(*pudp); @@ -257,7 +257,7 @@ static void alloc_init_cont_pmd(pud_t *pudp, unsigned long addr, if (pud_none(pud)) { phys_addr_t pmd_phys; BUG_ON(!pgtable_alloc); - pmd_phys = pgtable_alloc(); + pmd_phys = pgtable_alloc(PMD_SHIFT); __pud_populate(pudp, pmd_phys, PUD_TYPE_TABLE); pud = READ_ONCE(*pudp); } @@ -293,7 +293,7 @@ static inline bool use_1G_block(unsigned long addr, unsigned long next, static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end, phys_addr_t phys, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(void), + phys_addr_t (*pgtable_alloc)(int), int flags) { unsigned long next; @@ -303,7 +303,7 @@ static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end, if (pgd_none(pgd)) { phys_addr_t pud_phys; BUG_ON(!pgtable_alloc); - pud_phys = pgtable_alloc(); + pud_phys = pgtable_alloc(PUD_SHIFT); __pgd_populate(pgdp, pud_phys, PUD_TYPE_TABLE); pgd = READ_ONCE(*pgdp); } @@ -344,7 +344,7 @@ static void alloc_init_pud(pgd_t *pgdp, unsigned long addr, unsigned long end, static void __create_pgd_mapping(pgd_t *pgdir, phys_addr_t phys, unsigned long virt, phys_addr_t size, pgprot_t prot, - phys_addr_t (*pgtable_alloc)(void), + phys_addr_t (*pgtable_alloc)(int), int flags) { unsigned long addr, length, end, next; @@ -370,11 +370,23 @@ static void __create_pgd_mapping(pgd_t *pgdir, phys_addr_t phys, } while (pgdp++, addr = next, addr != end); } -static phys_addr_t pgd_pgtable_alloc(void) +static phys_addr_t pgd_pgtable_alloc(int shift) { void *ptr = (void *)__get_free_page(PGALLOC_GFP); - if (!ptr || !pgtable_page_ctor(virt_to_page(ptr))) - BUG(); + BUG_ON(!ptr); + + /* + * Call proper page table ctor in case later we need to + * call core mm functions like apply_to_page_range() on + * this pre-allocated page table. + * + * We don't select ARCH_ENABLE_SPLIT_PMD_PTLOCK if pmd is + * folded, and if so pgtable_pmd_page_ctor() becomes nop. + */ + if (shift == PAGE_SHIFT) + BUG_ON(!pgtable_page_ctor(virt_to_page(ptr))); + else if (shift == PMD_SHIFT) + BUG_ON(!pgtable_pmd_page_ctor(virt_to_page(ptr))); /* Ensure the zeroed page is visible to the page table walker */ dsb(ishst); From patchwork Sun Mar 10 01:19:05 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 10846073 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 530B714DE for ; Sun, 10 Mar 2019 01:19:17 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3F108289A1 for ; Sun, 10 Mar 2019 01:19:17 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2EE7E289C0; Sun, 10 Mar 2019 01:19:17 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AAC19289A1 for ; Sun, 10 Mar 2019 01:19:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0D5468E0004; Sat, 9 Mar 2019 20:19:13 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 05B8B8E0002; Sat, 9 Mar 2019 20:19:13 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E3DD38E0004; Sat, 9 Mar 2019 20:19:12 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-it1-f197.google.com (mail-it1-f197.google.com [209.85.166.197]) by kanga.kvack.org (Postfix) with ESMTP id B8A408E0002 for ; Sat, 9 Mar 2019 20:19:12 -0500 (EST) Received: by mail-it1-f197.google.com with SMTP id v125so1633327itc.4 for ; Sat, 09 Mar 2019 17:19:12 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=0N1MhcRbdixhLcojGjz1JbD9gYvKsU3sALAMDz5L+zQ=; b=k81MN8etVPllCQ2uVs9k9ZjyBQXGAy2c0Brtvji/bJ37h9VCuZNmJxcO8V6ZwQ+VUT 6Pz5UWrXDc/HRPfJyJozNAUqKLKEof7KtRe9hnMJP1APSx1ko0VBv1n1umzcrop2YQjX dRbpOk//WXWGCG8/IJeIcbSSMazmA9xyrrzZlOLvp4yAObcSM+10IzDp9Y/cwn7ij5ic CPEMHf9YdI/F8nVPiAOh+kiUZhYZTEBzR+3S1Wi+BZom0sox0eq11No0F/RNscB71oXc V62nu9VV0EyOrsD8wy7C51WFxgtLdNJWKTUVxAKBmRqGHDfG05RfGCrVyCWeVe/3Nd8g EHCQ== X-Gm-Message-State: APjAAAVszuVKz0/9u+8qYFM3S2Xcba4RdCKVf8MaVC+YV51wwSrQIccB fCh7gry4ZysaQAwUcokUG/KO6jAIa5qqjjvE6adGtV3tL1XpieocMqZAy+b2T95X6V45TksmxV/ kCI7rEUbUsE+7ENzuXTeqVveSoV6tcJiEWeG8NVfVYNdKJ9Imqma3uNbtSsUKVM/iOnbNjaTpsX DwUqW4QyFnJyjwIGtJZ7phWffXDuXUu5h6LGWvq2Z1aC5Xj6l3PSnV2tDYDsdSjRKsMnTkiVK8X 3yCE9FJoGSZPQPUCx2ElOuIDGhPFJKTUCIUCXjQIrPZqAlXD+LUbAIEw0DD1A76i3YlvviA85j8 +//KJ2tcZiLJsCIE+ev2EwJ+fXphG/CXjpICQ2S7k2MwcyfoQlEn7z3ktjXYgsCDgBZyyrXN7fM x X-Received: by 2002:a24:4741:: with SMTP id t62mr11675516itb.110.1552180752412; Sat, 09 Mar 2019 17:19:12 -0800 (PST) X-Received: by 2002:a24:4741:: with SMTP id t62mr11675500itb.110.1552180751461; Sat, 09 Mar 2019 17:19:11 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1552180751; cv=none; d=google.com; s=arc-20160816; b=N9/g3fbL+5TAsA4pzCunS/qAh/9REoF2A/CnoYQgZ/y6t60uottcGXgC3dZ7UJ+66V ZGpyN4ybp0nfsL1JhyIyitufOuIcCGsxrZhwhVElyjkJ6Ts5HFniV2GK18pNQS+gtlZB PE4iVZ2Li4zoSfgt4T0tv1eXsqfqwhUp0GrE1kg7aaPqjCAAMGTJXn37rJDXT3QWG1Vt 3SULu1lTR88M3DYeZvc9B8JU4sR0Pjje5FawOJPhnLRniJGrp48M/iDr094mYVIn2peG jUPL7K2EQIHTqaZeSXGYR7OlFUQ8oo8BbnjNiNdN1VepAZAP58r6hsYxbeaFGGjh7kAk 1BzA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:dkim-signature; bh=0N1MhcRbdixhLcojGjz1JbD9gYvKsU3sALAMDz5L+zQ=; b=YAyK+chjBOFyhvCo1e8jnmAj6IijOfcirjVRymX1/08a0C3fCv0vfSqzPJ39aBHEz6 D2yUPKzirPrq0HujjMygYh4mDYJGHc1o3+KNwoBvYwsnDlpnjG409/TEekFzAehgQ8AP 3Rch3xLn4fq38IshKAimWgpajAWYWeJzKQLXJ7W3VMS8DUbaJIS+JxasgSzIfDUEssz/ E1j9AQ12dnZEOMUya4uC0N2xuJG1fF7erq/P3d6CYRicUkWPeoWSZzzMie2HXsJij/i4 XSR3XJgG60vj2ZcsJTrc5jSKZP20UU7qMYlce1/j+Ev3ZwvMwu/H1HnuhFpAaZD7QqUv fG6g== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=EWYTqV2O; spf=pass (google.com: domain of 3d2aexaykcnyqmr92g8gg8d6.4gedafmp-eecn24c.gj8@flex--yuzhao.bounces.google.com designates 209.85.220.73 as permitted sender) smtp.mailfrom=3D2aEXAYKCNYQMR92G8GG8D6.4GEDAFMP-EECN24C.GJ8@flex--yuzhao.bounces.google.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from mail-sor-f73.google.com (mail-sor-f73.google.com. [209.85.220.73]) by mx.google.com with SMTPS id h12sor20043965itb.29.2019.03.09.17.19.11 for (Google Transport Security); Sat, 09 Mar 2019 17:19:11 -0800 (PST) Received-SPF: pass (google.com: domain of 3d2aexaykcnyqmr92g8gg8d6.4gedafmp-eecn24c.gj8@flex--yuzhao.bounces.google.com designates 209.85.220.73 as permitted sender) client-ip=209.85.220.73; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=EWYTqV2O; spf=pass (google.com: domain of 3d2aexaykcnyqmr92g8gg8d6.4gedafmp-eecn24c.gj8@flex--yuzhao.bounces.google.com designates 209.85.220.73 as permitted sender) smtp.mailfrom=3D2aEXAYKCNYQMR92G8GG8D6.4GEDAFMP-EECN24C.GJ8@flex--yuzhao.bounces.google.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=0N1MhcRbdixhLcojGjz1JbD9gYvKsU3sALAMDz5L+zQ=; b=EWYTqV2O2kn2zhIWcI3s1gmgxz1dXGxzOHPenWe89TDG4/8BxGz8vuObzzBtnXc8zn NjXgLS8HuTTvLpauBCevcESBrWxGSqM9g2cj56V1iqJmzZDUO9nPVoyciLxC6LzDzH3U UAbWy3JQJsk1VZSs7Nxw+1zz0XJAMe8cN4BhnZx3ojFoUW9IXD08xH/BzV3Tkwg0x1zk tnLSBiVVRYxYX+0fi4J0uoTiOugYrTudSji/00Whuzv81nFYtjnqB1W/h8Lc8kqO0L/m HU7+E5jphyonX4mohg03eaaVwiXnE+eC/FU8adSSlDYySBz4Y1BEqgHFtTl1zOHcicpu MlLg== X-Google-Smtp-Source: APXvYqxi62oUXl8WHRUPEaV20EFaBbDZc8udY7BAs1+oaagWSJIk8olRNZ0zRHkcRKgD12OX84VO25JPVVo= X-Received: by 2002:a24:cd07:: with SMTP id l7mr17617268itg.22.1552180751250; Sat, 09 Mar 2019 17:19:11 -0800 (PST) Date: Sat, 9 Mar 2019 18:19:05 -0700 In-Reply-To: <20190310011906.254635-1-yuzhao@google.com> Message-Id: <20190310011906.254635-2-yuzhao@google.com> Mime-Version: 1.0 References: <20190218231319.178224-1-yuzhao@google.com> <20190310011906.254635-1-yuzhao@google.com> X-Mailer: git-send-email 2.21.0.360.g471c308f928-goog Subject: [PATCH v3 2/3] arm64: mm: don't call page table ctors for init_mm From: Yu Zhao To: Catalin Marinas , Will Deacon , Mark Rutland Cc: "Aneesh Kumar K . V" , Andrew Morton , Nick Piggin , Peter Zijlstra , Joel Fernandes , "Kirill A . Shutemov" , Ard Biesheuvel , Chintan Pandya , Jun Yao , Laura Abbott , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, Yu Zhao X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP init_mm doesn't require page table lock to be initialized at any level. Add a separate page table allocator for it, and the new one skips page table ctors. The ctors allocate memory when ALLOC_SPLIT_PTLOCKS is set. Not calling them avoids memory leak in case we call pte_free_kernel() on init_mm. Acked-by: Mark Rutland Signed-off-by: Yu Zhao --- arch/arm64/mm/mmu.c | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index f704b291f2c5..d1dc2a2777aa 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -370,6 +370,16 @@ static void __create_pgd_mapping(pgd_t *pgdir, phys_addr_t phys, } while (pgdp++, addr = next, addr != end); } +static phys_addr_t pgd_kernel_pgtable_alloc(int shift) +{ + void *ptr = (void *)__get_free_page(PGALLOC_GFP); + BUG_ON(!ptr); + + /* Ensure the zeroed page is visible to the page table walker */ + dsb(ishst); + return __pa(ptr); +} + static phys_addr_t pgd_pgtable_alloc(int shift) { void *ptr = (void *)__get_free_page(PGALLOC_GFP); @@ -594,7 +604,7 @@ static int __init map_entry_trampoline(void) /* Map only the text into the trampoline page table */ memset(tramp_pg_dir, 0, PGD_SIZE); __create_pgd_mapping(tramp_pg_dir, pa_start, TRAMP_VALIAS, PAGE_SIZE, - prot, pgd_pgtable_alloc, 0); + prot, pgd_kernel_pgtable_alloc, 0); /* Map both the text and data into the kernel page table */ __set_fixmap(FIX_ENTRY_TRAMP_TEXT, pa_start, prot); @@ -1070,7 +1080,8 @@ int arch_add_memory(int nid, u64 start, u64 size, struct vmem_altmap *altmap, flags = NO_BLOCK_MAPPINGS | NO_CONT_MAPPINGS; __create_pgd_mapping(swapper_pg_dir, start, __phys_to_virt(start), - size, PAGE_KERNEL, pgd_pgtable_alloc, flags); + size, PAGE_KERNEL, pgd_kernel_pgtable_alloc, + flags); return __add_pages(nid, start >> PAGE_SHIFT, size >> PAGE_SHIFT, altmap, want_memblock); From patchwork Sun Mar 10 01:19:06 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yu Zhao X-Patchwork-Id: 10846075 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DF9751515 for ; Sun, 10 Mar 2019 01:19:19 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CAACA289A1 for ; Sun, 10 Mar 2019 01:19:19 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BED2C289BA; Sun, 10 Mar 2019 01:19:19 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_NONE, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 48EDA289A1 for ; Sun, 10 Mar 2019 01:19:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 912208E0005; Sat, 9 Mar 2019 20:19:14 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 89E598E0002; Sat, 9 Mar 2019 20:19:14 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 64EA28E0005; Sat, 9 Mar 2019 20:19:14 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) by kanga.kvack.org (Postfix) with ESMTP id 393288E0002 for ; Sat, 9 Mar 2019 20:19:14 -0500 (EST) Received: by mail-qt1-f200.google.com with SMTP id p5so1704629qtp.3 for ; Sat, 09 Mar 2019 17:19:14 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:date:in-reply-to:message-id :mime-version:references:subject:from:to:cc; bh=+4D1c6JWSWE8QT+yoi5CUv0ZKRIbRAhMzGEdnkssg9A=; b=bh9lZz1UP8MAGJbrgBj0SNbLtb3zjjBpIsMamy1Hp2NBsjoLY9q3rHbx05sWs2WnIe GNg4qBKFMDdtGvYOmHXu4XJLBByRV2iTuOmDAibhAc5nTRq2BmqlgRfaBj7WmQOSFd9T 2HDMPNcQ8OwxNptmb+MsmmuyEG3cdebaR4vP+ItG7wBAxLNFY8959BWuAQf/EvdfdC8d S791uF6QHiPcSEpskuZCzvVly3bvO2PbOCh2ug8qaPIP3Qbk4PF18sgbXQMOetJYd4i1 SSzQD54/dt5haAvko+KYRuCTJVD/EqmIR3Evu5g2A/BpWXY7EvKpLH8yjnCuaesHSjV/ 3Evw== X-Gm-Message-State: APjAAAUVG9RoJSLmlgYkoI4ejtXldlq+Ilt38KjEHMZENG3rbFvK8fj0 LnRas/OCllOwCJelkJQVOBiYcU32NARKIKNteSIWcTEGJveND3Owte3YL0a/5uQ7tWIRxOVILYK jk7T0bg05Gw+Q07UA+xZnXAIlr6jzJ2OO53Qnnxj0/5/LgNXWnq1YTR6OM2463P0JWMwExZq/6z JjHXyrxrx24Yn6l11HJOzHptfCM8ajNH3547cmGJGiIVgNXkfERgaZq2TZPLEMELCosQVME98y5 4qLD1C9JSYfPLWdQl3t4LupXuGEJJ04p8eqDg5FANJXjnV6LbOleJmCgjrPO0IoKXgIgCplTltc V5YImKEI8uRPoLUVk5pa1zqsnrYZzwiB+J07CBBGB2VSr69ztPF2UCIUurNGb+srNLTPX3fjZnM + X-Received: by 2002:ac8:29e8:: with SMTP id 37mr20906001qtt.153.1552180753970; Sat, 09 Mar 2019 17:19:13 -0800 (PST) X-Received: by 2002:ac8:29e8:: with SMTP id 37mr20905972qtt.153.1552180753312; Sat, 09 Mar 2019 17:19:13 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1552180753; cv=none; d=google.com; s=arc-20160816; b=Eu9PB13Ye4TB7r1JRUvl3eWTbbPo/ZlzAarWpR+DxV23hXWoWRsJyEl+c3GRwiRSQt 59Gl2YWwFH62yZoSq627WyejGqj/gPp8kzC2zjttnADqbblMkkOsHdjSy3RZWdzvLv6K 8c/gnj4rBoHVSuG/dQPCxXK3Rdx25mbR8PTK5tqoPSVeWcWXYNrvNknknEA9F4497keo xOenycNBdY5xTmF53j5PM4+wXriCvyfUnlZjvU8j3TWVygxnR+odvU7kGW1ewH17YXtF X7gwXVy/shE32i8Pbp81ZmqxxZ1M4SQTmfjWWIra9p2bV1tWBeT3suVuW542oUvhoWFf bXLw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=cc:to:from:subject:references:mime-version:message-id:in-reply-to :date:dkim-signature; bh=+4D1c6JWSWE8QT+yoi5CUv0ZKRIbRAhMzGEdnkssg9A=; b=bpMj79onHIpjrjYS2ku+Wg3AftT96EYq6UspXSxLs4Qbh6t7clmIqJPTBgs3VZnjy/ UaKYGZ9T0FDSHmrWMafiI4RWCvkto1/UbtPYv4RhhOVYVWjy2cucy9g2vOlB3VrPVxy7 jTaJNLxbdPYOT8IDIRw19yUUwxGHBRNX6myp62aWL+mX/xCM4jVIYtDS4FHxqWv9Hiym YPYMCsOAx5KR5+zOOgrcikJaxSENkR5RFsuO1u8xQKJKWPGYXgwMemFrmOVdEr9ePyK9 ME7QWp/z/bSVZLVDqgH1l665vjMNXkvxokFmsOpkB418HBlCkZ83kVpjTAEJtIA11wAy nDyw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=ne3qu3Fx; spf=pass (google.com: domain of 3ewaexaykcngsotb4iaiiaf8.6igfchor-ggep46e.ila@flex--yuzhao.bounces.google.com designates 209.85.220.73 as permitted sender) smtp.mailfrom=3EWaEXAYKCNgSOTB4IAIIAF8.6IGFCHOR-GGEP46E.ILA@flex--yuzhao.bounces.google.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com Received: from mail-sor-f73.google.com (mail-sor-f73.google.com. [209.85.220.73]) by mx.google.com with SMTPS id p10sor2239640qtf.13.2019.03.09.17.19.13 for (Google Transport Security); Sat, 09 Mar 2019 17:19:13 -0800 (PST) Received-SPF: pass (google.com: domain of 3ewaexaykcngsotb4iaiiaf8.6igfchor-ggep46e.ila@flex--yuzhao.bounces.google.com designates 209.85.220.73 as permitted sender) client-ip=209.85.220.73; Authentication-Results: mx.google.com; dkim=pass header.i=@google.com header.s=20161025 header.b=ne3qu3Fx; spf=pass (google.com: domain of 3ewaexaykcngsotb4iaiiaf8.6igfchor-ggep46e.ila@flex--yuzhao.bounces.google.com designates 209.85.220.73 as permitted sender) smtp.mailfrom=3EWaEXAYKCNgSOTB4IAIIAF8.6IGFCHOR-GGEP46E.ILA@flex--yuzhao.bounces.google.com; dmarc=pass (p=REJECT sp=REJECT dis=NONE) header.from=google.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=+4D1c6JWSWE8QT+yoi5CUv0ZKRIbRAhMzGEdnkssg9A=; b=ne3qu3FxuNnaxTHNaR0KvnSI+dxHillTroSbc2a23I9NeDzowB1PzvJmROLLgBDQkQ 4isyzzDBrnqmSk42wEbalVZQRdtGPCAh3G/B2qqSQtX3/BY5STPBC/0sf8VymLZxZnqv TFyb/EiMA0LE7HOd2jjMWMn2DLhBGtYXdC/SxMBWDrB+9bVk7z2HS1cqhsOLbpOYAqnq EWEylhDMVOFJ8n8MACWSvz57cSQhFCTOFpDGhLPApw5kwjO/4JcLuQBWDIBdVRtBa8MN 6AY17se4aFWkWszozRMszsYdPmcuPedaKhntA1DBHOnvx8MI+7RGhbs5ZQZ6UykgKoku kghg== X-Google-Smtp-Source: APXvYqzdCq/8JWJo7I6dgNrFQli0gtIFsccTph2MG2IS/kBQfoPBV4rNembCgH+TWILl3RUeNtEXGWzHp7A= X-Received: by 2002:ac8:1997:: with SMTP id u23mr15120422qtj.11.1552180753103; Sat, 09 Mar 2019 17:19:13 -0800 (PST) Date: Sat, 9 Mar 2019 18:19:06 -0700 In-Reply-To: <20190310011906.254635-1-yuzhao@google.com> Message-Id: <20190310011906.254635-3-yuzhao@google.com> Mime-Version: 1.0 References: <20190218231319.178224-1-yuzhao@google.com> <20190310011906.254635-1-yuzhao@google.com> X-Mailer: git-send-email 2.21.0.360.g471c308f928-goog Subject: [PATCH v3 3/3] arm64: mm: enable per pmd page table lock From: Yu Zhao To: Catalin Marinas , Will Deacon , Mark Rutland Cc: "Aneesh Kumar K . V" , Andrew Morton , Nick Piggin , Peter Zijlstra , Joel Fernandes , "Kirill A . Shutemov" , Ard Biesheuvel , Chintan Pandya , Jun Yao , Laura Abbott , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, Yu Zhao X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP Switch from per mm_struct to per pmd page table lock by enabling ARCH_ENABLE_SPLIT_PMD_PTLOCK. This provides better granularity for large system. I'm not sure if there is contention on mm->page_table_lock. Given the option comes at no cost (apart from initializing more spin locks), why not enable it now. We only do so when pmd is not folded, so we don't mistakenly call pgtable_pmd_page_ctor() on pud or p4d in pgd_pgtable_alloc(). (We check shift against PMD_SHIFT, which is same as PUD_SHIFT when pmd is folded). Signed-off-by: Yu Zhao --- arch/arm64/Kconfig | 3 +++ arch/arm64/include/asm/pgalloc.h | 12 +++++++++++- arch/arm64/include/asm/tlb.h | 5 ++++- 3 files changed, 18 insertions(+), 2 deletions(-) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index cfbf307d6dc4..a3b1b789f766 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -872,6 +872,9 @@ config ARCH_WANT_HUGE_PMD_SHARE config ARCH_HAS_CACHE_LINE_SIZE def_bool y +config ARCH_ENABLE_SPLIT_PMD_PTLOCK + def_bool y if PGTABLE_LEVELS > 2 + config SECCOMP bool "Enable seccomp to safely compute untrusted bytecode" ---help--- diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pgalloc.h index 52fa47c73bf0..dabba4b2c61f 100644 --- a/arch/arm64/include/asm/pgalloc.h +++ b/arch/arm64/include/asm/pgalloc.h @@ -33,12 +33,22 @@ static inline pmd_t *pmd_alloc_one(struct mm_struct *mm, unsigned long addr) { - return (pmd_t *)__get_free_page(PGALLOC_GFP); + struct page *page; + + page = alloc_page(PGALLOC_GFP); + if (!page) + return NULL; + if (!pgtable_pmd_page_ctor(page)) { + __free_page(page); + return NULL; + } + return page_address(page); } static inline void pmd_free(struct mm_struct *mm, pmd_t *pmdp) { BUG_ON((unsigned long)pmdp & (PAGE_SIZE-1)); + pgtable_pmd_page_dtor(virt_to_page(pmdp)); free_page((unsigned long)pmdp); } diff --git a/arch/arm64/include/asm/tlb.h b/arch/arm64/include/asm/tlb.h index 106fdc951b6e..4e3becfed387 100644 --- a/arch/arm64/include/asm/tlb.h +++ b/arch/arm64/include/asm/tlb.h @@ -62,7 +62,10 @@ static inline void __pte_free_tlb(struct mmu_gather *tlb, pgtable_t pte, static inline void __pmd_free_tlb(struct mmu_gather *tlb, pmd_t *pmdp, unsigned long addr) { - tlb_remove_table(tlb, virt_to_page(pmdp)); + struct page *page = virt_to_page(pmdp); + + pgtable_pmd_page_dtor(page); + tlb_remove_table(tlb, page); } #endif