From patchwork Sun Jan 24 08:22:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 12042147 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 223BFC433E0 for ; Sun, 24 Jan 2021 08:23:02 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 91F7722582 for ; Sun, 24 Jan 2021 08:23:01 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 91F7722582 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 10B826B0008; Sun, 24 Jan 2021 03:23:01 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0BF246B000A; Sun, 24 Jan 2021 03:23:01 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EC44D6B000C; Sun, 24 Jan 2021 03:23:00 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0229.hostedemail.com [216.40.44.229]) by kanga.kvack.org (Postfix) with ESMTP id D2DE16B0008 for ; Sun, 24 Jan 2021 03:23:00 -0500 (EST) Received: from smtpin13.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 8A7121EE6 for ; Sun, 24 Jan 2021 08:23:00 +0000 (UTC) X-FDA: 77739978120.13.birth84_0701d7e2757b Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin13.hostedemail.com (Postfix) with ESMTP id 6D64C18140B7B for ; Sun, 24 Jan 2021 08:23:00 +0000 (UTC) X-HE-Tag: birth84_0701d7e2757b X-Filterd-Recvd-Size: 6529 Received: from mail-pl1-f170.google.com (mail-pl1-f170.google.com [209.85.214.170]) by imf39.hostedemail.com (Postfix) with ESMTP for ; Sun, 24 Jan 2021 08:22:59 +0000 (UTC) Received: by mail-pl1-f170.google.com with SMTP id q2so2976025plk.4 for ; Sun, 24 Jan 2021 00:22:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=MjsjuAh+yTiFeI37LQ2ROLPGWlcSvFIMmiFEgOfHCAQ=; b=eoMW7GTVmr9o4bjF6Osj2CJExu/gpEdeNUUo1N+H3CelvPOleQu5cGtPoy2hMTKGQO iPWlEV6sb9Bkf/WxuWioVBu/Zf2xX/zuX6ec40GxcYSBHL08MDoqew6msLSgnv1m97Ri IOIwpbwjdWXqZeszC25ZkbyMeRlx+QsLLA90LzSryZnekHySSvJ6OAdJS52ZID+dLvXB olweo3yleJ3Mi97DWIyC5cYehEu80M3+BAU6/TF8bcuZGVZURMpq3XRZd26h8h2LFlun 2PnCuEhsytdNU+gGBrohRKdVTdqfA2ULKM3/9dHO0rWFYGKCvOH4EeA+/AIrVsTECyY2 jq+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=MjsjuAh+yTiFeI37LQ2ROLPGWlcSvFIMmiFEgOfHCAQ=; b=D7bQIUAKE/YXVI8nx9Cqv4c1TEms7+wP/BM1IENFHNwsNn1jTnNVv20PqMbwnj1skV EBqb1fMLQbpTKHlbZjnCwgW2YS/DqwUqPFoGxiP93Mst5BzYMOW7Hjs1nPuvEzV9dbE9 V7vgHnicQgjaRUpj3klXOVKldHBid2anPMakAFYMnv9pe9Cl1EMded4EpNzdxp3SNTZM 6VEk3vRO1Zb6Z0ImV8jaVj3wvcmUwcDTcfEqtbHOmQwb1tLwEkNERL1QnumXW8cPUzHt 11gZN2tl+8PblMgD4QZfONRnpCFvXer210mCG9HYpst5sdTk7vLirUt7SWGWjzPJ1nUu ckig== X-Gm-Message-State: AOAM532Bg8MdDiA9VbojPLNJLex/ntg5NDQab4R3G2DeEv2toKWp/ak8 bZ+6p3iOT+O4u/vl4GLDT1kOIzmGbcw= X-Google-Smtp-Source: ABdhPJw9eMUNU9DyqDg8M16Otuc3OE05frnBZNYjYRp7v9azdMe2JzhBv/W8zetDEcrCuHdciKeZ9A== X-Received: by 2002:a17:90a:d189:: with SMTP id fu9mr15168573pjb.98.1611476578988; Sun, 24 Jan 2021 00:22:58 -0800 (PST) Received: from bobo.ozlabs.ibm.com ([124.170.13.62]) by smtp.gmail.com with ESMTPSA id gb12sm11799757pjb.51.2021.01.24.00.22.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 24 Jan 2021 00:22:58 -0800 (PST) From: Nicholas Piggin To: linux-mm@kvack.org, Andrew Morton Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Zefan Li , Jonathan Cameron , Christoph Hellwig , Christophe Leroy , Rick Edgecombe , Ding Tianhong Subject: [PATCH v10 01/12] mm/vmalloc: fix vmalloc_to_page for huge vmap mappings Date: Sun, 24 Jan 2021 18:22:19 +1000 Message-Id: <20210124082230.2118861-2-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20210124082230.2118861-1-npiggin@gmail.com> References: <20210124082230.2118861-1-npiggin@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: vmalloc_to_page returns NULL for addresses mapped by larger pages[*]. Whether or not a vmap is huge depends on the architecture details, alignments, boot options, etc., which the caller can not be expected to know. Therefore HUGE_VMAP is a regression for vmalloc_to_page. This change teaches vmalloc_to_page about larger pages, and returns the struct page that corresponds to the offset within the large page. This makes the API agnostic to mapping implementation details. [*] As explained by commit 029c54b095995 ("mm/vmalloc.c: huge-vmap: fail gracefully on unexpected huge vmap mappings") Signed-off-by: Nicholas Piggin Reviewed-by: Christoph Hellwig --- mm/vmalloc.c | 41 ++++++++++++++++++++++++++--------------- 1 file changed, 26 insertions(+), 15 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index e6f352bf0498..62372f9e0167 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -34,7 +34,7 @@ #include #include #include - +#include #include #include #include @@ -343,7 +343,9 @@ int is_vmalloc_or_module_addr(const void *x) } /* - * Walk a vmap address to the struct page it maps. + * Walk a vmap address to the struct page it maps. Huge vmap mappings will + * return the tail page that corresponds to the base page address, which + * matches small vmap mappings. */ struct page *vmalloc_to_page(const void *vmalloc_addr) { @@ -363,25 +365,33 @@ struct page *vmalloc_to_page(const void *vmalloc_addr) if (pgd_none(*pgd)) return NULL; + if (WARN_ON_ONCE(pgd_leaf(*pgd))) + return NULL; /* XXX: no allowance for huge pgd */ + if (WARN_ON_ONCE(pgd_bad(*pgd))) + return NULL; + p4d = p4d_offset(pgd, addr); if (p4d_none(*p4d)) return NULL; - pud = pud_offset(p4d, addr); + if (p4d_leaf(*p4d)) + return p4d_page(*p4d) + ((addr & ~P4D_MASK) >> PAGE_SHIFT); + if (WARN_ON_ONCE(p4d_bad(*p4d))) + return NULL; - /* - * Don't dereference bad PUD or PMD (below) entries. This will also - * identify huge mappings, which we may encounter on architectures - * that define CONFIG_HAVE_ARCH_HUGE_VMAP=y. Such regions will be - * identified as vmalloc addresses by is_vmalloc_addr(), but are - * not [unambiguously] associated with a struct page, so there is - * no correct value to return for them. - */ - WARN_ON_ONCE(pud_bad(*pud)); - if (pud_none(*pud) || pud_bad(*pud)) + pud = pud_offset(p4d, addr); + if (pud_none(*pud)) + return NULL; + if (pud_leaf(*pud)) + return pud_page(*pud) + ((addr & ~PUD_MASK) >> PAGE_SHIFT); + if (WARN_ON_ONCE(pud_bad(*pud))) return NULL; + pmd = pmd_offset(pud, addr); - WARN_ON_ONCE(pmd_bad(*pmd)); - if (pmd_none(*pmd) || pmd_bad(*pmd)) + if (pmd_none(*pmd)) + return NULL; + if (pmd_leaf(*pmd)) + return pmd_page(*pmd) + ((addr & ~PMD_MASK) >> PAGE_SHIFT); + if (WARN_ON_ONCE(pmd_bad(*pmd))) return NULL; ptep = pte_offset_map(pmd, addr); @@ -389,6 +399,7 @@ struct page *vmalloc_to_page(const void *vmalloc_addr) if (pte_present(pte)) page = pte_page(pte); pte_unmap(ptep); + return page; } EXPORT_SYMBOL(vmalloc_to_page); From patchwork Sun Jan 24 08:22:20 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 12042149 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E29B9C433E0 for ; Sun, 24 Jan 2021 08:23:06 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 88AE422582 for ; Sun, 24 Jan 2021 08:23:06 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 88AE422582 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 2698B6B000A; Sun, 24 Jan 2021 03:23:06 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1F25B6B000C; Sun, 24 Jan 2021 03:23:06 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0E1616B000D; Sun, 24 Jan 2021 03:23:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0243.hostedemail.com [216.40.44.243]) by kanga.kvack.org (Postfix) with ESMTP id E8B916B000A for ; Sun, 24 Jan 2021 03:23:05 -0500 (EST) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id ABD808249980 for ; Sun, 24 Jan 2021 08:23:05 +0000 (UTC) X-FDA: 77739978330.19.peace40_2b062522757b Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin19.hostedemail.com (Postfix) with ESMTP id 91B1B1ACEA4 for ; Sun, 24 Jan 2021 08:23:05 +0000 (UTC) X-HE-Tag: peace40_2b062522757b X-Filterd-Recvd-Size: 6639 Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) by imf33.hostedemail.com (Postfix) with ESMTP for ; Sun, 24 Jan 2021 08:23:05 +0000 (UTC) Received: by mail-pl1-f176.google.com with SMTP id b8so5764666plx.0 for ; Sun, 24 Jan 2021 00:23:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=/x6cdpjaOH33+ELSrGwY3sQ3qRgzggzyFztG5hnhdsk=; b=aO4gEjmWuAuh0gZMGkfqR1QuVFpQRIH3UxpYZqGzPJi9zwsdd163c6MNj6YBKia+ZQ xaV2O1fDHJGBjZkyX1Xfps2OADbM1Seeotu2I7jMQkobgb5Dt2hquvM6Uci/8mrklLGi E3vNbCOlcObgpoQV0H1VtQUDw0Z9L4hzJPcTQ/g+XwFU5xpQdzmuAu8yuEkwkVfij7Vb 1v8AQi8CTlNtZZXthdGd+fm3tikUnrkoadMRBh/RdkuULLxDkVQRN3tU2QDzq1ZjegOY /Ctv4/l98syX528uaxkOcfQ2sCkUrorQ7xGxJ5LJRFAIenRQfUDOpKe2hOeJ6UUP9r5s 444w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=/x6cdpjaOH33+ELSrGwY3sQ3qRgzggzyFztG5hnhdsk=; b=ZtYjnhhX6D+RPdPn46XLcFmoV9ewXfiV2DojehE6AM04TiX3qriAuMb0zLiknR//7a E24lMvRtOswZs4i1d7ewrgUp40XrKCfhreZIuZ2tz2gKR8zSNvIMrK0ZYBc4R1oo76NO NNLaRDEREtEFVpPVSTOQlU38t0KLFib1Y2NRJbP5QETPa8D62/jBhoB6YwCyNzKpeCaS e0AQdxxB2tITQuPOB6ba9ePcfqfMlG64QkP117ldZklTWQbRia1Z0IirGpOd3ZtftJHB ZnPbWb7zwuqyry2CMLHJpRSNujght2+QgAVXLBDwMmaUDnY+Og3lX6xBlAQfX1NeQfeO 35/A== X-Gm-Message-State: AOAM530yOWd/CP5SItm+UtFAktvCAit8Z5qN0ZMAU3k/nJIozAlN3Fk5 l8rdkIfLna9cyn4ig6ee1GCy2sm1qQs= X-Google-Smtp-Source: ABdhPJyrsytE/NMqNg0eGkiocOJPB9OIhVhalaGRXuMoTtXPLltyqIr5T+D2zqmrI8yVhUBnUVlz2Q== X-Received: by 2002:a17:902:e80c:b029:de:a20b:7a9c with SMTP id u12-20020a170902e80cb02900dea20b7a9cmr13712689plg.12.1611476584165; Sun, 24 Jan 2021 00:23:04 -0800 (PST) Received: from bobo.ozlabs.ibm.com ([124.170.13.62]) by smtp.gmail.com with ESMTPSA id gb12sm11799757pjb.51.2021.01.24.00.22.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 24 Jan 2021 00:23:03 -0800 (PST) From: Nicholas Piggin To: linux-mm@kvack.org, Andrew Morton Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Zefan Li , Jonathan Cameron , Christoph Hellwig , Christophe Leroy , Rick Edgecombe , Ding Tianhong Subject: [PATCH v10 02/12] mm: apply_to_pte_range warn and fail if a large pte is encountered Date: Sun, 24 Jan 2021 18:22:20 +1000 Message-Id: <20210124082230.2118861-3-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20210124082230.2118861-1-npiggin@gmail.com> References: <20210124082230.2118861-1-npiggin@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: apply_to_pte_range might mistake a large pte for bad, or treat it as a page table, resulting in a crash or corruption. Add a test to warn and return error if large entries are found. Signed-off-by: Nicholas Piggin Reviewed-by: Christoph Hellwig --- mm/memory.c | 66 +++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 49 insertions(+), 17 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index feff48e1465a..672e39a72788 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -2440,13 +2440,21 @@ static int apply_to_pmd_range(struct mm_struct *mm, pud_t *pud, } do { next = pmd_addr_end(addr, end); - if (create || !pmd_none_or_clear_bad(pmd)) { - err = apply_to_pte_range(mm, pmd, addr, next, fn, data, - create, mask); - if (err) - break; + if (pmd_none(*pmd) && !create) + continue; + if (WARN_ON_ONCE(pmd_leaf(*pmd))) + return -EINVAL; + if (!pmd_none(*pmd) && WARN_ON_ONCE(pmd_bad(*pmd))) { + if (!create) + continue; + pmd_clear_bad(pmd); } + err = apply_to_pte_range(mm, pmd, addr, next, + fn, data, create, mask); + if (err) + break; } while (pmd++, addr = next, addr != end); + return err; } @@ -2468,13 +2476,21 @@ static int apply_to_pud_range(struct mm_struct *mm, p4d_t *p4d, } do { next = pud_addr_end(addr, end); - if (create || !pud_none_or_clear_bad(pud)) { - err = apply_to_pmd_range(mm, pud, addr, next, fn, data, - create, mask); - if (err) - break; + if (pud_none(*pud) && !create) + continue; + if (WARN_ON_ONCE(pud_leaf(*pud))) + return -EINVAL; + if (!pud_none(*pud) && WARN_ON_ONCE(pud_bad(*pud))) { + if (!create) + continue; + pud_clear_bad(pud); } + err = apply_to_pmd_range(mm, pud, addr, next, + fn, data, create, mask); + if (err) + break; } while (pud++, addr = next, addr != end); + return err; } @@ -2496,13 +2512,21 @@ static int apply_to_p4d_range(struct mm_struct *mm, pgd_t *pgd, } do { next = p4d_addr_end(addr, end); - if (create || !p4d_none_or_clear_bad(p4d)) { - err = apply_to_pud_range(mm, p4d, addr, next, fn, data, - create, mask); - if (err) - break; + if (p4d_none(*p4d) && !create) + continue; + if (WARN_ON_ONCE(p4d_leaf(*p4d))) + return -EINVAL; + if (!p4d_none(*p4d) && WARN_ON_ONCE(p4d_bad(*p4d))) { + if (!create) + continue; + p4d_clear_bad(p4d); } + err = apply_to_pud_range(mm, p4d, addr, next, + fn, data, create, mask); + if (err) + break; } while (p4d++, addr = next, addr != end); + return err; } @@ -2522,9 +2546,17 @@ static int __apply_to_page_range(struct mm_struct *mm, unsigned long addr, pgd = pgd_offset(mm, addr); do { next = pgd_addr_end(addr, end); - if (!create && pgd_none_or_clear_bad(pgd)) + if (pgd_none(*pgd) && !create) continue; - err = apply_to_p4d_range(mm, pgd, addr, next, fn, data, create, &mask); + if (WARN_ON_ONCE(pgd_leaf(*pgd))) + return -EINVAL; + if (!pgd_none(*pgd) && WARN_ON_ONCE(pgd_bad(*pgd))) { + if (!create) + continue; + pgd_clear_bad(pgd); + } + err = apply_to_p4d_range(mm, pgd, addr, next, + fn, data, create, &mask); if (err) break; } while (pgd++, addr = next, addr != end); From patchwork Sun Jan 24 08:22:21 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 12042151 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0D932C433DB for ; Sun, 24 Jan 2021 08:23:12 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9BD2A22582 for ; Sun, 24 Jan 2021 08:23:11 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9BD2A22582 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 37B1B6B000C; Sun, 24 Jan 2021 03:23:11 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 303FB6B000D; Sun, 24 Jan 2021 03:23:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1F2F36B000E; Sun, 24 Jan 2021 03:23:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0100.hostedemail.com [216.40.44.100]) by kanga.kvack.org (Postfix) with ESMTP id 0374A6B000C for ; Sun, 24 Jan 2021 03:23:10 -0500 (EST) Received: from smtpin30.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id B990F1EE6 for ; Sun, 24 Jan 2021 08:23:10 +0000 (UTC) X-FDA: 77739978540.30.lead37_6212af12757b Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin30.hostedemail.com (Postfix) with ESMTP id 9675D180B3C96 for ; Sun, 24 Jan 2021 08:23:10 +0000 (UTC) X-HE-Tag: lead37_6212af12757b X-Filterd-Recvd-Size: 6302 Received: from mail-pj1-f48.google.com (mail-pj1-f48.google.com [209.85.216.48]) by imf41.hostedemail.com (Postfix) with ESMTP for ; Sun, 24 Jan 2021 08:23:10 +0000 (UTC) Received: by mail-pj1-f48.google.com with SMTP id cq1so6440771pjb.4 for ; Sun, 24 Jan 2021 00:23:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=X5+HXwlV+fH03hyDjK/J9gg0LI2GnzY0puUeRChOcaU=; b=FJfdd+BvQdR6QZkiZ000WG4imKauyp2TKUd7ov3ARyWxKn9AdZ+Iu4WtGCe5ie6eDS vVUlTE6CnRm3Tadsj4m7umH0Ualy1+haHVe8kCZNtzl9/N4AfiAvBwK7MM8SVDRkcxx2 uIFx9idE1UVJeHCqo40pWH9whW3qPus1jrBhVSn7VsTMtSoq5vEiAKcLtbvAsvScqrMZ jd+hUZtOerguD/k/PIucqPjZFFFSamH4N15v7pMpk+eRk5kpVc+M0lyg9OPkFFsDD6tv EobQG4c2j9PZ8ppw8yGmN596LFVFT4SFzF6UBSPS8AGKM8et8PuJoy7zc8LxFndX5Mt2 rsjw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=X5+HXwlV+fH03hyDjK/J9gg0LI2GnzY0puUeRChOcaU=; b=VHzzTRqyHkOiuvaF8m/zHGxnjvpyHlQmkMR3Qn3ErjSzz4uoYyL9oxOyxp0l1sFR2Q u6DjJPay0AVHn9oSnZkSElCKwPWtPl74S4aFRaZH+RNOF06UxnwcgQO5089Tj7DKb87W b5k6Md3cod4gbXt4U+IWRpOx6yO6TkPIh+wIDxNQkLy4wfyyRWfHuJh2qEUteGwXApQS cCDQ6STl4fPXM2YeM3e3+7EHgxz8Ri6yy+I9vZ0NdG+3ZjTq1k1uPDLm4ejTcqsgC+p3 8S3GoIt57vaGfCWZtJVYH+KovHyg5jF+4o4uyMt8PuBtrnIlSilg1xIoPu4UFhRFN+rk oaDQ== X-Gm-Message-State: AOAM530U9tclZfC/+WZdEbdbePDSveL/5geMIR6sqwzJdyyVUHXFtGCs juT5NAiq9WaK+CNe0hI5B6YDWFZ+n9o= X-Google-Smtp-Source: ABdhPJxD5UwSXKWxq077HcNekHjHCxP9n4SGcDoENgLvY1RR9GktbirNa0o0jUS4EhthEUeob+SvKw== X-Received: by 2002:a17:90a:ce10:: with SMTP id f16mr1347598pju.136.1611476589276; Sun, 24 Jan 2021 00:23:09 -0800 (PST) Received: from bobo.ozlabs.ibm.com ([124.170.13.62]) by smtp.gmail.com with ESMTPSA id gb12sm11799757pjb.51.2021.01.24.00.23.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 24 Jan 2021 00:23:08 -0800 (PST) From: Nicholas Piggin To: linux-mm@kvack.org, Andrew Morton Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Zefan Li , Jonathan Cameron , Christoph Hellwig , Christophe Leroy , Rick Edgecombe , Ding Tianhong Subject: [PATCH v10 03/12] mm/vmalloc: rename vmap_*_range vmap_pages_*_range Date: Sun, 24 Jan 2021 18:22:21 +1000 Message-Id: <20210124082230.2118861-4-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20210124082230.2118861-1-npiggin@gmail.com> References: <20210124082230.2118861-1-npiggin@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The vmalloc mapper operates on a struct page * array rather than a linear physical address, re-name it to make this distinction clear. Signed-off-by: Nicholas Piggin Reviewed-by: Christoph Hellwig --- mm/vmalloc.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 62372f9e0167..7f2f36116980 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -189,7 +189,7 @@ void unmap_kernel_range_noflush(unsigned long start, unsigned long size) arch_sync_kernel_mappings(start, end); } -static int vmap_pte_range(pmd_t *pmd, unsigned long addr, +static int vmap_pages_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, int *nr, pgtbl_mod_mask *mask) { @@ -217,7 +217,7 @@ static int vmap_pte_range(pmd_t *pmd, unsigned long addr, return 0; } -static int vmap_pmd_range(pud_t *pud, unsigned long addr, +static int vmap_pages_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, int *nr, pgtbl_mod_mask *mask) { @@ -229,13 +229,13 @@ static int vmap_pmd_range(pud_t *pud, unsigned long addr, return -ENOMEM; do { next = pmd_addr_end(addr, end); - if (vmap_pte_range(pmd, addr, next, prot, pages, nr, mask)) + if (vmap_pages_pte_range(pmd, addr, next, prot, pages, nr, mask)) return -ENOMEM; } while (pmd++, addr = next, addr != end); return 0; } -static int vmap_pud_range(p4d_t *p4d, unsigned long addr, +static int vmap_pages_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, int *nr, pgtbl_mod_mask *mask) { @@ -247,13 +247,13 @@ static int vmap_pud_range(p4d_t *p4d, unsigned long addr, return -ENOMEM; do { next = pud_addr_end(addr, end); - if (vmap_pmd_range(pud, addr, next, prot, pages, nr, mask)) + if (vmap_pages_pmd_range(pud, addr, next, prot, pages, nr, mask)) return -ENOMEM; } while (pud++, addr = next, addr != end); return 0; } -static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, +static int vmap_pages_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, int *nr, pgtbl_mod_mask *mask) { @@ -265,7 +265,7 @@ static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, return -ENOMEM; do { next = p4d_addr_end(addr, end); - if (vmap_pud_range(p4d, addr, next, prot, pages, nr, mask)) + if (vmap_pages_pud_range(p4d, addr, next, prot, pages, nr, mask)) return -ENOMEM; } while (p4d++, addr = next, addr != end); return 0; @@ -306,7 +306,7 @@ int map_kernel_range_noflush(unsigned long addr, unsigned long size, next = pgd_addr_end(addr, end); if (pgd_bad(*pgd)) mask |= PGTBL_PGD_MODIFIED; - err = vmap_p4d_range(pgd, addr, next, prot, pages, &nr, &mask); + err = vmap_pages_p4d_range(pgd, addr, next, prot, pages, &nr, &mask); if (err) return err; } while (pgd++, addr = next, addr != end); From patchwork Sun Jan 24 08:22:22 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 12042153 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4C3E1C433DB for ; Sun, 24 Jan 2021 08:23:17 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E52B522252 for ; Sun, 24 Jan 2021 08:23:16 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E52B522252 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 85CAD6B000D; Sun, 24 Jan 2021 03:23:16 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 7E7786B000E; Sun, 24 Jan 2021 03:23:16 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6D4936B0012; Sun, 24 Jan 2021 03:23:16 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0232.hostedemail.com [216.40.44.232]) by kanga.kvack.org (Postfix) with ESMTP id 553EB6B000D for ; Sun, 24 Jan 2021 03:23:16 -0500 (EST) Received: from smtpin10.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 1F3D48249980 for ; Sun, 24 Jan 2021 08:23:16 +0000 (UTC) X-FDA: 77739978792.10.pull68_410a54e2757b Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin10.hostedemail.com (Postfix) with ESMTP id EB60816A4B1 for ; Sun, 24 Jan 2021 08:23:15 +0000 (UTC) X-HE-Tag: pull68_410a54e2757b X-Filterd-Recvd-Size: 9404 Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) by imf11.hostedemail.com (Postfix) with ESMTP for ; Sun, 24 Jan 2021 08:23:15 +0000 (UTC) Received: by mail-pl1-f178.google.com with SMTP id u11so5714213plg.13 for ; Sun, 24 Jan 2021 00:23:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=0Y/hT12bvV3dyctfA7PHWDL72BJBoDG5OIVFWS0o/KA=; b=Pa70CJQQElqUqOtQAq+G6CtPUrASz+vMAIlZA4qtaE1zD3ZFnFt1kOICn2NNz1XrEJ 8eZxK3p9b6KfR0XVPGAko6aht/ZVHip47T+i9WXnj3FMJrzHnKe1oPXP9bdJkpU9H8rm hod9wAqNtOH+ArRFWmK2CLs6I5N4d8h5LVrBM2VGOq+HvsgodnqJ2udzkklkNqoP2d6H O4oRaGsIPrEGFGL42kXhoWiy3uN64VviWYk50bqgzMDdXoKR2RmgcqCvQ2DYzsEQ06ax PbHXeeNANBM6u2tJpWwM3I7moN8fzWIIBJh8VV0FOacIijdhW2rEe0zjjSRoToprhPhW TMng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=0Y/hT12bvV3dyctfA7PHWDL72BJBoDG5OIVFWS0o/KA=; b=XNKplSOlOZJNdorosAFceAQjEQJCcUCUZU+WQ0mNwPRZCnx2VmulgjMdTqhBDqQUIl aQGianZnRIBZ2vW2vAfoste+hK6COkc8gYjjqrwe6Zj1eZfLOGSWeMn2D/Iw33x7NHM4 Dj8nGa30QrZnJZJSfZzLjQYrEzgBWFkKpEeVvVPDdwmIOSWiUv03ylnXyFqEQa7J8khE /fIdxg/B+sqtpG7t1u4YRHvfIQnTOfh5OOvteb0nT/Zx6dBsrAa69BroIDXNc/64IKTJ Qm0lWpJMmWBspni6KLE+ROhrHiZpRqE+rja6Tng6bbwpQ/Y9HwmQKWrbLF+vG04br1yK /ZDA== X-Gm-Message-State: AOAM533MkwV/DVh9Fx1dxlZmsGbg5DPfgMeyYn+O4S3wAjgvYBpkbmwP uUb0WIKBrRt/p7UVJUc3JRcPr8L1SLQ= X-Google-Smtp-Source: ABdhPJzJtg7jQQvosOIP/XkAdGf9EwFgBGrzT1C4WiO31v/tg/fKzBcOJcLi5f781HaEIq7zY+o8QA== X-Received: by 2002:a17:90b:1a87:: with SMTP id ng7mr370673pjb.211.1611476594481; Sun, 24 Jan 2021 00:23:14 -0800 (PST) Received: from bobo.ozlabs.ibm.com ([124.170.13.62]) by smtp.gmail.com with ESMTPSA id gb12sm11799757pjb.51.2021.01.24.00.23.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 24 Jan 2021 00:23:14 -0800 (PST) From: Nicholas Piggin To: linux-mm@kvack.org, Andrew Morton Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Zefan Li , Jonathan Cameron , Christoph Hellwig , Christophe Leroy , Rick Edgecombe , Ding Tianhong Subject: [PATCH v10 04/12] mm/ioremap: rename ioremap_*_range to vmap_*_range Date: Sun, 24 Jan 2021 18:22:22 +1000 Message-Id: <20210124082230.2118861-5-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20210124082230.2118861-1-npiggin@gmail.com> References: <20210124082230.2118861-1-npiggin@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This will be used as a generic kernel virtual mapping function, so re-name it in preparation. Signed-off-by: Nicholas Piggin --- mm/ioremap.c | 64 +++++++++++++++++++++++++++------------------------- 1 file changed, 33 insertions(+), 31 deletions(-) diff --git a/mm/ioremap.c b/mm/ioremap.c index 5fa1ab41d152..3f4d36f9745a 100644 --- a/mm/ioremap.c +++ b/mm/ioremap.c @@ -61,9 +61,9 @@ static inline int ioremap_pud_enabled(void) { return 0; } static inline int ioremap_pmd_enabled(void) { return 0; } #endif /* CONFIG_HAVE_ARCH_HUGE_VMAP */ -static int ioremap_pte_range(pmd_t *pmd, unsigned long addr, - unsigned long end, phys_addr_t phys_addr, pgprot_t prot, - pgtbl_mod_mask *mask) +static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + pgtbl_mod_mask *mask) { pte_t *pte; u64 pfn; @@ -81,9 +81,8 @@ static int ioremap_pte_range(pmd_t *pmd, unsigned long addr, return 0; } -static int ioremap_try_huge_pmd(pmd_t *pmd, unsigned long addr, - unsigned long end, phys_addr_t phys_addr, - pgprot_t prot) +static int vmap_try_huge_pmd(pmd_t *pmd, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot) { if (!ioremap_pmd_enabled()) return 0; @@ -103,9 +102,9 @@ static int ioremap_try_huge_pmd(pmd_t *pmd, unsigned long addr, return pmd_set_huge(pmd, phys_addr, prot); } -static inline int ioremap_pmd_range(pud_t *pud, unsigned long addr, - unsigned long end, phys_addr_t phys_addr, pgprot_t prot, - pgtbl_mod_mask *mask) +static int vmap_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + pgtbl_mod_mask *mask) { pmd_t *pmd; unsigned long next; @@ -116,20 +115,19 @@ static inline int ioremap_pmd_range(pud_t *pud, unsigned long addr, do { next = pmd_addr_end(addr, end); - if (ioremap_try_huge_pmd(pmd, addr, next, phys_addr, prot)) { + if (vmap_try_huge_pmd(pmd, addr, next, phys_addr, prot)) { *mask |= PGTBL_PMD_MODIFIED; continue; } - if (ioremap_pte_range(pmd, addr, next, phys_addr, prot, mask)) + if (vmap_pte_range(pmd, addr, next, phys_addr, prot, mask)) return -ENOMEM; } while (pmd++, phys_addr += (next - addr), addr = next, addr != end); return 0; } -static int ioremap_try_huge_pud(pud_t *pud, unsigned long addr, - unsigned long end, phys_addr_t phys_addr, - pgprot_t prot) +static int vmap_try_huge_pud(pud_t *pud, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot) { if (!ioremap_pud_enabled()) return 0; @@ -149,9 +147,9 @@ static int ioremap_try_huge_pud(pud_t *pud, unsigned long addr, return pud_set_huge(pud, phys_addr, prot); } -static inline int ioremap_pud_range(p4d_t *p4d, unsigned long addr, - unsigned long end, phys_addr_t phys_addr, pgprot_t prot, - pgtbl_mod_mask *mask) +static int vmap_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + pgtbl_mod_mask *mask) { pud_t *pud; unsigned long next; @@ -162,20 +160,19 @@ static inline int ioremap_pud_range(p4d_t *p4d, unsigned long addr, do { next = pud_addr_end(addr, end); - if (ioremap_try_huge_pud(pud, addr, next, phys_addr, prot)) { + if (vmap_try_huge_pud(pud, addr, next, phys_addr, prot)) { *mask |= PGTBL_PUD_MODIFIED; continue; } - if (ioremap_pmd_range(pud, addr, next, phys_addr, prot, mask)) + if (vmap_pmd_range(pud, addr, next, phys_addr, prot, mask)) return -ENOMEM; } while (pud++, phys_addr += (next - addr), addr = next, addr != end); return 0; } -static int ioremap_try_huge_p4d(p4d_t *p4d, unsigned long addr, - unsigned long end, phys_addr_t phys_addr, - pgprot_t prot) +static int vmap_try_huge_p4d(p4d_t *p4d, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot) { if (!ioremap_p4d_enabled()) return 0; @@ -195,9 +192,9 @@ static int ioremap_try_huge_p4d(p4d_t *p4d, unsigned long addr, return p4d_set_huge(p4d, phys_addr, prot); } -static inline int ioremap_p4d_range(pgd_t *pgd, unsigned long addr, - unsigned long end, phys_addr_t phys_addr, pgprot_t prot, - pgtbl_mod_mask *mask) +static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + pgtbl_mod_mask *mask) { p4d_t *p4d; unsigned long next; @@ -208,19 +205,19 @@ static inline int ioremap_p4d_range(pgd_t *pgd, unsigned long addr, do { next = p4d_addr_end(addr, end); - if (ioremap_try_huge_p4d(p4d, addr, next, phys_addr, prot)) { + if (vmap_try_huge_p4d(p4d, addr, next, phys_addr, prot)) { *mask |= PGTBL_P4D_MODIFIED; continue; } - if (ioremap_pud_range(p4d, addr, next, phys_addr, prot, mask)) + if (vmap_pud_range(p4d, addr, next, phys_addr, prot, mask)) return -ENOMEM; } while (p4d++, phys_addr += (next - addr), addr = next, addr != end); return 0; } -int ioremap_page_range(unsigned long addr, - unsigned long end, phys_addr_t phys_addr, pgprot_t prot) +static int vmap_range(unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot) { pgd_t *pgd; unsigned long start; @@ -235,8 +232,7 @@ int ioremap_page_range(unsigned long addr, pgd = pgd_offset_k(addr); do { next = pgd_addr_end(addr, end); - err = ioremap_p4d_range(pgd, addr, next, phys_addr, prot, - &mask); + err = vmap_p4d_range(pgd, addr, next, phys_addr, prot, &mask); if (err) break; } while (pgd++, phys_addr += (next - addr), addr = next, addr != end); @@ -249,6 +245,12 @@ int ioremap_page_range(unsigned long addr, return err; } +int ioremap_page_range(unsigned long addr, + unsigned long end, phys_addr_t phys_addr, pgprot_t prot) +{ + return vmap_range(addr, end, phys_addr, prot); +} + #ifdef CONFIG_GENERIC_IOREMAP void __iomem *ioremap_prot(phys_addr_t addr, size_t size, unsigned long prot) { From patchwork Sun Jan 24 08:22:23 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 12042155 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id ACCCEC433E0 for ; Sun, 24 Jan 2021 08:23:25 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 43C7722582 for ; Sun, 24 Jan 2021 08:23:25 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 43C7722582 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C7AD36B000E; Sun, 24 Jan 2021 03:23:24 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C057A6B0012; Sun, 24 Jan 2021 03:23:24 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A30046B0022; Sun, 24 Jan 2021 03:23:24 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0252.hostedemail.com [216.40.44.252]) by kanga.kvack.org (Postfix) with ESMTP id 841416B000E for ; Sun, 24 Jan 2021 03:23:24 -0500 (EST) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 4EC65363F for ; Sun, 24 Jan 2021 08:23:24 +0000 (UTC) X-FDA: 77739979128.25.name69_4f143ba2757b Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin25.hostedemail.com (Postfix) with ESMTP id 2E6871804E3B2 for ; Sun, 24 Jan 2021 08:23:24 +0000 (UTC) X-HE-Tag: name69_4f143ba2757b X-Filterd-Recvd-Size: 17533 Received: from mail-pf1-f173.google.com (mail-pf1-f173.google.com [209.85.210.173]) by imf27.hostedemail.com (Postfix) with ESMTP for ; Sun, 24 Jan 2021 08:23:23 +0000 (UTC) Received: by mail-pf1-f173.google.com with SMTP id 11so6598789pfu.4 for ; Sun, 24 Jan 2021 00:23:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=wrd9eNTrh6D6J8rGXcFB2/d+ZBemO5j2BbiFByaG1Bg=; b=DMRc8Nt1R2xShApvaTYwoyBDVoy9hUWONQaA2FOZ8yMnszxTbOxO/O1woi1IyNHjC6 tPmFBiNyUxUHUL1zAZEXSvqCQLiLNON8pLb6p+YwtUBkvv0P7xZejHN69lh/gJlHIVwa L/l08Ys0QPagBv6Jo1kgp92P9zkp4B+20GsjLCu1g/IkSNkckHPKg7p96NZ+sS49SjvN ih7bUyXU1iSOx2t1HZ7eAknyhOn3vrvEaWXdZHqUGP3hfoKACA237XxXMsGxzUWc0CCc 2TuCuvYdBhEvbqUBb8IggT29WTAB5yXEdyIVCC2+9YozwhTVKD+0uMvyN3piHfU1brfd Aj1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=wrd9eNTrh6D6J8rGXcFB2/d+ZBemO5j2BbiFByaG1Bg=; b=D6gSdYIUk/jl2J3V2T43UKNkUvZdb9Vz9eUvVXxSTbr7c2xL7PgfFfoHvjwUCEt4fQ YT+/+0Ny+xphxPSES8S+9FOtGJpOmme/UeIpbp3y3fGrfVGEbQ4g6/v5F4SQmfCOLAVQ /haCVGI2ew+e0alF61LWCaWgVFNRG3L5y2ardp2BRyQLSf20JymqHTCQ7wYPT8+ZMoPn /fFIaytIAC7rnY6liUqTaXqLNlpdBPaV+hUiwrfGLMfjbp6cZJSrvCk+s6bBCmGV+wn1 pTb0lN32d7fG5CghNGM6zukWuVmkvbzd9O6ku5+sGlp+w3FBLsBsDgEI2ZKw53mZJwOM i6PA== X-Gm-Message-State: AOAM532UJXOFpAsUF2Ih8YXGyew/2NbSZZzD4F3y/ZXUK8NgARFKWhps 8KdzdkflznATn921C3FQW3NyzgoWWMM= X-Google-Smtp-Source: ABdhPJyigOsY5nHKHAVc15tzqmbovl5J88NSq/7r4ByilDjYBAap/Q/0mRR68c8T7h/c6frqmUrOPg== X-Received: by 2002:a63:1f54:: with SMTP id q20mr12937683pgm.135.1611476602283; Sun, 24 Jan 2021 00:23:22 -0800 (PST) Received: from bobo.ozlabs.ibm.com ([124.170.13.62]) by smtp.gmail.com with ESMTPSA id gb12sm11799757pjb.51.2021.01.24.00.23.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 24 Jan 2021 00:23:21 -0800 (PST) From: Nicholas Piggin To: linux-mm@kvack.org, Andrew Morton Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Zefan Li , Jonathan Cameron , Christoph Hellwig , Christophe Leroy , Rick Edgecombe , Ding Tianhong , Catalin Marinas , Will Deacon , linux-arm-kernel@lists.infradead.org, Thomas Gleixner , Ingo Molnar , Borislav Petkov , x86@kernel.org, "H. Peter Anvin" Subject: [PATCH v10 05/12] mm: HUGE_VMAP arch support cleanup Date: Sun, 24 Jan 2021 18:22:23 +1000 Message-Id: <20210124082230.2118861-6-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20210124082230.2118861-1-npiggin@gmail.com> References: <20210124082230.2118861-1-npiggin@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This changes the awkward approach where architectures provide init functions to determine which levels they can provide large mappings for, to one where the arch is queried for each call. This removes code and indirection, and allows constant-folding of dead code for unsupported levels. This also adds a prot argument to the arch query. This is unused currently but could help with some architectures (e.g., some powerpc processors can't map uncacheable memory with large pages). Cc: linuxppc-dev@lists.ozlabs.org Cc: Catalin Marinas Cc: Will Deacon Cc: linux-arm-kernel@lists.infradead.org Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: x86@kernel.org Cc: "H. Peter Anvin" Acked-by: Catalin Marinas [arm64] Signed-off-by: Nicholas Piggin --- arch/arm64/include/asm/vmalloc.h | 8 +++ arch/arm64/mm/mmu.c | 10 +-- arch/powerpc/include/asm/vmalloc.h | 8 +++ arch/powerpc/mm/book3s64/radix_pgtable.c | 8 +-- arch/x86/include/asm/vmalloc.h | 7 ++ arch/x86/mm/ioremap.c | 12 ++-- include/linux/io.h | 9 --- include/linux/vmalloc.h | 6 ++ init/main.c | 1 - mm/ioremap.c | 88 +++++++++--------------- 10 files changed, 79 insertions(+), 78 deletions(-) diff --git a/arch/arm64/include/asm/vmalloc.h b/arch/arm64/include/asm/vmalloc.h index 2ca708ab9b20..597b40405319 100644 --- a/arch/arm64/include/asm/vmalloc.h +++ b/arch/arm64/include/asm/vmalloc.h @@ -1,4 +1,12 @@ #ifndef _ASM_ARM64_VMALLOC_H #define _ASM_ARM64_VMALLOC_H +#include + +#ifdef CONFIG_HAVE_ARCH_HUGE_VMAP +bool arch_vmap_p4d_supported(pgprot_t prot); +bool arch_vmap_pud_supported(pgprot_t prot); +bool arch_vmap_pmd_supported(pgprot_t prot); +#endif + #endif /* _ASM_ARM64_VMALLOC_H */ diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index ae0c3d023824..f6614c378792 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -1313,12 +1313,12 @@ void *__init fixmap_remap_fdt(phys_addr_t dt_phys, int *size, pgprot_t prot) return dt_virt; } -int __init arch_ioremap_p4d_supported(void) +bool arch_vmap_p4d_supported(pgprot_t prot) { - return 0; + return false; } -int __init arch_ioremap_pud_supported(void) +bool arch_vmap_pud_supported(pgprot_t prot); { /* * Only 4k granule supports level 1 block mappings. @@ -1328,9 +1328,9 @@ int __init arch_ioremap_pud_supported(void) !IS_ENABLED(CONFIG_PTDUMP_DEBUGFS); } -int __init arch_ioremap_pmd_supported(void) +bool arch_vmap_pmd_supported(pgprot_t prot) { - /* See arch_ioremap_pud_supported() */ + /* See arch_vmap_pud_supported() */ return !IS_ENABLED(CONFIG_PTDUMP_DEBUGFS); } diff --git a/arch/powerpc/include/asm/vmalloc.h b/arch/powerpc/include/asm/vmalloc.h index b992dfaaa161..105abb73f075 100644 --- a/arch/powerpc/include/asm/vmalloc.h +++ b/arch/powerpc/include/asm/vmalloc.h @@ -1,4 +1,12 @@ #ifndef _ASM_POWERPC_VMALLOC_H #define _ASM_POWERPC_VMALLOC_H +#include + +#ifdef CONFIG_HAVE_ARCH_HUGE_VMAP +bool arch_vmap_p4d_supported(pgprot_t prot); +bool arch_vmap_pud_supported(pgprot_t prot); +bool arch_vmap_pmd_supported(pgprot_t prot); +#endif + #endif /* _ASM_POWERPC_VMALLOC_H */ diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c index 98f0b243c1ab..743807fc210f 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -1082,13 +1082,13 @@ void radix__ptep_modify_prot_commit(struct vm_area_struct *vma, set_pte_at(mm, addr, ptep, pte); } -int __init arch_ioremap_pud_supported(void) +bool arch_vmap_pud_supported(pgprot_t prot) { /* HPT does not cope with large pages in the vmalloc area */ return radix_enabled(); } -int __init arch_ioremap_pmd_supported(void) +bool arch_vmap_pmd_supported(pgprot_t prot) { return radix_enabled(); } @@ -1182,7 +1182,7 @@ int pmd_free_pte_page(pmd_t *pmd, unsigned long addr) return 1; } -int __init arch_ioremap_p4d_supported(void) +bool arch_vmap_p4d_supported(pgprot_t prot) { - return 0; + return false; } diff --git a/arch/x86/include/asm/vmalloc.h b/arch/x86/include/asm/vmalloc.h index 29837740b520..094ea2b565f3 100644 --- a/arch/x86/include/asm/vmalloc.h +++ b/arch/x86/include/asm/vmalloc.h @@ -1,6 +1,13 @@ #ifndef _ASM_X86_VMALLOC_H #define _ASM_X86_VMALLOC_H +#include #include +#ifdef CONFIG_HAVE_ARCH_HUGE_VMAP +bool arch_vmap_p4d_supported(pgprot_t prot); +bool arch_vmap_pud_supported(pgprot_t prot); +bool arch_vmap_pmd_supported(pgprot_t prot); +#endif + #endif /* _ASM_X86_VMALLOC_H */ diff --git a/arch/x86/mm/ioremap.c b/arch/x86/mm/ioremap.c index 9e5ccc56f8e0..fbaf0c447986 100644 --- a/arch/x86/mm/ioremap.c +++ b/arch/x86/mm/ioremap.c @@ -481,24 +481,26 @@ void iounmap(volatile void __iomem *addr) } EXPORT_SYMBOL(iounmap); -int __init arch_ioremap_p4d_supported(void) +#ifdef CONFIG_HAVE_ARCH_HUGE_VMAP +bool arch_vmap_p4d_supported(pgprot_t prot) { - return 0; + return false; } -int __init arch_ioremap_pud_supported(void) +bool arch_vmap_pud_supported(pgprot_t prot) { #ifdef CONFIG_X86_64 return boot_cpu_has(X86_FEATURE_GBPAGES); #else - return 0; + return false; #endif } -int __init arch_ioremap_pmd_supported(void) +bool arch_vmap_pmd_supported(pgprot_t prot) { return boot_cpu_has(X86_FEATURE_PSE); } +#endif /* * Convert a physical pointer to a virtual kernel pointer for /dev/mem diff --git a/include/linux/io.h b/include/linux/io.h index 8394c56babc2..f1effd4d7a3c 100644 --- a/include/linux/io.h +++ b/include/linux/io.h @@ -31,15 +31,6 @@ static inline int ioremap_page_range(unsigned long addr, unsigned long end, } #endif -#ifdef CONFIG_HAVE_ARCH_HUGE_VMAP -void __init ioremap_huge_init(void); -int arch_ioremap_p4d_supported(void); -int arch_ioremap_pud_supported(void); -int arch_ioremap_pmd_supported(void); -#else -static inline void ioremap_huge_init(void) { } -#endif - /* * Managed iomap interface */ diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 80c0181c411d..00bd62bd701e 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -83,6 +83,12 @@ struct vmap_area { }; }; +#ifndef CONFIG_HAVE_ARCH_HUGE_VMAP +static inline bool arch_vmap_p4d_supported(pgprot_t prot) { return false; } +static inline bool arch_vmap_pud_supported(pgprot_t prot) { return false; } +static inline bool arch_vmap_pmd_supported(pgprot_t prot) { return false; } +#endif + /* * Highlevel APIs for driver use */ diff --git a/init/main.c b/init/main.c index c68d784376ca..bf9389e5b2e4 100644 --- a/init/main.c +++ b/init/main.c @@ -834,7 +834,6 @@ static void __init mm_init(void) pgtable_init(); debug_objects_mem_init(); vmalloc_init(); - ioremap_huge_init(); /* Should be run before the first non-init thread is created */ init_espfix_bsp(); /* Should be run after espfix64 is set up. */ diff --git a/mm/ioremap.c b/mm/ioremap.c index 3f4d36f9745a..c67f91164401 100644 --- a/mm/ioremap.c +++ b/mm/ioremap.c @@ -16,49 +16,16 @@ #include "pgalloc-track.h" #ifdef CONFIG_HAVE_ARCH_HUGE_VMAP -static int __read_mostly ioremap_p4d_capable; -static int __read_mostly ioremap_pud_capable; -static int __read_mostly ioremap_pmd_capable; -static int __read_mostly ioremap_huge_disabled; +static bool __ro_after_init iomap_max_page_shift = PAGE_SHIFT; static int __init set_nohugeiomap(char *str) { - ioremap_huge_disabled = 1; + iomap_max_page_shift = P4D_SHIFT; return 0; } early_param("nohugeiomap", set_nohugeiomap); - -void __init ioremap_huge_init(void) -{ - if (!ioremap_huge_disabled) { - if (arch_ioremap_p4d_supported()) - ioremap_p4d_capable = 1; - if (arch_ioremap_pud_supported()) - ioremap_pud_capable = 1; - if (arch_ioremap_pmd_supported()) - ioremap_pmd_capable = 1; - } -} - -static inline int ioremap_p4d_enabled(void) -{ - return ioremap_p4d_capable; -} - -static inline int ioremap_pud_enabled(void) -{ - return ioremap_pud_capable; -} - -static inline int ioremap_pmd_enabled(void) -{ - return ioremap_pmd_capable; -} - -#else /* !CONFIG_HAVE_ARCH_HUGE_VMAP */ -static inline int ioremap_p4d_enabled(void) { return 0; } -static inline int ioremap_pud_enabled(void) { return 0; } -static inline int ioremap_pmd_enabled(void) { return 0; } +#else /* CONFIG_HAVE_ARCH_HUGE_VMAP */ +static const bool iomap_max_page_shift = PAGE_SHIFT; #endif /* CONFIG_HAVE_ARCH_HUGE_VMAP */ static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, @@ -82,9 +49,13 @@ static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, } static int vmap_try_huge_pmd(pmd_t *pmd, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot) + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift) { - if (!ioremap_pmd_enabled()) + if (max_page_shift < PMD_SHIFT) + return 0; + + if (!arch_vmap_pmd_supported(prot)) return 0; if ((end - addr) != PMD_SIZE) @@ -104,7 +75,7 @@ static int vmap_try_huge_pmd(pmd_t *pmd, unsigned long addr, unsigned long end, static int vmap_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, phys_addr_t phys_addr, pgprot_t prot, - pgtbl_mod_mask *mask) + unsigned int max_page_shift, pgtbl_mod_mask *mask) { pmd_t *pmd; unsigned long next; @@ -115,7 +86,7 @@ static int vmap_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, do { next = pmd_addr_end(addr, end); - if (vmap_try_huge_pmd(pmd, addr, next, phys_addr, prot)) { + if (vmap_try_huge_pmd(pmd, addr, next, phys_addr, prot, max_page_shift)) { *mask |= PGTBL_PMD_MODIFIED; continue; } @@ -127,9 +98,13 @@ static int vmap_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, } static int vmap_try_huge_pud(pud_t *pud, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot) + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift) { - if (!ioremap_pud_enabled()) + if (max_page_shift < PUD_SHIFT) + return 0; + + if (!arch_vmap_pud_supported(prot)) return 0; if ((end - addr) != PUD_SIZE) @@ -149,7 +124,7 @@ static int vmap_try_huge_pud(pud_t *pud, unsigned long addr, unsigned long end, static int vmap_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end, phys_addr_t phys_addr, pgprot_t prot, - pgtbl_mod_mask *mask) + unsigned int max_page_shift, pgtbl_mod_mask *mask) { pud_t *pud; unsigned long next; @@ -160,21 +135,25 @@ static int vmap_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end, do { next = pud_addr_end(addr, end); - if (vmap_try_huge_pud(pud, addr, next, phys_addr, prot)) { + if (vmap_try_huge_pud(pud, addr, next, phys_addr, prot, max_page_shift)) { *mask |= PGTBL_PUD_MODIFIED; continue; } - if (vmap_pmd_range(pud, addr, next, phys_addr, prot, mask)) + if (vmap_pmd_range(pud, addr, next, phys_addr, prot, max_page_shift, mask)) return -ENOMEM; } while (pud++, phys_addr += (next - addr), addr = next, addr != end); return 0; } static int vmap_try_huge_p4d(p4d_t *p4d, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot) + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift) { - if (!ioremap_p4d_enabled()) + if (max_page_shift < P4D_SHIFT) + return 0; + + if (!arch_vmap_p4d_supported(prot)) return 0; if ((end - addr) != P4D_SIZE) @@ -194,7 +173,7 @@ static int vmap_try_huge_p4d(p4d_t *p4d, unsigned long addr, unsigned long end, static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, phys_addr_t phys_addr, pgprot_t prot, - pgtbl_mod_mask *mask) + unsigned int max_page_shift, pgtbl_mod_mask *mask) { p4d_t *p4d; unsigned long next; @@ -205,19 +184,20 @@ static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, do { next = p4d_addr_end(addr, end); - if (vmap_try_huge_p4d(p4d, addr, next, phys_addr, prot)) { + if (vmap_try_huge_p4d(p4d, addr, next, phys_addr, prot, max_page_shift)) { *mask |= PGTBL_P4D_MODIFIED; continue; } - if (vmap_pud_range(p4d, addr, next, phys_addr, prot, mask)) + if (vmap_pud_range(p4d, addr, next, phys_addr, prot, max_page_shift, mask)) return -ENOMEM; } while (p4d++, phys_addr += (next - addr), addr = next, addr != end); return 0; } static int vmap_range(unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot) + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift) { pgd_t *pgd; unsigned long start; @@ -232,7 +212,7 @@ static int vmap_range(unsigned long addr, unsigned long end, pgd = pgd_offset_k(addr); do { next = pgd_addr_end(addr, end); - err = vmap_p4d_range(pgd, addr, next, phys_addr, prot, &mask); + err = vmap_p4d_range(pgd, addr, next, phys_addr, prot, max_page_shift, &mask); if (err) break; } while (pgd++, phys_addr += (next - addr), addr = next, addr != end); @@ -248,7 +228,7 @@ static int vmap_range(unsigned long addr, unsigned long end, int ioremap_page_range(unsigned long addr, unsigned long end, phys_addr_t phys_addr, pgprot_t prot) { - return vmap_range(addr, end, phys_addr, prot); + return vmap_range(addr, end, phys_addr, prot, iomap_max_page_shift); } #ifdef CONFIG_GENERIC_IOREMAP From patchwork Sun Jan 24 08:22:24 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 12042157 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D06FCC433DB for ; Sun, 24 Jan 2021 08:23:30 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 668BB22582 for ; Sun, 24 Jan 2021 08:23:30 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 668BB22582 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id F3AB46B0012; Sun, 24 Jan 2021 03:23:29 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EC3CE6B0022; Sun, 24 Jan 2021 03:23:29 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D648A6B0023; Sun, 24 Jan 2021 03:23:29 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0253.hostedemail.com [216.40.44.253]) by kanga.kvack.org (Postfix) with ESMTP id BCDE86B0012 for ; Sun, 24 Jan 2021 03:23:29 -0500 (EST) Received: from smtpin26.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 898E3363F for ; Sun, 24 Jan 2021 08:23:29 +0000 (UTC) X-FDA: 77739979338.26.title52_0f12bc62757b Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin26.hostedemail.com (Postfix) with ESMTP id 6BBE61804B655 for ; Sun, 24 Jan 2021 08:23:29 +0000 (UTC) X-HE-Tag: title52_0f12bc62757b X-Filterd-Recvd-Size: 5643 Received: from mail-pj1-f48.google.com (mail-pj1-f48.google.com [209.85.216.48]) by imf31.hostedemail.com (Postfix) with ESMTP for ; Sun, 24 Jan 2021 08:23:28 +0000 (UTC) Received: by mail-pj1-f48.google.com with SMTP id u4so6595067pjn.4 for ; Sun, 24 Jan 2021 00:23:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=6FRU9caWwWjvnrqszVqRsss5Upxnc33B3OfV8qLPoO8=; b=H9TlhLYPPssMc0ceABPukOINSlBwNuToeKnAunjyAGtSAlzZAV8FrTD6Lyj2mHJJ4i y4AS8SccFUgbMgUGqxrJNvlnwcw1uUaJmyFR1GdTPu+H1SD3vOgb9oX/gJEc8JFUONWW EngXprGRbFCgLjCzEi9PH1wIDD7rNcuaLMsSVvjQIPwrYjUh/GE8q6+rLDgzYqn29KxF 6ekoVrlR8UiPjwHDPFHOXhQUE/LeKI5HZpBrCKcgkali9s8keF65rZq6LBMTdGyeNfjJ KZZe/RghMNEixmN7NhRMTNYeSgJlv5Z1+QQXs+zaH9IZ2Peo9ARAio4V+7lLiz5xkRjK 8xWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=6FRU9caWwWjvnrqszVqRsss5Upxnc33B3OfV8qLPoO8=; b=qQsN2wZ4wJ8Pr2g7mvFHr/IX0CuKm0qflHtGQ1o0kSWannVpM90ikwAgl7bcsI9Cga f1RspRqAv5MBGHNGt+dr5LgLPtMCI/Cd3dQGAmPaeoD0O+pHOHTe+gEQkB5JT9+2GIgm iIdaTJkjVDiIQ2AlBFsvIFFZwfzFh4HlFN903WccY6Q17Luc20mBiVYHEIVu3XW+k83N ROI+FBwjmfQYJ7mWIqm5+ZxLof3L+Ps9h4+RlXEOJ18Db0kSD/DKuCsgW71/SzglYloA n2fk5/HpP0XHTbA9ivN7AWMngZj5QlaNrNDjnZO4aGOUB+GDatVO3A5+MVQfC6Qo4d6o Avew== X-Gm-Message-State: AOAM533bDzPe3Wc1kcgKWvfLqyddcsDMHVrw19T0zNIvrp8RgcW+Ko26 VfUrtOw9rV3uScNpp/nKB5XGt3FCQ9Q= X-Google-Smtp-Source: ABdhPJwdJIoki6IUcecjOrHd9qF/DDTmYOojoF7Kmx2FLKYrVifE66eYZA1PnPDfTKB+XB3xG/u7JQ== X-Received: by 2002:a17:90b:1196:: with SMTP id gk22mr15333934pjb.168.1611476607794; Sun, 24 Jan 2021 00:23:27 -0800 (PST) Received: from bobo.ozlabs.ibm.com ([124.170.13.62]) by smtp.gmail.com with ESMTPSA id gb12sm11799757pjb.51.2021.01.24.00.23.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 24 Jan 2021 00:23:27 -0800 (PST) From: Nicholas Piggin To: linux-mm@kvack.org, Andrew Morton Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Zefan Li , Jonathan Cameron , Christoph Hellwig , Christophe Leroy , Rick Edgecombe , Ding Tianhong , Michael Ellerman Subject: [PATCH v10 06/12] powerpc: inline huge vmap supported functions Date: Sun, 24 Jan 2021 18:22:24 +1000 Message-Id: <20210124082230.2118861-7-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20210124082230.2118861-1-npiggin@gmail.com> References: <20210124082230.2118861-1-npiggin@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This allows unsupported levels to be constant folded away, and so p4d_free_pud_page can be removed because it's no longer linked to. Cc: linuxppc-dev@lists.ozlabs.org Acked-by: Michael Ellerman Signed-off-by: Nicholas Piggin --- arch/powerpc/include/asm/vmalloc.h | 19 ++++++++++++++++--- arch/powerpc/mm/book3s64/radix_pgtable.c | 21 --------------------- 2 files changed, 16 insertions(+), 24 deletions(-) diff --git a/arch/powerpc/include/asm/vmalloc.h b/arch/powerpc/include/asm/vmalloc.h index 105abb73f075..3f0c153befb0 100644 --- a/arch/powerpc/include/asm/vmalloc.h +++ b/arch/powerpc/include/asm/vmalloc.h @@ -1,12 +1,25 @@ #ifndef _ASM_POWERPC_VMALLOC_H #define _ASM_POWERPC_VMALLOC_H +#include #include #ifdef CONFIG_HAVE_ARCH_HUGE_VMAP -bool arch_vmap_p4d_supported(pgprot_t prot); -bool arch_vmap_pud_supported(pgprot_t prot); -bool arch_vmap_pmd_supported(pgprot_t prot); +static inline bool arch_vmap_p4d_supported(pgprot_t prot) +{ + return false; +} + +static inline bool arch_vmap_pud_supported(pgprot_t prot) +{ + /* HPT does not cope with large pages in the vmalloc area */ + return radix_enabled(); +} + +static inline bool arch_vmap_pmd_supported(pgprot_t prot) +{ + return radix_enabled(); +} #endif #endif /* _ASM_POWERPC_VMALLOC_H */ diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c index 743807fc210f..8da62afccee5 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -1082,22 +1082,6 @@ void radix__ptep_modify_prot_commit(struct vm_area_struct *vma, set_pte_at(mm, addr, ptep, pte); } -bool arch_vmap_pud_supported(pgprot_t prot) -{ - /* HPT does not cope with large pages in the vmalloc area */ - return radix_enabled(); -} - -bool arch_vmap_pmd_supported(pgprot_t prot) -{ - return radix_enabled(); -} - -int p4d_free_pud_page(p4d_t *p4d, unsigned long addr) -{ - return 0; -} - int pud_set_huge(pud_t *pud, phys_addr_t addr, pgprot_t prot) { pte_t *ptep = (pte_t *)pud; @@ -1181,8 +1165,3 @@ int pmd_free_pte_page(pmd_t *pmd, unsigned long addr) return 1; } - -bool arch_vmap_p4d_supported(pgprot_t prot) -{ - return false; -} From patchwork Sun Jan 24 08:22:25 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 12042159 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9D86BC433E0 for ; Sun, 24 Jan 2021 08:23:36 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 448B322252 for ; Sun, 24 Jan 2021 08:23:36 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 448B322252 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D3F796B0022; Sun, 24 Jan 2021 03:23:35 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CC8416B0023; Sun, 24 Jan 2021 03:23:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BB7AB6B0024; Sun, 24 Jan 2021 03:23:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id A50E26B0022 for ; Sun, 24 Jan 2021 03:23:35 -0500 (EST) Received: from smtpin20.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 74E228249980 for ; Sun, 24 Jan 2021 08:23:35 +0000 (UTC) X-FDA: 77739979590.20.night68_1b0da5d2757b Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin20.hostedemail.com (Postfix) with ESMTP id 56EC1180C07A3 for ; Sun, 24 Jan 2021 08:23:35 +0000 (UTC) X-HE-Tag: night68_1b0da5d2757b X-Filterd-Recvd-Size: 6249 Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by imf19.hostedemail.com (Postfix) with ESMTP for ; Sun, 24 Jan 2021 08:23:34 +0000 (UTC) Received: by mail-pl1-f180.google.com with SMTP id d4so5749968plh.5 for ; Sun, 24 Jan 2021 00:23:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=XdOBo5hMulSSxEYd/6YmwlJTZSrkouBXlfopwP0/Zc4=; b=o5l/DCVSCaxUEl6ZWA1anhN4/cxw3MkSyb+QZwdaivP6WeAYkl8rUCScGtQxmbJDDA n+MhvBH/u3vKGyrqfIs/NROmznkxDGeLh77YSkfE90KH4IAcp7XF40rgXHpPJXh/FtVx QBWJjW1+RxiBCRJswIDB+VySgrbidOM5UUl7stEEWOXo2UgtRpQTIF86xL5v8R1XiP0M O80Zq64Gro5u4QhH/tHJj1dbKyMjjc9THLxn2k4GnWiAtTsO44QJFz/28tOod0FOX6SX Q/wryIF0ZxiAFSz2jSvlmoJO/78asliunkKRZSorL/5BicTxWS1TTX2fr4PVWxmkPz6f Pqcg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=XdOBo5hMulSSxEYd/6YmwlJTZSrkouBXlfopwP0/Zc4=; b=Xpg2BCE8Ru1oJ4ESa+bRGJ5mojjJT7NCSDMx2uaWKGezSAyJ7Tm4Vuwx1CAGLzZW+n KrCrDuuI1FFOaHtEldV0YSE8A7tY6yftjTOxn6SW+LA36KJrzSHoWvtzJJJ5CG83yrsZ BKiz9H6qFeL6G7nrIvwjnYWKRsUAZBwgovWVRfeiRzpgZsYvd0Ec9ds/e3Jz2VSUSQLD lBkUNo96GW3Mom2v9nygA8StgkdlpA1umITjxHKF02CFtnrf1ZhOrRVpsq/AAqdKR25N N4Zm1S9bBwoz4A5gAZVJJwL/+Yvi8SM1SpUkT0wWagxrMD+kNQYP8xk7OKN03zxGl76/ prkQ== X-Gm-Message-State: AOAM5313ICnRUKxSZW43CAsOKOaJMTlpI3oE0DRcrb9bIlwlVmrl54nU e0JLegvWCtxG8nj5jKttaxSksfIgLZE= X-Google-Smtp-Source: ABdhPJx6GFbCQ6szgbTg/K7X7HKG3gfdSjXkRRdgzuwadHjoBdmePc1daU9pjAxjKKOpTJ6Vyv2z/Q== X-Received: by 2002:a17:902:d2c1:b029:de:7c38:f86a with SMTP id n1-20020a170902d2c1b02900de7c38f86amr4700511plc.83.1611476613887; Sun, 24 Jan 2021 00:23:33 -0800 (PST) Received: from bobo.ozlabs.ibm.com ([124.170.13.62]) by smtp.gmail.com with ESMTPSA id gb12sm11799757pjb.51.2021.01.24.00.23.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 24 Jan 2021 00:23:33 -0800 (PST) From: Nicholas Piggin To: linux-mm@kvack.org, Andrew Morton Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Zefan Li , Jonathan Cameron , Christoph Hellwig , Christophe Leroy , Rick Edgecombe , Ding Tianhong , Catalin Marinas , Will Deacon , linux-arm-kernel@lists.infradead.org Subject: [PATCH v10 07/12] arm64: inline huge vmap supported functions Date: Sun, 24 Jan 2021 18:22:25 +1000 Message-Id: <20210124082230.2118861-8-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20210124082230.2118861-1-npiggin@gmail.com> References: <20210124082230.2118861-1-npiggin@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This allows unsupported levels to be constant folded away, and so p4d_free_pud_page can be removed because it's no longer linked to. Cc: Catalin Marinas Cc: Will Deacon Cc: linux-arm-kernel@lists.infradead.org Acked-by: Catalin Marinas Signed-off-by: Nicholas Piggin --- arch/arm64/include/asm/vmalloc.h | 23 ++++++++++++++++++++--- arch/arm64/mm/mmu.c | 26 -------------------------- 2 files changed, 20 insertions(+), 29 deletions(-) diff --git a/arch/arm64/include/asm/vmalloc.h b/arch/arm64/include/asm/vmalloc.h index 597b40405319..fc9a12d6cc1a 100644 --- a/arch/arm64/include/asm/vmalloc.h +++ b/arch/arm64/include/asm/vmalloc.h @@ -4,9 +4,26 @@ #include #ifdef CONFIG_HAVE_ARCH_HUGE_VMAP -bool arch_vmap_p4d_supported(pgprot_t prot); -bool arch_vmap_pud_supported(pgprot_t prot); -bool arch_vmap_pmd_supported(pgprot_t prot); +static inline bool arch_vmap_p4d_supported(pgprot_t prot) +{ + return false; +} + +static inline bool arch_vmap_pud_supported(pgprot_t prot) +{ + /* + * Only 4k granule supports level 1 block mappings. + * SW table walks can't handle removal of intermediate entries. + */ + return IS_ENABLED(CONFIG_ARM64_4K_PAGES) && + !IS_ENABLED(CONFIG_PTDUMP_DEBUGFS); +} + +static inline bool arch_vmap_pmd_supported(pgprot_t prot) +{ + /* See arch_vmap_pud_supported() */ + return !IS_ENABLED(CONFIG_PTDUMP_DEBUGFS); +} #endif #endif /* _ASM_ARM64_VMALLOC_H */ diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index f6614c378792..ab9ba7c36dae 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -1313,27 +1313,6 @@ void *__init fixmap_remap_fdt(phys_addr_t dt_phys, int *size, pgprot_t prot) return dt_virt; } -bool arch_vmap_p4d_supported(pgprot_t prot) -{ - return false; -} - -bool arch_vmap_pud_supported(pgprot_t prot); -{ - /* - * Only 4k granule supports level 1 block mappings. - * SW table walks can't handle removal of intermediate entries. - */ - return IS_ENABLED(CONFIG_ARM64_4K_PAGES) && - !IS_ENABLED(CONFIG_PTDUMP_DEBUGFS); -} - -bool arch_vmap_pmd_supported(pgprot_t prot) -{ - /* See arch_vmap_pud_supported() */ - return !IS_ENABLED(CONFIG_PTDUMP_DEBUGFS); -} - int pud_set_huge(pud_t *pudp, phys_addr_t phys, pgprot_t prot) { pud_t new_pud = pfn_pud(__phys_to_pfn(phys), mk_pud_sect_prot(prot)); @@ -1425,11 +1404,6 @@ int pud_free_pmd_page(pud_t *pudp, unsigned long addr) return 1; } -int p4d_free_pud_page(p4d_t *p4d, unsigned long addr) -{ - return 0; /* Don't attempt a block mapping */ -} - #ifdef CONFIG_MEMORY_HOTPLUG static void __remove_pgd_mapping(pgd_t *pgdir, unsigned long start, u64 size) { From patchwork Sun Jan 24 08:22:26 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 12042161 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2A0DFC433E0 for ; Sun, 24 Jan 2021 08:23:43 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C2C1E22838 for ; Sun, 24 Jan 2021 08:23:42 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C2C1E22838 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 5C6E26B0023; Sun, 24 Jan 2021 03:23:42 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5260E6B0024; Sun, 24 Jan 2021 03:23:42 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3EE976B0025; Sun, 24 Jan 2021 03:23:42 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0057.hostedemail.com [216.40.44.57]) by kanga.kvack.org (Postfix) with ESMTP id 23C206B0023 for ; Sun, 24 Jan 2021 03:23:42 -0500 (EST) Received: from smtpin16.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id E7CD2363F for ; Sun, 24 Jan 2021 08:23:41 +0000 (UTC) X-FDA: 77739979842.16.plant96_3c04f162757b Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin16.hostedemail.com (Postfix) with ESMTP id C9E5B100E690C for ; Sun, 24 Jan 2021 08:23:41 +0000 (UTC) X-HE-Tag: plant96_3c04f162757b X-Filterd-Recvd-Size: 6561 Received: from mail-pj1-f41.google.com (mail-pj1-f41.google.com [209.85.216.41]) by imf38.hostedemail.com (Postfix) with ESMTP for ; Sun, 24 Jan 2021 08:23:41 +0000 (UTC) Received: by mail-pj1-f41.google.com with SMTP id kx7so6458961pjb.2 for ; Sun, 24 Jan 2021 00:23:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=80Po72abxrqXJsVTjT2d7/68A0xoEHKX9fHFXJoIAvk=; b=JYIpc/oqm/sAsWgaVw4nIYbYX6wxnaKxqEtZNm7YbjwC4PE+yNUmVWfFFc7/DDS1vu 3Bvb2n4Lkhye1htDxYySk01BKuFM42hGfIYYp/NSXryeL6E+Voz4SH123INkddKD0IBL Kqyr/eoCmpBjxN++rd5vW/WXKaozCgA4xl6bXF4b8ZYx64nHsbskvRHVLhXNSzxzHocQ wxRHD0kk3MvwDPGLuNi9tIE20PB+2I42NP6a6n8eSVI+UF0EM09H06R1p8Ny1J5sk1Wh 0nxk5XGh4iImvCFUaIO5Iy+J3AqypcC/fRSz/vfCtV/aTL23HF2BthbLHhB4rU0r8dai GszA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=80Po72abxrqXJsVTjT2d7/68A0xoEHKX9fHFXJoIAvk=; b=BxVsDHYscTl+rkAJwKJp1r/6QPGsHxvDPN63vrJAEuikcGt686TUGsYZiLWmNNwMO+ dIVA1zvquiwe4ZBsajIPnE1CoHUhUdSkkMvlO/fPfOuWncnQBEokmeY/BmDKahTdY5tA 4uZX3WzN2Sgux46QThnX7Xs5ExEjHRbAE+3vL7nrVLXPc2XpbffvVdKl2esvti6uzhXF vDBkG7tJE2NBhqrJdy9ak5JLznrsXS5FAGpkEGib4cuDPAZVN5GosJHqPqYnN38ag1ln qCNQkjlrmjlNE92ySRkj5/SjXtEdCRHkWVnTUTfb/XCM2uAWJuDjN3zIU2ODZNWGGfgZ hJLQ== X-Gm-Message-State: AOAM533Okc3h+4L3cqbA7e4H2A65rAiTZ3GE8Aw3+2coat8dADlFM2p5 YojhgHy6YiUpfdICoa9vO0cc0Sh42Nk= X-Google-Smtp-Source: ABdhPJyGxzt4jp0pHuRp4lsmYaC1aRXlAB8zf0kpATtdtfUdYJZXoyFDyRHWsaQurt0XqhzPzOtL+w== X-Received: by 2002:a17:902:9f89:b029:dc:3032:e439 with SMTP id g9-20020a1709029f89b02900dc3032e439mr1132590plq.35.1611476620596; Sun, 24 Jan 2021 00:23:40 -0800 (PST) Received: from bobo.ozlabs.ibm.com ([124.170.13.62]) by smtp.gmail.com with ESMTPSA id gb12sm11799757pjb.51.2021.01.24.00.23.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 24 Jan 2021 00:23:40 -0800 (PST) From: Nicholas Piggin To: linux-mm@kvack.org, Andrew Morton Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Zefan Li , Jonathan Cameron , Christoph Hellwig , Christophe Leroy , Rick Edgecombe , Ding Tianhong , Thomas Gleixner , Ingo Molnar , Borislav Petkov , x86@kernel.org, "H. Peter Anvin" Subject: [PATCH v10 08/12] x86: inline huge vmap supported functions Date: Sun, 24 Jan 2021 18:22:26 +1000 Message-Id: <20210124082230.2118861-9-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20210124082230.2118861-1-npiggin@gmail.com> References: <20210124082230.2118861-1-npiggin@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This allows unsupported levels to be constant folded away, and so p4d_free_pud_page can be removed because it's no longer linked to. Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: x86@kernel.org Cc: "H. Peter Anvin" Signed-off-by: Nicholas Piggin --- arch/x86/include/asm/vmalloc.h | 22 +++++++++++++++++++--- arch/x86/mm/ioremap.c | 21 --------------------- arch/x86/mm/pgtable.c | 13 ------------- 3 files changed, 19 insertions(+), 37 deletions(-) diff --git a/arch/x86/include/asm/vmalloc.h b/arch/x86/include/asm/vmalloc.h index 094ea2b565f3..e714b00fc0ca 100644 --- a/arch/x86/include/asm/vmalloc.h +++ b/arch/x86/include/asm/vmalloc.h @@ -1,13 +1,29 @@ #ifndef _ASM_X86_VMALLOC_H #define _ASM_X86_VMALLOC_H +#include #include #include #ifdef CONFIG_HAVE_ARCH_HUGE_VMAP -bool arch_vmap_p4d_supported(pgprot_t prot); -bool arch_vmap_pud_supported(pgprot_t prot); -bool arch_vmap_pmd_supported(pgprot_t prot); +static inline bool arch_vmap_p4d_supported(pgprot_t prot) +{ + return false; +} + +static inline bool arch_vmap_pud_supported(pgprot_t prot) +{ +#ifdef CONFIG_X86_64 + return boot_cpu_has(X86_FEATURE_GBPAGES); +#else + return false; +#endif +} + +static inline bool arch_vmap_pmd_supported(pgprot_t prot) +{ + return boot_cpu_has(X86_FEATURE_PSE); +} #endif #endif /* _ASM_X86_VMALLOC_H */ diff --git a/arch/x86/mm/ioremap.c b/arch/x86/mm/ioremap.c index fbaf0c447986..12c686c65ea9 100644 --- a/arch/x86/mm/ioremap.c +++ b/arch/x86/mm/ioremap.c @@ -481,27 +481,6 @@ void iounmap(volatile void __iomem *addr) } EXPORT_SYMBOL(iounmap); -#ifdef CONFIG_HAVE_ARCH_HUGE_VMAP -bool arch_vmap_p4d_supported(pgprot_t prot) -{ - return false; -} - -bool arch_vmap_pud_supported(pgprot_t prot) -{ -#ifdef CONFIG_X86_64 - return boot_cpu_has(X86_FEATURE_GBPAGES); -#else - return false; -#endif -} - -bool arch_vmap_pmd_supported(pgprot_t prot) -{ - return boot_cpu_has(X86_FEATURE_PSE); -} -#endif - /* * Convert a physical pointer to a virtual kernel pointer for /dev/mem * access diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index f6a9e2e36642..d27cf69e811d 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -780,14 +780,6 @@ int pmd_clear_huge(pmd_t *pmd) return 0; } -/* - * Until we support 512GB pages, skip them in the vmap area. - */ -int p4d_free_pud_page(p4d_t *p4d, unsigned long addr) -{ - return 0; -} - #ifdef CONFIG_X86_64 /** * pud_free_pmd_page - Clear pud entry and free pmd page. @@ -861,11 +853,6 @@ int pmd_free_pte_page(pmd_t *pmd, unsigned long addr) #else /* !CONFIG_X86_64 */ -int pud_free_pmd_page(pud_t *pud, unsigned long addr) -{ - return pud_none(*pud); -} - /* * Disable free page handling on x86-PAE. This assures that ioremap() * does not update sync'd pmd entries. See vmalloc_sync_one(). From patchwork Sun Jan 24 08:22:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 12042163 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3254CC433DB for ; Sun, 24 Jan 2021 08:23:49 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id BCF3B22D0A for ; Sun, 24 Jan 2021 08:23:48 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BCF3B22D0A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 59E6E6B0024; Sun, 24 Jan 2021 03:23:48 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5261C6B0025; Sun, 24 Jan 2021 03:23:48 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3EE386B0026; Sun, 24 Jan 2021 03:23:48 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0023.hostedemail.com [216.40.44.23]) by kanga.kvack.org (Postfix) with ESMTP id 254D46B0024 for ; Sun, 24 Jan 2021 03:23:48 -0500 (EST) Received: from smtpin06.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id DC28F180ACF84 for ; Sun, 24 Jan 2021 08:23:47 +0000 (UTC) X-FDA: 77739980094.06.home51_541689f2757b Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin06.hostedemail.com (Postfix) with ESMTP id BB2C210038E2B for ; Sun, 24 Jan 2021 08:23:47 +0000 (UTC) X-HE-Tag: home51_541689f2757b X-Filterd-Recvd-Size: 15022 Received: from mail-pg1-f179.google.com (mail-pg1-f179.google.com [209.85.215.179]) by imf18.hostedemail.com (Postfix) with ESMTP for ; Sun, 24 Jan 2021 08:23:47 +0000 (UTC) Received: by mail-pg1-f179.google.com with SMTP id z21so6880746pgj.4 for ; Sun, 24 Jan 2021 00:23:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=NjWfFRYeRlU/B850TknTaMl6GMUEdgcaAcsK7uRVp3w=; b=PPY5lnjhZG9a00xwbVV2XQ+djv+Z1w7sYF2K6fjn70hr8h6HQR46CJSSbADMibSYek JutMwhJQf4PP2spizB3clRLZW467+QmBqzaK8lyH5Ykve45YPNYEyRv490wG74JhFIsZ lhdl7X8rzaoGZq1OX50dxQ047lLAfhl3auoKkx35OAlF+VI+xpUooUjc7DHS+d2EBwj/ zLfAn+cTYv4cW2KoYaTlasAqoYvg+VvjdFgxNwyyiE7axiPQ1qjlPnA6rbNgKdEWaHTm v8ZV03BOiFeYmwaon8BXRyWUfoWv3gppSUKlHAyTHM2hGsZbjD3mdVmeniTqkP1h+7v6 h5QA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=NjWfFRYeRlU/B850TknTaMl6GMUEdgcaAcsK7uRVp3w=; b=jVlvz0iNDNwyeW6j9fYeNF+r40zsBQQdecnhVbpPuMc3oA6xYjY9mSVka1j1xwWDaP Bxmn/+/nOBFVO12hzW8cjXzKcTHieJ6bSv+Wv1S6w2qvb925LluInFBeUUXOP71ldOP1 vbuKWKiTmupdFRFyjPhoT1q1967D5nHgR/i6xSUAGP1vsohCK4HTCabORcnwd4Pf23IX aPX5e3DwGLLq3GTUsyf0mDwgEHcTYoBkddGdXPZj7Mn4k93V4FpXvqPatIzY74e+mc0c hv72l222O/s/FfVngf412IsryLlPy2CQrfWc7HyOXqmYwDqRcmsMLWPctQ8hWic4MHJx K6tg== X-Gm-Message-State: AOAM533OZNsIJLY77GtWZmVedQeox8qCKswMNVePC4yFT+oS3o9FyQVJ +JSV++6Wi4tm/TwoES/f6Qia4C9EpRo= X-Google-Smtp-Source: ABdhPJzSnnbzfBfajFRccGaJ8jhprS6fNStPS7yld5wZvHEKSVyxWtKjgQ0se0aHNa2WDIVdkfKa1g== X-Received: by 2002:a05:6a00:1506:b029:1bc:6f53:8eb8 with SMTP id q6-20020a056a001506b02901bc6f538eb8mr345934pfu.36.1611476626192; Sun, 24 Jan 2021 00:23:46 -0800 (PST) Received: from bobo.ozlabs.ibm.com ([124.170.13.62]) by smtp.gmail.com with ESMTPSA id gb12sm11799757pjb.51.2021.01.24.00.23.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 24 Jan 2021 00:23:45 -0800 (PST) From: Nicholas Piggin To: linux-mm@kvack.org, Andrew Morton Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Zefan Li , Jonathan Cameron , Christoph Hellwig , Christophe Leroy , Rick Edgecombe , Ding Tianhong Subject: [PATCH v10 09/12] mm: Move vmap_range from mm/ioremap.c to mm/vmalloc.c Date: Sun, 24 Jan 2021 18:22:27 +1000 Message-Id: <20210124082230.2118861-10-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20210124082230.2118861-1-npiggin@gmail.com> References: <20210124082230.2118861-1-npiggin@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is a generic kernel virtual memory mapper, not specific to ioremap. Signed-off-by: Nicholas Piggin Reviewed-by: Christoph Hellwig --- include/linux/vmalloc.h | 3 + mm/ioremap.c | 197 ---------------------------------------- mm/vmalloc.c | 196 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 199 insertions(+), 197 deletions(-) diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 00bd62bd701e..40649c4bb5a2 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -178,6 +178,9 @@ extern struct vm_struct *remove_vm_area(const void *addr); extern struct vm_struct *find_vm_area(const void *addr); #ifdef CONFIG_MMU +int vmap_range(unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift); extern int map_kernel_range_noflush(unsigned long start, unsigned long size, pgprot_t prot, struct page **pages); int map_kernel_range(unsigned long start, unsigned long size, pgprot_t prot, diff --git a/mm/ioremap.c b/mm/ioremap.c index c67f91164401..d1dcc7e744ac 100644 --- a/mm/ioremap.c +++ b/mm/ioremap.c @@ -28,203 +28,6 @@ early_param("nohugeiomap", set_nohugeiomap); static const bool iomap_max_page_shift = PAGE_SHIFT; #endif /* CONFIG_HAVE_ARCH_HUGE_VMAP */ -static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, - pgtbl_mod_mask *mask) -{ - pte_t *pte; - u64 pfn; - - pfn = phys_addr >> PAGE_SHIFT; - pte = pte_alloc_kernel_track(pmd, addr, mask); - if (!pte) - return -ENOMEM; - do { - BUG_ON(!pte_none(*pte)); - set_pte_at(&init_mm, addr, pte, pfn_pte(pfn, prot)); - pfn++; - } while (pte++, addr += PAGE_SIZE, addr != end); - *mask |= PGTBL_PTE_MODIFIED; - return 0; -} - -static int vmap_try_huge_pmd(pmd_t *pmd, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, - unsigned int max_page_shift) -{ - if (max_page_shift < PMD_SHIFT) - return 0; - - if (!arch_vmap_pmd_supported(prot)) - return 0; - - if ((end - addr) != PMD_SIZE) - return 0; - - if (!IS_ALIGNED(addr, PMD_SIZE)) - return 0; - - if (!IS_ALIGNED(phys_addr, PMD_SIZE)) - return 0; - - if (pmd_present(*pmd) && !pmd_free_pte_page(pmd, addr)) - return 0; - - return pmd_set_huge(pmd, phys_addr, prot); -} - -static int vmap_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, - unsigned int max_page_shift, pgtbl_mod_mask *mask) -{ - pmd_t *pmd; - unsigned long next; - - pmd = pmd_alloc_track(&init_mm, pud, addr, mask); - if (!pmd) - return -ENOMEM; - do { - next = pmd_addr_end(addr, end); - - if (vmap_try_huge_pmd(pmd, addr, next, phys_addr, prot, max_page_shift)) { - *mask |= PGTBL_PMD_MODIFIED; - continue; - } - - if (vmap_pte_range(pmd, addr, next, phys_addr, prot, mask)) - return -ENOMEM; - } while (pmd++, phys_addr += (next - addr), addr = next, addr != end); - return 0; -} - -static int vmap_try_huge_pud(pud_t *pud, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, - unsigned int max_page_shift) -{ - if (max_page_shift < PUD_SHIFT) - return 0; - - if (!arch_vmap_pud_supported(prot)) - return 0; - - if ((end - addr) != PUD_SIZE) - return 0; - - if (!IS_ALIGNED(addr, PUD_SIZE)) - return 0; - - if (!IS_ALIGNED(phys_addr, PUD_SIZE)) - return 0; - - if (pud_present(*pud) && !pud_free_pmd_page(pud, addr)) - return 0; - - return pud_set_huge(pud, phys_addr, prot); -} - -static int vmap_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, - unsigned int max_page_shift, pgtbl_mod_mask *mask) -{ - pud_t *pud; - unsigned long next; - - pud = pud_alloc_track(&init_mm, p4d, addr, mask); - if (!pud) - return -ENOMEM; - do { - next = pud_addr_end(addr, end); - - if (vmap_try_huge_pud(pud, addr, next, phys_addr, prot, max_page_shift)) { - *mask |= PGTBL_PUD_MODIFIED; - continue; - } - - if (vmap_pmd_range(pud, addr, next, phys_addr, prot, max_page_shift, mask)) - return -ENOMEM; - } while (pud++, phys_addr += (next - addr), addr = next, addr != end); - return 0; -} - -static int vmap_try_huge_p4d(p4d_t *p4d, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, - unsigned int max_page_shift) -{ - if (max_page_shift < P4D_SHIFT) - return 0; - - if (!arch_vmap_p4d_supported(prot)) - return 0; - - if ((end - addr) != P4D_SIZE) - return 0; - - if (!IS_ALIGNED(addr, P4D_SIZE)) - return 0; - - if (!IS_ALIGNED(phys_addr, P4D_SIZE)) - return 0; - - if (p4d_present(*p4d) && !p4d_free_pud_page(p4d, addr)) - return 0; - - return p4d_set_huge(p4d, phys_addr, prot); -} - -static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, - unsigned int max_page_shift, pgtbl_mod_mask *mask) -{ - p4d_t *p4d; - unsigned long next; - - p4d = p4d_alloc_track(&init_mm, pgd, addr, mask); - if (!p4d) - return -ENOMEM; - do { - next = p4d_addr_end(addr, end); - - if (vmap_try_huge_p4d(p4d, addr, next, phys_addr, prot, max_page_shift)) { - *mask |= PGTBL_P4D_MODIFIED; - continue; - } - - if (vmap_pud_range(p4d, addr, next, phys_addr, prot, max_page_shift, mask)) - return -ENOMEM; - } while (p4d++, phys_addr += (next - addr), addr = next, addr != end); - return 0; -} - -static int vmap_range(unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, - unsigned int max_page_shift) -{ - pgd_t *pgd; - unsigned long start; - unsigned long next; - int err; - pgtbl_mod_mask mask = 0; - - might_sleep(); - BUG_ON(addr >= end); - - start = addr; - pgd = pgd_offset_k(addr); - do { - next = pgd_addr_end(addr, end); - err = vmap_p4d_range(pgd, addr, next, phys_addr, prot, max_page_shift, &mask); - if (err) - break; - } while (pgd++, phys_addr += (next - addr), addr = next, addr != end); - - flush_cache_vmap(start, end); - - if (mask & ARCH_PAGE_TABLE_SYNC_MASK) - arch_sync_kernel_mappings(start, end); - - return err; -} - int ioremap_page_range(unsigned long addr, unsigned long end, phys_addr_t phys_addr, pgprot_t prot) { diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 7f2f36116980..5d79148b7fa7 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -68,6 +68,202 @@ static void free_work(struct work_struct *w) } /*** Page table manipulation functions ***/ +static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + pgtbl_mod_mask *mask) +{ + pte_t *pte; + u64 pfn; + + pfn = phys_addr >> PAGE_SHIFT; + pte = pte_alloc_kernel_track(pmd, addr, mask); + if (!pte) + return -ENOMEM; + do { + BUG_ON(!pte_none(*pte)); + set_pte_at(&init_mm, addr, pte, pfn_pte(pfn, prot)); + pfn++; + } while (pte++, addr += PAGE_SIZE, addr != end); + *mask |= PGTBL_PTE_MODIFIED; + return 0; +} + +static int vmap_try_huge_pmd(pmd_t *pmd, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift) +{ + if (max_page_shift < PMD_SHIFT) + return 0; + + if (!arch_vmap_pmd_supported(prot)) + return 0; + + if ((end - addr) != PMD_SIZE) + return 0; + + if (!IS_ALIGNED(addr, PMD_SIZE)) + return 0; + + if (!IS_ALIGNED(phys_addr, PMD_SIZE)) + return 0; + + if (pmd_present(*pmd) && !pmd_free_pte_page(pmd, addr)) + return 0; + + return pmd_set_huge(pmd, phys_addr, prot); +} + +static int vmap_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift, pgtbl_mod_mask *mask) +{ + pmd_t *pmd; + unsigned long next; + + pmd = pmd_alloc_track(&init_mm, pud, addr, mask); + if (!pmd) + return -ENOMEM; + do { + next = pmd_addr_end(addr, end); + + if (vmap_try_huge_pmd(pmd, addr, next, phys_addr, prot, max_page_shift)) { + *mask |= PGTBL_PMD_MODIFIED; + continue; + } + + if (vmap_pte_range(pmd, addr, next, phys_addr, prot, mask)) + return -ENOMEM; + } while (pmd++, phys_addr += (next - addr), addr = next, addr != end); + return 0; +} + +static int vmap_try_huge_pud(pud_t *pud, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift) +{ + if (max_page_shift < PUD_SHIFT) + return 0; + + if (!arch_vmap_pud_supported(prot)) + return 0; + + if ((end - addr) != PUD_SIZE) + return 0; + + if (!IS_ALIGNED(addr, PUD_SIZE)) + return 0; + + if (!IS_ALIGNED(phys_addr, PUD_SIZE)) + return 0; + + if (pud_present(*pud) && !pud_free_pmd_page(pud, addr)) + return 0; + + return pud_set_huge(pud, phys_addr, prot); +} + +static int vmap_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift, pgtbl_mod_mask *mask) +{ + pud_t *pud; + unsigned long next; + + pud = pud_alloc_track(&init_mm, p4d, addr, mask); + if (!pud) + return -ENOMEM; + do { + next = pud_addr_end(addr, end); + + if (vmap_try_huge_pud(pud, addr, next, phys_addr, prot, max_page_shift)) { + *mask |= PGTBL_PUD_MODIFIED; + continue; + } + + if (vmap_pmd_range(pud, addr, next, phys_addr, prot, max_page_shift, mask)) + return -ENOMEM; + } while (pud++, phys_addr += (next - addr), addr = next, addr != end); + return 0; +} + +static int vmap_try_huge_p4d(p4d_t *p4d, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift) +{ + if (max_page_shift < P4D_SHIFT) + return 0; + + if (!arch_vmap_p4d_supported(prot)) + return 0; + + if ((end - addr) != P4D_SIZE) + return 0; + + if (!IS_ALIGNED(addr, P4D_SIZE)) + return 0; + + if (!IS_ALIGNED(phys_addr, P4D_SIZE)) + return 0; + + if (p4d_present(*p4d) && !p4d_free_pud_page(p4d, addr)) + return 0; + + return p4d_set_huge(p4d, phys_addr, prot); +} + +static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift, pgtbl_mod_mask *mask) +{ + p4d_t *p4d; + unsigned long next; + + p4d = p4d_alloc_track(&init_mm, pgd, addr, mask); + if (!p4d) + return -ENOMEM; + do { + next = p4d_addr_end(addr, end); + + if (vmap_try_huge_p4d(p4d, addr, next, phys_addr, prot, max_page_shift)) { + *mask |= PGTBL_P4D_MODIFIED; + continue; + } + + if (vmap_pud_range(p4d, addr, next, phys_addr, prot, max_page_shift, mask)) + return -ENOMEM; + } while (p4d++, phys_addr += (next - addr), addr = next, addr != end); + return 0; +} + +int vmap_range(unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift) +{ + pgd_t *pgd; + unsigned long start; + unsigned long next; + int err; + pgtbl_mod_mask mask = 0; + + might_sleep(); + BUG_ON(addr >= end); + + start = addr; + pgd = pgd_offset_k(addr); + do { + next = pgd_addr_end(addr, end); + err = vmap_p4d_range(pgd, addr, next, phys_addr, prot, max_page_shift, &mask); + if (err) + break; + } while (pgd++, phys_addr += (next - addr), addr = next, addr != end); + + flush_cache_vmap(start, end); + + if (mask & ARCH_PAGE_TABLE_SYNC_MASK) + arch_sync_kernel_mappings(start, end); + + return err; +} static void vunmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, pgtbl_mod_mask *mask) From patchwork Sun Jan 24 08:22:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 12042165 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DDEAEC433E0 for ; Sun, 24 Jan 2021 08:23:53 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 8361A22838 for ; Sun, 24 Jan 2021 08:23:53 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8361A22838 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 1F2A56B0025; Sun, 24 Jan 2021 03:23:53 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 17BC86B0026; Sun, 24 Jan 2021 03:23:53 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 06C246B0027; Sun, 24 Jan 2021 03:23:53 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0203.hostedemail.com [216.40.44.203]) by kanga.kvack.org (Postfix) with ESMTP id E4F486B0025 for ; Sun, 24 Jan 2021 03:23:52 -0500 (EST) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id B04B2181AF5CA for ; Sun, 24 Jan 2021 08:23:52 +0000 (UTC) X-FDA: 77739980304.11.berry94_1b103e22757b Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin11.hostedemail.com (Postfix) with ESMTP id 89E4D180F8B82 for ; Sun, 24 Jan 2021 08:23:52 +0000 (UTC) X-HE-Tag: berry94_1b103e22757b X-Filterd-Recvd-Size: 4777 Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) by imf24.hostedemail.com (Postfix) with ESMTP for ; Sun, 24 Jan 2021 08:23:52 +0000 (UTC) Received: by mail-pl1-f176.google.com with SMTP id q2so2976563plk.4 for ; Sun, 24 Jan 2021 00:23:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=TVhRROVZ7z046qTJ4ejm1G9xGCLJycEigvvpkf2Te7E=; b=LD1AJ6Q7eOjSujKmrwu2p/+JbN0Zfr5LtB3ZkyFdzthlHqp5+5bXxUIPms0GZPIEI1 j8iC6ALOlnNPw7NvBm8g+wvNrzaRNyHGNbry4++c389NU654HOe92r7RSbw9ZyWG84gc kp2FgcVZUe9J6ouu30w7uKhA3EG4S+Ee1y+5Cad0k2E5Qg71u/s7n0t0KG/7ghJfui11 M9VJFibtXnmgzbuQat5H0d9DFSHOp4Kj27NTnsD2LZk65R7tFL3xKYvKblgjj1omjdo8 0laPVyur/H+mRgYddO4EAAw7KpMtM0p8cACx9kN1MlkXAqWRH/vc/B4wLdXNu0P0Y4t5 Rsbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=TVhRROVZ7z046qTJ4ejm1G9xGCLJycEigvvpkf2Te7E=; b=F03EWjpYW64lt1MTyxGA+Ddemi3qUPjmFx++vYUHqs/uyF7PbvCHJ4+7uK1csQ67ON WtEJLiVDP90Ik3ENHEe7dGFPCueSaH1Gtz22FbMehpXVlNVACe1jMFPa/j8XiF/0rBg7 ATRu6a3X51WT9o9IzcnHudwMG0JziEIWKEVP7ag3PR57OFg4qZd941p1k7CfZkG6SBa3 hpry0oCCwDTIH2duzLJo5Q+/ny8sEN5SM0nr9IPg1zQRZusbCIj1FkXJYCoH8IyIYANX MNYR3YUPcKpYzQNJ734uKBrzjFl/BXybf/rdysJd6813cL/h14PFAyvW/SbC/qbQf40M Qq4w== X-Gm-Message-State: AOAM5308xIQa+4/CL5q4uNH9iP0R81DJbWUJmDcV2bOqbF9tDqArveEN xjO/Ke/xF2jeCLFxDlnuWZxJszBmgYo= X-Google-Smtp-Source: ABdhPJwYhf2C3cY0UzUjCepD6a2ETZbd8HZtvD4C0FN52jeC88VQLzQc0VPtpNQam4pZa9M23WLv4Q== X-Received: by 2002:a17:902:bb8c:b029:dc:2e5e:2b2 with SMTP id m12-20020a170902bb8cb02900dc2e5e02b2mr13915452pls.10.1611476631232; Sun, 24 Jan 2021 00:23:51 -0800 (PST) Received: from bobo.ozlabs.ibm.com ([124.170.13.62]) by smtp.gmail.com with ESMTPSA id gb12sm11799757pjb.51.2021.01.24.00.23.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 24 Jan 2021 00:23:50 -0800 (PST) From: Nicholas Piggin To: linux-mm@kvack.org, Andrew Morton Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Zefan Li , Jonathan Cameron , Christoph Hellwig , Christophe Leroy , Rick Edgecombe , Ding Tianhong Subject: [PATCH v10 10/12] mm/vmalloc: add vmap_range_noflush variant Date: Sun, 24 Jan 2021 18:22:28 +1000 Message-Id: <20210124082230.2118861-11-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20210124082230.2118861-1-npiggin@gmail.com> References: <20210124082230.2118861-1-npiggin@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: As a side-effect, the order of flush_cache_vmap() and arch_sync_kernel_mappings() calls are switched, but that now matches the other callers in this file. Signed-off-by: Nicholas Piggin Reviewed-by: Christoph Hellwig --- mm/vmalloc.c | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 5d79148b7fa7..0377e1d059e5 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -235,7 +235,7 @@ static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, return 0; } -int vmap_range(unsigned long addr, unsigned long end, +static int vmap_range_noflush(unsigned long addr, unsigned long end, phys_addr_t phys_addr, pgprot_t prot, unsigned int max_page_shift) { @@ -257,14 +257,24 @@ int vmap_range(unsigned long addr, unsigned long end, break; } while (pgd++, phys_addr += (next - addr), addr = next, addr != end); - flush_cache_vmap(start, end); - if (mask & ARCH_PAGE_TABLE_SYNC_MASK) arch_sync_kernel_mappings(start, end); return err; } +int vmap_range(unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift) +{ + int err; + + err = vmap_range_noflush(addr, end, phys_addr, prot, max_page_shift); + flush_cache_vmap(addr, end); + + return err; +} + static void vunmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, pgtbl_mod_mask *mask) { From patchwork Sun Jan 24 08:22:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 12042167 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CDDE5C433DB for ; Sun, 24 Jan 2021 08:23:59 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5F12922838 for ; Sun, 24 Jan 2021 08:23:59 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5F12922838 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id E9E5E6B0007; Sun, 24 Jan 2021 03:23:58 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E25106B0026; Sun, 24 Jan 2021 03:23:58 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CF08F6B0027; Sun, 24 Jan 2021 03:23:58 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0162.hostedemail.com [216.40.44.162]) by kanga.kvack.org (Postfix) with ESMTP id B3D286B0026 for ; Sun, 24 Jan 2021 03:23:58 -0500 (EST) Received: from smtpin07.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 7FD77363F for ; Sun, 24 Jan 2021 08:23:58 +0000 (UTC) X-FDA: 77739980556.07.kitty96_19011f82757b Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin07.hostedemail.com (Postfix) with ESMTP id 644E91803F9A7 for ; Sun, 24 Jan 2021 08:23:58 +0000 (UTC) X-HE-Tag: kitty96_19011f82757b X-Filterd-Recvd-Size: 18051 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) by imf35.hostedemail.com (Postfix) with ESMTP for ; Sun, 24 Jan 2021 08:23:57 +0000 (UTC) Received: by mail-pl1-f182.google.com with SMTP id d4so5750215plh.5 for ; Sun, 24 Jan 2021 00:23:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=g0NovkKqfzZ3eblNSNRENOmeACxzzhmxFvZbeA7YSuQ=; b=Swy9wtpmyQBWsPfH+naEMLGk9buK3X6nzXYzTQLsb4LW9T1Uqv85S89gQXc0bO0vTe Ys3Sf354aBOKZ3DUnMALjSNA40tfisw2yXx16nCVjd0jCyvZF7XvoJnobEdSaoGjrGmm F/ZA/38LPUrqpwCaH2uCMSrQDPYKh4s4ubQUQZT2KCbngDGAAmXIinHWaQnJaQbV9psB xNEZbkzhz83CmkhgYgmW0NgO2uG67/OCGrEkzmp/RuQ1U3uSAA0XDHmWA4fAFu7I45aA pszNpL3orf4VhJpo/gK9cQqzZJMH4+jDvDuQ6HkR8nKZh7yOROupfElyFQCKtbEYVE2u dBXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=g0NovkKqfzZ3eblNSNRENOmeACxzzhmxFvZbeA7YSuQ=; b=BfVOcGNGFmOq/ghh7qPlKwGojBq6ZyRR1wW1IiD6Mo2Cg3SZSwI0AmWB2M2WXAXhm8 M+hSANmOTFZs1uBl1C8JxMEcWOFWQAB/6ZgeO3DZPLVzzRTX33ZOgJ/fLVjwzm44e59m t0T1cheCDAL+Tk3LwimpQWSHLXWjn3Vge6poGvrwyczBSTrtE9A7Mp9o6bbT9oDs2t7C OId63LXqU0TmFaVHziVldyIu4i3AGG3wHM2WnJUd9JPNrSCXl9E2tVYZS30jgMLCEU0W wao9fU4eWr4d6W1aLy+J6I9jsH2qfSMHsZTokJMMM1pEJpZFhnRqPVYOToXAA2fmEs5w KRyQ== X-Gm-Message-State: AOAM531CnGtV1Kpj7rwfkAXlkNo+AMcLym7f56w/iR6Os+tp65ibTsGW Ax90CzJdIdmw8G7CpoJhEjXR6TE3Ans= X-Google-Smtp-Source: ABdhPJxn8NZFJ60zzbe1LyWmAIknngAM/Y+lT4UhctnhdirFe+/e9UYq20V+bt3Qh/DUNUFEZWmRhg== X-Received: by 2002:a17:902:e886:b029:de:57b2:da69 with SMTP id w6-20020a170902e886b02900de57b2da69mr2941656plg.65.1611476636736; Sun, 24 Jan 2021 00:23:56 -0800 (PST) Received: from bobo.ozlabs.ibm.com ([124.170.13.62]) by smtp.gmail.com with ESMTPSA id gb12sm11799757pjb.51.2021.01.24.00.23.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 24 Jan 2021 00:23:56 -0800 (PST) From: Nicholas Piggin To: linux-mm@kvack.org, Andrew Morton Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Zefan Li , Jonathan Cameron , Christoph Hellwig , Christophe Leroy , Rick Edgecombe , Ding Tianhong Subject: [PATCH v10 11/12] mm/vmalloc: Hugepage vmalloc mappings Date: Sun, 24 Jan 2021 18:22:29 +1000 Message-Id: <20210124082230.2118861-12-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20210124082230.2118861-1-npiggin@gmail.com> References: <20210124082230.2118861-1-npiggin@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Support huge page vmalloc mappings. Config option HAVE_ARCH_HUGE_VMALLOC enables support on architectures that define HAVE_ARCH_HUGE_VMAP and supports PMD sized vmap mappings. vmalloc will attempt to allocate PMD-sized pages if allocating PMD size or larger, and fall back to small pages if that was unsuccessful. Architectures must ensure that any arch specific vmalloc allocations that require PAGE_SIZE mappings (e.g., module allocations vs strict module rwx) use the VM_NOHUGE flag to inhibit larger mappings. When hugepage vmalloc mappings are enabled in the next patch, this reduces TLB misses by nearly 30x on a `git diff` workload on a 2-node POWER9 (59,800 -> 2,100) and reduces CPU cycles by 0.54%. This can result in more internal fragmentation and memory overhead for a given allocation, an option nohugevmalloc is added to disable at boot. Signed-off-by: Nicholas Piggin --- arch/Kconfig | 10 +++ include/linux/vmalloc.h | 18 ++++ mm/page_alloc.c | 5 +- mm/vmalloc.c | 192 ++++++++++++++++++++++++++++++---------- 4 files changed, 177 insertions(+), 48 deletions(-) diff --git a/arch/Kconfig b/arch/Kconfig index 24862d15f3a3..f87feb616184 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -724,6 +724,16 @@ config HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD config HAVE_ARCH_HUGE_VMAP bool +config HAVE_ARCH_HUGE_VMALLOC + depends on HAVE_ARCH_HUGE_VMAP + bool + help + Archs that select this would be capable of PMD-sized vmaps (i.e., + arch_vmap_pmd_supported() returns true), and they must make no + assumptions that vmalloc memory is mapped with PAGE_SIZE ptes. The + VM_NOHUGE flag can be used to prohibit arch-specific allocations from + using hugepages to help with this (e.g., modules may require it). + config ARCH_WANT_HUGE_PMD_SHARE bool diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 40649c4bb5a2..2ba023daf188 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -25,6 +25,7 @@ struct notifier_block; /* in notifier.h */ #define VM_NO_GUARD 0x00000040 /* don't add guard page */ #define VM_KASAN 0x00000080 /* has allocated kasan shadow memory */ #define VM_MAP_PUT_PAGES 0x00000100 /* put pages and free array in vfree */ +#define VM_NOHUGE 0x00000200 /* force PAGE_SIZE pte mapping */ /* * VM_KASAN is used slighly differently depending on CONFIG_KASAN_VMALLOC. @@ -59,6 +60,7 @@ struct vm_struct { unsigned long size; unsigned long flags; struct page **pages; + unsigned int page_order; unsigned int nr_pages; phys_addr_t phys_addr; const void *caller; @@ -194,6 +196,18 @@ static inline void set_vm_flush_reset_perms(void *addr) if (vm) vm->flags |= VM_FLUSH_RESET_PERMS; } + +static inline bool is_vm_area_hugepages(const void *addr) +{ + /* + * This may not 100% tell if the area is mapped with > PAGE_SIZE + * page table entries, if for some reason the architecture indicates + * larger sizes are available but decides not to use them, nothing + * prevents that. This only indicates the size of the physical page + * allocated in the vmalloc layer. + */ + return (find_vm_area(addr)->page_order > 0); +} #else static inline int map_kernel_range_noflush(unsigned long start, unsigned long size, @@ -210,6 +224,10 @@ unmap_kernel_range_noflush(unsigned long addr, unsigned long size) static inline void set_vm_flush_reset_perms(void *addr) { } +static inline bool is_vm_area_hugepages(const void *addr) +{ + return false; +} #endif /* for /dev/kmem */ diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 027f6481ba59..b7a9661fa232 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -72,6 +72,7 @@ #include #include #include +#include #include #include @@ -8238,6 +8239,7 @@ void *__init alloc_large_system_hash(const char *tablename, void *table = NULL; gfp_t gfp_flags; bool virt; + bool huge; /* allow the kernel cmdline to have a say */ if (!numentries) { @@ -8305,6 +8307,7 @@ void *__init alloc_large_system_hash(const char *tablename, } else if (get_order(size) >= MAX_ORDER || hashdist) { table = __vmalloc(size, gfp_flags); virt = true; + huge = is_vm_area_hugepages(table); } else { /* * If bucketsize is not a power-of-two, we may free @@ -8321,7 +8324,7 @@ void *__init alloc_large_system_hash(const char *tablename, pr_info("%s hash table entries: %ld (order: %d, %lu bytes, %s)\n", tablename, 1UL << log2qty, ilog2(size) - PAGE_SHIFT, size, - virt ? "vmalloc" : "linear"); + virt ? (huge ? "vmalloc hugepage" : "vmalloc") : "linear"); if (_hash_shift) *_hash_shift = log2qty; diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 0377e1d059e5..eef61e0f5170 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -42,6 +42,19 @@ #include "internal.h" #include "pgalloc-track.h" +#ifdef CONFIG_HAVE_ARCH_HUGE_VMALLOC +static bool __ro_after_init vmap_allow_huge = true; + +static int __init set_nohugevmalloc(char *str) +{ + vmap_allow_huge = false; + return 0; +} +early_param("nohugevmalloc", set_nohugevmalloc); +#else /* CONFIG_HAVE_ARCH_HUGE_VMALLOC */ +static const bool vmap_allow_huge = false; +#endif /* CONFIG_HAVE_ARCH_HUGE_VMALLOC */ + bool is_vmalloc_addr(const void *x) { unsigned long addr = (unsigned long)x; @@ -477,31 +490,12 @@ static int vmap_pages_p4d_range(pgd_t *pgd, unsigned long addr, return 0; } -/** - * map_kernel_range_noflush - map kernel VM area with the specified pages - * @addr: start of the VM area to map - * @size: size of the VM area to map - * @prot: page protection flags to use - * @pages: pages to map - * - * Map PFN_UP(@size) pages at @addr. The VM area @addr and @size specify should - * have been allocated using get_vm_area() and its friends. - * - * NOTE: - * This function does NOT do any cache flushing. The caller is responsible for - * calling flush_cache_vmap() on to-be-mapped areas before calling this - * function. - * - * RETURNS: - * 0 on success, -errno on failure. - */ -int map_kernel_range_noflush(unsigned long addr, unsigned long size, - pgprot_t prot, struct page **pages) +static int vmap_small_pages_range_noflush(unsigned long addr, unsigned long end, + pgprot_t prot, struct page **pages) { unsigned long start = addr; - unsigned long end = addr + size; - unsigned long next; pgd_t *pgd; + unsigned long next; int err = 0; int nr = 0; pgtbl_mod_mask mask = 0; @@ -523,6 +517,65 @@ int map_kernel_range_noflush(unsigned long addr, unsigned long size, return 0; } +static int vmap_pages_range_noflush(unsigned long addr, unsigned long end, + pgprot_t prot, struct page **pages, unsigned int page_shift) +{ + unsigned int i, nr = (end - addr) >> PAGE_SHIFT; + + WARN_ON(page_shift < PAGE_SHIFT); + + if (page_shift == PAGE_SHIFT) + return vmap_small_pages_range_noflush(addr, end, prot, pages); + + for (i = 0; i < nr; i += 1U << (page_shift - PAGE_SHIFT)) { + int err; + + err = vmap_range_noflush(addr, addr + (1UL << page_shift), + __pa(page_address(pages[i])), prot, + page_shift); + if (err) + return err; + + addr += 1UL << page_shift; + } + + return 0; +} + +static int vmap_pages_range(unsigned long addr, unsigned long end, + pgprot_t prot, struct page **pages, unsigned int page_shift) +{ + int err; + + err = vmap_pages_range_noflush(addr, end, prot, pages, page_shift); + flush_cache_vmap(addr, end); + return err; +} + +/** + * map_kernel_range_noflush - map kernel VM area with the specified pages + * @addr: start of the VM area to map + * @size: size of the VM area to map + * @prot: page protection flags to use + * @pages: pages to map + * + * Map PFN_UP(@size) pages at @addr. The VM area @addr and @size specify should + * have been allocated using get_vm_area() and its friends. + * + * NOTE: + * This function does NOT do any cache flushing. The caller is responsible for + * calling flush_cache_vmap() on to-be-mapped areas before calling this + * function. + * + * RETURNS: + * 0 on success, -errno on failure. + */ +int map_kernel_range_noflush(unsigned long addr, unsigned long size, + pgprot_t prot, struct page **pages) +{ + return vmap_pages_range_noflush(addr, addr + size, prot, pages, PAGE_SHIFT); +} + int map_kernel_range(unsigned long start, unsigned long size, pgprot_t prot, struct page **pages) { @@ -2416,6 +2469,7 @@ static inline void set_area_direct_map(const struct vm_struct *area, { int i; + /* HUGE_VMALLOC passes small pages to set_direct_map */ for (i = 0; i < area->nr_pages; i++) if (page_address(area->pages[i])) set_direct_map(area->pages[i]); @@ -2449,11 +2503,12 @@ static void vm_remove_mappings(struct vm_struct *area, int deallocate_pages) * map. Find the start and end range of the direct mappings to make sure * the vm_unmap_aliases() flush includes the direct map. */ - for (i = 0; i < area->nr_pages; i++) { + for (i = 0; i < area->nr_pages; i += 1U << area->page_order) { unsigned long addr = (unsigned long)page_address(area->pages[i]); if (addr) { + unsigned long page_size = PAGE_SIZE << area->page_order; start = min(addr, start); - end = max(addr + PAGE_SIZE, end); + end = max(addr + page_size, end); flush_dmap = 1; } } @@ -2496,11 +2551,11 @@ static void __vunmap(const void *addr, int deallocate_pages) if (deallocate_pages) { int i; - for (i = 0; i < area->nr_pages; i++) { + for (i = 0; i < area->nr_pages; i += 1U << area->page_order) { struct page *page = area->pages[i]; BUG_ON(!page); - __free_pages(page, 0); + __free_pages(page, area->page_order); } atomic_long_sub(area->nr_pages, &nr_vmalloc_pages); @@ -2691,15 +2746,18 @@ EXPORT_SYMBOL_GPL(vmap_pfn); #endif /* CONFIG_VMAP_PFN */ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, - pgprot_t prot, int node) + pgprot_t prot, unsigned int page_shift, + int node) { const gfp_t nested_gfp = (gfp_mask & GFP_RECLAIM_MASK) | __GFP_ZERO; - unsigned int nr_pages = get_vm_area_size(area) >> PAGE_SHIFT; - unsigned long array_size; - unsigned int i; + unsigned int page_order = page_shift - PAGE_SHIFT; + unsigned long addr = (unsigned long)area->addr; + unsigned long size = get_vm_area_size(area); + unsigned int nr_small_pages = size >> PAGE_SHIFT; struct page **pages; + unsigned int i; - array_size = (unsigned long)nr_pages * sizeof(struct page *); + array_size = (unsigned long)nr_small_pages * sizeof(struct page *); gfp_mask |= __GFP_NOWARN; if (!(gfp_mask & (GFP_DMA | GFP_DMA32))) gfp_mask |= __GFP_HIGHMEM; @@ -2718,30 +2776,35 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, } area->pages = pages; - area->nr_pages = nr_pages; + area->nr_pages = nr_small_pages; + area->page_order = page_order; - for (i = 0; i < area->nr_pages; i++) { + /* + * Careful, we allocate and map page_order pages, but tracking is done + * per PAGE_SIZE page so as to keep the vm_struct APIs independent of + * the physical/mapped size. + */ + for (i = 0; i < area->nr_pages; i += 1U << page_order) { struct page *page; + int p; - if (node == NUMA_NO_NODE) - page = alloc_page(gfp_mask); - else - page = alloc_pages_node(node, gfp_mask, 0); - + page = alloc_pages_node(node, gfp_mask, page_order); if (unlikely(!page)) { /* Successfully allocated i pages, free them in __vfree() */ area->nr_pages = i; atomic_long_add(area->nr_pages, &nr_vmalloc_pages); goto fail; } - area->pages[i] = page; + + for (p = 0; p < (1U << page_order); p++) + area->pages[i + p] = page + p; + if (gfpflags_allow_blocking(gfp_mask)) cond_resched(); } atomic_long_add(area->nr_pages, &nr_vmalloc_pages); - if (map_kernel_range((unsigned long)area->addr, get_vm_area_size(area), - prot, pages) < 0) + if (vmap_pages_range(addr, addr + size, prot, pages, page_shift) < 0) goto fail; return area->addr; @@ -2749,7 +2812,7 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, fail: warn_alloc(gfp_mask, NULL, "vmalloc: allocation failure, allocated %ld of %ld bytes", - (area->nr_pages*PAGE_SIZE), area->size); + (area->nr_pages*PAGE_SIZE), size); __vfree(area->addr); return NULL; } @@ -2780,19 +2843,44 @@ void *__vmalloc_node_range(unsigned long size, unsigned long align, struct vm_struct *area; void *addr; unsigned long real_size = size; + unsigned long real_align = align; + unsigned int shift = PAGE_SHIFT; - size = PAGE_ALIGN(size); if (!size || (size >> PAGE_SHIFT) > totalram_pages()) goto fail; - area = __get_vm_area_node(real_size, align, VM_ALLOC | VM_UNINITIALIZED | + if (vmap_allow_huge && !(vm_flags & VM_NOHUGE) && + arch_vmap_pmd_supported(prot) && + (pgprot_val(prot) == pgprot_val(PAGE_KERNEL))) { + unsigned long size_per_node; + + /* + * Try huge pages. Only try for PAGE_KERNEL allocations, + * others like modules don't yet expect huge pages in + * their allocations due to apply_to_page_range not + * supporting them. + */ + + size_per_node = size; + if (node == NUMA_NO_NODE) + size_per_node /= num_online_nodes(); + if (size_per_node >= PMD_SIZE) { + shift = PMD_SHIFT; + align = max(real_align, 1UL << shift); + size = ALIGN(real_size, 1UL << shift); + } + } + +again: + size = PAGE_ALIGN(size); + area = __get_vm_area_node(size, align, VM_ALLOC | VM_UNINITIALIZED | vm_flags, start, end, node, gfp_mask, caller); if (!area) goto fail; - addr = __vmalloc_area_node(area, gfp_mask, prot, node); + addr = __vmalloc_area_node(area, gfp_mask, prot, shift, node); if (!addr) - return NULL; + goto fail; /* * In this function, newly allocated vm_struct has VM_UNINITIALIZED @@ -2806,8 +2894,18 @@ void *__vmalloc_node_range(unsigned long size, unsigned long align, return addr; fail: - warn_alloc(gfp_mask, NULL, + if (shift > PAGE_SHIFT) { + shift = PAGE_SHIFT; + align = real_align; + size = real_size; + goto again; + } + + if (!area) { + /* Warn for area allocation, page allocations already warn */ + warn_alloc(gfp_mask, NULL, "vmalloc: allocation failure: %lu bytes", real_size); + } return NULL; } From patchwork Sun Jan 24 08:22:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 12042169 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6FFF2C433DB for ; Sun, 24 Jan 2021 08:24:04 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 17ADF22838 for ; Sun, 24 Jan 2021 08:24:04 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 17ADF22838 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id AF32C6B0026; Sun, 24 Jan 2021 03:24:03 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AA3986B0027; Sun, 24 Jan 2021 03:24:03 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9B9FA6B0028; Sun, 24 Jan 2021 03:24:03 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0116.hostedemail.com [216.40.44.116]) by kanga.kvack.org (Postfix) with ESMTP id 85C646B0026 for ; Sun, 24 Jan 2021 03:24:03 -0500 (EST) Received: from smtpin24.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 5AF4D180ACF84 for ; Sun, 24 Jan 2021 08:24:03 +0000 (UTC) X-FDA: 77739980766.24.crowd13_0a084ec2757b Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin24.hostedemail.com (Postfix) with ESMTP id 3003F1A4A0 for ; Sun, 24 Jan 2021 08:24:03 +0000 (UTC) X-HE-Tag: crowd13_0a084ec2757b X-Filterd-Recvd-Size: 5545 Received: from mail-pl1-f173.google.com (mail-pl1-f173.google.com [209.85.214.173]) by imf05.hostedemail.com (Postfix) with ESMTP for ; Sun, 24 Jan 2021 08:24:02 +0000 (UTC) Received: by mail-pl1-f173.google.com with SMTP id r4so5721408pls.11 for ; Sun, 24 Jan 2021 00:24:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=X4PdAkKAj1M/Yo1TeEkwcmE5+wLYrx2vqWkun80SjU4=; b=SFxxpwF4aZqAtiyha8DpNwD8NhP2VPk/75SmbcOcUMe8/tsdfJEhfXIo72nG1U14DU /Lfme/sKGozWIGX0E2hSH8Mi7eKm/t+eYpIlLC1+AGsIAouFEPWAWEPQ5E2m1pe4ABGR 35pNfkvvqCINxLbhZT5Hq4lI7a3fhieQBBuWzw14T0cITiJIeHWoZrFD0AVaieTifYF3 jM6UOz0iA0NrXmVRl77vIVdAkSKXjl4Bf57i/GvL3hmvpx0acyI1x74L9491tfkkkeMK zEfDoR3CJ3WHkKTAVtJmYF4vJM1atUQiY108Xs6ol5GzVxQ9tLCHsdLKFzL4raSrcER4 X60g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=X4PdAkKAj1M/Yo1TeEkwcmE5+wLYrx2vqWkun80SjU4=; b=eYGqnxM82scQdznmyGJkrK+SlBeh/kXWqypn/C8STTo6/BgXd530+tvE1Ff2XdCiuV zxxfasettYD5yht4CRZ5uovWN6YLMpXVN8M9wMMXO/3C/0Zp0gAjrxF8vnuOcvZWOP0h HyoAPZnRm5L79bofN2r8C3pRZZJu0/C149Ou6CSySYMXDLISXhKBOjkS8xB4LZw5H5dh PsAQ0X+RngLKwR33b3UQ7TDEmPVp6+gD/lpYfKNcBpXHgewhBfHftiP9OBUbdEXKQdrB WTflV1t38P6DG0sfOVXXIfC67WuUmytL06VHTHfmW1UnTiIOEaj3lHel7TpS9wjP5JIs paUg== X-Gm-Message-State: AOAM532BgMOItcAxAISrVCXIhUreX4QcSaHGBxhjH855/odQHGgA/ybA My1k/4LMiH2NQjxBq97zXQZDdB/Ec38= X-Google-Smtp-Source: ABdhPJwWSI9slFc6FiGRew3Dj4/h6/vkxLw9qnQk1wa1Zw10/YWu1xYbDC+boiJwyVxb/47ASvxmaA== X-Received: by 2002:a17:902:e84e:b029:de:45bf:1296 with SMTP id t14-20020a170902e84eb02900de45bf1296mr7950370plg.49.1611476641776; Sun, 24 Jan 2021 00:24:01 -0800 (PST) Received: from bobo.ozlabs.ibm.com ([124.170.13.62]) by smtp.gmail.com with ESMTPSA id gb12sm11799757pjb.51.2021.01.24.00.23.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 24 Jan 2021 00:24:01 -0800 (PST) From: Nicholas Piggin To: linux-mm@kvack.org, Andrew Morton Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Zefan Li , Jonathan Cameron , Christoph Hellwig , Christophe Leroy , Rick Edgecombe , Ding Tianhong Subject: [PATCH v10 12/12] powerpc/64s/radix: Enable huge vmalloc mappings Date: Sun, 24 Jan 2021 18:22:30 +1000 Message-Id: <20210124082230.2118861-13-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20210124082230.2118861-1-npiggin@gmail.com> References: <20210124082230.2118861-1-npiggin@gmail.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Cc: linuxppc-dev@lists.ozlabs.org Signed-off-by: Nicholas Piggin --- Documentation/admin-guide/kernel-parameters.txt | 2 ++ arch/powerpc/Kconfig | 1 + arch/powerpc/kernel/module.c | 13 +++++++++++-- 3 files changed, 14 insertions(+), 2 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index a10b545c2070..d62df53e5200 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -3225,6 +3225,8 @@ nohugeiomap [KNL,X86,PPC,ARM64] Disable kernel huge I/O mappings. + nohugevmalloc [PPC] Disable kernel huge vmalloc mappings. + nosmt [KNL,S390] Disable symmetric multithreading (SMT). Equivalent to smt=1. diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index 107bb4319e0e..781da6829ab7 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -181,6 +181,7 @@ config PPC select GENERIC_GETTIMEOFDAY select HAVE_ARCH_AUDITSYSCALL select HAVE_ARCH_HUGE_VMAP if PPC_BOOK3S_64 && PPC_RADIX_MMU + select HAVE_ARCH_HUGE_VMALLOC if HAVE_ARCH_HUGE_VMAP select HAVE_ARCH_JUMP_LABEL select HAVE_ARCH_KASAN if PPC32 && PPC_PAGE_SHIFT <= 14 select HAVE_ARCH_KASAN_VMALLOC if PPC32 && PPC_PAGE_SHIFT <= 14 diff --git a/arch/powerpc/kernel/module.c b/arch/powerpc/kernel/module.c index a211b0253cdb..bc2695eeeb4c 100644 --- a/arch/powerpc/kernel/module.c +++ b/arch/powerpc/kernel/module.c @@ -92,8 +92,17 @@ void *module_alloc(unsigned long size) { BUILD_BUG_ON(TASK_SIZE > MODULES_VADDR); - return __vmalloc_node_range(size, 1, MODULES_VADDR, MODULES_END, GFP_KERNEL, - PAGE_KERNEL_EXEC, VM_FLUSH_RESET_PERMS, NUMA_NO_NODE, + /* + * Don't do huge page allocations for modules yet until more testing + * is done. STRICT_MODULE_RWX may require extra work to support this + * too. + */ + + return __vmalloc_node_range(size, 1, MODULES_VADDR, MODULES_END, + GFP_KERNEL, + PAGE_KERNEL_EXEC, + VM_NOHUGE | VM_FLUSH_RESET_PERMS, + NUMA_NO_NODE, __builtin_return_address(0)); } #endif