From patchwork Wed Mar 17 06:23:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 12144879 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A5ADCC433E0 for ; Wed, 17 Mar 2021 06:24:31 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 253D364F96 for ; Wed, 17 Mar 2021 06:24:31 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 253D364F96 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id AB5756B0073; Wed, 17 Mar 2021 02:24:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A64F06B0074; Wed, 17 Mar 2021 02:24:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 906D26B0075; Wed, 17 Mar 2021 02:24:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0146.hostedemail.com [216.40.44.146]) by kanga.kvack.org (Postfix) with ESMTP id 766546B0073 for ; Wed, 17 Mar 2021 02:24:30 -0400 (EDT) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 2FE24181AF5F7 for ; Wed, 17 Mar 2021 06:24:30 +0000 (UTC) X-FDA: 77928377100.17.796E318 Received: from mail-pl1-f177.google.com (mail-pl1-f177.google.com [209.85.214.177]) by imf07.hostedemail.com (Postfix) with ESMTP id BAD20A0000FC for ; Wed, 17 Mar 2021 06:24:29 +0000 (UTC) Received: by mail-pl1-f177.google.com with SMTP id e2so268691pld.9 for ; Tue, 16 Mar 2021 23:24:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=vAbu4uylyD/1yZsJCHQjYIcITelz2x29iNZ/QiXVtA4=; b=sXb2QLLIzWJSM+Zi+acrl+MEb1zX47TUFWvEPje56QooIkKUgZXMeMimDdGJUXgX9J 1WvPANwCYbJxwUTnOOm5W8F/OLX2VpjutsPFQg4uOs2mvGm8doD5GsJJ13j54ULp1C1P jh1im1vMwuhKWECyIqpMLcJr78DwSrzAYvzNcphn11O7eWgNETP1pyQWqOy/+s0bhR4m 2kEEAUNYMIrnYwc7qMhpZQnc8QyPT1k9gcefN4vnGFpxLM0SLUG3foeuVY1lqxMUz7y3 2PcMEPrmyVJAHyQwrHt5vdJB3xO8ebkFhXJ14d0rNpOs7F9G7ulHI5o+BKXIQLDVmLvh 4uhw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=vAbu4uylyD/1yZsJCHQjYIcITelz2x29iNZ/QiXVtA4=; b=m5/g4hKsoMRdtTUXa4Fd1ECkRnJ2fEWRtOq8d5GxT7uIB9zgQXIznx9n1QEHlZliIJ coQsc5v2HJI7P9b6jBvCS+i3pDcxBxcPkvKrk8huVLP8Atu9+kBYydVCHf3AaLf7UhW/ RXyvIZduFbjzwJUChxMDzyXPDPwMMShHq7hISDpZi3k75FN68o3fcbdOJXp6C7uewSm9 RoODctE1Sf5Q09q7QnK5NeVMMgCkX68WQ8/NMlsxSjMiQYvH4RT0+GYVTNpAKWmQ2F03 Up1nsQtEKVhlg37MjcuBLMFtEgr1JFlL+xLZOwWf8CSEWoyjA52dc87DIoazr1d4ruHb PbOQ== X-Gm-Message-State: AOAM532mn6zCOfMQAOaRoSjDyW2GHfp+e/vviY9oDnePjA7F2eNkUV4n 4TQZ9n9IM5zHsqTPIMvURtu6Jgnmyns= X-Google-Smtp-Source: ABdhPJz1uqgDCGDQAR9TanybbKMGcsepIcdLLEpecSZtq52NFToIck/CqpiGOQ15vW6GJFJxlSh+vQ== X-Received: by 2002:a17:902:ead5:b029:e5:bd04:bf48 with SMTP id p21-20020a170902ead5b02900e5bd04bf48mr3083048pld.38.1615962268349; Tue, 16 Mar 2021 23:24:28 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (58-6-239-121.tpgi.com.au. [58.6.239.121]) by smtp.gmail.com with ESMTPSA id s19sm17959620pfh.168.2021.03.16.23.24.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Mar 2021 23:24:27 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org, Andrew Morton Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, Jonathan Cameron , Christoph Hellwig , Christophe Leroy , Rick Edgecombe , Ding Tianhong , Russell King , linux-arm-kernel@lists.infradead.org Subject: [PATCH v13 01/14] ARM: mm: add missing pud_page define to 2-level page tables Date: Wed, 17 Mar 2021 16:23:49 +1000 Message-Id: <20210317062402.533919-2-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20210317062402.533919-1-npiggin@gmail.com> References: <20210317062402.533919-1-npiggin@gmail.com> MIME-Version: 1.0 X-Stat-Signature: gtmu6kppftgd46skg4pi4j4hx6sr9khg X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: BAD20A0000FC Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf07; identity=mailfrom; envelope-from=""; helo=mail-pl1-f177.google.com; client-ip=209.85.214.177 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1615962269-466889 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: ARM uses its own PMD folding scheme which is missing pud_page which should just pass through to pmd_page. Move this from the 3-level page table to common header. Cc: Russell King Cc: Ding Tianhong Cc: linux-arm-kernel@lists.infradead.org Signed-off-by: Nicholas Piggin --- arch/arm/include/asm/pgtable-3level.h | 2 -- arch/arm/include/asm/pgtable.h | 3 +++ 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/arch/arm/include/asm/pgtable-3level.h b/arch/arm/include/asm/pgtable-3level.h index 2b85d175e999..d4edab51a77c 100644 --- a/arch/arm/include/asm/pgtable-3level.h +++ b/arch/arm/include/asm/pgtable-3level.h @@ -186,8 +186,6 @@ static inline pte_t pte_mkspecial(pte_t pte) #define pmd_write(pmd) (pmd_isclear((pmd), L_PMD_SECT_RDONLY)) #define pmd_dirty(pmd) (pmd_isset((pmd), L_PMD_SECT_DIRTY)) -#define pud_page(pud) pmd_page(__pmd(pud_val(pud))) -#define pud_write(pud) pmd_write(__pmd(pud_val(pud))) #define pmd_hugewillfault(pmd) (!pmd_young(pmd) || !pmd_write(pmd)) #define pmd_thp_or_huge(pmd) (pmd_huge(pmd) || pmd_trans_huge(pmd)) diff --git a/arch/arm/include/asm/pgtable.h b/arch/arm/include/asm/pgtable.h index c02f24400369..d63a5bb6bd0c 100644 --- a/arch/arm/include/asm/pgtable.h +++ b/arch/arm/include/asm/pgtable.h @@ -166,6 +166,9 @@ extern struct page *empty_zero_page; extern pgd_t swapper_pg_dir[PTRS_PER_PGD]; +#define pud_page(pud) pmd_page(__pmd(pud_val(pud))) +#define pud_write(pud) pmd_write(__pmd(pud_val(pud))) + #define pmd_none(pmd) (!pmd_val(pmd)) static inline pte_t *pmd_page_vaddr(pmd_t pmd) From patchwork Wed Mar 17 06:23:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 12144881 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 29A9EC433DB for ; Wed, 17 Mar 2021 06:24:42 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id BE93564F8C for ; Wed, 17 Mar 2021 06:24:41 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BE93564F8C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 585226B0074; Wed, 17 Mar 2021 02:24:41 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 536F58D0001; Wed, 17 Mar 2021 02:24:41 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3AEB76B0078; Wed, 17 Mar 2021 02:24:41 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0077.hostedemail.com [216.40.44.77]) by kanga.kvack.org (Postfix) with ESMTP id 1FA026B0074 for ; Wed, 17 Mar 2021 02:24:41 -0400 (EDT) Received: from smtpin26.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id D3A55180AD82F for ; Wed, 17 Mar 2021 06:24:40 +0000 (UTC) X-FDA: 77928377520.26.44A95D7 Received: from mail-pf1-f178.google.com (mail-pf1-f178.google.com [209.85.210.178]) by imf17.hostedemail.com (Postfix) with ESMTP id 5ADF0407F8C4 for ; Wed, 17 Mar 2021 06:24:40 +0000 (UTC) Received: by mail-pf1-f178.google.com with SMTP id h3so405707pfr.12 for ; Tue, 16 Mar 2021 23:24:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=lKopX2yPijMVEzlZ+by8oeAN6QHVUP6629vS5TVZX0w=; b=ljBGC2U1NROLE07LGNWEtUl7p0/i4HyY6IUXQn2OjTwDQSErggxTgWPRanpnwkR4Kf 84x0GwHuWOFIX4QhYsgbVHSqH6wW7Vns7iWtHasyamaZrIKi4fuoGNIx5HQEZtX5/6bC BIq7+o0sQRSMV0BLXUGa+iRqUR6wiU1AkkwnvE5rpGoTZFAeAB1AlbCsk4EUpePOOZGX VQfjPnGR9Gz+xYivrdFmvF+RRV5Jm4nBJResii1pYeB7pM3dEjmztPYd4V2ly8hW0jgH mvQBhxSnXyWSueeld7+YWhFzrr3OOAMZicG+2dcseG+FGIfOIe17g4NLjPf/40/8r8A0 fFmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=lKopX2yPijMVEzlZ+by8oeAN6QHVUP6629vS5TVZX0w=; b=lGfONqxQ89wWvkEqlTOzBzy0cgeOe2efuffZ4t6LSFM5Bib8UAAuBjMtF+a/Korcdu mLJXIMJBJeE+O7TUvekwGKM4AsFd2lSqyCzetI+tNe2kvdkmHrHqoLfOTMROUlIAEc9l omXWFeto4ma4/6LnOv6arfkq+IoT1BxmoKchO08c2gEIPiakR6HHnt+6lhzO6iiz2bIF cwkKm5MhsNDD4/4C6j9DMv/cxzJVDjwj31N4mcpTBKibaen+AA4OnxHErY9RQSx+oQWB AYwy6aGTLYTGGyo4iybyYHfLy/VUMBO4Nlq2eP17gg/sRHs9Rvauf/gCIyeAIAaWiyl1 3rwQ== X-Gm-Message-State: AOAM532KfpBZGa6xtrWRMbl96OE8bImD2K3aUVrtJVK9E2ysp+Sp17Nn lRlhGRfBQMINg95SKMBBYB5kCyTrLtY= X-Google-Smtp-Source: ABdhPJz+3AmYll44tulolkLRLJTiuXsjmwqw8nQ4Q+GAOFyMSJVu9YQ6bjHhZwfR5lV89hqngDl0DQ== X-Received: by 2002:a63:1906:: with SMTP id z6mr1366452pgl.292.1615962278920; Tue, 16 Mar 2021 23:24:38 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (58-6-239-121.tpgi.com.au. [58.6.239.121]) by smtp.gmail.com with ESMTPSA id s19sm17959620pfh.168.2021.03.16.23.24.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Mar 2021 23:24:38 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org, Andrew Morton Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, Jonathan Cameron , Christoph Hellwig , Christophe Leroy , Rick Edgecombe , Ding Tianhong , Miaohe Lin , Christoph Hellwig Subject: [PATCH v13 02/14] mm/vmalloc: fix HUGE_VMAP regression by enabling huge pages in vmalloc_to_page Date: Wed, 17 Mar 2021 16:23:50 +1000 Message-Id: <20210317062402.533919-3-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20210317062402.533919-1-npiggin@gmail.com> References: <20210317062402.533919-1-npiggin@gmail.com> MIME-Version: 1.0 X-Stat-Signature: xpb4hirghoott74h3nwmo33tq7hf7pcx X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 5ADF0407F8C4 Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf17; identity=mailfrom; envelope-from=""; helo=mail-pf1-f178.google.com; client-ip=209.85.210.178 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1615962280-242870 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: vmalloc_to_page returns NULL for addresses mapped by larger pages[*]. Whether or not a vmap is huge depends on the architecture details, alignments, boot options, etc., which the caller can not be expected to know. Therefore HUGE_VMAP is a regression for vmalloc_to_page. This change teaches vmalloc_to_page about larger pages, and returns the struct page that corresponds to the offset within the large page. This makes the API agnostic to mapping implementation details. [*] As explained by commit 029c54b095995 ("mm/vmalloc.c: huge-vmap: fail gracefully on unexpected huge vmap mappings") Reviewed-by: Miaohe Lin Reviewed-by: Christoph Hellwig Signed-off-by: Nicholas Piggin --- mm/vmalloc.c | 41 ++++++++++++++++++++++++++--------------- 1 file changed, 26 insertions(+), 15 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 4f5f8c907897..98e697ac764c 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -34,7 +34,7 @@ #include #include #include - +#include #include #include #include @@ -343,7 +343,9 @@ int is_vmalloc_or_module_addr(const void *x) } /* - * Walk a vmap address to the struct page it maps. + * Walk a vmap address to the struct page it maps. Huge vmap mappings will + * return the tail page that corresponds to the base page address, which + * matches small vmap mappings. */ struct page *vmalloc_to_page(const void *vmalloc_addr) { @@ -363,25 +365,33 @@ struct page *vmalloc_to_page(const void *vmalloc_addr) if (pgd_none(*pgd)) return NULL; + if (WARN_ON_ONCE(pgd_leaf(*pgd))) + return NULL; /* XXX: no allowance for huge pgd */ + if (WARN_ON_ONCE(pgd_bad(*pgd))) + return NULL; + p4d = p4d_offset(pgd, addr); if (p4d_none(*p4d)) return NULL; - pud = pud_offset(p4d, addr); + if (p4d_leaf(*p4d)) + return p4d_page(*p4d) + ((addr & ~P4D_MASK) >> PAGE_SHIFT); + if (WARN_ON_ONCE(p4d_bad(*p4d))) + return NULL; - /* - * Don't dereference bad PUD or PMD (below) entries. This will also - * identify huge mappings, which we may encounter on architectures - * that define CONFIG_HAVE_ARCH_HUGE_VMAP=y. Such regions will be - * identified as vmalloc addresses by is_vmalloc_addr(), but are - * not [unambiguously] associated with a struct page, so there is - * no correct value to return for them. - */ - WARN_ON_ONCE(pud_bad(*pud)); - if (pud_none(*pud) || pud_bad(*pud)) + pud = pud_offset(p4d, addr); + if (pud_none(*pud)) + return NULL; + if (pud_leaf(*pud)) + return pud_page(*pud) + ((addr & ~PUD_MASK) >> PAGE_SHIFT); + if (WARN_ON_ONCE(pud_bad(*pud))) return NULL; + pmd = pmd_offset(pud, addr); - WARN_ON_ONCE(pmd_bad(*pmd)); - if (pmd_none(*pmd) || pmd_bad(*pmd)) + if (pmd_none(*pmd)) + return NULL; + if (pmd_leaf(*pmd)) + return pmd_page(*pmd) + ((addr & ~PMD_MASK) >> PAGE_SHIFT); + if (WARN_ON_ONCE(pmd_bad(*pmd))) return NULL; ptep = pte_offset_map(pmd, addr); @@ -389,6 +399,7 @@ struct page *vmalloc_to_page(const void *vmalloc_addr) if (pte_present(pte)) page = pte_page(pte); pte_unmap(ptep); + return page; } EXPORT_SYMBOL(vmalloc_to_page); From patchwork Wed Mar 17 06:23:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 12144883 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A15A5C433E0 for ; Wed, 17 Mar 2021 06:24:48 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1B8F364F6B for ; Wed, 17 Mar 2021 06:24:48 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1B8F364F6B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B03656B0075; Wed, 17 Mar 2021 02:24:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id ADA286B0078; Wed, 17 Mar 2021 02:24:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 97AF86B007B; Wed, 17 Mar 2021 02:24:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0130.hostedemail.com [216.40.44.130]) by kanga.kvack.org (Postfix) with ESMTP id 7EB966B0075 for ; Wed, 17 Mar 2021 02:24:47 -0400 (EDT) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 244425DD5 for ; Wed, 17 Mar 2021 06:24:47 +0000 (UTC) X-FDA: 77928377814.15.42EEBE3 Received: from mail-pj1-f51.google.com (mail-pj1-f51.google.com [209.85.216.51]) by imf13.hostedemail.com (Postfix) with ESMTP id AD715E0001AF for ; Wed, 17 Mar 2021 06:24:46 +0000 (UTC) Received: by mail-pj1-f51.google.com with SMTP id nh23-20020a17090b3657b02900c0d5e235a8so798473pjb.0 for ; Tue, 16 Mar 2021 23:24:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=++PkiUNNWPZSxFQ6Sb0mw9vxh3qzOWEkVwjCIjWdkRI=; b=ajvvPRf/jAipx/skovC3jeKAcpcNl2dzAog04Ee99vI+uwnUNL3Oa+xSiOerHr/y18 xETFawp087vMSzK0HIK8ULa9i34AwtpgV28b4PHaF7n+2bzqGF6WEtnuJ6obraTqUZbQ Q7Bf8L41d5AixGe+ZV+GYcKlVNeefTpggXcBzb5nUiRVVKjqLzPPBKR8A631nVi2+q9m P5a3WLa8z4YF6DI9eQYBHOVkMIJcuIUB9UA65Bx+VnTSXtTtxLTIJtQrqX+ZcXeRhKxI mVtOAo2kDmFO+5gc4GgYH7g0Dc9Zmw9+MNhvTjjqj6tQpZVfRGOzI2Cgg2a22gO8Y13b UCMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=++PkiUNNWPZSxFQ6Sb0mw9vxh3qzOWEkVwjCIjWdkRI=; b=YgOl55IJq0E/nfnVirmVsArBpgvBSsNa7k6LTNe7PmYPA6uN5v9D+Gxd/04Ej5rvQT 2wH72WUvpAw85KokEbxlagjaVloFv2Tz1ejHU4jRB0l3xc2+D+5w8I7nm46oq6T2wr45 2lbn00T0Tzt7sJplWrwo4oGObExzm1hVmGRr0R6911wATM9porZUE3MyAKmkoxof9t+l vEDhKV2QNlMV16+wLFzcL7Gqe7LuAy7OQI67pBc8o1ATmdVM01Wx9EcfF1qxt3oROwnC 9ofDOYfXeJx6yYoT/RtGUZEeOxllR+YHmmw5nYJxzwRY0MtwM/3R7L0frI6YQ8+lzyWB BknQ== X-Gm-Message-State: AOAM533hD3Jt/Z2iZYXUqB+F+1NT59YlibMTwx9MIneAiKK2EFe/qRpx n7XaQM8Mc/iuyBhvABmcDQewvP7tS8M= X-Google-Smtp-Source: ABdhPJxRO/kglfO5AeKNNHy2+ogUVcnIVIzFdCXy48JSpUWwjO5ZVjhDk8TJQyAPNU8ABmQPHBgM/A== X-Received: by 2002:a17:90a:6be4:: with SMTP id w91mr3084207pjj.68.1615962285498; Tue, 16 Mar 2021 23:24:45 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (58-6-239-121.tpgi.com.au. [58.6.239.121]) by smtp.gmail.com with ESMTPSA id s19sm17959620pfh.168.2021.03.16.23.24.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Mar 2021 23:24:44 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org, Andrew Morton Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, Jonathan Cameron , Christoph Hellwig , Christophe Leroy , Rick Edgecombe , Ding Tianhong , Miaohe Lin , Christoph Hellwig Subject: [PATCH v13 03/14] mm: apply_to_pte_range warn and fail if a large pte is encountered Date: Wed, 17 Mar 2021 16:23:51 +1000 Message-Id: <20210317062402.533919-4-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20210317062402.533919-1-npiggin@gmail.com> References: <20210317062402.533919-1-npiggin@gmail.com> MIME-Version: 1.0 X-Stat-Signature: hctm99n9fhnu5969hhwng6mpzm543sad X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: AD715E0001AF Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf13; identity=mailfrom; envelope-from=""; helo=mail-pj1-f51.google.com; client-ip=209.85.216.51 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1615962286-651459 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: apply_to_pte_range might mistake a large pte for bad, or treat it as a page table, resulting in a crash or corruption. Add a test to warn and return error if large entries are found. Reviewed-by: Miaohe Lin Reviewed-by: Christoph Hellwig Signed-off-by: Nicholas Piggin --- mm/memory.c | 66 +++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 49 insertions(+), 17 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 5efa07fb6cdc..ccaf74f070c0 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -2446,13 +2446,21 @@ static int apply_to_pmd_range(struct mm_struct *mm, pud_t *pud, } do { next = pmd_addr_end(addr, end); - if (create || !pmd_none_or_clear_bad(pmd)) { - err = apply_to_pte_range(mm, pmd, addr, next, fn, data, - create, mask); - if (err) - break; + if (pmd_none(*pmd) && !create) + continue; + if (WARN_ON_ONCE(pmd_leaf(*pmd))) + return -EINVAL; + if (!pmd_none(*pmd) && WARN_ON_ONCE(pmd_bad(*pmd))) { + if (!create) + continue; + pmd_clear_bad(pmd); } + err = apply_to_pte_range(mm, pmd, addr, next, + fn, data, create, mask); + if (err) + break; } while (pmd++, addr = next, addr != end); + return err; } @@ -2474,13 +2482,21 @@ static int apply_to_pud_range(struct mm_struct *mm, p4d_t *p4d, } do { next = pud_addr_end(addr, end); - if (create || !pud_none_or_clear_bad(pud)) { - err = apply_to_pmd_range(mm, pud, addr, next, fn, data, - create, mask); - if (err) - break; + if (pud_none(*pud) && !create) + continue; + if (WARN_ON_ONCE(pud_leaf(*pud))) + return -EINVAL; + if (!pud_none(*pud) && WARN_ON_ONCE(pud_bad(*pud))) { + if (!create) + continue; + pud_clear_bad(pud); } + err = apply_to_pmd_range(mm, pud, addr, next, + fn, data, create, mask); + if (err) + break; } while (pud++, addr = next, addr != end); + return err; } @@ -2502,13 +2518,21 @@ static int apply_to_p4d_range(struct mm_struct *mm, pgd_t *pgd, } do { next = p4d_addr_end(addr, end); - if (create || !p4d_none_or_clear_bad(p4d)) { - err = apply_to_pud_range(mm, p4d, addr, next, fn, data, - create, mask); - if (err) - break; + if (p4d_none(*p4d) && !create) + continue; + if (WARN_ON_ONCE(p4d_leaf(*p4d))) + return -EINVAL; + if (!p4d_none(*p4d) && WARN_ON_ONCE(p4d_bad(*p4d))) { + if (!create) + continue; + p4d_clear_bad(p4d); } + err = apply_to_pud_range(mm, p4d, addr, next, + fn, data, create, mask); + if (err) + break; } while (p4d++, addr = next, addr != end); + return err; } @@ -2528,9 +2552,17 @@ static int __apply_to_page_range(struct mm_struct *mm, unsigned long addr, pgd = pgd_offset(mm, addr); do { next = pgd_addr_end(addr, end); - if (!create && pgd_none_or_clear_bad(pgd)) + if (pgd_none(*pgd) && !create) continue; - err = apply_to_p4d_range(mm, pgd, addr, next, fn, data, create, &mask); + if (WARN_ON_ONCE(pgd_leaf(*pgd))) + return -EINVAL; + if (!pgd_none(*pgd) && WARN_ON_ONCE(pgd_bad(*pgd))) { + if (!create) + continue; + pgd_clear_bad(pgd); + } + err = apply_to_p4d_range(mm, pgd, addr, next, + fn, data, create, &mask); if (err) break; } while (pgd++, addr = next, addr != end); From patchwork Wed Mar 17 06:23:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 12144885 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7344FC433E0 for ; Wed, 17 Mar 2021 06:24:54 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 187FE64F96 for ; Wed, 17 Mar 2021 06:24:54 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 187FE64F96 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id A1A266B0078; Wed, 17 Mar 2021 02:24:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9C8B46B007B; Wed, 17 Mar 2021 02:24:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 869396B007D; Wed, 17 Mar 2021 02:24:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0139.hostedemail.com [216.40.44.139]) by kanga.kvack.org (Postfix) with ESMTP id 6E5206B0078 for ; Wed, 17 Mar 2021 02:24:53 -0400 (EDT) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 277DA181AF5F7 for ; Wed, 17 Mar 2021 06:24:53 +0000 (UTC) X-FDA: 77928378066.12.278DD7C Received: from mail-pj1-f48.google.com (mail-pj1-f48.google.com [209.85.216.48]) by imf01.hostedemail.com (Postfix) with ESMTP id B3BEF2000380 for ; Wed, 17 Mar 2021 06:24:52 +0000 (UTC) Received: by mail-pj1-f48.google.com with SMTP id k23-20020a17090a5917b02901043e35ad4aso2601732pji.3 for ; Tue, 16 Mar 2021 23:24:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=+xmdg+oguUa4M/qcJiA222ig1GYXN3tw39BVcZkinh4=; b=NysTe01Qi/lq0Ht3OBvm9MWdyBi269aHJZROee19b3GT6tTuVgso8DZWIypkSEWVei MIStA43kMPyyNIgYTc1aAkSORGUfWZTtezxniheDN0v8dDpOM8IvS8m9nqk9rbG0qIww jTnTF/fxLh97yIBfg/xeDwsbpENKUYgcTkpsGvrhM8jr4rVF458afW8p3tUQKc0JdpNr UuuKQ0vixeo2YBmZNqMk2u68kF/bCAfiV6EtiWC+nhBLpZxoWqb5Ovye0SmxH3MOs6+u T9vxKsbH9IVmUYLqKef3KrNPedjeqeqtMdF/YVhYFnGrX8efRXKdUkT4p2i0aTQAzxcs +Flw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=+xmdg+oguUa4M/qcJiA222ig1GYXN3tw39BVcZkinh4=; b=QKP7mVNwYkJR7KtuKpPO5zf9UHmMSKPSWRMfJ/g9szln44BtwBfZ092opfu41BhzcG T7zHksmEp49jGtZS69cPAfyqGbFZSPZ8fwbeT7B03kN5P7sDylGzB+LB7+osvLCLtkke Khd6cxaP9S+8i39tHuGm54Ps4aOYsBUi5wbCSWhDi1dQaVAm+F8T9ZPRRnJwro/cSGSG S37xXvYys0PYvRJu3EOCt1aFV9p/oKxpUAw5s6r59MYRxJWyvkolOGTWpNedC/pm0v7x mE8qUkyaQQpKVhNNjsxv4uKRHnKhQBQ6HdColrESEiovfBprk+7FxMdxY6K5c7WCT6AB KqVA== X-Gm-Message-State: AOAM530uuBRdyBvegBRJWcTvpKJ2SZ8G021qrDs3kcBM87JULdU/GGnN 1wT8o5kjyPewjg1UpuEwjm3AP6UcUwM= X-Google-Smtp-Source: ABdhPJwIpCK+UztE2dSFSAGTC2mdGpU/wz+pQ7ycJS9tKfG5lpPF/AfONMNNhsSTDkWz5X60+l9YJA== X-Received: by 2002:a17:90a:868c:: with SMTP id p12mr2816303pjn.82.1615962291549; Tue, 16 Mar 2021 23:24:51 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (58-6-239-121.tpgi.com.au. [58.6.239.121]) by smtp.gmail.com with ESMTPSA id s19sm17959620pfh.168.2021.03.16.23.24.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Mar 2021 23:24:51 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org, Andrew Morton Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, Jonathan Cameron , Christoph Hellwig , Christophe Leroy , Rick Edgecombe , Ding Tianhong , Miaohe Lin , Christoph Hellwig Subject: [PATCH v13 04/14] mm/vmalloc: rename vmap_*_range vmap_pages_*_range Date: Wed, 17 Mar 2021 16:23:52 +1000 Message-Id: <20210317062402.533919-5-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20210317062402.533919-1-npiggin@gmail.com> References: <20210317062402.533919-1-npiggin@gmail.com> MIME-Version: 1.0 X-Stat-Signature: qh6n4wocdanjwrqouo9y8woc5qetp5c6 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: B3BEF2000380 Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf01; identity=mailfrom; envelope-from=""; helo=mail-pj1-f48.google.com; client-ip=209.85.216.48 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1615962292-186856 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The vmalloc mapper operates on a struct page * array rather than a linear physical address, re-name it to make this distinction clear. Reviewed-by: Miaohe Lin Reviewed-by: Christoph Hellwig Signed-off-by: Nicholas Piggin --- mm/vmalloc.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 98e697ac764c..4693fab4f42a 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -189,7 +189,7 @@ void unmap_kernel_range_noflush(unsigned long start, unsigned long size) arch_sync_kernel_mappings(start, end); } -static int vmap_pte_range(pmd_t *pmd, unsigned long addr, +static int vmap_pages_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, int *nr, pgtbl_mod_mask *mask) { @@ -217,7 +217,7 @@ static int vmap_pte_range(pmd_t *pmd, unsigned long addr, return 0; } -static int vmap_pmd_range(pud_t *pud, unsigned long addr, +static int vmap_pages_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, int *nr, pgtbl_mod_mask *mask) { @@ -229,13 +229,13 @@ static int vmap_pmd_range(pud_t *pud, unsigned long addr, return -ENOMEM; do { next = pmd_addr_end(addr, end); - if (vmap_pte_range(pmd, addr, next, prot, pages, nr, mask)) + if (vmap_pages_pte_range(pmd, addr, next, prot, pages, nr, mask)) return -ENOMEM; } while (pmd++, addr = next, addr != end); return 0; } -static int vmap_pud_range(p4d_t *p4d, unsigned long addr, +static int vmap_pages_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, int *nr, pgtbl_mod_mask *mask) { @@ -247,13 +247,13 @@ static int vmap_pud_range(p4d_t *p4d, unsigned long addr, return -ENOMEM; do { next = pud_addr_end(addr, end); - if (vmap_pmd_range(pud, addr, next, prot, pages, nr, mask)) + if (vmap_pages_pmd_range(pud, addr, next, prot, pages, nr, mask)) return -ENOMEM; } while (pud++, addr = next, addr != end); return 0; } -static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, +static int vmap_pages_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, int *nr, pgtbl_mod_mask *mask) { @@ -265,7 +265,7 @@ static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, return -ENOMEM; do { next = p4d_addr_end(addr, end); - if (vmap_pud_range(p4d, addr, next, prot, pages, nr, mask)) + if (vmap_pages_pud_range(p4d, addr, next, prot, pages, nr, mask)) return -ENOMEM; } while (p4d++, addr = next, addr != end); return 0; @@ -306,7 +306,7 @@ int map_kernel_range_noflush(unsigned long addr, unsigned long size, next = pgd_addr_end(addr, end); if (pgd_bad(*pgd)) mask |= PGTBL_PGD_MODIFIED; - err = vmap_p4d_range(pgd, addr, next, prot, pages, &nr, &mask); + err = vmap_pages_p4d_range(pgd, addr, next, prot, pages, &nr, &mask); if (err) return err; } while (pgd++, addr = next, addr != end); From patchwork Wed Mar 17 06:23:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 12144887 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2DE3CC433E9 for ; Wed, 17 Mar 2021 06:25:00 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B850964F6B for ; Wed, 17 Mar 2021 06:24:59 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B850964F6B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 510666B007B; Wed, 17 Mar 2021 02:24:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4E79D6B007D; Wed, 17 Mar 2021 02:24:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 33C288D0001; Wed, 17 Mar 2021 02:24:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0130.hostedemail.com [216.40.44.130]) by kanga.kvack.org (Postfix) with ESMTP id 1AD716B007B for ; Wed, 17 Mar 2021 02:24:59 -0400 (EDT) Received: from smtpin07.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id DD4AD181AF5F7 for ; Wed, 17 Mar 2021 06:24:58 +0000 (UTC) X-FDA: 77928378276.07.9C4E192 Received: from mail-pj1-f45.google.com (mail-pj1-f45.google.com [209.85.216.45]) by imf16.hostedemail.com (Postfix) with ESMTP id 709018019140 for ; Wed, 17 Mar 2021 06:24:58 +0000 (UTC) Received: by mail-pj1-f45.google.com with SMTP id cl21-20020a17090af695b02900c61ac0f0e9so4779417pjb.1 for ; Tue, 16 Mar 2021 23:24:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=6i477Y4/Hjf81827AmPcW/5fBy/HecfjLNga9cgITPM=; b=o3Xg5+9gRBKCw3dH8UydF2AjmmqkR9F4upNpGkv3pzlzNe0SaBXYqkD4S5ejof4gJN kTR5Uw4MzB0HZRB48MUp847c6NoEeI7D4TG/P0rBm6H4kUX0jUKmLFq3QUzPFwkRe8oW mPzvv5fDjoGkLNEKP5B6CtNvwzT1Z3HqjPl891Ad6czw619BiRXpK8nd4DAsrG2LMsXM Q/vyqDJIWYdYzkpBt5Xtphb+8oYF793j3UBgyHxckvQTzPBDDBTq16r6pgQioBOe0od5 xBI0A1SavW8DrxdxeEhVmVc8/nm71Zqg5BzY5En0MVZ5RNVCKES/Ijf2/FLHQQtjlWBz 4Pwg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=6i477Y4/Hjf81827AmPcW/5fBy/HecfjLNga9cgITPM=; b=dIqESWFrk15bA8A/xFLK635MIO/Yf4dhUQm9nWP3TXDDZCSuignI9ftyzq+16yUWfN MjRzOIBtywLPXkLHR9MbZhLb0yuiLjGVc0bMA8lIF7GVyHVmweF5w6bpXCLwhvRPZ2KU YVGqstHEioqERRZ2NIVQYW5BneLH0hZft9VX6EVixh4csaG+cfQLsOcqSYmJ0tR65jQ8 5mAv4WREJjbYqm65Ba9sxRcf6LWx/ghd6WGdSKyHMxBvOJ3hsY7ZaTJ4omzzuA3IzXLT tXVVqwwVHbsBqLcGv0Hmuc+9BbE+t7o6QsjLF8D33gZXokXKizCJLHGnBx57siwB0Wki wr/w== X-Gm-Message-State: AOAM5338A30ex6GMfDc2D0Zewk6UsRDSws7AAqKbfDVUV2LPgrI0Gop6 ym3H5vtNq3QgopiVPk554HP5d1Nwldc= X-Google-Smtp-Source: ABdhPJyHqKMv5xXF3q6EMlXMMgZole3hFnV3RheN66PzOwhfuUfBqG4rgc0IBZ8yxhfySGklEfVmQA== X-Received: by 2002:a17:90a:2c4b:: with SMTP id p11mr3011472pjm.75.1615962297261; Tue, 16 Mar 2021 23:24:57 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (58-6-239-121.tpgi.com.au. [58.6.239.121]) by smtp.gmail.com with ESMTPSA id s19sm17959620pfh.168.2021.03.16.23.24.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Mar 2021 23:24:56 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org, Andrew Morton Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, Jonathan Cameron , Christoph Hellwig , Christophe Leroy , Rick Edgecombe , Ding Tianhong , Miaohe Lin , Christoph Hellwig Subject: [PATCH v13 05/14] mm/ioremap: rename ioremap_*_range to vmap_*_range Date: Wed, 17 Mar 2021 16:23:53 +1000 Message-Id: <20210317062402.533919-6-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20210317062402.533919-1-npiggin@gmail.com> References: <20210317062402.533919-1-npiggin@gmail.com> MIME-Version: 1.0 X-Stat-Signature: 5hpq18kaypcf9bsd7d4tkcpa8ypswcew X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 709018019140 Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf16; identity=mailfrom; envelope-from=""; helo=mail-pj1-f45.google.com; client-ip=209.85.216.45 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1615962298-641407 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This will be used as a generic kernel virtual mapping function, so re-name it in preparation. Reviewed-by: Miaohe Lin Reviewed-by: Christoph Hellwig Signed-off-by: Nicholas Piggin --- mm/ioremap.c | 64 +++++++++++++++++++++++++++------------------------- 1 file changed, 33 insertions(+), 31 deletions(-) diff --git a/mm/ioremap.c b/mm/ioremap.c index 5fa1ab41d152..3f4d36f9745a 100644 --- a/mm/ioremap.c +++ b/mm/ioremap.c @@ -61,9 +61,9 @@ static inline int ioremap_pud_enabled(void) { return 0; } static inline int ioremap_pmd_enabled(void) { return 0; } #endif /* CONFIG_HAVE_ARCH_HUGE_VMAP */ -static int ioremap_pte_range(pmd_t *pmd, unsigned long addr, - unsigned long end, phys_addr_t phys_addr, pgprot_t prot, - pgtbl_mod_mask *mask) +static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + pgtbl_mod_mask *mask) { pte_t *pte; u64 pfn; @@ -81,9 +81,8 @@ static int ioremap_pte_range(pmd_t *pmd, unsigned long addr, return 0; } -static int ioremap_try_huge_pmd(pmd_t *pmd, unsigned long addr, - unsigned long end, phys_addr_t phys_addr, - pgprot_t prot) +static int vmap_try_huge_pmd(pmd_t *pmd, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot) { if (!ioremap_pmd_enabled()) return 0; @@ -103,9 +102,9 @@ static int ioremap_try_huge_pmd(pmd_t *pmd, unsigned long addr, return pmd_set_huge(pmd, phys_addr, prot); } -static inline int ioremap_pmd_range(pud_t *pud, unsigned long addr, - unsigned long end, phys_addr_t phys_addr, pgprot_t prot, - pgtbl_mod_mask *mask) +static int vmap_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + pgtbl_mod_mask *mask) { pmd_t *pmd; unsigned long next; @@ -116,20 +115,19 @@ static inline int ioremap_pmd_range(pud_t *pud, unsigned long addr, do { next = pmd_addr_end(addr, end); - if (ioremap_try_huge_pmd(pmd, addr, next, phys_addr, prot)) { + if (vmap_try_huge_pmd(pmd, addr, next, phys_addr, prot)) { *mask |= PGTBL_PMD_MODIFIED; continue; } - if (ioremap_pte_range(pmd, addr, next, phys_addr, prot, mask)) + if (vmap_pte_range(pmd, addr, next, phys_addr, prot, mask)) return -ENOMEM; } while (pmd++, phys_addr += (next - addr), addr = next, addr != end); return 0; } -static int ioremap_try_huge_pud(pud_t *pud, unsigned long addr, - unsigned long end, phys_addr_t phys_addr, - pgprot_t prot) +static int vmap_try_huge_pud(pud_t *pud, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot) { if (!ioremap_pud_enabled()) return 0; @@ -149,9 +147,9 @@ static int ioremap_try_huge_pud(pud_t *pud, unsigned long addr, return pud_set_huge(pud, phys_addr, prot); } -static inline int ioremap_pud_range(p4d_t *p4d, unsigned long addr, - unsigned long end, phys_addr_t phys_addr, pgprot_t prot, - pgtbl_mod_mask *mask) +static int vmap_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + pgtbl_mod_mask *mask) { pud_t *pud; unsigned long next; @@ -162,20 +160,19 @@ static inline int ioremap_pud_range(p4d_t *p4d, unsigned long addr, do { next = pud_addr_end(addr, end); - if (ioremap_try_huge_pud(pud, addr, next, phys_addr, prot)) { + if (vmap_try_huge_pud(pud, addr, next, phys_addr, prot)) { *mask |= PGTBL_PUD_MODIFIED; continue; } - if (ioremap_pmd_range(pud, addr, next, phys_addr, prot, mask)) + if (vmap_pmd_range(pud, addr, next, phys_addr, prot, mask)) return -ENOMEM; } while (pud++, phys_addr += (next - addr), addr = next, addr != end); return 0; } -static int ioremap_try_huge_p4d(p4d_t *p4d, unsigned long addr, - unsigned long end, phys_addr_t phys_addr, - pgprot_t prot) +static int vmap_try_huge_p4d(p4d_t *p4d, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot) { if (!ioremap_p4d_enabled()) return 0; @@ -195,9 +192,9 @@ static int ioremap_try_huge_p4d(p4d_t *p4d, unsigned long addr, return p4d_set_huge(p4d, phys_addr, prot); } -static inline int ioremap_p4d_range(pgd_t *pgd, unsigned long addr, - unsigned long end, phys_addr_t phys_addr, pgprot_t prot, - pgtbl_mod_mask *mask) +static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + pgtbl_mod_mask *mask) { p4d_t *p4d; unsigned long next; @@ -208,19 +205,19 @@ static inline int ioremap_p4d_range(pgd_t *pgd, unsigned long addr, do { next = p4d_addr_end(addr, end); - if (ioremap_try_huge_p4d(p4d, addr, next, phys_addr, prot)) { + if (vmap_try_huge_p4d(p4d, addr, next, phys_addr, prot)) { *mask |= PGTBL_P4D_MODIFIED; continue; } - if (ioremap_pud_range(p4d, addr, next, phys_addr, prot, mask)) + if (vmap_pud_range(p4d, addr, next, phys_addr, prot, mask)) return -ENOMEM; } while (p4d++, phys_addr += (next - addr), addr = next, addr != end); return 0; } -int ioremap_page_range(unsigned long addr, - unsigned long end, phys_addr_t phys_addr, pgprot_t prot) +static int vmap_range(unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot) { pgd_t *pgd; unsigned long start; @@ -235,8 +232,7 @@ int ioremap_page_range(unsigned long addr, pgd = pgd_offset_k(addr); do { next = pgd_addr_end(addr, end); - err = ioremap_p4d_range(pgd, addr, next, phys_addr, prot, - &mask); + err = vmap_p4d_range(pgd, addr, next, phys_addr, prot, &mask); if (err) break; } while (pgd++, phys_addr += (next - addr), addr = next, addr != end); @@ -249,6 +245,12 @@ int ioremap_page_range(unsigned long addr, return err; } +int ioremap_page_range(unsigned long addr, + unsigned long end, phys_addr_t phys_addr, pgprot_t prot) +{ + return vmap_range(addr, end, phys_addr, prot); +} + #ifdef CONFIG_GENERIC_IOREMAP void __iomem *ioremap_prot(phys_addr_t addr, size_t size, unsigned long prot) { From patchwork Wed Mar 17 06:23:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 12144889 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CD8BBC433E0 for ; Wed, 17 Mar 2021 06:25:09 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5A0D964F8C for ; Wed, 17 Mar 2021 06:25:09 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5A0D964F8C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B2B746B007D; Wed, 17 Mar 2021 02:25:08 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B026F6B007E; Wed, 17 Mar 2021 02:25:08 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 953928D0001; Wed, 17 Mar 2021 02:25:08 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0064.hostedemail.com [216.40.44.64]) by kanga.kvack.org (Postfix) with ESMTP id 737786B007D for ; Wed, 17 Mar 2021 02:25:08 -0400 (EDT) Received: from smtpin21.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 2D1F1180AD82F for ; Wed, 17 Mar 2021 06:25:08 +0000 (UTC) X-FDA: 77928378696.21.D83CE49 Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) by imf27.hostedemail.com (Postfix) with ESMTP id A93788019148 for ; Wed, 17 Mar 2021 06:25:07 +0000 (UTC) Received: by mail-pl1-f182.google.com with SMTP id o10so265058plg.11 for ; Tue, 16 Mar 2021 23:25:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Jr4CGhz9eaDH3KnOkxuUPJTJC8uPeod4GqPDLE/A8Hg=; b=s8pVNX8ngaBvjNJlVIpAMeBCCHqrJI8GZ/m6YDrZ07wCW4I0tGUpK0VnzKMTxthPvV Q/gi3XrVguGmPeGfFsgpf7fj5g2lh9pjU1Qurs8ZdI9BSH/58yaPx6Uks8hE9a1phdAo 9sLx800ImEurdWwrqA5gTYIKagnJB87HwoBhCOfdBiJ2Mev/vI9yi8I8fOcf/tHTsqlD 22I6XIvQoHEBa8rP9/mXMvA3AEjITK7O3HhqUV+SXj3kiaGp0dw5e9j5wAC7fgaQ0Ou+ kyJZW4jTYcCsgJnYBuXaayrLAPde3tRwWo4lZuQ7IbCWT78GEKFuoAnypQeZgaZAG8gm q3Qg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Jr4CGhz9eaDH3KnOkxuUPJTJC8uPeod4GqPDLE/A8Hg=; b=acfNIJqXA0H82b0OglNPMKLmUpkX5yTQkqmMDNnlv+andCc0dYzofo42zwEPOV7otb BBECuMZ/3ke5P8NPARn3p45KE4Kxu7F1aekvpzv6oM2i6Gq1Vk0W4BASyYOnzsze9KIt O6I8+RKCLtmt3vBZIgbMegTCvV+YXdWm7spEGXimZBaqBCFABJrKlACgIUTb0BetDIqN 1Uvv3AGbAS/g8305Z2XYYkst4oykvMje1MaIHX529kYTmp/X//wZn/jKU3nm7xe0nAB3 dj/VU0Xg3Gs834PaohT+5GiPf6xelgS7rnpdHOE3O4+uspQj5SiiIwdvZSMb25YdhsHt A7mg== X-Gm-Message-State: AOAM530QgvjF7TkjKtzZGXoOt1kmZGHGv8HJQxjelFCMP18KvHws6gRi qYCY6yQ3ug+mbx9cdGxLiuSGUDo7qxo= X-Google-Smtp-Source: ABdhPJwCT4VAPxPeJykllG/8tlOSezNglaVPpaoZiYZxbcBf5l3mlF618dZ7dMr6BwWrTxOXiVNYSA== X-Received: by 2002:a17:902:ff0d:b029:e4:a497:fdad with SMTP id f13-20020a170902ff0db02900e4a497fdadmr2883390plj.33.1615962306443; Tue, 16 Mar 2021 23:25:06 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (58-6-239-121.tpgi.com.au. [58.6.239.121]) by smtp.gmail.com with ESMTPSA id s19sm17959620pfh.168.2021.03.16.23.24.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Mar 2021 23:25:05 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org, Andrew Morton Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, Jonathan Cameron , Christoph Hellwig , Christophe Leroy , Rick Edgecombe , Ding Tianhong , linuxppc-dev@lists.ozlabs.org, Catalin Marinas , Will Deacon , linux-arm-kernel@lists.infradead.org, Thomas Gleixner , Ingo Molnar , Borislav Petkov , x86@kernel.org, "H. Peter Anvin" Subject: [PATCH v13 06/14] mm: HUGE_VMAP arch support cleanup Date: Wed, 17 Mar 2021 16:23:54 +1000 Message-Id: <20210317062402.533919-7-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20210317062402.533919-1-npiggin@gmail.com> References: <20210317062402.533919-1-npiggin@gmail.com> MIME-Version: 1.0 X-Stat-Signature: 9kjgkh6bxn8d4ofykje7ztpopjqje4ua X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: A93788019148 Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf27; identity=mailfrom; envelope-from=""; helo=mail-pl1-f182.google.com; client-ip=209.85.214.182 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1615962307-626898 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This changes the awkward approach where architectures provide init functions to determine which levels they can provide large mappings for, to one where the arch is queried for each call. This removes code and indirection, and allows constant-folding of dead code for unsupported levels. This also adds a prot argument to the arch query. This is unused currently but could help with some architectures (e.g., some powerpc processors can't map uncacheable memory with large pages). Cc: linuxppc-dev@lists.ozlabs.org Cc: Catalin Marinas Cc: Will Deacon Cc: linux-arm-kernel@lists.infradead.org Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: x86@kernel.org Cc: "H. Peter Anvin" Reviewed-by: Ding Tianhong Acked-by: Catalin Marinas [arm64] Signed-off-by: Nicholas Piggin --- arch/arm64/include/asm/vmalloc.h | 8 ++ arch/arm64/mm/mmu.c | 10 +-- arch/powerpc/include/asm/vmalloc.h | 8 ++ arch/powerpc/mm/book3s64/radix_pgtable.c | 8 +- arch/x86/include/asm/vmalloc.h | 7 ++ arch/x86/mm/ioremap.c | 12 +-- include/linux/io.h | 9 --- include/linux/vmalloc.h | 6 ++ init/main.c | 1 - mm/debug_vm_pgtable.c | 4 +- mm/ioremap.c | 94 ++++++++++-------------- 11 files changed, 87 insertions(+), 80 deletions(-) diff --git a/arch/arm64/include/asm/vmalloc.h b/arch/arm64/include/asm/vmalloc.h index 2ca708ab9b20..597b40405319 100644 --- a/arch/arm64/include/asm/vmalloc.h +++ b/arch/arm64/include/asm/vmalloc.h @@ -1,4 +1,12 @@ #ifndef _ASM_ARM64_VMALLOC_H #define _ASM_ARM64_VMALLOC_H +#include + +#ifdef CONFIG_HAVE_ARCH_HUGE_VMAP +bool arch_vmap_p4d_supported(pgprot_t prot); +bool arch_vmap_pud_supported(pgprot_t prot); +bool arch_vmap_pmd_supported(pgprot_t prot); +#endif + #endif /* _ASM_ARM64_VMALLOC_H */ diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 7484ea4f6ba0..639b9de61b1d 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -1316,12 +1316,12 @@ void *__init fixmap_remap_fdt(phys_addr_t dt_phys, int *size, pgprot_t prot) return dt_virt; } -int __init arch_ioremap_p4d_supported(void) +bool arch_vmap_p4d_supported(pgprot_t prot) { - return 0; + return false; } -int __init arch_ioremap_pud_supported(void) +bool arch_vmap_pud_supported(pgprot_t prot) { /* * Only 4k granule supports level 1 block mappings. @@ -1331,9 +1331,9 @@ int __init arch_ioremap_pud_supported(void) !IS_ENABLED(CONFIG_PTDUMP_DEBUGFS); } -int __init arch_ioremap_pmd_supported(void) +bool arch_vmap_pmd_supported(pgprot_t prot) { - /* See arch_ioremap_pud_supported() */ + /* See arch_vmap_pud_supported() */ return !IS_ENABLED(CONFIG_PTDUMP_DEBUGFS); } diff --git a/arch/powerpc/include/asm/vmalloc.h b/arch/powerpc/include/asm/vmalloc.h index b992dfaaa161..105abb73f075 100644 --- a/arch/powerpc/include/asm/vmalloc.h +++ b/arch/powerpc/include/asm/vmalloc.h @@ -1,4 +1,12 @@ #ifndef _ASM_POWERPC_VMALLOC_H #define _ASM_POWERPC_VMALLOC_H +#include + +#ifdef CONFIG_HAVE_ARCH_HUGE_VMAP +bool arch_vmap_p4d_supported(pgprot_t prot); +bool arch_vmap_pud_supported(pgprot_t prot); +bool arch_vmap_pmd_supported(pgprot_t prot); +#endif + #endif /* _ASM_POWERPC_VMALLOC_H */ diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c index 98f0b243c1ab..743807fc210f 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -1082,13 +1082,13 @@ void radix__ptep_modify_prot_commit(struct vm_area_struct *vma, set_pte_at(mm, addr, ptep, pte); } -int __init arch_ioremap_pud_supported(void) +bool arch_vmap_pud_supported(pgprot_t prot) { /* HPT does not cope with large pages in the vmalloc area */ return radix_enabled(); } -int __init arch_ioremap_pmd_supported(void) +bool arch_vmap_pmd_supported(pgprot_t prot) { return radix_enabled(); } @@ -1182,7 +1182,7 @@ int pmd_free_pte_page(pmd_t *pmd, unsigned long addr) return 1; } -int __init arch_ioremap_p4d_supported(void) +bool arch_vmap_p4d_supported(pgprot_t prot) { - return 0; + return false; } diff --git a/arch/x86/include/asm/vmalloc.h b/arch/x86/include/asm/vmalloc.h index 29837740b520..094ea2b565f3 100644 --- a/arch/x86/include/asm/vmalloc.h +++ b/arch/x86/include/asm/vmalloc.h @@ -1,6 +1,13 @@ #ifndef _ASM_X86_VMALLOC_H #define _ASM_X86_VMALLOC_H +#include #include +#ifdef CONFIG_HAVE_ARCH_HUGE_VMAP +bool arch_vmap_p4d_supported(pgprot_t prot); +bool arch_vmap_pud_supported(pgprot_t prot); +bool arch_vmap_pmd_supported(pgprot_t prot); +#endif + #endif /* _ASM_X86_VMALLOC_H */ diff --git a/arch/x86/mm/ioremap.c b/arch/x86/mm/ioremap.c index 9e5ccc56f8e0..fbaf0c447986 100644 --- a/arch/x86/mm/ioremap.c +++ b/arch/x86/mm/ioremap.c @@ -481,24 +481,26 @@ void iounmap(volatile void __iomem *addr) } EXPORT_SYMBOL(iounmap); -int __init arch_ioremap_p4d_supported(void) +#ifdef CONFIG_HAVE_ARCH_HUGE_VMAP +bool arch_vmap_p4d_supported(pgprot_t prot) { - return 0; + return false; } -int __init arch_ioremap_pud_supported(void) +bool arch_vmap_pud_supported(pgprot_t prot) { #ifdef CONFIG_X86_64 return boot_cpu_has(X86_FEATURE_GBPAGES); #else - return 0; + return false; #endif } -int __init arch_ioremap_pmd_supported(void) +bool arch_vmap_pmd_supported(pgprot_t prot) { return boot_cpu_has(X86_FEATURE_PSE); } +#endif /* * Convert a physical pointer to a virtual kernel pointer for /dev/mem diff --git a/include/linux/io.h b/include/linux/io.h index 8394c56babc2..f1effd4d7a3c 100644 --- a/include/linux/io.h +++ b/include/linux/io.h @@ -31,15 +31,6 @@ static inline int ioremap_page_range(unsigned long addr, unsigned long end, } #endif -#ifdef CONFIG_HAVE_ARCH_HUGE_VMAP -void __init ioremap_huge_init(void); -int arch_ioremap_p4d_supported(void); -int arch_ioremap_pud_supported(void); -int arch_ioremap_pmd_supported(void); -#else -static inline void ioremap_huge_init(void) { } -#endif - /* * Managed iomap interface */ diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index df92211cf771..4b897a4a408b 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -78,6 +78,12 @@ struct vmap_area { }; }; +#ifndef CONFIG_HAVE_ARCH_HUGE_VMAP +static inline bool arch_vmap_p4d_supported(pgprot_t prot) { return false; } +static inline bool arch_vmap_pud_supported(pgprot_t prot) { return false; } +static inline bool arch_vmap_pmd_supported(pgprot_t prot) { return false; } +#endif + /* * Highlevel APIs for driver use */ diff --git a/init/main.c b/init/main.c index 53b278845b88..7382b80897f2 100644 --- a/init/main.c +++ b/init/main.c @@ -837,7 +837,6 @@ static void __init mm_init(void) pgtable_init(); debug_objects_mem_init(); vmalloc_init(); - ioremap_huge_init(); /* Should be run before the first non-init thread is created */ init_espfix_bsp(); /* Should be run after espfix64 is set up. */ diff --git a/mm/debug_vm_pgtable.c b/mm/debug_vm_pgtable.c index a9bd6ce1ba02..05efe98a9ac2 100644 --- a/mm/debug_vm_pgtable.c +++ b/mm/debug_vm_pgtable.c @@ -247,7 +247,7 @@ static void __init pmd_huge_tests(pmd_t *pmdp, unsigned long pfn, pgprot_t prot) { pmd_t pmd; - if (!arch_ioremap_pmd_supported()) + if (!arch_vmap_pmd_supported(prot)) return; pr_debug("Validating PMD huge\n"); @@ -385,7 +385,7 @@ static void __init pud_huge_tests(pud_t *pudp, unsigned long pfn, pgprot_t prot) { pud_t pud; - if (!arch_ioremap_pud_supported()) + if (!arch_vmap_pud_supported(prot)) return; pr_debug("Validating PUD huge\n"); diff --git a/mm/ioremap.c b/mm/ioremap.c index 3f4d36f9745a..3264d0203785 100644 --- a/mm/ioremap.c +++ b/mm/ioremap.c @@ -16,49 +16,16 @@ #include "pgalloc-track.h" #ifdef CONFIG_HAVE_ARCH_HUGE_VMAP -static int __read_mostly ioremap_p4d_capable; -static int __read_mostly ioremap_pud_capable; -static int __read_mostly ioremap_pmd_capable; -static int __read_mostly ioremap_huge_disabled; +static bool __ro_after_init iomap_max_page_shift = PAGE_SHIFT; static int __init set_nohugeiomap(char *str) { - ioremap_huge_disabled = 1; + iomap_max_page_shift = P4D_SHIFT; return 0; } early_param("nohugeiomap", set_nohugeiomap); - -void __init ioremap_huge_init(void) -{ - if (!ioremap_huge_disabled) { - if (arch_ioremap_p4d_supported()) - ioremap_p4d_capable = 1; - if (arch_ioremap_pud_supported()) - ioremap_pud_capable = 1; - if (arch_ioremap_pmd_supported()) - ioremap_pmd_capable = 1; - } -} - -static inline int ioremap_p4d_enabled(void) -{ - return ioremap_p4d_capable; -} - -static inline int ioremap_pud_enabled(void) -{ - return ioremap_pud_capable; -} - -static inline int ioremap_pmd_enabled(void) -{ - return ioremap_pmd_capable; -} - -#else /* !CONFIG_HAVE_ARCH_HUGE_VMAP */ -static inline int ioremap_p4d_enabled(void) { return 0; } -static inline int ioremap_pud_enabled(void) { return 0; } -static inline int ioremap_pmd_enabled(void) { return 0; } +#else /* CONFIG_HAVE_ARCH_HUGE_VMAP */ +static const bool iomap_max_page_shift = PAGE_SHIFT; #endif /* CONFIG_HAVE_ARCH_HUGE_VMAP */ static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, @@ -82,9 +49,13 @@ static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, } static int vmap_try_huge_pmd(pmd_t *pmd, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot) + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift) { - if (!ioremap_pmd_enabled()) + if (max_page_shift < PMD_SHIFT) + return 0; + + if (!arch_vmap_pmd_supported(prot)) return 0; if ((end - addr) != PMD_SIZE) @@ -104,7 +75,7 @@ static int vmap_try_huge_pmd(pmd_t *pmd, unsigned long addr, unsigned long end, static int vmap_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, phys_addr_t phys_addr, pgprot_t prot, - pgtbl_mod_mask *mask) + unsigned int max_page_shift, pgtbl_mod_mask *mask) { pmd_t *pmd; unsigned long next; @@ -115,7 +86,8 @@ static int vmap_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, do { next = pmd_addr_end(addr, end); - if (vmap_try_huge_pmd(pmd, addr, next, phys_addr, prot)) { + if (vmap_try_huge_pmd(pmd, addr, next, phys_addr, prot, + max_page_shift)) { *mask |= PGTBL_PMD_MODIFIED; continue; } @@ -127,9 +99,13 @@ static int vmap_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, } static int vmap_try_huge_pud(pud_t *pud, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot) + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift) { - if (!ioremap_pud_enabled()) + if (max_page_shift < PUD_SHIFT) + return 0; + + if (!arch_vmap_pud_supported(prot)) return 0; if ((end - addr) != PUD_SIZE) @@ -149,7 +125,7 @@ static int vmap_try_huge_pud(pud_t *pud, unsigned long addr, unsigned long end, static int vmap_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end, phys_addr_t phys_addr, pgprot_t prot, - pgtbl_mod_mask *mask) + unsigned int max_page_shift, pgtbl_mod_mask *mask) { pud_t *pud; unsigned long next; @@ -160,21 +136,27 @@ static int vmap_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end, do { next = pud_addr_end(addr, end); - if (vmap_try_huge_pud(pud, addr, next, phys_addr, prot)) { + if (vmap_try_huge_pud(pud, addr, next, phys_addr, prot, + max_page_shift)) { *mask |= PGTBL_PUD_MODIFIED; continue; } - if (vmap_pmd_range(pud, addr, next, phys_addr, prot, mask)) + if (vmap_pmd_range(pud, addr, next, phys_addr, prot, + max_page_shift, mask)) return -ENOMEM; } while (pud++, phys_addr += (next - addr), addr = next, addr != end); return 0; } static int vmap_try_huge_p4d(p4d_t *p4d, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot) + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift) { - if (!ioremap_p4d_enabled()) + if (max_page_shift < P4D_SHIFT) + return 0; + + if (!arch_vmap_p4d_supported(prot)) return 0; if ((end - addr) != P4D_SIZE) @@ -194,7 +176,7 @@ static int vmap_try_huge_p4d(p4d_t *p4d, unsigned long addr, unsigned long end, static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, phys_addr_t phys_addr, pgprot_t prot, - pgtbl_mod_mask *mask) + unsigned int max_page_shift, pgtbl_mod_mask *mask) { p4d_t *p4d; unsigned long next; @@ -205,19 +187,22 @@ static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, do { next = p4d_addr_end(addr, end); - if (vmap_try_huge_p4d(p4d, addr, next, phys_addr, prot)) { + if (vmap_try_huge_p4d(p4d, addr, next, phys_addr, prot, + max_page_shift)) { *mask |= PGTBL_P4D_MODIFIED; continue; } - if (vmap_pud_range(p4d, addr, next, phys_addr, prot, mask)) + if (vmap_pud_range(p4d, addr, next, phys_addr, prot, + max_page_shift, mask)) return -ENOMEM; } while (p4d++, phys_addr += (next - addr), addr = next, addr != end); return 0; } static int vmap_range(unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot) + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift) { pgd_t *pgd; unsigned long start; @@ -232,7 +217,8 @@ static int vmap_range(unsigned long addr, unsigned long end, pgd = pgd_offset_k(addr); do { next = pgd_addr_end(addr, end); - err = vmap_p4d_range(pgd, addr, next, phys_addr, prot, &mask); + err = vmap_p4d_range(pgd, addr, next, phys_addr, prot, + max_page_shift, &mask); if (err) break; } while (pgd++, phys_addr += (next - addr), addr = next, addr != end); @@ -248,7 +234,7 @@ static int vmap_range(unsigned long addr, unsigned long end, int ioremap_page_range(unsigned long addr, unsigned long end, phys_addr_t phys_addr, pgprot_t prot) { - return vmap_range(addr, end, phys_addr, prot); + return vmap_range(addr, end, phys_addr, prot, iomap_max_page_shift); } #ifdef CONFIG_GENERIC_IOREMAP From patchwork Wed Mar 17 06:23:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 12144891 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 38431C433DB for ; Wed, 17 Mar 2021 06:25:15 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D0D2164F8C for ; Wed, 17 Mar 2021 06:25:14 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D0D2164F8C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 670246B007E; Wed, 17 Mar 2021 02:25:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 646946B0080; Wed, 17 Mar 2021 02:25:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4E8386B0081; Wed, 17 Mar 2021 02:25:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0130.hostedemail.com [216.40.44.130]) by kanga.kvack.org (Postfix) with ESMTP id 303CD6B007E for ; Wed, 17 Mar 2021 02:25:14 -0400 (EDT) Received: from smtpin03.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id E80838249980 for ; Wed, 17 Mar 2021 06:25:13 +0000 (UTC) X-FDA: 77928378906.03.FFDBA34 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) by imf07.hostedemail.com (Postfix) with ESMTP id 77916A0009E3 for ; Wed, 17 Mar 2021 06:25:13 +0000 (UTC) Received: by mail-pl1-f171.google.com with SMTP id e2so269417pld.9 for ; Tue, 16 Mar 2021 23:25:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=6FRU9caWwWjvnrqszVqRsss5Upxnc33B3OfV8qLPoO8=; b=acOv1nNHq0DKxAmKf1vBi80AyxUYmYRKTh2Ey93TwUyRdkZCqWMNbWoGWU+5MekJGP bkdrfYR8csV+POZ7R2VaahbTBabix8QECkD93CNRXZRKrtJaqACa02+ligl4FVBcRJYf RjRL9pmF/JxgFVriXcGsY0lFBXYliNoOMi+qWVw74A0UwwlZXrE3SMGwlLdKOhtGwxyH 9lj/12FT4SHFaQn2nzfmOjdPxQPBiXUpk87Lrk7+lEBO/a2gWXWdccsTtyOdbLyQHQyx BvmaRP5WJYWW86mzlLmFL8NBGAmVMxsp5jpPJPAvUskm1r4Xns9NfRLajx7uwAQSirTg 9ZpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=6FRU9caWwWjvnrqszVqRsss5Upxnc33B3OfV8qLPoO8=; b=WupZWUpBBdfrVOGek2lDGmSxlFioOmVK3qTudn/a5VvqNYImMv9qHkf95r0LAhqpcE j2m5Zj6x+hwWBsZpPMSc+EfXXeDb6cfHZUoujBtNDgn47AkSWU7IB6Lrjwe54AhlOjjB YHXrBcwE7bQHGCz7HVZOdYkNRIT3Jw9vGe6P2OhlEiYuvjRndqhISiDetLk4OdbAbx1b svJ00Ubh9mftRMpBqeB35Zl5UBLgiTFYgebi/oYV+cJiaVRl3gktCTXiilA5SULdHAAe XG9P0v6NX4KqXLEScwClfwGmLPMY0Id7W4NJzHIsjC3QzX08qzP4X374FxFffckGlDAF So/w== X-Gm-Message-State: AOAM532HtpLswTzC4mog4MNQy9FrlsxSrd5J1YVDEDHWxHtxzSnUkP7A JlFhOTP/ylAh9xj+0fNCfPZdxQLoTKY= X-Google-Smtp-Source: ABdhPJwdKgWAmNPrq81aNVKAYJ1WAYd7AWSS4jrdMlA13JJ+pQXBcMpUBMbODVrBwNM0aQs0yHN98w== X-Received: by 2002:a17:90b:686:: with SMTP id m6mr2855807pjz.26.1615962312311; Tue, 16 Mar 2021 23:25:12 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (58-6-239-121.tpgi.com.au. [58.6.239.121]) by smtp.gmail.com with ESMTPSA id s19sm17959620pfh.168.2021.03.16.23.25.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Mar 2021 23:25:11 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org, Andrew Morton Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, Jonathan Cameron , Christoph Hellwig , Christophe Leroy , Rick Edgecombe , Ding Tianhong , linuxppc-dev@lists.ozlabs.org, Michael Ellerman Subject: [PATCH v13 07/14] powerpc: inline huge vmap supported functions Date: Wed, 17 Mar 2021 16:23:55 +1000 Message-Id: <20210317062402.533919-8-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20210317062402.533919-1-npiggin@gmail.com> References: <20210317062402.533919-1-npiggin@gmail.com> MIME-Version: 1.0 X-Stat-Signature: kzfbt3kwkki3m66i1gu1mhykewtxrpb8 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 77916A0009E3 Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf07; identity=mailfrom; envelope-from=""; helo=mail-pl1-f171.google.com; client-ip=209.85.214.171 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1615962313-17141 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This allows unsupported levels to be constant folded away, and so p4d_free_pud_page can be removed because it's no longer linked to. Cc: linuxppc-dev@lists.ozlabs.org Acked-by: Michael Ellerman Signed-off-by: Nicholas Piggin --- arch/powerpc/include/asm/vmalloc.h | 19 ++++++++++++++++--- arch/powerpc/mm/book3s64/radix_pgtable.c | 21 --------------------- 2 files changed, 16 insertions(+), 24 deletions(-) diff --git a/arch/powerpc/include/asm/vmalloc.h b/arch/powerpc/include/asm/vmalloc.h index 105abb73f075..3f0c153befb0 100644 --- a/arch/powerpc/include/asm/vmalloc.h +++ b/arch/powerpc/include/asm/vmalloc.h @@ -1,12 +1,25 @@ #ifndef _ASM_POWERPC_VMALLOC_H #define _ASM_POWERPC_VMALLOC_H +#include #include #ifdef CONFIG_HAVE_ARCH_HUGE_VMAP -bool arch_vmap_p4d_supported(pgprot_t prot); -bool arch_vmap_pud_supported(pgprot_t prot); -bool arch_vmap_pmd_supported(pgprot_t prot); +static inline bool arch_vmap_p4d_supported(pgprot_t prot) +{ + return false; +} + +static inline bool arch_vmap_pud_supported(pgprot_t prot) +{ + /* HPT does not cope with large pages in the vmalloc area */ + return radix_enabled(); +} + +static inline bool arch_vmap_pmd_supported(pgprot_t prot) +{ + return radix_enabled(); +} #endif #endif /* _ASM_POWERPC_VMALLOC_H */ diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c index 743807fc210f..8da62afccee5 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -1082,22 +1082,6 @@ void radix__ptep_modify_prot_commit(struct vm_area_struct *vma, set_pte_at(mm, addr, ptep, pte); } -bool arch_vmap_pud_supported(pgprot_t prot) -{ - /* HPT does not cope with large pages in the vmalloc area */ - return radix_enabled(); -} - -bool arch_vmap_pmd_supported(pgprot_t prot) -{ - return radix_enabled(); -} - -int p4d_free_pud_page(p4d_t *p4d, unsigned long addr) -{ - return 0; -} - int pud_set_huge(pud_t *pud, phys_addr_t addr, pgprot_t prot) { pte_t *ptep = (pte_t *)pud; @@ -1181,8 +1165,3 @@ int pmd_free_pte_page(pmd_t *pmd, unsigned long addr) return 1; } - -bool arch_vmap_p4d_supported(pgprot_t prot) -{ - return false; -} From patchwork Wed Mar 17 06:23:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 12144893 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AAB6BC433DB for ; Wed, 17 Mar 2021 06:25:21 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 3D03764F6B for ; Wed, 17 Mar 2021 06:25:21 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3D03764F6B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C62B86B0081; Wed, 17 Mar 2021 02:25:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C392D6B0082; Wed, 17 Mar 2021 02:25:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ADA318D0001; Wed, 17 Mar 2021 02:25:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0159.hostedemail.com [216.40.44.159]) by kanga.kvack.org (Postfix) with ESMTP id 94A866B0081 for ; Wed, 17 Mar 2021 02:25:20 -0400 (EDT) Received: from smtpin21.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 4E668815B for ; Wed, 17 Mar 2021 06:25:20 +0000 (UTC) X-FDA: 77928379200.21.A636F7F Received: from mail-pj1-f52.google.com (mail-pj1-f52.google.com [209.85.216.52]) by imf26.hostedemail.com (Postfix) with ESMTP id E58AA40001DE for ; Wed, 17 Mar 2021 06:25:19 +0000 (UTC) Received: by mail-pj1-f52.google.com with SMTP id kr3-20020a17090b4903b02900c096fc01deso766539pjb.4 for ; Tue, 16 Mar 2021 23:25:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=dj69ry3EYtQyAiX5oUHvY3dqq56j/eubUdI+6FA7GRk=; b=QXilW1y5Pw7owXm0BE0y5uqNYBU68jc9dy9LWT4Q4tnq+6acQ1CvSma676a1voPjVH /sChBOfCbIiES09oErIAYuIT1mjp1kTIdxSBNw+b4hbER9qN1QM3x3Dzuv4QutDlsxce zQmzk0jarXecDdnLDWz2Vpjng98h36IzQGxfzucAjGa5gpaRpWVQ6cKp+LBSwtVgMpN8 O9MSzlDfpIO4cGhMJQh0KgaT4bhTIpckhgdzgEh8eLjcmuzcSke0yvSo3Mscfpmm+dJ6 Sc34jJ+r9GUXSsCPtG6KuV7CoSuL2LOyFVnmqE4DgNTdNJ0QgVg3sv15OxNUPwyMqrRW rPSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=dj69ry3EYtQyAiX5oUHvY3dqq56j/eubUdI+6FA7GRk=; b=hQ7GuSYYwc7fddKnRUhI1B32/ppn8/Bo1ZBVfXA/KLjYWmZS13Uo/aHjHvNbLqk0n2 emjSde3Gx8hWohihQC7usFIFGy03R5lcgrKLIyEYGmR2kwBMd/tiF+xQbPXmD2eBbguD immbm7qximQq5Muv1s3sj3FbnQucZgKIe6CoZbQPBDOou5+vsrbScR4cKGTI5Emrdpok re6tc+rVfMm/XtER1tPoHzLMekQ3fpzzyfKkTVdviqGLHNIogNdoAo59kD6nfzjyN/PQ McTUztckanBhgS2r7nQFhCYFf6Rt0bmc7DHlGtF3B2Z3p9Fl3D/HlkYP9YzfCtjbaj/I ztMg== X-Gm-Message-State: AOAM533HSrrKdv1W78YEvm1VBTdUVKzH7JMD7FnbaJ0lE5xYlqtBYQYq HeZJigqSOhz+J+uOfQafGFkrEg7wOgk= X-Google-Smtp-Source: ABdhPJzKKmiML9ahyhDjVcYprqOolbC+HJd06JEicV3PL6AmH054w47Hun27ybXyvgmde5bmGpvF4g== X-Received: by 2002:a17:90a:1b08:: with SMTP id q8mr2789442pjq.203.1615962318728; Tue, 16 Mar 2021 23:25:18 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (58-6-239-121.tpgi.com.au. [58.6.239.121]) by smtp.gmail.com with ESMTPSA id s19sm17959620pfh.168.2021.03.16.23.25.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Mar 2021 23:25:18 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org, Andrew Morton Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, Jonathan Cameron , Christoph Hellwig , Christophe Leroy , Rick Edgecombe , Ding Tianhong , Catalin Marinas , Will Deacon , linux-arm-kernel@lists.infradead.org Subject: [PATCH v13 08/14] arm64: inline huge vmap supported functions Date: Wed, 17 Mar 2021 16:23:56 +1000 Message-Id: <20210317062402.533919-9-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20210317062402.533919-1-npiggin@gmail.com> References: <20210317062402.533919-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: E58AA40001DE X-Stat-Signature: oxmwqg4i3zjjzw4sn4biqzjesihxkbop Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf26; identity=mailfrom; envelope-from=""; helo=mail-pj1-f52.google.com; client-ip=209.85.216.52 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1615962319-458177 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This allows unsupported levels to be constant folded away, and so p4d_free_pud_page can be removed because it's no longer linked to. Cc: Catalin Marinas Cc: Will Deacon Cc: linux-arm-kernel@lists.infradead.org Acked-by: Catalin Marinas Signed-off-by: Nicholas Piggin --- arch/arm64/include/asm/vmalloc.h | 23 ++++++++++++++++++++--- arch/arm64/mm/mmu.c | 26 -------------------------- 2 files changed, 20 insertions(+), 29 deletions(-) diff --git a/arch/arm64/include/asm/vmalloc.h b/arch/arm64/include/asm/vmalloc.h index 597b40405319..fc9a12d6cc1a 100644 --- a/arch/arm64/include/asm/vmalloc.h +++ b/arch/arm64/include/asm/vmalloc.h @@ -4,9 +4,26 @@ #include #ifdef CONFIG_HAVE_ARCH_HUGE_VMAP -bool arch_vmap_p4d_supported(pgprot_t prot); -bool arch_vmap_pud_supported(pgprot_t prot); -bool arch_vmap_pmd_supported(pgprot_t prot); +static inline bool arch_vmap_p4d_supported(pgprot_t prot) +{ + return false; +} + +static inline bool arch_vmap_pud_supported(pgprot_t prot) +{ + /* + * Only 4k granule supports level 1 block mappings. + * SW table walks can't handle removal of intermediate entries. + */ + return IS_ENABLED(CONFIG_ARM64_4K_PAGES) && + !IS_ENABLED(CONFIG_PTDUMP_DEBUGFS); +} + +static inline bool arch_vmap_pmd_supported(pgprot_t prot) +{ + /* See arch_vmap_pud_supported() */ + return !IS_ENABLED(CONFIG_PTDUMP_DEBUGFS); +} #endif #endif /* _ASM_ARM64_VMALLOC_H */ diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 639b9de61b1d..1fb0035b0777 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -1316,27 +1316,6 @@ void *__init fixmap_remap_fdt(phys_addr_t dt_phys, int *size, pgprot_t prot) return dt_virt; } -bool arch_vmap_p4d_supported(pgprot_t prot) -{ - return false; -} - -bool arch_vmap_pud_supported(pgprot_t prot) -{ - /* - * Only 4k granule supports level 1 block mappings. - * SW table walks can't handle removal of intermediate entries. - */ - return IS_ENABLED(CONFIG_ARM64_4K_PAGES) && - !IS_ENABLED(CONFIG_PTDUMP_DEBUGFS); -} - -bool arch_vmap_pmd_supported(pgprot_t prot) -{ - /* See arch_vmap_pud_supported() */ - return !IS_ENABLED(CONFIG_PTDUMP_DEBUGFS); -} - int pud_set_huge(pud_t *pudp, phys_addr_t phys, pgprot_t prot) { pud_t new_pud = pfn_pud(__phys_to_pfn(phys), mk_pud_sect_prot(prot)); @@ -1428,11 +1407,6 @@ int pud_free_pmd_page(pud_t *pudp, unsigned long addr) return 1; } -int p4d_free_pud_page(p4d_t *p4d, unsigned long addr) -{ - return 0; /* Don't attempt a block mapping */ -} - #ifdef CONFIG_MEMORY_HOTPLUG static void __remove_pgd_mapping(pgd_t *pgdir, unsigned long start, u64 size) { From patchwork Wed Mar 17 06:23:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 12144895 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9276DC433E0 for ; Wed, 17 Mar 2021 06:25:29 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 02FC264F6B for ; Wed, 17 Mar 2021 06:25:28 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 02FC264F6B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 891C56B0071; Wed, 17 Mar 2021 02:25:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 868F38D0001; Wed, 17 Mar 2021 02:25:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6E3146B0083; Wed, 17 Mar 2021 02:25:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0121.hostedemail.com [216.40.44.121]) by kanga.kvack.org (Postfix) with ESMTP id 553046B0071 for ; Wed, 17 Mar 2021 02:25:28 -0400 (EDT) Received: from smtpin27.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 1973583FE for ; Wed, 17 Mar 2021 06:25:28 +0000 (UTC) X-FDA: 77928379536.27.8B4615D Received: from mail-pf1-f180.google.com (mail-pf1-f180.google.com [209.85.210.180]) by imf22.hostedemail.com (Postfix) with ESMTP id 9DC57C0001F7 for ; Wed, 17 Mar 2021 06:25:27 +0000 (UTC) Received: by mail-pf1-f180.google.com with SMTP id y5so408025pfn.1 for ; Tue, 16 Mar 2021 23:25:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=80Po72abxrqXJsVTjT2d7/68A0xoEHKX9fHFXJoIAvk=; b=JAmTOolWiVpo5iFh4Edp0oj5LSlEJYBtZtvpKoHs/S1GD0gx1eFqaVhYl9JEdsGNFS 739yXAV75PNjMjic5dq0IbTB2j4RWAD4v8Rnvmt4FxbkyOUukGOOFAjLoCiC2wzq05i4 5vQYHXbymK3fjEmZ7jcCrSauWdLhHskrIUX7jeC/PtYXGMOKQvWFNjr+00E1NV2SWvsK 4dpMWa9gDInoiDmvAl/1J80lsb0SPsTkrURLvuxpW7ocdCk4xpEMe4K5CWFvnCVJltKE /la9r99LSLY3b5B+bCQB2SYD8HjjPoGZLKZN9NUHaQ5VrkXLzgB5idoI8OPWE16CwrF7 SQuQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=80Po72abxrqXJsVTjT2d7/68A0xoEHKX9fHFXJoIAvk=; b=HI6rkvBMGh+udYtdUtSm8Oo7MK4CX70yX1zwcsDFffb6BDdfFZZb4+cBvSGSNiy3Gh EL/EfqjsqQliri4+gFJsWX/ps2C0PiWpFtLMS7bQa4oCm3H8Krf/Q47bi6/pYHbESdvF ZDyQwVTK71Q4h+rdb0ACnIlYvIXf1rIC9gLJOmOlqP9Nl21CaLBXq/5BTIDJa6tuXFIM v8vS1S6hG9Z6ufsS/g1mO+OY6LQ5NwVEfaBnvK8mSLDECa1azpEBKNV3NA47jq10WZs9 RKTJv66gLz82IRxRV//wffrpZiv/k40yfR8rMaMj7Ng4KlyYDUhu01DjDXs3RbETCWyg 1lpg== X-Gm-Message-State: AOAM531uDc4sW+lBYUu6iDGVL5FR12tCR/d/+uLlphWskAj/9DBSHYI8 IdtG5wW+K34uNm2oJGj4DbxJHRMKZdM= X-Google-Smtp-Source: ABdhPJwlFMT1KRAdDYK4lyGgozRTPX92bcYxejTdYdTXfuOCsRuG3aynQ5KZ8r0Agwq3wG940xYyag== X-Received: by 2002:a63:e5e:: with SMTP id 30mr1364996pgo.181.1615962326152; Tue, 16 Mar 2021 23:25:26 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (58-6-239-121.tpgi.com.au. [58.6.239.121]) by smtp.gmail.com with ESMTPSA id s19sm17959620pfh.168.2021.03.16.23.25.19 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Mar 2021 23:25:25 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org, Andrew Morton Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, Jonathan Cameron , Christoph Hellwig , Christophe Leroy , Rick Edgecombe , Ding Tianhong , Thomas Gleixner , Ingo Molnar , Borislav Petkov , x86@kernel.org, "H. Peter Anvin" Subject: [PATCH v13 09/14] x86: inline huge vmap supported functions Date: Wed, 17 Mar 2021 16:23:57 +1000 Message-Id: <20210317062402.533919-10-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20210317062402.533919-1-npiggin@gmail.com> References: <20210317062402.533919-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 9DC57C0001F7 X-Stat-Signature: ku1ej6miioq1qjb5brdxtqoqh3gqdypx Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf22; identity=mailfrom; envelope-from=""; helo=mail-pf1-f180.google.com; client-ip=209.85.210.180 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1615962327-424526 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This allows unsupported levels to be constant folded away, and so p4d_free_pud_page can be removed because it's no longer linked to. Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: x86@kernel.org Cc: "H. Peter Anvin" Signed-off-by: Nicholas Piggin --- arch/x86/include/asm/vmalloc.h | 22 +++++++++++++++++++--- arch/x86/mm/ioremap.c | 21 --------------------- arch/x86/mm/pgtable.c | 13 ------------- 3 files changed, 19 insertions(+), 37 deletions(-) diff --git a/arch/x86/include/asm/vmalloc.h b/arch/x86/include/asm/vmalloc.h index 094ea2b565f3..e714b00fc0ca 100644 --- a/arch/x86/include/asm/vmalloc.h +++ b/arch/x86/include/asm/vmalloc.h @@ -1,13 +1,29 @@ #ifndef _ASM_X86_VMALLOC_H #define _ASM_X86_VMALLOC_H +#include #include #include #ifdef CONFIG_HAVE_ARCH_HUGE_VMAP -bool arch_vmap_p4d_supported(pgprot_t prot); -bool arch_vmap_pud_supported(pgprot_t prot); -bool arch_vmap_pmd_supported(pgprot_t prot); +static inline bool arch_vmap_p4d_supported(pgprot_t prot) +{ + return false; +} + +static inline bool arch_vmap_pud_supported(pgprot_t prot) +{ +#ifdef CONFIG_X86_64 + return boot_cpu_has(X86_FEATURE_GBPAGES); +#else + return false; +#endif +} + +static inline bool arch_vmap_pmd_supported(pgprot_t prot) +{ + return boot_cpu_has(X86_FEATURE_PSE); +} #endif #endif /* _ASM_X86_VMALLOC_H */ diff --git a/arch/x86/mm/ioremap.c b/arch/x86/mm/ioremap.c index fbaf0c447986..12c686c65ea9 100644 --- a/arch/x86/mm/ioremap.c +++ b/arch/x86/mm/ioremap.c @@ -481,27 +481,6 @@ void iounmap(volatile void __iomem *addr) } EXPORT_SYMBOL(iounmap); -#ifdef CONFIG_HAVE_ARCH_HUGE_VMAP -bool arch_vmap_p4d_supported(pgprot_t prot) -{ - return false; -} - -bool arch_vmap_pud_supported(pgprot_t prot) -{ -#ifdef CONFIG_X86_64 - return boot_cpu_has(X86_FEATURE_GBPAGES); -#else - return false; -#endif -} - -bool arch_vmap_pmd_supported(pgprot_t prot) -{ - return boot_cpu_has(X86_FEATURE_PSE); -} -#endif - /* * Convert a physical pointer to a virtual kernel pointer for /dev/mem * access diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index f6a9e2e36642..d27cf69e811d 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -780,14 +780,6 @@ int pmd_clear_huge(pmd_t *pmd) return 0; } -/* - * Until we support 512GB pages, skip them in the vmap area. - */ -int p4d_free_pud_page(p4d_t *p4d, unsigned long addr) -{ - return 0; -} - #ifdef CONFIG_X86_64 /** * pud_free_pmd_page - Clear pud entry and free pmd page. @@ -861,11 +853,6 @@ int pmd_free_pte_page(pmd_t *pmd, unsigned long addr) #else /* !CONFIG_X86_64 */ -int pud_free_pmd_page(pud_t *pud, unsigned long addr) -{ - return pud_none(*pud); -} - /* * Disable free page handling on x86-PAE. This assures that ioremap() * does not update sync'd pmd entries. See vmalloc_sync_one(). From patchwork Wed Mar 17 06:23:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 12144897 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D0E43C433E0 for ; Wed, 17 Mar 2021 06:25:34 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5EBD864FA8 for ; Wed, 17 Mar 2021 06:25:34 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5EBD864FA8 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id F20DC6B0073; Wed, 17 Mar 2021 02:25:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EF7636B0082; Wed, 17 Mar 2021 02:25:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D72226B0083; Wed, 17 Mar 2021 02:25:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0103.hostedemail.com [216.40.44.103]) by kanga.kvack.org (Postfix) with ESMTP id BE51C6B0073 for ; Wed, 17 Mar 2021 02:25:33 -0400 (EDT) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 7D4A1815B for ; Wed, 17 Mar 2021 06:25:33 +0000 (UTC) X-FDA: 77928379746.01.E68BCC0 Received: from mail-pj1-f49.google.com (mail-pj1-f49.google.com [209.85.216.49]) by imf07.hostedemail.com (Postfix) with ESMTP id 169BDA0009DA for ; Wed, 17 Mar 2021 06:25:32 +0000 (UTC) Received: by mail-pj1-f49.google.com with SMTP id bt4so528199pjb.5 for ; Tue, 16 Mar 2021 23:25:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=oZYPzoCQS87jB7WmWdNUk0Mp+gSQWT5jT3H4y8xrQ2c=; b=TemPJTyQirvubOQVp3MTVqgduToCHRC+WNtWDBd6YrW98l1Pd+GeF5AvIKJwKAnEFa yoWOuLxS2MA+6CX8FJxOiK2ys/TT5n4G6fkmV7xRVbBVqBss3vGasubJo06SqMRyMcEC rUAxNhjzINp1/H7KVCEMcIAtX29iO4FV0nKYYbflWw+F34Vnxre2juX4Xpw0AX/kfJQp SEBRqHX0rVUPuOmzT35UM+FwVicG7cWE0RYYldxgae5QjWiIMZeQCqQyxtkA1izVvi5s BYCSCPYJwBDLG3YCkSIY9ln+PCe3pj3SchbvdduAJ/OIKyStEpLtwlTuzqiZ4jIgo5Y2 3sQg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=oZYPzoCQS87jB7WmWdNUk0Mp+gSQWT5jT3H4y8xrQ2c=; b=SqyJr7rzoSEsJy/VxNroU+eSrhBhnlwPbqn8+k8r9+7tFc0MdE1D40VIDgmw+PY45O QSzRW0b03XdL2L39OeGHGBJCIbPW25c3hh27WVLDBHjtBeuu0DltlILNBhU6xrt7u/0c em9bZMDWlpBT3lsj+Mcbj3ywOlWdkgUU7UEeas2H7ONMVQDelE87QU+zhcRO1oNJonAg DSeWG4QMMNMyrAzMjOC6QVb2lPFDd8Iw85pAupbO6pyCmEtCwAbROhWIFzxqdyrzhqQF YUveDXlwAbDYr3YuLwUBFaNuvfB1wXnBv3hVY+L3VUTYpg7/8UTph+eMdUBUABvCC0JB HE1g== X-Gm-Message-State: AOAM531Dr2CVOFVYid1A1SYbuJZw6r8VWHCPEwclUH5oRnuY1xflnIgN qYENZz2b8z2+V8d6SPFrZCoPu9wMJ0o= X-Google-Smtp-Source: ABdhPJxtEqE1tM+WOJOAPCcFt+4coTsAHZiBsWq0FVUjYiDsAWrpBV0tiygByrlVKECbzjDzhh0yOg== X-Received: by 2002:a17:90a:a898:: with SMTP id h24mr2906137pjq.9.1615962331915; Tue, 16 Mar 2021 23:25:31 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (58-6-239-121.tpgi.com.au. [58.6.239.121]) by smtp.gmail.com with ESMTPSA id s19sm17959620pfh.168.2021.03.16.23.25.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Mar 2021 23:25:31 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org, Andrew Morton Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, Jonathan Cameron , Christoph Hellwig , Christophe Leroy , Rick Edgecombe , Ding Tianhong , Christoph Hellwig Subject: [PATCH v13 10/14] mm/vmalloc: provide fallback arch huge vmap support functions Date: Wed, 17 Mar 2021 16:23:58 +1000 Message-Id: <20210317062402.533919-11-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20210317062402.533919-1-npiggin@gmail.com> References: <20210317062402.533919-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 169BDA0009DA X-Stat-Signature: ftyr37ijjbtiahsxufomdp13wmkoscyd Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf07; identity=mailfrom; envelope-from=""; helo=mail-pj1-f49.google.com; client-ip=209.85.216.49 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1615962332-378804 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: If an architecture doesn't support a particular page table level as a huge vmap page size then allow it to skip defining the support query function. Suggested-by: Christoph Hellwig Signed-off-by: Nicholas Piggin --- arch/arm64/include/asm/vmalloc.h | 7 +++---- arch/powerpc/include/asm/vmalloc.h | 7 +++---- arch/x86/include/asm/vmalloc.h | 13 +++++-------- include/linux/vmalloc.h | 24 ++++++++++++++++++++---- 4 files changed, 31 insertions(+), 20 deletions(-) diff --git a/arch/arm64/include/asm/vmalloc.h b/arch/arm64/include/asm/vmalloc.h index fc9a12d6cc1a..7a22aeea9bb5 100644 --- a/arch/arm64/include/asm/vmalloc.h +++ b/arch/arm64/include/asm/vmalloc.h @@ -4,11 +4,8 @@ #include #ifdef CONFIG_HAVE_ARCH_HUGE_VMAP -static inline bool arch_vmap_p4d_supported(pgprot_t prot) -{ - return false; -} +#define arch_vmap_pud_supported arch_vmap_pud_supported static inline bool arch_vmap_pud_supported(pgprot_t prot) { /* @@ -19,11 +16,13 @@ static inline bool arch_vmap_pud_supported(pgprot_t prot) !IS_ENABLED(CONFIG_PTDUMP_DEBUGFS); } +#define arch_vmap_pmd_supported arch_vmap_pmd_supported static inline bool arch_vmap_pmd_supported(pgprot_t prot) { /* See arch_vmap_pud_supported() */ return !IS_ENABLED(CONFIG_PTDUMP_DEBUGFS); } + #endif #endif /* _ASM_ARM64_VMALLOC_H */ diff --git a/arch/powerpc/include/asm/vmalloc.h b/arch/powerpc/include/asm/vmalloc.h index 3f0c153befb0..4c69ece52a31 100644 --- a/arch/powerpc/include/asm/vmalloc.h +++ b/arch/powerpc/include/asm/vmalloc.h @@ -5,21 +5,20 @@ #include #ifdef CONFIG_HAVE_ARCH_HUGE_VMAP -static inline bool arch_vmap_p4d_supported(pgprot_t prot) -{ - return false; -} +#define arch_vmap_pud_supported arch_vmap_pud_supported static inline bool arch_vmap_pud_supported(pgprot_t prot) { /* HPT does not cope with large pages in the vmalloc area */ return radix_enabled(); } +#define arch_vmap_pmd_supported arch_vmap_pmd_supported static inline bool arch_vmap_pmd_supported(pgprot_t prot) { return radix_enabled(); } + #endif #endif /* _ASM_POWERPC_VMALLOC_H */ diff --git a/arch/x86/include/asm/vmalloc.h b/arch/x86/include/asm/vmalloc.h index e714b00fc0ca..49ce331f3ac6 100644 --- a/arch/x86/include/asm/vmalloc.h +++ b/arch/x86/include/asm/vmalloc.h @@ -6,24 +6,21 @@ #include #ifdef CONFIG_HAVE_ARCH_HUGE_VMAP -static inline bool arch_vmap_p4d_supported(pgprot_t prot) -{ - return false; -} +#ifdef CONFIG_X86_64 +#define arch_vmap_pud_supported arch_vmap_pud_supported static inline bool arch_vmap_pud_supported(pgprot_t prot) { -#ifdef CONFIG_X86_64 return boot_cpu_has(X86_FEATURE_GBPAGES); -#else - return false; -#endif } +#endif +#define arch_vmap_pmd_supported arch_vmap_pmd_supported static inline bool arch_vmap_pmd_supported(pgprot_t prot) { return boot_cpu_has(X86_FEATURE_PSE); } + #endif #endif /* _ASM_X86_VMALLOC_H */ diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 4b897a4a408b..82b45e1f28ff 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -78,10 +78,26 @@ struct vmap_area { }; }; -#ifndef CONFIG_HAVE_ARCH_HUGE_VMAP -static inline bool arch_vmap_p4d_supported(pgprot_t prot) { return false; } -static inline bool arch_vmap_pud_supported(pgprot_t prot) { return false; } -static inline bool arch_vmap_pmd_supported(pgprot_t prot) { return false; } +/* archs that select HAVE_ARCH_HUGE_VMAP should override one or more of these */ +#ifndef arch_vmap_p4d_supported +static inline bool arch_vmap_p4d_supported(pgprot_t prot) +{ + return false; +} +#endif + +#ifndef arch_vmap_pud_supported +static inline bool arch_vmap_pud_supported(pgprot_t prot) +{ + return false; +} +#endif + +#ifndef arch_vmap_pmd_supported +static inline bool arch_vmap_pmd_supported(pgprot_t prot) +{ + return false; +} #endif /* From patchwork Wed Mar 17 06:23:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 12144899 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A6FDEC433E6 for ; Wed, 17 Mar 2021 06:25:41 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 3123764F8C for ; Wed, 17 Mar 2021 06:25:41 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3123764F8C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B9EEA6B0082; Wed, 17 Mar 2021 02:25:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B76036B0083; Wed, 17 Mar 2021 02:25:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9F05C6B0085; Wed, 17 Mar 2021 02:25:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0118.hostedemail.com [216.40.44.118]) by kanga.kvack.org (Postfix) with ESMTP id 850836B0082 for ; Wed, 17 Mar 2021 02:25:40 -0400 (EDT) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 477FE8149 for ; Wed, 17 Mar 2021 06:25:40 +0000 (UTC) X-FDA: 77928380040.02.E43A7CC Received: from mail-pj1-f50.google.com (mail-pj1-f50.google.com [209.85.216.50]) by imf26.hostedemail.com (Postfix) with ESMTP id BAFBE407F8EB for ; Wed, 17 Mar 2021 06:25:39 +0000 (UTC) Received: by mail-pj1-f50.google.com with SMTP id f2-20020a17090a4a82b02900c67bf8dc69so2659983pjh.1 for ; Tue, 16 Mar 2021 23:25:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=1Im/28hj7nXiVGvdYZWfdzJFJ0zjBM0pOjqUFFEb2DU=; b=vIwtqOp7lrhacyh/LDndRe/VXp9/J5WmVx9Q9GjRFHRV7ducB18RKrpq+MWH3/gwSF wlVtX17wZDzzF+XU2CkPVzzP7ttJeQnJxidb8jXwEgGsYFcQ8EMhBjDo3/CQp7v8ukfn Z3yvhddmD9gg0NXZez8tknVp5wuuL4MX+rArV9NbzwDVT+EObSQtMdKjIX2vuEGPeZrZ EWSQq5eCBwcPRKTrkYoOPGtGEtVm2+F6sDit7+sQW1QRuR0MGYOI0DQvrSrYOxUmn8I4 ZZN0PtRdPGy0ANcITuR2iDW6e5E0BeNJ+bfcWtPzf05eoPRWRWay1ZrkPoiVnJh9Z0a9 GR4g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=1Im/28hj7nXiVGvdYZWfdzJFJ0zjBM0pOjqUFFEb2DU=; b=ifgMJlo1QGzbl5cx7vFonbtiDk+aqHaY4dctlhiG1ZcnL8872/CpwenhjB/0aFvKhx 8zO+znhvcR33sWvY+/jy+pXAZ/DV7GeDJY/s7t7jHk1+kmZSiUQ6WIJuB3ez7ReV11TY 56xA4nrvz9n47PUqZw4mIiban7T5m5jrIoNgqRs+eCdk4d/stcMTIuoYXEWINOoE1C8S axfCo1k8ekBD4HSSlXJ7g9EZXg+bIbeU9r8ulT91evOdP5hYK7t9af38rWDAwsfhr0dQ sW00zNDHF34zP7GjkGG+4XScylnbgguoHga4XQQmhsN/WtHl+kz1NoZ2pKPIsaWjR6D4 Qf1A== X-Gm-Message-State: AOAM531pmItSTIZ16Eh1lksUFkx2a1rDwRAfjEptKA2Crg2D9HLpCH7E eLkHAg4cVfOFl3a5rMWxEL8LGNFYNhE= X-Google-Smtp-Source: ABdhPJygAEJKjxHZNCMNp5g8bHzfkjn1ID1kuyIYJCAS4jWMEsVVKenZGBZniTutej0CqhnHw9In+Q== X-Received: by 2002:a17:902:b908:b029:e6:3e0a:b3cc with SMTP id bf8-20020a170902b908b02900e63e0ab3ccmr3099643plb.68.1615962338410; Tue, 16 Mar 2021 23:25:38 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (58-6-239-121.tpgi.com.au. [58.6.239.121]) by smtp.gmail.com with ESMTPSA id s19sm17959620pfh.168.2021.03.16.23.25.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Mar 2021 23:25:37 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org, Andrew Morton Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, Jonathan Cameron , Christoph Hellwig , Christophe Leroy , Rick Edgecombe , Ding Tianhong , Christoph Hellwig Subject: [PATCH v13 11/14] mm: Move vmap_range from mm/ioremap.c to mm/vmalloc.c Date: Wed, 17 Mar 2021 16:23:59 +1000 Message-Id: <20210317062402.533919-12-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20210317062402.533919-1-npiggin@gmail.com> References: <20210317062402.533919-1-npiggin@gmail.com> MIME-Version: 1.0 X-Stat-Signature: u17um8oyt1cbj7s3ytuhw9pkt458cpz1 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: BAFBE407F8EB Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf26; identity=mailfrom; envelope-from=""; helo=mail-pj1-f50.google.com; client-ip=209.85.216.50 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1615962339-355582 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is a generic kernel virtual memory mapper, not specific to ioremap. Code is unchanged other than making vmap_range non-static. Reviewed-by: Christoph Hellwig Signed-off-by: Nicholas Piggin --- include/linux/vmalloc.h | 3 + mm/ioremap.c | 203 ---------------------------------------- mm/vmalloc.c | 202 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 205 insertions(+), 203 deletions(-) diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 82b45e1f28ff..1f6844e2670a 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -189,6 +189,9 @@ extern struct vm_struct *remove_vm_area(const void *addr); extern struct vm_struct *find_vm_area(const void *addr); #ifdef CONFIG_MMU +int vmap_range(unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift); extern int map_kernel_range_noflush(unsigned long start, unsigned long size, pgprot_t prot, struct page **pages); int map_kernel_range(unsigned long start, unsigned long size, pgprot_t prot, diff --git a/mm/ioremap.c b/mm/ioremap.c index 3264d0203785..d1dcc7e744ac 100644 --- a/mm/ioremap.c +++ b/mm/ioremap.c @@ -28,209 +28,6 @@ early_param("nohugeiomap", set_nohugeiomap); static const bool iomap_max_page_shift = PAGE_SHIFT; #endif /* CONFIG_HAVE_ARCH_HUGE_VMAP */ -static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, - pgtbl_mod_mask *mask) -{ - pte_t *pte; - u64 pfn; - - pfn = phys_addr >> PAGE_SHIFT; - pte = pte_alloc_kernel_track(pmd, addr, mask); - if (!pte) - return -ENOMEM; - do { - BUG_ON(!pte_none(*pte)); - set_pte_at(&init_mm, addr, pte, pfn_pte(pfn, prot)); - pfn++; - } while (pte++, addr += PAGE_SIZE, addr != end); - *mask |= PGTBL_PTE_MODIFIED; - return 0; -} - -static int vmap_try_huge_pmd(pmd_t *pmd, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, - unsigned int max_page_shift) -{ - if (max_page_shift < PMD_SHIFT) - return 0; - - if (!arch_vmap_pmd_supported(prot)) - return 0; - - if ((end - addr) != PMD_SIZE) - return 0; - - if (!IS_ALIGNED(addr, PMD_SIZE)) - return 0; - - if (!IS_ALIGNED(phys_addr, PMD_SIZE)) - return 0; - - if (pmd_present(*pmd) && !pmd_free_pte_page(pmd, addr)) - return 0; - - return pmd_set_huge(pmd, phys_addr, prot); -} - -static int vmap_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, - unsigned int max_page_shift, pgtbl_mod_mask *mask) -{ - pmd_t *pmd; - unsigned long next; - - pmd = pmd_alloc_track(&init_mm, pud, addr, mask); - if (!pmd) - return -ENOMEM; - do { - next = pmd_addr_end(addr, end); - - if (vmap_try_huge_pmd(pmd, addr, next, phys_addr, prot, - max_page_shift)) { - *mask |= PGTBL_PMD_MODIFIED; - continue; - } - - if (vmap_pte_range(pmd, addr, next, phys_addr, prot, mask)) - return -ENOMEM; - } while (pmd++, phys_addr += (next - addr), addr = next, addr != end); - return 0; -} - -static int vmap_try_huge_pud(pud_t *pud, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, - unsigned int max_page_shift) -{ - if (max_page_shift < PUD_SHIFT) - return 0; - - if (!arch_vmap_pud_supported(prot)) - return 0; - - if ((end - addr) != PUD_SIZE) - return 0; - - if (!IS_ALIGNED(addr, PUD_SIZE)) - return 0; - - if (!IS_ALIGNED(phys_addr, PUD_SIZE)) - return 0; - - if (pud_present(*pud) && !pud_free_pmd_page(pud, addr)) - return 0; - - return pud_set_huge(pud, phys_addr, prot); -} - -static int vmap_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, - unsigned int max_page_shift, pgtbl_mod_mask *mask) -{ - pud_t *pud; - unsigned long next; - - pud = pud_alloc_track(&init_mm, p4d, addr, mask); - if (!pud) - return -ENOMEM; - do { - next = pud_addr_end(addr, end); - - if (vmap_try_huge_pud(pud, addr, next, phys_addr, prot, - max_page_shift)) { - *mask |= PGTBL_PUD_MODIFIED; - continue; - } - - if (vmap_pmd_range(pud, addr, next, phys_addr, prot, - max_page_shift, mask)) - return -ENOMEM; - } while (pud++, phys_addr += (next - addr), addr = next, addr != end); - return 0; -} - -static int vmap_try_huge_p4d(p4d_t *p4d, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, - unsigned int max_page_shift) -{ - if (max_page_shift < P4D_SHIFT) - return 0; - - if (!arch_vmap_p4d_supported(prot)) - return 0; - - if ((end - addr) != P4D_SIZE) - return 0; - - if (!IS_ALIGNED(addr, P4D_SIZE)) - return 0; - - if (!IS_ALIGNED(phys_addr, P4D_SIZE)) - return 0; - - if (p4d_present(*p4d) && !p4d_free_pud_page(p4d, addr)) - return 0; - - return p4d_set_huge(p4d, phys_addr, prot); -} - -static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, - unsigned int max_page_shift, pgtbl_mod_mask *mask) -{ - p4d_t *p4d; - unsigned long next; - - p4d = p4d_alloc_track(&init_mm, pgd, addr, mask); - if (!p4d) - return -ENOMEM; - do { - next = p4d_addr_end(addr, end); - - if (vmap_try_huge_p4d(p4d, addr, next, phys_addr, prot, - max_page_shift)) { - *mask |= PGTBL_P4D_MODIFIED; - continue; - } - - if (vmap_pud_range(p4d, addr, next, phys_addr, prot, - max_page_shift, mask)) - return -ENOMEM; - } while (p4d++, phys_addr += (next - addr), addr = next, addr != end); - return 0; -} - -static int vmap_range(unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, - unsigned int max_page_shift) -{ - pgd_t *pgd; - unsigned long start; - unsigned long next; - int err; - pgtbl_mod_mask mask = 0; - - might_sleep(); - BUG_ON(addr >= end); - - start = addr; - pgd = pgd_offset_k(addr); - do { - next = pgd_addr_end(addr, end); - err = vmap_p4d_range(pgd, addr, next, phys_addr, prot, - max_page_shift, &mask); - if (err) - break; - } while (pgd++, phys_addr += (next - addr), addr = next, addr != end); - - flush_cache_vmap(start, end); - - if (mask & ARCH_PAGE_TABLE_SYNC_MASK) - arch_sync_kernel_mappings(start, end); - - return err; -} - int ioremap_page_range(unsigned long addr, unsigned long end, phys_addr_t phys_addr, pgprot_t prot) { diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 4693fab4f42a..53414959845d 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -68,6 +68,208 @@ static void free_work(struct work_struct *w) } /*** Page table manipulation functions ***/ +static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + pgtbl_mod_mask *mask) +{ + pte_t *pte; + u64 pfn; + + pfn = phys_addr >> PAGE_SHIFT; + pte = pte_alloc_kernel_track(pmd, addr, mask); + if (!pte) + return -ENOMEM; + do { + BUG_ON(!pte_none(*pte)); + set_pte_at(&init_mm, addr, pte, pfn_pte(pfn, prot)); + pfn++; + } while (pte++, addr += PAGE_SIZE, addr != end); + *mask |= PGTBL_PTE_MODIFIED; + return 0; +} + +static int vmap_try_huge_pmd(pmd_t *pmd, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift) +{ + if (max_page_shift < PMD_SHIFT) + return 0; + + if (!arch_vmap_pmd_supported(prot)) + return 0; + + if ((end - addr) != PMD_SIZE) + return 0; + + if (!IS_ALIGNED(addr, PMD_SIZE)) + return 0; + + if (!IS_ALIGNED(phys_addr, PMD_SIZE)) + return 0; + + if (pmd_present(*pmd) && !pmd_free_pte_page(pmd, addr)) + return 0; + + return pmd_set_huge(pmd, phys_addr, prot); +} + +static int vmap_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift, pgtbl_mod_mask *mask) +{ + pmd_t *pmd; + unsigned long next; + + pmd = pmd_alloc_track(&init_mm, pud, addr, mask); + if (!pmd) + return -ENOMEM; + do { + next = pmd_addr_end(addr, end); + + if (vmap_try_huge_pmd(pmd, addr, next, phys_addr, prot, + max_page_shift)) { + *mask |= PGTBL_PMD_MODIFIED; + continue; + } + + if (vmap_pte_range(pmd, addr, next, phys_addr, prot, mask)) + return -ENOMEM; + } while (pmd++, phys_addr += (next - addr), addr = next, addr != end); + return 0; +} + +static int vmap_try_huge_pud(pud_t *pud, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift) +{ + if (max_page_shift < PUD_SHIFT) + return 0; + + if (!arch_vmap_pud_supported(prot)) + return 0; + + if ((end - addr) != PUD_SIZE) + return 0; + + if (!IS_ALIGNED(addr, PUD_SIZE)) + return 0; + + if (!IS_ALIGNED(phys_addr, PUD_SIZE)) + return 0; + + if (pud_present(*pud) && !pud_free_pmd_page(pud, addr)) + return 0; + + return pud_set_huge(pud, phys_addr, prot); +} + +static int vmap_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift, pgtbl_mod_mask *mask) +{ + pud_t *pud; + unsigned long next; + + pud = pud_alloc_track(&init_mm, p4d, addr, mask); + if (!pud) + return -ENOMEM; + do { + next = pud_addr_end(addr, end); + + if (vmap_try_huge_pud(pud, addr, next, phys_addr, prot, + max_page_shift)) { + *mask |= PGTBL_PUD_MODIFIED; + continue; + } + + if (vmap_pmd_range(pud, addr, next, phys_addr, prot, + max_page_shift, mask)) + return -ENOMEM; + } while (pud++, phys_addr += (next - addr), addr = next, addr != end); + return 0; +} + +static int vmap_try_huge_p4d(p4d_t *p4d, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift) +{ + if (max_page_shift < P4D_SHIFT) + return 0; + + if (!arch_vmap_p4d_supported(prot)) + return 0; + + if ((end - addr) != P4D_SIZE) + return 0; + + if (!IS_ALIGNED(addr, P4D_SIZE)) + return 0; + + if (!IS_ALIGNED(phys_addr, P4D_SIZE)) + return 0; + + if (p4d_present(*p4d) && !p4d_free_pud_page(p4d, addr)) + return 0; + + return p4d_set_huge(p4d, phys_addr, prot); +} + +static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift, pgtbl_mod_mask *mask) +{ + p4d_t *p4d; + unsigned long next; + + p4d = p4d_alloc_track(&init_mm, pgd, addr, mask); + if (!p4d) + return -ENOMEM; + do { + next = p4d_addr_end(addr, end); + + if (vmap_try_huge_p4d(p4d, addr, next, phys_addr, prot, + max_page_shift)) { + *mask |= PGTBL_P4D_MODIFIED; + continue; + } + + if (vmap_pud_range(p4d, addr, next, phys_addr, prot, + max_page_shift, mask)) + return -ENOMEM; + } while (p4d++, phys_addr += (next - addr), addr = next, addr != end); + return 0; +} + +int vmap_range(unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift) +{ + pgd_t *pgd; + unsigned long start; + unsigned long next; + int err; + pgtbl_mod_mask mask = 0; + + might_sleep(); + BUG_ON(addr >= end); + + start = addr; + pgd = pgd_offset_k(addr); + do { + next = pgd_addr_end(addr, end); + err = vmap_p4d_range(pgd, addr, next, phys_addr, prot, + max_page_shift, &mask); + if (err) + break; + } while (pgd++, phys_addr += (next - addr), addr = next, addr != end); + + flush_cache_vmap(start, end); + + if (mask & ARCH_PAGE_TABLE_SYNC_MASK) + arch_sync_kernel_mappings(start, end); + + return err; +} static void vunmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, pgtbl_mod_mask *mask) From patchwork Wed Mar 17 06:24:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 12144901 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A1821C433DB for ; Wed, 17 Mar 2021 06:25:46 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4248164F8C for ; Wed, 17 Mar 2021 06:25:46 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4248164F8C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id CFE046B0074; Wed, 17 Mar 2021 02:25:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CDB946B0083; Wed, 17 Mar 2021 02:25:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B4F4C6B0085; Wed, 17 Mar 2021 02:25:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0144.hostedemail.com [216.40.44.144]) by kanga.kvack.org (Postfix) with ESMTP id 9A6A16B0074 for ; Wed, 17 Mar 2021 02:25:45 -0400 (EDT) Received: from smtpin30.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 622D2181AF5F7 for ; Wed, 17 Mar 2021 06:25:45 +0000 (UTC) X-FDA: 77928380250.30.6252538 Received: from mail-pf1-f180.google.com (mail-pf1-f180.google.com [209.85.210.180]) by imf04.hostedemail.com (Postfix) with ESMTP id 0622E12E for ; Wed, 17 Mar 2021 06:25:44 +0000 (UTC) Received: by mail-pf1-f180.google.com with SMTP id l3so408293pfc.7 for ; Tue, 16 Mar 2021 23:25:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=xD9CZCqfp6oOH0TENXfs67JbMzL04aaTtGQqYYdMF/Y=; b=ABPFYdpLbjkf6zwF7B72Zaoi9pvfhFCoeGzDoVnTUKtm1yxN+kjAmc19B52QXi3lvF wyZ8mEa7+Z+XStechgQYeu68Oxg3e6YrtrwlsgauDrOgV9/dLSYi6EFh0iLlvJi+xK1c kemMMu3Tm1xpoSPk3gB/phos27rP+OAcxHQ7tcFYNsjL2aTUapJFt6H4vDw+e4jgSoVh P1wUghjc6pah1T5/sOy26izKRVqvDKRh/l814l9hBzcXarn1BgQaL9gXxk9vF27KrvXd mAk8GKmMdZJbjov4CW8U/FxCgsvHMrxt70JhlVODRNo20Owg2NoTAnmKZfgZjc9cIdVf axog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=xD9CZCqfp6oOH0TENXfs67JbMzL04aaTtGQqYYdMF/Y=; b=FnrGVYnOtIvfmPqxlOMIokhc8W3P/fcSA8Mz+gEpSACrexk8NgMYMuNG43OBU/OowY ZctYFicJhYPq8mIFCO4BGvHl5R4Z1Rir8BFVGVitQegEpDe/s2dD6IaOyBEtomQSK2dB FpCbiQ/NYjIzqDZ/HPDts8dTPxdynmVPKe7WWjiqBltgoXdEIB0T9v+yH1q8GwwAQbL2 xF8lDN0qkLealtsLXCHhKcQDcOR3H+h+m4XOnwNORrsWWXb9FxVoSajL30wkFe5VF3/j PcuUUMMEP/jiuAPyh0FRdkqnrJaSOJSw2+nB1YE8hhs2M1swZNt5QZ77n05dHwzMDdGu Cw0g== X-Gm-Message-State: AOAM530892/AHr2sCc5OL/H4AjLlAEdQiLjnARPWqjMA/didPXhtH/Jc Smry8sO9h5p1TbWKkLCSx9dvKuSSZRg= X-Google-Smtp-Source: ABdhPJwLFoYHkHHTweADUvQkyZeGoreDRgE78Md0c3eS25/BwdXU+gAdtbvWckprWOjaLchflfPI+A== X-Received: by 2002:aa7:991a:0:b029:201:b736:c556 with SMTP id z26-20020aa7991a0000b0290201b736c556mr2961901pff.8.1615962343854; Tue, 16 Mar 2021 23:25:43 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (58-6-239-121.tpgi.com.au. [58.6.239.121]) by smtp.gmail.com with ESMTPSA id s19sm17959620pfh.168.2021.03.16.23.25.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Mar 2021 23:25:43 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org, Andrew Morton Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, Jonathan Cameron , Christoph Hellwig , Christophe Leroy , Rick Edgecombe , Ding Tianhong , Christoph Hellwig Subject: [PATCH v13 12/14] mm/vmalloc: add vmap_range_noflush variant Date: Wed, 17 Mar 2021 16:24:00 +1000 Message-Id: <20210317062402.533919-13-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20210317062402.533919-1-npiggin@gmail.com> References: <20210317062402.533919-1-npiggin@gmail.com> MIME-Version: 1.0 X-Stat-Signature: 3u9yfymgc3tu7m7zrsik6iab3xbi777j X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 0622E12E Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf04; identity=mailfrom; envelope-from=""; helo=mail-pf1-f180.google.com; client-ip=209.85.210.180 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1615962344-864686 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: As a side-effect, the order of flush_cache_vmap() and arch_sync_kernel_mappings() calls are switched, but that now matches the other callers in this file. Reviewed-by: Christoph Hellwig Signed-off-by: Nicholas Piggin --- mm/vmalloc.c | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 53414959845d..9455dba58b0e 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -240,7 +240,7 @@ static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, return 0; } -int vmap_range(unsigned long addr, unsigned long end, +static int vmap_range_noflush(unsigned long addr, unsigned long end, phys_addr_t phys_addr, pgprot_t prot, unsigned int max_page_shift) { @@ -263,14 +263,24 @@ int vmap_range(unsigned long addr, unsigned long end, break; } while (pgd++, phys_addr += (next - addr), addr = next, addr != end); - flush_cache_vmap(start, end); - if (mask & ARCH_PAGE_TABLE_SYNC_MASK) arch_sync_kernel_mappings(start, end); return err; } +int vmap_range(unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift) +{ + int err; + + err = vmap_range_noflush(addr, end, phys_addr, prot, max_page_shift); + flush_cache_vmap(addr, end); + + return err; +} + static void vunmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, pgtbl_mod_mask *mask) { From patchwork Wed Mar 17 06:24:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 12144903 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 06C03C433E0 for ; Wed, 17 Mar 2021 06:25:53 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 7F37664F8C for ; Wed, 17 Mar 2021 06:25:52 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7F37664F8C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 0EF836B0083; Wed, 17 Mar 2021 02:25:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0C6D66B0085; Wed, 17 Mar 2021 02:25:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E5B6C8D0001; Wed, 17 Mar 2021 02:25:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0115.hostedemail.com [216.40.44.115]) by kanga.kvack.org (Postfix) with ESMTP id C2B356B0083 for ; Wed, 17 Mar 2021 02:25:51 -0400 (EDT) Received: from smtpin20.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 80D858249980 for ; Wed, 17 Mar 2021 06:25:51 +0000 (UTC) X-FDA: 77928380502.20.D871587 Received: from mail-pj1-f54.google.com (mail-pj1-f54.google.com [209.85.216.54]) by imf13.hostedemail.com (Postfix) with ESMTP id D1E2EE0011CE for ; Wed, 17 Mar 2021 06:25:50 +0000 (UTC) Received: by mail-pj1-f54.google.com with SMTP id x7-20020a17090a2b07b02900c0ea793940so2654999pjc.2 for ; Tue, 16 Mar 2021 23:25:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=1uU5bcxzx01P8dHEfjYViSMSkdrX5rHnhOTXMkO32oE=; b=Dj87U9gZev1Rr5H6OFRHf6eB+X0ny+W2T+OPJf0i/fOrJmxO3O48S9Vx0Mbw6Gq6NX gfPOFq5B7JTJrzXg18w7QBdVeT+YXh3gNrSw25zVXb6ta4pysEGsbSEriZc/k2fEtFh9 t7E5K3yCYvFBW0G1nF3YU7MsJ0thlF3SKnEhydrd/GBiX3zAOo0o1vJDWgERNqSjd3zY EjfQeI25/XpweODnWb8UdiQdM5IwydmQ74Scg0xDwuUXCyLXdQhDWLBrpLzOGBvk2POz FceEazvOBdPhVPkXW+xYqQt1Lglvvv2EWea6GEhmIbSfe7RurmDdjhAEh4LZyN8LtnqI 6UmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=1uU5bcxzx01P8dHEfjYViSMSkdrX5rHnhOTXMkO32oE=; b=bQ0smCTh9lQqgmoPoAzO9rah7/df+7hekJb9RA0ptOYVcT7Xvelx3nu8RFnS11zov3 OLXYUil4HoyWIpWqyNYkHGPBt23oHybEYPLwig++fqzVTL3y1ldSHi5Nl/xqJ4e7lSXF 5OUvHO7tS9gjSfeIFcdI8QgX4QzUDsvehPOVVLaOuL1EcFf65FZ9MaHJl76drmCM2Wbh QasmML3ecADItOGGoAOkfRqVtPrb6/R2LgU8dt330khCpjnlRRye/Mr98ZWV79y828KM lgZ2u8ntN3uChLf8i3GGF9B+xq9uN5FLgzJd+Ja5Vpb++jehVuAbVDImxnxKRVcS8OrE TlrA== X-Gm-Message-State: AOAM5330NZcRW13abjZmnLMj/b5eeUYDOay2GGirTwL3KqeMrtVGjMnN gLg0SEXBBXMZG1711UtTtuKAT1AziXY= X-Google-Smtp-Source: ABdhPJxulUfRkKo4BI6n1Nb1qSvY1AXLNKJ0oLKh3JtExGE7ySc2ERfh6WLBwjpZzO7oJObIeRqEDg== X-Received: by 2002:a17:902:7888:b029:e6:b94d:c72 with SMTP id q8-20020a1709027888b02900e6b94d0c72mr3106960pll.8.1615962349361; Tue, 16 Mar 2021 23:25:49 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (58-6-239-121.tpgi.com.au. [58.6.239.121]) by smtp.gmail.com with ESMTPSA id s19sm17959620pfh.168.2021.03.16.23.25.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Mar 2021 23:25:48 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org, Andrew Morton Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, Jonathan Cameron , Christoph Hellwig , Christophe Leroy , Rick Edgecombe , Ding Tianhong Subject: [PATCH v13 13/14] mm/vmalloc: Hugepage vmalloc mappings Date: Wed, 17 Mar 2021 16:24:01 +1000 Message-Id: <20210317062402.533919-14-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20210317062402.533919-1-npiggin@gmail.com> References: <20210317062402.533919-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: D1E2EE0011CE X-Stat-Signature: 6aawy6umos7hffqi54cejfcx4hz58mzi Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf13; identity=mailfrom; envelope-from=""; helo=mail-pj1-f54.google.com; client-ip=209.85.216.54 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1615962350-262181 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Support huge page vmalloc mappings. Config option HAVE_ARCH_HUGE_VMALLOC enables support on architectures that define HAVE_ARCH_HUGE_VMAP and supports PMD sized vmap mappings. vmalloc will attempt to allocate PMD-sized pages if allocating PMD size or larger, and fall back to small pages if that was unsuccessful. Architectures must ensure that any arch specific vmalloc allocations that require PAGE_SIZE mappings (e.g., module allocations vs strict module rwx) use the VM_NOHUGE flag to inhibit larger mappings. This can result in more internal fragmentation and memory overhead for a given allocation, an option nohugevmalloc is added to disable at boot. Signed-off-by: Nicholas Piggin --- arch/Kconfig | 11 ++ include/linux/vmalloc.h | 21 ++++ mm/page_alloc.c | 5 +- mm/vmalloc.c | 216 +++++++++++++++++++++++++++++++--------- 4 files changed, 206 insertions(+), 47 deletions(-) diff --git a/arch/Kconfig b/arch/Kconfig index ecfd3520b676..b347102f2984 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -785,6 +785,17 @@ config HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD config HAVE_ARCH_HUGE_VMAP bool +# +# Archs that select this would be capable of PMD-sized vmaps (i.e., +# arch_vmap_pmd_supported() returns true), and they must make no assumptions +# that vmalloc memory is mapped with PAGE_SIZE ptes. The VM_NO_HUGE_VMAP flag +# can be used to prohibit arch-specific allocations from using hugepages to +# help with this (e.g., modules may require it). +# +config HAVE_ARCH_HUGE_VMALLOC + depends on HAVE_ARCH_HUGE_VMAP + bool + config ARCH_WANT_HUGE_PMD_SHARE bool diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 1f6844e2670a..8341964e6eb5 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -26,6 +26,7 @@ struct notifier_block; /* in notifier.h */ #define VM_KASAN 0x00000080 /* has allocated kasan shadow memory */ #define VM_FLUSH_RESET_PERMS 0x00000100 /* reset direct map and flush TLB on unmap, can't be freed in atomic context */ #define VM_MAP_PUT_PAGES 0x00000200 /* put pages and free array in vfree */ +#define VM_NO_HUGE_VMAP 0x00000400 /* force PAGE_SIZE pte mapping */ /* * VM_KASAN is used slighly differently depending on CONFIG_KASAN_VMALLOC. @@ -54,6 +55,9 @@ struct vm_struct { unsigned long size; unsigned long flags; struct page **pages; +#ifdef CONFIG_HAVE_ARCH_HUGE_VMALLOC + unsigned int page_order; +#endif unsigned int nr_pages; phys_addr_t phys_addr; const void *caller; @@ -188,6 +192,22 @@ void free_vm_area(struct vm_struct *area); extern struct vm_struct *remove_vm_area(const void *addr); extern struct vm_struct *find_vm_area(const void *addr); +static inline bool is_vm_area_hugepages(const void *addr) +{ + /* + * This may not 100% tell if the area is mapped with > PAGE_SIZE + * page table entries, if for some reason the architecture indicates + * larger sizes are available but decides not to use them, nothing + * prevents that. This only indicates the size of the physical page + * allocated in the vmalloc layer. + */ +#ifdef CONFIG_HAVE_ARCH_HUGE_VMALLOC + return find_vm_area(addr)->page_order > 0; +#else + return false; +#endif +} + #ifdef CONFIG_MMU int vmap_range(unsigned long addr, unsigned long end, phys_addr_t phys_addr, pgprot_t prot, @@ -205,6 +225,7 @@ static inline void set_vm_flush_reset_perms(void *addr) if (vm) vm->flags |= VM_FLUSH_RESET_PERMS; } + #else static inline int map_kernel_range_noflush(unsigned long start, unsigned long size, diff --git a/mm/page_alloc.c b/mm/page_alloc.c index cfc72873961d..2e2042f39b8b 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -72,6 +72,7 @@ #include #include #include +#include #include #include @@ -8222,6 +8223,7 @@ void *__init alloc_large_system_hash(const char *tablename, void *table = NULL; gfp_t gfp_flags; bool virt; + bool huge; /* allow the kernel cmdline to have a say */ if (!numentries) { @@ -8289,6 +8291,7 @@ void *__init alloc_large_system_hash(const char *tablename, } else if (get_order(size) >= MAX_ORDER || hashdist) { table = __vmalloc(size, gfp_flags); virt = true; + huge = is_vm_area_hugepages(table); } else { /* * If bucketsize is not a power-of-two, we may free @@ -8305,7 +8308,7 @@ void *__init alloc_large_system_hash(const char *tablename, pr_info("%s hash table entries: %ld (order: %d, %lu bytes, %s)\n", tablename, 1UL << log2qty, ilog2(size) - PAGE_SHIFT, size, - virt ? "vmalloc" : "linear"); + virt ? (huge ? "vmalloc hugepage" : "vmalloc") : "linear"); if (_hash_shift) *_hash_shift = log2qty; diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 9455dba58b0e..87e3bec51ddb 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -42,6 +42,19 @@ #include "internal.h" #include "pgalloc-track.h" +#ifdef CONFIG_HAVE_ARCH_HUGE_VMALLOC +static bool __ro_after_init vmap_allow_huge = true; + +static int __init set_nohugevmalloc(char *str) +{ + vmap_allow_huge = false; + return 0; +} +early_param("nohugevmalloc", set_nohugevmalloc); +#else /* CONFIG_HAVE_ARCH_HUGE_VMALLOC */ +static const bool vmap_allow_huge = false; +#endif /* CONFIG_HAVE_ARCH_HUGE_VMALLOC */ + bool is_vmalloc_addr(const void *x) { unsigned long addr = (unsigned long)x; @@ -483,31 +496,12 @@ static int vmap_pages_p4d_range(pgd_t *pgd, unsigned long addr, return 0; } -/** - * map_kernel_range_noflush - map kernel VM area with the specified pages - * @addr: start of the VM area to map - * @size: size of the VM area to map - * @prot: page protection flags to use - * @pages: pages to map - * - * Map PFN_UP(@size) pages at @addr. The VM area @addr and @size specify should - * have been allocated using get_vm_area() and its friends. - * - * NOTE: - * This function does NOT do any cache flushing. The caller is responsible for - * calling flush_cache_vmap() on to-be-mapped areas before calling this - * function. - * - * RETURNS: - * 0 on success, -errno on failure. - */ -int map_kernel_range_noflush(unsigned long addr, unsigned long size, - pgprot_t prot, struct page **pages) +static int vmap_small_pages_range_noflush(unsigned long addr, unsigned long end, + pgprot_t prot, struct page **pages) { unsigned long start = addr; - unsigned long end = addr + size; - unsigned long next; pgd_t *pgd; + unsigned long next; int err = 0; int nr = 0; pgtbl_mod_mask mask = 0; @@ -529,6 +523,66 @@ int map_kernel_range_noflush(unsigned long addr, unsigned long size, return 0; } +static int vmap_pages_range_noflush(unsigned long addr, unsigned long end, + pgprot_t prot, struct page **pages, unsigned int page_shift) +{ + unsigned int i, nr = (end - addr) >> PAGE_SHIFT; + + WARN_ON(page_shift < PAGE_SHIFT); + + if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMALLOC) || + page_shift == PAGE_SHIFT) + return vmap_small_pages_range_noflush(addr, end, prot, pages); + + for (i = 0; i < nr; i += 1U << (page_shift - PAGE_SHIFT)) { + int err; + + err = vmap_range_noflush(addr, addr + (1UL << page_shift), + __pa(page_address(pages[i])), prot, + page_shift); + if (err) + return err; + + addr += 1UL << page_shift; + } + + return 0; +} + +static int vmap_pages_range(unsigned long addr, unsigned long end, + pgprot_t prot, struct page **pages, unsigned int page_shift) +{ + int err; + + err = vmap_pages_range_noflush(addr, end, prot, pages, page_shift); + flush_cache_vmap(addr, end); + return err; +} + +/** + * map_kernel_range_noflush - map kernel VM area with the specified pages + * @addr: start of the VM area to map + * @size: size of the VM area to map + * @prot: page protection flags to use + * @pages: pages to map + * + * Map PFN_UP(@size) pages at @addr. The VM area @addr and @size specify should + * have been allocated using get_vm_area() and its friends. + * + * NOTE: + * This function does NOT do any cache flushing. The caller is responsible for + * calling flush_cache_vmap() on to-be-mapped areas before calling this + * function. + * + * RETURNS: + * 0 on success, -errno on failure. + */ +int map_kernel_range_noflush(unsigned long addr, unsigned long size, + pgprot_t prot, struct page **pages) +{ + return vmap_pages_range_noflush(addr, addr + size, prot, pages, PAGE_SHIFT); +} + int map_kernel_range(unsigned long start, unsigned long size, pgprot_t prot, struct page **pages) { @@ -2112,6 +2166,24 @@ EXPORT_SYMBOL(vm_map_ram); static struct vm_struct *vmlist __initdata; +static inline unsigned int vm_area_page_order(struct vm_struct *vm) +{ +#ifdef CONFIG_HAVE_ARCH_HUGE_VMALLOC + return vm->page_order; +#else + return 0; +#endif +} + +static inline void set_vm_area_page_order(struct vm_struct *vm, unsigned int order) +{ +#ifdef CONFIG_HAVE_ARCH_HUGE_VMALLOC + vm->page_order = order; +#else + BUG_ON(order != 0); +#endif +} + /** * vm_area_add_early - add vmap area early during boot * @vm: vm_struct to add @@ -2422,6 +2494,7 @@ static inline void set_area_direct_map(const struct vm_struct *area, { int i; + /* HUGE_VMALLOC passes small pages to set_direct_map */ for (i = 0; i < area->nr_pages; i++) if (page_address(area->pages[i])) set_direct_map(area->pages[i]); @@ -2431,6 +2504,7 @@ static inline void set_area_direct_map(const struct vm_struct *area, static void vm_remove_mappings(struct vm_struct *area, int deallocate_pages) { unsigned long start = ULONG_MAX, end = 0; + unsigned int page_order = vm_area_page_order(area); int flush_reset = area->flags & VM_FLUSH_RESET_PERMS; int flush_dmap = 0; int i; @@ -2455,11 +2529,14 @@ static void vm_remove_mappings(struct vm_struct *area, int deallocate_pages) * map. Find the start and end range of the direct mappings to make sure * the vm_unmap_aliases() flush includes the direct map. */ - for (i = 0; i < area->nr_pages; i++) { + for (i = 0; i < area->nr_pages; i += 1U << page_order) { unsigned long addr = (unsigned long)page_address(area->pages[i]); if (addr) { + unsigned long page_size; + + page_size = PAGE_SIZE << page_order; start = min(addr, start); - end = max(addr + PAGE_SIZE, end); + end = max(addr + page_size, end); flush_dmap = 1; } } @@ -2500,13 +2577,14 @@ static void __vunmap(const void *addr, int deallocate_pages) vm_remove_mappings(area, deallocate_pages); if (deallocate_pages) { + unsigned int page_order = vm_area_page_order(area); int i; - for (i = 0; i < area->nr_pages; i++) { + for (i = 0; i < area->nr_pages; i += 1U << page_order) { struct page *page = area->pages[i]; BUG_ON(!page); - __free_pages(page, 0); + __free_pages(page, page_order); } atomic_long_sub(area->nr_pages, &nr_vmalloc_pages); @@ -2697,15 +2775,19 @@ EXPORT_SYMBOL_GPL(vmap_pfn); #endif /* CONFIG_VMAP_PFN */ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, - pgprot_t prot, int node) + pgprot_t prot, unsigned int page_shift, + int node) { const gfp_t nested_gfp = (gfp_mask & GFP_RECLAIM_MASK) | __GFP_ZERO; - unsigned int nr_pages = get_vm_area_size(area) >> PAGE_SHIFT; + unsigned long addr = (unsigned long)area->addr; + unsigned long size = get_vm_area_size(area); unsigned long array_size; - unsigned int i; + unsigned int nr_small_pages = size >> PAGE_SHIFT; + unsigned int page_order; struct page **pages; + unsigned int i; - array_size = (unsigned long)nr_pages * sizeof(struct page *); + array_size = (unsigned long)nr_small_pages * sizeof(struct page *); gfp_mask |= __GFP_NOWARN; if (!(gfp_mask & (GFP_DMA | GFP_DMA32))) gfp_mask |= __GFP_HIGHMEM; @@ -2724,30 +2806,38 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, } area->pages = pages; - area->nr_pages = nr_pages; + area->nr_pages = nr_small_pages; + set_vm_area_page_order(area, page_shift - PAGE_SHIFT); - for (i = 0; i < area->nr_pages; i++) { - struct page *page; + page_order = vm_area_page_order(area); - if (node == NUMA_NO_NODE) - page = alloc_page(gfp_mask); - else - page = alloc_pages_node(node, gfp_mask, 0); + /* + * Careful, we allocate and map page_order pages, but tracking is done + * per PAGE_SIZE page so as to keep the vm_struct APIs independent of + * the physical/mapped size. + */ + for (i = 0; i < area->nr_pages; i += 1U << page_order) { + struct page *page; + int p; + /* Compound pages required for remap_vmalloc_page */ + page = alloc_pages_node(node, gfp_mask | __GFP_COMP, page_order); if (unlikely(!page)) { /* Successfully allocated i pages, free them in __vfree() */ area->nr_pages = i; atomic_long_add(area->nr_pages, &nr_vmalloc_pages); goto fail; } - area->pages[i] = page; + + for (p = 0; p < (1U << page_order); p++) + area->pages[i + p] = page + p; + if (gfpflags_allow_blocking(gfp_mask)) cond_resched(); } atomic_long_add(area->nr_pages, &nr_vmalloc_pages); - if (map_kernel_range((unsigned long)area->addr, get_vm_area_size(area), - prot, pages) < 0) + if (vmap_pages_range(addr, addr + size, prot, pages, page_shift) < 0) goto fail; return area->addr; @@ -2755,7 +2845,7 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, fail: warn_alloc(gfp_mask, NULL, "vmalloc: allocation failure, allocated %ld of %ld bytes", - (area->nr_pages*PAGE_SIZE), area->size); + (area->nr_pages*PAGE_SIZE), size); __vfree(area->addr); return NULL; } @@ -2786,19 +2876,43 @@ void *__vmalloc_node_range(unsigned long size, unsigned long align, struct vm_struct *area; void *addr; unsigned long real_size = size; + unsigned long real_align = align; + unsigned int shift = PAGE_SHIFT; - size = PAGE_ALIGN(size); if (!size || (size >> PAGE_SHIFT) > totalram_pages()) goto fail; - area = __get_vm_area_node(real_size, align, VM_ALLOC | VM_UNINITIALIZED | + if (vmap_allow_huge && !(vm_flags & VM_NO_HUGE_VMAP) && + arch_vmap_pmd_supported(prot)) { + unsigned long size_per_node; + + /* + * Try huge pages. Only try for PAGE_KERNEL allocations, + * others like modules don't yet expect huge pages in + * their allocations due to apply_to_page_range not + * supporting them. + */ + + size_per_node = size; + if (node == NUMA_NO_NODE) + size_per_node /= num_online_nodes(); + if (size_per_node >= PMD_SIZE) { + shift = PMD_SHIFT; + align = max(real_align, 1UL << shift); + size = ALIGN(real_size, 1UL << shift); + } + } + +again: + size = PAGE_ALIGN(size); + area = __get_vm_area_node(size, align, VM_ALLOC | VM_UNINITIALIZED | vm_flags, start, end, node, gfp_mask, caller); if (!area) goto fail; - addr = __vmalloc_area_node(area, gfp_mask, prot, node); + addr = __vmalloc_area_node(area, gfp_mask, prot, shift, node); if (!addr) - return NULL; + goto fail; /* * In this function, newly allocated vm_struct has VM_UNINITIALIZED @@ -2812,8 +2926,18 @@ void *__vmalloc_node_range(unsigned long size, unsigned long align, return addr; fail: - warn_alloc(gfp_mask, NULL, + if (shift > PAGE_SHIFT) { + shift = PAGE_SHIFT; + align = real_align; + size = real_size; + goto again; + } + + if (!area) { + /* Warn for area allocation, page allocations already warn */ + warn_alloc(gfp_mask, NULL, "vmalloc: allocation failure: %lu bytes", real_size); + } return NULL; } From patchwork Wed Mar 17 06:24:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 12144905 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8E3F5C433E6 for ; Wed, 17 Mar 2021 06:25:58 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 32AA064F8C for ; Wed, 17 Mar 2021 06:25:58 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 32AA064F8C Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C14186B0078; Wed, 17 Mar 2021 02:25:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BEA376B0085; Wed, 17 Mar 2021 02:25:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A8B8C8D0001; Wed, 17 Mar 2021 02:25:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 90D076B0078 for ; Wed, 17 Mar 2021 02:25:57 -0400 (EDT) Received: from smtpin27.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 427042C94 for ; Wed, 17 Mar 2021 06:25:57 +0000 (UTC) X-FDA: 77928380754.27.D95D042 Received: from mail-pj1-f44.google.com (mail-pj1-f44.google.com [209.85.216.44]) by imf17.hostedemail.com (Postfix) with ESMTP id C77B7407F8DF for ; Wed, 17 Mar 2021 06:25:56 +0000 (UTC) Received: by mail-pj1-f44.google.com with SMTP id k23-20020a17090a5917b02901043e35ad4aso2603017pji.3 for ; Tue, 16 Mar 2021 23:25:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=WZnEda0T3pGShx0QqEfgouMD/KA4MVCDJ6OoGinC0gE=; b=fLYlcSfNjjYf4im0CmwXX9uFu0R6/2elBdnUDdyYVOlkYQAe04v1WgcG7ZIGlpljT9 SLjBuUq0fzfMpOxz7InlDaQ4plALW4GVs7HLjWHat3QIhZxnYNHZKbTvtG9NtKWzC/5E /MHpVEnov74/4Jb2KNQKv3r6lRMpiz4IbCFj6ozscN3pw0QWY/OlApapX32lg8ev/Geb idkPjx22igBJf+xnGC6bLxyblUbSdNaRC7JGRPDPnfOVbd0+5lLwOns57KjSaIMeIuI7 UL6xeHwUjI8k8Z0g/0VRjA1ERirFFmiNfgZyWk+OUGKwFDiOhV1yCAI+/3GySAB0aDgU Xo8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=WZnEda0T3pGShx0QqEfgouMD/KA4MVCDJ6OoGinC0gE=; b=UK0ZSdgg8jZtCyX69wX6kqFCPv5nnx3uef4ArERLe0J7SzSF2cYjvIWZaptq7XbrmW NlMhMgAHnJOCCVzAi4Qm1NARvwRLQ/s7FvveDFGCaqgQfYeNHUCfBXW8CM7VDX/Lg10F yb1D2QLxtCNJ9IRXqpj3zhAh2KzxGIj3uO8XqP36wYpQQQLiwlozr9NXUlPfAfriA5yZ UXTnwIwHHzcRvC7fE/lnquxJRGxs3GXEX3h3Crij2rZydPyp2zJYJu3ZF4FlZPzbgyvB 4t4iV7SCYJ2eiw/eNhoTm5UxrvsCG3c+Ltow+dNx3nzxX83eBZ3nCMi4whQ2oV85peV6 UfSg== X-Gm-Message-State: AOAM531lifvoZrFt/8JbgVc997SJ/erHlVCnFtLs/OXpB+pI62FhC9Oa wBX/T5BLbr/AwETC1xT3SWAeSgGkKAc= X-Google-Smtp-Source: ABdhPJyRI+bGimVthTrlN06P4ADDSwiZIZdfQrOqluFmWVNVhhZm6lHvDqdPI9QCbgNXHKtlH+H10g== X-Received: by 2002:a17:902:b60d:b029:e6:7a9:7f4 with SMTP id b13-20020a170902b60db02900e607a907f4mr2850305pls.3.1615962355626; Tue, 16 Mar 2021 23:25:55 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (58-6-239-121.tpgi.com.au. [58.6.239.121]) by smtp.gmail.com with ESMTPSA id s19sm17959620pfh.168.2021.03.16.23.25.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 16 Mar 2021 23:25:55 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org, Andrew Morton Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, Jonathan Cameron , Christoph Hellwig , Christophe Leroy , Rick Edgecombe , Ding Tianhong , linuxppc-dev@lists.ozlabs.org, Michael Ellerman Subject: [PATCH v13 14/14] powerpc/64s/radix: Enable huge vmalloc mappings Date: Wed, 17 Mar 2021 16:24:02 +1000 Message-Id: <20210317062402.533919-15-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20210317062402.533919-1-npiggin@gmail.com> References: <20210317062402.533919-1-npiggin@gmail.com> MIME-Version: 1.0 X-Stat-Signature: ciha8kq5p1m8jhm1kb5g7cukjw6r7q9n X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: C77B7407F8DF Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf17; identity=mailfrom; envelope-from=""; helo=mail-pj1-f44.google.com; client-ip=209.85.216.44 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1615962356-953553 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This reduces TLB misses by nearly 30x on a `git diff` workload on a 2-node POWER9 (59,800 -> 2,100) and reduces CPU cycles by 0.54%, due to vfs hashes being allocated with 2MB pages. Cc: linuxppc-dev@lists.ozlabs.org Acked-by: Michael Ellerman Signed-off-by: Nicholas Piggin --- .../admin-guide/kernel-parameters.txt | 2 ++ arch/powerpc/Kconfig | 1 + arch/powerpc/kernel/module.c | 22 +++++++++++++++---- 3 files changed, 21 insertions(+), 4 deletions(-) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index 04545725f187..1f481f904895 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -3243,6 +3243,8 @@ nohugeiomap [KNL,X86,PPC,ARM64] Disable kernel huge I/O mappings. + nohugevmalloc [PPC] Disable kernel huge vmalloc mappings. + nosmt [KNL,S390] Disable symmetric multithreading (SMT). Equivalent to smt=1. diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index 386ae12d8523..b7cade9566da 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -181,6 +181,7 @@ config PPC select GENERIC_GETTIMEOFDAY select HAVE_ARCH_AUDITSYSCALL select HAVE_ARCH_HUGE_VMAP if PPC_BOOK3S_64 && PPC_RADIX_MMU + select HAVE_ARCH_HUGE_VMALLOC if HAVE_ARCH_HUGE_VMAP select HAVE_ARCH_JUMP_LABEL select HAVE_ARCH_KASAN if PPC32 && PPC_PAGE_SHIFT <= 14 select HAVE_ARCH_KASAN_VMALLOC if PPC32 && PPC_PAGE_SHIFT <= 14 diff --git a/arch/powerpc/kernel/module.c b/arch/powerpc/kernel/module.c index a211b0253cdb..cdb2d88c54e7 100644 --- a/arch/powerpc/kernel/module.c +++ b/arch/powerpc/kernel/module.c @@ -8,6 +8,7 @@ #include #include #include +#include #include #include #include @@ -87,13 +88,26 @@ int module_finalize(const Elf_Ehdr *hdr, return 0; } -#ifdef MODULES_VADDR void *module_alloc(unsigned long size) { + unsigned long start = VMALLOC_START; + unsigned long end = VMALLOC_END; + +#ifdef MODULES_VADDR BUILD_BUG_ON(TASK_SIZE > MODULES_VADDR); + start = MODULES_VADDR; + end = MODULES_END; +#endif + + /* + * Don't do huge page allocations for modules yet until more testing + * is done. STRICT_MODULE_RWX may require extra work to support this + * too. + */ - return __vmalloc_node_range(size, 1, MODULES_VADDR, MODULES_END, GFP_KERNEL, - PAGE_KERNEL_EXEC, VM_FLUSH_RESET_PERMS, NUMA_NO_NODE, + return __vmalloc_node_range(size, 1, start, end, GFP_KERNEL, + PAGE_KERNEL_EXEC, + VM_NO_HUGE_VMAP | VM_FLUSH_RESET_PERMS, + NUMA_NO_NODE, __builtin_return_address(0)); } -#endif