From patchwork Tue Aug 25 14:57:42 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11735839 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7FC48722 for ; Tue, 25 Aug 2020 14:58:11 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4C00820782 for ; Tue, 25 Aug 2020 14:58:11 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="dQccofU4" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4C00820782 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 691A58E000D; Tue, 25 Aug 2020 10:58:10 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 642958D0001; Tue, 25 Aug 2020 10:58:10 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4E2B88E000D; Tue, 25 Aug 2020 10:58:10 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0098.hostedemail.com [216.40.44.98]) by kanga.kvack.org (Postfix) with ESMTP id 328428D0001 for ; Tue, 25 Aug 2020 10:58:10 -0400 (EDT) Received: from smtpin10.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id D75761EE6 for ; Tue, 25 Aug 2020 14:58:09 +0000 (UTC) X-FDA: 77189396298.10.show51_42111ce2705c Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin10.hostedemail.com (Postfix) with ESMTP id 9ADC416A07E for ; Tue, 25 Aug 2020 14:58:09 +0000 (UTC) X-Spam-Summary: 1,0,0,1a10914ff3f5df75,d41d8cd98f00b204,npiggin@gmail.com,,RULES_HIT:41:69:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1431:1437:1515:1535:1543:1711:1730:1747:1777:1792:2393:2553:2559:2562:2693:3138:3139:3140:3141:3142:3308:3355:3865:3867:3868:3871:3872:3874:4117:4250:5007:6119:6261:6653:7514:7903:9036:9413:9592:10004:11026:11232:11473:11658:11914:12043:12219:12291:12295:12296:12297:12438:12517:12519:12555:12683:12895:13161:13229:13894:14181:14394:14687:14721:21080:21433:21444:21451:21627:21666:21990:30003:30054:30062:30070:30074:30090,0,RBL:209.85.210.194:@gmail.com:.lbl8.mailshell.net-62.50.0.100 66.100.201.100;04yfpwim8ajbctmx1suqrw7poynr9opwu1uepyi5f98ekzhbpbjw988g7yuwf7q.16pf537emigd1cnfmenaarp9iquwipafthxmuuio8amwyantu1zntonahicpsyh.1-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: show51_42111ce2705c X-Filterd-Recvd-Size: 6463 Received: from mail-pf1-f194.google.com (mail-pf1-f194.google.com [209.85.210.194]) by imf49.hostedemail.com (Postfix) with ESMTP for ; Tue, 25 Aug 2020 14:58:09 +0000 (UTC) Received: by mail-pf1-f194.google.com with SMTP id u20so7565302pfn.0 for ; Tue, 25 Aug 2020 07:58:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=6FG53WANq96sGtoLSKrRzKN8EzzN/tCT8YeKUFryJxY=; b=dQccofU4xh0H65J3wLVfbSsGxfXEbG4tZaPJjU1doX6AedshRyYkztWc2h50Qm3z93 KgscBr6qG9Fo44eG2OeRT2QZf2KmnmSNJ3lJqfRXlwMxr28f4//s+/k+WSkzhJG+zZXq 7e1A8Xe1Vb3oektoAvwsrr1HG/8T9+BhGHxeBxqhkUW/QAL+qYsEJzsjQGDLV40ToZlb cFFecAKq8Dmuuzx1yFyxfPa/KmQyWvfZIEsC7eIW5afSBgJPUE4uZ6p89M7FoGQqUVAQ TCo3qp5Y+8G9k+LVGo8fN1/xRq//BUSFIL+BRNmAN8TwMnHLloOkZCcwYUYknVes0/b3 AtTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=6FG53WANq96sGtoLSKrRzKN8EzzN/tCT8YeKUFryJxY=; b=ro5iIyNaEjDsXiTwCy/MB9wxWWqBUTu9LZcNi0ApdRK8K+IacoAadiIIt9UoIhohCw 6MBaVHvaz513KlPyCHbxQlxnRLdcDdL+TdO3DVJ8BtTfGU6vE34dssIAdfos3inAa/r5 otmCnCHnBnPc83ZGfX8E4b47UtjEPK/611f++UIfWN9suuF+n3Yh4GH3cQ8s9CvoaWw6 xPao3toC9BR1+yN6K3VsP0u+Ic2Y7CIkUf/MsWXM+jCn9BMQAZErxEcU9sOcman55OS/ uYlrQT1/53vhwiWlH3e1OlMrbFbvqntd7sv5qpSKz1IRQsxg/UnywjT5m5S3kciqL6ID VSww== X-Gm-Message-State: AOAM530io/QmnYoPuMON7/563IIYnqJAFLoTBuf6J4Az0D5h2iLromdR E5Z2O10ufIRsOkvBgtQarcErXEd8k1M= X-Google-Smtp-Source: ABdhPJxbIN9TEoKvgUhetFCwKB5gJVl/+6wG+sb6ODHZ7G+F/ibIU6KSibeJKKELCF6GtNzzgFkFGA== X-Received: by 2002:a05:6a00:1515:: with SMTP id q21mr8428082pfu.126.1598367488211; Tue, 25 Aug 2020 07:58:08 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (61-68-212-105.tpgi.com.au. [61.68.212.105]) by smtp.gmail.com with ESMTPSA id e29sm15755956pfj.92.2020.08.25.07.58.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Aug 2020 07:58:07 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org, Andrew Morton Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Zefan Li , Jonathan Cameron , Christoph Hellwig , Christophe Leroy Subject: [PATCH v7 01/12] mm/vmalloc: fix vmalloc_to_page for huge vmap mappings Date: Wed, 26 Aug 2020 00:57:42 +1000 Message-Id: <20200825145753.529284-2-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20200825145753.529284-1-npiggin@gmail.com> References: <20200825145753.529284-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 9ADC416A07E X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: vmalloc_to_page returns NULL for addresses mapped by larger pages[*]. Whether or not a vmap is huge depends on the architecture details, alignments, boot options, etc., which the caller can not be expected to know. Therefore HUGE_VMAP is a regression for vmalloc_to_page. This change teaches vmalloc_to_page about larger pages, and returns the struct page that corresponds to the offset within the large page. This makes the API agnostic to mapping implementation details. [*] As explained by commit 029c54b095995 ("mm/vmalloc.c: huge-vmap: fail gracefully on unexpected huge vmap mappings") Signed-off-by: Nicholas Piggin --- mm/vmalloc.c | 41 ++++++++++++++++++++++++++--------------- 1 file changed, 26 insertions(+), 15 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index b482d240f9a2..4e9b21adc73d 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -36,7 +36,7 @@ #include #include #include - +#include #include #include #include @@ -343,7 +343,9 @@ int is_vmalloc_or_module_addr(const void *x) } /* - * Walk a vmap address to the struct page it maps. + * Walk a vmap address to the struct page it maps. Huge vmap mappings will + * return the tail page that corresponds to the base page address, which + * matches small vmap mappings. */ struct page *vmalloc_to_page(const void *vmalloc_addr) { @@ -363,25 +365,33 @@ struct page *vmalloc_to_page(const void *vmalloc_addr) if (pgd_none(*pgd)) return NULL; + if (WARN_ON_ONCE(pgd_leaf(*pgd))) + return NULL; /* XXX: no allowance for huge pgd */ + if (WARN_ON_ONCE(pgd_bad(*pgd))) + return NULL; + p4d = p4d_offset(pgd, addr); if (p4d_none(*p4d)) return NULL; - pud = pud_offset(p4d, addr); + if (p4d_leaf(*p4d)) + return p4d_page(*p4d) + ((addr & ~P4D_MASK) >> PAGE_SHIFT); + if (WARN_ON_ONCE(p4d_bad(*p4d))) + return NULL; - /* - * Don't dereference bad PUD or PMD (below) entries. This will also - * identify huge mappings, which we may encounter on architectures - * that define CONFIG_HAVE_ARCH_HUGE_VMAP=y. Such regions will be - * identified as vmalloc addresses by is_vmalloc_addr(), but are - * not [unambiguously] associated with a struct page, so there is - * no correct value to return for them. - */ - WARN_ON_ONCE(pud_bad(*pud)); - if (pud_none(*pud) || pud_bad(*pud)) + pud = pud_offset(p4d, addr); + if (pud_none(*pud)) + return NULL; + if (pud_leaf(*pud)) + return pud_page(*pud) + ((addr & ~PUD_MASK) >> PAGE_SHIFT); + if (WARN_ON_ONCE(pud_bad(*pud))) return NULL; + pmd = pmd_offset(pud, addr); - WARN_ON_ONCE(pmd_bad(*pmd)); - if (pmd_none(*pmd) || pmd_bad(*pmd)) + if (pmd_none(*pmd)) + return NULL; + if (pmd_leaf(*pmd)) + return pmd_page(*pmd) + ((addr & ~PMD_MASK) >> PAGE_SHIFT); + if (WARN_ON_ONCE(pmd_bad(*pmd))) return NULL; ptep = pte_offset_map(pmd, addr); @@ -389,6 +399,7 @@ struct page *vmalloc_to_page(const void *vmalloc_addr) if (pte_present(pte)) page = pte_page(pte); pte_unmap(ptep); + return page; } EXPORT_SYMBOL(vmalloc_to_page); From patchwork Tue Aug 25 14:57:43 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11735841 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D1557722 for ; Tue, 25 Aug 2020 14:58:15 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9BE6620782 for ; Tue, 25 Aug 2020 14:58:15 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="vQtLeior" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9BE6620782 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id CACA48E000F; Tue, 25 Aug 2020 10:58:14 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id C5DCA8D0001; Tue, 25 Aug 2020 10:58:14 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B4BD48E000F; Tue, 25 Aug 2020 10:58:14 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0183.hostedemail.com [216.40.44.183]) by kanga.kvack.org (Postfix) with ESMTP id 9883E8D0001 for ; Tue, 25 Aug 2020 10:58:14 -0400 (EDT) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 57969181AEF23 for ; Tue, 25 Aug 2020 14:58:14 +0000 (UTC) X-FDA: 77189396508.01.nose22_3801ee12705c Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin01.hostedemail.com (Postfix) with ESMTP id 2329C100462EF for ; Tue, 25 Aug 2020 14:58:14 +0000 (UTC) X-Spam-Summary: 1,0,0,ddf028360dbeca7b,d41d8cd98f00b204,npiggin@gmail.com,,RULES_HIT:41:69:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1543:1711:1730:1747:1777:1792:2393:2559:2562:2898:3138:3139:3140:3141:3142:3355:3867:3868:3870:3872:4117:4321:5007:6119:6261:6653:7514:9413:10004:11026:11473:11658:11914:12043:12291:12296:12297:12438:12517:12519:12555:12683:12895:13255:13894:14096:14110:14181:14394:14687:14721:21080:21444:21451:21627:21666:21990:30054:30070,0,RBL:209.85.210.195:@gmail.com:.lbl8.mailshell.net-62.50.0.100 66.100.201.100;04yrekufukkswe1cyrqtt9eh4nz19oc8ha73weafds1bbqhgpowkew3iksrbkgd.n5sq39d8r3cbhitj4s7cj6gubf118u6i91ct1acxwxptzrobuwdfjzdz45qq9ww.o-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:25,LUA_SUMMARY:none X-HE-Tag: nose22_3801ee12705c X-Filterd-Recvd-Size: 6326 Received: from mail-pf1-f195.google.com (mail-pf1-f195.google.com [209.85.210.195]) by imf18.hostedemail.com (Postfix) with ESMTP for ; Tue, 25 Aug 2020 14:58:13 +0000 (UTC) Received: by mail-pf1-f195.google.com with SMTP id x25so7552279pff.4 for ; Tue, 25 Aug 2020 07:58:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=mSzVBpitc24aVru+Z5sqZmf0BwSZbFvBZc52kq4fALc=; b=vQtLeiorPiM+mBnb60jbbLATWdt1MHtxT3cLtj/RriJ8o1jdz7rkP3YQoG8mJWp8io LS8Dn6rf+7FDGByZFvEN2Fvj765mPOq6rs5PxtVpnNKAnP+HpRPZEGtxlZ4w7c+4GIgm ezgfbHhGIlzBhl9CqOSnbIsXRE5mnjIqpNFVmnXuNc6PzLygdIFXRQ5p2XVvInqQQ0lO 2yl8BqFp93HPQDWY6blFQJainUjJoxR/yKxm5FkwT715afuzUyduhXOJYRKlwe8TAjes P7xVYKEM9pYt5n5N5z4cdA2mKYCcpZb5B8iOuU36TX3QTyiEkX6JW7/V6JZZlxdFBQw7 8HRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=mSzVBpitc24aVru+Z5sqZmf0BwSZbFvBZc52kq4fALc=; b=AI498TVCHzWLTfshbcHfk1Eq/9dSMHa7MQMQc7wxriEWTrgYEA/52H+DbjiSbYy+ff 4PMTQnTtZC6/AoZuZblRLBnKYNGdK8SngpLOYJrUvZ4DiB2yCkf2ufqOIynzsidEbnVu HvJGGDRY3/Pq6V8nW2dQiiTNq/ugg9y3m+gaJsjLJOflb24nWZqhoBEJm2aJlCHWV937 kHxkYxAN1Y6DWSuksALRKIrFHr8Gd8mx9Y9G/KGaGGKCaZ224vbs47h3CBMH7MCqjshg N3fhXSuR0in0R3YErgtM6tx8JNfJZckohuknelckZ0nze7GLTn77uD6tl2SOHblVdFiU 8rTA== X-Gm-Message-State: AOAM533uW8fRTaj9tFqBmzKcDJBeZjxdk8Y4bmGmgS0c596tDf9dhweF Koj/pG45UxXKKiJC8NOZG19L1gTo3h8= X-Google-Smtp-Source: ABdhPJx8/sy89f7MYqPfGwfXfZYI+WaeFxgwudoUWCZ36ogQX+u6vunFQp0u1doW3+kcOJej62M5Pg== X-Received: by 2002:a17:902:7790:: with SMTP id o16mr7580684pll.299.1598367492650; Tue, 25 Aug 2020 07:58:12 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (61-68-212-105.tpgi.com.au. [61.68.212.105]) by smtp.gmail.com with ESMTPSA id e29sm15755956pfj.92.2020.08.25.07.58.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Aug 2020 07:58:12 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org, Andrew Morton Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Zefan Li , Jonathan Cameron , Christoph Hellwig , Christophe Leroy Subject: [PATCH v7 02/12] mm: apply_to_pte_range warn and fail if a large pte is encountered Date: Wed, 26 Aug 2020 00:57:43 +1000 Message-Id: <20200825145753.529284-3-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20200825145753.529284-1-npiggin@gmail.com> References: <20200825145753.529284-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 2329C100462EF X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: apply_to_pte_range might mistake a large pte for bad, or treat it as a page table, resulting in a crash or corruption. Add a test to warn and return error if large entries are found. Signed-off-by: Nicholas Piggin --- mm/memory.c | 60 +++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 44 insertions(+), 16 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 602f4283122f..995b2e790b79 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -2262,13 +2262,20 @@ static int apply_to_pmd_range(struct mm_struct *mm, pud_t *pud, } do { next = pmd_addr_end(addr, end); - if (create || !pmd_none_or_clear_bad(pmd)) { - err = apply_to_pte_range(mm, pmd, addr, next, fn, data, - create); - if (err) - break; + if (pmd_none(*pmd) && !create) + continue; + if (WARN_ON_ONCE(pmd_leaf(*pmd))) + return -EINVAL; + if (!pmd_none(*pmd) && WARN_ON_ONCE(pmd_bad(*pmd))) { + if (!create) + continue; + pmd_clear_bad(pmd); } + err = apply_to_pte_range(mm, pmd, addr, next, fn, data, create); + if (err) + break; } while (pmd++, addr = next, addr != end); + return err; } @@ -2289,13 +2296,20 @@ static int apply_to_pud_range(struct mm_struct *mm, p4d_t *p4d, } do { next = pud_addr_end(addr, end); - if (create || !pud_none_or_clear_bad(pud)) { - err = apply_to_pmd_range(mm, pud, addr, next, fn, data, - create); - if (err) - break; + if (pud_none(*pud) && !create) + continue; + if (WARN_ON_ONCE(pud_leaf(*pud))) + return -EINVAL; + if (!pud_none(*pud) && WARN_ON_ONCE(pud_bad(*pud))) { + if (!create) + continue; + pud_clear_bad(pud); } + err = apply_to_pmd_range(mm, pud, addr, next, fn, data, create); + if (err) + break; } while (pud++, addr = next, addr != end); + return err; } @@ -2316,13 +2330,20 @@ static int apply_to_p4d_range(struct mm_struct *mm, pgd_t *pgd, } do { next = p4d_addr_end(addr, end); - if (create || !p4d_none_or_clear_bad(p4d)) { - err = apply_to_pud_range(mm, p4d, addr, next, fn, data, - create); - if (err) - break; + if (p4d_none(*p4d) && !create) + continue; + if (WARN_ON_ONCE(p4d_leaf(*p4d))) + return -EINVAL; + if (!p4d_none(*p4d) && WARN_ON_ONCE(p4d_bad(*p4d))) { + if (!create) + continue; + p4d_clear_bad(p4d); } + err = apply_to_pud_range(mm, p4d, addr, next, fn, data, create); + if (err) + break; } while (p4d++, addr = next, addr != end); + return err; } @@ -2341,8 +2362,15 @@ static int __apply_to_page_range(struct mm_struct *mm, unsigned long addr, pgd = pgd_offset(mm, addr); do { next = pgd_addr_end(addr, end); - if (!create && pgd_none_or_clear_bad(pgd)) + if (pgd_none(*pgd) && !create) continue; + if (WARN_ON_ONCE(pgd_leaf(*pgd))) + return -EINVAL; + if (!pgd_none(*pgd) && WARN_ON_ONCE(pgd_bad(*pgd))) { + if (!create) + continue; + pgd_clear_bad(pgd); + } err = apply_to_p4d_range(mm, pgd, addr, next, fn, data, create); if (err) break; From patchwork Tue Aug 25 14:57:44 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11735843 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AA8B6722 for ; Tue, 25 Aug 2020 14:58:20 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 74A7420782 for ; Tue, 25 Aug 2020 14:58:20 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Xfp2e3Xt" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 74A7420782 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 6BA178E0010; Tue, 25 Aug 2020 10:58:19 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 66C7C8D0001; Tue, 25 Aug 2020 10:58:19 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 530888E0010; Tue, 25 Aug 2020 10:58:19 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0079.hostedemail.com [216.40.44.79]) by kanga.kvack.org (Postfix) with ESMTP id 37F1E8D0001 for ; Tue, 25 Aug 2020 10:58:19 -0400 (EDT) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id DEE348248D7C for ; Tue, 25 Aug 2020 14:58:18 +0000 (UTC) X-FDA: 77189396676.25.twig15_420b7bc2705c Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin25.hostedemail.com (Postfix) with ESMTP id A35361804E3A9 for ; Tue, 25 Aug 2020 14:58:18 +0000 (UTC) X-Spam-Summary: 1,0,0,15e39caae1f1c729,d41d8cd98f00b204,npiggin@gmail.com,,RULES_HIT:41:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1543:1711:1730:1747:1777:1792:2393:2559:2562:3138:3139:3140:3141:3142:3354:3865:3867:3872:4117:5007:6261:6653:7514:8603:9413:10004:11026:11473:11658:11914:12043:12297:12438:12517:12519:12555:12895:13161:13229:13894:14096:14181:14394:14687:14721:21080:21444:21451:21627:21666:21990:30003:30054,0,RBL:209.85.210.193:@gmail.com:.lbl8.mailshell.net-66.100.201.100 62.50.0.100;04yrz7rsmuufmggguwzmgc4bdi8kaochuuix8ukqnpyogge9qkahmqmyfz3kn5o.ay1wftg5tsr63ccox4ah67f3znc6zmf4q3cn7bkbgzn15pm77mmcrdt17h8813u.c-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: twig15_420b7bc2705c X-Filterd-Recvd-Size: 6238 Received: from mail-pf1-f193.google.com (mail-pf1-f193.google.com [209.85.210.193]) by imf21.hostedemail.com (Postfix) with ESMTP for ; Tue, 25 Aug 2020 14:58:18 +0000 (UTC) Received: by mail-pf1-f193.google.com with SMTP id k15so1082478pfc.12 for ; Tue, 25 Aug 2020 07:58:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=sv/9eriXs4YMtuUqEg5bncZ0rlcJBEbL+hLm5+QBb40=; b=Xfp2e3Xtmw0mMDWZ1ueqN80X0o3DlaU18t1Cz/zK/F/eG1y7CuKlm8YpZiO4eoodn+ pou5w1vEvYGji3W/Lm6246DlLB/o7In87TRc/AlIA0ZsHzwWyB0AsKNL0vlL6mmEPmR/ 0IFy4YCNvsAbiS9HKLsJN9yeZ1WKIzzFRSqG80JdkpP9JkQJa2GyYnw+7nlUPr8PhBrr TY8Tk3e6jBiHxthHc5rRMIvlJ2nuZy+ZQoGTKOPCVESC2mNUa1jBOftgRpfR8gVwsahg ndpNRGLeSb03UdhbXf5Oglnz1f2x6ei+zbwAPqXu52ALcrF+qkQbmNdZkrQCvBaKPAmB o9PQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=sv/9eriXs4YMtuUqEg5bncZ0rlcJBEbL+hLm5+QBb40=; b=Tp+S7rXaSFzkaLOk1xMwrRYIMNrabA+cOjAD5RyXXhvUHprWIwPPboFp8fzJ7WBipy usxntuSQ0NghIZprpX9Lv2+dRyKVRk5UW+jLI4tbGboF5fnMdbRDaDPA4T3hM6gnLVZQ OL4R+pNqQ2KwLcB25XIIwH5bUJjwgp4jxj+9MFv6MJD0/LMinMUNBRk+PGzy3mCo0u9Y ZQQIfYKboNjxncA4/y/3mFOPm8ZI+rz/9ZFKV4UCAY35ylNcd2bEYo+ZNP96pgShigRr Cza5QSHcVccHYBE+cohV7W92Dt4XqGCf/FkQ+TLvORNum9gqeMv9mzj5qPIofbQ9RPCY fprA== X-Gm-Message-State: AOAM530X4vQEuUmejzL7wYZa1K56rFwrTOsGr7ounEvYGrD24J0C6zBi 3Z6Hlp/QJUMTgRp7kgMItMbPJSpn+bU= X-Google-Smtp-Source: ABdhPJzgj9qz6s3F5P6AAw9oSEmK+WKjXVI/Yh0BhuKrx77hUIFnOe4T+XEIWRGHePjBI4hUL0PHmQ== X-Received: by 2002:a17:902:eb03:: with SMTP id l3mr5592962plb.296.1598367497180; Tue, 25 Aug 2020 07:58:17 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (61-68-212-105.tpgi.com.au. [61.68.212.105]) by smtp.gmail.com with ESMTPSA id e29sm15755956pfj.92.2020.08.25.07.58.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Aug 2020 07:58:16 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org, Andrew Morton Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Zefan Li , Jonathan Cameron , Christoph Hellwig , Christophe Leroy Subject: [PATCH v7 03/12] mm/vmalloc: rename vmap_*_range vmap_pages_*_range Date: Wed, 26 Aug 2020 00:57:44 +1000 Message-Id: <20200825145753.529284-4-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20200825145753.529284-1-npiggin@gmail.com> References: <20200825145753.529284-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: A35361804E3A9 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The vmalloc mapper operates on a struct page * array rather than a linear physical address, re-name it to make this distinction clear. Signed-off-by: Nicholas Piggin --- mm/vmalloc.c | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 4e9b21adc73d..45cd80ec7eeb 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -189,7 +189,7 @@ void unmap_kernel_range_noflush(unsigned long start, unsigned long size) arch_sync_kernel_mappings(start, end); } -static int vmap_pte_range(pmd_t *pmd, unsigned long addr, +static int vmap_pages_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, int *nr, pgtbl_mod_mask *mask) { @@ -217,7 +217,7 @@ static int vmap_pte_range(pmd_t *pmd, unsigned long addr, return 0; } -static int vmap_pmd_range(pud_t *pud, unsigned long addr, +static int vmap_pages_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, int *nr, pgtbl_mod_mask *mask) { @@ -229,13 +229,13 @@ static int vmap_pmd_range(pud_t *pud, unsigned long addr, return -ENOMEM; do { next = pmd_addr_end(addr, end); - if (vmap_pte_range(pmd, addr, next, prot, pages, nr, mask)) + if (vmap_pages_pte_range(pmd, addr, next, prot, pages, nr, mask)) return -ENOMEM; } while (pmd++, addr = next, addr != end); return 0; } -static int vmap_pud_range(p4d_t *p4d, unsigned long addr, +static int vmap_pages_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, int *nr, pgtbl_mod_mask *mask) { @@ -247,13 +247,13 @@ static int vmap_pud_range(p4d_t *p4d, unsigned long addr, return -ENOMEM; do { next = pud_addr_end(addr, end); - if (vmap_pmd_range(pud, addr, next, prot, pages, nr, mask)) + if (vmap_pages_pmd_range(pud, addr, next, prot, pages, nr, mask)) return -ENOMEM; } while (pud++, addr = next, addr != end); return 0; } -static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, +static int vmap_pages_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, pgprot_t prot, struct page **pages, int *nr, pgtbl_mod_mask *mask) { @@ -265,7 +265,7 @@ static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, return -ENOMEM; do { next = p4d_addr_end(addr, end); - if (vmap_pud_range(p4d, addr, next, prot, pages, nr, mask)) + if (vmap_pages_pud_range(p4d, addr, next, prot, pages, nr, mask)) return -ENOMEM; } while (p4d++, addr = next, addr != end); return 0; @@ -306,7 +306,7 @@ int map_kernel_range_noflush(unsigned long addr, unsigned long size, next = pgd_addr_end(addr, end); if (pgd_bad(*pgd)) mask |= PGTBL_PGD_MODIFIED; - err = vmap_p4d_range(pgd, addr, next, prot, pages, &nr, &mask); + err = vmap_pages_p4d_range(pgd, addr, next, prot, pages, &nr, &mask); if (err) return err; } while (pgd++, addr = next, addr != end); From patchwork Tue Aug 25 14:57:45 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11735845 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 524D4722 for ; Tue, 25 Aug 2020 14:58:25 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1201520782 for ; Tue, 25 Aug 2020 14:58:25 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="cQlfZOFx" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1201520782 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 30EE98E0011; Tue, 25 Aug 2020 10:58:24 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 2C3CA8D0001; Tue, 25 Aug 2020 10:58:24 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1AEC78E0011; Tue, 25 Aug 2020 10:58:24 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0141.hostedemail.com [216.40.44.141]) by kanga.kvack.org (Postfix) with ESMTP id 015F88D0001 for ; Tue, 25 Aug 2020 10:58:23 -0400 (EDT) Received: from smtpin03.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id A88A0181AEF21 for ; Tue, 25 Aug 2020 14:58:23 +0000 (UTC) X-FDA: 77189396886.03.list60_500d62d2705c Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin03.hostedemail.com (Postfix) with ESMTP id 5D2A828A4E8 for ; Tue, 25 Aug 2020 14:58:23 +0000 (UTC) X-Spam-Summary: 1,0,0,9b9b87375212bcb2,d41d8cd98f00b204,npiggin@gmail.com,,RULES_HIT:2:41:69:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1535:1605:1730:1747:1777:1792:1981:2194:2199:2393:2559:2562:2693:2898:3138:3139:3140:3141:3142:3870:3872:3876:4049:4120:4605:5007:6119:6261:6653:7514:7875:8603:9413:10004:11026:11473:11658:11914:12043:12291:12297:12438:12517:12519:12555:12683:12895:13894:14096:14110:14394:14687:21080:21444:21451:21611:21627:21666:21990:30054:30070,0,RBL:209.85.216.68:@gmail.com:.lbl8.mailshell.net-66.100.201.100 62.18.0.100;04y8o16rk98xe5r1j1ispgbdxcj76ypbn15mdcfei4rxy1xzjbaqk81xn86w7mc.bdhg4d7ts474bbqebnahn3yio4op7jorn1yebef5b3cepqwaadmz3cmmcmhgr9g.6-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: list60_500d62d2705c X-Filterd-Recvd-Size: 9333 Received: from mail-pj1-f68.google.com (mail-pj1-f68.google.com [209.85.216.68]) by imf02.hostedemail.com (Postfix) with ESMTP for ; Tue, 25 Aug 2020 14:58:22 +0000 (UTC) Received: by mail-pj1-f68.google.com with SMTP id mt12so1382922pjb.4 for ; Tue, 25 Aug 2020 07:58:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=0Y/hT12bvV3dyctfA7PHWDL72BJBoDG5OIVFWS0o/KA=; b=cQlfZOFxC9POJSsAPsU013/5SvZ8VquHCiQPYBs15lja1sRY9m0PfrBMaIg1XLZWcv p3kZ6oqN6DCIm2HvEax4UUKPiCB//PaUuvI8smd0nEuerBeL6E2tGvtPumsACkZ5ysso CDCfsLBjF8AYDsVcdnO8YD2fmQdKSaLT35JGbuQZasKci9nQJILSR1yfzQamY0bOttUB +RGvsK3qj29/8PWmXQTeNiCW2pdLS+4+/5QFudTaL8t0K/712PG54s7dYQO9AMX8Fg6C Rtkud9dnpS0h9tgeCnl+4oYIPUqQJ+ygF7KxTCgFHOmDfbGCK8Jy+0A4ZV7L0r66gU5p dY9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=0Y/hT12bvV3dyctfA7PHWDL72BJBoDG5OIVFWS0o/KA=; b=Pux7aFbVMSKTc81mAUMrOKDX7UF8fSvNzRcerHRfZ7Vl3Y4cb1r797gg2cBgAbXkmD SBLrgt3llvjnz9Dfksd6XXVlBW6Erhvjuwybx7ho0+hPGCDUJZo20EYpCaEWBe9xg6BE 9BPtS4f1aGRSl0Vf7bmDt+H9rur5ZokYUF9fJYO/7Og9fMixabrXDaCk1Bvs7eLE8ENZ xD++6Mcw7fLicGUf9Z99/Fw6lXvYvdfmxy0YlegmrsmvAcGYoTOvHax96y3+IgqIb8hK HX1sreospPHa6ohdX6YMRA1KWtKjwMyCbgdgvuScrRiXrNimPfM/5QgRex2rG5wg0FYX mc3w== X-Gm-Message-State: AOAM532aJ9gDrlBAXFa5qLozKugJTrm3klcqvDGnpvmEetZRK9t+mQpD lY53WhidNByyHm2IP3GWxqbsPmWsYQU= X-Google-Smtp-Source: ABdhPJw+i4LK7/5haXFGcH6N0zRmeZBuurVr4mpxZbjFi7Xsjr+zjEKDT0h7eZ/uzG52blM1wfwcQQ== X-Received: by 2002:a17:90b:28d:: with SMTP id az13mr2083415pjb.206.1598367501593; Tue, 25 Aug 2020 07:58:21 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (61-68-212-105.tpgi.com.au. [61.68.212.105]) by smtp.gmail.com with ESMTPSA id e29sm15755956pfj.92.2020.08.25.07.58.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Aug 2020 07:58:21 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org, Andrew Morton Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Zefan Li , Jonathan Cameron , Christoph Hellwig , Christophe Leroy Subject: [PATCH v7 04/12] mm/ioremap: rename ioremap_*_range to vmap_*_range Date: Wed, 26 Aug 2020 00:57:45 +1000 Message-Id: <20200825145753.529284-5-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20200825145753.529284-1-npiggin@gmail.com> References: <20200825145753.529284-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 5D2A828A4E8 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This will be used as a generic kernel virtual mapping function, so re-name it in preparation. Signed-off-by: Nicholas Piggin --- mm/ioremap.c | 64 +++++++++++++++++++++++++++------------------------- 1 file changed, 33 insertions(+), 31 deletions(-) diff --git a/mm/ioremap.c b/mm/ioremap.c index 5fa1ab41d152..3f4d36f9745a 100644 --- a/mm/ioremap.c +++ b/mm/ioremap.c @@ -61,9 +61,9 @@ static inline int ioremap_pud_enabled(void) { return 0; } static inline int ioremap_pmd_enabled(void) { return 0; } #endif /* CONFIG_HAVE_ARCH_HUGE_VMAP */ -static int ioremap_pte_range(pmd_t *pmd, unsigned long addr, - unsigned long end, phys_addr_t phys_addr, pgprot_t prot, - pgtbl_mod_mask *mask) +static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + pgtbl_mod_mask *mask) { pte_t *pte; u64 pfn; @@ -81,9 +81,8 @@ static int ioremap_pte_range(pmd_t *pmd, unsigned long addr, return 0; } -static int ioremap_try_huge_pmd(pmd_t *pmd, unsigned long addr, - unsigned long end, phys_addr_t phys_addr, - pgprot_t prot) +static int vmap_try_huge_pmd(pmd_t *pmd, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot) { if (!ioremap_pmd_enabled()) return 0; @@ -103,9 +102,9 @@ static int ioremap_try_huge_pmd(pmd_t *pmd, unsigned long addr, return pmd_set_huge(pmd, phys_addr, prot); } -static inline int ioremap_pmd_range(pud_t *pud, unsigned long addr, - unsigned long end, phys_addr_t phys_addr, pgprot_t prot, - pgtbl_mod_mask *mask) +static int vmap_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + pgtbl_mod_mask *mask) { pmd_t *pmd; unsigned long next; @@ -116,20 +115,19 @@ static inline int ioremap_pmd_range(pud_t *pud, unsigned long addr, do { next = pmd_addr_end(addr, end); - if (ioremap_try_huge_pmd(pmd, addr, next, phys_addr, prot)) { + if (vmap_try_huge_pmd(pmd, addr, next, phys_addr, prot)) { *mask |= PGTBL_PMD_MODIFIED; continue; } - if (ioremap_pte_range(pmd, addr, next, phys_addr, prot, mask)) + if (vmap_pte_range(pmd, addr, next, phys_addr, prot, mask)) return -ENOMEM; } while (pmd++, phys_addr += (next - addr), addr = next, addr != end); return 0; } -static int ioremap_try_huge_pud(pud_t *pud, unsigned long addr, - unsigned long end, phys_addr_t phys_addr, - pgprot_t prot) +static int vmap_try_huge_pud(pud_t *pud, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot) { if (!ioremap_pud_enabled()) return 0; @@ -149,9 +147,9 @@ static int ioremap_try_huge_pud(pud_t *pud, unsigned long addr, return pud_set_huge(pud, phys_addr, prot); } -static inline int ioremap_pud_range(p4d_t *p4d, unsigned long addr, - unsigned long end, phys_addr_t phys_addr, pgprot_t prot, - pgtbl_mod_mask *mask) +static int vmap_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + pgtbl_mod_mask *mask) { pud_t *pud; unsigned long next; @@ -162,20 +160,19 @@ static inline int ioremap_pud_range(p4d_t *p4d, unsigned long addr, do { next = pud_addr_end(addr, end); - if (ioremap_try_huge_pud(pud, addr, next, phys_addr, prot)) { + if (vmap_try_huge_pud(pud, addr, next, phys_addr, prot)) { *mask |= PGTBL_PUD_MODIFIED; continue; } - if (ioremap_pmd_range(pud, addr, next, phys_addr, prot, mask)) + if (vmap_pmd_range(pud, addr, next, phys_addr, prot, mask)) return -ENOMEM; } while (pud++, phys_addr += (next - addr), addr = next, addr != end); return 0; } -static int ioremap_try_huge_p4d(p4d_t *p4d, unsigned long addr, - unsigned long end, phys_addr_t phys_addr, - pgprot_t prot) +static int vmap_try_huge_p4d(p4d_t *p4d, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot) { if (!ioremap_p4d_enabled()) return 0; @@ -195,9 +192,9 @@ static int ioremap_try_huge_p4d(p4d_t *p4d, unsigned long addr, return p4d_set_huge(p4d, phys_addr, prot); } -static inline int ioremap_p4d_range(pgd_t *pgd, unsigned long addr, - unsigned long end, phys_addr_t phys_addr, pgprot_t prot, - pgtbl_mod_mask *mask) +static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + pgtbl_mod_mask *mask) { p4d_t *p4d; unsigned long next; @@ -208,19 +205,19 @@ static inline int ioremap_p4d_range(pgd_t *pgd, unsigned long addr, do { next = p4d_addr_end(addr, end); - if (ioremap_try_huge_p4d(p4d, addr, next, phys_addr, prot)) { + if (vmap_try_huge_p4d(p4d, addr, next, phys_addr, prot)) { *mask |= PGTBL_P4D_MODIFIED; continue; } - if (ioremap_pud_range(p4d, addr, next, phys_addr, prot, mask)) + if (vmap_pud_range(p4d, addr, next, phys_addr, prot, mask)) return -ENOMEM; } while (p4d++, phys_addr += (next - addr), addr = next, addr != end); return 0; } -int ioremap_page_range(unsigned long addr, - unsigned long end, phys_addr_t phys_addr, pgprot_t prot) +static int vmap_range(unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot) { pgd_t *pgd; unsigned long start; @@ -235,8 +232,7 @@ int ioremap_page_range(unsigned long addr, pgd = pgd_offset_k(addr); do { next = pgd_addr_end(addr, end); - err = ioremap_p4d_range(pgd, addr, next, phys_addr, prot, - &mask); + err = vmap_p4d_range(pgd, addr, next, phys_addr, prot, &mask); if (err) break; } while (pgd++, phys_addr += (next - addr), addr = next, addr != end); @@ -249,6 +245,12 @@ int ioremap_page_range(unsigned long addr, return err; } +int ioremap_page_range(unsigned long addr, + unsigned long end, phys_addr_t phys_addr, pgprot_t prot) +{ + return vmap_range(addr, end, phys_addr, prot); +} + #ifdef CONFIG_GENERIC_IOREMAP void __iomem *ioremap_prot(phys_addr_t addr, size_t size, unsigned long prot) { From patchwork Tue Aug 25 14:57:46 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11735847 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 85284722 for ; Tue, 25 Aug 2020 14:58:32 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 38CBB20782 for ; Tue, 25 Aug 2020 14:58:32 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="GAUBvlTx" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 38CBB20782 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 582138E0012; Tue, 25 Aug 2020 10:58:31 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 50C518D0001; Tue, 25 Aug 2020 10:58:31 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3F87C8E0012; Tue, 25 Aug 2020 10:58:31 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0071.hostedemail.com [216.40.44.71]) by kanga.kvack.org (Postfix) with ESMTP id 25FD98D0001 for ; Tue, 25 Aug 2020 10:58:31 -0400 (EDT) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id CB56D180ACF61 for ; Tue, 25 Aug 2020 14:58:30 +0000 (UTC) X-FDA: 77189397180.12.smash56_500dfd12705c Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin12.hostedemail.com (Postfix) with ESMTP id 648F518013E0D for ; Tue, 25 Aug 2020 14:58:30 +0000 (UTC) X-Spam-Summary: 1,0,0,45e885b66d7c30ec,d41d8cd98f00b204,npiggin@gmail.com,,RULES_HIT:4:41:69:355:379:541:800:960:966:968:973:988:989:1260:1311:1314:1345:1359:1431:1437:1515:1605:1730:1747:1777:1792:1801:2194:2196:2199:2200:2393:2559:2562:2639:2693:2898:3138:3139:3140:3141:3142:3167:3865:3866:3867:3868:3870:3871:4321:4385:4605:5007:6119:6261:6653:6742:7514:7875:8603:8660:9413:9592:10004:11026:11232:11473:11657:11658:11914:12043:12296:12297:12438:12517:12519:12555:12895:12986:13148:13230:13846:13894:14096:14394:14687:21080:21444:21451:21627:21666:21939:21966:21990:30012:30054:30070,0,RBL:209.85.210.195:@gmail.com:.lbl8.mailshell.net-66.100.201.100 62.50.0.100;04yreqhcwrshhccu1uc1i9bkxfpf6yc6187wn9pemjmja8a7q8tybtg8zikeqm6.fuwez7iqj1opey5oc9bds8rtndbdbpbbuij84u4fptzga3yz9yeujagqd91etsf.w-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:none,Custom_rules:0:0:0,LFtime:25,LUA_SUMMARY:none X-HE-Tag: smash56_500dfd12705c X-Filterd-Recvd-Size: 17352 Received: from mail-pf1-f195.google.com (mail-pf1-f195.google.com [209.85.210.195]) by imf40.hostedemail.com (Postfix) with ESMTP for ; Tue, 25 Aug 2020 14:58:29 +0000 (UTC) Received: by mail-pf1-f195.google.com with SMTP id t185so4440576pfd.13 for ; Tue, 25 Aug 2020 07:58:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=PoWZjfd9+LgKKwcwd/njX5fpO4STl9e4RnEY5y1S8j4=; b=GAUBvlTxy2A9BnS5ZutI8oNgc1UucAOxAr6pjScgXj537JKPb/P+AN4zkpWmUslPVv KeAu1QHH74L50dD3zuON5DE4zJDQ1XETD7rQUBjtBlejDQFS/17pyzT18QW5+QWpj027 OTp11JRuzSijnMU7y4j9ZkuE7oWhSUZzKT84vmtKGLQ8RRj4V08oPpB8xZ1YGxrUj5hr KYDmCUoIt6iwasFSr+o2CiZrRLR/szrTTas1tNoi6ouITdiNnnKrrEbZ9nleZie6ueVM EswVlgd2MdmX+Lim4FHgoaRM/uqJzi52ypnSZQQKaw/gpLHVfjORhy1n9JX6eWlrz1KQ zn3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=PoWZjfd9+LgKKwcwd/njX5fpO4STl9e4RnEY5y1S8j4=; b=PwENVdD9zB45w9srmfdjLItmy3zGBX/AqNiANL0AILInFfMgn8JCZvi92mp72EvF+Z Lycq3O6imBukFQGzS9PuMh6qlmb+R26UAKr6pmqaynXH8g/OXxrMDWYB0ZkTaGQH707K FxOHZZHvghm/mY4WLm78f8wIK5UgL+QOnppxX+H1S5TAIyGB+AXRhRrcudnbWSP/LBTt mwqQ2BhlZ6ufWbgTnRM2/s3e1ZOBuXW5J8xCRP7VGagsM2f9ZRAfvXBESeOTBHWI5GlD skegS9/wqdXMTYIETDNFner53WBjjUE7tXn+2U54RXr2sLk1ZV6HvTXvzAFiGGBt3X3M AbpQ== X-Gm-Message-State: AOAM533C1Boo4yewqCpC1lyJCZSUFtaBOGbUe0/ekD+xsI4+G0ubg0ec RtCKcn/3rrJEeqDllhuImOldvC8hA+E= X-Google-Smtp-Source: ABdhPJxciB6Foo8ZH9/2nA6YwZdwto57DtX1Azbp16OcKyr/UBX7pMIRifOCukZ277dSTR5A5TvAEg== X-Received: by 2002:aa7:8e9c:: with SMTP id a28mr933285pfr.20.1598367508687; Tue, 25 Aug 2020 07:58:28 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (61-68-212-105.tpgi.com.au. [61.68.212.105]) by smtp.gmail.com with ESMTPSA id e29sm15755956pfj.92.2020.08.25.07.58.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Aug 2020 07:58:28 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org, Andrew Morton Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Zefan Li , Jonathan Cameron , Christoph Hellwig , Christophe Leroy , Catalin Marinas , Will Deacon , linux-arm-kernel@lists.infradead.org, Thomas Gleixner , Ingo Molnar , Borislav Petkov , x86@kernel.org, "H. Peter Anvin" Subject: [PATCH v7 05/12] mm: HUGE_VMAP arch support cleanup Date: Wed, 26 Aug 2020 00:57:46 +1000 Message-Id: <20200825145753.529284-6-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20200825145753.529284-1-npiggin@gmail.com> References: <20200825145753.529284-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 648F518013E0D X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This changes the awkward approach where architectures provide init functions to determine which levels they can provide large mappings for, to one where the arch is queried for each call. This removes code and indirection, and allows constant-folding of dead code for unsupported levels. This also adds a prot argument to the arch query. This is unused currently but could help with some architectures (e.g., some powerpc processors can't map uncacheable memory with large pages). Cc: linuxppc-dev@lists.ozlabs.org Cc: Catalin Marinas Cc: Will Deacon Cc: linux-arm-kernel@lists.infradead.org Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: x86@kernel.org Cc: "H. Peter Anvin" Signed-off-by: Nicholas Piggin --- Ack or objection from arch maintainers if this goes via the -mm tree? arch/arm64/include/asm/vmalloc.h | 8 +++ arch/arm64/mm/mmu.c | 10 +-- arch/powerpc/include/asm/vmalloc.h | 8 +++ arch/powerpc/mm/book3s64/radix_pgtable.c | 8 +-- arch/x86/include/asm/vmalloc.h | 7 ++ arch/x86/mm/ioremap.c | 10 +-- include/linux/io.h | 9 --- include/linux/vmalloc.h | 6 ++ init/main.c | 1 - mm/ioremap.c | 88 +++++++++--------------- 10 files changed, 77 insertions(+), 78 deletions(-) diff --git a/arch/arm64/include/asm/vmalloc.h b/arch/arm64/include/asm/vmalloc.h index 2ca708ab9b20..597b40405319 100644 --- a/arch/arm64/include/asm/vmalloc.h +++ b/arch/arm64/include/asm/vmalloc.h @@ -1,4 +1,12 @@ #ifndef _ASM_ARM64_VMALLOC_H #define _ASM_ARM64_VMALLOC_H +#include + +#ifdef CONFIG_HAVE_ARCH_HUGE_VMAP +bool arch_vmap_p4d_supported(pgprot_t prot); +bool arch_vmap_pud_supported(pgprot_t prot); +bool arch_vmap_pmd_supported(pgprot_t prot); +#endif + #endif /* _ASM_ARM64_VMALLOC_H */ diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 75df62fea1b6..9df7e0058c78 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -1304,12 +1304,12 @@ void *__init fixmap_remap_fdt(phys_addr_t dt_phys, int *size, pgprot_t prot) return dt_virt; } -int __init arch_ioremap_p4d_supported(void) +bool arch_vmap_p4d_supported(pgprot_t prot) { - return 0; + return false; } -int __init arch_ioremap_pud_supported(void) +bool arch_vmap_pud_supported(pgprot_t prot); { /* * Only 4k granule supports level 1 block mappings. @@ -1319,9 +1319,9 @@ int __init arch_ioremap_pud_supported(void) !IS_ENABLED(CONFIG_PTDUMP_DEBUGFS); } -int __init arch_ioremap_pmd_supported(void) +bool arch_vmap_pmd_supported(pgprot_t prot) { - /* See arch_ioremap_pud_supported() */ + /* See arch_vmap_pud_supported() */ return !IS_ENABLED(CONFIG_PTDUMP_DEBUGFS); } diff --git a/arch/powerpc/include/asm/vmalloc.h b/arch/powerpc/include/asm/vmalloc.h index b992dfaaa161..105abb73f075 100644 --- a/arch/powerpc/include/asm/vmalloc.h +++ b/arch/powerpc/include/asm/vmalloc.h @@ -1,4 +1,12 @@ #ifndef _ASM_POWERPC_VMALLOC_H #define _ASM_POWERPC_VMALLOC_H +#include + +#ifdef CONFIG_HAVE_ARCH_HUGE_VMAP +bool arch_vmap_p4d_supported(pgprot_t prot); +bool arch_vmap_pud_supported(pgprot_t prot); +bool arch_vmap_pmd_supported(pgprot_t prot); +#endif + #endif /* _ASM_POWERPC_VMALLOC_H */ diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c index 28c784976bed..eca83a50bf2e 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -1134,13 +1134,13 @@ void radix__ptep_modify_prot_commit(struct vm_area_struct *vma, set_pte_at(mm, addr, ptep, pte); } -int __init arch_ioremap_pud_supported(void) +bool arch_vmap_pud_supported(pgprot_t prot) { /* HPT does not cope with large pages in the vmalloc area */ return radix_enabled(); } -int __init arch_ioremap_pmd_supported(void) +bool arch_vmap_pmd_supported(pgprot_t prot) { return radix_enabled(); } @@ -1234,7 +1234,7 @@ int pmd_free_pte_page(pmd_t *pmd, unsigned long addr) return 1; } -int __init arch_ioremap_p4d_supported(void) +bool arch_vmap_p4d_supported(pgprot_t prot) { - return 0; + return false; } diff --git a/arch/x86/include/asm/vmalloc.h b/arch/x86/include/asm/vmalloc.h index 29837740b520..094ea2b565f3 100644 --- a/arch/x86/include/asm/vmalloc.h +++ b/arch/x86/include/asm/vmalloc.h @@ -1,6 +1,13 @@ #ifndef _ASM_X86_VMALLOC_H #define _ASM_X86_VMALLOC_H +#include #include +#ifdef CONFIG_HAVE_ARCH_HUGE_VMAP +bool arch_vmap_p4d_supported(pgprot_t prot); +bool arch_vmap_pud_supported(pgprot_t prot); +bool arch_vmap_pmd_supported(pgprot_t prot); +#endif + #endif /* _ASM_X86_VMALLOC_H */ diff --git a/arch/x86/mm/ioremap.c b/arch/x86/mm/ioremap.c index 84d85dbd1dad..159bfca757b9 100644 --- a/arch/x86/mm/ioremap.c +++ b/arch/x86/mm/ioremap.c @@ -481,21 +481,21 @@ void iounmap(volatile void __iomem *addr) } EXPORT_SYMBOL(iounmap); -int __init arch_ioremap_p4d_supported(void) +bool arch_vmap_p4d_supported(pgprot_t prot) { - return 0; + return false; } -int __init arch_ioremap_pud_supported(void) +bool arch_vmap_pud_supported(pgprot_t prot) { #ifdef CONFIG_X86_64 return boot_cpu_has(X86_FEATURE_GBPAGES); #else - return 0; + return false; #endif } -int __init arch_ioremap_pmd_supported(void) +bool arch_vmap_pmd_supported(pgprot_t prot) { return boot_cpu_has(X86_FEATURE_PSE); } diff --git a/include/linux/io.h b/include/linux/io.h index 8394c56babc2..f1effd4d7a3c 100644 --- a/include/linux/io.h +++ b/include/linux/io.h @@ -31,15 +31,6 @@ static inline int ioremap_page_range(unsigned long addr, unsigned long end, } #endif -#ifdef CONFIG_HAVE_ARCH_HUGE_VMAP -void __init ioremap_huge_init(void); -int arch_ioremap_p4d_supported(void); -int arch_ioremap_pud_supported(void); -int arch_ioremap_pmd_supported(void); -#else -static inline void ioremap_huge_init(void) { } -#endif - /* * Managed iomap interface */ diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 0221f852a7e1..3f6bba4cc9bc 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -84,6 +84,12 @@ struct vmap_area { }; }; +#ifndef CONFIG_HAVE_ARCH_HUGE_VMAP +static inline bool arch_vmap_p4d_supported(pgprot_t prot) { return false; } +static inline bool arch_vmap_pud_supported(pgprot_t prot) { return false; } +static inline bool arch_vmap_pmd_supported(pgprot_t prot) { return false; } +#endif + /* * Highlevel APIs for driver use */ diff --git a/init/main.c b/init/main.c index ae78fb68d231..1c89aa127b8f 100644 --- a/init/main.c +++ b/init/main.c @@ -820,7 +820,6 @@ static void __init mm_init(void) pgtable_init(); debug_objects_mem_init(); vmalloc_init(); - ioremap_huge_init(); /* Should be run before the first non-init thread is created */ init_espfix_bsp(); /* Should be run after espfix64 is set up. */ diff --git a/mm/ioremap.c b/mm/ioremap.c index 3f4d36f9745a..c67f91164401 100644 --- a/mm/ioremap.c +++ b/mm/ioremap.c @@ -16,49 +16,16 @@ #include "pgalloc-track.h" #ifdef CONFIG_HAVE_ARCH_HUGE_VMAP -static int __read_mostly ioremap_p4d_capable; -static int __read_mostly ioremap_pud_capable; -static int __read_mostly ioremap_pmd_capable; -static int __read_mostly ioremap_huge_disabled; +static bool __ro_after_init iomap_max_page_shift = PAGE_SHIFT; static int __init set_nohugeiomap(char *str) { - ioremap_huge_disabled = 1; + iomap_max_page_shift = P4D_SHIFT; return 0; } early_param("nohugeiomap", set_nohugeiomap); - -void __init ioremap_huge_init(void) -{ - if (!ioremap_huge_disabled) { - if (arch_ioremap_p4d_supported()) - ioremap_p4d_capable = 1; - if (arch_ioremap_pud_supported()) - ioremap_pud_capable = 1; - if (arch_ioremap_pmd_supported()) - ioremap_pmd_capable = 1; - } -} - -static inline int ioremap_p4d_enabled(void) -{ - return ioremap_p4d_capable; -} - -static inline int ioremap_pud_enabled(void) -{ - return ioremap_pud_capable; -} - -static inline int ioremap_pmd_enabled(void) -{ - return ioremap_pmd_capable; -} - -#else /* !CONFIG_HAVE_ARCH_HUGE_VMAP */ -static inline int ioremap_p4d_enabled(void) { return 0; } -static inline int ioremap_pud_enabled(void) { return 0; } -static inline int ioremap_pmd_enabled(void) { return 0; } +#else /* CONFIG_HAVE_ARCH_HUGE_VMAP */ +static const bool iomap_max_page_shift = PAGE_SHIFT; #endif /* CONFIG_HAVE_ARCH_HUGE_VMAP */ static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, @@ -82,9 +49,13 @@ static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, } static int vmap_try_huge_pmd(pmd_t *pmd, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot) + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift) { - if (!ioremap_pmd_enabled()) + if (max_page_shift < PMD_SHIFT) + return 0; + + if (!arch_vmap_pmd_supported(prot)) return 0; if ((end - addr) != PMD_SIZE) @@ -104,7 +75,7 @@ static int vmap_try_huge_pmd(pmd_t *pmd, unsigned long addr, unsigned long end, static int vmap_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, phys_addr_t phys_addr, pgprot_t prot, - pgtbl_mod_mask *mask) + unsigned int max_page_shift, pgtbl_mod_mask *mask) { pmd_t *pmd; unsigned long next; @@ -115,7 +86,7 @@ static int vmap_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, do { next = pmd_addr_end(addr, end); - if (vmap_try_huge_pmd(pmd, addr, next, phys_addr, prot)) { + if (vmap_try_huge_pmd(pmd, addr, next, phys_addr, prot, max_page_shift)) { *mask |= PGTBL_PMD_MODIFIED; continue; } @@ -127,9 +98,13 @@ static int vmap_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, } static int vmap_try_huge_pud(pud_t *pud, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot) + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift) { - if (!ioremap_pud_enabled()) + if (max_page_shift < PUD_SHIFT) + return 0; + + if (!arch_vmap_pud_supported(prot)) return 0; if ((end - addr) != PUD_SIZE) @@ -149,7 +124,7 @@ static int vmap_try_huge_pud(pud_t *pud, unsigned long addr, unsigned long end, static int vmap_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end, phys_addr_t phys_addr, pgprot_t prot, - pgtbl_mod_mask *mask) + unsigned int max_page_shift, pgtbl_mod_mask *mask) { pud_t *pud; unsigned long next; @@ -160,21 +135,25 @@ static int vmap_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end, do { next = pud_addr_end(addr, end); - if (vmap_try_huge_pud(pud, addr, next, phys_addr, prot)) { + if (vmap_try_huge_pud(pud, addr, next, phys_addr, prot, max_page_shift)) { *mask |= PGTBL_PUD_MODIFIED; continue; } - if (vmap_pmd_range(pud, addr, next, phys_addr, prot, mask)) + if (vmap_pmd_range(pud, addr, next, phys_addr, prot, max_page_shift, mask)) return -ENOMEM; } while (pud++, phys_addr += (next - addr), addr = next, addr != end); return 0; } static int vmap_try_huge_p4d(p4d_t *p4d, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot) + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift) { - if (!ioremap_p4d_enabled()) + if (max_page_shift < P4D_SHIFT) + return 0; + + if (!arch_vmap_p4d_supported(prot)) return 0; if ((end - addr) != P4D_SIZE) @@ -194,7 +173,7 @@ static int vmap_try_huge_p4d(p4d_t *p4d, unsigned long addr, unsigned long end, static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, phys_addr_t phys_addr, pgprot_t prot, - pgtbl_mod_mask *mask) + unsigned int max_page_shift, pgtbl_mod_mask *mask) { p4d_t *p4d; unsigned long next; @@ -205,19 +184,20 @@ static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, do { next = p4d_addr_end(addr, end); - if (vmap_try_huge_p4d(p4d, addr, next, phys_addr, prot)) { + if (vmap_try_huge_p4d(p4d, addr, next, phys_addr, prot, max_page_shift)) { *mask |= PGTBL_P4D_MODIFIED; continue; } - if (vmap_pud_range(p4d, addr, next, phys_addr, prot, mask)) + if (vmap_pud_range(p4d, addr, next, phys_addr, prot, max_page_shift, mask)) return -ENOMEM; } while (p4d++, phys_addr += (next - addr), addr = next, addr != end); return 0; } static int vmap_range(unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot) + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift) { pgd_t *pgd; unsigned long start; @@ -232,7 +212,7 @@ static int vmap_range(unsigned long addr, unsigned long end, pgd = pgd_offset_k(addr); do { next = pgd_addr_end(addr, end); - err = vmap_p4d_range(pgd, addr, next, phys_addr, prot, &mask); + err = vmap_p4d_range(pgd, addr, next, phys_addr, prot, max_page_shift, &mask); if (err) break; } while (pgd++, phys_addr += (next - addr), addr = next, addr != end); @@ -248,7 +228,7 @@ static int vmap_range(unsigned long addr, unsigned long end, int ioremap_page_range(unsigned long addr, unsigned long end, phys_addr_t phys_addr, pgprot_t prot) { - return vmap_range(addr, end, phys_addr, prot); + return vmap_range(addr, end, phys_addr, prot, iomap_max_page_shift); } #ifdef CONFIG_GENERIC_IOREMAP From patchwork Tue Aug 25 14:57:47 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11735849 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D49D6138A for ; Tue, 25 Aug 2020 14:58:36 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A24BD20782 for ; Tue, 25 Aug 2020 14:58:36 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="uDNHH8Kh" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A24BD20782 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9C65B8D0006; Tue, 25 Aug 2020 10:58:35 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 9777D8D0001; Tue, 25 Aug 2020 10:58:35 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 866088D0006; Tue, 25 Aug 2020 10:58:35 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0019.hostedemail.com [216.40.44.19]) by kanga.kvack.org (Postfix) with ESMTP id 6B76E8D0001 for ; Tue, 25 Aug 2020 10:58:35 -0400 (EDT) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 1FEAF180AD806 for ; Tue, 25 Aug 2020 14:58:35 +0000 (UTC) X-FDA: 77189397390.12.cork70_5c0c8492705c Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin12.hostedemail.com (Postfix) with ESMTP id D80971800BEA1 for ; Tue, 25 Aug 2020 14:58:34 +0000 (UTC) X-Spam-Summary: 1,0,0,e438c4b0f771e722,d41d8cd98f00b204,npiggin@gmail.com,,RULES_HIT:41:69:355:379:541:800:960:966:968:973:988:989:1260:1311:1314:1345:1359:1431:1437:1515:1535:1542:1711:1730:1747:1777:1792:2196:2198:2199:2200:2393:2559:2562:3138:3139:3140:3141:3142:3353:3865:3867:3868:3870:3872:4321:4385:4605:5007:6119:6261:6653:7514:9413:9592:10004:11026:11232:11473:11657:11658:11914:12043:12297:12438:12517:12519:12555:12895:12986:13894:14096:14181:14394:14687:14721:21080:21444:21451:21627:21666:21990:30054,0,RBL:209.85.215.193:@gmail.com:.lbl8.mailshell.net-66.100.201.100 62.50.0.100;04yfcdhmbk7eihbasytoyf6m4gk7uocpwugqswer5zrawg36q1u1gyyp7b4iatg.gu4xc9pjbpgu9mw9ys8rqkq4z17cs8ifh4axh5dmiktyhsmxrdmeon54oecgztg.o-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: cork70_5c0c8492705c X-Filterd-Recvd-Size: 5540 Received: from mail-pg1-f193.google.com (mail-pg1-f193.google.com [209.85.215.193]) by imf40.hostedemail.com (Postfix) with ESMTP for ; Tue, 25 Aug 2020 14:58:34 +0000 (UTC) Received: by mail-pg1-f193.google.com with SMTP id w186so4898398pgb.8 for ; Tue, 25 Aug 2020 07:58:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=l67Z2dzho9T2ly5R7eUcCIHpsBNVy2xJSWIdCCUuaqc=; b=uDNHH8KhZtI7x0oo3/DGvUVgj15j0EFWeWkpDYBKCLJKsOMtuQ1xtfIlg3G2AhdBIn CYJgB2BvAfJMB4LL/pc1kFgvcDgNgqiXBYeStk8noBxkFGfXHwcn8jVawe4QiICKcdN1 T4XRPdl431VQo65ffcLocq0P2XPLAjeYshZGsAuExgdni/73Sg156xvcwRekGlVWJE2o AznQRe/Gzv5OMNjMBjT2abU3ehYKwFcnsxjKAURf/zpJ51iHpE+VyEuioFKDyhFwybyk ICUmDfnanBvbNsP5GjLLst5UsdUPbiQXprIPBIH4QHAFO1CzLtHvj7cNT80JD+I/PVED g5vQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=l67Z2dzho9T2ly5R7eUcCIHpsBNVy2xJSWIdCCUuaqc=; b=mHaStht7KH4U8WaAooBhuHd69rfL9ld7zZS9ofLtyLLN+fnhq/ksYlYbobdHefx8dm Ii1URhrE7hJ/DTbChqs6xgRcfNDLZ4SctDjBxNxWqtVJLsXyUlJvv5HHQAKkVhzcJ5ED CFblzJx4dpSrb+z5jWnqA8nR5N+P8s/b0XNSh0Iiy914m4BeX4jt0rnpSppKGjmUND7Q LNRACJT7lzmLLFb0oqxq6QSBHFE6+y8pSa90WKPN/uklsIMeEZ8cpBSZQzWeLfVfvojl M4JftZPXzQln/T+pCBr4UCwwgHSPdeiD/sC80dHp39BZFplfT45dwIp7GuOfU646PDrz We4A== X-Gm-Message-State: AOAM532AORsocTUle6hR6Q4NU/AOCXC4sINTwY7uopiLvuGw4yTkUzH3 A+0zndQTvKxzudlxefx0pilQEVK3Q8A= X-Google-Smtp-Source: ABdhPJzLMJzy9eSMALAieo2xYeCw4xNiaeoXz3Fa/uIpayqK5UpBelchAC7yK+azH4/CJat9OKGjQg== X-Received: by 2002:a17:902:407:: with SMTP id 7mr8173396ple.167.1598367513105; Tue, 25 Aug 2020 07:58:33 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (61-68-212-105.tpgi.com.au. [61.68.212.105]) by smtp.gmail.com with ESMTPSA id e29sm15755956pfj.92.2020.08.25.07.58.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Aug 2020 07:58:32 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org, Andrew Morton Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Zefan Li , Jonathan Cameron , Christoph Hellwig , Christophe Leroy Subject: [PATCH v7 06/12] powerpc: inline huge vmap supported functions Date: Wed, 26 Aug 2020 00:57:47 +1000 Message-Id: <20200825145753.529284-7-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20200825145753.529284-1-npiggin@gmail.com> References: <20200825145753.529284-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: D80971800BEA1 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This allows unsupported levels to be constant folded away, and so p4d_free_pud_page can be removed because it's no longer linked to. Cc: linuxppc-dev@lists.ozlabs.org Signed-off-by: Nicholas Piggin Acked-by: Michael Ellerman --- Ack or objection if this goes via the -mm tree? arch/powerpc/include/asm/vmalloc.h | 19 ++++++++++++++++--- arch/powerpc/mm/book3s64/radix_pgtable.c | 21 --------------------- 2 files changed, 16 insertions(+), 24 deletions(-) diff --git a/arch/powerpc/include/asm/vmalloc.h b/arch/powerpc/include/asm/vmalloc.h index 105abb73f075..3f0c153befb0 100644 --- a/arch/powerpc/include/asm/vmalloc.h +++ b/arch/powerpc/include/asm/vmalloc.h @@ -1,12 +1,25 @@ #ifndef _ASM_POWERPC_VMALLOC_H #define _ASM_POWERPC_VMALLOC_H +#include #include #ifdef CONFIG_HAVE_ARCH_HUGE_VMAP -bool arch_vmap_p4d_supported(pgprot_t prot); -bool arch_vmap_pud_supported(pgprot_t prot); -bool arch_vmap_pmd_supported(pgprot_t prot); +static inline bool arch_vmap_p4d_supported(pgprot_t prot) +{ + return false; +} + +static inline bool arch_vmap_pud_supported(pgprot_t prot) +{ + /* HPT does not cope with large pages in the vmalloc area */ + return radix_enabled(); +} + +static inline bool arch_vmap_pmd_supported(pgprot_t prot) +{ + return radix_enabled(); +} #endif #endif /* _ASM_POWERPC_VMALLOC_H */ diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c index eca83a50bf2e..27f5837cf145 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -1134,22 +1134,6 @@ void radix__ptep_modify_prot_commit(struct vm_area_struct *vma, set_pte_at(mm, addr, ptep, pte); } -bool arch_vmap_pud_supported(pgprot_t prot) -{ - /* HPT does not cope with large pages in the vmalloc area */ - return radix_enabled(); -} - -bool arch_vmap_pmd_supported(pgprot_t prot) -{ - return radix_enabled(); -} - -int p4d_free_pud_page(p4d_t *p4d, unsigned long addr) -{ - return 0; -} - int pud_set_huge(pud_t *pud, phys_addr_t addr, pgprot_t prot) { pte_t *ptep = (pte_t *)pud; @@ -1233,8 +1217,3 @@ int pmd_free_pte_page(pmd_t *pmd, unsigned long addr) return 1; } - -bool arch_vmap_p4d_supported(pgprot_t prot) -{ - return false; -} From patchwork Tue Aug 25 14:57:48 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11735851 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C2553722 for ; Tue, 25 Aug 2020 14:58:41 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 8F28120782 for ; Tue, 25 Aug 2020 14:58:41 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="b0AWgWGG" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8F28120782 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 8B4918E0013; Tue, 25 Aug 2020 10:58:40 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 8673C8D0001; Tue, 25 Aug 2020 10:58:40 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7547D8E0013; Tue, 25 Aug 2020 10:58:40 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0177.hostedemail.com [216.40.44.177]) by kanga.kvack.org (Postfix) with ESMTP id 5C1E58D0001 for ; Tue, 25 Aug 2020 10:58:40 -0400 (EDT) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 1FD011EE6 for ; Tue, 25 Aug 2020 14:58:40 +0000 (UTC) X-FDA: 77189397600.19.smile31_330a6332705c Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin19.hostedemail.com (Postfix) with ESMTP id CFD951AD1AD for ; Tue, 25 Aug 2020 14:58:39 +0000 (UTC) X-Spam-Summary: 1,0,0,7d9c17bbdd7c7f95,d41d8cd98f00b204,npiggin@gmail.com,,RULES_HIT:41:69:355:379:541:800:960:966:968:973:988:989:1260:1311:1314:1345:1359:1431:1437:1515:1535:1542:1711:1730:1747:1777:1792:2196:2198:2199:2200:2393:2559:2562:2693:3138:3139:3140:3141:3142:3353:3865:3866:3867:3868:3872:4117:4321:4385:4605:5007:6261:6653:6742:7514:8603:9389:9413:9592:10004:11026:11232:11473:11657:11658:11914:12043:12291:12296:12297:12438:12517:12519:12555:12895:12986:13894:14096:14181:14394:14687:14721:21080:21444:21451:21627:21666:21990:30054,0,RBL:209.85.215.193:@gmail.com:.lbl8.mailshell.net-66.100.201.100 62.50.0.100;04yf7ciijdmoo1siw111dnon84iwuopbud8e1tz3yt5si4b97rah8q1f7hrophw.nx8nj4ma3jezddmt9odts4wcex146cwjg3175mpjk55yog6ksms9knka3wffocj.g-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:none,Custom_rules:0:0:0,LFtime:26,LUA_SUMMARY:none X-HE-Tag: smile31_330a6332705c X-Filterd-Recvd-Size: 6131 Received: from mail-pg1-f193.google.com (mail-pg1-f193.google.com [209.85.215.193]) by imf42.hostedemail.com (Postfix) with ESMTP for ; Tue, 25 Aug 2020 14:58:39 +0000 (UTC) Received: by mail-pg1-f193.google.com with SMTP id o13so7013737pgf.0 for ; Tue, 25 Aug 2020 07:58:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=M/5eWAbEDQgOkiF6H06FGKZHT/EDb26p/JCBso4Tchw=; b=b0AWgWGGUkjUbVDCI2TEdIDTZ4JsxKTUttA8MH4l5sDv3h/rE3ZTwojNnF/PYUM1Mc rChp7wjU4B40W63KYGDwvx8DMJuTlmJIWmiOvJ24kwFppMGbxEe7UsbE/BSgegHkfb3j LONhu22IlfERWDw8KCGGDDHT35FyWcfSU82h3i6KidDfX79nz7QiE46/WdC5uZ4brzxu 8hrUEomJ1vnvGYG6UeDro5fRriPjk19sdTPUR0iAB0Omt6kUP5pVWMF3b7oZncCSELwC CgmHBgVY75MA6GwJqdeihvCoBXpuhmVIXDzN41XcjOhbeUcHiyD7Ay2fGa3VscHvt8OI U6Ug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=M/5eWAbEDQgOkiF6H06FGKZHT/EDb26p/JCBso4Tchw=; b=YwG3GiQovOPdGzndxvYb299K9xqfgDF6qO+vL1YTVW7dHdrUAGtpJhc2vHmhiqvlVT E9jKkRcvSjDYNph7GoqyBkw02EQgAcBlAR/zy0ZWZHTEtFZYFnuFL4tnVp4p86xo6k83 Ij6nwxbMrz4iWM53ndFlmMyNy+DGzr0rbYqpRrxM3f30eT6egUOTNWj8siEAOXQT7Jv0 dpUDHAlpa+UVcICJykL6MhU98n5Vky1yVoN4tvyVueCoEV0FbXwFbH0apw/mPH+dKw46 kJ/dY/rKNkc/nccJvQ7up6HUXH4Yzx1JV3/IQ4izB3krF3juRm6POp7NR3vfYlMbc1KX 6lfg== X-Gm-Message-State: AOAM531mtDdE84QGIumxZmlSdBAH/FVBQ1gewAnk1PyW52dZihxjKg+T KD71gg1tk1FvwFbSUZxl9ZxHcFCwOvw= X-Google-Smtp-Source: ABdhPJyoYG5YDMJT67BmAM16vft0/MgmMdJ09oTXgsuH1MNyBThmKV5NRHMl8mSRkrVyoySYFmRl/A== X-Received: by 2002:aa7:9f92:: with SMTP id z18mr8222689pfr.260.1598367518404; Tue, 25 Aug 2020 07:58:38 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (61-68-212-105.tpgi.com.au. [61.68.212.105]) by smtp.gmail.com with ESMTPSA id e29sm15755956pfj.92.2020.08.25.07.58.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Aug 2020 07:58:38 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org, Andrew Morton Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Zefan Li , Jonathan Cameron , Christoph Hellwig , Christophe Leroy , Catalin Marinas , Will Deacon , linux-arm-kernel@lists.infradead.org Subject: [PATCH v7 07/12] arm64: inline huge vmap supported functions Date: Wed, 26 Aug 2020 00:57:48 +1000 Message-Id: <20200825145753.529284-8-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20200825145753.529284-1-npiggin@gmail.com> References: <20200825145753.529284-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: CFD951AD1AD X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This allows unsupported levels to be constant folded away, and so p4d_free_pud_page can be removed because it's no longer linked to. Cc: Catalin Marinas Cc: Will Deacon Cc: linux-arm-kernel@lists.infradead.org Signed-off-by: Nicholas Piggin Acked-by: Catalin Marinas --- Ack or objection if this goes via the -mm tree? arch/arm64/include/asm/vmalloc.h | 23 ++++++++++++++++++++--- arch/arm64/mm/mmu.c | 26 -------------------------- 2 files changed, 20 insertions(+), 29 deletions(-) diff --git a/arch/arm64/include/asm/vmalloc.h b/arch/arm64/include/asm/vmalloc.h index 597b40405319..fc9a12d6cc1a 100644 --- a/arch/arm64/include/asm/vmalloc.h +++ b/arch/arm64/include/asm/vmalloc.h @@ -4,9 +4,26 @@ #include #ifdef CONFIG_HAVE_ARCH_HUGE_VMAP -bool arch_vmap_p4d_supported(pgprot_t prot); -bool arch_vmap_pud_supported(pgprot_t prot); -bool arch_vmap_pmd_supported(pgprot_t prot); +static inline bool arch_vmap_p4d_supported(pgprot_t prot) +{ + return false; +} + +static inline bool arch_vmap_pud_supported(pgprot_t prot) +{ + /* + * Only 4k granule supports level 1 block mappings. + * SW table walks can't handle removal of intermediate entries. + */ + return IS_ENABLED(CONFIG_ARM64_4K_PAGES) && + !IS_ENABLED(CONFIG_PTDUMP_DEBUGFS); +} + +static inline bool arch_vmap_pmd_supported(pgprot_t prot) +{ + /* See arch_vmap_pud_supported() */ + return !IS_ENABLED(CONFIG_PTDUMP_DEBUGFS); +} #endif #endif /* _ASM_ARM64_VMALLOC_H */ diff --git a/arch/arm64/mm/mmu.c b/arch/arm64/mm/mmu.c index 9df7e0058c78..07093e148957 100644 --- a/arch/arm64/mm/mmu.c +++ b/arch/arm64/mm/mmu.c @@ -1304,27 +1304,6 @@ void *__init fixmap_remap_fdt(phys_addr_t dt_phys, int *size, pgprot_t prot) return dt_virt; } -bool arch_vmap_p4d_supported(pgprot_t prot) -{ - return false; -} - -bool arch_vmap_pud_supported(pgprot_t prot); -{ - /* - * Only 4k granule supports level 1 block mappings. - * SW table walks can't handle removal of intermediate entries. - */ - return IS_ENABLED(CONFIG_ARM64_4K_PAGES) && - !IS_ENABLED(CONFIG_PTDUMP_DEBUGFS); -} - -bool arch_vmap_pmd_supported(pgprot_t prot) -{ - /* See arch_vmap_pud_supported() */ - return !IS_ENABLED(CONFIG_PTDUMP_DEBUGFS); -} - int pud_set_huge(pud_t *pudp, phys_addr_t phys, pgprot_t prot) { pud_t new_pud = pfn_pud(__phys_to_pfn(phys), mk_pud_sect_prot(prot)); @@ -1416,11 +1395,6 @@ int pud_free_pmd_page(pud_t *pudp, unsigned long addr) return 1; } -int p4d_free_pud_page(p4d_t *p4d, unsigned long addr) -{ - return 0; /* Don't attempt a block mapping */ -} - #ifdef CONFIG_MEMORY_HOTPLUG static void __remove_pgd_mapping(pgd_t *pgdir, unsigned long start, u64 size) { From patchwork Tue Aug 25 14:57:49 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11735853 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D880B722 for ; Tue, 25 Aug 2020 14:58:47 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A4696207D3 for ; Tue, 25 Aug 2020 14:58:47 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ld0qeCxb" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A4696207D3 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id AAFC48E0016; Tue, 25 Aug 2020 10:58:46 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A60C08D0001; Tue, 25 Aug 2020 10:58:46 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9509D8E0016; Tue, 25 Aug 2020 10:58:46 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0138.hostedemail.com [216.40.44.138]) by kanga.kvack.org (Postfix) with ESMTP id 7F93A8D0001 for ; Tue, 25 Aug 2020 10:58:46 -0400 (EDT) Received: from smtpin23.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 3F2D11EE6 for ; Tue, 25 Aug 2020 14:58:46 +0000 (UTC) X-FDA: 77189397852.23.badge34_23042832705c Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin23.hostedemail.com (Postfix) with ESMTP id C062237606 for ; Tue, 25 Aug 2020 14:58:45 +0000 (UTC) X-Spam-Summary: 1,0,0,e05b6db9824eb6d2,d41d8cd98f00b204,npiggin@gmail.com,,RULES_HIT:41:69:355:379:541:800:960:966:968:973:988:989:1260:1311:1314:1345:1359:1431:1437:1515:1535:1543:1711:1730:1747:1777:1792:2196:2198:2199:2200:2393:2559:2562:2895:2901:3138:3139:3140:3141:3142:3167:3354:3865:3867:3868:3870:3872:3874:4117:4385:4605:5007:6119:6261:6630:6653:6742:7514:7875:7903:9413:9592:10004:11026:11232:11473:11657:11658:11914:12043:12297:12438:12517:12519:12555:12895:13846:13894:14096:14181:14394:14687:14721:21080:21433:21444:21451:21627:21666:21990:30054:30070,0,RBL:209.85.210.195:@gmail.com:.lbl8.mailshell.net-62.50.0.100 66.100.201.100;04yrxy4x63xe8bjrshjueugnrzqpmycrgtpyz7af8mhexb7rpta9amkdwhqfeqd.rads6qqbbsakmwbwcjrq1jxpaj7utsrkdyrbs7parsd1kjgs87pwkoued3trgi1.o-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:none,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: badge34_23042832705c X-Filterd-Recvd-Size: 6450 Received: from mail-pf1-f195.google.com (mail-pf1-f195.google.com [209.85.210.195]) by imf14.hostedemail.com (Postfix) with ESMTP for ; Tue, 25 Aug 2020 14:58:45 +0000 (UTC) Received: by mail-pf1-f195.google.com with SMTP id m8so7566376pfh.3 for ; Tue, 25 Aug 2020 07:58:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=aQ1WXKhj5a18kdRmgSAyN5y+pR2QoRFll6u3oKCjOTs=; b=ld0qeCxbyfv+pIE1p9tMPi79FDatikVJ43DF5LbwBuNhVgpCDbF06GfECbrvsMP/rn j+9XBpaxT9i2UKZttKNrNGhxQ5lYz9qbzrdVtFq8EPll0QUhyKnG3TBNWuP5AAMgT0Z0 hcSRJXoE92dTPWqLbPVZu6V1RBvpNfMIMizISs/hwRqKOZ/uRUpmu2owYRuxktcPqJeo oL0zn6YvRj8eQU3/ys+70jRTBX7wIDQgr1OMzdCEnfXProl815qfC1bH2BGFW0+n/qNl zzVaksNuuUNfvcQDGz15exv/e/ZOcP4e+A5tRdVOtJd2sJUjt833+lmaWXG0LE3k5jWt aUVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=aQ1WXKhj5a18kdRmgSAyN5y+pR2QoRFll6u3oKCjOTs=; b=mjibtbDockj7F0hFB6/RbNVztMUbgPYuizFV+/hUfkialuL5QJK3UcgMVVSLHhWxk7 1vHiIBHk1OqZLMU0ewpf5OEp1u/vJQqG33qt9q+X1FOoDOEsd2rx+iRw411VCDw/XkEB 87JwMAGQ5usw7UAkBgugTnLunecpLp6tF+w9+EBlBlJdlx5zt6bdiEwBEuribDMEH4/Z 9mfVYw3+jy0K7ETLGHz8EhIhwKu2trzxJlRsQDqJj6Se+Y7607Zk8Gcv46AEd4vbI29H VvqP03ZaiTg1o5rne3Tn9LSalcNOJgn28HIGj6YubiE84hLoE1qI36cZ4llK71OiG+Rz QiUQ== X-Gm-Message-State: AOAM531i05KfMcet/SspWdnj7jpK/KvkSPUl0jEKN9Ja50XWvDo6PAdb 2KmzeHc6RPEMx3vVqtkH4vG3LPYeinE= X-Google-Smtp-Source: ABdhPJxgyt6v8I7ydZGOFbxNzJf2nKN/r4HPzy9rkAa1+avUfozgoPS8wf2IGvii68NG9fSXHus0bg== X-Received: by 2002:a63:7c9:: with SMTP id 192mr6935394pgh.181.1598367524426; Tue, 25 Aug 2020 07:58:44 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (61-68-212-105.tpgi.com.au. [61.68.212.105]) by smtp.gmail.com with ESMTPSA id e29sm15755956pfj.92.2020.08.25.07.58.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Aug 2020 07:58:44 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org, Andrew Morton Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Zefan Li , Jonathan Cameron , Christoph Hellwig , Christophe Leroy , Thomas Gleixner , Ingo Molnar , Borislav Petkov , x86@kernel.org, "H. Peter Anvin" Subject: [PATCH v7 08/12] x86: inline huge vmap supported functions Date: Wed, 26 Aug 2020 00:57:49 +1000 Message-Id: <20200825145753.529284-9-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20200825145753.529284-1-npiggin@gmail.com> References: <20200825145753.529284-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: C062237606 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This allows unsupported levels to be constant folded away, and so p4d_free_pud_page can be removed because it's no longer linked to. Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: x86@kernel.org Cc: "H. Peter Anvin" Signed-off-by: Nicholas Piggin --- Ack or objection if this goes via the -mm tree? arch/x86/include/asm/vmalloc.h | 22 +++++++++++++++++++--- arch/x86/mm/ioremap.c | 19 ------------------- arch/x86/mm/pgtable.c | 13 ------------- 3 files changed, 19 insertions(+), 35 deletions(-) diff --git a/arch/x86/include/asm/vmalloc.h b/arch/x86/include/asm/vmalloc.h index 094ea2b565f3..e714b00fc0ca 100644 --- a/arch/x86/include/asm/vmalloc.h +++ b/arch/x86/include/asm/vmalloc.h @@ -1,13 +1,29 @@ #ifndef _ASM_X86_VMALLOC_H #define _ASM_X86_VMALLOC_H +#include #include #include #ifdef CONFIG_HAVE_ARCH_HUGE_VMAP -bool arch_vmap_p4d_supported(pgprot_t prot); -bool arch_vmap_pud_supported(pgprot_t prot); -bool arch_vmap_pmd_supported(pgprot_t prot); +static inline bool arch_vmap_p4d_supported(pgprot_t prot) +{ + return false; +} + +static inline bool arch_vmap_pud_supported(pgprot_t prot) +{ +#ifdef CONFIG_X86_64 + return boot_cpu_has(X86_FEATURE_GBPAGES); +#else + return false; +#endif +} + +static inline bool arch_vmap_pmd_supported(pgprot_t prot) +{ + return boot_cpu_has(X86_FEATURE_PSE); +} #endif #endif /* _ASM_X86_VMALLOC_H */ diff --git a/arch/x86/mm/ioremap.c b/arch/x86/mm/ioremap.c index 159bfca757b9..1465a22a9bfb 100644 --- a/arch/x86/mm/ioremap.c +++ b/arch/x86/mm/ioremap.c @@ -481,25 +481,6 @@ void iounmap(volatile void __iomem *addr) } EXPORT_SYMBOL(iounmap); -bool arch_vmap_p4d_supported(pgprot_t prot) -{ - return false; -} - -bool arch_vmap_pud_supported(pgprot_t prot) -{ -#ifdef CONFIG_X86_64 - return boot_cpu_has(X86_FEATURE_GBPAGES); -#else - return false; -#endif -} - -bool arch_vmap_pmd_supported(pgprot_t prot) -{ - return boot_cpu_has(X86_FEATURE_PSE); -} - /* * Convert a physical pointer to a virtual kernel pointer for /dev/mem * access diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index dfd82f51ba66..801c418ee97d 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -780,14 +780,6 @@ int pmd_clear_huge(pmd_t *pmd) return 0; } -/* - * Until we support 512GB pages, skip them in the vmap area. - */ -int p4d_free_pud_page(p4d_t *p4d, unsigned long addr) -{ - return 0; -} - #ifdef CONFIG_X86_64 /** * pud_free_pmd_page - Clear pud entry and free pmd page. @@ -859,11 +851,6 @@ int pmd_free_pte_page(pmd_t *pmd, unsigned long addr) #else /* !CONFIG_X86_64 */ -int pud_free_pmd_page(pud_t *pud, unsigned long addr) -{ - return pud_none(*pud); -} - /* * Disable free page handling on x86-PAE. This assures that ioremap() * does not update sync'd pmd entries. See vmalloc_sync_one(). From patchwork Tue Aug 25 14:57:50 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11735855 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EA988138A for ; Tue, 25 Aug 2020 14:58:52 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9D6EE207D3 for ; Tue, 25 Aug 2020 14:58:52 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="EmY2bagV" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9D6EE207D3 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 77B698E0017; Tue, 25 Aug 2020 10:58:51 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 72C628D0001; Tue, 25 Aug 2020 10:58:51 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5F4418E0017; Tue, 25 Aug 2020 10:58:51 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0165.hostedemail.com [216.40.44.165]) by kanga.kvack.org (Postfix) with ESMTP id 48FEE8D0001 for ; Tue, 25 Aug 2020 10:58:51 -0400 (EDT) Received: from smtpin27.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 080313625 for ; Tue, 25 Aug 2020 14:58:51 +0000 (UTC) X-FDA: 77189398062.27.tray77_6012b9d2705c Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin27.hostedemail.com (Postfix) with ESMTP id C5CE63D663 for ; Tue, 25 Aug 2020 14:58:50 +0000 (UTC) X-Spam-Summary: 1,0,0,cdbb8e052c6d1c71,d41d8cd98f00b204,npiggin@gmail.com,,RULES_HIT:1:41:69:355:379:541:800:960:966:968:973:988:989:1260:1311:1314:1345:1359:1437:1515:1605:1730:1747:1777:1792:1981:2194:2196:2199:2200:2393:2559:2562:2637:2898:3138:3139:3140:3141:3142:3867:3871:4321:4385:4605:5007:6119:6261:6653:7514:7875:8603:9036:9413:9592:10004:11026:11232:11473:11658:11914:12043:12291:12296:12297:12438:12517:12519:12555:12683:12895:12986:13894:14096:14110:14394:14687:21080:21324:21444:21451:21611:21627:21666:21740:21990:30054:30070,0,RBL:209.85.215.194:@gmail.com:.lbl8.mailshell.net-62.18.0.100 66.100.201.100;04yrkzh9mbp1pcfjdef3axhaczpimopg37etga1q7iqtiwh4kj9ya3a3sn4gjz9.3fa5bjc6urpxgua57byd4psmn9sic57ew38fahi3yadbhj7x35y7111osympjtc.1-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: tray77_6012b9d2705c X-Filterd-Recvd-Size: 14906 Received: from mail-pg1-f194.google.com (mail-pg1-f194.google.com [209.85.215.194]) by imf08.hostedemail.com (Postfix) with ESMTP for ; Tue, 25 Aug 2020 14:58:50 +0000 (UTC) Received: by mail-pg1-f194.google.com with SMTP id l191so4898235pgd.5 for ; Tue, 25 Aug 2020 07:58:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=FErZs3hZSGjvqwNcDn/CTTFNbpmYi2cg1JR35qAXZTs=; b=EmY2bagVYcEwm/v7nmZnHTii9lxPoceyw//2dPjiejBidWt6EbSEd2mhPJcOgGVVd0 PETr9kZUt+8PZVFnxT357wUl0RQsyaZHtU+qKhbYjsBmz7rgaosotYUl7OsbvPMI9bP+ UZDtKe5/XCjGTipj5wmhZdxbeQ3j6sdUkaQR5Ml0JCBkGhCI3dxqQGKQKLhoIun4huda q4OQE81ocUORQul9f50fs9/wcB1vdEJBSX2m5sCh2xi9uGo7fyrRNKSQg+wmXebMj/cn Aq4N8+0aq7+kjVDVUfrN/0i9yojvl874l7U4a8ry0z/sRBrgmftKoWz/Xddx0R+MZU1N NUCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=FErZs3hZSGjvqwNcDn/CTTFNbpmYi2cg1JR35qAXZTs=; b=JyV1ataKspaja9u58aSd8+P3pd1TJni7GEc+NyFaDgsPFWWoJWrZoaMn07fAHKFj0C gjA7skguDEdf4XT0uGnC7KtVTXo6OfVh/cQbgWi2WJT4CTi+Y8nj6B5p0e5UTh09oaTl sBEEHac4dBNz0MvqckPx6oiN+Yk5wAbgPIspv2OAZWKwaS3v2fwLimDmZW536U+cv6tl W7LntxOrQkRQ6ldMsvuHa9grOJGNVIfSVIdC1Grsl00o/0yNt/C2bvC6bRNblmSfI5bO 5y9Nyb2TG6j0eSpMJj3zAxLFKLjTab4grNisc2AapeZtjwJ039o2CqqHbsODMoCrfnsF sfvA== X-Gm-Message-State: AOAM5331ItQn1wXaWlDf7uJVyub4jlc3E+cescZP0QKfraOVtWbwC63u /7foXDIhbTREzKsEtcH0GEEebTd44cU= X-Google-Smtp-Source: ABdhPJw31A0Uq06DkF6fTEYi8KgTmGz2Cy3Zchx+cG6HwJRAzukHgP51Ao6KdPPtqmJe9zVfFGPIFg== X-Received: by 2002:a17:902:8ecb:: with SMTP id x11mr7922476plo.13.1598367529029; Tue, 25 Aug 2020 07:58:49 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (61-68-212-105.tpgi.com.au. [61.68.212.105]) by smtp.gmail.com with ESMTPSA id e29sm15755956pfj.92.2020.08.25.07.58.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Aug 2020 07:58:48 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org, Andrew Morton Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Zefan Li , Jonathan Cameron , Christoph Hellwig , Christophe Leroy Subject: [PATCH v7 09/12] mm: Move vmap_range from mm/ioremap.c to mm/vmalloc.c Date: Wed, 26 Aug 2020 00:57:50 +1000 Message-Id: <20200825145753.529284-10-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20200825145753.529284-1-npiggin@gmail.com> References: <20200825145753.529284-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: C5CE63D663 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is a generic kernel virtual memory mapper, not specific to ioremap. Signed-off-by: Nicholas Piggin --- include/linux/vmalloc.h | 3 + mm/ioremap.c | 197 ---------------------------------------- mm/vmalloc.c | 196 +++++++++++++++++++++++++++++++++++++++ 3 files changed, 199 insertions(+), 197 deletions(-) diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 3f6bba4cc9bc..15adb9a14fb6 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -177,6 +177,9 @@ extern struct vm_struct *remove_vm_area(const void *addr); extern struct vm_struct *find_vm_area(const void *addr); #ifdef CONFIG_MMU +int vmap_range(unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift); extern int map_kernel_range_noflush(unsigned long start, unsigned long size, pgprot_t prot, struct page **pages); int map_kernel_range(unsigned long start, unsigned long size, pgprot_t prot, diff --git a/mm/ioremap.c b/mm/ioremap.c index c67f91164401..d1dcc7e744ac 100644 --- a/mm/ioremap.c +++ b/mm/ioremap.c @@ -28,203 +28,6 @@ early_param("nohugeiomap", set_nohugeiomap); static const bool iomap_max_page_shift = PAGE_SHIFT; #endif /* CONFIG_HAVE_ARCH_HUGE_VMAP */ -static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, - pgtbl_mod_mask *mask) -{ - pte_t *pte; - u64 pfn; - - pfn = phys_addr >> PAGE_SHIFT; - pte = pte_alloc_kernel_track(pmd, addr, mask); - if (!pte) - return -ENOMEM; - do { - BUG_ON(!pte_none(*pte)); - set_pte_at(&init_mm, addr, pte, pfn_pte(pfn, prot)); - pfn++; - } while (pte++, addr += PAGE_SIZE, addr != end); - *mask |= PGTBL_PTE_MODIFIED; - return 0; -} - -static int vmap_try_huge_pmd(pmd_t *pmd, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, - unsigned int max_page_shift) -{ - if (max_page_shift < PMD_SHIFT) - return 0; - - if (!arch_vmap_pmd_supported(prot)) - return 0; - - if ((end - addr) != PMD_SIZE) - return 0; - - if (!IS_ALIGNED(addr, PMD_SIZE)) - return 0; - - if (!IS_ALIGNED(phys_addr, PMD_SIZE)) - return 0; - - if (pmd_present(*pmd) && !pmd_free_pte_page(pmd, addr)) - return 0; - - return pmd_set_huge(pmd, phys_addr, prot); -} - -static int vmap_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, - unsigned int max_page_shift, pgtbl_mod_mask *mask) -{ - pmd_t *pmd; - unsigned long next; - - pmd = pmd_alloc_track(&init_mm, pud, addr, mask); - if (!pmd) - return -ENOMEM; - do { - next = pmd_addr_end(addr, end); - - if (vmap_try_huge_pmd(pmd, addr, next, phys_addr, prot, max_page_shift)) { - *mask |= PGTBL_PMD_MODIFIED; - continue; - } - - if (vmap_pte_range(pmd, addr, next, phys_addr, prot, mask)) - return -ENOMEM; - } while (pmd++, phys_addr += (next - addr), addr = next, addr != end); - return 0; -} - -static int vmap_try_huge_pud(pud_t *pud, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, - unsigned int max_page_shift) -{ - if (max_page_shift < PUD_SHIFT) - return 0; - - if (!arch_vmap_pud_supported(prot)) - return 0; - - if ((end - addr) != PUD_SIZE) - return 0; - - if (!IS_ALIGNED(addr, PUD_SIZE)) - return 0; - - if (!IS_ALIGNED(phys_addr, PUD_SIZE)) - return 0; - - if (pud_present(*pud) && !pud_free_pmd_page(pud, addr)) - return 0; - - return pud_set_huge(pud, phys_addr, prot); -} - -static int vmap_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, - unsigned int max_page_shift, pgtbl_mod_mask *mask) -{ - pud_t *pud; - unsigned long next; - - pud = pud_alloc_track(&init_mm, p4d, addr, mask); - if (!pud) - return -ENOMEM; - do { - next = pud_addr_end(addr, end); - - if (vmap_try_huge_pud(pud, addr, next, phys_addr, prot, max_page_shift)) { - *mask |= PGTBL_PUD_MODIFIED; - continue; - } - - if (vmap_pmd_range(pud, addr, next, phys_addr, prot, max_page_shift, mask)) - return -ENOMEM; - } while (pud++, phys_addr += (next - addr), addr = next, addr != end); - return 0; -} - -static int vmap_try_huge_p4d(p4d_t *p4d, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, - unsigned int max_page_shift) -{ - if (max_page_shift < P4D_SHIFT) - return 0; - - if (!arch_vmap_p4d_supported(prot)) - return 0; - - if ((end - addr) != P4D_SIZE) - return 0; - - if (!IS_ALIGNED(addr, P4D_SIZE)) - return 0; - - if (!IS_ALIGNED(phys_addr, P4D_SIZE)) - return 0; - - if (p4d_present(*p4d) && !p4d_free_pud_page(p4d, addr)) - return 0; - - return p4d_set_huge(p4d, phys_addr, prot); -} - -static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, - unsigned int max_page_shift, pgtbl_mod_mask *mask) -{ - p4d_t *p4d; - unsigned long next; - - p4d = p4d_alloc_track(&init_mm, pgd, addr, mask); - if (!p4d) - return -ENOMEM; - do { - next = p4d_addr_end(addr, end); - - if (vmap_try_huge_p4d(p4d, addr, next, phys_addr, prot, max_page_shift)) { - *mask |= PGTBL_P4D_MODIFIED; - continue; - } - - if (vmap_pud_range(p4d, addr, next, phys_addr, prot, max_page_shift, mask)) - return -ENOMEM; - } while (p4d++, phys_addr += (next - addr), addr = next, addr != end); - return 0; -} - -static int vmap_range(unsigned long addr, unsigned long end, - phys_addr_t phys_addr, pgprot_t prot, - unsigned int max_page_shift) -{ - pgd_t *pgd; - unsigned long start; - unsigned long next; - int err; - pgtbl_mod_mask mask = 0; - - might_sleep(); - BUG_ON(addr >= end); - - start = addr; - pgd = pgd_offset_k(addr); - do { - next = pgd_addr_end(addr, end); - err = vmap_p4d_range(pgd, addr, next, phys_addr, prot, max_page_shift, &mask); - if (err) - break; - } while (pgd++, phys_addr += (next - addr), addr = next, addr != end); - - flush_cache_vmap(start, end); - - if (mask & ARCH_PAGE_TABLE_SYNC_MASK) - arch_sync_kernel_mappings(start, end); - - return err; -} - int ioremap_page_range(unsigned long addr, unsigned long end, phys_addr_t phys_addr, pgprot_t prot) { diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 45cd80ec7eeb..256554d598e6 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -70,6 +70,202 @@ static void free_work(struct work_struct *w) } /*** Page table manipulation functions ***/ +static int vmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + pgtbl_mod_mask *mask) +{ + pte_t *pte; + u64 pfn; + + pfn = phys_addr >> PAGE_SHIFT; + pte = pte_alloc_kernel_track(pmd, addr, mask); + if (!pte) + return -ENOMEM; + do { + BUG_ON(!pte_none(*pte)); + set_pte_at(&init_mm, addr, pte, pfn_pte(pfn, prot)); + pfn++; + } while (pte++, addr += PAGE_SIZE, addr != end); + *mask |= PGTBL_PTE_MODIFIED; + return 0; +} + +static int vmap_try_huge_pmd(pmd_t *pmd, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift) +{ + if (max_page_shift < PMD_SHIFT) + return 0; + + if (!arch_vmap_pmd_supported(prot)) + return 0; + + if ((end - addr) != PMD_SIZE) + return 0; + + if (!IS_ALIGNED(addr, PMD_SIZE)) + return 0; + + if (!IS_ALIGNED(phys_addr, PMD_SIZE)) + return 0; + + if (pmd_present(*pmd) && !pmd_free_pte_page(pmd, addr)) + return 0; + + return pmd_set_huge(pmd, phys_addr, prot); +} + +static int vmap_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift, pgtbl_mod_mask *mask) +{ + pmd_t *pmd; + unsigned long next; + + pmd = pmd_alloc_track(&init_mm, pud, addr, mask); + if (!pmd) + return -ENOMEM; + do { + next = pmd_addr_end(addr, end); + + if (vmap_try_huge_pmd(pmd, addr, next, phys_addr, prot, max_page_shift)) { + *mask |= PGTBL_PMD_MODIFIED; + continue; + } + + if (vmap_pte_range(pmd, addr, next, phys_addr, prot, mask)) + return -ENOMEM; + } while (pmd++, phys_addr += (next - addr), addr = next, addr != end); + return 0; +} + +static int vmap_try_huge_pud(pud_t *pud, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift) +{ + if (max_page_shift < PUD_SHIFT) + return 0; + + if (!arch_vmap_pud_supported(prot)) + return 0; + + if ((end - addr) != PUD_SIZE) + return 0; + + if (!IS_ALIGNED(addr, PUD_SIZE)) + return 0; + + if (!IS_ALIGNED(phys_addr, PUD_SIZE)) + return 0; + + if (pud_present(*pud) && !pud_free_pmd_page(pud, addr)) + return 0; + + return pud_set_huge(pud, phys_addr, prot); +} + +static int vmap_pud_range(p4d_t *p4d, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift, pgtbl_mod_mask *mask) +{ + pud_t *pud; + unsigned long next; + + pud = pud_alloc_track(&init_mm, p4d, addr, mask); + if (!pud) + return -ENOMEM; + do { + next = pud_addr_end(addr, end); + + if (vmap_try_huge_pud(pud, addr, next, phys_addr, prot, max_page_shift)) { + *mask |= PGTBL_PUD_MODIFIED; + continue; + } + + if (vmap_pmd_range(pud, addr, next, phys_addr, prot, max_page_shift, mask)) + return -ENOMEM; + } while (pud++, phys_addr += (next - addr), addr = next, addr != end); + return 0; +} + +static int vmap_try_huge_p4d(p4d_t *p4d, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift) +{ + if (max_page_shift < P4D_SHIFT) + return 0; + + if (!arch_vmap_p4d_supported(prot)) + return 0; + + if ((end - addr) != P4D_SIZE) + return 0; + + if (!IS_ALIGNED(addr, P4D_SIZE)) + return 0; + + if (!IS_ALIGNED(phys_addr, P4D_SIZE)) + return 0; + + if (p4d_present(*p4d) && !p4d_free_pud_page(p4d, addr)) + return 0; + + return p4d_set_huge(p4d, phys_addr, prot); +} + +static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift, pgtbl_mod_mask *mask) +{ + p4d_t *p4d; + unsigned long next; + + p4d = p4d_alloc_track(&init_mm, pgd, addr, mask); + if (!p4d) + return -ENOMEM; + do { + next = p4d_addr_end(addr, end); + + if (vmap_try_huge_p4d(p4d, addr, next, phys_addr, prot, max_page_shift)) { + *mask |= PGTBL_P4D_MODIFIED; + continue; + } + + if (vmap_pud_range(p4d, addr, next, phys_addr, prot, max_page_shift, mask)) + return -ENOMEM; + } while (p4d++, phys_addr += (next - addr), addr = next, addr != end); + return 0; +} + +int vmap_range(unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift) +{ + pgd_t *pgd; + unsigned long start; + unsigned long next; + int err; + pgtbl_mod_mask mask = 0; + + might_sleep(); + BUG_ON(addr >= end); + + start = addr; + pgd = pgd_offset_k(addr); + do { + next = pgd_addr_end(addr, end); + err = vmap_p4d_range(pgd, addr, next, phys_addr, prot, max_page_shift, &mask); + if (err) + break; + } while (pgd++, phys_addr += (next - addr), addr = next, addr != end); + + flush_cache_vmap(start, end); + + if (mask & ARCH_PAGE_TABLE_SYNC_MASK) + arch_sync_kernel_mappings(start, end); + + return err; +} static void vunmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, pgtbl_mod_mask *mask) From patchwork Tue Aug 25 14:57:51 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11735857 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 20E28138A for ; Tue, 25 Aug 2020 14:58:57 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E1D2220782 for ; Tue, 25 Aug 2020 14:58:56 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="jFDbXQGW" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E1D2220782 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id BC2B88E0018; Tue, 25 Aug 2020 10:58:55 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id B728C8D0001; Tue, 25 Aug 2020 10:58:55 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A63A58E0018; Tue, 25 Aug 2020 10:58:55 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0247.hostedemail.com [216.40.44.247]) by kanga.kvack.org (Postfix) with ESMTP id 8BE1E8D0001 for ; Tue, 25 Aug 2020 10:58:55 -0400 (EDT) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 3A339181AEF21 for ; Tue, 25 Aug 2020 14:58:55 +0000 (UTC) X-FDA: 77189398230.11.beds73_39012c92705c Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin11.hostedemail.com (Postfix) with ESMTP id DEC91180F8B82 for ; Tue, 25 Aug 2020 14:58:54 +0000 (UTC) X-Spam-Summary: 1,0,0,b3e82260f0628319,d41d8cd98f00b204,npiggin@gmail.com,,RULES_HIT:41:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1541:1711:1730:1747:1777:1792:2198:2199:2393:2559:2562:2731:3138:3139:3140:3141:3142:3352:3865:3866:3868:3870:3874:4321:5007:6261:6653:7514:7875:9413:9592:10004:11026:11473:11658:11914:12043:12297:12438:12517:12519:12555:12895:13069:13161:13229:13311:13357:13894:14096:14181:14384:14394:14687:14721:21080:21444:21451:21627:21666:21990:30054,0,RBL:209.85.216.66:@gmail.com:.lbl8.mailshell.net-62.50.0.100 66.100.201.100;04yrh4xeid44ihdb17ude49gwk1r9oc71khjzgrret6jjxbm968kb1o7tqbr6oh.wayfyuxek4dsc75hxrzzi1fs4ai6dohy185ep76t6xabuhrhf38t46hw8jmmjho.6-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: beds73_39012c92705c X-Filterd-Recvd-Size: 4658 Received: from mail-pj1-f66.google.com (mail-pj1-f66.google.com [209.85.216.66]) by imf19.hostedemail.com (Postfix) with ESMTP for ; Tue, 25 Aug 2020 14:58:54 +0000 (UTC) Received: by mail-pj1-f66.google.com with SMTP id nv17so1388459pjb.3 for ; Tue, 25 Aug 2020 07:58:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Ik3l57t0FyOClvLjEe+sb/BhE/y0hhY0gnEqux751Uc=; b=jFDbXQGWGSUzk7/GJ+Z/Evd9q3iiU/BfiaMC4g83Na9QhLkZaoeFiiSi/FDNx+q0/o JzPfzJTgJWm/P9pW+kiyHgiJ3OxnIaNeOA/VQFt0osyMoqtV8dJGZoFOSLHI8ZJp2LQG uXQsCruf53J3N+40TqcYQYhQU5JOrltFKObIJtcT8dIGteWrwQ63Li+XqNWXfO/8TIcX O5Y3C++e90X5Ov1fyAwEZ0Pw1S8c0WlG2BrKjPLNiDRIjxzpQXbwnzWTA+RXKohUt+G4 zwS57mS4cvrbf9wiDvxEB6b1vNXZ5voiRSKhtoS77rvCe8CCQIRYn7SJtz8+EOdXbboI nZzw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Ik3l57t0FyOClvLjEe+sb/BhE/y0hhY0gnEqux751Uc=; b=GI3xu7BqGXo2e2QmMftWDOZ6lNtkVQBSsDI9Nx6REpdV/XPYVJ+S7EPZPKINx7HIvU OdBYlhLHlmc8yJYREpJUVeecW7TXBnZVXe2vIJTTL7sv8r3fSuuteMjUgUwmUK7037mB P0GBKG0ho19e+eVVNjjFi9Ks62IwTMtlH7QQk7yeY5ei2i0Y59QSiwko8Xh2kisLAiKA baBALioEmQoMHtbhoXTIveE+IUvfSA/yw/lzT/JYZp1WxiBWBXs/23KO94zv9VReENlq oF7eglXY8w7RLrXQBVa3eAUtR0PxdsfMmOjAQzuf+3ZJlqx5rweDkjR5am38ABDNfvWJ 8W1Q== X-Gm-Message-State: AOAM533MzdK7eUF+mLjKIcDKAOEmhd9DyTQiBp2tVSl7oK4l1kqf96OB pTadjOTITEnNisughvUb2nyKHjPWRLU= X-Google-Smtp-Source: ABdhPJzHo6/7MAKEoI/XmiBkadcGmQ+3ea7nzB3CCBEUR/YsmN15UifwlOMYYbga1Z5fsGVrMds0VQ== X-Received: by 2002:a17:90a:a101:: with SMTP id s1mr1788302pjp.205.1598367533484; Tue, 25 Aug 2020 07:58:53 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (61-68-212-105.tpgi.com.au. [61.68.212.105]) by smtp.gmail.com with ESMTPSA id e29sm15755956pfj.92.2020.08.25.07.58.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Aug 2020 07:58:53 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org, Andrew Morton Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Zefan Li , Jonathan Cameron , Christoph Hellwig , Christophe Leroy Subject: [PATCH v7 10/12] mm/vmalloc: add vmap_range_noflush variant Date: Wed, 26 Aug 2020 00:57:51 +1000 Message-Id: <20200825145753.529284-11-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20200825145753.529284-1-npiggin@gmail.com> References: <20200825145753.529284-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: DEC91180F8B82 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam04 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: As a side-effect, the order of flush_cache_vmap() and arch_sync_kernel_mappings() calls are switched, but that now matches the other callers in this file. Signed-off-by: Nicholas Piggin --- mm/vmalloc.c | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 256554d598e6..1d6cad16bda3 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -237,7 +237,7 @@ static int vmap_p4d_range(pgd_t *pgd, unsigned long addr, unsigned long end, return 0; } -int vmap_range(unsigned long addr, unsigned long end, +static int vmap_range_noflush(unsigned long addr, unsigned long end, phys_addr_t phys_addr, pgprot_t prot, unsigned int max_page_shift) { @@ -259,14 +259,24 @@ int vmap_range(unsigned long addr, unsigned long end, break; } while (pgd++, phys_addr += (next - addr), addr = next, addr != end); - flush_cache_vmap(start, end); - if (mask & ARCH_PAGE_TABLE_SYNC_MASK) arch_sync_kernel_mappings(start, end); return err; } +int vmap_range(unsigned long addr, unsigned long end, + phys_addr_t phys_addr, pgprot_t prot, + unsigned int max_page_shift) +{ + int err; + + err = vmap_range_noflush(addr, end, phys_addr, prot, max_page_shift); + flush_cache_vmap(addr, end); + + return err; +} + static void vunmap_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, pgtbl_mod_mask *mask) { From patchwork Tue Aug 25 14:57:52 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11735863 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3D4E817C7 for ; Tue, 25 Aug 2020 14:59:02 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id E3BDC20782 for ; Tue, 25 Aug 2020 14:59:01 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="G4/MMiKI" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E3BDC20782 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9C8038E0019; Tue, 25 Aug 2020 10:59:00 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 99E8A8D0001; Tue, 25 Aug 2020 10:59:00 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 865308E0019; Tue, 25 Aug 2020 10:59:00 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0100.hostedemail.com [216.40.44.100]) by kanga.kvack.org (Postfix) with ESMTP id 6EA888D0001 for ; Tue, 25 Aug 2020 10:59:00 -0400 (EDT) Received: from smtpin09.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 359E91EE6 for ; Tue, 25 Aug 2020 14:59:00 +0000 (UTC) X-FDA: 77189398440.09.flag95_341287b2705c Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin09.hostedemail.com (Postfix) with ESMTP id 0655B180AD801 for ; Tue, 25 Aug 2020 14:58:59 +0000 (UTC) X-Spam-Summary: 1,0,0,5ca799c4446aca61,d41d8cd98f00b204,npiggin@gmail.com,,RULES_HIT:4:41:69:355:379:541:800:960:966:968:973:988:989:1260:1311:1314:1345:1359:1431:1437:1515:1605:1730:1747:1777:1792:2194:2196:2198:2199:2200:2201:2393:2553:2559:2562:2638:2689:2693:2731:2897:2899:3138:3139:3140:3141:3142:3865:3866:3867:3868:3870:3871:3874:4250:4321:4385:4605:4641:5007:6119:6261:6653:7514:7875:7903:7904:8603:9036:9413:9592:10004:11026:11473:11658:11914:12043:12291:12296:12297:12438:12517:12519:12555:12679:12683:12895:12986:13894:14096:14394:14687:21080:21324:21444:21451:21611:21627:21666:21740:21990:30054:30056:30090,0,RBL:209.85.216.66:@gmail.com:.lbl8.mailshell.net-62.50.0.100 66.100.201.100;04y84q8x5s5moni1brs5h81irqzu9ycfg6btwtzq9hep1kcqo6waghiexx1ogpm.97ariu64b4bgc8ie99d8acg3iw6qhyp9ej1hgd3y6xwz4juixxjjbmgp87akoid.c-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFti me:24,LU X-HE-Tag: flag95_341287b2705c X-Filterd-Recvd-Size: 16164 Received: from mail-pj1-f66.google.com (mail-pj1-f66.google.com [209.85.216.66]) by imf07.hostedemail.com (Postfix) with ESMTP for ; Tue, 25 Aug 2020 14:58:59 +0000 (UTC) Received: by mail-pj1-f66.google.com with SMTP id g6so1376096pjl.0 for ; Tue, 25 Aug 2020 07:58:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=6KJeQP8NI0Gmq28+Z9mGp8SWZDiLwVnET1WJkFSU9fg=; b=G4/MMiKI3zAB9GZaRHWlNv9lxtyaITj1HOneVEnczu7qEBJuf+V670gslFcluD9DfV dKFhpN/WL4LNfDDfqNgVKdmCkWZtOQD8AR87AfG5JuTiroU9qsz6dQkIQISu8fQcNQpX brfM6EH/XQVEv4CyH06jAc2CUPxwD6+60MbRIOyzUHJi5XVsEEe8gkMYcR1WSQYp9r5u XP+Rkwj4ZIVDgOOaCKn9ONyOoYAOP4aBBQ9Gg8T/6H+kHiTY60o59ZUHO0d0sqG7Rd4P uU1lQsBRXasq3jcdCOP+kdgE/+A2BLKF21czW067Wh8qyrY+uN2Dd8I5PEt6zRFb/3NR QL3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=6KJeQP8NI0Gmq28+Z9mGp8SWZDiLwVnET1WJkFSU9fg=; b=hBK94iohxEBacZdY0LT/2uwz/l6AIFZslVqhzTfCByOs4QJVcaORYCoIGd2+FIxX4o NlHhc8wcCSsn4S5ZMpP7oV+wMBw6TYcHctpUugut3KwFbk8ZQCrp5VGPA4qm9deetmXT WZZ74feJkyYA3zsNu290UMAVXNkzzBtuvCPglxELlapY3iO6cDWVunaebWB/LZXV6qHV OGD02kN9O1HCHCy/NpHy5DgI/vusLWGTL7hGJU33Q/b+4Sr55I30YdvBjkXXtdEQHSYt 5Ux38+KQv+OaRT8GAiILTSQq9j/dVu753wgKVS2DoQLuz/L06Di+cmUZwi4hAqLS7kjw AdAA== X-Gm-Message-State: AOAM532DTfHWFq4BDZtyaHjfHMfNL5NmXxu8ILggiFydKPAinhvUfwoC h2S4hUMB6FtSVlGei1GSkIu5ywmO5ZY= X-Google-Smtp-Source: ABdhPJwxOdeZP69xC+JAEexbE9+iKYE0HYeXxKfVEDE0GXMN513IqNTakJmuHRjF51c8KGciI3/RNg== X-Received: by 2002:a17:90a:1e65:: with SMTP id w92mr1861914pjw.187.1598367538048; Tue, 25 Aug 2020 07:58:58 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (61-68-212-105.tpgi.com.au. [61.68.212.105]) by smtp.gmail.com with ESMTPSA id e29sm15755956pfj.92.2020.08.25.07.58.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Aug 2020 07:58:57 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org, Andrew Morton Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Zefan Li , Jonathan Cameron , Christoph Hellwig , Christophe Leroy Subject: [PATCH v7 11/12] mm/vmalloc: Hugepage vmalloc mappings Date: Wed, 26 Aug 2020 00:57:52 +1000 Message-Id: <20200825145753.529284-12-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20200825145753.529284-1-npiggin@gmail.com> References: <20200825145753.529284-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 0655B180AD801 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Support huge page vmalloc mappings. Config option HAVE_ARCH_HUGE_VMALLOC enables support on architectures that define HAVE_ARCH_HUGE_VMAP and supports PMD sized vmap mappings. vmalloc will attempt to allocate PMD-sized pages if allocating PMD size or larger, and fall back to small pages if that was unsuccessful. Allocations that do not use PAGE_KERNEL prot are not permitted to use huge pages, because not all callers expect this (e.g., module allocations vs strict module rwx). This reduces TLB misses by nearly 30x on a `git diff` workload on a 2-node POWER9 (59,800 -> 2,100) and reduces CPU cycles by 0.54%. This can result in more internal fragmentation and memory overhead for a given allocation, an option nohugevmalloc is added to disable at boot. Signed-off-by: Nicholas Piggin --- arch/Kconfig | 4 + include/linux/vmalloc.h | 1 + mm/page_alloc.c | 5 +- mm/vmalloc.c | 180 ++++++++++++++++++++++++++++++---------- 4 files changed, 145 insertions(+), 45 deletions(-) diff --git a/arch/Kconfig b/arch/Kconfig index af14a567b493..b2b89d629317 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -616,6 +616,10 @@ config HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD config HAVE_ARCH_HUGE_VMAP bool +config HAVE_ARCH_HUGE_VMALLOC + depends on HAVE_ARCH_HUGE_VMAP + bool + config ARCH_WANT_HUGE_PMD_SHARE bool diff --git a/include/linux/vmalloc.h b/include/linux/vmalloc.h index 15adb9a14fb6..a7449064fe35 100644 --- a/include/linux/vmalloc.h +++ b/include/linux/vmalloc.h @@ -58,6 +58,7 @@ struct vm_struct { unsigned long size; unsigned long flags; struct page **pages; + unsigned int page_order; unsigned int nr_pages; phys_addr_t phys_addr; const void *caller; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 0e2bab486fea..b6427cc7b838 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -69,6 +69,7 @@ #include #include #include +#include #include #include @@ -8102,6 +8103,7 @@ void *__init alloc_large_system_hash(const char *tablename, void *table = NULL; gfp_t gfp_flags; bool virt; + bool huge; /* allow the kernel cmdline to have a say */ if (!numentries) { @@ -8169,6 +8171,7 @@ void *__init alloc_large_system_hash(const char *tablename, } else if (get_order(size) >= MAX_ORDER || hashdist) { table = __vmalloc(size, gfp_flags); virt = true; + huge = (find_vm_area(table)->page_order > 0); } else { /* * If bucketsize is not a power-of-two, we may free @@ -8185,7 +8188,7 @@ void *__init alloc_large_system_hash(const char *tablename, pr_info("%s hash table entries: %ld (order: %d, %lu bytes, %s)\n", tablename, 1UL << log2qty, ilog2(size) - PAGE_SHIFT, size, - virt ? "vmalloc" : "linear"); + virt ? (huge ? "vmalloc hugepage" : "vmalloc") : "linear"); if (_hash_shift) *_hash_shift = log2qty; diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 1d6cad16bda3..8db53c2d7f72 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -44,6 +44,19 @@ #include "internal.h" #include "pgalloc-track.h" +#ifdef CONFIG_HAVE_ARCH_HUGE_VMALLOC +static bool __ro_after_init vmap_allow_huge = true; + +static int __init set_nohugevmalloc(char *str) +{ + vmap_allow_huge = false; + return 0; +} +early_param("nohugevmalloc", set_nohugevmalloc); +#else /* CONFIG_HAVE_ARCH_HUGE_VMALLOC */ +static const bool vmap_allow_huge = false; +#endif /* CONFIG_HAVE_ARCH_HUGE_VMALLOC */ + bool is_vmalloc_addr(const void *x) { unsigned long addr = (unsigned long)x; @@ -477,31 +490,12 @@ static int vmap_pages_p4d_range(pgd_t *pgd, unsigned long addr, return 0; } -/** - * map_kernel_range_noflush - map kernel VM area with the specified pages - * @addr: start of the VM area to map - * @size: size of the VM area to map - * @prot: page protection flags to use - * @pages: pages to map - * - * Map PFN_UP(@size) pages at @addr. The VM area @addr and @size specify should - * have been allocated using get_vm_area() and its friends. - * - * NOTE: - * This function does NOT do any cache flushing. The caller is responsible for - * calling flush_cache_vmap() on to-be-mapped areas before calling this - * function. - * - * RETURNS: - * 0 on success, -errno on failure. - */ -int map_kernel_range_noflush(unsigned long addr, unsigned long size, - pgprot_t prot, struct page **pages) +static int vmap_small_pages_range_noflush(unsigned long addr, unsigned long end, + pgprot_t prot, struct page **pages) { unsigned long start = addr; - unsigned long end = addr + size; - unsigned long next; pgd_t *pgd; + unsigned long next; int err = 0; int nr = 0; pgtbl_mod_mask mask = 0; @@ -523,6 +517,65 @@ int map_kernel_range_noflush(unsigned long addr, unsigned long size, return 0; } +static int vmap_pages_range_noflush(unsigned long addr, unsigned long end, + pgprot_t prot, struct page **pages, unsigned int page_shift) +{ + unsigned int i, nr = (end - addr) >> PAGE_SHIFT; + + WARN_ON(page_shift < PAGE_SHIFT); + + if (page_shift == PAGE_SHIFT) + return vmap_small_pages_range_noflush(addr, end, prot, pages); + + for (i = 0; i < nr; i += 1U << (page_shift - PAGE_SHIFT)) { + int err; + + err = vmap_range_noflush(addr, addr + (1UL << page_shift), + __pa(page_address(pages[i])), prot, + page_shift); + if (err) + return err; + + addr += 1UL << page_shift; + } + + return 0; +} + +static int vmap_pages_range(unsigned long addr, unsigned long end, + pgprot_t prot, struct page **pages, unsigned int page_shift) +{ + int err; + + err = vmap_pages_range_noflush(addr, end, prot, pages, page_shift); + flush_cache_vmap(addr, end); + return err; +} + +/** + * map_kernel_range_noflush - map kernel VM area with the specified pages + * @addr: start of the VM area to map + * @size: size of the VM area to map + * @prot: page protection flags to use + * @pages: pages to map + * + * Map PFN_UP(@size) pages at @addr. The VM area @addr and @size specify should + * have been allocated using get_vm_area() and its friends. + * + * NOTE: + * This function does NOT do any cache flushing. The caller is responsible for + * calling flush_cache_vmap() on to-be-mapped areas before calling this + * function. + * + * RETURNS: + * 0 on success, -errno on failure. + */ +int map_kernel_range_noflush(unsigned long addr, unsigned long size, + pgprot_t prot, struct page **pages) +{ + return vmap_pages_range_noflush(addr, addr + size, prot, pages, PAGE_SHIFT); +} + int map_kernel_range(unsigned long start, unsigned long size, pgprot_t prot, struct page **pages) { @@ -2400,6 +2453,7 @@ static inline void set_area_direct_map(const struct vm_struct *area, { int i; + /* HUGE_VMALLOC passes small pages to set_direct_map */ for (i = 0; i < area->nr_pages; i++) if (page_address(area->pages[i])) set_direct_map(area->pages[i]); @@ -2433,11 +2487,12 @@ static void vm_remove_mappings(struct vm_struct *area, int deallocate_pages) * map. Find the start and end range of the direct mappings to make sure * the vm_unmap_aliases() flush includes the direct map. */ - for (i = 0; i < area->nr_pages; i++) { + for (i = 0; i < area->nr_pages; i += 1U << area->page_order) { unsigned long addr = (unsigned long)page_address(area->pages[i]); if (addr) { + unsigned long page_size = PAGE_SIZE << area->page_order; start = min(addr, start); - end = max(addr + PAGE_SIZE, end); + end = max(addr + page_size, end); flush_dmap = 1; } } @@ -2480,11 +2535,11 @@ static void __vunmap(const void *addr, int deallocate_pages) if (deallocate_pages) { int i; - for (i = 0; i < area->nr_pages; i++) { + for (i = 0; i < area->nr_pages; i += 1U << area->page_order) { struct page *page = area->pages[i]; BUG_ON(!page); - __free_pages(page, 0); + __free_pages(page, area->page_order); } atomic_long_sub(area->nr_pages, &nr_vmalloc_pages); @@ -2623,9 +2678,12 @@ void *vmap(struct page **pages, unsigned int count, EXPORT_SYMBOL(vmap); static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, - pgprot_t prot, int node) + pgprot_t prot, unsigned int page_shift, int node) { struct page **pages; + unsigned long addr = (unsigned long)area->addr; + unsigned long size = get_vm_area_size(area); + unsigned int page_order = page_shift - PAGE_SHIFT; unsigned int nr_pages, array_size, i; const gfp_t nested_gfp = (gfp_mask & GFP_RECLAIM_MASK) | __GFP_ZERO; const gfp_t alloc_mask = gfp_mask | __GFP_NOWARN; @@ -2633,7 +2691,7 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, 0 : __GFP_HIGHMEM; - nr_pages = get_vm_area_size(area) >> PAGE_SHIFT; + nr_pages = size >> PAGE_SHIFT; array_size = (nr_pages * sizeof(struct page *)); /* Please note that the recursion is strictly bounded. */ @@ -2652,29 +2710,29 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, area->pages = pages; area->nr_pages = nr_pages; + area->page_order = page_order; - for (i = 0; i < area->nr_pages; i++) { + for (i = 0; i < area->nr_pages; i += 1U << page_order) { struct page *page; + int p; - if (node == NUMA_NO_NODE) - page = alloc_page(alloc_mask|highmem_mask); - else - page = alloc_pages_node(node, alloc_mask|highmem_mask, 0); - + page = alloc_pages_node(node, alloc_mask|highmem_mask, page_order); if (unlikely(!page)) { /* Successfully allocated i pages, free them in __vunmap() */ area->nr_pages = i; atomic_long_add(area->nr_pages, &nr_vmalloc_pages); goto fail; } - area->pages[i] = page; + + for (p = 0; p < (1U << page_order); p++) + area->pages[i + p] = page + p; + if (gfpflags_allow_blocking(gfp_mask)) cond_resched(); } atomic_long_add(area->nr_pages, &nr_vmalloc_pages); - if (map_kernel_range((unsigned long)area->addr, get_vm_area_size(area), - prot, pages) < 0) + if (vmap_pages_range(addr, addr + size, prot, pages, page_shift) < 0) goto fail; return area->addr; @@ -2682,7 +2740,7 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, fail: warn_alloc(gfp_mask, NULL, "vmalloc: allocation failure, allocated %ld of %ld bytes", - (area->nr_pages*PAGE_SIZE), area->size); + (area->nr_pages*PAGE_SIZE), size); __vfree(area->addr); return NULL; } @@ -2713,19 +2771,42 @@ void *__vmalloc_node_range(unsigned long size, unsigned long align, struct vm_struct *area; void *addr; unsigned long real_size = size; + unsigned long real_align = align; + unsigned int shift = PAGE_SHIFT; - size = PAGE_ALIGN(size); if (!size || (size >> PAGE_SHIFT) > totalram_pages()) goto fail; - area = __get_vm_area_node(real_size, align, VM_ALLOC | VM_UNINITIALIZED | + if (vmap_allow_huge && (pgprot_val(prot) == pgprot_val(PAGE_KERNEL))) { + unsigned long size_per_node; + + /* + * Try huge pages. Only try for PAGE_KERNEL allocations, + * others like modules don't yet expect huge pages in + * their allocations due to apply_to_page_range not + * supporting them. + */ + + size_per_node = size; + if (node == NUMA_NO_NODE) + size_per_node /= num_online_nodes(); + if (size_per_node >= PMD_SIZE) { + shift = PMD_SHIFT; + align = max(real_align, 1UL << shift); + size = ALIGN(real_size, 1UL << shift); + } + } + +again: + size = PAGE_ALIGN(size); + area = __get_vm_area_node(size, align, VM_ALLOC | VM_UNINITIALIZED | vm_flags, start, end, node, gfp_mask, caller); if (!area) goto fail; - addr = __vmalloc_area_node(area, gfp_mask, prot, node); + addr = __vmalloc_area_node(area, gfp_mask, prot, shift, node); if (!addr) - return NULL; + goto fail; /* * In this function, newly allocated vm_struct has VM_UNINITIALIZED @@ -2739,8 +2820,19 @@ void *__vmalloc_node_range(unsigned long size, unsigned long align, return addr; fail: - warn_alloc(gfp_mask, NULL, + if (shift > PAGE_SHIFT) { + free_vm_area(area); + shift = PAGE_SHIFT; + align = real_align; + size = real_size; + goto again; + } + + if (!area) { + /* Warn for area allocation, page allocations already warn */ + warn_alloc(gfp_mask, NULL, "vmalloc: allocation failure: %lu bytes", real_size); + } return NULL; } @@ -3739,7 +3831,7 @@ static int s_show(struct seq_file *m, void *p) seq_printf(m, " %pS", v->caller); if (v->nr_pages) - seq_printf(m, " pages=%d", v->nr_pages); + seq_printf(m, " pages=%d order=%d", v->nr_pages, v->page_order); if (v->phys_addr) seq_printf(m, " phys=%pa", &v->phys_addr); From patchwork Tue Aug 25 14:57:53 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nicholas Piggin X-Patchwork-Id: 11735865 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 19E2E722 for ; Tue, 25 Aug 2020 14:59:06 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D12E0207BC for ; Tue, 25 Aug 2020 14:59:05 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="CehPsFgN" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D12E0207BC Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 950608E001A; Tue, 25 Aug 2020 10:59:04 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 900E08D0001; Tue, 25 Aug 2020 10:59:04 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 81CBC8E001A; Tue, 25 Aug 2020 10:59:04 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0099.hostedemail.com [216.40.44.99]) by kanga.kvack.org (Postfix) with ESMTP id 65D3B8D0001 for ; Tue, 25 Aug 2020 10:59:04 -0400 (EDT) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 213048248D7C for ; Tue, 25 Aug 2020 14:59:04 +0000 (UTC) X-FDA: 77189398608.01.order31_1015e352705c Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin01.hostedemail.com (Postfix) with ESMTP id D17BE1004C6CD for ; Tue, 25 Aug 2020 14:59:03 +0000 (UTC) X-Spam-Summary: 1,0,0,fc608a1f7ee2e5c7,d41d8cd98f00b204,npiggin@gmail.com,,RULES_HIT:41:355:379:541:800:960:973:988:989:1260:1311:1314:1345:1359:1437:1515:1534:1541:1711:1730:1747:1777:1792:1801:2194:2199:2393:2559:2562:3138:3139:3140:3141:3142:3352:3867:4250:4321:4605:5007:6119:6261:6642:6653:7514:9413:10004:11026:11232:11473:11658:11914:12043:12297:12438:12517:12519:12555:12895:13069:13311:13357:13894:14181:14384:14394:14581:14687:14721:21080:21444:21451:21627:21666:30054:30089,0,RBL:209.85.210.194:@gmail.com:.lbl8.mailshell.net-66.100.201.100 62.50.0.100;04yghdx4ryfqx9zgbrcjr3jjjy8a4ycddu3htbxez758sx193u6u1y5gmd487si.g7zq36yz9assntd9zsmgnr44kf8w9u1bgjdsqnd1m5774imzmteebkyo1dbns3g.h-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:25,LUA_SUMMARY:none X-HE-Tag: order31_1015e352705c X-Filterd-Recvd-Size: 4484 Received: from mail-pf1-f194.google.com (mail-pf1-f194.google.com [209.85.210.194]) by imf31.hostedemail.com (Postfix) with ESMTP for ; Tue, 25 Aug 2020 14:59:03 +0000 (UTC) Received: by mail-pf1-f194.google.com with SMTP id t185so4441793pfd.13 for ; Tue, 25 Aug 2020 07:59:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Gv+PP/wjBFW/QutbN7MDl66u1IlF9/srIH5l1WMUz8Y=; b=CehPsFgNdfZPYRARvszK0iCwlAZvJIxorhXqUBbsMR1owd1LyaUhv6KkQvkxGhxABE h3c8oVY4EFa6rJUgIrnLjCkRStwzaM9eK3BEMzyYASBFUKLmA+35vORx8gBhNYtW5z1x 3s4yU9gOSNIa2NA3S2uwFMPR+o6NS9XnTAPYFEUJ/a2tvEAtbBjA1Mmp8fT2zjJBaCkA 4TWe+7FbzU6KUDpftLPy16UTk04L6fVCaWwJOTS0ymSpOh99vhufmMr4F4IOV+GRRGFF GHMVyMWmrmX2CZxkpNlbupR524BD2hb59sZczST12PHUj62mzy25W0DZVP/E9GAWUSNh X5zg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Gv+PP/wjBFW/QutbN7MDl66u1IlF9/srIH5l1WMUz8Y=; b=Dr5ed79uxVafhzYpEM2LRAj718fiaP2Kh4scnDU3IgOElbtFIDPT+Eb4OjQyqTGXq6 NcNgmCNzR5KbzF6NHT9Ul5bClNgALCFQbf3/3UZNW83GZwgOziO68fzyqjMqXwqATncq lGW2NRkk+sv30CHg4cjT5SRgHfN75ffGVjjxZKtmtE6rSCIyHCU5RfPQUNlaAHmiUxBh Rqba8hB/rDcYq7FFXZ0oQQ7km65rr7iw4ombDNbQLyhCRpEhtQ+9B6N7l0864NJz2IOc UI66I92SVzNYbifqsftSqL6T/IH3KbVCrYC++gShxlfoZmhCT1I1197A7CpeUmt4PLPz rBDA== X-Gm-Message-State: AOAM532TIePHPnusZXtJo7ELQ7ZU7JbVAFD9eFgPl0B0Qv+3sHr7WHKI YolFNHDYnE5IaJsFCgsp3DnvTbAs2pE= X-Google-Smtp-Source: ABdhPJw/xmLmJIGKGVYgH1x/f+iaOst/Nn2dxedzrtxUqC/Tzcj9oxAGZQvDlxbNYOCi6LTdFZvEGg== X-Received: by 2002:a62:6582:: with SMTP id z124mr7599734pfb.250.1598367542459; Tue, 25 Aug 2020 07:59:02 -0700 (PDT) Received: from bobo.ozlabs.ibm.com (61-68-212-105.tpgi.com.au. [61.68.212.105]) by smtp.gmail.com with ESMTPSA id e29sm15755956pfj.92.2020.08.25.07.58.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 25 Aug 2020 07:59:02 -0700 (PDT) From: Nicholas Piggin To: linux-mm@kvack.org, Andrew Morton Cc: Nicholas Piggin , linux-kernel@vger.kernel.org, linux-arch@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, Zefan Li , Jonathan Cameron , Christoph Hellwig , Christophe Leroy Subject: [PATCH v7 12/12] powerpc/64s/radix: Enable huge vmalloc mappings Date: Wed, 26 Aug 2020 00:57:53 +1000 Message-Id: <20200825145753.529284-13-npiggin@gmail.com> X-Mailer: git-send-email 2.23.0 In-Reply-To: <20200825145753.529284-1-npiggin@gmail.com> References: <20200825145753.529284-1-npiggin@gmail.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: D17BE1004C6CD X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Cc: linuxppc-dev@lists.ozlabs.org Signed-off-by: Nicholas Piggin --- Documentation/admin-guide/kernel-parameters.txt | 2 ++ arch/powerpc/Kconfig | 1 + 2 files changed, 3 insertions(+) diff --git a/Documentation/admin-guide/kernel-parameters.txt b/Documentation/admin-guide/kernel-parameters.txt index bdc1f33fd3d1..6f0b41289a90 100644 --- a/Documentation/admin-guide/kernel-parameters.txt +++ b/Documentation/admin-guide/kernel-parameters.txt @@ -3190,6 +3190,8 @@ nohugeiomap [KNL,X86,PPC] Disable kernel huge I/O mappings. + nohugevmalloc [PPC] Disable kernel huge vmalloc mappings. + nosmt [KNL,S390] Disable symmetric multithreading (SMT). Equivalent to smt=1. diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig index 1f48bbfb3ce9..9171d25ad7dc 100644 --- a/arch/powerpc/Kconfig +++ b/arch/powerpc/Kconfig @@ -175,6 +175,7 @@ config PPC select GENERIC_TIME_VSYSCALL select HAVE_ARCH_AUDITSYSCALL select HAVE_ARCH_HUGE_VMAP if PPC_BOOK3S_64 && PPC_RADIX_MMU + select HAVE_ARCH_HUGE_VMALLOC if HAVE_ARCH_HUGE_VMAP select HAVE_ARCH_JUMP_LABEL select HAVE_ARCH_KASAN if PPC32 && PPC_PAGE_SHIFT <= 14 select HAVE_ARCH_KASAN_VMALLOC if PPC32 && PPC_PAGE_SHIFT <= 14