From patchwork Wed Jul 1 08:38:36 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joerg Roedel X-Patchwork-Id: 11635645 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 99D28739 for ; Wed, 1 Jul 2020 08:38:49 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 7217D2074D for ; Wed, 1 Jul 2020 08:38:49 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7217D2074D Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=8bytes.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id BBDCA8D0025; Wed, 1 Jul 2020 04:38:47 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id B2C4E8D0024; Wed, 1 Jul 2020 04:38:47 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9A7F28D0025; Wed, 1 Jul 2020 04:38:47 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0253.hostedemail.com [216.40.44.253]) by kanga.kvack.org (Postfix) with ESMTP id 61AA08D0012 for ; Wed, 1 Jul 2020 04:38:47 -0400 (EDT) Received: from smtpin21.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 2EF91181AC9CC for ; Wed, 1 Jul 2020 08:38:47 +0000 (UTC) X-FDA: 76988856294.21.soup66_190dbad26e7f Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin21.hostedemail.com (Postfix) with ESMTP id B81BE180442C0 for ; Wed, 1 Jul 2020 08:38:43 +0000 (UTC) X-Spam-Summary: 1,0,0,dc4b63a0c37cd775,d41d8cd98f00b204,joro@8bytes.org,,RULES_HIT:41:355:379:541:800:960:973:988:989:1260:1345:1359:1437:1534:1542:1711:1730:1747:1777:1792:2393:2559:2562:2693:2898:3138:3139:3140:3141:3142:3354:3865:3867:3868:3870:3871:3872:4250:4321:5007:6119:6261:7576:10004:11026:11658:11914:12043:12291:12296:12297:12438:12555:12679:12683:12895:12986:13972:14181:14394:14721:21080:21627:21990:30054:30070,0,RBL:81.169.241.247:@8bytes.org:.lbl8.mailshell.net-62.2.6.100 64.100.201.201;04y8y5omx6i48688f3tzgwyxur194opmrboyoo6px9rm7usay6j33yfqe9xe4oq.oupyqwnt8bm1wu84ocbqajdqqa77cyh476jed19fjmf4coaror6uthsw44repo7.e-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:26,LUA_SUMMARY:none X-HE-Tag: soup66_190dbad26e7f X-Filterd-Recvd-Size: 3406 Received: from theia.8bytes.org (8bytes.org [81.169.241.247]) by imf26.hostedemail.com (Postfix) with ESMTP for ; Wed, 1 Jul 2020 08:38:42 +0000 (UTC) Received: by theia.8bytes.org (Postfix, from userid 1000) id 07570217; Wed, 1 Jul 2020 10:38:41 +0200 (CEST) From: Joerg Roedel To: x86@kernel.org Cc: hpa@zytor.com, Dave Hansen , Andy Lutomirski , Peter Zijlstra , Andrew Morton , Steven Rostedt , joro@8bytes.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Joerg Roedel Subject: [PATCH v2 1/3] x86/mm/64: Pre-allocate p4d/pud pages for vmalloc area Date: Wed, 1 Jul 2020 10:38:36 +0200 Message-Id: <20200701083839.19193-2-joro@8bytes.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200701083839.19193-1-joro@8bytes.org> References: <20200701083839.19193-1-joro@8bytes.org> X-Rspamd-Queue-Id: B81BE180442C0 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Joerg Roedel Pre-allocate the page-table pages for the vmalloc area at the level which needs synchronization on x86. This is P4D for 5-level and PUD for 4-level paging. Doing this at boot makes sure all page-tables in the system have these pages already and do not need to be synchronized at runtime. The runtime synchronizatin takes the pgd_lock and iterates over all page-tables in the system, so it can take quite long and is better avoided. Signed-off-by: Joerg Roedel --- arch/x86/mm/init_64.c | 52 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 52 insertions(+) diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c index dbae185511cd..e76bdb001460 100644 --- a/arch/x86/mm/init_64.c +++ b/arch/x86/mm/init_64.c @@ -1238,6 +1238,56 @@ static void __init register_page_bootmem_info(void) #endif } +/* + * Pre-allocates page-table pages for the vmalloc area in the kernel page-table. + * Only the level which needs to be synchronized between all page-tables is + * allocated because the synchronization can be expensive. + */ +static void __init preallocate_vmalloc_pages(void) +{ + unsigned long addr; + const char *lvl; + + for (addr = VMALLOC_START; addr <= VMALLOC_END; addr = ALIGN(addr + 1, PGDIR_SIZE)) { + pgd_t *pgd = pgd_offset_k(addr); + p4d_t *p4d; + pud_t *pud; + + p4d = p4d_offset(pgd, addr); + if (p4d_none(*p4d)) { + /* Can only happen with 5-level paging */ + p4d = p4d_alloc(&init_mm, pgd, addr); + if (!p4d) { + lvl = "p4d"; + goto failed; + } + } + + if (pgtable_l5_enabled()) + continue; + + pud = pud_offset(p4d, addr); + if (pud_none(*pud)) { + /* Ends up here only with 4-level paging */ + pud = pud_alloc(&init_mm, p4d, addr); + if (!pud) { + lvl = "pud"; + goto failed; + } + } + } + + return; + +failed: + + /* + * The pages have to be there now or they will be missing in + * process page-tables later. + */ + panic("Failed to pre-allocate %s pages for vmalloc area\n", lvl); +} + void __init mem_init(void) { pci_iommu_alloc(); @@ -1261,6 +1311,8 @@ void __init mem_init(void) if (get_gate_vma(&init_mm)) kclist_add(&kcore_vsyscall, (void *)VSYSCALL_ADDR, PAGE_SIZE, KCORE_USER); + preallocate_vmalloc_pages(); + mem_init_print_info(NULL); } From patchwork Wed Jul 1 08:38:37 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joerg Roedel X-Patchwork-Id: 11635647 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A448914B7 for ; Wed, 1 Jul 2020 08:38:51 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 72CDD207FB for ; Wed, 1 Jul 2020 08:38:51 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 72CDD207FB Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=8bytes.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 018248D0012; Wed, 1 Jul 2020 04:38:48 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id BF32A8D0026; Wed, 1 Jul 2020 04:38:47 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ADF408D0012; Wed, 1 Jul 2020 04:38:47 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0232.hostedemail.com [216.40.44.232]) by kanga.kvack.org (Postfix) with ESMTP id 884528D0024 for ; Wed, 1 Jul 2020 04:38:47 -0400 (EDT) Received: from smtpin03.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 4513A181ABE85 for ; Wed, 1 Jul 2020 08:38:47 +0000 (UTC) X-FDA: 76988856294.03.brain34_4716a3526e7f Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin03.hostedemail.com (Postfix) with ESMTP id 4546828A24F for ; Wed, 1 Jul 2020 08:38:43 +0000 (UTC) X-Spam-Summary: 1,0,0,f15b23228bb72ca3,d41d8cd98f00b204,joro@8bytes.org,,RULES_HIT:41:355:379:541:800:960:968:988:989:1260:1345:1359:1437:1534:1541:1711:1730:1747:1777:1792:2194:2199:2393:2559:2562:3138:3139:3140:3141:3142:3352:3865:3867:3868:3871:3872:3874:4250:4419:4605:5007:6117:6261:7576:7901:7903:9592:10004:11026:11473:11657:11658:11914:12043:12296:12297:12438:12555:12679:12895:13069:13311:13357:14096:14181:14384:14394:14721:21080:21451:21627:30012:30054,0,RBL:81.169.241.247:@8bytes.org:.lbl8.mailshell.net-62.2.6.100 64.100.201.201;04yrh4z56qqz1crh8w3yk9t9jb76hopadm3aq94h4qd18755mreduhpujiaqnrj.upj65qmwxgw7bxf8cdsun5xt9j5x9yzt33ky7i793cx44xy7trwddi1drwboaio.q-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:38,LUA_SUMMARY:none X-HE-Tag: brain34_4716a3526e7f X-Filterd-Recvd-Size: 2487 Received: from theia.8bytes.org (8bytes.org [81.169.241.247]) by imf41.hostedemail.com (Postfix) with ESMTP for ; Wed, 1 Jul 2020 08:38:42 +0000 (UTC) Received: by theia.8bytes.org (Postfix, from userid 1000) id 36549329; Wed, 1 Jul 2020 10:38:41 +0200 (CEST) From: Joerg Roedel To: x86@kernel.org Cc: hpa@zytor.com, Dave Hansen , Andy Lutomirski , Peter Zijlstra , Andrew Morton , Steven Rostedt , joro@8bytes.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Joerg Roedel Subject: [PATCH v2 2/3] x86/mm/64: Do not sync vmalloc/ioremap mappings Date: Wed, 1 Jul 2020 10:38:37 +0200 Message-Id: <20200701083839.19193-3-joro@8bytes.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200701083839.19193-1-joro@8bytes.org> References: <20200701083839.19193-1-joro@8bytes.org> X-Rspamd-Queue-Id: 4546828A24F X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Joerg Roedel Remove the code to sync the vmalloc and ioremap ranges for x86-64. The page-table pages are all pre-allocated now so that synchronizing the top-level page happens at page-table creation. Signed-off-by: Joerg Roedel --- arch/x86/include/asm/pgtable_64_types.h | 2 -- arch/x86/mm/init_64.c | 5 ----- 2 files changed, 7 deletions(-) diff --git a/arch/x86/include/asm/pgtable_64_types.h b/arch/x86/include/asm/pgtable_64_types.h index 8f63efb2a2cc..52e5f5f2240d 100644 --- a/arch/x86/include/asm/pgtable_64_types.h +++ b/arch/x86/include/asm/pgtable_64_types.h @@ -159,6 +159,4 @@ extern unsigned int ptrs_per_p4d; #define PGD_KERNEL_START ((PAGE_SIZE / 2) / sizeof(pgd_t)) -#define ARCH_PAGE_TABLE_SYNC_MASK (pgtable_l5_enabled() ? PGTBL_PGD_MODIFIED : PGTBL_P4D_MODIFIED) - #endif /* _ASM_X86_PGTABLE_64_DEFS_H */ diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c index e76bdb001460..e0cd2dfd333d 100644 --- a/arch/x86/mm/init_64.c +++ b/arch/x86/mm/init_64.c @@ -217,11 +217,6 @@ void sync_global_pgds(unsigned long start, unsigned long end) sync_global_pgds_l4(start, end); } -void arch_sync_kernel_mappings(unsigned long start, unsigned long end) -{ - sync_global_pgds(start, end); -} - /* * NOTE: This function is marked __ref because it calls __init function * (alloc_bootmem_pages). It's safe to do it ONLY when after_bootmem == 0. From patchwork Wed Jul 1 08:38:38 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joerg Roedel X-Patchwork-Id: 11635643 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A3AC0739 for ; Wed, 1 Jul 2020 08:38:47 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 7B6892078B for ; Wed, 1 Jul 2020 08:38:47 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7B6892078B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=8bytes.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 86DFB8D0023; Wed, 1 Jul 2020 04:38:45 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 81EF38D0012; Wed, 1 Jul 2020 04:38:45 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 70DEB8D0023; Wed, 1 Jul 2020 04:38:45 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0090.hostedemail.com [216.40.44.90]) by kanga.kvack.org (Postfix) with ESMTP id 5162E8D0012 for ; Wed, 1 Jul 2020 04:38:45 -0400 (EDT) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 1A8988248047 for ; Wed, 1 Jul 2020 08:38:45 +0000 (UTC) X-FDA: 76988856210.11.sun53_561741526e7f Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin11.hostedemail.com (Postfix) with ESMTP id 086F8180F8B8C for ; Wed, 1 Jul 2020 08:38:44 +0000 (UTC) X-Spam-Summary: 1,0,0,ba68f83efca2c1dc,d41d8cd98f00b204,joro@8bytes.org,,RULES_HIT:41:355:379:541:800:960:968:988:989:1260:1345:1359:1437:1534:1541:1711:1730:1747:1777:1792:2393:2559:2562:2897:3138:3139:3140:3141:3142:3352:3865:3867:3868:3870:3871:3872:4250:4321:4605:5007:6119:6261:7576:7903:9592:10004:11026:11657:11658:11914:12043:12297:12555:12679:12895:13069:13161:13229:13311:13357:14096:14181:14384:14394:14721:14777:21080:21433:21451:21627:30054,0,RBL:81.169.241.247:@8bytes.org:.lbl8.mailshell.net-64.100.201.201 62.2.6.100;04yfueug4q5haoboy1h358kfb8f4pycysh7ggagdiun7y343sd6rhapffpwdtzd.kyzeyirxiskt7utkzc5rc9pbi1pm3tsr43mp5k9b4kg97deg4cuuepguufihrw5.k-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: sun53_561741526e7f X-Filterd-Recvd-Size: 2475 Received: from theia.8bytes.org (8bytes.org [81.169.241.247]) by imf02.hostedemail.com (Postfix) with ESMTP for ; Wed, 1 Jul 2020 08:38:43 +0000 (UTC) Received: by theia.8bytes.org (Postfix, from userid 1000) id 68D2636B; Wed, 1 Jul 2020 10:38:41 +0200 (CEST) From: Joerg Roedel To: x86@kernel.org Cc: hpa@zytor.com, Dave Hansen , Andy Lutomirski , Peter Zijlstra , Andrew Morton , Steven Rostedt , joro@8bytes.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, Joerg Roedel Subject: [PATCH v2 3/3] x86/mm/64: Make sync_global_pgds() static Date: Wed, 1 Jul 2020 10:38:38 +0200 Message-Id: <20200701083839.19193-4-joro@8bytes.org> X-Mailer: git-send-email 2.17.1 In-Reply-To: <20200701083839.19193-1-joro@8bytes.org> References: <20200701083839.19193-1-joro@8bytes.org> X-Rspamd-Queue-Id: 086F8180F8B8C X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Joerg Roedel The function is only called from within init_64.c by now and can be static. Also remove it from pgtable_64.h. Signed-off-by: Joerg Roedel --- arch/x86/include/asm/pgtable_64.h | 2 -- arch/x86/mm/init_64.c | 2 +- 2 files changed, 1 insertion(+), 3 deletions(-) diff --git a/arch/x86/include/asm/pgtable_64.h b/arch/x86/include/asm/pgtable_64.h index 1b68d24dc6a0..95ac911b1a30 100644 --- a/arch/x86/include/asm/pgtable_64.h +++ b/arch/x86/include/asm/pgtable_64.h @@ -168,8 +168,6 @@ static inline void native_pgd_clear(pgd_t *pgd) native_set_pgd(pgd, native_make_pgd(0)); } -extern void sync_global_pgds(unsigned long start, unsigned long end); - /* * Conversion functions: convert a page and protection to a page entry, * and a page entry and page directory to the page they refer to. diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c index e0cd2dfd333d..e65b96f381a7 100644 --- a/arch/x86/mm/init_64.c +++ b/arch/x86/mm/init_64.c @@ -209,7 +209,7 @@ static void sync_global_pgds_l4(unsigned long start, unsigned long end) * When memory was added make sure all the processes MM have * suitable PGD entries in the local PGD level page. */ -void sync_global_pgds(unsigned long start, unsigned long end) +static void sync_global_pgds(unsigned long start, unsigned long end) { if (pgtable_l5_enabled()) sync_global_pgds_l5(start, end);