From patchwork Wed Nov 18 19:48:39 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Gleixner X-Patchwork-Id: 11915849 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E6206C63697 for ; Wed, 18 Nov 2020 20:58:37 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 40B502075B for ; Wed, 18 Nov 2020 20:58:37 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="4HJFTSRr"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="UXbqqWzH" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 40B502075B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linutronix.de Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 8E8AB6B005C; Wed, 18 Nov 2020 15:58:35 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8A0356B005D; Wed, 18 Nov 2020 15:58:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 737B76B0068; Wed, 18 Nov 2020 15:58:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0098.hostedemail.com [216.40.44.98]) by kanga.kvack.org (Postfix) with ESMTP id 414D26B005C for ; Wed, 18 Nov 2020 15:58:35 -0500 (EST) Received: from smtpin21.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id DFCC48249980 for ; Wed, 18 Nov 2020 20:58:34 +0000 (UTC) X-FDA: 77498752548.21.bone44_3c12f072733d Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin21.hostedemail.com (Postfix) with ESMTP id BBA0A18001C10 for ; Wed, 18 Nov 2020 20:58:34 +0000 (UTC) X-HE-Tag: bone44_3c12f072733d X-Filterd-Recvd-Size: 4438 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by imf01.hostedemail.com (Postfix) with ESMTP for ; Wed, 18 Nov 2020 20:58:34 +0000 (UTC) Message-Id: <20201118204006.869487226@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1605733112; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: references:references; bh=nCDGwh3SdMGSRFUr2TixpTbJs+qPfQ4mGy29He36FSU=; b=4HJFTSRrzyezlZsPToBtWv8leQtJ0MTHzKFJq9o42DBJgVcKgw+i/EzkU/Xu3X6MUOqgip kF+/QL8GARivdO4Vb7/vfEuBq0g4hbpTH6QlEtQlwamJ36b3KVMTjnsHteA7vXYZ4RLJ/a b3YrhuVTye5vTdl48rwxMdjfd3z9xDYCvmX0lajTAbqDUHSB7y+GTypDnX7rbC0ZHy48E/ F9xp/SQzShTSvEauNvYjsfheAOsTV5QAGv3H6B5MdRlTDHURo6/CV4UNlSfiVzG+r/lmET QJxbafIWg1X6GLNk4k96skCri49uU0DWrzbJCuzSAJIexDhJNLzLwqFgMxWEPA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1605733112; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: references:references; bh=nCDGwh3SdMGSRFUr2TixpTbJs+qPfQ4mGy29He36FSU=; b=UXbqqWzHxG8zdORi+anHS8vxHmSweUHPHldVZ8d7X+2hHEpvRgT6noPAN8OMPWrU0JIEOA ctDx7Fudx8RFo2Ag== Date: Wed, 18 Nov 2020 20:48:39 +0100 From: Thomas Gleixner To: LKML Cc: x86@kernel.org, Linus Torvalds , Christoph Hellwig , Matthew Wilcox , Daniel Vetter , Andrew Morton , linux-mm@kvack.org, Linus Torvalds , Peter Zijlstra , Ingo Molnar , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira Subject: [patch V4 1/8] mm/highmem: Provide and use CONFIG_DEBUG_KMAP_LOCAL References: <20201118194838.753436396@linutronix.de> MIME-Version: 1.0 Content-transfer-encoding: 8-bit X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: CONFIG_KMAP_LOCAL can be enabled by x86/32bit even if CONFIG_HIGHMEM is not enabled for temporary MMIO space mappings. Provide it as a seperate config option which depends on CONFIG_KMAP_LOCAL and let CONFIG_DEBUG_HIGHMEM select it. This won't increase the debug coverage of this significantly but it paves the way to do so. Signed-off-by: Thomas Gleixner --- V4: New patch --- include/asm-generic/kmap_size.h | 2 +- lib/Kconfig.debug | 8 ++++++++ mm/highmem.c | 4 ++-- 3 files changed, 11 insertions(+), 3 deletions(-) --- a/include/asm-generic/kmap_size.h +++ b/include/asm-generic/kmap_size.h @@ -3,7 +3,7 @@ #define _ASM_GENERIC_KMAP_SIZE_H /* For debug this provides guard pages between the maps */ -#ifdef CONFIG_DEBUG_HIGHMEM +#ifdef CONFIG_DEBUG_KMAP_LOCAL # define KM_MAX_IDX 33 #else # define KM_MAX_IDX 16 --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -849,9 +849,17 @@ config DEBUG_PER_CPU_MAPS Say N if unsure. +config DEBUG_KMAP_LOCAL + bool "Debug kmap_local temporary mappings" + depends on DEBUG_KERNEL && KMAP_LOCAL + help + This option enables additional error checking for the kmap_local + infrastructure. Disable for production use. + config DEBUG_HIGHMEM bool "Highmem debugging" depends on DEBUG_KERNEL && HIGHMEM + select DEBUG_KMAP_LOCAL help This option enables additional error checking for high memory systems. Disable for production systems. --- a/mm/highmem.c +++ b/mm/highmem.c @@ -368,10 +368,10 @@ EXPORT_SYMBOL(kunmap_high); static DEFINE_PER_CPU(int, __kmap_local_idx); /* - * With DEBUG_HIGHMEM the stack depth is doubled and every second + * With DEBUG_KMAP_LOCAL the stack depth is doubled and every second * slot is unused which acts as a guard page */ -#ifdef CONFIG_DEBUG_HIGHMEM +#ifdef CONFIG_DEBUG_KMAP_LOCAL # define KM_INCR 2 #else # define KM_INCR 1 From patchwork Wed Nov 18 19:48:40 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Gleixner X-Patchwork-Id: 11915851 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 108F8C63798 for ; Wed, 18 Nov 2020 20:58:40 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 7B5312467A for ; Wed, 18 Nov 2020 20:58:39 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="EeSRihQ0"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="ylOSmMyf" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7B5312467A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linutronix.de Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id AE4046B005D; Wed, 18 Nov 2020 15:58:36 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A47B26B0068; Wed, 18 Nov 2020 15:58:36 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8E6506B006C; Wed, 18 Nov 2020 15:58:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0177.hostedemail.com [216.40.44.177]) by kanga.kvack.org (Postfix) with ESMTP id 514D06B005D for ; Wed, 18 Nov 2020 15:58:36 -0500 (EST) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id E4585181AEF23 for ; Wed, 18 Nov 2020 20:58:35 +0000 (UTC) X-FDA: 77498752590.29.cows50_30142e42733d Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin29.hostedemail.com (Postfix) with ESMTP id C298118086CBE for ; Wed, 18 Nov 2020 20:58:35 +0000 (UTC) X-HE-Tag: cows50_30142e42733d X-Filterd-Recvd-Size: 5027 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by imf14.hostedemail.com (Postfix) with ESMTP for ; Wed, 18 Nov 2020 20:58:35 +0000 (UTC) Message-Id: <20201118204007.028261233@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1605733113; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: references:references; bh=7rStCz/mQFScHxK69Y+EQ2P9i7Zl599sIUrIWDCDz3Y=; b=EeSRihQ0Sba8l2old4dtzqJTYc/sprNUmo+vpnwrLXWFt4uNoHTGtAlnxY9ZRZjBcR4ntZ lLOJReiKRK+AKPfirtELCu5M91totpbCYhp/1UJlfRTvjoNVHcKtvjXxxS4A0U9z4X13Bv I6WNVRQEUr+qzSt0Om/tFWCSUMgzLKU0Gv4LQ/Rx5SIY6bYBjOdD03Rje1j2ZBtN0x1V3A TBtwQcU0wZfb10lwg1UoA8XvXtGAzR+c9Dyatw2MyUvTO4AJuZkbPnMMB0th7Fc/wBQpxe PRQgR5fSjqMHjid0TtpsBxpqA98kWoa0cr7HcmlMF/2dKkwxW0GnT3q9xqOjJg== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1605733113; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: references:references; bh=7rStCz/mQFScHxK69Y+EQ2P9i7Zl599sIUrIWDCDz3Y=; b=ylOSmMyfpk722YM3P/eT9vPJFIpXbquk8UhgYQ01gHXosiAAkOxmFtK8EgPdA/aoUpG5rk KijjZHafHfUrgBAA== Date: Wed, 18 Nov 2020 20:48:40 +0100 From: Thomas Gleixner To: LKML Cc: x86@kernel.org, Linus Torvalds , Christoph Hellwig , Matthew Wilcox , Daniel Vetter , Andrew Morton , linux-mm@kvack.org, Linus Torvalds , Peter Zijlstra , Ingo Molnar , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira Subject: [patch V4 2/8] mm/highmem: Provide CONFIG_DEBUG_KMAP_LOCAL_FORCE_MAP References: <20201118194838.753436396@linutronix.de> MIME-Version: 1.0 Content-transfer-encoding: 8-bit X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: CONFIG_DEBUG_KMAP_LOCAL, which is selected by CONFIG_DEBUG_HIGHMEM is only providing guard pages, but does not provide a mechanism to enforce the usage of the kmap_local() infrastructure. Provide CONFIG_DEBUG_KMAP_LOCAL_FORCE_MAP which forces the temporary mapping even for lowmem pages. This needs to be a seperate config switch because this only works on architectures which do not have cache aliasing problems. Suggested-by: Linus Torvalds Signed-off-by: Thomas Gleixner --- V4: New patch --- lib/Kconfig.debug | 14 ++++++++++++++ mm/highmem.c | 12 +++++++++++- 2 files changed, 25 insertions(+), 1 deletion(-) --- a/lib/Kconfig.debug +++ b/lib/Kconfig.debug @@ -856,9 +856,23 @@ config DEBUG_KMAP_LOCAL This option enables additional error checking for the kmap_local infrastructure. Disable for production use. +config ARCH_SUPPORTS_KMAP_LOCAL_FORCE_MAP + bool + +config DEBUG_KMAP_LOCAL_FORCE_MAP + bool "Enforce kmap_local temporary mappings" + depends on DEBUG_KERNEL && ARCH_SUPPORTS_KMAP_LOCAL_FORCE_MAP + select KMAP_LOCAL + select DEBUG_KMAP_LOCAL + help + This option enforces temporary mappings through the kmap_local + mechanism for non-highmem pages and on non-highmem systems. + Disable this for production systems! + config DEBUG_HIGHMEM bool "Highmem debugging" depends on DEBUG_KERNEL && HIGHMEM + select DEBUG_KMAP_LOCAL_FORCE_MAP if ARCH_SUPPORTS_KMAP_LOCAL_FORCE_MAP select DEBUG_KMAP_LOCAL help This option enables additional error checking for high memory --- a/mm/highmem.c +++ b/mm/highmem.c @@ -474,7 +474,12 @@ void *__kmap_local_page_prot(struct page { void *kmap; - if (!PageHighMem(page)) + /* + * To broaden the usage of the actual kmap_local() machinery always map + * pages when debugging is enabled and the architecture has no problems + * with alias mappings. + */ + if (!IS_ENABLED(CONFIG_DEBUG_KMAP_LOCAL_FORCE_MAP) && !PageHighMem(page)) return page_address(page); /* Try kmap_high_get() if architecture has it enabled */ @@ -494,6 +499,11 @@ void kunmap_local_indexed(void *vaddr) if (addr < __fix_to_virt(FIX_KMAP_END) || addr > __fix_to_virt(FIX_KMAP_BEGIN)) { + if (IS_ENABLED(CONFIG_DEBUG_KMAP_LOCAL_FORCE_MAP)) { + /* This _should_ never happen! See above. */ + WARN_ON_ONCE(1); + return; + } /* * Handle mappings which were obtained by kmap_high_get() * first as the virtual address of such mappings is below From patchwork Wed Nov 18 19:48:41 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Gleixner X-Patchwork-Id: 11915853 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 48FC0C56201 for ; Wed, 18 Nov 2020 20:58:42 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B5CAB246CA for ; Wed, 18 Nov 2020 20:58:41 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="XQy8/p7G"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="4by1Q+Zo" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B5CAB246CA Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linutronix.de Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D07EF6B0068; Wed, 18 Nov 2020 15:58:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C90326B006E; Wed, 18 Nov 2020 15:58:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AC2886B0070; Wed, 18 Nov 2020 15:58:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0084.hostedemail.com [216.40.44.84]) by kanga.kvack.org (Postfix) with ESMTP id 68D066B0068 for ; Wed, 18 Nov 2020 15:58:37 -0500 (EST) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 10115181AEF23 for ; Wed, 18 Nov 2020 20:58:37 +0000 (UTC) X-FDA: 77498752674.15.humor51_2d0f76b2733d Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin15.hostedemail.com (Postfix) with ESMTP id E0DD91814B0C7 for ; Wed, 18 Nov 2020 20:58:36 +0000 (UTC) X-HE-Tag: humor51_2d0f76b2733d X-Filterd-Recvd-Size: 5642 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by imf30.hostedemail.com (Postfix) with ESMTP for ; Wed, 18 Nov 2020 20:58:36 +0000 (UTC) Message-Id: <20201118204007.169209557@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1605733115; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: references:references; bh=I1RJ03FxdbQBuix0AFMPFpavs40u1cT6LSvrn3QgZ7A=; b=XQy8/p7GlgJ3dHq0xhwK0Hv31vgwkoS0c6MrnN94LckVmlEpXKxZ1+y+uCAt7d7q9wLtO/ /5bVHjJ/tqymFfxnrvv03YCLbeKSBLcB5a/+yhmyPd6PdDzzP97bccvBGHrlXkBVIHRcuM AMfaHJ6fh9JKoEuPmNoUVRJqaxCGwsHeba/87BOKpei72PIomBEhZcOz3c/QmTQaAq59A1 URFZDUA20v39aciJkfHBAOMNNXYKKZxv4sVVODrqWVrique+0E8vjQT1gq1p9qnMXm1RUO IfqmUddzEmQCGlQoZJDFH76oi5/BPvyp862dx6CyDclqfPaJu4WzaQy6wW6A2A== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1605733115; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: references:references; bh=I1RJ03FxdbQBuix0AFMPFpavs40u1cT6LSvrn3QgZ7A=; b=4by1Q+ZoAv1joVlLAjkmqwqynxa5rMU8TsHMLVejW5JQDqM2F19I0JMoSsby1aIMdNJDGy LbJcaMeAs1OuT1BQ== Date: Wed, 18 Nov 2020 20:48:41 +0100 From: Thomas Gleixner To: LKML Cc: x86@kernel.org, Linus Torvalds , Christoph Hellwig , Matthew Wilcox , Daniel Vetter , Andrew Morton , linux-mm@kvack.org, Linus Torvalds , Peter Zijlstra , Ingo Molnar , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira Subject: [patch V4 3/8] x86: Support kmap_local() forced debugging References: <20201118194838.753436396@linutronix.de> MIME-Version: 1.0 Content-transfer-encoding: 8-bit X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: kmap_local() and related interfaces are NOOPs on 64bit and only create temporary fixmaps for highmem pages on 32bit. That means the test coverage for this code is pretty small. CONFIG_KMAP_LOCAL can be enabled independent from CONFIG_HIGHMEM, which allows to provide support for enforced kmap_local() debugging even on 64bit. For 32bit the support is unconditional, for 64bit it's only supported when CONFIG_NR_CPUS <= 4096 as supporting it for 8192 CPUs would require to set up yet another fixmap PGT. If CONFIG_KMAP_LOCAL_FORCE_DEBUG is enabled then kmap_local()/kmap_atomic() will use the temporary fixmap mapping path. Signed-off-by: Thomas Gleixner --- V4: New patch --- arch/x86/Kconfig | 1 + arch/x86/include/asm/fixmap.h | 12 +++++++++--- arch/x86/include/asm/pgtable_64_types.h | 6 +++++- 3 files changed, 15 insertions(+), 4 deletions(-) --- a/arch/x86/Kconfig +++ b/arch/x86/Kconfig @@ -93,6 +93,7 @@ config X86 select ARCH_SUPPORTS_ACPI select ARCH_SUPPORTS_ATOMIC_RMW select ARCH_SUPPORTS_NUMA_BALANCING if X86_64 + select ARCH_SUPPORTS_KMAP_LOCAL_FORCE_MAP if NR_CPUS <= 4096 select ARCH_USE_BUILTIN_BSWAP select ARCH_USE_QUEUED_RWLOCKS select ARCH_USE_QUEUED_SPINLOCKS --- a/arch/x86/include/asm/fixmap.h +++ b/arch/x86/include/asm/fixmap.h @@ -14,13 +14,20 @@ #ifndef _ASM_X86_FIXMAP_H #define _ASM_X86_FIXMAP_H +#include + /* * Exposed to assembly code for setting up initial page tables. Cannot be * calculated in assembly code (fixmap entries are an enum), but is sanity * checked in the actual fixmap C code to make sure that the fixmap is * covered fully. */ -#define FIXMAP_PMD_NUM 2 +#ifndef CONFIG_DEBUG_KMAP_LOCAL_FORCE_MAP +# define FIXMAP_PMD_NUM 2 +#else +# define KM_PMDS (KM_MAX_IDX * ((CONFIG_NR_CPUS + 511) / 512)) +# define FIXMAP_PMD_NUM (KM_PMDS + 2) +#endif /* fixmap starts downwards from the 507th entry in level2_fixmap_pgt */ #define FIXMAP_PMD_TOP 507 @@ -31,7 +38,6 @@ #include #ifdef CONFIG_X86_32 #include -#include #else #include #endif @@ -92,7 +98,7 @@ enum fixed_addresses { FIX_IO_APIC_BASE_0, FIX_IO_APIC_BASE_END = FIX_IO_APIC_BASE_0 + MAX_IO_APICS - 1, #endif -#ifdef CONFIG_X86_32 +#ifdef CONFIG_KMAP_LOCAL FIX_KMAP_BEGIN, /* reserved pte's for temporary kernel mappings */ FIX_KMAP_END = FIX_KMAP_BEGIN + (KM_MAX_IDX * NR_CPUS) - 1, #ifdef CONFIG_PCI_MMCONFIG --- a/arch/x86/include/asm/pgtable_64_types.h +++ b/arch/x86/include/asm/pgtable_64_types.h @@ -143,7 +143,11 @@ extern unsigned int ptrs_per_p4d; #define MODULES_VADDR (__START_KERNEL_map + KERNEL_IMAGE_SIZE) /* The module sections ends with the start of the fixmap */ -#define MODULES_END _AC(0xffffffffff000000, UL) +#ifndef CONFIG_DEBUG_KMAP_LOCAL_FORCE_MAP +# define MODULES_END _AC(0xffffffffff000000, UL) +#else +# define MODULES_END _AC(0xfffffffffe000000, UL) +#endif #define MODULES_LEN (MODULES_END - MODULES_VADDR) #define ESPFIX_PGD_ENTRY _AC(-2, UL) From patchwork Wed Nov 18 19:48:42 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Gleixner X-Patchwork-Id: 11915855 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AFC9CC5519F for ; Wed, 18 Nov 2020 20:58:44 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1AA402467A for ; Wed, 18 Nov 2020 20:58:44 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="FUj46FGy"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="tzKY6zRD" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1AA402467A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linutronix.de Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9D2DC6B0070; Wed, 18 Nov 2020 15:58:39 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 965BA6B0071; Wed, 18 Nov 2020 15:58:39 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7B6AD6B0072; Wed, 18 Nov 2020 15:58:39 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0026.hostedemail.com [216.40.44.26]) by kanga.kvack.org (Postfix) with ESMTP id 4772F6B0070 for ; Wed, 18 Nov 2020 15:58:39 -0500 (EST) Received: from smtpin18.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id E70C8363B for ; Wed, 18 Nov 2020 20:58:38 +0000 (UTC) X-FDA: 77498752716.18.class64_2a0caac2733d Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin18.hostedemail.com (Postfix) with ESMTP id C5084100EDBE9 for ; Wed, 18 Nov 2020 20:58:38 +0000 (UTC) X-HE-Tag: class64_2a0caac2733d X-Filterd-Recvd-Size: 10771 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by imf03.hostedemail.com (Postfix) with ESMTP for ; Wed, 18 Nov 2020 20:58:37 +0000 (UTC) Message-Id: <20201118204007.269943012@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1605733116; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: references:references; bh=UzadNpr0ndJGp+OFzrQQljNjx0AFpfujd54lFJlXEXU=; b=FUj46FGya6FXvZsox4PUoQ4SbVEK3Cc5BlR+2K6kTs6q0qgseREgOEXLLPqYLvY8412TW4 jPrATE3tXRNjzaAufpkgKgnAfrwkg4sAgXJQ5EanSnlzVmmSYyHU3Gnh8gYnFCQyC2SYo6 OhMWuahni3NeahZ7pNl9lw9LCuuWdQnVSyUupeWEy73YTzXI3967CxFN3y3ASmFuer3zpf trE6zg9xSMJn4EqWxTK+io1x2NrGVfXdCR3al5PtxfXJXyGD8/UlYdn/LiTCt0OAt61sao J8p1SloUlqIhLDNOQTYiTqiwMChNtfu8vXphyxzuSsgvO1gHMAZ9PDd+mvF/lw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1605733116; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: references:references; bh=UzadNpr0ndJGp+OFzrQQljNjx0AFpfujd54lFJlXEXU=; b=tzKY6zRDXY4PT13iNIKyGcQBYChGgxflJNhLAGRVBdsWiWeRXLR1tAUoicXF7+11slBtBj H8BIyFaSARykX1DA== Date: Wed, 18 Nov 2020 20:48:42 +0100 From: Thomas Gleixner To: LKML Cc: x86@kernel.org, Linus Torvalds , Christoph Hellwig , Matthew Wilcox , Daniel Vetter , Andrew Morton , linux-mm@kvack.org, Peter Zijlstra , Ingo Molnar , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira , Linus Torvalds Subject: [patch V4 4/8] sched: Make migrate_disable/enable() independent of RT References: <20201118194838.753436396@linutronix.de> MIME-Version: 1.0 Content-transfer-encoding: 8-bit X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Thomas Gleixner Now that the scheduler can deal with migrate disable properly, there is no real compelling reason to make it only available for RT. There are quite some code pathes which needlessly disable preemption in order to prevent migration and some constructs like kmap_atomic() enforce it implicitly. Making it available independent of RT allows to provide a preemptible variant of kmap_atomic() and makes the code more consistent in general. FIXME: Rework the comment in preempt.h - Peter? Signed-off-by: Thomas Gleixner Cc: Peter Zijlstra Cc: Ingo Molnar Cc: Juri Lelli Cc: Vincent Guittot Cc: Dietmar Eggemann Cc: Steven Rostedt Cc: Ben Segall Cc: Mel Gorman Cc: Daniel Bristot de Oliveira --- include/linux/kernel.h | 21 ++++++++++++++------- include/linux/preempt.h | 38 +++----------------------------------- include/linux/sched.h | 2 +- kernel/sched/core.c | 45 +++++++++++++++++++++++++++++++++++---------- kernel/sched/sched.h | 4 ++-- lib/smp_processor_id.c | 2 +- 6 files changed, 56 insertions(+), 56 deletions(-) --- a/include/linux/kernel.h +++ b/include/linux/kernel.h @@ -204,6 +204,7 @@ extern int _cond_resched(void); extern void ___might_sleep(const char *file, int line, int preempt_offset); extern void __might_sleep(const char *file, int line, int preempt_offset); extern void __cant_sleep(const char *file, int line, int preempt_offset); +extern void __cant_migrate(const char *file, int line); /** * might_sleep - annotation for functions that can sleep @@ -227,6 +228,18 @@ extern void __cant_sleep(const char *fil # define cant_sleep() \ do { __cant_sleep(__FILE__, __LINE__, 0); } while (0) # define sched_annotate_sleep() (current->task_state_change = 0) + +/** + * cant_migrate - annotation for functions that cannot migrate + * + * Will print a stack trace if executed in code which is migratable + */ +# define cant_migrate() \ + do { \ + if (IS_ENABLED(CONFIG_SMP)) \ + __cant_migrate(__FILE__, __LINE__); \ + } while (0) + /** * non_block_start - annotate the start of section where sleeping is prohibited * @@ -251,6 +264,7 @@ extern void __cant_sleep(const char *fil int preempt_offset) { } # define might_sleep() do { might_resched(); } while (0) # define cant_sleep() do { } while (0) +# define cant_migrate() do { } while (0) # define sched_annotate_sleep() do { } while (0) # define non_block_start() do { } while (0) # define non_block_end() do { } while (0) @@ -258,13 +272,6 @@ extern void __cant_sleep(const char *fil #define might_sleep_if(cond) do { if (cond) might_sleep(); } while (0) -#ifndef CONFIG_PREEMPT_RT -# define cant_migrate() cant_sleep() -#else - /* Placeholder for now */ -# define cant_migrate() do { } while (0) -#endif - /** * abs - return absolute value of an argument * @x: the value. If it is unsigned type, it is converted to signed type first. --- a/include/linux/preempt.h +++ b/include/linux/preempt.h @@ -322,7 +322,7 @@ static inline void preempt_notifier_init #endif -#if defined(CONFIG_SMP) && defined(CONFIG_PREEMPT_RT) +#ifdef CONFIG_SMP /* * Migrate-Disable and why it is undesired. @@ -382,43 +382,11 @@ static inline void preempt_notifier_init extern void migrate_disable(void); extern void migrate_enable(void); -#elif defined(CONFIG_PREEMPT_RT) +#else static inline void migrate_disable(void) { } static inline void migrate_enable(void) { } -#else /* !CONFIG_PREEMPT_RT */ - -/** - * migrate_disable - Prevent migration of the current task - * - * Maps to preempt_disable() which also disables preemption. Use - * migrate_disable() to annotate that the intent is to prevent migration, - * but not necessarily preemption. - * - * Can be invoked nested like preempt_disable() and needs the corresponding - * number of migrate_enable() invocations. - */ -static __always_inline void migrate_disable(void) -{ - preempt_disable(); -} - -/** - * migrate_enable - Allow migration of the current task - * - * Counterpart to migrate_disable(). - * - * As migrate_disable() can be invoked nested, only the outermost invocation - * reenables migration. - * - * Currently mapped to preempt_enable(). - */ -static __always_inline void migrate_enable(void) -{ - preempt_enable(); -} - -#endif /* CONFIG_SMP && CONFIG_PREEMPT_RT */ +#endif /* CONFIG_SMP */ #endif /* __LINUX_PREEMPT_H */ --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -715,7 +715,7 @@ struct task_struct { const cpumask_t *cpus_ptr; cpumask_t cpus_mask; void *migration_pending; -#if defined(CONFIG_SMP) && defined(CONFIG_PREEMPT_RT) +#ifdef CONFIG_SMP unsigned short migration_disabled; #endif unsigned short migration_flags; --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -1696,8 +1696,6 @@ void check_preempt_curr(struct rq *rq, s #ifdef CONFIG_SMP -#ifdef CONFIG_PREEMPT_RT - static void __do_set_cpus_allowed(struct task_struct *p, const struct cpumask *new_mask, u32 flags); @@ -1768,8 +1766,6 @@ static inline bool rq_has_pinned_tasks(s return rq->nr_pinned; } -#endif - /* * Per-CPU kthreads are allowed to run on !active && online CPUs, see * __set_cpus_allowed_ptr() and select_fallback_rq(). @@ -2839,7 +2835,7 @@ void sched_set_stop_task(int cpu, struct } } -#else +#else /* CONFIG_SMP */ static inline int __set_cpus_allowed_ptr(struct task_struct *p, const struct cpumask *new_mask, @@ -2848,10 +2844,6 @@ static inline int __set_cpus_allowed_ptr return set_cpus_allowed_ptr(p, new_mask); } -#endif /* CONFIG_SMP */ - -#if !defined(CONFIG_SMP) || !defined(CONFIG_PREEMPT_RT) - static inline void migrate_disable_switch(struct rq *rq, struct task_struct *p) { } static inline bool rq_has_pinned_tasks(struct rq *rq) @@ -2859,7 +2851,7 @@ static inline bool rq_has_pinned_tasks(s return false; } -#endif +#endif /* !CONFIG_SMP */ static void ttwu_stat(struct task_struct *p, int cpu, int wake_flags) @@ -7881,6 +7873,39 @@ void __cant_sleep(const char *file, int add_taint(TAINT_WARN, LOCKDEP_STILL_OK); } EXPORT_SYMBOL_GPL(__cant_sleep); + +#ifdef CONFIG_SMP +void __cant_migrate(const char *file, int line) +{ + static unsigned long prev_jiffy; + + if (irqs_disabled()) + return; + + if (is_migration_disabled(current)) + return; + + if (!IS_ENABLED(CONFIG_PREEMPT_COUNT)) + return; + + if (preempt_count() > 0) + return; + + if (time_before(jiffies, prev_jiffy + HZ) && prev_jiffy) + return; + prev_jiffy = jiffies; + + pr_err("BUG: assuming non migratable context at %s:%d\n", file, line); + pr_err("in_atomic(): %d, irqs_disabled(): %d, migration_disabled() %u pid: %d, name: %s\n", + in_atomic(), irqs_disabled(), is_migration_disabled(current), + current->pid, current->comm); + + debug_show_held_locks(current); + dump_stack(); + add_taint(TAINT_WARN, LOCKDEP_STILL_OK); +} +EXPORT_SYMBOL_GPL(__cant_migrate); +#endif #endif #ifdef CONFIG_MAGIC_SYSRQ --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -1056,7 +1056,7 @@ struct rq { struct cpuidle_state *idle_state; #endif -#if defined(CONFIG_PREEMPT_RT) && defined(CONFIG_SMP) +#ifdef CONFIG_SMP unsigned int nr_pinned; #endif unsigned int push_busy; @@ -1092,7 +1092,7 @@ static inline int cpu_of(struct rq *rq) static inline bool is_migration_disabled(struct task_struct *p) { -#if defined(CONFIG_SMP) && defined(CONFIG_PREEMPT_RT) +#ifdef CONFIG_SMP return p->migration_disabled; #else return false; --- a/lib/smp_processor_id.c +++ b/lib/smp_processor_id.c @@ -26,7 +26,7 @@ unsigned int check_preemption_disabled(c if (current->nr_cpus_allowed == 1) goto out; -#if defined(CONFIG_SMP) && defined(CONFIG_PREEMPT_RT) +#ifdef CONFIG_SMP if (current->migration_disabled) goto out; #endif From patchwork Wed Nov 18 19:48:43 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Gleixner X-Patchwork-Id: 11915857 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AD4C7C56201 for ; Wed, 18 Nov 2020 20:58:47 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 2695F2075B for ; Wed, 18 Nov 2020 20:58:47 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="N4TFAvl1"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="flXm2zVt" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2695F2075B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linutronix.de Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 2A5926B0071; Wed, 18 Nov 2020 15:58:40 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 256396B0072; Wed, 18 Nov 2020 15:58:40 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 145726B0073; Wed, 18 Nov 2020 15:58:40 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0110.hostedemail.com [216.40.44.110]) by kanga.kvack.org (Postfix) with ESMTP id DAED96B0071 for ; Wed, 18 Nov 2020 15:58:39 -0500 (EST) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 7FE1A1EE6 for ; Wed, 18 Nov 2020 20:58:39 +0000 (UTC) X-FDA: 77498752758.12.alley86_39011e62733d Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin12.hostedemail.com (Postfix) with ESMTP id 58F361803B045 for ; Wed, 18 Nov 2020 20:58:39 +0000 (UTC) X-HE-Tag: alley86_39011e62733d X-Filterd-Recvd-Size: 11338 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by imf38.hostedemail.com (Postfix) with ESMTP for ; Wed, 18 Nov 2020 20:58:38 +0000 (UTC) Message-Id: <20201118204007.372935758@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1605733117; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: references:references; bh=iMrPjcxdjzJ6nBwnbJ/7bEevJXJleTg2rvmNQdnn/sM=; b=N4TFAvl1VSsvn7+DywqSgi0O8kJSvo40oLRVWy90xYk9MEVp45sohhEMT3rRylyNMgum9I 5mBVDT+Jt7DAQ7ANEbmaBI93rMXNa0ODO4qFEmof28ORws+HGSKfEBVW3P5VHxwSPkYdpO NH1G9fUt4tinnnb4LnS9WeQKanP4t1Cfrp+TunDl0zl4577zZhRT3CGEpU2nkKR2BD+PQ3 8UX1JM2eCADCRok8dI1aENLdHZysktNfRIkczABLmjJJAQzPwVAaQ5Ke4ymWoKPeFJnnA2 avQp/pIs0IgYa+9OZ9m0O4y7H5VOcFIglmPjGPFsklw1lTbKD5k0sqJ6SC1Lhw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1605733117; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: references:references; bh=iMrPjcxdjzJ6nBwnbJ/7bEevJXJleTg2rvmNQdnn/sM=; b=flXm2zVtGOL7YWecCTO8cbrvF1r12McrAs0TGhU/pXO9P5B+ayND3VNeVMLc1cKUAkpaw6 RO3vk68e7We9m3AQ== Date: Wed, 18 Nov 2020 20:48:43 +0100 From: Thomas Gleixner To: LKML Cc: x86@kernel.org, Linus Torvalds , Christoph Hellwig , Matthew Wilcox , Daniel Vetter , Andrew Morton , linux-mm@kvack.org, Linus Torvalds , Peter Zijlstra , Ingo Molnar , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira Subject: [patch V4 5/8] sched: highmem: Store local kmaps in task struct References: <20201118194838.753436396@linutronix.de> MIME-Version: 1.0 Content-transfer-encoding: 8-bit X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Thomas Gleixner Instead of storing the map per CPU provide and use per task storage. That prepares for local kmaps which are preemptible. The context switch code is preparatory and not yet in use because kmap_atomic() runs with preemption disabled. Will be made usable in the next step. The context switch logic is safe even when an interrupt happens after clearing or before restoring the kmaps. The kmap index in task struct is not modified so any nesting kmap in an interrupt will use unused indices and on return the counter is the same as before. Also add an assert into the return to user space code. Going back to user space with an active kmap local is a nono. Signed-off-by: Thomas Gleixner --- V4: Use the version which actually compiles and works V3: Handle the debug case correctly --- include/linux/highmem-internal.h | 10 +++ include/linux/sched.h | 9 +++ kernel/entry/common.c | 2 kernel/fork.c | 1 kernel/sched/core.c | 18 +++++++ mm/highmem.c | 99 +++++++++++++++++++++++++++++++++++---- 6 files changed, 129 insertions(+), 10 deletions(-) --- a/include/linux/highmem-internal.h +++ b/include/linux/highmem-internal.h @@ -9,6 +9,16 @@ void *__kmap_local_pfn_prot(unsigned long pfn, pgprot_t prot); void *__kmap_local_page_prot(struct page *page, pgprot_t prot); void kunmap_local_indexed(void *vaddr); +void kmap_local_fork(struct task_struct *tsk); +void __kmap_local_sched_out(void); +void __kmap_local_sched_in(void); +static inline void kmap_assert_nomap(void) +{ + DEBUG_LOCKS_WARN_ON(current->kmap_ctrl.idx); +} +#else +static inline void kmap_local_fork(struct task_struct *tsk) { } +static inline void kmap_assert_nomap(void) { } #endif #ifdef CONFIG_HIGHMEM --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -34,6 +34,7 @@ #include #include #include +#include /* task_struct member predeclarations (sorted alphabetically): */ struct audit_context; @@ -629,6 +630,13 @@ struct wake_q_node { struct wake_q_node *next; }; +struct kmap_ctrl { +#ifdef CONFIG_KMAP_LOCAL + int idx; + pte_t pteval[KM_MAX_IDX]; +#endif +}; + struct task_struct { #ifdef CONFIG_THREAD_INFO_IN_TASK /* @@ -1294,6 +1302,7 @@ struct task_struct { unsigned int sequential_io; unsigned int sequential_io_avg; #endif + struct kmap_ctrl kmap_ctrl; #ifdef CONFIG_DEBUG_ATOMIC_SLEEP unsigned long task_state_change; #endif --- a/kernel/entry/common.c +++ b/kernel/entry/common.c @@ -2,6 +2,7 @@ #include #include +#include #include #include @@ -194,6 +195,7 @@ static void exit_to_user_mode_prepare(st /* Ensure that the address limit is intact and no locks are held */ addr_limit_user_check(); + kmap_assert_nomap(); lockdep_assert_irqs_disabled(); lockdep_sys_exit(); } --- a/kernel/fork.c +++ b/kernel/fork.c @@ -930,6 +930,7 @@ static struct task_struct *dup_task_stru account_kernel_stack(tsk, 1); kcov_task_init(tsk); + kmap_local_fork(tsk); #ifdef CONFIG_FAULT_INJECTION tsk->fail_nth = 0; --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -4051,6 +4051,22 @@ static inline void finish_lock_switch(st # define finish_arch_post_lock_switch() do { } while (0) #endif +static inline void kmap_local_sched_out(void) +{ +#ifdef CONFIG_KMAP_LOCAL + if (unlikely(current->kmap_ctrl.idx)) + __kmap_local_sched_out(); +#endif +} + +static inline void kmap_local_sched_in(void) +{ +#ifdef CONFIG_KMAP_LOCAL + if (unlikely(current->kmap_ctrl.idx)) + __kmap_local_sched_in(); +#endif +} + /** * prepare_task_switch - prepare to switch tasks * @rq: the runqueue preparing to switch @@ -4073,6 +4089,7 @@ prepare_task_switch(struct rq *rq, struc perf_event_task_sched_out(prev, next); rseq_preempt(prev); fire_sched_out_preempt_notifiers(prev, next); + kmap_local_sched_out(); prepare_task(next); prepare_arch_switch(next); } @@ -4139,6 +4156,7 @@ static struct rq *finish_task_switch(str finish_lock_switch(rq); finish_arch_post_lock_switch(); kcov_finish_switch(current); + kmap_local_sched_in(); fire_sched_in_preempt_notifiers(current); /* --- a/mm/highmem.c +++ b/mm/highmem.c @@ -365,8 +365,6 @@ EXPORT_SYMBOL(kunmap_high); #include -static DEFINE_PER_CPU(int, __kmap_local_idx); - /* * With DEBUG_KMAP_LOCAL the stack depth is doubled and every second * slot is unused which acts as a guard page @@ -379,23 +377,21 @@ static DEFINE_PER_CPU(int, __kmap_local_ static inline int kmap_local_idx_push(void) { - int idx = __this_cpu_add_return(__kmap_local_idx, KM_INCR) - 1; - WARN_ON_ONCE(in_irq() && !irqs_disabled()); - BUG_ON(idx >= KM_MAX_IDX); - return idx; + current->kmap_ctrl.idx += KM_INCR; + BUG_ON(current->kmap_ctrl.idx >= KM_MAX_IDX); + return current->kmap_ctrl.idx - 1; } static inline int kmap_local_idx(void) { - return __this_cpu_read(__kmap_local_idx) - 1; + return current->kmap_ctrl.idx - 1; } static inline void kmap_local_idx_pop(void) { - int idx = __this_cpu_sub_return(__kmap_local_idx, KM_INCR); - - BUG_ON(idx < 0); + current->kmap_ctrl.idx -= KM_INCR; + BUG_ON(current->kmap_ctrl.idx < 0); } #ifndef arch_kmap_local_post_map @@ -464,6 +460,7 @@ void *__kmap_local_pfn_prot(unsigned lon pteval = pfn_pte(pfn, prot); set_pte_at(&init_mm, vaddr, kmap_pte - idx, pteval); arch_kmap_local_post_map(vaddr, pteval); + current->kmap_ctrl.pteval[kmap_local_idx()] = pteval; preempt_enable(); return (void *)vaddr; @@ -522,10 +519,92 @@ void kunmap_local_indexed(void *vaddr) arch_kmap_local_pre_unmap(addr); pte_clear(&init_mm, addr, kmap_pte - idx); arch_kmap_local_post_unmap(addr); + current->kmap_ctrl.pteval[kmap_local_idx()] = __pte(0); kmap_local_idx_pop(); preempt_enable(); } EXPORT_SYMBOL(kunmap_local_indexed); + +/* + * Invoked before switch_to(). This is safe even when during or after + * clearing the maps an interrupt which needs a kmap_local happens because + * the task::kmap_ctrl.idx is not modified by the unmapping code so a + * nested kmap_local will use the next unused index and restore the index + * on unmap. The already cleared kmaps of the outgoing task are irrelevant + * because the interrupt context does not know about them. The same applies + * when scheduling back in for an interrupt which happens before the + * restore is complete. + */ +void __kmap_local_sched_out(void) +{ + struct task_struct *tsk = current; + pte_t *kmap_pte = kmap_get_pte(); + int i; + + /* Clear kmaps */ + for (i = 0; i < tsk->kmap_ctrl.idx; i++) { + pte_t pteval = tsk->kmap_ctrl.pteval[i]; + unsigned long addr; + int idx; + + /* With debug all even slots are unmapped and act as guard */ + if (IS_ENABLED(CONFIG_DEBUG_HIGHMEM) && !(i & 0x01)) { + WARN_ON_ONCE(!pte_none(pteval)); + continue; + } + if (WARN_ON_ONCE(pte_none(pteval))) + continue; + + /* + * This is a horrible hack for XTENSA to calculate the + * coloured PTE index. Uses the PFN encoded into the pteval + * and the map index calculation because the actual mapped + * virtual address is not stored in task::kmap_ctrl. + * For any sane architecture this is optimized out. + */ + idx = arch_kmap_local_map_idx(i, pte_pfn(pteval)); + + addr = __fix_to_virt(FIX_KMAP_BEGIN + idx); + arch_kmap_local_pre_unmap(addr); + pte_clear(&init_mm, addr, kmap_pte - idx); + arch_kmap_local_post_unmap(addr); + } +} + +void __kmap_local_sched_in(void) +{ + struct task_struct *tsk = current; + pte_t *kmap_pte = kmap_get_pte(); + int i; + + /* Restore kmaps */ + for (i = 0; i < tsk->kmap_ctrl.idx; i++) { + pte_t pteval = tsk->kmap_ctrl.pteval[i]; + unsigned long addr; + int idx; + + /* With debug all even slots are unmapped and act as guard */ + if (IS_ENABLED(CONFIG_DEBUG_HIGHMEM) && !(i & 0x01)) { + WARN_ON_ONCE(!pte_none(pteval)); + continue; + } + if (WARN_ON_ONCE(pte_none(pteval))) + continue; + + /* See comment in __kmap_local_sched_out() */ + idx = arch_kmap_local_map_idx(i, pte_pfn(pteval)); + addr = __fix_to_virt(FIX_KMAP_BEGIN + idx); + set_pte_at(&init_mm, addr, kmap_pte - idx, pteval); + arch_kmap_local_post_map(addr, pteval); + } +} + +void kmap_local_fork(struct task_struct *tsk) +{ + if (WARN_ON_ONCE(tsk->kmap_ctrl.idx)) + memset(&tsk->kmap_ctrl, 0, sizeof(tsk->kmap_ctrl)); +} + #endif #if defined(HASHED_PAGE_VIRTUAL) From patchwork Wed Nov 18 19:48:44 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Gleixner X-Patchwork-Id: 11915861 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B0497C5519F for ; Wed, 18 Nov 2020 20:58:52 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1F4742075B for ; Wed, 18 Nov 2020 20:58:50 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="wv5xoqCg"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="U5mYYq4f" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1F4742075B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linutronix.de Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 69C5B6B0073; Wed, 18 Nov 2020 15:58:41 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 64D2A6B0075; Wed, 18 Nov 2020 15:58:41 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4ED5D6B0074; Wed, 18 Nov 2020 15:58:41 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0180.hostedemail.com [216.40.44.180]) by kanga.kvack.org (Postfix) with ESMTP id 216A86B0072 for ; Wed, 18 Nov 2020 15:58:41 -0500 (EST) Received: from smtpin06.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id BD2B7181AEF23 for ; Wed, 18 Nov 2020 20:58:40 +0000 (UTC) X-FDA: 77498752800.06.cough72_04128922733d Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin06.hostedemail.com (Postfix) with ESMTP id 9E7DD10048481 for ; Wed, 18 Nov 2020 20:58:40 +0000 (UTC) X-HE-Tag: cough72_04128922733d X-Filterd-Recvd-Size: 8892 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by imf28.hostedemail.com (Postfix) with ESMTP for ; Wed, 18 Nov 2020 20:58:39 +0000 (UTC) Message-Id: <20201118204007.468533059@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1605733118; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: references:references; bh=uloIfoEWS8rGSRg5i+oo5e3+6MFuyqJgSste+GxuC2c=; b=wv5xoqCgASbAT8uNifalcqb6U++yZ+uZ80ZCbfVTlCmtxYWdMwA6ItLhG8rZFjledd2oaa GwiD/2eKG56RHDkOiA5mXdzvGIVRxXfI3waeFUhlsCY7waCKgQpibSEUvnRCBd4Fe9rFx2 lLXb+kTW2kqMBELdxOQiVyyOQvempVYoBOL2cK8nOGCCm5hi9fuZUGuerkWS5fLLuUx+dv zoyprrk5CCtILj3fOtG8ShCCr2FH2eCptbhD/KhoPjQ2JqYdQvs6MwcbyJIsuR04LceRiZ +zOgjVLHwK08verEmB9P7AsTFfnrMvX1D5PnOdyMnGU8irxboN3WZE6tqPDOCg== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1605733118; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: references:references; bh=uloIfoEWS8rGSRg5i+oo5e3+6MFuyqJgSste+GxuC2c=; b=U5mYYq4f2t0t2g6MUw49vA6MYxd9I+aJOIQ3O+KKFHHCZ+9xzNr/t+Ls1eOvt3//Jyd/Vb 1soIRKWD10NR9LCw== Date: Wed, 18 Nov 2020 20:48:44 +0100 From: Thomas Gleixner To: LKML Cc: x86@kernel.org, Linus Torvalds , Christoph Hellwig , Matthew Wilcox , Daniel Vetter , Andrew Morton , linux-mm@kvack.org, Linus Torvalds , Peter Zijlstra , Ingo Molnar , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira Subject: [patch V4 6/8] mm/highmem: Provide kmap_local* References: <20201118194838.753436396@linutronix.de> MIME-Version: 1.0 Content-transfer-encoding: 8-bit X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Thomas Gleixner Now that the kmap atomic index is stored in task struct provide a preemptible variant. On context switch the maps of an outgoing task are removed and the map of the incoming task are restored. That's obviously slow, but highmem is slow anyway. The kmap_local.*() functions can be invoked from both preemptible and atomic context. kmap local sections disable migration to keep the resulting virtual mapping address correct, but disable neither pagefaults nor preemption. A wholesale conversion of kmap_atomic to be fully preemptible is not possible because some of the usage sites might rely on the preemption disable for serialization or on the implicit pagefault disable. Needs to be done on a case by case basis. Signed-off-by: Thomas Gleixner --- V3: Move migrate disable into the actual highmem mapping code so it only affects real highmem mappings. V2: Make it more consistent and add commentry --- include/linux/highmem-internal.h | 48 +++++++++++++++++++++++++++++++++++++++ include/linux/highmem.h | 43 +++++++++++++++++++++------------- mm/highmem.c | 6 ++++ 3 files changed, 81 insertions(+), 16 deletions(-) --- a/include/linux/highmem-internal.h +++ b/include/linux/highmem-internal.h @@ -68,6 +68,26 @@ static inline void kmap_flush_unused(voi __kmap_flush_unused(); } +static inline void *kmap_local_page(struct page *page) +{ + return __kmap_local_page_prot(page, kmap_prot); +} + +static inline void *kmap_local_page_prot(struct page *page, pgprot_t prot) +{ + return __kmap_local_page_prot(page, prot); +} + +static inline void *kmap_local_pfn(unsigned long pfn) +{ + return __kmap_local_pfn_prot(pfn, kmap_prot); +} + +static inline void __kunmap_local(void *vaddr) +{ + kunmap_local_indexed(vaddr); +} + static inline void *kmap_atomic_prot(struct page *page, pgprot_t prot) { preempt_disable(); @@ -140,6 +160,28 @@ static inline void kunmap(struct page *p #endif } +static inline void *kmap_local_page(struct page *page) +{ + return page_address(page); +} + +static inline void *kmap_local_page_prot(struct page *page, pgprot_t prot) +{ + return kmap_local_page(page); +} + +static inline void *kmap_local_pfn(unsigned long pfn) +{ + return kmap_local_page(pfn_to_page(pfn)); +} + +static inline void __kunmap_local(void *addr) +{ +#ifdef ARCH_HAS_FLUSH_ON_KUNMAP + kunmap_flush_on_unmap(addr); +#endif +} + static inline void *kmap_atomic(struct page *page) { preempt_disable(); @@ -181,4 +223,10 @@ do { \ __kunmap_atomic(__addr); \ } while (0) +#define kunmap_local(__addr) \ +do { \ + BUILD_BUG_ON(__same_type((__addr), struct page *)); \ + __kunmap_local(__addr); \ +} while (0) + #endif --- a/include/linux/highmem.h +++ b/include/linux/highmem.h @@ -60,24 +60,22 @@ static inline struct page *kmap_to_page( static inline void kmap_flush_unused(void); /** - * kmap_atomic - Atomically map a page for temporary usage + * kmap_local_page - Map a page for temporary usage * @page: Pointer to the page to be mapped * * Returns: The virtual address of the mapping * - * Side effect: On return pagefaults and preemption are disabled. - * * Can be invoked from any context. * * Requires careful handling when nesting multiple mappings because the map * management is stack based. The unmap has to be in the reverse order of * the map operation: * - * addr1 = kmap_atomic(page1); - * addr2 = kmap_atomic(page2); + * addr1 = kmap_local_page(page1); + * addr2 = kmap_local_page(page2); * ... - * kunmap_atomic(addr2); - * kunmap_atomic(addr1); + * kunmap_local(addr2); + * kunmap_local(addr1); * * Unmapping addr1 before addr2 is invalid and causes malfunction. * @@ -88,10 +86,26 @@ static inline void kmap_flush_unused(voi * virtual address of the direct mapping. Only real highmem pages are * temporarily mapped. * - * While it is significantly faster than kmap() it comes with restrictions - * about the pointer validity and the side effects of disabling page faults - * and preemption. Use it only when absolutely necessary, e.g. from non - * preemptible contexts. + * While it is significantly faster than kmap() for the higmem case it + * comes with restrictions about the pointer validity. Only use when really + * necessary. + * + * On HIGHMEM enabled systems mapping a highmem page has the side effect of + * disabling migration in order to keep the virtual address stable across + * preemption. No caller of kmap_local_page() can rely on this side effect. + */ +static inline void *kmap_local_page(struct page *page); + +/** + * kmap_atomic - Atomically map a page for temporary usage - Deprecated! + * @page: Pointer to the page to be mapped + * + * Returns: The virtual address of the mapping + * + * Effectively a wrapper around kmap_local_page() which disables pagefaults + * and preemption. + * + * Do not use in new code. Use kmap_local_page() instead. */ static inline void *kmap_atomic(struct page *page); @@ -101,12 +115,9 @@ static inline void *kmap_atomic(struct p * * Counterpart to kmap_atomic(). * - * Undoes the side effects of kmap_atomic(), i.e. reenabling pagefaults and + * Effectively a wrapper around kunmap_local() which additionally undoes + * the side effects of kmap_atomic(), i.e. reenabling pagefaults and * preemption. - * - * Other than that a NOOP for CONFIG_HIGHMEM=n and for mappings of pages - * in the low memory area. For real highmen pages the mapping which was - * established with kmap_atomic() is destroyed. */ /* Highmem related interfaces for management code */ --- a/mm/highmem.c +++ b/mm/highmem.c @@ -453,6 +453,11 @@ void *__kmap_local_pfn_prot(unsigned lon unsigned long vaddr; int idx; + /* + * Disable migration so resulting virtual address is stable + * accross preemption. + */ + migrate_disable(); preempt_disable(); idx = arch_kmap_local_map_idx(kmap_local_idx_push(), pfn); vaddr = __fix_to_virt(FIX_KMAP_BEGIN + idx); @@ -522,6 +527,7 @@ void kunmap_local_indexed(void *vaddr) current->kmap_ctrl.pteval[kmap_local_idx()] = __pte(0); kmap_local_idx_pop(); preempt_enable(); + migrate_enable(); } EXPORT_SYMBOL(kunmap_local_indexed); From patchwork Wed Nov 18 19:48:45 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Gleixner X-Patchwork-Id: 11915859 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 774D9C63697 for ; Wed, 18 Nov 2020 20:58:53 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D06182075B for ; Wed, 18 Nov 2020 20:58:52 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="wMGRnjEW"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="FUmzRTH8" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D06182075B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linutronix.de Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id D62386B007B; Wed, 18 Nov 2020 15:58:42 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id CB73C6B0078; Wed, 18 Nov 2020 15:58:42 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A93046B007B; Wed, 18 Nov 2020 15:58:42 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0140.hostedemail.com [216.40.44.140]) by kanga.kvack.org (Postfix) with ESMTP id 747E56B0074 for ; Wed, 18 Nov 2020 15:58:42 -0500 (EST) Received: from smtpin24.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 1A08C8249980 for ; Wed, 18 Nov 2020 20:58:42 +0000 (UTC) X-FDA: 77498752884.24.title94_570b5242733d Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin24.hostedemail.com (Postfix) with ESMTP id F3F011A4A5 for ; Wed, 18 Nov 2020 20:58:41 +0000 (UTC) X-HE-Tag: title94_570b5242733d X-Filterd-Recvd-Size: 8649 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by imf10.hostedemail.com (Postfix) with ESMTP for ; Wed, 18 Nov 2020 20:58:41 +0000 (UTC) Message-Id: <20201118204007.561220818@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1605733120; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: references:references; bh=+ZrRx/8oPKtbwxfPKSuA9uFhSRNHw0LXoGRTJJXvQqc=; b=wMGRnjEW7RLohGkm9TzjVU2Jv/8/fuvCDuUlbFsjnYUIrIlb6+Z8tYnVl80t5Xh4yK5h/x ZTW8xH+eJB95HzQD4tpGwvtb8LbbvsTS1MQOtIVq/WqBjDnvOXHONw5coe7Bljpfh/xbnH ACk0WjRR++c5UcXXoAeKCIimOr1sffBr2I8AhbdB75D72k0YUsvtAqT9YM9jtkk4rd25HM EOdR5NE+sPeRtPa5F0zwhqTRQx1UbFhW+79lMeZ7GTVxagAE7PYqNbL0KJm9vycjWnL4Xq TVJKThnz/J0tE1dsHthSM6VlAFxuuLXDfixDrCSwKNC9idyFLBPiE6bK3qOdmA== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1605733120; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: references:references; bh=+ZrRx/8oPKtbwxfPKSuA9uFhSRNHw0LXoGRTJJXvQqc=; b=FUmzRTH8UOdxIp1a1NUA4ayv2hSTU73Xu+3dn9nUlCCKISkX8InHxlwtmKukkTSdHS8s6t +ei2y27ls6Jrm9Cw== Date: Wed, 18 Nov 2020 20:48:45 +0100 From: Thomas Gleixner To: LKML Cc: x86@kernel.org, Linus Torvalds , Christoph Hellwig , Matthew Wilcox , Daniel Vetter , Andrew Morton , linux-mm@kvack.org, Linus Torvalds , Peter Zijlstra , Ingo Molnar , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira Subject: [patch V4 7/8] io-mapping: Provide iomap_local variant References: <20201118194838.753436396@linutronix.de> MIME-Version: 1.0 Content-transfer-encoding: 8-bit X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Thomas Gleixner Similar to kmap local provide a iomap local variant which only disables migration, but neither disables pagefaults nor preemption. Signed-off-by: Thomas Gleixner --- V3: Restrict migrate disable to the 32bit mapping case and update documentation. V2: Split out from the large combo patch and add the !IOMAP_ATOMIC variants --- Documentation/driver-api/io-mapping.rst | 76 +++++++++++++++++++------------- include/linux/io-mapping.h | 30 +++++++++++- 2 files changed, 74 insertions(+), 32 deletions(-) --- a/Documentation/driver-api/io-mapping.rst +++ b/Documentation/driver-api/io-mapping.rst @@ -20,55 +20,71 @@ as it would consume too much of the kern mappable, while 'size' indicates how large a mapping region to enable. Both are in bytes. -This _wc variant provides a mapping which may only be used -with the io_mapping_map_atomic_wc or io_mapping_map_wc. +This _wc variant provides a mapping which may only be used with +io_mapping_map_atomic_wc(), io_mapping_map_local_wc() or +io_mapping_map_wc(). + +With this mapping object, individual pages can be mapped either temporarily +or long term, depending on the requirements. Of course, temporary maps are +more efficient. They come in two flavours:: -With this mapping object, individual pages can be mapped either atomically -or not, depending on the necessary scheduling environment. Of course, atomic -maps are more efficient:: + void *io_mapping_map_local_wc(struct io_mapping *mapping, + unsigned long offset) void *io_mapping_map_atomic_wc(struct io_mapping *mapping, unsigned long offset) -'offset' is the offset within the defined mapping region. -Accessing addresses beyond the region specified in the -creation function yields undefined results. Using an offset -which is not page aligned yields an undefined result. The -return value points to a single page in CPU address space. - -This _wc variant returns a write-combining map to the -page and may only be used with mappings created by -io_mapping_create_wc +'offset' is the offset within the defined mapping region. Accessing +addresses beyond the region specified in the creation function yields +undefined results. Using an offset which is not page aligned yields an +undefined result. The return value points to a single page in CPU address +space. -Note that the task may not sleep while holding this page -mapped. +This _wc variant returns a write-combining map to the page and may only be +used with mappings created by io_mapping_create_wc() -:: +Temporary mappings are only valid in the context of the caller. The mapping +is not guaranteed to be globaly visible. - void io_mapping_unmap_atomic(void *vaddr) +io_mapping_map_local_wc() has a side effect on X86 32bit as it disables +migration to make the mapping code work. No caller can rely on this side +effect. + +io_mapping_map_atomic_wc() has the side effect of disabling preemption and +pagefaults. Don't use in new code. Use io_mapping_map_local_wc() instead. -'vaddr' must be the value returned by the last -io_mapping_map_atomic_wc call. This unmaps the specified -page and allows the task to sleep once again. +Nested mappings need to be undone in reverse order because the mapping +code uses a stack for keeping track of them:: -If you need to sleep while holding the lock, you can use the non-atomic -variant, although they may be significantly slower. + addr1 = io_mapping_map_local_wc(map1, offset1); + addr2 = io_mapping_map_local_wc(map2, offset2); + ... + io_mapping_unmap_local(addr2); + io_mapping_unmap_local(addr1); -:: +The mappings are released with:: + + void io_mapping_unmap_local(void *vaddr) + void io_mapping_unmap_atomic(void *vaddr) + +'vaddr' must be the value returned by the last io_mapping_map_local_wc() or +io_mapping_map_atomic_wc() call. This unmaps the specified mapping and +undoes the side effects of the mapping functions. + +If you need to sleep while holding a mapping, you can use the regular +variant, although this may be significantly slower:: void *io_mapping_map_wc(struct io_mapping *mapping, unsigned long offset) -This works like io_mapping_map_atomic_wc except it allows -the task to sleep while holding the page mapped. - +This works like io_mapping_map_atomic/local_wc() except it has no side +effects and the pointer is globaly visible. -:: +The mappings are released with:: void io_mapping_unmap(void *vaddr) -This works like io_mapping_unmap_atomic, except it is used -for pages mapped with io_mapping_map_wc. +Use for pages mapped with io_mapping_map_wc(). At driver close time, the io_mapping object must be freed:: --- a/include/linux/io-mapping.h +++ b/include/linux/io-mapping.h @@ -83,6 +83,21 @@ io_mapping_unmap_atomic(void __iomem *va } static inline void __iomem * +io_mapping_map_local_wc(struct io_mapping *mapping, unsigned long offset) +{ + resource_size_t phys_addr; + + BUG_ON(offset >= mapping->size); + phys_addr = mapping->base + offset; + return __iomap_local_pfn_prot(PHYS_PFN(phys_addr), mapping->prot); +} + +static inline void io_mapping_unmap_local(void __iomem *vaddr) +{ + kunmap_local_indexed((void __force *)vaddr); +} + +static inline void __iomem * io_mapping_map_wc(struct io_mapping *mapping, unsigned long offset, unsigned long size) @@ -101,7 +116,7 @@ io_mapping_unmap(void __iomem *vaddr) iounmap(vaddr); } -#else +#else /* HAVE_ATOMIC_IOMAP */ #include @@ -166,7 +181,18 @@ io_mapping_unmap_atomic(void __iomem *va preempt_enable(); } -#endif /* HAVE_ATOMIC_IOMAP */ +static inline void __iomem * +io_mapping_map_local_wc(struct io_mapping *mapping, unsigned long offset) +{ + return io_mapping_map_wc(mapping, offset, PAGE_SIZE); +} + +static inline void io_mapping_unmap_local(void __iomem *vaddr) +{ + io_mapping_unmap(vaddr); +} + +#endif /* !HAVE_ATOMIC_IOMAP */ static inline struct io_mapping * io_mapping_create_wc(resource_size_t base, From patchwork Wed Nov 18 19:48:46 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Thomas Gleixner X-Patchwork-Id: 11915863 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C881DC63798 for ; Wed, 18 Nov 2020 20:58:55 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 4229C2075B for ; Wed, 18 Nov 2020 20:58:55 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="eUC6bTUE"; dkim=permerror (0-bit key) header.d=linutronix.de header.i=@linutronix.de header.b="6Gj6t05x" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4229C2075B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linutronix.de Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 00E996B0074; Wed, 18 Nov 2020 15:58:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id EDE5E6B0078; Wed, 18 Nov 2020 15:58:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DA0E86B007D; Wed, 18 Nov 2020 15:58:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0187.hostedemail.com [216.40.44.187]) by kanga.kvack.org (Postfix) with ESMTP id A50056B0074 for ; Wed, 18 Nov 2020 15:58:43 -0500 (EST) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 4AEFC8249980 for ; Wed, 18 Nov 2020 20:58:43 +0000 (UTC) X-FDA: 77498752926.11.nerve24_3009c5c2733d Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin11.hostedemail.com (Postfix) with ESMTP id 213EB180F8B81 for ; Wed, 18 Nov 2020 20:58:43 +0000 (UTC) X-HE-Tag: nerve24_3009c5c2733d X-Filterd-Recvd-Size: 5174 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by imf36.hostedemail.com (Postfix) with ESMTP for ; Wed, 18 Nov 2020 20:58:42 +0000 (UTC) Message-Id: <20201118204007.670851839@linutronix.de> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1605733121; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: references:references; bh=n6ZnY4fkMJxZaMnKuAdf5ZenkVJW5B3lKuVHTxlEMsQ=; b=eUC6bTUEftA8VubxeJPfi/9RV3Ygjgx/syVTifwYyBPpgBjdlQO3p+Tzw/9zclPhp/oguL V8eWjtmVqFSElbYr6KDgwpygcHQvsZgRV7o+76zpYgZzWeUL3uF7Gf8Qu3T4zcE8aNG03c n11gDWPi6Psl/yvCMsgMY1kZO2qFOT+MTYMWMxeyM4Xm3bgZ2gmezcUmwJ/3vzPTSQ8zu1 9Pd0CB4FdrVTnDl3k0VzGFRrox46iOe3gPocn0qCiYEZzgzocPIib7OQMrq7yp6VVBqH4a vxPF6PZJlxoxos0zcXWJf0Dk+BJnyOHqxFPuFx4WCACiK4uLEpRHFpd68LKPXQ== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1605733121; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: references:references; bh=n6ZnY4fkMJxZaMnKuAdf5ZenkVJW5B3lKuVHTxlEMsQ=; b=6Gj6t05xdfPnftLBfQTZ+to85TmZSrrZNMp1Vt4RpSfwiFRxeihmtg7CYQ3+YvJG7vqQMO kUZoqldX72sngVCg== Date: Wed, 18 Nov 2020 20:48:46 +0100 From: Thomas Gleixner To: LKML Cc: x86@kernel.org, Linus Torvalds , Christoph Hellwig , Matthew Wilcox , Daniel Vetter , Andrew Morton , linux-mm@kvack.org, Linus Torvalds , Peter Zijlstra , Ingo Molnar , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira Subject: [patch V4 8/8] x86/crashdump/32: Simplify copy_oldmem_page() References: <20201118194838.753436396@linutronix.de> MIME-Version: 1.0 Content-transfer-encoding: 8-bit X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Thomas Gleixner Replace kmap_atomic_pfn() with kmap_local_pfn() which is preemptible and can take page faults. Remove the indirection of the dump page and the related cruft which is not longer required. Signed-off-by: Thomas Gleixner --- V3: New patch --- arch/x86/kernel/crash_dump_32.c | 48 ++++++++-------------------------------- 1 file changed, 10 insertions(+), 38 deletions(-) --- a/arch/x86/kernel/crash_dump_32.c +++ b/arch/x86/kernel/crash_dump_32.c @@ -13,8 +13,6 @@ #include -static void *kdump_buf_page; - static inline bool is_crashed_pfn_valid(unsigned long pfn) { #ifndef CONFIG_X86_PAE @@ -41,15 +39,11 @@ static inline bool is_crashed_pfn_valid( * @userbuf: if set, @buf is in user address space, use copy_to_user(), * otherwise @buf is in kernel address space, use memcpy(). * - * Copy a page from "oldmem". For this page, there is no pte mapped - * in the current kernel. We stitch up a pte, similar to kmap_atomic. - * - * Calling copy_to_user() in atomic context is not desirable. Hence first - * copying the data to a pre-allocated kernel page and then copying to user - * space in non-atomic context. + * Copy a page from "oldmem". For this page, there might be no pte mapped + * in the current kernel. */ -ssize_t copy_oldmem_page(unsigned long pfn, char *buf, - size_t csize, unsigned long offset, int userbuf) +ssize_t copy_oldmem_page(unsigned long pfn, char *buf, size_t csize, + unsigned long offset, int userbuf) { void *vaddr; @@ -59,38 +53,16 @@ ssize_t copy_oldmem_page(unsigned long p if (!is_crashed_pfn_valid(pfn)) return -EFAULT; - vaddr = kmap_atomic_pfn(pfn); + vaddr = kmap_local_pfn(pfn); if (!userbuf) { - memcpy(buf, (vaddr + offset), csize); - kunmap_atomic(vaddr); + memcpy(buf, vaddr + offset, csize); } else { - if (!kdump_buf_page) { - printk(KERN_WARNING "Kdump: Kdump buffer page not" - " allocated\n"); - kunmap_atomic(vaddr); - return -EFAULT; - } - copy_page(kdump_buf_page, vaddr); - kunmap_atomic(vaddr); - if (copy_to_user(buf, (kdump_buf_page + offset), csize)) - return -EFAULT; + if (copy_to_user(buf, vaddr + offset, csize)) + csize = -EFAULT; } - return csize; -} + kunmap_local(vaddr); -static int __init kdump_buf_page_init(void) -{ - int ret = 0; - - kdump_buf_page = kmalloc(PAGE_SIZE, GFP_KERNEL); - if (!kdump_buf_page) { - printk(KERN_WARNING "Kdump: Failed to allocate kdump buffer" - " page\n"); - ret = -ENOMEM; - } - - return ret; + return csize; } -arch_initcall(kdump_buf_page_init);