From patchwork Mon May 4 14:57:58 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 11526457 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9E30D15AB for ; Mon, 4 May 2020 14:59:50 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 600EF206D7 for ; Mon, 4 May 2020 14:59:50 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="B2pFBFVV" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 600EF206D7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 7DDCC8E002C; Mon, 4 May 2020 10:59:49 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 78F9F8E0024; Mon, 4 May 2020 10:59:49 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 67E008E002C; Mon, 4 May 2020 10:59:49 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0208.hostedemail.com [216.40.44.208]) by kanga.kvack.org (Postfix) with ESMTP id 4D0178E0024 for ; Mon, 4 May 2020 10:59:49 -0400 (EDT) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 0F2C1181AC9C6 for ; Mon, 4 May 2020 14:59:49 +0000 (UTC) X-FDA: 76779346098.01.trick85_388c2e0d4b85e X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,alexandre.chartre@oracle.com,,RULES_HIT:30054:30064:30067,0,RBL:156.151.31.86:@oracle.com:.lbl8.mailshell.net-62.18.0.100 64.10.201.10,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:ft,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: trick85_388c2e0d4b85e X-Filterd-Recvd-Size: 7463 Received: from userp2130.oracle.com (userp2130.oracle.com [156.151.31.86]) by imf34.hostedemail.com (Postfix) with ESMTP for ; Mon, 4 May 2020 14:59:48 +0000 (UTC) Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 044Elbbx116456; Mon, 4 May 2020 14:59:39 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2020-01-29; bh=gOU60KEGCpd90jFwTgBCZEZdFNyJRNJA9GfZLpt6zw8=; b=B2pFBFVVIkkc3kRYz76iRL1urniEgMM+pcMfb/sfNQ+kqocoURz+DfnjrDyBw0q84WPk rRFnStr5d0TiPQz8dTQe+RC1kIowLLuiJ+4pjuS0XEOdsqYJWbhuJkBu0+AZw3jdkIuE PrhMsEQ9QI3/CFfdW6JKLoHoGFaBkqE+RWrVGE7xrXPmvhbwJgrItDIl+Zc3WI8PxpZt FLN4bRgMWADYqsEx9LVm8MDmbon8NIMUHC1uUDb2M/SzYVSDH/EzkvBo7YFpVewBEJ1w tXRJyqnM3Lz4qCXl5BpTCj7x2of9DtyXiLgkTejBTo1+4YQQ+j4iZaEmIbZftiywvRQc LA== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by userp2130.oracle.com with ESMTP id 30s09qyfpj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 04 May 2020 14:59:39 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 044ElK3e033673; Mon, 4 May 2020 14:59:38 GMT Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by aserp3020.oracle.com with ESMTP id 30sjnaw0r4-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 04 May 2020 14:59:38 +0000 Received: from abhmp0001.oracle.com (abhmp0001.oracle.com [141.146.116.7]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id 044Exaia022105; Mon, 4 May 2020 14:59:36 GMT Received: from linux-1.home.com (/10.175.9.166) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 04 May 2020 07:59:36 -0700 From: Alexandre Chartre To: tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: pbonzini@redhat.com, konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, junaids@google.com, graf@amazon.de, rppt@linux.vnet.ibm.com, kuzuno@gmail.com, mgross@linux.intel.com, alexandre.chartre@oracle.com Subject: [RFC v4][PATCH part-2 01/13] mm/x86: Introduce decorated page-table (dpt) Date: Mon, 4 May 2020 16:57:58 +0200 Message-Id: <20200504145810.11882-2-alexandre.chartre@oracle.com> X-Mailer: git-send-email 2.18.2 In-Reply-To: <20200504145810.11882-1-alexandre.chartre@oracle.com> References: <20200504145810.11882-1-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9610 signatures=668687 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 adultscore=0 phishscore=0 mlxlogscore=999 bulkscore=0 malwarescore=0 spamscore=0 suspectscore=2 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2005040123 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9610 signatures=668687 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 mlxscore=0 lowpriorityscore=0 spamscore=0 adultscore=0 clxscore=1015 suspectscore=2 priorityscore=1501 malwarescore=0 mlxlogscore=999 phishscore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2005040123 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: A decorated page-table (dpt) encapsulates a native page-table (e.g. a PGD) and maintain additional attributes related to this page-table. It aims to be the base structure for providing useful functions to manage a page-table, such as tracking VA range mapped in a page-table or safely handling references to another page-table. Signed-off-by: Alexandre Chartre --- arch/x86/include/asm/dpt.h | 23 +++++++++++++ arch/x86/mm/Makefile | 2 +- arch/x86/mm/dpt.c | 67 ++++++++++++++++++++++++++++++++++++++ 3 files changed, 91 insertions(+), 1 deletion(-) create mode 100644 arch/x86/include/asm/dpt.h create mode 100644 arch/x86/mm/dpt.c diff --git a/arch/x86/include/asm/dpt.h b/arch/x86/include/asm/dpt.h new file mode 100644 index 000000000000..1da4d43d5e94 --- /dev/null +++ b/arch/x86/include/asm/dpt.h @@ -0,0 +1,23 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef ARCH_X86_MM_DPT_H +#define ARCH_X86_MM_DPT_H + +#include + +#include + +/* + * A decorated page-table (dpt) encapsulates a native page-table (e.g. + * a PGD) and maintain additional attributes related to this page-table. + */ +struct dpt { + spinlock_t lock; /* protect all attributes */ + pgd_t *pagetable; /* the actual page-table */ + unsigned int alignment; /* page-table alignment */ + +}; + +extern struct dpt *dpt_create(unsigned int pgt_alignment); +extern void dpt_destroy(struct dpt *dpt); + +#endif diff --git a/arch/x86/mm/Makefile b/arch/x86/mm/Makefile index e57af263e870..5b52d854a030 100644 --- a/arch/x86/mm/Makefile +++ b/arch/x86/mm/Makefile @@ -48,7 +48,7 @@ obj-$(CONFIG_NUMA_EMU) += numa_emulation.o obj-$(CONFIG_X86_INTEL_MEMORY_PROTECTION_KEYS) += pkeys.o obj-$(CONFIG_RANDOMIZE_MEMORY) += kaslr.o obj-$(CONFIG_PAGE_TABLE_ISOLATION) += pti.o -obj-$(CONFIG_ADDRESS_SPACE_ISOLATION) += asi.o +obj-$(CONFIG_ADDRESS_SPACE_ISOLATION) += asi.o dpt.o obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt.o obj-$(CONFIG_AMD_MEM_ENCRYPT) += mem_encrypt_identity.o diff --git a/arch/x86/mm/dpt.c b/arch/x86/mm/dpt.c new file mode 100644 index 000000000000..333e259c5b7f --- /dev/null +++ b/arch/x86/mm/dpt.c @@ -0,0 +1,67 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (c) 2019, 2020, Oracle and/or its affiliates. + * + */ + +#include + +#include + +/* + * dpt_create - allocate a page-table and create a corresponding + * decorated page-table. The page-table is allocated and aligned + * at the specified alignment (pgt_alignment) which should be a + * multiple of PAGE_SIZE. + */ +struct dpt *dpt_create(unsigned int pgt_alignment) +{ + unsigned int alloc_order; + unsigned long pagetable; + struct dpt *dpt; + + if (!IS_ALIGNED(pgt_alignment, PAGE_SIZE)) + return NULL; + + alloc_order = round_up(PAGE_SIZE + pgt_alignment, + PAGE_SIZE) >> PAGE_SHIFT; + + dpt = kzalloc(sizeof(*dpt), GFP_KERNEL); + if (!dpt) + return NULL; + + pagetable = (unsigned long)__get_free_pages(GFP_KERNEL_ACCOUNT | + __GFP_ZERO, + alloc_order); + if (!pagetable) { + kfree(dpt); + return NULL; + } + dpt->pagetable = (pgd_t *)(pagetable + pgt_alignment); + dpt->alignment = pgt_alignment; + + spin_lock_init(&dpt->lock); + + return dpt; +} +EXPORT_SYMBOL(dpt_create); + +void dpt_destroy(struct dpt *dpt) +{ + unsigned int pgt_alignment; + unsigned int alloc_order; + + if (!dpt) + return; + + if (dpt->pagetable) { + pgt_alignment = dpt->alignment; + alloc_order = round_up(PAGE_SIZE + pgt_alignment, + PAGE_SIZE) >> PAGE_SHIFT; + free_pages((unsigned long)(dpt->pagetable) - pgt_alignment, + alloc_order); + } + + kfree(dpt); +} +EXPORT_SYMBOL(dpt_destroy); From patchwork Mon May 4 14:57:59 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 11526477 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AD8341392 for ; Mon, 4 May 2020 15:02:04 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 78E7B20735 for ; Mon, 4 May 2020 15:02:04 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="aiDhQg48" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 78E7B20735 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 82A7E8E0036; Mon, 4 May 2020 11:02:03 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 7DA748E0024; Mon, 4 May 2020 11:02:03 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6EFBC8E0036; Mon, 4 May 2020 11:02:03 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0134.hostedemail.com [216.40.44.134]) by kanga.kvack.org (Postfix) with ESMTP id 529818E0024 for ; Mon, 4 May 2020 11:02:03 -0400 (EDT) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 185F9181AEF31 for ; Mon, 4 May 2020 15:02:03 +0000 (UTC) X-FDA: 76779351726.04.skate92_4c08823562e2a X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,alexandre.chartre@oracle.com,,RULES_HIT:30003:30054:30064:30070,0,RBL:156.151.31.86:@oracle.com:.lbl8.mailshell.net-64.10.201.10 62.18.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:ft,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:0,LUA_SUMMARY:none X-HE-Tag: skate92_4c08823562e2a X-Filterd-Recvd-Size: 8788 Received: from userp2130.oracle.com (userp2130.oracle.com [156.151.31.86]) by imf48.hostedemail.com (Postfix) with ESMTP for ; Mon, 4 May 2020 15:02:02 +0000 (UTC) Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 044ElbML116466; Mon, 4 May 2020 15:01:44 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2020-01-29; bh=mpXXd7uEdJ6fmVl0iUxBjNXdKzD8MJRJwDBOk9pZnT0=; b=aiDhQg48J1nTnS19xiB7JOTozw6IIJ6RDVQKmmdGkIqRuLpW8s76/frddUZtuAuBUjPy ElA+W64NLBZsxkRyNsTHCo/mD+yeVv56uxVANbu6S2URk2CukVo65oH/WwumHiwC9XLc QLGhAmMgjaDqXIMK2tNJ7Vprz/i/aInfcGOScTAJSLJvVJE6wJFVgf9hUJd/N0KarHKf iKzeHffR/dJ89EaaBaWr3bOJ5r6XSVp4fiR+K8C2nCHgNcngD6TILIBhQQQ8MlZsG6z+ pXc71NSWiGFr7xuMH6ygsa1m/aagaunI0X0rkwg6+vL6vOEyqgIYgPdHioAuVPhoN8i8 aw== Received: from userp3030.oracle.com (userp3030.oracle.com [156.151.31.80]) by userp2130.oracle.com with ESMTP id 30s09qyg76-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 04 May 2020 15:01:42 +0000 Received: from pps.filterd (userp3030.oracle.com [127.0.0.1]) by userp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 044ElbCv053434; Mon, 4 May 2020 14:59:42 GMT Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by userp3030.oracle.com with ESMTP id 30t1r2esd5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 04 May 2020 14:59:42 +0000 Received: from abhmp0001.oracle.com (abhmp0001.oracle.com [141.146.116.7]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id 044ExeBx022128; Mon, 4 May 2020 14:59:40 GMT Received: from linux-1.home.com (/10.175.9.166) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 04 May 2020 07:59:40 -0700 From: Alexandre Chartre To: tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: pbonzini@redhat.com, konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, junaids@google.com, graf@amazon.de, rppt@linux.vnet.ibm.com, kuzuno@gmail.com, mgross@linux.intel.com, alexandre.chartre@oracle.com Subject: [RFC v4][PATCH part-2 02/13] mm/dpt: Track buffers allocated for a decorated page-table Date: Mon, 4 May 2020 16:57:59 +0200 Message-Id: <20200504145810.11882-3-alexandre.chartre@oracle.com> X-Mailer: git-send-email 2.18.2 In-Reply-To: <20200504145810.11882-1-alexandre.chartre@oracle.com> References: <20200504145810.11882-1-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9610 signatures=668687 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 adultscore=0 suspectscore=2 spamscore=0 mlxlogscore=999 malwarescore=0 phishscore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2005040123 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9610 signatures=668687 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 mlxscore=0 lowpriorityscore=0 spamscore=0 adultscore=0 clxscore=1015 suspectscore=2 priorityscore=1501 malwarescore=0 mlxlogscore=999 phishscore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2005040123 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add functions to track buffers allocated for a decorated page-table. A page-table can have direct references to the kernel page table, at different levels (PGD, P4D, PUD, PMD). When freeing a page-table, we should make sure that we free parts actually allocated for the decorated page-table, and not parts of the kernel page table referenced from the page-table. To do so, we will keep track of buffers when building the page-table. Signed-off-by: Alexandre Chartre --- arch/x86/include/asm/dpt.h | 21 ++++++++++ arch/x86/mm/dpt.c | 82 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 103 insertions(+) diff --git a/arch/x86/include/asm/dpt.h b/arch/x86/include/asm/dpt.h index 1da4d43d5e94..b9cba051ebf2 100644 --- a/arch/x86/include/asm/dpt.h +++ b/arch/x86/include/asm/dpt.h @@ -3,9 +3,18 @@ #define ARCH_X86_MM_DPT_H #include +#include #include +enum page_table_level { + PGT_LEVEL_PTE, + PGT_LEVEL_PMD, + PGT_LEVEL_PUD, + PGT_LEVEL_P4D, + PGT_LEVEL_PGD +}; + /* * A decorated page-table (dpt) encapsulates a native page-table (e.g. * a PGD) and maintain additional attributes related to this page-table. @@ -15,6 +24,18 @@ struct dpt { pgd_t *pagetable; /* the actual page-table */ unsigned int alignment; /* page-table alignment */ + /* + * A page-table can have direct references to another page-table, + * at different levels (PGD, P4D, PUD, PMD). When freeing or + * modifying a page-table, we should make sure that we free/modify + * parts effectively allocated to the actual page-table, and not + * parts of another page-table referenced from this page-table. + * + * To do so, the backend_pages XArray is used to keep track of pages + * used for this page-table. + */ + struct xarray backend_pages; /* page-table pages */ + unsigned long backend_pages_count; /* pages count */ }; extern struct dpt *dpt_create(unsigned int pgt_alignment); diff --git a/arch/x86/mm/dpt.c b/arch/x86/mm/dpt.c index 333e259c5b7f..6df2d4fde8ec 100644 --- a/arch/x86/mm/dpt.c +++ b/arch/x86/mm/dpt.c @@ -8,6 +8,80 @@ #include +/* + * Get the pointer to the beginning of a page table directory from a page + * table directory entry. + */ +#define DPT_BACKEND_PAGE_ALIGN(entry) \ + ((typeof(entry))(((unsigned long)(entry)) & PAGE_MASK)) + +/* + * Pages used to build a page-table are stored in the backend_pages XArray. + * Each entry in the array is a logical OR of the page address and the page + * table level (PTE, PMD, PUD, P4D) this page is used for in the page-table. + * + * As a page address is aligned with PAGE_SIZE, we have plenty of space + * for storing the page table level (which is a value between 0 and 4) in + * the low bits of the page address. + * + */ + +#define DPT_BACKEND_PAGE_ENTRY(addr, level) \ + ((typeof(addr))(((unsigned long)(addr)) | ((unsigned long)(level)))) +#define DPT_BACKEND_PAGE_ADDR(entry) \ + ((void *)(((unsigned long)(entry)) & PAGE_MASK)) +#define DPT_BACKEND_PAGE_LEVEL(entry) \ + ((enum page_table_level)(((unsigned long)(entry)) & ~PAGE_MASK)) + +static int dpt_add_backend_page(struct dpt *dpt, void *addr, + enum page_table_level level) +{ + unsigned long index; + void *old_entry; + + if ((!addr) || ((unsigned long)addr) & ~PAGE_MASK) + return -EINVAL; + + lockdep_assert_held(&dpt->lock); + index = dpt->backend_pages_count; + + old_entry = xa_store(&dpt->backend_pages, index, + DPT_BACKEND_PAGE_ENTRY(addr, level), + GFP_KERNEL); + if (xa_is_err(old_entry)) + return xa_err(old_entry); + if (old_entry) + return -EBUSY; + + dpt->backend_pages_count++; + + return 0; +} + +/* + * Check if an offset in the page-table is valid, i.e. check that the + * offset is on a page effectively belonging to the page-table. + */ +static bool dpt_valid_offset(struct dpt *dpt, void *offset) +{ + unsigned long index; + void *addr, *entry; + bool valid; + + addr = DPT_BACKEND_PAGE_ALIGN(offset); + valid = false; + + lockdep_assert_held(&dpt->lock); + xa_for_each(&dpt->backend_pages, index, entry) { + if (DPT_BACKEND_PAGE_ADDR(entry) == addr) { + valid = true; + break; + } + } + + return valid; +} + /* * dpt_create - allocate a page-table and create a corresponding * decorated page-table. The page-table is allocated and aligned @@ -41,6 +115,7 @@ struct dpt *dpt_create(unsigned int pgt_alignment) dpt->alignment = pgt_alignment; spin_lock_init(&dpt->lock); + xa_init(&dpt->backend_pages); return dpt; } @@ -50,10 +125,17 @@ void dpt_destroy(struct dpt *dpt) { unsigned int pgt_alignment; unsigned int alloc_order; + unsigned long index; + void *entry; if (!dpt) return; + if (dpt->backend_pages_count) { + xa_for_each(&dpt->backend_pages, index, entry) + free_page((unsigned long)DPT_BACKEND_PAGE_ADDR(entry)); + } + if (dpt->pagetable) { pgt_alignment = dpt->alignment; alloc_order = round_up(PAGE_SIZE + pgt_alignment, From patchwork Mon May 4 14:58:00 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 11526459 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 63D7215AB for ; Mon, 4 May 2020 14:59:59 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 2FFC8206D7 for ; Mon, 4 May 2020 14:59:59 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="NYaPu6pU" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2FFC8206D7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 6095D8E002D; Mon, 4 May 2020 10:59:58 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 5DFD58E0024; Mon, 4 May 2020 10:59:58 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4F6CB8E002D; Mon, 4 May 2020 10:59:58 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0077.hostedemail.com [216.40.44.77]) by kanga.kvack.org (Postfix) with ESMTP id 37F7C8E0024 for ; Mon, 4 May 2020 10:59:58 -0400 (EDT) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id EBAF5180AD80F for ; Mon, 4 May 2020 14:59:57 +0000 (UTC) X-FDA: 76779346434.11.frame17_39d84dbdc0331 X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,alexandre.chartre@oracle.com,,RULES_HIT:30054:30064,0,RBL:156.151.31.86:@oracle.com:.lbl8.mailshell.net-62.18.0.100 64.10.201.10,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:ft,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: frame17_39d84dbdc0331 X-Filterd-Recvd-Size: 5872 Received: from userp2130.oracle.com (userp2130.oracle.com [156.151.31.86]) by imf13.hostedemail.com (Postfix) with ESMTP for ; Mon, 4 May 2020 14:59:57 +0000 (UTC) Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 044ElbDo116519; Mon, 4 May 2020 14:59:47 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2020-01-29; bh=ItAga/RAZZLJV2kYsxGyvAeBJnzaxmTXCE/C0Q2axKc=; b=NYaPu6pUwjVJ3gCuMmIeR4QjgnjJAa1EfrSnTxbil4a2AIEQnfp5a3IaEV4En82n/9yJ QKzIqppk1aCLias8xSIi3Hdf/5k6pH4iWeimowFwt0UtX25QaaQT/EMTM4/wBSVJBQpG T9+L70neL+NhewAs4vErWbCyeqzWqQniZWEeXy6rwJY8oXFgLs/cOg6ku6dPIzv21I4+ jNK2CnZJe5oI6nUgtW4FUZLA87Eu0ZlmOvrmHOsPv1TFt6KsiiqkCnrMiiSmaXbQhVWz clS6E1vVtFMO4hetYxBiUvXFOP/1e1ptoJxt8B8uUaARL0+MuP9uvDF86QPWIgaD3GwU zg== Received: from aserp3030.oracle.com (aserp3030.oracle.com [141.146.126.71]) by userp2130.oracle.com with ESMTP id 30s09qyfqf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 04 May 2020 14:59:47 +0000 Received: from pps.filterd (aserp3030.oracle.com [127.0.0.1]) by aserp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 044EkelF094657; Mon, 4 May 2020 14:59:46 GMT Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by aserp3030.oracle.com with ESMTP id 30sjdqq654-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 04 May 2020 14:59:46 +0000 Received: from abhmp0001.oracle.com (abhmp0001.oracle.com [141.146.116.7]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id 044Exi5M022173; Mon, 4 May 2020 14:59:44 GMT Received: from linux-1.home.com (/10.175.9.166) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 04 May 2020 07:59:44 -0700 From: Alexandre Chartre To: tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: pbonzini@redhat.com, konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, junaids@google.com, graf@amazon.de, rppt@linux.vnet.ibm.com, kuzuno@gmail.com, mgross@linux.intel.com, alexandre.chartre@oracle.com Subject: [RFC v4][PATCH part-2 03/13] mm/dpt: Add decorated page-table entry offset functions Date: Mon, 4 May 2020 16:58:00 +0200 Message-Id: <20200504145810.11882-4-alexandre.chartre@oracle.com> X-Mailer: git-send-email 2.18.2 In-Reply-To: <20200504145810.11882-1-alexandre.chartre@oracle.com> References: <20200504145810.11882-1-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9610 signatures=668687 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 suspectscore=0 mlxscore=0 bulkscore=0 adultscore=0 phishscore=0 mlxlogscore=999 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2005040123 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9610 signatures=668687 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 mlxscore=0 lowpriorityscore=0 spamscore=0 adultscore=0 clxscore=1015 suspectscore=0 priorityscore=1501 malwarescore=0 mlxlogscore=999 phishscore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2005040123 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add wrappers around the p4d/pud/pmd/pte offset kernel functions which ensure that page-table pointers are in the specified decorated page-table. Signed-off-by: Alexandre Chartre --- arch/x86/mm/dpt.c | 66 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 66 insertions(+) diff --git a/arch/x86/mm/dpt.c b/arch/x86/mm/dpt.c index 6df2d4fde8ec..44aad99bc3dc 100644 --- a/arch/x86/mm/dpt.c +++ b/arch/x86/mm/dpt.c @@ -82,6 +82,72 @@ static bool dpt_valid_offset(struct dpt *dpt, void *offset) return valid; } +/* + * dpt_pXX_offset() functions are equivalent to kernel pXX_offset() + * functions but, in addition, they ensure that page table pointers + * are in the specified decorated page table. Otherwise an error is + * returned. + */ + +static pte_t *dpt_pte_offset(struct dpt *dpt, + pmd_t *pmd, unsigned long addr) +{ + pte_t *pte; + + pte = pte_offset_map(pmd, addr); + if (!dpt_valid_offset(dpt, pte)) { + pr_err("DPT %p: PTE %px not found\n", dpt, pte); + return ERR_PTR(-EINVAL); + } + + return pte; +} + +static pmd_t *dpt_pmd_offset(struct dpt *dpt, + pud_t *pud, unsigned long addr) +{ + pmd_t *pmd; + + pmd = pmd_offset(pud, addr); + if (!dpt_valid_offset(dpt, pmd)) { + pr_err("DPT %p: PMD %px not found\n", dpt, pmd); + return ERR_PTR(-EINVAL); + } + + return pmd; +} + +static pud_t *dpt_pud_offset(struct dpt *dpt, + p4d_t *p4d, unsigned long addr) +{ + pud_t *pud; + + pud = pud_offset(p4d, addr); + if (!dpt_valid_offset(dpt, pud)) { + pr_err("DPT %p: PUD %px not found\n", dpt, pud); + return ERR_PTR(-EINVAL); + } + + return pud; +} + +static p4d_t *dpt_p4d_offset(struct dpt *dpt, + pgd_t *pgd, unsigned long addr) +{ + p4d_t *p4d; + + p4d = p4d_offset(pgd, addr); + /* + * p4d is the same has pgd if we don't have a 5-level page table. + */ + if ((p4d != (p4d_t *)pgd) && !dpt_valid_offset(dpt, p4d)) { + pr_err("DPT %p: P4D %px not found\n", dpt, p4d); + return ERR_PTR(-EINVAL); + } + + return p4d; +} + /* * dpt_create - allocate a page-table and create a corresponding * decorated page-table. The page-table is allocated and aligned From patchwork Mon May 4 14:58:01 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 11526461 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BE7B615AB for ; Mon, 4 May 2020 15:00:02 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 7F49C20721 for ; Mon, 4 May 2020 15:00:02 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="gO7Vc2hg" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7F49C20721 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 4A9258E002E; Mon, 4 May 2020 11:00:01 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 4596D8E0024; Mon, 4 May 2020 11:00:01 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 348FB8E002E; Mon, 4 May 2020 11:00:01 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0238.hostedemail.com [216.40.44.238]) by kanga.kvack.org (Postfix) with ESMTP id 1A2498E0024 for ; Mon, 4 May 2020 11:00:01 -0400 (EDT) Received: from smtpin15.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id D76FC4821 for ; Mon, 4 May 2020 15:00:00 +0000 (UTC) X-FDA: 76779346560.15.alley43_3a3de40ecd00f X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,alexandre.chartre@oracle.com,,RULES_HIT:30054:30064:30070,0,RBL:156.151.31.86:@oracle.com:.lbl8.mailshell.net-62.18.0.100 64.10.201.10,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:ft,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: alley43_3a3de40ecd00f X-Filterd-Recvd-Size: 7085 Received: from userp2130.oracle.com (userp2130.oracle.com [156.151.31.86]) by imf05.hostedemail.com (Postfix) with ESMTP for ; Mon, 4 May 2020 15:00:00 +0000 (UTC) Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 044ElbM9116466; Mon, 4 May 2020 14:59:51 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2020-01-29; bh=2TQALusUHTHT/8DPoOvtUCzsont8/NCzCLNnkMe9NN0=; b=gO7Vc2hgHxk88IgVrU2lqBvLzA+jn2bmMAufd4rine3T8INi4HeZhmkXoLx6pjLIrNSg 3rvg91zvnxZy9muQSvT0WujXuqivwBody/WOJAOSEreyAgEE7mYiK+CTKW7yvdHkNA+S r1rsemIlYLILCFDssrLQwAqmEvfcsLmP2B0fOiBMpIShK/bygslzyAGJvZ5xxfLfJ27U LUWAH2Lro5M5hZ18UjR7iXKNI7QQsqxfPBmmEEcXu+eFNMJTUuiujA2KMeUH2KyhdvVn U5XQwdcvMDo+rES/bx6XLZzppc3Dx8YxnL6ParysXVKxF97LXuD3EniDwlmZGQMJU99a iQ== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by userp2130.oracle.com with ESMTP id 30s09qyfr2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 04 May 2020 14:59:51 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 044EmVaC105750; Mon, 4 May 2020 14:59:51 GMT Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72]) by userp3020.oracle.com with ESMTP id 30sjjw0g03-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 04 May 2020 14:59:51 +0000 Received: from abhmp0001.oracle.com (abhmp0001.oracle.com [141.146.116.7]) by userv0121.oracle.com (8.14.4/8.13.8) with ESMTP id 044Exng3027530; Mon, 4 May 2020 14:59:49 GMT Received: from linux-1.home.com (/10.175.9.166) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 04 May 2020 07:59:48 -0700 From: Alexandre Chartre To: tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: pbonzini@redhat.com, konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, junaids@google.com, graf@amazon.de, rppt@linux.vnet.ibm.com, kuzuno@gmail.com, mgross@linux.intel.com, alexandre.chartre@oracle.com Subject: [RFC v4][PATCH part-2 04/13] mm/dpt: Add decorated page-table entry allocation functions Date: Mon, 4 May 2020 16:58:01 +0200 Message-Id: <20200504145810.11882-5-alexandre.chartre@oracle.com> X-Mailer: git-send-email 2.18.2 In-Reply-To: <20200504145810.11882-1-alexandre.chartre@oracle.com> References: <20200504145810.11882-1-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9610 signatures=668687 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=2 mlxscore=0 phishscore=0 bulkscore=0 malwarescore=0 spamscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2005040123 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9610 signatures=668687 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 mlxscore=0 lowpriorityscore=0 spamscore=0 adultscore=0 clxscore=1015 suspectscore=2 priorityscore=1501 malwarescore=0 mlxlogscore=999 phishscore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2005040123 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add functions to allocate p4d/pud/pmd/pte pages for an decorated page-table and keep track of them. Signed-off-by: Alexandre Chartre --- arch/x86/mm/dpt.c | 110 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 110 insertions(+) diff --git a/arch/x86/mm/dpt.c b/arch/x86/mm/dpt.c index 44aad99bc3dc..a2f54ba00255 100644 --- a/arch/x86/mm/dpt.c +++ b/arch/x86/mm/dpt.c @@ -4,6 +4,7 @@ * */ +#include #include #include @@ -148,6 +149,115 @@ static p4d_t *dpt_p4d_offset(struct dpt *dpt, return p4d; } +/* + * dpt_pXX_alloc() functions are equivalent to kernel pXX_alloc() functions + * but, in addition, they keep track of new pages allocated for the specified + * decorated page-table. + */ + +static pte_t *dpt_pte_alloc(struct dpt *dpt, pmd_t *pmd, unsigned long addr) +{ + struct page *page; + pte_t *pte; + int err; + + if (pmd_none(*pmd)) { + page = alloc_page(GFP_KERNEL | __GFP_ZERO); + if (!page) + return ERR_PTR(-ENOMEM); + pte = (pte_t *)page_address(page); + err = dpt_add_backend_page(dpt, pte, PGT_LEVEL_PTE); + if (err) { + free_page((unsigned long)pte); + return ERR_PTR(err); + } + set_pmd_safe(pmd, __pmd(__pa(pte) | _KERNPG_TABLE)); + pte = pte_offset_map(pmd, addr); + } else { + pte = dpt_pte_offset(dpt, pmd, addr); + } + + return pte; +} + +static pmd_t *dpt_pmd_alloc(struct dpt *dpt, pud_t *pud, unsigned long addr) +{ + struct page *page; + pmd_t *pmd; + int err; + + if (pud_none(*pud)) { + page = alloc_page(GFP_KERNEL | __GFP_ZERO); + if (!page) + return ERR_PTR(-ENOMEM); + pmd = (pmd_t *)page_address(page); + err = dpt_add_backend_page(dpt, pmd, PGT_LEVEL_PMD); + if (err) { + free_page((unsigned long)pmd); + return ERR_PTR(err); + } + set_pud_safe(pud, __pud(__pa(pmd) | _KERNPG_TABLE)); + pmd = pmd_offset(pud, addr); + } else { + pmd = dpt_pmd_offset(dpt, pud, addr); + } + + return pmd; +} + +static pud_t *dpt_pud_alloc(struct dpt *dpt, p4d_t *p4d, unsigned long addr) +{ + struct page *page; + pud_t *pud; + int err; + + if (p4d_none(*p4d)) { + page = alloc_page(GFP_KERNEL | __GFP_ZERO); + if (!page) + return ERR_PTR(-ENOMEM); + pud = (pud_t *)page_address(page); + err = dpt_add_backend_page(dpt, pud, PGT_LEVEL_PUD); + if (err) { + free_page((unsigned long)pud); + return ERR_PTR(err); + } + set_p4d_safe(p4d, __p4d(__pa(pud) | _KERNPG_TABLE)); + pud = pud_offset(p4d, addr); + } else { + pud = dpt_pud_offset(dpt, p4d, addr); + } + + return pud; +} + +static p4d_t *dpt_p4d_alloc(struct dpt *dpt, pgd_t *pgd, unsigned long addr) +{ + struct page *page; + p4d_t *p4d; + int err; + + if (!pgtable_l5_enabled()) + return (p4d_t *)pgd; + + if (pgd_none(*pgd)) { + page = alloc_page(GFP_KERNEL | __GFP_ZERO); + if (!page) + return ERR_PTR(-ENOMEM); + p4d = (p4d_t *)page_address(page); + err = dpt_add_backend_page(dpt, p4d, PGT_LEVEL_P4D); + if (err) { + free_page((unsigned long)p4d); + return ERR_PTR(err); + } + set_pgd_safe(pgd, __pgd(__pa(p4d) | _KERNPG_TABLE)); + p4d = p4d_offset(pgd, addr); + } else { + p4d = dpt_p4d_offset(dpt, pgd, addr); + } + + return p4d; +} + /* * dpt_create - allocate a page-table and create a corresponding * decorated page-table. The page-table is allocated and aligned From patchwork Mon May 4 14:58:02 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 11526465 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9628118EC for ; Mon, 4 May 2020 15:00:07 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5835721655 for ; Mon, 4 May 2020 15:00:07 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="kovaqTW9" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5835721655 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 1E94E8E002F; Mon, 4 May 2020 11:00:06 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 19C348E0024; Mon, 4 May 2020 11:00:06 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 089AE8E002F; Mon, 4 May 2020 11:00:06 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0062.hostedemail.com [216.40.44.62]) by kanga.kvack.org (Postfix) with ESMTP id E2A0D8E0024 for ; Mon, 4 May 2020 11:00:05 -0400 (EDT) Received: from smtpin09.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id A974E18F5D for ; Mon, 4 May 2020 15:00:05 +0000 (UTC) X-FDA: 76779346770.09.sea70_3aea0b4b77254 X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,alexandre.chartre@oracle.com,,RULES_HIT:30054:30064:30070,0,RBL:156.151.31.86:@oracle.com:.lbl8.mailshell.net-64.10.201.10 62.18.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:ft,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:26,LUA_SUMMARY:none X-HE-Tag: sea70_3aea0b4b77254 X-Filterd-Recvd-Size: 7757 Received: from userp2130.oracle.com (userp2130.oracle.com [156.151.31.86]) by imf48.hostedemail.com (Postfix) with ESMTP for ; Mon, 4 May 2020 15:00:04 +0000 (UTC) Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 044Elb0V116465; Mon, 4 May 2020 14:59:55 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2020-01-29; bh=sEQgjOdo4evI2vW/F7vLXsA4dt9pdUoHUZxHqkXr9QA=; b=kovaqTW9aibWjjOMYJyd5NFACyd7FB91PiCgvJWE159s4VLHkgk8SO2v0OxgU+W2LuYx 1crYD2Cy3Z+BDMesBUvZN4pir+op/dLUD78XgAlF7bc6fSjypaWY2GkqUuRP5CATgm/R lhEYlPvfgi5OIuZ3OiNMJgOqqwzbtXtR8wf4a1XKJApZK6Gvhl8cdqX0ITQzzPPoYaTG XWAqnccys3S3GzN5Plit7Fampc6/mwxd86NtLnJyZeYz49iC4pn0yoZfO8PfTAfdpayK UjfPWQiZs51ZKRYCc23YjN+4wj6DyheCs8UZ8dwUO4AhzMD6Hc49aq+41WVb+dizfi6u rw== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by userp2130.oracle.com with ESMTP id 30s09qyfrk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 04 May 2020 14:59:55 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 044EmVHV105667; Mon, 4 May 2020 14:59:54 GMT Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by userp3020.oracle.com with ESMTP id 30sjjw0g7j-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 04 May 2020 14:59:54 +0000 Received: from abhmp0001.oracle.com (abhmp0001.oracle.com [141.146.116.7]) by aserv0122.oracle.com (8.14.4/8.14.4) with ESMTP id 044ExrQj001259; Mon, 4 May 2020 14:59:53 GMT Received: from linux-1.home.com (/10.175.9.166) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 04 May 2020 07:59:52 -0700 From: Alexandre Chartre To: tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: pbonzini@redhat.com, konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, junaids@google.com, graf@amazon.de, rppt@linux.vnet.ibm.com, kuzuno@gmail.com, mgross@linux.intel.com, alexandre.chartre@oracle.com Subject: [RFC v4][PATCH part-2 05/13] mm/dpt: Add decorated page-table entry set functions Date: Mon, 4 May 2020 16:58:02 +0200 Message-Id: <20200504145810.11882-6-alexandre.chartre@oracle.com> X-Mailer: git-send-email 2.18.2 In-Reply-To: <20200504145810.11882-1-alexandre.chartre@oracle.com> References: <20200504145810.11882-1-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9610 signatures=668687 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxscore=0 phishscore=0 bulkscore=0 malwarescore=0 spamscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2005040123 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9610 signatures=668687 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 mlxscore=0 lowpriorityscore=0 spamscore=0 adultscore=0 clxscore=1015 suspectscore=0 priorityscore=1501 malwarescore=0 mlxlogscore=999 phishscore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2005040123 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add wrappers around the page table entry (pgd/p4d/pud/pmd) set functions which check that an existing entry is not being overwritten. Signed-off-by: Alexandre Chartre --- arch/x86/mm/dpt.c | 126 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 126 insertions(+) diff --git a/arch/x86/mm/dpt.c b/arch/x86/mm/dpt.c index a2f54ba00255..7a1b4cd53b03 100644 --- a/arch/x86/mm/dpt.c +++ b/arch/x86/mm/dpt.c @@ -258,6 +258,132 @@ static p4d_t *dpt_p4d_alloc(struct dpt *dpt, pgd_t *pgd, unsigned long addr) return p4d; } +/* + * dpt_set_pXX() functions are equivalent to kernel set_pXX() functions + * but, in addition, they ensure that they are not overwriting an already + * existing reference in the decorated page table. Otherwise an error is + * returned. + */ + +static int dpt_set_pte(struct dpt *dpt, pte_t *pte, pte_t pte_value) +{ +#ifdef DEBUG + /* + * The pte pointer should come from dpt_pte_alloc() or dpt_pte_offset() + * both of which check if the pointer is in the decorated page table. + * So this is a paranoid check to ensure the pointer is really in the + * decorated page table. + */ + if (!dpt_valid_offset(dpt, pte)) { + pr_err("DPT %p: PTE %px not found\n", dpt, pte); + return -EINVAL; + } +#endif + set_pte(pte, pte_value); + + return 0; +} + +static int dpt_set_pmd(struct dpt *dpt, pmd_t *pmd, pmd_t pmd_value) +{ +#ifdef DEBUG + /* + * The pmd pointer should come from dpt_pmd_alloc() or dpt_pmd_offset() + * both of which check if the pointer is in the decorated page table. + * So this is a paranoid check to ensure the pointer is really in the + * decorated page table. + */ + if (!dpt_valid_offset(dpt, pmd)) { + pr_err("DPT %p: PMD %px not found\n", dpt, pmd); + return -EINVAL; + } +#endif + if (pmd_val(*pmd) == pmd_val(pmd_value)) + return 0; + + if (!pmd_none(*pmd)) { + pr_err("DPT %p: PMD %px overwriting %lx with %lx\n", + dpt, pmd, pmd_val(*pmd), pmd_val(pmd_value)); + return -EBUSY; + } + + set_pmd(pmd, pmd_value); + + return 0; +} + +static int dpt_set_pud(struct dpt *dpt, pud_t *pud, pud_t pud_value) +{ +#ifdef DEBUG + /* + * The pud pointer should come from dpt_pud_alloc() or dpt_pud_offset() + * both of which check if the pointer is in the decorated page table. + * So this is a paranoid check to ensure the pointer is really in the + * decorated page table. + */ + if (!dpt_valid_offset(dpt, pud)) { + pr_err("DPT %p: PUD %px not found\n", dpt, pud); + return -EINVAL; + } +#endif + if (pud_val(*pud) == pud_val(pud_value)) + return 0; + + if (!pud_none(*pud)) { + pr_err("DPT %p: PUD %px overwriting %lx with %lx\n", + dpt, pud, pud_val(*pud), pud_val(pud_value)); + return -EBUSY; + } + + set_pud(pud, pud_value); + + return 0; +} + +static int dpt_set_p4d(struct dpt *dpt, p4d_t *p4d, p4d_t p4d_value) +{ +#ifdef DEBUG + /* + * The p4d pointer should come from dpt_p4d_alloc() or dpt_p4d_offset() + * both of which check if the pointer is in the decorated page table. + * So this is a paranoid check to ensure the pointer is really in the + * decorated page table. + */ + if (!dpt_valid_offset(dpt, p4d)) { + pr_err("DPT %p: P4D %px not found\n", dpt, p4d); + return -EINVAL; + } +#endif + if (p4d_val(*p4d) == p4d_val(p4d_value)) + return 0; + + if (!p4d_none(*p4d)) { + pr_err("DPT %p: P4D %px overwriting %lx with %lx\n", + dpt, p4d, p4d_val(*p4d), p4d_val(p4d_value)); + return -EBUSY; + } + + set_p4d(p4d, p4d_value); + + return 0; +} + +static int dpt_set_pgd(struct dpt *dpt, pgd_t *pgd, pgd_t pgd_value) +{ + if (pgd_val(*pgd) == pgd_val(pgd_value)) + return 0; + + if (!pgd_none(*pgd)) { + pr_err("DPT %p: PGD %px overwriting %lx with %lx\n", + dpt, pgd, pgd_val(*pgd), pgd_val(pgd_value)); + return -EBUSY; + } + + set_pgd(pgd, pgd_value); + + return 0; +} + /* * dpt_create - allocate a page-table and create a corresponding * decorated page-table. The page-table is allocated and aligned From patchwork Mon May 4 14:58:03 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 11527291 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EFE2315E6 for ; Mon, 4 May 2020 17:40:37 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id A94FE206A4 for ; Mon, 4 May 2020 17:40:37 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="Qc3jDiKW" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A94FE206A4 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id A71D48E0068; Mon, 4 May 2020 13:40:36 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id A23288E0058; Mon, 4 May 2020 13:40:36 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 911778E0068; Mon, 4 May 2020 13:40:36 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0175.hostedemail.com [216.40.44.175]) by kanga.kvack.org (Postfix) with ESMTP id 76B6E8E0058 for ; Mon, 4 May 2020 13:40:36 -0400 (EDT) Received: from smtpin26.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 36AAB18F61 for ; Mon, 4 May 2020 17:40:36 +0000 (UTC) X-FDA: 76779751272.26.hen80_525555453d4b X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,alexandre.chartre@oracle.com,,RULES_HIT:30054:30064:30070:30079,0,RBL:156.151.31.86:@oracle.com:.lbl8.mailshell.net-62.18.0.100 64.10.201.10,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:ft,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: hen80_525555453d4b X-Filterd-Recvd-Size: 10688 Received: from userp2130.oracle.com (userp2130.oracle.com [156.151.31.86]) by imf48.hostedemail.com (Postfix) with ESMTP for ; Mon, 4 May 2020 17:40:35 +0000 (UTC) Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 044Elblu116453; Mon, 4 May 2020 14:59:59 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2020-01-29; bh=OTEdK+JfE5NUVbySnV/SKcjHMMG3B870KclVub1fY4A=; b=Qc3jDiKW5iRpfH4T4cg4am63x66ajlrkrmv2BFaRP08O1ep940mwx/tYWRutDrS60i53 cQJLoLkzQnh6ewWeKHs2OO+9+2FJuC/VUFdFv1j/Rhlb201fHpvknSbo7Cr13/4l9Pvu z1kvUoKtAyBqHQLBrtjFzr8gCme0OsHDV+PhWXEJFPRWSiQZzpGcDBiB9EsCL+7a/6Zq t4s6YNtc4ZSuLkUfkOzL64yqQ7Gzyl8NTHZ9mFDY/ddc6jRwqHqo+PpUgPrMePm4IHP+ NwjBcTljbBPwzLJ7g1KRwFo2jygkIVLBcc5A9XzlPFFRFUwp1AckrrOZPmthVN9eTIYs Mw== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by userp2130.oracle.com with ESMTP id 30s09qyfs5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 04 May 2020 14:59:58 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 044EmVFx105642; Mon, 4 May 2020 14:59:58 GMT Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by userp3020.oracle.com with ESMTP id 30sjjw0gea-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 04 May 2020 14:59:58 +0000 Received: from abhmp0001.oracle.com (abhmp0001.oracle.com [141.146.116.7]) by aserv0122.oracle.com (8.14.4/8.14.4) with ESMTP id 044ExvW4001276; Mon, 4 May 2020 14:59:57 GMT Received: from linux-1.home.com (/10.175.9.166) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 04 May 2020 07:59:56 -0700 From: Alexandre Chartre To: tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: pbonzini@redhat.com, konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, junaids@google.com, graf@amazon.de, rppt@linux.vnet.ibm.com, kuzuno@gmail.com, mgross@linux.intel.com, alexandre.chartre@oracle.com Subject: [RFC v4][PATCH part-2 06/13] mm/dpt: Functions to populate a decorated page-table from a VA range Date: Mon, 4 May 2020 16:58:03 +0200 Message-Id: <20200504145810.11882-7-alexandre.chartre@oracle.com> X-Mailer: git-send-email 2.18.2 In-Reply-To: <20200504145810.11882-1-alexandre.chartre@oracle.com> References: <20200504145810.11882-1-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9610 signatures=668687 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxscore=0 phishscore=0 bulkscore=0 malwarescore=0 spamscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2005040123 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9610 signatures=668687 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 mlxscore=0 lowpriorityscore=0 spamscore=0 adultscore=0 clxscore=1015 suspectscore=0 priorityscore=1501 malwarescore=0 mlxlogscore=999 phishscore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2005040123 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Provide functions to copy page-table entries from the kernel page-table to a decorated page-table for a specified VA range. These functions are based on the copy_pxx_range() functions defined in mm/memory.c. A first difference is that a level parameter can be specified to indicate the page-table level (PGD, P4D, PUD PMD, PTE) at which the copy should be done. Also functions don't rely on mm or vma, and they don't alter the source page-table even if an entry is bad. Finally, the VA range start and size don't need to be page-aligned. Signed-off-by: Alexandre Chartre --- arch/x86/include/asm/dpt.h | 3 + arch/x86/mm/dpt.c | 205 +++++++++++++++++++++++++++++++++++++ 2 files changed, 208 insertions(+) diff --git a/arch/x86/include/asm/dpt.h b/arch/x86/include/asm/dpt.h index b9cba051ebf2..85d2c5051acb 100644 --- a/arch/x86/include/asm/dpt.h +++ b/arch/x86/include/asm/dpt.h @@ -40,5 +40,8 @@ struct dpt { extern struct dpt *dpt_create(unsigned int pgt_alignment); extern void dpt_destroy(struct dpt *dpt); +extern int dpt_map_range(struct dpt *dpt, void *ptr, size_t size, + enum page_table_level level); +extern int dpt_map(struct dpt *dpt, void *ptr, unsigned long size); #endif diff --git a/arch/x86/mm/dpt.c b/arch/x86/mm/dpt.c index 7a1b4cd53b03..0e725344b921 100644 --- a/arch/x86/mm/dpt.c +++ b/arch/x86/mm/dpt.c @@ -384,6 +384,211 @@ static int dpt_set_pgd(struct dpt *dpt, pgd_t *pgd, pgd_t pgd_value) return 0; } +static int dpt_copy_pte_range(struct dpt *dpt, pmd_t *dst_pmd, pmd_t *src_pmd, + unsigned long addr, unsigned long end) +{ + pte_t *src_pte, *dst_pte; + + dst_pte = dpt_pte_alloc(dpt, dst_pmd, addr); + if (IS_ERR(dst_pte)) + return PTR_ERR(dst_pte); + + addr &= PAGE_MASK; + src_pte = pte_offset_map(src_pmd, addr); + + do { + dpt_set_pte(dpt, dst_pte, *src_pte); + + } while (dst_pte++, src_pte++, addr += PAGE_SIZE, addr < end); + + return 0; +} + +static int dpt_copy_pmd_range(struct dpt *dpt, pud_t *dst_pud, pud_t *src_pud, + unsigned long addr, unsigned long end, + enum page_table_level level) +{ + pmd_t *src_pmd, *dst_pmd; + unsigned long next; + int err; + + dst_pmd = dpt_pmd_alloc(dpt, dst_pud, addr); + if (IS_ERR(dst_pmd)) + return PTR_ERR(dst_pmd); + + src_pmd = pmd_offset(src_pud, addr); + + do { + next = pmd_addr_end(addr, end); + if (level == PGT_LEVEL_PMD || pmd_none(*src_pmd) || + pmd_trans_huge(*src_pmd) || pmd_devmap(*src_pmd)) { + err = dpt_set_pmd(dpt, dst_pmd, *src_pmd); + if (err) + return err; + continue; + } + + if (!pmd_present(*src_pmd)) { + pr_warn("DPT %p: PMD not present for [%lx,%lx]\n", + dpt, addr, next - 1); + pmd_clear(dst_pmd); + continue; + } + + err = dpt_copy_pte_range(dpt, dst_pmd, src_pmd, addr, next); + if (err) { + pr_err("DPT %p: PMD error copying PTE addr=%lx next=%lx\n", + dpt, addr, next); + return err; + } + + } while (dst_pmd++, src_pmd++, addr = next, addr < end); + + return 0; +} + +static int dpt_copy_pud_range(struct dpt *dpt, p4d_t *dst_p4d, p4d_t *src_p4d, + unsigned long addr, unsigned long end, + enum page_table_level level) +{ + pud_t *src_pud, *dst_pud; + unsigned long next; + int err; + + dst_pud = dpt_pud_alloc(dpt, dst_p4d, addr); + if (IS_ERR(dst_pud)) + return PTR_ERR(dst_pud); + + src_pud = pud_offset(src_p4d, addr); + + do { + next = pud_addr_end(addr, end); + if (level == PGT_LEVEL_PUD || pud_none(*src_pud) || + pud_trans_huge(*src_pud) || pud_devmap(*src_pud)) { + err = dpt_set_pud(dpt, dst_pud, *src_pud); + if (err) + return err; + continue; + } + + err = dpt_copy_pmd_range(dpt, dst_pud, src_pud, addr, next, + level); + if (err) { + pr_err("DPT %p: PUD error copying PMD addr=%lx next=%lx\n", + dpt, addr, next); + return err; + } + + } while (dst_pud++, src_pud++, addr = next, addr < end); + + return 0; +} + +static int dpt_copy_p4d_range(struct dpt *dpt, pgd_t *dst_pgd, pgd_t *src_pgd, + unsigned long addr, unsigned long end, + enum page_table_level level) +{ + p4d_t *src_p4d, *dst_p4d; + unsigned long next; + int err; + + dst_p4d = dpt_p4d_alloc(dpt, dst_pgd, addr); + if (IS_ERR(dst_p4d)) + return PTR_ERR(dst_p4d); + + src_p4d = p4d_offset(src_pgd, addr); + + do { + next = p4d_addr_end(addr, end); + if (level == PGT_LEVEL_P4D || p4d_none(*src_p4d)) { + err = dpt_set_p4d(dpt, dst_p4d, *src_p4d); + if (err) + return err; + continue; + } + + err = dpt_copy_pud_range(dpt, dst_p4d, src_p4d, addr, next, + level); + if (err) { + pr_err("DPT %p: P4D error copying PUD addr=%lx next=%lx\n", + dpt, addr, next); + return err; + } + + } while (dst_p4d++, src_p4d++, addr = next, addr < end); + + return 0; +} + +static int dpt_copy_pgd_range(struct dpt *dpt, + pgd_t *dst_pagetable, pgd_t *src_pagetable, + unsigned long addr, unsigned long end, + enum page_table_level level) +{ + pgd_t *src_pgd, *dst_pgd; + unsigned long next; + int err; + + dst_pgd = pgd_offset_pgd(dst_pagetable, addr); + src_pgd = pgd_offset_pgd(src_pagetable, addr); + + do { + next = pgd_addr_end(addr, end); + if (level == PGT_LEVEL_PGD || pgd_none(*src_pgd)) { + err = dpt_set_pgd(dpt, dst_pgd, *src_pgd); + if (err) + return err; + continue; + } + + err = dpt_copy_p4d_range(dpt, dst_pgd, src_pgd, addr, next, + level); + if (err) { + pr_err("DPT %p: PGD error copying P4D addr=%lx next=%lx\n", + dpt, addr, next); + return err; + } + + } while (dst_pgd++, src_pgd++, addr = next, addr < end); + + return 0; +} + +/* + * Copy page table entries from the current page table (i.e. from the + * kernel page table) to the specified decorated page-table. The level + * parameter specifies the page-table level (PGD, P4D, PUD PMD, PTE) + * at which the copy should be done. + */ +int dpt_map_range(struct dpt *dpt, void *ptr, size_t size, + enum page_table_level level) +{ + unsigned long addr = (unsigned long)ptr; + unsigned long end = addr + ((unsigned long)size); + unsigned long flags; + int err; + + pr_debug("DPT %p: MAP %px/%lx/%d\n", dpt, ptr, size, level); + + spin_lock_irqsave(&dpt->lock, flags); + err = dpt_copy_pgd_range(dpt, dpt->pagetable, current->mm->pgd, + addr, end, level); + spin_unlock_irqrestore(&dpt->lock, flags); + + return err; +} +EXPORT_SYMBOL(dpt_map_range); + +/* + * Copy page-table PTE entries from the current page-table to the + * specified decorated page-table. + */ +int dpt_map(struct dpt *dpt, void *ptr, unsigned long size) +{ + return dpt_map_range(dpt, ptr, size, PGT_LEVEL_PTE); +} +EXPORT_SYMBOL(dpt_map); + /* * dpt_create - allocate a page-table and create a corresponding * decorated page-table. The page-table is allocated and aligned From patchwork Mon May 4 14:58:04 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 11526467 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9757617E8 for ; Mon, 4 May 2020 15:00:17 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6335C206D7 for ; Mon, 4 May 2020 15:00:17 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="WJAcFefZ" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6335C206D7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 4810A8E0030; Mon, 4 May 2020 11:00:16 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 4322E8E0024; Mon, 4 May 2020 11:00:16 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 348108E0030; Mon, 4 May 2020 11:00:16 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0062.hostedemail.com [216.40.44.62]) by kanga.kvack.org (Postfix) with ESMTP id 19EFE8E0024 for ; Mon, 4 May 2020 11:00:16 -0400 (EDT) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id C9E4C173DC for ; Mon, 4 May 2020 15:00:15 +0000 (UTC) X-FDA: 76779347190.04.hot71_3c619e077c12c X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,alexandre.chartre@oracle.com,,RULES_HIT:30054:30064:30079,0,RBL:141.146.126.78:@oracle.com:.lbl8.mailshell.net-64.10.201.10 62.18.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:ft,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: hot71_3c619e077c12c X-Filterd-Recvd-Size: 5021 Received: from aserp2120.oracle.com (aserp2120.oracle.com [141.146.126.78]) by imf25.hostedemail.com (Postfix) with ESMTP for ; Mon, 4 May 2020 15:00:14 +0000 (UTC) Received: from pps.filterd (aserp2120.oracle.com [127.0.0.1]) by aserp2120.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 044EnCoh024792; Mon, 4 May 2020 15:00:03 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2020-01-29; bh=hsMzy6wx1/DEdi8suQdgbXWe0Maq85/ZgmlxYBdz/2E=; b=WJAcFefZyCtjaUHMoZAZ8J2amgJ79c/i5nPycj0M7U47/tKdAIox7kCo9sAXBqN8v0pf Qdw7QAmJgjSU+Ooe6Yl9dhCs/R1IlNJ1eJ4lzQhKr3misdTrE9U/oEukuT4+3SE9HVV6 75V446d6RBXXGCy4ecWMmCgyhhQfFgh7hMtuEhNo8BNCOScXeh+KGq5VJGmnQpBK5VZz 3NbYqq38tpin9A/KCUAS57wDKY7XHS19vhpt4NUyX93XweuAlRmSBn5FbTh7v5kMxLpr B9ozcwXIjpAwKxCEQSeyTR6C7b/1aqePCDlrh163eHTTAGHxCO3qvmwfXDDCkFtVZsiR Pw== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by aserp2120.oracle.com with ESMTP id 30s0tm7dkr-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 04 May 2020 15:00:03 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 044ElM7o033871; Mon, 4 May 2020 15:00:03 GMT Received: from userv0122.oracle.com (userv0122.oracle.com [156.151.31.75]) by aserp3020.oracle.com with ESMTP id 30sjnaw244-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 04 May 2020 15:00:03 +0000 Received: from abhmp0001.oracle.com (abhmp0001.oracle.com [141.146.116.7]) by userv0122.oracle.com (8.14.4/8.14.4) with ESMTP id 044F01rW022337; Mon, 4 May 2020 15:00:01 GMT Received: from linux-1.home.com (/10.175.9.166) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 04 May 2020 08:00:00 -0700 From: Alexandre Chartre To: tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: pbonzini@redhat.com, konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, junaids@google.com, graf@amazon.de, rppt@linux.vnet.ibm.com, kuzuno@gmail.com, mgross@linux.intel.com, alexandre.chartre@oracle.com Subject: [RFC v4][PATCH part-2 07/13] mm/dpt: Helper functions to map module into a decorated page-table Date: Mon, 4 May 2020 16:58:04 +0200 Message-Id: <20200504145810.11882-8-alexandre.chartre@oracle.com> X-Mailer: git-send-email 2.18.2 In-Reply-To: <20200504145810.11882-1-alexandre.chartre@oracle.com> References: <20200504145810.11882-1-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9610 signatures=668687 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 adultscore=0 phishscore=0 mlxlogscore=999 bulkscore=0 malwarescore=0 spamscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2005040123 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9610 signatures=668687 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 mlxscore=0 priorityscore=1501 lowpriorityscore=0 spamscore=0 suspectscore=0 phishscore=0 clxscore=1015 bulkscore=0 mlxlogscore=999 adultscore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2005040123 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add helper functions to easily map a module into a decorated page-table. Signed-off-by: Alexandre Chartre --- arch/x86/include/asm/dpt.h | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/arch/x86/include/asm/dpt.h b/arch/x86/include/asm/dpt.h index 85d2c5051acb..5a38d97a70a8 100644 --- a/arch/x86/include/asm/dpt.h +++ b/arch/x86/include/asm/dpt.h @@ -2,6 +2,7 @@ #ifndef ARCH_X86_MM_DPT_H #define ARCH_X86_MM_DPT_H +#include #include #include @@ -44,4 +45,24 @@ extern int dpt_map_range(struct dpt *dpt, void *ptr, size_t size, enum page_table_level level); extern int dpt_map(struct dpt *dpt, void *ptr, unsigned long size); +static inline int dpt_map_module(struct dpt *dpt, char *module_name) +{ + struct module *module; + + module = find_module(module_name); + if (!module) + return -ESRCH; + + return dpt_map(dpt, module->core_layout.base, module->core_layout.size); +} + +/* + * Copy the memory mapping for the current module. This is defined as a + * macro to ensure it is expanded in the module making the call so that + * THIS_MODULE has the correct value. + */ +#define DPT_MAP_THIS_MODULE(dpt) \ + (dpt_map(dpt, THIS_MODULE->core_layout.base, \ + THIS_MODULE->core_layout.size)) + #endif From patchwork Mon May 4 14:58:05 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 11526469 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DD2C317E8 for ; Mon, 4 May 2020 15:00:20 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9B3C1206D7 for ; Mon, 4 May 2020 15:00:20 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="k3nxkLIW" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9B3C1206D7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 25F5C8E0031; Mon, 4 May 2020 11:00:19 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 210FA8E0024; Mon, 4 May 2020 11:00:19 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0D8BB8E0031; Mon, 4 May 2020 11:00:19 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0214.hostedemail.com [216.40.44.214]) by kanga.kvack.org (Postfix) with ESMTP id E97F28E0024 for ; Mon, 4 May 2020 11:00:18 -0400 (EDT) Received: from smtpin20.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id A332B181AC9CC for ; Mon, 4 May 2020 15:00:18 +0000 (UTC) X-FDA: 76779347316.20.loaf26_3cd2ab8bf5f0d X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,alexandre.chartre@oracle.com,,RULES_HIT:30003:30054:30064:30070,0,RBL:156.151.31.86:@oracle.com:.lbl8.mailshell.net-64.10.201.10 62.18.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:ft,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: loaf26_3cd2ab8bf5f0d X-Filterd-Recvd-Size: 8315 Received: from userp2130.oracle.com (userp2130.oracle.com [156.151.31.86]) by imf03.hostedemail.com (Postfix) with ESMTP for ; Mon, 4 May 2020 15:00:17 +0000 (UTC) Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 044ElbNQ116459; Mon, 4 May 2020 15:00:08 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2020-01-29; bh=K4WuIRzv6NwwGlBbEMaCvqf1ii2XJ+sibBt35OcpXCQ=; b=k3nxkLIW+pEF/Y1ilQRHwL3pob0Wn2rsWjG7ffWhhxZMxX3ArXWlk53vjW8dlpCraVDy imbEqF68auq3T0FZwjgk1kOkrKy/xfFtKr8jSV6L5g6z3OAzfru1i2z0TUkJfh+CpwQT NyFQkYbHWh8Ywk5aRwA/AEWEijwE2FmJCgklc2rjAOQ8pS7HE3+3PhS6s4xKa++fb5Wo oKeG1oYIf/DWcAAq7TKZoX3+2CdjEab8V32ZbOM9qAcB/X4opRlwbftrikc5VTofvRq0 LA+86v60a5hycNhDTXyeOZ6fHqTWUW3ZTqYMQmC294WtBK+U6JXpvL08u/sDuSUbsIiY Cg== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by userp2130.oracle.com with ESMTP id 30s09qyftp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 04 May 2020 15:00:08 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 044ElLhg033730; Mon, 4 May 2020 15:00:07 GMT Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by aserp3020.oracle.com with ESMTP id 30sjnaw2ej-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 04 May 2020 15:00:07 +0000 Received: from abhmp0001.oracle.com (abhmp0001.oracle.com [141.146.116.7]) by aserv0122.oracle.com (8.14.4/8.14.4) with ESMTP id 044F06lt001343; Mon, 4 May 2020 15:00:06 GMT Received: from linux-1.home.com (/10.175.9.166) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 04 May 2020 08:00:05 -0700 From: Alexandre Chartre To: tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: pbonzini@redhat.com, konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, junaids@google.com, graf@amazon.de, rppt@linux.vnet.ibm.com, kuzuno@gmail.com, mgross@linux.intel.com, alexandre.chartre@oracle.com Subject: [RFC v4][PATCH part-2 08/13] mm/dpt: Keep track of VA ranges mapped in a decorated page-table Date: Mon, 4 May 2020 16:58:05 +0200 Message-Id: <20200504145810.11882-9-alexandre.chartre@oracle.com> X-Mailer: git-send-email 2.18.2 In-Reply-To: <20200504145810.11882-1-alexandre.chartre@oracle.com> References: <20200504145810.11882-1-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9610 signatures=668687 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 adultscore=0 phishscore=0 mlxlogscore=999 bulkscore=0 malwarescore=0 spamscore=0 suspectscore=2 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2005040123 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9610 signatures=668687 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 mlxscore=0 lowpriorityscore=0 spamscore=0 adultscore=0 clxscore=1015 suspectscore=2 priorityscore=1501 malwarescore=0 mlxlogscore=999 phishscore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2005040123 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add functions to keep track of VA ranges mapped in a decorated page-table. This will be used when unmapping to ensure the same range is unmapped, at the same page-table level. This will also be used to handle mapping and unmapping of overlapping VA ranges. Signed-off-by: Alexandre Chartre --- arch/x86/include/asm/dpt.h | 12 ++++++++ arch/x86/mm/dpt.c | 60 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 72 insertions(+) diff --git a/arch/x86/include/asm/dpt.h b/arch/x86/include/asm/dpt.h index 5a38d97a70a8..0d74afb10141 100644 --- a/arch/x86/include/asm/dpt.h +++ b/arch/x86/include/asm/dpt.h @@ -16,6 +16,17 @@ enum page_table_level { PGT_LEVEL_PGD }; +/* + * Structure to keep track of address ranges mapped into a decorated + * page-table. + */ +struct dpt_range_mapping { + struct list_head list; + void *ptr; /* range start address */ + size_t size; /* range size */ + enum page_table_level level; /* mapping level */ +}; + /* * A decorated page-table (dpt) encapsulates a native page-table (e.g. * a PGD) and maintain additional attributes related to this page-table. @@ -24,6 +35,7 @@ struct dpt { spinlock_t lock; /* protect all attributes */ pgd_t *pagetable; /* the actual page-table */ unsigned int alignment; /* page-table alignment */ + struct list_head mapping_list; /* list of VA range mapping */ /* * A page-table can have direct references to another page-table, diff --git a/arch/x86/mm/dpt.c b/arch/x86/mm/dpt.c index 0e725344b921..12eb0d794d84 100644 --- a/arch/x86/mm/dpt.c +++ b/arch/x86/mm/dpt.c @@ -59,6 +59,24 @@ static int dpt_add_backend_page(struct dpt *dpt, void *addr, return 0; } +/* + * Return the range mapping starting at the specified address, or NULL if + * no such range is found. + */ +static struct dpt_range_mapping *dpt_get_range_mapping(struct dpt *dpt, + void *ptr) +{ + struct dpt_range_mapping *range; + + lockdep_assert_held(&dpt->lock); + list_for_each_entry(range, &dpt->mapping_list, list) { + if (range->ptr == ptr) + return range; + } + + return NULL; +} + /* * Check if an offset in the page-table is valid, i.e. check that the * offset is on a page effectively belonging to the page-table. @@ -563,6 +581,7 @@ static int dpt_copy_pgd_range(struct dpt *dpt, int dpt_map_range(struct dpt *dpt, void *ptr, size_t size, enum page_table_level level) { + struct dpt_range_mapping *range_mapping; unsigned long addr = (unsigned long)ptr; unsigned long end = addr + ((unsigned long)size); unsigned long flags; @@ -571,8 +590,36 @@ int dpt_map_range(struct dpt *dpt, void *ptr, size_t size, pr_debug("DPT %p: MAP %px/%lx/%d\n", dpt, ptr, size, level); spin_lock_irqsave(&dpt->lock, flags); + + /* check if the range is already mapped */ + range_mapping = dpt_get_range_mapping(dpt, ptr); + if (range_mapping) { + pr_debug("DPT %p: MAP %px/%lx/%d already mapped\n", + dpt, ptr, size, level); + err = -EBUSY; + goto done; + } + + /* map new range */ + range_mapping = kmalloc(sizeof(*range_mapping), GFP_KERNEL); + if (!range_mapping) { + err = -ENOMEM; + goto done; + } + err = dpt_copy_pgd_range(dpt, dpt->pagetable, current->mm->pgd, addr, end, level); + if (err) { + kfree(range_mapping); + goto done; + } + + INIT_LIST_HEAD(&range_mapping->list); + range_mapping->ptr = ptr; + range_mapping->size = size; + range_mapping->level = level; + list_add(&range_mapping->list, &dpt->mapping_list); +done: spin_unlock_irqrestore(&dpt->lock, flags); return err; @@ -611,6 +658,8 @@ struct dpt *dpt_create(unsigned int pgt_alignment) if (!dpt) return NULL; + INIT_LIST_HEAD(&dpt->mapping_list); + pagetable = (unsigned long)__get_free_pages(GFP_KERNEL_ACCOUNT | __GFP_ZERO, alloc_order); @@ -632,6 +681,7 @@ void dpt_destroy(struct dpt *dpt) { unsigned int pgt_alignment; unsigned int alloc_order; + struct dpt_range_mapping *range, *range_next; unsigned long index; void *entry; @@ -643,6 +693,16 @@ void dpt_destroy(struct dpt *dpt) free_page((unsigned long)DPT_BACKEND_PAGE_ADDR(entry)); } + list_for_each_entry_safe(range, range_next, &dpt->mapping_list, list) { + list_del(&range->list); + kfree(range); + } + + if (dpt->backend_pages_count) { + xa_for_each(&dpt->backend_pages, index, entry) + free_page((unsigned long)DPT_BACKEND_PAGE_ADDR(entry)); + } + if (dpt->pagetable) { pgt_alignment = dpt->alignment; alloc_order = round_up(PAGE_SIZE + pgt_alignment, From patchwork Mon May 4 14:58:06 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 11526471 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 00B5B15AB for ; Mon, 4 May 2020 15:00:31 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id B35A5206D7 for ; Mon, 4 May 2020 15:00:30 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="xPvV2kRQ" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org B35A5206D7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B607E8E0032; Mon, 4 May 2020 11:00:29 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id B37478E0024; Mon, 4 May 2020 11:00:29 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A4C638E0032; Mon, 4 May 2020 11:00:29 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0088.hostedemail.com [216.40.44.88]) by kanga.kvack.org (Postfix) with ESMTP id 86CEA8E0024 for ; Mon, 4 May 2020 11:00:29 -0400 (EDT) Received: from smtpin07.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 2D561181AEF31 for ; Mon, 4 May 2020 15:00:29 +0000 (UTC) X-FDA: 76779347778.07.group58_3e5bb232eb64e X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,alexandre.chartre@oracle.com,,RULES_HIT:30003:30054:30064:30070:30075,0,RBL:141.146.126.78:@oracle.com:.lbl8.mailshell.net-64.10.201.10 62.18.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:ft,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:0,LUA_SUMMARY:none X-HE-Tag: group58_3e5bb232eb64e X-Filterd-Recvd-Size: 8711 Received: from aserp2120.oracle.com (aserp2120.oracle.com [141.146.126.78]) by imf34.hostedemail.com (Postfix) with ESMTP for ; Mon, 4 May 2020 15:00:28 +0000 (UTC) Received: from pps.filterd (aserp2120.oracle.com [127.0.0.1]) by aserp2120.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 044En9bZ024708; Mon, 4 May 2020 15:00:15 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2020-01-29; bh=GsovF01BLBXADs9ogvbBPifuM8m9iNcoQzHGafVfZWQ=; b=xPvV2kRQBk3Nlt40ptkPC9KCMt9BYKDiPFbPmxAgFBng46AyCQUZGwjiTU6qKYSbb2/2 l97gXnRDxXCYqavU1u3WK4VV/AZNHg+kfGxqv+J/UnfAaUwh8rNebHujYO2WRXls2iYl 114hPLOOs7H28p6lsLWjbBNwf/V4UByCs1gZIi34GD0rfCtk65MN8dfHZ9GgAIxOrOQH YbevGLfZNHXwh7G256hpoyMmLjxRzr5zQFbAA9OavbQroFIMR8Pkr058C9HRLlW6NzjQ /nFSkSW+J/amLXkw+oMe6Yg6XDfeUU1zumGj32R0BB4W1kdlAqeDwSPsfKYeVBCwIPO+ Jw== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by aserp2120.oracle.com with ESMTP id 30s0tm7dn1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 04 May 2020 15:00:15 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 044EmVae105750; Mon, 4 May 2020 15:00:14 GMT Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72]) by userp3020.oracle.com with ESMTP id 30sjjw0hkh-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 04 May 2020 15:00:14 +0000 Received: from abhmp0001.oracle.com (abhmp0001.oracle.com [141.146.116.7]) by userv0121.oracle.com (8.14.4/8.13.8) with ESMTP id 044F0AuU027632; Mon, 4 May 2020 15:00:10 GMT Received: from linux-1.home.com (/10.175.9.166) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 04 May 2020 08:00:09 -0700 From: Alexandre Chartre To: tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: pbonzini@redhat.com, konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, junaids@google.com, graf@amazon.de, rppt@linux.vnet.ibm.com, kuzuno@gmail.com, mgross@linux.intel.com, alexandre.chartre@oracle.com Subject: [RFC v4][PATCH part-2 09/13] mm/dpt: Functions to clear decorated page-table entries for a VA range Date: Mon, 4 May 2020 16:58:06 +0200 Message-Id: <20200504145810.11882-10-alexandre.chartre@oracle.com> X-Mailer: git-send-email 2.18.2 In-Reply-To: <20200504145810.11882-1-alexandre.chartre@oracle.com> References: <20200504145810.11882-1-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9610 signatures=668687 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxscore=0 phishscore=0 bulkscore=0 malwarescore=0 spamscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2005040123 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9610 signatures=668687 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 malwarescore=0 mlxscore=0 priorityscore=1501 lowpriorityscore=0 spamscore=0 suspectscore=0 phishscore=0 clxscore=1015 bulkscore=0 mlxlogscore=999 adultscore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2005040123 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Provide functions to clear page-table entries in a decorated page-table for a specified VA range. Functions also check that the clearing effectively happens in the decorated page-table and there is no crossing of the decorated page-table boundary (through references to another page table), so that another page table is not modified by mistake. As information (address, size, page-table level) about VA ranges mapped to the decorated page-table is tracked, clearing is done with just specifying the start address of the range. Signed-off-by: Alexandre Chartre --- arch/x86/include/asm/dpt.h | 1 + arch/x86/mm/dpt.c | 135 +++++++++++++++++++++++++++++++++++++ 2 files changed, 136 insertions(+) diff --git a/arch/x86/include/asm/dpt.h b/arch/x86/include/asm/dpt.h index 0d74afb10141..01727ef0577e 100644 --- a/arch/x86/include/asm/dpt.h +++ b/arch/x86/include/asm/dpt.h @@ -56,6 +56,7 @@ extern void dpt_destroy(struct dpt *dpt); extern int dpt_map_range(struct dpt *dpt, void *ptr, size_t size, enum page_table_level level); extern int dpt_map(struct dpt *dpt, void *ptr, unsigned long size); +extern void dpt_unmap(struct dpt *dpt, void *ptr); static inline int dpt_map_module(struct dpt *dpt, char *module_name) { diff --git a/arch/x86/mm/dpt.c b/arch/x86/mm/dpt.c index 12eb0d794d84..c495c9b59b3e 100644 --- a/arch/x86/mm/dpt.c +++ b/arch/x86/mm/dpt.c @@ -636,6 +636,141 @@ int dpt_map(struct dpt *dpt, void *ptr, unsigned long size) } EXPORT_SYMBOL(dpt_map); +static void dpt_clear_pte_range(struct dpt *dpt, pmd_t *pmd, + unsigned long addr, unsigned long end) +{ + pte_t *pte; + + pte = dpt_pte_offset(dpt, pmd, addr); + if (IS_ERR(pte)) + return; + + do { + pte_clear(NULL, addr, pte); + } while (pte++, addr += PAGE_SIZE, addr < end); +} + +static void dpt_clear_pmd_range(struct dpt *dpt, pud_t *pud, + unsigned long addr, unsigned long end, + enum page_table_level level) +{ + unsigned long next; + pmd_t *pmd; + + pmd = dpt_pmd_offset(dpt, pud, addr); + if (IS_ERR(pmd)) + return; + + do { + next = pmd_addr_end(addr, end); + if (pmd_none(*pmd)) + continue; + if (level == PGT_LEVEL_PMD || pmd_trans_huge(*pmd) || + pmd_devmap(*pmd) || !pmd_present(*pmd)) { + pmd_clear(pmd); + continue; + } + dpt_clear_pte_range(dpt, pmd, addr, next); + } while (pmd++, addr = next, addr < end); +} + +static void dpt_clear_pud_range(struct dpt *dpt, p4d_t *p4d, + unsigned long addr, unsigned long end, + enum page_table_level level) +{ + unsigned long next; + pud_t *pud; + + pud = dpt_pud_offset(dpt, p4d, addr); + if (IS_ERR(pud)) + return; + + do { + next = pud_addr_end(addr, end); + if (pud_none(*pud)) + continue; + if (level == PGT_LEVEL_PUD || pud_trans_huge(*pud) || + pud_devmap(*pud)) { + pud_clear(pud); + continue; + } + dpt_clear_pmd_range(dpt, pud, addr, next, level); + } while (pud++, addr = next, addr < end); +} + +static void dpt_clear_p4d_range(struct dpt *dpt, pgd_t *pgd, + unsigned long addr, unsigned long end, + enum page_table_level level) +{ + unsigned long next; + p4d_t *p4d; + + p4d = dpt_p4d_offset(dpt, pgd, addr); + if (IS_ERR(p4d)) + return; + + do { + next = p4d_addr_end(addr, end); + if (p4d_none(*p4d)) + continue; + if (level == PGT_LEVEL_P4D) { + p4d_clear(p4d); + continue; + } + dpt_clear_pud_range(dpt, p4d, addr, next, level); + } while (p4d++, addr = next, addr < end); +} + +static void dpt_clear_pgd_range(struct dpt *dpt, pgd_t *pagetable, + unsigned long addr, unsigned long end, + enum page_table_level level) +{ + unsigned long next; + pgd_t *pgd; + + pgd = pgd_offset_pgd(pagetable, addr); + do { + next = pgd_addr_end(addr, end); + if (pgd_none(*pgd)) + continue; + if (level == PGT_LEVEL_PGD) { + pgd_clear(pgd); + continue; + } + dpt_clear_p4d_range(dpt, pgd, addr, next, level); + } while (pgd++, addr = next, addr < end); +} + +/* + * Clear page table entries in the specified decorated page-table. + */ +void dpt_unmap(struct dpt *dpt, void *ptr) +{ + struct dpt_range_mapping *range_mapping; + unsigned long addr, end; + unsigned long flags; + + spin_lock_irqsave(&dpt->lock, flags); + + range_mapping = dpt_get_range_mapping(dpt, ptr); + if (!range_mapping) { + pr_debug("DPT %p: UNMAP %px - not mapped\n", dpt, ptr); + goto done; + } + + addr = (unsigned long)range_mapping->ptr; + end = addr + range_mapping->size; + pr_debug("DPT %p: UNMAP %px/%lx/%d\n", dpt, ptr, + range_mapping->size, range_mapping->level); + dpt_clear_pgd_range(dpt, dpt->pagetable, addr, end, + range_mapping->level); + list_del(&range_mapping->list); + kfree(range_mapping); +done: + spin_unlock_irqrestore(&dpt->lock, flags); +} +EXPORT_SYMBOL(dpt_unmap); + /* * dpt_create - allocate a page-table and create a corresponding * decorated page-table. The page-table is allocated and aligned From patchwork Mon May 4 14:58:07 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 11526473 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D204B15AB for ; Mon, 4 May 2020 15:00:33 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9E83420721 for ; Mon, 4 May 2020 15:00:33 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="aFcw6hwQ" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9E83420721 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 915CF8E0033; Mon, 4 May 2020 11:00:31 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 8C5E88E0024; Mon, 4 May 2020 11:00:31 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7DC228E0033; Mon, 4 May 2020 11:00:31 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0020.hostedemail.com [216.40.44.20]) by kanga.kvack.org (Postfix) with ESMTP id 5FC448E0024 for ; Mon, 4 May 2020 11:00:31 -0400 (EDT) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 852A5181AC9CC for ; Mon, 4 May 2020 15:00:29 +0000 (UTC) X-FDA: 76779347778.28.eggs64_3e6d2be57f852 X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,alexandre.chartre@oracle.com,,RULES_HIT:30054:30064:30079,0,RBL:156.151.31.86:@oracle.com:.lbl8.mailshell.net-62.18.0.100 64.10.201.10,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:ft,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: eggs64_3e6d2be57f852 X-Filterd-Recvd-Size: 5992 Received: from userp2130.oracle.com (userp2130.oracle.com [156.151.31.86]) by imf34.hostedemail.com (Postfix) with ESMTP for ; Mon, 4 May 2020 15:00:28 +0000 (UTC) Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 044ElbMF116466; Mon, 4 May 2020 15:00:18 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2020-01-29; bh=uOEDtCE2eXREwp6VCLNufdVVxnWRr/1oL3nfszUxecU=; b=aFcw6hwQWBLe+EOVzoXsQrpKcsrLqOmJquMpCEwhVJUYEP30D5313pso+GeXEV0oMuzW dnqfSWgce2f4NBzUZ7P2K0hTB98uhXHu95SBNZSkmtxxK3O44DbOCkqMGzZJRZNxLaaW PpLbbHRRj9aNvCryMJgQ05quBQzNsTIjgPW0W/CiAepd5axRLlnSM8HTQrfXvbS0ICCl ktce+PsNiZg+74GBgkzV7xEoxPFfcCI8uj+v7dorYKwUb7LNIG3sNdUbeSg8b9PBNMI1 SlXwUqRb2slMxIPXEkaDoaU73Ls9rPVtwZhVTczhqz348w3eYe+f0s+pWlfh6Vu0h8Ia Rw== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by userp2130.oracle.com with ESMTP id 30s09qyfug-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 04 May 2020 15:00:16 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 044EmVQJ105634; Mon, 4 May 2020 15:00:16 GMT Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72]) by userp3020.oracle.com with ESMTP id 30sjjw0hv7-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 04 May 2020 15:00:16 +0000 Received: from abhmp0001.oracle.com (abhmp0001.oracle.com [141.146.116.7]) by userv0121.oracle.com (8.14.4/8.13.8) with ESMTP id 044F0EQJ027649; Mon, 4 May 2020 15:00:14 GMT Received: from linux-1.home.com (/10.175.9.166) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 04 May 2020 08:00:13 -0700 From: Alexandre Chartre To: tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: pbonzini@redhat.com, konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, junaids@google.com, graf@amazon.de, rppt@linux.vnet.ibm.com, kuzuno@gmail.com, mgross@linux.intel.com, alexandre.chartre@oracle.com Subject: [RFC v4][PATCH part-2 10/13] mm/dpt: Function to copy page-table entries for percpu buffer Date: Mon, 4 May 2020 16:58:07 +0200 Message-Id: <20200504145810.11882-11-alexandre.chartre@oracle.com> X-Mailer: git-send-email 2.18.2 In-Reply-To: <20200504145810.11882-1-alexandre.chartre@oracle.com> References: <20200504145810.11882-1-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9610 signatures=668687 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxscore=0 phishscore=0 bulkscore=0 malwarescore=0 spamscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2005040123 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9610 signatures=668687 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 mlxscore=0 lowpriorityscore=0 spamscore=0 adultscore=0 clxscore=1015 suspectscore=0 priorityscore=1501 malwarescore=0 mlxlogscore=999 phishscore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2005040123 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Provide functions to copy page-table entries from the kernel page-table to a decorated page-table for a percpu buffer. A percpu buffer have a different VA range for each cpu and all them have to be copied. Signed-off-by: Alexandre Chartre --- arch/x86/include/asm/dpt.h | 6 ++++++ arch/x86/mm/dpt.c | 38 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 44 insertions(+) diff --git a/arch/x86/include/asm/dpt.h b/arch/x86/include/asm/dpt.h index 01727ef0577e..fd8c1b84ffe2 100644 --- a/arch/x86/include/asm/dpt.h +++ b/arch/x86/include/asm/dpt.h @@ -78,4 +78,10 @@ static inline int dpt_map_module(struct dpt *dpt, char *module_name) (dpt_map(dpt, THIS_MODULE->core_layout.base, \ THIS_MODULE->core_layout.size)) +extern int dpt_map_percpu(struct dpt *dpt, void *percpu_ptr, size_t size); +extern void dpt_unmap_percpu(struct dpt *dpt, void *percpu_ptr); + +#define DPT_MAP_CPUVAR(dpt, cpuvar) \ + dpt_map_percpu(dpt, &cpuvar, sizeof(cpuvar)) + #endif diff --git a/arch/x86/mm/dpt.c b/arch/x86/mm/dpt.c index c495c9b59b3e..adc59f9ed876 100644 --- a/arch/x86/mm/dpt.c +++ b/arch/x86/mm/dpt.c @@ -771,6 +771,44 @@ void dpt_unmap(struct dpt *dpt, void *ptr) } EXPORT_SYMBOL(dpt_unmap); +void dpt_unmap_percpu(struct dpt *dpt, void *percpu_ptr) +{ + void *ptr; + int cpu; + + pr_debug("DPT %p: UNMAP PERCPU %px\n", dpt, percpu_ptr); + for_each_possible_cpu(cpu) { + ptr = per_cpu_ptr(percpu_ptr, cpu); + pr_debug("DPT %p: UNMAP PERCPU%d %px\n", dpt, cpu, ptr); + dpt_unmap(dpt, ptr); + } +} +EXPORT_SYMBOL(dpt_unmap_percpu); + +int dpt_map_percpu(struct dpt *dpt, void *percpu_ptr, size_t size) +{ + int cpu, err; + void *ptr; + + pr_debug("DPT %p: MAP PERCPU %px\n", dpt, percpu_ptr); + for_each_possible_cpu(cpu) { + ptr = per_cpu_ptr(percpu_ptr, cpu); + pr_debug("DPT %p: MAP PERCPU%d %px\n", dpt, cpu, ptr); + err = dpt_map(dpt, ptr, size); + if (err) { + /* + * Need to unmap any percpu mapping which has + * succeeded before the failure. + */ + dpt_unmap_percpu(dpt, percpu_ptr); + return err; + } + } + + return 0; +} +EXPORT_SYMBOL(dpt_map_percpu); + /* * dpt_create - allocate a page-table and create a corresponding * decorated page-table. The page-table is allocated and aligned From patchwork Mon May 4 14:58:08 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 11526481 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 678C215E6 for ; Mon, 4 May 2020 15:02:40 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 33A4D206D7 for ; Mon, 4 May 2020 15:02:40 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="cozJiluR" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 33A4D206D7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 62F9D8E0038; Mon, 4 May 2020 11:02:38 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 591208E0024; Mon, 4 May 2020 11:02:38 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 431528E0038; Mon, 4 May 2020 11:02:38 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0062.hostedemail.com [216.40.44.62]) by kanga.kvack.org (Postfix) with ESMTP id 26BCE8E0024 for ; Mon, 4 May 2020 11:02:38 -0400 (EDT) Received: from smtpin21.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id F040D1A4B9 for ; Mon, 4 May 2020 15:02:37 +0000 (UTC) X-FDA: 76779353154.21.shake12_511e181c05457 X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,alexandre.chartre@oracle.com,,RULES_HIT:30003:30054:30064:30070,0,RBL:156.151.31.86:@oracle.com:.lbl8.mailshell.net-62.18.0.100 64.10.201.10,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:ft,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:0,LUA_SUMMARY:none X-HE-Tag: shake12_511e181c05457 X-Filterd-Recvd-Size: 5574 Received: from userp2130.oracle.com (userp2130.oracle.com [156.151.31.86]) by imf40.hostedemail.com (Postfix) with ESMTP for ; Mon, 4 May 2020 15:02:37 +0000 (UTC) Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 044Elbmn116463; Mon, 4 May 2020 15:02:23 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2020-01-29; bh=Znxthj2nB/nosH+Hjc7m0rLCqlY1NvIB6ejFhn9Kd3E=; b=cozJiluRmq0O+l+y6gIqT1Prl0V56vFPgnUWM/9RBWpnAn3EMhRYfGrZ97Pa8nAseCif PtuJLdF2NDXMVxVSximHqbqY2YSjKkK9H5YmlpbBxhnjO9ZfzPH2mo2qIj9dQrx2nRqL tRbzgsIRubuKpk3sj0ukXWkU+bmkQwBAkv9/P8uzgAjjnT9F8uEHP+kwWHcnwdjoqhcV O78cxxvjUQHguahyjUzPdGHfBSRLb94EIKu+B2TDZxt8lDFae1/93Ev2q/KanGYSg2D9 XdVtz42LC8PFmnAv0EcHN6ouC1rf3EOzInCRsvjEu1/SH3RfCbGYIrYVOiS8vINSI9vs +Q== Received: from userp3030.oracle.com (userp3030.oracle.com [156.151.31.80]) by userp2130.oracle.com with ESMTP id 30s09qygch-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 04 May 2020 15:02:23 +0000 Received: from pps.filterd (userp3030.oracle.com [127.0.0.1]) by userp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 044ElaEu053343; Mon, 4 May 2020 15:00:23 GMT Received: from aserv0121.oracle.com (aserv0121.oracle.com [141.146.126.235]) by userp3030.oracle.com with ESMTP id 30t1r2eu6m-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 04 May 2020 15:00:23 +0000 Received: from abhmp0001.oracle.com (abhmp0001.oracle.com [141.146.116.7]) by aserv0121.oracle.com (8.14.4/8.13.8) with ESMTP id 044F0ImZ026075; Mon, 4 May 2020 15:00:18 GMT Received: from linux-1.home.com (/10.175.9.166) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 04 May 2020 08:00:17 -0700 From: Alexandre Chartre To: tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: pbonzini@redhat.com, konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, junaids@google.com, graf@amazon.de, rppt@linux.vnet.ibm.com, kuzuno@gmail.com, mgross@linux.intel.com, alexandre.chartre@oracle.com Subject: [RFC v4][PATCH part-2 11/13] mm/dpt: Add decorated page-table remap function Date: Mon, 4 May 2020 16:58:08 +0200 Message-Id: <20200504145810.11882-12-alexandre.chartre@oracle.com> X-Mailer: git-send-email 2.18.2 In-Reply-To: <20200504145810.11882-1-alexandre.chartre@oracle.com> References: <20200504145810.11882-1-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9610 signatures=668687 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 adultscore=0 suspectscore=0 spamscore=0 mlxlogscore=999 malwarescore=0 phishscore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2005040123 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9610 signatures=668687 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 mlxscore=0 lowpriorityscore=0 spamscore=0 adultscore=0 clxscore=1015 suspectscore=0 priorityscore=1501 malwarescore=0 mlxlogscore=999 phishscore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2005040123 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add a function to remap an already mapped buffer with a new address in a decorated page-table: the already mapped buffer is unmapped, and a new mapping is added for the specified new address. This is useful to track and remap a buffer which can be freed and then reallocated. Signed-off-by: Alexandre Chartre --- arch/x86/include/asm/dpt.h | 1 + arch/x86/mm/dpt.c | 25 +++++++++++++++++++++++++ 2 files changed, 26 insertions(+) diff --git a/arch/x86/include/asm/dpt.h b/arch/x86/include/asm/dpt.h index fd8c1b84ffe2..3234ba968d80 100644 --- a/arch/x86/include/asm/dpt.h +++ b/arch/x86/include/asm/dpt.h @@ -57,6 +57,7 @@ extern int dpt_map_range(struct dpt *dpt, void *ptr, size_t size, enum page_table_level level); extern int dpt_map(struct dpt *dpt, void *ptr, unsigned long size); extern void dpt_unmap(struct dpt *dpt, void *ptr); +extern int dpt_remap(struct dpt *dpt, void **mapping, void *ptr, size_t size); static inline int dpt_map_module(struct dpt *dpt, char *module_name) { diff --git a/arch/x86/mm/dpt.c b/arch/x86/mm/dpt.c index adc59f9ed876..9517e3081716 100644 --- a/arch/x86/mm/dpt.c +++ b/arch/x86/mm/dpt.c @@ -809,6 +809,31 @@ int dpt_map_percpu(struct dpt *dpt, void *percpu_ptr, size_t size) } EXPORT_SYMBOL(dpt_map_percpu); +int dpt_remap(struct dpt *dpt, void **current_ptrp, void *new_ptr, size_t size) +{ + void *current_ptr = *current_ptrp; + int err; + + if (current_ptr == new_ptr) { + /* no change, already mapped */ + return 0; + } + + if (current_ptr) { + dpt_unmap(dpt, current_ptr); + *current_ptrp = NULL; + } + + err = dpt_map(dpt, new_ptr, size); + if (err) + return err; + + *current_ptrp = new_ptr; + + return 0; +} +EXPORT_SYMBOL(dpt_remap); + /* * dpt_create - allocate a page-table and create a corresponding * decorated page-table. The page-table is allocated and aligned From patchwork Mon May 4 14:58:09 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 11526479 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2809A14C0 for ; Mon, 4 May 2020 15:02:38 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id CD753206D7 for ; Mon, 4 May 2020 15:02:37 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="LlwksrUj" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CD753206D7 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id E43A78E0037; Mon, 4 May 2020 11:02:36 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id DF5928E0024; Mon, 4 May 2020 11:02:36 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CE4078E0037; Mon, 4 May 2020 11:02:36 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0047.hostedemail.com [216.40.44.47]) by kanga.kvack.org (Postfix) with ESMTP id B64E08E0024 for ; Mon, 4 May 2020 11:02:36 -0400 (EDT) Received: from smtpin02.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 6528919B06 for ; Mon, 4 May 2020 15:02:36 +0000 (UTC) X-FDA: 76779353112.02.week47_50e9e275a4129 X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,alexandre.chartre@oracle.com,,RULES_HIT:6226:30001:30003:30034:30054:30064:30070:30079,0,RBL:156.151.31.85:@oracle.com:.lbl8.mailshell.net-64.10.201.10 62.18.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:ft,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: week47_50e9e275a4129 X-Filterd-Recvd-Size: 12339 Received: from userp2120.oracle.com (userp2120.oracle.com [156.151.31.85]) by imf14.hostedemail.com (Postfix) with ESMTP for ; Mon, 4 May 2020 15:02:35 +0000 (UTC) Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 044EliEO024627; Mon, 4 May 2020 15:02:25 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2020-01-29; bh=chwC6Qoj6vU2BtpkAhEtUbdF5FP79ZRqRWpFJfRAy6o=; b=LlwksrUjmqVcn1DFyrllRb6xoNpjN33QJHik5LcDtjxf0E/guw3MrcmImj7l4qRhGxaw xvTUH17OCFjpRI9t+m3PSuUbha++t6V6NcgrvCTDdE59OxGtJkRy2FxBBAalPM9uCzPm KBO3QcRo1GryhcX094C4/nyows7PelusYzeNgUTj2DL53S4l1SG8OigpdNCmRmQnRAYi SfXfQhmqKd8oe4tcbPD4l8RK07MRO5R0LBnDuxAnqd976MHFZD9E1/kU/sNA5SbIaX5/ ATCiadnJQSYJCOAXVm8p/UC4qAJrHfjt/tjUACg7qz/TK/4K0VfYPXE1eOmzV+XrNr4S 3Q== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by userp2120.oracle.com with ESMTP id 30s1gmy945-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 04 May 2020 15:02:25 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 044EmVhV105651; Mon, 4 May 2020 15:00:25 GMT Received: from userv0121.oracle.com (userv0121.oracle.com [156.151.31.72]) by userp3020.oracle.com with ESMTP id 30sjjw0jje-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 04 May 2020 15:00:25 +0000 Received: from abhmp0001.oracle.com (abhmp0001.oracle.com [141.146.116.7]) by userv0121.oracle.com (8.14.4/8.13.8) with ESMTP id 044F0NuL027832; Mon, 4 May 2020 15:00:23 GMT Received: from linux-1.home.com (/10.175.9.166) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 04 May 2020 08:00:22 -0700 From: Alexandre Chartre To: tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: pbonzini@redhat.com, konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, junaids@google.com, graf@amazon.de, rppt@linux.vnet.ibm.com, kuzuno@gmail.com, mgross@linux.intel.com, alexandre.chartre@oracle.com Subject: [RFC v4][PATCH part-2 12/13] mm/dpt: Handle decorated page-table mapped range leaks and overlaps Date: Mon, 4 May 2020 16:58:09 +0200 Message-Id: <20200504145810.11882-13-alexandre.chartre@oracle.com> X-Mailer: git-send-email 2.18.2 In-Reply-To: <20200504145810.11882-1-alexandre.chartre@oracle.com> References: <20200504145810.11882-1-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9610 signatures=668687 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 mlxscore=0 phishscore=0 bulkscore=0 malwarescore=0 spamscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2005040123 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9610 signatures=668687 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 suspectscore=0 mlxscore=0 spamscore=0 clxscore=1015 priorityscore=1501 bulkscore=0 phishscore=0 impostorscore=0 malwarescore=0 lowpriorityscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2005040123 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When mapping a buffer in a decorated page-table, data around the buffer can also be mapped if the entire buffer is not aligned with the page directory size used for the mapping. So, data can potentially leak into the decorated page-table. In such a case, print a warning that data are leaking. Also data effectively mapped can overlap with an already mapped buffer. This is not an issue when mapping data but, when unmapping, make sure data from another buffer don't get unmapped as a side effect. Signed-off-by: Alexandre Chartre --- arch/x86/include/asm/dpt.h | 1 + arch/x86/mm/dpt.c | 197 +++++++++++++++++++++++++++++++------ 2 files changed, 168 insertions(+), 30 deletions(-) diff --git a/arch/x86/include/asm/dpt.h b/arch/x86/include/asm/dpt.h index 3234ba968d80..e0adbf69dadf 100644 --- a/arch/x86/include/asm/dpt.h +++ b/arch/x86/include/asm/dpt.h @@ -25,6 +25,7 @@ struct dpt_range_mapping { void *ptr; /* range start address */ size_t size; /* range size */ enum page_table_level level; /* mapping level */ + int refcnt; /* reference count (for overlap) */ }; /* diff --git a/arch/x86/mm/dpt.c b/arch/x86/mm/dpt.c index 9517e3081716..d3d3c3de2943 100644 --- a/arch/x86/mm/dpt.c +++ b/arch/x86/mm/dpt.c @@ -9,6 +9,22 @@ #include + +static unsigned long page_directory_size[] = { + [PGT_LEVEL_PTE] = PAGE_SIZE, + [PGT_LEVEL_PMD] = PMD_SIZE, + [PGT_LEVEL_PUD] = PUD_SIZE, + [PGT_LEVEL_P4D] = P4D_SIZE, + [PGT_LEVEL_PGD] = PGDIR_SIZE, +}; + +#define DPT_RANGE_MAP_ADDR(r) \ + round_down((unsigned long)((r)->ptr), page_directory_size[(r)->level]) + +#define DPT_RANGE_MAP_END(r) \ + round_up((unsigned long)((r)->ptr + (r)->size), \ + page_directory_size[(r)->level]) + /* * Get the pointer to the beginning of a page table directory from a page * table directory entry. @@ -572,6 +588,70 @@ static int dpt_copy_pgd_range(struct dpt *dpt, return 0; } +/* + * Map a VA range, taking into account any overlap with already mapped + * VA ranges. On error, return < 0. Otherwise return the number of + * ranges the specified range is overlapping with. + */ +static int dpt_map_overlap(struct dpt *dpt, void *ptr, size_t size, + enum page_table_level level) +{ + unsigned long map_addr, map_end; + unsigned long addr, end; + struct dpt_range_mapping *range; + bool need_mapping; + int err, overlap; + + addr = (unsigned long)ptr; + end = addr + (unsigned long)size; + need_mapping = true; + overlap = 0; + + lockdep_assert_held(&dpt->lock); + list_for_each_entry(range, &dpt->mapping_list, list) { + + if (range->ptr == ptr && range->size == size) { + /* we are mapping the same range again */ + pr_debug("DPT %p: MAP %px/%lx/%d already mapped\n", + dpt, ptr, size, level); + return -EBUSY; + } + + /* check overlap with mapped range */ + map_addr = DPT_RANGE_MAP_ADDR(range); + map_end = DPT_RANGE_MAP_END(range); + if (end <= map_addr || addr >= map_end) { + /* no overlap, continue */ + continue; + } + + pr_debug("DPT %p: MAP %px/%lx/%d overlaps with %px/%lx/%d\n", + dpt, ptr, size, level, + range->ptr, range->size, range->level); + range->refcnt++; + overlap++; + + /* + * Check if new range is included into an existing range. + * If so then the new range is already entirely mapped. + */ + if (addr >= map_addr && end <= map_end) { + pr_debug("DPT %p: MAP %px/%lx/%d implicitly mapped\n", + dpt, ptr, size, level); + need_mapping = false; + } + } + + if (need_mapping) { + err = dpt_copy_pgd_range(dpt, dpt->pagetable, current->mm->pgd, + addr, end, level); + if (err) + return err; + } + + return overlap; +} + /* * Copy page table entries from the current page table (i.e. from the * kernel page table) to the specified decorated page-table. The level @@ -582,47 +662,48 @@ int dpt_map_range(struct dpt *dpt, void *ptr, size_t size, enum page_table_level level) { struct dpt_range_mapping *range_mapping; + unsigned long page_dir_size = page_directory_size[level]; unsigned long addr = (unsigned long)ptr; unsigned long end = addr + ((unsigned long)size); + unsigned long map_addr, map_end; unsigned long flags; - int err; + int overlap; - pr_debug("DPT %p: MAP %px/%lx/%d\n", dpt, ptr, size, level); + map_addr = round_down(addr, page_dir_size); + map_end = round_up(end, page_dir_size); - spin_lock_irqsave(&dpt->lock, flags); - - /* check if the range is already mapped */ - range_mapping = dpt_get_range_mapping(dpt, ptr); - if (range_mapping) { - pr_debug("DPT %p: MAP %px/%lx/%d already mapped\n", - dpt, ptr, size, level); - err = -EBUSY; - goto done; - } + pr_debug("DPT %p: MAP %px/%lx/%d -> %lx-%lx\n", dpt, ptr, size, level, + map_addr, map_end); + if (map_addr < addr) + pr_debug("DPT %p: MAP LEAK %lx-%lx\n", dpt, map_addr, addr); + if (map_end > end) + pr_debug("DPT %p: MAP LEAK %lx-%lx\n", dpt, end, map_end); - /* map new range */ + /* add new range */ range_mapping = kmalloc(sizeof(*range_mapping), GFP_KERNEL); - if (!range_mapping) { - err = -ENOMEM; - goto done; - } + if (!range_mapping) + return -ENOMEM; - err = dpt_copy_pgd_range(dpt, dpt->pagetable, current->mm->pgd, - addr, end, level); - if (err) { - kfree(range_mapping); - goto done; + spin_lock_irqsave(&dpt->lock, flags); + + /* + * Map the new range with taking overlap with already mapped ranges + * into account. + */ + overlap = dpt_map_overlap(dpt, ptr, size, level); + if (overlap < 0) { + spin_unlock_irqrestore(&dpt->lock, flags); + return overlap; } INIT_LIST_HEAD(&range_mapping->list); range_mapping->ptr = ptr; range_mapping->size = size; range_mapping->level = level; + range_mapping->refcnt = overlap + 1; list_add(&range_mapping->list, &dpt->mapping_list); -done: spin_unlock_irqrestore(&dpt->lock, flags); - - return err; + return 0; } EXPORT_SYMBOL(dpt_map_range); @@ -741,13 +822,72 @@ static void dpt_clear_pgd_range(struct dpt *dpt, pgd_t *pagetable, } while (pgd++, addr = next, addr < end); } + +/* + * Unmap a VA range, taking into account any overlap with other mapped + * VA ranges. + */ +static void dpt_unmap_overlap(struct dpt *dpt, struct dpt_range_mapping *range) +{ + unsigned long pgdir_size = page_directory_size[range->level]; + unsigned long chunk_addr, chunk_end; + unsigned long map_addr, map_end; + struct dpt_range_mapping *r; + unsigned long addr, end; + bool overlap; + + addr = DPT_RANGE_MAP_ADDR(range); + end = DPT_RANGE_MAP_END(range); + + lockdep_assert_held(&dpt->lock); + + /* + * Unmap the VA range by chunk to handle mapping overlap + * with any another range. + * XXX can be improved with a sorted range list + */ + chunk_addr = addr; + while (chunk_addr < end) { + overlap = false; + list_for_each_entry(r, &dpt->mapping_list, list) { + map_addr = DPT_RANGE_MAP_ADDR(r); + map_end = DPT_RANGE_MAP_END(r); + /* + * Check if there's an overlap and how far it goes. + */ + chunk_end = chunk_addr; + while (chunk_end >= map_addr && chunk_end < map_end) { + overlap = true; + chunk_end += pgdir_size; + if (chunk_end >= end) + break; + } + if (overlap) { + pr_debug("DPT %p: UNMAP %px/%lx/%d overlaps with %px/%lx/%d\n", + dpt, range->ptr, range->size, + range->level, + r->ptr, r->size, r->level); + break; + } + } + + if (!overlap) { + pr_debug("DPT %p: UNMAP CHUNK %lx/%lx/%d\n", dpt, + chunk_addr, pgdir_size, range->level); + chunk_end = chunk_addr + pgdir_size; + dpt_clear_pgd_range(dpt, dpt->pagetable, chunk_addr, + chunk_end, range->level); + } + chunk_addr = chunk_end; + } +} + /* * Clear page table entries in the specified decorated page-table. */ void dpt_unmap(struct dpt *dpt, void *ptr) { struct dpt_range_mapping *range_mapping; - unsigned long addr, end; unsigned long flags; spin_lock_irqsave(&dpt->lock, flags); @@ -758,13 +898,10 @@ void dpt_unmap(struct dpt *dpt, void *ptr) goto done; } - addr = (unsigned long)range_mapping->ptr; - end = addr + range_mapping->size; pr_debug("DPT %p: UNMAP %px/%lx/%d\n", dpt, ptr, range_mapping->size, range_mapping->level); - dpt_clear_pgd_range(dpt, dpt->pagetable, addr, end, - range_mapping->level); list_del(&range_mapping->list); + dpt_unmap_overlap(dpt, range_mapping); kfree(range_mapping); done: spin_unlock_irqrestore(&dpt->lock, flags); From patchwork Mon May 4 14:58:10 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Chartre X-Patchwork-Id: 11526475 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AADDD14C0 for ; Mon, 4 May 2020 15:00:40 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 69C9B20721 for ; Mon, 4 May 2020 15:00:40 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="iOX7z1yW" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 69C9B20721 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 3BFF38E0034; Mon, 4 May 2020 11:00:39 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 3956E8E0024; Mon, 4 May 2020 11:00:39 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2ABE28E0034; Mon, 4 May 2020 11:00:39 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0184.hostedemail.com [216.40.44.184]) by kanga.kvack.org (Postfix) with ESMTP id 128F08E0024 for ; Mon, 4 May 2020 11:00:39 -0400 (EDT) Received: from smtpin07.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id C9368824559C for ; Mon, 4 May 2020 15:00:38 +0000 (UTC) X-FDA: 76779348156.07.hour82_3fc9ed741300b X-Spam-Summary: 1,0,0,,d41d8cd98f00b204,alexandre.chartre@oracle.com,,RULES_HIT:30003:30012:30051:30054:30064:30070:30075,0,RBL:156.151.31.86:@oracle.com:.lbl8.mailshell.net-62.18.0.100 64.10.201.10,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:ft,MSBL:0,DNSBL:neutral,Custom_rules:0:1:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: hour82_3fc9ed741300b X-Filterd-Recvd-Size: 6803 Received: from userp2130.oracle.com (userp2130.oracle.com [156.151.31.86]) by imf40.hostedemail.com (Postfix) with ESMTP for ; Mon, 4 May 2020 15:00:38 +0000 (UTC) Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 044ElbG6116513; Mon, 4 May 2020 15:00:29 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references; s=corp-2020-01-29; bh=5ClwOk8hAM6M0zm7zYo3+G30LLjRZv1vfMafgBD0Iqw=; b=iOX7z1yWED8rz7gVbvpEo6SnpLCCF4ZeouDB99xr1IgkZ3XlsF/eK/xXi7ZGZ85VDgXS Sqvb1zzwJ+AKfjw42W+CeC616p7RVZvaA5RrEeX/l714yca5ICzhTIu6vhbZJF+yN9J+ rI4f3jqfKMCEZI2It1GBEPvkhminSBDaSVCuNJ0E89LbKVMd1VksNlNs7AxD08xXYNlC ugx6ChcQMdfgDuOMfXwWzs1aqmLmfRt+k5SxBhBUJQLx36BFNDibfSJa3VCzRGbnCHGO 4e57hhQGSWWWlA0oPBfFExmmUtEFuWxUfKzV2+Arxz4yJk4Z0Obpy+AQ6QBjW3NBa6M3 HQ== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by userp2130.oracle.com with ESMTP id 30s09qyfvv-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 04 May 2020 15:00:29 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 044ElLHX033714; Mon, 4 May 2020 15:00:28 GMT Received: from aserv0122.oracle.com (aserv0122.oracle.com [141.146.126.236]) by aserp3020.oracle.com with ESMTP id 30sjnaw42d-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 04 May 2020 15:00:28 +0000 Received: from abhmp0001.oracle.com (abhmp0001.oracle.com [141.146.116.7]) by aserv0122.oracle.com (8.14.4/8.14.4) with ESMTP id 044F0RUp001695; Mon, 4 May 2020 15:00:27 GMT Received: from linux-1.home.com (/10.175.9.166) by default (Oracle Beehive Gateway v4.0) with ESMTP ; Mon, 04 May 2020 08:00:26 -0700 From: Alexandre Chartre To: tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, hpa@zytor.com, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, x86@kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: pbonzini@redhat.com, konrad.wilk@oracle.com, jan.setjeeilers@oracle.com, liran.alon@oracle.com, junaids@google.com, graf@amazon.de, rppt@linux.vnet.ibm.com, kuzuno@gmail.com, mgross@linux.intel.com, alexandre.chartre@oracle.com Subject: [RFC v4][PATCH part-2 13/13] mm/asi: Function to init decorated page-table with ASI core mappings Date: Mon, 4 May 2020 16:58:10 +0200 Message-Id: <20200504145810.11882-14-alexandre.chartre@oracle.com> X-Mailer: git-send-email 2.18.2 In-Reply-To: <20200504145810.11882-1-alexandre.chartre@oracle.com> References: <20200504145810.11882-1-alexandre.chartre@oracle.com> X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9610 signatures=668687 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 adultscore=67 phishscore=0 mlxlogscore=999 bulkscore=0 malwarescore=0 spamscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2005040123 X-Proofpoint-Virus-Version: vendor=nai engine=6000 definitions=9610 signatures=668687 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 mlxscore=0 lowpriorityscore=0 spamscore=0 adultscore=58 clxscore=1015 suspectscore=0 priorityscore=1501 malwarescore=0 mlxlogscore=999 phishscore=0 impostorscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2005040123 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Core mappings are the minimal mappings we need to be able to enter isolation and handle an isolation abort or exit. This includes the kernel code, the GDT and the percpu ASI sessions. We also need a stack so we map the current stack when entering isolation and unmap it on exit/abort. Signed-off-by: Alexandre Chartre --- arch/x86/include/asm/asi.h | 2 ++ arch/x86/mm/asi.c | 57 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 59 insertions(+) diff --git a/arch/x86/include/asm/asi.h b/arch/x86/include/asm/asi.h index ac0594d4f549..eafed750e07f 100644 --- a/arch/x86/include/asm/asi.h +++ b/arch/x86/include/asm/asi.h @@ -45,6 +45,7 @@ #include #include +#include /* * ASI_NR_DYN_ASIDS is the same as TLB_NR_DYN_ASIDS. We can't directly @@ -150,6 +151,7 @@ extern void asi_destroy(struct asi *asi); extern void asi_set_pagetable(struct asi *asi, pgd_t *pagetable); extern int asi_enter(struct asi *asi); extern void asi_exit(struct asi *asi); +extern int asi_init_dpt(struct dpt *dpt); #ifdef CONFIG_PAGE_TABLE_ISOLATION DECLARE_ASI_TYPE(user); diff --git a/arch/x86/mm/asi.c b/arch/x86/mm/asi.c index b63a0a883293..8b670ed13729 100644 --- a/arch/x86/mm/asi.c +++ b/arch/x86/mm/asi.c @@ -162,6 +162,63 @@ void asi_set_pagetable(struct asi *asi, pgd_t *pagetable) } EXPORT_SYMBOL(asi_set_pagetable); +/* + * asi_init_dpt - Initialize a decorated page-table with the minimum + * mappings for using an ASI. Note that this function doesn't map any + * stack. If the stack of the task entering an ASI is not mapped then + * this will trigger a double-fault as soon as the task tries to access + * its stack. + */ +int asi_init_dpt(struct dpt *dpt) +{ + int err; + + /* + * Map the kernel. + * + * XXX We should check if we can map only kernel text, i.e. map with + * size = _etext - _text + */ + err = dpt_map(dpt, (void *)__START_KERNEL_map, KERNEL_IMAGE_SIZE); + if (err) + return err; + + /* + * Map the cpu_entry_area because we need the GDT to be mapped. + * Not sure we need anything else from cpu_entry_area. + */ + err = dpt_map_range(dpt, (void *)CPU_ENTRY_AREA_PER_CPU, P4D_SIZE, + PGT_LEVEL_P4D); + if (err) + return err; + + /* + * Map fixed_percpu_data to get the stack canary. + */ + if (IS_ENABLED(CONFIG_STACKPROTECTOR)) { + err = DPT_MAP_CPUVAR(dpt, fixed_percpu_data); + if (err) + return err; + } + + /* Map current_task, we need it for __schedule() */ + err = DPT_MAP_CPUVAR(dpt, current_task); + if (err) + return err; + + /* + * Map the percpu ASI tlbstate. This also maps the asi_session + * which is used by interrupt handlers to figure out if we have + * entered isolation and switch back to the kernel address space. + */ + err = DPT_MAP_CPUVAR(dpt, cpu_tlbstate); + if (err) + return err; + + return 0; +} +EXPORT_SYMBOL(asi_init_dpt); + /* * Update ASI TLB flush information for the specified ASI CR3 value. * Return an updated ASI CR3 value which specified if TLB needs to