From patchwork Wed Jan 13 09:41:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Janosch Frank X-Patchwork-Id: 12016297 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 93546C433E6 for ; Wed, 13 Jan 2021 09:42:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4B22F233F8 for ; Wed, 13 Jan 2021 09:42:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727389AbhAMJmY (ORCPT ); Wed, 13 Jan 2021 04:42:24 -0500 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:11380 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727345AbhAMJmW (ORCPT ); Wed, 13 Jan 2021 04:42:22 -0500 Received: from pps.filterd (m0187473.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 10D9XPpQ007838; Wed, 13 Jan 2021 04:41:41 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=OpoE8Y6bxTLR0nmKeCcXbkDvdT59x8h7/q2maMOCtC0=; b=IOFafx5V2oYaNd+lXH2BOJyekgSfMRDxAVB1VJ7R7qp5lp6xmm92Gp6yg+8bHDueXmuc iIEo9YzAdKhIfQbydwAtuLytiKR0bql4/OhM84Uvwn4lrr6M1WZez/x1HEdB3K+set1V V2bR0Nydb3t8ftUkwVU4D0NaW5ib+EHQPotu9Eaheu+GPMIJpPy6+capKz4yx/Ry5IiQ HdF+HPlPmmxJqID6WChWwnoZ5T6K/+K3IJEO4aZOVTeAeH9QNez7af2V5PldS7YF8/O0 s7iux0nyZpNGNIFY2JyZCj+mEt2zfuoiqm8JRxRWN9TNFFidEHnFzHmv35+tmVYrPmjM Sw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 361wdnsk1a-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 13 Jan 2021 04:41:41 -0500 Received: from m0187473.ppops.net (m0187473.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 10D9XRs1007869; Wed, 13 Jan 2021 04:41:40 -0500 Received: from ppma03ams.nl.ibm.com (62.31.33a9.ip4.static.sl-reverse.com [169.51.49.98]) by mx0a-001b2d01.pphosted.com with ESMTP id 361wdnsk0h-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 13 Jan 2021 04:41:40 -0500 Received: from pps.filterd (ppma03ams.nl.ibm.com [127.0.0.1]) by ppma03ams.nl.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 10D9QxQ9019232; Wed, 13 Jan 2021 09:41:38 GMT Received: from b06cxnps4076.portsmouth.uk.ibm.com (d06relay13.portsmouth.uk.ibm.com [9.149.109.198]) by ppma03ams.nl.ibm.com with ESMTP id 35y447vucb-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 13 Jan 2021 09:41:38 +0000 Received: from d06av23.portsmouth.uk.ibm.com (d06av23.portsmouth.uk.ibm.com [9.149.105.59]) by b06cxnps4076.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 10D9fZqB31654146 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 13 Jan 2021 09:41:35 GMT Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 8E924A4040; Wed, 13 Jan 2021 09:41:35 +0000 (GMT) Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 5AF0BA4053; Wed, 13 Jan 2021 09:41:35 +0000 (GMT) Received: from t46lp67.lnxne.boe (unknown [9.152.108.100]) by d06av23.portsmouth.uk.ibm.com (Postfix) with ESMTP; Wed, 13 Jan 2021 09:41:35 +0000 (GMT) From: Janosch Frank To: kvm@vger.kernel.org Cc: borntraeger@de.ibm.com, david@redhat.com, linux-s390@vger.kernel.org, imbrenda@linux.ibm.com Subject: [PATCH 01/14] s390/mm: Code cleanups Date: Wed, 13 Jan 2021 09:41:00 +0000 Message-Id: <20210113094113.133668-2-frankja@linux.ibm.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210113094113.133668-1-frankja@linux.ibm.com> References: <20210113094113.133668-1-frankja@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.343,18.0.737 definitions=2021-01-13_03:2021-01-13,2021-01-13 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 bulkscore=0 impostorscore=0 adultscore=0 malwarescore=0 phishscore=0 clxscore=1015 priorityscore=1501 spamscore=0 suspectscore=0 mlxlogscore=999 mlxscore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101130056 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Let's clean up leftovers before introducing new code. Signed-off-by: Janosch Frank Reviewed-by: David Hildenbrand Acked-by: Christian Borntraeger --- arch/s390/mm/gmap.c | 8 ++++---- arch/s390/mm/pgtable.c | 2 +- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/arch/s390/mm/gmap.c b/arch/s390/mm/gmap.c index 9bb2c7512cd5..f857104ca6c1 100644 --- a/arch/s390/mm/gmap.c +++ b/arch/s390/mm/gmap.c @@ -5,7 +5,7 @@ * Copyright IBM Corp. 2007, 2020 * Author(s): Martin Schwidefsky * David Hildenbrand - * Janosch Frank + * Janosch Frank */ #include @@ -2290,10 +2290,10 @@ static void gmap_pmdp_xchg(struct gmap *gmap, pmd_t *pmdp, pmd_t new, pmdp_notify_gmap(gmap, pmdp, gaddr); pmd_val(new) &= ~_SEGMENT_ENTRY_GMAP_IN; if (MACHINE_HAS_TLB_GUEST) - __pmdp_idte(gaddr, (pmd_t *)pmdp, IDTE_GUEST_ASCE, gmap->asce, + __pmdp_idte(gaddr, pmdp, IDTE_GUEST_ASCE, gmap->asce, IDTE_GLOBAL); else if (MACHINE_HAS_IDTE) - __pmdp_idte(gaddr, (pmd_t *)pmdp, 0, 0, IDTE_GLOBAL); + __pmdp_idte(gaddr, pmdp, 0, 0, IDTE_GLOBAL); else __pmdp_csp(pmdp); *pmdp = new; @@ -2523,7 +2523,7 @@ static inline void thp_split_mm(struct mm_struct *mm) * - This must be called after THP was enabled */ static int __zap_zero_pages(pmd_t *pmd, unsigned long start, - unsigned long end, struct mm_walk *walk) + unsigned long end, struct mm_walk *walk) { unsigned long addr; diff --git a/arch/s390/mm/pgtable.c b/arch/s390/mm/pgtable.c index 18205f851c24..5915f3b725bc 100644 --- a/arch/s390/mm/pgtable.c +++ b/arch/s390/mm/pgtable.c @@ -743,7 +743,7 @@ void ptep_zap_key(struct mm_struct *mm, unsigned long addr, pte_t *ptep) * Test and reset if a guest page is dirty */ bool ptep_test_and_clear_uc(struct mm_struct *mm, unsigned long addr, - pte_t *ptep) + pte_t *ptep) { pgste_t pgste; pte_t pte; From patchwork Wed Jan 13 09:41:01 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Janosch Frank X-Patchwork-Id: 12016325 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0DA74C43219 for ; Wed, 13 Jan 2021 09:42:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D054B233F8 for ; Wed, 13 Jan 2021 09:42:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727511AbhAMJmk (ORCPT ); Wed, 13 Jan 2021 04:42:40 -0500 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:41716 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727250AbhAMJmX (ORCPT ); Wed, 13 Jan 2021 04:42:23 -0500 Received: from pps.filterd (m0098414.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 10D9XDpx104317; Wed, 13 Jan 2021 04:41:41 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=hGAgBn+iRrQA7yN1MFcoZeireb0I12nN2upaHUFmkVk=; b=ZWdJrtWcfGAlHxxUbeLLyGO5uOyD0fq9/LpVLJ6Yl4uHuLb5eKFPbOh7q3ZmlUhb+iNt T83rJyvshEfhGa3ocASCRWbrH4N4mTBaz0o8s3IzAiMnlLgwpyiiHsPF8sgryd8AjkFt 3OP420hVXPDkLrvpMVatAErJlns9SnNG14cxYNcG/ezhM5yG/xJ9OO0WP+U9M04Qi6+t BBgD9uYDw9pnODrexOX0W3MYOAtEJAX9kkw1XPb1JtSLebhBGif5qlXHYNgc9adm/q17 GtzsdplkGFjd/SycUOT+bw7jkXVl1tsaXUqoLOKtaHN5RHA5WQ+F93Luc8T+3WMpOLvk eg== Received: from pps.reinject (localhost [127.0.0.1]) by mx0b-001b2d01.pphosted.com with ESMTP id 361wbnt2am-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 13 Jan 2021 04:41:41 -0500 Received: from m0098414.ppops.net (m0098414.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 10D9XwlD107749; Wed, 13 Jan 2021 04:41:40 -0500 Received: from ppma04ams.nl.ibm.com (63.31.33a9.ip4.static.sl-reverse.com [169.51.49.99]) by mx0b-001b2d01.pphosted.com with ESMTP id 361wbnt2a4-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 13 Jan 2021 04:41:40 -0500 Received: from pps.filterd (ppma04ams.nl.ibm.com [127.0.0.1]) by ppma04ams.nl.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 10D9Sevm001225; Wed, 13 Jan 2021 09:41:38 GMT Received: from b06cxnps4076.portsmouth.uk.ibm.com (d06relay13.portsmouth.uk.ibm.com [9.149.109.198]) by ppma04ams.nl.ibm.com with ESMTP id 35y448cuyk-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 13 Jan 2021 09:41:38 +0000 Received: from d06av23.portsmouth.uk.ibm.com (d06av23.portsmouth.uk.ibm.com [9.149.105.59]) by b06cxnps4076.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 10D9fZP431654148 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 13 Jan 2021 09:41:36 GMT Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id CB7E0A4040; Wed, 13 Jan 2021 09:41:35 +0000 (GMT) Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 99B6FA404D; Wed, 13 Jan 2021 09:41:35 +0000 (GMT) Received: from t46lp67.lnxne.boe (unknown [9.152.108.100]) by d06av23.portsmouth.uk.ibm.com (Postfix) with ESMTP; Wed, 13 Jan 2021 09:41:35 +0000 (GMT) From: Janosch Frank To: kvm@vger.kernel.org Cc: borntraeger@de.ibm.com, david@redhat.com, linux-s390@vger.kernel.org, imbrenda@linux.ibm.com Subject: [PATCH 02/14] s390/mm: Improve locking for huge page backings Date: Wed, 13 Jan 2021 09:41:01 +0000 Message-Id: <20210113094113.133668-3-frankja@linux.ibm.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210113094113.133668-1-frankja@linux.ibm.com> References: <20210113094113.133668-1-frankja@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.343,18.0.737 definitions=2021-01-13_03:2021-01-13,2021-01-13 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 clxscore=1015 mlxscore=0 spamscore=0 mlxlogscore=829 phishscore=0 impostorscore=0 bulkscore=0 suspectscore=0 malwarescore=0 adultscore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101130054 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org The gmap guest_table_lock is used to protect changes to the guest's DAT tables from region 1 to segments. Therefore it also protects the host to guest radix tree where each new segment mapping by gmap_link() is tracked. Changes to ptes are synchronized through the pte lock, which is easily retrievable, because the gmap shares the page tables with userspace. With huge pages the story changes. PMD tables are not shared and we're left with the pmd lock on userspace side and the guest_table_lock on the gmap side. Having two locks for an object is a guarantee for locking problems. Therefore the guest_table_lock will only be used for population of the gmap tables and hence protecting the host_to_guest tree. While the pmd lock will be used for all changes to the pmd from both userspace and the gmap. This means we need to retrieve the vmaddr to retrieve a gmap pmd, which takes a bit longer than before. But we can now operate on multiple pmds which are in disjoint segment tables instead of having a global lock. Signed-off-by: Janosch Frank --- arch/s390/include/asm/pgtable.h | 1 + arch/s390/mm/gmap.c | 70 ++++++++++++++++++++------------- arch/s390/mm/pgtable.c | 2 +- 3 files changed, 45 insertions(+), 28 deletions(-) diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtable.h index 794746a32806..b1643afe1a00 100644 --- a/arch/s390/include/asm/pgtable.h +++ b/arch/s390/include/asm/pgtable.h @@ -1519,6 +1519,7 @@ static __always_inline void __pudp_idte(unsigned long addr, pud_t *pudp, } } +pmd_t *pmd_alloc_map(struct mm_struct *mm, unsigned long addr); pmd_t pmdp_xchg_direct(struct mm_struct *, unsigned long, pmd_t *, pmd_t); pmd_t pmdp_xchg_lazy(struct mm_struct *, unsigned long, pmd_t *, pmd_t); pud_t pudp_xchg_direct(struct mm_struct *, unsigned long, pud_t *, pud_t); diff --git a/arch/s390/mm/gmap.c b/arch/s390/mm/gmap.c index f857104ca6c1..650c51749f4d 100644 --- a/arch/s390/mm/gmap.c +++ b/arch/s390/mm/gmap.c @@ -899,47 +899,62 @@ static void gmap_pte_op_end(spinlock_t *ptl) } /** - * gmap_pmd_op_walk - walk the gmap tables, get the guest table lock - * and return the pmd pointer + * gmap_pmd_op_walk - walk the gmap tables, get the pmd_lock if needed + * and return the pmd pointer or NULL * @gmap: pointer to guest mapping meta data structure * @gaddr: virtual address in the guest address space * * Returns a pointer to the pmd for a guest address, or NULL */ -static inline pmd_t *gmap_pmd_op_walk(struct gmap *gmap, unsigned long gaddr) +static inline pmd_t *gmap_pmd_op_walk(struct gmap *gmap, unsigned long gaddr, + spinlock_t **ptl) { - pmd_t *pmdp; + pmd_t *pmdp, *hpmdp; + unsigned long vmaddr; + BUG_ON(gmap_is_shadow(gmap)); - pmdp = (pmd_t *) gmap_table_walk(gmap, gaddr, 1); - if (!pmdp) - return NULL; - /* without huge pages, there is no need to take the table lock */ - if (!gmap->mm->context.allow_gmap_hpage_1m) - return pmd_none(*pmdp) ? NULL : pmdp; - - spin_lock(&gmap->guest_table_lock); - if (pmd_none(*pmdp)) { - spin_unlock(&gmap->guest_table_lock); - return NULL; + *ptl = NULL; + if (gmap->mm->context.allow_gmap_hpage_1m) { + vmaddr = __gmap_translate(gmap, gaddr); + if (IS_ERR_VALUE(vmaddr)) + return NULL; + hpmdp = pmd_alloc_map(gmap->mm, vmaddr); + if (!hpmdp) + return NULL; + *ptl = pmd_lock(gmap->mm, hpmdp); + if (pmd_none(*hpmdp)) { + spin_unlock(*ptl); + *ptl = NULL; + return NULL; + } + if (!pmd_large(*hpmdp)) { + spin_unlock(*ptl); + *ptl = NULL; + } + } + + pmdp = (pmd_t *) gmap_table_walk(gmap, gaddr, 1); + if (!pmdp || pmd_none(*pmdp)) { + if (*ptl) + spin_unlock(*ptl); + pmdp = NULL; + *ptl = NULL; } - /* 4k page table entries are locked via the pte (pte_alloc_map_lock). */ - if (!pmd_large(*pmdp)) - spin_unlock(&gmap->guest_table_lock); return pmdp; } /** - * gmap_pmd_op_end - release the guest_table_lock if needed + * gmap_pmd_op_end - release the pmd lock if needed * @gmap: pointer to the guest mapping meta data structure * @pmdp: pointer to the pmd */ -static inline void gmap_pmd_op_end(struct gmap *gmap, pmd_t *pmdp) +static inline void gmap_pmd_op_end(spinlock_t *ptl) { - if (pmd_large(*pmdp)) - spin_unlock(&gmap->guest_table_lock); + if (ptl) + spin_unlock(ptl); } /* @@ -1041,13 +1056,14 @@ static int gmap_protect_range(struct gmap *gmap, unsigned long gaddr, unsigned long len, int prot, unsigned long bits) { unsigned long vmaddr, dist; + spinlock_t *ptl = NULL; pmd_t *pmdp; int rc; BUG_ON(gmap_is_shadow(gmap)); while (len) { rc = -EAGAIN; - pmdp = gmap_pmd_op_walk(gmap, gaddr); + pmdp = gmap_pmd_op_walk(gmap, gaddr, &ptl); if (pmdp) { if (!pmd_large(*pmdp)) { rc = gmap_protect_pte(gmap, gaddr, pmdp, prot, @@ -1065,7 +1081,7 @@ static int gmap_protect_range(struct gmap *gmap, unsigned long gaddr, gaddr = (gaddr & HPAGE_MASK) + HPAGE_SIZE; } } - gmap_pmd_op_end(gmap, pmdp); + gmap_pmd_op_end(ptl); } if (rc) { if (rc == -EINVAL) @@ -2462,9 +2478,9 @@ void gmap_sync_dirty_log_pmd(struct gmap *gmap, unsigned long bitmap[4], int i; pmd_t *pmdp; pte_t *ptep; - spinlock_t *ptl; + spinlock_t *ptl = NULL; - pmdp = gmap_pmd_op_walk(gmap, gaddr); + pmdp = gmap_pmd_op_walk(gmap, gaddr, &ptl); if (!pmdp) return; @@ -2481,7 +2497,7 @@ void gmap_sync_dirty_log_pmd(struct gmap *gmap, unsigned long bitmap[4], spin_unlock(ptl); } } - gmap_pmd_op_end(gmap, pmdp); + gmap_pmd_op_end(ptl); } EXPORT_SYMBOL_GPL(gmap_sync_dirty_log_pmd); diff --git a/arch/s390/mm/pgtable.c b/arch/s390/mm/pgtable.c index 5915f3b725bc..a0e674a9c70a 100644 --- a/arch/s390/mm/pgtable.c +++ b/arch/s390/mm/pgtable.c @@ -429,7 +429,7 @@ static inline pmd_t pmdp_flush_lazy(struct mm_struct *mm, } #ifdef CONFIG_PGSTE -static pmd_t *pmd_alloc_map(struct mm_struct *mm, unsigned long addr) +pmd_t *pmd_alloc_map(struct mm_struct *mm, unsigned long addr) { pgd_t *pgd; p4d_t *p4d; From patchwork Wed Jan 13 09:41:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Janosch Frank X-Patchwork-Id: 12016321 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 30BD8C433E6 for ; Wed, 13 Jan 2021 09:42:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A3BFD233F6 for ; Wed, 13 Jan 2021 09:42:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727483AbhAMJmj (ORCPT ); Wed, 13 Jan 2021 04:42:39 -0500 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:45298 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727366AbhAMJmX (ORCPT ); Wed, 13 Jan 2021 04:42:23 -0500 Received: from pps.filterd (m0187473.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 10D9XQsN007843; Wed, 13 Jan 2021 04:41:43 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=ZChRM4aZYZ8sVIBlo4IguNAr6OULwH5eR1sz0dfVC/A=; b=H3NvlO6QcFDWUvdSqdq/+XP77L4k9pEVlkCZUdpZamUrsLlHDXeIinsUMA2ml6DEgG9x 6UqHHFldfG8sfudcmp1yyFYH288f/gGSVDNvYJPuQA7vczu+jW+x13Q/eqdGx9/6X+kF smzOB7SIEO5FyIunRwrThaDS8rYIe/ZNasB+zgDly+6QvRVJeTokaAtrMwKMT+7Fw9Yq WS/TGb++xi8ewtiZgELm/j1GWC7AgCE3N5/EyIroYEIV0fXNCb42n9W1PTKtxitzSkoz 6DlzlvFtDBjEXZV61ReLNaUpy2tU9fQ6+9Crvlid//CmJBm4W33l3/j+huUEQIW2B0Jh ZQ== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 361wdnsk1k-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 13 Jan 2021 04:41:42 -0500 Received: from m0187473.ppops.net (m0187473.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 10D9Y3tC009334; Wed, 13 Jan 2021 04:41:41 -0500 Received: from ppma04ams.nl.ibm.com (63.31.33a9.ip4.static.sl-reverse.com [169.51.49.99]) by mx0a-001b2d01.pphosted.com with ESMTP id 361wdnsk0q-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 13 Jan 2021 04:41:41 -0500 Received: from pps.filterd (ppma04ams.nl.ibm.com [127.0.0.1]) by ppma04ams.nl.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 10D9SHJt001148; Wed, 13 Jan 2021 09:41:39 GMT Received: from b06avi18878370.portsmouth.uk.ibm.com (b06avi18878370.portsmouth.uk.ibm.com [9.149.26.194]) by ppma04ams.nl.ibm.com with ESMTP id 35y448cuym-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 13 Jan 2021 09:41:38 +0000 Received: from d06av23.portsmouth.uk.ibm.com (d06av23.portsmouth.uk.ibm.com [9.149.105.59]) by b06avi18878370.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 10D9fV6B28049664 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 13 Jan 2021 09:41:31 GMT Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 13114A4055; Wed, 13 Jan 2021 09:41:36 +0000 (GMT) Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D6173A4053; Wed, 13 Jan 2021 09:41:35 +0000 (GMT) Received: from t46lp67.lnxne.boe (unknown [9.152.108.100]) by d06av23.portsmouth.uk.ibm.com (Postfix) with ESMTP; Wed, 13 Jan 2021 09:41:35 +0000 (GMT) From: Janosch Frank To: kvm@vger.kernel.org Cc: borntraeger@de.ibm.com, david@redhat.com, linux-s390@vger.kernel.org, imbrenda@linux.ibm.com Subject: [PATCH 03/14] s390/mm: Take locking out of gmap_protect_pte Date: Wed, 13 Jan 2021 09:41:02 +0000 Message-Id: <20210113094113.133668-4-frankja@linux.ibm.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210113094113.133668-1-frankja@linux.ibm.com> References: <20210113094113.133668-1-frankja@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.343,18.0.737 definitions=2021-01-13_03:2021-01-13,2021-01-13 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 bulkscore=0 impostorscore=0 adultscore=0 malwarescore=0 phishscore=0 clxscore=1015 priorityscore=1501 spamscore=0 suspectscore=0 mlxlogscore=997 mlxscore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101130056 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Locking outside of the function gives us the freedom of ordering locks, which will be important to not get locking issues for gmap_protect_rmap. Signed-off-by: Janosch Frank --- arch/s390/mm/gmap.c | 30 ++++++++++++++---------------- 1 file changed, 14 insertions(+), 16 deletions(-) diff --git a/arch/s390/mm/gmap.c b/arch/s390/mm/gmap.c index 650c51749f4d..c38f49dedf35 100644 --- a/arch/s390/mm/gmap.c +++ b/arch/s390/mm/gmap.c @@ -1017,25 +1017,15 @@ static int gmap_protect_pmd(struct gmap *gmap, unsigned long gaddr, * Expected to be called with sg->mm->mmap_lock in read */ static int gmap_protect_pte(struct gmap *gmap, unsigned long gaddr, - pmd_t *pmdp, int prot, unsigned long bits) + pte_t *ptep, int prot, unsigned long bits) { int rc; - pte_t *ptep; - spinlock_t *ptl = NULL; unsigned long pbits = 0; - if (pmd_val(*pmdp) & _SEGMENT_ENTRY_INVALID) - return -EAGAIN; - - ptep = pte_alloc_map_lock(gmap->mm, pmdp, gaddr, &ptl); - if (!ptep) - return -ENOMEM; - pbits |= (bits & GMAP_NOTIFY_MPROT) ? PGSTE_IN_BIT : 0; pbits |= (bits & GMAP_NOTIFY_SHADOW) ? PGSTE_VSIE_BIT : 0; /* Protect and unlock. */ rc = ptep_force_prot(gmap->mm, gaddr, ptep, prot, pbits); - gmap_pte_op_end(ptl); return rc; } @@ -1056,18 +1046,26 @@ static int gmap_protect_range(struct gmap *gmap, unsigned long gaddr, unsigned long len, int prot, unsigned long bits) { unsigned long vmaddr, dist; - spinlock_t *ptl = NULL; + spinlock_t *ptl_pmd = NULL, *ptl_pte = NULL; pmd_t *pmdp; + pte_t *ptep; int rc; BUG_ON(gmap_is_shadow(gmap)); while (len) { rc = -EAGAIN; - pmdp = gmap_pmd_op_walk(gmap, gaddr, &ptl); + pmdp = gmap_pmd_op_walk(gmap, gaddr, &ptl_pmd); if (pmdp) { if (!pmd_large(*pmdp)) { - rc = gmap_protect_pte(gmap, gaddr, pmdp, prot, - bits); + ptl_pte = NULL; + ptep = pte_alloc_map_lock(gmap->mm, pmdp, gaddr, + &ptl_pte); + if (ptep) + rc = gmap_protect_pte(gmap, gaddr, + ptep, prot, bits); + else + rc = -ENOMEM; + gmap_pte_op_end(ptl_pte); if (!rc) { len -= PAGE_SIZE; gaddr += PAGE_SIZE; @@ -1081,7 +1079,7 @@ static int gmap_protect_range(struct gmap *gmap, unsigned long gaddr, gaddr = (gaddr & HPAGE_MASK) + HPAGE_SIZE; } } - gmap_pmd_op_end(ptl); + gmap_pmd_op_end(ptl_pmd); } if (rc) { if (rc == -EINVAL) From patchwork Wed Jan 13 09:41:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Janosch Frank X-Patchwork-Id: 12016313 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C02FEC4332D for ; Wed, 13 Jan 2021 09:42:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9013A233F8 for ; Wed, 13 Jan 2021 09:42:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727434AbhAMJmd (ORCPT ); Wed, 13 Jan 2021 04:42:33 -0500 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:55640 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727351AbhAMJmZ (ORCPT ); Wed, 13 Jan 2021 04:42:25 -0500 Received: from pps.filterd (m0098399.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 10D9X8i9026893; Wed, 13 Jan 2021 04:41:42 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=UylKc0NUbhMzd6u0Mo2VbYCORVL1YYgg3OvT4tFfEzo=; b=U82NhEP/Te8gtdCtyueIwJ6JtopZ5lRK8FKMh8FvOVN/3XLDir6g/Yf2+LKmbftjSsOZ ULAIeXSzUbxIvT/BQ/wL/lRGv+OZqGKyDCInBt6iyMELcU75MoSacuVYPuYROQpANgL1 Jg+nmpqCsI9GORDDBb7HKU0QlqG+knLtifU+T1rtigUbUaVfg2Icc2ZyIFKP2KzW27Ht NbNLWvDP1/Ro6Q7/KyBGbbPtuGSr0E9ceqU951el0PSmejWjmN0GrShr0IKVbsG8aE78 rYp1ShLbAxCwju+UMSjhBAJ+SahSR0jA1d5uu3FMx61V2/BsOycUgJkXL8IJsiaMtB81 IA== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 361w5sajsh-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 13 Jan 2021 04:41:42 -0500 Received: from m0098399.ppops.net (m0098399.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 10D9Xa2a028356; Wed, 13 Jan 2021 04:41:41 -0500 Received: from ppma04ams.nl.ibm.com (63.31.33a9.ip4.static.sl-reverse.com [169.51.49.99]) by mx0a-001b2d01.pphosted.com with ESMTP id 361w5sajrr-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 13 Jan 2021 04:41:41 -0500 Received: from pps.filterd (ppma04ams.nl.ibm.com [127.0.0.1]) by ppma04ams.nl.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 10D9SJNQ001175; Wed, 13 Jan 2021 09:41:39 GMT Received: from b06avi18878370.portsmouth.uk.ibm.com (b06avi18878370.portsmouth.uk.ibm.com [9.149.26.194]) by ppma04ams.nl.ibm.com with ESMTP id 35y448cuyn-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 13 Jan 2021 09:41:39 +0000 Received: from d06av23.portsmouth.uk.ibm.com (d06av23.portsmouth.uk.ibm.com [9.149.105.59]) by b06avi18878370.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 10D9fVgL32244114 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 13 Jan 2021 09:41:31 GMT Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 601B0A4040; Wed, 13 Jan 2021 09:41:36 +0000 (GMT) Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 1EF32A4057; Wed, 13 Jan 2021 09:41:36 +0000 (GMT) Received: from t46lp67.lnxne.boe (unknown [9.152.108.100]) by d06av23.portsmouth.uk.ibm.com (Postfix) with ESMTP; Wed, 13 Jan 2021 09:41:36 +0000 (GMT) From: Janosch Frank To: kvm@vger.kernel.org Cc: borntraeger@de.ibm.com, david@redhat.com, linux-s390@vger.kernel.org, imbrenda@linux.ibm.com Subject: [PATCH 04/14] s390/mm: split huge pages in GMAP when protecting Date: Wed, 13 Jan 2021 09:41:03 +0000 Message-Id: <20210113094113.133668-5-frankja@linux.ibm.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210113094113.133668-1-frankja@linux.ibm.com> References: <20210113094113.133668-1-frankja@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.343,18.0.737 definitions=2021-01-13_03:2021-01-13,2021-01-13 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 adultscore=0 clxscore=1015 priorityscore=1501 malwarescore=0 suspectscore=0 impostorscore=0 bulkscore=0 mlxscore=0 lowpriorityscore=0 mlxlogscore=999 spamscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101130056 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Dirty tracking, vsie protection and lowcore invalidation notification are best done on the smallest page size available to avoid unnecessary flushing and table management operations. Hence we now split huge pages and introduce a page table if a notification bit is set or memory is protected via gmap_protect_range or gmap_protect_rmap. Signed-off-by: Janosch Frank --- arch/s390/include/asm/gmap.h | 18 +++ arch/s390/include/asm/pgtable.h | 3 + arch/s390/mm/gmap.c | 247 +++++++++++++++++++++++++------- arch/s390/mm/pgtable.c | 33 +++++ 4 files changed, 251 insertions(+), 50 deletions(-) diff --git a/arch/s390/include/asm/gmap.h b/arch/s390/include/asm/gmap.h index 40264f60b0da..a5711c189018 100644 --- a/arch/s390/include/asm/gmap.h +++ b/arch/s390/include/asm/gmap.h @@ -19,6 +19,11 @@ /* Status bits only for huge segment entries */ #define _SEGMENT_ENTRY_GMAP_IN 0x8000 /* invalidation notify bit */ #define _SEGMENT_ENTRY_GMAP_UC 0x4000 /* dirty (migration) */ +/* Status bits in the gmap segment entry. */ +#define _SEGMENT_ENTRY_GMAP_SPLIT 0x0001 /* split huge pmd */ + +#define GMAP_SEGMENT_STATUS_BITS (_SEGMENT_ENTRY_GMAP_UC | _SEGMENT_ENTRY_GMAP_SPLIT) +#define GMAP_SEGMENT_NOTIFY_BITS _SEGMENT_ENTRY_GMAP_IN /** * struct gmap_struct - guest address space @@ -62,6 +67,8 @@ struct gmap { struct radix_tree_root host_to_rmap; struct list_head children; struct list_head pt_list; + struct list_head split_list; + spinlock_t split_list_lock; spinlock_t shadow_lock; struct gmap *parent; unsigned long orig_asce; @@ -102,6 +109,17 @@ static inline int gmap_is_shadow(struct gmap *gmap) return !!gmap->parent; } +/** + * gmap_pmd_is_split - Returns if a huge gmap pmd has been split. + * @pmdp: pointer to the pmd + * + * Returns true if the passed huge gmap pmd has been split. + */ +static inline bool gmap_pmd_is_split(pmd_t *pmdp) +{ + return !!(pmd_val(*pmdp) & _SEGMENT_ENTRY_GMAP_SPLIT); +} + struct gmap *gmap_create(struct mm_struct *mm, unsigned long limit); void gmap_remove(struct gmap *gmap); struct gmap *gmap_get(struct gmap *gmap); diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtable.h index b1643afe1a00..6d6ad508f9c7 100644 --- a/arch/s390/include/asm/pgtable.h +++ b/arch/s390/include/asm/pgtable.h @@ -1147,6 +1147,9 @@ int ptep_shadow_pte(struct mm_struct *mm, unsigned long saddr, pte_t *sptep, pte_t *tptep, pte_t pte); void ptep_unshadow_pte(struct mm_struct *mm, unsigned long saddr, pte_t *ptep); +unsigned long ptep_get_and_clear_notification_bits(pte_t *ptep); +void ptep_remove_protection_split(struct mm_struct *mm, pte_t *ptep, + unsigned long gaddr); bool ptep_test_and_clear_uc(struct mm_struct *mm, unsigned long address, pte_t *ptep); int set_guest_storage_key(struct mm_struct *mm, unsigned long addr, diff --git a/arch/s390/mm/gmap.c b/arch/s390/mm/gmap.c index c38f49dedf35..41a5bbbc59e6 100644 --- a/arch/s390/mm/gmap.c +++ b/arch/s390/mm/gmap.c @@ -62,11 +62,13 @@ static struct gmap *gmap_alloc(unsigned long limit) INIT_LIST_HEAD(&gmap->crst_list); INIT_LIST_HEAD(&gmap->children); INIT_LIST_HEAD(&gmap->pt_list); + INIT_LIST_HEAD(&gmap->split_list); INIT_RADIX_TREE(&gmap->guest_to_host, GFP_KERNEL_ACCOUNT); INIT_RADIX_TREE(&gmap->host_to_guest, GFP_ATOMIC | __GFP_ACCOUNT); INIT_RADIX_TREE(&gmap->host_to_rmap, GFP_ATOMIC | __GFP_ACCOUNT); spin_lock_init(&gmap->guest_table_lock); spin_lock_init(&gmap->shadow_lock); + spin_lock_init(&gmap->split_list_lock); refcount_set(&gmap->ref_count, 1); page = alloc_pages(GFP_KERNEL_ACCOUNT, CRST_ALLOC_ORDER); if (!page) @@ -193,6 +195,10 @@ static void gmap_free(struct gmap *gmap) gmap_radix_tree_free(&gmap->guest_to_host); gmap_radix_tree_free(&gmap->host_to_guest); + /* Free split pmd page tables */ + list_for_each_entry_safe(page, next, &gmap->split_list, lru) + page_table_free_pgste(page); + /* Free additional data for a shadow gmap */ if (gmap_is_shadow(gmap)) { /* Free all page tables. */ @@ -547,6 +553,7 @@ int __gmap_link(struct gmap *gmap, unsigned long gaddr, unsigned long vmaddr) pud_t *pud; pmd_t *pmd; u64 unprot; + pte_t *ptep; int rc; BUG_ON(gmap_is_shadow(gmap)); @@ -597,9 +604,15 @@ int __gmap_link(struct gmap *gmap, unsigned long gaddr, unsigned long vmaddr) rc = radix_tree_preload(GFP_KERNEL_ACCOUNT); if (rc) return rc; + /* + * do_exception() does remove the pte index for huge + * pages, so we need to re-add it here to work on the + * correct pte. + */ + vmaddr = vmaddr | (gaddr & ~PMD_MASK); ptl = pmd_lock(mm, pmd); - spin_lock(&gmap->guest_table_lock); if (*table == _SEGMENT_ENTRY_EMPTY) { + spin_lock(&gmap->guest_table_lock); rc = radix_tree_insert(&gmap->host_to_guest, vmaddr >> PMD_SHIFT, table); if (!rc) { @@ -611,14 +624,24 @@ int __gmap_link(struct gmap *gmap, unsigned long gaddr, unsigned long vmaddr) *table = pmd_val(*pmd) & _SEGMENT_ENTRY_HARDWARE_BITS; } + spin_unlock(&gmap->guest_table_lock); } else if (*table & _SEGMENT_ENTRY_PROTECT && !(pmd_val(*pmd) & _SEGMENT_ENTRY_PROTECT)) { unprot = (u64)*table; unprot &= ~_SEGMENT_ENTRY_PROTECT; unprot |= _SEGMENT_ENTRY_GMAP_UC; gmap_pmdp_xchg(gmap, (pmd_t *)table, __pmd(unprot), gaddr); + } else if (gmap_pmd_is_split((pmd_t *)table)) { + /* + * Split pmds are somewhere in-between a normal and a + * large pmd. As we don't share the page table, the + * host does not remove protection on a fault and we + * have to do it ourselves for the guest mapping. + */ + ptep = pte_offset_map((pmd_t *)table, vmaddr); + if (pte_val(*ptep) & _PAGE_PROTECT) + ptep_remove_protection_split(mm, ptep, vmaddr); } - spin_unlock(&gmap->guest_table_lock); spin_unlock(ptl); radix_tree_preload_end(); return rc; @@ -860,7 +883,7 @@ static pte_t *gmap_pte_op_walk(struct gmap *gmap, unsigned long gaddr, } /** - * gmap_pte_op_fixup - force a page in and connect the gmap page table + * gmap_fixup - force memory in and connect the gmap table entry * @gmap: pointer to guest mapping meta data structure * @gaddr: virtual address in the guest address space * @vmaddr: address in the host process address space @@ -868,10 +891,10 @@ static pte_t *gmap_pte_op_walk(struct gmap *gmap, unsigned long gaddr, * * Returns 0 if the caller can retry __gmap_translate (might fail again), * -ENOMEM if out of memory and -EFAULT if anything goes wrong while fixing - * up or connecting the gmap page table. + * up or connecting the gmap table entry. */ -static int gmap_pte_op_fixup(struct gmap *gmap, unsigned long gaddr, - unsigned long vmaddr, int prot) +static int gmap_fixup(struct gmap *gmap, unsigned long gaddr, + unsigned long vmaddr, int prot) { struct mm_struct *mm = gmap->mm; unsigned int fault_flags; @@ -957,6 +980,76 @@ static inline void gmap_pmd_op_end(spinlock_t *ptl) spin_unlock(ptl); } +static pte_t *gmap_pte_from_pmd(struct gmap *gmap, pmd_t *pmdp, + unsigned long addr, spinlock_t **ptl) +{ + *ptl = NULL; + if (likely(!gmap_pmd_is_split(pmdp))) + return pte_alloc_map_lock(gmap->mm, pmdp, addr, ptl); + + return pte_offset_map(pmdp, addr); +} + +/** + * gmap_pmd_split_free - Free a split pmd's page table + * @pmdp The split pmd that we free of its page table + * + * If the userspace pmds are exchanged, we'll remove the gmap pmds as + * well, so we fault on them and link them again. We would leak + * memory, if we didn't free split pmds here. + */ +static inline void gmap_pmd_split_free(struct gmap *gmap, pmd_t *pmdp) +{ + unsigned long pgt = pmd_val(*pmdp) & _SEGMENT_ENTRY_ORIGIN; + struct page *page; + + if (gmap_pmd_is_split(pmdp)) { + page = pfn_to_page(pgt >> PAGE_SHIFT); + spin_lock(&gmap->split_list_lock); + list_del(&page->lru); + spin_unlock(&gmap->split_list_lock); + page_table_free_pgste(page); + } +} + +/** + * gmap_pmd_split - Split a huge gmap pmd and use a page table instead + * @gmap: pointer to guest mapping meta data structure + * @gaddr: virtual address in the guest address space + * @pmdp: pointer to the pmd that will be split + * @pgtable: Pre-allocated page table + * + * When splitting gmap pmds, we have to make the resulting page table + * look like it's a normal one to be able to use the common pte + * handling functions. Also we need to track these new tables as they + * aren't tracked anywhere else. + */ +static void gmap_pmd_split(struct gmap *gmap, unsigned long gaddr, + pmd_t *pmdp, struct page *page) +{ + unsigned long *ptable = (unsigned long *) page_to_phys(page); + pmd_t new; + int i; + + for (i = 0; i < 256; i++) { + ptable[i] = (pmd_val(*pmdp) & HPAGE_MASK) + i * PAGE_SIZE; + /* Carry over hardware permission from the pmd */ + if (pmd_val(*pmdp) & _SEGMENT_ENTRY_PROTECT) + ptable[i] |= _PAGE_PROTECT; + /* pmd_large() implies pmd/pte_present() */ + ptable[i] |= _PAGE_PRESENT | _PAGE_READ | _PAGE_WRITE; + /* ptes are directly marked as dirty */ + ptable[i + PTRS_PER_PTE] |= PGSTE_UC_BIT; + } + + pmd_val(new) = ((unsigned long)ptable | _SEGMENT_ENTRY | + (_SEGMENT_ENTRY_GMAP_SPLIT)); + spin_lock(&gmap->split_list_lock); + list_add(&page->lru, &gmap->split_list); + spin_unlock(&gmap->split_list_lock); + gmap_pmdp_xchg(gmap, pmdp, new, gaddr); +} + /* * gmap_protect_pmd - remove access rights to memory and set pmd notification bits * @pmdp: pointer to the pmd to be protected @@ -1045,7 +1138,8 @@ static int gmap_protect_pte(struct gmap *gmap, unsigned long gaddr, static int gmap_protect_range(struct gmap *gmap, unsigned long gaddr, unsigned long len, int prot, unsigned long bits) { - unsigned long vmaddr, dist; + struct page *page = NULL; + unsigned long vmaddr; spinlock_t *ptl_pmd = NULL, *ptl_pte = NULL; pmd_t *pmdp; pte_t *ptep; @@ -1054,12 +1148,12 @@ static int gmap_protect_range(struct gmap *gmap, unsigned long gaddr, BUG_ON(gmap_is_shadow(gmap)); while (len) { rc = -EAGAIN; + pmdp = gmap_pmd_op_walk(gmap, gaddr, &ptl_pmd); - if (pmdp) { + if (pmdp && !(pmd_val(*pmdp) & _SEGMENT_ENTRY_INVALID)) { if (!pmd_large(*pmdp)) { - ptl_pte = NULL; - ptep = pte_alloc_map_lock(gmap->mm, pmdp, gaddr, - &ptl_pte); + ptep = gmap_pte_from_pmd(gmap, pmdp, gaddr, + &ptl_pte); if (ptep) rc = gmap_protect_pte(gmap, gaddr, ptep, prot, bits); @@ -1071,25 +1165,37 @@ static int gmap_protect_range(struct gmap *gmap, unsigned long gaddr, gaddr += PAGE_SIZE; } } else { - rc = gmap_protect_pmd(gmap, gaddr, pmdp, prot, - bits); - if (!rc) { - dist = HPAGE_SIZE - (gaddr & ~HPAGE_MASK); - len = len < dist ? 0 : len - dist; - gaddr = (gaddr & HPAGE_MASK) + HPAGE_SIZE; + if (!page) { + /* Drop locks for allocation. */ + gmap_pmd_op_end(ptl_pmd); + ptl_pmd = NULL; + page = page_table_alloc_pgste(gmap->mm); + if (!page) + return -ENOMEM; + continue; + } else { + gmap_pmd_split(gmap, gaddr, + pmdp, page); + page = NULL; + gmap_pmd_op_end(ptl_pmd); + continue; } } gmap_pmd_op_end(ptl_pmd); } + if (page) { + page_table_free_pgste(page); + page = NULL; + } if (rc) { - if (rc == -EINVAL) + if (rc == -EINVAL || rc == -ENOMEM) return rc; /* -EAGAIN, fixup of userspace mm and gmap */ vmaddr = __gmap_translate(gmap, gaddr); if (IS_ERR_VALUE(vmaddr)) return vmaddr; - rc = gmap_pte_op_fixup(gmap, gaddr, vmaddr, prot); + rc = gmap_fixup(gmap, gaddr, vmaddr, prot); if (rc) return rc; } @@ -1172,7 +1278,7 @@ int gmap_read_table(struct gmap *gmap, unsigned long gaddr, unsigned long *val) rc = vmaddr; break; } - rc = gmap_pte_op_fixup(gmap, gaddr, vmaddr, PROT_READ); + rc = gmap_fixup(gmap, gaddr, vmaddr, PROT_READ); if (rc) break; } @@ -1255,7 +1361,7 @@ static int gmap_protect_rmap(struct gmap *sg, unsigned long raddr, radix_tree_preload_end(); if (rc) { kfree(rmap); - rc = gmap_pte_op_fixup(parent, paddr, vmaddr, PROT_READ); + rc = gmap_fixup(parent, paddr, vmaddr, PROT_READ); if (rc) return rc; continue; @@ -2170,7 +2276,7 @@ int gmap_shadow_page(struct gmap *sg, unsigned long saddr, pte_t pte) radix_tree_preload_end(); if (!rc) break; - rc = gmap_pte_op_fixup(parent, paddr, vmaddr, prot); + rc = gmap_fixup(parent, paddr, vmaddr, prot); if (rc) break; } @@ -2236,6 +2342,30 @@ static void gmap_shadow_notify(struct gmap *sg, unsigned long vmaddr, spin_unlock(&sg->guest_table_lock); } +/* + * ptep_notify_gmap - call all invalidation callbacks for a specific pte of a gmap + * @mm: pointer to the process mm_struct + * @addr: virtual address in the process address space + * @pte: pointer to the page table entry + * @bits: bits from the pgste that caused the notify call + * + * This function is assumed to be called with the guest_table_lock held. + */ +static void ptep_notify_gmap(struct gmap *gmap, unsigned long gaddr, + unsigned long vmaddr, unsigned long bits) +{ + struct gmap *sg, *next; + + if (!list_empty(&gmap->children) && (bits & PGSTE_VSIE_BIT)) { + spin_lock(&gmap->shadow_lock); + list_for_each_entry_safe(sg, next, &gmap->children, list) + gmap_shadow_notify(sg, vmaddr, gaddr); + spin_unlock(&gmap->shadow_lock); + } + if (bits & PGSTE_IN_BIT) + gmap_call_notifier(gmap, gaddr, gaddr + PAGE_SIZE - 1); +} + /** * ptep_notify - call all invalidation callbacks for a specific pte. * @mm: pointer to the process mm_struct @@ -2251,7 +2381,7 @@ void ptep_notify(struct mm_struct *mm, unsigned long vmaddr, { unsigned long offset, gaddr = 0; unsigned long *table; - struct gmap *gmap, *sg, *next; + struct gmap *gmap; offset = ((unsigned long) pte) & (255 * sizeof(pte_t)); offset = offset * (PAGE_SIZE / sizeof(pte_t)); @@ -2266,23 +2396,34 @@ void ptep_notify(struct mm_struct *mm, unsigned long vmaddr, if (!table) continue; - if (!list_empty(&gmap->children) && (bits & PGSTE_VSIE_BIT)) { - spin_lock(&gmap->shadow_lock); - list_for_each_entry_safe(sg, next, - &gmap->children, list) - gmap_shadow_notify(sg, vmaddr, gaddr); - spin_unlock(&gmap->shadow_lock); - } - if (bits & PGSTE_IN_BIT) - gmap_call_notifier(gmap, gaddr, gaddr + PAGE_SIZE - 1); + ptep_notify_gmap(gmap, gaddr, vmaddr, bits); } rcu_read_unlock(); } EXPORT_SYMBOL_GPL(ptep_notify); -static void pmdp_notify_gmap(struct gmap *gmap, pmd_t *pmdp, - unsigned long gaddr) +static inline void pmdp_notify_split(struct gmap *gmap, pmd_t *pmdp, + unsigned long gaddr, unsigned long vmaddr) { + int i = 0; + unsigned long bits; + pte_t *ptep = (pte_t *)(pmd_val(*pmdp) & PAGE_MASK); + + for (; i < 256; i++, gaddr += PAGE_SIZE, vmaddr += PAGE_SIZE, ptep++) { + bits = ptep_get_and_clear_notification_bits(ptep); + if (bits) + ptep_notify_gmap(gmap, gaddr, vmaddr, bits); + } +} + +static void pmdp_notify_gmap(struct gmap *gmap, pmd_t *pmdp, + unsigned long gaddr, unsigned long vmaddr) +{ + if (gmap_pmd_is_split(pmdp)) + return pmdp_notify_split(gmap, pmdp, gaddr, vmaddr); + + if (!(pmd_val(*pmdp) & _SEGMENT_ENTRY_GMAP_IN)) + return; pmd_val(*pmdp) &= ~_SEGMENT_ENTRY_GMAP_IN; gmap_call_notifier(gmap, gaddr, gaddr + HPAGE_SIZE - 1); } @@ -2301,8 +2442,9 @@ static void gmap_pmdp_xchg(struct gmap *gmap, pmd_t *pmdp, pmd_t new, unsigned long gaddr) { gaddr &= HPAGE_MASK; - pmdp_notify_gmap(gmap, pmdp, gaddr); - pmd_val(new) &= ~_SEGMENT_ENTRY_GMAP_IN; + pmdp_notify_gmap(gmap, pmdp, gaddr, 0); + if (pmd_large(new)) + pmd_val(new) &= ~GMAP_SEGMENT_NOTIFY_BITS; if (MACHINE_HAS_TLB_GUEST) __pmdp_idte(gaddr, pmdp, IDTE_GUEST_ASCE, gmap->asce, IDTE_GLOBAL); @@ -2327,11 +2469,13 @@ static void gmap_pmdp_clear(struct mm_struct *mm, unsigned long vmaddr, vmaddr >> PMD_SHIFT); if (pmdp) { gaddr = __gmap_segment_gaddr((unsigned long *)pmdp); - pmdp_notify_gmap(gmap, pmdp, gaddr); - WARN_ON(pmd_val(*pmdp) & ~(_SEGMENT_ENTRY_HARDWARE_BITS_LARGE | - _SEGMENT_ENTRY_GMAP_UC)); + pmdp_notify_gmap(gmap, pmdp, gaddr, vmaddr); + if (pmd_large(*pmdp)) + WARN_ON(pmd_val(*pmdp) & + GMAP_SEGMENT_NOTIFY_BITS); if (purge) __pmdp_csp(pmdp); + gmap_pmd_split_free(gmap, pmdp); pmd_val(*pmdp) = _SEGMENT_ENTRY_EMPTY; } spin_unlock(&gmap->guest_table_lock); @@ -2381,14 +2525,15 @@ void gmap_pmdp_idte_local(struct mm_struct *mm, unsigned long vmaddr) if (entry) { pmdp = (pmd_t *)entry; gaddr = __gmap_segment_gaddr(entry); - pmdp_notify_gmap(gmap, pmdp, gaddr); - WARN_ON(*entry & ~(_SEGMENT_ENTRY_HARDWARE_BITS_LARGE | - _SEGMENT_ENTRY_GMAP_UC)); + pmdp_notify_gmap(gmap, pmdp, gaddr, vmaddr); + if (pmd_large(*pmdp)) + WARN_ON(*entry & GMAP_SEGMENT_NOTIFY_BITS); if (MACHINE_HAS_TLB_GUEST) __pmdp_idte(gaddr, pmdp, IDTE_GUEST_ASCE, gmap->asce, IDTE_LOCAL); else if (MACHINE_HAS_IDTE) __pmdp_idte(gaddr, pmdp, 0, 0, IDTE_LOCAL); + gmap_pmd_split_free(gmap, pmdp); *entry = _SEGMENT_ENTRY_EMPTY; } spin_unlock(&gmap->guest_table_lock); @@ -2416,9 +2561,9 @@ void gmap_pmdp_idte_global(struct mm_struct *mm, unsigned long vmaddr) if (entry) { pmdp = (pmd_t *)entry; gaddr = __gmap_segment_gaddr(entry); - pmdp_notify_gmap(gmap, pmdp, gaddr); - WARN_ON(*entry & ~(_SEGMENT_ENTRY_HARDWARE_BITS_LARGE | - _SEGMENT_ENTRY_GMAP_UC)); + pmdp_notify_gmap(gmap, pmdp, gaddr, vmaddr); + if (pmd_large(*pmdp)) + WARN_ON(*entry & GMAP_SEGMENT_NOTIFY_BITS); if (MACHINE_HAS_TLB_GUEST) __pmdp_idte(gaddr, pmdp, IDTE_GUEST_ASCE, gmap->asce, IDTE_GLOBAL); @@ -2426,6 +2571,7 @@ void gmap_pmdp_idte_global(struct mm_struct *mm, unsigned long vmaddr) __pmdp_idte(gaddr, pmdp, 0, 0, IDTE_GLOBAL); else __pmdp_csp(pmdp); + gmap_pmd_split_free(gmap, pmdp); *entry = _SEGMENT_ENTRY_EMPTY; } spin_unlock(&gmap->guest_table_lock); @@ -2476,9 +2622,10 @@ void gmap_sync_dirty_log_pmd(struct gmap *gmap, unsigned long bitmap[4], int i; pmd_t *pmdp; pte_t *ptep; - spinlock_t *ptl = NULL; + spinlock_t *ptl_pmd = NULL; + spinlock_t *ptl_pte = NULL; - pmdp = gmap_pmd_op_walk(gmap, gaddr, &ptl); + pmdp = gmap_pmd_op_walk(gmap, gaddr, &ptl_pmd); if (!pmdp) return; @@ -2487,15 +2634,15 @@ void gmap_sync_dirty_log_pmd(struct gmap *gmap, unsigned long bitmap[4], bitmap_fill(bitmap, _PAGE_ENTRIES); } else { for (i = 0; i < _PAGE_ENTRIES; i++, vmaddr += PAGE_SIZE) { - ptep = pte_alloc_map_lock(gmap->mm, pmdp, vmaddr, &ptl); + ptep = gmap_pte_from_pmd(gmap, pmdp, vmaddr, &ptl_pte); if (!ptep) continue; if (ptep_test_and_clear_uc(gmap->mm, vmaddr, ptep)) set_bit(i, bitmap); - spin_unlock(ptl); + gmap_pte_op_end(ptl_pte); } } - gmap_pmd_op_end(ptl); + gmap_pmd_op_end(ptl_pmd); } EXPORT_SYMBOL_GPL(gmap_sync_dirty_log_pmd); diff --git a/arch/s390/mm/pgtable.c b/arch/s390/mm/pgtable.c index a0e674a9c70a..16896f936d32 100644 --- a/arch/s390/mm/pgtable.c +++ b/arch/s390/mm/pgtable.c @@ -739,6 +739,39 @@ void ptep_zap_key(struct mm_struct *mm, unsigned long addr, pte_t *ptep) preempt_enable(); } +unsigned long ptep_get_and_clear_notification_bits(pte_t *ptep) +{ + pgste_t pgste; + unsigned long bits; + + pgste = pgste_get_lock(ptep); + bits = pgste_val(pgste) & (PGSTE_IN_BIT | PGSTE_VSIE_BIT); + pgste_val(pgste) ^= bits; + pgste_set_unlock(ptep, pgste); + + return bits; +} +EXPORT_SYMBOL_GPL(ptep_get_and_clear_notification_bits); + +void ptep_remove_protection_split(struct mm_struct *mm, pte_t *ptep, + unsigned long gaddr) +{ + pte_t pte; + pgste_t pgste; + + pgste = pgste_get_lock(ptep); + pgste_val(pgste) |= PGSTE_UC_BIT; + pte = *ptep; + pte_val(pte) &= ~_PAGE_PROTECT; + + pgste = pgste_pte_notify(mm, gaddr, ptep, pgste); + ptep_ipte_global(mm, gaddr, ptep, 0); + + *ptep = pte; + pgste_set_unlock(ptep, pgste); +} +EXPORT_SYMBOL_GPL(ptep_remove_protection_split); + /* * Test and reset if a guest page is dirty */ From patchwork Wed Jan 13 09:41:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Janosch Frank X-Patchwork-Id: 12016301 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C25EDC433DB for ; Wed, 13 Jan 2021 09:42:41 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 77926233FB for ; Wed, 13 Jan 2021 09:42:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727409AbhAMJmZ (ORCPT ); Wed, 13 Jan 2021 04:42:25 -0500 Received: from mx0b-001b2d01.pphosted.com ([148.163.158.5]:51690 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1727342AbhAMJmX (ORCPT ); Wed, 13 Jan 2021 04:42:23 -0500 Received: from pps.filterd (m0098413.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 10D9Vpmc040528; Wed, 13 Jan 2021 04:41:41 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=9WZYSa7JbW9fugCMfA4EOQxhXDZxIz1Ry9oawJjpNWQ=; b=Jr5vG7/he/tEBmR8NGAJSpGeIaW3WN0Ui7MSV1LQ/3YqRXE+bAMvWH2V9gNP6owfY/9S HR0YjOukWhZRi4TsM9wXzq+jn1KiEbZc/BckvSM2xVe08qjLfgOAT9l1I88NhsBzo/hp YGcHmdAoHgnJb3DqkMp3LYlfNcjpYWWgMN4XEJFzOCu2dqlswlLpykR/s/19xNlUfic/ V3ZPHSEBRS9JTWM4s/nmVz2gQNL8vJvqJVuR60WY1rShJcpmYL5Unujr5u9vR4NI1QNI /TGigkKuRBSb+i+rRltqfNOxncHcuwBnmd5xpjjGyhnanHlHiqle+4KRinBzHbkA2TIi wQ== Received: from pps.reinject (localhost [127.0.0.1]) by mx0b-001b2d01.pphosted.com with ESMTP id 361wrysb56-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 13 Jan 2021 04:41:41 -0500 Received: from m0098413.ppops.net (m0098413.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 10D9VrCe040663; Wed, 13 Jan 2021 04:41:41 -0500 Received: from ppma06ams.nl.ibm.com (66.31.33a9.ip4.static.sl-reverse.com [169.51.49.102]) by mx0b-001b2d01.pphosted.com with ESMTP id 361wrysb4u-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 13 Jan 2021 04:41:41 -0500 Received: from pps.filterd (ppma06ams.nl.ibm.com [127.0.0.1]) by ppma06ams.nl.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 10D9S67B007667; Wed, 13 Jan 2021 09:41:39 GMT Received: from b06avi18878370.portsmouth.uk.ibm.com (b06avi18878370.portsmouth.uk.ibm.com [9.149.26.194]) by ppma06ams.nl.ibm.com with ESMTP id 35ydrdcehy-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 13 Jan 2021 09:41:39 +0000 Received: from d06av23.portsmouth.uk.ibm.com (d06av23.portsmouth.uk.ibm.com [9.149.105.59]) by b06avi18878370.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 10D9fVQo25821694 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 13 Jan 2021 09:41:31 GMT Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 9CB19A4051; Wed, 13 Jan 2021 09:41:36 +0000 (GMT) Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 6B113A4055; Wed, 13 Jan 2021 09:41:36 +0000 (GMT) Received: from t46lp67.lnxne.boe (unknown [9.152.108.100]) by d06av23.portsmouth.uk.ibm.com (Postfix) with ESMTP; Wed, 13 Jan 2021 09:41:36 +0000 (GMT) From: Janosch Frank To: kvm@vger.kernel.org Cc: borntraeger@de.ibm.com, david@redhat.com, linux-s390@vger.kernel.org, imbrenda@linux.ibm.com Subject: [PATCH 05/14] s390/mm: Split huge pages when migrating Date: Wed, 13 Jan 2021 09:41:04 +0000 Message-Id: <20210113094113.133668-6-frankja@linux.ibm.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210113094113.133668-1-frankja@linux.ibm.com> References: <20210113094113.133668-1-frankja@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.343,18.0.737 definitions=2021-01-13_03:2021-01-13,2021-01-13 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 lowpriorityscore=0 phishscore=0 clxscore=1015 malwarescore=0 adultscore=0 mlxlogscore=999 bulkscore=0 impostorscore=0 priorityscore=1501 suspectscore=0 spamscore=0 mlxscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101130054 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Right now we mark the huge page that is being written to as dirty although only a single byte may have changed. This means we have to migrate 1MB although only a very limited amount of memory in that range might be dirty. To speed up migration this patch splits up write protected huge pages into normal pages. The protection for the normal pages is only removed for the page that caused the fault. Signed-off-by: Janosch Frank --- arch/s390/mm/gmap.c | 34 ++++++++++++++++++++++++++++------ 1 file changed, 28 insertions(+), 6 deletions(-) diff --git a/arch/s390/mm/gmap.c b/arch/s390/mm/gmap.c index 41a5bbbc59e6..d8c9b295294b 100644 --- a/arch/s390/mm/gmap.c +++ b/arch/s390/mm/gmap.c @@ -532,6 +532,9 @@ void gmap_unlink(struct mm_struct *mm, unsigned long *table, static void gmap_pmdp_xchg(struct gmap *gmap, pmd_t *old, pmd_t new, unsigned long gaddr); +static void gmap_pmd_split(struct gmap *gmap, unsigned long gaddr, + pmd_t *pmdp, struct page *page); + /** * gmap_link - set up shadow page tables to connect a host to a guest address * @gmap: pointer to guest mapping meta data structure @@ -547,12 +550,12 @@ int __gmap_link(struct gmap *gmap, unsigned long gaddr, unsigned long vmaddr) { struct mm_struct *mm; unsigned long *table; + struct page *page = NULL; spinlock_t *ptl; pgd_t *pgd; p4d_t *p4d; pud_t *pud; pmd_t *pmd; - u64 unprot; pte_t *ptep; int rc; @@ -600,6 +603,7 @@ int __gmap_link(struct gmap *gmap, unsigned long gaddr, unsigned long vmaddr) /* Are we allowed to use huge pages? */ if (pmd_large(*pmd) && !gmap->mm->context.allow_gmap_hpage_1m) return -EFAULT; +retry_split: /* Link gmap segment table entry location to page table. */ rc = radix_tree_preload(GFP_KERNEL_ACCOUNT); if (rc) @@ -627,10 +631,25 @@ int __gmap_link(struct gmap *gmap, unsigned long gaddr, unsigned long vmaddr) spin_unlock(&gmap->guest_table_lock); } else if (*table & _SEGMENT_ENTRY_PROTECT && !(pmd_val(*pmd) & _SEGMENT_ENTRY_PROTECT)) { - unprot = (u64)*table; - unprot &= ~_SEGMENT_ENTRY_PROTECT; - unprot |= _SEGMENT_ENTRY_GMAP_UC; - gmap_pmdp_xchg(gmap, (pmd_t *)table, __pmd(unprot), gaddr); + if (page) { + gmap_pmd_split(gmap, gaddr, (pmd_t *)table, page); + page = NULL; + } else { + spin_unlock(ptl); + ptl = NULL; + radix_tree_preload_end(); + page = page_table_alloc_pgste(mm); + if (!page) + rc = -ENOMEM; + else + goto retry_split; + } + /* + * The split moves over the protection, so we still + * need to unprotect. + */ + ptep = pte_offset_map((pmd_t *)table, vmaddr); + ptep_remove_protection_split(mm, ptep, vmaddr); } else if (gmap_pmd_is_split((pmd_t *)table)) { /* * Split pmds are somewhere in-between a normal and a @@ -642,7 +661,10 @@ int __gmap_link(struct gmap *gmap, unsigned long gaddr, unsigned long vmaddr) if (pte_val(*ptep) & _PAGE_PROTECT) ptep_remove_protection_split(mm, ptep, vmaddr); } - spin_unlock(ptl); + if (page) + page_table_free_pgste(page); + if (ptl) + spin_unlock(ptl); radix_tree_preload_end(); return rc; } From patchwork Wed Jan 13 09:41:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Janosch Frank X-Patchwork-Id: 12016303 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2C1ABC4332B for ; Wed, 13 Jan 2021 09:42:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id D9A6F233FA for ; Wed, 13 Jan 2021 09:42:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727420AbhAMJm2 (ORCPT ); Wed, 13 Jan 2021 04:42:28 -0500 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:17612 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727364AbhAMJmY (ORCPT ); Wed, 13 Jan 2021 04:42:24 -0500 Received: from pps.filterd (m0187473.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 10D9XPRs007835; Wed, 13 Jan 2021 04:41:43 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=Lbo11U6FHjgeuaZyj0uVo3tgcVejkmwi/Njyi97reyE=; b=fXPpGMvWHpcfntuiAYHsMkO255v9siIIOtyhLFqXcOYK6EhZgaN9ZSapptr6iCQsW3wz ceaZdvZgalLO2xsq6fNdfkYm2jh25Ecoay6cEmXZVP6p96nCqNHP/3tsmMucmilV4atT bYzbnsBIqYAVX/Wo5aM+kBcy+RJCKxotrMMWFpcM/SAFfGx1DyRoWFxDR5d9ZwjTvHAG lV/BATzmG6li2OOE5JylY1IJ9j9OvCUvwzz9akiC2/29lObuvv754l5sLORIHN4THg1Q k6uMuPo2/R9pfqbgj0u/6FxZ8AxHzn7rn0DOrwou5sIFZiUsfXNXIfWV+eVkI/qPKWWi mg== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 361wdnsk1w-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 13 Jan 2021 04:41:42 -0500 Received: from m0187473.ppops.net (m0187473.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 10D9Y3tE009334; Wed, 13 Jan 2021 04:41:42 -0500 Received: from ppma03ams.nl.ibm.com (62.31.33a9.ip4.static.sl-reverse.com [169.51.49.98]) by mx0a-001b2d01.pphosted.com with ESMTP id 361wdnsk12-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 13 Jan 2021 04:41:41 -0500 Received: from pps.filterd (ppma03ams.nl.ibm.com [127.0.0.1]) by ppma03ams.nl.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 10D9ROHv019480; Wed, 13 Jan 2021 09:41:39 GMT Received: from b06avi18878370.portsmouth.uk.ibm.com (b06avi18878370.portsmouth.uk.ibm.com [9.149.26.194]) by ppma03ams.nl.ibm.com with ESMTP id 35y447vucc-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 13 Jan 2021 09:41:39 +0000 Received: from d06av23.portsmouth.uk.ibm.com (d06av23.portsmouth.uk.ibm.com [9.149.105.59]) by b06avi18878370.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 10D9fWV830867858 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 13 Jan 2021 09:41:32 GMT Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id D8FC1A4057; Wed, 13 Jan 2021 09:41:36 +0000 (GMT) Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id A7B64A4053; Wed, 13 Jan 2021 09:41:36 +0000 (GMT) Received: from t46lp67.lnxne.boe (unknown [9.152.108.100]) by d06av23.portsmouth.uk.ibm.com (Postfix) with ESMTP; Wed, 13 Jan 2021 09:41:36 +0000 (GMT) From: Janosch Frank To: kvm@vger.kernel.org Cc: borntraeger@de.ibm.com, david@redhat.com, linux-s390@vger.kernel.org, imbrenda@linux.ibm.com Subject: [PATCH 06/14] s390/mm: Provide vmaddr to pmd notification Date: Wed, 13 Jan 2021 09:41:05 +0000 Message-Id: <20210113094113.133668-7-frankja@linux.ibm.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210113094113.133668-1-frankja@linux.ibm.com> References: <20210113094113.133668-1-frankja@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.343,18.0.737 definitions=2021-01-13_03:2021-01-13,2021-01-13 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 bulkscore=0 impostorscore=0 adultscore=0 malwarescore=0 phishscore=0 clxscore=1015 priorityscore=1501 spamscore=0 suspectscore=0 mlxlogscore=999 mlxscore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101130056 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org It will be needed for shadow tables. Signed-off-by: Janosch Frank --- arch/s390/mm/gmap.c | 52 +++++++++++++++++++++++---------------------- 1 file changed, 27 insertions(+), 25 deletions(-) diff --git a/arch/s390/mm/gmap.c b/arch/s390/mm/gmap.c index d8c9b295294b..b7199c55f98a 100644 --- a/arch/s390/mm/gmap.c +++ b/arch/s390/mm/gmap.c @@ -530,10 +530,10 @@ void gmap_unlink(struct mm_struct *mm, unsigned long *table, } static void gmap_pmdp_xchg(struct gmap *gmap, pmd_t *old, pmd_t new, - unsigned long gaddr); + unsigned long gaddr, unsigned long vmaddr); static void gmap_pmd_split(struct gmap *gmap, unsigned long gaddr, - pmd_t *pmdp, struct page *page); + unsigned long vmaddr, pmd_t *pmdp, struct page *page); /** * gmap_link - set up shadow page tables to connect a host to a guest address @@ -632,7 +632,8 @@ int __gmap_link(struct gmap *gmap, unsigned long gaddr, unsigned long vmaddr) } else if (*table & _SEGMENT_ENTRY_PROTECT && !(pmd_val(*pmd) & _SEGMENT_ENTRY_PROTECT)) { if (page) { - gmap_pmd_split(gmap, gaddr, (pmd_t *)table, page); + gmap_pmd_split(gmap, gaddr, vmaddr, + (pmd_t *)table, page); page = NULL; } else { spin_unlock(ptl); @@ -952,19 +953,15 @@ static void gmap_pte_op_end(spinlock_t *ptl) * Returns a pointer to the pmd for a guest address, or NULL */ static inline pmd_t *gmap_pmd_op_walk(struct gmap *gmap, unsigned long gaddr, - spinlock_t **ptl) + unsigned long vmaddr, spinlock_t **ptl) { pmd_t *pmdp, *hpmdp; - unsigned long vmaddr; BUG_ON(gmap_is_shadow(gmap)); *ptl = NULL; if (gmap->mm->context.allow_gmap_hpage_1m) { - vmaddr = __gmap_translate(gmap, gaddr); - if (IS_ERR_VALUE(vmaddr)) - return NULL; hpmdp = pmd_alloc_map(gmap->mm, vmaddr); if (!hpmdp) return NULL; @@ -1047,7 +1044,7 @@ static inline void gmap_pmd_split_free(struct gmap *gmap, pmd_t *pmdp) * aren't tracked anywhere else. */ static void gmap_pmd_split(struct gmap *gmap, unsigned long gaddr, - pmd_t *pmdp, struct page *page) + unsigned long vmaddr, pmd_t *pmdp, struct page *page) { unsigned long *ptable = (unsigned long *) page_to_phys(page); pmd_t new; @@ -1069,7 +1066,7 @@ static void gmap_pmd_split(struct gmap *gmap, unsigned long gaddr, spin_lock(&gmap->split_list_lock); list_add(&page->lru, &gmap->split_list); spin_unlock(&gmap->split_list_lock); - gmap_pmdp_xchg(gmap, pmdp, new, gaddr); + gmap_pmdp_xchg(gmap, pmdp, new, gaddr, vmaddr); } /* @@ -1087,7 +1084,8 @@ static void gmap_pmd_split(struct gmap *gmap, unsigned long gaddr, * guest_table_lock held. */ static int gmap_protect_pmd(struct gmap *gmap, unsigned long gaddr, - pmd_t *pmdp, int prot, unsigned long bits) + unsigned long vmaddr, pmd_t *pmdp, int prot, + unsigned long bits) { int pmd_i = pmd_val(*pmdp) & _SEGMENT_ENTRY_INVALID; int pmd_p = pmd_val(*pmdp) & _SEGMENT_ENTRY_PROTECT; @@ -1099,13 +1097,13 @@ static int gmap_protect_pmd(struct gmap *gmap, unsigned long gaddr, if (prot == PROT_NONE && !pmd_i) { pmd_val(new) |= _SEGMENT_ENTRY_INVALID; - gmap_pmdp_xchg(gmap, pmdp, new, gaddr); + gmap_pmdp_xchg(gmap, pmdp, new, gaddr, vmaddr); } if (prot == PROT_READ && !pmd_p) { pmd_val(new) &= ~_SEGMENT_ENTRY_INVALID; pmd_val(new) |= _SEGMENT_ENTRY_PROTECT; - gmap_pmdp_xchg(gmap, pmdp, new, gaddr); + gmap_pmdp_xchg(gmap, pmdp, new, gaddr, vmaddr); } if (bits & GMAP_NOTIFY_MPROT) @@ -1168,10 +1166,14 @@ static int gmap_protect_range(struct gmap *gmap, unsigned long gaddr, int rc; BUG_ON(gmap_is_shadow(gmap)); + while (len) { rc = -EAGAIN; - - pmdp = gmap_pmd_op_walk(gmap, gaddr, &ptl_pmd); + vmaddr = __gmap_translate(gmap, gaddr); + if (IS_ERR_VALUE(vmaddr)) + return vmaddr; + vmaddr |= gaddr & ~PMD_MASK; + pmdp = gmap_pmd_op_walk(gmap, gaddr, vmaddr, &ptl_pmd); if (pmdp && !(pmd_val(*pmdp) & _SEGMENT_ENTRY_INVALID)) { if (!pmd_large(*pmdp)) { ptep = gmap_pte_from_pmd(gmap, pmdp, gaddr, @@ -1196,7 +1198,7 @@ static int gmap_protect_range(struct gmap *gmap, unsigned long gaddr, return -ENOMEM; continue; } else { - gmap_pmd_split(gmap, gaddr, + gmap_pmd_split(gmap, gaddr, vmaddr, pmdp, page); page = NULL; gmap_pmd_op_end(ptl_pmd); @@ -1214,9 +1216,6 @@ static int gmap_protect_range(struct gmap *gmap, unsigned long gaddr, return rc; /* -EAGAIN, fixup of userspace mm and gmap */ - vmaddr = __gmap_translate(gmap, gaddr); - if (IS_ERR_VALUE(vmaddr)) - return vmaddr; rc = gmap_fixup(gmap, gaddr, vmaddr, prot); if (rc) return rc; @@ -2441,6 +2440,7 @@ static inline void pmdp_notify_split(struct gmap *gmap, pmd_t *pmdp, static void pmdp_notify_gmap(struct gmap *gmap, pmd_t *pmdp, unsigned long gaddr, unsigned long vmaddr) { + BUG_ON((gaddr & ~HPAGE_MASK) || (vmaddr & ~HPAGE_MASK)); if (gmap_pmd_is_split(pmdp)) return pmdp_notify_split(gmap, pmdp, gaddr, vmaddr); @@ -2461,10 +2461,11 @@ static void pmdp_notify_gmap(struct gmap *gmap, pmd_t *pmdp, * held. */ static void gmap_pmdp_xchg(struct gmap *gmap, pmd_t *pmdp, pmd_t new, - unsigned long gaddr) + unsigned long gaddr, unsigned long vmaddr) { gaddr &= HPAGE_MASK; - pmdp_notify_gmap(gmap, pmdp, gaddr, 0); + vmaddr &= HPAGE_MASK; + pmdp_notify_gmap(gmap, pmdp, gaddr, vmaddr); if (pmd_large(new)) pmd_val(new) &= ~GMAP_SEGMENT_NOTIFY_BITS; if (MACHINE_HAS_TLB_GUEST) @@ -2612,7 +2613,8 @@ EXPORT_SYMBOL_GPL(gmap_pmdp_idte_global); * held. */ static bool gmap_test_and_clear_dirty_pmd(struct gmap *gmap, pmd_t *pmdp, - unsigned long gaddr) + unsigned long gaddr, + unsigned long vmaddr) { if (pmd_val(*pmdp) & _SEGMENT_ENTRY_INVALID) return false; @@ -2624,7 +2626,7 @@ static bool gmap_test_and_clear_dirty_pmd(struct gmap *gmap, pmd_t *pmdp, /* Clear UC indication and reset protection */ pmd_val(*pmdp) &= ~_SEGMENT_ENTRY_GMAP_UC; - gmap_protect_pmd(gmap, gaddr, pmdp, PROT_READ, 0); + gmap_protect_pmd(gmap, gaddr, vmaddr, pmdp, PROT_READ, 0); return true; } @@ -2647,12 +2649,12 @@ void gmap_sync_dirty_log_pmd(struct gmap *gmap, unsigned long bitmap[4], spinlock_t *ptl_pmd = NULL; spinlock_t *ptl_pte = NULL; - pmdp = gmap_pmd_op_walk(gmap, gaddr, &ptl_pmd); + pmdp = gmap_pmd_op_walk(gmap, gaddr, vmaddr, &ptl_pmd); if (!pmdp) return; if (pmd_large(*pmdp)) { - if (gmap_test_and_clear_dirty_pmd(gmap, pmdp, gaddr)) + if (gmap_test_and_clear_dirty_pmd(gmap, pmdp, gaddr, vmaddr)) bitmap_fill(bitmap, _PAGE_ENTRIES); } else { for (i = 0; i < _PAGE_ENTRIES; i++, vmaddr += PAGE_SIZE) { From patchwork Wed Jan 13 09:41:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Janosch Frank X-Patchwork-Id: 12016305 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 40E60C4332E for ; Wed, 13 Jan 2021 09:42:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 10860233F6 for ; Wed, 13 Jan 2021 09:42:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727415AbhAMJm2 (ORCPT ); Wed, 13 Jan 2021 04:42:28 -0500 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:16572 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727356AbhAMJmX (ORCPT ); Wed, 13 Jan 2021 04:42:23 -0500 Received: from pps.filterd (m0098410.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 10D9Y2N9096463; Wed, 13 Jan 2021 04:41:42 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=QvQGci/KkvJd2PvpbMksvlvyanuPqd0JO7N+F1gjXCk=; b=bvcD1au5qnfzBifVGMgpqsfCZ66gxnIZ0ZBeFaavPJu7nCuQaPp5bVlYnaRylsovHbSF Pd6KypKpuIZm4WXvngI7c8st4I3bGqemp25YMM6uG+IoHFbon3r+67BKU06q7c8qF5XD YBp+os8glkwHYWCLgbBDcZ3VoHcTU0e9fU0tIbSou6Zl6YUkeTXFZJuFMsopJl5qmgV5 FHU93AZfMOMx/6xcKZNKzii5zEpo0tX5QmluvS1OwjMYIBlQrrxHjyPaARArddfifzNn rQZKcJerQ8CHcJHTYMa2zb45vqZGzO2ddDGrTdqXnNe37bF4Rgx1KhJpGzrh05pAsv0M mw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 361vwcu004-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 13 Jan 2021 04:41:42 -0500 Received: from m0098410.ppops.net (m0098410.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 10D9Y3Mc096503; Wed, 13 Jan 2021 04:41:42 -0500 Received: from ppma06ams.nl.ibm.com (66.31.33a9.ip4.static.sl-reverse.com [169.51.49.102]) by mx0a-001b2d01.pphosted.com with ESMTP id 361vwctyy9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 13 Jan 2021 04:41:42 -0500 Received: from pps.filterd (ppma06ams.nl.ibm.com [127.0.0.1]) by ppma06ams.nl.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 10D9SO87007869; Wed, 13 Jan 2021 09:41:40 GMT Received: from b06avi18626390.portsmouth.uk.ibm.com (b06avi18626390.portsmouth.uk.ibm.com [9.149.26.192]) by ppma06ams.nl.ibm.com with ESMTP id 35ydrdcej0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 13 Jan 2021 09:41:40 +0000 Received: from d06av23.portsmouth.uk.ibm.com (d06av23.portsmouth.uk.ibm.com [9.149.105.59]) by b06avi18626390.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 10D9fWT627918750 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 13 Jan 2021 09:41:32 GMT Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 21C1FA4055; Wed, 13 Jan 2021 09:41:37 +0000 (GMT) Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id E45F4A4040; Wed, 13 Jan 2021 09:41:36 +0000 (GMT) Received: from t46lp67.lnxne.boe (unknown [9.152.108.100]) by d06av23.portsmouth.uk.ibm.com (Postfix) with ESMTP; Wed, 13 Jan 2021 09:41:36 +0000 (GMT) From: Janosch Frank To: kvm@vger.kernel.org Cc: borntraeger@de.ibm.com, david@redhat.com, linux-s390@vger.kernel.org, imbrenda@linux.ibm.com Subject: [PATCH 07/14] s390/mm: factor out idte global flush into gmap_idte_global Date: Wed, 13 Jan 2021 09:41:06 +0000 Message-Id: <20210113094113.133668-8-frankja@linux.ibm.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210113094113.133668-1-frankja@linux.ibm.com> References: <20210113094113.133668-1-frankja@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.343,18.0.737 definitions=2021-01-13_03:2021-01-13,2021-01-13 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxlogscore=956 impostorscore=0 malwarescore=0 clxscore=1015 mlxscore=0 lowpriorityscore=0 adultscore=0 suspectscore=0 phishscore=0 spamscore=0 bulkscore=0 priorityscore=1501 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101130056 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Introduce a function to do a idte global flush on a gmap pmd and remove some code duplication. Signed-off-by: Janosch Frank Reviewed-by: David Hildenbrand --- arch/s390/mm/gmap.c | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/arch/s390/mm/gmap.c b/arch/s390/mm/gmap.c index b7199c55f98a..f89e710c31af 100644 --- a/arch/s390/mm/gmap.c +++ b/arch/s390/mm/gmap.c @@ -1009,6 +1009,18 @@ static pte_t *gmap_pte_from_pmd(struct gmap *gmap, pmd_t *pmdp, return pte_offset_map(pmdp, addr); } +static inline void gmap_idte_global(unsigned long asce, pmd_t *pmdp, + unsigned long gaddr) +{ + if (MACHINE_HAS_TLB_GUEST) + __pmdp_idte(gaddr, pmdp, IDTE_GUEST_ASCE, asce, + IDTE_GLOBAL); + else if (MACHINE_HAS_IDTE) + __pmdp_idte(gaddr, pmdp, 0, 0, IDTE_GLOBAL); + else + __pmdp_csp(pmdp); +} + /** * gmap_pmd_split_free - Free a split pmd's page table * @pmdp The split pmd that we free of its page table @@ -2468,13 +2480,7 @@ static void gmap_pmdp_xchg(struct gmap *gmap, pmd_t *pmdp, pmd_t new, pmdp_notify_gmap(gmap, pmdp, gaddr, vmaddr); if (pmd_large(new)) pmd_val(new) &= ~GMAP_SEGMENT_NOTIFY_BITS; - if (MACHINE_HAS_TLB_GUEST) - __pmdp_idte(gaddr, pmdp, IDTE_GUEST_ASCE, gmap->asce, - IDTE_GLOBAL); - else if (MACHINE_HAS_IDTE) - __pmdp_idte(gaddr, pmdp, 0, 0, IDTE_GLOBAL); - else - __pmdp_csp(pmdp); + gmap_idte_global(gmap->asce, pmdp, gaddr); *pmdp = new; } @@ -2587,13 +2593,7 @@ void gmap_pmdp_idte_global(struct mm_struct *mm, unsigned long vmaddr) pmdp_notify_gmap(gmap, pmdp, gaddr, vmaddr); if (pmd_large(*pmdp)) WARN_ON(*entry & GMAP_SEGMENT_NOTIFY_BITS); - if (MACHINE_HAS_TLB_GUEST) - __pmdp_idte(gaddr, pmdp, IDTE_GUEST_ASCE, - gmap->asce, IDTE_GLOBAL); - else if (MACHINE_HAS_IDTE) - __pmdp_idte(gaddr, pmdp, 0, 0, IDTE_GLOBAL); - else - __pmdp_csp(pmdp); + gmap_idte_global(gmap->asce, pmdp, gaddr); gmap_pmd_split_free(gmap, pmdp); *entry = _SEGMENT_ENTRY_EMPTY; } From patchwork Wed Jan 13 09:41:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Janosch Frank X-Patchwork-Id: 12016317 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 58E16C433E0 for ; Wed, 13 Jan 2021 09:42:43 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 21537233F8 for ; Wed, 13 Jan 2021 09:42:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727473AbhAMJmf (ORCPT ); Wed, 13 Jan 2021 04:42:35 -0500 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:61392 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727376AbhAMJmY (ORCPT ); Wed, 13 Jan 2021 04:42:24 -0500 Received: from pps.filterd (m0098393.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 10D9Z8HY108770; Wed, 13 Jan 2021 04:41:43 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=k3OplJUc4AVcnR5ntOrQ76a0rAywIkaQOUKsli+701U=; b=Ixi4hB0RujVUNy/hubD6BaqG8u1U3vdwV9uZs6fs/BioJe6B5s0gO3Vn5sn11F3c09R/ Scdoh7UTIDGK76336RoZXzPfufnJPLNSF94tBJmeVcbF4uR0tPhH8WfG0u8KHJuQQWrk IIGjzV12pI3QLiBuxRYme4K1R//wPkrJ/5owqaEXBoS2iQDZmTJ//2qT12JwO7A/Rf+0 YyEJtDpFF3GVYQQvxv9bgPrrx63s3VGXFEbepGcSoxEU87td5TEnTrkGC2ff2L4YOHHi tE+D/SNQ05LjlCiHUPT9xrdsbfC4p5po0c0EfDRd0d8vV7ojLNPrQhuBDzTZffGvqwpr hA== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 361vq9bb9j-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 13 Jan 2021 04:41:43 -0500 Received: from m0098393.ppops.net (m0098393.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 10D9ZJl8109485; Wed, 13 Jan 2021 04:41:42 -0500 Received: from ppma02fra.de.ibm.com (47.49.7a9f.ip4.static.sl-reverse.com [159.122.73.71]) by mx0a-001b2d01.pphosted.com with ESMTP id 361vq9bb8c-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 13 Jan 2021 04:41:42 -0500 Received: from pps.filterd (ppma02fra.de.ibm.com [127.0.0.1]) by ppma02fra.de.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 10D9SUQu009883; Wed, 13 Jan 2021 09:41:40 GMT Received: from b06avi18626390.portsmouth.uk.ibm.com (b06avi18626390.portsmouth.uk.ibm.com [9.149.26.192]) by ppma02fra.de.ibm.com with ESMTP id 35y448jg08-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 13 Jan 2021 09:41:40 +0000 Received: from d06av23.portsmouth.uk.ibm.com (d06av23.portsmouth.uk.ibm.com [9.149.105.59]) by b06avi18626390.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 10D9fWkf29360446 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 13 Jan 2021 09:41:32 GMT Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 54C21A405B; Wed, 13 Jan 2021 09:41:37 +0000 (GMT) Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 2D230A405D; Wed, 13 Jan 2021 09:41:37 +0000 (GMT) Received: from t46lp67.lnxne.boe (unknown [9.152.108.100]) by d06av23.portsmouth.uk.ibm.com (Postfix) with ESMTP; Wed, 13 Jan 2021 09:41:37 +0000 (GMT) From: Janosch Frank To: kvm@vger.kernel.org Cc: borntraeger@de.ibm.com, david@redhat.com, linux-s390@vger.kernel.org, imbrenda@linux.ibm.com Subject: [PATCH 08/14] s390/mm: Make gmap_read_table EDAT1 compatible Date: Wed, 13 Jan 2021 09:41:07 +0000 Message-Id: <20210113094113.133668-9-frankja@linux.ibm.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210113094113.133668-1-frankja@linux.ibm.com> References: <20210113094113.133668-1-frankja@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.343,18.0.737 definitions=2021-01-13_03:2021-01-13,2021-01-13 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 suspectscore=0 spamscore=0 adultscore=0 mlxlogscore=866 lowpriorityscore=0 mlxscore=0 malwarescore=0 priorityscore=1501 impostorscore=0 phishscore=0 bulkscore=0 clxscore=1015 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101130054 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org For the upcoming support of VSIE guests on huge page backed hosts, we need to be able to read from large segments. Signed-off-by: Janosch Frank --- arch/s390/mm/gmap.c | 43 ++++++++++++++++++++++++++----------------- 1 file changed, 26 insertions(+), 17 deletions(-) diff --git a/arch/s390/mm/gmap.c b/arch/s390/mm/gmap.c index f89e710c31af..910371dc511d 100644 --- a/arch/s390/mm/gmap.c +++ b/arch/s390/mm/gmap.c @@ -1282,35 +1282,44 @@ EXPORT_SYMBOL_GPL(gmap_mprotect_notify); int gmap_read_table(struct gmap *gmap, unsigned long gaddr, unsigned long *val) { unsigned long address, vmaddr; - spinlock_t *ptl; + spinlock_t *ptl_pmd = NULL, *ptl_pte = NULL; + pmd_t *pmdp; pte_t *ptep, pte; int rc; - if (gmap_is_shadow(gmap)) - return -EINVAL; + BUG_ON(gmap_is_shadow(gmap)); while (1) { rc = -EAGAIN; - ptep = gmap_pte_op_walk(gmap, gaddr, &ptl); - if (ptep) { - pte = *ptep; - if (pte_present(pte) && (pte_val(pte) & _PAGE_READ)) { - address = pte_val(pte) & PAGE_MASK; - address += gaddr & ~PAGE_MASK; + vmaddr = __gmap_translate(gmap, gaddr); + if (IS_ERR_VALUE(vmaddr)) + return vmaddr; + pmdp = gmap_pmd_op_walk(gmap, gaddr, vmaddr, &ptl_pmd); + if (pmdp && !(pmd_val(*pmdp) & _SEGMENT_ENTRY_INVALID)) { + if (!pmd_large(*pmdp)) { + ptep = gmap_pte_from_pmd(gmap, pmdp, vmaddr, &ptl_pte); + if (ptep) { + pte = *ptep; + if (pte_present(pte) && (pte_val(pte) & _PAGE_READ)) { + address = pte_val(pte) & PAGE_MASK; + address += gaddr & ~PAGE_MASK; + *val = *(unsigned long *) address; + pte_val(*ptep) |= _PAGE_YOUNG; + /* Do *NOT* clear the _PAGE_INVALID bit! */ + rc = 0; + } + } + gmap_pte_op_end(ptl_pte); + } else { + address = pmd_val(*pmdp) & HPAGE_MASK; + address += gaddr & ~HPAGE_MASK; *val = *(unsigned long *) address; - pte_val(*ptep) |= _PAGE_YOUNG; - /* Do *NOT* clear the _PAGE_INVALID bit! */ rc = 0; } - gmap_pte_op_end(ptl); + gmap_pmd_op_end(ptl_pmd); } if (!rc) break; - vmaddr = __gmap_translate(gmap, gaddr); - if (IS_ERR_VALUE(vmaddr)) { - rc = vmaddr; - break; - } rc = gmap_fixup(gmap, gaddr, vmaddr, PROT_READ); if (rc) break; From patchwork Wed Jan 13 09:41:08 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Janosch Frank X-Patchwork-Id: 12016299 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0CFBBC433E9 for ; Wed, 13 Jan 2021 09:42:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A9553233F8 for ; Wed, 13 Jan 2021 09:42:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727425AbhAMJm3 (ORCPT ); Wed, 13 Jan 2021 04:42:29 -0500 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:47162 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727374AbhAMJmY (ORCPT ); Wed, 13 Jan 2021 04:42:24 -0500 Received: from pps.filterd (m0098409.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 10D9XXlm144448; Wed, 13 Jan 2021 04:41:43 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=Ytoz6F8EHZFNjnZPCEMDO45QXNzbmywtLCmbepU7XYc=; b=aSv2Rv0qPq4ChDz5wt+R48l64nBVjjoO1uf6nsRM+DooJpMP9g39jHI2Dr1wpVz7Vxss CxI8Pk9IDEk6NLdS/NEsLgfBzPCzQ11Ad8T8o0Tgn9RAmWk65wtrrJN0FikFxVQhwilQ 2xe/jt+hKNCs/sWzWWikR+I/d6oLs+Mes85MZeklB1LKlbVIIHJEwA9lGb/aeSLaQFHo GlGaXECI7YmBaqCbv75GI2FIdlZo87Ie8srGWz4fgXezgh2VQrAIAkxkZhsrjrflMN0g Ah2I0ptPlqO7k8n1kFDnwCe8n+rVbFipljqLD3k13kAsJfJWvkesdDGAV2ccfeQgzT5y Yg== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 361xb706rw-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 13 Jan 2021 04:41:43 -0500 Received: from m0098409.ppops.net (m0098409.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 10D9XxX1149994; Wed, 13 Jan 2021 04:41:43 -0500 Received: from ppma06ams.nl.ibm.com (66.31.33a9.ip4.static.sl-reverse.com [169.51.49.102]) by mx0a-001b2d01.pphosted.com with ESMTP id 361xb706qw-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 13 Jan 2021 04:41:43 -0500 Received: from pps.filterd (ppma06ams.nl.ibm.com [127.0.0.1]) by ppma06ams.nl.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 10D9SG7T007802; Wed, 13 Jan 2021 09:41:40 GMT Received: from b06avi18626390.portsmouth.uk.ibm.com (b06avi18626390.portsmouth.uk.ibm.com [9.149.26.192]) by ppma06ams.nl.ibm.com with ESMTP id 35ydrdcej1-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 13 Jan 2021 09:41:40 +0000 Received: from d06av23.portsmouth.uk.ibm.com (d06av23.portsmouth.uk.ibm.com [9.149.105.59]) by b06avi18626390.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 10D9fW5Z24641836 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 13 Jan 2021 09:41:32 GMT Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 9AF76A4053; Wed, 13 Jan 2021 09:41:37 +0000 (GMT) Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 6993CA4040; Wed, 13 Jan 2021 09:41:37 +0000 (GMT) Received: from t46lp67.lnxne.boe (unknown [9.152.108.100]) by d06av23.portsmouth.uk.ibm.com (Postfix) with ESMTP; Wed, 13 Jan 2021 09:41:37 +0000 (GMT) From: Janosch Frank To: kvm@vger.kernel.org Cc: borntraeger@de.ibm.com, david@redhat.com, linux-s390@vger.kernel.org, imbrenda@linux.ibm.com Subject: [PATCH 09/14] s390/mm: Make gmap_protect_rmap EDAT1 compatible Date: Wed, 13 Jan 2021 09:41:08 +0000 Message-Id: <20210113094113.133668-10-frankja@linux.ibm.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210113094113.133668-1-frankja@linux.ibm.com> References: <20210113094113.133668-1-frankja@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.343,18.0.737 definitions=2021-01-13_03:2021-01-13,2021-01-13 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 lowpriorityscore=0 suspectscore=0 bulkscore=0 priorityscore=1501 malwarescore=0 mlxscore=0 phishscore=0 clxscore=1015 impostorscore=0 adultscore=0 spamscore=0 mlxlogscore=825 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101130056 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org For the upcoming large page shadowing support, let's add the possibility to split a huge page and protect it with gmap_protect_rmap() for shadowing purposes. Signed-off-by: Janosch Frank --- arch/s390/mm/gmap.c | 93 +++++++++++++++++++++++++++++++++++---------- 1 file changed, 73 insertions(+), 20 deletions(-) diff --git a/arch/s390/mm/gmap.c b/arch/s390/mm/gmap.c index 910371dc511d..f20aa49c2791 100644 --- a/arch/s390/mm/gmap.c +++ b/arch/s390/mm/gmap.c @@ -1142,7 +1142,8 @@ static int gmap_protect_pmd(struct gmap *gmap, unsigned long gaddr, * Expected to be called with sg->mm->mmap_lock in read */ static int gmap_protect_pte(struct gmap *gmap, unsigned long gaddr, - pte_t *ptep, int prot, unsigned long bits) + unsigned long vmaddr, pte_t *ptep, + int prot, unsigned long bits) { int rc; unsigned long pbits = 0; @@ -1191,7 +1192,7 @@ static int gmap_protect_range(struct gmap *gmap, unsigned long gaddr, ptep = gmap_pte_from_pmd(gmap, pmdp, gaddr, &ptl_pte); if (ptep) - rc = gmap_protect_pte(gmap, gaddr, + rc = gmap_protect_pte(gmap, gaddr, vmaddr, ptep, prot, bits); else rc = -ENOMEM; @@ -1354,6 +1355,21 @@ static inline void gmap_insert_rmap(struct gmap *sg, unsigned long vmaddr, } } +static int gmap_protect_rmap_pte(struct gmap *sg, struct gmap_rmap *rmap, + unsigned long paddr, unsigned long vmaddr, + pte_t *ptep, int prot) +{ + int rc = 0; + + spin_lock(&sg->guest_table_lock); + rc = gmap_protect_pte(sg->parent, paddr, vmaddr, ptep, + prot, GMAP_NOTIFY_SHADOW); + if (!rc) + gmap_insert_rmap(sg, vmaddr, rmap); + spin_unlock(&sg->guest_table_lock); + return rc; +} + /** * gmap_protect_rmap - restrict access rights to memory (RO) and create an rmap * @sg: pointer to the shadow guest address space structure @@ -1370,16 +1386,15 @@ static int gmap_protect_rmap(struct gmap *sg, unsigned long raddr, struct gmap *parent; struct gmap_rmap *rmap; unsigned long vmaddr; - spinlock_t *ptl; + pmd_t *pmdp; pte_t *ptep; + spinlock_t *ptl_pmd = NULL, *ptl_pte = NULL; + struct page *page = NULL; int rc; BUG_ON(!gmap_is_shadow(sg)); parent = sg->parent; while (len) { - vmaddr = __gmap_translate(parent, paddr); - if (IS_ERR_VALUE(vmaddr)) - return vmaddr; rmap = kzalloc(sizeof(*rmap), GFP_KERNEL_ACCOUNT); if (!rmap) return -ENOMEM; @@ -1390,26 +1405,64 @@ static int gmap_protect_rmap(struct gmap *sg, unsigned long raddr, return rc; } rc = -EAGAIN; - ptep = gmap_pte_op_walk(parent, paddr, &ptl); - if (ptep) { - spin_lock(&sg->guest_table_lock); - rc = ptep_force_prot(parent->mm, paddr, ptep, PROT_READ, - PGSTE_VSIE_BIT); - if (!rc) - gmap_insert_rmap(sg, vmaddr, rmap); - spin_unlock(&sg->guest_table_lock); - gmap_pte_op_end(ptl); + vmaddr = __gmap_translate(parent, paddr); + if (IS_ERR_VALUE(vmaddr)) + return vmaddr; + vmaddr |= paddr & ~PMD_MASK; + pmdp = gmap_pmd_op_walk(parent, paddr, vmaddr, &ptl_pmd); + if (pmdp && !(pmd_val(*pmdp) & _SEGMENT_ENTRY_INVALID)) { + if (!pmd_large(*pmdp)) { + ptl_pte = NULL; + ptep = gmap_pte_from_pmd(parent, pmdp, paddr, + &ptl_pte); + if (ptep) + rc = gmap_protect_rmap_pte(sg, rmap, paddr, + vmaddr, ptep, + PROT_READ); + else + rc = -ENOMEM; + gmap_pte_op_end(ptl_pte); + gmap_pmd_op_end(ptl_pmd); + if (!rc) { + paddr += PAGE_SIZE; + len -= PAGE_SIZE; + radix_tree_preload_end(); + continue; + } + } else { + if (!page) { + /* Drop locks for allocation. */ + gmap_pmd_op_end(ptl_pmd); + ptl_pmd = NULL; + radix_tree_preload_end(); + kfree(rmap); + page = page_table_alloc_pgste(parent->mm); + if (!page) + return -ENOMEM; + continue; + } else { + gmap_pmd_split(parent, paddr, vmaddr, + pmdp, page); + gmap_pmd_op_end(ptl_pmd); + radix_tree_preload_end(); + kfree(rmap); + page = NULL; + continue; + } + + } + } + if (page) { + page_table_free_pgste(page); + page = NULL; } radix_tree_preload_end(); - if (rc) { - kfree(rmap); + kfree(rmap); + if (rc == -EAGAIN) { rc = gmap_fixup(parent, paddr, vmaddr, PROT_READ); if (rc) return rc; - continue; } - paddr += PAGE_SIZE; - len -= PAGE_SIZE; } return 0; } From patchwork Wed Jan 13 09:41:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Janosch Frank X-Patchwork-Id: 12016323 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7E1C1C43217 for ; Wed, 13 Jan 2021 09:42:43 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 4CB56233F6 for ; Wed, 13 Jan 2021 09:42:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727476AbhAMJmg (ORCPT ); Wed, 13 Jan 2021 04:42:36 -0500 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:22108 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727387AbhAMJmY (ORCPT ); Wed, 13 Jan 2021 04:42:24 -0500 Received: from pps.filterd (m0098409.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 10D9XXjU144531; Wed, 13 Jan 2021 04:41:44 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=EbRVPt4q1svCF2UehoIkBKLRJknMNrtbWp1ofIHLWHA=; b=NJgl7jMPTztT7Cv4WtZHJw91IFcK1JbTmClUmlSO/R7kntLRB9dydsQTLVOeqnhrnWST 9/kIlgGl0epmduxeahx2M6DLYuvKhFMtsUANZQHBW7yTwFhyyfBn6mmRKIy5YwR/+EYw gS1aY5mpImGjfPWvIxH0btxBh3OPfM+3foWCmhyW5Rre4jtyGxRrV08TM9KJJ2OxJxhL kGHM7roaYQFFOlUJRg037CBcbRuFsXm4zt+9wbKKHqPdIV3nl0nG460I1Rp6f+OPduO0 bjik1mZVtTkTBikKP5ZkgBBRUwddYoyOPQDEojmPFeFuVxni0SdOXfWGlgG9S4V989f6 cw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 361xb706s5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 13 Jan 2021 04:41:43 -0500 Received: from m0098409.ppops.net (m0098409.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 10D9Z5m2155217; Wed, 13 Jan 2021 04:41:43 -0500 Received: from ppma05fra.de.ibm.com (6c.4a.5195.ip4.static.sl-reverse.com [149.81.74.108]) by mx0a-001b2d01.pphosted.com with ESMTP id 361xb706rb-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 13 Jan 2021 04:41:43 -0500 Received: from pps.filterd (ppma05fra.de.ibm.com [127.0.0.1]) by ppma05fra.de.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 10D9R4nA014211; Wed, 13 Jan 2021 09:41:40 GMT Received: from b06avi18626390.portsmouth.uk.ibm.com (b06avi18626390.portsmouth.uk.ibm.com [9.149.26.192]) by ppma05fra.de.ibm.com with ESMTP id 35y448ag2c-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 13 Jan 2021 09:41:40 +0000 Received: from d06av23.portsmouth.uk.ibm.com (d06av23.portsmouth.uk.ibm.com [9.149.105.59]) by b06avi18626390.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 10D9fXoJ29426162 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 13 Jan 2021 09:41:33 GMT Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id CE76EA4055; Wed, 13 Jan 2021 09:41:37 +0000 (GMT) Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id A63DCA404D; Wed, 13 Jan 2021 09:41:37 +0000 (GMT) Received: from t46lp67.lnxne.boe (unknown [9.152.108.100]) by d06av23.portsmouth.uk.ibm.com (Postfix) with ESMTP; Wed, 13 Jan 2021 09:41:37 +0000 (GMT) From: Janosch Frank To: kvm@vger.kernel.org Cc: borntraeger@de.ibm.com, david@redhat.com, linux-s390@vger.kernel.org, imbrenda@linux.ibm.com Subject: [PATCH 10/14] s390/mm: Add simple ptep shadow function Date: Wed, 13 Jan 2021 09:41:09 +0000 Message-Id: <20210113094113.133668-11-frankja@linux.ibm.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210113094113.133668-1-frankja@linux.ibm.com> References: <20210113094113.133668-1-frankja@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.343,18.0.737 definitions=2021-01-13_03:2021-01-13,2021-01-13 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 lowpriorityscore=0 suspectscore=0 bulkscore=0 priorityscore=1501 malwarescore=0 mlxscore=0 phishscore=0 clxscore=1015 impostorscore=0 adultscore=0 spamscore=0 mlxlogscore=860 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101130056 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Let's factor out setting the shadow pte, so we can reuse that function for later huge to 4k shadows where we don't have a spte or spgste. Signed-off-by: Janosch Frank --- arch/s390/include/asm/pgtable.h | 1 + arch/s390/mm/pgtable.c | 24 ++++++++++++++++-------- 2 files changed, 17 insertions(+), 8 deletions(-) diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtable.h index 6d6ad508f9c7..d2c005f35c9c 100644 --- a/arch/s390/include/asm/pgtable.h +++ b/arch/s390/include/asm/pgtable.h @@ -1145,6 +1145,7 @@ void ptep_zap_unused(struct mm_struct *mm, unsigned long addr, void ptep_zap_key(struct mm_struct *mm, unsigned long addr, pte_t *ptep); int ptep_shadow_pte(struct mm_struct *mm, unsigned long saddr, pte_t *sptep, pte_t *tptep, pte_t pte); +void ptep_shadow_set(pte_t spte, pte_t *tptep, pte_t pte); void ptep_unshadow_pte(struct mm_struct *mm, unsigned long saddr, pte_t *ptep); unsigned long ptep_get_and_clear_notification_bits(pte_t *ptep); diff --git a/arch/s390/mm/pgtable.c b/arch/s390/mm/pgtable.c index 16896f936d32..6066c7ef027a 100644 --- a/arch/s390/mm/pgtable.c +++ b/arch/s390/mm/pgtable.c @@ -645,11 +645,24 @@ int ptep_force_prot(struct mm_struct *mm, unsigned long addr, return 0; } +void ptep_shadow_set(pte_t spte, pte_t *tptep, pte_t pte) +{ + pte_t tpte; + pgste_t tpgste; + + tpgste = pgste_get_lock(tptep); + pte_val(tpte) = (pte_val(spte) & PAGE_MASK) | + (pte_val(pte) & _PAGE_PROTECT); + /* don't touch the storage key - it belongs to parent pgste */ + tpgste = pgste_set_pte(tptep, tpgste, tpte); + pgste_set_unlock(tptep, tpgste); +} + int ptep_shadow_pte(struct mm_struct *mm, unsigned long saddr, pte_t *sptep, pte_t *tptep, pte_t pte) { - pgste_t spgste, tpgste; - pte_t spte, tpte; + pgste_t spgste; + pte_t spte; int rc = -EAGAIN; if (!(pte_val(*tptep) & _PAGE_INVALID)) @@ -660,12 +673,7 @@ int ptep_shadow_pte(struct mm_struct *mm, unsigned long saddr, !((pte_val(spte) & _PAGE_PROTECT) && !(pte_val(pte) & _PAGE_PROTECT))) { pgste_val(spgste) |= PGSTE_VSIE_BIT; - tpgste = pgste_get_lock(tptep); - pte_val(tpte) = (pte_val(spte) & PAGE_MASK) | - (pte_val(pte) & _PAGE_PROTECT); - /* don't touch the storage key - it belongs to parent pgste */ - tpgste = pgste_set_pte(tptep, tpgste, tpte); - pgste_set_unlock(tptep, tpgste); + ptep_shadow_set(spte, tptep, pte); rc = 1; } pgste_set_unlock(sptep, spgste); From patchwork Wed Jan 13 09:41:10 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Janosch Frank X-Patchwork-Id: 12016311 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B10D3C43332 for ; Wed, 13 Jan 2021 09:42:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 64068233F6 for ; Wed, 13 Jan 2021 09:42:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727412AbhAMJmb (ORCPT ); Wed, 13 Jan 2021 04:42:31 -0500 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:26932 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727398AbhAMJm1 (ORCPT ); Wed, 13 Jan 2021 04:42:27 -0500 Received: from pps.filterd (m0098404.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 10D9Xa0Q135311; Wed, 13 Jan 2021 04:41:44 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=Ao8nVR9oGbWjpWhpl+WELlhVaSBXRHDEfqo58IshcAQ=; b=IaEXcqWYhCeSwgdQ+lt59lNYkN0wXcmFgYRkuPNn8kYy32zRm4V8HZPDvthOGISA57ol k71CLkFheBMMAN/IIshQ5vPBWsANykXOpxdt1S8j7Z0f5lLms+9Y1J+PLaj9+/EEj5S1 cprX9wjF+nHM0v1h+G1/TLa9idYnDXoBatIBdjjDD5orD82gdCqjEI8Non/uNZwrpIh8 h+emX0ffetUm2ZldkcdAvWJv1tgW96nhIltCijBI8Kvn3e/oqULWAPp6Uua06Tn9Ls8q huCtbr+cBwpHUrLXq3sGLLCn2mEcGRUUJN3Uda3+MN0kZ28esktIUKXm7pRaB6ZgefyN zQ== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 361uq2vjar-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 13 Jan 2021 04:41:44 -0500 Received: from m0098404.ppops.net (m0098404.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 10D9Y8MM138191; Wed, 13 Jan 2021 04:41:44 -0500 Received: from ppma04ams.nl.ibm.com (63.31.33a9.ip4.static.sl-reverse.com [169.51.49.99]) by mx0a-001b2d01.pphosted.com with ESMTP id 361uq2vj9q-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 13 Jan 2021 04:41:43 -0500 Received: from pps.filterd (ppma04ams.nl.ibm.com [127.0.0.1]) by ppma04ams.nl.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 10D9SLhv001193; Wed, 13 Jan 2021 09:41:41 GMT Received: from b06cxnps3075.portsmouth.uk.ibm.com (d06relay10.portsmouth.uk.ibm.com [9.149.109.195]) by ppma04ams.nl.ibm.com with ESMTP id 35y448cuys-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 13 Jan 2021 09:41:41 +0000 Received: from d06av23.portsmouth.uk.ibm.com (d06av23.portsmouth.uk.ibm.com [9.149.105.59]) by b06cxnps3075.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 10D9fceh48365936 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 13 Jan 2021 09:41:38 GMT Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 2FE5AA4053; Wed, 13 Jan 2021 09:41:38 +0000 (GMT) Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id E3852A4059; Wed, 13 Jan 2021 09:41:37 +0000 (GMT) Received: from t46lp67.lnxne.boe (unknown [9.152.108.100]) by d06av23.portsmouth.uk.ibm.com (Postfix) with ESMTP; Wed, 13 Jan 2021 09:41:37 +0000 (GMT) From: Janosch Frank To: kvm@vger.kernel.org Cc: borntraeger@de.ibm.com, david@redhat.com, linux-s390@vger.kernel.org, imbrenda@linux.ibm.com Subject: [PATCH 11/14] s390/mm: Add gmap shadowing for large pmds Date: Wed, 13 Jan 2021 09:41:10 +0000 Message-Id: <20210113094113.133668-12-frankja@linux.ibm.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210113094113.133668-1-frankja@linux.ibm.com> References: <20210113094113.133668-1-frankja@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.343,18.0.737 definitions=2021-01-13_03:2021-01-13,2021-01-13 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 suspectscore=0 lowpriorityscore=0 priorityscore=1501 clxscore=1015 bulkscore=0 impostorscore=0 mlxlogscore=999 phishscore=0 adultscore=0 spamscore=0 mlxscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101130056 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Up to now we could only shadow large pmds when the parent's mapping was done with normal sized pmds. This is done by introducing fake page tables and effectively running the level 3 guest with a standard memory backing instead of the large one. With this patch we add shadowing when the host is large page backed. This allows us to run normal and large backed VMs inside a large backed host. Signed-off-by: Janosch Frank --- arch/s390/include/asm/gmap.h | 9 +- arch/s390/kvm/gaccess.c | 52 +++++- arch/s390/mm/gmap.c | 328 ++++++++++++++++++++++++++++------- 3 files changed, 317 insertions(+), 72 deletions(-) diff --git a/arch/s390/include/asm/gmap.h b/arch/s390/include/asm/gmap.h index a5711c189018..4133d09597a5 100644 --- a/arch/s390/include/asm/gmap.h +++ b/arch/s390/include/asm/gmap.h @@ -19,11 +19,12 @@ /* Status bits only for huge segment entries */ #define _SEGMENT_ENTRY_GMAP_IN 0x8000 /* invalidation notify bit */ #define _SEGMENT_ENTRY_GMAP_UC 0x4000 /* dirty (migration) */ +#define _SEGMENT_ENTRY_GMAP_VSIE 0x2000 /* vsie bit */ /* Status bits in the gmap segment entry. */ #define _SEGMENT_ENTRY_GMAP_SPLIT 0x0001 /* split huge pmd */ #define GMAP_SEGMENT_STATUS_BITS (_SEGMENT_ENTRY_GMAP_UC | _SEGMENT_ENTRY_GMAP_SPLIT) -#define GMAP_SEGMENT_NOTIFY_BITS _SEGMENT_ENTRY_GMAP_IN +#define GMAP_SEGMENT_NOTIFY_BITS (_SEGMENT_ENTRY_GMAP_IN | _SEGMENT_ENTRY_GMAP_VSIE) /** * struct gmap_struct - guest address space @@ -152,9 +153,11 @@ int gmap_shadow_sgt(struct gmap *sg, unsigned long saddr, unsigned long sgt, int fake); int gmap_shadow_pgt(struct gmap *sg, unsigned long saddr, unsigned long pgt, int fake); -int gmap_shadow_pgt_lookup(struct gmap *sg, unsigned long saddr, - unsigned long *pgt, int *dat_protection, int *fake); +int gmap_shadow_sgt_lookup(struct gmap *sg, unsigned long saddr, + unsigned long *pgt, int *dat_protection, + int *fake, int *lvl); int gmap_shadow_page(struct gmap *sg, unsigned long saddr, pte_t pte); +int gmap_shadow_segment(struct gmap *sg, unsigned long saddr, pmd_t pmd); void gmap_register_pte_notifier(struct gmap_notifier *); void gmap_unregister_pte_notifier(struct gmap_notifier *); diff --git a/arch/s390/kvm/gaccess.c b/arch/s390/kvm/gaccess.c index 6d6b57059493..d5f6b5c2c8de 100644 --- a/arch/s390/kvm/gaccess.c +++ b/arch/s390/kvm/gaccess.c @@ -981,7 +981,7 @@ int kvm_s390_check_low_addr_prot_real(struct kvm_vcpu *vcpu, unsigned long gra) */ static int kvm_s390_shadow_tables(struct gmap *sg, unsigned long saddr, unsigned long *pgt, int *dat_protection, - int *fake) + int *fake, int *lvl) { struct gmap *parent; union asce asce; @@ -1133,14 +1133,25 @@ static int kvm_s390_shadow_tables(struct gmap *sg, unsigned long saddr, if (ste.cs && asce.p) return PGM_TRANSLATION_SPEC; *dat_protection |= ste.fc0.p; + + /* Guest is huge page mapped */ if (ste.fc && sg->edat_level >= 1) { - *fake = 1; - ptr = ste.fc1.sfaa * _SEGMENT_SIZE; - ste.val = ptr; - goto shadow_pgt; + /* 4k to 1m, we absolutely need fake shadow tables. */ + if (!parent->mm->context.allow_gmap_hpage_1m) { + *fake = 1; + ptr = ste.fc1.sfaa * _SEGMENT_SIZE; + ste.val = ptr; + goto shadow_pgt; + } else { + *lvl = 1; + *pgt = ptr; + return 0; + + } } ptr = ste.fc0.pto * (PAGE_SIZE / 2); shadow_pgt: + *lvl = 0; ste.fc0.p |= *dat_protection; rc = gmap_shadow_pgt(sg, saddr, ste.val, *fake); if (rc) @@ -1169,8 +1180,9 @@ int kvm_s390_shadow_fault(struct kvm_vcpu *vcpu, struct gmap *sg, { union vaddress vaddr; union page_table_entry pte; + union segment_table_entry ste; unsigned long pgt; - int dat_protection, fake; + int dat_protection, fake, lvl = 0; int rc; mmap_read_lock(sg->mm); @@ -1181,12 +1193,35 @@ int kvm_s390_shadow_fault(struct kvm_vcpu *vcpu, struct gmap *sg, */ ipte_lock(vcpu); - rc = gmap_shadow_pgt_lookup(sg, saddr, &pgt, &dat_protection, &fake); + rc = gmap_shadow_sgt_lookup(sg, saddr, &pgt, &dat_protection, &fake, &lvl); if (rc) rc = kvm_s390_shadow_tables(sg, saddr, &pgt, &dat_protection, - &fake); + &fake, &lvl); vaddr.addr = saddr; + + /* Shadow stopped at segment level, we map pmd to pmd */ + if (!rc && lvl) { + rc = gmap_read_table(sg->parent, pgt + vaddr.sx * 8, &ste.val); + if (!rc && ste.i) + rc = PGM_PAGE_TRANSLATION; + ste.fc1.p |= dat_protection; + if (!rc) + rc = gmap_shadow_segment(sg, saddr, __pmd(ste.val)); + if (rc == -EISDIR) { + /* Hit a split pmd, we need to setup a fake page table */ + fake = 1; + pgt = ste.fc1.sfaa * _SEGMENT_SIZE; + ste.val = pgt; + rc = gmap_shadow_pgt(sg, saddr, ste.val, fake); + if (rc) + goto out; + } else { + /* We're done */ + goto out; + } + } + if (fake) { pte.val = pgt + vaddr.px * PAGE_SIZE; goto shadow_page; @@ -1201,6 +1236,7 @@ int kvm_s390_shadow_fault(struct kvm_vcpu *vcpu, struct gmap *sg, pte.p |= dat_protection; if (!rc) rc = gmap_shadow_page(sg, saddr, __pte(pte.val)); +out: ipte_unlock(vcpu); mmap_read_unlock(sg->mm); return rc; diff --git a/arch/s390/mm/gmap.c b/arch/s390/mm/gmap.c index f20aa49c2791..50dd95946d32 100644 --- a/arch/s390/mm/gmap.c +++ b/arch/s390/mm/gmap.c @@ -883,28 +883,6 @@ static inline unsigned long *gmap_table_walk(struct gmap *gmap, return table; } -/** - * gmap_pte_op_walk - walk the gmap page table, get the page table lock - * and return the pte pointer - * @gmap: pointer to guest mapping meta data structure - * @gaddr: virtual address in the guest address space - * @ptl: pointer to the spinlock pointer - * - * Returns a pointer to the locked pte for a guest address, or NULL - */ -static pte_t *gmap_pte_op_walk(struct gmap *gmap, unsigned long gaddr, - spinlock_t **ptl) -{ - unsigned long *table; - - BUG_ON(gmap_is_shadow(gmap)); - /* Walk the gmap page table, lock and get pte pointer */ - table = gmap_table_walk(gmap, gaddr, 1); /* get segment pointer */ - if (!table || *table & _SEGMENT_ENTRY_INVALID) - return NULL; - return pte_alloc_map_lock(gmap->mm, (pmd_t *) table, gaddr, ptl); -} - /** * gmap_fixup - force memory in and connect the gmap table entry * @gmap: pointer to guest mapping meta data structure @@ -1468,6 +1446,7 @@ static int gmap_protect_rmap(struct gmap *sg, unsigned long raddr, } #define _SHADOW_RMAP_MASK 0x7 +#define _SHADOW_RMAP_SEGMENT_LP 0x6 #define _SHADOW_RMAP_REGION1 0x5 #define _SHADOW_RMAP_REGION2 0x4 #define _SHADOW_RMAP_REGION3 0x3 @@ -1573,15 +1552,18 @@ static void __gmap_unshadow_sgt(struct gmap *sg, unsigned long raddr, BUG_ON(!gmap_is_shadow(sg)); for (i = 0; i < _CRST_ENTRIES; i++, raddr += _SEGMENT_SIZE) { - if (!(sgt[i] & _SEGMENT_ENTRY_ORIGIN)) + if (sgt[i] == _SEGMENT_ENTRY_EMPTY) continue; - pgt = (unsigned long *)(sgt[i] & _REGION_ENTRY_ORIGIN); + + if (!(sgt[i] & _SEGMENT_ENTRY_LARGE)) { + pgt = (unsigned long *)(sgt[i] & _SEGMENT_ENTRY_ORIGIN); + __gmap_unshadow_pgt(sg, raddr, pgt); + /* Free page table */ + page = pfn_to_page(__pa(pgt) >> PAGE_SHIFT); + list_del(&page->lru); + page_table_free_pgste(page); + } sgt[i] = _SEGMENT_ENTRY_EMPTY; - __gmap_unshadow_pgt(sg, raddr, pgt); - /* Free page table */ - page = pfn_to_page(__pa(pgt) >> PAGE_SHIFT); - list_del(&page->lru); - page_table_free_pgste(page); } } @@ -2188,7 +2170,7 @@ EXPORT_SYMBOL_GPL(gmap_shadow_sgt); /** * gmap_shadow_lookup_pgtable - find a shadow page table * @sg: pointer to the shadow guest address space structure - * @saddr: the address in the shadow aguest address space + * @saddr: the address in the shadow guest address space * @pgt: parent gmap address of the page table to get shadowed * @dat_protection: if the pgtable is marked as protected by dat * @fake: pgt references contiguous guest memory block, not a pgtable @@ -2198,32 +2180,64 @@ EXPORT_SYMBOL_GPL(gmap_shadow_sgt); * * Called with sg->mm->mmap_lock in read. */ -int gmap_shadow_pgt_lookup(struct gmap *sg, unsigned long saddr, - unsigned long *pgt, int *dat_protection, - int *fake) +void gmap_shadow_pgt_lookup(struct gmap *sg, unsigned long *sge, + unsigned long saddr, unsigned long *pgt, + int *dat_protection, int *fake) { - unsigned long *table; struct page *page; - int rc; + + /* Shadow page tables are full pages (pte+pgste) */ + page = pfn_to_page(*sge >> PAGE_SHIFT); + *pgt = page->index & ~GMAP_SHADOW_FAKE_TABLE; + *dat_protection = !!(*sge & _SEGMENT_ENTRY_PROTECT); + *fake = !!(page->index & GMAP_SHADOW_FAKE_TABLE); +} +EXPORT_SYMBOL_GPL(gmap_shadow_pgt_lookup); + +int gmap_shadow_sgt_lookup(struct gmap *sg, unsigned long saddr, + unsigned long *pgt, int *dat_protection, + int *fake, int *lvl) +{ + unsigned long *sge, *r3e = NULL; + struct page *page; + int rc = -EAGAIN; BUG_ON(!gmap_is_shadow(sg)); spin_lock(&sg->guest_table_lock); - table = gmap_table_walk(sg, saddr, 1); /* get segment pointer */ - if (table && !(*table & _SEGMENT_ENTRY_INVALID)) { - /* Shadow page tables are full pages (pte+pgste) */ - page = pfn_to_page(*table >> PAGE_SHIFT); - *pgt = page->index & ~GMAP_SHADOW_FAKE_TABLE; - *dat_protection = !!(*table & _SEGMENT_ENTRY_PROTECT); - *fake = !!(page->index & GMAP_SHADOW_FAKE_TABLE); - rc = 0; - } else { - rc = -EAGAIN; + if (sg->asce & _ASCE_TYPE_MASK) { + /* >2 GB guest */ + r3e = (unsigned long *) gmap_table_walk(sg, saddr, 2); + if (!r3e || (*r3e & _REGION_ENTRY_INVALID)) + goto out; + sge = (unsigned long *)(*r3e & _REGION_ENTRY_ORIGIN) + ((saddr & _SEGMENT_INDEX) >> _SEGMENT_SHIFT); + } else { + sge = (unsigned long *)(sg->asce & PAGE_MASK) + ((saddr & _SEGMENT_INDEX) >> _SEGMENT_SHIFT); } + if (*sge & _SEGMENT_ENTRY_INVALID) + goto out; + rc = 0; + if (*sge & _SEGMENT_ENTRY_LARGE) { + if (r3e) { + page = pfn_to_page(*r3e >> PAGE_SHIFT); + *pgt = page->index & ~GMAP_SHADOW_FAKE_TABLE; + *dat_protection = !!(*r3e & _SEGMENT_ENTRY_PROTECT); + *fake = !!(page->index & GMAP_SHADOW_FAKE_TABLE); + } else { + *pgt = sg->orig_asce & PAGE_MASK; + *dat_protection = 0; + *fake = 0; + } + *lvl = 1; + } else { + gmap_shadow_pgt_lookup(sg, sge, saddr, pgt, + dat_protection, fake); + *lvl = 0; + } +out: spin_unlock(&sg->guest_table_lock); return rc; - } -EXPORT_SYMBOL_GPL(gmap_shadow_pgt_lookup); +EXPORT_SYMBOL_GPL(gmap_shadow_sgt_lookup); /** * gmap_shadow_pgt - instantiate a shadow page table @@ -2305,6 +2319,95 @@ int gmap_shadow_pgt(struct gmap *sg, unsigned long saddr, unsigned long pgt, } EXPORT_SYMBOL_GPL(gmap_shadow_pgt); +int gmap_shadow_segment(struct gmap *sg, unsigned long saddr, pmd_t pmd) +{ + struct gmap *parent; + struct gmap_rmap *rmap; + unsigned long vmaddr, paddr; + spinlock_t *ptl = NULL; + pmd_t spmd, tpmd, *spmdp = NULL, *tpmdp; + int prot; + int rc; + + BUG_ON(!gmap_is_shadow(sg)); + parent = sg->parent; + + prot = (pmd_val(pmd) & _SEGMENT_ENTRY_PROTECT) ? PROT_READ : PROT_WRITE; + rmap = kzalloc(sizeof(*rmap), GFP_KERNEL); + if (!rmap) + return -ENOMEM; + rmap->raddr = (saddr & HPAGE_MASK) | _SHADOW_RMAP_SEGMENT_LP; + + while (1) { + paddr = pmd_val(pmd) & HPAGE_MASK; + vmaddr = __gmap_translate(parent, paddr); + if (IS_ERR_VALUE(vmaddr)) { + rc = vmaddr; + break; + } + rc = radix_tree_preload(GFP_KERNEL); + if (rc) + break; + rc = -EAGAIN; + + /* Let's look up the parent's mapping */ + spmdp = gmap_pmd_op_walk(parent, paddr, vmaddr, &ptl); + if (spmdp) { + if (gmap_pmd_is_split(spmdp)) { + gmap_pmd_op_end(ptl); + radix_tree_preload_end(); + rc = -EISDIR; + break; + } + spin_lock(&sg->guest_table_lock); + /* Get shadow segment table pointer */ + tpmdp = (pmd_t *) gmap_table_walk(sg, saddr, 1); + if (!tpmdp) { + spin_unlock(&sg->guest_table_lock); + gmap_pmd_op_end(ptl); + radix_tree_preload_end(); + break; + } + /* Shadowing magic happens here. */ + if (!(pmd_val(*tpmdp) & _SEGMENT_ENTRY_INVALID)) { + rc = 0; /* already shadowed */ + spin_unlock(&sg->guest_table_lock); + gmap_pmd_op_end(ptl); + radix_tree_preload_end(); + kfree(rmap); + break; + } + spmd = *spmdp; + if (!(pmd_val(spmd) & _SEGMENT_ENTRY_INVALID) && + !((pmd_val(spmd) & _SEGMENT_ENTRY_PROTECT) && + !(pmd_val(pmd) & _SEGMENT_ENTRY_PROTECT))) { + + pmd_val(*spmdp) |= _SEGMENT_ENTRY_GMAP_VSIE; + + /* Insert shadow ste */ + pmd_val(tpmd) = ((pmd_val(spmd) & + _SEGMENT_ENTRY_HARDWARE_BITS_LARGE) | + (pmd_val(pmd) & _SEGMENT_ENTRY_PROTECT)); + *tpmdp = tpmd; + gmap_insert_rmap(sg, vmaddr, rmap); + rc = 0; + } + spin_unlock(&sg->guest_table_lock); + gmap_pmd_op_end(ptl); + } + radix_tree_preload_end(); + if (!rc) + break; + rc = gmap_fixup(parent, paddr, vmaddr, prot); + if (rc) + break; + } + if (rc) + kfree(rmap); + return rc; +} +EXPORT_SYMBOL_GPL(gmap_shadow_segment); + /** * gmap_shadow_page - create a shadow page mapping * @sg: pointer to the shadow guest address space structure @@ -2322,7 +2425,8 @@ int gmap_shadow_page(struct gmap *sg, unsigned long saddr, pte_t pte) struct gmap *parent; struct gmap_rmap *rmap; unsigned long vmaddr, paddr; - spinlock_t *ptl; + spinlock_t *ptl_pmd = NULL, *ptl_pte = NULL; + pmd_t *spmdp; pte_t *sptep, *tptep; int prot; int rc; @@ -2347,26 +2451,46 @@ int gmap_shadow_page(struct gmap *sg, unsigned long saddr, pte_t pte) if (rc) break; rc = -EAGAIN; - sptep = gmap_pte_op_walk(parent, paddr, &ptl); - if (sptep) { - spin_lock(&sg->guest_table_lock); + spmdp = gmap_pmd_op_walk(parent, paddr, vmaddr, &ptl_pmd); + if (spmdp && !(pmd_val(*spmdp) & _SEGMENT_ENTRY_INVALID)) { /* Get page table pointer */ tptep = (pte_t *) gmap_table_walk(sg, saddr, 0); if (!tptep) { - spin_unlock(&sg->guest_table_lock); - gmap_pte_op_end(ptl); radix_tree_preload_end(); + gmap_pmd_op_end(ptl_pmd); break; } - rc = ptep_shadow_pte(sg->mm, saddr, sptep, tptep, pte); - if (rc > 0) { - /* Success and a new mapping */ - gmap_insert_rmap(sg, vmaddr, rmap); - rmap = NULL; - rc = 0; + + if (pmd_large(*spmdp)) { + pte_t spte; + if (!(pmd_val(*spmdp) & _SEGMENT_ENTRY_PROTECT)) { + spin_lock(&sg->guest_table_lock); + spte = __pte((pmd_val(*spmdp) & + _SEGMENT_ENTRY_ORIGIN_LARGE) + + (pte_index(paddr) << 12)); + ptep_shadow_set(spte, tptep, pte); + pmd_val(*spmdp) |= _SEGMENT_ENTRY_GMAP_VSIE; + gmap_insert_rmap(sg, vmaddr, rmap); + rmap = NULL; + rc = 0; + spin_unlock(&sg->guest_table_lock); + } + } else { + sptep = gmap_pte_from_pmd(parent, spmdp, paddr, &ptl_pte); + spin_lock(&sg->guest_table_lock); + if (sptep) { + rc = ptep_shadow_pte(sg->mm, saddr, sptep, tptep, pte); + if (rc > 0) { + /* Success and a new mapping */ + gmap_insert_rmap(sg, vmaddr, rmap); + rmap = NULL; + rc = 0; + } + spin_unlock(&sg->guest_table_lock); + gmap_pte_op_end(ptl_pte); + } } - gmap_pte_op_end(ptl); - spin_unlock(&sg->guest_table_lock); + gmap_pmd_op_end(ptl_pmd); } radix_tree_preload_end(); if (!rc) @@ -2380,6 +2504,75 @@ int gmap_shadow_page(struct gmap *sg, unsigned long saddr, pte_t pte) } EXPORT_SYMBOL_GPL(gmap_shadow_page); +/** + * gmap_unshadow_segment - remove a huge segment from a shadow segment table + * @sg: pointer to the shadow guest address space structure + * @raddr: rmap address in the shadow guest address space + * + * Called with the sg->guest_table_lock + */ +static void gmap_unshadow_segment(struct gmap *sg, unsigned long raddr) +{ + unsigned long *table; + + BUG_ON(!gmap_is_shadow(sg)); + /* We already have the lock */ + table = gmap_table_walk(sg, raddr, 1); /* get segment table pointer */ + if (!table || *table & _SEGMENT_ENTRY_INVALID || + !(*table & _SEGMENT_ENTRY_LARGE)) + return; + gmap_call_notifier(sg, raddr, raddr + HPAGE_SIZE - 1); + gmap_idte_global(sg->asce, (pmd_t *)table, raddr); + *table = _SEGMENT_ENTRY_EMPTY; +} + +static void gmap_shadow_notify_pmd(struct gmap *sg, unsigned long vmaddr, + unsigned long gaddr) +{ + struct gmap_rmap *rmap, *rnext, *head; + unsigned long start, end, bits, raddr; + + + BUG_ON(!gmap_is_shadow(sg)); + + spin_lock(&sg->guest_table_lock); + if (sg->removed) { + spin_unlock(&sg->guest_table_lock); + return; + } + /* Check for top level table */ + start = sg->orig_asce & _ASCE_ORIGIN; + end = start + ((sg->orig_asce & _ASCE_TABLE_LENGTH) + 1) * PAGE_SIZE; + if (!(sg->orig_asce & _ASCE_REAL_SPACE) && gaddr >= start && + gaddr < ((end & HPAGE_MASK) + HPAGE_SIZE - 1)) { + /* The complete shadow table has to go */ + gmap_unshadow(sg); + spin_unlock(&sg->guest_table_lock); + list_del(&sg->list); + gmap_put(sg); + return; + } + /* Remove the page table tree from on specific entry */ + head = radix_tree_delete(&sg->host_to_rmap, (vmaddr & HPAGE_MASK) >> PAGE_SHIFT); + gmap_for_each_rmap_safe(rmap, rnext, head) { + bits = rmap->raddr & _SHADOW_RMAP_MASK; + raddr = rmap->raddr ^ bits; + switch (bits) { + case _SHADOW_RMAP_SEGMENT_LP: + gmap_unshadow_segment(sg, raddr); + break; + case _SHADOW_RMAP_PGTABLE: + gmap_unshadow_page(sg, raddr); + break; + default: + BUG(); + } + kfree(rmap); + } + spin_unlock(&sg->guest_table_lock); +} + + /** * gmap_shadow_notify - handle notifications for shadow gmap * @@ -2431,6 +2624,8 @@ static void gmap_shadow_notify(struct gmap *sg, unsigned long vmaddr, case _SHADOW_RMAP_PGTABLE: gmap_unshadow_page(sg, raddr); break; + default: + BUG(); } kfree(rmap); } @@ -2514,10 +2709,21 @@ static inline void pmdp_notify_split(struct gmap *gmap, pmd_t *pmdp, static void pmdp_notify_gmap(struct gmap *gmap, pmd_t *pmdp, unsigned long gaddr, unsigned long vmaddr) { + struct gmap *sg, *next; + BUG_ON((gaddr & ~HPAGE_MASK) || (vmaddr & ~HPAGE_MASK)); if (gmap_pmd_is_split(pmdp)) return pmdp_notify_split(gmap, pmdp, gaddr, vmaddr); + if (!list_empty(&gmap->children) && + (pmd_val(*pmdp) & _SEGMENT_ENTRY_GMAP_VSIE)) { + spin_lock(&gmap->shadow_lock); + list_for_each_entry_safe(sg, next, &gmap->children, list) + gmap_shadow_notify_pmd(sg, vmaddr, gaddr); + spin_unlock(&gmap->shadow_lock); + } + pmd_val(*pmdp) &= ~_SEGMENT_ENTRY_GMAP_VSIE; + if (!(pmd_val(*pmdp) & _SEGMENT_ENTRY_GMAP_IN)) return; pmd_val(*pmdp) &= ~_SEGMENT_ENTRY_GMAP_IN; From patchwork Wed Jan 13 09:41:11 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Janosch Frank X-Patchwork-Id: 12016307 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7AFEBC43331 for ; Wed, 13 Jan 2021 09:42:42 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 3DFBA233F8 for ; Wed, 13 Jan 2021 09:42:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727430AbhAMJmb (ORCPT ); Wed, 13 Jan 2021 04:42:31 -0500 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:54050 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727394AbhAMJm0 (ORCPT ); Wed, 13 Jan 2021 04:42:26 -0500 Received: from pps.filterd (m0098399.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 10D9X7Uh026855; Wed, 13 Jan 2021 04:41:44 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=IPdx+oPF5xjDJ8zklp9aYhGjjg90Ic4dpd9q7bs1RHs=; b=OGLfcd4SydN2g5ZSUmi/2OCpba9bs2ZJGwgDy8MhCz1NaqVkXn/8+OZkFmNdritgLM15 koBBw4ZrM/BAlKRPEs6JniPaV20RQIGrDGNXsdZfCED27Kkjc1ibQBQ8n/PhVG8wm4Q9 2qnKtjkMVc+HdTkvfpHc3z8Az2EoRCQsB9M4j/y2QfZIYy8RDWuJMVYMmDhVMsNFlaH7 tENWVpsMEzYH3yN3kOCPumH8ieCFEnXJK4IWY4Sc41CwDEhD7bDy1OKDirPcNEnSq1nm TaXJ4eK4y0W7RV7HUd94ueVym4/5r+zHq8D7fU/vCkw/AKhyoZ+LcsD4+mV2+4kaYhDJ OA== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 361w5sajtg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 13 Jan 2021 04:41:44 -0500 Received: from m0098399.ppops.net (m0098399.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 10D9XWfA028055; Wed, 13 Jan 2021 04:41:44 -0500 Received: from ppma01fra.de.ibm.com (46.49.7a9f.ip4.static.sl-reverse.com [159.122.73.70]) by mx0a-001b2d01.pphosted.com with ESMTP id 361w5sajsm-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 13 Jan 2021 04:41:44 -0500 Received: from pps.filterd (ppma01fra.de.ibm.com [127.0.0.1]) by ppma01fra.de.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 10D9SNvL002604; Wed, 13 Jan 2021 09:41:41 GMT Received: from b06cxnps3075.portsmouth.uk.ibm.com (d06relay10.portsmouth.uk.ibm.com [9.149.109.195]) by ppma01fra.de.ibm.com with ESMTP id 35y4482ger-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 13 Jan 2021 09:41:41 +0000 Received: from d06av23.portsmouth.uk.ibm.com (d06av23.portsmouth.uk.ibm.com [9.149.105.59]) by b06cxnps3075.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 10D9fceR48365940 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 13 Jan 2021 09:41:38 GMT Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 720D7A4051; Wed, 13 Jan 2021 09:41:38 +0000 (GMT) Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 3AE8BA404D; Wed, 13 Jan 2021 09:41:38 +0000 (GMT) Received: from t46lp67.lnxne.boe (unknown [9.152.108.100]) by d06av23.portsmouth.uk.ibm.com (Postfix) with ESMTP; Wed, 13 Jan 2021 09:41:38 +0000 (GMT) From: Janosch Frank To: kvm@vger.kernel.org Cc: borntraeger@de.ibm.com, david@redhat.com, linux-s390@vger.kernel.org, imbrenda@linux.ibm.com Subject: [PATCH 12/14] s390/mm: Add gmap lock classes Date: Wed, 13 Jan 2021 09:41:11 +0000 Message-Id: <20210113094113.133668-13-frankja@linux.ibm.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210113094113.133668-1-frankja@linux.ibm.com> References: <20210113094113.133668-1-frankja@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.343,18.0.737 definitions=2021-01-13_03:2021-01-13,2021-01-13 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 adultscore=0 clxscore=1015 priorityscore=1501 malwarescore=0 suspectscore=0 impostorscore=0 bulkscore=0 mlxscore=0 lowpriorityscore=0 mlxlogscore=999 spamscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101130056 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org A shadow gmap and its parent are locked right after each other when doing VSIE management. Lockdep can't differentiate between the two classes without some help. Signed-off-by: Janosch Frank --- arch/s390/include/asm/gmap.h | 6 ++++++ arch/s390/mm/gmap.c | 40 +++++++++++++++++++----------------- 2 files changed, 27 insertions(+), 19 deletions(-) diff --git a/arch/s390/include/asm/gmap.h b/arch/s390/include/asm/gmap.h index 4133d09597a5..4edbeb012e2a 100644 --- a/arch/s390/include/asm/gmap.h +++ b/arch/s390/include/asm/gmap.h @@ -26,6 +26,12 @@ #define GMAP_SEGMENT_STATUS_BITS (_SEGMENT_ENTRY_GMAP_UC | _SEGMENT_ENTRY_GMAP_SPLIT) #define GMAP_SEGMENT_NOTIFY_BITS (_SEGMENT_ENTRY_GMAP_IN | _SEGMENT_ENTRY_GMAP_VSIE) + +enum gmap_lock_class { + GMAP_LOCK_PARENT, + GMAP_LOCK_SHADOW +}; + /** * struct gmap_struct - guest address space * @list: list head for the mm->context gmap list diff --git a/arch/s390/mm/gmap.c b/arch/s390/mm/gmap.c index 50dd95946d32..bc89fb974367 100644 --- a/arch/s390/mm/gmap.c +++ b/arch/s390/mm/gmap.c @@ -1339,7 +1339,7 @@ static int gmap_protect_rmap_pte(struct gmap *sg, struct gmap_rmap *rmap, { int rc = 0; - spin_lock(&sg->guest_table_lock); + spin_lock_nested(&sg->guest_table_lock, GMAP_LOCK_SHADOW); rc = gmap_protect_pte(sg->parent, paddr, vmaddr, ptep, prot, GMAP_NOTIFY_SHADOW); if (!rc) @@ -1874,7 +1874,7 @@ struct gmap *gmap_shadow(struct gmap *parent, unsigned long asce, /* only allow one real-space gmap shadow */ list_for_each_entry(sg, &parent->children, list) { if (sg->orig_asce & _ASCE_REAL_SPACE) { - spin_lock(&sg->guest_table_lock); + spin_lock_nested(&sg->guest_table_lock, GMAP_LOCK_SHADOW); gmap_unshadow(sg); spin_unlock(&sg->guest_table_lock); list_del(&sg->list); @@ -1946,7 +1946,7 @@ int gmap_shadow_r2t(struct gmap *sg, unsigned long saddr, unsigned long r2t, page->index |= GMAP_SHADOW_FAKE_TABLE; s_r2t = (unsigned long *) page_to_phys(page); /* Install shadow region second table */ - spin_lock(&sg->guest_table_lock); + spin_lock_nested(&sg->guest_table_lock, GMAP_LOCK_SHADOW); table = gmap_table_walk(sg, saddr, 4); /* get region-1 pointer */ if (!table) { rc = -EAGAIN; /* Race with unshadow */ @@ -1979,7 +1979,7 @@ int gmap_shadow_r2t(struct gmap *sg, unsigned long saddr, unsigned long r2t, offset = ((r2t & _REGION_ENTRY_OFFSET) >> 6) * PAGE_SIZE; len = ((r2t & _REGION_ENTRY_LENGTH) + 1) * PAGE_SIZE - offset; rc = gmap_protect_rmap(sg, raddr, origin + offset, len); - spin_lock(&sg->guest_table_lock); + spin_lock_nested(&sg->guest_table_lock, GMAP_LOCK_SHADOW); if (!rc) { table = gmap_table_walk(sg, saddr, 4); if (!table || (*table & _REGION_ENTRY_ORIGIN) != @@ -2030,7 +2030,7 @@ int gmap_shadow_r3t(struct gmap *sg, unsigned long saddr, unsigned long r3t, page->index |= GMAP_SHADOW_FAKE_TABLE; s_r3t = (unsigned long *) page_to_phys(page); /* Install shadow region second table */ - spin_lock(&sg->guest_table_lock); + spin_lock_nested(&sg->guest_table_lock, GMAP_LOCK_SHADOW); table = gmap_table_walk(sg, saddr, 3); /* get region-2 pointer */ if (!table) { rc = -EAGAIN; /* Race with unshadow */ @@ -2063,7 +2063,7 @@ int gmap_shadow_r3t(struct gmap *sg, unsigned long saddr, unsigned long r3t, offset = ((r3t & _REGION_ENTRY_OFFSET) >> 6) * PAGE_SIZE; len = ((r3t & _REGION_ENTRY_LENGTH) + 1) * PAGE_SIZE - offset; rc = gmap_protect_rmap(sg, raddr, origin + offset, len); - spin_lock(&sg->guest_table_lock); + spin_lock_nested(&sg->guest_table_lock, GMAP_LOCK_SHADOW); if (!rc) { table = gmap_table_walk(sg, saddr, 3); if (!table || (*table & _REGION_ENTRY_ORIGIN) != @@ -2114,7 +2114,7 @@ int gmap_shadow_sgt(struct gmap *sg, unsigned long saddr, unsigned long sgt, page->index |= GMAP_SHADOW_FAKE_TABLE; s_sgt = (unsigned long *) page_to_phys(page); /* Install shadow region second table */ - spin_lock(&sg->guest_table_lock); + spin_lock_nested(&sg->guest_table_lock, GMAP_LOCK_SHADOW); table = gmap_table_walk(sg, saddr, 2); /* get region-3 pointer */ if (!table) { rc = -EAGAIN; /* Race with unshadow */ @@ -2147,7 +2147,7 @@ int gmap_shadow_sgt(struct gmap *sg, unsigned long saddr, unsigned long sgt, offset = ((sgt & _REGION_ENTRY_OFFSET) >> 6) * PAGE_SIZE; len = ((sgt & _REGION_ENTRY_LENGTH) + 1) * PAGE_SIZE - offset; rc = gmap_protect_rmap(sg, raddr, origin + offset, len); - spin_lock(&sg->guest_table_lock); + spin_lock_nested(&sg->guest_table_lock, GMAP_LOCK_SHADOW); if (!rc) { table = gmap_table_walk(sg, saddr, 2); if (!table || (*table & _REGION_ENTRY_ORIGIN) != @@ -2203,7 +2203,7 @@ int gmap_shadow_sgt_lookup(struct gmap *sg, unsigned long saddr, int rc = -EAGAIN; BUG_ON(!gmap_is_shadow(sg)); - spin_lock(&sg->guest_table_lock); + spin_lock_nested(&sg->guest_table_lock, GMAP_LOCK_SHADOW); if (sg->asce & _ASCE_TYPE_MASK) { /* >2 GB guest */ r3e = (unsigned long *) gmap_table_walk(sg, saddr, 2); @@ -2270,7 +2270,7 @@ int gmap_shadow_pgt(struct gmap *sg, unsigned long saddr, unsigned long pgt, page->index |= GMAP_SHADOW_FAKE_TABLE; s_pgt = (unsigned long *) page_to_phys(page); /* Install shadow page table */ - spin_lock(&sg->guest_table_lock); + spin_lock_nested(&sg->guest_table_lock, GMAP_LOCK_SHADOW); table = gmap_table_walk(sg, saddr, 1); /* get segment pointer */ if (!table) { rc = -EAGAIN; /* Race with unshadow */ @@ -2298,7 +2298,7 @@ int gmap_shadow_pgt(struct gmap *sg, unsigned long saddr, unsigned long pgt, raddr = (saddr & _SEGMENT_MASK) | _SHADOW_RMAP_SEGMENT; origin = pgt & _SEGMENT_ENTRY_ORIGIN & PAGE_MASK; rc = gmap_protect_rmap(sg, raddr, origin, PAGE_SIZE); - spin_lock(&sg->guest_table_lock); + spin_lock_nested(&sg->guest_table_lock, GMAP_LOCK_SHADOW); if (!rc) { table = gmap_table_walk(sg, saddr, 1); if (!table || (*table & _SEGMENT_ENTRY_ORIGIN) != @@ -2359,7 +2359,7 @@ int gmap_shadow_segment(struct gmap *sg, unsigned long saddr, pmd_t pmd) rc = -EISDIR; break; } - spin_lock(&sg->guest_table_lock); + spin_lock_nested(&sg->guest_table_lock, GMAP_LOCK_SHADOW); /* Get shadow segment table pointer */ tpmdp = (pmd_t *) gmap_table_walk(sg, saddr, 1); if (!tpmdp) { @@ -2464,7 +2464,8 @@ int gmap_shadow_page(struct gmap *sg, unsigned long saddr, pte_t pte) if (pmd_large(*spmdp)) { pte_t spte; if (!(pmd_val(*spmdp) & _SEGMENT_ENTRY_PROTECT)) { - spin_lock(&sg->guest_table_lock); + spin_lock_nested(&sg->guest_table_lock, + GMAP_LOCK_SHADOW); spte = __pte((pmd_val(*spmdp) & _SEGMENT_ENTRY_ORIGIN_LARGE) + (pte_index(paddr) << 12)); @@ -2477,7 +2478,8 @@ int gmap_shadow_page(struct gmap *sg, unsigned long saddr, pte_t pte) } } else { sptep = gmap_pte_from_pmd(parent, spmdp, paddr, &ptl_pte); - spin_lock(&sg->guest_table_lock); + spin_lock_nested(&sg->guest_table_lock, + GMAP_LOCK_SHADOW); if (sptep) { rc = ptep_shadow_pte(sg->mm, saddr, sptep, tptep, pte); if (rc > 0) { @@ -2535,7 +2537,7 @@ static void gmap_shadow_notify_pmd(struct gmap *sg, unsigned long vmaddr, BUG_ON(!gmap_is_shadow(sg)); - spin_lock(&sg->guest_table_lock); + spin_lock_nested(&sg->guest_table_lock, GMAP_LOCK_SHADOW); if (sg->removed) { spin_unlock(&sg->guest_table_lock); return; @@ -2586,7 +2588,7 @@ static void gmap_shadow_notify(struct gmap *sg, unsigned long vmaddr, BUG_ON(!gmap_is_shadow(sg)); - spin_lock(&sg->guest_table_lock); + spin_lock_nested(&sg->guest_table_lock, GMAP_LOCK_SHADOW); if (sg->removed) { spin_unlock(&sg->guest_table_lock); return; @@ -2761,7 +2763,7 @@ static void gmap_pmdp_clear(struct mm_struct *mm, unsigned long vmaddr, rcu_read_lock(); list_for_each_entry_rcu(gmap, &mm->context.gmap_list, list) { - spin_lock(&gmap->guest_table_lock); + spin_lock_nested(&gmap->guest_table_lock, GMAP_LOCK_PARENT); pmdp = (pmd_t *)radix_tree_delete(&gmap->host_to_guest, vmaddr >> PMD_SHIFT); if (pmdp) { @@ -2816,7 +2818,7 @@ void gmap_pmdp_idte_local(struct mm_struct *mm, unsigned long vmaddr) rcu_read_lock(); list_for_each_entry_rcu(gmap, &mm->context.gmap_list, list) { - spin_lock(&gmap->guest_table_lock); + spin_lock_nested(&gmap->guest_table_lock, GMAP_LOCK_PARENT); entry = radix_tree_delete(&gmap->host_to_guest, vmaddr >> PMD_SHIFT); if (entry) { @@ -2852,7 +2854,7 @@ void gmap_pmdp_idte_global(struct mm_struct *mm, unsigned long vmaddr) rcu_read_lock(); list_for_each_entry_rcu(gmap, &mm->context.gmap_list, list) { - spin_lock(&gmap->guest_table_lock); + spin_lock_nested(&gmap->guest_table_lock, GMAP_LOCK_PARENT); entry = radix_tree_delete(&gmap->host_to_guest, vmaddr >> PMD_SHIFT); if (entry) { From patchwork Wed Jan 13 09:41:12 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Janosch Frank X-Patchwork-Id: 12016319 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3BC45C43333 for ; Wed, 13 Jan 2021 09:42:43 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id E810B233F6 for ; Wed, 13 Jan 2021 09:42:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727461AbhAMJmf (ORCPT ); Wed, 13 Jan 2021 04:42:35 -0500 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:25180 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727345AbhAMJmZ (ORCPT ); Wed, 13 Jan 2021 04:42:25 -0500 Received: from pps.filterd (m0098404.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 10D9XdNr135653; Wed, 13 Jan 2021 04:41:44 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=S6GFKR32bZeIaFLowyDYhSyXr4ztMq6nXbUXvw5LcTQ=; b=L1xbP4NpoBxqxDGyMfe2Y/NGgO8PCMPr40GZwdGGkFMGg7Kx4VC35jQolM3mHRl21niR tiBv6MYatDR1zy/XNF1pPp7IgkMHoRMdBR4gZIuaavLts9mIt77WuxoYCtzRUwnn0Ekl 9UY4ffUWxxtoUXeHosa34zV+n87Nim6MnMXjZEXfrageThqfN3eUCaoGFHrsLNNSyPca sbuoYdZ5ro6QKrtcg7QveA6IjVGSB0P74LeVsxSZxX/wmUV9jV0DSQeLd2iTxtE+B5G+ Hm4qJtQ2zfqE3AUj/vsmjlCNxCGJ/jknj+FwX16bFm7H0009nVyavopkFslMXSNY0ShL 6A== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 361uq2vjaf-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 13 Jan 2021 04:41:44 -0500 Received: from m0098404.ppops.net (m0098404.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 10D9YDPG138782; Wed, 13 Jan 2021 04:41:44 -0500 Received: from ppma04ams.nl.ibm.com (63.31.33a9.ip4.static.sl-reverse.com [169.51.49.99]) by mx0a-001b2d01.pphosted.com with ESMTP id 361uq2vj9p-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 13 Jan 2021 04:41:43 -0500 Received: from pps.filterd (ppma04ams.nl.ibm.com [127.0.0.1]) by ppma04ams.nl.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 10D9SHJu001148; Wed, 13 Jan 2021 09:41:41 GMT Received: from b06cxnps3075.portsmouth.uk.ibm.com (d06relay10.portsmouth.uk.ibm.com [9.149.109.195]) by ppma04ams.nl.ibm.com with ESMTP id 35y448cuyu-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 13 Jan 2021 09:41:41 +0000 Received: from d06av23.portsmouth.uk.ibm.com (d06av23.portsmouth.uk.ibm.com [9.149.105.59]) by b06cxnps3075.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 10D9fcF348365942 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 13 Jan 2021 09:41:38 GMT Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id AE4B0A4057; Wed, 13 Jan 2021 09:41:38 +0000 (GMT) Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 7CBA2A4053; Wed, 13 Jan 2021 09:41:38 +0000 (GMT) Received: from t46lp67.lnxne.boe (unknown [9.152.108.100]) by d06av23.portsmouth.uk.ibm.com (Postfix) with ESMTP; Wed, 13 Jan 2021 09:41:38 +0000 (GMT) From: Janosch Frank To: kvm@vger.kernel.org Cc: borntraeger@de.ibm.com, david@redhat.com, linux-s390@vger.kernel.org, imbrenda@linux.ibm.com Subject: [PATCH 13/14] s390/mm: Pull pmd invalid check in gmap_pmd_op_walk Date: Wed, 13 Jan 2021 09:41:12 +0000 Message-Id: <20210113094113.133668-14-frankja@linux.ibm.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210113094113.133668-1-frankja@linux.ibm.com> References: <20210113094113.133668-1-frankja@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.343,18.0.737 definitions=2021-01-13_03:2021-01-13,2021-01-13 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 suspectscore=0 lowpriorityscore=0 priorityscore=1501 clxscore=1015 bulkscore=0 impostorscore=0 mlxlogscore=999 phishscore=0 adultscore=0 spamscore=0 mlxscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101130056 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Not yet sure if I'll keep this. The walk should only walk and not check I, but then it looks way nicer. Signed-off-by: Janosch Frank --- arch/s390/mm/gmap.c | 17 +++++++---------- 1 file changed, 7 insertions(+), 10 deletions(-) diff --git a/arch/s390/mm/gmap.c b/arch/s390/mm/gmap.c index bc89fb974367..c4778ded8450 100644 --- a/arch/s390/mm/gmap.c +++ b/arch/s390/mm/gmap.c @@ -956,7 +956,8 @@ static inline pmd_t *gmap_pmd_op_walk(struct gmap *gmap, unsigned long gaddr, } pmdp = (pmd_t *) gmap_table_walk(gmap, gaddr, 1); - if (!pmdp || pmd_none(*pmdp)) { + if (!pmdp || pmd_none(*pmdp) || + pmd_val(*pmdp) & _SEGMENT_ENTRY_INVALID) { if (*ptl) spin_unlock(*ptl); pmdp = NULL; @@ -1165,7 +1166,7 @@ static int gmap_protect_range(struct gmap *gmap, unsigned long gaddr, return vmaddr; vmaddr |= gaddr & ~PMD_MASK; pmdp = gmap_pmd_op_walk(gmap, gaddr, vmaddr, &ptl_pmd); - if (pmdp && !(pmd_val(*pmdp) & _SEGMENT_ENTRY_INVALID)) { + if (pmdp) { if (!pmd_large(*pmdp)) { ptep = gmap_pte_from_pmd(gmap, pmdp, gaddr, &ptl_pte); @@ -1274,7 +1275,7 @@ int gmap_read_table(struct gmap *gmap, unsigned long gaddr, unsigned long *val) if (IS_ERR_VALUE(vmaddr)) return vmaddr; pmdp = gmap_pmd_op_walk(gmap, gaddr, vmaddr, &ptl_pmd); - if (pmdp && !(pmd_val(*pmdp) & _SEGMENT_ENTRY_INVALID)) { + if (pmdp) { if (!pmd_large(*pmdp)) { ptep = gmap_pte_from_pmd(gmap, pmdp, vmaddr, &ptl_pte); if (ptep) { @@ -1388,7 +1389,7 @@ static int gmap_protect_rmap(struct gmap *sg, unsigned long raddr, return vmaddr; vmaddr |= paddr & ~PMD_MASK; pmdp = gmap_pmd_op_walk(parent, paddr, vmaddr, &ptl_pmd); - if (pmdp && !(pmd_val(*pmdp) & _SEGMENT_ENTRY_INVALID)) { + if (pmdp) { if (!pmd_large(*pmdp)) { ptl_pte = NULL; ptep = gmap_pte_from_pmd(parent, pmdp, paddr, @@ -2378,8 +2379,7 @@ int gmap_shadow_segment(struct gmap *sg, unsigned long saddr, pmd_t pmd) break; } spmd = *spmdp; - if (!(pmd_val(spmd) & _SEGMENT_ENTRY_INVALID) && - !((pmd_val(spmd) & _SEGMENT_ENTRY_PROTECT) && + if (!((pmd_val(spmd) & _SEGMENT_ENTRY_PROTECT) && !(pmd_val(pmd) & _SEGMENT_ENTRY_PROTECT))) { pmd_val(*spmdp) |= _SEGMENT_ENTRY_GMAP_VSIE; @@ -2452,7 +2452,7 @@ int gmap_shadow_page(struct gmap *sg, unsigned long saddr, pte_t pte) break; rc = -EAGAIN; spmdp = gmap_pmd_op_walk(parent, paddr, vmaddr, &ptl_pmd); - if (spmdp && !(pmd_val(*spmdp) & _SEGMENT_ENTRY_INVALID)) { + if (spmdp) { /* Get page table pointer */ tptep = (pte_t *) gmap_table_walk(sg, saddr, 0); if (!tptep) { @@ -2886,9 +2886,6 @@ static bool gmap_test_and_clear_dirty_pmd(struct gmap *gmap, pmd_t *pmdp, unsigned long gaddr, unsigned long vmaddr) { - if (pmd_val(*pmdp) & _SEGMENT_ENTRY_INVALID) - return false; - /* Already protected memory, which did not change is clean */ if (pmd_val(*pmdp) & _SEGMENT_ENTRY_PROTECT && !(pmd_val(*pmdp) & _SEGMENT_ENTRY_GMAP_UC)) From patchwork Wed Jan 13 09:41:13 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Janosch Frank X-Patchwork-Id: 12016309 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1177FC432C3 for ; Wed, 13 Jan 2021 09:42:43 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id BBE0F233F9 for ; Wed, 13 Jan 2021 09:42:42 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727449AbhAMJme (ORCPT ); Wed, 13 Jan 2021 04:42:34 -0500 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:58382 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727396AbhAMJmZ (ORCPT ); Wed, 13 Jan 2021 04:42:25 -0500 Received: from pps.filterd (m0098404.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 10D9XaHq135299; Wed, 13 Jan 2021 04:41:44 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=zlHOQngHFVsjECDW2gjWgEiinuJbJTG62H3oRSZDlZ0=; b=YI+0L0/V8jG2Mh9jT3FUQx9rMMk8XewkSpc0vOmByy1cOMFlfsK7TrQ0bab5+VsF62XR RLaKSorTafwFQccp/VKpH2DnmK6XpguqG9dUr3LFCMVa4iMcr3hY9+fgHgxLE6n77g0h Vp11p6zqUpymZvMzM4SWFOVJf0xfc9epdm3k6DznjCkiZ8EQVU1J1DcYbzBOPLvmLb79 KoPCZYbGy7Ndq62955VLOSHGAh4PwfhI4/qPQ3ORm97OrMVDH8Cc9kru5oVR5tC4IJzx xeq5pz0dtDOVidmhvSWiRhtTZbuDq/bTuqnYMF7BThB+M3ux/TwPXpzw/BWtbF75zaPk RQ== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 361uq2vjas-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 13 Jan 2021 04:41:44 -0500 Received: from m0098404.ppops.net (m0098404.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 10D9Y0UN137579; Wed, 13 Jan 2021 04:41:44 -0500 Received: from ppma06fra.de.ibm.com (48.49.7a9f.ip4.static.sl-reverse.com [159.122.73.72]) by mx0a-001b2d01.pphosted.com with ESMTP id 361uq2vj9s-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 13 Jan 2021 04:41:44 -0500 Received: from pps.filterd (ppma06fra.de.ibm.com [127.0.0.1]) by ppma06fra.de.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 10D9S18i030792; Wed, 13 Jan 2021 09:41:42 GMT Received: from b06cxnps4074.portsmouth.uk.ibm.com (d06relay11.portsmouth.uk.ibm.com [9.149.109.196]) by ppma06fra.de.ibm.com with ESMTP id 361wgq814x-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 13 Jan 2021 09:41:41 +0000 Received: from d06av23.portsmouth.uk.ibm.com (d06av23.portsmouth.uk.ibm.com [9.149.105.59]) by b06cxnps4074.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 10D9fd3q28639546 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 13 Jan 2021 09:41:39 GMT Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id E0F4BA4055; Wed, 13 Jan 2021 09:41:38 +0000 (GMT) Received: from d06av23.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id B936FA4059; Wed, 13 Jan 2021 09:41:38 +0000 (GMT) Received: from t46lp67.lnxne.boe (unknown [9.152.108.100]) by d06av23.portsmouth.uk.ibm.com (Postfix) with ESMTP; Wed, 13 Jan 2021 09:41:38 +0000 (GMT) From: Janosch Frank To: kvm@vger.kernel.org Cc: borntraeger@de.ibm.com, david@redhat.com, linux-s390@vger.kernel.org, imbrenda@linux.ibm.com Subject: [PATCH 14/14] KVM: s390: Allow the VSIE to be used with huge pages Date: Wed, 13 Jan 2021 09:41:13 +0000 Message-Id: <20210113094113.133668-15-frankja@linux.ibm.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210113094113.133668-1-frankja@linux.ibm.com> References: <20210113094113.133668-1-frankja@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.343,18.0.737 definitions=2021-01-13_03:2021-01-13,2021-01-13 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 suspectscore=0 lowpriorityscore=0 priorityscore=1501 clxscore=1015 bulkscore=0 impostorscore=0 mlxlogscore=999 phishscore=0 adultscore=0 spamscore=0 mlxscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101130056 Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org Now that we have VSIE support for VMs with huge memory backing, let's make both features usable at the same time. Signed-off-by: Janosch Frank --- Documentation/virt/kvm/api.rst | 9 ++++----- arch/s390/kvm/kvm-s390.c | 14 ++------------ arch/s390/mm/gmap.c | 1 - 3 files changed, 6 insertions(+), 18 deletions(-) diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst index c136e254b496..c7fa31bbaa78 100644 --- a/Documentation/virt/kvm/api.rst +++ b/Documentation/virt/kvm/api.rst @@ -5876,15 +5876,14 @@ Do not enable KVM_FEATURE_PV_UNHALT if you disable HLT exits. :Architectures: s390 :Parameters: none -:Returns: 0 on success, -EINVAL if hpage module parameter was not set - or cmma is enabled, or the VM has the KVM_VM_S390_UCONTROL - flag set +:Returns: 0 on success, -EINVAL if cmma is enabled, or the VM has the + KVM_VM_S390_UCONTROL flag set With this capability the KVM support for memory backing with 1m pages through hugetlbfs can be enabled for a VM. After the capability is enabled, cmma can't be enabled anymore and pfmfi and the storage key -interpretation are disabled. If cmma has already been enabled or the -hpage module parameter is not set to 1, -EINVAL is returned. +interpretation are disabled. If cmma has already been enabled, -EINVAL +is returned. While it is generally possible to create a huge page backed VM without this capability, the VM will not be able to run. diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c index dbafd057ca6a..ad4fc84bb090 100644 --- a/arch/s390/kvm/kvm-s390.c +++ b/arch/s390/kvm/kvm-s390.c @@ -175,11 +175,6 @@ static int nested; module_param(nested, int, S_IRUGO); MODULE_PARM_DESC(nested, "Nested virtualization support"); -/* allow 1m huge page guest backing, if !nested */ -static int hpage; -module_param(hpage, int, 0444); -MODULE_PARM_DESC(hpage, "1m huge page backing support"); - /* maximum percentage of steal time for polling. >100 is treated like 100 */ static u8 halt_poll_max_steal = 10; module_param(halt_poll_max_steal, byte, 0644); @@ -551,7 +546,7 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext) break; case KVM_CAP_S390_HPAGE_1M: r = 0; - if (hpage && !kvm_is_ucontrol(kvm)) + if (!kvm_is_ucontrol(kvm)) r = 1; break; case KVM_CAP_S390_MEM_OP: @@ -761,7 +756,7 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm, struct kvm_enable_cap *cap) mutex_lock(&kvm->lock); if (kvm->created_vcpus) r = -EBUSY; - else if (!hpage || kvm->arch.use_cmma || kvm_is_ucontrol(kvm)) + else if (kvm->arch.use_cmma || kvm_is_ucontrol(kvm)) r = -EINVAL; else { r = 0; @@ -5042,11 +5037,6 @@ static int __init kvm_s390_init(void) return -ENODEV; } - if (nested && hpage) { - pr_info("A KVM host that supports nesting cannot back its KVM guests with huge pages\n"); - return -EINVAL; - } - for (i = 0; i < 16; i++) kvm_s390_fac_base[i] |= S390_lowcore.stfle_fac_list[i] & nonhyp_mask(i); diff --git a/arch/s390/mm/gmap.c b/arch/s390/mm/gmap.c index c4778ded8450..994688946553 100644 --- a/arch/s390/mm/gmap.c +++ b/arch/s390/mm/gmap.c @@ -1844,7 +1844,6 @@ struct gmap *gmap_shadow(struct gmap *parent, unsigned long asce, unsigned long limit; int rc; - BUG_ON(parent->mm->context.allow_gmap_hpage_1m); BUG_ON(gmap_is_shadow(parent)); spin_lock(&parent->shadow_lock); sg = gmap_find_shadow(parent, asce, edat_level);