From patchwork Wed Sep 19 08:47:50 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Janosch Frank X-Patchwork-Id: 10605499 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 34F3D5A4 for ; Wed, 19 Sep 2018 08:49:04 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2179A2B59F for ; Wed, 19 Sep 2018 08:49:04 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 157F02B5AA; Wed, 19 Sep 2018 08:49:04 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 79C402B59F for ; Wed, 19 Sep 2018 08:49:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731099AbeISOZy (ORCPT ); Wed, 19 Sep 2018 10:25:54 -0400 Received: from mx0a-001b2d01.pphosted.com ([148.163.156.1]:40948 "EHLO mx0a-001b2d01.pphosted.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730980AbeISOZy (ORCPT ); Wed, 19 Sep 2018 10:25:54 -0400 Received: from pps.filterd (m0098393.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w8J8iY7H024409 for ; Wed, 19 Sep 2018 04:49:01 -0400 Received: from e06smtp07.uk.ibm.com (e06smtp07.uk.ibm.com [195.75.94.103]) by mx0a-001b2d01.pphosted.com with ESMTP id 2mkgkmxh8n-1 (version=TLSv1.2 cipher=AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 19 Sep 2018 04:49:01 -0400 Received: from localhost by e06smtp07.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Wed, 19 Sep 2018 09:48:59 +0100 Received: from b06cxnps4076.portsmouth.uk.ibm.com (9.149.109.198) by e06smtp07.uk.ibm.com (192.168.101.137) with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted; (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256/256) Wed, 19 Sep 2018 09:48:56 +0100 Received: from d06av26.portsmouth.uk.ibm.com (d06av26.portsmouth.uk.ibm.com [9.149.105.62]) by b06cxnps4076.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id w8J8mtiY56754248 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=FAIL); Wed, 19 Sep 2018 08:48:55 GMT Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id CFC3CAE056; Wed, 19 Sep 2018 11:48:02 +0100 (BST) Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 29E42AE045; Wed, 19 Sep 2018 11:48:02 +0100 (BST) Received: from s38lp20.boeblingen.de.ibm.com (unknown [9.145.184.145]) by d06av26.portsmouth.uk.ibm.com (Postfix) with ESMTP; Wed, 19 Sep 2018 11:48:02 +0100 (BST) From: Janosch Frank To: kvm@vger.kernel.org Cc: linux-s390@vger.kernel.org, david@redhat.com, borntraeger@de.ibm.com, schwidefsky@de.ibm.com Subject: [RFC 02/14] s390/mm: Improve locking for huge page backings Date: Wed, 19 Sep 2018 10:47:50 +0200 X-Mailer: git-send-email 2.14.3 In-Reply-To: <20180919084802.183381-1-frankja@linux.ibm.com> References: <20180919084802.183381-1-frankja@linux.ibm.com> X-TM-AS-GCONF: 00 x-cbid: 18091908-0028-0000-0000-000002FB67A0 X-IBM-AV-DETECTION: SAVI=unused REMOTE=unused XFE=unused x-cbparentid: 18091908-0029-0000-0000-000023B52279 Message-Id: <20180919084802.183381-3-frankja@linux.ibm.com> X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2018-09-19_03:,, signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 priorityscore=1501 malwarescore=0 suspectscore=1 phishscore=0 bulkscore=0 spamscore=0 clxscore=1015 lowpriorityscore=0 mlxscore=0 impostorscore=0 mlxlogscore=472 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1807170000 definitions=main-1809190091 Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The gmap guest_table_lock is used to protect changes to the guest's DAT tables from region 1 to segments. Therefore it also protects the host to guest radix tree where each new segment mapping by gmap_link() is tracked. Changes to ptes are synchronized through the pte lock, which is easyly retrievable, because the gmap shares the page tables with userspace. With huge pages the story changes. PMD tables are not shared and we're left with the pmd lock on userspace side and the guest_table_lock on the gmap side. Having two locks for an object is a guarantee for locking problems. Therefore the guest_table_lock will only be used for population of the gmap tables and hence protecting the host_to_guest tree. While the pmd lock will be used for all changes to the pmd from both userspace and the gmap. This means we need to retrieve the vmaddr to retrieve a gmap pmd, which takes a bit longer than before. But we can now operate on multiple pmds which are in disjoint segment tables instead of having a global lock. Signed-off-by: Janosch Frank --- arch/s390/include/asm/pgtable.h | 1 + arch/s390/mm/gmap.c | 70 +++++++++++++++++++++++++---------------- arch/s390/mm/pgtable.c | 2 +- 3 files changed, 45 insertions(+), 28 deletions(-) diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtable.h index 0e7cb0dc9c33..c0abd57c5a21 100644 --- a/arch/s390/include/asm/pgtable.h +++ b/arch/s390/include/asm/pgtable.h @@ -1420,6 +1420,7 @@ static inline void __pudp_idte(unsigned long addr, pud_t *pudp, } } +pmd_t *pmd_alloc_map(struct mm_struct *mm, unsigned long addr); pmd_t pmdp_xchg_direct(struct mm_struct *, unsigned long, pmd_t *, pmd_t); pmd_t pmdp_xchg_lazy(struct mm_struct *, unsigned long, pmd_t *, pmd_t); pud_t pudp_xchg_direct(struct mm_struct *, unsigned long, pud_t *, pud_t); diff --git a/arch/s390/mm/gmap.c b/arch/s390/mm/gmap.c index 9ccd62cc7f37..04c24a284113 100644 --- a/arch/s390/mm/gmap.c +++ b/arch/s390/mm/gmap.c @@ -895,47 +895,62 @@ static void gmap_pte_op_end(spinlock_t *ptl) } /** - * gmap_pmd_op_walk - walk the gmap tables, get the guest table lock - * and return the pmd pointer + * gmap_pmd_op_walk - walk the gmap tables, get the pmd_lock if needed + * and return the pmd pointer or NULL * @gmap: pointer to guest mapping meta data structure * @gaddr: virtual address in the guest address space * * Returns a pointer to the pmd for a guest address, or NULL */ -static inline pmd_t *gmap_pmd_op_walk(struct gmap *gmap, unsigned long gaddr) +static inline pmd_t *gmap_pmd_op_walk(struct gmap *gmap, unsigned long gaddr, + spinlock_t **ptl) { - pmd_t *pmdp; + pmd_t *pmdp, *hpmdp; + unsigned long vmaddr; + BUG_ON(gmap_is_shadow(gmap)); - pmdp = (pmd_t *) gmap_table_walk(gmap, gaddr, 1); - if (!pmdp) - return NULL; - /* without huge pages, there is no need to take the table lock */ - if (!gmap->mm->context.allow_gmap_hpage_1m) - return pmd_none(*pmdp) ? NULL : pmdp; - - spin_lock(&gmap->guest_table_lock); - if (pmd_none(*pmdp)) { - spin_unlock(&gmap->guest_table_lock); - return NULL; + *ptl = NULL; + if (gmap->mm->context.allow_gmap_hpage_1m) { + vmaddr = __gmap_translate(gmap, gaddr); + if (IS_ERR_VALUE(vmaddr)) + return NULL; + hpmdp = pmd_alloc_map(gmap->mm, vmaddr); + if (!hpmdp) + return NULL; + *ptl = pmd_lock(gmap->mm, hpmdp); + if (pmd_none(*hpmdp)) { + spin_unlock(*ptl); + *ptl = NULL; + return NULL; + } + if (!pmd_large(*hpmdp)) { + spin_unlock(*ptl); + *ptl = NULL; + } + } + + pmdp = (pmd_t *) gmap_table_walk(gmap, gaddr, 1); + if (!pmdp || pmd_none(*pmdp)) { + if (*ptl) + spin_unlock(*ptl); + pmdp = NULL; + *ptl = NULL; } - /* 4k page table entries are locked via the pte (pte_alloc_map_lock). */ - if (!pmd_large(*pmdp)) - spin_unlock(&gmap->guest_table_lock); return pmdp; } /** - * gmap_pmd_op_end - release the guest_table_lock if needed + * gmap_pmd_op_end - release the pmd lock if needed * @gmap: pointer to the guest mapping meta data structure * @pmdp: pointer to the pmd */ -static inline void gmap_pmd_op_end(struct gmap *gmap, pmd_t *pmdp) +static inline void gmap_pmd_op_end(spinlock_t *ptl) { - if (pmd_large(*pmdp)) - spin_unlock(&gmap->guest_table_lock); + if (ptl) + spin_unlock(ptl); } /* @@ -1037,13 +1052,14 @@ static int gmap_protect_range(struct gmap *gmap, unsigned long gaddr, unsigned long len, int prot, unsigned long bits) { unsigned long vmaddr, dist; + spinlock_t *ptl = NULL; pmd_t *pmdp; int rc; BUG_ON(gmap_is_shadow(gmap)); while (len) { rc = -EAGAIN; - pmdp = gmap_pmd_op_walk(gmap, gaddr); + pmdp = gmap_pmd_op_walk(gmap, gaddr, &ptl); if (pmdp) { if (!pmd_large(*pmdp)) { rc = gmap_protect_pte(gmap, gaddr, pmdp, prot, @@ -1061,7 +1077,7 @@ static int gmap_protect_range(struct gmap *gmap, unsigned long gaddr, gaddr = (gaddr & HPAGE_MASK) + HPAGE_SIZE; } } - gmap_pmd_op_end(gmap, pmdp); + gmap_pmd_op_end(ptl); } if (rc) { if (rc == -EINVAL) @@ -2457,9 +2473,9 @@ void gmap_sync_dirty_log_pmd(struct gmap *gmap, unsigned long bitmap[4], int i; pmd_t *pmdp; pte_t *ptep; - spinlock_t *ptl; + spinlock_t *ptl = NULL; - pmdp = gmap_pmd_op_walk(gmap, gaddr); + pmdp = gmap_pmd_op_walk(gmap, gaddr, &ptl); if (!pmdp) return; @@ -2476,7 +2492,7 @@ void gmap_sync_dirty_log_pmd(struct gmap *gmap, unsigned long bitmap[4], spin_unlock(ptl); } } - gmap_pmd_op_end(gmap, pmdp); + gmap_pmd_op_end(ptl); } EXPORT_SYMBOL_GPL(gmap_sync_dirty_log_pmd); diff --git a/arch/s390/mm/pgtable.c b/arch/s390/mm/pgtable.c index 16d35b881a11..4b184744350b 100644 --- a/arch/s390/mm/pgtable.c +++ b/arch/s390/mm/pgtable.c @@ -410,7 +410,7 @@ static inline pmd_t pmdp_flush_lazy(struct mm_struct *mm, return old; } -static pmd_t *pmd_alloc_map(struct mm_struct *mm, unsigned long addr) +pmd_t *pmd_alloc_map(struct mm_struct *mm, unsigned long addr) { pgd_t *pgd; p4d_t *p4d;