From patchwork Tue Feb 2 09:11:14 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Aneesh Kumar K.V" X-Patchwork-Id: 12061177 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 01AE5C433DB for ; Tue, 2 Feb 2021 09:11:48 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 690F264E2B for ; Tue, 2 Feb 2021 09:11:47 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 690F264E2B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 291D36B0072; Tue, 2 Feb 2021 04:11:45 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 26AB56B0075; Tue, 2 Feb 2021 04:11:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E02BC6B0072; Tue, 2 Feb 2021 04:11:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0184.hostedemail.com [216.40.44.184]) by kanga.kvack.org (Postfix) with ESMTP id B70396B0074 for ; Tue, 2 Feb 2021 04:11:44 -0500 (EST) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 7C63EA8D8 for ; Tue, 2 Feb 2021 09:11:44 +0000 (UTC) X-FDA: 77772760128.19.thing32_4d13f28275c9 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin19.hostedemail.com (Postfix) with ESMTP id 5B1141AD1B3 for ; Tue, 2 Feb 2021 09:11:44 +0000 (UTC) X-HE-Tag: thing32_4d13f28275c9 X-Filterd-Recvd-Size: 7936 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf06.hostedemail.com (Postfix) with ESMTP for ; Tue, 2 Feb 2021 09:11:43 +0000 (UTC) Received: from pps.filterd (m0098399.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 11292Gbc028873; Tue, 2 Feb 2021 04:11:38 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-transfer-encoding; s=pp1; bh=Pr36thg1Hn0gsZexxZXZcL5GPtkl9LlFHSslf94bovU=; b=LsnBnnrwI/i1lBVoDCpMNaAqkjd75PQy0larIzI++1YIVxfZw+EZlgJy7PpCyarU1I6k O1w8/iHAj8CmNO8ynvT4vMW7vzf5ss9fIYg5UrWTFr1DHkk2WV6cIsdnX46Kl50RZ0tq 8WSyCWLpgBXapgM/g1+pW+SgIyZdAG7vcauqjan9cWwJ4bfRui4Pc2doeXVqsY58WvHp F268ONciRkqVy0A8E9qPoQFjjhAPfYumWAQS9xAPC2+H3spMn4OqkSir29wmq+8pVtd2 KkGTE03PZnqzCIgMSVJHg7EKx++nvWWQc4gAb2qqueddvK9RjW5U4VlAJrxGsGO/Da2f lA== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com with ESMTP id 36f3kc0h9u-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Feb 2021 04:11:37 -0500 Received: from m0098399.ppops.net (m0098399.ppops.net [127.0.0.1]) by pps.reinject (8.16.0.36/8.16.0.36) with SMTP id 11294hSk040872; Tue, 2 Feb 2021 04:11:37 -0500 Received: from ppma01dal.us.ibm.com (83.d6.3fa9.ip4.static.sl-reverse.com [169.63.214.131]) by mx0a-001b2d01.pphosted.com with ESMTP id 36f3kc0h9d-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Feb 2021 04:11:37 -0500 Received: from pps.filterd (ppma01dal.us.ibm.com [127.0.0.1]) by ppma01dal.us.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 112987EY013612; Tue, 2 Feb 2021 09:11:36 GMT Received: from b01cxnp23032.gho.pok.ibm.com (b01cxnp23032.gho.pok.ibm.com [9.57.198.27]) by ppma01dal.us.ibm.com with ESMTP id 36eu8qbpmh-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 02 Feb 2021 09:11:36 +0000 Received: from b01ledav001.gho.pok.ibm.com (b01ledav001.gho.pok.ibm.com [9.57.199.106]) by b01cxnp23032.gho.pok.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 1129BZAs30736846 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 2 Feb 2021 09:11:35 GMT Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 396C52805E; Tue, 2 Feb 2021 09:11:35 +0000 (GMT) Received: from b01ledav001.gho.pok.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 48BAF28058; Tue, 2 Feb 2021 09:11:32 +0000 (GMT) Received: from skywalker.ibmuc.com (unknown [9.199.35.92]) by b01ledav001.gho.pok.ibm.com (Postfix) with ESMTP; Tue, 2 Feb 2021 09:11:31 +0000 (GMT) From: "Aneesh Kumar K.V" To: linux-mm@kvack.org, akpm@linux-foundation.org Cc: mpe@ellerman.id.au, linuxppc-dev@lists.ozlabs.org, kaleshsingh@google.com, peterz@infradead.org, joel@joelfernandes.org, "Aneesh Kumar K.V" Subject: [RFC PATCH 4/6] mm/mremap: Use mmu gather interface instead of flush_tlb_range Date: Tue, 2 Feb 2021 14:41:14 +0530 Message-Id: <20210202091116.196134-4-aneesh.kumar@linux.ibm.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210202091116.196134-1-aneesh.kumar@linux.ibm.com> References: <20210202091116.196134-1-aneesh.kumar@linux.ibm.com> MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.369,18.0.737 definitions=2021-02-02_04:2021-01-29,2021-02-02 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxlogscore=999 malwarescore=0 adultscore=0 bulkscore=0 lowpriorityscore=0 suspectscore=0 clxscore=1015 mlxscore=0 priorityscore=1501 phishscore=0 impostorscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2102020059 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Some architectures do have the concept of page walk cache and only mmu gather interface supports flushing them. A fast mremap that involves moving page table pages instead of copying pte entries should flush page walk cache since the old translation cache is no more valid. Hence switch to mm gather to flush TLB and mark tlb.freed_tables = 1. No page table pages need to be freed here. With this the tlb flush is done outside page table lock (ptl). Signed-off-by: Aneesh Kumar K.V --- mm/mremap.c | 33 +++++++++++++++++++++++++++++---- 1 file changed, 29 insertions(+), 4 deletions(-) diff --git a/mm/mremap.c b/mm/mremap.c index 54fd2302b99d..14778d215011 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -217,6 +217,7 @@ static bool move_normal_pmd(struct vm_area_struct *vma, unsigned long old_addr, { spinlock_t *old_ptl, *new_ptl; struct mm_struct *mm = vma->vm_mm; + struct mmu_gather tlb; pmd_t pmd; /* @@ -245,11 +246,12 @@ static bool move_normal_pmd(struct vm_area_struct *vma, unsigned long old_addr, if (WARN_ON_ONCE(!pmd_none(*new_pmd))) return false; + tlb_gather_mmu(&tlb, mm, old_addr, PMD_SIZE); /* * We don't have to worry about the ordering of src and dst * ptlocks because exclusive mmap_lock prevents deadlock. */ - old_ptl = pmd_lock(vma->vm_mm, old_pmd); + old_ptl = pmd_lock(mm, old_pmd); new_ptl = pmd_lockptr(mm, new_pmd); if (new_ptl != old_ptl) spin_lock_nested(new_ptl, SINGLE_DEPTH_NESTING); @@ -258,13 +260,23 @@ static bool move_normal_pmd(struct vm_area_struct *vma, unsigned long old_addr, pmd = *old_pmd; pmd_clear(old_pmd); + /* + * Mark the range. We are not freeing page table pages nor + * regular pages. Hence we don't need to call tlb_remove_table() + * or tlb_remove_page(). + */ + tlb_flush_pte_range(&tlb, old_addr, PMD_SIZE); + tlb.freed_tables = 1; VM_BUG_ON(!pmd_none(*new_pmd)); pmd_populate(mm, new_pmd, (pgtable_t)pmd_page_vaddr(pmd)); - flush_tlb_range(vma, old_addr, old_addr + PMD_SIZE); if (new_ptl != old_ptl) spin_unlock(new_ptl); spin_unlock(old_ptl); + /* + * This will invalidate both the old TLB and page table walk caches. + */ + tlb_finish_mmu(&tlb, old_addr, PMD_SIZE); return true; } @@ -283,6 +295,7 @@ static bool move_normal_pud(struct vm_area_struct *vma, unsigned long old_addr, { spinlock_t *old_ptl, *new_ptl; struct mm_struct *mm = vma->vm_mm; + struct mmu_gather tlb; pud_t pud; /* @@ -292,11 +305,12 @@ static bool move_normal_pud(struct vm_area_struct *vma, unsigned long old_addr, if (WARN_ON_ONCE(!pud_none(*new_pud))) return false; + tlb_gather_mmu(&tlb, mm, old_addr, PUD_SIZE); /* * We don't have to worry about the ordering of src and dst * ptlocks because exclusive mmap_lock prevents deadlock. */ - old_ptl = pud_lock(vma->vm_mm, old_pud); + old_ptl = pud_lock(mm, old_pud); new_ptl = pud_lockptr(mm, new_pud); if (new_ptl != old_ptl) spin_lock_nested(new_ptl, SINGLE_DEPTH_NESTING); @@ -305,14 +319,25 @@ static bool move_normal_pud(struct vm_area_struct *vma, unsigned long old_addr, pud = *old_pud; pud_clear(old_pud); + /* + * Mark the range. We are not freeing page table pages nor + * regular pages. Hence we don't need to call tlb_remove_table() + * or tlb_remove_page(). + */ + tlb_flush_pte_range(&tlb, old_addr, PUD_SIZE); + tlb.freed_tables = 1; VM_BUG_ON(!pud_none(*new_pud)); pud_populate(mm, new_pud, (pmd_t *)pud_page_vaddr(pud)); - flush_tlb_range(vma, old_addr, old_addr + PUD_SIZE); + if (new_ptl != old_ptl) spin_unlock(new_ptl); spin_unlock(old_ptl); + /* + * This will invalidate both the old TLB and page table walk caches. + */ + tlb_finish_mmu(&tlb, old_addr, PUD_SIZE); return true; } #else