From patchwork Mon Jan 29 12:46:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Hildenbrand X-Patchwork-Id: 13535525 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9A501C47DA9 for ; Mon, 29 Jan 2024 12:48:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2899E6B00A6; Mon, 29 Jan 2024 07:48:13 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 239C76B00A7; Mon, 29 Jan 2024 07:48:13 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0B4D66B00A8; Mon, 29 Jan 2024 07:48:13 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id EBB0B6B00A6 for ; Mon, 29 Jan 2024 07:48:12 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id C240180A33 for ; Mon, 29 Jan 2024 12:48:12 +0000 (UTC) X-FDA: 81732326424.26.63E8B94 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf22.hostedemail.com (Postfix) with ESMTP id 0D57EC0027 for ; Mon, 29 Jan 2024 12:48:10 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=XUxetxEB; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf22.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706532491; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=d7wzDgEVUWhg5ZER45HLd710ycKB3fz8RPuXE32SMZA=; b=UuwTpksM5D1n4Wz2Xhjx/s3yET1Zhz8jvPI9x9awY19Ii+uyGploBvkVLINPM5Oe28mzJx Ds2/ZbaSNH9KfA6euYFreNkQ4z7IPWMab7NMUBX4Nd5Uo25Nc6BAkq058bUQL6UOdy0Az5 1bhqUrjwjbe9fHHgnDBdFEY/kk0GBUg= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=XUxetxEB; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf22.hostedemail.com: domain of david@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=david@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706532491; a=rsa-sha256; cv=none; b=s//Rgh17J7EJa2OOTBRbTDLV9j8YzKZF8eGE2noDJFGnbRk7Jw6AS9XXnaKcCGDUEVGhoE /82BQZHjH7CQ/LKHcOx7uHe5CelcAxbyfuOSuoFKrPXh2JUWqLBWqLpJi17kCXxxiWOOm1 sZgv6dHF4uQAk46pEDUpamI/uw9pm0I= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1706532490; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=d7wzDgEVUWhg5ZER45HLd710ycKB3fz8RPuXE32SMZA=; b=XUxetxEBhoBXne5mDBIHQ5yIZ+scYzI23fjLUGXliJN2/GYKLbTASZknuW4vj30nsdJ/EG n+EXsl5OfFfRp8hxr9Th7qqn6Di/7jQz8DnWp2mBZJ4cyiUmOMtoWFPiMt3MKOx2FWz24/ fgqUooRq3DMuB8L/hUWUySe/9Ea8Kxw= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-140-9ly1pTIPMpeLAWW3OrQO1g-1; Mon, 29 Jan 2024 07:48:04 -0500 X-MC-Unique: 9ly1pTIPMpeLAWW3OrQO1g-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.rdu2.redhat.com [10.11.54.5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 3AD4A3CBD50E; Mon, 29 Jan 2024 12:48:03 +0000 (UTC) Received: from t14s.fritz.box (unknown [10.39.194.46]) by smtp.corp.redhat.com (Postfix) with ESMTP id 2C0D28B; Mon, 29 Jan 2024 12:47:56 +0000 (UTC) From: David Hildenbrand To: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org, David Hildenbrand , Andrew Morton , Matthew Wilcox , Ryan Roberts , Russell King , Catalin Marinas , Will Deacon , Dinh Nguyen , Michael Ellerman , Nicholas Piggin , Christophe Leroy , "Aneesh Kumar K.V" , "Naveen N. Rao" , Paul Walmsley , Palmer Dabbelt , Albert Ou , Alexander Gordeev , Gerald Schaefer , Heiko Carstens , Vasily Gorbik , Christian Borntraeger , Sven Schnelle , "David S. Miller" , linux-arm-kernel@lists.infradead.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, sparclinux@vger.kernel.org Subject: [PATCH v3 11/15] mm/memory: factor out copying the actual PTE in copy_present_pte() Date: Mon, 29 Jan 2024 13:46:45 +0100 Message-ID: <20240129124649.189745-12-david@redhat.com> In-Reply-To: <20240129124649.189745-1-david@redhat.com> References: <20240129124649.189745-1-david@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.5 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 0D57EC0027 X-Stat-Signature: e3m99au1q3bm9kubzh8dq89io1xnuyym X-HE-Tag: 1706532490-633764 X-HE-Meta: U2FsdGVkX18oY4fpdaauV59XG7c/QZNJXLzg5DG7LoJStBb1pp4YU1VJnZxHnfjQfwFk6DgGTGN1IPTNqlKxYbskfgGh88yAptg4hKlkiGo1pe8u0wDXboTjq2mKQVR9EqN5bnUsD2FkPmYk1fX7QdVTJGm+6rwktJSgks4b4yXK8/VOPuWyUaFqWoGLW0pcufcInx7auQOGaDv4w+2Py9EZXwWT6Dv1t/KHYZGaos8JDZe4h7EXNv5dD8OHeDJil2hOkByvm6M38XIuK+7f7hbTvs/Cr8xbUQiXbwxNvOw9fSYsrcdStEB34lqzWRvI84bLB6XexdlCuz5+0g4Ej4K2ubAufthh7q3j7SHR/3MYDCfaF3SkDm8N+9L444EyLBNi9xar4JIPkeqmQeZXSPYdkUy79k7dB5KA7t7ZEBPb3VGUqCnjmTBtT4wXGJB0eXC6jKUMbSVx1nxYrlvb6hGaPlh0OOSiJuBKq2+2FhpDBYA0jXdkpY/lIPo1dM0CKN+c9WbSgpZ/H3awYq9upyaJqMBCbzIQNUmC2xk5/Ss27lrhWZneYvtfgIUoiDAFQL/j6vB022/l0Hp714yrUhNBb66gwdta1+SL4GCUp/JrsS+WF5yRvsabJKmMAlchvOEwC+YPzQx/1SpGugubrVRgbHrlhNNsBaXQoI+NAM/+tNSk4IVnSP8tWqJ0M4vUWsgdl7jV0LkJJc6LXbyUZr7HDop7zooQrleSMRoc2WTUfrQZBIw3mX1GdX82Z9REM9yHZCqZTlw2y84aui3vrLz7dN0lyoGqNxoGLc7Z0t5iLnyIryDlmONIPh9qhbHPZylfxLDXph44hPBTzXfM+E+14/ObqXQ8Wms1mzS/lrxdxiXj7KILTHVrnO7ISOnamlvM1qxvjYTuxDkcTPJWTfMTAwDPteIOSzoVWRwQeEpNzmU1DjlDHNVeQZqTZVp4cVK653Meyu8U1qzs5fK h/u8pgxs 4mdWXnk3ANVqnUVRSh3GMTXalOLOBFM7tq1wRJdfQn3bIzruB9akejRXokHdTqJ4SLy10dR5entDt6kbn+MxXq/YDvRVNGpIcLz/4A3N8a4Qr0GYLCyxCjQWd6/75tc+SJMS5VvyuUKn1iIWLgwO5gxHOyaTWinMtvd71KlXRH1YPM3YXq/p8rDD4VzD9Y/ajM9ubs0ojkHuyG10xx34GFaHamlpAT60ZBmd3vlVm+bceKczOfvgSWN3Wa9E5nmvuOk8mg+f1/V2yVEFuzud4fJCsGg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Let's prepare for further changes. Reviewed-by: Ryan Roberts Signed-off-by: David Hildenbrand Reviewed-by: Mike Rapoport (IBM) --- mm/memory.c | 63 ++++++++++++++++++++++++++++------------------------- 1 file changed, 33 insertions(+), 30 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 8d14ba440929..a3bdb25f4c8d 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -930,6 +930,29 @@ copy_present_page(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma return 0; } +static inline void __copy_present_pte(struct vm_area_struct *dst_vma, + struct vm_area_struct *src_vma, pte_t *dst_pte, pte_t *src_pte, + pte_t pte, unsigned long addr) +{ + struct mm_struct *src_mm = src_vma->vm_mm; + + /* If it's a COW mapping, write protect it both processes. */ + if (is_cow_mapping(src_vma->vm_flags) && pte_write(pte)) { + ptep_set_wrprotect(src_mm, addr, src_pte); + pte = pte_wrprotect(pte); + } + + /* If it's a shared mapping, mark it clean in the child. */ + if (src_vma->vm_flags & VM_SHARED) + pte = pte_mkclean(pte); + pte = pte_mkold(pte); + + if (!userfaultfd_wp(dst_vma)) + pte = pte_clear_uffd_wp(pte); + + set_pte_at(dst_vma->vm_mm, addr, dst_pte, pte); +} + /* * Copy one pte. Returns 0 if succeeded, or -EAGAIN if one preallocated page * is required to copy this pte. @@ -939,23 +962,23 @@ copy_present_pte(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, pte_t *dst_pte, pte_t *src_pte, unsigned long addr, int *rss, struct folio **prealloc) { - struct mm_struct *src_mm = src_vma->vm_mm; - unsigned long vm_flags = src_vma->vm_flags; pte_t pte = ptep_get(src_pte); struct page *page; struct folio *folio; page = vm_normal_page(src_vma, addr, pte); - if (page) - folio = page_folio(page); - if (page && folio_test_anon(folio)) { + if (unlikely(!page)) + goto copy_pte; + + folio = page_folio(page); + folio_get(folio); + if (folio_test_anon(folio)) { /* * If this page may have been pinned by the parent process, * copy the page immediately for the child so that we'll always * guarantee the pinned page won't be randomly replaced in the * future. */ - folio_get(folio); if (unlikely(folio_try_dup_anon_rmap_pte(folio, page, src_vma))) { /* Page may be pinned, we have to copy. */ folio_put(folio); @@ -963,34 +986,14 @@ copy_present_pte(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, addr, rss, prealloc, page); } rss[MM_ANONPAGES]++; - } else if (page) { - folio_get(folio); + VM_WARN_ON_FOLIO(PageAnonExclusive(page), folio); + } else { folio_dup_file_rmap_pte(folio, page); rss[mm_counter_file(folio)]++; } - /* - * If it's a COW mapping, write protect it both - * in the parent and the child - */ - if (is_cow_mapping(vm_flags) && pte_write(pte)) { - ptep_set_wrprotect(src_mm, addr, src_pte); - pte = pte_wrprotect(pte); - } - VM_BUG_ON(page && folio_test_anon(folio) && PageAnonExclusive(page)); - - /* - * If it's a shared mapping, mark it clean in - * the child - */ - if (vm_flags & VM_SHARED) - pte = pte_mkclean(pte); - pte = pte_mkold(pte); - - if (!userfaultfd_wp(dst_vma)) - pte = pte_clear_uffd_wp(pte); - - set_pte_at(dst_vma->vm_mm, addr, dst_pte, pte); +copy_pte: + __copy_present_pte(dst_vma, src_vma, dst_pte, src_pte, pte, addr); return 0; }