From patchwork Wed Feb 3 21:08:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 12065381 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.6 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EE64AC433DB for ; Wed, 3 Feb 2021 21:08:49 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9ACD864F60 for ; Wed, 3 Feb 2021 21:08:49 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9ACD864F60 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id A85666B0071; Wed, 3 Feb 2021 16:08:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9E8756B0072; Wed, 3 Feb 2021 16:08:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 813936B0073; Wed, 3 Feb 2021 16:08:46 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0226.hostedemail.com [216.40.44.226]) by kanga.kvack.org (Postfix) with ESMTP id 5D8876B0071 for ; Wed, 3 Feb 2021 16:08:46 -0500 (EST) Received: from smtpin11.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 1B88E1EF1 for ; Wed, 3 Feb 2021 21:08:46 +0000 (UTC) X-FDA: 77778195852.11.water87_3200f37275d6 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin11.hostedemail.com (Postfix) with ESMTP id E07D1180F8B81 for ; Wed, 3 Feb 2021 21:08:45 +0000 (UTC) X-HE-Tag: water87_3200f37275d6 X-Filterd-Recvd-Size: 8306 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [216.205.24.124]) by imf50.hostedemail.com (Postfix) with ESMTP for ; Wed, 3 Feb 2021 21:08:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1612386524; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=CAR5UYzoAPy14j76vk233Tw9s0t/A46HICdSqrKjEFQ=; b=Oxqf5uGskzjt3YIm3gS2xwucn05MJPScPWJI844xlTIodeBE/VT09EWnyvy2RZijylBPoC 7LsWW/TQ4PWZaVYnMBxFObe1mSeOZ2ItBueL4STmqLo3qi9UuttPBwmORVyVa4EE5KutEZ SvIgwha4pBNY2GhP2nAReRdnVdmALew= Received: from mail-qv1-f69.google.com (mail-qv1-f69.google.com [209.85.219.69]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-545-oyn7w1bPOPGWnHTnWVXx7w-1; Wed, 03 Feb 2021 16:08:43 -0500 X-MC-Unique: oyn7w1bPOPGWnHTnWVXx7w-1 Received: by mail-qv1-f69.google.com with SMTP id h13so468784qvo.18 for ; Wed, 03 Feb 2021 13:08:43 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=CAR5UYzoAPy14j76vk233Tw9s0t/A46HICdSqrKjEFQ=; b=QwBoXZA0hbl3HxqOQ8rBBnFbpCQMx6DSMJAop1nJ1VkI5JzoMBab0JAFU/mgTlCF4f GxOHT4ATghq4OsMoIedKIQvFXE/5cMoQUZ4ygWacK76uwICJbJqsvXj4O3Jt8uRB3UcO EmPligxNL1JRvMaC14Lp28FsAFTV51ebTFMb9KNxmTc5Sl1vp7pLrcEj9Zjo6YVIMrQN rcvwt0ObKkTiL8w6ImhqFJ8urAFgEmQzNRPUseFS+ZsCaHfuifAc6LOUXFG6XEaXzNDm 1Dh9CXg7Tw6WmJQSHvclGdZP/dwm36iXGYjTiwDg1+WE9O2WlLFvrImoSA4MYjn7FJTu mh+g== X-Gm-Message-State: AOAM533xnR+ByG9dfV2wtvjEqJusTl/ZMkc5nKdFrg6AaT2zbHaEwplp GfE4glnGrCQ7m9ya54LteZ05aCouLPgDBzg9Sj2atg5K5cKApa1DloL4zJp9pLSymSM5zNUIitK mmK+duF0ZdMc= X-Received: by 2002:ac8:82b:: with SMTP id u40mr4391918qth.332.1612386522676; Wed, 03 Feb 2021 13:08:42 -0800 (PST) X-Google-Smtp-Source: ABdhPJwVVt5kjHSDTzY5qynUdx3KoddfTDU7c0XF5xIku0zmBdUNWjFgD59RCmSGKUsLIoLJfGy42A== X-Received: by 2002:ac8:82b:: with SMTP id u40mr4391895qth.332.1612386522444; Wed, 03 Feb 2021 13:08:42 -0800 (PST) Received: from xz-x1.redhat.com (bras-vprn-toroon474qw-lp130-20-174-93-89-182.dsl.bell.ca. [174.93.89.182]) by smtp.gmail.com with ESMTPSA id t6sm2507659qti.2.2021.02.03.13.08.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 03 Feb 2021 13:08:41 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Wei Zhang , Matthew Wilcox , Linus Torvalds , Jason Gunthorpe , Gal Pressman , peterx@redhat.com, Christoph Hellwig , Andrea Arcangeli , Jan Kara , Kirill Shutemov , David Gibson , Mike Rapoport , Mike Kravetz , Kirill Tkhai , Jann Horn , Andrew Morton Subject: [PATCH 4/4] hugetlb: Do early cow when page pinned on src mm Date: Wed, 3 Feb 2021 16:08:32 -0500 Message-Id: <20210203210832.113685-5-peterx@redhat.com> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210203210832.113685-1-peterx@redhat.com> References: <20210203210832.113685-1-peterx@redhat.com> MIME-Version: 1.0 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=peterx@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This is the last missing piece of the COW-during-fork effort when there're pinned pages found. One can reference 70e806e4e645 ("mm: Do early cow for pinned pages during fork() for ptes", 2020-09-27) for more information, since we do similar things here rather than pte this time, but just for hugetlb. Signed-off-by: Peter Xu --- mm/hugetlb.c | 76 ++++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 71 insertions(+), 5 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 9e6ea96bf33b..931bf1a81c16 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -3734,11 +3734,27 @@ static bool is_hugetlb_entry_hwpoisoned(pte_t pte) return false; } +static void +hugetlb_copy_page(struct vm_area_struct *vma, pte_t *ptep, unsigned long addr, + struct page *old_page, struct page *new_page) +{ + struct hstate *h = hstate_vma(vma); + unsigned int psize = pages_per_huge_page(h); + + copy_user_huge_page(new_page, old_page, addr, vma, psize); + __SetPageUptodate(new_page); + ClearPagePrivate(new_page); + set_page_huge_active(new_page); + set_huge_pte_at(vma->vm_mm, addr, ptep, make_huge_pte(vma, new_page, 1)); + hugepage_add_new_anon_rmap(new_page, vma, addr); + hugetlb_count_add(psize, vma->vm_mm); +} + int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, struct vm_area_struct *vma) { pte_t *src_pte, *dst_pte, entry, dst_entry; - struct page *ptepage; + struct page *ptepage, *prealloc = NULL; unsigned long addr; int cow; struct hstate *h = hstate_vma(vma); @@ -3787,7 +3803,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, dst_entry = huge_ptep_get(dst_pte); if ((dst_pte == src_pte) || !huge_pte_none(dst_entry)) continue; - +again: dst_ptl = huge_pte_lock(h, dst, dst_pte); src_ptl = huge_pte_lockptr(h, src, src_pte); spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); @@ -3816,6 +3832,54 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, } set_huge_swap_pte_at(dst, addr, dst_pte, entry, sz); } else { + entry = huge_ptep_get(src_pte); + ptepage = pte_page(entry); + get_page(ptepage); + + if (unlikely(page_needs_cow_for_dma(vma, ptepage))) { + /* This is very possibly a pinned huge page */ + if (!prealloc) { + /* + * Preallocate the huge page without + * tons of locks since we could sleep. + * Note: we can't use any reservation + * because the page will be exclusively + * owned by the child later. + */ + put_page(ptepage); + spin_unlock(src_ptl); + spin_unlock(dst_ptl); + prealloc = alloc_huge_page(vma, addr, 0); + if (!prealloc) { + /* + * hugetlb_cow() seems to be + * more careful here than us. + * However for fork() we could + * be strict not only because + * no one should be referencing + * the child mm yet, but also + * if resources are rare we'd + * better simply fail the + * fork() even earlier. + */ + ret = -ENOMEM; + break; + } + goto again; + } + /* + * We have page preallocated so that we can do + * the copy right now. + */ + hugetlb_copy_page(vma, dst_pte, addr, ptepage, + prealloc); + put_page(ptepage); + spin_unlock(src_ptl); + spin_unlock(dst_ptl); + prealloc = NULL; + continue; + } + if (cow) { /* * No need to notify as we are downgrading page @@ -3826,9 +3890,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, */ huge_ptep_set_wrprotect(src, addr, src_pte); } - entry = huge_ptep_get(src_pte); - ptepage = pte_page(entry); - get_page(ptepage); + page_dup_rmap(ptepage, true); set_huge_pte_at(dst, addr, dst_pte, entry); hugetlb_count_add(pages_per_huge_page(h), dst); @@ -3842,6 +3904,10 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, else i_mmap_unlock_read(mapping); + /* Free the preallocated page if not used at last */ + if (prealloc) + put_page(prealloc); + return ret; }