From patchwork Mon Aug 26 20:43:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13778401 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B7BEDC5472C for ; Mon, 26 Aug 2024 20:44:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C2A236B009A; Mon, 26 Aug 2024 16:44:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B63D26B009B; Mon, 26 Aug 2024 16:44:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9522C6B009C; Mon, 26 Aug 2024 16:44:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 6FCC26B009A for ; Mon, 26 Aug 2024 16:44:17 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 3836E1A1330 for ; Mon, 26 Aug 2024 20:44:17 +0000 (UTC) X-FDA: 82495574154.20.938BAAC Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf25.hostedemail.com (Postfix) with ESMTP id 2F6AAA0012 for ; Mon, 26 Aug 2024 20:44:15 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="Mxon/CzT"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf25.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1724704985; a=rsa-sha256; cv=none; b=bKn3hAu/MXZ4zTLNOyEB69bdTTr3JqsAKOLx79k8nW42bZDjiLLlAnQfErWNBt1JE74nMa KWJFMc73NyICY/sHfwsLDbL1Wj/+qAiXxOy5skEO9hudCw+KAYOg68m5EGMrUDdOCHfhk2 RR+5zuC/S0CXEoA1gfVJt9+Us1NPSQ0= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="Mxon/CzT"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf25.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1724704985; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=xat90K+7laktWlKWrEYj2NKDd4TQak32OHmU5MtaKmE=; b=hCcxiOUUOpDWR79QtyipBr7u+M8KvEgQVIrhdjQ4cpL7yVfAduj4z7rMHuqEeShIrefhFH qJFtSICjFqZpWWBDFE5iUnqHq4TQLN9n4VE76Yg2gi3c74kHNP7BhpUpbW6+ZrIu7RpBds x1sjuQr9dS1sGJV/hU9j/W7PDtFMcwE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1724705054; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=xat90K+7laktWlKWrEYj2NKDd4TQak32OHmU5MtaKmE=; b=Mxon/CzTlrAUDYEC4tybCxt8knx5/r2HXsOm/8An5nLpqHxljJBr/sxpals4Ak84FhtDPW wxgqD1q4qxpCNpKU5PuWaIBLZXKyTnXna4aqGCxytVTBx01rn8U3MKOjLuizy2iPRtsC24 h/NvGX9Zbuw3ViIbYLAoiTd9lFp9x6c= Received: from mail-yw1-f198.google.com (mail-yw1-f198.google.com [209.85.128.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-91-9fwqpGmyNIO6g-0E-lMKJA-1; Mon, 26 Aug 2024 16:44:11 -0400 X-MC-Unique: 9fwqpGmyNIO6g-0E-lMKJA-1 Received: by mail-yw1-f198.google.com with SMTP id 00721157ae682-6adcb88da08so100204027b3.3 for ; Mon, 26 Aug 2024 13:44:11 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1724705051; x=1725309851; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xat90K+7laktWlKWrEYj2NKDd4TQak32OHmU5MtaKmE=; b=Fq2oYFApJeP79cAKjRKdAp40/TkV5XbV7Ke+U5TBgC6Y0tV/1VZIrAmHTKPv5JVlYY ne5FF21G+rifc6afurMwbq0dOsiYZDXrG1o+HOy0+7LCPzIXhrfR1nKPuL69xaR4AKGg gzqP8hrbfsVYARcK0sNyAyKIqy9QKtXRTDOXCdESlnUareeApAub7vGv5eee3rnbmTQD QTTCKWx0d33YxTYYZU5u8oPaysoxQKE1Uxq7t93kvV6Iri6yTsEfM/1IN2Zba2FP7/rw Z4EGfQ0GjM80L1m2CDsnQyNF/T3QYtlRM8/+xMb8hBggMU3XYvJufBS5fgspsP9Tsv6k IqwA== X-Forwarded-Encrypted: i=1; AJvYcCUtgtxZVFSeSvSswgEjXWdbavSIOo70Ntpw62IcjC0fM95iv8CJDUz6tQ67YnKCNp0vcKk8K01O0w==@kvack.org X-Gm-Message-State: AOJu0Yyl4QBgBOG2H9qeLPDJ+u+AhEUKFWwrtZCjk6Zu5Q2XP9uF8yV2 TSAsr0uB75bT4KqzNjPugsgSfA7tSVK/lIjxImt1By6lf6HCOmJPFv9+ZVBNY67sqd0famM9/zL D55y+1zRCVToY/57aPabAU3qXjAY+lq7eByKRxTrPIncRwPDQ X-Received: by 2002:a05:690c:380b:b0:6c1:2b6d:1964 with SMTP id 00721157ae682-6c62906557cmr136660707b3.38.1724705050751; Mon, 26 Aug 2024 13:44:10 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGVQdSmnkARvKpk+I4g97hETTq/PaNGcO/OaJoau0YQSMQm37evWRXkCU2N5l4bob/h1UXyLw== X-Received: by 2002:a05:690c:380b:b0:6c1:2b6d:1964 with SMTP id 00721157ae682-6c62906557cmr136660497b3.38.1724705050419; Mon, 26 Aug 2024 13:44:10 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id af79cd13be357-7a67f3fd6c1sm491055185a.121.2024.08.26.13.44.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 26 Aug 2024 13:44:09 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Gavin Shan , Catalin Marinas , x86@kernel.org, Ingo Molnar , Andrew Morton , Paolo Bonzini , Dave Hansen , Thomas Gleixner , Alistair Popple , kvm@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Sean Christopherson , peterx@redhat.com, Oscar Salvador , Jason Gunthorpe , Borislav Petkov , Zi Yan , Axel Rasmussen , David Hildenbrand , Yan Zhao , Will Deacon , Kefeng Wang , Alex Williamson Subject: [PATCH v2 07/19] mm/fork: Accept huge pfnmap entries Date: Mon, 26 Aug 2024 16:43:41 -0400 Message-ID: <20240826204353.2228736-8-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240826204353.2228736-1-peterx@redhat.com> References: <20240826204353.2228736-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 2F6AAA0012 X-Stat-Signature: 7o8ik9t5snsf115jghgoasta1nwist4d X-Rspam-User: X-HE-Tag: 1724705055-70743 X-HE-Meta: U2FsdGVkX182Lv31Q/FUCQNfhmKfEAJi7P9nM9kGf+251GMghc5ZpXLZ8DJ3En4syUYXC2OE1Tt4BpD/pBlNQk/pycnDGWB0eq+LDvwgyLEHRI7gLsYM4idzqgwrzj0s+gORIWC9DjHXK1THDnWPU75SkR21d5TRMxpT3+kZO9M7XHCJv2WTAYqrr9TN3X+mFd27/1RvKxheEdvx3hd0mXNOvxmdZrV5FZ57g4ThDk0rOOwV5j/a5yHVCANdU2sKxWFAXNWuBtRHD3cnn9fp2MLNIpb7Rz755Dcr1J8LG03Vw4MgXC2yuQTzuDsZS48qk3jSihfWOEY5lNrB3phNCmUwm3YqPCp8zOD7/NP6VJTophWTt4p/IR3SvFn4eB918BtEUcQXGhbuLW0KmmtUMUfHVHCmyfXkiQdzdIP91+L9U4OqtCREsLxbJv3isAYRACYvhIef7oTzqV9VdRNvG2k+aMdh5P1Mfltal/EQ0Za95p+PS0n/I7AIs+aIKjdYPGbbLZiW8LQ4PaIL3cS05zkbBZ0gzy19Wi/zlppXVdvXjcnrHdmlirRAt4qipQ/pksNyn2Q1w6L4QZBuCIuGMqAHr1RZ60s+q1lyjFRssuA3dWtcbWkDuDuD7cD040+IaXfQsNV15xa45pKooEZGJYXjeZZ060m7hlrq/PYiog5i7d+Fyso2EnTYmaMyVlqoKR/XS8Hei869HIDynOqkDMwJovJm1Am3+y6ToCqLqZ5s7B6an4+wS5x6hFyT+vR1NvjMe5nBtCcqkyYPclE32UwYi6ZsKwK1CjY6xjEgmsrqG1GSpsR7DCuiW2uQDPv97GvOLn9nkaLylP2uZ9qbIiM9517PsXKk0B5a+n+dQyj+cBXuGvvZjmacLiQjERA3N3/M5mDymdaqQKCthahidJrC9svUmu8AL4y0kkB//ACWlcyJR/QfhDTpFKBrFMpMc0bv7so+AeEs7YZo0hd b+rkymtG 8MzihfJltPHXfX7CKGYIswcQ70hIrC1ZoRm0KV2mvz3pzL2WoLKMhxrB3IYBgXUAJXFDkhIXfxYsJDQTV9dr4S2Sm7UabQNFxMvOedu94iTGs/v+3tIcDfMIbwZsyCu+B5UO+cFXbLhWVBifcicvydZCoLAD7QYxlohjcImQSw9nTpWsfMYJMDvRxd9c8EdqCEk2vai7bDiFntMoS3ePLGMJmqFQAGLRt9Zk//5Mm1BFGkfcHOM8MNtXegt1XnjibbFVS8R+535qTc8eIIzfzLM/K6wAlAc97LrzpGZx0O9WhTphqISRg1wf5pQSAkBd5euQqWqJcxDrNvEoaQNivqzlUxQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Teach the fork code to properly copy pfnmaps for pmd/pud levels. Pud is much easier, the write bit needs to be persisted though for writable and shared pud mappings like PFNMAP ones, otherwise a follow up write in either parent or child process will trigger a write fault. Do the same for pmd level. Signed-off-by: Peter Xu --- mm/huge_memory.c | 29 ++++++++++++++++++++++++++--- 1 file changed, 26 insertions(+), 3 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index e2c314f631f3..15418ffdd377 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1559,6 +1559,24 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, pgtable_t pgtable = NULL; int ret = -ENOMEM; + pmd = pmdp_get_lockless(src_pmd); + if (unlikely(pmd_special(pmd))) { + dst_ptl = pmd_lock(dst_mm, dst_pmd); + src_ptl = pmd_lockptr(src_mm, src_pmd); + spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); + /* + * No need to recheck the pmd, it can't change with write + * mmap lock held here. + * + * Meanwhile, making sure it's not a CoW VMA with writable + * mapping, otherwise it means either the anon page wrongly + * applied special bit, or we made the PRIVATE mapping be + * able to wrongly write to the backend MMIO. + */ + VM_WARN_ON_ONCE(is_cow_mapping(src_vma->vm_flags) && pmd_write(pmd)); + goto set_pmd; + } + /* Skip if can be re-fill on fault */ if (!vma_is_anonymous(dst_vma)) return 0; @@ -1640,7 +1658,9 @@ int copy_huge_pmd(struct mm_struct *dst_mm, struct mm_struct *src_mm, pmdp_set_wrprotect(src_mm, addr, src_pmd); if (!userfaultfd_wp(dst_vma)) pmd = pmd_clear_uffd_wp(pmd); - pmd = pmd_mkold(pmd_wrprotect(pmd)); + pmd = pmd_wrprotect(pmd); +set_pmd: + pmd = pmd_mkold(pmd); set_pmd_at(dst_mm, addr, dst_pmd, pmd); ret = 0; @@ -1686,8 +1706,11 @@ int copy_huge_pud(struct mm_struct *dst_mm, struct mm_struct *src_mm, * TODO: once we support anonymous pages, use * folio_try_dup_anon_rmap_*() and split if duplicating fails. */ - pudp_set_wrprotect(src_mm, addr, src_pud); - pud = pud_mkold(pud_wrprotect(pud)); + if (is_cow_mapping(vma->vm_flags) && pud_write(pud)) { + pudp_set_wrprotect(src_mm, addr, src_pud); + pud = pud_wrprotect(pud); + } + pud = pud_mkold(pud); set_pud_at(dst_mm, addr, dst_pud, pud); ret = 0;