From patchwork Fri Nov 18 01:10:13 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13047533 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 34881C4332F for ; Fri, 18 Nov 2022 01:10:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 733AC6B0072; Thu, 17 Nov 2022 20:10:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 71EA46B0073; Thu, 17 Nov 2022 20:10:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 58D038E0001; Thu, 17 Nov 2022 20:10:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 484646B0071 for ; Thu, 17 Nov 2022 20:10:33 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 1C34AC05A7 for ; Fri, 18 Nov 2022 01:10:33 +0000 (UTC) X-FDA: 80144782746.24.BB902E0 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf10.hostedemail.com (Postfix) with ESMTP id 8E5CAC0004 for ; Fri, 18 Nov 2022 01:10:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1668733831; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=bYQPkfNurDnqMfPJBlffYaCbgA2VoHiv3ENn4+XiB7o=; b=FUDeYrgiUOWLN+6z8n/PYEETLVPkU3FRk+5m+vO4WiLNPunJXPnoWr+3G0MG5MnRwp6qwS Uoc33iUWFjxFGh9ysICh/1WR+twnwD6LxoiGMM4s9zipkkTfwkSFvmnTYPmBJwQgb/h5bH rhvTyfNihwZpH2ksdDmfH95pdnSG/jY= Received: from mail-qt1-f199.google.com (mail-qt1-f199.google.com [209.85.160.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-644-SNvHEWvaO3KUP_ULKvxpbA-1; Thu, 17 Nov 2022 20:10:30 -0500 X-MC-Unique: SNvHEWvaO3KUP_ULKvxpbA-1 Received: by mail-qt1-f199.google.com with SMTP id fz10-20020a05622a5a8a00b003a4f466998cso3416517qtb.16 for ; Thu, 17 Nov 2022 17:10:29 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=bYQPkfNurDnqMfPJBlffYaCbgA2VoHiv3ENn4+XiB7o=; b=RKVkjDVSyLG/nbe2I7GhTw+rY/7vIXQZXvJdag60UEfFzlGIBdSfFFOPsxfGecC71N qR1QjWbSmkayr71ZsK958z6JjzkrVU+mLsAxnkcbOt771JH6d/SiGDVLCBkHcUrQ8kXG rHwHTRuDTDY3lk/cDPr7qGXONHhuMFQhgwhoAZt0r0vaNaeZkZiRgX0T8iS2m01M1nax t+P5WGCXeiGd//VpKVS5NLkzk5zif1RzVmwGE5kb8H9XuSI/wukzTP0PlRpC/RlJmTqU ErEH3Ksl5PG3wpNU5F2zjAWEHrThTlr6XzUS+jlnPlbdqPKv5sYxmf5/KOjnTZNK0pt6 6QMg== X-Gm-Message-State: ANoB5plY11/a/IaPJLJsA4c2FNmZJUiy/WqmOGQncwbqDZWcOvQK+uNh HByqlNlPuSL8qmM3a3UcP2h4iTlkmOrAKf0fHtmYCmAkb+8mzMAI9FNfSk9hDPWSGGWHn3XAXyk FSxsQgXbI//HGwUlk2rKYY+pAAsOd08H7HH6KTyL0Crf6WmqxGsgHOJFSITj/ X-Received: by 2002:ad4:5223:0:b0:4bb:59c1:3dec with SMTP id r3-20020ad45223000000b004bb59c13decmr4926703qvq.39.1668733829254; Thu, 17 Nov 2022 17:10:29 -0800 (PST) X-Google-Smtp-Source: AA0mqf6tBqdGqowiMujHJsML0L9StU8ix8zKxqShWsBrzFFmO4mXe4FZgt1bxkP0c1tgWyuzWpxagg== X-Received: by 2002:ad4:5223:0:b0:4bb:59c1:3dec with SMTP id r3-20020ad45223000000b004bb59c13decmr4926671qvq.39.1668733828859; Thu, 17 Nov 2022 17:10:28 -0800 (PST) Received: from x1n.redhat.com (bras-base-aurron9127w-grc-46-70-31-27-79.dsl.bell.ca. [70.31.27.79]) by smtp.gmail.com with ESMTPSA id u7-20020a05620a430700b006eed75805a2sm1491342qko.126.2022.11.17.17.10.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 17 Nov 2022 17:10:28 -0800 (PST) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Rik van Riel , Muchun Song , Andrew Morton , peterx@redhat.com, James Houghton , Nadav Amit , Andrea Arcangeli , David Hildenbrand , Miaohe Lin , Mike Kravetz Subject: [PATCH RFC v2 00/12] mm/hugetlb: Make huge_pte_offset() thread-safe for pmd unshare Date: Thu, 17 Nov 2022 20:10:13 -0500 Message-Id: <20221118011025.2178986-1-peterx@redhat.com> X-Mailer: git-send-email 2.37.3 MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1668733832; a=rsa-sha256; cv=none; b=6/ZTWRBuBf+s83yko7tzE+yO3MAjc6rzYYK0REV+2WVnG7BE0zgxb/pu/aszDc21h3LGAX uLIGg1b0iX1Qk4wIIK5qBCxhAVcWdBgjsz0JAia4vSPT1Fu3KUryQ4ZkyDgXgGuGsH0y0S QyXw2y/m+0jA4Ti4skwHBJllqfFja9w= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=FUDeYrgi; spf=pass (imf10.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1668733832; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=bYQPkfNurDnqMfPJBlffYaCbgA2VoHiv3ENn4+XiB7o=; b=el+NJndGEF7Q6/IRJZWWDXB4GUcrA+lSQJ+eB5uCQBS4BMOs+RBdgzGm5yHbEi5mmKPzyW DTodG15tumsbNyhCEAbZ/6FBaQp91GUhCz3gVNx/6/XAVOiog92djys/dCjVTojS2O7GKE kmCdurWELonUPNjx6WFWMYkGl0UeJiM= X-Stat-Signature: fh8kjxz8diay5bz6csog58n4m1wis4p1 X-Rspamd-Queue-Id: 8E5CAC0004 X-Rspamd-Server: rspam01 X-Rspam-User: Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=FUDeYrgi; spf=pass (imf10.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-HE-Tag: 1668733832-238505 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Based on latest mm-unstable (96aa38b69507). This can be seen as a follow-up series to Mike's recent hugetlb vma lock series for pmd unsharing, so this series also depends on that one. Hopefully this series can make it a more complete resolution for pmd unsharing. PS: so far no one strongly ACKed this, let me keep the RFC tag. But I think I'm already more confident than many of the RFCs I posted. PS2: there're a lot of changes comparing to rfcv1, so I'm just not adding the changelog. The whole idea is still the same, though. Problem ======= huge_pte_offset() is a major helper used by hugetlb code paths to walk a hugetlb pgtable. It's used mostly everywhere since that's needed even before taking the pgtable lock. huge_pte_offset() is always called with mmap lock held with either read or write. For normal memory types that's far enough, since any pgtable removal requires mmap write lock (e.g. munmap or mm destructions). However hugetlb has the pmd unshare feature, it means not only the pgtable page can be gone from under us when we're doing a walking, but also the pgtable page we're walking (even after unshared, in this case it can only be the huge PUD page which contains 512 huge pmd entries, with the vma VM_SHARED mapped). It's possible because even though freeing the pgtable page requires mmap write lock, it doesn't help us when we're walking on another mm's pgtable, so it's still on risk even if we're with the current->mm's mmap lock. The recent work from Mike on vma lock can resolve most of this already. It's achieved by forbidden pmd unsharing during the lock being taken, so no further risk of the pgtable page being freed. It means if we can take the vma lock around all huge_pte_offset() callers it'll be safe. There're already a bunch of them that we did as per the latest mm-unstable, but also quite a few others that we didn't for various reasons. E.g. it may not be applicable for not-allow-to-sleep contexts like FOLL_NOWAIT. Or, huge_pmd_share() is actually a tricky user of huge_pte_offset(), because even if we took the vma lock, we're walking on another mm's vma! Taking vma lock for all the vmas are probably not gonna work. I have totally no report showing that I can trigger such a race, but from code wise I never see anything that stops the race from happening. This series is trying to resolve that problem. Resolution ========== What this patch proposed, besides using the vma lock, is that we can also use other ways to protect the pgtable page from being freed from under us in huge_pte_offset() context. The idea is kind of similar to RCU fast-gup. Note that fast-gup is very safe regarding pmd unsharing even before vma lock, because fast-gup relies on RCU to protect walking any pgtable page, including another mm's. So fast-gup will never hit a freed page even if pmd sharing is possible. To apply similar same idea to huge_pte_offset(), it means with proper RCU protection the pte_t* pointer returned from huge_pte_offset() can also be always safe to access and de-reference, along with the pgtable lock that was bound to the pgtable page. Note that RCU will only work to protect pgtables if MMU_GATHER_RCU_TABLE_FREE=y. For the rest we need to disable irq. Of course, the whole locking idea is not needed if pmd sharing is not possible at all, or, on private hugetlb mappings. Patch Layout ============ Patch 1-3: cleanup, or dependency of the follow up patches Patch 4: the core patch to introduce hugetlb walker lock Patch 5-11: each patch resolves one possible race condition Patch 12: introduce hugetlb_walk() to replace huge_pte_offset() Tests ===== Only lightly tested on hugetlb kselftests including uffd. Comments welcomed, thanks. Peter Xu (12): mm/hugetlb: Let vma_offset_start() to return start mm/hugetlb: Move swap entry handling into vma lock for fault mm/hugetlb: Don't wait for migration entry during follow page mm/hugetlb: Add pgtable walker lock mm/hugetlb: Make userfaultfd_huge_must_wait() safe to pmd unshare mm/hugetlb: Protect huge_pmd_share() with walker lock mm/hugetlb: Use hugetlb walker lock in hugetlb_follow_page_mask() mm/hugetlb: Use hugetlb walker lock in follow_hugetlb_page() mm/hugetlb: Use hugetlb walker lock in hugetlb_vma_maps_page() mm/hugetlb: Use hugetlb walker lock in walk_hugetlb_range() mm/hugetlb: Use hugetlb walker lock in page_vma_mapped_walk() mm/hugetlb: Introduce hugetlb_walk() arch/s390/mm/gmap.c | 2 + fs/hugetlbfs/inode.c | 41 +++++++------- fs/proc/task_mmu.c | 2 + fs/userfaultfd.c | 24 ++++++--- include/linux/hugetlb.h | 112 +++++++++++++++++++++++++++++++++++++++ include/linux/pagewalk.h | 9 +++- include/linux/rmap.h | 4 ++ mm/hugetlb.c | 97 +++++++++++++++++---------------- mm/page_vma_mapped.c | 7 ++- mm/pagewalk.c | 6 +-- 10 files changed, 224 insertions(+), 80 deletions(-)