From patchwork Mon Feb 6 11:28:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Stevens X-Patchwork-Id: 13129690 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0363FC05027 for ; Mon, 6 Feb 2023 11:29:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6F5C76B0072; Mon, 6 Feb 2023 06:29:52 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6A5896B0073; Mon, 6 Feb 2023 06:29:52 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 51F1A6B0074; Mon, 6 Feb 2023 06:29:52 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 3E4B36B0072 for ; Mon, 6 Feb 2023 06:29:52 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id DC27AC0D0A for ; Mon, 6 Feb 2023 11:29:51 +0000 (UTC) X-FDA: 80436647382.27.DC3B2E9 Received: from mail-pj1-f45.google.com (mail-pj1-f45.google.com [209.85.216.45]) by imf22.hostedemail.com (Postfix) with ESMTP id 0C5ADC0013 for ; Mon, 6 Feb 2023 11:29:49 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=O3iioc3I; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf22.hostedemail.com: domain of stevensd@chromium.org designates 209.85.216.45 as permitted sender) smtp.mailfrom=stevensd@chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675682990; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=utDdyziP9bO+Fsgkcg9MC2JxaG0rPj2uHxgkJ2l7vzk=; b=rvlnrbHOTNRvGWiH1WkQI35DGJ0UUp1Wi+50aTnqRs0YVxug7ut9uxFAf+KOvaeuN61PMU yrFjapkVAi/M88aDx4KSt8VBqOadZe+FVwNdVpkX5B6Z0L0YB3ohekOyDNdlxnJZxJt3/A s7UQMGP2M2sITzD6EUnBq7HxPsmcjBg= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=O3iioc3I; dmarc=pass (policy=none) header.from=chromium.org; spf=pass (imf22.hostedemail.com: domain of stevensd@chromium.org designates 209.85.216.45 as permitted sender) smtp.mailfrom=stevensd@chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675682990; a=rsa-sha256; cv=none; b=oGHlpYLL+dbsABCicxaV49iNGas23wJqnHztxKaMW5NyN8GiGxmGiFSKoBXyPqvFskXndO tu2DfJoXx8TEHwL1eEityFNxEF+GPg5Y29VUyhEGi2Oc2C0gNXDclVvxxa9RHvWLoqhWPK ezYWk+q2zsQQIv2TdqDQ2VZTEDYBaHg= Received: by mail-pj1-f45.google.com with SMTP id o16-20020a17090ad25000b00230759a8c06so8009409pjw.2 for ; Mon, 06 Feb 2023 03:29:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=utDdyziP9bO+Fsgkcg9MC2JxaG0rPj2uHxgkJ2l7vzk=; b=O3iioc3IhBlAy1vP2an/YjZGVokv+0MZ7p6GDuiMlVKbiup1dzs1mB84FFkifLx1im IPgrQeDcAEouNtZZQRBPUr12fimPDJh6s3e9uODb3QlRiBgXZ8tdydrvU8CKuIMVXb73 IuVVUW7LfIeY3CSpm3noFrqMkB/OEcdsGGJoo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=utDdyziP9bO+Fsgkcg9MC2JxaG0rPj2uHxgkJ2l7vzk=; b=5L56NVu8vu1kWnrbKp+LBzWfRK60gM4rPh00ESEF0fuF+IDWjBUk4beGT8eVUjpCtx EDexxtYD8UVgCmH1m2lLZ2TmGThgxaDHSqctRkRAFMbHMss9PVi/7laySIYq46djNmfZ 5Qj+FRrXgSC+BfVlCRt3aq+FUUgPbw1eT7BfaNhISfK/oD6QFL4uns1s+3DUv7Sy6aj4 kJnMgPZ2u86J8pDOy2oqdaM/ghayH9nnFT8GmW7dp8QToWq/pBX+zKr0WA148DTQak1v /VMvyIP1fzxL8W/NNGm0uJVg1Np6z/d/0JDyg9hQPVo+vmQ1R3aY/IGdjHPLWJVGKWuT ETdg== X-Gm-Message-State: AO0yUKX9PHSvqFdMkzWqxHika34fyrbLvL8CIgIg1DdISgphMAHNEJGc PHvIQjpgXbdZz3RvByQ0W4tezoMatO4TrCGz X-Google-Smtp-Source: AK7set+PR0nogH0woEYsRyv10vuAB0a7gtb6hm6l7ukP1ANqZmj1N3Mjto6chdUMSCJfFdaUJkTjog== X-Received: by 2002:a17:902:cec1:b0:196:88e0:ea1a with SMTP id d1-20020a170902cec100b0019688e0ea1amr24354810plg.47.1675682988267; Mon, 06 Feb 2023 03:29:48 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:4a83:fb85:9f41:4c0c]) by smtp.gmail.com with UTF8SMTPSA id u2-20020a170902a60200b001962858f990sm6675831plq.164.2023.02.06.03.29.45 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 06 Feb 2023 03:29:47 -0800 (PST) From: David Stevens X-Google-Original-From: David Stevens To: linux-mm@kvack.org, Peter Xu Cc: Andrew Morton , "Kirill A . Shutemov" , Yang Shi , David Hildenbrand , Hugh Dickins , linux-kernel@vger.kernel.org, David Stevens Subject: [PATCH v2] mm/khugepaged: skip shmem with userfaultfd Date: Mon, 6 Feb 2023 20:28:56 +0900 Message-Id: <20230206112856.1802547-1-stevensd@google.com> X-Mailer: git-send-email 2.39.1.519.gcb327c4b5f-goog MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 0C5ADC0013 X-Stat-Signature: ia7artykqppyqnpqrc457wwmxps8mbe3 X-HE-Tag: 1675682989-982688 X-HE-Meta: U2FsdGVkX18daxvG25YuL+1NZtAT3LTWV93+hMO/Gigv9BMFp3TseKKujaw7M+4sCZD3FrOknnZQ/36isr1IlRiROnKH33qgiZsyezC61rQugeQclgUWtC3DAl4WORdV7otEJ4SBC9HdEyrpzkAqEbXo+t2NML3tPGZhvVPfjk//hYpL7eBijTRPSDgdbSK8KpSZIhBNgktgSo8MgeGbIfk2cAQk2qgNLMxlRHp/dclP859KiAeHm51wS2e+KooAOKYG+OhOgJaWF5OxsRS0nbQawV6jTrcLSa15p9I6QCNFrOcWRHTvzZwv1VaIVVL25I76fs+b4b9yynbve2EzRWHtylFdv7sJfCUjs0GquHgwznShgk4ZVRaXzVx5IwmAQrmJ8K4MRiw/+0bPZskpZbyO67X0sPOlDIbIqxfeczXpTZd0vScNQYkG5vHY7RH57qsi52+poPV5TnmkR5mz0GKJk/lSParAD7IsOc1n1QEZYxXuuYa5ItW/qvAXtiTDygdW69vqVtnqjL5crZJyINzR/+YN8tUwYSaOGM3g2Y+1zzzsiCNGp6VNkkaIhnBPAc/A+zVz709ONnJRUwp2eLf9bteuh+BPSiuEGbV+BXimFEJwfG0BEK+oZvM9cpQAa2/mYYjAGDenSu83Ofh+ctd9sXmostyg1zUh4DPHRvNCDkU8Jeyc0jo1HGOS05qLmx5awSfLdp6W//DFw98JzIbhl+8bPn1S+oNH4L0IEgc19oGk9VK+RBE1OtcqQTva0vAjf6m9ZC16lhM0/jqHepswEt7cl+MSO8NsB2o+iJaDp7T/kBSyzcJWACe36pifev8iChANb98LLxmBnJHtYPiF1Dchg5N5/zquOwEd0Nu5IijKMmaFdI2EWW9Z/IscAqAsWSx/oyQhynq5U2129/UBUoG/ivOUU2lXwMQPgTZ+6zlRFWArVuOT0FSFYueE6iY5APYNpEFnrU7/QMF SrmU4vax TVTToUUO2DfrG6RRstYG+7XEcCp5WoqWJKJJY2VIs+ksqKRVu6KOq6eQGHwgfEr4+jhUTgjxZ/9Ar6T1t3RpczgfI3bHKX11GEz7gNnvN0KTyrRo99Rl9YgCisNGtMCqARDhv8aFkoFgkpYesqX/VW+Oe7d6RT5QL8vtDWsKdW9egWpiiYvYiOULwquyPlHxvsFvX8hH8wsFwI1ABTb3b45yqXm1ws9ia0rlSpNHaVmokTeO3cd5CGSS0p86eRf+D8HQsZ0zpXTJDw8Hv7U7NUVrfi9QPYWC/S5GLWEKgjyv7uHxAJTcixYJ6kwpNM8fDIrfS X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: David Stevens Collapsing memory will result in any empty pages in the target range being filled by the new THP. If userspace has a userfaultfd registered with MODE_MISSING, for any page which it knows to be missing after registering the userfaultfd, it may expect a UFFD_EVENT_PAGEFAULT. Taking these two facts together, khugepaged needs to take care when collapsing pages in shmem to make sure it doesn't break the userfaultfd API. This change first makes sure that the intermediate page cache state during collapse is not visible by moving when gaps are filled to after the page cache lock is acquired for the final time. This is necessary because the synchronization provided by locking hpage is insufficient for functions which operate on the page cache without actually locking individual pages to examine their content (e.g. shmem_mfill_atomic_pte). This refactoring allows us to iterate over i_mmap to check for any VMAs with userfaultfds and then finalize the collapse if no such VMAs exist, all while holding the page cache lock. Since no mm locks are held, it is necessary to add smb_rmb/smb_wmb to ensure that userfaultfd updates to vm_flags are visible to khugepaged. However, no further locking of userfaultfd state is necessary. Although new userfaultfds can be registered concurrently with finalizing the collapse, any missing pages that are being replaced can no longer be observed by userspace, so there is no data race. This fix is targeted at khugepaged, but the change also applies to MADV_COLLAPSE. The fact that the intermediate page cache state before the rollback of a failed collapse can no longer be observed is technically a userspace-visible change (via at least SEEK_DATA and SEEK_END), but it is exceedingly unlikely that anything relies on being able to observe that transient state. Signed-off-by: David Stevens --- fs/userfaultfd.c | 2 ++ mm/khugepaged.c | 67 ++++++++++++++++++++++++++++++++++++++++-------- 2 files changed, 59 insertions(+), 10 deletions(-) diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c index cc694846617a..6ddfcff11920 100644 --- a/fs/userfaultfd.c +++ b/fs/userfaultfd.c @@ -114,6 +114,8 @@ static void userfaultfd_set_vm_flags(struct vm_area_struct *vma, const bool uffd_wp_changed = (vma->vm_flags ^ flags) & VM_UFFD_WP; vma->vm_flags = flags; + /* Pairs with smp_rmb() in khugepaged's collapse_file() */ + smp_wmb(); /* * For shared mappings, we want to enable writenotify while * userfaultfd-wp is enabled (see vma_wants_writenotify()). We'll simply diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 79be13133322..97435c226b18 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -55,6 +55,7 @@ enum scan_result { SCAN_CGROUP_CHARGE_FAIL, SCAN_TRUNCATED, SCAN_PAGE_HAS_PRIVATE, + SCAN_PAGE_FILLED, }; #define CREATE_TRACE_POINTS @@ -1725,8 +1726,8 @@ static int retract_page_tables(struct address_space *mapping, pgoff_t pgoff, * - allocate and lock a new huge page; * - scan page cache replacing old pages with the new one * + swap/gup in pages if necessary; - * + fill in gaps; * + keep old pages around in case rollback is required; + * - finalize updates to the page cache; * - if replacing succeeds: * + copy data over; * + free old pages; @@ -1747,6 +1748,7 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, XA_STATE_ORDER(xas, &mapping->i_pages, start, HPAGE_PMD_ORDER); int nr_none = 0, result = SCAN_SUCCEED; bool is_shmem = shmem_file(file); + bool i_mmap_locked = false; int nr = 0; VM_BUG_ON(!IS_ENABLED(CONFIG_READ_ONLY_THP_FOR_FS) && !is_shmem); @@ -1780,8 +1782,14 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, /* * At this point the hpage is locked and not up-to-date. - * It's safe to insert it into the page cache, because nobody would - * be able to map it or use it in another way until we unlock it. + * + * While iterating, we may drop the page cache lock multiple times. It + * is safe to replace pages in the page cache with hpage while doing so + * because nobody is able to map or otherwise access the content of + * hpage until we unlock it. However, we cannot insert hpage into empty + * indicies until we know we won't have to drop the page cache lock + * again, as doing so would let things which only check the presence + * of pages in the page cache see a state that may yet be rolled back. */ xas_set(&xas, start); @@ -1802,13 +1810,12 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, result = SCAN_TRUNCATED; goto xa_locked; } - xas_set(&xas, index); + xas_set(&xas, index + 1); } if (!shmem_charge(mapping->host, 1)) { result = SCAN_FAIL; goto xa_locked; } - xas_store(&xas, hpage); nr_none++; continue; } @@ -1967,6 +1974,46 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, put_page(page); goto xa_unlocked; } + + if (nr_none) { + struct vm_area_struct *vma; + int nr_none_check = 0; + + xas_unlock_irq(&xas); + i_mmap_lock_read(mapping); + i_mmap_locked = true; + xas_lock_irq(&xas); + + xas_set(&xas, start); + for (index = start; index < end; index++) { + if (!xas_next(&xas)) + nr_none_check++; + } + + if (nr_none != nr_none_check) { + result = SCAN_PAGE_FILLED; + goto xa_locked; + } + + /* + * If userspace observed a missing page in a VMA with an armed + * userfaultfd, then it might expect a UFFD_EVENT_PAGEFAULT for + * that page, so we need to roll back to avoid suppressing such + * an event. Any userfaultfds armed after this point will not be + * able to observe any missing pages, since the page cache is + * locked until after the collapse is completed. + * + * Pairs with smp_wmb() in userfaultfd_set_vm_flags(). + */ + smp_rmb(); + vma_interval_tree_foreach(vma, &mapping->i_mmap, start, start) { + if (userfaultfd_missing(vma)) { + result = SCAN_EXCEED_NONE_PTE; + goto xa_locked; + } + } + } + nr = thp_nr_pages(hpage); if (is_shmem) @@ -2000,6 +2047,8 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, xas_store(&xas, hpage); xa_locked: xas_unlock_irq(&xas); + if (i_mmap_locked) + i_mmap_unlock_read(mapping); xa_unlocked: /* @@ -2065,15 +2114,13 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, } xas_set(&xas, start); - xas_for_each(&xas, page, end - 1) { + end = index; + for (index = start; index < end; index++) { + xas_next(&xas); page = list_first_entry_or_null(&pagelist, struct page, lru); if (!page || xas.xa_index < page->index) { - if (!nr_none) - break; nr_none--; - /* Put holes back where they were */ - xas_store(&xas, NULL); continue; }