From patchwork Tue Jan 24 08:43:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muhammad Usama Anjum X-Patchwork-Id: 13113662 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 38DA2C25B4E for ; Tue, 24 Jan 2023 08:44:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6718F6B0072; Tue, 24 Jan 2023 03:44:42 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5FC7B6B0074; Tue, 24 Jan 2023 03:44:42 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 474616B0075; Tue, 24 Jan 2023 03:44:42 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 3885B6B0072 for ; Tue, 24 Jan 2023 03:44:42 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 04366A04C5 for ; Tue, 24 Jan 2023 08:44:41 +0000 (UTC) X-FDA: 80389056804.06.59DB611 Received: from madras.collabora.co.uk (madras.collabora.co.uk [46.235.227.172]) by imf10.hostedemail.com (Postfix) with ESMTP id 313A2C0011 for ; Tue, 24 Jan 2023 08:44:39 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=collabora.com header.s=mail header.b="WBW0lX/o"; spf=pass (imf10.hostedemail.com: domain of usama.anjum@collabora.com designates 46.235.227.172 as permitted sender) smtp.mailfrom=usama.anjum@collabora.com; dmarc=pass (policy=none) header.from=collabora.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674549880; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=LOGrTkEfcZgkiby0VbeQFHl5qANoxA+cj2slCCeX0rg=; b=jB1sOE/U90RG4CPTNktlL6zVnCHEm/o1Ja+qqADg8/Nx/MFNqpBW9IkRlb0ekZ/Zq2GCFn tbubWyfwhCyII4C6vedbQoKFC+hheZw9oZr1VjUhkTN8WgDiZmU7M8HwZ0UOenAR7GTztt ALMHZ20QTewxoThZMtSHMAom45kLL2w= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=collabora.com header.s=mail header.b="WBW0lX/o"; spf=pass (imf10.hostedemail.com: domain of usama.anjum@collabora.com designates 46.235.227.172 as permitted sender) smtp.mailfrom=usama.anjum@collabora.com; dmarc=pass (policy=none) header.from=collabora.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674549880; a=rsa-sha256; cv=none; b=TpGwZY9DbepNavluwbRg6CZnJPBd0GMoIEeFhR5v+Vn0fmKiUXP88bX/OLy84X8qPt2CX+ dPh5QT9xhThUbMsllI6BEaFVzr8eZD5KvfgYcA4vXaTvkeG7YR/T+/YdSYTA4yI4TFkDVT vhyAuumqD4sEsSho9eCPRy4s6ZDjZa4= Received: from localhost.localdomain (unknown [39.45.186.163]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: usama.anjum) by madras.collabora.co.uk (Postfix) with ESMTPSA id B4E1D6602E2B; Tue, 24 Jan 2023 08:44:31 +0000 (GMT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=collabora.com; s=mail; t=1674549879; bh=m/++XbQn9BatMFvanRp2XUIOa0zCFwRYse69T8hHq3A=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=WBW0lX/oQGiGf6RmlizOjOfaR7O9XvMYK0Ybxo3oXOnEJr2A80AStD0iglUEUaiE9 h6TIqMPwPx0t0gL+dH18QDgZgBgAghyuEI+veHKt7sOKaKKgaAgNviYWQvmpMxbZHl wuddkQF1DzY+FA+zg3B1Qxw9sYPr3cLs2km9v/Q3EFnUCV2++W21yrxVy/sv4ZrHky 1tWRq0JlGBvfQMpTBkforZU4Fghyfn0eh6blchhWnyp++sK7SFAkLTXCYPZG8IdW2F K1HKDDhFoSKH7UBSXDAsczZ+Qiw9ioI5FcVOibaRLRwwDeh8JFr0Mu5pGVDyzF+ydK yzbVjt8eJK/mg== From: Muhammad Usama Anjum To: Peter Xu , David Hildenbrand , Andrew Morton , =?utf-8?b?TWljaGHFgiBNaXJvc8WC?= =?utf-8?b?YXc=?= , Andrei Vagin , Danylo Mocherniuk , Paul Gofman , Cyrill Gorcunov Cc: Alexander Viro , Shuah Khan , Christian Brauner , Yang Shi , Vlastimil Babka , "Liam R . Howlett" , Yun Zhou , Suren Baghdasaryan , Alex Sierra , Muhammad Usama Anjum , Matthew Wilcox , Pasha Tatashin , Mike Rapoport , Nadav Amit , Axel Rasmussen , "Gustavo A . R . Silva" , Dan Williams , linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-kselftest@vger.kernel.org, Greg KH , kernel@collabora.com Subject: [PATCH v8 1/4] userfaultfd: Add UFFD WP Async support Date: Tue, 24 Jan 2023 13:43:20 +0500 Message-Id: <20230124084323.1363825-2-usama.anjum@collabora.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20230124084323.1363825-1-usama.anjum@collabora.com> References: <20230124084323.1363825-1-usama.anjum@collabora.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 313A2C0011 X-Stat-Signature: ekssxxe98xnm8s97qrt7wgtktw171zgi X-HE-Tag: 1674549879-362840 X-HE-Meta: U2FsdGVkX1/fsKWdrFswvK2qtRNa7c9kSZ2pgNnv4gHMrwITZN9CLKUoPciJov0jFR6035IL3gpQ1L7GI3RnSjMehGhZ4q746SDl2faw348NHIK95FdIjIxY5/J7E3Z+8zkc0R2vDlMHpU+ELXp2RdGESaWSB7Xh/vjpfkqBhrXMN+WEViriZQiQYnx+fw74xT6emdVzxfaFnxVSgk2Ww55KnGBoY30z/pVnMPkHq/TV/zo7fG4jLppa9TcLgwnxFuwpFeh7pqv8FrxbKa0W/U6vFgObhkmISYx5UPEAf2/enM7fEm7n9IOhh8ZC/jHUcxBUhGHhzeq5tc7u1Qx6nl1s3cXOFaq+oNxAu733iJyZHizHUWaj0jMnchkrQgH5sdMyb0X3DXw9YoJaNm3hLrJJrSIIKAK4dxg7QYX5V0cKV12vShHnN9QQMC5aNIS4qch8AT796/fDEDnPG3A7ErIAH4kXVL4Bgx1/x8vfe0u9X8UJCNDLNw3K2zx6ssPnOrBgR/sIka+YGVJD54VHO0IBBRrt/qlck037LQWjOHSO6zi7LI7Jrd4YI4Q21ZE1dASwwnCm+oY9BddiNlB3gmDj0Kv11HloC1EQI/CsqPoB3IcT2w9EesHSV9z4v1dVzNtFlJiOBj/KmKwQebHPEq64nkAd/1TqqScmpzgJVniM1uq0sVVCQC2B7FwvE9f5vszhevDwJDvuOE5vam7WdI6cDEakVD7an8YUzWlloZzT3peUuqOsZ1Nae+A+tqyu5MNvf/ZnykBGPQYIgNGrBNlwp5hsBLtKf2YHHxGEoVPLrNVJozq2vg8eRRcZyLQaVfM8jvxP6LG+Gk7P+Qk/yvhWlUb7heBgJSVrVPaRtC3YrxGjM0hudfsVDggQLj8ndGlgLV3MC9dCRL+cgF74LIu5cAnpXzOV8BnGNvf6ncWtW7MrkTzf6w1+1zUyxx0dK9H6C4PuVyJM7ikFgTd sxQfawGq G0EwuRixQ1iuEvHSSBtcrlQc4x8iBkdwJaay44eKtZkYr3q75BBu/qVsyY30hwo4J4/apyBiJhwdbQ4ReciRNcJQiulr/mdPsFP0PnTEfZm0T8v5NTtmiZ3pTGjxZwz1UnRohO+cjw1ivy6H563+4WVyNrZVPdA8BCIjw3MfSnQ6ADeyITwLOREddcS5Uw3yHQE1SwT7+xAX1fgnJ/XkuURihZ6HvS6sFG7+tSpaLyFJgIsbnW+l8KI3tbPjq2KDLAcC+kaC+VWvwGSX0j20HqdWxpCK0Axz7HefW X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add new WP Async mode (UFFD_FEATURE_WP_ASYNC) which resolves the page faults on its own. It can be used to track that which pages have been written-to from the time the pages were write-protected. It is very efficient way to track the changes as uffd is by nature pte/pmd based. UFFD synchronous WP sends the page faults to the userspace where the pages which have been written-to can be tracked. But it is not efficient. This is why this asynchronous version is being added. After setting the WP Async, the pages which have been written to can be found in the pagemap file or information can be obtained from the PAGEMAP_IOCTL. Suggested-by: Peter Xu Signed-off-by: Muhammad Usama Anjum --- Changes in v7: - Remove UFFDIO_WRITEPROTECT_MODE_ASYNC_WP and add UFFD_FEATURE_WP_ASYNC - Handle automatic page fault resolution in better way (thanks to Peter) --- fs/userfaultfd.c | 11 +++++++++++ include/linux/userfaultfd_k.h | 6 ++++++ include/uapi/linux/userfaultfd.h | 8 +++++++- mm/memory.c | 29 +++++++++++++++++++++++++++-- 4 files changed, 51 insertions(+), 3 deletions(-) diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c index 15a5bf765d43..b82af02092ce 100644 --- a/fs/userfaultfd.c +++ b/fs/userfaultfd.c @@ -1867,6 +1867,10 @@ static int userfaultfd_writeprotect(struct userfaultfd_ctx *ctx, mode_wp = uffdio_wp.mode & UFFDIO_WRITEPROTECT_MODE_WP; mode_dontwake = uffdio_wp.mode & UFFDIO_WRITEPROTECT_MODE_DONTWAKE; + /* Write protection cannot be disabled in case of aync WP */ + if (!mode_wp && (ctx->features & UFFD_FEATURE_WP_ASYNC)) + return -EINVAL; + if (mode_wp && mode_dontwake) return -EINVAL; @@ -1950,6 +1954,13 @@ static int userfaultfd_continue(struct userfaultfd_ctx *ctx, unsigned long arg) return ret; } +int userfaultfd_wp_async(struct vm_area_struct *vma) +{ + struct userfaultfd_ctx *ctx = vma->vm_userfaultfd_ctx.ctx; + + return (ctx && (ctx->features & UFFD_FEATURE_WP_ASYNC)); +} + static inline unsigned int uffd_ctx_features(__u64 user_features) { /* diff --git a/include/linux/userfaultfd_k.h b/include/linux/userfaultfd_k.h index 9df0b9a762cc..5db51fccae1d 100644 --- a/include/linux/userfaultfd_k.h +++ b/include/linux/userfaultfd_k.h @@ -179,6 +179,7 @@ extern int userfaultfd_unmap_prep(struct mm_struct *mm, unsigned long start, unsigned long end, struct list_head *uf); extern void userfaultfd_unmap_complete(struct mm_struct *mm, struct list_head *uf); +extern int userfaultfd_wp_async(struct vm_area_struct *vma); #else /* CONFIG_USERFAULTFD */ @@ -274,6 +275,11 @@ static inline bool uffd_disable_fault_around(struct vm_area_struct *vma) return false; } +int userfaultfd_wp_async(struct vm_area_struct *vma) +{ + return false; +} + #endif /* CONFIG_USERFAULTFD */ static inline bool pte_marker_entry_uffd_wp(swp_entry_t entry) diff --git a/include/uapi/linux/userfaultfd.h b/include/uapi/linux/userfaultfd.h index 005e5e306266..f4252ef40071 100644 --- a/include/uapi/linux/userfaultfd.h +++ b/include/uapi/linux/userfaultfd.h @@ -38,7 +38,8 @@ UFFD_FEATURE_MINOR_HUGETLBFS | \ UFFD_FEATURE_MINOR_SHMEM | \ UFFD_FEATURE_EXACT_ADDRESS | \ - UFFD_FEATURE_WP_HUGETLBFS_SHMEM) + UFFD_FEATURE_WP_HUGETLBFS_SHMEM | \ + UFFD_FEATURE_WP_ASYNC) #define UFFD_API_IOCTLS \ ((__u64)1 << _UFFDIO_REGISTER | \ (__u64)1 << _UFFDIO_UNREGISTER | \ @@ -203,6 +204,10 @@ struct uffdio_api { * * UFFD_FEATURE_WP_HUGETLBFS_SHMEM indicates that userfaultfd * write-protection mode is supported on both shmem and hugetlbfs. + * + * UFFD_FEATURE_WP_ASYNC indicates that userfaultfd write-protection + * asynchronous mode is supported in which the write fault is automatically + * resolved and write-protection is un-set. */ #define UFFD_FEATURE_PAGEFAULT_FLAG_WP (1<<0) #define UFFD_FEATURE_EVENT_FORK (1<<1) @@ -217,6 +222,7 @@ struct uffdio_api { #define UFFD_FEATURE_MINOR_SHMEM (1<<10) #define UFFD_FEATURE_EXACT_ADDRESS (1<<11) #define UFFD_FEATURE_WP_HUGETLBFS_SHMEM (1<<12) +#define UFFD_FEATURE_WP_ASYNC (1<<13) __u64 features; __u64 ioctls; diff --git a/mm/memory.c b/mm/memory.c index 4000e9f017e0..8c03b133d483 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3351,6 +3351,18 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf) if (likely(!unshare)) { if (userfaultfd_pte_wp(vma, *vmf->pte)) { + if (userfaultfd_wp_async(vma)) { + /* + * Nothing needed (cache flush, TLB invalidations, + * etc.) because we're only removing the uffd-wp bit, + * which is completely invisible to the user. This + * falls through to possible CoW. + */ + pte_unmap_unlock(vmf->pte, vmf->ptl); + set_pte_at(vma->vm_mm, vmf->address, vmf->pte, + pte_clear_uffd_wp(*vmf->pte)); + return 0; + } pte_unmap_unlock(vmf->pte, vmf->ptl); return handle_userfault(vmf, VM_UFFD_WP); } @@ -4812,8 +4824,21 @@ static inline vm_fault_t wp_huge_pmd(struct vm_fault *vmf) if (vma_is_anonymous(vmf->vma)) { if (likely(!unshare) && - userfaultfd_huge_pmd_wp(vmf->vma, vmf->orig_pmd)) - return handle_userfault(vmf, VM_UFFD_WP); + userfaultfd_huge_pmd_wp(vmf->vma, vmf->orig_pmd)) { + if (userfaultfd_wp_async(vmf->vma)) { + /* + * Nothing needed (cache flush, TLB invalidations, + * etc.) because we're only removing the uffd-wp bit, + * which is completely invisible to the user. This + * falls through to possible CoW. + */ + set_pmd_at(vmf->vma->vm_mm, vmf->address, vmf->pmd, + pmd_clear_uffd_wp(*vmf->pmd)); + return 0; + } else { + return handle_userfault(vmf, VM_UFFD_WP); + } + } return do_huge_pmd_wp_page(vmf); }