From patchwork Tue Jul 11 12:52:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muhammad Usama Anjum X-Patchwork-Id: 13308667 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 70B21EB64DD for ; Tue, 11 Jul 2023 12:53:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F12716B0074; Tue, 11 Jul 2023 08:53:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EC2296B0075; Tue, 11 Jul 2023 08:53:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D635C6B0078; Tue, 11 Jul 2023 08:53:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id C7BBD6B0074 for ; Tue, 11 Jul 2023 08:53:26 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 8ECD71C7ECE for ; Tue, 11 Jul 2023 12:53:26 +0000 (UTC) X-FDA: 80999322012.22.72BB57A Received: from madras.collabora.co.uk (madras.collabora.co.uk [46.235.227.172]) by imf24.hostedemail.com (Postfix) with ESMTP id ADB8F180018 for ; Tue, 11 Jul 2023 12:53:24 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=collabora.com header.s=mail header.b=bTSGMdLj; dmarc=pass (policy=quarantine) header.from=collabora.com; spf=pass (imf24.hostedemail.com: domain of usama.anjum@collabora.com designates 46.235.227.172 as permitted sender) smtp.mailfrom=usama.anjum@collabora.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689080004; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=e8HTDCgtNb7469DXumtszNr5ZT2YHvX/HJrNQANJT+w=; b=MdQiHEQqQF8z0840qvlpCScyg6JIZzEYEs/jKElZIFwQKrdawnlQfcapdWi1oYoBjDPSWx oBrV/bAyLrjXNIBSDqzWMVQQLw4TfORF68Zwj1Uy/g7duFuwzzTExMdf8A/bt4HD8rv9IL hft173v6y9way1N4venSlsSFSmVWJKA= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=collabora.com header.s=mail header.b=bTSGMdLj; dmarc=pass (policy=quarantine) header.from=collabora.com; spf=pass (imf24.hostedemail.com: domain of usama.anjum@collabora.com designates 46.235.227.172 as permitted sender) smtp.mailfrom=usama.anjum@collabora.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689080004; a=rsa-sha256; cv=none; b=MVQCse4K2ByG0DVZJ+0HhlN/MvBjM07V3dpeDNkJtnHAz+eGtC+6s/Nyn6mfBOEQyTilrY gClNwazxPFx9OJKkSmrgJZ+3VddXl96I40THTP3kTk8GUiUZ8uQWMetQhqoTZTyWHqiVG9 q1Z779IsIY3DKq30ez/K3WBS/KHa+sg= Received: from localhost.localdomain (unknown [119.155.40.122]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: usama.anjum) by madras.collabora.co.uk (Postfix) with ESMTPSA id D3ADC6607011; Tue, 11 Jul 2023 13:53:16 +0100 (BST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=collabora.com; s=mail; t=1689080003; bh=/J8HiHN8UsUk9lpcs8kZ/Et3MGSZkOMcYtXMhpxXRsQ=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=bTSGMdLj/ZbVHJS0gg6/aaffarJrp7VZQ5aXxnJfWkayqvXcw/6WrvsysfVb6qLfY y3vamGp93mNbDQ/8lljG3PfEW5Y9bjxfrNzfvhAfd3tF7U3dJBEPtCt8q6xa0XDUlB wF1sgr03kz3jKAy574gbz5z1PdDgOheZVHGaS1tU5RwFVFy5vWKskLwD2CZrDsEyRZ +vverEBqfCq3nJD6mUsypAH+oEIn+32EZ0LSxStPGqYmvKDjaCSGAwSuI2NEhw+0in LBY4MujEn4uG1SpPigN4zrAfUM9z08QmkJFTRTL5qbc+EQ/HvrWRsR6ByP6ma5xchR lWkYkZ6uPS3FA== From: Muhammad Usama Anjum To: Peter Xu , David Hildenbrand , Andrew Morton , =?utf-8?b?TWljaGHFgiBNaXJvc8WC?= =?utf-8?b?YXc=?= , Andrei Vagin , Danylo Mocherniuk , Paul Gofman , Cyrill Gorcunov , Mike Rapoport , Nadav Amit Cc: Alexander Viro , Shuah Khan , Christian Brauner , Yang Shi , Vlastimil Babka , "Liam R . Howlett" , Yun Zhou , Suren Baghdasaryan , Alex Sierra , Muhammad Usama Anjum , Matthew Wilcox , Pasha Tatashin , Axel Rasmussen , "Gustavo A . R . Silva" , Dan Williams , linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-kselftest@vger.kernel.org, Greg KH , kernel@collabora.com Subject: [PATCH v24 1/5] userfaultfd: UFFD_FEATURE_WP_ASYNC Date: Tue, 11 Jul 2023 17:52:37 +0500 Message-Id: <20230711125241.1587820-2-usama.anjum@collabora.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230711125241.1587820-1-usama.anjum@collabora.com> References: <20230711125241.1587820-1-usama.anjum@collabora.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: ADB8F180018 X-Stat-Signature: g39iffkqk66o9ujbi61cydjw7pznge6k X-HE-Tag: 1689080004-537602 X-HE-Meta: U2FsdGVkX1+tTsaIEbv8AxixIZok6AqCVh4KkeHODoIbELjC298ocJaJdMsLw3jf7ZRP8cFUdcO4mQVibyTzc2Cy3gB4T+CLosgU5H1CAxnzCRZugif6BxYwOq3sBgXbcqMURgBMQVj5DVF+q61PWB0teaLA3bxdqWztpuvhY/gzOwIBl3ttlPgAICvlbXPGoX/OviN5zgNEQXQH+2D9SXbQaKAdttQBTXJs4t77NvY1Bu11ZtRJghaMEMMKRnVYY5xa61F4oIcw2v0E6JVQ+FHduUUx2+Am+BWCHleUwyTWapKautsAMtWrd9x2Pyxy1KYXoW0jElfAJfUgEApykoXqF4kaM71JTxDzH5ac0lUhlpkTUxQ8VHy4oP97ZZx/+zGmgjR0RY0bz/yhjipFosdpdVXPRqt+hfXyptWIMK/61wekHyiM+adHwHbcn8Dis/AaQyjzNp/bsW6ya8BoN1fiWBX0WhK6pr7kGqBr9egyQKpXo7U2WPGFmpIQxTF3K0pRE1MyaZcjD+WT+8wKtFtU6vhXvjAtDTzFoMWDwq3w2zkZc2aKEAXSIQC8KktCVtSkufpQzNWfJvyZLI88/0s0tcEFj4mg/DzX9UouQMIzKMPkeCAhZveJoq9ZHLXN+fYAdvF9nkh/CPE7d5I2usEvxJidRoMLJwZtEHpX/1NwR77hRYI6dI9sWCQNOmJu2W2VulvcFwK8mP+snpJK2GogTnG2jGwlRF4gqQIYt6rTtqnAKJNSHc19/bJH5tvsRPrEzwNruMt24b5IeWJv+qazJvKPhkzdHiS0sdSwS0g2Rtto3gtlk/8ADp2vak7R7zXkaia5os7smsfD9ldUBHURIUGPNiky4kLOl5QMAZeiZ8c+YDmTPgtqfz0O5+NLc3SUVk9LeBCyCiQW6TdG1xVJ1MVQmEbt7cWt9Wm/A5ffHDkEoPMhZuLpwODBwVhdgZH1kOA6WEtPbiFHiJr 2qhx1ktz A1bGL2BMhR3BjmP9tqRc6dKiPEwhFroSywAEa1YRZDGm1qPpgFhDUj8wP7n6Kk5gouxXtMOx5umbcD05c/CddowJdzZc576S0jVaLt7KG2e8I1KbY5wJUKWNTWwjm5zLax9a7kNXOgbXwtcvx7Tk5h1Tdv+tG6/BkK7d19koxPpFu9FKhtQbsRXKpTZK1za4i1+TSnJleZVE8tnFzpGLddsEBisLQz+LAyhEOeJ//Y380F/YAGGEHE+5RuJqAEL3L0Qg1UdcVp0fTjajF+4mXG1WU8haBzKKEoiUhF6HzTcgt4npBmq0OHDhFNzVjoMbOBRF/kUnq13NEs7hcrcuk2JvarjbeTb5PYpMGPm1c7S1ueVwubdWZ+mSLAXm6FbkU3per X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Peter Xu This patch adds a new userfaultfd-wp feature UFFD_FEATURE_WP_ASYNC, that allows userfaultfd wr-protect faults to be resolved by the kernel directly. It can be used like a high accuracy version of soft-dirty, without vma modifications during tracking, and also with ranged support by default rather than for a whole mm when reset the protections due to existence of ioctl(UFFDIO_WRITEPROTECT). Several goals of such a dirty tracking interface: 1. All types of memory should be supported and tracable. This is nature for soft-dirty but should mention when the context is userfaultfd, because it used to only support anon/shmem/hugetlb. The problem is for a dirty tracking purpose these three types may not be enough, and it's legal to track anything e.g. any page cache writes from mmap. 2. Protections can be applied to partial of a memory range, without vma split/merge fuss. The hope is that the tracking itself should not affect any vma layout change. It also helps when reset happens because the reset will not need mmap write lock which can block the tracee. 3. Accuracy needs to be maintained. This means we need pte markers to work on any type of VMA. One could question that, the whole concept of async dirty tracking is not really close to fundamentally what userfaultfd used to be: it's not "a fault to be serviced by userspace" anymore. However, using userfaultfd-wp here as a framework is convenient for us in at least: 1. VM_UFFD_WP vma flag, which has a very good name to suite something like this, so we don't need VM_YET_ANOTHER_SOFT_DIRTY. Just use a new feature bit to identify from a sync version of uffd-wp registration. 2. PTE markers logic can be leveraged across the whole kernel to maintain the uffd-wp bit as long as an arch supports, this also applies to this case where uffd-wp bit will be a hint to dirty information and it will not go lost easily (e.g. when some page cache ptes got zapped). 3. Reuse ioctl(UFFDIO_WRITEPROTECT) interface for either starting or resetting a range of memory, while there's no counterpart in the old soft-dirty world, hence if this is wanted in a new design we'll need a new interface otherwise. We can somehow understand that commonality because uffd-wp was fundamentally a similar idea of write-protecting pages just like soft-dirty. This implementation allows WP_ASYNC to imply WP_UNPOPULATED, because so far WP_ASYNC seems to not usable if without WP_UNPOPULATE. This also gives us chance to modify impl of WP_ASYNC just in case it could be not depending on WP_UNPOPULATED anymore in the future kernels. It's also fine to imply that because both features will rely on PTE_MARKER_UFFD_WP config option, so they'll show up together (or both missing) in an UFFDIO_API probe. vma_can_userfault() now allows any VMA if the userfaultfd registration is only about async uffd-wp. So we can track dirty for all kinds of memory including generic file systems (like XFS, EXT4 or BTRFS). One trick worth mention in do_wp_page() is that we need to manually update vmf->orig_pte here because it can be used later with a pte_same() check - this path always has FAULT_FLAG_ORIG_PTE_VALID set in the flags. The major defect of this approach of dirty tracking is we need to populate the pgtables when tracking starts. Soft-dirty doesn't do it like that. It's unwanted in the case where the range of memory to track is huge and unpopulated (e.g., tracking updates on a 10G file with mmap() on top, without having any page cache installed yet). One way to improve this is to allow pte markers exist for larger than PTE level for PMD+. That will not change the interface if to implemented, so we can leave that for later. Co-developed-by: Muhammad Usama Anjum Signed-off-by: Muhammad Usama Anjum Signed-off-by: Peter Xu --- Changes in v24: - Rebase on top of next-20230710 - UFFD_FEATURE_WP_ASYNC has been defined to be (1 << 15) after the rebase Changes in v18: - Rebase on top of next-20230613 (use ptep_get) Changes in v17: - Rebase on top of next-20230525 Changes in v12: - Peter added the hugetlb support and revamped some other implementation - Transferred the authorship to Peter - Merge documentation to this patch Changes in v11: - Fix return code in userfaultfd_register() and minor changes here and there - Rebase on top of next-20230307 - Base patches on UFFD_FEATURE_WP_UNPOPULATED https://lore.kernel.org/all/20230306213925.617814-1-peterx@redhat.com - UFFD_FEATURE_WP_ASYNC depends on UFFD_FEATURE_WP_UNPOPULATED to work (correctly) Changes in v10: - Build fix - Update comments and add error condition to return error from uffd register if hugetlb pages are present when wp async flag is set Changes in v9: - Correct the fault resolution with code contributed by Peter Changes in v7: - Remove UFFDIO_WRITEPROTECT_MODE_ASYNC_WP and add UFFD_FEATURE_WP_ASYNC - Handle automatic page fault resolution in better way (thanks to Peter) --- Documentation/admin-guide/mm/userfaultfd.rst | 35 ++++++++++++++++++++ fs/userfaultfd.c | 26 ++++++++++++--- include/linux/userfaultfd_k.h | 21 +++++++++++- include/uapi/linux/userfaultfd.h | 9 ++++- mm/hugetlb.c | 32 ++++++++++-------- mm/memory.c | 27 +++++++++++++-- 6 files changed, 128 insertions(+), 22 deletions(-) diff --git a/Documentation/admin-guide/mm/userfaultfd.rst b/Documentation/admin-guide/mm/userfaultfd.rst index 4349a8c2b978..203e26da5f92 100644 --- a/Documentation/admin-guide/mm/userfaultfd.rst +++ b/Documentation/admin-guide/mm/userfaultfd.rst @@ -244,6 +244,41 @@ write-protected (so future writes will also result in a WP fault). These ioctls support a mode flag (``UFFDIO_COPY_MODE_WP`` or ``UFFDIO_CONTINUE_MODE_WP`` respectively) to configure the mapping this way. +If the userfaultfd context has ``UFFD_FEATURE_WP_ASYNC`` feature bit set, +any vma registered with write-protection will work in async mode rather +than the default sync mode. + +In async mode, there will be no message generated when a write operation +happens, meanwhile the write-protection will be resolved automatically by +the kernel. It can be seen as a more accurate version of soft-dirty +tracking and it can be different in a few ways: + + - The dirty result will not be affected by vma changes (e.g. vma + merging) because the dirty is only tracked by the pte. + + - It supports range operations by default, so one can enable tracking on + any range of memory as long as page aligned. + + - Dirty information will not get lost if the pte was zapped due to + various reasons (e.g. during split of a shmem transparent huge page). + + - Due to a reverted meaning of soft-dirty (page clean when uffd-wp bit + set; dirty when uffd-wp bit cleared), it has different semantics on + some of the memory operations. For example: ``MADV_DONTNEED`` on + anonymous (or ``MADV_REMOVE`` on a file mapping) will be treated as + dirtying of memory by dropping uffd-wp bit during the procedure. + +The user app can collect the "written/dirty" status by looking up the +uffd-wp bit for the pages being interested in /proc/pagemap. + +The page will not be under track of uffd-wp async mode until the page is +explicitly write-protected by ``ioctl(UFFDIO_WRITEPROTECT)`` with the mode +flag ``UFFDIO_WRITEPROTECT_MODE_WP`` set. Trying to resolve a page fault +that was tracked by async mode userfaultfd-wp is invalid. + +When userfaultfd-wp async mode is used alone, it can be applied to all +kinds of memory. + Memory Poisioning Emulation --------------------------- diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c index db57d5d4fcb7..08be70c7a57e 100644 --- a/fs/userfaultfd.c +++ b/fs/userfaultfd.c @@ -123,6 +123,11 @@ static bool userfaultfd_is_initialized(struct userfaultfd_ctx *ctx) return ctx->features & UFFD_FEATURE_INITIALIZED; } +static bool userfaultfd_wp_async_ctx(struct userfaultfd_ctx *ctx) +{ + return ctx && (ctx->features & UFFD_FEATURE_WP_ASYNC); +} + /* * Whether WP_UNPOPULATED is enabled on the uffd context. It is only * meaningful when userfaultfd_wp()==true on the vma and when it's @@ -1318,6 +1323,7 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx, bool basic_ioctls; unsigned long start, end, vma_end; struct vma_iterator vmi; + bool wp_async = userfaultfd_wp_async_ctx(ctx); pgoff_t pgoff; user_uffdio_register = (struct uffdio_register __user *) arg; @@ -1392,7 +1398,7 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx, /* check not compatible vmas */ ret = -EINVAL; - if (!vma_can_userfault(cur, vm_flags)) + if (!vma_can_userfault(cur, vm_flags, wp_async)) goto out_unlock; /* @@ -1453,7 +1459,7 @@ static int userfaultfd_register(struct userfaultfd_ctx *ctx, for_each_vma_range(vmi, vma, end) { cond_resched(); - BUG_ON(!vma_can_userfault(vma, vm_flags)); + BUG_ON(!vma_can_userfault(vma, vm_flags, wp_async)); BUG_ON(vma->vm_userfaultfd_ctx.ctx && vma->vm_userfaultfd_ctx.ctx != ctx); WARN_ON(!(vma->vm_flags & VM_MAYWRITE)); @@ -1553,6 +1559,7 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, unsigned long start, end, vma_end; const void __user *buf = (void __user *)arg; struct vma_iterator vmi; + bool wp_async = userfaultfd_wp_async_ctx(ctx); pgoff_t pgoff; ret = -EFAULT; @@ -1607,7 +1614,7 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, * provides for more strict behavior to notice * unregistration errors. */ - if (!vma_can_userfault(cur, cur->vm_flags)) + if (!vma_can_userfault(cur, cur->vm_flags, wp_async)) goto out_unlock; found = true; @@ -1623,7 +1630,7 @@ static int userfaultfd_unregister(struct userfaultfd_ctx *ctx, for_each_vma_range(vmi, vma, end) { cond_resched(); - BUG_ON(!vma_can_userfault(vma, vma->vm_flags)); + BUG_ON(!vma_can_userfault(vma, vma->vm_flags, wp_async)); /* * Nothing to do: this vma is already registered into this @@ -2005,6 +2012,11 @@ static inline int userfaultfd_poison(struct userfaultfd_ctx *ctx, unsigned long return ret; } +bool userfaultfd_wp_async(struct vm_area_struct *vma) +{ + return userfaultfd_wp_async_ctx(vma->vm_userfaultfd_ctx.ctx); +} + static inline unsigned int uffd_ctx_features(__u64 user_features) { /* @@ -2038,6 +2050,11 @@ static int userfaultfd_api(struct userfaultfd_ctx *ctx, ret = -EPERM; if ((features & UFFD_FEATURE_EVENT_FORK) && !capable(CAP_SYS_PTRACE)) goto err_out; + + /* WP_ASYNC relies on WP_UNPOPULATED, choose it unconditionally */ + if (features & UFFD_FEATURE_WP_ASYNC) + features |= UFFD_FEATURE_WP_UNPOPULATED; + /* report all available features and ioctls to userland */ uffdio_api.features = UFFD_API_FEATURES; #ifndef CONFIG_HAVE_ARCH_USERFAULTFD_MINOR @@ -2050,6 +2067,7 @@ static int userfaultfd_api(struct userfaultfd_ctx *ctx, #ifndef CONFIG_PTE_MARKER_UFFD_WP uffdio_api.features &= ~UFFD_FEATURE_WP_HUGETLBFS_SHMEM; uffdio_api.features &= ~UFFD_FEATURE_WP_UNPOPULATED; + uffdio_api.features &= ~UFFD_FEATURE_WP_ASYNC; #endif uffdio_api.ioctls = UFFD_API_IOCTLS; ret = -EFAULT; diff --git a/include/linux/userfaultfd_k.h b/include/linux/userfaultfd_k.h index ac8c6854097c..c98df391bfd8 100644 --- a/include/linux/userfaultfd_k.h +++ b/include/linux/userfaultfd_k.h @@ -161,11 +161,22 @@ static inline bool userfaultfd_armed(struct vm_area_struct *vma) } static inline bool vma_can_userfault(struct vm_area_struct *vma, - unsigned long vm_flags) + unsigned long vm_flags, + bool wp_async) { + vm_flags &= __VM_UFFD_FLAGS; + if ((vm_flags & VM_UFFD_MINOR) && (!is_vm_hugetlb_page(vma) && !vma_is_shmem(vma))) return false; + + /* + * If wp async enabled, and WP is the only mode enabled, allow any + * memory type. + */ + if (wp_async && (vm_flags == VM_UFFD_WP)) + return true; + #ifndef CONFIG_PTE_MARKER_UFFD_WP /* * If user requested uffd-wp but not enabled pte markers for @@ -175,6 +186,8 @@ static inline bool vma_can_userfault(struct vm_area_struct *vma, if ((vm_flags & VM_UFFD_WP) && !vma_is_anonymous(vma)) return false; #endif + + /* By default, allow any of anon|shmem|hugetlb */ return vma_is_anonymous(vma) || is_vm_hugetlb_page(vma) || vma_is_shmem(vma); } @@ -197,6 +210,7 @@ extern int userfaultfd_unmap_prep(struct vm_area_struct *vma, extern void userfaultfd_unmap_complete(struct mm_struct *mm, struct list_head *uf); extern bool userfaultfd_wp_unpopulated(struct vm_area_struct *vma); +extern bool userfaultfd_wp_async(struct vm_area_struct *vma); #else /* CONFIG_USERFAULTFD */ @@ -297,6 +311,11 @@ static inline bool userfaultfd_wp_unpopulated(struct vm_area_struct *vma) return false; } +static inline bool userfaultfd_wp_async(struct vm_area_struct *vma) +{ + return false; +} + #endif /* CONFIG_USERFAULTFD */ static inline bool userfaultfd_wp_use_markers(struct vm_area_struct *vma) diff --git a/include/uapi/linux/userfaultfd.h b/include/uapi/linux/userfaultfd.h index 62151706c5a3..0dbc81015018 100644 --- a/include/uapi/linux/userfaultfd.h +++ b/include/uapi/linux/userfaultfd.h @@ -40,7 +40,8 @@ UFFD_FEATURE_EXACT_ADDRESS | \ UFFD_FEATURE_WP_HUGETLBFS_SHMEM | \ UFFD_FEATURE_WP_UNPOPULATED | \ - UFFD_FEATURE_POISON) + UFFD_FEATURE_POISON | \ + UFFD_FEATURE_WP_ASYNC) #define UFFD_API_IOCTLS \ ((__u64)1 << _UFFDIO_REGISTER | \ (__u64)1 << _UFFDIO_UNREGISTER | \ @@ -216,6 +217,11 @@ struct uffdio_api { * (i.e. empty ptes). This will be the default behavior for shmem * & hugetlbfs, so this flag only affects anonymous memory behavior * when userfault write-protection mode is registered. + * + * UFFD_FEATURE_WP_ASYNC indicates that userfaultfd write-protection + * asynchronous mode is supported in which the write fault is + * automatically resolved and write-protection is un-set. + * It implies UFFD_FEATURE_WP_UNPOPULATED. */ #define UFFD_FEATURE_PAGEFAULT_FLAG_WP (1<<0) #define UFFD_FEATURE_EVENT_FORK (1<<1) @@ -232,6 +238,7 @@ struct uffdio_api { #define UFFD_FEATURE_WP_HUGETLBFS_SHMEM (1<<12) #define UFFD_FEATURE_WP_UNPOPULATED (1<<13) #define UFFD_FEATURE_POISON (1<<14) +#define UFFD_FEATURE_WP_ASYNC (1<<15) __u64 features; __u64 ioctls; diff --git a/mm/hugetlb.c b/mm/hugetlb.c index e4a28ce0667f..ee88a2a6649f 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6165,21 +6165,27 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, /* Handle userfault-wp first, before trying to lock more pages */ if (userfaultfd_wp(vma) && huge_pte_uffd_wp(huge_ptep_get(ptep)) && (flags & FAULT_FLAG_WRITE) && !huge_pte_write(entry)) { - struct vm_fault vmf = { - .vma = vma, - .address = haddr, - .real_address = address, - .flags = flags, - }; + if (!userfaultfd_wp_async(vma)) { + struct vm_fault vmf = { + .vma = vma, + .address = haddr, + .real_address = address, + .flags = flags, + }; - spin_unlock(ptl); - if (pagecache_folio) { - folio_unlock(pagecache_folio); - folio_put(pagecache_folio); + spin_unlock(ptl); + if (pagecache_folio) { + folio_unlock(pagecache_folio); + folio_put(pagecache_folio); + } + hugetlb_vma_unlock_read(vma); + mutex_unlock(&hugetlb_fault_mutex_table[hash]); + return handle_userfault(&vmf, VM_UFFD_WP); } - hugetlb_vma_unlock_read(vma); - mutex_unlock(&hugetlb_fault_mutex_table[hash]); - return handle_userfault(&vmf, VM_UFFD_WP); + + entry = huge_pte_clear_uffd_wp(entry); + set_huge_pte_at(mm, haddr, ptep, entry); + /* Fallthrough to CoW */ } /* diff --git a/mm/memory.c b/mm/memory.c index 0a265ac6246e..b9f8c32ddd28 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -3352,11 +3352,28 @@ static vm_fault_t do_wp_page(struct vm_fault *vmf) const bool unshare = vmf->flags & FAULT_FLAG_UNSHARE; struct vm_area_struct *vma = vmf->vma; struct folio *folio = NULL; + pte_t pte; if (likely(!unshare)) { if (userfaultfd_pte_wp(vma, ptep_get(vmf->pte))) { - pte_unmap_unlock(vmf->pte, vmf->ptl); - return handle_userfault(vmf, VM_UFFD_WP); + if (!userfaultfd_wp_async(vma)) { + pte_unmap_unlock(vmf->pte, vmf->ptl); + return handle_userfault(vmf, VM_UFFD_WP); + } + + /* + * Nothing needed (cache flush, TLB invalidations, + * etc.) because we're only removing the uffd-wp bit, + * which is completely invisible to the user. + */ + pte = pte_clear_uffd_wp(ptep_get(vmf->pte)); + + set_pte_at(vma->vm_mm, vmf->address, vmf->pte, pte); + /* + * Update this to be prepared for following up CoW + * handling + */ + vmf->orig_pte = pte; } /* @@ -4851,8 +4868,11 @@ static inline vm_fault_t wp_huge_pmd(struct vm_fault *vmf) if (vma_is_anonymous(vmf->vma)) { if (likely(!unshare) && - userfaultfd_huge_pmd_wp(vmf->vma, vmf->orig_pmd)) + userfaultfd_huge_pmd_wp(vmf->vma, vmf->orig_pmd)) { + if (userfaultfd_wp_async(vmf->vma)) + goto split; return handle_userfault(vmf, VM_UFFD_WP); + } return do_huge_pmd_wp_page(vmf); } @@ -4864,6 +4884,7 @@ static inline vm_fault_t wp_huge_pmd(struct vm_fault *vmf) } } +split: /* COW or write-notify handled on pte level: split pmd. */ __split_huge_pmd(vmf->vma, vmf->pmd, vmf->address, false, NULL); From patchwork Tue Jul 11 12:52:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muhammad Usama Anjum X-Patchwork-Id: 13308668 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F0A0BEB64DD for ; Tue, 11 Jul 2023 12:53:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 947316B007B; Tue, 11 Jul 2023 08:53:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 91D5F6B0078; Tue, 11 Jul 2023 08:53:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 797036B007B; Tue, 11 Jul 2023 08:53:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 6897F6B0075 for ; Tue, 11 Jul 2023 08:53:34 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 14DCD16024A for ; Tue, 11 Jul 2023 12:53:34 +0000 (UTC) X-FDA: 80999322348.26.3337907 Received: from madras.collabora.co.uk (madras.collabora.co.uk [46.235.227.172]) by imf08.hostedemail.com (Postfix) with ESMTP id 27507160002 for ; Tue, 11 Jul 2023 12:53:31 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=collabora.com header.s=mail header.b=jk0WM8YF; spf=pass (imf08.hostedemail.com: domain of usama.anjum@collabora.com designates 46.235.227.172 as permitted sender) smtp.mailfrom=usama.anjum@collabora.com; dmarc=pass (policy=quarantine) header.from=collabora.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689080012; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Ouqpx3GqEQgG5X/pZm3WiPYH3KWYFm+CfMB0X8+217o=; b=Vnvu8FL07TYoxo0VEjlq9KGpMB+EOSzcLw39/fVfx9EF+JRyFN8mCL8xNiG/2llc2XBzx4 +x788Te9KotN3OSdf180u7fEkmFk0V+jKt2It0W3pkk1eYOmgtgByYABFfOJB/u3JOuMk7 6IXv5z/OFlZad1ZfTA7nDugt2H7PTLw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689080012; a=rsa-sha256; cv=none; b=hLsik44PeAyjOBXM6urDYfGbj+hmlc6yq83aU8rTXF92rtjMy2hwgN465tGq8QEK41+NYb +oyY+Q1cCNKZDABoR5dfmSSK97hmZCnTo8EbkY+30hfBTSEQTdtpn2fkOYSmbOFs5KVEUW XCMMaYNkP7T3OTSrN0YBqp1rYLy1y8M= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=collabora.com header.s=mail header.b=jk0WM8YF; spf=pass (imf08.hostedemail.com: domain of usama.anjum@collabora.com designates 46.235.227.172 as permitted sender) smtp.mailfrom=usama.anjum@collabora.com; dmarc=pass (policy=quarantine) header.from=collabora.com Received: from localhost.localdomain (unknown [119.155.40.122]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: usama.anjum) by madras.collabora.co.uk (Postfix) with ESMTPSA id E28B56607010; Tue, 11 Jul 2023 13:53:23 +0100 (BST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=collabora.com; s=mail; t=1689080011; bh=5pZoAXBqrfC7o6fDfQj0GEkzj8IunB1a+vtWUG2GIFg=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=jk0WM8YF3AAZf676pb4HSgPCPbFqsQkO76gwCjpeJCPjc3G80D2xTz0I5D+j40yao FuhQT8rPDMzpAXSIsTnKxHk5QBCY7Y8PFAYmHT8Zo8mvJAJ+etJ129VpMKmG/eemNa kPlgPtuwfOzweNj7umtqBaBglRN5GfW3MPHfm0BcoFVHWcKAzzlxaY0YvKqPKOCEcz R4WguaWByOcToa1Fst+R9y04aKf8Wd8sg09tjJ8cxGUQ632sNs8DLDDTweyir1+8Hs pbDbhVxQi7+kuWlRafyUZRP17d/PyN1XKHSvyETPCOL5CVi0lC2LBYgDJUAk1X2pY/ 56qhOjC9H2mCg== From: Muhammad Usama Anjum To: Peter Xu , David Hildenbrand , Andrew Morton , =?utf-8?b?TWljaGHFgiBNaXJvc8WC?= =?utf-8?b?YXc=?= , Andrei Vagin , Danylo Mocherniuk , Paul Gofman , Cyrill Gorcunov , Mike Rapoport , Nadav Amit Cc: Alexander Viro , Shuah Khan , Christian Brauner , Yang Shi , Vlastimil Babka , "Liam R . Howlett" , Yun Zhou , Suren Baghdasaryan , Alex Sierra , Muhammad Usama Anjum , Matthew Wilcox , Pasha Tatashin , Axel Rasmussen , "Gustavo A . R . Silva" , Dan Williams , linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-kselftest@vger.kernel.org, Greg KH , kernel@collabora.com Subject: [PATCH v24 2/5] fs/proc/task_mmu: Implement IOCTL to get and optionally clear info about PTEs Date: Tue, 11 Jul 2023 17:52:38 +0500 Message-Id: <20230711125241.1587820-3-usama.anjum@collabora.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230711125241.1587820-1-usama.anjum@collabora.com> References: <20230711125241.1587820-1-usama.anjum@collabora.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 27507160002 X-Rspam-User: X-Stat-Signature: ta4p4h3r3mqi1c4xwshncexmjjscno6q X-Rspamd-Server: rspam03 X-HE-Tag: 1689080011-221344 X-HE-Meta: U2FsdGVkX1/NS10Xeky9Cs82giudGTbAO54RZPh0gbGTDbaqwCpb4kcdz6tD/pO4MTtV505WsB6QsAUShZqEUuqwZ1eQvBXQ/ggZBaxPxhtKeO0NT4yiT0xGkO769P66/qjVRAr8JotR3zXCA6RhkUOT8bUWOIGuWvMTCvhf4hlHtNkUdPrb+yU/CXSi7IzZIHr+436UnS1VK7Oxr9I39uVNCdDJJBa+OM+5tm84kxWpZm/8s/lSE0LFagn6eS+iaNHNnOp91/0GE2Fb2mpCe3UobKz8lJdJyXtiYhTQs2TJEYOzPQ+yMF7wkbVNYlJTpSbPldShJdqln0hI1QhrFLw1DcuoL6P8I1YFqAZnMz49EmZaakDRFSXPSzbCPE20+Jfiy3RIKVBhbmqn2OZ6MXdJC3wEnBgNQvqVWmeRuRmVZjbvWB/O7KdZuOIfppzlQrnmZ5o+bWPQ3n+UvuxnAFC6Mg9+HjVxU8099P+TupVILzWDLa2q4AgE5miu5eV2O9oSK5Slekno0W4JCoGzV3I+EiZQFeK7SkmSqAd8d8x53jxWwR4iFdsLzY1Cd57g+o7V45Zrx5bzQYeLMGK/vamI7rvkm3blpeUU0+dD6YAmajr6+tw4F1ShaoOaiJb8HgrzVJjUyPWubxlBu/jvXhWkJxed4wrfZMj9iol4ZJ54GkEzoBPxM30j/yBZxagjq4J2AXquD6tUyK8XDczV3teX4W4/cX5nDOtreRKrTKNAHDGWyb8z8f5UNiFJYlVQxm0KiGlN8n2jIvqCPWV3YqB5vTq3TvtJIo7P5v5Qi/EApXUKUEE9jGexQahAQPp6Hj4se3DCEv6+1mNtreyv0aMUzIE0nD1cewdxoJzy27MlVyDrtR6rEcair1q3EqPXHIMWZhTTa0i9kGp6yWeOLHyW25vi4JSgncVwKcFQQ435zDHdS5vU1WGlF4hEIo4tKNtquBW35s7yHGwFo5U gF484dCr HYV9KWEBjy4SlWamoPCiiqN25S/kfAXwwVd8GYHd0OVDtfyuxCaNJMu3GbwQq8LSksJp658xSWkyEoTtzbzdtHLcIlSGam0JJe6GegR6AmBNefWGgKzijlnzv++iqEWVSO9j55ddEigR2EQcwf9dwS28anrH6K44ydedze54yhqQXDzagb/MIW2IvE9M0sNPbm6IxtfbhmaPfS/MLrrXozA7yBNaxj3XBAMrCF67WfuYEX/6jC2rXnOptWOn7L84otKArKlchf5VGklPuqQKJXE5CzNqKwJFm0Q3j7lHLEqYF8qE= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: This IOCTL, PAGEMAP_SCAN on pagemap file can be used to get and/or clear the info about page table entries. The following operations are supported in this ioctl: - Get the information if the pages have been written-to (PAGE_IS_WRITTEN), file mapped (PAGE_IS_FILE), present (PAGE_IS_PRESENT), swapped (PAGE_IS_SWAPPED) or page has pfn zero (PAGE_IS_PFNZERO). - Find pages which have been written-to and/or write protect the pages (atomic PM_SCAN_OP_GET + PM_SCAN_OP_WP) This IOCTL can be extended to get information about more PTE bits. The entire address range passed by user [start, end) is scanned until either the user provided buffer is full or max_pages have been found. Signed-off-by: Muhammad Usama Anjum --- Changes in v24: - Place WP markers in case of hole as well Changes in v23: - Set vec_buf_index to 0 only when vec_buf_index is set - Return -EFAULT instead of -EINVAL if vec is NULL - Correctly return the walk ending address to the page granularity Changes in v22: - Interface change to return walk ending address to user: - Replace [start start + len) with [start, end) - Return the ending address of the address walk in start Changes in v21: - Abort walk instead of returning error if WP is to be performed on partial hugetlb - Changed the data types of some variables in pagemap_scan_private to long Changes in v20: - Correct PAGE_IS_FILE and add PAGE_IS_PFNZERO Changes in v19: - Interface changes such as renaming, return mask and WP can be used with any flags specified in masks - Internal code changes Changes in v18: - Rebased on top of next-20230613 - ptep_get() updates - remove pmd_trans_unstable() and add ACTION_AGAIN - Review updates (Micheal) Changes in v17: - Rebased on next-20230606 - Made make_uffd_wp_*_pte() better and minor changes Changes in v16: - Fixed a corner case where kernel writes beyond user buffer by one element - Bring back exclusive PM_SCAN_OP_WP - Cosmetic changes Changes in v15: - Build fix: - Use generic tlb flush function in pagemap_scan_pmd_entry() instead of using x86 specific flush function in do_pagemap_scan() - Remove #ifdef from pagemap_scan_hugetlb_entry() - Use mm instead of undefined vma->vm_mm Changes in v14: - Fix build error caused by #ifdef added at last minute in some configs Changes in v13: - Review updates - mmap_read_lock_killable() instead of mmap_read_lock() - Replace uffd_wp_range() with helpers which increases performance drastically for OP_WP operations by reducing the number of tlb flushing etc - Add MMU_NOTIFY_PROTECTION_VMA notification for the memory range Changes in v12: - Add hugetlb support to cover all memory types - Merge "userfaultfd: Define dummy uffd_wp_range()" with this patch - Review updates to the code Changes in v11: - Find written pages in a better way - Fix a corner case (thanks Paul) - Improve the code/comments - remove ENGAGE_WP + ! GET operation - shorten the commit message in favour of moving documentation to pagemap.rst Changes in v10: - move changes in tools/include/uapi/linux/fs.h to separate patch - update commit message Change in v8: - Correct is_pte_uffd_wp() - Improve readability and error checks - Remove some un-needed code Changes in v7: - Rebase on top of latest next - Fix some corner cases - Base soft-dirty on the uffd wp async - Update the terminologies - Optimize the memory usage inside the ioctl --- fs/proc/task_mmu.c | 583 ++++++++++++++++++++++++++++++++++++++++ include/linux/hugetlb.h | 1 + include/uapi/linux/fs.h | 55 ++++ mm/hugetlb.c | 2 +- 4 files changed, 640 insertions(+), 1 deletion(-) diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index cfab855fe7e9..4aadd046e0f8 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -19,6 +19,7 @@ #include #include #include +#include #include #include @@ -1768,11 +1769,593 @@ static int pagemap_release(struct inode *inode, struct file *file) return 0; } +#define PM_SCAN_REQUIRE_UFFD (1ULL << 63) + +#define PM_SCAN_FOUND_MAX_PAGES (1) +#define PM_SCAN_END_WALK (-256) + +#define PM_SCAN_BITS_ALL (PAGE_IS_WRITTEN | PAGE_IS_FILE | \ + PAGE_IS_PRESENT | PAGE_IS_SWAPPED | \ + PAGE_IS_PFNZERO) +#define PM_SCAN_OPS (PM_SCAN_OP_GET | PM_SCAN_OP_WP) +#define IS_PM_SCAN_GET(flags) (flags & PM_SCAN_OP_GET) +#define IS_PM_SCAN_WP(flags) (flags & PM_SCAN_OP_WP) + +#define PM_SCAN_FLAGS(wt, file, present, swap, pfnzero) \ + ((wt) | ((file) << 1) | ((present) << 2) | \ + ((swap) << 3) | ((pfnzero) << 4)) + +struct pagemap_scan_private { + struct page_region *vec_buf, cur_buf; + unsigned long vec_buf_len, vec_buf_index, max_pages, found_pages; + unsigned long long flags, required_mask, anyof_mask, excluded_mask, return_mask, end_addr; +}; + +static inline bool is_pte_uffd_wp(pte_t pte) +{ + return (pte_present(pte) && pte_uffd_wp(pte)) || + pte_swp_uffd_wp_any(pte); +} + +static inline void make_uffd_wp_pte(struct vm_area_struct *vma, + unsigned long addr, pte_t *pte) +{ + pte_t ptent = ptep_get(pte); + + if (pte_present(ptent)) { + pte_t old_pte; + + old_pte = ptep_modify_prot_start(vma, addr, pte); + ptent = pte_mkuffd_wp(ptent); + ptep_modify_prot_commit(vma, addr, pte, old_pte, ptent); + } else if (is_swap_pte(ptent)) { + ptent = pte_swp_mkuffd_wp(ptent); + set_pte_at(vma->vm_mm, addr, pte, ptent); + } else { + set_pte_at(vma->vm_mm, addr, pte, + make_pte_marker(PTE_MARKER_UFFD_WP)); + } +} + +static inline bool pagemap_scan_is_file(struct vm_area_struct *vma, pte_t ptent, + unsigned long addr) +{ + struct page *page = NULL; + swp_entry_t entry; + + if (pte_present(ptent)) { + page = vm_normal_page(vma, addr, ptent); + } else { + entry = pte_to_swp_entry(ptent); + if (is_pfn_swap_entry(entry)) + page = pfn_swap_entry_to_page(entry); + } + + if (page && !PageAnon(page)) + return true; + + return false; +} + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE +static inline bool is_pmd_uffd_wp(pmd_t pmd) +{ + return (pmd_present(pmd) && pmd_uffd_wp(pmd)) || + (is_swap_pmd(pmd) && pmd_swp_uffd_wp(pmd)); +} + +static inline void make_uffd_wp_pmd(struct vm_area_struct *vma, + unsigned long addr, pmd_t *pmdp) +{ + pmd_t old, pmd = *pmdp; + + if (pmd_present(pmd)) { + old = pmdp_invalidate_ad(vma, addr, pmdp); + pmd = pmd_mkuffd_wp(old); + set_pmd_at(vma->vm_mm, addr, pmdp, pmd); + } else if (is_migration_entry(pmd_to_swp_entry(pmd))) { + pmd = pmd_swp_mkuffd_wp(pmd); + set_pmd_at(vma->vm_mm, addr, pmdp, pmd); + } +} +#endif + +#ifdef CONFIG_HUGETLB_PAGE +static inline bool is_huge_pte_uffd_wp(pte_t pte) +{ + return (pte_present(pte) && huge_pte_uffd_wp(pte)) || + pte_swp_uffd_wp_any(pte); +} + +static inline void make_uffd_wp_huge_pte(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + pte_t ptent) +{ + if (is_hugetlb_entry_hwpoisoned(ptent) || is_pte_marker(ptent)) + return; + + if (is_hugetlb_entry_migration(ptent)) + set_huge_pte_at(vma->vm_mm, addr, ptep, + pte_swp_mkuffd_wp(ptent)); + else if (!huge_pte_none(ptent)) + huge_ptep_modify_prot_commit(vma, addr, ptep, ptent, + huge_pte_mkuffd_wp(ptent)); + else + set_huge_pte_at(vma->vm_mm, addr, ptep, + make_pte_marker(PTE_MARKER_UFFD_WP)); +} + +static inline bool pagemap_scan_is_huge_file(pte_t pte) +{ + if (pte_present(pte) && (!PageAnon(pte_page(pte)))) + return true; + + return false; +} +#endif + +static int pagemap_scan_test_walk(unsigned long start, unsigned long end, + struct mm_walk *walk) +{ + struct pagemap_scan_private *p = walk->private; + struct vm_area_struct *vma = walk->vma; + + if ((p->flags & PM_SCAN_REQUIRE_UFFD) && (!userfaultfd_wp_async(vma) || + !userfaultfd_wp_use_markers(vma))) + return -EPERM; + + if (vma->vm_flags & VM_PFNMAP) + return 1; + + return 0; +} + +static int pagemap_scan_output(unsigned long bitmap, + struct pagemap_scan_private *p, + unsigned long addr, unsigned int n_pages) +{ + struct page_region *cur_buf = &p->cur_buf; + + bitmap &= p->return_mask; + + if (cur_buf->flags == bitmap && + cur_buf->start + cur_buf->len * PAGE_SIZE == addr) { + cur_buf->len += n_pages; + p->found_pages += n_pages; + } else { + if (cur_buf->len) { + if (p->vec_buf_index >= p->vec_buf_len) { + p->end_addr = addr; + return PM_SCAN_END_WALK; + } + + memcpy(&p->vec_buf[p->vec_buf_index], cur_buf, + sizeof(*p->vec_buf)); + p->vec_buf_index++; + } + + cur_buf->start = addr; + cur_buf->len = n_pages; + cur_buf->flags = bitmap; + p->found_pages += n_pages; + } + + if (p->found_pages == p->max_pages) { + p->end_addr = addr + n_pages * PAGE_SIZE; + return PM_SCAN_FOUND_MAX_PAGES; + } + + return 0; +} + +static bool pagemap_scan_is_interesting_page(unsigned long bitmap, + struct pagemap_scan_private *p) +{ + if ((p->required_mask & bitmap) != p->required_mask) + return false; + if (p->anyof_mask && !(p->anyof_mask & bitmap)) + return false; + if (p->excluded_mask & bitmap) + return false; + + return true; +} + +static int pagemap_scan_pmd_entry(pmd_t *pmd, unsigned long start, + unsigned long end, struct mm_walk *walk) +{ + bool is_written, flush = false, interesting = true; + struct pagemap_scan_private *p = walk->private; + struct vm_area_struct *vma = walk->vma; + unsigned long bitmap, addr = end; + pte_t *pte, *orig_pte, ptent; + spinlock_t *ptl; + int ret = 0; + + arch_enter_lazy_mmu_mode(); + +#ifdef CONFIG_TRANSPARENT_HUGEPAGE + ptl = pmd_trans_huge_lock(pmd, vma); + if (ptl) { + unsigned long n_pages = (end - start)/PAGE_SIZE; + + if (n_pages > p->max_pages - p->found_pages) + n_pages = p->max_pages - p->found_pages; + + is_written = !is_pmd_uffd_wp(*pmd); + + bitmap = PM_SCAN_FLAGS(is_written, false, + pmd_present(*pmd), is_swap_pmd(*pmd), + pmd_present(*pmd) && is_zero_pfn(pmd_pfn(*pmd))); + + if (IS_PM_SCAN_GET(p->flags)) + interesting = pagemap_scan_is_interesting_page(bitmap, p); + + if (interesting) { + /* + * Break huge page into small pages if the WP operation + * need to be performed is on a portion of the huge page. + */ + if (is_written && IS_PM_SCAN_WP(p->flags) && + n_pages < HPAGE_SIZE/PAGE_SIZE) { + spin_unlock(ptl); + + split_huge_pmd(vma, pmd, start); + goto process_smaller_pages; + } + + if (IS_PM_SCAN_GET(p->flags)) + ret = pagemap_scan_output(bitmap, p, start, n_pages); + + if (IS_PM_SCAN_WP(p->flags) && is_written && ret >= 0) { + make_uffd_wp_pmd(vma, start, pmd); + flush_tlb_range(vma, start, end); + } + } + + spin_unlock(ptl); + arch_leave_lazy_mmu_mode(); + + return ret; + } + +process_smaller_pages: +#endif + + orig_pte = pte = pte_offset_map_lock(vma->vm_mm, pmd, start, &ptl); + if (!pte) { + walk->action = ACTION_AGAIN; + return 0; + } + + for (addr = start; addr < end && !ret; pte++, addr += PAGE_SIZE) { + ptent = ptep_get(pte); + is_written = !is_pte_uffd_wp(ptent); + + bitmap = PM_SCAN_FLAGS(is_written, pagemap_scan_is_file(vma, ptent, addr), + pte_present(ptent), is_swap_pte(ptent), + pte_present(ptent) && is_zero_pfn(pte_pfn(ptent))); + + if (IS_PM_SCAN_GET(p->flags)) { + interesting = pagemap_scan_is_interesting_page(bitmap, p); + if (interesting) + ret = pagemap_scan_output(bitmap, p, addr, 1); + } + + if (IS_PM_SCAN_WP(p->flags) && is_written && interesting && + ret >= 0) { + make_uffd_wp_pte(vma, addr, pte); + flush = true; + } + } + + if (flush) + flush_tlb_range(vma, start, addr); + + pte_unmap_unlock(orig_pte, ptl); + arch_leave_lazy_mmu_mode(); + + cond_resched(); + return ret; +} + +#ifdef CONFIG_HUGETLB_PAGE +static int pagemap_scan_hugetlb_entry(pte_t *ptep, unsigned long hmask, + unsigned long start, unsigned long end, + struct mm_walk *walk) +{ + unsigned long n_pages = (end - start)/PAGE_SIZE; + struct pagemap_scan_private *p = walk->private; + struct vm_area_struct *vma = walk->vma; + bool is_written, interesting = true; + struct hstate *h = hstate_vma(vma); + unsigned long bitmap; + spinlock_t *ptl; + int ret = 0; + pte_t ptent; + + if (IS_PM_SCAN_WP(p->flags) && n_pages < HPAGE_SIZE/PAGE_SIZE) { + p->end_addr = start; + return -EINVAL; + } + + if (n_pages > p->max_pages - p->found_pages) + n_pages = p->max_pages - p->found_pages; + + if (IS_PM_SCAN_WP(p->flags)) { + i_mmap_lock_write(vma->vm_file->f_mapping); + ptl = huge_pte_lock(h, vma->vm_mm, ptep); + } + + ptent = huge_ptep_get(ptep); + is_written = !is_huge_pte_uffd_wp(ptent); + + bitmap = PM_SCAN_FLAGS(is_written, pagemap_scan_is_huge_file(ptent), + pte_present(ptent), is_swap_pte(ptent), + pte_present(ptent) && is_zero_pfn(pte_pfn(ptent))); + + if (IS_PM_SCAN_GET(p->flags)) + interesting = pagemap_scan_is_interesting_page(bitmap, p); + + if (interesting) { + /* + * Partial hugetlb page clear isn't supported + */ + if (is_written && IS_PM_SCAN_WP(p->flags) && + n_pages < HPAGE_SIZE/PAGE_SIZE) { + ret = PM_SCAN_END_WALK; + p->end_addr = start; + goto unlock_and_return; + } + + if (IS_PM_SCAN_GET(p->flags)) + ret = pagemap_scan_output(bitmap, p, start, n_pages); + + if (IS_PM_SCAN_WP(p->flags) && is_written && ret >= 0) { + make_uffd_wp_huge_pte(vma, start, ptep, ptent); + flush_hugetlb_tlb_range(vma, start, end); + } + } + +unlock_and_return: + if (IS_PM_SCAN_WP(p->flags)) { + spin_unlock(ptl); + i_mmap_unlock_write(vma->vm_file->f_mapping); + } + + return ret; +} +#else +#define pagemap_scan_hugetlb_entry NULL +#endif + +static int pagemap_scan_pte_hole(unsigned long addr, unsigned long end, + int depth, struct mm_walk *walk) +{ + unsigned long n_pages = (end - addr)/PAGE_SIZE; + struct pagemap_scan_private *p = walk->private; + struct vm_area_struct *vma = walk->vma; + int ret = 0; + + if (!vma) + return 0; + + if (IS_PM_SCAN_GET(p->flags)) { + if (n_pages > p->max_pages - p->found_pages) + n_pages = p->max_pages - p->found_pages; + + ret = pagemap_scan_output(PM_SCAN_FLAGS(false, false, false, + false, false), p, addr, n_pages); + } + + if (IS_PM_SCAN_WP(p->flags) && + uffd_wp_range(vma, addr, end - addr, true) < 0) + ret = -EINVAL; + + return ret; +} + +static const struct mm_walk_ops pagemap_scan_ops = { + .test_walk = pagemap_scan_test_walk, + .pmd_entry = pagemap_scan_pmd_entry, + .pte_hole = pagemap_scan_pte_hole, + .hugetlb_entry = pagemap_scan_hugetlb_entry, +}; + +static int pagemap_scan_args_valid(struct pm_scan_arg *arg, unsigned long start, + unsigned long end, struct page_region __user *vec) +{ + /* Detect illegal size, flags, len and masks */ + if (arg->size != sizeof(struct pm_scan_arg)) + return -EINVAL; + if (!arg->flags) + return -EINVAL; + if (arg->flags & ~PM_SCAN_OPS) + return -EINVAL; + if (!(end - start)) + return -EINVAL; + if ((arg->required_mask | arg->anyof_mask | arg->excluded_mask | + arg->return_mask) & ~PM_SCAN_BITS_ALL) + return -EINVAL; + if (!arg->required_mask && !arg->anyof_mask && + !arg->excluded_mask) + return -EINVAL; + if (!arg->return_mask) + return -EINVAL; + + /* Validate memory range */ + if (!IS_ALIGNED(start, PAGE_SIZE)) + return -EINVAL; + if (!access_ok((void __user *)start, end - start)) + return -EFAULT; + + if (IS_PM_SCAN_GET(arg->flags)) { + if (arg->vec_len == 0) + return -EINVAL; + if (!vec) + return -EFAULT; + if (!access_ok((void __user *)vec, + arg->vec_len * sizeof(struct page_region))) + return -EFAULT; + } + + if (IS_PM_SCAN_WP(arg->flags) && !IS_PM_SCAN_GET(arg->flags) && + arg->max_pages) + return -EINVAL; + + return 0; +} + +static long do_pagemap_scan(struct mm_struct *mm, unsigned long __arg) +{ + struct pm_scan_arg __user *uarg = (struct pm_scan_arg __user *)__arg; + unsigned long long start, end, walk_start, walk_end; + unsigned long empty_slots, vec_index = 0; + struct mmu_notifier_range range; + struct page_region __user *vec; + struct pagemap_scan_private p; + struct pm_scan_arg arg; + int ret = 0; + + if (copy_from_user(&arg, uarg, sizeof(arg))) + return -EFAULT; + + start = untagged_addr((unsigned long)arg.start); + end = untagged_addr((unsigned long)arg.end); + vec = (struct page_region __user *)untagged_addr((unsigned long)arg.vec); + + ret = pagemap_scan_args_valid(&arg, start, end, vec); + if (ret) + return ret; + + p.max_pages = (arg.max_pages) ? arg.max_pages : ULONG_MAX; + p.found_pages = 0; + p.required_mask = arg.required_mask; + p.anyof_mask = arg.anyof_mask; + p.excluded_mask = arg.excluded_mask; + p.return_mask = arg.return_mask; + p.flags = arg.flags; + p.flags |= ((p.required_mask | p.anyof_mask | p.excluded_mask) & + PAGE_IS_WRITTEN) ? PM_SCAN_REQUIRE_UFFD : 0; + p.cur_buf.start = p.cur_buf.len = p.cur_buf.flags = 0; + p.vec_buf = NULL; + p.vec_buf_len = PAGEMAP_WALK_SIZE >> PAGE_SHIFT; + p.vec_buf_index = 0; + p.end_addr = 0; + + /* + * Allocate smaller buffer to get output from inside the page walk + * functions and walk page range in PAGEMAP_WALK_SIZE size chunks. As + * we want to return output to user in compact form where no two + * consecutive regions should be continuous and have the same flags. + * So store the latest element in p.cur_buf between different walks and + * store the p.cur_buf at the end of the walk to the user buffer. + */ + if (IS_PM_SCAN_GET(p.flags)) { + p.vec_buf = kmalloc_array(p.vec_buf_len, sizeof(*p.vec_buf), + GFP_KERNEL); + if (!p.vec_buf) + return -ENOMEM; + } + + /* + * Protection change for the range is going to happen. + */ + if (IS_PM_SCAN_WP(p.flags)) { + mmu_notifier_range_init(&range, MMU_NOTIFY_PROTECTION_VMA, 0, + mm, start, end); + mmu_notifier_invalidate_range_start(&range); + } + + walk_start = walk_end = start; + while (walk_end < end && !ret) { + if (IS_PM_SCAN_GET(p.flags)) { + /* + * All data is copied to cur_buf first. When more data + * is found, we push cur_buf to vec_buf and copy new + * data to cur_buf. Subtract 1 from length as the + * index of cur_buf isn't counted in length. + */ + empty_slots = arg.vec_len - vec_index; + p.vec_buf_len = min(p.vec_buf_len, empty_slots - 1); + } + + ret = mmap_read_lock_killable(mm); + if (ret) + goto out; + + walk_end = min((walk_start + PAGEMAP_WALK_SIZE) & PAGEMAP_WALK_MASK, end); + + ret = walk_page_range(mm, walk_start, walk_end, + &pagemap_scan_ops, &p); + mmap_read_unlock(mm); + + if (ret == PM_SCAN_FOUND_MAX_PAGES || ret == PM_SCAN_END_WALK) + arg.start = p.end_addr; + + if (ret && ret != PM_SCAN_FOUND_MAX_PAGES && + ret != PM_SCAN_END_WALK) + goto out; + + if (p.vec_buf_index) { + if (copy_to_user(&vec[vec_index], p.vec_buf, + p.vec_buf_index * sizeof(*p.vec_buf))) { + /* + * Return error even though the OP succeeded + */ + ret = -EFAULT; + goto out; + } + vec_index += p.vec_buf_index; + p.vec_buf_index = 0; + } + walk_start = walk_end; + } + + if (p.cur_buf.len) { + if (copy_to_user(&vec[vec_index], &p.cur_buf, sizeof(p.cur_buf))) { + ret = -EFAULT; + goto out; + } + vec_index++; + } + + ret = vec_index; + +out: + if (!p.end_addr) + arg.start = walk_start; + if (copy_to_user(&uarg->start, &arg.start, sizeof(arg.start))) + ret = -EFAULT; + + if (IS_PM_SCAN_WP(p.flags)) + mmu_notifier_invalidate_range_end(&range); + + kfree(p.vec_buf); + return ret; +} + +static long do_pagemap_cmd(struct file *file, unsigned int cmd, + unsigned long arg) +{ + struct mm_struct *mm = file->private_data; + + switch (cmd) { + case PAGEMAP_SCAN: + return do_pagemap_scan(mm, arg); + + default: + return -EINVAL; + } +} + const struct file_operations proc_pagemap_operations = { .llseek = mem_lseek, /* borrow this */ .read = pagemap_read, .open = pagemap_open, .release = pagemap_release, + .unlocked_ioctl = do_pagemap_cmd, + .compat_ioctl = do_pagemap_cmd, }; #endif /* CONFIG_PROC_PAGE_MONITOR */ diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 623e98d62df3..c44e7bde0339 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -259,6 +259,7 @@ long hugetlb_change_protection(struct vm_area_struct *vma, unsigned long cp_flags); bool is_hugetlb_entry_migration(pte_t pte); +bool is_hugetlb_entry_hwpoisoned(pte_t pte); void hugetlb_unshare_all_pmds(struct vm_area_struct *vma); #else /* !CONFIG_HUGETLB_PAGE */ diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h index b7b56871029c..ac684d99e68f 100644 --- a/include/uapi/linux/fs.h +++ b/include/uapi/linux/fs.h @@ -305,4 +305,59 @@ typedef int __bitwise __kernel_rwf_t; #define RWF_SUPPORTED (RWF_HIPRI | RWF_DSYNC | RWF_SYNC | RWF_NOWAIT |\ RWF_APPEND) +/* Pagemap ioctl */ +#define PAGEMAP_SCAN _IOWR('f', 16, struct pm_scan_arg) + +/* Bits are set in flags of the page_region and masks in pm_scan_args */ +#define PAGE_IS_WRITTEN (1 << 0) +#define PAGE_IS_FILE (1 << 1) +#define PAGE_IS_PRESENT (1 << 2) +#define PAGE_IS_SWAPPED (1 << 3) +#define PAGE_IS_PFNZERO (1 << 4) + +/* + * struct page_region - Page region with flags + * @start: Start of the region + * @len: Length of the region in pages + * @bitmap: Bits sets for the region + */ +struct page_region { + __u64 start; + __u64 len; + __u64 flags; +}; + +/* + * struct pm_scan_arg - Pagemap ioctl argument + * @size: Size of the structure + * @flags: Flags for the IOCTL + * @start: Starting address of the region + * (Ending address of the walk is also returned in it) + * @end: Ending address of the region + * @vec: Address of page_region struct array for output + * @vec_len: Length of the page_region struct array + * @max_pages: Optional max return pages + * @required_mask: Required mask - All of these bits have to be set in the PTE + * @anyof_mask: Any mask - Any of these bits are set in the PTE + * @excluded_mask: Exclude mask - None of these bits are set in the PTE + * @return_mask: Bits that are to be reported in page_region + */ +struct pm_scan_arg { + __u64 size; + __u64 flags; + __u64 start; + __u64 end; + __u64 vec; + __u64 vec_len; + __u64 max_pages; + __u64 required_mask; + __u64 anyof_mask; + __u64 excluded_mask; + __u64 return_mask; +}; + +/* Supported flags */ +#define PM_SCAN_OP_GET (1 << 0) +#define PM_SCAN_OP_WP (1 << 1) + #endif /* _UAPI_LINUX_FS_H */ diff --git a/mm/hugetlb.c b/mm/hugetlb.c index ee88a2a6649f..f7cefdcb114c 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -4981,7 +4981,7 @@ bool is_hugetlb_entry_migration(pte_t pte) return false; } -static bool is_hugetlb_entry_hwpoisoned(pte_t pte) +bool is_hugetlb_entry_hwpoisoned(pte_t pte) { swp_entry_t swp; From patchwork Tue Jul 11 12:52:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muhammad Usama Anjum X-Patchwork-Id: 13308669 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B970CEB64DC for ; Tue, 11 Jul 2023 12:53:42 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5752B6B0075; Tue, 11 Jul 2023 08:53:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 525C56B0078; Tue, 11 Jul 2023 08:53:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3C6E76B007D; Tue, 11 Jul 2023 08:53:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 2DD346B0075 for ; Tue, 11 Jul 2023 08:53:42 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 0642316020E for ; Tue, 11 Jul 2023 12:53:42 +0000 (UTC) X-FDA: 80999322684.13.8ED10F4 Received: from madras.collabora.co.uk (madras.collabora.co.uk [46.235.227.172]) by imf22.hostedemail.com (Postfix) with ESMTP id 31C4EC0011 for ; Tue, 11 Jul 2023 12:53:39 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=collabora.com header.s=mail header.b="KRhaIK0/"; dmarc=pass (policy=quarantine) header.from=collabora.com; spf=pass (imf22.hostedemail.com: domain of usama.anjum@collabora.com designates 46.235.227.172 as permitted sender) smtp.mailfrom=usama.anjum@collabora.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689080020; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=8wiGM7Q7eAABgqXCBfO0EzWj9ozwqucI7Vc0VSe2/BM=; b=ZTCS4WgD3m3Y0eX42+Et8DlUENxB2tcgpl5wIAxZcYcq9bCEHgKmnQ0j61Y0f3ZpLyUUfs Z+hqjTphv/VwOVA6YcCLx4W6eci2IVtu2KdpAFjU9nc8XvuZG5tK6wamN68QZgZ5FVYAD5 Zf9SDhmswQ4SPJj9NiZFJaAR2tfrCl0= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=collabora.com header.s=mail header.b="KRhaIK0/"; dmarc=pass (policy=quarantine) header.from=collabora.com; spf=pass (imf22.hostedemail.com: domain of usama.anjum@collabora.com designates 46.235.227.172 as permitted sender) smtp.mailfrom=usama.anjum@collabora.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689080020; a=rsa-sha256; cv=none; b=ooGU7eChZvfahvt1CQcRF1Dkh+Plx2L2MPRn0Hk9oxvvc9OOAcTOsc/4BOdRpO7Irs7Ldb +v6oh95f8vIkwYpXuV9jI4hBjrmwHmWJMByMySgPqfmIsiiwMszdHYmfRt4Fx5aW6wJBoR sPAsyN9rnzeLeYftz435ggaAczWBaK4= Received: from localhost.localdomain (unknown [119.155.40.122]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: usama.anjum) by madras.collabora.co.uk (Postfix) with ESMTPSA id 7A8EE6607013; Tue, 11 Jul 2023 13:53:31 +0100 (BST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=collabora.com; s=mail; t=1689080019; bh=5buco7QtB4jur/NsS/yDxCuMSRfJbuncKSNIdEednz0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=KRhaIK0/lH6VM/ilTrYSbtRQVut/Epn7J34PfRsOOuTr6NIzzLcBZ60c0h1ZXin47 wcVGslNjEOBColL26OJcDZEM9ZIvMbn8f2g4lwh/uVkU0YS85ITOx0780Z1ovfrOUd aEdw0kpimCLl4TuTfaasIkJHqveQjkrkGB9wF9dL/W9j8DPe9oEzXuHF/NTFf5RS7q uunzXTXufas3r3/GaJsTql7s7/mzSHQk6tcUv2Ne70zkCRhqVtft+MTjoo0Ys8+QE5 X3gYO0UTJEOW20/Q5yO+Lhqa6bKireAobpVXp/FqjIN1dHe9AwB4yl+h5GNHjgJLht w3MfsEIOHXvSg== From: Muhammad Usama Anjum To: Peter Xu , David Hildenbrand , Andrew Morton , =?utf-8?b?TWljaGHFgiBNaXJvc8WC?= =?utf-8?b?YXc=?= , Andrei Vagin , Danylo Mocherniuk , Paul Gofman , Cyrill Gorcunov , Mike Rapoport , Nadav Amit Cc: Alexander Viro , Shuah Khan , Christian Brauner , Yang Shi , Vlastimil Babka , "Liam R . Howlett" , Yun Zhou , Suren Baghdasaryan , Alex Sierra , Muhammad Usama Anjum , Matthew Wilcox , Pasha Tatashin , Axel Rasmussen , "Gustavo A . R . Silva" , Dan Williams , linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-kselftest@vger.kernel.org, Greg KH , kernel@collabora.com Subject: [PATCH v24 3/5] tools headers UAPI: Update linux/fs.h with the kernel sources Date: Tue, 11 Jul 2023 17:52:39 +0500 Message-Id: <20230711125241.1587820-4-usama.anjum@collabora.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230711125241.1587820-1-usama.anjum@collabora.com> References: <20230711125241.1587820-1-usama.anjum@collabora.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 31C4EC0011 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: 4zstzsywhd4j75pdxajzsk5jxmk3r7uu X-HE-Tag: 1689080019-327934 X-HE-Meta: U2FsdGVkX18hanIL4jLO3fuDuAMVXGFvdrxKBNuoo55Xgdzkoso1qUYJJOwsVxE2JheeNAvzYUhJbh1sc6SpKSk9bLHKQhDvnwnqZNDmszZYRaxDxKo83P6x/ziolOmzYTDsIGGh4XYw6Xu4hzMsKr8XJxFoK43gNgtQX1xLXzyCAsJNzNnICgO6T27mD1OO8NZBtQjZDXiq5KJnlcCdfJu3RMqCyiPW2sY1yeOWuS9FdK/4Tf7djLK04BQo4noXI5kL/4jWnKkNJdmrIGcJQtSVkyQQFx62GYqlT8Vvwwxam+5KcIPA/ARGqmmAP45mHRbiEW5PM8Uq1/jzjtUj3u5FfmRDJUsCc8gn6pyL9+/aQcNBE78fACQG3V0mvwHIKQ+OvnSnFYFfxSRjl94CV59ERpvUVKWF1m4lYzj3zGnTEqZAAULv0v7GC0XurHUgWnwrAPTcDVyUoNF2sPSzOHvqG5FLxSGHp8+2SuJOUhPt/D8R90LPpE/zUT2FigL0+lFAhcM11yviGSlAXSP91Och8dC80RGalvIoTkTwKnpyqpl7Q8ivqlx58k5lyfIivilOZ9K9EOyO3DNjS2w8Z6sOr/u67qBlQ7eCANXIJdLlHVK7LUMskePPjxRqkiIty+00MpGXck7RuwXoBKokRBzDVnDR6/+fSVO4jK49BX8CCZSNWwLwn5XP5u9W1/Vu6MK0nvR8EP7aLpbz04kN6f8LLXO8bH/1UvntW4C26x3ebQpyOakKKVO0hKFwrRAthUvbJ17bLwK2vkFEOL3SWsdohxEdIQZ3nqZIt0okRAX7P+p8mbDVYDjfiCHyDmKwNXVKs/xCkXrAd5eYj7h+NzPley4+rDMM318kSh7y814wEAxbAOYkTHaLTbon6LTnpDR51fBLJcqvuNeCj1HcPuzpNtnPrBnbiTpmM0IaBY4PaFTxbvIhtGuvu+4u1i7vxlzKs6K+AUiXObyPkqj 5lGDRcvv 79ZljgLVcMfm0xEKuVdDA5urjsEq+ay59wDbmU5Xmk6pGEF2iXgKik6s1NdlsKhLVxMz9i+F5tYBbsCEVeKR/Pb/HoKJbkyUljMrfQTLWEcP/E9liUCQRUPIH/A0hAi9tJdQIaq76asrjcGz8LHMfJT9aRmol0Ybr6Tdu/WtCDh/FwkhSntcoZeRPIGYnzPZhYavvWQTdfjZ74kX4qGJxq9ySfPQ7RGlxBOCT+4jqsK+63A2OBOK/g8rqpaWWm8iL9Fz7QJ+5p6EiXu4jSePMAQtXKb6stZ4tOyI75ow3qQslxkqZilFIINOPghmWVNyi5dDAQMhBiH6dExQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: New IOCTL and macros has been added in the kernel sources. Update the tools header file as well. Signed-off-by: Muhammad Usama Anjum --- Changes in v21: - Update tools/include/uapi/linux/fs.h Changes in v19: - Update fs.h accourding to precious patch --- tools/include/uapi/linux/fs.h | 55 +++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+) diff --git a/tools/include/uapi/linux/fs.h b/tools/include/uapi/linux/fs.h index b7b56871029c..ac684d99e68f 100644 --- a/tools/include/uapi/linux/fs.h +++ b/tools/include/uapi/linux/fs.h @@ -305,4 +305,59 @@ typedef int __bitwise __kernel_rwf_t; #define RWF_SUPPORTED (RWF_HIPRI | RWF_DSYNC | RWF_SYNC | RWF_NOWAIT |\ RWF_APPEND) +/* Pagemap ioctl */ +#define PAGEMAP_SCAN _IOWR('f', 16, struct pm_scan_arg) + +/* Bits are set in flags of the page_region and masks in pm_scan_args */ +#define PAGE_IS_WRITTEN (1 << 0) +#define PAGE_IS_FILE (1 << 1) +#define PAGE_IS_PRESENT (1 << 2) +#define PAGE_IS_SWAPPED (1 << 3) +#define PAGE_IS_PFNZERO (1 << 4) + +/* + * struct page_region - Page region with flags + * @start: Start of the region + * @len: Length of the region in pages + * @bitmap: Bits sets for the region + */ +struct page_region { + __u64 start; + __u64 len; + __u64 flags; +}; + +/* + * struct pm_scan_arg - Pagemap ioctl argument + * @size: Size of the structure + * @flags: Flags for the IOCTL + * @start: Starting address of the region + * (Ending address of the walk is also returned in it) + * @end: Ending address of the region + * @vec: Address of page_region struct array for output + * @vec_len: Length of the page_region struct array + * @max_pages: Optional max return pages + * @required_mask: Required mask - All of these bits have to be set in the PTE + * @anyof_mask: Any mask - Any of these bits are set in the PTE + * @excluded_mask: Exclude mask - None of these bits are set in the PTE + * @return_mask: Bits that are to be reported in page_region + */ +struct pm_scan_arg { + __u64 size; + __u64 flags; + __u64 start; + __u64 end; + __u64 vec; + __u64 vec_len; + __u64 max_pages; + __u64 required_mask; + __u64 anyof_mask; + __u64 excluded_mask; + __u64 return_mask; +}; + +/* Supported flags */ +#define PM_SCAN_OP_GET (1 << 0) +#define PM_SCAN_OP_WP (1 << 1) + #endif /* _UAPI_LINUX_FS_H */ From patchwork Tue Jul 11 12:52:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muhammad Usama Anjum X-Patchwork-Id: 13308670 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0B824EB64DC for ; Tue, 11 Jul 2023 12:53:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 797E36B0078; Tue, 11 Jul 2023 08:53:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 749966B007D; Tue, 11 Jul 2023 08:53:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6103B6B007E; Tue, 11 Jul 2023 08:53:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 50E746B0078 for ; Tue, 11 Jul 2023 08:53:51 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 24801140227 for ; Tue, 11 Jul 2023 12:53:51 +0000 (UTC) X-FDA: 80999323062.22.2AB3A2E Received: from madras.collabora.co.uk (madras.collabora.co.uk [46.235.227.172]) by imf10.hostedemail.com (Postfix) with ESMTP id B7D45C0009 for ; Tue, 11 Jul 2023 12:53:48 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=collabora.com header.s=mail header.b=MdARlwDF; dmarc=pass (policy=quarantine) header.from=collabora.com; spf=pass (imf10.hostedemail.com: domain of usama.anjum@collabora.com designates 46.235.227.172 as permitted sender) smtp.mailfrom=usama.anjum@collabora.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689080028; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Dw8wQ7eNGIBkh6JF897pShFRKy2LknhUMR1eis2YMj4=; b=rIx/dddpH+8EfjXi+GSzsfbM9KXFb9bQDnKww0CwrUxZVwuitWys3aP+LFHAMsjYUFEkWi DwP2gG4bXNC5OhnIuPtcyw/SGodZDpe76b1CEuvntb2n7VmDnOfaYwDBJ2PyxkmLpkLVa3 nd228aSNpUO1kQy7ORFmBxp8bNDHLfc= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=collabora.com header.s=mail header.b=MdARlwDF; dmarc=pass (policy=quarantine) header.from=collabora.com; spf=pass (imf10.hostedemail.com: domain of usama.anjum@collabora.com designates 46.235.227.172 as permitted sender) smtp.mailfrom=usama.anjum@collabora.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689080029; a=rsa-sha256; cv=none; b=Ryu/qJvlowvfwcUiMF8fc9l9laJzRX4mfzTobj2dXKbjW3KwtP+JrYYk73uO535SY11t1x pxFi6SoVBtdVhG5yTzqMVDhLg/CSrvo7XXSfYZvxzhp5d5RoFZx05UHiz0K9aLDEGwxq6V p4Yy9HC9ScoHy0UsGwj9zitznGmeYOg= Received: from localhost.localdomain (unknown [119.155.40.122]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: usama.anjum) by madras.collabora.co.uk (Postfix) with ESMTPSA id B61B5660217D; Tue, 11 Jul 2023 13:53:39 +0100 (BST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=collabora.com; s=mail; t=1689080027; bh=e3RwcWLS0DN4jGAQ1nXg3KEjRQ1a6mnaZglmii6vuAs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=MdARlwDFQDKScisXGCtEwJ9rzc9ku5dA1jLj2OZ1T00ExwO5tt2SmfX1hEFuAsd9+ 6LeHKU7sHQoFP2XdfPRRF6vEThylU/M7M/rM9KfOsOqm8k5/fgryA8ToQZ2t9iiJge cq3UOpVkRrhLNthVwSvFMjCp2MlXgw7HH+aj6Loi8jrF4WkHUqFK2Un4CafgcVW7sH 4A+iRmTMYrZpp5i/iKd+Gi5wyO6V04TBMnvbK579rqxsQJEnsn7ZzaQee8t0K7eA6y jPDvLFjgWAvV6USfK7nDQqjt3QHIky7ng7QUfXfC47leRaLQQTb4aNMXAyRfyuggIu JnpJFXyQF1ldw== From: Muhammad Usama Anjum To: Peter Xu , David Hildenbrand , Andrew Morton , =?utf-8?b?TWljaGHFgiBNaXJvc8WC?= =?utf-8?b?YXc=?= , Andrei Vagin , Danylo Mocherniuk , Paul Gofman , Cyrill Gorcunov , Mike Rapoport , Nadav Amit Cc: Alexander Viro , Shuah Khan , Christian Brauner , Yang Shi , Vlastimil Babka , "Liam R . Howlett" , Yun Zhou , Suren Baghdasaryan , Alex Sierra , Muhammad Usama Anjum , Matthew Wilcox , Pasha Tatashin , Axel Rasmussen , "Gustavo A . R . Silva" , Dan Williams , linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-kselftest@vger.kernel.org, Greg KH , kernel@collabora.com Subject: [PATCH v24 4/5] mm/pagemap: add documentation of PAGEMAP_SCAN IOCTL Date: Tue, 11 Jul 2023 17:52:40 +0500 Message-Id: <20230711125241.1587820-5-usama.anjum@collabora.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230711125241.1587820-1-usama.anjum@collabora.com> References: <20230711125241.1587820-1-usama.anjum@collabora.com> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: etq11abmn6w3atwwg9ebx9axzf1wqxyb X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: B7D45C0009 X-HE-Tag: 1689080028-528225 X-HE-Meta: U2FsdGVkX19qUva1bbz2RLY3s5eAFR1Mbcei8lj9fRNRrSpcykRA0013mGLldTlBf8zNRfMoJzloeN3RDSjdIxwnOIQ9TSs/W71beh/I8PwPW5yOI+RSI9hyPfoJfnVx4nDDj7nz48gOzefaud1xDX/PjxwjTbvUVrLZBY6uScqNyOYqlcr05RaYfwvOO49qrMnDtr322Q7ke6Yf41wi7sfwVlz/oUr2zeJ1UfFZqS7UCCVkBkWHHFAl44dSTsQ1Wrq2nJH31oDe/GTTDcwspfzKMpNt7Zd698GW2SpMUNILVsbwzwJIUE1l8+EOCMozj0aR/5QZbSWsbAL27TefSBfDwzrLnVejLqSmQVi6/Zs6f1LRAYrJvimGlmP5BDxHt0et+BNEQlv60lRdPO0vpJbC0YSqdDKk//R+Sz+H8ZwYecBSU2Ne0o5kPHYAl/0YHk4uhOYdZ0iaD0OsyDBaCxwr/Cg7+H8sXIlaBsMN1oZDeTLiuQ4SRnCwh5ywWVm/RGWilAnTSukuVlRzsy4xIqSVQMbOGcpm69HNOY1UJ7rAD2T5dGKxU8Xj2GpIsGacF8Y+Yl2t39d91ajLukbwXi2Nkm9AStObUrzR6vfrJ6eRd4pTDwzrIdUrHRg2BS1I6uY4lYjARpxJjj4UkEwNr3NijvhW97bAXDHhKnM03Joh71xM9bPlJWkK6pJl7y1DxLGxFfNjFm3sORJOtqogeN4SJsobFhLE9flzdVO3xfY54h3QzFjiT2Keg2QIk4ktAJ38mroc+M5DGCjqT3lI9K97fBFdjy0kDTa4t1LsOY2Mgr6hMJAIg13VwIxQLallE5WBufSwl+AM4SB+jAr9pOcEG6wvzCbAAWIp4yApwoP4qp3b5YFFN/UgkOgePeTKdBuLCyI2NWyySxEh+hQoK6m/OY3RmYd1e18JdeFtIn+gRO79EC/HElkktt9xoW5W/6l/wFNT1JOJfQ+8VGS s9gTQKqB OkqKaygZnHGSbwcr8I1/H634YXocyBDtHVuzGm0q4khU1CcwQw4P9ASWg3ZMGLcYIMxlxSfyr2u1KC7XIA8CyKMwUkzuu0L7QjTW1Q6kLWWk3Rc4YO4OIMY2QXXI2sauRHoUwBs5McJsISCZ68iqU7oLsS0yi/qdaz0OC+zl0EnNESSE7d7NjxtBnutVtzke4ZOtlxuXoGmOEHZFI/LKMxpJs5yaqo97nYLra3YvTqOGnNz9wF4jGMNcXioy8FwLeQ1pcY0o0gL5U000EKQdCNak1MoGCpFq3b5y6k96DYA4mQYo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add some explanation and method to use write-protection and written-to on memory range. Signed-off-by: Muhammad Usama Anjum --- Changes in v21: - Update according to changed interface Changes in v20: - Add new flag in documentation Changes in v19: - Improved documentatiom with punctuation marks Changes in v16: - Update the documentation Changes in v11: - Add more documentation rst --- Documentation/admin-guide/mm/pagemap.rst | 58 ++++++++++++++++++++++++ 1 file changed, 58 insertions(+) diff --git a/Documentation/admin-guide/mm/pagemap.rst b/Documentation/admin-guide/mm/pagemap.rst index c8f380271cad..4bdad8eff009 100644 --- a/Documentation/admin-guide/mm/pagemap.rst +++ b/Documentation/admin-guide/mm/pagemap.rst @@ -227,3 +227,61 @@ Before Linux 3.11 pagemap bits 55-60 were used for "page-shift" (which is always 12 at most architectures). Since Linux 3.11 their meaning changes after first clear of soft-dirty bits. Since Linux 4.2 they are used for flags unconditionally. + +Pagemap Scan IOCTL +================== + +The ``PAGEMAP_SCAN`` IOCTL on the pagemap file can be used to get or optionally +clear the info about page table entries. The following operations are supported +in this IOCTL: +- Get the information if the pages have been written to (``PAGE_IS_WRITTEN``), + file mapped (``PAGE_IS_FILE``), present (``PAGE_IS_PRESENT``), swapped + (``PAGE_IS_SWAPPED``) or page has pfn zero (``PAGE_IS_PFNZERO``). +- Find pages which have been written to and/or write protect the pages atomically + (atomic ``PM_SCAN_OP_GET + PM_SCAN_OP_WP``) + +The ``struct pm_scan_arg`` is used as the argument of the IOCTL. + 1. The size of the ``struct pm_scan_arg`` must be specified in the ``size`` + field. This field will be helpful in recognizing the structure if extensions + are done later. + 2. The flags can be specified in the ``flags`` field. The ``PM_SCAN_OP_GET`` + and ``PM_SCAN_OP_WP`` are the only added flags at this time. + 3. The range is specified through ``start`` and ``end``. + 4. The output buffer of ``struct page_region`` array and size is specified in + ``vec`` and ``vec_len``. + 5. The optional maximum requested pages are specified in the ``max_pages``. + 6. The masks are specified in ``required_mask``, ``anyof_mask``, + ``excluded_ mask`` and ``return_mask``. + 1. To find if ``PAGE_IS_WRITTEN`` flag is set for pages which have + ``PAGE_IS_FILE`` set and ``PAGE_IS_SWAPPED`` unset, ``required_mask`` + is set to ``PAGE_IS_FILE``, ``exclude_mask`` is set to + ``PAGE_IS_SWAPPED`` and ``return_mask`` is set to ``PAGE_IS_WRITTEN``. + The output buffer in ``vec`` and length must be specified in ``vec_len``. + 2. To find pages which have either ``PAGE_IS_FILE`` or ``PAGE_IS_SWAPPED`` + set, ``anyof_masks`` is set to ``PAGE_IS_FILE | PAGE_IS_SWAPPED``. + 3. To find written pages and engage write protect, ``PAGE_IS_WRITTEN`` is + specified in ``required_mask`` and ``return_mask``. In addition to + specifying the output buffer in ``vec`` and length in ``vec_len``, the + ``PM_SCAN_OP_WP`` is specified in ``flags`` to perform write protect + on the range as well. + +The ``PAGE_IS_WRITTEN`` flag can be considered as the better and correct +alternative of soft-dirty flag. It doesn't get affected by housekeeping chores +(VMA merging) of the kernel and hence the user can find the true soft-dirty pages +only. This IOCTL adds the atomic way to find which pages have been written and +write protect those pages again. This kind of operation is needed to efficiently +find out which pages have changed in the memory. + +To get information about which pages have been written to or optionally write +protect the pages, following must be performed first in order: + 1. The userfaultfd file descriptor is created with ``userfaultfd`` syscall. + 2. The ``UFFD_FEATURE_WP_UNPOPULATED`` and ``UFFD_FEATURE_WP_ASYNC`` features + are set by ``UFFDIO_API`` IOCTL. + 3. The memory range is registered with ``UFFDIO_REGISTER_MODE_WP`` mode + through ``UFFDIO_REGISTER`` IOCTL. + 4. Then any part of the registered memory or the whole memory region must + be write protected using ``PAGEMAP_SCAN`` IOCTL with flag ``PM_SCAN_OP_WP`` + or the ``UFFDIO_WRITEPROTECT`` IOCTL can be used. Both of these perform the + same operation. The former is better in terms of performance. + 5. Now the ``PAGEMAP_SCAN`` IOCTL can be used to either just find pages which + have been written to and/or optionally write protect the pages as well. From patchwork Tue Jul 11 12:52:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muhammad Usama Anjum X-Patchwork-Id: 13308671 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 61F4BEB64DC for ; Tue, 11 Jul 2023 12:54:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E9CD76B0071; Tue, 11 Jul 2023 08:53:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E4D106B0075; Tue, 11 Jul 2023 08:53:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C51808D0001; Tue, 11 Jul 2023 08:53:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id AE7A66B0075 for ; Tue, 11 Jul 2023 08:53:59 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 5C8D01A0253 for ; Tue, 11 Jul 2023 12:53:59 +0000 (UTC) X-FDA: 80999323398.06.D9B6DF0 Received: from madras.collabora.co.uk (madras.collabora.co.uk [46.235.227.172]) by imf20.hostedemail.com (Postfix) with ESMTP id E063B1C002F for ; Tue, 11 Jul 2023 12:53:56 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=collabora.com header.s=mail header.b=eVJhS77f; spf=pass (imf20.hostedemail.com: domain of usama.anjum@collabora.com designates 46.235.227.172 as permitted sender) smtp.mailfrom=usama.anjum@collabora.com; dmarc=pass (policy=quarantine) header.from=collabora.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689080037; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nRobQ2IlM+33fQ2PC84IrBYgNAm6UpBumawPglp3HIg=; b=4qrKDx6fCSee8gP+1T4S8jYmXx4ySCzp+azsKstEoNRZcnq8lOhC/esCatSPKEWbYTDWJA frDu3S0XnUAnADiCl9tql5CrjI1BrfUX7/kS5bpCnAQlin9J/w57UMWE2xreqkVPmJRiMC jq6btKP9vJGcKG9NWra1onT8WLz99u8= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689080037; a=rsa-sha256; cv=none; b=fC/+xsjb80phQtPxJKaIhKNxNk5dhUvJM8jlD0pH1uFL8Z9Zc/Gf9N66xsBsscVDPz/o5b 9t5XtWoyJfic0MtPn5WqAOUV0NWXQ2A9f3Fmzbl6lNg6fix00l8MwT8Iv1vP7019tXJK7F PH0pUxsQQpj7bXCNa38809xN7UnaQw4= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=collabora.com header.s=mail header.b=eVJhS77f; spf=pass (imf20.hostedemail.com: domain of usama.anjum@collabora.com designates 46.235.227.172 as permitted sender) smtp.mailfrom=usama.anjum@collabora.com; dmarc=pass (policy=quarantine) header.from=collabora.com Received: from localhost.localdomain (unknown [119.155.40.122]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) (Authenticated sender: usama.anjum) by madras.collabora.co.uk (Postfix) with ESMTPSA id B3A2A6607010; Tue, 11 Jul 2023 13:53:47 +0100 (BST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=collabora.com; s=mail; t=1689080035; bh=jYokwnCQtGHqLa0IJTMTK6ChhhTb5/rIiva5u53aJ6U=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=eVJhS77fuR2d5PHQMCybcA66cWUvgaGrG2PtZUkJxwwklc2t6QTAEEm6sEZ11iOGi L5vYg77vQSS+rHCARAYIpU93ieJoZO+mwkuSdCtHDULB/jcKd0cs69c/FbtLZFGnMJ KNCni6QA6E/HwFc5KnYRA2ig35xyOKSf9FuRERH/t4qb2rY7AVWxIrwX9KVzhbXgo7 QWdQxiTi+m5jLn9X8bMAb+KSAZUg51HHonGexhq47ErxfQsexlgNEWxGjq4D3TJGmi Gl4b0mz3yLgxEU4Ku6tsubNzrzgBimpzJlPxeR0httYdIFec91+7ncI4If6Im0yfFr NaADKSG/oEIQQ== From: Muhammad Usama Anjum To: Peter Xu , David Hildenbrand , Andrew Morton , =?utf-8?b?TWljaGHFgiBNaXJvc8WC?= =?utf-8?b?YXc=?= , Andrei Vagin , Danylo Mocherniuk , Paul Gofman , Cyrill Gorcunov , Mike Rapoport , Nadav Amit Cc: Alexander Viro , Shuah Khan , Christian Brauner , Yang Shi , Vlastimil Babka , "Liam R . Howlett" , Yun Zhou , Suren Baghdasaryan , Alex Sierra , Muhammad Usama Anjum , Matthew Wilcox , Pasha Tatashin , Axel Rasmussen , "Gustavo A . R . Silva" , Dan Williams , linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, linux-kselftest@vger.kernel.org, Greg KH , kernel@collabora.com Subject: [PATCH v24 5/5] selftests: mm: add pagemap ioctl tests Date: Tue, 11 Jul 2023 17:52:41 +0500 Message-Id: <20230711125241.1587820-6-usama.anjum@collabora.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230711125241.1587820-1-usama.anjum@collabora.com> References: <20230711125241.1587820-1-usama.anjum@collabora.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: E063B1C002F X-Rspam-User: X-Stat-Signature: ypgxzggb1gyia19wpjx968gja9b5yzkp X-Rspamd-Server: rspam03 X-HE-Tag: 1689080036-877870 X-HE-Meta: U2FsdGVkX1+vaORrxcCWY98b+lSBbRjnhqTIN1zJtUiTP4IngqKVUkgr59twjG1Z7YbIyt9ng5m+Xcr7nWAQxiW/zcq+0z/N3Ymqojrf75kzOjmhFmDQVI/7QeuK35Fv0F/8eR4LbGmaJuWZraNFulgJA0f8MDy54Yt7G0RESRjMjPwh+iJzQPf20Qa0e+wXQN4ioObfNH2iFnPmqXpFAJu25RXkvmJA6YL/qdkfl05HJ4xSqllkox+nJWyDZpvWPIEisrpFUsr0RXmC8IFeOM3Yb+xXXhAsxMTXy2ST5M7F4Gr99cDYy2aTYRsACKEg4Y4zbADzxii/8owWIMhlcXXFER3S5IsYFLGFI/q3wqxv0IVCEiHo0OioA/jWGBj/qYOJMtwZ9SrTyuThLv8p9x4ZFzwgoz/imKd/NTy2bW+dLpNNiLqnyUvwZ9yFloIEql65SQCe7jar2Yz5Mz3jp/KxJjbHcoNDGZndXCA1Q8go7jeIE41DRv8JDOTN25z6OHdvH0s4bhcFjbx5GqKH085N5ikTsjcMMdz5p3PxUGTPxDbMnRBhBVIlqc//E2YBp9fo/d4Oa/gYZrlsWI+fG0KxVeC+/3bmfg95vEAX9Wgl9Ocj5TyxLbch64Eyy1lqRDjaqzklBj/dBETfPfqTk2aacM6k6kka9NFwtGHRS6nJMv7q3Huat5E9kdpgTJ1Gtjb/4X746sZnJaHsZCUNJy4UfikSiQAocYWxk7tlwuuba26S0fiV5xtwrEHSseGHN16xhclUlbLKxiv/3ctNOkEl+4lYYYhQRiarAG1CioqbKnVmwdwjW0NheN+KVbOGOjRJNNSc/y3c9kkpFSfQtwL2YDxl1+NzVdw4JgrzHAJfJxY1wipM7fBxxgdPtsMCAtIPcEsztnWA3yMSAyb8H+2Z9dE2kzY4enMT45VkxbbyGA2yrKMRMg2dYWs1u0t9EACVMCmN1d5iVM1q8TY Vt3KU7vG o7EPnjlVyARGnB6AI8QAEdB/yt0I7X5Ybqjvwp9w8xiG6sOMuMBkR29sLd7BEwcR3Lrbyd9FNB+By4NH0Mnh3ccxQM6tKa2C69vzID2HC1/f19XUXP7FL9zXU4nOP9nje17T4gM+vSSjgX+ooKSYol2A5AHJlTre+FlClHkoJTMUcrego9H9kFH2jBkKGLc+SQf9nMt0bPNxQVOryLJpmaKR+rSS0wIXq93MhM8Ik4qXKRiaq103yDGTuq/0GCItL4iKehWTANro06CHhx3AvvTEwg0sdbty5WTJgCgecPkq1JkcF/4vN5s1H4ZI3Omc3L6ThHEE7ukqN5Zre+tZ9NhS3HFO9Q0qy1btORnAxZr7qETmjsLIwF9tmOg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add pagemap ioctl tests. Add several different types of tests to judge the correction of the interface. Signed-off-by: Muhammad Usama Anjum --- Changes in v19: - Incorporated interface changes and update tests Changes in v18: - Rebase on top of 20230613 (Resolve conflict in Makefile) - Add temp files to .gitignore Changes in v17: - Rebase on top of next-20230525 Changes in v16: - Added yet more tests which is a randomization test case to catch the corner cases - Add reset by exclusive PM_SCAN_OP_WP as well Changes in v13: - Update tests and rebase Makefile Changes in v12: - Updates and add more memory type tests Changes in v11: - Rebase on top of next-20230216 and update tests Chages in v7: - Add and update all test cases Changes in v6: - Rename variables Changes in v4: - Updated all the tests to conform to new IOCTL Changes in v3: - Add another test to do sanity of flags Changes in v2: - Update the tests to use the ioctl interface instead of syscall --- TAP version 13 1..92 ok 1 sanity_tests_sd memory size must be valid ok 2 sanity_tests_sd output buffer must be specified ok 3 sanity_tests_sd output buffer size must be valid ok 4 sanity_tests_sd wrong flag specified ok 5 sanity_tests_sd flag has extra bits specified ok 6 sanity_tests_sd no selection mask is specified ok 7 sanity_tests_sd no return mask is specified ok 8 sanity_tests_sd wrong return mask specified ok 9 sanity_tests_sd mixture of correct and wrong flag ok 10 sanity_tests_sd PAGEMAP_BITS_ALL cannot be specified with PM_SCAN_OP_WP ok 11 sanity_tests_sd Clear area with larger vec size ok 12 sanity_tests_sd Repeated pattern of written and non-written pages ok 13 sanity_tests_sd Repeated pattern of written and non-written pages in parts ok 14 sanity_tests_sd Repeated pattern of written and non-written pages max_pages ok 15 sanity_tests_sd only get 2 written pages and clear them as well ok 16 sanity_tests_sd Two regions ok 17 sanity_tests_sd Smaller max_pages ok 18 Smaller vec 46 50 ok 19 Page testing: all new pages must not be written (dirty) ok 20 Page testing: all pages must be written (dirty) ok 21 Page testing: all pages dirty other than first and the last one ok 22 Page testing: PM_SCAN_OP_WP ok 23 Page testing: only middle page dirty ok 24 Page testing: only two middle pages dirty ok 25 Large Page testing: all new pages must not be written (dirty) ok 26 Large Page testing: all pages must be written (dirty) ok 27 Large Page testing: all pages dirty other than first and the last one ok 28 Large Page testing: PM_SCAN_OP_WP ok 29 Large Page testing: only middle page dirty ok 30 Large Page testing: only two middle pages dirty ok 31 Huge page testing: all new pages must not be written (dirty) ok 32 Huge page testing: all pages must be written (dirty) ok 33 Huge page testing: all pages dirty other than first and the last one ok 34 Huge page testing: PM_SCAN_OP_WP ok 35 Huge page testing: only middle page dirty ok 36 Huge page testing: only two middle pages dirty ok 37 Hugetlb shmem testing: all new pages must not be written (dirty) ok 38 Hugetlb shmem testing: all pages must be written (dirty) ok 39 Hugetlb shmem testing: all pages dirty other than first and the last one ok 40 Hugetlb shmem testing: PM_SCAN_OP_WP ok 41 Hugetlb shmem testing: only middle page dirty ok 42 Hugetlb shmem testing: only two middle pages dirty ok 43 Hugetlb mem testing: all new pages must not be written (dirty) ok 44 Hugetlb mem testing: all pages must be written (dirty) ok 45 Hugetlb mem testing: all pages dirty other than first and the last one ok 46 Hugetlb mem testing: PM_SCAN_OP_WP ok 47 Hugetlb mem testing: only middle page dirty ok 48 Hugetlb mem testing: only two middle pages dirty ok 49 File memory testing: all new pages must not be written (dirty) ok 50 File memory testing: all pages must be written (dirty) ok 51 File memory testing: all pages dirty other than first and the last one ok 52 File memory testing: PM_SCAN_OP_WP ok 53 File memory testing: only middle page dirty ok 54 File memory testing: only two middle pages dirty ok 55 File anonymous memory testing: all new pages must not be written (dirty) ok 56 File anonymous memory testing: all pages must be written (dirty) ok 57 File anonymous memory testing: all pages dirty other than first and the last one ok 58 File anonymous memory testing: PM_SCAN_OP_WP ok 59 File anonymous memory testing: only middle page dirty ok 60 File anonymous memory testing: only two middle pages dirty ok 61 hpage_unit_tests all new huge page must not be written (dirty) ok 62 hpage_unit_tests all the huge page must not be written ok 63 hpage_unit_tests all the huge page must be written and clear ok 64 hpage_unit_tests only middle page written ok 65 hpage_unit_tests clear first half of huge page ok 66 hpage_unit_tests clear first half of huge page with limited buffer ok 67 hpage_unit_tests clear second half huge page ok 68 hpage_unit_tests get half huge page ok 69 hpage_unit_tests get half huge page ok 70 Test test_simple ok 71 mprotect_tests Both pages written ok 72 mprotect_tests Both pages are not written (dirty) ok 73 mprotect_tests Both pages written after remap and mprotect ok 74 mprotect_tests Clear and make the pages written ok 75 transact_test count 192 ok 76 transact_test count 0 ok 77 transact_test Extra pages 0 (0.0%), extra thread faults 0. ok 78 sanity_tests clear op can only be specified with PAGE_IS_WRITTEN ok 79 sanity_tests required_mask specified ok 80 sanity_tests anyof_mask specified ok 81 sanity_tests excluded_mask specified ok 82 sanity_tests required_mask and anyof_mask specified ok 83 sanity_tests Get sd and present pages with anyof_mask ok 84 sanity_tests Get all the pages with required_mask ok 85 sanity_tests Get sd and present pages with required_mask and anyof_mask ok 86 sanity_tests Don't get sd pages ok 87 sanity_tests Don't get present pages ok 88 sanity_tests Find written present pages with return mask ok 89 sanity_tests Memory mapped file ok 90 sanity_tests Read/write to private memory mapped file ok 91 unmapped_region_tests Get status of pages ok 92 userfaultfd_tests all new pages must not be written (dirty) # Totals: pass:92 fail:0 xfail:0 xpass:0 skip:0 error:0 --- tools/testing/selftests/mm/.gitignore | 2 + tools/testing/selftests/mm/Makefile | 3 +- tools/testing/selftests/mm/config | 1 + tools/testing/selftests/mm/pagemap_ioctl.c | 1464 ++++++++++++++++++++ tools/testing/selftests/mm/run_vmtests.sh | 4 + 5 files changed, 1473 insertions(+), 1 deletion(-) create mode 100644 tools/testing/selftests/mm/pagemap_ioctl.c mode change 100644 => 100755 tools/testing/selftests/mm/run_vmtests.sh diff --git a/tools/testing/selftests/mm/.gitignore b/tools/testing/selftests/mm/.gitignore index cdc9ce4426b9..cc920c79ff1c 100644 --- a/tools/testing/selftests/mm/.gitignore +++ b/tools/testing/selftests/mm/.gitignore @@ -18,6 +18,8 @@ mremap_dontunmap mremap_test on-fault-limit transhuge-stress +pagemap_ioctl +*.tmp* protection_keys protection_keys_32 protection_keys_64 diff --git a/tools/testing/selftests/mm/Makefile b/tools/testing/selftests/mm/Makefile index b7fce9073279..1dac4e8477e4 100644 --- a/tools/testing/selftests/mm/Makefile +++ b/tools/testing/selftests/mm/Makefile @@ -33,7 +33,7 @@ endif MAKEFLAGS += --no-builtin-rules CFLAGS = -Wall -I $(top_srcdir) $(EXTRA_CFLAGS) $(KHDR_INCLUDES) -LDLIBS = -lrt -lpthread +LDLIBS = -lrt -lpthread -lm TEST_GEN_PROGS = cow TEST_GEN_PROGS += compaction_test @@ -60,6 +60,7 @@ TEST_GEN_PROGS += mrelease_test TEST_GEN_PROGS += mremap_dontunmap TEST_GEN_PROGS += mremap_test TEST_GEN_PROGS += on-fault-limit +TEST_GEN_PROGS += pagemap_ioctl TEST_GEN_PROGS += thuge-gen TEST_GEN_PROGS += transhuge-stress TEST_GEN_PROGS += uffd-stress diff --git a/tools/testing/selftests/mm/config b/tools/testing/selftests/mm/config index be087c4bc396..4309916f629e 100644 --- a/tools/testing/selftests/mm/config +++ b/tools/testing/selftests/mm/config @@ -1,5 +1,6 @@ CONFIG_SYSVIPC=y CONFIG_USERFAULTFD=y +CONFIG_PTE_MARKER_UFFD_WP=y CONFIG_TEST_VMALLOC=m CONFIG_DEVICE_PRIVATE=y CONFIG_TEST_HMM=m diff --git a/tools/testing/selftests/mm/pagemap_ioctl.c b/tools/testing/selftests/mm/pagemap_ioctl.c new file mode 100644 index 000000000000..a2d2d9086b83 --- /dev/null +++ b/tools/testing/selftests/mm/pagemap_ioctl.c @@ -0,0 +1,1464 @@ +// SPDX-License-Identifier: GPL-2.0 +#define _GNU_SOURCE +#include +#include +#include +#include +#include +#include +#include "vm_util.h" +#include "../kselftest.h" +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#define PAGEMAP_BITS_ALL (PAGE_IS_WRITTEN | PAGE_IS_FILE | \ + PAGE_IS_PRESENT | PAGE_IS_SWAPPED) +#define PAGEMAP_NON_WRITTEN_BITS (PAGE_IS_FILE | PAGE_IS_PRESENT | \ + PAGE_IS_SWAPPED) + +#define TEST_ITERATIONS 100 +#define PAGEMAP "/proc/self/pagemap" +int pagemap_fd; +int uffd; +int page_size; +int hpage_size; + +static long pagemap_ioctl(void *start, int len, void *vec, int vec_len, int flag, + int max_pages, long required_mask, long anyof_mask, long excluded_mask, + long return_mask) +{ + struct pm_scan_arg arg; + + arg.start = (uintptr_t)start; + arg.end = (uintptr_t)(start + len); + arg.vec = (uintptr_t)vec; + arg.vec_len = vec_len; + arg.flags = flag; + arg.size = sizeof(struct pm_scan_arg); + arg.max_pages = max_pages; + arg.required_mask = required_mask; + arg.anyof_mask = anyof_mask; + arg.excluded_mask = excluded_mask; + arg.return_mask = return_mask; + + return ioctl(pagemap_fd, PAGEMAP_SCAN, &arg); +} + +int init_uffd(void) +{ + struct uffdio_api uffdio_api; + + uffd = syscall(__NR_userfaultfd, O_CLOEXEC | O_NONBLOCK); + if (uffd == -1) + ksft_exit_fail_msg("uffd syscall failed\n"); + + uffdio_api.api = UFFD_API; + uffdio_api.features = UFFD_FEATURE_WP_UNPOPULATED | UFFD_FEATURE_WP_ASYNC | + UFFD_FEATURE_WP_HUGETLBFS_SHMEM; + if (ioctl(uffd, UFFDIO_API, &uffdio_api)) + ksft_exit_fail_msg("UFFDIO_API\n"); + + if (!(uffdio_api.api & UFFDIO_REGISTER_MODE_WP) || + !(uffdio_api.features & UFFD_FEATURE_WP_UNPOPULATED) || + !(uffdio_api.features & UFFD_FEATURE_WP_ASYNC) || + !(uffdio_api.features & UFFD_FEATURE_WP_HUGETLBFS_SHMEM)) + ksft_exit_fail_msg("UFFDIO_API error %llu\n", uffdio_api.api); + + return 0; +} + +int wp_init(void *lpBaseAddress, int dwRegionSize) +{ + struct uffdio_register uffdio_register; + struct uffdio_writeprotect wp; + + uffdio_register.range.start = (unsigned long)lpBaseAddress; + uffdio_register.range.len = dwRegionSize; + uffdio_register.mode = UFFDIO_REGISTER_MODE_WP; + if (ioctl(uffd, UFFDIO_REGISTER, &uffdio_register)) + ksft_exit_fail_msg("ioctl(UFFDIO_REGISTER) %d %s\n", errno, strerror(errno)); + + if (!(uffdio_register.ioctls & UFFDIO_WRITEPROTECT)) + ksft_exit_fail_msg("ioctl set is incorrect\n"); + + wp.range.start = (unsigned long)lpBaseAddress; + wp.range.len = dwRegionSize; + wp.mode = UFFDIO_WRITEPROTECT_MODE_WP; + + if (ioctl(uffd, UFFDIO_WRITEPROTECT, &wp)) + ksft_exit_fail_msg("ioctl(UFFDIO_WRITEPROTECT)\n"); + + return 0; +} + +int wp_free(void *lpBaseAddress, int dwRegionSize) +{ + struct uffdio_register uffdio_register; + + uffdio_register.range.start = (unsigned long)lpBaseAddress; + uffdio_register.range.len = dwRegionSize; + uffdio_register.mode = UFFDIO_REGISTER_MODE_WP; + if (ioctl(uffd, UFFDIO_UNREGISTER, &uffdio_register.range)) + ksft_exit_fail_msg("ioctl unregister failure\n"); + return 0; +} + +int wp_addr_range(void *lpBaseAddress, int dwRegionSize) +{ + struct uffdio_writeprotect wp; + + if (rand() % 2) { + wp.range.start = (unsigned long)lpBaseAddress; + wp.range.len = dwRegionSize; + wp.mode = UFFDIO_WRITEPROTECT_MODE_WP; + + if (ioctl(uffd, UFFDIO_WRITEPROTECT, &wp)) + ksft_exit_fail_msg("ioctl(UFFDIO_WRITEPROTECT)\n"); + } else { + if (pagemap_ioctl(lpBaseAddress, dwRegionSize, NULL, 0, PM_SCAN_OP_WP, + 0, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN) < 0) + ksft_exit_fail_msg("error %d %d %s\n", 1, errno, strerror(errno)); + } + + return 0; +} + +void *gethugetlb_mem(int size, int *shmid) +{ + char *mem; + + if (shmid) { + *shmid = shmget(2, size, SHM_HUGETLB | IPC_CREAT | SHM_R | SHM_W); + if (*shmid < 0) + return NULL; + + mem = shmat(*shmid, 0, 0); + if (mem == (char *)-1) { + shmctl(*shmid, IPC_RMID, NULL); + ksft_exit_fail_msg("Shared memory attach failure\n"); + } + } else { + mem = mmap(NULL, size, PROT_READ | PROT_WRITE, + MAP_ANONYMOUS | MAP_HUGETLB | MAP_PRIVATE, -1, 0); + if (mem == MAP_FAILED) + return NULL; + } + + return mem; +} + +int userfaultfd_tests(void) +{ + int mem_size, vec_size, written, num_pages = 16; + char *mem, *vec; + + mem_size = num_pages * page_size; + mem = mmap(NULL, mem_size, PROT_NONE, MAP_PRIVATE | MAP_ANON, -1, 0); + if (mem == MAP_FAILED) + ksft_exit_fail_msg("error nomem\n"); + + wp_init(mem, mem_size); + + /* Change protection of pages differently */ + mprotect(mem, mem_size/8, PROT_READ|PROT_WRITE); + mprotect(mem + 1 * mem_size/8, mem_size/8, PROT_READ); + mprotect(mem + 2 * mem_size/8, mem_size/8, PROT_READ|PROT_WRITE); + mprotect(mem + 3 * mem_size/8, mem_size/8, PROT_READ); + mprotect(mem + 4 * mem_size/8, mem_size/8, PROT_READ|PROT_WRITE); + mprotect(mem + 5 * mem_size/8, mem_size/8, PROT_NONE); + mprotect(mem + 6 * mem_size/8, mem_size/8, PROT_READ|PROT_WRITE); + mprotect(mem + 7 * mem_size/8, mem_size/8, PROT_READ); + + wp_addr_range(mem + (mem_size/16), mem_size - 2 * (mem_size/8)); + wp_addr_range(mem, mem_size); + + vec_size = mem_size/page_size; + vec = malloc(sizeof(struct page_region) * vec_size); + + written = pagemap_ioctl(mem, mem_size, vec, 1, PM_SCAN_OP_GET | PM_SCAN_OP_WP, + vec_size - 2, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (written < 0) + ksft_exit_fail_msg("error %d %d %s\n", written, errno, strerror(errno)); + + ksft_test_result(written == 0, "%s all new pages must not be written (dirty)\n", __func__); + + wp_free(mem, mem_size); + munmap(mem, mem_size); + free(vec); + return 0; +} + +int get_reads(struct page_region *vec, int vec_size) +{ + int i, sum = 0; + + for (i = 0; i < vec_size; i++) + sum += vec[i].len; + + return sum; +} + +int sanity_tests_sd(void) +{ + int mem_size, vec_size, ret, ret2, ret3, i, num_pages = 10, total_pages = 0; + int total_writes, total_reads, reads, count; + struct page_region *vec, *vec2; + char *mem, *m[2]; + + vec_size = 100; + mem_size = num_pages * page_size; + + vec = malloc(sizeof(struct page_region) * vec_size); + if (!vec) + ksft_exit_fail_msg("error nomem\n"); + + vec2 = malloc(sizeof(struct page_region) * vec_size); + if (!vec2) + ksft_exit_fail_msg("error nomem\n"); + + mem = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0); + if (mem == MAP_FAILED) + ksft_exit_fail_msg("error nomem\n"); + + wp_init(mem, mem_size); + wp_addr_range(mem, mem_size); + + /* 1. wrong operation */ + ksft_test_result(pagemap_ioctl(mem, 0, vec, vec_size, PM_SCAN_OP_GET, + 0, PAGEMAP_BITS_ALL, 0, 0, PAGEMAP_BITS_ALL) < 0, + "%s memory size must be valid\n", __func__); + + ksft_test_result(pagemap_ioctl(mem, mem_size, NULL, vec_size, PM_SCAN_OP_GET, + 0, PAGEMAP_BITS_ALL, 0, 0, PAGEMAP_BITS_ALL) < 0, + "%s output buffer must be specified\n", __func__); + + ksft_test_result(pagemap_ioctl(mem, mem_size, vec, 0, PM_SCAN_OP_GET, + 0, PAGEMAP_BITS_ALL, 0, 0, PAGEMAP_BITS_ALL) < 0, + "%s output buffer size must be valid\n", __func__); + + ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, -1, + 0, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN) < 0, + "%s wrong flag specified\n", __func__); + + ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, + PM_SCAN_OP_GET | PM_SCAN_OP_WP | 0xFF, + 0, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN) < 0, + "%s flag has extra bits specified\n", __func__); + + ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET, + 0, 0, 0, 0, PAGE_IS_WRITTEN) < 0, + "%s no selection mask is specified\n", __func__); + + ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET, + 0, PAGE_IS_WRITTEN, PAGE_IS_WRITTEN, 0, 0) < 0, + "%s no return mask is specified\n", __func__); + + ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET, + 0, PAGE_IS_WRITTEN, 0, 0, 0x1000) < 0, + "%s wrong return mask specified\n", __func__); + + ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET | PM_SCAN_OP_WP, + 0, 0xFFF, PAGE_IS_WRITTEN, 0, PAGE_IS_WRITTEN) < 0, + "%s mixture of correct and wrong flag\n", __func__); + + ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET | PM_SCAN_OP_WP, + 0, 0, 0, PAGEMAP_BITS_ALL, PAGE_IS_WRITTEN) >= 0, + "%s PAGEMAP_BITS_ALL can be specified with PM_SCAN_OP_WP\n", __func__); + + /* 2. Clear area with larger vec size */ + ret = pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET | PM_SCAN_OP_WP, 0, + PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + ksft_test_result(ret >= 0, "%s Clear area with larger vec size\n", __func__); + + /* 3. Repeated pattern of written and non-written pages */ + for (i = 0; i < mem_size; i += 2 * page_size) + mem[i]++; + + ret = pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET, 0, PAGE_IS_WRITTEN, 0, + 0, PAGE_IS_WRITTEN); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + + ksft_test_result(ret == mem_size/(page_size * 2), + "%s Repeated pattern of written and non-written pages\n", __func__); + + /* 4. Repeated pattern of written and non-written pages in parts */ + ret = pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET | PM_SCAN_OP_WP, + num_pages/2 - 2, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + + ret2 = pagemap_ioctl(mem, mem_size, vec, 2, PM_SCAN_OP_GET, 0, PAGE_IS_WRITTEN, 0, 0, + PAGE_IS_WRITTEN); + if (ret2 < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret2, errno, strerror(errno)); + + ret3 = pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET | PM_SCAN_OP_WP, + 0, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (ret3 < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret3, errno, strerror(errno)); + + ksft_test_result((ret + ret3) == num_pages/2 && ret2 == 2, + "%s Repeated pattern of written and non-written pages in parts\n", + __func__); + + /* 5. Repeated pattern of written and non-written pages max_pages */ + for (i = 0; i < mem_size; i += 2 * page_size) + mem[i]++; + mem[(mem_size/page_size - 1) * page_size]++; + + ret = pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET | PM_SCAN_OP_WP, + num_pages/2, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + + ret2 = pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET | PM_SCAN_OP_WP, + 0, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (ret2 < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret2, errno, strerror(errno)); + + ksft_test_result(ret == num_pages/2 && ret2 == 1, + "%s Repeated pattern of written and non-written pages max_pages\n", + __func__); + + /* 6. only get 2 dirty pages and clear them as well */ + vec_size = mem_size/page_size; + memset(mem, -1, mem_size); + + /* get and clear second and third pages */ + ret = pagemap_ioctl(mem + page_size, 2 * page_size, vec, 1, PM_SCAN_OP_GET | PM_SCAN_OP_WP, + 2, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + + ret2 = pagemap_ioctl(mem, mem_size, vec2, vec_size, PM_SCAN_OP_GET, 0, + PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (ret2 < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret2, errno, strerror(errno)); + + ksft_test_result(ret == 1 && vec[0].len == 2 && + vec[0].start == (uintptr_t)(mem + page_size) && + ret2 == 2 && vec2[0].len == 1 && vec2[0].start == (uintptr_t)mem && + vec2[1].len == vec_size - 3 && + vec2[1].start == (uintptr_t)(mem + 3 * page_size), + "%s only get 2 written pages and clear them as well\n", __func__); + + wp_free(mem, mem_size); + munmap(mem, mem_size); + + /* 7. Two regions */ + m[0] = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0); + if (m[0] == MAP_FAILED) + ksft_exit_fail_msg("error nomem\n"); + m[1] = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0); + if (m[1] == MAP_FAILED) + ksft_exit_fail_msg("error nomem\n"); + + wp_init(m[0], mem_size); + wp_init(m[1], mem_size); + wp_addr_range(m[0], mem_size); + wp_addr_range(m[1], mem_size); + + memset(m[0], 'a', mem_size); + memset(m[1], 'b', mem_size); + + wp_addr_range(m[0], mem_size); + + ret = pagemap_ioctl(m[1], mem_size, vec, 1, PM_SCAN_OP_GET, 0, PAGE_IS_WRITTEN, 0, 0, + PAGE_IS_WRITTEN); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + + ksft_test_result(ret == 1 && vec[0].len == mem_size/page_size, + "%s Two regions\n", __func__); + + wp_free(m[0], mem_size); + wp_free(m[1], mem_size); + munmap(m[0], mem_size); + munmap(m[1], mem_size); + + free(vec); + free(vec2); + + /* 8. Smaller vec */ + mem_size = 1050 * page_size; + vec_size = mem_size/(page_size*2); + + vec = malloc(sizeof(struct page_region) * vec_size); + if (!vec) + ksft_exit_fail_msg("error nomem\n"); + + mem = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0); + if (mem == MAP_FAILED) + ksft_exit_fail_msg("error nomem\n"); + + wp_init(mem, mem_size); + wp_addr_range(mem, mem_size); + + ret = pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET | PM_SCAN_OP_WP, 0, + PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + + for (i = 0; i < mem_size/page_size; i += 2) + mem[i * page_size]++; + + ret = pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET | PM_SCAN_OP_WP, + mem_size/(page_size*5), PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + + total_pages += ret; + + ret = pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET | PM_SCAN_OP_WP, + mem_size/(page_size*5), PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + + total_pages += ret; + + ret = pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET | PM_SCAN_OP_WP, + mem_size/(page_size*5), PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + + total_pages += ret; + + ksft_test_result(total_pages == mem_size/(page_size*2), "%s Smaller max_pages\n", __func__); + + free(vec); + wp_free(mem, mem_size); + munmap(mem, mem_size); + total_pages = 0; + + /* 9. Smaller vec */ + mem_size = 10000 * page_size; + vec_size = 50; + + vec = malloc(sizeof(struct page_region) * vec_size); + if (!vec) + ksft_exit_fail_msg("error nomem\n"); + + mem = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0); + if (mem == MAP_FAILED) + ksft_exit_fail_msg("error nomem\n"); + + wp_init(mem, mem_size); + wp_addr_range(mem, mem_size); + + for (count = 0; count < TEST_ITERATIONS; count++) { + total_writes = total_reads = 0; + + for (i = 0; i < mem_size; i += page_size) { + if (rand() % 2) { + mem[i]++; + total_writes++; + } + } + + while (total_reads < total_writes) { + ret = pagemap_ioctl(mem, mem_size, vec, vec_size, + PM_SCAN_OP_GET | PM_SCAN_OP_WP, + 0, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + + if (ret > vec_size) + break; + + reads = get_reads(vec, ret); + total_reads += reads; + } + + if (total_reads != total_writes) + break; + } + + ksft_test_result(count == TEST_ITERATIONS, "Smaller vec %d %d\n", ret, vec_size); + + free(vec); + wp_free(mem, mem_size); + munmap(mem, mem_size); + + return 0; +} + +int base_tests(char *prefix, char *mem, int mem_size, int skip) +{ + int vec_size, written; + struct page_region *vec, *vec2; + + if (skip) { + ksft_test_result_skip("%s all new pages must not be written (dirty)\n", prefix); + ksft_test_result_skip("%s all pages must be written (dirty)\n", prefix); + ksft_test_result_skip("%s all pages dirty other than first and the last one\n", + prefix); + ksft_test_result_skip("%s PM_SCAN_OP_WP\n", prefix); + ksft_test_result_skip("%s only middle page dirty\n", prefix); + ksft_test_result_skip("%s only two middle pages dirty\n", prefix); + return 0; + } + + vec_size = mem_size/page_size; + vec = malloc(sizeof(struct page_region) * vec_size); + vec2 = malloc(sizeof(struct page_region) * vec_size); + + /* 1. all new pages must be not be written (dirty) */ + written = pagemap_ioctl(mem, mem_size, vec, 1, PM_SCAN_OP_GET | PM_SCAN_OP_WP, vec_size - 2, + PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (written < 0) + ksft_exit_fail_msg("error %d %d %s\n", written, errno, strerror(errno)); + + ksft_test_result(written == 0, "%s all new pages must not be written (dirty)\n", prefix); + + /* 2. all pages must be written */ + memset(mem, -1, mem_size); + + written = pagemap_ioctl(mem, mem_size, vec, 1, PM_SCAN_OP_GET, 0, PAGE_IS_WRITTEN, 0, 0, + PAGE_IS_WRITTEN); + if (written < 0) + ksft_exit_fail_msg("error %d %d %s\n", written, errno, strerror(errno)); + + ksft_test_result(written == 1 && vec[0].len == mem_size/page_size, + "%s all pages must be written (dirty)\n", prefix); + + /* 3. all pages dirty other than first and the last one */ + written = pagemap_ioctl(mem, mem_size, vec, 1, PM_SCAN_OP_GET | PM_SCAN_OP_WP, 0, + PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (written < 0) + ksft_exit_fail_msg("error %d %d %s\n", written, errno, strerror(errno)); + + memset(mem + page_size, 0, mem_size - (2 * page_size)); + + written = pagemap_ioctl(mem, mem_size, vec, 1, PM_SCAN_OP_GET | PM_SCAN_OP_WP, 0, + PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (written < 0) + ksft_exit_fail_msg("error %d %d %s\n", written, errno, strerror(errno)); + + ksft_test_result(written == 1 && vec[0].len >= vec_size - 2 && vec[0].len <= vec_size, + "%s all pages dirty other than first and the last one\n", prefix); + + written = pagemap_ioctl(mem, mem_size, vec, 1, PM_SCAN_OP_GET, 0, + PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (written < 0) + ksft_exit_fail_msg("error %d %d %s\n", written, errno, strerror(errno)); + + ksft_test_result(written == 0, + "%s PM_SCAN_OP_WP\n", prefix); + + /* 4. only middle page dirty */ + written = pagemap_ioctl(mem, mem_size, vec, 1, PM_SCAN_OP_GET | PM_SCAN_OP_WP, 0, + PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (written < 0) + ksft_exit_fail_msg("error %d %d %s\n", written, errno, strerror(errno)); + + mem[vec_size/2 * page_size]++; + + written = pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET, 0, PAGE_IS_WRITTEN, + 0, 0, PAGE_IS_WRITTEN); + if (written < 0) + ksft_exit_fail_msg("error %d %d %s\n", written, errno, strerror(errno)); + + ksft_test_result(written == 1 && vec[0].len >= 1, + "%s only middle page dirty\n", prefix); + + /* 5. only two middle pages dirty and walk over only middle pages */ + written = pagemap_ioctl(mem, mem_size, vec, 1, PM_SCAN_OP_GET | PM_SCAN_OP_WP, 0, + PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (written < 0) + ksft_exit_fail_msg("error %d %d %s\n", written, errno, strerror(errno)); + + mem[vec_size/2 * page_size]++; + mem[(vec_size/2 + 1) * page_size]++; + + written = pagemap_ioctl(&mem[vec_size/2 * page_size], 2 * page_size, vec, 1, PM_SCAN_OP_GET, + 0, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (written < 0) + ksft_exit_fail_msg("error %d %d %s\n", written, errno, strerror(errno)); + + ksft_test_result(written == 1 && vec[0].start == (uintptr_t)(&mem[vec_size/2 * page_size]) + && vec[0].len == 2, + "%s only two middle pages dirty\n", prefix); + + free(vec); + free(vec2); + return 0; +} + +void *gethugepage(int map_size) +{ + int ret; + char *map; + + map = memalign(hpage_size, map_size); + if (!map) + ksft_exit_fail_msg("memalign failed %d %s\n", errno, strerror(errno)); + + ret = madvise(map, map_size, MADV_HUGEPAGE); + if (ret) + return NULL; + + memset(map, 0, map_size); + + return map; +} + +int hpage_unit_tests(void) +{ + char *map; + int ret, ret2; + size_t num_pages = 10; + int map_size = hpage_size * num_pages; + int vec_size = map_size/page_size; + struct page_region *vec, *vec2; + + vec = malloc(sizeof(struct page_region) * vec_size); + vec2 = malloc(sizeof(struct page_region) * vec_size); + if (!vec || !vec2) + ksft_exit_fail_msg("malloc failed\n"); + + map = gethugepage(map_size); + if (map) { + wp_init(map, map_size); + wp_addr_range(map, map_size); + + /* 1. all new huge page must not be written (dirty) */ + ret = pagemap_ioctl(map, map_size, vec, vec_size, PM_SCAN_OP_GET | PM_SCAN_OP_WP, 0, + PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + + ksft_test_result(ret == 0, "%s all new huge page must not be written (dirty)\n", + __func__); + + /* 2. all the huge page must not be written */ + ret = pagemap_ioctl(map, map_size, vec, vec_size, PM_SCAN_OP_GET, 0, + PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + + ksft_test_result(ret == 0, "%s all the huge page must not be written\n", __func__); + + /* 3. all the huge page must be written and clear dirty as well */ + memset(map, -1, map_size); + ret = pagemap_ioctl(map, map_size, vec, vec_size, PM_SCAN_OP_GET | PM_SCAN_OP_WP, + 0, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + + ksft_test_result(ret == 1 && vec[0].start == (uintptr_t)map && + vec[0].len == vec_size && vec[0].flags == PAGE_IS_WRITTEN, + "%s all the huge page must be written and clear\n", __func__); + + /* 4. only middle page written */ + wp_free(map, map_size); + free(map); + map = gethugepage(map_size); + wp_init(map, map_size); + wp_addr_range(map, map_size); + map[vec_size/2 * page_size]++; + + ret = pagemap_ioctl(map, map_size, vec, vec_size, PM_SCAN_OP_GET, 0, + PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + + ksft_test_result(ret == 1 && vec[0].len > 0, + "%s only middle page written\n", __func__); + + wp_free(map, map_size); + free(map); + } else { + ksft_test_result_skip("%s all new huge page must be written\n", __func__); + ksft_test_result_skip("%s all the huge page must not be written\n", __func__); + ksft_test_result_skip("%s all the huge page must be written and clear\n", __func__); + ksft_test_result_skip("%s only middle page written\n", __func__); + } + + /* 5. clear first half of huge page */ + map = gethugepage(map_size); + if (map) { + wp_init(map, map_size); + wp_addr_range(map, map_size); + + memset(map, 0, map_size); + + wp_addr_range(map, map_size/2); + + ret = pagemap_ioctl(map, map_size, vec, vec_size, PM_SCAN_OP_GET, 0, + PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + + ksft_test_result(ret == 1 && vec[0].len == vec_size/2 && + vec[0].start == (uintptr_t)(map + map_size/2), + "%s clear first half of huge page\n", __func__); + wp_free(map, map_size); + free(map); + } else { + ksft_test_result_skip("%s clear first half of huge page\n", __func__); + } + + /* 6. clear first half of huge page with limited buffer */ + map = gethugepage(map_size); + if (map) { + wp_init(map, map_size); + wp_addr_range(map, map_size); + + memset(map, 0, map_size); + + ret = pagemap_ioctl(map, map_size, vec, vec_size, PM_SCAN_OP_GET | PM_SCAN_OP_WP, + vec_size/2, PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + + ret = pagemap_ioctl(map, map_size, vec, vec_size, PM_SCAN_OP_GET, 0, + PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + + ksft_test_result(ret == 1 && vec[0].len == vec_size/2 && + vec[0].start == (uintptr_t)(map + map_size/2), + "%s clear first half of huge page with limited buffer\n", + __func__); + wp_free(map, map_size); + free(map); + } else { + ksft_test_result_skip("%s clear first half of huge page with limited buffer\n", + __func__); + } + + /* 7. clear second half of huge page */ + map = gethugepage(map_size); + if (map) { + wp_init(map, map_size); + wp_addr_range(map, map_size); + + memset(map, -1, map_size); + + ret = pagemap_ioctl(map + map_size/2, map_size/2, vec, vec_size, + PM_SCAN_OP_GET | PM_SCAN_OP_WP, vec_size/2, PAGE_IS_WRITTEN, 0, + 0, PAGE_IS_WRITTEN); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + + ret = pagemap_ioctl(map, map_size, vec, vec_size, PM_SCAN_OP_GET, 0, + PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + + ksft_test_result(ret == 1 && vec[0].len == vec_size/2, + "%s clear second half huge page\n", __func__); + wp_free(map, map_size); + free(map); + } else { + ksft_test_result_skip("%s clear second half huge page\n", __func__); + } + + /* 8. get half huge page */ + map = gethugepage(map_size); + if (map) { + wp_init(map, map_size); + wp_addr_range(map, map_size); + + memset(map, -1, map_size); + usleep(100); + + ret = pagemap_ioctl(map, map_size, vec, 1, PM_SCAN_OP_GET | PM_SCAN_OP_WP, + hpage_size/(2*page_size), PAGE_IS_WRITTEN, 0, 0, + PAGE_IS_WRITTEN); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + + ksft_test_result(ret == 1 && vec[0].len == hpage_size/(2*page_size), + "%s get half huge page\n", __func__); + + ret2 = pagemap_ioctl(map, map_size, vec, vec_size, PM_SCAN_OP_GET, 0, + PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN); + if (ret2 < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret2, errno, strerror(errno)); + + ksft_test_result(ret2 == 1 && vec[0].len == (map_size - hpage_size/2)/page_size, + "%s get half huge page\n", __func__); + + wp_free(map, map_size); + free(map); + } else { + ksft_test_result_skip("%s get half huge page\n", __func__); + ksft_test_result_skip("%s get half huge page\n", __func__); + } + + free(vec); + free(vec2); + return 0; +} + +int unmapped_region_tests(void) +{ + void *start = (void *)0x10000000; + int written, len = 0x00040000; + int vec_size = len / page_size; + struct page_region *vec = malloc(sizeof(struct page_region) * vec_size); + + /* 1. Get written pages */ + written = pagemap_ioctl(start, len, vec, vec_size, PM_SCAN_OP_GET, 0, + PAGEMAP_NON_WRITTEN_BITS, 0, 0, PAGEMAP_NON_WRITTEN_BITS); + if (written < 0) + ksft_exit_fail_msg("error %d %d %s\n", written, errno, strerror(errno)); + + ksft_test_result(written >= 0, "%s Get status of pages\n", __func__); + + free(vec); + return 0; +} + +static void test_simple(void) +{ + int i; + char *map; + struct page_region vec; + + map = aligned_alloc(page_size, page_size); + if (!map) + ksft_exit_fail_msg("aligned_alloc failed\n"); + + wp_init(map, page_size); + wp_addr_range(map, page_size); + + for (i = 0 ; i < TEST_ITERATIONS; i++) { + if (pagemap_ioctl(map, page_size, &vec, 1, PM_SCAN_OP_GET, 0, + PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN) == 1) { + ksft_print_msg("written bit was 1, but should be 0 (i=%d)\n", i); + break; + } + + wp_addr_range(map, page_size); + /* Write something to the page to get the written bit enabled on the page */ + map[0]++; + + if (pagemap_ioctl(map, page_size, &vec, 1, PM_SCAN_OP_GET, 0, + PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN) == 0) { + ksft_print_msg("written bit was 0, but should be 1 (i=%d)\n", i); + break; + } + + wp_addr_range(map, page_size); + } + wp_free(map, page_size); + free(map); + + ksft_test_result(i == TEST_ITERATIONS, "Test %s\n", __func__); +} + +int sanity_tests(void) +{ + int mem_size, vec_size, ret, fd, i, buf_size; + struct page_region *vec; + char *mem, *fmem; + struct stat sbuf; + char *tmp_buf; + + /* 1. wrong operation */ + mem_size = 10 * page_size; + vec_size = mem_size / page_size; + + vec = malloc(sizeof(struct page_region) * vec_size); + mem = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0); + if (mem == MAP_FAILED || vec == MAP_FAILED) + ksft_exit_fail_msg("error nomem\n"); + + wp_init(mem, mem_size); + wp_addr_range(mem, mem_size); + + ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET | PM_SCAN_OP_WP, + 0, PAGEMAP_BITS_ALL, 0, 0, PAGEMAP_BITS_ALL) >= 0, + "%s WP op can be specified with !PAGE_IS_WRITTEN\n", __func__); + ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET, 0, + PAGEMAP_BITS_ALL, 0, 0, PAGEMAP_BITS_ALL) >= 0, + "%s required_mask specified\n", __func__); + ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET, 0, + 0, PAGEMAP_BITS_ALL, 0, PAGEMAP_BITS_ALL) >= 0, + "%s anyof_mask specified\n", __func__); + ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET, 0, + 0, 0, PAGEMAP_BITS_ALL, PAGEMAP_BITS_ALL) >= 0, + "%s excluded_mask specified\n", __func__); + ksft_test_result(pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET, 0, + PAGEMAP_BITS_ALL, PAGEMAP_BITS_ALL, 0, + PAGEMAP_BITS_ALL) >= 0, + "%s required_mask and anyof_mask specified\n", __func__); + wp_free(mem, mem_size); + munmap(mem, mem_size); + + /* 2. Get sd and present pages with anyof_mask */ + mem = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0); + if (mem == MAP_FAILED) + ksft_exit_fail_msg("error nomem\n"); + wp_init(mem, mem_size); + wp_addr_range(mem, mem_size); + + memset(mem, 0, mem_size); + + ret = pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET, 0, + 0, PAGEMAP_BITS_ALL, 0, PAGEMAP_BITS_ALL); + ksft_test_result(ret >= 0 && vec[0].start == (uintptr_t)mem && vec[0].len == vec_size && + vec[0].flags == (PAGE_IS_WRITTEN | PAGE_IS_PRESENT), + "%s Get sd and present pages with anyof_mask\n", __func__); + + /* 3. Get sd and present pages with required_mask */ + ret = pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET, 0, + PAGEMAP_BITS_ALL, 0, 0, PAGEMAP_BITS_ALL); + ksft_test_result(ret >= 0 && vec[0].start == (uintptr_t)mem && vec[0].len == vec_size && + vec[0].flags == (PAGE_IS_WRITTEN | PAGE_IS_PRESENT), + "%s Get all the pages with required_mask\n", __func__); + + /* 4. Get sd and present pages with required_mask and anyof_mask */ + ret = pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET, 0, + PAGE_IS_WRITTEN, PAGE_IS_PRESENT, 0, PAGEMAP_BITS_ALL); + ksft_test_result(ret >= 0 && vec[0].start == (uintptr_t)mem && vec[0].len == vec_size && + vec[0].flags == (PAGE_IS_WRITTEN | PAGE_IS_PRESENT), + "%s Get sd and present pages with required_mask and anyof_mask\n", + __func__); + + /* 5. Don't get sd pages */ + ret = pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET, 0, + 0, 0, PAGE_IS_WRITTEN, PAGEMAP_BITS_ALL); + ksft_test_result(ret == 0, "%s Don't get sd pages\n", __func__); + + /* 6. Don't get present pages */ + ret = pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET, 0, + 0, 0, PAGE_IS_PRESENT, PAGEMAP_BITS_ALL); + ksft_test_result(ret == 0, "%s Don't get present pages\n", __func__); + + wp_free(mem, mem_size); + munmap(mem, mem_size); + + /* 8. Find written present pages with return mask */ + mem = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0); + if (mem == MAP_FAILED) + ksft_exit_fail_msg("error nomem\n"); + wp_init(mem, mem_size); + wp_addr_range(mem, mem_size); + + memset(mem, 0, mem_size); + + ret = pagemap_ioctl(mem, mem_size, vec, vec_size, PM_SCAN_OP_GET | PM_SCAN_OP_WP, 0, + 0, PAGEMAP_BITS_ALL, 0, PAGE_IS_WRITTEN); + ksft_test_result(ret >= 0 && vec[0].start == (uintptr_t)mem && vec[0].len == vec_size && + vec[0].flags == PAGE_IS_WRITTEN, + "%s Find written present pages with return mask\n", __func__); + wp_free(mem, mem_size); + munmap(mem, mem_size); + + /* 9. Memory mapped file */ + fd = open(__FILE__, O_RDONLY); + if (fd < 0) + ksft_exit_fail_msg("%s Memory mapped file\n"); + + ret = stat(__FILE__, &sbuf); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + + fmem = mmap(NULL, sbuf.st_size, PROT_READ, MAP_PRIVATE, fd, 0); + if (fmem == MAP_FAILED) + ksft_exit_fail_msg("error nomem %ld %s\n", errno, strerror(errno)); + + tmp_buf = malloc(sbuf.st_size); + memcpy(tmp_buf, fmem, sbuf.st_size); + + ret = pagemap_ioctl(fmem, sbuf.st_size, vec, vec_size, PM_SCAN_OP_GET, 0, + 0, PAGEMAP_NON_WRITTEN_BITS, 0, PAGEMAP_NON_WRITTEN_BITS); + + ksft_test_result(ret >= 0 && vec[0].start == (uintptr_t)fmem && + vec[0].len == ceilf((float)sbuf.st_size/page_size) && + (vec[0].flags & PAGE_IS_FILE), + "%s Memory mapped file\n", __func__); + + munmap(fmem, sbuf.st_size); + close(fd); + + /* 10. Create and read/write to a memory mapped file */ + buf_size = page_size * 10; + + fd = open(__FILE__".tmp2", O_RDWR | O_CREAT, 0666); + if (fd < 0) + ksft_exit_fail_msg("Read/write to memory: %s\n", + strerror(errno)); + + for (i = 0; i < buf_size; i++) + if (write(fd, "c", 1) < 0) + ksft_exit_fail_msg("Create and read/write to a memory mapped file\n"); + + fmem = mmap(NULL, buf_size, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0); + if (fmem == MAP_FAILED) + ksft_exit_fail_msg("error nomem %ld %s\n", errno, strerror(errno)); + + wp_init(fmem, buf_size); + wp_addr_range(fmem, buf_size); + + for (i = 0; i < buf_size; i++) + fmem[i] = 'z'; + + msync(fmem, buf_size, MS_SYNC); + + ret = pagemap_ioctl(fmem, buf_size, vec, vec_size, PM_SCAN_OP_GET, 0, + PAGE_IS_WRITTEN, PAGE_IS_PRESENT | PAGE_IS_SWAPPED | PAGE_IS_FILE, 0, + PAGEMAP_BITS_ALL); + + ksft_test_result(ret >= 0 && vec[0].start == (uintptr_t)fmem && + vec[0].len == (buf_size/page_size) && + (vec[0].flags & PAGE_IS_WRITTEN), + "%s Read/write to memory\n", __func__); + + wp_free(fmem, buf_size); + munmap(fmem, buf_size); + close(fd); + + free(vec); + return 0; +} + +int mprotect_tests(void) +{ + int ret; + char *mem, *mem2; + struct page_region vec; + int pagemap_fd = open("/proc/self/pagemap", O_RDONLY); + + if (pagemap_fd < 0) { + fprintf(stderr, "open() failed\n"); + exit(1); + } + + /* 1. Map two pages */ + mem = mmap(0, 2 * page_size, PROT_READ|PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0); + if (mem == MAP_FAILED) + ksft_exit_fail_msg("error nomem\n"); + wp_init(mem, 2 * page_size); + wp_addr_range(mem, 2 * page_size); + + /* Populate both pages. */ + memset(mem, 1, 2 * page_size); + + ret = pagemap_ioctl(mem, 2 * page_size, &vec, 1, PM_SCAN_OP_GET, 0, PAGE_IS_WRITTEN, + 0, 0, PAGE_IS_WRITTEN); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + + ksft_test_result(ret == 1 && vec.len == 2, "%s Both pages written\n", __func__); + + /* 2. Start tracking */ + wp_addr_range(mem, 2 * page_size); + + ksft_test_result(pagemap_ioctl(mem, 2 * page_size, &vec, 1, PM_SCAN_OP_GET, 0, + PAGE_IS_WRITTEN, 0, 0, PAGE_IS_WRITTEN) == 0, + "%s Both pages are not written (dirty)\n", __func__); + + /* 3. Remap the second page */ + mem2 = mmap(mem + page_size, page_size, PROT_READ|PROT_WRITE, + MAP_PRIVATE|MAP_ANON|MAP_FIXED, -1, 0); + if (mem2 == MAP_FAILED) + ksft_exit_fail_msg("error nomem\n"); + wp_init(mem2, page_size); + wp_addr_range(mem2, page_size); + + /* Protect + unprotect. */ + mprotect(mem, page_size, PROT_NONE); + mprotect(mem, 2 * page_size, PROT_READ); + mprotect(mem, 2 * page_size, PROT_READ|PROT_WRITE); + + /* Modify both pages. */ + memset(mem, 2, 2 * page_size); + + /* Protect + unprotect. */ + mprotect(mem, page_size, PROT_NONE); + mprotect(mem, page_size, PROT_READ); + mprotect(mem, page_size, PROT_READ|PROT_WRITE); + + ret = pagemap_ioctl(mem, 2 * page_size, &vec, 1, PM_SCAN_OP_GET, 0, PAGE_IS_WRITTEN, + 0, 0, PAGE_IS_WRITTEN); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + + ksft_test_result(ret == 1 && vec.len == 2, + "%s Both pages written after remap and mprotect\n", __func__); + + /* 4. Clear and make the pages written */ + wp_addr_range(mem, 2 * page_size); + + memset(mem, 'A', 2 * page_size); + + ret = pagemap_ioctl(mem, 2 * page_size, &vec, 1, PM_SCAN_OP_GET, 0, PAGE_IS_WRITTEN, + 0, 0, PAGE_IS_WRITTEN); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + + ksft_test_result(ret == 1 && vec.len == 2, + "%s Clear and make the pages written\n", __func__); + + wp_free(mem, 2 * page_size); + munmap(mem, 2 * page_size); + return 0; +} + +/* transact test */ +static const unsigned int nthreads = 6, pages_per_thread = 32, access_per_thread = 8; +static pthread_barrier_t start_barrier, end_barrier; +static unsigned int extra_thread_faults; +static unsigned int iter_count = 1000; +static volatile int finish; + +static ssize_t get_dirty_pages_reset(char *mem, unsigned int count, + int reset, int page_size) +{ + struct pm_scan_arg arg = {0}; + struct page_region rgns[256]; + int i, j, cnt, ret; + + arg.size = sizeof(struct pm_scan_arg); + arg.start = (uintptr_t)mem; + arg.max_pages = count; + arg.end = (uintptr_t)(mem + count * page_size); + arg.vec = (uintptr_t)rgns; + arg.vec_len = sizeof(rgns) / sizeof(*rgns); + arg.flags = PM_SCAN_OP_GET; + if (reset) + arg.flags |= PM_SCAN_OP_WP; + arg.required_mask = PAGE_IS_WRITTEN; + arg.return_mask = PAGE_IS_WRITTEN; + + ret = ioctl(pagemap_fd, PAGEMAP_SCAN, &arg); + if (ret < 0) + ksft_exit_fail_msg("ioctl failed\n"); + + cnt = 0; + for (i = 0; i < ret; ++i) { + if (rgns[i].flags != PAGE_IS_WRITTEN) + ksft_exit_fail_msg("wrong flags\n"); + + for (j = 0; j < rgns[i].len; ++j) + cnt++; + } + + return cnt; +} + +void *thread_proc(void *mem) +{ + int *m = mem; + long curr_faults, faults; + struct rusage r; + unsigned int i; + int ret; + + if (getrusage(RUSAGE_THREAD, &r)) + ksft_exit_fail_msg("getrusage\n"); + + curr_faults = r.ru_minflt; + + while (!finish) { + ret = pthread_barrier_wait(&start_barrier); + if (ret && ret != PTHREAD_BARRIER_SERIAL_THREAD) + ksft_exit_fail_msg("pthread_barrier_wait\n"); + + for (i = 0; i < access_per_thread; ++i) + __atomic_add_fetch(m + i * (0x1000 / sizeof(*m)), 1, __ATOMIC_SEQ_CST); + + ret = pthread_barrier_wait(&end_barrier); + if (ret && ret != PTHREAD_BARRIER_SERIAL_THREAD) + ksft_exit_fail_msg("pthread_barrier_wait\n"); + + if (getrusage(RUSAGE_THREAD, &r)) + ksft_exit_fail_msg("getrusage\n"); + + faults = r.ru_minflt - curr_faults; + if (faults < access_per_thread) + ksft_exit_fail_msg("faults < access_per_thread"); + + __atomic_add_fetch(&extra_thread_faults, faults - access_per_thread, + __ATOMIC_SEQ_CST); + curr_faults = r.ru_minflt; + } + + return NULL; +} + +static void transact_test(int page_size) +{ + unsigned int i, count, extra_pages; + pthread_t th; + char *mem; + int ret, c; + + if (pthread_barrier_init(&start_barrier, NULL, nthreads + 1)) + ksft_exit_fail_msg("pthread_barrier_init\n"); + + if (pthread_barrier_init(&end_barrier, NULL, nthreads + 1)) + ksft_exit_fail_msg("pthread_barrier_init\n"); + + mem = mmap(NULL, 0x1000 * nthreads * pages_per_thread, PROT_READ | PROT_WRITE, + MAP_ANONYMOUS | MAP_PRIVATE, -1, 0); + if (mem == MAP_FAILED) + ksft_exit_fail_msg("Error mmap %s.\n", strerror(errno)); + + wp_init(mem, 0x1000 * nthreads * pages_per_thread); + wp_addr_range(mem, 0x1000 * nthreads * pages_per_thread); + + memset(mem, 0, 0x1000 * nthreads * pages_per_thread); + + count = get_dirty_pages_reset(mem, nthreads * pages_per_thread, 1, page_size); + ksft_test_result(count > 0, "%s count %d\n", __func__, count); + count = get_dirty_pages_reset(mem, nthreads * pages_per_thread, 1, page_size); + ksft_test_result(count == 0, "%s count %d\n", __func__, count); + + finish = 0; + for (i = 0; i < nthreads; ++i) + pthread_create(&th, NULL, thread_proc, mem + 0x1000 * i * pages_per_thread); + + extra_pages = 0; + for (i = 0; i < iter_count; ++i) { + count = 0; + + ret = pthread_barrier_wait(&start_barrier); + if (ret && ret != PTHREAD_BARRIER_SERIAL_THREAD) + ksft_exit_fail_msg("pthread_barrier_wait\n"); + + count = get_dirty_pages_reset(mem, nthreads * pages_per_thread, 1, + page_size); + + ret = pthread_barrier_wait(&end_barrier); + if (ret && ret != PTHREAD_BARRIER_SERIAL_THREAD) + ksft_exit_fail_msg("pthread_barrier_wait\n"); + + if (count > nthreads * access_per_thread) + ksft_exit_fail_msg("Too big count %d expected %d, iter %d\n", + count, nthreads * access_per_thread, i); + + c = get_dirty_pages_reset(mem, nthreads * pages_per_thread, 1, page_size); + count += c; + + if (c > nthreads * access_per_thread) { + ksft_test_result_fail(" %s count > nthreads\n", __func__); + return; + } + + if (count != nthreads * access_per_thread) { + /* + * The purpose of the test is to make sure that no page updates are lost + * when the page updates and read-resetting soft dirty flags are performed + * in parallel. However, it is possible that the application will get the + * soft dirty flags twice on the two consecutive read-resets. This seems + * unavoidable as soft dirty flag is handled in software through page faults + * in kernel. While the updating the flags is supposed to be synchronized + * between page fault handling and read-reset, it is possible that + * read-reset happens after page fault PTE update but before the application + * re-executes write instruction. So read-reset gets the flag, clears write + * access and application gets page fault again for the same write. + */ + if (count < nthreads * access_per_thread) { + ksft_test_result_fail("Lost update, iter %d, %d vs %d.\n", i, count, + nthreads * access_per_thread); + return; + } + + extra_pages += count - nthreads * access_per_thread; + } + } + + pthread_barrier_wait(&start_barrier); + finish = 1; + pthread_barrier_wait(&end_barrier); + + ksft_test_result_pass("%s Extra pages %u (%.1lf%%), extra thread faults %d.\n", __func__, + extra_pages, + 100.0 * extra_pages / (iter_count * nthreads * access_per_thread), + extra_thread_faults); +} + +int main(void) +{ + int mem_size, shmid, buf_size, fd, i, ret; + char *mem, *map, *fmem; + struct stat sbuf; + + ksft_print_header(); + ksft_set_plan(92); + + page_size = getpagesize(); + hpage_size = read_pmd_pagesize(); + + pagemap_fd = open(PAGEMAP, O_RDWR); + if (pagemap_fd < 0) + return -EINVAL; + + if (init_uffd()) + ksft_exit_fail_msg("uffd init failed\n"); + + /* + * Written (dirty) PTE bit tests + */ + + /* 1. Sanity testing */ + sanity_tests_sd(); + + /* 2. Normal page testing */ + mem_size = 10 * page_size; + mem = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0); + if (mem == MAP_FAILED) + ksft_exit_fail_msg("error nomem\n"); + wp_init(mem, mem_size); + wp_addr_range(mem, mem_size); + + base_tests("Page testing:", mem, mem_size, 0); + + wp_free(mem, mem_size); + munmap(mem, mem_size); + + /* 3. Large page testing */ + mem_size = 512 * 10 * page_size; + mem = mmap(NULL, mem_size, PROT_READ | PROT_WRITE, MAP_PRIVATE | MAP_ANON, -1, 0); + if (mem == MAP_FAILED) + ksft_exit_fail_msg("error nomem\n"); + wp_init(mem, mem_size); + wp_addr_range(mem, mem_size); + + base_tests("Large Page testing:", mem, mem_size, 0); + + wp_free(mem, mem_size); + munmap(mem, mem_size); + + /* 4. Huge page testing */ + map = gethugepage(hpage_size); + if (map) { + wp_init(map, hpage_size); + wp_addr_range(map, hpage_size); + base_tests("Huge page testing:", map, hpage_size, 0); + wp_free(map, hpage_size); + free(map); + } else { + base_tests("Huge page testing:", NULL, 0, 1); + } + + /* 5. Hugetlb page testing */ + mem_size = 2*1024*1024; + mem = gethugetlb_mem(mem_size, &shmid); + if (mem) { + wp_init(mem, mem_size); + wp_addr_range(mem, mem_size); + + base_tests("Hugetlb shmem testing:", mem, mem_size, 0); + + wp_free(mem, mem_size); + shmctl(shmid, IPC_RMID, NULL); + } else { + base_tests("Hugetlb shmem testing:", NULL, 0, 1); + } + + /* 6. Hugetlb page testing */ + mem = gethugetlb_mem(mem_size, NULL); + if (mem) { + wp_init(mem, mem_size); + wp_addr_range(mem, mem_size); + + base_tests("Hugetlb mem testing:", mem, mem_size, 0); + + wp_free(mem, mem_size); + } else { + base_tests("Hugetlb mem testing:", NULL, 0, 1); + } + + /* 7. file memory testing */ + buf_size = page_size * 10; + + fd = open(__FILE__".tmp0", O_RDWR | O_CREAT, 0777); + if (fd < 0) + ksft_exit_fail_msg("Create and read/write to a memory mapped file: %s\n", + strerror(errno)); + + for (i = 0; i < buf_size; i++) + if (write(fd, "c", 1) < 0) + ksft_exit_fail_msg("Create and read/write to a memory mapped file\n"); + + ret = stat(__FILE__".tmp0", &sbuf); + if (ret < 0) + ksft_exit_fail_msg("error %d %d %s\n", ret, errno, strerror(errno)); + + fmem = mmap(NULL, sbuf.st_size, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0); + if (fmem == MAP_FAILED) + ksft_exit_fail_msg("error nomem %ld %s\n", errno, strerror(errno)); + + wp_init(fmem, sbuf.st_size); + wp_addr_range(fmem, sbuf.st_size); + + base_tests("File memory testing:", fmem, sbuf.st_size, 0); + + wp_free(fmem, sbuf.st_size); + munmap(fmem, sbuf.st_size); + close(fd); + + /* 8. file memory testing */ + buf_size = page_size * 10; + + fd = memfd_create(__FILE__".tmp00", MFD_NOEXEC_SEAL); + if (fd < 0) + ksft_exit_fail_msg("Create and read/write to a memory mapped file: %s\n", + strerror(errno)); + + if (ftruncate(fd, buf_size)) + ksft_exit_fail_msg("Error ftruncate\n"); + + for (i = 0; i < buf_size; i++) + if (write(fd, "c", 1) < 0) + ksft_exit_fail_msg("Create and read/write to a memory mapped file\n"); + + fmem = mmap(NULL, buf_size, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0); + if (fmem == MAP_FAILED) + ksft_exit_fail_msg("error nomem %ld %s\n", errno, strerror(errno)); + + wp_init(fmem, buf_size); + wp_addr_range(fmem, buf_size); + + base_tests("File anonymous memory testing:", fmem, buf_size, 0); + + wp_free(fmem, buf_size); + munmap(fmem, buf_size); + close(fd); + + /* 9. Huge page tests */ + hpage_unit_tests(); + + /* 10. Iterative test */ + test_simple(); + + /* 11. Mprotect test */ + mprotect_tests(); + + /* 12. Transact test */ + transact_test(page_size); + + /* + * Other PTE bit tests + */ + + /* 1. Sanity testing */ + sanity_tests(); + + /* 2. Unmapped address test */ + unmapped_region_tests(); + + /* 3. Userfaultfd tests */ + userfaultfd_tests(); + + close(pagemap_fd); + return ksft_exit_pass(); +} diff --git a/tools/testing/selftests/mm/run_vmtests.sh b/tools/testing/selftests/mm/run_vmtests.sh old mode 100644 new mode 100755 index 9666c0c171ab..cca4dd8ee45d --- a/tools/testing/selftests/mm/run_vmtests.sh +++ b/tools/testing/selftests/mm/run_vmtests.sh @@ -56,6 +56,8 @@ separated by spaces: memory protection key tests - soft_dirty test soft dirty page bit semantics +- pagemap + test pagemap_scan IOCTL - cow test copy-on-write semantics example: ./run_vmtests.sh -t "hmm mmap ksm" @@ -324,6 +326,8 @@ fi CATEGORY="soft_dirty" run_test ./soft-dirty +CATEGORY="pagemap" run_test ./pagemap_ioctl + # COW tests CATEGORY="cow" run_test ./cow