From patchwork Thu Jun 3 18:32:16 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Axel Rasmussen X-Patchwork-Id: 12297837 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-26.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8CE0FC47082 for ; Thu, 3 Jun 2021 18:32:44 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 16367613B1 for ; Thu, 3 Jun 2021 18:32:44 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 16367613B1 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id A33186B0036; Thu, 3 Jun 2021 14:32:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A17C86B006C; Thu, 3 Jun 2021 14:32:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8AB1E6B006E; Thu, 3 Jun 2021 14:32:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0108.hostedemail.com [216.40.44.108]) by kanga.kvack.org (Postfix) with ESMTP id 58EA26B0036 for ; Thu, 3 Jun 2021 14:32:43 -0400 (EDT) Received: from smtpin17.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id DC27082499A8 for ; Thu, 3 Jun 2021 18:32:42 +0000 (UTC) X-FDA: 78213258564.17.EF79485 Received: from mail-qk1-f202.google.com (mail-qk1-f202.google.com [209.85.222.202]) by imf16.hostedemail.com (Postfix) with ESMTP id D5007801971D for ; Thu, 3 Jun 2021 18:32:17 +0000 (UTC) Received: by mail-qk1-f202.google.com with SMTP id j14-20020a05620a146eb02903a928cc3769so4931114qkl.7 for ; Thu, 03 Jun 2021 11:32:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:message-id:mime-version:subject:from:to:cc; bh=4PQMCxY9VZoCPKrMMqixmHLUOsEAJoUmXBoh17GhgWU=; b=OqTeCDLtqJYjjc6YAHAcR5yxO7/yVoA590JOswvab1ZYr+9Bh7V1e5y79CVCeO9kdp FLEUFiE9VvGRhP89sFzHqQMrdi09dtw56Gqk45S6RCMP4otxsOhIyHXGIhZb0LDFBz80 0XB6kaFoqHfaDbiFKsqpG3YXtV748UD28rdQOuRGeL/4v/qv8J2nJ6RFbp8zqu1w07Iz yeiVT8ULBcM3b4QbHdX7A15zXZTzfSC31luuz4qqJeQSE3dCqO2/6/1l9Q4DFeI+gOKK lWN1rPqEG8IOdBLAVzrdz/nHLjt4HCnyLWnXP3/3QMMtKjMbByGKG6ySlI1XejrxET0w wfXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:message-id:mime-version:subject:from:to:cc; bh=4PQMCxY9VZoCPKrMMqixmHLUOsEAJoUmXBoh17GhgWU=; b=PDd6agb8b8sopT0dAcKmAsjHhhUcOjFZqhZFRtQp3gvLJk0HLQ17BW9lRwSiiNpBbF Cfr4FR3W2fDdouOOLp5Lp+OT9DYHUdr+9XRxFtP3O6XKc43CyARYT5eCOxlVVLWM9vh9 Z+qDDt987sp0K09PDZr0FURDKY1MIskBztk6UZuK/3+v+S8OJGY0Nxnd3ltB6S4yrvOh EHp69nAI3ICd45XmK3SUCs+q82XSlknrsfb70dfvHMk9CQi+/fudEd8L/xRFaEQCCRc9 3ZoK3ZAbpmwrYko8MBQhz+Yid1Q1orM3LHHDtjy4M7IcxFYqs2DLBA2H9x3NkzC2hkgr wwzQ== X-Gm-Message-State: AOAM532rm+jw3bWrI67kDtx9yQCxpwN/Qx38WncDULbUcEhPn8atoI+F xNAqff8U9lcC3OG6XmD83fuQeEfwU7BdnRHZTTLm X-Google-Smtp-Source: ABdhPJyBQtTGgrMLS1ij4j6klC5VYHluJ2fynmgZZdZ/anwt+BswzR1xO+t92yBCzcm32gwgcLJaKphBSdSoAJTDMqG+ X-Received: from ajr0.svl.corp.google.com ([2620:15c:2cd:203:79cd:a91c:f1ad:1ead]) (user=axelrasmussen job=sendgmr) by 2002:a05:6214:2467:: with SMTP id im7mr849265qvb.59.1622745140406; Thu, 03 Jun 2021 11:32:20 -0700 (PDT) Date: Thu, 3 Jun 2021 11:32:16 -0700 Message-Id: <20210603183216.939169-1-axelrasmussen@google.com> Mime-Version: 1.0 X-Mailer: git-send-email 2.32.0.rc1.229.g3e70b5a671-goog Subject: [PATCH] ioctl_userfaultfd.2, userfaultfd.2: add minor fault mode From: Axel Rasmussen To: Andrea Arcangeli , Andrew Morton , Hugh Dickins , Mike Kravetz , Peter Xu Cc: linux-kernel@vger.kernel.org, linux-man@vger.kernel.org, linux-mm@kvack.org, Axel Rasmussen X-Rspamd-Queue-Id: D5007801971D Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20161025 header.b=OqTeCDLt; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf16.hostedemail.com: domain of 3NCC5YA0KCGUDaHOUDVPXVVHQJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--axelrasmussen.bounces.google.com designates 209.85.222.202 as permitted sender) smtp.mailfrom=3NCC5YA0KCGUDaHOUDVPXVVHQJRRJOH.FRPOLQXa-PPNYDFN.RUJ@flex--axelrasmussen.bounces.google.com X-Rspamd-Server: rspam03 X-Stat-Signature: f56o8tap6xn51medt4asdtywmy89egqn X-HE-Tag: 1622745137-695239 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Userfaultfd minor fault mode is supported starting from Linux 5.13. This commit adds a description of the new mode, as well as the new ioctl used to resolve such faults. The two go hand-in-hand: one can't resolve a minor fault without continue, and continue can't be used to resolve any other kind of fault. This patch covers just the hugetlbfs implementation (in 5.13). Support for shmem is forthcoming, but as it has not yet made it into a kernel release candidate, it will be added in a future commit. Signed-off-by: Axel Rasmussen --- man2/ioctl_userfaultfd.2 | 129 ++++++++++++++++++++++++++++++++++++--- man2/userfaultfd.2 | 79 +++++++++++++++++++----- 2 files changed, 186 insertions(+), 22 deletions(-) diff --git a/man2/ioctl_userfaultfd.2 b/man2/ioctl_userfaultfd.2 index 504f61d4b..9d4fc3a40 100644 --- a/man2/ioctl_userfaultfd.2 +++ b/man2/ioctl_userfaultfd.2 @@ -214,6 +214,10 @@ memory accesses to the regions registered with userfaultfd. If this feature bit is set, .I uffd_msg.pagefault.feat.ptid will be set to the faulted thread ID for each page-fault message. +.TP +.BR UFFD_FEATURE_MINOR_HUGETLBFS " (since Linux 5.13)" +If this feature bit is set, the kernel supports registering userfaultfd ranges +in minor mode on hugetlbfs-backed memory areas. .PP The returned .I ioctls @@ -240,6 +244,11 @@ operation is supported. The .B UFFDIO_WRITEPROTECT operation is supported. +.TP +.B 1 << _UFFDIO_CONTINUE +The +.B UFFDIO_CONTINUE +operation is supported. .PP This .BR ioctl (2) @@ -278,14 +287,8 @@ by the current kernel version. (Since Linux 4.3.) Register a memory address range with the userfaultfd object. The pages in the range must be "compatible". -.PP -Up to Linux kernel 4.11, -only private anonymous ranges are compatible for registering with -.BR UFFDIO_REGISTER . -.PP -Since Linux 4.11, -hugetlbfs and shared memory ranges are also compatible with -.BR UFFDIO_REGISTER . +What constitutes "compatible" depends on the mode(s) being used, as described +below. .PP The .I argp @@ -324,9 +327,16 @@ the specified range: .TP .B UFFDIO_REGISTER_MODE_MISSING Track page faults on missing pages. +Since Linux 4.3, only private anonymous ranges are compatible. +Since Linux 4.11, hugetlbfs and shared memory ranges are also compatible. .TP .B UFFDIO_REGISTER_MODE_WP Track page faults on write-protected pages. +Since Linux 5.7, only private anonymous ranges are compatible. +.TP +.B UFFDIO_REGISTER_MODE_MINOR +Track minor page faults. +Since Linux 5.13, only hugetlbfs ranges are compatible. .PP If the operation is successful, the kernel modifies the .I ioctls @@ -735,6 +745,109 @@ or not registered with userfaultfd write-protect mode. .TP .B EFAULT Encountered a generic fault during processing. +.\" +.SS UFFDIO_CONTINUE +(Since Linux 5.13.) +Used for resolving minor faults specifically. +Take the existing page(s) in the range registered with +.B UFFDIO_REGISTER_MODE_MINOR +and install page table entries for them. +.PP +The +.I argp +argument is a pointer to a +.I uffdio_continue +structure as shown below: +.PP +.in +4n +.EX +struct uffdio_continue { + struct uffdio_range range; /* Range to install PTEs for and continue */ + __u64 mode; /* Flags controlling the behavior of continue */ + __s64 mapped; /* Number of bytes mapped, or negated error */ +}; +.EE +.in +.PP +The following value may be bitwise ORed in +.IR mode +to change the behavior of the +.B UFFDIO_CONTINUE +operation: +.TP +.B UFFDIO_CONTINUE_MODE_DONTWAKE +Do not wake up the thread that waits for page-fault resolution. +.PP +The +.I mapped +field is used by the kernel to return the number of bytes +that were actually mapped, or an error in the same manner as +.BR UFFDIO_COPY . +If the value returned in the +.I mapped +field doesn't match the value that was specified in +.IR range.len , +the operation fails with the error +.BR EAGAIN . +The +.I mapped +field is output-only; +it is not read by the +.B UFFDIO_CONTINUE +operation. +.PP +This +.BR ioctl (2) +operation returns 0 on success. +In this case, the entire area was mapped. +On error, \-1 is returned and +.I errno +is set to indicate the error. +Possible errors include: +.TP +.B EAGAIN +The number of bytes mapped (i.e., the value returned in the +.I mapped +field) does not equal the value that was specified in the +.I range.len +field. +.TP +.B EINVAL +Either +.I range.start +or +.I range.len +was not a multiple of the system page size; or +.I range.len +was zero; or the range specified was invalid. +.TP +.B EINVAL +An invalid bit was specified in the +.IR mode +field. +.TP +.B EEXIST +One or more pages were already mapped in the given range. +In other words, not only did pages exist in the page cache, but page table +entries already existed for those pages and they were fully mapped. +.TP +.B ENOENT +The faulting process has changed its virtual memory layout simultaneously with +an outstanding +.B UFFDIO_CONTINUE +operation. +.TP +.B ENOMEM +Allocating memory needed to setup the page table mappings failed. +.TP +.B EFAULT +No existing page could be found in the page cache for the given range. +.TP +.BR ESRCH +The faulting process has exited at the time of a +.B UFFDIO_CONTINUE +operation. +.\" .SH RETURN VALUE See descriptions of the individual operations, above. .SH ERRORS diff --git a/man2/userfaultfd.2 b/man2/userfaultfd.2 index 593c189d8..07f53c6ff 100644 --- a/man2/userfaultfd.2 +++ b/man2/userfaultfd.2 @@ -78,7 +78,7 @@ all memory ranges that were registered with the object are unregistered and unread events are flushed. .\" .PP -Userfaultfd supports two modes of registration: +Userfaultfd supports three modes of registration: .TP .BR UFFDIO_REGISTER_MODE_MISSING " (since 4.10)" When registered with @@ -92,6 +92,18 @@ or an .B UFFDIO_ZEROPAGE ioctl. .TP +.BR UFFDIO_REGISTER_MODE_MINOR " (since 5.13)" +When registered with +.B UFFDIO_REGISTER_MODE_MINOR +mode, user-space will receive a page-fault notification +when a minor page fault occurs. +That is, when a backing page is in the page cache, but +page table entries don't yet exist. +The faulted thread will be stopped from execution until the page fault is +resolved from user-space by an +.B UFFDIO_CONTINUE +ioctl. +.TP .BR UFFDIO_REGISTER_MODE_WP " (since 5.7)" When registered with .B UFFDIO_REGISTER_MODE_WP @@ -212,9 +224,10 @@ a page fault occurring in the requested memory range, and satisfying the mode defined at the registration time, will be forwarded by the kernel to the user-space application. The application can then use the -.B UFFDIO_COPY +.B UFFDIO_COPY , +.B UFFDIO_ZEROPAGE , or -.B UFFDIO_ZEROPAGE +.B UFFDIO_CONTINUE .BR ioctl (2) operations to resolve the page fault. .PP @@ -318,6 +331,43 @@ should have the flag cleared upon the faulted page or range. .PP Write-protect mode supports only private anonymous memory. +.\" +.SS Userfaultfd minor fault mode (since 5.13) +Since Linux 5.13, userfaultfd supports minor fault mode. +In this mode, fault messages are produced not for major faults (where the +page was missing), but rather for minor faults, where a page exists in the page +cache, but the page table entries are not yet present. +The user needs to first check availability of this feature using +.B UFFDIO_API +ioctl against the feature bit +.B UFFD_FEATURE_MINOR_HUGETLBFS +before using this feature. +.PP +To register with userfaultfd minor fault mode, the user needs to initiate the +.B UFFDIO_REGISTER +ioctl with mode +.B UFFD_REGISTER_MODE_MINOR +set. +.PP +When a minor fault occurs, user-space will receive a page-fault notification +whose +.I uffd_msg.pagefault.flags +will have the +.B UFFD_PAGEFAULT_FLAG_MINOR +flag set. +.PP +To resolve a minor page fault, the handler should decide whether or not the +existing page contents need to be modified first. +If so, this should be done in-place via a second, non-userfaultfd-registered +mapping to the same backing page (e.g., by mapping the hugetlbfs file twice). +Once the page is considered "up to date", the fault can be resolved by +initiating an +.B UFFDIO_CONTINUE +ioctl, which installs the page table entries and (by default) wakes up the +faulting thread(s). +.PP +Minor fault mode supports only hugetlbfs-backed memory. +.\" .SS Reading from the userfaultfd structure Each .BR read (2) @@ -456,19 +506,20 @@ For the following flag may appear: .RS .TP -.B UFFD_PAGEFAULT_FLAG_WRITE -If the address is in a range that was registered with the -.B UFFDIO_REGISTER_MODE_MISSING -flag (see -.BR ioctl_userfaultfd (2)) -and this flag is set, this a write fault; -otherwise it is a read fault. +.B UFFD_PAGEFAULT_FLAG_WP +If this flag is set, then the fault was a write-protect fault. .TP +.B UFFD_PAGEFAULT_FLAG_MINOR +If this flag is set, then the fault was a minor fault. +.TP +.B UFFD_PAGEFAULT_FLAG_WRITE +If this flag is set, then the fault was a write fault. +.HP +If neither .B UFFD_PAGEFAULT_FLAG_WP -If the address is in a range that was registered with the -.B UFFDIO_REGISTER_MODE_WP -flag, when this bit is set, it means it is a write-protect fault. -Otherwise it is a page-missing fault. +nor +.B UFFD_PAGEFAULT_FLAG_MINOR +are set, then the fault was a missing fault. .RE .TP .I pagefault.feat.pid