From patchwork Fri Apr 4 15:43:47 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nikita Kalyazin X-Patchwork-Id: 14038635 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2185EC36010 for ; Fri, 4 Apr 2025 15:44:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4A8336B0027; Fri, 4 Apr 2025 11:44:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4554A6B0028; Fri, 4 Apr 2025 11:44:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 343EA6B0029; Fri, 4 Apr 2025 11:44:21 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 13A7C6B0027 for ; Fri, 4 Apr 2025 11:44:21 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id C9574804A2 for ; Fri, 4 Apr 2025 15:44:21 +0000 (UTC) X-FDA: 83296783122.28.BA241D4 Received: from smtp-fw-9105.amazon.com (smtp-fw-9105.amazon.com [207.171.188.204]) by imf16.hostedemail.com (Postfix) with ESMTP id A012D18000A for ; Fri, 4 Apr 2025 15:44:19 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=amazon.com header.s=amazon201209 header.b=KRg8Xetb; spf=pass (imf16.hostedemail.com: domain of "prvs=182d669d3=kalyazin@amazon.co.uk" designates 207.171.188.204 as permitted sender) smtp.mailfrom="prvs=182d669d3=kalyazin@amazon.co.uk"; dmarc=pass (policy=quarantine) header.from=amazon.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1743781459; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4KrCtE/sXZ3BNNe21aeAaGtv43DEVIENoUwdd2sqOFQ=; b=Gy+AW9bNKeE//p5+msE4Vgw6evyak6TVvhBrSLM+BJF6Pj/74K6BpRGxnN3Vot2wLyt6Cu 651CDAa73RbzDUSa7BwJ82mSar170uZo/PCmKsMQ9bX4KPBfBvr1UmJH6Ku+K1BAGUWSQp Vrhaz/GtPztRs//TY1utdoiXQ5Ka6Ms= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1743781459; a=rsa-sha256; cv=none; b=MUkzBum/RCVPaLWKmdh59yomd+5Ypz+MzqPt5AnlkTsUQeqg7CyRxLtCCyW858P4T8Zu/a 5ZYOU+FdvEiiA+MS2Nm5Mb6L2AAoEOm96UOLxKXzG5dbXzYwqT4V/Cc43EaIRsESPlDOma cZc2rUQxpRTUg7pA0GQUfa+4XXRNX34= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=amazon.com header.s=amazon201209 header.b=KRg8Xetb; spf=pass (imf16.hostedemail.com: domain of "prvs=182d669d3=kalyazin@amazon.co.uk" designates 207.171.188.204 as permitted sender) smtp.mailfrom="prvs=182d669d3=kalyazin@amazon.co.uk"; dmarc=pass (policy=quarantine) header.from=amazon.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1743781460; x=1775317460; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=4KrCtE/sXZ3BNNe21aeAaGtv43DEVIENoUwdd2sqOFQ=; b=KRg8Xetb3PGol4iMwQf4x2wZ9jMGFS9oKiW7kglCvPXlctg4HUcOFIo8 XtMJ+RuTWmCuhdmZobx0Uh/DXWB8njX+hNpvJLOKJSDPb9QoYgKEWUJQg hjVkTN+ShOYVf3bNpsTmAE2bd+PmyGXaGJ9RJyYdomRyBhpRRQfzECjoB o=; X-IronPort-AV: E=Sophos;i="6.15,188,1739836800"; d="scan'208";a="7643503" Received: from pdx4-co-svc-p1-lb2-vlan2.amazon.com (HELO smtpout.prod.us-west-2.prod.farcaster.email.amazon.dev) ([10.25.36.210]) by smtp-border-fw-9105.sea19.amazon.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 04 Apr 2025 15:44:14 +0000 Received: from EX19MTAUWA001.ant.amazon.com [10.0.38.20:31324] by smtpin.naws.us-west-2.prod.farcaster.email.amazon.dev [10.0.62.147:2525] with esmtp (Farcaster) id ea24aab9-5b16-4f64-955a-2db6935ecc80; Fri, 4 Apr 2025 15:44:12 +0000 (UTC) X-Farcaster-Flow-ID: ea24aab9-5b16-4f64-955a-2db6935ecc80 Received: from EX19D020UWA004.ant.amazon.com (10.13.138.231) by EX19MTAUWA001.ant.amazon.com (10.250.64.217) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1544.14; Fri, 4 Apr 2025 15:44:08 +0000 Received: from EX19MTAUEA001.ant.amazon.com (10.252.134.203) by EX19D020UWA004.ant.amazon.com (10.13.138.231) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1544.14; Fri, 4 Apr 2025 15:44:07 +0000 Received: from email-imr-corp-prod-iad-all-1b-85daddd1.us-east-1.amazon.com (10.43.8.2) by mail-relay.amazon.com (10.252.134.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA) id 15.2.1544.14 via Frontend Transport; Fri, 4 Apr 2025 15:44:07 +0000 Received: from dev-dsk-kalyazin-1a-a12e27e2.eu-west-1.amazon.com (dev-dsk-kalyazin-1a-a12e27e2.eu-west-1.amazon.com [172.19.103.116]) by email-imr-corp-prod-iad-all-1b-85daddd1.us-east-1.amazon.com (Postfix) with ESMTPS id A7CCB41FBE; Fri, 4 Apr 2025 15:44:05 +0000 (UTC) From: Nikita Kalyazin To: , , , , , , CC: , , , , , , , , , , , , , , , , , , , Subject: [PATCH v3 1/6] mm: userfaultfd: generic continue for non hugetlbfs Date: Fri, 4 Apr 2025 15:43:47 +0000 Message-ID: <20250404154352.23078-2-kalyazin@amazon.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250404154352.23078-1-kalyazin@amazon.com> References: <20250404154352.23078-1-kalyazin@amazon.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: A012D18000A X-Stat-Signature: fkfkyx1tpb1ikywjrsw4n4ppuhcx84ho X-HE-Tag: 1743781459-589148 X-HE-Meta: U2FsdGVkX192qZqQUyQ/GSaRuKFJReWS4GNDA+EoHotIMJx47JIXvO1wfIroP7VsW8GknHzq88EiOEQSk75ah7x9FNsjEW8gX15VgVeygo/LhwyFoa47WZkFc2nu6WWDcJmR6jYi/4TE/0dmCpt89tSPiSd1Ja/iaqqTCxw/X+o+YGM0UnkgyJGq2wBqb0IHK/m/NDkST+bL0lJH/KShB5A8IGDry0BLDejcBP+gPHJkfFkTv0h7ImhQm5z/QFEzmhih2yosoujYDFO5ONNnZECV7KO8x/T3gDyqA/Vm9s5DX+d/ryQ+2i2Z0ouBxW/juhlDNm9uqW7DDHmZfOBA+Rdokv4esMrzzy/icNDcB0GaTjdyZEHSbKHrGYLOCb7BJC38Yre4W/BpObx/GX7EcV6yJp1L9ETENbj9yrgvaVfq+22zkmsNTw/ek9s/44tY6trg9/hrr9KjYMW+dyCSV7mDDCjtAxeTnDs8YpUdibZNbAB0ZG/sjOnirMOtPdWwS6RN8WuzNZZ1mDXTQG2THyjO00QV6aylplXcmUKaxN6L+HfeUFYrR42jTl67PFqgpXdqdgmXHrbvhSKsJVJp9mXZAp8P0zzQlzAJYQEn65mjYWwWrgDIslxIcdETKP45YKvZP+73m3eWGQODlX0sjFstW84tbAb4CaZeGM+O2BPGMa5pbxmlUrAvwTvRkE/r/cd2R+5dc6Ohgw4JCyVMXNieLeVItbauyyfQzs3kPCJx4xNwLLC0DXC9OfodQR0KPX+2IFoE4iOPJddTSQzmkWlbeaayayti5wzNgxwpwbOiyvw1ygYuiYLp3P7G+LOkqi/D+Xg1pbvt+610/DeemtabsG2xukudVZAoja9NQAF7iUaXbT0CMauH5m/s5gz3rU6o+p2VMzitYZeMZXbVRild1sO4EOefNZcMjBNkbwq9LDRuF84T26ZrjCR1gK2NymUpWApZXWbkxrpbiKf +q5KaoUM dGtqdyMVuWkMmodt2HhiEhgCcG5IDNFHTFJRd5UtEBv0Hpc3fESx1fC4b9y8Gq1ZNOMQl92jt1Ip8pFEr3UEVzIZZ5+FV0yZTy5B1X4rDglHTcOoYJHX3C5iTX8fThta7Uiyrn3D5UhYgKZYlcBs69p52uH/nxyQsPfCoWNye2Xb2HGLlxmfp6mxHZdW/x0/p7SoPEwKD/1cFtkLpG2H0PUh2oGJ/jmZ/VbN1BlJ99ihbvCqyHYXzoekd4N+bUr5SERRrfg52VnW4jNTVaDlvn9tYEUcaXbC6tUtUbsQXJmx7XDRKgFWJY/R/Iyt0AaQLJKL44xuw/e+aSplGC0eCWeiPB06luXLClUoC1j0jhnyzW5ynQwXY0WZrfUgadjW4AUw4VmGGvqO9NOXntRpF0USQrGSMBC28y6uDdkMphyapfY1rELuMRohTgYiDW+uuq0ImltfiR/otR19fLp0H3cVU9AqK7pndIY4LgwS9hFJzWDF4iJVUPm3IyxfYkOXpsdyJkcYsX/pOAdtx8XgD+lz10Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Remove shmem-specific code from UFFDIO_CONTINUE implementation for non-huge pages by calling vm_ops->fault(). A new VMF flag, FAULT_FLAG_USERFAULT_CONTINUE, is introduced to avoid recursive call to handle_userfault(). Suggested-by: James Houghton Signed-off-by: Nikita Kalyazin --- include/linux/mm_types.h | 4 ++++ mm/hugetlb.c | 2 +- mm/shmem.c | 9 ++++++--- mm/userfaultfd.c | 37 +++++++++++++++++++++++++++---------- 4 files changed, 38 insertions(+), 14 deletions(-) diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index 0234f14f2aa6..2f26ee9742bf 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -1429,6 +1429,9 @@ enum tlb_flush_reason { * @FAULT_FLAG_ORIG_PTE_VALID: whether the fault has vmf->orig_pte cached. * We should only access orig_pte if this flag set. * @FAULT_FLAG_VMA_LOCK: The fault is handled under VMA lock. + * @FAULT_FLAG_USERFAULT_CONTINUE: The fault handler must not call userfaultfd + * minor handler as it is being called by the + * userfaultfd code itself. * * About @FAULT_FLAG_ALLOW_RETRY and @FAULT_FLAG_TRIED: we can specify * whether we would allow page faults to retry by specifying these two @@ -1467,6 +1470,7 @@ enum fault_flag { FAULT_FLAG_UNSHARE = 1 << 10, FAULT_FLAG_ORIG_PTE_VALID = 1 << 11, FAULT_FLAG_VMA_LOCK = 1 << 12, + FAULT_FLAG_USERFAULT_CONTINUE = 1 << 13, }; typedef unsigned int __bitwise zap_flags_t; diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 97930d44d460..c004cfdcd4e2 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6228,7 +6228,7 @@ static vm_fault_t hugetlb_no_page(struct address_space *mapping, } /* Check for page in userfault range. */ - if (userfaultfd_minor(vma)) { + if (userfaultfd_minor(vma) && !(vmf->flags & FAULT_FLAG_USERFAULT_CONTINUE)) { folio_unlock(folio); folio_put(folio); /* See comment in userfaultfd_missing() block above */ diff --git a/mm/shmem.c b/mm/shmem.c index 1ede0800e846..b4159303fe59 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -2467,7 +2467,8 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index, fault_mm = vma ? vma->vm_mm : NULL; folio = filemap_get_entry(inode->i_mapping, index); - if (folio && vma && userfaultfd_minor(vma)) { + if (folio && vma && userfaultfd_minor(vma) && + !(vmf->flags & FAULT_FLAG_USERFAULT_CONTINUE)) { if (!xa_is_value(folio)) folio_put(folio); *fault_type = handle_userfault(vmf, VM_UFFD_MINOR); @@ -2727,6 +2728,8 @@ static vm_fault_t shmem_falloc_wait(struct vm_fault *vmf, struct inode *inode) static vm_fault_t shmem_fault(struct vm_fault *vmf) { struct inode *inode = file_inode(vmf->vma->vm_file); + enum sgp_type sgp = vmf->flags & FAULT_FLAG_USERFAULT_CONTINUE ? + SGP_NOALLOC : SGP_CACHE; gfp_t gfp = mapping_gfp_mask(inode->i_mapping); struct folio *folio = NULL; vm_fault_t ret = 0; @@ -2743,8 +2746,8 @@ static vm_fault_t shmem_fault(struct vm_fault *vmf) } WARN_ON_ONCE(vmf->page != NULL); - err = shmem_get_folio_gfp(inode, vmf->pgoff, 0, &folio, SGP_CACHE, - gfp, vmf, &ret); + err = shmem_get_folio_gfp(inode, vmf->pgoff, 0, &folio, sgp, gfp, vmf, + &ret); if (err) return vmf_error(err); if (folio) { diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index d06453fa8aba..4b3dbc7dac64 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -380,30 +380,47 @@ static int mfill_atomic_pte_zeropage(pmd_t *dst_pmd, return ret; } -/* Handles UFFDIO_CONTINUE for all shmem VMAs (shared or private). */ +/* Handles UFFDIO_CONTINUE for all VMAs */ static int mfill_atomic_pte_continue(pmd_t *dst_pmd, struct vm_area_struct *dst_vma, unsigned long dst_addr, uffd_flags_t flags) { - struct inode *inode = file_inode(dst_vma->vm_file); - pgoff_t pgoff = linear_page_index(dst_vma, dst_addr); struct folio *folio; struct page *page; int ret; + struct vm_fault vmf = { + .vma = dst_vma, + .address = dst_addr, + .flags = FAULT_FLAG_WRITE | FAULT_FLAG_REMOTE | + FAULT_FLAG_USERFAULT_CONTINUE, + .pte = NULL, + .page = NULL, + .pgoff = linear_page_index(dst_vma, dst_addr), + }; + + if (!dst_vma->vm_ops || !dst_vma->vm_ops->fault) + return -EINVAL; - ret = shmem_get_folio(inode, pgoff, 0, &folio, SGP_NOALLOC); - /* Our caller expects us to return -EFAULT if we failed to find folio */ - if (ret == -ENOENT) +retry: + ret = dst_vma->vm_ops->fault(&vmf); + if (ret & VM_FAULT_ERROR) { ret = -EFAULT; - if (ret) goto out; - if (!folio) { - ret = -EFAULT; + } + + if (ret & VM_FAULT_NOPAGE) { + ret = -EAGAIN; goto out; } - page = folio_file_page(folio, pgoff); + if (ret & VM_FAULT_RETRY) + goto retry; + + page = vmf.page; + folio = page_folio(page); + BUG_ON(!folio); + if (PageHWPoison(page)) { ret = -EIO; goto out_release;