From patchwork Mon Jul 18 11:47:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 12921635 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BBB05C43334 for ; Mon, 18 Jul 2022 19:22:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5A8AF6B0075; Mon, 18 Jul 2022 15:22:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 50A8F6B0078; Mon, 18 Jul 2022 15:22:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 35D916B007B; Mon, 18 Jul 2022 15:22:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 22E2D6B0075 for ; Mon, 18 Jul 2022 15:22:43 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id F2A4033512 for ; Mon, 18 Jul 2022 19:22:42 +0000 (UTC) X-FDA: 79701192564.26.3A6D725 Received: from relay5.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by imf21.hostedemail.com (Postfix) with ESMTP id 968FB1C008D for ; Mon, 18 Jul 2022 19:22:42 +0000 (UTC) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 5026C20438 for ; Mon, 18 Jul 2022 19:22:42 +0000 (UTC) X-FDA: 79701192564.04.AA3260B Received: from mail-pf1-f179.google.com (mail-pf1-f179.google.com [209.85.210.179]) by imf01.hostedemail.com (Postfix) with ESMTP id E8C9C40067 for ; Mon, 18 Jul 2022 19:22:41 +0000 (UTC) Received: by mail-pf1-f179.google.com with SMTP id g126so11531594pfb.3 for ; Mon, 18 Jul 2022 12:22:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Ww8yAbYm93gsR7TAtQ/8ukDrr5zdDTnK385wVRiHFGY=; b=lmCGJim1YiEvbO+WT1twN6bmgaMWthf72oMjp34gdbGNhOw75RDN9th0CLrliZBNJl J3brxYlh8GQTkprkdHkD5i6WDCs2Oe9X4VSAolRysX6gyX0Pt7smhvp/3+NOyq3qrhnx 0DGlNkM0FcP2PZSCj9Wo/G7NJ69Zt0T4MG8eEQdfZleG8F3n/SgCodtczkEQbFxwnUmn zEonsiGMRe1vb/AUKPZSq7W05L0Zerf9zhbC699t27BEbqjYu7/E3CyJMQj+IafiqHTI 3Qm0tl8IQJa+GKLNB2nHSl2B9fxnk81RaZUmZf/kJcrjG4zM8J9699iXS/eKWBuQywVl Ev0w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Ww8yAbYm93gsR7TAtQ/8ukDrr5zdDTnK385wVRiHFGY=; b=eZ0TiAVZlaslVoYNjOAY2qRevutkimuku34VC14/qppIQp9RsmAORiC3Qac321Y8vn wmkED2hS3terkM+kT1Hlex12KvbUqjba7Eesz1w8L8WVcJOyL6ytMfIxwkC2L7huw3wP 6DsIWN9gZe3VHcKdf8dfnPMCtuLTTOpaoPDqsJa2Kwih9Eg7R3qPo/ZVSZ64DHVCai9R 6AXUV+H/Ca9sz66iAyCGvIFhQ3NjXtbugAm8LsBMazZqC3muS5qR1aiP47zDcAfYgOip OugoI04naIvpuUYwfDngo+Z3ZdDEtPQlFanMut6tJNa/h8ShX7f0MVUj8+wQ/mZwSTWy qz6Q== X-Gm-Message-State: AJIora9pgog+4f13jVvTy7M1oP0ldYF917mPyMjjOR+aGNlk0mQW8boc CHqgDYc66B2xH0xjR7wRx4jFZ14XRPE= X-Google-Smtp-Source: AGRyM1uY9QgJV/Sd0CdywlLsSm/mW3f90iJxrGW3gb9jU1ZnG+9/Q5X+9uZpEuCW0xBS28LVgO9yow== X-Received: by 2002:a05:6a00:1a92:b0:52b:ac3:7964 with SMTP id e18-20020a056a001a9200b0052b0ac37964mr29235872pfv.31.1658172160532; Mon, 18 Jul 2022 12:22:40 -0700 (PDT) Received: from sc2-haas01-esx0118.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id g9-20020a625209000000b0051bc5f4df1csm9613570pfb.154.2022.07.18.12.22.39 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 18 Jul 2022 12:22:40 -0700 (PDT) From: Nadav Amit X-Google-Original-From: Nadav Amit To: linux-mm@kvack.org Cc: Andrew Morton , Nadav Amit , David Hildenbrand , Mike Kravetz , Hugh Dickins , Axel Rasmussen , Mike Rapoport , Peter Xu Subject: [PATCH v2 4/5] userfaultfd: zero access/write hints Date: Mon, 18 Jul 2022 04:47:47 -0700 Message-Id: <20220718114748.2623-5-namit@vmware.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20220718114748.2623-1-namit@vmware.com> References: <20220718114748.2623-1-namit@vmware.com> MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1658172162; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Ww8yAbYm93gsR7TAtQ/8ukDrr5zdDTnK385wVRiHFGY=; b=43PdAVSsEiN3HXbf+jgsDE33Mvp2AaatFbRcOonVYaRCx5CDXgazaTCBmJzuCLpahq5c6p XBROvARAWwWxeaCizJfvhDvP/k6GOD7IN1PSEXoZTI9V2eQ9Wo8J4k42YBHOn/+rQtRsZZ giCn7xR+qRhV3v0XH/IQ8IuvZfDi+d0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1658172162; a=rsa-sha256; cv=none; b=u+BMoPu02971ZETYMdUGuWO0mssQaYjsV5ulgVSXN4clPc4tlmQTczPZnkBZAQIJidAKiE AhEt2dgwGvZLYuKhAoH2jUeDTECzBCc+d5HVas/i2l8krXjcAgjMbWdqylRHlNW9hkdnJH 6iQS0amnA67osOTPLK+LBqFDY+vjMkQ= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=lmCGJim1; dmarc=pass (policy=none) header.from=gmail.com; spf=none (imf21.hostedemail.com: domain of MAILER-DAEMON@hostedemail.com has no SPF policy when checking 216.40.44.14) smtp.mailfrom=MAILER-DAEMON@hostedemail.com X-Rspamd-Server: rspam10 X-Stat-Signature: maothbhg596i618gb3brr68yogfrm68i X-Rspam-User: X-Rspamd-Queue-Id: 968FB1C008D X-HE-Tag-Orig: 1658172161-183184 Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=lmCGJim1; dmarc=pass (policy=none) header.from=gmail.com; spf=none (imf21.hostedemail.com: domain of MAILER-DAEMON@hostedemail.com has no SPF policy when checking 216.40.44.14) smtp.mailfrom=MAILER-DAEMON@hostedemail.com X-HE-Tag: 1658172162-661646 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Nadav Amit When userfaultfd provides a zeropage in response to ioctl, it provides a readonly alias to the zero page. If the page is later written (which is the likely scenario), page-fault occurs and the page-fault allocator allocates a page and rewires the page-tables. This is an expensive flow for cases in which a page is likely be written to. Users can use the copy ioctl to initialize zero page (by copying zeros), but this is also wasteful. Allow userfaultfd users to efficiently map initialized zero-pages that are writable. IF UFFDIO_ZEROPAGE_MODE_WRITE_LIKELY is provided would map a clear page instead of an alias to the zero page. Suggested-by: David Hildenbrand Cc: Mike Kravetz Cc: Hugh Dickins Cc: Andrew Morton Cc: Axel Rasmussen Cc: Mike Rapoport Acked-by: Peter Xu Signed-off-by: Nadav Amit Reviewed-by: David Hildenbrand --- mm/userfaultfd.c | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index c15679f3eb6a..954c6980b29f 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -241,6 +241,37 @@ static int mfill_zeropage_pte(struct mm_struct *dst_mm, return ret; } +static int mfill_clearpage_pte(struct mm_struct *dst_mm, pmd_t *dst_pmd, + struct vm_area_struct *dst_vma, + unsigned long dst_addr, + uffd_flags_t uffd_flags) +{ + struct page *page; + int ret; + + ret = -ENOMEM; + page = alloc_zeroed_user_highpage_movable(dst_vma, dst_addr); + if (!page) + goto out; + + /* The PTE is not marked as dirty unconditionally */ + SetPageDirty(page); + __SetPageUptodate(page); + + if (mem_cgroup_charge(page_folio(page), dst_vma->vm_mm, GFP_KERNEL)) + goto out_release; + + ret = mfill_atomic_install_pte(dst_mm, dst_pmd, dst_vma, dst_addr, + page, true, uffd_flags); + if (ret) + goto out_release; +out: + return ret; +out_release: + put_page(page); + goto out; +} + /* Handles UFFDIO_CONTINUE for all shmem VMAs (shared or private). */ static int mcontinue_atomic_pte(struct mm_struct *dst_mm, pmd_t *dst_pmd, @@ -500,6 +531,10 @@ static __always_inline ssize_t mfill_atomic_pte(struct mm_struct *dst_mm, err = mcopy_atomic_pte(dst_mm, dst_pmd, dst_vma, dst_addr, src_addr, page, uffd_flags); + else if (!(uffd_flags & UFFD_FLAGS_WP) && + (uffd_flags & UFFD_FLAGS_WRITE_LIKELY)) + err = mfill_clearpage_pte(dst_mm, dst_pmd, dst_vma, + dst_addr, uffd_flags); else err = mfill_zeropage_pte(dst_mm, dst_pmd, dst_vma, dst_addr, uffd_flags);