From patchwork Wed Dec 4 11:09:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13893587 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B18CBE7716B for ; Wed, 4 Dec 2024 11:11:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4F84B6B009B; Wed, 4 Dec 2024 06:11:08 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4A8BA6B009C; Wed, 4 Dec 2024 06:11:08 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 349746B009D; Wed, 4 Dec 2024 06:11:08 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 1250F6B009B for ; Wed, 4 Dec 2024 06:11:08 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 9CE79C0EBF for ; Wed, 4 Dec 2024 11:11:07 +0000 (UTC) X-FDA: 82857009144.03.32F548E Received: from mail-pl1-f178.google.com (mail-pl1-f178.google.com [209.85.214.178]) by imf23.hostedemail.com (Postfix) with ESMTP id 30B11140018 for ; Wed, 4 Dec 2024 11:10:55 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=P6+l3lQO; spf=pass (imf23.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.178 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733310654; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YBcCGE4XlG5U77pGfU9yv3mqGKsFB2BwbFt8OgUUtNY=; b=sAlZYxYcN+rDv9U0kK1XZaOZ2voVWOCqb796aXwmsXdtlo4CO4biBPMyy+ETWPLbagfH2y KgYhvNIsDrdj7HPEnxgJM5MbROmDqLrsBxH0PlijWtuT5Z8AIdz5vB6JVNlhhXKGzLwkVH 4+FZYUvFESIFhpUvxAuMR7+kiaC2760= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=P6+l3lQO; spf=pass (imf23.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.178 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733310654; a=rsa-sha256; cv=none; b=3Xm+XlNSJnFWBBWZ8UlMXcrN5NOHIziBaoHwUtwPpXLlG+Koj0grlLypHpSTjzXS0Qk6dV C7wgfu4aY5LGKCijYwGzdYGXbxMTYQjxtNKbcAltq49tsPSVSq2g8onoGKIyxYG1clF64D 13inFiDVgMBHT/fG8M4FL4aVZEUv+to= Received: by mail-pl1-f178.google.com with SMTP id d9443c01a7336-215cc7b0c56so14459135ad.3 for ; Wed, 04 Dec 2024 03:11:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1733310664; x=1733915464; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=YBcCGE4XlG5U77pGfU9yv3mqGKsFB2BwbFt8OgUUtNY=; b=P6+l3lQOIrAsjmyU1ZHzulQZKfItp87fB6qhBX656k4BegJBX/MKlUAyLFDM3rfiyl 8164YA7CPhwup5t93hfnWU9FZh/rrrFPRhGxRjVvE6H+Ixt70PZZewex3BXMECrnQiLq x5H5BBY5KGVHyCILtOsz5t8P3f8zZIwdh+8K/3PnECMBsFDYlVmIBSZYT7DbKSbc0c/6 CKLLqw4JCt4ck9L3hpiabwCpKH3E/rkRcxel4pY7GirNTYwzwI5jWZm7J3W61oOXAioJ 2eBQJHbI3DCcVkgU3LZyDGyHF1kjbP1uUGO5KFcVuLAcrIpNJrMdwjdQbzBmmRZ4MRQe b8Lg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733310664; x=1733915464; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YBcCGE4XlG5U77pGfU9yv3mqGKsFB2BwbFt8OgUUtNY=; b=IhqUxi6ux7YSRtZp/bOSGXTB1ucSlZqqB4MJ6OeLOp2I6TobsYGYamLcR0N7OYGKzV vuNYRwf/wX+1/sDJ6tOXudrYrXHl9+K+qroR5GVk2gwWyER8miWBXM+GZ/dF+FK+eigX Zk3P3yvULjkQLy71nutihaLnt+6YOiS7C2eiu2DkU2VaxzQ25ZttTA0neCXAAxGtsLYT Y8wssDyehocstXQfw6R2meT2DnpsyLsx83uTZxjUgeI58vVX6GtZscYRTva4cFMute4B YMnI3dnEzorQ6MMjCMfOLzpD/RLPGZzrOR4CFQmaWklck9M1VyOLXEHp6GSjGvUD3L+i voQg== X-Forwarded-Encrypted: i=1; AJvYcCVtKGM8Fz7BQ6fWNRfqVuDELezx/mwnuuvWb/IzKK84IF3yCACDvaZvUL/FuaVztp/CdeFVrmACCA==@kvack.org X-Gm-Message-State: AOJu0YxBj4X5xtAWAhdze+sqnqAlj0ADiVDsOlfSy4W6O02mxrVwl3Tr f5BziG+az+ELFI6YTYI/tKN763kK8itkaW7tf8PizFmIkXCiC2YMocUMzfKQo60= X-Gm-Gg: ASbGncsolLnLU2pKLQUepOrY6FL1JzzTGqGbkqqJFKMkaMwtPdDS3qUvJ7cr1XSSKQU ssbDsT+p3NCCAz9n9Dnyi9tuEeqY7hLYkUD21NCbMlP2bYHgp6NeEGaycxFG+XACsN3YS4Wkpo2 8rcsKL3hGnto4/WFYkGWQf1fEjp0dEYIGmOpms30G0QwhBQOnNUeYSIXpTWuooWrba52fDs5dUc Lthf59lBHeKSngouApM+XKpWnsiolu5FkPZfdZsApVOnghCmy7dkxPf0NLVFMc2ZnT5lpLxwK/h 0/VwWNIvtqEVORk= X-Google-Smtp-Source: AGHT+IEq78EZW9DPKE8ZtBXaTwCJuq93zz+P96KIjyqh8unYy/RsaRnFkAa1Daiqvh42R9qprpb8TQ== X-Received: by 2002:a17:902:f687:b0:215:5a53:edee with SMTP id d9443c01a7336-215bcfc52f9mr64931185ad.9.1733310664497; Wed, 04 Dec 2024 03:11:04 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.148]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-21527515731sm107447495ad.192.2024.12.04.03.10.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Dec 2024 03:11:03 -0800 (PST) From: Qi Zheng To: david@redhat.com, jannh@google.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, peterx@redhat.com, akpm@linux-foundation.org Cc: mgorman@suse.de, catalin.marinas@arm.com, will@kernel.org, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, x86@kernel.org, lorenzo.stoakes@oracle.com, zokeefe@google.com, rientjes@google.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v4 06/11] mm: zap_install_uffd_wp_if_needed: return whether uffd-wp pte has been re-installed Date: Wed, 4 Dec 2024 19:09:46 +0800 Message-Id: <9d4516554724eda87d6576468042a1741c475413.1733305182.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: 30B11140018 X-Rspamd-Server: rspam12 X-Stat-Signature: gq9zuqkjq71mn61gza3tgkqm69gscido X-Rspam-User: X-HE-Tag: 1733310655-483463 X-HE-Meta: U2FsdGVkX1+tCvBcgJMSXCgBbLZlXIB/hPCJCWz0F3z5znDsgCKvVc8joq314CdLCVoAWS6T/sdx66APcWnRYY6Xi3zGem+rs/5b9//vNao9PJQrlQBQaO54vjT7iUOB2vcQ7r7Y+gEaSt9mtqjxMt1oUCTGwKxC0R9zNaF2gZPmgc7TmxPuYTVeBnILpJcHz+2Psfn2bGVMtx8QOioclRjbUm6rCq3Z/R8Bf9RS+109TUIFbamuIwfgj9+1VakFoYtFs+Wc8hAm56NrxtbVGeoT7mVlwDSk+0KeX+21ZVcbXOrWQYLXnjBVVVvDzMZFRVuXR4QwhQntrpbvXYSgp3jxfuw5jk05ag7QzPLGJKEhTuu/nqp77BtAhJY4rt7NEDiZdwkrA2V9jctnEEWDT9K8TsDCKH+0kNdlm1cC4WPisnBNWQWK0M0EFQRmkHVhbXM5dUw4+zy4CkI48UPJRXgl9De8/T3kW3mwFV3Bb+1FL55CEgedA93ULQCK/5CbIP+BD0y9eZkNG5u+q9CcXsgonA/PG3QdmSbfY5FzS90H4Iy9NPqeO1EfH+7lgcjhV3OnZ1MkDMjMQhZxuFgltdtbDzwfGROL7L4zxXKfQCXqb14HaVNvV5HrYLV0dq+JekG21zZ7b7LvR/Ps2K90IF7TGV+T9ZHmtmcvJUvXJp5llMRefvqDvjyyik2aQ26CfaD4ezz+tFMbpBvzyUeDtU8kGNhLKccUbR2a1BNsfbzFPPfyukNoGzs8q8fYObgt96YJLDc3GgnnZBymKUFumv3dmtl0XEOcvl+X4Z9z6JY5toTxhmNZ1LlUlwplqEqjqirC4eGCVmlUDKdzdgI/XId9+0NaC3OukilznjuQf1mg+hEw3I3C7Ev6burO1x7jCrhcd/fdlETaA0KqhSbwYrK148bFjfLLcptldqPU0X5GoZONnXWdPSdVDp/roooC16etFUF/mKzgvelH2F6 OmPYw+MV KAUlKNj9ZAhiKqgdnjXg/rtHK7FQJZjYUKXgcJX1WWX3U3iP8XUqKAw09fSCRb8cClvEJzXJCfDaC9bSFwt6KK77aKPyEdsy0BmcKxf0fOa9q2IyUm6Ai58uVwuOESLevxt82+Q64gSTU7t4KzIBmyWL0k36ok80TVnBLbFVAY46u2v1ZOy//i3MnDOCLnkjHT6g2k3ocxqHU9nw4LzqLzyRRbg5pV7wtTU4rXRWSuxM/yMAbTcGR4aDkpA1peUedQY+s2+mbtBQVTYj3fT0au0CLipmcKoeGIIhG6GSd6cC8Yhjcbun11+QCYKVxEUlyb5EbViFKOzuS7k4v/LKoDU0ZarjbASpAzUYk5vjSaXFaWmI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: In some cases, we'll replace the none pte with an uffd-wp swap special pte marker when necessary. Let's expose this information to the caller through the return value, so that subsequent commits can use this information to detect whether the PTE page is empty. Signed-off-by: Qi Zheng --- include/linux/mm_inline.h | 11 +++++++---- mm/memory.c | 16 ++++++++++++---- 2 files changed, 19 insertions(+), 8 deletions(-) diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h index 1b6a917fffa4b..34e5097182a02 100644 --- a/include/linux/mm_inline.h +++ b/include/linux/mm_inline.h @@ -564,9 +564,9 @@ static inline pte_marker copy_pte_marker( * Must be called with pgtable lock held so that no thread will see the none * pte, and if they see it, they'll fault and serialize at the pgtable lock. * - * This function is a no-op if PTE_MARKER_UFFD_WP is not enabled. + * Returns true if an uffd-wp pte was installed, false otherwise. */ -static inline void +static inline bool pte_install_uffd_wp_if_needed(struct vm_area_struct *vma, unsigned long addr, pte_t *pte, pte_t pteval) { @@ -583,7 +583,7 @@ pte_install_uffd_wp_if_needed(struct vm_area_struct *vma, unsigned long addr, * with a swap pte. There's no way of leaking the bit. */ if (vma_is_anonymous(vma) || !userfaultfd_wp(vma)) - return; + return false; /* A uffd-wp wr-protected normal pte */ if (unlikely(pte_present(pteval) && pte_uffd_wp(pteval))) @@ -596,10 +596,13 @@ pte_install_uffd_wp_if_needed(struct vm_area_struct *vma, unsigned long addr, if (unlikely(pte_swp_uffd_wp_any(pteval))) arm_uffd_pte = true; - if (unlikely(arm_uffd_pte)) + if (unlikely(arm_uffd_pte)) { set_pte_at(vma->vm_mm, addr, pte, make_pte_marker(PTE_MARKER_UFFD_WP)); + return true; + } #endif + return false; } static inline bool vma_has_recency(struct vm_area_struct *vma) diff --git a/mm/memory.c b/mm/memory.c index 7f8869a22b57c..1f149bc2c0586 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1466,27 +1466,35 @@ static inline bool zap_drop_markers(struct zap_details *details) /* * This function makes sure that we'll replace the none pte with an uffd-wp * swap special pte marker when necessary. Must be with the pgtable lock held. + * + * Returns true if uffd-wp ptes was installed, false otherwise. */ -static inline void +static inline bool zap_install_uffd_wp_if_needed(struct vm_area_struct *vma, unsigned long addr, pte_t *pte, int nr, struct zap_details *details, pte_t pteval) { + bool was_installed = false; + +#ifdef CONFIG_PTE_MARKER_UFFD_WP /* Zap on anonymous always means dropping everything */ if (vma_is_anonymous(vma)) - return; + return false; if (zap_drop_markers(details)) - return; + return false; for (;;) { /* the PFN in the PTE is irrelevant. */ - pte_install_uffd_wp_if_needed(vma, addr, pte, pteval); + if (pte_install_uffd_wp_if_needed(vma, addr, pte, pteval)) + was_installed = true; if (--nr == 0) break; pte++; addr += PAGE_SIZE; } +#endif + return was_installed; } static __always_inline void zap_present_folio_ptes(struct mmu_gather *tlb,