From patchwork Thu Nov 14 06:59:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13874612 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2D3ABD65C52 for ; Thu, 14 Nov 2024 07:00:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BB5C16B0092; Thu, 14 Nov 2024 02:00:53 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B18D86B0093; Thu, 14 Nov 2024 02:00:53 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 943456B0096; Thu, 14 Nov 2024 02:00:53 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 721CF6B0092 for ; Thu, 14 Nov 2024 02:00:53 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id D456A160F8B for ; Thu, 14 Nov 2024 07:00:52 +0000 (UTC) X-FDA: 82783801464.25.14EF2A5 Received: from mail-pl1-f174.google.com (mail-pl1-f174.google.com [209.85.214.174]) by imf07.hostedemail.com (Postfix) with ESMTP id 70CBF40431 for ; Thu, 14 Nov 2024 06:59:48 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=lYUcpL9F; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf07.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1731567520; a=rsa-sha256; cv=none; b=bozFawHpOF1BqeJ0VHUrEXkaZn9zVsX9aCwEr+Vz/Yv5IXHTHyMiQicjPC5A+Z0m3AVgCk I9O3DkWNDUyzyR1HwQeBLxzdQ1Vfm/Z0Xm+0fDTJPSEf7ksKF5EX4sh0bPRervcH66e/So sKv9kSIanIITCLfwgEPSCfUCm8030fI= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=lYUcpL9F; dmarc=pass (policy=quarantine) header.from=bytedance.com; spf=pass (imf07.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.214.174 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1731567520; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Injd64heqojhVahuoVzqrsqJ00krl1pndEuOREo5sHA=; b=uxIg9N8WZ7mUX+uEJPaUv3B27ETmrVLpRRKlnNejh0QDUovcaNkTIhJ9EHA4ojwRkPF44e NWzjNpmr0G0k0udsKVJD0mgXXWXWVNoKbZ7h5BbGS5QynWyVuqWWSMW43dKNzTGCrGy5V3 2rtPpKpelvrgE9xAXGwXBvWwVcOVxAw= Received: by mail-pl1-f174.google.com with SMTP id d9443c01a7336-20e576dbc42so2529805ad.0 for ; Wed, 13 Nov 2024 23:00:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1731567649; x=1732172449; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Injd64heqojhVahuoVzqrsqJ00krl1pndEuOREo5sHA=; b=lYUcpL9FMMQAvhNTcqPw9EzkOIeWqsqiLR94k6STE+mEJDQZ5jqAjoSKnHENMLTxwY m5aVGPvO9A/3Q1XdpFQyC5A/xQEtWNZcMmz5tpNmbyCdgliYrNywiSp7hWKiwE3I9PTd DFdM4qWLoD6ZdzIdUQ8619VswWsewTEx1GSD42aeqNzYquEF3/t+Q1IyBJMQmwmtKZT8 Yz2n3h7N1chwFIn9kyowFGkNZKyEf3efn/L4IWW0cf6icSf/Jtof6X1l2dbNIYRmIcdY rUG8dFGKVxSRlHKAaNN2E0WrRKI5fLCB3RSUVtLy1gO3lHt1e9V6DoFYl0LQLFtgM4I4 U8FQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731567649; x=1732172449; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Injd64heqojhVahuoVzqrsqJ00krl1pndEuOREo5sHA=; b=MKo8+sKDvFBKG3sPw+Y56N7m7QPfg9DX+RLIhKpqwOD7Xq5bjVd3RYp5FclEt2tRLg Tn104qFFdaMW1pNVAaVbeW0CWUHKoOnWSeBQr5Mj4I8cETgDE7vIXe5auiWB2n/JRqYy MuDc7kO15CVM5UKyOngOgkaOSt6WZ6PM61uuzs1wHgOTNLxie12FP7PM9dnNGG36CSJX SZUJwHSIWZfUfIf1OQeLxYo6AzqcFITnlJX5waic5comL/UAdSWp2R6Jc0GcTEAs3nCY jgk6RfFb+IPUMQp4Z9tgWXR+FXNUDOpz7pEw5zrlyDDPPFG9GFbyvcY5DpUZFuOABVg5 mHYg== X-Forwarded-Encrypted: i=1; AJvYcCWZdIjyyIdgMZ3+FlM+ROvG+sustXT5TzvOi1W+8Jcgy6sdiFvRqg7Ykk+kSZv29rVIQTzLlsuy0w==@kvack.org X-Gm-Message-State: AOJu0YxYOyw8mTrCpDS7c6GGjswFwE9ciU5ftwxsA/VkvqPR/nrECpcY YpJ4FPsjoL89WIDwLJKI5q2b1wSft54cr++f2RoqLMdidqs90pUDcDmin4O03+s= X-Google-Smtp-Source: AGHT+IF9J5Srma9m0kVIWNzHVFQU0S2TqIZpiUbAkjwNNaShq+S4YCS+Ol7MzG3HFBa6aKSnPA4QSw== X-Received: by 2002:a17:903:2310:b0:20c:aed1:813a with SMTP id d9443c01a7336-211b66023e2mr54146835ad.14.1731567648903; Wed, 13 Nov 2024 23:00:48 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([63.216.146.178]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-211c7d389c2sm4119065ad.268.2024.11.13.23.00.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 13 Nov 2024 23:00:48 -0800 (PST) From: Qi Zheng To: david@redhat.com, jannh@google.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, akpm@linux-foundation.org, peterx@redhat.com Cc: mgorman@suse.de, catalin.marinas@arm.com, will@kernel.org, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, x86@kernel.org, lorenzo.stoakes@oracle.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, zokeefe@google.com, rientjes@google.com, Qi Zheng Subject: [PATCH v3 3/9] mm: introduce zap_nonpresent_ptes() Date: Thu, 14 Nov 2024 14:59:54 +0800 Message-Id: <25e70f171e17370ec65159a301ff4f852991e14c.1731566457.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Queue-Id: 70CBF40431 X-Stat-Signature: e1e9cw4jefqafpkybrzgjxcikyq9i3cm X-Rspam-User: X-Rspamd-Server: rspam05 X-HE-Tag: 1731567588-957166 X-HE-Meta: U2FsdGVkX1/74X3q3e2tbwiNAq5vzgwUFJ4XhYvLA9WnB8yfRN0X8ClYZ/HLjO8LAn6aNdnv4elHVQykHHYwvCMK3fDskJRwJhM/o2c/lLeVhDjt7nBSgUkPutNMW8of5BfnpROM8mYT8X64yXQEt3VScCuUiLMPZDJz0vxTtByqZGSkZcBIpXFpYzqU8oHTi3S0gR3GRLMdvKqG3WdB3MMxk+d/U+ypBlX5oc1dW4/U4HvULzepWZe2GsFgPbudAw/FAQOrD5fd9n+BpTOr/GH5KNM89Fs4QVp7r6fL8ptwSEb9txVM7bTD7FG4T/LLqKaDMfZb4mRvW0eRKq5Bqq7Resc+j8U+iys6MabkcB78QjrY1o2a8rSuG999LeVW73rd3A6zn4RTnUUy+YV/xLJtOIP0Ij2osK+KGOJHJa6yiSNLWy1iBe8VyHQLDUQLpnRcx+Es59Q+UzQj3DvBiz9fYikvIk+CTaZVi0ILxr1lPogcyiJ4Qtj4D6qGT3UdzwycbN8+Fq3mI5M6TU2vAewShNKqzkn2/duqk9bghXnAniPVDGaiDuhtv2t2WJp/BHbAmlNiyN0/XEBpwEQLPUrpqNgxuuw/zeCLsswANClDIfeUyc2A3llhNOJuNRoo+ExVZykFXla0+bef3sV/I5E/95x4nyD+tT+dJsd2jjgZKTwkwG8h6Q15CCfZ5pB7w1pYXwXxMSShOFadtORPPFHQfbaUPKCbjy4RNCz4wqVoizFypw/C3Fzmk3zw4U3lRxoDSB5aW4Qu75h2lb/8R9ZOYnBO5ur6L/x6QMvAGz78bC2kImeKOfCZn6Zds9kYWFicaRTQECFlzuk2MIi6Zbm4enmBmJV334a/db/6YWYMp4wnXZGkgyjbs/3hBq8yaXPMCYcl/Uzq9+zWR+I6PA3hjGCT0//W4weHa5+Iba11Zyd0iFjLC/4WnFmwl+vB3mJg0mi4V4sjLIGEicU 9kNkHr3v PXKowF7m3UE5Tsp2uLMaab2P6hnF9PwEzaIQH1N2OHg1bqKmV/9rBtT3PIvnntl2t4mlwD3V1W0uJ9hAzuerIlMu+aaj9iIWiEbW9Lc5QwGAXu6Q5lrA3lRaAyEavCnBvYAz+ddJ/WiI91ggypc+G+fyRCnQg3FjWQW0zIBswxUM4I1UUnfqZCOs/a2yqf1Vwgd48CNu6tm567cFWjqFbu5mgjH6tC2FoKxMM1hx4dDxKCEP8TJ8nkPa+jebi2hxQeHbRM6FLJTe6ZEWpKL5dfhNqu0rGO5ncxbc5Hzz8ep6bME+iiSTSZjf/rJVEmvRcs6B+QtYdn1sZTccqqv+CBv2ogJnB0siD/bTczrjCXDITadnh/vS7FYr+c7eNDIurRIaLwKv1kJ2t/Ut+eUKM9eAeVQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Similar to zap_present_ptes(), let's introduce zap_nonpresent_ptes() to handle non-present ptes, which can improve code readability. No functional change. Signed-off-by: Qi Zheng Reviewed-by: Jann Horn Acked-by: David Hildenbrand --- mm/memory.c | 136 ++++++++++++++++++++++++++++------------------------ 1 file changed, 73 insertions(+), 63 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 209885a4134f7..bd9ebe0f4471f 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1587,6 +1587,76 @@ static inline int zap_present_ptes(struct mmu_gather *tlb, return 1; } +static inline int zap_nonpresent_ptes(struct mmu_gather *tlb, + struct vm_area_struct *vma, pte_t *pte, pte_t ptent, + unsigned int max_nr, unsigned long addr, + struct zap_details *details, int *rss) +{ + swp_entry_t entry; + int nr = 1; + + entry = pte_to_swp_entry(ptent); + if (is_device_private_entry(entry) || + is_device_exclusive_entry(entry)) { + struct page *page = pfn_swap_entry_to_page(entry); + struct folio *folio = page_folio(page); + + if (unlikely(!should_zap_folio(details, folio))) + return 1; + /* + * Both device private/exclusive mappings should only + * work with anonymous page so far, so we don't need to + * consider uffd-wp bit when zap. For more information, + * see zap_install_uffd_wp_if_needed(). + */ + WARN_ON_ONCE(!vma_is_anonymous(vma)); + rss[mm_counter(folio)]--; + if (is_device_private_entry(entry)) + folio_remove_rmap_pte(folio, page, vma); + folio_put(folio); + } else if (!non_swap_entry(entry)) { + /* Genuine swap entries, hence a private anon pages */ + if (!should_zap_cows(details)) + return 1; + + nr = swap_pte_batch(pte, max_nr, ptent); + rss[MM_SWAPENTS] -= nr; + free_swap_and_cache_nr(entry, nr); + } else if (is_migration_entry(entry)) { + struct folio *folio = pfn_swap_entry_folio(entry); + + if (!should_zap_folio(details, folio)) + return 1; + rss[mm_counter(folio)]--; + } else if (pte_marker_entry_uffd_wp(entry)) { + /* + * For anon: always drop the marker; for file: only + * drop the marker if explicitly requested. + */ + if (!vma_is_anonymous(vma) && !zap_drop_markers(details)) + return 1; + } else if (is_guard_swp_entry(entry)) { + /* + * Ordinary zapping should not remove guard PTE + * markers. Only do so if we should remove PTE markers + * in general. + */ + if (!zap_drop_markers(details)) + return 1; + } else if (is_hwpoison_entry(entry) || is_poisoned_swp_entry(entry)) { + if (!should_zap_cows(details)) + return 1; + } else { + /* We should have covered all the swap entry types */ + pr_alert("unrecognized swap entry 0x%lx\n", entry.val); + WARN_ON_ONCE(1); + } + clear_not_present_full_ptes(vma->vm_mm, addr, pte, nr, tlb->fullmm); + zap_install_uffd_wp_if_needed(vma, addr, pte, nr, details, ptent); + + return nr; +} + static unsigned long zap_pte_range(struct mmu_gather *tlb, struct vm_area_struct *vma, pmd_t *pmd, unsigned long addr, unsigned long end, @@ -1598,7 +1668,6 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, spinlock_t *ptl; pte_t *start_pte; pte_t *pte; - swp_entry_t entry; int nr; tlb_change_page_size(tlb, PAGE_SIZE); @@ -1611,8 +1680,6 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, arch_enter_lazy_mmu_mode(); do { pte_t ptent = ptep_get(pte); - struct folio *folio; - struct page *page; int max_nr; nr = 1; @@ -1622,8 +1689,8 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, if (need_resched()) break; + max_nr = (end - addr) / PAGE_SIZE; if (pte_present(ptent)) { - max_nr = (end - addr) / PAGE_SIZE; nr = zap_present_ptes(tlb, vma, pte, ptent, max_nr, addr, details, rss, &force_flush, &force_break); @@ -1631,67 +1698,10 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, addr += nr * PAGE_SIZE; break; } - continue; - } - - entry = pte_to_swp_entry(ptent); - if (is_device_private_entry(entry) || - is_device_exclusive_entry(entry)) { - page = pfn_swap_entry_to_page(entry); - folio = page_folio(page); - if (unlikely(!should_zap_folio(details, folio))) - continue; - /* - * Both device private/exclusive mappings should only - * work with anonymous page so far, so we don't need to - * consider uffd-wp bit when zap. For more information, - * see zap_install_uffd_wp_if_needed(). - */ - WARN_ON_ONCE(!vma_is_anonymous(vma)); - rss[mm_counter(folio)]--; - if (is_device_private_entry(entry)) - folio_remove_rmap_pte(folio, page, vma); - folio_put(folio); - } else if (!non_swap_entry(entry)) { - max_nr = (end - addr) / PAGE_SIZE; - nr = swap_pte_batch(pte, max_nr, ptent); - /* Genuine swap entries, hence a private anon pages */ - if (!should_zap_cows(details)) - continue; - rss[MM_SWAPENTS] -= nr; - free_swap_and_cache_nr(entry, nr); - } else if (is_migration_entry(entry)) { - folio = pfn_swap_entry_folio(entry); - if (!should_zap_folio(details, folio)) - continue; - rss[mm_counter(folio)]--; - } else if (pte_marker_entry_uffd_wp(entry)) { - /* - * For anon: always drop the marker; for file: only - * drop the marker if explicitly requested. - */ - if (!vma_is_anonymous(vma) && - !zap_drop_markers(details)) - continue; - } else if (is_guard_swp_entry(entry)) { - /* - * Ordinary zapping should not remove guard PTE - * markers. Only do so if we should remove PTE markers - * in general. - */ - if (!zap_drop_markers(details)) - continue; - } else if (is_hwpoison_entry(entry) || - is_poisoned_swp_entry(entry)) { - if (!should_zap_cows(details)) - continue; } else { - /* We should have covered all the swap entry types */ - pr_alert("unrecognized swap entry 0x%lx\n", entry.val); - WARN_ON_ONCE(1); + nr = zap_nonpresent_ptes(tlb, vma, pte, ptent, max_nr, + addr, details, rss); } - clear_not_present_full_ptes(mm, addr, pte, nr, tlb->fullmm); - zap_install_uffd_wp_if_needed(vma, addr, pte, nr, details, ptent); } while (pte += nr, addr += PAGE_SIZE * nr, addr != end); add_mm_rss_vec(mm, rss);