From patchwork Wed Dec 4 11:09:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Qi Zheng X-Patchwork-Id: 13893584 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B5263E7716D for ; Wed, 4 Dec 2024 11:10:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4AB9F6B0092; Wed, 4 Dec 2024 06:10:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 482BB6B0093; Wed, 4 Dec 2024 06:10:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 322B86B0095; Wed, 4 Dec 2024 06:10:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 09F6A6B0092 for ; Wed, 4 Dec 2024 06:10:44 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 9ADF9A0F2B for ; Wed, 4 Dec 2024 11:10:43 +0000 (UTC) X-FDA: 82857008598.02.EE57CFB Received: from mail-pg1-f175.google.com (mail-pg1-f175.google.com [209.85.215.175]) by imf29.hostedemail.com (Postfix) with ESMTP id E77D3120024 for ; Wed, 4 Dec 2024 11:10:20 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=K9pfmLoN; spf=pass (imf29.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.215.175 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733310635; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=uBIs+RyxQ8T92otn81RkrE5S0sfAaFlsvX0Ncx+wxck=; b=FS8u0uVxhXdS25V7hmtLU5j/c8qe28gzZqzvHy2S2HJsQ+I/a6SgrLeP41BzG1i/M3jh9B Hh3KJnFVl9YwWmxtHocUKntcHlu5hE/+uYtLXHqb5kM1CUwwWC6gpzNGQXbt6FMvAr2ZLB W6GzaJX1V1wPeGSzk3x7GtaZLLFTMOU= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733310635; a=rsa-sha256; cv=none; b=gSS1s9/sO/8qIWOXgEDOe/N3xbNMRPbVDvikhJAvKBdC50okdZnWlenEfAYaxFklY7Rj7X fh7mbqUDwdOoyzvgYDNWi51v6RFPSQulObif0sS73e2ktd8uxcy3KYu1+gbbQqXCIIjYVI kU8i/Q9h6hQN9iOMJLvPmjRnpwqYhVY= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=K9pfmLoN; spf=pass (imf29.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.215.175 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com Received: by mail-pg1-f175.google.com with SMTP id 41be03b00d2f7-7f46d5d1ad5so5079828a12.3 for ; Wed, 04 Dec 2024 03:10:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1733310640; x=1733915440; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=uBIs+RyxQ8T92otn81RkrE5S0sfAaFlsvX0Ncx+wxck=; b=K9pfmLoN7jgTNNrZW5wixeJUGQOwc3HiLFX5Wno1coDgHFddtdkylBZGur+OO6+okn h+pLO3MNOPu46LwysgnbjCyzp212iupE2ZX1Ncxt+r2KiHV4JFc7ZcG4SCQw5mI/rI1U kt2gVaio+QZwiCcqdgP24kpRv7TpVsMPI17U08DEeQciIbFOO4GNr28+XqiD5+iHJhS8 lcFQ5Dkcp62I/KnqbCeZirP3MoBicqVoWK5cSvH9nVVWfn+oYWEFxoTpwKTUsBW0KRlA awLDBidtqWqnk/RQLy4fsWdTt/9nU8TmVJX/ntIys9uX9MLwRrXJYEtW2JLhIfwUE0YO lDsQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733310640; x=1733915440; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=uBIs+RyxQ8T92otn81RkrE5S0sfAaFlsvX0Ncx+wxck=; b=Cd+XrIgiJeUvJc47r7Y+4qrk5fLn33VPugxxcqIfubvtNfRQsSASs4zxEECk/WMhcn 0uds2a3tFw5ZXWLGAjDaKc8C6kYtjAvdIz4//2nVDyqIIZ0dBKwuoEooMKpoW06RCjUl doTDpDsvDl+NgguNvmdwVwYJLgA9gpAJTqo0oHLWcZH0h2veLwLGq62HXf0OkQvrbtO/ 6/3DNqyRxBZplOW901Gle8C9M4TFnJON7FhsVQoZHTHE/kCDvnrBhtiVeeyQiweJZBD1 P9N/uaSDYz14bSJVvD1AblbixeK5yHFRSfCA6i3xWxlJcmv6ZCTfoeqW95BiWGGmpIIh UzUw== X-Forwarded-Encrypted: i=1; AJvYcCX9jMPbYPAZWVFXVN8KYnSXqpXtJvPePagj2+8S3G6fYhI2a5X4ICX3uiYh8FsoHJqlPprQ/+cxiQ==@kvack.org X-Gm-Message-State: AOJu0YxOInWfm7MagdlxMQPpXDYIZDpXxVBE8vZqnwmOQS3mToEroam/ VxF0g3woG2KEr1cuJYAEDnpgADCgtAQnp0iFuysp9y3R61cmxZRdYmNIefRgDq4= X-Gm-Gg: ASbGnctGDiigOPI9tKkqTzdxT30GFN4mYEqiiuFiJb9pQvssyClRYfQUaisyWFSdXvA iQaKXmYzaTHJ82mrnzKbo29xh7hKmh1SSnWVGejP0tifZurTaqOhXnNvGHLP9mkn4z1M5/AaFAu riO7FM1Y2X/Oq6gokkc1+lu9SFVb2hQsQdd0QoLWSsdxaO6lotPGLffjWIbHDXT3TkvwfmJJT0v whWLPbxRob3dvw4RGDd72Edkt5GRMPd6AdMiaG/4nHQ2yZQDbFqX2o3pCPvggxBXU6jNuY5ypfX Yw4VzW/OEvgHXLE= X-Google-Smtp-Source: AGHT+IER5lnoQQdWEC7gbQxZosi7N0mw2aeYdLAS9LZW7bMxGiW32vvaQ3PYOO0pQsVEtwB2CIp8WQ== X-Received: by 2002:a05:6a20:7f8e:b0:1e0:ca95:3cb2 with SMTP id adf61e73a8af0-1e1653a7c1fmr8526385637.8.1733310640114; Wed, 04 Dec 2024 03:10:40 -0800 (PST) Received: from C02DW0BEMD6R.bytedance.net ([203.208.167.148]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-21527515731sm107447495ad.192.2024.12.04.03.10.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Dec 2024 03:10:38 -0800 (PST) From: Qi Zheng To: david@redhat.com, jannh@google.com, hughd@google.com, willy@infradead.org, muchun.song@linux.dev, vbabka@kernel.org, peterx@redhat.com, akpm@linux-foundation.org Cc: mgorman@suse.de, catalin.marinas@arm.com, will@kernel.org, dave.hansen@linux.intel.com, luto@kernel.org, peterz@infradead.org, x86@kernel.org, lorenzo.stoakes@oracle.com, zokeefe@google.com, rientjes@google.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Qi Zheng Subject: [PATCH v4 03/11] mm: introduce zap_nonpresent_ptes() Date: Wed, 4 Dec 2024 19:09:43 +0800 Message-Id: <009ca882036d9c7a9f815489cfeafe0bdb79d62d.1733305182.git.zhengqi.arch@bytedance.com> X-Mailer: git-send-email 2.24.3 (Apple Git-128) In-Reply-To: References: MIME-Version: 1.0 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: E77D3120024 X-Stat-Signature: bnc46i9gzdura877ebq538jwsygutkx1 X-Rspam-User: X-HE-Tag: 1733310620-115304 X-HE-Meta: U2FsdGVkX18VyswK0ExdlWzMAmt3mae9MeQlEYwlkrnj4/o9u0v8FEsGjbBofQfWC23ukn71YaCs2eXNMKpMi8VPoJe67uSbA0MfMATY8ksi7OR/4OTk8o1xCnFNKYOlVYC3LUey/IUMnwlm+TbUH51IXb1rc7N5XW5JrAdM3hbqvdCv47CZ5kfiG+ancsxqNnQIf8c5GgLHS2TP1rqUOp8SKiBVlUh6v1IhRXRNIO8pxjctUdJbZkyr+xk66LlYTOq+tDLG6hapoPmp0m8B579xPae9FHM8igChKiuR/q644dI+IIkYC8MxCWtQNwFIyeoa6uSYrY5vJotdSFMVaBEfWTBvddneIGphUBz1F5L/y8hM7OKzkqEqfiLyNJdFGlgDtlhpaeiUuHcDhBhArQAegYj9A9zXpcKm8H6DMtDqGxH7ti/B+GOP2yLjeas319vr+ALCtTFQ3vThyikVk8CxxGgG8kMFHxs+C9JF9sHpAor4k8j18zxv5jQ6/NPWPFxDqP7WVuIWRY3cW9cQeE2Kn0YyXNPciQ5QneO6WdU2WzxUEiTlo4NBOIILO+02VNBM1wYWSBK9CCMIQs3fSUZO4yz3ZraMCokMEWJ4d1GBE7F9aLE5xjVq4M11E88qF3QEb+qYYQW276pflaEihioFS34w84II2z8uofOI8gxUd6iraoQRufv79RPSkw9vYVnFYEaTGjbCwVFQrpncy+fprdkddD6aGsPoP68l3kwelsi4VgfNhufSW+QgcAq5CD+9rChjRWZPO92YqLvpiHC+cBc3oEtYeCmjorU20vglpwBHpdWpDPVf9svHlhw/JwfDsvKe10qJYbryAo2MpHaFoCMDLYVJPx9+3o/LdJDi4Mw69fNDqyoVlgU7zvcZMnWvstcAmxhcocTgg3j77pX3AYFVrC7sRPiQzxRO4i1jc0PdabS0u4bPjXciAfmh9nx6bwPCuTkNyc3xjgL kfgXlfke l1jybsEPLwl+y9AmsyBg0p8AouQtAQfKRktRaz0M5vmMyW8FjubdIvHSGZVFfV8GzRWrC/7OWpNg0hcUH9lS4I/b8cgN8zd+H+fpDBnuq/2kCt4XqhW7DIZd0cgB4sLq8k4Ab4732J8QGcUPLPto8uJ+6uKcvlUuiPsVKuAyywafNhe+7qujSZPCwKBX78XS5ZWkP8NjVm33ZL2ZmesbaoeLlvL/SaAzsKBUDOe7g+Hqorb6QpoNbzOKMs7Jx/jZG2l3qOIZHBOuQSwyyFC9EYcrteHgcRQLtnZid0jj05lnumA6WU8h12t0ifraCqgTzr6Ckd2PAFf2n7NgdDw/rLQK6GCePyJwbGpm6Blsto3jyuG9dgu73sdLDMz4mnol1H6WpM+L4h1DcF7Rz8Kc7sbyNLw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Similar to zap_present_ptes(), let's introduce zap_nonpresent_ptes() to handle non-present ptes, which can improve code readability. No functional change. Signed-off-by: Qi Zheng Reviewed-by: Jann Horn Acked-by: David Hildenbrand --- mm/memory.c | 136 ++++++++++++++++++++++++++++------------------------ 1 file changed, 73 insertions(+), 63 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index d5a1b0a6bf1fa..5624c22bb03cf 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1587,6 +1587,76 @@ static inline int zap_present_ptes(struct mmu_gather *tlb, return 1; } +static inline int zap_nonpresent_ptes(struct mmu_gather *tlb, + struct vm_area_struct *vma, pte_t *pte, pte_t ptent, + unsigned int max_nr, unsigned long addr, + struct zap_details *details, int *rss) +{ + swp_entry_t entry; + int nr = 1; + + entry = pte_to_swp_entry(ptent); + if (is_device_private_entry(entry) || + is_device_exclusive_entry(entry)) { + struct page *page = pfn_swap_entry_to_page(entry); + struct folio *folio = page_folio(page); + + if (unlikely(!should_zap_folio(details, folio))) + return 1; + /* + * Both device private/exclusive mappings should only + * work with anonymous page so far, so we don't need to + * consider uffd-wp bit when zap. For more information, + * see zap_install_uffd_wp_if_needed(). + */ + WARN_ON_ONCE(!vma_is_anonymous(vma)); + rss[mm_counter(folio)]--; + if (is_device_private_entry(entry)) + folio_remove_rmap_pte(folio, page, vma); + folio_put(folio); + } else if (!non_swap_entry(entry)) { + /* Genuine swap entries, hence a private anon pages */ + if (!should_zap_cows(details)) + return 1; + + nr = swap_pte_batch(pte, max_nr, ptent); + rss[MM_SWAPENTS] -= nr; + free_swap_and_cache_nr(entry, nr); + } else if (is_migration_entry(entry)) { + struct folio *folio = pfn_swap_entry_folio(entry); + + if (!should_zap_folio(details, folio)) + return 1; + rss[mm_counter(folio)]--; + } else if (pte_marker_entry_uffd_wp(entry)) { + /* + * For anon: always drop the marker; for file: only + * drop the marker if explicitly requested. + */ + if (!vma_is_anonymous(vma) && !zap_drop_markers(details)) + return 1; + } else if (is_guard_swp_entry(entry)) { + /* + * Ordinary zapping should not remove guard PTE + * markers. Only do so if we should remove PTE markers + * in general. + */ + if (!zap_drop_markers(details)) + return 1; + } else if (is_hwpoison_entry(entry) || is_poisoned_swp_entry(entry)) { + if (!should_zap_cows(details)) + return 1; + } else { + /* We should have covered all the swap entry types */ + pr_alert("unrecognized swap entry 0x%lx\n", entry.val); + WARN_ON_ONCE(1); + } + clear_not_present_full_ptes(vma->vm_mm, addr, pte, nr, tlb->fullmm); + zap_install_uffd_wp_if_needed(vma, addr, pte, nr, details, ptent); + + return nr; +} + static unsigned long zap_pte_range(struct mmu_gather *tlb, struct vm_area_struct *vma, pmd_t *pmd, unsigned long addr, unsigned long end, @@ -1598,7 +1668,6 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, spinlock_t *ptl; pte_t *start_pte; pte_t *pte; - swp_entry_t entry; int nr; tlb_change_page_size(tlb, PAGE_SIZE); @@ -1611,8 +1680,6 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, arch_enter_lazy_mmu_mode(); do { pte_t ptent = ptep_get(pte); - struct folio *folio; - struct page *page; int max_nr; nr = 1; @@ -1622,8 +1689,8 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, if (need_resched()) break; + max_nr = (end - addr) / PAGE_SIZE; if (pte_present(ptent)) { - max_nr = (end - addr) / PAGE_SIZE; nr = zap_present_ptes(tlb, vma, pte, ptent, max_nr, addr, details, rss, &force_flush, &force_break); @@ -1631,67 +1698,10 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, addr += nr * PAGE_SIZE; break; } - continue; - } - - entry = pte_to_swp_entry(ptent); - if (is_device_private_entry(entry) || - is_device_exclusive_entry(entry)) { - page = pfn_swap_entry_to_page(entry); - folio = page_folio(page); - if (unlikely(!should_zap_folio(details, folio))) - continue; - /* - * Both device private/exclusive mappings should only - * work with anonymous page so far, so we don't need to - * consider uffd-wp bit when zap. For more information, - * see zap_install_uffd_wp_if_needed(). - */ - WARN_ON_ONCE(!vma_is_anonymous(vma)); - rss[mm_counter(folio)]--; - if (is_device_private_entry(entry)) - folio_remove_rmap_pte(folio, page, vma); - folio_put(folio); - } else if (!non_swap_entry(entry)) { - max_nr = (end - addr) / PAGE_SIZE; - nr = swap_pte_batch(pte, max_nr, ptent); - /* Genuine swap entries, hence a private anon pages */ - if (!should_zap_cows(details)) - continue; - rss[MM_SWAPENTS] -= nr; - free_swap_and_cache_nr(entry, nr); - } else if (is_migration_entry(entry)) { - folio = pfn_swap_entry_folio(entry); - if (!should_zap_folio(details, folio)) - continue; - rss[mm_counter(folio)]--; - } else if (pte_marker_entry_uffd_wp(entry)) { - /* - * For anon: always drop the marker; for file: only - * drop the marker if explicitly requested. - */ - if (!vma_is_anonymous(vma) && - !zap_drop_markers(details)) - continue; - } else if (is_guard_swp_entry(entry)) { - /* - * Ordinary zapping should not remove guard PTE - * markers. Only do so if we should remove PTE markers - * in general. - */ - if (!zap_drop_markers(details)) - continue; - } else if (is_hwpoison_entry(entry) || - is_poisoned_swp_entry(entry)) { - if (!should_zap_cows(details)) - continue; } else { - /* We should have covered all the swap entry types */ - pr_alert("unrecognized swap entry 0x%lx\n", entry.val); - WARN_ON_ONCE(1); + nr = zap_nonpresent_ptes(tlb, vma, pte, ptent, max_nr, + addr, details, rss); } - clear_not_present_full_ptes(mm, addr, pte, nr, tlb->fullmm); - zap_install_uffd_wp_if_needed(vma, addr, pte, nr, details, ptent); } while (pte += nr, addr += PAGE_SIZE * nr, addr != end); add_mm_rss_vec(mm, rss);