From patchwork Thu Jul 4 04:31:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oscar Salvador X-Patchwork-Id: 13723196 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 38458C30653 for ; Thu, 4 Jul 2024 04:32:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 96B846B00A7; Thu, 4 Jul 2024 00:32:16 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8F4A56B00A8; Thu, 4 Jul 2024 00:32:16 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 748026B00A9; Thu, 4 Jul 2024 00:32:16 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 52B926B00A7 for ; Thu, 4 Jul 2024 00:32:16 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 153A21C2667 for ; Thu, 4 Jul 2024 04:32:16 +0000 (UTC) X-FDA: 82300798272.20.C3D4381 Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.223.131]) by imf16.hostedemail.com (Postfix) with ESMTP id 02EF218001C for ; Thu, 4 Jul 2024 04:32:13 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=none; spf=pass (imf16.hostedemail.com: domain of osalvador@suse.de designates 195.135.223.131 as permitted sender) smtp.mailfrom=osalvador@suse.de; dmarc=pass (policy=none) header.from=suse.de ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1720067515; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=jf9TaAMeyGiTtzlVo2uGSB7dEGDehBnqIYrJsOaHQk8=; b=ZEiMaGVpzi0n+v2m5Nz3YQcv6CEVBpJsAM4VLFJHU/y/o0YMoQCSjyqr3L/hhlfd0tw/f6 y5wkA12JMHQ6NbNHmFm5QjpniNeXjd/JJPXUk25sDVgW/T8LNoFM8Of+YUtPytkNqJagmm aOBCqSSbLyYS3zErFWI0DTPql4nowh0= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=none; spf=pass (imf16.hostedemail.com: domain of osalvador@suse.de designates 195.135.223.131 as permitted sender) smtp.mailfrom=osalvador@suse.de; dmarc=pass (policy=none) header.from=suse.de ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1720067515; a=rsa-sha256; cv=none; b=uhQH0O2qslzvn0j6GNnbTaxYoyCyOvIc23j6PtepOTBsT8QpmFCmkFFQIqY+2rsGszEiQx 2sY608GrCqQMzYdXxzxv7BV0IB//7Zq8NDCU5J0oo+Uclv6nngK/OiwoRSTnQ/g0W97/ld ls0WMg3/m4IIaoWejjB87XvPknCAD10= Received: from imap1.dmz-prg2.suse.org (imap1.dmz-prg2.suse.org [IPv6:2a07:de40:b281:104:10:150:64:97]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id BB4731FCFA; Thu, 4 Jul 2024 04:32:12 +0000 (UTC) Received: from imap1.dmz-prg2.suse.org (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (No client certificate requested) by imap1.dmz-prg2.suse.org (Postfix) with ESMTPS id 9738313889; Thu, 4 Jul 2024 04:32:11 +0000 (UTC) Received: from dovecot-director2.suse.de ([2a07:de40:b281:106:10:150:64:167]) by imap1.dmz-prg2.suse.org with ESMTPSA id gC+BH8slhmbnDAAAD6G6ig (envelope-from ); Thu, 04 Jul 2024 04:32:11 +0000 From: Oscar Salvador To: Andrew Morton Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Peter Xu , Muchun Song , David Hildenbrand , SeongJae Park , Miaohe Lin , Michal Hocko , Matthew Wilcox , Christophe Leroy , Oscar Salvador Subject: [PATCH 18/45] fs/proc: Create pagemap_scan_pud_entry to handle PUD-mapped hugetlb vmas Date: Thu, 4 Jul 2024 06:31:05 +0200 Message-ID: <20240704043132.28501-19-osalvador@suse.de> X-Mailer: git-send-email 2.44.0 In-Reply-To: <20240704043132.28501-1-osalvador@suse.de> References: <20240704043132.28501-1-osalvador@suse.de> MIME-Version: 1.0 X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Rspamd-Action: no action X-Stat-Signature: ejybwg1t6hghzdri5toja6w9uay5r6bd X-Rspam-User: X-Rspamd-Queue-Id: 02EF218001C X-Rspamd-Server: rspam02 X-HE-Tag: 1720067533-271832 X-HE-Meta: U2FsdGVkX18XncctQ9mWTHbCtWV87+UwxSoQD4NBY18xMwD8oyDMK5dIciddqFOOcfAtn2dHTQu/2lmdBPEq9mwHNMqjXbCT6ruUdLGQvhY9VcKW2LmDfvAlHrD/NOidLOPkBu2hBriAKtSnagYEU1ox26SkbRfYutTOUHASMTYndQI1UjZEnB/FEQirW7ftmIkltt/QLvA6IwfPKKB/iPd5NkCKmeJRcmiClOcS5zPrU5VqJHQ1/r31yyUES2wfpPMkYgmJZcavcBo4kThs2UCGlhlV8LtxJqLa6FIRKPHlhOfs8DjC2xY0Uztg9FhuqzipOhuGg1AL13KZhrXV832+2htpz3GaUWIaRfRmYHbcg3MPvYYSRpr4hW3J1akMMja4r02W3Dsf2cpKEEIHg/6EUGacB8xjf8LYKx12ynXz0hhIkeo34rJKRNBcd4w0W0vBJ+bgHE4d0m3U3uJJ9ydH6JsLmqTRioeTBZ/i16Hh1T0IN4o9asBBq9zaHUF+vbjzqnbRDS3RULasPDt2QDMFrqoq3aeYHkiO3E2kqBOvtY+xEWLDVUhb9PPzNV2sXNBL3wIA+HudfI6uukDlpPs0CVT+NaDLE6Xf4N4usmZnGzQefHDQZ5LOW/LYxPOo58lst0LhefxBrvVV+/E+aI/udcyipLnALJop4upSG2NgUMm0wHVQlwdl9/guYaMK8ALqmp392W28LidB5I4lJ8PbpgQBJabGaRmlmiBDR3nRenZEdxkEsM0GchZwOQo+U6cI+911m8gMRH/biSdzVcw8Uk3o5GHPo+ukO35vkMdOjgXrHmtniJ8G5TG8wiN5MWHFvot8PtA/gh2AoDHZiAxbtkNdNgwh/39i2El4gL9ORaQdDLAAXM9ounaAvkuw180mq9LL7tgZlr8IQd0FLpAzMq6Bgcg/ZyYgvkYXy/4TYmzshwG0cSWrEv9WUs+n+HbBY5vPS316Oe1r83F DFJZnq9i dMOzQlYrgJ5ly4wYbkvDz+9/AyowVL2B7bFzusTIS9Za7FT1maan447Uxgs895/BWZ5ZQqvkr7GGRaLc69FuLqR4rpU3rjoK2Is5KTYvvSw+j0aqaah3S3P/yZCweD4S99Pv/J6YsxiUghn6R1pLGxi6oomb3q6dmSbM4V4s/9Etay9E= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Normal THP cannot be PUD-mapped (besides devmap) but hugetlb can, so create pagemap_scan_pud_entry in order to handle PUD-mapped hugetlb vmas. Signed-off-by: Oscar Salvador --- fs/proc/task_mmu.c | 104 ++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 103 insertions(+), 1 deletion(-) diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index df649f69ea2c..3785a44b97fa 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -1925,7 +1925,7 @@ static int pagemap_pmd_range(pmd_t *pmdp, unsigned long addr, unsigned long end, return err; } -#ifdef CONFIG_HUGETLB_PAGE +#ifdef CONFIG_PGTABLE_HAS_HUGE_LEAVES static int pagemap_pud_range(pud_t *pudp, unsigned long addr, unsigned long end, struct mm_walk *walk) { @@ -2324,6 +2324,59 @@ static void make_uffd_wp_pmd(struct vm_area_struct *vma, } #endif /* CONFIG_PGTABLE_HAS_HUGE_LEAVES */ +#ifdef CONFIG_PGTABLE_HAS_HUGE_LEAVES +static unsigned long pagemap_pud_category(struct pagemap_scan_private *p, + struct vm_area_struct *vma, + unsigned long addr, pud_t pud) +{ + unsigned long categories = PAGE_IS_HUGE; + + if (pud_present(pud)) { + struct page *page; + + categories |= PAGE_IS_PRESENT; + if (!pud_uffd_wp(pud)) + categories |= PAGE_IS_WRITTEN; + + if (p->masks_of_interest & PAGE_IS_FILE) { + page = vm_normal_page_pud(vma, addr, pud); + if (page && !PageAnon(page)) + categories |= PAGE_IS_FILE; + } + + if (is_zero_pfn(pud_pfn(pud))) + categories |= PAGE_IS_PFNZERO; + if (pud_soft_dirty(pud)) + categories |= PAGE_IS_SOFT_DIRTY; + } else if (is_swap_pud(pud)) { + swp_entry_t swp; + + categories |= PAGE_IS_SWAPPED; + if (!pud_swp_uffd_wp(pud)) + categories |= PAGE_IS_WRITTEN; + if (pud_swp_soft_dirty(pud)) + categories |= PAGE_IS_SOFT_DIRTY; + } + + return categories; +} + +static void make_uffd_wp_pud(struct vm_area_struct *vma, + unsigned long addr, pud_t *pudp) +{ + pud_t old, pud = *pudp; + + if (pud_present(pud)) { + old = pudp_invalidate_ad(vma, addr, pudp); + pud = pud_mkuffd_wp(old); + set_pud_at(vma->vm_mm, addr, pudp, pud); + } else if (is_migration_entry(pud_to_swp_entry(pud))) { + pud = pud_swp_mkuffd_wp(pud); + set_pud_at(vma->vm_mm, addr, pudp, pud); + } +} +#endif /* CONFIG_PGTABLE_HAS_HUGE_LEAVES */ + #ifdef CONFIG_HUGETLB_PAGE static unsigned long pagemap_hugetlb_category(pte_t pte) { @@ -2685,6 +2738,54 @@ static int pagemap_scan_pmd_entry(pmd_t *pmd, unsigned long start, return ret; } +#ifdef CONFIG_HUGETLB_PAGE +static int pagemap_scan_pud_entry(pud_t *pud, unsigned long start, + unsigned long end, struct mm_walk *walk) +{ + int ret = 0; + spinlock_t *ptl; + unsigned long categories; + struct vm_area_struct *vma = walk->vma; + struct pagemap_scan_private *p = walk->private; + + /* Only PUD-mapped hugetlb can reach here at this moment */ + ptl = pud_huge_lock(pud, vma); + if (!ptl) + return 0; + + categories = p->cur_vma_category | + pagemap_pud_category(p, vma, start, *pud); + + if (!pagemap_scan_is_interesting_page(categories, p)) + goto out_unlock; + + ret = pagemap_scan_output(categories, p, start, &end); + if (start == end) + goto out_unlock; + + if (~p->arg.flags & PM_SCAN_WP_MATCHING) + goto out_unlock; + if (~categories & PAGE_IS_WRITTEN) + goto out_unlock; + + if (end != start + PUD_SIZE) { + ret = 0; + pagemap_scan_backout_range(p, start, end); + p->arg.walk_end = start; + goto out_unlock; + } + + make_uffd_wp_pud(vma, start, pud); + flush_hugetlb_tlb_range(vma, start, end); + +out_unlock: + spin_unlock(ptl); + return ret; +} +#else +#define pagemap_scan_pud_entry NULL +#endif + #ifdef CONFIG_HUGETLB_PAGE static int pagemap_scan_hugetlb_entry(pte_t *ptep, unsigned long hmask, unsigned long start, unsigned long end, @@ -2772,6 +2873,7 @@ static int pagemap_scan_pte_hole(unsigned long addr, unsigned long end, static const struct mm_walk_ops pagemap_scan_ops = { .test_walk = pagemap_scan_test_walk, + .pud_entry = pagemap_scan_pud_entry, .pmd_entry = pagemap_scan_pmd_entry, .pte_hole = pagemap_scan_pte_hole, .hugetlb_entry = pagemap_scan_hugetlb_entry,