From patchwork Mon Jul 15 19:21:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13733839 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 851A8C3DA59 for ; Mon, 15 Jul 2024 19:22:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DE57D6B00A1; Mon, 15 Jul 2024 15:22:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D940E6B00A3; Mon, 15 Jul 2024 15:22:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B9C3A6B00A4; Mon, 15 Jul 2024 15:22:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 97C786B00A1 for ; Mon, 15 Jul 2024 15:22:10 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 3B246C01D7 for ; Mon, 15 Jul 2024 19:22:10 +0000 (UTC) X-FDA: 82342957620.08.1D1FEDC Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf25.hostedemail.com (Postfix) with ESMTP id 20D46A0020 for ; Mon, 15 Jul 2024 19:22:06 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=UI+hJyqU; spf=pass (imf25.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1721071307; a=rsa-sha256; cv=none; b=KUCtqh515ilWm2XQKSodY/bbCr54+ICKfc3jE0ovRCODS36W8VDMFhpKFEvVpHYz3kcjBZ +5S7be88e737lsQzmE1DqeNpCT7EcRYQn8qVjBQ186vZF7S5fDfKjIp36wPCtUP7CZmg2M N9UJWM4kD81uscOxE1gLRuW52p21Fqs= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=UI+hJyqU; spf=pass (imf25.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1721071307; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=MX5pdN1kDqdO60LUKursnvEFOQP0WFBlMnoZkGHINQQ=; b=BJspfY2tOQNnEvxLjxHartxEiVOgl8IIceIGdj/vg+xkOWNwRxHQBPkfxin1ALezmEm3Vc 08AgW3yDTd7Elre9GFDuC97lzSP3MZ7I9K+PDN2jUYe+GHn3OoKLRNlfFEMC/fsu+qj4gm SAmPOV/81HNoNChnXaYJHH0H9mFPytM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1721071326; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MX5pdN1kDqdO60LUKursnvEFOQP0WFBlMnoZkGHINQQ=; b=UI+hJyqUVVdqJoEJLwsMduIF4ljku5V6zHUvCiT05Vyc9jNV9FYpJdznGV2azGx+XzfNQs 1v498JyRoen0kqFKybyFRtk3GOuUmAeApBvUYs7WJNNYAM15B2PmX04Zu+/A9hnIm1nZXs BZHAcrt9Nqc0kow0MdUJEeTz02Mxy3Y= Received: from mail-qv1-f72.google.com (mail-qv1-f72.google.com [209.85.219.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-328-zaq30LGIMIC2uxjlzcFdBA-1; Mon, 15 Jul 2024 15:22:05 -0400 X-MC-Unique: zaq30LGIMIC2uxjlzcFdBA-1 Received: by mail-qv1-f72.google.com with SMTP id 6a1803df08f44-6b60afbf5d1so14885756d6.0 for ; Mon, 15 Jul 2024 12:22:05 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1721071323; x=1721676123; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MX5pdN1kDqdO60LUKursnvEFOQP0WFBlMnoZkGHINQQ=; b=NcmWHaIzmwHK633l/0++ZV7wn1wUYjTrsGfdI5CHYWj0ysZ7wD7sAxiOJXB0HBwDf0 n/x9Vai2/QSFpLsVIIFtAUFNpSCtsMclxl9ClXK4NgdID4RZANA/3d8Iep4/ZFu7Koo4 e3U3b0XKx6YN7hucaCTqRHvpu5KfEJCglUYCUsENmmfpoPPtquw9+Ekekm/YIAKNCf5C xs3bZx2rLJAmEkAjz8RTglwjh1T4sNEcGJ6C0EMYd8Qrz2pKMYW40SejayoghLiyEITx cWZNXbl/hSIM6Hip2Uu4oCwruRxssDvJOjNubwwmOPAmkP8BIYulAkEU31qvbtD5/8Pa +UtA== X-Gm-Message-State: AOJu0YyQ1jo67PR3JaZzaC1iEM43hsn+bZDn/a34JGojJ5oH6O4pYAdU vv4F0Thr9QYN5arI/a/fKnML/splF4CZ1bgyQbV+UobYfhDHdqmJzGO05a8fYoE/3ILDp+0uOrs aCzSN62sNcpIlAXgNsSbMz/C0j7L+P8rOwwBRxZNYj43TPTRjIpyaSXL5gKgpa2eYH+dEE6Mf6C R9IStRVmwlN40Ik7R7bF9LdmSaHqv+RA== X-Received: by 2002:a05:6214:3d9c:b0:6b7:586c:6db with SMTP id 6a1803df08f44-6b77e1a9bbbmr4347926d6.9.1721071323016; Mon, 15 Jul 2024 12:22:03 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGQhnzZI26fHLI1GP/caKDLHgduuPT+eLqK9zXdARycJ8bkdfse6d6/hJ9cmtHwcRv8UZEfEQ== X-Received: by 2002:a05:6214:3d9c:b0:6b7:586c:6db with SMTP id 6a1803df08f44-6b77e1a9bbbmr4347496d6.9.1721071322604; Mon, 15 Jul 2024 12:22:02 -0700 (PDT) Received: from x1n.redhat.com (pool-99-254-121-117.cpe.net.cable.rogers.com. [99.254.121.117]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6b761978d30sm24039356d6.31.2024.07.15.12.22.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 15 Jul 2024 12:22:02 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Dave Jiang , Rik van Riel , Dave Hansen , Michael Ellerman , linuxppc-dev@lists.ozlabs.org, Matthew Wilcox , Rick P Edgecombe , peterx@redhat.com, Oscar Salvador , Mel Gorman , Andrew Morton , Borislav Petkov , Christophe Leroy , Huang Ying , "Kirill A . Shutemov" , "Aneesh Kumar K . V" , Dan Williams , Thomas Gleixner , Hugh Dickins , x86@kernel.org, Nicholas Piggin , Vlastimil Babka , Ingo Molnar Subject: [PATCH v3 8/8] mm/mprotect: fix dax pud handlings Date: Mon, 15 Jul 2024 15:21:42 -0400 Message-ID: <20240715192142.3241557-9-peterx@redhat.com> X-Mailer: git-send-email 2.45.0 In-Reply-To: <20240715192142.3241557-1-peterx@redhat.com> References: <20240715192142.3241557-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Stat-Signature: m5wpuoydjc55tif31pkxn7txocorhdpw X-Rspamd-Queue-Id: 20D46A0020 X-Rspam-User: X-Rspamd-Server: rspam10 X-HE-Tag: 1721071326-671807 X-HE-Meta: U2FsdGVkX1/zMB+rPA5GeG3I4dI+uZtf4VkrBqjbCFXvFPA4hV5rpyjf6Ekz+FBmTZmVig0uEDAbemidYjZRrJSSCYELeU4YYfBIgsWc71FykONVdK+FrEgZ1c3oukraCD9WDHhduWTTE3H/JXFyT+bUnyxiRoXgYbaiJLCiqXXWG1NE2CQC/nMRydfW6XRgTEr1etE1hLdlpo5l/jigiolHX/a3MYNIblzizsAlCr0Ht2DVGBCfpJP9YANYkPriwZbdKMKkEN7ZocQ/GW+5NfFigeCF8FfZgNjqGNUp2PZ3KVeh54ZZkJkNJa+awCYlUsb33a7H7lCtPm6rRRCrgvGFkgiZBIm8LgLXZBY1FV+GSRMPtiX3o7bexTgTU27kOqsFfQck+iYRT4OMERavXl2O2pME8O3C8uADU1U86Fo4frBElGhwHnlCykfOzz1CK1KrYYvQmXpVziNZxGv0rDeJoYsxFYemVkMrahspIrLCNQpW607T68PKY8Gv99NHVSfoGyzlTeY8ANkqNPX4gctL9uh1EqJH70AhCevEUyRtV7tASvgEmG4kaWZptrsSGba7XOBXSoV/5TflzuCjMOptiINiiGMXntHqso2Xq08coiFB9HADlipgA4FZA47z/61/ZD365cNPkXNoy/5sMJOcoFdbFroK+HsYbkZIu6wH1Tsrf35MdJzRfE6Q2OkXXqxZ1vzsJexlBYM6nGB5cQfYjKrCO+m+jwk1QEkCWrFIIU1+cunAKx/s9lD0GLbYQKjpXS2K+Jbdk44wYiHUqlhX3AcvhE9molmFnQ/4Tg78st9lNix2db6+t815NxokBZicqYpppHWeq+VPCM8NE/TMHCJh+RQ8KJBBRIhUTH8jsadrZIuErD8oWSxByYAlyKEQmOoyAg6igIPlB1VYPn0rP0aWQoMB62mq7nlY1nJKDz/yOWFHTeivZ74xQx2OmFuNLmM76OuvtIexPv+ oX1Spl4R NoDRsdwR7GW8NH5aKWyz4ClXExjgRVYRdwW6E9M2nd7MtORZqPyzi+wzzDzK/o507N+MnQPJutIesFDtOah1uRV8GJd7TIzFtg5sG5iBU3OU5X5BhJfzi3svvrId/c6KagoH/E4t25VTbjvYvzsPFi2af2/8bh7NN89k/U2hUfah19zCnwBwlfnsiIeiiOvnG7TLtb98oehnIxf2NsPr6tpLk8nKlJwSKdb4UHinyJmhYMZjJWYhmzASmMgFBlvvQlQoRc7O0ZYfZV8CJzXk9utNd7UjJ8hFyP78+jnKj1jawhpbodkKYb/j93SkhFk5Gtkv8oLKiZEF7t111JtrXwYD+AcEWoEKJxDkhDi2O7WBOdDweLWBWXfToMRqenGnrwiQvM2lk199CeD707Zf4mgGMKpnJgK5qt3Amad7SXALSNKZG3j9QfowrkMeAlG42MUBUGmjxiTO74MLS5nlrW+I/0AUFtZ6d3wzkpuGdrMvORE3eKKBdb6NG/d1VzoI1Lc+2ii1qK8/1P9CD4Ozjrv92wkii8Nb+HTwSCqXaOkIStr2Y+fwmWIDHNgOw8TK4d397rYhLALfLvhX3dc+hXzzCZ0iTs9MRYHbu/3V836NKgQ+lM3KVnjaGPX0ZzkaAmNuN X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: This is only relevant to the two archs that support PUD dax, aka, x86_64 and ppc64. PUD THPs do not yet exist elsewhere, and hugetlb PUDs do not count in this case. DAX have had PUD mappings for years, but change protection path never worked. When the path is triggered in any form (a simple test program would be: call mprotect() on a 1G dev_dax mapping), the kernel will report "bad pud". This patch should fix that. The new change_huge_pud() tries to keep everything simple. For example, it doesn't optimize write bit as that will need even more PUD helpers. It's not too bad anyway to have one more write fault in the worst case once for 1G range; may be a bigger thing for each PAGE_SIZE, though. Neither does it support userfault-wp bits, as there isn't such PUD mappings that is supported; file mappings always need a split there. The same to TLB shootdown: the pmd path (which was for x86 only) has the trick of using _ad() version of pmdp_invalidate*() which can avoid one redundant TLB, but let's also leave that for later. Again, the larger the mapping, the smaller of such effect. Another thing worth mention is this path needs to be careful on handling "retry" event for change_huge_pud() (where it can return 0): it isn't like change_huge_pmd(), as the pmd version is safe with all conditions handled in change_pte_range() later, thanks to Hugh's new pte_offset_map_lock(). In short, change_pte_range() is simply smarter than change_pmd_range() now after the shmem thp collapse rework. For that reason, change_pud_range() will need proper retry if it races with something else when a huge PUD changed from under us. Cc: Dan Williams Cc: Matthew Wilcox Cc: Dave Jiang Cc: Hugh Dickins Cc: Kirill A. Shutemov Cc: Vlastimil Babka Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Dave Hansen Cc: Michael Ellerman Cc: Aneesh Kumar K.V Cc: Oscar Salvador Cc: x86@kernel.org Cc: linuxppc-dev@lists.ozlabs.org Fixes: a00cc7d9dd93 ("mm, x86: add support for PUD-sized transparent hugepages") Fixes: 27af67f35631 ("powerpc/book3s64/mm: enable transparent pud hugepage") Signed-off-by: Peter Xu --- include/linux/huge_mm.h | 24 +++++++++++++++++++ mm/huge_memory.c | 52 +++++++++++++++++++++++++++++++++++++++++ mm/mprotect.c | 34 ++++++++++++++++++++++----- 3 files changed, 104 insertions(+), 6 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index cff002be83eb..6e742680590a 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -336,6 +336,17 @@ void split_huge_pmd_address(struct vm_area_struct *vma, unsigned long address, void __split_huge_pud(struct vm_area_struct *vma, pud_t *pud, unsigned long address); +#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD +int change_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma, + pud_t *pudp, unsigned long addr, pgprot_t newprot, + unsigned long cp_flags); +#else +static inline int +change_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma, + pud_t *pudp, unsigned long addr, pgprot_t newprot, + unsigned long cp_flags) { return 0; } +#endif + #define split_huge_pud(__vma, __pud, __address) \ do { \ pud_t *____pud = (__pud); \ @@ -579,6 +590,19 @@ static inline int next_order(unsigned long *orders, int prev) { return 0; } + +static inline void __split_huge_pud(struct vm_area_struct *vma, pud_t *pud, + unsigned long address) +{ +} + +static inline int change_huge_pud(struct mmu_gather *tlb, + struct vm_area_struct *vma, pud_t *pudp, + unsigned long addr, pgprot_t newprot, + unsigned long cp_flags) +{ + return 0; +} #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ static inline int split_folio_to_list_to_order(struct folio *folio, diff --git a/mm/huge_memory.c b/mm/huge_memory.c index c10247bef08a..9a00c5955c0c 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2112,6 +2112,53 @@ int change_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, return ret; } +/* + * Returns: + * + * - 0: if pud leaf changed from under us + * - 1: if pud can be skipped + * - HPAGE_PUD_NR: if pud was successfully processed + */ +#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD +int change_huge_pud(struct mmu_gather *tlb, struct vm_area_struct *vma, + pud_t *pudp, unsigned long addr, pgprot_t newprot, + unsigned long cp_flags) +{ + struct mm_struct *mm = vma->vm_mm; + pud_t oldpud, entry; + spinlock_t *ptl; + + tlb_change_page_size(tlb, HPAGE_PUD_SIZE); + + /* NUMA balancing doesn't apply to dax */ + if (cp_flags & MM_CP_PROT_NUMA) + return 1; + + /* + * Huge entries on userfault-wp only works with anonymous, while we + * don't have anonymous PUDs yet. + */ + if (WARN_ON_ONCE(cp_flags & MM_CP_UFFD_WP_ALL)) + return 1; + + ptl = __pud_trans_huge_lock(pudp, vma); + if (!ptl) + return 0; + + /* + * Can't clear PUD or it can race with concurrent zapping. See + * change_huge_pmd(). + */ + oldpud = pudp_invalidate(vma, addr, pudp); + entry = pud_modify(oldpud, newprot); + set_pud_at(mm, addr, pudp, entry); + tlb_flush_pud_range(tlb, addr, HPAGE_PUD_SIZE); + + spin_unlock(ptl); + return HPAGE_PUD_NR; +} +#endif + #ifdef CONFIG_USERFAULTFD /* * The PT lock for src_pmd and dst_vma/src_vma (for reading) are locked by @@ -2342,6 +2389,11 @@ void __split_huge_pud(struct vm_area_struct *vma, pud_t *pud, spin_unlock(ptl); mmu_notifier_invalidate_range_end(&range); } +#else +void __split_huge_pud(struct vm_area_struct *vma, pud_t *pud, + unsigned long address) +{ +} #endif /* CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD */ static void __split_huge_zero_page_pmd(struct vm_area_struct *vma, diff --git a/mm/mprotect.c b/mm/mprotect.c index 2a81060b603d..694f13b83864 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -425,31 +425,53 @@ static inline long change_pud_range(struct mmu_gather *tlb, unsigned long end, pgprot_t newprot, unsigned long cp_flags) { struct mmu_notifier_range range; - pud_t *pud; + pud_t *pudp, pud; unsigned long next; long pages = 0, ret; range.start = 0; - pud = pud_offset(p4d, addr); + pudp = pud_offset(p4d, addr); do { +again: next = pud_addr_end(addr, end); - ret = change_prepare(vma, pud, pmd, addr, cp_flags); + ret = change_prepare(vma, pudp, pmd, addr, cp_flags); if (ret) { pages = ret; break; } - if (pud_none_or_clear_bad(pud)) + + pud = READ_ONCE(*pudp); + if (pud_none(pud)) continue; + if (!range.start) { mmu_notifier_range_init(&range, MMU_NOTIFY_PROTECTION_VMA, 0, vma->vm_mm, addr, end); mmu_notifier_invalidate_range_start(&range); } - pages += change_pmd_range(tlb, vma, pud, addr, next, newprot, + + if (pud_leaf(pud)) { + if ((next - addr != PUD_SIZE) || + pgtable_split_needed(vma, cp_flags)) { + __split_huge_pud(vma, pudp, addr); + goto again; + } else { + ret = change_huge_pud(tlb, vma, pudp, + addr, newprot, cp_flags); + if (ret == 0) + goto again; + /* huge pud was handled */ + if (ret == HPAGE_PUD_NR) + pages += HPAGE_PUD_NR; + continue; + } + } + + pages += change_pmd_range(tlb, vma, pudp, addr, next, newprot, cp_flags); - } while (pud++, addr = next, addr != end); + } while (pudp++, addr = next, addr != end); if (range.start) mmu_notifier_invalidate_range_end(&range);