From patchwork Mon May 22 04:49:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13249751 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2C07CC77B73 for ; Mon, 22 May 2023 04:49:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 769EE900004; Mon, 22 May 2023 00:49:52 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 719DC900002; Mon, 22 May 2023 00:49:52 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5E224900004; Mon, 22 May 2023 00:49:52 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 4E8C0900002 for ; Mon, 22 May 2023 00:49:52 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 796CB1A05C5 for ; Mon, 22 May 2023 04:49:51 +0000 (UTC) X-FDA: 80816663382.02.88BFC6F Received: from mail-yb1-f174.google.com (mail-yb1-f174.google.com [209.85.219.174]) by imf17.hostedemail.com (Postfix) with ESMTP id A3CB340012 for ; Mon, 22 May 2023 04:49:49 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=jjqeukn2; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf17.hostedemail.com: domain of hughd@google.com designates 209.85.219.174 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684730989; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=aPDEzBR49R6dDlw5Ac4B9CyjOOF4I7QDHwXmKNSpbHY=; b=xhmSTAyzPMeCN9corixMGHPcQc+ITQJXLoKiV3U9HRwKjmnTE8QwxVjQGQSB7zrjNyHwDb RxZZXtisRrw1gRv3iBJ1B7dMDqNCzUfspBgiiQA1ZM2VQCu13MVzRZFcIlcxrgoRFED6i2 qZhwimNPvU7jKHwtfgn0/nPjETCMucU= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=jjqeukn2; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf17.hostedemail.com: domain of hughd@google.com designates 209.85.219.174 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684730989; a=rsa-sha256; cv=none; b=S+MxnaJv6gY5J1zH68QYO8goPqXkLkVxpmkdxuqLLWmeoN/N4kIdusrXoGz52nsQcjWmjA I9rA8+EZ/LwHgOO00ZV8MmiDHsT3UI6ujbOFy46S1Qyyge9zkzExG40aT6Dj5R5/tIWlLK CpQ+yO/OEg2iwNb9V5OEq33+06qxulg= Received: by mail-yb1-f174.google.com with SMTP id 3f1490d57ef6-b9a7e639656so10756331276.0 for ; Sun, 21 May 2023 21:49:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1684730989; x=1687322989; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=aPDEzBR49R6dDlw5Ac4B9CyjOOF4I7QDHwXmKNSpbHY=; b=jjqeukn26805nHVLC5nBwMgzM4/Jmq9nujdcXAsHlTTa0MF+m2phch6zZaIztXO6Dz fns/H0GYdXEsi480XLUtkShE3/SCYxhfBiTmi4a8mWlPCmy6IGq/VUwbfbhJIIoyZO7w TE3oj47MvYeZYWdEl1lce5yvexgSm//EOtHJ9GZeQN0nTG0k7iqmz6ivXvrQT6/vTILo r9ZBlahSUgKAO7KElSgVquCd8eytGkW3ohhN44RmnSrwXqOowlkCJVfPWiZAiA+rPaIg Hemws7mTvH+9rnEvzhcxuWU8gWDXFk8QvfENWQsXDu8/h+u8uj+2BFe5jHCcgno98BaD /dgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684730989; x=1687322989; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=aPDEzBR49R6dDlw5Ac4B9CyjOOF4I7QDHwXmKNSpbHY=; b=WpTfJ2b2j2zdxHczFfBlWtBStvfRgeRHYcFfvJjF/zmxejFmuudz7pM23BJYZZojhq nQcJFjTh/7eRuvyZZ4l81Evar0F3px8PLedFSEKg9+X8QLq4OoUsnAYWLBdyKHy4HObL /iY8wyfzrXaj4TwWzvbttIlDJRwfxiPaVuoahbdCgEL8bBHB7BCoDG4RJpQcJijIpuyo LMUJ54gD+u6BnjGWlWZRsvKgJJzlRdW25kBDfdx743wRjcxb8T1OlM8iBMQMpOAf6ggq 2qrc80tN15afe0zkWIVQZRm8Owh+wiw6UukeyzmuZYmmSkskl25lAYmVBcO2DPDIDDZb 3Brw== X-Gm-Message-State: AC+VfDx22q/yreHHl1uADSMbMfhy/lF3oytlV5lphIdgiTTGpHWjN5Vu /hztO8I/vGHLCAvb45Q+hdzNCg== X-Google-Smtp-Source: ACHHUZ5ZcQphQAL7jjAMB8ykWMW22wF1U8iyf27N7XhXamyXhJP6vomdj/dc7m8k3wKxzVaO0p10GA== X-Received: by 2002:a25:d242:0:b0:b9a:66b7:673e with SMTP id j63-20020a25d242000000b00b9a66b7673emr11337327ybg.43.1684730988606; Sun, 21 May 2023 21:49:48 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id b206-20020a0dd9d7000000b00560f6704ee1sm1809847ywe.26.2023.05.21.21.49.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 21 May 2023 21:49:48 -0700 (PDT) Date: Sun, 21 May 2023 21:49:45 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 01/31] mm: use pmdp_get_lockless() without surplus barrier() In-Reply-To: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> Message-ID: <34467cca-58b6-3e64-1ee7-e3dc43257a@google.com> References: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> MIME-Version: 1.0 X-Rspam-User: X-Stat-Signature: eb6ej7s6ag93yxww9ut8q7gc8c1b3b96 X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: A3CB340012 X-HE-Tag: 1684730989-622857 X-HE-Meta: U2FsdGVkX1/IPyDhQQ7Nka5b7VJEeOiuR02qatzPjKtJ2vxbyLPs3QUwhDGCWA7X1t5rojv7FopSDGmGkdmYXqDcOA+UFyMofBx+/devFvL6b9AjlNeMJXCimCClyubKuBb4lA1PwzJy61twdpzWNkNAF92k1RocYmSOosQ3nMUcuEsoRCjywscLyX6LSrRdyc0Oyu09is/8oFveAkGXNN9/MvYQ5/4GSguI4s7s8JUuAF4octY4F48NMqk3EuGWIenvjJzpCMiOiwwpUEMJ0NkoYFAYfhWtOJ+OFru41S4nYbWvosTm1a4Kfpmmz4iA62KK/cUv28ERIju5HGzttoomerh4n5THvCYB3PSRH6VnamkNcmWL3yT6S7Km0hLeLwdZrayMCEq0ztRwLQKu52l0+pi5lYM1DUoT+UZ6Abf5mrM8ZO6jpm1omouYOeNvsHZzQsENXfmCbba5VF7+YR7WB9Qpn1yPLlJbmZpY9pNCUpDOthQdPru4xJb8VrNMQmKyE1oWUfuMA6ZuCELJQ79RF9DRSrns+yfv/rdEC+Wao+Mz/HYkwsa7W7PPgOGNjQr/xyagX+ILc3nUqXo8I7TK2tz15Jatg4JB9DXSN0K/jJ5eWclzEdRKB+3C/hDaOqpy0659sQw/om7A4yFWpVgSmZ4OoEU7qh1KVHhQdF9I1tcZBVM0SEz2zptkX/fJ8S+mBv4GQiqFIhFV4VKX2cjBP1UxriOoarLTkjZFHi+IkZLGkBMAH7vozOcAe56mO3imISaDwS4KL4XIfrYtZS24+yOF/imKUPHbulfkPicWoxAwhd2at6gZGTjEvq9XPf0cT1e2eD32ePLwONJdh64QLCIoEManq3+p1liKRb7FuC62JrvaWb9PRG4w5vaN2jEdd5ww0DD73d0/3TtIrjU1zyf+WnxzSIkcSAeE8EKrxty0o5fwH3AS+yA4pdkS6s10f0RtvyvYEHYbbOZ 77YmAMY2 mMoOmJJD0CU/y6ac5LAN7nikIFpTkAl+447vjnkWp/qyNAo9VhG6whhytsslDLi6a6Fvz42AKA2sKwauAPipynLSjrYy6NDXna3z8ysB3AowLBcRZ6n36uQNyvdzMk82FnMeT02hu02di1kInuxCkd0KTU1B9IdsSZ5xNkq+YqhvsvEYOXG0EDeFZ53KPEgdguuBoll7BM14QGqT5iKd0IMtrlcrllX5gjwIX3nHIjKUK9xqd46SjLszJM28KTS44qrChJNiOOcQfPrOv5/Ho0McKMVFvvdpDmyDOjseKs2690AL1Aoy7XQc4ZvX04KAKaiL7r2taAySCUXXUq/RdHTox6rdtfV/7ez3l6yiwo8pRzgx9P0WFs47CPqhjl7B88h+HOISXzYujgJ7BALLpRzVwpwIBiTUUpi2+u/rMh4ZcKL/ux8vpOiOi5w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use pmdp_get_lockless() in preference to READ_ONCE(*pmdp), to get a more reliable result with PAE (or READ_ONCE as before without PAE); and remove the unnecessary extra barrier()s which got left behind in its callers. HOWEVER: Note the small print in linux/pgtable.h, where it was designed specifically for fast GUP, and depends on interrupts being disabled for its full guarantee: most callers which have been added (here and before) do NOT have interrupts disabled, so there is still some need for caution. Signed-off-by: Hugh Dickins Acked-by: Yu Zhao Acked-by: Peter Xu --- fs/userfaultfd.c | 10 +--------- include/linux/pgtable.h | 17 ----------------- mm/gup.c | 6 +----- mm/hmm.c | 2 +- mm/khugepaged.c | 5 ----- mm/ksm.c | 3 +-- mm/memory.c | 14 ++------------ mm/mprotect.c | 5 ----- mm/page_vma_mapped.c | 2 +- 9 files changed, 7 insertions(+), 57 deletions(-) diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c index 0fd96d6e39ce..f7a0817b1ec0 100644 --- a/fs/userfaultfd.c +++ b/fs/userfaultfd.c @@ -349,15 +349,7 @@ static inline bool userfaultfd_must_wait(struct userfaultfd_ctx *ctx, if (!pud_present(*pud)) goto out; pmd = pmd_offset(pud, address); - /* - * READ_ONCE must function as a barrier with narrower scope - * and it must be equivalent to: - * _pmd = *pmd; barrier(); - * - * This is to deal with the instability (as in - * pmd_trans_unstable) of the pmd. - */ - _pmd = READ_ONCE(*pmd); + _pmd = pmdp_get_lockless(pmd); if (pmd_none(_pmd)) goto out; diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index c5a51481bbb9..8ec27fe69dc8 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -1344,23 +1344,6 @@ static inline int pud_trans_unstable(pud_t *pud) static inline int pmd_none_or_trans_huge_or_clear_bad(pmd_t *pmd) { pmd_t pmdval = pmdp_get_lockless(pmd); - /* - * The barrier will stabilize the pmdval in a register or on - * the stack so that it will stop changing under the code. - * - * When CONFIG_TRANSPARENT_HUGEPAGE=y on x86 32bit PAE, - * pmdp_get_lockless is allowed to return a not atomic pmdval - * (for example pointing to an hugepage that has never been - * mapped in the pmd). The below checks will only care about - * the low part of the pmd with 32bit PAE x86 anyway, with the - * exception of pmd_none(). So the important thing is that if - * the low part of the pmd is found null, the high part will - * be also null or the pmd_none() check below would be - * confused. - */ -#ifdef CONFIG_TRANSPARENT_HUGEPAGE - barrier(); -#endif /* * !pmd_present() checks for pmd migration entries * diff --git a/mm/gup.c b/mm/gup.c index bbe416236593..3bd5d3854c51 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -653,11 +653,7 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma, struct mm_struct *mm = vma->vm_mm; pmd = pmd_offset(pudp, address); - /* - * The READ_ONCE() will stabilize the pmdval in a register or - * on the stack so that it will stop changing under the code. - */ - pmdval = READ_ONCE(*pmd); + pmdval = pmdp_get_lockless(pmd); if (pmd_none(pmdval)) return no_page_table(vma, flags); if (!pmd_present(pmdval)) diff --git a/mm/hmm.c b/mm/hmm.c index 6a151c09de5e..e23043345615 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -332,7 +332,7 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp, pmd_t pmd; again: - pmd = READ_ONCE(*pmdp); + pmd = pmdp_get_lockless(pmdp); if (pmd_none(pmd)) return hmm_vma_walk_hole(start, end, -1, walk); diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 6b9d39d65b73..732f9ac393fc 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -961,11 +961,6 @@ static int find_pmd_or_thp_or_none(struct mm_struct *mm, return SCAN_PMD_NULL; pmde = pmdp_get_lockless(*pmd); - -#ifdef CONFIG_TRANSPARENT_HUGEPAGE - /* See comments in pmd_none_or_trans_huge_or_clear_bad() */ - barrier(); -#endif if (pmd_none(pmde)) return SCAN_PMD_NONE; if (!pmd_present(pmde)) diff --git a/mm/ksm.c b/mm/ksm.c index 0156bded3a66..df2aa281d49d 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -1194,8 +1194,7 @@ static int replace_page(struct vm_area_struct *vma, struct page *page, * without holding anon_vma lock for write. So when looking for a * genuine pmde (in which to find pte), test present and !THP together. */ - pmde = *pmd; - barrier(); + pmde = pmdp_get_lockless(pmd); if (!pmd_present(pmde) || pmd_trans_huge(pmde)) goto out; diff --git a/mm/memory.c b/mm/memory.c index f69fbc251198..2eb54c0d5d3c 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -4925,18 +4925,9 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) * So now it's safe to run pte_offset_map(). */ vmf->pte = pte_offset_map(vmf->pmd, vmf->address); - vmf->orig_pte = *vmf->pte; + vmf->orig_pte = ptep_get_lockless(vmf->pte); vmf->flags |= FAULT_FLAG_ORIG_PTE_VALID; - /* - * some architectures can have larger ptes than wordsize, - * e.g.ppc44x-defconfig has CONFIG_PTE_64BIT=y and - * CONFIG_32BIT=y, so READ_ONCE cannot guarantee atomic - * accesses. The code below just needs a consistent view - * for the ifs and we later double check anyway with the - * ptl lock held. So here a barrier will do. - */ - barrier(); if (pte_none(vmf->orig_pte)) { pte_unmap(vmf->pte); vmf->pte = NULL; @@ -5060,9 +5051,8 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, if (!(ret & VM_FAULT_FALLBACK)) return ret; } else { - vmf.orig_pmd = *vmf.pmd; + vmf.orig_pmd = pmdp_get_lockless(vmf.pmd); - barrier(); if (unlikely(is_swap_pmd(vmf.orig_pmd))) { VM_BUG_ON(thp_migration_supported() && !is_pmd_migration_entry(vmf.orig_pmd)); diff --git a/mm/mprotect.c b/mm/mprotect.c index 92d3d3ca390a..c5a13c0f1017 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -309,11 +309,6 @@ static inline int pmd_none_or_clear_bad_unless_trans_huge(pmd_t *pmd) { pmd_t pmdval = pmdp_get_lockless(pmd); - /* See pmd_none_or_trans_huge_or_clear_bad for info on barrier */ -#ifdef CONFIG_TRANSPARENT_HUGEPAGE - barrier(); -#endif - if (pmd_none(pmdval)) return 1; if (pmd_trans_huge(pmdval)) diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index 4e448cfbc6ef..64aff6718bdb 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -210,7 +210,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) * compiler and used as a stale value after we've observed a * subsequent update. */ - pmde = READ_ONCE(*pvmw->pmd); + pmde = pmdp_get_lockless(pvmw->pmd); if (pmd_trans_huge(pmde) || is_pmd_migration_entry(pmde) || (pmd_present(pmde) && pmd_devmap(pmde))) { From patchwork Mon May 22 04:51:00 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13249752 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 11EA2C7EE23 for ; Mon, 22 May 2023 04:51:08 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A2894900005; Mon, 22 May 2023 00:51:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9D964900002; Mon, 22 May 2023 00:51:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8A16C900005; Mon, 22 May 2023 00:51:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 79E47900002 for ; Mon, 22 May 2023 00:51:07 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 420FA4043D for ; Mon, 22 May 2023 04:51:07 +0000 (UTC) X-FDA: 80816666574.16.3501D4E Received: from mail-yb1-f174.google.com (mail-yb1-f174.google.com [209.85.219.174]) by imf07.hostedemail.com (Postfix) with ESMTP id 5A8A44000C for ; Mon, 22 May 2023 04:51:05 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="suZ7/fWy"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf07.hostedemail.com: domain of hughd@google.com designates 209.85.219.174 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684731065; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ANcMmc9BeTTDRJseGPHLnYq8DMAivc7OJ7zKS5AkOEU=; b=367zwI9YXnirYuV6xh70NEscmeezPHUwhMFRM6jMdN8KEyxZAarGifAPPj4zH/J+ub4Ful uNJE6F8ZiYnOhdXUqDTnrK/n+m/0FyTp6mw76Una36RWwIbfhWlAlJ+Jkyz2E7xQ2hhgOE pXvCKpmXYxprzWyRQFfB8K4AV/lmPd0= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="suZ7/fWy"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf07.hostedemail.com: domain of hughd@google.com designates 209.85.219.174 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684731065; a=rsa-sha256; cv=none; b=ocl5QYDeioc9ftkcMMYuUZg0oFggmsjW9+DgcyGzf3bDQl3ePnfbqWijEHaTZZT6unfW6E WsygjmnqC+ZwSPgvayoueBOLwak+woUhfgW0e7YfGFo28En6nbUeAgk3ut0UORPwXnGKB4 zcSZxBArQI2mkpWRFJBk/ScB+VHBh94= Received: by mail-yb1-f174.google.com with SMTP id 3f1490d57ef6-ba8afcc82c0so8138705276.2 for ; Sun, 21 May 2023 21:51:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1684731064; x=1687323064; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=ANcMmc9BeTTDRJseGPHLnYq8DMAivc7OJ7zKS5AkOEU=; b=suZ7/fWyjgtDmmUUG368PhOQSIoqVmOnxBEGz1k9zCkGOxssorwZrFR60X+8a4DQqr 7u71fp8c7x68EiyyW8ZSDwzHkHiPONOn7yDOgmy63jiQBTyUNcqNciC9w+0U347zF4Gf qZEb5FQPLn1+cE60AE8Q2hbQucuksdSO++m5CDyyBTN/iGnKQo4gRrU//els8iIiBdS7 ovYLm7Yj3qMDE0Qoy1tWZGPaThqr4eXUwvzaDxInjjwEGGfg2sUxlTqNWk6AaZWuz+i8 FTUjVygqWWxTs+smOosRSftvUI/03MEA0EZ3L2DHmfVjXa6bRJ1mwd0Lc0WFnRQmYQLX GTlA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684731064; x=1687323064; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ANcMmc9BeTTDRJseGPHLnYq8DMAivc7OJ7zKS5AkOEU=; b=Cl8s/txM+yn7DYmkGxSs/7t+BpEoi6uww1ak7B8SN7r0vb0cUEK/jcrOBIGGl/bAwd zFK1pHGCB3/mcpOZcczIxDVERE0hMenrW4ZNoVJpQrM6WWoRkLy2cCMZjKenydSljZ1a fGBnfvmpD53gO1pBo48bLeoJSeWZhJwAinSgesThmScDy55e4S8hHMcsN+ZUY+Hd60RK muEos2kfpka/uxHJBdl7su7qMGvYOrGLI2xF6tdaSpoKCnJg6fvKdJdrCm3OFCNeo51w 1v3P75CwHHfJMWoS5qwwAOXOZXuh7brdIuK33iGwJeRCQDv4hMa5v2ZX2nhICiMoDGTd HQfw== X-Gm-Message-State: AC+VfDwRvPk7jq6JJV6WnmG/9EQ0tCD8UUjMHfQxPDPjZk8UkEw6xeq1 oeEBbgdUJO3AbEL9swRp7cqH/A== X-Google-Smtp-Source: ACHHUZ6jO0XEKla2wlyWHkLbd6267BBIAu3WYemOi2dKQNM0AlHnDzPyGpw7fyb0X8v56kZQVrFU2A== X-Received: by 2002:a25:1042:0:b0:ba8:1807:9d7f with SMTP id 63-20020a251042000000b00ba818079d7fmr9116239ybq.58.1684731064262; Sun, 21 May 2023 21:51:04 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id a11-20020a81bb4b000000b0054e82b6a95esm1808098ywl.42.2023.05.21.21.51.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 21 May 2023 21:51:04 -0700 (PDT) Date: Sun, 21 May 2023 21:51:00 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 02/31] mm/migrate: remove cruft from migration_entry_wait()s In-Reply-To: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> Message-ID: <1659568-468a-6d36-c26-6a52a335ab59@google.com> References: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 5A8A44000C X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: j41h9ymcrtyih686in1umnb9mjjfh4qz X-HE-Tag: 1684731065-645011 X-HE-Meta: U2FsdGVkX1+S7AYFX9vBtFbDkQMvZ9lyNJ2IQxxIdUNGlEQbkwZOlf2ps2Czj8xb9BT1xWHRnt1pxTZ0+ez0JZSuHBLVBq0cQNC8702Tjl3EHJUEs/IdwW7lN97kO4r6t08HcXGsnC2m/GnsCUm/o/O6EPjwj1ROrz/N0ztjIloACy7IH8bLRCXO0JHYRBRFsjSQJVimGbQmv6Rd0jBMwIdsKyFWkoNEmy3CAtYOf95R+HZNXb/8cxOybgynbM5+S0h8oVIWxFDWeAqPVxgmL3jdNtq0K+BqIRR3TZor98d5wywDutoawD+XlzGB+67zMTI19da0wmGeaDbX+DDf6EpO66yDFPPkbLkgu91G8VE6YDVbSXLAonjb0dX2CffkqztFdZrsIDOLZFhRNTVz6VcsMmmgZU5TvVm9RT34CQ+8Gvkyg2igWNKwD1MutAcPi5BaMyPyGWff2JpeziRcTN3pFuM/nD4tCgWun2U+sd9zOP4N7UtEC7VvIrJFrxdEiM0GwY3F4qntpHBR8Oi/40P/LuzID/3xiI0xwC+E6ZLHhusQZM976XU5gPsFKM0QkkqJh6TT9GBGT38yyGNyz/JyxL7i1/DJmumCrHMeIBQkp9qJUUUy7sg2zA7/eW+ZM4zWM/cS1a/HHmXxrAaei+7rIKWJv2ZUHncaUfgOLL1GiXh8qkyo6gK4Rc45NiKKFrg3jv4eNGKRC0ljlWAT/a9yMcuiMLeGP9yB6wrygi0HqZK5X09RdrIonE9SCKho116ZS5Nh+mJ3NG9+3LlQ8OFMYzVxx8MeroQ7ETwnRpj6bg6m/EfRtAQBc6pUivJ+W9u0mKdyv16avbp6W9BfWLQzrZYiXlSQr+E5wJbD3S6hrc6yhqDgAFHuH+KTlANIBukro3MLANF/RS0XnuJKH2VxBgjtDXpOVRWRTZ9sHirpKMAPVJR3VQ0FtJXAeYVfTjcnl/XvB36TDKBkn9D I7ixwViA bXB7RLoIitPmW3aZKTPgT6MrDVk4gB4CUmsweHl+9W03YE4HFle5SsCLAGaPaxhvbpeUEu/aW2iwkdr6d0PO+uZ+vxX/PhY3aTzROhuVH23kdknJ2ZrFtUlzsJusHoCx5ieMTf6JAFhOatrTaChC0J6N5w6AlOUGI0NNY03f3sI5AdOVQBQxZ4VUX10xyL9Lba6f6EEuflOzfXG92a8j3TnfmRTZ6Sf3HG2arkxUmYuaj1KDdUw00+YeI8YG53LJJhXmYTiIK/lC+oNrMktSqXWiCz10rQDTi8pdK7f5uFfm7aMV7vFGoGxD7+CTW+p3bwF3765K7CimC1mLcJRcH1EBSFDaFs+ikSbknVXvRKIzSrdguchC2TZy0jKv4f62is+8UcEm7xZy3OzAhrxs7JQWbWiU50qWDB1d3skr/XPiTdXdBPBM61wGxqg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: migration_entry_wait_on_locked() does not need to take a mapped pte pointer, its callers can do the unmap first. Annotate it with __releases(ptl) to reduce sparse warnings. Fold __migration_entry_wait_huge() into migration_entry_wait_huge(). Fold __migration_entry_wait() into migration_entry_wait(), preferring the tighter pte_offset_map_lock() to pte_offset_map() and pte_lockptr(). Signed-off-by: Hugh Dickins Reviewed-by: Alistair Popple --- include/linux/migrate.h | 4 ++-- include/linux/swapops.h | 17 +++-------------- mm/filemap.c | 13 ++++--------- mm/migrate.c | 37 +++++++++++++------------------------ 4 files changed, 22 insertions(+), 49 deletions(-) diff --git a/include/linux/migrate.h b/include/linux/migrate.h index 6241a1596a75..affea3063473 100644 --- a/include/linux/migrate.h +++ b/include/linux/migrate.h @@ -75,8 +75,8 @@ bool isolate_movable_page(struct page *page, isolate_mode_t mode); int migrate_huge_page_move_mapping(struct address_space *mapping, struct folio *dst, struct folio *src); -void migration_entry_wait_on_locked(swp_entry_t entry, pte_t *ptep, - spinlock_t *ptl); +void migration_entry_wait_on_locked(swp_entry_t entry, spinlock_t *ptl) + __releases(ptl); void folio_migrate_flags(struct folio *newfolio, struct folio *folio); void folio_migrate_copy(struct folio *newfolio, struct folio *folio); int folio_migrate_mapping(struct address_space *mapping, diff --git a/include/linux/swapops.h b/include/linux/swapops.h index 3a451b7afcb3..4c932cb45e0b 100644 --- a/include/linux/swapops.h +++ b/include/linux/swapops.h @@ -332,15 +332,9 @@ static inline bool is_migration_entry_dirty(swp_entry_t entry) return false; } -extern void __migration_entry_wait(struct mm_struct *mm, pte_t *ptep, - spinlock_t *ptl); extern void migration_entry_wait(struct mm_struct *mm, pmd_t *pmd, unsigned long address); -#ifdef CONFIG_HUGETLB_PAGE -extern void __migration_entry_wait_huge(struct vm_area_struct *vma, - pte_t *ptep, spinlock_t *ptl); extern void migration_entry_wait_huge(struct vm_area_struct *vma, pte_t *pte); -#endif /* CONFIG_HUGETLB_PAGE */ #else /* CONFIG_MIGRATION */ static inline swp_entry_t make_readable_migration_entry(pgoff_t offset) { @@ -362,15 +356,10 @@ static inline int is_migration_entry(swp_entry_t swp) return 0; } -static inline void __migration_entry_wait(struct mm_struct *mm, pte_t *ptep, - spinlock_t *ptl) { } static inline void migration_entry_wait(struct mm_struct *mm, pmd_t *pmd, - unsigned long address) { } -#ifdef CONFIG_HUGETLB_PAGE -static inline void __migration_entry_wait_huge(struct vm_area_struct *vma, - pte_t *ptep, spinlock_t *ptl) { } -static inline void migration_entry_wait_huge(struct vm_area_struct *vma, pte_t *pte) { } -#endif /* CONFIG_HUGETLB_PAGE */ + unsigned long address) { } +static inline void migration_entry_wait_huge(struct vm_area_struct *vma, + pte_t *pte) { } static inline int is_writable_migration_entry(swp_entry_t entry) { return 0; diff --git a/mm/filemap.c b/mm/filemap.c index b4c9bd368b7e..28b42ee848a4 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1359,8 +1359,6 @@ static inline int folio_wait_bit_common(struct folio *folio, int bit_nr, /** * migration_entry_wait_on_locked - Wait for a migration entry to be removed * @entry: migration swap entry. - * @ptep: mapped pte pointer. Will return with the ptep unmapped. Only required - * for pte entries, pass NULL for pmd entries. * @ptl: already locked ptl. This function will drop the lock. * * Wait for a migration entry referencing the given page to be removed. This is @@ -1369,13 +1367,13 @@ static inline int folio_wait_bit_common(struct folio *folio, int bit_nr, * should be called while holding the ptl for the migration entry referencing * the page. * - * Returns after unmapping and unlocking the pte/ptl with pte_unmap_unlock(). + * Returns after unlocking the ptl. * * This follows the same logic as folio_wait_bit_common() so see the comments * there. */ -void migration_entry_wait_on_locked(swp_entry_t entry, pte_t *ptep, - spinlock_t *ptl) +void migration_entry_wait_on_locked(swp_entry_t entry, spinlock_t *ptl) + __releases(ptl) { struct wait_page_queue wait_page; wait_queue_entry_t *wait = &wait_page.wait; @@ -1409,10 +1407,7 @@ void migration_entry_wait_on_locked(swp_entry_t entry, pte_t *ptep, * a valid reference to the page, and it must take the ptl to remove the * migration entry. So the page is valid until the ptl is dropped. */ - if (ptep) - pte_unmap_unlock(ptep, ptl); - else - spin_unlock(ptl); + spin_unlock(ptl); for (;;) { unsigned int flags; diff --git a/mm/migrate.c b/mm/migrate.c index 01cac26a3127..3ecb7a40075f 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -296,14 +296,18 @@ void remove_migration_ptes(struct folio *src, struct folio *dst, bool locked) * get to the page and wait until migration is finished. * When we return from this function the fault will be retried. */ -void __migration_entry_wait(struct mm_struct *mm, pte_t *ptep, - spinlock_t *ptl) +void migration_entry_wait(struct mm_struct *mm, pmd_t *pmd, + unsigned long address) { + spinlock_t *ptl; + pte_t *ptep; pte_t pte; swp_entry_t entry; - spin_lock(ptl); + ptep = pte_offset_map_lock(mm, pmd, address, &ptl); pte = *ptep; + pte_unmap(ptep); + if (!is_swap_pte(pte)) goto out; @@ -311,18 +315,10 @@ void __migration_entry_wait(struct mm_struct *mm, pte_t *ptep, if (!is_migration_entry(entry)) goto out; - migration_entry_wait_on_locked(entry, ptep, ptl); + migration_entry_wait_on_locked(entry, ptl); return; out: - pte_unmap_unlock(ptep, ptl); -} - -void migration_entry_wait(struct mm_struct *mm, pmd_t *pmd, - unsigned long address) -{ - spinlock_t *ptl = pte_lockptr(mm, pmd); - pte_t *ptep = pte_offset_map(pmd, address); - __migration_entry_wait(mm, ptep, ptl); + spin_unlock(ptl); } #ifdef CONFIG_HUGETLB_PAGE @@ -332,9 +328,9 @@ void migration_entry_wait(struct mm_struct *mm, pmd_t *pmd, * * This function will release the vma lock before returning. */ -void __migration_entry_wait_huge(struct vm_area_struct *vma, - pte_t *ptep, spinlock_t *ptl) +void migration_entry_wait_huge(struct vm_area_struct *vma, pte_t *ptep) { + spinlock_t *ptl = huge_pte_lockptr(hstate_vma(vma), vma->vm_mm, ptep); pte_t pte; hugetlb_vma_assert_locked(vma); @@ -352,16 +348,9 @@ void __migration_entry_wait_huge(struct vm_area_struct *vma, * lock release in migration_entry_wait_on_locked(). */ hugetlb_vma_unlock_read(vma); - migration_entry_wait_on_locked(pte_to_swp_entry(pte), NULL, ptl); + migration_entry_wait_on_locked(pte_to_swp_entry(pte), ptl); } } - -void migration_entry_wait_huge(struct vm_area_struct *vma, pte_t *pte) -{ - spinlock_t *ptl = huge_pte_lockptr(hstate_vma(vma), vma->vm_mm, pte); - - __migration_entry_wait_huge(vma, pte, ptl); -} #endif #ifdef CONFIG_ARCH_ENABLE_THP_MIGRATION @@ -372,7 +361,7 @@ void pmd_migration_entry_wait(struct mm_struct *mm, pmd_t *pmd) ptl = pmd_lock(mm, pmd); if (!is_pmd_migration_entry(*pmd)) goto unlock; - migration_entry_wait_on_locked(pmd_to_swp_entry(*pmd), NULL, ptl); + migration_entry_wait_on_locked(pmd_to_swp_entry(*pmd), ptl); return; unlock: spin_unlock(ptl); From patchwork Mon May 22 04:52:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13249753 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A555FC7EE23 for ; Mon, 22 May 2023 04:52:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 286CB900003; Mon, 22 May 2023 00:52:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 21092900002; Mon, 22 May 2023 00:52:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 089A6900003; Mon, 22 May 2023 00:52:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id E7C64900002 for ; Mon, 22 May 2023 00:52:37 -0400 (EDT) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id AF43880443 for ; Mon, 22 May 2023 04:52:37 +0000 (UTC) X-FDA: 80816670354.22.733F4E7 Received: from mail-yw1-f170.google.com (mail-yw1-f170.google.com [209.85.128.170]) by imf26.hostedemail.com (Postfix) with ESMTP id E702D140016 for ; Mon, 22 May 2023 04:52:35 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=y0KD6rBK; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf26.hostedemail.com: domain of hughd@google.com designates 209.85.128.170 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684731155; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=lc/g6LXQOs93yTgCiGczTkkFK+sAgodNjTysgahDEVw=; b=kuZKu06LH+Ug6s3+JlhyLvihmBCedRg1sjpmIMRNKLzotumV3gCNi9cBJJ1GckrZoDHGqG +N5inzwpxT2sgpzL30C1dDVXk+fCDVHD0yjhRiFO4CZdtHiLtOo7uQp1bjSbIahYhdyEI0 a5nKvE0lrsIkvQT21D1rJx9R6iO1guM= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=y0KD6rBK; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf26.hostedemail.com: domain of hughd@google.com designates 209.85.128.170 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684731155; a=rsa-sha256; cv=none; b=LdNoyPZXMcITTO33cUT6xxuWZCoJGS4ImbL0COnsF4vNEhxEX4EmklRkV/ixlXlm0bqpbB 9L4nTPCObVSI8e+EnMAnf4N0pBE++EGNuEiXnnFhjraQch9f3JkYRy4/mDMvWJa0pdz04A FdbxTTISYptvg7OrQnCk+2GFKzTW99E= Received: by mail-yw1-f170.google.com with SMTP id 00721157ae682-561bb2be5f8so70077067b3.0 for ; Sun, 21 May 2023 21:52:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1684731155; x=1687323155; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=lc/g6LXQOs93yTgCiGczTkkFK+sAgodNjTysgahDEVw=; b=y0KD6rBKhVlt+/hQTMgxrgHr2ryOWmswWrxwVwW1aFZ79LkRBOC59dixF5L/cDZc4B qnUQhKlaq1KoTySTp2vqiuM8rApR8HI8YhApmutmEIgesAjWn3Zc6GfqRJFCVIZETmFo Ve2TuK1DkPJI4FLOOGkjpX3SgGZwTv9PexqmJixY6pwUW+8F8Muuz17KmV5M/4+hZQ/g NOfOuuy9xT5Aq5lXVwksnHlv6Vlink7vPREGhAD7sqE7v5xbbRTG+tMaRpWIb/f65tVQ 6KthNz49DESDlvL8PCw3g3sz72EdxO0SmuuSbMBDwQ700O/IFokzhDePdlcNOBT1Ipvs 7n6A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684731155; x=1687323155; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=lc/g6LXQOs93yTgCiGczTkkFK+sAgodNjTysgahDEVw=; b=haVwZfbRLtSGoBD51dbcn5KIsCYp0MwlgbGWs/6H4V91q1xWjG7wRVLaevvdaEiPPV NAAzsLDyLshZbg24BH8N/MO+5oiKN/KmXCCORUT/adzbjOAEDFcW+Uzz5jcgbLBesPO0 lb2s5ZlHYIGh381bzuHXje6fuseCPDrQJ6E/zhzvED6c00ADv+fx4v8KXTBrMg5OTRVB zIKwnWvnb6aFRsZPFOfxyc37BZmEGALBwh8rsj/EW3gJZjZm4cgblpKSw/E0At8dT33C xMisJie7MItwfHyXCj2CvUu98WzrgK1ipLoClbMJa0IhmcIDJw634YG2tXYyXWBo41EG miag== X-Gm-Message-State: AC+VfDwPpyrANXy2Tf+NePXOpMID6FR5Oic0xqc35wNyMTtJg9gL/C3A kltshFkJlRrclNG+xxw3C84cXA== X-Google-Smtp-Source: ACHHUZ5dIfhq7xIc6CspLmzSUsrpzIMl80pglZ3+j6+NrHQ8JwvnQ0G2n+LIPSBV/lbo+YJXvCTWKQ== X-Received: by 2002:a0d:d901:0:b0:55a:9e43:7efe with SMTP id b1-20020a0dd901000000b0055a9e437efemr10375067ywe.44.1684731154895; Sun, 21 May 2023 21:52:34 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id w6-20020a814906000000b0054f8b201c70sm1786111ywa.108.2023.05.21.21.52.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 21 May 2023 21:52:34 -0700 (PDT) Date: Sun, 21 May 2023 21:52:31 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 03/31] mm/pgtable: kmap_local_page() instead of kmap_atomic() In-Reply-To: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> Message-ID: <9df4aba7-fd2f-2da3-1543-fc6b4b42f5b9@google.com> References: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: E702D140016 X-Stat-Signature: 6r4hn85m6fh6txx6myjuy4yum5cy8hmy X-HE-Tag: 1684731155-892059 X-HE-Meta: U2FsdGVkX1+NrZCw7xC4p6F3jg76B2zSkWvss0GFOvmOb5S1pWvmMCuMgIUOMWTx0TS+ZxAtE24ggGw0vJ8d6+T3tsyiqWWNP7pHPg14i4aNY4Z9aP1znFFDiT6M67aJfFQInDbkv/K4OQrGpTU7vJGXfnlOZrdyzDBTzzbMHR2u2H7hgtk/7DJtOabFQgmyGPRf3nwoHihbVrvpAurU3WnabAQWw1gZOmZLIdhtInw/PA1oGjLXb2lg5mlGNghxKUheLVLKmUAT0SCG3JDDzcLGBQPOD7AovVxQ5QZTlWz584rpsfRThRjzuKEPXdkZKa/V66qxczX3Nvd19xTmURNmfw9f0qyiNCTChm5UjvjHrjYzFntK0MMwCwZEk096fOQsntt8beKni/eR2VAQ5Jxd/gnn6N7CCZ7MPZWwLQ+E/rgkvdLyExjOwtR7DLOVob0WnC5IJW1Xa+LCajqr2t5sNtDavIE7GHJHMVbGTR1+85iMEOzvNosPwTbaQvzvd34tO+idXiBVLrpX0FePYw8Q5tI9oFAqYcy976Jjkue2pEwylT5WikS8LtkjtnO3iFcszEa2wef9A8IK0atIgYr4K6b0jUKLcd5ydKLE3xZ28LLlv61DHE9Fz+MhWt+QwS4rPsajG1NeMONduIKVhAS39z3LgzpXCJFIkyiGB0rrC9XoHzjvmpfiTCq4kXGf8X7ZUbzKIzC13RCLqqi2AFMvk/3YbZycMn6qWWzYsfLYtYZ5buIgnhbV4uCAuORTM0Y/tos7QDQ9VVBRPGweO6z0sPuboDzKV2b9SHJ8ks9zlrjasKS/Q/iPzJjyzKEnotUU+9bp/NilHQOKLHFScLVS4GIyOJMCa5LxINeAp0kiXqnmxTtSE44eTckAgLNW4GLsT53n78TN28SHHgfBlVnsx+zcDwzuUKHtjpkRqrO4mS793rj1s34tp4cmT6wlVR6Dkf0yuTrKl30T4S8 5NdZXPEV aaPtS2cDv0Zg/g8BBAT36OBKbol7PqAucPUCMZ2OLMeg2f2+5pHWGMtBjc8114GfZaMjmdnimJK4pMaCpE8d5FWukE31ErZg6U/U16GJdIdoI3zrP14ZfT2MX1egkDa4f8iBbe/2pwclzKS4NoWMRr+3FfTWLSwGSZdxJQOO4JJ+VY0DmzDcSbuXWq+QHtap8LmjEXjLTAFyL6i/RuxOyM72TzSFvYEddiI+veZlo1w5nYmVAGKsils4Mw53dp8F+C92O991uSWsF7B1BPa9hVQ7NWGdT84m4NfCARX/QsJG7TshupFUR+g4bJXcO5AW7QBQpKoGbqj9FUXqDaTggVNQx4JVZurvhspUEGMRuoplviV/Y5AL18qBPM6Fl6PXPlVDBwmJ3bld9v2NdQexXoPzxBUnl+LaBiGrlThzbpL/P6D9AdO3i5W/zxg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: pte_offset_map() was still using kmap_atomic(): update it to the preferred kmap_local_page() before making further changes there, in case we need this as a bisection point; but I doubt it can cause any trouble. Signed-off-by: Hugh Dickins --- include/linux/pgtable.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 8ec27fe69dc8..94235ff2706e 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -96,9 +96,9 @@ static inline pte_t *pte_offset_kernel(pmd_t *pmd, unsigned long address) #if defined(CONFIG_HIGHPTE) #define pte_offset_map(dir, address) \ - ((pte_t *)kmap_atomic(pmd_page(*(dir))) + \ + ((pte_t *)kmap_local_page(pmd_page(*(dir))) + \ pte_index((address))) -#define pte_unmap(pte) kunmap_atomic((pte)) +#define pte_unmap(pte) kunmap_local((pte)) #else #define pte_offset_map(dir, address) pte_offset_kernel((dir), (address)) #define pte_unmap(pte) ((void)(pte)) /* NOP */ From patchwork Mon May 22 04:53:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13249754 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 675AAC7EE23 for ; Mon, 22 May 2023 04:53:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0B21E900006; Mon, 22 May 2023 00:53:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 03AE7900002; Mon, 22 May 2023 00:53:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DD1F0900006; Mon, 22 May 2023 00:53:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id C3A94900002 for ; Mon, 22 May 2023 00:53:35 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 97FBE140443 for ; Mon, 22 May 2023 04:53:35 +0000 (UTC) X-FDA: 80816672790.20.8BF2587 Received: from mail-yb1-f175.google.com (mail-yb1-f175.google.com [209.85.219.175]) by imf26.hostedemail.com (Postfix) with ESMTP id B109614000F for ; Mon, 22 May 2023 04:53:33 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=QlhUE6K0; spf=pass (imf26.hostedemail.com: domain of hughd@google.com designates 209.85.219.175 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684731213; a=rsa-sha256; cv=none; b=fCFp8OT91ROU62YfUzNpgwTj6kbXI+78vjFsrCmcv9kVxUKaDlGDbhXQCLzPCFyWqBOYVI UNtlZP8onEBTEUhrz8xmZ+zJF4dl1vgBOPbPG7lENmHuG1Qm4MnmRqGZZM05tLszS+py6N qeiI7gbaavmIv4laxYTnRSAuI8PubDU= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=QlhUE6K0; spf=pass (imf26.hostedemail.com: domain of hughd@google.com designates 209.85.219.175 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684731213; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=aQBwLL4HlECbSO5gR059+G6dephAddbWEgO6E2LDWZA=; b=jDUDXwIpIgSCONn3XO0lzR4EWXvhX64jE/Wnxjp90SX2aCrAcxJsE799L8Un/IW6pPDmL2 kzhnzL6DpKoNEDftgwkB9VvWedrwT1EBB6RJVsmu3OvLcnk9exswTdCztVNAIDK++t8hg3 ZHG37Dc/61Q3IsO4BEejlAJ5vBHWJic= Received: by mail-yb1-f175.google.com with SMTP id 3f1490d57ef6-b9daef8681fso4777470276.1 for ; Sun, 21 May 2023 21:53:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1684731213; x=1687323213; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=aQBwLL4HlECbSO5gR059+G6dephAddbWEgO6E2LDWZA=; b=QlhUE6K0ATPFlt6KGVkJhXqSKL2ZPZOFukSr2LTyU5JBI1yxIy3RC9hQaumaw7cyuQ ADgG035cRuNduod0FxNhIZ0JZ4NHX+U32x2wiEFQa00gNFi56c3mSK3uiJdpEyXYIUsT ROwP4RLPq5CG77EbsJtEYQtkmgcTOVN5CnUNanq9+UFDyx0pkaHUzcdHI1z5w0XPUSmi o2MufvhNsbrbye+c4cFuachrzWPeZh2+I5zinfMV/dyW4naX4xe0MJ9MR0i12bBzFRCj 8PLk4Zikwq2ioFjQFgXXXofFctLjXIybQaGA/r+no+6O/pmdY3s7wDE/uJygGJA7s2KN wLcg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684731213; x=1687323213; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=aQBwLL4HlECbSO5gR059+G6dephAddbWEgO6E2LDWZA=; b=TmjxKc7AG6Mx7mvv5LdAvfwZCcbj/vAqcsiNh6ITraYzWF6WuokFKaf6yxdXW73qLs kwEFD+TlOP41TPaoxxXAO4bk2kxSA//Mx8ApqLtm4iKzDWjIR3A9aO3cGsCG9qE5aSBG MuCDg4sUMJGaB3Qkqiq3QPzAE2xFmsQKRaYuMTZwvE15WNcyoy5PMb+swX3m2VhhrbNZ 6LNy/A6qluR6sdQZg/YNF8Gdf0HYes1IFodKwmnQ7DC86TYPkakhOQJMS59ha9vq8Roa Tkb34e79bDNje51E8y7QPC9YX6nilefF4OYuKAV+zw2DQu4sp91L5OTUO4azOYvKiQKw HCBA== X-Gm-Message-State: AC+VfDykt/VnHW945pmcrfawhTr9ozSe/Cmmdk9HNCZ59l1KbK+xzEZI o9NDwa5OC17ehg81n5uycoe4OA== X-Google-Smtp-Source: ACHHUZ4a/wUx7HBHObnjrL9fmwJm9cuLSPhXj2yvWOXUl7AxoJ4/x4VelqXRKeaAtFJsJWrg4u9aMA== X-Received: by 2002:a25:cfc6:0:b0:ba8:62ed:2221 with SMTP id f189-20020a25cfc6000000b00ba862ed2221mr9802761ybg.62.1684731212645; Sun, 21 May 2023 21:53:32 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id x7-20020a259a07000000b00b8f6ec5a955sm1255333ybn.49.2023.05.21.21.53.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 21 May 2023 21:53:32 -0700 (PDT) Date: Sun, 21 May 2023 21:53:28 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 04/31] mm/pgtable: allow pte_offset_map[_lock]() to fail In-Reply-To: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> Message-ID: <8218ffdc-8be-54e5-0a8-83f5542af283@google.com> References: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: B109614000F X-Stat-Signature: 386khxmz649xez63rfz7a65wphn735ym X-Rspam-User: X-HE-Tag: 1684731213-925835 X-HE-Meta: U2FsdGVkX19Dc5KBMnzvo6EKH6IF2w0w/daHfgx3kIANKrfEQrx3vOK/o51P3l/MSTOtV9nL7Xu5TXRmEV8D98KBSQp+b7yDE6mzl0FKokjbN5HZAG9JHlZTOvmMTqM5eBhWvnLg7/VtyAkF9hgiAzM+S87vue4dEEkxOgEsoIU/OVX3Pkxr3wb4htqOt31/WeyE6Vbisv4ZePAsNgu8MbidGl41fhljH6bkQhoELnmXte/BIHG4SX2cIvQjWMoQ8eFAxaTVVretdOX8R91+jJIhTtRosML2rsEn5p08LSW95Lc78rqykrYpyFA3cq2kGOA9UWTBTvSnMmB1vf5hZP+hmshQb1LGK665e0lhR4ERxq6FsChK12BvCq6GaAEQa4VnJyybHqCeJ/QuIraA+0g+oE6/+kyNHd9zQOXUBBkQZMvxUAUli6Xc/5/Ui7jlduShCt8RmncF+C5lZXxXnHn3EPXuqbctULdIcFtTGGCv8qvMYlwXqnLWAT640ibVgK/7lq+wlUgEcdAVh/tUta1UmL1JVwjzIuFRRV3g5T3KRfUPTdyKBnmuce1qa+1MBElLysEug+zyz3HC11tV7jTHqWLlcH+1s8s47uiCAklySLSxZ1x6LWTd63NDAp9MIgRh8wzVQhXgGZ13CHYMla7CWC/JHzSkuzXWgoi0tkZfk7fi87eS/TkmkmYwWlqBMoKXus5Ofm8t5TQcJyAYHL/nH3wqT7O9CbjQ1PauBarTg66QwGHelB/TE9e5qdF0idO0uF6IjvUEwRKtp+7I4TT4AbOYqB2WUqJXHdAgl+jfca4szs3djeyEGimfDPXvX6aZnSB2s7+BzzhXnllW/GOfl4HUX2CMCo4HCrveGAAFQlzKpiFlxm9GpZq9brKsI4RPky84uHLwT2jYaJsZKvcyd+/Es6Q6T3kwNYOikiXX2nQ25Yaypa6lvLmJKOiz1T3g2Z1vQWMun6kRpdi 9C9lGUYt vdfHjLSMInhC/QSEdbjubEmFDkunf29LlwrcN97KSpebAyRxFvrduDUulJm8n9uE4n/CeABuG6IOuFeZ+F4oP9pz9mYkLUaTSNUnLeyEhN9DBH9AcJzZS/u2BZazD8/ogDWtd4Q0N1ii88kLOfb1yo2sEF5Tps02hnfbID+dGGVpODbfZC37u1e5lWnnBwXpq4/cWZ0LTsTPysiqWEiBqSe3W2i857Hzm1zekiTlJcsnW66o+LNkUGoGXQvDArntp4zwQ+pduYOrW7zieYsX3ih+NaoMOT+hbV8stk4kdBJnGxsxrA+ELfrKSbeGgr5ASJmy4fIg4XbOAgWW9PBmFUT8cZANOcPcPqQEISu4IibaRJhiYfESBPtdLsTBDy85Kd3v8Dph8w3TxGh/7FRIxP3r3JFajhMJ5gQZ6aiPxj8fZCjiXsOMEInAxmg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Make pte_offset_map() a wrapper for __pte_offset_map() (optionally outputs pmdval), pte_offset_map_lock() a sparse __cond_lock wrapper for __pte_offset_map_lock(): those __funcs added in mm/pgtable-generic.c. __pte_offset_map() do pmdval validation (including pmd_clear_bad() when pmd_bad()), returning NULL if pmdval is not for a page table. __pte_offset_map_lock() verify pmdval unchanged after getting the lock, trying again if it changed. No #ifdef CONFIG_TRANSPARENT_HUGEPAGE around them: that could be done to cover the imminent case, but we expect to generalize it later, and it makes a mess of where to do the pmd_bad() clearing. Add pte_offset_map_nolock(): outputs ptl like pte_offset_map_lock(), without actually taking the lock. This will be preferred to open uses of pte_lockptr(), because (when split ptlock is in page table's struct page) it points to the right lock for the returned pte pointer, even if *pmd gets changed racily afterwards. Update corresponding Documentation. Do not add the anticipated rcu_read_lock() and rcu_read_unlock()s yet: they have to wait until all architectures are balancing pte_offset_map()s with pte_unmap()s (as in the arch series posted earlier). But comment where they will go, so that it's easy to add them for experiments. And only when those are in place can transient racy failure cases be enabled. Add more safety for the PAE mismatched pmd_low pmd_high case at that time. Signed-off-by: Hugh Dickins --- Documentation/mm/split_page_table_lock.rst | 17 ++++--- include/linux/mm.h | 27 +++++++---- include/linux/pgtable.h | 22 ++++++--- mm/pgtable-generic.c | 56 ++++++++++++++++++++++ 4 files changed, 101 insertions(+), 21 deletions(-) diff --git a/Documentation/mm/split_page_table_lock.rst b/Documentation/mm/split_page_table_lock.rst index 50ee0dfc95be..a834fad9de12 100644 --- a/Documentation/mm/split_page_table_lock.rst +++ b/Documentation/mm/split_page_table_lock.rst @@ -14,15 +14,20 @@ tables. Access to higher level tables protected by mm->page_table_lock. There are helpers to lock/unlock a table and other accessor functions: - pte_offset_map_lock() - maps pte and takes PTE table lock, returns pointer to the taken - lock; + maps PTE and takes PTE table lock, returns pointer to PTE with + pointer to its PTE table lock, or returns NULL if no PTE table; + - pte_offset_map_nolock() + maps PTE, returns pointer to PTE with pointer to its PTE table + lock (not taken), or returns NULL if no PTE table; + - pte_offset_map() + maps PTE, returns pointer to PTE, or returns NULL if no PTE table; + - pte_unmap() + unmaps PTE table; - pte_unmap_unlock() unlocks and unmaps PTE table; - pte_alloc_map_lock() - allocates PTE table if needed and take the lock, returns pointer - to taken lock or NULL if allocation failed; - - pte_lockptr() - returns pointer to PTE table lock; + allocates PTE table if needed and takes its lock, returns pointer to + PTE with pointer to its lock, or returns NULL if allocation failed; - pmd_lock() takes PMD table lock, returns pointer to taken lock; - pmd_lockptr() diff --git a/include/linux/mm.h b/include/linux/mm.h index 27ce77080c79..3c2e56980853 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2787,14 +2787,25 @@ static inline void pgtable_pte_page_dtor(struct page *page) dec_lruvec_page_state(page, NR_PAGETABLE); } -#define pte_offset_map_lock(mm, pmd, address, ptlp) \ -({ \ - spinlock_t *__ptl = pte_lockptr(mm, pmd); \ - pte_t *__pte = pte_offset_map(pmd, address); \ - *(ptlp) = __ptl; \ - spin_lock(__ptl); \ - __pte; \ -}) +pte_t *__pte_offset_map(pmd_t *pmd, unsigned long addr, pmd_t *pmdvalp); +static inline pte_t *pte_offset_map(pmd_t *pmd, unsigned long addr) +{ + return __pte_offset_map(pmd, addr, NULL); +} + +pte_t *__pte_offset_map_lock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, spinlock_t **ptlp); +static inline pte_t *pte_offset_map_lock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, spinlock_t **ptlp) +{ + pte_t *pte; + + __cond_lock(*ptlp, pte = __pte_offset_map_lock(mm, pmd, addr, ptlp)); + return pte; +} + +pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, spinlock_t **ptlp); #define pte_unmap_unlock(pte, ptl) do { \ spin_unlock(ptl); \ diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 94235ff2706e..3fabbb018557 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -94,14 +94,22 @@ static inline pte_t *pte_offset_kernel(pmd_t *pmd, unsigned long address) #define pte_offset_kernel pte_offset_kernel #endif -#if defined(CONFIG_HIGHPTE) -#define pte_offset_map(dir, address) \ - ((pte_t *)kmap_local_page(pmd_page(*(dir))) + \ - pte_index((address))) -#define pte_unmap(pte) kunmap_local((pte)) +#ifdef CONFIG_HIGHPTE +#define __pte_map(pmd, address) \ + ((pte_t *)kmap_local_page(pmd_page(*(pmd))) + pte_index((address))) +#define pte_unmap(pte) do { \ + kunmap_local((pte)); \ + /* rcu_read_unlock() to be added later */ \ +} while (0) #else -#define pte_offset_map(dir, address) pte_offset_kernel((dir), (address)) -#define pte_unmap(pte) ((void)(pte)) /* NOP */ +static inline pte_t *__pte_map(pmd_t *pmd, unsigned long address) +{ + return pte_offset_kernel(pmd, address); +} +static inline void pte_unmap(pte_t *pte) +{ + /* rcu_read_unlock() to be added later */ +} #endif /* Find an entry in the second-level page table.. */ diff --git a/mm/pgtable-generic.c b/mm/pgtable-generic.c index d2fc52bffafc..c7ab18a5fb77 100644 --- a/mm/pgtable-generic.c +++ b/mm/pgtable-generic.c @@ -10,6 +10,8 @@ #include #include #include +#include +#include #include #include @@ -229,3 +231,57 @@ pmd_t pmdp_collapse_flush(struct vm_area_struct *vma, unsigned long address, } #endif #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ + +pte_t *__pte_offset_map(pmd_t *pmd, unsigned long addr, pmd_t *pmdvalp) +{ + pmd_t pmdval; + + /* rcu_read_lock() to be added later */ + pmdval = pmdp_get_lockless(pmd); + if (pmdvalp) + *pmdvalp = pmdval; + if (unlikely(pmd_none(pmdval) || is_pmd_migration_entry(pmdval))) + goto nomap; + if (unlikely(pmd_trans_huge(pmdval) || pmd_devmap(pmdval))) + goto nomap; + if (unlikely(pmd_bad(pmdval))) { + pmd_clear_bad(pmd); + goto nomap; + } + return __pte_map(&pmdval, addr); +nomap: + /* rcu_read_unlock() to be added later */ + return NULL; +} + +pte_t *pte_offset_map_nolock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, spinlock_t **ptlp) +{ + pmd_t pmdval; + pte_t *pte; + + pte = __pte_offset_map(pmd, addr, &pmdval); + if (likely(pte)) + *ptlp = pte_lockptr(mm, &pmdval); + return pte; +} + +pte_t *__pte_offset_map_lock(struct mm_struct *mm, pmd_t *pmd, + unsigned long addr, spinlock_t **ptlp) +{ + spinlock_t *ptl; + pmd_t pmdval; + pte_t *pte; +again: + pte = __pte_offset_map(pmd, addr, &pmdval); + if (unlikely(!pte)) + return pte; + ptl = pte_lockptr(mm, &pmdval); + spin_lock(ptl); + if (likely(pmd_same(pmdval, pmdp_get_lockless(pmd)))) { + *ptlp = ptl; + return pte; + } + pte_unmap_unlock(pte, ptl); + goto again; +} From patchwork Mon May 22 04:54:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13249755 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 404D4C7EE23 for ; Mon, 22 May 2023 04:54:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D5C81900007; Mon, 22 May 2023 00:54:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CE4EC900002; Mon, 22 May 2023 00:54:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B8555900007; Mon, 22 May 2023 00:54:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id A6AA5900002 for ; Mon, 22 May 2023 00:54:31 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 631461C7587 for ; Mon, 22 May 2023 04:54:31 +0000 (UTC) X-FDA: 80816675142.29.2EA0827 Received: from mail-yw1-f169.google.com (mail-yw1-f169.google.com [209.85.128.169]) by imf07.hostedemail.com (Postfix) with ESMTP id 878BC4000C for ; Mon, 22 May 2023 04:54:29 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=J9ylJNba; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf07.hostedemail.com: domain of hughd@google.com designates 209.85.128.169 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684731269; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=DKO2MGlysYL+7+oQy8UdaapVjC7HViDdumwjMefUUFk=; b=GhunT1IQ7Dqy5wbnDfsWJQY7FQGfmSWw613ezKqiIjYUKz6fr9d4fmiqzKK36U9nEBFRwm 2gTIoVDSPNZy1B831qnaRTwKv5G2uaGuXu9SdDK+vYmoPElqImlFrgNECW5GLLlXgq5F4D W+g1XtMdc7JA1QL2w3qnQVb2vMA7A40= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=J9ylJNba; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf07.hostedemail.com: domain of hughd@google.com designates 209.85.128.169 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684731269; a=rsa-sha256; cv=none; b=hAcFnIuba5WZgkuClIEB3ksz6x90Vg0J3jgEBFN808OfBxNNmx6hcikMy6pMDzIX6LHDRh Cr17xgS/sV+3WdQq6lPkuuSh0dpdYje0GfzyxWwKsrTDTcUbUrTM20ffL0ZatvhfNpU5+E X0kcV7BQuhpQ51Z/rv3sfgMTOpBXy5Q= Received: by mail-yw1-f169.google.com with SMTP id 00721157ae682-561c46e07d7so76332627b3.2 for ; Sun, 21 May 2023 21:54:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1684731268; x=1687323268; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=DKO2MGlysYL+7+oQy8UdaapVjC7HViDdumwjMefUUFk=; b=J9ylJNbawMeH4tOfy11w+ZaO/6872J2rqfHP5lovPYJFW8zB72NKQpN6jeQPkFtYO5 7RHmvMjgUcPMPHqNGnXVyrAFqE5jQbr7ovn6/CLPgFbf3l4X7M/QuLcTo+6IociOTnvp 5CQ9bnI6x5YxJS4Rh5hvct/0Yo5JX0qxeMmK1yabPvgHVQgZYihchIYIgKApcKzJ3IRg ROh6CHyiaDDOL+Ex2mG/L8yyWOYTqU/rosesPvWEE4ijihe6qctq0T8BNRcL6ZpZfkCn M6FU+PpeRPkKqHsAb2AldyKg/94bE7H6WNY4ZFE5SlqswttpLgasmwxgkPJXNEXdyjtP 03EQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684731268; x=1687323268; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=DKO2MGlysYL+7+oQy8UdaapVjC7HViDdumwjMefUUFk=; b=aotFiMD8qgtbKoYsphnSnwyrykw94qIASsUUV3v54zk4jSVvvuqWWm04gRvaPwJy2Y bQr4sYgo0oTHmc3jiydjeBUBpP2gJbN72VTZ76hN4Ce1CPTUidG69yOXEsH5TpaGjAYv qOf40jlhzEbBydYWwToBtI11xXOYe3cbyYlMUG5S2LlGnfFupvnoyGzYVoW5sNA7fS8Y pRIht3nkaDGHT4kwhMaF69ABMGPIIjZB6M7qC5HNR7vno2uaZ31Ot/x82JvfRHFyOtjv 3vfvR5BKapBSocPur49aboOMgCu1eGWWZDEpHDFeJd9kTE72/wuHbaKQczpf0fOTQxp8 tW9Q== X-Gm-Message-State: AC+VfDxS/CTkYhvhYdTxk5ohliqb63diW13V7IJZuNaotkWyS9beKa2C HZwEVqTguI34LcYpRmPHMYVVGA== X-Google-Smtp-Source: ACHHUZ4551zDAoXcCIh8jhAd8f8X+Cxw/fLbkIrjHAC/+8ziZTZq6Fx4vtipE1QtRd7bp1hgyGTCog== X-Received: by 2002:a81:4e97:0:b0:561:e7bb:36a1 with SMTP id c145-20020a814e97000000b00561e7bb36a1mr8839265ywb.49.1684731268515; Sun, 21 May 2023 21:54:28 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id r190-20020a0de8c7000000b0056189f9ec2asm1803508ywe.133.2023.05.21.21.54.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 21 May 2023 21:54:28 -0700 (PDT) Date: Sun, 21 May 2023 21:54:25 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 05/31] mm/filemap: allow pte_offset_map_lock() to fail In-Reply-To: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> Message-ID: <3e6d4f8-9f4d-fa7e-304e-1494dddd45b@google.com> References: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 878BC4000C X-Stat-Signature: 1ptygqj3rnthtn7brhpb55mbonyi83c6 X-Rspam-User: X-HE-Tag: 1684731269-376775 X-HE-Meta: U2FsdGVkX19F3BBw2YslYZBSzFh1nmP0kLoOHXyNuRnPEdKYO1MpsIWhUFebi7aLpZwNFIHXCCTTfJB5b5GV5xKZ7xUjVOgt5P1b17T9rFWxdU9qw398ITyuw1uNVOaFndYVsgcI1KVdFjH9xWz9jHtPRnnBEgAbl1L6710LDovagqpuPjUt+y7414WfwJE918hjnLf06NCyrKmEFnL7xJNb0lDgVxMru9glE8ZLm4ZTPkaSaHmxpXNpjHRgwG3k+jXyfoj1gzHTdEKPNsZEr5aAQqGcVcRriyvRohopucMPNIiZ7h5hgi0Sf+Tr/CcNRH2wcHFjB9Tu4VXDUMcyUZKRbbpQ2+JbsBr03ozPJMOYIBMn8BcPLa/Bu2Z+IXnz8t/I5Ah/0/4Pc4BCSz+f3KBfZiySmknJP8rXE2TkETgkSibKTKuMELRQ8jlkfE7GHi+HSaJ1Q44omxjEwruei3Ci00c2djcDqK9GiqfcM3qAntB97wDI5U3oK+OPxYoOaTnMKwNA7y6ACF3X45XZE4v8fuVnOraarsbOgYPewyxTOLOj4Nnn/HsMkItOFhPTe/tKo9NdSHPBXjfoYdeKEcyopQcPGUPEOlfzdlHoWBFP5yXUkVo9L7mj+fgAbrJeXSZ3pPRHJRuKRnxpCAYv4AS2GlkFx19BPUgD7GB8DQJsEPZmYtBfwPiQsd5ybTICx6lzVeIV6rt4LS0BGXtuoQBPW4W05yaQ1aOvf0nSbO0bj7dm7RwjJ2gPbbXr2YIiNP5SmHaNK7GxaZLpNXMGDAj4YUfn1WUV18EOpaVlWNYVw4eWRcvpyfJTUdKm1+Z0xZywnC2aJz1JFXoW5wVUiuz1G1+mIgEYcnnCshViIT+c8TKQcmBqHHsIw+NE94/h3ebrT6QALI9Q7NbUMj10nMsXjd70AUpotzps2xOBnkU//0PHOmyU2mzBkSRMokHE/QaVUe5sUXLWD0nRZJJ /disuP/m qKZrXlRGx57QcZ9UIawSfc7AFGoPzSsjsvEnZZlFHtBUNGDRNtcrVchIIXjh3FxvBZ9fEaM2ahdHNbWDoNjEOIAsiYE1Sny7JUa4v8N8Ct7DsMAoL9yRcw/kDXIEH3D/geznje8RyzIQLwe+CB9ujy7ozPNdfQNPirI8/vzM4/wkTUdb+KjhPTV2+NoWRMpE8309dA62WE6Qnkg0oObmzL7ePyiE3K6jrx/3J/88lCIaM6VgRrts0/507j5f9DsULGs3lLJR3iyB40M5+dbpYe9M7zbo2+wujyZ24+catnUzStToFkhweRr5lIoDr7ayX4vlJQnYWFEVGGfBp/c6lcSHLpo5ADdHfw6Ov+sOccaugBvHLdZ+Pt5RXUtKZMSYQqbEWggtr5tZxfYzzAoyI9M71CcNzfX/pa5Nz5eOfZYdNBl5RWy3Eoxawyg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: filemap_map_pages() allow pte_offset_map_lock() to fail; and remove the pmd_devmap_trans_unstable() check from filemap_map_pmd(), which can safely return to filemap_map_pages() and let pte_offset_map_lock() discover that. Signed-off-by: Hugh Dickins --- mm/filemap.c | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 28b42ee848a4..9e129ad43e0d 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -3408,13 +3408,6 @@ static bool filemap_map_pmd(struct vm_fault *vmf, struct folio *folio, if (pmd_none(*vmf->pmd)) pmd_install(mm, vmf->pmd, &vmf->prealloc_pte); - /* See comment in handle_pte_fault() */ - if (pmd_devmap_trans_unstable(vmf->pmd)) { - folio_unlock(folio); - folio_put(folio); - return true; - } - return false; } @@ -3501,6 +3494,11 @@ vm_fault_t filemap_map_pages(struct vm_fault *vmf, addr = vma->vm_start + ((start_pgoff - vma->vm_pgoff) << PAGE_SHIFT); vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, addr, &vmf->ptl); + if (!vmf->pte) { + folio_unlock(folio); + folio_put(folio); + goto out; + } do { again: page = folio_file_page(folio, xas.xa_index); From patchwork Mon May 22 04:55:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13249756 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 97946C7EE23 for ; Mon, 22 May 2023 04:55:57 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 285B1900008; Mon, 22 May 2023 00:55:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 25D30900002; Mon, 22 May 2023 00:55:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 12520900008; Mon, 22 May 2023 00:55:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 0082E900002 for ; Mon, 22 May 2023 00:55:56 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id D5D2C1603FF for ; Mon, 22 May 2023 04:55:56 +0000 (UTC) X-FDA: 80816678712.02.1849696 Received: from mail-yb1-f179.google.com (mail-yb1-f179.google.com [209.85.219.179]) by imf03.hostedemail.com (Postfix) with ESMTP id 217752000F for ; Mon, 22 May 2023 04:55:54 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=cg30IRtt; spf=pass (imf03.hostedemail.com: domain of hughd@google.com designates 209.85.219.179 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684731355; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=zTm3eSBiFsuHpLmgEmMbxBID/kzy6WIASRGS/m4xNS4=; b=vq0F58KrcChOqdx346OYEM0Fc9Q9E+x4fzkEmQk/T98W0aBY9NqH7XbL7EhIuNhyCVlian RaaDvfGlvk3ylYCq4KKLvjWagvUSPmO+BfO4coHWZnzkRhzDh/TCNSVwRsLKicf+j6A9H4 /4uqK5V2df1sEf1F48yZCD7fR9viO3Q= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=cg30IRtt; spf=pass (imf03.hostedemail.com: domain of hughd@google.com designates 209.85.219.179 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684731355; a=rsa-sha256; cv=none; b=3rL0aUjcYaJZvY+FfXu1gMquSmMgHRXlk0AT18RtkYWlyQGB4UFIH9LW/wrFnNoZSAmNmM LXdJVfTmlK8QrLHXHnpVcKp09w6cld59YqxooEzDo/sG1+eD0HAjLbSC2duDk/UxTcirnY 51N4X+0lXIjSYE57fmPViScmEbRknm8= Received: by mail-yb1-f179.google.com with SMTP id 3f1490d57ef6-ba829e17aacso8512565276.0 for ; Sun, 21 May 2023 21:55:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1684731354; x=1687323354; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=zTm3eSBiFsuHpLmgEmMbxBID/kzy6WIASRGS/m4xNS4=; b=cg30IRtt7WWBZINDX0Jv402rJAxuzmQi7sNbiwo3sAYNeuv0iwLqCdxKXUV1666arB iODQXo3TXBZWNrbAGBgo7JPYM/uh3QZy2aOwzrHWKdaKeQxiaTgQr+B/h0WaJumRmkZS EwvnPxhlE/ygC1l9PvisFRAxceZsimtF2xJnCScbrcoMyL1u1L/6wf9rDZLxnaTA3LQc zI7g1VSQ5r9D+qngti0qturyyoRo7eal1Od9D8FGmOBxjDKgugY3ZMq1QggviidfFbw9 1sPnCORb05gZ3h3EwvicWZtgtg3JnoDyX2eiBh+EAi3PI8BUvp+6C/C1Nf/8YEhUgGVC fWYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684731354; x=1687323354; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=zTm3eSBiFsuHpLmgEmMbxBID/kzy6WIASRGS/m4xNS4=; b=XKJ2P1lDps65NGcFSEqHHvXWt8Ys3/Fufl4m6JTfjp8OyWszg8XoJj6aNcnTvhn80Z 66Cee9j1fJ9UH3j/kZNkzosN+ABeOdnA1ugLUbARY4jLGT0HeVYXLp9bau2tFj4E+tms RVTGx28CHhbVW3l4OMvAifmKHvoMHgcjhTe9bruJlKxjeXNRPJzn80/6RdyStbFvAhZz 8gYzAGI99iH041R/aqjmsF8ohmUPwUT0Z7VdV2b0jTdaizKzuuuXf4w5MbxQQEP4YNau af5rm41NBqUycpDH8y2Fr6VVeaHVRcRNBii2lR0By3K1GAozujfI0i1p4acKs2sOA0VO gFcQ== X-Gm-Message-State: AC+VfDxh2d3MugNy9LlUcVuAPFGZ82UDJlvYl4YKn07wr43+AH0qtK53 lwoX0eAsbA3DuknE3EdMTjTr5w== X-Google-Smtp-Source: ACHHUZ5T4JJRJ8fgbx7qxtgyN7oMDnzb0y/FhCeDxcmhIqNJgCW1YAFxxuHwFhGWjkNrnbHB22VBBQ== X-Received: by 2002:a25:fc28:0:b0:ba8:37bd:59db with SMTP id v40-20020a25fc28000000b00ba837bd59dbmr8298369ybd.34.1684731354145; Sun, 21 May 2023 21:55:54 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id 185-20020a2505c2000000b00babd051a405sm938658ybf.26.2023.05.21.21.55.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 21 May 2023 21:55:53 -0700 (PDT) Date: Sun, 21 May 2023 21:55:50 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 06/31] mm/page_vma_mapped: delete bogosity in page_vma_mapped_walk() In-Reply-To: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> Message-ID: <502d6743-b0bf-d848-596a-4b3f3e44de8b@google.com> References: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 217752000F X-Rspam-User: X-Stat-Signature: tdpmxex866paez1cq6bghukbcsakq8wm X-Rspamd-Server: rspam01 X-HE-Tag: 1684731354-587285 X-HE-Meta: U2FsdGVkX1/TKjD/Le/gpdJ4ayF6953MOh1UVyAGPbcA3aSg+5TZ9Q95Gco44WoF55Oc3nQqQ+eqvdEzU/NbxDVPjxuyUOps8/xJPnd2neTh9fgJuTC3ZPISO3ksXmf6oDftNJnjSPyiQlUiW2EDQwqSJerOoDXvYLRjGx47b/+3SnTVHaU/cWCfXIp9VproQ/OmN85+ldZSgzCm+qGto8g50Cbk/v4HdLZ3LU1+/hQkbN15hwTk24MEVqeaJEXhHg1csJwWIC9tOVANwd8Nl7dVezkvajzargAxAC0Ct+nHamiaaXAYR/7y+dlTN3qunlJiK0BncWZGiV7KcY0xI5EXiMZCRG7iRvpyVGgZaABFf6tYop38xx74dvpxqjyv39Knv0oekDyI6Z4z7WnfiJ24kHOSgr2xUV/XeB2PDFWGfB6fHigSXYJAYjb1Bq8DcgxvuYn35tZy0w3e2MXpMDjtJk0wvWTywZzUpU0f+wLPcLBQ2xkEotYlXh1OuXSL1Ygy3BDtNmF5yzKbwaV3vADRTksgRzPslMOUbLqNz1HGDWAefo2GOk6q2KoMAy4Kr+lfQp6rDx6pbJxiWgXLBL05wpXaLylSqWqtu7OaCKxumV3oeq4kK64w5+8LpBnJiGtPpUPQZ/131fq8GWt4SqNHdvSgeL7/Mc2ZpOCa6yEgcRkwPWtnz3yt5SuDUK/c3KgSzqRBWKok08tRlNBl9FKiFshok0JuF3i8uKkCn/iIU/e1YYi6CDYcO0CsohvaB3jJXS3iu7kCJ/95zDW7hL0sTV35fRIt3EoXO8FH8RNvype2FLtz8LCt9VME2r9tt+14oV5aK+WvlR6BamBQtBjPyvtc9qdI6jepo41g3Zayvmz/UeYApP/nvPkwLonKoaLHhrZgXX/R3UEJ5lEV1g6WbC402SFCU6l6G/I8A1nkfAOXECrqJn1AEcA2I61vubXyP3+dVm7w4GHXw94 H755to1v baKNMFqeyD9ndsOfIK18jCmL2jxawqAndvOmkFharaDhZUGx0MsMDN1PyMuWa2Uoxm9Nc4M1k3sBbydMFBWsR7Zqd4m7Plwy4ifnJtkVK6WOot/15rlpDXXGEh3hi9aQG7RmpYkbmFJct8mo2Qu0KCNYKWlKOtR88BOlNedgMqtJZAkV8rcgEtPlqtVH8ZSbxLshCmLNN319QUWTC2SLPkDtlzaFdg2ImxbOKhRuY6aGQsVxYleg7WQnuI5Ktv1/ysAbrgj1B9t6rjHlLVXyouOuh2TE8Y2rY5UzseIc3nqnjULkl0vxmBWzjZAx5jM6YOd5Mw5K63GaXaTUCwJQzNJsMocQnFI/QNFaYYvtfGzvnbpYKyRXq42RksYUDiKI8q/zYO0oysb0BxE8= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Revert commit a7a69d8ba88d ("mm/thp: another PVMW_SYNC fix in page_vma_mapped_walk()"): I was proud of that "Aha!" commit at the time, but in revisiting page_vma_mapped_walk() for pte_offset_map() failure, that block raised a doubt: and it now seems utterly bogus. The prior map_pte() has taken ptl unconditionally when PVMW_SYNC: I must have forgotten that when making the change. It did no harm, but could not have fixed a BUG or WARN, and is hard to reconcile with coming changes. Signed-off-by: Hugh Dickins --- mm/page_vma_mapped.c | 4 ---- 1 file changed, 4 deletions(-) diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index 64aff6718bdb..007dc7456f0e 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -275,10 +275,6 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) goto restart; } pvmw->pte++; - if ((pvmw->flags & PVMW_SYNC) && !pvmw->ptl) { - pvmw->ptl = pte_lockptr(mm, pvmw->pmd); - spin_lock(pvmw->ptl); - } } while (pte_none(*pvmw->pte)); if (!pvmw->ptl) { From patchwork Mon May 22 04:57:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13249757 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5C322C77B73 for ; Mon, 22 May 2023 04:57:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F07D3900009; Mon, 22 May 2023 00:57:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EB873900002; Mon, 22 May 2023 00:57:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DA71A900009; Mon, 22 May 2023 00:57:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id CA496900002 for ; Mon, 22 May 2023 00:57:32 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id A39F81C757B for ; Mon, 22 May 2023 04:57:32 +0000 (UTC) X-FDA: 80816682744.07.304B19F Received: from mail-yw1-f173.google.com (mail-yw1-f173.google.com [209.85.128.173]) by imf21.hostedemail.com (Postfix) with ESMTP id D50261C000A for ; Mon, 22 May 2023 04:57:30 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=CynFLjQo; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf21.hostedemail.com: domain of hughd@google.com designates 209.85.128.173 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684731450; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=dk0QCHZ/ZrkXov8P0kyfxgscgrLXPV5n2YV5/OCWJso=; b=MDw1dVaJ86hoRNJlc6rpn+AHFFqsgpxW5FHtgCH5ZZolWljX1xSfAc6ka8NawAR+DmrrVi WrHcL8XdUXoMK/JmbdLS5sdFqyznCpiUz/nuA8DW87rJeL2WbPeu17+PUgwCGRKc42M1CC CEft+fVqnHzdduC5vuQaWcIHCwcDSxU= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=CynFLjQo; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf21.hostedemail.com: domain of hughd@google.com designates 209.85.128.173 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684731450; a=rsa-sha256; cv=none; b=d9z/gnTxXR17HcwnBNDOKQVKXcKQN/cQxqlwvH7HNBKZsntJ4iay5qhcThszbBU4Npgexh KVKLKQvXysIwHK6NuCGsaKzHsPZWHQk4PiCXIt6dbT1tQmki1hDaGdmuFVeY33Ca8algHc 2o2HKEIjtQ4uTtlQJzCNvksNqr5ZI58= Received: by mail-yw1-f173.google.com with SMTP id 00721157ae682-561a33b6d63so74535137b3.1 for ; Sun, 21 May 2023 21:57:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1684731450; x=1687323450; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=dk0QCHZ/ZrkXov8P0kyfxgscgrLXPV5n2YV5/OCWJso=; b=CynFLjQo0B1iyGJQ1Hhn7p6Dwng2pZKqC1sA23EZv3JCM+ZbaT1ahTTm4sUvOZtzCZ BowaOQG29xMNLY0nY/iY+TITfx8rXjfCpXFh5g/iSfQ5lmiD0WCo6dz+EbU/qzow4T48 gIYfejZ6h5BXNTbcNV7JSAg8ejqfun0/YIkYFIX2j/yjnUq7/K2x26jXo8nJSL+Ur/DZ +SUB1nDtFHo1Zu1hnHccttoMBnxyawb0XgPXLVByf6aDR7smWV6B07cuec2Ws0N9yLdQ 8jqD+ttVDEwqJC0ct/mh9eZgT02XjAOsBx+WhPs+42uDF1lV5/G1Bt1qT41SuPS3wH4j Fx7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684731450; x=1687323450; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=dk0QCHZ/ZrkXov8P0kyfxgscgrLXPV5n2YV5/OCWJso=; b=KUZUDig6BbaKUJnciZLqzRZW+5Cl9Y/1ZvwZiMcBm4BerFcOMVtFFVJgEdCplcUssl Ey2f0exE45JCL/DtqCVGD8TWSJ82zBdjT06hGIK81ZbztQi6+wyyz+BFRwrWCZ82oXWf Tx/CEMnst4YFH4Rsp+VQfs/uLr0WtC/Qi+z5c0wxt9uJGQTuKzT8W6jQq1rlT6YsMm9J cOVBG/gZiL8f+jATrds5OaaShXqdw84Xj6osqAua4EfbDCYRS39uHibXBiDyGS8Rqv2J kpVev7QhnZVzQqypwweoTdN5taiZyduhqQPzbMgwKHnh7hB7waco8uCqyVuLUdU5LMD9 5b8A== X-Gm-Message-State: AC+VfDxEVvjkcEAJC2sP5NvD5UkJGra+TnSQ8uH2bTNLjclYZWMbUQla qm56OJG08AOeUAmELe4n9e8eNQ== X-Google-Smtp-Source: ACHHUZ6bURP043biciXrsTo4ID2f5FUMVgcYdQ4w5Fut/Rhs5NuMHxd9s5+H+9KhMfle9QGb5MKZdA== X-Received: by 2002:a81:9206:0:b0:55a:2ce1:2353 with SMTP id j6-20020a819206000000b0055a2ce12353mr9508264ywg.2.1684731449358; Sun, 21 May 2023 21:57:29 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id c130-20020a814e88000000b0056183cdb2d9sm1794222ywb.60.2023.05.21.21.57.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 21 May 2023 21:57:29 -0700 (PDT) Date: Sun, 21 May 2023 21:57:25 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 07/31] mm/page_vma_mapped: reformat map_pte() with less indentation In-Reply-To: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> Message-ID: <4d93bd9-346c-938f-45d0-e073372323f6@google.com> References: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: D50261C000A X-Stat-Signature: dd18kjupa61n4fnfdhit5k758ym4r8km X-HE-Tag: 1684731450-593313 X-HE-Meta: U2FsdGVkX1+VA7cbUwb76ttSBu/6+23D/IAOAsh5MPTJfj3cIGVQArlQjeXLGTOwNrY/ZGVNBpzVX3KSQ+n34Zs4tcpo0GzSNzy9GE19WQVWfnNwn3VgCBabIJwb2B5lS2qsuFkdN3jU8x6XaYbpQvWuVl/gRtYT64ULx2HqRJXWGbtQiyQBC3rsG+WYsQvvkm1PK+hM7PSuAVj/g8aeV1xiyD9CL5zfeQ3vTkInz0GLL3dEaAinKnUCGU/5w5O0DSq7L4wK4/lBzCdYfspaMqC7p7EXf6Dt67OjpmS3aA0/624GndN48odhNU6wYry5XKLZkW4JuEkQp6ODFntEsU96uJOnIPMaWFyyNpMnn3cjAvTFPPo4EQ4m2mdOA45b9grnMk20J/7ACAVYuLJy2O9rAZGnIx7E7uPBV+s9sxDc4ffVKK5wfpOF4FLTDQVaPNWlymhRIPWSjAvEHkPPMTGjOmeQ1SVgDBfJAzMOTqP4Qkn4UcNr5PlHGLCPtADOmP/of+ClhB+2HALgaF0danXREp4jG6n3vNEMxzYSPnYeP4r2Rdhbmqbg+kHdBWQ3s2ZljcSlGZ2DxWHBrruyc/VbdefgRQ5weMNxW6MP21xSau8bDh26wW2tPh4J+V6V5M6vaqvFTKs8nj81jXrbIcm5wL8PnpRpZNPLDax7d4zU0QddSfVWxpeB8RECk6mibEQZkQJZbzMIm9ydpdjXVzu8RTe3HvaPfTP6pTIDW63miLNxUgulf1QeH+aw2HgGelHTDRgNd/jJquzwvNRJHW5nJ6Nuufo1ybGIDBGtI0GvSP7h7shQPDkZrDrYjWDFfGMC3AcR9H6IjB3Q2Paq9z8lmJIPB7/L/tjZzGL6XsbzkPjA1flekBSdxuuGXxBOHB1givnVrdvH0h6mb9157sPwgA38sr+4kbjLAStTcYz0AVmy5XXhQaz6s/6AMYeV/j8RQqbG0yIvIU9gjGh l+e5upMo zHc9xksdtwsQqUQCxq3sSYRTd/YglwbgRG2aHpRkEwwOOLoOkEJSxTLtQ8fGhHD6kwy0JwtTcUsU6vHHdAxEzsuj8Y8Oic9dii4BRz6+vPiQ7RhUFYdyY3YlSpzZG78S1X7z18nKxAe3cfwe4yUefuNFvI+ypEv7KmIfumXD8RLgh18KIr3C1yflXnH1tulpaTfCECpZgumHAR9/zQDeobPoqpr1jYXqx9P3Kh9NflmM0NbCmCXMo71SSjxPrgvEDCB1QWs3aloCcOmYCy+nx3CkIdhlPVl8H5niE5ZgrSiwPV0AAC//7AHkLRQWmUBv0aIhOeYoo/OI8LDn38yFOrGkeomCUayhlBN5y+i9HBztzKDGCR9Sc1hFRYrFcz5Q7lmPtyEhgalRaMnV4J9uGOeGo4kezXJW+T9F4wBqfoCi1VzQSwpygokBp5A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: No functional change here, but adjust the format of map_pte() so that the following commit will be easier to read: separate out the PVMW_SYNC case first, and remove two levels of indentation from the ZONE_DEVICE case. Signed-off-by: Hugh Dickins --- mm/page_vma_mapped.c | 65 +++++++++++++++++++++++--------------------- 1 file changed, 34 insertions(+), 31 deletions(-) diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index 007dc7456f0e..947dc7491815 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -15,38 +15,41 @@ static inline bool not_found(struct page_vma_mapped_walk *pvmw) static bool map_pte(struct page_vma_mapped_walk *pvmw) { - pvmw->pte = pte_offset_map(pvmw->pmd, pvmw->address); - if (!(pvmw->flags & PVMW_SYNC)) { - if (pvmw->flags & PVMW_MIGRATION) { - if (!is_swap_pte(*pvmw->pte)) - return false; - } else { - /* - * We get here when we are trying to unmap a private - * device page from the process address space. Such - * page is not CPU accessible and thus is mapped as - * a special swap entry, nonetheless it still does - * count as a valid regular mapping for the page (and - * is accounted as such in page maps count). - * - * So handle this special case as if it was a normal - * page mapping ie lock CPU page table and returns - * true. - * - * For more details on device private memory see HMM - * (include/linux/hmm.h or mm/hmm.c). - */ - if (is_swap_pte(*pvmw->pte)) { - swp_entry_t entry; + if (pvmw->flags & PVMW_SYNC) { + /* Use the stricter lookup */ + pvmw->pte = pte_offset_map_lock(pvmw->vma->vm_mm, pvmw->pmd, + pvmw->address, &pvmw->ptl); + return true; + } - /* Handle un-addressable ZONE_DEVICE memory */ - entry = pte_to_swp_entry(*pvmw->pte); - if (!is_device_private_entry(entry) && - !is_device_exclusive_entry(entry)) - return false; - } else if (!pte_present(*pvmw->pte)) - return false; - } + pvmw->pte = pte_offset_map(pvmw->pmd, pvmw->address); + if (pvmw->flags & PVMW_MIGRATION) { + if (!is_swap_pte(*pvmw->pte)) + return false; + } else if (is_swap_pte(*pvmw->pte)) { + swp_entry_t entry; + /* + * Handle un-addressable ZONE_DEVICE memory. + * + * We get here when we are trying to unmap a private + * device page from the process address space. Such + * page is not CPU accessible and thus is mapped as + * a special swap entry, nonetheless it still does + * count as a valid regular mapping for the page + * (and is accounted as such in page maps count). + * + * So handle this special case as if it was a normal + * page mapping ie lock CPU page table and return true. + * + * For more details on device private memory see HMM + * (include/linux/hmm.h or mm/hmm.c). + */ + entry = pte_to_swp_entry(*pvmw->pte); + if (!is_device_private_entry(entry) && + !is_device_exclusive_entry(entry)) + return false; + } else if (!pte_present(*pvmw->pte)) { + return false; } pvmw->ptl = pte_lockptr(pvmw->vma->vm_mm, pvmw->pmd); spin_lock(pvmw->ptl); From patchwork Mon May 22 04:58:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13249758 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4CBFBC77B73 for ; Mon, 22 May 2023 04:59:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D818590000A; Mon, 22 May 2023 00:59:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D57D2900002; Mon, 22 May 2023 00:59:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BF7FA90000A; Mon, 22 May 2023 00:59:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id B1D0D900002 for ; Mon, 22 May 2023 00:59:04 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 788571C71C6 for ; Mon, 22 May 2023 04:59:04 +0000 (UTC) X-FDA: 80816686608.21.9301B8F Received: from mail-yw1-f182.google.com (mail-yw1-f182.google.com [209.85.128.182]) by imf01.hostedemail.com (Postfix) with ESMTP id A08F340014 for ; Mon, 22 May 2023 04:59:02 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=TGvacnGG; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf01.hostedemail.com: domain of hughd@google.com designates 209.85.128.182 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684731542; a=rsa-sha256; cv=none; b=1qbuEintHSm2TB9g2Bpi1V6QVTiysM/kg4xsjn4EAg4lr4cclKbiR9BygsWcnwZmc4+EAZ 1uzPRlKpBraTeZXpOyCPDIv8Op2b0ZzEBE1g9qgyz4rxQSuHDUxpVZbpMIo4mB8RH3ArIg JdXC+JLJMc+6gk2Zrm7z5qgTBHZQUZo= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=TGvacnGG; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf01.hostedemail.com: domain of hughd@google.com designates 209.85.128.182 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684731542; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9VYOFAJpcVHpYrrS1TQZcIVdYEqSjI0GrwCgA5USRTU=; b=6bmMIaF6z7XWfZarLf7fI9qeRo8t9rU/U7RhKOLIyAR0omqCPIlfMw0buIyh6no1jc0934 dV34BQz8dXfjwBBUkNCDdwJpjhMa5zkSE2kmCIAeBhwjbh1J0rBFYwNI4HYJ4b67Zupdup x5xmHAqzM0i1bmjjAP2yY6BuIE0wljc= Received: by mail-yw1-f182.google.com with SMTP id 00721157ae682-5621a279cbbso47448737b3.1 for ; Sun, 21 May 2023 21:59:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1684731541; x=1687323541; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=9VYOFAJpcVHpYrrS1TQZcIVdYEqSjI0GrwCgA5USRTU=; b=TGvacnGGV6vCjNoR60zApDSkblq671aMycg9KRLogEr52U+VDqJD6c9LnImDcePwXP E+oOLSRKY3QnKjySzwhyx6yXTUVprQkKKmDGsyxWBnNRAxybbHWPE+dZuUWqHGT3elWd KJzJqVN+sTGBo7NuVZnh1kpo/iER1/+XPOJgROkPPAZo27SecID8MF0vNrn6JwCjXzCa UJM1gcqwPlnlc9FLHaIRUz/YKXEVOCkEBbD5opmzBt0US/6C6qPiSdxH6LlPD5UCxU/z zjtg+pK6HvaDdyOshiCuMXpAmIJoOt9KExIMuIDdOzn/Rpr/BNUdc7KJhSnoFWJGvEir v0Cg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684731541; x=1687323541; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=9VYOFAJpcVHpYrrS1TQZcIVdYEqSjI0GrwCgA5USRTU=; b=d/Fip2g73q57bU5dK+06b9nS12uIAQzBkjcMecnxAbj+ZjRl8q/vb9Avv3LOG6IHnw 60vvjWgUQGgDkLsuYoXePbdQJ6FCHotleXyh+NxrPWeUZSGFuXA/kQS8fkN414TlrjbV v+FsRhAAJwjDQ3dXQzkYWs+TP9JK1BEURBB0zu/ydM45ehlmLqwlJsNVK/zzpqeUgCMN b2r9yv9AmzkKzdFRaw/Fsg3ClpvPwq/Mbtl2wicXuSq+1wIEq7/MZ2fEH9s3si5zupY1 iB/Nv7UTWo846BBFOPaBGB6EFUm0EawpI4tqQP/3B9WbYBcAbfXO2xGHJY7rWCwI4ENB 1YcQ== X-Gm-Message-State: AC+VfDwFHBd1E9s0kOzlMCxGs/QGEaN0tUDR3TEWkfm5Z7xP4dZ1YeyX xeVLVGGaiqNgxR6AbF3V1pEOpA== X-Google-Smtp-Source: ACHHUZ4Kg3oHxKZR/9CKF00axRzhTRl1agKzGrSrlAst7zhZTlHx2mXvNeak2L2x7OdVQNGzScOvHA== X-Received: by 2002:a81:1e45:0:b0:560:d022:53ac with SMTP id e66-20020a811e45000000b00560d02253acmr10367255ywe.5.1684731541606; Sun, 21 May 2023 21:59:01 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id g189-20020a8152c6000000b00555e1886350sm1827794ywb.78.2023.05.21.21.58.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 21 May 2023 21:59:01 -0700 (PDT) Date: Sun, 21 May 2023 21:58:58 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 08/31] mm/page_vma_mapped: pte_offset_map_nolock() not pte_lockptr() In-Reply-To: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> Message-ID: <8fa3fb6e-2e39-cbea-c529-ee9e64c7d2d0@google.com> References: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: A08F340014 X-Stat-Signature: 99nb36eyb65zihoki38bz46oddxingxk X-HE-Tag: 1684731542-256035 X-HE-Meta: U2FsdGVkX183xsEBgNccYX5Fj2sW6ZeBWg2MH3ma3yTqITt1zV7B+ZpE7A2H8JKaf2FwLP1tcBttWhRgaC2rwU06m6oZZHl+HwPJuiQC9bPJIgPhqmy/ilV8HsXRwixffj2YoH4VzvJw0A5BPV1eAMPuNVlQRXKplgUtVPstToIOSEwHcX+HlDUmntaDKp1flcTzgtnxFTfAXDbUhAZRq0hzzB/xN0NagXvelzJ7/WA6VCtmvphUiEowF1GoE3Wdq3WV3U2qHa8zGSoZLBXs6uQi1VXr/mg8MWuEVSnpaNvKRE/48PY/vaLHezryy+mbNOYvI0pLHxPYjvGQjMmRFwliiCrzMIkD5TEgG/F3bRQdc6nWdVHAgfTBLtaIhVXM92zqpP4HZRwuUgpP4aEd1wZ7L8WaIQT291SZfr8jxRd49ENBnSWSoxhu82+rSUtERwZxOxJ6H3BHK8VmAq6Q7Nn3u7ETUD72EBMRiZ62iNvOTmTTeWbnQNmoX2IOpPEnboKmAPr2k6seWU1ehCuzSFwMR+9cuNpG4ASaJUDfqfwhoO0q51xAgWI6bWcQdZ6jK+kYUdmM2DjkKlhs1ZJDx6lgP75e0wr4FxGAzvEiSlnOqV70p8F9SbL7MqnKMbeSjc5IWgqq9nPlR8N3jvdTESr/LVo/lQD7IPvOi/65vtazpuA+idGakoemUJpyorVlJarEqAJVggbiNNm2XT0KqaE0DlDEIO5HCNagotRiK1o3Xa2Q7D2rTQ739NGk5TrX6HCqdSjonaO847IOCQHQ94h4bDibpdqdrHS+qH/dQfTh6GhIEzPzwkj9FCNAs3d3zbvuqvR8dnuES2VXAeKW2KMSNYFeOr2mhqRc39oO8mab66DjmKoOD7sxDj4q0vKBH/2i3BbQZblvbeo+KOq/9bdb1pp5szQMHriRX0ltow0CSxIlTWXjyX06+2HA5fOVLyGS1IgUzPDjHC6yBDu +5KwA8Qh 8xlMaU4VtYsRzrmCqjRk6ZH8BvXsN81IXxdU68c/ucEzpQTgK8QhI0r/VT0KdbCqmhFloRPtYNtDald3JkSXayZRQqv5QlCcETYnOrlwhAsTpnOuEY96pl1jEtvXKLF9nB7ca/bh0ejpAMk0mS0pEyQ1hW+YnOUkaflW4ho04aaFR40rUeY6MCLtxJZNU737/TicZl7RKCRO35RG2BH5+GW2wdgdeADz9ZdQXnvKgGGts0AnQ/jEL84p5cn/sbiPp6F0uXE6n6GPhN7SDcJogz4LxBTYdhjaJTTMdy72hL4aX9xZSmZp1dq0LlC7/8s3VU1mwaBFXhpKLOX4Pbiuyr7tVPLD5YsZQibgh5IpD8eC1DVnjSZJcEip6OaRwp0Y0v+H04wNQpUT96hnCawvSZ0+OuTpg9feRUaE4Ck6ff7krI1IUPID2usiOPQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: map_pte() use pte_offset_map_nolock(), to make sure of the ptl belonging to pte, even if pmd entry is then changed racily: page_vma_mapped_walk() use that instead of getting pte_lockptr() later, or restart if map_pte() found no page table. Signed-off-by: Hugh Dickins --- mm/page_vma_mapped.c | 28 ++++++++++++++++++++++------ 1 file changed, 22 insertions(+), 6 deletions(-) diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index 947dc7491815..2af734274073 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -13,16 +13,28 @@ static inline bool not_found(struct page_vma_mapped_walk *pvmw) return false; } -static bool map_pte(struct page_vma_mapped_walk *pvmw) +static bool map_pte(struct page_vma_mapped_walk *pvmw, spinlock_t **ptlp) { if (pvmw->flags & PVMW_SYNC) { /* Use the stricter lookup */ pvmw->pte = pte_offset_map_lock(pvmw->vma->vm_mm, pvmw->pmd, pvmw->address, &pvmw->ptl); - return true; + *ptlp = pvmw->ptl; + return !!pvmw->pte; } - pvmw->pte = pte_offset_map(pvmw->pmd, pvmw->address); + /* + * It is important to return the ptl corresponding to pte, + * in case *pvmw->pmd changes underneath us; so we need to + * return it even when choosing not to lock, in case caller + * proceeds to loop over next ptes, and finds a match later. + * Though, in most cases, page lock already protects this. + */ + pvmw->pte = pte_offset_map_nolock(pvmw->vma->vm_mm, pvmw->pmd, + pvmw->address, ptlp); + if (!pvmw->pte) + return false; + if (pvmw->flags & PVMW_MIGRATION) { if (!is_swap_pte(*pvmw->pte)) return false; @@ -51,7 +63,7 @@ static bool map_pte(struct page_vma_mapped_walk *pvmw) } else if (!pte_present(*pvmw->pte)) { return false; } - pvmw->ptl = pte_lockptr(pvmw->vma->vm_mm, pvmw->pmd); + pvmw->ptl = *ptlp; spin_lock(pvmw->ptl); return true; } @@ -156,6 +168,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) struct vm_area_struct *vma = pvmw->vma; struct mm_struct *mm = vma->vm_mm; unsigned long end; + spinlock_t *ptl; pgd_t *pgd; p4d_t *p4d; pud_t *pud; @@ -257,8 +270,11 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) step_forward(pvmw, PMD_SIZE); continue; } - if (!map_pte(pvmw)) + if (!map_pte(pvmw, &ptl)) { + if (!pvmw->pte) + goto restart; goto next_pte; + } this_pte: if (check_pte(pvmw)) return true; @@ -281,7 +297,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) } while (pte_none(*pvmw->pte)); if (!pvmw->ptl) { - pvmw->ptl = pte_lockptr(mm, pvmw->pmd); + pvmw->ptl = ptl; spin_lock(pvmw->ptl); } goto this_pte; From patchwork Mon May 22 05:00:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13249759 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 86717C77B73 for ; Mon, 22 May 2023 05:00:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 283FD90000A; Mon, 22 May 2023 01:00:23 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2345A900002; Mon, 22 May 2023 01:00:23 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0D57E90000A; Mon, 22 May 2023 01:00:23 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id F13EA900002 for ; Mon, 22 May 2023 01:00:22 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id BBEAEC045B for ; Mon, 22 May 2023 05:00:22 +0000 (UTC) X-FDA: 80816689884.08.117DEC1 Received: from mail-yw1-f180.google.com (mail-yw1-f180.google.com [209.85.128.180]) by imf20.hostedemail.com (Postfix) with ESMTP id 6F6151C0021 for ; Mon, 22 May 2023 05:00:20 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=gG5CM9oz; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf20.hostedemail.com: domain of hughd@google.com designates 209.85.128.180 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684731620; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=UuVwG0QveaZLADVtUaHSgqKzTTEqSjST+aR027Vb5lo=; b=nFuug4z8RtzYiwfH+Fw3I5A2S/Y2J/puSTkzahZIG7nY1cYsIgljDWLb6mRqKE0vQy7KPg k0mQX1mGlmKgN93KpMuMVTJbEMLK2rR07lHwgkz8zQrL165VBT1p3bELnk7nyzp89r1pMo wglxqvtI5Ra01Du/HeLIyOzcxhO0S/o= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=gG5CM9oz; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf20.hostedemail.com: domain of hughd@google.com designates 209.85.128.180 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684731620; a=rsa-sha256; cv=none; b=3cRJB8QSQ2gG9Qbyecy/M5Lfob2OjiT+CCOCzRHuNVpr0cR2n747z6a/xOICndIoSygFN9 +LaZSWQzk7V4HamTpGoo2L2wJb3sN5uUj2/s53uOrva7kk4KhDPLQaAj2APlt25hZs6cxQ JwJbPkRBCvoSzFXtM3qp8qBfs/gZOEc= Received: by mail-yw1-f180.google.com with SMTP id 00721157ae682-561c5b5e534so74214067b3.2 for ; Sun, 21 May 2023 22:00:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1684731619; x=1687323619; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=UuVwG0QveaZLADVtUaHSgqKzTTEqSjST+aR027Vb5lo=; b=gG5CM9oziEo3d2xuTQZP2LZGLbyhx6n0Gd07VVIcPAplxjY92fK4ZBhk/PKkZAtjDN 5xbQCSYDeQd+vsPz3UctEBNNQiAW1PG9SnA0aX9yoaPh6erzg7iRa3/rTzxeaqzGLp7M lSM/wuLv0LneLKuxocS16HoYAdzNdVswju/csw1C62EZM/zSg7SkkxlvZsPb/qR4YDiI JxhcXWwQQZon4YC+DZWlfAheKEVu+DJEgm81Ja53+XH3TYiVcdE+XisUHVOIOH3c+bhB 84wXaTSFtqGDpcIfa2zJuBidBN4AfMzBHzdLZY7lpccTQVYK/Jpluc7HpY0MDlN7jzqs jXCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684731619; x=1687323619; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=UuVwG0QveaZLADVtUaHSgqKzTTEqSjST+aR027Vb5lo=; b=gaZ5GswZW38atTz6uHarHkuwJZH0cPmySeYk3RcSiCsW4ckJMjTBNHPsFoIPbgPV38 Wi8I/S6Bry3tgnQdYrUxxPzgTkxEnGo3xz+sBsWd17iOm+pfAcG2B5FXU4ItmDtl6oMX bxMFbBpk54CGQmtjHTGvVOK6R/77YAeRKm7KxU38bMQ3C2E+XEONYnnlzkBgt+W9OtOV Oww3Z2gQ3mi4TttoO6AAIsit9zEcHCtpqqwS2m2RYiKF1dhMdbV4QDCUlyzJmb1WH8Ju XAenoYhqgHLAxU+Y9WLH0DkN5nym5yZnUXVDgDWNICYkv7xsdPbPSUPTlTJS8A/zi8ZK ws5g== X-Gm-Message-State: AC+VfDzEWKB/g9PTD1AXyiDuTUouBOoWzaqEqH5T78uvEtSd1ggXe/3V A6ERdLJphEwZCyG6AE6Beh8Spg== X-Google-Smtp-Source: ACHHUZ7OcXDmRntzMQt9n+hv5KPWL1NcuSYc2shkgSn7zkVgG1IKDeMbUgq8Rk9jU0J7GbmzBFEDoA== X-Received: by 2002:a81:6d09:0:b0:561:902e:dc0a with SMTP id i9-20020a816d09000000b00561902edc0amr9837501ywc.32.1684731619134; Sun, 21 May 2023 22:00:19 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id r63-20020a815d42000000b00555df877a4csm1794565ywb.102.2023.05.21.22.00.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 21 May 2023 22:00:18 -0700 (PDT) Date: Sun, 21 May 2023 22:00:15 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 09/31] mm/pagewalkers: ACTION_AGAIN if pte_offset_map_lock() fails In-Reply-To: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> Message-ID: <6265ac58-6018-a8c6-cf38-69cba698471@google.com> References: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 6F6151C0021 X-Stat-Signature: tcg5gnptxnqtja9icmrf569r9troji8g X-Rspam-User: X-HE-Tag: 1684731620-213481 X-HE-Meta: U2FsdGVkX1++eZlvW2jAl/4dyQu0okHUrBM/PxMS8W5W0yqiYO8hW/i+7mB7JfL9zyGe+sxlRD6cJJFvUSqNFiZlxqurnJ4616RPmWzLGrFfDgZznTPWiSrvX3vnx3fY70PeK4azPVNGLwpY90A3SO/aTwiYCLwbG/5FabBzJWM8mzKYgWozVyA9WBC4f2uh5lBlzxdPUr+0L0A/eiSm+FqddyvRjBTsgo2wkRLtbIdHwgP/ZfhAuPPKxodu/jUYg8ruwFK6tkdT0W68+G8yLs/k6GpoFG1v1JZ1qe8A8ybIHoxRKxLQxQHibnRQVUGTN9bwvGC6jOdhKY0bgxxIz5GItcbdqoid80WZ82DD2ZqgOUw31zB3jT6LvWfRYIj0/W3KrKKQjS1oHaD77iS+G472JHW8+tcQeoVLQ1h3hx1Z+XEiEvyiN4zNk7dtWwOirFUQQSvffnS10EtEWPUGyoPoIZyj5pbscN7SLgZYQ4kvDtO+YMxzzRWchDmIvmjCovYxDfrnzU1p95TXOnEy6rDO8W66gUpKo/t3Cs38q0iFTEbCseimg0ZJ9GOgBbm67+sS2C8gnSBXxOX+mb1qcmVUaQkBCHqgJZhDKQCoA/1qcHEnoDV2xKy0bbxN9vOfoa0IL0eS+SvpOIYemkPoSjHlTZ9qtK2zZYt+wHPYiQtcwHElbBB/LgUsW3Dt4y0UpnY65PVDrAUGd3t3zGWV5aRByYAQuRPwTIQiWMxE9x/OKEahBbPxFO+lLNWskyxNM0kjCqugwXF+TqJ1l9VnqMSkjpHbZKNgqOgpUKvH6AJ9W1M0ocOVdxTXWR4bmyY4blF/VXg2l7PNntdVczAUSHylNL3uhZGaP/werI1JEm8tIDe78Gxlu/aqkf0Lo7o1Hm3V0pH0Ge9f1vzJNvv3kdVP1LDsaZcyNaKi37DUjrdxvLXIFJeHu/urffmTDxzQA6Vha68FaCWRvrnA1GB dqUhFNVd 6vEU/RhOy3yiW6VBaql9xuPM1MKdW6L0cMe6Q15iZvXILIv3VcHo2Axz7fvStwmOPV/zptyEcIkgKC+Go+++OSnX7NEhG8xmx8Jl5ysS/5Y6Z3E7/suHbd5FynJZTVvQtEVfFsUR756TqbaVeHqHazK1JaqQQmCCZa8G29S9BPI+phE9/E7/9xOufguyyRmrTDKBjOQA1OsGaVhi97zD0TjGp3cWjoOdeJcOtLGdgVGlVsafsTm6M1mmnXJoAcZm1vH0t1AY6kaGDqnV42jSTLhG8uAuUFMWporh22Y3sAkBfQM2uxm2dcjFQCu9iXXK7iDpDg0wP3Z5boMDvhKxitr+acBVUlLFn8M3chxT/qs+ZhtNzMPpyc+35BDoo8CwXeFcA6RUvGM1UBTMHnIYySstRZEg70tGwQ+B98cPOzZZer5BCSHCGCRF/fB7csYjUBEFNyXY8dtYEfMZjRssp11Zkqw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Simple walk_page_range() users should set ACTION_AGAIN to retry when pte_offset_map_lock() fails. No need to check pmd_trans_unstable(): that was precisely to avoid the possiblity of calling pte_offset_map() on a racily removed or inserted THP entry, but such cases are now safely handled inside it. Likewise there is no need to check pmd_none() or pmd_bad() before calling it. Signed-off-by: Hugh Dickins Reviewed-by: SeongJae Park --- fs/proc/task_mmu.c | 32 ++++++++++++++++---------------- mm/damon/vaddr.c | 12 ++++++++---- mm/mempolicy.c | 7 ++++--- mm/mincore.c | 9 ++++----- mm/mlock.c | 4 ++++ 5 files changed, 36 insertions(+), 28 deletions(-) diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 420510f6a545..dba5052ce09b 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -631,14 +631,11 @@ static int smaps_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, goto out; } - if (pmd_trans_unstable(pmd)) - goto out; - /* - * The mmap_lock held all the way back in m_start() is what - * keeps khugepaged out of here and from collapsing things - * in here. - */ pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); + if (!pte) { + walk->action = ACTION_AGAIN; + return 0; + } for (; addr != end; pte++, addr += PAGE_SIZE) smaps_pte_entry(pte, addr, walk); pte_unmap_unlock(pte - 1, ptl); @@ -1191,10 +1188,11 @@ static int clear_refs_pte_range(pmd_t *pmd, unsigned long addr, return 0; } - if (pmd_trans_unstable(pmd)) - return 0; - pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); + if (!pte) { + walk->action = ACTION_AGAIN; + return 0; + } for (; addr != end; pte++, addr += PAGE_SIZE) { ptent = *pte; @@ -1538,9 +1536,6 @@ static int pagemap_pmd_range(pmd_t *pmdp, unsigned long addr, unsigned long end, spin_unlock(ptl); return err; } - - if (pmd_trans_unstable(pmdp)) - return 0; #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ /* @@ -1548,6 +1543,10 @@ static int pagemap_pmd_range(pmd_t *pmdp, unsigned long addr, unsigned long end, * goes beyond vma->vm_end. */ orig_pte = pte = pte_offset_map_lock(walk->mm, pmdp, addr, &ptl); + if (!pte) { + walk->action = ACTION_AGAIN; + return err; + } for (; addr < end; pte++, addr += PAGE_SIZE) { pagemap_entry_t pme; @@ -1887,11 +1886,12 @@ static int gather_pte_stats(pmd_t *pmd, unsigned long addr, spin_unlock(ptl); return 0; } - - if (pmd_trans_unstable(pmd)) - return 0; #endif orig_pte = pte = pte_offset_map_lock(walk->mm, pmd, addr, &ptl); + if (!pte) { + walk->action = ACTION_AGAIN; + return 0; + } do { struct page *page = can_gather_numa_stats(*pte, vma, addr); if (!page) diff --git a/mm/damon/vaddr.c b/mm/damon/vaddr.c index 1fec16d7263e..b8762ff15c3c 100644 --- a/mm/damon/vaddr.c +++ b/mm/damon/vaddr.c @@ -318,9 +318,11 @@ static int damon_mkold_pmd_entry(pmd_t *pmd, unsigned long addr, spin_unlock(ptl); } - if (pmd_none(*pmd) || unlikely(pmd_bad(*pmd))) - return 0; pte = pte_offset_map_lock(walk->mm, pmd, addr, &ptl); + if (!pte) { + walk->action = ACTION_AGAIN; + return 0; + } if (!pte_present(*pte)) goto out; damon_ptep_mkold(pte, walk->mm, addr); @@ -464,9 +466,11 @@ static int damon_young_pmd_entry(pmd_t *pmd, unsigned long addr, regular_page: #endif /* CONFIG_TRANSPARENT_HUGEPAGE */ - if (pmd_none(*pmd) || unlikely(pmd_bad(*pmd))) - return -EINVAL; pte = pte_offset_map_lock(walk->mm, pmd, addr, &ptl); + if (!pte) { + walk->action = ACTION_AGAIN; + return 0; + } if (!pte_present(*pte)) goto out; folio = damon_get_folio(pte_pfn(*pte)); diff --git a/mm/mempolicy.c b/mm/mempolicy.c index 1756389a0609..4d0bcf6f0d52 100644 --- a/mm/mempolicy.c +++ b/mm/mempolicy.c @@ -514,10 +514,11 @@ static int queue_folios_pte_range(pmd_t *pmd, unsigned long addr, if (ptl) return queue_folios_pmd(pmd, ptl, addr, end, walk); - if (pmd_trans_unstable(pmd)) - return 0; - mapped_pte = pte = pte_offset_map_lock(walk->mm, pmd, addr, &ptl); + if (!pte) { + walk->action = ACTION_AGAIN; + return 0; + } for (; addr != end; pte++, addr += PAGE_SIZE) { if (!pte_present(*pte)) continue; diff --git a/mm/mincore.c b/mm/mincore.c index 2d5be013a25a..f33f6a0b1ded 100644 --- a/mm/mincore.c +++ b/mm/mincore.c @@ -113,12 +113,11 @@ static int mincore_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, goto out; } - if (pmd_trans_unstable(pmd)) { - __mincore_unmapped_range(addr, end, vma, vec); - goto out; - } - ptep = pte_offset_map_lock(walk->mm, pmd, addr, &ptl); + if (!ptep) { + walk->action = ACTION_AGAIN; + return 0; + } for (; addr != end; ptep++, addr += PAGE_SIZE) { pte_t pte = *ptep; diff --git a/mm/mlock.c b/mm/mlock.c index 40b43f8740df..9f2b1173b1b1 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -329,6 +329,10 @@ static int mlock_pte_range(pmd_t *pmd, unsigned long addr, } start_pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); + if (!start_pte) { + walk->action = ACTION_AGAIN; + return 0; + } for (pte = start_pte; addr != end; pte++, addr += PAGE_SIZE) { if (!pte_present(*pte)) continue; From patchwork Mon May 22 05:01:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13249760 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2E4E9C77B73 for ; Mon, 22 May 2023 05:02:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C478690000A; Mon, 22 May 2023 01:02:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BCFFE900002; Mon, 22 May 2023 01:02:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A991D90000A; Mon, 22 May 2023 01:02:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 95F87900002 for ; Mon, 22 May 2023 01:02:02 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 6D66A1406B1 for ; Mon, 22 May 2023 05:02:02 +0000 (UTC) X-FDA: 80816694084.17.215ECD0 Received: from mail-yw1-f180.google.com (mail-yw1-f180.google.com [209.85.128.180]) by imf02.hostedemail.com (Postfix) with ESMTP id 9C51680013 for ; Mon, 22 May 2023 05:02:00 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=Q+WLY7Ro; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf02.hostedemail.com: domain of hughd@google.com designates 209.85.128.180 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684731720; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=s/wneZk+WXTrjMA3fZuua9PXiv3fUuHnEiWoD80tgK8=; b=mhV/xeyYWEhdRXiGl/v2X560i4MBHD3gsaxPudVfcHZwkVfVcUO6FMjYFb+PgwiCiOBYQs 5WtF8/0P4aq2gldA2cQPj9DlqJ+ubDOeZVvlh/dyxphL3T+HBOIGtWOPRqHg+3zjTCA5jq jFkzfUmskflY8PRaBoFVOELJ0k99iVU= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=Q+WLY7Ro; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf02.hostedemail.com: domain of hughd@google.com designates 209.85.128.180 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684731720; a=rsa-sha256; cv=none; b=iFrEIgcnlZh8RSS2mRawXEapMNSjqNA0KK782I0DO031Kz5DKS5VrWH/mIEnjMeaTQpHOY CQsVb6YcSMT6LX1rNTcQSGDO4lcO2aMw+PA9gK5RdvtP75h63/XwYZN1p11bO2biZ2rA44 O8ZnbnAKf9Sm7hV+mQN8UzYlqBKtE2s= Received: by mail-yw1-f180.google.com with SMTP id 00721157ae682-561b7729a12so45689857b3.1 for ; Sun, 21 May 2023 22:02:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1684731720; x=1687323720; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=s/wneZk+WXTrjMA3fZuua9PXiv3fUuHnEiWoD80tgK8=; b=Q+WLY7Ro5SD5yNUAGTclzTje+VDS5OJwsYctg43k8TXzrg8fckZAOFDLbOpbv+Cz4d vusZX67zn6q4S5qUq6SfyoErvRul9hzMJrB3MRNnhQ6C5pPl/f+0Ug23z4gIrshnXudw uyL2XPYtPcLeaaYEa9djIcbbIiLW16k5noNko0Ixwnf5lma2IwkwvbUb6+ol9PyhD+iA SFpbS9+EhhrldJnU+af1fSiFWnx4LWMy4hftbnHOIsp4fFcbSzoqMd7Z+gmLZKCZGF9K xNYQslTPradcxr1kOt1BAZkUVCQ1AUWpnoyxrlCIdMtfrWBTqYrF1ShOS0ygR98FffYZ BJig== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684731720; x=1687323720; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=s/wneZk+WXTrjMA3fZuua9PXiv3fUuHnEiWoD80tgK8=; b=PkfbMPBxn9AZMAtVTrjNRdYIfdULvS02lkFAIAQoe6t4Sx5H1+vGbl0gK/k0SFV4LR GAbN0NTFwhyUAOJ+YgElw5PsnELvv68HLjw+hshg5r74gOjRnXOF5nJeR2Tno0aZyGxB eFN0/qoyHqCLrQWTZ7pNXq1IvJ/vFIP+4K5Dq9yA8Q8vnkyXIx8ej1ZusCm0YR+YDIEX C3VznxKgsIWio4eWJLXsLMQB0yc1AJGJjqlXfBnc3liENXsjRYWzYM5cnSXdDS1xFyki Mgm7Ke3/X1rfENJbVXZLhHWDUYrWf6gxpb/RTotYQFEQUfVkMuM2PKq06PRTgUMn8aT3 +zEQ== X-Gm-Message-State: AC+VfDxlpX1ZIXbyUqA89utZnh6/vSXn7CPWeyzNHoUVJdQxaFEl14ad rdwQWpGHwfELYAmKKFcvrKE+mg== X-Google-Smtp-Source: ACHHUZ62zTVVW00Kw9P1qNrMirZM+5qrpLkNDTDbJjpW5hqzfUZErYJorH+/umfKDMFCMPWuq8GGOg== X-Received: by 2002:a0d:cc45:0:b0:55a:3502:d2ca with SMTP id o66-20020a0dcc45000000b0055a3502d2camr10051671ywd.13.1684731719564; Sun, 21 May 2023 22:01:59 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id x67-20020a81a046000000b0054fcbf35b94sm1819620ywg.87.2023.05.21.22.01.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 21 May 2023 22:01:59 -0700 (PDT) Date: Sun, 21 May 2023 22:01:56 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 10/31] mm/pagewalk: walk_pte_range() allow for pte_offset_map() In-Reply-To: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> Message-ID: References: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 9C51680013 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: 36ef5n6m1cmo58xorbzdb961jt6tw45d X-HE-Tag: 1684731720-16481 X-HE-Meta: U2FsdGVkX1/M2EflRfGVu6FLEBlY4u1h8Az2bTX9xZ7i+4UovpMyKTkqznqJgOnoGCsdc6PJ2krYMHaJ+ZWO8sj31c3z4cNozMt1ybU43lP7YPsqtsbs9BYYsD9zJSrAOJZ3DhKpFuyrGgAhM8E+W1P4bR7Uv+MGHdc4+qJEUyJwaaO3TRz6xVY+i4oJaJSrqlIJk9pTWGNT5pEyVVDDTJDU8x5i77xXvv6hQMplX+8l6msm44N6MFs+4q3ERYpOa2Z9nf9CIPL4QRrKsSKLXZG5k7J9y3L3t/AuM7p2JyMpFvC48MHZnBTLAqolpBkt1hCAo8RfgSWGgUprCkoD7NdWAx3RCoT1sSUfHGEP6J7esGt+MHH+kjM4OdLwdVSyU+CL8zV868tyzc7y/Xy2JGO/LenIYxnI4VIyhstIlroAuv73InqNAWFRNC98lD4S6nFXerDVR+aUTyvaw7AweqnFaOM3aDN1a+ZSR9crzVNPMbOsQHPmEgNc4+L2jVCMHTOcz95zvqFhDtgtCJnf3me4SZHFPyTwUF2f7tcDTXDmLtRpox5BJi8Jx8eUrVrGzLml0L4yGlzBMNBPuMK/4JoMkj/EyojwDC7yhifR4DrlHXc0lWixpvldkFtfINyelzu+t88cly6M31GfAarf74vSUeS7pPup5J3IBiV3cDJarwL51S1WPXGRkz5qiMAe/0TgTUg/u18njfK2dzJrIpQMvHXBTJStiYT6QE40NO+09+wb1gAF5hPn9Jd0qovJCBsHf2FjEGhI8PtZP3XYnTjkL4dizPRBkjdB6btwmvG8zXaUad80+spmyipzbu6PkiaL8nCJGihmSFHJZgkBpzkuoX1LJ+vl58gwJ8kg+a7PtdSRYSSSe51Vc0XtzW++Md4g0crJIdtYJ0Cpx/D0APp8NYP0/2d2b0LVvOXq8T5eCF13aCQBaq8eSkSBKqjZzNgyF6JuJx6yn0Dt7po C45WBEyR 4sOCVCoD9lWtUYCAjK4SRyfC5xlRwEXexrAEwbcg41usM2lciMAuIOJzY7nO2BCgEtuRGJmAh0Y1Rc68UoIlYBXxbhrzZkaUBStEsEzEw1qwdoKTi0bAiUcVnJQDg215uDTU8ak1SM/cvgibcxuY4LgCKRziXc5pn/wa6YRJnFhI+ID2SriWgdd4bo+cLOr+IEVRoZILwwqIDuxSMhEUxZiDo/Au2X2YsrhoNS+vubwZT2i7hvQoMEchOECqm5r2RlBDtDln5f3nZRqTSGtYpLsQ3wcZwsCDFGHWEvj+9OWzzsCA8wSJ5HlchXaI33oV14sodXCkGCHADw1Xd1gokQlMDWRI/FGpt+UWS8WVhIzbG7N4jGIxpX1K0eqOrqEys0AC9Rsq+pi4MgYEMFTwfqXIGIjYWPVrUqahemVZYOyDJi+frc/nuhyAo1A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: walk_pte_range() has a no_vma option to serve walk_page_range_novma(). I don't know of any problem, but it looks safer to check for init_mm, and use pte_offset_kernel() rather than pte_offset_map() in that case: pte_offset_map()'s pmdval validation is intended for userspace. Allow for its pte_offset_map() or pte_offset_map_lock() to fail, and retry with ACTION_AGAIN if so. Add a second check for ACTION_AGAIN in walk_pmd_range(), to catch it after return from walk_pte_range(). Remove the pmd_trans_unstable() check after split_huge_pmd() in walk_pmd_range(): walk_pte_range() now handles those cases safely (and they must fail powerpc's is_hugepd() check). Signed-off-by: Hugh Dickins --- mm/pagewalk.c | 33 +++++++++++++++++++++++---------- 1 file changed, 23 insertions(+), 10 deletions(-) diff --git a/mm/pagewalk.c b/mm/pagewalk.c index cb23f8a15c13..64437105fe0d 100644 --- a/mm/pagewalk.c +++ b/mm/pagewalk.c @@ -46,15 +46,27 @@ static int walk_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, spinlock_t *ptl; if (walk->no_vma) { - pte = pte_offset_map(pmd, addr); - err = walk_pte_range_inner(pte, addr, end, walk); - pte_unmap(pte); + /* + * pte_offset_map() might apply user-specific validation. + */ + if (walk->mm == &init_mm) + pte = pte_offset_kernel(pmd, addr); + else + pte = pte_offset_map(pmd, addr); + if (pte) { + err = walk_pte_range_inner(pte, addr, end, walk); + if (walk->mm != &init_mm) + pte_unmap(pte); + } } else { pte = pte_offset_map_lock(walk->mm, pmd, addr, &ptl); - err = walk_pte_range_inner(pte, addr, end, walk); - pte_unmap_unlock(pte, ptl); + if (pte) { + err = walk_pte_range_inner(pte, addr, end, walk); + pte_unmap_unlock(pte, ptl); + } } - + if (!pte) + walk->action = ACTION_AGAIN; return err; } @@ -141,11 +153,8 @@ static int walk_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, !(ops->pte_entry)) continue; - if (walk->vma) { + if (walk->vma) split_huge_pmd(walk->vma, pmd, addr); - if (pmd_trans_unstable(pmd)) - goto again; - } if (is_hugepd(__hugepd(pmd_val(*pmd)))) err = walk_hugepd_range((hugepd_t *)pmd, addr, next, walk, PMD_SHIFT); @@ -153,6 +162,10 @@ static int walk_pmd_range(pud_t *pud, unsigned long addr, unsigned long end, err = walk_pte_range(pmd, addr, next, walk); if (err) break; + + if (walk->action == ACTION_AGAIN) + goto again; + } while (pmd++, addr = next, addr != end); return err; From patchwork Mon May 22 05:03:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13249761 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 88905C7EE23 for ; Mon, 22 May 2023 05:03:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2D0ED90000B; Mon, 22 May 2023 01:03:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2805E900002; Mon, 22 May 2023 01:03:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0FAA390000B; Mon, 22 May 2023 01:03:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id F0447900002 for ; Mon, 22 May 2023 01:03:12 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id BEC5F16042D for ; Mon, 22 May 2023 05:03:12 +0000 (UTC) X-FDA: 80816697024.25.BCA210B Received: from mail-yb1-f180.google.com (mail-yb1-f180.google.com [209.85.219.180]) by imf03.hostedemail.com (Postfix) with ESMTP id DE7932000A for ; Mon, 22 May 2023 05:03:10 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="Kc//GQWt"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf03.hostedemail.com: domain of hughd@google.com designates 209.85.219.180 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684731790; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=W3t6elq9R4Rh1GonIcnIp9HMVNr0t0fb96tM78h+6lY=; b=5ZjIvgWN2X1oxTOekAmJ6uFfJ8bH8aDlz+fccXxWy38HYApExd0Km/FV4fOp7vk5e3tMgp Mmebmuba/66v+nOnMLErEMX1tFNqlnCoF98ayUI1W6sjmKBWxGxloWJDBFlXnqcg9AypIy ep+9J1S/OQ6KjHI0m87lZIf594j0HhM= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="Kc//GQWt"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf03.hostedemail.com: domain of hughd@google.com designates 209.85.219.180 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684731790; a=rsa-sha256; cv=none; b=Yd8EoXHVgcDCEOW1WR42nppHP/Rlo6PNZKCStkE2KhcX0w50lKX66T00qWYEcQbCw1s/9+ em05HXnSL1mPGrU5j46lZZ5QWzAi/tuJws9Na5deM1i/8l436mo2ZUnEXzDnReGJhrKgYz B/MCJOWqztGW2G9rJNKH9w0ab2w0JBk= Received: by mail-yb1-f180.google.com with SMTP id 3f1490d57ef6-ba827a34ba8so8243189276.0 for ; Sun, 21 May 2023 22:03:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1684731790; x=1687323790; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=W3t6elq9R4Rh1GonIcnIp9HMVNr0t0fb96tM78h+6lY=; b=Kc//GQWtdcYt13ahGjffwsLxy+Q5bVB6WROYwiSc3JojtgWwDe3izjSyHuE6U0IG/O 5Fry5zLgCWeTO19pC/0eYwlnvGNsyrmto6ZIBbyVNO56nsRvrMOAOUvYspI147xhZOi+ xqdX2iixj9Eo1KLPuK7IDQx9rj9ZoPh6D5XKx6tya/9I/bhzim+Yk6xTIzKBifA2beI4 8qMCGwFCGRwoHjLeW/d0tZsZvr4CJThANuX9SXUFpqX1pEid4ilFDqnQwxhHJZTeMBQb PwATHquSI9R4s4VAZXrMlE6dKgd4X76gX9zjTUT4KksMHQf4ahMKcn/OqGVTbmF1mVyt 4G7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684731790; x=1687323790; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=W3t6elq9R4Rh1GonIcnIp9HMVNr0t0fb96tM78h+6lY=; b=AphhRIyq7Zt7/5CgYPsD/Cra24M23KbhDJqKbQFcMpdV0C/XvjYOlWGSlSL5mZs/MS oXxrN7hzHJDdjISxV8KKhNOrywPc+FQjlWARStgnfyuUAmlZN7QTGMxg/QUmRecCyw1S VN2oRUqQuve31PkSvPqvNE3hexEkfmVkonk4l/LoIGLvEqqa9ZIf88L0zzlMqbTCB+ow sRla+V1J+AQeydEPqnTh/KPeeS6DnzmuB99SplwkY6hJ+l/CrdgFo8oIb1NCKb9PPwkm Tk0CufKzaYX2wMD+fZLoCogAwahBkeL8JcPDTurix8AYnIi5PyyMc6ULboW23m6i1+5p vdqA== X-Gm-Message-State: AC+VfDxv3az/RJFPx4yb82dlwilzMI0gU483YJ+ObWNaT28OHhimt9fW x0ZCiT47rA7gG+4EY6I6BF4tsw== X-Google-Smtp-Source: ACHHUZ6OdnTYy1GhoQl2g/GfXGTjIkyQzrSYRsCypiToKGf2HILo2risNfGACl3kmPaMyL1d5bYIVQ== X-Received: by 2002:a25:aaac:0:b0:bab:eb8b:c484 with SMTP id t41-20020a25aaac000000b00babeb8bc484mr2926154ybi.14.1684731789855; Sun, 21 May 2023 22:03:09 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id e6-20020a25b046000000b00ba73c26f0d6sm1322602ybj.15.2023.05.21.22.03.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 21 May 2023 22:03:09 -0700 (PDT) Date: Sun, 21 May 2023 22:03:06 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 11/31] mm/vmwgfx: simplify pmd & pud mapping dirty helpers In-Reply-To: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> Message-ID: References: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: DE7932000A X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: 8s9w91h4oo7zxada1h7afteqjo61akt8 X-HE-Tag: 1684731790-404926 X-HE-Meta: U2FsdGVkX1+Zi30tWp/V6PkjUOluRUl180O2AKbmaLDlXwHMTfQHvd70e17Ap9yJ5MsMbzprIIrrZQn5YtolJcdMAheu/6bZPkWj4qKj1jPJDfc8bRCRE5Re+6gfo0Jcuwb22RMhSeB8+qx2Yz/EFClXUK0pHkB0cW59SBSy5p4NvgDcvHYCmwwNjJvdMVZ7JNJaR61Aj2kLtkKKkrVoV7jHZs7Do18sXZfaZnlG264d8U+cz5BYZ2fzeeKGj4iPh4KWeXFpZRbsbPrNld1dxh1B5Nnt9+fgjvogZtBXVXYtw1eibbBRDOBeCpwh7QG7Fr+y/hiOl6LoNaGwCOZ55onvdrb3ohcXDzzLLoFiVhntRYl1V7/+/1DEDn3WyxKBcZPI+Vw/WmzBI7N8og6D6PnFb/h6KC/bXFRKKW4zfMYO3+P99BFnkmWnooITih12GgWSaRI+/vhzhreQhSrhvTE83rpb6ficq7GfMq2oMbNgrpC9hwsn9ZRbPltZ9fhI403ndAg1YaaLyeiON9AmHpWRtfuI3OBR5Qg2o3Hq4VzN//UALy+fia2XuL0aD2eRK+/aDU2aluHlVF7FsaTM6NvBp/iLAmFSL2kqTiTHCRTOMQ0yOLK6Z64JhoxrJdTCUa01hBFJwJEdugsRf3mtndheoe2dSWkgZ94w3DOKtWPMlaq+EqHkO28evjfECicEjp9Sx91MgYRssu8HrhIRI/E2S+dH31ypFvviOgCVtEfgSgw6/vxAi10jCgtyUnFf3gjKvnW4eCIbGsSm9BAIq10KUVfNsphZyBThKahScJknNvWxNDH7HK6bFJWqBNGJNT0/SLUZMS5yOht58DE9cg/l2juBPFWO1pgzi2tyNxk3YgxXCxb6lJQt/UlZbhFBxTai3A+11xmL6ozkX+dskxMdGb1iv23+ugAfJoN/DRSofePqAFP1sxLNE+5vS1g5pVS904z8f3JH3bQy3iC HS/5Wy4K ZshQoZpS05b1lI/Tt/MNo8li8wEIEcKCteWMyajUkmAuk9LJwe5zdqH3Ee1m2PoP2B9kRUGwhCdQjCi2kJXzhbL/kQ4LE2u+5vEfYRiTb3qG+K5TOLgHFj4qpbU6lrF90Y2BWE7MPaQdVjaZvJ8uOaDrJPltbBSsqv3msAQMRVqMjGZHt16iIyvhhceV95To5bB6fYVSb0Jzdw5QBmIh/98iOv6B5PwBnUNcEbkhwjMEH/LNal3TYGPtp4bCBf+ji/F9542r5g9QB+8S3liccsEGJSPfQV53yqm06hVPjETRbeCdFFipsQWYT1lIAneRY4ZTHmDN6gh7P3cMx9T6mk/79UpGYaaXCo1hW6aSyzeQvwhRhKGuFLJT0Sb8DSjSfuIG2 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: wp_clean_pmd_entry() need not check pmd_trans_unstable() or pmd_none(), wp_clean_pud_entry() need not check pud_trans_unstable() or pud_none(): it's just the ACTION_CONTINUE when trans_huge or devmap that's needed to prevent splitting, and we're hoping to remove pmd_trans_unstable(). Is that PUD #ifdef necessary? Maybe some configs are missing a stub. Signed-off-by: Hugh Dickins --- mm/mapping_dirty_helpers.c | 34 +++++++++------------------------- 1 file changed, 9 insertions(+), 25 deletions(-) diff --git a/mm/mapping_dirty_helpers.c b/mm/mapping_dirty_helpers.c index e1eb33f49059..87b4beeda4fa 100644 --- a/mm/mapping_dirty_helpers.c +++ b/mm/mapping_dirty_helpers.c @@ -128,19 +128,11 @@ static int wp_clean_pmd_entry(pmd_t *pmd, unsigned long addr, unsigned long end, { pmd_t pmdval = pmdp_get_lockless(pmd); - if (!pmd_trans_unstable(&pmdval)) - return 0; - - if (pmd_none(pmdval)) { - walk->action = ACTION_AGAIN; - return 0; - } - - /* Huge pmd, present or migrated */ - walk->action = ACTION_CONTINUE; - if (pmd_trans_huge(pmdval) || pmd_devmap(pmdval)) + /* Do not split a huge pmd, present or migrated */ + if (pmd_trans_huge(pmdval) || pmd_devmap(pmdval)) { WARN_ON(pmd_write(pmdval) || pmd_dirty(pmdval)); - + walk->action = ACTION_CONTINUE; + } return 0; } @@ -156,23 +148,15 @@ static int wp_clean_pmd_entry(pmd_t *pmd, unsigned long addr, unsigned long end, static int wp_clean_pud_entry(pud_t *pud, unsigned long addr, unsigned long end, struct mm_walk *walk) { +#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD pud_t pudval = READ_ONCE(*pud); - if (!pud_trans_unstable(&pudval)) - return 0; - - if (pud_none(pudval)) { - walk->action = ACTION_AGAIN; - return 0; - } - -#ifdef CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD - /* Huge pud */ - walk->action = ACTION_CONTINUE; - if (pud_trans_huge(pudval) || pud_devmap(pudval)) + /* Do not split a huge pud */ + if (pud_trans_huge(pudval) || pud_devmap(pudval)) { WARN_ON(pud_write(pudval) || pud_dirty(pudval)); + walk->action = ACTION_CONTINUE; + } #endif - return 0; } From patchwork Mon May 22 05:04:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13249762 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4AC0CC77B73 for ; Mon, 22 May 2023 05:04:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C943690000C; Mon, 22 May 2023 01:04:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C1D78900002; Mon, 22 May 2023 01:04:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AE52390000C; Mon, 22 May 2023 01:04:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 9F780900002 for ; Mon, 22 May 2023 01:04:13 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 645FC12047E for ; Mon, 22 May 2023 05:04:13 +0000 (UTC) X-FDA: 80816699586.30.EE99227 Received: from mail-yw1-f175.google.com (mail-yw1-f175.google.com [209.85.128.175]) by imf03.hostedemail.com (Postfix) with ESMTP id 906BF20009 for ; Mon, 22 May 2023 05:04:11 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=yE9oBZ+c; spf=pass (imf03.hostedemail.com: domain of hughd@google.com designates 209.85.128.175 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684731851; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=gMNcXWmakk7P+zoeKodH9UUwH6aG3s17nHsuG14zY8Y=; b=A6pQKZLqmlS6fA2q9pSTFttLAf7XjpbQKZNXjmlKQ7sOutzl47kgvBzF9r5e0qnW1BS/k5 xq3QBzeAt/An74Go+p9FyagQzTDbx5MoU+Md6Jklu+7qCv+1Qqi7+um64zjsdjeOuxNDUB WOYfTIkeR0GfLH/zLkpP1mktzv9k98w= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684731851; a=rsa-sha256; cv=none; b=xkmOhIHf6Sg6BTiAR0I6M7Uo4qHJzNqFcVXBq/ur+DqZio6wUudWD8u4zHOY6ccrlZTlDR xC1q5Ce+Pk29T1qtNf80lB47e43pNR2V6ysuVpr70DnKYuz+rS0rPICqqWpE0yd02DXiRH iEIzeUvcmkImHDTLPtwPFkHoxptm4qY= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=yE9oBZ+c; spf=pass (imf03.hostedemail.com: domain of hughd@google.com designates 209.85.128.175 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f175.google.com with SMTP id 00721157ae682-561c1ae21e7so73851687b3.0 for ; Sun, 21 May 2023 22:04:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1684731850; x=1687323850; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=gMNcXWmakk7P+zoeKodH9UUwH6aG3s17nHsuG14zY8Y=; b=yE9oBZ+c+KHVypY5gza4GB1+80d52bTCDbLI/Y6WgFZjswF85OgTmRueNohIKa5HCn RIW2Wnvd56WBT/PaJWNYgidoUhB8V5KAp9zJm+23i0kvJLcqYbw0yNLwpWS8EoUIgMXO 0057efgR9JUAdgc/f2mkKLwhOjnHnkDDKtEeFpSuOpxEwvKhw8lEmNKs6VxGMTFXq9W5 sSyOJRzgEjdA5DausOy5f9zrEnebUpaQfXGLkpy0BqfSupfqQaLTpS0g/cbhzJZLF6eH CdCRTra6FNLJCs5X7UtLw6KjHZYdHYr0T5056rVVYaa+mJu1Od1zuvThKJk70HtbvQjx SxKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684731850; x=1687323850; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=gMNcXWmakk7P+zoeKodH9UUwH6aG3s17nHsuG14zY8Y=; b=GdMVR6P5EHfrwE2Ytn+Us/waDVPDTqBAyw2hlegv6DG9ggoL8vcDzd8Tbh0vW/BxPM s7BrU1urIx9AqnKstBBq+bQB1+GKY3ZhoQPh6JbnUedq+IXaAcgMbpc98xguXClVHYhf NyQR7PCTt6LMiWAMxDiV3Jd+JW6vLnqGAUodiOF9NZevmrlQHI86HN4U6u7/zdQ5WUNi cCvOE3y4/kIAkq6elyfTJLDyySIHeBjdmGmuic1y0inOEnAC38NqeR7ADHt9Ob2q3gx0 t8MNe9WuIZgS9GPcOKUnflre8nMs+x2sLoTvPqvqboX6uM/EVth/MVLbA63pBM8Bs0iT ON5Q== X-Gm-Message-State: AC+VfDzzsyZIQ+9CDJxiLh2ObJqABCMqfRIPpfM7fxszCSokuKeTTiri hZiHE/lFm0+oJd/4mGZGJ+mDHw== X-Google-Smtp-Source: ACHHUZ4s2TnDnXPeA62RlJKiZrtDTHYAyZbSNbhHfgawNyc1d9fDfp4flHO3RJFgNx0d20xXc/Hleg== X-Received: by 2002:a0d:d656:0:b0:55d:c8fb:8f61 with SMTP id y83-20020a0dd656000000b0055dc8fb8f61mr10119663ywd.7.1684731850557; Sun, 21 May 2023 22:04:10 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id t184-20020a0dd1c1000000b00555c30ec361sm1798344ywd.143.2023.05.21.22.04.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 21 May 2023 22:04:10 -0700 (PDT) Date: Sun, 21 May 2023 22:04:07 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 12/31] mm/vmalloc: vmalloc_to_page() use pte_offset_kernel() In-Reply-To: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> Message-ID: References: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 906BF20009 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: kg84jezq9emagy6yxrawx6p1knocq9sb X-HE-Tag: 1684731851-3645 X-HE-Meta: U2FsdGVkX18njK5MvqGCeQ85o8LFtgB8RhufKu3pEGoD4giKIw4/NBzXvK5b4LjP3Gd3AgXVVO2zI7zAQ8xGEG3qVXWq3q07nCWxiMsINJ3Z3RgnOut1B8KZID6bUX2Y1vY4qQQNKH+aSiuT4WTpKiaphe8TknC06DM9voogb5PKkK12+6W2DrHk0i3UfBk8jSxkB4e2YX9xBKp7m1Wdcn42e1KYurkx8bv83PlHxiFC7g5uhU6Ms+col5u7/VcrxLJk4zUO+77lmHJAMysbP1P+IVEXgMMpoyvD+dOJbZvhKffTNP3BCcsgXCrkZh3AhD1E/Sf54ajEB6xBH9hcb5wks+z7uXMf6MoG744fkRE9bAMDSjiRvM1QILRiEQjQah6O+Y/rYuUvzpno7Nnz38daDU7tvCTgRfT1m9ipalFJKN3hTcqW21effbv350fxXe5KymugO/JaMhEJkLC4ma+ighozsYXdObCH/HjboCq36BFcGZpHp/g5WsyOY3yerr5e5cEbmbPcZTeip8O0VPQIdr09lws9hXo2nFdWSUqOUdgz/LKv87dU2QM/oDg/VnbI2ChXt5pqnol5flMn/6hmZcoEqdpXcp/YnLe/Vbytjqw/fDIgVu6EGk/x50UiCa7VwDsesNYLPl1HNWXZ9ZkyO2OIMl7lVbPs4hScIISWHpIjQCwOYll+Y+5ny/SVnillluF3ZT54ouFuY7HEnCNFQ8q7SWJxmu5fVweCtKHo5dUJQT/ewCo1opxYAdMEZGZFR245Dk8KFF3895HoUHbSu2gtcNbCqIbC+yuW2G1ShWj1/2jwOzD8iy8JIOt3u+uNydFoaVazPn5RN0me6oDQzUiFEzu09mfYB/zoZjKnOAoUGPqsgKQAWpbAo/FUtDfZc0Oj/MzAxX+Ot+2D5XlYc3/X42FITea2uYZFNHRxsXQT4AoZ4IT8UZx1VX2cvTsgdNJwnO56+kzbNvN q2jiJY1/ x6REn9m0851aPodv+YISHo6xsPJQw4SGWEJhITSHJT6pLUWzBdsdhenje6HToNIJRNJwwUlEe5vBfF/UVHDayXoVrmqUy6TunMXHT0rk/cPzI3O0hlSG18vC7uBM1A2ysRJpJwrze7bMOSUwVtl6dIbGgS56ZUrKollCl5MPB2VURXLIDXj5XCoLgQCwxHGwj8X8B4Bbs3VJDiXI3QXl+oomZfUkAqpiEjvjZZp9nC1h6jXUvtfCvAL/JHm8rxK3FkhIKROlJDsfC+lBe5srYawALIiZ9wlI2ERWx6Wcn+wUNBBIurBGg/F6ISCVcskfuc8YUhX1RnnrQAUN5ZXlHRZ614F91nnnBUli8l2Pr4Sp0+qNQ7XCT2ZLrHTafnqjwRI2u16BfcvuXouwGAQBRcyxVXWMb+D0b0A7dIjQl6Ro6Ri7TrFQfJVMdAw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: vmalloc_to_page() was using pte_offset_map() (followed by pte_unmap()), but it's intended for userspace page tables: prefer pte_offset_kernel(). Signed-off-by: Hugh Dickins Reviewed-by: Lorenzo Stoakes --- mm/vmalloc.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 9683573f1225..741722d247d5 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -703,11 +703,10 @@ struct page *vmalloc_to_page(const void *vmalloc_addr) if (WARN_ON_ONCE(pmd_bad(*pmd))) return NULL; - ptep = pte_offset_map(pmd, addr); + ptep = pte_offset_kernel(pmd, addr); pte = *ptep; if (pte_present(pte)) page = pte_page(pte); - pte_unmap(ptep); return page; } From patchwork Mon May 22 05:05:15 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13249763 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9527EC7EE23 for ; Mon, 22 May 2023 05:05:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2FA9590000B; Mon, 22 May 2023 01:05:22 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2A7A6900002; Mon, 22 May 2023 01:05:22 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 16FDD90000B; Mon, 22 May 2023 01:05:22 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 075B4900002 for ; Mon, 22 May 2023 01:05:22 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id C1EEC140798 for ; Mon, 22 May 2023 05:05:21 +0000 (UTC) X-FDA: 80816702442.17.01F09F2 Received: from mail-yb1-f169.google.com (mail-yb1-f169.google.com [209.85.219.169]) by imf23.hostedemail.com (Postfix) with ESMTP id 00804140009 for ; Mon, 22 May 2023 05:05:19 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="OKj/gvd6"; spf=pass (imf23.hostedemail.com: domain of hughd@google.com designates 209.85.219.169 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684731920; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Q9fqf/aXpqllLgdqCnqVNZO09v0Ag5UU0wZCu2wmsUs=; b=RnlnsNXxJrV0ASknzJM7aHVFKAS7o9ADq7JNYFHSbMHMoTL3aGfk0zWG/c1EvtYgW4SjOa X6knkBpoan0CS6j7WnuP8MpNAAGKzR25iV7GYkOMdt6k9Fq5UWbuki4j41DhTopfMrgt/C ov8Dp+MycMDQmT80SSOmBhvD3q9rJOM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684731920; a=rsa-sha256; cv=none; b=ylQ70lqB7cQF6sgCRU08iHBN3bvK7nxX7agFzt78vdHRYSwD/d2mo3jeNA2IvxaHvlEjEZ cBHXNmIQVFOYgZ2SJBiByNJ7ShnxAgrTllWZ3yTjmcrTDNs9Ju5KAwNPfNE/pgCN+X3u9S T9XUwFLXrJ9vYoy7Ev72ED3T54+IU/g= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="OKj/gvd6"; spf=pass (imf23.hostedemail.com: domain of hughd@google.com designates 209.85.219.169 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f169.google.com with SMTP id 3f1490d57ef6-ba1815e12efso4775384276.3 for ; Sun, 21 May 2023 22:05:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1684731919; x=1687323919; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=Q9fqf/aXpqllLgdqCnqVNZO09v0Ag5UU0wZCu2wmsUs=; b=OKj/gvd6hObldwT6K4phcbvAmhbsjul6iPbLRzKjs71jzjUwd6r3e8xCWCNZVwPENy 0HaWB0qgO/fNnS/Ois67og8DBdTyq3v/IDNkLAR46vDKiCQUR3xPJoyU7eFLCu5OCUxu ene9fgjo5yHZ3AS4/uG54DbEfQUuxcpEJxYx8UgryAp+t99VA5jZFAwJV246EE9mtg1F OZdfnZLlti+W2fNxVTHpJKMKRYyeIyeNsz3sd1gR6zvPQCCVQ9jZz+INsTuvonpNhmYF Bm782lSdhp0xDJA2N1jyxGLFmtKZ57I/CBr/DUG8sWZnMiCyq1l9BTQfv5dwSU+wKfRJ U63g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684731919; x=1687323919; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Q9fqf/aXpqllLgdqCnqVNZO09v0Ag5UU0wZCu2wmsUs=; b=i64X65NeO7o9aKMJ1gs7zEeosUYDmTzVrxWHgsyGlDSCIJvjQ0+RVm/xzHziFfLDz8 mtQ94vxJ/fWV7tBkLh57ZaahYFlM6DUYfwheIS2GTf0TyNlcU+LWI+9u1evz2R3JOEK3 63CqOKd5Wlf7efiWnkhLcCGLN33tb/+WvqQEa0EArjNsJa6vuP2JunGlp+Xd+q4TMftw zi7BAtlfkXOEhDVu5FwZk5xsPUjvElPsNm9u4mjZkD5bOOVRoQwlmch6PKtDQ07XcqWy eTKRaUvwItUQdoQWqXxvYnKtro6boStfoKdYylQx+V0LZlitUFLY2vcgz+pqu/fI8Hu8 p/yQ== X-Gm-Message-State: AC+VfDzBGcw7jDNr3baBPo0g1jrUGur2LZg8Fo1rDCgagkWpZHRMal7Y WVsgjLXNh745qXi8xwPJUI0PSg== X-Google-Smtp-Source: ACHHUZ5Raj9TudeXZElbzzsJbxf03YA3/zAVWdObhAbA8Pr6yvbRJfnYmXwklG1UR7VtuXWIGmLBMQ== X-Received: by 2002:a25:fa12:0:b0:ba8:1c9e:c77f with SMTP id b18-20020a25fa12000000b00ba81c9ec77fmr9327198ybe.22.1684731919014; Sun, 21 May 2023 22:05:19 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id a12-20020a25938c000000b00ba87e9b5bf9sm1274482ybm.45.2023.05.21.22.05.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 21 May 2023 22:05:18 -0700 (PDT) Date: Sun, 21 May 2023 22:05:15 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 13/31] mm/hmm: retry if pte_offset_map() fails In-Reply-To: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> Message-ID: <2edc4657-b6ff-3d6e-2342-6b60bfccc5b@google.com> References: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 00804140009 X-Rspam-User: X-Stat-Signature: cm4wt57bkwxx4wjmnip8sz9fhnbokfum X-Rspamd-Server: rspam03 X-HE-Tag: 1684731919-931027 X-HE-Meta: U2FsdGVkX1+vv2H06uqtm2oIe+IhO6hfadC7DTuN4zvzPGi4kICbowlYYpbXsreumuz+jWCQ+LFsVJs1Wbvkbuzj4HrF8aJrROMFk1VpAM+x1of8SaplSthks/0wvB4cMu4amiaq658wN1RQtiaoQsIu0BAUHTJ9Dyh2nSH++ilsgeD6Ke6ZOQ5jWgLmvYXqlI95bsJE6on/i9JoflPAW8c4QB6wQdYibjoR3YjiPT4djgMkquzW4KqjRSSP8KHz0GsJqpOQwRTIT7rxuPFvJLbSDdvYMJwDLq4Hp/w9lHdwBUDkdS/69tu6eZ1OeCmId3hDpw8zHt0ZuBVLDFQCgUETqCQ2Opa/uhI6l8RsbUvZXVvWx55E5mNqJlChDfeoBggQ8XJ3TtdtPpSYL9Qg5OpdVEicj4k+w0p0SbtEdhXPi2XluM9QfPjNuGc9ZUfOnDF4w1IJtF0tRKhS5uLrSWyGiuFowwPHOXqx+UEyq+B2LYs+StvWAanTZdxddAhoTGBXVFFrsv5bOTs/5l1xFEhP14J8ZM2bXd3UXDKNsH3BpqufsvgFkA/8QkdO0FqmMGDrBkQqYYPuIZrTvPxKiGOpzEPri6/5U+fHXRMuhKogdrju4u8SNJjeDfR91wOA4J2S/HZ0OmlbqhrWYqKyYrwvX29DfATCqnIVwJhVD3jLESDZG3AU4vOd4Y+NNFCgtuFSTqsSN4iqizXMHnOQ73D8/FYaROPaLEN9gckeEZPIRdeEALGUR1BhLX+f3cx5MxCt4bmnEEWhh2bcVFMszqLLhK4lMOdD7aVYpVRrlEosBhlJKd20me03vPZIYyGxsUF4CtaOhQ7hz/ACTB1DGj+sTtQL7ScgMdj2Nyqe0nvYPtc81uRufs+kZ8Df7HdJ4Xf6jWGlINO0WQcGk3JBrvu+R8Ad3yLA52n0U+Op850jHr8ZlLepGbgl1vTw7rUV7yNEVZzY/1QNTQ/oea3 cc7GYvOX vLdWt/Dfw4tHLFPk62DmE2tczs8InIiVZ7cdAdtQ76rA/soRAH4ND8jOCXTu5BqofLoYMb+WKtyYyyj6jleZvQJDFPVekb78KgfT8NQ/aQbt6vGD/wvwtO0gJ87JNDQCWfyjEe/m+2M2az0jYWyzn6qs78Z14TK+BrST+PtqM0zCg0nGsM5YNNJ8Dfr2AQyTgkuD9tpEgIeBqRJrnJj0rHAmX2b9vbA8Ib6Y69WZ3CAYYBz+U/U2THc8XTLc8NbOsXIlGu0OPFNhP71yPY1Zhe831cBghjiRqdiDMs49K+f9SjNTptQg3d2se+/Xewrd2GZufpatDWK2GP/RR6iFEl56kispl4UxodCC9XyjdBYR6itK7AqZthBESAH8/E/HdC91OF0LbyKegMIKAb3Yby0w8JRUpmcdDbTllKP6OJ2fxR09u84e/iFFFwg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: hmm_vma_walk_pmd() is called through mm_walk, but already has a goto again loop of its own, so take part in that if pte_offset_map() fails. Signed-off-by: Hugh Dickins Reviewed-by: Alistair Popple --- mm/hmm.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/mm/hmm.c b/mm/hmm.c index e23043345615..b1a9159d7c92 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -381,6 +381,8 @@ static int hmm_vma_walk_pmd(pmd_t *pmdp, } ptep = pte_offset_map(pmdp, addr); + if (!ptep) + goto again; for (; addr < end; addr += PAGE_SIZE, ptep++, hmm_pfns++) { int r; From patchwork Mon May 22 05:06:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13249764 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3BEDC77B73 for ; Mon, 22 May 2023 05:06:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 51C3090000B; Mon, 22 May 2023 01:06:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4C963900002; Mon, 22 May 2023 01:06:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3B86490000B; Mon, 22 May 2023 01:06:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 2C2B7900002 for ; Mon, 22 May 2023 01:06:39 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id E59328044A for ; Mon, 22 May 2023 05:06:38 +0000 (UTC) X-FDA: 80816705676.12.61280AE Received: from mail-yw1-f175.google.com (mail-yw1-f175.google.com [209.85.128.175]) by imf05.hostedemail.com (Postfix) with ESMTP id 149F2100011 for ; Mon, 22 May 2023 05:06:36 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="xUO60/u0"; spf=pass (imf05.hostedemail.com: domain of hughd@google.com designates 209.85.128.175 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684731997; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=itm4lMYCv8hAr7oWbErSKKsiGUM8mghvSpbiblkFfbk=; b=W8qvnCrUXigBnKT3u7lRmtTQd/jtUU2DsHk4Ku7xDDwFlNOK40ykxE+WleNd3gw0yUDuc3 3Gz3gBDEyb5MgxkWth+p7gbG+sQm0W1jbCIxYFTEjJzgLpj8TRKLfLKYu+L4js6mbD5R+I QC8dqL5GLnHPY92vAMe1NvsKOVXbGaM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684731997; a=rsa-sha256; cv=none; b=fgOwQSFL//6WsjSih/cYJ6G6ccjux5T+j7KGCSA8K0JcGaHz8xt+RXdt/3LzfmodAGaynx 6eB44lQWcQgTUEQE9y45WmmZq9W0KxLYlMC/0dMS+IErCzRes3KSbI5AGV7Uc83mBNFAbd eEwhl+Oxt7seJaFyKy2sBYHyCqgXalA= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="xUO60/u0"; spf=pass (imf05.hostedemail.com: domain of hughd@google.com designates 209.85.128.175 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f175.google.com with SMTP id 00721157ae682-565014fc2faso7230947b3.1 for ; Sun, 21 May 2023 22:06:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1684731996; x=1687323996; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=itm4lMYCv8hAr7oWbErSKKsiGUM8mghvSpbiblkFfbk=; b=xUO60/u0hCL36TvKXHyURnnxqO5vn7ZybjY7ONPqMVFmXcRaGi5WJ1fjIK85dRtVJa +whV5tpXJtxlVEd0xUGkv7xuykXYLwzIAtD/A83Pd0ByM1utIbe7HqhKdkTngJ8uTZye nXdCtHMDNgLl+HSpG2o6gWxk89SbFvj51bDmDs5EUuxNtQO799f4KEwsfMDYpb1aA0JV G1qpsPEAFFQ60/jP3AFkOagxrcttoK3mZdLQdFsbhRCY1FyAsYOWEizdcI6h50IGdllG JcD8IVUHEVvPfu5cztMQwnxIE3n4xO4E81T6/znW0rbNwJzdKCpC1q5F2RM6c3EnqDt5 BYMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684731996; x=1687323996; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=itm4lMYCv8hAr7oWbErSKKsiGUM8mghvSpbiblkFfbk=; b=i2j3ciCFbcWXCB9g0ZlEredtfiva9qhzocP/Ri+jIeIK5C9/zuB4I1Y951oOc0mAhB 6ddkJrHOe3yYNmV1wmZXGcZjUYDVqaqPgFdjiABJvQNG3fiHDysjtElYYFxh1kvsX6op jEXh1QFewBZe2+ehK+n5SF6vJqupUlxv8VESWB4tQ1uFrFdQxfSez4woW9hksnQ1UBou LUzdMnlZYHoicqescB9akn1oqYeKvRsrJfRI6DN81Zb16tjPWAC1Bujwz6yAKVuXmMVY VEv11NNPf47krf57EhBUjGY7uGgEcyeTxoG9QjwkOGMUQQYmsY8zKiAzkJOTQDFa4ig8 LYGg== X-Gm-Message-State: AC+VfDyVvUJapA6jFcMLLOMOpTPgnuK5IZWJYCvQNbBFYOlfK+RbCUl0 BwsW7iVkw5ADThH1Nh6agRIL/A== X-Google-Smtp-Source: ACHHUZ78QIP2SjKpIXnCqaoasaQ8h+cR3FlbUtT2N1GVyk/3dI0nRSCd0E3wxVyacPDSOVPhhCpzYw== X-Received: by 2002:a0d:ed43:0:b0:561:9d6e:6f45 with SMTP id w64-20020a0ded43000000b005619d6e6f45mr10989561ywe.26.1684731996077; Sun, 21 May 2023 22:06:36 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id v184-20020a8148c1000000b0054f50f71834sm1805106ywa.124.2023.05.21.22.06.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 21 May 2023 22:06:35 -0700 (PDT) Date: Sun, 21 May 2023 22:06:32 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 14/31] fs/userfaultfd: retry if pte_offset_map() fails In-Reply-To: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> Message-ID: <424274a4-7c13-e14-b380-428fc69a45c5@google.com> References: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> MIME-Version: 1.0 X-Stat-Signature: heumjrqjgjwwf9h67zc6u18se5q1f8of X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 149F2100011 X-Rspam-User: X-HE-Tag: 1684731996-307835 X-HE-Meta: U2FsdGVkX1/GaVYkSiJi1gFyJAeAJnmx4HE66NOTF2Q9fw1gm0Ly04QSQ5rN2zJvb8D3VTl0IxS72vQVIYApYdy5zm8ujEuFEvYRnxlvA4q0E9vqmBNH/++yuHI5Z6CS1hwV5SNdRJZbFarmfF7QHgK4ybDSdi3M9l1AmHQtJ9rwx9/mCS93SENny/A8/u4vUnmBCOxn8/X9AQLi7yM/DAe6ZqNY7AVteIAwCJ7nBcW/ZjC+sJ63jDGkT1ChmmQKuNZB9BgGsSwoKopH03kmTbLoDFSRjansD+ymPk9Bx74DIuLNxrebvQUWZN3PY54SHG7FKnTGBz33FqRIC96ozPAxgGWjqkLhuO0zGns+4CtZ0VDXLObdiuwkvfNAODYQxeYkexBMaMXQ/ExtQgyWHM7+v2uGAYbisM17MjNmAYVi23RpAkMNVTRMzvgkVjonX6JNezF+e6rj7VPNptUgABEbuqCxpbm9bCZmuIISkXnX2ujFrtIpcq1QhAVwunNYkr/t1qHP2WEyCdQfighleLaCSxML+DcZvoDFKxU3Zq1EwwWxGNWjCMdT0aoelgPMxzVps+vDOTItuHL8C4z8WJ8kPpVBVtYshFkhfEEq1hu++L3XYijVw2GlS99i+GoTmb3HWw25CpVpGXmVR4/OYe4L0gdpU+ihl0sdyCnOlOAEiCVND2h6ICmcaz2r/lV+TtDOti/RIRBKphulBb3Lmhay2LsPFEbxcH+9uS2mSgCYMtSC1WGBokrKBIV8z1iN4fEcEMVDLs8kvH9N/0k5Y27aF8cYvY3OdCHKeb0abzk85R+syYk3Vjpo/DA5jXhCX/qmGm3B96EcRspzcJ1u/ddzGuvFjpvcI9IucSzraOp7HkESFxv7KZNQtL3JXNraALmbntnVOueenoQt9LboZjeZ1Clf+J+CIUWekgpzvkz+dn4GoMcKdFI771v443WqO1m5ku/ozGPrJ63BHsx goibLda6 QFzLmFkAD+nGxDkBcOYuEEiNEUGGSXi/SKxZGONdZ/fD92JE5TrhmXYS1Ufn6g0XVkEi7dQ0Eag/NIDs91IK+rzycEndI+FSFmr9/Qdqd6wJQ66tv1Ws23J5R1O/DDk9MOksVnKGduDkP/J5PrpcUxLOW9rpA6T5b51dtKhNykmHn7kwQE1Fwxp+cA3rRoYF3N0bzmm0OzCtFRIs3X0CcZLID8znH9MEJ5Rmpbr61p9jstg6DCiw7Qj7uKUDlL8HoIc13AEz98F7RUSNAnrk8fBSLC87mIGbF4YMcyMnKNi04nQkftT0IxjGs42FhlCgRDpoeZ36IyMWYKXZbmrTzzzqZ7GdzobdN/7KfYSEbNdFyCiwBXtQ+J8YtTmQqrv9baVI7bqpPCdVgmIyfBsYAn9/hLj8jst49gWYjnO2RGENc+ojMyFbAzXupqw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Instead of worrying whether the pmd is stable, userfaultfd_must_wait() call pte_offset_map() as before, but go back to try again if that fails. Risk of endless loop? It already broke out if pmd_none(), !pmd_present() or pmd_trans_huge(), and pte_offset_map() would have cleared pmd_bad(): which leaves pmd_devmap(). Presumably pmd_devmap() is inappropriate in a vma subject to userfaultfd (it would have been mistreated before), but add a check just to avoid all possibility of endless loop there. Signed-off-by: Hugh Dickins Acked-by: Peter Xu --- fs/userfaultfd.c | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c index f7a0817b1ec0..ca83423f8d54 100644 --- a/fs/userfaultfd.c +++ b/fs/userfaultfd.c @@ -349,12 +349,13 @@ static inline bool userfaultfd_must_wait(struct userfaultfd_ctx *ctx, if (!pud_present(*pud)) goto out; pmd = pmd_offset(pud, address); +again: _pmd = pmdp_get_lockless(pmd); if (pmd_none(_pmd)) goto out; ret = false; - if (!pmd_present(_pmd)) + if (!pmd_present(_pmd) || pmd_devmap(_pmd)) goto out; if (pmd_trans_huge(_pmd)) { @@ -363,11 +364,11 @@ static inline bool userfaultfd_must_wait(struct userfaultfd_ctx *ctx, goto out; } - /* - * the pmd is stable (as in !pmd_trans_unstable) so we can re-read it - * and use the standard pte_offset_map() instead of parsing _pmd. - */ pte = pte_offset_map(pmd, address); + if (!pte) { + ret = true; + goto again; + } /* * Lockless access: we're in a wait_event so it's ok if it * changes under us. PTE markers should be handled the same as none From patchwork Mon May 22 05:07:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13249773 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BB2B9C7EE23 for ; Mon, 22 May 2023 05:07:43 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B81D090000E; Mon, 22 May 2023 01:07:42 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B316D900002; Mon, 22 May 2023 01:07:42 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A52A090000E; Mon, 22 May 2023 01:07:42 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 9502E900002 for ; Mon, 22 May 2023 01:07:42 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 6EA7F1A0726 for ; Mon, 22 May 2023 05:07:42 +0000 (UTC) X-FDA: 80816708364.03.793A5BC Received: from mail-yb1-f182.google.com (mail-yb1-f182.google.com [209.85.219.182]) by imf06.hostedemail.com (Postfix) with ESMTP id AC43C180005 for ; Mon, 22 May 2023 05:07:40 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="X1s37wb/"; spf=pass (imf06.hostedemail.com: domain of hughd@google.com designates 209.85.219.182 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684732060; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Po1SIcw5OHs73xEeyHWdahQyrrpxMX/55AnymOheHdY=; b=YlVDMQolqrTUTUTWEkW+u3Ip4QAkbahbYGFVG34Z/vZLlayv+i7dq0qRzrf46xopv3TJ2E CZMfXaahLwIsp2K7OYSlQnEbWg5tIY+fE1EBhJU9Hszl3jQb6hK1qj25LUxEnI4K3gSvf8 c4ODYJ0B8DqbI/tZVVj2Nlqnyl9S524= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684732060; a=rsa-sha256; cv=none; b=H+g+gvnZlO9A7ZZXKAuvJ92Ii41sSwQTnALhxA5oFMSaNcZCFPlsnY5dxxSlv7v8/f78Tr BbBPNvbChUBuCpawpaGVyQpz0cBKTAonsQrCMG8Bzl9tE3ulYWm6OEVf+3/odFGmceIpO0 X/7KxhR5m/OaWU1B/CVB3bVqK1vddy4= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="X1s37wb/"; spf=pass (imf06.hostedemail.com: domain of hughd@google.com designates 209.85.219.182 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f182.google.com with SMTP id 3f1490d57ef6-ba81ded8d3eso8293228276.3 for ; Sun, 21 May 2023 22:07:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1684732060; x=1687324060; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=Po1SIcw5OHs73xEeyHWdahQyrrpxMX/55AnymOheHdY=; b=X1s37wb/ZRQK8wa5+7zJo8XqZvsoUBqIupbv2SaB+ZkVVeDTKpZ6QwcA2MkTOMLEB+ 04+Q8P7ixkmjr6uwabC/0/ae97Qd5WveKlzurD1MAgzXnZwGQUxZxpHSVC8+zTNRnrGz B25LFLPiXAeO2lmTa2nGOh3byiRj1pPd9axpciE/w6NeKi2mf190/nRmiWsGvrzrjmHJ BrIU26NxEn1SYQ6fIj/oe5nh1dTbf52uokYOvkdRrgGB7gx9TyZLzNcc0cxaPpUUftSN OOa+c2VRhh36Ov2ACw9ZnfHmqo02gnQWpWPgzuF/jjyarh/c5k63hfhpLNkRYyQl2O8o ZaQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684732060; x=1687324060; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Po1SIcw5OHs73xEeyHWdahQyrrpxMX/55AnymOheHdY=; b=R2n4SxLQXPtrvDBQXfq4wwrdhqyZaC7tDK8vs+vaWMj1GGnEMRO6PKAAITYOQSxzUQ K9qOWoDyTGvpqdzv29kJzzPyX1Pr0onZMIX1GxLvWdy9KfZbbfQE3cd7BJWC93scujgc TGpJoH6mHMW97yjW2uphgpog9GJ3gns1JzaMc6G/Ae4ug3EjuUm9+Hdbch//FnaY4s9p 8v1ic0N87n9g/wFLCnOhQefDeKuY0eaA+6Alk4cnaRyk1aNtYiLoGhzbKXcWH+7BAR2z qcTfKydgN4dD77RhZYgbu0QukaL0RTdZ/irePYQ7nRRn0HOunL0arLGChsTLVuJbKtlX W7Zg== X-Gm-Message-State: AC+VfDyTBuS7YypsHWYOQKSVo5Uh0RIAcg/6ygZp4p8giJ+Tk7j5uqOV +qJNZLwxNtgYeIQey6zV6p68/w== X-Google-Smtp-Source: ACHHUZ4m3WCKHCRUDIqxwtJbDgKim2PjP4Uj/Gl/wyaY7gpZzCtLpfaCmSlMhzQlWiwCXodOnzTDfw== X-Received: by 2002:a25:ada2:0:b0:ba8:fe6:8e3f with SMTP id z34-20020a25ada2000000b00ba80fe68e3fmr8924520ybi.5.1684732059390; Sun, 21 May 2023 22:07:39 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id r10-20020a0de80a000000b0054662f7b42dsm1801064ywe.63.2023.05.21.22.07.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 21 May 2023 22:07:39 -0700 (PDT) Date: Sun, 21 May 2023 22:07:35 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 15/31] mm/userfaultfd: allow pte_offset_map_lock() to fail In-Reply-To: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> Message-ID: <49d92b15-3442-4e84-39bd-c77c316bf844@google.com> References: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: AC43C180005 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: t6z7q6t3mcccy9aaxd8oixkjw69okgma X-HE-Tag: 1684732060-671223 X-HE-Meta: U2FsdGVkX1+f1fuxdIXjFzhkvRGlbU9n4EUbdH7301GqRmou/ph20RwHP96odIyw7BqwqLZFdNngPohrrreq4n/dfFxExDAtJfMpCdLVg/q6Rw8s8ZxPaN/9Pk2lkB3LeMvE9gc71t+xTadJ269gzae3Qhf0peMmGoxGP03gpYYf8HapNOgrKO4OqyWpPfL08pfiRP/sTE5ELcydW7B1di2948NR+57Qm9fnYSjNRDDJvzwgKxVA19UmF7He6kpJJyWPoENvZnrAxlUchGD3lxmlfXwTeUJNvIRS4npSahEKckvCDjBOiYzaZaH9gKiXRrEePDaerT0GZiwioFgjUsB9DcTY5ED/byKc63BBrheubsaiVfD0djdoZddyjUkpQXeLau8vOLPgt7lY914O8sMF3OmMSzTlUvJzRoGzvbgsgCcV031/fEY9m0uq2CYN6GjKTeHXXkztaHhbEox+ZrH+qKY+e+/q950qdoDaDqsYf85iSw6iXKYVp5b76N7saCPbCjzkP6ruxrW3WPL4kxZy7LAtfRCD45Zuyxn/cdwcKJAczuT5ZnvBIkNVKHae/NVcHglazTMowVh/k8yOzH3k4vKqQi74ro/GbGfJePXPhehnMOl/RW+hZzAScPDhPBVQd7SyYz0laFYn+BLwmVY05otGR0E9lwhXkrG053OWsvJxPrqk529vSLiGTGLor7tz+eKcmvSVs4HDwOz0+/oPfrF6OyIX6mKeTVYcSYzimBOHLIOTu6iuxmlr1fBLE1DDiN1iBLrzbVok95zlG7Cb9U/qfmx3NsfaF8tcIL/h4kekLqe9ybGbFQ0V4654jxGdX/Vyr7LFj1ub6j00/noSf7/tjz9Av26pQD9TuHG9L7AzhN3FhYRKRjjyeHQ5EsWGZfO6gBBS8QOYeYb1s9jqMtCU+Q1qH1Pu6jtYwVUvgbX10M/Y0PwkNTdPerVuSw2ATKiR5nf8NsS4rvH d8E1Mdmc wthtmiiF0eB2WZNQ+Tx5DQAm3p8UrqLlVGIB9DsJweDi/zxRJ7CEqnFGV/8uXcHg5T7GRUs9izrnZVxnnUq7lqMe3wAhuLCN6zOgpxlEd1XFG0Cd6Ttbb9F23RgUpHm+HJKv2AkTVz7NDEQIfJY1WBGcxZW9p8xk+uVxIR0OIZW3HJoqlf9HJPhpyH7pVdjuqHGbzb1iutEBDCT6vhY+PKfw6B4jUsG4iP0g/gwOreZAEJMd3twvDbxu8DMhDy/Sswvk/5S7WeZ7aWpL7TEA1Nnw8uhUlqBMZ16L/CkOTObG7+Qk6GR9mRyEXqI4tRmVtQgRRQ/QctfIT/rPaVPy5sWEwJS0d5jr/l7epMqQy3d5t3g5HOLrdjaIlinmqnlLofJ9TP6QWmm0EQpUOBX751OFLpDQXfsYLJ8DAnziFUbvLb11hTioNYMRqmQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: mfill_atomic_install_pte() and mfill_atomic_pte_zeropage() treat failed pte_offset_map_lock() as -EFAULT, with no attempt to retry. Signed-off-by: Hugh Dickins --- mm/userfaultfd.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c index e97a0b4889fc..b1554286a31c 100644 --- a/mm/userfaultfd.c +++ b/mm/userfaultfd.c @@ -76,14 +76,16 @@ int mfill_atomic_install_pte(pmd_t *dst_pmd, if (flags & MFILL_ATOMIC_WP) _dst_pte = pte_mkuffd_wp(_dst_pte); + ret = -EFAULT; dst_pte = pte_offset_map_lock(dst_mm, dst_pmd, dst_addr, &ptl); + if (!dst_pte) + goto out; if (vma_is_shmem(dst_vma)) { /* serialize against truncate with the page table lock */ inode = dst_vma->vm_file->f_inode; offset = linear_page_index(dst_vma, dst_addr); max_off = DIV_ROUND_UP(i_size_read(inode), PAGE_SIZE); - ret = -EFAULT; if (unlikely(offset >= max_off)) goto out_unlock; } @@ -121,6 +123,7 @@ int mfill_atomic_install_pte(pmd_t *dst_pmd, ret = 0; out_unlock: pte_unmap_unlock(dst_pte, ptl); +out: return ret; } @@ -212,13 +215,15 @@ static int mfill_atomic_pte_zeropage(pmd_t *dst_pmd, _dst_pte = pte_mkspecial(pfn_pte(my_zero_pfn(dst_addr), dst_vma->vm_page_prot)); + ret = -EFAULT; dst_pte = pte_offset_map_lock(dst_vma->vm_mm, dst_pmd, dst_addr, &ptl); + if (!dst_pte) + goto out; if (dst_vma->vm_file) { /* the shmem MAP_PRIVATE case requires checking the i_size */ inode = dst_vma->vm_file->f_inode; offset = linear_page_index(dst_vma, dst_addr); max_off = DIV_ROUND_UP(i_size_read(inode), PAGE_SIZE); - ret = -EFAULT; if (unlikely(offset >= max_off)) goto out_unlock; } @@ -231,6 +236,7 @@ static int mfill_atomic_pte_zeropage(pmd_t *dst_pmd, ret = 0; out_unlock: pte_unmap_unlock(dst_pte, ptl); +out: return ret; } From patchwork Mon May 22 05:08:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13249777 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6C396C77B75 for ; Mon, 22 May 2023 05:09:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0AC7B90000B; Mon, 22 May 2023 01:09:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 05BE2900002; Mon, 22 May 2023 01:09:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E3F4E90000B; Mon, 22 May 2023 01:08:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id D7498900002 for ; Mon, 22 May 2023 01:08:59 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id A071E4044B for ; Mon, 22 May 2023 05:08:59 +0000 (UTC) X-FDA: 80816711598.13.16F18EA Received: from mail-yw1-f179.google.com (mail-yw1-f179.google.com [209.85.128.179]) by imf06.hostedemail.com (Postfix) with ESMTP id DB2ED18000F for ; Mon, 22 May 2023 05:08:57 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=FWm+aAqC; spf=pass (imf06.hostedemail.com: domain of hughd@google.com designates 209.85.128.179 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684732137; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7rTGIbKCcYdvVJu9Ybd88rQGNMtFs0WCnevY0u3jIzc=; b=1zihbn1SA429hMCjykHb4lRv+Ld/7V8e/g92+sX6NLK5diqYfRUAGK8VgoPDwih47wUIcL L0JFhfCVULI/LFYYiIC/e78yC+z0opvhiPg2KmJ22lj3RMEuOhsnfuTMv+Xqfp9CTDVxi6 1/adIjr+p/mscRSkU1qsq+FxqqR6UNc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684732137; a=rsa-sha256; cv=none; b=kil4sXfU57BEtYEz1LZH+erpkPDPjd4swNyyFPOrl2kTL28hy7OtQa3472mGcZfjM2GLzm iQuRUaEi7N31YFJreoYuA5g/t0sv2soXy4mWkZrVLgexfeVnYDZ5ok8MT8oR3yNxPJFrY9 zYHYTIxPBBhGAeKHs94T9nacf7sdBmg= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=FWm+aAqC; spf=pass (imf06.hostedemail.com: domain of hughd@google.com designates 209.85.128.179 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f179.google.com with SMTP id 00721157ae682-565014fc2faso7246087b3.1 for ; Sun, 21 May 2023 22:08:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1684732137; x=1687324137; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=7rTGIbKCcYdvVJu9Ybd88rQGNMtFs0WCnevY0u3jIzc=; b=FWm+aAqCrTI8L5F1peTncvX2j+gJ1K5pdecyQ2SgwMPmXn8TR/t30T7yAueI1Z1Prx +ZNB4HFJeDGCEa3TY+DbZFPXAQMoW0lt0/cPYFkOtZn9pNtwNaAji3QwKEg49ie0wiso /aqMK319Yal/5gxVXR6hyaD4EBzJg8qItdk0tBwwmogEtt3sYbaEVCj54AVP/iyafLSN g3n7sbdycHd6xijbyr/rauTFVAs7KxoLkcQWJDCzbFut9bVjwaE8Xx0Mf3/CDuWrY3hk wIWhl9hSpGsCek8QZILjryCAxN2GrBpCITRg0NVy+fYzjrAIzOzQeSq16gHJmxidsl5V /Prg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684732137; x=1687324137; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=7rTGIbKCcYdvVJu9Ybd88rQGNMtFs0WCnevY0u3jIzc=; b=gEe5P6B6UrkcUzQue3reXSO31Gp/gozFM0otCW8wE3jYunExF7GjtRDwofVo1UjicU Lr1D9mUD8Etar01lO9+8yBHymYfIJiAHucK6liMH98tPjjO1mhxDotvnVldwSHdauD7F 2KEQ4qVkhoQCsCHttGvTZzwzkvRNsf09jya34TXGaYE3+NI/ExdR98TIxzkeYp79e9N4 9u1qaaDjljnMJbh0NPBmiU2EpTHAUci9wXlOJvDuY4GSITYMjlsz8P2KOkvkj3NZ+NcN 29fzSQlG+LzcwRY/XauyFOcm0VX3V4S/LJ2UTGIw1+8AOwH/5wSyNWVrP+jVTcbVBy// EChw== X-Gm-Message-State: AC+VfDwxNKsXRIszjJIhH+4VZAaAgbF9WyezTTrvf3DLjDCMWVQAm0gO mK7juQzFr+4Sg8DOdYzdylo9vA== X-Google-Smtp-Source: ACHHUZ7MEfF9deLoAIS2DPpctOmVhab2WCwoG7BFDeIpVNW1XvRRPWqoYkqwVwVotMomYmO1i86qzw== X-Received: by 2002:a0d:d74a:0:b0:55a:14df:5c10 with SMTP id z71-20020a0dd74a000000b0055a14df5c10mr12334350ywd.18.1684732136920; Sun, 21 May 2023 22:08:56 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id t67-20020a818346000000b0055a503ca1e8sm1804176ywf.109.2023.05.21.22.08.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 21 May 2023 22:08:56 -0700 (PDT) Date: Sun, 21 May 2023 22:08:53 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 16/31] mm/debug_vm_pgtable,page_table_check: warn pte map fails In-Reply-To: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> Message-ID: <87c0e8cc-85c0-806e-da9f-b7b3cacde7d@google.com> References: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> MIME-Version: 1.0 X-Stat-Signature: jbyzmaeuyr3p67dqjmi9myr93ubeqroh X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: DB2ED18000F X-Rspam-User: X-HE-Tag: 1684732137-33834 X-HE-Meta: U2FsdGVkX18+pV3NMQoRUo5jme5cu4qiWDUXFbSVra+RJxLvNCnw6m81c3bGnm4qgIm2OQnFzrds8TZoJGMoCPk62PrFcjiVJioBi/RFXcwH8EZUNayA+uTDy7zrQ/amJPHGQDEQmOA1Bz+TUzmylQucdxbw+saBQHCcF0Y2xmjjIuEhCSvzMFocLHySvtdUfqr/p7WKEEtcB3/7sJKsACzdPm6Vi34JEWiYf/q+/dv/aWPmFnf4Zvi9LwcHfzNNe7OhGha6FjEdWAHlph3XsHNn9mTc/l4dD4mggeB+Sd6DCs4aVQcWYdPQIyDrD9ZM4VIm8FvAr6U6Y19XfxU5ukhzLMeTzF91q6m9EdqMIKBSCytM0NBcWG89Ve7pF5RBDrU5z1pn2XtiUy+WjCp3fOD0baUy73ovy5j0mAks2KDgQjORSbDD6JRn7SGvsn+SPxlg60K35kPMTnqPR9zyB7iIq4dSM1SHaw6KWvAd6Ff3+lvJcV+4EVbHVD1erb9TZyr4kDzDcvWIkoVD2NZUUjp66351gPF8krCsvCzD4NJjF+hrncIz4XzW2VhUrTpySU4umSIY/CgyqVI2sdZi3d0UMWTFPUJCSUy+he426yh5YN5pifR3dFX5N5jwODwE7JyE4W9P7BnhofA3E/z+L3r5zdEXQYaXmI83yAurleIlESf609aKFbd2KN9VdimpD8kG5OTasdC/xqAtpQVQZklaXtrmSWqI42eL7VP7zuCSa+KVwFGQOnMc2+EDTAQQY+ALbSTfxcDmsZ6V5Attsf6/Eg77p7YeigwrVakTOYPzyncTLao8bdxSP4LZbKv/oqTtpYBrmqPy7JttFoZbycHDLzIe4DWPryw7Zmlr6fZBqYgkp7qPYBy88+mM0mN/yf+caiwUtV8XGgbSwyxYxmCm4TiiiclYMrryKZa00WtQaSSQkdga0NErtXVUBpu/0noXulHLMVVF7mCV4t0 guqLT0jD jZf1Rt8z+ih5kEibV3Ucev7p56s39tVH4UP5N3pd6mMNvu6pOE+OWcv/MAv8dw2UR4VhBYSoFutxSBKFZzmVYlg5EJC8hPSHgAX3bnDk/kpkyLKl+Jh3NwWcYfHd5AlUtgQiu9gLc09rRDLFYWBLMQqYk0E4Lafso56R/DpBEPhPJY4RYsyJqQ5cvs5+WolHZeD2mGQA+dd0zSomYrOnuPLBEUNRkUtezZ7RQf2XcaulFAFlFujhNzIWy8vlVac4yOsz4ldpdEFSBSbEyI5n5/uPmKHq3G8kxXDX4rclbm+PSyKbjqffaeC5h4lEtyw4BxwaSPiBBjXGSWfP12NsI+BWE6gBCX9qDFiEhGL2Djjz+PB1ZP1c53p67UByHL93o/7PxixUzZoi3K/k2PDbiYZFswEJUyKOGtZPBGLh4Cfnd1sSosgiIEaktng== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Failures here would be surprising: pte_advanced_tests() and pte_clear_tests() and __page_table_check_pte_clear_range() each issue a warning if pte_offset_map() or pte_offset_map_lock() fails. Signed-off-by: Hugh Dickins --- mm/debug_vm_pgtable.c | 9 ++++++++- mm/page_table_check.c | 2 ++ 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/mm/debug_vm_pgtable.c b/mm/debug_vm_pgtable.c index c54177aabebd..ee119e33fef1 100644 --- a/mm/debug_vm_pgtable.c +++ b/mm/debug_vm_pgtable.c @@ -138,6 +138,9 @@ static void __init pte_advanced_tests(struct pgtable_debug_args *args) return; pr_debug("Validating PTE advanced\n"); + if (WARN_ON(!args->ptep)) + return; + pte = pfn_pte(args->pte_pfn, args->page_prot); set_pte_at(args->mm, args->vaddr, args->ptep, pte); flush_dcache_page(page); @@ -619,6 +622,9 @@ static void __init pte_clear_tests(struct pgtable_debug_args *args) * the unexpected overhead of cache flushing is acceptable. */ pr_debug("Validating PTE clear\n"); + if (WARN_ON(!args->ptep)) + return; + #ifndef CONFIG_RISCV pte = __pte(pte_val(pte) | RANDOM_ORVALUE); #endif @@ -1377,7 +1383,8 @@ static int __init debug_vm_pgtable(void) args.ptep = pte_offset_map_lock(args.mm, args.pmdp, args.vaddr, &ptl); pte_clear_tests(&args); pte_advanced_tests(&args); - pte_unmap_unlock(args.ptep, ptl); + if (args.ptep) + pte_unmap_unlock(args.ptep, ptl); ptl = pmd_lock(args.mm, args.pmdp); pmd_clear_tests(&args); diff --git a/mm/page_table_check.c b/mm/page_table_check.c index 25d8610c0042..0c511330dbc9 100644 --- a/mm/page_table_check.c +++ b/mm/page_table_check.c @@ -240,6 +240,8 @@ void __page_table_check_pte_clear_range(struct mm_struct *mm, pte_t *ptep = pte_offset_map(&pmd, addr); unsigned long i; + if (WARN_ON(!ptep)) + return; for (i = 0; i < PTRS_PER_PTE; i++) { __page_table_check_pte_clear(mm, addr, *ptep); addr += PAGE_SIZE; From patchwork Mon May 22 05:10:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13249778 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66274C7EE23 for ; Mon, 22 May 2023 05:10:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 083BC6B0074; Mon, 22 May 2023 01:10:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 035066B0075; Mon, 22 May 2023 01:10:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E175F900002; Mon, 22 May 2023 01:10:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id D326C6B0074 for ; Mon, 22 May 2023 01:10:11 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id AB90C1C75DC for ; Mon, 22 May 2023 05:10:11 +0000 (UTC) X-FDA: 80816714622.16.FD981B2 Received: from mail-yw1-f181.google.com (mail-yw1-f181.google.com [209.85.128.181]) by imf29.hostedemail.com (Postfix) with ESMTP id E1CB2120013 for ; Mon, 22 May 2023 05:10:09 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=G8p2CzqF; spf=pass (imf29.hostedemail.com: domain of hughd@google.com designates 209.85.128.181 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684732209; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=qkatc+QCc6cccs9xXaqGfaqNL6OFGEuJiK0miGnv7G8=; b=PpkXVrvCP0HjXITwDPyUXtqYTywxpzZ5BkZNx9HjIG0T6kE7gppA+bC/VinMJUw8hDCVSt XkpGKWi4lY1DHFgilDeAqvuxk8a2iZHCTyyHmJYcqdKj9LkVc1f7gg1i6+Elx451TGlBxv ChubYUgSKWkavbzy9MEzJN9blR1yuMc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684732209; a=rsa-sha256; cv=none; b=WUX5GLq/74jXhHN9QUA9C/I4EzwTeXw0SKjKQ3+CYg39drtrWd03ZWlETiTzHvGj3cHdbw Rkj97Q1K5cvjINT9hzHEj8UWSAm89wxh7kcX2EfT6+4nSzZDqDGJ9vcJ+5cnFafEw1Iffw Jmcu9HDFkbvb1Cr6c+dr3GtmNBBqDWM= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=G8p2CzqF; spf=pass (imf29.hostedemail.com: domain of hughd@google.com designates 209.85.128.181 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f181.google.com with SMTP id 00721157ae682-561bb2be5f8so70193237b3.0 for ; Sun, 21 May 2023 22:10:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1684732209; x=1687324209; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=qkatc+QCc6cccs9xXaqGfaqNL6OFGEuJiK0miGnv7G8=; b=G8p2CzqFuweOTJakxAlGPjhKL7rDUMtnLNT7EUicI7w+jufYMVBoVuc81ZJQYoSTfb w6t5S+SR5Hp/NszCj/XtFj6CuqMt4ZlVp3PxVkJ3mlWRBt+Iifs7tpZrzkBlF0AC0HZQ qbtqENGFk4bM4644i8k7d4FbjPR3Um0C3R30rKDY1tPszjXHKCLCSRdB9GHFrQ6reUFc o/gs5XQG3ml90zH4fHmJbebDWsSKfY/AnaCH/uwICdx3ExQ+m1F+PTZVZDShUMb4Npmu GRvVy3RMuLlfKPCoJDA4zF1iEeyu+t7qtp3Nzn44IZZdc2OcH7AU7ec8Vtx87ucadbDG 6sAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684732209; x=1687324209; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=qkatc+QCc6cccs9xXaqGfaqNL6OFGEuJiK0miGnv7G8=; b=M7Sm8qPLFolyCeNsqRoph1zG2WUeG1FcNPslmuaQiRFcxqxwxUqDK0Q9uDXrP8fWEN A4n+PtGUNLldEIdYZ0KSS/NlZRxsxEOVC6ogAdEI68FY3Ge1AY6vGzDxYK8VmLE5SSyV 7whHigN/v8udFXul1BGV+SOy9B2qAdUs9BLa31QyQC1N2R6l3YU9Y0NLxSjI6XvQVMn0 F4U+fKB5hEoyJY6h76m5u4IG1F4ZJNTWBfL5OZDrFS0IXi3C6JsungnI4+NxPdJ0DKxO PemAIg7w9w4UXLDIVeppu6j9YBsUyq/GUdpWhv2qQnAqUGU2C31WXMrPATRarJp61Pw8 BkFw== X-Gm-Message-State: AC+VfDxmZqvxstl994GEm2tYNMe07P3RrmNGbxN7J5w23VRIKZUMf+Ur uDycA5rrPwpXs75KZndEqHlVJg== X-Google-Smtp-Source: ACHHUZ7YbSO8bhu9h9PxXguF69pafygjT3CZbvHofHpkGqbBtfy3DyuxisCr5F+9n6J8Jk2l8rmL9g== X-Received: by 2002:a0d:d595:0:b0:561:e910:52f5 with SMTP id x143-20020a0dd595000000b00561e91052f5mr12187363ywd.27.1684732208925; Sun, 21 May 2023 22:10:08 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id y185-20020a817dc2000000b00545a08184fdsm1790422ywc.141.2023.05.21.22.10.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 21 May 2023 22:10:08 -0700 (PDT) Date: Sun, 21 May 2023 22:10:05 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 17/31] mm/various: give up if pte_offset_map[_lock]() fails In-Reply-To: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> Message-ID: References: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: E1CB2120013 X-Rspam-User: X-Stat-Signature: bfm188bsxp3abzegt8tz8bswxxca5kj4 X-Rspamd-Server: rspam03 X-HE-Tag: 1684732209-872468 X-HE-Meta: U2FsdGVkX19RDbMxX8S24ynjHlMDJzsl0w4vTWJgHlYmBK9I3obIbUfcUi6z29s7x6cjPEC0id6lJzEPDK68Bg7YqeeyB/FCHlJlJHoSlwZzJepQyWjJASp+Z39w78Bd8paQWmXEv1O68V8l0b+VHqIpP0wOHIYNgyoZZ2q5kwex5ajXaYa13JRSak2xqE3RKLyizDetgPCc65MY+RgYlKtMAEbp0pw+fioc5QHzbJ/KBRrUmVMdSsAkWjzOoyOQAMbQaHmlpA7OH0ydUSar1dCNEPKAxJcYO01Bmj8yXbjTTqBzWOC0Qsz+LGXed4597+k/NDeZSGmtj+OsIszCI1AuyyO7X4cWHPgIYUb2csDUeAzbBCRHY/snKCG+pMxAe8rioS9ZZ/cL9805d/ml/q//z4+Bie8iS2hQFR7BxGpDG8s9wOEy34yySbjiv1+VjpgQFEHQiUfx2UPK42TaoYhtVlU9VMBNU1iYx3aBplCDpaY9ZBlfgakeTJ2yc3R9VQItvcz7LUy+L3MEbm/kwD5PIMin0CIPau46IfmZVgrpcOicbuyVgvcA75yjxA+1j3Os+azu+OnZ/o/U/WtoIn7kIuKRoPclWQieraS2Lt4B7eUpfP3GUAYAN/XJMINKeF7nlBrFVxk7SKs8A5X+AiVm/ub4T/IRTfTzMd7XD8iPN+xrXk/qvqvahdZOamseZVSRmL2kG594riujSTgrmimF3tRNLBpS8/xe0SbEGeYuHa1pRgmx+uxagVGrbJ80MoS87hrYxiEimTWhiWWEq4pxCGlAr2K5AhE1nMLXGyhX7T8pJU+UEZL5VwZN9C3hs2Br1NpJO7isnZpJ2dKmg5XqCVoMzjQzgaTOrda7+z7YSm6GakS1L1BnpIhLMDNz+OEi3re/JXh6tc9FdalO8uXMzEL9wqdiZwIU1TUrRizSDi6tnExRcQqyfN1mWkZpvoPX4hJMRQLvMisb+Ii ZGQ4LJx8 I+W/yO9NbPLAuWBjkHdvKhqvSXXEtErT0QtlQZ8jUPYF7ism0Dij/Y/bQJFxizgZVesgumUYgOmGp2MrRND46Io5HX+tCF1CJL0IAaFfq71paTruDCSMbFzMHqi4xw3VjIZRuM/x3l/X2ibVaHRL0FwqKBcJA8ahMcZFhpyjkHCEX0o0eDfjxKQYQRQKkWQtIIokmaexsqcOMsnjOaGrQFw7qT1WwMXAFd9qMiGfGgO6whSdWkK9PJkjBWrklEV3EUv7EF32vNxrrZhzELRCX4xcqmnqfvlAo8kXzZkwKTTToxrPWdwJtrovD9q7NPieuA7gWiE5LwvOX7INCrEB1OH5jjLf57Iyh+KtuYf/viT2QtshG/faMog9MZBGWWPfMSEINOQOMPJYKd5SRgXCrMlVKkihzWdCC7cTwY3Wo0EFNSvwIrgA6tZdEyg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Following the examples of nearby code, various functions can just give up if pte_offset_map() or pte_offset_map_lock() fails. And there's no need for a preliminary pmd_trans_unstable() or other such check, since such cases are now safely handled inside. Signed-off-by: Hugh Dickins --- mm/gup.c | 9 ++++++--- mm/ksm.c | 7 ++++--- mm/memcontrol.c | 8 ++++---- mm/memory-failure.c | 8 +++++--- mm/migrate.c | 3 +++ mm/swap_state.c | 3 +++ 6 files changed, 25 insertions(+), 13 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 3bd5d3854c51..bb67193c5460 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -544,10 +544,10 @@ static struct page *follow_page_pte(struct vm_area_struct *vma, if (WARN_ON_ONCE((flags & (FOLL_PIN | FOLL_GET)) == (FOLL_PIN | FOLL_GET))) return ERR_PTR(-EINVAL); - if (unlikely(pmd_bad(*pmd))) - return no_page_table(vma, flags); ptep = pte_offset_map_lock(mm, pmd, address, &ptl); + if (!ptep) + return no_page_table(vma, flags); pte = *ptep; if (!pte_present(pte)) goto no_page; @@ -851,8 +851,9 @@ static int get_gate_page(struct mm_struct *mm, unsigned long address, pmd = pmd_offset(pud, address); if (!pmd_present(*pmd)) return -EFAULT; - VM_BUG_ON(pmd_trans_huge(*pmd)); pte = pte_offset_map(pmd, address); + if (!pte) + return -EFAULT; if (pte_none(*pte)) goto unmap; *vma = get_gate_vma(mm); @@ -2377,6 +2378,8 @@ static int gup_pte_range(pmd_t pmd, pmd_t *pmdp, unsigned long addr, pte_t *ptep, *ptem; ptem = ptep = pte_offset_map(&pmd, addr); + if (!ptep) + return 0; do { pte_t pte = ptep_get_lockless(ptep); struct page *page; diff --git a/mm/ksm.c b/mm/ksm.c index df2aa281d49d..3dc15459dd20 100644 --- a/mm/ksm.c +++ b/mm/ksm.c @@ -431,10 +431,9 @@ static int break_ksm_pmd_entry(pmd_t *pmd, unsigned long addr, unsigned long nex pte_t *pte; int ret; - if (pmd_leaf(*pmd) || !pmd_present(*pmd)) - return 0; - pte = pte_offset_map_lock(walk->mm, pmd, addr, &ptl); + if (!pte) + return 0; if (pte_present(*pte)) { page = vm_normal_page(walk->vma, addr, *pte); } else if (!pte_none(*pte)) { @@ -1203,6 +1202,8 @@ static int replace_page(struct vm_area_struct *vma, struct page *page, mmu_notifier_invalidate_range_start(&range); ptep = pte_offset_map_lock(mm, pmd, addr, &ptl); + if (!ptep) + goto out_mn; if (!pte_same(*ptep, orig_pte)) { pte_unmap_unlock(ptep, ptl); goto out_mn; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 4b27e245a055..fdd953655fe1 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -6057,9 +6057,9 @@ static int mem_cgroup_count_precharge_pte_range(pmd_t *pmd, return 0; } - if (pmd_trans_unstable(pmd)) - return 0; pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); + if (!pte) + return 0; for (; addr != end; pte++, addr += PAGE_SIZE) if (get_mctgt_type(vma, addr, *pte, NULL)) mc.precharge++; /* increment precharge temporarily */ @@ -6277,10 +6277,10 @@ static int mem_cgroup_move_charge_pte_range(pmd_t *pmd, return 0; } - if (pmd_trans_unstable(pmd)) - return 0; retry: pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); + if (!pte) + return 0; for (; addr != end; addr += PAGE_SIZE) { pte_t ptent = *(pte++); bool device = false; diff --git a/mm/memory-failure.c b/mm/memory-failure.c index 5b663eca1f29..b3cc8f213fe3 100644 --- a/mm/memory-failure.c +++ b/mm/memory-failure.c @@ -414,6 +414,8 @@ static unsigned long dev_pagemap_mapping_shift(struct vm_area_struct *vma, if (pmd_devmap(*pmd)) return PMD_SHIFT; pte = pte_offset_map(pmd, address); + if (!pte) + return 0; if (pte_present(*pte) && pte_devmap(*pte)) ret = PAGE_SHIFT; pte_unmap(pte); @@ -800,11 +802,11 @@ static int hwpoison_pte_range(pmd_t *pmdp, unsigned long addr, goto out; } - if (pmd_trans_unstable(pmdp)) - goto out; - mapped_pte = ptep = pte_offset_map_lock(walk->vma->vm_mm, pmdp, addr, &ptl); + if (!ptep) + goto out; + for (; addr != end; ptep++, addr += PAGE_SIZE) { ret = check_hwpoisoned_entry(*ptep, addr, PAGE_SHIFT, hwp->pfn, &hwp->tk); diff --git a/mm/migrate.c b/mm/migrate.c index 3ecb7a40075f..308a56f0b156 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -305,6 +305,9 @@ void migration_entry_wait(struct mm_struct *mm, pmd_t *pmd, swp_entry_t entry; ptep = pte_offset_map_lock(mm, pmd, address, &ptl); + if (!ptep) + return; + pte = *ptep; pte_unmap(ptep); diff --git a/mm/swap_state.c b/mm/swap_state.c index b76a65ac28b3..db2ec85ef332 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -734,6 +734,9 @@ static void swap_ra_info(struct vm_fault *vmf, /* Copy the PTEs because the page table may be unmapped */ orig_pte = pte = pte_offset_map(vmf->pmd, faddr); + if (!pte) + return; + if (fpfn == pfn + 1) { lpfn = fpfn; rpfn = fpfn + win; From patchwork Mon May 22 05:12:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13249779 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6BE46C7EE23 for ; Mon, 22 May 2023 05:12:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F2B426B0074; Mon, 22 May 2023 01:12:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EB23E6B0075; Mon, 22 May 2023 01:12:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D53B7900002; Mon, 22 May 2023 01:12:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id BD5BC6B0074 for ; Mon, 22 May 2023 01:12:14 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 973C6ADAB8 for ; Mon, 22 May 2023 05:12:14 +0000 (UTC) X-FDA: 80816719788.06.702FA77 Received: from mail-yw1-f169.google.com (mail-yw1-f169.google.com [209.85.128.169]) by imf19.hostedemail.com (Postfix) with ESMTP id BD3621A0010 for ; Mon, 22 May 2023 05:12:12 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=Xury70gJ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf19.hostedemail.com: domain of hughd@google.com designates 209.85.128.169 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684732332; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7GAfzbPNj/2rvKRevzngE24N05kydgFSIoMtFyLCrv4=; b=DUChmWKKslp1V9N0xCTT0FkZkipI7Bb1yjTvJT4uIPKPUgUeptDdBaaST5ZDDj4yUyElFQ t5Y9vZYeO1c8ScoK1m1iR+mnW/oYVvvzlm/2ZmoTfhs7lquxLmcuA30xW/dVd3F0MkLvtJ JyXMIMt/yNiH8G5zMVR4BfX4umi6Ro4= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=Xury70gJ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf19.hostedemail.com: domain of hughd@google.com designates 209.85.128.169 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684732332; a=rsa-sha256; cv=none; b=YW2rTwC+GzwuNUnqVIXzk4Zto+l7Y01aqTXWYjpN8tGQhIUkulZvuWLKbweZ+7RNAmAx30 mfdpRlvKRpdzCwChlqmTeSSVY2gxugy2e15d5GIztUZf5PB0uT/dQDQYfhCcYjCb0SWpS4 Fija6Zvb9TvmHVVHdsuLhBKTMCqCaTA= Received: by mail-yw1-f169.google.com with SMTP id 00721157ae682-561d611668eso71645947b3.0 for ; Sun, 21 May 2023 22:12:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1684732332; x=1687324332; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=7GAfzbPNj/2rvKRevzngE24N05kydgFSIoMtFyLCrv4=; b=Xury70gJk/Ie7HEmpym3Ju1gfiqnvb3lYbEkOk+qdASbHtYcBtM6f7yvCnD5Qi5hF5 Xqo4lAQiP6+GN4sGJVvV6KOkXQkulHhobtYfAhUe9/l/2KsHbHtH2T86CZpNKlWRl63d IzMkj8FLns70Ingn1LKVytPrfZjfNvxgPvDTj2KhwHoFQ8Kx+CJ4VG6UzypdkQqqJSvO YWMudU4lJeyeTAt+pPO1YgidnSwVn5tO3duE8/EWyOVS7vVgZwJSIiPPVnzLsKahA0qM ZlEnIC4jzz74OlvLVDlS2OOEGCjrtDJzReiTZcHJL2Ft7dpNZsKSmX1OuM4OkIMXn0/q hnaQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684732332; x=1687324332; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=7GAfzbPNj/2rvKRevzngE24N05kydgFSIoMtFyLCrv4=; b=NjxRjC7l2FGfqWeAvysRVGodEdui+8vVIINo7SnKBS+s84NwQNG2FqD2gfxv2AnakI pI5LhRAgem3EviJFYRKvSndmtBaPd+xWQobXrjOjdqJZ3PkXRHoHGJzNRiOFMaARwNj/ AUtngGuyihCelyihgt399mryVZRUSAwlVlDF051C8SSqkK5iYJIxuj2v3DPkwCE5olYc 7OKp3iNAuFEirnv6pAAP2GjVo7xXEUK6ADSdAcTVzXEVX2kUMYx3wEuldIpMTiSRL2za zS0S1uy2wCakzg8B9bb6z/e1/YSF2db5//wBGi884qjtHmCpvVPPzf23ciz0nmD7ERJv 6DDA== X-Gm-Message-State: AC+VfDwx3vptmnuNDayNHXj6kEJhPW/hcGreBw58Z8OtpDMSYBe3e7cQ wBS6HepgM8wyIF807zsYlO5J/A== X-Google-Smtp-Source: ACHHUZ5YOFvYflvdD4FMdNwpV45kv2pl6g6+I95eOvlzunuPKVoSDfQex4to1CKCQVwtCrNkcGYjoQ== X-Received: by 2002:a0d:d107:0:b0:561:b4e3:5fc8 with SMTP id t7-20020a0dd107000000b00561b4e35fc8mr9743269ywd.37.1684732331752; Sun, 21 May 2023 22:12:11 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id o80-20020a0dcc53000000b00559f03541c6sm1814009ywd.132.2023.05.21.22.12.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 21 May 2023 22:12:11 -0700 (PDT) Date: Sun, 21 May 2023 22:12:08 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 18/31] mm/mprotect: delete pmd_none_or_clear_bad_unless_trans_huge() In-Reply-To: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> Message-ID: <4a834932-9064-9ed7-3cd1-99466f549486@google.com> References: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: BD3621A0010 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: qhzkj1abxce98ae6n837wjm4goyhxqna X-HE-Tag: 1684732332-143198 X-HE-Meta: U2FsdGVkX18u1LSdiZqb6nZ5hEIr4xADKULHAjGHyBlAHDkJE1fNKYcf3Cs4Bllq+zuCrwGFrW5+7TrTNuSHyzAxgWfHkm1rlV2f7Q9HW4ZmZtuuuH03y68a/vfWqHm1zTEo0nRh2xU4zRh+s+1JxohrM1lvNL18J6GWljUi2K2bgmRi0XSSTi6X18YDYawx0BQZ5ZVCZdvxePBMpu8/CHZSu+pCOYQrX9rKV4hH0y5K3LaXf//EkGUABeELYHyZQ0M7UZT0QRCqmPt6HIs0PJ4FGxY2kWiM08Xttc0GFKExAAlNhUpcZGmnBL05xRoAcffqJTb01m2Cab6shhXKFF2jE1W4lrWbDsYM4nQcNZh7RI8mXRMt5ahSh3cdM/6rDHTdXyxPdLDIDQHNzUbuZ2FxDkqIFbcfAWgTHP06irnr7usiKgK2NCj3xJbSetOA1K/ukiGa9aVKQfyb/Tl8ZVXffSwDw/IZWJW31atrBunU2Bm8a+oIkCwnU/bv9OinzCLGjC3b6vMuu1nH0CwI5e4/1QGvYr3wJ33csXkcZcNpwgkW2lnsCu+blDUlzDBqf2gulvHfXtrI7djErZPiHzRplsOeNpLOj3jEl087zwu4zS2ISbSzTg98YyYXZR4h5N+Oe1EeETHDpmFTOK1Mb5ovjB7hkmukijCDTts+Rxl2Wp6IKuRWoUNLvqPfSRP/o2PfKZp/H0/LG2rOQakPiSGCMJLZEkZ0Hx09fFDQZ4GYAJB3NUg9IRLROoD28w3FEhSNn4lPrY3Jm+L1pDF9TkOcdFz5rIuvTMu//dlBFJSdSqr4kAvG1FhrEhbI1hir+AEaagb6+t873QNvK9nB3Y5o6q6X9+7QcbUv+gbZN+ZAGDqpakDM5vJi9JRIUWmVG/xUnT5MilO36JX7s9952JsyofglpByoM2zNm4ve2z4FYeJgDMlcqijcABvFFO6DpOtIzOOzNTXn9W52Fx6 5g66K+cF vUpiWkWY4mAc3L0hcV/7LxUG+3cNh+3PjyDDDPsi2OVDU6mCtUwvt2X2m+Oz6blD1S0kE8mCjDg1nqfNXgA2+lQslhPIShdisBc3axfbzPmmcLCVUaoYDPYmGUPkUXlM7XYWytCSUX9u6aZMYL/KByqNS7ZMc6sJ0M/HzXrUeW95OkF2X2Yq2psKudVI+LqhWstA+jAozBK+KmiFtwF7uE8VSeE0B4ymp9C6QHBSiywLsD2pM7oCH3+hcKCaz71RiWzVn8acMt380NOEEm04BTlNOr35hHMA9wrHHqu3YZyKP3Js9srOb10kn1Y/ytu9Dnx2kYx68S8/YVFnlULG/t8yRRe1QZkpEMGNJ7ahcbWeiRP8hn4SqTtH8wj8rI6hKu5hHoimYVJu8uxtg2+dQmIGdDqeBAyyI0vmQoNBr0T50xdCjK4OiXONp3g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: change_pmd_range() had special pmd_none_or_clear_bad_unless_trans_huge(), required to avoid "bad" choices when setting automatic NUMA hinting under mmap_read_lock(); but most of that is already covered in pte_offset_map() now. change_pmd_range() just wants a pmd_none() check before wasting time on MMU notifiers, then checks on the read-once _pmd value to work out what's needed for huge cases. If change_pte_range() returns -EAGAIN to retry if pte_offset_map_lock() fails, nothing more special is needed. Signed-off-by: Hugh Dickins --- mm/mprotect.c | 74 ++++++++++++--------------------------------------- 1 file changed, 17 insertions(+), 57 deletions(-) diff --git a/mm/mprotect.c b/mm/mprotect.c index c5a13c0f1017..64e1df0af514 100644 --- a/mm/mprotect.c +++ b/mm/mprotect.c @@ -93,22 +93,9 @@ static long change_pte_range(struct mmu_gather *tlb, bool uffd_wp_resolve = cp_flags & MM_CP_UFFD_WP_RESOLVE; tlb_change_page_size(tlb, PAGE_SIZE); - - /* - * Can be called with only the mmap_lock for reading by - * prot_numa so we must check the pmd isn't constantly - * changing from under us from pmd_none to pmd_trans_huge - * and/or the other way around. - */ - if (pmd_trans_unstable(pmd)) - return 0; - - /* - * The pmd points to a regular pte so the pmd can't change - * from under us even if the mmap_lock is only hold for - * reading. - */ pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); + if (!pte) + return -EAGAIN; /* Get target node for single threaded private VMAs */ if (prot_numa && !(vma->vm_flags & VM_SHARED) && @@ -301,26 +288,6 @@ static long change_pte_range(struct mmu_gather *tlb, return pages; } -/* - * Used when setting automatic NUMA hinting protection where it is - * critical that a numa hinting PMD is not confused with a bad PMD. - */ -static inline int pmd_none_or_clear_bad_unless_trans_huge(pmd_t *pmd) -{ - pmd_t pmdval = pmdp_get_lockless(pmd); - - if (pmd_none(pmdval)) - return 1; - if (pmd_trans_huge(pmdval)) - return 0; - if (unlikely(pmd_bad(pmdval))) { - pmd_clear_bad(pmd); - return 1; - } - - return 0; -} - /* * Return true if we want to split THPs into PTE mappings in change * protection procedure, false otherwise. @@ -398,7 +365,8 @@ static inline long change_pmd_range(struct mmu_gather *tlb, pmd = pmd_offset(pud, addr); do { long ret; - + pmd_t _pmd; +again: next = pmd_addr_end(addr, end); ret = change_pmd_prepare(vma, pmd, cp_flags); @@ -406,16 +374,8 @@ static inline long change_pmd_range(struct mmu_gather *tlb, pages = ret; break; } - /* - * Automatic NUMA balancing walks the tables with mmap_lock - * held for read. It's possible a parallel update to occur - * between pmd_trans_huge() and a pmd_none_or_clear_bad() - * check leading to a false positive and clearing. - * Hence, it's necessary to atomically read the PMD value - * for all the checks. - */ - if (!is_swap_pmd(*pmd) && !pmd_devmap(*pmd) && - pmd_none_or_clear_bad_unless_trans_huge(pmd)) + + if (pmd_none(*pmd)) goto next; /* invoke the mmu notifier if the pmd is populated */ @@ -426,7 +386,8 @@ static inline long change_pmd_range(struct mmu_gather *tlb, mmu_notifier_invalidate_range_start(&range); } - if (is_swap_pmd(*pmd) || pmd_trans_huge(*pmd) || pmd_devmap(*pmd)) { + _pmd = pmdp_get_lockless(pmd); + if (is_swap_pmd(_pmd) || pmd_trans_huge(_pmd) || pmd_devmap(_pmd)) { if ((next - addr != HPAGE_PMD_SIZE) || pgtable_split_needed(vma, cp_flags)) { __split_huge_pmd(vma, pmd, addr, false, NULL); @@ -441,15 +402,10 @@ static inline long change_pmd_range(struct mmu_gather *tlb, break; } } else { - /* - * change_huge_pmd() does not defer TLB flushes, - * so no need to propagate the tlb argument. - */ - int nr_ptes = change_huge_pmd(tlb, vma, pmd, + ret = change_huge_pmd(tlb, vma, pmd, addr, newprot, cp_flags); - - if (nr_ptes) { - if (nr_ptes == HPAGE_PMD_NR) { + if (ret) { + if (ret == HPAGE_PMD_NR) { pages += HPAGE_PMD_NR; nr_huge_updates++; } @@ -460,8 +416,12 @@ static inline long change_pmd_range(struct mmu_gather *tlb, } /* fall through, the trans huge pmd just split */ } - pages += change_pte_range(tlb, vma, pmd, addr, next, - newprot, cp_flags); + + ret = change_pte_range(tlb, vma, pmd, addr, next, newprot, + cp_flags); + if (ret < 0) + goto again; + pages += ret; next: cond_resched(); } while (pmd++, addr = next, addr != end); From patchwork Mon May 22 05:13:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13249780 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C196EC7EE23 for ; Mon, 22 May 2023 05:13:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 69D986B0074; Mon, 22 May 2023 01:13:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 626506B0075; Mon, 22 May 2023 01:13:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4C8D7900002; Mon, 22 May 2023 01:13:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 3E94C6B0074 for ; Mon, 22 May 2023 01:13:40 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 08E60C0389 for ; Mon, 22 May 2023 05:13:40 +0000 (UTC) X-FDA: 80816723400.18.C0C87A2 Received: from mail-yw1-f170.google.com (mail-yw1-f170.google.com [209.85.128.170]) by imf19.hostedemail.com (Postfix) with ESMTP id 3B3961A0013 for ; Mon, 22 May 2023 05:13:38 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="u/pipKzV"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf19.hostedemail.com: domain of hughd@google.com designates 209.85.128.170 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684732418; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=dPoyGlkGpegFUTFofVDIHWRBddk7VxC7U3Cl7g+dttQ=; b=ulGvMNn20Y6PRHxG2295P1b6L9P4kTd3+P27X897fQTh+W5A/DxVjgtm0094+aCCKacJCZ aJxwuu0uKYwhQ36STpj5xTdcOqFO58io7+MlhAPbQyiMQpSpZ7ICXV5/17i8ymRGUgoeEF j239HBqe3MgpfQzru6ZngD+HpL4d3uQ= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b="u/pipKzV"; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf19.hostedemail.com: domain of hughd@google.com designates 209.85.128.170 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684732418; a=rsa-sha256; cv=none; b=Ye6AEd1auyLpHo3QsOaR0qWdUGOyUuBtbSrHhrBCDjF4JKiUoPLFuf6ka56sew+94HRKrS 9Ao5aY/8yEnm1UdyimeAlJW7pk/ZCo9FCTqwPAtRN/x7pAyaihL9FvJZoIuz5ahdGGEz4X Nre0WYYRiAHc5s6nFKeRrECLk7YQ4m4= Received: by mail-yw1-f170.google.com with SMTP id 00721157ae682-562108900acso48667207b3.2 for ; Sun, 21 May 2023 22:13:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1684732417; x=1687324417; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=dPoyGlkGpegFUTFofVDIHWRBddk7VxC7U3Cl7g+dttQ=; b=u/pipKzV3+7Fw72J6ZuzihOhd10w0gGlLGmoTYVNzOrFtguVpB16bQ3yHwI9CSCfkH 3Vl3zrzd94RD+ao7QuGXMU38bZgzZUL1/QZs1hHXZKn7GPINxEVThftcMMnY2o+Yudl0 U2nS3rhKugfJt5YxhFfn9PDdwtey86BwLjUPjmgewXJIUFBVGRQf2ahMAOU7Z/t1DHAm l66iDmXAruV3nmxv3A2DqZgGdZJpE3BSBsWI9c3hTvkUhwfiQW5QjWVbpPvVCeixeUrI GKrbNt5veHLdmjZBBDPCavEMUBfFme3RGuyXCTdVESH6VYwjwFZwaExB5zSeQBSZMndy lQew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684732417; x=1687324417; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=dPoyGlkGpegFUTFofVDIHWRBddk7VxC7U3Cl7g+dttQ=; b=A2CMOZAsoSknadow2Jlb8YiHb3E1YQFZcyWjy7XyueS1u/++cwUKBDIAs9iwFGEB/5 2iBS6ZB7ouznKiF5ItTlP20quhmGc3hk9TDuylePw0n+GyRGNOgT7KzG8Ep18p+Gezrb kdXwqefpcX9k2OtkArWh0n5lhQW1WHBYKR+rWT+pJm6nFLsXmE8L1YcE9Xu2Fs7q8K6A fREskdZD1rr2DLMmuK4uPyBIdPeJ3vaGLv/ynBo2RJWMu+V6wLvrMyQWFjgevZuDeg2u bL6ttug3iZcC9sWCLqIa06sKy0uBCffOgBE0/Yp4LHLAwwkg+Iqz99tecdWN+1hjnKdd mgww== X-Gm-Message-State: AC+VfDzAUAFPmBe8ay6HliG+/Ae6nZrefo4R3zJULBtzzVAjmEQiCWA8 jGyNrIzfsSIB1zeSITbTOgfZoQ== X-Google-Smtp-Source: ACHHUZ6QCzXgzIS4cFPDNtq3tlt+HJnjXO6ZXovLO9zv6wjiXs/oia4ZbqdffbL8veAaOUMs5BJwAQ== X-Received: by 2002:a0d:d595:0:b0:561:e910:52f5 with SMTP id x143-20020a0dd595000000b00561e91052f5mr12194393ywd.27.1684732417238; Sun, 21 May 2023 22:13:37 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id i67-20020a0df846000000b00559f1cb8444sm1824582ywf.70.2023.05.21.22.13.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 21 May 2023 22:13:36 -0700 (PDT) Date: Sun, 21 May 2023 22:13:33 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 19/31] mm/mremap: retry if either pte_offset_map_*lock() fails In-Reply-To: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> Message-ID: <2d3fbfea-5884-8211-0cc-954afe25ae9c@google.com> References: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 3B3961A0013 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: poekiaeerrwzh9fcic35yq15zph9qf41 X-HE-Tag: 1684732418-277827 X-HE-Meta: U2FsdGVkX18uUndIra/MLmNEob4aJE+DldIh9iztaPnLUSJUktwxdhuKJMMDRjvidDelFkCMFtRuoYfYWh+bDJmy/66ksS1WPA8FD2B6sUm2Czx0KkWyvnNZe5oqLuPLetj3Ud4QR5cBNRi9hkb1evPA/t9f9e3E94iwb5TRE3l2kSHegFz5Z8L+68+aR+gbIIv3T+mjt9LzfaD6k0TfTgzTefTGdKfPp+WxG08GeaMwdrYBKMSCqJMClsw+Fj1rEmDgnFb0qlydqGXSdnWdk8YQcRwPDY20aT668fcH1t2YDFWxwfTnDaw1ytjz85011s4UqkYhdAV0RYJ/j0nKecyiP2dgQ8crmn5Dqot2tUKBXasJYg4dNmAZWTduzGB3PVElw0PaOXeByJsxi0ieU3vRaCydD+kmdTwgu/9hSUnl88FjHSEYM+HVUJ4j9zG/Wycd5bEhtMjqaCd6tBBvkQWZXQ8ECr29YZjR+P7YtnofYh8z2A6flfXgYsbEfe4LZJi7x0giD7j9uwXHKOo8f83Yyxxg0vaus/eeWPssKRYJTWkgP2hdM00kJIlrc3PooeYnOaFkl4rp1W40fYxsD/jmeF3Oh0RyvZbJpo5CvL/JVveWGrFkDpRVwZ+VvnliQ2pHQG6QFYphNY+6RtvaYqIn3Ky3C+/l4HbcKd58T+iousYBPs+BN8EOJ0McuBWotu5/QEVBx2T6+VrobkM11gAgOs+216ViNPaRFWXYwgV6MIZofp98+TNZ9Ve2aJBlVWWJkMNtXvwlxV7lOjwkbhUavAPZLnE1ZWzI1QDJISstO/lyebv7lNiBG/UqfUQX15FwuFG4OnsUkLnq/ERN4O0iu9tAPK4+HnF89iGoDeSaG1cxGVqIzyY1i1seC2gx4ABhplt8+PnDdCG57Sx1I8tMgaj/Lcwr1oPp1ubIpUIgqo6ZJpqorr1wnattVS6IiwB6Tn+/5FSMvjcNG0v xbesMwSI p3C13q78EgUDPF95ORLkYtdR/IaMj7puLM5tf2peFfiANnEYNJHZYcIx+R6DL+Tz3WJCG6O6DJLAx+xnumomx4NP9ynFEeFhQnrSOA2C0IV/FckNu60CkQLUfnFFF41hswTN262xNIElACUW2YhpmCiSGylyLevcsF8RkY6eOIfSutuD+DwIt9N2W+LhBu5XR9M58PYP8CZ6byJXZvbyWzEqSt+ygPVVZiuYX6yD83ZT9+ftCnqdVBRzSh25dPT9BX70fLeHoYxNALH6IKGoJ3VC0QnBlxY1Iz4Gmp7WGZfIJqyDnyzlKulV6gjpaOm3kKsK2z9zjdITceu2FBSTK3IBz9kVkr9vpQJyTDex7vRzkNvFgoUiH37f9iHW5cFY4tAQiZK5LKADA+q7jZ3L01UQAixFoYNwHZWJ04dqqIOr+FX7ouwlRl1BY2g== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: move_ptes() return -EAGAIN if pte_offset_map_lock() of old fails, or if pte_offset_map_nolock() of new fails: move_page_tables() retry if so. But that does need a pmd_none() check inside, to stop endless loop when huge shmem is truncated (thank you to syzbot); and move_huge_pmd() must tolerate that a page table might have been allocated there just before (of course it would be more satisfying to remove the empty page table, but this is not a path worth optimizing). Signed-off-by: Hugh Dickins --- mm/huge_memory.c | 5 +++-- mm/mremap.c | 28 ++++++++++++++++++++-------- 2 files changed, 23 insertions(+), 10 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 624671aaa60d..d4bd5fa7c823 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -1760,9 +1760,10 @@ bool move_huge_pmd(struct vm_area_struct *vma, unsigned long old_addr, /* * The destination pmd shouldn't be established, free_pgtables() - * should have release it. + * should have released it; but move_page_tables() might have already + * inserted a page table, if racing against shmem/file collapse. */ - if (WARN_ON(!pmd_none(*new_pmd))) { + if (!pmd_none(*new_pmd)) { VM_BUG_ON(pmd_trans_huge(*new_pmd)); return false; } diff --git a/mm/mremap.c b/mm/mremap.c index b11ce6c92099..1fc47b4f38d7 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -133,7 +133,7 @@ static pte_t move_soft_dirty_pte(pte_t pte) return pte; } -static void move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd, +static int move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd, unsigned long old_addr, unsigned long old_end, struct vm_area_struct *new_vma, pmd_t *new_pmd, unsigned long new_addr, bool need_rmap_locks) @@ -143,6 +143,7 @@ static void move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd, spinlock_t *old_ptl, *new_ptl; bool force_flush = false; unsigned long len = old_end - old_addr; + int err = 0; /* * When need_rmap_locks is true, we take the i_mmap_rwsem and anon_vma @@ -170,8 +171,16 @@ static void move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd, * pte locks because exclusive mmap_lock prevents deadlock. */ old_pte = pte_offset_map_lock(mm, old_pmd, old_addr, &old_ptl); - new_pte = pte_offset_map(new_pmd, new_addr); - new_ptl = pte_lockptr(mm, new_pmd); + if (!old_pte) { + err = -EAGAIN; + goto out; + } + new_pte = pte_offset_map_nolock(mm, new_pmd, new_addr, &new_ptl); + if (!new_pte) { + pte_unmap_unlock(old_pte, old_ptl); + err = -EAGAIN; + goto out; + } if (new_ptl != old_ptl) spin_lock_nested(new_ptl, SINGLE_DEPTH_NESTING); flush_tlb_batched_pending(vma->vm_mm); @@ -208,8 +217,10 @@ static void move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd, spin_unlock(new_ptl); pte_unmap(new_pte - 1); pte_unmap_unlock(old_pte - 1, old_ptl); +out: if (need_rmap_locks) drop_rmap_locks(vma); + return err; } #ifndef arch_supports_page_table_move @@ -537,6 +548,7 @@ unsigned long move_page_tables(struct vm_area_struct *vma, new_pmd = alloc_new_pmd(vma->vm_mm, vma, new_addr); if (!new_pmd) break; +again: if (is_swap_pmd(*old_pmd) || pmd_trans_huge(*old_pmd) || pmd_devmap(*old_pmd)) { if (extent == HPAGE_PMD_SIZE && @@ -544,8 +556,6 @@ unsigned long move_page_tables(struct vm_area_struct *vma, old_pmd, new_pmd, need_rmap_locks)) continue; split_huge_pmd(vma, old_pmd, old_addr); - if (pmd_trans_unstable(old_pmd)) - continue; } else if (IS_ENABLED(CONFIG_HAVE_MOVE_PMD) && extent == PMD_SIZE) { /* @@ -556,11 +566,13 @@ unsigned long move_page_tables(struct vm_area_struct *vma, old_pmd, new_pmd, true)) continue; } - + if (pmd_none(*old_pmd)) + continue; if (pte_alloc(new_vma->vm_mm, new_pmd)) break; - move_ptes(vma, old_pmd, old_addr, old_addr + extent, new_vma, - new_pmd, new_addr, need_rmap_locks); + if (move_ptes(vma, old_pmd, old_addr, old_addr + extent, + new_vma, new_pmd, new_addr, need_rmap_locks) < 0) + goto again; } mmu_notifier_invalidate_range_end(&range); From patchwork Mon May 22 05:15:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13249781 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 505C9C7EE23 for ; Mon, 22 May 2023 05:15:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E8B636B0074; Mon, 22 May 2023 01:15:12 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E3BE96B0075; Mon, 22 May 2023 01:15:12 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CB674900002; Mon, 22 May 2023 01:15:12 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id BCA656B0074 for ; Mon, 22 May 2023 01:15:12 -0400 (EDT) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 8E3BD160427 for ; Mon, 22 May 2023 05:15:12 +0000 (UTC) X-FDA: 80816727264.25.699A9CD Received: from mail-yw1-f170.google.com (mail-yw1-f170.google.com [209.85.128.170]) by imf13.hostedemail.com (Postfix) with ESMTP id ACEAB20008 for ; Mon, 22 May 2023 05:15:10 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=cRcUhaoi; spf=pass (imf13.hostedemail.com: domain of hughd@google.com designates 209.85.128.170 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684732510; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wdvmzniCtM+nQioqQdrwu6IzqdoeozaKC+WtG0d7jzc=; b=4+goyj0JY9h0WVAff5qGXfM8IaeUZLZdF6GrR1hhlvyZRy7Zv0L1+NcE/r4whznZaihCwO 8AAwlm/YsN9x7exngM74k97Qv5vo7n/mppbjbUp8IC+fLYMMvWYku4uDnHawVWXmm0Sb9m ZcmGTjhHPkh68L3k6rJuSt1oTU97de8= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=cRcUhaoi; spf=pass (imf13.hostedemail.com: domain of hughd@google.com designates 209.85.128.170 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684732510; a=rsa-sha256; cv=none; b=Tyx8Z/6d7ZBPOnfonxL/bJeBtHMaf/VeIJi8ENXWcOFyJJ8j5eDOnYoq3HeCufSxqct4De WC0Di2nY+DD1yhdVp/eIbwNSnju5Yrcn6mbn/0fYnCnopXjN0dxVF9HF7lfJvYUdCNwrEA 8QvmR8JiJxOxtm4FVp9krTh7UX6xeLk= Received: by mail-yw1-f170.google.com with SMTP id 00721157ae682-561da492bcbso69668397b3.3 for ; Sun, 21 May 2023 22:15:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1684732509; x=1687324509; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=wdvmzniCtM+nQioqQdrwu6IzqdoeozaKC+WtG0d7jzc=; b=cRcUhaoiSgJ38o5WfT/WZeLX6uP++hPYcx9RZxFfGqfMkCU/m1hwojIEJua5ROp1t2 3gbVLqFCMhxh733PpSF/0qZ8YjtL+7/P2FqSYSUAY16Ac5W48ksKrjIJgFQ08/pbPPf3 q4tD4wBFecp/AobnrAUouJg1D3xu+hq/x/0yYcV3B7KprXoBoJD5VvXqNme3XhkF8MUs ptKdL72dy1i0NG2qsZVUCDca3Hwe/qFNH0+a+76pWfu+QubvXC6ErHyq6G5+PWt8MPaI 67e8Krgsis/Lu/L/JhEyvnDKGzJIK/fkyJS7kC+QKlqSYLfoaAZoCGiKnWGOJLttwSnL hGFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684732509; x=1687324509; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=wdvmzniCtM+nQioqQdrwu6IzqdoeozaKC+WtG0d7jzc=; b=j7zysr+oOxsaL1HB5cT8LWxGejyMwfvuSiNsjGW5K2rUNFuDFyhGZkBiwRsKukL7EI U1lTwb4sr8xzkI2zV4roqZghwYT531QrfhqvNy1A+BMKi7YHyMoy3+dZJuyYs+DUeuX0 RTLCiNbYrSA5SCkXJthNUPlSX1oZgethEdKKf+Me4W0sb5ljoWeUgBqQ76yTfU75CEXG eSpyt5nVsvX2NqCufB9c1z7yTDWvHJaTuh0wHoMUoRS1FNxP0AsVf6FjxZQA6yKjQYCV sS2zlCrHoFRAvFToCzMdD7mnnaubnIPFVpQ1b9WH1X6amAS4ehzcSC7AsRSx7UuMAUyj ea0w== X-Gm-Message-State: AC+VfDwdZBoDg57d4Ha/xKllQ6fuEWE5czmVLmiFkZJHejkJ7r8ZH8RI 7+kIEgyTAaZ2T2Q5KNpurPwMdw== X-Google-Smtp-Source: ACHHUZ4FV9kV2YREHnhwIbwygT+rPx4Et9mG+j5eesRA86qgE8Xs1bA3kk0cV5xsWSqbz756kifYvg== X-Received: by 2002:a0d:e807:0:b0:55a:4ff4:f97d with SMTP id r7-20020a0de807000000b0055a4ff4f97dmr10153523ywe.48.1684732509567; Sun, 21 May 2023 22:15:09 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id z73-20020a0dd74c000000b00559d9989490sm1828589ywd.41.2023.05.21.22.15.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 21 May 2023 22:15:09 -0700 (PDT) Date: Sun, 21 May 2023 22:15:06 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 20/31] mm/madvise: clean up pte_offset_map_lock() scans In-Reply-To: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> Message-ID: References: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: ACEAB20008 X-Rspam-User: X-Stat-Signature: s1q93nmrq7qt145e3qsi47abhhc9pr9n X-Rspamd-Server: rspam01 X-HE-Tag: 1684732510-148726 X-HE-Meta: U2FsdGVkX1+1qoytRbX0dNARK12VpEaeFxlm4qoXz7/OsbCxS0KuF0f64uBsnQS593FzLcPUUogVq/xDwMYYmkhI1Ep0kmcP+PvXmDw0MPoSK59vq+PQAIu8F1FUzJX6uLRq2rp1elV8LANLxUjhmHG7z2bO7STNVgHvgEcPdGLEnaKi7DhYNI5mQ///T/VA0xIdpc4L7cGeo5lL2xDpZaSuZMehjL5bPTP0/edXa8vG9l+5U3zWqRzgQTXkJiSPKDBdhN8DC5YkEbCAWFWIOdOdBGc6VrO/Z0f/iba+tuB1uzNbdlVld3DkJQAvqXf6xPuYYDUEObN6goCrbun0mwU1jK35jBXLriYKhzCs6rQpyZGzc8VDCNS2phJPA8SVmST3aXcTvqLnxm1vCkJfqCZSH3Y2g2ygmFcVo088tU08w2hz2oMpzbQvuyFrRfahI88s03YNZjdZ1TZ0gAP2pmVVY2y460tp7SXqV5V8f3WgxvwHpsrqmmN5vIypXK+dE3hfeFSoR7Me45bCbSOVyPIvXxnBPFMDR2devAJ6nHR44DR6Ri8+PkDL2xBAS2V/2O9lwzrz3WAiQcYZeIjwwZEXJNsg4K0F5W+TrtZV9kkvRgFtDhVnvlbNJDGi5Qn5s2Pmv7o7WjVc90laKVKxC9YSVJrecEvBwrisY1wx572Uk61SWJpLl+An4r/qX4W3XeANT4+rX6vSBDwyYZVWbWV5xmh1HzjYSkphIzkPXsY4laM9bPkDUhdP45eGERt+ww/yIo5ARa3fZ2H/c4Jtw4aMFdQ6ZWTfAgQwrhr6FJhxSacb6RozMrKbSYJ6P5wCE7lup8r4e3SnMhAZiEd//qHKx0BTLEydSsfZ2/FBjAHEmAzJ9LwQcSKO7Vj8HtH9RAbfaCvygPSv80fnSMdm+JCSmthTygw5lrObYHU2y6xh57WQO8L2pSknk4ERN77d2IZUkQrSKRkywWTtBQT be/CTf1v FnCFeel59Jjej/iZ46K08Ao3dUFNPkFArftCHHVjxqzyd49fxgiSshu5AKhYA+kf0SEF6v//+6HWgxlDuett60F0/cMIjdVmkgZoNaj2t1A23NApghshJvrB6VbWFahwoUQBGJ1YtanyC+RF21021cs1mjmJKhzT4+DPksQjSUcyX6GZ4V6i1BbmqUziPR05UG1t+PcuazXVyjprL0P5v7kPBGgeJOujuFaiBo+6zFznYqdIM17tIx34mnN/nlvwzSgDRJKY6Awv/3JIsDuUQlrjs4XSa6HWFMvawZi84A8Mq/pGv/ftZYRZJTdHF/i4+0dDEKMVRmYvxA7eS5NLQkhcOEA4hD2Imy+7a30KCdMam9ISBP6rcSFiUpZEi6X1V4lqAZcL1Urf4SCREMfaDY8tx2TQ8A1/uvH9S/y5Q0rwXEmnXkHmsh//L8A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Came here to make madvise's several pte_offset_map_lock() scans advance to next extent on failure, and remove superfluous pmd_trans_unstable() and pmd_none_or_trans_huge_or_clear_bad() calls. But also did some nearby cleanup. swapin_walk_pmd_entry(): don't name an address "index"; don't drop the lock after every pte, only when calling out to read_swap_cache_async(). madvise_cold_or_pageout_pte_range() and madvise_free_pte_range(): prefer "start_pte" for pointer, orig_pte usually denotes a saved pte value; leave lazy MMU mode before unlocking; merge the success and failure paths after split_folio(). Signed-off-by: Hugh Dickins --- mm/madvise.c | 122 ++++++++++++++++++++++++++++----------------------- 1 file changed, 68 insertions(+), 54 deletions(-) diff --git a/mm/madvise.c b/mm/madvise.c index b5ffbaf616f5..0af64c4a8f82 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -188,37 +188,43 @@ static int madvise_update_vma(struct vm_area_struct *vma, #ifdef CONFIG_SWAP static int swapin_walk_pmd_entry(pmd_t *pmd, unsigned long start, - unsigned long end, struct mm_walk *walk) + unsigned long end, struct mm_walk *walk) { struct vm_area_struct *vma = walk->private; - unsigned long index; struct swap_iocb *splug = NULL; + pte_t *ptep = NULL; + spinlock_t *ptl; + unsigned long addr; - if (pmd_none_or_trans_huge_or_clear_bad(pmd)) - return 0; - - for (index = start; index != end; index += PAGE_SIZE) { + for (addr = start; addr < end; addr += PAGE_SIZE) { pte_t pte; swp_entry_t entry; struct page *page; - spinlock_t *ptl; - pte_t *ptep; - ptep = pte_offset_map_lock(vma->vm_mm, pmd, index, &ptl); + if (!ptep++) { + ptep = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); + if (!ptep) + break; + } + pte = *ptep; - pte_unmap_unlock(ptep, ptl); - if (!is_swap_pte(pte)) continue; entry = pte_to_swp_entry(pte); if (unlikely(non_swap_entry(entry))) continue; + pte_unmap_unlock(ptep, ptl); + ptep = NULL; + page = read_swap_cache_async(entry, GFP_HIGHUSER_MOVABLE, - vma, index, false, &splug); + vma, addr, false, &splug); if (page) put_page(page); } + + if (ptep) + pte_unmap_unlock(ptep, ptl); swap_read_unplug(splug); cond_resched(); @@ -340,7 +346,7 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, bool pageout = private->pageout; struct mm_struct *mm = tlb->mm; struct vm_area_struct *vma = walk->vma; - pte_t *orig_pte, *pte, ptent; + pte_t *start_pte, *pte, ptent; spinlock_t *ptl; struct folio *folio = NULL; LIST_HEAD(folio_list); @@ -422,11 +428,11 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, } regular_folio: - if (pmd_trans_unstable(pmd)) - return 0; #endif tlb_change_page_size(tlb, PAGE_SIZE); - orig_pte = pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); + start_pte = pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); + if (!start_pte) + return 0; flush_tlb_batched_pending(mm); arch_enter_lazy_mmu_mode(); for (; addr < end; pte++, addr += PAGE_SIZE) { @@ -447,25 +453,28 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, * are sure it's worth. Split it if we are only owner. */ if (folio_test_large(folio)) { + int err; + if (folio_mapcount(folio) != 1) break; if (pageout_anon_only_filter && !folio_test_anon(folio)) break; + if (!folio_trylock(folio)) + break; folio_get(folio); - if (!folio_trylock(folio)) { - folio_put(folio); - break; - } - pte_unmap_unlock(orig_pte, ptl); - if (split_folio(folio)) { - folio_unlock(folio); - folio_put(folio); - orig_pte = pte_offset_map_lock(mm, pmd, addr, &ptl); - break; - } + arch_leave_lazy_mmu_mode(); + pte_unmap_unlock(start_pte, ptl); + start_pte = NULL; + err = split_folio(folio); folio_unlock(folio); folio_put(folio); - orig_pte = pte = pte_offset_map_lock(mm, pmd, addr, &ptl); + if (err) + break; + start_pte = pte = + pte_offset_map_lock(mm, pmd, addr, &ptl); + if (!start_pte) + break; + arch_enter_lazy_mmu_mode(); pte--; addr -= PAGE_SIZE; continue; @@ -510,8 +519,10 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd, folio_deactivate(folio); } - arch_leave_lazy_mmu_mode(); - pte_unmap_unlock(orig_pte, ptl); + if (start_pte) { + arch_leave_lazy_mmu_mode(); + pte_unmap_unlock(start_pte, ptl); + } if (pageout) reclaim_pages(&folio_list); cond_resched(); @@ -612,7 +623,7 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, struct mm_struct *mm = tlb->mm; struct vm_area_struct *vma = walk->vma; spinlock_t *ptl; - pte_t *orig_pte, *pte, ptent; + pte_t *start_pte, *pte, ptent; struct folio *folio; int nr_swap = 0; unsigned long next; @@ -620,13 +631,12 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, next = pmd_addr_end(addr, end); if (pmd_trans_huge(*pmd)) if (madvise_free_huge_pmd(tlb, vma, pmd, addr, next)) - goto next; - - if (pmd_trans_unstable(pmd)) - return 0; + return 0; tlb_change_page_size(tlb, PAGE_SIZE); - orig_pte = pte = pte_offset_map_lock(mm, pmd, addr, &ptl); + start_pte = pte = pte_offset_map_lock(mm, pmd, addr, &ptl); + if (!start_pte) + return 0; flush_tlb_batched_pending(mm); arch_enter_lazy_mmu_mode(); for (; addr != end; pte++, addr += PAGE_SIZE) { @@ -664,23 +674,26 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, * deactivate all pages. */ if (folio_test_large(folio)) { + int err; + if (folio_mapcount(folio) != 1) - goto out; + break; + if (!folio_trylock(folio)) + break; folio_get(folio); - if (!folio_trylock(folio)) { - folio_put(folio); - goto out; - } - pte_unmap_unlock(orig_pte, ptl); - if (split_folio(folio)) { - folio_unlock(folio); - folio_put(folio); - orig_pte = pte_offset_map_lock(mm, pmd, addr, &ptl); - goto out; - } + arch_leave_lazy_mmu_mode(); + pte_unmap_unlock(start_pte, ptl); + start_pte = NULL; + err = split_folio(folio); folio_unlock(folio); folio_put(folio); - orig_pte = pte = pte_offset_map_lock(mm, pmd, addr, &ptl); + if (err) + break; + start_pte = pte = + pte_offset_map_lock(mm, pmd, addr, &ptl); + if (!start_pte) + break; + arch_enter_lazy_mmu_mode(); pte--; addr -= PAGE_SIZE; continue; @@ -725,17 +738,18 @@ static int madvise_free_pte_range(pmd_t *pmd, unsigned long addr, } folio_mark_lazyfree(folio); } -out: + if (nr_swap) { if (current->mm == mm) sync_mm_rss(mm); - add_mm_counter(mm, MM_SWAPENTS, nr_swap); } - arch_leave_lazy_mmu_mode(); - pte_unmap_unlock(orig_pte, ptl); + if (start_pte) { + arch_leave_lazy_mmu_mode(); + pte_unmap_unlock(start_pte, ptl); + } cond_resched(); -next: + return 0; } From patchwork Mon May 22 05:17:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13249782 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AA5CCC7EE23 for ; Mon, 22 May 2023 05:17:36 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 255FA6B0074; Mon, 22 May 2023 01:17:36 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 206946B0075; Mon, 22 May 2023 01:17:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0F52F900002; Mon, 22 May 2023 01:17:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 00E556B0074 for ; Mon, 22 May 2023 01:17:35 -0400 (EDT) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id C2B041C6C84 for ; Mon, 22 May 2023 05:17:35 +0000 (UTC) X-FDA: 80816733270.30.F48F635 Received: from mail-yw1-f170.google.com (mail-yw1-f170.google.com [209.85.128.170]) by imf17.hostedemail.com (Postfix) with ESMTP id EB5D74000A for ; Mon, 22 May 2023 05:17:33 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=f6H5lzyE; spf=pass (imf17.hostedemail.com: domain of hughd@google.com designates 209.85.128.170 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684732654; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=hzsL+M6cdepZgQw4hIUfWIci6E11MaKNkaWhV1yArkY=; b=Z5tVv9SCeFMWQqxzcOmqyajzFINIUSkZs7h4iNDDH6wCzwJjqpis3QaXA2Q0fbTEFjYLVd 6mA37Tx1tpAOtIYP8ama33EQkpFmt6BAGIVPTMo47vRMksX21HNZR5yRbKpJ2TgAxED9Zc aZqkVHw7DKqdAe0ZYficamZvdGAQzZM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684732654; a=rsa-sha256; cv=none; b=jV+JmbS6ehBwHwfccjSO1asqMJ2ZYOAXt0HwnGS3lzw0QWwPWAxSE7As0lpSWDsTn9hIbu KNpNlnZ071CvLd6illMHLfbQR7cOhTbdwqOztm+joCK/duqHx9GHCTDoyoqQxEw6UtY28K 5v1t27ldSucLcGyVrir2Ol959Om8PQo= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=f6H5lzyE; spf=pass (imf17.hostedemail.com: domain of hughd@google.com designates 209.85.128.170 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f170.google.com with SMTP id 00721157ae682-561d611668eso71684067b3.0 for ; Sun, 21 May 2023 22:17:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1684732653; x=1687324653; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=hzsL+M6cdepZgQw4hIUfWIci6E11MaKNkaWhV1yArkY=; b=f6H5lzyEl/529vvBXo7cjHFatUfG3YtCbKZO/m7kkdLuuofbAW6JB7TNcMTLxhS2Fa oCscaMoZSrdZFIADmHM3vqeNvKHSfDv45YS3VG0xINc3KnCVeYBUvVx6KUlmN5IVqTQ3 eea7j8ExK19Opvsc5Spx705Pw81qEdSQc02WpxsCt2PFWo+K9ekVeiRw9nrsNx+U4KaQ G9GGOC4mudc6X5o4tr3EZNzo4YmhfN4UtuJyHpWp81+YRZq+yrHLruWf2ZHUJUdTErJk SGxCiEVvSkiDFU5p6ro7GOO7WaLMeSjUeJjhiSiayQPvsPwDptONLF8o1b8tv18mP3Yi FtUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684732653; x=1687324653; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=hzsL+M6cdepZgQw4hIUfWIci6E11MaKNkaWhV1yArkY=; b=IYH7krz4SNjqiTGHztgwh8EPyF0IQ9PmYfmKdDAP4JLEcsHMKnGFhAl17IKvLMbs5H zbuJ2mNOm9tp9DGYxsQ3ozZWT5XGcH4zgWrBxAzTuDumlcmPhQg62y8Kt86F9fkwrPDF lQHn2w5WmeaNEbSkfFpITnsO5IxkPe2cGdTnSca4Eg6J8DFL2IxpMWvHKntgJL0NcHph Io2UEUYFrtGBCo2hoFqJHFFebnhrmjupsb9XDbEabYYh2d2uN/qQ+aopW4oa123pwiBR ShBj0vS1gseeAmzc+TQBHeOBQ139Yo5BMEjUkwZZwpuwQ/6At8FlBm7NoH9kEXkiSRpo IPWw== X-Gm-Message-State: AC+VfDwgFJ8SiSxuvCCT24Drbpw0YQDKTerdRA1dFyFXezbjKygPKwjS KwFwkfC8yHcAFwxEpI9d6Z8K+A== X-Google-Smtp-Source: ACHHUZ7HPZHU/c0SUjH6d0BWcOB8eLdBcSfh1g/VsYEO0pbi+eAMTALQvB8Hsf5hIiR7CjUOpnbz8A== X-Received: by 2002:a81:6ec5:0:b0:561:e944:a559 with SMTP id j188-20020a816ec5000000b00561e944a559mr9577004ywc.31.1684732652960; Sun, 21 May 2023 22:17:32 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id e129-20020a0dc287000000b00545a08184f8sm1818483ywd.136.2023.05.21.22.17.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 21 May 2023 22:17:32 -0700 (PDT) Date: Sun, 21 May 2023 22:17:29 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 21/31] mm/madvise: clean up force_shm_swapin_readahead() In-Reply-To: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> Message-ID: References: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: EB5D74000A X-Rspam-User: X-Stat-Signature: d5cmpa1wofpbpuu14m8c51r54oxs7154 X-Rspamd-Server: rspam03 X-HE-Tag: 1684732653-219914 X-HE-Meta: U2FsdGVkX1+asbonAByMK0FABCsBQ34oKCqmHH43YKy0qR04w/KIPZKnzhvJdk4LSR9BCrReaQijZYHLDhV7ll0jcMhqjyRIHfScMeXTkb/93PdXKi7lxO1CNtI9yAT1VGA/eAaUI5wckp308/9BsyymEn/DjzJP1BsKfK6nVPfDMJSW2WEEvxcutsjUUVGyl+W6+NzAcW8M8OmyUFS6b9Jmx/LbDOSE6M6un0HKuaEh79uNABAdAg4RyNugajfnCez0c2h3QSlHyETmt7mp0gJHZcHxpg+OBWthJpWLJRxRSWuP68aHkkDwhi3on3UaQaMdbXMMxRbYuaNkLo6cTW3Mmxxw6leBEdkd+2ICwEalNVJwgxwFKRVPs+1/L7ZL/n1ckXwzA6IPiQDUVZFIcQXx6gKciofG8DBQRPTN1vu+lNywQne/yauH4aNogOsNPDEjW6djiaqweV2vCNjkYk+gRFQLwhF1xEo1m/TOWlLzZbUJc4wWngBxOyOmQxL6e9owPHhqblwhpplEQhEjJ2UYsTE7xABBe4hcmWhBMqEbk35FXTfmCCpuYIXWkvX9LcSrDKnaa8J0cHY1YjEt8eOo7hONM+Fbu/caUnpD6uUvht7s/CIquiIgj+N5CqQt+/GfnJ5rBShjIwPDtdVJ7w4W7Srs0qX9i7e10v8rURvkVOOj53yZioW98wbuM9t/+zLZqDmfXsejSNiBbM+kP7q351C5+38OzD3B+khcxIpudBusMPE2ZO0tQQIIGpvNqNePnr8D0AgX6cKTBywz5nZpVW/vow83te3t5cZFk2ij+HU+A/GQAyfe7LAIxNlBA8EcwP1Xv9Tns72k0qE9TtEl/FwO1e0qtDWx2sivenbylsUckpHY//R4N3g5t6x3gnI1KJPnX7cKeLNGWZ018R/OJowNLC+OgzKSGEt6BKup68etwRp5zjLu/I/HXSurbp3sc2Xuee1a90QXyK+ 8gqp0JjV hm1Wn2LRMoZtGichM35dMQcFJcrYlpLGPznXevicUwZ0+p+f6KN3a2mjIH19Orar56HRC4ECehRxfook83VTrAUskTiO42pDzIQXHX7HGzlz7p6oJHb9CHYznYVQT4hHx6MMowKtQr/QbocWXc/yOXFDjy6c2JOfi9k/i1H2m/oCkEdR1xupSZzu+o97tNha6+MYL9uEcRUFKP2KaAVdHpYkN6YvH4zJTQ3oOZCBDUdkbpU6hKpfKrle2H8GsmTNxXM+8X2y7B/gNAALE7ZHBqfYWaJlNc6x4U3a/X/syhX8p6GXJT+Tid0X1ZOBOJGuRmJAfGsdvw8rOynPHtedrT8yCCOuExUgJbq9X78IMnupWWiUseoQ+jjUOKfj66ys9EGAXT/WnMFyVbM+vm20My7fAFVY7RrZIZerlspIeCKcUSxzp4RTVhlJxrA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Some nearby MADV_WILLNEED cleanup unrelated to pte_offset_map_lock(). shmem_swapin_range() is a better name than force_shm_swapin_readahead(). Fix unimportant off-by-one on end_index. Call the swp_entry_t "entry" rather than "swap": either is okay, but entry is the name used elsewhere in mm/madvise.c. Do not assume GFP_HIGHUSER_MOVABLE: that's right for anon swap, but shmem should take gfp from mapping. Pass the actual vma and address to read_swap_cache_async(), in case a NUMA mempolicy applies. lru_add_drain() at outer level, like madvise_willneed()'s other branch. Signed-off-by: Hugh Dickins --- mm/madvise.c | 24 +++++++++++++----------- 1 file changed, 13 insertions(+), 11 deletions(-) diff --git a/mm/madvise.c b/mm/madvise.c index 0af64c4a8f82..9b3c9610052f 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -235,30 +235,34 @@ static const struct mm_walk_ops swapin_walk_ops = { .pmd_entry = swapin_walk_pmd_entry, }; -static void force_shm_swapin_readahead(struct vm_area_struct *vma, +static void shmem_swapin_range(struct vm_area_struct *vma, unsigned long start, unsigned long end, struct address_space *mapping) { XA_STATE(xas, &mapping->i_pages, linear_page_index(vma, start)); - pgoff_t end_index = linear_page_index(vma, end + PAGE_SIZE - 1); + pgoff_t end_index = linear_page_index(vma, end) - 1; struct page *page; struct swap_iocb *splug = NULL; rcu_read_lock(); xas_for_each(&xas, page, end_index) { - swp_entry_t swap; + unsigned long addr; + swp_entry_t entry; if (!xa_is_value(page)) continue; - swap = radix_to_swp_entry(page); + entry = radix_to_swp_entry(page); /* There might be swapin error entries in shmem mapping. */ - if (non_swap_entry(swap)) + if (non_swap_entry(entry)) continue; + + addr = vma->vm_start + + ((xas.xa_index - vma->vm_pgoff) << PAGE_SHIFT); xas_pause(&xas); rcu_read_unlock(); - page = read_swap_cache_async(swap, GFP_HIGHUSER_MOVABLE, - NULL, 0, false, &splug); + page = read_swap_cache_async(entry, mapping_gfp_mask(mapping), + vma, addr, false, &splug); if (page) put_page(page); @@ -266,8 +270,6 @@ static void force_shm_swapin_readahead(struct vm_area_struct *vma, } rcu_read_unlock(); swap_read_unplug(splug); - - lru_add_drain(); /* Push any new pages onto the LRU now */ } #endif /* CONFIG_SWAP */ @@ -291,8 +293,8 @@ static long madvise_willneed(struct vm_area_struct *vma, } if (shmem_mapping(file->f_mapping)) { - force_shm_swapin_readahead(vma, start, end, - file->f_mapping); + shmem_swapin_range(vma, start, end, file->f_mapping); + lru_add_drain(); /* Push any new pages onto the LRU now */ return 0; } #else From patchwork Mon May 22 05:18:24 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13249783 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 256ECC7EE23 for ; Mon, 22 May 2023 05:18:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B57C86B0074; Mon, 22 May 2023 01:18:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B2E636B0075; Mon, 22 May 2023 01:18:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A1CBB900002; Mon, 22 May 2023 01:18:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 92E1C6B0074 for ; Mon, 22 May 2023 01:18:30 -0400 (EDT) Received: from smtpin29.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 6F7658044A for ; Mon, 22 May 2023 05:18:30 +0000 (UTC) X-FDA: 80816735580.29.6F3B66D Received: from mail-yb1-f178.google.com (mail-yb1-f178.google.com [209.85.219.178]) by imf29.hostedemail.com (Postfix) with ESMTP id A91AC12000D for ; Mon, 22 May 2023 05:18:28 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=Rn20sER4; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf29.hostedemail.com: domain of hughd@google.com designates 209.85.219.178 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684732708; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7TjV4RyPitx7LIAI3Pj7MSABK+uluBQcp84mqUzVC6k=; b=S9VDjfcFzFCJx8ZAsA5raCRcS86VU1REqvvRvb0DmIwPaljk8kAe3m+j7R8a53FywLNvRQ bPRhw46+TQzfRKxmbr6qRr1eUphIpm4Ag92qYwYv+/B9+hX+Gp5rBU/BivAjQX85ItTNCu yNgkTY9zc4Gq0HqlcKNlssqA45c7tp0= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=Rn20sER4; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf29.hostedemail.com: domain of hughd@google.com designates 209.85.219.178 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684732708; a=rsa-sha256; cv=none; b=hd/J9+Wv3QS1TZqiKk62NJ7NUkQpSI+AQxyov0901slMhybyvRKQPdoO5pc/wjFnGG6gIU i8I6nexF7wKux5EbOZWyq+DDgQwRng47noOczYn1osuQyZ0LnNeSnA09jcG6EdfqrJiCsM p7lfTmhfxII2nAoFpLqdNU0ADLmpgjk= Received: by mail-yb1-f178.google.com with SMTP id 3f1490d57ef6-babb985f9c8so3642459276.1 for ; Sun, 21 May 2023 22:18:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1684732708; x=1687324708; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=7TjV4RyPitx7LIAI3Pj7MSABK+uluBQcp84mqUzVC6k=; b=Rn20sER4T4J8WT0bCSNgtKt+s3EuFbG8GtWiix1BcIQNeo/R8lvZKZibiwh3YZIsxd EnCfCzxhNamXvgrmaGBWZwQLLOogQl2zJkptkNsZuwZJP2V7Ysbew+9PodV2Eo/dF6Rz xAM/3hkG4T96NLraRyIuY8t8M3JyxrJk7Ta1kFceKAinzE25KLkprmyFpjg3Ymt8wWQE 3xmly1wR7KhvAsdh/yuCHd1F4QD89xMkxfEmWA4IIW5JNutOsZzL6TS+snO9aVVtcasO VB24BtEFsnlH0J+Nvd70D0Zxa1Rw4VEJiHNtQC31uldZTzMR/gntM0yqANgBYRGqCRYm U+Ow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684732708; x=1687324708; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=7TjV4RyPitx7LIAI3Pj7MSABK+uluBQcp84mqUzVC6k=; b=kqVCDVjceecSUvm7XKy2iZuECYyuI0R1z3Ly7h0t3VKs+FxySal4XS4LVc0Db6LBpS al1nhr8k6i4I4deCGj51EZe8RGqaCso9o2BoSpbNkp2nkjgeg3ykzoI4yu1xKKS+oC5R yk2+7FDEQCEK2kgXSFRT+ZqxfF3o9YJCV4cvDal4Nn4Pqn6t9V6raoQvkchyuhaUyoCI jM7GT+khLZ3aDhxT6pUbM9whbSQBNqAD7YRe0x/ChB2/636rb+SCaK8lUrSqSxlXhuWw SN1fMYZMkx5i9LQkwHJfSS03kKTysrd6ERxgUegURi0u8NoHGzaLNEkrOSQuoQPNfUaz ImkA== X-Gm-Message-State: AC+VfDzzBbdJMP9KY55wqzCot7U1Jv+Umrd2eD5dURM9ge1Xg98/a/SB Ww4vdjmlNBzukdHM/kvrUHL0Kg== X-Google-Smtp-Source: ACHHUZ6v2mlgLkozXZxDjiAaqAiQT2KR+fKn2zWVFJn2KWprg4vexj/qA5jscdHYSkrbKxnnqDZJ7A== X-Received: by 2002:a0d:df8b:0:b0:55a:8b11:5f6a with SMTP id i133-20020a0ddf8b000000b0055a8b115f6amr8985666ywe.19.1684732707681; Sun, 21 May 2023 22:18:27 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id u130-20020a816088000000b00552df52450csm1818581ywb.88.2023.05.21.22.18.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 21 May 2023 22:18:27 -0700 (PDT) Date: Sun, 21 May 2023 22:18:24 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 22/31] mm/swapoff: allow pte_offset_map[_lock]() to fail In-Reply-To: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> Message-ID: <619c27-d7b0-ae71-329e-9da3d3e7fc7@google.com> References: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: A91AC12000D X-Stat-Signature: os9p3uq7tfhcsuujf4dqo7qpcdqcwx4n X-HE-Tag: 1684732708-986405 X-HE-Meta: U2FsdGVkX19USTsfzTFZjJfrGda7uxq5E2WlCW6wzdfOhHgS3YQeJcEOW2KEp9IAcXHKb3TGdTSELPP66UWsXSqCcmwAugE9rl5oIL5hOvfOklJGQ6BHx0HzQJJDNlYUZldifZchmzJF1gDxCQQDgh/1K9qPa7vWMnH2Re3PIVERlbjtXd7/zC+lPiL+fsDqqBcgN+zY2U3PrbeePiVqrbbRtqzN0R8tFzZwESaiJNpZwyVeHh6H+IYd3wN0JKN7GeuLhSMiSD/2DL7gNKosU3pkNbfQZY9dpE2gNFk2ItNT73WzUrOh/ZEWvAXmneQDqr0O7v9uVi1xnve+ikVDq4POH4DWEXXTz4B8jOtLsFBC6Wmje2oZnNyKq/yw1WOyORPiMJGjTzAdf63cnwai7JwYQperYDd/rSKJTnYVTSCvCoH01U9CrZnYbkmVc98GVARRx/qhBHCrPpmtRkrw107sWWPsPXYyLKd3N+kqPkgfPLzMx3Krz7L0/7V92UY+LI5ON3kwTxeaI/wr195dBod91dCVtAy449lvUAmsnoy9JV+uxjk4HpIM+/WzDXHncXI/D50xSuYHuRnVEiZ8b7sOzh7QiDJwAxOwPG6o38OQ9phoTMQ5RltEebq0qzGQCWotVsZzQLT7A5nV5Iy5VMoKWl47M1Ml+DaTYBY554ZxzqyTRoQ2YtKI4o5KXiXDgTifMN7pUh18WFTdH5mA6gsNmBJJKX5XHIrgxFAjWszKJjwJdIj+mAFNLd8k+lthz0AT86wyQP298jTmEULIvl/gdOcF7A802WjzbYsRTjvQLggEqsWk834PgX5Ds93JtPDrARJPHSa2p4Bnqs1mFCGpeM/8UqTdQFIF2qWnw7GKvuhmNbjCtDqU9LaY+Rlw4/waJwWn40IkPQwSOLCKT/twh1QJwLmuhFFErFATWmrWT7HoEHYxhSgq9jJ3QDUJPi+0GyAZuKOHNIcnHYI NqQajYKN Q0FescSSU4AnUzYssza6IExAuKv1idgjrz4PeHwO+91YBAP5FXWZ0TLh4Gzvtc9j1MSzSbfQuw32Xf76PPPjcubMF1Nl3x31UjwN/8RZUIvQbY7dX1uijZ52nvddpF+KWcMrDeR66ZHDwh6v6miCNw7akhQj+aa/CMk5E4QVIP6q78c/NDKs5gf6N/BGEy0J+ihglx1qIENvTStwE6fJ/iqf2jCgIvM12NLPvb4E2b4B5SQNKgnFirN0PWix5idGY9LDUdaYFJpH/R0/HCm3RQoHGMw6JK039PN5j13hcWfmDmpC/YX2fkukdTe4gsePxhYNkSQiSlww+ZCc1DXrsBrV/i7DztlkPI78XeXHBepGay5cEjNJki19QdS7R3BxXllD3oX+/lpnDEh9qClGnQYn4zgq73gQpa+5X3vf+U4aYhvFrns+80tRoVw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Adjust unuse_pte() and unuse_pte_range() to allow pte_offset_map_lock() and pte_offset_map() failure; remove pmd_none_or_trans_huge_or_clear_bad() from unuse_pmd_range() now that pte_offset_map() does all that itself. Signed-off-by: Hugh Dickins --- mm/swapfile.c | 38 ++++++++++++++++++++------------------ 1 file changed, 20 insertions(+), 18 deletions(-) diff --git a/mm/swapfile.c b/mm/swapfile.c index 274bbf797480..12d204e6dae2 100644 --- a/mm/swapfile.c +++ b/mm/swapfile.c @@ -1774,7 +1774,7 @@ static int unuse_pte(struct vm_area_struct *vma, pmd_t *pmd, hwposioned = true; pte = pte_offset_map_lock(vma->vm_mm, pmd, addr, &ptl); - if (unlikely(!pte_same_as_swp(*pte, swp_entry_to_pte(entry)))) { + if (unlikely(!pte || !pte_same_as_swp(*pte, swp_entry_to_pte(entry)))) { ret = 0; goto out; } @@ -1827,7 +1827,8 @@ static int unuse_pte(struct vm_area_struct *vma, pmd_t *pmd, set_pte_at(vma->vm_mm, addr, pte, new_pte); swap_free(entry); out: - pte_unmap_unlock(pte, ptl); + if (pte) + pte_unmap_unlock(pte, ptl); if (page != swapcache) { unlock_page(page); put_page(page); @@ -1839,17 +1840,22 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd, unsigned long addr, unsigned long end, unsigned int type) { - swp_entry_t entry; - pte_t *pte; + pte_t *pte = NULL; struct swap_info_struct *si; - int ret = 0; si = swap_info[type]; - pte = pte_offset_map(pmd, addr); do { struct folio *folio; unsigned long offset; unsigned char swp_count; + swp_entry_t entry; + int ret; + + if (!pte++) { + pte = pte_offset_map(pmd, addr); + if (!pte) + break; + } if (!is_swap_pte(*pte)) continue; @@ -1860,6 +1866,8 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd, offset = swp_offset(entry); pte_unmap(pte); + pte = NULL; + folio = swap_cache_get_folio(entry, vma, addr); if (!folio) { struct page *page; @@ -1878,8 +1886,7 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd, if (!folio) { swp_count = READ_ONCE(si->swap_map[offset]); if (swp_count == 0 || swp_count == SWAP_MAP_BAD) - goto try_next; - + continue; return -ENOMEM; } @@ -1889,20 +1896,17 @@ static int unuse_pte_range(struct vm_area_struct *vma, pmd_t *pmd, if (ret < 0) { folio_unlock(folio); folio_put(folio); - goto out; + return ret; } folio_free_swap(folio); folio_unlock(folio); folio_put(folio); -try_next: - pte = pte_offset_map(pmd, addr); - } while (pte++, addr += PAGE_SIZE, addr != end); - pte_unmap(pte - 1); + } while (addr += PAGE_SIZE, addr != end); - ret = 0; -out: - return ret; + if (pte) + pte_unmap(pte); + return 0; } static inline int unuse_pmd_range(struct vm_area_struct *vma, pud_t *pud, @@ -1917,8 +1921,6 @@ static inline int unuse_pmd_range(struct vm_area_struct *vma, pud_t *pud, do { cond_resched(); next = pmd_addr_end(addr, end); - if (pmd_none_or_trans_huge_or_clear_bad(pmd)) - continue; ret = unuse_pte_range(vma, pmd, addr, next, type); if (ret) return ret; From patchwork Mon May 22 05:19:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13249784 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 09567C7EE23 for ; Mon, 22 May 2023 05:19:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 84A5B6B0075; Mon, 22 May 2023 01:19:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7F9206B0078; Mon, 22 May 2023 01:19:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6C158900002; Mon, 22 May 2023 01:19:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 5FD006B0075 for ; Mon, 22 May 2023 01:19:50 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 3707A1206FE for ; Mon, 22 May 2023 05:19:50 +0000 (UTC) X-FDA: 80816738940.21.F46CE79 Received: from mail-yb1-f179.google.com (mail-yb1-f179.google.com [209.85.219.179]) by imf08.hostedemail.com (Postfix) with ESMTP id 6CF35160014 for ; Mon, 22 May 2023 05:19:48 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=c16Z4sMp; spf=pass (imf08.hostedemail.com: domain of hughd@google.com designates 209.85.219.179 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684732788; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ooTKFpGgFIG6tpdJyTdSPdnhROWHUT8BFdRqrlDYiG0=; b=MC4PQPCELr8rX0FJYU7sO7QUvTc27QgUgPM8Be8kOC6DXiFhPjl8r0kS20hDTlpTLYvnHP uJP9vsE7YJWBYFkxEIy6kdF2gj49ufpjfjUzdp1nuveHVQAadHdLWHQdF08cDA27/DfaJ1 NPMLFiJ1ZQWDy1CK0ysoL+MJV5jjBU8= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684732788; a=rsa-sha256; cv=none; b=DgMdGqgAw/ioZOyeoTNtxOSJscr2L/am3rTUqhVYiH3moLttSZ1k0dA+6GZv5oHQwuZMDD 6Lhy+DuJIIURCGUpOjOMtRPX1YG+gwIG2gjkHPNGUtYxX/RhNlPexnJl8SVBtLbYy9MAoO 7hcRcoVR49J5Ovy3YiD6tjE3UqODY54= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=c16Z4sMp; spf=pass (imf08.hostedemail.com: domain of hughd@google.com designates 209.85.219.179 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f179.google.com with SMTP id 3f1490d57ef6-ba1815e12efso4782694276.3 for ; Sun, 21 May 2023 22:19:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1684732787; x=1687324787; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=ooTKFpGgFIG6tpdJyTdSPdnhROWHUT8BFdRqrlDYiG0=; b=c16Z4sMpjKlgLDYv6Gs/y1EYdzJZLSgcmaTkBaIf9Gh+M6R5Fj28OTQhOSCRw7EEv5 CoXdOMLH5hzAb7ctwTVEzalqQl9a+Um/b9wFPM2bKdaAzb1WrTE0ZJDEimE6nNgaQriX ekKs2u5sm5n3wdVCVhRqBQ64w5hCu6jc8AAKpTKma9h0EDNZhsm+d/uAlNMvz+IWh9Vi IdWQzCcYU/nxwQLTQZI99miGWNLqkFHHVO984XkRjgpZ7eBUTR4yIiwEf+nz+MbqBbbu qrDeOVuSWS3vwtPVFhgFnPNPF1tXG11tH/YIDKTxJb7j0INA2XTKzLvkIppZ9XEZec1f V+Og== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684732787; x=1687324787; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=ooTKFpGgFIG6tpdJyTdSPdnhROWHUT8BFdRqrlDYiG0=; b=IBtjuk9RY649N9PuOahvX2bsiuw1slStFGsiClsFzwrs9y0GGuodYzCFixZCeLSJNa BdUkhKhm8lpmH/tr9BbZR2IX92CjFRHUUk+w+R1yWfrB+W+CMDRc0PglsbbVL+Ojx8/h zZImen97TqLkBTZu2/79QimP7nP/cxAgYifjGpOP8vqrKT15tsfRIO3DGFjV4CulPcjj l1U0WZdh0ktPZ3N/m29qf8DMMbIeBHrKqycTpqzJZZjirHXY5sd1NhZCn8z+fzPbCCOT N7suJgPfCmGSP+D3MFwByuDtCJ8Vgr29H3zH2p4/+uQxD6TydL8dtPG04AGjRgcf0uUC IeJQ== X-Gm-Message-State: AC+VfDwocDr35x0YNFDDGemkM4qNfpCy1UqybJRSyaJBD6oawiEg8MDz C5AlebNNGc3fWxNInM1XE5bPQQ== X-Google-Smtp-Source: ACHHUZ4/iOac0pg6azmy6Y4+9ZIQvlTnkM+d3E/2233dtMLIq4uN1+AwZtw93iYqYZBi+HEHW+NHVQ== X-Received: by 2002:a0d:d743:0:b0:55a:59cb:4c1a with SMTP id z64-20020a0dd743000000b0055a59cb4c1amr9770012ywd.14.1684732787446; Sun, 21 May 2023 22:19:47 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id o9-20020a817309000000b00556aa81f615sm1809311ywc.68.2023.05.21.22.19.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 21 May 2023 22:19:47 -0700 (PDT) Date: Sun, 21 May 2023 22:19:44 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 23/31] mm/mglru: allow pte_offset_map_nolock() to fail In-Reply-To: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> Message-ID: <242721-1e64-845e-226a-bf2b2dc72dd@google.com> References: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> MIME-Version: 1.0 X-Stat-Signature: xxmgtxpiicfzbhzzt4gw1tagnanwcbrw X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 6CF35160014 X-Rspam-User: X-HE-Tag: 1684732788-93997 X-HE-Meta: U2FsdGVkX19TnBuv9XpdPF4iaYvkdG1dcOoI2XT3D5rQTCmzGH7wg0xsbfi/tKLwsa+kV8k1AW8Sil5f3HvS+8RA/89yjPNgr426yoyJGGb4aLJbS5GbQin40vwqYZ8pubEKun7WcJtcgcinjIB2nFIkhWmWu4mrjFH5ObfWU6am9Exg2c3a4tYKyQW76kX9+1a2jktUdKT2uLl7KjkjmpoPsZOHaI+uf04z9UQKqUI2KuZUnCEU1rH2owh7JK9TwEsPhW/5GnGmWlQVEwXHu0cuYZNqpmnSKSnknU0jnNXyq2bYr4OjoVYgfmY5taFyHjFnunFdndZ3aBLyKKhDiAvC3oIa29mmKi1rK1u7ci3xhT4r1yZduHF2OV6ifWCR+YJ8nIZgOSmQeUUz/fmbmNjRIy9OjfSFzpzSbjREXqWA87mfrsLW9WIBGgOStspiYGxnhJOvPT+mASAtAKc21eca1cj1SPFEdMqiYwZqD3znMUi5N7WpG6v5BKAAf2alfq8mtN/yJEJxODJnWAL2M1CWInqxzz3nlWyzdDjGvmWpzeGlJApIcEDYmboMNKUiRcMIeAh3cJ1H+iadNK2n291TcbwZ70hlepBjiHlVu5X6W+uB+AVslCPjBl8y0QYDU9f9rt8huDN5NRtuKB8mxYopgdS1CHbOEXOc8cv6uSf3pTNdB4aZ03n2zkYI1dGo3coWN0Dl0l6g6tw9lSwldrhkwD4P0AGtEj5fygMesK49Cxi5RGN0F0mzJYtQ//gqiBsNDcfTPdHHiN+dbhTbcCFpWOaJjpIg7jIBpam1yhLpygjkkbdZaq5e+RGbhtOzBFjpuxhQoV1FNpUJdqwuQf1vT9m2vLNhLXpC1eHlXNzWgnphvzqUB2jTVZh2kAW4uobMueCk72BJLnU7SnzlTXA03FvEoG+Jy++cVkho/bG2mJAfovNzLd6Jks6R/KENdgV1BLxmQiXAobJoWWK ITIWqoVn Y3GP6ISRZvV5HRYoxmN2g7u1GzlIHD4v1GhrvE2qv2wWtwsl6NLSqOn2ZdACUvNEljky7L0VErIfbc+FxMo/Cm7k4pF9cboO+4+4PCiXBHfQ9Ef85eG6QDZyvEXqj3Y0bIw6OLGBy8nsreZRy0UxoEcPmWEH+wpH0uRmNv7mzQhGQtMxcaUPUTXmqRfxFs7rtav4hupXyM+qqs0p5ysfjv6FtRIK46UDwXb2lPjp+J50+SmE6fthIv7JEtGBfJ5X9VDasC/u7d19QsWAX9IHhVdX/rJ0pu2q7PhuFls8xhFADSMgV5c9La0UHSbAMgrpQQuY31V8Ej0laKw3IaaKFEXw5CAfkEcd5VfONYYyG+6lXe+i0OBIpGkS/bAP6b2jVmBrNrP9rMcAExNZ9QQVxmU6Rp2bhH2qBRF8xQj1zHMBpLJ+01m666otGng== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: MGLRU's walk_pte_range() use the safer pte_offset_map_nolock(), rather than pte_lockptr(), to get the ptl for its trylock. Just return false and move on to next extent if it fails, like when the trylock fails. Remove the VM_WARN_ON_ONCE(pmd_leaf) since that will happen, rarely. Signed-off-by: Hugh Dickins Acked-by: Yu Zhao --- mm/vmscan.c | 16 +++++++--------- 1 file changed, 7 insertions(+), 9 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index d257916f39e5..1c344589c145 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -3992,15 +3992,15 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, struct pglist_data *pgdat = lruvec_pgdat(walk->lruvec); int old_gen, new_gen = lru_gen_from_seq(walk->max_seq); - VM_WARN_ON_ONCE(pmd_leaf(*pmd)); - - ptl = pte_lockptr(args->mm, pmd); - if (!spin_trylock(ptl)) + pte = pte_offset_map_nolock(args->mm, pmd, start & PMD_MASK, &ptl); + if (!pte) return false; + if (!spin_trylock(ptl)) { + pte_unmap(pte); + return false; + } arch_enter_lazy_mmu_mode(); - - pte = pte_offset_map(pmd, start & PMD_MASK); restart: for (i = pte_index(start), addr = start; addr != end; i++, addr += PAGE_SIZE) { unsigned long pfn; @@ -4041,10 +4041,8 @@ static bool walk_pte_range(pmd_t *pmd, unsigned long start, unsigned long end, if (i < PTRS_PER_PTE && get_next_vma(PMD_MASK, PAGE_SIZE, args, &start, &end)) goto restart; - pte_unmap(pte); - arch_leave_lazy_mmu_mode(); - spin_unlock(ptl); + pte_unmap_unlock(pte, ptl); return suitable_to_scan(total, young); } From patchwork Mon May 22 05:20:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13249785 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B5BB7C7EE23 for ; Mon, 22 May 2023 05:20:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5136F6B0075; Mon, 22 May 2023 01:20:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 49DEF6B0078; Mon, 22 May 2023 01:20:59 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3172E900002; Mon, 22 May 2023 01:20:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 1EFEB6B0075 for ; Mon, 22 May 2023 01:20:59 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id D70CAC0633 for ; Mon, 22 May 2023 05:20:58 +0000 (UTC) X-FDA: 80816741796.16.10774E9 Received: from mail-yb1-f180.google.com (mail-yb1-f180.google.com [209.85.219.180]) by imf23.hostedemail.com (Postfix) with ESMTP id 1165D140009 for ; Mon, 22 May 2023 05:20:56 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=YFRKfsKF; spf=pass (imf23.hostedemail.com: domain of hughd@google.com designates 209.85.219.180 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684732857; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=j6etDg/swU7AOCalaMxD05iIweXIhmZHtm6GGNoHnzQ=; b=C6jLFACHO54mBdO/H0fwMOLE6diQ5QOOns2HzuTILCUjDCvJOPKqiegjJ3OKOz0nFeKaRX 7oO3PF2jc1IiRVKKisM7Na1nKG26Puvt+z1TTU46aaBzKLXXkZRDJujHzcBeYV0lBvW2Tc 0iDmFlffCa4lYXAzW7XLh+XIVbVE22Y= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684732857; a=rsa-sha256; cv=none; b=D7elWljm1Ld1/HjW8cDopzs7J7VIBLsGLgn+EOfV9smPFDWHBU28EcnyVV7mRGbD4gsfKs Zht+VA8YwFrsH9azjFNyLaDJEOLJtAQyrhE8DpuQHHYzBPGDR9g17s9SKep1eA6sYBcu8N kyAeEWs/avSUk3G6eTT0OYRssZAwxaI= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=YFRKfsKF; spf=pass (imf23.hostedemail.com: domain of hughd@google.com designates 209.85.219.180 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f180.google.com with SMTP id 3f1490d57ef6-ba8afcc82c0so8176110276.2 for ; Sun, 21 May 2023 22:20:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1684732856; x=1687324856; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=j6etDg/swU7AOCalaMxD05iIweXIhmZHtm6GGNoHnzQ=; b=YFRKfsKFJp2z/omMF9kAApd0S0vxQuFTDhSSSfTCRKzoqcrub4PhM+O1boooV8+h+7 EnjnOkymlppoTkDw4nRAPps7rFoXTdCM2fk5vgu0nhCdGNkuTC3Ywg1Tg0z3smilw57a AYhYR543c86roHNURCr25oJ9c1fQFj3ujvNOV5N67eBFAEXegF1f2oEskky6yYw53RB6 1LMrFi8toFy6m/MnfxFA7DcCZIfsKlCmP01wxE3ks9bt8Xfi1FEm4Lsd/b5mwwUQxQvr vvJIRG6Xgs+dtaUMASMBUyR0SSOfetdWJ8Iw1sW/hI3XPXcU9B/tW4Jbs4RLezTnbIt4 iIvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684732856; x=1687324856; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=j6etDg/swU7AOCalaMxD05iIweXIhmZHtm6GGNoHnzQ=; b=Wz1CLYdhGy8UKs6kLbgPAEBVbOvebKUTpof9PTcjf3rluHuSEImVt0dPHKHr6JzlO9 xPzUP6rLzHkPmp6oTfX/Fl+jy3cmBl80TwCXXp/rqfK4ALDiK3gl/eVewORaehgcHDqF wYY3p0kxl6imitlSG5FHSFkpGdD/UaTrajd8EKEokIcb2t2DmHTqRcu1+AFhBoLAN7Ye +69e6UM9Fu5CSj+VWHRUE9dZTILTftD2FzGD33jW/n6u57YkkAbUu1qRFsn4Cer+QwOu uKzJfePZ19afNUeJRK6leVnM4aGXe5EDZjs2yU8c4pZ7ga3RRYshBvsKs3tpltAbLGH7 gu0A== X-Gm-Message-State: AC+VfDypWwNzAE9de9mlj0NJpxCxqU3bCQA50plXB3dYKyusYEzk/qjv hqHG5LEo8qDQOCNlNmg/MUccZg== X-Google-Smtp-Source: ACHHUZ4unls6ogV9SvyZExwOGgG/jyFNvH+bFxzEVTKzBIPGmZGTzZNTpUXsl12MimVOmHUoz8BiUw== X-Received: by 2002:a0d:ea93:0:b0:559:f0ef:aac0 with SMTP id t141-20020a0dea93000000b00559f0efaac0mr10004964ywe.30.1684732856030; Sun, 21 May 2023 22:20:56 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id i16-20020a0ddf10000000b0054f80928ea4sm1795763ywe.140.2023.05.21.22.20.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 21 May 2023 22:20:55 -0700 (PDT) Date: Sun, 21 May 2023 22:20:52 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 24/31] mm/migrate_device: allow pte_offset_map_lock() to fail In-Reply-To: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> Message-ID: References: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 1165D140009 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: 1zx11hzg5t6omxhyygfxjg48nwii8k7u X-HE-Tag: 1684732856-928958 X-HE-Meta: U2FsdGVkX1/J9Cr5q9uPj5PpPKvml8HpV/ML7WSd8z7LsdaxSkC+BxP5eNIGvJ2J4+XVrbeKOQeNs2ftc8+0qTTcThhu7Bz7ajTTNPtkuQS5SiQfzP2pOdlK4SS1Yp2JeoY8x7oNYCJVNNCk53WxL3sWZx0INpsugFPoxpj/g5Jz7ELte6PhyHuEkfacYxKZBr9nnjWZFzenbtdrWZ/SxcC22y60+0ovcohsXBlCTEP+iShAnmP0wHluvpaZtKnTtKARl+17OsU8MBtxMA3FSD+Ob0TpAdo9VXQKL+zvZZDSPmRYTcd7JNKpdvwOqygu2SevyeGlIBtNl40tWfNlJexFdG2SaNH+jGLftR/6UYsWUzjJTAEiNbLMOnhfLMWuoCaoAb4piti4XabMuOl3jGn5I0sxsKpnUKdBfRHIKwVK98MPsORf2Xe6hM4Igo1Id61x21YS9cwmVooZpjHbE2wjHzJfic9m2hN00CNTTnL97z6zQAq7ryc+5YRkcngxtJdzO6rUA2xG3SbN3iEHN2CTVGkcxGRjsp3Oqsea9TN3bQeA232K1iN9wfAR8oOXSErRrTaOqeaDhAJnL5w72y3f0PlUYvL/uotyQ1mjO81OpqiUxsWXixxvZJx/j+840PoT1oC3PZT3iqVf/TBKtFWYvJhGv17P5kab0U3ottpqJ47UCGzdfdYExe2MQwgLBZ40lYys3fqnOtM6XG4EkL4V18DPndmLN28V2HE3h+qt3sETjWqpJGKYiDwIXxWuE4LyTmv0IGu3fWViSHe76aGvOZoc0aWx+lXOgMh6tOcAs2i8kFs3zPhIrs4RiMQPulnn5HvqixVPU/IH+aIrANJhX4UsgLcQ1Dwm6BOvDeEiSUp2TpdD0fNV7p5tYZBz37gfJBsIfjBYFyJfYoraq76bFp6NoohnXhLZHldpWzzmFeyHczgxNGKfCVHPas4RG0FtpD8KURrLNbgJ6hg KAMX6fi4 yTRxL+9MRWdgZRCUwXCORXVX9Equi7bmkBl+Yb1itiUISbzaJeOhanV3+LHTWIaUx5PE1MOpLdHPAnmAaJ7MmCKHT1UEkuDRvmXp72qsk6+LcTYqZTI5Zwy/ukCrpC5jVly3hRE4J1VDEOLlstI/AZtopwD5YYBT6vIeXkw1/V8dBxLi9WJNDp73Q06STZIkmrfzD8I+4hcj+ds2xFL3+L/4P/tjeR0BmaaplN6vD3VoNSSFz1yj7vocUPlAmUWOPsgoPAwNZxczEHlGZ6CLexOK3aE5QYbMRpLdpAoKRpylRmsc21/6dfmWJUNh7aFPph/qpJVlkTbZmTsVlWXhGYqIZNH8qUTF/At7VJYlSJYg9CvDYAVdgWxRU04cxPxvNWkwBgbb2owjqiv783b+xUV3LvudKzL+8rVZYxIbTBVuPDc+6Lgqx7QxoaA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: migrate_vma_collect_pmd(): remove the pmd_trans_unstable() handling after splitting huge zero pmd, and the pmd_none() handling after successfully splitting huge page: those are now managed inside pte_offset_map_lock(), and by "goto again" when it fails. But the skip after unsuccessful split_huge_page() must stay: it avoids an endless loop. The skip when pmd_bad()? Remove that: it will be treated as a hole rather than a skip once cleared by pte_offset_map_lock(), but with different timing that would be so anyway; and it's arguably best to leave the pmd_bad() handling centralized there. migrate_vma_insert_page(): remove comment on the old pte_offset_map() and old locking limitations; remove the pmd_trans_unstable() check and just proceed to pte_offset_map_lock(), aborting when it fails (page has now been charged to memcg, but that's so in other cases, and presumably uncharged later). Signed-off-by: Hugh Dickins Reviewed-by: Alistair Popple --- mm/migrate_device.c | 31 ++++--------------------------- 1 file changed, 4 insertions(+), 27 deletions(-) diff --git a/mm/migrate_device.c b/mm/migrate_device.c index d30c9de60b0d..a14af6b12b04 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -83,9 +83,6 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, if (is_huge_zero_page(page)) { spin_unlock(ptl); split_huge_pmd(vma, pmdp, addr); - if (pmd_trans_unstable(pmdp)) - return migrate_vma_collect_skip(start, end, - walk); } else { int ret; @@ -100,16 +97,12 @@ static int migrate_vma_collect_pmd(pmd_t *pmdp, if (ret) return migrate_vma_collect_skip(start, end, walk); - if (pmd_none(*pmdp)) - return migrate_vma_collect_hole(start, end, -1, - walk); } } - if (unlikely(pmd_bad(*pmdp))) - return migrate_vma_collect_skip(start, end, walk); - ptep = pte_offset_map_lock(mm, pmdp, addr, &ptl); + if (!ptep) + goto again; arch_enter_lazy_mmu_mode(); for (; addr < end; addr += PAGE_SIZE, ptep++) { @@ -595,27 +588,10 @@ static void migrate_vma_insert_page(struct migrate_vma *migrate, pmdp = pmd_alloc(mm, pudp, addr); if (!pmdp) goto abort; - if (pmd_trans_huge(*pmdp) || pmd_devmap(*pmdp)) goto abort; - - /* - * Use pte_alloc() instead of pte_alloc_map(). We can't run - * pte_offset_map() on pmds where a huge pmd might be created - * from a different thread. - * - * pte_alloc_map() is safe to use under mmap_write_lock(mm) or when - * parallel threads are excluded by other means. - * - * Here we only have mmap_read_lock(mm). - */ if (pte_alloc(mm, pmdp)) goto abort; - - /* See the comment in pte_alloc_one_map() */ - if (unlikely(pmd_trans_unstable(pmdp))) - goto abort; - if (unlikely(anon_vma_prepare(vma))) goto abort; if (mem_cgroup_charge(page_folio(page), vma->vm_mm, GFP_KERNEL)) @@ -650,7 +626,8 @@ static void migrate_vma_insert_page(struct migrate_vma *migrate, } ptep = pte_offset_map_lock(mm, pmdp, addr, &ptl); - + if (!ptep) + goto abort; if (check_stable_address_space(mm)) goto unlock_abort; From patchwork Mon May 22 05:22:13 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13249786 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8861FC7EE23 for ; Mon, 22 May 2023 05:22:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 144946B0075; Mon, 22 May 2023 01:22:21 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0CC5A6B0078; Mon, 22 May 2023 01:22:21 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EB10C900002; Mon, 22 May 2023 01:22:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id DBB2A6B0075 for ; Mon, 22 May 2023 01:22:20 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id A27D3C0432 for ; Mon, 22 May 2023 05:22:20 +0000 (UTC) X-FDA: 80816745240.08.A9B638D Received: from mail-yb1-f176.google.com (mail-yb1-f176.google.com [209.85.219.176]) by imf01.hostedemail.com (Postfix) with ESMTP id CD8FF4000D for ; Mon, 22 May 2023 05:22:17 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=xxliRdiY; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf01.hostedemail.com: domain of hughd@google.com designates 209.85.219.176 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684732937; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=bf4ovlbip9o2eAgzZYzeIq4fNWAxMs47VtohjgZo/Hc=; b=EGtpdPsSfeulOzpSgsxFQpZQ1zMvDV1MUeq2WqkQyJpxhWvd3okek3JB/naaxtNd2RK6QN 814EC89WxHBOgv0liYh+WIrA/Sb5mj8mxFIE6Y30jmrpC2cgDcragyioylxU9lHY5A7Rem blJFDeP2WMJkG35Kh+v+HVr99FlAFtk= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=xxliRdiY; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf01.hostedemail.com: domain of hughd@google.com designates 209.85.219.176 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684732937; a=rsa-sha256; cv=none; b=hDiHtUTnXwyNLLjM9bKJRj5Y8St2XUnBz9tROMb2GrOXfUkBCKyz+Vzw1yHsBero6R/Mve R0bWAWdJad0BcpP67p0UAzucebgVuBHMHYEwW77ekSoJjS1grYH6CwjxEmB77wnhXVzzo/ NVdp9/6hx/x3mSLtdpOE51NC3cM6ldo= Received: by mail-yb1-f176.google.com with SMTP id 3f1490d57ef6-ba80dc46972so8132797276.2 for ; Sun, 21 May 2023 22:22:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1684732937; x=1687324937; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=bf4ovlbip9o2eAgzZYzeIq4fNWAxMs47VtohjgZo/Hc=; b=xxliRdiYq491ddtLAOp7IvlP6Tgs09+WixlM6YG4PIP9hmkihY34Mp7pqS8UXaXWhi RqkBsAewEsCIX9b0ejcCa5E2jV6uN+a3W57TZ3FlG9ZI/Xxbjh+C1MSqcwhmQFBu25PN rRzcJXLhWTtkRDhu7m269tBf4pXwAXhsMjAzMdb8HFMIiX2bUkp/5SZfLmQ8JxH4T0vk wKyQPeorZ9iGKOGhWx6oCTIHOln7Bg8pdmjG1PPm7ouSR3zthhZvslly93YlThiYWoSt Uq6if1E3nt9EtldOeJnlIf2uspBMjaO0EQ7qGUle/0SCGanLsme+EJIaES6VTaJRp/ZR yfYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684732937; x=1687324937; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=bf4ovlbip9o2eAgzZYzeIq4fNWAxMs47VtohjgZo/Hc=; b=YlIvjbOMdsEPSeDfv52FU74NR+7CKVIHP9nRN4H0qa1g25LQz9E5pXH+/CP9AQRADi 5/R3UFQkT2lNiN1h0G/fY3yepq3IB5PxNjGKXfc4JdTHbPOlRAhJmQb7KE2O51yPBGPg QEnVdNXSNFioAK6QZcW0FRi9J4pp4lTE/WEnbxAhEuVh79HZNrtYQ4xmeDNbRLrIqDlc UaxTn8ne3bcYNEuGmlE6ARcx+fVg2PqMvb8GwI68rjrwTC5AfNt39wlNXQiI99hgrmT7 xeMP3wNMjDjXO1tQpYuY2nNMygNYbYW0F6GNT7mCfMyeATALndWYNbnWt8J5VsR/WsCG txIw== X-Gm-Message-State: AC+VfDxRYg3Sxkoy6S+81zhPuoAPJPbRp4S/Pyfnpz10XFLviUcEhl3u sdZvh5xpXlo8AG7tjz411KNj1A== X-Google-Smtp-Source: ACHHUZ6vQXHbZLVvPerE0Fu3jg6OZ5EHbThB/3RtX3VnGbimP3VA+F9C2VHWinuyuwC+mVweYmLjPw== X-Received: by 2002:a25:f812:0:b0:ba8:3590:4302 with SMTP id u18-20020a25f812000000b00ba835904302mr10377914ybd.36.1684732936746; Sun, 21 May 2023 22:22:16 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id x7-20020a259a07000000b00b8f6ec5a955sm1266497ybn.49.2023.05.21.22.22.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 21 May 2023 22:22:16 -0700 (PDT) Date: Sun, 21 May 2023 22:22:13 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 25/31] mm/gup: remove FOLL_SPLIT_PMD use of pmd_trans_unstable() In-Reply-To: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> Message-ID: References: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: CD8FF4000D X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: wrt1kbty51bk5pj6ctx56inh388ttpst X-HE-Tag: 1684732937-824645 X-HE-Meta: U2FsdGVkX188zd3gnMvxtjV17dyUPyedETMov3ikzRDlxtKCfWfGqN3v9SGVPXJFOVgrnlxTm2XI771GuFh8o387RQrncQXpLE8IcOrwTUpybyvhZx9I6sZkEu9w9zoaWV7vIWBb6aRByqZ4ALuLSChrtNYeBLzouVYM4+IefR9AyzizEHSvsAYt+UJRnBMvDLkmcJMkYEyoz5t95haXZOupMnsfEcAsmQKSKddYjgCorwEe+klqv5ZhvB7gocV+jH7l/75VRBtyH2Hn8Nptp0F5GGYiYUctzpcK1ofW7q+ZTXj3HCjRO6bLeMvpcg7hIje/BqJuOJhQVyjvUUrJv4ljf/or/6oNODeQGPoVtA7yJupn6Q3dmucPMeJeu2TwOEZROVLglXkCDydklzobmJ28auJ3WZto2xpUP3ZatdAd8Dkh7p59kwcaSPTKJb6bApube1cQPUMffjJBoVI6lwvsgxIXss7VG5he44lHEpCuB3ulaNSqPLgU+3P3JiCQ9H/4K0e5SfNH66JzPLAhkOMY03fPmjL+L4WCe5AfCrjs4GVh5Dx46FujUd9tWFh03341TlaIF2i1jf0EhUCWN/S/BCj8mdz1QcZzPQPbSD8TpvWzpnW7rhomUWGFtP/lEREsMolVKc5XyEeP/BIfhftMqGTPrVR2GUQwZ1MYEmoEdcSQZlg9eue3OXay7X2yqcnMOL2j/2B9hqrTIfEG1flT5RWVBm0d4r3LzeJr75JxLvadqtj/atWeCviXKj7yWWX+mFGhNtSEpjQVB4xpeJKGCJrjZSe651VSKmjAbl7iI2o8pWMeK4iYf6n6rh5J1VY5un3iyMYrHgmDyamw9O3PTsihY+fkBktZc7AeXCZ+TgfaA9M+4RL9/s3ybUs4Bk1bheX/2rOs8xyUUgGKdmkmhRkib/QV+eXtfwMNTQMAZgaZ23O8fBoVwQUhd+bzLYSRgUTUCRKsqJNgje9 yFAV4P6e XFpNZ4dlEeeTP9qrrdldRi5uiBvWyIesbXUJtX7O8n7OvntAjp7pYvYuWiW3jFP5lH4lamvGJWm7L6pCKwn4QJi828SeU8tqZMu00WnKXU/UZT2Bv4/73j2bTJpqQMjtpBbRv84plZQ0USCIDG/ZcoQngCQtgwlxCj/EaZUVCAL6bnBkp92THHDO/D6L6Kn5AdX+w4ZgP64QPwjnXwbaGGWDpPcXw8fKI3/XV0Ky9Kd6o6gWrs+4HUt11kwyrW/hJvc4OeZoQd1c8eC/N49J5kpVq4+Xz8gozRr0QjuLyeDm86+wT/3kWDO7ZvUlFf89Yu40cDp91SjJrCNKWiwPxc7pwjAXUagtzQNr3C7hcrsQHzcmkaF0eDEFPjJdAeoh3IiLoN+JlC4bEDf0cdFFj++UG1DY/qK1XSYPGyCCnoWmD3PIsOTtHfwUWvw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: There is now no reason for follow_pmd_mask()'s FOLL_SPLIT_PMD block to distinguish huge_zero_page from a normal THP: follow_page_pte() handles any instability, and here it's a good idea to replace any pmd_none(*pmd) by a page table a.s.a.p, in the huge_zero_page case as for a normal THP. (Hmm, couldn't the normal THP case have hit an unstably refaulted THP before? But there are only two, exceptional, users of FOLL_SPLIT_PMD.) Signed-off-by: Hugh Dickins --- mm/gup.c | 19 ++++--------------- 1 file changed, 4 insertions(+), 15 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index bb67193c5460..4ad50a59897f 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -681,21 +681,10 @@ static struct page *follow_pmd_mask(struct vm_area_struct *vma, return follow_page_pte(vma, address, pmd, flags, &ctx->pgmap); } if (flags & FOLL_SPLIT_PMD) { - int ret; - page = pmd_page(*pmd); - if (is_huge_zero_page(page)) { - spin_unlock(ptl); - ret = 0; - split_huge_pmd(vma, pmd, address); - if (pmd_trans_unstable(pmd)) - ret = -EBUSY; - } else { - spin_unlock(ptl); - split_huge_pmd(vma, pmd, address); - ret = pte_alloc(mm, pmd) ? -ENOMEM : 0; - } - - return ret ? ERR_PTR(ret) : + spin_unlock(ptl); + split_huge_pmd(vma, pmd, address); + /* If pmd was left empty, stuff a page table in there quickly */ + return pte_alloc(mm, pmd) ? ERR_PTR(-ENOMEM) : follow_page_pte(vma, address, pmd, flags, &ctx->pgmap); } page = follow_trans_huge_pmd(vma, address, pmd, flags); From patchwork Mon May 22 05:23:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13249787 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8AA60C7EE23 for ; Mon, 22 May 2023 05:23:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2F0F26B0075; Mon, 22 May 2023 01:23:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2C7646B0078; Mon, 22 May 2023 01:23:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 18F2C900002; Mon, 22 May 2023 01:23:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 057B76B0075 for ; Mon, 22 May 2023 01:23:32 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id D24BEA0576 for ; Mon, 22 May 2023 05:23:31 +0000 (UTC) X-FDA: 80816748222.03.9DCF290 Received: from mail-yb1-f176.google.com (mail-yb1-f176.google.com [209.85.219.176]) by imf10.hostedemail.com (Postfix) with ESMTP id 0F7CCC0010 for ; Mon, 22 May 2023 05:23:29 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=OqBANCRQ; spf=pass (imf10.hostedemail.com: domain of hughd@google.com designates 209.85.219.176 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684733010; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=lzQPoLjWXuP3+Ka40j/FX+cgzbLo3MyR/78LE/Otflc=; b=8CAJz4mw80FWprNm1bNlakEzeEDqNVNv8J1pThmURmVBy/wHBzRHfYla+wmns5K9KDbJbi tc9CQi6yZziTj8lfnh2+TnkKK+ivwShc/Ji5HhUwvFi1jsuiKxWwIVosKtzLdeIK1X50T8 w9oZeyKM1khqVm5BQmSOlcF6mqOcwAY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684733010; a=rsa-sha256; cv=none; b=M3AAG64VYeB3gtTcmCw93fdupxGI9PXsFRFD5gWcWCsTbSDwfWskO3o9CqW67irV4ybLpo uYjnYlQENwKt7bQM5pD4bLyeC5W+PSqtU7mSAqx2kSu+WuEUgPCXNY1R/oYpyVjTrv+NfY A0Fx3SPzYhFGK1LgKj7tX0gS5O1Fgjs= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=OqBANCRQ; spf=pass (imf10.hostedemail.com: domain of hughd@google.com designates 209.85.219.176 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yb1-f176.google.com with SMTP id 3f1490d57ef6-b9daef8681fso4793342276.1 for ; Sun, 21 May 2023 22:23:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1684733009; x=1687325009; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=lzQPoLjWXuP3+Ka40j/FX+cgzbLo3MyR/78LE/Otflc=; b=OqBANCRQzvanaas2fLT/vDBYw2XZ+pto0Z3I0g1aU/R1OYHuij2QrMo2xy+Rr6C35x l86kCF5UL52/GbeJlQ6bHEBRSpOVELHuSaKXNybOZhqbs+opPxUSDPmtJNEm+jrMaklN ft3L6EaR3I8ziuMCU5sNY2OaZ+zGmtKvbNZ9Tsv9F9B6wECrNMIjTpzlO5d+Z255LDVp yEk+S1bLKTQfkqEB9/oNMA15Vbb0/7X2Ek3v5V8dnbfTlUQxMGQNWKaElfOhfI5B5peu qIbnRaHYRIRCfQSqqRnyf0yf2fu8VPlbHJkeNdMJAA3hkVwFBfoZ10oZW3ZVmkHCSkfx eCng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684733009; x=1687325009; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=lzQPoLjWXuP3+Ka40j/FX+cgzbLo3MyR/78LE/Otflc=; b=U8Psbvja4Mskh+xk/fLIO2neGkq9dfuSwqPEWurI4GzlMEMzo89yuGDjSkNrnLEY4k wvQ0qeAnEMlPWukjmMGoSNGQZ08J+lSRa69PNAL8jsXPgKVvigeY5IX6EJ5Wv2ZZ5l9o +1c2QdkDYZ9Zt9AM0G0r40cxQHr3qb5f85IEgO95g/G3t9WKnWAIbqGUxwELrC2s2VyZ Aop28p8m2ZOw3HUxH8Wfk1pB1xkW/KFwai9yIjSVECwrk3Pj5AI1g67W8pGs2LLF28fk n57/KRcAd5aziEbqdZt5d7+bm5y3Fss1mH2/us0oXlOMPVf4484cbGE6D9YRqEwpuQbg nJYA== X-Gm-Message-State: AC+VfDwQTm+73c7hZP/b/kRJTGhnb0PPB5KlJU5ARKGi/u6GRT8mro+f f6o3r6Jwnp4ZQOwp4F1hd1RHSQ== X-Google-Smtp-Source: ACHHUZ7qqz3U4uOnP6Ec8LA+Qz8AEibK4IaZAP5jwPoGGf6Fy3SqZkPf36XnHstJeJW2gmcrXMZvlw== X-Received: by 2002:a25:d2c5:0:b0:ba7:20a:3967 with SMTP id j188-20020a25d2c5000000b00ba7020a3967mr11645090ybg.43.1684733009011; Sun, 21 May 2023 22:23:29 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id g10-20020a056902134a00b00b9db62abff3sm1277036ybu.58.2023.05.21.22.23.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 21 May 2023 22:23:28 -0700 (PDT) Date: Sun, 21 May 2023 22:23:25 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 26/31] mm/huge_memory: split huge pmd under one pte_offset_map() In-Reply-To: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> Message-ID: <3f442a9c-af6d-573d-1ad1-f6f413b1abc9@google.com> References: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> MIME-Version: 1.0 X-Stat-Signature: n8tgmy887coshsenjisup9dwjnm4qme9 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 0F7CCC0010 X-Rspam-User: X-HE-Tag: 1684733009-920569 X-HE-Meta: U2FsdGVkX1//5JgjsBJLSPkXPmObNbiO++3UbRwyCrlFC1BEf3X4D8LM4iKdiN5R/htPFuPZzBBcqBOu5PcuqvsWMolt7GGh9agbHWWiSU1fU9meoXSfJUQbfXMLCdNNZ/5MSBcpUFf+1kdHHJciADJ+ceO+uuca0TpT5K64HyuZzW7upqh+Qk4lS6lBF1Cz1Uoc8nap5nbgMAHiPn12XyBAGrc4TnpUr6pkrJ8/5Abzfo+G33laupKODQxjHMS0xoEp/dAddAc1w+MNQfABndC1kLpQkUvLF0ZHxhb3mT4sKA9IJO9mPraxORlYXat/UrmPfvKkQp1PfRNSMEJcyQ7OGS9J2VvdGZ4in5aBxBfi6iVgQdu6u/TcNTZYmNlSLnrjZqF7dkWIgY9uEUgBuFTS6xr4FTMFQYAqnbL36o0KE0me6NdRPQUPksrMzJT/PAIXt3FLTua1ilARiQWMIVvLC7jd1SJv6b2OwZUSPs0MyYt00b8S6gd7we3Pc3NQGY6xhdyQPTcZClIeg4DiGVGHxLwbzHhXWWKO2XL4Eo/mpHPblgQCJzGkBCLnIi9E3dmQoQX/XUP56lnQf7WLQGTLrTzDtv9EESe3LNCQAM3XA1v+EkRYx49/RNl640MKgX/vdl1OFK/q5/SuW/A5uWrjpsdWjh5MPfvbYy1as5x7gzcyoJzOVmZXDo+C/uBPrFpY7apZ3uh9TyTD9p9lcjJ9VpF41sl9RqferoXEWCsQV3jHhzEzkfcVhxWxn9Vtu0Nillu9zkDnnFkVIdXfGOh0fKyG3GOeVXmM1aMz/4VPPWX627ctVPbiO1X+948gRVBOGiPtDQUsjmkF3VoZ1f9d3fNaF5r5Y7hXKQpiNjfzyHMNm5o9kFQg4ybvShu/Uwo3AlC0C01YTMrr/jcwabGRACqscLLz9wnUxoUBs8BRiLK9R0iZgbSnae5fNmz2ZPxyjvCnDMDIjeRpfPr 5LRbLiJE 5f90r+hbTJCaxWErfLnBWzama9IGJYVD2haNyrI2+QsbPcEAzGLZ3mC18J5dZOu7Z8q7qCIPPAEW63o0AYLfYuCqyoMyBrYsCpwA90PcqrDc2DOB2OjvqxG/JoJP1NHTkqVY04RPVMQ10eNx1ctYUDA/xYjMzMj1HzoTaK23qFBODWfTFNOLKYtJISx4bTIAq3YlCFaTO3yOLe2LNhiTGtxrgqD65wQeGjbQmQzX2gB7dHtnSwSagfgQP1LFj0WISEbrx94a9LadzdeQQgZm0CpdhiQoB34VgdHlABJDL9ars74VGBKLeLrr5oaII4kKoKaPyOOYrQ7qjpBE2r4qX0CExEynFvtYEg13NLSt6zhUWx+3piagxPsdjN0XikawcwcLAVWiWVKkdx+m4DrMuxcQy//HUK3cMXfllWmB4lFIMDDZbuMWeJ0Fg3Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: __split_huge_zero_page_pmd() use a single pte_offset_map() to sweep the extent: it's already under pmd_lock(), so this is no worse for latency; and since it's supposed to have full control of the just-withdrawn page table, here choose to VM_BUG_ON if it were to fail. And please don't increment haddr by PAGE_SIZE, that should remain huge aligned: declare a separate addr (not a bugfix, but it was deceptive). __split_huge_pmd_locked() likewise (but it had declared a separate addr); and change its BUG_ON(!pte_none) to VM_BUG_ON, for consistency with zero (those deposited page tables are sometimes victims of random corruption). Signed-off-by: Hugh Dickins Reviewed-by: Yang Shi --- mm/huge_memory.c | 28 ++++++++++++++++++---------- 1 file changed, 18 insertions(+), 10 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index d4bd5fa7c823..839c13fa0bbe 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2037,6 +2037,8 @@ static void __split_huge_zero_page_pmd(struct vm_area_struct *vma, struct mm_struct *mm = vma->vm_mm; pgtable_t pgtable; pmd_t _pmd, old_pmd; + unsigned long addr; + pte_t *pte; int i; /* @@ -2052,17 +2054,20 @@ static void __split_huge_zero_page_pmd(struct vm_area_struct *vma, pgtable = pgtable_trans_huge_withdraw(mm, pmd); pmd_populate(mm, &_pmd, pgtable); - for (i = 0; i < HPAGE_PMD_NR; i++, haddr += PAGE_SIZE) { - pte_t *pte, entry; - entry = pfn_pte(my_zero_pfn(haddr), vma->vm_page_prot); + pte = pte_offset_map(&_pmd, haddr); + VM_BUG_ON(!pte); + for (i = 0, addr = haddr; i < HPAGE_PMD_NR; i++, addr += PAGE_SIZE) { + pte_t entry; + + entry = pfn_pte(my_zero_pfn(addr), vma->vm_page_prot); entry = pte_mkspecial(entry); if (pmd_uffd_wp(old_pmd)) entry = pte_mkuffd_wp(entry); - pte = pte_offset_map(&_pmd, haddr); VM_BUG_ON(!pte_none(*pte)); - set_pte_at(mm, haddr, pte, entry); - pte_unmap(pte); + set_pte_at(mm, addr, pte, entry); + pte++; } + pte_unmap(pte - 1); smp_wmb(); /* make pte visible before pmd */ pmd_populate(mm, pmd, pgtable); } @@ -2077,6 +2082,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, bool young, write, soft_dirty, pmd_migration = false, uffd_wp = false; bool anon_exclusive = false, dirty = false; unsigned long addr; + pte_t *pte; int i; VM_BUG_ON(haddr & ~HPAGE_PMD_MASK); @@ -2205,8 +2211,10 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, pgtable = pgtable_trans_huge_withdraw(mm, pmd); pmd_populate(mm, &_pmd, pgtable); + pte = pte_offset_map(&_pmd, haddr); + VM_BUG_ON(!pte); for (i = 0, addr = haddr; i < HPAGE_PMD_NR; i++, addr += PAGE_SIZE) { - pte_t entry, *pte; + pte_t entry; /* * Note that NUMA hinting access restrictions are not * transferred to avoid any possibility of altering @@ -2249,11 +2257,11 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, entry = pte_mkuffd_wp(entry); page_add_anon_rmap(page + i, vma, addr, false); } - pte = pte_offset_map(&_pmd, addr); - BUG_ON(!pte_none(*pte)); + VM_BUG_ON(!pte_none(*pte)); set_pte_at(mm, addr, pte, entry); - pte_unmap(pte); + pte++; } + pte_unmap(pte - 1); if (!pmd_migration) page_remove_rmap(page, vma, true); From patchwork Mon May 22 05:24:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13249788 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DB037C77B75 for ; Mon, 22 May 2023 05:24:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7D2696B0074; Mon, 22 May 2023 01:24:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 7A9B66B0075; Mon, 22 May 2023 01:24:37 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 672EE900002; Mon, 22 May 2023 01:24:37 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 58A146B0074 for ; Mon, 22 May 2023 01:24:37 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 397711C76AB for ; Mon, 22 May 2023 05:24:36 +0000 (UTC) X-FDA: 80816750952.20.B6F94C4 Received: from mail-yw1-f178.google.com (mail-yw1-f178.google.com [209.85.128.178]) by imf22.hostedemail.com (Postfix) with ESMTP id 67296C000B for ; Mon, 22 May 2023 05:24:34 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=FCyyq681; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf22.hostedemail.com: domain of hughd@google.com designates 209.85.128.178 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684733074; a=rsa-sha256; cv=none; b=RIGT6A0bXs2pTr2Wrc7FLdCb0YK3pHh2vN/2lSxwrjbB74VD7SM+5MrOI8JWBdAX+TMvBR hS+jGHQfptxedMa2uCd7MyhT/To4Pil8e28UBJ/YoQm/dEfa5Q3+6eK43MKJGGveNlzx+D OEsatisV51h1H464yrKaEuwhEMfBxh4= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=FCyyq681; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf22.hostedemail.com: domain of hughd@google.com designates 209.85.128.178 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684733074; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=9aUzicIbTynIqE9awb1KUIJU7FYy6E6iJhesHPP8TVA=; b=r2ooazRXBCyyE6CT3Z2KWhambzVnhsiDCEqbm4IuDqVclv+ssDH65XbX8QRlJvG52kj65C TnJqHHlcA13G93Ss+65v2MViYZqsWCzBipya7FBoFe1m1936dqSWRbyFY6q4nygUdcMghK 1A3FQE6sJqLy5MEV/UvObClRYuosq2g= Received: by mail-yw1-f178.google.com with SMTP id 00721157ae682-561f23dc55aso56059007b3.3 for ; Sun, 21 May 2023 22:24:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1684733073; x=1687325073; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=9aUzicIbTynIqE9awb1KUIJU7FYy6E6iJhesHPP8TVA=; b=FCyyq681fiCl9+MbURk3V7VS6u/az3wH03OAXe0MvSy7niV7M+xCTcD3LFVHKqg1yQ ike7yvjl4pZ53eTIQqFm0t5pi4r6AsMUJr4uYJMWzl0YHx8ZPfzSAps1Y7Xpi29TZEWv oF3xEY1+E9kXiv/OXuel6wSjjlu4EU8KZchqn8ox3RdwN6yhtrdyc37IOWvkIHSmyayR wIwLertYdChnzsX/BWtNMS+OritTekYale8j0KLTkGM2OoyaSuh4ZrnXK1pihTZaxeir zEXvAqI7WXP5/kZeI5P2fpOTxrVe/RXgYq45Sc6JeGL/JmIWIWy78rlaXhstJBzFaLu3 cGzA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684733073; x=1687325073; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=9aUzicIbTynIqE9awb1KUIJU7FYy6E6iJhesHPP8TVA=; b=QJbqQIfVA7SazdEaIxX35k4HUDh5tgNiy+FHx0IbDwdKOQfQDik6tYsszVQ+UyXbOn cMVi5/bF4z2e+emeoly8bxBbqWK+o89mg2PP4zZr0xBlhztv7eSwaQdQlkRyRAVzIBWK s2/oTSt4ueBUNCjUY1QAVAI3CnPNEjCCW0KsmJOXRrCXW9gbPMJzELjK420VC5jbJW2O TiSgLZdA8serIZuF5irXhtEf9IMFMZa13CNRfEz8Bf3A9LhVyB0fAeca2Mp97mJnBAos K4i3BLoFC3lcBURJu2TLpsOVVf3KaAQVLh/fAVn2Dz8OxB7L0y+ZzbnvpEcsStCnjXD9 fZzQ== X-Gm-Message-State: AC+VfDxQnenlzMVyBJSgRW9f/QlWIiVFEFsi9zrAJIVzxwCEPfGpeaLv sjOfpjhGp1iRR0zCt+dE4L4LwA== X-Google-Smtp-Source: ACHHUZ5k0AE/kz26DrRpHOYfkL1f8tXo8j4DB4kKzH+Hlu6k3+yZrZ26Tvg/gU/iOROW9T7xGJmWkw== X-Received: by 2002:a81:8a01:0:b0:561:b5cc:e10a with SMTP id a1-20020a818a01000000b00561b5cce10amr9111246ywg.6.1684733073356; Sun, 21 May 2023 22:24:33 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id g189-20020a8152c6000000b00555e1886350sm1840019ywb.78.2023.05.21.22.24.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 21 May 2023 22:24:33 -0700 (PDT) Date: Sun, 21 May 2023 22:24:29 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 27/31] mm/khugepaged: allow pte_offset_map[_lock]() to fail In-Reply-To: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> Message-ID: References: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 67296C000B X-Stat-Signature: combg5n8chx35d4eqmsart38wpzmc56a X-HE-Tag: 1684733074-371172 X-HE-Meta: U2FsdGVkX19/n+FO/OqvxTMe9acQIqIrF/qDUBcmBPuADSSnG3uLbLDG0x7+4RKyQr7Za42qMTRHdZU9twfKU0y7Heyaf9IW/J7uWbTSBxB4alZf9NxWYQTMHqVW42/pfiFa5ISw9asIFI7ORGsprIThCmIpoB+4SzIJRPzUv04Bhd62fSrB8urV/9KkESGNpoSubXDjj7gBbRgyWZChySwcf6Ca/5kEP1xvsGThVW3qzetYl5PpqFLOh46l2OvJp+j1zzql5eoD1HORnnw8mQNXeJe7hGY/it5fF7AudjCQMnn5JhKAVbuB12hNN3ELii9PSdG4a3t54uLPh5N3M/4pxaEDGqkvAW2N0qSkY5AtsP2m6R4L+AjPBg1F0rTLqZg/tUcwnOcfEyjDSwf4YLYoFisEh+nrACMlNT1fPFuMG8H5rRRP3gjFlZNvg9fDbrGlXN0mXYJEOXmUag1XwgN5tcTwFKRLgk6FczNtw7/RTMiyNNp/O/RVxJ2qd2Lt3x11eC2oT+a0AWVYxEqDSziYd0Jv/ScSFl8cm8DsqW0qPdrx9cZpCmtc90WE4ogpXXWQN3A5nq/KDr92YbCHZSwUG82v2m6mLBTi1XL52sDCD6Or9KYy6eQ97gG9QsNbGj4Com7bw3lhOL/jPNOIq1qfLrqPKiO7AfafeZ+SXAYAu7E0Wmx83dHwhHIAbOjiqdz+ZSNpfCDJWih9Jhtax3IvaMHB3Eweo6lEA9h9bdtd5HD4uf0/SVMqchskc3Y0ACkzicAqseOhBPnOhyItPW1jn7or/DkARw2ukN13SuqHKzUD3C5DHSj05wPe0ZswdMdPrka1hlZV8XSQQjx7li78c05iqRP2hGzWnfyi61lsZ1Z4P3T4DHjZUnRQrO6OLf2eFzt/nLw1jnQuBrntNCQuSdy1yXOKKtmj1gqzB0tpgb90WSCuDfMv5KFLPYRwcLPDmvuLNMCVgKIEm0t dtcbBgGJ paGlwQZvFkkivZa1TPlA3rTJHrFKj3Y+E3BIa+Zcd7hSrcci4xknP+Fr0tkQ+Qg5mE8k3b4Bv4mC1dWeFsGF0KbTQ0r2/ZMr53tN2N56QkaSB8Tap2g/dbR3drHTgZGDzVhZrq3iXYUF40gerLLjXOYF3C9UMNnFHQC+rrqUV3iuP3jB+kdltIpiUzjmZOxFdr8xna4SHums6tkit5PG/bsXvCh+E5qcJdTTCesSY7ZlXre2hDCm3SJHg3ei516afzmfik+WRzJtMAekpLhU/4zRSIN/VYauEotfn1ahRE6cCe+TflTZTip5oVSovAgCJ5+I5Xonr60aT0EErHAM0ptAa66nsPr2gvJs/fY5+88Ch3BFGdjvWaXfoKVGNVjcqz9cOW8DgMb1cDCPXyDqj+OoaNRgcGuSCEss5RzG6hzS4xdKK9EB9xTjclg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: __collapse_huge_page_swapin(): don't drop the map after every pte, it only has to be dropped by do_swap_page(); give up if pte_offset_map() fails; trace_mm_collapse_huge_page_swapin() at the end, with result; fix comment on returned result; fix vmf.pgoff, though it's not used. collapse_huge_page(): use pte_offset_map_lock() on the _pmd returned from clearing; allow failure, but it should be impossible there. hpage_collapse_scan_pmd() and collapse_pte_mapped_thp() allow for pte_offset_map_lock() failure. Signed-off-by: Hugh Dickins Reviewed-by: Yang Shi --- mm/khugepaged.c | 72 +++++++++++++++++++++++++++++++++---------------- 1 file changed, 49 insertions(+), 23 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 732f9ac393fc..49cfa7cdfe93 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -993,9 +993,8 @@ static int check_pmd_still_valid(struct mm_struct *mm, * Only done if hpage_collapse_scan_pmd believes it is worthwhile. * * Called and returns without pte mapped or spinlocks held. - * Note that if false is returned, mmap_lock will be released. + * Returns result: if not SCAN_SUCCEED, mmap_lock has been released. */ - static int __collapse_huge_page_swapin(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long haddr, pmd_t *pmd, @@ -1004,23 +1003,35 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm, int swapped_in = 0; vm_fault_t ret = 0; unsigned long address, end = haddr + (HPAGE_PMD_NR * PAGE_SIZE); + int result; + pte_t *pte = NULL; for (address = haddr; address < end; address += PAGE_SIZE) { struct vm_fault vmf = { .vma = vma, .address = address, - .pgoff = linear_page_index(vma, haddr), + .pgoff = linear_page_index(vma, address), .flags = FAULT_FLAG_ALLOW_RETRY, .pmd = pmd, }; - vmf.pte = pte_offset_map(pmd, address); - vmf.orig_pte = *vmf.pte; - if (!is_swap_pte(vmf.orig_pte)) { - pte_unmap(vmf.pte); - continue; + if (!pte++) { + pte = pte_offset_map(pmd, address); + if (!pte) { + mmap_read_unlock(mm); + result = SCAN_PMD_NULL; + goto out; + } } + + vmf.orig_pte = *pte; + if (!is_swap_pte(vmf.orig_pte)) + continue; + + vmf.pte = pte; ret = do_swap_page(&vmf); + /* Which unmaps pte (after perhaps re-checking the entry) */ + pte = NULL; /* * do_swap_page returns VM_FAULT_RETRY with released mmap_lock. @@ -1029,24 +1040,29 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm, * resulting in later failure. */ if (ret & VM_FAULT_RETRY) { - trace_mm_collapse_huge_page_swapin(mm, swapped_in, referenced, 0); /* Likely, but not guaranteed, that page lock failed */ - return SCAN_PAGE_LOCK; + result = SCAN_PAGE_LOCK; + goto out; } if (ret & VM_FAULT_ERROR) { mmap_read_unlock(mm); - trace_mm_collapse_huge_page_swapin(mm, swapped_in, referenced, 0); - return SCAN_FAIL; + result = SCAN_FAIL; + goto out; } swapped_in++; } + if (pte) + pte_unmap(pte); + /* Drain LRU add pagevec to remove extra pin on the swapped in pages */ if (swapped_in) lru_add_drain(); - trace_mm_collapse_huge_page_swapin(mm, swapped_in, referenced, 1); - return SCAN_SUCCEED; + result = SCAN_SUCCEED; +out: + trace_mm_collapse_huge_page_swapin(mm, swapped_in, referenced, result); + return result; } static int alloc_charge_hpage(struct page **hpage, struct mm_struct *mm, @@ -1146,9 +1162,6 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, address + HPAGE_PMD_SIZE); mmu_notifier_invalidate_range_start(&range); - pte = pte_offset_map(pmd, address); - pte_ptl = pte_lockptr(mm, pmd); - pmd_ptl = pmd_lock(mm, pmd); /* probably unnecessary */ /* * This removes any huge TLB entry from the CPU so we won't allow @@ -1163,13 +1176,18 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, mmu_notifier_invalidate_range_end(&range); tlb_remove_table_sync_one(); - spin_lock(pte_ptl); - result = __collapse_huge_page_isolate(vma, address, pte, cc, - &compound_pagelist); - spin_unlock(pte_ptl); + pte = pte_offset_map_lock(mm, &_pmd, address, &pte_ptl); + if (pte) { + result = __collapse_huge_page_isolate(vma, address, pte, cc, + &compound_pagelist); + spin_unlock(pte_ptl); + } else { + result = SCAN_PMD_NULL; + } if (unlikely(result != SCAN_SUCCEED)) { - pte_unmap(pte); + if (pte) + pte_unmap(pte); spin_lock(pmd_ptl); BUG_ON(!pmd_none(*pmd)); /* @@ -1253,6 +1271,11 @@ static int hpage_collapse_scan_pmd(struct mm_struct *mm, memset(cc->node_load, 0, sizeof(cc->node_load)); nodes_clear(cc->alloc_nmask); pte = pte_offset_map_lock(mm, pmd, address, &ptl); + if (!pte) { + result = SCAN_PMD_NULL; + goto out; + } + for (_address = address, _pte = pte; _pte < pte + HPAGE_PMD_NR; _pte++, _address += PAGE_SIZE) { pte_t pteval = *_pte; @@ -1622,8 +1645,10 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, * lockless_pages_from_mm() and the hardware page walker can access page * tables while all the high-level locks are held in write mode. */ - start_pte = pte_offset_map_lock(mm, pmd, haddr, &ptl); result = SCAN_FAIL; + start_pte = pte_offset_map_lock(mm, pmd, haddr, &ptl); + if (!start_pte) + goto drop_immap; /* step 1: check all mapped PTEs are to the right huge page */ for (i = 0, addr = haddr, pte = start_pte; @@ -1697,6 +1722,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, abort: pte_unmap_unlock(start_pte, ptl); +drop_immap: i_mmap_unlock_write(vma->vm_file->f_mapping); goto drop_hpage; } From patchwork Mon May 22 05:25:25 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13249789 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 79009C7EE23 for ; Mon, 22 May 2023 05:25:32 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 142396B0075; Mon, 22 May 2023 01:25:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0F2C56B0078; Mon, 22 May 2023 01:25:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EFDEA900002; Mon, 22 May 2023 01:25:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id E06066B0075 for ; Mon, 22 May 2023 01:25:31 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id A5691140422 for ; Mon, 22 May 2023 05:25:31 +0000 (UTC) X-FDA: 80816753262.17.EF180E6 Received: from mail-yb1-f173.google.com (mail-yb1-f173.google.com [209.85.219.173]) by imf17.hostedemail.com (Postfix) with ESMTP id C564B4000A for ; Mon, 22 May 2023 05:25:29 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=Cd3nHwxa; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf17.hostedemail.com: domain of hughd@google.com designates 209.85.219.173 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684733129; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=NBl0ppCRA8l25HwCsBAYjZw+ZZ23COv4vEuhAo6MgQs=; b=4IAqIWJPSoSdBBtvRm+B0njJi/2YOY1UHopYQKLxS/MA04DC54qg7mA/E5W/dTv2TY473I nTiBPmJQ/YhOR164pKDgGBcCS2QX6e0fHBrsPl5Tg5xlPzvwNzJTUZwt461org+ZDUyBA0 I25GB/juQak9sac/Zq3FJP8EIvjJBgw= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=Cd3nHwxa; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf17.hostedemail.com: domain of hughd@google.com designates 209.85.219.173 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684733129; a=rsa-sha256; cv=none; b=O0hIg9BOa8Xc9sTNi6GvdEAHRelmGIG7pJIcAUcmGBaaSipzc1PfTo5UFkYfW4DykV1ZZn AN2S8kqf/LPjSXD/WEHfZmu/EohM74yKMXiBBIlSavva6hMiFCCCaLDQ6XMP1UrL3SfgRF zFCvjOTYpQtLruutinXNt/ATLMzT2I0= Received: by mail-yb1-f173.google.com with SMTP id 3f1490d57ef6-ba8afcc82c0so8181973276.2 for ; Sun, 21 May 2023 22:25:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1684733129; x=1687325129; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=NBl0ppCRA8l25HwCsBAYjZw+ZZ23COv4vEuhAo6MgQs=; b=Cd3nHwxaC/nh6HEQWhU6+zHLQfimMyJGpQmSRQFhhW/GPP873cSkYFOiS5uVG1tykM Ryshue+/dZ9gOhxIxMYC4NRprObjBBWLSBCffW9HayG++u4h8tLgWq5NECMQwyEn52g4 13GfY4O2VekWiqVxfPenJhxKSjbH9x7XqjT7KEDC9iyAKT/BVUfykNPLsCku8blfINLG XrU8Vy7pHm8Cw4m+KEkicTGibPauERe+Pk26G/c3KRes7WKtltas6Fuf97+EzswuIMvR NjmExJ27Xjr53poHCBGkRWEPVuUwRpPQE56LGrkD5LJsjNMZNI4MUNIfvWEA+VYa9OvY 6Vyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684733129; x=1687325129; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=NBl0ppCRA8l25HwCsBAYjZw+ZZ23COv4vEuhAo6MgQs=; b=N8wLMSh7h5q3OnJQdxmy4ckkiooKrYlfetRI14f2Xn3kx6c0fJPXRH9GxElL84itQU /pFD7MMoh+LQy8/az/C1/OuVNvmg4dwApbjNZLyL8+/NRD6aMhLnbcghcr5jinJLYT4o G9qCt+75vsI5WJGmNKYNBGkI1vjBPINkeq8x9n0/OrkFaRLnXcBhPet7EtFI4+yDLKqN qyknbWJp6aw324pRQUT1hLnUvSkd5O2ywPmrzFI4Xp1v97zIf/RylCqJQLQfKWQ4d/vb 82JLShrHvWRMxmhGDgAA8Z4FzOOXP6zJ7pIuSWjTwSON8UabOKONpsY2eqxks/gafsRP WGFw== X-Gm-Message-State: AC+VfDxTRaIDgsgR3FTEu/FUriqc1f5faXlLLH+yVa6TRbIAyL5Hfs7y i2G0+WS0SxuHSQh50weVS7dADw== X-Google-Smtp-Source: ACHHUZ73cLvyzajJn77JRRVp9Omy0BAayWsdbmLAc2xDyGvyYdthxJkt0VkVvx7l16HAMXlwEB2XiA== X-Received: by 2002:a05:6902:1101:b0:ba7:3df3:6df5 with SMTP id o1-20020a056902110100b00ba73df36df5mr12099433ybu.38.1684733128711; Sun, 21 May 2023 22:25:28 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id x7-20020a259a07000000b00b8f6ec5a955sm1267873ybn.49.2023.05.21.22.25.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 21 May 2023 22:25:28 -0700 (PDT) Date: Sun, 21 May 2023 22:25:25 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 28/31] mm/memory: allow pte_offset_map[_lock]() to fail In-Reply-To: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> Message-ID: References: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: C564B4000A X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: gom8gunymcsme5r1cyo35n9trs8brrtr X-HE-Tag: 1684733129-943386 X-HE-Meta: U2FsdGVkX18nGsvr4uQhvshnBzke0dblYHwrRt51lr269D4eX64cRaTvGDLr82z9ff90GYw/4bMQ9nRjNbmU6vKP/GOyRpgYG62fiUZc6jy3zXG3fzxEWU5syoPxHnpyGdWD+pmP17ci7cWBKWHeTWG6f1j9Hg5jVpKsH1B2QRS40N7LNZy9ouPQOR6CDwNxmzlQxHfU60Dt5RhsirH5MpeBz/YbG0+u4wiJw390TwWcqhwzpJ5YKd9CSGAMGPi2fWjX0IDI2O/1hGXYGv7hA4O98iJEDhOqD1UHqxN3wTg4weOEfxzi5BPnED+iFmS3rokZ1SPWCBSkiU4DRLhFviRirmzPT5+H29eZAdeQtrKn4fC/PEC/Uaz+c/WR8eJ2QjNvpdaMMiU9hLqkTqQqeMQcxasEh+XIp0oUq/QUaeCGsEOvia/kiAVWYk9usefAp+3jlYh84xwdeKpzM1dvGzuTmRjDeG1qZco6apZTlcpUXve9N7s+S7nAoS060vjoucau1GqUVPgMk56sT/KFpG2itjqiPQa4Fc8pomBOS6lPE2ZjVwwHqhZK6QdCvF312LvY/fWpU7lZPT+IrnM15jbKqYZmm9koeID2nvFewqOnt4Sc2TWMo6tW7qpa8KkclpUc/g9/sf6+bzCLCkuC7uXuSX5MTmKi9oOmW2ZEF5guivjrV2XGS+xUEwxByVaVAO2QTZW16LmdJmZlU/mOq8NnxBOuyDGr2kcdoDRXmpVSB5Gcj+8PjEtYJ73Gl4h7jBBmbClYFU6xE9Oa4Er9uIMvMHOxA8iU0e2EDvAnl5IIuCpVfR2quyNGm0MjgfiYti6rwcyr/6F0UW9/nbDFwJiHvUczA/zAEdQmYz3GIHKYGsE0H/2xLZDLZtTSEGlG0PtDCvmutOnzk9CmPN5b9KUxuf0ndokcMF6xBgIZk121+qjyjQvPAm5rGSkN59oC1Fxqvk8g+vaact0/Jbc TUmjKmoe rncZZDUqwo4aHYcK3SHFz1rAvlhRZxpOM6QBSRmsJfLD5UE8D7fjaZwrxS6jFsK1+hFCSahqJ8Y7OMuVdfacLtUmneGku2fw0ciqEQCMZn9EZD7i/GaVx72qOLdBdQs90Svj9euggZoPmaiKxsj4v2vQhz16CXdOqZQrWT9yBsJfpmS6N/MZoT8Hxh56vG6v38m6IhzqNfqWiIPLJN2kXt7ysMswczAySDx7Xn8N42rxLRQtmbM1rM6DlZVQLzufwmCWasuMAECUQC+2aQqMBVLkKi1rRONnHfGS49soeZw5atudkH3Fp8qZCT0CibCEo3lQpcyQfdPi0rHz9syFBI51uW75VaC6MOX8Tq+UVu0AIfDqHytpP8tfKCNA/nIEIDvhCaLcgLApCe5KjchULhaKQgFvI/SZ1T7ExImCPKXZE1WbgjfIsYwQNNQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: copy_pte_range(): use pte_offset_map_nolock(), and allow for it to fail; but with a comment on some further assumptions that are being made there. zap_pte_range() and zap_pmd_range(): adjust their interaction so that a pte_offset_map_lock() failure in zap_pte_range() leads to a retry in zap_pmd_range(); remove call to pmd_none_or_trans_huge_or_clear_bad(). Allow pte_offset_map_lock() to fail in many functions. Update comment on calling pte_alloc() in do_anonymous_page(). Remove redundant calls to pmd_trans_unstable(), pmd_devmap_trans_unstable(), pmd_none() and pmd_bad(); but leave pmd_none_or_clear_bad() calls in free_pmd_range() and copy_pmd_range(), those do simplify the next level down. Signed-off-by: Hugh Dickins --- mm/memory.c | 172 +++++++++++++++++++++++++--------------------------- 1 file changed, 82 insertions(+), 90 deletions(-) diff --git a/mm/memory.c b/mm/memory.c index 2eb54c0d5d3c..c7b920291a72 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1012,13 +1012,25 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, progress = 0; init_rss_vec(rss); + /* + * copy_pmd_range()'s prior pmd_none_or_clear_bad(src_pmd), and the + * error handling here, assume that exclusive mmap_lock on dst and src + * protects anon from unexpected THP transitions; with shmem and file + * protected by mmap_lock-less collapse skipping areas with anon_vma + * (whereas vma_needs_copy() skips areas without anon_vma). A rework + * can remove such assumptions later, but this is good enough for now. + */ dst_pte = pte_alloc_map_lock(dst_mm, dst_pmd, addr, &dst_ptl); if (!dst_pte) { ret = -ENOMEM; goto out; } - src_pte = pte_offset_map(src_pmd, addr); - src_ptl = pte_lockptr(src_mm, src_pmd); + src_pte = pte_offset_map_nolock(src_mm, src_pmd, addr, &src_ptl); + if (!src_pte) { + pte_unmap_unlock(dst_pte, dst_ptl); + /* ret == 0 */ + goto out; + } spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); orig_src_pte = src_pte; orig_dst_pte = dst_pte; @@ -1083,8 +1095,7 @@ copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, } while (dst_pte++, src_pte++, addr += PAGE_SIZE, addr != end); arch_leave_lazy_mmu_mode(); - spin_unlock(src_ptl); - pte_unmap(orig_src_pte); + pte_unmap_unlock(orig_src_pte, src_ptl); add_mm_rss_vec(dst_mm, rss); pte_unmap_unlock(orig_dst_pte, dst_ptl); cond_resched(); @@ -1388,10 +1399,11 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, swp_entry_t entry; tlb_change_page_size(tlb, PAGE_SIZE); -again: init_rss_vec(rss); - start_pte = pte_offset_map_lock(mm, pmd, addr, &ptl); - pte = start_pte; + start_pte = pte = pte_offset_map_lock(mm, pmd, addr, &ptl); + if (!pte) + return addr; + flush_tlb_batched_pending(mm); arch_enter_lazy_mmu_mode(); do { @@ -1507,17 +1519,10 @@ static unsigned long zap_pte_range(struct mmu_gather *tlb, * If we forced a TLB flush (either due to running out of * batch buffers or because we needed to flush dirty TLB * entries before releasing the ptl), free the batched - * memory too. Restart if we didn't do everything. + * memory too. Come back again if we didn't do everything. */ - if (force_flush) { - force_flush = 0; + if (force_flush) tlb_flush_mmu(tlb); - } - - if (addr != end) { - cond_resched(); - goto again; - } return addr; } @@ -1536,8 +1541,10 @@ static inline unsigned long zap_pmd_range(struct mmu_gather *tlb, if (is_swap_pmd(*pmd) || pmd_trans_huge(*pmd) || pmd_devmap(*pmd)) { if (next - addr != HPAGE_PMD_SIZE) __split_huge_pmd(vma, pmd, addr, false, NULL); - else if (zap_huge_pmd(tlb, vma, pmd, addr)) - goto next; + else if (zap_huge_pmd(tlb, vma, pmd, addr)) { + addr = next; + continue; + } /* fall through */ } else if (details && details->single_folio && folio_test_pmd_mappable(details->single_folio) && @@ -1550,20 +1557,14 @@ static inline unsigned long zap_pmd_range(struct mmu_gather *tlb, */ spin_unlock(ptl); } - - /* - * Here there can be other concurrent MADV_DONTNEED or - * trans huge page faults running, and if the pmd is - * none or trans huge it can change under us. This is - * because MADV_DONTNEED holds the mmap_lock in read - * mode. - */ - if (pmd_none_or_trans_huge_or_clear_bad(pmd)) - goto next; - next = zap_pte_range(tlb, vma, pmd, addr, next, details); -next: - cond_resched(); - } while (pmd++, addr = next, addr != end); + if (pmd_none(*pmd)) { + addr = next; + continue; + } + addr = zap_pte_range(tlb, vma, pmd, addr, next, details); + if (addr != next) + pmd--; + } while (pmd++, cond_resched(), addr != end); return addr; } @@ -1905,6 +1906,10 @@ static int insert_pages(struct vm_area_struct *vma, unsigned long addr, const int batch_size = min_t(int, pages_to_write_in_pmd, 8); start_pte = pte_offset_map_lock(mm, pmd, addr, &pte_lock); + if (!start_pte) { + ret = -EFAULT; + goto out; + } for (pte = start_pte; pte_idx < batch_size; ++pte, ++pte_idx) { int err = insert_page_in_batch_locked(vma, pte, addr, pages[curr_page_idx], prot); @@ -2572,10 +2577,10 @@ static int apply_to_pte_range(struct mm_struct *mm, pmd_t *pmd, mapped_pte = pte = (mm == &init_mm) ? pte_offset_kernel(pmd, addr) : pte_offset_map_lock(mm, pmd, addr, &ptl); + if (!pte) + return -EINVAL; } - BUG_ON(pmd_huge(*pmd)); - arch_enter_lazy_mmu_mode(); if (fn) { @@ -2804,7 +2809,6 @@ static inline int __wp_page_copy_user(struct page *dst, struct page *src, int ret; void *kaddr; void __user *uaddr; - bool locked = false; struct vm_area_struct *vma = vmf->vma; struct mm_struct *mm = vma->vm_mm; unsigned long addr = vmf->address; @@ -2830,12 +2834,12 @@ static inline int __wp_page_copy_user(struct page *dst, struct page *src, * On architectures with software "accessed" bits, we would * take a double page fault, so mark it accessed here. */ + vmf->pte = NULL; if (!arch_has_hw_pte_young() && !pte_young(vmf->orig_pte)) { pte_t entry; vmf->pte = pte_offset_map_lock(mm, vmf->pmd, addr, &vmf->ptl); - locked = true; - if (!likely(pte_same(*vmf->pte, vmf->orig_pte))) { + if (unlikely(!vmf->pte || !pte_same(*vmf->pte, vmf->orig_pte))) { /* * Other thread has already handled the fault * and update local tlb only @@ -2857,13 +2861,12 @@ static inline int __wp_page_copy_user(struct page *dst, struct page *src, * zeroes. */ if (__copy_from_user_inatomic(kaddr, uaddr, PAGE_SIZE)) { - if (locked) + if (vmf->pte) goto warn; /* Re-validate under PTL if the page is still mapped */ vmf->pte = pte_offset_map_lock(mm, vmf->pmd, addr, &vmf->ptl); - locked = true; - if (!likely(pte_same(*vmf->pte, vmf->orig_pte))) { + if (unlikely(!vmf->pte || !pte_same(*vmf->pte, vmf->orig_pte))) { /* The PTE changed under us, update local tlb */ update_mmu_tlb(vma, addr, vmf->pte); ret = -EAGAIN; @@ -2888,7 +2891,7 @@ static inline int __wp_page_copy_user(struct page *dst, struct page *src, ret = 0; pte_unlock: - if (locked) + if (vmf->pte) pte_unmap_unlock(vmf->pte, vmf->ptl); kunmap_atomic(kaddr); flush_dcache_page(dst); @@ -3110,7 +3113,7 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) * Re-check the pte - we dropped the lock */ vmf->pte = pte_offset_map_lock(mm, vmf->pmd, vmf->address, &vmf->ptl); - if (likely(pte_same(*vmf->pte, vmf->orig_pte))) { + if (likely(vmf->pte && pte_same(*vmf->pte, vmf->orig_pte))) { if (old_folio) { if (!folio_test_anon(old_folio)) { dec_mm_counter(mm, mm_counter_file(&old_folio->page)); @@ -3178,19 +3181,20 @@ static vm_fault_t wp_page_copy(struct vm_fault *vmf) /* Free the old page.. */ new_folio = old_folio; page_copied = 1; - } else { + pte_unmap_unlock(vmf->pte, vmf->ptl); + } else if (vmf->pte) { update_mmu_tlb(vma, vmf->address, vmf->pte); + pte_unmap_unlock(vmf->pte, vmf->ptl); } - if (new_folio) - folio_put(new_folio); - - pte_unmap_unlock(vmf->pte, vmf->ptl); /* * No need to double call mmu_notifier->invalidate_range() callback as * the above ptep_clear_flush_notify() did already call it. */ mmu_notifier_invalidate_range_only_end(&range); + + if (new_folio) + folio_put(new_folio); if (old_folio) { if (page_copied) free_swap_cache(&old_folio->page); @@ -3230,6 +3234,8 @@ vm_fault_t finish_mkwrite_fault(struct vm_fault *vmf) WARN_ON_ONCE(!(vmf->vma->vm_flags & VM_SHARED)); vmf->pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); + if (!vmf->pte) + return VM_FAULT_NOPAGE; /* * We might have raced with another page fault while we released the * pte_offset_map_lock. @@ -3591,10 +3597,11 @@ static vm_fault_t remove_device_exclusive_entry(struct vm_fault *vmf) vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); - if (likely(pte_same(*vmf->pte, vmf->orig_pte))) + if (likely(vmf->pte && pte_same(*vmf->pte, vmf->orig_pte))) restore_exclusive_pte(vma, vmf->page, vmf->address, vmf->pte); - pte_unmap_unlock(vmf->pte, vmf->ptl); + if (vmf->pte) + pte_unmap_unlock(vmf->pte, vmf->ptl); folio_unlock(folio); folio_put(folio); @@ -3625,6 +3632,8 @@ static vm_fault_t pte_marker_clear(struct vm_fault *vmf) { vmf->pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); + if (!vmf->pte) + return 0; /* * Be careful so that we will only recover a special uffd-wp pte into a * none pte. Otherwise it means the pte could have changed, so retry. @@ -3728,11 +3737,9 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) vmf->page = pfn_swap_entry_to_page(entry); vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); - if (unlikely(!pte_same(*vmf->pte, vmf->orig_pte))) { - spin_unlock(vmf->ptl); - goto out; - } - + if (unlikely(!vmf->pte || + !pte_same(*vmf->pte, vmf->orig_pte))) + goto unlock; /* * Get a page reference while we know the page can't be * freed. @@ -3807,7 +3814,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) */ vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); - if (likely(pte_same(*vmf->pte, vmf->orig_pte))) + if (likely(vmf->pte && pte_same(*vmf->pte, vmf->orig_pte))) ret = VM_FAULT_OOM; goto unlock; } @@ -3877,7 +3884,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) */ vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); - if (unlikely(!pte_same(*vmf->pte, vmf->orig_pte))) + if (unlikely(!vmf->pte || !pte_same(*vmf->pte, vmf->orig_pte))) goto out_nomap; if (unlikely(!folio_test_uptodate(folio))) { @@ -4003,13 +4010,15 @@ vm_fault_t do_swap_page(struct vm_fault *vmf) /* No need to invalidate - it was non-present before */ update_mmu_cache(vma, vmf->address, vmf->pte); unlock: - pte_unmap_unlock(vmf->pte, vmf->ptl); + if (vmf->pte) + pte_unmap_unlock(vmf->pte, vmf->ptl); out: if (si) put_swap_device(si); return ret; out_nomap: - pte_unmap_unlock(vmf->pte, vmf->ptl); + if (vmf->pte) + pte_unmap_unlock(vmf->pte, vmf->ptl); out_page: folio_unlock(folio); out_release: @@ -4041,22 +4050,12 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) return VM_FAULT_SIGBUS; /* - * Use pte_alloc() instead of pte_alloc_map(). We can't run - * pte_offset_map() on pmds where a huge pmd might be created - * from a different thread. - * - * pte_alloc_map() is safe to use under mmap_write_lock(mm) or when - * parallel threads are excluded by other means. - * - * Here we only have mmap_read_lock(mm). + * Use pte_alloc() instead of pte_alloc_map(), so that OOM can + * be distinguished from a transient failure of pte_offset_map(). */ if (pte_alloc(vma->vm_mm, vmf->pmd)) return VM_FAULT_OOM; - /* See comment in handle_pte_fault() */ - if (unlikely(pmd_trans_unstable(vmf->pmd))) - return 0; - /* Use the zero-page for reads */ if (!(vmf->flags & FAULT_FLAG_WRITE) && !mm_forbids_zeropage(vma->vm_mm)) { @@ -4064,6 +4063,8 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) vma->vm_page_prot)); vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); + if (!vmf->pte) + goto unlock; if (vmf_pte_changed(vmf)) { update_mmu_tlb(vma, vmf->address, vmf->pte); goto unlock; @@ -4104,6 +4105,8 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); + if (!vmf->pte) + goto release; if (vmf_pte_changed(vmf)) { update_mmu_tlb(vma, vmf->address, vmf->pte); goto release; @@ -4131,7 +4134,8 @@ static vm_fault_t do_anonymous_page(struct vm_fault *vmf) /* No need to invalidate - it was non-present before */ update_mmu_cache(vma, vmf->address, vmf->pte); unlock: - pte_unmap_unlock(vmf->pte, vmf->ptl); + if (vmf->pte) + pte_unmap_unlock(vmf->pte, vmf->ptl); return ret; release: folio_put(folio); @@ -4380,15 +4384,10 @@ vm_fault_t finish_fault(struct vm_fault *vmf) return VM_FAULT_OOM; } - /* - * See comment in handle_pte_fault() for how this scenario happens, we - * need to return NOPAGE so that we drop this page. - */ - if (pmd_devmap_trans_unstable(vmf->pmd)) - return VM_FAULT_NOPAGE; - vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, vmf->address, &vmf->ptl); + if (!vmf->pte) + return VM_FAULT_NOPAGE; /* Re-check under ptl */ if (likely(!vmf_pte_changed(vmf))) { @@ -4630,17 +4629,11 @@ static vm_fault_t do_fault(struct vm_fault *vmf) * The VMA was not fully populated on mmap() or missing VM_DONTEXPAND */ if (!vma->vm_ops->fault) { - /* - * If we find a migration pmd entry or a none pmd entry, which - * should never happen, return SIGBUS - */ - if (unlikely(!pmd_present(*vmf->pmd))) + vmf->pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd, + vmf->address, &vmf->ptl); + if (unlikely(!vmf->pte)) ret = VM_FAULT_SIGBUS; else { - vmf->pte = pte_offset_map_lock(vmf->vma->vm_mm, - vmf->pmd, - vmf->address, - &vmf->ptl); /* * Make sure this is not a temporary clearing of pte * by holding ptl and checking again. A R/M/W update @@ -5429,10 +5422,9 @@ int follow_pte(struct mm_struct *mm, unsigned long address, pmd = pmd_offset(pud, address); VM_BUG_ON(pmd_trans_huge(*pmd)); - if (pmd_none(*pmd) || unlikely(pmd_bad(*pmd))) - goto out; - ptep = pte_offset_map_lock(mm, pmd, address, ptlp); + if (!ptep) + goto out; if (!pte_present(*ptep)) goto unlock; *ptepp = ptep; From patchwork Mon May 22 05:26:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13249790 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AD956C7EE23 for ; Mon, 22 May 2023 05:26:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 49E716B0074; Mon, 22 May 2023 01:26:47 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 44E176B0078; Mon, 22 May 2023 01:26:47 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 31685900002; Mon, 22 May 2023 01:26:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 234596B0074 for ; Mon, 22 May 2023 01:26:47 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 00880140422 for ; Mon, 22 May 2023 05:26:46 +0000 (UTC) X-FDA: 80816756412.16.1A3E3E4 Received: from mail-yw1-f181.google.com (mail-yw1-f181.google.com [209.85.128.181]) by imf01.hostedemail.com (Postfix) with ESMTP id 2B34B40007 for ; Mon, 22 May 2023 05:26:44 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=pmFZ8eUZ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf01.hostedemail.com: domain of hughd@google.com designates 209.85.128.181 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684733205; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6V2+W0bna42qnZ43tvOf68yFS9Jt1uoq2b6WWLaZ+Rg=; b=J1Z8vTf9jIluS4X6HBplSqVhRowTG3oHIMWNFeie0zuwS2LVCKDjfptls8Jq5yM4QpbIfO SVVco9hlRmDhI4xoefp4wu+f8Q3BumfOJjG03idARoOKIT+TN0CX3fYVNLM+PCVWqJk+dk oZGzMcOJDw3/ZhvuymUoykFOLf2oNV4= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=pmFZ8eUZ; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf01.hostedemail.com: domain of hughd@google.com designates 209.85.128.181 as permitted sender) smtp.mailfrom=hughd@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684733205; a=rsa-sha256; cv=none; b=GR01mKwFIvwp5Y3QkArPoLazjbjCTK3e1DgyRkV2yzAiDYk1c1AVnCgFY+d7GXHqDRjkZv oPqPEXy936BsrVyR63oCI/Q5xDFfayv77OmwAMmBwclGDRoGujps0DjR/IehxVcY5qA6Yc oqYGAqrM/J4kLJhcac85KsFwT+G5P/A= Received: by mail-yw1-f181.google.com with SMTP id 00721157ae682-5619032c026so73114957b3.1 for ; Sun, 21 May 2023 22:26:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1684733204; x=1687325204; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=6V2+W0bna42qnZ43tvOf68yFS9Jt1uoq2b6WWLaZ+Rg=; b=pmFZ8eUZ5rWyUxNcjOAbP+PUX2zOF/4/kxaRJz4z7PKZCL9sh91Sp1fz2BDSpLA8va PshjIXySjEbJBMm+E5T0DV3GV1KfmGHUHW7NoxKfPJtzpKWDj4nZyySG6hAbDZeMssbe CpZvDb9v4tBFY4XeT7TABIe79HTh891UxQUA304GYCwmuhQf0Ddrievqmcn4LZJQZqo+ AyQ3f5iFkop7qEDUFMOD3A4f3L3NVlS89hjrz223HyzYGxeKrVNBksEbzG6kBQ75cK67 0udH2XswWGkcnuzDyqRllpE286zOHOwA3ju/MkmzjXbIYSmo/1Xbs4kh16I1DKzJVpox VJug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684733204; x=1687325204; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=6V2+W0bna42qnZ43tvOf68yFS9Jt1uoq2b6WWLaZ+Rg=; b=Nb4XTiI0wNhJ/t9qi3me6vWCyH9LV5sGlMzSP40HCqr4dH9t/GB0I6+g1yBXddL9Zq zykXspU5dn3yYDfgNHI/upd6l0FXl19SR0b0JnHN8QBwn2NshgXZGWEuwfA3eeba9vR+ mEswPfoPieNqOEZDJ8bNaRbaZMLmyVUSHIkaGnw3OHoymakXPqkt+Z5scn12ztWMFFBD 2f4BN8SMcNRZ2EmL827n62B3VD+87vMM2mcDRKMeMeWbZOjwJmQSTIsUh5tWnoxaaeJJ UAV71/FcWJQOQO/0brXZPFUM09ZxikPOIruxK3jDoqIkwWrtLS3/1BQfPhf170VeEMN/ eX7A== X-Gm-Message-State: AC+VfDy9fi3RRpnnEWJumQPYQGPI+nb2kby0dqlhCs8d4FhQOL54s1gv zaVaIfmDjRqhePVW88u1x+cI2A== X-Google-Smtp-Source: ACHHUZ6vQMBkQllrmMVkhqi7UHm68ru97kbfrpxbzO+74u3FaXGIYp1Xs+So1aiUfDsa50JYfZkYNw== X-Received: by 2002:a81:9383:0:b0:55a:7c7:c756 with SMTP id k125-20020a819383000000b0055a07c7c756mr11286138ywg.31.1684733204133; Sun, 21 May 2023 22:26:44 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id d206-20020a814fd7000000b0054605c23114sm1832452ywb.66.2023.05.21.22.26.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 21 May 2023 22:26:43 -0700 (PDT) Date: Sun, 21 May 2023 22:26:40 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 29/31] mm/memory: handle_pte_fault() use pte_offset_map_nolock() In-Reply-To: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> Message-ID: <5f10e87-c413-eb92-fc6-541e52c1f6be@google.com> References: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 2B34B40007 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: ambwq9t4eidhyhg4u1og65pa8tie5p98 X-HE-Tag: 1684733204-353796 X-HE-Meta: U2FsdGVkX1/RvORPsS96ore4RKLhILY5va2I2WBCl1I2d5bdAhIrumJ8YNZylcUuqtvFT7XKa1TWJmkxKhP0kti3KQpYza9hz03MjOC2IuYGAuSBHv6aLDu5nMWKemA78T1gSLKKE6lELQFyEX0A6OJvHK5WTIGSAEtpekDlhYDrWm8cskY/tO+utvTeUBt+xGRuRo5p5aXL3eEpeUKdalWEQGfsw/yUlD3BPsGA6OhF2kgeRRwnO57Bdxzb4DBv1GB1Icvyv58pWtdhoN8Gy+TpfNPE9REnzyxNvM2Qi5g1icjCf/yzx12+UyQ4dfXL/tenXdncPIvTEMRdi6ip2JKayljDUztEX3xBj6zL0zwgsxOtN3IYXY6W9mLwjFTA+5wvECEoMBrynnDA7hcV9w8T79xlpE15t9IRpWdOiiFmXYBxBBZpycyyvkCehzIqYMz5vikr0emQkHnP+z044I4jBDXD5bsvsaEsP2vCK9gqnd3HOKlHq73trgi37CMc2HARlKU8qKRS26XWM8gaSU9bou6I3nqlHyliL4PiDzNm04PJxgDbr7BgUlKeY2TdF8gY8fZEQIlGLDvUPRG2/dUMmu92lORIoG13+G1DwnDcIJS9lnsAzpMwlguJc1I6oYS7dFIvbZaWrV1YiAb4wcIvELzOc2NtAFf8nViZWx1zdDNyl0CHRICJAgMMAP75dvIF9Umvbuh/QHsU16HqG72piwJxpVhd4s0KozqEeEeWpwQuMqOEVCDsyrljg3Nf8jaqyZe8BoBnZhLpNejN3CHC5RIda5rlli0lNvNIDx9ujAIgoI9Ak6b/h3R5I+uY+uxCSgWLCq/73WK0Mk9Qm6RPSJqqZqj6NroqwrXN/l9mV0ipP54CQiVTlunuVligeF2DmzYN5WXSI03sFfetz7JlSBMs59ZAyJmjrkh4XXBF7V6t+yhKSJ/OV3nCe8mbCMSmJFBNJT9wgFzac9L zkrA+EnF AbIiKsHDGERaj7p9MEdKFy/KC/pgs4igGGbsPLVswqAvMUOLTiChUp/+VGqMmr5tr2HfOUlP8xmSr88nfO5KMCQoOGGefU5Bh5oS4NleKmG3MNt+10G5/wmjGll/tV2JoAI2KVAabxvrxU0Divh11Y6oMGsq9rNASdVscwgJlqSUUsSzT0rI0CeA+z72lmO9Vaf4ZfvkrEaxW+xJrHxuKJTHGvqtlxC7Jmw9wq3XrNf6YKkxh9z/9hVFuTkzwoQ34awBxc8jZmv5lsihIr/qyvKeMFEVtW1G7ztyUNv9r0Isqt111xOCBrmR2DH/LVn166r4lw2S6oMT6ShIIG1X/DKmhwaiHwmdRlq6BAsqfqx5KgNRSoGzoAr2OOd1FStavIimZ0BCpr2rftKQeZdPr902MYI3O1jTmwfxe3tF7MZAsYWzfKmFrqpP2Tg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: handle_pte_fault() use pte_offset_map_nolock() to get the vmf.ptl which corresponds to vmf.pte, instead of pte_lockptr() being used later, when there's a chance that the pmd entry might have changed, perhaps to none, or to a huge pmd, with no split ptlock in its struct page. Remove its pmd_devmap_trans_unstable() call: pte_offset_map_nolock() will handle that case by failing. Update the "morph" comment above, looking forward to when shmem or file collapse to THP may not take mmap_lock for write (or not at all). do_numa_page() use the vmf->ptl from handle_pte_fault() at first, but refresh it when refreshing vmf->pte. do_swap_page()'s pte_unmap_same() (the thing that takes ptl to verify a two-part PAE orig_pte) use the vmf->ptl from handle_pte_fault() too; but do_swap_page() is also used by anon THP's __collapse_huge_page_swapin(), so adjust that to set vmf->ptl by pte_offset_map_nolock(). Signed-off-by: Hugh Dickins --- mm/khugepaged.c | 6 ++++-- mm/memory.c | 38 +++++++++++++------------------------- 2 files changed, 17 insertions(+), 27 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 49cfa7cdfe93..c11db2e78e95 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1005,6 +1005,7 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm, unsigned long address, end = haddr + (HPAGE_PMD_NR * PAGE_SIZE); int result; pte_t *pte = NULL; + spinlock_t *ptl; for (address = haddr; address < end; address += PAGE_SIZE) { struct vm_fault vmf = { @@ -1016,7 +1017,7 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm, }; if (!pte++) { - pte = pte_offset_map(pmd, address); + pte = pte_offset_map_nolock(mm, pmd, address, &ptl); if (!pte) { mmap_read_unlock(mm); result = SCAN_PMD_NULL; @@ -1024,11 +1025,12 @@ static int __collapse_huge_page_swapin(struct mm_struct *mm, } } - vmf.orig_pte = *pte; + vmf.orig_pte = ptep_get_lockless(pte); if (!is_swap_pte(vmf.orig_pte)) continue; vmf.pte = pte; + vmf.ptl = ptl; ret = do_swap_page(&vmf); /* Which unmaps pte (after perhaps re-checking the entry) */ pte = NULL; diff --git a/mm/memory.c b/mm/memory.c index c7b920291a72..4ec46eecefd3 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -2786,10 +2786,9 @@ static inline int pte_unmap_same(struct vm_fault *vmf) int same = 1; #if defined(CONFIG_SMP) || defined(CONFIG_PREEMPTION) if (sizeof(pte_t) > sizeof(unsigned long)) { - spinlock_t *ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd); - spin_lock(ptl); + spin_lock(vmf->ptl); same = pte_same(*vmf->pte, vmf->orig_pte); - spin_unlock(ptl); + spin_unlock(vmf->ptl); } #endif pte_unmap(vmf->pte); @@ -4696,7 +4695,6 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) * validation through pte_unmap_same(). It's of NUMA type but * the pfn may be screwed if the read is non atomic. */ - vmf->ptl = pte_lockptr(vma->vm_mm, vmf->pmd); spin_lock(vmf->ptl); if (unlikely(!pte_same(*vmf->pte, vmf->orig_pte))) { pte_unmap_unlock(vmf->pte, vmf->ptl); @@ -4767,8 +4765,10 @@ static vm_fault_t do_numa_page(struct vm_fault *vmf) flags |= TNF_MIGRATED; } else { flags |= TNF_MIGRATE_FAIL; - vmf->pte = pte_offset_map(vmf->pmd, vmf->address); - spin_lock(vmf->ptl); + vmf->pte = pte_offset_map_lock(vma->vm_mm, vmf->pmd, + vmf->address, &vmf->ptl); + if (unlikely(!vmf->pte)) + goto out; if (unlikely(!pte_same(*vmf->pte, vmf->orig_pte))) { pte_unmap_unlock(vmf->pte, vmf->ptl); goto out; @@ -4897,27 +4897,16 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) vmf->pte = NULL; vmf->flags &= ~FAULT_FLAG_ORIG_PTE_VALID; } else { - /* - * If a huge pmd materialized under us just retry later. Use - * pmd_trans_unstable() via pmd_devmap_trans_unstable() instead - * of pmd_trans_huge() to ensure the pmd didn't become - * pmd_trans_huge under us and then back to pmd_none, as a - * result of MADV_DONTNEED running immediately after a huge pmd - * fault in a different thread of this mm, in turn leading to a - * misleading pmd_trans_huge() retval. All we have to ensure is - * that it is a regular pmd that we can walk with - * pte_offset_map() and we can do that through an atomic read - * in C, which is what pmd_trans_unstable() provides. - */ - if (pmd_devmap_trans_unstable(vmf->pmd)) - return 0; /* * A regular pmd is established and it can't morph into a huge - * pmd from under us anymore at this point because we hold the - * mmap_lock read mode and khugepaged takes it in write mode. - * So now it's safe to run pte_offset_map(). + * pmd by anon khugepaged, since that takes mmap_lock in write + * mode; but shmem or file collapse to THP could still morph + * it into a huge pmd: just retry later if so. */ - vmf->pte = pte_offset_map(vmf->pmd, vmf->address); + vmf->pte = pte_offset_map_nolock(vmf->vma->vm_mm, vmf->pmd, + vmf->address, &vmf->ptl); + if (unlikely(!vmf->pte)) + return 0; vmf->orig_pte = ptep_get_lockless(vmf->pte); vmf->flags |= FAULT_FLAG_ORIG_PTE_VALID; @@ -4936,7 +4925,6 @@ static vm_fault_t handle_pte_fault(struct vm_fault *vmf) if (pte_protnone(vmf->orig_pte) && vma_is_accessible(vmf->vma)) return do_numa_page(vmf); - vmf->ptl = pte_lockptr(vmf->vma->vm_mm, vmf->pmd); spin_lock(vmf->ptl); entry = vmf->orig_pte; if (unlikely(!pte_same(*vmf->pte, entry))) { From patchwork Mon May 22 05:27:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13249791 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E2669C7EE23 for ; Mon, 22 May 2023 05:28:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6E74E6B0074; Mon, 22 May 2023 01:28:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6973C6B0078; Mon, 22 May 2023 01:28:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5600A6B007B; Mon, 22 May 2023 01:28:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 4946E6B0074 for ; Mon, 22 May 2023 01:28:04 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id CBB491407C5 for ; Mon, 22 May 2023 05:28:03 +0000 (UTC) X-FDA: 80816759646.27.D89FC86 Received: from mail-yw1-f173.google.com (mail-yw1-f173.google.com [209.85.128.173]) by imf30.hostedemail.com (Postfix) with ESMTP id F3AC380002 for ; Mon, 22 May 2023 05:28:01 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=fVaNEVFF; spf=pass (imf30.hostedemail.com: domain of hughd@google.com designates 209.85.128.173 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684733282; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5QGRbISmNdiSrcRCSB3pJg16AgA7v+5ZDTS/B09Kn8o=; b=EavSP/oxVPYUvyg1cgSWTBVtzJw8fNHx7ZrGNrg06hVr0yaL2hDKC3RmvmcWyXkpe/9lzz pdv9VDLmWoL307iIzPVpJsufx1UNm7u8GMY4Ly/H/b1//7jwLKDd0TnWaHCF6aUfNBg1CP VKIGTrSaIr3rBMGu44Tq2rtgMBaPyV4= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684733282; a=rsa-sha256; cv=none; b=sKZ9hV+KZUpyn12BM40PWhU8fRyiuiVsYcdZB9+gkClF+EkD/G5Orvezkp/itqzUpFSM0P xFid6PZYHYa0xkyfo7YZO8R0e37e3WIvyTy+Xm3VrLLInYjrX/Ihfoz73PQvTEADRZPruX nhftU0MhOnxpWfALy1261+IgkaYp6Ns= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=fVaNEVFF; spf=pass (imf30.hostedemail.com: domain of hughd@google.com designates 209.85.128.173 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f173.google.com with SMTP id 00721157ae682-562191bcfb9so28191937b3.3 for ; Sun, 21 May 2023 22:28:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1684733281; x=1687325281; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=5QGRbISmNdiSrcRCSB3pJg16AgA7v+5ZDTS/B09Kn8o=; b=fVaNEVFFiBF4weUE59/pRbRilwRxpePXgogsf2ieUkcR0MbQmK9TAJMKwiY+Nl3wQv zKHe//Tgn/dCTvP7G0yKMDDCqiRigP7TN+cRzAhOy91u60ZDaLJj9pISGxGf7L/o9RQV RIdEDl0ysDGYdG9CQkCaefawPcW0Pu46VUAhGlWTpeI7kCMeXW2zSJd4jXBBPMxr+Sk7 BBLjIT+TptnFiPfiUV09WlYuflTUtcZ1khoOvaNMb55o7+CG56Ax+prHX5PMQDGLuiD5 5JiMmNrkUrTjybBbOntUDonvOeyTUvUndYAkHD0GpeLxqXMXoE3ar/hPCiH8QI5EkB9C IHZg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684733281; x=1687325281; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=5QGRbISmNdiSrcRCSB3pJg16AgA7v+5ZDTS/B09Kn8o=; b=iAE3650b4+waSPXHhMHiaRAt8/COuUAyFKBf3AzLlsTOOebz3vDvO0lXB2TnhoTcpM jK/11Tckvv0iWLbMT8aaiLmVXaIFfo07jRWEEeeZ6JzPOggKuvm5i5GFaokIL7cr8l3R 9z5g1zQtcqOh+eybDpHDrGKmI50DNhT9RrOzVOs1jotr5G8+VEZjOk8DDCFWOAfNe+F1 Z4C0dikAGmu7xpl9Z3jLMYyNgUkWTfsKRY6JAHn1mKfbZa/OsX1mociRqdLRLdGT8wD6 NhOXIWSH39ietQX4FbJe+6zQMT6smG+1PhGomo2Znk1BUJ2MzsXAWmpbYUT+W/wk1Z2I bFqA== X-Gm-Message-State: AC+VfDzLUrl2al9W8xrHJ61r4OwoTXxq7px64UjnMj+MJb2NhNUBlLFe UryGhDDEPc150JfAm8efeZJA0g== X-Google-Smtp-Source: ACHHUZ7e18kwUgmVMtp2lWAMCU5Id6EnMaAFL78obg4zT8QsdVgO5fiVzPuIAxqLocTFuTV77qh/ag== X-Received: by 2002:a81:a044:0:b0:561:e2df:c4d1 with SMTP id x65-20020a81a044000000b00561e2dfc4d1mr10124450ywg.9.1684733280842; Sun, 21 May 2023 22:28:00 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id k131-20020a816f89000000b0055a416529bbsm1831188ywc.24.2023.05.21.22.27.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 21 May 2023 22:28:00 -0700 (PDT) Date: Sun, 21 May 2023 22:27:57 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 30/31] mm/pgtable: delete pmd_trans_unstable() and friends In-Reply-To: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> Message-ID: References: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: F3AC380002 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: sdco4uthisfrww8wxyoa8bpxa1qji794 X-HE-Tag: 1684733281-269634 X-HE-Meta: U2FsdGVkX1+gLL+KUHp/vno0/Bcego8oU+ekc0NCUHKnEyytjt8ZV+isZVVlm3xm9uZgzafPWklJq8wPNJ3fsJIUrHLrar/08WgOkoJVPZcJ+RLqwQsSOdcNuloBtOvPsDYDBk2biWK72SWjT+mC7QboMferWrSC26tUy+hS/MTUP2q/ymLlbFU6x5T4sLOlxabnzDtOjWOhhlZR/Cog4sviGMRQKOGPmkJgyeU29CVIxRP5fRDKNrBhLkmqTXZPJzDiRCL2kTFl5cLGhZJAg3OJaA4uA3oNyYLeYUiotkLd95yX0cdVv7FIgqx7vUPL02L+ygx+ftVzg/WzMLrT7y5l09caI1TR+tVAvcYZXJuBAHIjElfOfwHH/er2wnes95hlAs3bzJpMk7lplbM22GImxZeNS4nFoWWywjMJOmw4K1hPu96vLFn6hO30MRK7hjIqBo8pIsOLYTxXcVNO5gONrUZuP80ed4ECe9KTVFVMeLEy5nRDzmC0LAFcHixlQABbwkrjzL/0oRGpgcmXRxiuOLpcxlU5L7F7KljKmaQU5xcRxDK0Amy9Z2eZKcsw8+VwA9ZkU3j5Jx/LybXyrNx5i3VNJEAArtIdAkgpnXgbKkntN22BYf8606W+jr2RibYLUOYaR9WSwjHng0CN+n8hwkfE4PqLXQ97LN4TSYRypyHWw2hDETvyh8yvWqvrw5roVj0U6NuJID8CdNBft7xIoxIuW9M+3oVl5VC8BTnIiGO12zpEld5qjDStaF7Pb4CR3ao440vV1LKlZa0LmawXiictaMtUgNtrIrE3cM7z49UpBZHRAxteJgIQQt8iIOhspj9PBUUz/ffzNXW4IH2FnupP0WhlVwCDWQkgNd1b1ClCawnYcheGqfvK+MnWFzyQeZsbxl7nCnA7wYzmJKw1+bgjVHuJ4VMdsgseypexhTjBlDNtvD6mN0WnxUIeklDRT61dyaD6dctk0ux Cybgvl50 CKajb0h+57x6gO6Z/te9lxT9JlC7qBvpBIPebWHIAmhp7XmF/ft3y54Pqv8MB+FR8rqQ7JoPd+R2nuNyreifF0zhmup7EZGbkB9byXx1SbJBkqA/eWxntf3gDfWDEe0jfRMziJ0Zw47IxZSWv5/p7GF8ttq1r11Zab7tiWtn5YGIdAn9mSgCAgWwNdWX9rDszfHyhP326+WAjr2bmm/HFLNixGJvll9OyrjCQbnb0L5TSMVipqq1qTJTu+gTrgxy83qOjFuVLR+SWW+pZYnwzH9EHU7LJjz3yh5Q3SSr3TkzhYrsys3uwa/TFsrKCqxwGg0onfwCJnhPyltnWnjXTEZxZXSDIX3YIJ8tkG+FGtvNTuw4kQB5kh2wnOK1LFs0TTUEMBQlEwrsCLVzvWY8zZENXQ4qUeKIGHSEgxr6FQZ7bfkD4XuGEdfN3NA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Delete pmd_trans_unstable, pmd_none_or_trans_huge_or_clear_bad() and pmd_devmap_trans_unstable(), all now unused. With mixed feelings, delete all the comments on pmd_trans_unstable(). That was very good documentation of a subtle state, and this series does not even eliminate that state: but rather, normalizes and extends it, asking pte_offset_map[_lock]() callers to anticipate failure, without regard for whether mmap_read_lock() or mmap_write_lock() is held. Retain pud_trans_unstable(), which has one use in __handle_mm_fault(), but delete its equivalent pud_none_or_trans_huge_or_dev_or_clear_bad(). While there, move the default arch_needs_pgtable_deposit() definition up near where pgtable_trans_huge_deposit() and withdraw() are declared. Signed-off-by: Hugh Dickins --- include/linux/pgtable.h | 103 +++------------------------------------- mm/khugepaged.c | 4 -- 2 files changed, 7 insertions(+), 100 deletions(-) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 3fabbb018557..a1326e61d7ee 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -599,6 +599,10 @@ extern void pgtable_trans_huge_deposit(struct mm_struct *mm, pmd_t *pmdp, extern pgtable_t pgtable_trans_huge_withdraw(struct mm_struct *mm, pmd_t *pmdp); #endif +#ifndef arch_needs_pgtable_deposit +#define arch_needs_pgtable_deposit() (false) +#endif + #ifdef CONFIG_TRANSPARENT_HUGEPAGE /* * This is an implementation of pmdp_establish() that is only suitable for an @@ -1300,9 +1304,10 @@ static inline int pud_trans_huge(pud_t pud) } #endif -/* See pmd_none_or_trans_huge_or_clear_bad for discussion. */ -static inline int pud_none_or_trans_huge_or_dev_or_clear_bad(pud_t *pud) +static inline int pud_trans_unstable(pud_t *pud) { +#if defined(CONFIG_TRANSPARENT_HUGEPAGE) && \ + defined(CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD) pud_t pudval = READ_ONCE(*pud); if (pud_none(pudval) || pud_trans_huge(pudval) || pud_devmap(pudval)) @@ -1311,104 +1316,10 @@ static inline int pud_none_or_trans_huge_or_dev_or_clear_bad(pud_t *pud) pud_clear_bad(pud); return 1; } - return 0; -} - -/* See pmd_trans_unstable for discussion. */ -static inline int pud_trans_unstable(pud_t *pud) -{ -#if defined(CONFIG_TRANSPARENT_HUGEPAGE) && \ - defined(CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE_PUD) - return pud_none_or_trans_huge_or_dev_or_clear_bad(pud); -#else - return 0; #endif -} - -#ifndef arch_needs_pgtable_deposit -#define arch_needs_pgtable_deposit() (false) -#endif -/* - * This function is meant to be used by sites walking pagetables with - * the mmap_lock held in read mode to protect against MADV_DONTNEED and - * transhuge page faults. MADV_DONTNEED can convert a transhuge pmd - * into a null pmd and the transhuge page fault can convert a null pmd - * into an hugepmd or into a regular pmd (if the hugepage allocation - * fails). While holding the mmap_lock in read mode the pmd becomes - * stable and stops changing under us only if it's not null and not a - * transhuge pmd. When those races occurs and this function makes a - * difference vs the standard pmd_none_or_clear_bad, the result is - * undefined so behaving like if the pmd was none is safe (because it - * can return none anyway). The compiler level barrier() is critically - * important to compute the two checks atomically on the same pmdval. - * - * For 32bit kernels with a 64bit large pmd_t this automatically takes - * care of reading the pmd atomically to avoid SMP race conditions - * against pmd_populate() when the mmap_lock is hold for reading by the - * caller (a special atomic read not done by "gcc" as in the generic - * version above, is also needed when THP is disabled because the page - * fault can populate the pmd from under us). - */ -static inline int pmd_none_or_trans_huge_or_clear_bad(pmd_t *pmd) -{ - pmd_t pmdval = pmdp_get_lockless(pmd); - /* - * !pmd_present() checks for pmd migration entries - * - * The complete check uses is_pmd_migration_entry() in linux/swapops.h - * But using that requires moving current function and pmd_trans_unstable() - * to linux/swapops.h to resolve dependency, which is too much code move. - * - * !pmd_present() is equivalent to is_pmd_migration_entry() currently, - * because !pmd_present() pages can only be under migration not swapped - * out. - * - * pmd_none() is preserved for future condition checks on pmd migration - * entries and not confusing with this function name, although it is - * redundant with !pmd_present(). - */ - if (pmd_none(pmdval) || pmd_trans_huge(pmdval) || - (IS_ENABLED(CONFIG_ARCH_ENABLE_THP_MIGRATION) && !pmd_present(pmdval))) - return 1; - if (unlikely(pmd_bad(pmdval))) { - pmd_clear_bad(pmd); - return 1; - } return 0; } -/* - * This is a noop if Transparent Hugepage Support is not built into - * the kernel. Otherwise it is equivalent to - * pmd_none_or_trans_huge_or_clear_bad(), and shall only be called in - * places that already verified the pmd is not none and they want to - * walk ptes while holding the mmap sem in read mode (write mode don't - * need this). If THP is not enabled, the pmd can't go away under the - * code even if MADV_DONTNEED runs, but if THP is enabled we need to - * run a pmd_trans_unstable before walking the ptes after - * split_huge_pmd returns (because it may have run when the pmd become - * null, but then a page fault can map in a THP and not a regular page). - */ -static inline int pmd_trans_unstable(pmd_t *pmd) -{ -#ifdef CONFIG_TRANSPARENT_HUGEPAGE - return pmd_none_or_trans_huge_or_clear_bad(pmd); -#else - return 0; -#endif -} - -/* - * the ordering of these checks is important for pmds with _page_devmap set. - * if we check pmd_trans_unstable() first we will trip the bad_pmd() check - * inside of pmd_none_or_trans_huge_or_clear_bad(). this will end up correctly - * returning 1 but not before it spams dmesg with the pmd_clear_bad() output. - */ -static inline int pmd_devmap_trans_unstable(pmd_t *pmd) -{ - return pmd_devmap(*pmd) || pmd_trans_unstable(pmd); -} - #ifndef CONFIG_NUMA_BALANCING /* * Technically a PTE can be PROTNONE even when not doing NUMA balancing but diff --git a/mm/khugepaged.c b/mm/khugepaged.c index c11db2e78e95..1083f0e38a07 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -946,10 +946,6 @@ static int hugepage_vma_revalidate(struct mm_struct *mm, unsigned long address, return SCAN_SUCCEED; } -/* - * See pmd_trans_unstable() for how the result may change out from - * underneath us, even if we hold mmap_lock in read. - */ static int find_pmd_or_thp_or_none(struct mm_struct *mm, unsigned long address, pmd_t **pmd) From patchwork Mon May 22 05:29:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13249792 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5FC20C77B75 for ; Mon, 22 May 2023 05:29:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 030946B0074; Mon, 22 May 2023 01:29:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id F23426B0078; Mon, 22 May 2023 01:29:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DEC41900002; Mon, 22 May 2023 01:29:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id D16096B0074 for ; Mon, 22 May 2023 01:29:13 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 96A2FC0420 for ; Mon, 22 May 2023 05:29:13 +0000 (UTC) X-FDA: 80816762586.28.C9B9424 Received: from mail-yw1-f171.google.com (mail-yw1-f171.google.com [209.85.128.171]) by imf12.hostedemail.com (Postfix) with ESMTP id A9C3340016 for ; Mon, 22 May 2023 05:29:11 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=KUYquKPQ; spf=pass (imf12.hostedemail.com: domain of hughd@google.com designates 209.85.128.171 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684733351; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=++qSmbn7bhjg3RT15iglmwv4R1gESGfbvlyrc9z9nPg=; b=8K7SYK09D+65Qq/GoJDKnaF4IkB3DdN4oEkHJd/PvJoH6hcnhBSOgWkL/hPdFDV5nJiMeL jqHFxp1wxco2GCoqWwNHzMiwJD4kHTPP+YGkLyYB0q0HfgfW1vQhhrH6YBOM66BpMh4Ryu Jsu9ZQwtQANmL4Dd/b1Iyby1ycOstYE= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684733351; a=rsa-sha256; cv=none; b=e8fG9fnE8ddHubA6ORYbGImte2cxbSxbmgfJ2YAL3bsFab0Y9TW9loy+OPjudtgi+bio75 W1UTq9TQtoZEVt02W5p9W1Pysmf8HqmjiQe44mzcyFziH7YeJ/vxPc6jwFzgT7pcT+mRNI 6DUkP5NsGJGGxY/bryK9aMMvi66vOLM= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=google.com header.s=20221208 header.b=KUYquKPQ; spf=pass (imf12.hostedemail.com: domain of hughd@google.com designates 209.85.128.171 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f171.google.com with SMTP id 00721157ae682-561b7729a12so45910687b3.1 for ; Sun, 21 May 2023 22:29:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1684733351; x=1687325351; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=++qSmbn7bhjg3RT15iglmwv4R1gESGfbvlyrc9z9nPg=; b=KUYquKPQFLz686sxc05IvSA2pptVmZo25juuDJPh5PU0X8mcUoMX85X6dQWcuCeLiy uZgK8UCVFiR0+JLDfI+BG2fZ4jpaUFacf9y4wexEGbBLRrMU7EGQ2WyCANVks/wifwjW WIWR1uH1YZ2eCpYGacQpMGZqDG1Rast16Vhqw6loQySVQm459lCNaPM2V4pX8ejuYDHJ iWokAPVtiNyJ1VEM+GanV1g70rryZ2Zn0pEeEaFvrZDQKTZaXrD5lh2IK6/6AvLj649e dxMB/Rriy/3qD/arXkivaOaOUE1co2k6CyJ03pNexh99nj5a3n2SbOmvYztGkHfCQKXx DWrw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684733351; x=1687325351; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=++qSmbn7bhjg3RT15iglmwv4R1gESGfbvlyrc9z9nPg=; b=Ywx2540gN8AGInE70AqTTJtBj01ejB/3NnO4XAX2iNUMLLRw29VrJqEBHcohc0FtSi 70id3qH/4oesfGfeUKx7tbVKzMQ1/CgkYqXjm3/NWJoK0Y/hubC5fEzC+rAipRfZUYPN CCADpLOXf8cDPcp0zbDVW5LXlf4vrqZFlzGAUXqXej/mAWOdeZNsfs08vqLLcHVtBoTS cnupKFsOct3c/++dwJ+o24gbkMHONqmf1GoJEI2KhZvwSHNMsw+cRUkPzCQvob1mFXHj e+Yxne89BKGQ5xXchD6scjwrP3gJtwLjFrrL1ZEMhLeRF3ARrdHJfFC1x+3kweSlqvCH Jgcw== X-Gm-Message-State: AC+VfDzYhf52z11F5JqJz4STpd/X2XxNasGgW/ORQtlq6lEyhkvBdl0b c6P+aUVnr1bgSwVRPoRRMU+5FQ== X-Google-Smtp-Source: ACHHUZ58DrOuEAqS21o31T6NkFTUrmPfkTQChFezlC+fyR5u9eV8KPIJhLKCfEeaQKdNd4WYmfpBlw== X-Received: by 2002:a0d:cac7:0:b0:55a:2084:9e05 with SMTP id m190-20020a0dcac7000000b0055a20849e05mr10730696ywd.23.1684733350763; Sun, 21 May 2023 22:29:10 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id x67-20020a81a046000000b0054fcbf35b94sm1832465ywg.87.2023.05.21.22.29.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 21 May 2023 22:29:10 -0700 (PDT) Date: Sun, 21 May 2023 22:29:07 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Mike Kravetz , Mike Rapoport , "Kirill A. Shutemov" , Matthew Wilcox , David Hildenbrand , Suren Baghdasaryan , Qi Zheng , Yang Shi , Mel Gorman , Peter Xu , Peter Zijlstra , Will Deacon , Yu Zhao , Alistair Popple , Ralph Campbell , Ira Weiny , Steven Price , SeongJae Park , Naoya Horiguchi , Christophe Leroy , Zack Rusin , Jason Gunthorpe , Axel Rasmussen , Anshuman Khandual , Pasha Tatashin , Miaohe Lin , Minchan Kim , Christoph Hellwig , Song Liu , Thomas Hellstrom , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 31/31] perf/core: Allow pte_offset_map() to fail In-Reply-To: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> Message-ID: References: <68a97fbe-5c1e-7ac6-72c-7b9c6290b370@google.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: A9C3340016 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: hnttjr6s1b3ozp4zf4f4zhwrmprw4fwx X-HE-Tag: 1684733351-711623 X-HE-Meta: U2FsdGVkX1/FipF5295d3bBuogxsURVGDKM+LMCcvGeS8Zmi71it0mLOdxF9oOixr9WlCwwvZeQWUR4vAI7igGb37SYTq22jejHJ2Ow7m7Kv5nMM5sFQhYqDQPD2ZIDSQxukx+addmQZZHb/a3+WI4rzhFFop5gcc0TV5NDLgvOP7yEI0W7itROidEuTJcv0gIbT7xcWNaif9wp9cBmT5J4uwHtsu2ygkgNeUuKEtDGzy0l4MLaa5Qztp61gF+tmFz7nGtEBx9RUUzbXG7VgAYCoZsfE6MV5WmSEQnhao+BYCVg6HA4Nsaf1nDOeYC8odBkqEybHXa4c6wVDlejQlmUNVyceXpnRs3p/eDdgIYfMDvGPAavWH86A/RiFEd2mlYs3rFL9MclGmVKrFnDvpXRS9Vzs9kZT34mqpa2SO9s8zaVf/uTumJJ6ywt4RiSHeCqpkIsCPvJEJrxGLLTIXEL2HjqMJR7TfSu65oi+7TlgVOPdiyEflP9hDT4wWDN6A1F/pd6Vz/0GNBnruBg70ngp/5IEZCHRCYcwHgBdyKuP27L5Opi2w8gWmJnw7/JZyeKldv1FpD+JKtX/ZoEcOH0EW6gcQJayDRSCIeU3W1jieWwD6nSa5IELVktSGv44/SxUEsGYAw0CeD+MK9E0Bv/7E8kRS4g0UdtxeIEJE59Rxg6bdh4nNINJwduQKLaAw86+k3jKgHTLXMNSO8gjhp7VPVRl34kBsy/iL3p2JRY3FpVwkgaVhAC2LSAZwhwqvqrCbw50z9WRNkSTcvYSowElkBMypFqp4c5AYTeiYIu+9aLJC9h8qWZ78mdtUDRrd04/dMtXanRs7uLFm7MbxgciUYG8uPGMtUG8yn07O2qPyLx4/zFK4vKvFrCQY0FOSNHghWLhfSHxnAH4Kca39oSmnCcMBRuG+HBhNmR8jILHISX2wz1tT/AAyq27zRXJmVym5HFUiLx3uBV/qKp PD9+GzHF j78QwDehSJR84bNaulOajC2CAvshLZu4mqjLAyy6bR4Pi34VOszKGIdEClI1WS7pajZSHsO770nQ3bZy90bI5u1xhp71m8ypW346WlidCdFn9oFY5zk8I5xvyZOhN02X4T+PYp4JeWOMhK/z+XHn0WJecDYvhGvSwJDF/D1fu89mT5w8L5syFNqSSdAS9shA8/qTWHSRAR3njBX3yNMTvL1P94cpB1ITri5BMUt55Fj/2D6p+FcFN36IoHeEL6b9RHfJ8FTcyJYdgZFFI9FhdmvdHOKMUXk0BYGOCG4OCJVDtwUqOgrLtDn0QuCI6Dix2MkcT0jQ5skP7qgY93Jcpfa6D/foUdaHb10oOylu9DWRcH36no6ViTxl/eIG+jSxo31GG0z1bSL0pTbUJ6945VAyQas2NGWhOmwiIeF0Aq8oF/My0X1WYT4Wuew== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In rare transient cases, not yet made possible, pte_offset_map() and pte_offet_map_lock() may not find a page table: handle appropriately. Signed-off-by: Hugh Dickins --- This is a perf patch, not an mm patch, and it will want to go in through the tip tree in due course; but keep it in this series for now, so that it's not missed, and not submitted before mm review. kernel/events/core.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/kernel/events/core.c b/kernel/events/core.c index db016e418931..174be710f3b3 100644 --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -7490,6 +7490,7 @@ static u64 perf_get_pgtable_size(struct mm_struct *mm, unsigned long addr) return pud_leaf_size(pud); pmdp = pmd_offset_lockless(pudp, pud, addr); +again: pmd = pmdp_get_lockless(pmdp); if (!pmd_present(pmd)) return 0; @@ -7498,6 +7499,9 @@ static u64 perf_get_pgtable_size(struct mm_struct *mm, unsigned long addr) return pmd_leaf_size(pmd); ptep = pte_offset_map(&pmd, addr); + if (!ptep) + goto again; + pte = ptep_get_lockless(ptep); if (pte_present(pte)) size = pte_leaf_size(pte);